\section{Introduction}

In the process of creating a neural network binary classifier, the network is often trained via the binary cross-entropy (BCE) objective function. The network's output is a probability value $p \in [0,1]$ that must be translated into a binary value $\{0,1\}$ indicating set membership to the positive class. The Heaviside step function $H$ is commonly used to determine set membership based on a heuristic threshold $\tau$, where $p \geq \tau$ are considered positive classification outcomes.

It is a common assumption that optimizing a desired evaluation metric is preferable to optimizing a different loss \cite{eban2016scalable, Rezatofighi_2019_CVPR, herschtal2004optimising, song2016training}. However, the Heaviside function, commonly used to compute confusion matrix set membership in terms of true positives, false positives, false negatives, true negatives, has a gradient with properties not conducive to optimization via gradient descent. The Heaviside function’s gradient is not defined at the threshold $\tau$ and is zero everywhere else (see Fig. \ref{fig:heaviside1} (left) for an example with $\tau = 0.7$). Therefore the gradient of the Heaviside function cannot be used to effectively back propagate errors and optimize a neural network according to metrics composed of confusion matrix set values, such as $F_1$-Score. To address this challenge, we propose a novel approximation of the Heaviside function that adheres to important properties of the original Heaviside function while allowing for optimization via gradient descent. Figure \ref{fig:heaviside2} (left) depicts our proposed approximation.

\begin{figure}[tb!p]
    \centering
    \begin{minipage}{.25\textwidth}
        \centering
        \includegraphics[width=0.7\linewidth]{plots/heaviside/heaviside.pdf}
    \end{minipage}%
    \begin{minipage}{0.25\textwidth}
        \centering
        \includegraphics[width=0.7\linewidth]{plots/heaviside/heaviside_sigmoid_approx.pdf}
    \end{minipage}
    \caption{Heaviside function $H$ (left) and \cite{kyurkchiev2015sigmoid} approximation (right) at $\tau=0.7$.}
    \label{fig:heaviside1}
    \vspace{-0.5em}
\end{figure}

\begin{figure}[tb!p]
    \centering
    \begin{minipage}{0.25\textwidth}
        \centering
        \includegraphics[width=0.7\linewidth]{plots/heaviside/heaviside_linear_approx_labeled.pdf}
    \end{minipage}%
    \begin{minipage}{0.25\textwidth}
        \centering
        \includegraphics[width=0.7\linewidth]{plots/heaviside/heaviside_sigmoid_fit_approx.pdf}
    \end{minipage}
    \caption{Proposed Heaviside function approximation $\mathcal{H}$ (left) and a sigmoid fit to our proposed Heaviside approximation (right) at $\tau=0.7$.}
    \label{fig:heaviside2}
    \vspace{-0.5em}
\end{figure}

The proposed approximation is flexible in that it can be used to directly optimize any metrics based on the confusion matrix sets without re-formulating the loss. One may simply substitute the step function with the proposed Heaviside approximation and directly optimize the metric. This flexibility is important because the optimal metric for binary classifiers often changes with the level of class imbalance of the data. For example, while accuracy is popular for evaluating classifiers, accuracy will over-estimate performance \cite{he2009learning} in datasets with high levels of class imbalance. In these cases, better estimators of performance include the harmonic mean of precision and recall ($F_\beta$-Score).

Though metrics based on confusion matrix set values are considered not decomposable, we find that when trained with an adequately large batch size our method performs as well as or better than the commonly used BCE objective function and other methods of directly optimizing the metric. Though this requires use of a large-enough batch size during training, we find these batch sizes to be well within limits of modern deep learning hardware when training on a variety of publicly available datasets. These datasets have varying levels of class balance, reflecting the fact that class imbalance is common in real-world data \cite{reed2001pareto}.

Our approach is scalable with the runtime equivalent to the metric's computational cost. Even for metrics that require computation over multiple thresholds, such as area under the receiver operating characteristic (AUROC) curve, we suggest a constant-time algorithm which ensures training runtime scales linearly with the number of samples in the dataset. Moreover, the proposed method places no limit on the maximum size of the batch used for training, as is the case in the adversarial approach proposed by \cite{fathony2020ap}.

In summary, our main contribution is a novel approximation of the Heaviside function to facilitate batch gradient descent when training classifiers that directly optimize evaluation metrics based on confusion-matrix sets. We explore the practical effects of batch size on non-decomposable metrics in light of our approach. Also, we show the flexibility of the approach by applying our direct optimization method to several confusion-matrix based metrics on various datasets





\section{Preliminaries}
\label{sec:preliminaries}

Once a binary classification network is trained, it is common to choose a threshold value $\tau$ to transform the output probability to a binary value. The threshold is often chosen by searching for an optimal value considering the carnality of the sets in the confusion matrix ($|TP|$, $|FP|$, $|FN|$, $|TN|$). These sets are computed using the Heaviside function $H$:
\begin{equation}
H(p, \tau) =
  \begin{cases}
    1 & p \geq \tau \\
    0 & p < \tau
  \end{cases}
\end{equation}

There are several important properties of the Heaviside function that must be considered when evaluating approximations. First, the complement of $H$,  $1-H$, is the reflection over the line $y=0.5$. Second, at the threshold $p=\tau$, the sample could be a member of either class with equal probability. In practice, to avoid ambiguity either the positive or negative class is chosen -- we chose the positive class in this work. Figure \ref{fig:heaviside1} (left) shows an example Heaviside function.

The confusion matrix set membership can then be computed for a prediction $p$ and for ground truth label $y$ via the set membership functions:
{\small
\begin{center}
    \noindent\begin{minipage}{.44\linewidth}
        $$
            tp = 
              \begin{cases}
                H(p, \tau) & y = 1 \\
                0 & \text{else}
              \end{cases}
        $$
    \end{minipage}%
    \begin{minipage}{.56\linewidth}
        $$
            fp = 
              \begin{cases}
                H(p, \tau) & y = 0 \\
                0 & \text{else}
              \end{cases}
        $$
    \end{minipage}
\end{center}
\begin{center}
    \noindent\begin{minipage}{.5\linewidth}
        $$
            fn = 
              \begin{cases}
                1-H(p, \tau) & y = 1 \\
                0 & \text{else}
              \end{cases}
        $$
    \end{minipage}%
    \begin{minipage}{.5\linewidth}
        $$
            tn = 
              \begin{cases}
                1-H(p, \tau) & y = 0 \\
                0 & \text{else}
              \end{cases}
        $$
    \end{minipage}
\end{center}
}

Consider a set of predictions $p \in [0,1]$, ground truth labels $y \in \{0,1\}$ and threshold value $\tau \in (0,1)$, e.g. $\{(p_1, y_1, \tau), (p_2, y_2, \tau), \cdots, (p_n, y_n, \tau)\}$. The cardinality of each confusion matrix set is computed as the sum over the set membership functions:
{\small
\begin{center}
    \noindent\begin{minipage}{.5\linewidth}
        $$
            |TP| = \sum_{i=1}^n tp(p_i, y_i, \tau)
        $$
    \end{minipage}%
    \begin{minipage}{.5\linewidth}
        $$
            |FP| = \sum_{i=1}^n fp(p_i, y_i, \tau)
        $$
    \end{minipage}
\end{center}
\begin{center}
    \noindent\begin{minipage}{.5\linewidth}
        $$
            |FN| = \sum_{i=1}^n fn(p_i, y_i, \tau)
        $$
    \end{minipage}%
    \begin{minipage}{.5\linewidth}
        $$
            |TN| = \sum_{i=1}^n tn(p_i, y_i, \tau)
        $$
    \end{minipage}
\end{center}
}
Common classification metrics based on the confusion matrix then build on the above cardinality sets. Two examples of these metrics are precision and recall. Precision, $|TP|/(|TP|+|FP|)$, is the proportion of positive results that are true positive results. Recall, $|TP|/(|TP|+|FN|)$, indicates the proportion of actual positives examples that are correctly identified.

Precision and recall represent a trade-off between classifier objectives; it is generally undesirable to optimize or evaluate for one while ignoring the other \cite{he2009learning}. This makes summary evaluation metrics that balance between these trade-offs popular. Commonly used objectives include accuracy and $F_1$-Score:
{\small
$$
\text{accuracy} = \frac{|TP|+|TN|}{|TP|+|TN|+|FP|+|FN|}
$$
$$
F_1\text{-Score}= \frac{2}{\text{precision}^{-1} + \text{recall}^{-1}}
$$
}
Accuracy is the rate of correct predictions to all predictions. $F_1$-Score is a specific instance of $F_\beta\text{-Score}=(1+\beta^2)\cdot(\text{precision} \cdot \text{recall})/(\beta^2 \cdot \text{precision}^{-1} + \text{recall}^{-1})$, which is the weighted harmonic mean of precision and recall. The parameter $\beta$ is chosen such that recall is considered $\beta$ times more important than precision.

When a binary classification neural network is trained in parts, independently of $\tau$ and the chosen metric, the network is unaware of the end goal and therefore is unlikely to optimize for the desired evaluation metric. This limitation often becomes evident when the goal is to balance between various confusion matrix set values, e.g.,  $F_\beta\text{-Score}$.  

Prior works have approximated the Heaviside function. For instance,  \cite{kyurkchiev2015sigmoid} proposed using a sigmoid function, as shown in Figure \ref{fig:heaviside1} (right). However, this prior work focused on minimizing the error between the approximation and $H$, not on a Heaviside approximation useful for back-propagation. Such sigmoid-based approximations do not adhere to the properties necessary for gradient descent. For example, the Heaviside approximation proposed in \cite{kyurkchiev2015sigmoid}, $s_0(k;p) = (1+e^{-kp})^{-1}$ can be reparameterized to account for $\tau$: $s_{\tau}(k;p) = (1+e^{-k(p-\tau)})^{-1}$. While this approximation approaches $H$ at the limit when $\tau=0.5$, with an appropriate value of $k$, it does not necessarily approach $H$ at the limit when $\tau$ nears the limits, for a given value of $k$. Moreover, with increasing values of $k$, while the error between $s_{\tau}$ and $H$ decreases, the range over which the derivative $s'_{\tau}=0$ increases. A non-zero gradient is important during training so the network can learn from similar samples belonging to different classes. Figure \ref{fig:heaviside1} (right) shows the \cite{kyurkchiev2015sigmoid} approximation. Our proposed approximation ensures, in addition to the above mentioned properties of  $H(p=\tau)=0.5$ and reflection over $y=0.5$, that the  gradient is non-zero and $\mathcal{H}$ approaches $H$ at the limit for any threshold $\tau$.





\section{Heaviside Piecewise Linear Approximation}
\label{sec:method}

To approximate the Heaviside function in a simple manner while adhering to the important properties noted in the prior section, we propose a five-point linearly interpolated function as shown in Figure \ref{fig:heaviside2} (left). Our approximation $\mathcal{H}$ is defined over $[0, 1]$ and is parameterized by a given threshold $\tau$ and a slope parameter $\delta$, which define three linear segments with slopes: $m_1$, $m_2$, and $m_3$. Let $\tau_m = \text{min} \{\tau, 1-\tau\}$ in order to ensure a gradient that adheres to properties desirable for optimization via back propagation. Then, the slope of each line segment is defined as:
\begin{center}
    \noindent\begin{minipage}{.33\linewidth}
    \centering
        $$
        m_1 = \frac{\delta}{\tau-\frac{\tau_m}{2}}
        $$
    \end{minipage}%
    \noindent\begin{minipage}{.33\linewidth}
    \centering
        $$
        m_2 = \frac{1-2\delta}{\tau_m}
        $$
    \end{minipage}%
    \noindent\begin{minipage}{.33\linewidth}
    \centering
        $$
        m_3 = \frac{\delta}{1-\tau-\frac{\tau_m}{2}}
        $$
    \end{minipage}
\end{center}
The Heaviside approximation is thus given by:
{\small
\begin{equation}
\mathcal{H} =
  \begin{cases}
    p \cdot m_1 & \text{if $p < \tau - \frac{\tau_m}{2}$} \\
    p \cdot m_3 +(1-\delta-m_3(\tau+\frac{\tau_m}{2})) & \text{if $p > \tau + \frac{\tau_m}{2}$} \\
    p \cdot m_2 + (0.5 - m_2\tau) & \text{otherwise}
  \end{cases}
\end{equation}
}
The above equation can be trivially derived by computing the slope and intercept for each segment $m_1, m_2, m_3$. Slope-intercept form is shown for clarity.

Including the threshold $\tau$ in our parameterization of the approximation is crucial in that many evaluation metrics compute confusion matrix set cardinality based on a given threshold value or across a range of thresholds to effectively evaluate classifier performance. For example, $F_1$-Score is usually computed at a specific threshold whereas area under the receiver operating characteristic curve (AUROC) is computed over a range of $\tau$ values. AUROC is a commonly used ranking performance measure defined in terms of the true positive rate (TPR) and false positive rate (FPR) as $\text{AUROC}=\int_{\tau=0}^1 \text{TPR}(\text{FPR}^{-1}(\tau)) d\tau$. Our approximation allows for the computation of either type of metric.

We define the slope parameter $\delta$ of $\mathcal{H}$ on the interval $(0,0.5)$. A larger value of $\delta$ providers a smoother derivative but further approximation from the Heaviside function. In practice, we found that this parameter was not crucial in our experiments and that a value of $\delta = 0.1$ works well. In the next sections we abuse notation for simplicity and denote our approximation as $\mathcal{H}(p, \tau)$, although it also depends on $\delta$.

The proposed approximation $\mathcal{H}$ adheres to the important properties of the Heaviside function including reflection over $y=0.5$ and $\mathcal{H}(p=\tau)=0.5$ as well as the properties necessary for effective back-propagation. At the limits, our approximation $\mathcal{H}$ approaches the Heaviside function $H$,
$$
\lim_{p \to \{0, 1\}} \mathcal{H}(p,\tau) = H(p,\tau) \hspace{1em} \forall \, \tau
$$
Our formulation also ensures that the derivative of $\mathcal{H}$ is non-zero: $\mathcal{H}'(p, \tau) \neq 0 \ ,\ \forall \, \tau$.

\subsection{Soft Set Membership with the Approximation}

We define soft set membership functions via the Heaviside approximation. Intuitively, the idea is to output a value proportional to the input prediction value and closer to $1$ when the prediction is a set member, and a value closer to $0$ in all other cases. In the negative cases where the set membership function yields the complement, we use $1-\mathcal{H}(p, \tau)$. Formally, we define the soft set membership functions, where $s$ is for ``soft," $tp_s$, $fp_s$, $fn_s$, and $tn_s$ based on the truth tables for each confusion matrix set:
\begin{align*}
tp_s(p, y, \tau) &= 
  \begin{cases}
    \mathcal{H}(p,\tau) & y = 1 \text{ or } p < \tau \\
    1-\mathcal{H}(p,\tau) & \text{else}
  \end{cases}\\
fp_s(p, y, \tau) &= 
  \begin{cases}
    \mathcal{H}(p,\tau) & y = 0 \text{ or } p < \tau \\
    1-\mathcal{H}(p,\tau) & \text{else}
  \end{cases}\\
fn_s(p, y, \tau) &= 
  \begin{cases}
    1-\mathcal{H}(p,\tau) & y = 1 \text{ or } p \geq \tau \\
    \mathcal{H}(p,\tau) & \text{else}
  \end{cases}\\
tn_s(p, y, \tau) &= 
  \begin{cases}
    1-\mathcal{H}(p,\tau) & y = 0 \text{ or } p \geq \tau \\
    \mathcal{H}(p,\tau) & \text{else}
  \end{cases}
\end{align*}
The properties of $\mathcal{H}$ ensure soft set membership weight is equally close to $1$ in the positive case as it is close to $0$ in the negative case. Positive or negative predictions should not disproportionately skew soft set membership magnitude.

Metrics such as accuracy and $F_1$-Score that combine cardinality values from the confusion matrix sets can then be approximated using the relevant soft set membership functions. For instance, we approximate precision using  $|TP_s| = \sum_{i=1}^n tp_s(p_i, y_i, \tau)$ and $|FP_s| = \sum_{i=1}^n fp_s(p_i, y_i, \tau)$. In practice, we sum these values at training time over members of mini-batches while optimizing via gradient descent. As gradient descent and its variants expect a small but representative sample of the broader data \cite{bottou2012stochastic}, the proposed method also expects a representative sample.

\subsection{Metrics and Losses}

Any metric composed of combinations of confusion matrix set values ($TP$, $FP$, $FN$, and $TN$) can be directly optimized using the proposed Heaviside approximation. In our experiments, we train and evaluate on accuracy, $F_1$-Score, AUROC, and $F_\beta$-Score  \cite{Rijsbergen}. Accuracy is a reasonable measure for balanced datasets. In the case of imbalanced data, accuracy tends to over-estimate classifier performance, easily maximizing performance by learning to predict only dominant-class samples achieving accuracy proportional to class imbalance \cite{he2009learning}. Popular alternative metrics for accuracy when dealing with imbalanced data include $F_\beta$-score and AUROC. Our experiments demonstrate that our proposed Heaviside approximation can be used to train a classifier with either objective using the soft membership functions.

\subsection{Computational Efficiency}

At training time, a loss that utilizes our proposed Heaviside approximation has a runtime linear with regard to the number of samples. In practice however, optimizing for a metric with $\mathcal{H}(p, \tau)$ over all samples in each batch could lead to increased run time due to the number of constant-time operations required to compute a metric. To mitigate this and further minimize run time, we observe that $\mathcal{H}(p, \tau)$ can be replaced with an reasonably-sized $O(1)$ lookup table by truncating $p$ to several decimal places and precomputing $\mathcal{H}$ for values of $p$ and $\tau$ over the range $[0,1]$. For example, if an interval between values of $\tau$ is set to $0.1$, as was done in our experiments, and $p$ is truncated at two decimal places, the lookup table has only $1,000$ elements. Using 8-bit storage, this table -- which only needs to be computed once -- consumes $1$kB of memory.





\section{Experiments}

This section presents four experiments to (1) understand the effect of different batch sizes in the proposed approach, (2) compare our proposed Heaviside approximation with a sigmoid approximation in the context of binary classification, (3) evaluate the performance of our approach against optimizing for other relevant classification metrics, and (4) evaluate optimizing directly for $F_\beta$-Score with our Heaviside approximation in order to balance precision and recall.

\subsection{Architecture and Training}

Unless otherwise noted, all experiments use the same network architecture and training scheme. The binary classifier is a feedforward neural network classifier consisting of three fully connected layers of 32 units, 16 units, and 1 unit. The first two layers have a Rectified Linear Unit (ReLU) activation \cite{nair2010rectified} and are followed by dropout \cite{hinton2012improving}. The final layer is a sigmoid-activated single-unit output. We use the ADAM optimizer \cite{kingma2014adam} with a learning rate of $0.001$ and a batch size of 2048. The network is trained using early stopping, monitoring validation loss for improvement over a sliding window of 100 epochs.
The maximum number of training iterations is set to 5000, a number high enough such that early stopping takes effect before finishing all epochs. The models for all losses were trained with TensorFlow, except for \cite{fathony2020ap} and AUROC via the proposed Heaviside approximation which were implemented in PyTorch. Note that we provide open source implementations of our proposed approach for both deep learning libraries. 

Training systems used either an NVIDIA Titan X or RTX 2080ti GPU, Intel i7 3.7GHz processor, and 32GB of RAM.

\subsubsection{Datasets}

Experiments were conducted on two synthetic datasets (Section \ref{sig-vs-lin-approx}) and four publicly available datasets with binary classification labels. The latter datasets were chosen for their varying levels of class imbalance in a variety of domains, as described in Section \ref{direct-optimization-experiment}. All of the datasets were minimally pre-processed, the details of which are described in the supplementary material. Features were centered and scaled to unit variance, and the data was split into separate train (64\%), test (20\%) and validation (16\%) sets. 
\subsection{Batch Size Effects}

Our first experiment explored the effects of batch size on approximating non-decomposable metrics, such as $F_1$-Score, for classification of microcalcifications in the Mammography dataset \cite{woods1993comparative}. The dataset is imbalanced, having only 2.32\% positive-class examples as detailed in the supplementary. 

Throughout training, $F_1$-Score was computed against both the current batch of the train split as well as the entire dataset using the network undergoing training. The difference between the batch $F_1$-Score and the $F_1$-Score of the entire dataset was then aggregated. Finally, we computed the absolute mean and standard deviation for the difference over all batches used during training.
\begin{table}[tb]
    \centering
    {\small
        \begin{tabular}{c|c|c|c}
            $B=128$ & $B=1024$ & $B=2048$ & $B=4096$\\ \hline
            $0.041\pm0.23$ & $0.005\pm0.06$ & $0.009\pm0.05$ & $0.006\pm0.03$
        \end{tabular}
    }
    \caption{Difference between batch $F_1$-Score and the $F_1$-Score of the entire Mammography dataset. Each column corresponds to a batch size $B$. See the text for more details.}
    \label{tbl:batch_results}
\end{table}
We performed this test for batch sizes of $\{128, 1024, 2048, 4096\}$ samples, which corresponds to $\{1\%, 9\%, 18\%, 37\%\}$ of the total samples of the dataset. As can be seen in Table \ref{tbl:batch_results}, the difference between batch $F_1$-Score and total $F_1$-Score decreases as the batch size increases. This trend is observed both in terms of absolute mean difference and standard deviation.

The relative performance of classifiers across batch sizes is shown in Figure \ref{fig:batch-size}. Directly optimizing Accuracy results in zero precision, recall, and $F_1$-Score because all predictions are negative. Training on $F_1$-Score, AUROC, and BCE produce non-zero $F_1$-Scores that are similar across batch sizes with the most noticeable difference being BCE, which is perhaps due to the regularizing effect when using small batches \cite{wilson2003inefficiency}

For the next experiments described in the paper, we chose a batch size of 1024, which worked well with our method and preliminary experiments with BCE.

\begin{figure}[tb!p]
    \centering
    \begin{minipage}{0.35\textwidth}
        \centering
        \includegraphics[width=1\linewidth]{plots/batch/batch_size_comparison.pdf}
    \end{minipage}
    \caption{Comparison of batch sizes across losses (rows) and metrics (columns). The Accuracy loss is computed via the proposed Heaviside approximation $\mathcal{H}$, $F_1$-Score via $\mathcal{H}$, AUROC via Wilcoxon-Mann-Whitney statistic \cite{yan2003optimizing} and BCE is binary cross-entropy loss. The top charts that seem empty indicate zero performance with the Accuracy loss. Bar height corresponds to the score in each case.}
    \label{fig:batch-size}
\end{figure}
\subsection{Sigmoid vs. Piecewise Linear Approximation}
\label{sig-vs-lin-approx}

Prior work proposed a sigmoid approximation for the Heaviside function \cite{kyurkchiev2015sigmoid}. Thus, we compare our proposed piecewise linear Heaviside approximation with a sigmoid parameterization fit to the piecewise function using non-linear least squares (Fig. \ref{fig:heaviside2} right). We fit a sigmoid to our piecewise linear approximation instead of using the original sigmoid approximation from \cite{kyurkchiev2015sigmoid} because the latter approximation poses challenges for gradient descent, as discussed in Sec. \ref{sec:preliminaries}. 

Two synthetic datasets were used for this experiment to control for class imbalance. Both synthetic datasets were generated by creating two isotropic gaussian blobs with $\sigma = 10$ in 3D space using a fixed random seed value and resulting in $10000$ samples each. We then removed a randomly-sampled proportion of the positive data points. This led to datasets with a positive to negative ratio of 1:1 (50\%) and 1:3 (33\%). The total number of samples in the final datasets were 7500 and 6250, respectively. 

We trained and evaluated models on accuracy and $F_{1}$-Score by directly optimizing for the respective metric over $10$ trials using the sigmoid or piecewise linear Heaviside approximation. The results in Table \ref{tbl:sigmoid-vs-linear} show similarity between approximations on the dataset with balanced classes and on the dataset with $33\%$ positive class balance. We argue in favor of the piecewise linear approximation since it has similar performance to the Sigmoid while avoiding the added complexity of curve fitting.
\begin{table}[b!p]
    \begin{center}
        \begin{tabular}{c|cc}
            \multicolumn{3}{c}{Synthetic 50\%} \\ \hline
            Loss & Accuracy Score & $F_1$-Score \\ \hline
           
            Sigmoid & $0.839 \pm 0.008$ & $0.845 \pm 0.008$ \\ \hline
           
            Linear* & $0.838 \pm 0.008$ & $0.846 \pm 0.008$ \\
        \end{tabular}
        \end{center}
    \begin{center}
        \begin{tabular}{c|cc}
            \multicolumn{3}{c}{Synthetic 33\%}  \\ \hline
            Loss & Accuracy Score & $F_1$-Score \\ \hline
            Sigmoid & $0.845 \pm 0.028$ & $0.782 \pm 0.020$ \\ \hline
            Linear* & $0.849 \pm 0.010$ & $0.779 \pm 0.010$ \\
        \end{tabular}
        \end{center}
        \caption{Networks trained to directly optimize for Accuracy score and $F_1$-Scores on synthetic datasets with even class balance (top) and 33\% positive samples (bottom). \textit{* indicates use of the proposed method.}}
        \label{tbl:sigmoid-vs-linear}
\end{table}
\subsection{Direct Optimization of Metrics Based on the Confusion Matrix Sets}
\label{direct-optimization-experiment}

Using four different datasets, we evaluate the performance of neural networks that are trained to optimize directly for accuracy, $F_{1}$-Score, and AUROC using our proposed Heaviside approximation. We compare these models to optimizing $F_1$-score via the adversarial approach by \cite{fathony2020ap}, optimizing AUROC via the Wilcoxon-Mann-Whitney (WMW) Statistic per \cite{yan2003optimizing}, and the classical binary cross entropy (BCE) loss. We evaluate the performance of these networks on accuracy score, $F_{1}$-Score, and AUROC over 10 repeated trials, which allow to measure the variability of the results.

The same batch size and learning rate are used for all methods except for \cite{fathony2020ap}, for which we use the hyperparameters suggested by the first author through personal communication. Direct comparison with \cite{fathony2020ap} is challenging as the maximum batch size for this method is 25 and in practice, set to only 20 samples due to the increased runtime associated with larger batch sizes.

Tables \ref{tbl:results-cocktailparty}, \ref{tbl:results-adult}, \ref{tbl:results-mammography}, and \ref{tbl:results-kaggle} show the results from the different datasets, which are further discussed in the next Sections. We report the mean of results over threshold values $\tau = \{0.1, 0.2, \cdots, 0.9\}$ across all trials.

\subsubsection{CocktailParty Dataset Results}

Results on dyadic, social interaction classification using the CocktailParty dataset are shown in Table \ref{tbl:results-cocktailparty}. This dataset has 4800 samples and a 30.29\% positive class balance making it both the smallest and the most class-balanced dataset among those considered in this experiment. Prior work has evaluated classification models for this dataset via pairwise accuracy \cite{hedayati2019}; we also consider other metrics.

Experimental results show training on accuracy via the proposed Heaviside approximation leads to the best accuracy and $F_1$-Score. When evaluating on the AUROC metric, BCE loss shows the best performance, possibly due to the small dataset size resulting in thresholds for which the true positive rate or false positive rate is ill-defined. In the case of $F_1$-Score evaluation, training on accuracy may outperform training on $F_1$-Score because of the reasonable class-balance in the dataset and the fact that accuracy incorporates the $TN$ set while $F_1$-Score does not take it into account.
\begin{table}[tb!p]
    \begin{center}
        {\small
            \begin{tabular}{c|ccc}
                \multicolumn{4}{c}{\textbf{CocktailParty} Results ($\mu\pm\sigma$)} \\
                \hline
                Loss & Accuracy & $F_1$-Score & AUROC \\ \hline
                Accuracy* & $0.846 \pm 0.01$ & $0.897 \pm 0.01$ & $0.776 \pm 0.02$ \\ \hline
                $F_1$-Score* & $0.782 \pm 0.07$ & $0.865 \pm 0.04$ & $0.650 \pm 0.12$ \\ \hline
                $F_1$-Score\dag & $0.764 \pm 0.01$ & $0.348 \pm 0.04$ & $0.615 \pm 0.01$ \\ \hline
                AUROC* & $0.637 \pm 0.04$ & $0.589 \pm 0.11$ & $0.707 \pm 0.07$ \\ \hline
                AUROC\ddag & $0.522 \pm 0.22$ & $0.424 \pm 0.41$ & $0.555 \pm 0.11$ \\ \hline
                BCE & $0.830 \pm 0.05$ & $0.876 \pm 0.05$ & $0.793 \pm 0.07$ \\ \hline
             \end{tabular}
         }
    \end{center}
    \caption{Losses (rows): Accuracy*, $F_1$-Score*, and AUROC* optimize the proposed linear Heaviside approximation $\mathcal{H}$. $F_1$-Score\dag\ uses the adversarial approach of \cite{fathony2020ap} and AUROC\ddag\ the WMW statistic \cite{yan2003optimizing}. BCE is the typical log loss. \textit{* indicates use of the proposed method.}}
    \label{tbl:results-cocktailparty}
\end{table}
\subsubsection{Adult Dataset Results}

Results on salary classification using the Adult dataset are shown in Table \ref{tbl:results-adult}. This dataset has 23.93\% positive data balance. Prior work has evaluated this dataset on accuracy score \cite{kohavi1996scaling}. In our experiments, when evaluating on accuracy, we obtain best performance by training on accuracy as a loss via the proposed Heaviside approximation. We believe accuracy is a reasonable evaluation metric for this dataset as the class balance is not extreme; however, accuracy may yet overestimate classifier performance \cite{he2009learning}. Evaluating on metrics that better deal with class imbalance, $F_1$-Score and AUROC, we find that optimizing on $F_1$-Score as a loss via $\mathcal{H}$ results in the best performance.
\begin{table}[tb!p]
    \begin{center}
        {\small
            \begin{tabular}{c|ccc}
                \multicolumn{4}{c}{\textbf{Adult} Results ($\mu\pm\sigma$)} \\
                \hline
                Loss & Accuracy & $F_1$-Score & AUROC \\ \hline
                Accuracy* & $0.813 \pm 0.01$ & $0.380 \pm 0.05$ & $0.617 \pm 0.02$ \\ \hline
                $F_1$-Score* & $0.772 \pm 0.04$ & $0.625 \pm 0.02$ & $0.781 \pm 0.02$ \\ \hline
                $F_1$-Score\dag & $0.788 \pm 0.00$ & $0.199 \pm 0.02$ & $0.562 \pm 0.01$ \\ \hline
                AUROC* & $0.470 \pm 0.05$ & $0.424 \pm 0.04$ & $0.586 \pm 0.02$ \\ \hline
                AUROC\ddag & $0.414 \pm 0.25$ & $0.353 \pm 0.15$ & $0.539 \pm 0.08$ \\ \hline
                BCE & $0.748 \pm 0.11$ & $0.315 \pm 0.14$ & $0.599 \pm 0.06$ \\ \hline
            \end{tabular}
        }
    \end{center}
    \caption{Losses (rows): Accuracy*, $F_1$-Score*, and AUROC* optimize the proposed linear Heaviside approximation $\mathcal{H}$. $F_1$-Score\dag\ uses the adversarial approach of \cite{fathony2020ap} and AUROC\ddag\ the WMW statistic \cite{yan2003optimizing}. BCE is the typical log loss. \textit{* indicates use of the proposed method.}}
    \label{tbl:results-adult}
\end{table}
\subsubsection{Mammography Dataset Results}
Results on classification of microcalcifications on the Mammography dataset are presented in Table \ref{tbl:results-mammography}. As the data balance is more skewed in this case (2.32\% positive), we expect accuracy to be less-representative of performance due to the classifier's propensity to learn to predict only dominant-class examples and achieve an accuracy proportional to the class imbalance \cite{he2009learning}. The results confirm this phenomenon with accuracy, highlighting the value of alternative evaluation measures on class-imbalanced data. 

Prior work with the Mammography dataset evaluates results on both precision and false positive rate independently \cite{woods1993comparative}. In lieu of a summary metric to directly optimize, we train and evaluate on $F_1$-Score. We find that directly optimizing the metric via the proposed Heaviside approximation results in close to or better than baseline performance. AUROC provides another summary metric, but considers only positive samples.
\begin{table}[tb!p]
    \begin{center}
        {\small
       
            \begin{tabular}{c|ccc}
                \multicolumn{4}{c}{\textbf{Mammography} Results ($\mu\pm\sigma$)} \\
                \hline
                Loss & Accuracy & $F_1$-Score & AUROC \\ \hline
                Accuracy* & $0.976 \pm 0.00$ & $0.000 \pm 0.00$ & $0.500 \pm 0.00$ \\ \hline
                $F_1$-Score* & $0.986 \pm 0.00$ & $0.699 \pm 0.04$ & $0.838 \pm 0.03$ \\ \hline
                $F_1$-Score\dag & $0.985 \pm 0.00$ & $0.554 \pm 0.15$ & $0.715 \pm 0.06$ \\ \hline
                AUROC* & $0.521 \pm 0.05$ & $0.105 \pm 0.06$ & $0.614 \pm 0.09$ \\ \hline
                AUROC\ddag & $0.636 \pm 0.42$ & $0.143 \pm 0.19$ & $0.606 \pm 0.15$ \\ \hline
                BCE & $0.984 \pm 0.00$ & $0.558 \pm 0.18$ & $0.743 \pm 0.11$ \\ \hline
            \end{tabular}
        }
    \end{center}
    \caption{Losses (rows): Accuracy*, $F_1$-Score*, and AUROC* optimize the proposed linear Heaviside approximation $\mathcal{H}$. $F_1$-Score\dag\ uses the adversarial approach of \cite{fathony2020ap} and AUROC\ddag\ the WMW statistic \cite{yan2003optimizing}. BCE is the typical log loss. \textit{* indicates use of the proposed method.}}
    \label{tbl:results-mammography}
\end{table}
\subsubsection{Kaggle Dataset Results}

The Kaggle Credit Card Fraud Detection dataset has the most extreme class imbalance of our datasets with only 0.17\% positive samples. Again, we see the effects of class imbalance on the accuracy metric and loss: Table \ref{tbl:results-kaggle} shows near perfect accuracy scores for all losses except AUROC. In the case of large class imbalance, AUROC is a challenging metric to directly optimize, as there are thresholds for which there are no positive samples and AUROC is ill-defined. One possible approach to addressing this is to apply ideas from sample mining, a common technique in deep metric learning \cite{Schroff2015facenet,wang2019deepmetriclearning}, selecting samples for a representative batch.

In line with \cite{davis2006relationship}, Kaggle acknowledges the class imbalance issue and recommends evaluation via AUPRC.\footnote{https://www.kaggle.com/mlg-ulb/creditcardfraud} However, the AUPRC summary metric is problematic due to incoherent scale assumptions \cite{flach2015precision}. We find that optimizing on $F_1$-Score directly via the heaviside approximation is effective in comparison with the baselines.
\begin{table}[tb!p]
    \begin{center}
        {\small
            \begin{tabular}{c|ccc}
                \multicolumn{4}{c}{\textbf{Kaggle} Results ($\mu\pm\sigma$)} \\
                \hline
                Loss & Accuracy & $F_1$-Score & AUROC \\ \hline
                Accuracy* & $0.998 \pm 0.00$ & $0.000 \pm 0.00$ & $0.500 \pm 0.00$ \\ \hline
                $F_1$-Score* & $0.999 \pm 0.00$ & $0.820 \pm 0.04$ & $0.901 \pm 0.02$ \\ \hline
                $F_1$-Score\dag & $0.999 \pm 0.00$ & $0.739 \pm 0.15$ & $0.833 \pm 0.07$ \\ \hline
                AUROC* & $0.479 \pm 0.08$ & $0.214 \pm 0.02$ & $0.694 \pm 0.06$ \\ \hline
                AUROC\ddag & $0.882 \pm 0.28$ & $0.470 \pm 0.35$ & $0.814 \pm 0.15$ \\ \hline
                BCE & $0.999 \pm 0.00$ & $0.553 \pm 0.26$ & $0.736 \pm 0.14$ \\ \hline
            \end{tabular}
        }
    \end{center}
    \caption{Losses (rows): Accuracy*, $F_1$-Score*, and AUROC* optimize the proposed linear Heaviside approximation $\mathcal{H}$. $F_1$-Score\dag\ uses the adversarial approach of \cite{fathony2020ap} and AUROC\ddag\ the WMW statistic \cite{yan2003optimizing}. BCE is the typical log loss. \textit{* indicates use of the proposed method.}}
    \label{tbl:results-kaggle}
\end{table}
\subsubsection{Discussion} The choice of evaluation metric for a given dataset depends on the dataset's class balance \cite{he2009learning, davis2006relationship} and the effects of this choice is amplified when optimizing the metric as a loss. Our results show the best classifiers tend to follow from networks trained on accuracy or $F_1$-score, depending class imbalance. Comparing the optimization of a given metric as a loss, our method tends to perform comparably or better than other methods for optimizing the metric as well as the BCE baseline. The AUROC results are unusual in that models trained with BCE tend to outperform both the proposed piecewise linear approximation method and the WMW statistic method for optimizing AUROC. AUROC is computed over a range of thresholds, which is problematic when positive samples for a given threshold do not exist. 

\subsection{Balancing between Precision and Recall}

Beyond the ability to directly optimize confusion matrix based metrics, our method also allows training-time optimization that balances between precision and recall using $F_\beta$-Score. This end-to-end approach to training is particularly useful in real-world scenarios where there is a high cost associated with missed detections. 

Table \ref{tbl:fb-results} shows the effectiveness of optimizing for $F_{\beta}$-Score on the Mammography dataset.  We find that directly optimizing $F_{\beta}$-Score is an effective way of maintaining maximum classifier performance measured via $F_1$-Score, while balancing between precision and recall at a ratio appropriate for the task at hand. Increasing values of $\beta$ correspond to an increased preference toward recall with small loss of total performance measured by $F_1$-Score.

\begin{table}[tb]
    \begin{center}
        
            \begin{tabular}{c|ccc}
                \multicolumn{4}{c}{\textbf{Mammography} $F_\beta$-Score Results ($\mu\pm\sigma$)} \\ \hline
                Loss & $F_1$-Score & Precision & Recall \\ \hline
                $F_1$-Score* & $0.679 \pm0.0$ & $0.727 \pm0.1$ & $0.645 \pm0.1$ \\ \hline
                $F_2$-Score* & $0.678 \pm0.0$ & $0.627 \pm0.1$ & $0.746 \pm0.1$ \\ \hline
                $F_3$-Score* & $0.579 \pm0.1$ & $0.475 \pm0.1$ & $0.752 \pm0.1$ \\ \hline
            \end{tabular}
        }
    \end{center}
    \caption{Balancing between precision and recall at training time by optimizing $F_\beta$-Score via the piecewise linear Heaviside approximation. Losses (rows): $F_\beta$-Score over $n=10$ trials, where $\beta = \{1,2,3\}$. \textit{* indicates use of the proposed method.}}
    \label{tbl:fb-results}
\end{table}





\section{Related Work}

Our work is inspired by a rich line of research focused on the direct optimization of evaluation metrics for binary classification. \cite{narasimhan2014statistical} explore plug-in algorithms that learn a classifier by applying an empirically determined threshold to suitable estimates of class probabilities. Their experiments demonstrate the applicability of their approach to optimizing $F_1$-Score with linear models. For metrics based on linear combinations of confusion matrix set cardinalities, \cite{koyejo2014consistent} identify an optimal plug-in classifier with a metric-dependent threshold. Also for linear models, \cite{kar2015surrogate} explore the precision@k metric in the context of ranking.

In the area of online learning, characterized by the sequential availability of data, works such as \cite{kar2014online, busa2015online} optimize specific metrics such as precision@k and $F_1$-Score. Our work does not address online learning, but instead batch methods. Our Heaviside approximation is applicable to any confusion matrix based metrics without reformulation.

Other prior work has focused on optimized a specific metric, like AUROC \cite{yan2003optimizing, herschtal2004optimising}, F-score \cite{dembczynski2011exact, zhao2013beyond}, and AUPRC in the context of ranking \cite{eban2016scalable}. We considered in particular the work by \cite{yan2003optimizing} as a baseline in our experiments. Our results showed that using our proposed Heaviside approximation for computing AUROC results in competitive results to using the WMW statistic for the AUROC loss.

Recently, adversarial approaches have emerged as another active area of research. \cite{wang2015adversarial} uses a structured support vector machine and reports on precision@k as well as $F_1$-Score. \cite{fathony2020ap} improves performance via a marginalization technique to ensure polynomial time convergence and evaluates on accuracy and $F_1$-Score, among other metrics, while reporting performance relative to BCE. One downside of the latter approach is small batch size, which was on the order of 25 samples in their evaluation and in our own experiments.

A parallel line of work has focused on dealing with class imbalance by dropping data in the over-represented class or augmenting the under-represented data by synthesizing new samples. Within this approach of re-sampling to deal with class imbalance, \cite{drummond2003c4} finds under-sampling preferential to over-sampling. However, in both cases it is vital to maintain representative distributions for both the over and under-represented classes \cite{he2009learning}. Therefore, strategic approaches toward handling imbalance via data preprocessing have been proposed. Synthetic Minority Over-sampling Technique (SMOTE) \cite{chawla2002smote} for example, generates synthetic data based on feature space similarity. However, recent work finds re-sampling based approaches inadequate and suggests a larger number of samples is necessary to improve results \cite{juba2019precision}. As an alternative to re-sampling, our work focuses on choosing an appropriate evaluation metric for the given level of class imbalance and directly optimizing this metric as a loss.

Our proposed method is unique in its flexibility. It allows optimization of any metric based on confusion matrix sets, even when computation of a range of thresholds is required, e.g., as in the case of AUROC. This means that if one chooses to optimize AUROC, but later finds that in the given application it would be more appropriate to specify the trade-off between precision and recall, one can easily switch to optimizing $F_\beta$-Score. 





\section{Conclusion}

We proposed a Heaviside function approximation that is useful for direct optimization of metrics based on confusion matrix set cardinalities. Our experiments showed the efficacy and flexibility of this approach in training end-to-end binary classifiers compared to classifiers that are trained with the traditional binary cross-entropy loss, unaware of their final evaluation metric. Further, our experiments demonstrated that our proposed method is competitive with other approaches for training binary classifiers that optimize for accuracy, $F_1$-Score, and AUROC. In the future, we are interested in studying the use of our Heaviside approximation on multi-class classification problems. 

Our work was motivated by the wide ranging impact that more flexible and robust binary classifiers would have across application domains, especially in the case of class imbalance. In our experiments, we showed applications related to social group dynamics, sociology, economics, and medicine. Improving the tools used in these areas of research has the potential to positively impact human quality of life. It is worth noting that end-to-end classifiers that directly optimize the evaluation metric as a loss remove the disconnect between training and evaluation, but are still susceptible to data bias. In real application scenarios, these tools must be used with care, keeping in mind the goal of building safe and responsible artificial intelligence systems.






