\section{Introduction}


Breast cancer is the second leading cause of cancer-related death among women in the United States. Screening mammography has been highly effective in reducing mortality rate but suffers from a high rate of false positive recalls, leading to added costs and stress. Better computer-aided diagnostic tools could improve patient outcomes by helping radiologists. Our goal is to build a system that provides highly accurate as well as interpretable predictions, in order to assist radiologists in cancer detection. To this end, we train an object detection network to predict the location of suspicious lesions and classify them.

\section{Data}

Our dataset \cite{NYU_dataset} comprises 229,426 screening mammography exams (1,001,093 images, with each exam containing the four standard views) from 141,473 unique patients. Among these, 5,832 exams had at least one biopsy performed within 120 days of a screening mammogram. Within this set of exams, 985 breasts had malignant findings, 5,556 had benign findings and 234 had both. For these exams, radiologists retrospectively annotated the locations of the biopsied lesions at a pixel level.


\section{Methods}

\label{sec:methods}

We used bounding boxes on the annotations of biopsied lesions to train a Feature-Pyramidal Network \cite{fpn}-based Faster-RCNN model \cite{faster_rcnn}. In most experiments, we used a ResNet-50 \citep[R-50;][]{resnet} backbone pretrained on ImageNet \cite{imagenet} to initialize our models. We also conducted experiments with ResNet-101 (R-101) and ResNeXt-101 \citep[X-101;][]{resnext} backbones. The final classifier predicts three classes: benign lesions, malignant lesions, or nothing. We used \texttt{maskrcnn-benchmark} \cite{mrcnn-benchmark} to train our models. We evaluated models after every 5,000 mini-batches and select the best models according to the score computed on the validation set. We used the standard setup of the framework, aside from the following exceptions.

\vspace{-1mm}
\paragraph{Use of non-annotated images} We used images without annotated lesions to train our model. These are treated as negatives for both the region-proposal network (RPN) and the classifier branch. The goal of this approach is to reduce overfitting on the comparatively small set of annotations. For our base experiment, during training, half of the images are sampled from an exam with a biopsy and half without.

\vspace{-1mm}
\paragraph{Resolution} We used a maximum resolution of $2200 \times 3000$ for the R-50 backbone, by resizing the mammograms with bilinear interpolation ($1700 \times 2700$ for R-101, $1300 \times 2100$ for X-101). Indeed, prior work \cite{multi-view} highlighted the importance of operating at a high resolution. Consequently, we also reduced the batch size to 4 (2 per GPU) due to our model's large memory requirements. 

\vspace{-1mm}
\paragraph{IoU thresholds} We also relaxed the IoU threshold for foreground objects in the RPN from 0.7 to 0.5, as in \cite{detecting_lesions, deformable}. Compared to natural images, our data set has (i) noisier annotations with less precise boundaries, and (ii) fewer annotations per image. To circumvent (i), we also tried isotropic bounding box rescaling by a factor drawn uniformly at random in $\left[0.8, 1.2\right]$. We set the IoU threshold of the final non-maximum suppression to 0.1, following the rationale of \citet{detecting_lesions} that for mammograms, ``overlapping detections are expected to happen less often than in usual object detection". 

\vspace{-1mm}
\paragraph{Optimization} We generally did not tune the learning rate schedules according to the advice of \citet{lr_schedule} (``recommended lr schedule'') despite lowering the batch size. Indeed, found that it did not strongly impact results while considerably increasing training time. We used gradient clipping (for norm $>3$) in all our experiments. We experimented with using batch normalization but found it not to help in our experiments.

\vspace{-1mm}
\paragraph{Inference}
During inference, we take the maximum malignant prediction among the bounding boxes of each view and then average over both views to get the predicted probability for the whole breast. We found this setup to perform better than others, including taking the maximum over views or the mean over boxes. In addition, we found it useful to decrease the score threshold at inference time from 0.05 to 0.001, as the tail-like behavior is important when computing AUC.


\section{Results}


\begin{table}[t]
\centering \small 
\begin{tabular}{lrlrl}
\toprule
\textbf{model} & \multicolumn{2}{c}{\textbf{test set}} & \multicolumn{2}{c}{\textbf{reader study}}  \\
\midrule
\textbf{\citet{pnas_paper} single model} &  0.886 & $\pm$ 0.003 & - & \\
\textbf{\citet{pnas_paper} ensemble} & 0.895 & & 0.876 \\
\hline
\textbf{Base setup} & 0.891 & $\pm$ 0.005 & 0.845 & $\pm$ 0.007 \\
\textbf{+ biopsy ratio 0.75} & 0.895 & $\pm$ 0.004 & 0.855 & $\pm$ 0.012 \\   
\textbf{+ biopsy ratio 1} & 0.887 & $\pm$ 0.011 & 0.855 & $\pm$ 0.013  \\
\textbf{+ bounding box scaling} & 0.890 & $\pm$ 0.006 & 0.841 & $\pm$ 0.010 \\
\textbf{+ classifier*5} & 0.903 & $\pm$ 0.007 & 0.859 & $\pm$ 0.013 \\
\textbf{+ bb scaling, classifier*5, ratio 0.75} & 0.888 & $\pm$ 0.006 & 0.855 & $\pm$ 0.006 \\ 
\textbf{+ recommended lr schedule} & 0.897 & $\pm$ 0.007 & 0.858 & $\pm$ 0.007 \\
\textbf{+ R-101 backbone} & 0.887 & $\pm$ 0.011 & 0.834 & $\pm$ 0.011 \\ 
\textbf{+ X-101 backbone} & \textbf{0.908} & $\pm$ 0.014 & \textbf{0.866} & $\pm$ 0.020 \\  
\hline
\textbf{Ensemble} & \textbf{0.919} & & \textbf{0.879} \\ 
\textbf{Ensemble + \citet{pnas_paper} ensemble} & \textbf{0.930} & & \textbf{0.895} \\
\bottomrule
\end{tabular}
\caption{AUC comparison against earlier results on the test set and reader study. Means and standard deviations for our models are computed over 3 random initializations. Base refers to the R-50 setup. For biopsy ratio, we increased the proportion of images from exams with biopsies in training. For classifier*5, we quintuple the weight on the classifier loss.}
\label{tab:main_results}

\end{table}

The results are shown in \autoref{tab:main_results}. Our methods compare favorably against the models of \citet{pnas_paper}. Indeed, we achieve a 23\% relative error reduction in AUC on the test set. Note that both models use pixel-level annotations. Due to high variance, we ensemble all runs together for our "Ensemble" setup. On the subpopulation of the test set used for the reader study in \citet{pnas_paper}, our results also compare favorably to both earlier models and radiologists in terms of ROC, although the improvements are smaller. The reader study setup contains more exams from patients who underwent biopsy. Thus we believe this model distinguishes better between negative cases and cases that need a biopsy but is not as good in distinguishing between benign and malignant cases within the biopsied population. Ensembling our method with \citet{pnas_paper} improves the reader study AUC further to 0.895, compared to 0.778 for the average radiologist.


\section{Analysis}

\begin{wrapfigure}{r}{0.5\textwidth}
  \captionsetup{format=plain}
  \begin{center}    
  \vspace{-8mm}
  \begin{tabular}{c c}
  \includegraphics[width=0.22\textwidth,trim=16mm 12mm 0mm 0mm]{354_mask.png} & \includegraphics[width=0.22\textwidth,trim=16mm 12mm 0mm 0mm]{354_preds.png}
  \end{tabular}
  \end{center}
  \vspace{-4mm}
  \caption{An example of an annotation of a malignant lesion (left) and predictions (right) from a X-101 model on a test set breast.}
  \label{fig:pred}
\end{wrapfigure}

Our validation metrics varied significantly between checkpoints. We believe that this is due to (i) the interaction of the components of the training loss, (ii) components of the training loss being only loosely related to the final metric, (iii) a small batch size rendering optimization unstable. This motivates ensembling all runs to produce our final model. 

We notice a clear trade-off between using larger networks and using a larger resolution, with the R-50, R-101 and X-101 setups performing competitively. Unfortunately, due to computational constraints, we were unable to test R-101 and X-101 on the same resolution as R-50. Overall, our ensemble combining different backbones and resolutions performed best.

In Figure~\ref{fig:pred}, we show an annotation and the corresponding prediction using an X-101 model. The model accurately predicts a malignant lesion with high probability (0.99). It also predicts a benign lesion with low probability (0.36) for which there is no ground-truth annotation. These bounding boxes can highlight suspicious regions and help radiologists understand predictions from our models.


\midlacknowledgments{We would like to thank Yiqiu Shen and Jungkyu Park for comments on earlier versions of this manuscript. We also gratefully acknowledge the support of Nvidia Corporation with the donation of some of the GPUs used in this research. This work was supported in part by grants from the National Institutes of Health (R21CA225175 and P41EB017183).}


