
\section{Introduction}


Convolutional neural networks (CNNs) have revolutionized medical image analysis~\cite{bekkers2018roto, ching2018opportunities, topol2019high, prevedello2019challenges, lutnick2019integrated, karimi2020deep, gaonkar2021eigenrank}. These networks achieve impressive results, but their outputs are often difficult to interpret, which is problematic for clinical decision making~\cite{yao2018weakly}. Designing explainable classification models is a challenge. In some applications, such as cancer diagnosis, interpretability can be achieved by localizing the regions of the input image that determine the output of the model~\cite{shen2020interpretable}.
Alternatively, detection and segmentation networks, such as U-Net~\cite{ronneberger2015u} and Faster R-CNN~\cite{ren2016faster} can be trained with annotations indicating regions relevant to diagnosis.
Unfortunately, acquiring such annotations is labor-intensive, and requires medical expertise. Moreover, learning under such supervision might bias the network to ignore lesions occult to radiologists, which a neural network can still identify. 




Given the above obstacle, weakly-supervised localization (WSL) has recently become an area of active research~\cite{diba2017weakly, singh2017hide, zhang2018adversarial, zhang2018self, cui2019weakly}. These approaches aim to identify image regions relevant to classification utilizing only image-level labels during training, based upon the observation that feature maps in the final convolutional layers of CNNs reveal the most influential regions of the input image~\cite{oquab2015object, zhou2016learning}. These methods are usually designed for natural images and applying them to medical images is challenging due to their unique characteristics. For example, mammography images have a much higher resolution ($\sim10^7$ pixels) than natural images ($\sim10^5$ pixels) in most benchmark datasets, such as ImageNet~\cite{deng2009imagenet}. Because of this, when applied to medical images, CNNs often aggressively downsample the input image~\cite{shen2019globally,shen2020interpretable} to accommodate GPU memory constraints, making the resulting localization too coarse. 
This is a crucial limitation for many medical diagnosis tasks, where regions of interest (ROIs) are often small (e.g. $\leq 1\%$ pixels). 





\begin{figure*}[htbp]
\centering
     \resizebox{0.7\linewidth}{!}{
    \begin{tabular}{c c c c c c}
        
         \LARGE Input & \LARGE  \LARGE Ground Truth & \LARGE  CAM & \LARGE GMIC & \LARGE GLAM (proposed) \\
                 \centering

         \includegraphics[ height=0.3\linewidth]{images/example_161_rcc/Input.png}
         &   
        \includegraphics[height=0.3\linewidth]{images/example_161_rcc/groundtruth.pdf} &   {
        \includegraphics[height=0.3\linewidth]{images/example_161_rcc/CAM.pdf} 
        } 
        &    {
        \includegraphics[height=0.3\linewidth]{images/example_161_rcc/GMIC.pdf} 
        } 
         &   {
        \includegraphics[height=0.3\linewidth]{images/example_161_rcc/Ours.pdf}} 
         \\
        \includegraphics[ height=0.3\linewidth]{images/example_78_rmlo/Input.pdf}   &   
        \includegraphics[height=0.3\linewidth]{images/example_78_rmlo/groundtruth_v2.pdf} &   {
        \includegraphics[height=0.3\linewidth]{images/example_78_rmlo/CAM.pdf} 
        } 
         &   {
        \includegraphics[height=0.3\linewidth]{images/example_78_rmlo/GMIC.pdf} 
        } 
         &   {
        \includegraphics[height=0.3\linewidth]{images/example_78_rmlo/Ours.pdf}} 
     \end{tabular}  
     }
\centering
\caption{Comparison of saliency maps generated by CAM~\cite{zhou2016learning}, GMIC~\cite{shen2020interpretable}, and the proposed method on a mammography image containing a malignant lesion (first row, red) and a benign lesion (second row, green). Both CAM and GMIC produce coarse saliency maps that fail to localize the lesion accurately. The proposed method generates a high-resolution saliency map that precisely localizes the lesions.}
\label{fig:compare_fig2}
\end{figure*}

In this work, we propose GLAM (Global-Local Activation Maps), a novel framework to generate fine-grained segmentation using only image-level labels. The proposed model processes high resolution medical images in a memory-efficient way. The main idea behind GLAM is to select informative regions (patches) that may contain ROIs via coarse-level localization and then to perform segmentation on selected patches rather than the entire image in a weakly supervised manner. We train and evaluate GLAM on a dataset containing more than one million mammography images. We demonstrate that the model outperforms existing baselines in segmentation of both benign and malignant lesions, improving the Dice similarity score relatively by 39.6\% and 20\%, respectively, while preserving classification accuracy. To achieve that, GLAM produces fine-grained saliency maps with a 300 times higher resolution than previous works~\cite{shen2020interpretable} ($736\times 480$ pixels for $2944\times1920$ pixels input images). In Figure~\ref{fig:compare_fig2}, we illustrate how the saliency maps generated by GLAM enable high-resolution segmentation of lesions relevant to breast cancer diagnosis. 


\section{Background}
\label{background}

WSL is the task of learning to locate ROIs (i.e. the \emph{objects}) in an image when only image-level labels are available during training. WSL methods are usually based on CNNs that produce saliency maps encoding the location of ROIs. To train the whole system using only image-level labels, the saliency maps are collapsed to predictions indicating the presence of each class using a pooling function. Once the CNN is trained, the saliency map can be used for localization~\cite{oquab2015object,zhou2016learning}. WSL
has been applied in a wide range of medical-imaging applications, including the detection of lung disease in chest X-ray images~\cite{wang2017chestx, yao2018weakly, tang2018attention, ma2019multi, liu2019align, guan2018diagnose}, diagnosis of injuries from pelvic X-ray images~\cite{wang2019weakly}, brain lesion segmentation~\cite{wu2019weakly}, breast MRI analysis~\cite{luo2019deep}, cancer detection in lung CT images~\cite{feng2017discriminative, schlemper2018attention}, and scan-plane detection in ultrasound~\cite{schlemper2018attention,baumgartner2017sononet}. 
 
The majority of these works focus on images that have relatively low resolution, that is, $512\times512$ pixels or less. Only a few works have considered higher resolution images, which are standard in some imaging procedures such as screening mammography~\cite{shen2019globally, shen2020interpretable}. 




In this work we focus on the diagnosis of breast cancer from screening mammography images.
Breast cancer is the second leading cause of cancer-related deaths among women~\cite{bray2018global} and screening mammography is the main tool for its early detection~\cite{marmot2013benefits}.
CNN classifiers have shown promise in diagnosis from mammograms~\cite{zhu2017deep, kim2018applying, ribli2018detecting, wu2019deep, geras2019artificial, mckinney2020international, shen2019globally, shen2020interpretable}. Accurate localization of suspicious lesions is crucial to aid clinicians in interpreting model outputs, and can provide guidance for future diagnostic procedures.
However, existing methods that explain their predictions, e.g. \citet{shen2019globally,shen2020interpretable}, 
offer only coarse localization. 
GLAM is inspired by recent works~\cite{yao2018weakly, shen2019globally, shen2020interpretable, shamout2020artificial}, which improve classification accuracy by processing image patches selected from coarse saliency maps. The main innovation of GLAM with respect to these works is that it generates a 
a high-resolution saliency map from the selected patches, which significantly improves lesion segmentation accuracy.










\section{Proposed Approach}

Our goal is to generate fine-grained saliency maps that localize objects of interest in high-resolution images using only image-level labels during training. 
We start this section by describing the inference pipeline of our approach in Section~\ref{sec:inference}. 
We then describe each module in detail in Section~\ref{sec:architecture}. Finally, we explain the training strategy in Section~\ref{sec:training}. 
\subsection{Inference pipeline}
\label{sec:inference}

\begin{figure}[]
   
    \centering
    \includegraphics[width=0.7\textwidth,trim=0mm 10mm 0mm 8mm]{images/overviewinferencev5.pdf}
       
    \caption{Inference pipeline of GLAM. 1) The global network $f_g$ is applied to the whole image $\mathbf{x}$ to obtain a coarse image-level segmentation map $\mathbf{S}_g$. 2) Based on this coarse-level segmentation, several patches are extracted from the input image. 3) The local network $f_l$ processes these patches to generate a high-resolution saliency map $\mathbf{S}_l$.
    }
    \label{fig:frameworkinference}
   
\end{figure} 

As illustrated in Figure~\ref{fig:frameworkinference}, during inference, our system processes an input $\mathbf{x}\in \mathbb{R}^{H,W}$ as follows:
\begin{enumerate}[leftmargin=*]
\itemsep0em 
    \item The image $\mathbf{x}$ is fed into the \emph{global module}, a memory-efficient CNN denoted by $f_g$, to produce an image-level coarse saliency map $\mathbf{S}_{g}$ and an image-level class prediction $\hat{y}_{g}$.

    \item We select $M$ patches from $\mathbf{x}$ based on $\mathbf{S}_{g}$. To do that, we greedily select the patches for which the sum of the entries in $\mathbf{S}_{g}$ is the largest (see Algorithm~\ref{alg:alg1} for a detailed description). 
    \item We feed the selected patches $\tilde{\mathbf{x}}_1, \ldots, \tilde{\mathbf{x}}_M$ 
    to the \emph{local module} $f_l$, another CNN which produces a fine-grained saliency map associated with each patch. We then remap the patch-level saliency maps back to their location in the original input image. We denote the saliency map obtained through this procedure by $\mathbf{S}_{l}$.
\end{enumerate}
GLAM produces an image-level saliency map $\mathbf{S}_{g}$ and a fine-grained multi-patch saliency map $\mathbf{S}_{l}$. These maps are aggregated through averaging to produce the final saliency map $\mathbf{S}_{c} = (\mathbf{S}_{g} + \mathbf{S}_{l})/2$. In addition, GLAM generates a classification output, which is produced by the global module (as this yields the best classification accuracy).

\subsection{Module parameterization}
\label{sec:architecture}

\paragraph{Global module}
The architecture of $f_g$ is based on the design of \citet{shen2019globally,shen2020interpretable}, which is similar to ResNet~\cite{he2016deep} with a reduced number of channels for memory efficiency. 
The main difference between $f_g$ and the global module of \citet{shen2019globally,shen2020interpretable} is that we combine saliency maps at different scales to generate the global saliency map, inspired by~\citet{sedai2018deep} who observed that using convolutional feature maps extracted only from the last layer of a CNN may be suboptimal in localization of small objects. We generate saliency maps at different scales ($\mathbf{S}_0$, $\mathbf{S}_1$ and $\mathbf{S}_2$) using a pyramidal hierarchy of feature maps (see Figure~\ref{fig:global} in Appendix~\ref{append:globalstructure}). Our image-level saliency map $\mathbf{S}_g$ is obtained by averaging them. For each $\mathbf{S}_n$ $ (n\in \{0,1,2\})$, we obtain a classification prediction $\tilde{y}_n$ associated with $\mathbf{S}_n$ using $\mathrm{top} \, t\%$ pooling (see Appendix~\ref{append:globalstructure}), where $t$ is a hyperparameter. The image-level classification prediction $\hat{y}_g$ is calculated by averaging $\tilde{y}_0$, $\tilde{y}_1$, and $\tilde{y}_2$. Additionally, we output a representation vector $z_g$ to feed it into a fusion module (described below) to enable joint training with the local module. 
See Appendix \ref{append:globalstructure} for more details. 


\paragraph{Local module}
Our local module is based on ResNet-34 with a reduced stride in the residual blocks to maintain a higher resolution. We replace the global average pooling and the last fully connected layer by a $1\times1$ convolution layer followed by a sigmoid non-linearity. 
The local module is applied to each of the selected patches $\tilde{\mathbf{x}}_k$ ($k \in 1,\ldots,K$) to extract a patch-level saliency map $\mathbf{A}_{k}$. As we only have image-level labels, we need to train the patch-level network without patch-level labels.
To address this challenge, we combine insights from weakly-supervised localization and multiple-instance learning to train the patch-level saliency maps hierarchically.
In multiple instance learning~\cite{maron1998framework} the labels are associated with bags of instances (the label is negative if all instances in a bag are negative, and positive otherwise). In our case each instance is a patch, and the patches from an image form a bag. 
We use a patch aggregation function $f_\text{p}$ to combine the information across all patches and form an image-level prediction $\hat{y}_{l}$. 
Formally, we have $\hat{y}_{l} = f_\text{p}(\mathbf{A_1},\ldots,\mathbf{A_k})$. We propose two different patch aggregation functions.  
\begin{itemize}[leftmargin=*]
\itemsep0em 
    \item {\textit{Concatenation-based aggregation}:} We concatenate the saliency maps spatially and apply pooling function $f_{\text{agg}}$ (\emph{i.e}\onedot} \def\Ie{\emph{I.e}\onedot $\mathrm{top} \, t\%$ pooling).
    The prediction is thus given by 
    $\hat{y}_{l} = f_{\text{agg}} (\mathrm{concat}(\mathbf{A}_1, \ldots, \mathbf{A}_K))$. 
    
    \item {\textit{Attention-based aggregation:}} $\mathrm{top} \, t\%$ pooling is applied to $\mathbf{A}_k$ to obtain a patch-level prediction $\hat{y}_k$. Additionally, we output a representation vector $z_k$ for each patch, which will be aggregated to $z_l$ and fed into the fusion module. We use the Gated Attention Mechanism~(\cite{ilse2018attention}) to
    combine the prediction and the representation vectors using attention weights $\alpha_i \in [0,1]$. The prediction is given by $\hat{y}_{l} = \sum_{i=1}^K \alpha_i \hat{y}_i$, and the representation vector by ${z}_{l} = \sum_{i=1}^K \alpha_i {z}_i$.
\end{itemize}
Section~\ref{sec:training} explains when we use each aggregation method. Refer to Appendix~\ref{append:localstructure} for more details.



\paragraph{Fusion module}
In order to jointly optimize the parameters of the global and local modules, we incorporate a  \emph{fusion module} consisting of one fully connected layer that combines the representation vectors from the global (${z}_{g}$) and local modules (${z}_{l}$) to produce a fusion prediction $\hat{y}_f$. Formally,  $\hat{y}_f = \mathrm{sigmoid}(w_f[z_g,z_l]^T)$, where $w_f$ is a vector of learnable parameters. 


\subsection{Training strategy}
\label{sec:training}
The training strategy that achieves the best experimental performance for GLAM is sequential. We first train the global module, then we train the local module and finally we train both together. This makes sense because in order to train the local module $f_l$ effectively, we need to select meaningful input patches. Since the selection relies on the global saliency map, this requires pretraining the global module. Our training strategy is as follows (see also Figure~\ref{fig:frameworktrain}). 

\begin{figure}[]
    \centering
    \includegraphics[width=0.7\textwidth,trim=0mm 10mm 0mm 8mm]{images/overviewtrainv4.pdf}
    \caption{Proposed training strategy. 1) Train the global module and select the best segmentation model. 2) Freeze the global module and use it to extract input patches for the local module. 3) Train the local module on the selected patches. 4) Joint training with the fusion module. We use BCE loss and sparsity loss to train the system. 
    }
    \label{fig:frameworktrain}
\end{figure} 

\begin{enumerate}[leftmargin=*]
\itemsep0em 
    \item Train the global module with the loss function {\small{$L_{g} = \sum_{n \in \{0,1,2\}} ( \operatorname{BCE}(y, \tilde{y}_{n}) + \lambda \sum_{(i,j)} |\mathbf{S}_{n}(i,j)| )$}} using the whole training set. Here, {\small{$\operatorname{BCE}(y, \tilde{y}_{n})$}} is the binary cross-entropy loss and  {\small{$\sum_{(i,j)} |\mathbf{S}_{n}(i,j)|$}} is an $\ell_1$ regularization enforcing sparsity of the saliency map. 
    The hyperparameter $\lambda$ balances the two terms. The model is selected based on segmentation performance on the validation set.

    \item Create training data for the local module. To produce negative examples, we select $K$ patches randomly from images with negative labels. The positive examples are generated by selecting $K$ patches from images with positive labels using Algorithm \ref{alg:alg1} applied to the saliency maps produced by the pretrained global module. 
    \item Train the local module minimizing the cost function $L_{l} = \operatorname{BCE}(y, \hat{y}_{l}) + \lambda \sum_{(i,j)} |\mathbf{S}_{l}(i,j)|$.
     Here we apply the concatenation-based aggregation described in Section~\ref{sec:architecture} because it yields the best results experimentally  and speeds up training. 
    \item Train the global and local modules jointly with the fusion module.
    Here we use the attention-based aggregation in the local module. The fusion module outputs a fusion prediction $\hat{y}_{f}$. The loss function for joint training is $L_{t} = L_{g} + L_{l} + L_{f}$, where $L_{f} = \operatorname{BCE}(y, \hat{y}_{f})$. 
\end{enumerate}


Note that the number of patches selected as input to the local module during training ($K$) and inference ($M$) do not need to be the same. In fact, we find empirically that it is beneficial to use $K$ as large as possible (within the constraints imposed by GPU memory), which is consistent with~\citet{shen2020interpretable}. However, choosing a large $M$ increases the false positive rate. We used $K=6$ and $M=1$, based on experiments reported in Appendix~\ref{append:numpatchtraining} and \ref{append:patchnuminference}.


\section{Experiments}
\subsection{Dataset and evaluation metrics} 
We trained and evaluated the proposed model on the NYU Breast Cancer Screening Dataset v1.0~\cite{NYU_dataset} that includes 229,426 exams (1,001,093 images) from 141,472 patients. Each exam contains at least four images with a resolution of $2944 \times 1920$ pixels, corresponding to the four standard views used in screening mammography: R-CC (right craniocaudal), L-CC (left craniocaudal), R-MLO (right mediolateral oblique) and L-MLO (left mediolateral oblique). The dataset is divided into disjoint training (186,816), validation (28,462) and test (14,148) sets, ensuring that each patient only belongs to one of the sets. 
Each breast has two binary labels indicating whether malignant or benign lesions are present. A subset of the images with lesions have pixel-level annotations provided by radiologists, which indicate the position of the lesions. 
Note that the dataset (458,852 breasts) contains more exams without lesions (452,311 compared to 5,556 with benign lesions, and 985  with malignant lesions). To account for this imbalance when training our models, at each epoch we use all exams with lesions and an equal number of randomly-sampled exams without lesions.


To measure classification performance, we report the area under the ROC curve (AUC) for identifying breasts with both malignant and benign lesions. To evaluate localization ability, we use the Dice similarity coefficient and pixel average precision (PxAP)~\cite{choe2020evaluating}. PxAP is the average of the area under the precision-recall curve for each pixel. The threshold to compute the precision and recall is either chosen for each image (image-level PxAP) or fixed for the whole test set (dataset-level PxAP). These metrics are described in more detail in Appendix~\ref{append:Evaluation metrics}. 


\subsection{Comparison to baselines}


We compare GLAM to two baselines: GMIC~\cite{shen2020interpretable}, a WSOL method specifically designed for high-resolution medical images, and CAM~\cite{zhou2016learning}, one of the most popular WSOL methods for natural images. 
We use the same backbone architecture for the two baselines and the global module of GLAM. 
The same hyperparameter tuning is applied to all models to ensure a fair comparison, as described in Appendix~\ref{append:Hyperparametertuning}. The models are selected based on segmentation performance, which is not necessarily equivalent to selection based on classification performance (see Appendix~\ref{Append:discussion} for further discussion). We also compare to a U-Net~\cite{ronneberger2015u} trained with strong supervision using the pixel-level annotations. This provides an upper bound for the segmentation performance of the WSL methods. In Table~\ref{t:comparision}, we report the performance of GLAM and the baselines. GLAM outperforms both GMIC and CAM in all segmentation evaluation metrics by a large margin, while achieving very similar classification accuracy. We observe a performance gap between GLAM and the strongly supervised U-Net model trained with ground-truth segmentation annotations, which indicates that there is room for further improvement.



\begin{table}[H]
\caption{Segmentation performance of our method (GLAM) and several baselines evaluated in terms of Dice (mean and standard deviation over the test set), image-level PxAP and dataset-level PxAP for malignant and benign lesions. We also report the classification AUC achieved by each model. GLAM outperforms the baselines, while achieving very similar classification accuracy. The performance of a model (U-Net) trained with segmentation annotations is also included for comparison.} 

\label{t:comparision}
\centering
\resizebox{\linewidth}{!}{
\begin{tabular}{lcccccccc}
\hline
                                                           & \multicolumn{2}{l}{Dice}                                   & \multicolumn{2}{l}{image-level PxAP}                        & \multicolumn{2}{l}{dataset-level PxAP}   &
                                \multicolumn{2}{l}{classification AUC}   \\ \hline
                                                           & \multicolumn{1}{l}{Malignant} &
                                \multicolumn{1}{l}{Benign} & \multicolumn{1}{l}{Malignant} &\multicolumn{1}{l}{Benign} & \multicolumn{1}{l}{Malignant} &
                                \multicolumn{1}{l}{Benign} &
                                \multicolumn{1}{l}{Malignant} &
                                \multicolumn{1}{l}{Benign} \\ \cmidrule(lr){2-3}\cmidrule(lr){4-5}\cmidrule(lr){6-7}\cmidrule(lr){8-9}
GLAM (proposed) & \textbf{0.390 $\pm$ 0.253} & 	\textbf{0.335 $\pm$ 0.259} &	\textbf{0.461 $\pm$ 0.296} &\textbf{0.396 $\pm$ 0.315} &	\textbf{0.341} &	\textbf{0.215}  &  0.882 & 0.770  \\
GMIC	&	0.325 $\pm$ 0.231&	0.240 $\pm$  0.175&	0.396 $\pm$ 0.275& 0.283 $\pm$ 0.244& 0.295 & 0.112 &0.886 & {0.780} \\
CAM & 0.250 $\pm$ 0.221 & 0.207 $\pm$ 0.180 & 0.279 $\pm$ 0.240 & 0.222 $\pm$  0.210 & 0.226 & 0.084 & {0.894} & 0.770 \\
U-Net (fully supervised) 	&	0.504 $\pm$ 0.283&	0.412 $\pm$ 0.316&	0.589 $\pm$ 0.329&	0.498 $\pm$ 0.357 & 0.452& 0.265 & - &- \\
\hline
\end{tabular}
}

\end{table}



\subsection{Ablation study}


We analyze different design choices in GLAM through an extensive ablation study. Due to space constraints, here we only discuss the advantages of  training the global and local modules jointly, and the segmentation properties of the global and local saliency maps. We defer results on the following choices to the appendices: design of the global module (Appendix~\ref{append:multi-level}), selection of training data for the local module  (Appendix~\ref{append:random}), design of the local module  (Appendix \ref{append:architecture}), number of input patches used by the local module during training (Appendix~\ref{append:numpatchtraining}) and inference (Appendix~\ref{append:patchnuminference}), the fusion module (Appendix~\ref{append:fusionmodule}), and hyper-parameter selection: 
(Appendix~\ref{append:tvalue}, {\color{green}  ~\ref{append:lambda}} and~\ref{append:combinehyper}).
\paragraph{Joint training of local and global modules.} Here we compare the GLAM saliency map $\mathbf{S}_{c-joint}$ obtained via joint training as described in Section~\ref{sec:training} with (1) the global saliency map $ \mathbf{S}_{g}$ from the global module pretrained in isolation, (2) the local saliency map $\mathbf{S}_{l}$ for the local module trained using patches selected using a frozen global module, (3) a saliency map $\mathbf{S}_{c-sep}$ obtained by averaging $ \mathbf{S}_{g}$ and $\mathbf{S}_{l}$. The results reported in Table \ref{t:ablation1} show that  $\mathbf{S}_{c-sep}$ is superior to $\mathbf{S}_{g}$ and $\mathbf{S}_{l}$, but joint training achieves better performance across all metrics.


\begin{table}
   
\caption{Segmentation performance of the GLAM saliency map $\mathbf{S}_{c-joint}$, the global ($\mathbf{S}_{g}$) and local saliency ($\mathbf{S}_{l}$) maps trained separately, and the average of $ \mathbf{S}_{g}$ and $\mathbf{S}_{l}$ with ($\mathbf{S}_{c-joint}$) and without ($\mathbf{S}_{c-sep}$) joint training. Averaging helps, joint training further improves.}


\label{t:ablation1}
\centering
\resizebox{0.8\linewidth}{!}{
\begin{tabular}{lcccccc}
\hline
                                                           & \multicolumn{2}{l}{Dice}                                   & \multicolumn{2}{l}{image-level PxAP}                        & \multicolumn{2}{l}{dataset-level PxAP}   \\ \hline
                                                           & \multicolumn{1}{l}{Malignant} &
                                \multicolumn{1}{l}{Benign} & \multicolumn{1}{l}{Malignant} &\multicolumn{1}{l}{Benign} & \multicolumn{1}{l}{Malignant} &
                                \multicolumn{1}{l}{Benign} \\ 
                          \cmidrule(lr){2-3}\cmidrule(lr){4-5}\cmidrule(lr){6-7}
$\mathbf{S}_{g}$                          & {0.325} $\pm$ 0.239   & {0.261} $\pm$ 0.185  & {0.363} $\pm$ 0.276      & {0.302} $\pm$  0.266 & {0.324} & {0.140 }                          \\
$\mathbf{S}_{l}$     & {0.343} $\pm$ 0.283	& {0.297} $\pm$ 0.287	&{0.444} $\pm$ 0.337	& {0.337} $\pm$ 0.310	&{0.207}	&{0.126  }                      \\
$\mathbf{S}_{c-sep}$ &                  {0.375} $\pm$ 0.264  &	{0.318} $\pm$ 0.243&	{0.449} $\pm$ 0.319 &	{0.382} $\pm$ 0.317 &	{0.340}  &	{0.191}   \\
$\mathbf{S}_{c-joint}$ & \textbf{0.390 $\pm$ 0.253} & 	\textbf{0.335 $\pm$ 0.259} &	\textbf{0.461 $\pm$ 0.296} &\textbf{0.396 $\pm$ 0.315} &	\textbf{0.341} &	\textbf{0.215}  \\
\hline
\end{tabular}
}
\end{table}


\paragraph{Segmentation properties of the global and local saliency maps} In Figure~\ref{fig:comp_global_vs_local}, we plot the Dice scores of the global ($\mathbf{S}_{g}$) and local ($\mathbf{S}_{l}$) saliency maps generated by GLAM for 400  randomly selected examples from the validation set. Each saliency map has different strengths and weaknesses. $\mathbf{S}_{l}$ fails completely for a subset of examples, this is because the patch-selection procedure did not select the correct patches (lower row in Figure~\ref{fig:failure_fig}). On the remaining examples, $\mathbf{S}_{l}$ tends to outperform $\mathbf{S}_{g}$ because it has a much higher resolution. However, in some cases it underperforms  $\mathbf{S}_{g}$, often in cases where the ground-truth segmentation is larger than the size of the patch (upper row in Figure~\ref{fig:failure_fig}). Averaging $\mathbf{S}_{l}$ and $\mathbf{S}_{g}$ (our strategy of choice in GLAM) achieves high-resolution segmentation, while hedging against the failure of the local module.



\begin{figure}[h!]
\begin{minipage}{0.46\linewidth}
    \centering
       
    \includegraphics[width=0.49\textwidth]{images/Dice_malignant_global_vs_local.pdf}
    \includegraphics[width=0.49\textwidth]{images/Dice_benign_global_vs_local.pdf}
   
    \caption{Scatter plot of Dice score of the global $\mathbf{S}_{g}$ and local $\mathbf{S}_{l}$ modules for 400 validation examples. $\mathbf{S}_{l}$ outperforms $\mathbf{S}_{g}$ for most small lesions, but may miss some larger lesions and fails entirely if the wrong patches are selected as input.
    }
    \label{fig:comp_global_vs_local}
\end{minipage}\hfill
\begin{minipage}{0.51\linewidth}
\centering
\setlength{\tabcolsep}{1.2pt}
   
    \begin{tabular}{c c c c }
\tiny{Ground Truth} & \tiny{$\mathbf{S}_{g}$} & \tiny{Patch Map} & \tiny{$\mathbf{S}_{l}$}  \\
        \includegraphics[width=0.21\linewidth,height=0.31\linewidth]{images/example_2614_lcc/groundtruth.png} & 
        \includegraphics[width=0.21\linewidth,height=0.31\linewidth]{images/example_2614_lcc/global.png} &
        \includegraphics[width=0.21\linewidth,height=0.31\linewidth]{images/example_2614_lcc/patchmap.png}&
        \includegraphics[width=0.21\linewidth,height=0.31\linewidth]{images/example_2614_lcc/localbranch.png} \\
        \includegraphics[width=0.21\linewidth,height=0.31\linewidth]{images/failure_case/175_rcc/idx_175_rcc_gt_v2.png} &        \includegraphics[width=0.21\linewidth,height=0.31\linewidth]{images/failure_case/175_rcc/idx_175_rcc_global.png} &        \includegraphics[width=0.21\linewidth,height=0.31\linewidth]{images/failure_case/175_rcc/idx_175_rcc_patch.png} &        \includegraphics[width=0.21\linewidth,height=0.31\linewidth]{images/failure_case/175_rcc/idx_175_rcc_local.png}   
    \end{tabular} 
    
   
\caption{Two failure cases of the local module. (Top) The lesion is larger than the input patch, so $\mathbf{S}_{l}$ only captures it partially. (Bottom) The input patches to $\mathbf{S}_{l}$ (in blue) do not cover the lesion. 
}
\label{fig:failure_fig}
\end{minipage}
\end{figure}



\section{Conclusion}


In this work, we propose a novel framework to perform weakly-supervised segmentation of images with very high resolution, which outperforms existing methods. Our results suggest that the general principle underlying GLAM (hierarchical selection of saliency maps) could be effective in other applications involving high-resolution images and videos. Another interesting question for future research is how to extend this principle to settings where high-resolution segmentation is desired, but (in contrast to breast cancer) the regions of interest do not tend to be small.



\newpage
\midlacknowledgments{
The authors would like to thank Mario Videna, Abdul Khaja and Michael Costantino for supporting our computing environment and Catriona C. Geras for proofreading the paper. We also gratefully acknowledge the support of Nvidia Corporation with the donation of some of the GPUs used in this research. C.F. was partially supported by NSF DMS grant 2009752. K.L. was partially supported by NIH grant R01 LM013316 and NSF NRT grant HDR-1922658. This work was supported in part by grants from the National Institutes of Health (P41EB017183, R21CA225175), the National Science Foundation (1922658) and the Gordon and Betty Moore Foundation (9683).
}






