\section{Introduction}

Referring Expression Grounding (REG), \textit{i}.\textit{e}., localizing the targeted instance (referent) in an image given a natural language description, is a longstanding task for multimodal understanding. Considering different granularities of localization, there are two sub-types of REG: 1) \emph{Referring Expression Comprehension (REC)}~\cite{hu2017modeling,hu2016natural,yu2016modeling,yu2017joint}, where the referents are localized by bounding boxes (bboxes). 2) \emph{Referring Expression Segmentation (RES)}~\cite{hu2016segmentation,liu2017recurrent,shi2018key,margffoy2018dynamic}, where the referents are localized by segmentation masks. Both tasks are important for many downstream high-level applications such as visual question answering~\cite{antol2015vqa}, navigation~\cite{chen2019touchdown}, and autonomous driving~\cite{kim2019grounding}.

State-of-the-art REG methods can be classified into two major categories: one-stage, proposal-free methods and two-stage, proposal-driven methods. For the one-stage methods~\cite{chen2018real,yang2019fast,liao2020real}, they regard REG as a generalized object detection (or segmentation) task, and the whole textual expression is treated as a specific object category. Although these one-stage methods achieve faster inference speed, their grounding performance, especially for complex expressions (\textit{e}.\textit{g}., RefCOCOg), is still behind the two-stage counterpart. The main reasons for the differences are two-fold: 1) The one-stage methods naturally focus on the local content, \textit{i}.\textit{e}., they fail to perform well in the expressions which need global reasoning. For example in Figure~\ref{fig:motivation} (a), when grounding "\emph{a cat laying down on a white towel next to some keys}", it is even difficult for humans to identify the referent \texttt{cat} without considering its contextual objects \texttt{towel} and \texttt{keys}. 2) The one-stage methods do not exploit the linguistic structure of expressions, \textit{i}.\textit{e}., they are not sensitive to linguistic variations in expressions. For instance, when changing the expression in Figure~\ref{fig:motivation} (a) to "\emph{cute cat with stripes looking toward the cat on a white towel}", they tend to refer to the same object (\#1)~\cite{akula2020words}. On the contrary, two-stage methods~\cite{yu2018mattnet,liu2019learning,liu2019improving} intuitively are more similar to the human way of reasoning: 1) detecting proposals with a detector, and then 2) grounding the referent to one of the proposals. In general, two-stage methods with perfect proposals\footnote{For the RefCOCO-like datasets, two-stage methods can use ground-truth regions in COCO~\cite{lin2014microsoft} as proposals.} can achieve more accurate and explainable grounding results than the one-stage methods.

\begin{figure}[t]
	\centering
	\includegraphics[width=0.95\linewidth]{images/motivation.pdf}
	\vspace{-0.5em}
	\caption{\textbf{(a):} A typical example from RefCOCOg dataset. \textbf{(b):} Recall of the referent (IoU>0.5) vs. number of proposals on the RefCOCO testB set. Real case denotes the actual situation in SOTA two-stage methods. \textbf{(c):} An example of filtered proposals from MAttNet~\cite{yu2018mattnet} with threshold 0.65. The dashline denotes the missing referent.}
	\vspace{-1.5em}
	\label{fig:motivation}
\end{figure}

Unfortunately, when using the results from off-the-shelf detectors as proposals, two-stage methods' performance drops dramatically. This is also the main weakness of two-stage solutions often criticized by competing methods in the literature, \textit{i}.\textit{e}., the performance of two-stage methods is heavily limited by the proposal quality. In this paper, we argue that this huge performance gap between the detected and ground-truth proposals is mainly caused by the \textbf{mismatch} between the roles of proposals in the two stages: \emph{the first-stage network generates proposals solely based on the detection confidence, while the second-stage network just assumes that the generated proposals will contain all right instances in the expression}. More specifically, for each image, a well pre-trained detector can detect hundreds of detections with a near-perfect recall of the referent and contextual objects (\textit{e}.\textit{g}., as shown in Figure~\ref{fig:motivation} (b), recall of the referent can reach up to 96.58\% with top-100 detections). However, to relieve the burden of the referent grounding step in the second stage, current two-stage methods always filter proposals simply based on their detection confidences. These heuristic rules result in a sharp reduction of the recall (\textit{e}.\textit{g}., decrease to 80.77\% as in Figure~\ref{fig:motivation} (b)), and bring in the mismatch negligently. To illustrate this further, we show a concrete example in Figure~\ref{fig:motivation} (c). To ground the referent at the second stage, we hope that the proposals contain the referent \texttt{person} and its contextual object \texttt{pizza}. In contrast, the first-stage network only keeps bboxes with high detection confidence (\textit{e}.\textit{g}., \texttt{knife}, \texttt{book}, and \texttt{cup}) as proposals, but actually misses the critical referent \texttt{person} (red bbox).


In this paper, we propose a novel algorithm Ref-NMS, to rectify the mismatch of detected proposals at the conventional first stage. In particular, for each expression, Ref-NMS regards all nouns in the expression as critical objects, and introduces a lightweight relatedness module to predict a probability score for each proposal to be a critical object. The higher predicted score denotes the higher relevance between a proposal and the expression. Then, we fuse the relatedness scores and classification scores, and exploit the fused scores as the suppression criterion in Non-Maximum Suppression (NMS). After NMS, we can filter out the proposals with little relevance to the expression. Finally, all proposals and the expression are fed into the second-stage grounding network, to obtain the referent prediction.


We demonstrate the significant performance gains of Ref-NMS on three challenging REG benchmarks. It's worth noting that the Ref-NMS can be generalized and easily integrated into any state-of-the-art two-stage method to further boost its performance on both REC and RES. Our method is robust and efficient, opening the door for many downstream applications such as multimodal summarization.

\section{Related Work}

\noindent\textbf{Referring Expression Comprehension (REC).} The current overwhelming majority of REC methods are in a two-stage manner~\cite{zhang2018grounding,zhuang2018parallel,wang2019neighbourhood,yang2019dynamic,yu2018mattnet,niu2019variational}: proposal generation and referent grounding. To the best of our knowledge, existing two-stage works all focus on the second stage. Specifically, they tend to design a more explainable reasoning process by structural modeling~\cite{yu2018mattnet,liu2019improving,liu2019learning,hong2019learning}, or more effective multi-modal interaction mechanism~\cite{yang2019dynamic,wang2019neighbourhood}. However, their performance is strictly limited by the proposals from the first stage. Recently, another emerging direction to solve REC is in a one-stage manner~\cite{chen2018real,yang2019fast,liao2020real}. Although one-stage methods achieve faster inference speed empirically, they come at a cost of lost interpretability and poor performance in composite expressions. In this paper, we focus on the two-stage REC methods, and try to rectify the overlooked mismatch.

Phrase grounding~\cite{akbari2019multi,dogan2019neural} is another closely related task to the REC. Similarly, there are two types of solutions: proposal-free and proposal-driven methods. In theory, Ref-NMS can also be applied to the proposal-driven methods and further boost their performance. We leave this for future works.

\noindent\textbf{Referring Expression Segmentation (RES).} Unlike REC, most of RES works are one-stage methods~\cite{hu2016segmentation,liu2017recurrent,shi2018key,margffoy2018dynamic,li2018referring}. They typically resort to FCN~\cite{long2015fully} and utilize a "concatenation-convolution" design to combine the two different modalities: they first concatenate the expression feature with visual features at each location, and then use several conv-layers to fuse the multimodal features for mask generation. To further improve mask qualities, they usually enhance their backbones with more effective features by multi-scale feature fusion~\cite{margffoy2018dynamic}, feature progressive refinement~\cite{li2018referring,chen2019see}, or novel attention mechanisms~\cite{shi2018key,ye2019cross}. Besides, with the development of two-stage instance segmentation networks (\textit{e}.\textit{g}., Mask R-CNN~\cite{he2017mask}), two-stage REC methods can be extended to solve the RES problem simply by replacing the object detection network at the second stage to an instance segmentation network. Analogously, Ref-NMS can be easily integrated into any two-stage RES method.

\noindent\textbf{Non-Maximum Suppression (NMS).} NMS is a de facto standard post-processing step adopted by numerous modern object detectors, which removes duplicate bboxes based on detection confidence. Except for the most prevalent GreedyNMS, multiple improved variants have been proposed in the last few years. Generally, they can be categorized into three groups: 1) Criterion-based~\cite{jiang2018acquisition,tychsen2018improving,tan2019learning,yang2019learning}: they utilize other scores instead of classification confidence as the criterion to remove bboxes by NMS, \textit{e}.\textit{g}., IoU scores. 2) Learning-based~\cite{hosang2017learning,hu2018relation}: they directly learn an extra network to remove duplicate bboxes. 3) Heuristic-based~\cite{bodla2017soft,liu2019adaptive}: they dynamically adjust the thresholds for suppression according to some heuristic rules. In this paper, we are inspired by the criterion-based NMS, and design the Ref-NMS, which uses both expression relatedness and detection confidence as the criterion.

\begin{figure}[t]
	\centering
	\includegraphics[width=0.98\linewidth]{images/two-stage-models.pdf}
	\caption{\textbf{Upper}: A typical two-stage REG framework, which uses heuristic filter rules to obtain expression-agnostic proposals at the first-stage, and feeds them into the second stage for referent grounding. \textbf{Below}: The Ref-NMS model can generate expression-aware proposals by considering the expression at the first stage.}
	\label{fig:two-stage-models}
\end{figure}

\section{Approach}

In this section, we first revisit the typical two-stage REG framework in Section~\ref{sec:revisit}, and then introduce the details about Ref-NMS in Section~\ref{sec:ref-nms}, including the relatedness module and training objectives.

\subsection{Revisiting Two-Stage REG Framework}\label{sec:revisit}
The two-stage framework is the most prevalent pipeline for REG. As shown in Figure~\ref{fig:two-stage-models}, it consists of two separate stages: proposal generation at the first-stage and referent grounding at the second-stage.

\noindent\textbf{Proposal Generation.} Given an image, current two-stage methods always resort to a well pre-trained detector to obtain a set of initially detected bboxes, and utilize an NMS to remove duplicate bboxes. However, even after NMS operation, there are still thousands of bboxes left (\textit{e}.\textit{g}., each image in RefCOCO has an average of 3,500 detections). To relieve the burden of the following referent grounding step, all existing works further filter these bboxes based on their detection confidences. Although this heuristic filter rule can reduce the number of proposals, it also results in a drastic drop in the recall of both the referent and contextual objects (Detailed results are reported in Table~\ref{tab:hit-rate}.).

\noindent\textbf{Referent Grounding.} In the training phase, two-stage methods usually use the ground-truth regions in COCO as proposals, and the number is quite small (\textit{e}.\textit{g}., each image in RefCOCO has an average of 9.84 ground-truth regions). For explainable grounding, state-of-the-art two-stage methods always compose these proposals into graph~\cite{yang2019dynamic,wang2019neighbourhood} or tree~\cite{liu2019learning,hong2019learning} structures, \textit{i}.\textit{e}., as the number of proposals increases linearly, the number of computation increases exponentially. Therefore, in the test phase, it is a must for them to filter detections at the first stage.

\begin{figure}[t]
	\centering
	\includegraphics[width=1.0\linewidth]{images/architecture.pdf}
	\caption{The overview of Ref-NMS model. Given an image, the model uses a pre-trained detector to generate thousands of initial bboxes. Then, hundreds of filtered bboxes and the expression are fed into the relatedness module to predict their relatedness scores. Lastly, the fused scores are used as the suppression criterion of NMS.} 
	\label{fig:architecture}
\end{figure}

\subsection{Ref-NMS}\label{sec:ref-nms}

\subsubsection{Relatedness Module}

An overview of the Ref-NMS model is shown in Figure~\ref{fig:architecture}. The core of Ref-NMS is the relatedness module. Given an image and a pre-trained detector, we can receive thousands of initial bboxes. To reduce the computation of the relatedness module, we first use a threshold $\delta$ to filter the bboxes with classification confidence, and obtain a filtered bbox set $\mathcal{B}$. For each bbox $\bm{b}_i \in \mathcal{B}$, we use a region visual encoder $e_v$ (\textit{i}.\textit{e}., an RoI Pooling layer and a convolutional head network) to extract the bbox feature $\bm{v}_i \in \mathbb{R}^v$. Meanwhile, for the referring expression $Q$, we use an expression encoder $e_q$ (\textit{i}.\textit{e}., a Bi-GRU) to output a set of word features $\{\bm{w}_1, ..., \bm{w}_{|Q|}\}$, where $\bm{w}_j \in \mathbb{R}^q$ is the $j$-th word feature. For each bbox $\bm{b}_i$, we use a soft-attention mechanism to calculate a unique expression feature $\bm{q}_i$ by:
\begin{equation} \label{eq:1}
\bm{v}^a_i = \text{MLP}_a (\bm{v}_i), \quad a_{ij} = \text{FC}_s ([\bm{v}^a_i; \bm{w}_j]), \quad \alpha_{ij} = \text{softmax}_j(a_{ij}),  \quad \bm{q}_i = \textstyle{\sum_j} \alpha_{ij} \bm{w}_j,
\end{equation}
where $\text{MLP}_a$ is a two-layer MLP mapping $\bm{v}_i \in \mathbb{R}^v$ to $\bm{v}^a_i \in \mathbb{R}^q$, $\text{FC}_s$ is a FC layer to calculate the similarity between bbox feature $\bm{v}^a_i$ and word feature $\bm{w}_j$, and $[;]$ is a concatenation operation. Then, we combine the two modal features and predict the relatedness score $r_i$:
\begin{equation}
\bm{v}^b_i = \text{MLP}_b(\bm{v}_i), \quad \bm{m}_i = \text{L2Norm}(\bm{v}^b_i \odot \bm{q}_i), \quad \hat{r}_i = \text{FC}_r(\bm{m}_i), \quad r_i = \text{sigmoid}(\hat{r}_i),
\end{equation}
where $\text{MLP}_b$ is a two-layer MLP mapping $\bm{v}_i \in \mathbb{R}^v$ to $\bm{v}^b_i \in \mathbb{R}^q$, $\odot$ is the element-wise multiplication, L2Norm represents $l_2$ normalization, and $\text{FC}_r$ is a FC layer mapping $\bm{m}_i \in \mathbb{R}^q$ to $\hat{r}_i \in \mathbb{R}$.

\noindent\textbf{Score Fusion.} After obtaining the relatedness score $r_i$ for bbox $\bm{b}_i$, we multiple $r_i$ with the classification confidence $c_i$ for bbox $\bm{b}_i$ from the original detector, and utilize the multiplication of two scores $s_i$ as the suppression criterion of the NMS operation, \textit{i}.\textit{e}., $s_i = r_i \times c_i$.

\subsubsection{Training Objectives for Ref-NMS}
To learn the relatedness score for each bbox, we need the ground-truth annotations for all mentioned instances (\textit{i}.\textit{e}., both referent and contextual objects) in the expression. However, current REG datasets only have annotations about the referent. Thus, we need to generate pseudo ground-truths for contextual objects. Specifically, we first assign POS tags to each word in the expression using the spaCy POS tager~\cite{spacy2} and extract all nouns in the expression. Then, we calculate the cosine similarity between GloVe embeddings~\cite{pennington2014glove}  of extracted nouns and categories of ground-truth regions in COCO\footnote{Two-stage methods always use a detector pretrained on COCO. Thus, we don't use extra or more annotations.}. Lastly, we use threshold $\gamma$ to filter regions as the pseudo ground-truths. 

In the training phase, we regard all the pseudo ground-truth bboxes and annotated referent bboxes as foreground bboxes. And we use two following types of training objectives for the Ref-NMS:

\noindent\textbf{Binary XE Loss.} For each bbox $\bm{b}_i \in \mathcal{B}$, if it has a high overlap (\textit{i}.\textit{e}., IoU>0.5) with any foreground bbox, its ground-truth relatedness score $r^*$ is set to 1, otherwise $r^*=0$. Then the relatedness score prediction becomes a binary classification problem. We can use the binary cross-entropy (XE) loss as the training objective:
\begin{equation}
\begin{aligned}
L = -\frac{1}{|\mathcal{B}|} \textstyle{\sum^{|\mathcal{B}|}_{i=1}} r^*_i \log(r_i) + (1-r^*_i) \log(1-r_i).
\end{aligned}
\end{equation}

\noindent\textbf{Ranking Loss.} Generally, if a bbox has a higher IoU with foregound bboxes, the relatedness between the bbox and expression should be higher, \textit{i}.\textit{e}., we can use the ranking loss as the training objectives: 
\begin{equation}
L = \frac{1}{N} \textstyle{\sum_{(\bm{b}_i, \bm{b}_j), \rho_i < \rho_j}} \max (0, r_i - r_j + \alpha),
\end{equation}
where $\rho_i$ denotes the largest IoU value between bbox $\bm{b}_i$ and foreground bboxes, $N$ is the total number of pos-neg training pairs, and $\alpha$ is a constant to control the ranking margin, set as 0.1. To select the pos-neg pair $(\bm{b}_i, \bm{b}_j)$, we follow the sampling-after-splitting strategy~\cite{tan2019learning}. Specifically, we first divide the bbox set $\mathcal{B}$ into 6 subsets based on a quantization $q$-value: $q_i = \lceil \text{max}(0, \rho_i - 0.5)/0.1  \rceil$, \textit{i}.\textit{e}., the bbox with higher IoU value has larger $q$-value. Then, all bboxes with $\rho > 0.5$ are selected as positive samples. For each positive sample, we rank the top-$h$ bboxes as negative samples based on predicted relatedness scores from the union of subsets with smaller $q$-value.


\subsubsection{Implementation Details}

\noindent\textbf{Language Settings.} We build a vocabulary for each dataset by filtering the words less than 2 times, and exploit the 300-d GloVe embeddings~\cite{pennington2014glove}  as the initialization of word embeddings. We use an "unk" symbol to replace all words out of the vocabulary. The largest length of sentences is set to 10 for RefCOCO and RefCOCO+, 20 for RefCOCOg. The hidden size of the encoder $e_q$ is set to 256.
\noindent\textbf{Visual Settings.} For visual encoder $e_v$, we use the same head network of the Mask R-CNN with ResNet-101 backbone\footnote{https://github.com/lichengunc/mask-faster-rcnn} as prior works~\cite{yu2018mattnet}, and utilize the pre-trained weights as initialization. The weights of the original detector (\textit{i}.\textit{e}., the gray part in Figure~\ref{fig:architecture} ) are fixed during the training phase.
\noindent\textbf{Parameter Settings.} The whole model is trained with Adam optimizer. The initial learning rate is initialized to 4e-4 and 5e-3 for the head network and the rest of network. We set the batch size as 8. The thresholds $\delta$ and $\gamma$ are set to 0.05 and 0.4, respectively. For ranking loss, the top-h is set to 100.

\section{Experiments}
\subsection{Datasets and Evaluation Metrics}

\noindent\textbf{Datasets.} We evaluate the Ref-NMS on three challenging REG datasets collected from the COCO images~\cite{lin2014microsoft}: 1) \textbf{RefCOCO}~\cite{yu2016modeling}:  It consists of 142,210 referring expressions for 50,000 objects in 19,994 images. These expressions are collected in an interactive game interface~\cite{kazemzadeh2014referitgame}, and the average length of each expression is 3.5 words. All expression-referent pairs are split into train, validation, testA, and testB sets. The testA set contains the images with multiple people and the testB set contains the images with multiple objects. 2) \textbf{RefCOCO+}~\cite{yu2016modeling}: It consists of 141,564 referring expressions for 49,856 objects in 19,992 images. Similar to RefCOCO, these expressions are collected from the same game interface, and have train, val, testA, and testB splits. Compared to RefCOCO, these expressions don't include absolute locations. 3) \textbf{RefCOCOg}~\cite{mao2016generation}: It consists of 104,560 referring expressions for 54,822 objects in 26,711 images. These expressions are collected in a non-interactive way, and the average length of each expression is 8.4 words. We follow the same split as~\cite{nagaraja2016modeling}.

\noindent\textbf{Evaluation Metrics.} For the REC task, we use the top-1 accuracy as the evaluation metric. When the IoU between bbox and ground truth is larger than 0.5, the prediction is correct. For the RES task, we use the overall IoU and Pr@X (the percentage of samples with IoU higher than X)\footnote{Due to the limited space, all RES results with the Pr@X metric are provided in the supplementary materials.} as metrics.


\subsection{Ablation Studies}

\subsubsection{Recall Analyses of the Referent and Contextual Objects}

\noindent\textbf{Settings.} To evaluate the effectiveness of the Ref-NMS to improve the recall of both referent and contextual objects, we compare Ref-NMS with plain GreedyNMS used in the baseline detector (\textit{i}.\textit{e}., ResNet-101 based Mask R-CNN). Since we only have annotated ground-truth bboxes for the referent, we calculate the recall of pseudo ground-truths to approximate the recall of contextual objects. The results are reported in Table~\ref{tab:hit-rate}, and more detailed results are provided in the supplementary materials.

\noindent\textbf{Results.} From Table~\ref{tab:hit-rate}, we have the following observations. When using top-100 bboxes as proposals, all three methods can achieve near-perfect recall ($\approx$ 97\%) for the referent and acceptable recall ($\approx$ 90\%) for the contextual objects, respectively. However, when the number of proposals decreases to a very small number (\textit{e}.\textit{g}., <10 in the real case), the recall of the baseline all drops significantly (\textit{e}.\textit{g}., 15.81\% for the referent and 20.34\% for the contextual objects on RefCOCO testB). In contrast, Ref-NMS can help narrow the gap over all dataset splits. Especially, the improvement is more obvious in the testB set (\textit{e}.\textit{g}., 7.51\% and 4.85\% absolute gains for the recall of referent on RefCOCO and RefCOCO+), where the categories of referents are more diverse and the recalls are relatively lower.


\addtolength{\tabcolsep}{-3pt}    
\begin{table}[t]
	\small
	\begin{center}
		\scalebox{0.82}{
			\begin{tabular}{| c| c | c c c | c c c | c c || c c c | c c c | c c |}
				\hline
				 & \multirow{3}{*}{Ref-NMS} & \multicolumn{8}{c||}{Referent} & \multicolumn{8}{c|}{Contextual Objects} \\
				\cline{3-18}
				& & \multicolumn{3}{c|}{RefCOCO} & \multicolumn{3}{c|}{RefCOCO+} & \multicolumn{2}{c||}{RefCOCOg} & \multicolumn{3}{c|}{RefCOCO} & \multicolumn{3}{c|}{RefCOCO+} & \multicolumn{2}{c|}{RefCOCOg} \\
				& & val & testA & testB &  val & testA & testB & val & test & val & testA & testB &  val & testA & testB & val & test \\
				\hline
				\parbox[t]{2mm}{\multirow{3}{*}{\rotatebox[origin=c]{90}{N=100}}} &  & 97.60 & 97.81 & 96.58 & 97.79 & 97.78 & 96.99 & 97.18 & 96.91 & 90.14 & 89.85 & 90.53 & 89.53 & 88.47 & 90.69 & 90.56 & 90.30 \\
				& B & 97.75 & 98.59 & 97.08 & 97.96 & 98.39 & 97.50 & 97.61 & 97.44 & 90.38 & 90.31 & 90.64 & 89.67 & 88.88 & 91.04 & 90.36 & 90.37  \\
				& R & 97.62 & 98.02 & 96.78 & 97.71 & 98.06 & 97.14 & 97.18 & 97.08 & 90.22 & 89.83 & 90.63 & 89.70 & 88.62 & 90.71 & 90.67 & 90.30  \\
				\hline
				\parbox[t]{2mm}{\multirow{3}{*}{\rotatebox[origin=c]{90}{Real}}} &  & 88.84 & 93.99 & 80.77 & 90.71 & 94.34 & 84.11 & 87.83 & 87.88 & 74.97 & 78.60 & 70.19 & 76.34 & 77.45 & 73.52 & 75.69 & 75.87 \\
				& B & \textbf{92.51} & \textbf{95.56} & \textbf{88.28} & \textbf{93.42} & \textbf{95.86} & \textbf{88.95} & \textbf{90.28} & \textbf{90.34} & \textbf{78.75} & \textbf{80.14} & \textbf{76.47} & \textbf{78.44} & \textbf{78.82} & \textbf{77.49} & 76.12 & 76.57  \\
				& R & 90.50 & 94.75 & 83.87 & 91.62 & 95.14 & 86.42 & 89.01 & 88.96 & 76.79 & 79.12 & 72.99 & 77.66 & 78.44 & 75.59 & \textbf{76.68} & \textbf{76.73}  \\
				\hline
			\end{tabular}
		}
	\end{center}
	\caption{Recall (\%) of the referent and contextual objects. The baseline detector is the ResNet-101 based Mask R-CNN with plain GreedyNMS. B denotes the Ref-NMS with binary XE loss, R denotes the Ref-NMS with ranking loss. Real denotes the real case used in the state-of-the-art two-stage methods.}
	\vspace{-1em}
	\label{tab:hit-rate}
\end{table}
\addtolength{\tabcolsep}{3pt}

\addtolength{\tabcolsep}{-3pt}    
\begin{table}[t]
	\small
	\begin{center}
		\scalebox{0.81}{
			\begin{tabular}{| l | c c c | c c c | c c || c c c | c c c | c c |}
				\hline
				\multirow{3}{*}{Models} & \multicolumn{8}{c||}{Referring Expression Comprehension} & \multicolumn{8}{c|}{Referring Expression Segmentation} \\
				\cline{2-17}
				& \multicolumn{3}{c|}{RefCOCO} & \multicolumn{3}{c|}{RefCOCO+} & \multicolumn{2}{c||}{RefCOCOg} & \multicolumn{3}{c|}{RefCOCO} & \multicolumn{3}{c|}{RefCOCO+} & \multicolumn{2}{c|}{RefCOCOg} \\
				& val & testA & testB &  val & testA & testB & val & test & val & testA & testB &  val & testA & testB & val & test \\
				\hline
				MAttNet~\cite{yu2018mattnet}  & 76.65 & 81.14 & 69.99 & 65.33 & 71.62 & 56.02 & 66.58 & 67.27 & 56.51 & 62.37 & 51.70 & 46.67 & 52.39 & 40.08 & 47.64 & 48.61 \\
				MAttNet$^\dagger$ & 76.92 & 81.19 & 69.58 & 65.90 & \textcolor{red}{\textbf{71.53}} & 57.23 & 67.52 & 67.55 & 57.14  &  62.34  & 51.48  & 47.30  & \textcolor{red}{\textbf{52.37}} & 41.14 & 48.28  & 49.01 \\
				~~+Ref-NMS B & \textcolor{red}{\textbf{78.82}} & \textcolor{red}{\textbf{82.71}} & \textcolor{red}{\textbf{73.94}} & \textcolor{red}{\textbf{66.95}} & 71.29 & \textcolor{red}{\textbf{58.40}} & \textcolor{blue}{68.89} & \textcolor{red}{\textbf{68.67}} & \textcolor{red}{\textbf{59.75}} & \textcolor{red}{\textbf{63.48}} & \textcolor{red}{\textbf{55.66}} & \textcolor{red}{\textbf{48.39}} & 51.57 & \textcolor{red}{\textbf{42.56}} & \textcolor{blue}{49.54} & \textcolor{red}{\textbf{50.38}} \\
				~~+Ref-NMS R & \textcolor{blue}{77.98} & \textcolor{blue}{82.02} & \textcolor{blue}{71.64} & \textcolor{blue}{66.64} & \textcolor{blue}{71.36} & \textcolor{blue}{58.01} & \textcolor{red}{\textbf{69.16}} & \textcolor{blue}{67.63} & \textcolor{blue}{58.32} & \textcolor{blue}{62.96} & \textcolor{blue}{53.68} & \textcolor{blue}{47.87} & \textcolor{blue}{51.85} & \textcolor{blue}{41.41} & \textcolor{red}{\textbf{50.13}} & \textcolor{blue}{49.07} \\
				\hline
				NMTree~\cite{liu2019learning} & 76.41 & 81.21 & 70.09 & 66.46 & 72.02 & 57.52 & 65.87 & 66.44 & 56.59 & 63.02 & 52.06 & 47.40 & 53.01 & 41.56 & 46.59 & 47.88 \\
				NMTree$^\dagger$ & 76.54 & 81.32 & 69.66 & 66.65 & 71.48 & 57.74 & 65.65 & 65.94 & 56.99 & \textcolor{blue}{62.88} & 51.90 & 47.75 & \textcolor{blue}{52.36} & 41.86 & 46.19 & 47.41 \\
				~~+Ref-NMS B & \textcolor{red}{\textbf{78.67}} & \textcolor{red}{\textbf{82.09}} & \textcolor{red}{\textbf{73.78}} & \textcolor{red}{\textbf{67.15}} & \textcolor{blue}{71.76} & \textcolor{blue}{58.70} & \textcolor{red}{\textbf{67.30}} & \textcolor{red}{\textbf{66.93}} & \textcolor{red}{\textbf{59.95}} & \textcolor{red}{\textbf{63.25}} & \textcolor{red}{\textbf{55.64}} & \textcolor{red}{\textbf{48.68}} & 52.30 & \textcolor{red}{\textbf{42.64}} & \textcolor{red}{\textbf{48.14}} & \textcolor{red}{\textbf{48.59}} \\
				~~+Ref-NMS R &  \textcolor{blue}{77.81} & \textcolor{blue}{81.69} & \textcolor{blue}{71.78} & \textcolor{blue}{67.03} & \textcolor{red}{\textbf{71.78}} & \textcolor{red}{\textbf{58.79}} & \textcolor{blue}{66.81} & \textcolor{blue}{66.31} & \textcolor{blue}{58.42} & 62.69 & \textcolor{blue}{53.60} & \textcolor{blue}{48.27} & \textcolor{red}{\textbf{52.65}} & \textcolor{blue}{42.18} & \textcolor{blue}{47.72} & \textcolor{blue}{48.09} \\
				\hline
				CM-A-E~\cite{liu2019improving}  & 78.35 &	83.14 & 71.32 &	68.09 &	73.65 &	58.03 & 67.99 & 68.67  & --  & -- & --  & --  & --  & --  & --  & -- \\
				CM-A-E$^\dagger$ & 78.35 & 83.12 & 71.32 & 68.19 & 73.04 & 58.27 & 69.10 & 69.20 & 58.23  & 64.60 & 53.14  & 49.65  & \textcolor{red}{\textbf{53.90}}  & 41.77  & 49.10  & 50.72 \\
				~~+Ref-NMS B & \textcolor{red}{\textbf{80.70}} & \textcolor{red}{\textbf{84.00}} & \textcolor{red}{\textbf{76.04}} & \textcolor{blue}{68.25} & \textcolor{red}{\textbf{73.68}} & \textcolor{red}{\textbf{59.42}} & \textcolor{red}{\textbf{70.55}} & \textcolor{red}{\textbf{70.62}} & \textcolor{red}{\textbf{61.46}} & \textcolor{red}{\textbf{65.55}} & \textcolor{red}{\textbf{57.41}} & \textcolor{blue}{49.76} & \textcolor{blue}{53.84} & \textcolor{red}{\textbf{42.66}} & \textcolor{red}{\textbf{51.21}} & \textcolor{red}{\textbf{51.90}}  \\
				~~+Ref-NMS R & \textcolor{blue}{79.55} & \textcolor{blue}{83.58} & \textcolor{blue}{73.62} & \textcolor{red}{\textbf{68.51}} & \textcolor{blue}{73.14} & \textcolor{blue}{58.38} & \textcolor{blue}{69.77} & \textcolor{blue}{70.01} & \textcolor{blue}{59.72} & \textcolor{blue}{64.87} & \textcolor{blue}{55.63} & \textcolor{red}{\textbf{49.86}} & 52.62 & \textcolor{blue}{41.87} & \textcolor{blue}{50.13} & \textcolor{blue}{51.44} \\
				\hline
			\end{tabular}
		}
	\end{center}
	\caption{Performances of different architectures on REC and RES. The metrics are top-1 accuracy (\%) for REC and overall IoU (\%) for RES. All baselines use the ResNet-101 based Mask R-CNN as first-stage networks. The \textcolor{red}{\textbf{best}} and \textcolor{blue}{second best} methods under each setting are marked in according formats. $^\dagger$ denotes our implementation.}
	\label{tab:model-agnostic}
\end{table}
\addtolength{\tabcolsep}{3pt} 

\subsubsection{Architecture Agnostic}

\noindent\textbf{Settings.} Since the Ref-NMS model is agnostic to the second stage network, it can be easily integrated into any referent grounding architectures. To evaluate the effectiveness and generality of Ref-NMS to boost the grounding performance of different backbones, we incorporated the Ref-NMS into multiple state-of-the-art two-stage methods including: \textbf{MAttNet}~\cite{yu2018mattnet} , \textbf{NMTree}~\cite{liu2019learning}, and \textbf{CM-A-E}~\cite{liu2019improving}. All results are reported in Table~\ref{tab:model-agnostic}.

\noindent\textbf{Results.} From Table~\ref{tab:model-agnostic}, we can observe that both variants of Ref-NMS can consistently improve the grounding performance over all three backbones on both REC and RES. The improvement is more significant on the testB set (\textit{e}.\textit{g}., 4.72\% and 3.23\% absolute performance gains for CM-A-E in REC and RES), which meets our expectation, \textit{i}.\textit{e}., the improvements of grounding performance have a strong positive correlation with the improvements of the recall of critical objects. Compared between two variants of Ref-NMS, in most of cases, Ref-NMS B achieves better grounding performance. We argue that the reason may come from the imbalance of positive and negative samples in each level.


\begin{table}[t]
	\small
	\begin{center}
		\scalebox{0.9}{
			\begin{tabular}{|l | l | c | c c c | c c c | c c|}
				\hline
					& \multirow{2}{*}{Models} & \multirow{2}{*}{Backbone} & \multicolumn{3}{c|}{RefCOCO} & \multicolumn{3}{c|}{RefCOCO+} & \multicolumn{2}{c|}{RefCOCOg}  \\
					&  &  & val & testA & testB & val & testA & testB & val & test \\
				\hline
					\parbox[t]{2mm}{\multirow{3}{*}{\rotatebox[origin=c]{90}{one-s.}}} & SSG~\cite{chen2018real}~\tiny{arXiv18} &  darknet53 & -- & 76.51 & 67.50 & -- & 62.14 & 49.27 & 58.80 & -- \\  
					& FAOA~\cite{yang2019fast}~\tiny{ICCV19} & darknet53 & 71.15 & 74.88 & 66.32 & 56.86 & 61.89 & 49.46 & 59.44 & 58.90 \\
					& RCCF~\cite{liao2020real}~\tiny{CVPR20} & dla34 & -- & 81.06 & 71.85 & -- & 70.35 & 56.32 & -- & 65.73 \\
				\hline
				\hline
					\parbox[t]{2mm}{\multirow{10}{*}{\rotatebox[origin=c]{90}{two-stage}}} & VC~\cite{zhang2018grounding}~\tiny{CVPR18} & vgg16 & -- & 73.33 & 67.44 & -- & 58.40 & 53.18 & -- & -- \\
					& ParalAttn~\cite{zhuang2018parallel}~\tiny{CVPR18} & vgg16 & -- & 75.31 & 65.52 & -- & 61.34 & 50.86 & -- & -- \\
					& LGRANs~\cite{wang2019neighbourhood}~\tiny{CVPR19} & vgg16 & -- & 76.60 & 66.40 & -- & 64.00 & 53.40 & -- & -- \\
					& DGA~\cite{yang2019dynamic}~\tiny{ICCV19} & vgg16 &	-- & 78.42 & 65.53 & -- & 69.07 & 51.99 & -- & 63.28 \\ 
					& NMTree~\cite{liu2019learning}~\tiny{ICCV19} & vgg16 &	71.65 &	74.81 &	67.34 &	58.00 &	61.09 & 53.45 & 61.01 & 61.46 \\
					& MAttNet~\cite{yu2018mattnet}~\tiny{CVPR18} &	res101 & 76.65 & 81.14 & 69.99 & 65.33 & 71.62 & 56.02 & 66.58 & 67.27 \\
					& RvG-Tree~\cite{hong2019learning}~\tiny{TPAMI19} & res101 & 75.06 & 78.61 & 69.85 & 63.51 & 67.45 & 56.66 & 66.95 & 66.51 \\
					& NMTree~\cite{liu2019learning}~\tiny{ICCV19} & res101 & 76.41 & 81.21 & 70.09 & 66.46 & 72.02 & 57.52 & 65.87 & 66.44 \\
					& CM-A-E~\cite{liu2019improving}~\tiny{CVPR19}	& res101 &	78.35 &	83.14 & 71.32 &	68.09 &	73.65 &	58.03 & 67.99 & 68.67 \\ 
					& \cellcolor{mygray-bg}{\textbf{CM-A-E+Ref-NMS}} & \cellcolor{mygray-bg}{res101} &
					\cellcolor{mygray-bg}{\textbf{80.70}} & \cellcolor{mygray-bg}{\textbf{84.00}} & \cellcolor{mygray-bg}{\textbf{76.04}} & \cellcolor{mygray-bg}{\textbf{68.25}} & \cellcolor{mygray-bg}{\textbf{73.68}} & \cellcolor{mygray-bg}{\textbf{59.42}} & \cellcolor{mygray-bg}{\textbf{70.55}} & \cellcolor{mygray-bg}{\textbf{70.62}} \\
				\hline
			\end{tabular}
		}
	\end{center}
	\caption{Top-1 accuracies (\%) of state-of-the-art models on referring expression comprehension.}
	\vspace{-1.5em}
	\label{tab:sota_rec}
\end{table}

\begin{table}[t]
	\small
	\begin{center}
		\scalebox{0.9}{
			\begin{tabular}{| l | l | c c c | c c c | c c |}
				\hline
				& \multirow{2}{*}{Models} & \multicolumn{3}{c|}{RefCOCO} & \multicolumn{3}{c|}{RefCOCO+} & \multicolumn{2}{c|}{RefCOCOg}  \\
				& & val & testA & testB & val & testA & testB &  val & test \\
				\hline
					\parbox[t]{2mm}{\multirow{6}{*}{\rotatebox[origin=c]{90}{one-stage}}} & RMI~\cite{liu2017recurrent}~\tiny{ICCV17} & 41.17 & 41.35 & 41.87 & 28.26 & 29.16 & 26.86 & -- & -- \\
					& DMN~\cite{margffoy2018dynamic}~\tiny{ECCV18} & 49.78 & 54.83 & 45.13 & 38.88 & 44.22 & 32.29 & -- & -- \\ 
					& RRN~\cite{li2018referring}~\tiny{CVPR18} & 55.33 & 57.26 & 53.95 & 39.75 & 42.15 & 36.11 & -- & -- \\
					& CMSA~\cite{ye2019cross}~\tiny{CVPR19} & 58.32 & 60.61 & 55.09 & 43.76 & 47.60 & 37.89 & -- & -- \\
					& STEP~\cite{chen2019see}~\tiny{ICCV19} & 60.04 & 63.46 & 57.97 & 48.19 & 52.33 & 40.41 & -- & -- \\
					& BRINet~\cite{hu2020bi}~\tiny{CVPR20} & 60.98 & 62.99 & 59.21 & 48.17 & 52.32 & 42.41 & -- & -- \\
				\hline
				\hline
					\parbox[t]{2mm}{\multirow{4}{*}{\rotatebox[origin=c]{90}{two-stage}}}  & MAttNet~\cite{yu2018mattnet}~\tiny{CVPR18} & 56.51 & 62.37 & 51.70 & 46.67 & 52.39 & 40.08 & 47.64 & 48.61 \\
					& NMTree~\cite{liu2019learning}~\tiny{ICCV19} & 56.59 & 63.02 & 52.06 & 47.40 & 53.01 & 41.56 & 46.59 & 47.88 \\
					& CM-A-E$^\dagger$~\cite{liu2019improving}~\tiny{CVPR19} & 58.23  & 64.60 & 53.14  & 49.65  & \textbf{53.90}  & 41.77  & 49.10  & 50.72 \\
					& \cellcolor{mygray-bg}{\textbf{CM-A-E+Ref-NMS}} &  \cellcolor{mygray-bg}{\textbf{61.46}} & \cellcolor{mygray-bg}{\textbf{65.55}} & \cellcolor{mygray-bg}{\textbf{57.41}} & \cellcolor{mygray-bg}{\textbf{49.76}} & \cellcolor{mygray-bg}{53.84} & \cellcolor{mygray-bg}{\textbf{42.66}} & \cellcolor{mygray-bg}{\textbf{51.21}} & \cellcolor{mygray-bg}{\textbf{51.90}} \\ 
					\hline
			\end{tabular}
		}
	\end{center}
	\caption{Overall IoU (\%) of state-of-the-art models on referring expression segmentation. All methods utilize ResNet-101 as backbone. $^\dagger$ denotes that the results are from our reimplementation using official released codes.}
	\vspace{-1.5em}
	\label{tab:sota_res}
\end{table}

\subsection{Comparisons with State-of-the-Arts}


We incorporate Ref-NMS (with the binary XE loss) into the model CM-A-E~\cite{liu2019improving}, which is dubbed \textbf{CM-A-E+Ref-NMS}, and compare it against the state-of-the-art methods on both REC and RES.

\textbf{Settings.} For the state-of-the-art REC methods, from the viewpoint of one-stage and two-stage, we can group them into: 1) Two-stage methods: \textbf{VC}~\cite{zhang2018grounding}, \textbf{ParalAttn}~\cite{zhuang2018parallel}, \textbf{LGRANs}~\cite{wang2019neighbourhood}, \textbf{DGA}~\cite{yang2019dynamic}, \textbf{NMTree}~\cite{liu2019learning}, \textbf{MAttNet}~\cite{yu2018mattnet}, \textbf{RvG-Tree}~\cite{hong2019learning}, and \textbf{CM-A-E}~\cite{liu2019improving}; 2) one-stage methods: \textbf{SSG}~\cite{chen2018real}, \textbf{FAOA}~\cite{yang2019fast}, and \textbf{RCCF}~\cite{liao2020real}. Analogously, for the state-of-the-art RES methods, we group them into: 1) Two-stage methods: \textbf{MAttNet}~\cite{yu2018mattnet}, \textbf{NMTree}~\cite{liu2019learning}, and \textbf{CM-A-E}~\cite{liu2019improving}; 2) one-stage methods: \textbf{RMI}~\cite{liu2017recurrent}, \textbf{DMN}~\cite{margffoy2018dynamic}, \textbf{RRN}~\cite{li2018referring}, \textbf{CMSA}~\cite{ye2019cross}, \textbf{STEP}~\cite{chen2019see}, and \textbf{BRINet}~\cite{hu2020bi}. 


\textbf{Results.} The REC and RES results are reported in Table~\ref{tab:sota_rec} and Table~\ref{tab:sota_res}. For the REC, CM-A-E+Ref-NMS achieves a new record-breaking performance that is superior to all existing REC methods on three benchmarks. Ref-NMS improves the strong baseline CM-A-E with an average of 2.64\%, 0.53\%, and 2.26\% absolute performance gains over RefCOCO, RefCOCO+, and RefCOCOg, respectively. For the RES, CM-A-E+Ref-NMS achieves a new state-of-the-art performance of two-stage methods over most of the dataset splits. Similarly, Ref-NMS improves CM-A-E with an average of 2.82\%, 0.31\%, and 1.65\% absolute performance gains over the three datasets.

\subsection{Qualitative Results}

We illustrate the qualitative results between CM-A-E+Ref-NMS  and baseline CM-A-E on REC in Figure~\ref{fig:visualization}. From the results in line (b), we can observe that Ref-NMS can assign high attention weights on more relevant words to individual referents (\textit{e}.\textit{g}., umbrella, man, and zebra). The results in line (c) show that the generated pseudo ground-truth bboxes can almost contain all contextual objects in the expression, except a few objects whose categories are far different from the categories of COCO (\textit{e}.\textit{g}., sweater, armrest, and grass). By comparing the results between line (d) and line (e), we have the following observations: 1) The baseline method always detects more false-positive proposals (\textit{i}.\textit{e}., the blue bboxes), and misses some critical objects (\textit{i}.\textit{e}., the red and green bboxes). Instead, Ref-NMS helps the model generate more expression-aware proposals. 2) Even for the failed cases in CM-A-E+Ref-NMS (\textit{i}.\textit{e}., the last two columns), Ref-NMS still generates more reasonable proposals (\textit{e}.\textit{g}., with less false positive proposals), and the grounding errors mainly come from the second stage.

\begin{figure}[t]
	\centering
	\includegraphics[width=1.0\linewidth]{images/visualization.pdf}
	\caption{Qualitative REC results on RefCOCOg showing comparisons between correct (green tick) and wrong referent grounds (red cross) by CM-A-E and CM-A-E+Ref-NMS. \textbf{(a):} The input image and referring expressions. \textbf{(b)}: The visualiation of word attention weights $\alpha$ (cf. Eq.~\eqref{eq:1}) for each referent object. \textbf{(c):} The annotated referent ground-truth bbox (red) and generated pseudo ground-truth bboxes for contextual objects (green). \textbf{(d)} and \textbf{(e)} denote the proposals and final grounding results from two methods. We only show the proposals and the final predicted referent bbox is illustrated in dash line. The denotations of bbox colors are as follows. \textbf{Red}: The bbox hits (IoU>0.5) the referent ground-truth bbox; \textbf{Green}: The bboxes hit the pseudo ground-truth bboxes; \textbf{Blue}: The false positive proposal predictions.}
	\label{fig:visualization}
\end{figure}

\section{Conclusions and Future Works}

In this paper, we focused on the two-stage referring expression grounding, and discussed the overlooked mismatch problem between the roles of proposals in different stages. Particularly, we proposed a novel approach dubbed Ref-NMS to calibrate this mismatch. Ref-NMS tackles the problem by considering the expression at the first stage, and learns a relatedness score between each detected proposal and the expression. The multiplication of the relatedness scores and classification scores serves as the suppression criterion for the NMS operation. Meanwhile, Ref-NMS is agnostic to the referent grounding step, and can be integrated into any state-of-the-art two-stage method. We validated the effectiveness through extensive comparative and ablative experiments. Moving forward, we plan to 1) extend Ref-NMS into bounding boxes from Selective Search or EdgeBox, instead of pre-trained detectors; and 2) apply Ref-NMS into other proposal-drive tasks which suffer from the same mismatch issue, \textit{e}.\textit{g}., phrase grounding, visual relationship detection, and VQA.


{\small
\bibliographystyle{plain}
