


\subsection{Temporal Attention Module}
Inspired by the non-local networks~\cite{wang2018non}, we propose a novel temporal attention module to refine the features of the current frame via aggregating information from other support frames. Different from the original non-local block, which aims to model the long-range dependencies by attention mechanism, our proposed temporal attention module focuses more on embedding information from other frames and use them to refine features of the current frame by cross-attention mechanism. Specifically, the temporal attention module has three steps: embedding of current frame embedding, embedding of support frames and features aggregation as described in Fig~\ref{fig:att}(a).

{\bf Embedding of current frame}. Given the features of the current frame $f_{C} \in \mathbb{R}^{C\times H \times W}$, where $H, W, C$ are the height, width, and the feature dimension of output feature map from the backbone network. We first feed it into a convolution layer to generate a feature map $f_{C}^{key}$, where $f_{C}^{key} \in \mathbb{R}^{\frac{C}{4} \times H \times W}$, and a non-linearity activation is applied. The $f_{C}^{key}$ can store the key features of the current frame including which and where objects may exist. Therefore, the feature map $f_{C}^{key}$ is learnt to encode the key information of visual semantics in the current frame, \ie object categories, object locations and masks.

{\bf Embedding of support frames}. Given a stack of feature maps obtained from support frames $\{f_{S_{t}} \in \mathbb{R}^{C\times H \times W}; t=1:T\}$ ($T$ is the number of the support frames), each feature map $f_{S_t}$ is first encoded into a pair of feature maps $f_{S_{t}}^{key}$ and $f_{S_{t}}^{value}$ by two parallel convolution layers, where $f_{S_{t}}^{key}, f_{S_{t}}^{value} \in \mathbb{R}^{\frac{C}{4} \times H \times W}$.  Non-linear activation is applied as well. If there are more than one support frames ($T>1$), we concatenate features of different frames along temporal dimension and obtain $f_S^{key}, f_S^{value}  \in \mathbb{R} ^{T \times \frac{C}{4} \times H \times W}$. $f_{S}^{key}$ contains the information of key features of support frames. The similarities between $f_{C}^{key}$ and $f_{S}^{key}$ shows when-and-where the features of support frames are suitable to be aggregated for the current frame. In addition, the feature map $f_{S}^{value}$ is learnt to represent the context information of all support frames.

{\bf Features aggregation}. In the feature aggregation step, we first compute the attention weights by similarities between all pixel in feature map $f_{C}^{key}$ and $f_{S}^{key}$. The similarities are computed as the correlation of every spatial location in the feature map $f_{C}^{key}$ and every spatial-temporal location in feature map $f_{S}^{key}$. Then the attention weights are used to aggregate features from $f_{S}^{value}$ to obtain context features from support frames. In addition, we perform a feature transformation by a $1 \times 1$ convolution layer. The whole process of feature aggregation for every position of the current frame can be summarized as the following equation,
\begin{equation}
F_{TA} \Leftrightarrow 
\label{eq1}
\left\{
\begin{aligned}
& X = f_S^{key} \odot f_{C}^{key} \\
& f_A(:, j) = F(f_S^{value} \odot \frac{exp(X(:, j))}{\sum_{i=1}^{N_p} exp(X(:, i))}) 
\end{aligned}
\right. 
\end{equation}
where $X$ is the similarity matrix between the current frame and support frames with size of $HWT \times HW$, $i$, $j$ are the indices of every position in the similarity matrix and the feature map, $N_p$ is the total number of positions in the feature map, $\odot$ is dot product, $F$ is a transformation function with non-linear activation, $f_A$ is the aggregated feature map after the transformation. Note that, all feature maps in above equation may be processed by some necessary reshaping or permutation operations. Finally, the refined feature map $\tilde{f_C}$ is obtained by summing up the aggregated feature map $f_A$ and the feature map of the current frame $f_C$.
 
After aggregation, we can obtain a feature map $\tilde{f_C}$, which not only preserves some information key visual semantics of current frame, but also extracts useful contextual information existing in other frames within the same video. 

\begin{figure}[t]
	\centering
	\includegraphics[width=0.4\textwidth]{figs/att.pdf}
	\caption{An illustration of (a) temporal attention module and (b) spatial attention module. Best viewed in color.}
	\label{fig:att}
\end{figure}

\subsection{Spatial Attention Module}
Besides aggregating temporal contextual information from support frames, we also propose a spatial attention module to model the spatial context of the current frame. The proposed spatial attention module is based on the non-local block~\cite{wang2018non} with several 
modifications. As shown in Fig~\ref{fig:att}(b), instead of computing the attention maps of all spatial positions for each channel, we simplify it by sharing a single attention weight for all spatial positions in each channel, which is known as the channel attention. As reported in~\cite{cao2019gcnet}, this simplified channel attention module can achieve very similar performance compared to original non-local block. Also, since it doesn't need to compute a specific attention weight for every spatial position, it can be more efficient. Then, we use two more convolution layers to transform the features. Finally, the channel attention is added back to the feature map of the current frame for global context feature aggregation. 

\subsection{Combining Two Attention Modules}
Furthermore, we integrate the two attention modules into a dual attention module. The dual attention model takes the features of the current frame and support frames as inputs. The features of the current frame $f_C$ is first processed by spatial attention model to obtain an attention map with context features of itself. And features of current frames $f_C$ and features of support frames $f_S$ are fed into the temporal attention module to generate another attention map with context information of support frames. Then, we aggregate the features of the current frame with two attention maps by adding them to the original feature map $f_C$ as follows,  
\begin{equation}
\begin{aligned}\label{eq2}
& f_{agg} = F_{TA}(f_C, f_S) + F_{SA}(f_C) + f_C \\
\end{aligned}
\end{equation}
where $F_{TA}, F_{SA}$ represent proposed temporal attention module and spatial attention module respectively.

\subsection{Attention Module on Object Level}\label{sec3.4}
Attention modules described above are all processed on frame level features, which means they aggregate the context information for the whole feature map from the current frame and support frames. However, for instance segmentation with a two-stage framework, aggregating context information for each object proposal is also critical. Object proposals act as valuable candidates for the final object predictions and they encode more focused features for the individual objects. Proper feature aggregation onto these object proposals can elucidate confusing feature representations and improve recognition accuracy. We attempt to extend the two proposed attention modules to object level by applying similar operations to the object features produced by ROI Align~\cite{he2017mask}. Since the proposed attention modules can be applied to feature maps with arbitrary size, we adopt the two proposed attention modules in a similar fashion. 
We denote the features of detected object proposals as $f_{C_p}$. $f_{C_p} \in \mathbb{R}^{P\times C \times h \times w}$, where $P$ is the number of object proposals, $C$ is the channel dimension, and $h$, $w$ are height and width of the proposal. The features of each proposal are fed into a dual attention module along with feature maps of support frames. Since the size of the features of proposals is much smaller than the whole feature map, aggregating features on object level can be very efficient. In the next section, we show the performance gain of adding object-level attention modules is comparable or even better than adding attention module on frame level. 

The proposed attention module is inspired by some early work of video object detection and video object segmentation~\cite{wu2019long,oh2019video}, where the non-local blocks are used for extracting self-attention features. However, the motivation of proposed attention module is different as it is designed to aggregate the features from other frames (support frames) to current frame through feature matching, but not to itself by self-attention mechanisms. In addition, we extend the proposed attention module to object level, which largely reduces the computational complexity while improves the performance as frame-level attention module. Furthermore, frame-level and object-level module can be integrated into a single framework for a better performance. To our best knowledge, there are few work on enhancing object level feature by other frames. And the experiments in Sec.~\ref{exp} show the effectiveness of our proposed attention module.

By performing proposed dual attention module on both frame level and object level, we achieve feature aggregation on both spatial and temporal dimension, and in both local and global granularities, which is a comprehensive approach to aggregate and enhance intermediate features in a video instance segmentation framework.

\subsection{Correlation-based Tracking Module}

\begin{figure}[t]
	\centering
	\includegraphics[width=0.4\textwidth]{figs/depth.pdf}
	\caption{An illustration of the correlation-based tracking module. The convolution layers are omitted here.}
	\label{fig:track}
\end{figure}

In order to track an object across frames in a more robust and consistent way, we propose a new correlation-based tracking module to generate both spatial likelihoods and object similarities, which is more powerful than the previous approach~\cite{yang2019video}.

Recent works on object tracking shows the effectiveness of correlation-based tracking which has achieved state-of-the-art performance on several tracking benchmarks. Inspired by SiamRPN++~\cite{li2019siamrpn++}, our proposed tracking module incorporates knowledge from both object similarities and cross-correlations of the target object and the search region. As shown in Fig~\ref{fig:track}, our tracking module can be abstracted into three procedures:(a) pair-wise similarity computation, which uses depth-wise correlation instead of matrix multiplication~\cite{yang2019video} to predict a similarity matrix between two groups of objects. (b) cross-correlation computation, which generates correlation maps with depth-wise correlation between a single object and the whole frame. To train the model to produce high quality correlation maps, we employ a pseudo likelihood map represented by a two dimensional gaussian distribution centered at the location of the object as the supervision signal. Then we compute the aggregated similarity vector for a detected object with ROI align within the corresponding bounding box on the correlation map. 
After that, we sum up the two similarity vectors to obtain the final depth-wise similarity vector. (c) The final similarity score is mapped from the similarity vector with  two $1 \times 1$ convolution layers. The proposed correlation-based tracking module considers both feature similarity and correlation-based similarity scores on the raw image features to increase the robustness of tracking.

\subsection{Data and Evaluation Metric}
{\bf Data} YouTube-VIS is the first and largest dataset for video instance segmentation, which is a subset of YouTube-VOS dataset~\cite{xu2018youtube}.
YouTube-VIS is comprised of 2,883 high resolution YouTube videos with 40 common object categories. In each video, several objects with bounding boxes and masks are labeled manually and the identities cross different frames are annotated as well. Since only the validation set is available for evaluation, all results reported in this paper are evaluated on the validation set.

{\bf Evaluation Metrics}
To evaluate the performance of the proposed method, we use the metrics mentioned in~\cite{yang2019video}, which are average precision(AP) and average recall(AR) based on a spatial-temporal Intersection-over-Union (IoU). Following the COCO evaluation, AP is computed by averaging over multiple IoU thresholds, \emph{e.g. } from from 50\% to 95\% at step 5\% and AR is the maximum recall given some fixed number of segmented instances per video. Both metrics are first calculated for each category and then averaged over 40 categories. 

\subsection{Implementation Details}

{\bf Training}. Our proposed method is built on Mask-RCNN~\cite{he2017mask}. The backbone network structure is ResNet50 with FPN~\cite{lin2017feature}, which is pre-trained on MSCOCO dataset~\cite{lin2014microsoft}. 
In the tracking branch, we use two convolution layers to refine the correlation features generated by depth-wise correlations, respectively. The first convolution layer has 256 channels which is the same dimension as the correlation features while the second one is used for correlation map prediction with only one output channel. 
Our model is implemented based on MMDetection~\cite{chen2019mmdetection} and the whole framework is trained end-to-end in 12 epochs with two NVIDIA 2080TI GPUs. We resize the original frame size to $640 \times 360$ for both training and testing. During training, the initial learning rate is set to 0.0125 and decays with a factor of 10 at epoch 8 and 11. For each input frame, we randomly select three frames from the same video, two used as support frames in the dual attention module and the other used as reference frame in the tracking module.

{\bf Testing}. During evaluation, the testing video is processed by the proposed method frame by frame in an online fashion. For each input frame, four additional frames are sampled from the testing video as support frames. Note that the number of support frames used in training and testing can be different, since testing with more support frames can help improve performance. We conduct an ablation study on the number of support frames in the following section. For the tracking head, we ignore the correlation map and use the predicted correlation score as the tracking score. Then we follow the inference procedure described in~\cite{yang2019video} to predict the category, bounding box and mask for the object instance. In addition, we combine other cues, \ie detection confidence, bounding box IoU, and category consistency, along with tracking scores to improve the tracking accuracy as a powerful post-processing.

\subsection{Ablation Study}
\begin{table}\setlength{\tabcolsep}{1pt}
\centering
\footnotesize
\begin{tabular} {l|c|c|c}
\hlineB{2}
Methods & AP & AP$_{0.5}$ & AP$_{0.75}$ \\ \hline 
Mask-Track RCNN~\cite{yang2019video} & 21.1 & 37.7 & 23.6 \\ \hline
Our Implementation & 20.9 & 37.9 & 21.6 \\ 
Our Implementation + MSCOCO & {\bf 24.1} & {\bf 42.6} & {\bf 24.9} \\ \hline
\hlineB{2}
\end{tabular}
\caption{Performance of the baseline model on YouTube-VIS validation set. ``Our Implementation" means our reproduced results of Mask-Track RCNN.``Our Implementation + MSCOCO" is used as the baseline model in all ablation studies hereafter.}
\label{exp:t1}
\end{table}


\begin{table}\setlength{\tabcolsep}{4pt}
\centering
\footnotesize
\begin{tabular} {l|c|c|c}
\hlineB{2}
\multicolumn{4}{c}{(a) {\bf Attention on Frame Level} }\\ 
& AP & AP$_{0.5}$ & AP$_{0.75}$ \\ \hline 
Baseline + Temporal Attention & 25.2 & 43.9 & 25.6 \\
Baseline + Spatial Attention & 24.9 & 43.4 & 25.2 \\
Baseline + Spatial-Temporal Attention & 25.8 & 44.5 & 27.0 \\ \hline
\multicolumn{4}{c}{(b) {\bf Attention on Object Level}}\\ 
& AP & AP$_{0.5}$ & AP$_{0.75}$ \\ \hline 
Baseline + Temporal Attention  & 25.4 & 44.4 & 25.9  \\ 
Baseline + Spatial Attention  & 24.8 & 43.3 & 25.1 \\ 
Baseline + Spatial-Temporal Attention & 26.1 & 45.6	& 26.7 \\ \hline
\multicolumn{4}{c}{(c) {\bf Attention on Both Frame and  Object Level}}\\ 
& AP & AP$_{0.5}$ & AP$_{0.75}$ \\ \hline 
Baseline + Spatial-Temporal Attention & {\bf 27.5} & {\bf 46.1} & {\bf 28.9} \\
\hlineB{2}
\end{tabular}

\caption{Ablation Study of our proposed attention module on YouTube-VIS validation set. The best results are highlighted in bold.}
\label{exp:t2}
\end{table}
{\bf Baseline Model and Data Augmentation}
Since our proposed method is built on Mask-Track RCNN~\cite{yang2019video}, we take Mask-Track RCNN as the baseline model to validate the effectiveness of each contribution. We first reproduce the Mask-Track RCNN with publicly available codes and the results are listed in Table~\ref{exp:t1}. Our result is close to the result reported in original paper. Note that all results in Table~\ref{exp:t1} are without any post-processing. 

In addition, we find that the number of object instances in YouTube-VIS dataset is limited. Since the proposed method does not depend on the temporal smoothness of a video, some image-based datasets can be adopted to increase the training samples. We choose MSCOCO~\cite{lin2014microsoft} as external data which has a large overlap on the object categories with YouTube-VIS. In order to make use of the image data, we generate support frames and reference frame based on a single image by some affine transformations, \ie \ie rotation, translation and shearing.
The identity annotations across different images can be generated automatically in this process. The performance after using external data is listed in Table~\ref{exp:t1} as well.
We use this model as a baseline model for all the following ablation experiments.



\begin{table}\setlength{\tabcolsep}{2pt}
\centering
\footnotesize
\begin{tabular} {l|c|c|c}
\hlineB{2}
Modules & AP & AP$_{0.5}$ & AP$_{0.75}$ \\ \hline 
Baseline + CM & 25.1 & 44.3 & 26.7 \\
Baseline + CM + FDA & 26.3 & 45.3 & 27.1 \\
Baseline + CM + BDA & 26.7 & 45.9 & 27.3 \\
Baseline + CM + FDA + BDA (CompFeat) & {\bf 28.4} & {\bf 47.4} & {\bf 30.7} \\
\hlineB{2}
\end{tabular}
\caption{Ablation Study of the proposed track module on YouTube-VIS validation set. CM, FDA, BDA denote the proposed tracking module with correlation map, the frame level dual attention module and object level dual attention module, respectively. The best results are highlighted in bold.}
\label{exp:t3}
\end{table}


\begin{table}\setlength{\tabcolsep}{3pt}
\centering
\footnotesize
\begin{tabular} {l|c|c|c|c}
\hlineB{2}
Train/Test(Uniform) &  2 frames &  3 frames &   3 frames & 4 frames \\ \hline 
Uniform 2 frames & 27.1 & 27.4 & 27.8 & 27.3  \\
Uniform 3 frames & 26.2 & 26.6 & 26.6 & 26.8  \\ 
Random 2 frames & 27.3 & 27.7 & 28.4 & 27.4\\
Random 3 frames  & 26.1 & 26.8 & 26.8 & 26.7 \\ \hline
\hlineB{2}
\end{tabular}
\caption{Performance of different sampling methods and different frames during training/testing on the validation set of Youtube-VIS. The performance is reported in AP.}
\label{exp:t6}
\end{table}

\begin{table*}[t]\setlength{\tabcolsep}{6pt}
\centering
\footnotesize
\begin{tabular} {c|c|c|c|c|c}
\hlineB{2}
Methods & AP & AP$_{0.5}$ & AP$_{0.75}$ & AR$_1$ & AR$_{10}$ \\ \hline
IoUTracker+~\cite{bochinski2017high}  & 23.6 & 39.2 & 25.5 & 26.2 & 30.9 \\
OSMN~\cite{yang2018efficient} & 27.5 & 45.1 & 29.1 & 28.6 & 33.1 \\
DeepSORT~\cite{wojke2017simple} & 26.1 & 42.9& 26.1 & 27.8 & 31.3 \\
SeqTracker & 27.5 & 45.7 & 28.7 & 29.7 & 32.5 \\
MaskTrack R-CNN~\cite{yang2019video} & 30.3 & 51.1 & 32.6 & 31.0 & 35.5 \\
SipMask~\cite{Cao_SipMask_ECCV_2020} & 32.5 & 53.0 & 33.3 & 33.5 & 38.9 \\
\hline
MaskTrack R-CNN + MSCOCO & 32.2 & 52.1 & 34.6 & 31.8 & 37.2 \\
CompFeat & {\bf 35.3} & {\bf 56.0} & {\bf 38.6} & {\bf 33.1} & {\bf 40.3} \\
\hlineB{2}
\end{tabular}
\caption{Comparison of the proposed approach with the state-of-the-arts on YouTube-VIS validation set. Note that all results in this Table including the post-processing. The best results are highlighted in bold.}
\label{exp:t4}
\end{table*}

{\bf Effectiveness of Attention Module}
We conduct ablation study to prove the effectiveness of the proposed attention module: temporal attention and spatial attention. The results are listed in Table~\ref{exp:t2}. Note all results here are without post-processing. We first evaluate each attention module on frame level in Table~\ref{exp:t2}(a). With the temporal attention module, we improve baseline model by 1.1\% in AP and  1.3\% in AP$_{0.5}$ respectively. Similarly, spatial attention module can also slightly improve the baseline performance by 1\%. When combining both temporal and spatial attention together as the dual attention module, we further boost the baseline model by 1.7\%, 1.9\% and 2.1\% in AP, AP$_{0.5}$ and AP$_{0.75}$. These experimental results prove that our proposed attention module can aggregate helpful context feature from both other frames and the input frame. 

We then experiment with temporal and spatial attention on object level. Table~\ref{exp:t2}(b) shows that proposed attention method on object level can always achieve comparable or even better results compared with the frame level one. For instance, by performing both temporal and spatial attention on object level, the performance gain becomes 2.0\%, 3.0\% and 1.8\% in AP, AP$_{0.5}$ and AP$_{0.75}$ respectively. 

Furthermore, Table~\ref{exp:t2}(c) lists the performance of combining attention modules on both frame level and object level. Comparing with the performance only on frame or object level, the combination one is superior. Specifically, with both attention on two different levels, we achieve AP/AP$_{0.5}$ $=27.5\%/46.1\%$, which outperforms the performance with attention module on frame level by 1.7\%/1.6\%. 
This further improvement shows that by using attention module on frame level and object level, we can aggregate context information in a global-to-local manner, which can greatly improve the baseline model by 3.4\%, 3.5\% and 4.0\% in AP, AP$_{0.5}$ and AP$_{0.75}$.

\begin{figure}[t]
\begin{center}
\bgroup 
 \def\arraystretch{0.1} 
 \setlength\tabcolsep{0.5pt}
\begin{tabular}{ccccc}
\includegraphics[width=0.2\linewidth]{figs/108/1.png} &
\includegraphics[width=0.2\linewidth]{figs/108/4.png} & 
\includegraphics[width=0.2\linewidth]{figs/108/7.png} & 
\includegraphics[width=0.2\linewidth]{figs/108/10.png} &
\includegraphics[width=0.2\linewidth]{figs/108/13.png} \\
\includegraphics[width=0.2\linewidth]{figs/40/2.png} &
\includegraphics[width=0.2\linewidth]{figs/40/3.png} &
\includegraphics[width=0.2\linewidth]{figs/40/4.png} &
\includegraphics[width=0.2\linewidth]{figs/40/5.png} &
\includegraphics[width=0.2\linewidth]{figs/40/6.png} \\
\includegraphics[width=0.2\linewidth]{figs/235/2.png} &
\includegraphics[width=0.2\linewidth]{figs/235/4.png} &
\includegraphics[width=0.2\linewidth]{figs/235/7.png} &
\includegraphics[width=0.2\linewidth]{figs/235/9.png} &
\includegraphics[width=0.2\linewidth]{figs/235/12.png} \\
\includegraphics[width=0.2\linewidth]{figs/96/18.png} &
\includegraphics[width=0.2\linewidth]{figs/96/19.png} &
\includegraphics[width=0.2\linewidth]{figs/96/20.png} &
\includegraphics[width=0.2\linewidth]{figs/96/21.png} &
\includegraphics[width=0.2\linewidth]{figs/96/23.png} \\
\includegraphics[width=0.2\linewidth]{figs/154/5.png} &
\includegraphics[width=0.2\linewidth]{figs/154/7.png} &
\includegraphics[width=0.2\linewidth]{figs/154/9.png} &
\includegraphics[width=0.2\linewidth]{figs/154/11.png} &
\includegraphics[width=0.2\linewidth]{figs/154/13.png} \\
\includegraphics[width=0.2\linewidth]{figs/215/1.png} &
\includegraphics[width=0.2\linewidth]{figs/215/2.png} &
\includegraphics[width=0.2\linewidth]{figs/215/5.png} &
\includegraphics[width=0.2\linewidth]{figs/215/10.png} &
\includegraphics[width=0.2\linewidth]{figs/215/13.png} \\
\end{tabular} \egroup
\end{center}
\caption{Visualization results of CompFeat. Each row has five sampled frames from a video sequence. Categories, bounding boxes and instance masks are shown for each object. Note objects with the same predicated identity across frames are marked with the same color. Zoom in to see details.}
\label{exp:v1}
\end{figure}


{\bf Effectiveness of Correlation-based Tracking Module}
The ablation study on the proposed correlation-based tracking module is shown in Table~\ref{exp:t3}. Again, the results are without post-processing. Compared to the baseline model, the tracking module with correlation map outperforms the baseline model by more than 1\% on AP, AP$_{0.5}$ and AP$_{0.75}$. This improvement indicates the cross-correlation map between objects and the whole frames contains more semantic information than the object features used by the baseline. In addition, when integrating it into the whole framework with dual attention on frame level or object level, we obtain consistently better performance. In particular, by using correlation maps and dual attention module on frame level, we improve the performance in AP from 25.8\% to 26.3\%. Finally, we evaluate the performance of video instance segmentation with all our proposed modules, \emph{e.g. } frame level dual attention module, object level dual attention module and correlation maps. As shown in the last row in Table~\ref{exp:t3}, we obtain the best performance. For instance, we achieve 28.4\% and 47.4\% on AP and $AP_{0.5}$ with all proposed modules, which surpasses the baseline model by 4.3\% and 4.8\%. 


{\bf Sampling Method.} The VIS performance under different sampling methods are shown in Table~\ref{exp:t6}. Note that, for fairness, all experiment use the uniform sampling during testing. From Table~\ref{exp:t6}, it can be observed that the random sampling is always powerful than the uniform one during training. The reason is that random sampling increases the variety of samples during training, which can lead our proposed ComFeat model discovers more temporal and spatial correspondences between current frames and support frames.




\subsection{Comparison with the State-of-the-Arts}
Mask-Track RCNN~\cite{yang2019video} is the first work on video instance segmentation. There are several work proposed in the Large-Scale Video Object Segmentation challenge~\cite{luiten2019video,wang2019empirical, dong2019temporal}, but it is hard to compare with them since they use different backbone networks and different external training data. We borrow experimental results of other approaches from~\cite{yang2019video}. The comparison results are presented in Table~\ref{exp:t4}. 
Mask-Track RCNN~\cite{yang2019video} is an online method which learns feature similarities for object matching. And SipMask~\cite{Cao_SipMask_ECCV_2020} shares the similar structure while replace the instance segmentation branch with an one stage instance segmentation module. Compared with these methods, our proposed CompFeat achieves the best performance under all evaluation metrics. Compared with our baseline Mask-Track RCNN, the proposed CompFeat outperform it by a large margin. 
Note this performance gain is not from the additional training data since MaskTrack R-CNN with the same training data only achieves 32.2\% and 52.1\% on AP and AP$_{0.5}$.

\subsection{Qualitative Results}
Fig.~\ref{exp:v1} shows some qualitative results of our proposed CompFeat on YouTube-VIS validation set. Each row represents the predicted results on different frames in a video. The objects with the same identity are shown in the same color. As shown, CompFeat makes accurate predictions on object categories, bounding boxes, masks and identities under challenging conditions, \ie  multiple similar objects (row 1, 2), moderate occlusions (row 3), and drastic appearance changes (row 4).
The last row shows a challenging case with six fish where our algorithm performs much better than MaskTrack-RCNN~\cite{yang2019video} although it misses a fish in the third image. 




\section{Introduction}
\input{TEX/1_intro.tex}
\section{Related Work}~\label{related}
\input{TEX/2_related.tex}
\section{Proposed Method}\label{method}
\input{TEX/3_method.tex}
\section{Experiments}\label{exp}
\input{TEX/4_experiment.tex}
\section{Conclusion}
\input{TEX/6_conclusion.tex}

