\section{Introduction}
\label{sec:introduction}

Colonoscopy is the gold standard for detecting colorectal lesions, since it enables colorectal polyps to be identified and removed in time,
thereby preventing further spread. 
Polyp segmentation, as a fundamental task in medical image analysis, aims to accurately locate polyps in the early stage, which is of great significance in the clinical prevention of rectal cancer. 
Traditional polyp segmentation methods mainly rely on low-level features, such as texture~\cite{fiori2014complete}, geometric features~\cite{mamonov2014automated}, simple linear iterative clustering super-pixels~\cite{maghsoudi2017superpixel}, \emph{etc}
However, these methods tend to yield low-quality segmentation performance and suffer from poor generalization ability.
With the development of deep learning in medical image analysis, polyp segmentation has achieved promising progress.
In particular, the U-shaped~\cite{ronneberger2015unet} has attracted significant attention due to its ability to adopt multi-level features for reconstructing high-resolution prediction results. 
PraNet~\cite{fan2020pranet} employs a two-stage segmentation approach, adopting a parallel decoder to predict rough regions, and an attention mechanism to restore the edges and internal structure of a polyp for fine-grained segmentation.
ThresholdNet~\cite{guo2020learn} is a confidence-guided data enhancement method based on a hybrid manifold for solving the problems caused by limited annotated data and imbalanced data distributions.


\begin{figure}[t!]
	\centering
    \small
	\begin{overpic}[width=.98\linewidth]{Imgs/first_map-min.pdf}
    \vspace{7pt}
	\put(7,-4.5){Image}
	\put(35,-4.5){GT}
	\put(58.5,-4.5){\color{red}Ours}
	\put(81.5,-4.5){SANet}
    \end{overpic}
    \vspace{7pt}
	\caption{
	The segmentation examples of our model and SANet~\cite{wei2021shallow} with different challenge cases, \emph{e.g.} camouflage ($1^{st}$ and $2^{nd}$ rows) and image acquisition influence ($3^{rd}$ row). 
	The images from top to bottom are from ClinicDB~\cite{bernal2015wm}, ETIS~\cite{silva2014toward}, and ColonDB~\cite{tajbakhsh2015automated}, which show that our model has better generalization ability. }
    \label{figure:first_map}
\end{figure}


Although these methods have greatly improved accuracy and generalization ability compared to traditional methods, it is still challenging for them to locate the boundaries of polyps, as shown as in \figref{figure:first_map}, for several reasons:
(1) \emph{Image noise.} During the data collection process, the lens rotates in the intestine to obtain polyp images from different angles, which also causes motion blur and reflector problems. As a result, this greatly increases the difficulty of polyp detection;
(2) \emph{Camouflage.} The color and texture of polyps are very similar to surrounding tissues, with low contrast, providing them with powerful camouflage properties~\cite{fan2021concealed,fan2020camouflaged}, and making them difficult to identify;
(3) \emph{Polycentric data.} Current models struggle to generalize to multicenter (or unseen) data with different domains/distributions.


To address the above issues, our contributions in this paper are threefold:
\begin{itemize}
\item We present a new polyp segmentation framework, termed \textbf{Polyp-PVT}. Different from existing CNN-based methods, we adapt the pyramid vision transformer as our encoder for extracting more powerful and robust features. 
\item To support our framework, we use three simple modules. 
Specifically, the cascaded fusion module (CFM) collects the semantic and location information of polyps from the high-level features through progressive integration. 
Meanwhile, the camouflage identification module (CIM) is applied to capture polyp cue disguised in low-level features, using an attention mechanism to pay more attention to potential polyps, which in turn reduces incorrect information in the lower features. 
We further introduce the similarity aggregation module (SAM) equipped with a non-local and a graph convolutional layer to mine local pixels and global semantic cues from the polyp area.

\item Finally, we conduct extensive experiments on five challenging benchmark datasets, including Kvasir-SEG~\cite{jha2020kvasir}, ClinicDB~\cite{bernal2015wm}, ColonDB~\cite{tajbakhsh2015automated}, Endoscene~\cite{vazquez2017benchmark}, and ETIS~\cite{silva2014toward}, to evaluate the performance of the proposed Polyp-PVT.
On ColonDB, our method achieves a mean Dice (mDic) of 0.808, which is 5.5\% higher than existing state-of-the-art method SANet~\cite{wei2021shallow}.
On the ETIS dataset, our model achieves a mean Dice (mDic) of 0.787, which is 3.7\% higher than SANet~\cite{wei2021shallow}.
\end{itemize}

\section{Related Works} 

\subsection{Polyp Segmentation}\label{subsec:polyp segmentation}
\textbf{Traditional Methods.} \revI{Computer-aided detection is an effective alternative to manual detection, and a detailed survey has been conducted on the detection of ulcers, polyps, and tumors in wireless capsule endoscopy imaging~\cite{rahim2020survey}.}
Early solutions for polyp segmentation were mainly based on low-level features, such as texture~\cite{mamonov2014automated}, geometric features~\cite{mamonov2014automated}, or simple linear iterative clustering superpixels~\cite{maghsoudi2017superpixel}.
However, due to the high similarity between polyps and surrounding tissues, these methods have a high risk of missed or false detection.

\textbf{Deep Learning-Based Methods.} Deep learning techniques~\cite{he2016deep,simonyan2015very,li2019selective,wang2018mixed,long2015fully} have greatly promoted
the development of polyp segmentation tasks. 
Akbari \textit{et al.}~\cite{akbari2018polyp} proposed a polyp segmentation model using a fully convolutional neural network, whose segmentation results are significantly better than traditional solutions.
Brandao \textit{et al.}~\cite{brandao2018towards} used the shape from shading strategy to restore depth, merging the result into an RGB model to provide richer feature representations.
More recently, encoder-decoder based models, such as U-Net~\cite{ronneberger2015unet}, UNet++~\cite{zhou2018unet++}, and ResUNet++~\cite{JhaSRJLHJ19}, have gradually come to dominate the field with excellent performance.
Sun \textit{et al.}~\cite{sun2019colorectal} introduced a dilated convolution to extract and aggregate high-level semantic features with resolution retention for improving the encoder network.
Psi-Net~\cite{murugesan2019psi} introduced a multi-task segmentation model that combines contour prediction and distance map estimation to assist segmentation mask prediction.
Hemin \textit{et al.}~\cite{qadir2019polyp} first attempted to use a deeper feature extractor to perform polyp segmentation based on Mask R-CNN~\cite{he2017mask}.

Different from the methods based on U-Net~\cite{ronneberger2015unet,zhou2018unet++,AlamTTJR20}, PraNet~\cite{fan2020pranet} uses reverse attention modules to mine boundary information with a global feature map, which is generated by a parallel partial decoder from high-level features.
\revI{Polyp-Net~\cite{banik2020polyp} proposed a dual-tree wavelet pooling CNN with a local gradient weighted embedding level set, which effectively avoids erroneous information in high signal areas, thereby significantly reducing the false positive rate.}
\revI{
Rahim \textit{et al.}~\cite{rahim2021deep} proposed to use different convolution kernels for the same hidden layer for deeper feature extraction with MISH and rectified linear unit activation functions for deep feature propagation and smooth non-monotonicity. In addition, they adopted joint generalized intersections, which overcomes scale invariance, rotation, and shape differences.}
Jha \textit{et al.}~\cite{jha2021real} designed a real-time polyp segmentation method called ColonSNet.
For the first time, Ahmed \textit{et al.}~\cite{ahmed2020generative} applied the generative adversarial network to the field of polyp segmentation.
Another interesting idea proposed by Thambawita~\textit{et al.}~\cite{ThambawitaHHR20} is introducing pyramid-based augmentation into the polyp segmentation task.
Further, Tomar \textit{et al.}~\cite{tomar2020ddanet} designed a dual decoder attention network based on ResUNet++ for polyp segmentation.
More recently, MSEG~\cite{huang2021hardnet} improved the PraNet and proposed a simple encoder-decoder structure. 
Specifically, they used Hardnet~\cite{chao2019hardnet} to replace the original backbone network Res2Net50 backbone network and removed the attention mechanism to achieve faster and more accurate polyp segmentation. 
As an early attempt, Transfuse~\cite{zhang2021transfuse} was the first to employ a two-branch architecture combining CNNs and transformers in a parallel style. 
DCRNet~\cite{yin2021duplex} uses external and internal context relations modules to separately estimate the similarity between each location and all other locations in the same and different images.
MSNet~\cite{Zhao_2021_MICCAI} introduced a multi-scale subtraction network to eliminate redundancy and complementary information between the multi-scale features.
Providing a comprehensive review on polyp segmentation is beyond the scope of this paper. 
In Tab.~\ref{tab: model_survey}, however, we give a brief survey of  representative works related to ours.

\begin{table*}[t!]
\small
  \centering
  \caption{
  A survey  on polyp segmentation. CL = CVC-CLINIC, EL = ETIS-Larib, C6 = CVC-612, AM = ASU-Mayo~\cite{zhou2017fine,tajbakhsh2016convolutional}, ES = EndoScene, DB = ColonDB, CV = CVC-VideoClinicDB, C = Colon, ED = Endotect 2020, KS = Kvasir-SEG, KCS = Kvasir Capsule-SEG, PraNet = same to datasets used in PraNet~\cite{fan2020pranet}, IS = image segmentation, VS = video segmentation, CF = classfication, OD = object detection, Own = private data. CSCPD~\cite{fiori2014complete}, APD~\cite{mamonov2014automated}, SBCP~\cite{maghsoudi2017superpixel}, FCN~\cite{akbari2018polyp}, D-FCN~\cite{brandao2018towards}, UNet++~\cite{zhou2018unet++}, Psi-Net~\cite{murugesan2019psi}, Mask R-CNN~\cite{qadir2019polyp}, UDC~\cite{sun2019colorectal}, ThresholdNet~\cite{guo2020learn}, MI2GAN~\cite{xie2020mi}, ACSNet~\cite{zhang2020ACSN}, PraNet~\cite{fan2020pranet}, GAN~\cite{ahmed2020generative}, APS~\cite{tomar2021automatic}, PFA~\cite{ThambawitaHHR20}, MMT~\cite{JhaHEJJLRH20}, U-Net-ResNet50~\cite{AlamTTJR20}, Survey~\cite{rahim2020survey}, Polyp-Net~\cite{banik2020polyp}, Deep CNN~\cite{rahim2021deep}, EU-Net~\cite{PatelBW21}, DSAS~\cite{lumini2021deep}, U-Net-MobileNetV2~\cite{branch2021polyp}, DCRNet~\cite{yin2021duplex}, MSEG~\cite{huang2021hardnet}, FSSNet~\cite{khadga2021few}, AG-CUResNeSt~\cite{sang2021ag}, MPAPS~\cite{yang2021mutual}, ResUNet++~\cite{JhaSJLJHR21}, NanoNet~\cite{jha2021nanonet}, ColonSegNet~\cite{jha2021real}, Segtran~\cite{segtran}, DDANet~\cite{tomar2020ddanet}, UACANet~\cite{kim2021uacanet}, DivergentNet~\cite{divergentNets}, DWHieraSeg~\cite{guo2021dynamic}, Transfuse~\cite{zhang2021transfuse}, SANet~\cite{wei2021shallow}, PNS-Net~\cite{ji2021pnsnet}}
    \renewcommand{\arraystretch}{1.0}
    \setlength\tabcolsep{4.0pt}
    \begin{tabular}{r|r|c|c|c|c|c|l}
    \hline
    No. & Model & Publication & Year & Code  & Type & Dataset & Core Components\\
    \hline   
    1&CSCPD & IJPRAI &2014 & N/A &   IS    & Own &  Adaptive-scale candidate \\

   2 &APD & TMI &2014 & N/A &   IS    & Own &  Geometrical analysis, binary classifier\\
    
    3&SBCP & SPMB &2017 & N/A &   IS    & Own &  Superpixel\\

    4&FCN & EMBC &2018 & N/A &   IS    & DB&  FCN and patch selection\\    
    
    5&D-FCN  & JMRR & 2018  & N/A    &  IS     & CL, EL, AM, and DB   & FCN and Shape-from-Shading (SfS)\\
    
   6 &UNet++ &DLMIA   &2018 & \href{https://github.com/MrGiovanni/UNetPlusPlus}{PyTorch} &   IS    & AM & Skip pathways and deep supervision\\  
    
    7&Psi-Net& EMBC  &2019 & \href{https://github.com/Bala93/Multi-task-deep-network}{PyTorch} &   IS    & Endovis &  Shape and boundary aware\\  
    
   8& Mask R-CNN & ISMICT  &2019 & N/A &   IS    & C6, EL, and DB &  Deep feature extractors \\ 
    
   9 &UDC & ICMLA   &2019 & N/A &   IS    & C6 and EL &  Dilation convolution \\
    
    10&ThresholdNet & TMI   & 2020  & \href{https://github.com/Guo-Xiaoqing/ThresholdNet}{PyTorch} &     IS  & ES and WCE   & \makecell[l]{Learn to threshold \\ Confidence-guided manifold mixup} \\
    
    11&MI2GAN  & MICCAI & 2020  & N/A    & IS      &C6 and EL  & GAN based model\\
   12 &ACSNet   & MICCAI & 2020  & \href{https://github.com/ReaFly/ACSNet }{PyTorch}& IS       &ES and KS   &  Adaptive context selection\\
    13&PraNet & MICCAI & 2020  & \href{https://github.com/DengPingFan/PraNet}{PyTorch} &  IS     & PraNet & Parallel partial decoder attention\\
    14&GAN  & MediaEval & 2020  & N/A    & IS      &  KS &  Image-to-image translation\\
    15&APS   & MediaEval & 2020  & N/A    &      IS & KS &Variants of U-shaped structure\\
    16&PFA & MediaEval &2020 & \href{https://github.com/vlbthambawita/pyra-pytorch}{PyTorch} &   IS    & KS &  Pyramid focus augmentation\\
    17&MMT  & MediaEval & 2020  & N/A    &   IS    & KS & Competition introduction \\
    
    18&U-Net-ResNet50 & MediaEval & 2020  & N/A    & IS       &  KS & Variants of U-shaped structure\\
   
    \revI{19}&\revI{Survey} & \revI{CMIG} & \revI{2021} & \revI{N/A}  & \revI{CF}     & \revI{Own} & \revI{Classification} \\
    
   \revI{20}&\revI{Polyp-Net} & \revI{TIM} & \revI{2020}  & \revI{N/A} & \revI{IS}      & \revI{DB and CV} &  \revI{Multimodel fusion network} \\
    
    \revI{21}&\revI{Deep CNN} & \revI{BSPC} & \revI{2021}  & \revI{N/A}& \revI{OD}     & \revI{EL} &   \revI{Convolutional neural network} \\
    
   
    22&EU-Net & CRV & 2021  & \href{https://github.com/rucv/Enhanced-U-Net}{PyTorch} & IS      & PraNet &  Semantic information enhancement \\
    23&DSAS & MIDL & 2021  & \href{https://github.com/LorisNanni/Deep-ensembles-based-on-Stochastic-Activation-Selection-for-Polyp-Segmentation}{Matlab} &  IS     &  KS & Stochastic activation selection\\
    24&U-Net-MobileNetV2 & arXiv & 2021  & N/A    &    IS   & KS & Variants of U-shaped structure \\
   25 &DCRNet & arXiv & 2021  & \href{https://github.com/PRIS-CV/DCRNet}{PyTorch}&    IS   &\makecell[c]{ES, KS, and\\PICCOLO}    &\makecell[l]{Within-image \\and cross-image contextual relations}   \\
    26&MSEG & arXiv & 2021  & \href{https://github.com/james128333/HarDNet-MSEG}{PyTorch} &IS       & PraNet & Hardnet and partial decoder\\
    27&FSSNet & arXiv & 2021  & N/A   &       IS & C6 and KS &Meta-learning\\
    28&AG-CUResNeSt& RIVF & 2021  & N/A    &    IS   & PraNet &  ResNeSt, attention gates\\
   29 &MPAPS & JBHI & 2021  & \href{https://github.com/CityU-AIM-Group/MPA-DA}{PyTorch}   &       IS &  DB, KS, and EL & Mutual-prototype adaptation network \\  
    30&ResUNet++ & JBHI   & 2021  & \href{https://github.com/DebeshJha/ResUNetPlusPlus-with-CRF-and-TTA}{PyTorch} &  IS, VS     & PraNet and AM & ResUNet++, CRF and TTA\\
   31 &NanoNet & CBMS  & 2021  & \href{https://github.com/DebeshJha/NanoNet}{PyTorch} &  IS, VS     & ED, KS, and KCS & Real-Time polyp segmentation\\
   32 &ColonSegNet  & Access & 2021  & \href{https://github.com/DebeshJha/ColonSegNet}{PyTorch} & IS   & KS   &  Residual block and SENet\\
    33&Segtran & IJCAI &2021 & \href{https://github.com/askerlee/segtran}{PyTorch}&   IS    & C6 and KS& Transformer\\
    34&DDANet & ICPR  & 2021  & 
    \href{https://github.com/nikhilroxtomar/DDANet }{PyTorch}&    IS   & KS & Dual decoder attention network\\
    35&UACANet& ACM MM & 2021  & \href{https://github.com/plemeri/UACANet}{PyTorch}    &       IS &PraNet  &\makecell[l]{ Uncertainty augmented \\Context attention network} \\
    36&DivergentNet & ISBI & 2021  & \href{https://github.com/vlbthambawita/divergent-nets}{PyTorch}    &       IS & EndoCV 2021 &Combine multiple models\\
    37&DWHieraSeg & MIA & 2021  & \href{https://github.com/CityU-AIM-Group/DW-HieraSeg}{PyTorch}    &       IS & ES &Dynamic-weighting\\
    38&Transfuse & MICCAI & 2021  & N/A  &IS       & PraNet & Transformer and CNN\\
   39 &SANet & MICCAI & 2021 & \href{https://github.com/weijun88/SANet}{PyTorch} & IS & PraNet & Shallow attention network \\
   40 &PNS-Net & MICCAI &2021 & \href{https://github.com/GewelsJI/PNS-Net}{PyTorch}&   VS    & C6, KS, ES, and AM &\makecell[l]{Progressively normalized \\ self-attention network}
    \\
    \hline
    \end{tabular}%
  \label{tab: model_survey}%
\end{table*}%


\subsection{Vision Transformer}\label{subsec:Transformer}
Transformers use multi-head self-attention (MHSA) layers to model long-term dependencies. 
Unlike the convolutional layer, the MHSA layer has dynamic weights and a global receptive field, making it more flexible and effective.
The transformer~\cite{vaswani2017attention} was first proposed by Vaswani \textit{et al.}~for the machine translation task, and has since had extensive influence on the natural language processing field.
To apply transformers to computer vision tasks, Dosovitskiy \textit{et al.}~\cite{dosovitskiy2020image} proposed a vision transformer (ViT), which was the first pure transformer for image classification.
ViT divides an image into multiple patches, which are sequentially sent to a transformer encoder after being encoded, and then an MLP is used to perform image classification. 
HVT~\cite{pan2021scalable} is based on a hierarchical progressive pooling method to compress the sequence length of a token and reduce the redundancy and number of calculations in ViT. 
The pooling-based vision transformer~\cite{heo2021rethinking} draws on the principle of CNNs whereby, as the depth increases, the number of feature map channels increases, and the spatial dimension decreases.
Yuan~\textit{et al.}~\cite{yuan2021tokens} pointed out that the simple token structure in ViT cannot capture important local features, such as edges and lines, which reduces the training efficiency and leads to redundant attention mechanisms.
T2T ViT was thus proposed to use layer-by-layer tokens-to-token transformation to gradually merge neighboring tokens and model local features, while reducing the token's length. 
TNT~\cite{han2021transformer} employs a transformer suitable for fine-grained image tasks, which further divides the original image patch and conducts self-attention mechanism calculations in smaller units. 
Meanwhile, external and internal transformers are used to extract global and local features. 




To adapt to dense prediction tasks such as semantic segmentation, 
several methods~\cite{wang2021pyramid,wang2021pvtv2,liu2021swin,cvt,coat,twins,levit} have also introduced the pyramid structure of CNNs to the design of transformer backbones.
For instance, PVT-based models~\cite{wang2021pyramid,wang2021pvtv2} use a hierarchical transformer with four stages, showing that a pure transformer backbone can be as versatile as its CNN counterparts, and performs better in detection and segmentation tasks.
In this work, \revI{we design a new transformer-based polyp segmentation framework}, which can accurately locate the boundaries of polyps even in extreme scenarios.

\begin{figure*}[t!]
	\centering
    \small
	\begin{overpic}[width=\linewidth]{Imgs/Polyp-PVT-min.pdf}
    \end{overpic}
    \vspace{-25pt}
	\caption{Framework of our Polyp-PVT, which consists of a pyramid vision transformer (PVT) (a) as the encoder network, (b) cascaded fusion module (CFM) for fusing the high-level feature, (c) camouflage identification module (CIM) to filter out the low-level information, and (d) similarity aggregation module (SAM) for integrating the high- and low-level features for the final output.}
    \label{figure:framework}
\end{figure*}

\section{Proposed Method}\label{sec:Method}
\subsection{Overall Architecture}
As shown in \figref{figure:framework}, the proposed 
Polyp-PVT~consists of four key modules: namely, a pyramid vision transformer (PVT) encoder, cascaded fusion module (CFM), camouflage identification module (CIM), and similarity aggregation module (SAM).
Specifically, the PVT is used to extract multi-scale long-range dependencies features from the input image.
The CFM is employed to collect the semantic cues and locate polyps by aggregating high-level features in a progressive manner.
The CIM is designed to remove noise and enhance low-level representation information of polyps, including texture, color, and edges.
The SAM is adopted to fuse the low- and high-level features provided by the CIM and CFM, effectively transmitting the information from pixel-level polyp to the entire polyp area.

Given an input image $I \in \mathbb{R}^{H \times W \times 3}$, we use the transformer-based backbone~\cite{wang2021pyramid} to extract four pyramid features $X_{i} \in \mathbb{R}^{\frac{H}{2^{i+1}} \times \frac{W}{2^{i+1}} \times C_i}$, where $C_i \in \{64,128,320,512\}$ and $i\in\{1,2,3,4\}$.
Then, we adjust the channel of three high-level features $X_2$, $X_3$ and $X_4$ to 32 through three convolutional units and feed them (\emph{i.e.} $X_2^{'}$, $X_3^{'}$, and $X_4^{'}$) to CFM to fuse, leading a feature map $T_1 \in \mathbb{R}^{\frac{H}{8} \times \frac{W}{8} \times 32}$.
Meanwhile, low-level features $X_{1}$ are converted to $T_2 \in \mathbb{R}^{\frac{H}{4} \times \frac{W}{4} \times 64}$ by the CIM.
After that, the $T_1$ and $T_2$ are 
aligned and fused
by SAM, yielding the final feature map $F \in \mathbb{R}^{\frac{H}{8} \times \frac{W}{8} \times 32}$.
Finally, $F$ is fed into a $1\times1$ convolutional layer to predict the polyp segmentation result $P_2$. We use the sum of $P_1$ and $P_2$ as the final prediction.
During training, we optimize the model with a main loss $\mathcal{L}_{\tt main}$ and an auxiliary loss $\mathcal{L}_{\tt aux}$. The main loss is calculated between the final segmentation result $P_2$ and the ground truth (GT), which is used to optimize the final polyp segmentation result.
Similarly, the auxiliary loss is used to supervise the intermediate result $P_1$ generated by the CFM.



\subsection{Transformer Encoder}\label{subsec:Transformer Encoder}
Due to uncontrolled factors in their acquisition, polyp images tend to contain significant noise, such as \textit{motion blur}, \textit{rotation}, and \textit{reflection}.
Some recent works~\cite{bhojanapalli2021understanding,xie2021segformer} have found that the vision transformer~\cite{dosovitskiy2020image,wang2021pyramid,wang2021pvtv2} demonstrates stronger performance and better robustness to input disturbances than CNNs~\cite{simonyan2015very,he2016deep}.
Inspired by this, we use a vision transformer as our backbone network to extract more robust and powerful features for polyp segmentation. 
Different from~\cite{liu2021swin,dosovitskiy2020image} that use fixed ``columnar'' structure or shifted windowing manner, the PVT~\cite{wang2021pyramid} is a pyramid architecture whose representation is calculated with spatial-reduction attention operations, thus it enables reduce the resource consumption.
Note that the proposed model is backbone-independent, other famous transformer backbones are feasible in our framework. Specifically, we adopt the PVTv2~\cite{wang2021pvtv2} which is the improved version of PVT with a more powerful feature extraction ability.
To adapt PVTv2 to the polyp segmentation task, we remove the last classification layer, and design a polyp segmentation head on top of four multi-scale feature maps (\emph{i.e.} $X_1$, $X_2$, $X_3$, and $X_4$) generated by different stages. 
Among these feature maps, $X_1$ gives detailed appearance information of polyps, and $X_2$, $X_3$, and $X_4$ provide high-level features.


\subsection{Cascaded Fusion Module}\label{subsec:cascaded fusion}
To balance the accuracy and computational resources, we follow recent popular practices~\cite{Wu_2019_CVPR,fan2020pranet} to implement the cascaded fuse module (CFM). 
Specifically, we define $\mathcal{F}(\cdot)$ as a convolutional unit composed of a $3\times 3$ convolutional layer with padding set to 1, batch normalization~\cite{ioffe2015batch} and ReLU~\cite{glorot2011deep}.
As shown in Fig. \ref{figure:framework} (b), the CFM mainly consists of two cascaded parts, as follows:

(1) In part one,
we up-sample the highest-level feature map $X_{4}^{'}$ to the same size as $X_{3}^{'}$ and then pass the result through two convolutional units {$\mathcal{F}_1(\cdot)$ and $\mathcal{F}_2(\cdot)$}, yieldings: $X_{4}^1$ and $X_{4}^2$.
Then, we multiply $X_{4}^1$ and $X_{3}^{'}$ and concatenate the result with $X_{4}^2$.
Finally, we use a convolution unit $\mathcal{F}_3(\cdot)$ to smooth the concatenated feature, yielding fused feature map $X_{34} \in \mathbb{R}^{\frac{H}{16} \times \frac{W}{16} \times 32}$.
The process can be summarized as Eqn.~\ref{eqn:part_1}.
\begin{equation}
X_{34} = \mathcal{F}_{3}(\text{Concat}(\mathcal{F}_1({{{X}}_4^{'}}) \odot {X_{3}^{'}}, \mathcal{F}_2({{{X}}_4^{'}}))),
\label{eqn:part_1}
\end{equation}
where ``$\odot$'' denotes the Hadamard product, and $\text{Concat}(\cdot)$ is the concatenation operation along the channel dimension.

(2) As shown Eqn.~\ref{eqn:part_2}, the second part follows a similar process to part one. 
Firstly, we up-sample $X_{4}^{'}$, $X_{3}^{'}$, $X_{34}$
to the same size as $X_{2}^{'}$, and smooth them using convolutional units $\mathcal{F}_{4}(\cdot)$, $\mathcal{F}_{5}(\cdot)$, and $\mathcal{F}_{6}(\cdot)$, respectively.
Then, we multiply the smoothed $X_{4}^{'}$ and $X_{3}^{'}$ with $X_2^{'}$, and concatenate the resulting map with 
up-sampled and smoothed $X_{34}$.
Finally, we feed the concatenated feature map into two convolutional units (\emph{i.e.} $\mathcal{F}_{7}(\cdot)$ and $\mathcal{F}_{8}(\cdot)$) to reduce the dimension, and obtain $T_{1} \in \mathbb{R}^{\frac{H}{8} \times \frac{W}{8} \times 32}$, which is also the output of the CFM.
\begin{equation}
T_{1} = \mathcal{F}_{8}(\mathcal{F}_{7}(\text{Concat}(\mathcal{F}_{4}(X_4^{'}) \odot \mathcal{F}_{5}(X_3^{'}) \odot {X_2^{'}}, \mathcal{F}_{6}(X_{34})))),
\label{eqn:part_2}
\end{equation}


\subsection{Camouflage Identification Module}\label{subsec:Camouflage Identification}
Low-level features often contain rich detail 
information, such as \textit{texture}, \textit{color}, and \textit{edges}.
However, polyps tend to be very similar in appearance to the background.
Therefore, we need a powerful extractor to identify the polyp details.

As shown in Fig.~\ref{figure:framework} (c), we introduce a camouflage identification module (CIM) to capture the details of polyps from different dimensions of the low-level feature map $X_1$.
Specifically, the CIM consists of a channel attention operation~\cite{woo2018cbam} $\text{Att}_{c}(\cdot)$ and a spatial attention operation~\cite{hu2018squeeze} $\text{Att}_{s}(\cdot)$, which can be formulated as:
\begin{equation}
T_{2} = \text{Att}_{s}\left(\text{Att}_{c}\left(X_1\right)\right),
\label{equ:att}
\end{equation}
The channel attention operation $\text{Att}_{c}(\cdot)$ can be written as follow:
\begin{equation}\label{equ:loss function}
\text{Att}_{c}(x) = \sigma\left(\mathcal{H}_1\left({P_{\rm max}}\left(x\right)\right) + \mathcal{H}_2\left({P_{\rm avg}}\left(x\right)\right)\right) \odot x,
\end{equation}
where $x$ is the input tensor and $\sigma(\cdot)$ is the Softmax function. $P_{\rm max}(\cdot)$ and $P_{\rm avg}(\cdot)$ denote adaptive maximum pooling and adaptive average pooling functions, respectively.
${\mathcal{H}_{i}(\cdot)}, i \in \{1,2\}$ shares parameters and consists of a convolutional layer with $1 \times 1$ kernel size to reduce the channel dimension 16 times, followed by a ReLU layer
and another $1 \times 1$ convolutional layer to recover the original channel dimension.
The spatial attention operation $\text{Att}_{s}(\cdot)$ can be formulated as:
\begin{equation}
\text{Att}_{s}(x) = \sigma(\mathcal{G}(\text{Concat} ({R_{\rm max}}(x), {R_{\rm avg}}(x)))) \odot x,
\label{equ:att_s}
\end{equation}
where $R_{\rm max}(\cdot)$ and $R_{\rm avg}(\cdot)$ represent the maximum and average values obtained along the channel dimension, respectively.
$\mathcal{G}(\cdot)$ is a $7 \times 7$ convolutional layer with padding set to \revI{3}.

\subsection{Similarity Aggregation Module}\label{subsec:GCN}
To explore high-order relations between the lower-level local features from CIM and higher-level cues from CFM. We introduce the non-local~\cite{wang2018non,te2020edge} operation under graph convolution domain~\cite{lu2019graph} to implement our similarity aggregation module (SAM). As a result, SAM can inject detailed appearance features into high-level semantic features using global attention.

\begin{figure}[t!]
	\centering
    \small
	\begin{overpic}[width=.98\linewidth]{Imgs/SAM.pdf}
    \end{overpic}
	\caption{Details of the SAM. The red arrow indicates a transpose. It is composed of GCN and non-local, which extend the pixel features of polyp regions with high-level semantic location cues to the entire region.}
    \label{figure:module ra}
\end{figure}

Given the feature map ${T_1}$, which contains high-level semantic information, and ${T_2}$ with rich appearance details, we fuse them through self-attention.
First, two linear mapping functions $W_\theta(\cdot)$ and $W_\phi(\cdot)$ are applied on ${T_1}$ to reduce the dimension and obtain 
feature maps $Q \in \mathbb{R}^{\frac{H}{8} \times \frac{W}{8} \times 16}$ and $K \in \mathbb{R}^{\frac{H}{8} \times \frac{W}{8} \times 16}$. 
Here, we take a convolution operation with a kernel size of $1\times1$ as the linear mapping process. 
This process can be expressed as:
\begin{equation}\label{equ:q}
Q = W_\theta (T_1),K =  W_\phi (T_1).
\end{equation}

For ${T_2}$, we use a convolutional unit $W_{g}(\cdot)$ to reduce the channel dimension to 32 and interpolate it to the same size as ${T_1}$. 
Then, we apply a Softmax function on the channel dimension and choose the second channel\footnote{\revI{The design choice is the same at \cite{te2020edge}; however, other channels are also feasible if we only update the weight of the selected channel.}} as the attention map, leading to ${T^{'}_2} \in \mathbb{R}^{\frac{H}{8} \times \frac{W}{8} \times 1}$.
These operations are represented as $\text{F}(\cdot)$ in Fig.~\ref{figure:module ra}. 
Next, we calculate the Hadamard product between $K$ and ${T^{'}_2}$.
This operation assigns different weights to different pixels, increasing the weight of edge pixels.
After that, we use an adaptive pooling operation to reduce the displacement of features, and apply a center crop on it to obtain the feature map
$V \in \mathbb{R}^{4 \times 4 \times 16}$.
In summary, the process can be formulated as follows:
\begin{equation}\label{equ:v}
V = \text{AP}(K \odot \text{F}(W_{g} ({T_2}))),
\end{equation}
where $\text{AP}(\cdot)$ denotes the pooling and crop operations.

Then, we establish the correlation between each pixel in $V$ and $K$ through an inner product, which is written as follows:
\begin{equation}\label{equ:f}
f= \sigma (V  \otimes K^\mathsf{T}),
\end{equation}
where "$\otimes$" denotes the inner product operation. $ K^\mathsf{T}$ is the transpose of $K$ and $f$ is the correlation attention map.

After obtaining the correlation attention map $f$, we multiply it with the feature map $Q$, and the result features are fed to the graph convolutional layer~\cite{te2020edge} $\text{GCN}(\cdot)$, leading to $G \in \mathbb{R}^{4 \times4 \times 16}$. 
Same to~\cite{te2020edge}, we calculate the inner product between $f$ and $G$ as Eqn.~\ref{equ:y}, reconstructing the graph domain features into the original structural features:
\begin{equation}\label{equ:y}
Y'= f^\mathsf{T} \otimes \text{GCN}(f  \otimes Q).
\end{equation}

The reconstructed feature map $Y'$ is adjusted to the same channel sizes with $Y$ by a convolutional layer $W_z(\cdot)$ with $1 \times 1$ kernel size, and then combined 
with the feature ${T_1}$ 
to obtain the final output $Z \in \mathbb{R}^{\frac{H}{8} \times \frac{W}{8} \times 32}$ of the SAM. 
Eqn.~\ref{equ:z} summarizes the details of this process:
\begin{equation}\label{equ:z}
Z = T_1 + W_z(Y').
\end{equation}


\subsection{Loss Function}\label{subsec:Supervision Strategy}

{Our loss function can be formulated as Eqn.~\ref{equ:loss}:}
\begin{equation}
\mathcal{L} = \mathcal{L}_{\rm main} + \mathcal{L}_{\rm aux},
\label{equ:loss}
\end{equation}
where $\mathcal{L}_{\rm main}$ and $\mathcal{L}_{\rm aux}$ are the main loss and auxiliary loss, respectively.
The main loss $\mathcal{L}_{\rm main}$ is calculated between the final segmentation result $P_2$ and ground truth $G$, which can be written as:
\begin{equation}
\mathcal{L}_{\rm main} = \mathcal{L}^{w}_{\text{IoU}}(P_2, G) + \mathcal{L}^{w}_{\text{BCE}}(P_2, G).
\label{equ:loss_main}
\end{equation}

{The auxiliary loss $\mathcal{L}_{\rm aux}$ is calculated between the intermediate result $P_1$ from the CFM and ground truth $G$, which can be formulated as:}
\begin{equation}
\mathcal{L}_{\rm aux} = \mathcal{L}^{w}_{\text{IoU}}(P_1, G) + \mathcal{L}^{w}_{\text{BCE}}(P_1, G).
\label{equ:loss_aux}
\end{equation}

$\mathcal{L}^{w}_{\text{IoU}}(\cdot)$ and $\mathcal{L}^{w}_{\text{BCE}}(\cdot)$ are the weighted intersection over union (IoU) loss \cite{wei2020f3net} and weighted binary cross entropy (BCE) loss \cite{wei2020f3net}, which restrict the prediction map in terms of the global structure (object-level) and local details (pixel-level) perspectives.
Unlike the standard BCE loss function, which treats all pixels equally, $\mathcal{L}^{w}_{\text{BCE}}(\cdot)$ considers the importance of each pixel and assigns higher weights to hard pixels.
Furthermore, compared to the standard IoU loss, $\mathcal{L}^{w}_{\text{IoU}}(\cdot)$ pays more attention to the hard pixels.


\subsection{Implementation Details}\label{subsection: Implementation Details}

We implement our Polyp-PVT~with the PyTorch framework and use a Tesla P100 to accelerate the calculations.
Considering the differences in the sizes of each polyp image, 
we adopt a multi-scale strategy~\cite{fan2020pranet,huang2021hardnet} in the training stage. 
The hyperparameter details are as follows.
To update the network parameters, we use the AdamW~\cite{loshchilov2019adamw} optimizer, which is widely used in transformer networks~\cite{wang2021pyramid,liu2021swin,wang2021pvtv2}. 
The learning rate is set to 1e-4 and the weight decay is adjusted to 1e-4 too. 
Further, we resize the input images to $352\times352$ with a mini-batch size of 16 for 100 epochs. \revI{More details about the training loss cures, parameter setting, and network parameters are shown in \figref{figure:loss}, \tabref{tab:train parameters} and \tabref{tab:net_paras}, respectively.}
The total training time is nearly 3 hours to achieve the best (\emph{e.g.} 30 epochs) performance.
For testing, we only resize the images to $352\times352$ without any post-processing optimization strategies.

\begin{figure}[t!]
 	\centering
 	\begin{overpic}[width=.98\linewidth]{./Imgs/loss.pdf}
    \end{overpic}
    \vspace{-5pt}
 	\caption{\revI{Loss curves under different training parameter settings.}}
     \label{figure:loss}
\end{figure}

\begin{table}[t!]
\small
  \centering
  \caption{\revI{Parameter setting during the training stage.}}
   \renewcommand{\arraystretch}{1.0}
   {
	\setlength\tabcolsep{6.0pt}
    \begin{tabular}{cccc}
    \hline
    Optimizer & Learning Rate (lr) & Multi-scale & Clip\\
   
     AdamW & 1e-4 & [0.75,1,1.25] & 0.5\\
    \hline
     Decay rate & Weight decay & Epochs & Input Size \\
   
     0.1 & 1e-4 & 100   & $352\times352$ \\
    \hline
    \end{tabular}%
  \label{tab:train parameters}}%
\end{table}%


\begin{table}[t!]
\small
  \centering
  \caption{\revI{Network parameters of each module. Note that the encoder parameters are the same as PVT without any changes. BasicConv2d and Conv2d with the parameters [in\_channel, out\_channel, kernel\_size, padding] and GCN [num\_state, num\_node].}}
  \setlength\tabcolsep{2.3pt}
	\renewcommand{\arraystretch}{1.0}
    \begin{tabular}{lc|lc}
    \hline
    \multicolumn{2}{c|}{\textbf{Encoder}} & \multicolumn{2}{c}{\textbf{SAM}} \\
    \hline
    patch\_size & [4]     & AvgPool2d & [6] \\
    embed\_dims & [64, 128, 320, 512] & Conv2d & [32,16,1,1] \\
    num\_heads & [1, 2, 5, 8] & Conv2d & [32,16,1,1] \\
    mlp\_ratios & [8, 8, 4, 4] & Conv2d & [16,32,1,1] \\
    depths & [3, 4, 18, 3] & GCN   & [16,16] \\
    sr\_ratios & [8, 4, 2, 1] & BasicConv2d & [64,32,1,0] \\
    drop\_rate & [0]     &       &  \\
    drop\_path\_rate & [0.1]   &       &  \\
    \hline
    \multicolumn{2}{c|}{\textbf{CFM}} & \multicolumn{2}{c}{\textbf{CIM}} \\
    \hline
    BasicConv2d & [32,32,3,1] & AvgPool2d & [1] \\
    BasicConv2d & [32,32,3,1] & AvgPool2d & [1] \\
    BasicConv2d & [32,32,3,1] & Conv2d & [64,4,1,0] \\
    BasicConv2d & [32,32,3,1] & ReLU  &  \\
    BasicConv2d & [64,64,3,1] & Conv2d & [4,64,1,0] \\
    BasicConv2d & [64,64,3,1] & Sigmoid &  \\
    BasicConv2d & [96,96,3,1] & Conv2d & [2,1,7,3] \\
    BasicConv2d & [96,32,3,1] & Sigmoid &  \\
    \hline
    \end{tabular}%
  \label{tab:net_paras}%
\end{table}%


\begin{table*}[t!]
\small
\centering
	\caption{Quantitative results of the test datasets, \emph{i.e.} Kvasir-SEG and ClinicDB. The best results are in \textbf{{boldface}}.
	}
	\renewcommand{\arraystretch}{1.0}
	\setlength\tabcolsep{3.1pt}
    \begin{tabular}{l|ccccccc|ccccccc}
    \hline  
          & \multicolumn{7}{c|}{Kvasir-SEG~\cite{jha2020kvasir}}                                  & \multicolumn{7}{c}{ClinicDB~\cite{bernal2015wm}} \\
\cline{2-15} 
Model & mDic& mIoU & ${ F}_{\beta }^{w}$   & ${S}_{\alpha}$    & $m{E}_{\xi}$   & $max{E}_{\xi}$  & MAE & mDic & mIoU & ${ F}_{\beta }^{w}$   & ${S}_{\alpha}$    & $m{E}_{\xi}$ & $max{E}_{\xi}$  & MAE   \\
    \hline
    MICCAI'15 U-Net  & 0.818  & 0.746  & 0.794  & 0.858  & 0.881    & 0.893 & 0.055  & 0.823  & 0.755  & 0.811  & 0.889  & 0.913    & 0.954 & 0.019    \\
    DLMIA'18 UNet++  & 0.821  & 0.743  & 0.808  & 0.862  & 0.886    & 0.909  & 0.048  & 0.794  & 0.729  & 0.785  & 0.873  & 0.891   & 0.931 & 0.022    \\
    MICCAI'19 SFA     & 0.723  & 0.611  & 0.670  & 0.782  & 0.834    & 0.849& 0.075  & 0.700  & 0.607  & 0.647  & 0.793  & 0.840   & 0.885  & 0.042  \\
    arXiv'21 MSEG & 0.897  & 0.839  & 0.885  & 0.912  & 0.942   & 0.948  & 0.028   & 0.909  & 0.864  & 0.907  & 0.938  & 0.961    & 0.969 & 0.007    \\
    arXiv'21 DCRNet  & 0.886  & 0.825  & 0.868  & 0.911  & 0.933   & 0.941 & 0.035  & 0.896  & 0.844  & 0.890  & 0.933  & 0.964   & 0.978 & 0.010   \\
    MICCAI'20 ACSNet  & 0.898  & 0.838  & 0.882  & 0.920  & 0.941   & 0.952   & 0.032   & 0.882  & 0.826  & 0.873  & 0.927  & 0.947   & 0.959 & 0.011   \\
    MICCAI'20 PraNet  & 0.898  & 0.840  & 0.885  & 0.915  & 0.944    & 0.948 & 0.030   & 0.899  & 0.849  & 0.896  & 0.936  & 0.963   & 0.979 & 0.009    \\
    CRV'21 EU-Net & 0.908  & 0.854  & 0.893  & 0.917  & 0.951  & 0.954  & 0.028    & 0.902  & 0.846  & 0.891  & 0.936  & 0.959   & 0.965 & 0.011    \\
    MICCAI'21 SANet & 0.904  & 0.847  & 0.892  & 0.915  & 0.949    & 0.953  & 0.028 & 0.916  & 0.859  & 0.909  & 0.939  & 0.971    & 0.976 & 0.012   \\
    \hline
    \rowcolor{gray!30}
    \textbf{Polyp-PVT~(Ours)} & \textbf{0.917} & \textbf{0.864} & \textbf{0.911} & \textbf{0.925} & \textbf{0.956}  & \textbf{0.962} & \textbf{0.023}  & \textbf{0.937} & \textbf{0.889} & \textbf{0.936} & \textbf{0.949} & \textbf{0.985}  & \textbf{0.989} & \textbf{0.006}\\
    \hline
    \end{tabular}
  \label{tab:test1_1}
 
\end{table*}%



\section{Experiments}\label{sec:Experiments}
\subsection{Evaluation Metrics} \label{sec:Metrics}
We employ six widely-used evaluation metrics, including Dice~\cite{milletari2016v}, IoU, mean absolute error (MAE), weighted F-measure (${ F}_{\beta }^{w}$){~\cite{margolin2014evaluate}}, S-measure (${S}_{\alpha}$)~\cite{Fan2021S-measure}, and E-measure (${E}_{\xi}$)~\cite{21Fan_HybridLoss,Fan2018Emeasure}  to evaluate the model performances.
Among these metrics, Dice and IoU are similarity measures at the regional level, which mainly focus on the internal consistency of segmented objects. 
Here, we report the mean value of Dice and IoU, denoted as mDic and mIoU, respectively. 
MAE is a pixel-by-pixel comparison indicator that represents the average value of the absolute error between the predicted value and the true value. 
Weighted F-measure (${ F}_{\beta }^{w}$) comprehensively considers the recall and precision, and eliminates the effect of considering each pixel equally in conventional indicators.
S-measure (${S}_{\alpha}$) focuses on the structural similarity of target prospects at the region and object level.
E-measure (${E}_{\xi}$) is used to evaluate the segmentation results at the pixel and image level.
We report the mean and max value of E-measure, denoted as $m{E}_{\xi}$ and $max{E}_{\xi}$, respectively.
The evaluation toolbox is derived from 
\url{ https://github.com/DengPingFan/PraNet}.



\subsection{Datasets and Compared Models}
\textbf{Datasets.} Following the experimental setups in PraNet~\cite{fan2020pranet}, we adopt five challenging public datasets, including Kvasir-SEG~\cite{jha2020kvasir}, ClinicDB~\cite{bernal2015wm}, ColonDB~\cite{tajbakhsh2015automated}, Endoscene~\cite{vazquez2017benchmark} and ETIS~\cite{silva2014toward} to verify the effectiveness of our framework.

\textbf{Models.} We collect several open source models from the field of polyp segmentation, for a total of nine comparative models, including U-Net~\cite{ronneberger2015unet}, UNet++~\cite{zhou2018unet++}, PraNet~\cite{fan2020pranet}, SFA~\cite{fang2019sfa}, MSEG~\cite{huang2021hardnet}, ACSNet~\cite{zhang2020ACSN}, DCRNet~\cite{yin2021duplex}, EU-Net~\cite{PatelBW21} and SANet~\cite{wei2021shallow}. 
For fair comparison, we use their open source codes to evaluate on the same training and testing sets. Note that the SFA results are generated using the released test model. 

\subsection{Analysis of Learning Ability}
\textbf{Settings.} We use the ClinicDB and Kvasir-SEG datasets to evaluate the learning ability of the proposed model. ClinicDB contains 612 images, which are extracted from 31 colonoscopy videos. Kvasir-SEG is collected from the polyp class in the Kvasir dataset, and includes 1,000 polyp images.
Following to the settings in PraNet, we adopt the same 900 and 548 images from ClinicDB and Kvasir-SEG datasets as the training set, and the remaining 64 and 100 images are employed as the respective test sets. 

\begin{figure}[t!]
	\centering
	\begin{overpic}[width=.98\linewidth]{Imgs/excelmap-min.pdf}
    \end{overpic}
	\caption{Evaluation of model generalization ability. We provide the max Dice results on ColonDB and ETIS.}
    \label{figure:generalization ability}
\end{figure}

\textbf{Results.} As can be seen in \tabref{tab:test1_1}, our model is superior to the current methods, demonstrating that it has a better learning ability.
On the Kvasir-SEG dataset, the mDic score of our model is 1.3\% higher than that of the second-best model, SANet, and 1.9\% higher than that of PraNet. 
On the ClinicDB dataset, the mDic score of our model is 2.1\% higher than that of SANet, and 3.8\% higher than that of PraNet.

\begin{table*}[t!]
\small
    \centering
    \renewcommand{\arraystretch}{1.0}
	\caption{Quantitative results of the test datasets ColonDB and ETIS. The SFA result is generated using the published code.}
    \setlength\tabcolsep{3.1pt}
    \begin{tabular}{l|ccccccc|ccccccc}
    \hline
    & \multicolumn{7}{c|}{ColonDB~\cite{tajbakhsh2015automated}}                                  & \multicolumn{7}{c}{ETIS~\cite{silva2014toward}} \\
    \cline{2-15}    
    Model & mDic & mIoU & ${ F}_{\beta }^{w}$   & ${S}_{\alpha}$    & $m{E}_{\xi}$   & $max{E}_{\xi}$ & MAE  & mDic & mIoU & ${ F}_{\beta }^{w}$   & ${S}_{\alpha}$    & $m{E}_{\xi}$  & $max{E}_{\xi}$  & MAE  \\
    \hline
    MICCAI'15 U-Net  & 0.512  & 0.444  & 0.498  & 0.712  & 0.696   & 0.776  & 0.061  & 0.398  & 0.335  & 0.366  & 0.684  & 0.643   & 0.740   & 0.036  \\
    DLMIA'18 UNet++ & 0.483  & 0.410  & 0.467  & 0.691  & 0.680    & 0.760 & 0.064   & 0.401  & 0.344  & 0.390  & 0.683  & 0.629    & 0.776 & 0.035  \\
    MICCAI'19 SFA   & 0.469  & 0.347  & 0.379  & 0.634  & 0.675   & 0.764    & 0.094 & 0.297  & 0.217  & 0.231  & 0.557  & 0.531    & 0.632  & 0.109   \\
    MICCAI'20 ACSNet & 0.716  & 0.649  & 0.697  & 0.829  & 0.839    & 0.851 & 0.039  & 0.578  & 0.509  & 0.530  & 0.754  & 0.737    & 0.764  & 0.059 \\
    arXiv'21 MSEG & 0.735  & 0.666  & 0.724  & 0.834  & 0.859    & 0.875 & 0.038   & 0.700  & 0.630  & 0.671  & 0.828  & 0.854   & 0.890  & 0.015  \\
   
    arXiv'21 DCRNet & 0.704  & 0.631  & 0.684  & 0.821  & 0.840   & 0.848 & 0.052  & 0.556  & 0.496  & 0.506  & 0.736  & 0.742   & 0.773  & 0.096  \\
    MICCAI'20 PraNet & 0.712  & 0.640  & 0.699  & 0.820  & 0.847   & 0.872  & 0.043   & 0.628  & 0.567  & 0.600  & 0.794  & 0.808   & 0.841 & 0.031    \\
    CRV'21 EU-Net & 0.756  & 0.681  & 0.730  & 0.831  & 0.863    & 0.872 & 0.045  & 0.687  & 0.609  & 0.636  & 0.793  & 0.807    & 0.841 & 0.067 \\
    MICCAI'21 SANet & 0.753  & 0.670  & 0.726  & 0.837  & 0.869   & 0.878  & 0.043  & 0.750  & 0.654  & 0.685  & 0.849  & 0.881   & 0.897   & 0.015 \\
    \hline
    \rowcolor{gray!30}
    \textbf{Polyp-PVT~(Ours)} & \textbf{0.808} & \textbf{0.727} & \textbf{0.795} & \textbf{0.865} & \textbf{0.913} & \textbf{0.919} & \textbf{0.031}  & \textbf{0.787} & \textbf{0.706} & \textbf{0.750} & \textbf{0.871} & \textbf{0.906}  & \textbf{0.910} & \textbf{0.013} \\
    \hline
    \end{tabular}%
  \label{tab:test1_2}
\end{table*}%

\begin{table}[t!]
 \centering
 \small
	\caption{Quantitative results of the test dataset Endoscene. The SFA result is generated using the published code.}\label{tab:test1_3}
	\renewcommand{\arraystretch}{1.0}
	\setlength\tabcolsep{1.7pt}
    \begin{tabular}{l|ccccccc}
    \hline
     & \multicolumn{7}{c}{Endoscene~\cite{vazquez2017benchmark}} \\
\cline{2-8}    Model & mDic & mIoU & ${ F}_{\beta }^{w}$   & ${S}_{\alpha}$    & $m{E}_{\xi}$    & $max{E}_{\xi}$ & MAE \\
    \hline
    U-Net  & 0.710  & 0.627  & 0.684  & 0.843  & 0.847    & 0.875& 0.022    \\
    UNet++ & 0.707  & 0.624  & 0.687  & 0.839  & 0.834  & 0.898  & 0.018   \\
    SFA   & 0.467  & 0.329  & 0.341  & 0.640  & 0.644    & 0.817 & 0.065  \\
    MSEG & 0.874  & 0.804  & 0.852  & 0.924  & 0.948    & 0.957 & 0.009   \\
    ACSNet & 0.863  & 0.787  & 0.825  & 0.923  & 0.939  & 0.968  & 0.013  \\
    DCRNet & 0.856  & 0.788  & 0.830  & 0.921  & 0.943   & 0.960   & 0.010  \\
    PraNet & 0.871  & 0.797  & 0.843  & 0.925  & 0.950   & 0.972 & 0.010    \\
    EU-Net & 0.837  & 0.765  & 0.805  & 0.904  & 0.919    & 0.933  & 0.015  \\
    SANet & 0.888  & 0.815  & 0.859  & 0.928  & 0.962   & 0.972  & 0.008  \\
    \hline
    \rowcolor{gray!30}
    \textbf{Polyp-PVT} & \textbf{0.900} & \textbf{0.833} & \textbf{0.884} & \textbf{0.935} & \textbf{0.973}& \textbf{0.981}   & \textbf{0.007}  \\
    \hline
    \end{tabular}
\end{table}


\subsection{Analysis of Generalization Ability}
\textbf{Settings.} To verify the generalization performance of the model, we test it on three unseen (\emph{i.e.} Polycentric) datasets, namely ETIS, ColonDB and EndoScene.
There are 196 images in ETIS, 380 images in ColonDB and 60 images in EndoScene.  
It is worth noting that the images in these datasets belong to different medical centers. In other words, the model has not seen their training data, which is different from the verification methods of ClinicDB and Kvasir-SEG.






\textbf{Results.} The results are shown in Tab.~\ref{tab:test1_2} and Tab.~\ref{tab:test1_3}. As can be seen, our Polyp-PVT~achieves a good generalization performance compared with the existing models.
On ColonDB, it is ahead of the second-best SANet and classical PraNet by 5.5\% and 9.6\%, respectively. On ETIS, we exceed the SANet and PraNet by 3.7\% and 15.9\%, respectively. In addition, on EndoScene, our model is better than SANet and PraNet by 1.2\% and 2.9\%, respectively. 
Moreover, to prove the generalization ability of Polyp-PVT, we present the max Dice results in \figref{figure:generalization ability},
where our model shows a steady improvement on both ColonDB and ETIS.
\revI{In addition, we show the standard deviation (SD) of the mean dice (mDic) between our model and others in \tabref{tab:sd}. As seen, there is not much difference in SD between our model and the comparison model, and they are both stable and balanced.}




\textbf{Effectiveness of CIM.} To demonstrate the ability of the CIM, we also remove it from Polyp-PVT, denoting this as ``Polyp-PVT (w/o CIM)''.
As shown in Tab.~\ref{tab:abalation study}, this variant performs worse than the overall Polyp-PVT. 
Specifically, removing the CIM causes the mDic to decrease by 1.8\% on Endoscene.
Meanwhile, it is obvious that the lack of the CIM introduces significant noise (please refer to Fig.~\ref{figure:Visualization_ablation study}).

\begin{figure*}[t!]
	\centering
	\begin{overpic}[width=.9\linewidth]{Imgs/visualization1-min.pdf}
	\small
	\put(5,-1.5){Image}
	\put(20,-1.5){GT}
	\put(33,-1.5){\color{red}Ours}
	\put(48,-1.5){SANet}
	\put(62,-1.5){PraNet}
	\put(75,-1.5){ACSNet}
	\put(90,-1.5){DCRNet}
    \end{overpic}
    \vspace{2pt}
	\caption{Visualization results with the current models. Green indicates a correct polyp. Yellow is the missed polyp. Red is the wrong prediction. As we can see, the proposed model can accurately locate and segment polyps, regardless of the number of size.}
    \label{figure:V1}
\end{figure*}

\begin{figure*}[t!]
	\centering
	\begin{overpic}[width=.9\linewidth]{Imgs/visualization2-min.pdf}
    \small
	\put(4,-1.5){Image}
	\put(17,-1.5){GT}
	\put(30,-1.5){\color{red}Ours}
	\put(41,-1.5){EU-Net}
	\put(53,-1.5){HarDNet}
	\put(67,-1.5){SFA}
	\put(79,-1.5){U-Net}
	\put(91,-1.5){UNet++}
    \end{overpic}
    \vspace{5pt}
	\caption{Visualization results with the current models.}
    \label{figure:V2}
\end{figure*}


\subsection{Qualitative Analysis}
Fig.~\ref{figure:V1} and Fig.~\ref{figure:V2} show the visualization results of our model and the compared models. We find that our results have two advantages. 
1) Our model is able to adapt 
to data under different conditions. That is, it maintains a stable recognition and segmentation ability under different acquisition environments, such as different lighting, contrast, reflection, motion blur, \emph{etc}
2) The model segmentation results have internal consistency, and predicted edges are closer to the ground-truth labels.
\revI{We also provide FROC curves on ColonDB in Fig.~\ref{figure:froc}, and our result is at the top, indicating that our effect achieves the best.}

\begin{table*}[t!]
  \centering
  \small
  \caption{{\revI{The standard deviation (SD) of the mean dice (mDic) of our model and the comparison models.}}}
  \setlength\tabcolsep{19.5pt}
	\renewcommand{\arraystretch}{1}
    \begin{tabular}{l|c|c|c|c|c}
    \hline 
    Datasets &Kvasir-SEG & ClinicDB &ColonDB & ETIS & Endoscene \\
    \hline 
    Metrics& mDic $ \pm$ SD    & mDic $ \pm$ SD    & mDic $ \pm$  SD    & mDic $ \pm$  SD    & mDic $ \pm$  SD \\
    \hline 
    U-Net  & .818 $ \pm$ .039  & .823  $ \pm$ .047  & .483 $ \pm$  .034  & .398  $ \pm$  .033  & .710  $ \pm$  .049  \\
    UNet++ & .821  $ \pm$ .040  & .794 $ \pm$ .044  & .456  $ \pm$  .037  & .401  $ \pm$  .057  & .707  $ \pm$  .053  \\
    SFA   & .723 $ \pm$  .052  & .701 $ \pm$  .054  & .444  $ \pm$  .037  & .297  $ \pm$  .025  & .468  $ \pm$  .050  \\
    MSEG  & .897  $ \pm$  .041  & .910  $ \pm$  .048  & .735  $ \pm$  .039  & .700  $ \pm$  .039  & .874  $ \pm$  .051  \\
    ACSNet & .898  $ \pm$  .045  & .882  $ \pm$  .048  & .716  $ \pm$  .040  & .578  $ \pm$  .035  & .863  $ \pm$ .055  \\
    DCRNet & .886  $ \pm$  .043  & .896  $ \pm$  .049  & .704  $ \pm$  .039  & .556  $ \pm$  .039  & .857  $ \pm$  .052  \\
    PraNet & .898  $ \pm$  .041  & .899  $ \pm$  .048  & .712  $ \pm$  .038  & .628 $ \pm$  .036  & .871  $ \pm$  .051  \\
    EU-Net & .908 $ \pm$  .042  & .902  $ \pm$  .048  & .756 $ \pm$  .040  & .687  $ \pm$ .039  & .837  $ \pm$  .049  \\
    SANet & .904  $ \pm$ .042  & .916  $ \pm$  .049  & .752  $ \pm$  .040  & .750  $ \pm$  .047  & .888  $ \pm$  .054  \\
    \hline 
    \rowcolor{gray!30}
    \textbf{Ours} & \textbf{.917}  $ \pm$  .042  & \textbf{.937}  $ \pm$  .050  & \textbf{.808}  $ \pm$  .043  & \textbf{.787}  $ \pm$  .044  & \textbf{.900} $ \pm$  .052  \\
    \hline 
    \end{tabular}%
  \label{tab:sd}%
\end{table*}%



\begin{table}[t!]
  \centering
  \small
	\caption{Quantitative results for ablation studies.}\label{tab:abalation study}
	\setlength\tabcolsep{1.7pt}
	\renewcommand{\arraystretch}{1.0}
    \begin{tabular}{lc|c|c|c|c|c}
    \hline
    Dataset & Metric & Bas.     & w/o CFM     & w/o CIM      & w/o SAM & Final\\
    \hline
    \multirow{2}[0]{*}{Endoscene} & mDic  & 0.869 & 0.892 & 0.882 & 0.874 & {0.900} \\
          & mIoU  & 0.792 & {0.826} & 0.808 & 0.801 & {0.833} \\
  \hline
    \multirow{2}[0]{*}{ClinicDB} & mDic  & 0.903 & 0.915 & 0.930 & 0.930 & {0.937} \\
          & mIoU  & 0.847 & 0.865 & 0.881& 0.877 & {0.889} \\
\hline
    \multirow{2}[0]{*}{ColonDB} & mDic  & 0.796  & 0.802 & 0.805 & 0.779 & {0.808} \\
          & mIoU  & 0.707   & 0.721 & 0.724 & 0.696 & {0.727} \\
  \hline
    \multirow{2}[0]{*}{ETIS} & mDic  & 0.759 & 0.771 & 0.785 & 0.778 & {0.787} \\
          & mIoU  & 0.668 & 0.690 & {0.711} & 0.693 & {0.706} \\
  \hline
    \multirow{2}[0]{*}{Kvasir-SEG} & mDic  & 0.910 & 0.922 & 0.910 & 0.910 & 0.917 \\
          & mIoU  & 0.856 & 0.872 & 0.858 & 0.853 & {0.864} \\
      \hline
    \end{tabular}
\end{table}%

\subsection{Ablation Study}
We describe in detail the effectiveness of each component on the overall model. The training, testing, and hyperparameter settings are the same as mentioned in \secref{subsection: Implementation Details}.
The results are shown in Tab.~\ref{tab:abalation study}.


\begin{figure}[t!]
 	\centering
 	\begin{overpic}[width=\linewidth]{./Imgs/ColonDB.pdf}
    \end{overpic}
    \vspace{-20pt}
 	\caption{\revI{FROC curves of different methods on ColonDB.}}
     \label{figure:froc}
\end{figure}


\textbf{Components.} We use PVTv2~\cite{wang2021pvtv2} as our baseline (Bas.) and evaluate module effectiveness by removing or replacing components from the complete Polyp-PVT and comparing the variants with the standard version. 
The standard version is denoted as ``Polyp-PVT (PVT+CFM+CIM+SAM)'', where ``CFM'', ``CIM'' and ``SAM'' indicate the usage of the CFM, CIM and SAM, respectively.

\textbf{Effectiveness of CFM.} To analyze the effectiveness of the CFM, a version of ``Polyp-PVT (w/o CFM)'' is trained. 
\tabref{tab:abalation study} shows that the model without the CFM drops sharply on all five datasets compared to the standard Polyp-PVT.
In particular, the mDic is reduced from 0.937 to 0.915 on ClinicDB. 

\textbf{Effectiveness of SAM.} Similarly, we test the effectiveness of the SAM module by removing it from the overall Polyp-PVT~and replacing it with an element-wise addition operation, which is denoted as ``Polyp-PVT (w/o SAM)''.
The performance of the complete Polyp-PVT~shows an improvement of 2.9\% and 3.1\% in terms of mDic and mIoU respectively, on ColonDB.
Fig.~\ref{figure:Visualization_ablation study} shows the benefits of SAM more intuitively.
It is found that the lack of the SAM leads to more detailed errors or even missed inspections.





\subsection{Video Polyp Segmentation}
To validate the superiority of the proposed model, we conduct experiments on the video polyp segmentation datasets.
For fair comparison, we re-train our model with the same training datasets and use the same testing set as PNS-Net~\cite{ji2021pnsnet}.
We compare our model on three standard benchmarks (\emph{i.e.} CVC-300-TV~\cite{bernal2012towards}, CVC-612-T~\cite{bernal2015wm}, and CVC-612-V~\cite{bernal2015wm}) against six \revI{SOTA} approaches, including
U-Net~\cite{ronneberger2015unet}, UNet++~\cite{zhou2018unet++}, ResUNet++~\cite{JhaSRJLHJ19}, ACSNet~\cite{zhang2020ACSN}, PraNet~\cite{fan2020pranet}, PNS-Net~\cite{ji2021pnsnet}, in \tabref{tab:VPS1} and \tabref{tab:VPS2}.
Note that all the prediction maps of the compared methods are provided by PNS-Net.
\revI{As seen}, our method is very competitive, and far ahead of the best existing model, PNS-Net, by 3.1\% and 6.7\% on CVC-612-V and CVC-300-TV, respectively, in terms of mDice.

\begin{figure*}[t!]
	\centering
	\begin{overpic}[width=\linewidth]{Imgs/ablation.pdf}
    \small
	\put(6,-2.5){Image}
	\put(21,-2.5){GT}
	\put(35,-2.5){Bas.}
	\put(46,-2.5){w/o CFM}
	\put(60,-2.5){w/o CIM}
	\put(75,-2.5){w/o SAM}
	\put(91,-2.5){\color{red}Ours}
    \end{overpic}
    \vspace{-5pt}
	\caption{Visualization of the ablation study results, which are converted from the output into heat maps. As can be seen, removing any module leads to missed or incorrectly detected results.}
    \label{figure:Visualization_ablation study}
\end{figure*}

\begin{table*}[t!]
\centering
\small
	\caption{The result of video polyp segmentation on the \emph{i.e.} CVC-612-T and CVC-612-V, where the best results are in \textbf{boldface}.
	}
	\renewcommand{\arraystretch}{1.0}
	\setlength\tabcolsep{3pt}

    \begin{tabular}{l|ccccccc|ccccccc}
    \hline   
          & \multicolumn{7}{c|}{CVC-612-T~\cite{bernal2015wm}}                                  & \multicolumn{7}{c}{CVC-612-V~\cite{bernal2015wm}} \\
\cline{2-15} 
Model & mDic& mIoU & ${ F}_{\beta }^{w}$   & ${S}_{\alpha}$    & $m{E}_{\xi}$    & $max{E}_{\xi}$ & MAE  & mDic & mIoU & ${ F}_{\beta }^{w}$   & ${S}_{\alpha}$    & $m{E}_{\xi}$    & $max{E}_{\xi}$ & MAE \\
    \hline
    MICCAI'15 U-Net  & 0.711  & 0.618  & 0.694  & 0.810  & 0.836    & 0.853 & 0.058 & 0.709  & 0.597  & 0.680  & 0.826  & 0.855    & 0.872  & 0.023 \\
    TMI'19 UNet++ & 0.697  & 0.603  & 0.688  & 0.800  & 0.817    & 0.865 & 0.059 & 0.668  & 0.557  & 0.642  & 0.805  & 0.830    & 0.846 & 0.025  \\
    ISM'19 ResUNet++ & 0.616  & 0.512  & 0.604  & 0.727  & 0.758   & 0.760 & 0.084   & 0.750  & 0.646  & 0.717  & 0.829  & 0.877    & 0.879  & 0.023 \\
     MICCAI'20 ACSNet & 0.780  & 0.697  & 0.772  & 0.838  & 0.864    & 0.866  & 0.053 & 0.801  & 0.710  & 0.765  & 0.847  & 0.887   & 0.890  & 0.054  \\
    MICCAI'20 PraNet & 0.833  & 0.767  & 0.834  & 0.886  & 0.904    & \textbf{0.926} & 0.038  & 0.857  & 0.793  & 0.855  & 0.915  & 0.936   & 0.965   & 0.013 \\
    MICCAI'21 PNS-Net & 0.837  & 0.765  & 0.838  & 0.903  & 0.903    & 0.923  & 0.038 & 0.851  & 0.769  & 0.836  & 0.923  & 0.944    & 0.962  & 0.012 \\
    \hline
    \rowcolor{gray!30}
    \textbf{Polyp-PVT~(Ours)} & \textbf{0.846} & \textbf{0.776} & \textbf{0.850} & \textbf{0.895} & \textbf{0.908} & \textbf{0.926} & \textbf{0.037} & \textbf{0.882} & \textbf{0.810} & \textbf{0.874} & \textbf{0.924} & \textbf{0.963} & \textbf{0.967} & \textbf{0.012} \\
    \hline
    \end{tabular}%
  \label{tab:VPS1}
 
\end{table*}%

\begin{table}[t!]
 \centering
	\caption{Video polyp segmentation results on the CVC-300-TV.}
	\renewcommand{\arraystretch}{1.0}
	\setlength\tabcolsep{2.0pt}
	\small
    \begin{tabular}{l|ccccccc}
    \hline
     & \multicolumn{7}{c}{CVC-300-TV~\cite{bernal2012towards}} \\
\cline{2-8}    Model & mDic & mIoU & ${ F}_{\beta }^{w}$   & ${S}_{\alpha}$    & $m{E}_{\xi}$   & $max{E}_{\xi}$  & MAE\\
    \hline
    U-Net & 0.631  & 0.516  & 0.567  & 0.793  & 0.826   & 0.849   & 0.027 \\
    UNet++ & 0.638  & 0.527  & 0.581  & 0.796  & 0.831   & 0.847  & 0.024 \\
    ResUNet++ & 0.533  & 0.410  & 0.469  & 0.703  & 0.718   & 0.720  & 0.052 \\
    ACSNet & 0.732  & 0.627  & 0.703  & 0.837  & 0.871    & 0.875  & 0.016 \\
    PraNet & 0.716  & 0.624  & 0.700  & 0.833  & 0.852   & 0.904 & 0.016   \\
    PNS-Net & 0.813  & 0.710  & 0.778  & 0.909  & 0.921    & 0.942 & 0.013  \\
     \hline
     \rowcolor{gray!30}
    \textbf{Ours} & \textbf{0.880} & \textbf{0.802} & \textbf{0.869} & \textbf{0.915} & \textbf{0.961}  & \textbf{0.965} & \textbf{0.011}\\
    \hline
    \end{tabular}
      \label{tab:VPS2}
\end{table}



\subsection{Limitations}
\revI{Although the proposed Polyp-PVT~model surpasses existing algorithms, it still performs poorly in certain cases. 
 %
We present some failure cases in Fig.~\ref{figure:limitation}.
As can be seen, one major limitation is the inability to detect accurate polyp boundaries with overlapping light and shadow ($1^{st}$ row).
Our model can identify the location information of polyps (green mask in $1^{st}$ row), but it regards the light and shadow part of the edge as the polyp (red mask in $1^{st}$ row). 
More deadly, our model incorrectly predicts the reflective point as a polyp (red mask in $2^{nd}$ and $3^{rd}$ rows).
We notice that the reflective points are very salient in the image. 
Therefore, we speculate that the prediction may be based on only these points.
More importantly, we believe that a simple way is to convert the input image into a gray image, which can eliminate the reflection and overlap of light and shadow to assist the model in the judgment.
}

\begin{figure*}[t!]
 	\centering
 	\begin{overpic}[width=.98\textwidth]{Imgs/limitation.pdf}
 	\small
	\put(6,-2){Image}
	\put(28,-2){GT}
	\put(48,-2){\color{red}Ours}
	\put(67,-2){SANet}
	\put(87,-2){PraNet}
    \end{overpic}
 	\caption{Visualization of some failure cases. Green indicates a correct polyp. Yellow is the missed polyp. Red is the wrong prediction.}
     \label{figure:limitation}
\end{figure*}


\section{Conclusion}
In this paper, we propose a new image polyp segmentation framework, named \textbf{Polyp-PVT}, which utilizes a pyramid vision transformer backbone as the encoder to explicitly extract more powerful and robust features.
Extensive experiments show that Polyp-PVT~consistently outperforms all current cutting-edge models on five challenging datasets without any pre-/post-processing.
In particular, for the unseen ColonDB dataset, the proposed model reaches a mean Dice score of above 0.8 for the first time.
Interestingly, we also surpass the current state-of-the-art PNS-Net in terms of video polyp segmentation task, demonstrating excellent learning ability. 
Specifically, we obtain the above-mention achievements by introducing three simple components, \emph{i.e.} a cascaded fusion module (CFM), a camouflage identification module (CIM), and a similarity aggregation module (SAM), which effectively extract high and low-level cues separately, 
and effectively fuse them for the final output. 
We hope this research will stimulate more novel ideas for solving the polyp segmentation task.


{
\bibliographystyle{IEEEtran}
