\section{Introduction} \label{sec:introduction}
Recent semantic segmentation algorithms \cite{DeeplabV3+:2018Chen,Wider:2019Wu} have successfully solved challenging benchmark datasets for semantic segmentation tasks like PASCAL VOC~\citep{VOC:Everingham2015} and MS COCO~\citep{COCO:Lin2014}.
To do so, however, they use a large number of pixel-level annotations,
which require extensive human labor to obtain.
Great attention in computer vision research has thus turned to \emph{weakly-supervised} learning \citep{OAA:2019Jiang,SEAM:2020Wang,MCIS:2020Sun}.
Weakly-supervised semantic segmentation aims to classify each pixel of test images, trained only on image-level labels
(whether a class is present in the image, but not its location).
Although weakly-supervised approaches have seen success in both semantic segmentation and object localization tasks, \citet{EvalWSOL:Choe2020} cast significant doubt on their validity and practicality.
They argue that although weakly-supervised learning algorithms are designed to be trained only on image-level labels, they inevitably use explicit pixel-level labels (or, equivalently, manual judgement of outputs) in hyperparameter tuning.
Since at least \emph{some} fully-supervised inputs are necessary,
\citeauthor{EvalWSOL:Choe2020} point out that simply using a small number of these,
{e.g}\onedot} \def\Eg{{E.g}\onedot five per class,
to train a fully-supervised localization model
substantially outperforms a weakly-supervised counterpart.
To still take advantage of less-expensive weakly-supervised data points, though,
perhaps the most natural change is
to the semi-weakly supervised semantic segmentation (SWSSS) task:
here only a small number of pixel-level labels are provided, as well as a large number of image-level labels.

Segmentation networks trained on a small number of pixel-level labels often confuse similar classes, {e.g}\onedot} \def\Eg{{E.g}\onedot \labelname{cat} and \labelname{horse}, as they architecturally tend to focus on local features rather than farther-away distinguishing areas.
Thus, the additional supervision from image-level labels can be potentially quite helpful.
Most existing SWSSS methods generate pseudo-labels from a classifier using class activation maps (CAMs) \cite{CAM:Zhou2015},
then train a segmentation network using both pseudo-labels and true pixel labels.
These pseudo-labels, however, are difficult to extract:
they tend to focus on small discriminative regions of objects,
ignoring less-distinctive bulks of objects and often including nearby pixels that are not part of objects.
Our analysis shows that as the baseline segmentation model improves with more training data,
the pseudo-labels quickly provide more incorrect than correct supervision to what the model already would have predicted.
Previous methods have thus employed additional information, such as saliency maps, or additional processing methods, adding complexity and many more hyperparameters to tune on a fully-supervised validation set.
We use weak supervision data differently,
without requiring any side information and introducing far fewer new hyperparameters.



To motivate our method, consider \cref{fig:gt_filtering}.
Baseline models with small training sets predict many classes which are not present in the image.
If we ignore the predictions for the class of any pixel which are not present in the image at all, which we term \textit{oracle filtering}, then the segmentation performance improves dramatically.
Inspired by this, we propose a simple algorithm we call \textit{prediction filtering}.
Prediction filtering uses a multi-label classifier trained on only image-level labels
to filter out segmentation predictions deemed very unlikely by the classifier, replacing predictions for those pixels with the next-most-likely class allowed through the filter.
It is compatible with any segmentation model, and
the threshold for ``very unlikely'' is the only new hyperparameter introduced.

Although the classifier is not perfect, because it is trained on a large weakly-supervised set, its predictions tend to be quite accurate.
Moreover, it is trying to solve an easier problem than the segmentation network, using a different architecture.
As we will see in the experiments, even without any additional weakly-supervised data, prediction filtering tends to improve the segmentation performance.
When applied to baseline segmentation models, the performance significantly improves; adding it to the baseline model variant from \citet{Dual:2020Luo} achieves (to our knowledge) the new highest performance on PASCAL VOC in the SWSSS regime.
As prediction filtering is so general, it can even be easily applied to models which already exploit weakly-supervised data via pseudo-labels;
doing so on the state-of-the-art SWSSS algorithms \citep{CCT:2020Ouali,Dual:2020Luo} yields a new model with significantly higher performance,
with more improvement for models trained on fewer fully-labeled images.






\begin{figure}[t!]
  \centering
  \includegraphics[width=0.48\textwidth]{figures/gt_filtering_smaller.pdf}
  \caption{Filtering this model's predicted classes drastically improves segmentation quality.}
  \label{fig:gt_filtering}
  \vspace*{-3.5mm}
\end{figure}


\section{Approaches to Semantic Segmentation}
\label{sec:related_work}

\paragraph{Fully-supervised semantic segmentation.}
In this task, we have a training set of images with pixel-level class labels:
$\mathcal{D}_\mathit{pixel} = \{(x_i, y_i)\}_{i=1}^M$, where
$x_i \in \mathbb{R}^{3 \times H_i\times W_i}$ are images
and $y_i \in \{0,1\}^{K\times H_i\times W_i}$ are pixel labels,
with $K$ the number of classes.
Our goal is to find a model that can predict pixel labels $y$ given a new image $x$.

Current approaches are mostly based on convolutional networks.
One important factor to their success is using larger receptive fields via dilated convolutional layers \citep{Dilated:2015Yu,DeeplabV3:2017Chen}.
Even so, state-of-the-art algorithms still misclassify many pixels
when multi-label classifiers on the same data obtain near-perfect accuracy.
We conjecture this is because segmentation models still miss global structure of the image when looking at an individual pixel.
We will exploit that a classifier ``looks at images differently.''


\paragraph{Weakly-supervised semantic segmentation.}
To avoid extensive human labor for pixel-level annotation, there have been many attempts to replace pixel-level labels with image-level labels:
$\mathcal{D}_\mathit{image} = \{(x_i, z_i)\}_{i=1}^N$,
with $z_i \in \{0,1\}^{K}$ a ``logical or'' of each channel of the unknown $y_i$.
We still want a model to produce $y$.
The most common pipeline for weakly-supervised semantic segmentation is to generate a class activation map (CAM) \citep{CAM:Zhou2015}, refine it with various post-processing methods, then use it as a pseudo-label to train a semantic segmentation network.
However, CAMs tend to focus only on discriminative regions of an object.
Prior work has attempted to expand the CAM to entire objects by masking out parts of an image \citep{HaS:Singh2017} or intermediate feature \citep{ACoL:Zhang2018,ADL:Choe2019};
these methods do indeed expand the resulting CAM, but often too much, in very unstable ways.
Another popular approach to grow the CAM regions, via methods proposed by \citet{CRF:Krahenbuhl2011}, \citet{DSRG:2018Huang}, or \citet{PSA:2018Ahn}, until it converges to the object region \cite{Boundary:2020Chen}.




\paragraph{Semi-weakly supervised semantic segmentation.}
\citet{EvalWSOL:Choe2020} point out fundamental problems with weakly-supervised learning, as discussed in \cref{sec:introduction}.
We thus consider combining a small number of pixel-annotated images $\mathcal{D}_\mathit{pixel}$ with many weakly-supervised images $\mathcal{D}_\mathit{image}$, which we refer to as semi-weakly supervised semantic segmentation (SWSSS).
Although they have not used exactly this name, many papers have already addressed this setting.
Broadly speaking, the most common approach is to generate pseudo-labels from a CAM, and use these in combination with true labels to train a segmentation network
\citep{WSSL:2015Papandreou,MDC:2018Wei,GAIN:2018Li,CCT:2020Ouali,Dual:2020Luo,AdvCAM:2021Lee,vessel2022dang,semi_context_const2021lai}.
(For lack of space, we unfortunately elide all details of these approaches.)
Because image-level labels have no spatial information, however, it is fundamentally difficult to make accurate pseudo-labels.
As we will now argue, as the number of pixel labels increases and base models improve, the benefit of pseudo-labels drastically diminishes.










\label{sec:problem}





\begin{figure*}[t]
    \centering
    \includegraphics[width=0.8\textwidth]{figures/problems_extended3.pdf}
    \caption{A segmentation network trained on few pixel-level labels confuses similar classes, as it does not capture the global features (panel a). To alleviate this issue, pseudo-labels have been widely used; this approach is sensitive to the values of additional hyperparameters, and the quality of its supervision drastically diminishes as the number of pixel labels increases (panel c).}
    \vspace*{-2mm}
    \label{fig:problems}
\end{figure*}



To demonstrate this, we train a DeeplabV1 segmentation network \citep{DeeplabV1:2015Chen} on $\mathcal{D}_\mathit{pixel}$ consisting of $1{,}464$ images from PASCAL VOC training set, and a VGG16 classifier on $\mathcal{D}_\mathit{image}$ containing all $10{,}582$ images in the full training set.
To examine what part of the image causes this prediction, we extract CAMs for \labelname{cat} and \labelname{horse} classes using GradCAM~\cite{Gradcam:Selvaraju2016}.
Although the classifier confidently predicts that only the \labelname{cat} class is present in the given image ($\Pr(\labelname{cat}) = 0.97$, $\Pr(\labelname{horse}) = 0.03$), \cref{fig:problems}(a) shows that the segmentation model predicts most of the cat's body as \labelname{horse} (pink region).
The CAM shows that the classifier makes this decision based on the most discriminative region of the object, {i.e}\onedot} \def\Ie{{I.e}\onedot the cat's head.
The segmentation model does the same at the green (top) location, correctly predicted as \labelname{cat};
at the yellow (middle) location, however, the \labelname{horse} prediction is based mostly on the more ambiguous body area.
As $\mathcal{D}_\mathit{pixel}$ grows, this phenomenon is largely alleviated; it seems $1{,}464$ images were not enough for the segmentation model to learn ``where to look.''


Supervision added by previous models from classifier CAMs, then,
will also tend to focus on discriminative regions of an object,
and can therefore be misleading.
To estimate the effectiveness of the pseudo-labels, we define a measure called \textit{mNet}, the mean over classes $c$ of
\[
    \mathit{net}_c =
   
    \frac{\sum_{i=1}^N \!\!\splitfrac{%
        \textcolor{myBlue}{\operatorname{area}\left(
            \left( \mathit{Pseudo}_{i,c} \setminus \mathit{Pred}_{i,c} \right)
            \cap \mathit{GT}_{i,c}
        \right)} }{%
        - \textcolor{myRed}{\operatorname{area}\left(
            \left( \mathit{Pseudo}_{i,c} \setminus \mathit{Pred}_{i,c} \right)
            \setminus \mathit{GT}_{i,c}
        \right))} }
    }{
        \sum_{i=1}^N \operatorname{area}(\mathit{Pseudo}_{i,c} \setminus \mathit{Pred}_{i,c})
    }
    \label{eq:net}
.\]
Here
the subscript $\cdot_{i,c}$ refers to the set of pixels of $i$-th training image whose label is $c$;
$\mathit{GT}$ refers to the ground truth labels $y$,
$\mathit{Pred}$ to the predicted labels from a baseline segmentation model,
and $\mathit{Pseudo}$ to the CAM-based pseudo-labels.
The first (blue) term in the numerator measures how much correct supervision the pseudo-label adds (see \cref{fig:problems}(b));
the second (red) term, how much incorrect supervision is added.
The denominator combines these two regions.
$\mathit{mNet}$ does not exactly measure how much the pseudo-labels help predictions, but it gives a rough sense of how correct their information is.

\Cref{fig:problems}(c) shows that ``plain'' CAM (in green, lowest) indeed helps when $\mathcal{D}_\mathit{pixel}$ is very small, but as it grows, $\mathit{mNet}$ even becomes negative.
It is possible to improve these predictions by, for instance, post-processing with a CRF (blue line, top).
This, however, requires a complicated structure with several additional hyperparameters to tune on a fully-supervised validation set;
setting these parameters correctly significantly affects performance,
as shown {e.g}\onedot} \def\Eg{{E.g}\onedot by the substantially worse $\mathit{mNet}$ when changing the threshold for a foreground region $\theta$ from $0.3$ to $0.2$ (orange line, middle).







   
   
   
   
   

\section{Prediction Filtering}
\label{sec:prediction_filtering}
\paragraph{Motivation.}
Given a segmentation network $f$,
whose output on the $i$-th image is $f(x_i) \in \mathbb{R}^{K \times H\times W}$,
the final prediction at each pixel $(h, w)$ is normally
\begin{align}
    \hat{y}_{h,w} = \argmax_{c \in \mathcal K} {f(x_i)_{c, h, w}}
    \label{eq:normal}
,\end{align}
where $\mathcal K = \{1, 2, \dots, K\}$.
\emph{Oracle filtering} (\cref{fig:gt_filtering}) instead only considers classes actually present in the image, maximizing over $\mathcal{\tilde K}_i = \{ c : z_{i,c} = 1 \}$.
This improves segmentation performance substantially;
the mIoU (mean intersection over union, the standard segmentation performance metric) 
of a DeeplabV1 segmentation network trained on $\mathcal{D}_\mathit{pixel}$ with $1{,}464$ images improves from $61.8$ to $70.6$.
We conjecture this is because the segmentation network has not learned to appropriately consider global features when predicting at each pixel of the image,
while the classifier, solving an easier problem with more data, can immediately identify relevant areas.






\begin{table*}[t!]
\centering
\fontsize{9.0}{12.0}\selectfont
\begin{tabular}{c|c|l|c|c|c|l}
\hline
Backbone & Add. 9.1K Images & \multicolumn{1}{c|}{Method} & Bkg. Cues & CRF & Pred. Filter & $\!$mIoU \\
\hline\hline
\multirow{8}{*}{VGG} 
  & -- & DeeplabV1~\citep{DeeplabV1:2015Chen}  & -- & \checkmark & -- &  61.8 \\
  \hhline{*{1}{~}*{6}{|-}}
  & \multirow{6}{*}{Image-level} & \cellcolor{myGray}DeeplabV1~\citep{DeeplabV1:2015Chen}  & \cellcolor{myGray}-- & \cellcolor{myGray}\checkmark & \cellcolor{myGray}\checkmark & \cellcolor{myGray}\ub{67.4} \\
  & & WSSL~\citep{WSSL:2015Papandreou}  & --  & \checkmark & -- & 64.6 \\
  & & \cellcolor{myGray}WSSL~\citep{WSSL:2015Papandreou} & \cellcolor{myGray}-- & \cellcolor{myGray}\checkmark & \cellcolor{myGray}\checkmark & \cellcolor{myGray}67.1 \\
  & & \citet{SemiGAN2017Souly}  & -- & -- & -- & 65.8$^*$ \\
  & & MDC~\citep{MDC:2018Wei}  & \checkmark   & \checkmark & -- & 65.7$^*$  \\
  & & FickleNet~\citep{FickleNet:2019Lee} & \checkmark  & \checkmark & -- & 65.8$^*$ \\ 
  \hhline{*{1}{~}*{6}{|-}}
  & Pixel-level & DeeplabV1~\citep{DeeplabV1:2015Chen}  & -- & \checkmark & -- & 69.0  \\
\Xhline{2\arrayrulewidth}
\multirow{4}{*}{VGG-W}
  & -- & DeeplabV1-W~\citep{Dual:2020Luo}  & -- & \checkmark & -- & 69.2 \\
  \hhline{*{1}{~}*{6}{|-}}
  & \multirow{3}{*}{Image-level} & \cellcolor{myGray}DeeplabV1-W~\citep{Dual:2020Luo}  & \cellcolor{myGray}-- & \cellcolor{myGray}\checkmark & \cellcolor{myGray}\checkmark & \cellcolor{myGray}73.8 \\
  & & DualNet~\citep{Dual:2020Luo} & \checkmark  & \checkmark & -- & 73.9 \\
  & & \cellcolor{myGray}DualNet~\citep{Dual:2020Luo} & \cellcolor{myGray}\checkmark & \cellcolor{myGray}\checkmark & \cellcolor{myGray}\checkmark & \cellcolor{myGray}\ub{75.1} \\ \Xhline{2\arrayrulewidth}
\multirow{9}{*}{ResNet}
  & \multirow{2}{*}{--} & DeeplabV3~\citep{DeeplabV3:2017Chen} & -- & \checkmark & -- & 72.4 \\
  & & \citet{semi_context_const2021lai} & -- & -- & -- & 74.5 \\
  \hhline{*{1}{~}*{6}{|-}}
  & \multirow{6}{*}{Image-level} & \cellcolor{myGray}DeeplabV3~\citep{DeeplabV3:2017Chen} & \cellcolor{myGray}-- & \cellcolor{myGray}\checkmark & \cellcolor{myGray}\checkmark & \cellcolor{myGray}75.3 \\ 
  & & \citet{semi_context_const2021lai} & -- & -- & -- & 76.1 \\
  & & CCT~\citep{CCT:2020Ouali} & -- & \checkmark & -- & 74.7 \\
  & & \cellcolor{myGray}CCT~\citep{CCT:2020Ouali} & \cellcolor{myGray}-- & \cellcolor{myGray}\checkmark & \cellcolor{myGray}\checkmark & \cellcolor{myGray}76.0 \\
  & & AdvCAM~\citep{AdvCAM:2021Lee} & -- & \checkmark & -- & 76.1 \\
  & & \cellcolor{myGray}AdvCAM~\citep{AdvCAM:2021Lee} & \cellcolor{myGray}-- & \cellcolor{myGray}\checkmark & \cellcolor{myGray}\checkmark & \cellcolor{myGray}\ub{77.1} \\ 
  \hhline{*{1}{~}*{6}{|-}}
  & Pixel-level & DeeplabV3~\citep{DeeplabV3:2017Chen} & -- & \checkmark &-- & 77.4 \\ 
\Xhline{2\arrayrulewidth}
\multirow{4}{*}{ResNet-W}
  & -- & DeeplabV3-W~\citep{Dual:2020Luo} & -- & \checkmark & -- & 76.2 \\ 
  \hhline{*{1}{~}*{6}{|-}}
  & \multirow{3}{*}{Image-level} & \cellcolor{myGray}DeeplabV3-W~\citep{Dual:2020Luo} & \cellcolor{myGray}-- & \cellcolor{myGray}\checkmark & \cellcolor{myGray}\checkmark & \cellcolor{myGray}\ub{77.5} \\ 
  & & DualNet~\citep{Dual:2020Luo} & \checkmark  & \checkmark & -- & 76.7 \\
  & & \cellcolor{myGray}DualNet~\citep{Dual:2020Luo} & \cellcolor{myGray}\checkmark  & \cellcolor{myGray}\checkmark & \cellcolor{myGray}\checkmark & \cellcolor{myGray}77.3 \\
\Xhline{2\arrayrulewidth}
\multirow{3}{*}{HRNetV2-W48}
  & -- & OCRNet~\citep{OCRNet:2021Strudel} & -- & -- & -- & 74.0 \\ 
  \hhline{*{1}{~}*{6}{|-}}
  & Image-level & \cellcolor{myGray}OCRNet~\citep{OCRNet:2021Strudel}& \cellcolor{myGray}-- & \cellcolor{myGray}-- & \cellcolor{myGray}\checkmark & \cellcolor{myGray}\ub{75.8} \\ 
  \hhline{*{1}{~}*{6}{|-}}
  & Pixel-level & OCRNet~\citep{OCRNet:2021Strudel}& -- & -- & -- & 77.7 \\ 
\Xhline{2\arrayrulewidth}
\end{tabular}
\vspace*{-1.0mm}
\caption{Comparison of the state-of-the-art methods on $1{,}464$ images with pixel labels. The ``Add. 9.1K Images'' column gives which type of supervision is used for the 9.1K additional images (augmented dataset). Numbers marked with $*$ are as reported by the corresponding paper.}
\label{tbl:sota}
\vspace*{-1.0mm}
\end{table*}

\newcolumntype{C}{>{\centering\arraybackslash}p{3.2em}}
\newcolumntype{D}{>{\centering\arraybackslash}p{6.1em}}
\begin{table*}[t!]
\setlength{\tabcolsep}{0.7pt}
\centering
\fontsize{6.5}{9.0}\selectfont
\begin{tabu}{D|CCCCCCCCCCCCCCCCCCCC|C}
\hline
Method & plane & bike & bird & boat & bottle & bus & car & cat & chair & cow & table & dog & horse & mbike & person & plant & sheep & sofa & train & tv & mIoU \\
\hline\hline
DeeplabV1 & 71.1 & 37.1 & 78.5 & 52.7 & 58.3 & 79.4 & 72.0 & 73.2 & 20.6 & 58.0 & 56.1 & 66.5 & 55.9 & 76.1 & 76.4 & 40.6 & 65.4 & 42.8 & 65.2 & 53.3 & 61.4 \\
+ Pred.\ Filter    & \ub{76.2} & \ub{38.9} & \ub{82.2} & \ub{58.2} & \ub{61.2} & \ub{85.1} & \ub{76.8} & \ub{84.4} & \ub{22.6} & \ub{73.0} & \ub{56.2} & \ub{79.3} & \ub{76.2} & \ub{82.4} & \ub{78.2} & \ub{46.0} & \ub{80.4} & \ub{43.6} & \ub{71.5} & \ub{55.2} & \ub{67.6} \\
\hline
DeeplabV3-W & \ub{89.3} & 60.2 & 80.5 & 56.4 & 73.7 & 92.5 & 83.8 & \ub{92.4} & 31.1 & 83.6 & \ub{69.9} & \ub{85.3} & 81.9 & 84.8 & \ub{85.4} & 63.2 & 84.2 & 52.7 & \ub{83.9} & \ub{69.8} & 76.1 \\
+ Pred.\ Filter  & \ub{89.3} & \ub{61.7} & \ub{81.5} & \ub{58.0} & \ub{73.8} & \ub{92.6} & \ub{84.3} & 91.3 & \ub{34.4} & \ub{84.9} & 69.8 & 85.1 & \ub{89.4} & \ub{86.2} & 85.1 & \ub{64.3} & \ub{89.0} & \ub{54.3} & 83.2 & 68.1 & \ub{77.2} \\
\Xhline{2\arrayrulewidth}
\end{tabu}
\vspace*{-1.0mm}
\caption{Evaluation results of DeeplabV1 (VGG-based) and DeeplabV3-W (ResNet-based) models on the test set.}
\vspace*{-4.0mm}
\label{tbl:test}
\end{table*}

\paragraph{Prediction filtering.}
Inspired by this phenomenon, we propose a simple post-processing method, \emph{prediction filtering}.
Given $D_{image}$ with a large number of images, we can train a highly accurate multi-label classifier $g$;
a ResNet50 achieves $99\%$ accuracy and $97.5\%$ average precision on PASCAL VOC.
Hence, we constrain predictions to come from the classifier's predictions instead of ground truth classes,
$\mathcal{\hat K}_i = \{ c : g(x_i)_c > \tau \}$,
where $g(x_i)_{c}$ is the output logit of $g$ for class $c$,
and $\tau$ is a threshold to determine the presence of a class in an image.
We provide full pseudocode in the appendix.




Compared to other SWSSS algorithms, prediction filtering has several advantages.
First, the architecture is simple. It only requires an additional classifier, which can be trained in parallel with the segmentation network; most existing methods require training a classifier first.
Second, it requires only a single additional hyperparameter, the threshold $\tau$, far fewer than required by other SWSSS algorithms.
For instance, MDC~\cite{MDC:2018Wei} requires two thresholds to determine the foreground and background regions, in addition to selecting the number of dilation layers with different rate for each layer.
(We provide a comprehensive comparison of hyperparameter counts in the appendix).
Prediction filtering thus minimizes the requirements on the additional fully-supervised validation set.
Third, it can be independently added to any segmentation algorithm, including existing SWSSS algorithms; we do so in our experiments. 









\paragraph{Effect on performance.}

Prediction filtering helps performance when an incorrect prediction is filtered out and the ``backup'' prediction is correct;
it hurts when a correctly-predicted object is incorrectly filtered.
It can also change an incorrect prediction to a different incorrect prediction;
this can in fact either increase or decrease the mIoU score.

For a non-perfect classifier and reasonable setting of $\tau$, it is conceivable for prediction filtering to hurt segmentation performance -- although we did not observe this in our experiments.
There is always a value of the threshold $\tau$, however, for which prediction filtering at least does not hurt:
just take $\tau \to -\infty$, in which case no predictions are changed.
As $\tau$ increases, it likely (though not certainly) removes incorrect predictions before it begins removing any correct predictions.
In another extreme, for a perfect classifier, prediction filtering approaches oracle filtering;
clearly, oracle filtering may not achieve perfect performance, but it can only help.





\begin{figure*}[t!]
    \centering
    \includegraphics[width=0.95\textwidth]{figures/various_supervision_resnet_6_v4.pdf}
    \caption{Performance of the prediction filtering on various models and levels of supervision (M).}
    \label{fig:various_supervision}
\end{figure*}


\newcolumntype{F}{>{\centering\arraybackslash}p{2.0em}}
\newcolumntype{E}{>{\centering\arraybackslash}p{4.0em}}
\begin{table*}[t!]
\begin{minipage}{.53\textwidth}
\centering
\fontsize{9.0}{12.0}\selectfont
\begin{tabu}{cc|EE|EE}
\hline
\multirow{2}{*}{Filtering} &
\multirow{2}{*}{CRF} &
\multicolumn{2}{c|}{DeeplabV1} &
\multicolumn{2}{c}{DeeplabV3}\\
\cline{3-6}
& & $M$=$500$ & $M$=$1{,}464$ & $M$=$500$ & $M$=$1{,}464$ \\
\hline
\hline
 & & 49.6 & 57.2 & 57.0 & 70.6 \\
 \checkmark & & 61.4 & 64.6 & 64.8 & 74.0  \\
 & \checkmark & 53.9 & 61.8 & 58.4 & 72.4 \\
 \checkmark & \checkmark & \ub{63.6} & \ub{67.4} & \ub{65.9} & \ub{75.3} \\
\Xhline{2\arrayrulewidth}
\end{tabu}
\end{minipage}%
\begin{minipage}{.52\textwidth}
\centering
\fontsize{9.0}{12.0}\selectfont
\begin{tabu}{cc|FFFF}
\hline
\multirow{2}{*}{Filtering} & \multirow{2}{*}{CRF} & \multicolumn{4}{c}{Pixel-level labels ($M$)} \\
\cline{3-6}
 & & 200 & 500 & 800 & 1,464 \\
\hline
\hline
 & & 43.3 & 49.9 & 52.7 & 57.2 \\
 \checkmark & & 46.1 & 54.0 & 57.1 & 62.0\\
 & \checkmark & 46.3 & 53.9 & 56.8 & 61.8 \\
 \checkmark & \checkmark & \ub{48.2} & \ub{56.5} & \ub{59.8} & \ub{64.9} \\
\Xhline{2\arrayrulewidth}
\end{tabu}
\end{minipage}%
\caption{{Left}: Performance of model variants with $\lvert\mathcal{D}_\mathit{pixel}\rvert = M$ and $\lvert\mathcal{D}_\mathit{image}\rvert = 10,582$. {Right}: the same, for a DeeplabV1 baseline, but with the classifier trained only on the same $M$ images in $D_{pixel}$.}
\vspace*{-1.0mm}
\label{tbl:crf_and_partial}
\end{table*}


\section{Experimental Evaluation}
\label{sec:experiment}


\paragraph{Dataset.}

We evaluate prediction filtering on PASCAL VOC 2012 \citep{VOC:Everingham2015} 
which contains $10{,}582$ training, $1{,}449$ validation, and $1{,}456$ test images.
For SWSSS, we follow the training splits of \citet{CCT:2020Ouali}, where $1{,}464$ images are used for $\mathcal{D}_\mathit{pixel}$.
As with previous work, we evaluate segmentation performance by mean Intersection over Union (mIoU),
generally on the validation set.
Test set performance is obtained from the PASCAL VOC evaluation server, without any tricks such as multi-scale or flipping.

\paragraph{Implementation.}
To verify the robustness of our method, we experiment with five semantic segmentation baselines: DeeplabV1 (based on VGG16), DeeplabV3 (based on ResNet-101), their deeper variants with wider receptive fields used by \citet{Dual:2020Luo} which we call DeeplabV1-W and DeeplabV3-W, and a Transformer model called OCRNet~\cite{OCRNet:2021Strudel} (based on HRNetV2-W48~\cite{HRNet2020Wang}).
For prediction filtering, we use a ResNet50 classifier.
We also apply prediction filtering to several existing SWSSS models.
Although CRF post-processing is no longer commonly used in semantic segmentation tasks,
in our experiments it still significantly improves the performance of models trained on a small number of pixel-level labels.
We thus apply CRFs as default except when otherwise specified.








\begin{figure*}[t!]
    \centering
    \begin{subfigure}[b]{0.47\textwidth}
        \includegraphics[width=\textwidth]{figures/orig_resnet.png}
        \caption{Baseline model}
        \label{fig:confusion-nofilter}
    \end{subfigure}
    \hfill
    \begin{subfigure}[b]{0.47\textwidth}
        \includegraphics[width=\textwidth]{figures/updated_resnet.png}
        \caption{With prediction filtering}
        \label{fig:confusion-filtered}
    \end{subfigure}
    \vspace*{-0.8mm}
    \caption{Pixel-level confusion matrices for DeeplabV1 models trained on $1{,}464$ pixel-level labels. Each entry shows the number of pixels (on a logarithmic scale; $0$ values are plotted as if they were $1$) whose true label is given according to the row, and whose predicted label is that in the column. Labels are sorted into rough categories to show block structure.}
     \vspace*{-1.0mm}
    \label{fig:confusion-matrices}
\end{figure*}

\begin{figure*}[h!]
   
    \centering
    \includegraphics[width=0.92\textwidth]{figures/qualitative4.pdf}
    \vspace*{-1.5mm}
    \caption{Qualitative results, the top row of a successful case and bottom a failure
   
    using DeeplabV1 trained on $1{,}464$ pixel-labeled images.
   
   
    }
     \vspace*{-3.0mm}
    \label{fig:qualitative}
\end{figure*}




\paragraph{Comparison with state-of-the-art.}
In \cref{tbl:sota}, we compare the performance of prediction filtering to existing methods when $\lvert\mathcal{D}_\mathit{pixel}\rvert$ is $1{,}464$.
We reproduce results for Deeplab, OCRNet, WSSL, CCT, DualNet, and AdvCAM.\footnote{%
Our AdvCAM result is substantially worse than reported by \citet{AdvCAM:2021Lee}, because we did not apply tricks such as multi-scale and flipping in inference time for fair comparison with the other state-of-the-art methods.}
Among VGG-based methods, DeeplabV1 with prediction filtering outperforms the other methods without using any additional information.
Similarly, filtering also improves models with stronger backbones, though the margin of improvement is less dramatic since the baseline is better.
Prediction filtering can help even when it does not involve adding any new training data:
it significantly helps WSSL, CCT, AdvCAM, and DualNet although they already use weakly-labeled data.
It is worth highlighting that simply adding prediction filtering to the DeeplabV3-W baseline achieves the new highest performance, slightly higher than DualNet (with a ResNet-W backbone) with prediction filtering, both of which are notably better than the previous state-of-the-art (DualNet with a ResNet-W backbone without prediction filtering).
Prediction filtering on top of DualNet also sets the new state-of-the-art for VGG-based models.









\paragraph{Results on the test set.}
We further evaluate prediction filtering on the test set of VOC2012.
In \cref{tbl:test}, we provide the performance of DeeplabV1 and DeeplabV3-W on the test set, as well as with prediction filtering applied.
We can observe that prediction filtering improves the intersection-over-union (IoU) scores for most of the $21$ classes,
leading to significant improvements in terms of mIoU (as on the validation set).









\paragraph{Various levels of supervision.}
\Cref{fig:various_supervision} shows the segmentation performance of DeeplabV1, DualNet with a VGG backbone, and DeeplabV3-W trained on $200$, $500$, $800$ and $1{,}464$ images with pixel labels, with and without prediction filtering.
The blue (bottom) line shows performance of the base model;
orange (middle) shows with prediction filtering;
green (top) is with oracle filtering, which upper-bounds the possibility of improvement of prediction filtering with a better classifier.
For smaller numbers of pixel labels, the performance gain from prediction filtering is drastically larger.
For example, at $800$ images with pixel labels,
DeeplabV1 goes from $56.8 \rightarrow 64.4$,
DeeplabV3-W from $72.3 \rightarrow 74.3$.
The improvements for $200$ pixel labels are  $12.4$, $1.8$, and $4.8$.




\paragraph{Relationship with CRF.}
CRFs adjust the prediction for each pixel by encouraging nearby, similar-colored pixels to have the same label. This usually improves the segmentation performance by refining detailed boundaries and making class predictions across an object more consistent.
The latter role overlaps to some extent with prediction filtering.
If the size of the wrong prediction is large, though, a CRF might expand the area of the wrong prediction, rather than remove it.
\Cref{tbl:crf_and_partial}, as well as the qualitative results to come shortly, shows that the methods complement one another.


\paragraph{Without image-level labels.}
Although image-level labels are easier to obtain than pixel-level labels, annotation effort is still nontrivial.
Our hypothesis about why prediction filtering works, however, is largely that classifiers ``looks at images differently'' than segmentation networks do.
It thus might help even if it does not use any additional data: $\mathcal{D}_\mathit{pixel}$ and $\mathcal{D}_\mathit{image}$ contain the same images.
\cref{tbl:crf_and_partial} (right) shows this is the case.
Even without introducing any actual new information, prediction filtering still improves mIoU significantly.










\paragraph{Changes between classes.}
In \cref{sec:problem}, we showed some qualitative evidence that the segmentation network with low pixel labels tends to be confused between similar classes,
and that prediction filtering can help to compensate it by looking at other parts of the image.
To further demonstrate this, \cref{fig:confusion-matrices} shows the pixel-level confusion matrix for a DeeplabV1 model with CRF before and after prediction filtering.
\cref{fig:confusion-matrices}(a) shows a strong block structure where pixels from one animal class are often confused for another animal class, or vehicles for vehicles.
In \cref{fig:confusion-matrices}(b), prediction filtering has dramatically reduced these types of mistakes.










\paragraph{Qualitative results.}
\Cref{fig:qualitative} shows a success (top) and failure (bottom) for prediction filtering.
At top, an object is mostly incorrectly predicted -- \labelname{bike} as \labelname{motor bike} -- and CRF only makes this worse.
The classifier in filtering, however, correctly identifies there is no \labelname{motor bike}, and the model's ``second choice'' prediction is largely correct.
In the failure case, an object (\labelname{table}) is mostly occluded;
the segmentation model still identifies it, but the classifier misses it.

\paragraph{Additional ablation studies.}
The appendix provides further experiments, including alternative approaches to prediction filtering, performance with model variants, and a runtime comparison between prediction filtering and CRFs. 



\section{Conclusion}
\label{sec:conclusion}
Most existing semi-weakly supervised semantic segmentation algorithms exploit the pseudo-labels extracted from a classifier.
Doing so, however, requires a complicated architecture and extensive hyperparameter tuning on fully-supervised validation sets.
We propose \textit{prediction filtering}, a simple post-processing method that only considers the classes for segmentation a classifier is confident are present.
Our experiments demonstrated adding this method to baselines achieves the new highest performance on PASCAL VOC in SWSSS regimes, and adding it to existing SWSSS algorithms uniformly improves their performance.
We expect prediction filtering can become a standard post-processing method for segmentation, along with CRFs, at least when a relatively large number of weakly-labeled images are available and the portion of class labels present in most images is low.

\bibliographystyle{named}
