
\section{The Elsevier article class}

\paragraph{Installation} If the document class \emph{elsarticle} is not available on your computer, you can download and install the system package \emph{texlive-publishers} (Linux) or install the \LaTeX\ package \emph{elsarticle} using the package manager of your \TeX\ installation, which is typically \TeX\ Live or Mik\TeX.

\paragraph{Usage} Once the package is properly installed, you can use the document class \emph{elsarticle} to create a manuscript. Please make sure that your manuscript follows the guidelines in the Guide for Authors of the relevant journal. It is not necessary to typeset your manuscript in exactly the same way as an article, unless you are submitting to a camera-ready copy (CRC) journal.

\paragraph{Functionality} The Elsevier article class is based on the standard article class and supports almost all of the functionality of that class. In addition, it features commands and options to format the
\begin{itemize}
\item document style
\item baselineskip
\item front matter
\item keywords and MSC codes
\item theorems, definitions and proofs
\item lables of enumerations
\item citation style and labeling.
\end{itemize}

\section{Front matter}

The author names and affiliations could be formatted in two ways:
\begin{enumerate}[(1)]
\item Group the authors per affiliation.
\item Use footnotes to indicate the affiliations.
\end{enumerate}
See the front matter of this document for examples. You are recommended to conform your choice to the journal you are submitting to.

\section{Bibliography styles}

There are various bibliography styles available. You can select the style of your choice in the preamble of this document. These styles are Elsevier styles based on standard styles like Harvard and Vancouver. Please use Bib\TeX\ to generate your bibliography and include DOIs whenever available.

Here are two sample references: \cite{Feynman1963118,Dirac1953888}.

\section*{References}


\section{Introduction}

	Most machine learning applications
	are hampered by need
	to collect large annotated datasets.
	Learning with incomplete supervision
	\cite{kolesnikov19cvpr,lee19cvpr}
	presents a great opportunity
	to speed up the development cycle
	and enable rapid adaptation
	to new environments.
	Semi-supervised learning
	\cite{tarvainen17nips,miyato19pami,xie19arxiv_uda}
	is especially relevant
	in the dense prediction context
	\cite{souly17iccv,hung18bmvc,mittal19pami}
	since pixel-level labels are very expensive,
	whereas unlabeled images are easily obtained.

	Dense prediction models produce
	a prediction for each input pixel.
	They typically operate on high input resolutions
	in order to be able to detect small objects
	\cite{cordts15cvprw,neuhold17iccv,maggiori17igarss}.
	Semantic segmentation predictions
	are dense $K$-class categorical distributions.
	Competitive semantic segmentation performance
	requires learning on large batches and large crops.
	This typically entails
	a large memory footprint during training,
	which constrains model capacity \cite{bulo18cvpr}.
	%
	Many semi-supervised algorithms introduce additional components
	to the training setup.
	For instance, training
	with surrogate classes \cite{dosovitskiy14nips}
	implies\noarticle{ an} infeasible size of the logit tensor.
	Ladder networks \cite{rasmus15nips}
	require an additional decoder and a twin encoder.
	GAN-based approaches require an additional
	generator \cite{salimans16nips,souly17iccv}
	or discriminator \cite{hung18bmvc,tsai18cvpr}.
	Some approaches use multiple model instances
	\cite{sajjadi16nips,qiao18eccv,tarvainen17nips,bortsova19miccai}
	or accumulated predictions across the dataset \cite{laine17iclr}.
	Such designs are less appropriate for dense prediction
	since they constrain model capacity.

	We study semi-supervised approaches
	\cite{sajjadi16nips,zheng16cvpr,laine17iclr,tarvainen17nips,xie19arxiv_uda}
	that require consistent predictions
	over input perturbations.
	We consider consistency
	in which input perturbations affect
	only one of two model instances,
	while the gradient is not propagated
	towards the model instance which
	operates on the clean input \cite{miyato19pami,xie19arxiv_uda}.
	For brevity, we refer to the two model instances
	as the \textit{perturbed} branch and the \textit{clean} branch.
	If the gradient is not computed in a branch,
	we refer to it as the \textit{teacher},
	and otherwise as the \textit{student}.
	Hence, we refer to the considered approach as
	one-way consistency with clean teacher.

	Let $\vec x$ be the input, $T$ an input perturbation,
	$h_{\vec\theta}$ the student,
	and $h_{\vec\theta'}$ the teacher,
	where $\vec\theta'$ denotes
	a frozen copy of the student parameters $\vec\theta$.
	Then, one-way consistency with
	clean teacher can be formulated as a divergence $D$
	between the two predictions:
	\begin{equation}
	L^\text{ct}_{\vec\theta}
	(\vec x) =
	D(h_{\vec{\theta}'}(\vec x),
	h_{\vec{\theta}}(T(\vec x))) \;.
	\label{eq:cons1w}
	\end{equation}
	Equation~\eqref{eq:cons1w} assumes that the output should be
	invariant to the perturbations of interest.
	However, that assumption is not suitable
	for many kinds of perturbations
	in the dense prediction case.
	For instance, rotated input should give rise to rotated predictions.
	Hence, we shall propose an extended formulation
	in order to accommodate a fairly general class
	of perturbations $T$
	that the correct model $f$ should be
	equivariant to:
	$f(T(x)) = T(f(x))$ \cite{lenc2018ijcv}.


	%









































	We argue that the clean teacher approach should be preferred
	in case of perturbations that are too strong
	for standard data augmentation. 
	In this setting \cite{french20bmvc}, perturbed inputs
	typically give rise to less reliable predictions
	than their clean counterparts.
	Figure~\ref{fig:moons-cons-variants} illustrates the advantage
	of the clean teacher approach in comparison with
	other kinds of consistency on the Two moons dataset.
	The clean student experiment (\protect\subref{fig:moons-1w-cs}) shows
	that many blue datapoints
	get classified into the red class
	due to influence of perturbed red datapoints.
	This aberration does not occur
	when the teacher inputs are clean (\protect\subref{fig:moons-1w-ct})
	since there we only use clean datapoints
	as consistency targets.
	Two-way consistency \cite{laine17iclr} (\protect\subref{fig:moons-2w-p1})
	can be viewed as a superposition
	of the two one-way approaches
	and works worse than one-way consistency
	with clean teacher.
	All experiments use KL divergence for $D$.

	\begin{figure}[htb]
		\centering
		\begin{subfigure}[t]{0.4\textwidth}
			\centering
			\includegraphics[width=1\textwidth]{
				figs/semi_toy/semi_toy_kl_detach_clean_detach_pert_20000_2.pdf}
			\caption{no consistency loss}
			\label{fig:moons-sup}
		\end{subfigure}
		\begin{subfigure}[t]{0.4\textwidth}
			\centering
			\includegraphics[width=1\textwidth]{
				figs/semi_toy/semi_toy_kl_detach_pert_20000_2.pdf}
			\caption{one-way, clean student:
				$D(h_{\vec{\theta}}(\vec x),h_{\vec{\theta}'}(T(\vec x)))$}
			\label{fig:moons-1w-cs}
		\end{subfigure}

		\begin{subfigure}[t]{0.4\textwidth}
			\centering
			\includegraphics[width=1\textwidth]{
				figs/semi_toy/semi_toy_kl_detach_clean_20000_2.pdf}
			\caption{one-way, clean teacher:
				$D(h_{\vec{\theta}'}(\vec x),h_{\vec{\theta}}(T(\vec x)))$}
			\label{fig:moons-1w-ct}
		\end{subfigure}
		\begin{subfigure}[t]{0.4\textwidth}
			\centering
			\includegraphics[width=1\columnwidth]{
				figs/semi_toy/semi_toy_kl_20000_2.pdf}
			\caption{two-way, one input clean:
				$D(h_{\vec{\theta}}(\vec x), h_{\vec{\theta}}(T(\vec x)))$}
			\label{fig:moons-2w-p1}
		\end{subfigure}
		\caption{
			A toy 2D classification problem
			with 6 labeled (red, blue)
			and many unlabeled datapoints (white).
			All setups involve 20\,000 epochs
			of semi-supervised training
			with cross-entropy and default Adam
			optimization hyperparameters.
			The consistency loss
			was set to none (\protect\subref{fig:moons-sup}),
			one-way with clean student (\protect\subref{fig:moons-1w-cs}),
			one-way with clean teacher (\protect\subref{fig:moons-1w-ct}),
			two-way with one input clean (\protect\subref{fig:moons-2w-p1}).
			One-way consistency with clean teacher
			outperforms other formulations.
		}
		\label{fig:moons-cons-variants}
	\end{figure}

	One-way consistency
	is especially advantageous
	in the dense prediction context
	since it requires caching latent activations
	in only one model instance.
	This allows training of better models
	in many practical cases where
	the model capacity is limited by GPU memory \cite{bulo18cvpr,kreso20tits}.
	In comparison with two-way consistency
	\cite{laine17iclr,bortsova19miccai},
	the proposed approach both improves generalization
	and halves the training memory footprint.

	This paper is an extended version of our
	preliminary conference report \cite{grubisic21mva}.
	It exposes the elements of our method
	in much more detail and complements
	them with several new experiments.
	In particular, we show a detailed analysis
	of a large-scale experiment which compares
	the contribution of coarse labels dataset
	with semi-supervised learning on unlabeled images.

	The consolidated paper proposes
	a simple and effective method
	for semi-supervised semantic segmentation.
	One-way consistency with clean teacher
	\cite{miyato19pami,xie20nips,french20bmvc}
	outperforms the two-way formulation
	in our validation experiments.
	In addition, it retains the memory footprint
	of supervised training
	because teacher activations
	do not have to be cached
	for gradient computation.
	Experiments with the standard
	convolutional architecture \cite{chen18pami}
	reveal that our
	photometric and geometric perturbations
	lead to competitive generalization performance
	and outperform their counterpart
	from a recent related work
	\cite{french20bmvc}.
	We observe a similar advantage
	in experiments with a recent
	efficient architecture \cite{orsic19cvpr},
	which offers a similar performance
	while requiring an order of magnitude
	less computation.
	To our knowledge, this is the first account of evaluation of
	semi-supervised algorithms for dense prediction
	with a model capable of real-time inference.
	This contributes to the goals
	of Green AI \cite{schwartz20cacm}
	by enabling competitive research with less environmental damage.


	\section{Related work}

	\subsection{Data augmentation}

	The typical goal of data augmentation is to regularize learning
	by artificially extending the training dataset.
	MixUp \cite{zhang18iclr} constructs new training examples
	by blending two images and their one-hot labels.
	AugMix \cite{hendrycks20iclr} applies a composition
	of simple transformations to original images
	and proposes a consistency loss
	based on three-way Jensen-Shannon divergence
	between predictions in two augmented and one original image.
	AutoAugment \cite{cubuk19cvpr} optimizes the augmentation policy with reinforcement learning.
	RandAugment \cite{cubuk19arxiv} reduces
	the search space of AutoAugment
	through uniform sampling and choosing a constant magnitude
	for all transformations.
	TrivialAugment \cite{muller2021iccv} demonstrates that randomly choosing a single transformation for each input can work as well as a more complex augmentation policy.



	\subsection{Dense prediction}

	Image classification models usually achieve efficiency,
	spatial invariance, and integration of contextual information
	by gradual downsampling of representations
	and use of global spatial pooling operations.
	However, dense prediction also requires location accuracy,
	while contextual information is still important.
	Hence, dense prediction models often search for a trade-off between efficiency and quality of high-resolution features.
	Some common designs use a classification backbone as a feature encoder and attach a decoder that increases spatial dimensions.
	Many approaches seek to enhance contextual information, starting with FCN-8s \cite{long15cvpr}.
	UNet \cite{ronneberger15unet} improves spatial details by directly using earlier representations of the encoder as inputs to corresponding representations in a symmetric decoder.
	Semantic segmentation training gains much from ImageNet pre-trained encoders \cite{kreso20tits}. HRNet \cite{sun2019cvpr} maintains the full resolution throughout the whole model and incrementally introduces parallel lower-resolution branches that exchange information between stages.
	Further work increases efficiency with lighter decoders \cite{kreso20tits,zhao18eccv}.
	Some models use context aggregation modules such as spatial pyramid pooling \cite{zhao17pspnet} and multi-scale inference \cite{zhao18eccv}.
	DeepLab \cite{chen2018deeplab} uses dilated convolutions to increase the receptive field and improves spatial details through CRF postprocessing.





	\subsection{Semi-supervised learning}




	Semi-supervised methods often rely on some of
	the following assumptions about the data distribution \cite{chappelle10mit}: (1)
	similar inputs in high density regions correspond to similar outputs (smoothness assumption), (2)
	inputs form clusters separated by low-density regions and inputs within clusters are likely to correspond to similar outputs (cluster assumption), and (3) the data roughly lies on a lower-dimensional manifold (manifold assumption).
	%
	Semi-supervised methods devise various inductive biases that exploit such regularities for learning from unlabeled data.

	Entropy minimization \cite{grandvalet05nips},
	and pseudo-labeling (or self-training) \cite{yarowsky95hlt,mcclosky06aaai,lee13icmlw}
	encourage high confidence in unlabeled inputs.
	Such designs push decision boundaries
	towards low-density regions,
	under assumptions of clustered data
	and prediction smoothness.
	They usually do not achieve
	competitive performance on their own \cite{oliver18nips},
	but can be effective in conjunction with other approaches \cite{xie19arxiv_noisy,miyato19pami}.


	Many approaches exploit the smoothness assumption
	by enforcing prediction consistency
	across different versions of the same input
	or different model instances.
	Exemplar training \cite{dosovitskiy14nips}
	associates patches with their original images
	(each image is a separate surrogate class).
	Temporal ensembling \cite{laine17iclr}
	enforces per-datapoint consistency
	between the current prediction
	and a moving average of past predictions.

	Mean Teacher \cite{tarvainen17nips} encourages consistency between two model instances, the teacher and the student.
	Instead of averaging predictions,
	the teacher uses a moving average
	of the student parameters.
	A loss term encourages the consistency
	between the student and the teacher.

	Clusterization of latent representations
	can be promoted by considering walks
	walks which start in a labeled example,
	pass over an unlabeled example, and end in another labeled one \cite{hausser17cvpr}.
	Their loss penalizes such walks
	between different labels.

	MixMatch \cite{berthelot19nips} encourages consistency
	between predictions in different
	MixUp perturbations of the same input.
	The average prediction is used as a pseudo-label
	for all variants of the input.
	Deep co-training \cite{qiao18eccv} produces complementary models by encouraging them to be consistent while each is being given adversarial examples of other models as training inputs.

	Consistency losses
	may encourage trivial solutions,
	where all data give rise to the same model output.
	This is not much of a problem in semi-supervised learning
	since there the trivial solution is inhibited
	through the supervised objective.
	Interestingly, recent work \cite{chen2020simsiam,tian2021arxiv} 
	shows that a variant of simple one-way consistency
	evades trivial solutions even
	in the context of self-supervised
	representation learning.
























	Virtual adversarial training (VAT) \cite{miyato19pami}
	encourages one-way consistency
	between predictions in original datapoints
	and their worst-case perturbations.
	These perturbations are recovered
	by maximizing a quadratic approximation of the prediction divergence
	in a small $\mathrm{L}^2$ ball around the input.
	Better performance is often obtained by encouraging low-entropy predictions \cite{grandvalet05nips}.
	Unsupervised data augmentation (UDA) \cite{xie19arxiv_uda} alo
	uses a one-way consistency loss.
	However, instead of adversarial additive perturbations,
	they use random augmentations generated by RandAugment.
	Different from all previous approaches,
	we explore one-way consistency
	in the dense prediction context,
	propose a novel perturbation model,
	and explain advantages of perturbing only the student
	with respect to other forms of consistency.

	\subsection{Semi-supervised semantic segmentation}

	In the classic semi-supervised GAN (SGAN) setup,
	the classifier also acts as a discriminator
	which distinguishes between real data
	(both labeled and unlabeled)
	and fake data produced by the generator \cite{salimans16nips}.
	This approach has been adapted for dense prediction
	by expressing the discriminator as a segmentation network
	that produces dense C+1-way logits \cite{souly17iccv}.
	KE-GAN \cite{qi19cvpr} additionally enforces
	semantic consistency of neighbouring predictions
	by leveraging label-similarity
	recovered from a large
	text corpus (MIT ConceptNet).
	A semantic segmentation model can also
	be trained as a GAN generator (AdvSemSeg) \cite{hung18bmvc}.
	In this setup,
	the discriminator guesses whether
	its input is ground truth or generated
	by the segmentation network.
	It is also used to choose better
	predictions for use as pseudo-labels
	for semi-supervised training.
	s4GAN + MLMT \cite{mittal19pami}
	additionally post-processes the recovered dense predictions
	by emphasizing classes identified
	by an image-wide classifier trained with Mean Teacher
	\cite{tarvainen17nips}.
	The authors note that
	the image-wide classification
	component is not appropriate for datasets such as Cityscapes,
	where almost all images contain a large number of classes.
	\delete{Universal semantic segmentation \cite{kalluri19iccv}
		pulls features from unlabeled images towards centroids
		obtained by training on multiple datasets.}

	A recent approach \cite{ouali20cvpr}
	enforces consistency between outputs of redundant decoders
	with noisy intermediate representations.
	Other recent work studies pseudo-labeling successfully
	in the dense prediction context
	\cite{zhu20arxiv,chen20arxiv,mendel20eccvf}.
	Zhu et al. \cite{zhu20arxiv} observe advantages in
	hard pseudo-labels.




	A recent approach \cite{bortsova19miccai}
	proposes a two-way consistency loss,
	which is related to the $\mathrm{\Pi}$ model
	\cite{laine17iclr},
	and perturbs both inputs with geometric warps.
	However, we show that
	perturbing only the student branch
	generalizes better
	and has a smaller training footprint.
	A concurrent approach \cite{lai2021cac} successfully applies
	an altered contrastive loss \cite{oord18cpc,he20moco}
	between two branches
	which receive overlapping crops.
	They propose pixel-dependent gradient stopping --
	the pixel with higher confidence is treated as the teacher.
	Mean Teacher consistency
	with CutMix perturbations
	achieves state-of-the-art performance
	on half-resolution Cityscapes  \cite{french20bmvc}.
	Different than most presented approaches
	and similar to \cite{zhu20arxiv,chen20arxiv, french20bmvc},
	our method does not increase
	the training footprint \cite{bulo18cvpr}.
	In comparison with \cite{zhu20arxiv,chen20arxiv},
	our teacher is updated in each training step,
	which eliminates the need
	for multiple training episodes.
	In comparison with \cite{french20bmvc},
	we use a perturbation model
	which results in better generalization.
	None of the previous approaches
	addresses semi-supervised training
	of efficient dense prediction models
	\cite{zhao18eccv,chao19iccv,orsic19cvpr,nie20accv}.


	\section{One-way consistency for dense prediction}

	We formulate dense consistency
	as a mean pixel-wise divergence
	between corresponding predictions
	in the clean image and its perturbed version.
	We perturb images by a composition of
	photometric and geometric transformations.
	Photometric transformations do not disturb
	the spatial layout of the input image.
	Geometric transformations affect
	the spatial layout of the input image,
	but the same kind of disturbance
	is expected at the model output.
	Ideally, our models should exhibit
	invariance to photometric transformations
	and equivariance \cite{lenc2018ijcv} to the geometric ones.

	\subsection{Notation}

	We typeset vectors and arrays in bold, sets in blackboard bold, and we underline random variables.
	$\P[\rvar y|\rvec x=\vec x]$ denotes the distribution
	of a random variable $\rvar y |\vec x$,
	while $\P(y \vert \vec x)$ is a shorthand
	for the probability $\P(\rvar y=y \vert \rvec x=\vec x)$.
	We use Python-like indexing notation.

	We denote the labeled dataset as $\set D_\text{l}$,
	and the unlabeled dataset as $\set D_\text{u}$.
	We consider input images
	$\vec{x}\in [0,1]^{H\times W\times 3}$ and dense labels
	$\vec{y}\in \cbr{1\mkern1.5mu{..}\mkern1.5mu C}^{H\times W}$.
	A model instance
	maps an image to per-pixel class probabilities:
	$h_{\vec{\theta}}(\vec{x})_\ind{i,j,c} = \P(\rvec{y}_\ind{i,j}=c \vert \vec{x}, \vec{\theta})$.
	For convenience, we identify output vectors
	of class probabilities with distributions:
	$h_{\vec{\theta}}(\vec{x})_\ind{i,j} \equiv
	\P[\rvec{y}_\ind{i,j} \vert \vec{x}, \vec{\theta}]$.

	\subsection{Dense one-way consistency}

	We adapt one-way consistency 
	\cite{miyato19pami,xie19arxiv_uda}
	for dense prediction under
	our perturbation model
	$T_{\vec\tau}=
	T^\text{G}_{\vec\gamma} \circ
	T^\text{P}_{\vec\varphi}$, which is a composition
	of a geometric warp $T^\text{G}_{\vec\gamma}$
	and a per-pixel photometric perturbation
	$T^\text{P}_{\vec\varphi}$
	with parameters $\vec\tau=(\vec\gamma,\vec\varphi)$.
	$T^\text{G}_{\vec\gamma}$ displaces pixels
	with respect to a dense deformation field.




	We apply the same geometric warp
	to the student input
	and the teacher output.
	Figure~\ref{fig:1w-ct-graph} illustrates
	the computational graph of
	the resulting dense consistency loss.
	In simple one-way consistency,
	the teacher parameters $\vec\theta'$
	are a frozen copy of
	the student parameter $\vec\theta$.
	In Mean Teacher, $\vec\theta'$
	is a moving average of $\vec\theta$.
	In simple two-way consistency,
	both branches use the same $\vec\theta$
	and are subject to gradient propagation.


	\begin{figure}[htb]
		\centering
		\begin{tikzpicture}[scale=1, every node/.style={scale=1}, x=5em, y=5em]
		\draw (0,0.5) node[nrect] (x) [] {$\vec x$};
		\draw (1,1) node[nrect] (y) [] {$h_{\vec\theta'}$};
		\draw (2,1) node[nrect] (tp) [] {$T^\text{G}_{\vec\gamma}$};
		\draw (1,0) node[nrect] (xp) [] {$T_{\vec\tau}$};
		\draw (2,0) node[nrect, themeblue, fill=white!95!themeblue] (yp) [] {$h_{\vec\theta}$};
		\draw (3,0.5) node[nrect, themeblue, fill=white!95!themeblue] (D) [] {$D$};
	
	

		\node[] (imx) [left=0 of x] {\includegraphics[width=4.9em]{figs/oneway/0000_x.png}};
		\node (imy) [above=0 of y] {\includegraphics[width=4.9em]{figs/oneway/0000_out.png}};
		\node (imtp) [above=0 of tp] {\includegraphics[width=4.9em]{figs/oneway/0000_out_po.png}};
		\node (imxp) [below=0 of xp] {\includegraphics[width=4.9em]{figs/oneway/0000_x_p.png}};
		\node (imyp) [below=0 of yp] {\includegraphics[width=4.9em]{figs/oneway/0000_out_p.png}};
		\node (imerr) [right=0 of D] {\includegraphics[width=4.9em]{figs/oneway/0000_errors.png}};

		\path (x) edge [dedge] node {} (y);
		\path (y) edge [dedge] node {} (tp);
		\path (x) edge [dedge] node {} (xp);
		\path (xp) edge [dedge] node {} (yp);
		\path (tp) edge [dedge] node {} (D);
		\path (yp) edge [dedge, themeblue] node {} (D);
		\end{tikzpicture}
		\caption{Unsupervised dense prediction loss based on one-way consistency with clean teacher.
			Top branch: the input image is fed to the teacher $h_{\vec\theta'}$. The resulting predictions are perturbed with geometric perturbations $T^\text{G}_{\vec\gamma}$.
			Bottom branch: the input image is perturbed with geometric and photometric perturbations and fed to the student $h_{\vec\theta}$. The loss $D$ corresponds to the average pixel-wise KL divergence between the two branches. Gradients are computed only in the blue part of the graph.
			(Best viewed in color.)
		}
		\label{fig:1w-ct-graph}
	\end{figure}

	A general semi-supervised training criterion
	$L(\vec\theta ; \set D_\text{l}, \set D_\text{u})$
	can be expressed as a weighted sum of
	a supervised term $L_\text{s}$
	and an unsupervised consistency term $L_\text{c}$:
	\begin{equation}
	L(\vec\theta ; \set D_\text{l}, \set D_\text{u}) =
	\E_{(\vec x,\vec y)\in \set D_\text{l}}
	L_\text{s}(\vec\theta ; \vec x, \vec y)
	+
	\alpha
	\E_{\vec x\in \set D_\text{u}} \E_{\rvec\tau}
	L_\text{c}(\vec\theta ; \vec{x}, \vec\tau)
	\;.
	\label{eq:semisup-error}
	\end{equation}
	In our experiments, $L_\text{s}$ is the usual
	mean per-pixel cross entropy
	with $\mathrm{L}^2$ regularization.
	We stochastically estimate
	the expectation over perturbation parameters $\rvec\tau$
	with one sample per training step.

	We formulate the unsupervised term $L_\text{c}$
	at pixel $(i,j)$
	as a one-way divergence $D$ between
	the prediction in the perturbed image
	and its interpolated correspondence

	in the clean image.
	The proposed loss encourages the trained model
	to be equivariant with respect to $T^\text{G}_{\vec\gamma}$ and
	invariant with respect to $T^\text{P}_{\vec\varphi}$:
	\begin{align}
	L_{\text c}^{ij}(\vec\theta; \vec x, \vec\tau) =
	D(
	T^\text{G}_{\vec{\gamma}}(
	h_{\vec\theta'}(\vec x))_\ind{i,j},
	h_{\vec\theta}(
	(T^\text{G}_{\vec\gamma} \circ T^\text{P}_{\vec\varphi})

	(\vec x))_\ind{i,j}
	)











	\;.
	\label{eq:1w-ct-objective}
	\end{align}
	We use a validity mask $\vec v^{\vec\gamma}\in\cbr{0,1}^{H\times W}$,
	$\vec v^{\vec\gamma}_\ind{i,j} =
	\enbbracket{
		T^\text{G}_{\vec{\gamma}}(\cvec 1_{H\times W})_\ind{i,j} = 1
	}$ to ensure that the loss is unaffected by padding,
	which may be sampled from outside of $\sbr{1,H}\times\sbr{1,W}$.
	A vector produced by
	$T^\text{G}_{\vec{\gamma}}
	(h_{\vec\theta}(\vec x))_\ind{i,j}$
	represents a valid distribution
	wherever $\vec v^{\vec\gamma}_\ind{i,j}=1$.
	Finally, we express the consistency term $L_\text{c}$
	as a mean contribution over all pixels:
	\begin{equation}
	L_\text{c}(\vec\theta; \vec x, \vec\tau)
	=
	\frac{1}{\sum(\vec v^{\vec\gamma})}
	\sum_{i,j} \vec v^{\vec\gamma}_\ind{i,j}
	L_{\text{c}}^{i, j}
	(\vec\theta ; \vec x, \vec\tau)
	\;
	.







	\label{eq:cons-loss}
	\end{equation}

	Recall that the gradient is not computed
	with respect to $\vec\theta'$.
	Consequently, $L_\text{c}$
	allows gradient propagation
	only towards the perturbed image.
	We refer to training with loss~\eqref{eq:1w-ct-objective}
	as one-way consistency with clean teacher (and perturbed student).
	Such formulation provides two distinct advantages
	with respect to other kinds of consistency.
	First, predictions in perturbed images are
	pulled towards predictions in clean images.
	This improves generalization
	when the perturbations are stronger
	than data augmentations used in $L_\text{s}$
	(cf. Figures~\ref{fig:1w_motivation} and \ref{fig:moons-cons-variants}).
	Second, we do not have to cache
	teacher activations during training
	since the gradients propagate
	only towards the student branch.
	Hence, the proposed semi-supervised objective
	does not constrain model complexity
	with respect to the supervised baseline.











	We use KL divergence as a principled choice
	for $\mathit{D}$:
	\begin{equation}
	\mathit{D}(\rvec y, \tilde{\rvec y})
	= \E_{\rvec y}
	\ln\frac{\P(\rvec y=\vec y)}
	{\P(\tilde{\rvec y}=\vec y)}
	= \H_{\tilde{\rvec y}}(\rvec y) - \H(\rvec y) \;.
	\label{eq:kldiv}
	\end{equation}
	Note that the entropy term $-\H(\rvec y)$
	has no effect on parameter updates
	since the gradients are not propagated through $\vec\theta'$.
	Hence, one-way consistency does not encourage
	increasing entropy of model predictions in clean images.
	Several researchers have observed improvement
	after adding an additional entropy minimization term \cite{grandvalet05nips}
	to the consistency loss \cite{miyato19pami,xie19arxiv_uda}.
	This practice did not prove beneficial in our initial experiments.

	Note that two-way consistency \cite{laine17iclr,bortsova19miccai}
	would be obtained by replacing
	$\vec\theta'$ with $\vec\theta$.
	It would require caching latent activations
	for both model instances.
	As a consequence, we would have to endure
	a two-fold increase of the training footprint
	with respect to the supervised baseline.
	This would be undesirable due to constraining
	feasible capacity of the deployed models
	\cite{pleiss17arxiv,bulo18cvpr}.

	We argue that consistency with clean teacher
	generalizes better than consistency with clean student
	since strong perturbations
	may push inputs
	beyond the natural manifold and
	spoil predictions (cf. Figure~\ref{fig:moons-cons-variants}).
	We also observe that perturbing both branches sometimes results
	in learning to map all perturbed pixels to similar
	arbitrary predictions (e.g. always the same class) \cite{grill2020byol}.
	Figure~\ref{fig:1w_motivation} illustrated that
	consistency training has the best chance to succeed
	if the teacher is applied to the clean image,
	and the student learns on the perturbed image.

	\begin{figure}[thb]
		\centering
	
	
		\includegraphics[width=0.8\textwidth]{figs/cons1w_v2}
		\caption{
			Two variants of one-way consistency training
			on one clean image (left)
			and its perturbed version (right).
			The proposed clean-teacher formulation
			trains in the perturbed pixels ($\mathrm{A_p}$)
			according to the corresponding predictions
			in the clean image ($\mathrm{A_c}$).
			The reverse formulation
			(training in $\mathrm{B_c}$ according to the prediction in $\mathrm{B_p}$)
			leads to worse performance
			since strongly perturbed images
			often give rise to less accurate predictions.
		}
		\label{fig:1w_motivation}
	\end{figure}


	\subsection{Photometric component of the proposed perturbation model}
	\label{subsec:photometric}

	We express pixel-level photometric transformations
	as a composition of five simple perturbations
	with five image-wide parameters
	$\vec\varphi = (b,s,h,c,\vec\pi)$.
	These simple perturbations
	are applied in each pixel
	in the following order:
	(1) brightness is shifted by adding $b$ to all channels,
	(2) saturation is multiplied with $s$,
	(3) hue is shifted by addition with $h$,
	(4) contrast is modulated
	by multiplying all channels with $c$, and
	(5) RGB channels are randomly perturbed
	according to $\vec\pi$.
	The resulting compound transformation
	$T^\text{P}_{\vec\varphi}$
	is independently applied
	to all image pixels.

	Our training procedure randomly picks image-wide parameters $\vec\varphi$
	for each unlabeled image.
	We sample the four parameters as follows:
	$b\sim\mathcal U(-0.25, 0.25)$,
	$s\sim\mathcal U(0.25, 2)$,
	$h\sim\mathcal U(-36^\circ, 36^\circ)$,
	$c\sim\mathcal U(0.25, 2)$.


	\subsection{Geometric component of the proposed perturbation model}
	\label{subsec:tps}

	We formulate a fairly general class
	of parametric geometric transformations
	by leveraging thin plate splines (TPS) \cite{duchon77ctfsv,bookstein89pami}.
	We consider the 2-D TPS warp $f\colon \cset R^2\to\cset R^2$
	which maps each image coordinate pair $\vec q$
	to the relative 2D displacement
	of its correspondence $\vec q'$:
	\begin{equation}
	f(\vec q) = \vec q' - \vec q \;.
	\end{equation}
	TPS warps minimize the bending energy (curvature)
	$\int_{\dom(f)} \norm{\pd[2]{f(\vec q)}{\vec q}}_\text{F}^2 \dif \vec q$
	for a given a set of control points
	and their displacements
	$\cbr{(\vec c_i, \vec d_i)\colon i=1\mkern1.5mu{..}\mkern1.5mu n} \subset \cset R^2\times\cset R^2$.
	In simple words, a TPS warp produces
	a smooth deformation field which
	optimally satisfies all constraints
	$f(\vec c_i) = \vec d_i$.
	In particular, a 2-D TPS warp has the form
	\begin{equation} \label{eq:tps_2d}
	f(\vec q)

	=

	\vec A \cdot
	\begin{bmatrix} 1 \\ \vec q \end{bmatrix}
	+

	\vec W \cdot
	\begin{bmatrix}\phi(\norm{\vec q - \vec c_i})\end{bmatrix}_{i=1\mkern1.5mu{..}\mkern1.5mu n}^\transpose \;.
	\end{equation}
	In the above equation, $\vec q$ denotes
	a 2D coordinate vector to be transformed,
	$\vec A$ is a $2\times 3$ affine transformation matrix,
	$\vec W$ is a $2\times n$
	control point coefficient matrix,
	and $\phi(r)=r^2\ln(r)$.
	Such a 2-D TPS warp is equivariant
	with respect to rotation and translation \cite{bookstein89pami}.
	That is, $f(T(\vec q)) = T(f(\vec q))$
	for every composition of
	rotation and translation $T$.

	TPS parameters $\vec A$ and $\vec W$ can be determined
	as a solution of a standard linear system
	which enforces deformation constraints
	$(\vec c_i, \vec d_i)$,
	and square-integrability of second derivatives of $f$.
	When we determine $\vec A$ and $\vec W$,
	we can easily transform entire images.

	We first consider images
	as continuous domain functions
	and later return to images as arrays from $[0,1]^{H\times W\times 3}$.
	Let $\dom(I)=\intcc{0,W}\times\intcc{0,H}$ denote the domain and $I \colon \dom(I)\to\intcc{0,1}^3$ the original image of size $(W, H)$.
	Then the transformed image $I'$ can be expressed as
	\begin{equation}
	I'(\vec q+f(\vec q)) = \begin{cases} I(\vec q), & \vec q \in \dom(I), \\ \cvec{0}, & \text{otherwise.} \end{cases}
	\end{equation}

	The resulting formulation is known as
	forward warping \cite{szeliski2010computer}
	and is tricky to implement.
	We therefore prefer to recover
	the reverse transformation $\tilde f$,
	which can be done by
	by replacing each control point $\vec c_i$
	with $\vec c_i'=\vec c_i+\vec d_i$.
	Then the transformed image is:
	\begin{equation}
	\tilde{I}(\vec q') =
	\begin{cases}
	I(\vec q' - \tilde f(\vec q')),
	& \vec q' - \tilde f(\vec q') \in \dom(I),
	\\
	\cvec{0}, & \text{otherwise.}
	\end{cases}
	\end{equation}
	This formulation is known as
	backward warping \cite{szeliski2010computer}.
	It can be easily implemented
	for discrete images
	by leveraging bilinear interpolation.
	Contemporary frameworks already include
	the implementations for GPU hardware.
	Hence, the main difficulty is
	to determine the TPS parameters
	by solving two linear systems
	with $(n+3)\times (n+3)$ variables \cite{bookstein89pami}.

	In our experiments, we use $n=4$ control points
	corresponding to centers of the four image quadrants:
	$\del{\vec c_1', \mkern1.5mu{..}\mkern1.5mu, \vec c_4'} =
	\del{
		\sbr{\frac{1}{4}H, \frac{1}{4}W}^\transpose,
		\mkern1.5mu{..}\mkern1.5mu,
		\sbr{\frac{3}{4}H,\frac{3}{4}W}^\transpose}$.
	The parameters of our geometric transformation
	are four 2D displacements
	$\vec\gamma = \del{\vec d_1, \mkern1.5mu{..}\mkern1.5mu, \vec d_4}$.
	Let $f_{\vec\gamma}$ denote the resulting TPS warp.
	Then we can express our transformation as

	$T^\text{G}_{\vec\gamma}(\vec x) = \mathrm{backward\_warp}(
	\vec x, f_{\vec\gamma})$.


	Our training procedure picks a random $\vec\gamma$
	for each unlabeled image.
	We sample each displacement from
	a bivariate normal distribution
	$\mathcal N\del{\cvec 0_2, r\cvec I_2}$,
	where $r$ is the maximum
	L$^\infty$ norm of the displacement.
	We choose $r=0.05\cdot H$ by visual inspection,
	where $H$ is the image height.

	\subsection{Training procedure}

	Algorithm~\ref{alg:training} sketches a procedure for recovering
	gradients of the proposed semi-supervised loss~\eqref{eq:semisup-error}
	on a mixed batch of labeled
	and unlabeled examples.
	For simplicity, we make the following changes in notation here:
	$\vec x_\text{l}$ and $\vec y_\text{l}$ are batches of size $B_\text{l}$, $\vec x_\text{u}$, $\vec\gamma$ and $\vec\varphi$ batches of size $B_\text{u}$, and all functions are applied to batches.
	The algorithm computes the gradient of the supervised loss,
	discards cached activations,
	computes the teacher prediction,
	applies the consistency loss~\eqref{eq:1w-ct-objective}, and
	finally accumulates the gradient contributions of the two losses.
	Back-propagation through one-way consistency with clean teacher
	requires roughly the same extent of caching
	as in the supervised baseline.
	Hence, our approach does not constrain the model complexity.

	\begin{algorithm}[%
		caption={Evaluation of the proposed semi-supervised loss with respect to perturbation parameters ($\vec\gamma$, $\vec\varphi$) on a mixed batch of labeled ($\vec x_\text{l}$, $\vec y_\text{l}$) and unlabeled ($\vec x_\text{u}$) examples. $\mathrm{CE}$ denotes mean cross entropy, while $\mathrm{KL\_masked}$ denotes mean masked KL-divergence over valid pixels.},
		label={alg:training}]
		# $\vec x_\text{u}$, $\vec\gamma$, $\vec\varphi$, $\vec p_\text{t}$, $\vec p_\text{s}$, and $\vec v$ contain batches of $B_\text{u}$ elements.
		# $\vec x_\text{l}$, $\vec y$ and $\vec p_\text{l}$ contain batches of $B_\text{l}$ elements.
		procedure compute_loss_gradient($h, \vec\theta, \vec x_\text{l}, \vec y_\text{l}, \vec x_\text{u}, \vec\gamma, \vec\varphi$):
		$\vec\theta' \gets \mathrm{frozen\_copy}(\vec\theta)$  # simple one-way

		# supervised loss
		$\vec p_\text{l} \gets h_{\vec\theta}(\vec x_\text{l})$
		$L_\text{s} \gets \mathrm{CE}(\vec y_\text{l}, \vec p_\text{l})$
		$\vec g \gets \nabla_{\vec\theta}L_\text{s}$  # clears cached activations

		# unsupervised loss
		with autodiff.no_grad():  # activations not cached here
		$\vec p_\text{t} \gets T^\text{G}_{\vec\gamma}(h_{\vec\theta'}(\vec x_\text{u}))$  # clean teacher
		$\vec p_\text{s} \gets h_{\vec\theta}((T^\text{G}_{\vec\gamma} \circ T^\text{P}_{\vec\varphi})(\vec x_\text{u}))$  # perturbed student
		$\vec v \gets \lfloor T^\text{G}_{\vec\gamma}(\cvec 1_{B_\text{u}\times H\times W}) \rfloor$  # validity mask
		$L_\text{c} \gets \alpha\cdot \mathrm{KL\_masked}(\vec p_\text{t}, \vec p_\text{s}, \vec v)$
		$\vec g \gets \vec g + \nabla_{\vec\theta}L_\text{c}$

		return $\vec g$
	\end{algorithm}



	\section{Experiments}











	Our experiments evaluate one-way consistency with clean teacher
	and our perturbations. We compare its performance
	with other kinds of consistency
	and state of the art in semi-supervised
	semantic segmentation.
	We denote simple one-way consistency as "simple",
	Mean Teacher \cite{tarvainen17nips} as "MT",
	and our perturbations as "PhTPS".
	In experiments that compare consistency variants,
	"1w" denotes one-way, "2w" denotes two way, "ct" denotes clean teacher,
	"cs" denotes clean student, and "2p" denotes both inputs perturbed.
	We present semi-supervised experiments
	in several semantic segmentation setups
	as well as in image-classification setups on CIFAR-10.







	\subsection{Experimental setup}

	We perform semantic segmentation on
	Cityscapes \cite{cordts15cvprw}, and image classification on CIFAR-10.
	Cityscapes contains
	2975 training, 500 validation and
	1525 testing images
	with resolution $1024\times2048$.
	Images are acquired from a moving vehicle
	during daytime and fine weather conditions.
	We present half-resolution and
	full-resolution experiments.
	We use bilinear interpolation for images
	and nearest neighbour subsampling for labels.
	Some experiments on Cityscapes also use
	the coarsely labeled Cityscapes subset ("train-extra")
	that contains 19998 images.
	CIFAR-10 consists of 50000 training
	and 10000 test images
	of resolution $32\times32$.

	We apply the consistency loss to all training images
	(including the labeled ones).
	We train on batches of $B_\text{l}$ labeled
	and $B_\text{u}$ unlabeled images.
	We perform
	$\left\lfloor
	\envert{\set D_\text{l}}/B_\text{l}
	\right\rfloor$
	training steps per epoch.
	We always report generalization performance
	at the end of training.
	Our performance metrics correspond to
	means and variances
	over several runs of the experiment.
	We use the same perturbation model
	across all datasets and tasks
	(TPS displacements are proportional to image size),
	which is likely suboptimal \cite{cubuk19arxiv}.
	Unless stated otherwise,
	batch normalization statistics are updated
	only in forward passes with clean inputs.
	In Mean Teacher, the teacher uses a moving average
	of the student's estimated population statistics.


	Cityscapes experiments involve the following models:
	SwiftNet with ResNet-18 (SwiftNet-RN18) or ResNet-34 (SwiftNet-RN34), and
	DeepLab v2 with a ResNet-101 backbone.
	We initialize the backbones with
	ImageNet pre-trained parameters.
	%
	We apply random scaling, cropping and horizontal flipping
	to all inputs and segmentation labels.
	%
	We schedule the learning rate according to
	$e\mapsto\eta \cos(e\pi/2)$,
	where $e\in\sbr{0..1}$ is the fraction of epochs completed.
	This alleviates the generalization drop
	at the end of training with
	standard cosine annealing \cite{loschilov16sgdr}.
	We use learning rates $\eta=4\cdot10^{-4}$ for randomly initialized parameters
	and $\eta=10^{-4}$ for pre-trained parameters.
	We use Adam with $(\beta_1,\beta_2)=(0.9, 0.999)$.
	The L$^2$ regularization weight in supervised experiments is
	$10^{-4}$ for randomly initialized and
	$2.5\cdot10^{-5}$ for pre-trained parameters \cite{orsic19cvpr}.
	We have found that such L$^2$ regularization is too strong
	for our full-resolution semi-supervised experiments.
	Thus, we use a $4$ times smaller weight there.
	Due to early validation experiments,
	we set $\alpha=0.5$ unless stated otherwise.
	Batch sizes are $(B_\text{l}, B_\text{u})=(8,8)$
	for SwiftNet-RN18 \cite{orsic19cvpr}
	and $(B_\text{l}, B_\text{u})=(4,4)$
	for DeepLab v2 (ResNet-101 backbone) \cite{chen18pami}.
	The batch size in corresponding supervised experiments
	is $B_\text{l}$.

	In half-resolution Cityscapes experiments
	the size of crops is $448\times 448$
	and the scaling factor is sampled from $\mathcal{U}(1.5^{-1},1.5)$.
	%
	We train all models for 74400 iterations.
	This corresponds to about $200\cdot\frac{2975}{\envert{\set D_\text{l}}}$ epochs with SwiftNet
	and $100\cdot\frac{2975}{\envert{\set D_\text{l}}}$ epochs with DeepLab v2
	(when all labels are used, $\frac{2975}{\envert{\set D_\text{l}}}=1$).
	%
	In comparison with SwiftNet-RN18,
	DeepLab v2 incurs a 12-fold per-image slowdown
	during supervised training.
	However, it also requires less epochs
	since it has very few parameters
	with random initialization.
	Hence, semi-supervised DeepLab v2
	trains in 30h on RTX 2080 Ti,
	which is more than 5 times
	slower than SwiftNet-RN18.


	In full-resolution experiments,
	we only use SwiftNet models.
	The crop size is $768\times 768$
	and the scaling is sampled from $\mathcal{U}(2^{-1},2)$.
	The number of epochs is 250.
	The batch size is $8$ in supervised experiments,
	and $(B_\text{l}, B_\text{u})=(8,8)$
	in semi-supervised experiments.
	\delete{Due to an oversight, batch normalization statistics are also updated in the perturbed branch in full-resolution experiments.}

	Classification experiments target CIFAR-10
	and involve the Wide ResNet model WRN-28-2
	with standard hyperparameters \cite{zagoruyko16bmvc}.
	We augment all training images
	with random flips, padding and random cropping.
	We use all training images (including labeled images) in $\set D_\text{u}$ for the consistency loss.
	Batch sizes are $(B_\text{l}, B_\text{u})=(128,640)$.
	Thus, the number of iterations per epoch is
	$\left\lfloor\frac{\envert{\set D_\text{l}}}{128}\right\rfloor$.

	For example, only one iteration is performed
	if $\envert{\set D_\text{l}}=250$.
	We run $1000\cdot \frac{4000}{\envert{\set D_\text{l}}}$
	epochs in semi-supervised, and $100$ epochs in supervised training.
	We use default VAT hyper-parameters $\xi=10^{-6}$, $\epsilon=10$, $\alpha=1$ \cite{miyato19pami}.
	We perform photometric perturbations as described, and sample TPS displacements from $\mathcal N(\cvec 0, 3.2\cvec I)$.

	\subsection{Semantic segmentation on half-resolution Cityscapes}

	Table~\ref{tab:cs_half} compares our models
	with the state of the art
	on half-resolution Cityscapes val.
	The top section presents the previous work
	\cite{mittal19pami,hung18bmvc,mendel20eccv,french20bmvc}.
	The middle section presents
	our experiments based on DeepLab v2 \cite{chen18pami}.
	All these experiments use the same splits
	in order to ensure fair comparison.
	The first row shows that our experiment
	with the public code \cite{french20bmvc}
	reproduces the mIoU performance from their paper.
	The last three rows show
	experiments with our code.
	Note that we use more involved training
	than the previous work
	since that would be a method of choice
	in all practical applications.
	Hence, our performance is consistently
	greater than in the top section of the table.
	We enable fair comparison with \cite{french20bmvc}
	by plugging their method into
	our training procedure.
	Under these conditions,
	our MT-PhTPS outperforms MT-CutMix
	with L2 loss and confidence thresholding
	for 0.8 percentage points (pp) with 1/4 labels,
	1.4\,pp with 1/2 labels and
	1.0\,pp with all labels,
	while underperforming by 1.7\,pp
	with 1/8 labels.


	The bottom section presents
	experiments based on SwiftNet-RN18
	as mean and standard deviations
	across three different subsets.
	The last two rows again show that
	our perturbation model outperforms CutMix
	when $1/4$ or more labels are available.
	Notably, our perturbation model
	succeeds to improve upon
	the fully supervised baseline
	for both backbones.
	We observe that DeepLab v2
	and Swiftnet-RN18
	get very similar benefits from the consistency loss.
	Note that DeepLab v2 has a much stronger backbone
	(ResNet-101 vs ResNet-18).
	Nevertheless, SwiftNet-RN18 comes out
	as a method of choice for applications
	due to about 12 times faster inference.
	We also note that Mean Teacher performs
	comparably to simple consistency
	in experiments with all labels.

	\begin{table}[h]
		\centering
		\begin{tabular}{@{\;}l@{\;\;}c@{\;\;}
				c@{\;\;}c@{\;\;}c@{\;\;}c@{\;}}
			\noalign{\smallskip}
			\hline
			\noalign{\smallskip}
			& \multicolumn{4}{c}{Label proportion}
			\\
			Method & 1/8  &  1/4  &  1/2  & 1/1 \\
			\noalign{\smallskip}
			\hline
			\noalign{\smallskip}
			DLv2-RN101 supervised \cite{mittal19pami,french20bmvc} &
			$56.2_{\phantom{0.0}}$ & $60.2_{\phantom{0.0}}$ & -- & $66.0_{\phantom{0.0}}$
			\\
			DLv2-RN101 s4GAN \cite{mittal19pami} &
			$59.3_{\phantom{0.0}}$ & $61.9_{\phantom{0.0}}$ & -- & $65.8_{\phantom{0.0}}$
			\\
			DLv2-RN101 supervised \cite{hung18bmvc} &
			$55.5_{\phantom{0.0}}$  & $59.9_{\phantom{0.0}}$  & $64.1_{\phantom{0.0}}$ & $66.4_{\phantom{0.0}}$
			\\
			DLv2-RN101 AdvSemSeg \cite{hung18bmvc} &
			$58.8_{\phantom{0.0}}$  & $62.3_{\phantom{0.0}}$  & $65.7_{\phantom{0.0}}$ & $67.7_{\phantom{0.0}}$
			\\
		
		
		
		
		
		
			DLv2-RN101 supervised \cite{mendel20eccv} &
			$56.0_{\phantom{0.0}}$ & $60.5_{\phantom{0.0}}$ & -- & $66.0_{\phantom{0.0}}$
			\\
			DLv2-RN101 ECS \cite{mendel20eccv} &
			$\underline{60.3}_{\phantom{0.0}}$ & $\underline{63.8}_{\phantom{0.0}}$ & -- & $\underline{67.7}_{\phantom{0.0}}$
			\\
		
		
		
			DLv2-RN101 MT-CutMix \cite{french20bmvc} &
			$\underline{60.3}_{1.2}$ & $\underline{63.9}_{0.7}$ & -- & $\underline{67.7}_{0.4}$
			\\
			\hline
			DLv2-RN101 supervised\textsubscript{\cite{french20bmvc}} &
			$54.7_{\phantom{0.0}}$ & $59.7_{\phantom{0.0}}$ & $64.6_{\phantom{0.0}}$ & $67.5_{\phantom{0.0}}$
			\\
		
		
		
		
		
		
		
		
		
		
		
		
			DLv2-RN101 supervised &
			$56.4_{0.4}$ & $61.9_{1.1}$ & $66.6_{0.6}$ & $69.8_{0.4}$
			\\
			DLv2-RN101 MT-CutMix\textsubscript{$\sim$\cite{french20bmvc}} &
			$\mathbf{63.2}_{1.4}$  & $65.6_{0.8}$  & $67.6_{0.4}$ & $70.0_{0.3}$
			\\
			DLv2-RN101 MT-PhTPS &
			$61.5_{1.0}$  & $\mathbf{66.4}_{1.1}$ & $\mathbf{69.0}_{0.6}$ & $\mathbf{71.0}_{0.7}$
			\\
			\hline
			SN-RN18 supervised &
			$55.5_{0.9}$ & $61.5_{0.5}$ & $66.9_{0.7}$ & $70.5_{0.6}$
			\\
			SN-RN18 simple-CutMix &
			$59.8_{0.5}$ & $63.8_{1.2}$ & $67.0_{1.4}$ & $69.3_{1.1}$
			\\
			SN-RN18 simple-PhTPS &
			$61.1_{1.6}$ & $65.3_{1.9}$ & $68.5_{0.6}$ & $\mathbf{71.4}_{0.6}$
			\\
			SN-RN18 MT-CutMix\textsubscript{$\sim$\cite{french20bmvc}} &
			$61.6_{0.9}$  & $64.6_{0.5}$ & $67.6_{0.7}$ & $69.9_{0.6}$
			\\
			SN-RN18 MT-CutMix &
			$59.3_{1.3}$  & $63.3_{1.0}$ & $66.8_{0.6}$ & $69.7_{0.5}$
			\\
			SN-RN18 MT-PhTPS &
			$\underline{62.0}_{1.3}$ & $\mathbf{66.0}_{1.0}$ & $\mathbf{69.1}_{0.5}$ & $\mathbf{71.2}_{0.7}$
			\\
			\hline
		\end{tabular}
		\caption{
			Semi-supervised semantic segmentation accuracy (mIoU/\%)
			on half-resolution Cityscapes val
			with different proportions of labeled data.
			%
			The top section reviews experiments
			from previous work.
			%
			The middle section presents our experiments with DeepLab v2 with ResNet-101.
			The first row
		
			uses the implementation from \cite{french20bmvc},
			while other rows use our implementation.
			%
			The bottom section presents
			our experiments
			with SwiftNet-RN18 and
			different consistency variants.
			We run experiments
			across 5 different dataset splits
			and report mean mIoUs with standard deviations.
			%
			The subscript "$\sim$\cite{french20bmvc}"
			denotes training with $\mathrm{L^2}$ loss,
			confidence thresholding, and $\alpha=1$,
			as proposed in \cite{french20bmvc}
			instead of KL divergence with $\alpha=0.5$.
			Best results overall are bold,
			and best results within sections are underlined.
		}
		\label{tab:cs_half}
	\end{table}

	Table~\ref{tab:cs_half_ablation} considers several ablations
	of the perturbation model
	in experiments with SwiftNet-RN18 and simple-PHTPS consistency.
	We present experiments with
	only photometric perturbations (simple-Ph)
	and only geometric perturbations (simple-TPS).
	The table also includes supervised training with
	PhTPS augmentations
	in one half of each mini-batch
	in addition to standard jittering.
	Perturbing the whole mini-batch
	with PhTPS in supervised training
	did not improve upon the baseline.
	We observe that perturbing half of each mini-batch
	with PhTPS in addition to standard jittering
	in supervised training improves generalization performance,
	but quite less than semi-supervised training.
	Semi-supervised experiments suggest that
	photometric perturbations (Ph) alone always contribute,
	geometric perturbations (TPS) alone contribute
	only when $1/4$ or less labels are used,
	and PhTPS always outperforms Ph,
	except when all labels are used.

	\begin{table}[h]
		\centering
		\begin{tabular}{@{\;}l@{\;\;}c@{\;\;}
				c@{\;\;}c@{\;\;}c@{\;\;}c@{\;}}
			\noalign{\smallskip}
			\hline
			\noalign{\smallskip}
			& \multicolumn{4}{c}{Label proportion}
			\\
			Method & 1/8  &  1/4  &  1/2  & 1/1 \\
			\noalign{\smallskip}
			\hline
			\noalign{\smallskip}
			SN-RN18 supervised &
			$55.5_{\phantom{+0.0}}$ & $61.5_{\phantom{+0.0}}$ & $66.9_{\phantom{+0.0}}$ & $70.5_{\phantom{+0.0}}$
			\\
			SN-RN18 supervised PhTPS-aug &
			$56.2_{+1.5}$ & $62.2_{+0.7}$ & $67.4_{+0.5}$ & $70.4_{-0.1}$
		
			\\
		
		
		
			SN-RN18 simple-Ph &
			$59.2_{+3.7}$ & $64.9_{+3.4}$ & $68.3_{+1.4}$ & $\mathbf{71.8}_{+1.3}$
			\\
		
		
		
			SN-RN18 simple-TPS &
			$58.2_{+3.1}$ & $63.4_{+1.9}$ & $66.7_{-0.2}$ & $70.1_{-0.4}$
			\\
			SN-RN18 simple-PhTPS &
			$\mathbf{61.1}_{+5.6}$ & $\mathbf{65.3}_{+3.8}$ & $\mathbf{68.5}_{+1.6}$ & $\mathbf{71.4}_{+0.9}$
			\\
			\hline
		\end{tabular}
		\caption{
			Ablation experiments
			on half-resolution Cityscapes val (mIoU/\%)
			with SwiftNet-RN18.
			%
			The label "supervised PhTPS-aug" denotes
			supervised training where
			half of each mini-batch
			is perturbed with PhTPS.
			The bottom three rows compare PhTPS with
			Ph (only photometric) and TPS (only geometric)
			under simple one-way consistency.
			We present means of experiments
			on 5 different dataset splits.
			Numerical subscripts are differences
			with respect to the supervised baseline.
		}
		\label{tab:cs_half_ablation}
	\end{table}

	\subsection{Semantic segmentation on full-resolution Cityscapes}

	Table~\ref{tab:cs_full} presents our full resolution experiments
	in setups like in table~\ref{tab:cs_half},
	and comparison with previous work.
	but with full-resolution images and labels.







	In comparison with
	KE-GAN \cite{qi19cvpr} and
	ECS \cite{mendel20eccv},
	we underperform with $1/8$ labeled images,
	but outperform with $1/2$ labeled images.
	Note that KE-GAN \cite{qi19cvpr}
	also trains on a large text corpus
	(MIT ConceptNet) as well as
	that ECS DLv3$^+$-RN50 \cite{mendel20eccv}
	requires 22\,GB of GPU memory with batch size 6,
	while our SN-RN18 simple-PhTPS
	requires only 6.7\,GB of GPU memory
	with batch size 8
	and can be trained on affordable GPU hardware.

	We note that the concurrent approach DLv3$^+$-RN50 CAC \cite{lai2021cac}
	outperforms our method with 1/8 and 1/1 labels.
	However, ResNet-18 has significantly
	less capacity than ResNet-50.
	Therefore, the bottom section
	applies our method to the SwiftNet model
	with a ResNet-34 backbone,
	which still has less capacity than ResNet-50.
	The resulting model outperforms \cite{lai2021cac}
	across most configurations.
	This shows that our method
	consistently improves
	when more capacity is available.

	We note that training DLv3$^+$-RN50 CAC
	requires three RTX 2080Ti GPUs \cite{lai2021cac},
	while our SN-RN34 simple-PhTPS setup
	requires 7.7\,GiB of GPU memory
	and fits on a single such GPU.
	Moreover, SN-RN34 has about 4 times faster inference
	than DLv3$^+$-RN50 on RTX 2080Ti.





	\begin{table}[htb]
		\centering
		\begin{tabular}{@{\;}l@{\;\;}
				c@{\;\;}c@{\;\;}c@{\;\;}c@{\;}}
			\noalign{\smallskip}
			\hline
			\noalign{\smallskip}
			& \multicolumn{4}{c}{Label proportion}
			\\
			Method & 1/8  &  1/4  &  1/2  & 1/1 \\
			\noalign{\smallskip}
			\hline
			\noalign{\smallskip}
			DLv3$^+$-RN50 KE-GAN \cite{qi19cvpr} &
			$66.9_{\phantom{0.0}}$  & $70.6_{\phantom{0.0}}$ & $72.2_{\phantom{0.0}}$ & $75.3_{\phantom{0.0}}$
			\\
			DLv3$^+$-RN50 supervised \cite{mendel20eccv} &
			$63.2_{\phantom{0.0}}$ & $68.4_{\phantom{0.0}}$ & $72.9_{\phantom{0.0}}$ & $74.8_{\phantom{0.0}}$
			\\
			DLv3$^+$-RN50 ECS \cite{mendel20eccv} &
			$67.4_{\phantom{0.0}}$ & $70.7_{\phantom{0.0}}$ & $72.9_{\phantom{0.0}}$ & $74.8_{\phantom{0.0}}$
			\\
			DLv3$^+$-RN50 supervised \cite{lai2021cac} &
			$63.9_{\phantom{0.0}}$ & $68.3_{\phantom{0.0}}$ & $71.2_{\phantom{0.0}}$ & $76.3_{\phantom{0.0}}$
			\\
			DLv3$^+$-RN50 CAC \cite{lai2021cac} &
			$\mathbf{70.1}_{\phantom{0.0}}$ & $\underline{72.4}_{\phantom{0.0}}$ & $\underline{74.0}_{\phantom{0.0}}$ & $\underline{76.5}_{\phantom{0.0}}$
			\\
			\hline
			SN-RN18 supervised &
			$61.1_{0.4}$ & $67.3_{1.1}$ & $71.9_{0.1}$ & $75.4_{0.4}$
			\\
			SN-RN18 simple-PhTPS &
			$66.3_{1.0}$ & $71.0_{0.5}$ & $\underline{74.3}_{0.7}$ & $\underline{75.8}_{0.4}$
			\\
			SN-RN18 MT-PhTPS &
			$\underline{68.6}_{0.6}$ & $\underline{72.0}_{0.3}$ & $73.8_{0.4}$ & $75.0_{0.4}$
			\\
			\hline
			SN-RN34 supervised &
			$64.9_{0.8}$ & $69.8_{1.0}$ & $73.8_{1.4}$ & $76.1_{0.8}$
			\\
			SN-RN34 simple-PhTPS &
			$69.2_{0.8}$ & $73.1_{0.7}$ & $\mathbf{76.3}_{0.7}$ & $\mathbf{77.9}_{0.2}$
			\\
			SN-RN34 MT-PhTPS &
			$\mathbf{70.8}_{1.5}$ & $\mathbf{74.3}_{0.5}$ & $\mathbf{76.0}_{0.5}$ & $77.2_{0.4}$
			\\
			\hline
		\end{tabular}
		\caption{
			Semi-supervised semantic segmentation accuracy (mIoU/\%)
			on full-resolution Cityscapes val
			with different proportions of labeled data.
			We compare
			simple-PhTPS and MT-PhTPS (ours)
			with supervised training
			and previous work.
			DLv3$^+$-RN50 stands for DeepLab v3$^+$ with ResNet-50,
			and SN for SwiftNet.
			We run experiments
			across 5 different dataset splits
			and report mean mIoUs with standard deviations.
			Best results overall are bold,
			and best results within sections are underlined.
		}
		\label{tab:cs_full}
	\end{table}

	Finally, we present experiments
	in the large-data regime
	where we place the whole
	fine subset into $\set D_\text{l}$.
	In some of these experiments we also train
	on the large coarsely labeled subset.
	We denote extent of supervision
	with subscripts "l" (labeled) and "u" (unlabeled).
	Hence, $\mathrm{C_\text{u}}$ in the table
	denotes the coarse subset without labels.
	Table~\ref{tab:cs-coarse} investigates
	the impact of the coarse subset
	on SwiftNet performance on
	full-resolution Cityscapes val.
	%
	We observe that semi-supervised learning
	brings considerable improvement
	with respect to fully supervised learning
	on fine labels only
	(columns $\mathrm{F_\text{l}}$ vs
	$\mathrm{F_\text{l}} \cup \mathrm{C_\text{u}}$).
	%
	It is also interesting to compare
	the proposed semi-supervised setup
	($\mathrm{F_\text{l}} \cup \mathrm{C_\text{u}}$)
	with classic fully supervised
	learning on both subsets
	($\mathrm{F_\text{l}}, \mathrm{C_\text{l}}$).
	We observe that semi-supervised learning
	with SN-RN18
	comes close to supervised
	learning with coarse labels.
	Moreover, semi-supervised learning prevails
	when we plug in SN-RN34.
	These experiments suggest that
	unsupervised training represents
	an attractive alternative
	to coarse labels and large annotation efforts.

	\begin{table}[htb]
		\centering
		\begin{tabular}{@{\;}l@{\;\;}c@{\;\;}c@{\;\;}c@{\;\;}c@{\;\;}c@{\;}}
			\noalign{\smallskip}
			\hline
			\noalign{\smallskip}
			Method &
			\multicolumn{1}{c}{$\mathrm{F_\text{l}}$} &
			\multicolumn{1}{c}{$(\mathrm{F_\text{l}},\mathrm{F_\text{u}})$} &
			\multicolumn{1}{c}{$(\mathrm{F_\text{l}},
				\mathrm{F_\text{u}}\cup\mathrm{C_\text{u}})$} &
			\multicolumn{1}{c}{$(\mathrm{F_\text{l}},
				\mathrm{C_\text{l}})$} \\
			\noalign{\smallskip}
			\hline
			\noalign{\smallskip}
			SN-RN18 simple-PhTPS &
			\multirow{2}{*}{$75.4_{0.4}$} & $75.8_{0.4}$ & $76.5_{0.3}$ & \multirow{2}{*}{$76.9_{0.3}$} \\
			SN-RN18 MT-PhTPS &
			& $75.0_{0.4}$ & $75.5_{0.3}$ & \\
			\hline
			SN-RN34 simple-PhTPS &
			$76.1_{0.8}$ & $77.2_{0.4}$ & $78.5_{0.4}$ & $77.7_{0.4}$ \\
		
		
			\hline
		\end{tabular}
		\caption{
			Impact of the training setup on
			mIoU/\% performance on
			full-resolution Cityscapes val.
			Tags F and C denote fine and
			coarse subsets, respectively.
			Subset indices denote whether
			we train with labels ($\textrm{l}$)
			or one-way consistency ($\textrm{u}$).
		}
		\label{tab:cs-coarse}
	\end{table}

	Figure~\ref{fig:cs_examples} illustrates
	qualitative results of
	semi-supervised training on Cityscapes val.
	The left column displays images from Cityscapes val perturbed using $T_{\vec\tau}$.


	The middle and bottom row visualize predictions
	for supervised
	and semi-supervised training, respectively.
	We observe that our method
	develops a substantial resilience
	to heavy perturbations like those
	used during training.

	\newcommand{0.4\textwidth}{0.4\textwidth}

	\newcommand{3.2cm}{3.2cm}
	\begin{figure}[htb]
		\centering
		\includegraphics[width=0.4\textwidth]
		{figs/examples/cs/img_frankfurt_000001_017101.png}\,%
		\includegraphics[width=0.4\textwidth]
		{figs/examples/cs/img_frankfurt_000001_082087.png}  \\
		\includegraphics[width=0.4\textwidth]
		{figs/examples/cs/baseline_frankfurt_000001_017101.png}\,%
		\includegraphics[width=0.4\textwidth]
		{figs/examples/cs/baseline_frankfurt_000001_082087.png} \\
		\includegraphics[width=0.4\textwidth]
		{figs/examples/cs/phtps_frankfurt_000001_017101.png}\,%
		\includegraphics[width=0.4\textwidth]
		{figs/examples/cs/phtps_frankfurt_000001_082087.png}
		\caption{
			Qualitative results on Cityscapes val.
			Images perturbed with $T_{\vec\tau}$
			are presented at the model input (top).
			Models trained with the standard
			supervised loss produce
			many incorrect predictions (middle).
			Models trained with semi-supervised consistency
			achieve much better generalization performance (bottom).
		}
		\label{fig:cs_examples}
	\end{figure}


	\subsection{Validation of consistency variants}

	Table~\ref{tab:cons-variants} compares
	supervised baselines with
	4 kinds of unsupervised consistency
	on CIFAR-10 and half-resolution Cityscapes.
	We investigate the following kinds of consistency:
	one-way with clean teacher (1w-ct),
	one-way with clean student (1w-cs),
	two-way with one clean input (2w-c1), and
	one-way with both inputs perturbed (1w-p2).
	Note that two-way consistency is not
	possible with Mean Teacher.
	Also, when both inputs are perturbed (1w-p2),
	we have to use the inverse geometric transformation
	on dense predictions \cite{bortsova19miccai}.
	We use forward warping \cite{niklaus2020softsplat}
	with the same displacement field for this.
	Two-way consistency with both inputs perturbed (2w-p2)
	is possible as well.
	We expect it to behave similarly to 1w-2p
	because it could be seen as a superposition
	of two opposite one-way consistencies,
	and our preliminary experiments suggest so.

	All experiments use PhTPS perturbations.
	CIFAR-10 experiments train
	on 4000 labels and 50000 images.
	Cityscapes experiments correspond to
	the setup from Table~\ref{tab:cs_half}
	based on SwiftNet-RN18.
	The table suggests that 1w-ct performs best,
	while 2w-c1 performs in-between 1w-ct and 1w-cs,
	as well as that predictions from unperturbed inputs
	represent better targets for our unsupervised loss.
	The 1w-p2 setup underperforms
	with respect to 1w-ct,
	but often outperforms 1w-cs.
	A closer inspection reveals
	that 1w-p2 sometimes learns
	to cheat the consistency loss
	by outputting similar predictions
	in all perturbed images.
	This is more likely to occur when
	batch normalization uses the batch statistics
	estimated during training.


	A closer inspection of 1w-cs Cityscapes experiments
	reveals severe overfitting to the training dataset
	as well as consistency-cheating.










	\begin{table}[htb]
		\centering
		\begin{tabular}{@{}l@{\;\;}l@{\;\;}|c@{\;\;}
				c@{\;\;}c@{\;\;}c@{\;\;}c@{}}
			Dataset & Algorithm & sup. &  {1w-ct} &
			{1w-cs} & {2w-c1} & {1w-p2} \\
			\hline
			CIFAR-10, 4k & simple-PhTPS & $80.8_{0.4}$ & $\mathbf{90.8}_{0.3}$ & $69.3_{4.2}$ & $72.9_{2.6}$ & $73.3_{7.0}$ \\
			CIFAR-10, 4k & MT-PhTPS & $80.8_{0.4}$ & $\mathbf{90.8}_{0.4}$ & $80.5_{0.5}$ & - & $73.4_{1.4}$ \\
			CS-half, 1/4 & simple-PhTPS & $61.5_{0.5}$ & $\mathbf{65.3}_{1.9}$ & $\phantom{0}1.6_{1.0}$ & $16.7_{3.0}$ & $61.6_{0.5}$ \\
			CS-half, 1/4 & MT-PhTPS & $61.5_{0.5}$ & $\mathbf{66.0}_{1.0}$ & $61.5_{1.4}$ & - & $62.0_{1.1}$ \\
		
		
		\end{tabular}	\caption{
			Comparison of 4 consistency variants under
			PhTPS perturbations:
			one-way with clean teacher (1w-ct),
			one-way with clean student (1w-cs),
			two-way with one input clean (2w-c1), and
			one-way with both inputs perturbed (1w-p2).
			Algorithms are evaluated on
			CIFAR-10 test (accuracy/\%)
			while training on 4000 out of 50000 labels (CIFAR-10, $2/25$)
			and half-resolution Cityscapes val (mIoU/\%)
			while training on $1/4$ of labels
			from Cityscapes train
			with SwiftNet-RN18 (CS-$1/4$).
		
		
		
		
		}
		\label{tab:cons-variants}
	\end{table}











	%


	\subsection{Image classification on CIFAR-10}

	Table~\ref{tab:cifar10} evaluates image classification performance
	of two supervised baselines
	and 4 semi-supervised algorithms on CIFAR-10.
	The first supervised baseline uses only labeled data
	with standard data augmentation.
	The second baseline
	additionally uses our perturbations
	for data augmentation.
	The third algorithm is VAT
	with entropy minimization \cite{miyato19pami}.
	The simple-PhTPS approach outperforms
	supervised approaches and VAT.
	Two-way consistency
	results in worst generalization performance.
	Perturbing the teacher input
	results in accuracy below 17\%
	for 4000 or less labeled examples,
	and is not displayed.
	Note that somewhat better performance can be achieved
	by complementing consistency with other techniques
	that are either unsuitable for dense prediction or out of scope of this paper \cite{cubuk19arxiv,xie19arxiv_uda,berthelot19nips}.

	\begin{table}[htb]
		\centering
		\begin{tabular}
			{@{\,}l@{\;\;}c@{\;\;}c@{\;\;}c@{\;\;}c@{}}
			\noalign{\smallskip}
			\hline\noalign{\smallskip}
			& \multicolumn{4}{@{}c@{}}
			{Number of labeled examples}
			\\
			Method &  250  &  1000  &  4000  & 50000
			\\
			\noalign{\smallskip}
			\hline
			\noalign{\smallskip}
			supervised
		
			& $31.8_{0.6}$ & $59.3_{1.4}$
			& $81.0_{0.2}$ & $94.7$
			\\
			supervised PhTPS-aug
			& $48.7_{0.9}$ & $67.2_{0.5}$
			& $81.7_{0.2}$ & $95.0$
			\\
			\hline
			VAT + entropy minimization
			& $41.0_{2.5}$ & $73.2_{1.5}$
			& $84.2_{0.4}$ & $90.5_{0.2}$
			\\
		
		
		
		
		
		
		
		
		
		
		
		
		
		
		
		
		
		
		
		
		
		
		
		
		
			1w-cs simple-PhTPS
			& $27.7_{5.9}$ & $51.7_{3.5}$
			& $69.3_{4.2}$ & $91.6_{1.5}$
			\\
			2w-c1 simple-PhTPS
			& $30.3_{1.8}$ & $54.8_{1.5}$
			& $72.9_{2.6}$ & $95.9_{0.2}$
			\\
			1w-ct simple-PhTPS
			& $\mathbf{68.8}_{5.4}$ & $\mathbf{84.2}_{0.4}$
			& $\mathbf{90.6}_{0.4}$ & $\mathbf{96.2}_{0.2}$
			\\
			\hline
		\end{tabular}
		\caption{
			Classification accuracy [\%]
			on CIFAR-10 test with WRN-28-2.
			We compare two supervised approaches (top),
			VAT with entropy minimization \cite{miyato19pami}
			(middle),
			and two-way and one-way consistency
			with our perturbations
			(bottom three rows).
			We report means and standard deviations of $5$ runs.
			The label "supervised PhTPS-aug" denotes
			supervised training where
			half of each mini-batch
			is perturbed with PhTPS.
		}
		\label{tab:cifar10}
	\end{table}

	\section{Conclusion}

	We have presented the first comprehensive study
	of one-way consistency for semi-supervised dense prediction,
	and proposed a novel perturbation model
	which leads to competitive generalization performance on Cityscapes.
	Our study clearly shows that
	one-way consistency with clean teacher
	outperforms other forms of consistency
	(e.g.~clean student or two-way)
	both in terms of generalization performance
	and training footprint.
	We explain this by observing
	that predictions in perturbed images
	tend to be less reliable targets.

	The proposed perturbation model is a composition
	of a photometric transformation and a geometric warp.
	These two kinds of perturbations
	have to be treated differently since we desire
	invariance with respect to the former
	and equivariance with respect to the latter.
	Our perturbation model outperforms CutMix
	both in standard experiments with DeepLabv2-RN101
	and in combination with
	recent efficient models (SwiftNet-RN18 and Swiftnet-RN34).

	We consider two teacher formulations.
	In the simple formulation the teacher
	is a frozen copy of the student.
	In the Mean Teacher formulation,
	the teacher is a moving average of student parameters.
	Mean Teacher outperforms simple consistency
	in low data regimes (half resolution, few labels).
	However, experiments with more data
	suggest that the simple one-way formulation
	scales significantly better.

	To the best of our knowledge,
	this is the first account of
	semi-supervised semantic segmentation
	with efficient models.
	This combination is essential
	for many practical real-time applications
	where there is a lack of large datasets
	with suitable pixel-level groundtruth.
	Many of our experiments are based on SwiftNet-RN18,
	which behaves similarly to DeepLabv2-RN101,
	while offering
	about $9\times$ faster inference on half-resolution images,
	and about $15\times$ faster inference on full-resolution images
	on RTX 2080Ti.
	Experiments on Cityscapes coarse
	reveal that semi-supervised learning
	with one-way consistency
	can come close and exceed
	full supervision with coarse annotations.
	Simplicity, competitive performance and speed of training
	make this approach a very attractive baseline
	for evaluating future semi-supervised approaches
	in the dense prediction context.


	

\section{Qualitative results}

Figure~\ref{fig:cs_examples} illustrates 
qualitative results of
semi-supervised and supervised training evaluated 
on half-resolution Cityscapes val.
The left column displays clean and perturbed versions of images from Cityscapes val.
The second column displays corresponding fround truth.
The last two columns display predictions of
supervised training and semi-supervised MT-PhTPS training.
MT-PhTPS results in less errors on clean images and much more robustness to $T_{\vec\tau}$ perturbations.

\newcommand{\imwidth}{0.24\textwidth} 
\begin{figure*}[!htb]
  \centering
\includegraphics[width=\imwidth]{figs/cs_half_1_4/0000_x.png}\,\includegraphics[width=\imwidth]{figs/cs_half_1_4/0000_y.png}\,\includegraphics[width=\imwidth]{figs/cs_half_1_4/0000_out.png}\,\includegraphics[width=\imwidth]{figs/cs_half_1_4/0000_out_sup.png}\,\\
\includegraphics[width=\imwidth]{figs/cs_half_1_4/0000_x_p.png}\,\includegraphics[width=\imwidth]{figs/cs_half_1_4/0000_y_p.png}\,\includegraphics[width=\imwidth]{figs/cs_half_1_4/0000_out_p.png}\,\includegraphics[width=\imwidth]{figs/cs_half_1_4/0000_out_p_sup.png}\,\\
\includegraphics[width=\imwidth]{figs/cs_half_1_4/0001_x.png}\,\includegraphics[width=\imwidth]{figs/cs_half_1_4/0001_y.png}\,\includegraphics[width=\imwidth]{figs/cs_half_1_4/0001_out.png}\,\includegraphics[width=\imwidth]{figs/cs_half_1_4/0001_out_sup.png}\,\\
\includegraphics[width=\imwidth]{figs/cs_half_1_4/0001_x_p.png}\,\includegraphics[width=\imwidth]{figs/cs_half_1_4/0001_y_p.png}\,\includegraphics[width=\imwidth]{figs/cs_half_1_4/0001_out_p.png}\,\includegraphics[width=\imwidth]{figs/cs_half_1_4/0001_out_p_sup.png}\,\\
\includegraphics[width=\imwidth]{figs/cs_half_1_4/0002_x.png}\,\includegraphics[width=\imwidth]{figs/cs_half_1_4/0002_y.png}\,\includegraphics[width=\imwidth]{figs/cs_half_1_4/0002_out.png}\,\includegraphics[width=\imwidth]{figs/cs_half_1_4/0002_out_sup.png}\,\\
\includegraphics[width=\imwidth]{figs/cs_half_1_4/0002_x_p.png}\,\includegraphics[width=\imwidth]{figs/cs_half_1_4/0002_y_p.png}\,\includegraphics[width=\imwidth]{figs/cs_half_1_4/0002_out_p.png}\,\includegraphics[width=\imwidth]{figs/cs_half_1_4/0002_out_p_sup.png}\,\\
\includegraphics[width=\imwidth]{figs/cs_half_1_4/0003_x.png}\,\includegraphics[width=\imwidth]{figs/cs_half_1_4/0003_y.png}\,\includegraphics[width=\imwidth]{figs/cs_half_1_4/0003_out.png}\,\includegraphics[width=\imwidth]{figs/cs_half_1_4/0003_out_sup.png}\,\\
\includegraphics[width=\imwidth]{figs/cs_half_1_4/0003_x_p.png}\,\includegraphics[width=\imwidth]{figs/cs_half_1_4/0003_y_p.png}\,\includegraphics[width=\imwidth]{figs/cs_half_1_4/0003_out_p.png}\,\includegraphics[width=\imwidth]{figs/cs_half_1_4/0003_out_p_sup.png}\,\\
\includegraphics[width=\imwidth]{figs/cs_half_1_4/0004_x.png}\,\includegraphics[width=\imwidth]{figs/cs_half_1_4/0004_y.png}\,\includegraphics[width=\imwidth]{figs/cs_half_1_4/0004_out.png}\,\includegraphics[width=\imwidth]{figs/cs_half_1_4/0004_out_sup.png}\,\\
\includegraphics[width=\imwidth]{figs/cs_half_1_4/0004_x_p.png}\,\includegraphics[width=\imwidth]{figs/cs_half_1_4/0004_y_p.png}\,\includegraphics[width=\imwidth]{figs/cs_half_1_4/0004_out_p.png}\,\includegraphics[width=\imwidth]{figs/cs_half_1_4/0004_out_p_sup.png}\,\\
\caption{
    Qualitative results on images from half-resolution Cityscapes val with SwiftNet-RN18 trained on half-resolution Cityscapes train with $1/4$ of the labels.
    The colums are as follows: (1) input image (perturbed with $T_{\vec\tau}$ in odd rows), (2) corresponding ground truth label (warped with $T^{\text G}_{\vec\gamma}$ in even rows), (3) segmentation predicted when trained with MT-PhTPS, and (4) segmentation predicted when trained only on labeled images (supervised).
    Models trained with semi-supervised consistency 
    produce better segmentations in clean images and 
    are much more robust to perturbations used during training.
  }
  \label{fig:cs_examples}
\end{figure*}



