\section{Introduction}
\IEEEPARstart{M}{any} autonomous agents, \emph{e.g.}\xspace mobile robots and driverless vehicles, rely on scene understanding tasks for navigation and interaction with the environment and semantic segmentation is one of the basic, yet most important visual perception tasks~\cite{DeepLab, RefineNet, HRNet,  PSPNet}. In recent years, the significant developments in machine learning and neural networks have been conducive to supervised semantic segmentation regime~\cite{DeepLab, RefineNet, FCN}. However, the superior performance of these supervised techniques is highly subject to the availability of a large-scale and high-quality annotated dataset, and providing such a pixel-level annotation for a dataset from a novel domain, \emph{e.g.}\xspace satellite or underwater images, can be a very challenging and laborious task~\cite{ARL5, ARL17, IIC38}.

Recently unsupervised learning techniques such as deep clustering methods show promises in dealing with image-level tasks such as unsupervised image classification problems~\cite{DEC, VaDE, InfoGAN, IMSAT, AE+CE}. More specifically, maximizing mutual information plays a remarkable role in the core of recent works, and many algorithms have successfully utilized this concept for unsupervised image classification~\cite{DEC, VaDE, InfoGAN, IMSAT, JULE, DAC}. However, these techniques cannot be naively applied to semantic segmentation tasks, which require spatially smooth and consistent label predictions at the pixel-levels.

\begin{figure}
    \centering
    \begin{subfigure}[t]{0.11\textwidth}
      \includegraphics[width=\textwidth]{images/teaser/90-000000239093/img-in.png}
    \end{subfigure}
    \begin{subfigure}[t]{0.11\textwidth}
      \includegraphics[width=\textwidth]{images/teaser/88-000000499738/img-in.png}
    \end{subfigure}
    \begin{subfigure}[t]{0.11\textwidth}
      \includegraphics[width=\textwidth]{images/teaser/93-000000478785/img-in.png}
    \end{subfigure}
    \begin{subfigure}[t]{0.11\textwidth}
      \includegraphics[width=\textwidth]{images/teaser/93-000000323476/img-in.png}
    \end{subfigure}
    
    
    \par\smallskip
    \centering
    \begin{subfigure}[t]{0.11\textwidth}
      \includegraphics[width=\textwidth]{images/teaser/90-000000239093/img-gt.png}
    \end{subfigure}
    \begin{subfigure}[t]{0.11\textwidth}
      \includegraphics[width=\textwidth]{images/teaser/88-000000499738/img-gt.png}
    \end{subfigure}
    \begin{subfigure}[t]{0.11\textwidth}
      \includegraphics[width=\textwidth]{images/teaser/93-000000478785/img-gt.png}
    \end{subfigure}
    \begin{subfigure}[t]{0.11\textwidth}
      \includegraphics[width=\textwidth]{images/teaser/93-000000323476/img-gt.png}
    \end{subfigure}
    
    \par\smallskip
    \centering
    \begin{subfigure}[t]{0.11\textwidth}
      \includegraphics[width=\textwidth]{images/teaser/90-000000239093/img-sp.png}
    \end{subfigure}
    \begin{subfigure}[t]{0.11\textwidth}
      \includegraphics[width=\textwidth]{images/teaser/88-000000499738/img-sp.png}
    \end{subfigure}
    \begin{subfigure}[t]{0.11\textwidth}
      \includegraphics[width=\textwidth]{images/teaser/93-000000478785/img-sp.png}
    \end{subfigure}
    \begin{subfigure}[t]{0.11\textwidth}
      \includegraphics[width=\textwidth]{images/teaser/93-000000323476/img-sp.png}
    \end{subfigure}
    
    \caption{\textbf{Segmentation Result}. (first row) Some examples from COCO-Stuff datasets~\cite{Cocostuff}, (second row) their truth, and (third row) the predicted segmentation masks using our proposed method (InMARS$^{+}$).}
    \label{fig:teaser:coco-st3}
  \end{figure}
  
Recently, there have been few attempts to tackle this challenging problem by conducting various strategies and domain knowledge~\cite{WNet, IIC, SegSort, ARL}. However, these techniques are more susceptible to degenerate/trivial solutions, and their superior performance are highly dependent on their incorporated data augmentation strategy. \par
This paper introduces a novel pixel-wise and fully unsupervised learning method for semantic segmentation. Our proposed approach uses an adversarially trained information maximization technique over the features extracted from a set of segmented regions, also known as superpixels.
By extracting features from these grouped pixels, we, indeed, reformulate unsupervised image segmentation as a sub-region unsupervised classification task and transform decision space from pixel-level into sub-region domain. Next, we utilize Mutual Information Maximization for maintaining maximum information between input and output, which enforces semantically similar and different data to be assigned to the same and distinct clusters, respectively. We also incorporate an adversarial training strategy to act as a regularizer for our model, helping it learn a robust and generalizable representation for clustering the extracted features. To customize our learning regime for the segmentation task, we also propose a novel adversarial strategy by bundling a learnable adversarial pixel intensity noise with a set of random geometric perturbations on the superpixels.
We test and evaluate the proposed method on four subsets of two commonly used unsupervised image segmentation benchmark datasets: Potsdam \cite{Potsdam}, and COCO-stuff \cite{Cocostuff}. We demonstrate the superior performance of our proposed method, outperforming the existing state-of-the-art methods on these datasets. Moreover, we perform an ablation study to show each component's contribution to the proposed method. In summary, the main contributions of this work can be summarized as follows:

\begin{enumerate}
\itemsep0em
\item We propose a new unsupervised image segmentation approach based on adversarial training and mutual information,
\item We introduce a novel sub-region-level training strategy for image segmentation and incorporate customized adversarial pixel noise followed by spatial perturbations to make the model robust to the  photometrical and geometrical variations,
\item Our approach achieves the state-of-the-art results on commonly used unsupervised semantic segmentation datasets.
\end{enumerate}\par


\section{Related works} \label{sec:ReWo}
\textbf{Supervised Semantic Segmentation}: The recent advances in deep learning and convolutional neural architectures, along with the availability of large-scale annotated datasets, have been helpful to the impressive performance of the recent semantic segmentation algorithms~\cite{ReviewSS2020}. Fully Convolutional Network (FCNs)~\cite{FCN} is a successful generation of convolutional neural network (CNN) architectures that achieved a significant improvement over classical approaches. Several notable architectural variants of FCNs are later proposed by exploiting various strategies. For instance, \cite{RefineNet} and \cite{PSPNet} proposed different multi-scale and pyramid FCN architectures to learn multi-scale feature representation resulting in a better segmentation performance for various region sizes' resolution. Chen \emph{et al.}\xspace~\cite{DeepLab} also proposed atrous spatial pyramid pooling to segment objects at multiple scales robustly while applying Conditional Random Field (CRF) additionally at the end of the network to smooth out the pixel-wise label predictions. Jun \emph{et al.}\xspace~\cite{DANet} suggested an attention mechanism to integrate local features with their global dependencies. Recently, \cite{HRNet} introduced a computational tractable multi-stage architecture to extract a fast but high-resolution feature representation. Despite the progressive success of supervised techniques, their great performance depends on the availability of large-scale annotated data.\par

\textbf{Deep Clustering}: 
Data clustering is a classical pattern recognition problem, and several model-based approaches, \emph{e.g.}\xspace K-means~\cite{He_2013_CVPR}, Gaussian Mixture~\cite{GMM-Cl},  Kernel-based~\cite{NIPS2004_2602} and spectral clustering~\cite{On-spectral-clustering}, have been developed to tackle this fundamental problem. However, their performance considerably degrades on highly non-linear data and/or in a high dimensional space, \emph{e.g.}\xspace raw and unprocessed images. Deep learning has provided an opportunity for these clustering techniques to overcome their drawbacks by clustering on a proper embed (and possibly low-dimensional) feature space, encoded by these neural models. Deep Embedded Clustering (DEC)~\cite{DEC} is one of the pioneering works, which combines one of the classical clustering methods with deep learning. This method combines a reconstruction loss on an Auto-Encoder (AE) with a clustering objective on the encoded features to construct and cluster the rich low-dimensional encoded features, simultaneously. Other similar approaches also use AE reconstruction loss but followed by a K-means~\cite{kmean+AE}, or a cross-entropy objective between multiple network predictions~\cite{AE+CE} to perform the clustering task on the deep embed space. The other generative models such as Variational AE (VAE) ~\cite{VaDE} and InfoGAN~\cite{InfoGAN} are also used in combination with clustering objectives to learn a clustering-friendly latent space. 

Unlike the earlier works, IMSAT~\cite{IMSAT} considers the clustering task as a discrete representation problem, where each discrete representation would be a code for the assignment of input data to a specific cluster. Then an objective is defined to maximize mutual information between inputs and outputs while regularizing this representation by an adversarial perturbation. In spite of their success in the data clustering problems, these deep clustering methods cannot be naively used for the unsupervised semantic segmentation problem where the task is to cluster semantically same pixels with high spatial/local dependencies using fine-grained deep features.\par

\textbf{Unsupervised Semantic Segmentation}:
Being a very challenging problem, there have been a few recent attempts to tackle unsupervised image segmentation. 
Hwang \emph{et al.}\xspace~\cite{SegSort} proposed a method by combining two sequential clustering modules for both pixel-level and segment-level to perform the task. Notwithstanding its satisfactory performance, this approach relies on mean-shift clustering, making it susceptible to the degeneracy issue, \emph{i.e.}\xspace assigning random labels to different candidates. 
Generative model-based approaches such as~\cite{ARL4, ReDrawGAN} generate a semantic mask for an image by perturbing~\cite{ARL4} or redrawing~\cite{ReDrawGAN} the generated foreground and background masks. Despite their great performance, their application is limited to the cases where each image only contains a single foreground object.\par
Perhaps the most relevant and competitive approaches to ours are the recent works of IIC~\cite{IIC} and AC~\cite{ARL}, which similarly uses the mutual information maximization concept to tackle unsupervised segmentation task. Different from these frameworks, our proposed method, inspired by~\cite{IMSAT}, models the unsupervised segmentation task as maximizing mutual information between the inputs and a discrete representation, \emph{i.e.}\xspace the cluster assignments.   
Moreover, while these methods rely on heuristic data augmentation strategies for their unsupervised model, our proposed framework incorporates a wisely customized adversarial training strategy as a learning-based data augmentation, which is jointly optimized with mutual information maximization objectives. Additionally, we suggest a new overlook into the problem by suggesting to directly cluster spatially connected groups of pixels, \emph{i.e.}\xspace superpixels, instead of the pixels. This strategy facilitates training, reduces the computational cost, and increases statistics' reliability during training by increasing batch size.

\section{Proposed method} \label{sec:PrMe}
\subsection{Overview}
\textcolor{black}}{{The} semantic segmentation task can be considered as a mapping function $F:X\rightarrow Y$ from input image $X\in \mathbb{R}^{(w\times h\times c)}$ to the output map $Y\in \mathbb{B}^{(w\times h\times k)}$ where $w$ and $h$ define the input image's width and height and $c$ and $k$ represent the color and the semantic label channels of the input and the output, respectively. Note that in this representation, $Y$ is a discrete binary tensor, and the label of each pixel in the image $X$ is encoded as a one-hot binary vector in $Y$. Therefore, a group of pixels with an identical one-hot binary code represents the same semantic object class. Without loss of generality, we can assume there always exists a proper mapping function between the inputs and outputs, which can be estimated using a deep neural network\footnote{ This claim can be supported by this fact that deep learning has favorably applied to the supervised semantic segmentation task, and many algorithms successfully tackled this problem \cite{DeepLab, RefineNet, PSPNet, FCN}}. \par

In an unsupervised setting, the main task is to find a proper function $F$ using a deep model, which learns to perform this mapping without any annotation. To tackle this, we build a framework (Fig.\xspace~\ref{fig:usn}) based on three major components, including Region-Wise Embedding (RWE), Mutual Information Maximization (MIM), and Adversarial Regularization, which are separately elaborated in the followings.\par
\begin{figure}[t]
\begin{center}
\includegraphics[width=1\linewidth]{images/RA-Letter-diagram-Final.png}
\vspace{-2.5em}
\end{center}
   \caption{Building blocks of our proposed method. The \colorbox{myYellow}{yellow} and \colorbox{myRed}{red} colors indicate original data and augmented version of original data, respectively. The input image $X$ and its adversarial augmented version $X'$ are passed from the network and generate response $Y$ and $Y'$, respectively. The network is combined of three modules, Backbone, Superpixel, and RWE. Backbone is a fully convolutional network. Superpixel module extract sub-region from the input image. RWE map output of Backbone to the sub-regions to generate fixed-length feature. The clustering loss $I(X;Y)$ is computed between input $X$ and its corresponding embedding $Y$. The regularization loss $R(\theta)$ controls the network to generate rich embedding via \textit{Adversarial Augmentation} module.}
\label{fig:usn}
\end{figure}
\textbf{From Pixel to Region}: 
At the pixel-level, extracting semantically meaningful features is more complicated than the image-level, as we need to ensure maintaining the spatial smoothness and connectivity between the semantic label of neighboring pixels while generating an embedding for each pixel.
To this end, we propose to exploit segmentation region proposals, \emph{e.g.}\xspace superpixels, and to assign the same label to all the pixels in each region to ensure holding their spatial label smoothness while reducing the problem's complexity. These superpixels can be directly extracted using any unsupervised, model-based, or deep learning-based frameworks~\cite{SLIC, SSN, SP_Affinity_Loss, SP-FCN}. In fact, each superpixel provides a hypothetically good initial guess for the spatial connectivity of the semantically meaningful pixels in proximity without any supervision. Therefore, from each of these superpixel regions, one feature embedding can be extracted as the representative. The module extracting these features is called Region-Wise-Embedding (RWE) in our framework. RWE receives an embedding map from the backbone's output and produces a fixed-length embedding for each superpixel using a pooling strategy. The effect of the superpixel parameter selection and pooling strategy for this module will be discussed in the ablation study. Note that the RWE module might resemble the Region Proposal Network (RPN) in Faster R-CNN detector~\cite{FasterRCNN}.\par

\textbf{Mutual Information Maximization and Adversarial Regularization}: Training a deep network requires optimizing a loss function. In unsupervised settings, a loss function should optimize the network parameters without any annotated data. Hence, the design of the loss function capturing a proper source of information for the task plays a remarkable role in tackling the problem. In the absence of an annotated data, the input measurements and the model predictions are the only available source of information for training the model. In addition, data augmentation, as a very popular strategy, can be conducive to the unsupervised training scheme, \emph{e.g.}\xspace by generating additional pseudo labeled data. This is because the augmented version of input can hypothetically have the same label as the original input. Using this pair match, the model is encouraged to generate a similar output for this pair, \emph{i.e.}\xspace the input data, and its augmented version.\par
Selecting a proper data augmentation strategy with a reasonable perturbation/augmentation magnitude has a crucial impact on the quality of optimization convergence, \emph{e.g.}\xspace simple data augmentation or a minor perturbation may not help the optimization to converge to a good solution, and a high perturbation or incorrect augmentation strategy may result in optimization divergence,~\cite{dataAug}. To this end, the previous works~\cite{IIC, UDA} have also attempted to adopt a proper augmentation strategy for their methods to generate reasonable results. Despite this popularity, designing proper augmentation is still challenging since the perturbation should not alter the meaning of data.\par
To address this challenge, we utilize adversarial training in our data augmentation module in order to generate adversarial examples while bound the amount of perturbation in a safe region~\cite{IMSAT}. A safe region is an area where adding perturbation does not alter the meaning of input data, and it is a hyper-parameter in adversarial training. As depicted in Figure.~\ref{fig:usn}, the adversarial augmentation module gets an image $X$ and generate its augmented version $X^{'}$. This secondary input passes through the network and result in $Y^{'}$. The adversarial objective is then computed as a deviation between $Y^{'}$ and $Y$. Mutual information is an information-theoretic criterion to evaluate the correlation between two variables, so it is a valuable measure to train a deep network in the lack of annotation. We exploit this concept to maximize the dependency between input and output and encourage the model to hold maximum information between them. We show this criterion in the Figure \ref{fig:usn} between $X$ and $Y$.
The final loss function is a weighted average of the adversarial and mutual information terms.

\subsection{Method} This paper proposes a novel approach to find the proper mapping function called \textbf{In}formation \textbf{M}aximization and \textbf{A}dversarial \textbf{R}egularization \textbf{S}egmentation (\textbf{InMARS}) to segment input image into a predefined number of clusters without any label or supervision. To explain the pipeline, we start by providing further details about RWE and the variant of the extracted features. Then, as our clustering technique, we elaborate more on the Mutual Information Maximization objective. Finally, we present our customized adversarial training as a regularizer .

\textbf{Region Wise Embedding (RWE)}:
As mentioned earlier, in this stage, our goal is to produce candidate region/sub-regions to preserve spatial information to some extent and to generate smooth label predictions for each sub-region while generating a reliable feature from this candidate for the next stage. Moreover, we need to ensure that this RWE module can extract a fixed-length feature representation from each superpixel with arbitrary sizes before inputting them into the clustering objective. To this end, we evaluate \textcolor{black}}{{three} different strategies to generate the fixed-length embedding from variable-size inputs as follow: (1) Sub-region Pooling, (2) Spatial Pyramid Pooling (SPP)~\cite{SPP}, and (3) Interpolation.\\ \textcolor{black}}{{In the sub-region pooling strategy, we produce a single embedding by applying max/average-pooling to each superpixel separately. SPP is a fixed-length map that tunes 2D max-pooling parameters based on input size in multiple scales to generate a fixed size output for any input size's region. We utilize a zero mask in the SPP to zero out pixels' embedding that fall into SPP's rectangular region but do not belong to the selected superpixel. Then, we passed SPP's output from a series of fully connected (FC) layers to generate the final embedding. Interpolation is another strategy to build an equal size embedding for all candidates. For this, we resize superpixel regions into a predefined size. Then, similar to SPP, we apply a zero mask and feed a flattened version of these equal-sized regions to the fully connected layers.} Their effect on the performance will be studied in the ablation part.

\textbf{Unsupervised Segmentation by Maximizing Mutual Information and Adversarial Training}: 
Let $Y_i\in \mathbb{B}^{(w\times h)}$ be $i^{th}$ channel of $Y$ as a discrete representation in the network's output where $1\leq i \leq k$ and $k$ is the maximum number of the semantic labels/clusters. Also, let $x \sim X$ and $y_i \sim Y_i$ be random variables for the input and output, respectively. Given $x$ and considering each label as a conditionally independent variable, we can formulate the joint distribution of the output by $p_{\theta}\left(y_{1},\ldots, y_{k} \mid x\right)=\prod_{i=1}^{k} p_{\theta}\left(y_{i} \mid x\right)$, where $\theta$ is the parameters of the model, \emph{i.e.}\xspace deep neural network. During training, mutual information between input $X$ and output $Y$ is maximized using marginal entropy and conditional entropy, while the training is regularized via adversarial training. To this end, we have the following objective to minimize:

\vspace*{-0.75em}
\begin{equation} \label{eq:mainLoss}
   
    \textcolor{black}}{{R(\theta)}-\lambda I(X;Y)
\end{equation}
\vspace*{-1.0em}

Where \textcolor{black}}{{$R(\theta)$} is the regularization term that enforces the deep network to produce informative and reliable features. $I(X; Y )$ is the mutual information between input and embedding of a network, and $\lambda$ is a trade-off between two terms.

\underline{\emph{Mutual Information Objective}}: 
There are two ways to deal with the data clustering problem using mutual information maximization (MIM), 1) $MIM(X, Y)$ between the inputs $X$ and the outputs $Y$ and 2) $MIM(Y,Y^{\prime})$ between the output $Y$ and the prediction of an augmented version of the input $\hat{Y}$. In both approaches, the goal is to train a network that maps similar inputs into similar representations~\cite{IMSAT,IIC}. The $MIM(Y,Y^{\prime})$ requires a pair of the original input and augmented version, yet the $MIM(X, Y)$ does not depend on any additional pairs and can be readily computed from the network's output. Following~\cite{IMSAT}, we selected $MIM(X, Y)$ strategy as our strategy. In terms of entropy, mutual information can be described as $I(X; Y)=H(Y)-H(Y\mid X)$, where $H(Y)$ and $H(Y\mid X)$ are entropy and conditional entropy, respectively, defined by:

\vspace*{-0.75em}
\begin{equation} \label{eq:8}
H(Y) \equiv h(p_{\theta}(y)) = h\left(\frac{1}{N} \sum_{j=1}^{N} p_{\theta}\left(y \mid x_{j}\right)\right)
\end{equation}
\vspace*{-0.75em}
\begin{equation} \label{eq:9}
H(Y \mid X)=\frac{1}{N} \sum_{j=1}^{N} h\big(p_{\theta}\left(y \mid x_{j}\right)\big)
\end{equation}
\vspace*{-1.0em}

where $h(.)$ is the entropy function and $x_j$ is each training data input. \textcolor{black}}{{According to \cite{Gomes_RIM}, we can incorporate prior knowledge, \emph{e.g.}\xspace a distribution over the sample size in each class in the case of a class imbalance in the data, by rewriting $H(Y)$ as $log(K) - KL[p_{\theta}(y)|| q(y)]$ where $K$ is the number of clusters, $q(y)$ is the prior distribution and $KL[.||.]$ is the Kullback-Leibler divergence~\cite{KLD}. In our experiments for simplicity, we assume the prior $q(y)$ is a uniform distribution.} \par
To better understand $MIM$ as an objective function, we could evaluate marginal entropy $H(Y)$ and conditional entropy $H(Y\mid X)$. The marginal entropy pushes cluster assignment to be uniform (distributing data to all clusters) while the conditional entropy forces cluster assignment to be explicit (push the same data to a single cluster).\par

\underline{\emph{Adversarial Training}}: To augment each data sample with adversarial training,  an additional regularization term $R(\theta)$ is incorporated into the loss function:

\vspace*{-0.75em}
\begin{equation} \label{eq:adv}
\textcolor{black}}{{R(\theta ;\ T)}= KL\left(p_{\theta}\left(y_{i} \mid x\right), p_{\theta}\left(y_{i} \mid T(x)\right)\right)
\end{equation}
\vspace*{-1.0em}

where $KL,~p_{\theta}\left(y_{i} \mid x\right)$ and $p_{\theta}\left(y_{i} \mid T(x)\right)$ are KL divergence, the network's embedding for the original data $x$, and the transformed (augmented) data $T(x)$, respectively.\par
This regularization objective penalizes embedding dissimilarity between data and augmented versions and enforces the consistency of embedding on the network's output. We select pixel noise to increase the invariance of the extracted features against perturbations \textcolor{black}}{{$r$ as $T(x) = x + r$.} Then, we regularize the network against adversarial samples while keeping the decision boundary in the low-density region of data distribution by confining adversarial perturbations. The confined adversarial samples are mined according to the following criteria.

\vspace*{-0.75em}
\begin{equation} \label{eq:t}
r=\underset{r^{'}}{\operatorname{argmax}}\left\{R_{a d v}(\theta ;\ x, T(x)) ; \|r^{'}\|_{2} \leq \varepsilon\right\}
\end{equation}
\vspace*{-1.0em}

Where $r$ is the additive noise \textcolor{black}}{{and $\varepsilon$ is a hyper-parameter that controls the range of the local perturbation}. We can approximate adversarial term $r$ within the training process by an additional forward/backward pass. \textcolor{black}}{{To this end, we called $R_{a d v}$ as adversarial form of $R$ when we compute $T(x)$ in adversarial fashion.} In addition, we also \textcolor{black}}{{add} random affine perturbations to include geometrical invariance to the network as in \cite{IMSAT}.

\underline{\emph{Training Objective}}: Combining adversarial and mutual information terms result in the final objective function. Recall that our main objective (Equation \ref{eq:mainLoss}) consist of two terms $R_{a d v}(\theta)$ and $I(X;Y)$. Therefore, we can write ${\mathcal{L}}_{R}$ and ${\mathcal{L}}_{MI}$ for these terms as follow:
\vspace*{-0.5em}
\begin{equation} \label{eq:loss}
\begin{split}
    \textcolor{black}}{{
            \mathcal{L}_{R}} & \textcolor{black}}{{= \frac{1}{2} R_{a d v}+\frac{1}{2} R_{g e o}
            } \\
    {\mathcal{L}}_{MI} & = H(Y\mid X) - \gamma H(Y)\\
    {\mathcal{L}}_{total} & = \textcolor{black}}{{\mathcal{L}_{R}} + \lambda {\mathcal{L}}_{MI}
\end{split}
\end{equation}

where \textcolor{black}}{{$\mathcal{L}_{R}$} and ${\mathcal{L}}_{MI}$ are \textcolor{black}}{{regularization} loss and mutual information loss, respectively. \textcolor{black}}{{$\lambda$ and $\gamma$ are the hyper-parameters. We fix the $\lambda$ during training but decrease the $\gamma$ gradually (see Sec. IV - Hyper-parameters, for more details). The higher value of $\gamma$ at initial stages helps to uniformly generate class in the output while the decay of this coefficient gradually assigns more importance on the explicit class generation in the final stages.} $R_{a d v}$ and \textcolor{black}}{{$R_{g e o}$} are adversarial additive pixel noise and \textcolor{black}}{{random affine perturbation, respectively. For $R_{g e o}$ we use random scaling, translation, rotation and shearing similar to \cite{IMSAT}.} \textcolor{black}}{{All hyper-parameters are tuned in a non-overlapping subset with the train and validation sets}, and the computed parameters are used for the final training and evaluation. Minimizing loss function ${\mathcal{L}_{total}}$ is equivalent to minimizing \textcolor{black}}{{$R$} and maximizing $I(X;Y)$, simultaneously.\par

\section{Experiments} \label{sec:Ex}
\textbf{Dataset}: Ji \emph{et al.}\xspace~\cite{IIC} has released two benchmark from COCO-Stuff~\cite{Cocostuff} and Potsdam~\cite{Potsdam} datasets, which has become the most popular datasets for evaluating the existing unsupervised image segmentation methods. Potsdam dataset consists of 8550 RGBIR $200\times200$ pixel satellite images with 3150 unlabelled samples. The first subset (Potsdam) has six classes (roads and cars, vegetation and trees, buildings, and clutter), while the second one (Potsdam-3) has three classes. Potsdam-3 is created by merging six classes of the first subset into three \textcolor{black}}{{coarser} classes. COCO-Stuff~\cite{Cocostuff} is another challenging dataset that contains stuff classes. The first subset of COCO-Stuff has 15-\textcolor{black}}{{coarse} classes with 52K images. These images at least contain $75\%$ of stuff pixels. The second subset (COCO-Stuff-3) is generated by taking 36K images from the first subset with the sky, ground, and plant classes.\par

\textbf{Implementation Details}: We conducted all experiments with randomly initialized networks. We used Hourglass~\cite{hourglass} as the main backbone of our segmentation network; however, this backbone can be replaced by any other fully convolutional networks, \emph{e.g.}\xspace the UNet~\cite{UNet}, as shown in our ablation study (Table~\ref{AS-backbone:long}). Hourglass or “stacked hourglass” network is built on the successive steps of pooling and upsampling done to produce a final set of predictions. Each hourglass module is closely connected to fully convolutional networks, and the connection of modules could process spatial information at multiple scales for dense prediction~\cite{hourglass}. We used two stacks of the hourglass with two residual blocks in each hourglass as our backbone. Then, the RWE head is employed on the top of the backbone to map pixels' embedding to regions. In RWE, we picked SLIC Superpixel \cite{SLIC} with parameters so that each candidate region conveys meaningful spatial information and has self-determining features. In fact, the form and spatial size of the ground truth classes can be important to choose proper superpixel parameters. Therefore, the parameters are set such that \emph{(i)} the number of superpixels is not less than the number of semantic classes; \emph{(ii)} Since under a general superpixel parameters' setup, there is no guarantee that each generated superpixel region carries the pixels with the same semantic labels, over-segmentation is essential in our framework. By over-segmentation, our framework can merge the superpixels with the same semantic classes. We implemented the RWE module with three different approaches:~(1) sub-region pooling, (2) SPP, and (3) interpolation. In the ablation part, we studied the impact of each approach on the network's performance (Table~\ref{AS-RWE:long}). In our final result, we chose SPP as the best module for the RWE module.

Our proposed method (InMARS) has three variants, namely InMARS$^{b}$, InMARS and InMARS$^{+}$. InMARS$^{b}$ is the argmaxed outputs from the backbone component of our model during inference (Fig.\xspace~\ref{fig:usn}). Note that in our implementations, we ensured that the number of the backbone's output channels is equal to the number of ground-truth classes. InMARS is the inference output of our proposed model (Fig.\xspace~\ref{fig:usn}) including RWE module, trained with $2*N_{gt}$ superpixels. InMARS$^{+}$ is another variant (multi-scale) of InMARS, incorporating all three superpixel resolutions (Table~\ref{AS-SP:long}) during inference to generate multiple outputs for each pixel. Then, the final prediction is formed by pixel-wise averaging of the three embeddings. We compared the performance of our method against \cite{ IIC, ARL,Pedregosa2011scikit-learn, Lowe:2004:DIF:993451.996342, Doersch, isola2015learning, DeepCluster}. We also reported the results for a supervised baseline, trained using the UNet~\cite{UNet}, as the best performing backbone from the possible backbones we used in our experiment.\par

\begin{figure*}[htb]
    \centering
    \begin{subfigure}[t]{0.125\textwidth}
      \includegraphics[width=1\textwidth]{images/coco-st3-97-000000129656/img-in.png}
    \end{subfigure}
    \begin{subfigure}[t]{0.125\textwidth}
      \includegraphics[width=1\textwidth]{images/coco-st3-97-000000129656/iic.png}
    \end{subfigure}
    \begin{subfigure}[t]{0.125\textwidth}
      \includegraphics[width=1\textwidth]{images/coco-st3-97-000000129656/sp.png}
    \end{subfigure}
    \begin{subfigure}[t]{0.125\textwidth}
      \includegraphics[width=1\textwidth]{images/coco-st3-97-000000129656/img-emb.png}
     \end{subfigure}
    \begin{subfigure}[t]{0.125\textwidth}
      \includegraphics[width=1\textwidth]{images/coco-st3-97-000000129656/img-sp.png}
    \end{subfigure}
    \begin{subfigure}[t]{0.125\textwidth}
      \includegraphics[width=1\textwidth]{images/coco-st3-97-000000129656/sup.png}
    \end{subfigure} 
    \begin{subfigure}[t]{0.125\textwidth}
      \includegraphics[width=1\textwidth]{images/coco-st3-97-000000129656/img-gt.png}
    \end{subfigure}
    
    \par\smallskip
    \centering
    \begin{subfigure}[t]{0.125\textwidth}
      \includegraphics[width=1\textwidth]{images/coco-st3-94-000000320508/img-in.png}
    \end{subfigure}
    \begin{subfigure}[t]{0.125\textwidth}
      \includegraphics[width=1\textwidth]{images/coco-st3-94-000000320508/iic.png}
    \end{subfigure}    
    \begin{subfigure}[t]{0.125\textwidth}
      \includegraphics[width=1\textwidth]{images/coco-st3-94-000000320508/sp.png}
    \end{subfigure}    
    \begin{subfigure}[t]{0.125\textwidth}
      \includegraphics[width=1\textwidth]{images/coco-st3-94-000000320508/img-emb.png}
     \end{subfigure}
    \begin{subfigure}[t]{0.125\textwidth}
      \includegraphics[width=1\textwidth]{images/coco-st3-94-000000320508/img-sp.png}
    \end{subfigure}
    \begin{subfigure}[t]{0.125\textwidth}
      \includegraphics[width=1\textwidth]{images/coco-st3-94-000000320508/sup.png}
    \end{subfigure}
    \begin{subfigure}[t]{0.125\textwidth}
      \includegraphics[width=1\textwidth]{images/coco-st3-94-000000320508/img-gt.png}
    \end{subfigure}

    \par\smallskip
    \centering
    \begin{subfigure}[t]{0.125\textwidth}
      \includegraphics[width=1\textwidth]{images/91-000000437239/img-in.png}
    \end{subfigure}
    \begin{subfigure}[t]{0.125\textwidth}
      \includegraphics[width=1\textwidth]{images/91-000000437239/iic.png}
    \end{subfigure}
    \begin{subfigure}[t]{0.125\textwidth}
      \includegraphics[width=1\textwidth]{images/91-000000437239/sp.png}
    \end{subfigure}
    \begin{subfigure}[t]{0.125\textwidth}
      \includegraphics[width=1\textwidth]{images/91-000000437239/img-emb.png}
     \end{subfigure}
    \begin{subfigure}[t]{0.125\textwidth}
      \includegraphics[width=1\textwidth]{images/91-000000437239/img-sp.png}
    \end{subfigure}
    \begin{subfigure}[t]{0.125\textwidth}
      \includegraphics[width=1\textwidth]{images/91-000000437239/sup.png}
    \end{subfigure}
    \begin{subfigure}[t]{0.125\textwidth}
      \includegraphics[width=1\textwidth]{images/91-000000437239/img-gt.png}
    \end{subfigure}   
    
    \par\smallskip
    \centering
    \begin{subfigure}[t]{0.125\textwidth}
      \includegraphics[width=1\textwidth]{images/85-000000350023/img-in.png}
    \end{subfigure}
    \begin{subfigure}[t]{0.125\textwidth}
      \includegraphics[width=1\textwidth]{images/85-000000350023/iic.png}
    \end{subfigure}
    \begin{subfigure}[t]{0.125\textwidth}
      \includegraphics[width=1\textwidth]{images/85-000000350023/sp.png}
    \end{subfigure}
    \begin{subfigure}[t]{0.125\textwidth}
      \includegraphics[width=1\textwidth]{images/85-000000350023/img-emb.png}
     \end{subfigure}
    \begin{subfigure}[t]{0.125\textwidth}
      \includegraphics[width=1\textwidth]{images/85-000000350023/img-sp.png}
    \end{subfigure}
    \begin{subfigure}[t]{0.125\textwidth}
      \includegraphics[width=1\textwidth]{images/85-000000350023/sup.png}
    \end{subfigure}
    \begin{subfigure}[t]{0.125\textwidth}
      \includegraphics[width=1\textwidth]{images/85-000000350023/img-gt.png}
    \end{subfigure} 
    
    \par\smallskip
    \centering
    \begin{subfigure}[t]{0.125\textwidth}
      \includegraphics[width=1\textwidth]{images/91-000000139871/img-in.png}
      \subcaption{Input}
    \end{subfigure}
    \begin{subfigure}[t]{0.125\textwidth}
      \includegraphics[width=1\textwidth]{images/91-000000139871/iic.png}
      \subcaption{IIC\cite{IIC}}
    \end{subfigure}
    \begin{subfigure}[t]{0.125\textwidth}
      \includegraphics[width=1\textwidth]{images/91-000000139871/sp.png}
      \subcaption{Superpixel}
    \end{subfigure}
    \begin{subfigure}[t]{0.125\textwidth}
      \includegraphics[width=1\textwidth]{images/91-000000139871/img-emb.png}
      \subcaption{InMARS$^{b}$}
     \end{subfigure}
    \begin{subfigure}[t]{0.125\textwidth}
      \includegraphics[width=1\textwidth]{images/91-000000139871/img-sp.png}
      \subcaption{InMARS$^{+}$}
    \end{subfigure}
    \begin{subfigure}[t]{0.125\textwidth}
      \includegraphics[width=1\textwidth]{images/91-000000139871/sup.png}
      \subcaption{Supervised}
    \end{subfigure}    
    \begin{subfigure}[t]{0.125\textwidth}
      \includegraphics[width=1\textwidth]{images/91-000000139871/img-gt.png}
      \subcaption{GT}
    \end{subfigure}
    
    \caption{\textbf{Segmentation Results from COCO-Stuff-3 \cite{Cocostuff}}. (a) Few input images examples, (b) The IIC~\cite{IIC} outputs. The figures (c)-(e) represent the component of our model, \emph{i.e.}\xspace (c) Superpixels in the finest resolution (\emph{i.e.}\xspace $N_{gt}^{2}$), (d) The argmaxed version of the our backbone outputs ( InMARS$^{b}$) (e) The outputs of the multi-scale variant of our model (InMARS$^{+}$). The outputs of the supervised baseline and the Ground-truths are shown in (f) and (g), respectively.}
    \vspace{-1em}
    \label{fig:vis:coco-st3}
  \end{figure*}

\textbf{Hyper-parameters}: All the hyper-parameters in our experiments were tuned in a non-overlapping subset of the dataset (\%10 randomly selected), and they were fixed during optimization and evaluation on the dataset on which we evaluated and reported the final results. We selected Adam optimizer~\cite{Adam} with the initial learning rate $lr_{init}=10^{-5}$ and exponential decays for the learning rate by $\alpha=0.96$ in every epoch to do all the experiments. The learning rate in every epoch is derived by $lr_{i} = \alpha^{i}*lr_{init}$. In this equation $lr_{init}$ is the initial learning rate, $i$ is the numerator of epoch and $lr_{i}$ is the learning rate in the epoch $i$. We trained the network for 100 epochs and picked the last model for evaluation. We selected ($5$, $2*N_{gt}$) for compactness and number of SLIC superpixels during training. To speed up the training process, we extracted all superpixels once offline. In training, invalid pixels (pixels that do not belong to any of the classes) were discarded, and the loss function was calculated just on valid pixels. We followed the prior works for image size ($200\times200$ and $128\times128$ pixel images for Potsdam and COCO-Stuff, respectively. For hourglass, we resized the network's inputs from $200\times200$ to $192\times192$ pixels, as with depth equal to 4, the input should be divisible by 16 ($2^{depth}$). We used PyTorch library~\cite{PyTorch} for implementation and NVidia Titan X and GTX 1080ti GPUs for training. The hyper-parameters $\lambda$, $\gamma$ and $\varepsilon$ are set to $4$, $1$ and $.25$, respectively. We fixed the $\lambda$ and $\varepsilon$ during training but decreased the $\gamma$ from $1$ to $0.1$.\par

\subsection{Evaluation}
As in IIC~\cite{IIC} and AC~\cite{ARL}, we used the same evaluation measure (pixel accuracy) to report the performance of our method, \emph{i.e.}\xspace $\label{eq:ACC}
ACC=\max _{m} \frac{\sum_{n=1}^{N} 1\left\{l_{n}=m\left(c_{n}\right)\right\}}{N}$
where $l_n$ and $c_n$ are the ground-truth label and cluster assignment for input $x_n$, respectively. $m$ ranges over all possible one-to-one mapping between cluster assignments and ground-truth classes. The maximum operator is effectively computed using linear assignment~\cite{Kuhn1955Hungarian} to find the best one-to-one mapping between the predicted classes and ground-truth labels. In both the assignment and evaluation step, we discarded invalid pixels as in visualization (Figure \ref{fig:vis:coco-st3}).

Table \ref{table-acc:long}. demonstrates the superior performance of our InMARS$^{+}$ compared to the state-of-the-art methods in the three datasets, validating the efficacy of our suggested modules for this task, \emph{i.e.}\xspace unsupervised segmentation.

\begin{table}
\begin{center}
\scriptsize

\caption{\textbf{Accuracy of Unsupervised Image Segmentation}}
\label{table-acc:long}
\label{table-acc:onecol}
\begin{tabular}{|l|c|c|c|c|}
\hline
& COCO & COCO & Potsdam-3 & Potsdam\\
& Stuff-3 & Stuff & & \\
\hline\hline
\textcolor{black}}{{Supervised Baseline} & \textcolor{black}}{{84.7} & \textcolor{black}}{{89.7} & \textcolor{black}}{{80.2} & \textcolor{black}}{{82.4} \\
\hdashline
K-means \cite{Pedregosa2011scikit-learn}
& 52.2 & 14.1 & 45.7 & 35.3 \\
SIFT \cite{Lowe:2004:DIF:993451.996342}
& 38.1 & 20.2 & 38.2 & 28.5 \\
Doersch 2015 \cite{Doersch}
& 47.5 & 23.1 & 49.6 & 37.2 \\
Isola 2016 \cite{isola2015learning}
& 54.0 & 24.3 & 63.9 & 44.9 \\
Deep Cluster 2018 \cite{DeepCluster}
& 41.6 & 19.9 & 41.7 & 29.2 \\
IIC 2019 \cite{IIC}
& 72.3 & 27.7 & 65.1 & 45.4 \\
AC 2020 \cite{ARL}
& \underline{72.9} & \underline{30.8} & 66.5 & \textbf{49.3} \\
\hline
\textbf{InMARS$^{b}$} \textit{(Ours)}
& 72.2 & 29.2 & 68.5 & 46.2 \\
\textbf{InMARS} \textit{(Ours)}
& 72.8 & 30.3 &  \underline{68.8} & 46.9 \\
\textbf{InMARS$^{+}$} \textit{(Ours)}
&\textbf{73.1} &\textbf{31.0} &\textbf{70.1} & \underline{47.3} \\
\hline

\end{tabular}
\footnotesize{\\\textsuperscript{b} InMARS backbone with $N_{gt}$ output classes.}
\footnotesize{\\\textsuperscript{+} InMARS with multi-scale test by different number of superpixels.}
\end{center}
\end{table}
\par

\textbf{Qualitative Result}: Figure \ref{fig:vis:coco-st3}. illustrates the output of InMARS (output of the backbone) and InMARS$^{+}$ (output of the RWE) on the COCO-Stuff-3 dataset. The RWE module has spatial smoothness ability and performs well on the top of the backbone to produce a continuous segmentation mask while preserving semantic information. The superpixels with the finest resolution ($N_{gt}^2$) are displayed in Fig.\xspace~3-(c). In addition, we included the output of the fully supervised UNet~\cite{UNet} and IIC~\cite{IIC} as two baselines for visual comparison. Note that similar to~\cite{IIC, UNet} in all the results, invalid pixels (pixels that do not belong to any of the classes) were discarded and are shown with dark colors. In Figure \ref{fig:failure:coco-st3} some failure cases presented. The first row is an ambiguous sample of the ground class (gray color in the ground-truth). Our method failed to distinguish this sample (artificial turf) from natural class. In fact, this level of perception requires a higher level of knowledge which is only available in the supervised objective. The second row shows a sample with a cluttered background. Our method failed to understand the natural class since it cannot understand this level of clutter.

\begin{figure}[!tb]
    \centering
    \begin{subfigure}[t]{0.11\textwidth}
      \includegraphics[width=\textwidth]{images/Failure Case/4-000000509459/img-in.png}
    \end{subfigure}
    \begin{subfigure}[t]{0.11\textwidth}
      \includegraphics[width=\textwidth]{images/Failure Case/4-000000509459/img-emb.png}
     \end{subfigure}
    \begin{subfigure}[t]{0.11\textwidth}
      \includegraphics[width=\textwidth]{images/Failure Case/4-000000509459/img-sp.png}
    \end{subfigure}
    \begin{subfigure}[t]{0.11\textwidth}
      \includegraphics[width=\textwidth]{images/Failure Case/4-000000509459/img-gt.png}
    \end{subfigure}
    
    \par\smallskip
    \centering
    \begin{subfigure}[t]{0.11\textwidth}
      \includegraphics[width=\textwidth]{images/Failure Case/8-000000291962/img-in.png}
      \subcaption{Input}
    \end{subfigure}
    \begin{subfigure}[t]{0.11\textwidth}
      \includegraphics[width=\textwidth]{images/Failure Case/8-000000291962/img-emb.png}
      \subcaption{InMARS$^{b}$}
     \end{subfigure}
    \begin{subfigure}[t]{0.11\textwidth}
      \includegraphics[width=\textwidth]{images/Failure Case/8-000000291962/img-sp.png}
      \subcaption{InMARS$^{+}$}      
    \end{subfigure}
    \begin{subfigure}[t]{0.11\textwidth}
      \includegraphics[width=\textwidth]{images/Failure Case/8-000000291962/img-gt.png}
      \subcaption{GT}
    \end{subfigure}
    \caption{\textbf{Failure Cases}. Some failure cases of the proposed method from COCO-Stuff-3 \cite{Cocostuff}. The proposed algorithm fails to recognize some classes due to the similarity of classes (first row) and clutter in the background (second row)}
    \label{fig:failure:coco-st3}
\end{figure}

\begin{figure}[!tb]
    
    \centering
    \crule[colorSky]{.15cm}{.15cm}
    \small{Sky}
    \crule[colorPlants]{.15cm}{.15cm}
    \small{Plants}
    \crule[colorGround]{.15cm}{.15cm}
    \small{Ground}
    
    \vspace*{0in}
    \begin{subfigure}[t]{0.115\textwidth}
    
     \includegraphics[width=\textwidth, trim={0.25cm 0.25cm 0 0},clip]{TSNE/cluster-tSNE-0.0.pdf}
    \vspace*{-\baselineskip}
    \end{subfigure}
    \begin{subfigure}[t]{0.115\textwidth}
      \includegraphics[width=\textwidth, trim={0.25cm 0.25cm 0 0},clip]{TSNE/cluster-tSNE-5p.pdf}
    \vspace*{-\baselineskip}
    \end{subfigure}
    \begin{subfigure}[t]{0.115\textwidth}
      \includegraphics[width=\textwidth, trim={0.25cm 0.25cm 0 0},clip]{TSNE/cluster-tSNE-0.25.pdf}
    \vspace*{-\baselineskip}
    \end{subfigure}
    \begin{subfigure}[t]{0.115\textwidth}
      \includegraphics[width=\textwidth, trim={0.25cm 0.25cm 0 0},clip]{TSNE/cluster-tSNE-0.5.pdf}
    \vspace*{-\baselineskip}
    \end{subfigure}    
    \caption{\textbf{TSNE Visualization}. TSNE plot for the superpixel embedding on a subset of COCO-Stuff dataset using our proposed method (InMARS$^{+}$). From left to right, before training with random weights, after training, and trained model for images with $IoU > .25$ and $IoU > .5$, respectively.}
    \label{fig:tsne:coco-st3}
  \end{figure}

\textbf{Cluster Visualization}: Figure \ref{fig:tsne:coco-st3} illustrated the TSNE visualization for superpixels' embedding in the output of the network. It is worth to mention that TSNE is a 2D visualization that map high-dimensional data to lower dimension. This mapping cannot reflect the high dimensional separation capability of the method solely. However, we tried to highlight the network's performance by reducing the image with low average Intersection-over-Union (IoU) with ground-truth. Figure \ref{fig:tsne:coco-st3} (a) is the network's output at the initial stage, before training. As we expected, samples fall in a random pattern. Figure \ref{fig:tsne:coco-st3} (b) is the network's output after training. Figure \ref{fig:tsne:coco-st3} (c) and (d) are obtained by including image with IoU more than .25\% and .50\%, respectively. The TSNE of the trained model in Figure \ref{fig:tsne:coco-st3} (c) and (d) confirm that InMARS$^{+}$ can separate classes in feature space properly.

\subsection{Ablation Study}
In this part, we assessed the role of each component in the proposed method. Table \ref{AS-backbone:long} - Table \ref{AS-SP:long} present the ablation study of different components conducted on 10\% of randomly selected images of datasets.\par


\begin{table}[!tb]
\centering
\scriptsize
\vspace{-0.5em}
\captionof{table}{\textbf{Backbone}.We trained different backbone architectures with the same loss function and RWE parameters in the reduced version of COCO-Stuff-3 and Potsdam-3 datasets to compare their effects on the proposed method.}
\label{AS-backbone:long}
\label{AS-backbone:onecol}
\begin{tabular}{|l|c|c|}
\hline
& COCO-Stuff-3 & Potsdam-3\\
\hline\hline
UNet \cite{UNet} & 52.5 & 50.2 \\
Hourglass \cite{hourglass} & \textbf{54.5} & \textbf{51.4} \\
\hline
\end{tabular}
\end{table}


\begin{table}[!tb]

\centering
\scriptsize
\captionof{table}{\textbf{Region-Wise Embedding \textit{(RWE)}}. We tested three configurations for the RWE head on the top of the backbone to build a fixed-length embedding from superpixels. Max and average pooling are two kinds of sub-region pooling. SPP is spatial pyramid pooling, and FC stands for fully connected layers. The heads with the FC layer perform better since they can model complex tasks.}
\label{AS-RWE:long}
\label{AS-RWE:onecol}
\begin{tabular}{|l|c|c|}
\hline
& COCO-Stuff-3 & Potsdam-3\\
\hline\hline
max-pooling & 54.5 & 51.4 \\
mean-pooling & 50.1 & 51.0 \\
SPP~\cite{SPP}+FC & 57.3 & \textbf{55.5} \\
Interpolation+FC & \textbf{59.8} & 55.4 \\
\hline
\end{tabular}

\end{table}

\begin{table}[!tb]

\centering
\scriptsize
\textcolor{black}}{{\captionof{table}{\textbf{Adversarial Regularization}. we illustrated the accuracy of the trained network \textcolor{black}}{{without any regularization ($no-adversarial$)}, with \textcolor{black}}{{adversarial} additive noise \textcolor{black}}{{($R_{a d v}$)}, \textcolor{black}}{{random geometrical} affine perturbation \textcolor{black}}{{($R_{g e o}$)} and, their combination. The backbone is hourglass and \textcolor{black}}{{max-pooling} module selected for RWE part.}
\label{AS-ADV:onecol}
\label{AS-ADV:long}
\begin{tabular}{|l|c|c|}
\hline
& COCO-Stuff-3 & Potsdam-3\\
\hline\hline
$no-adversarial$& 44.2 & 42.3 \\
\textcolor{black}}{{$R_{a d v}$}& 54.5 & 51.4 \\
\textcolor{black}}{{$R_{g e o}$}& 61.2 & 59.1 \\
\textcolor{black}}{{$\frac{1}{2} R_{a d v}+\frac{1}{2} R_{g e o}$} & \textbf{61.6} & \textbf{60.4} \\
\hline
\end{tabular}}
\end{table}


\begin{table}[!tb]
\centering
\scriptsize
\captionof{table}{\textbf{Superpixel}. we inspected the effect of number of superpixel. $(N_{gt})$ is the number of ground-truth classes.}
\label{AS-SP:long}
\begin{tabular}{|l|c|c|}
\hline
(number of superpixels) & COCO-Stuff-3 & Potsdam-3\\
\hline\hline
$N_{gt}$ & 54.5 & 51.4 \\
$2*N_{gt}$ & \textbf{55.3} & 52.1 \\
$N_{gt}^{2}$ & 55.2 & \textbf{52.6} \\
\hline
\end{tabular}
\end{table}

\textbf{Backbone}: The backbone is a fully convolutional deep network, like FCN \cite{FCN} which gets the input image and generates an embedding map of equal size to the input image. In Table~\ref{AS-backbone:long}, we compared the effect of choosing different backbone architectures on the accuracy of the proposed method in the Potsdam-3 and COCO-Stuff-3 dataset. We chose UNet \cite{UNet} and Hourglass \cite{hourglass} backbone in our experiments, while other semantic segmentation backbones can be used instead. The results depicted that the hourglass backbone is performed best among others.\par

\textbf{RWE}: 
We designed three different ways to generate fixed-length embedding for superpixels: (1) Sub-region pooling, (2) SPP, and (3) interpolation. 
In Table~\ref{AS-RWE:long}, we reported the result of experiments with different RWE modules. In the InMARS, we chose SPP+FC for our final model.\par

\textbf{Adversarial Regularization}: The regularization term plays a remarkable role in the performance of the proposed method. Table~\ref{AS-ADV:long}. compares two \textcolor{black}}{{different} strategies for regularization: \textcolor{black}}{{adversarial} additive noise and \textcolor{black}}{{random geometrical} affine perturbation (similar to~\cite{IMSAT}). The best accuracy achieved by a combination of two terms according to the Equation $\textcolor{black}}{{\frac{1}{2} R_{a d v}+\frac{1}{2} R_{g e o}}$.
  
\textbf{Superpixel}: Table \ref{AS-SP:long}. shows various parameter selection for superpixel candidates. The extent and size of target classes define the approximate number of superpixels. The number of superpixels is selected regarding the number of ground-truth classes $(N_{gt})$. Since superpixel is not an ideal proposal, over-segmentation performs better. Finally, we selected combination of $N_{gt}$, $2N_{gt}$ and $N_{gt}^{2}$ in InMARS$^{+}$.\par




\section{Conclusion}
We have proposed a fully unsupervised semantic segmentation network, which utilizes the mutual information as the supervision and adversarial training as a regularization. The proposed method leads to an unsupervised semantic segmentation model that can partition the scene into semantic classes while generating good boundaries in the output by exploiting the off-the-shelf superpixel method. We further demonstrate the state-of-the-art results on benchmarks comparable with supervised methods and wish to reduce the gap with these methods. In the future, improvement of the method to handle more semantic classes in complex datasets would be fruitful.




















\ifCLASSOPTIONcaptionsoff
  \newpage
\fi





\bibliographystyle{IEEEtran}
