\section{Introduction}
\noindent
Unsupervised Salient Object Detection (USOD) aims to find the most conspicuous objects in images without manual annotation.
USOD methods can be utilized as pre-process for other computer vision tasks, such as object recognition \cite{rec1,rec2}, object detection \cite{det1,det2}, and semantic segmentation \cite{seg1,seg2}.
To produce more precise object masks, USOD methods are expected to correctly distinguish salient objects as well as their boundaries.
However, natural images usually contain diverse objects, complex background, different lighting conditions, and other challenging conditions, which bring severe challenges to detect salient objects in an unsupervised manner.

\begin{figure}[!t]
\centering
\includegraphics[width=0.47 \textwidth]{image/compare.pdf}\\
\caption{Performance comparison between SOD methods. The circle and triangle indicate training on MSRA-B and DUTS-TR datasets, respectively.}
\label{fig:compare}
\end{figure}


Recently, USOD methods have demonstrated impressive performance owing to the development of Deep-Learning techniques \cite{resnet,cityscape,imagenet}, as shown in Fig. \ref{fig:compare}.
There are, however, mainly two limitations in these methods.
First, unsupervised saliency modeling in images is not best served by current learning strategies.
Specifically, most Deep-Learning-based (DL-based) USOD methods \cite{sbf,mnl,usps,dcfd} learn image saliency based on noisy prior knowledge provided by several traditional SOD methods \cite{hs,dsr,rbd,mc}.
For example, USPS \cite{usps} uses the saliency cues extracted by traditional SOD methods to train deep networks in a fully-supervised manner.
However, traditional SOD methods typically misidentify some background regions with distinctive colors as saliency regions, as shown in Fig. \ref{fig:pseudo}(c-d), because they model the image saliency using low-level features, such as color vectors.
Using a simpler training strategy like USPS cannot effectively eliminate these regions (Fig. \ref{fig:pseudo}(e)), thus reducing the quality of pseudo labels.
A2S \cite{a2s} is able to eliminate these regions in the generated pseudo label (f-g) by modeling image saliency using high-level representations.
However, a few patches in the generated pseudo labels are incorrectly classified, suggesting that some detailed saliency information has not yet been thoroughly uncovered.
Second, there is still untapped potential for using appearance information to enhance segmentation performance.
Existing DL-based USOD methods only use appearance information as post-process \cite{densecrf} to refine the network predictions.
Although it refine the boundaries of target objects (Fig. \ref{fig:pseudo} (e-h)), but is powerless when facing localization error in predictions.
Instead, appearance information can be formulated as a supervision signal for deep networks to find more precise boundary of target objects during training.
In summary, solving the above two issues is a promising direction to improve the performance of DL-based USOD methods.

\begin{figure}[!t]
\centering
\begin{minipage}{0.49 \textwidth}
\subfloat[Image]{
\includegraphics[width=0.82in,height=0.6in]{maps/image/3_118_118210.png}}
\subfloat[GT]{
\includegraphics[width=0.82in,height=0.6in]{maps/gt/3_118_118210.png}}
\subfloat[RBD]{
\includegraphics[width=0.82in,height=0.6in]{maps/rbd/3_118_118210.png}}
\subfloat[HS]{
\includegraphics[width=0.82in,height=0.6in]{maps/hs/3_118_118210.png}}
\end{minipage}
\vspace{-0.1in}

\begin{minipage}{1 \textwidth}
\subfloat[USPS]{
\includegraphics[width=0.82in,height=0.6in]{maps/usps/3_118_118210.png}}
\subfloat[A2S-U]{
\includegraphics[width=0.82in,height=0.6in]{maps/a2s_moco/3_118_118210.png}}
\subfloat[A2S-S]{
\includegraphics[width=0.82in,height=0.6in]{maps/a2s_in/3_118_118210.png}}
\subfloat[Ours]{
\includegraphics[width=0.82in,height=0.6in]{maps/our/3_118_118210.png}}
\end{minipage}

\caption{Saliency maps generated by USOD methods. 
Two results of A2S are produced by using unsupervised (-U) and supervised (-S) encoders, respectively.}
\label{fig:pseudo}
\end{figure}

For solving the first issue, we propose an effective way to formulate saliency modeling problem in an unsupervised fashion, which can better take advantage of week saliency knowledge and gradually transfer them to more fine-level saliency labels.
Actually, a robust learning strategy that can learn from subsets of high-confidence data and possibly avoid the chaos derived from the large amount of ambiguous data under such an unsupervised setting tends to be critical. 
This is because salient regions are always concealed inside many easily confused image regions in practical cases.
Inspired by the self-paced learning (SPL) theory, we embed an Attentive Self-Paced Learning (ASPL) paradigm into our framework to alleviate the data ambiguity and guide a robust learning manner in complex scenarios. 
To gradually learn from the reliable training samples to the ambiguous ones, the original SPL model assigns a binary weight to each sample based on sample easiness and current training progress. 
Moreover, a sample regularizer is employed to learn more robust knowledge by increasing the sample diversity in learning.
In this paper, we further leverage the sample easiness to simulate a continuous attention weight for each sample.
Our weight generation process, like the original SPL, takes current training progress into account, which means that samples with the same easiness will acquire different weights at various stages of training. 
Our adaptive strategy leads to a more rational sample weighting scheme compared to the binary one as in SPL.

 

Concerning the second issue, we elucidate the natural relationship between object segmentation and pixel appearance information, and accordingly utilize the latter to facilitate low-level guidance for obtaining more precise object boundaries in the former problem.
Specifically, intra-object consistency and boundary distinctiveness are two general characteristics of object appearance in the color image.
This prior knowledge has been extensively studied in conventional non-learning segmentation methods \cite{hs,mc}.	
Many learning-based methods \cite{lsc,mfnet} with a weakened supervision signal also use such prior to localize the boundaries of target objects.
Specifically, by aligning the edges both in the color images and saliency predictions, they can produce high-quality boundary without using precise but expensive annotations.
Such strategy bridges the connection between the appearance information and the supervision signal for training deep networks, like LSC \cite{lsc}.
However, knowledge learned from boundary pixels may be cancelled by other non-boundary pixels, resulting in a degraded segmentation performance.
In our method, our framework can perceive coarse saliency regions by using the above ASPL paradigm that addresses the first issue.
Most edges inside objects or the background have already been well settled, and therefore can be ignored in the proposed appearance guidance. 
By eliminating the effect of these pixels, our framework imposes more attention on refining boundaries using local appearance information.
In addition, appearance information also can be interpreted as the feature vectors in other modalities, such as depth map, thermal image or optical flow.
Our framework is capable of improving salient detection accuracy by aggregating multi-modality data.

Based on the above discussion, we propose a novel appearance-guided attentive self-paced learning framework for Unsupervised Salient Object Detection (USOD).
By integrating the Self-Paced Learning (SPL) paradigm and the appearance guidance into a unified model, the proposed framework is capable of discovering the faithful and precise saliency patterns from the large number of ambiguous prior knowledge.
Specifically, we propose an Attentive Self-Paced Learning (ASPL) paradigm as a more effective alternative to traditional SPL.
Our ASPL introduces an adaptive weighting mechanism that assigns an attention factor for each sample conditioned on its current prediction and training progress.
Furthermore, we propose an Appearance Guidance Module (AGM) to align the edges in both color images and saliency predictions, resulting in more robust segmentations in real complex scenarios. 
Benefit from the appearance information in multi-modality data, the proposed framework generates more precise saliency predictions in an unsupervised manner. 
Experiments on RGB, RGB-D, RGB-T and video SOD benchmarks prove that our framework achieves state-of-the-art performance against existing USOD methods and is comparable to the latest supervised SOD methods.

The main contributions are summarized as follows:
\begin{enumerate}
\item We propose an appearance-guided attentive self-paced learning framework for Unsupervised Salient Object Detection (USOD) by integrating both self-paced learning (SPL) and appearance guidance into a unified framework.
\item We propose an attentive self-paced learning paradigm that employs an adaptive weighting mechanism for training from simple samples to more complex ones conditioned on their current predictions and training progress.
\item We propose an Appearance Guidance Module (AGM) that leverages local low-level information to align the edges in color images and saliency predictions.
\item Experiments on RGB, RGB-D, RGB-T and video SOD benchmarks prove that our framework achieves state-of-the-art performance against existing USOD methods and is comparable to the latest supervised SOD methods.
\end{enumerate}

\section{Related Works}
\subsection{RGB Salient Object Detection}
\textbf{Supervised Methods.}
Researchers have developed a large family of fully-supervised Salient Object Detection (SOD) algorithms \cite{picanet,cpd,afnet,brn,cag,page,pagrn,ucf} in the past few years.
Ronneberger et al. \cite{unet} proposed a U-shape structure that progressively upsamples and concatenates the smaller features to the larger ones.
Zhang et al. \cite{amulet} integrated multi-level features into a single feature, and used it to enhance the learned features.
Hou et al. \cite{dss} introduced a weighted aggregation strategy to produce more precise predictions by fusing multi-level outputs.
Qin et al. \cite{basnet} proposed a novel loss function to supervise the network from pixel-, patch- and image-level simultaneously.
Zhao et al. \cite{gate} developed a gate-based network to surpass the learned features in previous stages.

To ease the annotation burden, some researchers trained saliency detectors by using some low-cost annotations.
Zeng et al. \cite{msw} used multi-source annotations to produce pseudo labels for unlabeled images.
Piao et al. \cite{mfnet} employed several directive filters to synthesize more accurate saliency cues from multiple noisy pseudo labels.
Zhang et al. \cite{wssa} relabeled the DUTS-TR dataset \cite{duts} with scribbles and leveraged the edge detection task to tackle boundary localization.
Yu et al. \cite{lsc} proposed a local coherence loss to find precise boundary based on scribbles annotations.
They employed all appearance information in images as guidance for training deep networks, where edges inside object or background may distract the learned saliency.

Although these methods have achieved significant performance, they train saliency detectors using numerous human annotations, which are very expensive to collect.

\textbf{Unsupervised Methods.}
Existing DL-based USOD methods have proposed two pipelines to generate pseudo labels from images.
First, most USOD methods \cite{sbf,mnl,usps,edns,dcfd} focused on refining the coarse saliency cues extracted by several traditional SOD methods.
For example, 
Zhang et al. \cite{mnl} designed a noise modeling module to deal with noises in these saliency cues.
Nguyen et al. \cite{usps} incrementally refine these saliency cues by using the self-supervision technique.
Zhang et al. \cite{edns} proposed an encoder-decoder network to learn saliency information from these saliency cues.
Second, to prevent the localization errors caused by traditional SOD methods, Zhou et al. \cite{a2s} proposed a novel framework to extract high-quality saliency cues using a deep network.
Although the above methods have achieved remarkable performance, there are two limitations in these methods.
First, existing methods used a relatively simple training strategy to learned from all pixels in images, where reliable pixels and ambiguous pixels are coexisting.
Second, low-level information, which is crucial for improving the segmentation performance, is utilized as post-process for network predictions.


In our work, we integrate self-paced learning and appearance guidance into a unified network.
Specifically, we propose a attentive self-paced learning paradigm to better schedule the learning progress of our framework.
Moreover, the appearance information is formulated as a supervision signal for obtaining precise object boundary.

\subsection{Multi-modality Salient Object Detection}
Multi-modality SOD means that using extra data from other modalities to improve the SOD performance, such as depth map, thermal image and optical flow.

\textbf{Supervised Methods.}
To better integrate the auxiliary information, abundant methods were proposed recently \cite{dsn2,ssav,adf,dsa2f,spnet,digr,pcsa,tenet,stvs,mied,midd}.
Most of these methods employ a two-stream encoder-decoder structure to aggregate the appearance information from multi-modality data.
To reduce the need of pixel-level annotations, Zhao et al. \cite{wvsod} proposed a VSOD dataset with scribbles to indicate the location of salient objects.

Similar to RGB SOD task, these methods still need extra efforts on annotating images, which is very laborious.

\textbf{Unsupervised Methods.}
A few works focus on tackling multi-modality SOD task in an unsupervised manner.
For example, Ji et al. \cite{dsu} refined the saliency maps extracted by traditional SOD methods to produce more accurate saliency predictions.
However, the same as RGB-based USOD methods, this method is sensitive to the quality of traditional SOD methods.
Moreover, it has not well utilize appearance information during the network training process.

To the best of our knowledge, our work is the first in tackling various multi-modality Salient Object Detection (SOD) tasks by using a single unified framework.
By modeling saliency information in images, our framework surpasses existing USOD methods and is comparable to the fully-supervised multi-modality SOD methods.


\begin{figure*}[!t]
\centering
\includegraphics[width=1 \textwidth]{image/framework.pdf}\\
\caption{Overview of our appearance-guided attentive self-paced learning framework. 
The network learns to produce high-quality saliency predictions from image in an unsupervised manner by using two novel supervision signals, Attentive Self-Paced Learning (ASPL) and Appearance Guidance Module (AGM). 
First, the ASPL paradigm promotes the network to excavate more detailed saliency information from simple samples to more difficult ones. 
Second, the AGM utilizes the appearance information from multi-modality data to refine the boundary of saliency predictions. 
We use a constant map to fill in the missing modalities of each image.
The $\rho$ is a parameter that gradually increases from 0 to 1 as the training progress.}
\label{fig:framewwork}
\end{figure*}

\subsection{Self-Paced Learning}
Learning from examples organized in a purposeful order that illustrates gradually more concepts is a natural learning process of humans and animals.
Bengio et al. \cite{SPL} proposed the self-paced (or curriculum) learning paradigm in response to this observation, in order to learn the model iteratively from simple to complex examples.
To learn the model parameters $\textbf{w}$ and weight variable $\textbf{v}$, the SPL minimizes:
\begin{equation}
\begin{split}
\label{eqn:spl}
\mathbb{E}(\textbf{w},\textbf{v};\lambda)=&\sum_{i=1}^{n}v_il(y_i,f(\textbf{x}_i,\textbf{w}))-\lambda \sum_{i=1}^{n}v_i, \\
&s.t. \textbf{v}\in [0,1]^n,
\end{split}
\end{equation}
where $\lambda$ is a parameter for controlling the learning pace.
The objective of SPL is to minimize the weighted training loss together with the negative $l_1$-norm regularizier $-\|\textbf{v}\|_1=-\sum_{i=1}^{n}v_i$.
In \cite{SPL5,SPL6}, the ACS (Alternative Convex Search) is proposed to solve Eqn. \ref{eqn:spl} by using an iterative method for biconvex optimization.
How to measure the easiness of samples is critical for the effectiveness of Self-Paced Learning (SPL).
In original SPL, the easiness is estimated based on the loss term $l(y_i,f(\textbf{x}_i,\textbf{w}))$, where $f(\textbf{x}_i,\textbf{w}))=\phi_{\textbf{w}} (\textbf{x}_i)$.
For example, Jiang et al. \cite{jiang_spl} assigned binary weights $v$ for training samples to indicates whether they are token into account in the training process. 
These weights are conditioned on the following principles:
1) samples with $l(y_i,\phi_{\textbf{w}} (\textbf{x}_i)) < T_l$ will be selected in training ($v_i=1$);
2) samples with $l(y_i,\phi_{\textbf{w}} (\textbf{x}_i)) > T_h$ will not be selected in training ($v_i=0$);  
3)
the weights for other samples will be decided by comparing their losses to the threshold $T_l+\frac{T_h}{\sqrt{i}+\sqrt{i+1}}$, where $i$ is the sample's rank w.r.t. its loss value in group.
$T_l$ and $T_h$ are hyperparameters and rise adaptively with the training process.

Such an intuitive learning strategy has been validated in a great deal of computer vision tasks.
For example, Ghasedi et al. \cite{SPL1} proposed a balanced self-paced learning algorithm to adopt the generative adversarial network for clustering.
Soviany et al. \cite{SPL2} leveraged SPL to tackle cross-domain object detection by progressively learning unlabeled samples in target domains.
Zhu et al. \cite{SPL3} tackled the spectrum reconstruction by using SPL to select the training samples in each iteration.
Zhang et al. \cite{SPL4} integrated multiple-instance learning and SPL into a unified network to address co-saliency detection.

In this paper, we adapt the SPL paradigm to the USOD task based on the specific sample distributions.
Moreover, we build an adaptive weighting mechanism for all samples conditioned on current predictions and training progress.

\section{Our Approach}
\subsection{Framework Overview}
Graphical illustration of the proposed framework is shown in Fig. \ref{fig:framewwork}.
First, we propose an Attentive Self-Paced Learning (ASPL) paradigm to excavate finer saliency knowledge gradually from simple examples to the complex ones.
Second, we propose an Appearance Guidance Module (AGM) to formulate the local appearance contrast of each pixel as the probability of saliency boundary and finds the potential boundary of the target objects by maximizing the probability.
In addition, multi-scale training is employed during training and we ensure the scale consistency using a L2 loss $L_{ms}$.
Overall, the loss function used to train our framework is:
\begin{equation}
\label{eqn:lambda}
L_{s1} = \lambda_s L_{aspl} + \lambda_s \hat L_{aspl} + \lambda_a L_{agm} + \lambda_a \hat L_{agm} +\lambda_m L_{ms},
\end{equation}
where all $\lambda$ are hyperparameters.
The $\hat{L}$ indicates the loss for the predictions of a reference scale.

\subsection{Attentive Self-Paced Learning}
A prerequisite of conventional SPL is that all labels for samples are known beforehand.
In this case, assigning different weights for samples may vary the gradients but still along to the optimal.
However, for USOD, our initial labels for each pixels are noisy.
Learning from partial ambiguous pixels are prone to distract the learned saliency modeling patterns in deep networks, resulting in a weakened detection performance.
To address this issue, the proposed ASPL employs soft weights to aggregate saliency information from all samples for modeling more accurate saliency patterns in images. 
Moreover, our ASPL employs a dynamic weighting strategy conditioned on the current learning process to learn from easy samples to the complex ones.

To learn saliency in images, our ASPL paradigm approximates the distribution of saliency predictions to an ideal distribution.
Specifically, in the ideal case, the SOD results follow a Bernoulli distribution.
While in practice, the saliency prediction of our network follows a continuous distribution.
Such a continuous distribution can be considered as a combination of a clean Bernoulli distribution and random noise.
By driving the network predictions to converge to a Bernoulli distribution, we can recover more accurate saliency information.
In our ASPL, we use the L1 loss to reducing the distance for each pixel to its potential label by:
\begin{equation}
l(\tilde y_i,\phi_{\textbf{w}} (\textbf{x}_i)) = |\tilde y_i -\phi_{\textbf{w}} (\textbf{x}_i)| \Leftrightarrow -|\phi_{\textbf{w}} (\textbf{x}_i) - 0.5|,
\end{equation}
where $\tilde y_i$ is the potential label, which is defined as:
\begin{equation}
\tilde y_i =\begin{cases}
1,&\text{if} \quad \phi_{\textbf{w}} (\textbf{x}_i) > 0.5;\\
0,&\text{if} \quad \phi_{\textbf{w}} (\textbf{x}_i) \le 0.5.
\end{cases}
\end{equation}
Minimizing the distance between predictions and potential labels by $|\tilde y_i -\phi_{\textbf{w}} (\textbf{x}_i)|$ is equivalent to maximize $|\phi_{\textbf{w}} (\textbf{x}_i) - 0.5|$, where $0.5$ is the center between two classes, 0 and 1.

How to formulate the weights of samples is critical for the effectiveness of the SPL-based paradigms.
Intuitively, pixels closed to target distribution are easy samples and contains more reliable saliency information. 
Learning from these pixels can conclude some general saliency patterns in images.
For other pixels of greater difficulty, saliency information is hidden within noise.
A sophisticated training strategy is required to reveal the hidden saliency information in such noisy pixels.
At the beginning of the training phase, we assign high weights to the easy samples to learn more reliable saliency information and reduce interest in difficult samples to prevent noise from interfering with the learned saliency patterns.
Since the network has learned reliable saliency patterns as the training process progresses, we can gradually increase the weights of difficult samples to uncover more detailed saliency information while having little effect on the patterns already learned.
Based on the above analysis, the weight $v$ in ASPL is defined as:
\begin{equation}
v_i = |\phi_{\textbf{w}} (\textbf{x}_i) - 0.5|^{(2^{1-\rho}-1)},
\end{equation}
where $\rho$ is a parameter that increases linearly from 0 to 1 with the training process.
As for the regularizer term in the Eqn. \ref{eqn:spl}, its aim is to increase the number of samples involved in the training of the model.
This item is removed in our ASPL because all samples contribute to the training process to varying degrees. 

\begin{figure}[!t]
\includegraphics[width=0.45 \textwidth]{image/aspl.jpg}
\caption{Gradient comparison between ADB \cite{a2s} and our ASPL losses. The solid and dotted lines are gradient curves for $L_{aspl}$ and $L_{adb}$, respectively.}
\label{fig:doloss}
\end{figure}


Finally, the overall loss function of our ASPL can be summarized as:
\begin{equation}
L_{aspl} = \sum_{i=1}^{n}v_il(y_i,f(\textbf{x}_i,\textbf{w})) = -|\phi_{\textbf{w}} (\textbf{x}_i) - 0.5|^{(2^{1-\rho})},
\end{equation}
The partial derivative of our $L_{aspl}$ over $k_i = \phi_{\textbf{w}} (\textbf{x}_i)-0.5$ is
\begin{equation}
        \frac{\partial L_{dfs}}{\partial k_i} = -sign(k_i)2^{(1-\rho)}|k_i|^{2^{(1-\rho)}-1},
\end{equation}
where $sign(k_i) \in [-1, 1]$ .
The gradient curves of $\frac{\partial L_{dfs}}{\partial k_i}$ for different $|k_i|$ are visualized in Fig. \ref{fig:doloss}(b).
For pixels with small $|k|$, their gradients increase quickly during the training process.
While the gradients of pixels with large $|k|$ are relative stable for the whole training process.
In this way, our ASPL guides our saliency generator to extract coarse-to-fine saliency cues from images by dynamically shifting its attention from easy samples to more complex ones.

\begin{figure*}[!t]
\centering
\includegraphics[width=0.95 \textwidth]{image/bac.pdf}
\caption{Graphical illustration of $L_{agm}$.
Our $L_{agm}$ consists of three terms: boundary mask $b_i$, saliency difference $s_{i,j}$, and appearance similarity $d_{i,j}$.
Arrows with deep colors indicate large values.
We select three pixels as examples.
The topmost example tends to be salient, as most neighbors with similar appearances are salient.
Bottomleft example is eliminated because it is the internal edge of salient object.
}
\label{fig:acloss}
\end{figure*}

\subsection{Appearance Guidance Module}
Appearance information is essential for distinguishing between object boundaries and surrounding background.
For example, many traditional segmentation methods \cite{slic,graphcut,bilateral} used this information to generate precise boundaries for target objects.
Inspired by these works, we propose a Appearance Guidance Module (AGM) to reduce the saliency differences between a neighborhood of a given pixel.
As shown in Fig. \ref{fig:acloss}, our AGM bridges the correlation between saliency prediction and appearance information by multiplying three terms:
\begin{equation}
\label{eqn:acl}
L_{agm} = \sum_{i}\sum_{j \in K_i}b_is_{i, j}d_{i, j},
\end{equation}
where $b_i$ and $K_i$ is the boundary mask and $11\times 11$ neighbors of $i$-th pixel, respectively.
First, we introduce a boundary mask into our AGM to avoid the negative effects caused by internal edges in the background or objects.
The boundary masks are generated by using two $5\times 5$ max-pooling operations on our saliency predictions:
\begin{equation}
B = maxpool(Y) * maxpool(1-Y),
\label{eqn:bmask}
\end{equation}
where $Y$ and $B$ are the predicted saliency maps and corresponding boundary masks, respectively.
The $b_i$ is the pixel sample in $B$.
Second, the saliency difference is measured using L1 distance:
\begin{equation}
s_{i, j} = |y_i - y_j|,
\end{equation}
where $y_i$ is the $i$-th pixel in $Y$, while $y_j$ is a neighbor pixel around $y_i$.
Third, for pixels close to an object boundary, their appearances are often different from surrounding pixels.
An exponential distance is employed to measure the appearance similarity by:
\begin{equation}
\label{eqn:dis}
d_{i, j} = exp(-\alpha \|\textbf{x}_{i} - \textbf{x}_{j}\|^2),
\end{equation}
where $\textbf{x}$ is a color vector and $\|\cdot\|^2$ is the Euclidean distance.
$\alpha$ is a hyperparameter that controls the effect of appearance similarity.

From a broader perspective, appearance also can be interpreted as features in other modality data, such as depth map, optical flow and thermal image. 
For salient regions that are not distinctive to surrounding background pixels in color image, they may be distinctive in other modality data.
Therefore, integrating the appearance information in multi-modality data can improve the segmentation accuracy.
Defining $X^m, m \in [1,M] $ as the data from the $m$-th modality, we re-formulate the appearance similarity defined in Eqn. \ref{eqn:dis} as:
\begin{equation}
d_{i, j} = exp(-\alpha \sum_{m=1}^{M}\|\textbf{x}^{m}_{i} - \textbf{x}^{m}_{j}\|^2),
\end{equation}
where $\textbf{x}^{m}_{i}$ is the $i$-th feature from $m$-th modality.  
However, in practical scenarios, it is extremely hard to collect all multi-modality data for the same image.
To tackle the missing modality data, we can simply embed a constant map to our framework.
In this case, $\|\textbf{x}^{m}_{i} - \textbf{x}^{m}_{j}\|^2 = 0$ will not affect the calculation of appearance similarity.

\subsection{Multi-scale Training}
Salient objects are consistent for the same images of different scales.
Therefore, we resize input images to a reference scale and encourage our network to produce consistent predictions for multi-scale inputs.
The predictions $\hat p$ of reference scale are resized to the original size and compute its difference with $p$ by:
\begin{equation}
L_{ms} = \sum_{i}|y_i - resize(\hat y_i)|^2.
\end{equation}
where $y_i$ and $\hat y_i$ are saliency predictions in original and reference scales, respectively.


\subsection{Network Construction.}
We build a simple network $\phi$ to produce one-channel predictions for each image, which is defined as:
\begin{equation}
\phi(X) = Y,
\end{equation}
where $X$ and $Y$ are input image and prediction, respectively.
Following previous method \cite{a2s}, our network consists of an ImageNet pre-trained encoder and several Squeeze-and-Excitation (SE) blocks.
The formulas of our network $\phi$ are shown as follows:
\begin{equation}
\label{eqn:phi}
\begin{split}
E_3, E_4, E_5 &= Encoder(X),\\
F_i &= SE(E_i), i=3,4,5,\\
H &= SE(concat(F_3, F_4, F_5)),\\
Y &= Sigmoid(sum(H - \bar H)),
\end{split}
\end{equation}
where $\bar H$ is the global mean of $H$.
Specifically, our encoder is instantiated as the ResNet-50 \cite{resnet} pretrained by MoCo-v2 \cite{mocov2}, which does not require any human annotation for training. 
We select the features from stage 3 to 5 of the encoder to modeling saliency because they have concluded more global statistics, which are crucial for localizing salient objects.
We further reduce the channel of each feature to 64 and reorganize its learned knowledge using an SE block.
Subsequently, an additional SE block is employed to integrate the knowledge of all three features after concatenation, denoted as $H$.
After that, $H$ is subtracted by its global mean to ensure the coexisting of positive and negative samples, and summed over the channel dimension to produce one-channel predictions.
Finally, a Sigmoid function is appended to produce the preliminary saliency predictions.

\begin{figure}[!t]
\includegraphics[width=0.49 \textwidth]{image/detector_1.pdf}
\caption{The network structure used in our framework.}
\label{fig:network}
\end{figure}

\subsection{Training a Generalized Detector}
Our framework can produce high-quality saliency predictions for training samples by modeling intrinsic knowledge in the visible images. 
However, during testing, it may not work well for the unseen images without fine-tuning based on the intrinsic knowledge.
Therefore, following previous methods \cite{sbf,usps}, we employ a second stage to train a generalized saliency detector using the pseudo labels generated by our framework.
The saliency detector can be any existing fully-supervised SOD network.
IOU loss is employed as the loss function for training the detector:
\begin{equation}
\begin{split}
    L_{s2} = L_{iou}(P, \ddot{Y}) = 1 &- \frac{sum(P*\ddot{Y})}{sum(P+\ddot{Y}-P*\ddot{Y})},\\
    \ddot{Y} &= CRF(Y)
\end{split}
\end{equation}
where $P$ and $Y$ are the prediction of detector and our pseudo labels, respectively.

\section{Experiments}
\subsection{Experiment settings}
\textbf{Setup.}
All experiments are implemented on a single GTX 1080 Ti GPU using PyTorch \cite{Pytorch}.
The batch size is 8 and images are resized to $320^2$.
The reference scale in multi-scale training is randomly selected from $(192^2, 256^2, 384^2, 448^2)$.
For data augmentation, horizontal flipping is employed during the training process.
SGD is utilized as the optimizer to train our framework for 20 epochs.
The initial learning rate is set to 0.1 and decayed using a linear schedule.
In addition, $\alpha$ in Eqn. \ref{eqn:dis} is set to 200.
Meanwhile, $\lambda_d$, $\lambda_b$ and $\lambda_m$ in Eqn. \ref{eqn:lambda} are set to 1, 0.05 and 1, respectively
In the generalized salient object detection phase, we use a learning rate of 0.005 to train our saliency detector for 10 epochs.
For RGB SOD, the same network as previous method \cite{a2s} is trained on the generated pseudo labels.
For multi-modality SOD tasks, we use the generated pseudo labels to train the MIDD \cite{midd} network.
Noted that all networks in our framework use MoCo-v2 pretrained ResNet-50 as backbone to ensure that no manual labels are involved in the whole training process. 

\textbf{Datasets.}
To make a fair comparison with existing SOD methods, we train our framework on the train subsets of MSRA-B \cite{msra} or DUTS \cite{duts} datasets separately.
ECSSD \cite{ecssd}, PASCAL-S \cite{pascal-S}, HKU-IS \cite{hku-is}, DUTS-TE \cite{duts}, DUT-O \cite{DUT-OMRON} as well as the test subset of MSRA-B are employed to evaluate the performance of the proposed framework.
They have 1000, 850, 4447, 5019, 5168, and 2000 images, respectively.

For multi-modality SOD tasks, including RGB-D, RGB-T and video SOD, we use the same datasets as existing fully-supervised methods \cite{dsu,stvs,ccfe}.
Specifically, for RGB-D SOD, we choose the same 700 samples from NLPR \cite{nlpr} and 1485 samples from NJUD \cite{njud} to train our framework.
Our method is evaluated on RGBD135 \cite{rgbd135}, NJUD, NLPR and SIP \cite{sip}, which contain 135, 500, 300 and 929 images, respectively.	
For RGB-T SOD, we choose the same 2500 images in VT5000 \cite{vt5000} as our train set, while the rest 2500 images in VT5000, 1000 images in VT1000 \cite{vt1000} and 821 images in VT821 \cite{vt821} are employed as the test sets.
For video SOD, we choose the train splits of DAVIS \cite{davis} and DAVSOD \cite{ssav} to train our framework.
Due to the high sample rate of DAVSOD, we randomly sample 5 frames of each video to avoid overfitting.
Moreover, the test splits of DAVIS, DAVSOD, SegV2 \cite{segv2} and FBMS \cite{fbms} are employed as the test sets, which contains 1376, 5332, 1025 and 367 images, respectively.

\begin{table*}[t]
\caption{Experiment results on SOD benchmarks.
``E\&S'' indicates the supervised signals used to train encoders and SOD methods.
``F'', ``W'' and ``U'' mean fully-supervised, weakly-supervised and unsupervised, respectively.
Best scores are in bold.
}
\label{tab:result}
\centering
\renewcommand\tabcolsep{4pt}
\begin{tabular}{l|c|c|ccc|ccc|ccc|ccc|ccc|ccc}
\hline
\multirow{2}{*}{Methods} & \multirow{2}{*}{Year} & \multirow{2}{*}{E\&S} & \multicolumn{3}{c|}{ECSSD} & \multicolumn{3}{c|}{MSRA-B} & \multicolumn{3}{c|}{DUT-O} & \multicolumn{3}{c|}{PASCAL-S}& \multicolumn{3}{c|}{DUTS-TE}& \multicolumn{3}{c}{HKU-IS} \\\cline{4-21}
& & & $F_{\beta}\negmedspace \uparrow$ & $E_{\xi}\negmedspace \uparrow$ & $\mathcal{M}\negmedspace \downarrow$ & $F_{\beta}\negmedspace \uparrow$ & $E_{\xi}\negmedspace \uparrow$ & $\mathcal{M}\negmedspace \downarrow$ & $F_{\beta}\negmedspace \uparrow$ & $E_{\xi}\negmedspace \uparrow$ & $\mathcal{M}\negmedspace \downarrow$ & $F_{\beta}\negmedspace \uparrow$ & $E_{\xi}\negmedspace \uparrow$ & $\mathcal{M}\negmedspace \downarrow$ & $F_{\beta}\negmedspace \uparrow$ & $E_{\xi}\negmedspace \uparrow$ & $\mathcal{M}\negmedspace \downarrow$ & $F_{\beta}\negmedspace \uparrow$ & $E_{\xi}\negmedspace \uparrow$ & $\mathcal{M}\negmedspace \downarrow$   \\\hline
\multicolumn{21}{c}{Trained on DUTS-TR} \\ \hline
PiCANet \cite{picanet} & 2018 & F\&F & .886 & .927 & .046 & .874 & .933 & .055 & .717 & .848 & .065 & .804 & .862 & .076 & .759 & .873 & .051 & .870 & .941 & .044 \\
BASNet \cite{basnet} & 2019 & F\&F & .879 & .921 & .037 & .896 & .942 & \textbf{.036} & .756 & .869 & .056 & .781 & .853 & .077 & .791 & .884 & .048 & .898 & .947 & .033 \\
CPD \cite{cpd} & 2019 & F\&F & .917 & .949 & .037 & .899 & .947 & .038 & .747 & .873 & .056 & .831 & .887 & .072 & .805 & .904 & .043 & .891 & .952 & .034 \\
SCRN \cite{scrn} & 2019 & F\&F & .918 & .942 & .037 & .868 & .929 & .063 & .746 & .869 & .056 & .839 & .888 & .065 & .809 & .901 & .040 & .897 & .954 & .033 \\
ITSD \cite{itsd} & 2020 & F\&F & .905 & .933 & .035 & .902 & .948 & .038 & .750 & .863 & .059 & .817 & .868 & .066 & .808 & .897 & .040 & .899 & .953 & .030 \\
MINet \cite{minet} & 2020 & F\&F & .924 & \textbf{.953} & .033 & .903 & .948 & .038 & .756 & .873 & .055 & .842 & .899 & .064 & .828 & .917 & .037 & .908 & \textbf{.961} & .028 \\
LDF \cite{ldf} & 2020 & F\&F & .930 & .951 & .034 & .902 & .944 & .037 & .773 & .881 & .052 & \textbf{.853} & \textbf{.903} & \textbf{.062} & .855 & .929 & .034 & .914 & .960 & .028 \\
KRN \cite{krn} & 2021 & F\&F & \textbf{.931} & .951 & \textbf{.032} & \textbf{.911} & \textbf{.950} & \textbf{.036} & \textbf{.793} & \textbf{.893} & \textbf{.050} & .851 & .894 & .068 & \textbf{.865} & \textbf{.934} & \textbf{.033} & \textbf{.920} & \textbf{.961} & \textbf{.027} \\\hline
WSS \cite{wss} & 2017 & F\&W & .767 & .769 & .108 & -- & -- & -- & .590 & .729 & .110 & .698 & .690 & .184 & .633 & .806 & .100 & .773 & .819 & .079 \\
ASMO \cite{asmo} & 2018 & F\&W & .762 & .792 & .068 & -- & -- & -- & .641 & .761 & .100 & .653 & .647 & .206 & .569 & .690 & .116 & .763 & .800 & .089 \\
MSW \cite{msw} & 2019 & F\&W & .761 & .788 & .098 & .861 & .924 & .073 & .597 & .728 & .109 & .685 & .693 & .178 & .648 & .742 & .091 & .734 & .786 & .084 \\
WSSA \cite{wssa} & 2020 & F\&W & .870 & .917 & .059 & .869 & .929 & .049 & .703 & .845 & .068 & .785 & .855 & .096 & .742 & .869 & .062 & .860 & .932 & .047 \\
MFNet \cite{mfnet} & 2021 & F\&W & .844 & .889 & .084 & .872 & .923 & .059 & .621 & .784 & .098 & .756 & .824 & .115 & .693 & .832 & .079 & .839 & .919 & .058 \\
SCW \cite{lsc} & 2021 & F\&W & \textbf{.900} & \textbf{.931} & \textbf{.049} & \textbf{.898} & \textbf{.940} & \textbf{.040} & \textbf{.758} & \textbf{.862} & \textbf{.060} & \textbf{.827} & \textbf{.879} & \textbf{.080} & \textbf{.823} & \textbf{.890} & \textbf{.049} & \textbf{.896} & \textbf{.943} & \textbf{.038} \\ \hline
EDNS \cite{edns} & 2020 & F\&U & .872 & .906 & .068 & .880 & .932 & .051 & .682 & .821 & .076 & .801 & .846 & .097 & .735 & .847 & .065 & .874 & .933 & .046 \\
DCFD \cite{dcfd} & 2022 & F\&U & .888 & .915 & .059 & .888 & .930 & .045 & .710 & .837 & .070 & .795 & .860 & .090 & .764 & .855 & .064 & .889 & .935 & .042 \\
Ours & 2022 & U\&U & \textbf{.916} & \textbf{.938} & \textbf{.044} & \textbf{.904} & \textbf{.944} & \textbf{.039} & \textbf{.745} & \textbf{.863} & \textbf{.061} & \textbf{.830} & \textbf{.882} & \textbf{.074} & \textbf{.810} & \textbf{.901} & \textbf{.047} & \textbf{.902} & \textbf{.947} & \textbf{.037} \\\hline

\multicolumn{21}{c}{Trained on MSRA-B} \\ \hline
SBF \cite{sbf} & 2017 & F\&U & .812 & .878 & .087 & .867 & .929 & .058 & .611 & .771 & .106 & .711 & .795 & .131 & .627 & .785 & .105 & .805 & .895 & .074 \\
MNL \cite{mnl} & 2018 & F\&U & .874 & .906 & .069 & .881 & .932 & .053 & .683 & .821 & .076 & \textbf{.792} & \textbf{.846} & \textbf{.091} & -- & -- & -- & .874 & .932 & .047 \\
USPS \cite{usps} & 2019 & F\&U & .875 & .903 & \textbf{.064} & .896 & .938 & .042 & .715 & .839 & .069 & .770 & .828 & .107 & .730 & .840 & .072 & .880 & .933 & .043 \\
A2S \cite{a2s} & 2021 & F\&U & \textbf{.888} & \textbf{.911} & \textbf{.064} & .902 & \textbf{.941} & \textbf{.041} & .719 & \textbf{.841} & .069 & .790 & .838 & .106 & \textbf{.750} & \textbf{.860} & \textbf{.065} & \textbf{.887} & \textbf{.937} & \textbf{.042} \\
DCFD \cite{dcfd} & 2022 & F\&U & .880 & .900 & \textbf{.064} & \textbf{.903} & .938 & \textbf{.041} & \textbf{.731} & .838 & \textbf{.064} & .773 & .830 & .105 & .744 & .832 & .068 & \textbf{.887} & .926 & .044 \\\hline
A2S \cite{a2s} & 2021 & U\&U & .882 & .921 & \textbf{.056} & .886 & .939 & .041 & .688 & .818 & .079 & .778 & .842 & .100 & .729 & .847 & .069 & .868 & .936 & \textbf{.041} \\
Ours & 2022 & U\&U & \textbf{.902} & \textbf{.923} & \textbf{.056} & \textbf{.912} & \textbf{.948} & \textbf{.036} & \textbf{.731} & \textbf{.851} & \textbf{.065} & \textbf{.803} & \textbf{.848} & \textbf{.099} & \textbf{.767} & \textbf{.871} & \textbf{.061} & \textbf{.891} & \textbf{.939} & \textbf{.041} \\\hline
\end{tabular}
\end{table*}

\begin{table*}[!t]
\begin{center}
\caption{Setting comparison among USOD methods.
Note that some methods exclude the time for extracting saliency cues using traditional methods.
``F'' and ``U'' indicate fully-supervised and unsupervised training strategies.
``IN'' and ``CS'' are ImageNet \cite{imagenet} and CityScape \cite{cityscape} datasets, respectively.
}
\renewcommand\tabcolsep{11pt}
\label{tab:compare}
\begin{tabular}{c|ccccccc}
\hline
Method & Train set & Input & Encoder & Pretrain & Saliency cues & Train time \\ \hline
EDNS \cite{edns} & DUTS-TR & $352\times 352$ & VGG-16 & F-IN & \cite{rbd, mr, gs} & $\textgreater$8h \\
DCFD \cite{dcfd} & DUTS-TR & -- & ResNet-50 & F-IN & \cite{dsr} & -- \\
Ours & DUTS-TR & $320\times 320$ & ResNet-50 & U-IN & No & 4.5h \\\hline
SBF \cite{sbf} & MSRA-B  & $224\times 224$& VGG-16 & F-IN & \cite{mb+, bms, cssd} & $\textgreater$3h \\
MNL \cite{mnl} & MSRA-B  & $425\times 425$& ResNet-101 & F-IN & \cite{rbd, dsr, mc, cssd} & $\textgreater$4h \\
USPS \cite{usps} & MSRA-B  & $432\times 432$ & ResNet-101 & F-CS & \cite{rbd, dsr, mc, cssd} & $\textgreater$30h \\
DCFD \cite{dcfd} & MSRA-B & -- & ResNet-101 & F-CS & \cite{dsr} & -- \\
A2S \cite{a2s} & MSRA-B  & $320\times 320$ & ResNet-50 & U-IN & No & 1h \\
Ours & MSRA-B & $320\times 320$ & ResNet-50 & U-IN & No & 1.3h \\
\hline
\end{tabular}
\end{center}
\end{table*}

\textbf{Metrics.}
We adopt three criteria in our experiments, including ave-$F_{\beta}$, Mean Absolute Error (MAE) and Enhanced-alignment Measure ($E_{\xi}$).
$F_{\beta}$ is computed by:
\begin{equation}
F_{\beta} = \frac{(1+\beta^{2})\times Precision \times Recall}{\beta^{2}\times Precision + Recall},
\end{equation}
where $\beta^{2}$ is set to 0.3 \cite{THUR15K} in general.
The ave-$F_{\beta}$ is the $F_{\beta}$ scores by setting the threshold as two times average value of the predictions.
In addition, MAE is calculated by:
\begin{equation}
    \mathcal{M} = \frac{1}{W \times H} \sum_{i=1}^{W}\sum_{j=1}^{H} |p_i - g_i|,
\end{equation}
where $p_i$ and $g_i$ are prediction and ground truth, respectively.
Enhanced-alignment Measure ($E_{\xi}$) \cite{Emeasure} captures global statistics and local pixel matching information by:
\begin{equation}
E_{\xi} = \frac{1}{W \times H} \sum_{i=1}^{W}\sum_{j=1}^{H} \phi (i, j),
\end{equation}
where $\phi(i, j)$ is the enhanced alignment matrix.


\begin{figure*}[t]
\centering
\includegraphics[width=1 \textwidth]{image/visual.pdf}
\caption{Visual comparison with state-of-the-art SOD methods.}
\label{fig:visual}
\end{figure*}

\subsection{Results on RGB SOD}
We compare the proposed framework with \textbf{fully-supervised} methods, PiCANet \cite{picanet}, BASNet \cite{basnet}, CPD \cite{cpd}, SCRN \cite{scrn}, ITSD \cite{itsd}, MINet \cite{minet}, LDF \cite{ldf}, and KRN \cite{krn}, \textbf{weakly-supervised} methods, WSS \cite{wss}, ASMO \cite{asmo}, MSW \cite{msw}, WSSA \cite{wssa}, MFNet \cite{mfnet}, and SCW \cite{lsc}, and \textbf{unsupervised} methods, SBF \cite{sbf}, MNL \cite{mnl}, USPS \cite{usps}, EDNS \cite{edns}, A2S \cite{a2s}, and DCFD \cite{dcfd}.
The results of these methods on six SOD benchmarks are exhibited in Tab. \ref{tab:result}.


Either training on MSRA-B or DUTS-TR, the proposed framework achieves significant improvements compared to existing USOD methods.
These results prove the effectiveness and generalization ability of the proposed framework.
Compared to weakly-supervised SOD methods, our unsupervised framework still surpasses most of them and achieves similar results to state-of-the-art SCW \cite{lsc} method.
It proves that modeling multi-level representations of images can produce more precise saliency information than low-cost human annotations.
In addition, the performance of our framework is comparable to the latest fully-supervised SOD methods.
These results imply that unsupervised USOD methods have the potential to achieve more competitive performance against fully-supervised SOD methods.
It conveys an encouraging message that human annotation may not be required for training a robust saliency detector.

\begin{table*}[t]
\caption{The quality of pseudo labels for multi-modality SOD datasets.
}
\label{tab:mm}
\centering
\renewcommand\tabcolsep{7pt}
\small
\begin{tabular}{l|ccc|ccc|ccc|ccc}
\hline
\multirow{2}{*}{Training data} & \multicolumn{3}{c|}{RGB} & \multicolumn{3}{c|}{RGB-D} & \multicolumn{3}{c|}{VSOD} & \multicolumn{3}{c}{RGB-T} \\\cline{2-13}
& $F_{\beta}\negmedspace \uparrow$ & $E_{\xi}\negmedspace \uparrow$ & $\mathcal{M}\negmedspace \downarrow$ & $F_{\beta}\negmedspace \uparrow$ & $E_{\xi}\negmedspace \uparrow$ & $\mathcal{M}\negmedspace \downarrow$ & $F_{\beta}\negmedspace \uparrow$ & $E_{\xi}\negmedspace \uparrow$ & $\mathcal{M}\negmedspace \downarrow$ & $F_{\beta}\negmedspace \uparrow$ & $E_{\xi}\negmedspace \uparrow$ & $\mathcal{M}\negmedspace \downarrow$   \\\hline
MM-split  & .917 & .945 & .038 & .672 & .693 & .135 & .557 & .725 & .099 & .932 & .951 & .029  \\
Only-RGB & .917 & .941 & .040 & .804 & .874 & .068 & .613 & .784 & .089 & .920 & .950 & .030 \\
MM-joint & .918 & .941 & .040 & .826 & .892 & .062 & .637 & .805 & .079 & .923 & .952 & .029  \\
\hline
\end{tabular}
\end{table*}

Based on the setting comparison between USOD methods in Tab. \ref{tab:compare}, we have the following conclusions.
First, the proposed framework achieves better performance than other methods under more disadvantaged settings, \textit{e.g.}, smaller input size, and weakened encoders.
Second, the latest MNL \cite{mnl}, USPS \cite{usps} and DCFD \cite{dcfd} use ResNet-101 \cite{resnet} as encoder, which is more powerful than our ResNet-50.
Moreover, their encoders are pretrained using image- or pixel-level labels, while our encoder is pretrained without using any human annotation.
Third, no human annotation is involved in the training process of our framework, including the pretraining of encoders.
Fourth, similar to A2S \cite{a2s}, our framework does not rely on any traditional SOD method to extract coarse saliency cues from images.
Last, training efficiency of our framework is much higher than most USOD methods.
A2S is slightly faster than us, but its performance is significantly lower.

A qualitative comparison of SOD methods on several challenging cases is illustrated in Fig. \ref{fig:visual}.
First, our framework better differentiates salient objects with surrounding backgrounds and produces distinct boundaries than existing USOD methods.
For example, in the first image, multiple SOD methods fail to capture the tiny salient object.
Our framework and two fully-supervised SOD methods well segment this object with precise boundaries.
Second, as shown in the sixth example, existing USOD methods are struggling to capture entire salient objects.
On the contrary, the proposed framework well captures entire salient objects as fully-supervised methods.
Last, our framework can produce more clean background than other methods.
In the last image, most SOD methods consider those characters may be salient because of their similar colors with the dog.
Our saliency prediction is much more distinct than these methods.


\begin{figure}[t]
\centering
\includegraphics[width=0.48 \textwidth ,height=2.9in]{image/mm_vis.pdf}
\caption{Examples of the generated pseudo labels. The ``D/T/O'' means depth map, thermal image or optical flow.}
\label{fig:mm_show}
\end{figure}



\begin{table*}[t]
\caption{Results on RGB-D SOD datasets. ``F'' and ``U'' mean fully-supervised and unsupervised, respectively.
}
\label{tab:rgbd}
\centering
\renewcommand\tabcolsep{7pt}
\small
\begin{tabular}{l|c|c|ccc|ccc|ccc|ccc}
\hline
\multirow{2}{*}{Methods} & \multirow{2}{*}{Year} & \multirow{2}{*}{Sup.} & \multicolumn{3}{c|}{RGBD135} & \multicolumn{3}{c|}{NJUD} & \multicolumn{3}{c|}{NLPR} & \multicolumn{3}{c}{SIP} \\\cline{4-15}
& & & $F_{\beta}\negmedspace \uparrow$ & $E_{\xi}\negmedspace \uparrow$ & $\mathcal{M}\negmedspace \downarrow$ & $F_{\beta}\negmedspace \uparrow$ & $E_{\xi}\negmedspace \uparrow$ & $\mathcal{M}\negmedspace \downarrow$ & $F_{\beta}\negmedspace \uparrow$ & $E_{\xi}\negmedspace \uparrow$ & $\mathcal{M}\negmedspace \downarrow$ & $F_{\beta}\negmedspace \uparrow$ & $E_{\xi}\negmedspace \uparrow$ & $\mathcal{M}\negmedspace \downarrow$   \\\hline
DSNet \cite{dsn2} & 2021 & F & .899 & .969 & .020 & -- & -- & -- & .886 & .957 & .024 & .864 & .920 & .052  \\
DSA2F \cite{dsa2f} & 2021 & F & .899 & .958 & .021 & .901 & .937 & .039 & .897 & .953 & .024 & -- & -- & --  \\
SPNet \cite{spnet} & 2021 & F & .927 & .984 & .013 & -- & -- & -- & .903 & .959 & .021 & .893 & .931 & .043  \\
DIGR \cite{digr} & 2022 & F & .898 & .969 & .019 & -- & -- & -- & .889 & .956 & .023 & .878 & .918 & .053 \\
CCFE \cite{ccfe} & 2022 & F & .911 & .964 & .020 & .914 & .953 & .032 & .907 & .962 & .021 & .889 & .923 & .047  \\
\hline
DSU \cite{dsu}  & 2022 & U & .767 & .895 & .061 & .719 & .797 & .135 & .745 & .879 & .065 & .619 & .774& .156  \\
MM-split (Ours) & 2022 & U & .845 & .910 & .051 & .709 & .722 & .148 & .836 & .903 & .055 & .650 & .677 & .151  \\
MM-joint (Ours) & 2022 & U & .877 & .946 & .029 & .862 & .908 & .060 & .852 & .931 & .034 & .873 & .925 & .051 \\
\hline
\end{tabular}
\end{table*}

\begin{table*}[t]
\caption{Results on VSOD datasets. ``F'', ``W'' and ``U'' mean fully-supervised, weakly-supervised and unsupervised, respectively.
}
\label{tab:vsod}
\centering
\renewcommand\tabcolsep{7pt}
\small
\begin{tabular}{l|c|c|ccc|ccc|ccc|ccc}
\hline
\multirow{2}{*}{Methods} & \multirow{2}{*}{Year} & \multirow{2}{*}{Sup.} & \multicolumn{3}{c|}{DAVSOD} & \multicolumn{3}{c|}{DAVIS} & \multicolumn{3}{c|}{SegV2} & \multicolumn{3}{c}{FBMS} \\\cline{4-15}
& & & $F_{\beta}\negmedspace \uparrow$ & $E_{\xi}\negmedspace \uparrow$ & $\mathcal{M}\negmedspace \downarrow$ & $F_{\beta}\negmedspace \uparrow$ & $E_{\xi}\negmedspace \uparrow$ & $\mathcal{M}\negmedspace \downarrow$ & $F_{\beta}\negmedspace \uparrow$ & $E_{\xi}\negmedspace \uparrow$ & $\mathcal{M}\negmedspace \downarrow$ & $F_{\beta}\negmedspace \uparrow$ & $E_{\xi}\negmedspace \uparrow$ & $\mathcal{M}\negmedspace \downarrow$   \\\hline
SSAV \cite{ssav} & 2019 & F & .540 & .742 & .083 & .762 & .902 & .029 & .720 & .895 & .024 & .793 & .875 & .043  \\
PCSA \cite{pcsa} & 2020 & F & .556 & .749 & .077 & .794 & .922 & .022 & .789 & .931 & .019 & .783 & .868 & .042 \\
TENet \cite{tenet}  & 2020 & F & .595 & .773 & .067 & .821 & .941 & .017 & -- & -- & -- & .851 & .915 & .026  \\
STVS \cite{stvs}  & 2021 & F & .563 & .764 & .080 & .812 & .940 & .022 & .835 & .950 & .016 & .821 & .903 & .042  \\
DCFNet \cite{dcfnet}  & 2021 & F & .599 & .783 & .065 & .864 & .964 & .016 & .863 & .954 & .013 & -- & -- & --  \\
WSVSOD \cite{wvsod}  & 2021 & W & .492 & .710 & .103 & .731 & .900 & .036 & .711 & .909 & .031 & .736 & .840 & .084  \\
\hline
MM-split (Ours)  & 2022 & U & .534 & .747 & .084 & .751 & .913 & .042 & .751 & .914 & .033 & .732 & .794 & .100  \\
MM-joint (Ours) & 2022 & U & .547 & .762 & .085 & .756 & .908 & .037 & .808 & .927 & .021 & .795 & .876 & .060 \\
\hline
\end{tabular}
\end{table*}

\begin{table*}[t]
\caption{Results on RGB-T SOD datasets. ``F'' and ``U'' mean fully-supervised and unsupervised, respectively.}
\label{tab:rgbt}
\centering
\renewcommand\tabcolsep{7pt}
\small
\begin{tabular}{l|c|c|ccc|ccc|ccc}
\hline
\multirow{2}{*}{Methods} & \multirow{2}{*}{Year} & \multirow{2}{*}{Sup.} & \multicolumn{3}{c|}{VT5000} & \multicolumn{3}{c|}{VT1000} & \multicolumn{3}{c}{VT821} \\\cline{4-12}
& & & $F_{\beta}\negmedspace \uparrow$ & $E_{\xi}\negmedspace \uparrow$ & $\mathcal{M}\negmedspace \downarrow$ & $F_{\beta}\negmedspace \uparrow$ & $E_{\xi}\negmedspace \uparrow$ & $\mathcal{M}\negmedspace \downarrow$ & $F_{\beta}\negmedspace \uparrow$ & $E_{\xi}\negmedspace \uparrow$ & $\mathcal{M}\negmedspace \downarrow$   \\\hline
ADF \cite{adf}   & 2020 & F & .778 & .891 & .048 & .847 & .922 & .034 & .717 & .845 & .077 \\
MIED \cite{mied} & 2020 & F & .761 & .880 & .050 & .853 & .928 & .030 & .760 & .877 & .050  \\
MIDD \cite{midd}  & 2021 & F & .801 & .899 & .043 & .882 & .942 & .027 & .805 & .898 & .045  \\
APNet \cite{apnet} & 2021 & F & .821 & .918 & .035 & .885 & .951 & .021 & .818 & .912 & .034 \\
CCFE \cite{ccfe}  & 2022 & F & .859 & .937 & .030 & .906 & .963 & .018 & .857 & .934 & .027  \\
\hline
MM-split (Ours)  & 2022 & U & .810 & .904 & .046 & .885 & .939 & .031 & .805 & .900 & .043   \\
MM-joint (Ours) & 2022 & U & .807 & .903 & .047 & .881 & .939 & .032 & .805 & .899 & .044 \\
\hline
\end{tabular}
\end{table*}


\subsection{Results on Multi-modality SOD}
To validate the generalization ability of the proposed framework, we conduct more experiments on multi-modality SOD tasks, including RGB-D, RGB-T and video.
Noted that for the video SOD task, we pre-compute the optical flow for each frame as an extra modality data.
Since there are different modality data, we train our framework under three sets of data:
1) \textit{Task-specific multi-modality data (``MM-split'').}
For example, for RGB-T SOD task, we only use 2500 RGB-thermal image pairs to train our framework.
Other multi-modality SOD tasks follow a similar setting.
2) \textit{RGB images from all tasks (``Only-RGB'').}
3) \textit{All multi-modality data from all tasks (``MM-joint'').}
In Tab. \ref{tab:mm}, we show the scores of saliency predictions on corresponding training sets.

\begin{figure*}[!t]
\begin{minipage}{1 \textwidth}
\includegraphics[width=.98in,height=0.7in]{labels/image/ILSVRC2012_test_00002588.png}
\includegraphics[width=.98in,height=0.7in]{labels/gt/ILSVRC2012_test_00002588.png}
\includegraphics[width=.98in,height=0.7in]{labels/baseline/ILSVRC2012_test_00002588.png}
\includegraphics[width=.98in,height=0.7in]{labels/v1/ILSVRC2012_test_00002588.png}
\includegraphics[width=.98in,height=0.7in]{labels/v2/ILSVRC2012_test_00002588.png}
\includegraphics[width=.98in,height=0.7in]{labels/v3/ILSVRC2012_test_00002588.png}
\includegraphics[width=.98in,height=0.7in]{labels/stage2/ILSVRC2012_test_00002588.png}
\end{minipage}
\vspace{0.05in}

\begin{minipage}{1 \textwidth}
\includegraphics[width=.98in,height=0.7in]{labels/image/ILSVRC2012_test_00007277.png}
\includegraphics[width=.98in,height=0.7in]{labels/gt/ILSVRC2012_test_00007277.png}
\includegraphics[width=.98in,height=0.7in]{labels/baseline/ILSVRC2012_test_00007277.png}
\includegraphics[width=.98in,height=0.7in]{labels/v1/ILSVRC2012_test_00007277.png}
\includegraphics[width=.98in,height=0.7in]{labels/v2/ILSVRC2012_test_00007277.png}
\includegraphics[width=.98in,height=0.7in]{labels/v3/ILSVRC2012_test_00007277.png}
\includegraphics[width=.98in,height=0.7in]{labels/stage2/ILSVRC2012_test_00007277.png}
\end{minipage}
\vspace{0.05in}

\begin{minipage}{1 \textwidth}
\includegraphics[width=.98in,height=0.7in]{labels/image/ILSVRC2012_test_00011755.png}
\includegraphics[width=.98in,height=0.7in]{labels/gt/ILSVRC2012_test_00011755.png}
\includegraphics[width=.98in,height=0.7in]{labels/baseline/ILSVRC2012_test_00011755.png}
\includegraphics[width=.98in,height=0.7in]{labels/v1/ILSVRC2012_test_00011755.png}
\includegraphics[width=.98in,height=0.7in]{labels/v2/ILSVRC2012_test_00011755.png}
\includegraphics[width=.98in,height=0.7in]{labels/v3/ILSVRC2012_test_00011755.png}
\includegraphics[width=.98in,height=0.7in]{labels/stage2/ILSVRC2012_test_00011755.png}
\end{minipage}
\vspace{0.02in}

\begin{minipage}{1 \textwidth}
\begin{minipage}{.98in} \centering Image \end{minipage}
\begin{minipage}{.98in} \centering GT \end{minipage}
\begin{minipage}{.98in} \centering Baseline \end{minipage}
\begin{minipage}{.98in} \centering V1 \end{minipage}
\begin{minipage}{.98in} \centering V2 \end{minipage}
\begin{minipage}{.98in} \centering V3 \end{minipage}
\begin{minipage}{.98in} \centering V4 \end{minipage}
\end{minipage}

\caption{Examples of the generated pseudo labels.}
\label{fig:pseudo_show}
\end{figure*}

Overall, training on the union set of all multi-modality data (``MM-joint'') reports a more generalized performance on all datasets.
An interesting observation is that training on task-specific data (``MM-split'') obtains slightly better performance on RGB and RGB-T datasets, while is significantly inferior on RGB-D and VSOD datasets.
Under the unsupervised setting, training on all multi-modality data may report slightly worse performance on a subset of training set to learn more generalized saliency information from other data.
Meanwhile, due to a low diversity of data in RGB-D and VSOD dataset, training on such data is prone to focus on some discriminative regions of target objects, as shown in Fig. \ref{fig:mm_show}. 
In addition, the results on ``Only-RGB'' prove that multi-modality data can guide our framework to learn more accurate saliency information.

The above experiments provide an quantitative validation on the quality of pseudo labels produced by using different training data.
For a more intuitive illustration, we employ these pseudo labels of task-specific dataset to train a task-specific saliency detector like conventional fully-supervised methods.
For the RGB-D SOD task in Tab. \ref{tab:rgbd}, using the labels of ``MM-split'' can obtain slightly better performance than the recently proposed unsupervised RGB-D SOD method DSU \cite{dsu}.
Moreover, using the labels of ``MM-joint'' can achieve a competitive performance against the latest fully-supervised RGB-D SOD methods.
For the VSOD task in Tab. \ref{tab:vsod}, training with our pseudo labels achieves better performance than weakly-supervised method WSVSOD \cite{wvsod}.
Furthermore, training on the labels of ``MM-joint'' reports a comparable performance against the latest fully-supervised video SOD methods.
For the RGB-T SOD task in Tab. \ref{tab:rgbt}, training with ``MM-split'' achieves better results than ``MM-joint'' because the quality of pseudo labels is higher.
In addition, their performances also are comparable to the latest fully-supervised RGB-T SOD methods.

\subsection{Ablation Study}
\textbf{Comparison between Initial Saliency Cues.}
Existing DL-based USOD methods take advantage of traditional SOD methods and features extracted by an ImageNet pretrained network as their initial saliency cues.
We evaluate the accuracy of these saliency cues on MSRA-B, as shown in Tab. \ref{tab:initial}.

In conclusion, traditional SOD methods can produce significantly more accurate saliency maps than ImageNet pretrained networks.
This observation is reasonable because the pretrained networks are trained for other related vision tasks without pixel-level labels.
As shown in Tab. \ref{tab:compare}, most existing DL-based USOD methods \cite{sbf,mnl,usps,dcfd} achieve impressive performances by integrating the prior knowledge from both traditional SOD methods and supervised pretrained networks. 
Different with these methods, our framework can produce high-quality pseudo labels by refining the saliency cues of an unsupervised pretrained network, which obtains the worst performance compared to other methods.
This experiment well prove that our framework can better model the saliency in images, instead of refining multiple noisy labels as existing USOD methods.

\begin{table}[t]
\begin{center}
\caption{The quality of initial saliency cues used in DL-based USOD methods.
``IN-S'' and ``IN-U'' mean supervised and unsupervised pretraining on ImageNet, respectively.}
\renewcommand\tabcolsep{9pt}
\label{tab:initial}
\begin{tabular}{c|c|ccc}
\hline
Methods & Type & $F_{\beta} \uparrow$ & $E_{\xi} \uparrow$ & $\mathcal{M} \downarrow$ \\\hline
MC \cite{mc}       & Hand-crafted  & .810 & .877 & .145  \\
HS \cite{hs}       & Hand-crafted  & .774 & .843 & .161  \\
DSR \cite{dsr}     & Hand-crafted  & .780 & .867 & .118  \\
RBD \cite{rbd}     & Hand-crafted  & .803 & .883 & .109  \\\hline
IN-S \cite{resnet} & DL-based      & .474 & .665 & .321 \\
IN-U \cite{mocov2} & DL-based      & .451 & .659 & .353 \\\hline
\end{tabular}
\end{center}
\end{table}


\begin{table}[t]
\begin{center}
\caption{Ablation study on our framework. We list the scores for the generated pseudo labels compared to human annotations in the DUTS-TR dataset.
}
\renewcommand\tabcolsep{3pt}
\label{tab:ablation}
\begin{tabular}{c|ccccc|ccc}
\hline
&$L_{adb}$ & $L_{mse}$ & $L_{aspl}$ & $L_{agm}$ & Stage 2 & $F_{\beta} \uparrow$ & $E_{\xi} \uparrow$ & $\mathcal{M} \downarrow$ \\\hline
Baseline & \checkmark & & & & & .882 & .915 & .071  \\\hline
V1 & \checkmark & \checkmark & & & & .891 & .921 & .066  \\
V2 & & \checkmark & \checkmark & & & .908 & .937 & .050  \\
V3 & & \checkmark & \checkmark & \checkmark & & .917 & .945 & .038  \\\hline
V4 & & \checkmark & \checkmark & \checkmark & \checkmark &  .920 & .944 & .037 \\\hline
\end{tabular}
\end{center}
\end{table}


\textbf{Effectiveness of loss functions.}
To validate the effectiveness of each loss in the first stage, we report the scores of pseudo labels generated by different supervised signals in Tab. \ref{tab:ablation}.
Compared to baseline, the scale consistency loss $L_{ms}$ slightly improves the quality of pseudo labels.
Compared to $L_{adb}$ \cite{a2s}, our framework with $L_{aspl}$ further excavates the latent saliency information, and thereby more complete objects are segmented.
Furthermore, the proposed $L_{agm}$ refines the learned saliency cues using low-level representations, which is the complete method in our first stage.
Last, the results of V4 prove that the network learns the similar saliency information, but in a more generalized manner.

We also show some pseudo labels generated by these variants in Fig. \ref{fig:pseudo_show}.
The baseline localizes salient objects but loses too many details.
The pseudo labels generated by our framework are much more similar to human annotations.
It proves that our framework extracts more precise saliency information from images by modeling multi-level representations.


\begin{table}[t]
\begin{center}
\caption{Ablation study on the proposed $L_{aspl}$.
Several existing losses are utilized for training our saliency detector with the cooperation of $L_{agm}$ and $L_{ms}$.}
\renewcommand\tabcolsep{11pt}
\label{tab:dfloss}
\begin{tabular}{c|ccc}
\hline
Loss & $F_{\beta} \uparrow$ & $E_{\xi} \uparrow$ & $\mathcal{M} \downarrow$ \\\hline
$L2$ & .558 & .540 & .208  \\
$L1$ & .895 & .928 & .044  \\
$L_{adb}$ \cite{a2s} & .903 & .934 & .042  \\
$L_{spl}$ \cite{SPL} & .910 & .938 & .041  \\
$L_{aspl}$  & .917 & .945 & .038 \\\hline
\end{tabular}
\end{center}
\end{table}

\textbf{Learning strategy comparison.}
Unlike supervised SOD methods, localizing salient objects in images without human annotations is challenging.
We test multiple losses to validate their effectiveness on this goal, including $L2$ loss, $L1$ loss, $L_{adb}$ \cite{a2s}, original $L_{spl}$ \cite{SPL} and our $L_{aspl}$.
The results of using one of these losses to train our framework with the cooperation of $L_{agm}$ and $L_{ms}$ are listed in Tab. \ref{tab:dfloss}.

In conclusion, training with $L_{aspl}$ loss achieves the best performance among these losses.
$L_{adb}$ and $L_{spl}$ report lower scores than ours because they fail to learn more latent saliency information from hard samples.
However, reliable saliency information in these hard samples is much less than easy samples.
Therefore, $L1$ loss obtains inferior results because it pays more attention to these hard samples from the beginning.
The more convincing proof is that $L2$ loss reports the worst results because our saliency generator learns saliency information mainly based on these hard samples.

\begin{table}[t]
\begin{center}
\caption{Different designs of the proposed $L_{agm}$. }
\renewcommand\tabcolsep{7pt}
\label{tab:acl}
\begin{tabular}{c|c|ccc}
\hline
Tag & Description & $F_{\beta} \uparrow$ & $E_{\xi} \uparrow$ & $\mathcal{M} \downarrow$ \\\hline
A1 & Linear appearance similarity & .735 & .819 & .120  \\
A2 & $L_{lsc}$ \cite{lsc} & .906 & .923 & .052 \\
A3 & Without boundary mask & .907 & .926 & .051 \\
A4 & Ours & .917 & .945 & .038 \\\hline
\end{tabular}
\end{center}
\end{table}

\textbf{Design of $L_{agm}$.}
The design of our $L_{agm}$ loss also is crucial for our framework.
We build some variants of $L_{agm}$ loss:
1) We measure the appearance similarity in Eqn. \ref{eqn:dis} using the L1 distance, denoted as $A_1$;
2) We employ the appearance-guided loss $L_{lsc}$ proposed in \cite{lsc}, denoted as $A_2$
3) We remove the boundary masks $B$ in Eqn. \ref{eqn:bmask}, denoted as $A_3$.
All results are listed in Tab. \ref{tab:acl}, including our full method $A_4$.

Overall, $A_4$ achieves the best results among these variants.
The results of $A_1$ demonstrate that linear distance can not effectively eliminate the impact of pixels with low similarities.
$A_2$ and $A_3$ report inferior performances because edges inside the background or objects introduce noisy information into our training process.

\begin{table}[t]
\begin{center}
\caption{Effect of different hyperparameters.}
\renewcommand\tabcolsep{3pt}
\label{tab:param}
\begin{tabular}{c|c|ccc||c|c|ccc}
\hline
Param. & Value & $F_{\beta} \uparrow$ & $E_{\xi} \uparrow$ & $\mathcal{M} \downarrow$ &Param. & Value & $F_{\beta} \uparrow$ & $E_{\xi} \uparrow$ & $\mathcal{M} \downarrow$ \\\hline
            & 100  & .911 & .932 & .046 &             & 0.5 & .911 & .937 & .042 \\
            & 150  & .914 & .937 & .043 &             & 0.7 & .915 & .942 & .039 \\
$\alpha$    & 200  & .917 & .945 & .038 & $\lambda_s$ & 1   & .917 & .945 & .038 \\
            & 250  & .915 & .942 & .039 &             & 1.2 & .914 & .943 & .038 \\
            & 300  & .914 & .943 & .039 &             & 1.5 & .913 & .941 & .039 \\\hline
            & 0.01 & .869 & .915 & .054 &             & 0.5 & .915 & .943 & .038 \\
            & 0.03 & .910 & .938 & .042 &             & 0.75 & .915 & .945 & .038 \\
$\lambda_a$ & 0.05 & .917 & .945 & .038 & $\lambda_m$ & 1   & .917 & .945 & .038 \\
            & 0.07 & .914 & .945 & .039 &             & 1.25 & .915 & .943 & .039 \\
            & 0.09 & .908 & .942 & .040 &             & 1.5 & .914 & .941 & .039 \\\hline
\end{tabular}
\end{center}
\vspace{-0.2in}
\end{table}

\textbf{Effect of hyperparameters.}
The performance of our framework is affected by several hyperparameters, including $\lambda_s$, $\lambda_a$ and $\lambda_m$ in Eqn. \ref{eqn:lambda}, as well as $\alpha$ in Eqn. \ref{eqn:dis}.
We vary these hyperparameters and exhibit their results in Tab. \ref{tab:param}.

As the results shown, the values $\alpha=200$, $\lambda_s=1$, $\lambda_a=0.05$ and $\lambda_m=1$ work best in practice.
Our framework is robust to $\alpha \in [100, 300]$, and reports the best performance for $\alpha = 200$.
Moreover, our framework achieves comparable performance for various $\lambda_s$ and $\lambda_m$ values within $[0.5, 1.5]$.
On the contrary, our framework is sensitive to $\lambda_a$.
Although we observe robust performance for $\lambda_a \in [0.03, 0.07]$, $\lambda_a$ outside this range (\textit{e.g.}, $\lambda_a = 0.01$) seems to induce significant performance drops.
We attribute this performance drop to the reduced effect of pixels with highly similar appearances.


\begin{table}[t]
\begin{center}
\caption{Different losses for the second stage. }
\renewcommand\tabcolsep{5pt}
\label{tab:loss_s2}
\begin{tabular}{c|ccc|ccc}
\hline
\multirow{2}{*}{Loss} & \multicolumn{3}{c|}{DUT-OMRON} & \multicolumn{3}{c}{ECSSD}  \\\cline{2-7}
 & $F_{\beta} \uparrow$ & $E_{\xi} \uparrow$ & $\mathcal{M} \downarrow$ & $F_{\beta} \uparrow$ & $E_{\xi} \uparrow$ & $\mathcal{M} \downarrow$  \\\hline
BCE & .708 & .834 & .066 & .891 & .924 & .047 \\
BCE+IOU & .726 & .846 & .065 & .894 & .923 & .048 \\
BCE+IOU+SSIM & .716 & .838 & .067 & .886 & .919 & .049 \\
CTLoss & .743 & .862 & .061 & .907 & .914 & .057 \\\hline
IOU & .745 & .863 & .061 & .916 & .938 & .044 \\\hline
\end{tabular}
\end{center}
\end{table}

\textbf{Loss for the second stage.}
Multiple losses have proved their effectiveness on fully-supervised SOD task, such as BCE loss \cite{dss,amulet}, BCE+IOU loss \cite{egnet,minet}, CTLoss \cite{itsd,contour}, BCE+IOU+SSIM loss \cite{basnet}.
The results of using these losses to train our saliency detector with the generated pseudo labels are exhibited in Tab. \ref{tab:loss_s2}.

In summary, training our detector using IOU loss achieves the best results among these losses.
BCE and CTLoss provide pixel-wise supervised signals, which means that training with these losses is easy to overfit the noises and thus degrade the generalization ability of our detector.
Similarly, SSIM is based on regional statistics and thus is sensitive to noisy regions in pseudo labels.
Unlike the above losses, IOU is robust to pixel-level or region-level noises because it bases on global statistics.

\section{Conclusion}
In this paper, we propose an appearance-guided attentive self-paced learning framework for Unsupervised Salient Object Detection.
The proposed framework integrates both self-paced learning (SPL) and appearance guidance into a unified learning framework.
Specifically, we propose an Attentive Self-Paced Learning (ASPL) paradigm that organizes the training samples in a meaningful order to excavate gradually more detailed saliency information.
Moreover, we propose an Appearance Guidance Module (AGM), which formulates the local appearance contrast of each pixel as the probability of saliency boundary and finds the potential boundary of the target objects by maximizing the probability.
Since the appearance can also be interpreted as the feature vector of other modalities, such as depth map, thermal image or optical flow, we further extend our framework to other multi-modality SOD tasks.
Experiments on RGB, RGB-D, RGB-T, video SOD benchmarks prove that our framework achieves state-of-the-art performance against existing USOD methods and is comparable to the latest supervised SOD methods.



\bibliographystyle{IEEEtran}
