\section{Introduction}
\label{introd}
\vspace{-.25cm}
The recent development of RGB-D salient object detection, \textit{i.e.}, RGB-D SOD, is especially fueled by the increasingly accessible 3D imaging sensors~\citep{GiaValSal:book18} and their diverse applications, including image caption generation~\citep{xu2015show}, image retrieval~\citep{ko2004svm,shao2006specific} and video analysis~\citep{liu2008generic,wang2018revisiting}, to name a few. Provided with multi-modality input of an RGB image and its depth map, the task of RGB-D SOD is to effectively identify and segment the most distinctive objects in a scene. 


The state-of-the-art RGB-D SOD approaches~\citep{Li_2021_HAINet,CoNet2020,ji2022dmra,DPANet2020,CMMS2020,li2021joint} typically entail an image-to-mask mapping pipeline that is based on the powerful deep learning paradigms of \textit{e.g.}, VGG16~\citep{VGG15} or ResNet50~\citep{he2016deep}. This strategy has led to excellent performance. 
On the other hand, these RGB-D SOD methods are fully supervised, thus demand a significant amount of pixel-level training annotations. 
This however becomes much less appealing in practical scenarios, owing to the laborious and time-consuming process in obtaining manual annotations.  
It is therefore natural and desirable to contemplating unsupervised alternatives.
Unfortunately, existing unsupervised RGB-D SOD methods, such as global priors~\citep{GP2015}, center prior~\citep{CDCP2017}, and depth contrast prior~\citep{NJU2K}, rely primarily on handcrafted feature representations.
This is in stark contrast to the deep representations learned by their supervised SOD counterparts, which in effect imposes severe limitations on the feature representation power that may otherwise benefit greatly from the potentially abundant unlabeled RGB-D images.



 





\begin{figure*}[t]
\centering
\vspace{-1.05cm}
\includegraphics[width=1\textwidth]{graph/Intro.pdf} 
\vspace{-0.6cm}
\caption{(a) An illustration of deep unsupervised RGB-D saliency detection. ‘Initial label’ is generated by a traditional method. `Baseline' shows the saliency map generated by saliency network trained with initial pseudo-labels. ‘Ours’ shows our final results. (b) Efficiency and effectiveness comparison over a wide range of unsupervised SOD methods on the NLPR benchmark. \textbf{Promoting Saliency From Depth:} our approach achieves a large-margin improvement over the baseline, by engaging depth information to improve pseudo-labels in the training process, without introducing additional computational cost during inference, shown in red arrow.}
\vspace{-0.5cm}
\label{intro}
\end{figure*}



These observations motivate us to explore a new problem of \textit{deep unsupervised RGB-D saliency detection}:
given an unlabeled set of RGB-D images, deep neural network is trained to predict saliency without any laborious human annotations in the training stage.
A relatively straightforward idea is to exploit the outputs from traditional RGB-D method as pseudo-labels, which are internally employed to train the saliency prediction network (`baseline'). 
Moreover, the input depth map may serve as a complementary source of information in refining the pseudo-labels, as it contains cues of spatial scene layout that may help in exposing the salient objects.
Nevertheless, practical examination reveals two main issues: \textit{(1) Inconsistency and large variations in raw depth maps}: as illustrated in Fig.~\ref{intro} (a), similar depth values are often shared by a salient object and its surrounding, making it very difficult in extracting the salient regions from depth without explicit pixel-level supervision; \textit{(2) Noises from unreliable pseudo-labels}: unreliable pseudo-labels may inevitably bring false positive into training, resulting in severe damage in its prediction performance.




To address the above challenges, the following \textit{two} key components are considered in our approach.
First, a depth-disentangled saliency update (DSU) framework is proposed to iteratively refine \& update the pseudo-labels by engaging the depth knowledge.
Here a depth-disentangled network is devised to explicitly learn the discriminative saliency cues and non-salient background from raw depth map, denoted as saliency-guided depth $D_{Sal}$ and non-saliency-guided depth $D_{NonSal}$, respectively.
This is followed by a depth-disentangled label update (DLU) module that takes advantage of $D_{Sal}$ to emphasize saliency response from pseudo-label; it also utilizes $D_{NonSal}$ to eliminate the background influence, thus facilitating more trustworthy supervision signals in training the saliency network.
Note the DSU module is not engaged at test time. Therefore, at test time, our trained model takes as input only an RGB image, instead of involving both RGB and depth as input and in the follow-up computation.
Second, an attentive training strategy is introduced to alleviate the issue of noisy pseudo-labels; it is achieved by re-weighting the training samples in each training batch to focus on those more reliable pseudo-labels.
As demonstrated in Fig.~\ref{intro} (b), our approach works effectively and efficiently in practice. It significantly outperforms existing unsupervised SOD methods on the widely-used NLPR benchmark. Specifically, it improves over the baseline by 37\%, a significant amount without incurring extra computation cost. 
Besides, the test time execution of our approach is at 35 frame-per-second (FPS), the fastest among all RGB-D unsupervised methods, and on par with the most efficient RGB-based methods.
In summary, our main contributions are as follows:



\vspace{-0.2cm}
\begin{itemize}
\item
To our knowledge, our work is the first in exploring deep representation to tackle the problem of unsupervised RGB-D saliency detection. 
This is enabled by two key components in the training process, namely the DSU strategy to produce \& refine pseudo-labels, and the attentive training strategy to alleviate the influence of noisy pseudo-labels.
It results in a light-weight architecture that engages only RGB data at test time (\textit{i.e.,} w/o depth map), achieving a significant improvement without extra computation cost.
\item
Empirically, our approach outperforms state-of-the-art unsupervised methods on four public benchmarks. 
Moreover, it runs in real time at 35 FPS, much faster than existing unsupervised RGB-D SOD methods, and at least on par with the fastest RGB counterparts.
\item 
Our approach could be adapted to work with fully-supervised scenario. As demonstrated in Sec.~\ref{ablfully}, augmented with our proposed DSU module, the empirical results of existing RGB-D SOD models have been notably improved. 
\end{itemize}











\vspace{-.45cm}
\section{Related Work}
\vspace{-.3cm}
\RQ{Remarkable progresses have been made recently in salient object detection~\citep{ma2021pyramidal,xu2021locate,MINet,GateNet,tsai2018image,liu2018picanet},} where the performance tends to deteriorate in the presence of cluttered or low-contrast backgrounds.
Moreover, promising results are shown by a variety of recent efforts~\citep{chen2021depth,Graph2020,chen2020improved,SSLSOD,desingh2013depth,ASIF-Net,2020mmnet,shigematsu2017learning} that integrate effective depth cues to tackle these issues. Those methods, however, typically demand extensive annotations, which are labor-intensive and time-consuming.
This naturally leads to the consideration of unsupervised SODs that do not rely on such annotations.

Prior to the deep learning era~\citep{wang2019sodsurvey,really2020}, traditional RGB-D saliency methods are mainly based on the manually-crafted RGB features and depth cues to infer a saliency map.
Due to their lack of reliance on manual human annotation, these traditional methods can be regarded as early manifestations of unsupervised SOD.
Ju \textit{et al}.~\citep{NJU2K} consider the incorporation of a prior induced from anisotropic center-surround operator, and an additional depth prior.
Ren \textit{et al}.~\citep{GP2015} also introduce two priors: normalized depth prior and the global-context surface orientation prior.
In contrary to direct utilization of the depth contrast priors, local background enclosure prior is explicitly developed by~\citep{LBE2016}.
Interested readers may refer to \citep{D3Net2020,NLPR,zhou2020rgb} for more in-depth surveys and analyses.
However, by relying on manually-crafted priors, these methods tend to have inferior performance.


\RQ{Meanwhile, considerable performance gain has been achieved by recent efforts in RGB-based unsupervised SOD~\citep{SBF2017,USD2018,DeepUSPS2019,hsu2018co},} which instead construct automated feature representations using deep learning. 
A typical strategy is to leverage the noisy output produced by traditional methods as pseudo-label (\textit{i.e.}, supervisory signal) for training saliency prediction net.
The pioneering work of Zhang \textit{et al}.~\citep{SBF2017} fuses the outputs of multiple unsupervised saliency models as guiding signals in CNN training. 
In~\citep{USD2018}, competitive performance is achieved by fitting a noise modeling module to the noise distribution of pseudo-label. 
Instead of directly using pseudo-labels from handcrafted methods, Nguyen \textit{et al}.~\citep{DeepUSPS2019} further refine pseudo-labels via a self-supervision iterative process. 
\RQ{Besides, deep unsupervised learning has been considered by~\citep{hsu2018unsupervised,tsai2019deep} for the co-saliency task, where superior performance has been obtained by engaging a fusion-learning scheme~\citep{hsu2018unsupervised}, or utilizing dedicated loss terms~\citep{tsai2019deep}.}
These methods demonstrate that the incorporation of powerful deep neural network brings better feature representation than those unsupervised SOD counterparts based on handcrafted features.


In this work, a principled research investigation on \textit{deep unsupervised RGB-D saliency detection}.
is presented for the first time. 
Different from existing deep RGB-based unsupervised SOD~~\citep{SBF2017,USD2018} that use pseudo-labels from the handcrafted methods directly,
our work aims to refine and update pseudo-labels to remove the undesired noise. 
And compared to the refinement method~\citep{DeepUSPS2019} using self-supervision technique, our approach instead adopts disentangled depth information to promote pseudo-labels.  


\let\thefootnote\relax\footnotetext{\hspace{-.57cm}In this paper, unsupervised learning refers to learning without using human annotation~\citep{SBF2017}.}



\vspace{-.3cm}
\section{Methodology}
\vspace{-.3cm}
\label{ddsu}
\subsection{Overview} 
\vspace{-.2cm}
\label{ooverview}
Fig.~\ref{overview} presents an overview of our Depth-disentangled Saliency Update (DSU) framework.
Overall our DSU strives to significantly improve the quality of pseudo-labels, that leads to more trustworthy supervision signals for training the saliency network.
It consists of three key components.
\textbf{First} is a saliency network responsible for saliency prediction, whose initial supervision signal is provided by traditional handcrafted method without using human annotations.
\textbf{Second}, a depth network and a depth-disentangled network are designed to decompose depth cues into saliency-guided depth $D_{Sal}$ and non-saliency-guided depth $D_{NonSal}$, to explicitly depict saliency cues in the spatial layout.
\textbf{Third}, a depth-disentangled label update (DLU) module is devised to refine and update the pseudo-labels, by engaging the learned $D_{Sal}$ and $D_{NonSal}$.
The updated pseudo-labels could in turn provide more trustworthy supervisions for the saliency network.
Moreover, an attentive training strategy (ATS) is incorporated when training the saliency network, tailored for noisy unsupervised learning by mitigating the ambiguities caused by noisy pseudo-labels.
We note that the DSU and ATS are not performed at test time, so does not affect the inference speed. In other words, the inference stage involves only the black dashed portion of the proposed network architecture in Fig.~\ref{overview}, \textit{i.e.}, only RGB images are used for predicting saliency, which enables very efficient detection.



\begin{figure*}[t]
\centering
\vspace{-.7cm}
\includegraphics[width=1\textwidth]{graph/overall} 
\vspace{-0.7cm}
\caption{Overview of the proposed method. The saliency network is trained with the iteratively updated pseudo-labels. The depth network and depth-disentangled network are designed to decompose raw depth into saliency-guided depth $D_{Sal}$ and non-saliency-guided depth $D_{NonSal}$, which are subsequently fed into the depth-disentangled label update (DLU) module to refine and update pseudo-labels. The inference stage involves only the black dashed portion.

\vspace{-0.4cm}
\label{overview}
\end{figure*}





\vspace{-0.2cm}
\subsection{Depth-disentangled Saliency Update}
\vspace{-0.1cm}

The scene geometry embedded in the depth map can serve as a complementary information source to refine the pseudo-labels. Nevertheless, as discussed in the introduction section, a direct adoption of the raw depth may not necessarily lead to good performance due to the large value variations and inherit noises in raw depth map. Without explicit pixel-wise supervision, the saliency model may unfortunately be confused by nearby background regions having the same depth but with different contexts; meanwhile, depth variations in salient or background regions may also result in false responses. These observations motivate us to design a depth-disentangled network to effectively capture discriminative saliency cues from depth, and a depth-disentangled label update (DLU) strategy to improve and refine pseudo-labels by engaging the disentangled depth knowledges.



\vspace{-0.2cm}
\subsubsection{Depth-disentangled Network}
\vspace{-0.1cm}
\label{ddnet}
The depth-disentangled network aims at capturing valuable saliency as well as redundant non-salient cues from raw depth map.
As presented in the bottom of Fig.~\ref{overview}, informative depth feature $F^{Depth}$ is first extracted from the depth network under the supervision of raw depth map, using the mean square error (MSE) loss function, \textit{i.e.,} $l_{d,1}$. 
The $F^{Depth}$ is then decomposed into saliency-guided depth $D_{Sal}$ and non-saliency-guided depth $D_{NonSal}$ following two principles: 1) explicitly guiding the model to learn saliency-specific cues from depth; 2) ensuring the coherence between the disentangled and original depth features.


Specifically, in the bottom right of Fig.~\ref{overview}, we first construct the spatial supervision signals for the depth-disentangled network. Given the rough saliency prediction $Sal_{pred}$ from the saliency network and the raw depth map $D_{map}$, the (non-)saliency-guided depth masks, \textit{i.e.,} $D_{Sal}^{Label}$ and $D_{NonSal}^{Label}$, can be obtained by multiplying $Sal_{pred}$ (or $1-Sal_{pred}$) and depth map $D_{map}$ in a spatial attention manner. Since the predicted saliency may contain errors introduced from the inaccurate pseudo-labels,
we employ a holistic attention (HA) operation~\citep{CPD2019} to smooth the coverage area of the predicted saliency, so as to effectively perceive more saliency area from depth. 
Formally, the (non-)saliency-guided depth masks are generated by:
\begin{equation}
D_{Sal}^{Label}= \Psi_{\max}(\mathcal{F}_{G}(Sal_{pred}, k), Sal_{pred}) \otimes D_{map}, 
\label{eq1}
\end{equation}
\begin{equation}
D_{NonSal}^{Label}= \Psi_{\max}(\mathcal{F}_{G}(1-Sal_{pred}, k), 1-Sal_{pred}) \otimes D_{map},
\label{eq2}
\end{equation}
where
$\mathcal{F}_{G}(\cdot, k)$ represents the HA operation, which is implemented using the convolution operation with Gaussian kernel $k$ and zero bias; the size and standard deviation of the Gaussian kernel $k$ are initialized with 32 and 4, respectively, which are then finetuned through the training procedure;
$\Psi_{\max}(\cdot, \cdot)$ is a maximum function to preserve the higher values from the Gaussian filtered map and the original map;
$\otimes$ denotes pixel-wise multiplication.


Building upon the guidance of $D_{Sal}^{Label}$ and $D_{NonSal}^{Label}$, $F^{Depth}$ is fed into D-Sal CNN and D-NonSal CNN to explicitly learn valuable saliency and redundant non-salient cues from depth map, generating $D_{Sal}$ and $D_{NonSal}$, respectively. The loss functions here (\textit{i.e.,} $l_{Sal}$ and $l_{NonSal}$) are MSE loss. Detailed structures of D-Sal and D-NonSal CNNs are in the appendix.
To further ensure the coherence between the disentangled and original depth features, a consistency loss, ${l}_{con}$, is employed as:
\begin{equation}
{{l}}_{con} = {\frac{1}{{H}{\times}{W}{\times}{C}}}{\sum_{i=1}^{H} \sum_{j=1}^{W} \sum_{k=1}^{C} {\parallel}{F}_{i, j, k}^{Depth}, \tilde{F}_{i, j, k}^{Depth}{\parallel}_{2}},
\end{equation}
where
$\tilde{F}^{Depth}$ is the sum of the disentangled ${F}^{Depth}_{Sal}$ and ${F}^{Depth}_{NonSal}$, which denotes the regenerated depth feature;
 $H$, $W$ and $C$ are the height, width and channel of ${F}^{Depth}$ and $\tilde{F}^{Depth}$;

${\parallel}{\cdot}{\parallel}_{2}$ represents Euclidean norm. Here, $\tilde{F}^{Depth}$ is also under the supervision of depth map, using MSE loss, \textit{i.e.,} $l_{d,2}$.

Then, the overall training objective for the depth network and the depth-disentangled network is as: 
\begin{equation}
\mathcal{L}_{depth} = \frac{1}{5N} \sum_{n=1}^{N} ( l_{d, 1}^n + l_{d, 2}^n + l_{Sal}^n + l_{NonSal}^n + {\lambda}l_{con}^n),
\end{equation}
where $n$ denotes the $n_{th}$ sample in a mini-batch with $N$ training samples;
$\lambda$ is set to 0.02 in the experiments to balance the consistency loss $l_{con}$ and other loss terms.
As empirical evidences in Fig.~\ref{disdepthvisual} suggested, comparing to the raw depth map, $D_{Sal}$ is better at capturing discriminative cues of the salient objects, while $D_{NonSal}$ better depicts the complementary background context.


In the next step, the learned $D_{Sal}$ and $D_{NonSal}$ are fed into our Depth-disentangled Label Update, to obtain the improved pseudo-labels.



\begin{figure*}[t]
\centering
\vspace{-0.6cm}
\includegraphics[width=1\textwidth]{graph/abl_depth} 
\vspace{-.66cm}
\caption{The internal inspections of the proposed DSU. It is observed that the updated label exhibits more reliable saliency signals than initial pseudo-label.}
\vspace{-.26cm}
\label{disdepthvisual}
\end{figure*}







\vspace{-0.2cm}
\subsubsection{Depth-disentangled Label Update}
\vspace{-0.1cm}
To maintain more reliable supervision signals in training, a depth-disentangled label update (DLU) strategy is devised to iteratively refine \& update pseudo-labels.
Specifically, as shown in the upper stream of Fig.~\ref{overview}, using the obtained $Sal_{pred}$, $D_{Sal}$ and $D_{NonSal}$, the DLU simultaneously highlights the salient regions in the coarse saliency prediction by the sum of $Sal_{pred}$ and $D_{Sal}$,
and
suppresses the non-saliency negative responses by subtracting $D_{NonSal}$ in a pixel-wise manner. This process can be formulated as:
\begin{equation}
\mathcal{S}_{temp} = Sal_{pred}^{i, j} + D_{Sal}^{i, j} - D_{NonSal}^{i, j}  \bigg|_{i \in [1, H]; j \in [1, W]}.
\end{equation}

To avoid the value overflow of the obtained $\mathcal{S}_{temp}$ (\textit{i.e.}, removing negative numbers and normalizing the results to the range of [0, 1]), a thresholding operation and a normalization process are performed as:
\begin{equation}
\mathcal{S}_\mathcal{N} = \frac{\mathcal{S}_{n}^{i, j} - min(\mathcal{S}_{n})}{max(\mathcal{S}_{n}) - min(\mathcal{S}_{n})}, 
{\rm where}\ \mathcal{S}_{n} =
\begin{cases}
0,  & \mbox{if }\mathcal{S}_{temp}^{i, j} < 0 \\
\mathcal{S}_{temp}^{i, j}, & \mbox{others}
\end{cases},
i \in [1, H]; j \in [1, W],
\end{equation}
\iffalse
\begin{equation}
\mathcal{S}_\mathcal{N} = \frac{\mathcal{S}_{n}^{i, j} - min(\mathcal{S}_{n})}{max(\mathcal{S}_{n}) - min(\mathcal{S}_{n})}  \bigg|_{i \in [1, H]; j \in [1, W]}
\end{equation}
\fi
where $min(\cdot)$ and $max(\cdot)$ denote the minimum and maximum functions. 
Finally, a fully-connected conditional random field (CRF~\citep{hou2017deeply}) is applied to $\mathcal{S}_\mathcal{N}$, to generate the enhanced saliency map $\mathcal{S}_{map}$ as the updated pseudo-labels.
The empirical result of the proposed DSU step is showcased in Fig.~\ref{disdepthvisual}, with quantitative results presented in Table~\ref{tabupdate}. 
These internal evidences suggest that the DSU leads to noticeable improvement after merely several iterations.


\iffalse
\vspace{-0.3cm}
\subsubsection{Discussion}
\vspace{-0.1cm}

\begin{wrapfigure}{r}{8.1cm}
  \centering
\vspace{-1cm}
  \tabcaption{\footnotesize{Discussion on different pseudo-label updating settings.}}
  \vspace{0.1cm}
   \resizebox{!}{0.98 cm}{
    \begin{tabular}{cc|cc|cc}
     \hline \toprule
    \multicolumn{2}{c|}{\multirow{2}[2]{*}{{Different Pseudo-label Updating Setups}}} & \multicolumn{2}{c|}{\textbf{NJUD}} & \multicolumn{2}{c}{\textbf{NLPR}} \\
  
  & & \multicolumn{1}{c}{$F_{\beta}\uparrow$}  & \multicolumn{1}{c|}{$\mathcal{M}\downarrow$ }    & \multicolumn{1}{c}{$F_{\beta}\uparrow$}  & \multicolumn{1}{c}{$\mathcal{M}\downarrow$ } \\ \midrule
    \multicolumn{2}{l|}{(a) DSU w/o $D_{NonSal}$ }  	                & 0.698	& 0.143 	& 0.715 	& 0.072   \\ 
     \multicolumn{2}{l|}{(b) DSU using $D^{Label}_{Sal}$, $D^{Label}_{NonSal}$}  	& 0.691	& 0.154 	& 0.707 	& 0.079   \\ 
    \multicolumn{2}{l|}{(c) DSU using $D_{Sal}$, $D_{NonSal}$ (\textbf{Ours})}  		& 0.719 	& 0.135 	& 0.745	& 0.065  \\
     \bottomrule \hline 
    \end{tabular}%
  \label{ABL}}%
  \vspace{-0.3cm}
\end{wrapfigure}
\textit{Is depth-disentanglement necessary?}
In our DSU, we disentangle the raw depth map into valuable salient parts as well as redundant non-salient areas to simultaneously highlight the salient regions and suppress the non-salient negative responses. To verify the necessity of depth disentanglement, we remove the $D_{NonSal}$ branch in the depth-disentangled network and only use the $D_{Sal}$ to update pseudo-labels in DLU. Empirical results in Table~\ref{ABL} reveal that removing $D_{NonSal}$ (row (a)) leads to significant inferior performance when comparing with the original DSU (row (c)), which proves the superiority of our DSU design.



\textit{How about using raw depth labels in DSU?}
We also consider to directly utilize the raw disentangled depth labels, \textit{i.e.,} $D^{Label}_{Sal}$ and $D^{Label}_{NonSal}$, to update pseudo-labels in DSU. 
However, a direct adoption of the raw depth labels does not lead to good performance, as empirically exhibited in Table~\ref{ABL} (b).
The DSU using raw labels performs worse compared to our original DSU design in row (c).
This is partly due to the large value variations and intrinsic label noises in raw depth map as discussed before.
On the other hand, benefiting from the capability of deep-learning-based networks to learn general representations from large-scale data collections, using $D_{Sal}$ and $D_{NonSal}$ in our DSU strategy is able to eliminate the potential biases in a single raw depth label.
Visual evidences as exhibited in the 4$^{th}$-7$^{th}$ columns of Fig.~\ref{disdepthvisual} also support our claim.
\fi





\iffalse
\begin{table}[t]
  \centering
  \caption{Discussion on different pseudo-label updating settings.}
   \resizebox{!}{1 cm}{
    \begin{tabular}{cc|m{0.5cm}<{\centering}m{0.5cm}<{\centering}|m{0.5cm}<{\centering}m{0.5cm}<{\centering}}
     \hline \toprule
    \multicolumn{2}{c|}{\multirow{2}[2]{*}{{Different Pseudo-label Updating Setups}}} & \multicolumn{2}{c|}{\textbf{NJUD}} & \multicolumn{2}{c}{\textbf{NLPR}} \\
  
  & & \multicolumn{1}{c}{$F_{\beta}\uparrow$}  & \multicolumn{1}{c|}{$\mathcal{M}\downarrow$ }    & \multicolumn{1}{c}{$F_{\beta}\uparrow$}  & \multicolumn{1}{c}{$\mathcal{M}\downarrow$ } \\ \midrule
    \multicolumn{2}{l|}{(a) DSU w/o $D_{NonSal}$ }  	                & 0.698	& 0.143 	& 0.715 	& 0.072   \\ 
     \multicolumn{2}{l|}{(b) DSU using $D^{Label}_{Sal}$, $D^{Label}_{NonSal}$}  	& 0.691	& 0.154 	& 0.707 	& 0.079   \\ 
    \multicolumn{2}{l|}{(c) DSU using $D_{Sal}$, $D_{NonSal}$ (\textbf{Ours})}  		& 0.719 	& 0.135 	& 0.745	& 0.065  \\
     \bottomrule \hline 
    \end{tabular}%
  \label{ABL}}%
\end{table}%
\fi




\vspace{-.2cm}
\subsection{Attentive Training Strategy}
\vspace{-.1cm}
\label{ATStrain}
When training the saliency network using pseudo-labels, an attentive training strategy (ATS) is proposed to tailor for the deep unsupervised learning context, to reduce the influence of ambiguous pseudo-labels, and concentrate on the more reliable training examples.
This strategy is inspired by the human learning process of understanding new knowledge, that is, from general to specific understanding cycle~\citep{dixon1999organizational,peltier2005reflective}.
The ATS alternates between two steps to re-weigh the training instances in a mini-batch. 
\iffalse
To reduce the influence of ambiguous pseudo-labels, and concentrate on the more reliable training examples, an attentive training strategy is proposed here to tailor for our deep unsupervised learning context.
This is achieved by alternating between two steps to re-weigh training instances in a mini-batch for the saliency network. 

This alternative way is specially designed for our DSU.  when performing each label update, the model first learn general representations from updated pseudo-labels. 
\fi

To be specific, we first start by settling the related loss functions. For the $n_{th}$ sample in a mini-batch with $N$ training samples, we define the binary cross-entropy loss between the predicted saliency $Sal_{pred}^{n}$ and the pseudo-label $\mathcal{S}_{map}^{n}$ as:
\begin{equation}
l_n = - (\mathcal{S}_{map}^{n} \cdot \log{Sal_{pred}^{n}} + (1 - \mathcal{S}_{map}^{n}) \cdot \log(1 - Sal_{pred}^{n})).
\end{equation}
Then, the training objective for the saliency network in current mini-batch is defined as an attentive binary cross-entropy loss $\mathcal{L}_{sal}$, which can be represented as follows:
\begin{equation}
\mathcal{L}_{sal} = \frac{1}{\sum_{n=1}^{N} \alpha_n} \sum_{n=1}^{N} (\alpha_n \cdot l_n),
\mathcal{\alpha}_{n} =
\begin{cases}
1,  & \mbox{step one,}  \\
\frac{ \sum_{i \in N}^{ i \ne n} e^{l_i}} {\sum_{i \in N} e^{l_i}} , & \mbox{step two,}
\end{cases}
\label{eq8}
\end{equation}
where $\alpha_n$ represents the weight of the $n_{th}$ training sample at current training mini-batch.

\begin{wrapfigure}{r}{7cm}
\centering
\vspace{-.4cm}
\includegraphics[width=.5\textwidth]{graph/loss}
\vspace{-.8cm}
\caption{Analysis of the proposed attentive training strategy, when evaluated on NJUD testset. {The `backbone' refers to the saliency network trained with initial pseudo-labels.}}
\vspace{-.5cm}
\label{trainloss}
\end{wrapfigure}
The ATS starts from a uniform weight for each training sample in step one to learn general representations across a lot of training data.
Step two decreases the importance of ambiguous training instances through the imposed attentive loss; the higher the loss value, the less weight an instance is to get.

In this paper, we define step one and two as a training round ($2\tau$ epochs).
During each training round, the saliency loss $\mathcal{L}_{sal}$ and depth loss $\mathcal{L}_{depth}$ are optimized simultaneously to train their network parameters. 
The proposed DLU is taken at the end of each training round to update the pseudo-labels for the saliency network;
meanwhile, $D_{Sal}^{Label}$ and $D_{NonSal}^{Label}$ in Eqs.\ref{eq1} and \ref{eq2} are also updated using the improved $Sal_{pred}$.
This allows the network to make efficient use of the updated pseudo labels.
As suggested by Fig.~\ref{trainloss} and Table~\ref{tabats}, the use of our attentive training strategy leads to more significant error reduction compared with the backbone solely using uniform weight or attentive weight for each instance.
Besides, when our DSU and the attentive training strategy are both incorporated to the backbone, better results are achieved.







\vspace{-0.2cm}
\section{Experiments and Analyses}
\vspace{-0.1cm}
\label{experi}



\label{sec:setup}

\vspace{-0.2cm}
\subsection{Datasets and Evaluation Metrics}
\vspace{-0.1cm}
Extensive experiments are conducted over four large-scale RGB-D SOD benchmarks.   
NJUD~\citep{NJU2K} in its latest version consists of 1,985 samples, that are collected from the Internet and 3D movies;  
NLPR~\citep{NLPR} has 1,000 stereo images collected with Microsoft Kinect; 
STERE~\citep{STERE} contains 1,000 pairs of binocular images downloaded from the Internet; 
DUTLF-Depth~\citep{DMRA2019} has 1,200 real scene images captured by a Lytro2 camera. 
We follow the setup of~\citep{D3Net2020} to construct the training set, which includes 1,485 samples from NJUD and 700 samples from NLPR, respectively. 
Data augmentation is also performed by randomly rotating, cropping and flipping the training images to avoid potential overfitting. 
The remaining images are reserved for testing. 


Here, five widely-used evaluation metrics are adopted: E-measure ($E_{\xi}$) \citep{fan2018enhanced}, weighed F-measure ($F_{\beta}^{w}$)~\citep{margolin2014evaluate}, F-measure ($F_{\beta}$)~\citep{achanta2009frequency}, Mean Absolute Error (MAE or $\mathcal{M}$)~\citep{borji2015salient}, and inference time(s) or FPS (Frames Per Second). 

Implementation details are presented in the Appx~\ref{imple}. \textit{\href{https://github.com/jiwei0921/DSU}{Source code is publicly available.}}





\begin{figure*}[htbp]
\centering
\vspace{-0.3cm}
\includegraphics[width=1\textwidth]{graph/UnSOD} 
\vspace{-0.8cm}
\caption{\footnotesize{Qualitative comparison with unsupervised saliency detection methods. GT denotes ground-truth.}
\vspace{-0.5cm}
\label{unsodvis}
\end{figure*}





\begin{table*}[htbp]
  \centering

  \caption{\footnotesize{Quantitative comparison with unsupervised SOD methods. {`Backbone'} refers to the saliency feature extraction network ~\citep{CPD2019} adopted in our pipeline, \textit{i.e.} the one without the two proposed key components. 
  The RGB-based methods are specifically marked by $^\dagger$. UnSOD is shorthand for unsupervised SOD. We also provide the results of existing fully supervised methods that can be referenced in Table~\ref{fully_SOD}. 
  }}
 
\resizebox{!}{2.65 cm}{
    \begin{tabular}{c|c|c|m{0.55cm}<{\centering}m{0.55cm}<{\centering}m{0.55cm}<{\centering}m{0.6cm}<{\centering}|m{0.55cm}<{\centering}m{0.55cm}<{\centering}m{0.55cm}<{\centering}m{0.6cm}<{\centering}|m{0.55cm}<{\centering}m{0.55cm}<{\centering}m{0.55cm}<{\centering}m{0.6cm}<{\centering}|m{0.55cm}<{\centering}m{0.55cm}<{\centering}m{0.55cm}<{\centering}m{0.56cm}<{\centering}}
     \hline \toprule
    \multicolumn{2}{c|}{\multirow{2}[1]{*}{*}} & \multicolumn{1}{c|}{{Inference}} & \multicolumn{4}{c|}{\textbf{NJUD}} & \multicolumn{4}{c|}{\textbf{NLPR}} & \multicolumn{4}{c|}{\textbf{STERE}} & \multicolumn{4}{c}{\textbf{DUTLF-Depth}} \\
    \multicolumn{2}{c|}{} & Time(s)$\downarrow$ & $E_{\xi} \uparrow$  & $F_{\beta}^{w}\uparrow$  & $F_{\beta}\uparrow$  & $\mathcal{M}\downarrow$    & $E_{\xi} \uparrow$  & $F_{\beta}^{w}\uparrow$  & $F_{\beta}\uparrow$  & $\mathcal{M}\downarrow$   & $E_{\xi} \uparrow$  & $F_{\beta}^{w}\uparrow$  & $F_{\beta}\uparrow$  & $\mathcal{M}\downarrow$  & $E_{\xi} \uparrow$  & $F_{\beta}^{w}\uparrow$  & $F_{\beta}\uparrow$  & $\mathcal{M}\downarrow$  \\  
\midrule 
   
    \multirow{11}[1]{*}{\emph{\rotatebox{90}{\textbf{Handcrafted UnSOD}}}}
          & RBD$^\dagger$~\citep{RBD2014}     & 0.189      & .684	&.387	&.556	&.256	&.765	&.388	&.590	&.211	&.730	&.443	&.610	&.223	&.733	&.447	&.619	&.222 \\
          & MST$^\dagger$~\citep{MST2016}     & 0.030     & .670 & .291 & .436 & .281 & .762 & .257 & .491 & .199 & .681 & .312 & .447 & .269 & .678 & .254 & .401 & .279 \\
          & BSCA$^\dagger$~\citep{BSCA2015}  & 2.665     & .756 & .446 & .623 & .216 & .745 & .376 & .554 & .178 & .803 & .497 & .676 & .179 & .808 & .479 & .682 & .181 \\
          & DSR$^\dagger$~\citep{DSR2013}     & 0.376     &.739	 &.436	&.594	&.196	&.757	&.451	&.545	&.120	&.785	&.486	&.645	&.165	&.797	&.478	&.640	&.164 \\
          & ACSD~\citep{NJU2K}     & 0.718    & .790 & .448 & .696 & .198 & .751 & .327 & .547 & .171 & .793 & .425 & .661 & .200 & .250 & .210 & .188 & .668 \\
          & DES~\citep{RGBD135}    & 7.790      & .421 & .241 & .165 & .448 & .735 & .259 & .583 & .301 & .673 & .383 & .592 & .297 & .733 & .386 & .668 & .280 \\
          & LHM~\citep{NLPR}    & 2.130   & .722 & .311 & .625 & .201 & .772 & .320 & .520 & .119 & .772 & .360 & .703 & .171 & .767 & .350 & .659 & .174 \\
          & GP~\citep{GP2015}    & 12.98  & .730 & .323 & .666 & .204 & .813 & .347 & .670 & .144 & .785 & .371 & .710 & .182 & -     & -     & -     & - \\          
          & CDB~\citep{CDB2018}    & 0.600    & .752	& .408	& .650	& .200	& .810	& .388	& .618	& .108	& .808	& .436	& .713	& .166 & - & - & -	 & - \\
          & SE~\citep{SE2016}       & 1.570   & .780 & .518 & .735 & .164 & .853 & .578 & .701 & .085 & .825 & .546 & .747 & .143 & .730 & .339 & .474 & .196 \\
          & DCMC~\citep{DCMC}    & 1.210     & .796 & .506 & .715 & .167 & .684 & .265 & .328 & .196 & .832 & .529 & .743 & .148 & .712 & .290 & .406 & .243 \\
          & MB~\citep{MB2017}      & -     & .643 & .369 & .492 & .202 & .814 & .574 & .637 & .089 & .693 & .455 & .572 & .178 & .691 & .464 & .577 & .156 \\
          & CDCP~\citep{CDCP2017}     & 5.720    & .751 & .522 & .618 & .181 & .785 & .512 & .591 & .114 & .797 & .596 & .666 & .149 & .794 & .530 & .633 & .159 \\
    \midrule \midrule

    \multirow{4}[4]{*}{\emph{\rotatebox{90}{\textbf{Deep UnSOD}}}}
    & USD$^\dagger$~\citep{USD2018}    & 0.0180      & .768     & .565     & .630     & .163     & .786     & .536    & .580    & .119    & .796     & .572     & .670    & .146     & .795     & .545     & .650    & .157 \\
          & DeepUSPS$^\dagger$~\citep{DeepUSPS2019}    & 0.0292    & .771     & .576     & .647     & .159     & .809     & .622     & .639     & .088     & .806     & .632    & .682     & .124    & .798    & .573     & .654    & .149 \\
\cmidrule{2-19}          & Backbone    & 0.0286    & .759 & .510 & .627 & .186 & .760 & .479 & .570 & .126 & .794 & .555 & .666 & .158 & .798 & .512 & .644 & .167 \\
        & \cellcolor[rgb]{ .851,  .851,  .851}{$\Delta$ gains}    & \cellcolor[rgb]{ .851,  .851,  .851}-    & \cellcolor[rgb]{ .851,  .851,  .851}{{\color{red}$\uparrow$5\%}}     & \cellcolor[rgb]{ .851,  .851,  .851}{{\color{red}$\uparrow$17\%}}     & \cellcolor[rgb]{ .851,  .851,  .851}{{\color{red}$\uparrow$15\%}}     & \cellcolor[rgb]{ .851,  .851,  .851}{{\color{red}$\downarrow$27\%}}     & \cellcolor[rgb]{ .851,  .851,  .851}{{\color{red}$\uparrow$16\%}}     & \cellcolor[rgb]{ .851,  .851,  .851}{{\color{red}$\uparrow$37\%}}     & \cellcolor[rgb]{ .851,  .851,  .851}{{\color{red}$\uparrow$31\%}}     & \cellcolor[rgb]{ .851,  .851,  .851}{{\color{red}$\downarrow$48\%}}     & \cellcolor[rgb]{ .851,  .851,  .851}{{\color{red}$\uparrow$8\%}}     & \cellcolor[rgb]{ .851,  .851,  .851}{{\color{red}$\uparrow$22\%}}     & \cellcolor[rgb]{ .851,  .851,  .851}{{\color{red}$\uparrow$16\%}}     & \cellcolor[rgb]{ .851,  .851,  .851}{{\color{red}$\downarrow$37\%}}     & \cellcolor[rgb]{ .851,  .851,  .851}{{\color{red}$\uparrow$7\%}}     & \cellcolor[rgb]{ .851,  .851,  .851}{{\color{red}$\uparrow$27\%}}     & \cellcolor[rgb]{ .851,  .851,  .851}{{\color{red}$\uparrow$18\%}}     & \cellcolor[rgb]{ .851,  .851,  .851}{{\color{red}$\downarrow$36\%}} \\
          & \textbf{Ours}     & 0.0286    & \textbf{.797} & \textbf{.597} & \textbf{.719} & \textbf{.135} & \textbf{.879} & \textbf{.657} & \textbf{.745} & \textbf{.065} & \textbf{.857} & \textbf{.678} & \textbf{.774} & \textbf{.099} & \textbf{.854} & \textbf{.650} & \textbf{.763} & \textbf{.107} \\
     \bottomrule \hline 
    \end{tabular}}%
  \label{comUnSOD}%
  \vspace{-0.4cm}
\end{table*}%







\vspace{-0.3cm}
\subsection{Comparison with the State-of-the-arts}
\label{sec:com}
\vspace{-0.2cm}

Our approach is compared with 15 unsupervised SOD methods, \textit{i.e.}, without using any human annotations. Their results are either directly furnished by the authors of the respective papers, or generated by re-running their original implementations.
{In this paper, we make the first attempt to address deep-learning-based unsupervised RGB-D SOD. Since existing unsupervised RGB-D methods are all based on handcrafted feature representations, we additionally provide several RGB-based methods (\textit{e.g.}, USD and DeepUSPS) for reference purpose only. 
This gives more observational evidences for the related works. These RGB-based methods are specifically marked by $^\dagger$ in Table~\ref{comUnSOD}.}




Quantitative results are listed in Table~\ref{comUnSOD}, where our approach clearly outperforms the state-of-the-art unsupervised SOD methods in both RGB-D and RGB only scenarios. 
This is due to our DSU framework that leads to trustworthy supervision signals for saliency network.
Furthermore, our network design leads to a light-weight architecture in the inference stage, shown as the black dashed portion in Fig.~\ref{overview}. This enables efficient \& effective detection of salient objects and brings a large-margin improvement over the backbone network without introducing additional depth input and computational costs, as shown in Fig.~\ref{intro} and Table~\ref{comUnSOD}.
Qualitatively, saliency predictions of competing methods are exhibited in Fig.~\ref{unsodvis}. These results consistently proves the superiority of our method.



\vspace{-.3cm}
\subsection{Analysis of the Empirical Results}
\vspace{-.2cm}
\label{experanalysis}
The focus here is on the evaluation of the contributions from each of the components, and the evaluation of the obtained pseudo-labels as intermediate results. 







\textbf{Effect of each component.}
In Table~\ref{tableabl}, we conduct ablation study to investigate the contribution of each component.
To start with, we consider the backbone (a), where the saliency network is trained with initial pseudo-labels.
As our proposed ATS and DSU are gradually incorporated, increased performance has been observed on both datasets.
Here we first investigate the benefits of ATS by applying it to the backbone and obtaining (b). We observe increased F-measure scores of 3\% and 5.8\% on NJUD and NLPR benchmarks, respectively.
This clearly shows that the proposed ATS can effectively improve the utilization of reliable pseudo-labels, by re-weighting ambiguous training data.
We then investigate in detail all components in our DSU strategy.
The addition of the entire DSU leads to (f), which significantly improves the F-measure metric by 11.3\% and 23.5\% on each of the benchmarks, while reducing the MAE by 22.4\% and 41.9\%, respectively.
This verifies the effectiveness of the DSU strategy to refine and update pseudo-labels.
Moreover, as we gradually exclude the consistency loss ${l}_{con}$ (row (e)) and HA operation (row (d)), degraded performances are observed on both datasets.
For an extreme case where we remove the DLU and only maintain CRF to refine pseudo-labels, it is observed that much worse performance is achieved.
These results consistently demonstrate that all components in the DSU strategy are beneficial for generating more accurate pseudo-labels.


\begin{wrapfigure}{r}{9.16cm}
  \centering
\vspace{-0.73cm}
  \tabcaption{\footnotesize{Ablation study of our deep unsupervised RGB-D SOD pipeline, using the F-measure and MAE metrics.}}
   \resizebox{!}{1.28 cm}{
    \begin{tabular}{c|cc|cc|cc}
    \bottomrule \hline
    \multirow{2}[2]{*}{{Index}} & \multicolumn{2}{c|}{\multirow{2}[2]{*}{{Model Setups}}} & \multicolumn{2}{c|}{\textbf{NJUD}} & \multicolumn{2}{c}{\textbf{NLPR}} \\
      \cline{4-5} \cline{6-7}
          & \multicolumn{2}{c|}{} & $F_{\beta}\uparrow$  & $\mathcal{M}\downarrow$     & $F_{\beta}\uparrow$  & $\mathcal{M}\downarrow$  \\
\hline
    (a)     & \multicolumn{2}{c|}{Backbone} & 0.627 & 0.186 & 0.570  & 0.126 \\ \hline
    (b)     & \multicolumn{2}{c|}{(a) + attentive training strategy} & 0.646      &  0.174     &   0.603    &0.112  \\ \hline
    (c)     & \multicolumn{1}{c|}{} & (b) + CRF  &  0.674     &  0.160     &   0.663    &0.093  \\
\cline{1-1}\cline{3-7}    	(d)     & \multicolumn{1}{c|}{\multirow{2}[1]{*}{DSU strategy}} & (b) + DSU (w/o ${l}_{con} \& HA$)  &  0.703     &  0.141     &   0.716    &0.074  \\
\cline{1-1}\cline{3-7}      	(e)    & \multicolumn{1}{c|}{} & (b) + DSU (w/o ${l}_{con}$) &   0.712    &  0.137     &   0.735    & 0.068  \\
\cline{1-1}\cline{3-7}    	(f)     & \multicolumn{1}{c|}{} & (b) + DSU (\textbf{Ours}) & 0.719 & 0.135 & 0.745 & 0.065  \\  
     \hline \toprule
    \end{tabular}%
  \label{tableabl}}%
  \vspace{-0.3cm}
\end{wrapfigure}
We also display the visual evidence of the updated pseudo-labels obtained from the DSU strategy in Fig.~\ref{UpdateLabel}. 
It is shown that the initial pseudo-labels unfortunately tend to miss important parts as well as fine-grained details. 
The application of CRF helps to filter away background noises, while salient parts could still be missing out.
By adopting our DSU and attentive training strategy, the missing parts could be retrieved in the updated pseudo-labels, with the object silhouette also being refined. 
These numerical and visual results consistently verify the effectiveness of our pipeline in deep unsupervised RGB-D saliency detection.

\begin{wrapfigure}{r}{9cm}
\vspace{-.76cm}
  \centering
  \tabcaption{\footnotesize{Analyzing attentive training strategy (ATS) with different settings. `Setting 1' is backbone + uniform weight + DSU, and `Setting 2' is backbone + attentive weight + DSU. The last four columns show backbone + ATS + DSU with different alternation interval $\tau$.
 
  }}
    \resizebox{!}{1.1 cm}{
    \begin{tabular}{c|c|c|c|c|c|c|c}
    \bottomrule \hline
    \multicolumn{2}{c|}{\multirow{2}[4]{*}{*}} & \multicolumn{2}{c|}{Analysis of the ATS} & \textbf{Ours} & \multicolumn{3}{c}{Analysis of the interval $\tau$} \\
\cline{3-4}\cline{6-8}    \multicolumn{2}{c|}{} & Setting 1 & Setting 2 & $\tau=3$  & $\tau=1$   & $\tau=5$   & $\tau=10$ \\   \hline
    \multirow{1}[4]{*}{\textbf{NJUD}} & $F_{\beta} \uparrow$     & 0.695 & 0.707 & \textbf{0.719} & 0.711 & 0.712 & 0.687 \\
\cline{2-8}          & $\mathcal{M} \downarrow$                                 & 0.148 & 0.142 & \textbf{0.135} & 0.143 & 0.138 & 0.147 \\  \hline
    \multirow{1}[4]{*}{\textbf{NLPR}} & $F_{\beta} \uparrow$      & 0.706 & 0.737 & \textbf{0.745} & 0.742 & 0.739 & 0.698 \\
\cline{2-8}          & $\mathcal{M} \downarrow$                                & 0.071 & 0.067 & \textbf{0.065} & 0.069 & 0.066 & 0.073 \\
  \hline  \toprule
    \end{tabular}}%
  \label{tabats}%
  \vspace{-.5cm}
\end{wrapfigure}
\textbf{Detailed analysis of the ATS.}
The influence of the attentive training strategy (ATS) is studied in detail here. The error reduction curves in Fig.~\ref{trainloss} have shown that the use of ATS could lead to greater test error reduction compared with the backbone that uses uniform weight or attentive weight for each instance. 
When combining the ATS with the DSU strategy as shown in Table~\ref{tabats}, the ATS also enables the network to learn more efficiently than using uniform weight or attentive weight alone, by comparing `Ours' with `Setting 1' and `Setting 2'.
In addition, we discuss the effect of different alternation intervals $\tau$ in ATS. As listed in Table~\ref{tabats}, the larger or smaller interval leads to inferior performance due to the insufficient or excessive learning of saliency models.






\iffalse
\begin{figure}[htbp]
\centering
\includegraphics[width=.47\textwidth]{graph/update} 
\caption{Visual examples of the intermediate pseudo-labels used in our approach. 
`Initial' is the initial pseudo-label generated by existing handcrafted method. 
`+CRF’ refers to the filtered pseudo-label output after applying fully-connected CRF. 
Update 1-2 represent updated pseudo-labels produced in our pipeline. `GT' is the ground truth, which is displayed here only for reference purpose.}
\label{UpdateLabel}
\end{figure}



\begin{table}[htbp]
  \centering
  \caption{Internal mean absolute errors, each is evaluated between current pseudo-labels and the corresponding true labels (only used for evaluation purpose) during the training process.}
   \resizebox{!}{0.45 cm}{    
\begin{tabular}{crcccc}
     \hline \toprule
    Training Label Update & \multicolumn{1}{c}{Initial} & Update 1     & Update 2     &  Update 3     & Update 4 \\  \midrule
    {Mean absolute error} & \multicolumn{1}{c}{0.162} & 0.124 & 0.117 & 0.116  & 0.116 \\
     \bottomrule \hline 
    \end{tabular}}%
    \vspace{1cm}
  \label{tabupdate}%
\end{table}%

\begin{table}[t]
  \centering
  \caption{Accuracy of refined pseudo-label under different settings. `CRF' refers to fully-connected CRF. `OTSU’ is the standard Otsu image thresholding method.}
   \resizebox{!}{1.3 cm}{
    \begin{tabular}{c|cccc}
   \hline \toprule
    \multicolumn{1}{c|}{Label Accuray} & $E_{\xi} \uparrow$  & $F_{\beta}^{w}\uparrow$  & $F_{\beta}\uparrow$  & $\mathcal{M}\downarrow$  \\ \midrule
    \multicolumn{1}{c|}{Initial pseudo-label} & 0.760 & 0.526 & 0.614 & 0.162 \\
    \multicolumn{1}{c|}{Initial pseudo-label + CRF} & 0.763 & 0.578 & 0.634 & 0.144 \\ 
    Our updated training label  & \textbf{0.792} & \textbf{0.635} & \textbf{0.708} & \textbf{0.116} \\ \midrule
    \multicolumn{1}{c|}{Depth map} & 0.419 & 0.284 & 0.164 & 0.414 \\
    \multicolumn{1}{c|}{Depth map + OTSU} & 0.465 & 0.398 & 0.429 & 0.332 \\ 
  \bottomrule \hline 
    \end{tabular}}%
  \label{training_Set_ablation}%
  \vspace{-0.5cm}
\end{table}%
\fi



\begin{figure}[h]
	\centering
	\vspace{-0.5cm}
	\begin{minipage}{0.53\textwidth}
\begin{minipage}{1\textwidth}
\centering
\tabcaption{\footnotesize{Internal mean absolute errors, each is evaluated between current pseudo-labels and the corresponding true labels (only used for evaluation purpose) during the training process.}}
\vspace{0.25cm}
   \resizebox{!}{0.44 cm}{    
\begin{tabular}{crcccc}
     \hline \toprule
    Pseudo-label Update & \multicolumn{1}{c}{Initial} & Update 1     & Update 2     &  Update 3     & Update 4 \\  \midrule
    {Mean absolute error} & \multicolumn{1}{c}{0.162} & 0.124 & 0.117 & 0.116  & 0.116 \\
     \bottomrule \hline 
    \end{tabular}}%
          \label{tabupdate}
		\end{minipage}
	\hspace{0.1in}
	\begin{minipage}{1\textwidth}
\centering
\tabcaption{\footnotesize{Comparison of different pseudo-label generation variants. `CRF' refers to fully-connected CRF. `OTSU’ represents the standard Otsu image thresholding method.}}
\vspace{0.15cm}
     \resizebox{!}{1.3 cm}{
    \begin{tabular}{c|m{0.7cm}<{\centering}m{0.7cm}<{\centering}m{0.7cm}<{\centering}m{0.7cm}<{\centering}}
   \hline \toprule
    \multicolumn{1}{c|}{Label Accuray} & $E_{\xi} \uparrow$  & $F_{\beta}^{w}\uparrow$  & $F_{\beta}\uparrow$  & $\mathcal{M}\downarrow$  \\ \midrule
    \multicolumn{1}{c|}{Initial pseudo-label} & 0.760 & 0.526 & 0.614 & 0.162 \\
    \multicolumn{1}{c|}{Initial pseudo-label + CRF} & 0.763 & 0.578 & 0.634 & 0.144 \\ 
    Our DSU  & \textbf{0.792} & \textbf{0.635} & \textbf{0.708} & \textbf{0.116} \\ \midrule
    \multicolumn{1}{c|}{Depth map} & 0.419 & 0.284 & 0.164 & 0.414 \\
    \multicolumn{1}{c|}{Depth map + OTSU} & 0.465 & 0.398 & 0.429 & 0.332 \\ 
  \bottomrule \hline 
    \end{tabular}
    \label{training_Set_ablation}
		\end{minipage}
		\end{minipage}
	\hspace{0.1in}
		\begin{minipage}[h]{0.44\linewidth}
			\centering
			\includegraphics[scale=0.32]{graph/update}
			\vspace{-0.25cm}
	        \figcaption{\footnotesize{Visual examples of the intermediate pseudo-labels used in our approach. 
`Initial' shows the initial pseudo-labels generated by traditional handcrafted method. 
`+CRF’ refers to the pseudo-labels after applying fully-connected CRF. 
Update 1\&2 represent the updated pseudo-labels produced in our pipeline over two training rounds. `GT' means the ground truth, used for reference purpose only.}}
\vspace{-0.45cm}
	        \label{UpdateLabel}
		\end{minipage}%
\vspace{-0.1cm}
\end{figure}




\textbf{Analysis of pseudo-labels.}
\label{ablDSU}
We analyze the quality of pseudo-labels over the training process in Table~\ref{tabupdate},
where the mean absolute error scores between the pseudo-labels at different update rounds and the ground-truth labels are reported.
It is observed that the quality of pseudo-labels is significantly improved during the first two rounds, which then remains stable in the consecutive rounds.
Fig.~\ref{UpdateLabel} also shows that the initial pseudo-label is effectively refined, where the updated pseudo-label is close to the true label. This provides more reliable guiding signals for training the saliency network.

\textbf{Comparison of different pseudo-label variants.}
In Table~\ref{training_Set_ablation}, we investigate other possible pseudo-label generation variants, including, initial pseudo-label with CRF refinement, raw depth map,  and raw depth map together with OTSU thresholding~\citep{otsu1979threshold}.
It is shown that, compared with the direct use of CRF, our proposed DSU is able to provide more reliable pseudo-labels, by disentangling depth to promote saliency.
It is worth noting that a direct application of the raw depth map or together with an OTSU adaptive thresholding of the depth map, may nevertheless lead to awful results.
We conjecture this is because of the large variations embedded in raw depth, and the fact that foreground objects may be affected by nearby background stuffs that are close in depth. 







\begin{wrapfigure}{r}{8.1cm}
  \centering
\vspace{-0.7cm}
  \tabcaption{\footnotesize{Discussion on different pseudo-label updating settings.}}
  \vspace{0.2cm}
   \resizebox{!}{0.95 cm}{
    \begin{tabular}{cc|cc|cc}
     \hline \toprule
    \multicolumn{2}{c|}{\multirow{2}[2]{*}{{Different Pseudo-label Updating Setups}}} & \multicolumn{2}{c|}{\textbf{NJUD}} & \multicolumn{2}{c}{\textbf{NLPR}} \\
  
  & & \multicolumn{1}{c}{$F_{\beta}\uparrow$}  & \multicolumn{1}{c|}{$\mathcal{M}\downarrow$ }    & \multicolumn{1}{c}{$F_{\beta}\uparrow$}  & \multicolumn{1}{c}{$\mathcal{M}\downarrow$ } \\ \midrule
    \multicolumn{2}{l|}{(a) DSU w/o $D_{NonSal}$ }  	                & 0.698	& 0.143 	& 0.715 	& 0.072   \\ 
     \multicolumn{2}{l|}{(b) DSU using $D^{Label}_{Sal}$, $D^{Label}_{NonSal}$}  	& 0.691	& 0.154 	& 0.707 	& 0.079   \\ 
    \multicolumn{2}{l|}{(c) DSU using $D_{Sal}$, $D_{NonSal}$ (\textbf{Ours})}  		& 0.719 	& 0.135 	& 0.745	& 0.065  \\
     \bottomrule \hline 
    \end{tabular}%
  \label{ABL}}%
  \vspace{-0.3cm}
\end{wrapfigure}
\textbf{Is depth-disentanglement necessary?}
In our DSU, we disentangle the raw depth map into valuable salient parts as well as redundant non-salient areas to simultaneously highlight the salient regions and suppress the non-salient negative responses. To verify the necessity of depth disentanglement, we remove the $D_{NonSal}$ branch in the depth-disentangled network and only use the $D_{Sal}$ to update pseudo-labels in DLU. Empirical results in Table~\ref{ABL} reveal that removing $D_{NonSal}$ (row (a)) leads to significant inferior performance when comparing with the original DSU (row (c)), which proves the superiority of our DSU design.


\textbf{How about using raw depth labels in DSU?}
We also consider to directly utilize the raw disentangled depth labels, \textit{i.e.,} $D^{Label}_{Sal}$ and $D^{Label}_{NonSal}$, to update pseudo-labels in DSU. 
However, a direct adoption of the raw depth labels does not lead to good performance, as empirically exhibited in Table~\ref{ABL} (b).
The DSU using raw labels performs worse compared to our original DSU design in row (c).
This is partly due to the large value variations and intrinsic label noises in raw depth map as discussed before.
On the other hand, benefiting from the capability of deep-learning-based networks to learn general representations from large-scale data collections, using $D_{Sal}$ and $D_{NonSal}$ in our DSU strategy is able to eliminate the potential biases in a single raw depth label.
Visual evidences as exhibited in the 4$^{th}$-7$^{th}$ columns of Fig.~\ref{disdepthvisual} also support our claim.






\vspace{-0.4cm}
\subsection{Application to Fully-supervised Setting}
\vspace{-0.3cm}
\label{ablfully}

\begin{wrapfigure}{r}{6.5cm}
\vspace{-.72cm}
  \centering
\tabcaption{\footnotesize{Applying our DSU to existing fully-supervised RGB-D SOD methods.}}
\vspace{0.1cm}
 \resizebox{!}{1.83 cm}{  
    \begin{tabular}{cr|cccc}
     \hline \toprule
    \multicolumn{2}{c}{\multirow{2}[1]{*}{*}} & \multicolumn{2}{c}{\textbf{NJUD}} & \multicolumn{2}{c}{\textbf{NLPR}} \\ \cmidrule(r){3-4} \cmidrule(r){5-6}
    \multicolumn{2}{r}{}  & $F_{\beta}\uparrow$  & $\mathcal{M}\downarrow$    & $F_{\beta}\uparrow$  & $\mathcal{M}\downarrow$ \\
    \midrule
    \multicolumn{2}{c}{DMRA~\citep{DMRA2019}}	& 0.872 & 0.051 & 0.855 & 0.031 \\
    \multicolumn{2}{c}{\textbf{+ Our DSU}}  	& 0.893 & 0.044 & 0.879 & 0.026 \\ \midrule
    \multicolumn{2}{c}{CMWN~\citep{CMWN2020}}	& 0.878 & 0.047 & 0.859 & 0.029 \\
    \multicolumn{2}{c}{\textbf{+ Our DSU}} 		& 0.901 & 0.041 & 0.882 & 0.025 \\ \midrule
    \multicolumn{2}{c}{FRDT~\citep{FRDT2020}}	& 0.879 & 0.048 & 0.868 & 0.029 \\
    \multicolumn{2}{c}{\textbf{+ Our DSU}} 		& 0.903 & 0.038 & 0.901 & 0.023 \\ \midrule
    \multicolumn{2}{c}{CPD~\citep{CPD2019}}		& 0.873 & 0.045 & 0.866 & 0.028 \\
    \multicolumn{2}{c}{\textbf{+ Our DSU}} 		& 0.909 & 0.036 & 0.907 & 0.022 \\
     \bottomrule \hline 
    \end{tabular}}%
          \label{tabfullyabl}
  \vspace{-.5cm}
\end{wrapfigure}
To show the generic applicability of our approach, a variant of our DSU is applied to several cutting-edge fully-supervised SOD models to improve their performance.
This is made possible by redefining the quantities $D^{Label}_{Sal} = \mathcal{S}_{GT} \otimes D_{map}$ and $D^{Label}_{NonSal} = (1 - \mathcal{S}_{GT}) \otimes D_{map}$, with $\mathcal{S}_{GT}$ being the ground-truth saliency. 
Then the saliency network (\textit{i.e.,} existing SOD models) and the depth-disentangled network are retrained by $\mathcal{S}_{GT}$ and the new $D^{Label}_{Sal}$ and $D^{Label}_{NonSal}$, respectively. After training, the proposed DLU is engaged to obtain the final improved saliency.
In Table~\ref{tabfullyabl}, we report the original results of four SOD methods and the new results of incorporating our DSU strategy on two popular benchmarks.
It is observed that our supervised variants have consistent performance improvement comparing to each of existing models.
For example, the average MAE score of four SOD methods on NJUD benchmark is reduced by 18.0\%.
We attribute the performance improvement to our DSU strategy that can exploit the learned $D_{Sal}$ to facilitate the localization of salient object regions in a scene, as well as suppress the redundant background noises by subtracting $D_{NonSal}$. More experiments are presented in the Appx~\ref{MA-abl}.








\vspace{-.5cm}
\section{Conclusion and Outlook}
\vspace{-.4cm}
This paper tackles the new task of deep unsupervised RGB-D saliency detection.
Our key insight is to internally engage and refine the pseudo-labels. This is realized by two key modules, the depth-disentangled saliency update in iteratively fine-tuning the pseudo-labels, and the attentive training strategy in addressing the issue of noisy pseudo-labels. 
Extensive empirical experiments demonstrate the superior performance and realtime efficiency of our approach. 
For future work, we plan to extend our approach to scenarios involving partial labels.
\vspace{.1cm}
\\
\small{
\textbf{Acknowledgement.} This research was partly supported by the University of Alberta Start-up Grant, UAHJIC Grants, and NSERC Discovery Grants (No. RGPIN-2019-04575).
}

\iffalse
\textbf{\textit{Ethics Statement.}} 
This work does not involve any negative societal and ethical impacts, and we believe the most proximate impacts of this work will be positive. In particular, class-agnostic unsupervised segmentation problems like RGB-D salient object detection problem present a major obstacle to the requirement of a significant amount of laborious pixel-level human annotations. The proposed Depth-disentangled Saliency Update framework can produce more reliable supervision signal for training saliency network, which exhibits an appealing prospect and tradeoff between model accuracy and human consumption. Furthermore, our method achieves fast inference speed and good generic applicability, which is useful and appealing for light-weight deployment in practice.

\textbf{\textit{Reproducibility Statement.}} 
As for social responsibility, the project team member will responsible for the quality of the project, like no cheating, being honest and academic integrity. We also promise that our source code with easy-to-use instruction will be publicly available upon paper acceptance.
\fi



\small{

