\section{Introduction}


\IEEEPARstart {A}{s} an economical and convenient alternative to depth sensors, monocular depth estimation
 has wide applications in obstacle avoidance \cite{Mancini2016FastRM}, simultaneous localization and mapping (SLAM) \cite{Tateno2017CNNSLAMRD,Guo2020SemanticHB}, robot navigation \cite{mendes2020deep}, etc., and thus has captured much attention for a long time \cite{Hu2019VisualizationOC,hu2019analysis,laina2016deeper,Eigen2014depth}. 
In the past few years, it has further gained significant progress due to the fast development of deep learning. 


\begin{figure}[t]
\vspace{-5mm}
\centering
\includegraphics[width=0.53\textwidth]{figs/para-accu.pdf}
\caption{The total parameters and accuracy of different methods for depth estimation. As seen that, there is a trade-off between accuracy and model's capacity. Our method achieves competitive performance compared with state-of-the-art methods with only very a small number of parameters.}
\label{fig_config}
\end{figure}


Most of the previous works focused solely on the improvement of estimation accuracy \cite{fu2018deep,laina2016deeper,hu2019revisiting}.
However, the depth estimation has to be both computationally efficient and accurate.
It is essential for robotic systems with limited computation resources.
Although several prior works have attempted to improve the computational efficiency with light-weight networks \cite{Wofk2019FastDepthFM,Nekrasov2019RealTimeJS}, the cost is a significant degradation of inference accuracy.  
To the best of our knowledge, there are no prior works of light-weight depth estimation that can achieve the same level or at least comparable performance as state-of-the-art methods.

\begin{figure*}[t]
\centering
\subfigure {\includegraphics[width=\linewidth]{figs/framework2.pdf}}
\caption{The left side shows the depth histogram of NYU-v2 training set and ScanNet validation set, respectively. Both of the two histograms exhibit a long-tailed distribution and they are highly similar. The right side shows the configuration of knowledge distillation (KD) considered in this paper. 
(a) is the standard method that applies KD with the original labeled set. (b) applies KD with only auxiliary unlabeled set. (c) applies KD with the original labeled set and the auxiliary unlabeled set. (d) applies KD with both the original and auxiliary labeled set.
}
\label{fig_frame}
\end{figure*}

In this paper, we delve into monocular depth estimation from the aspects of inference accuracy and computational efficiency. We aim to achieve a maximum improvement of the accuracy with the minimum cost of computation time.
Towards this end, we first introduce a light-weight network with a particular target on model compression.
In contrast to traditional encoder-decoder architecture that gradually up-samples the intermediate features,
we first compress feature maps extracted by multi-layers of the encoder to a fixed small number of channel at each scale, and then upsample them to the same resolution. These feature maps are then concatenated and fed to two convolutional layers to yield a final depth map. Thanks to the above design, the network built on MobileNet-v2 only has 1.7 M parameters, and thus yields the minimum model complexity than previous methods.

We then consider how to tackle the low inference accuracy when using a light-weight network. 
A common solution in image classification is the use of knowledge distillation (KD) to boost a small network, i.e. student, with the guidance from a large network, i.e. teacher. 
Previous studies tend to introduce complex strategies to improve KD, such as distilling intermediate features \cite{Romero2015FitNetsHF}, multi-teacher distillation \cite{Tarvainen2017MeanTA,Liu2019KnowledgeFI}, adversarial distillation \cite{Wang2020DeceivingIT}.



We consider a more intuitive way to promote KD with auxiliary data.  Our motivations are two-fold: first, as the depth estimation is a non-linear mapping from RGB space to depth space, KD approximates this function in a data driven approach. Therefore, the more valuable data we have, the more accurate we can approximate the mapping. Second, unlike image recognition, data without same categories is considered as out-of-distribution data which is usually harmful to KD. Since depth estimation is a regression task, auxiliary data is easy to collect. For instance, indoor scenarios have similar structures and scene scales, as seen in Fig.~\ref{fig_frame}.

 We particularly study two cases, i.e. labeled and unlabeled data.
In labeled case, ground truths can be obtained by depth sensors, e.g. Kinect. In unlabeled case, auxiliary data can be simply collected by visual camera in scenarios with similar scene scales.
Therefore, it is more practical for depth estimation to boost the performance with auxiliary data in real-world applications.
In this paper, we propose to take the following learning strategies for these two specific cases. 
\begin{itemize}
    \item When auxiliary data is unlabeled, we first train the teacher with the original labeled set, and then
    apply KD with the original labeled set and the auxiliary unlabeled set. \vspace{2mm}
    \item When auxiliary data is labeled, we first train the teacher on the combination of the original set and auxiliary set. It will provide us a more discriminative teacher. KD is then applied to further boost the student with the mixed dataset.
\end{itemize}

The use of auxiliary data can actually improve light-weight depth estimation.
As shown in Fig.~\ref{fig_config}, our method can achieve comparable performance against state-of-the-art methods while only using $1\%$ parameters and outperforms other light-weight approaches by a large margin.
We conduct a number of experiments to evaluate the effectiveness of our approach, and confirm through the experiments that:
\begin{itemize}
    \item  Even without access to the original training set, we can still successfully apply KD with enough auxiliary unlabeled samples as long as they have similar scene scales to the original training samples. \vspace{2mm}
    \item  When applying KD with the original trained set and auxiliary unlabeled set together, the performance improvement is significant as the gap between the teacher and student is better filled.
    \vspace{2mm}
    \item  The improvement of the light-weight network is limited if we directly train it with the mixed dataset of both original and auxiliary labeled data due to its low capacity. On the other hand, the two-stage learning strategy, i.e. training a larger teacher and then apply KD is more effective.
    
   
   

    
   
   
   
\end{itemize}














\section{Related Work}
 
 \begin{figure*}[t]
\centering
\subfigure {\includegraphics[width=\linewidth]{figs/network.png}}
\caption{Diagram of the proposed network. The core components are the four feature fusion and compression layers (FFC). The FFC layer consists of a channel-wise attention layer and a convolutional layer, such that it can automatically attribute more weights to more important features and then fuse them to fewer channels. 
}
\label{fig_arch}
\vspace{-5mm}
\end{figure*}

\subsection{Monocular Depth Estimation}
In previous studies, monocular depth estimation was solved in a supervised learning fashion by penalizing pixel-wise loss between prediction and ground truth depth \cite{Eigen2014depth,hu2019revisiting}. Similar to other vision tasks, e.g. semantic segmentation, different networks have been continuously proposed. Starting from the basic encoder-decoder network \cite{laina2016deeper}, networks assembled with skip connections \cite{hu2019revisiting}, dilated convolution \cite{fu2018deep}, pyramid pooling \cite{Mendes2021OnDL}, etc. have shown advanced performance. 
It can be also formulated as an unsupervised learning problem complying with the geometry consistency of multi-view images \cite{Zhou2017UnsupervisedLO,Zhan2018UnsupervisedLO}. 
Currently, the performance of unsupervised learning approaches is still inferior to supervised learning methods. 

Several approaches have attempted to estimate depth in real time.  In \cite{Wofk2019FastDepthFM,Nekrasov2019RealTimeJS}, light-weight networks built on MobileNet and MobileNet-v2 were introduced for fast depth estimation. The networks are trained in a traditional supervised fashion. Also, a light-weight network with  recurrent modules was proposed to solve the unsupervised depth estimation \cite{Liu2020MiniNetAE}. The methods using small networks demonstrate superior performance in terms of computation speed, however, the accuracy tends to drop significantly compared to those built upon large networks.




\subsection{Learning with Auxiliary Data}
To boost the performance of learning based methods, a recent tendency is to leverage extra labeled training datasets. As demonstrated by image recognition on ImageNet \cite{deng2009imagenet}, this strategy is indeed effective. The top1 accuracy demonstrated by those approaches \cite{pham2020meta,foret2020sharpness} is greater than $5\%$ compared with methods without utilizing extra data.

This strategy is also employed for depth estimation. In \cite{Chen2020ImprovingMD}, six auxiliary datasets are employed to handle hard cases including low light, reflecting surface, spurious edges, etc., which demonstrates a superior performance for indoor depth estimation.
Also, several methods employ multi-domain datasets, e.g. indoor and outdoor scenes, synthesized and real-world images to learn a universal network \cite{romanov2020towards,Lasinger2020TowardsRM,yin2020diversedepth,Aleotti2021RealTimeSI}. However, these methods omit the scale difference across datasets and can only estimate a normalized depth map. In this paper, we focus on indoor depth estimation and aim to reconstruct the real scale of a scene.

 Note that auxiliary unlabeled data is also commonly leveraged in image recognition under the setting of semi-supervised learning \cite{Wang2021KnowledgeDA,Xie2020SelfTrainingWN}.
In image recognition, auxiliary data has to be carefully collected such that its semantic attribute falls into categories of model's prediction.
As depth estimation is formulated as a regression task, the auxiliary data is easier to be collected in practical scenarios.


\subsection{Knowledge Distillation}
Knowledge distillation has been extensively studied in recent years. it is originally proposed to transfer the knowledge from a teacher to a student on image recognition \cite{Hinton2015DistillingTK}. Usually, the teacher is large and the student is relatively small.
Recent studies have attempted to improve KD more effectively.
Mirzadeh et al. proposed to use an assistant network between the teacher and the student  \cite{Mirzadeh2020ImprovedKD}. Some works have employed intermediate features to guide the learning of the student \cite{Huang2017LikeWY,Liu2020StructuredKD}. In \cite{Tarvainen2017MeanTA,Liu2019KnowledgeFI}, the strategy of distillation from multiple teachers has been proposed. 
Another type of method to improve KD is to augment training set, such as applying GAN to generate more data \cite{Snelson2005SparseGP}, or employing a large amount of extra data on the cloud \cite{Xu2019PositiveUnlabeledCO}.

A few works also apply KD to boost depth estimation \cite{Pilzer2019RefineAD,Aleotti2021RealTimeSI}. 
Unlike these methods that simply apply existing methods of KD developed for image recognition, we propose to utilize auxiliary unlabeled/labeled data to boost KD based on an observation that many scenarios have similar scene scales in real-world.




\section{Methodology}
 In this section, we first give the details of the light-weight network. Then, we present the learning strategy of applying KD with auxiliary data.




\subsection{Light-weight Network}

Most of the previous works employ a symmetric encoder-decoder to deal with pixel to pixel regression tasks \cite{laina2016deeper,ma2017sparse,Hu2020ATU}. Such network yields low inference efficiency as it requires lots of GPU memories during the computation procedure. 
As shown in \cite{netdissect2017}, there is a high redundancy inside CNNs, multiple filters tend to capture same or similar feature representations. 
To push the envelope of light-weight depth estimation, in this paper, we introduce an extremely compact design of network architecture.

The proposed network is shown in Fig.~\ref{fig_arch}. The core components are
four feature fusion and compression layers (FFC). The FFC layer is employed to efficiently fuse feature maps. It consists of a channel-wise attention mechanism and a convolutional layer.
 To be specific,
given a set of feature maps
extracted by a convolutional block, e.g. residual block,  we first apply channel-wise attention \cite{hu2018senet} to automatically attribute weights to them. Then, they are fused by a $5 \times 5$ convolutional layer. To reduce the model's complexity, these feature maps are compressed to a fixed number of channels (16 channels). 
For features extracted with an encoder at multi-scales, we apply FFC at each scale. 
Then, the outputted feature maps are upsampled to have the same size by $\times2,4,8$ and 16, respectively. Finally, they are concatenated and fed into two  $5 \times 5$ convolutional layers (denoted as \textbf{convs} in Fig.~\ref{fig_arch}) to yield a final depth map. 

We use ResNet-34 and MobileNet-v2 as backbone networks for the teacher and student, respectively. As a result, the teacher and student have 21.9 M and 1.7 M parameters, respectively. The detailed information of model parameters is given in Table.~\ref{net_paras}.

\begin{table}[!t]
\begin{center} 
\caption{The details of model parameters for the teacher and student.}
\label{net_paras}
\begin{tabular}{|l|c|c|c|c|}
\hline
Network  & Backbone & Encoder (M) & Decoder (M) & Total (M)\\
\hline
Teacher & ResNet-34 & 21.3 &0.6 &21.9 \\
Student & MobileNet-v2 & 1.4 &0.3 &1.7 \\
\hline
\end{tabular}
\end{center}
\end{table}




\subsection{Applying KD with Auxiliary Data}

\subsubsection{Standard KD}

We consider a classical knowledge distillation with a well pre-trained teacher on a labeled set  $\mathcal{X}$. 
The supervision for training the student comes from the loss against ground truth depths and estimation from the teacher net.
Here, we use $N_t$ and $N_s$ to represent the teacher and student net, respectively.
Then, the loss for training the student is:
\begin{equation}
    \mathcal{L} = \frac{1}{\mathcal{X}} \sum_{x_i, g_i \in \mathcal{X}} (\lambda  L (N_s(x_i), N_t(x_i))  + (1- \lambda) L (N_s(x_i), g_i) )
\end{equation}
where $g_i$ denotes the ground truth for $x_i$, $L$ is an error measure between two depth maps.

We employ the error measure proposed in \cite{hu2019revisiting}, where the sum of three losses are used:
\begin{equation}
 L  = l_{\rm depth} +  l_{\rm grad} + l_{\rm normal} 
\end{equation}
where $l_{\rm depth}=\frac{1}{n}\sum_{i=1}^n F(e_i)$, 
$l_{\rm grad}=\frac{1}{n}\sum_{i=1}^n (F(\nabla_{x}(e_i))+F(\nabla_{y}(e_i)))$, and $ l_{\rm normal} = \frac{1}{n}\sum_{i=1}^n
    \left(1-\cos\theta_i\right),$ 
where $F(e_i) = \ln(e_i+0.5)$; $e_i = \|y_i - \hat{y_i}\|_1$; $y_i$ and $\hat{y_i}$ are two depth maps; and $\theta_i$ is the angle between the surface normals computed from them.


\subsubsection{Learning with Auxiliary Data}

\begin{figure}[t]
\centering
\subfigure {\includegraphics[width=0.98\linewidth]{figs/ax_res2.pdf}}
\caption{Results of the student network on NYU-v2 test set. From left to right: result of the student that learned via KD with the training set of ImageNet, result with supervised learning on NYU-v2 training set, result of the student that learned via KD with training set of ScanNet.}
\label{fig_ax_res}
\vspace{-5mm}
\end{figure}


 We assume auxiliary data is effective to KD for depth estimation as long as it has similar scene scales.
 We conduct a preliminary experiment to verify our assumption. Specifically,
 given a teacher network trained on the NYU-v2 dataset, we perform KD with cross-domain datasets.
 \begin{equation}
    \mathcal{L} =  \frac{1}{\mathcal{U}} \sum_{u_j \in \mathcal{U}} (\lambda  L (N_s(u_j), N_t(u_j))  
\end{equation}
We choose ImageNet and ScanNet for the purpose, the former is an out-of-distribution dataset and the latter is also an indoor dataset.
As seen in Fig.~\ref{fig_ax_res}, we can see that the results of KD with ScanNet is slightly better than the original result and there is $15.5\%$ accuracy drop if we perform KD with ImageNet. The experimental details are given in Sec.~\ref{experimental_details}.
 
 The result of this preliminary experiment supports our assumption very well that even without access to the original training data, using unlabeled data with similar scene scales can still successfully perform KD for depth estimation.
It also reveals that applying KD with original set as well as auxiliary data together, the gap between the teacher and student can be better filled. Thereby, the performance of the light-weight network can be further improved.
Specifically, we consider two situations of using auxiliary data. 
\paragraph{The use of auxiliary unlabeled data}
The teacher is trained with the original labeled set $\mathcal{X}$, which is denoted as $N_t$. We can access $\mathcal{X}$ as well as an auxiliary unlabeled set $\mathcal{U}$ during the procedure of KD. Then the loss for the student is formulated as:
\begin{equation}
\begin{split}
    \mathcal{L} = & 
    \frac{1}{\mathcal{X}} \sum_{x_i, g_i \in \mathcal{X}} (\lambda  L (N_s(x_i), N_t(x_i))  + (1- \lambda) L (N_s(x_i),g_i) ) + \\ 
    &  \frac{1}{\mathcal{U}} \sum_{u_j \in \mathcal{U}}  (L (N_s(u_j), N_t(u_j))
\end{split}    
\end{equation}

\paragraph{The use of auxiliary labeled data}
In this case, the auxiliary data $\mathcal{U}^{'}$ is fully labeled.
The teacher is trained on a mixed dataset, i.e. $\mathcal{X} \cup \mathcal{U}^{'}$, and it will be highly discriminative than the one trained on $\mathcal{X}$ only. 
The student is then learned via KD  with mixed data: 
\begin{equation}
\begin{split}
    \mathcal{L} = & 
    \frac{1}{\mathcal{X}} \sum_{x_i, g_i \in \mathcal{X}} (\lambda  L (N_s(x_i), N_t(x_i))  + (1- \lambda) L (N_s(x_i),g_i) ) + \\ 
    &  \frac{1}{\mathcal{U}^{'}} \sum_{u_j, g_j \in \mathcal{U}^{'}}  (\lambda L (N_s(u_j), N_t(u_j))  + (1- \lambda) L (N_s(u_j),g_j) )
\end{split}    
\end{equation}






\section{Experiments}




\subsection{Experimental Setting}



We use the NYU-v2 dataset \cite{Silberman2012IndoorSA} for all the experiments. The dataset consists of a variety of indoor scenes and is used in most of the previous studies. We use the standard  procedure  for preprocessing \cite{Eigen2014depth,laina2016deeper,ma2017sparse}. To be specific, the official splits of 464 scenes are used, i.e., 249 scenes for training and 215 scenes for testing. 
This results in approximately 50K unique pairs of an image and a depth map of 640$\times$480 pixel size. 
The images are then resized down to 320$\times$240 pixels using bilinear interpolation, and then crop their central parts of 304$\times$228 pixels, which are used as inputs to networks. The depth maps are resized to 
$152\times 114$ pixels. 
For testing, following the previous studies, we use the same small subset of 654 samples.
The auxiliary data is taken from the ScanNet dataset \cite{dai2017scannet}. We randomly select 204K images from 1513 scenarios. 

\subsubsection{Implementation Details}
\label{experimental_details}
For the teacher net, we use the network built on ResNet-34. 
For the student net, we use MobileNet-v2 as the encoder. 
Both the teacher and student are trained for 20 epochs. The parameter $\lambda$ for calculating $\mathcal{L}$ is set to 0.1.
The encoder module in the network is initialized by a model pretrained with the ImageNet dataset \cite{deng2009imagenet}. The other layers in the network are randomly initialized. 
We use Adam optimizer with an initial learning rate of 0.0001, and reduce it to 10$\%$ for every 5 epochs. We set $\beta_{1}=0.9$, $\beta_{2}=0.999$, and use weight decay of 0.0001.




\begin{figure*}[!t]
\centering
\subfigure []{\includegraphics[width=0.45\linewidth]{figs/res_wo_KD.pdf}}\quad
\subfigure []{ \includegraphics[width=0.45\linewidth]{figs/res_w_kd.pdf}}
\caption{(a) Results of the teacher and student network trained with supervised learning. The blue color denotes results trained on $\mathcal{X}$ and the green color denotes results with $\mathcal{X}\cup \mathcal{U'}$.  (b) Results of the student network learned with KD. The blue color denotes results of  $N_s(\mathcal{X})$, $N_s(\mathcal{U})$ and $N_s(\mathcal{X \cup U})$, respectively, and the green color denotes results of $N_s(\mathcal{X \cup U'})$.}
\label{w_wo_kd}
\end{figure*}


\begin{table*}[!t]
\begin{center} 
\caption{Quantitative comparisons between our method and other approaches built on large networks on the NYU-v2 dataset.}
\label{nyu_depth_acc}
\begin{tabular}{|l|l|c|c|c|c|c|c|c|}
\hline
Method & Backbone & Original set & Auxiliary set & Params (M) $\downarrow$  & RMSE $\downarrow$ & REL $\downarrow$ &$\log10$ $\downarrow$ & $\delta_1$ $\uparrow$ \\
\hline
Laina et al. \cite{laina2016deeper}&ResNet-50 &NYU-v2 &-- &60.6 &0.573  &0.127 &0.055 &0.811 \\ 
Hu et al. \cite{hu2019revisiting}&ResNet-50 &NYU-v2 &-- &63.6 &0.555 &0.126 &0.054 &0.843 \\ 
Zhang et al. \cite{Zhang2019PatternAffinitivePA} &ResNet-50 &NYU-v2 &-- &95.4 &0.497 &0.121 &- &0.846  \\
Fu et al. \cite{fu2018deep}&ResNet-101 &NYU-v2 &-- &110.0 &0.509 &0.115 &0.051 &0.828 	 \\    

Hu et al.  \cite{hu2019revisiting}&SeNet-154 &NYU-v2 &-- &149.8	&0.530 &0.115 &0.051 &0.866 	 \\
Chen et al. \cite{Chen2019StructureAwareRP}&SeNet-154 &NYU-v2 &-- &210.3 &0.514 & 0.111 &0.048 & 0.878  \\ 
Chen et al.\cite{Chen2020ImprovingMD}&ResNet-101  &NYU-v2 & HC  labeled data &163.4 &\textbf{0.376} &\textbf{0.098} &\textbf{0.042} & \textbf{0.899} \\ \hline
Ours $N_s(\mathcal{X} \cup \mathcal{U})$ &MobileNet-V2 &NYU-v2 & Unlabeled ScanNet &1.7 &0.482 &0.131 &0.056  &0.837\\ 
Ours $N_s(\mathcal{X} \cup \mathcal{U}^{'})$  &MobileNet-V2 &NYU-v2 & Labeled ScanNet &1.7 &\textbf{0.461} &\textbf{0.121} &\textbf{0.052}  &\textbf{0.855} \\ \hline
\end{tabular}
\end{center}
\end{table*}

\begin{figure*}[t]
\centering  
\begin{tabular}
{p{0.15\textwidth}<{\centering}p{0.15\textwidth}<{\centering}p{0.15\textwidth}<{\centering}p{0.15\textwidth}<{\centering}p{0.15\textwidth}<{\centering}} 
\IncG[ width=1.1in]{./figs/imgs/20.png}
&\IncG[ width=1.1in]{./figs/gt/gt20.png}
&\IncG[ width=1.1in]{./figs/ma/20.png}
&\IncG[ width=1.1in]{./figs/netra/20.png}
&\IncG[ width=1.1in]{./figs/ours/out20.png}
\\
\IncG[ width=1.1in]{./figs/imgs/29.png}
&\IncG[ width=1.1in]{./figs/gt/gt29.png}
&\IncG[ width=1.1in]{./figs/ma/29.png}
&\IncG[ width=1.1in]{./figs/netra/29.png}
&\IncG[ width=1.1in]{./figs/ours/out29.png}
\\
\IncG[ width=1.1in]{./figs/imgs/36.png}
&\IncG[ width=1.1in]{./figs/gt/gt36.png}
&\IncG[ width=1.1in]{./figs/ma/36.png}
&\IncG[ width=1.1in]{./figs/netra/36.png}
&\IncG[ width=1.1in]{./figs/ours/out36.png}
\\
\IncG[ width=1.1in]{./figs/imgs/40.png}
&\IncG[ width=1.1in]{./figs/gt/gt40.png}
&\IncG[ width=1.1in]{./figs/ma/40.png}
&\IncG[ width=1.1in]{./figs/netra/40.png}
&\IncG[ width=1.1in]{./figs/ours/out40.png}
\\
\IncG[ width=1.1in]{./figs/imgs/54.png}
&\IncG[ width=1.1in]{./figs/gt/gt54.png}
&\IncG[ width=1.1in]{./figs/ma/54.png}
&\IncG[ width=1.1in]{./figs/netra/54.png}
&\IncG[ width=1.1in]{./figs/ours/out54.png}
\\
\IncG[ width=1.1in]{./figs/imgs/67.png}
&\IncG[ width=1.1in]{./figs/gt/gt67.png}
&\IncG[ width=1.1in]{./figs/ma/67.png}
&\IncG[ width=1.1in]{./figs/netra/67.png}
&\IncG[ width=1.1in]{./figs/ours/out67.png}
\\
\footnotesize{(a) Input images} & \footnotesize{(b) Ground truth} & \footnotesize{(c) Fast-depth \cite{Wofk2019FastDepthFM}}& \footnotesize{(d) Joint-depth \cite{Nekrasov2019RealTimeJS}} &(e) \footnotesize{Ours} \\
\end{tabular}
\caption{Qualitative comparison of different methods for light-weight depth estimation. }
\label{fig1}
\end{figure*}



\subsection{Quantitative Evaluation}
For simplicity, we use $\mathcal{X}$, $\mathcal{U}$ and $\mathcal{U'}$ to denote the NYU-v2, unlabeled ScanNet, and labeled ScanNet, respectively.
The teacher trained on $\mathcal{X}$ and $\mathcal{X} \cup \mathcal{U}$ are denoted as $N_t(\mathcal{X})$ and $N_s(\mathcal{X} \cup \mathcal{U})$, respectively. Similarly, the student trained on  $\mathcal{X}$, $\mathcal{X} \cup \mathcal{U}$ and $\mathcal{X} \cup \mathcal{U}^{'}$ are denoted as 
$N_s(\mathcal{X}), N_s(\mathcal{X} \cup \mathcal{U})$, $N_s(\mathcal{X} \cup \mathcal{U}^{'})$, respectively, throughout the paper. 


\subsubsection{Performance without KD}
We first quantify the teacher and student network with pure supervised learning. We perform experiments on  $\mathcal{X}$ and the mixed dataset $\mathcal{X}\cup\mathcal{U'}$, respectively. The results are shown in Fig.~\ref{w_wo_kd} (a).
It is clear that the performance can be improved by training with more labeled samples, as the teacher is improved from o.845 to 0.874 and the student is improved from 0.802 to 0.825, respectively.
On the other hand, the performance gap between the teacher and student is significant, e.g. 0.845 vs 0.802 and 0.874 vs 0.825.


\subsubsection{Performance with KD}




We conduct several experiments to validate the proposed method. We first train the teacher net $N_t(\mathcal{X})$ and $N_t(\mathcal{X} \cup \mathcal{U}^{'})$ on the $\mathcal{X}$ and $\mathcal{X} \cup \mathcal{U}^{'}$, respectively. Then, we train the student in four different settings as follows:
\begin{enumerate}
    \item Given a trained teacher on the original set, we apply KD with the original training set, i.e $N_t(\mathcal{X}) \rightarrow N_s(\mathcal{X})$.
    \item Given a trained teacher on the original set, we apply KD with the auxiliary unlabeled set, i.e $N_t(\mathcal{X}) \rightarrow N_s(\mathcal{U})$.
    \item Given a trained teacher on the original set, we apply KD with the original training set and auxiliary unlabeled set, i.e $N_t(\mathcal{X}) \rightarrow N_s(\mathcal{X}\cup \mathcal{U})$.
    \item  Given a trained teacher on the original training set and auxiliary labeled set, we apply KD with the mixed labeled set, i.e $N_t(\mathcal{X}\cup \mathcal{U'}) \rightarrow N_s(\mathcal{X}\cup \mathcal{U'})$.
\end{enumerate}

The results are shown in Fig.~\ref{w_wo_kd} (b), we can see a large performance gap between the teacher and the student when applying standard KD as the setting of 1), the performance dropped from 0.845 to 0.808. An intriguing
observation is that the performance is even better when using only auxiliary unlabeled data, i.e. setting 2), compared to standard KD. As expected, the combination of them, i.e. setting 3), brings a significant performance boost. 

As seen in Fig.~\ref{w_wo_kd} (a) $N_s(\mathcal{X}\cup\mathcal{U'})$,  when the auxiliary data is labeled, the light-weight network can be improved by supervised learning though, the improvement is limited due to low capacity of the small network. Its performance is inferior to the one trained with KD with auxiliary unlabeled data, i.e. result of $N_s(\mathcal{X}\cup\mathcal{U})$ in Fig.~\ref{w_wo_kd} (b).
On the other hand, we can learn a more accurate teacher, thus the light-weight network can also be further improved by applying KD, as seen in $N_s(\mathcal{X}\cup\mathcal{U'})$ of Fig.~\ref{w_wo_kd} (b).





\subsection{Comparison with Previous Methods.}
\paragraph{Comparison with Large Networks }
The comparison against methods built on large networks is shown in Table~\ref{nyu_depth_acc}.  
A clear tendency of accuracy improvement is demonstrated by previous methods built on different backbone networks, i.e. from ResNet-50 to SeNet-154. 
As seen that when using auxiliary unlabeled data,
our method is comparable to \cite{Zhang2019PatternAffinitivePA, hu2019revisiting} and even better than \cite{fu2018deep,laina2016deeper} with only 1.7 M parameters.


When comparing methods using extra labeled data, the best performance shown in Table~\ref{nyu_depth_acc} is \cite{Chen2020ImprovingMD} where the method is trained with six auxiliary datasets. The number of extra training data is 120K in total. Note that the auxiliary data used in \cite{Chen2020ImprovingMD} is carefully collected to handle the hard cases for depth estimation, such as spurious edges and reflecting surface. The extra data of our method is randomly selected from the ScanNet dataset \footnote{The auxiliary data used in \cite{Chen2020ImprovingMD} is not publicly available. We believe the performance of our method can be further improved when utilizing similar data for handling hard cases.}.


\paragraph{Comparison with Light-weight Networks }

\setlength{\tabcolsep}{1.0pt}
\begin{table}[!t]
\begin{center}
\caption{Quantitative comparison of light-weight approaches on the NYU-v2 dataset. The best and the second best results are highlighted in red and blue, respectively.}
\label{nyu_depth_acc_light}
\begin{tabular}{|l|l|c|c|c|c|c|c|c|c|}
\hline
Method & Backbone & Params (M) & GPU [ms]  & REL & $\delta_1$  \\
\hline
Fast-depth \cite{Wofk2019FastDepthFM} & MobileNet &3.9 &7 &- &0.775\\
Joint-depth \cite{Nekrasov2019RealTimeJS} &MobileNet-V2 &3.1 &21 &0.149 &0.790\\ 
Ours $N_s(\mathcal{X} \cup \mathcal{U})$     &MobileNet-V2 &1.7 &11  &{\color{blue}\textbf{0.131}} &{\color{blue}\textbf{0.837}}\\ 
Ours $N_s(\mathcal{X} \cup \mathcal{U}^{'})$   &MobileNet-V2 &1.7 &11  &{\color{red}\textbf{0.121}}  &{\color{red}\textbf{0.855}}\\ 
\hline
\end{tabular}
\end{center}
\end{table}


\setlength{\tabcolsep}{2.5pt}
\begin{figure*}[t]
\centering  
\begin{tabular}
{p{0.05\textwidth}<{\centering}p{0.16\textwidth}<{\centering}p{0.16\textwidth}<{\centering}p{0.16\textwidth}<{\centering}p{0.16\textwidth}<{\centering}p{0.16\textwidth}<{\centering}} 
\footnotesize{RGB}
&\IncG[ width=1.03in]{./figs/tum/rgb/1305031809.649447.png}
&\IncG[ width=1.03in]{./figs/tum/rgb/1305031455.491637.png}
&\IncG[ width=1.03in]{./figs/tum/rgb/1305031533.275405.png}
&\IncG[ width=1.03in]{./figs/tum/rgb/1305031245.461734.png}
&\IncG[ width=1.03in]{./figs/tum/rgb/1305031104.943262.png}
\\
\footnotesize{Ground truth}
&\IncG[ width=1.15in]{./figs/tum/gt/360_gt.png}
&\IncG[ width=1.15in]{./figs/tum/gt/desk_gt.png}
&\IncG[ width=1.15in]{./figs/tum/gt/desk2_gt.png}
&\IncG[ width=1.2in]{./figs/tum/gt/rpy_gt.png}
&\IncG[ width=1.15in]{./figs/tum/gt/xyz_gt.png}
\\
\vspace{-5mm}
\footnotesize{Ours}
&\IncG[ width=1.15in]{./figs/tum/ours/360_pred.png}
&\IncG[ width=1.15in]{./figs/tum/ours/desk_pred.png}
&\IncG[ width=1.15in]{./figs/tum/ours/desk2_pred.png}
&\IncG[ width=1.2in]{./figs/tum/ours/rpy_pred.png}
&\IncG[ width=1.15in]{./figs/tum/ours/xyz_pred.png}
\\
&\footnotesize{(a) fr1/360} & \footnotesize{(b) fr1/desk } & \footnotesize{(c) fr1/desk2} & \footnotesize{(d) fr1/rpy}  & \footnotesize{(e) fr1/xyz} \\
\end{tabular}
\caption{Qualitative comparison of predicted point clouds on the five sequences from TUM dataset. }
\label{fig_tum}
\end{figure*}


\begin{figure}[t]
\centering
\subfigure {\includegraphics[width=\linewidth]{figs/num.pdf}}
\caption{Results of KD that applies  $N_t^\mathcal{X} \rightarrow N_s(\mathcal{X \cup U})$ with different number of training samples from $\mathcal{U}$.}
\label{fig_num}
\vspace{-5mm}
\end{figure}


We compare the proposed method with two previous approaches for light-weight depth estimation, including a traditional encoder-decoder net, i.e. Fast-depth \cite{Wofk2019FastDepthFM}, joint learning of semantic and depth, i.e. Joint-depth \cite{Nekrasov2019RealTimeJS}. 
The quantitative results are given in Table~\ref{nyu_depth_acc_light}. As seen that our method outperforms other two methods by a significant margin even with only about half parameters. The $\delta_1$ accuracy of our method $N_s(\mathcal{X} \cup \mathcal{U})$ is $83.7\%$ which outperforms Joint-depth and Fast-depth by $4.7\%$ and $6.2\%$, respectively. When the auxiliary data is labeled, the improvement is more significant, as the accuracy of $N_s(\mathcal{X} \cup \mathcal{U'})$ is 85.5\%, yielding $6.5\%$ and $8\%$ improvement.
The qualitative comparisons are shown in Fig.~\ref{fig1} where the estimated depth maps of our method are more accurate and have finer details.

We also compare the GPU time for inferring a depth map from an input image. We use a  computer  with  Intel(R)  Xeon(R)  CPU  E5-2690  v3  and a  GT1080Ti  GPU  card. 
For the other two methods, we use the official implementations to calculate the computation time. The experimental results show our method infers a depth map using only 11 ms of GPU time, which is much faster than Joint-depth. On the other hand, Fast-depth achieves the smallest inference speed at the expense of  degradation of accuracy. It demonstrates the worst accuracy among the three methods. 

\subsection{Effect of Varying the number of Auxiliary Data}
We conduct an ablation study to quantify the light-weight network while varying the number of auxiliary data. Specifically, we use the teacher trained on the original labeled set and apply KD with different number of auxiliary unlabeled samples taken from $\mathcal{U}$. In our experiments, we evaluate on the 11.6K, 22.0K, 40.2K, 67.6K,153.0K and 204.2K samples.

The results are shown in Fig.~\ref{fig_num}. Generally, the more auxiliary data we use, the better KD we can apply, and there will be no more improvement when the auxiliary samples reach a certain number. 
\subsection{Cross-dataset Evaluation}
To measure the generalization performance of the light-weight model, we conduct a cross-dataset evaluation on the SUNRGBD \cite{Song2015SUNRA} and TUM \cite{sturm12iros} datasets. 
We directly apply $N_s(\mathcal{X}\cup\mathcal{U})$ and  $N_s(\mathcal{X}\cup\mathcal{U'})$ without any fine-tuning to evaluate on these two datasets.
Note that the comparison between $N_s(\mathcal{X}\cup\mathcal{U'})$ and other methods is not fair as it employs auxiliary labeled data. 
Nevertheless, we still show it here to provide an experimental support to the community that it is probably the most effective and reliable way to boost KD in a data-driven fashion.
Results for each dataset are shown as follows.

\subsubsection{Results on SUNRGBD}
The SUNRGBD dataset is commonly employed to test the generalization performance in previous works \cite{huynh2020guiding}. The results are shown in Table.~\ref{sunrgbd_evaluation} where the best and second best results are shown in red and blue, respectively.  Our method demonstrates the lowest RMSE and REL error. Joint-depth is best in $\delta_1$ and also is second best in REL. 


\begin{table}[t]
\begin{center}
\caption{The results of different methods on the SUNRGBD dataset. The best and the second best results are highlighted in red and blue, respectively.}
\label{sunrgbd_evaluation}
\begin{tabular}
{|l|ccccccccccc|}
\hline
Method  & RMSE && && & REL & && && $\delta_1$\\ \hline
Fast-depth \cite{Wofk2019FastDepthFM}  &0.662  &&&&&0.376 &&& &&0.404 \\
Joint-depth \cite{Nekrasov2019RealTimeJS} &0.634&&  &&&{\color{blue} \textbf{0.338}}  &&&&&{\color{red} \textbf{0.454}} \\
Ours $N_s(\mathcal{X} \cup \mathcal{U})$ & {\color{blue} \textbf{0.577}} &&& & &{\color{blue} \textbf{0.338}}& & && &0.430\\
Ours $N_s(\mathcal{X} \cup \mathcal{U'})$  &{\color{red} \textbf{0.531}}&&  &&&{\color{red} \textbf{0.306}} &  &&&&{\color{blue} \textbf{0.446}}\\
\hline
\end{tabular}
\end{center}
\end{table}



\begin{table}[t]
\begin{center}
\caption{The $\delta_1$ accuracy of different methods on the five sequences from TUM dataset. The best and the second best results are highlighted in red and blue, respectively.}
\label{cross_evaluation}
\begin{tabular}
{|l|ccccc|}
\hline
Method &fr1/360 &fr1/desk &fr1/desk2 &fr1/rpy &fr1/xyz \\ \hline
Fast-depth \cite{Wofk2019FastDepthFM}   &0.548 &0.308 &0.358 &0.333 &0.287 \\
Joint-depth \cite{Nekrasov2019RealTimeJS} &0.512 &0.410 &0.441 &0.552 &{\color{blue} \textbf{0.583}}\\
Ours $N_s(\mathcal{X} \cup \mathcal{U})$  &{\color{blue} \textbf{0.615}}  &{\color{blue} \textbf{0.442}} &{\color{blue}\textbf{0.498}} &{\color{blue}\textbf{ 0.611}} &0.486 \\
Ours $N_s(\mathcal{X} \cup \mathcal{U'})$  &{\color{red} \textbf{0.854}} &{\color{red} \textbf{0.695}} &{\color{red} \textbf{0.772}}  &{\color{red} \textbf{0.679}} &{\color{red} \textbf{0.905}}\\
\hline
\end{tabular}
\end{center}
\end{table}


\subsubsection{Results on TUM}
We select five sequences including fr1/360, fr1/desk, fr1/desk2, fr1/rpy and fr1/xyz as \cite{Czarnowski2020DeepFactorsRP}. 
The experimental results are shown in Table~\ref{cross_evaluation}. The depth accuracy is measured by $\delta_1$.
As seen that our method outperforms the other methods by a large margin, showing a satisfactory generalization performance. 
The average accuracy (the mean of $\delta_1$) of Joint-depth and Fast-depth is 0.494 and 0.369, respectively, while our method $N_s(\mathcal{X} \cup \mathcal{U})$ and $N_s(\mathcal{X} \cup \mathcal{U'})$ achieve an average of 0.530 and 0.781, respectively. 
The qualitative comparison between predicted point clouds and ground truths is shown in Fig.~\ref{fig_tum}.




\section{Conclusion}
In this paper, 
we revisit monocular depth estimation from the aspects of inference  accuracy and computational efficiency. We point out the inherent difficulty is the trade-off between the accuracy and model's capacity. Our method first introduces a light-weight network with a goal of using much fewer parameters. Then, we quantitatively show that employing auxiliary training data with similar scene scales is effective to improve the light-weight network, through the study of two specific cases, i.e. auxiliary unlabeled/labeled data when applying knowledge distillation. 
As a result, our method  achieves a comparable performance against methods built on large networks with about only 1\% parameters and outperforms other light-weight methods by a large margin. 


\bibliographystyle{IEEEtranS}
