\section{Introduction}

Estimating reliable depth maps is an essential task for the planning of unmanned aerial vehicles (UAV) trajectories. However, lightweight, reliable for a wide range of depths, and energy efficient depth sensors for outdoor use do not exist to date. Distances between objects and the camera therefore need to be inferred rather than be measured. Except for some pathological configurations and when in presence of motion blur, depth can be inferred from vehicle motion and video stream of an on-board RGB camera. Theoretically, for a static scene, depth can be perfectly calculated by triangulation if the exact camera motion and the frame-to-frame displacement of each pixel are known, which is not achievable in practice. 

\begin{figure}[tbp]
\centerline{\includegraphics[width=.85\linewidth]{img/graphical_abstract.pdf}}
\caption{Schematic view of our network for inferring depth from RGB video sequences, called M4Depth, that recycles previous depth map estimates and uses the drone motion to estimate the next depth map.}
\label{fig}
\end{figure}

Research has been carried to exploit recent progress in deep learning to infer depth directly from a sequence of images without having to rely on a physical model for the camera displacement. The results of these methods are encouraging on datasets and benchmarks created for autonomous car applications~\cite{Geiger2012AreWe}. However, discarding all motion information has a major drawback; depth estimation methods then have to rely on some sort of semantic priors to produce their outputs, which makes them scene specific. In particular, the constrained camera motion and the particularities of an environment of autonomous cars applications reduces the complexity of the depth estimation task. It is therefore unsure if these methods would perform equally well for to a wider variety of motion types and environments such as the ones encountered during drone flights.

When using temporal information, depth estimation can be seen as a triangulation problem. Mathematical algorithms for sparse point triangulation such as the one used for visual odometry include a frame-to-frame keypoint tracker and an iterative algorithm that performs the triangulation itself based on the displacement of both the camera and the keypoints~\cite{Scaramuzza2020Aerial}. Since triangulation is based on motion, the use of semantics is limited to keypoints detection and tracking. These algorithms are, however, computationally expensive when tracking a large number of points and are not able to take scene constraints into account due to their sparsity. This makes them poor candidates for dense depth estimation.

In this work, we look at the benefits that motion information can bring to depth estimation for drone applications when it is embedded inside of a Convolutional Neural Network (CNN) instead of being considered as an additional input. More precisely, we propose a new modular architecture that is designed to use motion and temporal information inside of the network.

The paper is organized as follows. First, we precisely formulate the addressed problem and review a number of the works published in the field of depth estimation. We then present our method, a modular deep CNN for end-to-end depth estimation that embeds a spatial re-projection layer and uses a time-recurrent feedback in each of its levels. Next, we present our experimental setup and analyze the performance of our method. For this, we establish a baseline comprising several state-of-the-art methods for depth estimation on the Mid-Air dataset~\cite{Fonder2019MidAir} and compare our method to this baseline. We also discuss our results in this section before concluding our paper.


\section{Problem Statement}

Our problem statement is as follows. We consider a drone equipped with an RGB camera and an Inertial Measurement Unit (IMU) flying in an unseen environment that is assumed to be completely static. The camera is rigidly attached to the drone and its intrinsic parameters are supposed to be known and constant. We introduce the following components and notations:
\begin{itemize}[leftmargin=*,topsep=6pt,itemsep=1pt]
    \item $I_t$ is an RGB image of size $H \times W$ recorded by the camera at timestep $t$. Images are assumed to have the following properties: 1) motion blur and rolling shutter artifacts are negligible; 2) the camera focal length $f$ is known and constant over a flight; 3) camera shutter speed and gain are unknown and can change over time.
    \item $\transformationMatrix{t}{}$ is the transformation matrix encoding the motion of optical center of the camera from timestep $t-1$ to $t$. This matrix is assumed to be known.
    \item $d_{i j,t}$ is the distance (in meters) of the surface recorded by the pixel at coordinates $(i ,j)$ of the frame $I_t$ to the camera along its $z$ axis.
\end{itemize}

Using these notation, a depth map is defined as 
\begin{equation}
\mathbf{d}_t=\{ d_{i j,t} \; | \; i \in [1,H], \; j\in [1,W] \} . 
\end{equation}

We denote by  $h_t$  the complete history of the information recorded by the drone up to timestep $t$, from the beginning of a flight. We define a set $\mathcal{D}$ of functions $D$ that are able to estimate a depth map for $\mathbf{d}$ from $h_t$ as follows:
\begin{equation}
\hat{\mathbf{d}_t} = D(h_t) ,
\end{equation}
such that $\ D\in \mathcal{D}, \  
\text{with} \  h_t= \left[I _0, [I _1, \transformationMatrix{1}{}], \ ... \ , \ [I _t, \transformationMatrix{t}{}]\right]$.

We want to find the function $D$ in this set that best estimates $\hat{\mathbf{d}_t}$ with respect to the ground truth $\mathbf{d}_t$. By defining our neural network, we implicitly select a subspace of $\mathcal{D}$ in which the parameters for $D$ will be searched for by gradient descent.

Several metrics were introduced by Eigen~\etal~\cite{Eigen2014Depth} to assess the performance of a depth estimation method. Since we are considering autonomous vehicles applications, errors on the estimate for closer objects have a higher impact than the errors occurring for objects in the background of the scene. We therefore want to minimize the error relatively to the distance of the object. The $\text{RMSE }\log$ distance metric~\cite{Eigen2014Depth} has this property. We therefore search for the candidate $D$ that minimizes this metric when comparing its outputs to the ground truth.

The network is trained in a supervised fashion on a dataset made of recorded drone trajectories for which every RGB frame of a trajectory comes with the corresponding ground-truth depth map and camera position. In accordance with good machine learning practices, the performance of the proposed architecture is assessed on a set of data that remains unknown to the training phase. 

\section{Dataset}
Our problem statement expresses the need for a dataset to explicitely perform the training and the testing of our method. A large majority of datasets providing RGB+D and motion data focus on ground vehicles (see ~\cite{Geiger2012AreWe,Ros2016TheSynthia,Sturm2012ABenchmark}), which makes them unsuitable for our needs although  most depth estimation methods are exclusively tested on them. Indeed, in ground vehicles datasets, images feature limited motion and the visual content is constrained. These datasets are not appropriate for our purpose since we target drone applications and the challenges that come with them, among which the exploitation of a 6 degrees of freedom (6DoF) motion and varied visual environments, .

To the best of our knowledge, only two datasets provide data appropriate for drone applications, namely EuRoC MAV~\cite{Burri2016TheEuroc} and Mid-Air~\cite{Fonder2019MidAir}. The former features recordings from real sensors and was entirely recorded in two rooms of an industrial building. The latter is a synthetic dataset recorded in unstructured environments under varied weather and lighting conditions.

While the testing of our method on real data is potentially attractive, the EuRoC MAV dataset has two major drawbacks: (1) with only 22 minutes of recordings, it falls short compared to other larger datasets and (2), more importantly, the train and test data are recorded in the same place which makes it impossible to predict the performance for places never seen during training. On the other hand, with its 79 minutes of flight replicated in varying climate conditions, the Mid-Air dataset is much larger. Additionally, all trajectories are recorded in different places of the virtual environments which makes it much more suitable to train and test methods for robustness to previously unseen visual content. For these reasons, we have decided to validate our method on the Mid-Air dataset.

\section{Related work}
\label{sec:rel-work}

Being recent, the Mid-Air dataset currently lacks a proper baseline for depth estimation. In this section, we briefly review the state of the art in depth estimation and select a few methods that will compose our baseline on this dataset. The chosen methods are representative of different families detailed hereafter.
\\
\textbf{Depth from a single picture.}
Estimating depth from a single picture has a long history. Its main focus consists in on estimating depth from a single RGB picture. If the first methods were fully handcrafted with unsatisfactory results, the growth of machine learning and the development of CNNs has led to a massive improvement in the quality of the depth estimates. All current state-of-the-art methods are indeed based on CNNs, the main difference between them being the architecture and the used training procedure. As this field of research has been surveyed multiple times~\cite{Godard2019DiggingIS,Godard2017UnsupervisedMD, Ming2021DeepLearning, Xiaogang2020MonocularDE, Zhao2020MonocularDepth}, we refer the reader to these surveys for comprehensively detailed descriptions of the methods.

Some general observations have been made in these survey papers which are worth noticing. Estimating depth from a single picture comes with a major drawback in general and especially for autonomous vehicle applications. Since the problem is ill-posed, networks have to heavily rely  on priors to compute a suitable proposal. Such dependency on priors leads to a lack of robustness and generalization. Therefore, methods of this family need to be fine-tuned for every new scenario or environment encountered in order to be able to produce reasonable estimates.

A good method for single image depth estimation is Monodepth, a network proposed in~\cite{Godard2017UnsupervisedMD}. We select it for our baseline since it has already been used as a baseline in the presentation paper of Mid-Air. 
\\
\textbf{Depth from an image sequence.}
Attempts to use the temporal information featured in an image sequence for depth estimation are newer. Recent proposals~\cite{Kumar2018DepthNetAR, Patil2020DontFT,Wang2019RecurrentNN,Zhang2019ExploitingTC} have been made to include recurrence in the networks to make use of temporal information for improving estimates. Current proposals mainly replace certain layers by recurrent layers in existing CNN architectures.

As such,  current  depth estimation from sequence methods still badly  use any kind of  temporal  information. Furthermore,  the  lack  of  input  about  the  real  camera  motion makes these methods completely unsuitable for estimating the proper scale for depth without relying on semantic priors. Even worse, there is no proof that the scale of the outputs remains constant over the sequence. This is particularly problematic for autonomous vehicle applications.

We select the ST-CLSTM network proposed by~\cite{Zhang2019ExploitingTC} for our baseline because this method achieved state-of-the-art performances in this category on the KITTI dataset~\cite{Geiger2012AreWe} when it was published and because there exists an implementation of it on GitHub~\cite{Weihaox2020}.
\\
\textbf{Use of motion information.}
When motion is used by depth estimation methods, it is exclusively exploited to build a loss function for unsupervised training~\cite{Godard2019DiggingIS,Luo2020EveryPC,Mahjourian2018Unsupervised,Patil2020DontFT,Wang2019RecurrentNN}. In these cases, motion is learnt by a separate network, also in an unsupervised fashion, and depth is still estimated without any direct motion information. As the core of the depth estimation networks does not change for these methods, estimations suffer from the same issue as their counterparts that do not use motion.

One notable exception to these observations is the idea proposed by~\cite{Luo2020ConsistentVD}. This method uses the unsupervised loss based on motion estimation to fine-tune the network at test time. This method features outstanding performance, although they are achieved at the cost of a large computational burden. Furthermore, due to its architecture, this method  cannot estimate depth before the whole video sequence is available. It is therefore not designed to run in an online fashion, which makes it inappropriate for autonomous vehicle applications.

For this category, we use Monodepth2~\cite{Godard2019DiggingIS} and the method proposed by Wang \etal. in~\cite{Wang2019RecurrentNN} as baselines. The former is built upon on Monodepth~\cite{Godard2017UnsupervisedMD} while the latter proposes a supervised method showing promising results. Additionally, the code for both methods was released by their respective authors.
\\
\textbf{Structure from motion.} Structure from motion is a research field that developed in parallel with depth estimation. The idea here consists in reconstructing 3D shapes from a batch of 2D pictures that capture the scene to reconstruct with different points of view. Reconstruction is achieved by explicitly modeling the relative camera position between the pictures of the set. Approaches for performing this task are varied~\cite{Ozyesil2017ASurvey} and are, by their nature, not all suitable for real-time depth estimation. Some of them however appear to be compatible for depth estimation on sequences~\cite{Ummenhofer2017DeMoNDA,Yao2018mvsnet} while others were specifically designed to work on image sequences in real time~\cite{Duzcceker2020DeepVideoMVSMS, Teed2020DeepV2DVT}.

The approaches proposed by~\cite{Duzcceker2020DeepVideoMVSMS} and~\cite{Teed2020DeepV2DVT} are similar. They both propose a three-stage network. Their stages are an image-encoding network followed by the computation of a cost volume that is finally processed by a depth estimation network. The purpose of the cost volume consists of providing the costs for matching a point in an image with a series of candidates in another image. The cost volume in both~\cite{Duzcceker2020DeepVideoMVSMS} and~\cite{Teed2020DeepV2DVT} is built by a plane-sweeping method~\cite{Collins1996ASA,Gallup2007RealTimePS}. This relies totally on a precise relative camera positioning. Any imprecision in the camera positioning will skew the cost volume and severely alter the depth estimation that follows.

While the multi-stage nature of these two methods makes them arduous to train, they present the major advantage of estimating depth without directly using any semantic cues. This naturally makes them resilient to visual novelties. They would have made a interesting addition to our baseline, but their extremely long training time (one whole week according to~\cite{Teed2020DeepV2DVT}) convinced us that it would be hard to get their performance on Mid-Air in a reasonable period of time.

\section{Our M4Depth method}

Similar to the ideas developed for structure from motion, we base our method on a notion of cost volumes generated by using motion information to decouple depth estimation from semantic information. However, we use an approach different than the plane-sweeping method~\cite{Collins1996ASA,Gallup2007RealTimePS} to build them. Instead of using a multi-stage network, we propose to build a lighter multi-level architecture that can be trained in an end-to-end fashion. In this section, we detail our complete architecture and the loss function used for training.

\subsection{Network definition}

\begin{figure}[t!]
\centerline{\includegraphics[width=.88\linewidth]{img/architecture-overview.pdf}}
\caption{Overall architecture overview of M4Depth with 3 levels. The blue convolution blocks encode the image while the orange ones are in charge of producing depth estimates. }
\label{fig:global-overview}
\end{figure}

Depth estimation can be seen as a point triangulation problem when camera motion information is paired with the video sequence. Approaching depth estimation as such allows for a reduction in the dependency of estimates on semantic priors. Point triangulation is inherently an iterative process. Instead of simply iterating on a full network as proposed in~\cite{Ummenhofer2017DeMoNDA} or bypassing the iteration by proposing a full range of candidates as in~\cite{Duzcceker2020DeepVideoMVSMS,Teed2020DeepV2DVT}, we propose approaching the iterative process in a similar manner as the PWC-Net~\cite{Sun2018PWCNetCF}.

PWC-Net~\cite{Sun2018PWCNetCF} is a neural network that was developed for optical flow. Optical flow is defined as the pattern of apparent motion of image objects between two consecutive frames caused by the movement of these objects and the camera. It takes the form of a 2D vector field where each vector is a displacement vector showing the movement of points from the first frame to the second one. Methods capable of recovering the optical flow are, by definition, able to perform frame-to-frame dense point tracking.

As for point triangulation, optical flow estimation can be approached as an iterative process. In PWC-Net, the iterative process is embedded in the architecture itself. The architecture is a modified version of the U-Net architecture with skip connections introduced by~\cite{Ronneberger2015UNet} where each level of the decoder has to produce an optical flow estimate. This estimate is used to compute a cost volume based on the feature maps produced by the encoder for the two frames between which the optical flow has to be computed. The cost volume and the estimate are then passed to the next level of the decoder whose purpose consists of refining the estimate. The levels of this architecture are generic and can be stacked at will to obtain the desired network depth.

Based on the idea of PWC-Net, our approach consists of a modified version of the U-Net architecture, where each level in the decoder part of the network has to refine the depth estimate produced by the level preceding it in the architecture. We use the same encoder as PWC-Net, our contribution is located in the decoder part of the network.

\begin{figure}[t!]
\centerline{\includegraphics[width=1\linewidth]{img/level-details-updated.pdf}}
\caption{Details for one of our architecture level. 3D boxes represent data with a spatial dimension. Inputs are highlighted in orange, outputs in blue and recurrent information in yellow. The rounded boxes represent convolutional layers with their filter size, their stride and their number of filters.}
\label{fig:lvl-overview}
\end{figure}

At its input, each level of the decoder receives information from current and previous time steps. This information is first ingested by a preprocessing unit that has it converted in such a way
 that allows the decoder to focus on the refinement of the depth estimate it receives. This preprocessing unit does not contain any learnable parameters. Once processed, the data is fed to a small convolutional subnetwork whose purpose is to estimate depth from the data it receives.  We give a block overview of our architecture in Fig.~\ref{fig:global-overview}.

The operations performed by a preprocessing unit are further detailed in Fig.~\ref{fig:lvl-overview}. The first operation it performs consists in using the depth estimate and the motion information to spatially reproject the feature and depth maps inherited from the previous time step. This allows one to spatially realign data according to the depth estimate. If the estimate is imprecise, the realignment will not be perfect. A cost volume is then computed between the spatially reprojected feature map and the feature map coming from the encoder for the current frame. Finally, all the information that will be fed to the learning part of the decoder is concatenated. 

Let us now describe the building blocks of our architecture and motivate their design.
\\
\textbf{Reprojection Layer.} This layer does not have any learnable parameters. It spatially moves vectors of a feature map to some new location on the sensor plane, the one to which the spatial point encoded by each vector would be projected when seen from a different point of view by the camera. In our architecture, we reproject vectors of the previous timestep to their new location in the current frame.

The camera motion between consecutive frames is encoded by the transformation matrix $\transformationMatrix{t}{}$ and can be broken down in a rotation matrix $\mathbf{R}_t$ and a translation vector $\mathbf{t}_t$:
\begin{equation}
    \transformationMatrix{t}{}=[\mathbf{R}_t|\mathbf{t}_t] .
\end{equation}
Taking into account this camera motion $\transformationMatrix{t}{}$ and for the pinhole camera model, a point in space projected at coordinates $(i _{t},j _{t})$ in the current frame $t$ will be linked to its previous coordinates $(i _{t-1},j _{t-1})$ in frame $t-1$ by the following relation:
    \begin{equation}
    \resizebox{.9\columnwidth}{!}{$
    \mathbf{d}_{t-1}(i_{t-1},j _{t-1}) \begin{bmatrix}i_{t-1}\\_{t-1}\\1\end{bmatrix}= \mathbf{K} \mathbf{R}_t \left( \begin{bmatrix} (i_{t}-c_x)/f\\ (j_{t}-c_y)/f\\ 1 \end{bmatrix} \mathbf{d}_t(i_{t},j_{t}) + \mathbf{t}_t \right),
    \label{equ:reproj}
    $}
    \end{equation}
   
    where $\mathbf{K}$ is the $3 \times 3$ camera intrinsic matrix, $f$ the camera focal length, and $(c_x,c_y)$ the coordinates of the principal point of the camera.
    
    Moving the vectors across the frame is performed trough a warping operation using bilinear interpolation, as described in \cite{Sun2018PWCNetCF}. For the $l$-th level of our architecture, the warping of a feature map $\mathbf{f}$ can be expressed as follows:
    \begin{equation}
        \mathbf{f}^l_w(i _{t},j _{t}) =  \mathbf{f}^l(i _{t-1},j _{t-1}),
    \end{equation}
    where the coordinates $(i_{t-1},j_{t-1})$ are computed using Equ.~\ref{equ:reproj} and the upscaled depth map produced by the previous level, $\mathbf{d}^{l+1}_t$. Any imprecision on $\mathbf{d}^{l+1}_t$ will lead to an imprecise spatial reprojection. When detected properly, these reprojection inaccuracies can be used to detect and correct inaccuracies in the depth estimate.
    
    As such, this layer leads to gradient explosions during training. Instabilities are due to the gradients computed on the depth map used for computing the reprojection. To prevent this issue and to allow the network to train properly, we do not propagate the gradients of the reprojection layer to the depth map.
    \\
    \textbf{Cost Volume Layer.} This layer does not have any learnable parameters. Its purpose consists in detecting any reprojection imprecision induced by the reprojection layer. For this, it builds a volume where each value represents the cost of matching two given feature vectors. The cost of matching two feature vectors $\textbf{x}_1$ and $\textbf{x}_2$ is defined as their correlation: 
    \begin{equation}
        \text{cost}(\textbf{x}_1, \textbf{x}_2) = \frac{1}{L} \textbf{x}_1^T \textbf{x}_2 ,
    \end{equation}
    where $L$ is the length of the feature vectors. The volume is built by computing this cost for each vector of a feature map $\mathbf{f_1}$ with all vectors located within a spatial neighbourhood $r$ in another feature map $\mathbf{f_2}$ of same dimensions:
    \begin{equation}
    \resizebox{.9\columnwidth}{!}{$
    \begin{aligned}
        &\text{cost\_vol}(\mathbf{f_1}, \mathbf{f_2}, r)(i,j) = \\
        &\  \left[\ \text{cost}\left(\mathbf{f_1}(i,j), \mathbf{f_2}(i+k_i,j+k_j)\right) \; \forall \; k_i,k_j\in \{-r,\ldots,r\} \ \right] .
    \end{aligned}
    $}
    \end{equation}
    
    When computed for feature maps of two successive frames, the cost can be used as a similarity metric for matching points between these two frames and therefore detect potential misalignements left by the reprojection layer. The purpose of each level of our architecture being to correct the residual depth error, we assume that matching feature vectors are located in each others neighbourhood after reprojection. For each level $l$ of our architecture, we compute a cost volume for a neighbourhood of 4 pixels:
    \begin{equation}
        \mathbf{cv}^l_t = \text{cost\_vol}\left(\mathbf{f_{enc}}^l_t, \  \mathbf{f_{enc}}^l_{t-1,w}, \ 4\right).
    \end{equation}
    It is important to note that a range $r$ at the $l$-th level of the architecture will correspond to a range of $2^{lr}$ in the input image due to the reduction of the spatial dimension induced by the image encoding layers.
    
    As our cost volume is built by taking into account the entire neighbourhood of a pixel, it is robust to potential reprojection errors that could occur because of imprecision in motion information. Corresponding vectors in consecutive frames are, indeed, likely to be matched together despite an error on motion information with our architecture design. The same cannot be said for cost volumes built based on the plane-sweeping method. When compared to the plane-sweeping method, our method however has one drawback: as opposed to the ones built with the former, matches in our cost volumes do not provide direct information on the corresponding depth. The position of a match has to be combined with motion information and previous depth estimates to derive information about depth.
    \\
    \textbf{Depth Estimator.} This sub-network consists of seven convolutional layers in charge of producing a depth estimate. The inputs of the depth estimator are the feature map of the current image, the cost volume $\mathbf{cv}^l_t$, the wrapped depth estimate produced at the previous time step $\mathbf{d}^l_{t-1,w}$, and the upscaled depth estimate produced by the previous level of the architecture. Additionally, each pixel is given its grid coordinates on the sensor plane and camera motion information.
    
    The convolutional layers have the same number of filters for all levels of the architecture (128, 128, 128, 96, 64, 32, 1) and are followed by a leaky ReLU activation unit, excepted the last one that remains inactivated. All convolution filters have a kernel size of $3 \times 3$ and a stride of 1. In order to ease the convergence, depth maps are encoded in a log space at the input of the sub-network. The depth estimator is taught to produce its output in a log space.

\input{tables/performance_comparison}
    
\subsection{Loss function definition}
Since the levels of our architecture are stackable at will, the architecture can have any depth. In the following paragraph, we detail our loss function for a network that is made of $M$ levels.

We could train this network by using our performance metric directly as a loss function. It is however possible to design a loss function that features better convergence properties. In order to train such a deep architecture efficiently, we use a custom loss function that aggregates several weighted loss terms.

For each frame in a sequence, we compute the L1 distance on the logarithm of the depths for the estimate produced by each level. Doing so eases the training of the deepest layers in the architecture. Since intermediate depth maps have a lower resolution, we resize the ground truths by bilinear interpolation to match the dimensions of the estimates. The resulting terms are aggregated together through a weighted sum:
\begin{equation}
    \mathcal{L}_{1t} = \frac{\alpha}{HW} \sum_{l=1}^{M} \sum_{\dpix_{ij}\in\mathbf{d}_t^l} 2^{l+1} \ |\log(\dpix_{ij}) - \log (\hat{\dpix_{ij}})| ,
\end{equation}
where $\alpha$ is a weighting parameter. We set it arbitrarily to $0.64$.

We use this loss function to define the total loss for the complete sequence. For a sequence on length $N$, the total training loss for our depth estimation network is computed as follows:
\begin{equation}
    \mathcal{L}_{\text{tot}} = \frac{1}{N} \sum_{t=1}^{N}  \ (\mathcal{L}_{1t}) + \gamma|\mathbf{\Theta}|_2 ,
\end{equation}
where $|\mathbf{\Theta}|_2$ is an L2 regularising term on the weights of the parameters in network and where $\gamma$ is the weighting parameter of this regulariser. We set it arbitrarily to $0.0004$. 

\section{Experiments} 

\subsection{Experimental setup}
In the next paragraphs, we detail the dataset settings used for testing our architecture and give the complete set of specifications of the training parameters for our network.
\\
\textbf{Dataset.}
With the test set proposed in the original paper~\cite{Fonder2019MidAir} being quite small, we propose an alternate train/test split. In order to guarantee a representative test set, we divide the train set proposed in the original paper as follows: trajectories whose number is a multiple of 3 (including the 0 one) are allocated to the test set. This de facto creates a two-third/one-third split of the original train set for our train/test sets.

We subsample the frame rate by a factor 4 (from 25 to $6.25$ fps) to increase the motion experienced by the camera between two frames, and we divide the long trajectories in non-overlapping sub-sequences of 8 frames in both the train and test sets. This results in $8,704$ trajectories for training and $4,352$ for testing. For all our experiments in this work, we resize the pictures and the depth maps to a size of $384 \times 384$ pixels. We use bilinear interpolation to resize color images and the nearest neighbour method for depth maps.
\\
\textbf{Training.}
With everything else properly defined, we can detail our training procedure. We use a He initialization~\cite{He2015DelvingDI} for our variables and an Adam optimizer~\cite{Kingma2015Adam} for the training itself. We keep the default parameters proposed by both of them. All our trainings are performed on GPUs with 12~GB of VRAM with a batch size of 3 sequences and for 200~k iterations. For the learning rate, we begin the training process by setting it to $10^{-4}$. We then divide it by 2 every 60~k iterations. Depending on the network parameters, a full training can require up to 40 hours to complete. In order to assess the raw capabilities of our method, no data augmentation was used during training.

\subsection{Baseline}

With the Mid-Air dataset being relatively new, there are currently no performance reports for depth estimation methods on it. A baseline is however required in order to assess the benefits of our approach. We motivated a selection of candidate methods to use as references in section \ref{sec:rel-work}. As a reminder, we chose to use four methods, \ie~\cite{Godard2019DiggingIS,Godard2017UnsupervisedMD,Wang2019RecurrentNN,Zhang2019ExploitingTC}, to build our baseline. The full code for~\cite{Godard2019DiggingIS,Godard2017UnsupervisedMD,Wang2019RecurrentNN} was released by their authors~\cite{Godard2017Monodepth,Godard2019Monodepth2,Wang2019RNN}. We rely on their work for building our baseline. The authors of~\cite{Zhang2019ExploitingTC} only released a pretrained network. For this method, we use an unofficial implementation released on GitHub~\cite{Weihaox2020} for our baseline.

In order to obtain the fairest comparison possible, we trained the networks for the same number of steps and with batches containing a similar number of pictures. We chose a batch size of 18 for methods performing single-image depth estimation~\cite{Godard2019DiggingIS,Godard2017UnsupervisedMD}. We trained methods working on sequences~\cite{Wang2019RecurrentNN,Zhang2019ExploitingTC} on batches of 3 sequences of length 5. We trained all networks five times for 200~k steps before reporting their performance.

We kept the default parameters for~\cite{Godard2017UnsupervisedMD} and~\cite{Wang2019RecurrentNN}. It must, however, be noted that~\cite{Godard2017UnsupervisedMD} failed to produce any output for three trainings out of the five we performed. Performance obtained with the default learning rate were extremely poor for~\cite{Zhang2019ExploitingTC}. We obtained better results by reducing it to $10^{-4}$. Finally, we could not get a good convergence for~\cite{Godard2019DiggingIS} on Mid-Air. Training would simply fail on sequences and lead  to poor performances when trained on stereo data. Reducing the learning rate did not help in any way in this case. Results are reported in Table~\ref{tab:perf_scores}.

\subsection{Performance analysis}
With our architecture levels being generically stackable, we trained our network for different architecture depths unrolled for 6 time steps and reported their performance in Table~\ref{tab:perf_scores}. It appears that our network proposal consistently outperforms the baseline by a large margin. As expected, increasing the depth of the architecture leads to better performances, but even the shallowest architecture performs better than the baseline. A qualitative comparison of the outputs of the different methods is shown in Fig.~\ref{fig:qualitative-overview}.

\input{img/results/image_results-grid}

In order to assess the use of temporal information by our network, we perform an additional set of tests. We unroll our network over several numbers of time steps for training and report their performance on the test set for different sequence lengths. To ensure the coherence of the obtained scores, we compute the performance only for the last image of each of the $4,352$ test sequences. When testing a network on a sequence of length $N$, we therefore give it the last $N$ frames of each test sequence.

\input{tables/seq_len_grid-small}

The results of this experiment are reported in Table~\ref{tab:seq_len_test}. They show that temporal information is important to get good performances. Networks working on sequences perform far better than the ones working with a single image. Performances get marginally better as the length of sequences increases. They however reach a plateau passed a sequence length of 4 frames. This could indicate that our architecture is limited in its ability to use recurrent information. Additionally, increasing the length of the training sequences does not seem to lead to better performances.

At inference time and without taking data loading operations into account, an NVidia Tesla V100 GPU needs $29$~ms to process a single frame for a non-optimized TensorFlow implementation of our network with 6 levels. This corresponds to $34$~frames per second and therefore already meets real-time constraints. In its current implementation, our network requires $808$~MB of memory to run.

\subsection{Discussion}

The gap between performances of our method and that of the baseline raises a fundamental question. Since some baseline methods also work on video sequences in a supervised fashion~\cite{Wang2019RecurrentNN,Zhang2019ExploitingTC}, why does our model perform so well compared to them? In the next paragraphs, we propose some explanations.

In the methods we chose for our baseline, two of them work on sequences. They should therefore perform better than the other ones. This intuition is verified on the KITTI dataset. The performances reported in the original papers show a clear gap between the different methods, with the ones predicting depth from a single image achieving the worst scores. However, we observe that three out of the four baseline methods achieve almost the same performance when compared on the Mid-Air dataset.

A reasonable hypothesis is that current recurrent methods work well for constrained motion, but fail to properly use temporal information for more varied motion. When compared to a drone flight for which the camera motion has up to 6DoF, the motion of a camera mounted on the rooftop of a car is much more constrained. A car driving on a flat road has indeed only 3 degrees of freedom in practice, 2 for translations in the horizontal plane and one for rotations around the vertical axis, everything being constrained by the physics of the car.

This hypothesis is supported by the scores shown in Table~\ref{tab:seq_len_test}. Our network, when deprived of temporal information, performs similarly to the baseline. This further supports the idea that the ability of current recurrent networks to properly use temporal information on Mid-Air is limited and highlights the benefits motion information can bring to the quality of the estimates. 

\section{Conclusion}
In this work, we presented a new method, named M4Depth, for performing depth estimation on sequences recorded by a camera whose motion has up to six degrees of freedom. Our method uses motion information as well as time recurrence to produce its estimates. We made this possible by embedding spatial feature reprojection layers at each level of our architecture. Our network is relatively lightweight, runs in real time, and can be trained in an end-to-end fashion.

In order to analyse the performance of M4Depth, we have established a baseline on Mid-Air, a drone dataset, by retraining state-of-the-art depth estimation methods on it. Results show that M4Depth substantially outperforms the baseline, even in its most basic form. Our experiments show that the use of temporal information is the key contributing factor for this performance improvement.

Despite this improvement, we have noticed that our architecture has a limited ability to exploit the whole trajectory history. In the future, we will investigate alternative architectures that could better use this history. Also, we will test if data augmentation is useful for M4Depth and possible variants. 

\input{appendix/main}

{\small
\bibliographystyle{ieee_fullname}
\typeout{} 
\section{Camera model and geometry}
Our work relies on some hypotheses on the optics of the camera. In this section, we proceed to detail the mathematical model used for our camera.

\begin{figure}[ht]
  \centering
 
  \includegraphics[width=.48\textwidth]{appendix/img/pinhole.pdf}  
  \caption{Pinhole camera model visualization}
\label{fig:pinhole}
\end{figure}

We use the pinhole camera model. With this model the camera is simply represented by a sensor plane and a focal point, the optical centre of the camera, that is used as the camera origin, as illustrated in Fig.~\ref{fig:pinhole}. In a perfect camera, the focal point is located at some point on the principal axis, the axis intersecting the sensor plane perpendicularly on its central point. In computer vision, the distance separating the focal point from the sensor plane is called the focal length and is expressed as a multiple of a sensor pixel width.

A pinhole camera is fully characterised by 5 intrinsic parameters that are:
\begin{itemize}
    \item $f_x$ and $f_y$, the focal lengths along the x and y axis, respectively;
    \item $s$, the skew factor of a pixel;
    \item $(c_x,c_y)$, the coordinates of the principal point on the camera sensor
\end{itemize}
In a perfect camera pixels are square. In this case, $f_x$ and $f_y$ are equal to each other and $s$ is equal to zero. In a perfect camera, the principal point is located at the centre of the sensor. These parameters can be assembled in a projection matrix $\mathbf{K}$ as follows:
\begin{equation}
    \mathbf{K} = \begin{bmatrix}f_x & s&c_x\\ 0&f_y&c_y\\ 0&0&1\end{bmatrix}.
\end{equation}

The projection of a point in space on the sensor plane is located at the intersection of the line joining this point to the focal point with the sensor plane. For the pinhole camera model, the pixel coordinates $(i,j)$ resulting from the projection of a point located at coordinates $(x,y,z)$ in the camera referential are obtained by using the camera intrinsic matrix $\mathbf{K}$ and the right-angle theorem as follows:
\begin{equation}
    (i,j)=\left(\frac{\alpha}{z},\frac{\beta}{z}\right) \quad \text{with} \quad \begin{bmatrix}\alpha\\ \beta\\ z \\ 1\end{bmatrix}=  \begin{bmatrix}\mathbf{K} & \mathbf{0} \\ \mathbf{0} & 1 \end{bmatrix} \begin{bmatrix} x\\ y\\ z\\ 1 \end{bmatrix}.
    \label{equ:pinhole}
\end{equation}
In practice, the 3D coordinates of a point are rarely expressed within the camera frame of reference. It is more common to know the position of a point with respect to a given frame of reference and to know the camera pose with respect to the same frame of reference. If we express the camera position by a vector $\mathbf{p}$ of size 3 and the camera orientation by a $3\times 3$ rotation matrix $\mathbf{R}$, the position $(X,Y,Z)$ of a point in space can be expressed with respect to the camera frame of reference by applying the following transformation:
\begin{equation}
    \begin{bmatrix}x\\ y\\ z \\ 1\end{bmatrix}= \underbrace{\begin{bmatrix} \mathbf{R}^T & -\mathbf{R}^T\mathbf{p}\\ \mathbf{0}&1\end{bmatrix}}_\mathbf{T} \begin{bmatrix} X\\ Y\\ Z\\ 1 \end{bmatrix},
\end{equation}
where $T$ denotes the matrix transpose operator. In this equation, $\mathbf{T}$ is referred as the transformation matrix.

\section{Reprojection layer mathematical details}

In this section, we further detail and motivate our reprojection layer. The reprojection layer involves two distinct poses of the same camera. We denote these poses by $\text{C}_1$ and $\text{C}_2$. $\text{C}_2$'s pose is expressed relative to the pose of $\text{C}_1$ and is encoded by the transformation matrix $\mathbf{T}_2$ with:
\begin{equation}
    \mathbf{T}_2 = \begin{bmatrix} \mathbf{R} & \mathbf{t}\\ \mathbf{0}&1\end{bmatrix}.
\end{equation}

Let us assume that some visual information is encoded for each pixel of $\text{C}_1$ and that this information is not directly available to $\text{C}_2$. A point $\text{P}$ seen by $\text{C}_1$ and projected on the sensor plane in $(i_1,j_1)$ has a high probability of also being seen by $\text{C}_2$, but is projected in a different location on the sensor, $(i_2,j_2)$. When P is visible to both cameras, the information encoded for this point should be similar for both camera poses. The purpose of the reprojection layer is to transfer the encoded information from $\text{C}_1$ to $\text{C}_2$ by using the geometrical constraints of the system. 

\begin{figure}[ht]
  \centering
 
  \includegraphics[width=.3\textwidth]{appendix/img/cameras-triang.pdf}  
  \caption{Illustration of the setup used by the reprojection layer.}
\label{fig:triangulation}
\end{figure}

To be able to transfer information from one projection to the other, we first need to find the relation that links $(i_1,j_1)$ and $(i_2,j_2)$. From Equ.\ref{equ:pinhole}, it can be seen that recovering the full 3D coordinates of a point whose projection coordinates $(i,j)$ and depth $z$ are known is trivial if the intrinsic matrix is known (which is often the case in computer vision). Assuming that P is located at a depth $d_2$ of $\text{C}_2$, its 3D coordinates with respect of $\text{C}_2$ are given by:
\begin{equation}
    \mathbf{P}_{\text{C}_2} = \begin{bmatrix} (i_{2}-c_x)/f\\ (j_{2}-c_y)/f\\ 1 \end{bmatrix} d_2,
    \label{equ:reproj_1}
\end{equation}
if we assume that $f_x$ and $f_y$ are both equal to $f$ and that the skew factor $s$ is negligible.
 
These coordinates are expressed with respect to the $\text{C}_2$ referential. Their expression in $\text{C}_1$ is given by:
\begin{equation}
    \mathbf{P}_{\text{C}_1} = [\mathbf{R}|\mathbf{R}\mathbf{t}] \begin{bmatrix}\mathbf{P}_{\text{C}_2} \\ 1 \end{bmatrix} =  \mathbf{R} (\mathbf{P}_{\text{C}_2} + \mathbf{t}).
    \label{equ:reproj_2}
\end{equation}
Computing the projection coordinates $(i_1,j_1)$ for P then simply consists of applying the camera projection equation:
\begin{equation}
    d_1 \begin{bmatrix}i_{1} & j_{1} & 1\end{bmatrix}^T= \mathbf{K} \mathbf{P}_{\text{C}_1}.
    \label{equ:reproj_3}
\end{equation}
When Equ.~\ref{equ:reproj_1}, \ref{equ:reproj_2} and \ref{equ:reproj_3} are combined together, we have:

\begin{equation}
    d_1 \begin{bmatrix}i_{1}\\j_{1}\\1\end{bmatrix}= \mathbf{K} \mathbf{R} \left( \begin{bmatrix} (i_{2}-c_x)/f\\ (j_{2}-c_y)/f\\ 1 \end{bmatrix} d_2 + \mathbf{t} \right).
    \label{equ:reproj_tot}
    \end{equation}

Equation~\ref{equ:reproj_tot} would have been enough to move information on the sensor plane from $\text{C}_1$ to $\text{C}_2$ if visual information was defined continuously on the sensor plane. However, this is not the case. In practice, the sensor plane is defined as a discrete grid where each element of this grid is a pixel of the image. It has therefore to be indexed by integer coordinates. Equation~\ref{equ:reproj_tot} shows that, if one of the coordinates $(i_1,j_1)$ or $(i_2,j_2)$ can be defined on a discrete grid, the other cannot since transformations are defined in the real domain. This leads to an issue for applying Equ.~\ref{equ:reproj_tot}.  

As a reminder, we want to transfer information from $\text{C}_1$ to $\text{C}_2$. This means that the sensor plane of $\text{C}_1$ is populated with the information that is to be transferred. In such a configuration, the information on the sensor plane can be made continuous by using interpolation. Indexing the sensor plane of $\text{C}_1$ with coordinates defined in the real domain therefore becomes possible and Equ.~\ref{equ:reproj_tot}  can be used without any modification.

In summary, our reprojection layer computes coordinates $(i_1,j_1)$ from $(i_2,j_2)$ and a depth estimate $d_2$, samples the information located at these coordinates on the sensor plane of $\text{C}_1$ by using bilinear interpolation, and then copies it to the coordinates $(i_2,j_2)$ for $\text{C}_2$. In practice, in our architecture, we respectively use the notations $\text{C}_{t-1}$ and $\text{C}_t$ instead of $\text{C}_1$ and $\text{C}_2$. 



