\section{Introduction}
\IEEEPARstart{S}{imultaneous} localization and mapping (SLAM) is one of the most fundamental tasks in the field of computer vision and robotics, with applications ranging from augmented reality (AR), virtual reality (VR) to autonomous driving. In AR applications, SLAM is often used to provide accurate localization to facilitate users to place virtual objects~\cite{Ye2021SuperPlane}, while the dense reconstruction is increasingly needed for better interaction with the environment. Monocular dense SLAM~\cite{deepfactors, tateno2017cnn} has received much attention due to the simplicity of monocular video acquisition, yet it is a much more difficult task compared to RGB-D SLAM~\cite{wang2021DymSLAM, Li2022DGS-SLAM, yu2018dsslam, dai2020rgb, henein2020dynamic}.

\begin{figure}[h]
  \centering
  \includegraphics[width=\linewidth]{pictures/dual-flow_representation.png}
  \caption{\textbf{Dual-Flow Representation.} The estimated optical flow is decomposed into a static flow caused by the camera poses and a dynamic flow caused by the dynamic object's motion.}
  \label{fig: dual-flow representation}
  \vspace{-1em}
\end{figure}

Impressive progress has been seen in geometry-based~\cite{Engel2014lsd, mur2015orb, forster2014svo, engel2017direct}, learning-based \cite{Zhou2017SfMLearner, wang2018end, wang2019improving}, and hybrid approaches \cite{wang2017non, wang2017noniterative, brahmbhatt2018geometry}. However, developing robust and reliable SLAM methods for real-world AR applications is still a challenging problem, especially in dynamic scenarios. To meet the challenges of such dynamic environments, some approaches~\cite{bescos2018dynaslam, xiao2019dynamic} filter out dynamic objects in advance by introducing information such as semantics, e.g., using Mask R-CNN~\cite{He_2017_ICCV} to segment out potentially moving vehicles and pedestrians, and then run monocular SLAM systems such as ORB-SLAM~\cite{mur2015orb}. However, due to the slow running speed and high memory consumption of Mask-RCNN, it is difficult to meet the real-time requirement of SLAM. What's more, there are limits in practical applications, for not all dynamic objects exist in the detector's training data and may lead to catastrophic system failure~\cite{wang2021DymSLAM}.
Besides, simply throwing away dynamic information and only constructing sparse feature maps makes such SLAM methods incapable of handling tasks like planning and interaction.
Some approaches \cite{yang2019cubeslam, huang2020clustervo} combine multi-object detection and SLAM, making it possible to add target-aware constraints to mitigate the interference of dynamic objects, but the generalization of such SLAM systems is limited as well due to the detector.

Recently, a learning-based dense SLAM system, DROID-SLAM~\cite{teed2021droid}, has been proposed, which demonstrates better accuracy and robustness than traditional methods. However, we find that it has relatively large errors in the pose estimation in some stronger dynamic scenarios such as sequence 09 of KITTI~\cite{geiger2013vision}. DROID-SLAM~\cite{teed2021droid} presents a dense bundle adjustment layer to iteratively update the residual of inverse depths and camera poses using the estimated optical flow and weights, while the presence of dynamic objects may lead to ambiguity in optical flow estimation if the system failed to discover them. For this reason, we propose a novel scene motion representation, named dual-flow, that decomposes the estimated optical flow into a static flow caused by the camera poses and a dynamic flow of the dynamic object's own motion, as shown in Fig. \ref{fig: dual-flow representation}. Such a representation largely mimics the way humans perceive the real world~\cite{villacorta2021static}. In addition, DROID-SLAM introduces massive supervision for optical flow, depth, and pose estimation during training, which severely restricts the possibility of finetuning in new scenes if it performs poorly in new scenes. On the contrary, we explore to train our dual-flow-based dynamic SLAM in a self-supervised way.






Our insight is that for the vast majority of scenes, dynamic objects may not be dominant and their motion can be decomposed by the dominant static objects. The representation based on scene motion decomposition has natural advantages: 1) we can easily optimize static fields in a similar way as DROID-SLAM does for the poses and depths.  2) For dynamic fields, we can obtain consistent luminosity by warping the current frame to adjacent frames.  3) This holds promise for obtaining a self-supervised network model, which is also more interpretable. 
Based on these observations, we propose a dynamic update module for DeFlowSLAM, in which the dual-flow representation is cleverly embedded, shown in Fig. ~\ref{fig:pipeline}, and detailed in Sec \ref{sec:overview}. Directly warping the dynamic object field to the neighbor images may result in occlusion, shown in Fig. \ref{fig:maskagg}, we introduce a dynamic mask aggregation operator in the dynamic update module, named Mask-Agg to remove the incorrect alignment effect. Specifically, the Mask-Agg operator iteratively updates the residuals of the dynamic mask via convolutional gated recurrent unit~(ConvGRU). The final dynamic mask can be obtained by summing the aggregated dynamic mask residuals with the original mask. The obtained dynamic mask will be combined with the estimated weights and fed into the dense bundle adjustment (DBA) Layer to optimize the residuals of pose and depth, shown in Fig. \ref{fig:update module}.


We first verify our hypothesis on a highly dynamic dataset, VKITTI2~\cite{cabon2020virtual}, where ablation experiments demonstrate that a dual-flow representation can achieve better performance. In addition, DeFlowSLAM can make full use of information from all pixels to solve the camera pose, resulting in gains compared to potential approaches that filter dynamic objects by masking. Applying our knowledge to the challenging SLAM dataset of TartanAir, the trained model exhibits good generalization and achieves comparable accuracy to DROID-SLAM in both static and slightly dynamic scenes. While in highly dynamic scenes, our method significantly outperforms DROID-SLAM, and even the error in pose estimation is sharply reduced to half of DROID-SLAM. Then, we simply extended it to stereo and RGB-D datasets, showing the high scalability of the system. In AR applications, DeFlowSLAM shows relatively accurate pose estimation results, as shown in Fig. \ref{fig:AR demo}.
 In summary, our contributions are three-fold:
\begin{itemize}
    \item We propose a novel dual-flow scene motion representation that decomposes the optical flow into a static flow field and a dynamic flow field.
    \item We build a dynamic dense SLAM, DeFlowSLAM, which outperforms state-of-the-art methods in dynamic scenes.
    \item We propose a self-supervised training method instead of the strong supervision in DROID-SLAM.
   
   
   
\end{itemize}

\begin{figure*}[t]
    \centering
    \includegraphics[width=.95\linewidth]{pictures/DeFlowSLAM_framework.pdf} 
    \vspace{-1em}
    \caption{\textbf{DeFlowSLAM Overview.} DeFlowSLAM takes the image sequence as input, extracts features to construct a correlation volume, and feeds it with the initial static flow, optical flow, and dynamic mask into the dynamic update module to iteratively optimize the residual of the pose, inverse depth, static optical flow and dynamic optical flow, and finally outputs the estimation of the camera pose and 3D structure. The optimization process is performed by creating a covisibility graph and updating the existing covisibility graph.}
    \label{fig:pipeline}
    \vspace{-1em}
\end{figure*}

\vspace{-1em}

\section{Related Work}
\subsection{Scene Motion Decomposition}
Scene motion estimation has received increasing attention for 3D perception. It is the task of obtaining the 3D structure and 3D motion of dynamic scenes. Many scene motion estimation approaches have been proposed recently based on different types of input data, such as 3D point clouds~\cite{liu:2019:flownet3d, HPLFlowNet}, stereo images~\cite{huguet2007variational, vogel2013piecewise, zhang20013d}, or RGB-D images~\cite{hornacek2014sphereflow, hadfield2011kinecting, Lv18eccv, quiroga2014dense}.  Monocular scene flow estimation is a highly ill-posed problem and comparatively few approaches have been suggested so far~\cite{brickwedde2019mono}. Here, we focus on monocular videos and the scene motion estimation can be regarded as optical flow~\cite{dosovitskiy2015flownet, ilg2017flownet, sun2018pwc, ranjan2019competitive}. RAFT~\cite{teed2020raft} presents a Recurrent All-pairs Field Transforms network for optical flow estimation, which does not consider the scene motion decomposition. While EffiScene~\cite{Jiao_2021_CVPR} presents an unsupervised scene flow estimation method by jointly learning four low-level vision sub-tasks: optical flow, depth, camera pose, and motion segmentation.
Different from the forward and backward flow proposed by DF-VO~\cite{zhan2021dfvo}, we leverage the property \cite{Baur2021ICCV} of the optical flow, where it inherently separates every pixel into multiple moving agents and a large class of points following a single rigid sensor motion and try to decompose the optical flow to the static field caused by the camera pose and dynamic field caused by the objects' own motion. 
\vspace{-1em}
\subsection{Dynamic SLAM}
SLAM is a fundamental capability of many intelligent systems to provide accurate pose estimation and mapping. Humans live in a dynamic environment, intelligent systems should also have the ability to deal with dynamic environments, which are required to recognize the dynamic contents from the static environments. Traditional approaches largely filter out the interference of dynamic objects by introducing a prior~\cite{tan2013robust} or RANSAC methods~\cite{huang2020dual}. Some recent approaches try to use segmentation to filter out potential dynamic objects~\cite{vincent2020dynamic, bescos2018dynaslam, xiao2019dynamic} and then run the SLAM system, or unify object detection and SLAM into a multi-task system~\cite{huang2020clustervo, nair2020multi, Ye2022PVO} or add the object constraint to the SLAM system~\cite{yang2019cubeslam, strecke2019fusion, brasch2018semantic}. Unlike existing approaches, this paper uses a learning-based approach to identify dynamic fields at the pixel level, which does not require explicit supervision of object detection and better simulates the way humans see the world.

\vspace{-1em}

\subsection{Dense Reconstruction}
Reconstructing dense volumetric scenes is an important task in robotics with many applications such as city modeling, augmented reality navigation~\cite{Wasenmller2016AugmentedR3}, and cultural heritage preservation~\cite{10.1145/2770877}. Most of the existing systems~\cite{reijgwart2019voxgraph, oleynikova2017voxblox, millane2018c, slavcheva2017killingfusion, bozic2020deepdeform} need to use depth or lidar information for dense reconstruction, this paper does not have this limitation, and can support data formats such as monocular video, stereo, and RGB-D, and can maintain a good reconstruction effect in the dynamic environment. To improve the reconstruction performance, some approaches propose collaborative reconstruction by multiple robots~\cite{li2017corb, schmuck2019ccm, Liu2021Coxgraph}. As an example, Coxgraph~\cite{Liu2021Coxgraph} extends the single-robot reconstruction system, Voxgraph~\cite{reijgwart2019voxgraph}, to a multi-robot dense reconstruction system, and additionally adds more constraints to ensure low-error reconstruction. Our system is able to maintain robust dense reconstruction with less constraint. Some methods focus on dense depth reconstruction~\cite{wimbauer2020monorec, ranftl2016dense, russell2014video}, while our DeFlowSLAM focuses on pose estimation although it can produce the intermediate depth and flow.


\vspace{-1em}
\section{Methodology}
\label{sec:overview}
Fig. \ref{fig:pipeline} shows an overview of our novel dual-flow-based SLAM method, DeFlowSLAM, which takes a sequence of images as input and outputs the estimation of the camera pose and the 3D map of the environments. DeFlowSLAM has an end-to-end differentiable architecture that combines the strengths of both classical approaches and deep networks. It can robustly cope with challenging scenarios such as dynamic scenes, thanks to our proposed dual-flow representation, iterative dynamic update module, and factor graph optimization based on co-visibility between frames. Specifically, unlike DROID-SLAM~\cite{teed2021droid}, which iteratively updates camera pose and depth, we additionally update a dynamic mask and a dynamic flow. And each update of camera poses in DeFlowSLAM is performed by the optimization of the estimated static flow field rather than the optical flow in DROID-SLAM. Next, we will first review DROID-SLAM for understanding and then elaborate on the details of our approach.

\vspace{-1em}

\subsection{Preliminaries: DROID-SLAM}
DROID-SLAM~\cite{teed2021droid} operates on a sequence of images $\{\mathbf{I}_t\}_{t=0}^N$. For each image $t$, it maintains two state variables: camera pose $\mathbf{G}_t \in SE(3)$ and inverse depth $\mathbf{d}_t \in \mathbb{R}_{+}^{H\times W}$, which get iteratively updated during inference as new frames are processed. DROID-SLAM~\cite{teed2021droid} adopts a frame graph $\mathcal{G} = (\mathcal{V}, \mathcal{E})$ to represent co-visibility between frames. The nodes are input images and the edge $(i, j) \in \mathcal{E}$ implies that the images ${I}_i$ and ${I}_j$ have overlapped views. 

\noindent\subsubsection{Feature Extraction and Correlation}
Following RAFT \cite{teed2020raft}, the input images are first fed to the feature extraction module, and the relationship between the two images will be computed.

\noindent\textbf{Feature Extraction.}
First, each of the input images is processed by an image encoder, which consists of six residual blocks and three downsampling layers. The image encoder produces a dense image feature map, which is 1/8 of the original resolution, and feature maps from pairs of input images are used for later correlation volume construction.

\noindent\textbf{Correlation Pyramid.}
Given pairs of image features for each edge $(i, j) \in \mathcal{E}$ in the frame graph $\mathcal{G}$, DROID-SLAM\cite{teed2021droid} generates the correlation volume by taking the dot product between all-pairs of image feature vectors in $\mathbf{f}_\theta(I_i)_{u_i v_i}$ and $\mathbf{f}_\theta(I_j)_{u_j v_j}$, which has 4 dimensions.
It then computes the correlation volume $\mathbf{C}^{ij} \in \mathbb{R}^{H\times W\times H\times W}$ with each element computed as
\begin{equation}
    C_{u_i v_i u_j v_j}^{ij} = \langle \mathbf{f}_\theta(I_i)_{u_i v_i},\  \mathbf{f}_\theta(I_j)_{u_j v_j} \rangle.
\end{equation}
where $u_i,v_i,u_j,v_j$ are the pixel coordinates for image $I_i, I_j$ respectively and $< . >$ stands for the dot product.
The last two dimensions of the correlation matrix are fed to the average pooling layers with four different kernel sizes (1,2,4,8), forming a 4-level correlation pyramid.

\noindent\textbf{Correlation Lookup.}
DROID-SLAM \cite{teed2021droid} defines a correlation lookup function that uses a coordinate grid with radius $r$ to index the correlation volume $L_r:
\mathbb{R}^{H\times W\times H\times W} \times \mathbb{R}^{H\times W \times 2} \mapsto \mathbb{R}^{H\times W \times (r+1)^2}$.
This function takes an $H\times W$ grid of coordinates as input, which is the dimension of the optical flow field.
Then it retrieves the relative values from the correlation volume by using a bi-linear interpolation. The correlation lookup function is applied to every correlation volume matrices in the pyramid. The final output feature is combined by concatenating the lookup results at each level.

\noindent\subsubsection{Update Module}
DROID-SLAM~\cite{teed2021droid} proposes an update operator, a $3\times3$ convolutional GRU which updates the hidden state$\mathbf{h}$, and additionally produces a pose update, $\Delta \boldsymbol \xi^{(k)}$, and depth update, $\Delta \mathbf{d}^{(k)}$. In the update module, the pose and depth updates are applied to the current depth and pose estimates through retraction on the SE3 manifold and vector addition respectively
\begin{equation}
    \mathbf{G}^{(k+1)} = \Exp(\Delta \boldsymbol \xi^{(k)}) \circ \mathbf{G}^{(k)}, \qquad
    \mathbf{d}^{(k+1)} = \Delta \mathbf{d}^{(k)} + \mathbf{d}^{(k)}.
\end{equation}
The update operator iteratively produces a sequence of poses and depths, with the expectation of converging to a fixed point $\{\mathbf{G}^{(k)}\} \rightarrow \mathbf{G}^*$, $\{\mathbf{d}^{(k)}\} \rightarrow \mathbf{d}^*$, reflecting the true reconstruction.

\begin{figure*}[t]
    \centering
    \includegraphics[width=0.95\linewidth]{pictures/dynamic_update_module.pdf}
   
    \caption{\textbf{Dynamic Update Module.} The correlation feature of the static optical flow is looked up by correlation volumes. The obtained features will be fed into two convolutional layers together with the optical flow and dynamic mask, resulting in intermediate features. These features will be fed to ConvGRU, followed by two convolution layers with dynamic mask residual and confidence. The iterative dynamic mask residual plus the original mask to obtain the new dynamic mask termed Mask-Agg Operator. In addition, static flow revision and dynamic flow are obtained from the static and dynamic convolution layers. The static revision flow plus the original static flow is fed into the DBA layer that combines the dynamic mask and confidence to optimize the depth and pose. Finally, the new static and dynamic flows from the dynamic convolutional layer are summed to the optical flow.}
    \label{fig:update module}
    \vspace{-1em}
\end{figure*}

\vspace{-1em}
\subsection{Dual-Flow Representation}
The core concept of the proposed dynamic SLAM network is the dual-flow representation and the self-supervised training scheme. In contrast to DROID-SLAM \cite{teed2021droid}, which uses optical flow as an intermediate motion representation, we propose a novel scene motion representation by decomposing optical flow into a static flow caused by camera motion and a dynamic flow caused by the motion of dynamic objects themselves, shown in Fig. \ref{fig: dual-flow representation}. Such a representation can distinguish between static and dynamic object motions, thus having better interpretability and making the network traceable during the training process. We can directly use the dense static flow to estimate camera motion without masking out the pixels belonging to dynamic objects. 
The optical flow $\mathbf{F_{o}}_t \in \mathbb{R}^{H\times W\times 2}$, static flow $\mathbf{F_{s}}_t \in \mathbb{R}^{H\times W\times 2}$ and dynamic flow $\mathbf{F_{d}}_t \in \mathbb{R}^{H\times W\times 2}$ are a set of vectors, where the static flow plus the dynamic flow equals the optical flow:
\begin{equation}
 \mathbf{F_{o}}_t = \mathbf{F_{s}}_t + \mathbf{F_{d}}_t.
 \label{equ: flow equ}
\end{equation}

Our network operates on a sequence of images $\{\mathbf{I}_t\}_{t=0}^N$. As new frames being processed, the set of camera poses $\{\mathbf{G}_t\}_{t=0}^N \in SE(3)$, inverse depths $\{\mathbf{d}_t\}_{t=0}^N \in \mathbb{R}_{+}^{H\times W}$, static flows $\{\mathbf{F_{s}}_t\}_{t=0}^N \in \mathbb{R}^{H\times W\times 2}$ , dynamic flows $\{\mathbf{F_{d}}_t\}_{t=0}^N\in \mathbb{R}^{H\times W\times 2}$ and the binary dynamic masks $\{\mathbf{M_{d}}_t\}_{t=0}^N \in \mathbb{R}_{+}^{H\times W \times 2 }$ are update iteratively.
In $\mathbf{M_{d}}_t$, we let 0 indicate dynamic while 1 indicate static. 
As a reminder in this paper, when we refer to depth, please note that we are using the inverse depth parameterization. 

Similar to DROID-SLAM~\cite{teed2021droid}, we also adopt a frame graph $\mathcal{G} = (\mathcal{V}, \mathcal{E})$ to represent co-visibility between frames. 
For example, as shown in the co-visibity graph optimization in Fig. \ref{fig:pipeline}, the white nodes indicate each image, and the edges indicate two images with a co-visibility relationship, where the red points indicate image ${I}_i$ and image ${I}_j$, and the blue edges indicate the co-visibility relationship in the frame graph. 
The frame graphs are built dynamically during the training and inference. After each pose or depth update with the revision static flow field, 
we can update the frame graph with new co-visibilities. 
When the camera returns to a previously visited place, a long-distance connection is added to the graph, and loop closure is performed. 

\vspace{-1em}
\subsection{Dynamic Update Module}
Fig. \ref{fig:update module} shows the dynamic update module of our DeFlowSLAM.
The module is a $3\times3$ ConvGRU with a hidden state $\mathbf{h}$. Different from the update module in DROID-SLAM \cite{teed2021droid}, which works directly on the revision optical flow, our dynamic update module works on the decomposed static and dynamic flow fields respectively. We update the revision static flow field in a similar way to DROID-SLAM \cite{teed2021droid}, while for the dynamic flow field, we add it to the static flow field to obtain the optical flow, which is fed into the flow encoder as a new optimization term in the next iteration.
Each application of the operator updates the hidden state, and additionally produces a pose increment, depth increment, dynamic mask increment, and dynamic flow. The pose increments are applied to the current pose through retraction on the SE3 manifold, 
\begin{equation}
    \mathbf{G}^{(k+1)} = \Exp(\Delta \boldsymbol \xi^{(k)}) \circ \mathbf{G}^{(k)}.
\end{equation}
While the depth and the dynamic mask increments are added to the current depth and dynamic mask respectively
\begin{equation}
    \Xi^{(k+1)} = \Delta \Xi^{(k)} + \Xi^{(k)}, \, \Xi \in \{\mathbf{d}, \mathbf{M_d}\}.
\end{equation}
where $\mathbf{F_d}^{(k+1)}$ is directly given a new value in every iteration. With the updated static flow $\mathbf{F_{s}}^{(k+1)}$ transformed from $\mathbf{G}^{(k+1)}$ and $\mathbf{d}^{(k+1)}$, the final optical flow can be computed using Equation \ref{equ: flow equ}.

Iterative operations of the dynamic update module produce a sequence of poses, depths, dynamic masks, dynamic flows, and complete optical flows with the expectation of converting to an optimal position, such as
$\{\mathbf{G}^{(k)}\} \rightarrow \mathbf{G}^*$, $\{\mathbf{d}^{(k)}\} \rightarrow \mathbf{d}^*$, $\{\mathbf{M_{d}}^{(k)}\} \rightarrow \mathbf{M_{d}}^*$, $\{\mathbf{F_{d}}^{(k)}\} \rightarrow \mathbf{F_{d}}^*$, $\{\mathbf{F_{o}}^{(k)}\} \rightarrow \mathbf{F_{o}}^*$,
representing the true reconstruction.



\subsubsection{Correspondence}
At the beginning of each iteration, we use the current pose and depth estimates to search for correspondence. Given a grid of pixel coordinates, $\mathbf{p}_i \in \mathbb{R}^{H \times W \times 2}$ in frame $i$, we compute the dense correspondence field $\mathbf{p}_{ij}$ 
\begin{equation}
    \mathbf{p}_{ij} = \Pi_c(\mathbf{G}_{ij} \circ \Pi_c^{-1}(\mathbf{p}_i, \mathbf{d}_i)), 
   
    \  \mathbf{G}_{ij} = \mathbf{G}_j \circ \mathbf{G}_i^{-1}.
\end{equation}
for each edge $(i, j) \in \mathcal{E}$ in the frame graph. Here $\Pi_c$ is the camera model that maps a set of 3D points onto the image, and $\Pi_c^{-1}$ is the inverse projection function that maps the inverse depth map $\mathbf{d}$ and the coordinate grid $\mathbf{p}_i$ to the 3D point cloud. $\mathbf{p}_{ij}$ represents the coordinates of pixel $\mathbf{p}_i$ mapped to $j$ frames using the current pose and depth estimates.

\subsubsection{Inputs}
We use the dense correspondence field to index the correlation volumes. For each edge of $(i,j) \in \mathcal{E}$, we use $\mathbf{p}_{ij}$ to lookup from the correlation volume $\mathbf{C}^{ij}$ to retrieve the correlation features. In addition, we use the dense correspondence field to derive the static optical flow caused by the camera motion, i.e., the difference between $\mathbf{p}_{ij} - \mathbf{p}_j$, which is $\mathbf{F_{s}}_{ij}$. For the initial dynamic mask $\mathbf{M_d}_{ij}$ and the initial dynamic flow $\mathbf{F_d}_{ij}$, we simply initialize them to zeros. 

\subsubsection{Dynamic Update}
We extract the global context by averaging the hidden states over the spatial dimensions of the image and use this feature vector as an additional input to the ConvGRU.
ConvGRU generates an updated hidden state $\mathbf{h}^{(k+1)}$. Instead of directly predicting updates in depth or pose, we predict updates in the space of dense flow fields. We map the hidden states by two additional convolution layers to produce four outputs: (1) a revised static flow field $\mathbf{r_s}_{ij} \in \mathbb{R}^{H \times W \times 2}$, (2) an updated dynamic flow field $\mathbf{F_d}_{ij} \in \mathbb{R}^{H \times W \times 2}$,  (3) a correlation confidence map $\mathbf{w}_{ij} \in \mathbb{R}^{H \times W \times 2}$, (4) an updated dynamic mask increment field $\Delta \mathbf{M_d}_{ij} \in \mathbb{R}^{H \times W \times 2}$. The revision $\mathbf{r}_{ij}$ is a correction term for the network prediction, which is used to correct errors in the dense correspondence fields. We express the corrected correspondence as $\mathbf{p_s}_{ij}^* = \mathbf{r_s}_{ij} + \mathbf{p_s}_{ij}$.

We then pool the hidden state's overall features that share the same source view $i$  and predict the pixel-level damping factor $\lambda$. We use the softplus operator to ensure that the damping term is positive. In addition, we use pooled features to predict an 8x8 mask that can be used to upsample the inverse depth estimation.



\subsubsection{Dense Bundle Adjustment Layer (DBA)}
The dense bundle adjustment layer (DBA) maps a set of stream revisions to a set of pose and pixel depth updates. We define the cost function on the whole frame diagram
\begin{equation}
    \mathbf E(\mathbf{G}', \mathbf{d}') = \sum_{(i,j) \in \mathcal{E}} \norm{\mathbf{p_s}_{ij}^* - \Pi_c(\mathbf{G}'_{ij} \circ \Pi_c^{-1}(\mathbf{p}_i, \mathbf{d}'_i)) }_{\Sigma_{ij}}^2
    \label{eqn:objective},
\end{equation}
\begin{equation}
\Sigma_{ij} = \diag \mathbf{w_d}_{ij},
\end{equation}
\begin{equation}
    \mathbf{w_d}_{ij}=\operatorname{sigmoid}(\mathbf{w}_{ij}+(1-\mathbf{M_d}_{ij})\cdot\eta).
\end{equation}
where $\eta$ is set as 10. And $\norm{\cdot}_{\Sigma}$ is the Mahalanobis distance, which weights the error term according to the combined confidence $\mathbf{w_d}_{ij}$. The combined confidence makes sure that the optimization considers all points.
The addition of these points can effectively improve the accuracy of BA operation in dynamic scenes. Equation \ref{eqn:objective} shows that we want an updated pose $\mathbf{G}'$ and depth $\mathbf{d}'$ such that the reprojected points match the modified correspondence $\mathbf{p_s}_{ij}^*$ predicted by the update operator.
\begin{figure}[t]
  \centering
  \includegraphics[width=\linewidth]{pictures/maskagg.png}
  \caption{\textbf{Mask-Agg Illustration}. If there are dynamic objects in the scene, directly matching pixels using static flow will cause erroneous results, making photometric loss invalid. Using the aggregated mask predicted by the network, we can filter out these invalid pixels (e.g. the pink mask) in geometry photometric loss, which is shown in the right bottom picture.}
  \label{fig:maskagg}
  \vspace{-1em}
\end{figure}
We use a local parameterization to linearize Equation \ref{eqn:objective} and use the Gauss-Newton algorithm to solve for the update ($\Delta \boldsymbol\xi$, $\Delta \mathbf d$). Since each term in Equation \ref{eqn:objective} contains only a single depth variable, the Hessian matrix has a block diagonal structure. Separating the camera pose and depth variables allows the system to be solved efficiently using Schur's complement and adding the pixel damping factor $\lambda$ to the depth block.
The DBA layer is implemented as part of the computational graph and backpropagation is performed through this layer during training.










\vspace{-1em}
\subsection{Loss Function}
In this section, we elaborate on the self-supervised training scheme for DeFlowSLAM in detail.


\subsubsection{Geometry Photometric Loss}
To supervise the geometric predictions using images, we introduce a photometric reprojection loss to guide the network's optimization. We refer to \cite{2020Digging} to build our photometric loss function.
Given the predicted pose $G_{ij}$ and the predicted depth $\mathbf{\hat{d}}_i$, we can get the corresponding coordinates of pixels in image $I_i$ in image $I_j$. We then use bi-linear sampling to sample the image $I_j$, getting a new-sampled image $I_{j \rightarrow i}$.
\begin{equation}
    I_{j \rightarrow i}=I_{j}\left\langle\Pi_c(\mathbf{G}_{ij} \circ \Pi_c^{-1}(\mathbf{p}_i, \mathbf{\hat{{d}}}_i))\right\rangle.
\end{equation}
Then we can use photometric loss on source image $I_i$ and new image $I_{j \rightarrow i}$.
\begin{equation}
    \mathcal{L}_{\text {geo\_ph}}=\frac{1}{N}\sum_{ij} p e\left(I_{i}, I_{j \rightarrow i}\right).
\end{equation}
We introduce $L_1$ loss and $SSIM$\cite{2004Image} loss to form our geometry photometric loss, setting $\alpha=0.85$.
\begin{equation}
    p e\left(I_{a}, I_{b}\right)=\frac{\alpha}{2}\left(1-\operatorname{SSIM}\left(I_{a}, I_{b}\right)\right)+(1-\alpha)\left\|I_{a}-I_{b}\right\|_{1}.
\end{equation}

\subsubsection{Mask Aggregation Operator for Geometry}
In the self-supervised training policy, when using the photometric loss to supervise camera poses and depths, the direct use of the dual-flow representation may result in pixel mismatches due to the object's motion, making the geometry photometric loss less accurate. So we propose a mask aggregation operator, i.e. Mask-Agg, to filter out the wrong pixel matches to enhance our dual-flow representation, as shown in Fig. \ref{fig:maskagg}. 
The aggregated mask $\mathbf{M_{d_i}^{Agg}}$ for frame $I_i$ is
computed by gathering the dynamic masks estimated for all the frames which are connected to frame 
$I_i$ in the frame graph.
Then, the final geometry photometric loss function is
\begin{equation}
     \mathcal{L}_{\text {geo\_ph}}=\frac{1}{N^{'}}\sum_{ij} p e\left(I_{i}, I_{j \rightarrow i}\right) \cdot \mathbf{M_{d_i}^{Agg}}.
\end{equation}
where $N^{'}$ means the count of pixels whose $\mathbf{M_{d_i}^{Agg}}$ value is 1. Specifically, the Mask-Agg operator iteratively updates the residuals of the dynamic mask via ConvGRU. The final dynamic mask can be obtained by summing the aggregated dynamic mask residuals with the original old mask.
Tab. \ref{table:vkitti ablation study} shows the Mask-Agg module helps filter out the ambiguous matches in self-supervised training, obtaining better results.

\subsubsection{Optical Flow Photometric Loss}
The geometry photometric loss is used to supervise the static flow caused by camera motion. So we also introduce the optical flow photometric loss to supervise the complete scene motion, including camera motion and objects' motion. Through the update module, we can get the optical flow results $\mathbf{F_o}_{ij}$ by adding static flow and dynamic flow. Similar to the geometry photometric loss, we use $\mathbf{F_o}_{ij}$ to generate corresponding coordinates between images.
\begin{equation}
    I_{j \rightarrow i}=I_{j}\left\langle \mathbf{F_o}_{ij}+ \mathbf{p_{ij}}\right\rangle.
\end{equation}
Then we still use bi-linear sampling to sample from the source image, evaluate their photometric errors:
\begin{equation}
    \mathcal{L}_{\text {flow\_ph}}=\sum_{ij} p e\left(I_{i}, I_{j \rightarrow i}\right).
\end{equation}
where the $pe$ function here is just $L_1$ loss:
\begin{equation}
    p e\left(I_{a}, I_{b}\right)=\left\|I_{a}-I_{b}\right\|_{1}.
\end{equation}

\subsubsection{Supervised Mask Loss}
When dynamic mask labels are available, we can directly supervise our predicted masks using a simple cross-entropy classification loss:
\begin{equation}
    \mathcal{L}_{\text{gt\_mask}} = -\frac{1}{\left|\mathcal N\right|} \sum_{\mathbf{p}_{i} \in \mathcal N} \mathbf{M}_{i} \log \mathbf{\hat M}_i + \left(1-\mathbf{M}_{i}\right) \log \left(1-\mathbf{\hat M}_i\right).
\end{equation}
where $\mathbf{M}_{i}$ is the ground truth mask labels, $\mathbf{\hat M}_i$ is the predicted masks.

\subsubsection{Artificial Mask Loss}
When the dynamic mask labels are not available, we design a method to artificially build referred masks for self-supervised dynamic mask learning. Referred to SLIM~\cite{Baur2021ICCV}, we propose the artificial mask loss to achieve this goal. Using the camera pose, depth, and optical flow we have already gotten, we can first infer the target coordinate of the pixel $p$ following the equations below:
\begin{equation}
    \mathbf{p}_{\text {cam}} =\Pi_c(\mathbf{G}_{ij} \circ \Pi_c^{-1}(\mathbf{p}_i, \mathbf{\hat{{d}}}_i)), \\
    \mathbf{p}_{\text {flow}} =\mathbf{p}_i+ \mathbf{\hat F_o{}}_{ij}.
\end{equation}
where $\mathbf{p}_{\text{cam}}$ is the target coordinate calculated by projection, $\mathbf{p}_{\text{flow}}$ is the target coordinate calculated by optical flow.
Then we use the difference between these two coordinates to form our artificial mask labels
\begin{equation}
    \mathbf{M^{art}_{i}}=[\left\|\mathbf{p}_{\text {cam}}-\mathbf{p}_{\text {flow}}\right\|_{2}\le \mu],
\end{equation}
where $\mu$ is set as 0.5. This artificial mask label has the same format compared with ground truth mask labels, so our final loss function should be
\begin{align}
\mathcal{L}_{\text{art\_mask}} = &-\frac{1}{\left|\mathcal N\right|} \sum_{\mathbf{p}_{i} \in \mathcal N} \mathbf{M^{art}_{i}} \log \mathbf{\hat M}_i + \nonumber\\ 
    &\left(1-\mathbf{M^{art}_{i}}\right) \log \left(1-\mathbf{\hat M}_i\right).
\end{align}
\begin{figure}[t]
  \centering
  \includegraphics[width=\linewidth]{pictures/slam.pdf}
   
  \caption{\textbf{Qualitative Results of DeFlowSLAM}. DeFlowSLAM can generalize to new datasets. We show results from ETH3D, TUM RGB-D, ScanNet and Virtual KITTI2.}
  \label{fig:slam reconstruction}
  \vspace{-1em}
\end{figure}

\subsubsection{Final Loss Function}
We use $\mathcal{L}_{\text {geo\_ph}}$, $\mathcal{L}_{\text {flow\_ph}}$ and $\mathcal{L}_{\text{art\_mask}}$ for self-supervised training. The supervised final loss function is
\begin{equation}
    \mathcal{L}_{\text{self-sup}}=\lambda_0\mathcal{L}_{\text{geo\_ph}}+\lambda_1\mathcal{L}_{\text{flow\_ph}}+\lambda_2\mathcal{L}_{\text{art\_mask}}.
\end{equation}
where $\lambda_0=100$, $\lambda_1=5$, and $\lambda_2=0.05$. Since the network has several update iterations, we use $\gamma=0.9$ to apply the loss to the output of each iteration with exponentially increasing weights. For semi-supervised learning, we modify the $\mathcal{L}_{\text{art\_mask}}$ to the $\mathcal{L}_{\text{gt\_mask}}$.













\vspace{-1em}
\subsection{Implementation Details}
\subsubsection{Training Details}
DeFlowSLAM is implemented in PyTorch, and we use the LieTorch extension~\cite{teed2021tangent} to perform back-propagation in the tangent space of all groups elements. In the ablation study, we trained DeFlowSLAM on the VKITTI2 dataset~\cite{cabon2020virtual} with 2 RTX-3090 GPUs for 80,000 steps, which took about 2 days. While we trained our network on TartanAir dataset ~\cite{tartanair2020iros} for 250k steps with a batch size of 4 and resolution of $344\times464$. Training on 4 RTX-3090 GPUs takes 5 days. In the monocular setup, we fix the first two poses as the true pose for each training sequence as DROID-SLAM\cite{teed2021droid}, for the network can only recover the trajectory of the camera to the similarity transformation.
In addition, there are still normative degrees of uncertainty during training, which may have an impact on the stability in terms of regulation and gradients.
During training, we set each training example consisting of a 6-frame video sequence. We keep the same constructing strategy in \cite{teed2021droid}, making sure that the average distance between adjacent sequence frames is between 8 pixels and 96 pixels. 





\subsubsection{SLAM System Details}
In the inference process, we integrate our networks into a full SLAM system. The SLAM system takes the video stream as input and performs simultaneous localization and mapping. Our system contains a frontend thread and a backend thread, both running at the same time. The mission of the frontend thread is to receive new frames, select keyframes, extract features, and perform local bundle adjustment. While the backend thread performs global bundle adjustment for the entire history of keyframes.
\begin{table}[t]
    \caption{\textbf{Ablation Study of DeFlowSLAM Trained and Tested on Vitual KITTI2 (VK) Dataset.} SS means self-supervised, SM means semi-supervised, SF means single flow, DF means dual flow, and MA means Mask-Agg. 
   
    }
    \resizebox{\linewidth}{!}{%
        \begin{tabular}{l|c c c c c}
            Monocular & VK01 & VK02 & VK06 & VK18 & VK20 \\
            \toprule
            DROID-SLAM*~\cite{teed2021droid} & \underline{1.091} & \textbf{0.025} & 0.113 & 1.156 & 8.285 \\
            \hline
            Ours (SM, DF) & \textbf{0.761} & \underline{0.069} & \underline{0.11} & \underline{0.737} & \textbf{2.546}\\
            \hline
           
            Ours (SS, SF) & 4.278 & 0.187 & 0.142 & 1.248 & 10.487 \\
            Ours (SS, DF) & 1.099 & 0.094 & 1.984 & 0.983 & 6.918 \\
            \textbf{Ours (SS, DF, MA)} & 1.341 & 0.089 & \textbf{0.092} & \textbf{0.376} & \underline{5.975} \\
           
            \bottomrule
        \end{tabular}
        \label{table:vkitti ablation study}
    }
    \vspace{-1em}
\end{table}

\begin{table}[t]
    \caption{\textbf{Ablation Study of Dynamic Threshold $\mu$ of DeFlowSLAM Trained and Tested on Vitual KITTI2 (VK) Dataset in Self-Supervised Setting.} $\mu=0.5$ achieves the best results.
   
    }
    \resizebox{\linewidth}{!}{%
        \begin{tabular}{l|c c c c c}
            $\mu$ & VK01 & VK02 & VK06 & VK18 & VK20 \\
            \toprule
            0.3 & 0.763 & 0.117 & 0.126 & 36.10 & 114.5 \\
            0.5 & 1.341 & \textbf{0.089} & \textbf{0.092} & \textbf{0.376} & \textbf{5.975} \\
            0.7 & \textbf{0.738} & 0.274 & 0.324 & 27.41 & 104.1 \\
           
            \bottomrule
        \end{tabular}
        \label{table:dynamic threshold}
    }
    \vspace{-1em}
\end{table}



\begin{figure}[t]
   \centering
    \includegraphics[width=\linewidth]{pictures/weight.pdf} 
    \vspace{-2em}
    \caption{\textbf{DeFlowSLAM Confidence.} We visualize the confidence of our DeFlowSLAM vs. DROID-SLAM. Our method can exploit the dynamic pixels to solve the camera pose. The darker color means the greater weight. DeFlowSLAM also takes  advantage of more information about the parked vehicles on the roadside.}
    \label{fig:weight}
    \vspace{-1em}
\end{figure}
\begin{table}[t]
    \centering
    \caption{\textbf{Dynamic SLAM Results on KITTI (K) \& Virtual KITTI2 (VK) Datasets with Metric: ATE[m] Trained on TartanAir Dataset.} We achieve the best results.}
   
    \resizebox{\linewidth}{!}
    {%
    \begin{tabular}{l c c c c c c c}
        \toprule
        Method & K09 & K10 & VK01 & VK02 & VK06 & VK18 & VK20 \\
        \midrule
        DSO~\cite{engel2017direct} & \underline{28.1} & 24.0 & - & - & - & - & - \\
        DynaSLAM*~\cite{bescos2018dynaslam} & 41.91 & \underline{7.519} & 27.830 & X & X & X & \underline{2.807} \\
        DROID-SLAM*~\cite{teed2021droid} & 47.1 & 11.0 & \underline{2.259} & \underline{0.049} & \underline{0.136} & \underline{1.170} & 6.998 \\
        \midrule
        Ours & \textbf{27.8} & \textbf{4.2} & \textbf{0.591} & \textbf{0.021} & \textbf{0.13} & \textbf{0.400} & \textbf{1.039} \\
        \bottomrule
    \end{tabular}
    }
    \label{tab:Kitti}
    \vspace{-1em}
\end{table}

\begin{figure}[t]
\centerline{
\begin{tabular}{cc}
\multicolumn{2}{c}{\includegraphics[width=0.9\linewidth]{pictures/vkitti01.jpg}} \\
\includegraphics[width=0.47\linewidth]{pictures/vkitti02.png} &
\includegraphics[width=0.47\linewidth]{pictures/vkitti20.png} \\
\end{tabular}
}
    \caption{\textbf{Comparison Trajectory Results of Our Method with DROID-SLAM on VKITTI2 Sequences 01 (Top), 02 (Bottom-Left), and 20 (Bottom-right).} In these dynamic sequences, our method performs better than DROID-SLAM, having better trajectory estimation results.}
  \label{fig: trajetory of vkitti2}
  \vspace{-1em}
\end{figure}

\begin{figure}[t]
    \centering
    \includegraphics[width=\linewidth]{pictures/kitti0910.png}
    \label{fig:sub2}
     \vspace{-2em}
    \caption{\textbf{Trajectory Comparison between Our Method and DROID-SLAM.} In KITTI sequences 09 (Left) and 10 (Right), our trajectories are closer to the ground truth.}
    \vspace{-1em}
\end{figure}

\noindent\textbf{Initialization.}
Following DROID-SLAM~\cite{teed2021droid}, our system keeps collecting frames until having 12 frames in total. The system saves the old frame only if its estimated static flow is greater than 16 pixels. Once the system has gotten 12 frames, it initializes the factor graph by creating edges between keyframes within 3 timesteps. Then the system will run the dynamic update operator 10 times using edges in this graph.

\noindent\textbf{Frontend.}
The frontend directly processes the incoming video stream, which maintains a set of keyframes and a frame graph storing edges between co-visible keyframes, where the keyframe poses and depths are being actively optimized. When a new frame comes, we first extract its feature map, then add it to the frame graph, building edges with its 3 nearest neighbors. The distance used here is measured by averaging static flow. The new frame's pose is initialized by a linear motion model, and several update iterations are performed to optimize the keyframe poses and depths. We fix the first two poses to eliminate the scale uncertainty while treating all depths as free variables. After tracking the new frame, we use the static flow-based distance to remove redundant frames and remove the oldest keyframe if the buffer is overloaded.

\noindent\textbf{Backend.}
The backend performs global bundle adjustment over the entire history of keyframes~\cite{teed2021droid}. 
During each iteration, we first choose the edge whose containing frames are temporally adjacent. We then choose the new edge according to the distance map, which is made by exploiting the averaging static flow between all keyframe pairs. Similar to DROID-SLAM~\cite{teed2021droid}, for each chosen edge, we suppress the adjacent edges within a distance of 2, where the distance here is defined as the Chebyshev distance between index pairs $||(i,j) - (k,l)||_\infty = \max(|i-k|, |j-l|)$. We then apply the dynamic update operator to the new complete frame graph. 

\noindent\textbf{Stereo and RGB-D Inputs.}
\label{sec:stereo and rgbd}
Our system can also use stereo and RGB-D inputs. For RGB-D, we still treat depth as a variable to optimize, since the sensor depth can be noisy. We simply add a term to the optimization process which is shown in Eq.~(\ref{eqn:objective}). It will then penalize the squared distance between the predicted depths and the sensor depths. For stereo, we use the same system, except for doubling the frames and fixing the relative pose between the left and right frames in the DBA layer.
We can exploit the stereo evidence by adding crossed camera edges in the optimization graph~\cite{teed2021droid}.







\vspace{-1em}

\section{Experiments}

We first validate the effectiveness of our method in the highly dynamic scenarios of Virtual KITTI2~\cite{cabon2020virtual} in the ablation study. Further, we train DeFlowSLAM from scratch with the same strategy on a larger dataset, TartanAir~\cite{tartanair2020iros}, and test the generalization of our method on different dynamic datasets, such as Virtual KITTI2~\cite{cabon2020virtual}, KITTI~\cite{geiger2013vision}, and dynamic sequences of TUM-RGBD\cite{schubert2018tum}. We also test monocular or stereo datasets, such as static scenes of TUM-RGBD~\cite{schubert2018tum} and EuRoc~\cite{burri2016euroc}.
Following previous works~\cite{teed2021droid}, we use absolute trajectory error (ATE)~\cite{tumrgbd} to evaluate the accuracy of the estimated camera trajectories. In particular, we compare DeFlowSLAM to DynaSLAM~\cite{bescos2018dynaslam}, etc in dynamic scenes, demonstrating the effectiveness of our method. We also compare DeFlowSLAM to DROID-SLAM in AR applications, the results further prove the robustness of our pose estimation.



\begin{table*}[t]
	\centering
	\caption{\textbf{Dynamic SLAM Results on TUM Dynamic Sequences with Metric: ATE[m].} The best results are shown in bold. DeFlowSLAM achieves competitive and even best performance. Note that DVO SLAM, ORB-SLAM2 and PointCorr use the RGB-D dataset, while our method and DROID-SLAM only use the monocular RGB dataset.
	}
	\label{tab:tum_dy}
	\resizebox{14cm}{!}
	{
	\begin{tabular}{cc|ccc|cc}
		\toprule
		\multicolumn{2}{c|}{\multirow{1}{*}{Sequences}}         & \multicolumn{5}{c}{ Trans. RMSE of  trajectory alignment [m]}    \\
		\multicolumn{2}{c|}{}       &  DVO SLAM~\cite{kerl2013robust} 
    &   
     ORB-SLAM2~\cite{orbslam2}  
		&  
     PointCorr~\cite{dai2020rgb} & DROID-SLAM~\cite{teed2021droid} & Ours
     \\
		\midrule
		\multirow{5}{*}{slightly dynamic}  & fr2/desk-person   &0.104 &\textbf{0.006} &0.008  & 0.017          & 0.013  \\
		& fr3/sitting-static     &0.012      & 0.008  & 0.010  & \textbf{0.007} &\textbf{0.007}   \\
		& fr3/sitting-xyz        &0.242      & 0.010  & \textbf{0.009}   &0.016  &0.015  \\
		& fr3/sitting-rpy       &0.176    & 0.025    &\textbf{0.023}   & 0.029  & 0.027   \\
		& fr3/sitting-halfsphere  &0.220   & 0.025  & 0.024  & \textbf{0.022}  & 0.025  \\
		\midrule
		\multirow{4}{*}{highly dynamic} & fr3/walking-static   &0.752 & 0.408 & 0.011 & 0.016 & \textbf{0.007}  \\
		& fr3/walking-xyz      &1.383    &0.722  & 0.087  & 0.019 & \textbf{0.018}  \\ 
		& fr3/walking-rpy     &1.292     &0.805 & 0.161  &0.059 &  \textbf{0.057}    \\
		& fr3/walking-halfsphere  &1.014   & 0.723   & \textbf{0.035} & 0.312  &   0.42   \\
		\bottomrule
	\end{tabular}
	}
	 \vspace{-1em}
\end{table*}


\begin{table*}[h]
    \centering
    \caption{
        \textbf{Monocular SLAM Results on TartanAir Monocular Benchmark with Metric: ATE[m]}. Bold stands for best results and underlined for the second best. * means the results are generated by running the official pretrained model in our environment to ensure evaluation consistency. X means the system has failed here. - means lack of result. We achieve the best results.
    }
    \resizebox{.8\linewidth}{!}{%
        \begin{tabular}{l|cccccccc | c}
            Monocular & MH000 & MH001 & MH002 & MH003 & MH004 & MH005 & MH006 & MH007 & Avg \\
            \toprule
            ORB-SLAM~\cite{mur2015orb} & 1.30 & \textbf{0.04} & 2.37 & 2.45 & X & X & 21.47 & 2.73 & - \\
            DeepV2D~\cite{deepv2d} & 6.15 & 2.12 & 4.54 & 3.89 & \underline{2.71} & 11.55 & 5.53 & 3.76 & 5.03 \\
            TartanVO~\cite{tartanvo2020corl} & 4.88	& 0.26 & 2.00 & 0.94 & \textbf{1.07} & 3.19 & 1.00 & 2.04 & 1.92 \\
            DROID-SLAM*~\cite{teed2021droid} & \textbf{0.04} & 0.69 & \underline{0.03} & \underline{0.02} & 3.73 & \underline{0.62} & \underline{0.38} & \textbf{0.07} & \underline{0.70} \\
            Ours & \underline{0.63} & \underline{0.06} & \textbf{0.02} & \textbf{0.01} & 2.80 & \textbf{0.20} & \textbf{0.31} & \underline{0.45} & \textbf{0.56} \\
            \midrule
        \end{tabular}
        \label{table:TartanAir mono}
    }
    \vspace{-1em}
\end{table*}


\begin{table*} [h]
    \caption{
        \textbf{Monocular SLAM Results on EuRoC Dataset with Metric: ATE[m]}. $^\dagger$ denotes visual odometry methods. X means the system has failed here. - means lack of result. We achieve comparable results on par with DROID-SLAM.
    }
    \centering
    \resizebox{.9\linewidth}{!}{%
        \begin{tabular}{cl| ccccc | ccc | ccc | c}
            & & MH01 & MH02 & MH03 & MH04 & MH05 & V101 & V102 & V103 & V201 & V202 & V203 & Avg \\
            \toprule
            \multirow{4}{*}{\rotatebox[origin=c]{90}{Deep/Hyb.}}
            & DeepFactors~\cite{deepfactors} & 1.587 & 1.479 & 3.139 & 5.331 & 4.002 & 1.520 & 0.679 & 0.900 & 0.876 & 1.905 & 1.021 & 2.040 \\
            & DeepV2D~\cite{deepv2d}$^\dagger$ & 0.739 & 1.144 & 0.752 & 1.492 & 1.567 & 0.981 & 0.801 & 1.570 & 0.290 & 2.202 & 2.743 & 1.298 \\
            & DeepV2D (Tartan Air)$^\dagger$ & 1.614 & 1.492 & 1.635 & 1.775 & 1.013 & 0.717 & 0.695 & 1.483 & 0.839 & 1.052 & 0.591 & 1.173 \\
            & TartanVO~\cite{tartanvo2020corl}$^\dagger$ & 0.639 & 0.325 & 0.550 & 1.153 & 1.021 & 0.447 & 0.389 & 0.622 & 0.433 & 0.749 & 1.152 & 0.680 \\
            & D3VO + DSO~\cite{d3vo}$^\dagger$ & - & - & 0.08 \ \ & - & 0.09 \ \ & - & - & 0.11 \ \ & - & 0.05 \ \ & \underline{0.19} \ \ & - \\
            \midrule
            \multirow{5}{*}{\rotatebox[origin=c]{90}{Classical}}
            & ORB-SLAM~\cite{mur2015orb} & 0.071 & 0.067 & 0.071 & 0.082 & 0.060 & \textbf{0.015} & 0.020 & X & 0.021 & \underline{0.018} & X & - \\
            & DSO~\cite{dso}$^\dagger$ & 0.046 & 0.046 & 0.172 & 3.810 & 0.110 & 0.089 & 0.107 & 0.903 & 0.044 & 0.132 & 1.152 & 0.601 \\
            & SVO~\cite{svo}$^\dagger$ & 0.100 & 0.120 & 0.410 & 0.430 & 0.300 & 0.070 & 0.210 & X & 0.110 & 0.110 & 1.080 & - \\
            & DSM~\cite{dsm} & 0.039 & 0.036 & 0.055 & \underline{0.057} & 0.067 & 0.095 & 0.059 & 0.076 & 0.056 & 0.057 & 0.784 & \underline{0.126} \\
            & ORB-SLAM3~\cite{orbslam3} & \underline{0.016} & \underline{0.027} & 0.028 & 0.138 & 0.072 & 0.033 & 0.015 & \underline{0.033} & 0.023 & 0.029 & X & - \\
            \midrule
            & DROID-SLAM*~\cite{teed2021droid} & \textbf{0.013} & \textbf{0.014} & \underline{0.022} & \textbf{0.043} & \textbf{0.043} & 0.037 & \underline{0.012} & \textbf{0.020} & \underline{0.017} & \textbf{0.013} & \textbf{0.014} & \textbf{0.022} \\
            & Ours & 0.018 & 0.037 & \textbf{0.020} & 0.060 & \underline{0.048} & \underline{0.031} & \textbf{0.008} & 0.083 & \textbf{0.016} & 0.035 & 1.137 & 0.136 \\
            \midrule
        \end{tabular}
    }
    \label{table:EurocMono}
     \vspace{-1em}
\end{table*}

\begin{table*}[h]
    \caption{
        \textbf{Monocular SLAM Results on TUM-RGBD Dataset with Metric: ATE[m]}. X means the system has failed here. - means lack of result. We achieve comparable results on par with DROID-SLAM.
    }
    \centering
    \resizebox{.75\linewidth}{!}{%
        \begin{tabular}{l|ccccccccc | c}
            & 360 & desk & desk2 & floor & plant & room & rpy & teddy & xyz & avg \\
            \toprule
            ORB-SLAM2~\cite{orbslam2} & X & 0.071 & X & \underline{0.023} & X & X & X & X & 0.010 & - \\
            ORB-SLAM3~\cite{orbslam3} & X & \underline{0.017} & 0.210 & X & \underline{0.034} & X & X & X & \textbf{0.009} & - \\
            \midrule
            DeepTAM~\cite{deeptam} & \textbf{0.111} & 0.053 & 0.103 & 0.206 & 0.064 & \underline{0.239} & 0.093 & 0.144 & 0.036 & 0.116 \\
            TartanVO~\cite{tartanvo2020corl} & 0.178 & 0.125 & 0.122 & 0.349 & 0.297 & 0.333 & 0.049 & 0.339 & 0.062 & 0.206\\
            DeepV2D~\cite{deepv2d} & 0.243 & 0.166 & 0.379 & 1.653 & 0.203 & 0.246 & 0.105 & 0.316 & 0.064 & 0.375 \\
            DeepV2D (TartanAir) & 0.182 & 0.652 & 0.633 & 0.579 & 0.582 & 0.776 & 0.053 & 0.602 & 0.150 & 0.468 \\
            DeepFactors~\cite{deepfactors} & 0.159 & 0.170 & 0.253 & 0.169 & 0.305 & 0.364 & 0.043 & 0.601 & 0.035 & 0.233\\
            \midrule
            DROID-SLAM*~\cite{teed2021droid} & \textbf{0.111} & 0.018 & \underline{0.042} & \textbf{0.021} & 
            \textbf{0.016} & \textbf{0.049} & \underline{0.026} & \underline{0.048} & 0.012 & \textbf{0.038} \\
            Ours  & 0.159 & \textbf{0.016} & \textbf{0.030} & 0.169 & 
            0.048 & 0.538 & \textbf{0.021} & \textbf{0.039} & \textbf{0.009} & \underline{0.114} \\
            \midrule
        \end{tabular}
    }
    \label{table:TUMmono}
     \vspace{-1em}
\end{table*}


\begin{table*}[h]
    \centering
    \caption{
        \textbf{Stereo SLAM results on TartanAir Stereo Benchmark with Metric: ATE[m]}. X means the system has failed here. - means lack of result. We achieve comparable results on par with DROID-SLAM.
    }
    \resizebox{.8\linewidth}{!}{%
        \begin{tabular}{l|cccccccc | c}
            Stereo & SH000 & SH001 & SH002 & SH003 & SH004 & SH005 & SH006 & SH007 & Avg \\
            \toprule
            ORB-SLAM2~\cite{orbslam2} & \textbf{0.05} & 6.67 & X & X & X & X & 0.10 & X & -\\
            TartanVO~\cite{tartanvo2020corl} & 2.52 & 1.61 & 3.65 & 0.29 & 3.36 & \underline{4.74} & 3.72 & 3.06 & 2.87 \\
            DROID-SLAM*~\cite{teed2021droid} & 0.44 & \textbf{0.08} & \textbf{0.13} & \underline{0.20}  & \underline{0.16} & \textbf{3.29} & \underline{0.38} & \underline{0.18} & \textbf{0.61} \\
            Ours & \underline{0.14} & \underline{0.10} & \textbf{0.13} & \textbf{0.08}
             & \textbf{0.09} & 7.60 & \textbf{0.03} & \textbf{0.02} & \underline{1.02} \\
            \midrule
        \end{tabular}
    }
    \label{table:TartanAir stereo}
     \vspace{-1em}
\end{table*}


\begin{table*} [h]
    \caption{
        \textbf{Stereo SLAM results on EuRoC Dataset with Metric: ATE[m]}. - means lack of result. We achieve comparable results on par with DROID-SLAM.
    }
    \centering
    \resizebox{.9\linewidth}{!}{%
        \begin{tabular}{l| ccccc | ccc | ccc | c}
            & MH01 & MH02 & MH03 & MH04 & MH05 & V101 & V102 & V103 & V201 & V202 & V203 & Avg \\
            \toprule
            D3VO + DSO~\cite{d3vo} & - & - & 0.08 \ \ & - & 0.09 \ \ & - & - & 0.11 \ \ & - & 0.05 \ \ & - & - \\
            ORB-SLAM2~\cite{orbslam2} & 0.035 & 0.018 & 0.028 & 0.119 & 0.060 & \underline{0.035} & 0.020 & \underline{0.048} & \underline{0.037} & 0.035 & - & - \\
            VINS-Fusion~\cite{vinsf} & 0.540 & 0.460 & 0.330 & 0.780 & 0.500 & 0.550 & 0.230 & - & 0.230 & 0.200 & - & - \\
            SVO~\cite{svo} & 0.040 & 0.070 & 0.270 & 0.170 & 0.120 & 0.040 & 0.040 & 0.070 & 0.050 & 0.090 & 0.790 & 0.159 \\
            ORB-SLAM3~\cite{orbslam3} & 0.029 & 0.019 & \underline{0.024} & 0.085 & 0.052 & \underline{0.035} & 0.025 & 0.061 & 0.041 & \underline{0.028} & 0.521 & 0.084 \\
            \midrule
            DROID-SLAM*~\cite{teed2021droid} & \textbf{0.015} & \underline{0.013} & 0.035 & \textbf{0.048} & \textbf{0.040} & 0.037 & \textbf{0.011} & \textbf{0.020} & \textbf{0.018} & \textbf{0.015} & \textbf{0.017} & \textbf{0.024} \\
            Ours & \underline{0.018} & \textbf{0.011} & \textbf{0.020} & \underline{0.061} & \underline{0.042} & \textbf{0.031} & \textbf{0.011} & 0.090 & 0.039 & 0.037 & \underline{0.478} & \underline{0.075} \\
            \midrule
        \end{tabular}
    }
    \label{table:EurocStereo}
     \vspace{-1em}
\end{table*}
\vspace{-1em}
\subsection{Datasets}
\noindent\textbf{Virtual KITTI2}~\cite{cabon2020virtual} consists of 5 sequence clones from the KITTI tracking benchmark, and provides different variants of these sequences such as modified weather conditions (e.g. fog, rain) or modified camera configurations (e.g. rotated by $15^{\circ}$). 
In the ablation study, we use the split \textit{clone} as the training set, the split \textit{15-degree rotation} as the validation set, and the split \textit{30-degree rotation} of as the test set. 

\noindent\textbf{TartanAir}~\cite{tartanair2020iros} is a challenging dataset for robot navigation tasks and SLAM. The data is collected in photo-realistic simulation environments with the presence of moving objects, changing light, and various weather conditions. We use the official test split ~\cite{tartanair2020iros} and calculate ATE in all sequences. 

\noindent\textbf{KITTI}~\cite{geiger2013vision} is a dataset capturing real-world traffic situations and ranges from freeways over rural areas to inner-city scenes with many static and dynamic objects. It contains object labels in the form of 3D tracklets that provides online benchmarks for stereo, optical flow, object detection, and visual odometry. 

\noindent\textbf{TUM-RGBD}~\cite{schubert2018tum} consists of indoor scenes captured with a handheld camera. This is a notoriously difficult dataset for monocular methods due to rolling shutter artifacts, motion blur, and heavy rotation~\cite{teed2021droid}. We use the dynamic sequences to evaluate the effectiveness of DeFlowSLAM in dynamic scenes and exploit the static sequences in monocular or stereo settings.

\noindent\textbf{EuRoc}~\cite{burri2016euroc} is a dataset containing 11 sequences classified as simple, medium and difficult based on illumination and camera motion. The dataset \cite{d3vo} is very challenging due to the strong motion and the significant illumination changes between stereo and temporal images. It is a widely used benchmark to evaluate SLAM systems. 
















\vspace{-1em}
\subsection{Ablation Study}
We conduct an ablation study to verify the effectiveness of our dual-flow representation in dynamic scenes and further explore the self-supervised training to train our model. We chose Virtual KITTI2 dataset \cite{cabon2020virtual} for ablation study. 


\noindent\textbf{Framework Design.} To remove the interference of dynamic objects, an easy way to implement this is to add a module to DROID-SLAM that removes dynamic objects, which is considered a mask-removal approach. Instead, we use our dual-flow representation to keep more pixels for solving the pose, decomposing the static components even for regions of dynamic objects. Fig.~\ref{fig:weight} shows that DeFlowSLAM can exploit the dynamic pixels to solve the camera pose, compared with DROID-SLAM~\cite{teed2021droid}. Tab.~\ref{table:vkitti ablation study} shows that the dual-flow representation is better than the rough single-flow method. Notice that SS means self-supervised, SM means semi-supervised, SF means single flow, and DF means dual flow.






\noindent\textbf{Supervision Policy.} We explore the possibility of training DeFlowSLAM in a self-supervised manner. Compared with the supervised approach, which requires a large number of complicated loss functions and is difficult to adjust the appropriate loss weights, and the semi-supervised approach, which requires additional masks for dynamic objects, the self-supervised approach achieves the transformation of different loss functions, allowing a less restricted adaptation to new scenarios and demonstrating better generalization ability. Tab. \ref{table:vkitti ablation study} demonstrates that the self-supervised approach can achieve comparable or better accuracy than the supervised approach.

\noindent\textbf{Mask-Agg Setting.}
In self-supervised training, we introduce the Mask-Agg technique to reduce the influence of error matching of the pixels. Because the predicted dynamic masks in self-supervised training are not converged in the early stage of training, we only introduce Mask-Agg at the last $1/10$ epochs. Tab. \ref{table:vkitti ablation study} demonstrates that Mask-Agg helps to improve pose estimation, where MA means Mask-Agg.

\noindent\textbf{Dynamic Threshold $\mu$.}
In self-supervised setting, we perform an ablation study of dynamic threshold $\mu$ on Virtual KITTI2 (VK) dataset. Experiments demonstrate that $\mu=0.5$ achieves the better performance, as shown in Tab.~\ref{table:dynamic threshold}.

















































\vspace{-1em}
\subsection{Generalization} After verifying the validity of our hypothesis, we trained our DeFlowSLAM on TartanAir dataset~\cite{tartanair2020iros} from scratch and test our DeFlowSLAM on other popular SLAM datasets, such as Virtual KITTI2 (VKITTI2) \cite{cabon2020virtual} and KITTI \cite{geiger2013vision} for autonomous driving scenarios with dynamic objects, EuRoc \cite{burri2016euroc} for drones with strong motion and significant illumination changes, and TUM RGB-D \cite{schubert2018tum} for hand-held SLAM with motion blur and heavy rotations. The results demonstrate that our model generalizes well to different datasets, shown in Figure \ref{fig:slam reconstruction}. Next, we show the effectiveness of DeFlowSLAM in several settings.

\begin{figure*}[t]
  \centering
  \includegraphics[width=\linewidth]{pictures/AR_gt_deflow_droid_original_18.png}
  \vspace{-2em}
  \caption{\textbf{AR Application.} DeFlowSLAM deals with the dynamic objects in the scene very well while DROID-SLAM exhibits significant drifts~(the red boxes). We augment the original video with a virtual tree, a car, and a street lamp. From left to right and from top to down: Ground-Truth, DeFlowSLAM, DROID-SLAM, Original Image. }
  \label{fig:AR demo}
  \vspace{-1em}
\end{figure*}
\vspace{-1em}

\begin{table}[t]
    \caption{\textbf{Motion Segmentation.} We show the motion segmentation results of DeFlowSLAM on Virtual KITTI2 dataset.
   
    }
    \resizebox{\linewidth}{!}{%
        \begin{tabular}{l|c c c c c}
            Monocular & VK01 & VK02 & VK06 & VK18 & VK20 \\
            \toprule
            Ours  &0.538585	&0.528356	&0.562788	&0.739282	&0.654025\\
            \bottomrule
        \end{tabular}
        \label{table:vkitti motion segmentation}
    }
    \vspace{-1em}
\end{table}



\begin{figure}[t]
\label{fig:motion segmentation}
  \centering
  \includegraphics[width=\linewidth]{pictures/motion_segmentation_gt_pt.png}
    \vspace{-2em}
  \caption{\textbf{Motion Segmentation.} From left to right: Ground-Truth dynamic mask, Predicted dynamic mask larger than the dynamic threshold $\mu=0.5$.}
  \vspace{-1em}
\end{figure}

\subsection{Dynamic SLAM}
We test the performance of the proposed DeFlowSLAM on sequences 09 and 10 from KITTI dataset \cite{geiger2013vision} and all sequences from Virtual KITTI2 dataset \cite{cabon2020virtual}. Here we provide an additional trajectory for this experiment. The ATE results are shown in Table \ref{tab:Kitti}. Compared with DynaSLAM \cite{bescos2018dynaslam} which uses Mask-RCNN for dynamic environment and DROID-SLAM \cite{teed2021droid}, our DeFlowSLAM is far more accurate and robust in dynamic scenes. We also perform evaluations on TUM RGB-D dynamic sequences with different dynamic proportions and the comparison results in Tab.~\ref{tab:tum_dy} shows that DeFlowSLAM achieves competitive and even best performance. Note that DVO SLAM~\cite{kerl2013robust} , ORB-SLAM2~\cite{orbslam2} and PointCorr~\cite{dai2020rgb} use the RGB-D dataset, while our method and DROID-SLAM~\cite{teed2021droid} only use the monocular RGB dataset.




\vspace{-1em}
\subsection{Monocular SLAM}
In monocular setting, we test our trained DeFlowSLAM on TartanAir test sets, EuRoC, and TUM RGB-D dataset.
As shown in Tab.~\ref{table:TartanAir mono}, DeFlowSLAM achieves the best results. 
Tab.~\ref{table:EurocMono} and Tab.~\ref{table:TUMmono} show that our method achieves comparable even better results than the SOTA supervised method, DROID-SLAM~\cite{teed2021droid} in most sequences. The results also demonstrate our DeFlowSLAM is more robust than the classical SLAM algorithms as they failed in many sequences. Specifically, we achieve an average ATE of 0.136m on EuRoC dataset in the monocular setting, and an average ATE of 0.114m on TUM-RGBD static sequences, outperforming most supervised methods.













\vspace{-1em}
\subsection{Stereo SLAM}
Under stereo setup, our trained DeFlowSLAM is also tested on TartanAir test dataset and EuRoC stereo dataset. Tab.~\ref{table:TartanAir stereo} illustrates  DeFlowSLAM achieves comparable results on par with DROID-SLAM~\cite{teed2021droid}, with an average ATE of 1.02m on TartanAir stereo test dataset, outperforming TartanVO~\cite{tartanvo}. Tab.~\ref{table:EurocStereo} shows that DeFlowSLAM exhibits comparable results on EuRoc dataset in the stereo setting with DROID-SLAM~\cite{teed2021droid}, outperforming most supervised methods and traditional SLAM, ORB-SLAM3~\cite{orbslam3}. In most sequences, our method is on the same order of magnitude as DROID-SLAM~\cite{teed2021droid}, which shows the effectiveness of our method.  




























\vspace{-1em}
\subsection{AR Applications}
We conduct extensive experiments on AR applications to demonstrate the robustness of DeFlowSLAM. As shown in Fig. \ref{fig:AR demo}, we augment the original video with a virtual tree, a car, and a street lamp. Our DeFlowSLAM can deal with the dynamic objects in the scene very well while DROID-SLAM exhibits significant drifts (the red boxes). 


\vspace{-1em}
\subsection{Motion Segmentation}
Although our method focuses on the SLAM system, our dual-flow representation can be well applied to motion segmentation. We simply set a threshold value (such as $\mu=0.5$) for motion and visualize the pixel points of the dynamic field larger than this threshold to obtain the result of motion segmentation, as shown in Fig.~\ref{fig:motion segmentation} and Tab.~\ref{table:vkitti motion segmentation}.






\vspace{-1em}
\section{Discussion and Limitations}
Although our system is versatile to many SLAM settings and is more robust to the challenges of dynamic scenes, it can be further improved in the following directions. DeFlowSLAM performs slightly weaker in some scenarios than DROID-SLAM, probably because we use a certain fixed dynamic threshold. We can explore dynamic threshold estimation methods to the challenges of different scenarios.
As with droid-SLAM, we have high memory requirements for longer sequences and larger scenes, and our DeFlowSLAM system needs to run in segments. A lightweight and efficient SLAM system is a potential research direction. 
For some loop closure sequences, we can add loop closure constraints to reduce the drift. 
Our system focuses more on solving the camera pose, and the depth and optical flow obtained are only 1/8 of the original image size, which is not ideal for tasks like depth estimation and optical flow estimation. An efficient and versatile SLAM system is worthwhile researching. We can also explore dynamic object reconstruction with dynamic dense SLAM. 
\vspace{-1em}
\section{Conclusions}
We present a novel dual-flow representation that decomposes the optical flow into a static flow field caused by the camera poses and a dynamic field caused by the dynamic objects' motion. We present dynamic update module with dual-flow representation to compose the full SLAM system, DeFlowSLAM.
We explore a self-supervised method to train our dual-flow based dynamic dense SLAM system, which outperforms DROID-SLAM in highly dynamic scenes and achieves comparable performance in static and slightly dynamic scenes. 

\vspace{-1em}

\section*{Acknowledgments}
This work was partially supported by NSF of China (No. 61932003) and ZJU-SenseTime Joint Lab of 3D Vision.


\bibliographystyle{IEEEtran}
