\section*{MULTIMEDIA MATERIAL} 
\label{MULTIMEDIA MATERIAL}
{\footnotesize \textbf{Website}: \url{https://github.com/arclab-hku/Event_based_VO-VIO-SLAM}.}

{\footnotesize \textbf{Demo Video}: \url{https://youtu.be/XqAm1q0alNY}.}

\vspace{-0.5em
\section{INTRODUCTION}
\label{INTRODUCTION}

\IEEEPARstart
{E}{vent} cameras are novel bio-inspired sensors\cite{CPYHKU:EVENT-SURVEY}, which have a high dynamic range (140 dB compared to 60 dB of standard cameras) to handle broad illumination conditions.
\begin{figure}[htb] 
       
       
        \centering
        \captionsetup{justification=justified
        \includegraphics[width=1.0\columnwidth]{firstpic.png}
        \caption{Our ESVIO combines events, images, and IMU to provide robust onboard control for drones under aggressive motion. Event streams provide rich and reliable features, while only few features are tracked in image frames due to motion blur. 
        \textbf{Left bottom:} stereo event-based feature tracking. 
        \textbf{Right bottom:} stereo image-based feature tracking. } 
        \label{firstpic}
        \vspace{-2.0em
\end{figure}%
Unlike standard cameras that output fixed-rate image frames, event cameras respond to pixel-level intensity changes and output asynchronous event streams at the latency of microsecond level, which endows these novel sensors to tackle high-speed motion without motion blur.

Most of the existing event-based visual odometers (VO) use monocular event camera\cite{CPYHKU:EVO,CPYHKU:Ultimate-SLAM,CPYHKU:PL-EVIO}, while few research on visual odometry based on stereo event cameras\cite{CPYHKU:ESVO}\cite{CPYHKU:Feature-based-ESVO}. 
Since event cameras output asynchronous event streams rather than fixed-rate image frames, the traditional image-based instantaneous matching cannot be directly implemented on event streams.
For consecutive stereo event streams, merely relying on temporal constraints to extract and match event-corner features might lead to many false correspondences.
Temporal deviations between event streams, noise impacts, different contrast sensitivity of sensors, etc. cause the above problem.
Therefore, it is crucial to extract apposite event-corner features and design proper constraints to achieve the matching and tracking of stereo event-corner features.


Compared with standard cameras, event cameras do not suffer from motion blur under aggressive motion.
However, when the relative motion between event cameras and scenes is restricted, e.g. in the stationary state, event streams might not be reliably generated and transmitted, whereas standard cameras are able to provide rich information most of the time (e.g. low-speed motion and well-lit scenes). 
Observing this complementarity, leveraging both of the advantages of the aforementioned different sensors in combination with an inertial measurement unit (IMU) results in a robust and accurate visual-inertial odometry (VIO) pipeline\cite{CPYHKU:Ultimate-SLAM} \cite{CPYHKU:PL-EVIO}.




In this paper, we propose, to the best of our knowledge, the first published event-based stereo visual-inertial odometry (ESVIO). Our contributions are summarized as follows:
\begin{enumerate}
\item 
In order to achieve robust state estimation under aggressive motion and low-light scenarios, we propose the first purely event-based stereo VIO (ESIO) pipeline with sliding windows graph-based optimization, and further extend it with image-aided (ESVIO) which tightly integrated stereo event streams, stereo image frames, and IMU together. 

\item
To tackle the problem of event-based stereo feature tracking and matching, we designed geometry-based temporal event-based feature tracking method and instantaneous event-based stereo feature matching method in the consecutive stereo event streams. These two methods ensure accurate and reliable state estimation.

\item
We evaluate that our ESVIO can achieve state-of-the-art performance on publicly available datasets.
We also release a very challenging event-based VO/VIO dataset, featuring aggressive motion and HDR scenarios, during the evaluation of our methods. 
Finally, we perform onboard closed-loop quadrotor flight using our ESVIO as the estimator.

\end{enumerate}

The remainder of the paper is organized as follows:
Section \ref{Related Works} introduces the related works. 
Section \ref{Methodology} introduces the methodology of our methods.
Section \ref{Evaluation} presents the experiments and results.
Finally, the conclusion is given in Section \ref{CONCLUSIONS}.

\vspace{-0.5em
\section{Related Works}
\label{Related Works}

\subsection{Event-based Monocular Visual Odometry}
Event-based monocular VO has been intensively researched for challenging scenarios in recent years.
\cite{CPYHKU:kueng2016low} is the first work using feature tracks to achieve event-based VO, which detects features firstly from grayscale frames and then uses event streams tracked features asynchronously. 
EVO\cite{CPYHKU:EVO} proposed a monocular event-based parallel tracking-and-mapping philosophy which applies the image-to-model alignment for tracking and Event-based Multi-View Stereo (EMVS)\cite{CPYHKU:EMVS} for mapping. 
\cite{CPYHKU:Event-based-visual-inertial-odometry} proposed the first algorithm to fuse an event-based camera with an IMU that provides accurate 6-DoF state estimation based on Extended Kalman Filter (EKF).
\cite{CPYHKU:ETH-EIO} proposed a tightly-coupled VIO for event cameras, which utilizes the camera motion and the scene structure to fuse events stream in a spatio-temporal window.
Ultimate SLAM\cite{CPYHKU:Ultimate-SLAM} furthered the aforementioned research by combining events, images and IMU measurements with nonlinear optimization, which leverages the complementary advantages of event cameras and standard cameras. 
\cite{CPYHKU:Continuous-time-visual-inertial-odometry-for-event-cameras} adopted a continuous-time framework for trajectory estimation and fused the event camera and IMU together.
DEVO\cite{CPYHKU:DEVO} proposed a novel VO based on a hybrid setup of depth and event cameras, which construct a semi-dense depth map by thresholding time-surface maps.
EKLT-VIO\cite{CPYHKU:EKLT-VIO} integrated an accurate state-of-the-art event-based feature tracker EKLT\cite{CPYHKU:EKLT} with EKF backend to achieve event-based state estimation on Mars-like datasets. 
\cite{CPYHKU:GuanEVIO} proposed a real-time monocular event-based VIO based on graph optimization, which directly utilizes the asynchronous raw events for feature detection.
PL-EVIO\cite{CPYHKU:PL-EVIO} extended the above method to leverage the complementary advantages of standard and event cameras, which tightly combined event-based point features, event-based line features, image-based point features, and IMU measurements.


\subsection{Event-based Stereo Visual Odometry} 
In contrast to monocular VO, which requires sufficient parallax to recover the depth of corner features, stereo VO can directly obtain the depth of features at the current timestamp.
This solves scale uncertainty and even tracking failure caused by insufficient parallax.
However, most of the recent research on stereo event cameras has focused on depth estimation and constructing semi-dense or dense maps\cite{tulyakov2019learning}\cite{nam2022stereo}, with less research on VO/SLAM fields.
ESVO\cite{CPYHKU:ESVO} proposed the first event-based stereo VO pipeline, which achieves parallel 3D semi-dense mapping thread and tracking thread by maximizing the spatio-temporal consistency of stereo event streams. 
\cite{CPYHKU:Feature-based-ESVO} adopted stereo feature detection and matching with the geometry method, which adopts reprojection error minimization to achieve pose estimation. 
However, these algorithms merely use events to estimate the state, which might result in tracking failure when the system is stationary.
In addition, these aforementioned approaches without combining IMU might lead to losses of visual tracks under textureless areas.
Our work fills a gap based on combining stereo event streams, stereo image frames and IMU together, which can operate in real-time under high-resolution event streams with better performance than previously proposed methods. 

\vspace{-0.5em
\section{Methodology}
\label{Methodology}
A brief framework of our ESVIO is presented in Fig. \ref{framework}. 
The core of our system lies in the pre-processing of raw event streams using motion compensation and the tracking and matching of consecutive stereo event streams in temporal and spatial.
Meanwhile, we demonstrate the event-based constraint residuals for graph optimization.
Please note that we also propose ESIO, a pure event-based VIO that only combines stereo event streams and IMU measurements together.

\begin{figure}[htb] 
       
        \vspace{-1.0em
        \centering
        \captionsetup{justification=justified
        \includegraphics[width=0.9\columnwidth]{framework.png}
        \caption{Framework illustrating the brief pipeline of the proposed ESVIO. 
        The front end starts with pre-processing different sensor measurements. 
        The back end tightly fuses instantaneous stereo event residual, temporal stereo event residual, image residual and IMU residual together.} 
        \label{framework}
        \vspace{-1.0em
\end{figure}%

\subsection{Motion Compensation for Event Streams} 
Motion compensation corrects the curved event streams by aligning events corresponding to the same scene edge.
\cite{CPYHKU:ETH_SCI_Ro_avo} only uses the angular velocity of IMU to achieve rotational compensation, while ignoring the effect of translation. 
\cite{CPYHKU:FAST-Dynamic-Vision} achieve the rotational and translational compensation by IMU and depth camera respectively. 
We design a motion compensation approach to correct each raw event position, which uses the angular velocity from the IMU sensor and the linear velocity from our ESVIO back-end to achieve rotational and translational compensation respectively. 
\begin{figure}[htb] 
       
        \vspace{-1.0em
        \centering
        \captionsetup{justification=justified
        \includegraphics[width=0.9\columnwidth]{motion_compensation.png}
        \caption{The event streams before and after motion compensation.} 
        \label{motion_compensation}
        \vspace{-1.0em
\end{figure}%

Given the $k$th event as $e_{k}=\{l_{k},t_{k},p_{k}\}$, where $l_{k}=\{x_{k},y_{k}\}$ represents the pixel location of the event $e_{k}$. $t_{k}$ is the timestamp and $p_{k}$ represents its polarity.
The event $e_{k}$ is warped from $t_{k}$ to $t_{ref}$, and the compensated location $l_{k}^{'}$ define as
\begin{equation}
\begin{split}
\label{compensated_location}
l_{k}^{'} = \mathcal{M} [x_{k},y_{k},t_{k},\Theta]
\end{split}
\end{equation}
where $\mathcal{M}$ is the motion compensation function. $\Theta$ represents compensation parameters.
Since the short time interval $\delta t$ between $t_{ref}$ and $t_{k}$, we assume that the motion during this period is uniform motion, thereby the ego-motion estimation of each event can be formulated as follow:
\begin{equation}
\begin{split}
\label{motion_compensation_eq}
R(\delta t)  & = \text{exp}(\omega({t_{k}}) \delta t)\\
L^{'}(t_{k}) & = R(\delta t) L(t_{k}) + v(t_{ref})\delta t\\
\end{split}
\end{equation}
where $\text{exp}$ denotes the exponential map from $se(3)$ to $SE(3)$. $R({\delta t})$ is the rotation matrix converted from the euler angle $\omega({t_{k}})\delta t$, $\omega (t_{k})= \tilde{\omega}(t_{k})-b_{g}(t_{k})-n_{g}(t_{k})$. $\tilde{\omega}(t_{k})$ is the measurement of gyroscope, while $b_{g}(t_{k})$ and $n_{g}(t_{k})$ are bias and noise variable of gyroscope respectively. $L(t_{k})=\{x_{k},y_{k},1\}$ is homogeneous matrix that extended from $l_{k}$. $v(t_{ref})$ represents the velocity of our ESVIO back-end at $t_{ref}$ timestamp. Finally, we convert $L^{'}(t_{k})$ to homogeneous matrix and obtain the compensated location $l^{'}_{k}$.
Fig. \ref{motion_compensation} compares raw event streams and motion-compensated event streams, where raw event streams produce a certain extent of motion blur and the compensated event streams show clear contours of scenes.

\subsection{Stereo Event-corner Feature Detection, Temporal Tracking and Instantaneous Matching}
The stereo raw event streams are compensated to generate surface of active event (SAE), which is a 2D array used to store event pixels and timestamps. 
Event corners are extracted on the SAE using the publicly available feature detection algorithm ARC*\cite{CPYHKU:ARC*}. 
For a new arrival event, the detector inspects a circular set of events around its position and searches for arc-like corners in the SAE.
Furthermore, to reduce the influence of noisy events, we apply the time surface to further filter event-corner features. 
Firstly, we only select those features whose timestamps are within a short interval from that of the current time surface, thereby retaining features at the dense event streams. 
Secondly, we set the minimum distance $d$ between adjacent event-corner features to distribute them uniformly.

\begin{figure}[htb] 
       
        \vspace{-1.0em
        \centering
        \captionsetup{justification=justified
        \includegraphics[width=1.0\columnwidth]{tracking.png}
        \caption{Stereo event-corner features:
    (a) Geometry of stereo vision;
    (b) Event corner temporal tracking and instantaneous matching on the time surface;
    (c) Event corner tracking on left event streams. } 
        \label{tracking}
        \vspace{-1.0em
\end{figure}%
To obtain the 3D coordinate of event-corner features and perform nonlinear optimization, we adopt both temporal feature tracking and instantaneous stereo matching. The geometry of the temporal tracking and instantaneous matching is depicted in Fig. \ref{tracking}(a).
To ensure that the matched features between the left and right event streams lie along the epipolar line, we use stereo-rectified time surfaces for instantaneous stereo matching.
Stereo event-corner features, $\mathcal{F}_{l}^{i}$ and $\mathcal{F}_{r}^{i}$, are instantaneously matched by forward and inverse LK optical flow\cite{CPYHKU:LK_optical_flow} between the left and right time surface at the current timestamp $i$. 
Meanwhile, the temporal feature tracking module also uses forward and inverse optical flow to track event-corner features of left event streams, $\mathcal{F}_{l}^{i}$ and $\mathcal{F}_{l}^{j}$, at consecutive two timestamps $i$ and $j$. 
In Fig. \ref{tracking}(b), red and blue dots represent event-corner features extracted on the left and right time surface at timestamp $i$ respectively, while yellow dots denote the features on the left time surface at the previous timestamp $j$.
Note that the left bottom and right bottom pictures form instantaneous stereo matching, while the left bottom and left top pictures form temporal feature tracking. 
Green lines connect matched event-corner features between the left and right time surfaces at the same timestamp, while yellow lines connect tracked features with two consecutive left time surfaces at $i$ and $j$.
Fig. \ref{tracking}(c) shows the event-corner features tracking on event streams. 
The result shows our proposed method can achieve correct temporal tracking and instantaneous matching of stereo event-corner features.

Finally, we recover the inverse depth of corner features that are successfully tracked and matched with consecutive left and right event streams. 
As depicted in Fig. \ref{tracking}(a), the matched corner features of the 3D point $P$ on the imaging plane of left and right event cameras at timestamp $i$ are $p_{L}^{i}$ and $p_{R}^{i}$ respectively. The inverse depth of $P$ can be formulated through epipolar geometry and triangulation between $p_{L}^{i}$ and $p_{R}^{i}$.
Meanwhile, the event-corner feature of left event camera $p_{L}^{i}$ and $p_{L}^{j}$ at timestamps $i$ and $j$ can also be triangulated to recover the inverse depth.

\subsection{The Construction of Event-based Residual Constraint for the Graph-based Optimization} 

The full state vector in the sliding window is defined as
\begin{equation}
\begin{split}
\label{state}
\chi &= [x_{b_{0}},...,x_{b_{n}}, x^{b}_{e}, x^{b}_{c}, \Lambda_{et},
\Lambda_{em},
\Lambda_{c}]\\
x_{b_{k}} &= [ p_{b_{k}}^{w}, q_{b_{k}}^{w},v_{b_{k}}^{w},b_{a_{k}},b_{g_{k}} ] \quad k\in [0,n]\\
\end{split}
\end{equation}
where $x_{b_{k}}$ is the state of IMU at timestamp $k$ in the world frame, which consists of the position $p_{b_{k}}^{w}$, the orientation quaternion $q_{b_{k}}^{w}$, the velocity $v_{b_{k}}^{w}$, the accelerometer bias $b_{a_{k}}$ and the gyroscope bias $b_{g_{k}}$. $x^{b}_{e}$ and $x^{b}_{c}$ are the extrinsic transformation from event camera and standard camera to IMU respectively. $\Lambda_{et} = [\lambda_{et_{0}},\lambda_{et_{1}},...,\lambda_{et_{l}}]$, $\Lambda_{em} = [\lambda_{em_{0}},\lambda_{em_{1}},...,\lambda_{em_{l}}]$, $\lambda_{et_{l}}, \lambda_{em_{l}}$ represent the inverse depth of the event-corner features $et_{l}$, $em_{l}$ respectively.
$\Lambda_{c} = [\lambda_{c_{0}},\lambda_{c_{1}},...,\lambda_{c_{l}}]$, $\lambda_{c_{l}}$ is the inverse depth of the image-based features $c_{l}$. $n$ is the total number of keyframes, and $l$ is the total number of features in the sliding window.

Combining event, image, and IMU residual terms, the event-visual-inertial odometry can be formulated as the joint nonlinear optimization problem as follow
\begin{equation}
\begin{split}
\label{Joint_nonlinear_optimization}
&\mathop{\mathrm{min}}\limits_{\chi} \Bigg(
\sum_{k \in b}\left \|r_{b}(\hat{z}_{b_{k+1}}^{b_{k}}, \chi)\right \|^{2}_{W_{b}}
+ \sum_{(l,k) \in c}\left \|r_{c}(\hat{z}_{c_{k}}^{l}, \chi)\right \|^{2}_{W_{c}} \\
&+\sum_{(l,k) \in et}\left \|r_{et}(\hat{z}_{et_{k}}^{l}, \chi)\right \|^{2}_{W_{et}}
+\sum_{(l,k) \in em}\left \|r_{em}(\hat{z}_{em_{k}}^{l}, \chi)\right \|^{2}_{W_{em}} \Bigg)
\end{split}
\end{equation}
where $r_{b}(\hat{z}_{b_{k+1}}^{b_{k}}, \chi)$ is the residual for IMU measurement with weight $W_{b}$. $r_{c}(\hat{z}_{c_{k}}^{l}, \chi)$ represents the residuals for standard cameras measurement with weight $W_{c}$. $r_{et}(\hat{z}_{et_{k}}^{l}, \chi)$ and $r_{em}(\hat{z}_{em_{k}}^{l}, \chi)$ are the residuals for event-based temporal tracking and instantaneous matching measurement respectively. The detailed definition of event-based residuals will be presented below, other residual terms can be found in \cite{CPYHKU:PL-EVIO}.

For the stereo event cameras, we construct the temporal tracking factor $r_{et}(\hat{z}_{et_{k}}^{l}, \chi)$ in the consecutive event streams and the instantaneous matching factor $r_{em}(\hat{z}_{em_{k}}^{l}, \chi)$ between the left and right event streams.
Consider the $l$th feature that is first observed in the $i$th left event stream, the residual for the event-corner feature observation in the $k$th left event stream is defined as:
\begin{equation}
\begin{split}
\setlength{\arraycolsep}{1.2pt}
        r_{et}=\left[
                \begin{array}{c}
                u^{l}_{et_{k}}\\
                v^{l}_{et_{k}}\\
                \end{array} 
                \right]-\pi_{e}\bigg((T^{b}_{le})^{-1} T^{b_{k}}_{w} T^{w}_{b_{i}}T^{b}_{le} \pi^{-1}_{e}(\frac{1}{\lambda_{et}},\left[
                        \begin{array}{c}
                        u^{l}_{et_{i}}\\
                        v^{l}_{et_{i}}\\
                        \end{array} 
                        \right])\bigg)
 \label{re-projection-event-temporal-tracking}
\end{split}
\end{equation}
where $[u^{l}_{e_{k}}, v^{l}_{e_{k}}]$ is the first observation of the $l$th event-corner feature in the $k$th event stream. $[u^{l}_{e_{i}}, v^{l}_{e_{i}}]$ is the same event-corner feature in the $i$th event stream. $\pi_{e}$ and $\pi^{-1}_{e}$ are the projection and back-projection functions of the event camera respectively. 
$T_{le}^{b}$ represents the extrinsic transformation from left event camera to body coordinate. 
$T^{w}_{b_{i}}$ indicates the pose of the body center related to the world frame at timestamp $i$, $T^{b_{k}}_{w}$ is the transpose of the pose of body coordinate in the world frame at the $k$th keyframe.

Consider the $l$th feature that is observed in the $i$th right event stream, the residual for the event-corner feature observation in the $i$th left event stream is defined as:
\begin{equation}
\begin{split}
\setlength{\arraycolsep}{1.2pt}
        r_{em}=\left[
                \begin{array}{c}
                u^{l}_{lem_{i}}\\
                v^{l}_{lem_{i}}\\
                \end{array} 
                \right]-\pi_{e}\bigg(T^{le}_{re} \pi^{-1}_{e}(\frac{1}{\lambda_{em}},\left[
                        \begin{array}{c}
                        u^{l}_{rem_{i}}\\
                        v^{l}_{rem_{i}}\\
                        \end{array} 
                        \right])\bigg)
 \label{re-projection-event-instantaneous-matching}
\end{split}
\end{equation}
where $[u^{l}_{lem_{i}}, v^{l}_{lem_{i}}]$ is the first observation of the $l$th event-corner feature in the $i$th left event stream. $[u^{l}_{rem_{i}}, v^{l}_{rem_{i}}]$ is the same event-corner feature in the $i$th right event stream. 
$T_{re}^{le}$ represents the extrinsic transformation from the right event camera to the left event camera. 

\vspace{-0.5em
\section{Evaluation}
\label{Evaluation}
We perform both dataset and real-world experiments to evaluate our ESVIO.
We firstly evaluate our proposed ESIO and ESVIO in the self-collected dataset which is acquired by two DAVIS346 ($346 \times 260$, event-sensor, image-sensor, IMU sensor) and VICON.
It contains extremely fast 6-Dof motion and scenes with HDR. 
In subsection \ref{Evaluation of Our ESVIO on Public Datasets}, we compare our methods with other event-based and image-based methods on two publicly available datasets: MVSEC \cite{CPYHKU:MVSEC} and VECtor \cite{CPYHKU:VECtor}.
We perform numerical analysis to evaluate the accuracy of our system.
The accuracy is measured with absolute trajectory error (ATE) aligning the estimated trajectory with ground truth using 6-DOF transformation (in SE3), which is calculated by the tool \cite{CPYHKU:evo_package}.
Finally, in subsection \ref{Indoor Quadrotor Flighting Evaluation} and \ref{Outdoor Large-scale Evaluation}, we evaluate our ESVIO in the onboard quadrotor flighting and outdoor large-scale environment, respectively.
All experiments run in real-time on an Intel NUC computer equipped with Intel i7-1260P, 32GB RAM, and Ubuntu 20.04 operation system.

\begin{figure}[htb] 
        \vspace{-1.0em
       
        \vspace{-1.0em
        \centering
        \captionsetup{justification=justified
        \includegraphics[width=0.85\columnwidth]{quadrotor_flight.png}
        \caption{Self-designed quadrotor with stereo event cameras and NUC.} 
        \label{quadrotor_flight}
        \vspace{-2.0em
\end{figure}%

\subsection {Evaluation of Our ESVIO in Challenging Situations}
\label{Evaluation of Our ESVIO in Challenge Situation}

\begin{table*}[htbp] 
       
       
        \renewcommand\arraystretch{1.2}
       
        \begin{center}
        \caption{Accuracy Comparison of Our ESVIO on HKU Dataset}
        \label{vicon_Comparison_with_ESVIO}
        \resizebox{\columnwidth*2}{!}
        { 
        \begin{threeparttable}
       
        \begin{tabular}{c|cccccccc} 
        \hline 
        \multicolumn{1}{c|}{Sequence} 
    & \makecell{ORB-SLAM3 \cite{CPYHKU:ORB-SLAM3} \\Stereo VIO} 
    & \makecell{VINS-Fusion \cite{CPYHKU:VINS-Fusion} \\Stereo VIO}
   
   
    &\makecell{USLAM \cite{CPYHKU:ETH-EIO} \\Mono EIO} 
    &\makecell{USLAM \cite{CPYHKU:Ultimate-SLAM} \\Mono EVIO}
    &\makecell{PL-EVIO \cite{CPYHKU:PL-EVIO} \\Mono EVIO}   
    & \makecell{\textbf{Our ESIO} \\\textbf{Stereo EIO}}
    & \makecell{\textbf{Our ESIO+}  \\\textbf{Stereo EIO}}
    &  \makecell{\textbf{Our ESVIO} \\\textbf{Stereo EVIO}}\\  
   
   
\hline
hku\_agg\_translation            & 0.095& 0.069& 10.41& 0.38& \textbf{0.048}& 0.38& 0.35 &0.063 \\
hku\_agg\_rotation             & 0.23& 0.88& \textit{failed}& 2.06 & 0.15 & 0.87& 0.51& \textbf{0.11}\\
hku\_agg\_flip            & \textbf{0.14}& 0.45& 4.32& 2.66& 0.15 & 1.47& 1.23& \textbf{0.14}\\
hku\_agg\_walk            & \textit{failed}&\textit{failed}& \textit{failed}& 1.75& 0.37 & 1.31& 1.14& \textbf{0.27}\\
hku\_hdr\_circle    & 0.083& 2.52& 0.46& 0.66& \textbf{0.068}& 0.69& 0.23& 0.081\\
hku\_hdr\_slow    & 0.086& 0.073& \textit{failed}& 1.52& 0.069& 0.16& 0.17& \textbf{0.059}\\
hku\_hdr\_tran\_rota    & 0.20& 0.075& \textit{failed}& 1.74& 0.068 & 0.55& 0.60& \textbf{0.065}\\
hku\_hdr\_agg    & 0.28& 1.18& \textit{failed}& 2.40& 0.14& 2.27& 1.37& \textbf{0.10}\\
hku\_dark\_normal           & \textit{failed}& 0.80& \textit{failed}& 2.01& 1.25& \textbf{0.28}& 0.32 & 0.39\\
\hline 
Average                & 0.16& 0.76& 5.06 & 1.69& 0.26& 0.89& 0.66& \textbf{0.14}  \\
\hline        
        \end{tabular}
        \begin{tablenotes} 
        \item \textit{*EIO means purely event-based VIO, EVIO means event-based VIO with image-aided} 
        \end{tablenotes} 
        \end{threeparttable} 
        }
        \end{center}
        \vspace{-3.0em}
\end{table*}

\subsubsection{\bf Experiment Data Description}

The dataset contains stereo event data at 60HZ and stereo image frames at 30Hz with resolution in $346 \times 260$, as well as IMU data at 1000Hz.
Timestamps between all sensors are synchronized in hardware.
We also provide ground truth poses from a motion capture system VICON at 50Hz during the beginning and end of each sequence, which can be used for accuracy performance. 
The data consists of handheld sequences including rapid motion and HDR scenarios.
The full setup including the attached infrared filter can be seen in Fig. \ref{quadrotor_flight}.
These two DAVIS346 are rigidly attached with the baseline of 6.0 cm and USB 3.0 interfaces are used to transmit sensor measurements to the NUC.
However, since the limitation of our hardware and cost, we use DAVIS346-COLOR and DAVIS346-MONO for the data collection.
Although this might introduce some artificial inconsistency, we think it is acceptable for the method evaluation.
The DAVIS comprises an image camera and event camera on the same pixel array, thus calibration can be done using standard image-based methods, such as Kalibr\footnote{\url{https://github.com/ethz-asl/kalibr}}, on the image frames and then are applied to the event camera.
For the benefit of the research community, we also release the dataset and the configuration files on our project website.

\subsubsection{\bf Methods Evaluation}

Table \ref{vicon_Comparison_with_ESVIO} compares the performance of our ESIO with the most relevant event-based or image-based systems.
Our ESIO has good performance, especially for the sequence \textit{hku\_agg\_walk} and \textit{hku\_dark\_normal}, our ESIO still can produce reliable and accurate pose estimation even when the state-of-the-art image-based VIO method, ORB-SLAM3, fail. 
Even if it does not perform too much improvement compared with our previous work PL-EVIO \cite{CPYHKU:PL-EVIO} in most of the sequence, ESIO and ESVIO are still a breakthrough for event-based stereo VIO.
For the motion compensation version (ESIO+), it shows effective improvement in most of the data sequence, which is opposite to the conclusion from Ref.\cite{CPYHKU:PL-EVIO}.
This might thank to the well-designed motion compensated methods that process the reliable and optimized compensated measurements from the ESVIO back-end.

Note that we also evaluate EVO\cite{CPYHKU:EVO} and ESVO\cite{CPYHKU:ESVO} in our datasets, but they failed in all sequences.
This might be caused by three factors: 
Firstly, both EVO and ESVO have strict initialization requirements. For example, EVO requires running in a scene that is planar to the sensor, up to several seconds, for bootstrapping the system. 
Secondly, they are sensitive to parameter tuning, even in their open-source project, they use different parameters for different sequences in the same scenarios. We might fail to correctly tune parameters for their successful running.
Finally, our dataset is so challenging that only the robust methods can perform well.
We also release the dataset on our project website for peer's performance comparison.

\subsection{Evaluation of Our ESVIO on Public Datasets}
\label{Evaluation of Our ESVIO on Public Datasets}

In this section, we evaluate our ESVIO on publicly available datasets and compare it with other image-based or event-based VO/VIO methods. 
The VECtor \cite{CPYHKU:VECtor} dataset consists of hardware-synchronized sensor suite that includes stereo event cameras, stereo standard cameras, an RGB-D sensor, a LiDAR, and an IMU.
It covers the full spectrum of 6 DoF motion dynamics, environment complexities, and illumination conditions for both small and large-scale scenarios.
To the best of our knowledge, we provide the first results on this new event-based dataset.
For the MVSEC \cite{CPYHKU:MVSEC}, we select the sequence which captured in the indoor flying room.
We used the stereo event camera ($640 \times 480$) and the regular stereo camera ($1224 \times 1024$) from the VECtor, and the DAVIS346 ($346 \times 260$ for both event and image) from the MVSEC, for evaluation, respectively.

\begin{table*}[htbp] 
       
       
        \renewcommand\arraystretch{1.2}
       
        \LARGE
        \begin{center}
        \caption{Accuracy Comparison of Our ESVIO with Other Image-based or Event-based Methods}
        \label{public_dataset}
        \resizebox{\columnwidth*2}{!}
        { 
        \begin{threeparttable}
       
        \begin{tabular}{c|c|ccccccc} 
        \hline  
    \multicolumn{2}{c|}{Sequence}  
    & \makecell{ORB-SLAM3 \cite{CPYHKU:ORB-SLAM3} \\Stereo VIO} 
    & \makecell{VINS-Fusion \cite{CPYHKU:VINS-Fusion} \\Stereo VIO}
    & \makecell{EVO \cite{CPYHKU:EVO} \\Mono EO}
    & \makecell{ESVO \cite{CPYHKU:ESVO} \\Stereo EO} 
    & \makecell{Ultimate SLAM \cite{CPYHKU:Ultimate-SLAM}\\Mono EVIO } 
    & \makecell{PL-EVIO \cite{CPYHKU:PL-EVIO} \\Mono EVIO} 
    & \makecell{\textbf{Our ESVIO} \\\textbf{Stereo EVIO}} \\
   
\hline
\multirow{17}*{\makecell{VECtor\cite{CPYHKU:VECtor}}}
  & corner-slow  & 0.012 & 0.013& 0.035 & 0.039& 0.039 & 0.017& \textbf{0.012}\\
~ & robot-normal  & 0.029& \textbf{0.023}& 0.13 &\textit{failed}& 0.047& 0.027 & 0.043 \\
~ & robot-fast & 0.15 & \textit{failed}& \textit{failed} &\textit{failed}& 0.35&  \textbf{0.037}& 0.042 \\
~ & desk-normal  & \textbf{0.039} & 0.040& \textit{failed} &\textit{failed}& 0.19  & 0.31& 0.052\\
~ & desk-fast  & 0.099 & 0.10& \textit{failed} &\textit{failed} &  0.34  & 0.043& \textbf{0.042} \\
~ & sofa-normal  & \textbf{0.044} & 0.038& \textit{failed} &0.53& 1.72&  0.058& 0.047\\
~ & sofa-fast  & 0.064 & 0.17& \textit{failed} &\textit{failed}& 0.76  & \textbf{0.050} & 0.052\\
~ & mountain-normal  & \textbf{0.026} & 0.30& \textit{failed} & \textit{failed}& 0.27 & 0.32& 0.044 \\
~ & mountain-fast  & 0.52& \textit{failed} & \textit{failed} &\textit{failed}& 1.02 & \textbf{0.031}& 0.039\\
~ & hdr-normal  &0.019 & 0.038& \textit{failed} &\textit{failed} & 0.17&  0.12& \textbf{0.017}\\
~ & hdr-fast  & 0.040& 0.055& \textit{failed} &\textit{failed} & 0.48 & \textbf{0.036} & 0.039\\
~ & corridors-dolly &\textbf{0.80} & 1.46& \textit{failed} &\textit{failed}  & \textit{failed} & 1.23& 0.88  \\
~ & corridors-walk  &1.03 & 0.39& \textit{failed} &\textit{failed}  & \textit{failed} & 0.72& \textbf{0.34}\\
~ & school-dolly     &0.92 &1.79 & \textit{failed} &13.71  & \textit{failed}  & 3.11 & \textbf{0.53} \\
~ & school-scooter   &0.75 & \textbf{0.56}& \textit{failed} &9.83  & 6.83 & 1.39& 0.63\\
~ & units-dolly    &18.06 & 10.39& \textit{failed} &\textit{failed}  & \textit{failed} & 13.82& \textbf{8.12}\\
~ & units-scooter  &14.50 & 11.47& \textit{failed} &\textit{failed}  & \textit{failed} & 11.66 & \textbf{6.64}\\
\hline
\multirow{4}*{\makecell{MVSEC\cite{CPYHKU:MVSEC} }}
    &Indoor Flying 1 & 1.42 & 0.40 & 1.36 & 1.07 & \textit{failed} &  0.36 & \textbf{0.25} \\
~ &Indoor Flying 2 & 1.70 & 2.10 & \textit{failed} & 1.10 & \textit{failed} & 0.30 & \textbf{0.30}\\
~ &Indoor Flying 3 & 1.54 & 0.39 & 1.37 & 0.91 &  \textit{failed} & 0.34 & \textbf{0.25}\\
~ &Indoor Flying 4 & 0.58 & 0.30 & \textit{failed} &\textit{failed}  & \textbf{0.23} & 0.44 & 0.46\\
\hline
        \end{tabular}
       
       
       
        \end{threeparttable} 
        }
        \end{center}
        \vspace{-1.0em}
\end{table*}

As can be seen in Table \ref{public_dataset}, our proposed ESVIO can achieve fairly good result in most of the sequence.
Although the performance of ORB-SLAM3 is slightly better than ours in some sequences (e.g. \textit{robot-normal}, \textit{desk-normal}, \textit{mountain-normal}), our ESVIO provides more reliable and accurate in most of the sequences in harsh situations with HDR or aggressive motion.
The proposed ESVIO is more precise than our previous PL-EVIO, especially in large-scale environment, this might due to the better event-corner depth estimation.
However, the traditional event-based methods \cite{CPYHKU:EVO}  \cite{CPYHKU:Ultimate-SLAM} \cite{CPYHKU:ESVO} failed in most of the sequences in these two datasets.

Please note that we think that parameter tuning is infeasible. 
Therefore, we evaluate our methods using fixed parameters for all sequences during the evaluations. 
We also use this criterion when evaluating other methods as a comparison.
However, the generalization capability of \cite{CPYHKU:EVO}\cite{CPYHKU:ESVO} is slightly poor, they need finely parameter tuning for different \textit{rosbag}.
Meanwhile, the performance of ESVO \cite{CPYHKU:ESVO} in Table \ref{public_dataset} is worse than that of the original paper, which is caused by the intercept of the time period from the original \textit{rosbag}.
For example, ESVO \cite{CPYHKU:ESVO} only can work well during [0-27s] of sequence \textit{Indoor Flying 3} (the total duration is 94s), but it has significant drifts after 27s. Instead we use the whole sequence \textit{Indoor Flying 3} without any timestamp modification. The absolute mean error of the original paper and our results are 0.19m and 0.91m respectively.

Besides, we emphasize real-time performance when evaluating our methods, while the computational burden of EVO \cite{CPYHKU:EVO} and ESVO \cite{CPYHKU:ESVO} is so large that we had to slow down the \textit{rosbag} such as $\times 0.2$ or $\times 0.5$ data-speed, during the evaluations. 
Other compared methods are evaluated using the original data-speed of the \textit{rosbag}.
We also provide a visual comparison between our method and the other methods in the accompanying video, and all the video records of our ESVIO during the evaluations can be obtained on our website.

Last but not the least, although our proposed ESVIO achieves satisfactory results, it still has limitations in the low-texture environment.
For example, the scenarios in sequence \textit{units-dolly} and \textit{units-scooter} is so special that the visual-only method might be easy to degenerate or mismatch during the loop-closure detection.
This also indicates that either event camera or standard camera has limitations, although event cameras play a complementary role to the traditional image-based method, multi-sensor fusion, especially vision-based and non-vision-based, should be developed to exploit the complementary advantage of different sensors.

\vspace{-1.5em}
\subsection {Indoor Quadrotor Flight Evaluation}
\label{Indoor Quadrotor Flighting Evaluation}

To further demonstrate the practicability of our ESVIO, we perform real-world experiments on a self-designed quadrotor platform.
We choose Pixracer autopilot as our flight platform with T-Motor F90 PRO, as shown in Fig. \ref{quadrotor_flight}. 
The states estimate from our ESVIO is used to provide onboard pose feedback control for the quadrotor.
The quadrotor is commanded to follows a circular pattern eight times continuously during the experiment.
The robust and accurate onboard state estimates of our ESVIO enable real-time feedback control.
Meanwhile, we also record the ground truth from VICON for further quantitative evaluation.

\begin{figure*}[htb] 
   
   
    
    \subfigbottomskip=-3pt
    \subfigcapskip=-6pt
    \captionsetup{justification=justified
    \centering
    \subfigure[]{
            \begin{minipage}[t]{1.8\columnwidth}
            \centering
            \includegraphics[width=1.0\columnwidth]{hdr_flight_evaluation_new.png}
            \label{flighting_hdr}
            \end{minipage}%
    }%
    
    \subfigure[]{
            \begin{minipage}[t]{1.8\columnwidth}
            \centering
            \includegraphics[width=1.0\columnwidth]{agg_flight_evaluation_new.png}
            \label{flighting_aggressive}
            \end{minipage}%
    }%
    \caption{
    The position (X, Y, Z), orientation (Roll, Pitch, Yaw), and the corresponding errors of our ESVIO in onboard flight compared with the ground truth from VICON:
    (a) onboard quadrotor flight in low-illumination conditions;
    (b) onboard quadrotor flight in aggressive motion. 
    
    \label{flighting_performance}
    \vspace{-2.0em
\end{figure*}%

\subsubsection{\bf Quadrotor Flight in HDR Scenarios}

We give the translational and rotational results of our ESVIO against the VICON in Fig. \ref{flighting_hdr}.
The total trajectory length is 56.0m. The translation errors in the X and Y axis are within 0.3m, while the error in the Z axis is within 0.1m.
For the rotational part, the error of roll and pitch are both within $6^{\circ}$.
The error in yaw at 55s is larger than $10^{\circ}$, which is caused by rapid change in yaw at that moment resulting in the estimated pose being slightly slower than VICON.
The root mean square error (RMSE) of ATE in HDR flight is 0.17m.
The video demo can be seen in\footnote{\url{https://b23.tv/wcZiKzG}}.

\subsubsection{\bf Quadrotor Flight in Aggressive Motion}

In this section, the yaw angle of the commanded pattern is changed drastically, for aggressive motion.
The performance of our ESVIO is qualitatively demonstrate in Fig. \ref{firstpic} and quantitatively evaluated in Fig. \ref{flighting_aggressive}.
Note that it would have some outliers during the comparison with the VICON. 
For example, there is an error that is more than $10^{\circ}$ in the roll and yaw axis at 22s.
This is caused by VICON's ball is not well observed during aggressive flight, resulting in an inaccurate measurement of the VICON at that moment.
However, our reliable ESVIO state estimator still provides robust and accurate onboard pose feedback for the quadrotor.
The video demo can be seen in\footnote{\url{https://b23.tv/nQUbMjy}}.

\vspace{-1.0em}
\subsection {Outdoor Large-scale Evaluation}
\label{Outdoor Large-scale Evaluation}
A large-scale experiment is carried out on the HKU campus to illustrate the long-time practicability of our ESVIO, features with large-scale, indoor-outdoor conversion, pedestrians in the scene generating outlier events, etc. 
The path length of the outdoor evaluation is around 1866m and the duration is 34.9 minutes. 
The evaluation covers the place around 310m in length, 170m in width, and 55m in height changes.
Since the VICON is not available outdoors, we only show the qualitative performance and the estimated trajectory overlaid with the Google map for visual comparison, as shown in Fig. \ref{google_map}.
More details of this outdoor large-scale evaluation can be seen on our website.

\begin{figure}[htb] 
       
       
        \centering
        \captionsetup{justification=justified
        \includegraphics[width=1.0\columnwidth]{HKU_outdoor.jpg}
        \caption{The estimated trajectory of our ESVIO in the HKU outdoor. 
        We also visualize the detection and tracking situation of the stereo event-corner features, and stereo image features, during the experiment.
        The complementarity of event-based and image-based measurements ensures the robustness of the system.} 
        \label{google_map}
        \vspace{-1.5em
\end{figure}%

\vspace{-1.0em}
\section{CONCLUSIONS}
\label{CONCLUSIONS}
In this paper, we proposed a robust, real-time event-based stereo VIO, ESVIO, which tightly fuses the stereo event streams, stereo image frames, and IMU measurements using sliding windows graph-based optimization. 
Geometry-based temporal tracking and instantaneous matching between consecutive stereo event streams are designed to ensure robust state estimation.
In addition, the motion compensation approach corrects the curved event streams with IMU and ESVIO back-end to emphasize the contour of scenes.
Extensive evaluations demonstrate that our ESVIO achieves superior performance compared to other state-of-the-art algorithms on public datasets and our self-collected challenging datasets.
Furthermore, we also perform various onboard closed-loop flights using the proposed ESVIO under low-light scenes and aggressive motion.
In our future work, we might further explore the event-based mapping for the quadrotor system which should be able to support autonomous navigation and obstacle avoidance.

\vspace{-1.0em}
\bibliographystyle{IEEEtran}
