\section{Introduction}

3D object detection is a fundamental problem in computer vision, multimedia and many applications, including autonomous driving~\cite{geiger2012we, Sun_2020_CVPR}, augmented reality~\cite{park2008multiple} and robotics~\cite{oh2002development}. For example, detecting both static (e.g., poles, traffic lights) and dynamic objects (e.g., vehicles) in traffic scenes is important for autonomous driving. Similarly, AR (augmented reality) recognition applications attach a synthetic 3D model to a real-world object. There has been considerable progress in object detection over the last decade. Moreover, different methods have been proposed for various sensors, including monocular cameras, depth cameras, LiDAR, and radars ~\cite{qi2017pointnet,qi2017pointnet++,qi2018frustum,yang2019std,ye2020hvnet,yang20203dssd,shi2020pv,yin2020center}. 2D object detection deals with detecting objects from RGB images and videos, while 3D object detection utilizes point cloud-based representations obtained from LiDARs or other sensors. Moreover, it is known that point cloud data obtained from LiDAR sensors tends to be more accurate than RGB images and videos~\cite{caesar2020nuscenes}. Consequently, point clouds are being widely used for scene understanding in autonomous driving and AR. 


\begin{figure}[t]
    \centering
    \includegraphics[width=\linewidth]{LaTeX/ACMMM/figures/coverpic.pdf}
    \caption{M3 Transformers used in our approach: Left: Multi-representation transformers; Middle: Multi-scale transformers; Right: Mutual-relation transformers. We use these characteristics to present a novel 3D object detection architecture.}
    \label{fig:coverpic.p}
\vspace{-0.5cm}
\end{figure}

There is extensive work in object detection. Moreover, 
state-of-the-art methods for 3D object detection base on different networks~\cite{yin2020center,shi2020pv}. However, there are two key limitations:

\noindent{\textbf{Ineffective point cloud representations:}} The three major techniques used to process point clouds are based on voxels~\cite{zhou2018voxelnet,yan2018second}, raw point clouds~\cite{qi2017pointnet,qi2017pointnet++,qi2018frustum,shi2019pointrcnn,yang2019std,liu2019point, qin2020weakly}, and bird's-eye-view ~\cite{chen2017multi, ku2018joint, yang2018pixor}. Each representation has a unique advantage and it has been shown that combining these representations can result in terms of detection accuracy \cite{qi2018frustum,shi2020pv,chen2019fast}. However, fusing these representations is non-trivial. First, the architectures corresponding to the VoxelNets, the PointNets, and the 2D convolutional neural networks are  different. Moreover, raw point clouds need to be converted to voxels and pixels before techniques based on VoxelNets and 2D convolutional neural networks can be applied. The differences between the inputs of these three neural models can result in semantic gaps. Previous works~\cite{chen2019fast,shi2020pv} tend to use feature concatenation and attention modules to fuse multi-representation features. However, the correlation between features of different representations has not been addressed.


\noindent{\textbf{Insufficient modeling of multi-scale features:}} Fusing multi-scale feature maps is a well-known technique used for improving the detection performance of 2D object detection. In terms of 3D object detection, current approaches tend to use multi-scale feature pyramids~\cite{qi2017pointnet,qi2017pointnet++,yang20203dssd,lan2019modeling,lang2019pointpillars,ye2020hvnet}. However, fusing these multiple feature pyramids is non-trivial because the higher resolution and the larger receptive fields are conflicting \cite{lin2017feature, qi2017pointnet++}. Existing methods~\cite{qi2017pointnet++,ye2020hvnet,shi2020pv} fuse the multi-scale features using bi-linear down-sampling/up-sampling and concatenation. Although these approaches~\cite{qi2017pointnet++,ye2020hvnet,shi2020pv} can improve the accuracy by a large margin, there are many challenges with respect to the underlying fusion method  in terms of the correlation between feature maps of different scales.

A key issue in terms of designing a good approach for object detection is exploiting the correlation between different representations and the large size of the receptive fields.  Our approach is motivated by use of transformers~\cite{vaswani2017attention}, a form of neural network based on attention that has been used in natural language processing ~\cite{hu2020iterative,tan2019lxmert,su2019vl,lu2019vilbert}. Specifically, transformers use multi-head attention to narrow the semantic gap between different representations by adapting to informative features and eliminating noise.




Another key aspect of 3D object detection is to model the mutual relationships between different points in the point cloud data~\cite{qi2017pointnet,qi2017pointnet++,lan2019modeling,thomas2019kpconv}. Modeling mutual relationships can enhance the ability to recognize the fine-grained patterns and can generalize to complex scenes.
Prior works in 3D object detection have modeled these relationships using multi-layer perceptrons~\cite{qi2017pointnet,qi2017pointnet++,qi2018frustum,shi2020pv}, farthest point sampling layers~\cite{qi2017pointnet++,shi2019pointrcnn,shi2020pv}, max pooling layers~\cite{lan2019modeling,qi2017pointnet++,shi2019pointrcnn,shi2020pv}, and graph convolutional networks~\cite{wang2019dynamic}. However, a key challenge is to model these mutual relationships along  with fusing different representations and multi-scale features.














\subsection*{Main Results}
We  present \textsc{M3DeTR}, a novel two-stage architecture for 3D object detection for different applications. Given raw 3D point cloud data, our approach can localize static and dynamic obstacles with state-of-the-art accuracy. As illustrated in Figure~\ref{fig:coverpic.p}, the key components of our approach are M3 transformers, which are used to combine different feature representations. Conceptually, each of the M3 transformers are used for aggregating point cloud representations, multi-scale representations and mutual relationships among a subset of points in the point cloud data.

The novel components of our work include:

\begin{enumerate}
    \item \textsc{M3DeTR} is the first unified architecture for 3D object detection with transformers that accounts for multi-representation, multi-scale, mutual-relation models of point clouds in an end-to-end manner.

    \item \textsc{M3DeTR} is robust and insensitive with respect to the hyper-parameters of transformer architectures.
    We test multiple variants with different transformer blocks designs.
    We demonstrate improved performance of \textsc{M3DeTR}  regardless of hyper-parameters.

    \item Our unified architecture achieves state-of-the-art performance on KITTI 3D Object Detection Dataset~\cite{geiger2012we} and Waymo Open Dataset~\cite{Sun_2020_CVPR}. We outperform the previous state-of-the-art approaches by 2.86\% mAP for car class on the Waymo validation set and 1.48\% mAP for all classes on the Waymo test set.
\end{enumerate}









\section{Related Work}


\textbf{Multi-representation modeling.}
Existing techniques for modeling 3D point cloud data include bird-eye-view (BEV), volumetric, and point-wise representations. 
Generally, BEV-based approaches ~\cite{chen2017multi, ku2018joint, liang2018deep, liang2019multi} first project 3D point clouds into 2D BEV space and then adopt the standard 2D object detectors to generate 3D object proposals from projected 2D feature maps. To deal with the irregular format of input point clouds, voxel-based architectures~\cite{zhou2018voxelnet, yan2018second, lang2019pointpillars} use equally spaced 3D voxels to encode the point clouds such that the volumetric representation can be consumed by the region proposal network (RPN) ~\cite{ren2015faster}. Inspired by the PointNet/PointNet++ approach~\cite{qi2017pointnet, qi2017pointnet++}, which is invariant under transformation,~\cite{qi2018frustum, shi2019pointrcnn} extend this method to the task of 3D object detection and directly process the raw point clouds to infer 3D bounding boxes. However, these methods are typically limited due to either information loss or high computation cost. Recently, many approaches~\cite{yang2019std, liu2019point, tang2020searching, shi2020pv, yin2020center} have combined the advantages of speed (of voxel-based representation) and efficiency (of point-based representation) by fusing point-voxel features for 3D object prediction.




\textbf{Multi-scale modeling.} Modeling multi-scale features is an important procedure in deep learning-based computer vision~\cite{lin2017feature, cai2019guided, liu2019l2g, zhang2020feature, liao2020mmnet, zhu18b,qi2017pointnet++, hu2020you, wang2020infofocus} because it is able to enlarge the receptive field and increase resolution. In 3D representation, modeling multi-scale features is also popular and important. PointNet++~\cite{qi2017pointnet++} proposes the set abstraction module to model local features of a cluster of point clouds. To model multi-scale patterns of point clouds, they use 3 different sampling ranges and radii with 3 parallel PointNets and thus fuse the multi-scale. In 3D object detection,~\cite{hu2020you, wang2020infofocus, ye2020hvnet} adopt different detection heads with multi-scale feature maps to handle both large and small object classes. 

\textbf{Mutual-relation modeling.} 2D Convolutional Neural Networks \cite{He_2016_CVPR} are commonly used to process mutual relations in 2D images. As point clouds are scattered and lacking structure, passing information from one point to its neighbors is not trivial. PointNets~\cite{qi2017pointnet++} proposes the set abstraction module to model the local context by using the subsampling and grouping layers. After this well-known work, many convolution-like operators~\cite{thomas2019kpconv,lan2019modeling,wang2019dynamic} on point clouds have been proposed to model the local context and the mutual relation between points. Recently, transformers~\cite{zhao2020point,engel2020point,guo2020pct,pan20203d} have been introduced in PointNets to model mutual relation. However, those previous works mainly focus on the local and global contexts of point clouds by applying mutual-relation transformers on points. Instead, our approach not only models the mutual relation between points, but it also models multi-scale and multi-representation features of point clouds.
\begin{figure*}[t]
    \centering
    \includegraphics[width=\linewidth]{LaTeX/ACMMM/figures/arch.pdf}
    \caption{\textbf{An overview of our \textsc{M3DeTR} architecture: \textsc{M3DeTR} is a transformer based framework for object detection in a coarse-to-fine manner.} It consists of three parts. PointNets, VoxelNet, and 2D ConvNets modules enable individual multi-representation feature learning.  M3 Transformers enable inter-intra multi-representation, multi-scale, multi-location feature attention. With the Region Proposal Network (RPN), the initial box proposals are generated. R-CNN captures and refines region-wise feature representations from M3 transformer output to improve detection performance. }
    \label{fig:3MDetr_network}
    \vspace{-0.2cm}
\end{figure*}

\textbf{Transformers in computer vision.}
Inspired by their success in machine translation ~\cite{vaswani2017attention}, transformer-based architectures have recently become effective and popular in a variety of computer vision tasks. Particularly, the design of self-attention and cross-attention mechanisms in transformers has been successful in modeling dependencies and learning richer information.~\cite{dosovitskiy2020image} leverages the direct application of transformers on the image recognition task without using convolution.~\cite{carion2020end, zhu2020deformable} apply transformers to eliminate the need for many handcrafted components in conventional object detection and achieve impressive detection results. ~\cite{yang2020learning, zeng2020learning} explore transformers on image and video synthesis tasks.~\cite{zhao2020point} investigates the self-attention networks on 3D point cloud segmentation task. 

Recently, there have been several joint representation fusion research attempts. PointPainting~\cite{vora2020pointpainting} proposes a novel method that accepts both images and point clouds as inputs to do 3D object detection by appending 2D semantic segmentation labels to LiDAR points. In the visual question answering task,~\cite{hu2020iterative} jointly fuses and reasons over three different modality representations, including question words, visual objects, and text tokens to answer the question. ~\cite{mittal2020m3er} combines multimodal information to solve robust emotion recognition problems. Building on a multi-representation and multi-scale transformer, our proposed model addresses the voxel-wise, point-wise, and BEV-wise feature representation gap and enables effective cross-representation interactions with different levels of semantic features. Coupled with a point-wise mutual-relation transformer, our framework learns to capture deeper local-global structures and richer geometric relationships among point clouds.



\section{Our Approach}
\textsc{M3DeTR} takes point cloud data as input and generates 3D boxes for different object categories with state-of-the-art accuracy as shown in Figure~\ref{fig:3MDetr_network}. Our goal is to perform multi-representation, multi-scale, and mutual-relation fusion with transformers over a joint embedding space. Our method consists of three main steps:

\begin{enumerate}
    \item Generate feature embeddings for different point cloud representations using VoxelNet, PointNet, and 2D ConvNet.
   
    
    \item Fuse these embeddings using \textit{M3 transformer} that leverages multi-representation and multi-scale feature embedding and models mutual relationships between points.
    
    \item Perform 3D detection using detection heads network, including RPN and R-CNN stages.
\end{enumerate}


In the rest of this section, we discuss each step in detail in Sections~\ref{backbone}, ~\ref{m3transformer}, and~\ref{dethead}, respectively. We conclude by defining the loss functions used for training the M3 transformer in Section~\ref{loss}.

\subsection{Multi-Representation Feature Embeddings}
\label{backbone}
Our network processes the raw input point clouds $\bm{P} = \{\bm{p}_1\text{,} \bm{p}_2\text{, ...,} \bm{p}_n\}$ and encodes them into three different embedding spaces, namely, voxel-, point-, and BEV-based feature representations. We discuss the embedding process for each representation in detail.

\noindent\textbf{Voxels:}
Voxel-wise feature extraction is divided into two steps: (1) the voxelization layer, which is driven by VoxelNet~\cite{zhou2018voxelnet}, takes the raw input point clouds $\bm{P}$ and converts them into equally spaced 3D voxels $\bm{v} \in \mathbb{R}^{L \times W \times H}$; (2) voxel-wise features $\bm{f}^{voxel}$ at different scales are extracted with 3D sparse convolutions, which is visualized in \ref{fig:3MDetr_network}. We denote the size of each discretized voxel as $L \times W \times H \times C$, where $L, W, H$ indicate the length, width, and height of the voxel grid and $C$ represents the channel of the voxel features. We adopt the average of the point-wise features from all the points to represent the whole non-empty voxel feature. Normally, the input voxel feature is encoded as the sampled points' 3D coordinates and reflectance, while 0 is padded if the voxel is empty. After voxelization, the input feature is propagated through a series of $3 \times 3 \times 3$ sparse cubes, including four consecutive blocks of 3D sparse convolution with downsampled sizes of $1 \times$, $2 \times$, $4 \times$, $8 \times$, using convolution operations of stride 2. Specifically, each sparse convolutional block includes a 3D convolution layer followed by a LayerNorm layer and a ReLU layer. Unlike ~\cite{zhou2018voxelnet, yan2018second}, all four different scales of obtained 3D voxel CNN features, $\bm{f}^{voxel} = \{\bm{f}^{voxel\ 1\times},\  \bm{f}^{voxel\ 2\times},\  \bm{f}^{voxel\ 4\times},\  \bm{f}^{voxel\ 8\times} \}$ are passed to the transformer stage as shown in Figure~\ref{fig:3MDetr_network}. 

\noindent\textbf{Bird's-eye-view:}
2D ConvNets takes the $8 \times$ downsampled voxel-based feature map $\bm{f}^{voxel\ 8\times}$ from the 3D Voxel CNN branch and generates a corresponding BEV-representation embedding. To directly apply 2D convolution, we start by combining the z-dimension and the channel dimension of the input voxel features into a single dimension. The structure of 2D ConvNets includes two encoder-decoder blocks, where each block consists of 2D down-sampling convolutional layers to produce top-down features, as well as de-convolutional layers to upsample to the input feature size. Specifically, both encoder and decoder paths are composed of a number of 2D convolutional layers with the kernel size of $3 \times 3$ followed by a BatchNorm layer and a ReLU layer. The output BEV-based feature from 2D ConvNets, denoted as $\bm{f}^{bev} \in \mathbb{R}^{L_{bev}\times W_{bev}\times C_{bev}}$, was further converted into keypoint feature $\bm{F}^{bev}\in \mathbb{R}^{n\times c_{bev}}$ through bi-linear interpolation.


\noindent\textbf{Points:}
Typically, there are more than 10K raw points inside an entire point cloud scene. In order to cover the entire point set effectively without large memory consumption, we apply Furthest-Point-Sampling (FPS) algorithm to sample $n$ keypoints, denoted as $\bm{\hat{P}} \subset \bm{P}$. Adopted from PointNet++~\cite{qi2017pointnet++} and PV-RCNN~\cite{shi2020pv}, Set Abstraction and Voxel Set Abstraction (VSA) module take raw point coordinates $\bm{P}$ and the 3D voxel-based features $\bm{f}^{voxel}$, respectively, to generate keypoint features for $\bm{\hat{P}}$. In particular, the corresponding keypoint features from raw points $\bm{P}$ is $\bm{F}^{point} \in \mathbb{R}^{n\times c_{point}}$, and the corresponding keypoint features from voxels $\bm{f}^{voxel}$ are $\{\bm{F}^{voxel\ 1\times}$, $\bm{F}^{voxel\ 2\times}$, $\bm{F}^{voxel 4\times}$,  $\bm{F}^{voxel\ 8\times} \}$, where $\bm{F}^{i}\in \mathbb{R}^{n\times c_{i}}$.







\subsection{Multi-representation, Multi-scale, and Mutual-relation Transformer}
\label{m3transformer}
Once the three feature embedding sequences, $\bm{F}^{voxel}$, $\bm{F}^{point}$ and $\bm{F}^{bev}$ are generated, 
the feature embedding sequences to extract mutually enhanced and robust semantic features. With the multi-head self-attention mechanism of transformers, the embeddings among inter-points and intra-points are able to dynamically and intelligently attend to each other, generating final cross-representations, cross-scales, and cross-points descriptive feature representations as shown in Figure \ref{fig:3MDetr_network}. In the remainder of this section, we will briefly review transformers and the self-attention mechanism followed by discussing the multi-representation, multi-scale, and mutual-relation transformer layers. 



\noindent\textbf{Transformer basics.}
A transformer is a stacked encoder-decoder architecture relying on a self-attention mechanism to compute representations of its input and output~\cite{vaswani2017attention}. Each transformer layer consists of one Multi-head Self-attention (MHSA) module and a few linear layers, as well as normalization and activation layers.

Consider two input matrices $\bm{X}_l \in \mathbb{R}^{l\times d_{in}}$ and $\bm{X}_s \in \mathbb{R}^{s \times d_{in}}$, where $l$ and $s$ are the lengths of the input sequence of dimension $d_{in}$. The attention layer output is defined as: \vspace{-3.5 mm}

$$\textrm{Attention}(\bm{Q}, \bm{V}, \bm{K}) = \textrm{Softmax}(\frac{\bm{Q}\bm{K}^{T}}{\sqrt{d_{in}}})\bm{V} \in \mathbb{R}^{l \times d_{out}},$$ 

$\textrm{where}\  \bm{Q} = \bm{X}_l \bm{W}_q,\  \bm{K} = \bm{X}_l \bm{W}_k\  and\  \bm{V} = \bm{X}_s \bm{W}_v.$

The matrices $\bm{Q}$, $\bm{K}$ and $\bm{V}$ represent query, key and value respectively, obtained by projection matrices $\bm{W}_q, \bm{W}_k\in \mathbb{R}^{d_{in}\times d_{h}}$ and $\bm{W}_v\in \mathbb{R}^{d_{in}\times d_{out}}$, where $d_{h}$ is the hidden dimension, and $d_{out}$ is the output dimension. When two input matrices $\bm{X}_l$ and $\bm{X}_s$ represent the same feature maps, the attention module is usually referred to as self-attention. MHSA with $N$ heads and the input matrix $\bm{X}$ is defined as follows: \vspace{-3.5 mm}

\[\textrm{MultiHead}(\bm{Q}, \bm{K}, \bm{V}) = \textrm{Concatenate} [\textrm{head}_1, ..., \textrm{head}_N] \bm{W} ^O, \]

\noindent where $ \textrm{head}_i = \textrm{Attention}(\bm{QW}_{i}^{q}, \bm{KW}_{i}^{k}, \bm{VW}_{i}^{v})$ and $\bm{Q}, \bm{K}, \bm{V}$ are linear projections of X. The projection matrices $\bm{W}^{q}_i \in \mathbb{R}^{d_h\times d_{q}},\  \bm{W}^{k}_i \in \mathbb{R}^{d_h\times d_{k}},\  \bm{W}^{v}_i \in \mathbb{R}^{d_h\times d_{v}},\  \bm{W}^O \in \mathbb{R}^{d_v\times d_{out}}$ are learnable parameters in the network that correspond to each of the attention heads, where $d_k$, $d_v$ and $d_q$ are the hidden dimensions of each attention head for $\bm{K}$, $\bm{V}$, and $\bm{Q}$, respectively. 


Now, we present the proposed transformers used to capture the inter- and intra- interactions among input features. Specifically, we propose two stacked transformer encoder layers named M3 Transformers, as shown in Figure \ref{fig:3MDetr_transformer}: (1) the multi-representation and multi-scale transformer, and (2) the mutual-relation transformer.




\noindent\textbf{Multi-representation and multi-scale transformer layer.}
First, we focus on the intra-point representation fusion. For each individual point, the input sequence to this transformer layer is its several different corresponding point cloud features, including various scales and distinctive representations. The input sequence to the transformer is $\bm{F} = [\bm{F}^{voxel\ 1\times}$, $ \bm{F}^{voxel\ 2\times}$, $ \bm{F}^{voxel\ 4\times}$, 
$\bm{F}^{voxel\ 8\times}$, $\bm{F}^{point}$, $\bm{F}^{bev}]$, where each $\bm{F}^{i}\in \mathbb{R}^{n\times c_i}$. After explicitly modeling all element-wise interactions among those features for each point separately, the output from the block is the updated feature vectors after aggregating the information from all the input sequence $\bm{F}$.

The multi-representation and multi-scale transformer layer takes 6 different inputs: $\bm{F}^{point}$, $\bm{F}^{voxel 1\times}$, $\bm{F}^{voxel 2\times}$, $\bm{F}^{voxel 4\times}$, $\bm{F}^{voxel 8\times}$,  $\bm{F}^{bev}$. As different inputs may have different feature dimensions, we use the single-layer perceptron to apply feature reduction on the input features to align the feature dimensions of each feature embeddings. The outputs of the feature reduction layer are $\bm{\hat{F}} = [\bm{\hat{F}}^{voxel\ 1\times}$, $ \bm{\hat{F}}^{voxel\ 2\times}$, $ \bm{\hat{F}}^{voxel\ 4\times}$, $\bm{\hat{F}}^{voxel\ 8\times}$, $\bm{\hat{F}}^{point}$, $\bm{\hat{F}}^{bev}]$, where the output dimension of each feature is equivalent to $\hat{c}$.

After the feature reduction layer, the multi-representation and multi-scale transformer layer takes $\bm{\hat{F}}$ as inputs and generates self-attention features $\bm{T}^{voxel\ 1\times}$, $\bm{T}^{voxel\ 2\times}$,$\bm{T}^{voxel\ 4\times}$,$\bm{T}^{voxel\ 8\times}$, $\bm{T}^{point}$, $\bm{T}^{bev}$, which corresponds to $\bm{\hat{F}}^{voxel\ 1\times}$, $\bm{\hat{F}}^{voxel\ 2\times}$,$\bm{\hat{F}}^{voxel\ 4\times}$,$\bm{\hat{F}}^{voxel\ 8\times}$, $\bm{\hat{F}}^{point}$, $\bm{\hat{F}}^{bev}$. We visualize the multi-representation and multi-scale transformer in Figure \ref{fig:3MDetr_transformer}.



\noindent\textbf{Mutual-relation transformer layer.}
Inspired by~\cite{qi2017pointnet, qi2017pointnet++, pan20203d}, inter-points feature fusion within a spatial neighboring space is leveraged in the mutual-relation layer of the transformer. Our goal is to attend to and aggregate the neighboring information in an attention manner for each point with an enriched feature.

From the first transformer block output, we obtain aggregated features $\bm{T}$ of different scales and representations after the first transformer block. We concatenate channels along the point dimension and rearrange those points into a sequence as the input of the mutual-relation transformer. Let the learned concatenated feature $\bm{T} = 
\rm{concat}$ $\{\bm{T}^{voxel\ 1\times}$, $ \bm{T}^{voxel\ 2\times}$, $ \bm{F}^{voxel\ 4\times}$, $\bm{T}^{voxel\ 8\times}$, $\bm{T}^{point}$, $\bm{T}^{bev}\} \in\mathbb{R}^{n\times c_T}$, where $c_{T}$ is the sum of all channel size in $\bm{T}$.  Note that, the first dimension of $\bm{T}$ is the number of keypoints $n$. To model the mutual-relation between keypoints, we need to split $\bm{T}$ into $n$ point-wise feature $\bm{T} = \{ \bm{T}_i\}$, where $\bm{T}_i \in \mathbb{R}^{c_{T}}$. The concatenation and split module is showed in Figure \ref{fig:3MDetr_transformer}.

The mutual-relation transformer takes point-wise features of $n$ keypoints as inputs and uses the multi-head self-attention head to model the mutual relationship between keypoints. The outputs of the mutual-relation transformer are $\bm{\hat{T}}$.


\begin{figure}[t]
    \centering
    \includegraphics[width=0.85\linewidth]{figures/Transformer_block.pdf}
    \caption{\textbf{M3 Transformers} consist of two parts: a multi-representation and multi-scale transformer, and a mutual-relation transformer. Multi-representation and multi-scale transformer takes the different feature embedding and generates enriched cross-representations and cross-scales embedding. On top of it, the mutual-relation transformer further models point-wise feature relationship to extract the refined features.}
    \label{fig:3MDetr_transformer}
    \vspace{-0.3cm}
\end{figure}

\noindent\textbf{Comparison with previous point-based transformers}
Prior work has explored the transformer application on the task of point cloud processing. First, \cite{guo2020pct, engel2020point, zhao2020point} leverage the inherent permutation invariance of transformers to capture local context within the point cloud on the shape classification and segmentation tasks, while our \textsc{M3DeTR} mainly investigates the strong attention ability of transformers between input embeddings for the 3D object detection task. Pointformer \cite{pan20203d} is the approach most related to our method because we both address the 3D object detection task by capturing the dependencies among points' features. However, Pointformer adopts a single PointNet branch to extract points feature, while \textsc{M3DeTR} considers all three different representations and also applies the transformer to learn aggregated representation-based features.


\subsection{Detection Heads Network}
\label{dethead}
After we obtain the enriched embedding $\bm{\hat{T}}$ from the M3 transformer, the detection network is composed of two stages that predict the 3D bounding box class, localization, and orientation in a coarse-to-fine manner, including RPN and R-CNN. Please refer to PV-RCNN~\cite{shi2020pv} for more details.



\noindent\textbf{RPN:} Region Proposal Networks take the deep semantic features $\bm{f}^{bev}$ produced by the 2D ConvNets as inputs and generate high-quality 3D object proposals $\bm{B}$, as shown in Figure \ref{fig:3MDetr_network}. A 3D object box $\bm{B}_i$ is parameterized as $(x, y, z, l, h, w, \theta) $, where $(x, y, z)$ is the center of the box, $(l , h, w)$ is the dimension of the box, and $\theta$ is the orientation in bird's-eye-view. Similar to the conventional RPN in 2D object detection~\cite{ren2015faster} each position on the deep feature map is placed by predefined bounding box anchors denoted as ($x^a$, $y^a$, $z^a$, $l^a$, $h^a$, $w^a$, $\theta^a$). In addition, each anchor box is assigned a one-hot vector of classification targets. Then initial proposals are generated by predicting the relative offset of an object's 3D ground truth bounding box ($x^{gt}$, $y^{gt}$, $z^{gt}$, $l^{gt}$, $h^{gt}$, $w^{gt}$, $\theta^{gt}$). The specific outputs of RPN are the candidate proposals with object classification labels, bounding box regression, and orientation prediction. 



\noindent\textbf{R-CNN:}
R-CNN serves as the second stage of the detection network and it takes the initial region proposals $\bm{B}$ from RPN as input to conduct further proposal refinement. For each input proposal box, the RoI-grid pooling module~\cite{shi2020pv} is adopted to extract the corresponding proposal-specific grid points' features from the transformer-based embeddings $\bm{\hat{T}}$. Compared with previous works, \textsc{M3DeTR} leverages the richer embedding information from the learned transformers for the fine-grained proposal refinement. Initially, the RoI-grid pooling module was proposed to aggregate the keypoint features to the RoI-grid points. It takes 3D region proposals (RoI) as inputs, and the output is a list of a fixed number of grid points' feature vectors. As the main component to extract refined features, the RoI-grid pooling module uniformly samples $ N \times N \times N $ grid points per 3D proposal. For each grid point, the output feature is generated by applying a PointNet-block \cite{qi2017pointnet} on a small number of surrounding keypoints, $M$, within its spatial surrounding region with a radius of $r$. Specifically, keypoints are the subset of input points that are sampled using the Furthest-point-sampling algorithm to cover the entire point set. 

\begin{table*}[ht]
    \centering
   
\input{LaTeX/ACMMM/table/waymo_test_all_classes}
\end{table*}

\begin{table*}[ht]
    \centering
   
\input{table/waymo_test}
\end{table*}

Finally, the refined representations of each proposal are first passed to two fully connected layers and then generate the box 3D Intersection-over-Union (IoU) guided confidence scoring of class prediction and location refinement of regression targets, $\bm{\hat{B}}$. Compared with the traditional classification-guided box scoring, 3D IoU guided confidence scoring considers the IoU between the proposal box and its corresponding ground truth box. Empirically, \cite{shi2020points,shi2020pv} show that it achieves better results compared with the traditional classification confidence based techniques. \vspace{-2 mm}

\subsection{Loss Functions} 
\label{loss}
In this section, we define the loss function used in our approach.
The bounding box regression target for both RPN and R-CNN stages is calculated as the relative offsets between the anchors and the ground truth as:
$\triangle x = \frac{x^{gt} - x^a}{d^a},  \triangle y = \frac{y^{gt} - y^a}{d^a}$,  
$\triangle z = \frac{z^{gt} - z^a}{h^a}$
$\triangle h = \log (\frac{h^{gt}}{{h^{a}}})$,  
$\triangle w = \log (\frac{w^{gt}}{{w^{a}}})$,
$\triangle \theta = \theta^{gt} - \theta^{a}$,
 where $ d^a $ = $\sqrt{(w^a)^2 + (l^a)^2} $. Similar to~\cite{shi2020pv, shi2020points}, the focal loss is applied \cite{lin2017focal} for the classification loss, $\mathcal{L}_{cls}$. Smooth L1 loss \cite{girshick2014rich} is adopted for the box localization regression target's losses, $\mathcal{L}_{reg}$ and $\mathcal{L}_{ref}$. In addition, 3D IoU loss \cite{shi2020pv, shi2020points} is used for $\mathcal{L}_{iou}$. Please refer to PV-RCNN~\cite{shi2020pv} for the details about $\mathcal{L}_{iou}$.

Similar to PV-RCNN \cite{shi2020pv}, we formally define a multi-task loss for both the RPN and R-CNN stages, 

\begin{equation}
\begin{aligned}
 & \mathcal{L}_{cls} = -\alpha_{a}(1 - p^{a})^\gamma \log p^{a}, \\
 & \mathcal{L}_{reg} = \sum_{b \in (x, y, z, w, l, h, \theta)} \mathcal{L}_s (\triangle b), \\
 & \mathcal{L} = \mathcal{L}_{cls} + \beta_{reg} \mathcal{L}_{reg} + \beta_{iou} \mathcal{L}_{iou} + \beta_{ref} \mathcal{L}_{ref},
\end{aligned}
\end{equation}

where $p^{a}$ is the model's estimated class probability for an anchor box, and $\beta_{reg}$, $\beta_{iou} $ and $\beta_{ref}$ are chosen to balance the weights between classification loss, IoU loss and regression loss for RPN stage and R-CNN stage. We adopt the default $\alpha = 0.25$, and $\gamma = 2.0$ from the parameters of focal loss~\cite{lin2017focal}.


\section{Experiments}
In this section, we evaluate \textsc{M3DeTR} both qualitatively and quantitatively on the Waymo Open Dataset~\cite{Sun_2020_CVPR} and the KITTI Dataset~\cite{geiger2012we} in the task of LiDAR-based 3D object detection. Our main results include achieving a state-of-the-art accuracy on these datasets and robustness to hyper-parameter tuning.

We begin by giving a brief overview of the datasets and evaluation metric in Section~\ref{dataset} followed by presenting the implementation details in Section~\ref{impl}. We present our main results on the Waymo Open Dataset and the KITTI Dataset in Section~\ref{result}. Finally, we present a series of ablations to demonstrate the effectiveness of each component of our approach in Section~\ref{ablation} and conclude by demonstrating the robustness of our method in Section~\ref{robust}. 

\subsection{Datasets and Evaluation Metric}
\label{dataset}
\noindent \textbf{Waymo Open Dataset:}
The Waymo Open Dataset~\cite{Sun_2020_CVPR} is a large-scale autonomous driving dataset containing $1000$ scenes of $20$s duration each, with $798$ scenes for training and $202$ scenes for validation. Each scene is sampled at a frequency of $10$Hz. Overall, the dataset includes $12$M labeled objects and thus we only use one fifth of the training scenes for the following experiment. We consider LiDAR data as the input to our approach. The evaluation protocol on the Waymo dataset consists of the mean average precision (mAP) and mean average precision weighted by heading (mAPH). For each object category, the detection outcomes are evaluated based on two difficulty levels: LEVEL\_1 denotes the annotated bounding box with more than 5 points and LEVEL\_2 represents the annotated bounding box with more than 1 point.

\textbf{KITTI dataset}
The KITTI 3D object detection benchmark~\cite{geiger2012we} is another popular dataset for autonomous driving. It contains $7,481$ training and $7,518$ testing LiDAR scans. We follow the standard split on the training ($3,712$ samples) and validation sets ($3,769$ samples). For each object category, the detection outcomes are evaluated based on three difficulty levels based on the object size, occlusion state, and truncation level. \vspace{-2.5 mm}

\begin{table*}[t]
    \centering
\input{LaTeX/ACMMM/table/waymo_val}
\end{table*}

\subsection{Implementation Details}\label{impl}
Based on~\cite{shi2020pv}, we use PyTorch to implement our M3 transformer modules and integrate them into the PV-RCNN network. 

\noindent \textbf{Backbone:}
The 3D voxel CNN branch consists of 4 blocks of 3D sparse convolutions with output feature channel dimensions of 16, 32, 64, 64. Those 4 different voxel representations of different scales, as well as point features from point cloud input, are used to refine keypoint features by PointNet~\cite{qi2017pointnet++} through set abstraction and voxel set abstraction~\cite{shi2020pv}. The number of sampled keypoints $n$ is 2,048 for both Waymo and KITTI. In order to sample the keypoints effectively and accurately, we uses FPS on points within the range of top 1000 or 1500 initial proposals with a radius $r$ of 2.4.

For the Waymo Open Dataset, the voxel size is [0.1, 0.1, 0.15], and we focus on the input LiDAR point cloud range with [-75.2, 75.2], [-75.2, 75.2], and [-2, 4] meters in x, y, and z axis, respectively. The 2D ConvNets output size is $188 \times 188 \times 512$.

For the KITTI dataset, the voxel size is [0.05, 0.05, 0.1], and we focus on the input LiDAR point cloud range with [0, 70.4], [-40, 40], and [-3, 1] meters in x, y, and z axis, respectively.  The 2D ConvNets output size is $200 \times 176 \times 512$.

\noindent\textbf{M3 Transformer}
We project the embeddings obtained from the backbone network with different scales and representations to 256 channels, as the input of the multi-representation and multi-scale transformer requires, and project the output features back to their original dimensions before passing into the mutual-relation transformer. Due to the GPU memory constraint, we experiment with two types of MHSA module designs: 2 encoder layers with 4 attention heads and 1 encoder layer with 8 attention heads. 

\noindent\textbf{Detection Heads}
The RPN anchor size for each object category is set by computing the average of the corresponding objects from the annotated training set. Uniformly, the RoI-grid pooling module samples $ 6 \times 6 \times 6$ grid points within each initial 3D proposal to form a refined features. Specifically, the number of surrounding points used to extract the grid point's feature, M, is 16.
During the training phase, 512 proposals are generated from RPN to R-CNN, where non-maximum suppression (NMS) with a threshold of 0.8 is applied to remove the overlapping proposals. In the validation phase, 100 proposals are fed into R-CNN, where the NMS threshold is 0.7.


\begin{figure*}[t]
    \centering
    \includegraphics[width=1\linewidth]{LaTeX/ACMMM/figures/Visualization_4.pdf}
    \caption{We highlight the 3D detection results of \textbf{M3DeTR} on the Waymo Open Dataset. The 3D ground truth bounding boxes are in green, while the detection bounding box are shown in blue.}
    \label{fig:visualization}
\end{figure*}

\noindent\textbf{Training Parameters}
Models are trained from scratch on $4$ NVIDIA P6000 GPUs. We use the Adam optimizer with a fixed weight decay of $0.01$ and use a one-cycle scheduler proposed in~\cite{smith2018disciplined}. For the Waymo Open Dataset, we train our models for $45$ epochs with a batch size of $8$ scenes and a learning rate $0.01$, which takes around $50$ hours. For the KITTI dataset, we train our models for $80$ epochs with a batch size of $8$ scenes per and a learning rate $0.01$, which takes around $15$ hours. 


\begin{table}[t]
    \centering
    \setlength\tabcolsep{3px}
\input{LaTeX/ACMMM/table/kitti_test_car_cyclist}
\vspace{-0.2cm}
\end{table}



\subsection{Results}
\label{result}
\noindent\textbf{Waymo Open Dataset:}
We first present our object detection results for the vehicle, pedestrian, and cyclist classes on the test set of Waymo Open Dataset in Table~\ref{tab:waymo_test_all_classes} compared with PV-RCNN~\cite{shi2020pv}. We evaluate our method at both LEVEL\_1 and LEVEL\_2 difficulty levels. Note that we reproduce the baseline PV-RCNN with a single frame input since they recently adopt two-frames input on the test set. As we can see, PV-RCNN achieves 69.57\% and 63.65\% on average in LEVEL\_1 mAP and LEVEL\_2 mAP, respectively, while \textsc{M3DeTR} improves them by 1.48\% and 1.85\%, respectively. Without bells and whistles, our approach works better than PV-RCNN~\cite{shi2020pv}. Furthermore, we compare our framework on the vehicle class for different distances with state-of-the-art methods, including StarNet~\cite{ngiam2019starnet}, PointPillars~\cite{lang2019pointpillars}, RCD~\cite{bewley2020range}, Det3D~\cite{zhu2019class}, RangeDet~\cite{fan2021rangedet} and PV-RCNN~\cite{shi2020pv}. As shown in Table~\ref{tab:waymo_test}, \textsc{M3DeTR} outperforms PV-RCNN significantly in both LEVEL\_1 and LEVEL\_2 difficulty levels across all distances, demonstrating the effectiveness of the newly proposed framework. Moreover, we visualize the detection results of \textsc{M3DeTR} in Figure ~\ref{fig:visualization}. 

Compared with the PV-RCNN shown in Figure~\ref{fig:visualization_comparison}, \textsc{M3DeTR} successfully captures the inter- and intra- interactions among input features and effectively helps the model generate high-quality box proposals. To the best of our knowledge, \textsc{M3DeTR} achieves the state-of-the-art in the Vehicle class in both LEVEL\_1 and LEVEL\_2 difficulty levels among all the published papers with a single frame LiDAR input.

We also evaluate the overall object detection performance with an IoU of 0.7 for the Vehicle class on the full Waymo Open Dataset validation set as in Table~\ref{tab:waymo_val}, further proving that our architecture is more efficient for jointly modeling the input features.

\noindent\textbf{KITTI dataset:}
We also compare our approach with the state-of-the-art methods on the KITTI test set~\cite{shi2020pv,chen2020object}. We compute the mAP on three difficult types of both car and  cyclist classes in 3D detection metric. Table~\ref{tab:kitti_test_car_cyclist} shows that \textsc{M3DeTR} achieves state-of-the-art performance and outperforms the previous work by a large margin especially on the cyclist class. In particular, HotSpotNet~\cite{chen2020object} achieves $82.59\%$ in the ``easy'' categories of 3D detection metric, while \textsc{M3DeTR} improves these results by significant $1.24\%$.  \vspace{-2 mm}

\begin{table}[t]
    \centering
    \setlength\tabcolsep{3px}
\input{LaTeX/ACMMM/table/kitti_ablation_transformer}
\vspace{-0.5cm}
\end{table}

\subsection{Ablation Studies}
\label{ablation}
To demonstrate the individual benefits of the multi-representation, multi-scale, and mutual-relation layers of the M3 transformer, we perform ablation experiments and tabulate the results in Table~\ref{tab:kitti_ablation_transformer}. All experiments are conducted on the validation set of KITTI dataset.
With the single multi-representation and multi-scale transformer layer, we can achieve 4.07\% and 1.71\% on the moderate difficulty in car class with 11 and 40 recall positions, respectively compared with the PV-RCNN baseline. On the other side, with the single mutual-relation transformer layer, the performance gain are 4.47\% and 1.99\% compared with the PV-RCNN baseline. Without hyper-parameter tuning, \textsc{M3DeTR} benefits from unifying multiple point cloud representations, feature scales, and model mutual-relations simultaneously which results in the best performance.  \vspace{-2 mm}











\subsection{Robustness of \textsc{M3DeTR}}
\label{robust}
To demonstrate the robustness of \textsc{M3DeTR} to hyper-parameter tuning, we perform a series of tests by varying the sampling size, number of detection heads, and number of transformer encoder layers. We present the results of these tests in Figure~\ref{fig:M3DeTR_robustness}, where we observe that \textsc{M3DeTR} performs consistently well for the ``car'' category with IoU threshold of 0.7 for both 11 and 40 recall positions on the KITTI validation set.

\begin{figure}[t]
    \centering
   
    \centerline{
    \includegraphics[width=1\linewidth]{LaTeX/ACMMM/figures/Comparison_pvrcnn_M3DeTR_wo.pdf}
    }
    \caption{We visualize the 3D detection results for the same input point cloud between PV-RCNN (left) and \textbf{M3DeTR} (right) on Waymo Open Dataset. We highlight the false negative boxes from PV-RCNN in red.}
   
    \label{fig:visualization_comparison}
\end{figure}


\begin{figure}[t]
    \centering
    \includegraphics[width=0.95\linewidth]{LaTeX/ACMMM/figures/M3DeTR_Robustness.pdf}
    \caption{Performance comparison of different \textbf{M3 Transformers} variants in Car class on KITTI validation set. We show the mAP results with IoU threshold of 0.7 for both 11 and 40 recall positions.  Note that "l" and "h" represent layer number and head dimension of M3 Transformers, respectively. "Top" denotes the number of proposals used for keypoint sampling from RPN stage.}
    \label{fig:M3DeTR_robustness}
    \vspace{-0.5cm}
\end{figure}
 
\section{Conclusions, Limitations and Future Works}
In this paper, we present \textsc{M3DeTR}, a novel transformer-based framework for object detection with LiDAR point clouds. \textsc{M3DeTR} is designed to simultaneously model multi-representation, multi-scale, mutual-relation features through the proposed M3 Transformers. Overall, the first transformer integrates features with different scales and representations, and the second transformer aggregates information from all keypoints. Experimental results show that \textsc{M3DeTR} outperforms previous work by a large margin on the Waymo Open Dataset and the KITTI dataset. Without bells and whistles, \textsc{M3DeTR} is demonstrated to be invariant to the hyper-parameters of transformer.

However, constrained by the hardware resources, we are unable to train and validate \textsc{M3DeTR} with the full training set from the large-scale Waymo Open Dataset. As part of the future work, designing a completely different architecture that leverages the attention mechanism for information fusion would be an interesting direction.  


\bibliographystyle{ACM-Reference-Format}

\section{Introduction}
ACM's consolidated article template, introduced in 2017, provides a
consistent \LaTeX\ style for use across ACM publications, and
incorporates accessibility and metadata-extraction functionality
necessary for future Digital Library endeavors. Numerous ACM and
SIG-specific \LaTeX\ templates have been examined, and their unique
features incorporated into this single new template.

If you are new to publishing with ACM, this document is a valuable
guide to the process of preparing your work for publication. If you
have published with ACM before, this document provides insight and
instruction into more recent changes to the article template.

The ``\verb|acmart|'' document class can be used to prepare articles
for any ACM publication --- conference or journal, and for any stage
of publication, from review to final ``camera-ready'' copy, to the
author's own version, with {\itshape very} few changes to the source.

\section{Template Overview}
As noted in the introduction, the ``\verb|acmart|'' document class can
be used to prepare many different kinds of documentation --- a
double-blind initial submission of a full-length technical paper, a
two-page SIGGRAPH Emerging Technologies abstract, a ``camera-ready''
journal article, a SIGCHI Extended Abstract, and more --- all by
selecting the appropriate {\itshape template style} and {\itshape
  template parameters}.

This document will explain the major features of the document
class. For further information, the {\itshape \LaTeX\ User's Guide} is
available from
\url{https://www.acm.org/publications/proceedings-template}.

\subsection{Template Styles}

The primary parameter given to the ``\verb|acmart|'' document class is
the {\itshape template style} which corresponds to the kind of publication
or SIG publishing the work. This parameter is enclosed in square
brackets and is a part of the {\verb|documentclass|} command:
\begin{verbatim}
  \documentclass[STYLE]{acmart}
\end{verbatim}

Journals use one of three template styles. All but three ACM journals
use the {\verb|acmsmall|} template style:
\begin{itemize}
\item {\verb|acmsmall|}: The default journal template style.
\item {\verb|acmlarge|}: Used by JOCCH and TAP.
\item {\verb|acmtog|}: Used by TOG.
\end{itemize}

The majority of conference proceedings documentation will use the {\verb|acmconf|} template style.
\begin{itemize}
\item {\verb|acmconf|}: The default proceedings template style.
\item{\verb|sigchi|}: Used for SIGCHI conference articles.
\item{\verb|sigchi-a|}: Used for SIGCHI ``Extended Abstract'' articles.
\item{\verb|sigplan|}: Used for SIGPLAN conference articles.
\end{itemize}

\subsection{Template Parameters}

In addition to specifying the {\itshape template style} to be used in
formatting your work, there are a number of {\itshape template parameters}
which modify some part of the applied template style. A complete list
of these parameters can be found in the {\itshape \LaTeX\ User's Guide.}

Frequently-used parameters, or combinations of parameters, include:
\begin{itemize}
\item {\verb|anonymous,review|}: Suitable for a ``double-blind''
  conference submission. Anonymizes the work and includes line
  numbers. Use with the \verb|\acmSubmissionID| command to print the
  submission's unique ID on each page of the work.
\item{\verb|authorversion|}: Produces a version of the work suitable
  for posting by the author.
\item{\verb|screen|}: Produces colored hyperlinks.
\end{itemize}

This document uses the following string as the first command in the
source file:
\begin{verbatim}
\documentclass[sigconf]{acmart}
\end{verbatim}

\section{Modifications}

Modifying the template --- including but not limited to: adjusting
margins, typeface sizes, line spacing, paragraph and list definitions,
and the use of the \verb|\vspace| command to manually adjust the
vertical spacing between elements of your work --- is not allowed.

{\bfseries Your document will be returned to you for revision if
  modifications are discovered.}

\section{Typefaces}

The ``\verb|acmart|'' document class requires the use of the
``Libertine'' typeface family. Your \TeX\ installation should include
this set of packages. Please do not substitute other typefaces. The
``\verb|lmodern|'' and ``\verb|ltimes|'' packages should not be used,
as they will override the built-in typeface families.

\section{Title Information}

The title of your work should use capital letters appropriately -
\url{https://capitalizemytitle.com/} has useful rules for
capitalization. Use the {\verb|title|} command to define the title of
your work. If your work has a subtitle, define it with the
{\verb|subtitle|} command.  Do not insert line breaks in your title.

If your title is lengthy, you must define a short version to be used
in the page headers, to prevent overlapping text. The \verb|title|
command has a ``short title'' parameter:
\begin{verbatim}
  \title[short title]{full title}
\end{verbatim}

\section{Authors and Affiliations}

Each author must be defined separately for accurate metadata
identification. Multiple authors may share one affiliation. Authors'
names should not be abbreviated; use full first names wherever
possible. Include authors' e-mail addresses whenever possible.

Grouping authors' names or e-mail addresses, or providing an ``e-mail
alias,'' as shown below, is not acceptable:
\begin{verbatim}
  \author{Brooke Aster, David Mehldau}
  \email{dave,judy,steve@university.edu}
  \email{firstname.lastname@phillips.org}
\end{verbatim}

The \verb|authornote| and \verb|authornotemark| commands allow a note
to apply to multiple authors --- for example, if the first two authors
of an article contributed equally to the work.

If your author list is lengthy, you must define a shortened version of
the list of authors to be used in the page headers, to prevent
overlapping text. The following command should be placed just after
the last \verb|\author{}| definition:
\begin{verbatim}
  \renewcommand{\shortauthors}{McCartney, et al.}
\end{verbatim}
Omitting this command will force the use of a concatenated list of all
of the authors' names, which may result in overlapping text in the
page headers.

The article template's documentation, available at
\url{https://www.acm.org/publications/proceedings-template}, has a
complete explanation of these commands and tips for their effective
use.

Note that authors' addresses are mandatory for journal articles.

\section{Rights Information}

Authors of any work published by ACM will need to complete a rights
form. Depending on the kind of work, and the rights management choice
made by the author, this may be copyright transfer, permission,
license, or an OA (open access) agreement.

Regardless of the rights management choice, the author will receive a
copy of the completed rights form once it has been submitted. This
form contains \LaTeX\ commands that must be copied into the source
document. When the document source is compiled, these commands and
their parameters add formatted text to several areas of the final
document:
\begin{itemize}
\item the ``ACM Reference Format'' text on the first page.
\item the ``rights management'' text on the first page.
\item the conference information in the page header(s).
\end{itemize}

Rights information is unique to the work; if you are preparing several
works for an event, make sure to use the correct set of commands with
each of the works.

The ACM Reference Format text is required for all articles over one
page in length, and is optional for one-page articles (abstracts).

\section{CCS Concepts and User-Defined Keywords}

Two elements of the ``acmart'' document class provide powerful
taxonomic tools for you to help readers find your work in an online
search.

The ACM Computing Classification System ---
\url{https://www.acm.org/publications/class-2012} --- is a set of
classifiers and concepts that describe the computing
discipline. Authors can select entries from this classification
system, via \url{https://dl.acm.org/ccs/ccs.cfm}, and generate the
commands to be included in the \LaTeX\ source.

User-defined keywords are a comma-separated list of words and phrases
of the authors' choosing, providing a more flexible way of describing
the research being presented.

CCS concepts and user-defined keywords are required for for all
articles over two pages in length, and are optional for one- and
two-page articles (or abstracts).

\section{Sectioning Commands}

Your work should use standard \LaTeX\ sectioning commands:
\verb|section|, \verb|subsection|, \verb|subsubsection|, and
\verb|paragraph|. They should be numbered; do not remove the numbering
from the commands.

Simulating a sectioning command by setting the first word or words of
a paragraph in boldface or italicized text is {\bfseries not allowed.}

\section{Tables}

The ``\verb|acmart|'' document class includes the ``\verb|booktabs|''
package --- \url{https://ctan.org/pkg/booktabs} --- for preparing
high-quality tables.

Table captions are placed {\itshape above} the table.

Because tables cannot be split across pages, the best placement for
them is typically the top of the page nearest their initial cite.  To
ensure this proper ``floating'' placement of tables, use the
environment \textbf{table} to enclose the table's contents and the
table caption.  The contents of the table itself must go in the
\textbf{tabular} environment, to be aligned properly in rows and
columns, with the desired horizontal and vertical rules.  Again,
detailed instructions on \textbf{tabular} material are found in the
\textit{\LaTeX\ User's Guide}.

Immediately following this sentence is the point at which
Table~\ref{tab:freq} is included in the input file; compare the
placement of the table here with the table in the printed output of
this document.

\begin{table}
  \caption{Frequency of Special Characters}
  \label{tab:freq}
  \begin{tabular}{ccl}
    \toprule
    Non-English or Math&Frequency&Comments\\
    \midrule
    \O & 1 in 1,000& For Swedish names\\
    $\pi$ & 1 in 5& Common in math\\
    \$ & 4 in 5 & Used in business\\
    $\Psi^2_1$ & 1 in 40,000& Unexplained usage\\
  \bottomrule
\end{tabular}
\end{table}

To set a wider table, which takes up the whole width of the page's
live area, use the environment \textbf{table*} to enclose the table's
contents and the table caption.  As with a single-column table, this
wide table will ``float'' to a location deemed more
desirable. Immediately following this sentence is the point at which
Table~\ref{tab:commands} is included in the input file; again, it is
instructive to compare the placement of the table here with the table
in the printed output of this document.

\begin{table*}
  \caption{Some Typical Commands}
  \label{tab:commands}
  \begin{tabular}{ccl}
    \toprule
    Command &A Number & Comments\\
    \midrule
    \texttt{{\char'134}author} & 100& Author \\
    \texttt{{\char'134}table}& 300 & For tables\\
    \texttt{{\char'134}table*}& 400& For wider tables\\
    \bottomrule
  \end{tabular}
\end{table*}

Always use midrule to separate table header rows from data rows, and
use it only for this purpose. This enables assistive technologies to
recognise table headers and support their users in navigating tables
more easily.

\section{Math Equations}
You may want to display math equations in three distinct styles:
inline, numbered or non-numbered display.  Each of the three are
discussed in the next sections.

\subsection{Inline (In-text) Equations}
A formula that appears in the running text is called an inline or
in-text formula.  It is produced by the \textbf{math} environment,
which can be invoked with the usual
\texttt{{\char'134}begin\,\ldots{\char'134}end} construction or with
the short form \texttt{\$\,\ldots\$}. You can use any of the symbols
and structures, from $\alpha$ to $\omega$, available in
\LaTeX~\cite{Lamport:LaTeX}; this section will simply show a few
examples of in-text equations in context. Notice how this equation:
\begin{math}
  \lim_{n\rightarrow \infty}x=0
\end{math},
set here in in-line math style, looks slightly different when
set in display style.  (See next section).

\subsection{Display Equations}
A numbered display equation---one set off by vertical space from the
text and centered horizontally---is produced by the \textbf{equation}
environment. An unnumbered display equation is produced by the
\textbf{displaymath} environment.

Again, in either environment, you can use any of the symbols and
structures available in \LaTeX\@; this section will just give a couple
of examples of display equations in context.  First, consider the
equation, shown as an inline equation above:
\begin{equation}
  \lim_{n\rightarrow \infty}x=0
\end{equation}
Notice how it is formatted somewhat differently in
the \textbf{displaymath}
environment.  Now, we'll enter an unnumbered equation:
\begin{displaymath}
  \sum_{i=0}^{\infty} x + 1
\end{displaymath}
and follow it with another numbered equation:
\begin{equation}
  \sum_{i=0}^{\infty}x_i=\int_{0}^{\pi+2} f
\end{equation}
just to demonstrate \LaTeX's able handling of numbering.

\section{Figures}

The ``\verb|figure|'' environment should be used for figures. One or
more images can be placed within a figure. If your figure contains
third-party material, you must clearly identify it as such, as shown
in the example below.
\begin{figure}[h]
  \centering
  \includegraphics[width=\linewidth]{sample-franklin}
  \caption{1907 Franklin Model D roadster. Photograph by Harris \&
    Ewing, Inc. [Public domain], via Wikimedia
    Commons. (\url{https://goo.gl/VLCRBB}).}
  \Description{A woman and a girl in white dresses sit in an open car.}
\end{figure}

Your figures should contain a caption which describes the figure to
the reader.

Figure captions are placed {\itshape below} the figure.

Every figure should also have a figure description unless it is purely
decorative. These descriptions convey what’s in the image to someone
who cannot see it. They are also used by search engine crawlers for
indexing images, and when images cannot be loaded.

A figure description must be unformatted plain text less than 2000
characters long (including spaces).  {\bfseries Figure descriptions
  should not repeat the figure caption – their purpose is to capture
  important information that is not already provided in the caption or
  the main text of the paper.} For figures that convey important and
complex new information, a short text description may not be
adequate. More complex alternative descriptions can be placed in an
appendix and referenced in a short figure description. For example,
provide a data table capturing the information in a bar chart, or a
structured list representing a graph.  For additional information
regarding how best to write figure descriptions and why doing this is
so important, please see
\url{https://www.acm.org/publications/taps/describing-figures/}.

\subsection{The ``Teaser Figure''}

A ``teaser figure'' is an image, or set of images in one figure, that
are placed after all author and affiliation information, and before
the body of the article, spanning the page. If you wish to have such a
figure in your article, place the command immediately before the
\verb|\maketitle| command:
\begin{verbatim}
  \begin{teaserfigure}
    \includegraphics[width=\textwidth]{sampleteaser}
    \caption{figure caption}
    \Description{figure description}
  \end{teaserfigure}
\end{verbatim}

\section{Citations and Bibliographies}

The use of \BibTeX\ for the preparation and formatting of one's
references is strongly recommended. Authors' names should be complete
--- use full first names (``Donald E. Knuth'') not initials
(``D. E. Knuth'') --- and the salient identifying features of a
reference should be included: title, year, volume, number, pages,
article DOI, etc.

The bibliography is included in your source document with these two
commands, placed just before the \verb|\end{document}| command:
\begin{verbatim}
  \bibliographystyle{ACM-Reference-Format}
  
\section{Additional Ablation Study}
To further explore the effectiveness of M3DeTR, we conduct ablation on the Waymo Open Dataset. Specifically, we demonstrate the individual benefits of the multi-representation,
multi-scale, and mutual-relation layers of the M3 transformer. As shown in the Table~\ref{}, M3DeTR 

\section{More Implementation Details}
\subsection{RoIPool}
As pointed out in the Section 

\section{Additional Qualitative Results}
As shown in the Figure~\ref{}, we add some additional qualitative visualizations to further demonstrate the effectiveness of M3DeTR, compared to PV-RCNN~\cite{shi2020pv}. 







\bibliographystyle{ACM-Reference-Format}











\section{Introduction}
3D object detection is a fundamental problem in computer vision and many applications, including autonomous driving~\cite{geiger2012we, Sun_2020_CVPR}, augmented reality~\cite{park2008multiple} and robotics~\cite{oh2002development}.
Moreover, different methods have been proposed for various sensors, including monocular cameras, depth cameras, LiDAR, and radars ~\cite{qi2017pointnet,qi2017pointnet++,qi2018frustum,yang2019std,ye2020hvnet,yang20203dssd,shi2020pv,yin2020center}. 2D object detection deals with detecting objects from RGB images and videos, while 3D object detection utilizes point cloud-based representations obtained from LiDARs or other sensors. Moreover, it is known that point cloud data obtained from LiDAR sensors tends to be more accurate than RGB images and videos~\cite{caesar2020nuscenes}. Consequently, point clouds are being widely used for scene understanding in autonomous driving and AR. 

\begin{figure}[t]
    \centering
    \includegraphics[width=\linewidth]{LaTeX/ACMMM/figures/coverpic.pdf}
    \caption{M3 Transformers used in our approach: Left: Multi-representation transformers; Middle: Multi-scale transformers; Right: Mutual-relation transformers. We use these characteristics to present a novel 3D object detection architecture.}
    \label{fig:coverpic.p}
\vspace{-0.5cm}
\end{figure}

The previous
state-of-the-art methods for 3D object detection base on different networks~\cite{yin2020center,shi2020pv}. However, there are two key limitations:

\noindent{\textbf{Ineffective point cloud representations:}} The three major techniques used to process point clouds are based on voxels~\cite{zhou2018voxelnet,yan2018second}, raw point clouds~\cite{qi2017pointnet,qi2017pointnet++,qi2018frustum,shi2019pointrcnn,yang2019std,liu2019point, qin2020weakly}, and bird's-eye-view ~\cite{chen2017multi, ku2018joint, yang2018pixor}. Each representation has a unique advantage and it has been shown that combining these representations can result in terms of detection accuracy \cite{qi2018frustum,shi2020pv,chen2019fast}. However, fusing these representations is non-trivial. First, the architectures corresponding to the VoxelNets, the PointNets, and the 2D convolutional neural networks are  different. Moreover, raw point clouds need to be converted to voxels and pixels before techniques based on VoxelNets and 2D convolutional neural networks can be applied. The differences between the inputs of these three neural models can result in semantic gaps. Previous works~\cite{chen2019fast,shi2020pv} tend to use feature concatenation and attention modules to fuse multi-representation features. However, the correlation between features of different representations has not been addressed.

\noindent{\textbf{Insufficient modeling of multi-scale features:}} Fusing multi-scale feature maps is a well-known technique used for improving the detection performance of 2D object detection. In terms of 3D object detection, current approaches tend to use multi-scale feature pyramids~\cite{qi2017pointnet,qi2017pointnet++,yang20203dssd,lan2019modeling,lang2019pointpillars,ye2020hvnet}. However, fusing these multiple feature pyramids is non-trivial because the higher resolution and the larger receptive fields are conflicting \cite{lin2017feature, qi2017pointnet++}. Existing methods~\cite{qi2017pointnet++,ye2020hvnet,shi2020pv} fuse the multi-scale features using bi-linear down-sampling/up-sampling and concatenation. Although these approaches~\cite{qi2017pointnet++,ye2020hvnet,shi2020pv} can improve the accuracy by a large margin, there are many challenges with respect to the underlying fusion method  in terms of the correlation between feature maps of different scales.

A key issue in terms of designing a good approach for object detection is exploiting the correlation between different representations and the large size of the receptive fields.  Our approach is motivated by use of transformers~\cite{vaswani2017attention}, a form of neural network based on attention that has been used in natural language processing ~\cite{hu2020iterative,tan2019lxmert,su2019vl,lu2019vilbert}. Specifically, transformers use multi-head attention to narrow the semantic gap between different representations by adapting to informative features and eliminating noise.

Another key aspect of 3D object detection is to model the mutual relationships between different points in the point cloud data~\cite{qi2017pointnet,qi2017pointnet++,lan2019modeling,thomas2019kpconv}. Modeling mutual relationships can enhance the ability to recognize the fine-grained patterns and can generalize to complex scenes.
Prior works in 3D object detection have modeled these relationships using multi-layer perceptrons~\cite{qi2017pointnet,qi2017pointnet++,qi2018frustum,shi2020pv}, farthest point sampling layers~\cite{qi2017pointnet++,shi2019pointrcnn,shi2020pv}, max pooling layers~\cite{lan2019modeling,qi2017pointnet++,shi2019pointrcnn,shi2020pv}, and graph convolutional networks~\cite{wang2019dynamic}. However, a key challenge is to model these mutual relationships along  with fusing different representations and multi-scale features.

We  present \textsc{M3DeTR}, a novel two-stage architecture for 3D object detection task. Given raw 3D point cloud data, our approach can localize static and dynamic obstacles with state-of-the-art accuracy. As illustrated in Figure~\ref{fig:coverpic.p}, the key components of our approach are M3 transformers, which are used to combine different feature representations. Conceptually, each of the M3 transformers are used for aggregating point cloud representations, multi-scale representations and mutual relationships among a subset of points in the point cloud data.
\\

\noindent \textbf{Summary of contributions:}
\begin{itemize}
    \item \textsc{M3DeTR} is the first unified architecture for 3D object detection with transformers that accounts for multi-representation, multi-scale, mutual-relation models of point clouds in an end-to-end manner.

    \item \textsc{M3DeTR} is robust and insensitive with respect to the hyper-parameters of transformer architectures.
    We test multiple variants with different transformer blocks designs.
    We demonstrate improved performance of \textsc{M3DeTR}  regardless of hyper-parameters.

    \item Our unified architecture achieves state-of-the-art performance on KITTI 3D Object Detection Dataset~\cite{geiger2012we} and Waymo Open Dataset~\cite{Sun_2020_CVPR}. We outperform the previous state-of-the-art approaches by 2.86\% mAP for car class on the Waymo validation set and 1.48\% mAP for all classes on the Waymo test set.
\end{itemize}

\section{Related Work}
\textbf{Multi-representation modeling.}
Existing techniques for modeling 3D point cloud data include bird-eye-view (BEV), volumetric, and point-wise representations. 
Generally, BEV-based approaches ~\cite{chen2017multi, ku2018joint, liang2018deep, liang2019multi} first project 3D point clouds into 2D BEV space and then adopt the standard 2D object detectors to generate 3D object proposals from projected 2D feature maps. To deal with the irregular format of input point clouds, voxel-based architectures~\cite{zhou2018voxelnet, yan2018second, lang2019pointpillars} use equally spaced 3D voxels to encode the point clouds such that the volumetric representation can be consumed by the region proposal network (RPN) ~\cite{ren2015faster}. Inspired by the PointNet/PointNet++ approach~\cite{qi2017pointnet, qi2017pointnet++}, which is invariant under transformation,~\cite{qi2018frustum, shi2019pointrcnn} extend this method to the task of 3D object detection and directly process the raw point clouds to infer 3D bounding boxes. However, these methods are typically limited due to either information loss or high computation cost. Recently, many approaches~\cite{yang2019std, liu2019point, tang2020searching, shi2020pv, yin2020center} have combined the advantages of speed (of voxel-based representation) and efficiency (of point-based representation) by fusing point-voxel features for 3D object prediction.

\textbf{Multi-scale modeling.} Modeling multi-scale features is an important procedure in deep learning-based computer vision~\cite{lin2017feature, cai2019guided, liu2019l2g, zhang2020feature, liao2020mmnet, zhu18b,qi2017pointnet++, hu2020you, wang2020infofocus} because it is able to enlarge the receptive field and increase resolution. In 3D representation, modeling multi-scale features is also popular and important. PointNet++~\cite{qi2017pointnet++} proposes the set abstraction module to model local features of a cluster of point clouds. To model multi-scale patterns of point clouds, they use 3 different sampling ranges and radii with 3 parallel PointNets and thus fuse the multi-scale. In 3D object detection,~\cite{hu2020you, wang2020infofocus, ye2020hvnet} adopt different detection heads with multi-scale feature maps to handle both large and small object classes. 

\textbf{Mutual-relation modeling.} 2D Convolutional Neural Networks \cite{He_2016_CVPR} are commonly used to process mutual relations in 2D images. As point clouds are scattered and lacking structure, passing information from one point to its neighbors is not trivial. PointNets~\cite{qi2017pointnet++} proposes the set abstraction module to model the local context by using the subsampling and grouping layers. After this well-known work, many convolution-like operators~\cite{thomas2019kpconv,lan2019modeling,wang2019dynamic} on point clouds have been proposed to model the local context and the mutual relation between points. Recently, transformers~\cite{zhao2020point,engel2020point,guo2020pct,pan20203d} have been introduced in PointNets to model mutual relation. However, those previous works mainly focus on the local and global contexts of point clouds by applying mutual-relation transformers on points. Instead, our approach not only models the mutual relation between points, but it also models multi-scale and multi-representation features of point clouds.
\begin{figure*}[t]
    \centering
    \includegraphics[width=\linewidth]{LaTeX/ACMMM/figures/arch.pdf}
    \caption{\textbf{An overview of our \textsc{M3DeTR} architecture: \textsc{M3DeTR} is a transformer based framework for object detection in a coarse-to-fine manner.} It consists of three parts. PointNets, VoxelNet, and 2D ConvNets modules enable individual multi-representation feature learning.  M3 Transformers enable inter-intra multi-representation, multi-scale, multi-location feature attention. With the Region Proposal Network (RPN), the initial box proposals are generated. R-CNN captures and refines region-wise feature representations from M3 transformer output to improve detection performance. }
    \label{fig:3MDetr_network}
    \vspace{-0.2cm}
\end{figure*}

\textbf{Transformers in computer vision.}
Inspired by their success in machine translation ~\cite{vaswani2017attention}, transformer-based architectures have recently become effective and popular in a variety of computer vision tasks. Particularly, the design of self-attention and cross-attention mechanisms in transformers has been successful in modeling dependencies and learning richer information.~\cite{dosovitskiy2020image} leverages the direct application of transformers on the image recognition task without using convolution.~\cite{carion2020end, zhu2020deformable} apply transformers to eliminate the need for many handcrafted components in conventional object detection and achieve impressive detection results. ~\cite{yang2020learning, zeng2020learning} explore transformers on image and video synthesis tasks.~\cite{zhao2020point} investigates the self-attention networks on 3D point cloud segmentation task. 

Recently, there have been several joint representation fusion research attempts. PointPainting~\cite{vora2020pointpainting} proposes a novel method that accepts both images and point clouds as inputs to do 3D object detection by appending 2D semantic segmentation labels to LiDAR points. In the visual question answering task,~\cite{hu2020iterative} jointly fuses and reasons over three different modality representations. ~\cite{mittal2020m3er} combines multimodal information to solve robust emotion recognition problems. Building on a multi-representation and multi-scale transformer, our proposed model addresses the voxel-wise, point-wise, and BEV-wise feature representation gap and enables effective cross-representation interactions with different levels of semantic features. Coupled with a point-wise mutual-relation transformer, our framework learns to capture deeper local-global structures and richer geometric relationships among point clouds.



\section{Our Approach}
\textsc{M3DeTR} takes point cloud data as input and generates 3D boxes for different object categories with state-of-the-art accuracy as shown in Figure~\ref{fig:3MDetr_network}. Our goal is to perform multi-representation, multi-scale, and mutual-relation fusion with transformers over a joint embedding space. Our method consists of three main steps:

\begin{itemize}
    \item Generate feature embeddings for different point cloud representations using VoxelNet, PointNet, and 2D ConvNet.
   
    
    \item Fuse these embeddings using \textit{M3 transformer} that leverages multi-representation and multi-scale feature embedding and models mutual relationships between points.
    
    \item Perform 3D detection using detection heads network, including RPN and R-CNN stages.
\end{itemize}


\subsection{Multi-Representation Feature Embeddings}
\label{backbone}
Our network processes the raw input point clouds $\bm{P} = \{\bm{p}_1\text{,} \bm{p}_2\text{, ...,} \bm{p}_n\}$ and encodes them into three different embedding spaces, namely, voxel-, point-, and BEV-based feature representations. We discuss the embedding process for each representation in detail.

\noindent\textbf{Voxels:}
Voxel-wise feature extraction is divided into two steps: (1) the voxelization layer, which is used by VoxelNet~\cite{zhou2018voxelnet}, takes the raw input point clouds $\bm{P}$ and converts them into equally spaced 3D voxels $\bm{v} \in \mathbb{R}^{L \times W \times H}$; (2) voxel-wise features $\bm{f}^{voxel}$ at different scales are extracted with 3D sparse convolutions, which is visualized in Figure~\ref{fig:3MDetr_network}. Unlike ~\cite{zhou2018voxelnet, yan2018second}, all four different scales of obtained 3D voxel CNN features, $\bm{f}^{voxel} = \{\bm{f}^{voxel\ 1\times},\  \bm{f}^{voxel\ 2\times},\  \bm{f}^{voxel\ 4\times},\  \bm{f}^{voxel\ 8\times} \}$ are passed to the transformer stage as shown in Figure~\ref{fig:3MDetr_network}. More details will be introduced in the supplementary materials. 


\noindent\textbf{Bird's-eye-view:}
2D ConvNets takes the $8 \times$ downsampled voxel-based feature map $\bm{f}^{voxel\ 8\times}$ from the 3D Voxel CNN branch and generates a corresponding BEV-representation embedding. To directly apply 2D convolution, we start by combining the z-dimension and the channel dimension of the input voxel features into a single dimension. The structure of 2D ConvNets includes two encoder-decoder blocks, where each block consists of 2D down-sampling convolutional layers to produce top-down features, as well as de-convolutional layers to upsample to the input feature size. Specifically, both encoder and decoder paths are composed of a number of 2D convolutional layers with the kernel size of $3 \times 3$ followed by a BatchNorm layer and a ReLU layer. The output BEV-based feature from 2D ConvNets, denoted as $\bm{f}^{bev} \in \mathbb{R}^{L_{bev}\times W_{bev}\times C_{bev}}$, was further converted into keypoint feature $\bm{F}^{bev}\in \mathbb{R}^{n\times c_{bev}}$ through bi-linear interpolation. 

\noindent\textbf{Points:}
Typically, there are more than 10K raw points inside an entire point cloud scene. In order to cover the entire point set effectively without large memory consumption, we apply Furthest-Point-Sampling (FPS) algorithm to sample $n$ keypoints, denoted as $\bm{\hat{P}} \subset \bm{P}$. Adopted from PointNet++~\cite{qi2017pointnet++} and PV-RCNN~\cite{shi2020pv}, Set Abstraction and Voxel Set Abstraction (VSA) module take raw point coordinates $\bm{P}$ and the 3D voxel-based features $\bm{f}^{voxel}$, respectively, to generate keypoint features for $\bm{\hat{P}}$. In particular, the corresponding keypoint features from raw points $\bm{P}$ is $\bm{F}^{point} \in \mathbb{R}^{n\times c_{point}}$, and the corresponding keypoint features from voxels $\bm{f}^{voxel}$ are $\{\bm{F}^{voxel\ 1\times}$, $\bm{F}^{voxel\ 2\times}$, $\bm{F}^{voxel 4\times}$,  $\bm{F}^{voxel\ 8\times} \}$, where $\bm{F}^{i}\in \mathbb{R}^{n\times c_{i}}$.

\subsection{Multi-representation, Multi-scale, and Mutual-relation Transformer}
\label{m3transformer}
Once the three feature embedding sequences, $\bm{F}^{voxel}$, $\bm{F}^{point}$ and $\bm{F}^{bev}$ are generated, they are able to dynamically and intelligently attend to each other, generating final cross-representations, cross-scales, and cross-points descriptive feature representations as shown in Figure \ref{fig:3MDetr_network}. In the remainder of this section, we will briefly review transformers basics followed by discussing the multi-representation, multi-scale, and mutual-relation transformer layers. 

\noindent\textbf{Transformer basics.}
A transformer is a stacked encoder-decoder architecture relying on a self-attention mechanism to compute representations of its input and output~\cite{vaswani2017attention}. Consider two input matrices $\bm{X}_l \in \mathbb{R}^{l\times d_{in}}$ and $\bm{X}_s \in \mathbb{R}^{s \times d_{in}}$, where $l$ and $s$ are the lengths of the input sequence of dimension $d_{in}$. The attention layer output is defined as: \vspace{-3.5 mm}

$$\textrm{Attention}(\bm{Q}, \bm{V}, \bm{K}) = \textrm{Softmax}(\frac{\bm{Q}\bm{K}^{T}}{\sqrt{d_{in}}})\bm{V} \in \mathbb{R}^{l \times d_{out}},$$ 

$\textrm{where}\  \bm{Q} = \bm{X}_l \bm{W}_q,\  \bm{K} = \bm{X}_l \bm{W}_k\  and\  \bm{V} = \bm{X}_s \bm{W}_v.$

The matrices $\bm{Q}$, $\bm{K}$ and $\bm{V}$ represent query, key and value respectively, obtained through projection matrices $\bm{W}_q, \bm{W}_k\in \mathbb{R}^{d_{in}\times d_{h}}$ and $\bm{W}_v\in \mathbb{R}^{d_{in}\times d_{out}}$, where $d_{h}$ is the hidden dimension, and $d_{out}$ is the output dimension. When two input matrices $\bm{X}_l$ and $\bm{X}_s$ represent the same feature maps, the attention module is usually referred to as self-attention. Each transformer layer consists of one Multi-head Self-attention (MHSA) module and a few linear layers, as well as normalization and activation layers. More details will be introduced in the supplementary materials.





Now, we present the proposed transformers used to capture the inter- and intra- interactions among input features. Specifically, we propose two stacked transformer encoder layers named M3 Transformers, as shown in Figure \ref{fig:3MDetr_transformer}: (1) the multi-representation and multi-scale transformer, and (2) the mutual-relation transformer.


\noindent\textbf{Multi-representation and multi-scale transformer layer.}
First, we focus on the intra-point representation fusion. For each individual point, the input sequence to this transformer layer is its several different corresponding point cloud features, including various scales and distinctive representations. The input sequence to the transformer is $\bm{F} = [\bm{F}^{voxel\ 1\times}$, $ \bm{F}^{voxel\ 2\times}$, $ \bm{F}^{voxel\ 4\times}$, 
$\bm{F}^{voxel\ 8\times}$, $\bm{F}^{point}$, $\bm{F}^{bev}]$, where each $\bm{F}^{i}\in \mathbb{R}^{n\times c_i}$. After explicitly modeling all element-wise interactions among those features for each point separately, the output from the block is the updated feature vectors after aggregating the information from all the input sequence $\bm{F}$.

The multi-representation and multi-scale transformer layer takes 6 different inputs: $\bm{F}^{point}$, $\bm{F}^{voxel 1\times}$, $\bm{F}^{voxel 2\times}$, $\bm{F}^{voxel 4\times}$, $\bm{F}^{voxel 8\times}$,  $\bm{F}^{bev}$. As different inputs may have different feature dimensions, we use the single-layer perceptron to apply feature reduction on the input features to align the feature dimensions of each feature embeddings. The outputs of the feature reduction layer are $\bm{\hat{F}} = [\bm{\hat{F}}^{voxel\ 1\times}$, $ \bm{\hat{F}}^{voxel\ 2\times}$, $ \bm{\hat{F}}^{voxel\ 4\times}$, $\bm{\hat{F}}^{voxel\ 8\times}$, $\bm{\hat{F}}^{point}$, $\bm{\hat{F}}^{bev}]$, where the output dimension of each feature is equivalent to $\hat{c}$.

After the feature reduction layer, the multi-representation and multi-scale transformer layer takes $\bm{\hat{F}}$ as inputs and generates self-attention features $\bm{T}^{voxel\ 1\times}$, $\bm{T}^{voxel\ 2\times}$,$\bm{T}^{voxel\ 4\times}$,$\bm{T}^{voxel\ 8\times}$, $\bm{T}^{point}$, $\bm{T}^{bev}$, which corresponds to $\bm{\hat{F}}^{voxel\ 1\times}$, $\bm{\hat{F}}^{voxel\ 2\times}$,$\bm{\hat{F}}^{voxel\ 4\times}$,$\bm{\hat{F}}^{voxel\ 8\times}$, $\bm{\hat{F}}^{point}$, $\bm{\hat{F}}^{bev}$. We visualize the multi-representation and multi-scale transformer in Figure \ref{fig:3MDetr_transformer}.


\noindent\textbf{Mutual-relation transformer layer.}
Inspired by~\cite{qi2017pointnet, qi2017pointnet++, pan20203d}, inter-points feature fusion within a spatial neighboring space is leveraged in the mutual-relation layer of the transformer. Our goal is to attend to and aggregate the neighboring information in an attention manner for each point with an enriched feature.

From the first transformer block output, we obtain aggregated features $\bm{T}$ of different scales and representations after the first transformer block. We concatenate channels along the point dimension and rearrange those points into a sequence as the input of the mutual-relation transformer. Let the learned concatenated feature $\bm{T} = 
\rm{concat}$ $\{\bm{T}^{voxel\ 1\times}$, $ \bm{T}^{voxel\ 2\times}$, $ \bm{F}^{voxel\ 4\times}$, $\bm{T}^{voxel\ 8\times}$, $\bm{T}^{point}$, $\bm{T}^{bev}\} \in\mathbb{R}^{n\times c_T}$, where $c_{T}$ is the sum of all channel size in $\bm{T}$.  Note that, the first dimension of $\bm{T}$ is the number of keypoints $n$. To model the mutual-relation between keypoints, we need to split $\bm{T}$ into $n$ point-wise feature $\bm{T} = \{ \bm{T}_i\}$, where $\bm{T}_i \in \mathbb{R}^{c_{T}}$. The concatenation and split module is showed in Figure \ref{fig:3MDetr_transformer}.

The mutual-relation transformer takes point-wise features of $n$ keypoints as inputs and uses the multi-head self-attention head to model the mutual relationship between keypoints. The outputs of the mutual-relation transformer are $\bm{\hat{T}}$.


\begin{figure}[t]
    \centering
    \includegraphics[width=0.85\linewidth]{LaTeX/ACMMM/figures/Transformer_block.pdf}
    \caption{\textbf{M3 Transformers} consist of two parts: a multi-representation and multi-scale transformer, and a mutual-relation transformer. Multi-representation and multi-scale transformer takes the different feature embedding and generates enriched cross-representations and cross-scales embedding. On top of it, the mutual-relation transformer further models point-wise feature relationship to extract the refined features.}
    \label{fig:3MDetr_transformer}
    \vspace{-0.3cm}
\end{figure}

\noindent\textbf{Comparison with previous point-based transformers}
Prior work has explored the transformer application on the task of point cloud processing. First, \cite{guo2020pct, engel2020point, zhao2020point} leverage the inherent permutation invariance of transformers to capture local context within the point cloud on the shape classification and segmentation tasks, while our \textsc{M3DeTR} mainly investigates the strong attention ability of transformers between input embeddings for the 3D object detection task. Pointformer \cite{pan20203d} is the approach most related to our method because we both address the 3D object detection task by capturing the dependencies among points' features. However, Pointformer adopts a single PointNet branch to extract points feature, while \textsc{M3DeTR} considers all three different representations and also applies the transformer to learn aggregated representation-based features.

\subsection{Detection Heads Network}
\label{dethead}
After we obtain the enriched embedding $\bm{\hat{T}}$ from the M3 transformer, the detection network is composed of two stages that predict 3D bounding box class, localization, and orientation in a coarse-to-fine manner, including RPN and R-CNN. Please refer to PV-RCNN~\cite{shi2020pv} for more details.

\noindent\textbf{RPN:} Region Proposal Networks take the deep semantic features $\bm{f}^{bev}$ produced by the 2D ConvNets as inputs and generate high-quality 3D object proposals $\bm{B}$, as shown in Figure \ref{fig:3MDetr_network}. A 3D object box $\bm{B}_i$ is parameterized as $(x, y, z, l, h, w, \theta) $, where $(x, y, z)$ is the center of the box, $(l , h, w)$ is the dimension of the box, and $\theta$ is the orientation in bird's-eye-view. Similar to the conventional RPN in 2D object detection~\cite{ren2015faster} each position on the deep feature map is placed by predefined bounding box anchors denoted as ($x^a$, $y^a$, $z^a$, $l^a$, $h^a$, $w^a$, $\theta^a$). Then initial proposals are generated by predicting the relative offset of an object's 3D ground truth bounding box ($x^{gt}$, $y^{gt}$, $z^{gt}$, $l^{gt}$, $h^{gt}$, $w^{gt}$, $\theta^{gt}$).



\noindent\textbf{R-CNN:}
R-CNN serves as the second stage and it takes the initial region proposals $\bm{B}$ from RPN as input to conduct further proposal refinement. For each input proposal box, the RoI-grid pooling module~\cite{shi2020pv} is adopted to extract the corresponding proposal-specific grid points' features from the transformer-based embeddings $\bm{\hat{T}}$. Compared with previous works, \textsc{M3DeTR} leverages the richer embedding information from the learned transformers for the fine-grained proposal refinement. As the main component to extract refined features, the RoI-grid pooling module uniformly samples $ N \times N \times N $ grid points per 3D proposal. For each grid point, the output feature is generated by applying a PointNet-block \cite{qi2017pointnet} on a small number of surrounding keypoints, $M$, within its spatial surrounding region with a radius of $r$. Specifically, keypoints are the subset of input points that are sampled using the Furthest-point-sampling algorithm to cover the entire point set. 


\begin{table*}[ht]
    \centering
   
\input{LaTeX/ACMMM/table/waymo_test_all_classes}
\end{table*}

\begin{table*}[ht]
    \centering
   
\input{LaTeX/ACMMM/table/waymo_test}
\end{table*}

Finally, the refined representations of each proposal are first passed to two fully connected layers and then generate the box 3D Intersection-over-Union (IoU) guided confidence scoring of class prediction and location refinement of regression targets, $\bm{\hat{B}}$. Compared with the traditional classification-guided box scoring, 3D IoU guided confidence scoring considers the IoU between the proposal box and its corresponding ground truth box. Empirically, \cite{shi2020points,shi2020pv} show that it achieves better results compared with the traditional classification confidence based techniques. \vspace{-2 mm}

\begin{table*}[t]
    \centering
\input{LaTeX/ACMMM/table/waymo_val}
\end{table*}

\subsection{Loss Functions} 
\label{loss}
In this section, we define the loss function. The bounding box regression target for both RPN and R-CNN stages is calculated as the relative offsets between the anchors and the ground truth as:
$\triangle x = \frac{x^{gt} - x^a}{d^a},  \triangle y = \frac{y^{gt} - y^a}{d^a}$,  
$\triangle z = \frac{z^{gt} - z^a}{h^a}$
$\triangle h = \log (\frac{h^{gt}}{{h^{a}}})$,  
$\triangle w = \log (\frac{w^{gt}}{{w^{a}}})$,
$\triangle \theta = \theta^{gt} - \theta^{a}$,
 where $ d^a $ = $\sqrt{(w^a)^2 + (l^a)^2} $. Similar to~\cite{shi2020pv, shi2020points}, the focal loss is applied \cite{lin2017focal} for the classification loss, $\mathcal{L}_{cls}$. Smooth L1 loss \cite{girshick2014rich} is adopted for the box localization regression target's losses, $\mathcal{L}_{reg}$ and $\mathcal{L}_{ref}$. In addition, 3D IoU loss \cite{shi2020pv, shi2020points} is used for $\mathcal{L}_{iou}$. 

Similar to PV-RCNN \cite{shi2020pv}, we formally define a multi-task loss for both the RPN and R-CNN stages, 
\vspace{1pt}
\begin{equation}
\begin{aligned}
 & \mathcal{L}_{cls} = -\alpha_{a}(1 - p^{a})^\gamma \log p^{a}, \\
 & \mathcal{L}_{reg} = \sum_{b \in (x, y, z, w, l, h, \theta)} \mathcal{L}_s (\triangle b), \\
 & \mathcal{L} = \mathcal{L}_{cls} + \beta_{reg} \mathcal{L}_{reg} + \beta_{iou} \mathcal{L}_{iou} + \beta_{ref} \mathcal{L}_{ref},
\end{aligned}
\end{equation}

where $p^{a}$ is the model's estimated class probability for an anchor box, and $\beta_{reg}$, $\beta_{iou} $ and $\beta_{ref}$ are chosen to balance the weights between classification loss, IoU loss and regression loss for RPN stage and R-CNN stage. We adopt the default $\alpha = 0.25$, and $\gamma = 2.0$ from the parameters of focal loss~\cite{lin2017focal}.


\section{Experiments}
In this section, we evaluate \textsc{M3DeTR} both qualitatively and quantitatively on the Waymo Open Dataset~\cite{Sun_2020_CVPR} and the KITTI Dataset~\cite{geiger2012we} in the task of LiDAR-based 3D object detection. Our main results include achieving a state-of-the-art accuracy on these datasets and robustness to hyper-parameter tuning.


\subsection{Datasets and Evaluation Metric}
\label{dataset}
\noindent \textbf{Waymo Open Dataset:}
The Waymo Open Dataset~\cite{Sun_2020_CVPR} is a large-scale autonomous driving dataset containing $1000$ scenes of $20$s duration each, with $798$ scenes for training and $202$ scenes for validation. Each scene is sampled at a frequency of $10$Hz. Overall, the dataset includes $12$M labeled objects and thus we only use one fifth of the training scenes for the following experiment. We consider LiDAR data as the input to our approach. The evaluation protocol on the Waymo dataset consists of the mean average precision (mAP) and mean average precision weighted by heading (mAPH). For each object category, the detection outcomes are evaluated based on two difficulty levels: LEVEL\_1 denotes the annotated bounding box with more than 5 points and LEVEL\_2 represents the annotated bounding box with more than 1 point.

\textbf{KITTI dataset}
The KITTI 3D object detection benchmark~\cite{geiger2012we} is another popular dataset for autonomous driving. It contains $7,481$ training and $7,518$ testing LiDAR scans. We follow the standard split on the training ($3,712$ samples) and validation sets ($3,769$ samples). For each object category, the detection outcomes are evaluated based on three difficulty levels based on the object size, occlusion state, and truncation level. \vspace{-2.5 mm}


\subsection{Implementation Details}\label{impl}
Closely following the codebase\footnote{https://github.com/open-mmlab/OpenPCDet.}, we use PyTorch to implement our M3 transformer modules and integrate them into the PV-RCNN network~\cite{shi2020pv}. More details about backbone and detection heads network will be introduced in the supplementary materials.




\noindent\textbf{M3 Transformer}
We project the embeddings obtained from the backbone network with different scales and representations to 256 channels, as the input of the multi-representation and multi-scale transformer requires, and project the output features back to their original dimensions before passing into the mutual-relation transformer. Due to the GPU memory constraint, we experiment with two types of MHSA module designs: 2 encoder layers with 4 attention heads and 1 encoder layer with 8 attention heads. 



\begin{figure*}[t]
    \centering
    \includegraphics[width=1\linewidth]{LaTeX/ACMMM/figures/Visualization_4.pdf}
    \caption{We highlight the 3D detection results of \textbf{M3DeTR} on the Waymo Open Dataset. The 3D ground truth bounding boxes are in green, while the detection bounding box are shown in blue.}
    \label{fig:visualization}
\end{figure*}

\noindent\textbf{Training Parameters}
Models are trained from scratch on $4$ NVIDIA P6000 GPUs. We use the Adam optimizer with a fixed weight decay of $0.01$ and use a one-cycle scheduler proposed in~\cite{smith2018disciplined}. For the Waymo Open Dataset, we train our models for $45$ epochs with a batch size of $8$ scenes and a learning rate $0.01$, which takes around $50$ hours. For the KITTI dataset, we train our models for $80$ epochs with a batch size of $8$ scenes per and a learning rate $0.01$, which takes around $15$ hours. 

\begin{table}[t]
    \centering
    \setlength\tabcolsep{3px}
\input{LaTeX/ACMMM/table/kitti_test_car_cyclist}
\vspace{-0.2cm}
\end{table}

\subsection{Results}
\label{result}
\noindent\textbf{Waymo Open Dataset:}
We first present our object detection results for the vehicle, pedestrian, and cyclist classes on the test set of Waymo Open Dataset in Table~\ref{tab:waymo_test_all_classes} compared with PV-RCNN~\cite{shi2020pv}. We evaluate our method at both LEVEL\_1 and LEVEL\_2 difficulty levels. Note that we reproduce the baseline PV-RCNN with a single frame input since they recently adopt two-frames input on the test set. As we can see, PV-RCNN achieves 69.57\% and 63.65\% on average in LEVEL\_1 mAP and LEVEL\_2 mAP, respectively, while \textsc{M3DeTR} improves them by 1.48\% and 1.85\%, respectively. Without bells and whistles, our approach works better than PV-RCNN~\cite{shi2020pv}. Furthermore, we compare our framework on the vehicle class for different distances with state-of-the-art methods, including StarNet~\cite{ngiam2019starnet}, PointPillars~\cite{lang2019pointpillars}, RCD~\cite{bewley2020range}, Det3D~\cite{zhu2019class}, RangeDet~\cite{fan2021rangedet} and PV-RCNN~\cite{shi2020pv}. In Table~\ref{tab:waymo_test}, \textsc{M3DeTR} outperforms PV-RCNN significantly in both LEVEL\_1 and LEVEL\_2 difficulty levels across all distances, demonstrating the effectiveness of the newly proposed framework. Moreover, we visualize the detection results of \textsc{M3DeTR} in Figure ~\ref{fig:visualization}. 

Compared with the PV-RCNN shown in Figure~\ref{fig:visualization_comparison}, \textsc{M3DeTR} successfully captures the inter- and intra- interactions among input features and effectively helps the model generate high-quality box proposals. To the best of our knowledge, \textsc{M3DeTR} achieves the state-of-the-art in the Vehicle class in both LEVEL\_1 and LEVEL\_2 difficulty levels among all the published papers with a single frame LiDAR input.

We also evaluate the overall object detection performance with an IoU of 0.7 for the Vehicle class on the full Waymo Open Dataset validation set as in Table~\ref{tab:waymo_val}, further proving that our architecture is more efficient for jointly modeling the input features.

\noindent\textbf{KITTI dataset:}
We compare our approach with the state-of-the-art methods on the KITTI test set~\cite{shi2020pv,chen2020object}. We compute the mAP on three difficult types of both car and  cyclist classes in 3D detection metric. Table~\ref{tab:kitti_test_car_cyclist} shows that \textsc{M3DeTR} achieves state-of-the-art performance and outperforms the previous work by a large margin especially on the cyclist class. In particular, HotSpotNet~\cite{chen2020object} achieves $82.59\%$ in the ``easy'' categories of 3D detection metric, while \textsc{M3DeTR} improves these results by significant $1.24\%$.  \vspace{-2 mm}

\begin{table}[t]
    \centering
    \setlength\tabcolsep{3px}
\input{LaTeX/ACMMM/table/kitti_ablation_transformer}
\vspace{-0.5cm}
\end{table}

\subsection{Ablation Studies}
\label{ablation}
To demonstrate the individual benefits of the multi-representation, multi-scale, and mutual-relation layers of the M3 transformer, we perform ablation experiments and tabulate the results in Table~\ref{tab:kitti_ablation_transformer}. All experiments are conducted on the validation set of KITTI dataset.


With the single multi-representation and multi-scale transformer layer, we can achieve 4.07\% and 1.71\% on the moderate difficulty in car class with 11 and 40 recall positions, respectively compared with the PV-RCNN baseline. On the other side, with the single mutual-relation transformer layer, the performance gain are 4.47\% and 1.99\% compared with the PV-RCNN baseline. Without hyper-parameter tuning, \textsc{M3DeTR} benefits from unifying multiple point cloud representations, feature scales, and model mutual-relations simultaneously which results in the best performance.  \vspace{-2 mm}

\subsection{Robustness of M3DeTR}
\label{robust}
To demonstrate the robustness of \textsc{M3DeTR} to hyper-parameter tuning, we perform a series of tests by varying the sampling size, number of detection heads, and number of transformer encoder layers. We present the results of these tests in Figure~\ref{fig:M3DeTR_robustness}, where we observe that \textsc{M3DeTR} performs consistently well for the ``car'' category with IoU threshold of 0.7 for both 11 and 40 recall positions on the KITTI validation set.

\begin{figure}[t]
    \centering
   
    \centerline{
    \includegraphics[width=1\linewidth]{LaTeX/ACMMM/figures/Comparison_pvrcnn_M3DeTR_wo.pdf}
    }
    \caption{We visualize the 3D detection results for the same input point cloud between PV-RCNN (left) and \textbf{M3DeTR} (right) on Waymo Open Dataset. We highlight the false negative boxes from PV-RCNN in red.}
   
    \label{fig:visualization_comparison}
\end{figure}

\begin{figure}[t]
    \centering
    \includegraphics[width=0.95\linewidth]{LaTeX/ACMMM/figures/M3DeTR_Robustness.pdf}
    \caption{Performance comparison of different \textbf{M3 Transformers} variants in Car class on KITTI validation set. We show the mAP results with IoU threshold of 0.7 for both 11 and 40 recall positions.  Note that "l" and "h" represent layer number and head dimension of M3 Transformers, respectively. "Top" denotes the number of proposals used for keypoint sampling from RPN stage.}
    \label{fig:M3DeTR_robustness}
    \vspace{-0.5cm}
\end{figure}
 
\section{Conclusions}
In this paper, we present \textsc{M3DeTR}, a novel transformer-based framework for object detection with LiDAR point clouds. \textsc{M3DeTR} is designed to simultaneously model multi-representation, multi-scale, mutual-relation features through the proposed M3 Transformers. Overall, the first transformer integrates features with different scales and representations, and the second transformer aggregates information from all keypoints. Experimental results show that \textsc{M3DeTR} outperforms previous work by a large margin on the Waymo Open Dataset and the KITTI dataset. Without bells and whistles, \textsc{M3DeTR} is demonstrated to be invariant to the hyper-parameters of transformer.






{\small
\bibliographystyle{ieee_fullname}

\section{Introduction}
3D object detection is a fundamental problem in computer vision and many applications, including autonomous driving~\cite{geiger2012we, Sun_2020_CVPR}, augmented reality~\cite{park2008multiple} and robotics~\cite{oh2002development}.
Moreover, different methods have been proposed for various sensors, including monocular cameras, depth cameras, LiDAR, and radars ~\cite{qi2017pointnet,qi2017pointnet++,qi2018frustum,yang2019std,ye2020hvnet,yang20203dssd,shi2020pv,yin2020center}. 2D object detection deals with detecting objects from RGB images and videos, while 3D object detection utilizes point cloud-based representations obtained from LiDARs or other sensors. Moreover, it is known that point cloud data obtained from LiDAR sensors tends to be more accurate than RGB images and videos~\cite{caesar2020nuscenes}. Consequently, point clouds are being widely used for scene understanding in autonomous driving and AR. 

\begin{figure}[t]
    \centering
    \includegraphics[width=\linewidth]{LaTeX/ACMMM/figures/coverpic.pdf}
    \caption{M3 Transformers used in our approach: Left: Multi-representation transformers; Middle: Multi-scale transformers; Right: Mutual-relation transformers. We use these characteristics to present a novel 3D object detection architecture.}
    \label{fig:coverpic.p}
\vspace{-0.5cm}
\end{figure}

The previous
state-of-the-art methods for 3D object detection base on different networks~\cite{yin2020center,shi2020pv}. However, there are two key limitations:

\noindent{\textbf{Ineffective point cloud representations:}} The three major techniques used to process point clouds are based on voxels~\cite{zhou2018voxelnet,yan2018second}, raw point clouds~\cite{qi2017pointnet,qi2017pointnet++,qi2018frustum,shi2019pointrcnn,yang2019std,liu2019point, qin2020weakly}, and bird's-eye-view ~\cite{chen2017multi, ku2018joint, yang2018pixor}. Each representation has a unique advantage and it has been shown that combining these representations can result in terms of detection accuracy \cite{qi2018frustum,shi2020pv,chen2019fast}. However, fusing these representations is non-trivial. First, the architectures corresponding to the VoxelNets, the PointNets, and the 2D convolutional neural networks are  different. Moreover, raw point clouds need to be converted to voxels and pixels before techniques based on VoxelNets and 2D convolutional neural networks can be applied. The differences between the inputs of these three neural models can result in semantic gaps. Previous works~\cite{chen2019fast,shi2020pv} tend to use feature concatenation and attention modules to fuse multi-representation features. However, the correlation between features of different representations has not been addressed.

\noindent{\textbf{Insufficient modeling of multi-scale features:}} Fusing multi-scale feature maps is a well-known technique used for improving the detection performance of 2D object detection. In terms of 3D object detection, current approaches tend to use multi-scale feature pyramids~\cite{qi2017pointnet,qi2017pointnet++,yang20203dssd,lan2019modeling,lang2019pointpillars,ye2020hvnet}. However, fusing these multiple feature pyramids is non-trivial because the higher resolution and the larger receptive fields are conflicting \cite{lin2017feature, qi2017pointnet++}. Existing methods~\cite{qi2017pointnet++,ye2020hvnet,shi2020pv} fuse the multi-scale features using bi-linear down-sampling/up-sampling and concatenation. Although these approaches~\cite{qi2017pointnet++,ye2020hvnet,shi2020pv} can improve the accuracy by a large margin, there are many challenges with respect to the underlying fusion method  in terms of the correlation between feature maps of different scales.

A key issue in terms of designing a good approach for object detection is exploiting the correlation between different representations and the large size of the receptive fields.  Our approach is motivated by use of transformers~\cite{vaswani2017attention}, a form of neural network based on attention that has been used in natural language processing ~\cite{hu2020iterative,tan2019lxmert,su2019vl,lu2019vilbert}. Specifically, transformers use multi-head attention to narrow the semantic gap between different representations by adapting to informative features and eliminating noise.

Another key aspect of 3D object detection is to model the mutual relationships between different points in the point cloud data~\cite{qi2017pointnet,qi2017pointnet++,lan2019modeling,thomas2019kpconv}. Modeling mutual relationships can enhance the ability to recognize the fine-grained patterns and can generalize to complex scenes.
Prior works in 3D object detection have modeled these relationships using multi-layer perceptrons~\cite{qi2017pointnet,qi2017pointnet++,qi2018frustum,shi2020pv}, farthest point sampling layers~\cite{qi2017pointnet++,shi2019pointrcnn,shi2020pv}, max pooling layers~\cite{lan2019modeling,qi2017pointnet++,shi2019pointrcnn,shi2020pv}, and graph convolutional networks~\cite{wang2019dynamic}. However, a key challenge is to model these mutual relationships along  with fusing different representations and multi-scale features.

We  present \textsc{M3DETR}, a novel two-stage architecture for 3D object detection task. Given raw 3D point cloud data, our approach can localize static and dynamic obstacles with state-of-the-art accuracy. As illustrated in Figure~\ref{fig:coverpic.p}, the key components of our approach are M3 transformers, which are used to combine different feature representations. Conceptually, each of the M3 transformers are used for aggregating point cloud representations, multi-scale representations and mutual relationships among a subset of points in the point cloud data.
\\

\noindent \textbf{Summary of contributions:}
\begin{itemize}
    \item \textsc{M3DETR} is the first unified architecture for 3D object detection with transformers that accounts for multi-representation, multi-scale, mutual-relation models of point clouds in an end-to-end manner.

    \item \textsc{M3DETR} is robust and insensitive with respect to the hyper-parameters of transformer architectures.
    We test multiple variants with different transformer blocks designs.
    We demonstrate improved performance of \textsc{M3DETR}  regardless of hyper-parameters.

    \item Our unified architecture achieves state-of-the-art performance on KITTI 3D Object Detection Dataset~\cite{geiger2012we} and Waymo Open Dataset~\cite{Sun_2020_CVPR}. We outperform the previous state-of-the-art approaches by 2.86\% mAP for car class on the Waymo validation set and 1.48\% mAP for all classes on the Waymo test set.
\end{itemize}

\section{Related Work}
\textbf{Multi-representation modeling.}
Existing techniques for modeling 3D point cloud data include bird-eye-view (BEV), volumetric, and point-wise representations. 
Generally, BEV-based approaches ~\cite{chen2017multi, ku2018joint, liang2018deep, liang2019multi} first project 3D point clouds into 2D BEV space and then adopt the standard 2D object detectors to generate 3D object proposals from projected 2D feature maps. To deal with the irregular format of input point clouds, voxel-based architectures~\cite{zhou2018voxelnet, yan2018second, lang2019pointpillars} use equally spaced 3D voxels to encode the point clouds such that the volumetric representation can be consumed by the region proposal network (RPN) ~\cite{ren2015faster}. Inspired by the PointNet/PointNet++ approach~\cite{qi2017pointnet, qi2017pointnet++}, which is invariant under transformation,~\cite{qi2018frustum, shi2019pointrcnn} extend this method to the task of 3D object detection and directly process the raw point clouds to infer 3D bounding boxes. However, these methods are typically limited due to either information loss or high computation cost. Recently, many approaches~\cite{yang2019std, liu2019point, tang2020searching, shi2020pv, yin2020center} have combined the advantages of speed (of voxel-based representation) and efficiency (of point-based representation) by fusing point-voxel features for 3D object prediction.

\textbf{Multi-scale modeling.} Modeling multi-scale features is an important procedure in deep learning-based computer vision~\cite{lin2017feature, cai2019guided, liu2019l2g, zhang2020feature, liao2020mmnet, zhu18b,qi2017pointnet++, hu2020you, wang2020infofocus} because it is able to enlarge the receptive field and increase resolution. In 3D representation, modeling multi-scale features is also popular and important. PointNet++~\cite{qi2017pointnet++} proposes the set abstraction module to model local features of a cluster of point clouds. To model multi-scale patterns of point clouds, they use 3 different sampling ranges and radii with 3 parallel PointNets and thus fuse the multi-scale. In 3D object detection,~\cite{hu2020you, wang2020infofocus, ye2020hvnet} adopt different detection heads with multi-scale feature maps to handle both large and small object classes. 

\begin{figure*}[t]
    \centering
    \includegraphics[width=\linewidth]{LaTeX/ACMMM/figures/arch.pdf}
    \caption{\textbf{An overview of our \textsc{M3DETR} architecture: \textsc{M3DETR} is a transformer based framework for object detection in a coarse-to-fine manner.} It consists of three parts. PointNets, VoxelNet, and 2D ConvNets modules enable individual multi-representation feature learning.  M3 Transformers enable inter-intra multi-representation, multi-scale, multi-location feature attention. With the Region Proposal Network (RPN), the initial box proposals are generated. R-CNN captures and refines region-wise feature representations from M3 transformer output to improve detection performance. }
    \label{fig:3MDETR_network}
    \vspace{-0.2cm}
\end{figure*}

\textbf{Mutual-relation modeling.} 2D Convolutional Neural Networks \cite{He_2016_CVPR} are commonly used to process mutual relations in 2D images. As point clouds are scattered and lacking structure, passing information from one point to its neighbors is not trivial. PointNets~\cite{qi2017pointnet++} proposes the set abstraction module to model the local context by using the subsampling and grouping layers. After this well-known work, many convolution-like operators~\cite{thomas2019kpconv,lan2019modeling,wang2019dynamic} on point clouds have been proposed to model the local context and the mutual relation between points. Recently, transformers~\cite{zhao2020point,engel2020point,guo2020pct,pan20203d} have been introduced in PointNets to model mutual relation. However, those previous works mainly focus on the local and global contexts of point clouds by applying mutual-relation transformers on points. Instead, our approach not only models the mutual relation between points, but it also models multi-scale and multi-representation features of point clouds.


\textbf{Transformers in computer vision.}
Inspired by their success in machine translation ~\cite{vaswani2017attention}, transformer-based architectures have recently become effective and popular in a variety of computer vision tasks. Particularly, the design of self-attention and cross-attention mechanisms in transformers has been successful in modeling dependencies and learning richer information.~\cite{dosovitskiy2020image} leverages the direct application of transformers on the image recognition task without using convolution.~\cite{carion2020end, zhu2020deformable} apply transformers to eliminate the need for many handcrafted components in conventional object detection and achieve impressive detection results. ~\cite{yang2020learning, zeng2020learning} explore transformers on image and video synthesis tasks.~\cite{zhao2020point} investigates the self-attention networks on 3D point cloud segmentation task. 

Recently, there have been several joint representation fusion research attempts. PointPainting~\cite{vora2020pointpainting} proposes a novel method that accepts both images and point clouds as inputs to do 3D object detection by appending 2D semantic segmentation labels to LiDAR points. In the visual question answering task,~\cite{hu2020iterative} jointly fuses and reasons over three different modality representations. ~\cite{mittal2020m3er} combines multimodal information to solve robust emotion recognition problems. Building on a multi-representation and multi-scale transformer, our proposed model addresses the voxel-wise, point-wise, and BEV-wise feature representation gap and enables effective cross-representation interactions with different levels of semantic features. Coupled with a point-wise mutual-relation transformer, our framework learns to capture deeper local-global structures and richer geometric relationships among point clouds.



\section{Our Approach}
\textsc{M3DETR} takes point cloud data as input and generates 3D boxes for different object categories with state-of-the-art accuracy as shown in Figure~\ref{fig:3MDETR_network}. Our goal is to perform multi-representation, multi-scale, and mutual-relation fusion with transformers over a joint embedding space. Our method consists of three main steps:

\begin{itemize}
    \item Generate feature embeddings for different point cloud representations using VoxelNet, PointNet, and 2D ConvNet.
   
    
    \item Fuse these embeddings using \textit{M3 transformer} that leverages multi-representation and multi-scale feature embedding and models mutual relationships between points.
    
    \item Perform 3D detection using detection heads network, including RPN and R-CNN stages.
\end{itemize}


\subsection{Multi-Representation Feature Embeddings}
\label{backbone}
Our network processes the raw input point clouds $\bm{P} = \{\bm{p}_1\text{,} \bm{p}_2\text{, ...,} \bm{p}_n\}$ and encodes them into three different embedding spaces, namely, voxel-, point-, and BEV-based feature representations. We discuss the embedding process for each representation in detail.

\noindent\textbf{Voxels:}
Voxel-wise feature extraction is divided into two steps: (1) the voxelization layer, which is used by VoxelNet~\cite{zhou2018voxelnet}, takes the raw input point clouds $\bm{P}$ and converts them into equally spaced 3D voxels $\bm{v} \in \mathbb{R}^{L \times W \times H}$; (2) voxel-wise features $\bm{f}^{voxel}$ at different scales are extracted with 3D sparse convolutions, which is visualized in Figure~\ref{fig:3MDETR_network}. Unlike ~\cite{zhou2018voxelnet, yan2018second}, all four different scales of obtained 3D voxel CNN features, $\bm{f}^{voxel} = \{\bm{f}^{voxel\ 1\times},\  \bm{f}^{voxel\ 2\times},\  \bm{f}^{voxel\ 4\times},\  \bm{f}^{voxel\ 8\times} \}$ are passed to the transformer stage as shown in Figure~\ref{fig:3MDETR_network}. More details will be introduced in the supplementary materials. 


\noindent\textbf{Bird's-eye-view:}
2D ConvNets takes the $8 \times$ downsampled voxel-based feature map $\bm{f}^{voxel\ 8\times}$ from the 3D Voxel CNN branch and generates a corresponding BEV-representation embedding. To directly apply 2D convolution, we start by combining the z-dimension and the channel dimension of the input voxel features into a single dimension. The structure of 2D ConvNets includes two encoder-decoder blocks, where each block consists of 2D down-sampling convolutional layers to produce top-down features, as well as de-convolutional layers to upsample to the input feature size. Specifically, both encoder and decoder paths are composed of a number of 2D convolutional layers with the kernel size of $3 \times 3$ followed by a BatchNorm layer and a ReLU layer. The output BEV-based feature from 2D ConvNets, denoted as $\bm{f}^{bev} \in \mathbb{R}^{L_{bev}\times W_{bev}\times C_{bev}}$, was further converted into keypoint feature $\bm{F}^{bev}\in \mathbb{R}^{n\times c_{bev}}$ through bi-linear interpolation. 

\noindent\textbf{Points:}
Typically, there are more than 10K raw points inside an entire point cloud scene. In order to cover the entire point set effectively without large memory consumption, we apply Furthest-Point-Sampling (FPS) algorithm to sample $n$ keypoints, denoted as $\bm{\hat{P}} \subset \bm{P}$. Adopted from PointNet++~\cite{qi2017pointnet++} and PV-RCNN~\cite{shi2020pv}, Set Abstraction and Voxel Set Abstraction (VSA) module take raw point coordinates $\bm{P}$ and the 3D voxel-based features $\bm{f}^{voxel}$, respectively, to generate keypoint features for $\bm{\hat{P}}$. In particular, the corresponding keypoint features from raw points $\bm{P}$ is $\bm{F}^{point} \in \mathbb{R}^{n\times c_{point}}$, and the corresponding keypoint features from voxels $\bm{f}^{voxel}$ are $\{\bm{F}^{voxel\ 1\times}$, $\bm{F}^{voxel\ 2\times}$, $\bm{F}^{voxel 4\times}$,  $\bm{F}^{voxel\ 8\times} \}$, where $\bm{F}^{i}\in \mathbb{R}^{n\times c_{i}}$.

\subsection{Multi-representation, Multi-scale, and Mutual-relation Transformer}
\label{m3transformer}
Once the three feature embedding sequences, $\bm{F}^{voxel}$, $\bm{F}^{point}$ and $\bm{F}^{bev}$ are generated, they are able to dynamically and intelligently attend to each other, generating final cross-representations, cross-scales, and cross-points descriptive feature representations as shown in Figure \ref{fig:3MDETR_network}. In the remainder of this section, we will briefly review transformers basics followed by discussing the multi-representation, multi-scale, and mutual-relation transformer layers. 

\noindent\textbf{Transformer basics.}
A transformer is a stacked encoder-decoder architecture relying on a self-attention mechanism to compute representations of its input and output~\cite{vaswani2017attention}. Consider two input matrices $\bm{X}_l \in \mathbb{R}^{l\times d_{in}}$ and $\bm{X}_s \in \mathbb{R}^{s \times d_{in}}$, where $l$ and $s$ are the lengths of the input sequence of dimension $d_{in}$. The attention layer output is defined as: \vspace{-3.5 mm}

$$\textrm{Attention}(\bm{Q}, \bm{V}, \bm{K}) = \textrm{Softmax}(\frac{\bm{Q}\bm{K}^{T}}{\sqrt{d_{in}}})\bm{V} \in \mathbb{R}^{l \times d_{out}},$$ 

$\textrm{where}\  \bm{Q} = \bm{X}_l \bm{W}_q,\  \bm{K} = \bm{X}_l \bm{W}_k\  and\  \bm{V} = \bm{X}_s \bm{W}_v.$

The matrices $\bm{Q}$, $\bm{K}$ and $\bm{V}$ represent query, key and value respectively, obtained through projection matrices $\bm{W}_q, \bm{W}_k\in \mathbb{R}^{d_{in}\times d_{h}}$ and $\bm{W}_v\in \mathbb{R}^{d_{in}\times d_{out}}$, where $d_{h}$ is the hidden dimension, and $d_{out}$ is the output dimension. When two input matrices $\bm{X}_l$ and $\bm{X}_s$ represent the same feature maps, the attention module is usually referred to as self-attention. Each transformer layer consists of one Multi-head Self-attention (MHSA) module and a few linear layers, as well as normalization and activation layers. More details will be introduced in the supplementary materials.

\begin{figure}[t]
    \centering
    \includegraphics[width=0.85\linewidth]{LaTeX/ACMMM/figures/Transformer_block_v1.pdf}
    \caption{\textbf{M3 Transformers} consist of two parts: a multi-representation and multi-scale transformer, and a mutual-relation transformer. Multi-representation and multi-scale transformer takes the different feature embedding and generates enriched cross-representations and cross-scales embedding. On top of it, the mutual-relation transformer further models point-wise feature relationship to extract the refined features.}
    \label{fig:3MDETR_transformer}
    \vspace{-0.3cm}
\end{figure}





Now, we present the proposed transformers used to capture the inter- and intra- interactions among input features. Specifically, we propose two stacked transformer encoder layers named M3 Transformers, as shown in Figure \ref{fig:3MDETR_transformer}: (1) the multi-representation and multi-scale transformer, and (2) the mutual-relation transformer.



\noindent\textbf{Multi-representation and multi-scale transformer layer.}
First, we focus on the intra-point representation fusion. For each individual point, the input sequence to this transformer layer is its several different corresponding point cloud features, including various scales and distinctive representations. The input sequence to the transformer is $\bm{F} = [\bm{F}^{voxel\ 1\times}$, $ \bm{F}^{voxel\ 2\times}$, $ \bm{F}^{voxel\ 4\times}$, 
$\bm{F}^{voxel\ 8\times}$, $\bm{F}^{point}$, $\bm{F}^{bev}]$, where each $\bm{F}^{i}\in \mathbb{R}^{n\times c_i}$. After explicitly modeling all element-wise interactions among those features for each point separately, the output from the block is the updated feature vectors after aggregating the information from all the input sequence $\bm{F}$.

The multi-representation and multi-scale transformer layer takes 6 different inputs: $\bm{F}^{point}$, $\bm{F}^{voxel 1\times}$, $\bm{F}^{voxel 2\times}$, $\bm{F}^{voxel 4\times}$, $\bm{F}^{voxel 8\times}$,  $\bm{F}^{bev}$. As different inputs may have different feature dimensions, we use the single-layer perceptron to apply feature reduction on the input features to align the feature dimensions of each feature embeddings. The outputs of the feature reduction layer are $\bm{\hat{F}} = [\bm{\hat{F}}^{voxel\ 1\times}$, $ \bm{\hat{F}}^{voxel\ 2\times}$, $ \bm{\hat{F}}^{voxel\ 4\times}$, $\bm{\hat{F}}^{voxel\ 8\times}$, $\bm{\hat{F}}^{point}$, $\bm{\hat{F}}^{bev}]$, where the output dimension of each feature is equivalent to $\hat{c}$.

After the feature reduction layer, the multi-representation and multi-scale transformer layer takes $\bm{\hat{F}}$ as inputs and generates self-attention features $\bm{T}^{voxel\ 1\times}$, $\bm{T}^{voxel\ 2\times}$,$\bm{T}^{voxel\ 4\times}$,$\bm{T}^{voxel\ 8\times}$, $\bm{T}^{point}$, $\bm{T}^{bev}$, which corresponds to $\bm{\hat{F}}^{voxel\ 1\times}$, $\bm{\hat{F}}^{voxel\ 2\times}$,$\bm{\hat{F}}^{voxel\ 4\times}$,$\bm{\hat{F}}^{voxel\ 8\times}$, $\bm{\hat{F}}^{point}$, $\bm{\hat{F}}^{bev}$. We visualize the multi-representation and multi-scale transformer in Figure \ref{fig:3MDETR_transformer}.



\noindent\textbf{Mutual-relation transformer layer.}
Inspired by~\cite{qi2017pointnet, qi2017pointnet++, pan20203d}, inter-points feature fusion within a spatial neighboring space is leveraged in the mutual-relation layer of the transformer. Our goal is to attend to and aggregate the neighboring information in an attention manner for each point with an enriched feature.

From the first transformer block output, we obtain aggregated features $\bm{T}$ of different scales and representations after the first transformer block. We concatenate channels along the point dimension and rearrange those points into a sequence as the input of the mutual-relation transformer. Let the learned concatenated feature $\bm{T} = 
\rm{concat}$ $\{\bm{T}^{voxel\ 1\times}$, $ \bm{T}^{voxel\ 2\times}$, $ \bm{F}^{voxel\ 4\times}$, $\bm{T}^{voxel\ 8\times}$, $\bm{T}^{point}$, $\bm{T}^{bev}\} \in\mathbb{R}^{n\times c_T}$, where $c_{T}$ is the sum of all channel size in $\bm{T}$.  Note that, the first dimension of $\bm{T}$ is the number of keypoints $n$. To model the mutual-relation between keypoints, we need to split $\bm{T}$ into $n$ point-wise feature $\bm{T} = \{ \bm{T}_i\}$, where $\bm{T}_i \in \mathbb{R}^{c_{T}}$. The concatenation and split module is showed in Figure \ref{fig:3MDETR_transformer}.

The mutual-relation transformer takes point-wise features of $n$ keypoints as inputs and uses the multi-head self-attention head to model the mutual relationship between keypoints. The outputs of the mutual-relation transformer are $\bm{\hat{T}}$.




\noindent\textbf{Comparison with previous point-based transformers}
Prior work has explored the transformer application on the task of point cloud processing. First, \cite{guo2020pct, engel2020point, zhao2020point} leverage the inherent permutation invariance of transformers to capture local context within the point cloud on the shape classification and segmentation tasks, while our \textsc{M3DETR} mainly investigates the strong attention ability of transformers between input embeddings for the 3D object detection task. Pointformer \cite{pan20203d} is the approach most related to our method because we both address the 3D object detection task by capturing the dependencies among points' features. However, Pointformer adopts a single PointNet branch to extract points feature, while \textsc{M3DETR} considers all three different representations and also applies the transformer to learn aggregated representation-based features.

\subsection{Detection Heads Network}
\label{dethead}
After we obtain the enriched embedding $\bm{\hat{T}}$ from the M3 transformer, the detection network is composed of two stages that predict 3D bounding box class, localization, and orientation in a coarse-to-fine manner, including RPN and R-CNN. Please refer to PV-RCNN~\cite{shi2020pv} for more details.

\noindent\textbf{RPN:} Region Proposal Networks take the deep semantic features $\bm{f}^{bev}$ produced by the 2D ConvNets as inputs and generate high-quality 3D object proposals $\bm{B}$, as shown in Figure \ref{fig:3MDETR_network}. A 3D object box $\bm{B}_i$ is parameterized as $(x, y, z, l, h, w, \theta) $, where $(x, y, z)$ is the center of the box, $(l , h, w)$ is the dimension of the box, and $\theta$ is the orientation in bird's-eye-view. Similar to the conventional RPN in 2D object detection~\cite{ren2015faster} each position on the deep feature map is placed by predefined bounding box anchors denoted as ($x^a$, $y^a$, $z^a$, $l^a$, $h^a$, $w^a$, $\theta^a$). Then initial proposals are generated by predicting the relative offset of an object's 3D ground truth bounding box ($x^{gt}$, $y^{gt}$, $z^{gt}$, $l^{gt}$, $h^{gt}$, $w^{gt}$, $\theta^{gt}$).







\noindent\textbf{R-CNN:}
R-CNN serves as the second stage and it takes the initial region proposals $\bm{B}$ from RPN as input to conduct further proposal refinement. For each input proposal box, the RoI-grid pooling module~\cite{shi2020pv} is adopted to extract the corresponding proposal-specific grid points' features from the transformer-based embeddings $\bm{\hat{T}}$. 
Compared with previous works, \textsc{M3DETR} leverages the richer embedding information from the learned transformers for the fine-grained proposal refinement. As the main component to extract refined features, the RoI-grid pooling module uniformly samples $ N \times N \times N $ grid points per 3D proposal. For each grid point, the output feature is generated by applying a PointNet-block \cite{qi2017pointnet} on a small number of surrounding keypoints, $M$, within its spatial surrounding region with a radius of $r$. Specifically, keypoints are the subset of input points that are sampled using the Furthest-point-sampling algorithm to cover the entire point set. 

\begin{table*}[ht]
    \centering
   
\input{LaTeX/ACMMM/table/waymo_test_all_classes}
\end{table*}
\begin{table*}[ht]
    \centering
   
\input{LaTeX/ACMMM/table/waymo_test}
\end{table*}
\begin{table*}[ht]
    \centering
\input{LaTeX/ACMMM/table/waymo_val}
\end{table*}




Finally, the refined representations of each proposal are first passed to two fully connected layers and then generate the box 3D Intersection-over-Union (IoU) guided confidence scoring of class prediction and location refinement of regression targets, $\bm{\hat{B}}$. Compared with the traditional classification-guided box scoring, 3D IoU guided confidence scoring considers the IoU between the proposal box and its corresponding ground truth box. Empirically, \cite{shi2020points,shi2020pv} show that it achieves better results compared with the traditional classification confidence based techniques. \vspace{-2 mm}



\subsection{Loss Functions} 
\label{loss}
In this section, we define the loss function. The bounding box regression target for both RPN and R-CNN stages is calculated as the relative offsets between the anchors and the ground truth as:
$\triangle x = \frac{x^{gt} - x^a}{d^a},  \triangle y = \frac{y^{gt} - y^a}{d^a}$,  
$\triangle z = \frac{z^{gt} - z^a}{h^a}$
$\triangle h = \log (\frac{h^{gt}}{{h^{a}}})$,  
$\triangle w = \log (\frac{w^{gt}}{{w^{a}}})$,
$\triangle \theta = \theta^{gt} - \theta^{a}$,
 where $ d^a $ = $\sqrt{(w^a)^2 + (l^a)^2} $. Similar to~\cite{shi2020pv, shi2020points}, the focal loss is applied \cite{lin2017focal} for the classification loss, $\mathcal{L}_{cls}$. Smooth L1 loss \cite{girshick2014rich} is adopted for the box localization regression target's losses, $\mathcal{L}_{reg}$ and $\mathcal{L}_{ref}$. In addition, 3D IoU loss \cite{shi2020pv, shi2020points} is used for $\mathcal{L}_{iou}$. 

Similar to PV-RCNN \cite{shi2020pv}, we formally define a multi-task loss for both the RPN and R-CNN stages, 
\vspace{1pt}
\begin{equation}
\begin{aligned}
 & \mathcal{L}_{cls} = -\alpha_{a}(1 - p^{a})^\gamma \log p^{a}, \\
 & \mathcal{L}_{reg} = \sum_{b \in (x, y, z, w, l, h, \theta)} \mathcal{L}_s (\triangle b), \\
 & \mathcal{L} = \mathcal{L}_{cls} + \beta_{reg} \mathcal{L}_{reg} + \beta_{iou} \mathcal{L}_{iou} + \beta_{ref} \mathcal{L}_{ref},
\end{aligned}
\end{equation}

where $p^{a}$ is the model's estimated class probability for an anchor box, and $\beta_{reg}$, $\beta_{iou} $ and $\beta_{ref}$ are chosen to balance the weights between classification loss, IoU loss and regression loss for RPN stage and R-CNN stage. We adopt the default $\alpha = 0.25$, and $\gamma = 2.0$ from the parameters of focal loss~\cite{lin2017focal}.


\section{Experiments}
In this section, we evaluate \textsc{M3DETR} both qualitatively and quantitatively on the Waymo Open Dataset~\cite{Sun_2020_CVPR} and the KITTI Dataset~\cite{geiger2012we} in the task of LiDAR-based 3D object detection. Our main results include achieving a state-of-the-art accuracy on these datasets and robustness to hyper-parameter tuning.


\subsection{Datasets and Evaluation Metric}
\label{dataset}
\noindent \textbf{Waymo Open Dataset:}
The Waymo Open Dataset~\cite{Sun_2020_CVPR} is a large-scale autonomous driving dataset containing $1000$ scenes of $20$s duration each, with $798$ scenes for training and $202$ scenes for validation. Each scene is sampled at a frequency of $10$Hz. Overall, the dataset includes $12$M labeled objects and thus we only use one fifth of the training scenes for the following experiment. We consider LiDAR data as the input to our approach. The evaluation protocol on the Waymo dataset consists of the mean average precision (mAP) and mean average precision weighted by heading (mAPH). For each object category, the detection outcomes are evaluated based on two difficulty levels: LEVEL\_1 denotes the annotated bounding box with more than 5 points and LEVEL\_2 represents the annotated bounding box with more than 1 point.

\textbf{KITTI dataset}
The KITTI 3D object detection benchmark~\cite{geiger2012we} is another popular dataset for autonomous driving. It contains $7,481$ training and $7,518$ testing LiDAR scans. We follow the standard split on the training ($3,712$ samples) and validation sets ($3,769$ samples). For each object category, the detection outcomes are evaluated based on three difficulty levels based on the object size, occlusion state, and truncation level. \vspace{-2.5 mm}


\subsection{Implementation Details}\label{impl}
Closely following the codebase\footnote{https://github.com/open-mmlab/OpenPCDet.}, we use PyTorch to implement our M3 transformer modules and integrate them into the PV-RCNN network~\cite{shi2020pv}. More details about backbone and detection heads network will be introduced in the supplementary materials.




\noindent\textbf{M3 Transformer}
We project the embeddings obtained from the backbone network with different scales and representations to 256 channels, as the input of the multi-representation and multi-scale transformer requires, and project the output features back to their original dimensions before passing into the mutual-relation transformer. Due to the GPU memory constraint, we experiment with two types of MHSA module designs: 2 encoder layers with 4 attention heads and 1 encoder layer with 8 attention heads. 



\begin{figure*}[t]
    \centering
    \includegraphics[width=1\linewidth]{LaTeX/ACMMM/figures/Visualization_4.pdf}
   
    \caption{We highlight the 3D detection results of \textbf{M3DETR} on the Waymo Open Dataset. The 3D ground truth bounding boxes are in green, while the detection bounding box are shown in blue.}
    \label{fig:visualization}
    \vspace{-0.4cm}
\end{figure*}

\noindent\textbf{Training Parameters}
Models are trained from scratch on $4$ NVIDIA P6000 GPUs. We use the Adam optimizer with a fixed weight decay of $0.01$ and use a one-cycle scheduler proposed in~\cite{smith2018disciplined}. For the Waymo Open Dataset, we train our models for $45$ epochs with a batch size of $8$ scenes and a learning rate $0.01$, which takes around $50$ hours. For the KITTI dataset, we train our models for $80$ epochs with a batch size of $8$ scenes per and a learning rate $0.01$, which takes around $15$ hours. 

\subsection{Results}
\label{result}
\noindent\textbf{Waymo Open Dataset:}
We first present our object detection results for the vehicle, pedestrian, and cyclist classes on the test set of Waymo Open Dataset in Table~\ref{tab:waymo_test_all_classes} compared with PV-RCNN~\cite{shi2020pv}. We evaluate our method at both LEVEL\_1 and LEVEL\_2 difficulty levels. Note that we reproduce the baseline PV-RCNN with a single frame input since they recently adopt two-frames input on test set. As we can see, PV-RCNN achieves 69.57\% and 63.65\% on average in LEVEL\_1 mAP and LEVEL\_2 mAP, respectively, while \textsc{M3DETR} improves them by 1.48\% and 1.85\%, respectively. Without bells and whistles, our approach works better than PV-RCNN~\cite{shi2020pv}. Furthermore, we compare our framework on the vehicle class for different distances with state-of-the-art methods, including StarNet~\cite{ngiam2019starnet}, PointPillars~\cite{lang2019pointpillars}, RCD~\cite{bewley2020range}, Det3D~\cite{zhu2019class}, RangeDet~\cite{fan2021rangedet} and PV-RCNN~\cite{shi2020pv}. In Table~\ref{tab:waymo_test}, \textsc{M3DETR} outperforms PV-RCNN significantly in both LEVEL\_1 and LEVEL\_2 difficulty levels across all distances, demonstrating the effectiveness of newly proposed framework. Moreover, we visualize the detection results of \textsc{M3DETR} in Figure ~\ref{fig:visualization}. 



Compared with the PV-RCNN shown in Figure~\ref{fig:visualization_comparison}, \textsc{M3DETR} successfully captures the inter- and intra- interactions among input features and effectively helps the model generate high-quality box proposals. To the best of our knowledge, \textsc{M3DETR} achieves the state-of-the-art in the Vehicle class in both LEVEL\_1 and LEVEL\_2 difficulty levels among all the published papers with a single frame LiDAR input.

We also evaluate the overall object detection performance with an IoU of 0.7 for Vehicle class on the full Waymo Open Dataset validation set as in Table~\ref{tab:waymo_val}, further proving that our architecture is more efficient for jointly modeling the input features.

\noindent\textbf{KITTI dataset:}
We compare our approach with the state-of-the-art methods on the KITTI test set~\cite{shi2020pv,chen2020object}. We compute the mAP on three difficult types of both car and  cyclist classes in 3D detection metric. Table~\ref{tab:kitti_test_car_cyclist} shows that \textsc{M3DETR} achieves state-of-the-art performance and outperforms the previous work by a large margin especially on the cyclist class. In particular, HotSpotNet~\cite{chen2020object} achieves $82.59\%$ in the ``easy'' categories of 3D detection metric, while \textsc{M3DETR} improves these results by significant $1.24\%$.  \vspace{-2 mm}

\begin{table}[t]
    \centering
    \setlength\tabcolsep{3px}
\input{LaTeX/ACMMM/table/kitti_test_car_cyclist}
\vspace{-0.2cm}
\end{table}

\begin{table}[t]
    \centering
    \setlength\tabcolsep{3px}
\input{LaTeX/ACMMM/table/kitti_ablation_transformer}
\vspace{-0.5cm}
\end{table}

\subsection{Ablation Studies}
\label{ablation}
To demonstrate the individual benefits of the multi-representation, multi-scale, and mutual-relation layers of the M3 transformer, we perform ablation experiments and tabulate the results in Table~\ref{tab:kitti_ablation_transformer}. All experiments are conducted on the validation set of KITTI dataset.


With the single multi-representation and multi-scale transformer layer, we can achieve 4.07\% and 1.71\% on the moderate difficulty in car class with 11 and 40 recall positions, respectively compared with the PV-RCNN baseline. On the other side, with the single mutual-relation transformer layer, the performance gain are 4.47\% and 1.99\% compared with the PV-RCNN baseline. Without hyper-parameter tuning, \textsc{M3DETR} benefits from unifying multiple point cloud representations, feature scales, and model mutual-relations simultaneously which results in the best performance.  \vspace{-2 mm}

\subsection{Robustness of M3DETR}
\label{robust}
To demonstrate the robustness of \textsc{M3DETR} to hyper-parameter tuning, we perform a series of tests by varying the sampling size, number of detection heads, and number of transformer encoder layers. We present the results of these tests in Figure~\ref{fig:M3DETR_robustness}, where we observe that \textsc{M3DETR} performs consistently well for the ``car'' category with IoU threshold of 0.7 for both 11 and 40 recall positions on the KITTI validation set.

\begin{figure}[t]
\vspace{-0.5cm}
    \centering
   
    \centerline{
    \includegraphics[width=1\linewidth]{LaTeX/ACMMM/figures/Comparison_pvrcnn_M3DETR_wo.pdf}
    }
    \vspace{-1mm}
    \caption{We visualize the 3D detection results for the same input point cloud between PV-RCNN (left) and \textbf{M3DETR} (right) on Waymo Open Dataset. We highlight the false negative boxes from PV-RCNN in red.}
   
    \label{fig:visualization_comparison}
    \vspace{-0.4cm}
\end{figure}

\begin{figure}[t]
    \centering
    \includegraphics[width=0.95\linewidth]{LaTeX/ACMMM/figures/M3DETR_Robustness.pdf}
    \vspace{-1mm}
    \caption{Performance comparison of different \textbf{M3 Transformers} variants in Car class on KITTI validation set. We show the mAP results with IoU threshold of 0.7 for both 11 and 40 recall positions.  Note that "l" and "h" represent layer number and head dimension of M3 Transformers, respectively. "Top" denotes the number of proposals used for keypoint sampling from RPN stage.}
    \label{fig:M3DETR_robustness}
    \vspace{-0.6cm}
\end{figure}
 
\section{Conclusions}
In this paper, we present \textsc{M3DETR}, a novel transformer-based framework for object detection with LiDAR point clouds. \textsc{M3DETR} is designed to simultaneously model multi-representation, multi-scale, mutual-relation features through the proposed M3 Transformers. Overall, the first transformer integrates features with different scales and representations, and the second transformer aggregates information from all keypoints. Experimental results show that \textsc{M3DETR} outperforms previous work by a large margin on the Waymo Open Dataset and the KITTI dataset. Without bells and whistles, \textsc{M3DETR} is demonstrated to be invariant to the hyper-parameters of transformer.


\noindent\textbf{Acknowledgement.} This work was supported in part by ARO Grants W911NF1910069, W911NF2110026  and U.S. Army Grant No. W911NF2120076.




{\small
\bibliographystyle{ieee_fullname}

\section{More Our Approach Details}
We further discuss our approach in the following. 

\subsection{Voxel Representation in Multi-Representation Feature Embeddings}
For the voxel-wise feature extraction from raw point clouds input, there are two steps, voxelization using voxelization layer and feature extraction using 3D sparse convolutions. We denote the size of each discretized voxel as $L \times W \times H \times C$, where $L, W, H$ indicate the length, width, and height of the voxel grid and $C$ represents the channel of the voxel features. We adopt the average of the point-wise features from all the points to represent the whole non-empty voxel feature. After voxelization, the input feature is propagated through a series of $3 \times 3 \times 3$ sparse cubes, including four consecutive blocks of 3D sparse convolution with downsampled sizes of $1 \times$, $2 \times$, $4 \times$, $8 \times$, using convolution operations of stride 2. Specifically, each sparse convolutional block includes a 3D convolution layer followed by a LayerNorm layer and a ReLU layer.








\subsection{Multi-head Self-attention Basics}
Building on the attention mechanism, Multi-head Self-attention (MHSA) with $N$ heads and the input matrix $\bm{X}$ is defined as follows: \vspace{-5mm}

\[\textrm{MHSA}(\bm{Q}, \bm{K}, \bm{V}) = \textrm{Concatenate} [\textrm{head}_1, ..., \textrm{head}_N] \bm{W} ^O, \]\vspace{-3mm}


\noindent where $ \textrm{head}_i = \textrm{Attention}(\bm{QW}_{i}^{q}, \bm{KW}_{i}^{k}, \bm{VW}_{i}^{v})$ and $\bm{Q}, \bm{K}, \bm{V}$ are linear projections of X. The projection matrices $\bm{W}^{q}_i \in \mathbb{R}^{d_h\times d_{q}},\  \bm{W}^{k}_i \in \mathbb{R}^{d_h\times d_{k}},\  \bm{W}^{v}_i \in \mathbb{R}^{d_h\times d_{v}},\  \bm{W}^O \in \mathbb{R}^{d_v\times d_{out}}$ are learnable parameters in the network that correspond to each of the attention heads, where $d_k$, $d_v$ and $d_q$ are the hidden dimensions of each attention head for $\bm{K}$, $\bm{V}$, and $\bm{Q}$, respectively. 

\section{More Implementation Details}
As mentioned in the Section 4.2, we give more details on our implementation for reproduction of our result. Our code will also be released later, including trained models that can match the performance that was included in the paper. 

\subsection{Backbone}
The 3D voxel CNN branch consists of 4 blocks of 3D sparse convolutions with output feature channel dimensions of 16, 32, 64, 64. Those 4 different voxel representations of different scales, as well as point features from point cloud input, are used to refine keypoint features by PointNet~\cite{qi2017pointnet++} through set abstraction and voxel set abstraction~\cite{shi2020pv}. The number of sampled keypoints $n$ is 2,048 for both Waymo and KITTI. In order to sample the keypoints effectively and accurately, we uses FPS on points within the range of top 1000 or 1500 initial proposals with a radius $r$ of 2.4.

\textbf{Waymo:} The voxel size is [0.1, 0.1, 0.15], and we focus on the input LiDAR point cloud range with [-75.2, 75.2], [-75.2, 75.2], and [-2, 4] meters in x, y, and z axis, respectively. The 2D ConvNets output size is $188 \times 188 \times 512$.

\textbf{KITTI:} The voxel size is [0.05, 0.05, 0.1], and we focus on the input LiDAR point cloud range with [0, 70.4], [-40, 40], and [-3, 1] meters in x, y, and z axis, respectively.  The 2D ConvNets output size is $200 \times 176 \times 512$.

\subsection{Detection Heads}
The RPN anchor size for each object category is set by computing the average of the corresponding objects from the annotated training set. The RoI-grid pooling module samples $ 6 \times 6 \times 6$ grid points within each initial 3D proposal to form a refined features. The number of surrounding points used to extract the grid point's feature, M, is 16.

During the training phase, 512 proposals are generated from RPN to R-CNN, where non-maximum suppression (NMS) with a threshold of 0.8 is applied to remove the overlapping proposals. In the validation phase, 100 proposals are fed into R-CNN, where the NMS threshold is 0.7.

{\small
\bibliographystyle{ieee_fullname}

\section{Introduction}

\section{Related Work}

\section{Method}

\section{Experiments}



{\small
\bibliographystyle{ieee_fullname}

\section{Introduction}

3D object detection is one of the fundamental tasks in autonomous driving and robotics. A primary factor contributing to the success of 3D object detection lies in the type of sensor used. For instance, point cloud data read from LiDAR sensors are more accurate than RGB images and videos obtained via cameras ~\cite{caesar2020nuscenes}. As LiDAR sensors are increasingly becoming commercially available, point clouds are being widely used in perception research for autonomous driving. The three major ways to process pointclouds are via voxels~\cite{zhou2018voxelnet,yan2018second}, raw pointclouds~\cite{qi2017pointnet,qi2017pointnet++,qi2018frustum,shi2019pointrcnn,yang2019std,liu2019point}, and bird's-eye-view (BEV) ~\cite{chen2017multi, ku2018joint, yang2018pixor}. Each representation has a unique advantage and it has been shown that combining these representations yields better results in terms of detection accuracy. However, fusing these representations is non-trivial because ... \rohan{fill}. In this work, we propose a novel method for fusing the three different representations of pointclouds for improved 3D object detection.


\rohan{In second paragraph, (1) start by discussing how prior work have fused these 3 representations. Discuss the PV-RCNN method. (2) State why it is inefficient or why it is problematic. (3) Then end the 2nd para by introducing the 3 M's and stating that they are essential for SOTA fusion.}

The VoxelNet contains convolutional neural networks, which are proved to be powerful tools in modeling vision feature, but the inputs for the VoxelNet must be voxelized and that would cause information loss. The PointNets take the raw points as inputs, but it is challenging to model points directly as the points are structureless.  2D Convolutional Neural Networks takes psuedo images as inputs. Similar with voxelization, pseudo images are generated with information lost, which hurts the accuracy of the architecture. Recent advances in 3D object detection ~\cite{tang2020searching,yang2019std} show that combining various representations of voxel, point, and BEV features and making use of VoxelNet, PointNets, and 2D Convolution neural network can significantly boost the accuracy of 3D object detection. The  Previous works ~\cite{
vora2020pointpainting, ku2018joint} also show that using various representations of 3D objects (VoxelNets, PointNets, 2D CNNs) can benefit 3D object detection, because different representations have complimentary and varying capabilities in modeling points. However, utilizing different representations of point cloud is not trivial. First, a semantic gap naturally exists between features generated by various network architectures (VoxelNet, PointNets, CNN). Next, fusing multi-scale features of PointNets and 2D/3D convolution neural networks is challenging due to variance of receptive fields. Third, convolutional neural networks are translation-invariant, but the PointNets are translation-sensitive, which may increase the difficulty of modeling multi-representation features of 3D objects.

\begin{figure}[t]
    \centering
    \includegraphics[width=\linewidth]{LaTeX/ICCV/figures/coverpic.pdf}
    \caption{Left: Multi-representation transformers; Middle: Multi-scale transformers; Right: Mutual-relation transformers.}
    \label{fig:coverpic.p}
\end{figure}


\rohan{Third paragraph should begin with how transformers can mitigate some of the problems mentioned in the second para.}

On the other hand, transformers ~\cite{vaswani2017attention} are powerful tools for modeling text and language. Transformers~\cite{hu2020iterative} also push the performance boundaries of multi-modality representations of vision and language. The mechanism of transformers is the multi-head self-attention layer, which can help the model adaptively attend to informative features and eliminate noise. Recently, transformers have been introduced in 2D object detection ~\cite{carion2020end} and 3D point cloud modeling ~\cite{pan20203d} and have made great progress in these tasks.
However, previous transformers solely focus on mutual relation~\cite{pan20203d} or spatial-temporal information~\cite{yin2020lidar} in 3D object detection. Modeling multi-representation, multi-scale features is less frequently addressed with transformers. Although transformers are capable of solving various modeling issues, there is no unified architecture that models multi-representation, multi-scale, mutual-relation features with transformers.

\subsection{Main Contributions}
In this work, we propose our approach, Multi-representation, Multi-scale, Mutual-relation 3D Object Detection with Transformers (\textsc{M3Detr}). Our approach uses transformers to fuse multimodal, multiscale, multi-location features processed by VoxelNets and 2D/3D Convolutional Neural Networks and outputs unified point-level features used for 3D object detection. \textsc{M3Detr} is also a two-stage 3D object detector. In the first stage, we use the bird's-eye-view feature map to generate rough 3D bounding boxes. In the second stage, we use point-level features to refine and filter the rough proposals from the first stage. Both the bird's-eye-view feature map and point-level features are enhanced by our proposed \textsc{M3Detr}. Our technical contributions can be summarized as follows:

\begin{enumerate}
    \item To the best of our knowledge, our proposed \textsc{M3Detr} is the first unified architecture of 3D object detection with transformers that model multi-representation, multi-scale, mutual-relation of point clouds in an end-to-end manner.
    \item Our proposed approach achieves state-of-the-art performance on KITTI 3D Object Detection Dataset ~\cite{geiger2012we} and Waymo Open Dataset ~\cite{Sun_2020_CVPR}. We outperform the previous state-of-the-art approaches by XX\%mAP on the KITTI dataset and XX\%mAP on the Waymo Open Dataset.
    \item Our proposed \textsc{M3Detr} is more robust to rotation of point clouds, which makes our architecture more reliable in real-world applications.
\end{enumerate}










\section{Related Work}

\textbf{Multi-representation modeling.}
3D point clouds have multiple data representations, including multi-view, volumetric, and point sets. Based on the intermediate representations, existing 3D object detectors in point clouds can be divided into three classes: multi-view-based, voxel-based, and point-based. Generally, multi-view-based approaches ~\cite{chen2017multi, ku2018joint, liang2018deep, liang2019multi} first project 3D point clouds into multi-view 2D images and then adopt the standard 2D object detectors to generate 3D object proposals from projected 2D feature maps. To deal with the irregular format of the input point clouds, voxel-based architectures ~\cite{zhou2018voxelnet, yan2018second, lang2019pointpillars} use equally spaced 3D voxels to encode the point clouds such that the volumetric representation can be consumed by the region proposal network (RPN). Inspired by the powerful PointNet/PointNet++~\cite{qi2017pointnet, qi2017pointnet++} whose point-based representation is invariant under transformation, ~\cite{qi2018frustum, shi2019pointrcnn} extend this method to the task of 3D object detection and directly process the raw point clouds to infer 3D bounding boxes. However, all those methods are typically limited due to either information loss or high computation cost. Recently, there is a trend to take advantage of the speed of voxel-based feature representation and the effectiveness of point-based feature representation, as shown in ~\cite{yang2019std, liu2019point, tang2020searching, shi2020pv, yin2020center}, which jointly integrate and fuse point-voxel features to make 3D object prediction.


\textbf{Multi-scale modeling.} Modeling multi-scale features is very important in computer vision. In 2D object detection, Feature Pyramid Networks (FPN) ~\cite{lin2017feature} exploits multiple scale feature maps to enlarge the receptive field and various feature maps for object detection. Inspired by FPN, Zhu et al. propose bi-directional FPN~\cite{zhu18b} to enhance the feature pyramid maps. In 3D representation, modeling multi-scale features is also popular and important. PointNet++~\cite{qi2017pointnet++} proposes the set abstraction module to model local features of a cluster of point clouds. To model multi-scale patterns of point clouds, they use 3 different sampling ranges and radii with 3 parallel Pointnets and thus fuse the multi-scale. In 3D object detection, ~\cite{hu2020you, wang2020infofocus} adopt different detection heads with multi-scale feature maps to take care of both large and small object classes. 

\textbf{Mutual-relation modeling.} 2D Convolutional Neural Networks~\cite{He_2016_CVPR} are commonly used to process mutual relation in 2D images. As point clouds are scattered and structureless, passing information from one point to its neighbours is not trivial. PointNets~\cite{qi2017pointnet++} proposes the set abstraction module to model the local context by using the subsampling and grouping layers. After this well-known work, many convolution-like operators~\cite{thomas2019kpconv,lan2019modeling,wang2019dynamic} on point clouds have been proposed to model the local context and the mutual relation between points. Recently, transformers~\cite{zhao2020point,engel2020point,guo2020pct,pan20203d} have been introduced in PointNets to model mutual relation. However, those previous works mainly focus on the local and global context of point clouds by applying mutual-relation transformers on points. Instead, our approach not only models the mutual relation between points, but it also models multi-scale and multi-representation features of point clouds.

\textbf{Transformers in computer vision.}
Inspired by its success in machine translation ~\cite{vaswani2017attention}, transformer-based architectures have recently become effective and popular in a variety of computer vision tasks. Particularly, the design of self-attention and cross-attention mechanisms in transformers has been successful in modeling dependencies and learning richer information.  ~\cite{dosovitskiy2020image} leverages the direct application of transformers on the image recognition task without using convolution. ~\cite{carion2020end, zhu2020deformable} apply transformers to eliminate the need for many handcrafted components in conventional object detection and achieve impressive detection results. ~\cite{yang2020learning, zeng2020learning} explore transformers on image and video synthesis tasks. ~\cite{zhao2020point} investigates the self-attention networks on the 3D point cloud segmentation task. 

Recently, there have been several joint representation fusion research attempts. PointPainting ~\cite{vora2020pointpainting} proposes a novel method that accepts both images and point clouds as inputs to do 3D object detection by appending 2D semantic segmentation labels to lidar points. In the visual question answering task, ~\cite{hu2020iterative} jointly fuses and reasons over three different modality representations, including question words, visual objects, and text tokens to answer the question. ~\cite{mittal2020m3er} combines multimodal information to solve robust emotion recognition problems. Building on a multi-representation and multi-scale transformer, our proposed model addresses the voxel-wise, point-wise, and BEV-wise feature representation gap and enables effective cross-representation interactions with different levels of semantic features. Coupled with a point-wise mutual-relation transformer, our framework learns to capture deeper local-global structures and richer geometric relationships among point clouds.


\section{Proposed Approach}
In this paper, we propose the M3Detr, which takes point clouds as inputs and generates accurate 3D boxes for different object categories. Our goal is to perform multi-representation, multi-scale, and mutual-relation fusion with transformers over a joint embedding space. As illustrated in Figure \ref{fig:3MDetr_network}, our method consists of three main stages: (1) a 3D voxel CNN-based, PointNet-based, and 2D ConvNets-based backbone network to encode the point cloud with rich representations; (2) a transformer-based stage to jointly fuse generated embeddings; and (3) detection heads to generate 3D object box predictions. 





\begin{figure*}[t]
    \centering
    \includegraphics[width=0.85\linewidth]{ICCV/figures/M3Detr_architecture.pdf}
    \caption{\textbf{M3Detr is a transformer network for object detection in a coarse-to-fine manner.} PointNet, VoxelNet, and 2D Convs modules enable individual multi-representation feature learning.  M3 Transformers enable inter-intra multi-representation, multiscale, multi-location feature attention. With the Region Proposal Network, the initial box proposals are generated. R-CNN captures and refines feature representations to improve detection performance.}
    \label{fig:3MDetr_network}
\end{figure*}

\subsection{Backbone Network}
\label{backbone}
The backbone network operates on the input point clouds and encodes them into three different embedding spaces, including voxel-based feature representation, point-based feature representation, and BEV-based feature representation. 

\textbf{3D Voxel CNN}
Voxel-wise feature extraction is divided into two steps: (1) the voxelization layer takes the raw input point clouds and converts them into equally spaced 3D voxels; (2) voxel-wise features at different scales are extracted with 3D sparse convolution. We denote the discretized voxel with size of $L \times W \times H \times C$, where L, W, H indicate the length, width, and height of the voxel grid and C represents the channel of the voxel features. We adopt the average of the point-wise features from all the points to represent the whole non-empty voxel feature. Normally, the input voxel feature is encoded as the sampled points' 3D coordinates and reflectance, while 0 is padded if the voxel is empty. After voxelization, the input feature is propagated through a series of $3 \times 3 \times 3$ 3D sparse blocks, including four consecutive blocks of 3D sparse convolution with downsampled sizes of $1 \times$, $2 \times$, $4 \times$, $8 \times$, respectively, to extract deeper features of different levels. Specifically, each sparse convolutional block includes a 3D convolution layer followed by a LayerNorm layer and a ReLU layer. Unlike ~\cite{zhou2018voxelnet, yan2018second}, all four different scales of obtained 3D voxel CNN features, $F^{voxel} = \{F^{voxel}_{1\times}, F^{voxel}_{2\times}, F^{voxel}_{4\times}, F^{voxel}_{8\times} \}$ are fed into a later stage, the transformer stage, in our framework. 


\textbf{PointNets}
Typically, there are more than 10k raw points inside an entire point cloud scene. Following ~\cite{qi2017pointnet++, shi2020pv}, in order to capture the entire point set effectively without large memory consumption, we apply Furthest-Point-Sampling (FPS) algorithm to sample a number $n$ of keypoints denoted as $K = \{p_1, p_2, ... p_n \}$ from the whole set of raw points. The Voxel Set Abstraction (VSA) module \cite{shi2020pv} is adopted to encode the 3D voxel-based feature to the keypoints. For example, building on the voxel-wise features, we apply a PointNet-like module to extract the aggregated features for every single keypoint. Please refer to ~\cite{shi2020pv} for more details.




\textbf{2D ConvNets}
2D ConvNets takes the $8 \times$ downsampled voxel-based feature map $F^{voxel}_{8\times}$ from the 3D Voxel CNN branch and generates a corresponding BEV-representation embedding. To directly apply 2D convolution, we start by combining the z-dimension and the channel dimension of the input voxel features into a single dimension. The structure of 2D ConvNets includes two encoder-decoder blocks, where each block consists of 2D down-sampling convolutional layers to produce top-down features and de-convolutional layers to upsample back to the input feature size. Specifically, both the encoder and decoder path are composed of a number of 2D convolutional layers with the kernel size of $3 \times 3$ followed by a BatchNorm layer and a ReLU layer. The channel of both decoder output feature maps is $C\_bev$ and the output BEV-based feature from 2D ConvNets is denoted as $F^{bev}$ with size of $L\_bev, W\_bev, C\_bev$.


\subsection{M3 Transformers}
Our transformer stage receives a sequence of elements as input, including voxel-based features with different scales represented as $\{F^{voxel}_{1\times}, F^{voxel}_{2\times}, F^{voxel}_{4\times}, F^{voxel}_{8\times}\}$, point-based features denoted as $F^{point}$, and BEV-based features as $F^{bev}$ from the output of the backbone network discussed in \ref{backbone}. Our aim is to apply a series of transformer layers over the list of all \{$F^{voxel}_{1\times}$, $F^{voxel}_{2\times}$, $F^{voxel}_{4\times}$, $F^{voxel}_{8\times}$, $F^{point}$, $F^{voxel}$\} embedding sequences to extract mutually enhanced and robust semantic features. With the multi-head self-attention mechanism of transformers, the embeddings among inter-points and intra-points is able to dynamically and freely attend to each other, generating a final cross-reprensentation, cross-scales, and cross-points descriptive feature representations.

Specifically, we propose two stacked transformer encoder layers named M3 Transformers as shown in Figure \ref{fig:3MDetr_transformer}: (1) multi-representation and multi-scale transformer layer, and (2) mutual-relation transformer layer. 






\textbf{Multi-representation and multi-scale transformer layer}
First, we focus on the intra-point representation fusion. For each individual point, the input sequence to this transformer layer is its corresponding several different point cloud features, including various scales and distinctive representations. The multi-representation and multi-scale transformer layer combines various features from \{$F^{voxel}_{1\times}, F^{voxel}_{2\times}, F^{voxel}_{4\times}, F^{voxel}_{8\times}$, $F^{point}$, $F^{voxel}$\} with each limited representation power. After the explicit modeling of all element-wise interactions among those features for each point separately, the output from the block is the updated feature element after aggregating the information from the whole input sequence.

Multi-representation and multi-scale transformer layer handles the different input channel dimensions in three steps. For a single point, the transformer layer first linearly projects all input feature representations into a common representation space by using fully connected convolution layers. Second, the transformer encoder takes those projected features as inputs and produce the corresponding enriched updated features with the same size as each of the input element being the same size. Finally, each of those features is projected back to their original input feature dimensions to preserve the size.



\textbf{Mutual-relation transformer layer}
Furthermore, we consider the pair-wise feature fusion between points. The multi-representation and multi-scale transformer is mainly motivated by the embedding representation gap for each individual point. However, inspired by ~\cite{qi2017pointnet, qi2017pointnet++, pan20203d}, inter-points feature fusion within a spatial neighboring space is leveraged in this layer of the transformer. Our goal is to attend to and aggregate the neighboring information in a attention manner for each point with an enriched feature.


From the first transformer block output, we obtain aggregated features of different scales and representations after fusion. Let the learned concatenated feature for the $i_{th}$ point $f_i = {\rm concat} \{f^{voxel}_{1\times_i}, f^{voxel}_{2\times_i}, f^{voxel}_{4\times_i}, f^{voxel}_{8\times_i}, f^{point}_i, f^{voxel}_i \}$. We concatenate channels in the point dimension and rearrange those points into a sequence as the input of the mutual-relation transformer. The mutual-relation transformer layer would take different points' concatenated features, $f = \{f_1, f_2, ... f_n \}$ as the input sequence and aggregate pair-wise information across different points.

\begin{figure}[t]
    \centering
    \includegraphics[width=0.85\linewidth]{LaTeX/ICCV/figures/Transformer_block.pdf}
    \caption{\textbf{M3 Transformers} consists of two parts: multi-representation and multi-scales transformer layer, and mutual-relation transformer layer.}
    \label{fig:3MDetr_transformer}
\end{figure}

\textbf{Comparison with previous point-based transformers}
Prior work has explored the transformer application on the task of point cloud processing. First, \cite{guo2020pct, engel2020point, zhao2020point} leverage the inherently permutation invariant of transformer to capture local context within the point cloud on the shape classification and segmentation tasks, while our M3Detr mainly investigates the strong attention ability of transformers between input embeddings for the 3D object detection task. Pointformer\cite{pan20203d} is the most related approach to our method, because we both address the 3D object detection task by capturing the dependencies among points' features. However, Pointformer adopts a single PointNet branch to extract points feature, while 3MDetr consider all three different representation and apply transformer to learn jointly aggregated representation-based features as well.


\subsection{Detection Heads}
Through the representation-aware, scale-aware, and point-wise relation-aware self-attention mechanism of M3 transformer, the embeddings dynamically attends to each other regardless of whether they are the same representation or not. After obtaining the enriched embedding from the proposed M3 transformer, the head network is composed of two stages that predict the 3D bounding box class, localization, and orientation in a coarse-to-fine manner, including RPN and R-CNN.

\textbf{RPN}
takes the aggregated semantic features produced by the M3 transformer as inputs and generates high-quality 3D object proposals. Similar to the conventional RPN in 2D object detection, each position on the aggregated feature map is placed by predefined bounding box anchors. Then proposals are generated by predicting the relative offset of an object's 3D bounding box. The specific output of RPN is the candidate proposals with object classification labels, bounding box regression, and orientation prediction. Note that the input to RPN is the combined features between the original 2D-CNN BEV embedding and the transformer-based embedding. Compared with previous works, 3MDetr leverages the richer embedding information from the learned transformers to provide better candidate region proposals. 

Here, we adopt the RoI-grid pooling module from ~\cite{shi2020pv} to extract each position's corresponding transformer-based features on the 2D-CNN BEV feature map. Previously, the RoI-grid pooling module was proposed to aggregate the keypoint features to the RoI-grid points. It takes 3D region proposals (RoI) as inputs, and the output is a list of a fixed number of grid points' feature vectors. As the main component to extract refined features, the RoI-grid pooling module uniformly samples $ N \times N \times N $ grid points per proposal. For each grid point, the output feature is generated by applying a PointNet-block \cite{qi2017pointnet} on a small number of surrounding keypoints, $M$, within its spatial surrounding region with a radius of $r$. Specifically, keypoints are the subset of input points that are sampled using the Furthest-point-sampling algorithm in order to cover the entire point set. In our work, we consider the whole input scene as a pseudo region proposal and would like to capture similar richer contextual information for the input 3D point cloud. Intuitively, the x-y plane dimension of grid points is chosen to be same as the 2D-CNN BEV feature map such that they can later be naturally fused together to form an aggregated feature map. Note that there is no need to control the grids in the z dimension. After obtaining each grid's transformer-based features, we combine it with its corresponding 2D-CNN BEV features to be the grid's final aggregated features.

\textbf{R-CNN}
serves as the second part of the detection heads, which takes the initial region proposals from RPN and conducts further proposal refinement. For each input proposal box, the RoI-grid pooling module from RPN is reused to extract the corresponding proposal-specific grid points' features from the transformer-based embeddings. Finally, the refined representations of each proposal are first fed into two fully connected layers and then predicts the box 3D Intersection-over-Union (IoU) guided confidence scoring and location refinement. Compared with the traditional classification guided box scoring, 3D IoU guided confidence scoring considers the IoU between the proposal box and its corresponding ground truth box. Empirically, \cite{shi2020points,shi2020pv} show that it achieves better results.

\begin{table*}[ht]
    \centering
   
\input{LaTeX/ICCV/table/waymo_test}
\end{table*}

\subsection{Losses}
In this section, we describe the loss function that 3MDetr adopts. The anchor and ground truth boxes are parameterized as $ (x, y, z, l, h, w, \theta) $, where $(x, y, z)$ is the center of box, $(l , h, w)$ is the dimension of box, and $\theta$ is the orientation in bird's-eye-view. The bounding box regression target is calculated as the relative offsets between the anchors and the ground truth as follows:

\begin{equation}
\begin{split}
    \triangle x = \frac{x^{gt} - x^a}{d^a},  \triangle y &= \frac{y^{gt} - y^a}{d^a},  
\triangle z = \frac{z^{gt} - z^a}{h^a}, \\
\triangle l = \log (\frac{l^{gt}}{{l^{a}}}),  \triangle h &= \log (\frac{h^{gt}}{{h^{a}}}),  
\triangle w = \log (\frac{w^{gt}}{{w^{a}}}), \\
\triangle \theta &= \theta_{gt} - \theta_{a}
\end{split}
\end{equation} 

where $x^{gt}$ and $x^a$ denote ground truth and anchor box, respectively, and $ d^a = \sqrt{(w^a)^2 + (l^a)^2} $. Similar to ~\cite{shi2020pv, shi2020points}, the focal loss is applied \cite{lin2017focal} for the classification loss. Smooth L1 loss \cite{girshick2014rich} is adopted for the box regression target's loss. In addition, 3D IoU loss \cite{shi2020pv, shi2020points} is used.

Similar to PV-RCNN \cite{shi2020pv}, we formally define a multi-task loss for both the RPN and RCNN stages, 

\begin{equation}
 L_{stage\_RPN} = L_{cls} + \beta L_{reg\_RPN},
\end{equation}

\begin{equation}
 L_{stage\_RCNN} = L_{iou} + L_{reg\_RCNN},
\end{equation}

where $\beta$ is chosen to balance the weights between classification loss and regression loss for the RPN stage. 




\section{Experiments}
In this section, we evaluate M3Detr both qualitatively and quantitatively on two challenging datasets for the LiDAR-based 3D object detection task, including the Waymo Open Dataset and the KITTI dataset. Our model outperforms previous work by a large margin on both datasets, showing the effectiveness of M3Detr for modeling and capturing richer semantic information.

\subsection{Datasets and Evaluation Metric}
\textbf{Waymo Open Dataset}
Waymo Open dataset is one of the large-scale autonomous driving datasets. In total, there are 1000 scenes of 20s duration each, sampled at 10Hz, including 12M labeled objects annotated with 798 scenes for training and 202 scenes for validation. For our experiments, we take the LiDAR input as the only input to our approach. Officially, Waymo Open Dataset evaluates the detection accuracy based on mean average precision (mAP) and mean average precision weighted by heading (mAPH). For each object category, the detection outcomes are evaluated based on two difficulty levels: LEVEL\_1 denotes the annotated bounding box with more than 5 points, and LEVEL\_2 represents the annotated bounding box with more than 1 point.


\textbf{KITTI dataset}
KITTI 3D object detection benchmark is another popular dataset for autonomous driving. It contains 7,481 training and 7,518 testing samples. We follow the standard split on the training (3,712 samples) and validation sets (3,769 samples). For each object category, the detection outcomes are evaluated based on three difficulty levels based on the object size, occlusion state, and truncation level. 


\begin{table*}[t]
    \centering
\input{LaTeX/ICCV/table/waymo_val}
\end{table*}

\subsection{Implementation Details}
Closely following the codebase ~\cite{shi2020pv}, we use PyTorch to implement our M3 transformer modules and integrate them into the vanilla PV-RCNN network. 

\textbf{Backbone Network}
The 3D voxel CNN branch consists of 4 blocks of 3D sparse convolutions with output feature channel dimensions of 16, 32, 64, 64, respectively. For the Waymo Open dataset, the voxel size is [0.1, 0.1, 0.15], and we focus on the input LiDAR point cloud range with [-75.2, 75.2], [-75.2, 75.2], and [-2, 4] meters in x, y, z axis, respectively. The sampled subset number of raw points $n$ is 4,096. For the KITTI dataset, the voxel size is [0.05, 0.05, 0.1], and we focus on the input LiDAR point cloud range with [0, 70.4], [-40, 40], and [-3, 1] meters in x, y, z axis, respectively. For the Waymo Open Dataset, the 2D ConvNets output size is $188 \times 188 \times 512$. The sampled subset number of raw points, $n$, is 4,096. For the KITTI dataset, the 2D ConvNets output size is $200 \times 176 \times 512$. The sampled subset number of raw points $n$ is 2,048. 

\textbf{M3 Transformer}
For M3 transformers, we project all the input embeddings from backbone network to be same 256-d output channels. For the multi-representation and multi-scales transformer, we adopt 4 encoder layers with 2 attention heads. On the other side, we use 4 encoder layers with 1 attention head for mutual-relation transformer. 

\textbf{Detection Heads}
The RPN anchor size for each object category is set by computing the average of the corresponding objects from the annotated training set. As mentioned before, the input to RPN is the aggregated features. We apply the RoI-grid pooling module to extract the transformer-based embedding. However, after obtaining each grid's transformer-based features, the output dimension might be different from its corresponding BEV feature dimension, so we apply two MLP layers to project the generated features to be the same dimensions as the BEV feature map. During the training phase, 512 proposals are generated from RPN to R-CNN, where NMS with a threshold of 0.8 is applied to remove the overlapping proposals. In the validation phase, 100 proposals are fed into R-CNN, where the NMS threshold is 0.7.



\textbf{Training Parameters}
Models are trained from scratch on four NVIDIA P6000 GPUs. We adopt an ADAM optimizer with a fixed weight decay of 0.01 and use a one-cycle scheduler proposed in ~\cite{smith2018disciplined}. For the Waymo Open Dataset, we train our models for 45 epochs with a batch size of 8 scenes each, which takes around 50 hours. For the KITTI dataset, we train our models for 80 epochs with a batch size of 8 scenes per each, which takes around 15 hours. 

\begin{table*}[t]
    \centering
    \setlength\tabcolsep{3px}
\input{LaTeX/ICCV/table/kitti_abaltion}
\end{table*}

\begin{table}[t]
    \centering
    \setlength\tabcolsep{3px}
\input{LaTeX/ICCV/table/kitti_test_cyclist}
\end{table}

\subsection{Main Results}
\textbf{Waymo Open Dataset}
We first present our object detection results on the test set of Waymo Open Dataset in Table~\ref{tab:waymo_test}. Particularly, we compare our framework with the state-of-the-art methods, including StarNet~\cite{ngiam2019starnet}, PointPillars~\cite{lang2019pointpillars}, RCD~\cite{bewley2020range}, Det3D~\cite{zhu2019class}, and recently published PV-RCNN~\cite{shi2020pv}. We adopt both LEVEL\_1 and LEVEL\_2 difficulty levels to evaluate object detection results on Vehicle class. Note that we reproduce the baseline PV-RCNN since they recently adopt two-frames input on the test set. As we can see from table, the bseline PV-RCNN has an mAP for LEVEL\_1 $\times \times \times$. 3MDetr improves the performance of PV-RCNN significantly, demonstrating the effectiveness of the newly proposed framework. Moreover, we visualize the detection results of 3MDetr in Figure ~\ref{}. Compared with the baseline, 3MDetr helps to ***. As of ICCV deadline, 3MDetr achieves the state-of-the-art in Vehicle class in both LEVEL\_1 and LEVEL\_2 difficulty levels among all the published paper with single frame LiDAR input.

We also evaluate the overall object detection performance with an IoU of 0.7 on the Waymo Open Dataset validation set as in Table~\ref{tab:waymo_val}, further proving that our architecture is more efficient for jointly modeling the input features.

\textbf{KITTI dataset}
Moreover, We  compare  our  framework  with  the  state-of-the-art  methods  on  the KITTI validation set, including the vanilla PV-RCNN ~\cite{shi2020pv} as our baseline, and recetly published HotSpotNet~\cite{chen2020object}. We compute the mAP on three difficult types of cyclist. Table ~\ref{tab:kitti_test} shows that 3MDetr outperforms the previous work by a large margin, especially on the BEV object detection metric. In particular, recent HotSpotNet achieves 82.59\% on the Easy level of 3D object detection and 83.29\% on the Easy level of BEV metric, 3MDetr improves it by a massive 1.24\% and 1.74\%.  




\subsection{Ablation Studies}
To understand the contribution of individual transformer layers and hyper-parameters of transformer block to the success of 3MDetr, Table. ~\ref{} summarizes the performance serveral variants of our framework. All experiments are conducted on KITTI dataset.



\textbf{M3 transformers layers}
Table ~\ref{tab:kitti_ablation} shows that the performance with different combination of transformer layers with other hyper-parameters the same. With the single multi-representation and multi-scale transformer layer, we can achieve *** compared with PV-RCNN baseline. On the other side, with the single mutual-relation transformer layer, the performance is ***. To further investigate the effects of hyperparameters design chosen in our transformer, including RCNN head design, number of heads, number of transformer encoder layers, skip connection and normalization. Table ~\ref{tab:kitti_ablation} shows that we always get the detection improvement if we use a combined BEV-based feature from the transformer. In this case, the object proposals from RCNN stage encodes richer mutually enhanced features, thus is high-quality. However, when we vary the number of heads, we get a comparable results depending on the recall rate. 

\textbf{Rotation-variant Inputs}
To further validate our model is more robust compared with the baseline, we conduct experments. Table ~\ref{} shows that ***. We argue that since 3MDetr framework involves the characteristic of multi-representation, multi-scale and mutual-position, so our model 3MDetr is very robust to noise. 








\section{Conclusions}
The proposed M3Detr is a novel transformer-based 3D object detector for LiDAR point clouds. It jointly models multi-representation, multi-scale, mutual-relation features through attention mechanism in transformer architecture. Experimental results show that M3Detr outperforms previous work on the Waymo Open Dataset and the KITTI dataset. 

{\small
\bibliographystyle{ieee_fullname}






\section{Introduction}

Please follow the steps outlined below when submitting your manuscript to
the IEEE Computer Society Press.  This style guide now has several
important modifications (for example, you are no longer warned against the
use of sticky tape to attach your artwork to the paper), so all authors
should read this new version.

\subsection{Language}

All manuscripts must be in English.

\subsection{Dual submission}

Please refer to the author guidelines on the ICCV 2021 web page for a
discussion of the policy on dual submissions.

\subsection{Paper length}
Papers, excluding the references section,
must be no longer than eight pages in length. The references section
will not be included in the page count, and there is no limit on the
length of the references section. For example, a paper of eight pages
with two pages of references would have a total length of 10 pages.
{\bf There will be no extra page charges for ICCV 2021.}

Overlength papers will simply not be reviewed.  This includes papers
where the margins and formatting are deemed to have been significantly
altered from those laid down by this style guide.  Note that this
\LaTeX\ guide already sets figure captions and references in a smaller font.
The reason such papers will not be reviewed is that there is no provision for
supervised revisions of manuscripts.  The reviewing process cannot determine
the suitability of the paper for presentation in eight pages if it is
reviewed in eleven.  

\subsection{The ruler}
The \LaTeX\ style defines a printed ruler which should be present in the
version submitted for review.  The ruler is provided in order that
reviewers may comment on particular lines in the paper without
circumlocution.  If you are preparing a document using a non-\LaTeX\
document preparation system, please arrange for an equivalent ruler to
appear on the final output pages.  The presence or absence of the ruler
should not change the appearance of any other content on the page.  The
camera ready copy should not contain a ruler. (\LaTeX\ users may uncomment
the \verb'\iccvfinalcopy' command in the document preamble.)  Reviewers:
note that the ruler measurements do not align well with lines in the paper
--- this turns out to be very difficult to do well when the paper contains
many figures and equations, and, when done, looks ugly.  Just use fractional
references (e.g.\ this line is $095.5$), although in most cases one would
expect that the approximate location will be adequate.

\subsection{Mathematics}

Please number all of your sections and displayed equations.  It is
important for readers to be able to refer to any particular equation.  Just
because you didn't refer to it in the text doesn't mean some future reader
might not need to refer to it.  It is cumbersome to have to use
circumlocutions like ``the equation second from the top of page 3 column
1''.  (Note that the ruler will not be present in the final copy, so is not
an alternative to equation numbers).  All authors will benefit from reading
Mermin's description of how to write mathematics:
\url{http://www.pamitc.org/documents/mermin.pdf}.

\subsection{Blind review}

Many authors misunderstand the concept of anonymizing for blind
review.  Blind review does not mean that one must remove
citations to one's own work---in fact it is often impossible to
review a paper unless the previous citations are known and
available.

Blind review means that you do not use the words ``my'' or ``our''
when citing previous work.  That is all.  (But see below for
tech reports.)

Saying ``this builds on the work of Lucy Smith [1]'' does not say
that you are Lucy Smith; it says that you are building on her
work.  If you are Smith and Jones, do not say ``as we show in
[7]'', say ``as Smith and Jones show in [7]'' and at the end of the
paper, include reference 7 as you would any other cited work.

An example of a bad paper just asking to be rejected:
\begin{quote}
\begin{center}
    An analysis of the frobnicatable foo filter.
\end{center}

   In this paper we present a performance analysis of our
   previous paper [1], and show it to be inferior to all
   previously known methods.  Why the previous paper was
   accepted without this analysis is beyond me.

   [1] Removed for blind review
\end{quote}

An example of an acceptable paper:

\begin{quote}
\begin{center}
     An analysis of the frobnicatable foo filter.
\end{center}

   In this paper we present a performance analysis of the
   paper of Smith \etal [1], and show it to be inferior to
   all previously known methods.  Why the previous paper
   was accepted without this analysis is beyond me.

   [1] Smith, L and Jones, C. ``The frobnicatable foo
   filter, a fundamental contribution to human knowledge''.
   Nature 381(12), 1-213.
\end{quote}

If you are making a submission to another conference at the same time,
which covers similar or overlapping material, you may need to refer to that
submission in order to explain the differences, just as you would if you
had previously published related work.  In such cases, include the
anonymized parallel submission~\cite{Authors14} as additional material and
cite it as
\begin{quote}
[1] Authors. ``The frobnicatable foo filter'', F\&G 2014 Submission ID 324,
Supplied as additional material {\tt fg324.pdf}.
\end{quote}

Finally, you may feel you need to tell the reader that more details can be
found elsewhere, and refer them to a technical report.  For conference
submissions, the paper must stand on its own, and not {\em require} the
reviewer to go to a tech report for further details.  Thus, you may say in
the body of the paper ``further details may be found
in~\cite{Authors14b}''.  Then submit the tech report as additional material.
Again, you may not assume the reviewers will read this material.

Sometimes your paper is about a problem which you tested using a tool which
is widely known to be restricted to a single institution.  For example,
let's say it's 1969, you have solved a key problem on the Apollo lander,
and you believe that the ICCV70 audience would like to hear about your
solution.  The work is a development of your celebrated 1968 paper entitled
``Zero-g frobnication: How being the only people in the world with access to
the Apollo lander source code makes us a wow at parties'', by Zeus \etal.

You can handle this paper like any other.  Don't write ``We show how to
improve our previous work [Anonymous, 1968].  This time we tested the
algorithm on a lunar lander [name of lander removed for blind review]''.
That would be silly, and would immediately identify the authors. Instead
write the following:
\begin{quotation}
\noindent
   We describe a system for zero-g frobnication.  This
   system is new because it handles the following cases:
   A, B.  Previous systems [Zeus et al. 1968] didn't
   handle case B properly.  Ours handles it by including
   a foo term in the bar integral.

   ...

   The proposed system was integrated with the Apollo
   lunar lander, and went all the way to the moon, don't
   you know.  It displayed the following behaviours
   which show how well we solved cases A and B: ...
\end{quotation}
As you can see, the above text follows standard scientific convention,
reads better than the first version, and does not explicitly name you as
the authors.  A reviewer might think it likely that the new paper was
written by Zeus \etal, but cannot make any decision based on that guess.
He or she would have to be sure that no other authors could have been
contracted to solve problem B.
\medskip

\noindent
FAQ\medskip\\
{\bf Q:} Are acknowledgements OK?\\
{\bf A:} No.  Leave them for the final copy.\medskip\\
{\bf Q:} How do I cite my results reported in open challenges?
{\bf A:} To conform with the double blind review policy, you can report results of other challenge participants together with your results in your paper. For your results, however, you should not identify yourself and should not mention your participation in the challenge. Instead present your results referring to the method proposed in your paper and draw conclusions based on the experimental comparison to other results.\medskip\\

\begin{figure}[t]
\begin{center}
\fbox{\rule{0pt}{2in} \rule{0.9\linewidth}{0pt}}
  
\end{center}
   \caption{Example of caption.  It is set in Roman so that mathematics
   (always set in Roman: $B \sin A = A \sin B$) may be included without an
   ugly clash.}
\label{fig:long}
\label{fig:onecol}
\end{figure}

\subsection{Miscellaneous}

\noindent
Compare the following:\\
\begin{tabular}{ll}
 \verb'$conf_a$' &  $conf_a$ \\
 \verb'$\mathit{conf}_a$' & $\mathit{conf}_a$
\end{tabular}\\
See The \TeX book, p165.

The space after \eg, meaning ``for example'', should not be a
sentence-ending space. So \eg is correct, {\em e.g.} is not.  The provided
\verb'\eg' macro takes care of this.

When citing a multi-author paper, you may save space by using ``et alia'',
shortened to ``\etal'' (not ``{\em et.\ al.}'' as ``{\em et}'' is a complete word.)
However, use it only when there are three or more authors.  Thus, the
following is correct: ``
   Frobnication has been trendy lately.
   It was introduced by Alpher~\cite{Alpher02}, and subsequently developed by
   Alpher and Fotheringham-Smythe~\cite{Alpher03}, and Alpher \etal~\cite{Alpher04}.''

This is incorrect: ``... subsequently developed by Alpher \etal~\cite{Alpher03} ...''
because reference~\cite{Alpher03} has just two authors.  If you use the
\verb'\etal' macro provided, then you need not worry about double periods
when used at the end of a sentence as in Alpher \etal.

For this citation style, keep multiple citations in numerical (not
chronological) order, so prefer \cite{Alpher03,Alpher02,Authors14} to
\cite{Alpher02,Alpher03,Authors14}.

\begin{figure*}
\begin{center}
\fbox{\rule{0pt}{2in} \rule{.9\linewidth}{0pt}}
\end{center}
   \caption{Example of a short caption, which should be centered.}
\label{fig:short}
\end{figure*}

\section{Formatting your paper}

All text must be in a two-column format. The total allowable width of the
text area is $6\frac78$ inches (17.5 cm) wide by $8\frac78$ inches (22.54
cm) high. Columns are to be $3\frac14$ inches (8.25 cm) wide, with a
$\frac{5}{16}$ inch (0.8 cm) space between them. The main title (on the
first page) should begin 1.0 inch (2.54 cm) from the top edge of the
page. The second and following pages should begin 1.0 inch (2.54 cm) from
the top edge. On all pages, the bottom margin should be 1-1/8 inches (2.86
cm) from the bottom edge of the page for $8.5 \times 11$-inch paper; for A4
paper, approximately 1-5/8 inches (4.13 cm) from the bottom edge of the
page.

\subsection{Margins and page numbering}

All printed material, including text, illustrations, and charts, must be kept
within a print area 6-7/8 inches (17.5 cm) wide by 8-7/8 inches (22.54 cm)
high.

Page numbers should be included for review submissions but not for the 
final paper. Review submissions papers should have page numbers in the 
footer with numbers centered and .75 inches (1.905 cm) from the bottom 
of the page and start on the first page with the number 1.

Page numbers will be added by the publisher to all camera-ready papers 
prior to including them in the proceedings and before submitting the 
papers to IEEE Xplore. As such, your camera-ready submission should 
not include any page numbers. Page numbers should automatically be 
removed by uncommenting (if it's not already) the line
\begin{verbatim}
\end{verbatim}
near the beginning of the .tex file.

\subsection{Type-style and fonts}

Wherever Times is specified, Times Roman may also be used. If neither is
available on your word processor, please use the font closest in
appearance to Times to which you have access.

MAIN TITLE. Center the title 1-3/8 inches (3.49 cm) from the top edge of
the first page. The title should be in Times 14-point, boldface type.
Capitalize the first letter of nouns, pronouns, verbs, adjectives, and
adverbs; do not capitalize articles, coordinate conjunctions, or
prepositions (unless the title begins with such a word). Leave two blank
lines after the title.

AUTHOR NAME(s) and AFFILIATION(s) are to be centered beneath the title
and printed in Times 12-point, non-boldface type. This information is to
be followed by two blank lines.

The ABSTRACT and MAIN TEXT are to be in a two-column format.

MAIN TEXT. Type main text in 10-point Times, single-spaced. Do NOT use
double-spacing. All paragraphs should be indented 1 pica (approx. 1/6
inch or 0.422 cm). Make sure your text is fully justified---that is,
flush left and flush right. Please do not place any additional blank
lines between paragraphs.

Figure and table captions should be 9-point Roman type as in
Figures~\ref{fig:onecol} and~\ref{fig:short}.  Short captions should be centered.

\noindent Callouts should be 9-point Helvetica, non-boldface type.
Initially capitalize only the first word of section titles and first-,
second-, and third-order headings.

FIRST-ORDER HEADINGS. (For example, {\large \bf 1. Introduction})
should be Times 12-point boldface, initially capitalized, flush left,
with one blank line before, and one blank line after.

SECOND-ORDER HEADINGS. (For example, { \bf 1.1. Database elements})
should be Times 11-point boldface, initially capitalized, flush left,
with one blank line before, and one after. If you require a third-order
heading (we discourage it), use 10-point Times, boldface, initially
capitalized, flush left, preceded by one blank line, followed by a period
and your text on the same line.

\subsection{Footnotes}

Please use footnotes\footnote {This is what a footnote looks like.  It
often distracts the reader from the main flow of the argument.} sparingly.
Indeed, try to avoid footnotes altogether and include necessary peripheral
observations in
the text (within parentheses, if you prefer, as in this sentence).  If you
wish to use a footnote, place it at the bottom of the column on the page on
which it is referenced. Use Times 8-point type, single-spaced.

\subsection{References}

List and number all bibliographical references in 9-point Times,
single-spaced, at the end of your paper. When referenced in the text,
enclose the citation number in square brackets, for
example~\cite{Authors14}.  Where appropriate, include the name(s) of
editors of referenced books.

\begin{table}
\begin{center}
\begin{tabular}{|l|c|}
\hline
Method & Frobnability \\
\hline\hline
Theirs & Frumpy \\
Yours & Frobbly \\
Ours & Makes one's heart Frob\\
\hline
\end{tabular}
\end{center}
\caption{Results.   Ours is better.}
\end{table}

\subsection{Illustrations, graphs, and photographs}

All graphics should be centered.  Please ensure that any point you wish to
make is resolvable in a printed copy of the paper.  Resize fonts in figures
to match the font in the body text, and choose line widths which render
effectively in print.  Many readers (and reviewers), even of an electronic
copy, will choose to print your paper in order to read it.  You cannot
insist that they do otherwise, and therefore must not assume that they can
zoom in to see tiny details on a graphic.

When placing figures in \LaTeX, it's almost always best to use
\verb+\includegraphics+, and to specify the  figure width as a multiple of
the line width as in the example below
{\small\begin{verbatim}
   \usepackage[dvips]{graphicx} ...
   \includegraphics[width=0.8\linewidth]
                   {myfile.eps}
\end{verbatim}
}

\subsection{Color}

Please refer to the author guidelines on the ICCV 2021 web page for a discussion
of the use of color in your document.

\section{Final copy}

You must include your signed IEEE copyright release form when you submit
your finished paper. We MUST have this form before your paper can be
published in the proceedings.

{\small
\bibliographystyle{ieee_fullname}

\section{Introduction}
3D object detection is a fundamental problem in computer vision and many applications, including autonomous driving~\cite{geiger2012we, Sun_2020_CVPR}, augmented reality~\cite{park2008multiple} and robotics~\cite{oh2002development}.
Moreover, different methods have been proposed for various sensors, including monocular cameras, depth cameras, LiDAR, and radars ~\cite{qi2017pointnet,qi2017pointnet++,qi2018frustum,yang2019std,ye2020hvnet,yang20203dssd,shi2020pv,yin2020center}. 2D object detection deals with detecting objects from RGB images and videos, while 3D object detection utilizes point cloud-based representations obtained from LiDARs or other sensors. Moreover, it is known that point cloud data obtained from LiDAR sensors tends to be more accurate than RGB images and videos~\cite{caesar2020nuscenes}. Consequently, point clouds are being widely used for scene understanding in autonomous driving and AR. 

\begin{figure}[t]
    \centering
    \includegraphics[width=\linewidth]{LaTeX/ACMMM/figures/coverpic.pdf}
    \caption{M3 Transformers used in our approach: Left: Multi-representation transformers; Middle: Multi-scale transformers; Right: Mutual-relation transformers. We use these characteristics to present a novel 3D object detection architecture.}
    \label{fig:coverpic.p}
\vspace{-0.5cm}
\end{figure}

The previous
state-of-the-art methods for 3D object detection base on different networks~\cite{yin2020center,shi2020pv}. However, there are two key limitations:

\noindent{\textbf{Ineffective point cloud representations:}} The three major techniques used to process point clouds are based on voxels~\cite{zhou2018voxelnet,yan2018second}, raw point clouds~\cite{qi2017pointnet,qi2017pointnet++,qi2018frustum,shi2019pointrcnn,yang2019std,liu2019point, qin2020weakly}, and bird's-eye-view ~\cite{chen2017multi, ku2018joint, yang2018pixor}. Each representation has a unique advantage and it has been shown that combining these representations can result in terms of detection accuracy \cite{qi2018frustum,shi2020pv,chen2019fast}. However, fusing these representations is non-trivial. First, the architectures corresponding to the VoxelNets, the PointNets, and the 2D convolutional neural networks are  different. Moreover, raw point clouds need to be converted to voxels and pixels before techniques based on VoxelNets and 2D convolutional neural networks can be applied. The differences between the inputs of these three neural models can result in semantic gaps. Previous works~\cite{chen2019fast,shi2020pv} tend to use feature concatenation and attention modules to fuse multi-representation features. However, the correlation between features of different representations has not been addressed.

\noindent{\textbf{Insufficient modeling of multi-scale features:}} Fusing multi-scale feature maps is a well-known technique used for improving the detection performance of 2D object detection. In terms of 3D object detection, current approaches tend to use multi-scale feature pyramids~\cite{qi2017pointnet,qi2017pointnet++,yang20203dssd,lan2019modeling,lang2019pointpillars,ye2020hvnet}. However, fusing these multiple feature pyramids is non-trivial because the higher resolution and the larger receptive fields are conflicting \cite{lin2017feature, qi2017pointnet++}. Existing methods~\cite{qi2017pointnet++,ye2020hvnet,shi2020pv} fuse the multi-scale features using bi-linear down-sampling/up-sampling and concatenation. Although these approaches~\cite{qi2017pointnet++,ye2020hvnet,shi2020pv} can improve the accuracy by a large margin, there are many challenges with respect to the underlying fusion method  in terms of the correlation between feature maps of different scales.

A key issue in terms of designing a good approach for object detection is exploiting the correlation between different representations and the large size of the receptive fields.  Our approach is motivated by use of transformers~\cite{vaswani2017attention}, a form of neural network based on attention that has been used in natural language processing ~\cite{hu2020iterative,tan2019lxmert,su2019vl,lu2019vilbert}. Specifically, transformers use multi-head attention to narrow the semantic gap between different representations by adapting to informative features and eliminating noise.

Another key aspect of 3D object detection is to model the mutual relationships between different points in the point cloud data~\cite{qi2017pointnet,qi2017pointnet++,lan2019modeling,thomas2019kpconv}. Modeling mutual relationships can enhance the ability to recognize the fine-grained patterns and can generalize to complex scenes.
Prior works in 3D object detection have modeled these relationships using multi-layer perceptrons~\cite{qi2017pointnet,qi2017pointnet++,qi2018frustum,shi2020pv}, farthest point sampling layers~\cite{qi2017pointnet++,shi2019pointrcnn,shi2020pv}, max pooling layers~\cite{lan2019modeling,qi2017pointnet++,shi2019pointrcnn,shi2020pv}, and graph convolutional networks~\cite{wang2019dynamic}. However, a key challenge is to model these mutual relationships along  with fusing different representations and multi-scale features.

We  present \textsc{M3DETR}, a novel two-stage architecture for 3D object detection task. Given raw 3D point cloud data, our approach can localize static and dynamic obstacles with state-of-the-art accuracy. As illustrated in Figure~\ref{fig:coverpic.p}, the key components of our approach are M3 transformers, which are used to combine different feature representations. Conceptually, each of the M3 transformers are used for aggregating point cloud representations, multi-scale representations and mutual relationships among a subset of points in the point cloud data.
\\

\noindent \textbf{Summary of contributions:}
\begin{itemize}
    \item \textsc{M3DETR} is the first unified architecture for 3D object detection with transformers that accounts for multi-representation, multi-scale, mutual-relation models of point clouds in an end-to-end manner.

    \item \textsc{M3DETR} is robust and insensitive with respect to the hyper-parameters of transformer architectures.
    We test multiple variants with different transformer blocks designs.
    We demonstrate improved performance of \textsc{M3DETR}  regardless of hyper-parameters.

    \item Our unified architecture achieves state-of-the-art performance on KITTI 3D Object Detection Dataset~\cite{geiger2012we} and Waymo Open Dataset~\cite{Sun_2020_CVPR}. We outperform the previous state-of-the-art approaches by 2.86\% mAP for car class on the Waymo validation set and 1.48\% mAP for all classes on the Waymo test set.
\end{itemize}

\section{Related Work}
\textbf{Multi-representation modeling.}
Existing techniques for modeling 3D point cloud data include bird-eye-view (BEV), volumetric, and point-wise representations. 
Generally, BEV-based approaches ~\cite{chen2017multi, ku2018joint, liang2018deep, liang2019multi} first project 3D point clouds into 2D BEV space and then adopt the standard 2D object detectors to generate 3D object proposals from projected 2D feature maps. To deal with the irregular format of input point clouds, voxel-based architectures~\cite{zhou2018voxelnet, yan2018second, lang2019pointpillars} use equally spaced 3D voxels to encode the point clouds such that the volumetric representation can be consumed by the region proposal network (RPN) ~\cite{ren2015faster}. Inspired by the PointNet/PointNet++ approach~\cite{qi2017pointnet, qi2017pointnet++}, which is invariant under transformation,~\cite{qi2018frustum, shi2019pointrcnn} extend this method to the task of 3D object detection and directly process the raw point clouds to infer 3D bounding boxes. However, these methods are typically limited due to either information loss or high computation cost. Recently, many approaches~\cite{yang2019std, liu2019point, tang2020searching, shi2020pv, yin2020center} have combined the advantages of speed (of voxel-based representation) and efficiency (of point-based representation) by fusing point-voxel features for 3D object prediction.

\textbf{Multi-scale modeling.} Modeling multi-scale features is an important procedure in deep learning-based computer vision~\cite{lin2017feature, cai2019guided, liu2019l2g, zhang2020feature, liao2020mmnet, zhu18b,qi2017pointnet++, hu2020you, wang2020infofocus} because it is able to enlarge the receptive field and increase resolution. In 3D representation, modeling multi-scale features is also popular and important. PointNet++~\cite{qi2017pointnet++} proposes the set abstraction module to model local features of a cluster of point clouds. To model multi-scale patterns of point clouds, they use 3 different sampling ranges and radii with 3 parallel PointNets and thus fuse the multi-scale. In 3D object detection,~\cite{hu2020you, wang2020infofocus, ye2020hvnet} adopt different detection heads with multi-scale feature maps to handle both large and small object classes. 

\begin{figure*}[t]
    \centering
    \includegraphics[width=\linewidth]{LaTeX/ACMMM/figures/arch.pdf}
    \caption{\textbf{An overview of our \textsc{M3DETR} architecture: \textsc{M3DETR} is a transformer based framework for object detection in a coarse-to-fine manner.} It consists of three parts. PointNets, VoxelNet, and 2D ConvNets modules enable individual multi-representation feature learning.  M3 Transformers enable inter-intra multi-representation, multi-scale, multi-location feature attention. With the Region Proposal Network (RPN), the initial box proposals are generated. R-CNN captures and refines region-wise feature representations from M3 transformer output to improve detection performance. }
    \label{fig:3MDETR_network}
    \vspace{-0.2cm}
\end{figure*}

\textbf{Mutual-relation modeling.} 2D Convolutional Neural Networks \cite{He_2016_CVPR} are commonly used to process mutual relations in 2D images. As point clouds are scattered and lacking structure, passing information from one point to its neighbors is not trivial. PointNets~\cite{qi2017pointnet++} proposes the set abstraction module to model the local context by using the subsampling and grouping layers. After this well-known work, many convolution-like operators~\cite{thomas2019kpconv,lan2019modeling,wang2019dynamic} on point clouds have been proposed to model the local context and the mutual relation between points. Recently, transformers~\cite{zhao2020point,engel2020point,guo2020pct,pan20203d} have been introduced in PointNets to model mutual relation. However, those previous works mainly focus on the local and global contexts of point clouds by applying mutual-relation transformers on points. Instead, our approach not only models the mutual relation between points, but it also models multi-scale and multi-representation features of point clouds.


\textbf{Transformers in computer vision.}
Inspired by their success in machine translation ~\cite{vaswani2017attention}, transformer-based architectures have recently become effective and popular in a variety of computer vision tasks. Particularly, the design of self-attention and cross-attention mechanisms in transformers has been successful in modeling dependencies and learning richer information.~\cite{dosovitskiy2020image} leverages the direct application of transformers on the image recognition task without using convolution.~\cite{carion2020end, zhu2020deformable} apply transformers to eliminate the need for many handcrafted components in conventional object detection and achieve impressive detection results. ~\cite{yang2020learning, zeng2020learning} explore transformers on image and video synthesis tasks.~\cite{zhao2020point} investigates the self-attention networks on 3D point cloud segmentation task. 

Recently, there have been several joint representation fusion research attempts. PointPainting~\cite{vora2020pointpainting} proposes a novel method that accepts both images and point clouds as inputs to do 3D object detection by appending 2D semantic segmentation labels to LiDAR points. In the visual question answering task,~\cite{hu2020iterative} jointly fuses and reasons over three different modality representations. ~\cite{mittal2020m3er} combines multimodal information to solve robust emotion recognition problems. Building on a multi-representation and multi-scale transformer, our proposed model addresses the voxel-wise, point-wise, and BEV-wise feature representation gap and enables effective cross-representation interactions with different levels of semantic features. Coupled with a point-wise mutual-relation transformer, our framework learns to capture deeper local-global structures and richer geometric relationships among point clouds.



\section{Our Approach}
\textsc{M3DETR} takes point cloud data as input and generates 3D boxes for different object categories with state-of-the-art accuracy as shown in Figure~\ref{fig:3MDETR_network}. Our goal is to perform multi-representation, multi-scale, and mutual-relation fusion with transformers over a joint embedding space. Our method consists of three main steps:

\begin{itemize}
    \item Generate feature embeddings for different point cloud representations using VoxelNet, PointNet, and 2D ConvNet.
   
    
    \item Fuse these embeddings using \textit{M3 transformer} that leverages multi-representation and multi-scale feature embedding and models mutual relationships between points.
    
    \item Perform 3D detection using detection heads network, including RPN and R-CNN stages.
\end{itemize}


\subsection{Multi-Representation Feature Embeddings}
\label{backbone}
Our network processes the raw input point clouds $\bm{P} = \{\bm{p}_1\text{,} \bm{p}_2\text{, ...,} \bm{p}_n\}$ and encodes them into three different embedding spaces, namely, voxel-, point-, and BEV-based feature representations. We discuss the embedding process for each representation in detail.

\noindent\textbf{Voxels:}
Voxel-wise feature extraction is divided into two steps: (1) the voxelization layer, which is used by VoxelNet~\cite{zhou2018voxelnet}, takes the raw input point clouds $\bm{P}$ and converts them into equally spaced 3D voxels $\bm{v} \in \mathbb{R}^{L \times W \times H}$; (2) voxel-wise features $\bm{f}^{voxel}$ at different scales are extracted with 3D sparse convolutions, which is visualized in Figure~\ref{fig:3MDETR_network}. Unlike ~\cite{zhou2018voxelnet, yan2018second}, all four different scales of obtained 3D voxel CNN features, $\bm{f}^{voxel} = \{\bm{f}^{voxel\ 1\times},\  \bm{f}^{voxel\ 2\times},\  \bm{f}^{voxel\ 4\times},\  \bm{f}^{voxel\ 8\times} \}$ are passed to the transformer stage as shown in Figure~\ref{fig:3MDETR_network}. More details will be introduced in the supplementary materials. 


\noindent\textbf{Bird's-eye-view:}
2D ConvNets takes the $8 \times$ downsampled voxel-based feature map $\bm{f}^{voxel\ 8\times}$ from the 3D Voxel CNN branch and generates a corresponding BEV-representation embedding. To directly apply 2D convolution, we start by combining the z-dimension and the channel dimension of the input voxel features into a single dimension. The structure of 2D ConvNets includes two encoder-decoder blocks, where each block consists of 2D down-sampling convolutional layers to produce top-down features, as well as de-convolutional layers to upsample to the input feature size. Specifically, both encoder and decoder paths are composed of a number of 2D convolutional layers with the kernel size of $3 \times 3$ followed by a BatchNorm layer and a ReLU layer. The output BEV-based feature from 2D ConvNets, denoted as $\bm{f}^{bev} \in \mathbb{R}^{L_{bev}\times W_{bev}\times C_{bev}}$, was further converted into keypoint feature $\bm{F}^{bev}\in \mathbb{R}^{n\times c_{bev}}$ through bi-linear interpolation. 

\noindent\textbf{Points:}
Typically, there are more than 10K raw points inside an entire point cloud scene. In order to cover the entire point set effectively without large memory consumption, we apply Furthest-Point-Sampling (FPS) algorithm to sample $n$ keypoints, denoted as $\bm{\hat{P}} \subset \bm{P}$. Adopted from PointNet++~\cite{qi2017pointnet++} and PV-RCNN~\cite{shi2020pv}, Set Abstraction and Voxel Set Abstraction (VSA) module take raw point coordinates $\bm{P}$ and the 3D voxel-based features $\bm{f}^{voxel}$, respectively, to generate keypoint features for $\bm{\hat{P}}$. In particular, the corresponding keypoint features from raw points $\bm{P}$ is $\bm{F}^{point} \in \mathbb{R}^{n\times c_{point}}$, and the corresponding keypoint features from voxels $\bm{f}^{voxel}$ are $\{\bm{F}^{voxel\ 1\times}$, $\bm{F}^{voxel\ 2\times}$, $\bm{F}^{voxel 4\times}$,  $\bm{F}^{voxel\ 8\times} \}$, where $\bm{F}^{i}\in \mathbb{R}^{n\times c_{i}}$.

\subsection{Multi-representation, Multi-scale, and Mutual-relation Transformer}
\label{m3transformer}
Once the three feature embedding sequences, $\bm{F}^{voxel}$, $\bm{F}^{point}$ and $\bm{F}^{bev}$ are generated, they are able to dynamically and intelligently attend to each other, generating final cross-representations, cross-scales, and cross-points descriptive feature representations as shown in Figure \ref{fig:3MDETR_network}. In the remainder of this section, we will briefly review transformers basics followed by discussing the multi-representation, multi-scale, and mutual-relation transformer layers. 

\noindent\textbf{Transformer basics.}
A transformer is a stacked encoder-decoder architecture relying on a self-attention mechanism to compute representations of its input and output~\cite{vaswani2017attention}. Consider two input matrices $\bm{X}_l \in \mathbb{R}^{l\times d_{in}}$ and $\bm{X}_s \in \mathbb{R}^{s \times d_{in}}$, where $l$ and $s$ are the lengths of the input sequence of dimension $d_{in}$. The attention layer output is defined as: \vspace{-3.5 mm}

$$\textrm{Attention}(\bm{Q}, \bm{V}, \bm{K}) = \textrm{Softmax}(\frac{\bm{Q}\bm{K}^{T}}{\sqrt{d_{in}}})\bm{V} \in \mathbb{R}^{l \times d_{out}},$$ 

$\textrm{where}\  \bm{Q} = \bm{X}_l \bm{W}_q,\  \bm{K} = \bm{X}_l \bm{W}_k\  and\  \bm{V} = \bm{X}_s \bm{W}_v.$

The matrices $\bm{Q}$, $\bm{K}$ and $\bm{V}$ represent query, key and value respectively, obtained through projection matrices $\bm{W}_q, \bm{W}_k\in \mathbb{R}^{d_{in}\times d_{h}}$ and $\bm{W}_v\in \mathbb{R}^{d_{in}\times d_{out}}$, where $d_{h}$ is the hidden dimension, and $d_{out}$ is the output dimension. When two input matrices $\bm{X}_l$ and $\bm{X}_s$ represent the same feature maps, the attention module is usually referred to as self-attention. Each transformer layer consists of one Multi-head Self-attention (MHSA) module and a few linear layers, as well as normalization and activation layers. More details will be introduced in the supplementary materials.

\begin{figure}[t]
    \centering
    \includegraphics[width=0.85\linewidth]{LaTeX/ACMMM/figures/Transformer_block_v1.pdf}
    \caption{\textbf{M3 Transformers} consist of two parts: a multi-representation and multi-scale transformer, and a mutual-relation transformer. Multi-representation and multi-scale transformer takes the different feature embedding and generates enriched cross-representations and cross-scales embedding. On top of it, the mutual-relation transformer further models point-wise feature relationship to extract the refined features.}
    \label{fig:3MDETR_transformer}
    \vspace{-0.3cm}
\end{figure}





Now, we present the proposed transformers used to capture the inter- and intra- interactions among input features. Specifically, we propose two stacked transformer encoder layers named M3 Transformers, as shown in Figure \ref{fig:3MDETR_transformer}: (1) the multi-representation and multi-scale transformer, and (2) the mutual-relation transformer.



\noindent\textbf{Multi-representation and multi-scale transformer layer.}
First, we focus on the intra-point representation fusion. For each individual point, the input sequence to this transformer layer is its several different corresponding point cloud features, including various scales and distinctive representations. The input sequence to the transformer is $\bm{F} = [\bm{F}^{voxel\ 1\times}$, $ \bm{F}^{voxel\ 2\times}$, $ \bm{F}^{voxel\ 4\times}$, 
$\bm{F}^{voxel\ 8\times}$, $\bm{F}^{point}$, $\bm{F}^{bev}]$, where each $\bm{F}^{i}\in \mathbb{R}^{n\times c_i}$. After explicitly modeling all element-wise interactions among those features for each point separately, the output from the block is the updated feature vectors after aggregating the information from all the input sequence $\bm{F}$.

The multi-representation and multi-scale transformer layer takes 6 different inputs: $\bm{F}^{point}$, $\bm{F}^{voxel 1\times}$, $\bm{F}^{voxel 2\times}$, $\bm{F}^{voxel 4\times}$, $\bm{F}^{voxel 8\times}$,  $\bm{F}^{bev}$. As different inputs may have different feature dimensions, we use the single-layer perceptron to apply feature reduction on the input features to align the feature dimensions of each feature embeddings. The outputs of the feature reduction layer are $\bm{\hat{F}} = [\bm{\hat{F}}^{voxel\ 1\times}$, $ \bm{\hat{F}}^{voxel\ 2\times}$, $ \bm{\hat{F}}^{voxel\ 4\times}$, $\bm{\hat{F}}^{voxel\ 8\times}$, $\bm{\hat{F}}^{point}$, $\bm{\hat{F}}^{bev}]$, where the output dimension of each feature is equivalent to $\hat{c}$.

After the feature reduction layer, the multi-representation and multi-scale transformer layer takes $\bm{\hat{F}}$ as inputs and generates self-attention features $\bm{T}^{voxel\ 1\times}$, $\bm{T}^{voxel\ 2\times}$,$\bm{T}^{voxel\ 4\times}$,$\bm{T}^{voxel\ 8\times}$, $\bm{T}^{point}$, $\bm{T}^{bev}$, which corresponds to $\bm{\hat{F}}^{voxel\ 1\times}$, $\bm{\hat{F}}^{voxel\ 2\times}$,$\bm{\hat{F}}^{voxel\ 4\times}$,$\bm{\hat{F}}^{voxel\ 8\times}$, $\bm{\hat{F}}^{point}$, $\bm{\hat{F}}^{bev}$. We visualize the multi-representation and multi-scale transformer in Figure \ref{fig:3MDETR_transformer}.



\noindent\textbf{Mutual-relation transformer layer.}
Inspired by~\cite{qi2017pointnet, qi2017pointnet++, pan20203d}, inter-points feature fusion within a spatial neighboring space is leveraged in the mutual-relation layer of the transformer. Our goal is to attend to and aggregate the neighboring information in an attention manner for each point with an enriched feature.

From the first transformer block output, we obtain aggregated features $\bm{T}$ of different scales and representations after the first transformer block. We concatenate channels along the point dimension and rearrange those points into a sequence as the input of the mutual-relation transformer. Let the learned concatenated feature $\bm{T} = 
\rm{concat}$ $\{\bm{T}^{voxel\ 1\times}$, $ \bm{T}^{voxel\ 2\times}$, $ \bm{F}^{voxel\ 4\times}$, $\bm{T}^{voxel\ 8\times}$, $\bm{T}^{point}$, $\bm{T}^{bev}\} \in\mathbb{R}^{n\times c_T}$, where $c_{T}$ is the sum of all channel size in $\bm{T}$.  Note that, the first dimension of $\bm{T}$ is the number of keypoints $n$. To model the mutual-relation between keypoints, we need to split $\bm{T}$ into $n$ point-wise feature $\bm{T} = \{ \bm{T}_i\}$, where $\bm{T}_i \in \mathbb{R}^{c_{T}}$. The concatenation and split module is showed in Figure \ref{fig:3MDETR_transformer}.

The mutual-relation transformer takes point-wise features of $n$ keypoints as inputs and uses the multi-head self-attention head to model the mutual relationship between keypoints. The outputs of the mutual-relation transformer are $\bm{\hat{T}}$.




\noindent\textbf{Comparison with previous point-based transformers}
Prior work has explored the transformer application on the task of point cloud processing. First, \cite{guo2020pct, engel2020point, zhao2020point} leverage the inherent permutation invariance of transformers to capture local context within the point cloud on the shape classification and segmentation tasks, while our \textsc{M3DETR} mainly investigates the strong attention ability of transformers between input embeddings for the 3D object detection task. Pointformer \cite{pan20203d} is the approach most related to our method because we both address the 3D object detection task by capturing the dependencies among points' features. However, Pointformer adopts a single PointNet branch to extract points feature, while \textsc{M3DETR} considers all three different representations and also applies the transformer to learn aggregated representation-based features.

\subsection{Detection Heads Network}
\label{dethead}
After we obtain the enriched embedding $\bm{\hat{T}}$ from the M3 transformer, the detection network is composed of two stages that predict 3D bounding box class, localization, and orientation in a coarse-to-fine manner, including RPN and R-CNN. Please refer to PV-RCNN~\cite{shi2020pv} for more details.

\noindent\textbf{RPN:} Region Proposal Networks take the deep semantic features $\bm{f}^{bev}$ produced by the 2D ConvNets as inputs and generate high-quality 3D object proposals $\bm{B}$, as shown in Figure \ref{fig:3MDETR_network}. A 3D object box $\bm{B}_i$ is parameterized as $(x, y, z, l, h, w, \theta) $, where $(x, y, z)$ is the center of the box, $(l , h, w)$ is the dimension of the box, and $\theta$ is the orientation in bird's-eye-view. Similar to the conventional RPN in 2D object detection~\cite{ren2015faster} each position on the deep feature map is placed by predefined bounding box anchors denoted as ($x^a$, $y^a$, $z^a$, $l^a$, $h^a$, $w^a$, $\theta^a$). Then initial proposals are generated by predicting the relative offset of an object's 3D ground truth bounding box ($x^{gt}$, $y^{gt}$, $z^{gt}$, $l^{gt}$, $h^{gt}$, $w^{gt}$, $\theta^{gt}$).







\noindent\textbf{R-CNN:}
R-CNN serves as the second stage and it takes the initial region proposals $\bm{B}$ from RPN as input to conduct further proposal refinement. For each input proposal box, the RoI-grid pooling module~\cite{shi2020pv} is adopted to extract the corresponding proposal-specific grid points' features from the transformer-based embeddings $\bm{\hat{T}}$. 
Compared with previous works, \textsc{M3DETR} leverages the richer embedding information from the learned transformers for the fine-grained proposal refinement. As the main component to extract refined features, the RoI-grid pooling module uniformly samples $ N \times N \times N $ grid points per 3D proposal. For each grid point, the output feature is generated by applying a PointNet-block \cite{qi2017pointnet} on a small number of surrounding keypoints, $M$, within its spatial surrounding region with a radius of $r$. Specifically, keypoints are the subset of input points that are sampled using the Furthest-point-sampling algorithm to cover the entire point set. 

\begin{table*}[ht]
    \centering
   
\input{LaTeX/ACMMM/table/waymo_test_all_classes}
\end{table*}
\begin{table*}[ht]
    \centering
   
\input{LaTeX/ACMMM/table/waymo_test}
\end{table*}
\begin{table*}[ht]
    \centering
\input{LaTeX/ACMMM/table/waymo_val}
\end{table*}




Finally, the refined representations of each proposal are first passed to two fully connected layers and then generate the box 3D Intersection-over-Union (IoU) guided confidence scoring of class prediction and location refinement of regression targets, $\bm{\hat{B}}$. Compared with the traditional classification-guided box scoring, 3D IoU guided confidence scoring considers the IoU between the proposal box and its corresponding ground truth box. Empirically, \cite{shi2020points,shi2020pv} show that it achieves better results compared with the traditional classification confidence based techniques. \vspace{-2 mm}



\subsection{Loss Functions} 
\label{loss}
In this section, we define the loss function. The bounding box regression target for both RPN and R-CNN stages is calculated as the relative offsets between the anchors and the ground truth as:
$\triangle x = \frac{x^{gt} - x^a}{d^a},  \triangle y = \frac{y^{gt} - y^a}{d^a}$,  
$\triangle z = \frac{z^{gt} - z^a}{h^a}$
$\triangle h = \log (\frac{h^{gt}}{{h^{a}}})$,  
$\triangle w = \log (\frac{w^{gt}}{{w^{a}}})$,
$\triangle \theta = \theta^{gt} - \theta^{a}$,
 where $ d^a $ = $\sqrt{(w^a)^2 + (l^a)^2} $. Similar to~\cite{shi2020pv, shi2020points}, the focal loss is applied \cite{lin2017focal} for the classification loss, $\mathcal{L}_{cls}$. Smooth L1 loss \cite{girshick2014rich} is adopted for the box localization regression target's losses, $\mathcal{L}_{reg}$ and $\mathcal{L}_{ref}$. In addition, 3D IoU loss \cite{shi2020pv, shi2020points} is used for $\mathcal{L}_{iou}$. 

Similar to PV-RCNN \cite{shi2020pv}, we formally define a multi-task loss for both the RPN and R-CNN stages, 
\vspace{1pt}
\begin{equation}
\begin{aligned}
 & \mathcal{L}_{cls} = -\alpha_{a}(1 - p^{a})^\gamma \log p^{a}, \\
 & \mathcal{L}_{reg} = \sum_{b \in (x, y, z, w, l, h, \theta)} \mathcal{L}_s (\triangle b), \\
 & \mathcal{L} = \mathcal{L}_{cls} + \beta_{reg} \mathcal{L}_{reg} + \beta_{iou} \mathcal{L}_{iou} + \beta_{ref} \mathcal{L}_{ref},
\end{aligned}
\end{equation}

where $p^{a}$ is the model's estimated class probability for an anchor box, and $\beta_{reg}$, $\beta_{iou} $ and $\beta_{ref}$ are chosen to balance the weights between classification loss, IoU loss and regression loss for RPN stage and R-CNN stage. We adopt the default $\alpha = 0.25$, and $\gamma = 2.0$ from the parameters of focal loss~\cite{lin2017focal}.


\section{Experiments}
In this section, we evaluate \textsc{M3DETR} both qualitatively and quantitatively on the Waymo Open Dataset~\cite{Sun_2020_CVPR} and the KITTI Dataset~\cite{geiger2012we} in the task of LiDAR-based 3D object detection. Our main results include achieving a state-of-the-art accuracy on these datasets and robustness to hyper-parameter tuning.


\subsection{Datasets and Evaluation Metric}
\label{dataset}
\noindent \textbf{Waymo Open Dataset:}
The Waymo Open Dataset~\cite{Sun_2020_CVPR} is a large-scale autonomous driving dataset containing $1000$ scenes of $20$s duration each, with $798$ scenes for training and $202$ scenes for validation. Each scene is sampled at a frequency of $10$Hz. Overall, the dataset includes $12$M labeled objects and thus we only use one fifth of the training scenes for the following experiment. We consider LiDAR data as the input to our approach. The evaluation protocol on the Waymo dataset consists of the mean average precision (mAP) and mean average precision weighted by heading (mAPH). For each object category, the detection outcomes are evaluated based on two difficulty levels: LEVEL\_1 denotes the annotated bounding box with more than 5 points and LEVEL\_2 represents the annotated bounding box with more than 1 point.

\textbf{KITTI dataset}
The KITTI 3D object detection benchmark~\cite{geiger2012we} is another popular dataset for autonomous driving. It contains $7,481$ training and $7,518$ testing LiDAR scans. We follow the standard split on the training ($3,712$ samples) and validation sets ($3,769$ samples). For each object category, the detection outcomes are evaluated based on three difficulty levels based on the object size, occlusion state, and truncation level. \vspace{-2.5 mm}


\subsection{Implementation Details}\label{impl}
Closely following the codebase\footnote{https://github.com/open-mmlab/OpenPCDet.}, we use PyTorch to implement our M3 transformer modules and integrate them into the PV-RCNN network~\cite{shi2020pv}. More details about backbone and detection heads network will be introduced in the supplementary materials.




\noindent\textbf{M3 Transformer}
We project the embeddings obtained from the backbone network with different scales and representations to 256 channels, as the input of the multi-representation and multi-scale transformer requires, and project the output features back to their original dimensions before passing into the mutual-relation transformer. Due to the GPU memory constraint, we experiment with two types of MHSA module designs: 2 encoder layers with 4 attention heads and 1 encoder layer with 8 attention heads. 



\begin{figure*}[t]
    \centering
    \includegraphics[width=1\linewidth]{LaTeX/ACMMM/figures/Visualization_4.pdf}
   
    \caption{We highlight the 3D detection results of \textbf{M3DETR} on the Waymo Open Dataset. The 3D ground truth bounding boxes are in green, while the detection bounding box are shown in blue.}
    \label{fig:visualization}
    \vspace{-0.4cm}
\end{figure*}

\noindent\textbf{Training Parameters}
Models are trained from scratch on $4$ NVIDIA P6000 GPUs. We use the Adam optimizer with a fixed weight decay of $0.01$ and use a one-cycle scheduler proposed in~\cite{smith2018disciplined}. For the Waymo Open Dataset, we train our models for $45$ epochs with a batch size of $8$ scenes and a learning rate $0.01$, which takes around $50$ hours. For the KITTI dataset, we train our models for $80$ epochs with a batch size of $8$ scenes per and a learning rate $0.01$, which takes around $15$ hours. 

\subsection{Results}
\label{result}
\noindent\textbf{Waymo Open Dataset:}
We first present our object detection results for the vehicle, pedestrian, and cyclist classes on the test set of Waymo Open Dataset in Table~\ref{tab:waymo_test_all_classes} compared with PV-RCNN~\cite{shi2020pv}. We evaluate our method at both LEVEL\_1 and LEVEL\_2 difficulty levels. Note that we reproduce the baseline PV-RCNN with a single frame input since they recently adopt two-frames input on test set. As we can see, PV-RCNN achieves 69.57\% and 63.65\% on average in LEVEL\_1 mAP and LEVEL\_2 mAP, respectively, while \textsc{M3DETR} improves them by 1.48\% and 1.85\%, respectively. Without bells and whistles, our approach works better than PV-RCNN~\cite{shi2020pv}. Furthermore, we compare our framework on the vehicle class for different distances with state-of-the-art methods, including StarNet~\cite{ngiam2019starnet}, PointPillars~\cite{lang2019pointpillars}, RCD~\cite{bewley2020range}, Det3D~\cite{zhu2019class}, RangeDet~\cite{fan2021rangedet} and PV-RCNN~\cite{shi2020pv}. In Table~\ref{tab:waymo_test}, \textsc{M3DETR} outperforms PV-RCNN significantly in both LEVEL\_1 and LEVEL\_2 difficulty levels across all distances, demonstrating the effectiveness of newly proposed framework. Moreover, we visualize the detection results of \textsc{M3DETR} in Figure ~\ref{fig:visualization}. 



Compared with the PV-RCNN shown in Figure~\ref{fig:visualization_comparison}, \textsc{M3DETR} successfully captures the inter- and intra- interactions among input features and effectively helps the model generate high-quality box proposals. To the best of our knowledge, \textsc{M3DETR} achieves the state-of-the-art in the Vehicle class in both LEVEL\_1 and LEVEL\_2 difficulty levels among all the published papers with a single frame LiDAR input.

We also evaluate the overall object detection performance with an IoU of 0.7 for Vehicle class on the full Waymo Open Dataset validation set as in Table~\ref{tab:waymo_val}, further proving that our architecture is more efficient for jointly modeling the input features.

\noindent\textbf{KITTI dataset:}
We compare our approach with the state-of-the-art methods on the KITTI test set~\cite{shi2020pv,chen2020object}. We compute the mAP on three difficult types of both car and  cyclist classes in 3D detection metric. Table~\ref{tab:kitti_test_car_cyclist} shows that \textsc{M3DETR} achieves state-of-the-art performance and outperforms the previous work by a large margin especially on the cyclist class. In particular, HotSpotNet~\cite{chen2020object} achieves $82.59\%$ in the ``easy'' categories of 3D detection metric, while \textsc{M3DETR} improves these results by significant $1.24\%$.  \vspace{-2 mm}

\begin{table}[t]
    \centering
    \setlength\tabcolsep{3px}
\input{LaTeX/ACMMM/table/kitti_test_car_cyclist}
\vspace{-0.2cm}
\end{table}

\begin{table}[t]
    \centering
    \setlength\tabcolsep{3px}
\input{LaTeX/ACMMM/table/kitti_ablation_transformer}
\vspace{-0.5cm}
\end{table}

\subsection{Ablation Studies}
\label{ablation}
To demonstrate the individual benefits of the multi-representation, multi-scale, and mutual-relation layers of the M3 transformer, we perform ablation experiments and tabulate the results in Table~\ref{tab:kitti_ablation_transformer}. All experiments are conducted on the validation set of KITTI dataset.


With the single multi-representation and multi-scale transformer layer, we can achieve 4.07\% and 1.71\% on the moderate difficulty in car class with 11 and 40 recall positions, respectively compared with the PV-RCNN baseline. On the other side, with the single mutual-relation transformer layer, the performance gain are 4.47\% and 1.99\% compared with the PV-RCNN baseline. Without hyper-parameter tuning, \textsc{M3DETR} benefits from unifying multiple point cloud representations, feature scales, and model mutual-relations simultaneously which results in the best performance.  \vspace{-2 mm}

\subsection{Robustness of M3DETR}
\label{robust}
To demonstrate the robustness of \textsc{M3DETR} to hyper-parameter tuning, we perform a series of tests by varying the sampling size, number of detection heads, and number of transformer encoder layers. We present the results of these tests in Figure~\ref{fig:M3DETR_robustness}, where we observe that \textsc{M3DETR} performs consistently well for the ``car'' category with IoU threshold of 0.7 for both 11 and 40 recall positions on the KITTI validation set.

\begin{figure}[t]
\vspace{-0.5cm}
    \centering
   
    \centerline{
    \includegraphics[width=1\linewidth]{LaTeX/ACMMM/figures/Comparison_pvrcnn_M3DETR_wo.pdf}
    }
    \vspace{-1mm}
    \caption{We visualize the 3D detection results for the same input point cloud between PV-RCNN (left) and \textbf{M3DETR} (right) on Waymo Open Dataset. We highlight the false negative boxes from PV-RCNN in red.}
   
    \label{fig:visualization_comparison}
    \vspace{-0.4cm}
\end{figure}

\begin{figure}[t]
    \centering
    \includegraphics[width=0.95\linewidth]{LaTeX/ACMMM/figures/M3DETR_Robustness.pdf}
    \vspace{-1mm}
    \caption{Performance comparison of different \textbf{M3 Transformers} variants in Car class on KITTI validation set. We show the mAP results with IoU threshold of 0.7 for both 11 and 40 recall positions.  Note that "l" and "h" represent layer number and head dimension of M3 Transformers, respectively. "Top" denotes the number of proposals used for keypoint sampling from RPN stage.}
    \label{fig:M3DETR_robustness}
    \vspace{-0.6cm}
\end{figure}
 
\section{Conclusions}
In this paper, we present \textsc{M3DETR}, a novel transformer-based framework for object detection with LiDAR point clouds. \textsc{M3DETR} is designed to simultaneously model multi-representation, multi-scale, mutual-relation features through the proposed M3 Transformers. Overall, the first transformer integrates features with different scales and representations, and the second transformer aggregates information from all keypoints. Experimental results show that \textsc{M3DETR} outperforms previous work by a large margin on the Waymo Open Dataset and the KITTI dataset. Without bells and whistles, \textsc{M3DETR} is demonstrated to be invariant to the hyper-parameters of transformer.


\noindent\textbf{Acknowledgement.} This work was supported in part by ARO Grants W911NF1910069, W911NF2110026  and U.S. Army Grant No. W911NF2120076.

\pagebreak


{\small
\bibliographystyle{ieee_fullname}
