\section*{Acknowledgement}
This work is supported in part by National Key Research and Development Program of China under Grant 2020AAA0107400, Zhejiang Provincial Natural Science Foundation of China under Grant LR19F020004, National Natural Science Foundation of China under Grant U20A20222, National Science Foundation for Distinguished Young Scholars under Grant 62225605, Alibaba-Zhejiang University Joint Research Institute of Frontier Technologies, Ant Group, and sponsored by CAAI-HUAWEI MindSpore Open Fund.



\section{Conclusion}


\section{Conclusion}
In this paper, we develop a unified panoptic segmentation framework, dubbed \ours, for point cloud data, which is capable to exclusively predict panoptic results without any hand-crafted post-processing and achieves state-of-the-art performance. \ours allocates a set of classifiers to learn how to group coherent points directly and introduces bipartite matching to enable end-to-end training. Moreover, \ours employs a transformer decoder to refine the groupings and resolve class imbalance problem by designing a context-aware cutmix augmentation. \ours is the first to provide a holistic and end-to-end solution for point cloud panoptic segmentation. 
We hope that \ours can inspire more researchers to delve into the development of unified segmentation for point cloud, which is beneficial to promoting autonomous vehicles.


\section{Experiment}
    To evaluate \ours, we conduct experiments on two popular LiDAR point cloud datasets: SemanticKITTI\cite{SemanticKITTI} and nuScenes\cite{nuScenesSeg}.
    
    \subsection{Datasets and Evaluation Metric}
    \label{sec:dataset_metric}
        \subsubsection{SemanticKITTI} \label{sec:semantickitti}
        proposes the first panoptic segmentation challenge on point cloud data. It contains 22 data sequences splited into 3 parts: 10 for training, 1 for validation and 11 for testing. There are 8 \emph{thing} classes and 11 \emph{stuff} classes.
        
        \subsubsection{nuScenes} \label{sec:nuscenes}
        is a large-scale dataset for autonomous driving, which contains LiDAR data of 1000 scenes. The 1000 scenes are divided into 3 parts: 750 for training, 100 scenes for validation and 150 scenes for testing. There are 10 \emph{thing} classes and 6 \emph{stuff} classes.
        
        \subsubsection{Evaluation Metric} \label{sec:metric}
        Mean Panoptic Quality (PQ) \cite{panopticsegmentation} is adopted as the primary evaluation metric for our experiment. As shown in Equation \ref{eq:pqc}, PQ of a specific class can be decomposed into Segmentation Quality (SQ) and Recognition Quality (RQ):
        
        \begin{equation}
            \label{eq:pqc}
            \textrm{PQ}_c = \underbrace{\frac{\sum_{(p, g)\in \textrm{TP}_c}  \textrm{IoU}(p, g) }{|\textrm{TP}_c|}}_{\textrm{segmentation quality (SQ)}} 
            \times \underbrace{\frac{|\textrm{TP}_c|}{|\textrm{TP}_c| + \frac{1}{2} |\textrm{FP}_c| + \frac{1}{2} |\textrm{FN}_c|}}_{\textrm{recognition quality (RQ)} },
        \end{equation}
        where $\textrm{TP}_c$ is the set of matched predicted masks and ground truth masks of class $c$, $\textrm{FP}_c$ is the set of unmatched predicted masks of class $c$, $\textrm{FN}_c$ is set of unmatched ground truth masks of class c, and $\textrm{IoU}(p, g)$ is the intersection-over-union of predicted mask $p$ and ground truth mask $g$. 
       
       
       
       
       
       
       
        Mean PQ is the average of PQ of all classes and we additionally report $\textrm{PQ}^{\textrm{Th}}$, $\textrm{SQ}^{\textrm{Th}}$ and $\textrm{RQ}^{\textrm{Th}}$ of \emph{thing} classes, $\textrm{PQ}^{\textrm{St}}$, $\textrm{SQ}^{\textrm{St}}$ and $\textrm{RQ}^{\textrm{St}}$ of \emph{stuff} classes and $\textrm{PQ}^{\dagger}$ \cite{pqdagger}, where $\textrm{PQ}$ of \emph{stuff} classes is replaced by their $\textrm{IoU}$ in the calculation.
        
        
    \subsection{Implementation Details}
    \label{sec:implementation_details}
    \subsubsection{Settings and Hyper-parameters}
    Our implementation is based on MMDetection3D~\cite{mmdet3d2020}. Specifically, we train our models for 80 epochs with a batch size of 4. The learning rate is set to 0.002 initially and decrease with a factor of 0.1 after 50 epochs. We adopt AdamW~\cite{adamw} with a weight decay of 0.05 as our optimizer. In addition to our proposed CutMix augmentation, we apply random flipping along x- and y- axis, random rotation along z- axis and random scaling. Unless specified, the point feature dimension is set to 128 and the number of classifiers is set to 100. The number of refinement stages is 3. As for training, the losses are included in Equation~\ref{eq:match_loss} and the coefficients $\alpha$, $\beta$, $\gamma$ are set to 4, 1, 1 respectively. 
   
    
        \footnotetext{Our 1st place performance is assessed on Aug 12th, 2022 in \emph{https://competitions.codalab.org/competitions/24025\#results}.}
    \subsubsection{Backbone}
    We employ the backbone of RPVNet~\cite{RPVNet} in \ours. It fuse points, voxels and range-view feature, and extract representative features. We follow the same backbone architecture as RPVNet but change the output feature dimension to 128. The voxel size is set to 5 cm and 10 cm for SemanticKITTI and nuScenes respectively.
        \input{tables/nus_val_mean}
    
        \input{tables/ablation_num_stages}
    \input{tables/ablation_components}
    \subsection{Main Results}
    \label{sec:main_results}
        \subsubsection{Results on SemanticKITTI} 
        \label{sec:result_sem}
       
        As shown in Table~\ref{tab:sem_test_mean} and \ref{tab:sem_val_mean}, we surpass all existing methods in PQ of both test set and validation set, and show significant advantages in the performance of \emph{thing} classes. As for test set, we improve PQ of Panoptic-PHNet~\cite{Panoptic_PHNet} from $61.5\%$ to $62.2\%$ and achieve a gain of $1.9\%$ in $\textrm{PQ}^{\textrm{Th}}$. As for validation set, we outperform GP-S3Net~\cite{GP-S3Net} by a margin of $1.1\%$ in PQ and $2.8\%$ in $\textrm{PQ}^{\textrm{Th}}$.
        Compared with clustering-based methods DS-Net~\cite{DS_Net}, and Panoptic-PolarNet~\cite{Panoptic-PolarNet} in addition to Panoptic-PHNet and GP-S3Net, our method achieve an increase of over $6\%$ in PQ of test set. With respect to range-image-based method EfficientLPS~\cite{EfficientLPS}, \ours outperforms by $4.8\%$ in PQ of test set. The results of combined methods (row 1 and row 2) presented in the tables are obtained by training a detection head and semantic head as stated in the dataset. 
        Moreover, following Panoptic-PHNet, we report results of model ensemble and test-time augmentation.
        Additionally, we provide class-wise performance of \ours in supplementary material.
        
        \subsubsection{Results on nuScenes} 
        \label{sec:result_nus}
        In this section, we compare the results of \ours on nuScenes with results of previous methods. As listed in Table~\ref{tab:nus_val_mean}, our method achieves state-of-the-art results on validation set.
        
    \subsection{Ablation Study}
   
    
        \subsubsection{Ablation on Network Components}
        To verify the effectiveness of \ours, we gradually apply our proposed components to a vanilla network. As shown in Table~\ref{tab:ablation_components}, M1 refers to a vanilla network with no refinement on the classifier or CutMix augmentation. M2 is trained with classifier refinement and M3 is trained with context-aware CutMix. The performance of M4 shows that both classifier refinement and context-aware CutMix contribute to the high performance.
        
        
        \subsubsection{Ablation on Number of Stages} As shown in Table~\ref{tab:ablation_num_stages}, trials with different number of stages reveals that \ours with 3 stages achieves the best result. We observe that there exists over-fitting concerning the decrease of training loss as the number of stages increase. It suggests that models for larger datasets may benefit from more stages.
        
        \subsubsection{Ablation on Number of Classifiers} Table~\ref{tab:ablation_num_classifiers} contains result from different number of classifiers. It shows that 100 classifiers achieve the best result. On the one hand, insufficient number of classifiers is harmful to performance on both \emph{thing} classes and \emph{stuff} classes since bipartite assignment may assign the classifiers to inconsistent semantic. On the other hand, excessive number of classifiers benefit from consistency in assignment and perform better in \emph{thing} classes. 
       
        However, since the number of classifiers for background classes is fixed, excessive instance classifiers may lead to under-segmented background.
        
       
       
        
       
       
        
        \input{tables/ablation_num_classifiers}
        
        \subsubsection{Ablation on CutMix Strategies} In addition to our proposed context-aware CutMix, there is another CutMix strategy in point cloud object detection and segmentation: random CutMix~\cite{SECOND, RPVNet, Panoptic_PHNet}. They alleviate class imbalance by randomly mixing the sampled instances in the current scan. To validate the effectiveness of our context-aware CutMix, we compare performance by applying the strategies on M1 in Table~\ref{tab:ablation_strategy}. As shown in Table~\ref{tab:ablation_strategy}, our context-aware CutMix achieve a gain of 5.1\% in PQ and outperform by a large margin on \emph{thing} classes. It verifies our design on preserving context information of instances to enhance performance.
        
        \input{tables/ablation_strategy}
        
    \subsection{Analysis and Visualization}
    
        \subsubsection{Spatial Distributions of Predictions}
        As stated in Section~\ref{sec:bipartite_matching} and ~\ref{sec:classifier_refine}, our classifiers are able to distinguish between instances, hence being capable of predicting panoptic segmentation results directly. Considering that the predictions are in 3D space, it is better to present an illustration more intuitively. 
        Therefore, we plot the centers of the predictions in bird-eye view (BEV). As shown in Figure~\ref{fig:spatial}, each subplot stands for the spatial distribution of a classifier's predictions on \emph{car} in a 100m $\times$ 100m square. The distributions follow certain patterns: 1) the arc-shaped patterns reveal accordance with the rotation of LiDAR sensors. 2) the positions where dense predictions located demonstrate spatial spacing, verifying the ability of the classifiers to predict exclusive instances.
       
       
       
       
       
       
       
        
        \begin{figure}[h]
            \centering
           
           
           
            \includegraphics[width=0.45\textwidth]{figures/spatial_locations_alpha0.05.pdf}
           
            \caption{Spatial distributions of predictions by classifiers. The centers are projected into a 100m $\times$ 100m x-y plane. Results are obtained from SemanticKITTI test set.}
            \label{fig:spatial}
        \end{figure}
        
       
        
       
\section{Introduction}
   
    As one of the most challenging problems in computer vision, panoptic segmentation~\cite{panopticsegmentation} seeks a holistic solution to both semantic segmentation and instance segmentation. Shortly after the researchers proposed the question in image data, two LiDAR datasets~\cite{SemanticKITTI, nuScenesSeg} for autonomous driving extend the research area to point cloud data. 
   
    These emerging challenges aim at assigning points with groupings of countable \emph{thing} instances and uncountable \emph{stuff} classes, revealing that perception system of autonomous vehicles demands understanding of the environment in terms of both semantic level and instance level through point cloud sensors.
    
    To solve point cloud panoptic segmentation, previous efforts can be divided into two streams: proposal-based methods and proposal-free methods. As the name indicates, proposal-based methods rely on proposals generated by an object detection head to get instance segmentation
   
    and employ an extra semantic branch for semantic segmentation. Besides their cascaded structure, this stream of methods involves lots of hand-crafted components such as proposals and non-maximum suppression (NMS). As for proposal-free methods, they introduce clustering-based methods in their instance branch based on the predicted offsets to instance centers. Similarly, an extra semantic branch is attached for semantic segmentation. 
   
    Although outstanding results have been achieved by these methods on different benchmarks, there are two main drawbacks as shown in the upper part of Figure~\ref{fig:intro}: 1) they treat semantic and instance segmentation as surrogate tasks, which does not truly solve panoptic segmentation holistically; 2) their instance branch involves many hand-crafted components and post-processing, which is complicated and time-consuming.
    \begin{figure}[t]
        \centering
       
       
       
       
       
        \includegraphics[width=0.4\textwidth]{figures/intro6.pdf}
        \caption{Illustration of previous framework and \ours.}
       
        \label{fig:intro}
    \end{figure}
 
 
   
    
   
    
   
   
    
    
   
   
   
    
    Inspired by recent developments in image segmentation~\cite{cheng2021maskformer, max-deeplab, panoptic_segformer, zhang2021knet}, we propose \ours, a simple but effective \textbf{p}oint cloud \textbf{u}nified  \textbf{p}anoptic \textbf{s}egmentation framework to solve the challenges above. In essence, the aim of point cloud panoptic segmentation is to predict groupings of coherent points. \ours unifies point cloud instance and semantic segmentation as a classifier-assigning problem. More specifically, \ours allocates a set of point-level classifiers, learning to assign them to exclusive instances or semantic classes. By utilizing bipartite matching in the training phase, we achieve \ours in an end-to-end manner and it is able to predict exclusive groupings with no hand-crafted design or post-processing as shown in Figure~\ref{fig:intro}.

    
     

    
    
   
   
    
   
   
   
    
   
    In addition to predicting exclusive groupings, we adopt two designs to produce better grouping results.
    First, we utilize a transformer decoder to refine the classifiers. In each stages of refinement, our point-level classifiers query point features from backbone and generate refined classifiers. Afterwards, the classifiers integrate the feature of corresponding instances and semantics, enhancing their ability to distinguish between groupings. Moreover, we employ a classifier self-attention to incorporate global relations into the classifiers. After the classifiers are refined, new point groupings are produced and can be further used to refine the classifiers in the next stage together with the refined classifiers.
    Second, in order to alleviate class imbalance and train the classifiers more sufficiently, we design a context-aware CutMix~\cite{SECOND, RPVNet, Panoptic_PHNet} augmentation. We cut instances from training scans and mix them with the instances in the current scan based on their background to avoid damaging their context so that performance is improved in return.
   
   
    
    To evaluate the effectiveness of our proposals, we conduct extensive experiments on two point cloud panoptic segmentation datasets. Our method ranks 1st on the leader board of SemanticKITTI~\cite{SemanticKITTI} 
   
    and achieves state-of-the-art results on nuScenes~\cite{nuScenes}.
    
    To sum up, the contributions of this paper are listed below:
    \begin{itemize}
       
       
        \item To the best of our knowledge, \ours is the first simple but effective \textbf{p}oint cloud \textbf{u}nified \textbf{p}anoptic \textbf{s}egmentation framework, using a set of point-level classifiers to directly predict semantic and instance groupings.
        \item To get rid of post-processing and hand-crafted designs, we introduce bipartite matching in our training so that the classifiers are able to exclusively predict groupings.
        \item We utilize a transformer decoder to iteratively refine the classifiers with point features to produce more accurate groupings of points.
        \item To encounter class imbalance, we adopt a context-aware CutMix strategy to enhance the performance of segmentation by preserving the context of instances.
        \item We achieve rank 1 performance on the leader board of SemanticKITTI panoptic segmentation task and SOTA results on nuScenes.
    \end{itemize}
    
    
\section{Method}
\begin{figure*}[t]
    \centering
    \includegraphics[width=\textwidth]{figures/network_arch7.pdf}
   
   
   
   
   
   
    \caption{Pipeline of \ours. Point features are first encoded by a RPV backbone~\cite{RPVNet} and fed into the unrefined classifiers to get initial groupings and semantics. Then, point features activated by corresponding groupings are integrated into the classifiers and a self-attention is applied to the classifiers to produce refined classifiers. For simplicity, we omit the superscripts of classifiers in Section~\ref{sec:classifier_refine}. With the refined classifiers, more accurate groupings and semantics are obtained. To clarify, groupings and semantics of all stages will be supervised by ground-truth with bipartite matching in training and only the groupings and semantics of the last stage will be used to output segmentation results in inference.}
    \label{fig:pipeline}
\end{figure*}


In this section, we first state the definition of point cloud panoptic segmentation in Section~\ref{sec:problem_formulation}. Then, we present the network architecture of \ours in Section~\ref{sec:pups}, along with its two core components, i.e., bipartite matching (Section~\ref{sec:bipartite_matching}) and classifier refinement (Section~\ref{sec:classifier_refine}). Lastly, we show a context-aware CutMix for instances in Section~\ref{sec:cutmix}.


\subsection{Problem Formulation}
\label{sec:problem_formulation}
Point cloud panoptic segmentation aims at grouping a point cloud $P \in \mathbb{R}^{K\times 4}$ of $K$ points into a set of \emph{thing} instances and \emph{stuff} classes, among which \emph{thing} instances refer to countable objects (e.g. person, car, bicycle) and \emph{stuff} classes refers to uncountable backgrounds (e.g. road, terrain, vegetation). As shown in Equation~\ref{eq:problem_formulation}, ground-truth groupings in a point cloud are defined as:
\begin{equation}
    \label{eq:problem_formulation}
    \{y_i\}_{i=1}^M = \{(g_i, c_i)\}_{i=1}^M,
\end{equation}
where $g_i \in \{0,1\}^K$ is a ground truth binary mask indicating which points belong to group $i$, $c_i$ is the semantic class of group $i$, and $M$ is the number of ground truth groupings in the point cloud. 
Note that the groupings are mutually exclusive, i.e.
, each point in a point cloud belongs to either an instance of \emph{things} or a background \emph{stuff}. In this way, each point is assigned to a group ID and a semantic class.


\subsection{Point Cloud Unified Panoptic Segmentation}
\label{sec:pups}

Given the definition in Section~\ref{sec:problem_formulation}, we allocate $N$ learnable point-level classifiers to predict the groupings for both distinct \emph{thing} instances and background \emph{stuff} in a unified manner. We denote the learnable parameters of the classifiers as $\theta = \{\theta_i \;|\; \theta_i \in \mathbb{R}^C \}_{i=1}^N$, where $C$ is the number of channels. As shown in the inference pipeline of Figure~\ref{fig:pipeline}, $K$ points are fed into a backbone to extract point-wise feature $F$. Using the feature $F$ and the parameters $\theta$, \ours generate two vital scores for panoptic segmentation results: grouping scores $G = \{\hat{g}_i \;|\; \hat{g}_i \in [0,1]^K\}_{i=1}^N$ and semantic scores $\Delta = \{\Delta_i \;|\; \Delta_i \in \mathbb{R}^{T}\}_{i=1}^N$, where $T$ is the number of \emph{thing} and \emph{stuff} classes. 

First of all, grouping scores $G$ indicate the probability of the $K$ points belonging to $N$ groups, which are used to decide the group ID for each point. Similarly, semantic scores $\Delta$ indicates the probability of the $N$ groupings belonging to $T$ semantic classes, which are used to assign semantic classes to groupings, and further decide semantic classes for each point.


Specifically, with the parameters $\theta$ of classifiers and the feature $F$ of points, we utilize a simple matrix multiplication and a sigmoid, denoted as $\delta(\cdot, \cdot)$, to obtain grouping scores $G$ for each point with respect to each classifier:

\begin{equation}
    \label{eq:grouping_scores}
    \hat{g}_i = \delta(\theta_i, F), i=1,\hdots,N,
\end{equation}
where $F \in \mathbb{R}^{K \times C}$ is the point-level features of $K$ points. 

Similarly, we utilize $\delta(\cdot, \cdot)$ to predict a semantic score $\Delta_i \in \mathbb{R}^{T}$ for each point-level classifier, where $T$ is the number of \emph{thing} and \emph{stuff} classes:
\begin{equation}
    \label{eq:semantic_scores}
    \Delta_i = \delta(\psi, \theta_i), i=1,\hdots,N.
\end{equation}
To clarify, $\psi \in \mathbb{R}^C$ stands for a set of learnable parameters. 

To output the result of panoptic segmentation, PUPS assigns semantic class $c_i$ to $\hat{g}_i$ and groupings to points by:
\begin{gather}
    \label{eq:output}
    \hat{c}_i = \arg \max \Delta_i, \\
    \hat{z}_{i,k} = 
        \begin{cases}
            1 & \text{if} \quad \hat{g}_{i,k} = \underset{i}{\max} \, (\hat{g}_{i,k}) \\
            0 & \text{otherwise}
        \end{cases},\\
    \{\hat{y}_i\}_{i=1}^N = \{(\hat{z}_i, \hat{c}_i)\}_{i=1}^N,
\end{gather}
Our prediction is organized similarly to Equation~\ref{eq:problem_formulation} and group ID and semantic class for each point is obtained.


Now, \ours is able to predict the groupings directly. Instead of adding post-processing at test time, we use bipartite matching (Section~\ref{sec:bipartite_matching}) in our training pipeline to prevent multiple classifiers from predicting the same instance.


\subsection{Bipartite Matching}
\label{sec:bipartite_matching}
One solution to the aforementioned problem is to make our point-level classifiers learn from exclusive ground truth groupings.
Therefore, a one-to-one mapping from the $M$ ground-truth groupings to the $N$ classifiers is needed.
Inspired by recent application of bipartite matching in object detection~\cite{DETR} and image segmentation~\cite{cheng2021maskformer, panoptic_segformer, zhang2021knet}, bipartite matching is able to assign one ground-truth to only one prediction according to a cost matrix. This one-to-one rule plays a vital role in exclusively predicting the point cloud panoptic segmentation results for the reason that there are no classifier assigned to learn the same \emph{thing} instance or \emph{stuff} class, reducing the possibility of duplicate predictions. It also prevents the classifiers from only focusing on easy groupings because all ground truths are mapped, reducing the bias of the model. Since instance ID prediction is not required for \emph{stuff} classes, several classifiers are constantly mapped to the ground truth of \emph{stuff} classes.
\subsubsection{Cost Computation} To match the predictions $\{\hat{s}_i\}_{i=1}^N=\{(\hat{g}_i, \Delta_i)\}_{i=1}^N$ and ground-truths $\{y_i\}_{i=1}^M\{(g_i, c_i)\}_{i=1}^M$, we compute the cost matrix based on their pairwise accordance in terms of both points and groups. For simplicity, we term the match cost and the training loss between prediction and ground-truth with the same notation:
\begin{equation}
    \label{eq:match_loss}
    \mathcal{L}_{\text{match}} = \alpha \mathcal{L}_{\text{dice}} +  \beta \mathcal{L}_{\text{focal}} + \gamma \mathcal{L}_{\text{CE}},
\end{equation}
where dice loss $\mathcal{L}_{\text{dice}}$~\cite{diceloss} and cross entropy loss ($\mathcal{L}_{\text{CE}}$) are for point-wise accordance between $\hat{g}_i$ and $g_j$ ($\forall 1 \le i \le N, 1 \le j \le M$). The focal loss~\cite{2017Focal} is for group classification between $\Delta_i$ and $c_j$ ($\forall 1 \le i \le N, 1 \le j \le M$). For classifiers that are not assigned with any ground truth, they are masked as negative.


In summary, Section~\ref{sec:pups} resolves the problem of predicting the groupings and Section~\ref{sec:bipartite_matching} enables the classifiers to learn from exclusive ground truth so that there is no need to apply post-processing to remove duplicate predictions or clustering algorithms to coalesce segmented groupings.


\subsection{Classifier Refinement}
\label{sec:classifier_refine}
Although the process is simple as stated in Section~\ref{sec:pups}, it is challenging to accurately classify the unordered points into groups. Being different from classifying points into semantic classes, panoptic segmentation demands a further step to discriminate instance information within one semantic class. Moreover, the predicting process is purely point-based, i.e., points are treated as isolated only with implicit spatial information, and appearance encoded in their backbone features. 
Thus, classifying the points into groups only once may introduce noises from other groups


To fulfill the aforementioned demand and overcome the problem, we employ a transformer decoder with $S$ stages to refine the point-level classifiers with point features as shown in Figure~\ref{fig:pipeline}. 
In each stage of the transformer decoder, the process of refinement are divided into three parts: 1) classifier feature query. It gathers point features for each classifier. 2) classifier update. It updates the classifiers with the gathered features. 3) classifier self-attention. It further models the context information between classifiers.


\subsubsection{Classifier Feature Query}
First, we query the point features with grouping scores from Equation~\ref{eq:grouping_scores} using the parameters of the point-level classifiers fed into this stage. The grouping score serves as an attention map for the $i$th classifier with respect to every single point in the point cloud so that the most related features are collected as instance and semantic information.
With the point features and their attention, the discriminative instance and semantic feature $F^{\theta_i} \in \mathbb{R}^{C}$ of the grouping $i$ is obtained by:
\begin{equation}
    \label{eq:aggregate_features}
    F_{\theta_i} = \frac{1}{K} \sum_{k=1}^K \hat{g}_{i,k} \cdot F_k.
\end{equation}
For simplicity, we omit the superscript used in Figrue~\ref{fig:pipeline} and  $\theta = \{\theta_i\}_{i=1}^N$ stands for the parameters of the classifiers fed into this stage.



\subsubsection{Classifier Update}
Since the objective of classifiers is to learn the distinct groupings of points, we integrate the discriminative instance and semantic features into the parameters of the classifiers so that they are able to retrieve instance points that are missing in the current groups and rule out noisy ones, thus producing more accurate results. Specifically, we first project the features into the space of classifiers' parameter and employ a learnable momentum $m$ to control the extent of integration. The calculation of the projection and momentum $m$ is as followed:
\begin{align}
    \label{eq:refine_momentum}
    m = 1 & - \sigma (\varphi_1({F}_{\theta_i})) \\
    \Tilde{\theta}_i = (1-m) &\cdot \varphi_2({F}_{\theta_i}) + m \cdot \theta_i,
\end{align}
where $\sigma$ is a non-linear function sigmoid and $\varphi_1$, $\varphi_2$ are linear transformations.






\subsubsection{Classifier Self-attention}
Lastly, besides the integration of classifier parameters and their corresponding local information gathered by the attention maps, we apply self-attention to incorporate global relation into the parameters of the classifiers. We utilize a multi-head self-attention~\cite{2017Attention} to model the relation between classifiers. The relation helps classifiers distinguish between each other and understand the context of the point cloud, reducing the probability that their groupings share a large overlap and enhancing the grouping accuracy.

Eventually, the point features and the refined classifiers are again feed into the next stage of the decoder. The refined grouping scores generated by Equation~\ref{eq:grouping_scores} in the next stage is able to gather more points from the instance and suppress noisy ones more accurately, producing better instance feature. The refined semantic scores produced by Equation~\ref{eq:semantic_scores} assign better semantic classes for the groupings.




\input{tables/sem_test_mean}


\subsection{Context-aware CutMix}
\label{sec:cutmix}
In object detection and segmentation, class imbalance is a common issue, leading to performance degradation in the minor classes. A trivial solution to this issue is to cut objects out of training set to form a sample database. Before an input is fed into a network, the objects are sampled from the database and mix with the existing ones. 




\subsubsection{Context-aware Mixing}
We suggest mixing instances in accordance with their context in light of the aforementioned scenario.
After an instance is sampled from the database, context-aware mixing translate the instance to the nearest contextual point. Contextual points are the points that most possibly exist underneath an instance, e.g., \emph{car} instances are most possibly on top of \emph{road} and \emph{parking} but not \emph{pole}. 
Panoptic segmentation methods are able to model the relation between instances and background since their objective is to distinguish among them. Thus, preserving the context is beneficial to the recognition of instances.
As a result, the grouping results of the mixed classes are enhanced.
\input{tables/sem_val_mean}
\section{Related Work}
Panoptic Segmentation aims to divide an input sample into countable \emph{thing} instances or uncountable \emph{stuff} classes. The output of panoptic segmentation is to assign element-wise label with both instance ID and semantic class. For input modal of LiDAR point cloud or image, panoptic segmentation models follow two typical frameworks: proposal-based and proposal-free.
\subsection{LiDAR Point Cloud Panoptic Segmentation}
    \subsubsection{Proposal-based methods} This kind of methods are usually formulated in a two-stage manner: segmentation after detection~\cite{LPSAD, MOPT}. Moreover, SemanticKITTI~\cite{SemanticKITTI} and nuScenes~\cite{nuScenes} report results by joining state-of-the-art point cloud object detection methods and point cloud semantic segmentation methods. Taking in range-view images, EfficientLPS~\cite{EfficientLPS} utilizes a instance branch to predict classes, bounding boxes and masks for \emph{thing} classes and fuse semantic feature to predict \emph{stuff} classes. After post-processing, the range-view result is projected back to point-wise result. It is worth noting that hand-crafted components such as anchors or NMS are often involved in these methods. 
    \subsubsection{Proposal-free methods} As for proposal-free ones, they usually predict instance centers and point-wise offset to centers to output panoptic segmentation result~\cite{Panoptic-PolarNet, DS_Net}. Recently, Panoptic-PHNet~\cite{Panoptic_PHNet} introduces a K-NN transformer to predict more accurate offsets. Additionally, proposal-free methods~\cite{Panoster, DS_Net, Panoptic_PHNet} often involve clustering algorithms to cluster points to instances. GP-S3Net~\cite{GP-S3Net} propose a novel graph-based clustering method to effectively predict instances from over-segmented clusters. 
    
    \ours directly groups point cloud without any bounding box proposals, hand-crafted post-processing or clustering algorithms. It is worth noting that these hand-crafted components in both proposal-based/free methods require lots of computation and careful tuning.
\subsection{Image Panoptic Segmentation}
    \subsubsection{Proposal-based methods} This stream of methods follow the pipeline that bounding boxes are first obtained and masks of each bounding box are predicted afterwards, such as Panoptic-FPN~\cite{Panoptic-FPN}. These methods fuse their masks of \emph{thing} classes and masks of \emph{stuff} classes with merging modules~\cite{Liu_2019_CVPR, Li_2019_CVPR, Porzi_2019_CVPR}.
    \subsubsection{Proposal-free methods} Proposal-free methods solves panoptic segmentation by employing two separate branches to predict semantic masks and group pixels to instances. One of the most popular grouping methods is instance center regression, which predicts pixel-level offsets to instance centers~\cite{Neven_2019_CVPR, cheng2020panoptic}. Recently, following DETR~\cite{DETR}, multiple works introduce bipartite matching in their training~\cite{zhang2021knet, max-deeplab, cheng2021maskformer, li2022maskdino}, simplifying the process of panoptic segmentation.
    
    Inspire by the methods that introduce bipartite matching into their training pipeline, we propose \ours, the first framework on point cloud data which is able to exclusive predict panoptic groupings of points and can be trained in an end-to-end manner.













