
\section{Conclusion}
\label{sec:conclusion}

This paper reported on 6D pose estimation from a single RGB image by introducing novel primitive learning associated with each object. This paper presents a substantial improvement in the pose estimation even in the occluded case and comparable performance even using only synthetic images for training. The proposed method was validated using three public benchmark datasets yielding \ac{SOTA} performance for occluded and small objects.

\section{Experiment}
\label{sec:experiment}

In this section, we evaluate the proposed method using three different datasets.

\input{tab_linemod_occ.tex}

\subsection{Dataset}

In total, three datasets were used for the evaluation: LINEMOD \cite{hinterstoisser2012model}, Occlusion LINEMOD \cite{hinterstoisser2012model}, and YCB-Video \cite{xiang2017posecnn}. The LINEMOD and Occlusion LINEMOD datasets are widely exploited benchmarks for 6D object pose estimation and cover various challenges, including occlusion and texture-less. The LINEMOD dataset comprises 13 objects with 1,312 rendered images on each for the training and about 1,200 images per object for the test. The Occlusion-LINEMOD dataset shares the training images with the LINEMOD dataset. For the test, they provide pose information for eight occluded objects. A recently released dataset, the YCB-Video dataset is composed of 21 high-quality 3D models, offering 92 annotated video sequences per frame. These video sequences include various lighting conditions, noise in the capture, and occlusion.

\subsection{Training Detail}

We sampled 50,000 object poses and rendered 50,000 training images per object together with their associated primitives. The rendered object images were further augmented using the method proposed by \cite{mjeon-2019-cams} using domain randomization. We train our network mainly from synthetic images and additionally train with 180 real images for three types of dataset.

All of the networks were trained with the Adam optimizer on Titan V. The ResNet backbone was initialized with pre-trained weights provided in PyTorch. All networks were trained with a batch size of 50 and a 0.0001 learning rate for 40 epochs.

\subsection{Evaluation Metric}

\textbf{(i) 2D Projection error metric} To evaluate the pose estimation in terms of the 2D projection error we also use the same metric as in \cite{pvnet-2019-cvpr} and measure the mean pixel distance between projection of the 3D model and the image pixel points. We also count the estimation is correct if the error is less than 5 pixels.

\textbf{(ii) 3D Projection error metric} Similar to \cite{xiang2017posecnn}, two metrics were selected for evaluation. We chose the average distance metric (ADD)~\cite{hinterstoisser2012model} as the evaluation metric, which computes the distance of transformed 3D model points between the ground truth pose and predicted pose using ADD(-S) metric.
When the average distance value is less than 10\% of the 3D model's diameter, the predicted 6D pose is considered to be correct. This ADD metric, however, is not suitable to use with symmetrical objects. For symmetrical objects, the average distance is computed between the closest points.

\textbf{(iii) Pose estimation metric} For the original LINEMOD dataset, we directly measure the \ac{MAE} for rotation and translation together with other metrics.

\input{tab_ycb.tex}
\vspace{-3mm}

\subsection{6D Object Pose Estimation Evaluation}

We evaluate the performance of our model on the three datasets. The proposed method is compared against other \ac{SOTA} methods from holistic, keypoint-based and part-based approaches. We evaluate our method when training only with synthetic images (PrimA6D-S) and when training using 180 additional real images (PrimA6D-SR). Other existing methods were trained using additional real images. Qualitative results for all three datasets are presented in \figref{fig:linemod_result_fig} and \figref{fig:linemod_occ_result_fig}. Please also refer to \texttt{prima6d.mp4}. We excluded the comparison to the case without the primitive decoder due to drastically large error when no primitive decoder was used.

\subsubsection{LINEMOD Dataset Results}

\tabref{tab:linemod_add} and \tabref{tab:linemod_2dp} exhibit the evaluation for all 13 objects in the LINEMOD dataset. Among the keypoint or the part-based approaches, PVNet is the most recently released method with \ac{SOTA} performance and thus used as the main baseline method for comparison.

Over the LINEMOD dataset, PrimA6D-SR outperformed PVNet in terms of the ADD(-S) score and presented a comparable result in terms of the 2D projection metric. The different performance can be understood from the \ac{MAE} metric in \tabref{tab:linemod_2dp}. If the object center point in the image is found properly, the 2D projection metric is affected more by the rotational estimation accuracy and ADD(-S) is more critical to the translation inference performance. As can be seen, PVNet exhibited higher performance than PrimA6D-S in terms of the rotation \ac{MAE} metric, which results in a performance difference on the 2D projection metric. On the contrary, PrimA6D-S outperformed PVNet in terms of the translation \ac{MAE} metric, which results in a performance difference on the ADD(-S) score.

In addition, the PrimA6D(-S/-SR) presents the strength for small and texture-less objects that the drawback in the PVNet such as ape and cat. Especially, the PrimA6D(-S/-SR) has the forte on the translation estimation.

\subsubsection{Occlusion LINEMOD Dataset Results}

Tables in \tabref{tab:occ_add} list the performance on the Occlusion LINEMOD dataset. Here, we only evaluate the performance of models that do not have the refinement step. As can be seen in two tables, the PrimA6D(-S/-SR) outperformed PVNet in terms of the ADD(-S) score and presented a comparable result in terms of the 2D projection metric. As the PrimA6D(-S/-SR) reconstruct the corresponding object and primitive, we can affirm notable improvement on the occluded case. Moreover, the PrimA6D(-S/-SR) demonstrates striking accuracy for objects that are difficult to recognize by other methods due to small size such as cat and holepuncher.

\subsubsection{YCB-Video Dataset Result}

We further evaluate the pose estimation performance over the YCB dataset as shown in \tabref{tab:ycb}. The PrimA6D(-S/-SR) achieved enhanced inference capability even when trained solely from synthetic dataset (PrimA6D-S). The proposed method is capable of inferring 6D pose of objects that are difficult to recognize by other methods such as wood block.

Another notable point in YCB dataset is the overall down performance from LINEMOD; all approaches including PVNET and ours shows lower accuracy in YCB than LINEMOD. This is because of the challenging and realistic dataset nature of YCB. Despite the challenging dataset, the proposed method shows the smallest performance drop from LINEMOD both in terms of ADD(-S) and 2D projection error even when solely trained from synthetic data. This reveals the generalization capability of the proposed method while others lacking.

\subsection{Ablation Studies}

We further evaluate the proposed method by conducting ablation studies in terms of the effect of real images, effect of training with \ac{GAN} and bounding box accuracy.

\textbf{1) Effect of real image} The proposed method was designed to exploit synthetic data from CAD model. However, considering the potential discrepancy between CAD and real data, additional weight refinement using real training image shows very meaningful performance improvement. We train the network only from synthetic data (PrimA6D-S) and compare the performance improvement by adding real images in the training set (PrimA6D-SR). This comparison is provided for three datasets. Obviously we witness accuracy improvement in all metrics by adding real images in the training set.


\begin{figure}[!t]
	\centering
	\includegraphics[width=0.95\columnwidth]{figs/fig_5.pdf}

	\caption{\bl{Effect of GAN on occluded and symmetrical object.}}

	\label{fig:effect_gan}
	\vspace{-4mm}
\end{figure}



\textbf{2) Training with GAN} The adversarial loss and training strategy using \ac{GAN} provides a detailed refinement of the object and primitive reconstruction. As can be seen in \figref{fig:effect_gan}, the reconstructed result shows a blurry and obscure reconstruction when without using \ac{GAN}. This indicates the potential ambiguity in the rotation and translation estimation. Additional training with \ac{GAN} enables the \ac{VAE} to build discriminability. \bl{Also, regarding symmetrical objects with ambiguity for one axis, incorporating adversarial training can solve the potential axis ambiguity in the reconstructed primitive and completely form three orthogonal axes. In the experiment, we confirmed that the ambiguous axis in the primitive becomes the clear orthogonal axis for the other two axes according to the qualitative result.} Testing over Occlusion LINEMOD in \tabref{tab:occ_add} reveals this substantial improvement from the adversarial loss.

\input{tab_bb_ablation.tex}

\textbf{3) Bounding box accuracy} The input of the network is a cropped image from an object detection module hence the PrimA6D estimation performance may be dependent on the detection accuracy. We have examined the effect of the detected bounding box accuracy by reducing the \ac{IoU} to the original bounding box in terms of  ADD(-S) as this is more sensitive to the translational estimation accuracy. As can be seen in \tabref{tab:bb_ablation}, the performance drops as we reduce the \ac{IoU} from 1.0 to 0.75. As can be seen from \tabref{tab:bb_ablation}, \tabref{tab:linemod_add} and \tabref{tab:linemod_2dp}, the PrimA6D-SR still outperforms all other existing methods despite the performance degradation. More notably, the PrimA6D-S shows the exceptional result on the Occlusion LINEMOD dataset even with decreased IoU, being solely trained from synthetic data.

\subsection{Inference Time}

Since our method is applied after object detection, the total inference time is also affected by the running time of the object detection model. For example, we used M2Det \cite{DBLP:journals/corr/abs-1811-04533} that runs at 33.4 \ac{fps}. For the 6D object pose estimation of a single object in an image, the total process takes 31ms on average using a GTX 1080Ti GPU. The entire 6D pose estimation supports 16 to 17 \ac{fps}.

\subsection{Failure Cases}

\figref{fig:fail} shows failure cases when one axis is entirely obscured by the other two axes of the rotation primitive. This hidden primitive axis prevents the algorithm from extracting the correct keypoint yielding a larger rotation error. In future work, we aim to include estimated depth from the primitive to resolve this issue.


\begin{figure}[!t]
	\centering
	\includegraphics[width=0.85\columnwidth]{figs/fig_6.jpg}

	\caption{The examples of failure case.}

	\label{fig:fail}
	\vspace{-5mm}
\end{figure}

\begin{figure*}[!t]
	\centering
	\includegraphics[trim=0 60 0 0, clip, width=0.95\textwidth]{figs/fig_7.png}
    \caption{Qualitative pose estimation evaluation on the LINEMOD dataset. The red box indicates the ground truth pose while blue is the estimated pose using PrimA6D. Next to the inferred pose is reconstructed object and the predicted primitive associated to each object.}

	\label{fig:linemod_result_fig}
	\vspace{-4mm}
\end{figure*}

\begin{figure*}[!t]
	\centering
	\includegraphics[width=0.95\textwidth]{figs/fig_8.png}
    \caption{Qualitative pose estimation evaluation on the Occlusion LINEMOD dataset. The red box indicates the ground truth pose while blue is the estimated pose using PrimA6D. Next to the inferred pose is reconstructed object and the predicted primitive associated to each object.}

	\label{fig:linemod_occ_result_fig}
	\vspace{-4mm}
\end{figure*}


\section{Introduction}
\label{sec:intro}


The 6D object pose estimation intends to recover the orientation and translation of the object relative to the camera.
When solving 6D pose estimation, reaching accurate, fast and robust solutions has been a key element of augmented reality \cite{pvnet-2019-cvpr}, robotic tasks \cite{billings2018silhonet, periyasamy2018robust, wang2019densefusion}, and autonomous vehicles \cite{manhardt2019roi}. Despite the widespread use of RGB-D cameras, single image-based pose estimation is still essential when an RGB-D camera is not applicable.

Recent advances in deep learning-based object pose estimation has been groundbreaking. Direct end-to-end 6D pose regression suffered from the non-linearity of the rotation, and studies tackled this issue through orientation representation \cite{saxena-2009-icra} or combining \ac{PnP} and \ac{RANSAC}. The latter approaches reported current \acf{SOTA} performance by leveraging a two-step approach for pose estimation \cite{pvnet-2019-cvpr, brachmann2017dsac, learning-less-2017}. These approaches extract keypoints from the RGB image using deep learning and compute the 6D pose of the object using 2D-3D correspondence matching, such as the \ac{PnP} algorithm.



\begin{figure}[!t]
	\centering
	\includegraphics[width=0.9\columnwidth]{figs/fig_1.pdf}

	\caption{We train VAE to reconstruct objects and the associated primitive followed by the keypoints extraction over the inferred primitive. Keypoints are extracted from each corner plus the center, the same as the object center (i.e., 21).}

	\label{fig:oveview}
	\vspace{-5mm}
\end{figure}

We were also partially motivated by the finding in \cite{sundermeyer2018implicit} that an \ac{AAE} could implicitly represent a rotation when training an \ac{AAE} from a 3D CAD. The main advantage of this approach is that there is no need for manual annotation to prepare ground pose labeled training images. While \cite{sundermeyer2018implicit} only solved for the orientation retrieval given a discretized orientation set, we reconstruct a primitive from which keypoints are extracted under occlusion and complete a full orientation regression.

This paper solves for the direct 6D pose regression by introducing a novel primitive descriptor in \figref{fig:oveview}. The proposed solution, primitive associated 6D (PrimA6D) pose estimation, combines a direct and holistic understanding of an image with keypoint based PnP for the final orientation regression. More specifically, we newly introduce the primitive decoder in addition to the \ac{VAE} to increase the discriminability of the orientation inference. Via this step, like the aforementioned studies, we aim to find reliable keypoints in an image for pose regression extracting from the reconstructed primitives. Differing from previous methods, our method presents the following contributions.


\begin{figure*}[!t]
	\centering
	\includegraphics[width=0.95\textwidth]{figs/fig_2.pdf}

	\caption{The reconstruction stage of PrimA6D. In the reconstruction stage, an encoder using ResNext50 \cite{DBLP:journals/corr/XieGDTH16} and an reconstruction decoder consisted of fully deconvolutional layers form the \ac{VAE}. We reconstruct both object and primitive in a separate decoder. To increase the discriminability, we additionally introduce adversarial loss and train using \ac{GAN}.}

	\label{fig:network}
	\vspace{-4mm}
\end{figure*}

\begin{itemize}

	\item We propose a novel 6D object pose estimation network called PrimA6D, which introduces a rotation primitive reconstruction and its associated keypoints to enhance the orientation inference.

	\item The proposed estimation scheme mitigates the manual effort of preparing label annotated images for training. The proposed primitive is straightforward to generate given rotation in 3D CAD; the trained \ac{VAE} allows us to learn from synthetic images via domain randomization robust to occlusion and symmetry.

	\item We verify the meaningful performance improvement of 6D object pose estimation compared to the existing \ac{SOTA} 6D pose regression methods. The proposed method improved estimation over LINEMOD as 97.62\% (PVNet 86.27\%), Occlusion LINEMOD as 59.77\% (PVNet 40.77\%), and YCB-Video as 94.43\% (PVNet 73.78\%) in terms of ADD(-S).

	\item The proposed method is more generalizable than existing methods showing less performance variance between datasets. This superior generalization capability is essential for robotic manipulation and automation tasks.

\end{itemize}

\section{Method}
\label{sec:method}


\begin{figure}[!t]
	\centering
	\includegraphics[width=1\columnwidth]{figs/fig_3.pdf}
  \caption{The rotation and translation inference. We use keypoints learned from the reconstructed primitive for the orientation estimation using PnP. In the translation inference stage, the network input is created using the output of the reconstruction decoder through object relocalization. }

	\label{fig:pose}
	\vspace{-1mm}
\end{figure}


Given an RGB image, we compute the 6D pose of an object relative to the camera in terms of the its relative transform ${\mathbf{T}}^{C}_{O}$ between the camera coordinate frame ${C}$ and the object coordinate frame ${O}$. The 6D pose is represented by quaternion and translation vectors. Similar to \cite{pix2pose} and \cite{sundermeyer2018implicit}, the proposed method can be accompanied with other detection modules such as M2Det \cite{DBLP:journals/corr/abs-1811-04533}. We use a cropped image using a bounding box detected from other methods as input to our network.



\subsection{Training Set Prepration}

As indicated earlier, our aim is at enabling 6D pose regression without requiring the real images. For each pair of a 3D CAD object and the associated camera pose, we prepare the training set consists of a bounding box, rendered primitive, the target keypoints (the 20 corner points and a center point), and rendered training image. The primitive images represent the rotation of the object having different colors per axis as in \figref{fig:oveview}.

\subsection{Object and Primitive Reconstruction}

Using the training data, the first phase of the proposed method consists of object and primitive reconstruction as shown in \figref{fig:network}. We train the \ac{VAE} having an encoder and two reconstruction decoders. This step is partially similar to \ac{AAE} in \cite{sundermeyer2018implicit}. In \cite{sundermeyer2018implicit}, the authors verified that a trained \ac{AAE} using a geometric augmentation technique learns the representation of the object's orientation and generates the latent code with data for the object orientation. We adopt the same technique in the training phase by rendering the training images with various poses of the object. The training images are augmented using the domain randomization technique \cite{mjeon-2019-cams} to bridge the gap between real and rendered images. Augmented images are fed into the encoder, and clean target images are obtained through the reconstruction decoder using four losses as follows. Differing from \cite{sundermeyer2018implicit}, our training strategy is to use \ac{GAN} by introducing a discriminator for the reconstructed object and primitive.

\subsubsection{Object Reconstruction Loss}

The first loss focuses on the reconstruction of an object by comparing the pixel-wise loss. To prevent overfitting in the reconstruction stage, we use the top-k pixel-wise L2 loss as follow.
\begin{equation}
	\label{eq:reconst_loss}
	\mathcal{L}_{O}=\frac{1}{K}\sum_{i=1}^{K} \ell_{[i]}(\left \| x-\hat{x} \right \|^2),
\end{equation}
where $\ell_{[i]}$ is a function that extracts the pixel with the $i^\text{th}$ largest error \cite{atkloss-2017-nips}. We use $K=128$ in this paper. In \eqref{eq:reconst_loss}, $x$ is the target image pixel and $\hat{x}$ is the predicted image pixel respectively. This initial latent code encompasses the orientation but it is not sufficiently discriminative for regression. To increase the discriminability, we introduce the rotational primitive decoder to embolden the orientation of the code.

\subsubsection{Primitive Reconstruction Loss}

The rendered primitive image becomes the prediction target of the primitive decoder. Using the color-coded axes in primitive image, we designed the color based axis-aligning loss function for the primitive decoder. When computing the loss for the channels, we consider a per-pixel intensity difference $d_c$ for each channels $c \in \mathcal{C} = [R,G,B]$.
\begin{eqnarray}
	\label{eq:primitive}
	d_{c} &=& x_{c}-\hat{x}_{c}\\
	S_{c} &=&\frac{1}{K}\sum_{i=1}^{K}\ell_{[i]}(\exp({\alpha \left | d_c \right |}) \cdot \left \|  d_c \right \|^{2}) \\
	C_{c} &=&\frac{1}{K}\sum_{i=1}^{K}\ell_{[i]}( \left \|  d_c \right \|^{2})\\
	\mathcal{L}_{P}&=&\sum_{c \in \mathcal{C}}S_{c} \exp \left( \frac{C_{c}}{\sum_{k \in \mathcal{C}}C_{k}} \right)
\end{eqnarray}
Here, the $S_{c}$ measures misalignment of each axis in terms of the pixel intensity difference per channel. The $C_{c}$ measures the overall misalignment per channel and discerns between channels by penalizing the channel showing a larger error when the primitive reconstruction loss $L_P$ is constructed. The $\alpha$ is a weight constant ($\alpha=5$ in this paper).

\subsubsection{Overall VAE Loss}

Using these two reconstruction losses with \ac{KL} divergence loss, we train the \ac{VAE} to generate both an object image and a primitive image. By adopting \ac{VAE}, the network encodes input image $x$ to the parameters of a Gaussian distribution $q(z|x)$. In doing so, we minimize the \ac{KL} divergence between $q(z|x)$ and $\mathcal{N}(0,I)$. The overall loss function for the reconstruction stack becomes as below.
\begin{equation}
	\label{eq:loss_kld}
	\mathcal{L}_{R} = \mathcal{L}_{O} + \mathcal{L}_{P} + D_{KL} \left( q(z|x)||\mathcal{N}(0,I) \right)
\end{equation}


\subsubsection{Adversarial Loss}

We further aim to improve the quality of the reconstruction using training strategy of \ac{GAN}. For instance, the reconstructed primitive sometimes becomes ambiguous as shown in \figref{fig:effect_gan} and may result in incorrect keypoints learning. Regarding this issue, introducing adversarial loss improved the primitive/object prediction accuracy and thereby the overall pose inference performance. For the target primitive/object image $X$ and the reconstructed image $\hat{X}$, we consider the following loss to train using \ac{GAN} \cite{goodfellow2014generative}.
\begin{equation}
	\label{eq:loss_adv}
\mathcal{L}_{A} = \mathbb{E}_{\hat{X}}[\log D(\hat{X})] + \mathbb{E}_{X}[\log (1-D(G(X)))].
\end{equation}
Here we use $D{(\cdot)}$ to distinguish if the predicted primitive/object is the target primitive/object from 3D model (true) or generated by $G{(\cdot)}$ (fake). Following the training strategy of \ac{GAN}, we solve for $G$ and $D$ by minimizing $\mathcal{L}^G_{A}=\mathbb{E}_{\hat{X}}[\log D(\hat{X})]$ and maximizing $\mathcal{L}^D_{A}=\mathbb{E}_{X}[\log (1-D(G(X)))]$.


\begin{figure}[!t]
	\centering
	\includegraphics[width=0.8\columnwidth]{figs/fig_4.pdf}
	\caption{The input image of the reconstruction network is ($\kappa$=1.3) larger than the actual bounding-box allowing complete object reconstruction even for inaccurate bounding-box.}

	\label{fig:synthetic_dataset}
	\vspace{-4mm}
\end{figure}

\subsection{Orientation Regression via PnP}

From the reconstructed primitive we estimate the rotation first by learning the keypoints from it. Using generated keypoints in the training set preparation and via pixel-wise L2 loss, we adopt ResNet18 \cite{resnet} and extract the keypoints of the reconstructed primitive using the loss as
\begin{equation}
	\label{eq:pnp_R}
	\mathcal{L}_{rot}=\sum\left \| x_k-\hat{x_k} \right \|^2.
\end{equation}
Since the keypoints consists of corners and the center point of the object which is the same as the primitive center, we utilize this estimated center point, $C=x_c = (u_c,v_c)$ for the translation estimation.

\subsection{Translation Regression}

\input{tab_linemod_add.tex}
\input{tab_linemod_2dp_mae.tex}

Given the estimated orientation, we then infer the 3D translation by computing the scene depth $T_{z}$ of the object center (i.e., primitive center). The reconstructed image is relocated to the original image using the bounding box of the object. This process is called \textit{object relocalization} (\figref{fig:pose}).

As the sample illustration in \figref{fig:synthetic_dataset} shows, the reconstruction occurs \textit{without} assuming the bounding box is centered and \bl{thereby is robust to inaccurate object detection.} The reconstruction phase handles the inaccurate bounding box location by properly reconstructing the object even at the off-centered position. The inflation scaling factor $\kappa$ was chosen to handle the detected bounding box with \ac{IoU} \bl{0.75}. A small $\kappa$ fails to handle inaccuracy in objection detection phase, and a large $\kappa$ might deteriorate the performance with increased empty space near borders. Because this re-localization replaces an object with a reconstructed object, the translation can be regressed over occlusion-free objects. From this relocalized full image, we again use ResNet18 to regress the scene depth $T_{z}$ to the object center.

Using $T_{z}$, the bounding-box, and the camera intrinsic matrix $K$, we compute the $T_{x}$ and $T_{y}$ components of translation as follows from the estimated object center $(u_c, v_c)$. The focal length $f$ and principal point $(u_p,v_p)$ are obtained from the calibration matrix while the bounding box is known when preparing the training data.
\begin{eqnarray}
	\label{eq:trans}
	u_c=f_{u}\frac{T_x}{T_z}+u_p \quad\text{and}\quad
	u_v=f_{v}\frac{T_y}{T_z}+v_p.
\end{eqnarray}






\section{Related Works}
\label{sec:related}



PnP based approaches establish 2D-3D correspondences from keypoints or patches and solve for the pose from these correspondences. Keypoint-based approaches define specific features to infer the pose from an image, such as the corner points of 3D bounding-box. These approaches chiefly involve two steps: first, the 2D keypoint of an object in the image is computed, and then an object pose is regressed using the \ac{PnP} algorithm. In BB8 \cite{rad2017bb8}, the authors predicted the 2D projection corner points of the object's 3D bounding box. Then, they computed the object pose from 2D-3D correspondences using \ac{PnP} algorithm. Similarly, \cite{tekin2018real} additionally added the central point of object to the keypoints of BB8 \cite{rad2017bb8}. Unfortunately, \cite{rad2017bb8,tekin2018real} revealed the weakness to the occlusion that prevents defining the correct bounding box of an object. To alleviate this issue, PVNet \cite{pvnet-2019-cvpr} predicted the unit vectors that point to predefined keypoints for each pixel in an object. By computing the locations of these keypoints using unit vector voting, the authors substantially improved their accuracy. However, these keypoint-based approaches may suffer when an object lacks texture, whereas the proposed primitive can be robustly reconstructed even for small and textureless objects.


The part-based approaches extract parts of an object called patches for pose regression. This method is  frequently used with depth data. For instance, \cite{brachmann2016uncertainty} jointly predicted the object labels and object coordinates for every pixel, then computed the object pose from the predicted 2D-3D correspondence matching. In \cite{oberweger2018making}, the authors extracted heatmaps of multiple keypoints from local patches then computed the object pose using the \ac{PnP} algorithm. Depth data may not always be available and we propose using a single RGB as input. Recently, \cite{pix2pose} exploited \ac{AE} to generate an image representing 3D coordinate per each pixel. Together with the reconstructed image containing 3D information, the expected error per pixel was also estimated to sort meaningful 2D-3D correspondence.






Unlike the aforementioned approaches, which consist of steps, holistic approaches utilize the overall shape of the object appearing in an image to directly regress the object pose relative to the camera. In Deep-6DPose \cite{do2018deep}, the authors found the regions where the objects were located through the \ac{RPN}. Despite solving for both object detection and 6D pose estimation, their method was vulnerable to small and symmetrical objects since they relied on the detected regions. In SSD-6D \cite{kehl2017ssd}, the authors used the discrete 6D pose space to represent the discriminable viewpoint. SSD-6D is a \ac{SSD}-style pose estimation model that extends the work of \cite{liu2016ssd}. Their method had a faster inference time than other methods though it sacrificed accuracy.

Incorporating a mask has been widely adopted. For example, \cite{billings2018silhonet,xiang2017posecnn,periyasamy2018robust,wu2018real} utilized an mask of the object to solve the pose estimation task. The well-known PoseCNN \cite{xiang2017posecnn} required an \ac{ICP} refinement for better accuracy. Stemming from PoseCNN, \cite{wu2018real} enabled a real-time performance with RGB data. The authors of \cite{xiang2017posecnn,wu2018real} also reported meaningful results, but their works were sensitive to the issue of object occlusion. Tackling the occlusion issue, SilhoNet \cite{billings2018silhonet} constructed an occlusion-free mask even from the occluded region, through CNN. This type of mask level restoration enhanced the performance over occlusion, however, yielded low accuracy for symmetrical objects. Unlike these methods relying on masks, we exploit the reconstructed object to not only regress the translation but further tackle the occlusion issue.


All the abovementioned methods require true pose-labeled images for training. Another stream of studies has focused on leveraging latent code without the need for labeled training images via \ac{AAE} to solve this issue. Without requiring the pose label from object 6D pose estimation, \cite{sundermeyer2018implicit} utilized the latent code generated from the object recovery process using \ac{AE}. The object pose was determined by a similarity comparison with the predefined latent codebook for each pose, but the accuracy was dominated by the size of the codebook with a discretized orientation class. We have overcome this accuracy limitation and achieved substantially improved performance from a synthetic dataset.



\section*{ACKNOWLEDGMENT}

\balance
\small
\bibliographystyle{IEEEtranN}







