\subsection{Dataset Statistics}


\subsection{Evaluation of the Annotation Method}
\label{sec:evalAnno}

For validating the accuracy of our  annotation method, we manually annotated the
3D locations  of the  3D joints in  randomly selected frames  of a  sequence, by
relying on  the consolidated point cloud  from the 5 cameras.   We then compared
these locations to the ones predicted with our method (explained in Section~\ref{sec:multiCameraSetup}) using the multi-camera setup.

As shown in the last column of Table~\ref{tab:table2}, our method achieves an
average joint error accuracy of lower than 8mm on average, with an Area Under the Curve
metric~(AUC) of 0.79. This metric is comparable with the results reported
for the recent FreiHAND dataset~\cite{zimmermann2019freihand}~(AUC=0.791). Note that the
occlusions in our dataset are higher due to larger objects and that we do not use green screens.

To analyze the influence of the different terms in Eq.~\eqref{eq:global}, we run
the optimization  of Eq.~\eqref{eq:global}  by enabling only  a subset  of these
terms, and  report the results  in Table~\ref{tab:table2}.  While  $E_\text{silh}$ and
$E_\text{dpt}$ terms alone  cannot provide good pose estimates,  together they provide
better estimates  as it leads  to a loss function  with less local  minima.  The
$E_\text{3D}$  term provides  a  minor improvement  in estimates  but  speeds up  the
convergence.  Though  the physical plausibility  term $E_\text{phy}$ does not  help in
improving the pose  estimates, it results in more natural  grasps.  The last two
columns  show the  effect of  multi-frame based  joint optimization  compared to
single-frame  based  optimization  when  all   the  terms  are  considered.  The
multi-frame  multi-camera based  optimization over  all the  terms improves  the
accuracy by about 15\%.

The  accuracy of  the single  camera based  annotation method  is calculated  by
considering the annotations from the multi-camera method as ground truth for a given
sequence. More specifically, for a sequence of
1000  frames, we compute the average difference   between  hand+object  mesh   vertices  obtained  from  single  and
multi-camera setups. Further, we calculate the  accuracy after each stage of the
single-camera  setup.  The  results  are given  in  Table~\ref{tab:table5}.  The
estimated poses  with these two  methods are consistent  with each other  with an
average mesh error  of 0.77cm and 0.45cm for hand  and object, respectively. The
final refinement stage yields a 15\% improvement in accuracy.





















\begin{table}[]
  \begin{center}
    \resizebox{0.47\textwidth}{!}{%
      \begin{tabular}{lcccc}
	\toprule
	Method & \specialcell{Mesh Error$\downarrow$} & F@5mm$\uparrow$ & F@15mm$\uparrow$  & \specialcell{Joint Error$\downarrow$}\\
	\midrule
	Joints2D & 1.14 & 0.49 & 0.93 & 3.14 \\
	Joints2D + Dir. Vec. & \textbf{1.06} & \textbf{0.51} & \textbf{0.94} & \textbf{3.04} \\

	\cite{hasson2019learning}  & 1.10 & 0.46 & 0.93 & 3.18 \\
	\bottomrule
    \end{tabular}}
    \vspace{-2mm}
    \caption{Evaluation  of  different  methods   for  single  frame  hand  pose
      prediction. The Mesh Error (in cm) and \textit{F}-score are obtained after
      aligning the  predicted meshes  with ground truth  meshes. The  Mean joint
      error (in  cm) is obtained after  aligning the position of  the root joint
      and overall scale with the ground  truth. Hand pose prediction using joint
      direction  predictions along  with  2D joint  predictions provides  better
      accuracy    than   directly    predicting the   MANO    parameters   as    in
      \cite{hasson2019learning}.}
    \vspace{-6mm}
    \label{tab:tablePoseEst}
  \end{center}
  
\end{table}


\subsection{Evaluation of Hand Pose Prediction Method}

We  trained our  single frame  hand  pose prediction  method explained  in
Section~\ref{sec:handposeSingleFrameSec} on 66,034  frames from our \datasetname
dataset.  We evaluated it  on a test set of 13  sequences captured from different
viewpoints  and  totaling  11,524   frames.  The  test  set  sequences also  contain
subjects and objects  not present  in  the training  set.

We  report three  different metrics  from previous  works: Mean  joint position
error after aligning the position of the root joint and global scale with ground
truth~\cite{Zimmermann2017}; Mesh error measuring the average Euclidean distance
between predicted  and ground truth mesh vertices~\cite{zimmermann2019freihand};  and the
$F$-score~\cite{zimmermann2019freihand},  defined as  the harmonic  mean between
recall and  precision between two meshes  given a distance threshold.   The mesh
error  and \textit{F}-score  are obtained  after aligning  the predicted  meshes
using  Procrustes alignment  with the  ground truth  meshes and  hence does  not
measure the accuracy of wrist joint rotation.  The mean joint error on the other
hand considers wrist joint location  as the 3D points  are not rotated
before evaluation.

To understand the effect of joint direction predictions on the overall accuracy,
we evaluate  the results  of the  MANO fitting  by dropping  the second  term in
Eq.~\eqref{eq:manoFitEst}.  We also compare our  results with the hand branch of
\cite{hasson2019learning}, a  very recent work  that predicts the MANO  pose and
shape parameters directly from a single RGB image, retrained on our dataset.  As
shown in Table~\ref{tab:tablePoseEst}, predicting joint directions along with 2D
joint locations  significantly improves the  hand pose estimation  accuracy.  It
can also be inferred that predicting 2D hand joint locations and fitting MANO model to them is  more   accurate  than   direct   MANO  parameter   predictions  as   in
\cite{hasson2019learning}.      Qualitative     results     are     shown     in
Fig.~\ref{fig:baseline_qualitative}. The last row shows that our method robustly
predicts hand poses even when interacting with unknown objects.


\comment{
  \begin{figure}
    \begin{center}

      \includegraphics[page=17, width=0.3\linewidth, trim=6cm 1.35cm 6cm 1.9cm,clip]{figures/singleFramePose/handposes}
      \includegraphics[page=9, width=0.3\linewidth, trim=6cm 1.35cm 6cm 1.9cm,clip]{figures/singleFramePose/handposes}
      \includegraphics[page=6, width=0.3\linewidth, trim=6cm 1.35cm 6cm 1.9cm,clip]{figures/singleFramePose/handposes} \\
      \includegraphics[page=10, width=0.3\linewidth, trim=6cm 1.35cm 6cm 1.9cm,clip]{figures/singleFramePose/handposes}
      \includegraphics[page=19, width=0.3\linewidth, trim=6cm 1.35cm 6cm 1.9cm,clip]{figures/singleFramePose/handposes}
      \includegraphics[page=7, width=0.3\linewidth, trim=6cm 1.35cm 6cm 1.9cm,clip]{figures/singleFramePose/handposes} \\
      \includegraphics[page=18, width=0.3\linewidth, trim=6cm 1.35cm 6cm 1.9cm,clip]{figures/singleFramePose/handposes}
      \includegraphics[page=20, width=0.3\linewidth, trim=6cm 1.35cm 6cm 1.9cm,clip]{figures/singleFramePose/handposes}
      \includegraphics[page=12, width=0.3\linewidth, trim=6cm 1.35cm 6cm 1.9cm,clip]{figures/singleFramePose/handposes}
    \end{center}
    \vspace{-2mm}
    \caption{Qualitative results for single RGB frame based hand pose estimation method. We recover hand poses when it is heavily occluded by objects and in cluttered scenes.}
    \label{fig:baseline_qualitative}
    \vspace{-5mm}
  \end{figure}
}

\begin{figure}
  \begin{center}
    
    \includegraphics[page=17, width=0.24\linewidth, trim=6cm 1.35cm 6cm 1.9cm,clip]{figures/singleFramePose/handposes2}
    \includegraphics[page=9, width=0.24\linewidth, trim=6cm 1.35cm 6cm 1.9cm,clip]{figures/singleFramePose/handposes2}
    \includegraphics[page=6, width=0.24\linewidth, trim=6cm 1.35cm 6cm 1.9cm,clip]{figures/singleFramePose/handposes2} 
    \includegraphics[page=10, width=0.24\linewidth, trim=6cm 1.35cm 6cm 1.9cm,clip]{figures/singleFramePose/handposes2}\\
    \includegraphics[page=19, width=0.24\linewidth, trim=6cm 1.35cm 6cm 1.9cm,clip]{figures/singleFramePose/handposes2}
    \includegraphics[page=7, width=0.24\linewidth, trim=6cm 1.35cm 6cm 1.9cm,clip]{figures/singleFramePose/handposes2} 
    \includegraphics[page=18, width=0.24\linewidth, trim=6cm 1.35cm 6cm 1.9cm,clip]{figures/singleFramePose/handposes2}
    \includegraphics[page=20, width=0.24\linewidth, trim=6cm 1.35cm 6cm 1.9cm,clip]{figures/singleFramePose/handposes2}\\
    \includegraphics[page=22, width=0.24\linewidth, trim=6cm 1.35cm 6cm 1.9cm,clip]{figures/singleFramePose/handposes2}
    \includegraphics[page=24, width=0.24\linewidth, trim=6cm 1.35cm 6cm 1.9cm,clip]{figures/singleFramePose/handposes2}
    \includegraphics[page=25, width=0.24\linewidth, trim=6cm 1.35cm 6cm 1.9cm,clip]{figures/singleFramePose/handposes2}
    \includegraphics[page=28, width=0.24\linewidth, trim=6cm 1.35cm 6cm 1.9cm,clip]{figures/singleFramePose/handposes2}
  \end{center}
  \vspace{-2mm}
  \caption{Qualitative results of  our single color image  hand pose estimation
    method. It can recover hand poses even when the hand is heavily occluded by objects and in
    cluttered scenes. The last row shows it can handle unseen objects.}
  \label{fig:baseline_qualitative}
  \vspace{-5mm}
\end{figure}



\comment{
  \paragraph{PCK metric: Percentage of Correct Keypoints in a Side View.}
  We use  a second  RGB camera registered  with respect  to the  RGB-D camera  used to
  capture  the sequences.   This camera  has a  viewpoint that  is approximately
  orthogonal to  the RGB-D  camera.  We use  the first camera  to obtain  the pose
  annotations and the second camera for  validation only. We manually annotate the
  hand pose  with the 2D  joint locations of visible  joints in the  second camera
  view for 10 randomly chosen frames  of 15 different sequences, totaling over 150
  frames.   Further, we  annotated  the object  pose by  providing  the 2D  corner
  locations of visible selected points in the second camera view totaling over 100
  frames.

  To evaluate the PCK metric, we project  the estimated 3D pose annotations on the
  second camera, and  compute the 2D distances between the  manual annotations and
  the projected automatic  annotations. Fig.~\ref{fig:anno_frameswithin} shows the
  PCK metric  for varying  distance thresholds, \ie, the  percentage of  3D points
  that have an error smaller than  a threshold. The final stage of the global optimization discussed in Section \ref{GO} increases the accuracy of annotations by about 7\% by refining the poses. Qualitative results before and after the final stage of global optimization are shown in Fig.~\ref{fig:global_qualitative}. More qualitative examples on different objects, persons and poses are shown in Fig.~\ref{fig:anno_qualitative}. 



  \begin{figure}
    \begin{center}
      \includegraphics[width=0.45\linewidth]{plots/PCK_Dataset_Hand}
      \includegraphics[width=0.45\linewidth]{plots/PCK_Dataset_Object}
    \end{center}
    \vspace{-5mm}
    \caption{PCK metric (before and after final refinement stage) for evaluating the proposed pose annotation method on 15 sequences with  varying distance thresholds.  Left: Hand.  Right: Object.}
    \label{fig:anno_frameswithin}
  \end{figure}


  \begin{figure}
    \begin{center}
      \includegraphics[width=0.28\linewidth]{figures/mask_0045pngimg2_SM1_b.png}
      \includegraphics[width=0.28\linewidth]{figures/mask_0304pngimg2_MC1_b.png}
      \includegraphics[width=0.28\linewidth]{figures/mask_0652jpgimg2_SM2_b.jpg} \\
      \includegraphics[width=0.28\linewidth]{figures/mask_0045jpgimg2_SM1_a.jpg}
      \includegraphics[width=0.28\linewidth]{figures/mask_0304pngimg2_MC1_a.jpg}
      \includegraphics[width=0.28\linewidth]{figures/mask_0652jpgimg2_SM2_a.jpg}
    \end{center}
    \vspace{-2mm}
    \caption{Qualitative results of the proposed annotation
      method. Each column shows the 3D models of the hand and
      object superimposed on the images from the side view camera
      before and after refinement stage, respectively. The final
      refinement still improves the estimated poses.}
    \vspace{-6mm}
    \label{fig:global_qualitative}
  \end{figure}


  \begin{figure}
    \hspace{-2mm}
    \begin{center}
      \includegraphics[width=0.6\linewidth]{plots/Dataset_depth_errors}
    \end{center}
    \vspace{-4mm}
    \caption{Reconstruction error. We plot the fraction of depth residuals where the depth residual is smaller than a threshold. A large fraction of errors are well below 10~mm, which shows that the articulated models can explain the captured data accurately.}
    \label{fig:depth_errors}
  \end{figure}

  \begin{figure}
    \begin{center}
      \includegraphics[width=0.65\linewidth]{plots/PCK_HONet_hand}
    \end{center}
    \vspace{-5mm}
    \caption{Evaluation of our baseline for hand pose estimation. We plot the percentage of correct keypoints~(PCK) over distance threshold. The dashed curve shows that the synthetic images used to train the image segmentation are clearly not sufficient to train this baseline, even if they work well for the segmentation task. }
    \label{fig:baseline_hand}
    \vspace{-5mm}
  \end{figure}
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

  \vspace{-6mm}

  \paragraph{Reconstruction error.}
  As an additional  evaluation, we rely on  a reconstruction error.  To  do so, we
  compute the  difference between the  captured depth maps  and the the  depth maps
  generated  using the  rendered  models of  the  hand and  the  object under  the
  estimated  poses.    Fig.~\ref{fig:depth_errors}  shows  the   evaluation for three different objects \textit{Mustard bottle, Cracker box} and \textit{Sugar box}.   The
  majority of all  errors, around 90\%, are below 10~mm.  The remaining errors can
  be attributed to  depth noise from the  sensor and the wrist  and forearm, which
  are not present in our model.
}




\subsection{3D Hand and Object Poses}
\label{sec:poses}

We    aim   to    estimate    the   3D    poses    $\mathcal{P}   =    \{(\text{p}_\text{h}^t,
\text{p}_\text{o}^t)\}^{N_F}_{t=1}$ for both  the hand and the object in  all the images
of the sequence.  We adopt the MANO hand model~\cite{romero2017embodied} and use
the objects from the YCB-Video dataset~\cite{posecnn2018} as their corresponding
3D models are available and of good quality. The MANO hand pose $\text{p}_\text{h}^t \in
\mathbb{R}^{51}$ consists of 45 DoF~(3 DoF for each of the 15 finger joints) plus 6 DoF
for rotation  and translation of the  wrist joint.  The 15  joints together with
the wrist joint form  a kinematic tree with  the wrist joint node as  the first parent
node.  In  addition to  the pose  parameters $\text{p}_\text{h}^t$,  the hand  model has
shape parameters $\boldsymbol{\beta} \in \mathbb{R}^{10}$ that are  fixed for a given person and we
follow a  method similar to  \cite{Tan_2016_CVPR} to estimate  these parameters.
More  details  about  the  shape   parameter  estimation  are  provided  in  the
supplementary material. The object  pose $\text{p}_\text{o}^t \in \text{SE}(3)$ consists
of 6 DoF for global rotation and translation.





\subsection{Cost Function}
\label{sec:cost}

We  formulate   the  hand+object  pose   estimation  as  an   energy 
minimization problem:
\begin{equation}
\label{eq:global}
\hat{\mathcal{P}} = \arg\min\limits_{\mathcal{P}} \sum^{N_F}_{t=1} 
\big(E_\mathcal{D}(\text{p}_\text{h}^t, \text{p}_\text{o}^t) + E_\mathcal{C}(\text{p}_\text{h}^t, \text{p}_\text{o}^t)\big)\> ,
\end{equation}
where $E_\mathcal{D}$ and  $E_\mathcal{C}$ represent the energy  from data terms
and constraints, respectively.  We define $E_\mathcal{D}$ as,
\begin{equation}
  \begin{array}{r}
    \!\!\!\!\mvleft
    E_\mathcal{D}(\text{p}_\text{h}^t, \text{p}_\text{o}^t) = \sum\limits^{N_C}_{c=1} \Big( \alpha E_\text{mask}(\text{I}^t_c,\text{p}_\text{h}^t,\text{p}_\text{o}^t) \:+ \beta E_\text{dpt}(\text{D}^t_c, \text{p}_\text{h}^t,\text{p}_\text{o}^t)  + \\[0.5em]
   \!\!\!\! \gamma E_\text{j2D}(\text{I}^t_c, \text{p}_\text{h}^t) \Big) \> + \delta E_\text{3D}(\{\text{D}^t_c\}_{c=1..N_C}, \text{p}_\text{h}^t,\text{p}_\text{o}^t) \> ,
\label{eq:dataTerm}
\end{array}
\end{equation}

 
 
 
 
 




\noindent where $E_\text{mask}(\cdot)$ is a  silhouette discrepancy term, $E_\text{dpt}(\cdot)$ a
depth  residual  term,  $E_\text{j2D}(\cdot)$  a 2D  error  in  hand  joint
locations,  and $E_\text{3D}(\cdot)$  a  3D  distance term.  This  last  term is  not
absolutely necessary,  however, we  observed  that it  significantly  speeds up  convergence.
$\alpha$, $\beta$,  $\gamma$, $\delta$ are weights.


The constraints energy $E_\mathcal{C}$ is defined as,
\begin{align}
E_\mathcal{C}(\text{p}_\text{h}^t, \text{p}_\text{o}^t)  = \; & \epsilon E_\text{joint}(\text{p}_\text{h}^t) \; + \zeta E_\text{phy}(\text{p}_\text{h}^t,\text{p}_\text{o}^t) \; + \nonumber\\
& \eta E_\text{tc}(\text{p}_\text{h}^t,\text{p}_\text{o}^t,\text{p}_\text{h}^{t-1},\text{p}_\text{o}^{t-1},\text{p}_\text{h}^{t-2},\text{p}_\text{o}^{t-2}) \> ,
\end{align}
where $E_\text{joint}(\cdot)$  denotes a prior on  the hand pose to  prevent unnatural
poses, $E_\text{phy}(\cdot)$ is a physical plausibility term ensuring the hand and the
object  do not  interpenetrate,  and $E_\text{tc}(\cdot)$  is  a temporal  consistency
term. The  terms are weighted by  parameters $\epsilon$, $\zeta$ and  $\eta$.

We detail  each of the terms  in $E_\mathcal{D}$ and $E_\mathcal{C}$  below. For
simplicity, we  omit the  frame index  $t$ from our  above notation  except when
necessary.


\vspace{-0.3cm}
\paragraph{Silhouette discrepancy term $E_\text{mask}$. }
The $E_\text{mask}(\cdot)$ term compares the silhouettes  of the hand and the object models
rendered  with the  current estimated  poses and  their segmentation  masks.  We
obtain a segmentation $S_c(\text{I})$ of the hand and the object in the color image $\text{I}$ of camera $c$ using
DeepLabv3~\cite{deeplabv2017}  trained   on  images  created   by  synthetically
over-laying  and under-laying  images of  hands on  YCB objects.  More
  details about this step are given in the supplementary material. The hand and
object  models  are  rendered  on   the  camera  plane  using  a  differentiable
renderer~\cite{Henderson2019},  which  enables   computing  the  derivatives  of
$E_\text{mask}$ with  respect to the pose  parameters. The silhouette of  the hand and
object rendered on  camera $c$ is denoted by $\text{RS}_c(\text{p}_\text{h},\text{p}_\text{o})$  and the silhouette
discrepancy is defined as,
\begin{align}
E_\text{mask}(\text{I}_c, \text{p}_\text{h}, \text{p}_\text{o}) = \lVert \text{RS}_c(\text{p}_\text{h}, \text{p}_\text{o}) - S(\text{I}_c) \rVert^2 \> .
\end{align}

\paragraph{Depth residual term $E_\text{dpt}$. }
The depth residual term is similar to the segmentation discrepancy term:
\begin{equation}
E_\text{dpt}(\text{D}_c, \text{p}_\text{h}, \text{p}_\text{o}) = \text{Tukey}(\lVert \text{RD}_c(\text{p}_\text{h}, \text{p}_\text{o}) - \text{D}_c \rVert) \> ,
\end{equation}
where $\text{RD}_c(\text{p}_\text{h}, \text{p}_\text{o})$ is the depth rendering of the hand and
the object under their current estimated poses $\text{p}_\text{h}$ and $\text{p}_\text{o}$.  The
Tukey function is a robust estimator that  is similar to the $\ell_2$ loss close
to  0, and  constant after  a threshold.   It is  useful to  be robust  to small
deviations in the scale  and shape of the hand and object  models and also noise
in the  captured depth maps.   $E_\text{dpt}$ is differentiable as we  employ a
differentiable renderer~\cite{Henderson2019} for rendering the depth maps.

\vspace{-0.3cm}
\paragraph{2D Joint error term $E_\text{j2D}$. }


The 2D joint error term is defined as,
\begin{equation}
E_\text{j2D}(\text{I}_c, \text{p}_\text{h}) = \sum_{i=1}^{21} \text{h}[i] \Big\lVert
\text{proj}_c(\text{J}_{\text{p}_\text{h}}[i]) - \text{K}_c[i] \Big\rVert^2 \> ,
\label{eq:j2dErr}
\end{equation}
where $\text{J}_{\text{p}_\text{h}}[i]$ denotes the $i^\text{th}$ 3D hand joint location under pose $\text{p}_\text{h}$, the $\text{proj}_c(\cdot)$ operator projects it onto camera $c$, $\text{K}_c[i]$ is its predicted 2D location, and $\text{h}[i]$ its confidence.  The 21 hand joints in $E_\text{j2D}(\cdot)$ consist of 15 finger joints, 5 finger tips, and the wrist joint.

In practice,  we take  the $\text{K}_c[i]$ as  the locations of  the maximum  values of
heatmaps, and  the $\text{h}[i]$  as the  maximum values  themselves. To  predict these
heatmaps, we trained a CNN based  on the architecture of \cite{Wei16}.  Training
data  come   from  an  initial   dataset~\cite{ho3d_old}  we  created   using  a
semi-automatic method. This  dataset is made of 15,000 frames  from 15 sequences
in a single camera setup. We manually initialized the grasp pose and object pose
for the first  frame of each sequence.  The manipulators  were asked to maintain
their grasp poses as rigid as possible to make the registration easier.  We then
ran the optimization  stages for the single camera case  described in Section~\ref{sec:singleCameraSetup}.  After
optimization, we  augmented the  resulting dataset by  scaling and  rotating the
images,     and      adding     images     from     the      Panoptic     Studio
dataset~\cite{xiang2019monocular}, which contain 3D annotations for hands.


\vspace{-0.3cm}
\paragraph{3D error term $E_\text{3D}$. }
This term  is not  absolutely necessary as  the depth  information from  all the  cameras is
already exploited by $E_\text{dpt}$, however it accelerates the convergence by guiding
the optimization towards the  minimum even from far away.  We  build a point cloud
$\text{P}$ by  merging the depth maps  from the RGB-D cameras  after transforming
them  to   a  common  reference  frame.    More  details  on  the   point  cloud
reconstruction can be found in the supplementary material.
  
We segment  $\text{P}$ into  an object  point cloud $\text{P}_\oo$  and a  hand point
cloud $\text{P}_\hh$ using  the segmentation mask $S_c(\text{I})$ in  each camera image.
At  each iteration  of the  optimization, for  each point  $\text{P}_\oo[j]$ of  the
object point cloud, we look for the closest vertex $\text{V}_\text{o}[j^*]$ on the object
mesh, and for each point $\text{P}_\hh[k]$ of  the hand point cloud, we look for the
closest vertex  $\text{V}_\text{h}[k^*]$ on  the hand mesh.   $E_\text{3D}(\text{P}, \text{p}_\text{h},
\text{p}_\text{o})$ is then defined as,
\begin{align}
\sum_j &\big\lVert \text{P}_\oo[j] - \text{V}_\text{o}[j^*]\big\rVert^2 + \sum_k \big\lVert \text{P}_\hh[k] - \text{V}_\text{h}[k^*]\big\rVert^2 \> .
\end{align}









\vspace{-0.5cm}
\paragraph{Joint angle constraint $E_\text{joint}$. }
This  term imposes  restrictions on  the 15  joints of  the hand  to ensure  the
resulting  pose is  natural.   The  three-dimensional  rotation  of  a joint  is
parameterized using the axis-angle representation in MANO model, resulting in 45
joint angle  parameters.  A common  solution when using MANO model is to optimize in
the PCA space of 3D joint angles with an $\ell_2$ regularizer~\cite{zimmermann2019freihand, boukhayma20193d} for pose coefficients. However, we  observed  in practice  that
optimizing Eq.~\ref{eq:global} in PCA space had less expressibility: some of the complex grasp poses in our dataset could not be accurately expressed in the PCA space, which was constructed with relatively simpler grasp poses and free-hand gestures.
  Instead, we optimize on joint angles directly and derive our
  own limits  for each  of the 45  joint parameters~(please  refer to supplementary
  material for these limits).  As in \cite{Zhou16c}, the joint angle constraint
term $E_\text{joint}(\text{p}_\text{h}^t)$ is given by,
\begin{equation}
\sum_{i=1}^{45} \max(\underline{a_i} - a[i], 0) + \max(a[i] -
\overline{a_i}, 0) \> ,
\label{eq:ejoint}
\end{equation}
where $a[i]$  denotes the $i^{th}$ joint  angle parameter for pose  $\text{p}_\text{h}$, and
$\underline{a_i}$ and $\overline{a_i}$ correspond to its lower and upper limits.


\begin{figure*}[ht]
  \begin{center}
    \includegraphics[width=0.85\linewidth, trim=.2cm 3.5cm .0cm 2cm,clip]{figures/singleview_pipeline}
  \end{center}
\vspace{-4mm}
\caption{The different stages of the multi-camera and single camera setups. See Section \ref{sec:fullOpt} for more details.}
\label{fig:methodOverview}
\vspace{-10px}
\end{figure*}	



\vspace{-0.3cm}
\paragraph{Physical plausibility term $E_\text{phy}$. }
During optimization, the  hand model might penetrate the object  model, which is
physically not possible.  To avoid this, we add a repulsion term that pushes the
object and  the hand  apart if  they interpenetrate each  other.  For  each hand
vertex $V_\text{h}[m]$, the amount of penetration $\Gamma[m]$ is taken as, 
\begin{equation}
\Gamma[m] = \max(-\mathbf{n}_\text{o}\big(\text{V}_\text{o}[m^*]\big)^T\big(\text{V}_\text{h}[m] -\text{V}_\text{o}[m^*]\big), 0) \> , 
\end{equation}
where  $\text{V}_\text{o}[m^*]$  is   the  vertex  on  object  closest   to  hand  vertex
$\text{V}_\text{h}[m]$, and the $\mathbf{n}_\text{o}(\cdot)$ operator  provides the normal vector for
a vertex.   In words, the amount  of penetration is estimated  by projecting the
vector  joining a  hand vertex  and its  nearest object  vertex onto  the normal
vector at  the object vertex location.   The physical plausibility term  is then
defined as,
\begin{equation}
E_\text{phy}(\text{p}^t_\text{h},\text{p}^t_\text{o}) = \sum_m \exp{\big(w~\Gamma[m]\big)} \> .
\end{equation}
We use  $w = 5$ in  practice, and only a subsampled  set of vertices of  the hand to
compute $E_\text{phy}$ efficiently.



\paragraph{Temporal consistency term $E_\text{tc}$.}

The previous  terms are all applied  to each frame independently.   The temporal
consistency term $E_\text{tc}$  allows us to constrain together the  poses for all the
frames. We apply a 0-th and 1-st order  motion model on both the hand and object
poses:
\begin{equation}
\begin{array}{l}
E_\text{tc}(\text{p}_\text{h}^t,\text{p}_\text{o}^t,\text{p}_\text{h}^{t-1},\text{p}_\text{o}^{t-1},\text{p}_\text{h}^{t-2},\text{p}_\text{o}^{t-2}) = \\[0.5em] \nonumber 
\lVert \Delta_h^t \rVert^2 + \lVert \Delta_o^t \rVert^2 + 
\lVert \Delta_h^t - \Delta_h^{t-1} \rVert^2 + \lVert \Delta_o^{t} -
\Delta_o^{t-1} \rVert^2 \> ,
\end{array}
\end{equation}
where $\Delta_h^t = \text{p}_\text{h}^t - \text{p}_\text{h}^{t-1}$ and $\Delta_o^t = \text{p}_\text{o}^t
- \text{p}_\text{o}^{t-1}$.  Since we  optimize a sum of these terms  over the sequence,
this effectively constrains all the poses together.







\section{Introduction}

\input{intro.tex}


\section{Related Work}

\input{rel_work.tex}


\section{3D Annotation Method}
\label{sec:method}

\input{method.tex}


\section{Optimization}
\label{sec:fullOpt}

\input{optimization.tex}


\section{Monocular RGB based 3D Hand Pose
\label{sec:handposeSingleFrameSec}

\input{method2.tex}




\section{Benchmarking \datasetname}

\input{eval.tex}


\section{Conclusion}

We introduced a fully automatic method to annotate images of a hand manipulating
an  object with  their  3D poses,  even under  large  occlusions, by  exploiting
temporal  consistency. We also introduced the first  markerless  dataset of  color images  for
benchmarking 3D  hand+object pose estimation.  To demonstrate the  usefulness of
our dataset, we  proposed a method  for predicting the 3D pose  of the hand
from  a single  color  image. Another  future  application is  the  joint estimation  of
hand+object poses from a single RGB frame.

The lack  of high quality  segmentations (we had to  use a synthetic  dataset as
explained in  Section~\ref{sec:method}) of the hand  sometimes affects accuracy.
Improving  these segmentations  and/or introducing  a attraction  term and  physics
constraints as  in~\cite{RealtimeHO_ECCV2016, Tzionas2016} would  further
improve our annotations.


\section{Acknowledgement}
This work  was supported  by the  Christian Doppler  Laboratory for  Semantic 3D
Computer Vision, funded in part by Qualcomm Inc.

\title{HOnnotate: A method for 3D Annotation of Hand and Object Poses \\ Supplementary Material}
\maketitle
\setcounter{section}{0}
\input{supplementarySec.tex}

{\small
\bibliographystyle{ieee_fullname}

\subsection{2D Joint Detection}


\subsection{Multi-Camera Setup}
\label{sec:multiCameraSetup}

\paragraph{Initialization.}
In the multi-camera setup, we obtain a first estimate $\tilde{\text{p}}_\text{h:w}^0$ for the
hand pose in the first frame $(t=0)$ as,
\begin{equation}
\label{eq:multiViewInit}
\tilde{\text{p}}_\text{h:w}^0   =  \arg\min\limits_{\text{p}_\text{h:w}}   \sum_{c=1}^{N_C}
E_\text{j2D}(\text{I}_c^0, \text{p}_\text{h:w}) + \nu E_\text{joint}(\text{p}_\text{h:w}) \> .
\end{equation}
We use the Dogleg  optimizer~\cite{ConnGoulToin00} to perform this optimization.
A  first estimate  $\tilde{\text{p}}_\text{o}^0$  for  the object  pose  in  this frame  is
obtained using  \cite{Rad17} trained by  synthetically over-laying hands  on YCB
objects as explained in Section~\ref{sec:cost}.

\paragraph{Single-frame joint pose optimization.}
We then obtain  estimates $\tilde{\text{p}}_\text{h:w}^t$ and $\tilde{\text{p}}_\text{o}^t$  for all the
other  frames ($t = 1..N_F$)  by   tracking.   We   minimize  $\big(
E_\mathcal{D}(\text{p}_\text{h:w}^t,  \text{p}_\text{o}^t)  + E_\mathcal{C}(\text{p}_\text{h:w}^t,  \text{p}_\text{o}^t)  \big)$
w.r.t.   $\text{p}_\text{h:w}^t$  and   $\text{p}_\text{o}^t$,  using  $\tilde{\text{p}}_\text{h:w}^{t-1}$  and
$\tilde{\text{p}}_\text{o}^{t-1}$ for initialization.  


\paragraph{Multi-frame joint pose optimization.}
We finally  perform a  full optimization of  Eq.~\eqref{eq:global} w.r.t. $\text{p}_\text{h}^t$ and 
$\text{p}_\text{o}^t$ for $t=0..N_F$ over  all the
frames simultaneously using  estimates $\tilde{\text{p}}_\text{h:w}^t$ and $\tilde{\text{p}}_\text{o}^t$
for    initialization.     Due    to    memory    constraints,    we    optimize
Eq.~\eqref{eq:global}  in  batches instead  of  considering  all the  frames  in
sequence.   We use  a  batch size  of 20  frames  with $\alpha=20$,  $\beta=20$,
$\gamma=5\times10^{-5}$,    $\delta=50$,    $\epsilon=100$,   $\zeta=50$,    and
$\eta=100$,  and  the  Adam  optimizer  with  learning  rate  of  0.01  for  100
iterations.


\subsection{Single-Camera Setup}
\label{sec:singleCameraSetup}

\paragraph{Initialization.}
In the single camera  setup, as we assume that the  grasp pose varies marginally
across the sequence,  we initially make the assumption that  it remains constant
throughout the sequence.  In order to account for the minor changes which occur
in practice, we relax this assumption  in the latter stages of the optimization.
We thus obtain initial estimates $\tilde{\text{p}}_\text{h:w}^t$ for the hand poses as,
\begin{align}
\label{eq:singleViewPoseEst}
\{ \tilde{\text{p}}_\text{h:w}^t\}_t = \arg\min_{\{\text{p}_\text{h:w}^t\}_t} \sum_t E_\text{j2D}(\text{I}^t, \text{p}_\text{h:w}^t) + \nu E_\text{joint}(\text{p}_\text{h:w}^t) \> ,
\end{align}
where the  joint angle parameters  are constrained to be  the same over  all the
frames, and only the rotation and translation parameters for the wrist joint can
be different.   In practice,  we perform  this optimization  only over  a random
subset $\Omega$ of the frames to save time.  We set $\nu = 50$, size of $\Omega$
to  20  and use  the  Dogleg  optimizer~\cite{ConnGoulToin00}.  First  estimates
$\tilde{\text{p}}_\text{o}^t$ for  the object  poses in  $\Omega$ are  obtained as  for the
multi-camera setup.

From the $\tilde{\text{p}}_\text{h:w}^t$ and the $\tilde{\text{p}}_\text{o}^t$, we can compute the grasp
pose in the  \emph{object coordinate system} $\tilde{\text{p}}_\text{h:o}^t$,  which is assumed
to be constant at this stage. The initial estimate of the constant grasp pose $\tilde{\text{p}}_\text{h:o}$  is taken as the average of $\{\tilde{\text{p}}_\text{h:o}^t\}_{t\in\Omega}$





\vspace{-0.5px}
\paragraph{Grasp pose estimation.}
We obtain  a better estimate of  the grasp pose $\hat{\text{p}}_\text{h:o}$  and object poses
$\hat{\text{p}}_\text{o}^t$ under fixed grasp pose assumption as,
\begin{equation}
\begin{array}{r}
\hat{\text{p}}_\text{h:o}, \{\hat{\text{p}}_\text{o}^t\}_{t \in \Omega} = \arg\min_{\text{p}_\text{h:o},\{\text{p}_\text{o}^t\}_{t \in \Omega}} \sum_{t \in \Omega} E_\mathcal{D}(f_{\text{p}_\text{o}^t}(\text{p}_\text{h:o}), \text{p}_\text{o}^t) + \\[0.5em]
\zeta E_\text{phy}(f_{\text{p}_\text{o}^t}(\text{p}_\text{h:o}), \text{p}_\text{o}^t) + \epsilon E_\text{joint}(\text{p}_\text{h:o}), \\ [0.5em]
\end{array}
\end{equation}
wrt $\text{p}_\text{h:o}$ and $\text{p}_\text{o}^t$ over the frames  in $\Omega$, using $\tilde{\text{p}}_\text{h:o}$  and $\tilde{\text{p}}_\text{o}^t$
for  initialization. $f_{\text{p}_\text{o}^t}(\cdot)$ converts the grasp pose to the hand pose in the world coordinate system given object pose $\text{p}_\text{o}^t$.  This  optimization  accounts for  the  mutual  occlusions between hand and  object.
\vspace{-0.4cm}
\paragraph{Object pose estimation.}
Having a good estimate $\hat{\text{p}}_\text{h:o}$ for  the grasp pose, we obtain object pose
estimates    $\hat{\text{p}}_\text{o}^t$    for    all    the    frames    by    minimizing
$E_\mathcal{D}\big(f_{\text{p}_\text{o}^t}(\hat{\text{p}}_\text{h:o}),\text{p}_\text{o}^t\big)$ wrt $\text{p}_\text{o}^t$   over  each   frame  independently. We  use
$\hat{\text{p}}_\text{o}^{t-1}$  to  initialize  the  optimization  over  $\text{p}_\text{o}^t$ at frame $t$,
except for $\hat{\text{p}}_\text{o}^0$ where $\tilde{\text{p}}_\text{o}^0$  is used.  Note that the hand
pose is not optimized in this stage.

\vspace{-0.3cm}
\paragraph{Multi-frame joint hand+object pose refinement.}
In this  final stage, we  allow variations in the  grasp pose across  frames and
introduce temporal constraints.  We thus optimize Eq.~\eqref{eq:global} w.r.t. $\{(\text{p}_\text{h}^t,
\text{p}_\text{o}^t)\}^{N_F}_{t=1}$ over  all the frames  simultaneously, using  pose parameters
  $\hat{\text{p}}_\text{o}^t$, $\hat{\text{p}}_\text{h}^t=f_{\hat{\text{p}}_\text{o}^t}(\hat{\text{p}}_\text{h:o})$ estimated  in  the  previous   stages  as initialization for $\text{p}_\text{o}^t$ and $\text{p}_\text{h}^t$.

\subsection{3D Object Pose Estimation}


Estimating the  3D pose of  an object from  a single frame  is still one  of the
fundamental problems of Computer Vision. Some  methods are now robust to partial
occlusions~\cite{Oberweger2018,peng2019pvnet,hu2019segpose}, but many works rely on RGB-D data to
handle this problem~\cite{Buch2017,Mitash2017,Zhang2017,Kehl16},  by fitting the
3D object model to depth data. This  can fail when a hand grasps the object,
since the surface of the hand can be mistaken for the surface of the object.



\subsection{3D Hand Pose Estimation}

Single image hand pose estimation is also a very popular problem in Computer Vision, and approaches can be divided into discriminative and generative methods. Discriminative approaches directly predict the joint locations from RGB or RGB-D images.  Recent works based on Deep Networks \cite{Tompson14b,Xu16,Neverova17,Oberweger17,mueller2017real,Zimmermann2017,ge2018hand} show remarkable performance, compared to early works based on Random Forests such as \cite{keskin2012hand}.  However, discriminative methods perform poorly in case of partial occlusion.

Generative approaches take advantage of a hand model and its kinematic structure to generate hand pose hypotheses that are physically plausible~\cite{sridhar2013interactive,Qian2014realtime,sharp2015accurate,Tzionas2014capturing,melax2013dynamics,de2011model,ye2016spatial}.  \cite{Mueller2018,Panteleris2018} predict 2D joint locations and then lift them to 3D.  Generative approaches are usually accurate and can be made robust to partial occlusions.  They typically rely on some pose prior, which may require manual initialization or result in drift when tracking.


Our work is  related to both discriminative and generative  approaches: we use a
generative approach within a global  optimization framework to generate the pose
annotations,  and  use  a  discriminative  method  to  initialize  this  complex
optimization.  \cite{Wan_2019_CVPR}  also combines generative  and discriminative
approaches to train  a network in a self-supervised setting.   However, they only
consider hands.   We also train  a discriminative  method using our  dataset, to
predict the hand poses which are robust to occlusions from interacting objects.



\subsection{Synthetic Images for 3D Pose Estimation}

Being able to train discriminative methods on synthetic data is valuable as it is difficult to acquire annotations for real images~\cite{Zimmermann2017}.  \cite{hasson2019learning,Rad2018b} show that because of the domain gap between synthetic and real images, training on synthetic images only results in sub-optimal performance.  A GAN method was used in \cite{Mueller2018} to make synthetic images of hands more realistic.  While using synthetic images remains appealing for many problems, creating the virtual scenes can be expensive and time-consuming.  Generating animated realistic hand grasps of various objects, as it would be required to solve the problem considered in this paper remains challenging. Being able to use real sequences for training has thus also its advantages. Moreover, evaluation should be performed on real images.


\subsection{Joint Hand+Object Pose Estimation}

Early         approaches        for         joint        hand+object         pose
estimation~\cite{Oikonomidis2011full,Wang11d,Ballan2012motion}  typically relied
on  multi-view camera  setups, and  frame-by-frame tracking  methods, which  may
require      careful      initialization      and     drift      over      time.
\cite{panteleris20153d,Tzionas_2015_ICCV}  propose generative  methods to  track
finger   contact    points   for   in-hand   RGB-D    object   shape   scanning.
\cite{pham2015capturing,Pham_2015_CVPR} consider sensing from vision to estimate
contact forces during hand+object interactions  using a single RGB-D camera, and
then estimate the hand and the  object pose.  However, these methods are limited
to small occlusions.

\cite{kyriazis2013physically,Tzionas2016} propose to use a physics simulator and
a  3D renderer  for  frame-to-frame tracking  of hands  and  objects from  RGB-D.
\cite{Kyriazis14} uses  an ensemble  of Collaborative Trackers  for multi-object
and multiple  hand tracking from  RGB-D images.   The accuracy of  these methods
seems to  be qualitatively  high, but  as the ground  truth acquisition in
a real-world is known to be hard, they evaluate the proposed method on
synthetic datasets, or by measuring the  standard deviation of the difference in
hand/object poses during a grasping scenario.

\cite{Tsoli_2018_ECCV} considers the problem of  tracking a deformable object in
interaction with a hand, by optimizing  an energy function on the appearance and
the kinematics  of the hand,  together with hand+object  contact configurations.
However, it is  evaluated quantitatively only on synthetic  images, which points
to the difficulty  of evaluation on real data.  In  addition, they only consider
scenarios where the  hand is visible from  a top view, restricting  the range of
the hand poses and not allowing occlusions.


Very recently,  \cite{kokic2019learning} uses a  coarse hand pose  estimation to
retrieve the 3D pose and shape of hand-held objects. However, they only consider
a   specific  type   of   object  and   do  not   estimate   the  object   pose.
\cite{hasson2019learning}  presents a  model  with contact  loss that  considers
physically feasible hand+object interaction  to improve grasp quality.  However,
to  estimate 3D  hand pose,  they  predict PCA  components for  the pose,  which
results  in  lower   accuracy  compared  to  ours,   as  our  experiments
show. \cite{tekin2019h}  proposes a deep model  to jointly predict 3D  hand and
object poses from egocentric views, but the absence of physical constraints might
result in infeasible grasps.


\subsection{Hand+Object Datasets}

Several datasets for  hand+object interactions have already  been proposed. Many
works    provide    egocentric   RGB    or    RGB-D    sequences   for    action
recognition~\cite{bullock2015yale,cai2015scalable,fathi2011learning,bambach2015lending,rogez2015understanding}\comment{,luo2017scene,Tzionas_2015_ICCV}.
However,
they  focus  on   grasp  and  action  labels  and  do   not  provide  3D  poses.
\cite{choi2017robust,rogez20143d,mueller2017real,Tsoli_2018_ECCV}  
generate synthetic datasets with 3D hand pose  annotations, but fine interaction between a
hand and an object remains difficult to generate accurately.


\cite{Tzionas2016,Tzionas2014capturing}  captured sequences  in  the context  of
hand+hand  and   hand+object  interaction,   with  2D  hand   annotations  only.
\cite{Myanganbayar18} collected  a dataset of  real RGB images of  hands holding
objects.  They  also provide 2D joint  annotations of pairs of  non-occluded and
occluded hands,  by removing  the object  from the grasp  of the  subject, while
maintaining  their hand  in  the same  pose.   \cite{goudie20173d} proposes  two
datasets, a hand+object  segmentation dataset, and a  hand+object pose estimation
dataset. However,  for both  datasets, the  background pixels  have been  set to
zero, and the training  images only consist of a hand  interacting with a tennis
ball.   They provide  hand  pose  annotations and  object  positions, by  manually
labelling the joints and using a generative method to refine the joint positions.
\cite{JooSS18} generate a large scale dataset with full body pose and hand pose annotations in a multi-view setup. They use a generative approach to fit the body and hand models to 3D keypoints and point cloud. However, their dataset focuses on total body pose annotation and not hand+object interactions exclusively and does not provide object pose annotations.


\cite{RealtimeHO_ECCV2016} proposed  an RGB-D dataset  of a hand  manipulating a
cube, which  contains manual ground  truth for  both fingertip positions  and 3D
poses of the  cube.  \cite{pham2018hand} collected a dataset  where they measure
motion and force under different  object-grasp configurations using sensors, but
do   not   provide  3D   poses.    In   contrast   to  these   previous   works,
\cite{garcia2018first} provides a dataset of hand and object interactions with 3D annotations
for both hand joints and object pose.  They used a motion capture system made of
magnetic sensors  attached to  the user's hand and  to the  object in  order to
obtain hand 3D pose annotations in  RGB-D video sequences.  However, this changes
the appearance of the hand in the color images as the sensors and the tape attaching
them are visible.

Very recently,  \cite{hasson2019learning} introduced  ObMan, a large  dataset of
images of  hands grasping  objects. The images in ObMan dataset are synthetic  and the
grasps  are generated  using an  algorithm from  robotics.  Even  more recently,
\cite{zimmermann2019freihand} proposed a multi-view RGB dataset, FreiHAND, which
includes hand-object interactions.  However, the annotations are  limited to the
3D poses and shapes of  the hand.  Further, \cite{zimmermann2019freihand} uses a
human-in-the-loop method  to obtain annotations  from multiple RGB cameras  in a
green-screen  background environment.  Our method,  on  the other  hand is  fully
automatic, capable of working even on a single RGBD-camera setup and does not make
any assumption  on the background.  The objects in  our dataset are  also larger
than those  in FreiHAND, thus  resulting in a  more challenging scenario  as the
occlusions are larger. The annotation  accuracy of our method is comparable
to \cite{zimmermann2019freihand} as described in Section~\ref{sec:evalAnno}.

As illustrated  in Fig.~\ref{fig:one} and  Table~\ref{tab:hand_obj_dataset}, our
\datasetname  dataset is  the first  markerless dataset  providing both  3D hand
joints and 3D  object pose annotations for  real images, while the  hand and the
object are heavily occluded by each other.



\begin{table}
  \begin{center}
   
    \resizebox{.47\textwidth}{!}{%
      \begin{tabular}{lccccccc}
	\toprule
	Dataset & \specialcell{No. of\\Frames} & \specialcell{3D Object\\Pose} & \specialcell{Marker-\\less} & \specialcell{Real\\Images}  & Labels & \specialcell{No. of\\Objects} & \specialcell{No. of\\Subjects}\\
	\midrule
	PAN~\cite{JooSS18} & 675K  & - & + & + & automatic & - & 70\\
	GAN~\cite{Mueller2018} & 300K & - & + & - & synthetic & - & -\\
	FPHA~\cite{garcia2018first} & 100K  & + (23K frames) & - & + & automatic & 26 (4 models) & 6\\
	ObMan~\cite{hasson2019learning} & 150K  & + & + & -  & synthetic & 2.7K & 20\\
	Freihand~\cite{zimmermann2019freihand} & 37K  & - & + & + &  hybrid & 27 & 35\\
	\textbf{HO-3D (ours)} & 78K  & + & + & + &  automatic & 10 & 10\\
	\bottomrule
    \end{tabular}}
    \vspace{-2mm}
    \caption{ Comparison of hand+object datasets.}
    \vspace{-9mm}
    \label{tab:hand_obj_dataset}
  \end{center}
\end{table}


\section{Hand Pose Estimation from Single Color Image}

Fig.~\ref{fig:net_arch} shows the architecture of our hand pose estimator from a
single frame. Given an image of the hand centered in the image window, we first
extract features using the convolutional layers of VGG~\cite{Simonyan14c},  and
then similar to \cite{Cao2017realtime} using a multi-stage CNN, we predict
heatmaps for the 2D hand joint locations and finally joint direction vectors with respect to wrist joint. The hand detection can be done using segmentation as described in Section~\ref{sec:hand_obj_seg}.


\begin{figure*}
	\begin{center}
		\includegraphics[width=0.95\linewidth]{figures/hand_pose_arch.pdf}
	\end{center}
	\vspace{-2mm}
	\caption{Architecture of our hand pose estimator from a single color
          image. Given an input image of hand centered in the image, we extract
          the features using the convolutional layers of VGG~\cite{Simonyan14c}
          (\texttt{Conv1\_1} to \texttt{Conv4\_4}). Similarly
          to~\cite{Cao2017realtime}, we then predict heatmaps for the joint
          locations in multi-stages. The architecture for the different stages are all the same.  \texttt{C} denotes a convolutional layer with the number of filters and the filter size inscribed; \texttt{FC}, a fully-connected layer with the number of neurons; \texttt{P} and \texttt{AP} denote max-pooling and average pooling with their sizes, respectively.}
	\label{fig:net_arch}
\end{figure*}

\section{Hand Pose Estimation for Hand Interaction with Unseen Objects}
Knowing the objects in advance can help to improve the performances of the
estimated 3D hand pose while hand interacts with objects, however, in practice,
the hand can manipulate any arbitrary objects. We have tested our hand pose
estimator trained on our annotations, and tested on sequences where a hand is
manipulating objects not present in the annotated images. As shown in Fig.~\ref{fig:unseen_objects}, our pose estimator performs well on these sequences.


\section{Hand Shape Estimation}
The MANO hand shape parameters $\beta \in \mathbb{R}^{10}$ were estimated for each human manipulator in our \datasetname dataset. The shape parameters are estimated from a sequence $\Phi$ of hand only poses using a method similar to \cite{Tan_2016_CVPR} in two steps. More exactly, the pose of hand $p_h^t$ in the sequence is first estimated for each frame $t$ using a mean pose $\beta_{mean}$ as $\hat{p}_h^{t} = \argmin_{p_h} E_{H}(p_h, \beta_{mean})$, where, 
\begin{align}
E_{H}(p_h, \beta_{mean}) = &E_\mathcal{D}(p_h, \beta_{mean}) + \epsilon E_\text{joint}(p_h) + \\ \nonumber 
&\eta E_\text{tc}(p_h, p_h^{t-1}, p_h^{t-2}).
\end{align}
$E_\mathcal{D}(p_h, \beta_{mean})$ represents the data term defined in Eq. 2 of the paper where hand is rendered with pose parameters $p_h$ and shape parameters $\beta_{mean}$. $E_\text{joint}$ and $E_\text{tc}$ are explained in Section 3.2 of the paper. At each frame, the pose parameters are initialized with $p_h^{t-1}$. 
The personalized hand shape parameters are then obtained as,
\begin{equation}
\beta^* = \argmin_{\beta} \sum_{t\in \Phi} \min_{p_h^t} E_{H}(p_h^t, \beta),
\end{equation}
where the pose parameters are initialized with the values obtain in the first step ($\hat{p}_h^{t}$).

\section{Joint Angle Constraints}
The maximum and minimum limits on the joint angle parameters used in Eq.~(8) of the paper are provided in  Table~\ref{tab:table1}.
\begin{table}
	\begin{center}
		\resizebox{0.47\textwidth}{!}{%
			\begin{tabular}{lccccc}
				\toprule
				Joint & Index & Middle & Pinky  & Ring & Thumb\\
				\midrule
				MCP & \begin{tabular}{@{}c@{}}
					(0.00, 0.45)\\
					(-0.15, 0.20)\\
					(0.10, 1.80)\\
				\end{tabular} & \begin{tabular}{@{}c@{}}
					(0.00, 0.00)\\
					(-0.15, 0.15)\\
					(0.10, 2.00)\\
				\end{tabular} & \begin{tabular}{@{}c@{}}
					(-1.50, -0.20)\\
					(-0.15, 0.60)\\
					(-0.10, 1.60)\\
				\end{tabular} & \begin{tabular}{@{}c@{}}
					(-0.50, -0.40)\\
					(-0.25, 0.10)\\
					(0.10, 1.80)\\
				\end{tabular} & \begin{tabular}{@{}c@{}}
					(0.00, 2.00)\\
					(-0.83, 0.66)\\
					(0.00, 0.50)\\
				\end{tabular} \\
				\hline
			
				PIP & \begin{tabular}{@{}c@{}}
					(-0.30, 0.20)\\
					(0.00, 0.00)\\
					(0.00, 0.20)\\
				\end{tabular} & \begin{tabular}{@{}c@{}}
					(-0.50, -0.20)\\
					(0.00, 0.00)\\
					(0.00, 2.00)\\
				\end{tabular} & \begin{tabular}{@{}c@{}}
					(0.00, 0.00)\\
					(-0.50, 0.60)\\
					(0.00, 2.00)\\
				\end{tabular} & \begin{tabular}{@{}c@{}}
					(-0.40, -0.20)\\
					(0.00, 0.00)\\
					(0.00, 2.00)\\
				\end{tabular} & \begin{tabular}{@{}c@{}}
					(-0.15, 1.60)\\
					(0.00, 0.00)\\
					(0.00, 0.50)\\
				\end{tabular} \\
				\hline	
				DIP & \begin{tabular}{@{}c@{}}
					(0.00, 0.00)\\
					(0.00, 0.00)\\
					(0.00, 1.25)\\
				\end{tabular} & \begin{tabular}{@{}c@{}}
					(0.00, 0.00)\\
					(0.00, 0.00)\\
					(0.00, 1.25)\\
				\end{tabular} & \begin{tabular}{@{}c@{}}
					(0.00, 0.00)\\
					(0.00, 0.00)\\
					(0.00, 1.25)\\
				\end{tabular} & \begin{tabular}{@{}c@{}}
					(0.00, 0.00)\\
					(0.00, 0.00)\\
					(0.00, 1.25)\\
				\end{tabular} & \begin{tabular}{@{}c@{}}
					(0.00, 0.00)\\
					(-0.50, 0.00)\\
					(-1.57, 1.08)\\
				\end{tabular} \\
				\bottomrule
		\end{tabular}}
		\caption{Empirically derived minimum and maximum values for the joint angle parameters used in our implementation.}
		\label{tab:table1}
	\end{center}
\end{table}


\section{Point Cloud from Multiple Cameras}
The $E_\text{3D}$ term in Section~3.2 of the paper uses the combined point cloud $P$ from all the RGB-D cameras. Let $P_c$ denote the point cloud corresponding to camera $c$ and $M_{c_1,c_2}$ denote the relative pose between two cameras $c_1$ and $c_2$. The consolidated point cloud $P$ is then obtained as,
\begin{equation}
P = [P_0,~M_{c_1,c_0}\cdot P_1,~M_{c_2,c_0}\cdot P_2, ... ,~M_{c_{N},c_0}\cdot
  P_{N}] \> ,
\end{equation}
where $[\cdot, \cdot]$ represents concatenation of point clouds.


\section{Hand-Object Segmentation Network}
\label{sec:hand_obj_seg}
The segmentation maps for the hand and object are obtained from a DeepLabV3~\cite{deeplabv2017} network trained on synthetic images of hand and objects. The synthetic images are obtained by
over-laying and under-laying images of hands on images of objects at random locations and scales. We use the object masks provided by~\cite{ycbonline}.
The segmented hands were obtained using an RGB-D camera by applying simple depth
thresholding. We also use additional synthetic hand images from the RHD
dataset~\cite{Zimmermann2017}. A few example images from the training data are
shown in Fig.~\ref{fig:fig12}. We use 100K training images with
augmentations. Fig.~\ref{fig:samples_seg} shows segmentation of hand and object
using the trained DeepLabV3 network.

\begin{figure}
	\begin{center}
		\begin{tabular}{ccc}
			\includegraphics[width=0.32\linewidth]{figures/handObjSeg1} & \hspace{-4mm}
			\includegraphics[width=0.32\linewidth]{figures/handObjSeg3} & \hspace{-4mm}
			\includegraphics[width=0.32\linewidth]{figures/handObjSeg4}
		\end{tabular}
	\end{center}
	\vspace{-2mm}
\caption{Synthetic training images used for training the hand-object segmentation network.}
\label{fig:fig12}
\end{figure}

\begin{figure}
	\begin{center}
	\begin{tabular}{cc}
		\includegraphics[width=0.47\linewidth]{figures/Shreyas_Mustard_1_color_0018} &  \hspace{-4mm}
		\includegraphics[width=0.47\linewidth]{figures/Shreyas_Mustard_1_0018_prediction_graybg} \\
		\end{tabular}
	\end{center}
	\vspace{-2mm}
	\caption{Example  of   hand  and   object  segmentation   obtained  with
		DeepLabV3.  Left: input image; Right: hand (green) and object (purple) 
		segmentation.}
	\vspace{-5mm}
	\label{fig:samples_seg}
\end{figure}








\section{Automatic Initialization}
\begin{figure}[ht]
	\begin{center}
		\includegraphics[width=0.8\linewidth]{figures/OS_Ours_compare}
	\end{center}
	\caption{Accuracy of keypoint prediction, described in~Section 3.2 of
          the paper when trained with PAN~\cite{JooSS18} dataset alone and PAN +
          our annotations. The accuracy is measured in percentage of correct 2D
          keypoints given a threshold. Only 15,000 images from our \datasetname dataset are used in training. Due to the presence of object occlusions, a network trained on hands-only dataset is less accurate in predicting keypoints when compared with a network trained with hand+object data.}
	\label{fig:pckCurve}
\end{figure}

As explained in Section~4.1 of the paper, a keypoint prediction network based on convolutional pose machine~\cite{Wei16} is used to obtain initialization for hand poses. Such a network is trained with our initial hand+object dataset of 15,000 images together with images from hand-only PAN~\cite{JooSS18} dataset. Fig. \ref{fig:pckCurve} compares the accuracy of network in predicting keypoints in hand-object interaction scenarios when trained with hands-only dataset and hands+object dataset. Our initial \datasetname dataset helps in obtaining a more accurate network for predicting keypoints and hence results in better initialization.



\section{Dataset Details}
We annotated 77,558 frames of 68 sequences hand-object interaction of 10 persons with different hand shape. On average there are 1200 frames per sequences. 16 sequences are captured and annotated in a single camera, and 52 sequences for the multi-camera setup. 

\paragraph{Hand+Object.}
The participants are asked to perform actions with objects. 
The grasp poses vary between frames in a sequence in the multi-camera setup and remain almost rigid in the single camera setup.

\paragraph{Participants.}
The participants are between 20 and 40 years old, 7 of them are males and 3 are
females. In total, 10 hand shapes are considered for the annotations.

\paragraph{Objects.}
We aimed to choose 10 different objects from the YCB dataset~\cite{posecnn2018} that are used in daily life. As shown in Fig.~\ref{fig:ycb_objects},
we have a wide variety of sizes such as large objects (e.g. Bleach)  that cause large hand occlusion, or the objects that make grasping and manipulation difficult (e.g. Scissors), while these are not the case in the existing hand+object datasets.

\paragraph{Multi-Camera Setup.} We use 5 calibrated RGB-D cameras, in our multi-camera setup. The cameras are located at different angles and locations. Our cameras are synchronized with a precision of about 5 ms. The scenes are cluttered with objects, and the backgrounds vary between scenes.

Figs.~\ref{fig:HO_3D_1} and \ref{fig:HO_3D_2} show some examples of the 3D annotated frames for both hand and object from our proposed dataset, HO-3D.



\begin{figure*}
	\begin{center}
		\begin{tabular}{cccc}
			\includegraphics[width=0.24\linewidth,trim=0.8cm 3cm 0.8cm 3cm,clip]{figures/suppDataset/unseenObjects/00435_concat.jpg} & \hspace{-3mm}\includegraphics[width=0.24\linewidth,trim=0.8cm 3cm 0.8cm 3cm,clip]{figures/suppDataset/unseenObjects/00556_concat.jpg} & \hspace{-3mm}\includegraphics[width=0.24\linewidth,trim=0.8cm 3cm 0.8cm 3cm,clip]{figures/suppDataset/unseenObjects/00594_concat.jpg} & \hspace{-3mm}\includegraphics[width=0.24\linewidth,trim=0.8cm 3cm 0.8cm 3cm,clip]{figures/suppDataset/unseenObjects/00646_concat.jpg} \\
			
			\includegraphics[width=0.24\linewidth,trim=0.8cm 3cm 0.8cm 3cm,clip]{figures/suppDataset/unseenObjects/00746_concat.jpg} & \hspace{-3mm}\includegraphics[width=0.24\linewidth,trim=0.8cm 3cm 0.8cm 3cm,clip]{figures/suppDataset/unseenObjects/00760_concat.jpg} & \hspace{-3mm}\includegraphics[width=0.24\linewidth,trim=0.8cm 3cm 0.8cm 3cm,clip]{figures/suppDataset/unseenObjects/00769_concat.jpg} & \hspace{-3mm}\includegraphics[width=0.24\linewidth,trim=0.8cm 3cm 0.8cm 3cm,clip]{figures/suppDataset/unseenObjects/00787_concat.jpg} \\
			
			\includegraphics[width=0.24\linewidth,trim=0.8cm 3cm 0.8cm 3cm,clip]{figures/suppDataset/unseenObjects/00809_concat.jpg} & \hspace{-3mm}\includegraphics[width=0.24\linewidth,trim=0.8cm 3cm 0.8cm 3cm,clip]{figures/suppDataset/unseenObjects/00882_concat.jpg} & \hspace{-3mm}\includegraphics[width=0.24\linewidth,trim=0.8cm 3cm 0.8cm 3cm,clip]{figures/suppDataset/unseenObjects/01002_concat.jpg} & \hspace{-3mm}\includegraphics[width=0.24\linewidth,trim=0.8cm 3cm 0.8cm 3cm,clip]{figures/suppDataset/unseenObjects/01022_concat.jpg} \\
			
			\includegraphics[width=0.24\linewidth,trim=0.8cm 3cm 0.8cm 3cm,clip]{figures/suppDataset/unseenObjects/01072_concat.jpg} & \hspace{-3mm}\includegraphics[width=0.24\linewidth,trim=0.8cm 3cm 0.8cm 3cm,clip]{figures/suppDataset/unseenObjects/01108_concat.jpg} & \hspace{-3mm}\includegraphics[width=0.24\linewidth,trim=0.8cm 3cm 0.8cm 3cm,clip]{figures/suppDataset/unseenObjects/01155_concat.jpg} & \hspace{-3mm}\includegraphics[width=0.24\linewidth,trim=0.8cm 3cm 0.8cm 3cm,clip]{figures/suppDataset/unseenObjects/01174_concat.jpg} \\
			
			\includegraphics[width=0.24\linewidth,trim=0.8cm 3cm 0.8cm 3cm,clip]{figures/suppDataset/unseenObjects/01353_concat.jpg} & \hspace{-3mm}\includegraphics[width=0.24\linewidth,trim=0.8cm 3cm 0.8cm 3cm,clip]{figures/suppDataset/unseenObjects/01395_concat.jpg} & \hspace{-3mm}\includegraphics[width=0.24\linewidth,trim=0.8cm 3cm 0.8cm 3cm,clip]{figures/suppDataset/unseenObjects/01409_concat.jpg} & \hspace{-3mm}\includegraphics[width=0.24\linewidth,trim=0.8cm 3cm 0.8cm 3cm,clip]{figures/suppDataset/unseenObjects/01791_concat.jpg} \\
			
			\includegraphics[width=0.24\linewidth,trim=0.8cm 3cm 0.8cm 3cm,clip]{figures/suppDataset/unseenObjects/01823_concat.jpg} & \hspace{-3mm}\includegraphics[width=0.24\linewidth,trim=0.8cm 3cm 0.8cm 3cm,clip]{figures/suppDataset/unseenObjects/01907_concat.jpg} & \hspace{-3mm}\includegraphics[width=0.24\linewidth,trim=0.8cm 3cm 0.8cm 3cm,clip]{figures/suppDataset/unseenObjects/01994_concat.jpg} & \hspace{-3mm}\includegraphics[width=0.24\linewidth,trim=0.8cm 3cm 0.8cm 3cm,clip]{figures/suppDataset/unseenObjects/02036_concat.jpg} \\
			
			\includegraphics[width=0.24\linewidth,trim=0.8cm 3cm 0.8cm 3cm,clip]{figures/suppDataset/unseenObjects/02061_concat.jpg} & \hspace{-3mm}\includegraphics[width=0.24\linewidth,trim=0.8cm 3cm 0.8cm 3cm,clip]{figures/suppDataset/unseenObjects/02107_concat.jpg} & \hspace{-3mm}\includegraphics[width=0.24\linewidth,trim=0.8cm 3cm 0.8cm 3cm,clip]{figures/suppDataset/unseenObjects/02136_concat.jpg} & \hspace{-3mm}\includegraphics[width=0.24\linewidth,trim=0.8cm 3cm 0.8cm 3cm,clip]{figures/suppDataset/unseenObjects/02195_concat.jpg} \\
			
		
			
		\end{tabular}
	\end{center}
	\vspace{-2mm}
	\caption{Qualitative results of 3D hand pose estimation of hand manipulating unseen objects. Our pose estimator trained on the \datasetname dataset is still able to predict accurate 3D poses when interacting with new objects.}
	\label{fig:unseen_objects}
\end{figure*}

\begin{figure*}
	\begin{center}
	\begin{tabular}{ccc}
		\includegraphics[width=0.33\linewidth]{figures/suppDataset/banana/annoVis_01046_3.jpg} & \hspace{-3mm}
		\includegraphics[width=0.33\linewidth]{figures/suppDataset/banana/annoVis_01558_1.jpg} & \hspace{-3mm}
		\includegraphics[width=0.33\linewidth]{figures/suppDataset/banana/annoVis_01993_0.jpg} \\

		\includegraphics[width=0.33\linewidth]{figures/suppDataset/bleach/annoVis_00495_4.jpg} & \hspace{-3mm}
		\includegraphics[width=0.33\linewidth]{figures/suppDataset/bleach/annoVis_01230_0.jpg} & \hspace{-3mm}
		\includegraphics[width=0.33\linewidth]{figures/suppDataset/bleach/annoVis_00805_1.jpg} \\

		\includegraphics[width=0.33\linewidth]{figures/suppDataset/cracker/annoVis_0480.jpg} & \hspace{-3mm}
		\includegraphics[width=0.33\linewidth]{figures/suppDataset/cracker/annoVis_0809.jpg} & \hspace{-3mm}
		\includegraphics[width=0.33\linewidth]{figures/suppDataset/cracker/annoVis_0897.jpg} \\

		\includegraphics[width=0.33\linewidth]{figures/suppDataset/mug/annoVis_01131_1.jpg} & \hspace{-3mm}
		\includegraphics[width=0.33\linewidth]{figures/suppDataset/mug/annoVis_01543_2.jpg} & \hspace{-3mm}
		\includegraphics[width=0.33\linewidth]{figures/suppDataset/mug/annoVis_1545.jpg} \\

		\includegraphics[width=0.33\linewidth]{figures/suppDataset/mustard/annoVis_0624.jpg} & \hspace{-3mm}
		\includegraphics[width=0.33\linewidth]{figures/suppDataset/mustard/annoVis_0833.jpg} & \hspace{-3mm}
		\includegraphics[width=0.33\linewidth]{figures/suppDataset/mustard/annoVis_0870.jpg} \\

	\end{tabular}
	\end{center}
	\vspace{-2mm}
	\caption{Some examples of the 3D annotated frames for both hand and object from our proposed dataset, HO-3D.}
	\label{fig:HO_3D_1}
\end{figure*}

\begin{figure*}
	\begin{center}
	\begin{tabular}{ccc}
		
		\includegraphics[width=0.33\linewidth]{figures/suppDataset/pitcher/annoVis_0393.jpg} & \hspace{-3mm}
		\includegraphics[width=0.33\linewidth]{figures/suppDataset/pitcher/annoVis_1004.jpg} & \hspace{-3mm}
		\includegraphics[width=0.33\linewidth]{figures/suppDataset/pitcher/annoVis_1736.jpg} \\

		\includegraphics[width=0.33\linewidth]{figures/suppDataset/pottedmeat/annoVis_00431_3.jpg} & \hspace{-3mm}
../		\includegraphics[width=0.33\linewidth]{figures/suppDataset/pottedmeat/annoVis_01133_4.jpg} & \hspace{-3mm}
		\includegraphics[width=0.33\linewidth]{figures/suppDataset/pottedmeat/annoVis_01255_1.jpg} \\

		\includegraphics[width=0.33\linewidth]{figures/suppDataset/powerdrill/annoVis_0009.jpg} & \hspace{-3mm}
		\includegraphics[width=0.33\linewidth]{figures/suppDataset/powerdrill/annoVis_00563_2.jpg} & \hspace{-3mm}
		\includegraphics[width=0.33\linewidth]{figures/suppDataset/powerdrill/annoVis_01039_3.jpg} \\

		\includegraphics[width=0.33\linewidth]{figures/suppDataset/scissor/annoVis_00624_0.jpg} & \hspace{-3mm}
		\includegraphics[width=0.33\linewidth]{figures/suppDataset/scissor/annoVis_01617_3.jpg} & \hspace{-3mm}
		\includegraphics[width=0.33\linewidth]{figures/suppDataset/scissor/annoVis_01855_2.jpg} \\

		\includegraphics[width=0.33\linewidth]{figures/suppDataset/sugar/annoVis_0500.jpg} & \hspace{-3mm}
		\includegraphics[width=0.33\linewidth]{figures/suppDataset/sugar/annoVis_0687.jpg} & \hspace{-3mm}
		\includegraphics[width=0.33\linewidth]{figures/suppDataset/sugar/annoVis_0841.jpg} \\
	\end{tabular}
	\end{center}
	\vspace{-2mm}
	\caption{Some examples of the 3D annotated frames for both hand and object from our proposed dataset, HO-3D.}
	\label{fig:HO_3D_2}
\end{figure*}

\begin{figure*}[ht]
	\begin{center}
		\begin{tabular}{ccccc}
			\includegraphics[width=0.18\linewidth]{figures/ycb/ycb_banana.png} & 
			\includegraphics[width=0.18\linewidth]{figures/ycb/ycb_bleach.png} & 
			\includegraphics[width=0.18\linewidth]{figures/ycb/ycb_cracker.png} & 
			\includegraphics[width=0.18\linewidth]{figures/ycb/ycb_driller.png} & 
			\includegraphics[width=0.18\linewidth]{figures/ycb/ycb_mug.png} \\
			
			Banana & Bleach & Cracker Box & Driller & Mug \\
			
			\includegraphics[width=0.18\linewidth]{figures/ycb/ycb_mustard.png} & 
			\includegraphics[width=0.18\linewidth]{figures/ycb/ycb_pitcher.png} & 
			\includegraphics[width=0.18\linewidth]{figures/ycb/ycb_potted.png} & 
			\includegraphics[width=0.18\linewidth]{figures/ycb/ycb_scissors.png} & 
			\includegraphics[width=0.18\linewidth]{figures/ycb/ycb_sugar.png} \\
			
			Mustard Bottle & Pitcher & Potted Meat & Scissors & Sugar Box \\
		\end{tabular}
	\end{center}
	\vspace{-2mm}
	\caption{ 10 objects of the YCB dataset~\cite{posecnn2018} that we use for our dataset HO-3D.}
	\label{fig:ycb_objects}
\end{figure*}





