

\section{Introduction}\label{sec:intro}

Part-structure provides a compact and meaningful intermediate shape representation of articulated objects. Co-part segmentation, which aims to label semantic part belonging for each pixel of the objects in an image, is an important problem in computer vision. Such capability can directly serve various higher-level tasks such as marker-less motion tracking, action recognition and prediction, robot manipulation, and human-machine interaction.

 With the advent of deep learning, and the availability of large amount of annotated motion datasets, supervised learning-based approaches have led to superior performances over traditional part segmentation methods; the most success has been achieved for human pose estimation, e.g., ~\cite{guler2018densepose, kanazawa2018end}. However, this approach assumes significant domain knowledge, and highly depends on the specific dataset used for training, thus making it difficult to generalize to objects with different appearance, lighting or pose, not to mention unseen subjects. 


A video sequence is viewed as a spatio-temporal intensity volume that contains all structural and motion information of the action, including poses of the subject at any time as well as the dynamic transitions between the poses. Our goal in this research is to extract a general part-based representation from videos. Compared with single image-based segmentation, our work intends to aggregate shape correlation information from multiple images to improve the segmentation of individual images. The capability of consistently detecting object parts are important for motion tracking of creatures of various topology, and ultimately extracting their skeletal structures. 

A successful line of recent works in this direction formulates the task as an image generation problem, where segmented parts are globally warped to form the final image. There, part-segmentation becomes the essential intermediate step, because: the better you can segment (parts), the better you can generate (the image). In this paper, we follow the same image-generation concept, but introduce a dual procedure of part-assembly to form a closed loop with part-segmentation, which ensures more consistent, also more compact and meaningful part segmentation. Specifically, we generate the final image through blending each part's warped image, instead of a global image warping. In essence, our image based assembly operation effectively constrains the manifold of each individual part, resulting in improved results.  

We take an unsupervised learning approach. Like many recent works about unsupervised / self-supervised part segmentation, we believe shape correlation information between different frames can be leveraged for achieving semantic consistency. Our approach is similar to PSD~\cite{psd}, Motion Co-part~\cite{lathuiliere2020motion} and Flow Capsule~\cite{sabour2020unsupervised}, in the use of motion cues embedded in different frames for co-part segmentation. We go beyond the existing techniques in multiple ways:
\begin{enumerate}[(1)]
\setlength{\itemsep}{0pt}
	\item In our method, the supervision is attained by introducing a novel dual-procedure of part-assembly to form a close loop with part-segmentation.  
	\item The learned parts and their transformation have clear explainable physical meaning. 
	\item Our method doesn't require any field-based global warping operation, which enables handling dramatically dynamic motions. 
\end{enumerate}

We demonstrate the advantages of our method both visually and quantitatively. Extensive experiments have been conducted on datasets showcasing challenges due to the change in appearance, occlusions, scale and background.
We also compared with recent works including NEM~\cite{NIPS2017_nem}, R-NEM~\cite{steenkiste2018relational_RNEM}, PSD~\cite{psd}, SCOPS~\cite{hung2019scops}, Motion Co-part~\cite{lathuiliere2020motion} and Flow Capsule~\cite{sabour2020unsupervised}. Our method outperforms state-of-the-art methods in quantitative evaluation. 
 
 



\section{Related work}\label{sec:related_work}
\paragraph{Part-based Representation}
In analyzing images, describing object as a collection of parts, each with a range of possible states, is a classical framework for learning an object representation in computer vision ~\cite{ross2006learning, nguyen2013learning}. The states can be computed based on different evidences, such as visual and semantic features~\cite{wang2015semantic},  geometric shape and its behavior under viewpoint changes~\cite{eslami2012generative} and object articulation~\cite{sun2011articulated}, resulting in a large variation of part partition. Our work performs motion-based part segmentation, where each part is constituted with a group of pixels moving together.

\paragraph{Motion-based Co-part Segmentation}
Motion-based Co-part segmentation has been an important problem in understanding and reconstructing dynamic scenes.
Articulated object can be naturally segmented as a group of rigid parts, if prior knowledge on underlying kinematic structure is known. However, this assumption does not hold in our problem setting. 
Most of the traditional computer vision technology recovers rigid part and kinematic structure by exploiting motion information, in particular, RGB image sequences with feature points tracked over time. There have been three main approaches: (i) motion segmentation and factorization~\cite{Yan08}, (ii) probabilistic graphical model~\cite{Ross10, Sturm11}, and (iii) cost function based optimization methods~\cite{Ochs14, Keuper15}. The work from Chang and Demiris~\cite{Chang18} achieved state-of-the-art performance on the reconstruction of articulated structures from 2D image sequences. There, the segmentation was executed on tracked key-points, rather than all pixels like in our approach; the method, however, is prone to image noise, occlusions, deformations and cannot deal with articulated structures of high complexity.

\paragraph{Unsupervised Co-part Segmentation}
 With the popularity of deep neural networks, motion part segmentation has achieved superior performance in domains where labeled data are abundant, such as faces~\cite{khan2015multi} and human bodies~\cite{guler2018densepose, kanazawa2018end}.
Parts segmentation can also be learned in an entirely unsupervised fashion. 
Nonnegative Matrix Factorization (NMF) ~\cite{lee1999learning} learned features that exhibit sparse part-based representation of data to disentangle the hidden structure of data. \cite{collins2018deep} further proposed deep feature factorization (DFF) to estimate the common part segments in images through NMF. 
Leveraging on semantic consistency in an image collection of single object category, Hung et. al. \cite{hung2019scops} proposed a self-supervised network SCOPS to predict part segmentation based on the pre-trained CNN features. 
\cite{psd} proposed a deep model to discover object parts and the associated hierarchical structure and dynamical model from unlabeled videos. However, they assume that pre-computed motion information is available. 
\cite{lathuiliere2020motion} proposed a model to leverage motion information with the purpose of obtaining segments that correspond to group of pixels associated to object parts moving together. But the transformation between parts in different frame is not explainable, and the motion merge to flow rather than on each part.
\cite{sabour2020unsupervised} proposed exploit motion as a powerful perceptual cue for part definition, using an expressive decoder for part generation and layered image formation with occlusion. But they still rely on flow to warp image, instead of considering that each part has independent motion.








\section{Method}\label{sec:method}

Our goal is to train a deep neural network to compute part segmentations and estimate part motions from a single input image.
We train our network using an image collection of the same object category, which can be extracted from videos of an animating object.
Our unsupervised training process guides the network to identify object parts in the images by observing their motions.
To facilitate the training, we assemble the generated parts to recover the input images, which can be considered a dual-procedure of part segmentation. 
In what follows, we will first introduce our segmentation model in detail, then discuss the objectives and the training process.

\subsection{Model}\label{sec:method:model}

Our part segmentation network consists of an {\emph{image encoder}} network and a \emph{segment decoder} network. The image encoder encodes a given image into a set of latent feature maps, each corresponding to an object part. The segment decoder then decodes these part feature maps into part segments, which are later assembled to reconstruct the input image. Figure~\ref{fig:overview} provides an overview of the proposed network structure. 

\paragraph{Image Encoder}
Given an image $I\in\mathbf{R}^{H\times{}W\times{}C}$ as input, the image encoder, $\mathcal{E}$, computes latent representations of $K$ part segments, each represented by a feature map, $V_k\in{} \mathbf{R}^{H'\times{}W'\times{}C'}$, where $k\in\mathcal{K}$ and $\mathcal{K}=\{1,\dots,K\}$ is the set of part indices. 
These latent feature maps implicitly capture the shapes, appearances, locations, and poses of the corresponding part segments in the input image.
We treat the background region as a special part segment and represent the corresponding feature map as $V_0$.

The encoder also estimates a set of affine transformations, $\{T_k\}$, for each part $k\in\mathcal{K}$. We assume there exists a set of \emph{canonical} parts located at the center of the image, which are shared by all the images and can be transformed by $\{T_k\}$ to match the current parts in the input image. We use $V_k^*$ to represent the feature map of a canonical part.


In our implementation, a transformation, $T$, is given by a 6-tuple
\begin{equation}\label{eqn:transformation}
    T = (s_x, s_y, s_{\theta}, c_{\theta}, t_x, t_y)
\end{equation}
where $(s_x, s_y)$ and $(t_x, t_y)$ represent the scaling and the translation of the transformation. 
To avoid the continuity issue of angle representations \cite{zhou_continuity_2019}, we use two variables $(s_{\theta}, c_{\theta})$ to represent a rotation $\theta$, which correspond to the sine and cosine of $\theta$ respectively. The transformation matrix of $T$ is then given by
\begin{equation}
    T = \left[\begin{array}{cc}
         A & \bm{t}  \\
         \bm{0} & 1 
    \end{array} \right]
\end{equation}
where
\begin{equation*}
    A = 
    \left[\begin{array}{cc}
        \bar{c}_{\theta} & -\bar{s}_{\theta} \\
        \bar{s}_{\theta} & \bar{c}_{\theta} 
    \end{array}\right]
    \left[\begin{array}{cc}
        s_x & 0  \\
        0 & s_y 
    \end{array}\right],
    \bm{t} = \left[\begin{array}{c}
        t_x \\ t_y
    \end{array}\right]
\end{equation*}
and $(\bar{s}_{\theta},\bar{c}_{\theta}) = (s_{\theta}, c_{\theta})/\parallel(s_{\theta}, c_{\theta})\parallel_2$. 


\paragraph{Segment Decoder} 
The \emph{segment decoder} network, $\mathcal{D}$, is trained to convert a latent feature map ${V_k}$ into a part image, $P_k\in{}\mathbf{R}^{H\times{}W\times{}C}$, which recovers the appearance of the part in the original image.

We use the same decoder network $\mathcal{D}$ to convert feature maps of all the object parts and the background into part images. The decoder $\mathcal{D}$ also outputs a {depth map}, $D_k\in{}\mathbf{R}^{H\times{}W}$, for each part. With $\bm{u}$ representing the coordinates of a pixel, $D_k(\bm{u})$ is a scalar that specifies the relative inverse depth of the corresponding pixel located at $\bm{u}$ in part image $P_k$.
We assume that the object is composed of opaque parts, so that a part with smaller inverse depth (thus farther from the camera) will be partially occlude by the parts with larger inverse depth. 
The part mask $M_k\in\bm{R}^{H\times{}W}$ is thus a pixel-wise visibility mask indicating whether a pixel in the part image $P_k$ is visible in the original image, which can be computed as
\begin{equation}\label{eqn:part_mask}
    M_k(\bm{u})=\softmax\limits_{l\in{}\{0\} \cup{} \mathcal{K}} D_l(\bm{u})
\end{equation}



\paragraph{Part Assembly}
We train the segmentation network by assembling the part images together and reconstructing the input image. This is achieved by gathering the visible pixels from all the part images. Specifically, the reconstructed image $\tilde{I}$ is computed as
    $\tilde{I} = \sum_{k=0}^{K}\tilde{I}_k$
where $\tilde{I}_k$ is the visible part of part image $P_k$ as specified by the part mask $M_k$. We compute $\tilde{I}_k$ using
    $\tilde{I}_k = {M}_k \odot{} P_k$,
where $\odot$ is the Hadamard (pixel-wise) product between two arrays.








\subsection{Training}\label{sec:training}
We train our co-part segmentation network using image pairs randomly selected from the input image collection. During training, we require the network to reconstruct one image of a pair (the \emph{source} image, $I^s$) as accurate as possible, while using the other image of the pair (the \emph{target} image, $I^t$) to cross-validate the latent representation of the parts and the segmentation results. 
This validation is performed by constructing the target image using the part segments extracted from the source image.
{Unlike the existing works that warp the source image using optical flow \cite{siarohin2020first,siarohin2020motion,sabour2020unsupervised}, we transform the parts in latent space directly and decode the transformed latent features of the parts to generate the target image.} 

Figure~\ref{fig:training} provides an overview of this training process. In more details, the encoder network $\mathcal{E}$ takes the two images $I^s$ and $I^t$ as input and computes latent feature maps and part transformations for both of them. The results are denoted as $\{V^s_0, (V_k^s, T_k^s)\}$ and $\{V^t_0, (V_k^t, T_k^t)\}$ respectively. 
We transform each source latent feature map $V_k^s$ using the corresponding transformations $T_k^s$ and $T_k^t$, and assemble the resulting latent maps $\{\tilde{V}^{s\rightarrow{}t}_0, \tilde{V}^{s\rightarrow{}t}_k\}$ using the segment decoder $\mathcal{D}$ to produce the retargeted image $\tilde{I}^{s\rightarrow{}t}$. Note that we assume the background is static and use $\tilde{V}^{s\rightarrow{}t}_0 = V^s_0$ in this transformation.

This retargeting is performed in two steps. First, we inversely transform $V_k^s$ using $T_k^s$ to compute the canonical latent feature map $V_k^*=({T_k^s})^{-1}\circ{}V_k^s$, which is assumed to be shared by both the source and the target. 
The transformation operation $\circ{}$ is defined as 
\begin{equation}
(T\circ{}V)(\bm{u}) = V(T^{-1}\bm{u})
\end{equation}
where $\bm{u}$ and its transformed counterpart $T^{-1}\bm{u}$ are both coordinates of pixels.
Then the target transformation $T_k^t$ is applied to $V_k^*$ to compute the retargeted feature map $\tilde{V}^{s\rightarrow{}t}_k={T_k^t}\circ{}{V_k^*}=[{T_k^t}(T_k^s)^{-1}]\circ{}{V_k^*}$. 
We also recover the source feature map from $V_k^*$ as $\tilde{V}^{s}_k={T_k^s}\circ{}{V_k^*}$. We find this additional procedure helpful in facilitating the training at the early stage of the process.

The resulting feature maps $\{\tilde{V}^{s}_0, \tilde{V}^{s}_k\}$ and $\{\tilde{V}^{s\rightarrow{}t}_0, \tilde{V}^{s\rightarrow{}t}_k\}$ are then input to the decoder $\mathcal{D}$ to assemble the reconstructed image $\tilde{I}^s$ and the retargeted images $\tilde{I}^{s\rightarrow{}t}$ respectively. 
In the meanwhile, the corresponding part masks $\{\tilde{M}^s_k\}$, $\{\tilde{M}^{s\rightarrow{}t}_k\}$, and $\{M^*_k\}$, computed using $\mathcal{D}$ and Equation~\eqref{eqn:part_mask}, are recorded as well, which are used as a part of the training objective as described below.
We train our segmentation network in an end-to-end fashion with an objective formulated as a weighted sum of several losses.

\paragraph{Image Reconstruction Loss}
The main driving loss of our training is the image reconstruction loss, which penalizes the difference between the generated images and the corresponding inputs.
The difference between images is measured based on the perceptual loss of~\cite{johnson2016vggloss}, where a pretrained VGG-19 network~\cite{simonyan2015vgg} is used to extract features from the images for comparison. The difference between two images $I$ and $\tilde{I}$ is then computed as
\begin{equation}\label{eqn:rec_loss}
\begin{aligned}
    \mathcal{L}(I, \tilde{I}) &= {\lambda_1}\parallel{}I - \tilde{I}\parallel_1
     + {\lambda_2}\parallel \nabla{}{I} - \nabla{}{\tilde{I}} \parallel_1 \\
    &+{\lambda_3}\parallel \phi_{\text{vgg}}({I}) - \phi_{\text{vgg}}({\tilde{I}})\parallel_1
\end{aligned}
\end{equation}
where $\nabla\cdot$ computes image gradient {as suggested by~\cite{eigen_predicting_2015}}, and $\phi_{\text{vgg}}(\cdot)$ extracts VGG-19 features from the image.
The reconstruction loss is applied to both the reconstructed source image $\tilde{{I}^s}$ and the regargeted image $\tilde{I}^{s\rightarrow{}t}$. The total loss is thus
\begin{equation}
    \mathcal{L}_\text{rec} = \lambda_{s}\mathcal{L}(I^s, \tilde{I}^s) + \lambda_{t}\mathcal{L}(I^t, \tilde{I}^{s\rightarrow{}t})
\end{equation}


\input{vis_hand_horse_and_arm}

\paragraph{Background Loss}
As described in the last section, we treat the background as a special part segment and compute its features map and part mask. In practice, however, we observe that some background pixels can appear in the other part masks, causing noisy part segmentation.
To address this problem, we include a novel background loss in the training to encourage clear partition between the object parts and the background.
This is achieved by encouraging the background part to occupy as much image as possible, forcing the object parts to shrink into the most relevant region in the image.
The background loss is thus defined as
\begin{equation}
\mathcal{L}_{\text{bg}} = \lambda_{\text{bg}}
    \left(\parallel M_0 - 1 \parallel_1 + \sum_{k=1}^{K} \parallel {M_k}\parallel_1 \right)
\end{equation}
which drives the values of the background mask close to one and the values of the other part masks close to zero.
We apply this background loss to both the source parts $M^s_k$ and the retargeted parts $M^{s\rightarrow{}t}_k$. We find this loss essential for precise part segmentation with tight boundaries.


\paragraph{Transformation Loss}
In our system, we expect each part transformation $T_k$ to be strongly correlated with the absolute pose of the part $k$ in the input image and thus has a clearly explainable physical meaning. 
More specifically, each part transformation $T_k= \left[A_k | \bm{t}_k \right]$ defines a coordinate system with the origin at $\bm{t}_k$ and the axes defined by $A_k$. We assume $\bm{t}_k$ to be located at the center of the part $k$ and the axis $A_k$ align with the longest and the shortest dimensions of the object part. We enforce such property in the training using a novel transformation loss defined as $\mathcal{L}_\text{trf}=\mathcal{L}_\text{tran}+\mathcal{L}_\text{rots}$, where
\begin{align}
    \mathcal{L}_\text{tran}&=
    \lambda_\text{tran}\sum_{k=1}^K\parallel \bm{t}_k - \bm{\hat{u}}_k \parallel_1 \\
    \mathcal{L}_\text{rots}&=
    \lambda_\text{rots}\sum_{k=1}^K\parallel A_k A_k^T - \Sigma_k \parallel_F
\end{align}
Since we do not have the ground-truth part poses, we estimate the reference transformation using the mean and the covariance of the part mask $M_k$ as 
\begin{align}
    \bm{\hat{u}}_k&=\frac{1}{z_k}{\sum_{\bm{u}\in{}\mathcal{U}}\bm{u}{M_k}(\bm{u})} \label{eq:COM}\\
    \Sigma_k&=\frac{1}{z_k} {\sum_{\bm{u}\in{}\mathcal{U}}
        (\bm{u}-\bm{\hat{u}}_k)(\bm{u}-\bm{\hat{u}}_k)^T{M_k}(\bm{u})}
    \label{eq:part_cov}
\end{align}
where $M_k(\bm{u})$ represents the mask value of the pixel located at $\bm{u}\in\mathcal{U}$ and $z_k=\sum_{\bm{u}\in{}\mathcal{U}}{M_k}(\bm{u})$ is a normalization constant. In this estimation, we only consider the pixels that have been clearly identified as part $k$ with a threshold $\zeta$, so that $\mathcal{U}=\{\bm{u}|M_k(\bm{u})>\zeta\}$. {We choose $\zeta=0.02$ empirically in our implementation.}

\input{vis_part_image}

\paragraph{Equivariance Loss}
The estimation of the part transformations should be consistent across images and show equivariance to image transformations. Following the common practice of unsupervised landmark detection~\cite{jakab_unsupervised_2018,zhang_unsupervised_2018,siarohin2020first}, we employ an equivariance loss in our training. 
Specifically, we transform the input image $I$ using a random transformation $T^w$. The encoder $\mathcal{E}$ then estimates part transformations for both the original image $I$ and the transformed image $I^w$. 
The equivariance loss is then defined as
\begin{equation}\label{eqn:eq_loss}
    \mathcal{L}_{\text{eq}} = \lambda_{\text{eq}} \sum_{k=1}^K \parallel T^wT_k - T^w_k \parallel_1
\end{equation}
where $T_k$ and $T^w_k$ are the part transformations estimated from $I$ and $I^w$ respectively, and the 1-norm is computed with the transformation matrix treated as a vector.


\paragraph{Concentration Loss}
To encourage the pixels belonging to the same object part to form a connected and concentrated component,
we employ the geometry concentration loss suggested in SCOPS~\cite{hung2019scops} to regularize the shape of part mask $M_k$. Specially, this concentration loss is computed as
\begin{equation}\label{eqn:con_loss}
    \mathcal{L}_{\text{con}} =\lambda_{\text{con}} \sum_{k=1}^{K} \sum_{\bm{u}} \parallel \bm{u}  - \bm{\bar{u}}_k \parallel^2_2 \cdot {M_k}(\bm{u})/{z_k}
\end{equation}
where 
\begin{equation}\label{eqn:com_all}
    \bm{\bar{u}}_k=\frac{1}{z_k}\sum_{\bm{u}}\bm{u}{M_k}(\bm{u})
\end{equation}
is the center of gravity of the part mask $M_k$, and $z_k=\sum_{\bm{u}}{M_k}(\bm{u})$ is a normalization constant. Unlike the transformation loss, we consider all the pixels in $M_k$ in this concentration loss. 
Note that the summation of Equation~\eqref{eqn:con_loss} excludes the part mask corresponding to the background.
As shown in Fig.~\ref{fig:training}, we apply concentration loss to the part masks of the source parts $M^s_k$, the retargeted parts $M^{s\rightarrow{}t}_k$, and the {canonical} parts $M^*_k$.


\section{Experiments}\label{sec:experiments}

Our model is implemented using the the standard U-Net architecture. We include the details about the network structure and the training settings in the supplementary materials.
We visually demonstrate the effectiveness of our co-part segmentation method on several test cases with large variation, including human, hand, quadruped, and robot arms in Figure~\ref{fig:other}, where the resulting part segments are rendered with different colors
\llb{with the corresponding masks computed using \emph{hard}max $M_k(\bm{u})=\max_{l\in{0}\cup{} \mathcal{K}}D_l(\bm{u})$ instead of Equation~\eqref{eqn:part_mask}}.
Additionally, Figure~\ref{fig:vis_part_image} illustrates the individual segment images computed by our network.  

\llb{
Our method is designed to extract parts that exhibit different affine transformations in the training image pairs, which is consistent with the behavior of semantically meaningful segmentation of a subject, such as a human body.
The order of these parts is not determined in the unsupervised learning process. As in previous works, we manually label those parts after the model training. Notably, we only need to label the parts once, and the labels are consistent over all test images.
}



In the rest part of this section, we will introduce the ablation studies we performed to analyze the effectiveness of each loss component in our framework, and also the comparison with state-of-the-art co-part segmentation techniques. 

\input{taichi_vis}
\input{vox_vis}
\input{taichi_evaluation} 


\subsection{Datasets}
\paragraph{Tai-Chi-HD.} Tai-Chi-HD dataset~\cite{siarohin2020first} is a collection of short videos with full-body Tai-Chi movements. $2981$ Tai-Chi videos were downloaded from YouTube. The videos were cropped and resized to a fixed resolution of $128 \times 128$, while preserving the aspect ratio. There are $2746$ training videos and $235$ test videos. This dataset contains $5300$ images with ground truth landmarks ($18$ joints) generated using the method from Cao et al.\cite{cao2019openpose}. Only $300$ images with ground truth foreground segmentation mask are available.

\input{exercise_evaluation}

\paragraph{VoxCeleb.} The VoxCeleb dataset~\cite{nagrani2017voxceleb} is a large scale face dataset, which consists of $22496$ videos, extracted from YouTube. We follow the preprocessing described in ~\cite{siarohin2020first} to crop original video into several short sequences to guarantee that face can move freely in the image space with reasonable scale. All the cropped videos are then resized to $128 \times 128$, again, preserving the aspect ratio. After the preprocessing, our dataset contains $15103$ training videos and $443$ test videos. The length of each video varies from $64$ to $1024$ frames. 
This dataset contains $5300$ images with ground truth landmarks ($68$ keypoints) generated using the method from Bulat et al.\cite{bulat2017far}. Similarly, only $300$ images with ground truth foreground segmentation mask are available.

\paragraph{Exercise.} Exercise dataset is a collection of paired images from two consecutive frames for full human body performing Yoga exercises. This dataset is originally collected by \cite{xue2016visual} from YouTube, and preprocessed with motion stabilization. We use the reorganized version of this dataset provided by \cite{psd}, which contains $49356$ pairs of images for training. For this dataset, only $30$ images with ground truth part segmentation masks are available.


\subsection{Metrics}
\paragraph{Intersection Over Union}
We use the commonly adopted mean intersection over union (IoU) metric to evaluate how similar our predicted segmentation is to the ground truth. The average IoU across all frames of the dataset is practically used. 
We use foreground IoU for the test on VoxCeleb and Tai-Chi-HD datasets due to the shortage of ground-truth part segmentation mask; while part IoU is used for the test on Exercise dataset.
\paragraph{Landmark Regression MAE}
We use the landmark regression MAE metric to evaluate whether our method can generate consistent semantic part segmentation on different images. Following 
~\cite{hung2019scops}, We first fit a linear regression model from the parts' center of mass $\bm{\bar{u}}_k$ to ground truth landmarks using $5000$ annotated images, where $\bm{\bar{u}}_k$ is calculated using Equation~\ref{eqn:com_all}, and then on the other $300$ images, we compute the mean average error (MAE) between regressed and ground truth landmark positions as the evaluation metric. 



\input{generalization}

\subsection{Quantitative Comparison}
We quantitatively compare our result with that of the state of the art methods for co-part segmentation, SCOPS~\cite{hung2019scops}, Motion Co-part~\cite{siarohin2020motion}, PSD~\cite{psd}, NEM~\cite{NIPS2017_nem}, N-REM~\cite{steenkiste2018relational_RNEM} and Flow Capsules~\cite{sabour2020unsupervised}, using both IoU and Landmark metrics. 




We first compared our method with SCOPS and Motion Co-part using landmark regression and foreground IoU metrics on VoxCeleb and Tai-Chi-HD dataset, which have rich variation on background texture, actor's appearance and body proportions, etc. To make a fair comparison, we train our model with $K=10$ in this test, which is consistent with the settings of the other methods. 
The results in Table~\ref{taichi} indicate that our method significantly improves the accuracy of foreground segmentation and can achieve more consistent and precise segmentation. As illustrated in Figures~\ref{fig:taichi} and \ref{fig:VoxCeleb}, our approach achieves more consistent part segmentation than SCOPS, where the main objects are clearly separated from the background. In the meanwhile, our results are aligned with image silhouette more tightly than Motion~Co-part.



We further compare our method with PSD, NEM, R-NEM and Flow Capsule on the accuracy of co-part segmentation using the IoU metric on Exercise dataset. We train our model with $K=15$ segments in this comparison. 
As reported in Table~\ref{table:yoga}, 
our model achieves a consistently better
performance than the baselines.



\subsection{Generalization}

\llb{Our model can be trained with both single-video settings (hand, quadruped, and robot arm) and multiple-video settings (Tai-Chi-HD, VoxCeleb, and Exercise datasets), where in the latter case, each pair of images are extracted from the same random video during training. 
Models trained with multiple videos can be generalized to images of the same category but with difference appearance. For example, without further training, the models trained on Tai-Chi-HD and VoxCeleb datasets can be applied directly to videos downloaded from YouTube, as shown in the top two rows in Figure~\ref{fig:generalization}.
}

\llb{
Moreover, it is rather straightforward to extend our method to support multiple subjects. The bottom row of Figure~\ref{fig:generalization} demonstrates an example of such ability, where a model is trained on a video with two persons, and $K = 17$ is used to accommodate additional potential part segments.
}

\input{abalation_study}

\subsection{Ablation Study}
\input{ablation}
We perform ablation studies to
validate the contribution of each loss employed in our training. The comparison is conducted between the full training objective proposed in Sec.\ref{sec:training} and its variants, each with one of the losses ($\mathcal{L}_\text{vgg}$, $\mathcal{L}_\text{bg}$,  $\mathcal{L}_\text{rots}$, $\mathcal{L}_\text{tran}$, $\mathcal{L}_\text{con}$) disabled.
\llb{We use Tai-Chi-HD dataset in these experiments, and the results are reported in terms of foreground IoU and landmark regression accuracy.}
Note that in the ablation study with $\mathcal{L}_\text{vgg}$, we only remove the VGG features from $\mathcal{L}_\text{rec}$ and keep the rest terms of Equation~\eqref{eqn:rec_loss} unchanged.
The results summarized in Table~\ref{ablation_study} reveal that all the losses are beneficial towards effective learning. 

$\mathcal{L}_\text{vgg}$ is the most significant one among them, which influences both foreground extraction and part segmentation. 
Unlike the methods that generate images using field-based global warping operation~\cite{siarohin2020first,sabour2020unsupervised}, our model cannot utilize the pixels of the input image directly to generate the target image, where the VGG features significantly facilitate the training and help achieve a good performance.

$\mathcal{L}_\text{bg}$ is another critical term of our objective design, which enforces all the background information to be embedded in the background channel, thus consequently ensuring the segmentation tightly aligned with the foreground silhouette. As shown in Figure~\ref{fig:difloss}, the segmentation trained without this loss can be noisy with background pixels mislabeled as a part of object segments.

Similar to the results reported in~\cite{hung2019scops}, we find $\mathcal{L}_\text{con}$ guarantees the semantic correctness of the segmented parts through penalizing vague and scattered partition in each channel. This conforms to our ablation study that $\mathcal{L}_\text{con}$ has obvious effect on Landmark regression accuracy. More visual comparison result can be found in Figure~\ref{fig:difloss}.  

As illustrated in Figure~\ref{fig:rotation_vis}, $\mathcal{L}_\text{rots}$ and $\mathcal{L}_\text{tran}$ play a critical role in bringing the estimated part transformation with explainable meaning. The part transformations learned without these losses only loosely correlate to the global pose of the parts, while the transformations estimated using our model align with the motion of the object parts.
\llb{
We find these loss terms very effective when the transformation of each part can be clearly defined, such as in the case of human limbs in the Tai-Chi-HD dataset.
Nonetheless, these loss terms take only marginal effect on faces in the VoxCeleb dataset, though we kept this term as a part of a uniform training process.
}


\input{vis_rotation}





\section{\llb{Discussion}}
In this paper, we have proposed an unsupervised Co-part segmentation approach, which leverages shape correlation information between different frames in the video to achieve semantic part segmentation. We have designed a novel network structure which achieves self-supervision through a dual procedure of part-assembly to form a closed loop with part-segmentation. 
Additionally, we have developed several new loss functions
that ensure consistent, compact and meaningful part segmentation and the intermediate transformations with clear explainable physical meaning. We have demonstrated the advantages of our method through a host of studies. 

\input{failure_case}

\llb{
We empirically choose weights in our training to balance the magnitude of each loss term in a preliminary training, except for the image reconstruction loss, which is an order of magnitude larger than the other regularization terms due to its critical role in the training. The performance of our model is not sensitive to the specific choice of these loss weights, and similar video categories can share the loss weights.}
\llb{In order to make a fair comparison, we deliberately use different $K$ in some of the experiments to ensure consistency with the baseline methods, but the performance of the method is not sensitive to the specific value of a large enough $K$. For example, we can achieve correct segmentation using an empirical value of $K = 15$ in all the test cases discussed in the paper.
}

Several lines are open for future research. 
\llb{First, As showed in Figure~\ref{fig:failure_case}, due to the lack of temporal information, our method can fail in inferring occluded parts and may incorrectly label the limbs when a person turns around. It would be a valuable extension of the current framework to train with image sequences or videos to address such inconsistency issues using the time coherence information embedded in the video.}
\llb{%
Second, although our method allows moderate background motions as exhibited in Tai-Chi-HD and VoxCeleb datasets during training, dramatic background change can interfere the training stage and degrade the performance. Extending our method to support training on videos with dramatic background change is a viable future work.%
}
Another interesting direction would be additionally identifying joint positions, which would significantly support a more diverse range of applications. Lastly, extending the study of co-part segmentation to 3D makes another meaningful future work.

\section*{Acknowledgements}
\llb{
We thank the anonymous reviewers for their constructive comments. 
This work was supported in part by National Key R$\&$D Program of China (2020AAA0105200, 2019YFF0302900) and Beijing Academy of Artificial Intelligence (BAAI).%
}


\subsection{Model}\label{sec:model}

Given an image $I$, We train a network to predict the segmentation $M$,as in Fig.~\ref{fig:overview}.
The Definition of each module is in the following.

\subsubsection*{Encoder}
Given an image I, the Encoder predicts the feature V and transformation T
\begin{equation}
{\rm{E(I)  = \{ (}}{{ V}_{0}}{,}{{T}_{^{_0}}}{),(}{{ V}_1}{\rm{, }}{{T}_1}{)},...{(}{{ V}_k}{,}{{T}_k}{)}\} \label{encoder}
\end{equation}

we assume V come from latent ${{{V_{lat}}}}$ by transformation matrix 
T, the transformation includes translation $t$,rotation $\theta$ and scale $s$. (now we predict $cos(\theta) $ and $sin(\theta)$
\begin{equation}
{{V  =  T( V_{lat})}}
\end{equation} 
\begin{equation}
{{V_{lat}  =  T^{-1}( V)}}
\label{latent}
\end{equation} 

\subsection{Decoder}
Given a feature ${{ V}_i}$, the decoder predicts the part scalar ${\rm{d}}_i$ that specifies relative inverse depth (i.e larger for foreground objects and image information ${{P}}_i$
\begin{equation}
{\rm{D}}({{ V}_i}{{)  =  (}}{{{d}}_i},{{P}}_i)
\end{equation}
we apply a softmax across the ${{d}}_i$  at each pixel to generate a visibility mask for each part.
\begin{equation} M= soft{\max _k}({{d)}}\end{equation}
the final output $\widetilde I$
\begin{equation}
\widetilde I = \sum\limits_{i = 1}^k {{M_i}{P_i}} 
\end{equation}
\subsection{Training process}
As shown in Fig.~\ref{sturcture_overview}.
Given two images $I^s$ and $I^t$, the encoder predicts $(V^s,T^s)$ and $(V^t,T^t)$ respectively, we  can get  $V^{s}_{lat}$ with $(V^s,T^s)$ by equation \eqref{latent}.
We assume the source and target images image share the same latent feature, so 
\begin{equation}
\widetilde V^t = {T^{^t}}V_{lat}^s
\end{equation}
We employ {\rm{D}} to reconstructs the target frame by transforming the source frame features according to the affine transformation,
so feed $\widetilde V^t$ to decoder ${\rm{D}}$, we can obtain the final predict image
${\widetilde I^t}$,$\widetilde M^t$.
For any input $I^s$, we can obtain the same output ${\widetilde I^t}$, therefore, it forces the network to learn the same latent feature $V^{s}_{lat}$.



\section{Do \emph{not} have an appendix here}

\textbf{\emph{Do not put content after the references.}}
Put anything that you might normally include after the references in a separate
supplementary file.s

We recommend that you build supplementary material in a separate document.
If you must create one PDF and cut it up, please be careful to use a tool that
doesn't alter the margins, and that doesn't aggressively rewrite the PDF file.
pdftk usually works fine. 

\textbf{Please do not use Apple's preview to cut off supplementary material.} In
previous years it has altered margins, and created headaches at the camera-ready
stage. 










\subsection{training on single/multiple videos (R1, R6)}

As we have shown in the paper, our model can be trained with both \textbf{single-video settings} (hand, quadruped, and robot arm) and \textbf{multiple-video settings} (Tai-Chi-HD, VoxCeleb, and Exercise datasets). We will clarify this point in the revision

\input{figures/rebuttal_view}
\subsection{generalization to unseen subjects (R6), multiple subjects and moving backgrounds (R1)}

Models trained with multiple videos can be generalized to images of the same category but with difference appearance. For example, the images shown in Figs.~5 and 6 were not seen by the networks during training. 
In addition, without further training, the models trained on Tai-Chi-HD and VoxCeleb datasets can be applied directly to videos downloaded from YouTube, as shown in the top two rows of Fig.~\ref{fig:rebuttal_generalization}.
Moreover, it is rather straightforward to extend our method to accommodate multiple subjects by increasing $K$.
The bottom row of Fig.~\ref{fig:rebuttal_generalization} demonstrates an example of such ability, where a model is trained on a video with two persons
with $K=17$.

Our method allows moderate background motions as exhibited in Tai-Chi-HD and VoxCeleb datasets during training;
but dramatic background change can interfere the training stage and degrade the performance.
However, in the evaluation stage, our trained model takes single images as input, thus can be applied to videos with moving background.
Extending our method to support training on videos with dramatic background change is a viable future work.




\subsection{trivial solutions (R3, R5), train on image pairs (R1) }


We formulate the training as a retargeting problem between pairs of images, so that the network is forced to extract parts from one image and {separately} transform them to reconstruct the other image. 
Pairs of random frames in a video often contain significant per-part movement, making the motion of the whole object impossible to be reconstructed using a single image-to-image affine transformation.
Therefore, trivial solutions where the whole object is marked as a single part can cause significant image reconstruction loss and will be eliminated during training.


\subsection{optimum $K$ (R5), part semantic annotation (R1)}


We can achieve correct segmentation using an empirical value of $K=15$ in all the test cases discussed in the paper.
In order to make a fair comparison, we deliberately use different $K$ in some of the experiments to ensure consistency with the baseline methods.
Nevertheless, automatically choosing an optimum $K$ for a given video category is indeed a valuable problem in future research. 




Our method is designed to extract parts that exhibit different affine transformations in the training image pairs, which is consistent with the behavior of semantically meaningful segmentation of a subject, such as a human body.
The order of these parts is not determined in the unsupervised learning process. As in previous works, we manually label those parts after the model training. Notably, we only need to label the parts once, and the labels are consistent over all test images.

\subsection{failure cases (R3)}

Due to the lack of temporal information, our method can fail in inferring occluded parts and may incorrectly label the limbs when a person turns around.
We would like to expand the limitation section with these failure cases.


\subsection{loss weights (R3), transformation loss (R5)}

We empirically choose weights to balance the magnitude of each loss term in a preliminary training, except for the image reconstruction loss, which is an order of magnitude larger than the other regularization terms due to its critical role in the training.
The performance of our model is not sensitive to the specific choice of these loss weights,
and similar video categories can share the loss weights.
For the Tai-Chi-HD dataset, we can take advantage of the transformation loss because the transformation of each limb can be clearly defined. Nonetheless, this loss term takes only marginal effect on faces in the VoxCeleb dataset, though we kept this term as a part of a uniform training process. 
We will clarify these points in the paper.



\end{document}




\section{Outline}
	This document contains more implementation details and experimental results that could not be included in the main paper due to space constraints. Note we demonstrate around 50 examples both on IIW/SAW real data and from unseen data source, in order to show the robustness of our algorithm and its superior performance over existing approaches. The remaining content is organized as follows.
	\begin{itemize}
		\itemsep-0.21em
		\item \textbf{Section \ref{sec:implementation_details}:} Implementation details and network structure.
		\item \textbf{Section \ref{sec:refinement_module}:} Illustration and Justification of our image refinement module.
		\item \textbf{Section \ref{sec:network_analysis}:} Feature analysis of the simple two-network solution.
		\item \textbf{Section \ref{sec:dataset_comparison}:} More examples from our proposed dataset, compared with CGIntrinsics \cite{li2018cgintrinsics}  and InteriorNet \cite{li2018interiornet} dataset.
		\item \textbf{Section \ref{sec:results_iiw_saw}:} More visual comparisons on the IIW/SAW dataset.
		\item \textbf{Section \ref{sec:results_outdoor_scene}:} More visual comparisons on the unseen data source.
	\end{itemize}
	
	\clearpage
	
	
	
	
	
	
	\section{Implementation details and network structure} \label{sec:implementation_details}
	
	In this section, we introduce our implementation details and network structure.
	
	\textbf{Implementation Details.} Our framework is implemented in PyTorch framework with mini-batch size of 8. Our network is optimized by Adam. Its initial learning rate is set as 0.01, and decreased to 0.0005 while finetuning IIW/SAW dataset. The learning rate is maintained to be the same as 0.01 while finetuing MIT dataset, since it is more different from our synthetic training data in all aspects than IIW/SAW dataset.
	
	\textbf{Network Structure.}  The intrinsic estimation network consists of 20 successive convolution layers, of which the first three encode the input image into high dimensional deep features, and the last three decode the intermediate features back to output image. All the other convolution layers are arranged into residual blocks \cite{he2016deep} that have been widely used in the deep learning community. Moreover, we deploy dilated convolution to explicitly increase the receptive field of the neural network. The reason for this design is to meet the specific characteristics of intrinsic images, for example, either piece-wise constancy or shadow removal for albedo require consideration of distant pixels. We also downsample the feature maps before the residual blocks by one fourth and upsample them back to the original resolution after the residual blocks to accelerate the training procedure. All the convolution layers share the same 3$\times$3 kernel size, output 64 channels of feature maps, and are followed by instance normalization (IN) and ReLU layers, except for the last one. 
	
	Its detailed network structure is shown in Table \ref{table:estimation_network}, while its corresponding intrinsic decision network $f_{ID}$  is revealed in Table \ref{table:decision_network}. 
	
	\input{table-estimation-network.tex}
	\input{table-decision-network.tex}
	
	\clearpage
	
	
	
	
	
	\section{Illustration and justification of the image refinement module $f_{IR}$} \label{sec:refinement_module}
	
	\subsection{ How is the image refinement module $f_{IR}$ trained and evaluated? }
	In Figure \ref{figure:refinement_module}, we illustrate the difference between the training and testing phase of our proposed image refinement module. In the training stage, we feed our rendered input image ($I$) and an image structure map ($E_R$) calculated from the label ($R^*$), which is \cite{L1smooth}'s processed image of $I$, into the neural network, and generate its output ($R$). In the evaluation stage, we feed the albedo image ($A$) and the albedo structure ($E_A$) predicted from the intrinsic estimation network to the refinement module to generate the final albedo ($A_R$).
	
	As can be seen, the refined image shares the similar image structure with the input structure map, and similar color information with the input image. It learns to refine the input image guided by the input structure map implicitly in the deep neural network, which is dramatically different from any previous guided image filtering process.
	
	\begin{figure*}[htp]
		\centering
		\includegraphics[scale=0.48]{refinement_network.png}
		\vspace{-2mm}
		\caption{Training and evaluation phase of the refinement module $f_{IR}$.}
		\label{figure:refinement_module}
	\end{figure*}
	
	\subsection{Significance of the  image refinement module}
	\cite{fan2018revisiting} integrated a guided domain filter in its pipeline as a post-processed refinement step, which is the closest to ours in spirit. Therefore, in order to justify the effectiveness of our image refinement module, we replace it with the domain filter \cite{fan2018revisiting} and evaluate its performance on the IIW test data, which achieves \textbf{20.90} on the WHDR error metric. It lags behind our \textbf{17.10} by a large margin.
	
	Note their domain filter is supervised by the albedo and jointly trained within the whole framework, while our refinement module doesn't learn any prior of the intrinsic information and is independently trained.
	
	\clearpage
	
	
	
	
	
	\section{Feature analysis of two-network solution} \label{sec:network_analysis}
	
	We further analyze the influence of different feature channels to the intrinsic image prediction by training two fully convolution neural networks, which share the same network structure with our intrinsic estimation network, for albedo and shading prediction separately. We illustrate the activeness of different feature channels in Figure \ref{figure:analysis}, similarly to Figure 6 in the main paper.
	
	We can see that the feature distribution of albedo and shading are dramatically different from each other, which conflicts with the phenomenon observed in our proposed framework. Moreover, by leveraging this two-network solution, its performance lags behind ours in both IIW and SAW dataset as shown in Table 3 of the main paper. This performance difference may come from the special design of our AONet that unifies the important features of albedo and shading together, which potentially benefits the albedo and shading prediction from each other.
	
	\begin{figure*}[htp]
		\centering
		\includegraphics[scale=0.47]{analysis_albedo.pdf}\\
		\includegraphics[scale=0.47]{analysis_shading.pdf}
		\vspace{-2mm}
		\caption{Influence of different feature channels to albedo and shading prediction. The error is computed as the difference between the original network output and the one generated by disabling one feature channel of a single layer.}
		\label{figure:analysis}
	\end{figure*}
	\clearpage
	
	
	
	
	
	\section{More examples in our dataset, CGIntrinsics \cite{li2018cgintrinsics} and InteriorNet \cite{li2018interiornet} dataset}\label{sec:dataset_comparison}
	
	We further demonstrate more examples from our proposed dataset from Figure \ref{figure:our-data1} to \ref{figure:our-data2}, compared with CGIntrinsics in Figure \ref{figure:CGI-data} and InteriorNet in Figure \ref{figure:Interior-data}. As can be seen, our synthetic data features more complex and realistic lighting environment, richer textures and indoor-scene layouts.
	

	

	
	
	\input{figure-our-data1.tex}
	\input{figure-our-data4.tex}
	\input{figure-our-data2.tex}

	\input{figure-CGI-data.tex}

	\input{figure-Interior-data.tex}

	\clearpage
	
	
	
	
	
	
	\section{More qualitative comparisons on the IIW/SAW dataset} \label{sec:results_iiw_saw}
	
	From Figure \ref{figure:iiw_results1} to \ref{figure:iiw_results9}, we demonstrate around 17 groups of visual comparisons with \cite{fan2018revisiting,li2018cgintrinsics} on the IIW/SAW dataset. Our results with ``*'' are generated from the network finetuned on the IIW/SAW dataset.
	
	\input{figure-iiw-results1.tex}
	\input{figure-iiw-results2.tex}
	\input{figure-iiw-results3.tex}

	\input{figure-iiw-results7.tex}


	\input{figure-iiw-results9.tex}
	
	\clearpage
	
	
	
	
	
	\section{More visual comparisons on the unseen data source}\label{sec:results_outdoor_scene}
	
	We demonstrate the intrinsic decompositions on around 30 self-collected real-world images which are unseen in the training data from Figure \ref{figure:real_results1} to \ref{figure:real_results12}. Our results are predicted from the network without finetuning on IIW/SAW real data.
	
	\input{figure-real-results1.tex}
	\input{figure-real-results2.tex}
	\input{figure-real-results3.tex}
	\input{figure-real-results16.tex}

	\input{figure-real-results6.tex}

	\input{figure-real-results11.tex}
	\input{figure-real-results4.tex}
	\input{figure-real-results18.tex}
	\input{figure-real-results19.tex}


	\input{figure-real-results12.tex}
	
	\clearpage
	{
		\bibliographystyle{ieee}
		


\section{Outline}
	This document contains implementation details of our method. The content is organized as follows.
	\begin{itemize}
		\itemsep-0.21em
		\item \textbf{Section \ref{sec:structure}:} Network structure.
		\item \textbf{Section \ref{sec:Trainingprogress}:} Implementation details.
	
	  
	\end{itemize}
	
	
	\section{Network Structure} \label{sec:structure}
	
	As stated in the main paper, our segmentation network consists of two major modules: image encoder $\mathcal{E}$ and segment decoder $\mathcal{D}$. Their structures are shown in Figure~\ref{figure:network}. 
	In this document, we assume the shape of a tensor is in the format of $C\times H \times W$, where $C$ is the number of channels, $H$ and $W$ are the height and the width of the tensor, respectively.
	
	
	\subsection{Image Encoder}
	The image encoder $\mathcal{E}$ includes three parts: feature extractor, feature map estimator, and transformation estimator.

	\textbf{Feature Extractor.}  
	The feature extractor takes an image $I$ of size $(3 \times 128 \times 128)$ as input and computes a feature image $f$ of size $((128\times (K+1)) \times 32\times 32)$, where $(K+1)$ corresponds to $K$ foreground parts and one background part.
	
	The network is constructed based on the standard U-Net architecture~\cite{ronneberger2015unet}, where three cascaded downsampling blocks and the corresponding upsampling blocks are employed. The initial feature count is $512$ and the max feature count is set to $1024$ in our implementation.
	%
	Before being input into the U-Net module, an input image will first pass through three convolutional layers, each followed by a ELU layer and an instance normalization layer, and two max-pooling layers. 
	%
	The output of the U-Net will pass through an additional convolutional layer to compute the final feature image.
	The parameters of these layers can be found in Table~\ref{table:Feature}.
	

	%

	%


	%


   
    
    \begin{figure*}[htp]
		\centering
		\includegraphics[scale=0.48]{supplementary/network.pdf}
		\vspace{-2mm}
		\caption{Two major modules of our network: image encoder $\mathcal{E}$ and segment decoder $\mathcal{D}$. The image encoder  $\mathcal{E}$ is composes of three parts: feature extractor, feature map estimator and transformation estimator.}
		\label{figure:network}
	\end{figure*}
    \input{supplementary/table_feature_part}
	
    \textbf{Feature Map Estimator}. 
    Based on the output feature image $f$ of the feature extractor, the feature map estimator computes $(K+1)$ feature maps $\{V_k\}, k\in\{0,\dots{},K\}$, each of size $10\times32\times32$.
    %
    We employ two convolutional layers in this module, where a leaky ReLU with $0.2$ negative slope is used as the activation function of the first convolutional layer.
    %
    The parameters of these layers are shown in Table~\ref{table:partmap}.
    \input{supplementary/table_partmap}
    
    \textbf{Transformation Estimator}. 
    Similar to the feature map estimator, the transformation estimator also takes the feature image $f$ as input and computes the transformations of every part. 
    %
    The network first convert $f$ into a one-dimensional feature vector using three convolutional layers, then three separate MLPs are employed to estimate the scaling $(s_x, s_y)$, rotation $(s_{\theta},  c_{\theta})$, and translation $(t_x, t_y)$ of the transformation.
    %
   
    Note that we only estimate transformations of the $K$ foreground parts. The transformations of the background part is assumed to be identity.
    Table~\ref{table:transformation} reports the parameters of these layers.
    
   
    
   
   
    
    \input{supplementary/table_transformation}
    
    
    \subsection{Segment Decoder}
    
    \textbf{Segment Decoder}. The segment decoder $\mathcal{D}$ takes a feature map $V_k$ $(10\times32\times32)$ as input, feeds it  through two convolutional  layers, one residual block~\cite{he2016resnet},  and  two upsampling blocks~\cite{johnson2016vggloss}, then finally outputs a tensor with size  $(4\times128\times128)$. The first three channels of this tensor are considered as the part image $P_k$ $(3\times128\times128)$, while the fourth channel is used as the depth map $D_k$ $(1\times128\times128)$. 
    The details are shown in Table ~\ref{table:decoder}.
    \input{supplementary/table_decoder}
    \input{supplementary/table_parameters}
    
    
    \section{Implementation Details}\label{sec:Trainingprogress}
    
   
    
    Our system is implemented in the PyTorch framework. We train our networks using Adam~\cite{adam2014}, and the learning rate is set to $0.00005$.
    We use batch size $6$ for $K=10$ and $4$ for $K=15$. The training is performed on one Titan X GPU with $12$GB memory. 
   
    We terminate the training when the learning progress stalls or exceeds $500,000$ iterations.
    
    In experiment, we set $K$ same to previous work to compare. In practice, the $K$ is not sensitive to the result when $K$ is big enough. The model can automatically select which channels to segment and set redundant channels to be empty. For hand, quadruped, and robot arm, we set the $K$ is $16$, it works well.
    
    We empirically choose weights to balance the magnitude of each loss term in a preliminary training, except for the image reconstruction loss, which is an order of magnitude larger than the other regularization terms due to its critical role in the training.The performance of our model is not sensitive to the specific choice of these loss weights, and similar video categories can share the loss weights.
    
    Table~\ref{table:Parameters} reports the weights of the training losses as described in Section~3.2 of the main paper. Specifically, the vgg loss in the image reconstruction loss is computed in five different resolutions: $128\times 128$, $64\times 64$, $32\times 32$, $16\times 16$, and $8\times 8$. 
    %
    The weights of the corresponding loss terms are set to  $[1.0/32, 1.0/16, 1.0/8, 1.0/4, 1.0]$ as suggested in~\cite{wang2018pix2pixHD}. 
    %
    We use a smaller weight $\lambda_{con}/10$ for the concentration loss computed on canonical parts $M^*_k$. 
    %
    The random transformations used to compute the equivariance loss are generated uniformly in the scaling range $[0.8,1.05]$, the rotation range $[-180 ^\circ{}, 180 ^\circ{}]$, and the translation range $[-0.2,0.2]$.

    
   
    
    


	
    We train separate networks for each dataset used in this work. The two images of an image pair are both randomly selected from the same video clip.
    %
    We augment the training data using random rotations (in the range of $[-15^\circ, 15^\circ]$) and color jittering to increase the robustness of the network, where the same augmentation is applied to both the images of an image pair.
    
   
   
   
    
   
	

	
	We have included some of our results and comparisons to the state-of-the-art approaches in the main paper. For additional results and comparison, please refer to Figure~\ref{figure:vis_taichi1}, Figure~\ref{figure:vis_taichi2}, Figure~\ref{figure:vis_vox1}, Figure~\ref{figure:vis_vox2}, Figure~\ref{figure:exercise1}, and Figure~\ref{figure:exercise2} in this document, as well as the the supplementary video.
	

	\input{supplementary/vis_taichi.tex}
    \input{supplementary/vis_vox.tex}
    \input{supplementary/vis_exercise}
\clearpage
\newpage{}

\section{\blue{**Rebuttal is in the next page}}

\section{Experiments}
\begin{itemize}
    \item generalization: (finished)
    \begin{itemize}
        \item apply TaiChi model to Yoga
        \item apply Yoga model to TaiChi
        \item find a video on Youtube and apply TaiChi/Yoga to it
        \item find a video on Youtube and apply Face model to it
        
        \item (optional) train model with all the videos
    \end{itemize}
    
    \item moving background - training on DAVIS dataset
    \begin{itemize}
        \item train on single video
        \item train on several videos and show generalization
    \end{itemize}
    
    \item multiple object - training on DAVIS dataset/ other videos
    \begin{itemize}
        \item try our model on videos with mulitple characters/faces 
        \item train on single video with two characters (DAVIS or other video)
        \item or train on single video with some other objects
    \end{itemize}
    
    \item ablation study of Transformation loss on Face dataset
    
    \item apply erosion to Motion Co-Part result, possibly showing that the segments becomes apart, or showing that we cannot make the mask consistently tightly.
    
    \item ablation study of K
    
    \item check the order of magnitude of every loss term 
    (around 0.1)
    
\end{itemize}



\clearpage

We thank all the reviewers for taking the time to read our submission and provide positive and constructive comments. Below we first address common concerns raised in the reviews then answer each individual questions.

\section{generalization of the method, training on single/multiple videos (R1, R6)}
\input{figures/rebuttal_view}

Our model can be trained using either a single video or a collection of videos. We only assume that the two images in an image pair have similar appearance, but these image pairs can be collected from multiple videos of different subjects. We have shown in the paper both the results trained with \textbf{single-video settings} (hand, quadruped, and robot arm) and the results with \textbf{multiple-video settings} (Tai-Chi-HD, VoxCeleb, and Exercise datasets).

Models trained with multiple videos can generalize well to images of the same category but with difference appearance. For example, the images shown in Fig.~5 and Fig.~6 of our paper are from the test set of Tai-Chi-HD and VoxCeleb datasets respectively, which were not seen by the networks during training. Without further training, the models trained on these two datasets can be applied to videos downloaded from YouTube, as shown in Fig.~\ref{fig:rebuttal_generalization} above.

\section{trivial solutions (R3, R5), images pairs (R1), part correspondence (R1), optimum $K$ (R5)}
During training, we require the network to reconstruct source image and target image as accurate as possible, the trivial solutions could not get the accurate target image. And if we do not use images pairs, it maybe get trivial solution.

Our method is unsupervised, the segmentation does not necessarily correspond to the human annotated object parts. Like previous works,  we manually select our parts to correspond the ground-truth in Exercise dataset. In Tai-Chi-HD and VoxCeleb datasets, we use two metrics without correspondence.

In experiment, we set $K$ same to previous work to compare. In practice, the $K$ is not sensitive to the result when $K$ is big enough. For hand, quadruped, and robot arm, we set the $K$ is $16$, it works well.

\section{failure cases (R3)}
Our method can not distinguish left and right of human. When person cross  legs or arms, the correspondence is  wrong. 
\section{loss weights (R3), background loss (R5), transformation loss (R5)}
For loss weights setting, we set order of magnitude of every loss term (except reconstruction loss) to be same (0.1) based on experience. 
transformation loss does not have clear effect on VoxCeleb. 
\blue{Background is same to  trivial solutions}

\section{erosion (R1)}

\section{moving background, DAVIS dataset (R1)}
In our method, we assume that the background is static.  In experiment, the slight translation can be handle.  In Tai-Chi-HD and  VoxCeleb datasets, the background is not totally static. But in DAVIS dataset, the background has more change, our method is not suitable for it without supervision.  
\section{multiple objects (R1)}
In our setting, we do not limit the number of objects in the video, it can use the same way to train. 


\section{Electronic Submission}
\label{submission}

Submission to ICML 2021 will be entirely electronic, via a web site
(not email). Information about the submission process and \LaTeX\ templates
are available on the conference web site at:
\begin{center}
\textbf{\texttt{http://icml.cc/}}
\end{center}

The guidelines below will be enforced for initial submissions and
camera-ready copies. Here is a brief summary:
\begin{itemize}
\item Submissions must be in PDF\@.
\item Submitted papers can be up to eight pages long, not including references, plus unlimited space for references. Accepted papers can be up to nine pages long, not including references, to allow authors to address reviewer comments. Any paper exceeding this length will automatically be rejected. 
\item \textbf{Do not include author information or acknowledgements} in your
    initial submission.
\item Your paper should be in \textbf{10 point Times font}.
\item Make sure your PDF file only uses Type-1 fonts.
\item Place figure captions \emph{under} the figure (and omit titles from inside
    the graphic file itself). Place table captions \emph{over} the table.
\item References must include page numbers whenever possible and be as complete
    as possible. Place multiple citations in chronological order.
\item Do not alter the style template; in particular, do not compress the paper
    format by reducing the vertical spaces.
\item Keep your abstract brief and self-contained, one paragraph and roughly
    4--6 sentences. Gross violations will require correction at the
    camera-ready phase. The title should have content words capitalized.
\end{itemize}

\subsection{Submitting Papers}

\textbf{Paper Deadline:} The deadline for paper submission that is
advertised on the conference website is strict. If your full,
anonymized, submission does not reach us on time, it will not be
considered for publication. 

\textbf{Anonymous Submission:} ICML uses double-blind review: no identifying
author information may appear on the title page or in the paper
itself. Section~\ref{author info} gives further details.

\textbf{Simultaneous Submission:} ICML will not accept any paper which,
at the time of submission, is under review for another conference or
has already been published. This policy also applies to papers that
overlap substantially in technical content with conference papers
under review or previously published. ICML submissions must not be
submitted to other conferences and journals during ICML's review
period.
Informal publications, such as technical
reports or papers in workshop proceedings which do not appear in
print, do not fall under these restrictions.

\medskip

Authors must provide their manuscripts in \textbf{PDF} format.
Furthermore, please make sure that files contain only embedded Type-1 fonts
(e.g.,~using the program \texttt{pdffonts} in linux or using
File/DocumentProperties/Fonts in Acrobat). Other fonts (like Type-3)
might come from graphics files imported into the document.

Authors using \textbf{Word} must convert their document to PDF\@. Most
of the latest versions of Word have the facility to do this
automatically. Submissions will not be accepted in Word format or any
format other than PDF\@. Really. We're not joking. Don't send Word.

Those who use \textbf{\LaTeX} should avoid including Type-3 fonts.
Those using \texttt{latex} and \texttt{dvips} may need the following
two commands:

{\footnotesize
\begin{verbatim}
dvips -Ppdf -tletter -G0 -o paper.ps paper.dvi
ps2pdf paper.ps
\end{verbatim}}
It is a zero following the ``-G'', which tells dvips to use
the config.pdf file. Newer \TeX\ distributions don't always need this
option.

Using \texttt{pdflatex} rather than \texttt{latex}, often gives better
results. This program avoids the Type-3 font problem, and supports more
advanced features in the \texttt{microtype} package.

\textbf{Graphics files} should be a reasonable size, and included from
an appropriate format. Use vector formats (.eps/.pdf) for plots,
lossless bitmap formats (.png) for raster graphics with sharp lines, and
jpeg for photo-like images.

The style file uses the \texttt{hyperref} package to make clickable
links in documents. If this causes problems for you, add
\texttt{nohyperref} as one of the options to the \texttt{icml2021}
usepackage statement.


\subsection{Submitting Final Camera-Ready Copy}

The final versions of papers accepted for publication should follow the
same format and naming convention as initial submissions, except that
author information (names and affiliations) should be given. See
Section~\ref{final author} for formatting instructions.

The footnote, ``Preliminary work. Under review by the International
Conference on Machine Learning (ICML). Do not distribute.'' must be
modified to ``\textit{Proceedings of the
$\mathit{38}^{th}$ International Conference on Machine Learning},
Online, PMLR 139, 2021.
Copyright 2021 by the author(s).''

For those using the \textbf{\LaTeX} style file, this change (and others) is
handled automatically by simply changing
$\mathtt{\backslash usepackage\{icml2021\}}$ to
$$\mathtt{\backslash usepackage[accepted]\{icml2021\}}$$
Authors using \textbf{Word} must edit the
footnote on the first page of the document themselves.

Camera-ready copies should have the title of the paper as running head
on each page except the first one. The running title consists of a
single line centered above a horizontal rule which is $1$~point thick.
The running head should be centered, bold and in $9$~point type. The
rule should be $10$~points above the main text. For those using the
\textbf{\LaTeX} style file, the original title is automatically set as running
head using the \texttt{fancyhdr} package which is included in the ICML
2021 style file package. In case that the original title exceeds the
size restrictions, a shorter form can be supplied by using

\verb|\icmltitlerunning{...}|

just before $\mathtt{\backslash begin\{document\}}$.
Authors using \textbf{Word} must edit the header of the document themselves.

\section{Format of the Paper}

All submissions must follow the specified format.

\subsection{Dimensions}




The text of the paper should be formatted in two columns, with an
overall width of 6.75~inches, height of 9.0~inches, and 0.25~inches
between the columns. The left margin should be 0.75~inches and the top
margin 1.0~inch (2.54~cm). The right and bottom margins will depend on
whether you print on US letter or A4 paper, but all final versions
must be produced for US letter size.

The paper body should be set in 10~point type with a vertical spacing
of 11~points. Please use Times typeface throughout the text.

\subsection{Title}

The paper title should be set in 14~point bold type and centered
between two horizontal rules that are 1~point thick, with 1.0~inch
between the top rule and the top edge of the page. Capitalize the
first letter of content words and put the rest of the title in lower
case.

\subsection{Author Information for Submission}
\label{author info}

ICML uses double-blind review, so author information must not appear. If
you are using \LaTeX\/ and the \texttt{icml2021.sty} file, use
\verb+\icmlauthor{...}+ to specify authors and \verb+\icmlaffiliation{...}+ to specify affiliations. (Read the TeX code used to produce this document for an example usage.) The author information
will not be printed unless \texttt{accepted} is passed as an argument to the
style file.
Submissions that include the author information will not
be reviewed.

\subsubsection{Self-Citations}

If you are citing published papers for which you are an author, refer
to yourself in the third person. In particular, do not use phrases
that reveal your identity (e.g., ``in previous work \cite{langley00}, we
have shown \ldots'').

Do not anonymize citations in the reference section. The only exception are manuscripts that are
not yet published (e.g., under submission). If you choose to refer to
such unpublished manuscripts \cite{anonymous}, anonymized copies have
to be submitted
as Supplementary Material via CMT\@. However, keep in mind that an ICML
paper should be self contained and should contain sufficient detail
for the reviewers to evaluate the work. In particular, reviewers are
not required to look at the Supplementary Material when writing their
review.

\subsubsection{Camera-Ready Author Information}
\label{final author}

If a paper is accepted, a final camera-ready copy must be prepared.
For camera-ready papers, author information should start 0.3~inches below the
bottom rule surrounding the title. The authors' names should appear in 10~point
bold type, in a row, separated by white space, and centered. Author names should
not be broken across lines. Unbolded superscripted numbers, starting 1, should
be used to refer to affiliations.

Affiliations should be numbered in the order of appearance. A single footnote
block of text should be used to list all the affiliations. (Academic
affiliations should list Department, University, City, State/Region, Country.
Similarly for industrial affiliations.)

Each distinct affiliations should be listed once. If an author has multiple
affiliations, multiple superscripts should be placed after the name, separated
by thin spaces. If the authors would like to highlight equal contribution by
multiple first authors, those authors should have an asterisk placed after their
name in superscript, and the term ``\textsuperscript{*}Equal contribution"
should be placed in the footnote block ahead of the list of affiliations. A
list of corresponding authors and their emails (in the format Full Name
\textless{}email@domain.com\textgreater{}) can follow the list of affiliations.
Ideally only one or two names should be listed.

A sample file with author names is included in the ICML2021 style file
package. Turn on the \texttt{[accepted]} option to the stylefile to
see the names rendered. All of the guidelines above are implemented
by the \LaTeX\ style file.

\subsection{Abstract}

The paper abstract should begin in the left column, 0.4~inches below the final
address. The heading `Abstract' should be centered, bold, and in 11~point type.
The abstract body should use 10~point type, with a vertical spacing of
11~points, and should be indented 0.25~inches more than normal on left-hand and
right-hand margins. Insert 0.4~inches of blank space after the body. Keep your
abstract brief and self-contained, limiting it to one paragraph and roughly 4--6
sentences. Gross violations will require correction at the camera-ready phase.

\subsection{Partitioning the Text}

You should organize your paper into sections and paragraphs to help
readers place a structure on the material and understand its
contributions.

\subsubsection{Sections and Subsections}

Section headings should be numbered, flush left, and set in 11~pt bold
type with the content words capitalized. Leave 0.25~inches of space
before the heading and 0.15~inches after the heading.

Similarly, subsection headings should be numbered, flush left, and set
in 10~pt bold type with the content words capitalized. Leave
0.2~inches of space before the heading and 0.13~inches afterward.

Finally, subsubsection headings should be numbered, flush left, and
set in 10~pt small caps with the content words capitalized. Leave
0.18~inches of space before the heading and 0.1~inches after the
heading.

Please use no more than three levels of headings.

\subsubsection{Paragraphs and Footnotes}

Within each section or subsection, you should further partition the
paper into paragraphs. Do not indent the first line of a given
paragraph, but insert a blank line between succeeding ones.

You can use footnotes\footnote{Footnotes
should be complete sentences.} to provide readers with additional
information about a topic without interrupting the flow of the paper.
Indicate footnotes with a number in the text where the point is most
relevant. Place the footnote in 9~point type at the bottom of the
column in which it appears. Precede the first footnote in a column
with a horizontal rule of 0.8~inches.\footnote{Multiple footnotes can
appear in each column, in the same order as they appear in the text,
but spread them across columns and pages if possible.}

\begin{figure}[ht]
\vskip 0.2in
\begin{center}
\centerline{\includegraphics[width=\columnwidth]{icml_numpapers}}
\caption{Historical locations and number of accepted papers for International
Machine Learning Conferences (ICML 1993 -- ICML 2008) and International
Workshops on Machine Learning (ML 1988 -- ML 1992). At the time this figure was
produced, the number of accepted papers for ICML 2008 was unknown and instead
estimated.}
\label{icml-historical}
\end{center}
\vskip -0.2in
\end{figure}

\subsection{Figures}

You may want to include figures in the paper to illustrate
your approach and results. Such artwork should be centered,
legible, and separated from the text. Lines should be dark and at
least 0.5~points thick for purposes of reproduction, and text should
not appear on a gray background.

Label all distinct components of each figure. If the figure takes the
form of a graph, then give a name for each axis and include a legend
that briefly describes each curve. Do not include a title inside the
figure; instead, the caption should serve this function.

Number figures sequentially, placing the figure number and caption
\emph{after} the graphics, with at least 0.1~inches of space before
the caption and 0.1~inches after it, as in
Figure~\ref{icml-historical}. The figure caption should be set in
9~point type and centered unless it runs two or more lines, in which
case it should be flush left. You may float figures to the top or
bottom of a column, and you may set wide figures across both columns
(use the environment \texttt{figure*} in \LaTeX). Always place
two-column figures at the top or bottom of the page.

\subsection{Algorithms}

If you are using \LaTeX, please use the ``algorithm'' and ``algorithmic''
environments to format pseudocode. These require
the corresponding stylefiles, algorithm.sty and
algorithmic.sty, which are supplied with this package.
Algorithm~\ref{alg:example} shows an example.

\begin{algorithm}[tb]
   \caption{Bubble Sort}
   \label{alg:example}
\begin{algorithmic}
   \STATE {\bfseries Input:} data $x_i$, size $m$
   \REPEAT
   \STATE Initialize $noChange = true$.
   \FOR{$i=1$ {\bfseries to} $m-1$}
   \IF{$x_i > x_{i+1}$}
   \STATE Swap $x_i$ and $x_{i+1}$
   \STATE $noChange = false$
   \ENDIF
   \ENDFOR
   \UNTIL{$noChange$ is $true$}
\end{algorithmic}
\end{algorithm}

\subsection{Tables}

You may also want to include tables that summarize material. Like
figures, these should be centered, legible, and numbered consecutively.
However, place the title \emph{above} the table with at least
0.1~inches of space before the title and the same after it, as in
Table~\ref{sample-table}. The table title should be set in 9~point
type and centered unless it runs two or more lines, in which case it
should be flush left.


\begin{table}[t]
\caption{Classification accuracies for naive Bayes and flexible
Bayes on various data sets.}
\label{sample-table}
\vskip 0.15in
\begin{center}
\begin{small}
\begin{sc}
\begin{tabular}{lcccr}
\toprule
Data set & Naive & Flexible & Better? \\
\midrule
Breast    & 95.9$\pm$ 0.2& 96.7$\pm$ 0.2& $\surd$ \\
Cleveland & 83.3$\pm$ 0.6& 80.0$\pm$ 0.6& $\times$\\
Glass2    & 61.9$\pm$ 1.4& 83.8$\pm$ 0.7& $\surd$ \\
Credit    & 74.8$\pm$ 0.5& 78.3$\pm$ 0.6&         \\
Horse     & 73.3$\pm$ 0.9& 69.7$\pm$ 1.0& $\times$\\
Meta      & 67.1$\pm$ 0.6& 76.5$\pm$ 0.5& $\surd$ \\
Pima      & 75.1$\pm$ 0.6& 73.9$\pm$ 0.5&         \\
Vehicle   & 44.9$\pm$ 0.6& 61.5$\pm$ 0.4& $\surd$ \\
\bottomrule
\end{tabular}
\end{sc}
\end{small}
\end{center}
\vskip -0.1in
\end{table}

Tables contain textual material, whereas figures contain graphical material.
Specify the contents of each row and column in the table's topmost
row. Again, you may float tables to a column's top or bottom, and set
wide tables across both columns. Place two-column tables at the
top or bottom of the page.

\subsection{Citations and References}

Please use APA reference format regardless of your formatter
or word processor. If you rely on the \LaTeX\/ bibliographic
facility, use \texttt{natbib.sty} and \texttt{icml2021.bst}
included in the style-file package to obtain this format.

Citations within the text should include the authors' last names and
year. If the authors' names are included in the sentence, place only
the year in parentheses, for example when referencing Arthur Samuel's
pioneering work \yrcite{Samuel59}. Otherwise place the entire
reference in parentheses with the authors and year separated by a
comma \cite{Samuel59}. List multiple references separated by
semicolons \cite{kearns89,Samuel59,mitchell80}. Use the `et~al.'
construct only for citations with three or more authors or after
listing all authors to a publication in an earlier reference \cite{MachineLearningI}.

Authors should cite their own work in the third person
in the initial version of their paper submitted for blind review.
Please refer to Section~\ref{author info} for detailed instructions on how to
cite your own papers.

Use an unnumbered first-level section heading for the references, and use a
hanging indent style, with the first line of the reference flush against the
left margin and subsequent lines indented by 10 points. The references at the
end of this document give examples for journal articles \cite{Samuel59},
conference publications \cite{langley00}, book chapters \cite{Newell81}, books
\cite{DudaHart2nd}, edited volumes \cite{MachineLearningI}, technical reports
\cite{mitchell80}, and dissertations \cite{kearns89}.

Alphabetize references by the surnames of the first authors, with
single author entries preceding multiple author entries. Order
references for the same authors by year of publication, with the
earliest first. Make sure that each reference includes all relevant
information (e.g., page numbers).

Please put some effort into making references complete, presentable, and
consistent. If using bibtex, please protect capital letters of names and
abbreviations in titles, for example, use \{B\}ayesian or \{L\}ipschitz
in your .bib file.

\section*{Software and Data}

If a paper is accepted, we strongly encourage the publication of software and data with the
camera-ready version of the paper whenever appropriate. This can be
done by including a URL in the camera-ready copy. However, \textbf{do not}
include URLs that reveal your institution or identity in your
submission for review. Instead, provide an anonymous URL or upload
the material as ``Supplementary Material'' into the CMT reviewing
system. Note that reviewers are not required to look at this material
when writing their review.

\section*{Acknowledgements}

\textbf{Do not} include acknowledgements in the initial version of
the paper submitted for blind review.

If a paper is accepted, the final camera-ready version can (and
probably should) include acknowledgements. In this case, please
place such acknowledgements in an unnumbered section at the
end of the paper. Typically, this will include thanks to reviewers
who gave useful comments, to colleagues who contributed to the ideas,
and to funding agencies and corporate sponsors that provided financial
support.


\nocite{langley00}












\section*{Acknowledgements}

\textbf{Do not} include acknowledgements in the initial version of
the paper submitted for blind review.

If a paper is accepted, the final camera-ready version can (and
probably should) include acknowledgements. In this case, please
place such acknowledgements in an unnumbered section at the
end of the paper. Typically, this will include thanks to reviewers
who gave useful comments, to colleagues who contributed to the ideas,
and to funding agencies and corporate sponsors that provided financial
support.

