
\section{Introduction}
\input{tex/intro.tex}

\section{Related Work}
\input{tex/related_work.tex}


\section{Approach}
\input{tex/approach.tex}

\section{Experiments}
\input{tex/exp.tex}

\section{Conclusion}
\input{tex/conclusion.tex}

\section*{Acknowledgements}
This work was supported in part by the National Natural Science Foundation of China under Grant U1813218, Grant 61822603, Grant U1713214, Grant 61672306, and Grant 61572271. We sincerely thank Tianyu Zhao, Yu Zheng, Kai Zhong for valuable discussions.

{\small
\bibliographystyle{ieee}






















\section{Network architecture}
Our network architecture has two parts, one for embedding the conditional input $r$ to model uncertainty and the other for encoding and decoding input images. Figure \ref{fig::arch_supp} shows the encoder-decoder network architecture. For the embedding part, we set random input $r$ as a 128-dimensional vector. At the training stage and at the initialization of test stage, each dimension is sampled from a Gaussian distribution $N(0,1)$. Then we use two fully-connected layers (with 256 and 2304 output channels respectively), a reshape layer (from 2304 to 24$\times$32$\times$3) and two 3x3 convolutional layers (with 32 and 128 output channels respectively). For the encoder-decoder part, we use the two-branch version proposed in \cite{Fan_2017_CVPR}. This encoder-decoder network consists of two prediction branches, one is used for capturing high level structures and the other learns geometric continuity. 

First, the input image is encoded into an intermediate latent variable $z_i$ and the random input is embedded as $z_r$. Both $z_i$ and $z_r$ have the equal shapes of 24$\times$32$\times$128. Then, channel-wise concatenation is performed on the two encoded latent variables. Finally, the concatenated feature is decoded into the output point cloud.

\section{More Implementation Details}
\subsection{Datasets}
\paragraph{ShapeNet \cite{shapenet2015}} contains 57386 CAD models across 55 different categories. We randomly took 80\% of the objects for training and the rest for testing. For multi-view images rendering, we used the off-the-shelf renderer\footnote{\url{https://github.com/shubhtuls/mvcSnP/tree/master/preprocess/synthetic/rendering}} provided by \cite{Tulsiani_2018_CVPR}. For the groundtruth point clouds, we used the data\footnote{\url{https://github.com/optas/latent_3d_points}} provided by \cite{achlioptas2017learning}. Each point cloud consists of 2048 points uniformly sampled from the mesh on the dataset. We used "chair" for single-category experiments and the 13 popular categories following 3D-R2N2 \cite{choy20163d} for multi-category experiments. 

\paragraph{Stanford Online Products \cite{Song_2016_CVPR}}
is an online repository initially released to accelerate the field of metric learning. It contains automatically downloaded data from \url{https://www.ebay.com}. We used ``chair" and ``sofa" in our experiments for multi-view reconstruction on real world images. 

\subsection{Baseline approaches}
We reproduced several benchmark results of these methods on the datasets with their released code. In this section, we will show some details on these experiments.

\paragraph{3D-R2N2 \cite{choy20163d}}
For the 32$\times$32$\times$32 voxelized groundtruth for 3D-R2N2 \cite{choy20163d}, we directly used the provided voxels from their repositories. Following the paper, we applied two-stage training for 20k and 40k iterations on the training data. For Chamfer Distance computation, we uniformly sampled point clouds on the predicted voxels using their off-the-shelf functions.

\paragraph{PTN \cite{NIPS2016_6206}}
Similar to 3D-R2N2 \cite{choy20163d}, we uniformly sampled point clouds on the predicted voxels to enable comparsion with the groundtruth point clouds.

\paragraph{PSGN \cite{Fan_2017_CVPR}}
For fair comparison, we used the two-branch version of the network architecture described in \cite{Fan_2017_CVPR} with an output of 2048 points. We trained the fully-supervised deterministic model for 100k iterations with an Adam initial learning rate 1e-4. 

\paragraph{Lin \emph{et al.} \cite{lin2017learning}} We followed the two-stage training strategy in their paper. For the depth map rendered from the fixed 8 poses, we used their off-the-shelf released data described at \url{https://github.com/chenhsuanlin/3D-point-cloud-generation}. As they only released data for single-category experiments, we did not reproduce their 13-category results. Note that our input images and the groundtruth shapes are different from theirs (our groundtruth consist of 2048 points for each shape, which differs from their 10k dense point clouds). This made us unable to directly compare our 13-category performance with that reported in their main paper.
\input{table/number.tex}

\subsection{Highly diverse generative model design}
We present details on the highly diverse model used in the final paragraph of Section 4.4 in our main paper. To better demonstrate the positive correlation between the consistency loss and the 3D reconstruction error, we trained a highly diverse conditional generative model on multi-view images. Specifically, we applied diversity constraint on the whole concatenated point clouds at the second stage of the training. We used $\alpha=15.0$ and $\beta=0.5$ in this experiment. Similar to the main experiment, we trained the model for 40,000 iterations using Adam with an initial learning rate 1e-4.

From the Table 6 in the main paper we can infer the positive correlation between the consistency loss and CD at inference stage. Moreover, it is shown that applying the diversity constraint to the single-view predicted point clouds rather than the concatenated results gives much higher performance (3.37 vs. 4.66 for CD). 


\section{More Ablation Studies}

\subsection{Ablation on number of input images}
We conducted experiments with different number of input views. We randomly sample $n$ views and run inference on both single and multiple categories. Results are shown in Table \ref{tab::number}. When we input only one view , the consistency loss is unable to work, so the performance of the conditional model is relatively poor. With more views observed, the performance becomes consistently better.

\subsection{Runtime analysis}
We did not use any type of connectivity on the view-based sampling layer. On 2048 (10k) points, our layer, which takes 6.6ms (10.2ms) on average, is an $O(n)$ approximation with $10.4\%$ ($3.6\%$) hidden parts included. The accurate mesh-based sampling is at least $O(nlgn)$ with a large constant. Generating a triangle mesh from 2048 (10k) points already takes 209.8ms (1.12s), which becomes the speed bottleneck at both training and inference. 

Note that in some cases the current system suffers from the problem of empty faces. It could be due to that the current view-based sampling is an approximation form which might wrongly samples the points from the back part. This will get the wrongly sampled points to be closer to the front, resulting in unbalanced density. Developing efficient usage of connectivity to better approximate the view-based sampling process might help resolve this issue.


\section{More Qualitative Results}
In this section, we show more samples of qualitative results on single-view conditional predictions and multi-view reconstruction.


\clearpage
\input{fig/supp_vis_1.tex}

\clearpage
\input{fig/supp_vis_2.tex}

\subsection{Overview}
The problem of single-view shape reconstruction was conventionally formulated as a one-to-one mapping $\phi: I \rightarrow S$, where $I$ denotes the input RGB image and $S$ denotes the predicted shape. This one-to-one generative model was widely used to output either voxels \cite{choy20163d} or point clouds \cite{Fan_2017_CVPR} via cross entropy loss and differentiable distance metrics. Most existing methods took the implicit assumption that the input image is sufficient to predict the whole shape structure. 

Consider the probabilistic model $p(S|I)$, where $S$ is a random shape conditioned on the input image $I$. In perfect conditions where useful knowledge is completely learned from the groundtruth shape, the existing deterministic architecture $\phi: I \rightarrow S$ can learn the most probable shape $S^*$, where

\begin{equation}
\phi (I) = S^* = E_S[p(S|I)]
\end{equation}

Most existing single-view reconstruction methods utilized this deterministic formulation and could generalize relatively well to the test set. This is probably due to the fact that most objects in the widely used ShapeNet \cite{shapenet2015} dataset have a symmetric or category-specific structure which enables reasonable inference. However, this is arguably not true especially in complex scenarios. In fact, the structure of the occluded back part is usually relatively ambiguous. To better model this inherent ambiguity, we introduce a conditional generative model $f: I\times r \rightarrow S$, where the image-based generation is conditioned on a Gaussian input vector $r$. We aim to learn a mapping to approximate the probabilistic model $p(S|I)$ in the reasonable shape space. 

However, different from the scenarios of generation in CGAN \cite{mirza2014conditional}, we only have limited groundtruth (in fact, only one shape per image) which cannot span the reasonable shape space. Motivated by the fact that the front part can be sufficiently inferred from the input image while the back part is relatively ambiguous, we propose to train the conditional generative model in a partially supervised manner. Furthermore, we introduce a diversity constraint to help span the reasonable space. 

Let us take a step further into a multi-view scenario, where the problem comes to a many-to-one mapping. Considering only three input views for simplicity here, we have $I_1\times I_2\times I_3 \rightarrow S$. As most parts of the object are covered by different input views, we assume that multi-view reconstruction can be viewed as a deterministic inference with sufficient information. Given a conditional generative model $S=f(I,r)$, the output $S^*$ follow the constraint:

\begin{equation}
\label{eq::consis}
S^*=f(I_1,r_1)=f(I_2,r_2)=...=f(I_n,r_n)
\end{equation}

From the Bayesian perspective, Single-view and multi-view reconstruction can be formulated as to approximate $p(S|I_i)$ and $p(S|\{I_i\}_{i=1}^n)$. In this paper,
the idea is to first implicitly approximate $p(S|I_i)$ with a conditional model and then apply it to deterministic multi-view synthesis, which differs from conventional RNN-based methods \cite{choy20163d,NIPS2017_6640}. This choice has two main reasons. 1) The data scale is limited and only one 3D groundtruth exists for every image, making it not easy to explicitly parameterize $p(S|I_i)$. 2) For a 3D shape, rendered images from different views is correlated. Thus, it is relatively intractable to formulate $p(S|\{I_i\}_{i=1}^n)$ with $p(S|I_i)$ and directly optimize maximum likelihood. Same problem exists in many other research directions eg. multi-view pose estimation. We propose an alternate conceptual idea to get intersections of manifolds conditioned on different views with assistance of pair-wise distance minimization. Figure \ref{fig::pipeline} summarizes an overview of our proposed approach.

\subsection{Modeling the Uncertainty for Single-view Reconstruction}
\label{sec::uncertainty}

Given a specific architecture on a conditional generative model $S=f(I,r)$, we aim to learn the ambiguity of single-view reconstruction from limited groundtruth data. In this section, we first briefly review differentiable distance metric in the shape space, then introduce two of our proposed differentiable constraints to help learn the conditional model.

\paragraph{Distance Metric:}
Two existing differentiable distance metrics between point sets were originally used in \cite{Fan_2017_CVPR}. These metrics are Chamfer Distance (CD) and Earth Mover's distance (EMD) \cite{rubner2000earth}. CD finds the nearest neighbor and is formulated as below\footnote[3]{We use the first-order version of Chamfer Distance following \cite{lin2017learning}.} in Eq.(\ref{eq::chamfer}), while EMD learns a optimal transport between two point sets in Eq.(\ref{eq::emd}). 

\begin{equation}
\label{eq::chamfer}
d_{CD}(S_1, S_2)=\sum_{x \in S_1}{\min\limits_{y \in S_2} \Vert x-y \Vert_2}+\sum_{x \in S_2}{\min\limits_{y \in S_1} \Vert x-y \Vert_2}
\end{equation}

\begin{equation}
\label{eq::emd}
d_{EMD}(S_1, S_2)=\min\limits_{\phi: S_1 \rightarrow S_2} \sum_{x \in S_1} \Vert x-\phi (x) \Vert_2^2
\end{equation}

We use both of these two metrics in our training pipeline. Following \cite{lin2017learning}, the two terms in Chamfer Distance were jointly reported as pred$\rightarrow$GT and GT$\rightarrow$pred at test stage. 

\paragraph{Front Constraint:}
We propose a \emph{front constraint} along with a new differentiable operation: \emph{view based sampling}, which enables the conditional model to learn in a partially supervised manner. Different from recently proposed point cloud downsampling strategies \cite{Li_CVPR_2018, NIPS2017_7095} which aims to uncover inner relationship for coarse-to-fine understanding, our proposed \emph{view based sampling} layer outputs a set of points which consist the front part of the shape from a specific view.

The overview of the \emph{front constraint} is shown in Figure \ref{fig:sampling}. In the proposed approach, we get the generative model to focus more on the front $N_1$ points, while the $N-N_1$ remaining points are conditionally generated. For the \emph{view based sampling} layer, we first render the point cloud onto a 2D depth map with the intrinsic and extrinsic parameters given. Then, we sample all of the points which contribute to the rendered map. This strategy enables that all sampled points are on the front side of the object from the view. Note that because pixel-wise loss on the depth map used in \cite{lin2017learning} is only differentiable on the rendered $z$ axis, it will not work in our single-view training scenario (See Section \ref{sec::ablation}).

By either applying view based sampling to the groundtruth point cloud or applying inverse-projection to the pre-rendered depth map, we can get the groundtruth front part. Then, CD or EMD \cite{rubner2000earth} can be used to acquire the loss of the front constraint $loss_{front}$ and differentiably guide the sampled point cloud.

\paragraph{Diversity Constraint:}
Because for one input image, only one groundtruth shape is available at the training stage, simply training the conditional generative model with groundtruth constraints will hardly get the model to span the reasonable shape space. For different input vectors $r$, we aim to get different predictions which all satisfy the front constraint. With the hinge loss as in the widely used Triplet Loss \cite{schroff2015facenet} in face verification, we propose a \emph{diversity constraint} which uses the Euclidean distance of input $r$ as the distance margin in 3D space. 

Specifically, considering paired input vectors $r_1$, $r_2$ for a single training image $I$, we have two predicted point clouds $S_1=f(I, r_1)$ and $S_2=f(I, r_2)$. The loss of the \emph{diversity constraint} is formulated as in Eq.(\ref{eq::div}) below. 

\begin{equation}
loss_{div}=\max (0, \ \Vert r_1-r_2 \Vert_2 - \alpha EMD(S_1, S_2))
\label{eq::div}
\end{equation}

Because the counts of both point clouds are equal, we use EMD \cite{rubner2000earth} to measure the distance between $S_1$ and $S_2$. The hyper-parameter $\alpha$ helps control the diversity of the predicted point clouds.

\input{fig/partial-supervision.tex}

\paragraph{Latent Space Discriminator:}
Combining the front constraint and the diversity constraint forms a initial paradigm for modeling the ambiguity of the single-view reconstruction. However, this paradigm puts little pressure on the back part. Thus, motivated by the generative adversarial networks \cite{goodfellow2014explaining} and recently proposed representation learning method \cite{achlioptas2017learning} on point clouds, we propose to add a latent space discriminator to better learn the shape priors. Specifically, we first train an auto-encoder on the point cloud domain. Then, we transfer the decoder to the end of our architecture and get it fixed. Finally, we apply WGAN-GP \cite{WGAN-GP} on the top of the latent space. Take $E_S$ as the encoder from the point cloud domain to the latent space, and $E_I$ as the encoder we use from the input image $I_i$ and the random noise $r_i$ to the latent variable $z_i$, the loss is formulated as below, where $S$ is the sampled point cloud from the dataset. 

\begin{align}
    \label{eq::check_gan}
    loss_{gan}&= -\mathbb{E}_{I_i \sim p_{data}, r_i\sim p(r)}[D(E_I(I_i, r_i))] \nonumber\\& +\mathbb{E}_{S \sim p_{data}}[D(E_S(S))] \\&- \lambda\mathbb{E}_{\hat{z} \sim p_{\hat{z}}}[({||\nabla _{\hat{z}}D(\hat{z})||}_2-1)^2]\nonumber
\end{align}

\paragraph{Training on Single-view Images:}
As discussed, we can train a conditional generative model using single input image with the optimization objective in Eq.(\ref{eq::loss}) at the training stage. $\beta$, $\gamma$ denotes the relative loss weight of the diversity loss and the GAN loss respectively.

\begin{equation}
\label{eq::loss}
loss=loss_{front}+\beta loss_{div} + \gamma loss_{gan}
\end{equation}
The training is performed in an iterative min-max manner as the widely-used GAN training strategy. The hyper-parameter $\alpha$ and $\beta$ modulates how far the generative model goes beyond the observed groundtruth. 

\subsection{Synthesizing Multi-view Predictions}
\label{sec::integration}

\paragraph{Finetuning on Multi-view Images:}
To get the network to learn more clues on the high level structure of the object, we finetune the single-view pretrained model on multi-view conditions. For synthesizing the multi-view point clouds at the training stage, we simply concatenate predicted point clouds from different views. Then, $loss_{front}$ was computed in different views on the concatenated results and $loss_{div}$ was computed in different random inputs to guide the training process. Specifically, for each shape, 8 views and 5 random inputs for each is used to train the model. Similar to the single-view training stage, we use the combined $loss$ in Eq.(\ref{eq::loss}) as our minimization objective.

\paragraph{Inference:}
As shown in Eq.(\ref{eq::consis}), from the deterministic perspective the multi-view reconstruction can be viewed as taking the intersection of the reasonable shape space conditioned on each input image. Thus, we propose a consistency constraint directly on the shape level. Consider a set of results $\{S_i\}_{i=1}^n$ from $n$ different views, where $S_i = f(I_i, r_i)$. The consistency loss is formulated in Eq.(\ref{eq::loss_consis}).

\begin{equation}
\label{eq::loss_consis}
loss_{consis}=\frac{2}{n(n-1)}\sum_{i=1}^{n-1}\sum_{j=i+1}^n CD(S_i, S_j)
\end{equation}

Figure \ref{fig:fgsm} shows our inference method. The method of freezing the inference model and adjusting the input is popular in the field of adversarial attacks \cite{goodfellow2014explaining}. By online minimizing $loss_{consis}$ with respect to the input vectors $\{r_i\}_{i=1}^n$ in the conditional model, we get more consistent results. To prevent the optimization from local minimum, we use heuristic search in the $\{r_i\}_{i=1}^n$ initialization. \textbf{Algorithm \ref{alg::inference}} shows our detailed inference pipeline. Our method does not require camera calibration at inference. 

\subsection{Experimental Settings}
\paragraph{Network Architecture:}
Figure \ref{fig::arch} briefly shows our network architecture. For the encoder-decoder branch, we used the two-branch version of the point set generation network in \cite{Fan_2017_CVPR}. We set random input $r$ as a 128-dimensional vector. The embedding branch employs a structure with two fully connected layers and two convolutional layers. Channel-wise concatenation is performed on the embedded vector $z_r$ and the encoded features $z_i$. For more details on the two-branch network in \cite{Fan_2017_CVPR}, refer to our supplementary material.

\input{alg/alg_inference.tex}
\input{fig/fig_fgsm.tex}
\input{fig/arch.tex}

\vspace{-8pt}
\paragraph{Implementation Details:}
We trained our conditional generative network for two stages on a GTX 1080 GPU. The input images were rendered from ShapeNetCore.v1 \cite{shapenet2015} with the toolkit provided by \cite{Tulsiani_2018_CVPR}. To cover the entire object, we uniformly sampled the rendered views along the horizontal circle with a random longitudinal perturbation. We took 80\% of the data for training and the rest for testing. We used $\lambda = 10$ and $\gamma = 0.1$ for adversarial learning. At the first training stage, we trained the model using single-view images for 40,000 iterations with a batch size 16 and 5 random inputs for each image. $\alpha_1 = 0.2$, $\beta_1 = 10.0$. Then, we finetuned our model for 100,000 iterations on multi-view images. There were 2 shapes in each batch, 8 views for each shape, and 5 random inputs for each view. $\alpha_2 = 0.1$, $\beta_2 = 1.0$. We used Adam with an initial learning rate 1e-4 in both stages. At test stage, we used 8 views to reconstruct the point clouds. The range of the longitudinal perturbation is a degree of $[-20,40]$. Following \cite{lin2017learning}, we scaled the reconstruction error CD by a factor of 100. Code is made available\footnote{https://github.com/weiyithu/OptimizeMVS}.

\input{fig/vis_conditional.tex}
\input{table/uncertainty.tex}

\subsection{Multiple Predictions on a Single Image}
Our generative model is able to predict multiple plausible shapes conditioned on the random input $r$. As discussed in Section \ref{sec::uncertainty}, the front constraint guides the generation of the front part, and the diversity constraint enables the conditional model to span the shape space. 

\paragraph{Qualitative Visualization:}
Figure \ref{fig::vis_uncertainty} visualizes the multiple predictions on a single input image conditioned on randomly sampled $r$. It is shown that our conditional model generates plausible shapes with a large diversity. The front part from the view of the input RGB image is predicted in a relatively more deterministic manner while the back part is mainly controlled by the random input $r$. 

\paragraph{Evaluation on Uncertainty Modeling:}
We conducted experiments to better verify the generating diversity of the proposed conditional generative model. We took one single image $I$ and randomly sampled 10 inputs $\{r_i\}_{i=1}^{10}$. Then, we fed the model with $S_i=f(I,r_i)$ and computed $loss_{consis}$ in Eq.(\ref{eq::consis}) on the predicted shape set $\{S_i\}_{i=1}^{10}$. We re-implemented the fully-supervised MoN method in \cite{Fan_2017_CVPR}. For fair comparison, we used the conditional model after the first training stage in this experiment. Table \ref{tab::uncertainty} shows the results. The partial supervision boosts the diversity of the predicted shapes. Moreover, it is demonstrated that when the loss weight $\beta$ of the diversity loss rises, the generating diversity gets consistent increase.

\input{table/sing_cat.tex}
\subsection{Multi-view Shape Reconstruction}
\input{fig/fig_vis.tex}

\paragraph{Evaluation Metric:}
The most widely used metric on evaluating point cloud generation is the Chamfer Distance in Eq.(\ref{eq::chamfer}). For comparison, we use the same protocol with \cite{lin2017learning}. However, it is worth noting that CD computation under different numbered point clouds is relatively confusing. Thus, we also report FPS-CD where we used farthest point sampling \cite{NIPS2017_7095} to get same-numbered point clouds.

\paragraph{Single-category Experiments:}
In this experiment, we applied our conditional generative model on the task of single-category multi-view shape reconstruction on ShapeNet \cite{shapenet2015} "chairs". We re-implemented several widely used image-based reconstruction methods including 3D-R2N2 (5 views) \cite{choy20163d}, PTN \cite{NIPS2016_6206}, PSGN \cite{Fan_2017_CVPR} and Lin \emph{et al.} \cite{lin2017learning} on our synthetic dataset. We converted the voxels predicted by \cite{choy20163d, NIPS2016_6206} to point clouds in the experiment. For the groundtruth point clouds, we used the uniformly sampled point clouds directly from \cite{achlioptas2017learning}. Note that our idea is also complementary to voxel-based deterministic methods (eg. MarrNet \cite{NIPS2017_6657}), where metrics can be developed on voxel space and back-propagation of cross-entropy loss is performed only from the front. Here we use PSGN \cite{Fan_2017_CVPR} with the point cloud outputs for direct comparison. Table \ref{tab::single-cat} shows the experimental results. It is reported that although our conditional generative method is not only partially supervised but also without explicit 3D supervision at training stage, our approach outperforms all baseline methods.
\input{table/multi_cat.tex}
\input{table/noinput.tex}

\paragraph{Multi-category Experiments:}
We tested our model in multi-category experiments following \cite{choy20163d} on 13 popular categories on ShapeNet \cite{shapenet2015} dataset. As shown in Table \ref{tab::multicat}, our proposed method outperforms two baseline methods 3D-R2N2 \cite{choy20163d} and PSGN \cite{Fan_2017_CVPR} by a relatively large margin. 

\paragraph{Qualitative Results:}
For qualitative analysis, in Figure \ref{fig::vis} we visualize the predicted shapes for two state-of-the-art baseline methods: 3D-R2N2 \cite{choy20163d} and PSGN \cite{Fan_2017_CVPR}. It is shown that our partially supervised conditional generative model can infer reasonable shapes which are dense and accurate. More details are generated due to the specific aim on the front parts of the objects.

\subsection{Ablation Studies}
\label{sec::ablation}

\paragraph{Conditional vs. Deterministic:}
To demonstrate the effectiveness of the conditional model, we implemented a deterministic model $S=f_d(I)$. For fair comparison, we used an encoder-decoder structure similar with our network and trained the deterministic model for two stages with the front constraint. Single-category experiment was conducted on the deterministic model. Table \ref{tab::noinput} shows the results. Although the shape in ShapeNet \cite{shapenet2015} dataset often has symmetric structure, the conditional generative model outperforms the deterministic counterpart by 0.25 on CD. 

\input{table/ablation.tex}

\paragraph{Analysis on different features in the framework:}
We performed ablation analysis on three different features: two-stage training, diversity constraint at multi-view training stage and consistency loss during inference. As shown in Table \ref{tab::ablation}, all features achieve consistent gain on the final performance.

\paragraph{Front constraint vs. Projection loss:}
Our conditional model can be trained on single-view images with the front constraint and the diversity constraints. For comparison, we directly applied the projection loss used on multi-view  images training in \cite{lin2017learning} on single-view images, the training did not converge. Because the pixel-wise loss on the depth map suffers from non-differentiable quantization in the render process, the projection loss can only get gradients from the rendering axis. Our \emph{view based sampling} enables valid gradients on the x and y axis from the view. 

\input{fig/fig_real.tex}
\input{fig/exp_consistency.tex}

\paragraph{Correlation between consistency loss and reconstruction error:}
In this part, we study the positive correlation between $loss_{consis}$ and the reconstruction error CD. First, we sampled $loss_{consis}$ and CD simultaneously at test stage. As shown in Figure \ref{fig::consis}, these two metrics show strong patterns of positive correlation. We further demonstrate this consistency on a highly diverse model (refer to our supplementary material for details). Table \ref{tab::consis} shows the experimental results. Minimizing the consistency loss gives consistent decrease on the CD metric with respect to the groundtruth shape. This demonstrates the fact that with $loss_{consis}$ as the mutual constraints inside the framework, the model will infer a more accurate shape at test stage. This verifies our interpretation on the success of applying the conditional generative model to the task of multi-view shape reconstruction.

\subsection{Reconstructing Real World Images}
The idea of multi-view reconstruction with our conditional generative model has great generalization ability. We conducted experiments on \emph{Stanford Online Products} dataset \cite{Song_2016_CVPR} for reconstructing real world images. Figure \ref{fig::real} visualizes our predictions. Our model generates surprisingly reasonable shapes by observing multi-view images in real world scenarios.

To further demonstrate the necessity of conditional modeling, Figure \ref{fig:unsymmetric} shows visual results on unsymmetric real data sampled from \cite{Song_2016_CVPR}. While \cite{Fan_2017_CVPR} sticks to the symmetry prior and fails to generalize, our model generates a much realistic prediction.

\input{fig/unsymmetric.tex}
\input{table/consis.tex}


