


















\section{#1}}

\newcommand{\mathrm{Var}}{\mathrm{Var}}

\usepackage[width=122mm,left=12mm,paperwidth=146mm,height=193mm,top=12mm,paperheight=217mm]{geometry}


\begin{document}

\pagestyle{headings}
\mainmatter
\def\ECCV18SubNumber{467} 

\title{COCO-FUNIT:  \\ Few-Shot Unsupervised Image Translation with \\
a Content Conditioned Style Encoder}


\titlerunning{COCO-FUNIT}

\authorrunning{Saito et al.}

\author{Kuniaki Saito$^{1,2}$ \and
Kate Saenko$^{1}$\and
Ming-Yu Liu$^{2}$}
\authorrunning{Saito et al.}
\institute{Boston University$^{1}$ \hspace{0.5in} NVIDIA$^{2}$ \\
\email{\{keisaito,saenko\}@bu.edu}, \ \email{mingyul@nvidia.com}}
\maketitle


\input{introduction.tex}
\input{related.tex}
\input{method.tex}
\input{experiments.tex}
\input{conclusion.tex}
\input{acknowledgement}
\bibliographystyle{splncs}





\section{Learning Objective Function}\label{sec::learning}

We train our model using a similar objective function as in the FUNIT\xspace work. Below, we first describe the individual objective terms and then present the overall optimization problem. Note that we do not utilize the gradient penalty term~\cite{mescheder2018training} used in the FUNIT\xspace work.

\mysubsec{Adversarial loss}.
$\mathcal{L}_{\text{GAN}}(D,G)$ denotes class conditional GAN loss. We use the loss to ensure both photorealism and domain-faithfulness of image translation. Following the projection discriminator design~\cite{miyato2018cgans}, we use the hinge version of the adversarial loss. 

\mysubsec{Reconstruction loss}. 
The loss encourages the model to reconstruct images when both the content and the style are from the same domain. This loss helps regularize the learning and is given by 
\begin{equation}
\mathcal{L}_{\text{R}} (G)= E_{x_c}[||x_c - \bar{\bm{x_c}}||_1], 
\end{equation}
where $\bar{\bm{x_c}} = G (x_c, x_c)$. 

\mysubsec{Discriminator Feature matching loss}.
Minimizing the feature distance between real and fake samples in the discriminator feature space can stabilize the training of adversarial learning and contribute to the performance of a model as used in the FUNIT\xspace work. We take the feature before the last linear layer and performed spatial average pooling. Let the feature computing function as $D_f$. The feature matching loss is given by 
\begin{equation}
\mathcal{L}_{\text{FM}} (G) =  E_{x_s, x_c}[||D_f(x_s) - D_f(\bar{\bm{x_s}})||_1], 
\end{equation}
where $\bar{\bm{x_s}} = G (x_c, x_s)$. 

Overall our training objective is given by, 
\begin{align}
\min_{D}\max_{G} \mathcal{L}_{\text{GAN}}(D,G) + 
\lambda_{\text{R}} \mathcal{L}_{\text{R}}(G) + \lambda_{\text{F}} \mathcal{L}_{\text{FM}}(G), \label{eqn::learning} 
\end{align}
where $\lambda_{\text{R}}$ and $\lambda_{\text{F}}$ denote trade-off parameters for two losses. We set $\lambda_{\text{R}}$ 0.1 and $\lambda_{\text{F}}$ 1.0 in all experiments.



\mysec{Performance Metrics}\label{sec::metrics}

Here, we describe the details of the evaluation metrics that we use to measure style faithfulness, content preservation, and human preference of the translation outputs.

\mysubsec{Style faithfulness.} We use the Frechet Inception Distance (FID)~\cite{heusel2017gans} to measure distance between the distribution of the translated images and the distribution of real unseen images, based on the InceptionV3~\cite{szegedy2016rethinking} network. We compute FID for each of the target class and report their mean (mFID).

\mysubsec{Content preservation.} An ideal translation should keep the structure of input content image unchanged. We measure the content preservation by comparing the body-part segmentation mask of a content image and that of a translated image. Since we do not have ground-truth body-part annotations, we estimate the body-part segmentation masks by using a DeeplabV3~\cite{chen2017rethinking} network trained on the Pascal body part dataset~\cite{everingham2015pascal}. We note similar approaches are used in the other image synthesis prior  works~\cite{wang2018high,park2019semantic,wang2018video,wang2019few}. We obtain the body-part segmentation masks for both content and translated images. Then, we calculate pixel accuracy (PAcc) and mean intersection-over-union (mIoU) by treating the mask of the content image as the ground-truth annotation.

\mysubsec{Human preference.} Using Amazon Mechanical Turk (AMT), we perform a subjective visual test to gauge the quality of few-shot translation results. We conduct two studies: content preservation and style faithfulness. In the first study, each question contains the content image and the translation results from two competing methods, and the AMT worker is asked to choose which translation result better preserves the pose content from the content image. In the second study, each question contains the style image and the translation results from two competing methods, and the AMT worker is asked to choose which translation result rendering an object resembles more to the one in the style image. We generate 1000 questions per dataset per study. Each question is answered by 3 different AMT workers with a high approval rating. We use the preference score for evaluation. These are the counterpart of the automatic evaluation metrics described above.




