\section{Introduction}

Exemplar-based image translation is a subdivision of image translation, which aims to generate a image with exemplar image style, while maintaining the content information of the input image. Such methods have a wide range of promising applications, such as style transfer, face editing, scene transformation, etc. 



Previous extensive studies \cite{ma2018exemplar,wang2019example,huang2017arbitrary,park2019semantic,zhang2020cross,zhou2021cocosnet,zhan2021unbalanced} have made great progress in exemplar-based image translation. However, current methods mostly focus on learning the correspondence between the two input domains and ignore the potentially useful information interactions within each domain. In this paper, we propose a more general approach that combines the two domain features of the inputs into a whole, and then uses a transformer-based model to learn both the inter-domain correspondence and the information interactions within each domain. 

Recently, a number of transformer-based models \cite{dosovitskiy2020vit,carion2020end,wang2020axial,liu2021swin} have been applied to vision tasks. These works outperform CNN-based methods in their respective fields. Nevertheless, training a pure transformer model is time-consuming and highly dependent on the amount of data. Therefore, we utilize a hybrid structure of CNN and transformer to generate better results with limited amount of data. We first use CNN-based encoders to obtain features of the input image and the exemplar image and concatenate them into a whole. Then we propose a novel Cross-domain Feature Fusion Transformer (CFFT) to fuse the whole feature. The CFFT consists of a Feature Fusion Network and a Hierarchical Transformer Network. The former is used for feature fusion and the latter for learning fine-grained spatial alignment at multiple scales. By the improved transformer-based network, CFFT is able to continuously learn self-attention and cross-attention between two input features, so that the model can learn better inter-domain correspondence and potentially beneficial correlation information within each domain.







Based on the CFFT model, we propose CFFT-GAN, a general exemplar-based image translation method. It decouples and fuses features from different domains and leverages generative adversarial network to learn image translation. In particular, we employ encoders to extract features from different image domains and perform feature fusion through CFFT. Then, the fused feature is fed into the spatially-adaptive generator to synthesize the translation results. In addition, our CFFT model is a pluggable design, and we can achieve multi-domain image translation by cascading CFFT modules. 

We validate the different types of exemplar-based image translation capabilities of our method on many high-resolution image datasets, including pose-to-image, edge-to-image, sketch-to-image, and mask-to-image. These adequate and diverse experiments have demonstrated the validity of our model and show that our approach has better performance than state-of-the-art methods. Ablation studies and feature visualization results indicate the importance of our proposed CFFT module. Extended multi-domain experiments reflect the potential of our approach.

Our contributions can be summarized as follows:
\begin{itemize}
   
    
    \item We present a general exemplar-based image translation method, CFFT-GAN, which using Cross-domain Feature Fusion Transformer module to learn both intra-domain and inter-domain correlation for more effective image translation.
    
    \item We propose a Hierarchical Transformer (Hiformer) network in CFFT for fine-grained spatial alignment at multiple scales.
   
    \item Benefiting from the pluggable nature of CFFT, we utilize cascaded CFFT modules to achieve multi-domain image translation.
   
    \item Rich validation and comparison experiments show that CFFT-GAN can achieve remarkable results in various image translation tasks and also demonstrate the superiority of our approach over SOTA methods.
    
   
     
   
    
   
    
   
    
\end{itemize}



\begin{figure*}[t]
\centering
\includegraphics[width=0.96\linewidth]{model.pdf}
\caption{Overall of our CFFT-GAN model for exemplar-based image translation. Two Encoders are used to extract different domain features. The CFFT module first concatenates different domain features as a whole feature, and performs feature fusion by FFN, and then learns fine-grained spatial alignment of the feature by Hiformer network. Finally, the output feature from CFFT is input to the spatially-adaptive generator to synthesize translation result.}

\label{fig:model}
\end{figure*}


\section{Related Work}

\subsection{Image Translation}
Image translation aims to learn the transformation between different image domains by training a conditional generative adversarial network. Some methods \cite{isola2017image,wang2018high,park2019semantic} use paired data for explicit supervision, while others \cite{zhu2017unpaired,choi2018stargan,choi2020stargan,yi2017dualgan,park2020contrastive} implement unsupervised training without paired data. Our method works well under both conditions. In addition image translation has been used in a variety of scenarios, such as image editing \cite{yu2018generative,wu2020cascade,shen2020interpreting}, domain adaptation \cite{liu2021source,li2020model}, pose transfer \cite{siarohin2018deformable,li2019dense,ren2020deep,ma2021must}. Recently, 
exemplar-based image translation \cite{ma2018exemplar,huang2018multimodal,wang2019example,saito2020coco,zhang2020cross,zhou2021cocosnet,zhan2021unbalanced} has received more and more attention because of its flexibility and applicability. Usually exemplar-based image translation refers to the transfer of stylistic features from a exemplar image to some type of label image, such as edges \cite{zhu2017toward,lee2018diverse}, key points \cite{ma2017pose,men2020controllable,ma2021must}, segmentation masks \cite{wang2018high,park2019semantic}, etc. Some approaches \cite{huang2018multimodal,men2020controllable,ma2021must} use AdaIN \cite{huang2017arbitrary} to transfer style information from the exemplar image into the label image. More recently, CoCosNet v1 \cite{zhang2020cross} uses correlation matrices to learn cross-domain correspondences. UNITE \cite{zhan2021unbalanced} introduces optimal transport for proper
feature alignment and faithful style control in image translation. CoCosNet v2 \cite{zhou2021cocosnet} learn a full-resolution correspondence with a hierarchical GRU-assisted PatchMatch \cite{barnes2009patchmatch}. In this paper, we propose a new method using the transformer-based network to learn cross-domain feature fusion. 



\subsection{Transformer}


Transformer was first proposed by \cite{vaswani2017attention} for machine translation. Early research on applying transformer to vision tasks has not received extensive attention, while the recent proposal of ViT \cite{dosovitskiy2020vit} model has set off a wave of research on visual transformer models. The idea of ViT is very straightforward, which is to transfer the original transformer model as invariant as possible to the image classification task. After ViT, more and more vision tasks started to use the transformer model and its variants. DETR \cite{carion2020end} applies transformer to object detection task with remarkable results. Axial-deeplab \cite{wang2020axial} applies axial attention to image segmentation. ViViT \cite{arnab2021vivit} proposes a video classification model based purely on visual transformer. Swin transformer \cite{liu2021swin} improves the performance of transformer for various visual tasks by further improving the computation of image patch window. Although transformer has achieved good results in various classical vision tasks, there are relatively few studies related to the use of transformer in dense estimation tasks like image generation. TransGAN \cite{jiang2021transgan} uses a pure transformer structure to design the generator and discriminator to train the adversarial model, but the resolution of the generated images is small. Recently, ViTGAN \cite{lee2021vitgan} combines ViT and GAN and designs a new regularization technique to achieve comparable results to CNN-based GAN models on many standard image generation benchmarks. These studies demonstrate that transformer models can also play an effective role in image generation. However, the mainstream image translation approaches are still using CNN-based networks. Because image translation requires that the generated image has accurate semantic information about the conditional input as well as style information about the exemplar image, which demands learning the correspondence between different image domains. In this paper, we propose a cross-domain feature fusion transformer model for exemplar-based image translation. This model can better learn the correspondences between different domains and the potential connections within each domain.








\section{Approach}
Exemplar-based image translation refers to transfer an input image $x_{A}$ from domain $A$ to domain $B$, and make it have the stylistic texture of exemplar $y_{B}$, while maintaining the content of $x_{A}$. Our proposed CFFT-GAN first transform $x_{A}$ and $y_{B}$ into feature space $F$ by two CNN-based encoders, then fuse them using Cross-domain Feature Fusion Transformer, and generate the translation result via a SPADE-based \cite{park2019semantic} residual generator. The overall of our method is illustrated in Figure \ref{fig:model}.




\subsection{Encoders}
\label{subsection1}
For exemplar-based image translation, the inputs are two different types of image domains, content input domain $A$ and style exemplar domain $B$. There is large semantic gaps between them, which is not conducive to the feature fusion later. To learn correspondence better, we need to map the two image domains into the same latent feature space $F$. Specifically, we let $\mathcal{E}_{A \rightarrow S}$ and $\mathcal{E}_{B \rightarrow S}$ represent the encoders of two domains, then the domain aligned latent features can be expressed as:
\begin{equation}
\begin{aligned}
&x_{F}=\mathcal{E}_{A \rightarrow F}\left(x_{A} ; \theta_{\mathcal{E_{A}}}\right), \\
&y_{F}=\mathcal{E}_{B \rightarrow F}\left(y_{B} ; \theta_{\mathcal{E_{B}}}\right),
\end{aligned}
\end{equation}
where $\theta$ is the learnable parameters for each encoder, and $x_{F}$ and $y_{S} \in \mathbb{R}^{C \times H \times W}$ represent the learned domain aligned features, which are used for later feature fusion. The domain alignment of features is trained by Equation \ref{domain_align}.







\subsection{Cross-domain Feature Fusion Transformer}
We propose a novel Cross-domain Feature Fusion Transformer (CFFT) model, shown on the Figure \ref{fig:model}. The CFFT is designed to learn not only inter-domain correspondences, but also feature correlations within domains. Therefore, we can more fully utilize the features of both domains for image translation. Cross-domain feature pairs are first flattened into feature vectors. Then we concatenate the flattened features into a whole along the channel dimension. There are a total of $L=H \times W$ tokens, and the dimension of each token is $1 \times 2C$. The purpose of this operation is that the transformer-based network can learn the correlation information within each feature and also indirectly learn the correspondence between different domain features.

\noindent \textbf{Feature Fusion Network.}
After adding the position embedding, we then use Feature Fusion Network (FFN) to learn feature fusion and semantic coarse matching. FFN consists of two Transformer Encoders (TE) \cite{vaswani2017attention}, each TE has a module layer number N=3. The two TEs are connected through a MLP layer for compressing feature dimension from 2C to C. In this way, FFN achieves a feature fusion role, and based on the transformer's self-attention mechanism, the module is able to learn both the inter-domain and intra-domain correlations within the total feature.


\noindent \textbf{Hierarchical Transformer.}
To decouple and match the two features more finely, we propose a Hierarchical Transformer (Hiformer) after FFN. It presents a stepped structure that continuously splits features in spatial dimension and performs self-learning optimization for each spatial part of the feature layer by layer. For each TE in Hiformer, we use a module layer number N=2. This approach implements a transformer-based feature learning process from coarse to fine. The number of feature splitting levels is adjustable, and we use 3 levels in this paper. We then gradually fuse the local features by same structure. 

With the above network structure, we can obtain the output feature that have been decoupled and spatially aligned, as shown in the feature visualizations of Figure \ref{fig:model}. 



 








\begin{figure}[t]
\centering
\includegraphics[width=1.0\linewidth]{multi_domain.pdf}
\caption{Our method can achieve multi-domain image translation by cascading CFFT modules.}
\label{fig:model2}
\end{figure}


\subsection{Spatially-adaptive Generator}
The role of the generator network is to synthesize $\hat{x}_{B}$ with the content of input image and the style of exemplar image. In the previous step, we have obtained the fused and aligned feature named $\mathbf{f}$, we just need to feed it into the generator network. Therefore, we introduce a spatially-adaptive denormalization (SPADE) \cite{park2019semantic} structure as the generator. We concatenate the feature $\mathbf{f}$ with the scaled input image into $\mathbf{w}$, where scaled input image can help the initial training of the model. Then we feed $\mathbf{w}$ into the SPADE-based residual blocks as modulation parameters layer by layer. Specifically, for a particular feature layer $F^{i} \in \mathbb{R}^{C_{i} \times H_{i} \times W_{i}}$ in the generator, we have the following formula,
\begin{equation}
F_{c, h, w}^{i} = \gamma_{h, w}^{i}(\mathbf{w}_{i}) \times \frac{F_{c, h, w}^{i}-\mu_{h, w}^{i}}{\sigma_{h, w}^{i}}+\beta_{h, w}^{i}(\mathbf{w}_{i}),
\end{equation}
where $\mu_{h, w}^{i}$ and $\sigma_{h, w}^{i}$ are the calculated mean and standard deviation across channel dimension, $\gamma_{h, w}^{i}$ and $\beta_{h, w}^{i}$ are learnable weight networks, $\mathbf{w}_{i}$ is a copy of $\mathbf{w}$. Finally, the generated result of the image translation can be expressed as:
\begin{equation}
\hat{x}_{B}=\mathcal{G}\left(\mathbf{w}; \theta_{\mathcal{G}}\right),
\end{equation}
where $\theta_{\mathcal{G}}$ is the learnable parameters of the generator $\mathcal{G}$.

\subsection{Multi-domain Image Translation}
\label{CFFT-cascade}

Our proposed CFFT is a pluggable module design. When the inputs have more than two image domains, we can also easily decouple and fuse information from multiple domains by cascading CFFT modules. As shown in schematic Figure \ref{fig:model2}, we first learn a feature fusion using CFFT1 for domains A and B, then introduce a third domain C and jointly learn a feature fusion for all three domains by CFFT2. This way of learning facilitates the progressive fusion of features, allowing for better correspondence learning between several different domains. As shown in Figure \ref{fig:animation}, our method can introduce a third domain (pose) on top of the dual-domain image translation to achieve both image translation and face animation.





\subsection{Loss Functions}
We train the CFFT-GAN model end-to-end. Usually, semantically aligned data for different domains is accessible, for example $x_{A}$ and $x_{B}$, whereas triplets data $(x_{A}$,$y_{B};x_{B})$ may not be accessible. We therefore construct the pseudo-tuple $(x_{A},\widetilde{x}_{B};x_{B})$, where $\widetilde{x}_{B}=\mathcal{A}(x_{B})$, $\mathcal{A}$ denotes a collection of data augmentation operations, such as horizontal flip, geometric deformation, random crop, etc. The following loss functions are used for the training process.

\noindent \textbf{Domain Alignment Loss.} We need the features $x_{F}$ and $y_{F}$ learned by the encoders to be in the same feature domain, as explained in subsection "Encoders". Therefore, we let different domain features under the same semantic be aligned.
\begin{equation}
\mathcal{L}_{\text {align}}=\left\|\mathcal{E}_{A \rightarrow S}\left(x_{A}\right)-\mathcal{E}_{B \rightarrow S}\left(x_{B}\right)\right\|_{1}.
\label{domain_align}
\end{equation}

\noindent \textbf{Feature Matching Loss.} Following previous work \cite{johnson2016perceptual,zhang2020cross}, we used the pre-trained VGG-19 network to extract the features of the generated translation result and the corresponding Ground Truth separately, and let the features at each layer match.
\begin{equation}
\mathcal{L}_{\text {match}}=\sum_{l} \mu_{l}\left\|\phi_{l}\left(\hat{x}_{B}\right)-\phi_{l}\left(x_{B}\right)\right\|_{1},
\end{equation}
where $\phi_{l}$ is the output features of layer $l$ in the VGG-19 network, and $\mu_{l}$ is the corresponding weight for each layer.

\noindent \textbf{Translation Loss.} We want the model to output translation image that has semantic content of the input image and the stylized texture of the exemplar image. Therefore, We utilize perceptual loss to reduce semantic differences and contextual loss \cite{mechrez2018contextual} to constrain the stylistic consistency. Specifically, the perceptual loss is shown as follows,
\begin{equation}
\mathcal{L}_{\text {perc}}=\left\|\phi_{h}\left(\hat{x}_{B}\right)-\phi_{h}\left(x_{B}\right)\right\|_{2},
\end{equation}
where $\phi_{h}$ represents the output of a high-level layer in the pre-trained VGG-19 network, and this layer contains mainly high-level semantic information about the input image. The contextual loss (CX) is written as follows,
\begin{equation}
\mathcal{L}_{CX}=\sum_{l} \omega_{l}\left[-\log \left(C X\left(\phi_{l}\left(\hat{x}_{B}\right), \phi_{l}\left(y_{B}\right)\right)\right)\right],
\end{equation}
where CX stands for contextual similarity, $\phi_{l}$ is the $l$th layer of low-level features in the pre-trained VGG-19 network, which contain mainly the rich stylized texture of the image, and $\omega_{l}$ is the weight coefficients of the different layers.

\noindent \textbf{Adversarial Loss.} We also use a discriminator network to distinguish between the real samples and the generated samples, and train it together with our image translation model. The adversarial loss function is defined as,
\begin{equation}
\begin{aligned}
\mathcal{L}_{a d v}^{\mathcal{D}} &=-\mathbb{E}\left[h\left(\mathcal{D}\left(y_{B}\right)\right)\right]-\mathbb{E}\left[h\left(-\mathcal{D}\left(\mathcal{G}\left(x_{A}, y_{B}\right)\right)\right)\right], \\
\mathcal{L}_{a d v}^{\mathcal{G}} &=-\mathbb{E}\left[\mathcal{D}\left(\mathcal{G}\left(x_{A}, y_{B}\right)\right)\right],
\end{aligned}
\end{equation}
where $h(t)=min(0,-1+t)$ represents the hinge function to regularize the discriminator.

\noindent \textbf{Total Loss.} In conclusion, we optimize the following overall objective.
\begin{equation}
\begin{aligned}
\mathcal{L} &= \lambda_{\text{align}} \mathcal{L}_{\text {align }}+\lambda_{\text{match}} \mathcal{L}_{\text {match}}+\lambda_{\text{perc}} \mathcal{L}_{\text{perc}} \\
&+\lambda_{\text{CX}}\mathcal{L}_{\text {CX}}+\lambda_{\text{adv}}\left(\mathcal{L}_{\text {adv }}^{\mathcal{D}}+\mathcal{L}_{\text {adv }}^{\mathcal{G}}\right),
\end{aligned}
\end{equation}
where weights $\lambda$ are used to balance the individual objectives.







\begin{table*}[t]
\centering
\begin{tabular}{l|ccc|cc|cc|cc}
\hline
& \multicolumn{3}{c|}{Deepfashion} & \multicolumn{2}{|c|}{CelebA-HQ} & \multicolumn{2}{|c|}{Metfaces} & \multicolumn{2}{|c}{ADE20K}\\
\cline{2-10} 
& \text { FID $\downarrow$ } & \text { SWD $\downarrow$ } & \text { LPIPS $\downarrow$ }  & \text { FID $\downarrow$ } & \text { SWD $\downarrow$ } & \text { FID $\downarrow$ } & \text { SWD $\downarrow$ } & \text { FID $\downarrow$ } & \text { SWD $\downarrow$ } \\
\hline
{CoCosNet v1}  & {14.4} & {29.0} & {0.2386} & {19.4} & {22.3} & {25.6} & { 24.3} & {26.4} & {10.5} \\
{UNITE}        & {13.1} & {26.6} & {0.2371} & {18.3} & {21.6} & {24.1} & {23.5} & \textbf{25.1} & {10.1}\\
{CoCosNet v2}  & {12.2} & {24.6} & {0.2245} & {16.2} & {19.7} & {23.3} & {22.4} & {25.2} & \textbf{9.9} \\
{Ours}   & \textbf{8.8} & \textbf{19.0} & {\textbf{0.2070}} & \textbf{12.4} & \textbf{18.1} & \textbf{22.6} & \textbf{21.3} & {25.4} & {10.1}\\
\hline
\end{tabular}
\caption{Quantitative comparison results of image quality between our method and SOTA methods.}
\label{table:metrics1}
\end{table*}









\section{Experiments}
\textbf{Datasets.} To verify the generality of our approach, we perform experiments on several widely used public datasets.
\begin{itemize}
    \item Deepfashion \cite{liu2016deepfashion} contains 52,712 human images with various appearances and poses. Human image paired data is used in our experiments. We also utilize Openpose \cite{cao2017realtime} to extract the key points of the human image as input pose.
    \item CelebA-HQ \cite{liu2015deep} is comprised of 30,000 high resolution images of faces in a variety of styles. We connect its face landmarks to form the face edge, and use the Canny edge detection algorithm to get the background edge, and finally combine them as the input.
    \item MetFaces \cite{karras2020training} is a high resolution art-style portrait dataset, which contains 1,336 images of human faces. We use HED \cite{xie2015holistically} to extract the sketch of the face images.
    \item ADE20K \cite{zhou2017scene} contains 25,574 scene images with 150-class semantic segmentation annotation. We use this dataset to validate the image translation from mask to scene.
    \item AFHQ \cite{choi2020stargan} is a animal faces dataset consisting of 15,000 high-quality images. We also use HED to extract the sketch of the animal faces.
\end{itemize}
For better comparison with the SOTA methods and to demonstrate the good performance of our method, all datasets are experimented with 512 resolution images, except for ADE20K where 256 resolution images are used.



\noindent \textbf{Baselines.} We compare our method with SOTA exemplar-based image translation methods. 1) CoCosNet v1 \cite{zhang2020cross} summarized various types of image translation tasks into exemplar-based image translation for the first time. It proposes a cross-domain correspondence learning method, which deforms the exemplar image by calculating the correlation matrix of two domain features. 2) UNITE \cite{zhan2021unbalanced} introduces optimal transport for proper feature alignment and faithful style control in image translation. 3) CoCosNet v2 \cite{zhou2021cocosnet} learn a full-resolution correspondence, and purpose a hierarchical GRU-assisted PatchMatch method for efficient correspondence computation. Because we propose a general image translation method, there is no comparison with task-specific methods. Besides, we do not compare with the approaches that directly learn image translation and fail to use an exemplar.


\noindent \textbf{Evaluation Metrics.} We adopt the commonly used image generation evaluation metrics to assess the performance of the various methods. Fréchet Inception Score (FID) \cite{heusel2017gans} is the metric used to evaluate the quality and diversity of generated images. Sliced Wasserstein distance (SWD) \cite{karras2017progressive} is applied to measure the statistical distance of low-level patch distribution between synthesized images and real images. Learned Perceptual Image Patch Similarity (LPIPS) is a semantic similarity evaluation metric, and we use it to evaluate the decoupling and semantic alignment of the models on Deepfashion dataset, since only Deepfashion has paired data to compute this metric.

For exemplar-based image translation, the generated image should have the same semantic content as the input and have the consistent style and texture as the exemplar. We therefore calculate two additional metrics for evaluating semantic consistency and style similarity. We utilize the pre-trained VGG model on ImageNet and use its high-level feature maps, $relu3\_ 2$, $relu4\_ 2$ and $relu5\_ 2$, to represent high-level semantics. We then calculate the high-level semantics average cosine similarity between the generated and content input images as a semantic consistency metric. Similarly, the style similarity metric can be calculated using the low-level feature maps, $relu1\_ 2$, $relu2\_ 2$. 


\begin{table}[t]
\centering
\setlength{\tabcolsep}{0.4mm}{
\begin{tabular}{l|c|c|c|c}
\hline
& {Deepfashion} & {CelebA-HQ} & {Metfaces} & {ADE20K}\\

\hline
{CoCosNet v1}  & {0.924} & {0.945} & {0.941} & {0.862}  \\
{UNITE}        & {0.944} & {0.948} & {0.956} & {0.839} \\
{CoCosNet v2}  & {0.959} & {0.955} & {0.963} & {0.877}  \\
{Ours}         & \textbf{0.975} & \textbf{0.961} & \textbf{0.968} & \textbf{0.878}\\
\hline
\end{tabular}}
\caption{Quantitative evaluation of semantic consistency. The higher the score, the better the evaluation metric.}
\label{table:metrics2}
\end{table}


\begin{figure}[t]
\centering
\includegraphics[width=1.0\linewidth]{compare.pdf}
\caption{Qualitative comparison results of our method with SOTAs on Deepfashion dataset, CelebA-HQ dataset, MetFaces dataset, and ADE20K dataset, respectively.}
\label{fig:compare}
\end{figure}



\begin{table}[t]
\centering
\setlength{\tabcolsep}{0.4mm}{
\begin{tabular}{l|c|c|c|c}
\hline
& {Deepfashion} & {CelebA-HQ} & {Metfaces} & {ADE20K}\\
\hline
{CoCosNet v1}  & {0.959} & {0.947} & {0.944} & {0.952} \\
{UNITE}        & {0.966} & {0.953} & {0.965} & {0.955} \\
{CoCosNet v2}  & {0.974} & {0.966} & {0.964} & \textbf{0.959} \\
{Ours}         & \textbf{0.982} & \textbf{0.974} & \textbf{0.971} & 0.954\\
\hline
\end{tabular}}
\caption{Quantitative evaluation of style similarity. The higher the score, the better the evaluation metric.}
\label{table:metrics3}
\end{table}

\begin{figure}[t]
\centering
\includegraphics[width=1.0\linewidth]{animation.pdf}
\caption{Multi-domain image translation results. Through the figure \ref{fig:model2} structure, input edge and exemplar we can get translation result, and then input head pose we can get face animation result based on the translation result.}
\label{fig:animation}
\end{figure}

\begin{figure}[t]
\centering
\includegraphics[width=0.98\linewidth]{cross.pdf}
\caption{Portrait stylization. Our method can transfer image styles from the MetFaces to the faces of the CelebA-HQ. The model is only trained on the
MetFaces dataset. The model input is the edge image of input face.}
\label{fig:cross}
\end{figure}


\begin{figure}[t]
\centering
\includegraphics[width=0.98\linewidth]{vis_feature.pdf}
\caption{Visualization results of CFFT's input and output feature maps, which demonstrate the decoupling and semantic alignment capabilities of CFFT.}
\label{fig:hotmap}
\end{figure}

\noindent \textbf{Implementation Details.}
We use the TTUR \cite{heusel2017gans} learning strategy with a generator and discriminator learning rate of 1e-4 and 4e-4, respectively. We utilize Adam solver with $\beta_1 = 0$ and $\beta_2 = 0.999$. Our experiments are carried out on 8 32GB Tesla V100 GPUs. The input and output feature scales for the CFFT module are $(C,H,W)=(64,64,64)$. The weights of the total loss are $\lambda_{\text{align}}=10$, $\lambda_{\text{match}}=10$, $\lambda_{\text{perc}}=0.001$, $\lambda_{\text{CX}}=10$, $\lambda_{\text{adv}}=10$.




\begin{table}[t]
\centering
\begin{tabular}{l|c|c|c}
\hline
& {FID} & {SWD} & {Params} \\
\hline
{CFFT$\rightarrow$ CM} & {14.3} & {28.8} & {4.8M}  \\
{CFFT$\rightarrow$ AdaIN}  & {16.3} & {30.2} & {0.02M}  \\ 
{CFFT w/o Hiformer}  & {11.3} & {25.0} & {1.37M} \\
{CFFT 32$\times$32}  & {9.5} & {23.7} &  {2.22M}  \\
{Ours}               & \textbf{8.8} & \textbf{19.0} & 2.62M \\
\hline
\end{tabular}
\caption{Ablation studies of our method. Evaluate the FID and SWD metrics on the Deepfashion dataset, and compute the parameters of each module.}
\label{table:ablation}
\end{table}




\subsection{Experimental Results}

\noindent \textbf{Quantitative Evaluation.} For the quantitative comparison experiments, we first used FID and SWD to assess the quality of the generated images for each model, and use LPIPS to evaluate the model decoupling and semantic alignment ability of each model on the Deepfashion dataset. As shown in Table \ref{table:metrics1}, our approach outperforms the SOTA methods in most datasets. This is due to the fact that our proposed CFFT module is able to better learn the correspondence between different domains and the potentially beneficial information within each domain, allowing the generator to obtain richer information.

In addition to evaluating the quality of the generated images, we quantitatively assess the semantic consistency of the generated images and content inputs and the style similarity to the exemplar images, as shown in Table \ref{table:metrics2} and \ref{table:metrics3}. Compared with the SOTA methods, our model also achieves higher performance.


\noindent \textbf{Qualitative Comparison.} We make a qualitative comparison with SOTA methods, as shown in Figure \ref{fig:compare}. Our method can better fuse the style of the exemplar and the semantic content of the input. Our approach can also obtain more refined results when the semantic difference between exemplar and input is large, as shown in the third row. In addition to the comparative results, Figure \ref{fig:show} also show the more diverse generative results of our method. As can be seen from the figures our approach shows strong performance in a wide range of image translation tasks and the diversity of the generated results reflects the generalizability of our model.

\noindent \textbf{Applications.} Our method has good potential for application. Firstly, we can decouple and fuse information from multiple domains by cascading CFFT modules. We test a specific task, face animation. As shown in Figure \ref{fig:animation}, we first learn exemplar-based image translation between edge input and RGB exemplar, and then add head pose to learn the correspondence between three domains. The head pose is extracted via DECA \cite{feng2021learning}. The whole training process does not use paired data, and it is self-supervised training. Our generated results have the shape, style, and pose corresponding to the each input domain. By cascading CFFT modules, our approach can progressively learn the feature fusion from multiple image domains to achieve more complex image translation. Secondly, we test the portrait stylization experiment, as shown in Figure \ref{fig:cross}. We demonstrate the generalizability and excellent performance of our method through this cross-dataset experiment.

More comparative results and visualizations are presented in the supplementary material.



\subsection{Ablation Study}
We perform the ablation studies of CFFT, the key module of our model, on the Deepfashion dataset. The quantitative results are shown in Table \ref{table:ablation}. The first row of the table represents that we replace the CFFT module with the Correlation Matrix in CoCosNet. This method requires a larger amount of parameters and does not achieve good performance. In the second row of the table we replace the CFFT with the AdaIN \cite{huang2017arbitrary} structure, which is a very common method of feature transfer. Although it has few parameters, its performance is low for complex correspondence learning. In the third row, we remove the Hiformer in the CFFT, and the performance of the model has dropped significantly. In the fourth row, we decrease the spatial scale of the latent features in CFFT, from 64$\times$64 to 32$\times$32. But the model is still able to maintain good performance. In addition, we visualise the input and output features of the CFFT module, as shown in Figure \ref{fig:hotmap}. The results show that the CFFT is capable of good feature decoupling and semantic alignment.




\section{Conclusion}
This paper proposes CFFT-GAN, an exemplar-based image translation that learns the feature fusion of different domains by transformer-based module CFFT, which is able to not only learn the correspondence between domains better but also extract potential beneficial information within each domain. Our proposed CFFT has strong generality, and we can achieve multi-domain image translation by cascading CFFT. Quantitative and qualitative experiments demonstrate the superiority of our approach over the SOTA methods, and our ablation studies on the key module CFFT also reflect its core effect. There is still room for improvement in our approach, e.g., our model still has a performance deficit in the task of mask-to-image. We will optimize this task in future work.


\section{Acknowledgments}
This work is supported by the National Key Research and Development Program of China under Grant No. 2021YFC3320103.





