\section{Introduction}
Conditional image synthesis aims to create an image according to the given control signals. 
With the increasing demand for flexible conditional image synthesis, various kinds of control signals have been introduced into this field, which can be divided into three main modalities: (i) \emph{textual controls (TC)}, including the class labels~\cite{brock2018large} and natural language descriptions~\cite{xu2018attngan,tao2020df}; (ii) \emph{visual controls (VC)}, such as a spatially-aligned sketch map for reference~\cite{ghosh2019interactive,xia2019cali} or 
another image for style transfer~\cite{gatys2016image,johnson2016perceptual}; (iii) \emph{preservation controls (PC)}, which require the synthesized image to preserve some given image blocks, e.g., image outpainting and inpainting~\cite{yu2019free,zhang2020text}.

However, control signals of various modalities possess different characteristics.
Existing works~\cite{xu2018attngan, isola2017image,xia2020tedigan} hence typically design separate methods customized for each control modality.
Moreover, most of these approaches only utilize one type of control signal and cannot simultaneously combine multiple types of controls in a concise and versatile model.
This begs the question: {can we integrate any number of multi-modal control signals into a unified framework for flexible conditional image synthesis?} 
There are two inevitable challenges in this setting: 
(i) how to unify the multi-modal controls and represent them in a unified form, especial when employing multiple control signals from different modalities concurrently;
(ii) how to guarantee the fulfillment of the multi-modal controls while ensuring the fidelity of the synthesized image.

\begin{figure}
\centering
\includegraphics[width=1.0\textwidth]{exp1}
    \vspace{-0.4cm}
\caption{The three main modalities of control signals for conditional image synthesis: Textual Controls (TC), Visual Controls (VC), and Preservation Controls (PC).}\label{fig:exp}
    \vspace{-0.5cm}
\end{figure}

Recently, two-stage image synthesis~\cite{oord2017neural,razavi2019generating,chen2020generative,esser2020taming,ramesh2021zero} has made great progress. The first stage learns a convolutional autoencoder with quantized latent representations for converting an image into a sequence of discrete tokens, e.g., for compressing a 256$\times$256 image into a sequence of $32\times32$ tokens where each token correlates mainly with an $8\times8$ block of the image.
Converting a sequence of tokens back into an image is also supported.
The second stage then typically adopts an autoregressive model, e.g., PixelCNN~\cite{oord2016conditional} or a unidirectional Transformer decoder~\cite{vaswani2017attention}, to capture the distribution over sequences of tokens. Particularly, the Transformer-based methods~\cite{chen2020generative,esser2020taming,ramesh2021zero} exploit the global expressivity of Transformer to capture long-range relationships between local constituents.

In this paper, we make two key observations about the two-stage framework.
First, the two-stage framework has the advantage that it can potentially unify the multi-modal control signals and the generated image into a single sequence of discrete tokens. 
However, existing works~\cite{oord2017neural,razavi2019generating,esser2020taming,ramesh2021zero}
largely neglect this advantage of the two-stage framework over the traditional one-stage approaches such as those based mainly on the generative adversarial networks (GAN)~\cite{goodfellow2014generative}.
Second, the \emph{autoregressive (AR)} approach to sequence generation, adopted by the existing two-stage methods such as DALL-E~\cite{ramesh2021zero} and VQGAN~\cite{esser2020taming}, brings undesirable shortcomings:
(i) the token-by-token synthesis procedure leads to slow generation speed, especially for the heavyweight Transformer~\cite{chen2020generative,esser2020taming,ramesh2021zero};
(ii) each generated token can only catch sight of the previously generated tokens and cannot incorporate bidirectional contexts, which may affect the holistic consistency of image synthesis;
(iii) the fixed left-to-right order of autoregressive decoding cannot respond to the preservation control signals unless the image blocks to be preserved are at the beginning of the sequence.
Notably, different from AR generation, \emph{non-autoregressive (NAR)} sequence generation with bidirectional Transformer, i.e., BERT~\cite{devlin2019bert}, can naturally avoid the three shortcomings. 

Based on the aforementioned observations, we propose UFC-BERT, a novel BERT-based two-stage framework to {\bf U}ni{\bf F}y any number of multi-modal {\bf C}ontrols for conditional image synthesis.
Concretely, the textual, visual, and preservation control signals, as well as the generated image, are uniformly represented as a sequence of discrete tokens, as shown in Figure~\ref{fig:framework}. The textual control consists of word tokens for class labels or natural language descriptions.
The visual control(s) and the generated image are both represented as discrete tokens due to the first stage, where each token corresponds to a block within the reference image(s) or the generated image.
Zero, one, or more reference images are supported.
To preserve a given image block within the generated image, we encode the given image block into discrete tokens and fix corresponding parts of the generated sequence to the tokens.

We train UFC-BERT via the masked sequence modeling task, which predicts a masked subset of the target image's tokens conditioned on both the multi-modal control signals and the generation target's unmasked tokens. During inference, we adopt Mask-Predict, a NAR generation algorithm~\cite{ghazvininejad2019mask,guo2020incorporating,cho2020x}, which predicts all target tokens at the first iteration and then iteratively re-mask and re-predict a subset of tokens with low confidence scores.

To further improve upon the NAR generation algorithm,
we exploit the discriminative capability of the BERT architecture~\cite{dong2019unified,zhou2020unified} and add two estimators (see Figure~\ref{fig:framework}), where one estimator estimates the relevance between the generated image and the control signals, and the other one estimates the image's fidelity.
The two estimators help improve the quality of the synthesized image, because at each iteration we can generate multiple samples and keep only the highly-scored one before starting the next iteration.
The two estimators also help save the number of iterations needed, since the algorithm can dynamically terminate if running for more iterations no longer improves the scores.


\mjxdel{
In conclusion, the main contributions of our UFC-BERT are summarized as follows:
\begin{itemize}
\item To the best of our knowledge, UFC-BERT is the first work that unifies any number of multi-modal controls by a universal form for conditional image synthesis.
\item The non-autoregressive generation with BERT can improve the inference speed, enhance the holistic consistency of images and support the preservation controls for image inpainting.
\item The progressive NAR generation algorithm with relevance and fidelity estimators ensures the effectiveness of multi-modal controls and the fidelity of the synthesized image.
\end{itemize}
}


The extensive experiments on M2C-Fashion, a newly collected clothing dataset with tens of millions of image-text pairs, as well as on Multi-Modal CelebA-HQ~\cite{karras2017progressive,xia2020tedigan}, a public facial dataset, demonstrate UFC-BERT can synthesize high-quality images that comply with various multi-modal controls.




\section{Related Works}

We have discussed the connection between our work and \textbf{Two-Stage Image Synthesis} in the Introduction.
In this section, we further discuss related works from other fields.

\textbf{Conditional Image Synthesis.}
A variety of control signals have been introduced into conditional image synthesis.
The class-conditional generation task~\cite{brock2018large,miyato2018cgans} adopts class labels as control signals.
The text-to-image synthesis task~\cite{zhang2017stackgan,zhang2018stackgan++,li2019controllable,xu2018attngan,zhu2020sean,tao2020df} further employs natural language descriptions as controls.
The image-to-image translation task generates photo-realism images from visual controls, such as a sketch map~\cite{ghosh2019interactive,xia2019cali}, semantic label map~\cite{isola2017image,chen2017photographic,wang2018high}, human pose~\cite{ma2017pose} or another image for style transfer~\cite{gatys2016image,johnson2016perceptual}. 
Moreover, image outpainting and inpainting~\cite{iizuka2016let,yu2019free} can be regarded as image synthesis conditioned on preservation control signals, where some image blocks of the desired image are already specified and need to be preserved in the generated image.
However, these works only utilize one kind of control signal and design their methods customized for each kind of control.
Text-guided image manipulation~\cite{dong2017semantic,nam2018text,zhang2020text,li2020manigan,xia2020tedigan} semantically edits an image, where the text description and the original image serve as control signals.
But they still fail to unify multiple modalities in a universal form and cannot easily extend to more control modalities.
To promote versatility and extensibility, we propose UFC-BERT to unify any number of multi-modal controls.

\mjxdel{
\textbf{Two-Stage Image Synthesis.}
Recently, two-stage image synthesis~\cite{oord2017neural,razavi2019generating,chen2020generative,esser2020taming,ramesh2021zero} has made great progress, which first represents the images as discrete codes (i.e. image tokens) from a learned codebook at stage 1 and then models the distributions over the discrete sequence at stage 2. 
The first work VQ-VAE~\cite{oord2017neural} proposes the Vector Quantised Variational AutoEncoder to learn discrete representations of images, and apply a convolutional model PixelCNN~\cite{oord2016conditional} to autoregressively generate the codes. And VQ-VAE-2 proposes a hierarchy of learned representations to extend this method. 
Further, current methods~\cite{chen2020generative,esser2020taming,ramesh2021zero} 
develop the unidirectional Transformer~\cite{vaswani2017attention} to capture long-range interactions between discrete codes at the second stage. Among them, \cite{chen2020generative} uses a shallow VQVAE with the small receptive field at stage 1, but VQGAN~\cite{esser2020taming} design a more powerful first stage for high-resolution image synthesis, which captures as much context as possible in the learned codebook. And DALL.E~\cite{ramesh2021zero} endow this framework with the large-scale data, model size and computational resources, and surprisingly achieves zero-shot text-to-image synthesis in open domains. 
Different from these methods, we apply the two-stage framework to unify multi-modal controls and employ the non-autoregressive generation (NAR) at the second stage to improve the synthesis speed, enhance the image coherence and support the preservation control. 
}

\textbf{Visual-Language Transformer.} With great progress in language tasks~\cite{vaswani2017attention,radford2018improving,radford2019language, brown2020language}, the transformer architecture is being rapidly transferred to other fields such as vision~\cite{chen2020generative, dosovitskiy2020image} and audio~\cite{child2019generating}. Recently, pretraining visual-language transformer~\cite{qi2020imagebert,huang2020pixel,zhou2020unified,cho2020x,lu2019vilbert,tan2019lxmert} (e.g. multi-modal BERT) has achieved significant improvements on a variety of downstream tasks, e.g. visual question answering, image captioning~\cite{zhou2020unified}, and text-to-image generation~\cite{cho2020x}. Among them, the single-stream architecture~\cite{su2019vl,li2020unicoder,rahman2019m,chen2019uniter,qi2020imagebert,huang2020pixel} uses a single transformer to jointly model a pair of text and image, while the two-stream architecture~\cite{lu2019vilbert,lu202012,tan2019lxmert} applies two transformers to separately learn the representations of the text and the image, respectively.
Our UFC-BERT is also a variant of the single-stream visual-language transformer, but focuses on flexible multi-modal image synthesis instead of multi-modal pretraining.




\textbf{Non-Autoregressive Sequence Generation.}
Though it is natural to autoregressively predict tokens from left to right when generating a sequence, autoregressive decoding suffers from the slow speed and sequential error accumulation.
Thus, the non-autoregressive generation (NAR) paradigm is proposed to avoid these drawbacks in neural machine translation~\cite{gu2017non,guo2019non,lee2018deterministic,ghazvininejad2019mask}, image captioning~\cite{gao2019masked,guo2020non,zhou2020unified}, and speech synthesis~\cite{ren2019fastspeech,ren2020fastspeech}.
These approaches often employ the bidirectional Transformer (i.e. BERT) as it is not trained with a specific generation order.
Our progressive NAR generation algorithm improves upon the Mask-Predict non-autoregressive algorithm~\cite{wang2019bert,ghazvininejad2019mask,mansimov2019generalized,liao2020probabilistically}, by introducing the relevance estimator and the fidelity estimator to facilitate sample selection and dynamic termination.




\newcommand{\mathbb{R}}{\mathbb{R}}
\newcommand{\mathcal{Z}}{\mathcal{Z}}
\newcommand{\mathcal{W}}{\mathcal{W}}
\newcommand{\mathcal{Z}_V}{\mathcal{Z}_V}
\newcommand{\mathcal{Z}_I}{\mathcal{Z}_I}
\newcommand{{n_z}}{{n_z}}
\newcommand{D}{D}
\newcommand{E}{E}
\newcommand{\mathbf{q}}{\mathbf{q}}
\newcommand{{\bf Z}}{{\bf Z}}










\section{UFC-BERT For Multi-Modal Image Synthesis}



\subsection{Background: Two-Stage Image Synthesis}
In this section, we review the two-stage architecture~\cite{oord2017neural,razavi2019generating,esser2020taming,ramesh2021zero} for image synthesis.

At the first stage, a codebook $\mathcal{Z} = \{{\bf z}_k\}_{k=1}^K$ for vector quantization is learned, where ${\bf z}_k \in \mathbb{R}^{n_z}$ is the $k$-th code-word in the codebook and $K$ is the number of code-words. 
An image ${\bf X} \in \mathbb{R}^{H \times W \times 3}$ can be transformed into (or from) a collection of code-words ${\bf Z} \in \mathbb{R}^{h \times w
\times {n_z}}$. 
Concretely, a convolutional encoder $E$ first encodes the original image ${\bf X}$ as $  \hat{{\bf Z}} = E({\bf X}) \in \mathbb{R}^{h\times w \times {n_z}}$.
Then an element-wise quantization step $\mathbf{q}(\cdot)$ is applied to each element $\hat{{\bf Z}}_{ij}$ to obtain the element's closest code-word ${\bf z}_k$, i.e., $ \mathbf{q}( \hat{{\bf Z}}_{ij}) = {\rm arg\, min}_{{\bf z}_k\in \mathcal{Z}} \Vert \hat{{\bf Z}}_{ij} - {\bf z}_k \Vert$. 
For reconstruction, a convolutional decoder $D$ is also learned for recovering image $\hat{\bf X}\in \mathbb{R}^{H \times W \times 3}$ from $ {{\bf Z}}$ such that $\hat{\bf X}$ is close to ${\bf X}$. 
The first stage can be denoted by
\begin{equation}
 {\bf Z} = \mathbf{q}(E({\bf X})),\ \hat{\bf X} = D({\bf Z} ).
\end{equation}
Due to the convolutional layers, each of the $h\times w$ elements of $\hat{{\bf Z}}$ mainly correlates with an $\frac{H}{h}\times \frac{W}{w}$ block of the image, though its receptive field may be larger if multiple convolutions are stacked.

At the second stage, image ${\bf X}$'s quantized representation ${\bf Z}$ can be rewritten as a sequence of codes ${\bf I} \in \{0, \dots, \vert \mathcal{Z} \vert - 1 \}^{N_I}$, composed of $N_I$ ($=h \times w$) indices from the codebook $\mathcal{Z}$.
Thus, image synthesis can be formulated as autoregressive sequence generation, i.e. predicting the distribution $\Pr({I}_i  \vert {\bf I}_{<i}, {\bf C})$ of the next token ${I}_i$ conditioned on the preceding tokens ${\bf I}_{<i}$ and the control signals ${\bf C}$. 
The distribution is typically modeled using a unidirectional Transformer.
The likelihood is then $\Pr({\bf I}\vert {\bf C})=\prod_i \Pr({I}_i \vert {\bf I}_{<i}, {\bf C})$.
Parameters are learned by minimizing $\mathcal{L}_{\text{AR}} = \mathbb{E}_{{\bf I} \sim data} \left[ -\log \Pr({\bf I\vert {\bf C}}) \right].$


We focus on improving the second stage.
Specifically, the autoregressive paradigm adopted by the existing two-stage works~\cite{oord2017neural,razavi2019generating,esser2020taming,ramesh2021zero} suffers from slow generation speed, fails to capture bidirectional contexts, and cannot fully support preservation control signals.
We thus propose UFC-BERT, a novel NAR approach for stage two, to unify any number of multi-modal controls and tackle the shortcomings of AR.
As for stage one, we directly follow VQGAN's design~\cite{ramesh2021zero}, which improves upon VQVAE~\cite{oord2017neural} by incorporating a perceptual loss~\cite{johnson2016perceptual} and patch-based adversarial training~\cite{isola2017image}.



\begin{figure}
\centering
\includegraphics[width=1\textwidth]{framework}
    \vspace{-0.5cm}
\caption{The framework of UFC-BERT,
where the textual control (TC), vsiual control (VC), and preservation control (PC), as well as the image to generate, collectively form a sequence of tokens.
}\label{fig:framework}
    \vspace{-0.5cm}

\end{figure}


\subsection{Problem Formulation}
Conditional image synthesis aims to generate an image that satisfies a set of control signals ${\bf C}$.
We consider three major modalities of control signals. 
A \emph{Textual Control (TC)} consists of a sequence of words ${\bf T} \in \{0, \dots, \vert \mathcal{W} \vert - 1 \}^{N_T}$, where $\mathcal{W}$ is the vocabulary and $N_T$ is the number of words in the text.
In the two-stage framework, an image can be converted into a sequence of code-words (i.e. tokens) based on stage one's encoder $E$ and codebook $\mathcal{Z}$.
Thus, a \emph{Visual Control (VC)} is denoted by a sequence ${\bf V} \in \{0, \dots, \vert \mathcal{Z} \vert - 1 \}^{N_V}$ consisting of code-words from the codebook $\mathcal{Z}$, where $N_V$ is the sequence length.
Similarly, the target (i.e., the image to generate) is a sequence of code-words ${\bf I} \in \{0, \dots, \vert \mathcal{Z} \vert - 1 \}^{N_I}$.
We support zero, one, or multiple visual controls for flexibility.
As for the \emph{Preservation Control (PC)}, it is a sequence of binary masks ${\bf P} \in \{0, 1\}^{N_I}$ with the same length as ${\bf I}$, where $1$ means that the token is known (i.e., $I_i$ is ground-truth if $P_i=1$) while $0$ means the token needs to be predicted.
We aim to design a model at the second stage to synthesize the target image's sequence ${\bf I}$ conditioned on ${\bf C}$, i.e., a combination of any number of control signals from $\{{\bf T},{\bf V},{\bf P}\}$. 

\subsection{Model Inputs}
As shown in Figure~\ref{fig:framework}, our UFC-BERT modifies the original BERT model~\cite{devlin2019bert} to accommodate any number of multi-modal controls.
Similar to BERT, the backbone is a multi-layer bidirectional Transformer encoder, enabling the dependency modeling between all input elements. 
The input sequence of UFC-BERT always starts with two special tokens [REL] and [FDL] for relevance estimation and fidelity estimation, then goes on with the word sequence ${\bf T}$ of textual controls, code sequence ${\bf V}$ of visual controls, and ends with the code sequence ${\bf I}$ of the target image to generate.
Two special separation tokens [EOT] and [EOV] are appended to the end of the textual and the visual control sequences, respectively. If there are multiple visual controls, another special token [SEP] is inserted to separate them.
The sequence ${\bf I}$ of the target image to generate may be partially or fully masked by a special token [MASK].
When the preservation control ${\bf P}$ is present and $P_{i}=1$, token $I_{i}$ in ${\bf I}$ is always set to the code-word corresponding to the given image block to be preserved.


Each input token's representation is the sum of the position and token embeddings:

{\bf Position Embedding.} Our UFC-BERT learns independent sets of position embeddings for the different kinds of the inputs to achieve better distinguishment between the various modalities.
The position embeddings for the word sequence are the same as BERT, i.e., we use sequential position embeddings.
For a visual control or the target image, the position embedding of each token is decided according to where this token lies on the $h\times w$ grid, i.e., we use spatial position embeddings.

{\bf Token Embedding.} For textual controls, we use Byte-Pair Encoding~\cite{sennrich2015neural} to segment each word into sub-words and then learn sub-word embeddings. Each special token, e.g., [REL] or [MASK], is assigned a dedicated embedding.
For visual controls and the target image, we learn an embedding for each code-word. We do not directly use the embeddings from stage one's codebook due to the decoupling of the two stages.

\subsection{Training: Masked Sequence Modeling with Relevance and Fidelity Estimation}
As shown in Figure~\ref{fig:framework}, we train UFC-BERT via masked sequence modeling, i.e., predicting the masked tokens in the target image conditioned on the controls.
A relevance estimator and a fidelity estimator are also trained in the process, and will be key to our progressive NAR generation algorithm.

{\bf Task 1: Masked Sequence Modeling.} 
This task is similar to Masked Language Modeling (MLM) in BERT, but incorporates multi-modal control signals when predicting the masked tokens.
To construct training samples, we mask parts of the target image ${\bf I}$ to predict using four strategies:
(1) randomly decide the number of tokens to mask, and then randomly mask the desired number of tokens;
(2) mask all tokens; 
(3) mask the tokens within some boxed areas of the image, where the number of boxes and the box sizes are randomly decided;
(4) mask the tokens outside some random boxed areas of the image.
We use the four strategies with probability 0.70, 0.10, 0.10, and 0.10, respectively.\mjxdel{The third and fourth strategies are designed for preservation controls as the conventional image inpainting often fill in the box area based on other contents, or repair other parts given a box area. }
To construct multi-modal control signal ${\bf C}$ for each training sample, there are four different combinations: {\it <TC, VC>, <TC>, <VC>, <empty>}, where
{\it <TC, VC>} means the textual and visual controls are simultaneously employed, {\it <TC>} or {\it <VC>} means only a textual or visual signal is used, and {\it<empty>} means no textual or visual control is present. 
Note that the preservation control is already included in the masked sequence modeling task.
Since our dataset does not contain ground-truth pairs of visual controls and target images, we crop one or multiple regions of a target image to construct VC for the target image.
Because image synthesis from solely textual controls is more challenging than from other signals, we use the four combinations with probability 0.20, 0.55, 0.20, 0.05, respectively, where textual controls get more attention.
We feed UFC-BERT's outputs at each position of ${\bf I}$ into a softmax classifier over the codebook $\mathcal{Z}$, which produces a probability score $Y_i = \Pr({I}_i \vert {\bf I}_{{U}}, {\bf C})$ for each position $i\in M$, where $M$ is the set of masked positions and $U$ is the unmasked set.
Finally, the masked sequence modeling task minimizes the softmax cross-entropy loss $\mathcal{L}_{\text{MSM}} = \mathbb{E}_{{\bf I}_M,{\bf I}_U} \left[ -\log \Pr({\bf I}_M \vert {\bf I}_{{U}}, {\bf C}) \right]$, where $\Pr({\bf I}_{M} \vert {\bf I}_{{U}}, {\bf C} )=\prod_{i \in {M}} Y_i$.


{\bf Task 2: Relevance Estimation.} 
This task is to learn a binary classifier that judges whether the generated image is relevant or irrelevant to the given multi-modal control ${\bf C}$. Briefly, we add a linear layer on the output corresponding to the special token [REL].
The linear layer outputs a scalar representing the logit, and a binary cross-entropy loss $\mathcal{L}_{\text{REL}}$ is added.
During training, the training samples from Task 1 serve as the positive instances (i.e. relevant pairs). We construct negative instances (i.e. irrelevant pairs) by swapping the control signals of two training samples. 

{\bf Task 3: Fidelity Estimation.} 
This task aims to distinguish whether the generated image is realistic from the view of human visual cognition. Similar to relevance estimation, we feed the output corresponding to [FDL] into a linear layer for binary classification and add another binary cross-entropy loss $\mathcal{L}_{\text{FDL}}$. Since the low-fidelity images (i.e. negative instances) do not exist in the dataset, 
we run UFC-BERT from previous epochs to synthesize images based solely on textual control signals, and use the synthesized images as negative instances.


We combine the three tasks' losses to train UFC-BERT, i.e.,
\begin{equation}
\mathcal{L}_{\text{UFC-BERT}} = {\lambda}_1 {\mathcal L}_{\text{MSM}} +  {\lambda}_2 {\mathcal L}_{\text{REL}} + {\lambda}_3 {\mathcal L}_{\text{FDL}},
\end{equation}
where ${\lambda}_1$, ${\lambda}_2$ and ${\lambda}_3$ are set to 1.0, 0.5, and 0.5 to balance the three losses.
The masked sequence modeling task ignores the negative instances from the other two tasks, i.e., irrelevant pairs or unrealistic instances. And the fidelity estimation task is added only after a certain number of epochs.


\subsection{Inference: Progressive Non-Autoregressive Generation}
We design a Progressive Non-Autoregressive Generation (PNAG) algorithm for conditional image synthesis after training, which improves upon Mask-Predict~\cite{ghazvininejad2019mask,guo2020incorporating,cho2020x}. 
Mask-Predict predicts all target tokens when given a fully-masked sequence at the first iteration, and then iteratively re-mask and re-predict a subset of tokens with low probability scores for a constant number of iterations.
However, Mask-Predict cannot ensure the efficacy of multi-modal controls and the fidelity of the synthesized images, and requires determining the number of iterations.
Our PNAG tackles its drawbacks via sample selection and dynamic termination, based on the relevance and fidelity estimators.



Each iteration of our PNAG algorithm consists of a {\it Mask} step and then a {\it Predict} step.
Let ${\bf I}^{(t, in)} = ({I}_1^{(t, in)}, \dots, {I}_{N_I}^{(t, in)})$ 
and ${\bf I}^{(t, out)} = ({I}_1^{(t, out)}, \dots, {I}_{N_I}^{(t, out)})$ 
be the state of the target image's sequence before and after the $t$-th iteration, respectively.
The tokens in ${\bf I}^{(0, out)}$ for $t=0$ is all set to [MASK] except for the positions that are controlled by the preservation signals, i.e., except for ${I}_i^{(0, out)}$ that has $P_i=1$.
If a preservation control is present, i.e.\ $P_i=1$, we always set ${I}_i^{(t, in)}$ and ${I}_i^{(t, out)}$ for all $t$ to be the code-word that corresponds to the provided image block to be preserved.


{\bf Mask Step.} 
At the beginning of iteration $t$ ($t\ge 1$), we construct the input sequence ${\bf I}^{(t, in)}$ by (re-)masking a subset of tokens in the generated sequence ${\bf I}^{(t-1, out)}$ from the last iteration.
Similar to beam search, we construct $B$ parallel input sequences $\{ {\bf I}^{(t, in)}_{1}, \dots, {\bf I}^{(t, in)}_{B} \}$ at each iteration.
Specifically, to produce each ${\bf I}^{(t, in)}_{b}$, we mask $n$ tokens of ${\bf I}^{(t-1, out)}$ by sampling from a multinomial distribution proportional to the probability scores ${\bf Y}^{(t-1)}=\{ {Y}^{(t-1)}_i\}_{i=1}^{N_I}$ (see Equation~\ref{eq:1}).
Here $n = N_I \cdot (\alpha + \frac{T-t}{T-1}\cdot{\beta})$, where $\alpha, \beta$ are two hyper-parameters and $T$ is the maximum possible number of iterations, such that the number of tokens to re-mask gradually decreases after every iteration.



{\bf Predict Step.} Given the control ${\bf C}$ and an input sequence ${\bf I}^{(t, in)}_{b}$, UFC-BERT estimates a distribution $\Pr(\hat{I}_{i}  \vert {\bf I}^{(t, in)}_{b}, {\bf C})$ for each masked position $i$. 
UFC-BERT also estimates the relevance score ${S}^{R}_b$ and fidelity score ${S}^{F}_b$ regarding the image that it is about to synthesize, and summarizes the scores into a comprehensive score ${S}^{(t)}_b = \sigma {S}^{R}_b + (1- \sigma ) {S}^{F}_b $, where $\sigma$ is a coefficient for adjusting the importance of the two.
We perform \emph{sample selection} based on ${S}^{(t)}_b$, i.e., we select the $b$-th sequence ${\bf I}^{(t, in)}_{b}$ with the highest ${S}^{(t)}_b$, discard the others, and then generate ${\bf I}^{(t, out)}$ based on the selected ${\bf I}^{(t, in)}_{b}$ as follows:
\begin{align}
  {I}^{(t, out)}_i \sim  \Pr(\hat{I}_{i}  \vert {\bf I}^{(t, in)}_{b}, {\bf C}),
  \quad\quad
  {Y}^{(t)}_i \leftarrow   \Pr(\hat{I}_{i} = {I}^{(t, out)}_i \vert {\bf I}^{(t, in)}_{b}, {\bf C}), 
  \label{eq:1} 
\end{align}
where each token ${I}^{(t, out)}_i$ is sampled from the multinomial distribution $\Pr(\hat{I}_{i}  \vert {\bf I}^{(t, in)}_{b}, {\bf C})$ and the corresponding probability is assigned to ${Y}^{(t)}_i$.
Note that we predict tokens for \emph{all} masked positions regardless of the predictions' confidence.
We also implement \emph{dynamic termination} based on ${S}^{(t)}_b$.
Specifically, if the current iteration's score ${S}^{(t)}_b$ is higher than $S_{max}$ (initialized as zero), we set $S_{max}$ to ${S}^{(t)}_b$ and record the current iteration as $t_{max}$.
If $S_{max}$ does not increase after three consecutive iterations, we select ${\bf I}^{(t_{max}, out)}$ as the final result and terminate our generation algorithm.


\mjxdel{Instead of unconsciously inference of MNAG, our PNAG method guides the inference towards a better direction, i.e. progressively enhance the relevance and fidelity of synthesized images, where the dynamic termination further avoids redundant/insufficient iterations. 
If we need to synthesize multiple images simultaneously, we sample multiple values and probabilities for each code at the first iteration by Equation~\ref{eq:1}, and then conduct parallel generation during subsequent iterations.
}






\section{Experiments} 

\subsection{Datasets and Hyperparameters}
In experiments, we focus on two practical fields of image synthesis: fashionable clothing and human faces. 
We collect a very large-scale clothing dataset M2C-Fashion with Chinese text descriptions, which contains tens of millions of image-text pairs, much larger than the commonly used text-to-image datasets COCO~\cite{lin2014microsoft} and CUB~\cite{wah2011caltech}. Details of the dataset are provided in the supplementary material. We additionally use another high-resolution facial dataset Multi-Modal CelebA-HQ~\cite{karras2017progressive,xia2020tedigan}.

Following the model setting of VQGAN~\cite{esser2020taming}, we use the $256 \times 256$ image size on the two datasets and transform each image to a discrete sequence of $16 \times 16$ codes, where the codebook size $\vert \mathcal{Z} \vert$ is set to 1024. For the BERT model, we set the number of layers, hidden size, and the number of attention heads to 24, 1024, and 16, respectively. Our UFC-BERT has 307M parameters, same as the Transformer used by VQGAN. 
As for hyper-parameters of PNAG, we set the parallel decoding number $B$ to 5 and the balance coefficient $\sigma$ to 0.5. We set the initial mask ratio ${\alpha}$, the minimum mask ratio $\beta$, and the maximum iteration number $T$ to 0.8, 0.2, and 10, respectively. 



\begin{figure*}[t]
\centering
\includegraphics[width=1\textwidth]{multicontrols}
\vspace{-0.4cm}
  \caption{Images generated by our UFC-BERT under various combinations of textual controls (TC), visual controls (VC), and preservation controls (PC).
  Please see the supplemental material for more showcases, where we also include a study on the diversity of the images generated by UFC-BERT and analyze how the multiple control signals interfere with each other.
  }
\label{fig:controls}
\end{figure*}


\begin{figure*}[htp]
\centering
\includegraphics[width=1\textwidth]{multivisual}          
\vspace{-0.2cm}
  \caption{Image synthesis with multiple visual controls, where we crop regions from $2\sim3$ images to serve as the visual controls. UFC-BERT synthesizes images that naturally fuse the visual elements.}
\label{fig:multivisual}
\end{figure*}



\subsection{Flexibility of Multi-Modal Controls for Conditional Image Synthesis}
In this section, we qualitatively verify the synthesis ability of UFC-BERT with three modalities of control signals, i.e., textual, \mjxdel{unaligned} visual, and preservation controls. 
The textual controls are the texts paired with the images, which are already provided by the two datasets, while the\mjxdel{ unaligned} visual controls are code sequences of cropped regions, e.g. regions that represent logos or texture of clothes.
\mjxdel{Unlike spatially aligned visual signals such as a sketch map~\cite{ghosh2019interactive,xia2019cali} or semantic map~\cite{isola2017image,wang2018high}, unaligned visual signals are hardly studied in the previous works, which requires learning the reasonable utilization of the given visual elements without the spatial alignment.}






In Figure~\ref{fig:controls}, we synthesize images conditioned on combinations of the three types of control signals. The results demonstrate UFC-BERT can unify any number of multi-modal controls to synthesize high-quality images. 
Further, UFC-BERT supports one or multiple visual controls for more flexible synthesis, as shown in Figure~\ref{fig:multivisual} where we generate images given 2$\sim$3 visual controls. 
We observe that UFC-BERT can reasonably fuse multiple visual elements and produce a harmonious image.







\subsection{Quantitative Comparison to Existing Methods for Text-to-Image Synthesis}
In this section, we investigate how our UFC-BERT quantitatively compares to existing models. Considering most existing methods only utilize one control signal, we select the most common and challenging task {\it text-to-image synthesis} to compare the synthesis ability. 

\begin{table}[tp]
    \centering
   
    \caption{Comparisons with GAN baselines for text-to-image synthesis on Multi-Modal CelebA-HQ.}
              \vspace{-0.4cm}

    \begin{adjustbox}{max width = \textwidth}
    \begin{threeparttable}
    \label{table:gan}
   
        \begin{tabular}{c|cccccc}
            \toprule
            Method & AttnGAN~\cite{xu2018attngan} & ControlGAN~\cite{li2019controllable} & DF-GAN~\cite{tao2020df}& DM-GAN~\cite{zhu2019dm} & TediGAN~\cite{xia2020tedigan} & UFC-BERT (our) \\ 

            \midrule
              FID $\downarrow$ &125.98&116.32&137.60&131.05&106.37&{\bf 66.72} \\
            \midrule
              LPIPS $\downarrow$  &0.512&0.522&0.581&0.544&{ 0.456}&{\bf 0.448} \\
            \bottomrule

        \end{tabular}
   
      \end{threeparttable}
  \end{adjustbox}
\end{table}



\begin{table}[!tp]
    \centering
   
    \caption{Comparisons with the autoregressive two-stage method VQGAN for text-to-image synthesis. $\downarrow$ means the lower the better, while $\uparrow$ means the opposite. We evaluate speed on the same V100 GPU.}
          \vspace{-0.4cm}

    \begin{adjustbox}{max width = \textwidth}
    \begin{threeparttable}
    \label{table:ar}
   
        \begin{tabular}{cc ccc ccc c}
            \toprule
               \multirow{2}{*}{Datasets} & \multirow{2}{*}{Methods} & \multicolumn{4}{c}{Automatic Metrics} & \multicolumn{2}{c}{Human Pairwise Study} & \multirow{2}{*}{Inference Speed} \\
              \cmidrule(lr){3-6} \cmidrule(lr){7-8}
             & & FID$\downarrow$  & LPIPS $\downarrow$  & PSNR$\uparrow$ & SSIM$\uparrow$& Relevance & Fidelity& \\
            \midrule
             \multirow{2}{*}{M2C-Fashion} 
              &VQGAN (AR) &12.48&0.483&10.80&0.56&38.6\%&44.2\%& 8.73 sec/sample\\
            \cmidrule(lr){2-9}
              &UFC-BERT (NAR) &{\bf 11.53}&{\bf 0.461}&{\bf 13.14}&{\bf 0.58}&{\bf 61.4\%}&{\bf 55.8\%}& {\bf 0.81} sec/sample \\
            \midrule

             \multirow{2}{0.15\textwidth}{\centering Multi-Modal CelebA-HQ}
              &VQGAN (AR) &{\bf 52.63}&{0.503}&{8.98}&{0.28}&{42.7\%}&{46.9\%}& 8.66 sec/sample\\
            \cmidrule(lr){2-9}
              &UFC-BERT (NAR) &{66.72}&{\bf 0.448}&{\bf 9.56}&{\bf 0.29}&{\bf57.3\%}&{\bf 53.1\%}& {\bf 0.79} sec/sample \\
            \bottomrule

        \end{tabular}
   
      \end{threeparttable}
  \end{adjustbox}
       \vspace{-0.4cm}

\end{table}





First, we compare our UFC-BERT with GAN-based text-to-image models AttnGAN~\cite{xu2018attngan}, ControlGAN~\cite{li2019controllable}, DF-GAN~\cite{tao2020df}, DM-GAN~\cite{zhu2019dm} and TediGAN~\cite{xia2020tedigan} on the Multi-Modal CelebA-HQ dataset. For evalution, we adopt two automatic metrics FID~\cite{heusel2017gans} and LPIPS~\cite{zhang2018unreasonable}.
We report the results on Table~\ref{table:gan} and our UFC-BERT achieves the best performance on the two metrics, even outperforming the TediGAN that uses slow and complex instance-level optimization. This demonstrates the two-stage architecture and non-autoregressive generation of UFC-BERT are suitable for text-to-image synthesis.




\begin{wrapfigure}{r}{0.6\textwidth}
    \vspace{-0.4cm}
    \includegraphics[width=0.6\textwidth]{ARNAR}

    \setlength{\abovecaptionskip}{-0.1cm}  
    \setlength{\belowcaptionskip}{-0.1cm} 
    \caption{Typical examples of UFC-BERT and VQGAN for text-to-image synthesis, including a counterfactual case.}
    \label{fig:arnar}
   
\end{wrapfigure}


Besides, we compare our UFC-BERT with the autoregressive two-stage method VQGAN  from three aspects: (i) the automatic metrics FID for image quality, as well as LPIPS, PSNR~\cite{wang2004image} and SSIM~\cite{wang2004image} for the similarity between the generated image and the ground truth; (ii) the Relevance and Fidelity metrics are evaluated through a user study, where the users are asked to judge which model's output is more relevant to the textual descriptions, and more photo-realistic; (iii) the synthesis speed of the two approaches.
Note that the autoregressive inference implementation of VQGAN has been optimized by caching the preceding computation as in Transformer-XL~\cite{dai2019transformer},
and UFC-BERT and VQGAN have the same parameter number (307M) for fair comparison.
For the user study, the two models receive the same textual signals, and each generates 50 images.
We collect the pairwise comparison results from five volunteers.

As shown in Table~\ref{table:ar}, our UFC-BERT achieves better performance for almost all criteria with about 11$\times$ speedup.
This suggests our non-autoregressive UFC-BERT with progressive NAR generation algorithm can synthesize high-fidelity images relevant to textual descriptions.
As for the FID metric, UFC-BERT outperforms VQGAN on M2C-Fashion, but has worse performance on Multi-Modal CelebA-HQ, it may be due to the fact that the autoregressive VQGAN can more easily memorize the pattern of a small dataset (only 30,000 facial images).
In Figure~\ref{fig:arnar}, we further show typical generated examples to intuitively display the difference between the two approaches, including a case of counterfactual generation. We find that UFC-BERT can synthesize high-quality images, even for the counterfactual case.













\subsection{The Effectiveness of Our Progressive NAR Generation Algorithm}
\vspace{-0.1cm}

In this section, we first visualize in Figure~\ref{fig:pnag} the iterative process of our PNAG inference method based on the relevance and fidelity estimators. The images with red bounding boxes are the final outputs that match the textual control signals.
We can find that the fidelity and relevance of the images increase after a few iterations, verifying our PNAG algorithm can guide the inference process towards a better direction
and synthesize more realistic images that match the control signals.

\begin{figure}
\centering
\includegraphics[width=1\textwidth]{PNAG}
\vspace{-0.6cm}
\caption{The iterative inference process of our PNAG algorithm. The red bounding box means the image has the highest comprehensive score and is selected as the final output result.}\label{fig:pnag}
\end{figure}


\begin{table}[t]
    \centering
   
    \caption{Ablation studies of our PNAG inference algorithm. PNAG(w/o.\ REF) and PNAG(w/o.\ FDL) set $B$ to the default value 5. MNAG is the original Mask-Predict algorithm~\cite{ghazvininejad2019mask}.}
    \vspace{-0.3cm}
    \begin{adjustbox}{max width = \textwidth}
    \begin{threeparttable}
    \label{table:pnag}
   
        \begin{tabular}{cccccccc}
            \toprule
            \multirow{1}{*}{Dataset}   &  \multirow{1}{*}{Metrics}& MNAG~\cite{ghazvininejad2019mask}& PNAG(w/o. REF) & PNAG(w/o. FDL)&  PNAG($B$=1) & PNAG($B$=5)& PNAG($B$=10)\\ 
           
            \midrule
            \multirow{3}{*}{M2C-Fashion} &  FID $\downarrow$ &14.77&12.17&13.14&12.72&11.53&{\bf 11.14} \\
            \cmidrule(lr){2-8}
              &  LPIPS $\downarrow$ &0.488  &0.477&0.469&0.479&0.461& {\bf 0.456} \\
            \midrule
            \multirow{2}{0.15\textwidth}{\centering Multi-Modal CelebA-HQ} &  FID $\downarrow$ &72.04&68.90&70.32&69.49&66.72&{\bf 65.30} \\
            \cmidrule(lr){2-8}
              &  LPIPS $\downarrow$  &0.514&0.469&0.463&0.475&0.448& {\bf 0.445} \\
            \bottomrule

        \end{tabular}
   
      \end{threeparttable}
  \end{adjustbox}
\end{table}



We then conduct ablation studies of PNAG. As shown in Table~\ref{table:pnag}, we develop three ablated inference methods PNAG(w/o. REF), PNAG(w/o. FDL) and MNAG, where PNAG(w/o. REF)  and PNAG(w/o. FDL) discard the relevance estimator and the fidelity estimator, respectively, and MNAG is the original Mask-Predict method~\cite{ghazvininejad2019mask} without any estimator. The results demonstrate that the two estimators effectively utilize the discriminative capability of UFC-BERT and do help improve the synthesis quality. Additionally, we vary the crucial hyper-parameter of PNAG $B$ (i.e.\ the parallel decoding number during inference) from 1 to 10, and the results in Table~\ref{table:pnag} show that a larger $B$ is beneficial to the synthesis quality\mjxdel{ while affecting the synthesized speed}.

\vspace{-0.4cm}
\section{Conclusions}
\vspace{-0.2cm}

We proposed UFC-BERT to unify any number of multi-modal controls in a universal form for conditional image synthesis. We utilized non-autoregressive generation to improve inference speed, enhance holistic consistency, and support preservation controls. Further, we designed a progressive generation algorithm based on relevance and fidelity estimators to ensure relevance and fidelity.




\medskip


{\small
\bibliographystyle{ieee_fullname}
