\section{Introduction}

We take inspiration from the recent work of DALL-E 2\cite{Ramesh2022-ai} which trains a denoising diffusion model\cite{Song2020-cd,Ho2020-dk,Sohl-Dickstein2015-hz} to generate CLIP image embeddings from CLIP text embeddings. We generalise this approach to train a diffusion model to generate latent codes for a generative model conditional on CLIP image embeddings. 

In effect, we map between the latent spaces of two different pre-trained models to enable controllable generation from a previous unconditional generative model. In this case we choose to generate StyleGAN2/3\cite{Karras2019-sj,Karras2021-sg} latent codes conditional on CLIP\cite{Radford2021-ru} image/text embeddings to enable text conditional synthesis using the pre-trained StyleGAN model. As both the text encoder and image synthesiser are frozen this method is quick to train, and by leveraging the shared embedding space of images and text in CLIP we are able to train it with no external data.

At inference time our model generates high-quality and high-resolution outputs, well aligned to a text prompt, with minimal images artefacts. We also show that we can benefit from the hierarchical and controllable nature of StyleGAN's existing latent space which can further be used to exert control over the generated images, see Section \ref{sec:using}.

In summary, our main contributions are:
\begin{itemize}
  \item A conditional diffusion model that connects pre-trained CLIP and StyleGAN models into a text-to-image model.  
  \item A training scheme that requires no text-image pairs.
  \item A model which is quick to train, and can produce mega-pixel samples in under a second.
\end{itemize}

Source code and trained models used in this work are available at the following url: \url{https://github.com/justinpinkney/clip2latent}.

\section{Related work}

\subsection{Text-to-image generation}

Recent months have seen a tremendous advance in the quality of text-to-image generative models. The majority of these models are trained using large amounts of text-image paired data where, image generation is conditioned on the text input. To expedite the training and improve the quality of generation, several of the state of the art approaches make use of pre-trained models. Some use a pre-trained text encoder to leverage the semantic knowledge captured by large-scale text\cite{Saharia2022-ry} or multi-modal pre-training\cite{Ramesh2022-ai}. Others have used a pre-trained image auto-encoder to reduce the effective spatial dimensionality required for acceptable resolution\cite{Rombach2021-bu,Gu2021-tt} and in some case to work with an image representation more amenable to the image synthesis model\cite{Ramesh2021-ro,Ding2022-gr}.  In contrast to our work however, the actual image synthesis model rarely makes use of an entirely pre-trained generator. 

Early work in leveraging only pre-trained models was inspired by feature visualisation approaches\cite{Olah2017-lu}, applying a variety of image parameterisations and regularisation allowed a successful iterative optimisation to maximise the similarity of an image to a given text prompt\cite{murdock1}. As well as conventional image parameterisations, this approach has also been applied to generative models where the latent code is directly optimised to produce model outputs which best match a given text prompt\cite{murdock2}. This approach is conceptually closest to ours, where we replace the time consuming iterative optimisation with sampling from a conditional diffusion model which generates latent codes. 

Another notable approach which leverages only pre-trained models is CLIP guided diffusion, where the classifier guidance approach for sampling a diffusion model can be applied using CLIP as the classifier\cite{crowson1}. However, this classifier guidance approach has since been shown to be limited by the lack of pre-trained CLIP models trained on noised images and both approaches are outperformed by classifier-free guidance for text-to-image generation\cite{Nichol2021-be}.

\subsection{Text driven manipulation of pre-trained StyleGAN}

Although there has been limited work in using a pre-trained StyleGAN purely for text-to-image generation, there has been exploration of text driven \textit{editing} using StyleGAN models. StyleCLIP\cite{Patashnik2021-zm}, CLIP2StyleGAN\cite{Abdal2021-lv}, StyleMC\cite{Kocasari2021-wv}, and others\cite{Xu2021-ev,Gabbay2021-go,Tzelepis2022-fj} use a variety of methods to extract editing directions in the latent space of a pre-trained StyleGAN based on text descriptions via CLIP. Although not the main application of our work, we note that our approach can also be applied to discovery of edit directions and show a small proof-of-concept example in Appendix F.

Finally, StyleGAN-Nada\cite{Gal2021-ir} performs domain adaptation of a pre-trained StyleGAN model by fine-tuning the generator based on a text description of the desired domain encoded by CLIP. Mind the Gap\cite{Zhu2021-bw} takes a similar approach, but uses CLIP image encodings to perform domain adaptation to a new visual style. Unlike our approach this requires fine-tuning of the generator which modifies its entire output to match a single text/image prompt without adding any further text based control.

\subsection{Language-Free training}

One of our key contributions is the ability to train an text-to-image model without any text labelled training data. Two previous works have shown similar results. LAFITE\cite{Zhou2021-dl} trains a GAN conditioned on CLIP image embeddings, we compare our results to their language-free text-to-image model in Section \ref{sec:results}. Note the method of LAFITE requires training of the generator from scratch, whereas our method freezes the pre-trained StyleGAN generator. CLIP-GEN\cite{Wang2022-cf} similarly trains an auto-regressive transformer model to predict VQGAN tokens conditioned only on CLIP image embeddings and show that this approach generalises to CLIP text embeddings. In essence our method takes this, and similar\cite{Rombach2021-bu,Ramesh2021-ro}, approaches to the extreme: predicting a single StyleGAN latent vector rather than VQGAN tokens. This enables our model to be far smaller (48M vs 307M-1.6B parameters) and train much faster (500k vs 2M iterations) in exchange for being constrained closely to the original domain of the generative model.

\section{Approach}

Our approach is to map the latent spaces of pre-trained CLIP and StyleGAN models using a diffusion model we call clip2latent. 

StyleGAN is a powerful generative model which can be sampled with a normally distributed latent vector $z \in\mathbb{R}^{512}$. From this vector StyleGAN applies a 'mapping network' to map to a second latent space, $w \in\mathbb{R}^{512}$, finally this $w$ vector is used to generate an image. To generate our dataset we randomly sample $z$ latents and generate images from these with StyleGAN. Next we use the image encoder component of CLIP to encode StyleGAN generated images into the CLIP embedding space $e_i \in\mathbb{R}^{512}$. After encoding the StyleGAN generated images using CLIP, we can discard the generated images and original $z$ latents to leave our training data $(w, e_i)$ tuples, see Figure \ref{fig:training}a.

We then train the clip2latent model using the same approach as the diffusion based prior from DALL-E 2\cite{Ramesh2022-ai}, Figure \ref{fig:training}b. Briefly we train a Gaussian diffusion model to generate $w$ vectors conditioned on $e_i$ embeddings. As in DALL-E 2 we train a transformer based model on a sequence consisting of: the CLIP image embedding $e_i$, an embedding representing the timestep in the diffusion process, the noised $w$ latent vector, and a final learned embedding whos output predicts the denoised $w$.

Once we have trained clip2latent, we rely on the fact that CLIP can embed images and text into a shared latent space. At inference time we generate a CLIP embedding for a text description ($e_t$) and use this as the conditioning to generate a StyleGAN latent vector, from which we can create a high-resolution image using the StyleGAN generator, illustrated in Figure \ref{fig:training}c.

\begin{figure}[htbp]
    \centering
    \includegraphics[width=\textwidth]{figs/schematic.png}
    \caption{A schematic diagram of: a. synthetic data generation b. clip2latent diffusion model training, and c. text-to-image inference. Triangle and rectangle blocks indicate models, vertical striped bars indicate latent vectors or embeddings. $z$ and $w$ are the StyleGAN $z$ and $w$ latent spaces, $e_i$ and $e_t$ denote the CLIP embeddings space for images and text respectively.}
    \label{fig:training}
\end{figure}

\section{Training details}

\subsection{Data generation}

To generate paired training samples without the need for external images and text labels we use the pre-trained StyleGAN 2 model (config-F) trained on FFHQ\cite{Karras2019-sj}. We sample 1 million random latent vectors with no truncation and generate the CLIP embeddings for each image using the ViT-B/32 pre-trained CLIP\cite{Radford2021-ru}. We resize the generated StyleGAN image to 224x224 pixels and apply the CLIP image normalisation. For each image we generate a single embedding and do not perform any data augmentation on the generated images.

\subsection{Latent prior training}

We then train a denoising diffusion model to generate the latent vectors conditioned on the CLIP image embeddings. We choose the same architecture as the prior in DALLE-2\cite{Ramesh2022-ai} and use the open source implementation DALL-E 2-Pytorch\cite{wang1}. Our diffusion model is analogous to that presented in DALLE-2 where the CLIP image embeddings are replaced by the StyleGAN latent codes and the CLIP text embedding are replaced by the corresponding CLIP image embeddings, as such we could consider this model to be a \emph{latent prior}.

During training we substitute the image embeddings with zeros with a probability of 0.2 to allow us to use classifier-free guidance\cite{Ho2021-ck} during inference. Rather than directly use the $w$ latent vectors for training we subtract the mean and divide by the estimated standard deviation of samples from $w$ space to better match the variance expected by the diffusion model. This also aligns with the common practice of learning StyleGAN latent prediction models relative to the mean latent vector\cite{Richardson2020-og}.

To improve the ability of the model to generalise to text embeddings and avoid over-fitting on image embeddings we take the approach introduced in LAFITE\cite{Zhou2021-dl} where "pseudo-text" embeddings are generated from image embeddings using scaled Gaussian noise, see Section \ref{sec:noise} and Appendix A.1.

We train our diffusion model for 1 million iterations at a batch size of 512, for full hyperparameter details see Appendix A. We select the best checkpoint as judged by generating 4 samples each for a set of 16 text prompts and computing the mean CLIP cosine similarity score between the text embedding and the CLIP image embedding of the generated image. In practice, we find that this peak validation score generally occurs at around 500 thousand iterations which takes approximately 9 hours on a single A100-80GB GPU.

\subsection{Text-to-image synthesis}

At inference time we generate the CLIP embedding for a text input and use this as the condition for the clip2latent network. By performing denoising diffusion, the clip2latent network generates a StyleGAN latent code which can then be used for image synthesis. During the denoising process we optionally employ timestep respacing\cite{Nichol2021-jd} and super conditioning\cite{Ho2021-ck} to decrease the sampling time and increase the CLIP similarity respectively. We do not employ any clipping or renormalisation of the denoised outputs or any of the recent method to reduce super-conditioned artefacts\cite{Karras2022-wf,Wang2022-ic}. In practice, with our shortest denoising schedule, end-to-end text to image synthesis can be performed in under a quarter of a second on a single A100 GPU for a single image (see Table \ref{table:compare}). As our image generation pipeline is fast and scales well to batching, we employ CLIP re-ranking to obtain the best CLIP similarity scoring sample from a batch of 16 candidates.

\section{Results}
\label{sec:results}

\begin{figure}[tbp]
    \centering
    \makebox[\textwidth][c]{\includegraphics[width=\textwidth]{figs/compare-wdalle.jpg}}
    \caption{Sample generations from our model (clip2latent) compared to direct optimisation of the StyleGAN latent vectors, LAFITE, and DALLE-2. For details of the comparison methods see Appendix C. Note, LAFITE generates samples at 256x256 pixels, whereas clip2latent and direct optimisation generate samples at 1024x1024. All prompts were prefixed with "A photograph of".}
    \label{fig:compare}
\end{figure}


\renewcommand{\arraystretch}{1.2}
\begin{table}

    \centering
    \caption{CLIP similarity scores and inference times for our methods compared to previous work. Similarity scores are measured by generating samples for 64 text prompts and measuring the mean CLIP similarity score. For a full list of prompts see Appendix B, some samples could not be run in DALLE-2 due to the content policy. Inference times are measured using and A100-80GB PCIe GPU, for generation of a single sample including CLIP re-ranking and do not take advantage of any batching. We note the DALLE-2-pytorch library does not batch the two calls to forward when using classifier free guidance and could be optimised further. Inference times are not included for DALLE-2 as it is not available to run on comparable hardware.}
    \label{table:compare}

    \begin{tabular}{lS[table-format=1.3]S[table-format=2.3]} 
     \toprule
     \textbf{Method} & \textbf{CLIP score} & \textbf{run-time (s)}  \\ [0.5ex] 
     \midrule
     clip2latent (ours)  & 0.316 & 11.760 \\
     clip2latent (ours) + timestep respacing   & 0.315 & 0.244 \\
     Direct Optimisation & 0.321 & 19.191  \\ 
     LAFITE & 0.278 & 0.045 \\ 
     DALLE-2 & 0.291{$^{*}$} & \textemdash \\
     \bottomrule
    \end{tabular}

\end{table}
    
Our method produces high quality, high resolution images based on text prompts without the need for paired image-text data during training. Figure \ref{fig:compare} shows examples of text-to-image generation via our method, compared against LAFITE and direct optimisation\cite{nshepperd1} of the $w$ vector (for further details of the baseline comparison methods see Appendix C). We judge the similarity of generated image and text prompts using the standard CLIP similarity score, and show that our approach acheives CLIP socres close to optimisation methods whilst largely avoiding image artefacts and performing inference more rapidly, see Table \ref{table:compare}.

\subsection{Importance of embedding augmentation}
\label{sec:noise}

Although CLIP encodes both images and text into a shared latent space it has been noted that the encodings for the two modalities are disjoint in this space\cite{hoppe1,Liang2022-cv}. As we wish to train a text to image model with no external data we are restricted to using the CLIP image embeddings of StyleGAN generated images. Although, previous works has shown that simply training models conditional on image embeddings and substituting text embeddings at inference time is sufficient to generate good quality results \cite{Wang2022-cf,Ramesh2022-ai}, we find the the addition of scaled Gaussian noise to the condition embedding vectors (as suggested in LAFITE\cite{Zhou2021-dl}) is crucial in allowing the model to generalise across both image and text embeddings produced by CLIP. 

Although the authors of LAFITE describe this as a method for producing "psuedo text encodings", we consider this to be more analogous to a method of data augmentation, noting that the "pseudo text" embeddings and original image embedding have a similarity score of around 0.8 (given a noise scaling of 0.75 using in LAFITE), far higher than any real text embedding with a well matched image embedding (typically in range 0.3-0.4\cite{hoppe1}). Without the addition of noise, the model learns to produce latent codes corresponding very closely to the input image, but fails to generalise to text embeddings. At very high noise levels the conditioning is less informative and the model performance deteriorates. We find the optimal level of noise scaling to be 1.0, see Appendix Figure 1.

\subsection{Making use of StyleGAN}
\label{sec:using}

\begin{figure}[htbp]
    \centering
    \includegraphics[width=\textwidth]{figs/using-sg.jpg}
    \caption{Demonstrating the application of existing StyleGAN editing techniques to add additional levels of control. a: Artefact correction using truncation, (top row original, bottom row truncation 0.8) for images of "a British politician laughing happily", "a Nigerian professor of economics", "person with very tight curly blonde hair". b: Style mixing to create variations of colour, lighting, and texture for "a university graduate". c: Age, Pose, and Smile editing using InterfaceGAN directions on a generated sample (top-left) from the prompt "an arctic explorer". }
    \label{fig:using}
\end{figure}


The image synthesis portion of our text-to-image pipeline uses the well studied and understood StyleGAN generator. This means we can leverage many of the well known and favourable properties of StyleGAN to exert further control over our generated images. In the following section we demonstrate how our method can take advantage of: truncation for artefact removal, style mixing for improving diversity, and the well-known editability of StyleGAN's latent space.

\subsubsection{Truncation}

We find that at high guidance scales our latent vectors can sometimes fall outside the typical domain of StyleGAN latent space, generating unnatural artefacts. One method to alleviate this is to use the well-known technique of truncation\cite{Brock2018-at} to move the generated latent closer to the mean latent vector. This trades off image-text similarity and diversity for image fidelity, see Figure \ref{fig:using}a. Although in some cases this can actually \textit{increase} the CLIP similarity score as reducing artefacts brings the image closer to the domain of real face image.

\subsubsection{Style Mixing}

We can also take advantage of the extended latent space of StyleGAN ($w+$) where the latent vector for each layer controls the generated image at a different resolution scale\cite{Karras2019-sj}. One straightforward way to utilise this structure is to add extra colour and lighting diversity to our generated images by performing Style Mixing with our generated latent vector for the lower resolution layers which control most of the semantic appearance of the generated image, with a randomly sampled latent for higher resolution layers which predominantly control colour and lighting, see Figure \ref{fig:using}b.

\subsubsection{Latent Editing}

Finally we can also leverage the extensive literature on finding semantically meaningful edits in StyleGAN latent space to edit our generated images, adding an extra level of control to our text-to-image pipeline. This allows us to make use of the wealth of existing high-quality facial editing directions available for StyleGAN. We show the application of some well known InterfaceGAN\cite{Shen2020-ur} directions to our generated images in Figure \ref{fig:using}c. It has previously been shown that high-quality edits require latent vectors to lie well within the typical domain of StyleGAN's latent space\cite{Tov2021-le}. The generative nature of the clip2latent diffusion model ensures that text-generated latents are within the domain of StyleGAN without the need for additional losses such as the "latent discriminator" in \textit{Tov et al.}\cite{Shen2020-ur}. This ensure that our method can generate high-quality edits with few artefacts, in contrast to alternatives such as direct optimisation which generates latents with poor editability, see Appendix Figure 2.

\section{Discussion}

We note our method is not restricted to StyleGAN in particular and in theory could be applied to any generative model. In practice our method appears to favour models with a relatively low dimensional latent space (see Appendix E) and clearly is well suited to those models for which high quality pre-trained networks are available. To demonstrate the generality of our method beyond faces we also train a text-to-image model using a 256x256 StyleGAN3 landscape model\cite{Pinkney_Awesome_Pretrained_StyleGAN3} trained on the LHQ dataset\cite{Skorokhodov2021-kw}, see Figure \ref{fig:headline}, bottom row. Throughout this work we have chosen the StyleGAN family of architectures due to the wide range of pre-trained models available\cite{Pinkney_Awesome_pretrained_StyleGAN2} and its high-resolution, fast inference and state of the art performance in many domains\cite{Sauer2022-bi,Kang2022-ax}. However, there is no reason a different GAN (e.g. BigGAN\cite{Brock2018-at}) or entirely different class of generative model (e.g. VAE\cite{Child2020-rx}) couldn't also be used.

Although we have chosen to take a simple data-free approach to training our model, which does not rely on any exist text-image paired data, there is now wide availability of such datasets across a variety of domains thanks to the open-source efforts of LAION\cite{Schuhmann2021-qv}.  To make use of such paired data would require embedding the real-world images into the latent space of StyleGAN for which there exist many methods\cite{Richardson2020-og,Abdal2019-nj,Alaluf2021-xu}. We leave this as an avenue for exploration for future work.

Our generated images are constrained to lie withing the $w$ space of StyleGAN, although this can produce a large variety of images its expensiveness is limited. Previous work has shown that the $w+$ space is vastly more capable in what images it can represent\cite{Abdal2019-nj}. We show some limited exploration in applying our model to other StyleGAN latent spaces in Appendix E, but leave extensions to the other StyleGAN latent spaces to future work.

As well as generalisation to other generative model latent spaces we believe that the application of powerful diffusion models can allow the addition of conditional sampling to previously unconditional models based on any image encoding, for example facial recognition/attribute networks or other classification models. We look forward to future applications of diffusion models as tools for arbitrarily mapping between latent spaces of pre-trained models. 

\section{Conclusion}

Our method allows sampling from a pre-trained generative model conditioned on CLIP image and text embeddings, providing a lightweight way to train and inference an effective text-to-image generation model without access to external data. Our method generates high quality results with satisfactory CLIP scores whilst largely avoiding image artefacts from earlier methods. We hope our work sheds a light on how the connection between text and image enables more powerful use of pre-trained generative models.

