\subsubsection*{Author Contributions}

\section*{Acknowledgments}
We would like to thank Niki Parmar for initial discussions. Special thanks to Gabriel Bender and Thang Luong for reviewing the paper and providing constructive feedback. We appreciate the efforts of Kevin Murphy and David Fleet for advising the project and providing feedback throughout. We are grateful to Evan Rapoport, Douglas Eck and Zoubin Ghahramani for supporting this work in a variety of ways. Tim Salimans and Chitwan Saharia helped us with brainstorming and coming up with shared benchmarks. Jason Baldridge was instrumental for bouncing ideas. Alex Rizkowsky was very helpful in keeping things organized, while Erica Moreira and Victor Gomes ensured smooth resourcing for the project. Sarah Laszlo and Kathy Meier-Hellstern have greatly helped us incorporate important responsible AI practices into this project, which we are immensely grateful for. Finally, Blake Hechtman and Anselm Levskaya were generous in helping us debug a number of JAX issues.









\section{Related Works}

This paper is closely related to auto-regressive methods for text conditioned image and video generation. DALL-E~\cite{DALLE} \textit{translates} text tokens to discrete image embeddings learnt using a VQVAE~\cite{vqvae}. Parti~\cite{PARTI} has a similar architecture but can generate higher quality images by predicting tokens from a ViT-VQGAN~\cite{vit_vqgan} using a 21B parameters transformer. Similar architectures have been used for generating videos as well. GODIVA~\cite{godiva} uses a transformer to map text tokens to video tokens from a image based VQVAE. Given the large number of tokens from multiple frames, GODIVA relied on a local-attention mechanism. Similarly, NUWA~\cite{nuwa} and NUWA-Infinity~\cite{wu2022nuwa} both employ auto-regressive architectures to generate videos and images from text. NUWA generates fixed size outputs, while NUWA-Infinity introduces a second layer of auto-regressive computation to support variable size videos. Likewise, CogVideo~\cite{cogvideo} argues the main reason behind low quality video generation is the scarcity of good text-video data and tried to leverage pre-trained text to images models to generate high quality video. 

While Phenaki\xspace sticks to the same architecture principles, it has major differences with previous work. Most notably, NUWA, NUWA-Infinity and CogVideo treat videos as a sequence of independent images. This can lead to poor modeling of dynamics and generate motion artifacts. To combat this, NUWA-infinity used the previous frame during decoding to combat this. In Phenaki\xspace, we go further and treat videos as a temporal sequence of images which substantially decreases the number of video tokens given the redundancy in video generation, and results in a much lower training cost. The auto-regressive nature of the \model also allows us to effectively condition on previous frames and generates longer videos as detailed in Section~\ref{sec:method}.




Diffusion models are another class of models which recently have been used for conditional and unconditional video generation, which we call VDM~\cite{ho2022video}. In VDM, authors proposed replacing the conventional U-Net architectures for 2D image modeling with a 3D space-time model to run the diffusion process directly on pixels.
While this approach provides an effective formulation for modeling videos, it is limited to fixed size videos. To address this issue, VDM provides an auto-regressive extension, which allows the model to generate longer videos but it is typically impractical due to high sampling time of diffusion models.

Text conditional video generation is a relatively new field of research, nonetheless, image conditional video generation, commonly known as video prediction, and unconditional video generation have been studied more comprehensively. These papers include deterministic methods using a combination of recurrent and convolutional networks~\cite{ranzato2014video,srivastava2015unsupervised,finn2016unsupervised, wang2017predrnn}, variational based stochastic methods~\cite{babaeizadeh2017stochastic,svg,villegas2019high,fitvid} and more recently by learning a discrete representation~\cite{videovqvae,rakhimov2020latent,transframer}, auto-regressive models~\cite{videotranf,videogpt,ccvs,harp}, diffusion models~\cite{mcvd,harvey2022flexible,yang2022diffusion,diff4pred} flow based models~\cite{kumar2019videoflow}, and finally adversarial based methods~\cite{vondrick2016generating,saito2017temporal,tulyakov2018mocogan,clark2019adversarial,saito2020train,trIVD}. These works mostly consider limited domain (e.g. robotic videos) prediction/generation, or short fixed size clips. Section~\ref{sec:exp} provides comparison with some of these models.

\section{Hyper-Parameters}
\begin{table}[hbtp]
    \centering
    \begin{tabular}{l|c|l}
         Symbol & Value & Description \\\hline
         $t_x, w_x, h_x, c_x$ & $11, 128, 128, 3$ & Video dimensions \\
         $t_p, w_p, h_p, c_p$ & $2, 8, 8, 3$ & Patches dimensions (all frames except the first one) \\
         $t_z, w_z, h_z$ & $6, 16, 16$ & Video tokens dimension (before linear projection) \\
         $h_z$ & 512 & Hidden size in the transformer layer \\
         $d_z$ & 32 & Embedding dimension (after linear projection) \\
         $-$ & 4 & Number of layers for spatial transformer \\
         $-$ & 4 & Number of layers for temporal transformer \\
         $-$ & 2048 & MLP size \\
         $|E|$ & 8192 & Codebook size \\ 
         \hline
         - & AdamW &  Optimizer \\
         $\beta_1$ & 0.9 & first moment of gradient \\
         $\beta_2$ & 0.99 & second moment of gradient \\
         - & 1e-4 & Learning rate \\ 
         - & 1e-4 & Weight decay \\ 
         - & Cosine decay & Learning rate scheduler \\ 
         - & 1M & Target number of training steps for learning rate scheduler \\
         - & 100K & Warmup steps \\
         - & 10 & Gradient clipping magnitude \\
         - & 1028 & Batch size \\
         
    \end{tabular}
    \caption{Hyperparamters used for C-\vivit\xspace architecture and optimizer.}
    \label{tab:hparams_cvivit}
\end{table}

\begin{table}[hbtp]
    \centering
    \begin{tabular}{l|c|l}
         Symbol & Value & Description \\\hline
         $|\mathbf{z}|$ & 1536 & Sequence Length \\
         - & 24 & Number of layer \\
         - & 2048 & Embedding dimension \\
         - & 8192 & MLP dimension \\
         - & 32 & Number of heads \\
         \hline
         - & AdamW &  Optimizer \\
         $\beta_1$ & 0.9 & first moment of gradient \\
         $\beta_2$ & 0.99 & second moment of gradient \\
         - & 1e-4 & Learning rate \\ 
         - & 1e-4 & Weight decay \\ 
         - & Cosine decay & Learning rate scheduler \\ 
         - & 4M & Target number of training steps for learning rate scheduler \\
         - & 10K & Warmup steps \\
         - & 10 & Gradient clipping magnitude \\
         - & 512 & Batch size \\
    \end{tabular}
    \caption{Hyperparamters used for MaskGIT architecture and optimizer.}
    \label{tab:hparams_maskgit}
\end{table}

\section{Details of Experiments}

\subsection{Video Quantization}
\label{app:sec:vid_quant}
\subsubsection{Network architecture}
All encoder-decoder baselines have approximately $50\text{M}$ parameters.
The Convolutional baseline encoder architecture consists of $5$ convolutional blocks with channel multipliers of $[1, 1, 2, 2, 4]$, $2$ residual layers and $128$ hidden units per block, and embedding dimension of 256.
The ViT baseline encoder architecture consists of an image patchification step over non-overlapping $8\times8$ spatial patches which are linearly transformed into image tokens.
Next, we follow with $8$ transformer layers with $512$ hidden units, $8$ attention heads, $2048$ mlp units, and embedding dimension of $32$.
C-\vivit\xspace encoder architecture patches the first frame to non-overlapping $8\times8$ patches, and then the rest of the frames to non-overlapping $2\times8\times8$ spatio-temporal patches which are linearly transformed into video embeddings.
Next, C-\vivit\xspace encoder architecture consists of $4$ spatial and $4$ temporal transformer layers with $512$ hidden units, $8$ attention heads, $2048$ mlp hidden units, and embedding dimension of $32$.
The decoder architecture for all models is the same as the encoder but in reverse to put the latent embeddings back to image space.
The VQ objective is trained with commitment loss of $\beta=0.25$ and codebook size of $8192$.
The discriminator architecture is the StyleGAN \cite{stylegan} discriminator with blur resample, and channel multiplier of $1$.


\subsubsection{Training}
We train all encoder-decoder baselines and with StyleGAN~\citep{stylegan} discriminators with a batch size of $128$ using Adam optimizer~\cite{adam} with $\beta_1=0.9$ and $\beta_2=0.99$. We use a linear learning rate warmup to a peak value of $1 \times 10^{-4}$ over $100,000$ steps and then decaying over the remaining $900,000$ steps with a cosine schedule, and use a decoupled weight decay \cite{adamw} of $1 \times 10^{-4}$ for the encoder-decoder and discriminator. To capture longer time horizons during training and better evaluate temporal coherence, we downsample the MiT dataset from 25 FPS to 6 FPS and evaluate on videos of 10 frames at spatial resolution of $128\times128$.

\subsection{Image conditional video generation}
\label{app:sec:vid_bair}


\subsubsection{BAIR Robot Push C-\vivit\xspace architecture} \label{app:sec:bair}
We use a similar setup as in Section \ref{app:sec:vid_quant}, but the video tokenization step is done over $4\times4$ spatial patches on the first image and $2\times4\times4$ spatio-temporal patches in the rest of the video. The spatial encoder consists of 8 layers and the temporal encoder consists of 6 layers.

\subsubsection{Kinetics-600 C-\vivit\xspace architecture}
We use a similar setup as in Section \ref{app:sec:bair}, but both the spatial encoder and temporal encoder consist of 8 layers.

\subsubsection{MaskGIT architecture}
To perform video prediction in latent space in the BAIR Robot Push and Kinetics-600 datasets, we use an unconditional transformer architecture consisting of $24$ layers, $768$ hidden
units, $16$ attention heads, dropout and attention dropout rate of $0.1$, $3072$ mlp hidden units.

\subsubsection{Training and Inference}
As described in Table~\ref{tab:hparams_maskgit}, we train C-\vivit\xspace with the same optimizer setup as in Sec~\ref{app:sec:vid_quant}, but we do not downsample the FPS of any of the datasets in this section for fair comparison with the video prediction baselines. We train MaskGIT on the video tokens extracted using C-\vivit\xspace in an unconditional setting, that is, we do not assume frames or text inputs to be given. During training, we use the Adam \cite{adam} optimizer with $\beta_1=0.9$ and $\beta_2=0.99$. We use a linear learning rate warmup up to a peak value of $1\times10^{-4}$ over $10,000$ steps, and constant learning rate schedule for ${~\sim}2M$ steps. At inference time, we initialize MaskGIT given a number of input frames, and predict the rest of the frames depending on the dataset on which we evaluate.

\subsection{Text conditional video generation}
\label{app:sec:text2vid}

\subsubsection{Architecture}
In our text conditional video generation, we use the same C-\vivit\xspace architecture and training described in Section \ref{app:sec:vid_quant}.
To train MaskGIT, we include a text conditioning in the form of T5X embeddings \cite{roberts2022scaling} which are used as input through the use of cross attention with the video tokens. 
We reduce the number of parameters of our base model for fairness in the quantitative comparisons against NUWA. The MaskGIT architecture used against NUWA consists of $20$ transformer layers with $1536$ hidden units, $24$ attention heads, and $6144$ MLP hidden units, resulting in $0.9\text{B}$ parameters similar to NUWA. For the main experiments in this paper, we use a larger architecture that consists of consists of $24$ transformer layers with $2048$ hidden units, $32$ attention heads, and $8192$ mlp hidden units, resulting in $1.8\text{B}$ parameters.

\subsubsection{Training and inference}
For all our text-conditional video generation, we use the training parameters Table~\ref{tab:hparams_maskgit}.

\subsubsection{Inference parameters against NUWA}
We use $\lambda=0.1$, $12$ MaskGIT iterations, and temperature of $4.0$. 

\subsubsection{Inference parameters for ablation of image and video data for training.}
We use $\lambda=6$, $24$ MaskGIT iterations, and temperature of $4.0$. 

\subsubsection{Inference parameters for all videos in the paper.}
We use $\lambda=12$, $48$ MaskGIT iterations, and temperature of $8.0$. 

\input{figures/story2}

\section{Experiments}
\label{sec:exp}
To evaluate Phenaki\xspace, we test it on the following tasks: 1)~text conditional video generation, 2)~text-image conditional video generation, 3)~time variable text conditional video generation (i.e.) story\xspace mode, 4)~video quantization and 5)~image conditional video generation a.k.a. video prediction. To the best of our knowledge, 3)~time variable text conditional video generation has not been explored in prior work. Given the dynamic nature of videos, we highly encourage readers to visit \href{https://phenaki.github.io/}{phenaki.github.io}\ to check the generated videos. The website also includes qualitative comparisons to a subset of the prompts from the CogVideo paper~\cite{cogvideo}. While the focus is on the text to video generation tasks, it is remarkable that \model is still competitive on the more traditional video tasks despite not being developed explicitly for these tasks. We implemented \model in JAX~\cite{jax2018github} using FLAX~\cite{flax2020github} library.
\subsection{Text conditional video generation} 
\label{sec:t2v}
Currently there is no established benchmark for evaluating text to video methods.
This makes comparing \model to recent methods such as NUWA~\cite{nuwa}, CogVideo~\cite{cogvideo}, NUWA-Infinity~\cite{wu2022nuwa} and video diffusion models \cite{ho2022video} difficult. 

Unless specified otherwise, we train a 1.8B parameter \model model on a corpus of ${\sim}15$M text-video pairs at 8 FPS mixed with ${\sim}50$M text-images plus ${\sim}400$M pairs of LAION-400M~\cite{schuhmann2021laion} (more details in Appendix~\ref{app:sec:text2vid}).  The model used in the visualisations in this paper was trained for 1 million steps at a batch size of 512, which took less than 5 days. In this setup 80\% of the training data came from the video dataset and each image dataset contributed 10\%.  

\paragraph{Qualitative evaluation:} Samples from this model can be seen in Figure~\ref{fig:compos} and additional samples are provided at \href{https://phenaki.github.io/}{phenaki.github.io}. We observe that there is a high degree of control over both the actors and the background dynamics in the videos. The appearance of the actors and the video style can be adjusted by the text prompt as well (e.g. a regular video, a cartoon or a pencil drawing). 

On \href{https://phenaki.github.io/}{phenaki.github.io}\ we provide examples from prompts that were provided in the CogVideo~\cite{cogvideo} demo. Since there are substantial differences between these methods it is hard to compare them on an equal footing. As an example, there are massive differences in scale: 9B parameters for CogVideo and 1.8B for our model. Additionally, the training data is different. Finally, we do not know how representative the prompts in the CogVideo demo are for the general performance of the CogVideo. 

\paragraph{Quantative comparison:} 
The NUWA~\cite{nuwa} paper provided a qualitative evaluation on Kinetics-400. Since the NUWA model is only $0.9$B parameters we also use a model of the same size. Our model was trained on 50\% video and 50\% image data in this experiment. The NUWA model fine-tuned on Kinetics but the \model model is not: it is evaluated in a \emph{zero shot setting}.  The results in Table~\ref{table:t2v_comp} show that \model achieves comparable generation quality, in a zero-shot setting, compared to previous text to video methods that were actually trained or finetuned on this dataset. 

\paragraph{On the importance of joint text-to-image and text-to-video training}
\input{tables/t2v_joint}
While there are some text-video datasets, text-image datasets dominate the internet in terms of quality and quantity \cite{videocc}. Consequently, there is simply not enough video data available to cover all the concepts present in text-image datasets. For example using only our video data, concepts such as pencil drawings or different painting styles cannot be learned. To be able to learn a model that can combine video dynamics with these additional concepts we have to combine training on image and video data. In Table~\ref{tab:video_image}, we evaluate the performance of using different ratios of video and images. We start with data splits of only video, and vary the ratio of image and video datasets up to using $50\%$ image and $50\%$ video datasets. In our results, we find that there is a trade-off in performance between models trained with only video video (i.e., significantly better FVD), and models trained with more image data (i.e., better text-video and text-image alignment, and significantly better FID in image datasets). On \href{https://phenaki.github.io/}{phenaki.github.io}\ we show samples from different models side by side where this trade-off between control over the content and the quality of the dynamics can be seen. We believe that the trade-off between concepts and dynamics will be improved as the quality and size of text-video datasets increases in the future.


\subsection{Text-Image conditional video generation}
Given that \model can be conditioned on both still images and text, an interesting setup is to \textit{animate} existing images given a text prompt. For this experiment, we use the same model from Section~\ref{sec:t2v} but conditioned on unseen pictures (captured with our phones from local subjects) and a related prompt. As it can be seen in Figure~\ref{fig:cat} the model can generate coherent videos starting from the given images, while following the given prompts. 
\input{figures/composition}%

\input{figures/cat}%

\subsection{Visual story telling by dynamic text inputs}
A notable and useful feature of \model is that it is auto-regressive in time. This allows for generating long videos, while the prompt changes over time. Time variable prompts can be thought of as a \textit{story\xspace}; a narration of the entire video where each prompt corresponds to a scene from the video. This allows for creating dynamically changing scenes. To the best our knowledge, this paper is the first work to generate such videos. 
An example of this can be seen in Fig.~\ref{fig:story} and on \href{https://phenaki.github.io/}{phenaki.github.io}. The way it works is that we generate a video with the first prompt and then extend it in time by conditioning a possibly new prompt and on the last $N$, typically 5, previously generated frames.

\subsection{Video Encoding}
\label{sec:vid_quant}
To evaluate the video encoding and reconstruction performance of C-\vivit\xspace, we use the Moments-in-Time (MiT)~\citep{monfortmoments} dataset. MiT contains ${\sim}802$K training, ${\sim}33$K validation and ${\sim}67$K test videos at 25 FPS. The MiT dataset, in contrast to other publicly available video datasets, is a high quality balanced dataset with high coverage and density of verbs depicting moments of a few seconds~\cite{monfortmoments}. We compare C-\vivit\xspace against per-frame image based encoder-decoders that have been used as video quantizers for conditional video generation~\citep{harp,nuwa,cogvideo,nuwa,cogvideo,godiva}: a ViT \cite{vit_vqgan} and a convolutional VQ-GAN\cite{vqgan}. The experimental details can be found in the Appendix~\ref{app:sec:vid_quant}. 

\input{tables/quant_results}

As demonstrated in Table~\ref{tab:quant}, we evaluate the video reconstruction quality using FID~\cite{heusel2017gans} and FVD~\cite{unterthiner2018towards}. Both FID and FVD compare the distribution of generated videos (or images) to the ground truth distribution. The FID ignores temporal coherency, while the FVD measures how well the spatio-temporal dynamics of the videos are reconstructed. Results in Table~\ref{tab:quant} show that per-frame image based methods slightly outperform our video method (indicated by marginally higher FID of C-\vivit\xspace), however, they do poorly at modeling the spatio-temporal dynamics in video (significantly lower FVD of C-\vivit\xspace). This is expected as C-\vivit\xspace has spatio-temporal connections between patches in each frame, allowing space and time to be modeled together. In addition, C-\vivit\xspace compresses the video into fewer tokens per video compared to the image based baselines. This is crucial as the number of tokens drastically impacts the computational cost of the transformer in downstream tasks. Furthermore, C-\vivit\xspace tokens are auto-regressive in time which enables variable length videos to be modeled with the same encoder which is important for video extrapolation conditioned on previously generated frames.

\subsection{Image conditional video generation a.k.a Video prediction}
\label{sec:vid_bair}

\input{tables/vidpred_results}
To evaluate the learnt video representation of C-\vivit\xspace beyond reconstruction, we test it on the task of frame-conditioned video generation, also commonly known as video prediction~\cite{fitvid}. In this experiment, we test \model on BAIR Robot Pushing benchmark~\cite{BAIRRobotPush} where the task is to generate 15 frames conditioned on a given single frame. For open domain videos, we test \model on Kinetics-600~\cite{Kinetics600} where the task is to predict 11 frames given 5 frames. More details about these experiments can be found in Appendix~\ref{app:sec:vid_bair}. \cref{tab:bair,tab:kinetics} show the results of these experiments. Note that \model is not specifically designed for video prediction, therefore, it lacks components such as skip connections in U-Nets which are known to improve the performance for video prediction methods~\cite{svg,villegas2019high,fitvid}. Nevertheless, our method is competitive on these benchmarks with SOTA video prediction methods. Overall, these experiments show that \model is strong at modeling dynamics of the videos which is required for generating coherent videos from text.







\section{Introduction}
\afterpage{%
\input{figures/story}%
}

It is now possible to generate realistic high resolution images given a description \cite{DALLE,ramesh2022hierarchical,nichol2021glide,saharia2022photorealistic,PARTI}, but generating high quality videos from text remains challenging. In essence, videos are just a sequence of images, but this does not mean that generating a long coherent video is easy.  In practice, it is a significantly harder task because there is much less high quality data available and the computational requirements are much more severe~\cite{clark2019adversarial}. For image generation, there are datasets with billions of image-text pairs (such as LAION-5B~\cite{schuhmann2021laion} and JFT4B~\cite{zhai2022scaling}) while the text-video datasets are substantially smaller~e.g. WebVid~\cite{bain2021frozen} with ${\sim}10$M videos, which is not enough given the higher complexity of open domain videos. As for computation, training current state-of-the-art image generation models is already pushing the state-of-the-art computational capabilities~\cite{PARTI}, leaving little to no room for generating videos, particularly videos of variable length.

To make the matters worse, one can argue that a single short text prompt is not sufficient to provide a complete description of a video (except for short clips), and instead, a generated video must be conditioned on a sequence of prompts, or \textit{a story\xspace}, which narrates what happens over time. Ideally, a video generation model must be able to generate videos of arbitrary length, all the while having the capability of conditioning the generated frames at time $t$ on prompts at time $t$ that can vary over time. Such capability can clearly distinguish the \textit{video} from a ``moving image" and open up the way to real-world creative applications in art, design and content creation. To the best our knowledge, story\xspace based conditional video generation has never been explored before and this is the first paper to take early steps towards that goal. A traditional deep learning approach of simply learning this task from data is not possible, since there is no story-based dataset to learn from. Instead, to achieve this we rely on a model that is designed specifically with this capability in mind.

In this paper, we introduce \model, a text to video model trained on both text to video and text to image data that can:
\begin{itemize}[leftmargin=0.2in]
    \item[--] Generate temporally coherent and diverse videos conditioned on open domain prompts even when the prompt is a new composition of concepts (Fig.~\ref{fig:compos}). The videos can be long (minutes) even though the model is trained on 1.4 seconds videos (at 8 fps).
    \item[--] Generate videos conditioned on a story\xspace (i.e. a sequence of prompts), e.g.~Fig.~\ref{fig:story} and Fig.~\ref{fig:story2}.
   
\end{itemize}

To enable these capabilities, we could not rely on current video encoders, because they either can only decode fixed size videos or they encode frames independently. Hence, we introduce C-\vivit\xspace, a novel encoder-decoder architecture that:
\begin{itemize}[leftmargin=0.2in]
\item[--] Exploits temporal redundancy in videos to improve reconstruction quality over a per frame model while compressing the number of video tokens by 40\% or more.
\item[--] Allows encoding and decoding of variable length videos given its causal structure.
\end{itemize}

\input{figures/models_diagram}


\section{The \model model}
\label{sec:method}
Inspired by the previous work in auto-regressive text to image~\cite{DALLE,PARTI,saharia2022photorealistic} and text to video~\cite{nuwa,wu2022nuwa,cogvideo}, Phenaki\xspace is designed with two main components (see Figure~\ref{fig:models}): an encoder-decoder model which compresses videos to discrete embeddings (i.e. tokens) and a transformer model to \textit{translate} text embeddings to video tokens. To get the text embeddings, Phenaki\xspace uses a pre-trained language model, T5X~\cite{roberts2022scaling}. We will discuss each one of these components in the following subsections.
 

\subsection{Encoder-decoder video model: C-\vivit\xspace}
One of the primary challenges for generating video from text, is to get a compressed representation of videos. Previous work on text to video either use per-frame image encoders~\cite{cogvideo,nuwa,harp} such as VQ-GAN~\cite{vqgan} or fixed length video encoders~\cite{godiva} such as VideoVQVAE~\cite{videovqvae}. The former allows for generating videos of arbitrary length, however in practice, the videos have to be short because the encoder does not compress the videos in time and the tokens are highly redundant in consecutive frames. The latter is more efficient in the number of tokens but it does not allow to generate variable length videos. In Phenaki\xspace, our goal is to generate videos of variable length while keeping the number of video tokens to a minimum so they can be modeled with a transformer within current computational limitations. To do so, we introduce C-\vivit\xspace, a causal variation of ViViT\xspace~\cite{vivit} with additional architectural changes for video generation, which can compress the videos in temporal and spatial dimensions, while staying auto-regressive in time, This capability allows for generating videos of arbitrary length auto-regressively.

\paragraph{Encoder architecture:}
As illustrated in Figure \ref{fig:models}, we start with a video sequence of $t_x + 1$ frames with a resolution of $w_x\times h_x$ and $c_x$ channels: $\mathbf{x} \in \mathds{R}^{(t_x + 1) \times h_x \times w_x \times c_x}$. This sequence will be compressed into a token representation of size $(t_z + 1)\times w_z \times h_z$ where the first $w_z\times h_z$ tokens represent the first frame independently from the rest of the video, and the remaining tokens represent spatio-temporal video tokens that auto-regressively depend on previous frames. 
To do so, we extract non-overlapping image patches of size $w_p \times h_p \times c_p$ from the first frame and video patches of size $t_p\times w_p \times h_p \times c_p$ from the rest of the video.
We typically use all channels at once such that the number of patches equals the number of video tokens $t_z=\frac{t_x}{t_p}$, $w_z=\frac{w_x}{w_p}$ and  $h_z=\frac{h_x}{h_p}$.
Each of these patches is flattened and linearly projected into a $d_z$ dimensional space.
We combine the spatial dimensions to have a tensor of shape $(t_z + 1)\times w_z*h_z \times d_z$ where the spatial and temporal dimensions are separated.
Then multiple transformer layers are applied along the spatial dimensions with all-to-all attention.
This is followed by multiple transformer layers over the temporal dimension with causal attention such that each spatial token only observes spatial tokens from previous frames in an auto-regressive manner.
The effect of this is that the first frame can be completely independently encoded.
This opens up the possibility of text to image training to be embedded naturally into our video model.
The second advantage is that we can condition the video generation process on a number of starting frames. 
The resulting patch embeddings $\mathbf{z}$ of shape $t_z\times w_z \times h_z \times d_z$ are then tokenized into learned codewords $c_z$ by vector quantization.
The codebook learning will be discussed later together with the losses.

\paragraph{Decoder architecture:}
The C-\vivit\xspace decoder is simply an upside down version of the encoder. First tokens are transformed into embeddings. This is followed by the temporal transformer, then the spatial transformer.
After the output of the spatial transformer, we apply a single linear projection without activation to map the tokens back to pixel space.

\paragraph{Quantization and Losses:}
To learn a discrete latent space, we quantize our encoder outputs into the entries of a learned codebook via the vector quantization (VQ) objective in VQVAEs \citep{vqvae},
\begin{equation}
    L_{\textit{VQ}} = \| \text{sg}(\mathbf{z}) - \mathbf{e} \|^2_2 + \beta \| \mathbf{z} - sg(\mathbf{e}) \|^2_2,
\end{equation}
where $\text{sg}(x) \equiv x$, and $\frac{\text{d}}{\text{d}x} \text{sg}(x) \equiv 0$ is the stop-gradient operator, $\beta$ is the commitment loss weight, and $\mathbf{e}$ is a codebook vector from codebook $\mathbf{E}$. The index to the codebook vector closest to $\mathbf{z}$ is found by $i = \text{argmin}_j \| \mathbf{z} - \mathbf{E}_j \|^2_2$. 
In addition to the VQ objective, we adopt the factorized and $\ell_2$-normalized codes from ViT-VQGAN~\citep{vit_vqgan} to improve codebook usage and reconstruction quality.

To train our model, we use a combination of $L_2$ loss, image perceptual loss $L_{\textit{IP}}$~\citep{JohnsonPerceptual2016,ZhangPerceptual2018}, video perceptual loss $L_{\textit{VP}}$ by using the I3D network~\citep{i3d} as feature extractor, and adversarial loss $L_{\textit{Adv}}$ with StyleGAN architecture~\citep{stylegan}.
As training objective, we use the following
\begin{equation}
    L = L_{\textit{VQ}} + 0.1\times L_{\textit{Adv}} + 0.1\times L_{\textit{IP}} + 1.0\times L_{\textit{VP}} + 1.0\times L_2.
\end{equation}

\paragraph{Novelty over the ViViT\xspace architecture:}
While our proposed C-\vivit\xspace architecture is inspired by the factorized encoder in ViViT\xspace~\cite{vivit}, we modify their architecture to enable self-supervised learning from unlabeled videos. We first remove the \texttt{[CLS]} tokens in the spatial and the temporal transformers. Next, we apply temporal transformer for all spatial tokens computed by the spatial encoder, in contrast to  single run of the temporal transformer over the \texttt{[CLS]} tokens in ViViT\xspace.
Most importantly, the ViViT\xspace encoder requires a fixed length video input due to the all-to-all attention in time. Therefore, we apply causal attention instead such that our C-\vivit\xspace encoder becomes auto-regressive and allows for a variable number of input frames which are necessary to learn from image datasets, and auto-regressively extrapolate video or single frames into the future. 












\subsection{Text-to-video generation with bidirectional transformers}
In this stage, the text-to-video task can be formulated as a sequence-to-sequence problem to predict video tokens given the paired text embeddings. Most of recent methods \citep{DALLE, PARTI, nuwa, cogvideo} adopt a  transformer model for these sequence-to-sequence tasks. In their models, they use an auto-regressive transformer which predicts the image or video tokens sequentially given the encoded text features.
As a result, the sampling time scales linearly with the sequence length, even when caching is used. This becomes impractical for long video sequence generation. 

\paragraph{Masked bidirectional transformer:}

In this work, we aim to reduce the sampling time by having a small and fixed sampling step disregarding different video sequence lengths. Inspired by previous work for image generation~\citep{MASKGIT}, we use a bidirectional transformer since it can predict different video tokens simultaneously. For training step $i$, we first sample a mask ratio $\gamma_{i}$ from 0 to 1 and randomly replace $\lceil \gamma_{i} \cdot N \rceil$ tokens with the special token \texttt{[MASK]}, where $N$ is the video sequence length. Then
we learn the model parameters by minimizing the cross entropy loss on those masked tokens given the encoded text embeddings and unmasked video tokens. During inference, we first label all of the video tokens as the special token \texttt{[MASK]}. Then, at each inference step, we predict all the masked (unknown) video tokens in parallel conditioned on the text embeddings and unmasked (predicted) video tokens. We keep a ratio $\beta_{i}$ of the predicted tokens at sampling step $i$ and the remaining tokens are re-masked and re-predicted in the next step. 

As discussed in MaskGIT~\citep{MASKGIT}, the masking schedule $\gamma_{i}$ and sampling schedule $\beta_{i}$ have a significant effect on the samples quality therefore we follow the same strategies. Compared to an auto-regressive transformer, the number of sampling steps is an order-of-magnitude smaller (typically we use values in the range of 12 to 48). Generally speaking, more sampling steps improves the quality.

\paragraph{Losses and training strategies:}
Given a pre-trained C-\vivit\xspace, videos are encoded into codebook ids $\textbf{a}$ of shape $(t_z + 1)\times w_z \times h_z$ which are flattened into a long vector using the raster ordering from \cite{vit_vqgan}. We then model the text-conditional video token distribution using \textit{Masked Visual Token Modeling} (MVTM)~\cite{MASKGIT}:
\begin{equation}
    L_{\text{mask}}= - \sum\nolimits_{\forall i\in[1,N],m_i=1} \log p(a_i | \textbf{a}_{\bar{M}}, \mathbf{p}),
\end{equation}
where $\textbf{a}_{\bar{M}}$ represents the masked version of $\textbf{a}$, $m_i$ is a binary variable indicating whether $a_i$ is masked or not, $N$ is the number of video tokens, and $\mathbf{p}$ is the text condition embedding.
In addition to the MVTM objective, we train using classifier-free guidance by dropping the text condition $10\%$ of the time during training~\cite{cfg,PARTI} .
Finally, we dynamically adjust the MVTM objective during training to allow the use of image and video datasets as a single large dataset.
We achieve this by only applying the masking ratio and objective on the first $w_z \times h_z$ tokens if only a single frame is given or over all video tokens if a full video is given. 
This mixed image and video dataset training strategy allows our models to learn concepts only present in image datasets, and transfer them to concepts present video datasets (e.g., the pencil drawing styled video of the panda in Figure.\ref{fig:compos}).

\paragraph{Inference and auto-regressive generation of long videos:}
At inference time, we sample videos tokens by the same iterative process used in \cite{MASKGIT} with classifier-free guidance scale $\lambda$ to control alignment between the generation and the text condition. Once the first video is generated, we can extrapolate additional frames auto-regressively by encoding the last $K$ generated frames in the last video using C-\vivit\xspace, initializing MaskGIT with the tokens computed by our C-\vivit\xspace encoder, and proceed to generate the remaining video tokens conditioned on a text input. During video extrapolation, the text condition can be the same or a different one which enables our model to dynamically create visual transitions between the previous and current text condition visual content, effective generating a visual story an described by the input text.


\section{Conclusion}

We introduced \model, a model which is capable of generating variable length videos conditioned on a sequence of open domain text prompts. \model uses C-\vivit\xspace as video encoder. C-\vivit\xspace is a new model which provides temporal-spatial compression while being auto-regressive in time.
The C-\vivit\xspace model is a crucial part of \model that allows it to generate variable length videos. We demonstrate how joint training on images and videos can improve the generation quality, and diversity, given the existence of much larger image-text dataset with order of magnitude more samples. The \model model achieves good performance on video prediction, it can be used as to generate long videos conditioned on a text prompt. Additionally it is able to condition on both text and a starting frame. Finally, \model is not limited to generating a video depicting a single concept or caption. It is actually able to generate longer coherent video stories based on a sequence of text prompts. The more complex narratives it can visualize demonstrate how this can become a great creative tool for story telling.

\section*{Ethics Statement}
While we have not explored potential downstream applications of the generative models described in this work, we believe Phenaki can have a positive impact in a variety of creative settings. In general, many of the samples from the model will not perfectly correspond to the input caption or the user’s intent; however, the end-user is likely to gain considerable time savings even if only one of the generated samples aligns with their intent. We thus foresee Phenaki being useful in eventually empowering users to accelerate their creativity, especially since the model can so quickly generate videos. Phenaki and similar models will be part of an ever-broad toolset for artists and non-artists alike, providing new and exciting ways to express creativity.

The flip-side of this acceleration and ease-of-use is the potential for harmful impact, as with many of the prior or concurrent work in generative modeling. An easy-to-use system like Phenaki can be repurposed for generating maliciously fake content and enable spreading of such content much easier. While the quality of the videos generated by Phenaki is not yet indistinguishable from real videos, getting to that bar for a specific set of samples is within the realm of possibility, even today. This can be particularly harmful if Phenaki is to be used to generate videos of someone without their consent and knowledge.

Like DALLE-2~\cite{ramesh2022hierarchical}, Imagen~\cite{saharia2022photorealistic}, Parti~\cite{PARTI} and others, Phenaki is trained on a collection of datasets that is known to encode a number of undesirable biases. LAION-400M~\cite{schuhmann2021laion} specifically has a variety of issues regarding violence, pornography, gore. While our primary image and video datasets have minimal traits like this, we did incorporate LAION-400M into our training and observed better results. In a currently training version of Phenaki, we use a set of datasets that minimizes such problems.

Taken together, these issues contribute to our decision not to release the underlying models, code, data or interactive demo at this time. Before we can do that, we want to focus our efforts on better understanding of data, prompt and output filtering. We would also like to more explicitly measure the biases encoded in the outputs of Phenaki, so that we can further mitigate them actively, either in the data, models or pre/post-processing steps.






