\section{Acknowledgements}
This work was in part supported by Panasonic through BAIR Commons, Darpa RACER, the Hong Kong Centre for Logistics Robotics, and BMW. In addition, we thank the TRC program (\href{https://sites.research.google/trc/about/}{https://sites.research.google/trc/about/}) and Cirrascale Cloud Services (\href{https://cirrascale.com/}{https://cirrascale.com/}) for providing compute resources.
\section{Samples}

\subsection{DMLab}

\begin{figure}[H]
    \centering
    \includegraphics[width=\linewidth]{figures/samples/dmlab_cond.pdf}
    \caption{156 frames generated conditioned on 144 (action-conditioned)}
\end{figure}

\begin{figure}[H]
    \centering
    \includegraphics[width=\linewidth]{figures/samples/dmlab_uncond.pdf}
    \caption{264 frames generated conditioned on 36 (\textbf{no} action-conditioning)}
\end{figure}

\clearpage

\begin{figure}[H]
    \centering
    \includegraphics[width=\linewidth]{figures/samples/dmlab_3d_more.pdf}
    \caption{3D visualizations of the resulting generated DMLab mazes}
    \label{fig:dmlab_3d_more}
\end{figure}

\clearpage


\clearpage

\subsection{Minecraft}
\begin{figure}[H]
    \centering
     \includegraphics[width=\linewidth]{figures/samples/minecraft_cond.pdf}
    \caption{156 frames generated conditioned on 144 (action-conditioned)}
\end{figure}

\begin{figure}[H]
    \centering
     \includegraphics[width=\linewidth]{figures/samples/minecraft_uncond.pdf}
    \caption{264 frames generated conditioned on 36 (action-conditioned)}
\end{figure}

\clearpage

\subsection{Habitat}

\begin{figure}[H]
    \centering
    \includegraphics[width=\linewidth]{figures/samples/habitat_cond.pdf}
    \caption{156 frames generated conditioned on 144 (action-conditioned)}
\end{figure}

\begin{figure}[H]
    \centering
    \includegraphics[width=\linewidth]{figures/samples/habitat_uncond.pdf}
    \caption{264 frames generated conditioned on 36 (\textbf{no} action-conditioning)}
\end{figure}


\clearpage

\subsection{Kinetics-600}

\begin{figure}[H]
    \centering
     \includegraphics[width=\linewidth]{figures/samples/kinetics_no_top_k.pdf}
    \caption{80 frames generated conditioned on 20 (no top-k sampling)}
\end{figure}

\begin{figure}[H]
    \centering
    \includegraphics[width=\linewidth]{figures/samples/kinetics_top_k.pdf}
    \caption{80 frames generated conditioned on 20 (with top-k sampling)}
\end{figure}

\clearpage

\section{Performance versus Horizon}

\begin{figure}[H]
\begin{subfigure}[b]{\linewidth}%
\rlap{\includegraphics[width=\linewidth]{figures/over_time/dl_maze_over_time.pdf}}
\caption{DMLab}
\end{subfigure}\hfill%
\begin{subfigure}[b]{\linewidth}%
\rlap{\includegraphics[width=\linewidth]{figures/over_time/minerl_over_time.pdf}}
\caption{Minecraft}
\end{subfigure}\hfill%
\begin{subfigure}[b]{\linewidth}%
\rlap{\includegraphics[width=\linewidth]{figures/over_time/habitat_over_time.pdf}}
\caption{Habitat}
\end{subfigure}\hfill%

\caption{All plots shows PSNR, SSIM, and LPIPS on 150 predicted frames conditioned on 144 frames. The 144 conditioned frames are not shown on the graphs and timestep 0 corresponds to the first predicted frame}
\label{fig:horizon}
\end{figure}
\Cref{fig:horizon} shows PSNR, SSIM, and LPIPS as a function of prediction horizon for each dataset. Generally, each plot reflected the corresponding aggregated metrics in \Cref{table:main}. For DMLab, TECO shows much better temporal consistency for the full trajectory, with Latent FDM coming in second. CW-VAE is able retain some consistency but drops fairly quickly. Lastly, FitVid and Perceiver AR lose consistency very quickly. We see a similar trend in Minecraft, with Latent FDM coming closer in matching TECO. For Habitat, all methods generally have trouble producing consistent predictions, primarily due to the difficulty of the environment.

\clearpage

\section{Performance versus Training Sequence Length}
\begin{figure}[H]
\begin{subfigure}[b]{0.45\linewidth}
\rlap{\includegraphics[width=\linewidth]{figures/train_seq_len/dl_maze_fvd.pdf}}
\end{subfigure}
\hfill
\begin{subfigure}[b]{0.45\linewidth}
\includegraphics[width=\linewidth]{figures/train_seq_len/dl_maze_lpips.pdf}
\end{subfigure}

\begin{subfigure}[b]{0.45\linewidth}
\rlap{\includegraphics[width=\linewidth]{figures/train_seq_len/dl_maze_psnr.pdf}}
\end{subfigure}
\hfill
\begin{subfigure}[b]{0.45\linewidth}
\includegraphics[width=\linewidth]{figures/train_seq_len/dl_maze_ssim.pdf}
\end{subfigure}
\caption{DMLab}
\label{fig:scaling_dmlab}
\end{figure}

\begin{figure}[H]
\begin{subfigure}[b]{0.45\linewidth}
\rlap{\includegraphics[width=\linewidth]{figures/train_seq_len/minerl_fvd.pdf}}
\end{subfigure}
\hfill
\begin{subfigure}[b]{0.45\linewidth}
\includegraphics[width=\linewidth]{figures/train_seq_len/minerl_lpips.pdf}
\end{subfigure}

\begin{subfigure}[b]{0.45\linewidth}
\rlap{\includegraphics[width=\linewidth]{figures/train_seq_len/minerl_psnr.pdf}}
\end{subfigure}
\hfill
\begin{subfigure}[b]{0.45\linewidth}
\includegraphics[width=\linewidth]{figures/train_seq_len/minerl_ssim.pdf}
\end{subfigure}
\caption{Minecraft}
\label{fig:scaling_minecraft}
\end{figure}

\Cref{fig:scaling_dmlab} and \Cref{fig:scaling_minecraft} show plots comparing performance with training models on different sequence lengths. Under a fixed compute budget and batch size, training on shorter videos enables us to scale to larger models. This can also be interpreted as model capacity or FLOPs allocated per image. In general, training on shorter videos enables higher quality frames (per-image) but at a cost of worse temporal consistency due to reduced context length. We can see a very clear trend in DMLab, in that TECO is able to better scale on longer sequences, and correspondingly benefits from it. Latent FDM has trouble when training on full sequences. We hypothesize that this may be due to diffusion models being less amenable towards downsamples, it it needs to model and predict noise. In Minecraft, we see the best performance at around 50-100 training frames, where a model has higher fidelity image predictions, and also has sufficient context.

\section{Sampling}
\begin{figure}[H]
\begin{subfigure}[b]{0.5\linewidth}
\rlap{\includegraphics[width=\linewidth]{figures/sampling_fvd.pdf}}
\end{subfigure}
\hfill
\begin{subfigure}[b]{0.5\linewidth}
\includegraphics[width=\linewidth]{figures/sampling_lpips.pdf}
\end{subfigure}
\end{figure}


\begin{table}[H]
\centering
\begin{tabular}{@{}lc@{}}
\toprule
             & Sampling Time per Frame (ms) \\ \midrule
TECO (ours)  & $186$                        \\
Latent FDM   & $3606$                       \\
Perceiver-AR & $8443$                       \\
CW-VAE       & $0.062$                      \\
FitVid       & $0.074$                      \\ \bottomrule
\end{tabular}
\end{table}

\section{Ablations}

\begin{table}[H]
    \begin{subtable}[h]{0.6\linewidth}
        \centering
        \begin{tabular}{@{}ccc@{}}
        \toprule
        DropLoss Rate & FVD & Train Step (ms)                    \\ \midrule
        0.8             & $187$ &    $125$                \\
        0.6             & $186$  &  $143$                 \\
        0.4             & $184$  & $155$ \\
        0.2             & $184$   &    $167$              \\
        0.0             & $182$ & $182$ \\ \bottomrule
        \end{tabular}
        \caption{DropLoss Rates}
    \end{subtable}
    \hfill
    \begin{subtable}[h]{0.4\linewidth}
        \centering
        \begin{tabular}{@{}lr@{}}
        \toprule
        Posteriors                                   & FVD                       \\
        \midrule
        VQ (+ MaskGit prior) (ours)                                    & $189$                     \\
        OneHot (+ MaskGit prior) & $199$ \\
        OneHot (+ Block AR prior)                    & $209$ \\
        OneHot (+ Independent prior)                  & $228$                     \\
        Argmax (+ MaskGit prior)                  & $336$                     \\
        \bottomrule
        \end{tabular}
        \caption{Posteriors}
    \end{subtable}
    \begin{subtable}[h]{0.33\linewidth}
        \centering
        \begin{tabular}{@{}lr@{}}
            \toprule
            Dynamics Prior & FVD                       \\ \midrule
            MaskGit (ours) & $189$                     \\
            Independent    & $220$ \\ 
            Autoregressive & $207$                     \\
            \bottomrule
        \end{tabular}
        \caption{Prior Networks}
    \end{subtable}
    \begin{subtable}[h]{0.33\linewidth}
        \centering
        \begin{tabular}{@{}lr@{}}
        \toprule
        Conditional Encoding       & FVD \\ \midrule
        Yes (ours)                    & 189 \\
        No & 208 \\ \bottomrule
        \end{tabular}
        \caption{Conditional Encoding}
    \end{subtable}
    \begin{subtable}[h]{0.33\linewidth}
    \centering
        \begin{tabular}{@{}cc@{}}
            \toprule
            Number of Codes & FVD \\ \midrule
            64              & 191 \\
            256             & 195 \\
            1024            & 186 \\
            4096            & 200 \\ \bottomrule
            \end{tabular}
            \caption{VQ Codebook Size}
    \end{subtable}
    \caption{Ablations comparing alternative prior, posterior, and codebook designs}
    \label{table:abl_arch}
\end{table}

\begin{table}[H]
    \begin{subtable}[h]{0.28\linewidth}
        \centering
        \begin{tabular}{@{}lcl@{}}
        \toprule
                  & \multicolumn{2}{c}{FVD}   \\
                        Size & $2\times 2$ & $4\times 4$ \\ \midrule
        Base            & 204         & 189         \\
        Small Enc & 214         & 191         \\
        Small Dec & 232         & 198         \\ \bottomrule
        \end{tabular}
        \caption{Encoder and Decoder}
    \end{subtable}
    \begin{subtable}[h]{0.36\linewidth}
        \centering
        \begin{tabular}{@{}cccc@{}}
        \toprule
               &       & \multicolumn{2}{c}{FVD}   \\
        Layers & Width & $2\times 2$ & $4\times 4$ \\ \midrule
        8      & 768   & 204         & 189         \\
        8      & 384   & 260         & 196         \\
        2      & 768   & 216         & 202         \\ \bottomrule
        \end{tabular}
        \caption{Temporal Transformer}
    \end{subtable}
    \begin{subtable}[h]{0.36\linewidth}
        \centering
        \begin{tabular}{@{}cccc@{}}
        \toprule
               &       & \multicolumn{2}{c}{FVD}   \\
        Layers & Width & $2\times 2$ & $4\times 4$ \\ \midrule
        8      & 768   & 204         & 189         \\
        8      & 384   & 228         & 193         \\
        2      & 768   & 228         & 201         \\ \bottomrule
        \end{tabular}
        \caption{MaskGit Prior}
    \end{subtable}
    \caption{Ablations on scaling different parts of TECO.}
    \label{table:abl_scaling}
\end{table}

\begin{table}[H]
\centering
\begin{tabular}{@{}lccccc@{}}
\toprule
               & FVD ($\downarrow$) & PSNR ($\uparrow$) & SSIM ($\uparrow$) & LPIPS ($\downarrow$) & Train Step Time (ms) \\ \midrule
TECO (ours)    & $\textbf{27.5}$             & $\mathbf{22.5}$   & $\mathbf{0.709}$  & $\mathbf{0.155}$     & $\mathbf{151}$       \\
MaskGit        & $75.5$             & $21.4$            & $0.681$           & $0.184$              & $170$                \\
Autoregressive & $45.7$    & $20.0$            & $0.635$           & $0.203$              & $267$                \\ \bottomrule
\end{tabular}
\caption{DMLab dataset comparisons against similar model as TECO without latent dynamics, and Maskgit or AR model on VQ-GAN tokens directly.}
\label{table:abl_dynamics}
\end{table}
\Cref{table:abl_dynamics} shows comparisons between TECO and alternative architectures that do not use latent dynamics. Architecturally, MaskGit and Autoregressive are very similar to TECO, with a few small changes: (1) there is no CNN decoder and (2) MaskGit and AR directly predict the VQ-GAN latents (as opposed to the learned VQ latents in TECO). In terms of training time, MaskGit and AR are a little slower since they operate on $16 \times 16$ latents instead of $8\times 8$ latents for TECO. In addition, conditioning for the AR model is done using cross attention, as channel-wise concatenation does not work well due to unidirectioal masking. Both models without latent dynamics have worse temporal consistency, as well as overall sample quality. We hypothesize that TECO has better temporal consistency due to weak bottlenecking of latent representation, as a lot of time can be spent modeling likelihood of imperceptible image / video statistics.
\clearpage

\section{Full Experimental Results}
\label{section:full_results}
\begin{table}[H]
\hspace*{-1.5cm}\begin{tabular}{@{}lcccccc@{}}
\toprule
             & TPU-v3 Days & Params & FVD $\downarrow$         & PSNR $\uparrow$           & SSIM $\uparrow$             & LPIPS $\uparrow$            \\ \midrule
TECO (ours)  & $32$        & 169M   & $\mathbf{27.5 \pm 1.77}$ & $\mathbf{22.4 \pm 0.368}$ & $\mathbf{0.709 \pm 0.0119}$ & $\mathbf{0.155 \pm 0.00958}$ \\
Latent FDM   & $32$        & 31M    & $181 \pm 2.20$           & $17.8 \pm 0.111$          & $0.588 \pm 0.00453$         & $0.222 \pm 0.00493$         \\
Perceiver-AR & $32$        & 30M    & $96.3 \pm 3.64$          & $11.2 \pm 0.00381$        & $0.304 \pm 0.0000456$       & $0.487 \pm 0.00123$         \\
CW-VAE       & $32$        & 111M   & $125 \pm 7.95$           & $12.6 \pm 0.0585$         & $0.372 \pm 0.000330$        & $0.465 \pm 0.00156$         \\
FitVid       & $32$        & 165M   & $176 \pm 4.86$           & $12.0 \pm 0.0126$         & $0.356 \pm 0.00171$         & $0.491 \pm 0.00108$         \\ \bottomrule
\end{tabular}
\caption{DMLab}
\end{table}

\begin{table}[H]
\hspace*{-1.5cm}\begin{tabular}{@{}lcccccc@{}}
\toprule
             & TPU-v3 Days & Params & FVD $\downarrow$         & PSNR $\uparrow$            & SSIM $\uparrow$              & LPIPS $\uparrow$             \\ \midrule
TECO (ours)  & $80$        & 274M   & $116\pm 5.08$            & $\mathbf{15.4 \pm 0.0603}$ & $\mathbf{0.381 \pm 0.00192}$ & $\mathbf{0.340 \pm 0.00264}$ \\
Latent FDM   & $80$        & 33M    & $167 \pm 6.26$           & $13.4 \pm 0.0904$          & $0.349 \pm 0.00327$          & $0.429 \pm 0.00284$          \\
Perceiver-AR & $80$        & 166M   & $\mathbf{76.3 \pm 1.72}$ & $13.2 \pm 0.0711$          & $0.323 \pm 0.00336$          & $0.441 \pm 0.00207$          \\
CW-VAE       & $80$        & 140M   & $397 \pm 15.5$           & $13.4 \pm 0.0610$          & $0.338 \pm 0.00274$          & $0.441 \pm 0.00367$          \\
FitVid       & $80$        & 176M   & $956 \pm 15.8$           & $13.0 \pm 0.00895$         & $0.343 \pm 0.00380$          & $0.519 \pm 0.00367$          \\ \bottomrule
\end{tabular}
\caption{Minecraft}
\end{table}

\begin{table}[H]
\hspace*{-1.5cm}\begin{tabular}{@{}lcccccc@{}}
\toprule
             & TPU-v3 Days & Params & FVD $\downarrow$        & PSNR $\uparrow$           & SSIM $\uparrow$              & LPIPS $\uparrow$              \\ \midrule
TECO (ours)  & $275$       & 386M   & $\mathbf{76.3\pm 1.72}$ & $\mathbf{12.8\pm 0.0139}$ & $0.363\pm 0.00122$           & $0.604\pm 0.00451$            \\
Latent FDM   & $275$       & 87M    & $433 \pm 2.67$          & $12.5 \pm 0.0121$         & $0.311 \pm 0.000829$         & $\mathbf{0.582 \pm 0.000492}$ \\
Perceiver-AR & $275$       & 200M   & $164 \pm 12.6$          & $\mathbf{12.8\pm 0.0423}$ & $\mathbf{0.405 \pm 0.00248}$ & $0.676 \pm 0.00282$           \\ \bottomrule
\end{tabular}
\caption{Habitat}
\end{table}


\begin{table}[H]
\hspace*{-1.5cm}\begin{subtable}[h]{0.5\linewidth}
    \begin{tabular}{@{}cccc@{}}
    \toprule
                 & TPU-v3 Days & Params & FVD $\downarrow$        \\ \midrule
    TECO (ours)  & $640$       & 1.09B  & $649 \pm 16.5$          \\
    Latent FDM   & $640$       & 831M   & $960 \pm 52.7$ \\
    Perceiver-AR & $640$       & 1.06B  &  $\mathbf{607 \pm 6.98}$         \\ \bottomrule
    \end{tabular}
    \caption{Using top-k sampling for Perceiver AR and TECO}
\end{subtable}
\hspace*{1.5cm}\begin{subtable}[h]{0.5\linewidth}
    \begin{tabular}{@{}cccc@{}}
    \toprule
                 & TPU-v3 Days & Params & FVD $\downarrow$        \\ \midrule
    TECO (ours)  & $640$       & 1.09B  & $\mathbf{799 \pm 23.4}$          \\
    Latent FDM   & $640$       & 831M   & $960 \pm 52.7$ \\
    Perceiver-AR & $640$       & 1.06B  & $1022 \pm 32.4$          \\ \bottomrule
    \end{tabular}
    \caption{No top-k sampling}
\end{subtable}
\caption{Kinetics}
\label{table:kinetics_full}
\end{table}

\begin{figure}[H]
    \centering
    \includegraphics[width=0.5\linewidth]{figures/fvd_top_k.pdf}
    \caption{FVD on Kinetics-600 with different top-k values for Perceiver-AR and TECO}
    \label{fig:kinetics_top_k}
\end{figure}
\clearpage


\section{Scaling Results}

\begin{table}[H]
\hspace*{-1.5cm}\begin{tabular}{@{}cccccccc@{}}
\toprule
                             & \begin{tabular}[c]{@{}c@{}}TPU-v3 \\ Days\end{tabular} & \begin{tabular}[c]{@{}c@{}}Train \\ Seq Len\end{tabular} & Params & FVD $\downarrow$         & PSNR $\uparrow$           & SSIM $\uparrow$             & LPIPS $\downarrow$  \\ \midrule
\multirow{4}{*}{TECO (ours)} & \multirow{4}{*}{$32$}                                  & 300                                                      & 169M   & $\mathbf{48.2 \pm 2.02}$ & $\mathbf{21.9 \pm 0.368}$ & $\mathbf{0.703 \pm 0.0114}$ & $\mathbf{0.157 \pm 0.0119}$  \\
                             &                                                        & 200                                                      & 169M   & $59.7 \pm 2.29$          & $19.9 \pm 0.186$          & $0.628 \pm 0.00821$         & $0.187\pm 0.00460$  \\
                             &                                                        & 100                                                      & 86M    & $63.9\pm 7.84$           & $15.4\pm 0.199$           & $0.476 \pm 0.00745$         & $0.322 \pm 0.00792$ \\
                             &                                                        & 50                                                       & 195M   & $52.7 \pm 6.23$          & $13.9 \pm 0.0311$         & $0.418 \pm 0.000659$        & $0.383\pm 0.000302$ \\ \midrule
\multirow{4}{*}{Latent FDM}  & \multirow{4}{*}{$32$}                                  & 300                                                      & 31M    & $181 \pm 2.20$           & $17.8 \pm 0.111$          & $0.588 \pm 0.00453$         & $0.222 \pm 0.00493$ \\
                             &                                                        & 200                                                      & 62M    & $66.4 \pm 3.31$          & $17.7 \pm 0.114$          & $0.561 \pm 0.00623$         & $0.253 \pm 0.00550$ \\
                             &                                                        & 100                                                      & 80M    & $55.6 \pm 1.36$          & $15.5 \pm 0.233$          & $0.468 \pm 0.00776$         & $0.336 \pm 0.00511$ \\
                             &                                                        & 50                                                       & 110M   & $68.3 \pm 3.19$          & $14.0 \pm 0.0445$         & $0.414 \pm 0.424$           & $0.385 \pm 0.00151$ \\ \bottomrule
\end{tabular}
\caption{DM Lab scaling}
\end{table}

\begin{table}[H]
\hspace*{-1.5cm}\begin{tabular}{@{}cccccccc@{}}
\toprule
                             & \begin{tabular}[c]{@{}c@{}}TPU-v3 \\ Days\end{tabular} & \begin{tabular}[c]{@{}c@{}}Train \\ Seq Len\end{tabular} & Params & FVD $\downarrow$ & PSNR $\uparrow$            & SSIM $\uparrow$              & LPIPS $\downarrow$           \\ \midrule
\multirow{4}{*}{TECO (ours)} & \multirow{4}{*}{$80$}                                  & 300                                                      & 274M   & $116 \pm 5.08$   & $15.4 \pm 0.0603$          & $0.381 \pm 0.00192$          & $0.340 \pm 0.00264$          \\
                             &                                                        & 200                                                      & 261M   & $109.5 \pm 1.46$ & $15.4 \pm 0.0906$          & $0.379 \pm 0.00263$          & $0.343 \pm 0.00148$          \\
                             &                                                        & 100                                                      & 257M   & $85.1 \pm 4.09$  & $\mathbf{15.7 \pm 0.0516}$ & $0.385 \pm 0.00244$          & $\mathbf{0.325 \pm 0.00121}$ \\
                             &                                                        & 50                                                       & 140M   & $80.7 \pm 1.42$  & $14.8 \pm 0.0404$          & $0.369 \pm 0.00197$          & $0.360 \pm 0.00133$          \\ \midrule
\multirow{4}{*}{Latent FDM}  & \multirow{4}{*}{$80$}                                  & 300                                                      & 33M    & $167 \pm 6.26$   & $13.4 \pm 0.0904$          & $0.349 \pm 0.00327$          & $0.429 \pm 0.00284$          \\
                             &                                                        & 200                                                      & 80M    & $104.9 \pm 3.21$ & $15.0 \pm 0.0701$          & $0.384 \pm 0.00320$          & $0.366 \pm 0.00311$          \\
                             &                                                        & 100                                                      & 69M    & $92.8 \pm 4.40$  & $15.1 \pm 0.0866$          & $\mathbf{0.390 \pm 0.00281}$ & $0.358 \pm 0.00250$          \\
                             &                                                        & 50                                                       & 186M   & $85.6 \pm 2.25$  & $14.8 \pm 0.0578$          & $0.378 \pm 0.00144$          & $0.372 \pm 0.000966$         \\ \bottomrule
\end{tabular}
\caption{Minecraft scaling}
\end{table}
\clearpage


\input{related_work}
\clearpage


\section{Dataset Details}
\label{section:dataset_details}
\subsection{DMLab}
\label{section:dataset_dmlab_details}
We generate random $7\times 7$ mazes split into four quadrants, with each quadrant containing a random combination of wall and floor textures. We generate 40k trajectories of 300 frames, each $64\times 64$ images. Actions in this environment consist of $20^{\circ}$ left turn, $20^{\circ}$ right turn, and walk forward. In order to maximally traverse the maze, we code an agent that traverses to the furthest unvisited point in the maze, with some added noise for stochasticity. Since the maze is a grid, we can easily hard-code a navigation policy to move to any specified point in the maze.

For 3D visualizations, we also collect depth, camera intrinsics and camera extrinsics (pose) for each timestep. Given this information, we can project RGB points into a 3D coordinate space and reconstruct the maze as a 3D pointcloud. Note that since videos are generated \text{only using RGB} as input, they do not have groundtruth depth and pose. Therefore, we train depth and pose estimators that are used during evaluation. Specifically, we train a depth estimator to map from RGB frame to depth, and a pose estimator that takes in two adjacent RGB frames and predicts the relative change in orientation. During evaluation, we are given an initial ground truth orientation that we apply sequentially to predicted frames.

Although the GQN Mazes~\citep{eslami2018neural} already exists as a video prediction dataset, it is difficult to properly measure temporal consistency. The 3D scenes are relatively simple, and it does not have actions to help reduce stochasticity in using metrics such as PSNR, SSIM, and LPIPS. As a result, FVD is the reliable metric used in GQN Mazes, but tends to be sensitive to noise in video predictions. In addition, we perform 3D visualizations using our dataset that are not possible with GQN Mazes.

\subsection{Minecraft}
We generate 200k trajectories (each of a different Minecraft world) of 300 $128\times 128$ frames in the Minecraft marsh biome. We hardcode an agent to randomly traverse the surroundings by taking left, right, and forward actions with different probabilities. In addition, we let the agent constantly jump, which we found to help traverse simple hills, and prevent itself from drowning. We specifically chose the marsh biome, as it contains hilly turns with sparse collections of trees that act as clear landmarks for consistent generation. Forest and jungle biomes tend to be too dense for any meanginfully clear consistency, as all surroundings look nearly identical. On the other hand, plains biomes had the opposite issue where the surroundings were completely flat. Mountain biomes were too hilly and difficult to traverse.

We opt to introduce an alternative to the MineRL Navigate~\citep{guss2019minerl} since this dataset primarily consists of human demonstrations of people navigating to specific points. This means that trajectories usually follow a relatively straight line, so there are not many long-term dependencies in this dataset, as only a few past frames of context are necessary for prediction.

\subsection{Habitat}
Habitat is a 3D simulator that can render realistic trajectories in scans of 3D scenes. We compile roughly 1400 3D scans from HM3D~\citep{ramakrishnan2021habitat}, MatterPort3D~\citep{chang2017matterport3d} and Gibson~\citep{xia2018gibson}, and generate a total of 200k trajectories of $300$ $128 \times 128$ frames. We use the in-built path traversal algorithm provided in Habitat to construct action trajectories that allow our agent to move between randomly sampled locations in the 3D scene. Similar to Minecraft and DMLab, the agent action space consists of left turn, right turn, and move forward.

\clearpage


\section{Hyperparameters}
\label{section:hyperparams}
\subsection{VQ-GAN \& VAE}
\begin{table}[H]
\centering
\begin{tabular}{@{}lcc@{}}
\toprule
                       & DMLab / Minecraft       & Habitat / Kinetics-600  \\ \midrule
GPU Days               & 16                      & 32                      \\
Resolution             & 64 / 128                & 128                     \\
Batch Size             & 64                      & 64                      \\
LR                     & $3\times 10^{-4}$       & $3\times 10^{-4}$       \\
Num Res Blocks           & 2                       & 2                       \\
Attention Resolutions  & 16                      & 16                      \\
Channel Mult           & 1,2,2,2                 & 1,2,3,4                 \\
Base Channels          & 128                     & 128                     \\
Latent Size (VQ-GAN)   & $16 \times 16$          & $16 \times 16$          \\
Embedding Dim (VQ-GAN) & 256                     & 256                     \\
Codebook Size (VQ-GAN) & 1024                    & 8192                    \\
Latent Size (VAE)      & $16 \times 16 \times 4$ & $16 \times 16 \times 8$ \\ \bottomrule
\end{tabular}
\end{table}

\subsection{TECO}
\begin{table}[H]
\centering
\begin{tabular}{@{}llcccc@{}}
\toprule
\multicolumn{2}{c}{Hyperparameters}                                                                    & DMLab             & Minecraft         & Habitat           & Kinetics-600      \\ \midrule
                                                                                & TPU-v3 Days          & 32                & 80                & 275               & 640               \\
                                                                                & Params               & 169M              & 274M              & 386M              & 1.09B             \\
                                                                                & Resolution           & 64                & 128               & 128               & 128               \\
                                                                                & Batch Size           & 32                & 32                & 32                & 32                \\
                                                                                & Sequence Length      & 300               & 300               & 300               & 100               \\
                                                                                & LR                   & $1\times 10^{-4}$ & $1\times 10^{-4}$ & $1\times 10^{-4}$ & $1\times 10^{-4}$ \\
                                                                                & LR Schedule          & cosine            & cosine            & cosine            & cosine            \\
                                                                                & Warmup Steps         & 10k               & 10k               & 10k               & 10k               \\
                                                                                & Total Training Steps & 1M                & 1M                & 1M                & 1M                \\
                                                                                & DropLoss Rate        & 0.9               & 0.9               & 0.9               & 0.9               \\ \midrule
Encoder                                                                         & Depths               & 256, 512          & 256, 512          & 256, 512          & 256, 512          \\
                                                                                & Blocks               & 2                 & 4                 & 4                 & 8                 \\ \midrule
\multirow{2}{*}{Codebook}                                                       & Size                 & 1024              & 1024              & 1024              & 1024              \\
                                                                                & Embedding Dim        & 32                & 32                & 32                & 32                \\ \midrule
Decoder                                                                         & Depths               & 256, 512          & 256, 512          & 256, 512          & 256, 512          \\
                                                                                & Blocks               & 4                 & 8                 & 8                 & 10                \\ \midrule
\multirow{6}{*}{\begin{tabular}[c]{@{}l@{}}Temporal\\ Transformer\end{tabular}} & Downsample Factor    & 8                 & 8                 & 4                 & 2                 \\
                                                                                & Hidden Dim           & 1024              & 1024              & 1024              & 1536              \\
                                                                                & Feedforward Dim      & 4096              & 4096              & 4096              & 6144              \\
                                                                                & Heads                & 16                & 16                & 16                & 24                \\
                                                                                & Layers               & 8                 & 12                & 8                 & 24                \\
                                                                                & Dropout              & 0                 & 0                 & 0                 & 0                 \\ \midrule
\multirow{6}{*}{MaskGit}                                                        & Mask Schedule        & cosine            & cosine            & cosine            & cosine            \\
                                                                                & Hidden Dim           & 512               & 768               & 1024              & 1024              \\
                                                                                & Feedforward Dim      & 2048              & 3072              & 4096              & 4096              \\
                                                                                & Heads                & 8                 & 12                & 16                & 16                \\
                                                                                & Layers               & 8                 & 6                 & 16                & 24                \\
                                                                                & Dropout              & 0                 & 0                 & 0                 & 0                 \\ \bottomrule
\end{tabular}
\end{table}

\begin{table}[H]
\centering
\begin{tabular}{@{}llcccc@{}}
\toprule
                                                                                &                      & \multicolumn{4}{c}{\begin{tabular}[c]{@{}c@{}}Train Sequence Length\\ (Fewer FLOPs per Frame)\end{tabular}} \\
\multicolumn{2}{c}{Hyperparameters}                                                                    & 300                       & 200                       & 100                      & 50                       \\ \midrule
                                                                                & TPU-v3 Days          & 32                        & 32                        & 32                       & 32                       \\
                                                                                & Params               & 169M                      & 169M                      & 86M                      & 195M                     \\
                                                                                & Resolution           & 64                        & 64                        & 64                       & 64                       \\
                                                                                & Batch Size           & 32                        & 32                        & 32                       & 32                       \\
                                                                                & LR                   & $1\times 10^{-4}$         & $1\times 10^{-4}$         & $1\times 10^{-4}$        & $1\times 10^{-4}$        \\
                                                                                & LR Schedule          & cosine                    & cosine                    & cosine                   & cosine                   \\
                                                                                & Warmup Steps         & 10k                       & 10k                       & 10k                      & 10k                      \\
                                                                                & Total Training Steps & 1M                        & 1M                        & 1M                       & 1M                       \\
                                                                                & DropLoss Rate        & 0.9                       & 0.85                      & 0.85                     & 0.85                     \\ \midrule
Encoder                                                                         & Depths               & 256, 512                  & 256, 512                  & 256, 512                 & 256, 512                 \\
                                                                                & Blocks               & 2                         & 2                         & 2                        & 2                        \\ \midrule
\multirow{2}{*}{Codebook}                                                       & Size                 & 1024                      & 1024                      & 1024                     & 1024                     \\
                                                                                & Embedding Dim        & 32                        & 32                        & 32                       & 32                       \\ \midrule
Decoder                                                                         & Depths               & 256, 512                  & 256, 512                  & 256, 512                 & 256, 512                 \\
                                                                                & Blocks               & 4                         & 4                         & 4                        & 4                        \\ \midrule
\multirow{6}{*}{\begin{tabular}[c]{@{}l@{}}Temporal\\ Transformer\end{tabular}} & Downsample Factor    & 8                         & 8                         & 2                        & 2                        \\
                                                                                & Hidden Dim           & 1024                      & 1024                      & 512                      & 1024                     \\
                                                                                & Feedforward Dim      & 4096                      & 4096                      & 2048                     & 4096                     \\
                                                                                & Heads                & 16                        & 16                        & 8                        & 16                       \\
                                                                                & Layers               & 8                         & 8                         & 8                        & 8                        \\
                                                                                & Dropout              & 0                         & 0                         & 0                        & 0                        \\ \midrule
\multirow{6}{*}{MaskGit}                                                        & Mask Schedule        & cosine                    & cosine                    & cosine                   & cosine                   \\
                                                                                & Hidden Dim           & 512                       & 512                       & 512                      & 768                      \\
                                                                                & Feedforward Dim      & 2048                      & 2048                      & 2048                     & 3072                     \\
                                                                                & Heads                & 8                         & 8                         & 8                        & 12                       \\
                                                                                & Layers               & 8                         & 8                         & 8                        & 8                        \\
                                                                                & Dropout              & 0                         & 0                         & 0                        & 0                        \\ \bottomrule
\end{tabular}
\caption{Hyperparameters for scaling TECO on DMLab}
\end{table}

\begin{table}[H]
\centering
\begin{tabular}{@{}llcccc@{}}
\toprule
                                                                                &                      & \multicolumn{4}{c}{\begin{tabular}[c]{@{}c@{}}Train Sequence Length\\ (Fewer FLOPs per Frame)\end{tabular}} \\
Hyperparameters &                                                                    & 300                       & 200                       & 100                      & 50                       \\ \midrule
                                                                                & TPU-v3 Days          & 80                        & 80                        & 80                       & 80                       \\
                                                                                & Params               & 274M                      & 261M                      & 257M                     & 140M                     \\
                                                                                & Resolution           & 128                       & 128                       & 128                      & 128                      \\
                                                                                & Batch Size           & 32                        & 32                        & 32                       & 32                       \\
                                                                                & LR                   & $1\times 10^{-4}$         & $1\times 10^{-4}$         & $1\times 10^{-4}$        & $1\times 10^{-4}$        \\
                                                                                & LR Schedule          & cosine                    & cosine                    & cosine                   & cosine                   \\
                                                                                & Warmup Steps         & 10k                       & 10k                       & 10k                      & 10k                      \\
                                                                                & Total Training Steps & 1M                        & 1M                        & 1M                       & 1M                       \\
                                                                                & DropLoss Rate        & 0.9                       & 0.85                      & 0.25                     & 0.25                     \\ \midrule
Encoder                                                                         & Depths               & 256, 512                  & 256, 512                  & 256, 512                 & 256, 512                 \\
                                                                                & Blocks               & 4                         & 4                         & 4                        & 4                        \\ \midrule
\multirow{2}{*}{Codebook}                                                       & Size                 & 1024                      & 1024                      & 1024                     & 1024                     \\
                                                                                & Embedding Dim        & 32                        & 32                        & 32                       & 32                       \\ \midrule
Decoder                                                                         & Depths               & 256, 512                  & 256, 512                  & 256, 512                 & 256, 512                 \\
                                                                                & Blocks               & 8                         & 8                         & 8                        & 8                        \\ \midrule
\multirow{6}{*}{\begin{tabular}[c]{@{}l@{}}Temporal\\ Transformer\end{tabular}} & Downsample Factor    & 8                         & 4                         & 2                        & 2                        \\
                                                                                & Hidden Dim           & 1024                      & 1024                      & 1024                     & 512                      \\
                                                                                & Feedforward Dim      & 4096                      & 4096                      & 4096                     & 2048                     \\
                                                                                & Heads                & 16                        & 16                        & 16                       & 8                        \\
                                                                                & Layers               & 12                        & 12                        & 12                       & 12                       \\
                                                                                & Dropout              & 0                         & 0                         & 0                        & 0                        \\ \midrule
\multirow{6}{*}{MaskGit}                                                        & Mask Schedule        & cosine                    & cosine                    & cosine                   & cosine                   \\
                                                                                & Hidden Dim           & 768                       & 768                       & 768                      & 768                      \\
                                                                                & Feedforward Dim      & 3072                      & 3072                      & 3072                     & 3072                     \\
                                                                                & Heads                & 12                        & 12                        & 12                       & 12                       \\
                                                                                & Layers               & 6                         & 6                         & 6                        & 8                        \\
                                                                                & Dropout              & 0                         & 0                         & 0                        & 0                        \\ \bottomrule
\end{tabular}
\caption{Hyperparameters for scaling TECO on Minecraft}
\end{table}

\subsection{Latent FDM}

\begin{table}[H]
\centering
\begin{tabular}{@{}lcccc@{}}
\toprule
Hyperparameters       & DMLab             & Minecraft         & Habitat           & Kinetics-600      \\ \midrule
TPU-v3 Days           & 32                & 80                & 275               & 640               \\
Params                & 31M               & 33M               & 87M               & 831M              \\
Resolution            & 64                & 128               & 128               & 128               \\
Batch Size            & 32                & 32                & 32                & 32                \\
LR                    & $1\times 10^{-4}$ & $1\times 10^{-4}$ & $1\times 10^{-4}$ & $1\times 10^{-4}$ \\
LR Schedule           & cosine            & cosine            & cosine            & cosine            \\
Optimizer             & Adam              & Adam              & Adam              & Adam              \\
Warmup Steps          & 10k               & 10k               & 10k               & 10k               \\
Total Training Steps  & 1M                & 1M                & 1M                & 1M                \\
Base Channels         & 128               & 128               & 128               & 256               \\
Num Res Blocks        & 1,1,1,2           & 1,1,2,2           & 1,2,2,4           & 2,2,2,2           \\
Head Dim              & 64                & 64                & 64                & 64                \\
Attention Resolutions & 4,2               & 4,2               & 4,2               & 8,4,2             \\
Dropout               & 0                 & 0                 & 0                 & 0                 \\
Channel Mult          & 1,1,1,2           & 1,2,2,2           & 1,2,2,4           & 1,2,3,8           \\ \bottomrule
\end{tabular}
\caption{Hyperparameters for Latent FDM}
\end{table}

\begin{table}[H]
\centering
\begin{tabular}{@{}lcccc@{}}
\toprule
                      & \multicolumn{4}{c}{\begin{tabular}[c]{@{}c@{}}Train Sequence Length\\ (Fewer FLOPs per Frame)\end{tabular}} \\
Hyperparameters       & 300                       & 200                       & 100                      & 50                       \\ \midrule
TPU-v3 Days           & 32                        & 32                        & 32                       & 32                       \\
Params                & 31M                       & 62M                       & 80M                      & 110M                     \\
Resolution            & 64                        & 64                        & 64                       & 64                       \\
Batch Size            & 32                        & 32                        & 32                       & 32                       \\
LR                    & $1\times 10^{-4}$         & $1\times 10^{-4}$         & $1\times 10^{-4}$        & $1\times 10^{-4}$        \\
LR Schedule           & cosine                    & cosine                    & cosine                   & cosine                   \\
Optimizer             & Adam                      & Adam                      & Adam                     & Adam                     \\
Warmup Steps          & 10k                       & 10k                       & 10k                      & 10k                      \\
Total Training Steps  & 1M                        & 1M                        & 1M                       & 1M                       \\
Base Channels         & 128                       & 128                       & 128                      & 192                      \\
Num Res Blocks        & 1,1,1,2                   & 1,1,2,2,4                 & 2,2,2,2                  & 3,3,3,3                  \\
Head Dim              & 64                        & 64                        & 64                       & 64                       \\
Attention Resolutions & 4,2                       & 4,1                       & 4,2                      & 8,4,2                    \\
Dropout               & 0                         & 0                         & 0                        & 0                        \\
Channel Mult          & 1,1,1,2                   & 1,1,2,2,4                 & 1,2,3,4                  & 1,2,3,4                  \\ \bottomrule
\end{tabular}
\caption{Hyperparameters for scaling Latent FDM on DMLab}
\end{table}

\begin{table}[H]
\centering
\begin{tabular}{@{}lcccc@{}}
\toprule
                      & \multicolumn{4}{c}{\begin{tabular}[c]{@{}c@{}}Train Sequence Length\\ (Fewer FLOPs per Frame)\end{tabular}} \\
Hyperparameters       & 300                       & 200                       & 100                      & 50                       \\ \midrule
TPU-v3 Days           & 80                        & 80                        & 80                       & 80                       \\
Params                & 33M                       & 80M                       & 69M                      & 186M                     \\
Resolution            & 128                       & 128                       & 128                      & 128                      \\
Batch Size            & 32                        & 32                        & 32                       & 32                       \\
LR                    & $1\times 10^{-4}$         & $1\times 10^{-4}$         & $1\times 10^{-4}$        & $1\times 10^{-4}$        \\
LR Schedule           & cosine                    & cosine                    & cosine                   & cosine                   \\
Optimizer             & Adam                      & Adam                      & Adam                     & Adam                     \\
Warmup Steps          & 10k                       & 10k                       & 10k                      & 10k                      \\
Total Training Steps  & 1M                        & 1M                        & 1M                       & 1M                       \\
Base Channels         & 128                       & 128                       & 128                      & 192                      \\
Num Res Blocks        & 1,1,2,2                   & 2,2,2,2                   & 3,3,3,3                  & 2,2,2,2                  \\
Head Dim              & 64                        & 64                        & 64                       & 64                       \\
Attention Resolutions & 4,2                       & 4,2                       & 8,4,2                    & 8,4,2                    \\
Dropout               & 0                         & 0                         & 0                        & 0                        \\
Channel Mult          & 1,2,2,2                   & 1,2,3,4                   & 1,2,2,3                  & 1,2,3,4                  \\ \bottomrule
\end{tabular}
\caption{Hyperparameters for scaling Latent FDM on Minecraft}
\end{table}
\clearpage

\subsection{CW-VAE}
\begin{table}[H]
\centering
\begin{tabular}{@{}llcc@{}}
\toprule
\multicolumn{2}{l}{Hyperparameters}              & DMLab             & Minecraft         \\ \midrule
                          & TPU-v3 Days          & 32                & 80                \\
                          & Params               & 111M              & 140M              \\
                          & Resolution           & 64                & 128               \\
                          & Batch Size           & 32                & 32                \\
                          & LR                   & $1\times 10^{-4}$ & $1\times 10^{-4}$ \\
                          & LR Schedule          & cosine            & cosine            \\
                          & Optimizer            & Adam              & Adam              \\
                          & Warmup Steps         & 10k               & 10k               \\
                          & Total Training Steps & 1M                & 1M                \\ \midrule
\multirow{2}{*}{Encoder}  & Kernels              & 4,4,4             & 4,4,4             \\
                          & Filters              & 256,512,1024      & 256,512,1024      \\ \midrule
\multirow{2}{*}{Decoder}  & Depths               & 256,512           & 256,512           \\
                          & Blocks               & 4                 & 8                 \\ \midrule
\multirow{8}{*}{Dynamics} & Levels               & 3                 & 3                 \\
                          & Abs Factor           & 6                 & 6                 \\
                          & Enc Dense Layers     & 3                 & 3                 \\
                          & Enc Dense Embed      & 1024              & 1024              \\
                          & Cell Stoch Size      & 128               & 256               \\
                          & Cell Deter Size      & 1024              & 1024              \\
                          & Cell Embed Size      & 1024              & 1024              \\
                          & Cell Min Stddev      & 0.001             & 0.001             \\ \bottomrule
\end{tabular}
\caption{Hyperparameters for CW-VAE}
\end{table}

\subsection{FitVid}
\begin{table}[H]
\centering
\begin{tabular}{@{}lcc@{}}
\toprule
Hyperparameters      & DMLab             & Minecraft         \\ \midrule
TPU-v3 Days          & 32                & 80                \\
Params               & 165M              & 176M              \\
Resolution           & 64                & 128               \\
Batch Size           & 32                & 32                \\
LR                   & $1\times 10^{-4}$ & $1\times 10^{-4}$ \\
LR Schedule          & cosine            & cosine            \\
Optimizer            & Adam              & Adam              \\
Warmup Steps         & 10k               & 10k               \\
Total Training Steps & 1M                & 1M                \\
g Dim                & 256               & 256               \\
RNN Size             & 512               & 768               \\
z Dim                & 64                & 128               \\
Filters              & 128,128,256,512   & 128,128,256,512   \\ \bottomrule
\end{tabular}
\caption{Hyperparameters for FitVid}
\end{table}
\section{Preliminaries}

\subsection{VQ-GAN}
VQ-GAN~\citep{esser2021taming, van2017neural} is an autoencoder that learns to compress data into a set of discrete latents, consisting of an encoder $E$, decoder $G$, codebook $C$, and discriminator $D$. Given an image $x\in\mathbb{R}^{H\times W\times 3}$, the encoder $E$ maps $x$ to its latent representation $h\in\mathbb{R}^{H'\times W'\times D}$, which is quantized by nearest neighbors lookup in a codebook of embeddings $C = \{e_i\}^K_{i=1}$ to produce $z \in \mathbb{R}^{H'\times W' \times D}$. The discretized latent $z$ is fed through decoder $G$ to reconstruct $x$. A straight-through estimator~\citep{bengio2013estimating} is used to maintain gradient flow through the quantization step. The codebook optimizes the following loss:

\begingroup\removeParBefore\endlinechar=32 \eqinner{
    \label{eq:vq}
    \mathcal{L}_{\mathrm{VQ}} &= \norm{\operatorname{sg}(h) - e}_2^2 + \beta\norm{h - \operatorname{sg}(e)}_2^2
}

where $\beta=0.25$ is a hyperparameter, and $e$ is the corresponding nearest-neighbors embedding from codebook $C$. For reconstruction, VQ-GAN replaces the original $\ell_2$ loss with a perceptual loss~\citep{zhang2018unreasonable}, $\mathcal{L}_{\mathrm{LPIPS}}$. Finally, in order to encourage higher-fidelity samples, patch-level discriminator $D$ is trained to classify between real and reconstructed images, with.

\begingroup\removeParBefore\endlinechar=32 \eqinner{
    \mathcal{L}_{\mathrm{GAN}} &= \log{D(x)} + \log(1 - D(\hat{x}))
}

Overall, VQ-GAN optimizes the following combination of losses:
\begingroup\removeParBefore\endlinechar=32 \eqinner{
    \min_{E,G,C}\max_{D} \,\,\, \mathcal{L}_{\mathrm{LPIPS}} + \mathcal{L}_{\mathrm{VQ}} + \lambda \mathcal{L}_{\mathrm{GAN}}
}
where $\lambda = \frac{\norm{\nabla_{G_L}\mathcal{L}_{\mathrm{LPIPS}}}_2}{\norm{\nabla_{G_L}\mathcal{L}_{\mathrm{GAN}}}_2 + \delta}$ is an adaptive weight, $G_L$ is the last decoder layer, and $\delta = 10^{-6}$.

\subsection{MaskGit}
MaskGit~\citep{chang2022maskgit} is a generative model that models distributions over tokens, such as produced by a VQ-GAN. Instead of autoregressively modelling the sequence of tokens, MaskGit generates images with competitive sample quality at a fraction of the sampling cost by using a masked token prediction objective during training. Formally, we denote $z \in \mathbb{Z}^{H\times W}$ as the discrete latent tokens representing an image. For each training step, we uniformly sample $t\in [0, 1)$ and randomly generate a mask $m\in \{0,1\}^{H\times W}$ with $N=\lceil \gamma HW\rceil$ masked values, where $\gamma = \cos\left(\frac{\pi}{2}t\right)$. Then, MaskGit learns to predict the masked tokens with the following objective
\begingroup\removeParBefore\endlinechar=32 \eqinner{
    \mathcal{L}_{\mathrm{mask}} = -\mathbb{E}_{z\in\mathcal{D}}[\log \p(z|z\odot m)].
}
During inference, because MaskGit has been trained to model any set of unconditional and conditional probabilities, we can sample any subset of tokens per sampling iteration, from the extreme case of sampling all tokens (independent) to sampling one token at a time (autoregressive). \cite{chang2022maskgit} introduces a confidence-based sampling mechanism whereas other work~\citep{lee2022draft} proposes iterative sample-and-revise approaches.

\section{Discussion}
\vspace*{-1ex}

We introduced TECO, an efficient video prediction model that leverages hundreds of frames of temporal context. Our evaluation demonstrated that TECO accurately incorporates long-range context, outperforming SOTA baselines across a wide range of datasets. In addition, we introduce several difficult video datasets, which we hope make it easier to evaluate temporal consistency in future video prediction models. We identify several limitations as directions for future work:
\begin{itemize}
    \item Although we show that PSNR, SSIM, and LPIPS can be reliable metrics to measure consistency when video models are properly conditioned, there remains room for better evaluation metrics that provide a reliable signal as the prediction horizon grows, since new parts of a scene that are generated are unlikely to correlate with ground truth.
    
    \item Our focus was on learning a compressed tokens and an expressive prior, which we combined with a simple full attention transformer as the sequence model. Leveraging prior work on efficient sequence models~\citep{choromanski2020rethinking,wang2020linformer,zhai2021attention,gu2021efficiently,hawthorne2022general} would likely allow for further scaling. 
   
    
    \item We trained all models on top of pretrained VQ-GAN codes to reduce the data dimensionality. This compression step lets us train on longer sequences at a cost of reconstruction error, which causes noticeable artifacts in Kinetics-600, such as corrupted text and incoherent faces. Although TECO can train directly on pixels, a $\ell_2$ loss results in slightly blurry predictions. Training directly on pixels with diffusion or GAN losses would be promising.
  
\end{itemize}

\section{Experiments}



\subsection{Datasets}
We introduce three challenging video datasets to better measure long-range consistency in video prediction. We design these benchmarks around 3D environments in DMLab~\citep{beattie2016deepmind}, Minecraft~\citep{guss2019minerl}, and Habitat~\citep{habitat19iccv}, with videos of agents randomly traversing different scenes of varying difficulty. These datasets require video prediction models to re-produce observed parts of scenes, and newly generate unobserved parts of the scene. In contrast, many existing video benchmarks do not have strong long-range dependencies, where a model with limited context is sufficient. Refer to \Cref{section:dataset_details} for further details on each dataset.

\textbf{DMLab}\quad DeepMind Lab is a simulator that procedurally generates random 3D mazes with random floor and wall textures. We generate 40k action-conditioned $64\times 64$ videos of $300$ frames of an agent randomly traversing $7\times 7$ mazes by choosing random points in the maze and navigating to them via the shortest path. We train all models for both action-conditioned and unconditional prediction (by periodically masking out actions) to enable both types of generations. We use both modes to evaluate since a video model may generate new parts of a scene that do not correlate with the action (e.g. run into a wall) which results in out-of-distribution errors. However, action-conditioning is useful with enough conditioned past context, and substantially lowers variance on PSNR, SSIM, and LPIPS evaluations.

\textbf{Minecraft}\quad This popular game features procedurally generated 3D worlds that contain complex terrain such as hills, forests, rivers, and lakes. We collect 200k action-conditioned videos of length $300$ and resolution $128 \times 128$ in Minecraft's marsh biome. The player iterates between walking forward for a random number of steps and randomly rotating left or right, resulting in parts of the scene going out of view and coming back into view later. We train action-conditioned for all models for ease of interpreting and evaluating, though it is generally easy for video models to unconditionally learn these discrete actions.

\textbf{Habitat}\quad Habitat is a simulator for rendering trajectories through scans of real 3D scenes. We compile $\sim$1400 indoor scans from HM3D~\citep{ramakrishnan2021habitat}, MatterPort3D~\citep{chang2017matterport3d}, and Gibson~\citep{xia2018gibson} to generate 200k action-conditioned videos of $300$ frames at a resolution of $128 \times 128$ pixels. We use the in-built path traversal algorithm provided in Habitat to construct action trajectories that allow our agent to move between randomly sampled locations in the 3D scene. Similar to DMLab, we train all video models to perform both unconditional and action-conditioned prediction.

\textbf{Kinetics-600}\quad Kinetics-600~\citep{carreira2017quo} is a highly complex real-world video dataset, originally proposed for action recognition. The dataset contains $\sim$400k videos of varying length of up to 300 frames. We evaluate our method in the video prediction without actions (as they do not exist), generating 80 future frames conditioned on 20. In addition, we filter out videos shorter than 100 frames, leaving 392k videos that are split for training and evaluation. We use a resolution of $128 \times 128$ pixels. Although Kinetics-600 does not have many long-range dependencies, we evaluate our method on this dataset to show that it can scale to complex, natural video.

\begin{table}[!t]
\vspace*{-3ex}
\caption{Quantitative evaluation on all four datasets. Detailed results in \Cref{section:full_results}.}
\label{table:main}
\vspace*{-1ex}
\begin{tabular}{@{}lccccccccc@{}}
\toprule
                                 & \multicolumn{4}{c}{DMLab}                                            & \multicolumn{4}{c}{Minecraft}                                              \\
Method                       & FVD $\downarrow$ & PSNR $\uparrow$ & SSIM $\uparrow$  & LPIPS $\downarrow$ & FVD $\downarrow$ & PSNR $\uparrow$ & SSIM $\uparrow$  & LPIPS $\downarrow$ \\ \midrule
FitVid                       & $176$            & $12.0$          & $0.356$          & $0.491$            & $956$            & $13.0$          & $0.343$          & $0.519$            \\
CW-VAE                       & $125$            & $12.6$          & $0.372$          & $0.465$            & $397$            & $13.4$          & $0.338$          & $0.441$            \\
Perceiver \rlap{AR}                 & $\hphantom{0}96$           & $11.2$          & $0.304$          & $0.487$            & $\hphantom{0}\textbf{76}$            & $13.2$          & $0.323$          & $0.441$            \\ 
Latent FDM  & $181$            & $17.8$          & $0.588$          & $0.222$            & $167$            & $13.4$          & $0.349$          & $0.429$            \\ 
TECO (ours) & $\hphantom{0}\mathbf{28}$  & $\mathbf{22.4}$ & $\mathbf{0.709}$ & $\mathbf{0.155}$   & $116$            & $\mathbf{15.4}$          & $\mathbf{0.381}$          & $\mathbf{0.340}$            \\
\bottomrule
\toprule
             & \multicolumn{4}{c}{Habitat}                                                & \multicolumn{4}{c}{Kinetics-600}                                           \\
Method       & FVD $\downarrow$ & PSNR $\uparrow$ & SSIM $\uparrow$  & LPIPS $\downarrow$ & FVD $\downarrow$ & PSNR $\uparrow$ & SSIM $\uparrow$  & LPIPS $\downarrow$ \\ \midrule
Perceiver AR & $164$            & $\mathbf{12.8}$ & $\mathbf{0.405}$ & $0.676$            & $\llap{1}022$           & $13.4$          & $0.310$          & $0.404$            \\
Latent FDM   & $433$            & $12.5$          & $0.311$          & $\mathbf{0.582}$   & $960$            & $13.2$          & $0.334$          & $0.413$            \\
TECO (ours)  & $\hphantom{0}\mathbf{73}$    & $\mathbf{12.8}$ & $0.363$          & $0.604$            & $\mathbf{799}$   & $\mathbf{13.8}$ & $\mathbf{0.341}$ & $\mathbf{0.381}$   \\ \bottomrule
\end{tabular}
\end{table}

\subsection{Baselines}
We compare against SOTA baselines selected from several different families of models: latent-variable-based variational models, autoregressive likelihood models, and diffusion models. In addition, for more fair comparisons, we train all models on VQ codes using the same VQ-GAN as our method. For our diffusion baseline, we follow \cite{rombach2022high} and use a pretrained VAE instead of a VQ-GAN. Note that we do not have any GANs for our baselines, since to the best of our knowledge, there does not exist a GAN that trains on latent space instead of raw pixels, an important aspect for properly scaling to long video sequences.

\textbf{FitVid}\quad
FitVid~\citep{babaeizadeh2021fitvid} is a state-of-the-art variational video prediction model based on CNNs and LSTMs that scales to complex video by leveraging efficient architectural design choices in its encoder and decoder.

\textbf{Clockwork VAE}\quad
CW-VAE \citep{saxena2021clockwork} is also a variational video prediction model that is designed to better learn long-range dependencies through a hierarchies of latent variables with exponentially slower tick speeds for each new level. 

\textbf{Perceiver AR}\quad
We use Perceiver AR~\citep{hawthorne2022general} as our AR baseline over VQ-GAN discrete latents, which has been show to be an effective generative model that can efficiently incorporate long-range sequential dependencies. Conceptually, this baseline is similar to HARP~\citep{seo2022harp} with a Perceiver AR as the prior instead of a sparse transformer~\citep{child2019generating}. We choose Perceiver AR over other autoregressive baselines such as VideoGPT~\citep{yan2021videogpt} or TATS~\citep{ge2022long} primarily due to the prohibitive costs of transformers when applied to tens of thousands of tokens.

\textbf{Latent FDM}\quad
For our diffusion baseline, we train a Latent FDM model with frame-wise autoregressive sampling. Although FDM~\citep{harvey2022flexible} is originally trained on pixel observations, we also train in latent space for a more fair comparison with our method and other baselines, as training on long sequences in pixel space is too expensive. We follow LDM~\citep{rombach2022high} to separately train an autoencoder to encode each frame into a set of continuous latents. 

\subsection{Experimental Setup}
\textbf{Training}\quad
All of our models are trained for 1 million iterations under fixed compute budget (measured in TPU v3 days) allocated for each dataset. Models are trained on TPU-v3 instances, ranging from v3-8 to v3-128 TPU pods (similar to 4 V100s to 64 V100s) with training times of roughly 3-5 days. For DMLab, Minecraft, and Habitat we train all models on full 300 frames videos, and 100 frames for Kinetics-600. Our VQ-GANs are trained on 8 A5000 GPUs, taking about 2-4 days for each dataset, and downsample all videos to $16\times 16$ grids of discrete latents per frame regardless of original video resolution. More details on exact hyperparameters and compute budgets for each dataset can be found in \Cref{section:hyperparams}.

\textbf{Evaluation}\quad
We evaluate our models using a combination of standard video prediction metrics such as PSNR~\citep{huynh2008scope}, SSIM~\citep{wang2004image}, LPIPS~\citep{zhang2018unreasonable}, and FVD~\citep{unterthiner2019fvd}. For DMLab, Minecraft, and Habitat, we measure FVD on 300 frame videos, conditioned on 36 frames (264 predicted frames). For Kinetics-600, we evaluate FVD on 100 frame videos, conditioned on 20 frames (80 predicted frames). To evaluate temporal consistency, we measure PSNR, SSIM, and LPIPS on video predictions conditioned on 144 frames (156 predicted frames), and action condition for all models. Conditioning on a large portion of the video ensures that the model can observe a large part of the scene, and combined with action-conditioning, the model with temporally-consistent predictions should generate future frames close to the ground truth. Due to this reduced stochasticity, we only sample one prediction for computing PSNR, SSIM, and LPIPS. We compute all metrics over batches of $256$ examples, averaged over 4 runs to make $1024$ total samples.

\subsection{Benchmark Results}
\textbf{DMLab \& Minecraft}\quad
\Cref{table:main} shows quantitative results on the DMLab and Minecraft datasets. TECO performs the best across all metrics for both datasets when training on the full 300 frame videos.  \Cref{fig:dm_lab_3d_traj} shows sample trajectories and 3D visualizations of the generated DMLab mazes, where TECO is able to generate more stable and consistent 3D mazes. For both datasets, CW-VAE, FitVid, and Perceiver AR can produce sharp predictions, but do not model long-horizon context well, with per-frame metrics sharply dropping as the future prediction horizon increases as seen in \Cref{fig:horizon}. Latent FDM has consistent predictions, but high FVD most likely due to FVD being sensitive to high frequency errors.

In order to better investigate scaling properties of our models, \Cref{fig:scaling_dmlab} and \Cref{fig:scaling_minecraft} compare TECO and Latent FDM on different training sequence lengths. Intuitively, under a fixed computation budget and batch size, models that train on shorter sequence lengths can scale larger, with more FLOPs allocated per frame. In general, this is reflected in model architectures through computations at higher spatial resolutions (e.g. less downsampling). For DMLab, we see that in terms of per-frame metrics, models generally benefit from training on longer videos, where more computation per image has less of an effect due to saturation in image quality because of relatively simple visual complexity. For Minecraft, we observe that models generally perform best when training with 100 frames of context, which have better per-image sample quality compared to training on 300 frames due to higher downsampling required for longer sequences. Models trained on 300 frames generally have more distortion in predictions compared to 100 frames. Theoretically, as the compute budget is increased, training on 300 frames would eventually outperform models trained on 100 frames.


\begin{figure}[t]
    \centering
    \vspace*{-5ex}
    \includegraphics[width=0.95\linewidth]{figures/samples/dmlab_3d.pdf}
    \caption{3D visualization of predicted trajectories in DMLab for each model, generating 264 frames conditioned on 36. Note that predictions do not necessarily match ground truth since new parts of a maze can be generated given limited context. \textbf{Video predictions use only the first-person RGB frames}. Refer to \Cref{section:dataset_dmlab_details} for more details on 3D evaluation. A video corresponding to this figure is available at: \url{https://wilson1yan.github.io/teco}.}
    \label{fig:dm_lab_3d_traj}
    \vspace*{-3ex}
\end{figure}





\textbf{Habitat}\quad
\Cref{table:main}
shows results for our Habitat dataset. We only evaluate our strongest baselines, Perceiver AR and Latent FDM due to the need to implement model parallel variants for all of our models. Because of high complexity of Habitat videos, all models generally perform equally as bad in per-frame metrics. However, TECO has significantly better FVD. Qualitatively, Latent FDM quickly collapses to blurred predictions with poor sample quality, and although Perceiver AR can generate high quality frames, they are less temporally consistent than TECO: agents in Habitat videos navigate to far points in the scene and back whereas Perceiver AR tends to generate samples where the agent is constantly turning. TECO generates traversals of a scene that match the data distribution more closely.

\textbf{Kinetics-600}\quad
\Cref{table:main} shows FVD for predicting $80$ $128\times128$ frames conditioned on $20$ for Kinetics-600. Although Kinetics-600 does not have many long-range dependencies, we found that TECO is able to produce more stable generations that degrade slower by incorporating longer contexts. In contrast, Perceiver AR tends to degrade quickly, with Latent FDM performing in between. \Cref{fig:kinetics_top_k} and \Cref{table:kinetics_full} include further investigations using top-k sampling for Perceiver AR and TECO. \Cref{table:main} does not use top-k sampling for a fair comparison against Latent FDM. With top-k sampling, Perceiver AR outperforms our method at $k = 8$. However, resulting videos tend to be uninteresting with little to no dynamics movement.

\textbf{Sampling Speed}\quad
\Cref{fig:main_exp} compares sampling speed for all models. We report sampling speed on Minecraft and observed similar results for the different model sizes used on other datasets. FitVid and CW-VAE are both significantly faster that other methods, but have poor sample quality. On the other end, Perceiver AR and Latent FDM can produce high quality samples, but are 20-60x slower than TECO, which has comparably fast sampling speed while retaining high sample quality.

\vspace*{-.5ex}
\subsection{Ablations}
\vspace*{-1ex}
In this section, we perform ablations on various architectural decisions of our model. For simplicity, we evaluate our methods on short sequences of 16 frames from Something-Something-v2 (SSv2). We choose SSv2 as it provides insight into scaling our method on complex real-world data more similar to Kinetics-600 while being computationally cheaper to run. 

\Cref{table:abl_arch} shows several ablations comparing posterior, prior, and various architectural design choices. We demonstrate that using VQ-latent dynamics with a MaskGit prior proves better compared to alternative formulations for latent dynamics models, such as popular variational methods. In addition, we show that conditional encodings learn better representations for video predictions. We also ablate the codebook size, showing that although there exists an optimal codebook size, it does not matter too much as along as there are not too many codes, which may make it more difficult for the prior to learn. Lastly, we show the benefits of DropLoss, with up to 60\% faster training and a minimal increase in FVD. The benefits are greater for longer sequences, and allow video models to better account for long horizon context with little cost in performance.

\Cref{table:abl_scaling} shows ablations on scaling different parts of our model, such as the encoder, decoder, temporal transformer, and prior. In general, it is more beneficial to have an imbalanced encoder decoder architecture, with more parameters in the decoder. For the temporal transformer, it is more beneficial to have larger resolution features ($4\times 4$), especially for more complex data like SSv2, and less useful for visually simpler datasets such as DMLab or Minecraft. Similarly, having a larger width is more beneficial than more layers due to increased capacity to represent each frame. Lastly, for scaling the MaskGit prior, more layers is better than larger width networks.

\section{Introduction}
Recent work in video prediction has seen tremendous progress~\citep{ho2022video,clark2019adversarial,yan2021videogpt,le2021ccvs,ge2022long,tian2021good,luc2020transformation} in producing high-fidelity and diverse samples on complex video data. This can largely be attributed to a combination of increased computational resources and more compute efficient high-capacity neural architectures. However, much of this progress has focused on generating short videos, where models can perform well by basing their predictions on only a handful of previous frames.

Video prediction models with short context windows can generate long videos in a sliding window fashion. While the resulting videos can look impressive at first sight, they lack temporal consistency. We would like models to predict temporally consistent videos --- where the same content is generated if a camera pans back to a previously observed location. On the other hand, the model should imagine a new part of the scene for locations that have not yet been observed, and future predictions should remain consistent to this newly imagined part of the scene. 

Prior work has investigated techniques for modeling long-term dependencies, such as temporal hierarchies~\citep{saxena2021clockwork} and strided sampling with frame-wise interpolation~\citep{ge2022long,hong2022cogvideo}. Other methods train on sparse sets of frames selected out of long videos~\citep{harvey2022flexible,skorokhodov2021stylegan,clark2019adversarial,saito2018tganv2,yu2022generating}, or model videos via compressed representations~\citep{yan2021videogpt,rakhimov2020latent,le2021ccvs,seo2022harp,gupta2022maskvit,walker2021predicting}. Refer to \Cref{section:related_work} for more detailed discussion on related work.

Despite this progress, many methods still have difficulty scaling to datasets with many long-range dependencies. While Clockwork-VAE~\citep{saxena2021clockwork} trains on long sequences, it is limited by training time (due to a recurrent architecture) and difficult to scale to more complex data. On the other hand, transformer-based methods over latent spaces~\citep{yan2021videogpt} scale poorly to long videos due to quadratic complexity in attention, with long videos containing tens of thousands of tokens. Methods that train on subsets of tokens are limited by truncated backpropagation through time~\citep{hutchins2022block,rae2019compressive,dai2019transformer} or naive temporal operations~\citep{hawthorne2022general}.

In this paper, we introduce \textbf{Te}mporally \textbf{Co}nsistent Video Transformer (TECO), a vector-quantized latent dynamics model that effectively models long-term dependencies in a compact representation space using efficient transformers. The key contributions are summarized as follows: 
\begin{itemize}
    \item We introduce TECO, an efficient and scalable video prediction model that learns a set of compressed VQ-latents to allow for efficient training and generation.
    
    \item We propose several long-length video prediction datasets centered around 3D scenes in DMLab~\citep{beattie2016deepmind}, Minecraft~\citep{guss2019minerl}, and Habitat~\citep{szot2021habitat,habitat19iccv} to help better evaluate temporal consistency in video predictions.
    
    \item We show that TECO has strong performance on a variety of difficult video prediction tasks, and is able to leverage long-term temporal context to generate high quality videos with consistency.
    
    \item We provide several ablations providing intuition for why TECO is able to generate more temporally consistency predictions, and how these insights can extend to future work in long-term video prediction.
\end{itemize}


\section{\method{}}
Generating temporally consistent videos requires training on long videos to correctly learn long-term temporal dependencies between frames. However, computational and memory requirements remain the primary bottleneck in preventing from doing so. We present \textbf{Te}mporally \textbf{Co}nsistent Video Transformer (\method{}), a video generation model that more efficiently scales to training on longer horizon videos. 

First, we train a VQ-GAN to spatially compress our video data. Shown in prior work~\citep{seo2022harp}, this is an important step for video prediction in a more efficient and scalable manner. However, even in latent space, existing methods are still limited to modeling short sequences of 16--24 frames, which can be attributed to the quadratics costs of transformer layers as sequence length grows. With 256 tokens per frame, 16 frame videos already consist of 4096 tokens, and scaling to longer videos of 100s frames is prohibitively expensive, where resulting videos have tens of thousands of tokens. Therefore, in the following sections, we propose several key design choices to building a more efficient video prediction model.

\subsection{Vector-Quantized Latent Dynamics} 
Our proposed framework shown in \cref{fig:arch} follows similarly to prior work in latent dynamics models~\citep{hafner2019learning,hafner2020mastering,saxena2021clockwork}, with several key differences in architectural and latent variable design. Let $x_{1:T}$ consist of a sequence of video frames encoded using a pretrained VQ-GAN. In the following sections, we motivate each component for our model, with several specific design choices to ensure efficiency and scalability. TECO consists of four components:

\begingroup\removeParBefore\endlinechar=32 \eqinner{
    &\text{Encoder:}\quad &&z_t = E(x_t, x_{t-1})\qquad &\text{Temporal Transformer:} &&h_t = H(z_{<t}) \\
    &\text{Dynamics Prior:}\quad && \p(z_t\mid h_t) \quad\, &\text{Decoder:} && \p(x_t\mid z_t,h_t)
}

\textbf{Encoder}\quad
Although VQ-GAN exploits spatial redundancies, we can achieve more compressed representations by leveraging temporal redundancy in video data. To do this, we learn a CNN encoder $z_t = E(x_t, x_{t-1})$ which encodes the current frame $x_t$ conditioned on the previous frame by channel-wise concatenating $x_{t-1}$, and then quantizes the output using codebook $C$ to produce $z_t$. We apply the VQ loss defined in \cref{eq:vq} per timestep. In addition, we $\ell_2$-normalize the codebook and embeddings to encourage higher codebook usage~\citep{yu2021vector}. Conditionally encoding nearby frames lets the model learn smaller latents, and provides a general way to take advantage of temporal redundancy. The first frame is concatenated with zeros and does not quantize $z_1$ to prevent information loss. As we focus on video prediction, there is always at least 1 frame to condition on, so we do not need to predict the un-quantized representation of the first frame when computing decoding and dynamics losses. Intuitively, this also does not burden the dynamics model to learn an unconditional prior.

\textbf{Temporal Transformer}\quad
Compressed, discrete latents are more lossy and tend to require higher spatial resolutions compared to continuous latents. Therefore, before modeling temporal information, we apply a single strided convolution to downsample each discrete latent $z_t$, where visually simpler datasets allow for more downsampling and visually complex datasets require less downsampling. Afterwards, we learn a large transformer to model temporal dependencies, and then apply a transposed convolution to upsample our representation back to the original resolution of $z_t$. In summary, we use the following architecture:

\begingroup\removeParBefore\endlinechar=32 \eqinner{
    h_t &= H(z_{<t}) = \text{ConvTranspose}(\text{Transformer}(\text{Conv}(z_{<t})))
}

\textbf{Decoder}\quad
The decoder is a upsampling CNN that reconstructs $\hat{x_t} = D(z_t, h_t)$, where $z_t$ can be interpreted as the posterior of timestep $t$, and $h_t$ is the output of the temporal transformer which summarizes information from previous timesteps. $z_t$ and $h_t$ are concatenated channel-wise before being fed into the decoder. Together with the encoder, the decoder optimizes the following cross entropy reconstruction loss

\begingroup\removeParBefore\endlinechar=32 \eqinner{
    \mathcal{L}_{\mathrm{recon}} &= -\textstyle\frac{1}{T}\sum_{t=1}^T \log p(x_t \mid z_t, h_t).
}

which encourages $z_t$ features to encode relative information between frames since the temporal transformer can aggregate information over time.

\textbf{Dynamics Prior}\quad
Lastly, we use a MaskGit~\citep{chang2022maskgit} to model the dynamics prior, $p(z_t \mid h_t)$. In our experiments, we show that using a MaskGit prior allows for not just faster but also higher quality sampling compared to an autoregressive prior. During every training iteration, we use the same process as prior work to sample a random mask $m_t$ and optimize

\begingroup\removeParBefore\endlinechar=32 \eqinner{
    \mathcal{L}_{\mathrm{prior}} = -\textstyle\frac{1}{T}\sum_{t=1}^T \log\p(z_t \mid z_t \odot m_t).
}

where $h_t$ is concatenated channel-wise with masked $z_t$ to predict the masked tokens. During generation, we follow \citet{lee2022draft}, where we initially generate each frame in chunks of 8 at a time and then go through 2 revise rounds of re-generating half the tokens each time.

\textbf{Training Objective}\quad
The final objective is the sum of these losses:
\begin{align}
    \mathcal{L}_{\mathrm{TECO}} &= \mathcal{L}_{\mathrm{VQ}} + \mathcal{L}_{\mathrm{recon}} + \mathcal{L}_{\mathrm{prior}}
\end{align}


\subsection{DropLoss} 

To train the model efficiently on long videos, we propose DropLoss, a simple trick to allow for more scalable and efficient training. Due to its architecture design, TECO can be separated into two components: (1) learning temporal representations, consisting of the encoder and the temporal transformer, and (2) predicting future frames, consisting of the dynamics prior and decoder. We can increase training efficiency by dropping out random timesteps that are not decoded and thus omitted from the reconstruction loss. For example, given a video of T frames, we compute $h_t$ for all $t \in \{1,\dots, T\}$, and then compute the losses $\mathcal{L}_{\mathrm{prior}}$ and $\mathcal{L}_{\mathrm{recon}}$ for only 10\% of the indices. Because random indices are selected each iteration, the model still needs to learn to accurately predict all timesteps. This reduces training costs significantly because the decoder and dynamics prior require non-trivial computations. DropLoss is applicable to both a wide class of architectures and to tasks beyond video prediction.

\section{Related Work}
\label{section:related_work}
\textbf{Video Generation}\quad
Prior video generation methods can be divided into a few classes of models: variational models, exact likelihood models, and GANs. SV2P~\citep{babaeizadeh2017stochastic}, SVP~\citep{denton2018stochastic}, SVG~\citep{villegas2019high}, and FitVid~\cite{babaeizadeh2021fitvid} are variational video generation methods models videos through stochastic latent dynamics, optimized using the ELBO~\citep{kingma2013auto} objective extended in time. SAVP~\citep{lee2018stochastic} adds an adversarial~\citep{goodfellow2014generative} loss to encourage more realistic and high-fidelity generation quality. Diffusion models~\citep{ho2020denoising,sohldickstein2014thermodynamics} have recently emerged as a powerful class of variational generative models which learn to iteratively denoise an initial noise sample to generate high-quality images. There have been several recent works that extend diffusion models to video, through temporal attention~\citep{ho2022video,harvey2022flexible}, 3D convolutions~\citep{hoppe2022diffusion}, or channel stacking~\citep{voleti2022masked}. Unlike variational models, autoregressive models (AR) and flows~\citep{kumar2019videoflow} model videos by optimizing exact likelihood. Video Pixel Networks~\citep{kalchbrenner2017video} and Subscale Video Transformers~\citep{weissenborn2019scaling} autoregressively model each pixel. For more compute efficient training, some prior methods~\citep{yan2021videogpt,le2021ccvs,seo2022harp,rakhimov2020latent,walker2021predicting} propose to learn an AR model in a spatio-temporally compressed latent space of a discrete autoencoder, which has shown to be orders of magnitudes more efficient compared to pixel-based methods. Instead of a VQ-GAN, \cite{le2021ccvs}, learns a frame conditional autoencoder through a flow mechanism. Lastly, GANs~\citep{goodfellow2014generative} offer an alternative method to training video models. MoCoGAN~\citep{tulyakov2018mocogan} generates videos by disentangling style and motion. MoCoGAN-HD~\citep{tian2021good} can efficiently extend to larger resolutions by learning to navigate the latent space of a pretrained image generator. TGANv2~\citep{saito2018tganv2}, DVD-GAN~\citep{clark2019adversarial}, StyleGAN-V~\citep{skorokhodov2021stylegan}, and TrIVD-GAN~\citep{luc2020transformation} introduce various methods to scale to complex video, such as proposing sparse training, or more efficient discriminator design. 

The main focus of this work lies with video prediction, a specific interpretation of conditional video generation. Most prior methods are trained autoregressive in time, so they can be easily extended to video prediction. Video Diffusion, although trained unconditionally proposes reconstruction guidance for prediction. GANs generally require training a separate model for video prediction. However, some methods such as MoCoGAN-HD  and DI-GAN can approximate frame conditioning by inverting the generator to compute a corresponding latent for a frame. 

\textbf{Long-Horizon Video Generation}\quad
CW-VAE~\citep{saxena2021clockwork} learns a hierarchy of stochastic latents to better model long term temporal dynamics, and is able to generate videos with long-term consistency for hundreds of frames. TATS~\citep{ge2022long} extends VideoGPT which allows for sampling of arbitrarily long videos using a sliding window. In addition, TATs and CogVideo~\citep{hong2022cogvideo} propose strided sampling as a simple method to incorporate longer horizon contexts. StyleGAN-V~\citep{skorokhodov2021stylegan} and DI-GAN~\citep{yu2022generating} learn continuous-time representations for videos which allow for sampling of arbitrary long videos as well. \cite{brooks2022generating} proposes an efficient video GAN architecture that is able to generate high resolution videos of 128 frames on complex video data for dynamic scenes and horseback riding. FDM~\citep{harvey2022flexible} proposes a diffusion model that is trained to be able to flexibly condition on a wide range of sampled frames to better incorporate context of arbitrarily long videos. \cite{lee2021revisiting} is able to leverage a hierarchical prediction framework using semantic segmentations to generate long videos.
\section{Reproducibility}
We provide several resources in order to aim for better reproducilbity. We include anonymized code in the supplementary materials for our models, baselines, and datasets. In addition, \Cref{section:hyperparams} details hyperparameters and compute requirements for all models.
