\section{Introduction}
Unsupervised learning of compositional representation is a core ability of human intelligence~\citep{analysisbysynthesis,frankland2020concepts}. Observing a visual scene, we perceive it not simply as a monolithic entity but as a geometric  composition of key components such as objects, borders, and space~\citep{kulkarni2015deep,analysisbysynthesis,epstein2017cognitive,whatiscognitivemap}. Furthermore, this understanding of the structure of the scene composition enables the ability for \textit{zero-shot imagination}, i.e., composing a novel, counterfactual, or systematically manipulated scenes that are significantly different from the training distribution.
As such, realizing this ability has been considered a core challenge in building a human-like AI system~\citep{lake2017building}.

DALL$\cdot$E\xspace~\citep{dalle} has recently shown an impressive ability to systematically generalize for zero-shot image generation. Trained with a dataset of text-image pairs, it can generate plausible images even from an unfamiliar text prompt such as ``avocado chair'' or ``lettuce hedgehog'', a form of systematic generalization~\citep{lake2018generalization,bahdanau2018systematic} in the text-to-image domain. However, from the perspective of compositionality, this success 
is somewhat expected 
because the text prompt already brings the composable structure. That is, the text is already discretized into a sequence of concept modules, i.e., words, which are composable, reusable, and encapsulated.
DALL$\cdot$E\xspace then learns to produce an image with global consistency by smoothly stitching over the discrete concepts via an Image GPT~\citep{igpt} decoder. 

Building from the success of DALL$\cdot$E\xspace, an important question is if we can achieve such systematic generalization for zero-shot image generation  only from images without text. This would require realizing an ability lacking in DALL$\cdot$E\xspace: extracting a set of composable representations from an image to play a similar role as that of word tokens. 
The most relevant  direction towards this is object-centric representation learning~\citep{iodine,slotattention,space,onbinding}.
While these can obtain a set of slots from an input image and reconstruct the same image from the slots, we argue that its systematic generalization ability to handle arbitrary \textit{reconfiguration} of slots is significantly limited due to the way it decodes the slots. 

In this paper, we propose a model that can achieve composition-based systematic generalization for image generation like DALL$\cdot$E\xspace but by learning compositional representations from images instead of providing a text prompt. We call this model, Illiterate-DALL$\cdot$E\xspace. To do this, we first introduce the slot-decoding dilemma and the pixel independence problem observed in the pixel-mixture decoder of traditional object-centric representation models. Then, we hypothesize that achieving compositional systematic generalization in image generation requires not only compositional slots but also a powerful expressive decoder that can model complex interactions among the slots and other pixels. To this end, we propose the \textit{slot-to-sequence} (Slot2Seq) decoder, which is an Image GPT decoder that takes the extracted slots as input instead of text. In experiments, we show that this simple architecture achieves the desired properties.

The main contribution of the paper is to show for the first time that the compositional systematic generalization, shown in DALL$\cdot$E\xspace, can be achieved without text. We also show that making a powerful and expressive decoder work with slot representations is crucial in achieving this. In this regard, we introduce a new architecture, Illiterate-DALL$\cdot$E\xspace, and a new slot decoding framework, Slot2Seq. The paper also introduces a new dataset, CLEVR-Mirror, and other tasks useful to evaluate the systematic generalization ability in image generation. Lastly, the experiment results strongly support the benefits of our model, and suggest that future object-centric models may benefit from building on the advantages of the proposed framework alongside the pixel-mixture decoder.



\section{Preliminaries}
\begin{figure}[t]
    \centering
    \hspace{-4mm}
    \includegraphics[width=0.8\textwidth]{images/images_lowres/arch-comparisons-013.png}
    \caption{\textbf{Overview of our proposed model with respect to prior works.} \textbf{Left:} In DALL$\cdot$E\xspace, words in the input text act as the composable units for generating the desired novel image. The generated images have global consistency because each pixel depends non-linearly on all previous pixels and the input word embeddings. \textbf{Middle:} Unlike DALL$\cdot$E\xspace that requires text supervision, Slot Attention provides an auto-encoding framework in which object slots act as the composable units inferred purely from raw images. However, during rendering, object slots are composed via a simple weighted sum of pixels obtained without any dependency on other pixels and slots which harms image consistency and quality. 
    \textbf{Right:} Our model combines the best of both models. Like Slot Attention our model is free of text-based supervision and like DALL$\cdot$E\xspace, produces novel image compositions with global consistency using the proposed decoder \textit{Slot2Seq}.
   
    }
    \label{fig:abstract-arch}
\end{figure}


\subsection{Object-Centric Representation Learning with Pixel-Mixture Decoder}
\label{sec:bg:objectcentric}
A common framework for learning object-centric representations is via a form of auto-encoders \citep{slotattention, monet,space}. An encoder takes an input image to return a \textit{set} of object representation vectors or slot vectors $\smash{\{\bs_1,\dots,\bs_N\} = f_\phi(\bx)}$. A decoder then reconstructs the image by composing the decoded result of each slot $\smash{\hat{\bx}  = g_\ta(\bs_{1:N})}$. To encourage the emergence of object concepts in the slots, the decoder usually uses an architecture implementing an inductive bias about the scene composition such as the pixel-mixture decoders.

\textbf{Pixel-mixture decoders} are the most common approach to construct an image from slots. This method assumes that the final image is constructed by the pixel-wise weighted mean of the slot images. Specifically, each slot $n$ generates its own slot image $\bmu_n$ and its corresponding alpha-mask $\bpi_n$. Then, the slot images are weighted-summed with the alpha-mask weights as follows:
\begin{align}
    \hat{\bx}_i = \sum_{n=1}^N \pi_{n,i} \cdot \boldsymbol{\mu}_{n,i} 
    &&\text{where}&& 
    \boldsymbol{\mu}_n = g^\text{RGB}_\ta(\bs_n)  
    &&\text{and}&& 
    \boldsymbol{\pi}_n = \frac{\exp g^\text{mask}_\ta(\bs_n)}{\sum_{m=1}^N \exp g^\text{mask}_\ta(\bs_m)} \ , \nn
\end{align}
where $i \in [1, HW]$ is the pixel index with $H$ and $W$ the image height and width, respectively. 

\subsection{Limitations of Pixel-Mixture Decoders}
\label{sec:bg:limitations}
The pixel-mixture decoder has two main limitations. The first is what we term as the \textbf{slot-decoding dilemma}. As there is no explicit incentive for the encoder to make a slot focus on the pixel area corresponding to an object, it is usually required to employ a weak decoder for $g_\ta^\text{RGB}$ such as the Spatial Broadcast Network~\citep{watters2019spatial} to make object concepts emerge in the slots. By limiting the capacity of the slot-decoder, it prevents a slot from modeling multiple objects or the entire image and instead incentivizes the slot to focus on a local area that shows a statistical regularity such as an object of a simple color. However, this comes with a side effect: the weak decoder can make the generated image blur the details. When we use a more expressive decoder such as a CNN to capture such details, the object disentanglement tends to fail with encoder producing slots that capture the whole image, hence a dilemma (See Appendix \ref{ax:sbd-discussion}). 

The second problem is the \textbf{pixel independence}. In pixel-mixture decoders, each slot's contribution to a generated pixel, i.e., $\pi_{n,i}$ and $\bmu_{n,i}$, is independent of the other slots and pixels. This may not be an issue if the aim of the slots is only to reconstruct the input image. However, it becomes an issue when we consider that an important desired property for object representations is to use them as \textit{concept modules} i.e., to be able to \textit{reuse} them in an arbitrary configuration in the same way as word embeddings can be reused to compose an arbitrary sentence.
The lack of flexibility of the decoder due to the pixel independence, however, prevents the slots 
from having such reconfigurability. For example, when decoding an arbitrary set of slots, the rendered image would look like a mere superposition of individual object patches without \textit{global semantic consistency}, producing a Frankenstein-like image as the examples in Section \ref{sec:exp} shall show. In contrast, this zero-shot global semantic consistency for concept reconfiguration is achieved in DALL$\cdot$E\xspace by using the Image GPT decoder. 


\subsection{Image GPT and DALL$\cdot$E\xspace}

\textbf{Image GPT} \citep{chen2020generative} is an autoregressive generative model for images implemented using Transformer \citep{transformer, gpt3}. For computational efficiency, Image GPT first down-scales the image of size $H\times W$ by a factor of $K$ using a VQ-VAE encoder \citep{vqvae}. 
This turns an image $\bx$ into a sequence of discrete image tokens  $\{{\bz}_i\}$ where $i$ indexes the tokens in a raster-scan order from 1 to $HW/K^2$. The transformer is then trained to model an auto-regressive distribution over this token sequence by maximizing the log-likelihood $\sum_i \log p_\ta({\bz}_i| {\bz}_{<i})$. During generation, the transformer samples the image tokens sequentially conditioning on the previously generated tokens, i.e., $\hat{\bz}_i \sim p_\ta(\hat{\bz}_i| \hat{\bz}_{<i})$ where $\hat{\bz}_i$ is the image token generated for the position $i$ in the sequence. Lastly, a VQ-VAE decoder takes all the generated tokens and produces the final image $\hat{\bx}$. Crucially, due to the autoregressive generation of $\bz_i$, unlike the pixel-mixture decoder, the generated pixels are not independent. The autoregressive generation powered by the direct memory access of the transformer makes Image GPT one of the most powerful decoder models in image generation and completion tasks. However, Image GPT does not provide a way to learn high-level semantic representations such as object-centric representations that can provide a way to semantically control image generation.

\textbf{DALL$\cdot$E\xspace} \citep{dalle} shows that the image generation of Image GPT can be controlled by conditioning on a text prompt. In particular, it shows the impressive ability of zero-shot generation \textit{with global semantic consistency} for rather an arbitrary reconfiguration of the words in the text prompt. DALL$\cdot$E\xspace can be seen as a conditional Image GPT model, $\hat{\bz}_i \sim p_\ta(\hat{\bz}_i| \hat{\bz}_{<i}, \bc_{1:N})$, where $\bc_{1:N}$ are the text prompt tokens. The key limitation from the perspective of this work is that it requires supervision in terms of text-image pairs. This means that it does not need to learn the compositional structure by itself since the discrete concepts are inherently provided by the prompt text even though it learns the word embeddings provided the structure. Therefore, it is important to investigate if we can make a text-free DALL$\cdot$E\xspace by learning to infer such compositional structure and representation only from images.


\section{Illiterate DALL$\cdot$E\xspace with Slot-To-Sequence Decoder}
\label{sec:method}

In this section, we propose a method that can provide global semantic consistency in the zero-shot image generation setting without text by learning object slots from images. To do this, we shall lift the use of inductive biases in the slot decoder about object composition by making use of a transformer \citep{transformer} as our image decoder while replacing the text prompt with a concept library constructed from image slots. Our central hypothesis is that a powerful auto-regressive decoder such as Transformer after training with a sufficiently diverse set of raw images should learn the rules of composition without domain specifications. 

\textbf{Obtaining Image Tokens using DVAE.} To make the training of transformer computationally feasible for high-resolution images, we first downscale the input image $\bx$ of size $H\times W$ by a factor of $K$ using Discrete VAE \citep{dvae}. To do this, we split the image into $K\times K$-sized patches resulting in $T$ patches where $T=HW/K^2$. We provide each patch $\bx_i$ as input to an encoder network $f_\phi$ to return log probabilities (denoted as $\bo_i$) for a categorical distribution with $V$ classes. With these log probabilities, we use a relaxed categorical distribution \citep{jang2016categorical} with a temperature $\tau$ to sample a relaxed one-hot vector $\bz_i^{\text{soft}}$. We then decode these soft vectors to obtain patch reconstructions. This process is summarized as follows:
\begin{align*}
    \bo_i = f_\phi(\bx_i)
    &&
    \Longrightarrow
    &&
    \bz_i^{\text{soft}} &\sim \text{RelaxedCategorical}(\bo_i; \tau)
    &&
    \Longrightarrow
    &&
    \tilde{\bx}_i = g_\ta(\bz_i^{\text{soft}})\ .
\end{align*}
By minimizing an MSE reconstruction objective for the image patches i.e. $\cL_\text{DVAE} = \sum_{i=1}^T (\tilde{\bx}_i - \bx_i)^2$, we train the DVAE encoder $f_\phi(\cdot)$ and the decoder $g_\ta(\cdot)$ networks.

\textbf{Inferring Object Slots.} 
To infer the object slots from a given image $\bx$, we first use the DVAE encoder as described above to obtain a discrete token $\bz_i$ for each patch $i$. Next, we map each token $\bz_i$ to an embedding by using a learned dictionary. To incorporate the position information into these patch embeddings, we sum them with learned positional embeddings. This results in an embedding $\bu_i$ which now has both the content and the position information for the patch. These embeddings $\bu_{1:T}$ are then given as input to a Slot-Attention encoder \citep{slotattention} with $N$ slots. 
\begin{align*}
    \bo_i &= f_\phi(\bx_i)
    &\Longrightarrow&&
    \bz_i &\sim \text{Categorical}(\bo_i) &\Longrightarrow \\
    \bu_i &= \text{Dictionary}_\phi(\bz_i) + \bp_{\phi,i}
    &\Longrightarrow&&
    \bs_{1:N}, A_{1:N} &= \text{SlotAttention}_\phi(\bu_{1:T})\ .
\end{align*}
This results in $N$ object slots $\bs_{1:N}$ and $N$ attention maps $A_{1:N}$ from the last refinement iteration of the Slot-Attention encoder. 

\textbf{Reconstruction using Transformer.} To reconstruct the input image, we decode the slots $\bs_{1:N}$ and first reconstruct the DVAE tokens $\hat{\bz}_{1:T}$ using a Transformer. We then use the DVAE decoder $g_\ta$ to decode the DVAE tokens and reconstruct the image patches $\hat{\bx}_i$ and therefore the image $\hat{\bx}$.
\begin{align*}
   \hat{\bo}_i = \text{Transformer}_\ta(\hat{\bu}_{<i} ; \bs_{1:N})
   &&\Longrightarrow&&
   \hat{\bz}_i = \arg \max_{v\in[1, V]} \hat{o}_{i,v}
   &&\Longrightarrow&&
   \hat{\bx}_i = g_\ta({\hat{\bz}_i})\ .
\end{align*}
where $\hat{\bu}_{i} = \text{Dictionary}_\phi(\hat{\bz}_i) + \bp_{\phi,i}$ and $\hat{o}_{i,v}$ is the log probability of token $v$ among the $V$ classes of the categorical distribution represented by $\hat{\bo}_i$. To train the transformer, the slot attention encoder and the embeddings, we minimize the cross-entropy of predicting each token $\bz_i$ conditioned on the preceding tokens $\bz_{<i}$ and the slots $\bs_{1:N}$. Let the predicted log-probabilities for the token at position $i$ be $\bar{\bo}_i &= \text{Transformer}_\ta({\bu}_{<i} ; \bs_{1:N})$. Then the cross-entropy training objective can be written as $\smash{\cL_\text{ST} = \sum_{i=1}^T \text{CrossEntropy}(\bz_i,\bar{\bo}_i)}$.

\textbf{Learning Objective and Training.} The complete training objective is given by $\cL = \cL_\text{ST} + \cL_\text{DVAE}$ and all modules of our model are trained jointly. We apply a decay on the DVAE temperature $\tau$ from 1.0 to 0.1 at the start of the training. We also apply a learning rate warm-up for the parameters of Slot-Attention and Transformer at the start of the training as we found that this led to stable training.


\textbf{Multi-headed Slot Attention.}
For representing images, we found that slots of standard Slot Attention encoder \citep{slotattention} can suffer in expressiveness when representing objects with complex shapes and textures. This is because the slots collect information from the input cells via dot-product attention in which the attended input values are pooled using a simple weighted sum. As such, this pooling method can be too weak for representing complex objects. To address this, we propose an extension of Slot Attention called \textit{Multi-headed Slot Attention} in which each slot attends to the input cells via multiple heads. Thus each slot can attend to different parts of the same object. When the slot state is computed, the attended values of different heads are concatenated which allows these partial object features to interact more flexibly and produce a significantly better object representation. Full details of the implementation and an ablation experiment to show the benefits of multiple heads are provided in Appendix \ref{ax:mhsa}.

\subsection{Visual Concept Library}
\label{sec:concept-lib}
The above model extracts a set of concepts or slots belonging to a particular image whereas an intelligent agent would build a \textit{library of reusable concepts} from diverse experience. In DALL$\cdot$E\xspace, a vocabulary of word embeddings plays the role of this library and provides reusable concepts. Similarly, to build a library of visual concepts from the dataset, we use the following simple approach based on $K$-means clustering to construct a library of reusable visual concepts. To do this, (i) collect $N$ slots $\cS_i = \{\bs^i_1,\dots,\bs^i_N\}$ and their attention maps $\cA_i = \{A^i_1,\dots,A^i_N\}$ from each image $i$ in the training dataset. (ii) Run $K$-means clustering on $\smash{\cS = \bigcup_i \cS_i}$ with cosine similarity between slots as a distance metric where $K$ is the number of total concepts in the library. If the object position is more important for building the concept library, cluster the slots using IOU between the attention maps as a similarity metric. (iii) Use each cluster as a concept and the slots assigned to the cluster as the instantiations of the concept. (iv) To compose an arbitrary image, choose the concepts from the library and randomly choose a slot for each concept and build a \textit{slot prompt} similar to a text prompt in DALL$\cdot$E\xspace. (v) Provide this prompt to the decoder to compose and generate the image. 



\section{Related Work}
\textbf{Compositional Generation via Object-Centric Learning.} Self-supervised object-centric approaches \citep{monet, iodine, slotattention, kabra2021simone, greff2017neural, engelcke2020genesis, engelcke2021genesis, air, spair, space, gnm, sqair, scalor, silot, gswm, deng2021generative, anciukevicius2020object, kugelgen2020towards, ocvt} typically perform mixture-based alpha compositing for rendering with contents of each slot decoded independently. 
\cite{savarese2021information} and \cite{yang2020learning} rely on minimizing mutual information between predicted object segments as a learning signal while \cite{yang2021self} leverage optical flow. However, these approaches either cannot compose novel scenes or require specialized losses for object discovery unlike ours which uses a simple reconstruction loss. DINO \citep{caron2021emerging} combining Vision Transformer \citep{dosovitskiy2021an} and a self-supervised representation learning objective \citep{chen2020a, he2020momentum, grill2020bootstrap, touvron2021training, hinton2021how, oord2018representation} discovers object-centric attention maps but, unlike ours, cannot compose novel scenes. TIMs \citep{lamb2021transformers, goyal2021recurrent} with an Image GPT decoder show object-centric attention maps but, unlike ours, do not investigate compositional generation.

\textbf{Compositional Generation using GANs.}
\cite{chai2021using} show compositional generation using a collage of patches encoded and decoded using a pretrained GAN. However, the collage needs to be provided manually and object discovery requires a specialized process unlike our model.  
\cite{bielski2019emergence} and \cite{chen2019unsupervised} show object discovery from given images by randomly re-drawing or adding a random jitter \citep{voynov2021big} to the foreground followed by an adversarial loss. As these models use alpha compositing, their compositional ability is limited. Some works \citep{donahue2016adversarial, donahue2019large} infer abstract representations for images that emphasize the high-level semantics, but these, unlike ours, cannot be used for compositional image generation. 
\cite{niemeyer2021giraffe}, \cite{blockgan, chen2016infogan}, \cite{steenkiste2020investigating}, \cite{liao2020towards} and \cite{ehrhardt2020relate} introduce GANs which generate images conditioned on object-wise or factor-wise noise and optionally on camera pose. Lacking an inference module, these cannot perform compositional editing of a given image unlike ours. While \cite{kwak2016generating} provide an encoder to discover objects, the decoding relies on alpha compositing. \cite{reed2016learning} show compositional generation in GANs and improved generation quality \citep{johnson2018image, hinz2019generating} via an object-centric or a key-point based pathway for scene rendering. However, these require supervision for the bounding boxes and the keypoints.

\textbf{Compositional Generation with Latent Variable Models and Text.} 
Early approaches focused on disentangling the independent factors of the observation generating process using $\beta$-VAE \citep{higgins2017beta} or ensuring that independently sampling each latent factor should produce images indistinguishable from the training distribution \citep{kim2018disentangling, kumar2017variational, chen2018isolating}. However unlike ours, these approaches providing a monolithic single vector representation can suffer in multi-object scenes due to \textit{superposition catastrophe} \citep{binding,gnm}. Another line of approaches learn to map text to images to compose novel scenes \citep{dalle, reed2016generative, li2019object, higgins2018scan} or map attribute-value pairs to images \citep{sohn2015learning, yan2016attribute2image}. However such approaches rely on supervision.







\section{Experiments}
\label{sec:exp}


In experiments, we evaluate the benefits of Slot2Seq over pixel-mixture decoder in: 1) generating novel images from an arbitrary configuration of slots, 2) image reconstruction, and 3) generating novel images not belonging to the training image distribution. As the baseline, we compare with an unsupervised object-centric framework having a slot attention encoder like ours but using a mixture decoder for decoding instead of Slot2Seq. Hereafter, we refer to this model as simply Slot Attention \citep{slotattention}. We evaluate the models on four datasets: 3D Shapes \citep{3dshapes18}, CLEVR-Mirror which we develop from the CLEVR~\citep{clevr} dataset by adding a mirror in the scene, Shapestacks \citep{groth2018shapestacks} and Bitmoji \citep{bitmoji}. For brevity, we shall refer to CLEVR-Mirror as simply CLEVR. In each dataset, a scene can be described in terms of composable entities such as blocks, walls, floors, hair, or face. 
Our models take only raw images without any other supervision or annotations.
 

\subsection{Compositional Image Generation} 
\label{sec:exp:compgen}
\begin{table}[t]
\begin{subtable}{.42\linewidth}\centering
{\begin{tabular}{@{}lcc@{}}
\toprule
            & \multicolumn{2}{c}{FID $(\downarrow)$}                                     \\ \cmidrule(l){2-3}
Dataset     & SA & Ours \\ \midrule
3D Shapes   &  44.14      &  \textbf{36.75}    \\
CLEVR       &  63.76        & \textbf{41.62}    \\
Shapestacks &  155.74        &   \textbf{51.27}    \\
Bitmoji     &  71.43       &  \textbf{15.83}    \\ \bottomrule
\end{tabular}}
\caption{Compositional Generation}\label{tab:1a}
\end{subtable}%
\begin{subtable}{.58\linewidth}\centering
{\begin{tabular}{@{}lcccc@{}}
\toprule
           
            & \multicolumn{2}{c}{MSE $(\downarrow)$} & \multicolumn{2}{c}{FID $(\downarrow)$} \\ \cmidrule(l){2-5} 
Dataset     & SA               & Ours            & SA         & Ours        \\ \midrule
3D Shapes   & 16.65             & \textbf{10.49} & 44.71                & \textbf{38.20}      \\
CLEVR       & \textbf{29.59} &  49.95     & 55.95          &  \textbf{37.42}          \\
Shapestacks & 233.72        & \textbf{111.86}    & 139.72                     &   \textbf{30.22}  \\
Bitmoji     & 388.72        & \textbf{261.10}    & 67.66              &   \textbf{11.89}       \\ \bottomrule
\end{tabular}}
\caption{Image Reconstruction}\label{tab:1b}
\end{subtable}
\caption{Comparison of Compositional Generation (left) and Image Reconstruction (right) between Slot-Attention (SA) and our model. For comparison of compositional generation, we report the FID score. For image reconstruction quality, we report the MSE and the FID score.}\label{tab:1}
\end{table}
\begin{figure}[t]
    \centering
    \includegraphics[valign=t,width=1.0\textwidth]{images/plot-discriminator-007.pdf}   
    \caption{Comparison of training curves of a discriminator to compare the quality of compositional generation between Slot-Attention and our model. A CNN discriminator tries to discriminate between real images and model-generated images. We note that the curves converge more slowly for our model than for Slot Attention and show that our generations are more realistic.}
    \label{fig:discriminator-training-curves}
\end{figure}





\begin{figure}[t]
    \centering
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
    \includegraphics[width=1.0\textwidth]{images/compositional_gen_3dshapes_bitmoji_joint.png}
    \caption{\textbf{
   
    Comparison of compositional generation between Slot Attention and our model in 3D Shapes and Bitmoji datasets.
    } We visualize the slot prompts provided to Slot2Seq and the resulting compositions. To visualize the prompt slots, we show the image from which the slot was taken masked using its input attention map. Note that these attention maps had emerged in the source images implicitly without any supervision. We note that our decoder performs zero-shot generalization to novel slot inputs while the mixture decoder fails to compose arbitrary slots well.  
    }
    \label{fig:qual-compgen-bitmoji-3shapes-001}
\end{figure}
\begin{figure}[t]
    \centering
    \includegraphics[valign=t,width=1.0\textwidth]{images/bitmoji-traversal-diagram-hair-and-face-3.png}   
    \caption{\textbf{Object Replacement and Out-of-Distribution Compositions in Bitmoji.} We show that in our model, it is possible to edit the image by taking a specific slot and replacing it with an arbitrary slot drawn from the concept library for the same concept. We also show OOD compositions by replacing the slot with those from the opposite gender.}
    \label{fig:replacements_bitmoji}
\end{figure}

We evaluate how effectively the known slots can be reused to generate novel scenes. For this, we evaluate the quality of image generations produced by our model from arbitrarily composed slot prompts. To build these prompts, we first build a concept library using $K$-means as described in Section \ref{sec:concept-lib}. In 3D Shapes and Bitmoji, we apply $K$-means clustering on the slots. In 3D Shapes, this results in clusters for \textit{wall}, \textit{object} and \textit{floor} and in Bitmoji for \textit{dress}, \textit{hair}, \textit{neck}, \textit{eyes}, \textit{face} and \textit{collar}. These clusters make up our concept library and are visualized in Appendix \ref{ax:additional_qual}. To build novel prompts from this library, we simply pick one slot from each cluster. In CLEVR and Shapestacks datasets, to build a prompt, it is more natural to pick the object positions first and then sample a slot from the library for that position on the canvas. Hence, for CLEVR, we pick a set of random positions keeping a certain distance between the picked positions. For Shapestacks, we pick a set of random positions in a vertical tower-like configuration. 
In this way, we generate $40 000$ prompts for each dataset and render the corresponding images. We then evaluate the quality of the images by computing the FID score with respect to the true images. To support this metric, we also report the training curves of a CNN discriminator that tries to classify between real and model-generated images. If generated images are close to the true data distribution, then it should be harder for the discriminator to discriminate and the resulting training curve should converge more slowly. 

From Table \ref{tab:1a} and Figure \ref{fig:discriminator-training-curves}, we note that our compositional generations are significantly more realistic than the generations from the mixture decoder. The qualitative samples shown in Figures \ref{fig:qual-compgen-bitmoji-3shapes-001}, \ref{fig:replacements_bitmoji}, \ref{fig:additions_clevr} and \ref{fig:qual-compgen-shapestacks-001} show that our decoder can compose the desired image accurately and make an effective reuse of the known slots. Additional qualitative results are provided in Appendix \ref{ax:additional_qual}. We also note that for visually complex Shapestacks and Bitmoji datasets, the improvement in the FID score provided by our model is much larger than that for the visually simpler CLEVR and 3D Shapes datasets. 
 
These results provide evidence that \textit{i)} our model resolves the \textit{slot-decoding dilemma} and consequently, it benefits from the flexibility of the Image GPT decoder that can render the fine details such as floor texture in Shapestacks and complex facial features in Bitmoji while the mixture decoder suffers. Notice that the attention segments of the visualized slots effectively focus on the specific object regions of the respective images that were provided during concept library creation. This emergent object localization occurs despite the powerful decoder, thus resolving the dilemma. We visualize the attention maps of the slots for a given input image in Figures \ref{fig:qual-recon-shapestacks-ours-000} -- \ref{fig:qual-recon-3dshapes-ours-000} in Appendix. Because our slots attend on the DVAE tokens, hence our attention is patch-level and our attention segments may appear coarser than those of Slot Attention in which slots attend on the CNN feature cells having the same resolution as the input image. 
\textit{ii)} Because the mixture decoder has the \textit{pixel-independence assumption}, we see in Figure \ref{fig:qual-compgen-bitmoji-3shapes-001} that it suffers when slots are combined in arbitrary configurations. In 3D Shapes, the object shadows are controlled by the floor component of the mixture and are either too large or too small for the 3D object. This is because after arbitrarily choosing a different 3D object, the floor slot has no knowledge that the object which casts the shadow has changed and it continues to draw the shadow mask of the source image which has now become obsolete in the new slot configuration. 
Slot Attention suffers even more severely in Bitmoji as the composed mixture components and masks are mutually incompatible in the new slot configuration and produce an inaccurate alpha composition. In contrast, Slot2Seq is robust to this problem because the slots and pixels can interact. For instance, in 3D Shapes, as the pixels of the floor are being rendered, the predictor is aware of the object that will cast the shadow via the previous pixels and the input slots and can thus predict a globally consistent pixel value.



 















\begin{figure}[t]
    \centering
    \includegraphics[valign=t,width=1.0\textwidth]{images/clevr-qual-adder-compare-002.png}   
    \caption{\textbf{Compositional Generation in CLEVR.} We show compositional generation in CLEVR by adding object slots one at a time. In training, only 3-6 objects were shown. However, we note that our model can generalize in generating 1-2 and 7-8 objects including the empty scene. We also note that our mirror reflections are significantly more clear than those of Slot Attention.}
    \label{fig:additions_clevr}
\end{figure}
\begin{figure}[t]
    \centering
    \includegraphics[valign=t,width=1.0\textwidth]{images/compositional_gen_sa_ours_compare_001.png}   
    \caption{\textbf{Compositional generation in Shapestacks Dataset.} We show compositional generation in Shapestacks by composing arbitrary blocks drawn from the concept library in a tower configuration. We note that Slot2Seq can render the details such as the texture of the floor and the wall significantly better than the mixture decoder.}
    \label{fig:qual-compgen-shapestacks-001}
\end{figure}





\subsection{Reconstruction Quality}

A natural test about whether an auto-encoder learns accurate representations of the given input is through an evaluation of the reconstruction error. For this, we report two metrics: 1) MSE to evaluate how well the reconstructed image preserves the contents of the original image and 2) FID score to show how realistic is the reconstructed image with respect to the true image distribution. We compute these metrics on a held-out set and report in Table \ref{tab:1b}. We find that our model outperforms Slot Attention in all datasets in terms of FID which shows that the image quality of our model is better due to its powerful decoder. 
In MSE, our model outperforms the baseline in all datasets except for CLEVR. We note that the performance gain from our model, in both MSE and FID, increases as the visual complexity of the dataset increases from 3D Shapes and CLEVR to Shapestacks and Bitmoji. Despite the better MSE of the mixture decoder in CLEVR, the gap is not severe given the broader limitations of the mixture decoder in handling novel slot configurations. We also believe that the MSE of our model is affected by the DVAE patch size which can affect the rendering of small-sized objects of CLEVR. To test this hypothesis, we compared using a coarse 
patch grid and a fine 
patch grid and we found the latter to reduce the MSE gap with Slot Attention. Thus, in the future, with more compute that can enable a finer DVAE grid, this discrepancy may be resolved.
























\subsection{Generalization to Out-of-Distribution (OOD) Slot Reconfigurations}
Because the Slot2Seq decoder is based on Transformer and does not have an explicit inductive bias about scene composition, we test whether it can generalize in composing slots when their configurations are different from the training distribution.
For this, \textit{i)} in Shapestacks, we compose slot prompts to render two towers while only single-tower images were shown during training. \textit{ii)} In CLEVR, we compose prompts to render fewer or more objects than were shown during training. \textit{iii)} Lastly, in Bitmoji, we annotated the gender of 500 images and used it to construct 1600 OOD slot prompts by composing hair and face only from opposite genders. By doing this, we change the distribution of the slot configurations with respect to the training distribution. Note that such slot prompts were already a part of the Bitmoji compositions in Section \ref{sec:exp:compgen}. However, with this experiment, we evaluate the specific OOD subset of those prompts. We show the results in Figures \ref{fig:replacements_bitmoji}, \ref{fig:additions_clevr} and \ref{fig:qual-compgen-shapestacks-ood-001} and a comparison of FID score with respect to the mixture decoder in Table \ref{tab:ood-table}. 
From these, we note that our decoder is able to generalize to the OOD slot configurations and it also outperforms the mixture decoder in image quality.



\begin{figure}[t]
\begin{minipage}[t]{0.68\linewidth}
\vspace{0mm}
\centering
\includegraphics[width=1.0\linewidth]{images/ood-shapestacks-ours-001.png}
\captionof{figure}{\textbf{Out-of-Distribution Generalization in Shapestacks Dataset.} We provide slot prompts to render two towers instead of one. The models were trained only on images with one tower.}
\label{fig:qual-compgen-shapestacks-ood-001}
\end{minipage}
\hfill
\begin{minipage}[t]{0.30\linewidth}
\vspace{0mm}
\centering
\small
\begin{tabular}{@{}lcc@{}}
\toprule
            & \multicolumn{2}{c}{FID $(\downarrow)$}                                     \\ \cmidrule(l){2-3}
Dataset     & SA & Ours \\ \midrule
Shapestacks   &   172.82     &  \textbf{82.38}  \\
CLEVR  &   62.44     &   \textbf{43.77}  \\
Bitmoji & 81.66 & \textbf{22.75} \\
\bottomrule
\end{tabular}
\captionof{table}{FID Comparison for OOD Compositions.}
\label{tab:ood-table}
\end{minipage}
\end{figure}









\section{Discussion}

We presented Illiterate DALL$\cdot$E\xspace, a model for text-free zero-shot imagination by learning slot representations only from images. Our model combines the best of DALL$\cdot$E\xspace and object-centric representation learning via the proposed decoder \textit{Slot2Seq}. 
It 
provides a significant improvement over pixel-mixture decoders in reconfigurability of slots for zero-shot imagination. These results suggest that future object-centric models may benefit from building on the advantages of the proposed framework alongside the pixel-mixture decoder.

The proposed Slot2Seq, with its Transformer-based implementation, can also be seen as a graphics engine that learns to mimic the physical laws governing the rendering of the image through its exposure to a large dataset. Thus to render a scene, the only information that the Transformer requires and requests from the encoder is the abstract object-level \textit{what} and \textit{where} in the form of slots. Knowing these objects, the Transformer then fills in the rest and implicitly handles the mechanisms that give rise to the observed object appearance, occlusions, shadows, reflections, and transparency.

An interesting future direction is to consider that an intelligent agent may collect new experience indefinitely, and thus our approach of building the library of reusable concepts can be made more formal by learning discrete concepts and growing this library using online clustering~\citep{onlinekmeans} or Bayesian non-parametric clustering~\citep{dpmm}. Another direction is to learn a slot-level prior for fast density modeling and sampling scenes at the representation level. 


\section*{Ethics Statement}
For future applications, negative societal consequences should also be considered. The current model does not generate images realistic enough to deceive humans if used with malicious intent. However future versions of the model after scaling up its size and the training dataset can have such potential. Despite these considerations, these consequences are not imminent.
 









\section{Introduction}
Unsupervised learning of compositional representation is a core ability of human intelligence~\citep{analysisbysynthesis,frankland2020concepts}. Observing a visual scene, we perceive it not simply as a monolithic entity but as a geometric  composition of key components such as objects, borders, and space~\citep{kulkarni2015deep,analysisbysynthesis,epstein2017cognitive,whatiscognitivemap}. Furthermore, this understanding of the structure of the scene composition enables the ability for \textit{zero-shot imagination}, i.e., composing a novel, counterfactual, or systematically manipulated scenes that are significantly different from the training distribution.
As such, realizing this ability has been considered a core challenge in building a human-like AI system~\citep{lake2017building}.

DALL$\cdot$E\xspace~\citep{dalle} has recently shown an impressive ability to systematically generalize for zero-shot image generation. Trained with a dataset of text-image pairs, it can generate plausible images even from an unfamiliar text prompt such as ``avocado chair'' or ``lettuce hedgehog'', a form of systematic generalization in the text-to-image domain~\citep{lake2018generalization,bahdanau2018systematic}. However, from the perspective of compositionality, this success
is somewhat expected
because the text prompt already brings the composable structure. That is, the text is already discretized into a sequence of concept modules, i.e., words, which are composable, reusable, and encapsulated.
DALL$\cdot$E\xspace then learns to produce an image with global consistency by smoothly stitching over the discrete concepts via an Image GPT~\citep{igpt} decoder.

Building from the success of DALL$\cdot$E\xspace, an important question is if we can achieve such systematic generalization for zero-shot image generation  only from images without text. This would require realizing an ability lacking in DALL$\cdot$E\xspace: extracting a set of composable representations from an image to play a similar role as that of word tokens.
The most relevant  direction towards this is object-centric representation learning~\citep{iodine,slotattention,space,onbinding}.
While this approach can obtain a set of slots from an image by reconstructing the image from the slots, we argue that its ability to systematically generalize in such a way to handle arbitrary \textit{reconfiguration} of slots is significantly limited due to the way it decodes the slots.

In this work, we propose a slot-based autoencoder architecture, called SLot Attention TransformEr (SLATE). SLATE combines the best of DALL$\cdot$E\xspace and object-centric representation learning. That is, compared to the previous object-centric representation learning, our model significantly improves the ability of composition-based systematic generalization in image generation. However, unlike DALL$\cdot$E\xspace we achieve this by learning object-centric representations from images alone instead of taking a text prompt, resulting in a model called Illiterate DALL$\cdot$E\xspace (I-DALL$\cdot$E\xspace). To do this, we first analyze that the existing pixel-mixture decoders for learning object-centric representations suffer from the \textit{slot-decoding dilemma} and the \textit{pixel independence problem} and thus show limitations in achieving compositional systematic generalization. Inspired from DALL$\cdot$E\xspace, we then hypothesize that resolving these limitations requires not only learning composable slots but also a powerful and expressive decoder that can model complex interactions among the slots and the pixels. 
Based on this investigation, we propose the SLATE architecture, a simple but novel slot autoencoding architecture that uses a slot-conditioned Image GPT decoder. Also, we propose a method to build a library of visual concepts from the learned slots. Similar to the text prompt in DALL$\cdot$E\xspace, this allows us to program a form of `sentence' made up of visual concepts to compose an image. Furthermore, the proposed model is also simple, easy to implement, can be plugged in into many other models. In experiments, we show that this simple architecture achieves various benefits such as significantly improved generation quality in both the in-distribution and the out-of-distribution settings. This suggests that more attention and investigation on the proposed framework is required in future object-centric models.

The main contribution of the paper can be seen as three folds: (1) achieving the first text-free DALL$\cdot$E\xspace model, (2) the first object-centric representation learning model based on a transformer decoder, and (3) showing that it significantly improves the systematic generalization ability of object-centric representation models. (4) While achieving better performance, the proposed model is much simpler than previous approaches.





\section{Preliminaries}
\begin{figure}[t]
    \centering
    \hspace{-4mm}
    \includegraphics[width=1.0\textwidth]{images/images_lowres/arch-comparisons-015.png}
    \caption{\textbf{Overview of our proposed model with respect to prior works.} \textbf{Left:} In DALL$\cdot$E\xspace, words in the input text act as the composable units for generating the desired novel image. The generated images have global consistency because each pixel depends non-linearly on all previous pixels and the input word embeddings because of its transformer decoder. \textbf{Middle:} Unlike DALL$\cdot$E\xspace that requires text supervision, Slot Attention provides an auto-encoding framework in which object slots act as the composable units inferred purely from raw images. However, during rendering, object slots are composed via a simple weighted sum of pixels obtained without any dependency on other pixels and slots which harms image consistency and quality.
    \textbf{Right:} Our model, SLATE, combines the best of both models. Like Slot Attention our model is free of text-based supervision and like DALL$\cdot$E\xspace, produces novel image compositions with global consistency.
   
    }
    \label{fig:abstract-arch}
\end{figure}


\subsection{Object-Centric Representation Learning with Pixel-Mixture Decoder}
\label{sec:bg:objectcentric}
A common framework for learning object-centric representations is via a form of auto-encoders \citep{slotattention, monet,space}. In this framework, an encoder takes an input image to return a \textit{set} of object representation vectors or slot vectors $\smash{\{\bs_1,\dots,\bs_N\} = f_\phi(\bx)}$. A decoder then reconstructs the image by composing the decodings $g_\ta(\bs_n)$ of each slot, $\smash{\hat{\bx}  = f_\text{compose}(g_\ta(\bs_1), \dots, g_\ta(\bs_N))}$. To encourage the emergence of object concepts in the slots, the decoder usually uses an architecture implementing an inductive bias about the scene composition such as the pixel-mixture decoders.

\textbf{Pixel-mixture decoders} are the most common approach to construct an image from slots. This method assumes that the final image is constructed by the pixel-wise weighted mean of the slot images. Specifically, each slot $n$ generates its own slot image $\bmu_n$ and its corresponding alpha-mask $\bpi_n$. Then, the slot images are weighted-summed with the alpha-mask weights as follows:
\begin{align}
    \hat{\bx}_i = \sum_{n=1}^N \pi_{n,i} \cdot \boldsymbol{\mu}_{n,i}
    &&\text{where}&&
    \boldsymbol{\mu}_n = g^\text{RGB}_\ta(\bs_n)
    &&\text{and}&&
    \boldsymbol{\pi}_n = \frac{\exp g^\text{mask}_\ta(\bs_n)}{\sum_{m=1}^N \exp g^\text{mask}_\ta(\bs_m)} \ , \nn
\end{align}
where $i \in [1, HW]$ is the pixel index with $H$ and $W$ the image height and width, respectively.

\subsection{Limitations of Pixel-Mixture Decoders}
\label{sec:bg:limitations}
The pixel-mixture decoder has two main limitations. The first is what we term as the \textbf{slot-decoding dilemma}. As there is no explicit incentive for the encoder to make a slot focus on the pixel area corresponding to an object, it is usually required to employ a weak decoder for $g_\ta^\text{RGB}$ such as the Spatial Broadcast Network~\citep{watters2019spatial} to make object concepts emerge in the slots. By limiting the capacity of the slot-decoder, it prevents a slot from modeling multiple objects or the entire image and instead incentivizes the slot to focus on a local area that shows a statistical regularity such as an object of a simple color. However, this comes with a side effect: the weak decoder can make the generated image blur the details. When we use a more expressive decoder such as a CNN to prevent this, the object disentanglement tends to fail, e.g., by producing slots that capture the whole image, hence a dilemma (See Appendix \ref{ax:sbd-discussion}).

The second problem is the \textbf{pixel independence}. In pixel-mixture decoders, each slot's contribution to a generated pixel, i.e., $\pi_{n,i}$ and $\bmu_{n,i}$, is independent of the other slots and pixels. This may not be an issue if the aim of the slots is to reconstruct only the input image. However, it becomes an issue when we consider that an important desired property for object representations is to use them as \textit{concept modules} i.e., to be able to \textit{reuse} them in an arbitrary configuration in the same way as word embeddings can be reused to compose an arbitrary sentence.
The lack of flexibility of the decoder due to the pixel independence, however, prevents the slots
from having such reconfigurability. For example, when decoding an arbitrary set of slots taken from different images, the rendered image would look like a mere superposition of individual object patches without \textit{global semantic consistency}, producing a Frankenstein-like image as the examples in Section \ref{sec:exp} shall show. In contrast, this zero-shot global semantic consistency for concept reconfiguration is achieved in DALL$\cdot$E\xspace by using the Image GPT decoder.


\subsection{Image GPT and DALL$\cdot$E\xspace}

\textbf{Image GPT} \citep{chen2020generative} is an autoregressive generative model for images implemented using a transformer \citep{transformer, gpt3}. For computational efficiency, Image GPT first down-scales the image of size $H\times W$ by a factor of $K$ using a VQ-VAE encoder \citep{vqvae}.
This turns an image $\bx$ into a sequence of discrete image tokens  $\{{\bz}_i\}$ where $i$ indexes the tokens in a raster-scan order from 1 to $HW/K^2$. The transformer is then trained to model an auto-regressive distribution over this token sequence by maximizing the log-likelihood $\sum_i \log p_\ta({\bz}_i| {\bz}_{<i})$. During generation, the transformer samples the image tokens sequentially conditioning on the previously generated tokens, i.e., $\hat{\bz}_i \sim p_\ta(\hat{\bz}_i| \hat{\bz}_{<i})$ where $\hat{\bz}_i$ is the image token generated for the position $i$ in the sequence. Lastly, a VQ-VAE decoder takes all the generated tokens and produces the final image $\hat{\bx}$. Crucially, due to the autoregressive generation of $\bz_i$, unlike the pixel-mixture decoder, the generated pixels are not independent. The autoregressive generation powered by the direct context attention of the transformer makes Image GPT one of the most powerful decoder models in image generation and completion tasks. However, Image GPT does not provide a way to learn high-level semantic representations such as object-centric representations that can provide a way to semantically control image generation.

\textbf{DALL$\cdot$E\xspace} \citep{dalle} shows that the image generation of Image GPT can be controlled by conditioning on a text prompt. In particular, it shows the impressive ability of zero-shot generation \textit{with global semantic consistency} for rather an arbitrary reconfiguration of the words in the text prompt. DALL$\cdot$E\xspace can be seen as a conditional Image GPT model, $\hat{\bz}_i \sim p_\ta(\hat{\bz}_i| \hat{\bz}_{<i}, \bc_{1:N})$, where $\bc_{1:N}$ are the text prompt tokens. The key limitation from the perspective of this work is that it requires supervision in terms of text-image pairs. This means that it does not need to learn the compositional structure by itself since the discrete concepts are inherently provided by the prompt text even though it learns the word embeddings provided the structure. Therefore, it is important to investigate if we can make a text-free DALL$\cdot$E\xspace by learning to infer such compositional structure and representation only from images.


\section{SLATE: Slot Attention Transformer}
\label{sec:method}

In this section, we propose a method that can provide global semantic consistency in the zero-shot image generation setting without text by learning object slots from images alone. In doing this, we shall lift the use of inductive biases in the image decoder about slot composition by making use of a transformer \citep{transformer} as our image decoder while replacing the text prompt with a slot prompt drawn from a concept library constructed from a given set of images. Our central hypothesis is that a powerful auto-regressive decoder such as a transformer after training with a sufficiently diverse set of raw images should learn the slot representations and the rules of composition jointly in a way to achieve systematic generalization of zero-shot image generation. In the following, we describe the details of the model architecture.

\textbf{Obtaining Image Tokens using DVAE.} To make the training of transformer computationally feasible for high-resolution images, we first downscale the input image $\bx$ of size $H\times W$ by a factor of $K$ using Discrete VAE \citep{dvae}. To do this, we split the image into $K\times K$-sized patches resulting in $T$ patches where $T=HW/K^2$. We provide each patch $\bx_i$ as input to an encoder network $f_\phi$ to return log probabilities (denoted as $\bo_i$) for a categorical distribution with $V$ classes. With these log probabilities, we use a relaxed categorical distribution \citep{jang2016categorical} with a temperature $\tau$ to sample a relaxed one-hot encoding $\bz_i^{\text{soft}}$ for the patch $i$. We then decode these soft vectors to obtain patch reconstructions. This process is summarized as follows:
\begin{align*}
    \bo_i = f_\phi(\bx_i)
    &&
    \Longrightarrow
    &&
    \bz_i^{\text{soft}} &\sim \text{RelaxedCategorical}(\bo_i; \tau)
    &&
    \Longrightarrow
    &&
    \tilde{\bx}_i = g_\ta(\bz_i^{\text{soft}})\ .
\end{align*}
By minimizing an MSE reconstruction objective for the image patches i.e. $\cL_\text{DVAE} = \sum_{i=1}^T (\tilde{\bx}_i - \bx_i)^2$, we train the DVAE encoder $f_\phi(\cdot)$ and the decoder $g_\ta(\cdot)$ networks.

\textbf{Inferring Object Slots.}
To infer the object slots from a given image $\bx$, we first use the DVAE encoder as described above to obtain a discrete token $\bz_i$ for each patch $i$. Next, we map each token $\bz_i$ to an embedding by using a learned dictionary. To incorporate the position information into these patch embeddings, we sum them with learned positional embeddings. This results in an embedding $\bu_i$ which now has both the content and the position information for the patch. These embeddings $\bu_{1:T}$ are then given as input to a Slot-Attention encoder \citep{slotattention} with $N$ slots.
\begin{align*}
    \bo_i &= f_\phi(\bx_i)
    &\Longrightarrow&&
    \bz_i &\sim \text{Categorical}(\bo_i) &\Longrightarrow \\
    \bu_i &= \text{Dictionary}_\phi(\bz_i) + \bp_{\phi,i}
    &\Longrightarrow&&
    \bs_{1:N}, A_{1:N} &= \text{SlotAttention}_\phi(\bu_{1:T})\ .
\end{align*}
This results in $N$ object slots $\bs_{1:N}$ and $N$ attention maps $A_{1:N}$ from the last refinement iteration of the Slot-Attention encoder.

\textbf{Reconstruction using Transformer.} To reconstruct the input image, we decode the slots $\bs_{1:N}$ and first reconstruct the DVAE tokens $\hat{\bz}_{1:T}$ using a transformer. We then use the DVAE decoder $g_\ta$ to decode the DVAE tokens and reconstruct the image patches $\hat{\bx}_i$ and therefore the image $\hat{\bx}$.
\begin{align*}
   \hat{\bo}_i = \text{Transformer}_\ta(\hat{\bu}_{<i} ; \bs_{1:N})
   &&\Longrightarrow&&
   \hat{\bz}_i = \arg \max_{v\in[1, V]} \hat{o}_{i,v}
   &&\Longrightarrow&&
   \hat{\bx}_i = g_\ta({\hat{\bz}_i})\ .
\end{align*}
where $\hat{\bu}_{i} = \text{Dictionary}_\phi(\hat{\bz}_i) + \bp_{\phi,i}$ and $\hat{o}_{i,v}$ is the log probability of token $v$ among the $V$ classes of the categorical distribution represented by $\hat{\bo}_i$. To train the transformer, the slot attention encoder and the embeddings, we minimize the cross-entropy of predicting each token $\bz_i$ conditioned on the preceding tokens $\bz_{<i}$ and the slots $\bs_{1:N}$. Let the predicted log-probabilities for the token at position $i$ be $\bar{\bo}_i = \text{Transformer}_\ta({\bu}_{<i} ; \bs_{1:N})$. Then the cross-entropy training objective can be written as $\smash{\cL_\text{ST} = \sum_{i=1}^T \text{CrossEntropy}(\bz_i,\bar{\bo}_i)}$.

\textbf{Learning Objective and Training.} The complete training objective is given by $\cL = \cL_\text{ST} + \cL_\text{DVAE}$ and all modules of our model are trained jointly. We apply a decay on the DVAE temperature $\tau$ from 1.0 to 0.1 at the start of the training. We also apply a learning rate warm-up for the parameters of slot-attention encoder and the transformer at the start of the training as we found that this led to stable training.


\textbf{Multi-headed Slot Attention.}
For representing images, we found that slots of standard Slot Attention encoder \citep{slotattention} can suffer in expressiveness when representing objects with complex shapes and textures. This is because the slots collect information from the input cells via dot-product attention in which the attended input values are pooled using a simple weighted sum. As such, this pooling method can be too weak for representing complex objects. To address this, we propose an extension of Slot Attention called \textit{Multi-Headed Slot Attention} (MHSA) in which each slot attends to the input cells via multiple heads. Thus each slot can attend to different parts of the same object. When the slot state is computed, the attended values of different heads are concatenated which allows these partial object features to interact more flexibly and produce a significantly better object representation. Full details of the implementation and an ablation experiment to show the benefits of multiple heads are provided in Appendix \ref{ax:mhsa}.

\subsection{Visual Concept Library}
\label{sec:concept-lib}
The above model enables extracting a set of concepts or slots from a particular image whereas an intelligent agent would build a \textit{library of reusable concepts} from diverse experience such as a stream of images. In DALL$\cdot$E\xspace, a vocabulary of word embeddings plays the role of this library and provides reusable concepts. Similarly, to build a library of visual concepts from the dataset, we use the following simple approach based on $K$-means clustering to construct a library of reusable visual concepts. To do this, we propose the following steps: (i) Collect $N$ slots $\cS_i = \{\bs^i_1,\dots,\bs^i_N\}$ and their attention maps $\cA_i = \{A^i_1,\dots,A^i_N\}$ from each image $i$ in the training dataset. (ii) Run $K$-means clustering on $\smash{\cS = \bigcup_i \cS_i}$ with cosine similarity between slots as a distance metric where $K$ is the number of total concepts in the library. If the object position is more important for building the concept library, cluster the slots using IOU between the attention maps as a similarity metric. (iii) Use each cluster as a concept and the slots assigned to the cluster as the instantiations of the concept. (iv) To compose an arbitrary image, choose the concepts from the library and randomly choose a slot for each concept and build a \textit{slot prompt} similar to a text prompt in DALL$\cdot$E\xspace. (v) Provide this prompt to the decoder to compose and generate the image.



\section{Related Work}
\textbf{Compositional Generation via Object-Centric Learning.} Self-supervised object-centric approaches \citep{monet, iodine, slotattention, kabra2021simone, greff2017neural, engelcke2020genesis, engelcke2021genesis, air, spair, space, gnm, sqair, scalor, silot, gswm, deng2021generative, anciukevicius2020object, kugelgen2020towards, ocvt} typically perform mixture-based alpha compositing for rendering with contents of each slot decoded independently.
\cite{savarese2021information} and \cite{yang2020learning} rely on minimizing mutual information between predicted object segments as a learning signal while \cite{yang2021self} leverage optical flow. However, these approaches either cannot compose novel scenes or require specialized losses for object discovery unlike ours which uses a simple reconstruction loss. DINO \citep{caron2021emerging} combining Vision Transformer \citep{dosovitskiy2021an} and a self-supervised representation learning objective \citep{chen2020a, he2020momentum, grill2020bootstrap, touvron2021training, hinton2021how, oord2018representation} discovers object-centric attention maps but, unlike ours, cannot compose novel scenes. TIMs \citep{lamb2021transformers, goyal2021recurrent} with an Image GPT decoder show object-centric attention maps but, unlike ours, do not investigate compositional generation.

\textbf{Compositional Generation using GANs.}
\cite{chai2021using} show compositional generation using a collage of patches encoded and decoded using a pretrained GAN. However, the collage needs to be provided manually and object discovery requires a specialized process unlike our model.
\cite{bielski2019emergence} and \cite{chen2019unsupervised} show object discovery from given images by randomly re-drawing or adding a random jitter \citep{voynov2021big} to the foreground followed by an adversarial loss. As these models use alpha compositing, their compositional ability is limited. Some works \citep{donahue2016adversarial, donahue2019large} infer abstract representations for images that emphasize the high-level semantics, but these, unlike ours, cannot be used for compositional image generation.
\cite{niemeyer2021giraffe}, \cite{blockgan, chen2016infogan}, \cite{steenkiste2020investigating}, \cite{liao2020towards} and \cite{ehrhardt2020relate} introduce GANs which generate images conditioned on object-wise or factor-wise noise and optionally on camera pose. Lacking an inference module, these cannot perform compositional editing of a given image unlike ours. While \cite{kwak2016generating} provide an encoder to discover objects, the decoding relies on alpha compositing. \cite{reed2016learning} show compositional generation in GANs and improved generation quality \citep{johnson2018image, hinz2019generating} via an object-centric or a key-point based pathway for scene rendering. However, these require supervision for the bounding boxes and the keypoints.

\textbf{Compositional Generation with Latent Variable Models and Text.}
Early approaches focused on disentangling the independent factors of the observation generating process using $\beta$-VAE \citep{higgins2017beta} or ensuring that independently sampling each latent factor should produce images indistinguishable from the training distribution \citep{kim2018disentangling, kumar2017variational, chen2018isolating}. However unlike ours, these approaches providing a monolithic single vector representation can suffer in multi-object scenes due to \textit{superposition catastrophe} \citep{binding,gnm}. Another line of approaches learn to map text to images to compose novel scenes \citep{dalle, reed2016generative, li2019object, higgins2018scan} or map attribute-value pairs to images \citep{sohn2015learning, yan2016attribute2image}. However such approaches rely on supervision.







\section{Experiments}
\label{sec:exp}


In experiments, we evaluate the benefits of SLATE having a transformer decoder over the traditional pixel-mixture decoder in: 1) generating novel images from an arbitrary configuration of slots, 2) image reconstruction, and 3) generating novel images not belonging to the training image distribution. As the baseline, we compare with an unsupervised object-centric framework having a slot attention encoder like ours but using a mixture decoder for decoding instead of transformer. Hereafter, we refer to this model as simply Slot Attention \citep{slotattention}. We evaluate the models on four datasets: 3D Shapes \citep{3dshapes18}, CLEVR-Mirror which we develop from the CLEVR~\citep{clevr} dataset by adding a mirror in the scene, Shapestacks \citep{groth2018shapestacks} and Bitmoji \citep{bitmoji}. For brevity, we shall refer to CLEVR-Mirror as simply CLEVR. In each dataset, a scene can be described in terms of composable entities such as blocks, walls, floors, hair, or face.
Our models take only raw images without any other supervision or annotations.


\subsection{Compositional Image Generation}
\label{sec:exp:compgen}
\begin{table}[t]
\begin{subtable}{.42\linewidth}\centering
{\begin{tabular}{@{}lcc@{}}
\toprule
            & \multicolumn{2}{c}{FID $(\downarrow)$}                                     \\ \cmidrule(l){2-3}
Dataset     & SA & Ours \\ \midrule
3D Shapes   &  44.14      &  \textbf{36.75}    \\
CLEVR       &  63.76        & \textbf{41.62}    \\
Shapestacks &  155.74        &   \textbf{51.27}    \\
Bitmoji     &  71.43       &  \textbf{15.83}    \\ \bottomrule
\end{tabular}}
\caption{Compositional Generation}\label{tab:1a}
\end{subtable}%
\begin{subtable}{.58\linewidth}\centering
{\begin{tabular}{@{}lcccc@{}}
\toprule
           
            & \multicolumn{2}{c}{MSE $(\downarrow)$} & \multicolumn{2}{c}{FID $(\downarrow)$} \\ \cmidrule(l){2-5}
Dataset     & SA               & Ours            & SA         & Ours        \\ \midrule
3D Shapes   & 16.65             & \textbf{10.49} & 44.71                & \textbf{38.20}      \\
CLEVR       & \textbf{29.59} &  49.95     & 55.95          &  \textbf{37.42}          \\
Shapestacks & 233.72        & \textbf{111.86}    & 139.72                     &   \textbf{30.22}  \\
Bitmoji     & 388.72        & \textbf{261.10}    & 67.66              &   \textbf{11.89}       \\ \bottomrule
\end{tabular}}
\caption{Image Reconstruction}\label{tab:1b}
\end{subtable}
\caption{Comparison of Compositional Generation (left) and Image Reconstruction (right) between Slot-Attention (SA) and our model. For comparison of compositional generation, we report the FID score. For image reconstruction quality, we report the MSE and the FID score.}\label{tab:1}
\end{table}
\begin{figure}[t]
    \centering
    \includegraphics[valign=t,width=1.0\textwidth]{images/plot-discriminator-007.pdf}
    \caption{Comparison of training curves of a discriminator to compare the quality of compositional generation between Slot-Attention and our model. A CNN discriminator tries to discriminate between real images and model-generated images. We note that the curves converge more slowly for our model than for Slot Attention and show that our generations are more realistic.}
    \label{fig:discriminator-training-curves}
\end{figure}





\begin{figure}[t]
    \centering
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
    \includegraphics[width=1.0\textwidth]{images/compositional_gen_3dshapes_bitmoji_joint.png}
    \caption{\textbf{
   
    Comparison of compositional generation between Slot Attention and our model in 3D Shapes and Bitmoji datasets.
    } We visualize the slot prompts provided to the decoder and the resulting compositions. To visualize the prompt slots, we show the image from which the slot was taken masked using its input attention map. Note that these attention maps had emerged in the source images implicitly without any supervision. We note that our decoder performs zero-shot generalization to novel slot inputs while the mixture decoder fails to compose arbitrary slots well.
    }
    \label{fig:qual-compgen-bitmoji-3shapes-001}
\end{figure}
\begin{figure}[t]
    \centering
    \includegraphics[valign=t,width=1.0\textwidth]{images/bitmoji-traversal-diagram-hair-and-face-3.png}
    \caption{\textbf{Object Replacement and Out-of-Distribution Compositions in Bitmoji.} We show that in our model, it is possible to edit the image by taking a specific slot and replacing it with an arbitrary slot drawn from the concept library for the same concept. We also show OOD compositions by replacing the slot with those from the opposite gender.}
    \label{fig:replacements_bitmoji}
\end{figure}

We evaluate how effectively the known slots can be reused to generate novel scenes. For this, we evaluate the quality of image generations produced by our model from arbitrarily composed slot prompts. To build these prompts, we first build a concept library using $K$-means as described in Section \ref{sec:concept-lib}. In 3D Shapes and Bitmoji, we apply $K$-means clustering on the slots. In 3D Shapes, this results in clusters for \textit{wall}, \textit{object} and \textit{floor} and in Bitmoji for \textit{dress}, \textit{hair}, \textit{neck}, \textit{eyes}, \textit{face} and \textit{collar}. These clusters make up our concept library and are visualized in Appendix \ref{ax:additional_qual}. To build novel prompts from this library, we simply pick one slot from each cluster. In CLEVR and Shapestacks datasets, to build a prompt, it is more natural to pick the object positions first and then sample a slot from the library for that position on the canvas. Hence, for CLEVR, we pick a set of random positions keeping a certain distance between the picked positions. For Shapestacks, we pick a set of random positions in a vertical tower-like configuration.
In this way, we generate $40 000$ prompts for each dataset and render the corresponding images. We then evaluate the quality of the images by computing the FID score with respect to the true images. To support this metric, we also report the training curves of a CNN discriminator that tries to classify between real and model-generated images. If generated images are close to the true data distribution, then it should be harder for the discriminator to discriminate and the resulting training curve should converge more slowly.

From Table \ref{tab:1a} and Figure \ref{fig:discriminator-training-curves}, we note that our compositional generations are significantly more realistic than the generations from the mixture decoder. The qualitative samples shown in Figures \ref{fig:qual-compgen-bitmoji-3shapes-001}, \ref{fig:replacements_bitmoji}, \ref{fig:additions_clevr} and \ref{fig:qual-compgen-shapestacks-001} show that our decoder can compose the desired image accurately and make an effective reuse of the known slots. Additional qualitative results are provided in Appendix \ref{ax:additional_qual}. We also note that for visually complex Shapestacks and Bitmoji datasets, the improvement in the FID score provided by our model is much larger than that for the visually simpler CLEVR and 3D Shapes datasets.

These results provide evidence that \textit{i)} our model resolves the \textit{slot-decoding dilemma} and consequently, it benefits from the flexibility of the Image GPT decoder that can render the fine details such as floor texture in Shapestacks and complex facial features in Bitmoji while the mixture decoder suffers. Notice that the attention segments of the visualized slots effectively focus on the specific object regions of the respective images that were provided during concept library creation. This emergent object localization occurs despite the powerful decoder, thus resolving the dilemma. We visualize the attention maps of the slots for a given input image in Figures \ref{fig:qual-recon-shapestacks-ours-000} -- \ref{fig:qual-recon-3dshapes-ours-000} in Appendix. Qualitatively, we note that our attention maps are comparable to those of Slot Attention and arguably better in the case of Bitmoji as they attend to the semantically meaningful regions such as \textit{hair}, \textit{face}, \textit{eyes}, \textit{clothes} or \textit{collar} compared to the slots of Slot Attention.  Because our slots attend on the DVAE tokens, hence our attention is patch-level and our attention segments may appear coarser than those of Slot Attention in which slots attend on the CNN feature cells having the same resolution as the input image.
\textit{ii)} Because the mixture decoder has the \textit{pixel-independence assumption}, we see in Figure \ref{fig:qual-compgen-bitmoji-3shapes-001} that it suffers when slots are combined in arbitrary configurations. In 3D Shapes, the object shadows are controlled by the floor component of the mixture and are either too large or too small for the 3D object. This is because after arbitrarily choosing a different 3D object, the floor slot has no knowledge that the object which casts the shadow has changed and it continues to draw the shadow mask of the source image which has now become obsolete in the new slot configuration.
Slot Attention suffers even more severely in Bitmoji as the composed mixture components and masks are mutually incompatible in the new slot configuration and produce an inaccurate alpha composition. In contrast, the SLATE decoder is robust to this problem because the slots and pixels can interact. For instance, in 3D Shapes, as the pixels of the floor are being rendered, the predictor is aware of the object that will cast the shadow via the previous pixels and the input slots and can thus predict a globally consistent pixel value.



















\begin{figure}[t]
    \centering
    \includegraphics[valign=t,width=1.0\textwidth]{images/clevr-qual-adder-compare-002.png}
    \caption{\textbf{Compositional Generation in CLEVR.} We show compositional generation in CLEVR by adding object slots one at a time. In training, only 3-6 objects were shown. However, we note that our model can generalize in generating 1-2 and 7-8 objects including the empty scene. We also note that our mirror reflections are significantly more clear than those of Slot Attention.}
    \label{fig:additions_clevr}
\end{figure}
\begin{figure}[t]
    \centering
    \includegraphics[valign=t,width=1.0\textwidth]{images/compositional_gen_sa_ours_compare_001.png}
    \caption{\textbf{Compositional generation in Shapestacks Dataset.} We show compositional generation in Shapestacks by composing arbitrary blocks drawn from the concept library in a tower configuration. We note that SLATE can render the details such as the texture of the floor and the wall significantly better than the mixture decoder.}
    \label{fig:qual-compgen-shapestacks-001}
\end{figure}





\subsection{Reconstruction Quality}

A natural test about whether an auto-encoder learns accurate representations of the given input is through an evaluation of the reconstruction error. For this, we report two metrics: 1) MSE to evaluate how well the reconstructed image preserves the contents of the original image and 2) FID score to show how realistic is the reconstructed image with respect to the true image distribution. We compute these metrics on a held-out set and report in Table \ref{tab:1b}. We find that our model outperforms Slot Attention in all datasets in terms of FID which shows that the image quality of our model is better due to its powerful decoder.
In MSE, our model outperforms the baseline in all datasets except for CLEVR. We note that the performance gain from our model, in both MSE and FID, increases as the visual complexity of the dataset increases from 3D Shapes and CLEVR to Shapestacks and Bitmoji. Despite the better MSE of the mixture decoder in CLEVR, the gap is not severe given the broader limitations of the mixture decoder in handling novel slot configurations. We also believe that the MSE of our model is affected by the DVAE patch size which can affect the rendering of small-sized objects of CLEVR. To test this hypothesis, we compared using a coarse
patch grid and a fine
patch grid and we found the latter to reduce the MSE gap with Slot Attention. Thus, in the future, with more compute that can enable a finer DVAE grid, this discrepancy may be resolved.
























\subsection{Generalization to Out-of-Distribution (OOD) Slot Reconfigurations}
Because the SLATE decoder is based on transformer and does not have an explicit inductive bias about scene composition, we test whether it can generalize in composing slots when their configurations are different from the training distribution.
For this, \textit{i)} in Shapestacks, we compose slot prompts to render two towers while only single-tower images were shown during training. \textit{ii)} In CLEVR, we compose prompts to render fewer or more objects than were shown during training. \textit{iii)} Lastly, in Bitmoji, we annotated the gender of 500 images and used it to construct 1600 OOD slot prompts by composing hair and face only from opposite genders. By doing this, we change the distribution of the slot configurations with respect to the training distribution. Note that such slot prompts were already a part of the Bitmoji compositions in Section \ref{sec:exp:compgen}. However, with this experiment, we evaluate the specific OOD subset of those prompts. We show the results in Figures \ref{fig:replacements_bitmoji}, \ref{fig:additions_clevr} and \ref{fig:qual-compgen-shapestacks-ood-001} and a comparison of FID score with respect to the mixture decoder in Table \ref{tab:ood-table}.
From these, we note that our decoder is able to generalize to the OOD slot configurations and it also outperforms the mixture decoder in image quality.



\begin{figure}[t]
\begin{minipage}[t]{0.68\linewidth}
\vspace{0mm}
\centering
\includegraphics[width=1.0\linewidth]{images/ood-shapestacks-ours-001.png}
\captionof{figure}{\textbf{Out-of-Distribution Generalization in Shapestacks Dataset.} We provide slot prompts to render two towers instead of one. The models were trained only on images with one tower.}
\label{fig:qual-compgen-shapestacks-ood-001}
\end{minipage}
\hfill
\begin{minipage}[t]{0.30\linewidth}
\vspace{0mm}
\centering
\small
\begin{tabular}{@{}lcc@{}}
\toprule
            & \multicolumn{2}{c}{FID $(\downarrow)$}                                     \\ \cmidrule(l){2-3}
Dataset     & SA & Ours \\ \midrule
Shapestacks   &   172.82     &  \textbf{82.38}  \\
CLEVR  &   62.44     &   \textbf{43.77}  \\
Bitmoji & 81.66 & \textbf{22.75} \\
\bottomrule
\end{tabular}
\captionof{table}{FID Comparison for OOD Compositions.}
\label{tab:ood-table}
\end{minipage}
\end{figure}









\section{Discussion}

We presented a new model, SLATE, for text-free zero-shot imagination by learning slot representations only from images. Our model combines the best of DALL$\cdot$E\xspace and object-centric representation learning.
It
provides a significant improvement over pixel-mixture decoders in reconfigurability of slots for zero-shot imagination. These results suggest that future object-centric models may benefit from building on the advantages of the proposed framework alongside the pixel-mixture decoder.

The decoder of SLATE, with its transformer-based implementation, can also be seen as a graphics engine that learns to mimic the physical laws governing the rendering of the image through its exposure to a large dataset. Thus to render a scene, the only information that the transformer requires and requests from the encoder is the abstract object-level \textit{what} and \textit{where} in the form of slots. Knowing these objects, the transformer then fills in the rest and implicitly handles the mechanisms that give rise to the observed object appearance, occlusions, shadows, reflections, and transparency.

An interesting future direction is to consider that an intelligent agent may collect new experience indefinitely, and thus our approach of building the library of reusable concepts can be made more formal by learning discrete concepts and growing this library using online clustering~\citep{onlinekmeans} or Bayesian non-parametric clustering~\citep{dpmm}. Another direction is to learn a slot-level prior for fast density modeling and sampling scenes at the representation level.


\section*{Ethics Statement}
For future applications, negative societal consequences should also be considered. The current model does not generate images realistic enough to deceive humans if used with malicious intent. However future versions of the model after scaling up its size and the training dataset can have such potential. Despite these considerations, these consequences are not imminent.













