\section{Introduction}

In object-centric representation learning, we make the assumption that visual scenes are composed of multiple entities or objects that interact with each other, and exploit this compositional property as inductive bias for neural networks. Informally, the goal is to find transformations $r$ of the data $\mathbf{x}$ into a \emph{set} of vector representations $r(\mathbf{x}) = \{\mathbf{z}_k\}$ each corresponding to an individual object, without supervision \citep{eslami2016air, greff2017neural,kosiorek2018sequential,crawford2019spatially, greff2019multi, burgess2019monet, engelcke2020genesis, lin2020space, locatello2020object, chen2020learning,weis2020unmasking,mnih2014recurrent,gregor2015draw,yuan2019generative}.
Relying on this inductive bias, object-centric representations are conjectured to be more robust than distributed representations, and to enable the systematic generalization typical of symbolic systems while retaining the expressiveness of connectionist approaches \cite{bengio2013representation,lake2017building,greff2020binding,scholkopf2021toward}.
Grounding for these claims comes mostly from cognitive psychology and neuroscience \cite{spelke1990principles, teglas2011pure, wagemans2015oxford}. E.g., infants learn about the physical properties of objects as entities that behave consistently over time \cite{baillargeon1985object,spelke2007core} and are able to re-apply their knowledge to new scenarios involving previously unseen objects \cite{dehaene2020how}. Similarly, in complex machine learning tasks like physical modelling and reinforcement learning, it is common to train from the internal representation of a simulator \cite{battaglia2016interaction, sanchez-gonzalez2020learning} or of a game engine \cite{vinyals2019grandmaster, openai2019dota} rather than from raw pixels, as more abstract representations facilitate learning.
Finally, learning to represent objects separately is a crucial step towards learning causal models of the data from high-dimensional observations, as objects can be interpreted as causal variables that can be manipulated independently~\cite{scholkopf2021toward}. Such causal models are believed to be crucial for human-level generalization~\cite{pearl2009causality,peters2017elements}, but traditional causality research assumes causal variables to be given rather than learned~\cite{scholkopf2019causality}.

As object-centric learning developed recently as a subfield of representation learning, we identify three key hypotheses and design systematic experiments to test them. 
(1)~\emph{The unsupervised learning of objects as pretraining task is useful for downstream tasks.} Besides learning to separate objects without supervision, current approaches are expected to separately represent information about each object's properties, so that the representations can be useful for arbitrary downstream tasks. 
(2)~\emph{In object-centric models, distribution shifts affecting a single object do not affect the representations of other objects.}
If objects are to be represented independently of each other to act as compositional building blocks for higher-level cognition~\cite{greff2020binding}, changes to one object in the input should not affect the representation of the unchanged objects. This should hold even if the change leads to an object being out of distribution (OOD).
(3)~\emph{Object-centric models are generally robust to distribution shifts, even if they affect global properties of the scene.} 
Even if the whole scene is OOD---e.g., if it contains more objects than in the training set---object-centric approaches should be robust thanks to their inductive bias.\looseness=-1

In this paper, we systematically investigate these three concrete hypotheses by re-implementing popular unsupervised object discovery approaches and testing them on five multi-object datasets.\footnote{Training and evaluating all the models for the main study requires approximately 1.44 GPU years on NVIDIA V100.} 
We find that:
(1)~Object-centric models achieve good downstream performance on property prediction tasks. We also observe a strong correlation between segmentation metrics, reconstruction error, and downstream property prediction performance, suggesting potential model selection strategies. 
(2)~If a single object is out of distribution, the overall segmentation performance is not strongly impacted. Remarkably, the downstream prediction of in-distribution (ID) objects is mostly unaffected. 
(3)~Under more global distribution shifts, the ability to separate objects depends significantly on the model and shift at hand, and downstream performance may be severely affected.

As an additional contribution, we provide a library\footnote{\url{https://github.com/addtt/object-centric-library}}
for benchmarking object-centric representation learning, which can be extended with more datasets, methods, and evaluation tasks. We hope this will foster further progress in the learning and evaluation of object-centric representations.



\section{Study design and hypotheses}\label{sec:study_design}

\textbf{Problem definition:} 
Vanilla deep learning architectures learn distributed representations that do not capture the compositional properties of natural scenes---see, e.g., the ``superposition catastrophe''~\cite{von1986thinking,bowers2014neural,greff2020binding}. Even in disentangled representation learning~\cite{higgins2016beta, kim2018disentangling, chen2018isolating, ridgeway2018learning, kumar2017variational, eastwood2018framework}, factors of variations are encoded in a vector representation that is the output of a standard CNN encoder. This introduces an unnatural ordering of the objects in the scene and fails to capture its compositional structure in terms of objects.
Formally defining objects is challenging \citep{greff2020binding} and there is no consensus even outside of machine learning \citep{smith1996origin,green2019theory}. \citet{greff2020binding} put forth three properties for object-centric representations: \emph{separation}, i.e., object features in the set of vectors $r(\mathbf{x})$ do not interact with each other, and each object is individually captured in a single element of $r(\mathbf{x})$; \emph{common format}, i.e., each element of $r(\mathbf{x})$ shares the same representational format; and \emph{disentanglement}, i.e., each element of $r(\mathbf{x})$ is represented in a disentangled format that exposes the factors of variation. 
In this paper, we consider representations $r(\mathbf{x})$ that are sets of vectors with each element sharing the representational format. 
We take a pragmatic perspective and focus on two clear desiderata for object-centric approaches:

\textbf{Desideratum 1: } \emph{Object embodiment.}
The representation should contain information about the object's location and its embodiment in the scene.
As we focus on unsupervised object discovery, this translates to segmentation masks. This is related to separation and common format, as the decoder is applied to the elements of $r(\mathbf{x})$ with shared parameters.

\textbf{Desideratum 2:} \emph{Informativeness of the representation.} Instead of learning disentangled representations of objects, which is challenging even in single-object scenarios \citep{locatello2018challenging}, we want the representation to contain useful information for downstream tasks, not necessarily in a disentangled format. We define objects through their properties as annotated in the datasets we consider, and predict these properties from the representations. Note that this may not be the only way to define objects (e.g., defining faces and edges as objects and deducing shapes as composition of those). The fact that existing models learn informative representations is our first hypothesis (see below).

\textbf{Design principle:} These desiderata offer well-defined quantitative evaluations for object-centric approaches and we want to understand the implications of learning such representations. To this end, we train four different state-of-the-art methods on five datasets, taking hyperparameter configurations from the respective publications and adapting them to improve performance when necessary. Assuming these models succeeded in learning an object-centric representation, we investigate the following hypotheses.

\textbf{Hypothesis 1:} \emph{The unsupervised learning of objects as pretraining task is useful for downstream tasks.}  Existing empirical evaluations largely focus on \emph{Desideratum 1} and measure performance in terms of segmentation metrics. The hope, however, is that the learned representation would be useful for other downstream tasks besides segmentation (\emph{Desideratum 2}). We test this hypothesis by training small downstream models on the frozen object-representations 
to predict the object properties. We match the predictions to the ground-truth properties with the Hungarian algorithm \cite{kuhn1955hungarian} following \citet{locatello2020object}.

\textbf{Hypothesis 2:} 
\emph{In object-centric models, distribution shifts affecting a single object do not affect the representations of other objects.}
A change in the properties of one object in the input should not affect the representation of the other objects. Even OOD objects with previously unseen properties should be segmented correctly by a network that learned the notion of objects~\citep{greff2020binding,scholkopf2021toward}.
We test this hypothesis by (1)~evaluating the segmentation of the scene after the distribution shift, and (2)~training downstream models to predict object properties, and evaluating them on representations extracted from scenes with one OOD object. More specifically, we test changes in the shape, color, or texture of one object.

\textbf{Hypothesis 3:} \emph{Object-centric models are generally robust to distribution shifts, even if they affect global properties of the scene.} Early evidence~\citep{romijnders2021representation} points to the conjecture that learning object-centric representations biases the network towards learning more robust representations of the overall scene. Intuitively, the notion of objects is an additional inductive bias for the network to exploit to maintain accurate predictions if simple global properties of the scene are altered. We test this hypothesis by training downstream models to predict object properties, and evaluating them on representations of scenes with OOD global properties. In this case, we test robustness by cropping, introducing occlusions, and increasing the number of objects.\looseness=-1


\begin{figure}
    \centering
    \captionsetup{belowskip=-15pt}
    \includegraphics[width=\linewidth]{figures/datasets_and_shifts}
    \caption{\textbf{Top}: examples from the five datasets in this study. \textbf{Bottom}: distribution shifts in CLEVR.}
    \label{fig:datasets}
\end{figure}



\section{Experimental setup}
\label{sec:experimental_setup}

Here we provide an overview of our experimental setup. After introducing the relevant models and datasets, we outline the evaluation protocols for segmentation accuracy (Desideratum~1) and downstream task performance (Desideratum~2). Then, we discuss the distribution shifts that we use to test robustness---the aforementioned evaluations are repeated once again under these distribution shifts.
We conclude with a discussion on the limitations of this study.

\textbf{Models and datasets.}
We implement four state-of-the-art object-centric models---MONet \cite{burgess2019monet}, GENESIS~\cite{engelcke2020genesis}, Slot Attention~\cite{locatello2020object}, and SPACE~\cite{lin2020space}---as well as vanilla variational autoencoders (VAEs) \citep{kingma2013auto,rezende2014stochastic} as baselines for distributed representations. 
We use one VAE variant with a broadcast decoder \cite{watters2019spatial} and one with a regular convolutional decoder.
See \cref{app:models} for an overview of the models with implementation details.
We then collect five popular multi-object datasets: \emph{Multi-dSprites}, \emph{Objects Room}, and \emph{Tetrominoes} from DeepMind's Multi-Object Datasets collection~\cite{multiobjectdatasets19}, \emph{CLEVR}~\cite{johnson2017clevr}, and \emph{Shapestacks}~\cite{groth2018shapestacks}. The datasets are shown in \cref{fig:datasets} (top row) and described in detail in \cref{app:datasets}.
For each dataset, we define train, validation, and test splits. The test splits, which always contain at least $\num{2000}$ images, are exclusively used for evaluation.
We train each model on all datasets, using 10 random seeds for object-centric models and 5 for each VAE variant, resulting in 250 models in total.

\textbf{Metrics.} We evaluate the segmentation accuracy of object-centric models with the Adjusted Rand Index (ARI)~\cite{hubertComparingPartitions1985}, Segmentation Covering (SC)~\cite{arbelaez2010contour}, and mean Segmentation Covering (mSC)~\cite{engelcke2020genesis}. For all models, we additionally evaluate reconstruction quality via the mean squared error (MSE).
\cref{app:evaluation_metrics} includes detailed definitions of these metrics.

\textbf{Downstream property prediction.}
We evaluate object-centric representations by training downstream models to predict ground-truth object properties from the representations. More specifically, exploiting the fact that object slots share a common representational format, a single downstream model $f$ can be used to predict the properties of each object independently: for each slot representation $\mathbf{z}_k$ we predict a vector of object properties $\hat{\mathbf{y}}_k = f(\mathbf{z}_k)$.
As in previous work on object property prediction~\cite{locatello2020object}, each model simultaneously predicts all properties of an object. For learning, we use the cross-entropy loss for categorical properties and MSE for numerical properties, and denote by $\ell(\hat{\mathbf{y}}_k, \mathbf{y}_m)$ the overall loss for a single object, where $\mathbf{y}_m$ are its ground-truth properties. Here $k \in \{1,\ldots,K\}$ and $m \in \{1,\ldots,M\}$ with $K$ the number of slots and $M$ the number of objects.
In order to optimize the downstream models, the vector $\hat{\mathbf{y}}_k$ (the properties predicted from the $k$th representational slot) needs to be matched to the ground-truth properties $\mathbf{y}_m$ of the $m$th object. This is done by computing a $M \times K$ matrix of \emph{matching losses} for each slot--object pair, and then solving the assignment problem using the Hungarian algorithm~\cite{kuhn1955hungarian} to minimize the total matching loss, which is the sum of $\min(M, K)$ terms from the loss matrix.
As matching loss we use either the negative cosine similarity between predicted and ground-truth masks (as in \citet{greff2019multi}), or the downstream loss $\ell(\hat{\mathbf{y}}_k, \mathbf{y}_m)$ itself (as in \citet{locatello2020object}). In the following, we will refer to these strategies as \emph{mask matching} and \emph{loss matching}, respectively.
For property prediction, we use 4 different downstream models: a linear model, and MLPs with up to 3 hidden layers of size 256 each.
Given a pretrained object-centric model, we train each downstream model on the representations of $\num{10000}$ images. The downstream models are then tested on $\num{2000}$ held-out images from the test set, which may exhibit distribution shifts as discussed below.
Further details on this evaluation are provided in \cref{app:evaluation_downstream}



\textbf{Evaluating distributed representations.}
Since in non-slot-based models, such as classical VAEs, the representations of the single objects are not readily available, matching representations to objects for downstream property prediction is not trivial.
Although this is an inherent limitation of distributed representations, we are nevertheless interested in evaluating their usefulness.
Using the matching framework presented above, we require the downstream model $f$ to output the predicted properties of \emph{all} objects, and then match these with the true object properties to evaluate prediction quality.
Our downstream model in this case will thus take as input the entire representation $\mathbf{z} = r(\mathbf{x})$ (which is now a single vector rather than a set of vectors) and output the predictions for all objects together as a vector $f(\mathbf{z})$. Finally, we split $f(\mathbf{z})$ into $K$ vectors $\{\hat{\mathbf{y}}_k\}_{k=1}^K$, where $K$ loosely corresponds to the number of slots in object-centric models. At this point, we can compute the loss $\ell(\hat{\mathbf{y}}_k, \mathbf{y}_m)$ for each pair, as usual.
We now consider two matching strategies: As before, \emph{loss matching} simply defines the matching loss of a slot--object pair as the prediction loss itself. In the \emph{deterministic matching} strategy, following \citet{greff2019multi}, we lexicographically sort objects according to a canonical order of object properties.
Calling $\pi$ the permutation that defines this sorting, the $k$th slot is deterministically matched with the $m$th object, where $m = \pi^{-1}(k)$.


\textbf{Baselines.}
To correctly assess performance on downstream tasks, it is fundamental to compare with sensible baselines. Here we consider as baseline the best performance that can be achieved by a downstream model that outputs a constant vector that does not depend on the image. When predicting properties independently for each object (in slot-based models), the optimal solution is to predict the mean of continuous properties and the majority class for categorical ones. When using deterministic matching in the distributed case, the downstream model can exploit the predefined total order to predict more accurately than random guessing even without using information from the input (this effect is non-negligible only for the properties that are most significant in the order). Finally, in a few cases, loss matching for distributed representations can be significantly better than deterministic matching.\footnote{Intuitively, a (constant) diverse set of uninformed predictions $\{\hat{\mathbf{y}}_k\}$ might be sufficient for the matching algorithm to find suitable enough objects for most predictions.} 
For simplicity, for both matching strategies in the distributed case, we directly learn a vector $\hat{\mathbf{y}}$ by gradient descent to minimize the prediction loss. As this depends on random initialization and optimization dynamics, we repeat this for 10 random seeds and report error bars in the plots.\looseness=-1


\textbf{Distribution shifts.}
We test the robustness of the learned representations under two classes of distribution shifts: one where one object goes OOD, and one where global properties of the scene are changed. All such distribution shifts occur at test time, i.e., the unsupervised models are always trained on the original datasets.
To evaluate generalization to distribution shifts affecting a \textit{single object}, we systematically induce changes in the color, shape, and texture of objects.
To change color, we apply a random color shift to one random object in the scene, using the available masks (we do not do this in Multi-dSprites, as the training distribution covers the entire RGB color space).
To test robustness to unseen textures, we apply neural style transfer~\cite{gatys_neural_2016} to one random object in each scene, using \emph{The Great Wave off Kanagawa} as style image.
When either a new color or a new texture is introduced, prediction of material (in CLEVR only) and color is not performed.
To introduce a new shape, we select images from Multi-dSprites that have at most 4 objects (in general, they have up to 5), and add a randomly colored triangle, in a random position, at a random depth in the object stack. In this case, shape prediction does not apply.
Finally, to test robustness to \textit{global} changes in the scene, we change the number of objects (in CLEVR only), introduce occlusions (a gray square at a random location), or crop images at the center and restore their original size via bilinear interpolation.
See \cref{fig:datasets} for examples, and \cref{app:evaluation_shifts} for further details.


\textbf{Limitations of this study.}
While we aim to conduct a sound and informative experimental study to answer the research questions from \cref{sec:experimental_setup}, inevitably there are limitations regarding datasets, models, and evaluations.
Although the datasets considered here vary significantly in complexity and visual properties, they all consist of synthetic images where object properties are independent of each other and independent between objects.
Regarding object-centric models, we only focus on autoencoder-based approaches that model a scene as a mixture of components.
As official implementations are not always available, and none of the methods in this work has been applied to all the datasets considered here, we re-implement these methods and choose hyperparameters following a best-effort approach.
Finally, we only consider the downstream task of object property prediction, and assess generalization using only a few representative single-object and global distribution shifts.



\section{Results}

In this section, we highlight our findings with plots that are representative of our main results. The full experimental results are presented in \cref{app:results}. In \cref{sec:results/metrics} we focus on the different evaluation metrics and the performance we obtained re-training the methods considered in this study. We then focus on our three hypotheses in \cref{sec:results/hyp1,sec:results/hyp2,sec:results/hyp3}.


\subsection{Learning and evaluating object discovery}
\label{sec:results/metrics}

    \begin{figure}
        \centering
        \captionsetup{belowskip=-14pt}
        \includegraphics[width=\linewidth,trim={10pt 10pt 5pt 10pt}]{figures/indistrib/metrics/barplot_ARI}
        \caption{ARI of all models and datasets on $\num{2000}$ test images. Medians and 95\% confidence intervals with 10 seeds.}
        \label{fig:main_text/indistrib/metrics/box_plots}
    \end{figure}

Since all methods included in our study were originally evaluated only on a subset of the datasets and metrics considered here, we first test how well these models perform.

\cref{fig:main_text/indistrib/metrics/box_plots} shows the segmentation performance of the models in terms of ARI across models, datasets, and random seeds. \cref{fig:indistrib/metrics/box_plots} in \cref{app:results} provides an overview of the reconstruction MSE and all segmentation metrics. Although these results are in line with published work, we observe substantial differences in the ranking between models depending on the metric. This indicates that, in practice, these metrics are not equivalent for measuring object discovery.

This is confirmed in \cref{fig:indistrib_correlation/metrics_metrics}, which shows rank correlations between metrics on different datasets (aggregating over different models).
We also observe a strong negative correlation between ARI and MSE across models and datasets, suggesting that models that learn to more accurately reconstruct the input tend to better segment objects according to the ARI score.
This trend is less consistent for the other segmentation metrics, as MSE significantly correlates with mSC in only three datasets (Multi-dSprites, Objects Room, and CLEVR), and with SC in two (Multi-dSprites and Objects Room).
SC and mSC measure very similar segmentation notions and therefore are significantly correlated in all datasets, although to a varying extent. However, they correlate with ARI only on two and three datasets, respectively (the same datasets where they correlate with the MSE).

\textbf{Summary:} We observe strong differences in performance and ranking between the models depending on the evaluation metric. In the tested datasets, we find that the ARI, which requires ground-truth segmentation masks to compute, correlates particularly well with the MSE, which is unsupervised and provides training signal. 


    
\subsection{Usefulness for downstream tasks (Hypothesis 1)}
\label{sec:results/hyp1}
To test Hypothesis 1, we first evaluate whether frozen object-centric representations can be used to train downstream models measuring {Desideratum 2} from \cref{sec:study_design}. As discussed in \cref{sec:experimental_setup}, this type of downstream task requires matching the true object properties with the predictions of the downstream model. In the following, we will only present results obtained with \emph{loss matching}, and show results for other matching strategies in \cref{app:results}.

    \begin{figure}
        \centering
        \captionsetup{belowskip=-8pt}
        \includegraphics[width=\linewidth,trim={9pt 10pt 9pt 10pt}]{figures/indistrib/metrics/corr_metrics_metrics}
        \caption{Spearman rank correlations between evaluation metrics across models and random seeds (color-coded only when p\textless 0.05).}
        \label{fig:indistrib_correlation/metrics_metrics}
    \end{figure}
    
    \begin{figure*}
        \centering
        \includegraphics[width=\linewidth]{figures/indistrib/downstream/barplots_loss_MLP1_main}
        \caption{Comparison of downstream property prediction performance for object-centric (slot-based) and distributed (VAE) representations, using an MLP with one hidden layer as downstream model. The metric is accuracy for categorical properties or $R^2$\xspace for numerical ones. The baselines in gray indicate the best performance that can be achieved by a model that outputs a constant vector that does not depend on the input. The bars show medians and 95\% confidence intervals with 10 random seeds.}
        \label{fig:indistrib/downstream/barplots_loss_linear_main}
    \end{figure*}

\cref{fig:indistrib/downstream/barplots_loss_linear_main} shows downstream prediction performance on all datasets and models, when the downstream model is a single-layer MLP.
Although results vary across datasets and models, accurate prediction of object properties seems to be possible in most of the scenarios considered here.
\cref{fig:indistrib/downstream/box_plots} in \cref{app:results} shows similar results when using a linear model or MLPs with up to 3 hidden layers.


In \cref{fig:indistrib/downstream/barplots_loss_linear_main}, we also compare the downstream prediction performance from object-centric and distributed representations. We observe that VAE representations tend to achieve lower scores in downstream prediction, although not always by a large margin. In particular, color and size in CLEVR and color in Tetrominoes are predicted relatively well, and significantly better than the baseline. On the other hand, in many cases where VAE representations perform well, they have in fact a considerable advantage if we take the baselines into account (scale in Multi-dSprites, color in Shapestacks, x~and~y in CLEVR, Multi-dSprites, and Tetrominoes). Moreover, performance from distributed representations often does not improve significantly when using a larger downstream model (see \cref{fig:indistrib/downstream/barplots_compare_downstream_models_loss}).
In conclusion, although the two classes of representations are difficult to compare on this task, these results suggest that the quantities of interest are present in the VAE representations, but they appear to be less explicit and less easily usable.

    \begin{figure}
        \centering
        \captionsetup{belowskip=-10pt}
        \includegraphics[width=\linewidth,trim={10pt 10pt 10pt 10pt}]{figures/indistrib/downstream/corr_metrics_downstream_loss_MLP1}
        \caption{Spearman rank correlations between evaluation metrics and downstream performance with an MLP. The correlations are color-coded only when p\textless 0.05.}
        \label{fig:indistrib_correlations/metrics_downstream}
    \end{figure}

Finally, we investigate the relationship between downstream performance and evaluation metrics.
\cref{fig:indistrib_correlations/metrics_downstream} shows the Spearman rank correlation of the segmentation and reconstruction metrics with the test performance of downstream predictors.
For all datasets and 
object 
properties, downstream performance is strongly correlated with the ARI.
On the other hand, SC and mSC exhibit inconsistent trends across datasets.
Models that correctly separate objects according to the ARI are therefore useful for downstream object property prediction, confirming Hypothesis 1.
Downstream prediction performance is also significantly correlated with the reconstruction MSE in all datasets. This is not particularly surprising, since the representation of a model that cannot properly reconstruct the input might not contain the information necessary for property prediction. However, the correlation is generally stronger with the ARI than with the MSE, suggesting that having a notion of objects is more important for downstream tasks than reconstruction accuracy. This is consistent with the findings by \citet{papa2022inductive}, where the ARI still correlates strongly with downstream performance when objects have complex textures, while the MSE does not. When segmentation masks are available for validation, ARI should therefore be the preferred metric to select useful representations for downstream tasks.
\cref{fig:indistrib/downstream/corr_metrics_downstream_all} in \cref{app:results} shows analogous results for mask matching and for the three other downstream models---these results are broadly similar, except that correlations with ARI tend to be stronger when using mask matching (perhaps unsurprisingly) or larger downstream models.


\textbf{Summary:}
Models that accurately segment objects allow for good downstream prediction performance.
Despite often having an advantage, distributed representations generally perform worse,
but not always significantly: the information is present but less easily accessible. 
The ARI is consistently correlated with downstream performance, and is therefore useful for model selection when masks are available. The MSE can be a practical unsupervised alternative on these datasets, but it may be less robust on complex textures.


{
\spacedstyle{-10}
\subsection{Generalization with one OOD object (Hypothesis 2)}
\label{sec:results/hyp2}
}
To test Hypothesis 2, we construct settings where a single object is OOD and the others are ID. We change the object style with neural style transfer, change the color of one object at random (only in CLEVR, Tetrominoes, and Shapestacks), or introduce a new shape (only in Multi-dSprites).
The unsupervised models are always trained on the original datasets. Then we train downstream models to predict the object properties from the learned representations. 
We consider two scenarios for this task: (1)~train the predictors on the original datasets and test them on the variants with a modified object, (2)~train and test the predictors on each variant.
In both cases, we test the predictors on representations that might be inaccurate, because the representation function (encoder) is OOD. However, since in case (2) the downstream model is \emph{trained} under distribution shift, this experiment quantifies the extent to which the representation can still be used by a downstream task that is allowed to adapt to the shift---although the representation might no longer represent objects faithfully, it could still contain useful information.\looseness=-1



    \begin{figure*}
        \centering
        \captionsetup{belowskip=0pt}
        \includegraphics[width=\linewidth,trim={4pt 8pt 4pt 8pt}]{figures/ood/metrics/only_ARI_ood_object/barplots_ARI}
        \caption{Effect of single-object distribution shifts on the ARI. Medians and 95\% confidence intervals with 10 random seeds.}
        \label{fig:main_text/ood/metrics/generalization_compositional/box_plots}
    \end{figure*}

For {Desideratum 1}, we observe in \cref{fig:main_text/ood/metrics/generalization_compositional/box_plots} that the models are generally robust to distribution shifts affecting a single object.
Introducing a new color or a new shape typically does not affect segmentation quality (but note Slot Attention on Tetrominoes), while changing the texture of an object via neural style transfer leads to a moderate drop in ARI in some cases.
In \cref{fig:ood/metrics/generalization_compositional/box_plots} (\cref{app:results}) we observe that SC and mSC show a compatible but less pronounced trend, while the MSE more closely mirrors the ARI. 
We conclude that the encoder is still partially able to separate objects when one object undergoes a distribution shift at test time.

    \begin{figure}
        \centering
        \captionsetup{belowskip=-10pt}
        \includegraphics[width=\linewidth,trim={0pt 5pt 0pt 8pt}]{figures/ood/downstream/slotted/match_loss/main_text_scatter_object_MLP1}
        \caption{\textbf{ID vs OOD downstream performance} with single-object distribution shifts. All datasets, models, and object properties are shown. Metrics: accuracy for categorical attributes, $R^2$\xspace for numerical attributes. The downstream model (an MLP with one hidden layer) is tested zero-shot out-of-distribution (left) or retrained after the distribution shift has occurred (right).}
        \label{fig:main_text/ood/downstream/generalization_compositional/scatter}
    \end{figure}

For {Desideratum 2}, we observe in \cref{fig:main_text/ood/downstream/generalization_compositional/scatter} (left) that property prediction performance for objects that underwent distribution shifts (color, shape, or texture) is often significantly worse than in the original dataset, whereas the prediction of ID objects is largely unaffected.
This is in agreement with Hypothesis 2: changes to one object do not affect the representation of other objects, even when these objects are OOD.
Extensive results, including further splits and all downstream models, are shown in \cref{fig:ood/downstream/slotted/match_loss/object/retrain_False} in \cref{app:results}.
On the right plot in \cref{fig:main_text/ood/downstream/generalization_compositional/scatter}, we observe that retraining the downstream models after the distribution shifts does not lead to significant improvements. This suggests that the shifts introduced here negatively affect not only the downstream model, but also the representation itself.
This result also holds with different downstream models and with mask matching (see \cref{fig:ood/downstream/slotted/match_loss/object/retrain_True,fig:ood/downstream/slotted/match_mask/object/retrain_True}). 
While in principle we observe a similar trend for VAEs (see e.g. \cref{fig:ood/downstream/vae/match_loss/object/retrain_False,fig:ood/downstream/vae/match_loss/object/retrain_True} in \cref{app:results}), their performance is often too close to the respective baseline (\cref{fig:indistrib/downstream/barplots_loss_linear_main}) for a definitive conclusion to be drawn.\looseness=-1

\textbf{Summary:}
The models are generally robust to distribution shifts affecting a single object.
Downstream prediction is largely unaffected for ID objects, but may be severely affected for OOD objects.
Finally, there seems to be no clear benefit in retraining downstream models after the shifts, indicating that the deteriorated representations cannot easily be adjusted \emph{post hoc}.


\subsection{Robustness to global shifts (Hypothesis 3)}
\label{sec:results/hyp3}
Finally, we investigate the robustness of object-centric models to transformations changing the global properties of a scene at test time.
Here, we consider variants of the datasets with occlusions, cropping, or more objects (only on CLEVR).
We train downstream predictors on the original datasets and report their test performance on the dataset variants with global shifts. As before, we also report results of downstream models retrained on the OOD datasets.

    \begin{figure*}
        \centering
        \captionsetup{belowskip=-4pt}
        \includegraphics[width=\linewidth,trim={4pt 5pt 4pt 8pt}]{figures/ood/metrics/only_ARI_ood_global/barplots_ARI}
        \caption{Effect of distribution shifts on global scene properties on the ARI. Medians and 95\% confidence intervals with 10 seeds.}
        \label{fig:figures/ood/metrics/only_ARI_ood_global/barplots_ARI}
    \end{figure*}

For {Desideratum 1}, \cref{fig:figures/ood/metrics/only_ARI_ood_global/barplots_ARI} shows that segmentation quality is generally only marginally affected by occlusion, but cropping often leads to a significant degradation. In CLEVR, the effect on the ARI of increasing the number of objects is comparable to the effect of occlusions, which suggests that learning about objects is useful for this type of systematic generalization.
These trends persist when considering SC and mSC, but appear less pronounced and less consistent across datasets (see \cref{fig:ood/metrics/generalization_global/box_plots} in \cref{app:results} for detailed results).
As might be expected, when the number of objects is increased in CLEVR, the MSE increases more conspicuously for VAEs than for object-centric models (\cref{fig:ood/metrics/generalization_global/box_plots}, bottom left), likely due to their explicit modeling of objects. However, \cref{fig:ood_visualizations_clevr} shows that VAEs may, in fact, generalize relatively well to an unseen number of objects, although not nearly as well as some object-centric models. 

    \begin{figure}
        \centering
        \captionsetup{belowskip=-10pt}
        \includegraphics[width=\linewidth]{figures/ood/downstream/slotted/match_loss/main_text_scatter_global_MLP1}
        \caption{\textbf{ID vs OOD downstream performance} with global distribution shifts. All datasets, models, and object properties are shown. Metrics: accuracy for categorical attributes, $R^2$\xspace for numerical attributes. The downstream model (an MLP with one hidden layer) is tested zero-shot out-of-distribution (left) or retrained after the distribution shift has occurred (right).}
        \label{fig:main_text/ood/downstream/generalization_global/scatter}
    \end{figure}
    
For {Desideratum 2}, we train a downstream model on the original dataset and test it under global distribution shifts. These shifts generally have a negative effect on downstream property prediction (\cref{fig:main_text/ood/downstream/generalization_global/scatter}, left), although this is comparable to the effect on OOD objects when only one object is OOD. This is in agreement with the observation made in \cref{sec:results/hyp2} that these shifts negatively affect the representation, which is no longer accurate because the encoder is OOD (cf. the ``OOD2'' scenario in \citet{dittadi2020transfer}).
When retraining the downstream models on the OOD datasets while keeping the representation frozen, the performance improves slightly but does not reach the corresponding results on the training distribution (\cref{fig:main_text/ood/downstream/generalization_global/scatter}, right), as in \cref{sec:results/hyp2}. 
These observations also hold for different downstream models and with mask matching (\cref{fig:ood/downstream/slotted/match_loss/global/retrain_False,fig:ood/downstream/slotted/match_mask/global/retrain_False,fig:ood/downstream/slotted/match_loss/global/retrain_True,fig:ood/downstream/slotted/match_mask/global/retrain_True}), as well as for distributed representations (see, e.g., \cref{fig:ood/downstream/vae/match_loss/global/retrain_False/scatter}) although with similar caveats as in \cref{sec:results/hyp2}.

\textbf{Summary:}
The impact of global distribution shifts on the segmentation capability of object-centric models depends on the chosen shift; e.g., cropping consistently has a significant effect. Moreover, the usefulness for downstream tasks decreases substantially in many cases, and the performance of downstream prediction models cannot be satisfactorily recovered by retraining them.





\section{Other related work}

Recent years have seen a number of systematic studies on disentangled representations~\citep{locatello2018challenging,locatello2019fairness,van2019disentangled,trauble2020independence}, some of which focusing on their effect on generalization~\citep{gondal2019transfer,dittadi2020transfer,montero2021the,pmlr-v89-esmaeili19a,traeuble2022role}.
In the context of object-centric learning, \citet{engelcke2020reconstruction} investigate their reconstruction bottlenecks to understand how these models can separate objects from the input in an unsupervised manner. In contrast, we specifically test some key implications of learning object-centric representations.

Slot-based object-centric models can be classified according to their approach to separating the objects at a representational level~\cite{greff2020binding}.
In models that use \emph{instance slots}~\cite{greff_tagger_2016,greff2017neural,van2018relational,greff2019multi,locatello2020object,lowe2020learning, huang2020better,le2011learning,goyal2019recurrent,van2020investigating,NEURIPS2019_32bbf7b2,yang2020learning,kipf2019contrastive,racah2020slot,kipf2021conditional}, each slot is used to represent a different part of the input. This introduces a routing problem, because all slots are identical but they cannot all represent the same object, so a mechanism needs to be introduced to allow slots to communicate with each other.
In models based on \emph{sequential slots}~\cite{eslami2016air,stelzner2019supair,kosiorek2018sequential,stove,burgess2019monet,engelcke2020genesis,engelcke2021genesisv2,vonkugelgen2020econ}, the representational slots are computed in a sequential fashion, which solves the routing problem and allows to dynamically change the number of slots, but introduces dependencies between slots.
In models based on \emph{spatial slots}~\cite{nash2017multi,crawford2019spatially,jiang2020scalor,crawford2020exploiting,lin2020space,deng2021generative,lin2020improving,dittadi2019lavae}, a spatial coordinate is associated with each slot, introducing a dependency between slot and spatial location. 
In this work, we focus on four scene-mixture models as representative examples of approaches based on instance slots (Slot Attention), sequential slots (MONet and GENESIS), and spatial slots (SPACE). 

\section{Conclusions}

In this paper, we identify three key hypotheses in object-centric representation learning: learning about objects is useful for downstream tasks, it facilitates strong generalization, and it improves overall robustness to distribution shifts.
To investigate these hypotheses,
we re-implement and systematically evaluate four state-of-the-art unsupervised object-centric learners on a suite of five common multi-object datasets.
We find that object-centric representations are generally useful for downstream object property prediction, and downstream performance is strongly correlated with segmentation quality and reconstruction error.
Regarding generalization, we observe that when a single object undergoes distribution shifts the overall segmentation quality and downstream performance for in-distribution objects is largely unaffected.
Finally, we find that object-centric models can still relatively robustly separate objects even under global distribution shifts. However, this may depend on the specific shift, and downstream performance appears to be more severely affected.

An interesting avenue for future work is to continue our systematic investigation of object-centric learning on more complex data with diverse textures, as well as a wide range of more challenging downstream tasks. 
Furthermore, it would be interesting to compare object-centric and non-object-centric models more fairly: while learning about objects offers clear advantages, the full potential of distributed representations in this context is still not entirely clear, particularly when scaling up datasets and models.
Finally, while we limit our study to unsupervised object discovery, it would be relevant to consider methods that leverage some form of supervision when learning about objects.
We believe our benchmarking library will facilitate progress along these and related lines of research.



\section*{Acknowledgements}
We would like to thank Thomas Brox, Dominik Janzing, Sergio Hernan Garrido Mejia, Thomas Kipf, and Frederik Träuble for useful comments and discussions, and the anonymous reviewers for valuable feedback.


