\section{Introduction}
Web-scale image diffusion models, such as OpenAI's DALL-E 2 \cite{dalle2}, have been one of the most exciting recent breakthroughs in machine learning. By training over hundreds of millions of image-caption pairs from the Web, these models learn a language-conditioned distribution over natural images, from which novel images can be generated given a text prompt. Large language models \cite{bert, gpt-3}, which are also trained on web-scale data, have recently been applied to robotics \cite{saycan} to enable generalisation of language-conditioned policies to novel language commands. Given these successes, in this paper we ask the following question: \textbf{Can web-scale text-to-image diffusion models, such as DALL-E, be exploited for real-world robotics?}

Since these models can generate realistic images of scenes, they must also understand how to arrange objects in a natural way. E.g. generating images with a ``kitchen" prompt would be more likely to display plates neatly placed on a table or a shelf, than scattered across a sofa or the floor. This leads to a clear application to robotics: predicting goal states for object rearrangement tasks, a canonical challenge in robotics \cite{rearrangement}. Manually aligning goal states with human values is brittle and cumbersome, so this is where web-scale learning offers a powerful solution: we can implicitly model natural distributions of objects in a scalable, unsupervised manner.

\begin{figure}[h!]
    \centerline{\includegraphics[width=0.95\linewidth]{figures/teaser.pdf}}
    \caption{The robot prompts DALL-E with the list of objects it detects, which generates an image with a human-like arrangement of those objects. The robot then rearranges the scene via pick-and-place to match the generated image.}
    \label{fig:teaser}
\end{figure}

In this paper, we propose DALL-E-Bot, the first method to explore web-scale image diffusion models for robotics. We design a framework which enables DALL-E to be used to predict a goal state for object rearrangement, given an image of an initial, disorganised scene, as shown in Fig.~\ref{fig:teaser}. The pipeline converts the initial image into a text caption, which is then passed into DALL-E to generate a new image, from which we then obtain goal poses for each object. Note that we use publicly-available DALL-E as it is, without requiring any further data collection or training. This is important: it allows for zero-shot autonomous rearrangement, going beyond prior work which often requires collecting examples of desirable arrangements and training a model specifically for those scenes \cite{jiang-human-context,neatnet,structformer,adaptable-planners}.

With real-world experiments across three everyday re-arrangement tasks, we studied two abilities of DALL-E-Bot: (1) to re-arrange a full scene by generating an entire image, and (2) to precisely place a single item into an existing scene using DALL-E's inpainting feature. By evaluating with human users, we found that DALL-E-Bot arranges scenes in a more appealing way to humans than simple baselines, and we also explored the impact of our design choices with ablation studies. Finally, we present the limitations of current web-scale diffusion models for robotics, and propose recommendations to the text-to-image community for better aligning these models with robotics applications.


\section{Related Work}

\subsection{Predicting Goal Arrangements}

We now highlight prior approaches to predicting goal poses for rearrangement tasks. Some methods view the prediction of goal poses as a classification problem, by choosing from a set of discrete options for an object's placement. For house-scale rearrangement, a pre-trained language model can be used to predict goal receptacles such as tables \cite{housekeep}, and out-of-place objects can be detected automatically \cite{tidee}. At a room level, the correct drawer or shelf can be classified \cite{organisational-principles-classification}, taking preferences into account \cite{abdo-shelves}. Lower-level prediction from a dense set of goal poses can be achieved with a graph neural network \cite{efficient-interpretable} or a preference-aware transformer \cite{adaptable-planners}. Our framework uses high-resolution images of how objects should be placed, thus not requiring a set of discrete options to be pre-defined, and providing more precise guidance than with language-based methods.

Learning to predict continuous object poses can be done using example arrangements by encoding spatial preferences with a graph VAE \cite{neatnet}, or using an autoregressive language-conditioned transformer \cite{structformer}, or by learning gradient fields \cite{targf}. Other methods use full demonstrations \cite{cliport,transporters}, or leverage priors such as human pose context \cite{jiang-human-context}. When the goal image is given, rearrangement is possible even with unknown objects \cite{ifor}. However, unlike these works, our proposed framework does not require collecting and training on a dataset of rearrangement examples, which often restricts these methods to a specific set of objects and scenes. Instead, exploiting existing web-scale image diffusion models enables zero-shot, autonomous rearrangement.

\subsection{Web-Scale Diffusion Models}

Generating images with web-scale diffusion models such as DALL-E is at the heart of our method. Diffusion models \cite{sohl2015} are trained to reverse a single step of added noise to a data sample. By starting from random noise and iteratively running many of these small, learned denoising steps, this can generate a sample from the learned distribution of data. These models have been used to generate images \cite{ddpms,improved-ddpms,beat-gans}, text-conditioned images \cite{dalle2,glide,imagen,stable-diffusion}, robot trajectories \cite{levine-diffusion}, and audio waves \cite{audio}. We use DALL-E 2 \cite{dalle2} in this work, although our framework could be used  with other text-to-image models.

\section{Method}

\begin{figure*}[h]
    \centerline{\includegraphics[width=1\linewidth]{figures/overview_3.pdf}}
    \caption{DALL-E-Bot creates a human-like arrangement of objects in the scene using a modular approach. First, the initial observation image is converted into a per-object description consisting of a segmentation mask, an object caption, and a CLIP visual feature vector. Next, a text prompt is constructed describing the objects in the scene and is passed into DALL-E to create a goal image for the rearrangement task, where the objects are arranged in a human-like way. Then, the objects in the initial and generated images are matched using their CLIP visual features, and their poses are estimated by aligning their segmentation masks. Finally, a robot rearranges the scene based on the estimated poses to realise the predicted arrangement.
   
    }
    \label{fig:pipeline}
\end{figure*}

\subsection{Overview}\label{ss:overview}
We address the problem of predicting a goal pose for each object in a scene, such that objects can then be rearranged in an appealing way. We propose to predict goal poses zero-shot from a single RGB image $I_I$ of the initial scene.

To achieve this, we propose a modular pipeline shown in Fig.~\ref{fig:pipeline}.  At the heart of our method is a web-scale image diffusion model DALL-E 2 \cite{dalle2}, which, given a text description $\ell$ of a scene, generates high-quality variations of images $\{I_G^i\}_{i=0}^N$, each representing a plausible arrangement of objects.
We convert an initial RGB observation into a more relevant object-level representation to individually reason about the objects in the scene. This representation consists of text captions of crops of individual objects $c_i$ (used to construct a text prompt $\ell$) together with their segmentation masks $M_i$, and visual-semantic feature vector $v_i$ acquired using the CLIP model \cite{clip}.
We also convert generated images into object-level representations and select the image that has the same number of objects as the initial scene, and best matches the objects in the initial scene semantically.
Using an Iterative Closest Point (ICP) \cite{icp} algorithm in image space, we then register corresponding segmentation masks to obtain transformations that need to be applied to the individual objects to achieve the desired arrangement. Finally, we convert these transformations from image space to Cartesian space using a depth camera observation, and deploy a real Franka Emika Panda robot equipped with a compliant suction gripper to re-arrange the scene.

\subsection{Object-level Representation}\label{ss:object-detection}
 
To reason about the poses of individual objects in the observed scene, we need to convert the initial RGB observation into a more functional, object-level representation. We use the Mask R-CNN model \cite{mask-r-cnn} from the Detectron2 library \cite{detectron2} to detect objects in an image and generate segmentation masks $\{M_i\}^n_i$. This model was pre-trained on the LVIS dataset \cite{lvis}, which has $1200$ object classes, being more than sufficient for many rearrangement tasks. For each object, Mask R-CNN provides us with a bounding box, a segmentation masks, and a class label. However, we found that whilst the bounding box and segmentation mask predictions are usually high quality and can be used for pose estimation (described in Section~\ref{ss:pose-estimation}), the predicted class labels are often incorrect due to the large number of classes in the training dataset.

As we are using labels of objects in the scene (described in Section~\ref{ss:prompt-generation}) to construct a prompt for an image diffusion model, it is crucial for these labels to be accurate. Therefore, instead of directly using predicted object class labels, we pass RGB crops around each object's bounding box through an OFA image-to-text captioning model \cite{ofa}, and acquire a text description of the objects in the initial scene observation, $\{c_i\}^n_i$. Generally, this approach allows us to more accurately predict object class labels and go beyond the objects in Mask R-CNN's training distribution, and even obtain their visual characteristics such as colour or shape. Finally, we also pass each object crop through a CLIP visual model \cite{clip}, giving each object a 512-dimensional visual-semantic feature vector $v_i$. These features will be used later for matching objects between the initial scene image and the generated image.

In summary, by the end of this stage, we have converted an initial scene RGB observation $I_I$ into an object-level representation of the scene $\{M_i, c_i, v_i\}^n_i$, which represents each object by a segmentation mask, a text caption, and a semantic feature vector.

\subsection{Goal Image Generation}\label{ss:prompt-generation}

Our method relies on the ability to generate images of natural and human-like arrangements, given their text descriptions. To this end, we exploit recent advances in text-to-image generation and web-scale diffusion models, by using the publicly-available DALL-E 2 \cite{dalle2} model from OpenAI. This has been trained on a vast number of image-caption pairs from the Web, and represents the conditional distribution $p_\theta(I_G |y, I_M)$. Here, $I_G$ is an image generated by the model, $\ell$ is a text prompt, and $I_M$ is an image mask that can be used to prevent the model from changing the values of certain pixels in the image. A large portion of distribution $p_\theta$ represents images with scenes arranged by humans in a natural and usable way. Therefore, by sampling from this distribution, we can generate images representing our desired scenes and realise the arrangements by 
moving objects to the same poses as in the generated images. Additionally, the ability to condition this distribution on an image mask $I_M$ allows us to address scenarios where not all objects in the scene need to be, or can be, moved by the robot.

To generate an image using DALL-E, we first need to construct a text prompt $\ell$ describing the desired scene. To this end, we use object captions from our object-level representation. Although full captions, including their visual characteristics, could be used to generate images with identical objects in the scene, in this work, we only use the nouns describing the object's class and leave including visual characteristics for future work. Thus, we extract the class of each object from the caption of its object crop, i.e. we extract ``apple'' from ``a red apple on a wooden table''. We do this by passing the object captions through the Part-of-Speech tagging model  \cite{pos-tagging} from the Flair NLP library \cite{flair}, which tags each word as a noun, a verb, etc. From this list of classes, we construct a prompt that makes minimal assumptions about the scene, to allow DALL-E to arrange it in the most natural way. In this work, our experiments are based on tabletop scenes, with observations captured by a camera mounted on a robot's wrist pointing downwards towards the table. Therefore, we added a ``top-down'' phrase to the prompt to better align the initial and generated images. An example prompt we use would be ``A fork, a knife, a plate, and a spoon, top-down'' (as in Fig.~\ref{fig:pipeline}).

We use DALL-E's ability to condition distribution $p_\theta$ on image masks in three ways. First, if there are objects in the scene that a robot is not allowed to move, we add their contours to $I_M$. This prevents DALL-E from generating these objects in different poses while still allowing for other objects to be placed on top or in them (e.g. a basket can not be moved, but other objects can be placed inside it). Second, we add a mask of the tabletop's edges in our scene to $I_M$ to visually ground the generated images. This prevents objects from being placed on the edge of the generated image. Additionally, we found throughout our experiments that it incentivises DALL-E to create objects of appropriate sizes. Third, we subtract segmentation masks of all the movable objects from $I_M$, with enlarged masks to avoid any shadows remaining. Avoiding shadows is essential, as if DALL-E sees any shadows of objects in their original poses, it will generate objects in the same poses to fit with those shadows, hindering its ability to generate novel and diverse images.

Using the prompt $\ell$ and the image mask $I_M$, we sample a batch of images from the conditional distribution $p_\theta(I_G |y, I_M)$, representing the text-to-image model. We do so using an automated script and OpenAI's web API.

\subsection{Image Selection \& Object Matching}\label{ss:object-matching}

In the batch of generated images (four per batch), not all will be desirable for the rearrangement task; some may have artefacts which make object detection difficult, others may include extra objects that were not part of the text prompt, etc. Therefore, we need to select the generated image $I_G$ which best matches the real-world initial image $I_I$.

For each generated image, we obtain segmentation masks and a CLIP semantic feature vector for each object, using the procedure in Section~\ref{ss:object-detection}. Then, we filter out generated images where the number of objects is different to the initial scene. If there are no images with the same number of objects in the generated images, we sample another batch with DALL-E. We then match objects between the generated image and initial image. This is non-trivial since the generated objects are different instances to the real objects, often with a very different appearance. Inspired by \cite{vgg-object-matching}, we compute a similarity score between any two objects (one from $I_I$, and one from $I_G$) using the cosine similarity between their CLIP feature vectors. Since greedy matching is not guaranteed to yield optimal results in general, we use the Hungarian Matching algorithm \cite{hungarian-matching} to compute an assignment of each object in the initial image to an object in the generated image, such that the total similarity score is maximised. Then, we select the generated image $I_G$ which has the best overall score with the initial image $I_I$. This image contains the most similar set of objects to the real scene, and therefore gives the best opportunity for rearranging the real scene.

\subsection{Object Pose Estimation}\label{ss:pose-estimation}

For each object in the initial image, we now know its segmentation mask in the initial image and the corresponding segmentation mask in the generated image. By aligning these masks, we can estimate a transformation from the initial pose (in the initial image) to the goal pose (in the generated image). We rescale each initial segmentation mask, such that the dimensions of its bounding box equal those in the generated image, and then use the Iterative Closest Point algorithm \cite{icp} to align the two masks, taking each pixel to be a point. This gives us a 3-DoF $(x, y, \theta)$ transformation in pixel space between the initial and goal pose.

The scale of the objects in the generated image can be significantly different to the initial image, leading to the predicted arrangement resulting in collisions, or being unnaturally spaced out. Therefore, we adjust the poses of the objects in the scene based on the size difference of objects between the initial and generated images. We move all the objects closer to or further from the one with the minimum cumulative distance to all the other ones. Additionally, if collisions occur in the found arrangement, we move objects away from the central one until there are no more collisions.

Next, we use a wrist-mounted depth camera to project the pixel-space poses into 3D space on the tabletop, to obtain a transformation for each object which would move it from the initial real-world pose to the goal real-world pose. Finally, the robot executes these transformations by performing a sequence of pick-and-place operations using a suction gripper. We also designed a simple planner which first moves objects that would cause collisions into intermediate poses to the side, before later moving them to their goal poses.

\begin{figure*}[h]
    \centerline{\includegraphics[width=1\linewidth]{figures/Generated_3.pdf}}
    \caption{Examples of scenes rearranged by the robot using different methods. Columns for the methods that use DALL-E generated images include both an image of the final arrangement (left) and a generated image (right).}
    \label{fig:Generated}
\end{figure*}

\section{Experiments}

\subsection{Zero-Shot Autonomous Rearrangement}\label{s:qual}

In our experiments, we evaluate the ability of our method to create human-like arrangements using both subjective (Section~\ref{s:qual}) and objective (Section~\ref{s:quant}) metrics. First, we explore the following question: \textbf{can DALL-E-Bot arrange a set of objects in a human-preferred way?}

We evaluate on 3 everyday tabletop rearrangement tasks: \textbf{dining}, \textbf{office}, and \textbf{fruit basket} (Fig.~\ref{fig:Generated}), where the robot should arrange the objects in a human-like way in each scene. The dining scene involves four objects (a knife, a fork, a spoon, and a plate). The office scene includes a stationary object which the robot can see but is not allowed to move (an iPad), and three movable objects (a keyboard, a mouse, and a mug). The fruit basket scene contains two apples and an orange, as well as a stationary basket.

In real-world applications of DALL-E-Bot, users would only see the outcome of the real-world rearrangement, which would include all the errors that might accumulate through the pipeline. To simulate this experience for our evaluation, we create the predicted arrangements using a real-world Franka Emika robot equipped with a compliant suction gripper. Then, we record the outcome as an RGB image of a tabletop scene, using a camera on the wrist of the robot.

Since DALL-E-Bot is the first method to predict precise goal poses for rearrangement in a way which is zero-shot and autonomous, we designed two baselines which are also zero-shot for a fair comparison. The \method{Rand-No-Coll} baseline places objects randomly in the environment while ensuring they do not overlap. The \method{Geometric} baseline puts all the objects evenly in a straight line such that they are not colliding, and aligns the objects so that they are parallel using their bounding boxes. In addition, we compare our method \method{DALL-E-Bot} to two ablated variants. \method{Abl-Mask-RCNN} uses labels predicted by Mask R-CNN to construct the text prompt for DALL-E, instead of using the captioning model, to understand the sensitivity to incorrect object classifications. \method{Abl-No-VG} (no visual grounding) does not use the edge crops around the table, to understand the effect of the visual grounding, which was described in Section~\ref{ss:prompt-generation}.

Since we aim to create arrangements which are appealing to humans, the most appropriate evaluation is to directly ask humans for feedback. Therefore, we showed human users images of the final real-world scene created by the robot, and asked them the following question: \textit{``If the robot made this arrangement for you, how happy would you be?''}. The user provided a score for each method on a Likert Scale from 1 (very unhappy) to 10 (very happy), while being shown arrangements made by each method side-by-side in a web-based questionnaire. We recruited 17 users with ages ranging from 20 to 71  (ten male, six female, and one preferred not to say). For each of the 3 scenes, and each of the 5 methods, the robot created 5 different rearrangements, where each re-arrangement had a different random initialisation, for a total of 1275 scores. All users were shown the same images. Examples of these images are in Fig.~\ref{fig:Generated}.

The results of this user study are in Table~\ref{tab:qual}, and videos are available at: \textcolor{blue}{\href{https://www.robot-learning.uk/dall-e-bot}{https://www.robot-learning.uk/dall-e-bot}}. \method{DALL-E-Bot} received high user scores, showing that it can create satisfactory arrangements in a zero-shot fashion without requiring task-specific training. It beats the heuristic baselines, showing that \textbf{users do value semantic correctness for arranging scenes beyond just simple geometric alignment}, which justifies the use of web-scale learning for capturing these subtle semantic arrangement rules. This is especially evident in the dining scene, where DALL-E recognises the semantic structure which can be created from those objects. The \method{Abl-Mask-RCNN} \cite{mask-r-cnn} ablation baseline performs poorly in the dining scene, since it often predicts the wrong classes for the objects, e.g. ``frisbee'' instead of ``plate''. This makes the prompt to DALL-E unusual, resulting in unnatural images being generated. This justifies our use of a dedicated captioning model trained on a larger-scale dataset, rather than being restricted by Mask R-CNN classes. \method{DALL-E-Bot} also outperforms the \method{Abl-No-VG} ablation baseline on some scenes, showing that there is some value in including the edges of the scene in the inpainting mask, which makes DALL-E less likely to generate partial objects at the edge of the image. Qualitative results in Fig.~\ref{fig:Generated} also show that the realised arrangements for \method{DALL-E-Bot} closely match the diffusion-generated images for most objects, despite the differences between object instances. Note that although the generated plate is square, the real plate is round, and the objects are correctly matched using CLIP semantic features.

\begin{table}[h]
    \centering
    \begin{tabular}{|l|c|c|c|c|c|}
        \hline
        {\scriptsize Method} & {\scriptsize Dining Scene} & {\scriptsize Office Scene} & {\scriptsize Fruit Scene} & {\scriptsize Mean} \\
        \hline
        {\scriptsize Rand-No-Coll} & \mean{2.21}{2.24} & \mean{3.42}{2.47} & \mean{3.26}{2.00} & \justmean{2.96} \\
        \hline
        {\scriptsize Geometric} & \mean{4.14}{1.80} & \mean{3.47}{2.23} & \mean{2.85}{1.62} & \justmean{3.49} \\
        \hline
        {\scriptsize Abl-Mask-RCNN} & \mean{3.99}{2.60} & \mean{7.53}{2.04} & \mean{5.86}{3.40} & \justmean{5.79} \\
        \hline
        {\scriptsize Abl-No-VG} & \mean{\textbf{7.45}}{1.81} & \mean{7.13}{1.99} & \mean{5.48}{3.56} & \justmean{6.69} \\
        \hline
        {\scriptsize DALL-E-Bot} & \mean{7.14}{2.13} & \mean{\textbf{7.71}}{2.01} & \mean{\textbf{9.55}}{0.90} & \justmean{\textbf{8.13}} \\
        \hline
    \end{tabular}
    \vspace{0.2cm}
    \caption{User scores for arrangements made by each method. Higher numbers are better. Each cell shows the mean and standard deviation across all users and scene initialisations. The best result for each column is in bold.}
    \label{tab:qual}
\end{table}


\subsection{Placing Missing Objects with Inpainting}\label{s:quant}

In the next experiment, we use objective metrics to answer the following question: \textbf{can DALL-E-Bot precisely complete an arrangement which was partially made by a human?} For this, we ask DALL-E-Bot to find a suitable pose for an object that has been masked out from a human-made scene, while the other objects are kept fixed. We study this using the dining scene, because it has the most rigid structure for semantic correctness and thus is most suitable for quantitative, objective evaluation. To create these scenes initially, we recruited ten users (both left and right-handed) and asked them the following: ``\textit{Imagine you are sitting down here for dinner. Can you please arrange these objects so that you are happy with the arrangement?}''. As there can be multiple suitable poses for any single object in the scene, for each of the four objects we asked the users to provide any alternative poses that they would still be happy with, while keeping other objects fixed. 

Given the image of the arrangement made by a user, we mask out everything except the fixed objects. This means that DALL-E cannot change the pixels belonging to fixed objects. The method must then predict the pose of the missing object. DALL-E-Bot does this by inpainting the missing object somewhere in the image. For a given user, the predicted pose for the missing object is compared against the actual pose in their arrangement. This is done by aligning two segmentation masks of the missing object, one from the actual scene and one at the predicted pose. Since this is for two poses of exactly the same object instance, we find the alignment is highly accurate and can be used to estimate the error between the actual and predicted pose. From this transformation, we take the orientation and distance errors projected into the workspace as our metrics. This is repeated for every object as the missing object, and across all the users.

We compare our method to two zero-shot heuristic baselines, \method{Rand-No-Coll} and \method{Geometric}. \method{Rand-No-Coll} places the missing object randomly within the bounds of the image, ensuring it does not collide with the fixed objects. \method{Geometric} first finds a line defined by centroids of segmentation maps of two fixed objects. Then it places the considered object on that line such that it is as close to the fixed objects as possible, does not collide with them, and its orientation is aligned with the orientation of the closest object.

We compare the predicted pose against each of the acceptable poses provided by the user, and report the position and orientation errors from the closest acceptable pose in Table~\ref{tab:quant}. The distribution of acceptable poses is multimodal. Therefore, we present the median error across all users, which is less dominated by outliers than the mean and is a more informative representation of the aggregate performance.

\begin{table}[h]
    \scriptsize
    \centering
\begin{tabular}{|l|c|c|c|c|}
\hline
                    & Fork          & Plate         & Spoon         & Knife        \\ \hline
Method              & cm / deg      & cm / deg      & cm / deg      & cm / deg     \\ \hline
Rand-No-Coll        & 25.85 / 70.32 & 10.78 / - & 27.47 / 42.56 & 23.51 / 99.32 \\ \hline
Geometric & 15.59 / 40.57 & 2.29 / -  & 23.83 / 86.11 & 11.58 / \textbf{1.47} \\ \hline
DALL-E-Bot          & \textbf{4.95} / \textbf{1.26}   & \textbf{1.28} / -  & \textbf{2.13} / \textbf{2.72}   & \textbf{2.1} / 3.27   \\ \hline
\end{tabular}
    \vspace{0.2cm}
    \caption{Position and orientation errors between predicted and user preferred object poses. Each cell shows the median across all users. The best result for each column is in bold.}
    \label{tab:quant}
\end{table}

DALL-E-Bot outperforms the baselines, and is able to accurately place the missing objects with a low average error across the different users. This implies that it is successfully conditioning on the placement of the other objects in the scene using inpainting, and that the human and robot can create an arrangement collaboratively. 

\section{Discussion}

\subsection{Limitations}

\textbf{Robustness of cross-instance object alignment.} To estimate each object's transformation from the initial image to the generated image, we align the two binary segmentation masks with ICP. However, this is difficult for symmetric objects such as knives or keyboards, which can be aligned with 180-degree error. This can be alleviated by aligning semantic feature maps instead of binary segmentation masks.

\textbf{Top-down pick-and-place}. Our experiments focus on 3-DoF rearrangement tasks, which is sufficient for many everyday tasks. However, future work can extend to 6-DoF poses with more complex interactions, e.g. to stack shelves. This could draw from recent works on collision-aware manipulation \cite{where-to-start} and learning of skills beyond grasping \cite{johns2021coarse-to-fine}.

\textbf{Object-centric framework}. Our method reasons about pose transformations to solve everyday rearrangement tasks. Thus, as individual components (e.g. segmentation, pose estimation) improve, overall performance will also improve. However, some tasks, such as folding deformable fabrics or sweeping small particles, are not within this method's scope.

\subsection{Future Work}

\textbf{Personal preferences}. If objects placed by users are visible in the inpainting mask, DALL-E may implicitly condition images on inferred preferences (e.g. left/right-handedness). Future work could extend to conditioning on preferences inferred in previous scenes arranged by users \cite{neatnet}.

\textbf{Prompt engineering}. Adding terms such as ``neat, precise, ordered, geometric'' for the dining scene improved the apparent neatness of the generated image. As found in other works \cite{lets-think}, there is significant scope to explore this further.

\textbf{Language-conditioned rearrangement}. Instructions provided by a human could be added to the text prompt, e.g. ``plates stacked'' vs ``plates laid out''. Prior work shows that specifying spatial relationships such as ``left of'' is difficult \cite{dalle-relations}, but future diffusion models could overcome this.

\subsection{Recommendations to the Text-To-Image Community}

As this is the first work to explore web-scale diffusion models for robotics, we now provide our findings on the strengths and limitations of existing diffusion models for robotics, with the aim of guiding the text-to-image community when targetting applications to robotics.

\textbf{Everyday scenes in training datasets}. We found that Stable Diffusion~\cite{stable-diffusion} trained on LAION-Aesthetics is proficient at generating aesthetically pleasing images, but DALL-E is better suited for robotic applications, because the training dataset includes ``ordinary" images. Taking this further, training \textit{only} on everyday scenes could be important for robotics.

\textbf{Batch sampling and rejection}. Many of the generated images are not suitable as goal images (e.g. wrong number of objects). We found that the best images came from sampling larger batches and designing an algorithm to reject invalid samples. Diffusion model systems and tools which allow for automated rejection based on the text prompt could be useful.

\textbf{Visual conditioning}. Rather than just conditioning on language descriptions of objects to be generated, it would be useful to condition on image features of the real objects, but still allow the diffusion model to arrange them differently. This would help with matching between the initial and generated images. Textual inversion~\cite{textual-inversion} looks promising for making the generated objects better match the real instances.

\textbf{Outpainting}. We found that objects are frequently generated at the image edge and only partially in view, making pose estimation more difficult. Outpainting, available in some text-to-image models including DALL-E, can help with this.

\textbf{Guidance scale.} Some interfaces (e.g. Stable Diffusion~\cite{stable-diffusion}) allow a trade-off between image realism and text prompt adherence. This is useful for robotics, since generating images that adhere to the text prompt is much more important than generating attractive or realistic images.


\subsection{Conclusions}

In this paper, we show for the first time that web-scale diffusion models like DALL-E have significant potential as ``imagination engines'' for robots, acting like an aesthetic prior for how to arrange objects in a human-like way. This allows for zero-shot, autonomous rearrangement, using DALL-E out-of-the-box, without requiring collecting datasets of example arrangements for specific scenes, and without any additional training. In other words, \textbf{our system gives web-scale diffusion models an embodiment to realise the scenes that they imagine}. Studies with human users showed that they are happy with the created arrangements for everyday rearrangement tasks, and that the inpainting feature of diffusion models is useful for conditioning on pre-placed objects. Web-scale diffusion models are a recent and active research frontier, and so we have also provided recommendations for further aligning these models with robotics. We believe that this is an exciting direction for the future of robot learning, as diffusion models continue to impress and inspire complementary research communities.

\clearpage
\addtolength{\textheight}{-12cm}  
                                 
                                 
                                 
                                 
                                 












\bibliographystyle{IEEEtran}
