\section{Introduction}\label{sec:intro}

Manipulating deformable objects is a long-standing challenge in robotics with a wide range of real-world applications. In contrast to rigid object manipulation, deformable object manipulation presents additional challenges due to more complex configuration spaces, dynamics, and sensing.

While several prior works have made progress largely centered around tasks with 1D (e.g.,  rope~\cite{nair_rope_2017}) or 2D (e.g.,  fabric~\cite{seita_fabrics_2020}) deformable structures, little prior work has addressed generalizable vision-based methods for manipulating 3D deformable structures such as the task of: ``insert objects into a bag, and then carry the bag away''.  These types of tasks are especially challenging with diverse goal conditioning: the goal states of deformable objects are not easily specified, for example, by compact pose representations.

In this work, we propose a new suite of benchmark tasks, called \textit{DeformableRavens}, to test manipulation of cables, fabrics, and bags spanning 1D, 2D, and 3D deformables. For several tasks in the benchmark, we propose to tackle them using novel goal-conditioned variants of Transporter Network~\cite{zeng_transporters_2020} architectures. 
Our experiments also significantly extend results of using Transporter Networks for deformable manipulation tasks --- while~\cite{zeng_transporters_2020} demonstrated one 1D deformable task, we show results on 12 tasks, including those with fabrics and bags.


\begin{figure}[t]
\center
\includegraphics[width=0.49\textwidth]{teaser_v02.png}
\caption{
Example of a trained Transporter Network policy in action on the \emph{bag-items-1} task (see Table~\ref{tab:tasks}). The setup involves a simulated UR5 robot arm, a green target zone, a drawstring-style bag, and a red cube. The starting configuration is shown in the top left frame. The robot, with suction cup, must grasp and sufficiently open the bag to insert a cube, then (bottom row) picks and pulls upwards, enclosing the cube. The robot concludes by bringing the bag with the cube in the target zone. We overlay arrows to indicate the movement of the robot's arm just before a given frame.
}
\vspace*{-15pt}
\label{fig:teaser}
\end{figure}




The main contributions of this paper are: (i) an open-source simulated benchmark, \textit{DeformableRavens}, with 12 tasks manipulating 1D, 2D, and 3D deformable objects to help accelerate research progress in robotic manipulation of deformables, (ii) end-to-end goal-conditioned Transporter Network architectures that learn vision-based multi-step manipulation of deformable 1D, 2D, and 3D structures, and (iii) experiments demonstrating that the proposed vision-based architectures are competitive with or superior to baselines that consume ground truth simulated pose and vertex information. We also discuss the shortcomings of the system, which point to interesting areas for future research. The project website contains supplementary material, including the appendix, code, data, and videos.



\begin{figure*}[t]
\center
\includegraphics[width=1.00\textwidth]{tasks_v06-min.png}
\caption{
The 12 tasks in the proposed \emph{DeformableRavens} benchmark (see Table~\ref{tab:tasks}) with suction cup gripper and deformable objects. \textbf{Top row}: (a) cable-ring, (b) cable-ring-notarget, (c) cable-shape, (d) cable-shape-notarget, (e) cable-line-notarget, (f) fabric-cover. \textbf{Bottom row}: (g) fabric-flat, (h) fabric-flat-notarget, (i) bag-alone-open, (j) bag-items-1, (k) bag-items-2, (l) bag-color-goal. Of the tasks shown, 4 are ``goal-conditioned,'' i.e.,  cable-shape-notarget, cable-line-notarget, fabric-flat-notarget, bag-color-goal. These use a separate goal image $\mathbf{o}_g$ for each episode to specify a success configuration. The other 8 tasks do not use goal images. Examples of successful configurations for each task are shown with overlaid (cropped) images to the top left.
}
\vspace*{-15pt}
\label{fig:tasks}
\end{figure*}


\section{Related Work}\label{sec:rw}


\subsection{Deformable Object Manipulation}

In this work, deformables~\cite{manip_deformable_survey_2018} refers to 1D structures such as ropes and cables (we use the terms interchangeably), 2D structures such as fabrics, and 3D structures such as bags.

Rope has been extensively explored in early robotics literature, such as for knot-tying~\cite{case_study_knots_1991,knot_planning_2003}. In recent years, learning-based approaches for manipulation have grown in popularity to improve the generalization of knot-tying and also to manipulate rope towards general target configurations~\cite{schulman_isrr_2013,nair_rope_2017,ZSVI_2018,priya_2020,state_estimation_LDO,wang_visual_planning_2019}. The approach we propose is focused primarily in manipulating cables to target locations, which may be specified by a target zone or a target image, and does not require modeling rope physics.

Much research on robotic fabric manipulation~\cite{grasp_centered_survey_2019,manip_deformable_survey_2018,corona_2018} uses bilateral robots and gravity to expose corners for ease of manipulation, which enables iteratively re-grasping the fabric's lowest hanging point~\cite{osawa_2007,kita_2009_iros,kita_2009_icra}. Subsequent work~\cite{maitin2010cloth,cusumano2011bringing,unfolding_rf_2014}, generalized to a wider variety of initial configurations of new fabrics. Other approaches focus on fabric smoothing.
For example, Sun~et~al.~\cite{heuristic_wrinkles_2014,cloth_icra_2015} attempt to detect and then pull perpendicular to the largest wrinkle. Another task of interest is wrapping rigid objects with fabric, which bridges the gap between manipulating 2D and 3D structures. Hayashi~et~al.~\cite{planning_wrapping_fabric_2017,fabric_wrapping_2014} present an approach for wrapping fabric around a cylinder using a bimanual robot.



Work on 3D deformable structures such as bags has been limited due to the complexities of manipulating bags. Early work focused on mechanical design of robots suitable for grasping~\cite{grasping_sacks_2005} or unloading~\cite{unloading_sacks_2008} sacks. Some work assumes that a sturdy, brown bag is already open for item insertion, as with a grocery checkout robot~\cite{checkout_robot_2011}, or uses reinforcement learning for zipping bags~\cite{contour_ziplock_2018} in constrained setups. 

These works generally focus on optimizing a specific problem in deformable object manipulation, often with task-specific algorithms or manipulators. In contrast, we propose a simulation setup with a single robot arm that applies to a wide range of tasks with 1D, 2D, and 3D deformables. In independent and concurrent work, Lin~et~al.~\cite{corl2020softgym}  propose \emph{SoftGym}, a set of environments with cables, fabrics, and liquids simulated using NVIDIA FleX, and use it to benchmark standard reinforcement learning algorithms.


\subsection{Data-Driven Robot Manipulation}

Robot manipulation using learned, data-driven techniques has become a highly effective paradigm for robot manipulation, as exemplified in the pioneering work of Levine~et~al.~\cite{levine_finn_2016}, Mahler~et~al.~\cite{mahler2019learning}, Florence~et~al.~\cite{dense_object_nets_2018}, Kalashnikov~et~al.~\cite{QT-Opt}, and others. Tools used often involve either Imitation Learning (IL)~\cite{argall2009survey} or Reinforcement Learning (RL)~\cite{Sutton_2018}, and in recent years, such techniques have been applied for manipulation of deformables.

For example, IL is used with human~\cite{seita-bedmaking} and scripted~\cite{seita_fabrics_2020} demonstrators for fabric smoothing, while RL is applied for smoothing and folding in~\cite{sim2real_deform_2018,rishabh_2020,lerrel_2020}. Using model-based RL,~\cite{visual_foresight_2018} were able to train a video prediction model to enable robots to fold pants and fabric, and a similar approach was investigated further in~\cite{fabric_vsf_2020} for model-based fabric manipulation. Some approaches combine IL and RL~\cite{balaguer2011combining}, and others use techniques such as latent space planning~\cite{latent_space_roadmap_2020,yan_fabrics_latent_2020} or using dense object descriptors~\cite{descriptors_fabrics_2020}. In this work, we use IL.

As described in Section~\ref{sec:PS}, the deep architecture we propose for image-based manipulation uses Fully Convolutional Neural Networks (FCNs)~\cite{fcn_2015} to produce per-pixel scores in an image, where each pixel corresponds to an action. This technique~\cite{thesis_andyzeng} has shown promising results for robotic warehouse order picking~\cite{zeng_icra_2018}, pushing and grasping~\cite{zeng_iros_2018}, tossing objects~\cite{zeng_tossing_2019}, kit assembly~\cite{form2fit_2020}, and mobile manipulation~\cite{spatial_action_maps_2020}. We show that similar approaches may be effective for picking and placing of deformables. In independent and concurrent work, Lee~et~al.~\cite{folding_fabric_fcn_2020} show how an approach can be used for fabric folding.


\section{Background}\label{sec:PS}

We first describe the problem formulation, followed by background on Transporter Networks~\cite{zeng_transporters_2020}. Section~\ref{sec:goal-conditioned} then describes a novel goal-conditioned framework.

\subsection{Problem Formulation}\label{ssec:problem-formulation}

We formulate the problem of rearranging deformable objects as learning a policy $\pi$ that sequences pick and place actions $\mathbf{a}_t \in \mathcal{A}$ with a robot from visual observations $\mathbf{o}_t \in \mathcal{O}$:
\begin{equation}\label{eq:action}
\pi(\mathbf{o}_t)\rightarrow{\mathbf{a}_t}=(\mathcal{T}_\textrm{pick},\mathcal{T}_\textrm{place})\in{\mathcal{A}}
\end{equation}
where $\mathcal{T}_\textrm{pick}$ is the pose of the end effector when grasping part of an object, and $\mathcal{T}_\textrm{place}$ is the pose of the end effector when releasing the grasp.
Since deformable objects may require more than a single pick and place action to reach a desired configuration, it is important that the policy $\pi$ learns from closed-loop visual feedback. Some rearrangement tasks~\cite{rearrangement_2020} may also be specified by a target goal observation $\mathbf{o}_g$, in which case we refer to the policy as goal-conditioned: 
\begin{equation}\label{eq:action_goal}
\pi(\mathbf{o}_t, \mathbf{o}_g)\rightarrow{\mathbf{a}_t}=(\mathcal{T}_\textrm{pick},\mathcal{T}_\textrm{place})\in{\mathcal{A}}.
\end{equation}
In this work, we consider tabletop manipulation tasks where both poses $\mathcal{T}_\textrm{pick}$ and $\mathcal{T}_\textrm{place}$ are defined in SE(2). Picking and placing positions are sampled from a fixed-height planar surface, while rotations are defined around the z-axis (aligned with gravity direction). The $\mathcal{T}_\textrm{pick}$ and $\mathcal{T}_\textrm{place}$ both parameterize a motion primitive \cite{frazzoli2005maneuver} that controls the end effector to approach $\mathcal{T}_\textrm{pick}$ until contact is detected,
activates the end effector to execute a grasp, moves upward to a fixed z-height, approaches $\mathcal{T}_\textrm{place}$, and lowers the end effector until contact is detected, then releases the grasp.
While this discrete-time planar action parameterization has its limitations, we find that it remains sufficiently expressive for a number of tabletop tasks involving manipulation of bags, in which gravity helps to induce a natural adhesive force that brings objects back towards the tabletop plane. 

To train the policy, we assume access to a small dataset of $N$ stochastic expert demonstrations $\mathcal{D} = \{\xi_i\}_{i=1}^N$, where each \emph{episode} $\xi_i$ of length $T_i$ (i.e.,  the number of actions) consists of a sequence of observations and actions:
\begin{equation}\label{eq:tau}
\xi_i = (\mathbf{o}_1, \mathbf{a}_1, \mathbf{o}_2, \ldots, 
\mathbf{o}_{T_i}, \mathbf{a}_{T_i}, \mathbf{o}_{T_i + 1})
\end{equation}
used to supervise $\pi$, which may be goal-conditioned.

\begin{figure*}[t]
\center
\includegraphics[width=0.95\textwidth]{transporter-goal_v04.png} 
\caption{
The proposed \emph{Transporter-Goal-Split}, applied to an example on \emph{bag-color-goal}, where given the current image $\mathbf{o}_t$ and goal $\mathbf{o}_g$, the objective is to insert the block in the correct, open bag; the goal $\mathbf{o}_g$ specifies that the yellow bag (with the block) is the target, not the blue bag. Transporter Networks use three Fully Convolutional Networks (FCNs), $f_\textrm{pick}$ for the attention module, and $\Phi_\textrm{query}$ and $\Phi_\textrm{key}$ for the transport module. \emph{Transporter-Goal-Split} considers a separate goal image $\mathbf{o}_g$ and passes that through a new FCN, $\Phi_\textrm{goal}$. These are then combined with other deep features (labeled as ``logits'') through element-wise multiplication. Finally, the standard cross-convolution operator is applied. \emph{Transporter-Goal-Stack} dispenses with $\Phi_\textrm{goal}$ and instead stacks $\mathbf{o}_t$ and $\mathbf{o}_g$ as one image, and does not use $\Phi_\textrm{goal}$). All tested Transporter architectures produce $\mathcal{Q}_{\rm pick}$ and $\mathcal{Q}_{\rm place}$, which are each $320{\times}160$ dimensional heat maps colored so that darker pixels are low values and lighter pixels are high. The largest value in a heat map represents the \emph{pixel} to pick or place.
}
\vspace*{-10pt}
\label{fig:transporter-goal}
\end{figure*}

\subsection{Background: Transporter Networks}\label{ssec:vanilla-transporter}

\emph{Transporter Networks}~\cite{zeng_transporters_2020} is a model architecture for manipulation that learns to rearrange objects by (i) attending to a local region of interest, then (ii) predicting its target spatial displacement by cross-correlating its dense deep visual features over the scene. It can parameterize actions for pick and place, and has shown to work well for variety of tabletop manipulation tasks including stacking, sweeping, and kit assembly~\cite{form2fit_2020}. 

Transporter Networks consist of 3 Fully Convolutional Networks (FCNs). The first FCN $f_\textrm{pick}$ takes as input the visual observation $\mathbf{o}_t$, and outputs a dense per-pixel prediction of action-values $\mathcal{Q}_\textrm{pick}$ that correlate with picking success: $\mathcal{T}_\textrm{pick} = \argmax_{(u, v)} \ \mathcal{Q}_\textrm{pick}((u,v)|\mathbf{o}_t)$ where each pixel $(u, v)$ corresponds to a picking action at that location via camera-to-robot calibration. The second FCN $\Phi_\textrm{key}$ also takes as input $\mathbf{o}_t$, while the third FCN $\Phi_\textrm{query}$ takes as input a partial crop $\mathbf{o}_t[\mathcal{T}_\textrm{pick}]$ from $\mathbf{o}_t$ centered on $\mathcal{T}_\textrm{pick}$. Both the second and third FCNs output dense feature embeddings, which are then cross-correlated with each other to output a dense per-pixel prediction of action-values $\mathcal{Q}_\textrm{place}$ that correlate with placing success: $\mathcal{T}_\textrm{place} = \argmax_{\{\Delta\tau_i\}} \ \mathcal{Q}_\textrm{place}(\Delta\tau_i|\mathbf{o}_t,\mathcal{T}_\textrm{pick})$ where
\begin{equation}
\mathcal{Q}_\textrm{place}(\Delta\tau|\mathbf{o}_t,\mathcal{T}_\textrm{pick}) = \Phi_\textrm{query}(\mathbf{o}_t[\mathcal{T}_\textrm{pick}])*\Phi_\textrm{key}(\mathbf{o}_t)[\Delta\tau]
\end{equation}
and $\Delta\tau$ covers the space of all possible placing poses.

A key aspect of Transporter Networks is that the visual observations $\mathbf{o}_t$ must be spatially consistent, so that the 3D structure of the data is preserved under different visuo-spatial transforms $\Delta\tau$. In practice, this property also improves training data augmentation (since rotations and translations of the orthographic image appear as different configurations of objects in the scene). To leverage this, observations $\mathbf{o}_t$ are top-down orthographic images of the tabletop scene, where each pixel represents a vertical column of 3D space.

\section{Goal-Conditioned Transporter Networks}\label{sec:goal-conditioned}

In some tasks, it is more natural to specify a success criteria by passing in a goal observation $\mathbf{o}_g$ showing objects in a desired configuration. In this case, we assume $\mathbf{o}_g$ is fixed and available as an extra input with the current observation $\mathbf{o}_t$, and that $\mathbf{o}_t$ and $\mathbf{o}_g$ have the same pixel resolution.

\subsection{Goal-Conditioned Transporter Networks}\label{ssec:goal-conditioned}

We propose two goal-conditioned architectures based on Transporter Networks. The first, \emph{Transporter-Goal-Stack}, stacks the current $\mathbf{o}_t$ and goal $\mathbf{o}_g$ images channel-wise, then passes it as input through a standard Transporter Network.
The second, \emph{Transporter-Goal-Split}, separates processing of the goal image $\mathbf{o}_g$ through a fourth FCN $\Phi_\textrm{goal}$ to output dense features that are combined with features from the query $\Phi_\textrm{query}$ and key $\Phi_\textrm{key}$ networks using the Hadamard product:
\begin{equation}
\psi_\textrm{query}(\mathbf{o}_t) = \Phi_\textrm{query}(\mathbf{o}_t)\odot\Phi_\textrm{goal}(\mathbf{o}_t)
\end{equation}
\begin{equation}
\psi_\textrm{key}(\mathbf{o}_t) = \Phi_\textrm{key}(\mathbf{o}_t)\odot\Phi_\textrm{goal}(\mathbf{o}_t)
\end{equation}
\begin{equation}
\mathcal{Q}_\textrm{place}(\Delta\tau|\mathbf{o}_t,\mathcal{T}_\textrm{pick}) = \psi_\textrm{query}(\mathbf{o}_t)[\mathcal{T}_\textrm{pick}]*\psi_\textrm{key}(\mathbf{o}_t)[\Delta\tau]
\end{equation} We hypothesize that this separation is beneficial for learning, since convolutional kernels may otherwise struggle to disambiguate which channels correspond to $\mathbf{o}_t$ or $\mathbf{o}_g$.
Figure~\ref{fig:transporter-goal} shows the architecture of \emph{Transporter-Goal-Split}.
For consistency, all Transporter modules $\Phi_\textrm{query}$, $\Phi_\textrm{key}$, and $\Phi_\textrm{goal}$ employ a 43-layer, 9.9M parameter FCN~\cite{fcn_2015} with residual connections~\cite{resnets_2016}, given that similar architectures have been used for FCN-based picking and placing~\cite{zeng_icra_2018,zeng_transporters_2020}.

A key aspect of the goal-conditioned model is that the spatial structure of the goal images are preserved in the architecture. This prior encourages the deep networks to learn features that infer pick and place actions anchored on visual correspondences between objects in the current and goal images. This is useful for manipulating deformable objects, where changes between current and desired configurations are better captured with dense correspondences \cite{descriptors_fabrics_2020,dense_object_nets_2018}.

\subsection{Training Details for Goal Conditioned Models}

To train both goal-conditioned architectures, we use an approach based on Hindsight Experience Replay~\cite{HER_2017}. For each task (Table~\ref{tab:tasks}), we assume a dataset of $N$ demonstrations $\mathcal{D} = \{\xi_i\}_{i=1}^N$, where each episode $\xi_i$ (see Eq.~\ref{eq:tau}) is of length $T_i$ with final observation $\mathbf{o}_g = \mathbf{o}_{T_i + 1}$.
In standard Transporter Networks, a single sample $(\mathbf{o}_k, \mathbf{a}_k)$ containing the observation and the resulting action at time $k$, is drawn at random from $\mathcal{D}$. For the goal-conditioned Transporter Networks, we use the same procedure to get $(\mathbf{o}_k, \mathbf{a}_k)$, then additionally use the corresponding observation $\mathbf{o}_g$ after the last action from the demonstration episode containing $\mathbf{o}_k$, to get training sample $(\mathbf{o}_k, \mathbf{a}_k, \mathbf{o}_g)$. 

As described in Section~\ref{ssec:vanilla-transporter}, because observations $\mathbf{o}_t$ are top-down orthographic images and are spatially consistent with rotations and translations, this enables data augmentation by randomizing a rotation and translation for each training image. To handle the goal-conditioned case, both $\mathbf{o}_t$ and $\mathbf{o}_g$ are augmented using the same random rotation and translation, to ensure consistency.


\section{Simulator and Tasks}\label{sec:simulator-tasks}


\begin{table}
  \setlength\tabcolsep{3.0pt}
 
  \centering
  \caption{
  \textbf{DeformableRavens}. Tasks involve rearranging deformable objects (e.g.,  cables, fabrics, and bags). Each comes with a scripted expert demonstrator that succeeds with high probability, except for the four bag tasks which are challenging; for these, we filter to use only successful episodes in training. Some require \emph{precise placing} to trigger a success. Tasks with a \emph{visible zone} will have a green target zone on the workspace to indicate where items should be placed (e.g., a square target zone that a fabric must cover); other \emph{goal-conditioned} tasks use a separate goal image to specify the success criteria for object rearrangement. See Figure~\ref{fig:tasks} for visualizations.
  }
  \footnotesize
  \begin{tabular}{@{}lrccc}
  \toprule
  Task                  & demos$\;$  & precise & visible & goal  \\
  (Max. Episode Length) & stats(\%)  & placing & zone    & cond. \\
  \midrule
  (a) cable-ring$^\mathsection$ (20)          &  99.1 & \xmark & \cmark & \xmark \\
  (b) cable-ring-notarget$^\mathsection$ (20) &  99.3 & \xmark & \xmark & \xmark \\
  (c) cable-shape$^*$ (20)                    &  98.8 & \cmark & \cmark & \xmark \\
  (d) cable-shape-notarget$^*$ (20)           &  99.1 & \cmark & \xmark & \cmark \\
  (e) cable-line-notarget$^*$ (20)            & 100.0 & \cmark & \xmark & \cmark \\
  \midrule
  (f) fabric-cover (2)                        &  97.0 & \xmark & \xmark & \xmark \\
  (g) fabric-flat$^{\dagger}$ (10)            &  98.3 & \cmark & \cmark & \xmark \\
  (h) fabric-flat-notarget$^{\dagger}$ (10)   &  97.4 & \cmark & \xmark & \cmark \\
  \midrule
  (i) bag-alone-open$^{\mathsection}$ (8)     &  60.2 & \xmark & \xmark & \xmark \\
  (j) bag-items-1 (8)                         &  41.7 & \xmark & \cmark & \xmark \\
  (k) bag-items-2 (9)                         &  32.5 & \xmark & \cmark & \xmark \\
  (l) bag-color-goal (8)                      &  89.1 & \xmark & \xmark & \cmark \\
  \bottomrule
  \end{tabular}
  \\$^\mathsection$evaluated based on maximizing the convex hull area of a ring.
  \\$^*$evaluated by the percentage of a cable within a target zone.
  \\$^\dagger$evaluated using fabric coverage, as in Seita~et~al.~\cite{seita_fabrics_2020,seita-bedmaking}.
  \vspace{-2.0em}
  \label{tab:tasks}
\end{table}

We evaluate the system on \emph{DeformableRavens}, a novel suite of simulated manipulation tasks involving cables, fabrics, and bags, using PyBullet~\cite{coumans2019} with an OpenAI gym~\cite{gym} interface. See Table~\ref{tab:tasks} and Figure~\ref{fig:tasks} for overviews.

\subsection{Deformable Objects (Soft Bodies) in PyBullet}\label{ssec:simulator}

Data-driven methods in robot learning often require substantial amounts of data, which can be expensive to obtain in the real-world~\cite{QT-Opt}.
While simulators have been used in robotics to help alleviate this difficulty, such as for locomotion~\cite{sim-to-real-locomotion} and rigid object manipulation~\cite{openai-dactyl}, many simulated benchmarks for manipulation focus on rigid objects, partially because of difficulties in simulation of deformables~\cite{manipulation_cloth_liu_2016}.

Motivated by these challenges, we provide support for deformable objects (called ``soft bodies'') in PyBullet~\cite{coumans2019}, a widely-used publicly available simulator for robotics research. While prior work with soft bodies in PyBullet~\cite{deep_dressing_2018,assistive_gym_2020,sim2real_deform_2018} use position-based dynamics solvers, we use new soft body physics simulation based on the Finite Element Method~\cite{bathe2006finite} with mass-springs and self-collisions among vertices~\cite{cloth-cloth-collisions}. Contact and friction constraints between soft bodies and multi bodies are solved in a unified constraint solver at the velocity level. Implicit damping of the velocity uses a Krylov style method and soft body contact and friction is based on Conjugate Gradient for symmetric positive definite systems and Conjugate Residual for indefinite systems~\cite{Verschoor2019}.

In simulation, cables consist of a sequence of rigid bodies (``beads''). Fabrics and bags are soft bodies, where each consists of a set of vertices. For fabrics and bags, we create a plane and a sphere object, respectively, using Blender~\cite{blender}. (For bags, we remove vertices to form an opening.) We then import the object mesh files into PyBullet. To improve physics stability of drawstring-style bagging behavior in PyBullet simulation, we additionally attach a ring of rigid beads at the bag's opening, using the same beads that form cables. See Figure~\ref{fig:pybullet-sim} for an overview.

\begin{figure}[t]
\center
\includegraphics[width=0.47\textwidth]{bags_v03-min.png}
\caption{
\textbf{Top row}: examples of physical bags. The bags we use follow a design similar to the sack (top left) and drawstring (top middle). Bags with handles (e.g.,  top right) or with more stiffness will be addressed in future work. \textbf{Bottom row}: to make bags, we create a sphere in Blender and remove vertices above a certain height (bottom left). We import the mesh into PyBullet and add a series of beads at the bag opening. For bag tasks, we initialize the bag by randomly sampling a starting pose (bottom middle), applying a small force, and allowing the bag to crumple (bottom right).
}
\vspace*{-15pt}
\label{fig:pybullet-sim}
\end{figure}

\subsection{Benchmark for Manipulating Deformable Objects}\label{ssec:benchmark}

We design 12 tasks with deformables, listed in Table~\ref{tab:tasks}. For consistency, each uses a standardized setup: a UR5 arm, a $0.5{\times}1$m tabletop workspace, and 3 calibrated RGB-D cameras diagonally overlooking the workspace, producing top-down observations $\mathbf{o}_t \in \mathbb{R}^{320\times 160\times 6}$ of pixel resolution $320{\times}160$, where each pixel represents a $3.125{\times}3.125$mm vertical column of 3D space. Observations contain 3 RGB channels and 3 channel-wise depth values. The goal is to train a policy that executes a sequence of pick and place actions in the workspace to achieve an objective, learned from demonstrations with behavior cloning~\cite{Pomerleau_behavior_cloning}. Some tasks (e.g.,  \emph{cable-shape-notarget} and \emph{fabric-flat-notarget}) are specified by goal images of a target scene configuration, while others are described solely by the distribution of demonstrations (e.g.,  \emph{cable-ring-notarget} and \emph{bag-alone-open}).

The grasping motion primitive (introduced in Section~\ref{sec:PS}) is similar to the implementations in~\cite{fabric_vsf_2020,seita_fabrics_2020} and approximates a pinch-grasp on a deformable by indexing the nearest vertex that comes in contact with the end effector tip, and locking in its degrees of freedom to match that of the end effector using fixed constraints. Certain tasks are divided into different stages, such as \emph{bag-items-1} and \emph{bag-items-2} which involve opening a bag, inserting items inside, and lifting a bag. Depending on the task stage and the gripped item, actions lift the gripped object to a different hard-coded height. For example, pulling the bag upwards requires a higher vertical displacement than opening the bag.

\section{Experiments}\label{sec:experiments}

We use scripted, stochastic demonstrator policies to get 1000 demonstrations (i.e.,  episodes) per task, and train policies using 1, 10, 100, or all 1000 demonstrations. We provide a detailed overview of the data generation process in the Appendix. We test the following models:

\noindent\textbf{Transporter.} This model is directly from~\cite{zeng_transporters_2020}.

\noindent\textbf{Transporter-Goal-Split.} The Transport model that includes a separate goal module $\Phi_\textrm{goal}$ to process $\mathbf{o}_g$.

\noindent\textbf{Transporter-Goal-Stack.} A model that stacks $\mathbf{o}_t$ and $\mathbf{o}_g$ channel-wise to form a 12-channel image, then passes it as input to standard Transporter Networks.

\noindent\textbf{GT-State MLP.} A model that uses ground truth pose information as observations $\mathbf{o}_t$, and does not use images. It processes its input with a multi-layer perception (MLP) and directly regresses $\mathcal{T}_{\rm pick}$ and $\mathcal{T}_{\rm place}$.

\noindent\textbf{GT-State MLP 2-Step.} A baseline similar to the prior ground truth baseline, except it regresses in a two-step fashion, first regressing $\mathcal{T}_{\rm pick}$, and then concatenates this result again to the ground truth information to regress $\mathcal{T}_{\rm place}$.

We test \emph{Transporter} on non-goal conditioned tasks, whereas for those that use goals $\mathbf{o}_g$, we test with \emph{Transporter-Goal-Split} and \emph{Transporter-Goal-Stack}. Ground truth baselines are tested in both settings, where in the goal-conditioned case, we concatenate the ground truth pose information in both the current and goal configurations to form a single input vector. Following~\cite{zeng_transporters_2020}, to handle multi-modality of picking and placing distributions, the models output a mixture density~\cite{mixture_density_networks} represented by a 26-D multivariate Gaussian. During training, all models use the same data augmentation procedures for a fair comparison.

\noindent\textbf{Evaluation metrics.}
Episodes for \emph{cable-shape}, \emph{cable-shape-notarget}, and \emph{cable-line-notarget}, are successful if all cable beads have reached designated target poses.
For \emph{cable-ring}, \emph{cable-ring-notarget}, and \emph{bag-alone-open}, success is triggered when the convex hull area of the ring of beads (forming the cable or the bag opening) exceeds a threshold.
Episodes for \emph{fabric-cover} are successful if the fabric covers the cube.
\emph{Fabric-flat} and \emph{fabric-flat-notarget} are evaluated based on coverage~\cite{seita_fabrics_2020}.
\emph{Bag-items-1} and \emph{bag-items-2} consider a success when all blocks have been transported to the target zone with the bag.
Finally, \emph{bag-color-goal} evaluates success if the item has been inserted into the correct colored bag.

\begin{table*}[t]
  \setlength\tabcolsep{5.0pt}
  \caption{
  \textbf{Results.} Task success rate (mean \% over 60 test-time episodes in simulation of the best saved snapshot) vs. \# of demonstration episodes (1, 10, 100, or 1000) used in training. For the first eight tasks listed, we benchmark with Transporter Networks~\cite{zeng_transporters_2020} (``Transporter'') and two baselines that use ground-truth pose information instead of images as input. The last row of four tasks tests goal-conditioned policies, and we test with proposed goal-conditioned Transporter Network architectures (see Section~\ref{ssec:goal-conditioned}), along with similar ground-truth baselines. We do not test the last four tasks with ``Transporter'' since the architecture does not support including an extra goal image as input. Section~\ref{sec:experiments} and the Appendix contain more details.
  }
  \centering
  \footnotesize
 
  \begin{tabular}{@{}lrrrrrrrrrrrrrrrr@{}}
  \toprule
  & \multicolumn{4}{c}{cable-ring} & \multicolumn{4}{c}{cable-ring-notarget} & \multicolumn{4}{c}{cable-shape} & \multicolumn{4}{c}{fabric-cover}  \\
 \cmidrule(lr){2-5} \cmidrule(lr){6-9} \cmidrule(lr){10-13} \cmidrule(lr){14-17}
 Method & 1 & 10 & 100 & 1000 & 1 & 10 & 100 & 1000 & 1 & 10 & 100 & 1000 & 1 & 10 & 100 & 1000 \\
 \midrule
 GT-State MLP            & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 1.7 & 3.3 & 5.0 & 0.4 & 0.8 & 1.0 & 0.5 & 3.3 & 25.0 & 18.3 & 21.7  \\
 GT-State MLP 2-Step     & 0.0 & 1.7 & 1.7 & 0.0 & 1.7 & 0.0 & 0.0 & 1.7 & 0.7 & 0.6 & 0.9 & 0.5 & 3.3 & 16.7 & 6.7 & 3.3  \\
 Transporter             & 16.7 & 50.0 & 55.0 & 68.3 & 15.0 & 68.3 & 73.3 & 70.0 & 75.6 & 80.6 & 90.1 & 86.5 & 85.0 & 100.0 & 100.0 & 100.0  \\
 \midrule
 & \multicolumn{4}{c}{fabric-flat} & \multicolumn{4}{c}{bag-alone-open} & \multicolumn{4}{c}{bag-items-1} & \multicolumn{4}{c}{bag-items-2}  \\
 \cmidrule(lr){2-5} \cmidrule(lr){6-9} \cmidrule(lr){10-13} \cmidrule(lr){14-17}
 Method & 1 & 10 & 100 & 1000 & 1 & 10 & 100 & 1000 & 1 & 10 & 100 & 1000 & 1 & 10 & 100 & 1000 \\
 \midrule
 GT-State MLP            & 26.0 & 45.6 & 65.6 & 71.3 & 15.0 & 16.7 & 35.0 & 43.3 & 1.7 & 20.0 & 30.0 & 31.7 & 0.0 & 0.0 & 6.7 & 8.3  \\
 GT-State MLP 2-Step     & 21.8 & 30.9 & 45.5 & 41.7 & 11.7 & 15.0 & 18.3 & 26.7 & 0.0 & 8.3 & 28.3 & 31.7 & 0.0 & 1.7 & 6.7 & 11.7  \\
 Transporter             & 42.1 & 86.5 & 89.5 & 88.8 & 18.3 & 50.0 & 61.7 & 63.3 & 25.0 & 36.7 & 48.3 & 51.7 & 5.0 & 30.0 & 41.7 & 46.7  \\
 \midrule
 & \multicolumn{4}{c}{cable-line-notarget} & \multicolumn{4}{c}{cable-shape-notarget} & \multicolumn{4}{c}{fabric-flat-notarget} & \multicolumn{4}{c}{bag-color-goal}  \\
 \cmidrule(lr){2-5} \cmidrule(lr){6-9} \cmidrule(lr){10-13} \cmidrule(lr){14-17}
 Method & 1 & 10 & 100 & 1000 & 1 & 10 & 100 & 1000 & 1 & 10 & 100 & 1000 & 1 & 10 & 100 & 1000  \\
 \midrule
 GT-State MLP            & 11.1 & 44.5 & 72.7 & 77.4 & 11.1 & 42.7 & 66.0 & 65.4 & 14.8 & 49.8 & 62.1 & 63.2 & 0.8 & 0.8 & 10.0 & 14.9  \\
 GT-State MLP 2-Step     & 8.5 & 39.4 & 58.8 & 65.4 & 9.4 & 44.9 & 54.9 & 56.4 & 23.0 & 51.1 & 59.6 & 61.9 & 4.9 & 5.0 & 5.0 & 15.0  \\
 Transporter-Goal-Stack  & 63.5 & 82.8 & 53.0 & 54.4 & 54.4 & 47.5 & 45.1 & 45.6 & 16.5 & 26.3 & 25.3 & 20.1 & 12.4 & 21.6 & 65.4* & 70.3*  \\
 Transporter-Goal-Split  & 74.9 & 95.6 & 53.9 & 99.2 & 48.4 & 75.1 & 64.9 & 76.4 & 27.6 & 35.6 & 30.1 & 77.0 & 10.0 & 63.1 & 40.1* & 49.8*  \\ 
  \bottomrule \vspace{-0.8em} \\
  *trained with 40K iterations.
  \end{tabular}
  \label{tab:results}
  \vspace{-2.0em}
\end{table*}


\section{Results}\label{sec:results}

We summarize results in Table~\ref{tab:results}. For each model type and demo count $N \in \{1, 10, 100, 1000\}$, we train 3 models randomly initialized with different TensorFlow~\cite{tensorflow2015-whitepaper} seeds, except for \emph{bag-color-goal}, where we report results from 1 model since the task is computationally expensive. Models are trained for 20K iterations, with a batch size of 1 for the three Transporter models and 128 for the two ground truth models (to strengthen the baseline). We save 10 snapshots throughout training at equally spaced intervals, and for each, we roll out 20 evaluation episodes. This gives 60 metrics (Section~\ref{sec:experiments}) for each snapshot, due to training 3 models. For each of the 10 snapshots and their associated metric, we average the 60 metrics and report the maximum over the 10 in Table~\ref{tab:results}. The Appendix has more extensive analysis.

\subsection{Non-Goal Conditioned Tasks}

In these tasks, \emph{Transporter} generally achieves orders of magnitude better sample efficiency than ground truth models. It performs reliably on \emph{fabric-cover} with 100\% success rates with just 10 demos (compared to 25\% or less for all ground truth models), and does well on \emph{cable-shape} ($\ge$~90.1\%) and \emph{fabric-flat} ($\ge$~86.5\%) given enough demos. On \emph{bag-alone-open}, \emph{Transporter} attains performance of 61.7\% and 63.3\% with 100 and 1000 demos, which is comparable to the scripted demonstrator performance of 60.2\% (1000 successes out of 1661) before data filtering. Similarly, on \emph{bag-items-1} and \emph{bag-items-2}, the best raw performance numbers (with 1000 demos) are 51.7\% and 46.7\%, which exceed demonstrator performance of 41.7\% and 32.5\% (see Table~\ref{tab:tasks}).

Qualitatively, the learned policies may generalize to new starting configurations from the same data distribution. Figure~\ref{fig:teaser} shows an example successful rollout on \emph{bag-items-1}, where the robot opens the bag, inserts the cube, picks and pulls the bag (which encloses the cube in it), then deposits the bag to the target zone. The project website contains videos.

\subsection{Goal Conditioned Tasks}

For \emph{cable-line-notarget} and \emph{cable-shape-notarget} with 1 and 10 demos, \emph{Transporter-Goal-Stack} and \emph{Transporter-Goal-Split} outperform ground-truth baselines, but the performance gap narrows with larger datasets, perhaps because ground-truth models should eventually do well given sufficient data. Furthermore, we observe from training curves (see the supplementary material) that at around 16-20K training iterations, both goal-conditioned Transporter models with larger datasets (100 and 1000 demos) start to get significantly higher test-time performance across the goal-conditioned tasks (especially \emph{bag-color-goal}). From inspecting test roll outs, we hypothesize that these models need to train longer to disambiguate $\mathbf{o}_t$ and $\mathbf{o}_g$ for the picking network $f_{\rm pick}$.


\begin{figure}[t]
\vspace{-0.5em}
\center
\includegraphics[width=0.46\textwidth]{failures_v02.png}
\caption{
Failure cases we observe from trained Transporter policies on bag tasks. Left: in all bag tasks, a failure case may result from covering up the bag opening; these are hard for subsequent actions to recover from. Right: even if items are inserted into the bag, they may not be fully enclosed and can fall out when the bag is lifted. We overlay frames with orange arrows to indicate the direction of motion of the robot's end effector.
}
\vspace*{-15pt}
\label{fig:failure}
\end{figure}


\section{Limitations and Future Work}
\label{sec:limitations}

In this work, we present a new suite of tasks that evaluate end-to-end vision-based manipulation of deformable objects, spanning cables, fabrics, and bags. In addition, we propose goal-conditioned model architectures with Transporter Networks for rearranging objects to match goal configurations.

Manipulating deformable objects is challenging, and while the proposed method yields promising results, its limitations point to interesting directions for future work.
For example, Figure~\ref{fig:failure} shows several failure modes in the bag tasks.
The bag opening step is challenging because a single counterproductive action that visually occludes the bag opening makes it difficult to recover.
In addition, for \emph{bag-items-1} and \emph{bag-items-2}, items may fail to remain enclosed in the bag when lifted vertically.
Future work may investigate higher rates of control that learn recovery policies to react in real-time, or stateful systems that can track objects under visual occlusions, or bimanual manipulation policies, which may help prevent items from falling out of bags. We also observe that the performance of policies on goal-conditioned tasks are slightly below those that explicitly rely on visual anchors in the workspace (e.g.,  \emph{fabric-flat} vs. \emph{fabric-flat-notarget}), suggesting a performance gap. Finally, given the COVID-19 pandemic which limited access to robots, as conditions improve, we will conduct physical experiments with bags.


\section*{Acknowledgments}

\small Daniel Seita is supported by the Graduate Fellowships for STEM Diversity. We thank Xuchen Han for assistance with deformables in PyBullet, and Julian Ibarz for helpful feedback on writing.



\bibliographystyle{IEEEtranS}
