\section{Introduction}
Traditionally, it is believed that infants have very little understanding of the physical world due to their poor performance on object-manipulation tasks \cite{piaget1954construction,piaget1952origins}. However, with further research conducted via new visual-attention methods, it has been revealed that, contrary to traditional beliefs, even young infants possess a wealth of knowledge about the physical world and have the innate ability to perform causal reasoning over physical interactions between objects \cite{baillargeon1995model,hespos2001infants,needham1997object,sperber1995causal}. In addition, young infants develop their sensitivity to kinetic information for depth in the very early stages of development, which allows them to foster a response for expansion and contraction patterns specifying approach to three-dimensional objects \cite{kayed2000timing,schmuckler1998looming,shirai2004asymmetry}. Beyond that, infants can also learn to recognise the objects from those interactions when they are as young as nine months old \cite{shinskey2014picturing}. Thus, as a result of all these attributes, infants can observe a wide range of physical interactions (e.g. containment, stability and contact) between objects of various geometric shapes and further develop a causal understanding for fundamental physical interactions in their preoperational stage of development \cite{huitt2003piaget}. Furthermore, according to the theory of accommodation \cite{piaget1976piaget}, infants can further modify their basic understanding of physical interactions into higher order tasks which incorporate a mixture of those fundamental physical interactions. Hence, it is of paramount importance for infants to have fully developed their innate ability in understanding those fundamental physical interactions before moving on to the higher order tasks. Equivalently, it is vital for AI models that aim to improve their understanding and reasoning capabilities of real-world object interactions, to be first trained on those fundamental physical interactions. 

\begin{figure}[t]
\begin{center}
\includegraphics[width=\linewidth]{dataexample.png}
\end{center}
  \caption{ Top row: visual data attributes for one example frame comprises of RGB, object segmentation, optical flow, depth, and surface normal vector. Bottom three rows: example frames from the three physical interactions.}
\label{fig:long3}
\end{figure}

This paper aims to study the significance of causal reasoning in fundamental physical interactions between objects in a 3D environment. Our work draws inspiration from a few recent 3D visual reasoning datasets \cite{baradel2019cophy,girdhar2019cater,yi2019clevrer} in terms of their methodologies for synthesizing large-scale physical interaction datasets in a 3D environment. As such, the paper composed of two parts: 1) the SPACE simulator for synthesizing of various physical interactions between objects in 3D environment with a large degree of conditional variances and accurate physics-based interactions, and 2) the SPACE dataset, a large-scale video dataset with over 15k unique videos and over 2 million frames of the three fundamental physical interactions: containment, stability and contact, as shown in Figure \ref{fig:long3}. Furthermore, we evaluate the SPACE dataset on PhyDNet \cite{leguen20phydnet}, a state-of-the-art model for physics-based future frame prediction and show that by pre-training on our SPACE dataset, we can improve the physics-based model's performance on higher order real-world human action datasets after fine-tuning in a way similar to curriculum learning \cite{10.1145/1553374.1553380}.



\section{Related Work} 
The notion of drawing inspiration from conventional concepts in neuroscience and psychology to understand human perception of physical dynamics and causal reasoning has been around for some time \cite{aggarwal2011human,cao2020long,kong2018human,liang2019peeking}. However, those works primarily focused on human intuitions of recognizing or predicting motions. But with the advancement and rise of deep learning, computer graphics, and embodied AI \cite{duan2021survey,kobbelt2004survey,lecun2015deep}, there has been a paradigm shift towards generating synthetic datasets that range from simple 2D cartoons \cite{gordon2016commonsense,netanyahu2021phase,zitnick2014adopting} to realistic interaction in 3D environments \cite{baradel2019cophy,duan2020actionet,girdhar2019cater,johnson2017clevr,shu2021agent,yi2019clevrer}, all with the aim to explore machine perception of physics and causal reasoning on a deeper level. However, only datasets from CLEVRER \cite{yi2019clevrer}, CoPhy \cite{baradel2019cophy}, and CATER \cite{girdhar2019cater} are most relevant to our work.

\begin{figure}[t]
\begin{center}
  \includegraphics[width=\linewidth]{setup.png}
\end{center}
  \caption{Environment setup, and all the available interactable objects and containment holders (for containment task only) spawned for the two physical events.}
\label{fig:long2}
\end{figure}
CLEVRER \cite{yi2019clevrer} is a collision-based video dataset comprising over 20,000 synthetic videos of object collision and 300,000 questions and answers on four newly proposed categories of questions: descriptive, explanatory, predictive, and counterfactual. A Neuro-Symbolic Dynamic Reasoning model was further proposed to tackles the dataset.

CoPhy \cite{baradel2019cophy} developed a model for causal physical reasoning in a synthetic 3D environment. However, they propose a model that learns the physical dynamics from a counterfactual setting and based on the given scenarios to make the next frame prediction. The CoPhy framework allows for the generation of 300k different scenarios in a 3D environment, such as a tower of blocks falling or objects colliding. 


CATER \cite{girdhar2019cater} built a video dataset with observable and controllable objects in a 3D environment and requires the model to have spatiotemporal understanding to solve it. It consists of three different tasks: atomic action recognition, compositional action recognition, and snitch localization (an object permanence task where the goal is to classify the final location of the snitch object within a 2D grid space), with 5,500 videos for each of the tasks. 


Even though our SPACE dataset might share similar visual modalities as the three datasets, but the nature of its physical interactions and the complexity in dynamics are very different. The purpose of the SPACE dataset is to serve as the building blocks for curriculum learning \cite{10.1145/1553374.1553380} in more complex real-world object interaction tasks. Hence, our SPACE dataset is rich in complexity and diversity, making it a suitable dataset for training AI models to learn the intuitive physics behind object interactions.

\section{SPACE Dataset}
\subsection{Overview}
Figure \ref{fig:long3} shows the composition of the SPACE dataset, which is made up of three novel video datasets that are generated from the scenarios synthesized based on three fundamental physical interactions: \emph{containment}, \emph{stability}, and \emph{contact} in a 3D environment. Each of the physical interaction scenarios is synthesized by our SPACE simulator, which is developed using Blender \cite{blender}, an open-source 3D computer graphics software with a Python API. The SPACE simulator allows users to generate various scenarios for each of the physical interactions with different cinematic conditions, object classes and position settings as shown in Figure \ref{fig:long2}. Each physical interaction scenario generated has several parameters such as object classes $O$, number of objects spawned $N$, spawn locations $L$, and containment holders $C$ (only applicable to the containment task). These parameters are selected because they affect the physical dynamics significantly, thus helping to create more diverse set of scenarios for each category of the physical interactions.


\textbf{Containment.} This physical interaction is demonstrated by synthesizing up to three objects of various object classes to be spawned at different locations with predefined heights which are directly above one of the containment holders as illustrated in Figure \ref{fig:long3}. The data will capture the final coordinates of each object to determine if it is contained within the containment holder $label_{containment}=\{0,1\}$.



\textbf{Stability.} This physical interaction is demonstrated via having up to three objects of various object classes spawning at different locations with predefined heights which are directly above the origin of the plane as shown in Figure \ref{fig:long3}. The stability $label_{stability}=\{0,1\}$ of each object can be determined through the differences in the angular rotational motion of each object measured throughout the scene.  


\textbf{Contact.} This physical interaction is demonstrated via having up to three objects of various object classes spawned at different locations and at a ground level height as shown in Figure \ref{fig:long3}. After the scene has been initialized, a ball of fixed mass and shape will be spawned and moved across the plane at a speed of $1m/s$ with a fixed trajectory. Whether each object has been contacted $label_{contact}=\{0,1\}$ are determined by the changes in its initial and final positions.

\subsection{Procedural Generation}
To synthesize all the physical interaction scenarios, the SPACE simulator will sample from a range of one to three objects $N=\{1,2,3\}$ to be spawned in the scene, and the class of each object are then sampled from $O=$ \{\emph{cylinder, cone, inverted cone, cube, torus, sphere, flipped cylinder}\}.

However, there are differences in certain settings between each category of the physical interactions. For containment, the object(s) will be spawned directly above the containment holder with height differences of $\pm0.5$ between two consecutive objects. The containment holder is sampled from $C=$ \{\emph{wine glass, glass, mug, pot, box}\}.
For stability, the object(s) will be spawned directly above the origin $(0,0,z)$ with its \emph{x} and \emph{y} coordinates sampled from $\{0.3,0.2,0.1,0,-0.1,-0.2,-0.3\}$.
For contact, the object(s) will be spawned within a $6\times6$ grid with an incremental or decremental value of 1 for both the \emph{x} and \emph{y} axes at ground level.

\begin{figure}[t]
\begin{center}
  \includegraphics[width=\linewidth]{plots.jpg}
\end{center}
  \caption{Dataset analysis for number of objects and success rate of physical interactions by object classes.}
\label{fig:long}
\end{figure}


\subsection{Dataset Structure}
 There are 15,000 unique scene instances generated for the SPACE simulator, with 5,000 scenes for each category of physical interactions. From there, we collect 15,000 videos lasting 3 seconds with a frame rate of 50 frames per second (FPS), which total up to 2,250,000 frames. Besides the RGB frames, we also provide the segmentation map, optical flow map, depth map and surface normal vector map as shown in Figure \ref{fig:long3}. We further perform an in-depth analysis of the composition of each geometric object within each category of physical interactions and analyze the success rate for each of the geometric objects within the scene. The success rate for each of the geometric objects within their physical interaction task is a clear indication of how the geometric shape of the object might have on the overall outcome of physical interactions, as seen in Figure \ref{fig:long}.


\section{Experiments}
In our experiments, we aim to show that the SPACE dataset is useful in helping physics-based deep models learn robust physical dynamics for downstream tasks.

\subsection{Task \& Dataset}
In this paper, we illustrate the efficacy of our dataset for improving physics-based models by comparing two different experimental setups on the future frame prediction task \cite{liu2018future}. For this task, we make use of our SPACE dataset and the UCF101 dataset \cite{soomro2012ucf101}. The UCF101 dataset consists of realistic action videos, collected from YouTube, with 101 action categories.

Specifically, we want to show that a transfer learning setup that leverages on pre-training with the SPACE dataset outperforms one that is limited to pre-training on the UCF101 dataset, when both models are subsequently fine-tuned and evaluated on the UCF101 dataset. Our setup is also inspired by curriculum learning \cite{10.1145/1553374.1553380}, where data is presented to the model in a way that gradually illustrates more complex data. In this case, UCF101 represents the increased complexity, since it is situated in the real world while our SPACE dataset is in a synthetic environment.

\subsection{Network Architecture}
We select a state-of-the-art physics-based model, PhyDNet \cite{leguen20phydnet} for our experiments. PhyDNet introduces a two-branch deep architecture that disentangles the learning of physical dynamics and unknown complementary information that do not correspond to any prior model. In PhyDNet, physics knowledge is represented by the learning of partial differential equations (PDE) and is used to enforce physical constraints for future frame prediction. Unlike previous models, PhyDNet allows for a generic class of linear PDEs through the varying of differential orders, enabling it to account for more classical models (e.g. the heat equation, the wave equation and etc). To learn the unknown complementary information, PhyDNet uses a generic ConvLSTM \cite{xingjian2015convolutional}.

\subsection{Experimental Settings}
\label{sec:experimental_settings}
For all experiments, we use a batch size of 2 and a sequence length of 50 frames. We remove UCF101 samples that have less than 50 frames. Within the 50 frames, the first 5 frames are set as the input sequence, and the remaining 45 are to be generated. We train for 20 and 30 epochs for pre-training and fine-tuning respectively. We use an adaptive learning rate that follows the original PhyDNet setup. During fine-tuning for both SPACE and UCF101, we set the starting learning rate to be the final learning rate in the UCF101 pre-training, and maintain the adaptive learning rate. Like in the original PhyDNet setup, we also employ teacher forcing \cite{williams1989learning}, we set the reduction rate in teacher forcing probability threshold to be the same as that of PhyDNet with each epoch. We reset this threshold at the start of the fine-tuning process with the same reduction rate.

For SPACE, we sample 50 frames by taking 1 frame every 3 frames, since each scene has a sequence of 150 frames. We take 1,000 instances for each task in SPACE: containment, stability and contact for a total of 3,000 instances. For each task, we take 200 samples as validation data in a 8:2 ratio. Likewise, we split UCF101 into training and validation data by categories in a 8:2 ratio.

We vary the dataset composition in our experiments for both SPACE and UCF101 to test the importance of the three SPACE tasks in helping physics-based deep models learn robust physical dynamics. Specifically, we have two scenarios:
1) 3 UCF101 categories are filmed with a fixed camera, and encapsulate physical dynamics from at least two of the three SPACE tasks (UCF3), while having minimal human intervention/actions throughout the video,
and 2) individual SPACE tasks evaluated on scenario 2 during fine-tuning.

\subsection{Evaluation Metric}
We evaluate our experimental setups using an evaluation metric commonly used in video prediction: Mean Squared Error (MSE). This metric computes the quality of generated frames in comparison to their respective ground-truth frames. The MSE is averaged for each frame of the output sequence. Lower MSE indicates better performance.
\begin{figure}[t]
\begin{center}
  \includegraphics[width=\linewidth]{result.png}
\end{center}
  \caption{Experimental results for scenarios in Section \ref{sec:experimental_settings}. For scenario 1, we compare the effects of UCF3 and SPACE pre-training (cyan and red lines), and for scenario 2, we compare the effects of individual SPACE tasks (green, orange and pink lines).}
\label{fig:result}
\end{figure}
\subsection{Results}
Our results show that the SPACE dataset helps physics-based model, PhyDNet to learn robust physical dynamics. In Figure \ref{fig:result}, we show that the validation loss in experimental scenario 1 (where we train the models on UCF3) is lower in the case where the model is pre-trained on the SPACE dataset. We also show that each of the three fundamental physical interactions in the SPACE dataset is essential for the learning of robust physical dynamics, since the experimental results for scenario 2 are worse than when they are combined in scenario 1. Furthermore, experimental scenario 1 with SPACE pre-training has the overall lowest validation MSE loss after 30 epochs of fine-tuning on UCF3.

\subsection{Conclusion}
Children learn the causality and intuitive physics behind object interaction by first learning about fundamental physical interaction concepts such as contact, containment and stability. Similarly, for AI models to improve their understanding of object interactions, they should also be trained in an approach like curriculum learning. In this work, we propose SPACE, a simulator for physical interactions and causal learning in 3D environments, that composes a large-scale synthetic video dataset for three fundamental physical interaction tasks. Our experiments show that the pre-training done the on SPACE dataset improves a state-of-the-art physics-based deep learning model's performance on real-world object interaction tasks.

\section*{Acknowledgments}
This research is supported by the Agency for Science, Technology and Research (A*STAR), Singapore under its AME
Programmatic Funding Scheme (Award \#A18A2b0046) and the National Research Foundation, Singapore under its NRF-ISF Joint Call (Award NRF2015-NRF-ISF001-2541). We would also like to thank the Machine Learning and Data Analytics Lab at School of Electrical and Electronic Engineering, Nanyang Technological University and DeCLaRe Lab, Singapore University of Technology and Design for the computational resources used for this work.



\bibliographystyle{IEEEbib}
