\section{Introduction}
\vspace{-3mm}

Efficient exploration in the absence of dense reward signals is a long-standing problem in reinforcement learning~\citep{vecerik2017leveraging}. Without dense extrinsic signals, a promising alternative is to define suitable auxiliary \textit{intrinsic} signals that can help the agent in exploring its environment~\citep{laud2004theory}. Recently, \textit{curiosity} has emerged as a promising computational framework for modeling intrinsic reward and has brought major advances in many sparse-reward domains~\citep{pathak2017curiosity,burda2018large,burda2018exploration,pathak2019self,dean2020see}. While the algorithmic details of different methods vary, the core idea is to use changes in the observed state as the intrinsic reward to encourage agents to explore their environment. Despite their strong performance on many sparse-reward tasks, these existing approaches tend to rely on a holistic view of state transitions and do not allow for a targeted understanding of specific aspects of the environment. However, not all states are equally interesting but such information is not available to the agent a priori. On the contrary, humans rely on extensive knowledge about the world when exploring the environment. Language serves as a powerful medium for encoding this knowledge. A particular type of language that humans use is \textit{question} -- in an unfamiliar environment, humans often start the exploration by asking what can be done in the environment.
Based on this observation, we hypothesize that \textit{language-based question answering} may provide a grounded and targeted medium to probe specific knowledge about the current state in order to solve the task at hand.

As a step towards more structured and flexible curiosity-driven learning, we develop a novel form of curiosity, \textsc{Ask \& Explore}\ (\textsc{AnE}), that leverages \textit{grounded question answering} to encourage the agent to ask questions about the environment and be curious when the answers to these questions change. These questions can capture physical properties of the objects (e.g., \textit{Is the large sphere green in color?}) as well as their spatial relationships with other objects (e.g., \textit{Are there any blue spheres behind the cyan ball?}). By using language as a compositional medium to uncover specific knowledge about the environment, we are able to train an agent that explores and solves challenging long-horizon sparse-reward tasks. In addition to our qualitative results, we perform an in-depth study of what type of questions are useful under what scenarios to provide empirical guidelines for applying our method.







\vspace{-3mm}
\section{Background}
\vspace{-3mm}

\textbf{Exploration bonuses and curiosity-driven exploration}. Exploration bonuses motivate agents to explore their environment even when extrinsic reward $r_t^e$ is sparse (or zero) by training the policy to maximize a new reward $r_t = r_t^e + r_t^i$, where $r_t^i$ is the exploration bonus or the intrinsic reward at time $t$~\citep{krebs2009novelty,dayan1996exploration,sutton1990integrated}. The intrinsic reward $r_t^i$ is designed to be higher in novel states in order to encourage the agents to explore less frequently visited states. In recent years, several promising algorithms in this family include: 1) Curiosity-driven exploration by self-supervised prediction~\citep{pathak2017curiosity,burda2018large,pathak2019self}, which formulates an intrinsic reward that encourages the agent to favor transitions with high prediction error using dynamics-based learning, and 2) Random Network Distillation (RND)~\citep{burda2018exploration}, which encourages novelty by training the policy to minimize the prediction error of a predictor neural network as it tries to mimic a randomly initialized target neural network. Several other approaches have used count-based exploration~\citep{bellemare2016unifying, tang2017exploration} and multimodal signals~\citep{dean2020see} to encourage exploration.

\textbf{CLEVR-Robot Environment:} We perform experiments using the CLEVR-Robot Environment~\citep{jiang2019language}, an open-source object interaction environment built using the MuJoCo physics engine~\citep{todorov2012mujoco} and CLEVR language engine~\citep{johnson2017clevr}. The environment is designed to serve as a testbed for studying grounded language understanding and object manipulation. To succeed in this environment, the agent must be able to handle a varying number of objects with diverse visual and physical properties (see details in Appendix~\ref{app_dataset}).

\begin{figure}[tbp]
\centering
\vspace{-6mm}
\includegraphics[width=\linewidth]{images/approach.pdf}
\caption{\textbf{\textsc{Ask \& Explore}}: Our approach proposes a curiosity formulation that leverages grounded question answering to query specific knowledge about the environment. The agent is encouraged to ask questions and be curious about transitions when the answer to a question changes (details in Section~\ref{sec: approach}).\vspace{-6mm}}
\label{fig:approach}
\end{figure}

\vspace{-3mm}
\section{\textsc{Ask \& Explore}}
\label{sec: approach}
\vspace{-3mm}



Our goal is to develop an intrinsic reward that leverages the knowledge about the physical properties of objects and how objects in the environment relate to each other. Such intrinsic reward may bridge the gap between passive pattern recognition and active decision making. To design the intrinsic reward, we chose grounded language as a flexible medium for encoding this knowledge. The CLEVR-Robot environment provides an ideal testbed for using grounded language as a source of intrinsic reward as it provides functionalities to generate scenes and language (in the form of questions) that can be evaluated as the agent interacts with the environment. Note that access to the true state and language is not required. Indeed, for an agent in the real world, such assumptions do not hold. Nonetheless, just like humans can describe a scene with language, the agent can also be equipped with a parameterized visuolinguistic model such as a visual question answering model (VQA) or image captioning model. We plan to explore these directions in future works.


We formulate an intrinsic reward that aims to generate the agent's curiosity about transitions when the answers to the agent's questions grounded in the environment change. At every step, the agent has access to $n$ questions, $q_1, q_2, ... q_n$. For question $q_k$, the difference in the answer before the transition $A(s_t, q_k)$, and after the transition $A(s_{t+1}, q_k)$ contributes to the curiosity signal corresponding to that action (Figure \ref{fig:approach}). To evaluate $A(s_t, q_k)$, we experiment with two types of intrinsic rewards that leverage language. One of them uses the labeling function of the CLEVR-Robot environment and the other utilizes a parameterized VQA model (experiments with the latter can be found in Appendix \ref{app_vqa}). The intrinsic reward at time $t$, $r^i_t$, expressed as
\begin{equation}
     r_t^i = \sum_{k=1}^n \mathbbm{1}[A(s_t, q_k) \neq A(s_{t+1}, q_k)]
\end{equation}

Further algorithmic details such as how the questions are selected can be found in Appendix \ref{imp_details}.


\begin{figure}[tbp]
\centering
\begin{subfigure}[t]{0.24\columnwidth}
\centering
\includegraphics[width=0.99\linewidth, height=0.8\linewidth]{images/simplegoal_init.png}
\vspace{-0.5cm}
\caption{\scriptsize \centering Goal is: ``{\it There is a \textbf{\textcolor{green}{green}} sphere; are there any rubber \textbf{\textcolor{cyan}{cyan}} balls \textbf{in front of} it?}".}
\end{subfigure}
\hfill
\begin{subfigure}[t]{0.24\columnwidth}
\centering
\includegraphics[width=0.99\linewidth, height=0.8\linewidth]{images/simplegoal_transition.png}
\vspace{-0.5cm}
\caption{\scriptsize \centering Agent performs actions and interacts with the environment and tries to satisfy goal.}
\end{subfigure}
\hfill
\begin{subfigure}[t]{0.24\columnwidth}
\centering
\includegraphics[width=0.99\linewidth, height=0.8\linewidth]{images/simplegoal_end.png}
\vspace{-0.5cm}
\caption{\scriptsize \centering Resulting state: Agent receives \textbf{+1} reward.}
\end{subfigure}
\hfill
\begin{subfigure}[t]{0.24\columnwidth}
\centering
\includegraphics[width=0.99\linewidth, height=0.8\linewidth]{images/order.png}
\vspace{-0.5cm}
\caption{\scriptsize \centering Object ordering}
\end{subfigure}
\caption{We use the CLEVR-Robot environment and consider both dense (a-c) and sparse (d) reward settings. The global location of the objects vary across episodes (images from~\citet{jiang2019language}).\vspace{-4mm}}
\label{fig:clevr_env}
\end{figure}

\vspace{-3mm}
\section{Experiments}
\vspace{-3mm}

We design our experiments to understand the following overarching question: \textit{Does an agent with grounded language understanding explore the environment in a more structured and efficient manner?} To answer different aspects of this question, we first evaluate our approach in tasks with different reward sparsity (Section \ref{sec: varying_sparsity}). Then, we evaluate the impact of different types of grounded language understanding on the performance and how the impact differs in settings with varying reward sparsity (Section \ref{sec: varying_complexity}). Finally, we study the effect of the linguistic feedback's density on the efficacy of exploration (due to space limit, we defer the details to Appendix \ref{app_multiple_ques}).

We compare our approach to three baselines:

1. Proximal Policy Optimization (PPO)~\citep{schulman2017proximal} (no exploration bonus)

2. Intrinsic Curiosity Module (ICM)~\citep{pathak2017curiosity}

3. Random Network Distillation (RND)~\citep{burda2018exploration}.

We use the same optimized hyperparameters from the original papers~\citep{pathak2017curiosity, burda2018exploration}. The agent is trained using PPO in all experiments with the same hyperparameters\footnote{The original ICM uses A3C but we used PPO similar to~\cite{burda2018large}.}. We perform three independent runs of each algorithm without any tuning of random seeds, and plot the mean and standard deviation across the three runs (see details in Appendix~\ref{app_details}).


\vspace{-2mm}
\subsection{Varying Degree of Reward Sparsity}
\label{sec: varying_sparsity}
\vspace{-2mm}

To study in what scenarios grounded language understanding can help exploration, we test our approach in two tasks with drastically different reward sparsity.

\textbf{Dense reward setting.} \quad In this setting, the agent needs to complete an object alignment goal where the spatial relationship between two objects in the environment is specified, for example, \textit{"There is a green sphere; are there any rubber cyan balls in front of it?"} (Figure \ref{fig:clevr_env} (a-c)). Goal descriptions take the form of questions which can be evaluated on the state to assess if the goal has been met successfully. The agent receives a reward of $+1$ if it manipulates the objects to achieve the desired spatial arrangement. When the environment is reset before every episode, it is ensured that the goal state is not satisfied initially. 

\textbf{Sparse reward setting.} \quad In the \textbf{object ordering} task (Figure \ref{fig:clevr_env} (d)), the agent needs to order the objects by color in a single line, for example, \textit{“arrange the objects so that their
colors range from blue to green in the horizontal direction, and keep the objects close vertically"}. The ordering of colors we specify is: \textit{cyan, purple, green, blue, red} from left to right. The agent is given a $+10$ reward if it is able to successfully order the objects in this arrangement, and $0$ otherwise. The rationale behind opting for a $+10$ reward instead of $+1$ is to compensate for the extreme reward sparsity which led to the agent not making any significant progress.

The results for the two settings are shown in Figure \ref{fig:sparsity_plots}. We observe in the dense reward setting, PPO with no exploration bonus outperforms all curiosity-driven methods, which highlights that curiosity does not provide a significant advantage when the reward is dense. In the sparse reward task, we find that while all existing baseline methods struggle to make meaningful progress, \textsc{AnE}\ significantly outperforms the baselines in this setting using a single question ($n = 1$) at each step. This confirms our hypothesis -- an intrinsic reward that leverages grounded language understanding is better at exploring the environment. The exploration results in a wider coverage of relevant states and helps the agent learn to solve the task more efficiently
compared to existing novelty-based exploration methods.
In addition, we study the impact of scaling to multiple questions in both dense and sparse reward environments in Appendix \ref{app_multiple_ques}, and the performance of different approaches in the absence of any extrinsic reward in Appendix \ref{app_intrinsic_only}.

\begin{figure}[tbp]
\begin{minipage}{.49\textwidth}
\centering
\vspace{-2mm}
\includegraphics[width=\linewidth]{images/regular_setting_1.png}
(a) Dense reward setting
\end{minipage}%
\hspace{.02\textwidth}
\begin{minipage}{.49\textwidth}
\centering
\vspace{-2mm}
\includegraphics[width=\linewidth]{images/sparse_setting_1.png}
(b) Sparse reward setting
\end{minipage}%
\caption{\textsc{AnE}\ significantly outperforms the baselines (PPO, ICM and RND) in the sparse reward setting which demonstrates the effectiveness of an intrinsic reward based on grounded question answering.\vspace{-6mm}}
\label{fig:sparsity_plots}
\end{figure}

\vspace{-2mm}
\subsection{Varying complexity of questions}
\label{sec: varying_complexity}
\vspace{-2mm}

To better understand which questions are most useful for the task, we test the performance of \textsc{AnE}\ using three types of questions querying varying complexity of spatial relationships between objects in the environment - \textbf{one}, \textbf{two} and \textbf{three} ``hop'' questions:

\vspace{-1mm}
\textbf{One-hop}: \textit{``There is a red metallic sphere; are there any green matte balls left of it?''}

\vspace{-1mm}
\textbf{Two-hop}: \textit{``Are there any purple rubber balls that are on the left side of the red sphere that is behind the blue matte ball?''}

\vspace{-1mm}
\textbf{Three-hop}: \textit{``There is a green rubber ball behind the red metallic sphere; are there any blue balls in front of the purple matte sphere?''}

\begin{figure}[!ht]
\begin{minipage}{.49\textwidth}
\centering
\vspace{-2mm}
\includegraphics[width=\linewidth]{images/regular_setting_hops_1.png}
(a) Dense reward setting
\end{minipage}%
\hspace{.02\textwidth}
\begin{minipage}{.49\textwidth}
\centering
\vspace{-2mm}
\includegraphics[width=\linewidth]{images/sparse_setting_hops_1.png}
(b) Sparse reward setting
\end{minipage}%
\caption{Comparing the performance of \textsc{AnE}\ for questions of varying complexity. \vspace{-4mm}}
\label{fig:complexity_plots}
\end{figure}

We observe in Figure \ref{fig:complexity_plots} how the performance of the agent varies with increasing complexity of questions (in terms of a greater number of pair-wise relationships between objects). It is interesting to note that the relationship between language and agent performance is not the same across different task settings. The extrinsic reward increases in the dense reward setting as we move from one-hop to two-hop questions in the environment and then decreases as we progress to three-hop questions. While in the sparse task, the success rate is highest using one-hop questions and decreases as we increase the number of object relationships, although all improve over the baselines. Therefore, even simple probes of spatial relationships are sufficient as a curiosity signal.









\vspace{-3mm}
\section{Conclusion}
\vspace{-3mm}

In this paper, we proposed a novel form of curiosity by encouraging the agent to ask questions about the environment and be curious when the answers to their questions change. We show that this formulation of intrinsic reward probes targeted knowledge about the physical properties of the objects as well as their spatial relationships with other objects, achieving significantly better performance than existing curiosity methods on highly sparse reward tasks.



{\small


