\section{Hyperparameters} \label{sec:hyperparams}
We use the Adam optimizer \citep{adam} in all experiments.

\begin{table}[h!]
\centering
\caption{PPO Hyperparameters}
\begin{tabular}{lcc}
\hline ENV. DISTRIBUTION MODE & HARD \\
\hline$\gamma$ & $.999$ \\
$\lambda$ & $.95$ \\
LEARNING RATE & $0.0005$ \\
\# TIMESTEPS PER ROLLOUT & $256$ \\
EPOCHS PER ROLLOUT & $3$ \\
\# MINIBATCHES PER EPOCH & $8$ \\
MINIBATCH SIZE & $2048$ \\
ENTROPY BONUS $\left(k_{H}\right)$ & $.01$ \\
PPO CLIP RANGE & $.2$ \\
REWARD NORMALIZATION? & YES \\
LEARNING RATE & $5 \times 10^{-4}$ \\
\# WORKERS & 4  \\
\# ENVIRONMENTS PER WORKER & 64 \\
TOTAL TIMESTEPS & $200 \mathrm{M}$ \\
ARCHITECTURE & Impala \\
LSTM? & No  \\
FRAME STACK? & No \\
\hline
\end{tabular}
\end{table}


   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   




\section{Introduction} \label{sec:intro}
A fundamental problem in machine learning is out of distribution (OOD) robustness: performing well on test data that is not distributed identically to the training set \citep{ood-in-ml}.
This problem has been studied under various names (OOD generalization, distributional shift, domain generalization). It is crucial in many applications, particularly those where it is not feasible to collect data distributed identically to that which the model will encounter in deployment.
In this work, we focus on a particularly concerning type of OOD robustness problem called \emph{objective robustness}, a term coined by \citet{clarifying}, which we study in the deep RL setting. Usually, when an RL model is deployed off-distribution, the model either performs well or simply fails to take useful actions. However, there exists an alternative failure mode 
in which the agent pursues an objective other than the true reward while retaining all or most of the capabilities it had on the training distribution.
We call this kind of failure an \emph{objective robustness} failure and distinguish it from \emph{capability robustness} failures. To highlight and illustrate this class of failures, we provide empirical demonstrations of the phenomenon in deep RL agents---to our knowledge, this is the first time it has been demonstrated empirically.

An important inspiration for this paper is work on learned optimizers by \citet{mesa-optimizers}, who identify objective robustness as an important challenge to the future use of machine learning systems in complex domains.
Indeed, while capability robustness failures are concerning, objective robustness failures are potentially more dangerous: the only risks from an incapable agent are those of accidents from its incompetence, but the same is not true for an agent that capably pursues an incorrect objective, which can leverage its learned capabilities to visit potentially arbitrarily bad states (as assessed by the actual reward).
For example, for most reward functions, the optimal policy will try to avoid being shut down \citep{turner_optimal}.
Our main contributions are:
\begin{itemize}
  \item We highlight the class of objective robustness failures, differentiate it from other robustness problems, and discuss why it is important to address (Section~\ref{sec:or}).
  \item We demonstrate that objective robustness failures occur in practice by training deep RL agents on the Procgen benchmark \citep{procgen}, a set of diverse procedurally generated environments designed to induce robust generalization, and deploying them on slightly modified environments (Section~\ref{sec:experiments}).
\end{itemize}


\begin{figure}\label{fig:coinrun}
 \centering
 \includegraphics[width=\textwidth]{coinrun.png}
  \caption{
    A simple demonstration of objective robustness failure.
    \textbf{Left:} An agent trained on procedurally generated levels in CoinRun reliably avoids obstacles and navigates to the coin (the reward), which is always located at the right side of the level.
    \textbf{Right:} When the same agent is deployed on a modified version of the environment in which the coin position is randomized (but accessible), the agent ignores the coin and instead competently navigates to the end of the level. 
    }
\end{figure}


\section{Objective Robustness} \label{sec:or}

Our focus is on a sub-problem of the more general problem of out-of-distribution (OOD) robustness, which is usually studied in the supervised learning setting, where it is defined as achieving good test error even when the test data is sampled from a different distribution than the training data. It is widely known that deep learning systems may fail in unexpected ways when deployed out-of-distribution; for example, a classifier trained on images of cows in Alpine pastures will typically fail to recognize an image of a cow that is taken on a sandy beach
\citep{beery,ood-in-ml}.

We focus on the reinforcement learning setting \citep{sutton}, in which a system is trained to take actions in an environment in order to maximize a given reward. In this setting, the problem is to achieve good reward out-of-distribution. OOD robustness problems frequently arise in RL---here are a few examples:
\begin{enumerate}
  \item A deep RL agent is trained to beat a video game. As it trains, the agent is able to visit new states due to newly learned capabilities and thus effectively induces its own distributional shift \citep{krueger}.
  \item A robot is trained in a simulation where data is plentiful. It is then deployed in the real world, where it encounters subtle differences in physical parameters \citep{openai-robot-hand}.
  \item A factory wants to use deep RL for a process control application. The agent is trained on episodes of fixed length. When it is deployed in the real world, the episode never ends (there are no episode resets in the real world), and the distribution of observations slowly drifts over time as its context in the world changes.
\end{enumerate}

\subsection{Defining objective robustness}
A deep RL agent is trained to maximize reward $R \colon S \times A \times S \to \mathbb R$, where $S$ and $A$ are the sets of all valid states and actions, respectively. Assume that the agent is deployed under distributional shift; that is, an aspect of the environment (and therefore the distribution of observations) changes at test time. If the agent now achieves low reward in the new environment because it continues to act capably yet appears to optimize a different objective $R' \neq R$, then we say the agent underwent an \emph{objective robustness failure}. We call $R'$ the \emph{behavioral objective} of the agent.

We must specify what counts as a behavioral objective (for example, we wish to avoid ascribing a behavioral objective to an agent that simply acts randomly). To do this, 
we understand the behavioral objective as equivalent to the notion of a ``goal'' under the \emph{intentional stance} \citep{dennett}: a system has one if describing the system as pursuing it is useful for predicting that system's behavior. In particular, we do not assume that the behavioral objective is represented explicitly by the agent, although this may sometimes be the case.

Our primary motivation for identifying this class of failures is that objective robustness failures---failures in which systems take competent yet misaligned actions when deployed off-distribution---have the potential to be particularly dangerous.
Other types of failures are bounded in the damage they can do: the worst that can happen is an accident resulting from incompetence (say, a self-driving car failing to brake). In contrast, an agent that pursues an incorrect objective can perform arbitrarily badly relative to the true reward and indeed becomes more dangerous the more capable it is.

\subsection{Properties and causes of objective robustness failures} \label{subsec:causes}

When should we expect models to to be objective robust? We begin by identifying two prerequisites for objective robustness failure:
\begin{enumerate}
  \item The training environment must be diverse enough to learn sufficiently robust capabilities. If this is not the case, then RL algorithms tend to memorize simple action sequences that work in the training environment but are not robust under distributional shift.
  \item There must exist some proxy $R': S \times A \times S \to \mathbb R$ that approximately tracks the true reward on the training distribution.
  \item The proxy and the true reward come apart on the OOD test environment.
\end{enumerate}
While these are plausible necessary conditions for objective robustness failure, they are by no means sufficient since, by themselves, they do not guarantee that the model learns to follow the proxy reward $R'$ instead of the true reward. However, assumptions (1) and (2) are also very weak: almost every real-world problem requires a diverse training environment (to achieve capability robustness), and proxies are common in complex environments. 
Thus objective robustness failure depends mostly on whether the inductive biases of the model prime it to learn a proxy that then (3) diverges from the true objective on the test set.

What proxies then do models tend to learn? Following \citet{mesa-optimizers}, we (non-exhaustively) list factors that increase the likelihood of learning a particular proxy $R'$.

\begin{itemize}
  \item The proxy $R'$ is simpler than the true reward $R$.
  \item The true reward $R$ is sparse, but the proxy $R'$ is dense \citep{gradient-starvation}.
  \item The true reward $R$ is computationally hard to predict without using the proxy, while the proxy is easily predicable.
  \item If $R$ is hard to predict, then large model capacity may make objective robustness failure \emph{less} likely, because the model is better able to model the true reward.
\end{itemize}

As an example for such proxies, consider human evolution. 
While very different from SGD, biological evolution is (to an approximation) a local optimization process that maximizes inclusive genetic fitness. 
Humans, however, generally have no desire to maximize the number of our descendants. Instead, we pursue objectives which in the ancestral environment were good proxies for fitness, such as friendship, food, and love. This illustrates a general phenomenon: given a challenging objective, complex environments are rife with proxies for and sub-goals of that objective, many of which are more dense or easier to predict than the true objective. In addition, human evolution illustrates how such proxies can come apart from the true objective under distributional shift:
in the ancestral environment humans were driven to eat high-caloric foods, which in the modern world often leads to obesity.
We illustrate this sort of shift in a simple experiment in Section~\ref{subsec:keys-and-chests} that relies on sub-goals. This suggests that objective robustness will be a relevant problem in complex, real-world tasks.

Our experiments have been designed to show as clearly as possible that the robustness failures demonstrated are different from those usually considered in the literature. For example, a natural interpretation of our results in CoinRun in Section~\ref{sec:coinrun} is that the agent has learned a \emph{robust capability} to avoid obstacles and navigate the levels,\footnote{After all, Procgen \citep{procgen} was designed to test (capability) generalization in deep RL.} but a \emph{non-robust objective}, since its behavior off-distribution is better described by ``get to the end of the level'' than ``go to the coin''.

By empirically demonstrating this kind of failure for the first time, we hope to illustrate how it is different from the kind of robustness failure that is usually considered in the literature and help others think more clearly about objective robustness. 
Though it may seem unreasonable to expect the agent to identify the coin, not the end of the level, as the marker of the true reward when the two were indistinguishable during training, it is precisely this unidentifiability of the true reward 
that makes objective robustness a difficult problem. More complex and realistic environments are also likely to have proxies which are reliably correlated with the reward during training but may cease to be correlated with it off-distribution. In this way, objective robustness is similar to the problem of unidentifability in IRL \citep{unidentifiability}, as discussed in Section~\ref{sec:related}.

\section{Experiments} \label{sec:experiments}
Here, we provide simple empirical demonstrations of objective robustness failures. Each experiment is an example of an agent that learns to perform capably when deployed out-of-distribution, but pursues a behavioral objective that is different from the objective for which it was optimized. Video examples of objective robustness failures in all of the following environments can be found in the supplementary materials.

All environments are adapted from the Procgen benchmark \citep{procgen}. This benchmark is built to study sample efficiency and generalization to within-distribution tasks. Agents are tasked with performing well in an arcade-like video game from pixel observations. The environments are procedurally generated and thus diverse; to perform well, an agent is forced to learn strategies that work in a wide range of task settings and difficulty and cannot rely on e.g.\ memorizing a small number of trajectories to solve a fixed set of levels.

\paragraph{Implementational Details} For all environments, we use an Actor-Critic architecture using Proximal Policy Optimization (PPO) \citep{ppo}. Hyperparameters can be found in the Appendix. All models are implemented in PyTorch \citep{pytorch}, and our implementation is based on a codebase by \citet{pg-pt}. Unless otherwise stated, models are trained on $100$k procedurally generated levels for 200M timesteps.
Each training run required approximately 30 GPU hours of compute on a V100.

\paragraph{Different kinds of failure} The experiments illustrate different flavors of objective robustness failures. \emph{Action space proxies} (CoinRun and Maze I): the agent substitutes a simple action space proxy (``move right'') for the true reward, which could have been identified in terms of a simple feature in its input space (the yellow coin/cheese).
\emph{Observation ambiguity} (Maze II): The observations contain multiple features that identify the goal state, which come apart in the OOD test distribution.
\emph{Instrumental goals} (Keys and Chests): The agent learns an objective (collecting keys) that is only instrumentally useful to acquiring the true reward (opening chests).

\subsection{CoinRun} \label{sec:coinrun}
In CoinRun, a platformer, the agent spawns on the left side of the level and has to avoid enemies and obstacles to get to a coin (the reward) at the far right of the level. 
We modify the original CoinRun to create a test environment that induces an objective robustness failure.

The agent is trained on vanilla CoinRun, where the coin is always found at the end of the level.
We then deploy the agent in the test environment. This test environment differs from the training environment in two ways: 1) the test environment is newly generated, meaning the agent has not seen the structure of any specific level during training, and 2) instead of being located at the end, the coin position is randomized.
When deployed in this test environment, the agent generally ignores the coin completely. While the agent sometimes runs into the coin by accident, it often misses it and proceeds to the end of the level, as shown in Figure~\ref{fig:coinrun}. It is clear from this demonstration that the agent has not learned to go after the coin; instead, it has learned the proxy "reach the far right end of the level." It competently achieves this objective, but test reward is low.

\subsubsection{Varying how often the coin is randomly placed in training}\label{sec:pct-rand}
To test how stable the objective robustness failure is, we train a series of agents on environments which vary in how often the coin is placed randomly.
We then deploy those agents in the test environment in which coin position is always randomized.
Results can be seen in Figure~\ref{fig:ablations}, which shows the frequencies of two different outcomes,
1) failure of capability: the agent dies or gets stuck, thus neither getting the coin nor to the end of the level, and 2) failure of objective: the agent misses the coin and navigates to the end of the level. As expected, as the diversity of the training environment increases, the proportion of objective robustness failures decreases, as the model learns to pursue the coin instead of to going to the end of the level.

\begin{figure}[ht]
\vskip 0.2in
\begin{center}
\centerline{\includegraphics[width=10cm]{img/ablations.pdf}}
\caption{
 
  How diverse does the training distribution need to be to induce objective robustness? We train agents in a CoinRun environment in which the coin is placed randomly $\{0, 1, 2, 5, 10\}\%$ of the time, and keep track of how often the agent navigates to the end of the level while ignoring the coin.
 
}
\label{fig:ablations}
\end{center}
\vskip -0.2in
\end{figure}




\subsection{Maze} \label{sec:maze}

\begin{figure}
 \centering
 \includegraphics[width=\textwidth]{maze.png}
  \caption{Objective robustness failure in maze environments.
  The agent is trained to navigate mazes to reach its reward.
  \textbf{Left:} The agent is trained on procedurally generated mazes where the reward (a piece of cheese) is always in the top right of the maze. When it is deployed in an environment where the cheese is placed randomly, it ignores it and goes to the top right of the maze.
  \textbf{Right:} The agent is trained to navigate to a yellow gem (the reward). In the test environment, it must choose between a yellow star and a red gem, and it consistently navigates to the yellow star. This would be an undesirable generalization of its objective if it is a gem-collecting agent.
  }\label{fig:maze}
\end{figure}


\subsubsection{Variant 1} \label{subsec:mazeI}
We modify the Procgen Maze environment in order to implement an idea from \citet{hubinger-investigation}. 
In this environment, a maze is generated using Kruskal's algorithm \citep{kruskal}, and the agent is trained to navigate towards a piece of cheese located at a random spot in the maze.
Instead of training on the original environment, we train on a modified version in which the cheese is always located in the upper right corner of the maze (Figure~\ref{fig:maze}).

When deployed in the original Maze environment at test time, the agent does not perform well; it ignores the randomly placed objective, instead navigating to the lower right corner of the maze as usual. Using the terminology of Section~\ref{sec:or}: the training objective is to reach the cheese, but the behavioral objective of the learned policy is to navigate to the lower right corner.

\subsubsection{Variant 2}
We hypothesize that in CoinRun, the policy that always navigates to the end of the level is preferred because it is simple in terms of its action space: simply move as far right as possible. The same is true for the Maze experiment in Section~\ref{subsec:mazeI}, where the agent has learned to navigate to the top right corner. In both experiments, the objective robustness failure arises because a visual feature (coin / cheese) and a positional feature (right / top right) come apart at test time, and the inductive biases of the model favor the latter.
However, objective robustness failures can also arise due to other kinds of distributional shift. To illustrate this, we present a simple setting in which there is no \emph{positional} feature that favors one objective over the other; instead, the agent is forced to choose between two ambiguous visual cues.

We train an RL agent on a version of the Procgen Maze environment where the reward is a randomly placed \emph{yellow gem}. At test time, we deploy it on a modified environment featuring two randomly placed objects: a yellow \emph{star} and a \emph{red} gem; the agent is forced to choose between consistency in shape or in color (Figure~\ref{fig:maze}).
Except for occasionally getting stuck in a corner, the agent always successfully pursues the yellow star, thus generalizing in favor of color rather than shape consistency.

\subsection{Keys and Chests} \label{subsec:keys-and-chests}

\begin{figure}
 \centering
 \includegraphics[width=\textwidth]{keyschests.png}
  \caption{Objective robustness failure on the "keys and chests" task. 
  The agent must collect keys in order to open chests and is only rewarded for opening chests. 
  \textbf{Left:} The agent is trained on procedurally generated mazes in which there are twice as many chests as keys. 
  \textbf{Right:} At test time, there are instead twice as many keys as chests. The agent more highly values collecting keys than opening chests; it routinely goes out of the way to collect all the keys before opening any remaining chests despite the fact that doing so offers no benefit to its actual return (in fact, it would \emph{decrease} its time-discounted return).
  }\label{fig:keyschests}
\end{figure}

So far, our experiments featured environments in which there is a perfect proxy for the true reward. 
The Keys and Chests environment, first suggested by \citet{keys-and-chests}, provides a different type of example. This environment, which we implement by adapting the Heist environment from Procgen, is a maze with two kinds of objects: keys and chests. 
Whenever the agent comes across a key it is added to a key inventory. 
When an agent with at least one key in its inventory comes across a chest, the chest is opened and a key is deleted from the inventory. 
The agent is rewarded for every chest it opens.

The objective robustness failure arises due to the following distributional shift between training and test environments: in the training environment, there are twice as many chests as keys, while in the test environment there are twice as many keys as chests. The basic task facing the agent is the same (the reward is only given upon opening a chest), but the circumstances are different.

We observe that an agent trained on the ``many chests'' distribution goes out of its way to collect all the keys before opening the last chest on the ``many keys'' distribution (Figure~\ref{fig:keyschests}), even though only half of them are even instrumentally useful for the true reward; occasionally, it even gets distracted by the keys in the inventory (which are displayed in the top right corner) and spends the rest of the episode trying to collect them instead of opening the remaining chest(s).
Applying the intentional stance, we describe the agent as having learned a simple behavioral objective: collect as many keys as possible, while sometimes visiting chests. This strategy leads to high reward in an environment where chests are plentiful and the agent can thus focus on looking for keys. 
However, this proxy objective fails under distributional shift when keys are plentiful and chests are no longer easily available. 




\section{Related Work} \label{sec:related}

\paragraph{Learned optimization}
The main inspiration for this paper is the work of \citet{mesa-optimizers} on learned optimization. Its premise is that for many tasks, the simplest way to solve them is to perform search according to an objective function (e.g.\ to perform well at chess, an agent must search through possible moves and evaluate them). This leads to a problem: when ML systems learn to perform search according to an objective, how do we make sure that the learned objective is equivalent to the training objective, even off-distribution?
If many complex tasks require learning a planner+objective, then solving this problem would be crucial for applying deep learning systems to difficult real-world problems.

We do not focus on the alignment of learned optimizers, though it is certainly a noteworthy special case of objective robustness. Instead, we study objective robustness more generally, and
provide the first (to our knowledge) explicit empirical demonstrations of objective robustness failures.

\paragraph{Objective robustness}
Recent work has introduced and discussed the concept of objective robustness.
Though they do not use the term, \citet{Leike} refer to the difference between what a model was optimiz\emph{ed} for and what it appears to be optimiz\emph{ing} as the \emph{reward-result gap}. 
A related distinction is made by \citet{2d-robustness} between 1) robustness failures in which a model fails to generalize capably and 2) those in which a model generalizes capably but pursues a different objective than the one on which it was trained.
The term \emph{objective robustness} is then introduced by \citet{clarifying} to refer to the latter failure mode. These works collectively identify a type of robustness failure that is qualitatively distinct from the kind usually studied in the literature (where models perform poorly off-distribution because they did not learn general enough capabilities).
To our knowledge, we are the first to empirically study objective robustness.




\paragraph{Reward misspecification} 
The work on objective robustness identifies a problem beyond reward (mis-)specification \citep{concrete-problems, faulty-reward}, the problem of specifying a reward that captures the behavior we want. While reward misspecification may already be a problem on the training distribution, objective robustness problems only become visible once we move out-of distribution.
That is, even with a perfectly specified reward function, objective robustness is still an issue.\footnote{Failures due to objective misspecification, like reward hacking, occur when the model behaves in an unintended/undesirable way that nevertheless scores highly on the reward function; in failures of objective robustness, on the other hand, models score \emph{poorly} on the objective function because they are capably pursuing an incorrect objective.} 

\paragraph{Adversarial Robustness} Adversarial examples are a well-known kind of capability robustness failure. In image classification, adversarial examples are typically found by optimizing small $l_p$ norm perturbations to images to cause misclassification, despite the changes being imperceptible to humans \citep{szegedy, goodfellow2015explaining, goodfellowopenai}. More recent works argue that real-world attackers are not limited to small perturbations \citep{gilmer} and that the norm-perturbation model is only a useful local approximation to the true worst-case risk \citep{uesato}. For these reasons, \citet{brown} introduced a contest to evaluate ML systems against \emph{unconstrained} (as opposed to norm-constrained) adversaries. In reinforcement learning, agents have also been shown to be vulnerable to small perturbations on input images \citep{huang, kos}, as well as vulnerable to adversarial policies: policies that attack the agent simply by taking actions in the same environment and thereby modifying the agent's observations to cause poor performance \citep{gleave}.

\paragraph{Out-of-Distribution Robustness} Adversarial examples and objective robustness failures are types of OOD robustness failures; in OOD robustness, the goal is to optimize worst-case performance over a \emph{perturbation set} of possible test domains (environments) $\mathcal{F}$. As in the case of adversarial examples \citep{ilyas}, the reason why ML models usually fail OOD in general is that they tend to learn features that are highly predictive on the training domain(s) but are brittle to distributional shift, also known as \emph{spurious correlations}.

Generalizing well to arbitrary distributions of inputs is impossible, so the perturbation set represents the encoding of our assumptions about which environments might be encountered. Usually this is done by assuming access to a set of $m$ training environments $\varepsilon$ and some relationship between $\varepsilon$ and $\mathcal{F}$. In the case of supervised learning, a few possible assumptions can be made about the relationship between inputs $X$ and targets $Y$. Shifts across environments in the distribution of $P_e(X,Y)$ can be attributed to shifts in $P(X)$ (\emph{covariate shift}), shifts in $P(Y|X)$ (\emph{concept shift}), or a combination of the two \citep{rex}. One possible assumption is that $P(Y|X)$ is fixed, or that only covariate shift can happen (the \emph{fixed relationship assumption, FRA}) \citep{rex}, as is done in the case of adversarial examples. Another possibility is to take the causal approach to domain generalization and assume that the causal mechanism of $Y$ is invariant but $X$ is subject to arbitrary interventions on its mechanisms (the \emph{fixed mechanism assumption, FMA}) \citep{rex}. The idea is that learning a relationship between inputs and targets that is invariant between distributions will enable OOD generalization by avoiding spurious correlations (since the actual causal mechanism of the target is invariant across distributions). Invariant Risk Minimization (IRM) \citep{irm} and Risk Extrapolation (REx) \citep{rex} are examples of such approaches.

The existing work on OOD robustness is largely complementary to our work here on objective robustness. For example, the work of \citet{irm} and \citet{rex} indicate that one approach that might alleviate objective robustness failures is learning invariant representations across a sufficient diversity of environments, which could help the model learn the true causes leading to it receiving the reward. Our CoinRun ablation over how often the coin is randomly placed in training (Section~\ref{sec:pct-rand}) also indicates that training with greater environmental diversity can help identify the true objective. Overall, the OOD robustness literature indicates that models tend to misgeneralize because they learn spurious correlations. The distinction between capability and objective robustness highlights that when a model fails to generalize due to learning spurious correlations, it may do so in two different ways that have notably different consequences: it might generalize completely incapably, or it might generalize capably but pursue an incorrect objective.


\paragraph{Unidentifiability}
Objective robustness failures tend to arise when there are multiple possible reward functions that are indistinguishable from the true reward. This type of unidentifiability is analogous to the one encountered in inverse reinforcement learning (IRL).
In their seminal paper on IRL, \citet{irl} note that identifying the exact reward function an optimal agent optimizes with its behavior is in general impossible since there may be infinitely many reward functions consistent with any observed policy $\pi$ in a fixed environment. \citet{unidentifiability} separate the causes for this unidentifiability into two classes: \emph{representational unidentifiability}, where a set of reward functions is behaviorally invariant under certain arithmetic operations (e.g., re-scaling), and \emph{experimental unidentifiability}, where $\pi$'s observed behavior is insufficient to distinguish between two or more reward functions which both rationalize the behavior of the agent (the behavior is optimal under both); they also demonstrate that experimental unidentifiability can be alleviated if the agent can be observed in other MDPs with different transition dynamics.
In the context of objective robustness, the reward function is unidentifiable by the agent. Given a history of trajectories and rewards in the training environment(s), many possible reward functions fit the data that will generalize OOD differently; this is analogous to experimental unidentifiability.

\section{Discussion} \label{sec:discussion}

In summary, we provide concrete examples of reinforcement learning agents that fail in a particular way: their capabilities generalize to an out-of-distribution environment, whereupon they pursue the wrong objective. While deep RL practitioners may generally be aware that this type of failure is possible, there has not yet been much discussion of objective robustness as distinct from other types of out-of-distribution robustness. We argue that objective robustness is a natural category since, much like adversarial robustness failures, objective robustness failures have distinct causes and pose distinct problems. By introducing the objective robustness problem to a broader audience, we hope to spark interest in it as an avenue for future research.


Additionally, there is much space for further empirical work on the factors that might influence the likelihood of objective robustness failures described in Section~\ref{subsec:causes}. For instance, what kinds of proxy objectives are agents most likely to learn? A better understanding here could inform the choice of an adequate perturbation set over environments to enable the training of objective robust models.
Scaling up the study of objective robustness failures to more complex environments than the toy examples presented here should also facilitate a better understanding of the kinds of behavioral objectives our agents are likely to learn in real-world tasks.




Finally, a more rigorous and technical understanding of the concept of the behavioral objective seems obviously desirable. In this paper, we understood it more informally as equivalent to a goal or objective under the intentional stance \citep{dennett} because humans already intuitively understand and reason about the intentions of other systems through this lens and because formally specifying a behavioral definition of objectives or goals fell outside the scope of this project. However, a more rigorous definition could enable the formalization of properties we could try to verify in our models with e.g. interpretability techniques.

\subsection{Limitations}\label{sec:limitations}

\begin{itemize}
  \item Our experiments demonstrate the existence of objective robustness failures and illustrate some of their causes; they do not prove that they will occur in problems of interest. We do however think that there are good arguments for this (Section~\ref{subsec:causes}).
  \item The notion of objective robustness is quite recent. With time, the objective vs.\ capability robustness distinction may be superseded by a categorization that better captures the problems that occur in practice, is more grounded in theory, or both. However, it seems clear that there are different kinds of possible robustness failures, and understanding how they differ in their causes and consequences is important for building safe and capable AI systems.
\end{itemize}

\section*{Broader Impact}\label{sec:impact}

Our motivation in writing this paper was to address a failure mode in deep reinforcement learning systems that we see as particularly concerning. Highly capable agents taking actions in the service of misaligned objectives could be catastrophic when deployed in the real world. Our hope is that making this risk more tangible with concrete demonstrations of objective robustness failures in modern systems in relatively simple environments will spark further research into both the problem and the space of potential solutions. Ensuring that our systems have actually the objectives we want them to is critical for being able to usefully deploy AI in the world.

We therefore expect research into objective robustness to have a robustly beneficial impact in general. One scenario where this might fail to be true is if a better understanding of objective robustness failures makes it easier to purposefully induce such failures. Perhaps adversaries could induce a distributional shift where an agent's behavioral objective comes apart from the training objective.

\acksection

Special thanks to Rohin Shah and Evan Hubinger for their guidance and feedback throughout the course of this project. Thanks also to Max Chiswick for assistance adapting the code for training the agents, Adam Gleave for helpful feedback on drafts of this paper, and the organizers of the AI Safety Camp for bringing the authors of this paper together: Remmelt Ellen, Nicholas Goldowsky-Dill, Rebecca Baron, Max Chiswick, and Richard Möhn.

This work was supported by funding from the AI Safety Camp and from the Machine Intelligence Research Institute.

