\section{Introduction}
Actor-critic algorithms \citep{Barto1983,Konda1999} learn an explicit actor policy that executes actions in the environment, while the critic evaluates the actions selected by the actor. Despite promising stability and convergence guarantees, actor-critic methods \citep{Wang2016b,Gruslys2017,Schulman2017} are still outperformed by critic-only ones \citep{Schaul2015,Anschel2017,Medina2018}. In our opinion, the weakness of actor-critic algorithms lies in their critic being \emph{on-actor}, i.e., it learns $Q^{\pi}$, the function that evaluates the current actor $\pi$, instead of $Q^*$, the optimal Q-function. Producing accurate estimates of $Q^{\pi}$ is difficult, and limits the extent to which experience replay can be used \citep{Wang2016b}, which reduces sample-efficiency.

In this paper, we present the Actor-Advisor, an architecture in which a Policy Gradient actor learns from its Monte-Carlo returns, instead of Q-Values provided by its critic, and a Double DQN \citep{Hasselt2016} critic advises the actor before an action is selected. To let the critic advise the actor, Policy Shaping \citep{Fernandez2006,Griffith2013} is used. Policy Shaping allows an agent's policy to be directly influenced by an external advisory policy by sampling actions from their mixture. Until now, Policy Shaping could not be used with Policy Gradient, since the action being executed by a vanilla Policy Gradient agent has to be drawn from the exact probability distribution output by the policy. Otherwise, learning diverges \cite{Sutton2000}. Our contribution (Section \ref{sec:contribution}) addresses this convergence issue, and allows the Softmax policy arising from the critic to be mixed with the actor's policy, without impairing convergence.

We compare our contribution, the Actor-Advisor, to state-of-the-art actor-critic algorithms available in the OpenAI baselines, such as Proximal Policy Optimization \citep{Schulman2017} and A3C \citep{Mnih2016}. We also compare the Actor-Advisor, which components are Policy Gradient and Double DQN, with its critic-only counterpart, i.e., Double DQN. Although it is not current state-of-the-art, Double DQN is readily available, easy to implement and tune, and largely outperforms state-of-the-art actor-critic algorithms. Our results demonstrate that the Actor-Advisor's performance is much superior to conventional actor-critic algorithms', and shows better sample-efficiency than critic-only Double DQN \citep{Hasselt2016}. We further note that our Actor-Advisor algorithmic architecture is compatible with any type of critic. In addition to advice emanating from a sample-efficient off-policy critic, we also demonstrate the applicability of the Actor-Advisor to other settings, where advice comes from various sources: a safety-critical task in which the advisor is a proven-safe backup policy; a complex tasks for which a simple and sub-optimal heuristic policy is available; and a transfer learning setting in which a previously learned policy in a given environment advises the agent in a similar but distinct environment.


\begin{comment}
Reinforcement Learning has proven to be effective in many settings, from games to robotics \citep{Silver2017,Levine2016}. However, a well-known downside of RL is that it requires a lot of interactions with the environment to learn a good policy, and while learning, the necessary exploration may lead to unsafe situations. To mitigate this, we might want to leverage existing knowledge we may already have about the problem, such as domain knowledge from human experts or from previously-encountered similar problems, to make learning faster and safer. Policy Shaping, mixing the policy of a Reinforcement Learning agent with an external advisory policy \citep{Griffith2013}, is an effective way to achieve this. Policy Shaping allows for the implementation of two ways to advise the agent: \emph{directives}, a deterministic form of advice that determines the action taken by the agent, and \emph{soft advice}, which amounts to suggesting actions to the agents, rather than determining them. Desirable traits of this implementation are the ability for a designer to force the agent to submit to strict \emph{safety} guidelines using a proven-correct backup policy, a potential increase of sample-\emph{efficiency} and quality of the learned policy thanks to soft advice, and, once it has learned, the ability of an agent to show \emph{robustness} in the face of suboptimal soft advice. While not fulfilling all of these goals, \cite{Griffith2013} show that Policy Shaping is much more readily amenable to these benefits than the much more studied Reward Shaping. Reward Shaping \citep{Ng1999} incorporates human or domain knowledge by considering it as merely an additional element of the reward signal, without intervening on the policy itself. In this paper, we present the Actor-Advisor, based on Policy Gradient, that allows safe backup policies to be strictly obeyed, and, when soft advice is used, is efficient and robust to mistakes in the advice. We analyze its theoretical properties, then experimentally demonstrate its wide applicability to four sub-fields of Reinforcement Learning: safe RL, efficient RL with heuristics and domain knowledge, transfer learning, and Actor-Critic methods.






\end{comment}

\section{Background}
\label{sec:background}

In this section, we formally introduce the concepts at the basis of Actor-Advisor, that is Markov Decision Processes (MDPs), value-based methods, Policy Gradient compared to actor-critic methods and how policies in the form of probability distributions are combined to allow Policy Shaping.

\subsection{Markov Decision Processes}
\label{sec:background_mdp}

A discrete-time Markov Decision Process (MDP) \citep{Bellman1957} with discrete actions is defined by the tuple $\langle S, A, R, T \rangle$: a possibly-infinite set $S$ of states; a finite set $A$ of actions; a reward function $R(s_t, a_t, s_{t+1}) \in  \mathbb{R}$ returning a scalar reward $r_t$ for each state transition; and a transition function $T(s_{t+1} | s_t, a_t) \in [0, 1]$ taking as input a state-action pair $(s_t, a_t)$ and returning a probability distribution over new states $s_{t+1}$.

A stochastic stationary policy $\pi(a_t | s_t) \in [0, 1]$ maps each state to a probability distribution over actions. At each time-step, the agent observes $s_t$, selects $a_t \sim \pi(s_t)$, then observes $r_{t+1}$ and $s_{t+1}$. The $(s_t, a_t, r_{t+1}, s_{t+1})$ tuple is called an \emph{experience} tuple. An optimal policy $\pi^*$ maximizes the expected cumulative discounted reward $E_{\pi^*}[\sum_t \gamma^t r_t]$, where $\gamma$ is a discount factor. The goal of the agent is to find $\pi^*$ based on its experiences within the environment.

\subsection{Value-based methods}
\label{sec:background_value-based}

Q-Learning \citep{Watkins1989} learns the optimal action-value function $Q^* (s, a) = E_{\pi^*} [\sum_{k=0}^{\infty} \gamma^k r_{t+k} | s_t = s, a_t = a]$, i.e., the optimal expected return for each action in each state. Value-based methods, such as Q-Learning and SARSA \citep{Rummery1994}, select actions according to the Q-Values and some exploration strategy. Then, the resulting experience tuple is used to update the Q-function according to this formula:

\begin{align}
    \label{eq:qlearning}
    Q_{k+1}(s_t, a_t) &= Q_k (s_t, a_t) + \alpha \delta, \\
    \nonumber
    \delta &= r_{t+1} + \gamma \max_a Q_k (s_{t+1}, a) - Q_k (s_t, a_t)
\end{align}
The SARSA update rule is obtained by replacing $\max_{a} Q_k(s_{t+1}, a)$ with $Q_k(s_{t+1}, a_{t+1})$, where $a_{t+1}$ is the action actually performed by the agent in the environment. This changes the Q-function being learned, that becomes $Q^{\pi}$ instead of $Q^*$, with $\pi$ the policy that represents the actual behavior of the agent

\subsection{Policy Gradient and Actor-Critic Algorithms}
\label{sec:background_pg}
Instead of choosing actions according to Q-Values, Policy Gradient methods \citep{Williams1992,Sutton2000} explicitly learn an \emph{actor} policy $\pi_\theta(a_t | s_t) \in [0, 1]$, parametrized by a weights vector $\theta$, such as the weights of a neural network. The objective of the agent is to maximize the expected cumulative discounted reward $E_{\pi}[\sum_t \gamma^t r_t]$, which translates to the minimization of the following equation \citep{Sutton2000}:

\begin{align}
	\label{eq:pg}
	\mathcal{L}(\pi_{\theta}) &= -\sum\limits_{t=0}^{T} \left\{
	\begin{array}{cl}
		\mathcal{R}_t \\
		Q^{\pi_{\theta}}(s_t, a_t)
	\end{array}
	\right\} \log (\pi_{\theta} (a_t | s_t))
\end{align}

\noindent
with $a_t \sim \pi_{\theta}(s_t)$ the action executed at time $t$, $\mathcal{R}_t = \sum_{\tau=t}^{T} \gamma^{\tau} r_{\tau}$ the Monte-Carlo return at time $t$, and $r_{\tau} = R(s_{\tau}, a_{\tau}, s_{\tau+1})$ the immediate reward. At every training epoch, experiences are used to compute the gradient $\frac{\partial \mathcal{L}}{\partial \theta}$ of Equation \ref{eq:pg}, then the weights of the policy are adjusted one small step in the opposite direction of the gradient. A second gradient update requires fresh experiences \citep{Sutton2000}, which prevents Policy Gradient to be used with experience replay to improve sample-efficiency.

Actor-critic methods allow the actor to use Q-Values instead of Monte-Carlo returns (the returns $\mathcal{R}_t$ are replaced by $Q^{\pi_{\theta}}(s_t, a_t)$ in Equation \ref{eq:pg}), which leads to a gradient of lower variance. Such methods achieve impressive results on several challenging tasks \citep{Wang2016b,Gruslys2017,Mnih2016,Schulman2017}. However, actor-critic algorithms rely on their critic being on-actor, i.e., the learned $Q^{\pi_{\theta}}$ must be accurate for the current actor. Maintaining a close and healthy relationship between the actor and the critic requires careful design, as discussed by \cite{Konda1999} and \cite{Sutton2000}.


\begin{comment}
\subsection{Policy Gradient}
\label{sec:background_pg}
Policy Gradient methods \citep{Williams1992,Sutton2000} explicitly learn a policy $\pi_\theta(a_t | s_t) \in [0, 1]$ parametrized by a weights vector $\theta$. The agent's goal is to maximize the expected cumulative discounted reward $E_{\pi}[\sum_t \gamma^t r_t]$, which is achieved by to minimizing the following loss \citep{Sutton2000}:

\begin{align}
    \label{eq:pg}
    \mathcal{L}(\pi) &= -\sum\limits_{t=0}^{T} \mathcal{R}_t \log (\pi_{\theta} (a_t | s_t))
\end{align}

\noindent
where $a_t \sim \pi_{\theta}(s_t)$ is the action executed at time $t$, and $\pi_{\theta} (a_t | s_t)$ the probability that action $a_t$ is executed. The return $\mathcal{R}_t = \sum_{\tau=t}^{T} \gamma^{\tau} r_{\tau}$, with $r_{\tau} = R(s_{\tau}, a_{\tau}, s_{\tau+1})$, is a simple discounted sum of future rewards. Intuitively, the loss leads to past actions with the highest return to be executed more often in the future, hence constantly improving the policy. At every training epoch, experiences are used to compute the gradient $\frac{\partial L}{\partial \theta}$ of Equation \ref{eq:pg}, then the weights of the policy are adjusted one step in the direction of the gradient.
\end{comment}

\subsection{On-Policy, Off-Policy and On-Actor}

The definitions of on-policy, off-policy and \emph{on-actor} are central concepts in this paper. Their definitions are as follows. Off-policy algorithms learn the Q-function of a policy that is not the one being executed by the agent. For instance, Q-Learning learns the optimal Q-function $Q^* \equiv Q^{\pi*}$ while the agent executes another policy $\pi$. On the contrary, on-policy algorithms, like SARSA, learn the Q-function $Q^{\pi}$ that evaluates the executed policy $\pi$. While generally accepted, this definition is sometimes unclear when it comes to describing the critic being learned by actor-critic algorithms, since two policies intervene in this setting: the actor $\pi$, and the behavior policy $\mu$. While $\pi = \mu$ in most cases, recent work allows $\mu$ to be distinct from $\pi$ \citep{Degris2012,Wang2016b,Gu2017b}. We therefore denote as \emph{on-actor} an algorithm that learns $Q^{\pi}$ evaluating the actor's policy $\pi$ (regardless of what $\mu$ is), and \emph{off-actor} an algorithm that learns $Q^*$.

\subsection{Policy Shaping}
\label{sec:background_mixing}

Policy Shaping \citep{Griffith2013} allows the agent's learned policy $\pi^L(s_t)$ to be influenced by an external advisory policy $\pi^E(s_t)$. The agent samples actions from the mixture of the two policies $\pi^L(s_t)$ and $\pi^E(s_t)$, computed by performing an element-wise multiplication of $\pi^L(s_t)$ with $\pi^E(s_t)$, divided by their dot product, as follows:

\begin{align}
    \label{eq:pss}
    a_t \sim  \frac{\pi^L(s_t) \pi^E(s_t)} {\pi^L(s_t) \cdot \pi^E(s_t)}
\end{align}

\noindent
Even if first proposed for integrating human feedback in the learning process \citep{Griffith2013}, this simple method can be applied to a larger variety of problems.

\section{The Actor-Advisor}
\label{sec:contribution}

Policy Gradient requires the actions taken by the agent to be directly sampled from the policy $\pi$ it is learning \citep{Sutton2000}, which prevents any exploration mechanism (e.g., $\varepsilon$-Greedy), backup, heuristic or advisory policy to be leveraged by the agent. Our main contribution, the Actor-Advisor architecture, consists of a solution to this problem, that allows the policy executed by the agent to be directly influenced by an external advisory policy, without impairing convergence. We first present the foundation of our contribution, then analyze its theoretical properties, and finally discuss the various ways it can be applied to real-world problems.

\subsection{Policy Gradient with External Advice}

The Actor-Advisor assumes a parametric policy represented by a neural network, trained using Policy Gradient \citep{Sutton2000}. The neural network has two inputs: the state $s_t$ and an \emph{advice} $\pi^E$, a probability distribution over actions. The state is used to compute a probability distribution over actions $\pi^L_{\theta}$, using dense, convolutional or any other kind of \textit{trainable} layers. Then, the output $\pi_{\theta}$ of the network is computed by element-wise multiplying $\pi^L_{\theta}$ and $\pi^E$, followed by a normalization \citep{Griffith2013}:

\vspace{-4mm}
\begin{align}
	\nonumber
    \pi_{\theta}(s_t, \pi^E(s_t)) &= \frac{\pi^L_{\theta}(s_t) \pi^E(s_t)} {\sum_{a \in A} \pi^L_{\theta}(a|s_t) \pi^E(a|s_t)} \\
    \label{eq:pss2}
   	&= \frac{\pi^L_{\theta}(s_t) \pi^E(s_t)} {\pi^L_{\theta}(s_t) \cdot \pi^E(s_t)}
\end{align}

\noindent where $\pi^L_{\theta}(s_t) \cdot \pi^E(s_t)$ is the dot product between the two vectors, and $\pi_{\theta}(s_t, \pi^E(s_t))$ denotes the parametric policy $\pi_{\theta}$ from which the action $a_t$ is actually sampled. The key insight is that the gradient is calculated based on the mixed policy $\pi_{\theta}(s_t, \pi^E(s_t))$, rather than just on the learned parametrized policy $\pi^L_{\theta}$. As a result, the distribution $\pi_{\theta}$ output by the network is a mixture of $\pi^L_{\theta}$ and some advice, and actions can therefore be directly sampled from that distribution. The element-wise multiplication, without any square root or other kind of transformation, has the nice property that an all-ones $\pi^E$ has a neutral effect on $\pi^L_{\theta}$, therefore disabling advice in an elegant way. The network is trained using the standard Policy Gradient loss \citep{Sutton2000}:

\begin{align}
    \label{eq:pgg}
    \mathcal{L}(\pi) &= -\sum\limits_{t=0}^{T} \mathcal{R}_t \log (\pi_{\theta}(a_t | s_t, \pi^E(s_t)))
\end{align}

\noindent with $\pi_{\theta}(a_t | s_t, \pi^E(s_t))$ the probability to execute action $a_t$ at time $t$, given as input the state $s_t$ and some state-dependent advice $\pi^E(s_t)$, and the return $\mathcal{R}_t = \sum_{\tau=t}^{T} \gamma^{\tau} r_{\tau}$, with $r_\tau = \mathcal{R}(s_\tau, a_\tau, s_{\tau+1})$, a simple discounted sum of future rewards. Note that the network training (or optimization) problem is not made harder by the use of $\pi^E$, since $\pi^E$ is not fed to the trainable part of the network, is not parametric, and as such cannot be considered as an extension of the state-space. This neural architecture, inspired by how variable action-spaces are implemented in \cite{Steckelmacher2018}, meets all the Policy Gradient assumptions, yet the behavior of the agent can be directly altered by an external advisory policy from any source. Moreover, our experimental results in Section \ref{sec:experiments} demonstrate that Actor-Advisor is able to leverage advice to learn faster.

\subsection{Stochastic and Deterministic Advice}
\label{subsec:contribution_StocDet}
Depending on the values of $\pi^E(a | s)$, two distinct forms of advice (in this section, we consider that almost deterministic advice is still stochastic) arise, leading to distinct potential uses of the advice:

\begin{description}
\item[Stochastic] advice ensures that each action has a non-zero probability, and can thus be selected by the agent with an arbitrarily high or low probability, depending on $\pi^L$. Stochastic advice can be seen as an optional, \emph{soft} advice, as it will merely bias the action selection rather than determining it. Stochastic advice can be used to allow a heuristic, a human teacher, or a previously learned policy (in a transfer learning setting) to help the agent learn the task. In the case of our new Actor-Critic formulation, the Actor-Advisor, in which a Policy Gradient actor is advised by a Double DQN critic, the advisory policy arising from the critic is also stochastic. In the above-mentioned cases, stochastic advice can be suboptimal, hence it is desirable that the agent learns to ignore it whenever following the advice endangers its performance. A property we believe inherent to Policy Gradient (on which the Actor-Advisor is based) is that the higher the entropy of the advice, the easier it is for the agent to learn to ignore it. Our experiments in Section \ref{sec:experiments} empirically confirms this intuition.
\item[Deterministic] advice has one 1 for a particular action, and a zero probability for all other actions. We consider that deterministic advice, or \emph{directives}, are used to enforce proven-safe mandatory guidelines that must be respected by the agent, in a safe RL task.
\end{description}

\subsection{Analyzing the Gradient}

Conventional Policy Gradient learns a policy that directly maps states to actions, as do our $\pi^L_{\theta}$ policy. In contrast to vanilla Policy Gradient, the gradient computed by Actor-Advisor is not only based on the policy learned $\pi^L_{\theta}$, but on a mixture of $\pi^L_{\theta}$ with the advice $\pi^E$, which is then normalized. In this section, we derive the gradient flowing into $\pi^L_{\theta}$, show that it is modified by $\pi^E$, and detail how $\pi^E$ influences the gradient.

We consider two cases: when the advice is deterministic, we show that the gradient flowing in $\pi^L_{\theta}$ is zero. When the advice is uniform (so, no actual advice is given), we show that the gradient flowing in $\pi^L_{\theta}$ is exactly the vanilla Policy Gradient. The general case, when the advice is stochastic but non-uniform, cannot be compared to vanilla Policy Gradient in our opinion: the Actor-Advisor gradient is different from the vanilla Policy Gradient, which is fortunate because vanilla Policy Gradient is not compatible with external advice. Nevertheless, our empiric results suggest that the Actor-Advisor finds a local optimum (just as vanilla Policy Gradient does), even if stochastic advice may change the path towards it. An in-depth analysis, that considers the whole learning dynamics of the agent instead of point estimates of the gradient, is beyond the scope of this paper.

The first step of our analysis consists of plugging Equation \ref{eq:pss2} in Equation \ref{eq:pgg}, which produces the general Actor-Advisor gradient. For clarity, we omit $s_t$ in our notations, which makes $\pi_{\theta}(a_t | s_t)$ read as $\pi_{\theta}(a_t)$:

\begin{align*}
    \nabla_{\theta} \mathcal{L}(\pi_{\theta}) &= -\nabla_{\theta} \sum\limits_{t=0}^{T} \mathcal{R}_t \log (\pi_{\theta}(a_t | s_t, \pi^E(s_t))) \\
    &= -\nabla_{\theta} \sum\limits_{t=0}^{T} \mathcal{R}_t \log \left( \frac{\pi^L_{\theta}(a_t) \pi^E(a_t)} {\pi^L_{\theta} \cdot \pi^E} \right) \\
    &= \begin{array}{ll}
    	-\nabla_{\theta} \sum\limits_{t=0}^{T} \mathcal{R}_t (\log(\pi^L_{\theta}(a_t)) &\cancel{+ \log(\pi^E(a_t))} \\
    	&- \log(\pi^L_{\theta} \cdot \pi^E))
    \end{array}
\end{align*}

We now consider two cases: i) the advice $\pi^E$ is deterministic; and ii) $\pi^E$ is uniform.

\paragraph{Deterministic} The advice $\pi^E$ is a deterministic vector with a single one, and the other actions have zeros; in other words, $a_t = \operatorname{argmax}_a \pi^E$, the only action allowed by $\pi^E$, and $\pi^E(a_t) = 1$. For compactness, we denote as $\nabla_{\theta} \mathcal{L}(\pi_{\theta})(t)$ the loss at time-step $t$, with $\nabla_{\theta} \mathcal{L}(\pi_{\theta}) = \sum_{t=0}^{T} \nabla_{\theta} \mathcal{L}(\pi_{\theta})(t)$:

\begin{align*}
	\nabla_{\theta} \mathcal{L}(\pi_{\theta})(t) &= -\nabla_{\theta} \mathcal{R}_t (\log(\pi^L_{\theta}(a_t)) - \log(\pi^L_{\theta}(a_t) 1)) \\
	&= -\nabla_{\theta} ~ 0
\end{align*}

In states where $\pi^E$ is deterministic, the gradient becomes zero, which prevents $\pi^L_{\theta}$ from changing. This is expected, as in those states, $\pi^E$ fully determines the actions being executed, and $\pi^L_{\theta}$ is ignored. This property is important, as it ensures that the parameters $\theta$ do not \emph{drift} when deterministic advice is used. Covering big parts of the environment with a backup policy is therefore safe, and does not impair learning in the other parts of the environment.

\paragraph{Uniform} We now consider the case where the advice $\pi^E$ is uniform, be it normalized or not, such that $\forall a: \pi^E(a) = P$. We assume that $\pi^L_{\theta}$ is normalized, such that $\sum_a \pi^L_{\theta}(a) = 1$ for any $\theta$.

\begin{align*}
	\nabla_{\theta} \mathcal{L}(\pi_{\theta})(t) &= -\nabla_{\theta} \mathcal{R}_t (\log(\pi^L_{\theta}(a_t)) - \log(\sum_a P \pi^L_{\theta}(a))) \\
	&= -\nabla_{\theta} \mathcal{R}_t (\log(\pi^L_{\theta}(a_t)) - \cancel{\log(P)}) \\
    &= -\nabla_{\theta} \mathcal{R}_t (\log(\pi^L_{\theta}(a_t)))
\end{align*}

In this case, $\log(P)$ does not depend on $\pi^L_{\theta}$, and can thus be removed from the gradient. We also note that, because  $\pi^L_{\theta}$ is normalized, the gradient of its norm is zero. When uniform advice is used, only the standard Policy Gradient remains, and the agent learns as if our contribution was not implemented.

This concludes our analysis of the gradient when uniform or deterministic advice is used. Our experimental results in Section \ref{sec:experiments} demonstrate the robustness of the Actor-Advisor to all forms of advice, even stochastic non-uniform one, which complements the analysis of this section.



\section{Application Domains}
\label{sec:review}

In this section, we review three subfields of reinforcement learning to which our main contribution, presented in Section \ref{sec:contribution}, can be applied. The three sub-fields are safe RL, efficient leverage of domain knowledge, and transfer learning.

\subsection{Safe RL}

A reinforcement learning agent is considered \emph{safe} if it does not behave in an undesirable and harmful manner \citep{Amodei2016}; in other words, it will not take actions leading to undesirable transitions (with ``undesirable" defined by some designer-provided constraints). According to \cite{Garcia2015} and \cite{Amodei2016}, to tackle the safe RL problem, we can either change the agent's optimization criterion, or, what we focus on in this paper, change its action selection strategy, possibly using a safe backup policy \citep{Hans2008}. Backup policies, regarded by engineers as easy to design, forcibly take control of the agent in certain circumstances, and move it to safe regions of the state-space. In our experiments, we consider a safety-critical task in which a robot navigates on a table, and is prevented from falling by a designer-provided backup policy (Section \ref{subsec:envs}). When the robot is too close to the edge of the table, the backup policy forces the robot to turn until it does not face the edge of the table any more.

\subsection{Leveraging Domain Knowledge}

Our use of advice through Policy Shaping is similar to dynamic Reward Shaping \citep{Ng1999}, which allows an expert designer, or human teacher, to help an RL agent learn faster in environments where rewards are sparse, by giving it additional rewards.
However, Reward Shaping only allows to express feedback \citep{Thomaz2006,Knox2009,Christiano2017,Pilarski2017}, i.e., evaluations of the actions performed by the agent \emph{after} their execution. Hence, in the case where the agent performs an undesirable action, it is only possible to complain a posteriori, and there is no way to prevent the agent from wrong doing.
Policy Shaping has been used to let a human critique policy, extracted from human-delivered feedback, influence the agent's actions by mixing it with the agent's policy \citep{Griffith2013}. However, because the critique policy is slowly learned from feedback, the undesirable action prevention problem remains. In our experiments, we instead consider \emph{advice} \citep{Clouse1992,Maclin1996,Thomaz2008}, provided by a heuristic, rather than feedback. In our setting, at every time-step, our heuristic can advise the agent on which action to choose before the agent has selected any action (see Section \ref{subsec:safe}). In contrast to using Reward Shaping and feedback, advising the agent through Policy Shaping makes it possible to halt or deflect the agent's behavior \emph{before} it engages in undesirable actions, by expressing a directive, i.e., deterministic advice (Section \ref{subsec:contribution_StocDet}).

\subsection{Transfer Learning}

From an RL perspective, transfer learning allows to generalize across tasks, and to reuse a policy learned in a given environment in a new similar environment \citep{Taylor2009}. \cite{Fernandez2006} propose an algorithm inspired by Policy Shaping, that, at each timestep, reuses with probability $\psi$ a previously learned policy, and greedily exploits the currently learned policy with probability $1-\psi$. Our contribution, the Actor-Advisor, can be applied in a transfer learning setting, and has a significant advantage on the algorithm of \citeauthor{Fernandez2006}: it does not require the $\psi$ hyperparameter to be defined, the mixing happens automatically.

Distillation is closely related to transfer learning. A large classifier generates samples used to train a smaller one, in the hope that the smaller classifier will be as good as the big one, but much more compact \citep{Bucila2006}. We show in Section \ref{sec:experiments_transfer} that Actor-Advisor naturally implements a robust distillation scheme. An external policy is distilled in the current one, even if the task these two policies solve is different (but on the same action set). The Actor-Advisor makes no assumption about where the \textit{old} policy comes from, or how the \textit{new} policy is represented, and can therefore be used to distil a policy to a much simpler encoding.






\section{Experiments}
\label{sec:experiments}


Our main contribution, the Actor-Advisor architecture, allows an external policy to guide and influence the policy being learned by the agent. This section illustrates how this general framework naturally applies to the following areas:

\begin{enumerate}
  \item \textbf{actor-critic:} We introduce a combination of Policy Gradient with Double DQN, made possible by our contribution, allowing Policy Gradient to be advised by the critic, instead of directly using the critic's Q-Values (Section \ref{subsec:dqn+pg}).
  \item \textbf{safety \& domain knowledge:} We evaluate the Actor-Advisor on a safety-critical task, in which a safe backup policy prevents the agent from executing a dangerous action such as falling off a table, and some additional heuristic advice helps the agent when it is close to the goal (Section \ref{subsec:safe}).
  \item \textbf{transfer learning:} We illustrate how an agent, that has to navigate in a house, is able to leverage advice from a policy learned in another house to learn faster (Section \ref{sec:experiments_transfer}).

\end{enumerate}

\subsection{Environments}
\label{subsec:envs}

Our experiments take place in two environments which we introduce below. Table has continuous states, complex dynamics and is difficult to explore. Five Rooms is a smaller environment, but that leads to challenging transfer learning tasks.

\begin{figure}
	\centering
	\includegraphics[width=45mm]{table.pdf}
	\caption{The Table environment: the agent (in black) must align itself to the charging station (in blue) to plug itself to it, without falling from the table by getting too close to the edges.}
    \label{fig:table}
\end{figure}


\noindent \textbf{Table} is a continuous-states and discrete-actions environment in which an agent must find and plug itself to a charging station, resulting in a +100 reward, without getting too close to the edges and fall off the table, which results in a -50 reward. The table is a one-by-one unit square. Three actions are available to the agent: go forward 0.005 units, turn left 0.1 radians, and turn right 0.1 radians. The agent observes its $(x, y)$ position and its current orientation $\theta$, expressed in radians. It starts in position $(0.1, 0.1)$ with $\theta=0.1$. The episode terminates either if the agent successfully docks itself on the charging station, i.e., if its position is $(0.5 \pm 0.05, 0.5 \pm 0.05)$ and $\theta=\frac{\pi}{4} \pm 0.3$, or after 2000 unfruitful timesteps, or if the agent falls off the table. Because the agent moves slowly compared to the size of the table, and that the reward signal is sparse, this task is extremely challenging. It is more difficult to explore than most Gym tasks, and possibly comparable in difficulty to some Atari games.

\begin{figure}
	\centering
	\includegraphics{grid1_grid2.pdf}
	\caption{The two $20 \times 18$ Five Rooms grids, ideal for transfer learning tasks. Black cells represent walls; the agent starts in the initial state S, and must reach the goal G. The difference between grid 1 and grid 2 is the location of the two doors on the path from S to G.}
	\label{fig:grids}
\end{figure}

\noindent \textbf{Five Rooms} is a $20 \times 18$ cells grid world environment (Figure \ref{fig:grids}). It is divided into five rooms, and each room is accessible via one of the four one-cell-wide doors. This, in addition to its size, makes exploration difficult. The agent can move one cell up, down, left or right, unless the target cell is a wall (then the agent does not move). The agent starts in the top-left corner of the grid, and must reach the bottom right corner, where it receives a reward of $+100$; it receives $-1$ in every other cell. The episode terminates either once the goal has been reached, or after 500 unfruitful time-steps. The optimal policy takes $35$ time-steps to reach the goal, and obtains a cumulative reward of $65$. For the transfer learning experiment, we use two slightly different variants of the same grid: in grid 1, the two main doors are centred; in grid 2, the doors are shifted.

\subsection{Policy Gradient Advised by Double DQN}
\label{subsec:dqn+pg}

\begin{figure}[t]
	\centering
	\includegraphics{plot_pg_vs_dqnpg.pdf}
	\includegraphics{results_pg_vs_dqnpg_grid.pdf}
	\caption{Our Actor-Advisor agent, in which the Policy Gradient actor is advised by the Softmax policy of a Double DQN critic, is much more sample-efficient than A3C, PPO and ACER (ACER learns after 10K episodes, so has been omitted from the plot), and outperforms the Double DQN critic used without an actor.}
	\label{fig:dqn+pg}
\end{figure}

We propose Actor-Advisor, an alternative to Actor-Critic, in which a Double DQN \citep{Hasselt2016} critic learns $Q^*$ from experiences, and provides advice to a Policy Gradient actor learning the same task. The Double DQN critic maintains an experience buffer of 20000 experiences, performs a training iteration every 16 time-steps, and replays 512 experiences per training iteration. This configuration, and intense experience replay, leads to stable and highly sample-efficient learning. At every timestep, the Softmax policy (with a temperature of 0.1) produced by the critic described above is mixed with the policy of the Policy Gradient agent. The Policy Gradient agent has a learning rate of 0.0001, and performs a gradient step every 16 episodes. Contrary to conventional Actor-Critic (Section \ref{sec:background_pg}), we keep the standard Policy Gradient loss unchanged (see Equation \ref{eq:pgg}), letting Policy Gradient learn from its returns instead of replacing them with Q-Values, that would then need to be on-policy. This addresses two important problems: it removes the bias introduced by potentially incorrect Q-Values, and it allows our highly sample-efficient off-policy critic to be used. The Actor-Advisor is much simpler than previous attempts at an actor-critic algorithm with experience replay \citep{Schulman2017,Mnih2016,Gruslys2017}, and achieves significantly higher sample-efficiencies.

We compare the Actor-Advisor against A3C \citep{Mnih2016}, ACER \cite{Wang2016b} and PPO \citep{Schulman2017}, three state-of-the-art actor-critic algorithms,\footnote{PPO optimizes an advanced policy loss, combined with a critic loss, which makes it actor-critic.} on the two environments described in Section \ref{subsec:envs}. In Figure \ref{fig:dqn+pg},\footnote{Results are averaged over 8 runs.} we show that the Actor-Advisor agent is as sample-efficient as plain Double DQN, and even displays a statistically significantly better sample-efficiency at the beginning of learning (p=1.9e-16 for Table and p=3.6e-26 for Four Rooms\footnote{We performed a Wilconxon test on 320 points in the [140, 180] episode interval for Table, and in the [40, 80] episode interval for Four Rooms.}). Additional results on the LunarLander environment with discrete actions \citep{Gym} confirm this, but were not added to this paper due to space constraints. The Actor-Advisor is, to our knowledge, the first actor-critic-like method that successfully exploits the sample-efficiency of an off-policy critic. Compared to A3C, ACER and PPO, the Actor-Advisor bridges the gap between actor-critic and critic-only algorithms, and we are confident that future improvements to the Actor-Advisor will enable it to compare more and more favorably to critic-only algorithms. In the next section, we evaluate the Actor-Advisor in other settings, where advice comes from sources other than a critic, and demonstrate its wide and successful applicability to various problems.


\subsection{Safe Backup Policy and Advice}
\label{subsec:safe}

\begin{figure}
	\centering
	\includegraphics{plot_hack_vs_nohack.pdf}
	\caption{Reinforcement Learning with a backup policy and advice, on \emph{Table}: when advice and a backup policy are used, our Actor-Advisor (\textit{Advice + Our PG}) outperforms vanilla Policy Gradient (\textit{Advice + Vanilla PG}). The advisory and backup policies are crucial to the learning performance, and our Actor-Advisor architecture is able to leverage them to learn faster and more effectively.}
	\label{fig:backup}
\end{figure}


The Table environment is an example of a simulated robotic task in which a safe backup policy, as well as some heuristic advice, could be used by the agent. We consider: i) a designer-provided backup policy, that deflects the trajectory of the agent by forcing it to choose the turn action whenever it gets dangerously close to the edges of the table (the backup policy only kicks in when the agent is too close to an edge, and facing it); and ii) an advisory policy produced by a heuristic that, when the agent is close to the goal but in the incorrect orientation, makes it turn until the (successful) end of the episode. This advisory policy is simple and illustrates the benefits to be obtained from even the most basic heuristics. Both the safe backup policy and the heuristic advice are provided to the agent in the form of directives (Section \ref{subsec:contribution_StocDet}). When none of the heuristic or backup policies is activated, a neutral advice of all ones is given to it, thereby providing no direction at all.

In Figure \ref{fig:backup}, we compare our agent (\textit{Advice + Our PG}), advised by the safe backup and the heuristic policies, with vanilla Policy Gradient advised by the same policies (\textit{Advice + Vanilla PG}). In the \textit{Advice + Vanilla PG} setting, the actions of Policy Gradient are simply overridden by the external policies whenever they intervene, and the gradient is computed only based on the agent's learned policy (Section \ref{sec:contribution}). The \textit{No advice} setting learns the task slowly; the performance of the Actor-Advisor is superior to the naive \textit{Advice + Vanilla PG} setting. This demonstrates the necessity of our method to allow Policy Shaping to be used with Policy Gradient, while still allowing the optimal policy to be learned.

\subsection{Transfer Learning}
\label{sec:experiments_transfer}

Agents advising each other can be an effective way to transfer knowledge from one task to another. We evaluate the Actor-Advisor in a transfer learning setting, where an agent learns a policy to navigate in grid 1 (Five Rooms environment, Figure \ref{fig:grids}), policy that then advises a new agent learning in the second grid. This task illustrates the real-world setting of a robot that has to move in a house that changes over time, or that has been trained in one house and has to be moved to another one. We first train a conventional Policy Gradient agent on grid 1, and save its learned policy $\pi_1$ once its entropy almost reaches 0. Then, we launch our Actor-Advisor agent on grid 2, and let the policy $\pi_1$ advise our agent on grid 2. To make sure that our agent can ignore the advice from $\pi_1$ in the parts of the environment where it is irrelevant, we use $\pi^E(s, a) = \pi_1(s, a) + 1$, with the 1 ensuring that the entropy of the advice is high enough (the Actor-Advisor always normalizes the mixture of $\pi^E$ and $\pi^L$, so an unnormalized $\pi^E$ can be used as-is). In this configuration, the actor learns to find the goal in grid 2, while the advisor $\pi_1$ remains fixed ($\pi_1$ is not updated with new data from grid 2).

We compare our Actor-Advisor architecture to a more naive approach to transferring knowledge from grid 1 to grid 2, which is to simply relaunch the PG agent confident on grid 1 (and that has learned $\pi_1$), on grid 2, and let it try to re-adjust itself to its new environment. Surprisingly, an agent confident in some task shows great difficulty to re-adjust itself to a few changes in its environment. Our Actor-Advisor agent, on the other hand, shows robustness to the partially irrelevant stochastic advice given by its advisor $\pi_1$, and successfully leverages correct advice, resulting in a much faster learning than a new PG agent freshly launched on grid 2 (Figure \ref{fig:transfer}).

\begin{figure}
	\centering
	\includegraphics{plot_transfer.pdf}
	\caption{Transfer learning: three agents evaluated on grid 2: \textit{fresh B} is a fresh Policy Gradient agent; \textit{A seeds B} is a PG agent having learned to navigate in grid 1, and then launched as is on grid 2; \textit{A advises B} is our Actor-Advisor agent freshly launched on grid 2 while being advised by a policy learned on grid 1.}
	\label{fig:transfer}
\end{figure}

\section{Conclusion and Future Work}

Despite strong theoretical convergence guarantees and regret bounds, Actor-Critic suffers from its use of an on-actor critic that learns $Q_{\pi}$, the function evaluating the current actor $\pi$, instead of $Q^*$. This prevents actor-critic algorithms from using experience replay to improve sample-efficiency, and might be the reason why well-designed critic-only algorithms exhibit much higher sample-efficiency than actor-critic ones. We propose the Actor-Advisor, a novel combination of a Policy Gradient actor with a Q-Learning-based critic, in which the actor is advised by its critic before action selection, using Policy Shaping. We extend Policy Gradient so that it can be used with Policy Shaping without convergence issue. The Actor-Advisor allows the critic to fully leverage sample-efficient off-policy algorithms with experience replay, and to advise the actor about the optimal Q-function $Q^*$. We empirically show that the Actor-Advisor, combining Policy Gradient with Double DQN, outperforms well-known actor-critic algorithms, such as Proximal Policy Optimization and A3C, and even displays better sample-efficiency than Double DQN. To demonstrate the wide applicability of the Actor-Advisor, we evaluate it in three important RL subfields, by varying the source of advice: safe RL; efficient leverage of domain knowledge; and transfer learning. We believe that this work will lead to many more future research opportunities and applications.




\section*{Acknowledgements}

The first and second authors are funded by the Science Foundation of Flanders (FWO, Belgium), respectively as 1SA6619N Applied Researcher, and 1129319N Aspirant.


\bibliographystyle{aaai}

\section{Introduction}
Policy Shaping, mixing the policy of a Reinforcement Learning agent with an external advisory policy, allows the advisory policy to immediately influence the actions taken by the agent, as opposed to reward-shaping that merely helps the agent learn faster. In this paper, we focus on two desirable properties of policy shaping: the advisory policy can force the probability of an action to be zero, therefore preventing it from being executed (\textit{safety}), and the agent, once it has learned its own policy, can assign any probability it wishes to an action having a non-zero advisory probability (\textit{robustness}). While less studied than reward shaping, policy shaping with the two above-mentioned properties has a wide application scope, depending on the source of the advisory policy: \emph{safety --} an expert designer implements a safe backup policy, that prevents the agent from performing bad actions while still allowing it to learn a good policy for the task to solve; \emph{learning from human intervention --} a person gives (sparse) orders to the agent, which has been shown to be a natural way for people to teach agents \citep{Thomaz2006}; \emph{transfer learning --} a previously learned policy advises the new one, with the new one being able to progressively ignore the advice as it becomes more confident in its own behavior; \emph{domain knowledge --} a heuristic algorithm provides a good but sub-optimal policy that helps the agent explore the environment in the early stages of learning; \emph{generalized actor-critic --} other policies, learned in parallel with the main one, using other learning algorithms (value or policy-based), provide advice to the main policy.

Until now, Policy Shaping was a practice restricted to value-based methods \cite{Griffith2013,Cederborg2015,Fernandez2006}, such as Q-learning and SARSA. However, value-based methods are incapable of ignoring a phony advisory policy, in contrast to Policy Gradient. This disqualifies critic-only algorithms (a critic being a value-based method such as Q-learning), as well as actor-critic algorithms (an actor generally being a Policy Gradient-based method); actor-only is the only option left. Unfortunately, vanilla Policy Gradient is strongly on-policy, hence requiring the actions taken to be exclusively sampled from the policy currently learned by the agent. Our contribution, Directed Policy Gradient (DPG), makes it possible for any Policy Gradient-based method to allow an external advisory policy to directly influence or determine the actions selected by the agent, without endangering convergence.

We illustrate the performance of DPG with three experiment scenarios: 1) a \emph{safety}-critical task in which the agent is prevented from falling from a platform by a backup policy, while being sporadically helped to reach the goal by (possibly human-provided) \emph{advice}; 2) a navigation task in which the agent leverages knowledge of a similar environment \emph{transferred} from another agent; 3) a novel design of the \emph{actor-critic} architecture, in which the critic no longer critiques but advises the actor. In the latter scenario, the ``actor-advisor" algorithm developed fundamentally diverges from the actor-critic concept. In our actor-advisor algorithm, the advisor directly advises which action should be taken by the actor (which the actor can ignore if needed); in actor-critic, the critic irrevocably biases the actor's estimation of the return towards potentially erroneous Q-values. Our above-mentioned experiment scenarios consist in just a first look at the novelty and application potential of Directed Policy Gradient.


\section{Background}
\label{sec:background}

In this section, we formally introduce Markov Decision Processes (MDPs), how probability distributions (in our case, policies) are combined to allow Policy Shaping, and why Policy Gradient is a more natural fit for Policy Shaping than Value-based methods. We review the work related to the various fields (namely safe Reinforcement Learning, Reinforcement Learning from human intervention, transfer learning and actor-critic algorithms) tackled by our contribution in Section \ref{sec:review}.

\subsection{Markov Decision Processes}
\label{sec:background_mdp}

A discrete-time Markov Decision Process (MDP) \cite{Bellman1957} with discrete actions is defined by the tuple $\langle S, A, R, T \rangle$: a possibly-infinite set $S$ of states; a finite set $A$ of actions; a reward function $R(s_t, a_t, s_{t+1}) \in \mathcal{R}$ returning a scalar reward $r_t$ for each state transition; and a transition function $T(s_{t+1} | s_t, a_t) \in [0, 1]$ taking as input a state-action pair $(s_t, a_t)$ and returning a probability distribution over new states $s_{t+1}$.

A stochastic stationary policy $\pi(a_t | s_t) \in [0, 1]$ maps each state to a probability distribution over actions. At each time-step, the agent observes $s_t$, selects $a_t \sim \pi(s_t)$, then observes $r_{t+1}$ and $s_{t+1}$. The $(s_t, a_t, r_{t+1}, s_{t+1})$ tuple is called an \emph{experience} tuple. An optimal policy $\pi^*$ maximizes the expected cumulative discounted reward $E_{\pi^*}[\sum_t \gamma^t r_t]$. The goal of the agent is to find $\pi^*$ based on its experiences within the environment.

\subsection{Policy Shaping}
\label{sec:background_mixing}

As described in \cite{Griffith2013}, Policy Shaping allows the agent's learned policy $\pi_L(s_t)$ to be influenced by another advisory fixed policy $\pi_H(s_t)$. The agent then samples actions from the mixture of the two policies, $\pi(s_t)$, as follows:

\begin{align} 
    \label{eq:ps}
    a_t \sim \pi(s_t) = \frac{\pi_L(s_t) \, \times \, \pi_H(s_t)} {\sum_{a \in A} \pi_L(a|s_t) \, \times \, \pi_H(a|s_t)}
\end{align}

Even if first proposed for integrating human feedback in the learning process \cite{Griffith2013}, this simple method can be applied to a larger variety of problems. In addition to a human, the advisory policy $\pi_H(s_t)$ can be a backup policy that prevents the agent from behaving undesirably, another's agent policy from which to transfer knowledge, or a policy to imitate. If the advisory policy $\pi_H(s_t)$ is deterministic (concretely, $\pi_H(s_t)$ is a vector of zeroes and one 1 for the forced action), it becomes a directive and the agent has no other choice but to follow it. On the other hand, if $\pi_H(s_t)$ is stochastic (i.e., each action in $\pi_H(s_t)$ has a non-zero probability), then it can be seen as an advice (which is softer than a directive), as it will merely bias the action selection rather than determining it. In this paper, we will use the terms advice or advisory policy, as these terms are more general, and precise that it is deterministic/stochastic when necessary.  

\subsection{Policy Gradient  versus Value-based methods}
\label{sec:background_pg}

Policy Gradient methods \cite{Williams1992,Sutton2000} explicitly learn a policy $\pi_\theta(a_t | s_t) \in [0, 1]$ parametrized by a weights vector $\theta$. The agent's goal is to maximize the expected cumulative discounted reward $E_{\pi}[\sum_t \gamma^t r_t]$, which consists in minimizing the following loss \cite{Sutton2000}:

\begin{align}
    \label{eq:pg}
    \mathcal{L}(\pi) &= -\sum\limits_{t=0}^{T} \mathcal{R}_t \log (\pi_{\theta} (a_t | s_t))
\end{align}

\noindent
where $a_t \sim \pi(a_t | s_t)$ is the action executed at time $t$. The return $\mathcal{R}_t = \sum_{\tau=t}^{T} \gamma^{\tau} r_{\tau}$, with $r_{\tau} = R(s_{\tau}, a_{\tau}, s_{\tau+1})$, is a simple discounted sum of future rewards. Intuitively, the loss leads to the probability of past actions with the highest return to be executed more often in the future, leading to a constantly improving policy. At every training epoch, experiences are used to compute the gradient $\frac{\partial L}{\partial \theta}$ of Equation \ref{eq:pg}, then the weights of the policy are adjusted one step in the direction of the gradient. Policy Gradient is said to be strongly \emph{on-policy}: it only converges if the actions executed by the agent are drawn from the exact probability distribution produced by $\pi$, which prevents any form of explicit exploration or advice. 




Q-Learning \cite{Watkins1989} maintains an action-value function $Q_{\pi} (s, a) = E_{\pi} [\sum_{k=0}^{\inf} \gamma^k r_{t+k} | s_t = s, a_t = a]$, also called a \emph{critic}, which estimates the expected return for each action in each state. In critic-only algorithms, such as Q-Learning \cite{Watkins1989} and SARSA \cite{Rummery1994}, the agent selects actions according to the Q-Values and some exploration strategy. Then, the resulting experience tuple is used to update the Q function according to this formula:

\begin{align}
    \label{eq:qlearning}
    Q_{k+1}(s_t, a_t) &= Q_k (s_t, a_t) + \alpha \delta, \\
    \nonumber
    \delta &= r_{t+1} + \gamma \max_a Q_k (s_{t+1}, a) - Q_k (s_t, a_t)
\end{align}
The SARSA update rule is obtained by replacing $\max_{a'} Q_k(s_{t+1}, a')$ with   $Q_k(s_{t+1}, a_{t+1})$. 

In contrast to Policy Gradient-based methods, off-policy algorithms (e.g., Q-Learning) tolerate the sampling of actions from a mixture (see Section \ref{sec:background_mixing}) of the agent's own policy (obtained via applying a Softmax on its Q-values) with an external advisory policy. However, due to its quest for the optimal policy, Q-Learning is incapable of avoiding deterministic advice, forcing it to choose suboptimal actions. Although, being on-policy, it does not theoretically allow for Policy Shaping, SARSA is able to avoid deterministic phony advice, as it learns based on the actions actually taken by the agent. However, since SARSA requires an artificial exploratory mechanism (such as $\epsilon$-Greedy), it is incapable of ignoring phony stochastic advice. Policy Gradient, on the other hand, being strongly on-policy and converging towards a deterministic policy, blocks the influence of phony advice both in its deterministic and stochastic form. Our main contribution, Directed Policy Gradient, extends Policy Gradient so that it can leverage external advisory policies without any convergence issue. 



\section{Related Work}
\label{sec:review}

\subsection{Safe RL}
By \emph{safety} as not entering undesirable states
Safety is inherently difficult for RL because the agent's goal is generally to maximize a reward function, which often does not incorporate safety concerns.

\subsection{Learning from Human Intervention}

RL agents have been leveraging human knowledge through Reward Shaping \citep{Ng1999}, allowing humans to teach the agent by providing feedback \citep{Thomaz2006,Knox2009,Griffith2013,Cederborg2015,Christiano2017,Pilarski2017}. However, Reward Shaping with human feedback originates from the Reinforcement Learning viewpoint, in which the human is merely a part of the environment, providing a reward signal that can easily be incorporated to the agent's reward function, regardless of how humans actually teach. Moreover, teaching through rewards is a slow process that only biases the actions selected by the agent, instead of immediately changing its behavior.

Policy Shaping has been used to let a human teacher provide feedback on the agent's actions \cite{Griffith2013,Cederborg2015}. The human feedback, expressed as binary positive or negative evaluations of the actions, is collected while the agent acts, and a critique policy is extracted from it. During action selection, this critique policy is then mixed (see Section \ref{sec:background_mixing}) with the agent's policy, learned with standard variations of the Q-Learning algorithm. In this paper, we consider human-provided advice \citep{Clouse1992,Maclin1996,Thomaz2008}, not feedback. In our setting, at every time-step, the human provides a fresh advice to the agent with a probability $L$ before any option has been selected. Preferring advice to feedback is motivated by the fact that, according to previous work analysing how people teach \cite{Thomaz2006,Kaochar2011}, humans actually seldom use feedback as a teaching tool; they would rather tell the agent \emph{what} to do, instead of whether it did well.  


\subsection{Transfer Learning}
From a RL perspective, Transfer Learning allows to generalize across tasks, and to reuse a previously learned policy in a given environment in a new similar environment \cite{Taylor2009}. Policy Shaping has been used to allow an agent to leverage a previously learned policy, while learning a new tasks \citep{Fernandez2006}. In \cite{Fernandez2006}, a past policy is reused by a Q-Learning agent with probability $\psi$, and greedily exploits the currently learned policy with probability $1-\psi$. However, compared to the mixture formula we use (see Section \ref{sec:background_mixing}), this alternative presents an important issue: it requires the setting of a hyperparameter $psi$, forcing the designer to tune and fix the impact that the external policy will have. In contrast, simply mixing policies together, combined with the capability of Policy Gradient to self-tune its exploration, leads to an automatic, natural convergence towards an optimal policy, unpolluted by fixed hyperparameters. 

Nevertheless, it must be noted that Policy Shaping remains a limited solution for transfer learning, as it requires the old task and the new task to share the same state and action spaces. 

\subsection{Actor-Critic Algorithms}

Actor-Critic methods \cite{Barto1983,Konda1999} learn both a policy (actor) and a value function (critic). The policy selects actions to be executed in the environment, while the value function evaluates the actor's current policy. Current actor-critic algorithms learn their critic using Q-Learning (see Equation \ref{eq:qlearning}), and their actor using a variant of Policy Gradient, where the return (see Equation \ref{eq:pg}) is replaced with a Q-Value, such as:

\begin{align}
	\label{eq:ac}
	\mathcal{L}(\pi) &= -\sum\limits_{t=0}^{T} Q(s_t, a_t) \log (\pi_{\theta} (a_t | s_t))
\end{align}

The main advantage of Actor-Critic methods is that they allow experience replay to be used by the critic, thereby improving its data-efficiency. The actor, by using Q-Values (low-variance estimates of future returns) instead of actual returns of higher variance, is able to compute a more informative gradient. As such, actor-critic algorithms have obtained state-of-the-art results on several challenging tasks \cite{Wang2016b,Gruslys2017,Mnih2016,Schulman2017}. Unfortunately, Q-Learning is known for its tendency to overestimate its action values, which can harm performance. This bias towards overestimation is analysed in \cite{Hasselt2010}, and a solution is proposed, said in \cite{Anschel2017} to actually result in an underestimation of action values. 

We propose a novel way of combining Policy Gradient with Q-Learning, in which the Q-Learning agent (called advisor instead of critic) provides advice to the Policy Gradient agent (actor), while learning in parallel. In our setting, Equation \ref{eq:qlearning} is left unchanged, leading to a twofold advantage: 1) Policy Gradient learns quicker by leveraging advice coming from action values; and 2) Policy Gradient has the possibility to converge towards a policy optimizing its returns, regardless of the biased action values. Letting Policy Gradient learns from its returns ensures that it remains capable of blocking the influence of wrong Q-values, which is not guaranteed by Actor-Critic algorithms. Moreover, this Actor-Advisor architecture allows for experience replay to be used by the Advisor. 

\section{Directed Policy Gradient}
\label{sec:contribution}

Because Policy Gradient allows a policy to be learned directly from experiences, without needing a value function, it is a perfect fit for Policy Shaping. However, Policy Gradient requires the actions taken by the agent to be directly sampled from the policy $\pi$ \cite{Sutton2000}, which prevents any artificial exploration or backup policy to be implemented.

Directed Policy Gradient assumes a parametric policy represented by a neural network. The neural network has two inputs: the state $s_t$ and an \emph{advice} $\pi_H$. The state is used to compute a probability distribution $\pi_L$ over actions, using dense, convolutional or any other kind of trainable layers. Then, the output $\pi$ of the network is computed by element-wise multiplying $\pi_L$ and $\pi_H$, as suggested in the Policy Shaping literature \cite{Griffith2013}. As a result, the distribution $\pi$ output by the network is a mixture of the agent's policy and some advice, and actions can therefore be directly sampled from that distribution. Note that $\pi_H$ is not fed to the trainable part of the network, and as such cannot be considered as an extension of the state-space. Thus, the network training (or optimization) problem is not made harder by the use of $\pi_H$. In this neural architecture, inspired from how variable action-spaces are implemented in \cite{Steckelmacher2018}, all the Policy Gradient assumptions are met, yet the behavior of the agent can be directly altered by a teacher or a backup policy. Moreover, our experimental results in Section \ref{sec:experiments} demonstrate that Directed Policy Gradient is able to leverage advice to learn faster, instead of merely obeying instructions and then learning as if nothing happened.

More specifically, the policy over options $\pi$ is represented as a feed-forward neural network (with one hidden layer of 100 neurons in our experiments), trained using Policy Gradient \cite{Sutton2000} and the Adam optimizer \cite{Kingma2014}. The neural network $\pi$ takes two inputs: $\mathbf{s}$ the state observed from the environment, and a $\mathbf{\pi_H}$ probability distribution over actions used for Policy Shaping. While $\mathbf{\pi_H}$ usually comes from a human (in the form of a deterministic distribution corresponding to the action advised by the human), it can also be a backup policy, a policy to be imitated, actions from demonstration trajectories, or a mixture of all that. If no advice is available, $\pi_H$ is set to the uniform distribution, which cancels its effect and allows Policy Gradient to learn as usual. The output of the $\pi$ network is computed as follow:

\begin{align} 
    \label{eq:ps+pg}
    \mathbf{h_1} = \tanh(\mathbf{W_1} \mathbf{s} + \mathbf{b_1}), \\
    \mathbf{\hat{y}} = \sigma(\mathbf{W_2 h_1} + \mathbf{b_2}) \circ \mathbf{\pi_H}, \\
    \mathbf{y} = \frac{\mathbf{\hat{y}}} {\mathbf{1^T \hat{y}}},
\end{align}

with $\mathbf{W_i}$ and $\mathbf{b_i}$ the weights and biases of layer $i \in [1,2]$, $\sigma$ the sigmoid activation function, and $\circ$ denoting the element-wise product of two vectors. The network is trained using the standard Policy Gradient loss \cite{Sutton2000}:

\begin{align}
    \label{eq:pgg}
    \mathcal{L}(\pi) &= -\sum\limits_{t=0}^{T} \mathcal{R}_t \log (\pi(a_t | s_t, \pi_H))
\end{align}

with $a_t \sim \pi(s_t)$ the action taken at time $t$, the return $\mathcal{R}_t = \sum\limits_{\tau=t}^{T} \gamma^\tau r_\tau$, with $r_\tau = \mathcal{R}(s_\tau, a_\tau, s_{\tau+1})$, a simple discounted sum of future rewards. We now demonstrate the applicability of Directed Policy Gradient to various settings, and illustrate how much less human interventions are needed for Directed Policy Gradient to be useful, compared to reward-based approaches.

\section{Experiments}
\label{sec:experiments}

Because our main contribution, Directed Policy Gradient, does not rely on rewards, values, classifiers or value functions, its properties are quite different from existing algorithms, that we review in Section \ref{sec:review}. Our experiments therefore thoroughly explore the properties of Directed Policy Gradient, and are divided in three groups:

\begin{enumerate}
  \item We compare Directed Policy Gradient with policy shaping from a simulated human advisory policy, to simple Policy Gradient with reward shaping from a simulated human reward function, and to Policy Gradient without any human intervention (Section \ref{subsec:expcompare}).
  \item We quantify the impact on Directed Policy Gradient of varying the hyper-parameters of Policy Shaping: the probability of giving advice $L$ and the probability that the advice is correct $P(right)$ (Section \ref{subsec:expparameters}). 
  \item We consider a bounded amount of human advice, compatible with a real-world deployment of Directed Policy Gradient, and show that the agent can still learn even if the amount of advice is heavily reduced (Section \ref{subsec:expstop}).
\end{enumerate}

\subsection{Five Rooms Environment}
\label{subsec:env}



The evaluation of our method takes place in the Five Rooms environment (see Figure \ref{fig:fiverooms}a), inspired from the well-known Four Rooms environment used to evaluate options without human intervention \cite{options}. Five Rooms is a 29 cells high and 27 cells wide grid world, divided by walls. Each of the five rooms are accessible via one of the four one-cell-wide doors. This, in addition to its size, makes exploration difficult. In a conventional setting, the agent can move one cell up, down, left or right, unless the target cell is a wall (then the agent does not move). The agent starts in the top-left corner of the grid, and must reach the bottom right corner, where it receives a reward of $+100$; it receives $-0.1$ in every other cell. The episode terminates either once the goal has been reached, or after 500 unfruitful time-steps. The optimal policy takes $54$ time-steps to reach the goal, and obtains a cumulative reward of $100 - 54 \times (-0.1) = 94.6$.

Five rooms is a fairly large (in comparison to Four rooms in Figure \ref{fig:fiverooms}b) and difficult to sufficiently explore, due to the sparse reward. As the original Four Rooms environment \cite{options}, and because of its sheer size, Five rooms is well-suited to the use of options. Our use of a complicated environment, and options, is motivated by the nature of real-world robotic tasks in human-populated environments, that are big, complex, and already rely on options. Note that options are not necessary for Directed Policy Gradient to work, but our experiments show, using this environment, that our method is compatible with Hierarchical Reinforcement Learning. We see this as an important benefit, as it is natural for humans to provide feedback on the level of options. We implemented 5 options: 4 door-options, each associated to one of the 4 doors, driving the agent directly to that door from any neighboring room, and then terminating. One option goes to the goal from any cell of the bottom room. Instead of choosing among the four \texttt{up, down, left, right} low-level actions, the agent's policy selects an option among these 5 ones. If the chosen option is defined for the agent's current position, the option leads directly to its respective destination. Otherwise, a random action is executed, after which the option immediately terminates and returns control to the policy over options. As a result, the agent visits potentially every cell of the grid, instead of just hopping from door to door, keeping the problem challenging\footnote{The task is much more challenging than if initiation sets \ref{sec:background_options} were used, as the agent has access to every option at any moment}.

\subsection{Simulated Human Advice}
\label{subsec:advice}


In our experiments, we use a function simulating a human teacher providing advice to the agent in a deterministic fashion. To design this function, we got inspiration from a small sample of actual people giving advice to a learner. Before the agent chooses an option, the human has the possibility of telling the agent which option to execute. This human advice is represented as a probability distribution over options, with a probability 1 for the option that the human wants the agent to execute, and 0 for every other option. The $\pi_H(s_t)$ advice is then mixed to the learned policy $\pi_L(s_t)$, as detailed in Section \ref{sec:contribution}.

Because real humans may not always be present, or attentive, the human has a probability $0 \le L \le 1$ of providing advice to the agent, and a probability $0 \le P(right) \le 1$ of providing correct advice. A \emph{wrong advice} consists in the teacher advising the agent to take the door leading to the middle left room, regardless of the agent's current location, which takes the agent away from the direct path to the goal.

\subsection{Directed Policy Gradient and Reward Shaping}
\label{subsec:expcompare}

\begin{table}[b]
	\centering
	\setlength\tabcolsep{3pt}
	\begin{tabular}{|l|l|c|c|}
		\hline
		Test & Episode Number & Human reward & Human advice \\
		\hhline{|=|=|=|=|}
		t-test & [180, 220] & 0.343 & 0.001  \\
        & [960, 1000] & 0.035 & 1.464e-06 \\
		\hline
		Wilcoxon & [180, 220] & 0.419 & 1.369e-03 \\
        & [960, 1000] & 0.007 & 2.396e-09\\
        \hline
	\end{tabular}
    \vspace{3mm}
	\caption{P-values comparing the returns obtained by Policy Gradient (PG) to PG + Human Reward and PG + Human Advice, in the early and late stages of learning. The returns obtained by PG + Human Advice are significantly higher than the ones obtained by PG + Human Reward, both in early and late stages of learning. Each test is performed on 41 (episodes) $\times$ 16 (runs) samples of cumulative reward values, taken in one of two given episode intervals.}
	\label{tab:pvalues}
\end{table}

In this section, we compare Directed Policy Gradient to simple Policy Gradient with reward shaping, and Policy Gradient without any form of human intervention.

Our deterministic simulated human reward shaping works as follows, and is given to the agent only on some time-steps, with a probability of $L = 0.05$ (this leads to about 1000 human interventions over 1000 episodes). At time-step $t$, the agent chooses an option $a_t$. The human teacher has access to the agent's current position in the grid, and knows which option $a^*$ is the correct one given that position. She allows (by giving 0) or punishes (by giving -5) the agent depending on whether $a_t = a^*$. This numerical human reward is then added to the environmental reward. Although it is not potential-based, this does not prevent the agent from learning the task; and making it potential-based \cite{Harutyunyan2015} would require modeling the human's reward function, which might alter it. Human advice is given to Directed Policy Gradient with the same probability $L = 0.05$, which also results in about 1000 human interventions over 1000 episodes.

Results in Figure \ref{fig:PGvsHPvsHR} show that, in the beginning, the performance of the agent is not improved by human reward; however performance seems to be slightly improved towards the end of the 1000 episodes (p-values are given in Table \ref{tab:pvalues}). On the other hand, Directed Policy Gradient leads to much higher returns early on, and continues to dominate reward shaping and simple Policy Gradient until the end of the 1000 episodes. This experiment shows that, in contrast to reward-based methods, even a small amount of human advice significantly improves the agent's performance. The next experiments explores the behavior of Directed Policy Gradient when various amounts of human advice is given.

\subsection{Sensitivity to Hyper-Parameters}
\label{subsec:expparameters}



Different settings lead to varying kinds of human intervention given to the agent. In this experiment, we vary the value of $L$, the probability that the teacher gives advice, and $P(right)$, the probability that the advice is correct. We show that the most interesting results are obtained for values of $L$ that are not too large. If $L$ is too large, the agent is controlled by the human and learns to imitate her, but this is only useful if we consider the human to be the main controller, and the agent is only used to ``fill in the gaps'' when the human is unavailable. Hence, we keep the value of $L$ small ($L=0.2$ maximum) in order to fully emphasize the learning ability of Directed Policy Gradient, in a setting where imitation learning is not enough.

Figure \ref{fig:hyperparams} compares various configuration of Directed Policy Gradient, with a high or low probability of giving advice, and a high or low probability of giving correct advice. The main results are:

\begin{enumerate}
  \item When the advice delivered to the agent is correct, larger values of $L$ lead to higher returns. Similarly and unsurprisingly, when incorrect advice is delivered, a small $L$ allows the agent to overcome bad advice and still learn a good policy.
  \item The agent always manages to learn a reasonable policy, even when most of the advice is incorrect ($P(wrong) = 0.8$), regardless of $L$.
\end{enumerate}

The second point is the most interesting, as it demonstrates that Directed Policy Gradient is robust to wrong advice. Adding advice to an agent is therefore free: scarce advice can only improve the performance of the agent, and wrong advice does not prevent the optimal policy from being eventually learned. The next experiment shows that Directed Policy Gradient is even robust to non-stationary advice, contrary to most reward-based approaches (see Section \ref{sec:review_challenges}).

\subsection{Non-Stationary Advice}
\label{subsec:expstop}

\begin{table}[b]
	\centering
	\setlength\tabcolsep{3pt}
	\begin{tabular}{|l|c|}
		\hline
		$L$ & p-value (Wilcoxon)  \\
		\hhline{|=|=|}
		0.05 & 0.047   \\
        \hline
        0.2 & 1.294e-07 \\
        \hline
        1 & 2.035e-08 \\
        \hline
	\end{tabular}
    \vspace{3mm}
	\caption{P-values comparing the returns obtained by Policy Gradient (PG) to PG + Human Advice when advice is interrupted after 700 human interventions, on 41 (episodes) $\times$ 16 (runs) samples in the late stage of learning (episode interval [960, 1000]). The performance of PG + Human Advice is significantly higher than of Policy Gradient for the highest values of $L$. Interestingly, there is little significance difference between $L=0.2$ and $L=1$.}
	\label{tab:pvalues2}
\end{table}


In many real-world settings, humans are not always available to help the agent learn. In this experiment, we interrupt advice after 700 human interventions. This assesses the robustness of the agent to advice that changes (abruptly) over time, and mimics human advice obtained over one or two hours of on-the-field training, before the human trainer leaves, possibly to set-up another agent in another company.


In Figure \ref{fig:stopafter700}, the performance of the agent is very high while it is trained, as human advice can be directly followed, leading to high returns. This shows that the trainer is able to completely override the agent's behavior, which is crucial for safe Reinforcement Learning. When advice stops, the agent's performance drops, then consistently recovers, and even surpasses Policy Gradient's performance without human intervention (as is suggested by the statistical significance test values in Table \ref{tab:pvalues2}). This empirical result shows the robustness of our technique and suggests that Policy Shaping allows users to quit helping the learner whenever their interest fades: the agent will still be able to discover a good policy. More importantly, the final performance of the agent is higher when advice has been available at some point, which demonstrates that Directed Policy Gradient learns from the advice, instead of merely following it. Interestingly, the more intensively the agent is helped (i.e., the higher the $L$), the better it recovers afterwards. In a real-world setting, intense and dedicated advice (but over a short period of time) therefore seems to be the way to go.


In contrast to reward shaping from human feedback, interrupting advice does not necessarily result in a performance loss in the long run. In Figure \ref{fig:rewardvsadvice_interruption}, we compare human advice, interrupted after 700 interventions, to reward shaping, interrupted after 10,000 punishments (so, much more than 10,000 time-steps during which the human had to watch the agent). Once the 10,000 human punishments have been consumed, the performance of Policy Gradient with human reward slightly decreases, then plateaus until the end. Using only 700 pieces of advice, Directed Policy Gradient manages to learn a policy marginally better than the one obtained with reward shaping, even if reward shaping requires one order of magnitude more human dedication. Furthermore, Directed Policy Gradient is simpler to implement (see Section \ref{sec:contribution}) than reward shaping, which ideally requires a potential-based reward function to be designed \cite{Ng1999,Harutyunyan2015} to avoid biasing the learned policy. 


\section{Conclusion and Future Work}

This paper presents Directed Policy Gradient, an extension of Policy Gradient that allows an advisory policy $\pi_H$ to directly influence the actions selected by the agent. We illustrate Directed Policy Gradient in a human-agent cooperation setting, where the advisory policy is defined by a human. We show that Directed Policy Gradient allows good policies to be learned from scarce advice, is robust to errors in the advice, and leads to higher returns than no advice, or reward-based approaches. Contrary to other approaches, reviewed in Section \ref{sec:review}, Directed Policy Gradient uses the environmental reward, and still allows the optimal policy to be learned. Finally, although we used the example of a human advisory policy, and compared our work to other human-based approaches, it is important to note that any advisory policy can be used to shape a learner's policy, such as expert demonstrations, policies to be distilled from other agents, backup policies for Safe Reinforcement Learning, or a mix of all the above. Directed Policy Gradient is therefore a straightforward, effective, and widely applicable approach to policy shaping.

On a higher level, Directed Policy Gradient is an original approach to learning from human intervention. It uses immediate advice instead of feedback or demonstrations, and incorporates that advice in the learning process by directly shaping a policy, instead of producing rewards or a value function. Directed Policy Gradient is also easier to implement. For instance, a theoretically correct implementation of reward-shaping requires that the reward function is potential-based, which requires advanced algorithmic precautions to be implemented \cite{Harutyunyan2015}. Such precautions are not needed by Directed Policy Gradient. Moreover, incorporating human input through the reward function leads to slowly changing the agent's policy, in contrast to directly influencing the agent's policy. In this regard, our work is pushing forward two promising (but not yet well-investigated) paradigms, namely advice as a tool for humans to teach agents, and policy shaping as a method to learn from external directives.

Finally, as future work, we will build on the most important property of Policy Gradient, its ability to work with continuous actions, to develop a continuous-action version of Directed Policy Gradient. Such an extension would be applicable to many real-world problems, such as autonomous cars, fork-lift robots, or prosthetics, as these settings are mostly continuous-action.

\begin{acks}

The first author is supported by the Vrije Universiteit Brussel.
The second author is ``Aspirant'' with the Science Foundation of Flanders (FWO, Belgium), grant number 1129317N.

\end{acks}

