\section{Story.}




\section{Introduction}
\label{submission}


Multi-agent reinforcement learning (MARL) addresses the sequential decision-making of two or more autonomous 
agents that operate in a common environment, each of which aims to optimize its own long-term return 
by interacting with the environment and with other agents \citep{busoniu2008comprehensive}. 
MARL is becoming more common in many real-world applications such as robotics, as well as in other applications involving
complex, dynamic environments, such as video games.
Largely, 
MARL algorithms can be placed into two categories depending on the type of setting they address.
In the \emph{cooperative} setting, agents collaborate to optimize a shared long-term return
whereas in the \emph{competitive} setting, an advantage for one agent results in a loss for another.


Early research in cooperative MARL focused on agents that operated independently and that did not explicitly communicate \cite{IPPO, IAC, IQL}. However, when each agent has only a partial view of the environment, agents benefit from exchanging information with one another, allowing
them to construct a more complete belief state and improve decision making.
Even in fully observed environments, inter-agent communication can be beneficial to coordinate behavior.
Not surprisingly, performance advantages are obtained when agents have the ability to learn a communication protocol, whether implicitly 
or explicitly \cite{qmix,vdn,lowe2017multi,IPPO}. 
Recent research on communication for deep MARL adopts an end-to-end training procedure based on a differentiable communication channel 
\citep{sukhbaatar2016learning, foerster2016learning, gupta2017cooperative, singh2018learning, das2019tarmac}.  
Essentially, each agent has the capability of generating messages and these messages can influence other agents' policies.
In these works, the message-generation subnet is trained using the gradient of other agents' policy or critic losses.

In designing a MARL architecture, a key decision involves the nature of the communication channel.
The simplest scenario involves direct agent-to-agent communication, where each of the $N$ agents can receive messages from all other 
agents.  Communication costs are quadratic in $N$ and each agent faces the challenge of interpreting $N-1$ simultaneous messages.
Communication costs can be reduced by restricting the communication topology \citep{wang2017nonlocal}. 
Message processing can be simplified using a learned key-value attention mechanism that condenses messages at either the side
of the sender \citep{kim2020communication} or recipient \citep{das2019tarmac}.

\textbf{Intelligent communication channel.}
In previous approaches, the communication channel is \emph{passive}, by which we mean that its role is to convey whatever message passes through the channel without alteration.  We describe an approach in which the communication channel is 
\emph{active} in that it can interpret and transform signals from one or more agents, and it is \emph{stateful} in  that its interpretations can depend on the recent history of messages transmitted.  Because we endow the channel itself 
with intelligence, message communication complexity is reduced from quadratic to linear in $N$.
In essence, the channel is a specialist agent aiming to \emph{facilitate} coordination among the other agents.
We refer to the channel as \emph{{\textsc{saf}}}, an acronym on \emph{stateful, active facilitator}.
{\textsc{saf}}\ is itself adaptive and learns to improve the collective performance of the agents. 













\begin{figure*}[ht!]
    \centering
    \includegraphics[width=\linewidth]{figures/approach_figure.pdf}
    \caption{Agents communicate through the \emph{{\textsc{saf}}} and pick a policy from the shared policy pool. First, each agent generates a message and competes for write-access into the \emph{{\textsc{saf}}}. Next, all agents read messages from the \emph{{\textsc{saf}}} and use it with their internal state to pick a policy from the shared policy pool.}
    \label{fig:illust1}
\end{figure*}

\textbf{Maximizing agent independence.}
By endowing the communication channel with intelligence, there is a risk that {\textsc{saf}}\ may simply become a centralized controller 
dictating actions to the agents, which is antagonistic to a multi-agent architecture. We therefore need to encourage 
independence of the agents, which could also lead to specialization of labor and thus faster learning. To the extent that independent agents can solve a task, independence is a clear advantage because the 
learning problem can be decomposed into smaller problems, i.e., each agent can learn without concern about the behavior of other agents.  However, most tasks require \emph{some} coordination among agents. We therefore pursue an approach 
that attempts to regularize toward independence: agents pay a penalty for modulating their behavior based on the information 
provided by {\textsc{saf}}. This penalty, which is added to the primary task-based reward, discourages unnecessary use of the 
communication channel.




The penalty is expressed as the conditional mutual information between an agent's behavior, $B$, and the information, $M$,
obtained from {\textsc{saf}}, given the agent's current belief state, $S$, denoted $I\!\left(B; M \mid S\right)$. 
Previous work has shown how to optimize $I\!\left(B;M \mid S\right)$ using the framework of KL-Regularized RL  \citep{teh2017distral,  galashov2019information, goyal2019infobot, tirumala2020behavior}. 
This optimization encourages the agent to act according to a \emph{default} or \emph{prior} policy that is insensitive to {\textsc{saf}}.
 To see that minimizing $I\!\left(B;M \mid S\right)$ is achieved by minimizing the KL divergence between an agent's policy
 and the default one, note that
$I\!\left(B;M \mid S\right) = \mathbb{E}_{\pi_\theta}\left[D_{\text{KL}}\!\left[\pi_\theta \!\left(B \mid S,M\right) \mid \pi_0 \!\left(B \mid S\right) \right]\right]$, 
where $\pi_\theta \!\left(B \mid S,M\right)$ is the agent's {\textsc{saf}}-sensitive policy,
$\pi_0 \!\left(B \mid S\right)= \pi_\theta \!\left(B \mid S\right)$  is the default policy,
$\mathbb{E}_{\pi_\theta}$ denotes an expectation over trajectories generated by $\pi_\theta$, and
$D_{\text{KL}}$ is the  Kullback-Leibler divergence.
In past work, the information asymmetry obtained by prohibiting the default policy access to shared information
has been shown to improve transfer and generalization
\citep{galashov2019information, goyal2019infobot, tirumala2020behavior}. 

\textbf{Policy switching}. We were deliberately vague in describing $B$ as an agent's behavior.  $B$ might refer to the actions an agent takes \citep[as it has in][]{goyal2019infobot, galashov2019information, tirumala2020behavior}, but in the present work, we endow agents with a discrete set of distinct policies to select from, and the agents have an explicit decision-making
component that selects a policy stochastically conditioned on $S$ and $M$.  The communicated information $M$ is used only
to select a policy, not to select actions conditioned on the policy. This mixture-of-policies approach 
limits the manner in which {\textsc{saf}}\ can micromanage an agent's behavior, and it has been shown to be effective in
endowing agents with different behavior modes \citep{goyal2021scoff,tirumala2020behavior}.

In the previous paragraphs, we introduced three key ideas that operate synergistically. First, communication among agents is via an intelligent channel, {\textsc{saf}}. Second, each agent is incentivized to act independently and avoid relying on {\textsc{saf}}. Third, agents
operate according to a mixture-of-policies, where {\textsc{saf}}\ provides the signal to select the policy.
To better appreciate how these ideas work in tandem, consider the naturalistic example of a herd of deer coordinating their
behavior. By default, each animal's policy is to graze in a field. But when one of the animals senses danger, the herd
needs to escape. They need to move in unison because if they split up, it will be easier for a predator to trap them.
Suppose that they can escape either to the north or the east, each characterized by a different policy. {\textsc{saf}}\ in
this case will relay the danger alert and will collect information from the individuals and suggest a direction so that
the deer can escape in coordination. Each deer is responsible for avoiding obstacles and avoiding running into other deer,
and thus they operate autonomously with only the high-level guidance from {\textsc{saf}}.

\vspace{-3mm}
\paragraph{Contributions.} The key contributions of our work are as follows: (a) we propose a novel architecture for multi-agent RL. Instead of agents communicating directly between one another, communication is mediated by a facilitator, {\textsc{saf}}, which itself uses the history of communication and active computation to improve the collective performance of all the agents, (b) To ensure agent autonomy, different agents are incentivized to minimize the influence of {\textsc{saf}}\ on their behavior, (c) To further promote autonomy, agent behavior is only coarsely modulated by {\textsc{saf}}, much in the way that a Ph.D. student's research direction is only guided at a high level by their advisor. In the case of our MARL architecture, this modulation  comes in the form of policy selection, which is made explicit via a policy mixture model with a component that switches among policies (see Figure \ref{fig:illust1}), (d) We show the performance of the proposed method on different MARL environments in cooperative setting. We also conduct various ablations and understand the role of different components namely the use of intelligent channel \emph{{\textsc{saf}}}, maximizing agent independence and the use of policy pool. We show that the collective performance of the agent which uses all the three components is higher as compared to an agent which only uses one of the components. 

\iffalse
\begin{itemize}
    \item We propose a novel architecture for multi-agent RL. Instead of agents communicating directly between one another, communication is mediated by a facilitator, {\textsc{saf}}, which itself uses the history of communication and active computation to improve the collective performance of all the agents. 
    \item To ensure agent autonomy, they are incentivized to minimize the influence of {\textsc{saf}}\ on their behavior.
    \item To further promote autonomy, agent behavior is only coarsely modulated by {\textsc{saf}}, much in the way that a Ph.D. student's
    research direction is only guided at a high level by their advisor. In the case of our MARL architecture, this modulation 
    comes in the form of policy selection, which is made explicit via a policy mixture model with a component that switches 
    among policies (see Figure \ref{fig:illust1}).
    \item We show the performance of the proposed method on different MARL environments in cooperative setting. We also conduct various ablations and understand the role of different components namely the use of intelligent channel \emph{{\textsc{saf}}}, maximizing agent independence and the use of policy pool. We show that the collective performance of the agent which uses all the three components is more as compared to an agent which only uses one of the components. 
   
    
   
   
   
\end{itemize}

\fi






 
















\section{Background and Notation}

In this work, we consider a multi-agent version of Markov decision processes (MDPs) with partially observable Markov environments (POMDP).
The environment is defined as $(\mathcal{N},\mathcal{S},\mathcal{O},\mathcal{A},\Pi,R,\gamma)$. $\mathcal{N} = \{1,...,N\}$ denotes the set of $N>1$ agents. $\mathcal{S}$ describes all possible states of all agents.  $\mathcal{A} = A_1\times\dots\times A_N$ denotes the joint action space and $a_{i,t}\in A_i$ refers to the action of agent $i$ at time step $t$. $\mathcal{O}=O_{1}\times\dots\times O_{N}$
denotes the set of partial observation where $o_{i,t}\in O_i$ stands for partial observation of agent $i$ at time step $t$. $\Pi$ is the set of policies available to the agents. To choose actions, agent $i$ uses a stochastic policy $\pi_{\theta_i}:O_i \times A_i \mapsto [0,1]$. Actions from all agents together produces the transition to the next state according to transition function $\mathcal{T}:(s_t, a_{1,t}, \dots, a_{N,t}) \mapsto s_{t+1}$ where $s_t\in\mathcal{S}$ is the joint state at timestep $t$. 
$R:\mathcal{S}\times\mathcal{A}\mapsto \mathbb{R}$ is the global reward function conditioned on the joint state and actions. At timestep $t$, the agent team receives a reward $r_t=R(s_t,a_{1,t},\dots,a_{N,t})$ based on the current total state $s_t$ and joint action $a_{1,t},\dots, a_{N,t}$. $\gamma$ is the discount factor for future rewards. In this study, we focus on cooperative MARL with partial
observations.









\section{MARL with a Facilitator}

We present our approach, first in terms of a high level overview of the mechanisms enabling inter-agent communication
(Section~\ref{sec:hlo}), and then with a detailed description that steps through the algorithm (Section~\ref{sec:da}).
For further details, see Algorithm \ref{alg:attention} in Appendix~\ref{appx:alg}.


\subsection{High Level Overview}
\label{sec:hlo}



 \textbf{Stateful active facilitator.} The facilitator, \emph{{\textsc{saf}}}, consists of 
$l$ stateful memory \emph{slots}, each a vector of $d_m$ elements. The slots are randomly initialized at the beginning of an episode. The state of the \emph{{\textsc{saf}}} is updated once per time step, where a time step corresponds to all agents performing one action. In this update, \emph{{\textsc{saf}}} integrates information provided by the agents into its slot memory and then
outputs a message $M$ that any agent can use.


\textbf{Policy switching using a shared policy pool.} Although all agents operate in the same environment, 
their contexts and objectives may vary. A policy pool $\Pi = \{ \pi^1, \ldots, \pi^U\}$, shared among agents, 
enables agents to exhibit diverse behaviors and have distinct goals. Each policy $\pi^u \in\Pi$ is associated with a 
learned signature key  $k_{\pi^u}$. Using differentiable hard attention with  Gumbel-softmax \citep{jang2016categorical},
an agent can  dynamically select one of  the policies at each time step via a query formed from its belief state, $S$, 
and the message conveyed by {\textsc{saf}}, $M$. 

\textbf{Reducing agent dependence on {\textsc{saf}}.} To reduce an agent's dependence on {\textsc{saf}}, each agent is penalized
according to the KL divergence $D_\text{KL} [ \Pr(Z \mid S,M) \mid\mid \Pr(Z \mid S) ]$, where $Z$ is the agent's policy choice.  







\subsection{Detailed Algorithm}
\label{sec:da}
\textbf{Step 1: Agents pass messages to {\textsc{saf}}.} At step $t$, each agent $i$ receives a partial observation $o_{i, t}$ 
of the environment. This observation is used to update the agent's belief state, $s_{i, t}$, which in turn is
used to generate a message for {\textsc{saf}}.  The message,  $m'_{i, t} = g_{enc}(s_{i, t})$ is a vector of dimensionality
${d_m}$. We denote the set of  messages generated by the agents at time step $t$ as 
${\bm{M}}'_t \equiv \{m'_{i, t} | 1 \leq i \leq N\}  \in \mathbb{R}^{N\times d_m}$.





\textbf{Step 2: \emph{{\textsc{saf}}} integrates information from the agents.} 
\emph{{\textsc{saf}}} dynamically integrates the information from all the agents and incorporates the information 
that it finds interesting into its state representation. The \emph{{\textsc{saf}}} slot memory at time step $t$ is a set of vectors 
$F_t \in \mathbb{R}^{l \times d_m}$, each row representing one of the $l$ slots. The state of 
\emph{{\textsc{saf}}} is updated based on agent messages, ensuring that only the important 
information is incorporated.  {\textsc{saf}}\  achieves this via the use of query-key-value attention mechanism 
\citep{bahdanau2014neural, vaswani2017attention}.  In this case, the query is a function of {\textsc{saf}}'s state 
(a set of slots), represented by matrix ${\bm{F}}_{t}$ (with one row per slot), i.e., $\widetilde{{\bm{Q}}} = {\bm{F}}_{t} \widetilde{{\bm{W}}}^{q}$.  Keys and values are a function of the messages from individual agents. Dot product attention is applied
to obtain the updated state of the slots:  ${\bm{F}}_{t} \leftarrow \mathrm{softmax}\left(\frac{\widetilde{{\bm{Q}}}({\bm{M}}'_{t} \widetilde{{\bm{W}}}^{e})^\mathrm{T}}{\sqrt{d_e}}\right){\bm{M}}'_{t} \widetilde{{\bm{W}}}^{v}$. The use of softmax to write into $l$ slots leads to a standard soft competition among agents to influence the state of \emph{{\textsc{saf}}}. Next, self-attention
is applied over the slots of {\textsc{saf}}\ to obtain its updated state.



\textbf{Step 3: \emph{{\textsc{saf}}}  modulates the behavior of other agents.} \emph{{\textsc{saf}}} makes the updated state available to the agents should they deem to use it. We again utilize an attention mechanism to perform this reading operation. All the agents create queries $Q^s_{t} = \{q^s_{i, t} | 1 \leq i \leq N\}\in\mathbb{R}^{N\times d_e}$ where $q^{s}_{i,t}$ is query generated using the encoded partial observations of agent $i$: $q^{s}_{i, t} = W^{q}_{write}s_{i, t}$. Generated queries are matched with the keys $\kappa = F_t {\bm{W}}^{e}\in\mathbb{R}^{l\times d_e}$ from the updated state of \emph{{\textsc{saf}}} (a set of slots), forming the attention matrix
\mbox{$\alpha= \mathrm{softmax} \left( \frac{Q^s_t \kappa^T
   }{\sqrt{d_e}}\right)$}. 
The slot values generated by each slot of \emph{{\textsc{saf}}}'s state  and the attention weights are then made available to all the agents: $$
{\bm{M}}_t = \mathrm{softmax} \left( \frac{Q^s_t \kappa^T
   }{\sqrt{d_e}}\right) {\bm{F}}_t W^v 
$$

Here ${\bm{M}}_t=  \{m_{i, t} | 1 \leq i \leq N\}$, where $m_{i, t}$ is the message made available to the agent $i$.


\textbf{Step 4: Policy switching.} The encoded partial observation of an agent $s_{i, t}$ and the information made available to  each agent $m_{i, t}$ is used to select a policy $\pi^{u}$ for that agent from the pool of policies $\Pi$ via a straight-through Gumbel-softmax attention mechanism to make a differentiable approximately one-hot selection. Each policy has an associated signature key which is initialized randomly at the start of the training: $K^{\Pi}_t=\{K^{\pi^{u}}_{t} | 1 \leq u \leq U\}$. These keys are matched against the queries computed as deterministic function of encoded partial observation and the information made available to each agent.  
$q^{policy}_{i, t} = g_{psel}(s_{i, t},  m_{i, t})$  where $ g_{psel} $ is parameterized as a neural network.

\[ \mathrm{index}_i= \mathrm{GumbelSoftmax}\left(\frac{q^{policy}_{i, t} (K^{\Pi}_t)^T}{\sqrt{d_m}}\right)\]

As a result of this attention procedure, agent $i$ selects a policy $\pi^{index_{i}}$. This operation is performed independently for each agent, i.e. each agent selects a policy from the policy pool.


\textbf{Step 5: Maximizing agent independence.}  We minimize the dependence of agent on the information made available to each agent. We do this by optimizing the conditional mutual information by upper bounding the KL and penalizing $I\!\left(Z; M\mid S\right)$ \footnote{$I\!\left(Z; M\mid S\right) \geq I\!\left(A;M\mid S\right)$ \citep{galashov2019information, goyal2019infobot} such that $\pi_{\theta}\!\left(A \mid S, M\right) = \sum_z p_\text{enc}\!\left(z\mid S, M\right) p_\text{dec}\!\left(A\mid S,z\right)$}.

Thus, we can instead maximize this lower bound on $J\!\left(\theta\right)$:
\begin{equation}
\begin{aligned}
J\!\left(\theta\right) &\geq \mathbb{E}_{\pi_\theta}\!\left[r\right]-\beta I\!\left(Z; M\mid S\right) 
\label{eq:information_bottleneck}
\end{aligned}
\end{equation}

where $\mathbb{E}_{\pi_\theta}$ denotes an expectation over trajectories across different agents, $\beta > 0$ is a trade-off parameter and $r=\sum_{t=0}^T r_t$ is the total return up to the horizon $T$.



\section{Related Work}

\textbf{Centralized training decentralized execution (CTDE).} These approaches are among the most commonly adopted variations for MARL in cooperative tasks. They usually involve a centralized critic which takes in global information, i.e. information from multiple agents and decentralized policies which are guided by the critic. \citet{Foerster2018CounterfactualMP} uses the standard centralized critic decentralized actors framework with a \textit{counterfactual baseline}. \citet{lowe2017multi} and \citet{Yu2021TheSE} propose the extension of single-agent DDPG and single-agent PPO respectively, to a multi-agent framework with a centralized critic and decentralized policies during training and completely decentralized execution. \citet{li2021celebrating} uses an information theory based objective to promote novel behaviours in CTDE based approaches. Value Decomposition (\cite{Sunehag2018ValueDecompositionNF}, \citet{Rashid2018QMIXMV}, \citet{wang2020qplex}, \citet{Mahajan2019MAVENMV}) approaches learn a factorized action-value function. \citet{Sunehag2018ValueDecompositionNF} proposes Value Decomposition Networks (VDN) which simply add the action-value function of each agent to get the final action value function. \citet{Rashid2018QMIXMV} uses a mixing network to combine the action-value functions of each agent in a non-linear fashion.

\textbf{Communication in MARL.} Several approaches use emergent communication protocols for MARL. Communication involves deciding the message to be shared and determining how the message sending process is implemented. \citet{foerster2016learning} and \citet{Sukhbaatar2015} have done work on learnable inter-agent communication protocols. \citet{Jiang2018LearningAC} first proposed using attention for communication where attention is used for integrating the received information as well as determining when communication is needed.  \citet{das2019tarmac} uses multiple rounds of direct pairwise inter-agent communication in addition to the centralized critic where the messages sent by each agent are formed by encoding it's partial observation, location information, etc., and the messages received by each agent are integrated into it's current state by using a soft-attention mechanism.  \citet{kim2020communication} uses intentions represented as encoded imagined trajectories as messages where the encoding is done via a soft-attention mechanism with the messages received by the agent. \citet{wang2021tom2c} trains a model for each agent to infer the intentions of other agents in a supervised manner, where the communicated message denotes the intentions of each agent. The above mentioned approaches require a computational complexity which is quadratic in the number agents. Our approach has a computational complexity which is linear in the number of agents. Moreover, we show that our approach is able to outperform several standard baselines using messages which can be computed as simply the encoded partial observations of each agent. 

\textbf{Independent learning.} Independent Learning in MARL consists of each agent optimizing its policy locally using its observation and in the absence of any communication or centralized controller (as in CTDE). These approaches mainly consist of the extension of single-agent RL approaches to multi-agent settings where each agent learns independently using local observations, considering other agents as part of the environment. \citet{Tan1993MultiAgentRL} proposed Independent Q-Learning (IQL) where each agent independently learns it's own action-value function. \citet{Witt2020IsIL} demonstrates that PPO, when used for independent learning in multi-agent settings (called Independent PPO or IPPO) is in fact capable of beating several state of the art approaches in MARL on competitive benchmarks such as StarCraft and can hence serve as a strong baseline.


\textbf{Communication bottleneck.} With the emergence of modular deep learning architectures \citep{vaswani2017attention, goyal2021neural, scarselli2008graph, bronstein2017geometric, kipf2018neural, battaglia2018relational}, which require communication among different model components, there have been development of methods which introduce a bottleneck in this communication to a fixed bandwidth which helps communicating only the relevant information. \citet{DianboLiu2021DiscreteValuedNC} uses a VQ-VAE \citep{oord2017neural} to discretize the information being communicated. Inspired by the theories in cognitive neuroscience \citep{Baars1988-BAAACT, shanahan2006cognitive, Dehaene-et-al-2017}, \citet{goyal2021coordination} proposes the use of a generic \textit{shared workspace} which acts as a bottleneck for communication among different components of multi-component architectures and promotes the emergence of \textit{specialist} components. We use a \emph{{\textsc{saf}}} similar to the shared workspace in which different agents compete to write information to and read information from. 
 



\iffalse
\subsection{Shared Knowledge Source}
We call the list of vectors used to coordinate messages a shared workspace. Attention mechanism is used for shared workspace mechanisms as in our previous paper. The query is a function of the state of the memory matrix $M_t$ at time step t (with one row per slot of the memory), $Q_t=M_t W^q$. In this study, each row in $M_{j,t}=E_j(mean(O_t))$ where $E_j$ is an MLP based encoder model for GWS slot j. Different E models do not share parameters.
$O_t$ is the message matrix at time step t with each row representing message sent from an agent. $mean(O_t)$ is the mean vector of messages from all agents. The each message type varies among different MARL algorithms (see results section for details). The key for GWS writing attention is $K_t=O_t W^k$. Updated shared workspace slots are $M'_{t}=softmax(\frac{Q_t K_t}{\sqrt{d}})O_t$ where d is the key dimension.

After information writing into shared workspace, the information is broadcasted from the shared workspace using attention mechanism to all agents. $Q^{read}_t=O_t W'^q$ and $K^{read}_t=M'_t W'^k$. Messages sent to agents are $O'_{t}=softmax(\frac{Q^{read}_t K^{read}_t}{\sqrt{d_m}})M'_t$



\subsection{Dynamic Policy Selection}
Each agent has access to a pool of more than one policies and select only one of the policies to choose action by following the these steps: 1) Parsing the state of an agent $j$. 2) Selectively receive message from the shared workspace via attention mechanism and 3) Select policy by attention mechanism with Gumbel-softmax. 4) Apply the policy and take action 5) Update the policy


\subsection{Communication Cost}
$m'_{i,t}$ is the message received by agent $i$ from the shared workspace at time step $t$. $s_{i,t}$ is the internal state that come from partial observation of the agent. $\pi_{i,t}$ is the policy picked by agent $i$ at time step $t$. 

\begin{align}
R_i &= R^{env}_{i} + R^{independence}_i \\
R_i &= \mathbb{E}\left[\sum_{t} \gamma^t r(s_{i,t},a_{i,t}) -\beta \gamma^t KL[ \pi_{i,t}(a_{i,t}|s_{i,t},m'_{i,t})||\pi_{i,t}(a_{i,t}|s_{i,t})]\right]
\end{align}

the KL term penalize divergence of $\pi_{i,t}(a_t|s_{i,t})$ from $\pi_{i,t}(a_t|s_{i,t},m'_{i,t})$. Therefore, minimize influence of the message on the action prediction and enhancer independence of each agent.

\fi





\section{Experiments}
We investigate how well \emph{{\textsc{saf}}}, policy switching, and maximizing independence work together to improve cooperative MARL. Next, we compare our method with a state-of-the-art cognitive science-inspired approach used in multi-agent communication and cooperation\citep{wang2021tom2c}. Lastly, to understand if our method can be adapted to MARL methods that send different messages among agents, we integrate our machinery into a MARL algorithm that iteratively sends hypothetical actions among cooperative agents.

\subsection{Environments}
In this section, we describe the various MARL environments which we considered for our experiments..


\vspace{-3mm}
\paragraph{GhostRun environment.} We use the \textbf{GhostRun Environment} which is an adaptation of the Drone environment available from \citet{shuo2019maenvs}. The environment consists of multiple drones, each with a partial view of the ground below them. The ground consists of ghosts - represented by red dots, trees - depicted by green dots, and obstacles - depicted by black dots. The ghosts move about randomly whereas the trees and obstacles are stationary (see Figure \ref{fig:Env} (a)). The task at hand is a cooperative task where the agents must work together to escape from ghosts. Hence, the number of ghosts in each agent's partial observation of the environment must be minimized. The reward received by each agent at each time step is the negative of the total number of ghosts in the view of all the agents and a step cost of -1 for each step taken.

\begin{figure*}[htp]
    \centering
    \subfigure[GhostRun Environment]{
    \includegraphics[width=0.2\linewidth]{figures/Drone.png}}\hspace{15mm}
    \subfigure[PistonBall Environment]{
    \includegraphics[width=0.1\linewidth]{figures/PistonBall.png}}\hspace{15mm}
    \subfigure[MSTC Environment]{
    \includegraphics[width=0.2\linewidth]{figures/pose_environment.png}}
    
    \caption{GhostRun, PistonBall and MSTC environments. In the GhostRun environment (left panel) Each agent has its own partial view of the environment and the reward is to escape from ghosts (red dots). Different agents can communicate their encoded views with each other. In the PistonBall environment (middle panel) all agents (pistons) work together to move the ball from one side to the other. In the MSTC environment (right panel), sensors (gray dots) need to cover as many targets as possible (red dots). MSTC figure adapted from \citet{wang2021tom2c}}
    \label{fig:Env}
\end{figure*}

\vspace{-3mm}
\paragraph{Multi-Sensor Target Coverage.} We use the multi-sensor multi-target tracking (MSTC) environment from \citet{wang2021tom2c}. There are sensors (which are the agents) and targets. The goal is for the sensors to observe as many targets as possible at once (see Figure \ref{fig:Env} (c)). Each sensor has a partial observation of its surroundings and its view may be obstructed by obstacles. The targets can move according to one of two rules: according to a random walk, or along the shortest path to reach a previously sampled destination. At the beginning of each episode, the location of targets, sensors and obstacles is randomly sampled. The maximum episode length is $100$ steps and the reward is defined as $r=\frac{1}{m}\sum_q T_q$ where $T_q=1$ if the target $T_q$ is observed by any sensor and $0$ otherwise. If no target is observed (i.e. $T_q=0\quad\forall q$) then $r=-0.1$.

\vspace{-3mm}
\paragraph{PistonBall Environment.} This is a simple physics-based cooperative game where the goal is to move the ball to the left wall of the game border by activating the vertically moving pistons. Each agent’s observation is an RGB image of the two pistons (or the wall) next to the agent and the space above them (see Figure \ref{fig:Env} (b)). Every piston can be acted on at any time. The action space can be discrete: 0 to move down, 1 to stay still, and 2 to move up. Alternatively, the action space can be continuous: the action value in the range $[-1, 1]$ is proportional to the amount by which the pistons are raised or lowered. Continuous actions are scaled by a factor of $4$, so that in both the discrete and continuous action space, the action $1$ will move a piston $4$ pixels up, and $-1$ will move pistons 4 pixels down.





\subsection{Baselines: Varying Along Different Axes}


In our experiments, we consider multiple baselines which use some aspects of the proposed method, such as the use of stateful and active facilitator \emph{{\textsc{saf}}}, shared pool of policies and maximizing agent independence. In particular, we try to disentangle the contributions of these different components of the proposed architecture.
The following baselines are evaluated:

\textit{Multiple Agents with no communication} [\textsc{I}]: There are multiple independent agents with no communication between them.

\textit{Multiple Agents with pairwise communication} [\textsc{P}]: 
Every pair of agents can communicate with each other via self-attention \citep{vaswani2017attention}.

\textit{Multiple Agents with pairwise communication and shared pool of policies} [\textsc{P + SP}]: Every pair of agents can communicate with each other via self-attention and each agent can select a different policy from the shared pool of policies. The difference from the proposed method would be that this variant does not make use of \emph{{\textsc{saf}}}. 

\textit{Multiple Agents with \textsc{\emph{{\textsc{saf}}}}} [\textsc{\emph{{\textsc{saf}}}}]: Here, we consider multiple agents such that different agents communicate with each other via \emph{{\textsc{saf}}}, and all agents share the same policy.

\textit{Multiple Agents with \textsc{\emph{{\textsc{saf}}}} and policy pool} [\textsc{\emph{{\textsc{saf}}} + SP}]: the  agents can communicate with each other via \emph{{\textsc{saf}}}, and also share the pool of policies such that each agent can dynamically select a policy from the policy pool. 

\textit{Multiple Agents with \textsc{\emph{{\textsc{saf}}}}, policy pool and maximizing agent dependence.} [\textsc{\emph{{\textsc{saf}}} + SP + KL}]: The agents can communicate with each other via \emph{{\textsc{saf}}}, share the pool of policies, and also agents minimize the dependence of the policy on the information made available by \emph{{\textsc{saf}}}.




\vspace{-4mm}
\paragraph{Hyperparameters.}  We consider two hyper-parameters in the proposed model: (1) number of policies in the pool ($U=N_{policies}$), (2) number of slots in the \emph{{\textsc{saf}}} ($l=N_{slots}$).  For all our experiments, we use $N_{policies}=3$.




\begin{figure*}[ht!]
    \centering
    \subfigure[2 agents]{
    \includegraphics[width=0.3\linewidth]{figures/Scale_agents2_test.pdf}
    }
    \subfigure[5 agents]{
    \includegraphics[width=0.3\linewidth]{figures/Scale_agents5_test.pdf}
    }\\
    \subfigure[15 agents]{
    \includegraphics[width=0.3\linewidth]{figures/Scale_agents15_test.pdf}
    }
    \subfigure[30 agents]{
    \includegraphics[width=0.3\linewidth]{figures/Scale_agents30_test.pdf}
    }
    
    
   
   
   
    \caption{\textsc{{\textsc{saf}}}, shared policy pool and maximizing agent's independence in cooperative MARL bring significant improvement in performance when there are more cooperative agents in the same environment in the training and evaluation process. (a) 2 cooperative agents  (b) 5 cooperative agents (c) 15 cooperative agents (d) 30 cooperative agents. Our method (\emph{{\textsc{saf}}} + SP) performs better when there is a  larger number of agents in the environment .All experiments in this figure were conducted in GhostRun environment}
    \label{fig:Scale}
\end{figure*}



\vspace{-3mm}
\paragraph{Stateful and Active communication Facilitator (\emph{{\textsc{saf}}}) improves communication in MARL.}
We investigate the use of \emph{{\textsc{saf}}} for coordinating among different agents in the Ghost Run environment by comparing different communication schemes. (a). We compare three communication strategies [\textsc{I}], [\textsc{P}] and using \emph{{\textsc{saf}}}. We find that the proposed \emph{{\textsc{saf}}} based approach outperforms the other two settings and shows an improvement of about \textbf{45\%} over the other two baselines as shown in figure \ref{fig:GhostRun}(a). This shows that introducing the \emph{{\textsc{saf}}} helps improve inter-agent communication. The \emph{{\textsc{saf}}} acts as a specialist agent / communication bottleneck for the messages sent by all agents. It filters out only the relevant information for providing the relevant \textit{context} to each agent making it more robust, which may not be possible in pairwise communication i.e., where every pair of agents communicate with each other. 

We also compare [\emph{{\textsc{saf}}}+SP] against a graph-based communication baseline \citep{wang2021tom2c} that we simply name \emph{\textsc{Graph}}, on the MSTC environment. In \emph{\textsc{Graph}}, the agents generate a message from their observation and communication is done through a fully-connected graph whose nodes are the updated messages. These are used top optimize the agents' policies. We show the test reward  in Table \ref{tom-saf}.

\begin{figure*}[ht!]
    \centering
    \subfigure[Communication Methods]{
    \includegraphics[width=0.2\linewidth]{figures/CommunicationMethod_test.pdf}
    }
    \subfigure[Policy Selection]{
    \includegraphics[width=0.2\linewidth]{figures/PolicySelection_test.pdf}
    }
    \subfigure[Varying beta]{
    \includegraphics[width=0.2\linewidth]{figures/BetaKL_test.pdf}
    }
    \subfigure[Maximizing Independence]{
    \includegraphics[width=0.2\linewidth]{figures/KLTarget_OOD.pdf}
    }
    \caption{\emph{{\textsc{saf}}}, shared policy pool and maximizing agent's independence in cooperative MARL. (a) Comparison between different communication setups (b) Comparison between different policy selection strategies (c) Comparison between coefficients for KL Loss. (d) Comparison between penalizing policy selection and penalizing action selection}
    \label{fig:GhostRun}
\end{figure*}







\begin{figure*}[t]
    \centering
    \includegraphics[width=0.26\linewidth]{figures/PistonBall_reward.pdf}
    \includegraphics[width=0.26\linewidth]{figures/PistonBall_length.pdf}
    \caption{\emph{{\textsc{saf}}}, shared policy pool and maximising agent's independence in cooperative MARL with agents communicating hypothetical actions as messages in PistonBall environment. Our method has higher cumulative rewards while decreasing the average episode's length compared to the baselines.}
    \label{fig:HypotheticalActions}
\end{figure*}


\begin{table}[]
\caption{Average reward on the test setting in the MSTC environment for the [\emph{{\textsc{saf}}}+SP] architecture and the \emph{\textsc{Graph}} model}
\label{tom-saf}
\vskip 0in
\begin{center}
\begin{small}
\begin{sc}
\begin{tabular}{lll}
\toprule
 & [\emph{{\textsc{saf}}}+SP] & \emph{\textsc{Graph}} \\
\midrule
Expected Return   & $71.26 (\pm 5.81)$ &   $61.22 (\pm 8.15) $    \\
\bottomrule
\end{tabular}
\end{sc}
\end{small}
\end{center}
\vskip -0.1in
\end{table}

\vspace{-3mm}
\paragraph{Policy switching using shared policy pool improves performance.} We investigate the significance of using policy switching using a shared pool of policies for each agent. We experiment with two settings: \textsc{\emph{{\textsc{saf}}}} and \textsc{\emph{{\textsc{saf}}} + SP}.  The inter-agent communication in both settings is facilitated via the use of \emph{{\textsc{saf}}}. The results are shown in figure \ref{fig:GhostRun}(b). Using a shared pool of policies with dynamic selection shows a relative improvement of \textbf{25\%} over the fixed policy setting.  This shows that the mixture of policies helps to transfer knowledge about different behaviors across different agents. 





\vspace{-3mm}
\paragraph{Minimizing dependence on \emph{{\textsc{saf}}}.} Here, we investigate the effect of minimizing the dependence of each agent's policy on the information that is available from the \emph{{\textsc{saf}}} in the form of its state, quantified by the conditional mutual information $I\!\left(Z;M \mid S\right)$, where $Z$ is the agent's policy choice. One can directly optimize it  as discussed in eq.\ref{eq:information_bottleneck}. We also experimented with different values of the coefficients ($\beta$) of the KL based regularization loss. Figure \ref{fig:GhostRun}(c) shows that minimizing the dependence of each agent on the information from \emph{{\textsc{saf}}} helps improve convergence. 

We also compare the effect of directly optimizing the conditional mutual information $I\!\left(A;M \mid S\right)$ i.e., where the information from \emph{{\textsc{saf}}} is used to also select actions (as compared when information from \emph{{\textsc{saf}}} is only used to select the policy) and the agent is optimized to minimize dependence on information from \emph{{\textsc{saf}}}. We call the method where we optimize the mutual information directly on actions as \textit{regularized action selection}, and the method where we optimize the upper bound on mutual information as \textit{regularized policy selection}. Figure \ref{fig:GhostRun}(d) compares the effect of regularizing action as compared to regularizing the policy selection, and shows that regularizing the policy selection (i.e., upper bound) achieves better results as compared to regularizing the action selection. 



\paragraph{\textsc{SAF} is significantly helpful for MARL training with large number of agents.} Here we investigate how the proposed method scales with increasing the number of agents. Learning a stateful, active facilitator \emph{{\textsc{saf}}} which integrates information across agents should scale better as compared to using a communication protocol where every agent interacts with every other agent. To test this hypothesis, we compare the performance of five different approaches: \emph{{\textsc{saf}}}, [\emph{{\textsc{saf}}}+\textsc{SP}], [\textsc{P}], [\textsc{P} + \textsc{SP}], [\textsc{I}] by varying the number of agents. Figure \ref{fig:Scale} shows the result of these different methods by varying the number of agents. As the number of agents increases, the relative difference between the proposed method and the different baselines ([\textsc{I}], [\textsc{P}], [\textsc{P}+\textsc{SP}]) increase significantly. 

\paragraph{Flexible plug-in tool for many different MARL settings.} In the previous sections, we showed that \emph{{\textsc{saf}}}, shared policy, and maximizing independence boost performance on cooperative MARL tasks in which agents communicate messages generated as a function of the observed environment. In this section, we investigate the possibility of applying the proposed method on cooperative MARL algorithms in which agents iteratively communicate \textit{hypothetical actions} with each other before taking actions. The performance of the proposed method is compared with two recent approaches that communicate hypothetical actions, namely consensus update and iterative reasoning (IR) \cite{zhang2018fully}. Consensus update conducts graph-based local communication and IR communicate hypothetical actions among agents such that an agent's policy is conditional on its teammates. Our experimental results show that our method achieves higher cumulative rewards in the PistonBall environment while doing so in a lower number of iterations, both of which indicate better performance (see Figure \ref{fig:HypotheticalActions}). This suggests the potential of the proposed method as a flexible plug-in tool in many MARL settings. 



\section{Future Work}
Here, we  introduced three key ideas that operate synergistically. First, communication among agents is conducted via an intelligent channel, {\textsc{saf}}. Second, each agent is incentivized to act independently and avoid relying on {\textsc{saf}}. Third, agents
operate according to a mixture-of-policies, where {\textsc{saf}}\ provides the signal to select the policy. Through extensive experiments we show the utility of all the different components. We also show that the proposed method scales better on increasing the number of the agents and achieves higher returns as compared to various different baselines. Future work may involve scaling the proposed method to more complex multi-agent problems like Starcraft \citep{vinyals2019grandmaster} and Google Research Football \citep{gfootball}.





























\section{Story.}




\section{Introduction}
\label{submission}

As reinforcement learning (RL) methods begin to tackle applications in the real world and other complex, dynamic environments,
the participation of more than one agent is often required, e.g., robotics and video game applications.
Multi-agent RL (MARL) addresses the sequential decision-making problem of two or more autonomous 
agents that operate in a common environment, each of which aims to optimize its own long term return 
by interacting with the environment and with other agents \citep{busoniu2008comprehensive}. Largely, 
MARL algorithms can be placed into three categories depending on the type of setting they address: fully cooperative, 
fully competitive, and a mix of the two. In the cooperative setting, agents collaborate to optimize a shared long-term return
whereas in the competitive setting, agent returns are zero sum. We focus on cooperative MARL with a partially
observed environment.  


Early research in cooperative MARL focused on agents that operated independently and that did not explicitly communicate \cite{???}.
However, when each agent has only a partial view of the environment, it becomes beneficial for agents to communicate with one another to 
augment their belief states. Even in fully observed environments, inter-agent communication can be beneficial to coordinate behavior.
Not surprisingly, enabling inter-agent communication yields performance advantages \cite{???}. Recent research on communication for deep MARL
adopts an end-to-end training procedure based on a differentiable communication channel \cite{???}. Essentially, each agent possesses a
subnetwork that generates messages and these messages modulate other agents' policy or critic networks via the communication channel.
The message-generation subnet is trained using the gradient of other agents' policy or critic losses.



In designing a multi-agent communication architecture, the key decisions concern properties of the communication channel: Are there bandwidth limitations on the message transmission or the processing of received messages? Can messages be modulated by the channel.  The simplest scenario involves agent-to-agent direct
communication, where each of $N$ agents can receive messages from the $N-1$ other agents. Because message complexity grows with $N$, 


OH NO. I DIDN"T KNOW THIS.  
are the first couple of paragraphs the same?
75\% OF IT.o

ok, i'll integrate.  thanks for catching.  Can I move this file to an OLD directory to prevent this from happening again?

manner in which messages are communicated and the capacity of the messaging
pathway.
Most recent studies assume direct agent-to-agent communication. In \citet{das2019tarmac}, messages are broadcast from each agent to all other agents,
and each agent uses soft attention to integrate received messages.
broadcast messages from each agent to all other agents, and each
agent uses soft attention to integrate messages
and use soft-attention to integrate messages at each receiver.
\citep{kim2020communication} make each agent send messages to all other agents. \citep{wang2021tom2c} train a graph neural network model for each agent to decide which other agents it is sending message to. In iterated reasoning method, each agent is sending messages only to its physical neighbours.
(1) when an agent should send messages, (2) the manner in which messages are transmitted, and (3) how an agent uses received messages.


What message to share across different agents have been on of the major focus in MARL communication of recent studies. \citep{das2019tarmac} use encoded information from each agent's partial observation and previously received messages as the message. \citep{kim2020communication} shared imagined trajectories of agents. \citep{wang2021tom2c} train a model in each to infer intention of other agents and use the inferred intentions as messages. In iterated reasoning method, the each agent send its own hypothetical action to other agents and repeat the process a couple of steps before taking actions in each time step.

Another important factor to consider is how messages sending process are coordinated among different agents. Most recent studies use an agent-to-agent strategy. \citep{das2019tarmac} broadcasted message from each agent to all other agents and use soft-attention to integrate messages at each receiver.
\citep{kim2020communication} make each agent send messages to all other agents. \citep{wang2021tom2c} train a graph neural network model for each agent to decide which other agents it is sending message to. In iterated reasoning method, each agent is sending messages only to its physical neighbours.

To the best of our knowledge, in most algorithms, when an agent receives messages from other agents, it either directly combines the messages with internal states and use it as input for policy or combine with internal state after some aggregation process a such soft attention \cite{das2019tarmac,kim2020communication}.
	
	


In this study, we propose a inter-agent communication scheme for MARL that includes both a new approach of how to communicate messages among agents and a new method to utilize received messages. Our method include a coordinated message broadcasting router, a state and message based policy selection system and an additional ELBO based reward that enhance independence of each agent. In this study, we do not specify how messages are generated from each agent. Therefore, our method can be plugged into any communication schema in MARL.


whereas communication channels are passive and stateless, the facilitator is active (information seeking) and stateful
the facilitator works to coordinate communication
not a central controller

	







\section{Related works}

\begin{itemize}
    \item \textbf{Centralized Training Decentralized Execution}: CTDE is one of the most commonly adopted paradigms for MARL. \citep{lowe2017multi} extends DDPG to a multi-agent framework with a centralized critic and decentralized policies during training and decentralized execution. \citep{li2021celebrating} uses an information theory based objective to promote novel behaviours in CTDE based approaches
    \end{itemize}








\begin{algorithm}
    \caption{MARL with dynamic policy selection, shared information source and cost of communication}
   \label{alg:method}
   \begin{algorithmic
   \footnotesize

   \item[]
   
   \STATE{\bfseries \itshape } The whole MARL game is defined as $(\mathcal{N},\mathcal{S},\mathcal{A},P,R,\gamma)$. $\mathcal{N} = \{1,...,N\}$ denotes the set of $N>1$ agents. $S_t=\{s_{1,t},s_{2,t},...,s_{N,t}\}$ denotes the state of each agent at time step $t$. $A_{i}^t$ denotes the action of agent $i$ at time step $t$ . $P : \mathcal{S} \times \mathcal{A} \times \mathcal{S} \rightarrow [0,1]$ denotes the state transition probability. $R^i :\mathcal{S}\times \mathcal{A} \times \mathcal{S} \rightarrow \mathbb{R}$ is the reward function that determines the immediate reward received by agent $i$ for a transition from $(s;a)$ to $s'$ and $\gamma$ is the discount factor. In addition, there are $U$ different policies $\Pi={\pi_1,\pi_2,...,\pi_U}$ available for each agent. $K^{\pi}_{u}$ is the randomly initialized key for policy $\pi_u$. There is a shared information source $I_{t}\in \mathbb{R}^{J \times d_I}$ where $J$ is the number of slots in the share information source at time step $t$ and $d_I$ is the dimension of the slot.  Each of the $J$ slots is associated with a writing query $q^I_{j,t}$ at time step t.
   
    \BlankLine 
    
    
    \For{ time step $t$ in $1$ to $T$}  {
    
    \textbf{Step 1}: Each agent $i$ has its state $s_{i,t}$ (encoded partial observation) and generate messages $m_{i,t}$     \BlankLine 
    
    The mechanism generating the message vary among different algorithms \BlankLine 

    
    $M_t=(m_{0,t},m_{1,t},m_{2,t}...,m_{N,t})$ is the collection of all messages from all agents at time step t.  \BlankLine 
    

        
    \textbf{Step 2}:All agents write their messages in to the share information source via soft attention \BlankLine 
    
    
     the shared information source
    $I_{t}=\mbox{softmax}\left(\frac{Q^I_{t} K^M_t}{\sqrt{d_m}}\right)M_t$ \BlankLine
    where $Q^I_{t}=(q^I_{0,t},q^I_{1,t},q^I_{2,t},...,q^I_{J,t})$ ,$O_{t}$ is the generated shared information that contain messages from all agents. $K^M_t=(k^M_{0,t},k^M_{1,t},k^M_{2,t},....,k^M_{N,t})$ is the key matrix for messages. Each $K^M_t=W^M M_t$\BlankLine 
    
    \For{ agent $i$ in $1$ to $N$}  {
    
    \textbf{Step 3}: Next,each agent read from the $I_t$:
    
    $m'_{i,t}=\mbox{softmax}\left(\frac{q^s_{i,t}I_t }{\sqrt{d_m}}\right)I_t$ \BlankLine
    the resulted vector $m'_{i,t}$ is the message agent $i$ received from the shared information source and $q^s_{i,t}=W^{s}s_{i,t}$ is the query from each agent based on its state\BlankLine
    
    
    \textbf{Step 4}: Each agent $i$ select policy from the pool using attention and gumbel-softmax:
    
    $q^{policy}_{i,t}=q^s_{i,t}+m'_{i,t}$ is the query generated using both internal state of the agent and message received from other agent 
    
    $q^{policy-S}_{i,t}=q^s_{i,t}$ is the query generated using only internal state of the agent
    
    $\mbox{AttScore}_{i,t}=GumbelSoftmax(q^{policy}_{i,t},K^{pi}_t)$ 
    
    $AttScore^s_{i,t}=GumbelSoftmax(q^{policy-s}_{i,t},K^{pi}_t)$ 
    
    
    Each agent $i$ picks one policy $\pi _ {i,t}$ based on the one hot vector $AttScore_{i,t}$ 
    
    Each agent $i$ takes action $a_{i,t}\get \pi_{i,t}(s_{i,t},m'_{i,t})$ \BlankLine
    
    
    
    Environment reward $r_{i,t}=r(a_{i,t},s_{i,t})$ \\

    \textbf{Step 5}: KL loss calculated using policy picking\BlankLine
    $L^{KL}_{i,t}=-KL[AttScore_{i,t}||AttScore^s_{i,t}]$
    }
    
   }
\end{algorithmic}
\end{algorithm}



\section{Method}

\subsection{Shared behavior priors}

Each agent has access to a pool of more than one policies and select only one of the policies to choose action by following the these steps: 1) Parsing the state of an agent $j$. 2) Selectively receive message from the shared workspace via attention mechanism and 3) Select policy by attention mechanism with Gumbel-softmax. 4) Apply the policy and take action 5) Update the policy

\subsection{Shared knowledge source}
We call the list of vectors used to coordinate messages a shared workspace. Attention mechanism is used for shared workspace mechanisms as in our previous paper. The query is a function of the state of the memory matrix $M_t$ at time step t (with one row per slot of the memory), $Q_t=M_t W^q$. In this study, each row in $M_{j,t}=E_j(mean(O_t))$ where $E_j$ is an MLP based encoder model for GWS slot j. Different E models do not share parameters.
$O_t$ is the message matrix at time step t with each row representing message sent from an agent. $mean(O_t)$ is the mean vector of messages from all agents. The each message type varies among different MARL algorithms (see results section for details). The key for GWS writing attention is $K_t=O_t W^k$. Updated shared workspace slots are $M'_{t}=softmax(\frac{Q_t K_t}{\sqrt{d}})O_t$ where d is the key dimension.

After information writing into shared workspace, the information is broadcasted from the shared workspace using attention mechanism to all agents. $Q^{read}_t=O_t W'^q$ and $K^{read}_t=M'_t W'^k$. Messages sent to agents are $O'_{t}=softmax(\frac{Q^{read}_t K^{read}_t}{\sqrt{d}})M'_t$




\subsection{Reward for independence}
$m'_{i,t}$ is the message received by agent $i$ from the shared workspace at time step $t$. $s_{i,t}$ is the internal state that come from partial observation of the agent. $\pi_{i,t}$ is the policy picked by agent $i$ at time step $t$. 

\begin{align}
R_i &= R^{env}_{i} + R^{independence}_i \\
R_i &= \mathbb{E}\left[\sum_{t} \gamma^t r(s_{i,t},a_{i,t}) -\beta \gamma^t KL[ \pi_{i,t}(a_{i,t}|s_{i,t},m'_{i,t})||\pi_{i,t}(a_{i,t}|s_{i,t})]\right]
\end{align}

the KL term penalize divergence of $\pi_{i,t}(a_t|s_{i,t})$ from $\pi_{i,t}(a_t|s_{i,t},m'_{i,t})$. Therefore, minimize influence of the message on the action prediction and enhancer independence of each agent.




\begin{figure*}
    \centering
    \includegraphics[width=0.8\linewidth]{figures/MARL.png}

    \caption{GWS-NPS in multi-agent reinforcement learning. Each agent only has a partial view of the environment. The agents need to cooperate with each other to achieve the same goal. The agents need to communicate messages with each other at each time step to coordinate. In our study, we use a shared workspace to mediate communication and a dynamic policy selection at each time step to accomodate non-stationary environment }
    \label{fig:MARL}
\end{figure*}


\section{Experiment}




\subsection{shared behavior priors helps in MARL}


\subsection{shared knowledge source helps in MARL}


\subsection{Including cost of communication helps in MARL}


\subsection{shared knowledge source helps}

\subsection{Random sample of agents writing into shared knowledge source improve generalization}


\subsection{multi-agent minigrid environment vs baselines}
In this section, we compare performance of our method with several well known methods in MARL literature including IPPO, MADDPG, QMIX and MAPPO in an multi-agent treasure seatching game.

\subsection{Pose environment our method  vs baselines }
In this section, we compare performance of our method with several well known methods in MARL literature including IPPO, MADDPG, QMIX and MAPPO in an multi-agent cooperative action game.

\subsection{Drone environment our method  vs baselines }
In this section, we compare performance of our method with several well known methods in MARL literature including IPPO, MADDPG, QMIX and MAPPO in an multi-agent cooperative escapling game.


\subsection{Particle environment our method vs baselines}
In this section, we compare performance of our method with several well known methods in MARL literature including IPPO, MADDPG, QMIX and MAPPO in an multi-agent cooperative target reaching game.











\begin{figure*}
    \centering
    \includegraphics[width=0.2\linewidth]{figures/Drone.png}

    \caption{Drone environment. Each drone has its own partial view of the environment and the reward is to escape from human (red dots). Different agents can communicate encoded view with each other  }
    \label{fig:Drone}
\end{figure*}



\begin{figure*}
    \centering
    \includegraphics[width=0.6\linewidth]{figures/Drone_line.png}
    \includegraphics[width=0.4\linewidth]{figures/Drone_box_indis.png}
    \includegraphics[width=0.4\linewidth]{figures/Drone_box_OOD.png}
    \caption{Performance of different models in Drone environment.Upper panel: Reward curve on OOD test environment of different methods. Lower left:Reward in in-distribution test environment. In general method in which agents communicating via GWS and pick policy by attention from a shared pool of polcies performed best. Lower right:Reward in ODD test environment where the  number of objects is different from training environment. In general method in which agents communicating via GWS and pick policy by attention from a shared pool of polcies performed best }
    \label{fig:DronePerformance}
\end{figure*}







\begin{figure*}
    \centering
    \includegraphics[width=0.6\linewidth]{figures/TOM.png}

    \caption{Machine Theory of Mind model.Each agent infer intentions of other agents and send messages to each other in order to achieve the co-operative goal such as picking up objects in a room }
    \label{fig:TOM}
\end{figure*}



\begin{figure*}
    \centering
    \includegraphics[width=0.9\linewidth]{figures/TOM_comm.png}

    \caption{Performance of different methods in the ToM frame work using "comm" strategy. The method in which agents send messages to each other via GWS and pick policy using attention (comm-attention-GWS) achieved highest reward in initial stage of the training }
    \label{fig:TOMPerformance}
\end{figure*}






\begin{figure*}
    \centering
    \includegraphics[width=0.6\linewidth]{figures/InterationReasoning.png}

    \caption{MARL under iterative reasoning framework, in which agents communicate hypothetical actions with each other }
    \label{fig:Piston}
\end{figure*}



\begin{figure*}
    \centering
    \includegraphics[width=0.3\linewidth]{figures/PistonBall.png}
    \includegraphics[width=0.7\linewidth]{figures/PistonReward.png}
    \includegraphics[width=0.7\linewidth]{figures/Piston_iterationLength.png}

    \caption{Performance of different methods in the iterative reasoning framework. The method in which agents send messages to each other via GWS and pick policy using attention from a shared pool achieved highest reward in and finished the games in lowest number of steps }
    \label{fig:PistonBallPerformance}
\end{figure*}

\section*{Software and Data}

If a paper is accepted, we strongly encourage the publication of software and data with the
camera-ready version of the paper whenever appropriate. This can be
done by including a URL in the camera-ready copy. However, \textbf{do not}
include URLs that reveal your institution or identity in your
submission for review. Instead, provide an anonymous URL or upload
the material as ``Supplementary Material'' into the CMT reviewing
system. Note that reviewers are not required to look at this material
when writing their review.



\nocite{langley00}


\section{Introduction}
In recent years, deep learning has moved towards structural models with separate sub-components handling different pieces of information. 

In our previous shared workspace work inspired by global workspace theory in cognitive neurosciences, different functionally specialized model components share a common bandwidth-limited communication channel, which function as a communication bottleneck that put a restriction on which components can broadcast information to others. Different specialists compete for write access to the common workspace. The contents of the workspace, in turn,
are broadcast to all specialist modules simultaneously.The results suggested shared workspace encourage specialization and compositionality of different components, and synchronization among them. 






\section{Method}


\begin{algorithm}
    \caption{ NPS+GWT+DVNC}
   \label{alg:method}
   \begin{algorithmic
   \footnotesize
  
   \item[]
   
   \STATE{\bfseries \itshape Input:} Input sequence $\{ {\bm{x}}^1, \ldots, {\bm{x}}^t, \ldots,  {\bm{x}}^T\}$, set of embeddings describing the rules $\Vec{\vR_i}$, and set of MLPs $(MLP_{i})$ corresponding to each rule $\vR_{1 \ldots N}$. Hyper-parameters specific to \textsc{ModelName}\ are the number of stages $K$, the number of slots $M$, and the
   number of rules $N$. $W^q$, $\widetilde{W}^k$, and $\widetilde{W}^q$ are learnable weights.
  

     
    \BlankLine 
    \item[]
    \For{each input element ${\bm{x}}^t$ with $t \gets 1$ to $T$}  {
    
    \vspace{2mm}
    \STATE{{\bfseries \itshape Step 1:} \itshape Update or infer the entity state in each slot $j$, ${\bm{V}}_j^{t,0}$, from
    the previous state, ${\bm{V}}_j^{t-1,K}$ and the current input ${\bm{x}}_t$.}
  
   
    
    Competition among different slots to write in share workspace. $H_Q$ is the embedding of actions or state representation of time step t-1\\
    $Q=W_Q H_Q$ \\
    $K_i=W_k V$ \\
    ${\bm{M}}_t=topK(softmax(\frac{QK^T}{\sqrt{d}}))V$ \\
    ${\bm{M}}^{'}_t=MultiHeadQuantization({\bm{M}}_t)$
    \\
    
    
    
    
   \item[]
  
   \vspace{2mm}
   \For{each stage $h \gets 0$ to $K-1$}  {                  
        \item[]
   \STATE{{\bfseries \itshape Step 2:} \itshape Select \{rule, primary slot\} pair}
  
  \STATE \bull ${\bm{k}}_{i} = \Vec{\vR}_{i}  \quad \forall i \in \{1, \ldots, N\}$
  \STATE \bull ${\bm{q}}_j = {\bm{V}}_{j}^{t,h} {\bm{W}}^q \quad \forall j \in \{1, \ldots, M\}$
  \STATE \bull $r,p = \mathrm{argmax}_{i, j}\left({\bm{q}}_{j}{\bm{k}}_i + \gamma \right)$
  \STATE $~~~~~~~~~~~~~~~\text{ where } \gamma \sim \mathrm{Gumbel} (0,1)$
  
 
   \STATE{{\bfseries \itshape Step 3:} \itshape Apply selected rule to primary slot conditioned on global information vector M}
   
   \STATE \bull $R = \mathrm{MLP}_{r}(\mathrm{Concatenate}([{\bm{V}}_{p}^{t,h}, {\bm{M}}^{'}_t]))$
   
   \STATE \bull ${\bm{V}}_{p}^{t,h+1} = {\bm{V}}_{p}^{t,h} + R$
  
    }
   }

\end{algorithmic}
\end{algorithm}



\subsection{Shared workspace}
Attention mechanism is used for shared workspace mechanisms as in our previous paper.The query is a function of the state of the memory matrix M (with one row per
slot of the memory),$Q=MW^q$. Dot product attention is applied to update the memory matrix. Information is broadcast from the shared workspace using attention mechanism as well with $q_k=h^k_t W^q$, $k_j=(m_jW^e)^T$ and $s_{k,j}=softmax(\frac{q_k k_j}{\sqrt{d_e}})$. Hidden states from each memory slots are updated by $h^k_t=h^k_t+\sum_j s_{kj} v_j$

\subsection{communication discretization}
The process of converting data with continuous attributes into data with discrete attributes is called discretization \citep{chmielewski1996global}. In this study, we use discrete latent variables to quantize information communicated among different modules in a similar manner as in Vector Quantized Variational AutoEncoder (VQ-VAE) \citep{oord2017neural}. Similar to VQ-VAE, we introduce a discrete latent space vector $e \in \RR^{L \times m}$ where $L$ is the size of the discrete latent space (i.e., an $L$-way categorical variable), and $m$ is the dimension of each latent embedding vector $e_j$. Here, $L$ and $m$  are both hyperparameters. In addition, by dividing each target vector into $G$ segments or discretization heads, we separately quantize each head and concatenate the results (Figure~\ref{fig:Discretization}). More concretely, the discretization process for each vector $h  \in \Hcal\subset \RR^{m}$ is described as follows. First, we divide a vector $h$ into $G$ segments $s_1,s_2,\dots,s_G$ with $h=\textsc{concatenate}(s_1,s_2,\dots,s_G),$ where each segment $s_i \in \RR^{m/G}$ with $\frac{m}{G} \in \NN^+$. Second, we discretize each segment $s_i$ separately: 
$$
e_{o_i}=\textsc{discretize}(s_i), \quad \text{ where } o_i=\argmin_{j \in \{1,\dots,L\}} ||s_{i}-e_j||.
$$
Finally, we concatenate the discretized results to obtain the final discretized vector $Z$ as
\[Z=\textsc{concatenate}(\textsc{discretize}(s_1),\textsc{discretize}(s_2),...,\textsc{discretize}(s_G)).\]
The multiple steps described above can be summarized by $Z=q(h,L,G)$, 
\subsection{combination of shared workspace and DvNC}


\subsection{Combination of shared workspace and communication}
Shared workplace method and communication naturally complement each other. In this study, we introduce discretization to the broadcast message from shared workspace. i.e. $h^k_t=h^k_t+q(\sum_j s_{kj} v_j)$

\subsection{Neural production system}
In neural production system, there are the following several steps.1)Parsing input data into slots 2)select primary slot and rules by attention mechanism with Gumbel softmax. 3)Select contextual slot by attention with Gumbel softmax and 4) Apply the rule MLP and update the primary 


\subsection{Combination of shared workspace, communication discretization and NPS}

In addition to the the primary slot-rule-conditional slot sets, in the original NPS method , we introduce another set of "global information", which allows all the slots ( objects) write into a bottlenecked shared work space with competition. The information, discretization, from the workspace is then broadcasted to all objects. These "global information" can be thought as things like gravity or wind speed that applies to all objects or the whole environment instead of object pairs.






\begin{figure*}
    \centering
    \includegraphics[width=0.8\linewidth]{figures/Method.png}
    \caption{Method that combines shared workspace, discretization and NPS}
    \label{fig:jointmethod}
\end{figure*}


\section{Experiment}
To understand if combination of the bottlenecking effects from both shared workspace and discretized represetation will help inter-component communication in a model with multiple specialists on relevant information,we designed experiments in several reasoning tasks. We compared performance of transformer with 4-layers with shared workspace with top-k attention and discretized communication in the last 2 layers with transformer with only shared workspace and transformer with only discretized representation.

\subsection{CIFAR10 Image Scrambling Classification} In order to
evaluate the importance of coherence, we consider a task of classifying a CIFAR-10 image where the image has been
scrambled in a patch-wise manner according to a fixed permutation.

\subsection{text-image reasoning task}
In order to understand whether our method performance well in text-image reasoning task, we design a simple text-image environment in which the text describes locations of different objects in the environment.

\begin{figure*}
    \centering
    \includegraphics[width=0.6\linewidth]{figures/DLshapes.png}
    \caption{DLshapes}
    \label{fig:text-image locational reasoning}
\end{figure*}



\subsection{Sort-of-CLEVR}
Sort-of-CLEVR. In relational reasoning,
the model is tasked with answering questions about
certain properties of various objects and their relations with other objects. The model is presented with an image and a question for that image.This task has a clear sparse structure as in order to answer the questions correctly, it needs to only reason about a specific subset of objects that the question
mentions. For this task, we use the Sort-of-CLEVR dataset.Each image in Sort-of-CLEVR is a size 75  75 image containing 6 randomly placed geometrical shapes of 6 possible colors and 2 possible shapes. Each image comes with 10 relational questions and 10 non-relational questions. Nonrelational questions only consider properties of individual objects. For example, what is the shape of the red object?
is a non-relational question. On the other hand, relational
questions consider relations among multiple objects. For
example, what is the shape of the object closest to the red
object? is a relational question.


\begin{figure*}
    \centering
    \includegraphics[width=0.8\linewidth]{figures/SortOfClevr.png}
    \caption{Sort-of-clevr}
    \label{fig:Sort-of-clevr}
\end{figure*}



\subsection{Physical vision}
While machine learning algorithms excel at many challenging visual tasks, it is
unclear that they can make predictions about commonplace real world physical
events. \citep{bear2021physion} designed a physical reasoning environment where model are trained to conduct physical reasoning and predict result of physical intervention \url{https://github.com/cogtoolslab/physics-benchmarking-neurips2021}


\begin{figure*}
    \centering
    \includegraphics[width=0.3\linewidth]{figures/physicalvision.png}
    \caption{Physical vision}
    \label{fig:PhysicalVision}
\end{figure*}

\subsection{action based condition changes prediction}
We uses action-based condition changes same as in c-SWM paper for experiments.

\subsection{virtual tool games}
This game asks players to place one of several objects (“tools”) into a two-dimensional (2D) dynamic physical environment to achieve a goal: getting a
red object into a green region (Fig. 1D). This goal is the same for every level, but what is required to achieve it varies greatly. Once a single tool is placed, the physics of the world are enabled so that players see the effect of the action they took. they are limited to a single action on each attempt. the authors
designed 30 levels—20 for the original experiment. In this project , we train models to predict final state (end positions of all objects) of the games based on initial state , tool chosen and initial position of the tool. Due to existance of phyiscal rules, the physical reasoning is relatively difficult. 

\begin{figure*}
    \centering
    \includegraphics[width=0.8\linewidth]{figures/TrialErrors.png}
    \caption{virtual tool games}
    \label{fig:virtualtoolgames}
\end{figure*}



\subsection{Reinforcement learning task}
In order to test our method in reinforcement learning, we used  RL environment proposed \citep{janner2018reasoning} that construct castle of blocks of certain shapes using MuJoCo environment.


\begin{figure*}
    \centering
    \includegraphics[width=0.2\linewidth]{figures/RL.png}
    \caption{Reinforcement learning task for castle construction}
    \label{fig:RL}
\end{figure*}




\section{Story.}




\section{Introduction}
Multi-agent reinforcement learning (MARL) addresses the sequential decision-making of two or more autonomous 
agents that operate in a common environment, each of which aims to optimize its own long-term return 
by interacting with the environment and with other agents \citet{busoniu2008comprehensive}. 
MARL is becoming more common in many real-world applications such as robotics, as well as in other applications involving
complex, dynamic environments, such as video games.
Largely, 
MARL algorithms can be placed into two categories depending on the type of setting they address.
In the \emph{cooperative} setting, agents collaborate to optimize a shared long-term return
whereas in the \emph{competitive} setting, an advantage for one agent results in a loss for another.


Early research in cooperative MARL focused on agents that operated independently and that did not explicitly communicate \cite{IPPO, IAC, IQL}. However, when each agent has only a partial view of the environment, agents benefit from exchanging information with one another, allowing
them to construct a more complete belief state and improve decision making.
Even in fully observed environments, inter-agent communication can be beneficial to coordinate behavior.
Not surprisingly, performance advantages are obtained when agents have the ability to learn a communication protocol, whether implicitly 
or explicitly \cite{qmix,vdn,lowe2017multi,IPPO}. 
Recent research on communication for deep MARL adopts an end-to-end training procedure based on a differentiable communication channel 
\citet{sukhbaatar2016learning, foerster2016learning, gupta2017cooperative, singh2018learning, das2019tarmac}.  
Essentially, each agent has the capability of generating messages and these messages can influence other agents' policies.
In these works, the message-generation subnet is trained using the gradient of other agents' policy or critic losses.

In designing a MARL architecture, a key decision involves the nature of the communication channel.
The simplest scenario involves direct agent-to-agent communication, where each of the $N$ agents can receive messages from all other 
agents.  Communication costs are quadratic in $N$ and each agent faces the challenge of interpreting $N-1$ simultaneous messages.
Communication costs can be reduced by restricting the communication topology \citet{wang2017nonlocal}. 
Message processing can be simplified using a learned key-value attention mechanism that condenses messages at either the side
of the sender \citet{kim2020communication} or recipient \citet{das2019tarmac}.

\textbf{Intelligent communication channel.}
In previous approaches, the communication channel is \emph{passive}, by which we mean that its role is to convey whatever message passes through the channel without alteration.  We describe an approach in which the communication channel is 
\emph{active} in that it can interpret and transform signals from one or more agents, and it is \emph{stateful} in  that its interpretations can depend on the recent history of messages transmitted.  Because we endow the channel itself 
with intelligence, message communication complexity is reduced from quadratic to linear in $N$.
In essence, the channel is a specialist agent aiming to \emph{facilitate} coordination among the other agents.
We refer to the channel as \emph{{\textsc{saf}}}, an acronym on \emph{stateful, active facilitator}.
{\textsc{saf}}\ is itself adaptive and learns to improve the collective performance of the agents. 














\begin{figure}
  \centering
  \includegraphics[width=\linewidth]{figures/approach_figure.pdf}
  \caption{Agents communicate through the \emph{{\textsc{saf}}} and pick a policy from the shared policy pool. First, each agent generates a message and competes for write-access into the \emph{{\textsc{saf}}}. Next, all agents read messages from the \emph{{\textsc{saf}}} and use it with their internal state to pick a policy from the shared policy pool.}
    \label{fig:illust1}
\end{figure}

\textbf{Maximizing agent independence.}
By endowing the communication channel with intelligence, there is a risk that {\textsc{saf}}\ may simply become a centralized controller 
dictating actions to the agents, which is antagonistic to a multi-agent architecture. We therefore need to encourage 
independence of the agents, which could also lead to specialization of labor and thus faster learning. To the extent that independent agents can solve a task, independence is a clear advantage because the 
learning problem can be decomposed into smaller problems, i.e., each agent can learn without concern about the behavior of other agents.  However, most tasks require \emph{some} coordination among agents. We therefore pursue an approach 
that attempts to regularize toward independence: agents pay a penalty for modulating their behavior based on the information 
provided by {\textsc{saf}}. This penalty, which is added to the primary task-based reward, discourages unnecessary use of the 
communication channel.




The penalty is expressed as the conditional mutual information between an agent's behavior, $B$, and the information, $M$,
obtained from {\textsc{saf}}, given the agent's current belief state, $S$, denoted $I\!\left(B; M \mid S\right)$. 
Previous work has shown how to optimize $I\!\left(B;M \mid S\right)$ using the framework of KL-Regularized RL  \citet{teh2017distral,  galashov2019information, goyal2019infobot, tirumala2020behavior}. 
This optimization encourages the agent to act according to a \emph{default} or \emph{prior} policy that is insensitive to {\textsc{saf}}.
 To see that minimizing $I\!\left(B;M \mid S\right)$ is achieved by minimizing the KL divergence between an agent's policy
 and the default one, note that
$I\!\left(B;M \mid S\right) = \mathbb{E}_{\pi_\theta}\left[D_{\text{KL}}\!\left[\pi_\theta \!\left(B \mid S,M\right) \mid \pi_0 \!\left(B \mid S\right) \right]\right]$, 
where $\pi_\theta \!\left(B \mid S,M\right)$ is the agent's {\textsc{saf}}-sensitive policy,
$\pi_0 \!\left(B \mid S\right)= \pi_\theta \!\left(B \mid S\right)$  is the default policy,
$\mathbb{E}_{\pi_\theta}$ denotes an expectation over trajectories generated by $\pi_\theta$, and
$D_{\text{KL}}$ is the  Kullback-Leibler divergence.
In past work, the information asymmetry obtained by prohibiting the default policy access to shared information
has been shown to improve transfer and generalization
\citet{galashov2019information, goyal2019infobot, tirumala2020behavior}. 

\textbf{Policy switching}. We were deliberately vague in describing $B$ as an agent's behavior.  $B$ might refer to the actions an agent takes \citet[as it has in][]{goyal2019infobot, galashov2019information, tirumala2020behavior}, but in the present work, we endow agents with a discrete set of distinct policies to select from, and the agents have an explicit decision-making
component that selects a policy stochastically conditioned on $S$ and $M$.  The communicated information $M$ is used only
to select a policy, not to select actions conditioned on the policy. This mixture-of-policies approach 
limits the manner in which {\textsc{saf}}\ can micromanage an agent's behavior, and it has been shown to be effective in
endowing agents with different behavior modes \citet{goyal2021scoff,tirumala2020behavior}.

In the previous paragraphs, we introduced three key ideas that operate synergistically. First, communication among agents is via an intelligent channel, {\textsc{saf}}. Second, each agent is incentivized to act independently and avoid relying on {\textsc{saf}}. Third, agents
operate according to a mixture-of-policies, where {\textsc{saf}}\ provides the signal to select the policy.
To better appreciate how these ideas work in tandem, consider the naturalistic example of a herd of deer coordinating their
behavior. By default, each animal's policy is to graze in a field. But when one of the animals senses danger, the herd
needs to escape. They need to move in unison because if they split up, it will be easier for a predator to trap them.
Suppose that they can escape either to the north or the east, each characterized by a different policy. {\textsc{saf}}\ in
this case will relay the danger alert and will collect information from the individuals and suggest a direction so that
the deer can escape in coordination. Each deer is responsible for avoiding obstacles and avoiding running into other deer,
and thus they operate autonomously with only the high-level guidance from {\textsc{saf}}.

\vspace{-3mm}
\paragraph{Contributions.} The key contributions of our work are as follows: (a) we propose a novel architecture for multi-agent RL. Instead of agents communicating directly between one another, communication is mediated by a facilitator, {\textsc{saf}}, which itself uses the history of communication and active computation to improve the collective performance of all the agents, (b) To ensure agent autonomy, different agents are incentivized to minimize the influence of {\textsc{saf}}\ on their behavior, (c) To further promote autonomy, agent behavior is only coarsely modulated by {\textsc{saf}}, much in the way that a Ph.D. student's research direction is only guided at a high level by their advisor. In the case of our MARL architecture, this modulation  comes in the form of policy selection, which is made explicit via a policy mixture model with a component that switches among policies (see Figure \ref{fig:illust1}), (d) We show the performance of the proposed method on different MARL environments in cooperative setting. We also conduct various ablations and understand the role of different components namely the use of intelligent channel \emph{{\textsc{saf}}}, maximizing agent independence and the use of policy pool. We show that the collective performance of the agent which uses all the three components is higher as compared to an agent which only uses one of the components. 

\iffalse
\begin{itemize}
    \item We propose a novel architecture for multi-agent RL. Instead of agents communicating directly between one another, communication is mediated by a facilitator, {\textsc{saf}}, which itself uses the history of communication and active computation to improve the collective performance of all the agents. 
    \item To ensure agent autonomy, they are incentivized to minimize the influence of {\textsc{saf}}\ on their behavior.
    \item To further promote autonomy, agent behavior is only coarsely modulated by {\textsc{saf}}, much in the way that a Ph.D. student's
    research direction is only guided at a high level by their advisor. In the case of our MARL architecture, this modulation 
    comes in the form of policy selection, which is made explicit via a policy mixture model with a component that switches 
    among policies (see Figure \ref{fig:illust1}).
    \item We show the performance of the proposed method on different MARL environments in cooperative setting. We also conduct various ablations and understand the role of different components namely the use of intelligent channel \emph{{\textsc{saf}}}, maximizing agent independence and the use of policy pool. We show that the collective performance of the agent which uses all the three components is more as compared to an agent which only uses one of the components. 
   
    
   
   
   
\end{itemize}

\fi






 

















\section{Background and Notation}
In this work, we consider a multi-agent version of Markov decision processes (MDPs) with partially observable Markov environments (POMDP).
The environment is defined as $(\mathcal{N},\mathcal{S},\mathcal{O},\mathcal{A},\Pi,R,\gamma)$. $\mathcal{N} = \{1,...,N\}$ denotes the set of $N>1$ agents. $\mathcal{S}$ describes all possible states of all agents.  $\mathcal{A} = A_1\times\dots\times A_N$ denotes the joint action space and $a_{i,t}\in A_i$ refers to the action of agent $i$ at time step $t$. $\mathcal{O}=O_{1}\times\dots\times O_{N}$
denotes the set of partial observation where $o_{i,t}\in O_i$ stands for partial observation of agent $i$ at time step $t$. $\Pi$ is the set of policies available to the agents. To choose actions, agent $i$ uses a stochastic policy $\pi_{\theta_i}:O_i \times A_i \mapsto [0,1]$. Actions from all agents together produces the transition to the next state according to transition function $\mathcal{T}:(s_t, a_{1,t}, \dots, a_{N,t}) \mapsto s_{t+1}$ where $s_t\in\mathcal{S}$ is the joint state at timestep $t$. 
$R:\mathcal{S}\times\mathcal{A}\mapsto \mathbb{R}$ is the global reward function conditioned on the joint state and actions. At timestep $t$, the agent team receives a reward $r_t=R(s_t,a_{1,t},\dots,a_{N,t})$ based on the current total state $s_t$ and joint action $a_{1,t},\dots, a_{N,t}$. $\gamma$ is the discount factor for future rewards. In this study, we focus on cooperative MARL with partial
observations.








\section{MARL with a Facilitator}

We present our approach, first in terms of a high level overview of the mechanisms enabling inter-agent communication
(Section~\ref{sec:hlo}), and then with a detailed description that steps through the algorithm (Section~\ref{sec:da}).
For further details, see Algorithm \ref{alg:attention} in Appendix~\ref{appx:alg}.


\subsection{High Level Overview}
\label{sec:hlo}



 \textbf{Stateful active facilitator.} The facilitator, \emph{{\textsc{saf}}}, consists of 
$l$ stateful memory \emph{slots}, each a vector of $d_m$ elements. The slots are randomly initialized at the beginning of an episode. The state of the \emph{{\textsc{saf}}} is updated once per time step, where a time step corresponds to all agents performing one action. In this update, \emph{{\textsc{saf}}} integrates information provided by the agents into its slot memory and then
outputs a message $M$ that any agent can use.


\textbf{Policy switching using a shared policy pool.} Although all agents operate in the same environment, 
their contexts and objectives may vary. A policy pool $\Pi = \{ \pi^1, \ldots, \pi^U\}$, shared among agents, 
enables agents to exhibit diverse behaviors and have distinct goals. Each policy $\pi^u \in\Pi$ is associated with a 
learned signature key  $k_{\pi^u}$. Using differentiable hard attention with  Gumbel-softmax \citet{jang2016categorical},
an agent can  dynamically select one of  the policies at each time step via a query formed from its belief state, $S$, 
and the message conveyed by {\textsc{saf}}, $M$. 

\textbf{Reducing agent dependence on {\textsc{saf}}.} To reduce an agent's dependence on {\textsc{saf}}, each agent is penalized
according to the KL divergence $D_\text{KL} [ \Pr(Z \mid S,M) \mid\mid \Pr(Z \mid S) ]$, where $Z$ is the agent's policy choice.  







\subsection{Detailed Algorithm}
\label{sec:da}
\textbf{Step 1: Agents pass messages to {\textsc{saf}}.} At step $t$, each agent $i$ receives a partial observation $o_{i, t}$ 
of the environment. This observation is used to update the agent's belief state, $s_{i, t}$, which in turn is
used to generate a message for {\textsc{saf}}.  The message,  $m'_{i, t} = g_{enc}(s_{i, t})$ is a vector of dimensionality
${d_m}$. We denote the set of  messages generated by the agents at time step $t$ as 
${\bm{M}}'_t \equiv \{m'_{i, t} | 1 \leq i \leq N\}  \in \mathbb{R}^{N\times d_m}$.





\textbf{Step 2: \emph{{\textsc{saf}}} integrates information from the agents.} 
\emph{{\textsc{saf}}} dynamically integrates the information from all the agents and incorporates the information 
that it finds interesting into its state representation. The \emph{{\textsc{saf}}} slot memory at time step $t$ is a set of vectors 
$F_t \in \mathbb{R}^{l \times d_m}$, each row representing one of the $l$ slots. The state of 
\emph{{\textsc{saf}}} is updated based on agent messages, ensuring that only the important 
information is incorporated.  {\textsc{saf}}\  achieves this via the use of query-key-value attention mechanism 
\citet{bahdanau2014neural, vaswani2017attention}.  In this case, the query is a function of {\textsc{saf}}'s state 
(a set of slots), represented by matrix ${\bm{F}}_{t}$ (with one row per slot), i.e., $\widetilde{{\bm{Q}}} = {\bm{F}}_{t} \widetilde{{\bm{W}}}^{q}$.  Keys and values are a function of the messages from individual agents. Dot product attention is applied
to obtain the updated state of the slots:  ${\bm{F}}_{t} \leftarrow \mathrm{softmax}\left(\frac{\widetilde{{\bm{Q}}}({\bm{M}}'_{t} \widetilde{{\bm{W}}}^{e})^\mathrm{T}}{\sqrt{d_e}}\right){\bm{M}}'_{t} \widetilde{{\bm{W}}}^{v}$. The use of softmax to write into $l$ slots leads to a standard soft competition among agents to influence the state of \emph{{\textsc{saf}}}. Next, self-attention
is applied over the slots of {\textsc{saf}}\ to obtain its updated state.



\textbf{Step 3: \emph{{\textsc{saf}}}  modulates the behavior of other agents.} \emph{{\textsc{saf}}} makes the updated state available to the agents should they deem to use it. We again utilize an attention mechanism to perform this reading operation. All the agents create queries $Q^s_{t} = \{q^s_{i, t} | 1 \leq i \leq N\}\in\mathbb{R}^{N\times d_e}$ where $q^{s}_{i,t}$ is query generated using the encoded partial observations of agent $i$: $q^{s}_{i, t} = W^{q}_{write}s_{i, t}$. Generated queries are matched with the keys $\kappa = F_t {\bm{W}}^{e}\in\mathbb{R}^{l\times d_e}$ from the updated state of \emph{{\textsc{saf}}} (a set of slots), forming the attention matrix
\mbox{$\alpha= \mathrm{softmax} \left( \frac{Q^s_t \kappa^T
   }{\sqrt{d_e}}\right)$}. 
The slot values generated by each slot of \emph{{\textsc{saf}}}'s state  and the attention weights are then made available to all the agents: $$
{\bm{M}}_t = \mathrm{softmax} \left( \frac{Q^s_t \kappa^T
   }{\sqrt{d_e}}\right) {\bm{F}}_t W^v 
$$

Here ${\bm{M}}_t=  \{m_{i, t} | 1 \leq i \leq N\}$, where $m_{i, t}$ is the message made available to the agent $i$.


\textbf{Step 4: Policy switching.} The encoded partial observation of an agent $s_{i, t}$ and the information made available to  each agent $m_{i, t}$ is used to select a policy $\pi^{u}$ for that agent from the pool of policies $\Pi$ via a straight-through Gumbel-softmax attention mechanism to make a differentiable approximately one-hot selection. Each policy has an associated signature key which is initialized randomly at the start of the training: $K^{\Pi}_t=\{K^{\pi^{u}}_{t} | 1 \leq u \leq U\}$. These keys are matched against the queries computed as deterministic function of encoded partial observation and the information made available to each agent.  
$q^{policy}_{i, t} = g_{psel}(s_{i, t},  m_{i, t})$  where $ g_{psel} $ is parameterized as a neural network.

\[ \mathrm{index}_i= \mathrm{GumbelSoftmax}\left(\frac{q^{policy}_{i, t} (K^{\Pi}_t)^T}{\sqrt{d_m}}\right)\]

As a result of this attention procedure, agent $i$ selects a policy $\pi^{index_{i}}$. This operation is performed independently for each agent, i.e. each agent selects a policy from the policy pool.


\textbf{Step 5: Maximizing agent independence.}  We minimize the dependence of agent on the information made available to each agent. We do this by optimizing the conditional mutual information by upper bounding the KL and penalizing $I\!\left(Z; M\mid S\right)$ \footnote{$I\!\left(Z; M\mid S\right) \geq I\!\left(A;M\mid S\right)$ \citet{galashov2019information, goyal2019infobot} such that $\pi_{\theta}\!\left(A \mid S, M\right) = \sum_z p_\text{enc}\!\left(z\mid S, M\right) p_\text{dec}\!\left(A\mid S,z\right)$}.

Thus, we can instead maximize this lower bound on $J\!\left(\theta\right)$:
\begin{equation}
\begin{aligned}
J\!\left(\theta\right) &\geq \mathbb{E}_{\pi_\theta}\!\left[r\right]-\beta I\!\left(Z; M\mid S\right) 
\label{eq:information_bottleneck}
\end{aligned}
\end{equation}

where $\mathbb{E}_{\pi_\theta}$ denotes an expectation over trajectories across different agents, $\beta > 0$ is a trade-off parameter and $r=\sum_{t=0}^T r_t$ is the total return up to the horizon $T$.


\section{Related Work}

\textbf{Centralized training decentralized execution (CTDE).} These approaches are among the most commonly adopted variations for MARL in cooperative tasks. They usually involve a centralized critic which takes in global information, i.e. information from multiple agents and decentralized policies which are guided by the critic. \citet{Foerster2018CounterfactualMP} uses the standard centralized critic decentralized actors framework with a \textit{counterfactual baseline}. \citet{lowe2017multi} and \citet{Yu2021TheSE} propose the extension of single-agent DDPG and single-agent PPO respectively, to a multi-agent framework with a centralized critic and decentralized policies during training and completely decentralized execution. \citet{li2021celebrating} uses an information theory based objective to promote novel behaviours in CTDE based approaches. Value Decomposition (\cite{Sunehag2018ValueDecompositionNF}, \citet{Rashid2018QMIXMV}, \citet{wang2020qplex}, \citet{Mahajan2019MAVENMV}) approaches learn a factorized action-value function. \citet{Sunehag2018ValueDecompositionNF} proposes Value Decomposition Networks (VDN) which simply add the action-value function of each agent to get the final action value function. \citet{Rashid2018QMIXMV} uses a mixing network to combine the action-value functions of each agent in a non-linear fashion.

\textbf{Communication in MARL.} Several approaches use emergent communication protocols for MARL. Communication involves deciding the message to be shared and determining how the message sending process is implemented. \citet{foerster2016learning} and \citet{Sukhbaatar2015} have done work on learnable inter-agent communication protocols. \citet{Jiang2018LearningAC} first proposed using attention for communication where attention is used for integrating the received information as well as determining when communication is needed.  \citet{das2019tarmac} uses multiple rounds of direct pairwise inter-agent communication in addition to the centralized critic where the messages sent by each agent are formed by encoding it's partial observation, location information, etc., and the messages received by each agent are integrated into it's current state by using a soft-attention mechanism.  \citet{kim2020communication} uses intentions represented as encoded imagined trajectories as messages where the encoding is done via a soft-attention mechanism with the messages received by the agent. \citet{wang2021tom2c} trains a model for each agent to infer the intentions of other agents in a supervised manner, where the communicated message denotes the intentions of each agent. The above mentioned approaches require a computational complexity which is quadratic in the number agents. Our approach has a computational complexity which is linear in the number of agents. Moreover, we show that our approach is able to outperform several standard baselines using messages which can be computed as simply the encoded partial observations of each agent. 

\textbf{Option-critic in Multi-agent Hierarchical Reinforcement learning}
 A classical approach within the Hierarchical Reinforcement Learning (HRL) \citet{Pateria2021Hierarchical} literature is modeling agents' intentions as options \citet{Sutton1999Between}, temporally extended actions that aim to achieve subgoals in a finite time horizon. Recently, \citet{Bacon2017The} proposed an end-to-end option-critic architecture capable of learning both options and the related policy. However, despite the advantages brought by using options, due to the temporally-extended nature of options, agents' responses can be inconsistent when the environment or other agents' behaviour change. To tackle this problem, \citet{Han2019Multi} proposed a dynamical termination scheme which allows an agent to flexibly terminate its current option.
 Although both option-critic and our approach use a pool of actors, while in the former case actors model options, in the latter one actors model policies, preventing agents' inconsistent behaviours. Moreover, although within the option-critic framework the optimality of the learned hierarchical policy can be theoretically guaranteed \citet{Chakravorty2020Option}, the learned options cannot be easily transferred to other tasks \citet{Pateria2021Hierarchical}. Furthermore, many works proposed within the option-critic framework (\citet{Klissarov2017Learning},\citet{Riemer2018LearningAO},\citet{Khetarpal2020OptionsOI}) perform poorly on sparse reward tasks (\citet{Bagaria2020Option},\citet{Nachum2018Data}), while experimental results show that our approach presents comparable performances with SOTA baselines even on that case (e.g., Waterworld Environment \citet{gupta2017cooperative})   



\textbf{Independent learning.} Independent Learning in MARL consists of each agent optimizing its policy locally using its observation and in the absence of any communication or centralized controller (as in CTDE). These approaches mainly consist of the extension of single-agent RL approaches to multi-agent settings where each agent learns independently using local observations, considering other agents as part of the environment. \citet{Tan1993MultiAgentRL} proposed Independent Q-Learning (IQL) where each agent independently learns it's own action-value function. \citet{Witt2020IsIL} demonstrates that PPO, when used for independent learning in multi-agent settings (called Independent PPO or IPPO) is in fact capable of beating several state of the art approaches in MARL on competitive benchmarks such as StarCraft and can hence serve as a strong baseline.


\textbf{Communication bottleneck.} With the emergence of modular deep learning architectures \citet{vaswani2017attention, goyal2021neural, scarselli2008graph, bronstein2017geometric, kipf2018neural, battaglia2018relational}, which require communication among different model components, there have been development of methods which introduce a bottleneck in this communication to a fixed bandwidth which helps communicating only the relevant information. \citet{DianboLiu2021DiscreteValuedNC} uses a VQ-VAE \citet{oord2017neural} to discretize the information being communicated. Inspired by the theories in cognitive neuroscience \citet{Baars1988-BAAACT, shanahan2006cognitive, Dehaene-et-al-2017}, \citet{goyal2021coordination} proposes the use of a generic \textit{shared workspace} which acts as a bottleneck for communication among different components of multi-component architectures and promotes the emergence of \textit{specialist} components. We use a \emph{{\textsc{saf}}} similar to the shared workspace in which different agents compete to write information to and read information from. 
 



\iffalse
\subsection{Shared Knowledge Source}
We call the list of vectors used to coordinate messages a shared workspace. Attention mechanism is used for shared workspace mechanisms as in our previous paper. The query is a function of the state of the memory matrix $M_t$ at time step t (with one row per slot of the memory), $Q_t=M_t W^q$. In this study, each row in $M_{j,t}=E_j(mean(O_t))$ where $E_j$ is an MLP based encoder model for GWS slot j. Different E models do not share parameters.
$O_t$ is the message matrix at time step t with each row representing message sent from an agent. $mean(O_t)$ is the mean vector of messages from all agents. The each message type varies among different MARL algorithms (see results section for details). The key for GWS writing attention is $K_t=O_t W^k$. Updated shared workspace slots are $M'_{t}=softmax(\frac{Q_t K_t}{\sqrt{d}})O_t$ where d is the key dimension.

After information writing into shared workspace, the information is broadcasted from the shared workspace using attention mechanism to all agents. $Q^{read}_t=O_t W'^q$ and $K^{read}_t=M'_t W'^k$. Messages sent to agents are $O'_{t}=softmax(\frac{Q^{read}_t K^{read}_t}{\sqrt{d_m}})M'_t$



\subsection{Dynamic Policy Selection}
Each agent has access to a pool of more than one policies and select only one of the policies to choose action by following the these steps: 1) Parsing the state of an agent $j$. 2) Selectively receive message from the shared workspace via attention mechanism and 3) Select policy by attention mechanism with Gumbel-softmax. 4) Apply the policy and take action 5) Update the policy


\subsection{Communication Cost}
$m'_{i,t}$ is the message received by agent $i$ from the shared workspace at time step $t$. $s_{i,t}$ is the internal state that come from partial observation of the agent. $\pi_{i,t}$ is the policy picked by agent $i$ at time step $t$. 

\begin{align}
R_i &= R^{env}_{i} + R^{independence}_i \\
R_i &= \mathbb{E}\left[\sum_{t} \gamma^t r(s_{i,t},a_{i,t}) -\beta \gamma^t KL[ \pi_{i,t}(a_{i,t}|s_{i,t},m'_{i,t})||\pi_{i,t}(a_{i,t}|s_{i,t})]\right]
\end{align}

the KL term penalize divergence of $\pi_{i,t}(a_t|s_{i,t})$ from $\pi_{i,t}(a_t|s_{i,t},m'_{i,t})$. Therefore, minimize influence of the message on the action prediction and enhancer independence of each agent.

\fi

\section{Experiments}
We investigate how well \emph{{\textsc{saf}}}, policy switching, and maximizing independence work together to improve cooperative MARL. Next, we compare our method with a state-of-the-art cognitive science-inspired approach used in multi-agent communication and cooperation\citet{wang2021tom2c}. Lastly, to understand if our method can be adapted to MARL methods that send different messages among agents, we integrate our machinery into a MARL algorithm that iteratively sends hypothetical actions among cooperative agents.

\subsection{Environments}
In this section, we describe the various MARL environments which we considered for our experiments..


\vspace{-3mm}
\paragraph{GhostRun environment.} We use the \textbf{GhostRun Environment} which is an adaptation of the Drone environment available from \citet{shuo2019maenvs}. The environment consists of multiple drones, each with a partial view of the ground below them. The ground consists of ghosts - represented by red dots, trees - depicted by green dots, and obstacles - depicted by black dots. The ghosts move about randomly whereas the trees and obstacles are stationary (see Figure \ref{fig:Env} (a)). The task at hand is a cooperative task where the agents must work together to escape from ghosts. Hence, the number of ghosts in each agent's partial observation of the environment must be minimized. The reward received by each agent at each time step is the negative of the total number of ghosts in the view of all the agents and a step cost of -1 for each step taken.

\begin{figure*}[htp]
    \centering
    \subfigure[GhostRun Environment]{
    \includegraphics[width=0.2\linewidth]{figures/Drone.png}}\hspace{15mm}
    \subfigure[PistonBall Environment]{
    \includegraphics[width=0.1\linewidth]{figures/PistonBall.png}}\hspace{15mm}
    \subfigure[MSTC Environment]{
    \includegraphics[width=0.2\linewidth]{figures/pose_environment.png}}
    
    \caption{GhostRun, PistonBall and MSTC environments. In the GhostRun environment (left panel) Each agent has its own partial view of the environment and the reward is to escape from ghosts (red dots). Different agents can communicate their encoded views with each other. In the PistonBall environment (middle panel) all agents (pistons) work together to move the ball from one side to the other. In the MSTC environment (right panel), sensors (gray dots) need to cover as many targets as possible (red dots). MSTC figure adapted from \citet{wang2021tom2c}}
    \label{fig:Env}
\end{figure*}

\vspace{-3mm}
\paragraph{Multi-Sensor Target Coverage.} We use the multi-sensor multi-target tracking (MSTC) environment from \citet{wang2021tom2c}. There are sensors (which are the agents) and targets. The goal is for the sensors to observe as many targets as possible at once (see Figure \ref{fig:Env} (c)). Each sensor has a partial observation of its surroundings and its view may be obstructed by obstacles. The targets can move according to one of two rules: according to a random walk, or along the shortest path to reach a previously sampled destination. At the beginning of each episode, the location of targets, sensors and obstacles is randomly sampled. The maximum episode length is $100$ steps and the reward is defined as $r=\frac{1}{m}\sum_q T_q$ where $T_q=1$ if the target $T_q$ is observed by any sensor and $0$ otherwise. If no target is observed (i.e. $T_q=0\quad\forall q$) then $r=-0.1$.

\vspace{-3mm}
\paragraph{PistonBall Environment.} This is a simple physics-based cooperative game where the goal is to move the ball to the left wall of the game border by activating the vertically moving pistons. Each agent’s observation is an RGB image of the two pistons (or the wall) next to the agent and the space above them (see Figure \ref{fig:Env} (b)). Every piston can be acted on at any time. The action space can be discrete: 0 to move down, 1 to stay still, and 2 to move up. Alternatively, the action space can be continuous: the action value in the range $[-1, 1]$ is proportional to the amount by which the pistons are raised or lowered. Continuous actions are scaled by a factor of $4$, so that in both the discrete and continuous action space, the action $1$ will move a piston $4$ pixels up, and $-1$ will move pistons 4 pixels down.





\subsection{Baselines: Varying Along Different Axes}


In our experiments, we consider multiple baselines which use some aspects of the proposed method, such as the use of stateful and active facilitator \emph{{\textsc{saf}}}, shared pool of policies and maximizing agent independence. In particular, we try to disentangle the contributions of these different components of the proposed architecture.
The following baselines are evaluated:

\textit{Multiple Agents with no communication} [\textsc{I}]: There are multiple independent agents with no communication between them.

\textit{Multiple Agents with pairwise communication} [\textsc{P}]: 
Every pair of agents can communicate with each other via self-attention \citet{vaswani2017attention}.

\textit{Multiple Agents with pairwise communication and shared pool of policies} [\textsc{P + SP}]: Every pair of agents can communicate with each other via self-attention and each agent can select a different policy from the shared pool of policies. The difference from the proposed method would be that this variant does not make use of \emph{{\textsc{saf}}}. 

\textit{Multiple Agents with \textsc{\emph{{\textsc{saf}}}}} [\textsc{\emph{{\textsc{saf}}}}]: Here, we consider multiple agents such that different agents communicate with each other via \emph{{\textsc{saf}}}, and all agents share the same policy.

\textit{Multiple Agents with \textsc{\emph{{\textsc{saf}}}} and policy pool} [\textsc{\emph{{\textsc{saf}}} + SP}]: the  agents can communicate with each other via \emph{{\textsc{saf}}}, and also share the pool of policies such that each agent can dynamically select a policy from the policy pool. 

\textit{Multiple Agents with \textsc{\emph{{\textsc{saf}}}}, policy pool and maximizing agent dependence.} [\textsc{\emph{{\textsc{saf}}} + SP + KL}]: The agents can communicate with each other via \emph{{\textsc{saf}}}, share the pool of policies, and also agents minimize the dependence of the policy on the information made available by \emph{{\textsc{saf}}}.




\vspace{-4mm}
\paragraph{Hyperparameters.}  We consider two hyper-parameters in the proposed model: (1) number of policies in the pool ($U=N_{policies}$), (2) number of slots in the \emph{{\textsc{saf}}} ($l=N_{slots}$).  For all our experiments, we use $N_{policies}=3$.




\begin{figure*}[ht!]
    \centering
    \subfigure[2 agents]{
    \includegraphics[width=0.3\linewidth]{figures/Scale_agents2_test.pdf}
    }
    \subfigure[5 agents]{
    \includegraphics[width=0.3\linewidth]{figures/Scale_agents5_test.pdf}
    }\\
    \subfigure[15 agents]{
    \includegraphics[width=0.3\linewidth]{figures/Scale_agents15_test.pdf}
    }
    \subfigure[30 agents]{
    \includegraphics[width=0.3\linewidth]{figures/Scale_agents30_test.pdf}
    }
    
    
   
   
   
    \caption{\textsc{{\textsc{saf}}}, shared policy pool and maximizing agent's independence in cooperative MARL bring significant improvement in performance when there are more cooperative agents in the same environment in the training and evaluation process. (a) 2 cooperative agents  (b) 5 cooperative agents (c) 15 cooperative agents (d) 30 cooperative agents. Our method (\emph{{\textsc{saf}}} + SP) performs better when there is a  larger number of agents in the environment .All experiments in this figure were conducted in GhostRun environment}
    \label{fig:Scale}
\end{figure*}



\vspace{-3mm}
\paragraph{Stateful and Active communication Facilitator (\emph{{\textsc{saf}}}) improves communication in MARL.}
We investigate the use of \emph{{\textsc{saf}}} for coordinating among different agents in the Ghost Run environment by comparing different communication schemes. (a). We compare three communication strategies [\textsc{I}], [\textsc{P}] and using \emph{{\textsc{saf}}}. We find that the proposed \emph{{\textsc{saf}}} based approach outperforms the other two settings and shows an improvement of about \textbf{45\%} over the other two baselines as shown in figure \ref{fig:GhostRun}(a). This shows that introducing the \emph{{\textsc{saf}}} helps improve inter-agent communication. The \emph{{\textsc{saf}}} acts as a specialist agent / communication bottleneck for the messages sent by all agents. It filters out only the relevant information for providing the relevant \textit{context} to each agent making it more robust, which may not be possible in pairwise communication i.e., where every pair of agents communicate with each other. 

We also compare [\emph{{\textsc{saf}}}+SP] against a graph-based communication baseline \citet{wang2021tom2c} that we simply name \emph{\textsc{Graph}}, on the MSTC environment. In \emph{\textsc{Graph}}, the agents generate a message from their observation and communication is done through a fully-connected graph whose nodes are the updated messages. These are used top optimize the agents' policies. We show the test reward  in Table \ref{tom-saf}.

\begin{figure*}[ht!]
    \centering
    \subfigure[Communication Methods]{
    \includegraphics[width=0.2\linewidth]{figures/CommunicationMethod_test.pdf}
    }
    \subfigure[Policy Selection]{
    \includegraphics[width=0.2\linewidth]{figures/PolicySelection_test.pdf}
    }
    \subfigure[Varying beta]{
    \includegraphics[width=0.2\linewidth]{figures/BetaKL_test.pdf}
    }
    \subfigure[Maximizing Independence]{
    \includegraphics[width=0.2\linewidth]{figures/KLTarget_OOD.pdf}
    }
    \caption{\emph{{\textsc{saf}}}, shared policy pool and maximizing agent's independence in cooperative MARL. (a) Comparison between different communication setups (b) Comparison between different policy selection strategies (c) Comparison between coefficients for KL Loss. (d) Comparison between penalizing policy selection and penalizing action selection}
    \label{fig:GhostRun}
\end{figure*}







\begin{figure*}[t]
    \centering
    \includegraphics[width=0.26\linewidth]{figures/PistonBall_reward.pdf}
    \includegraphics[width=0.26\linewidth]{figures/PistonBall_length.pdf}
    \caption{\emph{{\textsc{saf}}}, shared policy pool and maximising agent's independence in cooperative MARL with agents communicating hypothetical actions as messages in PistonBall environment. Our method has higher cumulative rewards while decreasing the average episode's length compared to the baselines.}
    \label{fig:HypotheticalActions}
\end{figure*}


\begin{table}[]
\caption{Average reward on the test setting in the MSTC environment for the [\emph{{\textsc{saf}}}+SP] architecture and the \emph{\textsc{Graph}} model}
\label{tom-saf}
\vskip 0in
\begin{center}
\begin{small}
\begin{sc}
\begin{tabular}{lll}
\toprule
 & [\emph{{\textsc{saf}}}+SP] & \emph{\textsc{Graph}} \\
\midrule
Expected Return   & $71.26 (\pm 5.81)$ &   $61.22 (\pm 8.15) $    \\
\bottomrule
\end{tabular}
\end{sc}
\end{small}
\end{center}
\vskip -0.1in
\end{table}

\vspace{-3mm}
\paragraph{Policy switching using shared policy pool improves performance.} We investigate the significance of using policy switching using a shared pool of policies for each agent. We experiment with two settings: \textsc{\emph{{\textsc{saf}}}} and \textsc{\emph{{\textsc{saf}}} + SP}.  The inter-agent communication in both settings is facilitated via the use of \emph{{\textsc{saf}}}. The results are shown in figure \ref{fig:GhostRun}(b). Using a shared pool of policies with dynamic selection shows a relative improvement of \textbf{25\%} over the fixed policy setting.  This shows that the mixture of policies helps to transfer knowledge about different behaviors across different agents. 





\vspace{-3mm}
\paragraph{Minimizing dependence on \emph{{\textsc{saf}}}.} Here, we investigate the effect of minimizing the dependence of each agent's policy on the information that is available from the \emph{{\textsc{saf}}} in the form of its state, quantified by the conditional mutual information $I\!\left(Z;M \mid S\right)$, where $Z$ is the agent's policy choice. One can directly optimize it  as discussed in eq.\ref{eq:information_bottleneck}. We also experimented with different values of the coefficients ($\beta$) of the KL based regularization loss. Figure \ref{fig:GhostRun}(c) shows that minimizing the dependence of each agent on the information from \emph{{\textsc{saf}}} helps improve convergence. 

We also compare the effect of directly optimizing the conditional mutual information $I\!\left(A;M \mid S\right)$ i.e., where the information from \emph{{\textsc{saf}}} is used to also select actions (as compared when information from \emph{{\textsc{saf}}} is only used to select the policy) and the agent is optimized to minimize dependence on information from \emph{{\textsc{saf}}}. We call the method where we optimize the mutual information directly on actions as \textit{regularized action selection}, and the method where we optimize the upper bound on mutual information as \textit{regularized policy selection}. Figure \ref{fig:GhostRun}(d) compares the effect of regularizing action as compared to regularizing the policy selection, and shows that regularizing the policy selection (i.e., upper bound) achieves better results as compared to regularizing the action selection. 



\paragraph{\textsc{SAF} is significantly helpful for MARL training with large number of agents.} Here we investigate how the proposed method scales with increasing the number of agents. Learning a stateful, active facilitator \emph{{\textsc{saf}}} which integrates information across agents should scale better as compared to using a communication protocol where every agent interacts with every other agent. To test this hypothesis, we compare the performance of five different approaches: \emph{{\textsc{saf}}}, [\emph{{\textsc{saf}}}+\textsc{SP}], [\textsc{P}], [\textsc{P} + \textsc{SP}], [\textsc{I}] by varying the number of agents. Figure \ref{fig:Scale} shows the result of these different methods by varying the number of agents. As the number of agents increases, the relative difference between the proposed method and the different baselines ([\textsc{I}], [\textsc{P}], [\textsc{P}+\textsc{SP}]) increase significantly. 

\paragraph{Flexible plug-in tool for many different MARL settings.} In the previous sections, we showed that \emph{{\textsc{saf}}}, shared policy, and maximizing independence boost performance on cooperative MARL tasks in which agents communicate messages generated as a function of the observed environment. In this section, we investigate the possibility of applying the proposed method on cooperative MARL algorithms in which agents iteratively communicate \textit{hypothetical actions} with each other before taking actions. The performance of the proposed method is compared with two recent approaches that communicate hypothetical actions, namely consensus update and iterative reasoning (IR) \cite{zhang2018fully}. Consensus update conducts graph-based local communication and IR communicate hypothetical actions among agents such that an agent's policy is conditional on its teammates. Our experimental results show that our method achieves higher cumulative rewards in the PistonBall environment while doing so in a lower number of iterations, both of which indicate better performance (see Figure \ref{fig:HypotheticalActions}). This suggests the potential of the proposed method as a flexible plug-in tool in many MARL settings. 



\section{Future Work}
Here, we  introduced three key ideas that operate synergistically. First, communication among agents is conducted via an intelligent channel, {\textsc{saf}}. Second, each agent is incentivized to act independently and avoid relying on {\textsc{saf}}. Third, agents
operate according to a mixture-of-policies, where {\textsc{saf}}\ provides the signal to select the policy. Through extensive experiments we show the utility of all the different components. We also show that the proposed method scales better on increasing the number of the agents and achieves higher returns as compared to various different baselines. Future work may involve scaling the proposed method to more complex multi-agent problems like Starcraft \citet{vinyals2019grandmaster} and Google Research Football \citet{gfootball}.























