
\section*{Appendix}\label{sec:additional}

We present additional results and discussion regarding the behavior of our network and baselines under various conditions.

\vspace{-1mm}
\subsection*{\bf Varying History Window $H$}

In Tables \ref{tab:action_prediction_A1} and \ref{tab:action_prediction_A2}, we present results of training our networks with various history windows $H \in \{10,5,1\}$, using static and dynamic rollouts from the synthetic dataset initial layouts. As expected, lowering the history window size lowers performance of all networks, but \textsc{MGpi}{} either outperforms all baselines, or performs comparably to our strongest baseline Social Pooling. 

The strong performance of Social Pooling is due to the explicit spatial integration of neighbor information in a grid around the agent of interest, resulting in the implicit learning of sensory gating. Note that Social Pooling and Average Pooling networks have no data left out in their inputs - they each get the same data as our proposed network, since relative gaze and pose history is included in their visual history encoding. Furthermore, Social Pooling is made stronger by spatial binning instead of naive overall averaging. While our network must learn to appropriately weight each neighbor for pooling, Social Pooling may instead achieve the learning of attention by learning weighting over the grid. Moreover, propagating each grid's pooled messages forward results in a higher preservation of information than in our network. 

As expected, this advantage is visible in results at lower history window sizes in the static agent case, when it becomes harder to correctly learn neighbor weighting (Table \ref{tab:action_prediction_A1} in this document, Figure 6 in main submission). However, when neighbors are dynamic (Table \ref{tab:action_prediction_A2}), \textsc{MGpi}{} still outperforms all baselines, even at lower history windows. This is because the explicit sensory gating required to infer when to pay attention to a moving agent is advantageous over a static weighting learned over a grid, as in Social Pooling. 

 

Further, we analyze the effect of varying number of grids in Social Pooling, at history $H = 15$. Results in Figure \ref{fig:socpool_compare} show that \textsc{MGpi}{} outperforms Social Pooling even with larger numbers of grids under the mean average precision metric, while performing similarly under the accuracy metric. Further, \textsc{MGpi}{} furnishes us with a natural way to infer group membership in real-world data (Section \ref{sec:expt_group}), which Social Pooling does not.  






\vspace{-1mm}
\subsection*{{\bf Varying Initial Layouts}}

In Tables \ref{tab:action_prediction_B1}  and \ref{tab:action_prediction_B2} we present performance results using datasets generated from synthetic and real-world initial layouts in Synthetic \cite{Cristani2011}, Coffee Break \cite{Cristani2011} and Cocktail Party \cite{Zen2010} datasets. As expected, \textsc{MGpi}{} outperforms all baselines in the dynamic case, while performing comparably to Social Pooling in the static case. This comparison has been discussed in the previous subsection. 

\vspace{-1mm}
\subsection*{\bf Confusion Matrices}
As mentioned in Section \ref{sec:sim}, confusion matrices of action prediction for all networks (Figure \ref{fig:cmat}) show that \texttt{NSO} and \texttt{SSO} work complementarily: \texttt{NSO} predicts better if agents make strong or weak addressing decisions by observing others. On the other hand, \texttt{SSO} performs better when observation of the self is necessary, {\it i.e.}, in order to predict if agents keep speaking, listening, are distracted or responding. \textsc{MGpi}{} outperforms baselines for most actions, both in the static and dynamic case, with lower ambiguity.


\subsection*{\bf Learned Gating Outputs}

We visualize outputs of the KPM-Gate for an agent $A^m$ at various surrounding locations of a neighbor agent $A^{n_i}$ while fixing relative gaze direction. We do so for all cases of initial layout source (Synthetic, Coffee Break or Cocktail Party) and conditions (static or dynamic agents) in Figure \ref{fig:kpm_compare}. Similar to Figure \ref{fig:visualization1}, the black circles represent $A^m$ looking in the $-x$-direction. $A^{n_i}$ is assumed to be looking in the $+x$-direction. Warmer colors indicate higher outputs from the gating function at those corresponding positions of $A^{n_i}$.

As described in Section \ref{sec:expt_group}, such outputs are used to infer group membership of a neighbor agent $A^{n_i}$ with respect to the agent under consideration $A^m$. As can be seen in Figure \ref{fig:kpm_compare}, our network learns appropriate gating in all cases by the $20^{\text{th}}$ epoch of training. Variations between the learned gating under various conditions leads to appropriate learning of the distance function $D(m,n_i)$ for clustering. This custom informative distance function leads to group detection performance comparable to state-of-the-art methods (Table 3).


\begin{table*}[!htbp]
\caption{Static Scenario: Action Prediction (mAP) for $H = 10, 5, 1$, Synthetic \cite{Cristani2011} initial layouts}
\centering
\scalebox{.7}{
\begin{tabular}{lcccc}
\toprule
      Model {\bf (mAP's, $H=10$)} &  $J=2$& 4&8&12 \\ 
     \midrule
     Neighbor States Only \hfill (\texttt{NSO}) & 0.65 & 0.68 & 0.66 & 0.65 \\
     Self States Only \hfill (\texttt{SSO}) & 0.72 & 0.72 & 0.72 & 0.72 \\
     Equal Pooling \hfill (\texttt{EQPOOL}) &0.85 & 0.85& 0.83 & 0.83\\
     Social Pooling \hfill (\texttt{SOCPOOL}) &0.85 & 0.85 & 0.84 & 0.83\\
     \midrule
    
     {\bf \textsc{MGpi}{} Network} & {\bf 0.87} & {\bf 0.88} & {\bf 0.86} & {\bf 0.85}\\
\bottomrule
\end{tabular}
}
\scalebox{.7}{
\begin{tabular}{lcccc}
\toprule
      Model {\bf (mAP's, $H=5$)} &  $J=2$& 4&8&12 \\ 
     \midrule
     Neighbor States Only \hfill (\texttt{NSO}) & 0.59 & 0.60 & 0.60 & 0.59 \\
     Self States Only \hfill (\texttt{SSO}) & 0.59 & 0.59 & 0.59 & 0.59 \\
     Equal Pooling \hfill (\texttt{EQPOOL}) &0.80 & 0.79& 0.76 & 0.76\\
     Social Pooling \hfill (\texttt{SOCPOOL}) &{\bf 0.80} & {\bf 0.80} & {\bf 0.80} & 0.79\\
     \midrule
    
     {\bf \textsc{MGpi}{} Network} & 0.79 & {\bf 0.80} & {\bf 0.80} & {\bf 0.80}\\
\bottomrule
\end{tabular}
}
\scalebox{.7}{
\begin{tabular}{lcccc}
\toprule
      Model {\bf (mAP's, $H=1$)} &  $J=2$& 4&8&12 \\ 
     \midrule
     Neighbor States Only \hfill (\texttt{NSO}) & 0.48 & 0.48 & 0.49 & 0.47 \\
     Self States Only \hfill (\texttt{SSO}) & 0.49 & 0.49 & 0.49 & 0.49 \\
     Equal Pooling \hfill (\texttt{EQPOOL}) &0.64 & 0.63& 0.62 & 0.60\\
     Social Pooling \hfill (\texttt{SOCPOOL}) &{\bf 0.69} & {\bf 0.70} & {\bf 0.69} & {\bf 0.69}\\
     \midrule
    
     {\bf \textsc{MGpi}{} Network} &  {0.68} & {0.68} & {0.67} & {0.66}\\
\bottomrule
\end{tabular}
}\label{tab:action_prediction_A1}
\end{table*}


\begin{table*}[!htbp]
\caption{Dynamic Scenario: Action Prediction (mAP) for $H = 10, 5, 1$, Synthetic \cite{Cristani2011} initial layouts}
\centering
\scalebox{.7}{
\begin{tabular}{lcccc}
\toprule
      Model {\bf (mAP's, $H=10$)} &  $J=2$& 4&8&12 \\ 
     \midrule
     Neighbor States Only \hfill (\texttt{NSO}) & 0.55 & 0.58 & 0.62 & 0.62 \\
     Self States Only \hfill (\texttt{SSO}) & 0.73 & 0.73 & 0.73 & 0.73 \\
     Equal Pooling \hfill (\texttt{EQPOOL}) &0.82 & 0.83& 0.82 & 0.81\\
     Social Pooling \hfill (\texttt{SOCPOOL}) &0.82 & 0.83 & 0.83 & 0.83\\
     \midrule
    
     {\bf \textsc{MGpi}{} Network} & {\bf 0.83} & {\bf 0.85} & {\bf 0.85} & {\bf 0.86}\\
\bottomrule
\end{tabular}
}
\scalebox{.7}{
\begin{tabular}{lcccc}
\toprule
      Model {\bf (mAP's, $H=5$)} &  $J=2$& 4&8&12 \\ 
     \midrule
     Neighbor States Only \hfill (\texttt{NSO}) & 0.50 & 0.55 & 0.58 & 0.58 \\
     Self States Only \hfill (\texttt{SSO}) & 0.63 & 0.63 & 0.63 & 0.63 \\
     Equal Pooling \hfill (\texttt{EQPOOL}) &0.75 & 0.75& 0.74 & 0.75\\
     Social Pooling \hfill (\texttt{SOCPOOL}) &{\bf 0.76} & 0.77 & 0.78 & 0.78\\
     \midrule
    
     {\bf \textsc{MGpi}{} Network} & {\bf 0.76} & {\bf 0.78} & {\bf 0.79} & {\bf 0.79}\\
\bottomrule
\end{tabular}
}
\scalebox{.7}{
\begin{tabular}{lcccc}
\toprule
      Model {\bf (mAP's, $H=1$)} &  $J=2$& 4&8&12 \\ 
     \midrule
     Neighbor States Only \hfill (\texttt{NSO}) & 0.37 & 0.41 & 0.44 & 0.43 \\
     Self States Only \hfill (\texttt{SSO}) & 0.55 & 0.55 & 0.55 & 0.55 \\
     Equal Pooling \hfill (\texttt{EQPOOL}) &0.62 & 0.62& 0.61 & 0.61\\
     Social Pooling \hfill (\texttt{SOCPOOL}) &{0.63} & {0.64} & {0.64} & {0.64}\\
     \midrule
    
     {\bf \textsc{MGpi}{} Network} &  {\bf 0.64} & {\bf 0.65} & {\bf 0.66} & {\bf 0.66}\\
\bottomrule
\end{tabular}
}
\label{tab:action_prediction_A2}
\end{table*}



\begin{figure*}[h]
\centering
\includegraphics[width=0.9\linewidth]{figures/socpool_vs_magdam.pdf}
\caption{{\bf Change in Social Pooling performance} with varying grid size, history window $H = 15$. \textsc{MGpi}{} performs similarly to \texttt{SOCPOOL} under the accuracy metric, and outperforms \texttt{SOCPOOL}  under the mean average precision metric, even with larger grid sizes.}
\label{fig:socpool_compare}
\end{figure*}



\begin{table*}[!htbp]
\caption{Static Scenario: Action Prediction (mAP) for $H = 15$, Synthetic \cite{Cristani2011}, Coffee Break \cite{Cristani2011}, Cocktail Party \cite{Zen2010} initial layouts}
\centering
\scalebox{.7}{
\begin{tabular}{lcccc}
\toprule
      Model {\bf (mAP's, Synthetic)} &  $J=2$& 4&8&12 \\ 
     \midrule
     Neighbor States Only \hfill (\texttt{NSO}) & 0.67 & 0.69 & 0.70 & 0.68 \\
     Self State Only \hfill (\texttt{SSO}) & 0.76 & 0.76 & 0.76 & 0.76 \\
     Equal Pooling \hfill (\texttt{EQPOOL}) &0.87 & 0.86& 0.85 & 0.85\\
     Social Pooling \hfill (\texttt{SOCPOOL}) &0.85 & 0.84 & 0.84 & 0.84\\
     \midrule
    
     {\bf \textsc{MGpi}{} Network} & {\bf 0.88} & {\bf 0.88} & {\bf 0.89} & {\bf 0.88}\\
\bottomrule
\end{tabular}
}
\scalebox{.7}{
\begin{tabular}{lcccc}
\toprule
      Model {\bf (mAP's, Coffee Break)} &  $J=2$& 4&8&12 \\ 
     \midrule
     Neighbor States Only \hfill (\texttt{NSO}) & 0.71 & 0.72 & 0.71 & 0.70 \\
     Self States Only \hfill (\texttt{SSO}) & 0.75 & 0.75 & 0.75 & 0.75 \\
     Equal Pooling \hfill (\texttt{EQPOOL}) &0.89 & 0.88& 0.87 & 0.86\\
     Social Pooling \hfill (\texttt{SOCPOOL}) &{\bf 0.91} & {\bf 0.92} & {\bf 0.91} & {\bf 0.91}\\
     \midrule
    
     {\bf \textsc{MGpi}{} Network} & {0.90} & {0.91} & {\bf 0.91} & {\bf 0.91}\\
\bottomrule
\end{tabular}
}
\scalebox{.7}{
\begin{tabular}{lcccc}
\toprule
      Model {\bf (mAP's, Cocktail Party)} &  $J=2$& 4&8&12 \\ 
     \midrule
     Neighbor States Only \hfill (\texttt{NSO}) & 0.72 & 0.79 & 0.79 & 0.79 \\
     Self States Only \hfill (\texttt{SSO}) & 0.73 & 0.73 & 0.73 & 0.73 \\
     Equal Pooling \hfill (\texttt{EQPOOL}) &0.91 & 0.94& 0.94 & 0.94\\
     Social Pooling \hfill (\texttt{SOCPOOL}) &{\bf 0.92} & {\bf 0.95} & {0.94} & 0.94\\
     \midrule
    
     {\bf \textsc{MGpi}{} Network} & {\bf 0.92} & {\bf 0.95} & {\bf 0.95} & {\bf 0.95}\\
\bottomrule
\end{tabular}
}
\label{tab:action_prediction_B1} 
\end{table*}


\begin{table*}[h]
\caption{Dynamic Scenario: Action Prediction (mAP) for $H = 15$, Synthetic \cite{Cristani2011}, Coffee Break \cite{Cristani2011}, Cocktail Party \cite{Zen2010} initial layouts}
\centering
\scalebox{.7}{
\begin{tabular}{lcccc}
\toprule
      Model {\bf (mAP's)} &  $J=2$& 4&8&12 \\ 
     \midrule
     Neighbor States Only \hfill (\texttt{NSO}) & 0.57 & 0.62 & 0.65 & 0.64 \\
     Self State Only \hfill (\texttt{SSO}) & 0.78 & 0.78 & 0.78 & 0.78 \\
     Equal Pooling \hfill (\texttt{EQPOOL}) &0.85 & 0.86 & 0.85 & 0.85\\
     Social Pooling \hfill (\texttt{SOCPOOL}) &0.85 & 0.85 & 0.86 & 0.85\\
     \midrule
    
     {\bf \textsc{MGpi}{} Network} & {\bf 0.87} & {\bf 0.87} & {\bf 0.88} & {\bf 0.88}\\
\bottomrule
\end{tabular}
}
\scalebox{.7}{
\begin{tabular}{lcccc}
\toprule
      Model {\bf (mAP's, Coffee Break)} &  $J=2$& 4&8&12 \\ 
     \midrule
     Neighbor States Only \hfill (\texttt{NSO}) & 0.60 & 0.65 & 0.68 & 0.68 \\
     Self States Only \hfill (\texttt{SSO}) & 0.78 & 0.78 & 0.78 & 0.78 \\
     Equal Pooling \hfill (\texttt{EQPOOL}) &0.86 & 0.87& 0.87 & 0.86\\
     Social Pooling \hfill (\texttt{SOCPOOL}) &0.86 & 0.87 & 0.87 & 0.87\\
     \midrule
    
     {\bf \textsc{MGpi}{} Network} & {\bf 0.87} & {\bf 0.88} & {\bf 0.89} & {\bf 0.89}\\
\bottomrule
\end{tabular}
}
\scalebox{.7}{
\begin{tabular}{lcccc}
\toprule
      Model {\bf (mAP's, Cocktail Party)} &  $J=2$& 4&8&12 \\ 
     \midrule
     Neighbor States Only \hfill (\texttt{NSO}) & 0.59 & 0.73 & 0.75 & 0.74 \\
     Self States Only \hfill (\texttt{SSO}) & 0.78 & 0.78 & 0.78 & 0.78 \\
     Equal Pooling \hfill (\texttt{EQPOOL}) &0.89 & 0.91& 0.91 & 0.91\\
     Social Pooling \hfill (\texttt{SOCPOOL}) &{0.87} & {0.90} & {0.90} & 0.90\\
     \midrule
    
     {\bf \textsc{MGpi}{} Network} & {\bf 0.89} & {\bf 0.92} & {\bf 0.93} & {\bf 0.92}\\
\bottomrule
\end{tabular}
}
\label{tab:action_prediction_B2} 
\end{table*}

\begin{figure*}[h]
\centering
\includegraphics[width=0.9\linewidth]{figures/heatmap_best_syn.pdf}
\includegraphics[width=\linewidth]{figures/heatmap_best_dyn.pdf}
\caption{{\bf Confusion matrices on action prediction:} Top ($6\times6$) - Static agents case. Bottom ($7\times7$) - Dynamic agents case. SP: Speaking, LI: Listening, DI: Distracted, SA: Strong Addressing, WA: Weak Addressing, RE: Responding, MV: Moving.}
\label{fig:cmat}
\end{figure*}

\begin{figure*}[h]
\centering
Synthetic:\;\;\;\;\;\;\;\;
\begin{subfigure}[]{0.4\textwidth}
    \centering
    \includegraphics[width=\linewidth]{figures/gate_plot_syntheticstat_lookright.pdf}
\end{subfigure}
\hspace{2mm}
\begin{subfigure}[]{0.4\textwidth}
    \centering
    \includegraphics[width=\linewidth]{figures/gate_plot_syntheticdyn_lookright.pdf}
\end{subfigure}

Coffee Break:\;\;
\begin{subfigure}[]{0.4\textwidth}
    \centering
    \includegraphics[width=\linewidth]{figures/gate_plot_coffeestat_lookright.pdf}
\end{subfigure}
\hspace{2mm}
\begin{subfigure}[]{0.4\textwidth}
    \centering
    \includegraphics[width=\linewidth]{figures/gate_plot_coffeedyn_lookright.pdf}
\end{subfigure}

Cocktail Party:
\begin{subfigure}[]{0.4\textwidth}
    \centering
    \includegraphics[width=\linewidth]{figures/gate_plot_cocktailstat_lookright.pdf}
    \caption{Static Agents Case}
\end{subfigure}
\hspace{2mm}
\begin{subfigure}[]{0.4\textwidth}
    \centering
    \includegraphics[width=\linewidth]{figures/gate_plot_cocktaildyn_lookright.pdf}
    \caption{Dynamic Agents Case}
\end{subfigure}


\caption{{\bf Learned gating output for static agents (left) and dynamic agents (right) cases} for Synthetic, Coffee Break and Cocktail Party initial layouts, respectively. Each are displayed at training epochs 10, 15 and 20.}
\label{fig:kpm_compare}
\end{figure*}


\clearpage





\section{Conclusion}

In this work, we have presented \textsc{MGpi}{}, a deep neural network model of non-verbal and conversational interactions among multiple people and multiple groups. In the absence of real-world annotated data or simulators of multigroup face-to-face conversational behavior ({\it e.g.}, speaking, listening, responding), we have designed our own social interaction simulator. While we consider discrete conversational actions among agents, we believe this is a necessary first step toward designing more complex models of real-world multigroup social intelligence. 

We have demonstrated the necessity and efficacy of each interpretable component of \textsc{MGpi}{}, by evaluating several ablative baselines on their ability to predict the next appropriate action. We have demonstrated \textsc{MGpi}{}'s superior performance in scenarios with static and dynamically evolving group assignments. We reiterate that training \textsc{MGpi}{} requires no explicit group annotations. Instead, its Kinesic-Proxemic-Message gate (KPM gate) learns to express social attention based on non-verbal cues, as a result of training for a socially intelligent policy. 

We have demonstrated the remarkable emergence of KPM gate's ability to identify groups in real-world data, and achieved state-of-the-art results with our direct, unsupervised method, without using complex layout assumptions or hand-defined parameters.

In a dearth of real-world data on multiagent multigroup conversational interactions, valuable future work would involve large-scale collection of non-verbal and verbal exchanges between multiple groups of people. The availability of more diverse data of human-to-human exchange ({\it e.g.}, body orientation, gestures, facial expressions, {\it etc}) would equip us to better design computational models like \textsc{MGpi}{}. Furthermore, such data would equip our models to better reason about social perception and propriety, by allowing the training of deep modules for a wider range of classes of behavioral actions, {\it e.g.}, illustrators, emblems and attitudes \cite{vinciarelli2009social}. 

\vspace{3mm}

{\noindent\bf Acknowledgment:} This project was sponsored in part by JST CREST (JPMJCR14E1), NSF NRI (1637927) and IARPA (D17PC00340).
\pagebreak
\section{Experiments: Social Interaction Simulator}
\label{sec:sim}

\paragraph{\bf Conversational Action Data} 
We aim to learn computational models that can encode the kinesics and proxemics in group formation and conversational actions ({\it e.g.}, responding, distraction, {\it etc}) in multiagent multigroup scenarios. Unfortunately, there are no public datasets of large-scale demonstrations of the physical positions, gaze directions, group assignments, and communication actions of multiple socially interacting people. Existing datasets \cite{Cristani2011,Robicquet2016,Zen2010} only provide sequences of the physical positions, gaze directions and group assignments (but annotated only sparsely). More importantly, \emph{no conversational action annotations are provided}. Similarly, there are no social interaction simulators for the kind of face-to-face behavioral actions of interest to us. In order to evaluate the capacity of our proposed model, we simulate multigroup human communication, drawing inspiration from several prior studies and observations of small-group conversational dynamics \cite{Gatica2006, katzenmaier2004identifying, gatica2005detecting, heylen2006annotating, mast2002dominance}. While we simulate sequences of non-verbal interactions, we point out that the physical layouts of agents are from real-world datasets.

\paragraph{\bf Initial Agent Layout Data:} We experiment with both artificially constructed and real-world data to form our initial multigroup layouts - specifically, we use (a) Synthetic ~\cite{Cristani2011}\footnote{\url{http://profs.sci.univr.it/~cristanm/ssp/}}, (b) Coffee Break ~\cite{Cristani2011}, and (c) Cocktail Party \cite{Zen2010} datasets to provide 100, 119, and 320 non-identical, independent layouts respectively. Multiple people (6-12 per layout) are organized into several groups (Fig. \ref{fig:party}).

\begin{figure}[h]
\centering
    \begin{subfigure}[]{0.22\textwidth}
    \centering
    \includegraphics[width=\columnwidth]{figures/simulate1.pdf}
    \end{subfigure}
    \centering
    \begin{subfigure}[]{0.22\textwidth}
    \centering
    \includegraphics[width=\columnwidth]{figures/simulate2.pdf}
    \end{subfigure}
\caption{{\bf Example physical layout} \cite{Cristani2011} and subsequent conversational interaction. Arrows represent gaze directions, head colors indicate conversational action - red: speaking, yellow: distracted, green: listening. }
\label{fig:synth}
\vspace{-4mm}
\end{figure}

\paragraph{\bf Rules of Interaction:} 
To design multigroup communication protocols, we draw inspiration from several observations of small-group conversational dynamics in literature, namely: speaking and turn-taking,  \cite{Gatica2006}, addressing \cite{katzenmaier2004identifying}, interest in meetings \cite{gatica2005detecting, heylen2006annotating}, and the relation of speaking time and dominance \cite{mast2002dominance}. 
We design per-group rules for two scenarios: {\bf (1) Static scenario}, where agents stay within a social group and do not transition to other groups. Six conversational actions are considered $\mathcal{U}^{stat} = \{$\textit{Speaking}, \textit{Listening}, \textit{Distracted}, \textit{Strongly Addressing}, \textit{Weakly Addressing}, \textit{Responding}$\}$.  {\bf (2) Dynamic scenario}, where agents transition from group to group, leading to a temporally evolving group assignments. Seven conversational actions are considered $\mathcal{U}^{dyn} = \mathcal{U}^{stat} \cup \{$\textit{Moving}$\}$. Based on observations from literature described above, we design per-group probabilistic rules conditioned on a temporal history of past group actions, to decide the evolution of each agent's conversational actions. The same rules are applied to every group in the scene. We simulate 600-step interaction episodes starting from each set of initial layouts ((a),(b) and (c) above) to obtain training data. Figure~\ref{fig:synth} shows an example of initial multigroup layout and subsequent interaction{\footnote{Example simulator videos may be found here: \url{https://github.com/navysanghvi/MGpi}}}.

We reiterate: group assignment information is used \emph{only for evaluation} and \emph{not} for supervisory training signals to \textsc{MGpi}{}. Instead, \textsc{MGpi}{} implicitly identifies groups using the KPM gate.  We hypothesize that our automatic KPM gating mechanism will learn to discover relevant neighbors in both static and dynamic scenarios. While we recognize the gap between reality and our simulations, we believe that, in the absence of existing data or other simulators, this is a necessary first step towards understanding how to model multiagent multigroup interaction.

\paragraph{\bf Two-Agent Simulation Example} Using Figure \ref{fig:tt_example}, we illustrate our rules in a simple two-agent case. In the static scenario, as shown in Figure \ref{fig:tt_example}(a), one agent is \textit{Speaking} while the other is \textit{Listening}. The \textit{Listening} agent may become \textit{Distracted} (Figure \ref{fig:tt_example}(b)) with a certain probability. After some time steps, the \textit{Speaking} agent may start \textit{Strongly Addressing} the group, to draw back the attention of the \textit{Distracted} member (Figure  \ref{fig:tt_example}(c)). Once the \textit{Speaking} agent has spoken for some time, it yields to the other. The other agent is then \textit{Responding} followed by \textit{Speaking} (Figure \ref{fig:tt_example}(d)). The \textit{Speaking} agent might transition to \textit{Weakly Addressing} (Figure \ref{fig:tt_example}(e)). Additionally, in a dynamic scenario, a group member who is \textit{Distracted} may start \textit{Moving} toward another group. When a \textit{Moving} agent joins a new group, it is welcomed as the next speaker.

\paragraph{\bf Baselines}  We now describe experiments and results of applying the \textsc{MGpi}{} network to predict the communication actions of simulated social agents. We compare the \textsc{MGpi}{} network with the following baselines, including communication models used in prior work:

\textbf{{Neighbor State Only (\texttt{NSO}):}} We omit Self-STM Encoder $C'$ so as to only use cues from neighbors. This helps us test whether observations of the self are necessary for modeling social interaction.

\textbf{{Self State Only (\texttt{SSO}):}} We omit the Social Signal Gating Module so as to use cues of the agent's self alone.  This helps us test whether observations of nearby agents' social signals are necessary for modeling social interaction.

\textbf{{Equal Pooling (\texttt{EQPOOL}):}} Prior work studies the role of message broadcasting in scenarios where one agent simply averages messages it receives from other agents \cite{Foerster2016,Mordatch2017,Sukhbaatar2016}. We implement this message broadcasting baseline by omitting the KPM gate $K$ so that all neighbor encoded `messages' $N$ and $C$ are \textit{equally weighted}, such that $\vct{x}_t^{(n_i \leftarrow m)} = \left[N\left(\vct{P}_{t,hist}^{n_i}\right) \;\; C\left(\vct{D}_{t,hist}^{n_i}\right)\right]$.  This helps us test if equal social perception to neighbor agents is sufficient for modeling social interaction.

\textbf{{Social Pooling (\texttt{SOCPOOL}):}} We adopt social pooling \cite{Alahi2016,Lee2017} as a second message broadcasting baseline. It uses a grid that divides the 2D world into non-overlapping regions and pools messages by region. In our experiments, we use a $4\times 4$ grid around target locations, each of size $50\times 50$ pixels (this configuration yielded best performance). For each region, we average $\vct{x}_t^{(n_i \leftarrow m)}$ over neighbors $A^{n_i}$ located in that region. For this baseline, the KPM gate and original Signal Pooling Mechanism of \textsc{MGpi}{} are replaced by an operator that performs grid-wise pooling and concatenates pooled messages. We expect this baseline to learn to implicitly gate messages by down-weighting messages in far away grids.


\paragraph{\bf Training/Evaluation Scheme} Demonstrations of multigroup communication were collected as described previously. For each layout in each dataset (Synthetic, Coffee Break and Cocktail Party), agents changed modes 600 times, {\it i.e.}, $T=600$. We split demonstrations into two subsets with equal number of layouts from each dataset, and learned models on one subset to test the other. All metrics are averaged over the two test subsets for two-fold cross-validation. All models were trained for 30 epochs with mini-batches of size 4096, various numbers of agents $J \in\{2,4,8,12\}$, and a history window $H = 15$, and we confirmed that neither further training nor smaller mini-batches improved performance. Each network was trained to minimize categorical cross entropy (also called negative log likelihood) between predicted probabilities of actions $\{\{\hat{\vct{U}}_{t}\}^T_{t=1}\}_{\mathcal{D}}$ and actual demonstrated actions  $\{\{\vct{U}_{t}\}^T_{t=1}\}_{\mathcal{D}}$ via Adam \cite{Diederik2014}. 
We report the cross-entropy loss on testing subsets. As every model assigns maximum probability to one of six (static scenario) or seven (dynamic scenario) actions, we also evaluate the model on mean average precision (mAP) over actions.


\begin{table*}[htbp!]
\caption{{ Evaluation on group detection}: Precision, Recall and F1 scores under the $|G|$ criterion. \\State-of-the-art methods DS~\cite{Hung2011}, HVFF-ms~\cite{Setti2013}, GCFF~\cite{Setti2015}, and GRUPO~\cite{Vazquez2015} are compared against a simple baseline and the KPM gate (ours). Standard deviation across 5 training runs are shown in brackets. Due to averaging over a lesser number of datasets, italicized results for DS and GRUPO cannot be taken into account for a fair comparison}
\vspace{-2mm}
\centering
\scalebox{1}{
\begin{tabular}{c|c|cccc||c|cc}
\toprule
     \multicolumn{2}{c|}{Method$\Rightarrow$} & \multirow{2}{*}{DS} & \multirow{2}{*}{HVFF-ms} &  \multirow{2}{*}{GCFF} & \multirow{2}{*} {GRUPO} & {Pose-only} & \multicolumn{2}{c}{\bf KPM gate (ours)}\\\cline{1-2}\cline{8-9}
     {Dataset $\Downarrow$} & {Metric $\Downarrow$} & & & & & {Baseline} & {\bf Static Scenario}&{\bf Dynamic Scenario} \\
     \midrule
     \multirow{3}{*}{Synthetic} & Precision & {0.68} & {0.72} & {0.91} & {N/A} & 0.88 & {\bf 1.00} {\footnotesize{(0.000)}}  & {\bf 1.00} {\footnotesize{(0.000)}}\\
     & Recall & 0.80 & 0.73 & 0.91 & N/A & 0.58 & {\bf 0.96} {\footnotesize{(0.008)}}& {\bf 0.98} {\footnotesize{(0.000)}} \\
     & F1 Score & {0.74} & {0.73} & {0.91} & {N/A} & 0.70 & {\bf 0.98} {\footnotesize{(0.004)}}  & {\bf 0.99} {\footnotesize{(0.000)}} \\
     \midrule
     \multirow{3}{*}{Coffee Break} & Precision  & 0.40 & 0.40 & 0.61 & N/A & 0.53 & {\bf 0.63} {\footnotesize{(0.004)}} & {\bf 0.61} {\footnotesize{(0.004)}}\\
     & Recall & 0.38 & 0.38 & {\bf 0.64} & N/A & 0.46 & {0.63} {\footnotesize{(0.010)}} & {0.62} {\footnotesize{(0.005)}} \\
     & F1 Score &  0.39 & 0.39 & {\bf 0.63} & N/A & 0.49 & {\bf 0.63} {\footnotesize{(0.005)}} & {0.62} {\footnotesize{(0.001)}} \\
     \midrule
     \multirow{3}{*}{Cocktail Party} & Precision & N/A & 0.30 & {0.63} & {\bf 0.65} & 0.29 & 0.60 {\footnotesize{(0.004)}} & {0.63} {\footnotesize{(0.010)}}\\
     & Recall & N/A & 0.30 & {\bf 0.65} & 0.63 & 0.27 & 0.56 {\footnotesize{(0.007)}} & 0.55 {\footnotesize{(0.003)}} \\
     & F1 Score & N/A & 0.30 & {\bf 0.64} & {\bf 0.64} & 0.28 & 0.58 {\footnotesize{(0.005)}} & 0.59 {\footnotesize{(0.005)}} \\
     \midrule
     \midrule
     \multirow{3}{*}{ Overall Mean} & { Precision} & {\emph{0.54}} & 0.47 & 0.71 & {\emph{0.65}} & 0.57 & {\bf 0.74} {\footnotesize{(0.002)}} & {\bf 0.75} {\footnotesize{(0.004)}}\\
     & { Recall} & {\emph{0.59}} & 0.47 & {\bf 0.73} & {\emph{0.63}} & 0.44 & 0.72 {\footnotesize{(0.008)}} & 0.72 {\footnotesize{(0.002)}} \\
     & { F1 Score} & {\emph{0.56}} & 0.47 & 0.72 & {\emph{0.64}} & 0.49 & {\bf 0.73} {\footnotesize{(0.004)}} & {\bf 0.73} {\footnotesize{(0.002)}} \\
    \bottomrule
\end{tabular}
}
\label{tab:group}
\vspace{-3mm}
\end{table*}

\paragraph{\bf Results} Tables \ref{tab:action_prediction1} and \ref{tab:action_prediction2} shows all model performances in communication action prediction, with $H=15$, and various numbers of neighbors $J$. We emphasize that $J$ is only fixed during training; since a single Social-STM Encoder is learned, \emph{any} number of neighbors may be considered at every time step during testing - in fact, \emph{different} numbers of neighbors may be considered for each agent in the scene. 

As hypothesized, \textsc{MGpi}{} effectively learns protocols in both static and dynamic scenarios, outperforming strong baselines in terms of both mean average precision (mAP) and test loss, validating our choices of various encoding modules and demonstrating each one's necessity. Confusion matrices of action prediction for all networks
(omitted here for spcae) show that \texttt{NSO} and \texttt{SSO} work complementarily: \texttt{NSO} predicts better if agents make strong or weak addressing decisions by observing others. On the other hand, \texttt{SSO} performs better when observation of the self is necessary, {\it i.e.}, in order to predict if agents keep speaking, listening, are distracted or responding. \textsc{MGpi}{} outperforms baselines for most actions, both in the static and dynamic case, with lower ambiguity. 

\begin{figure}[h]
\vspace{-1mm}
\centering
\includegraphics[width=0.25\linewidth]{figures/grid_lookright.pdf}
\includegraphics[width=0.74\linewidth]{figures/gate_plot_history_lookright.pdf}
\caption{{\bf Social Attention of KPM Gate:} For various positions of a neighbor $A^{n_i}$ ({\it e.g.}, any blue figure) in a fine grid around $A^m$ (magenta figure, black circles), warmer colors indicate higher outputs from the learned KPM gate $K$. Gaze directions $\vct{g}^{m}_t = [-1,0]$,\;\; $\vct{g}^{n_i}_t = [+1,0]$.\;\; History windows $H \in \{5, 10, 15\}$.}
\label{fig:visualization1}
\vspace{-3mm}
\end{figure}

\paragraph{\bf Learned Social Attention} Figure~\ref{fig:visualization1} represents physical constraints learned by the \textsc{MGpi}{} Network's KPM gate $K$ to judge if neighbor agent $A^{n_i}$ is in the same group as agent $A^m$. These visuals are obtained after training in the static scenario. We visualize the output from $K$ at various locations around $A^m$ while fixing relative gaze $g^{(n_i \leftarrow m)}$. As expected, we see improved learning with larger history windows $H$, and choose $H = 15$ in our experiments. We confirm that increasing the history window does not improve performance, suggesting that, once trained on real-world scenarios, \textsc{MGpi}{} might be able to \emph{reflect the optimal length of short-term memory exhibited in humans}. Also as expected, we see a higher output from our KPM gate when agents are looking toward each other, i.e., when the neighbor $A^{n_i}$ is toward the left of the image, looking in $+x$ direction, \emph{i.e.}, toward $A^m$, who is looking in $-x$ direction.


\section{Experiments: Group Identification in Real Data}\label{sec:expt_group}

We have demonstrated the effectiveness of the \textsc{MGpi}{} network in learning group communication policy with simulated social agents. We now test the effectiveness of our learned automatic social signal gating mechanism, the KPM gate, in identifying communication groups in real-world data ~\cite{Cristani2011,Zen2010}. 

\paragraph{\bf Training \textsc{MGpi}{}} Similar to Section \ref{sec:sim}, we train \textsc{MGpi}{} using a dataset of simulated interactions from a mixture of initial layouts from the Synthetic~\cite{Cristani2011}, Coffee Break~\cite{Cristani2011} and Cocktail Party~\cite{Zen2010} datasets. For the sake of equal representation in our training data, we use 100 initial layouts from each dataset, for a total of 300 demonstrations of $T=600$ steps each. Each dataset has several continuous frames with various numbers of people annotated with positions, orientations, and group assignments (Fig. \ref{fig:party}). \textsc{MGpi}{} is trained for 20 epochs, in both static and dynamic scenarios, setting $J = 12$, which is sufficiently larger than group membership (6 at most) in the datasets. This ensures that the network must learn to filter out irrelevant social cues from non-group agents, strengthening the contribution of the KPM gate to good network performance. Further, this enhances the chances that KPM gate's learned social attention is effective for group identification.

\paragraph{\bf Unsupervised Group Identification} As described in Section \ref{sec:imitation}, once \textsc{MGpi}{} is trained, we define a distance $D(n,j)$ between two agents $n,j$ based on the output of learned gating function $K$: $D(n,j) = 1 - \frac{1}{2}(K(\vct{g}^{(j \leftarrow n)}, \vct{l}^{(j \leftarrow n)}) + K(\vct{g}^{(n \leftarrow j)}, \vct{l}^{(n \leftarrow j)}))$. For every initial layout in real-world datasets Synthetic, Coffee Break, and Cocktail Party \cite{Cristani2011,Zen2010}, we use this simply computed distance measure to form an affinity matrix of pairwise distances between agents in the scene. Using this affinity matrix, we run the DBSCAN clustering algorithm~\cite{Ester1996} to cluster people into conversational groups.

\paragraph{\bf Evaluation} Group detection performance is measured by the F1 score (harmonic mean of Precision and Recall) under the $|G|$ condition~\cite{Cristani2011,Setti2013}. Per frame, a group is judged to be detected correctly if {\emph{all}} constituent people are grouped into a single cluster. Precision, Recall and F1 scores are averaged over frames for each dataset (Synthetic, Coffee Break, Cocktail Party) separately, as well as over all datasets.
Due to the stochastic nature of training \textsc{MGpi}{}, we show average and standard deviation of KPM gate's performance over 5 training runs. 

\paragraph{\bf Comparisons} We define a pose-only baseline, which runs DBSCAN using actual euclidean distances between agents in each scene, scaled by the maximum such distance. Unlike \textsc{MGpi}{} and other state-of-the-art methods, this does not use gaze or body orientation information. In addition to this simple baseline, several state-of-the-art group identification methods DS\cite{Hung2011}, HVFF-ms \cite{Setti2013}, GCFF \cite{Setti2015}, GRUPO \cite{Vazquez2015} are chosen for comparison. These methods are based on complex \emph{o-space} estimation under assumptions of particular spatial layouts called F-formations \cite{Kendon1990}, often using heuristic parameters. By contrast, KPM gate offers a direct, unsupervised approach to group identification. 


\paragraph{\bf Results} From Table~\ref{tab:group} we see that our learning process of the KPM gate is very stable and has very low standard deviation across training runs. Our simple method takes a few miliseconds to run and could be used for real-time group identification. On the Synthetic dataset, our approach significantly outperforms all state-of-the-art methods in terms of all metrics. On the Coffee Break dataset, our approach outperforms other methods in terms of Precision. and performs as well as the best (GCFF) in terms of F1 score. On the Cocktail Party dataset, our method performs slightly worse than the best (GCFF and GRUPO), but better than HVFF-ms.  Overall, our method outperforms all methods in terms of Precision and F1 score, and performs comparably to the best (GCFF) in terms of Recall. 

\paragraph{\bf Discussion} Our work improves over prior work in ways that are even more significant than our superior overall performance: \\{\emph {(1) Imitation leads to group detection:}} Most importantly, unlike prior work, our model is not trained explicitly for group detection. Instead, our focus is on behavioral imitation (learning a model that mimics human interaction). The surprising result is that, after training for behavior imitation, the output of KPM gate also performs successfully on the task of group detection. \\{\emph {(2) Data-driven approach, features, parameters:}} All methods we compare with rely on the assumption that people in groups position themselves in specific layouts (F-formations) developed in interactional socio-linguistics. By contrast, our models use no prior knowledge about layouts - we learn spatial layouts from data. Methods like GRUPO \cite{Vazquez2015} use hand-defined parameters, features (e.g, stride, lambda, gaussian mixtures), and expensive iterative mode-finding sub-algorithms. By contrast, our method has a single learned feature (KPM gate) and performs low-cost O($N^2$) unsupervised clustering. GRUPO uses the additional feature of lower body orientation, which our method does not.

In summary, we have demonstrated state-of-the-art performance using a simple, direct method without the use of hand-crafted parameters, features or layout assumptions. The remarkable \emph{emergence} of social perception in the form of group identification has been compared with various \emph{explicit} methods for this task. 



\section{Imitation Learning and Group Identification}
\label{sec:imitation}

By modeling the \textsc{MGpi}{} Network's modules $N$, $C$, $G$, $C'$, and $\pi$ as fully-differentiable functions, we ensure that it is trainable end-to-end via back-propagation, using demonstrations of multiagent multigroup state and action sequences {\footnote{Our code may be found here: \url{https://github.com/navysanghvi/MGpi}}}.

\paragraph{\bf Imitation Learning} We perform imitation learning to learn a policy from several `expert' demonstrations. In this work, we adopt the approach of behavior cloning, which enables \textsc{MGpi}{} to learn from a set of demonstrations $\mathcal{D}$ directly, via supervised learning without needing to explicitly learn state transitions or reward functions. Similar approaches can be found in recent work on multiagent behavior prediction~\cite{Alahi2016,Robicquet2016}. We explain how these demonstrations are obtained in Section \ref{sec:sim}.
Each demonstration $D \in \mathcal{D}$ is of length $T$, and denoted by $D=\left\{(\vct{S}_{t}, \vct{U}_{t})\right\}^{T}_{t=1}$, where $\vct{S}_t$ and $\vct{U}_t$ are the joint state and joint action of the $M$ interacting agents at time $t$. Therefore, $\vct{S}_t = \left\{\vct{S}^m_t\right\}^M_{m=1}$, and $\vct{U}_t = \left\{\vct{d}^m_t\right\}^M_{m=1}$, where $\vct{S}^m_t$ and $\vct{d}^m_t$ are individual agent state and action, as described in the notation in Section \ref{sec:architecture}. Note that $\vct{S}_{t}$ contains no information about the group each agent belongs to.

\paragraph{\bf Group Assignments} We emphasize again that training of the \textsc{MGpi}{} Network requires {no supervision with respect to group assignments}. Moreover, we identify these assignments implicitly using the KPM gate module $K$. For every agent $A^m$, in order to minimize the error between \textsc{MGpi}{}'s predicted action $\hat{\vct{d}}^m_{t}$ and demonstrated action $\vct{d}^m_{t}$, KPM gate $K$ must implicitly learn spatial attention. The higher the `importance' weight assigned by $K$ to a neighbor's encoded non-verbal and conversational history, the more likely the neighbor is to be in the same group as the $A^m$, and the greater its influence on $A^m$'s resulting policy $\pi$. 

\paragraph{\bf Group Identification} Once the \textsc{MGpi}{} network parameters are learned, we can use the KPM gate activations to {identify groups}. In order to identify groups, at each time step, we define a distance $D(n,j)$ between two agents $n,j$ based on the output of learned gating function $K$: $D(n,j) = 1 - \frac{1}{2}(K(\vct{g}^{(j \leftarrow n)}, \vct{l}^{(j \leftarrow n)}) + K(\vct{g}^{(n \leftarrow j)}, \vct{l}^{(n \leftarrow j)}))$. The distance is simply computed from the weighted average of the bi-directional weights computed by the KPM gate. We can compose an affinity matrix of pairwise distances between all agents and run the DBSCAN clustering algorithm~\cite{Ester1996} to cluster people into conversational groups. As mentioned in our Introduction, this is a relatively simple and straight-forward approach. It is compared in Section \ref{sec:expt_group} with state-of-the-art approaches based on complex \emph{o-space} estimation under assumptions of particular spatial layouts called F-formations \cite{Kendon1990}.
\section{Social Interaction Management: The \textsc{MGpi}{} Network \label{sec:architecture}}

Our goal is to design a computational model for multiagent multigroup interactions that incorporates signal gating, adaptation to dynamic group sizes and short-term memory encoding. Towards this goal, we propose the \textsc{MGpi}{} architecture (Figures \ref{fig:overview}, \ref{fig:details}) In our design of \textsc{MGpi}{}, we consider several insights from social science literature: the importance of `situational awareness' and `presence' \cite{albrecht2006social}, and the impact of non-verbal social cues `kinesics' \cite{birdwhistell1952introduction} and `proxemics' \cite{edward1966hall}.
\vspace{-3mm}
\paragraph{\bf{Notation}} We denote the $m$-th agent by $A^m$ and its $J_t$ neighbors at time $t$ by $\{A^{n_i}\}^{J_t}_{i=1}$.
At time $t$, $A^m$ and its neighbor $A^{n_i}$ have gaze directions $\vct{g}_t^m, \vct{g}_t^{n_i} \in \mathbb{R}^2$, and positions $\vct{l}_t^m, \vct{l}_t^{n_i} \in \mathbb{R}^2$.  $R(\phi)$ is the rotation matrix associated with angle $\phi = \arctan{(\vct{g}_t^m)}$. Relative gaze direction and relative rotated position of $A^{n_i}$ w.r.t. $A^m$ are respectively given by:
\begin{align*} 
\vct{g}_t^{(n_i \leftarrow m)} = R(\phi)\vct{g}_t^{n_i} {\text{\;\;\;\;\;\;and\;\;\;\;\;\;}} \vct{l}_t^{(n_i \leftarrow m)} = R(\phi) (\vct{l}_t^{n_i} - \vct{l}_t^m).
\end{align*}
In our model, we use a past history of features. The history length or horizon is denoted as $H$. The histories of relative gaze directions and relative rotated positions of $A^{n_i}$ w.r.t. $A^m$ are respectively given by: 
\begin{align*}
\vct{G}_{t,hist}^{(n_i \leftarrow m)} &= \begin{bmatrix}
\vct{g}_{t-H+1}^{(n_i \leftarrow m)} & . & . & . & \vct{g}_t^{(n_i \leftarrow m)}
\end{bmatrix} \in \mathbb{R}^{2\times H},\\
\vct{L}_{t,hist}^{(n_i \leftarrow m)} &= \begin{bmatrix}
\vct{l}_{t-H+1}^{(n_i \leftarrow m)} & . & . & . & \vct{l}_t^{(n_i \leftarrow m)}
\end{bmatrix} \in \mathbb{R}^{2\times H}.
\end{align*}
The conversational action space of an agent is denoted by $\mathcal{U}$. The conversational action of $A^m$ and $A^{n_i}$ at time $t$ are represented by $|\mathcal{U}|$-dimensional one-hot vectors $\vct{d}_t^{m}$ and $\vct{d}_t^{n_i}$ respectively. The histories of conversational actions of $A^m$ and $A^{n_i}$ are respectively given by:
\begin{align*}
    \vct{D}_{t,hist}^{m} &= [\vct{d}_{t-H+1}^{m}, ..., \vct{d}_t^{m}] \in \{1,0\}^{|\mathcal{U}|\times H}\\
    \vct{D}_{t,hist}^{n_i} &= [\vct{d}_{t-H+1}^{n_i}, ..., \vct{d}_t^{n_i}] \in \{1,0\}^{|\mathcal{U}|\times H}
\end{align*}

The state of agent $A^m$ at time $t$ is given by 
\begin{align*}
    S_t^m = \left[\left\{\vct{G}_{t,hist}^{(n_i \leftarrow m)}\right\}^{J_t}_{i=1},
    \left\{\vct{L}_{t,hist}^{(n_i \leftarrow m)}\right\}^{J_t}_{i=1},
    \left\{\vct{D}_{t,hist}^{n_i}\right\}^{J_t}_{i=1},
    \vct{D}_{t,hist}^{m}\right]
\end{align*} 

\paragraph{\bf Modules} We now present details of our \textsc{MGpi}{} network's components. As shown in Figure \ref{fig:overview}, the \textsc{MGpi}{} network consists of three modules:  (a) \emph{Social Signal Gating Module:} This module aggregates observations from surrounding agents in the environment into an internal encoding; (b) \emph{Self Short-Term Memory Encoder:} This encodes each agent's own  communication action history; (c) \emph{Interaction Policy:} This module decides the agent's next action based on the agent's encoded observations of its neighbors and of itself. Specifically, the policy incorporates the outputs of the Social Signal Gating Module and Self-STM Encoder and generates a probability vector over the next action of the agent.




\subsection{Social Signal Gating Module}

Conceptually, the Social Signal Gating (SSG) Module allows the agent to select only the most relevant social signals coming from other agents. Computationally, the SSG generates an encoding for the part of available information that depends on the neighboring agents (the part that depends on itself is described later).
As shown in blue in Figure \ref{fig:details}, SSG consists of three successive computational units: (1) Social Short-Term Memory Encoder, (2) Kinesic-Proxemic-Message Gate and (3) Signal Pooling.




\paragraph{\bf Social Short-Term Memory Encoder (Social-STM)} 

For agent $A^m$, at any time $t$, the role of Social-STM is to generate an encoding for each neighbor $A^{n_i}$'s social signals. 
As shown in red in Figure \ref{fig:details}, the Social-STM contains two encoders. (1) Non-Verbal Encoder $N$ encodes a history of relative gaze directions and rotated positions $\vct{P}_{t,hist}^{n_i} = \begin{bmatrix}\vct{G}_{t,hist}^{n_i} \\ \vct{L}_{t,hist}^{n_i}\end{bmatrix} \in \mathbb{R}^{4\times H}$\; (2) Conversational Encoder $C$ encodes a history of the neighbor's conversational actions $\vec{D}_{t,hist}^{n_i}$.

Non-Verbal Encoder $N\left(\vct{P}_{t,hist}^{n_i}\right)$ summarizes $A^m$'s observations of neighbor $A^{n_i}$'s non-verbal communication history. Conversational Encoder $C\left(\vct{D}_{t,hist}^{n_i}\right)$ summarizes $A^m$'s observations of $A^{n_i}$'s past conversation.
As shown in Figure \ref{fig:details}, $N$ and $C$ are both linearly-activated gated recurrent units (GRU) \cite{Chung2014}. Each encoder learns the associated temporal dynamics and outputs neighbor `messages' $N \in \mathbb{R}^{64}$ and $C \in \mathbb{R}^{64}$. $J_t$ pairs of messages are output, one pair for each neighbor. Next, these messages are weighted by the KPM gate's importance score, pooled and passed to the interaction policy.


\paragraph{\bf Kinesic-Proxemic-Message (KPM) gate} The role of the KPM  gate is to decide how much attention to give to each neighboring agent's concatenated social signal $\left[ N\left(\vct{P}_{t,hist}^{n_i}\right), C\left(\vct{D}_{t,hist}^{n_i}\right) \right]$. The KPM gate $K\left(\vct{g}_t^{(n_i \leftarrow m)}, \vct{l}_t^{(n_i \leftarrow m)}\right)$ is a function of the relative rotated position and orientation between $A^m$ and neighbor $A^{n_i}$ at time $t$. The KPM gate is a multi-layer perceptron, consisting of two feed-forward layers, respectively activated by an exponential linear unit (ELU) \cite{Clevert2015} and a hard-sigmoid function \cite{Courbariaux2016}. For each neighbor $A^{n_i}$, it outputs a scalar `importance' weight $K \in [0,1]$, reflecting the neighbor's degree of influence on $A^m$'s next conversational action.

\paragraph{\bf Signal Pooling} The role of the signal pooling operator is to aggregate the weighted social signals for all $J_t$ neighbors of $A^m$ at time $t$. As shown in Figure \ref{fig:details}, each neighbor $A^{n_i}$'s concatenated social signal is weighted by the KPM gate's respective importance score to obtain:
\begin{align*}
    \vct{x}_t^{(n_i \leftarrow m)} = K\left(\vct{g}_t^{(n_i \leftarrow m)}, \vct{l}_t^{(n_i \leftarrow m)}\right) \;.\; \left[N\left(\vct{P}_{t,hist}^{n_i}\right) \;\; C\left(\vct{D}_{t,hist}^{n_i}\right)\right]
\end{align*}
Next, similar to \cite{Sukhbaatar2016}, we employ average pooling of weighted messages from neighbors. The pooled message $\vct{x}_t^m = \frac{1}{J_t} \sum\limits^{J_t}_{i=1} \vct{x}_t^{(n_i \leftarrow m)}$ is passed to the interaction policy $\pi$, along with the Self Short-Term Memory Encoder message (described next), to decide $A^m$'s next conversational action.


\subsection{Self Short-Term Memory Encoder (Self-STM)} 

The role of the Self-STM is to encode the agents recollection of her own past actions at time $t$. Similar to the Conversational Encoder, the Self-STM $C'$ encodes a history of the agent $A^m$'s own conversational actions $\vct{D}_{t,hist}^{m}$. As shown in yellow in Figure \ref{fig:details}, $C'(\vct{D}_{t,hist}^m)$ is a linearly-activated GRU, yielding a self `message' $C' \in \mathbb{R}^{64}$. This is concatenated with the pooled neighbor message $\vec{x}^m_t$ and passed to the interaction policy to decide $A^m$'s next conversational action.


\subsection{Interaction Policy} 

The final module of \textsc{MGpi}{} is the interaction policy $\pi$ which, at time $t$, decides the agent's next action based on the agent's observations of its neighbors and itself. Formally, the policy $\pi$ takes as input the concatenated outputs $\left[\vct{x}_t^m,\;\; C'\left(\vct{D}_{t,hist}^m\right)\right]$ from the SSG Module and the Self-STM (shown in green in Figures \ref{fig:overview}, \ref{fig:details}). The interaction policy $\pi\left(\left[\vct{x}_t^m,\;\; C'\left(\vct{D}_{t,hist}^m\right)\right]\right)$ is modeled as two fully-connected layers, respectively activated by an ELU and soft-max function. It yields a $|\mathcal{U}|$-dimensional output, representing a probability distribution over the agent $A^m$'s next conversational action.

\section{Related Work} 

The analysis of intelligent agent grouping, behavior and communication is of broad interest to many disciplines including Human-Robot Interaction \cite{breazeal2003toward, breazeal2005learning, sanghvi2011automatic, castellano2012detecting, fiore2013toward, salam2016fully, yun2018automatic}, Multiagent Systems and Machine Learning \cite{Busoniu2008, Stone2000, Foerster2016, Sukhbaatar2016, Foerster2017, Le2017, Lowe2017, Peng2017, Mordatch2017, wagner2011acting, brambilla2013swarm, sanghvi2017exploiting}, Neurobiology \cite{cromwell2008sensory, freedman1987neurobiological, bronkhorst2000cocktail, shapiro1997personal, hillyard1987sensory, colino2014tactile}, Computer Vision \cite{cristani2010socially, cristani2013human, Zen2010, bazzani2013social, Setti2013, Setti2015, vascon2014game} and Psychology \cite{birdwhistell1952introduction, edward1966hall, mehrabian1967inference,davies1995mental, meltzoff2003imitation, albrecht2006social}. Since this work is focused on developing computational models of multiagent communication and identifying social groups, we elaborate on selected related work.

\vspace{-2mm}
\paragraph{\bf Human-Robot Interaction:} The effects of gaze and proxemics in human-robot interaction and social signal perception has been widely explored \cite{fiore2013toward}. Recognizing these non-verbal cues' social importance, we incorporate the perception of such cues as a central component in our model. Dialogue systems work \cite{singh2000reinforcement, serban2016building} concentrates on appropriately responding to conversational cues, while our work additionally deals with the non-verbal aspects of human communication.  Detection and automatic analysis of a human's engagement in social human-robot scenarios has been comprehensively studied \cite{sanghvi2011automatic, castellano2012detecting, salam2016fully, yun2018automatic} using various non-verbal cues such as body attitudes, facial video signals, and personality traits. Related to this, our work presents a method to automatically infer an interacting agent's attention from non-verbal and conversational cues. Sociable robots have been defined and studied in the context of their degree of interaction with humans and their functionality \cite{breazeal2003toward, dautenhahn2007socially}. Toward enabling socially intelligent robots, we present a model of multiagent multigroup interaction. The importance of simulation theory and imitative interaction in the social understanding of neighbors by robots has been studied \cite{breazeal2005learning}. We use this insight in order to learn a computational model for socially intelligent  robots through imitation of simulated interactions. 


\vspace{-2mm}
\paragraph{\bf Multiagent Systems and Machine Learning:} Emergent collective behaviors have been studied in multiagent organized swarms \cite{brambilla2013swarm}. By contrast, we model individual behavior in a socially intelligent multigroup setting. Adversarial deceptive interactions have been studied between individuals \cite{wagner2011acting} and groups of intelligent robots \cite{sanghvi2017exploiting}. Instead, we assume a cooperative scenario, where interactions imitate multigroup human-to-human social reactions. Modeling multiagent communication is an active area of research in machine learning \cite{Busoniu2008,Stone2000}. Recent work makes use of deep reinforcement learning or imitation learning to discover policies for agent-to-agent communication, {\it e.g.}, \cite{Foerster2016,Sukhbaatar2016,Foerster2017,Le2017,Lowe2017,Peng2017,Mordatch2017}. The motivation for these works is often collaboratively solving a particular task which might involve partial observability. Communication protocols best fit to accomplish that task are learned. By contrast, we learn to imitate social communication protocols of humans in a multigroup scenario. Some works assume the communication of internal states between agents, while others learn to communicate a particular `message' in order to solve the task. Neighbor messages might be aggregated using a pre-defined pooling operation. By contrast, we explicitly encode particular social signals between agents and learn signal gating to appropriately weight incoming signals, showing superior performance to prior pooling strategies.
 


\vspace{-2mm}
\paragraph{\bf Neurobiology and Signal Gating:} The concept of social signal gating introduced in our model is inspired by the neurobiological process of \emph{sensory gating} \cite{cromwell2008sensory,freedman1987neurobiological}. Sensory gating is the ability to filter out unnecessary or irrelevant external stimuli. The cocktail party effect ~\cite{bronkhorst2000cocktail} is one example of auditory sensory gating where a person is able so focus auditory attention to a specific target while ignoring other irrelevant audio input.  Similar sensory gating is observed in other senses as well, to prevent overwhelming the primary cortical areas \cite{shapiro1997personal,hillyard1987sensory, colino2014tactile}. Recognizing the importance of modelling sensory gating in socially intelligent computational systems, we use our KPM gate to mimic this phenomenon when considering the importance of nearby agents' social signals.

\vspace{-2mm}
\paragraph{\bf Social Group Interactions:} For the purpose of evaluating our model, there are no large-scale public datasets of states and conversational actions of people interacting in multiple groups. Existing datasets \cite{Cristani2011,Robicquet2016,Zen2010} only provide sequences of physical positions, gaze directions and group assignments. More importantly, no conversational action annotations ({\it e.g.}, listening, speaking, {\it etc}) are provided. Closely related to our ideas, a social simulator PsychSim \cite{marsella2004psychsim} adopts a theory-of-mind approach to modelling interactions in scenarios with agent attributes such as power and hardship. By contrast, we consider scenarios of face-to-face interactions in social situations, and require agents to have conversational roles such as responding to being addressed. Further, we aim to automatically learn the levels of reasoning required to behave appropriately in a social situation. In the absence of prior work on simulators or real-world datasets with conversational action annotations, we design our own social interaction simulator (Section \ref{sec:sim}). Inspired by several works which study the multi-modal nature of small-group conversational dynamics \cite{Gatica2006, katzenmaier2004identifying, gatica2005detecting, heylen2006annotating, mast2002dominance}, we design simulated interaction rules in order to generate data to train our network.

\vspace{-2mm}
\paragraph{\bf Social Group Identification:} There is much work on identifying the members of a group in a social situation \cite{Hung2011,Setti2013,Setti2015,Vazquez2015}, and more recently \cite{swofford2019dante}. This prior work typically assumes agents' arrangement in spatial layouts called F-formations ~\cite{Kendon1990}, often attempting to find \emph{o-space} centers \cite{Kendon1990} using heuristic strategies and hand-crafted parameters. By contrast, under no such layout assumptions, we learn parameters for a straightforward, automatic group identification mechanism (Section \ref{sec:expt_group}).


\section{Introduction}\label{sec:intro}

In order to develop next-generation robots that can interact socially with humans, there is first a need for expressive computational models that can encode social intelligence \cite{albrecht2006social, goleman2007social, dautenhahn2007socially, vinciarelli2009social}. Broadly, an agent's social intelligence is its ability to understand and respond appropriately to others, {\it i.e.}, its: (1) social perception and (2) social interaction management skill. Social perception is the ability to analyze other agents' social signals including non-verbal behavioral information such as facial or postural expressions \cite{sanghvi2011automatic,mehrabian1967inference}, physical distance ({\em{proxemics}}) \cite{edward1966hall} and body gestures ({\em{kinesics}}) \cite{birdwhistell1952introduction}. Social interaction management is the ability to take appropriate action in response to information retrieved via social perception.

\begin{figure}[t!]
\vspace{2mm}
    \centering
    \includegraphics[width=0.45\textwidth]{figures/party.pdf}
    \includegraphics[height=27mm]{figures/cb_1.pdf}
    \includegraphics[height=27mm]{figures/cp_1.pdf}
    \caption{Top: Multiagent multigroup social intelligence at a party. Bottom: Coffee Break \cite{Cristani2011} and Cocktail Party \cite{Zen2010} dataset images.}
    \label{fig:party}
    \vspace{-5mm}
\end{figure}

Toward the ultimate goal of robotic social intelligence, we propose a computational model of social interaction management based on social perception of behavioral signals in the context of multiagent multigroup conversation. Consider a crowded social party like the ones depicted in Figure \ref{fig:party} to understand some important components of social intelligence. 
\begin{enumerate}
    \item \textbf{Group Formation and Social Signal Gating.} When many people are gathered together, it is natural for people to form smaller groups and engage in conversation. In order to participate effectively in a group conversation, each person needs to be able to pay attention to the people in the group, while suppressing information (both verbal and non-verbal) from other nearby groups. This type of social attention mechanism, which we term social signal gating, is a critical aspect of social perception and is closely tied to group identification (discussed later).
    \item \textbf{Dynamic Group Size and Influence.} At a social party, people dynamically move from group to group to start new conversations and leave old ones. The ability to adapt to {dynamic group size} and a {dynamic degree of influence} from nearby people is an important aspect of social intelligence.
    \item \textbf{Short-Term Memory.} Communication within a group is successful when there is an appropriate balance of conversational actions. At every moment, each group member decides their future conversational actions based on the past course of interaction (\emph{e.g.}, speak, respond or leave). For example, if one person has been talking very long, group members might become disinterested and disengage from the group. In other words, a social agent's {short-term memory} of past non-verbal and conversational interactions with neighbors is influential in determining its future social interactions. 
\end{enumerate}

\begin{figure}[h]
\vspace{-4mm}
  \begin{center}
    \includegraphics[width=0.4\textwidth]{figures/MGpi_overview.pdf}
  \end{center}
  \caption{Overview of \textsc{MGpi}{} Network}
  \label{fig:overview}
  \vspace{-3mm}
\end{figure}


The example above highlights critical features of a multiagent multigroup interaction model. Keeping in mind these features, we propose {\bf \textsc{MGpi}{}}, a \textbf{M}ultiagent \textbf{G}roup \textbf{P}erception and \textbf{I}nteraction Network. As shown in Figure \ref{fig:overview}, \textsc{MGpi}{} is a deep neural network consisting of the following modules: 
\begin{enumerate}
\item {\bf Social Signal Gating Module} expresses appropriate social perception by processing incoming and past social signals from neighboring agents (both non-verbal and conversational). It decides their degree of influence using a social attention function, the Kinesic-Proxemic-Message (KPM) gate. To enable reasoning with a dynamic number of influencing neighbors, it uses a pooling operation to aggregate encoded social signals. 
\item {\bf Short-term Memory (STM) Encoders} encode the agent's short-term memory of past interactions with each neighbor (Social-STM) and its own actions (Self-STM). 
\item{\bf Interaction Policy Module} processes outputs from aforementioned components to decide the agent's next conversational action.
\end{enumerate}


The \textsc{MGpi}{} network enables us to model social agents that make action choices in a decentralized manner. The parameters of \textsc{MGpi}{} are learned end-to-end via imitation learning, using a set of demonstrated group interaction sequences. In this work, demonstrations are generated by a social interaction simulator, but \textsc{MGpi}{} can also be learned from annotated real-world social interactions.  
Since modeling the full extent of social intelligence is very complex ({\it e.g.}, power, dialogue, trust), in this work, we use abstracted expressions of social iteraction ({\it e.g.}, discrete conversational actions). We believe this simplification is a necessary step towards designing more complex models of real-world social interactions.

As mentioned earlier, social perception is closely tied to group identification. Thus, we further hypothesize that, if our proposed social signal gating module is modeled correctly, we will be able to map the learned social attention directly to group membership. We demonstrate the ability of the KPM gate to identify groups in a direct, unsupervised manner, achieving state-of-the-art performance against group detection methods that use hand-defined parameters, features and complex spatial assumptions on how people arrange themselves in social situations (F-formations \cite{Kendon1990}). 

\noindent In summary, our contributions are as follows:
\begin{enumerate}
\item We present \textsc{MGpi}{}, a {\bf computational model} of multiagent multigroup social interactions (Section \ref{sec:architecture}).
\item In a dearth of prior work on multiagent multigroup social interaction simulators and in the absence of real-world datasets with multigroup behavioral annotations ({\it e.g.}, listen, respond, strongly address), we design our own {\bf social interaction simulator}, seeking inspiration from several works which study the multi-modal nature of small-group conversational dynamics \cite{Gatica2006, katzenmaier2004identifying, gatica2005detecting, heylen2006annotating, mast2002dominance}. We use this simulated data to train and evaluate our network (Section \ref{sec:sim}).
\item We demonstrate how the ability of {\bf group identification} \emph{emerges} as a result of learning a social interaction policy, achieving competitive performance against state-of-the-art methods with the \emph{explicit} aim of detecting groups. Unlike prior work, we do so without the use of explicit group annotations, layout assumptions or hand-defined parameters (Sections \ref{sec:imitation}, \ref{sec:expt_group}). 
\end{enumerate}



