\section{Background \& Related Work}

Learning a language requires interacting with both the environment and other agents~\cite{bisk2020experience}. Language games represent one common example of this~\cite{wittgenstein1953philosophische}, as seen by the important role of play in L1 child language acquisition~\cite{hainey2016systematic} as well as L2 learners~\cite{godwin2014games}. 

Among the language games defined in the literature~\cite{steels2015talking}, \textit{guessing games} represent the first step in a curriculum for language learning. For example, in GuessWhat?!~\cite{de2017guesswhat}, two agents interact with each other: a \textit{Questioner} generates questions aimed at finding a hidden object in the scene and an \textit{Oracle}, aware of the target object, answers the questions supporting the Questioner in playing the game. Different from other language games~\cite{das2017visual}, guessing games have a specific goal which represents a clear incentive for learning. In addition, they require that the \textit{Questioner} masters both natural language generation and understanding with a focus on object categories and attributes. 
For humans, concepts learned in this way are generic and generalisable to new tasks and domains where grounded reasoning is important~\cite{hampton1979polymorphous}. However, \textit{how well can AI agents generalise with concepts acquired from visual guessing games?}

The literature has not explored if representations built from self-play are transferable, focusing instead on large scale self-supervised learning. 
For instance, large scale image captioning datasets have been used to train multi-modal Transformers
~\cite{lu2019vilbert,li2019visualbert,tan2019lxmert,chen2019uniter}. 
Multi-task learning~\cite{lu202012} has been used to leverage the diversity of training signals provided combining datasets, but only for discriminative tasks. 
While some dialogue work \cite{cogswell2020dialog} aims to bootstrap a conversing agent from VQA datasets, most work on GuessWhat?!~\cite{de2017guesswhat,shekhar2019beyond,strub2017end} has designed bespoke models for the task, ignoring the utility of this dataset for other Vision+Language tasks.

We propose self-play as a mechanism for learning general grounded representations. We seed our approach with the \textit{GuessWhat?!} corpus of questions and objects, and demonstrate how to generalise to other downstream tasks. We propose two different strategies to exploit these data. First, a supervised learning phase is undertaken to learn a Questioner and Oracle model able to play guessing games. Second, the trained agents can be used to play guessing games on images requiring only object annotations as supervision.
We show that an agent trained on GuessWhat?! dialogues can use self-play to adapt to new and harder tasks. Specifically, we investigate models' generalisation performance and quality of the learned representations on the CompGuessWhat?! benchmark~\cite{suglia-etal-2020-compguesswhat}, a more extensive evaluation suite for GuessWhat?!. Furthermore, we study how the learned representation help solve VQA on the dataset TDIUC~\cite{kafle2017tdiuc}. We show overall comparable performance with state-of-the-art models and improvements for specific question types that require object attribute information to be answered correctly. 

\section{Methodology}\label{sec:methodology}
Our proposed transfer/fine-tuning procedure requires a training set of guessing games $D_g$ from which we learn a Questioner $Q$ and an Oracle $O$ via supervised learning. Given a set of images $\mathcal{I}$, it is possible to use the trained models $Q$ and $O$ to run the self-play procedure for $n$ epochs obtaining the model $Q^n$. Finally, given a downstream task $t$ and an associated dataset $\mathcal{D}_t$ based on images from $\mathcal{I}$, we use $Q^n$'s parameters as initialisation for the training procedure on $\mathcal{D}_t$.  

To apply this procedure, both the \emph{Questioner} and the \emph{Oracle} require a \textit{multi-modal encoder} $\Gamma$ able to generate $d$-dimensional representations for the textual tokens $\mathbf{h}_t$, for the objects $\mathbf{h}_o$, as well as fusing the visual and textual modalities in a representation of the current context $\mathbf{h}_c$. After the self-play procedure, only the encoder $\Gamma$ of the model $Q^n$ is used in the fine-tuning process on the downstream task $t$ using the dataset $\mathcal{D}_t$. It is important to underline that the presented self-play procedure does not depend on a specific implementation of the multi-modal encoder $\Gamma$. A possible implementation is presented in Section~\ref{sec:vlp_implementation} and it is used in the experimental evaluation of this paper. 
\begin{figure*}[h]
\centering
\includegraphics[width=0.95\textwidth]{img/multitask_vlp.pdf}
\caption{We use the single-stream VLP model as a backbone multi-modal encoder for our task. The visual features tokens (marked in red) are the FastRCNN features associated with the objects in the image, the history tokens (marked in blue) and the tokens to be generated (marked in yellow) are given in input to the model. A \textit{Guesser head} uses the learned contextual object representations to generate a probability distribution over the objects $P(o_i | \mathbf{h}_{o_i})$, whereas the \textit{Generator head} is used to incrementally predict the masked tokens.}
\label{fig:multitask_vlp}
\end{figure*}
\subsection{Oracle design}

The Oracle task is cast as a Visual Question Answering (VQA) task conditioned on the image $I$, the current question $q$ and on the target object $\hat{o}$. We follow common practice in vocabulary-based VQA~\cite{antol2015vqa} and we treat the problem as a multi-class classification task over the classes $\{Yes, No, N/A\}$. We use $\mathbf{h}_c$ as input to a multi-layer feedforward neural network to obtain a probability distribution over the label set.

\subsection{Questioner design}

The Questioner must play two roles: question generation and target object prediction~\cite{de2017guesswhat}. It is beneficial to jointly learn the two tasks because the representations learned by each task are complementary. In addition, they better encode attributes, which favours better generalisation to unseen object categories~\cite{suglia-etal-2020-compguesswhat}. 

To solve the two specific tasks in a multi-task fashion, we design two different heads on top of the shared encoder $\Gamma$: 
1) the \textit{guesser head}, produces a probability distribution over every object $o_i$ using the encoded representations $\mathbf{h}_{o_i}$ passed through an MLP; 
2) the \textit{generator head}, a multi-modal decoder, also implemented as an MLP,
which predicts a probability distribution over the vocabulary $V$ given the context representation generated by $\Gamma$.

We include two losses in our model: 1) the negative log-likelihood of the probability associated by the guesser head with the target object $\hat{o}$~\cite{shekhar2019beyond}; 2) a sequence-to-sequence cross-entropy loss~\cite{sutskever2014sequence} for the generated question tokens.
Unlike previous work that trains a separate module to learn to stop~\cite{shekhar2018ask}, we add a special token \texttt{[STOP]} to the input data so that it learns when to stop more efficiently as part of the question generation task.

Training an agent to solve tasks of different complexity and size is challenging. 
The procedure presented in \cite{shekhar2019beyond} alternates between tasks, updating the hardest task more often. For this technique, finding the right schedule is cumbersome and requires fine-tuning. We rely on a more systematic training procedure based on random dataset-proportional batch sampling inspired by \cite{sanh2019hierarchical}. This represents a hard-parameter sharing multi-task training procedure that avoids interference between tasks and favours a more stable training, which  mitigates catastrophic forgetting~\cite{french1999catastrophic}. 

\subsection{Self-Play via Iterated Experience Learning (SPIEL)}

Inspired by iterated learning ~\cite{kirby2014iterated}, we design a process by which the Questioner learns from games previously generated by other instances of the Questioner agent. We call our training procedure \textit{Self-play via Iterated Experience Learning} (SPIEL).

\begin{algorithm}
\footnotesize
\caption{SPIEL: Self-Play via Iterated Experience Learning}\label{alg:self_play}
\begin{algorithmic}[1]
\Procedure{self\_play}{$Q_0, O, \mathcal{I}, n$}
    \State $\mathcal{D}_q\gets \Call{read\_gold\_games}{\null}$
    \State $\mathcal{E}_g\gets []$ \Comment{Initialise the experience buffer}
    \For{$e\gets 1, n$}
        \LineComment{Interactive phase}
        \State $Q\gets Q^e$ \Comment{load latest weights}
        \State $\mathcal{G}^e\gets \Call{generate\_games}{\mathcal{I}}$
        \State $\mathcal{G}^e\gets \Call{play\_games}{Q, O, \mathcal{G}^e}$
        \State \Call{append}{$\mathcal{E}_g, \mathcal{G}^e$}
        \State $\mathcal{D}^e_g\gets []$ 
                
        \LineComment{Transmission phase}
        \For{$i\gets0, len(\mathcal{E}_g)$}
            \State $g\gets \mathcal{E}_g[i]$ \Comment{Priority to the latest games}
            \If{\Call{is\_valid\_game}{$g$}}
                \State \Call{append}{$\mathcal{D}^e_g$, $g$}
            \EndIf
            \If{\Call{Len}{$\mathcal{D}_g^e$} == \Call{Len}{$\mathcal{D}_q$}}
                \textbf{break}
            \EndIf
        \EndFor
                        
        \LineComment{Learning phase}
        \State $Q^{e+1}\gets \Call{train}{Q, \mathcal{D}_q, \mathcal{D}^e_g}$
    \EndFor
\EndProcedure
\end{algorithmic}
\end{algorithm}

In SPIEL, described in Algorithm~\ref{alg:self_play}, we assume access to a set of images $\mathcal{I}$ and the bounding boxes $\mathcal{O}_I$ of the objects therein.\footnote{Object annotations intended as either gold bounding boxes or predicted bounding boxes from an object detector.} In every gameplay, there is a Questioner $Q$ and an Oracle $O$, initialised with agents $Q^0$ and $O$, respectively, that were trained with Supervised Learning using gold successful dialogues.\footnote{The Oracle is fixed during this learning procedure.} We consider every iteration $e$ of the algorithm as a \textit{self-play} epoch. In a single self-play epoch, we alternate 3 phases: 

\paragraph{Interactive} phase: the agents play guessing games with novel combinations of image and target object. The generated dialogue can be successful if the predicted target object is equal to the target object. Every played dialogue is stored in an experience buffer $\mathcal{E}_g$.

\paragraph{Transmission} phase: in this phase the datasets for the multi-task learning procedure for the Questioner are created. The generator head dataset $\mathcal{D}_q$ is fixed in advance while the dataset for the guesser head $\mathcal{D}_g^e$ is created from the experience buffer $\mathcal{E}_g$ by selecting the \emph{unique} and \emph{valid} dialogues.

\paragraph{Learning} phase: the same multi-task learning procedure used in the supervised learning phase is used to fine-tune the Questioner parameters using the datasets $\mathcal{D}_g^e$ and $\mathcal{D}_q$ collected for the current epoch $e$. This procedure is repeated $n$ times or until a halting condition is reached (e.g. early stopping based on validation metric). 

See Appendix~\ref{appendix:self_play} for implementation details. At the end of the SPIEL procedure, we obtain the model $Q^n$ whose parameters can be reused in other tasks. Particularly, we use the parameters of $Q^n$'s shared encoder $\Gamma$ as initialisation for the fine-tuning on the downstream task $t$ using dataset $\mathcal{D}_t$. 

\subsection{Implementation}\label{sec:vlp_implementation}

We implement a shared multi-modal encoder $\Gamma$ using VLP~\cite{zhou2020vlp}, a single-stream multi-modal Transformer for captioning depicted in Figure~\ref{fig:multitask_vlp}.
During the GuessWhat?! fine-tuning, we extend VLP by including dialogue context in the input together with the features associated with the objects in the image. We learn two new segment ids to represent the question/answer exchanges in the dialogue, as described in \cite{wolf2019transfertransfo}. 
The question is generated by incrementally replacing \texttt{[MASK]} tokens until the end of sequence is generated. 
See Appendix~\ref{appendix:vlp} for more details.
SPIEL training is run on a set of images $\mathcal{I}$ from GuessWhat?! and TDIUC dataset with corresponding object annotations. We make sure that GuessWhat?! test images are not contained in $\mathcal{I}$. This is not an issue for TDIUC test images because the downstream task annotations (QA pairs) are not used by the model during this phase. Once the model has been trained with SPIEL, we use the parameters of the shared encoder $\Gamma$ as a backbone for a VQA model that is fine-tuned on the TDIUC dataset. 

\section{Experimental Evaluation}
To assess the generality of our learned representations, we include two evaluation paradigms:
1) \emph{in-domain evaluation} and 2) \emph{transfer evaluation}. We evaluate several variants of our model: 1) \texttt{VLP-SL}: VLP-based model trained on GuessWhat?! data using multi-task learning; 2) \texttt{SPIEL-gs}: \texttt{VLP-SL} model fine-tuned with our SPIEL procedure where the generator head uses only gold successful games (gs); 3) \texttt{SPIEL-gm}: same as 2) but both successful and failed gold games are used by the generator head. In both SPIEL variants, the guesser head is trained using failed and successful generated games because it is important for the guesser head to be exposed to both types of signal to learn a more robust policy. We decided to investigate the two variants \texttt{SPIEL-gs} and \texttt{SPIEL-gm} to get more insights about the effect that successful and failed games have on the generator head ability to produce effective dialogues.

\begin{figure*}[h]
\centering
\includegraphics[width=0.91\textwidth]{img/multi_prediction.pdf}
\caption{We show the ability of the model to play guessing games with the \emph{bowl} as target object (highlighted in \emph{red}). Given the generated dialogue, we use the probing classifier trained for CompGuessWhat?! to predict the \emph{bowl}'s attributes. Predictions on TDIUC questions associated with the current image are reported as well.}
\label{fig:multi_prediction}
\end{figure*}

\subsection{In-domain evaluation}

We use the CompGuessWhat?! evaluation suite~\cite{suglia-etal-2020-compguesswhat} to assess the ability of the Questioner to play guessing games and learn visually grounded representations in the process.
It complements an evaluation based only on gameplay accuracy~\cite{de2017guesswhat} with 2 auxiliary tasks: target object 1) attribute-prediction expressed in terms of abstract attributes (A), situated-attributes (SO), abstract+situated attributes (AS), and location attributes (L); 2) zero-shot gameplay with near-domain accuracy (ND) and out-of-domain accuracy (OD).  Table~\ref{tab:cgw_results} shows the comparison with previous state-of-the-art models on this benchmark such as \citet{de2017guesswhat} (\texttt{DV-*}) and \citet{shekhar2019beyond} (\texttt{GDSE-*}). \texttt{VLP-SL} has a greater advantage in terms of representation power compared to previous models. This is reflected in all the tasks of the CompGuessWhat?! evaluation.
Particularly, we see better performance even for the zero-shot gameplay (ND: $+5.6$, OD: $+15.2$). This is because VLP associates a vector of probabilities that represents a distribution over the VisualGenome object classes with every object. This helps VLP to cope with the issue of unseen objects and helps the model to generalise. Learning to play is key to gameplay performance, leading to an increase of $+4.4$ over \texttt{VLP-SL} and $+7.9$ over \texttt{GDSE-CL}. In this setup, the difference between the versions \texttt{SPIEL-gs} and \texttt{SPIEL-gm} is very minimal ($0.1$). However, when analysed in more detail, we can see that training the questioner with gold successful data only improves attribute prediction while using mixed data improves overall generalisation in the zero-shot evaluation. 

\begin{table}[]
\begin{footnotesize}
\centering
\begin{tabular}{@{}l@{\hspace{4pt}}c
        @{\hspace{8pt}}c@{\hspace{4pt}}c@{\hspace{4pt}}c@{\hspace{4pt}}c
        @{\hspace{5pt}}l@{\hspace{5pt}}
        c@{\hspace{4pt}}c@{\hspace{4pt}}c@{}}
\toprule
     &  & \multicolumn{4}{@{}c@{\hspace{5pt}}}{Attribute Pred.} &  
     &     \multicolumn{2}{@{}c@{\hspace{3pt}}}{ZShot} & Score \\
     \cmidrule{3-6}
     \cmidrule{8-9}
Models              & Acc.  & A     & SO            & AS           & L             & & ND     & OD    &      \\ \midrule
\texttt{Random}              & 15.8  & 15.1  &\phantom{0}0.1 &\phantom{0}7.8&\phantom{0}2.8 & & 16.8   & 18.6  & 13.3 \\
\texttt{DV-SL}          & 41.5  & 46.8  & 39.1          & 48.5         & 42.7          & & 31.3   & 28.4  & 38.5 \\
\texttt{DV-RL}          & 53.5  & 45.2  & 38.9          & 47.2         & 43.5          & & 43.9   & 38.7  & 46.2 \\
\texttt{GDSE-SL}             & 49.1  & 59.9  & 47.6          & 60.1         & 48.3          & & 29.8   & 22.3  & 43.0 \\
\texttt{GDSE-CL}             & 59.8  & 59.5  & 47.6          & 59.8         & 48.1          & & 43.4   & 29.8  & 50.1 \\
\midrule
\texttt{VLP-SL}              & 59.5  & 59.2  & 48.2          & 59.7         & 49.3          & & 49.0   & 45.0  & 53.5 \\
\texttt{SPIEL-gs}           & 64.1  & \textbf{61.3}  & \textbf{49.6}          & \textbf{61.6}         & \textbf{51.1}          & & 54.9   & 51.9  & 57.8 \\
\texttt{SPIEL-gm}           & \textbf{64.6}  & 60.8  & 48.3          & 59.5         & 51.0          & & \textbf{55.3}   & \textbf{52.9}  & \textbf{57.9} \\ 
\bottomrule
\end{tabular}
\end{footnotesize}
\caption{F1 scores for attribute prediction and accuracies for zero-shot evaluation on CompGuessWhat?!.}
\label{tab:cgw_results}
\end{table}

\subsection{Transfer evaluation}

For the transfer evaluation, we use the VQA dataset TDIUC~\cite{kafle2017tdiuc}.
It provides a finer-grained way to assess the quality of the representations learned by our guessing game transfer technique
in terms of several question types including object categories and their attributes.
Specifically, we were interested in improving on the following question types: 1) Positional reasoning; 2) Counting; 3) Object presence; 4) Utility/Affordances; 5) Attribute; 6) Color; and 7) Object recognition. TDIUC is evaluated using the arithmetic mean accuracy per question type (A-MPT), as well as the harmonic mean (H-MPT) that better captures the skewed question-type distribution.
In Table~\ref{tab:tdiuc_results}, we report a comparison between variants trained on guessing games data (\texttt{VLP+SL} and \texttt{SPIEL-*}), the original model VLP trained on Conceptual Captions (\texttt{VLP+CC}) and other state-of-the-art models \textit{specifically} designed for the VQA task such as \texttt{MUREL}~\cite{cadene2019murel}, \texttt{RAU}~\cite{noh2016rau}, \texttt{NMN}~\cite{andreas2016neural}, \texttt{MCB-*}~\cite{fukui2016multimodal}. The full set of results is available in the Appendix, Table~\ref{tab:full_tdiuc_results}.

Among 
them,
\texttt{MUREL} achieves the best scores across the board, due to a custom iterative reasoning mechanism and a non-linear fusion module.
However, all our models have a more balanced overall performance which results in better harmonic means (H-MPT, $+5$ points over \texttt{MUREL}). 
Specifically, this improvement is favoured by an increase in accuracy on the \emph{Utility/Affordances} question type ($+20.7$). As shown by the attribute prediction in the CompGuessWhat?! and depicted in Figure~\ref{fig:multi_prediction} (c), our models learn better representations than competitors specifically for abstract attributes among which there are object affordances. Particularly, we can see how it is able to understand that certain objects can \emph{contain} things (e.g. ``the one with the soup in it?"), that objects have specific functions (e.g. ``are the contents of the plate edible?") or that they have specific properties (e.g. ``a spoon is made of wood").

The effectiveness of the proposed fine-tuning procedure is confirmed by the improved performance across all the question types compared to our baseline \texttt{VLP+CC}. Models such as \texttt{MUREL} and \texttt{MCB-*} equipped with specific VQA modules have an advantage on specific question (e.g., positional reasoning) compared to VLP that relies only on BERT self-attention layers~\cite{devlin2019bert}. In addition, when comparing the two SPIEL variants, a similar trend showed in the in-domain evaluation can be observed. Particularly, \texttt{SPIEL-gm} benefits from being exposed to more language data coming from successful and failed guessing games. 
 
\newcommand{\SH}[1]{\begin{scriptsize}#1\end{scriptsize}}
\begin{table}[!t]
\centering
\begin{footnotesize}
\begin{tabular}{@{}l@{\hspace{2pt}}c@{\hspace{3pt}}c@{\hspace{3pt}}c@{\hspace{3pt}}c@{\hspace{3pt}}c@{\hspace{3pt}}c@{\hspace{3pt}}c@{\hspace{3pt}}|@{\hspace{3pt}}c@{\hspace{3pt}}c@{\hspace{3pt}}c@{}}
\textsc{Model}  & {\rotatebox{65}{\textsc{\SH{Position}}}} & {\rotatebox{65}{\textsc{\SH{Count}}}} & {\rotatebox{65}{\textsc{\SH{Presence}}}} & {\rotatebox{65}{\textsc{\SH{Afford.}}}} & {\rotatebox{65}{\SH{\textsc{Attr.}}}} & {\rotatebox{65}{\textsc{\SH{Color}}}} & {\rotatebox{65}{\textsc{\SH{Recog.}}}} &  \rotatebox{65}{\textsc{\SH{A-MPT}}}               & \rotatebox{65}{\textsc{\SH{H-MPT}}}               \\ 
\toprule
\texttt{RAU}        & 35.3         & 48.4         & 94.4         & 31.6         & 56.5         & 66.9         & 86.1        & 67.8         & 59.0        \\
\texttt{NMN}        & 27.9         & 49.2         & 92.5         & 25.2         & 47.7         & 54.9         & 82.0        &  62.6         & 51.9        \\
\texttt{MCB-A}      & 55.4         & 51.0         & 93.6         &\textbf{35.1} & 56.7         & 68.5         & 81.9        & 67.9         & \textbf{60.5}        \\
\texttt{MCB}        & 33.3         & 50.3         & 91.8         & 33.9         & 53.2         & 56.9         & 84.6        & 65.8         & 58.0        \\ 
\texttt{MUREL}      & \textbf{41.2}& \textbf{61.8}& \textbf{95.8}& 21.4         & \textbf{58.2}& \textbf{74.4}&\textbf{89.4}& \textbf{71.2}& 59.3        \\
\midrul
\texttt{VLP}      & & & & & & & & & &\\
\hspace{5pt}+\texttt{CC}     & 36.9         & 55.3         & 94.7         & 31.0         & 55.4         & 67.3         & 85.8        & 68.8         & 60.1        \\
\hspace{5pt}+\texttt{SL}  & 39.0         &\textbf{57.6} &\textbf{94.8} & \textbf{42.1}& 54.3         & 69.0         & 86.1        & 70.5         & 64.0        \\
\texttt{SPIEL-gs}  &\textbf{40.9} & 57.5         &\textbf{94.8} & 36.3         & 56.9         & 69.2         &\textbf{86.3}& 70.4         & 63.3        \\
\texttt{SPIEL-gm}  & 40.6         & 57.0         &\textbf{94.8} & 39.2         &\textbf{57.0} &\textbf{69.4} & 86.2        & \textbf{70.9}         &\textbf{64.3}\\ 
\bottomrule
\end{tabular}
\end{footnotesize}
\caption{Results for the transfer evaluation on TDIUC. The models are divided in two categories: (top) Models specifically designed for VQA and (bottom) our VLP-based implementations. We report only the question types that we believe will benefit from the guessing games fine-tuning procedure. For the full set of results please refer to Appendix, Table~\ref{tab:full_tdiuc_results}.
}
\label{tab:tdiuc_results}
\end{table}

\section{Conclusions}

In this work, we verified that representations learned while playing guessing games can be transferred to other downstream tasks such as VQA. We presented two ways of learning from guessing games data namely multi-task learning and SPIEL. Models using SPIEL performed better both on in-domain evaluation on CompGuessWhat?! as well as on the transfer task TDIUC. Our self-play procedure was able to learn useful and finer-grained object representations such as object affordances, thus demonstrating that \textit{learning to guess helps learning to ground}.

The current study showed how we can apply the SPIEL training procedure to a VQA dataset such as TDIUC. We believe that this work can be extended to other datasets because the SPIEL procedure only requires a set of images and associated object bounding boxes. These could be either gold or generated by a trained object detector therefore classifying guessing games as a \textit{holistic self-training procedure} for multi-modal datasets.

