\section{Introduction}\label{introduction}

Choosing the right loss function is crucial for getting the expected behavior from deep learning based 
models trained for any task. While the 
token-level cross-entropy (CE) loss continues to excel in training natural 
language generation (NLG) models for various tasks, including dialog-response 
generation \citep{roller2020recipes, zhang2019dialogpt}, it is well accepted among 
NLP researchers that CE is not the most appropriate loss function to use for
training dialog generation models. Thus, finding the right loss function for training 
dialog generation is still an open problem and an active area of research \citep{shen2017conditional,zhao2017learning,saleh2020hierarchical,li2021improving}.

Both unconditional or conditional NLG models,
including dialog generation models, are almost universally trained using
the token-level cross-entropy (CE) loss. The CE loss for language generation models is computed by
comparing the predicted token probabilities to the ground truth target sequence
from the dataset. This computation process of CE is unconditional or
context-free as it does not depend on the available prompt/context in the
case of conditional NLG tasks like dialog generation. Without the context it is not possible to grade the generated response for qualities that human users truly care about, like relevance, coherence, etc. Another major concern with training dialog generation models using the CE loss is the assumption of a fixed target. While training any NLG model using the CE loss,
  the probability of the ground truth response is maximized. Here, we
  make an implicit assumption that the ground truth is the only response
  possible for the given context. But this property does not hold for most dialogs where each context may have a large number of possible responses \citep{dou2021multitalk}\footnote{These problems are
specific to dialog systems only and may or may not apply to other NLG
tasks.}.

Previous attempts in training Seq2Seq dialog generation models using the
CE loss have led to various complications. Mode collapse is one of the
most common issues when training a Seq2Seq model with the CE loss,
mainly at smaller scales \citep{li2016deep,li2021improving}. Here, the model will just assign a
high probability to one or more generic and bland responses e.g. ``I
don't know", ``I have a problem", ``Yes" etc., irrespective of the context.
Previous research
works have also explored various augmentations to the model architecture \citep{serban2016building,zhao2019rethinking}
and/or the loss function \citep{serban2017hierarchical,shen2017conditional,zhao2017learning,li2021improving} to resolve the common problems with CE, as
mentioned above.

In order to facilitate a training process that aims to maximize some user-expected quality in dialog generation models, we propose to directly optimize an estimate of human perceived quality of a context-response pair. \citet{Sinha2020LearningAU,yeh2021comprehensive} have
shown that the output score of retrieval models correlates strongly with human
perception of dialog response quality (given that it was trained on a
dataset of domain similar to the target domain of application). 
Further,~\citet{santra2021representation} showed that representations learned by a
response retrieval (using binary cross-entropy or a contrastive loss) capture
important dialog understanding features even better than large-scale dialog generation models trained using CE. 

These observations triggered us to propose a novel \textbf{Contextual Response Retrievability} loss function or \textbf{CORAL} that circumvents these
issues with CE loss and relies on a
response retrieval model.  
Further, we leverage CORAL to design a learning framework for dialog generation models that treats score calculated by retrieval models as a reward function, \textbf{Response Retrievability Reward} ($R_3$). We then use reinforcement learning (RL) to train dialog models that maximizes above reward between context and generations from the optimized network. 
To test the proposed loss function, we train transformer-based Seq2Seq models \citep{vaswani2017attention}
using CORAL for open-domain dialog generation using DailyDialog dataset \citep{li2017dailydialog} and for domain-specific dialog generation using DSTC7-Ubuntu dataset \citep{yoshino2019dialog}. We compared the performance against state-of-the-art pretrained CE-based Seq2Seq models and various other baselines using both automatic metrics and through manual analysis. To summarize, our contributions are as follows.
\begin{enumerate}
  \item We propose CORAL loss function for training dialog generation models by directly optimizing for an estimate of human preference of a $\langle$context,response$\rangle$ pair. To the best of our knowledge, we are the first to propose a loss function for dialog generation model that relies upon the context also.
    \item Further, we propose a recipe to train improved seq2seq dialog models which uses the CORAL loss in a reinforcement learning setup.
  \item We experimentally prove the effectiveness of CORAL against other dialog generation models that use CE and its variants.
\end{enumerate}

\section{Literature Review}\label{literature-review}

The use of the Seq2Seq model for training chitchat (now known as
open-domain) dialog generation model was first proposed by \citet{ritter2011data}. It was realized very soon that, unlike other NLG problems such
as NMT, training dialog generation models using the CE loss leads to
various complications. Researchers found that at, smaller scales, it was
almost impossible to train a dialog generation model \citep{serban2016building,serban2017hierarchical,zhao2017learning,li2016deep}. \citet{serban2016building} argued that standard RNNs fail to encode the context properly and
proposed a hierarchical encoder-based architecture, HRED, that
captures utterance-level and context-level information using separate
RNNs. To capture the notion of multiple possible responses per context,
\citet{serban2017hierarchical} further augmented their HRED model with a VAE to
propose VHRED. Contemporary work also proposed a VAE based grounded
response generation model kg-CVAE \citep{zhao2017learning}.

Several prior works have also tried to formulate reward functions for training dialog generation models using Reinforcement Learning. To generate coherent and engaging responses, \citet{li2016deep} proposed to maximize expected rewards from future turns estimated using a self-play mechanism in a simulated environment. 
\citet{sankar2019deep} proposes a space of interpretable, discrete and composed attributes for dialogs as the action space for an RL based solution to the open domain dialog generation problem.
\citet{saleh2020hierarchical} uses a hierarchical reinforcement learning framework to allow the model to learn a proper credit assignment at the utterance level instead of the more common word level assignment. They also used self-play for training the model.
\citet{zhao2019rethinking} propose a learnable latent action space for applying RL algorithms to train conversational agents. This relaxes the researcher from making the choice for action spaces when training RL-based dialog systems. Usual choices are dialog acts or output vocabulary.

Although at smaller scales CE-based training leads to degenerate dialog
generation models, at (extremely) large scales of training data and
model-parameter-size, we do get models that can generate diverse,
coherent and interesting responses. Various large-scale dialog models
have been built by researchers for a wide range of model sizes, from
100s of millions to 10s of billions of parameters. DialoGPT \citep{zhang2019dialogpt} is
based on the GPT-2 \citep{radford2019language} language model (LM) further finetuned on a large
conversational corpus crawled from Reddit. Blenderbot \citep{roller2020recipes}
focuses on generating high-quality responses and is based on a
Transformer-based Seq2Seq architecture \citep{vaswani2017attention}. To improve the quality of
responses, Blenderbot finetuned their model on the Blended-Skill-Talk
dataset (combination of multiple datasets, \citealp{smith2020together}) after pretraining on
Reddit. DialogRPT \citep{gao2020dialogrpt} proposes a sample-and-rank method for
candidates generated from DialoGPT. The ranker was trained using a
dataset of upvotes/downvotes and the number of replies on Reddit
comments. Another family of recently proposed dialog generation models \citep{wu2019response,komeili2021internet} work
based on the idea of retrieve-and-refine for efficient training large language models. 

In this paper, we focus on testing the proposed CORAL loss mainly at
smaller scales because 1) the changes are more apparent for smaller scale models, and 2) it becomes more difficult to judge the efficiency and robustness of
  the approach at larger scales where all loss functions seem to work
  upto the same extent. Although we train small-scale models only, our small models outperform several large-sized models across multiple metrics. 
  
 
\section{Methods}\label{sec:methods}

\begin{figure}
    \centering
    \includegraphics[width=\textwidth]{images/coral-compare-losses.drawio2.pdf}
   
    \caption{Schematics of Cross-Entropy (CE) and CORAL Losses at train time. The main idea of CORAL loss is to optimize a measure of the compatibility between the context and a candidate response. For our implementation, we have used a response retrieval model for measuring this compatibility. Another main difference between CE and CORAL is that, CE is more strict and trains the model only based on true response targets. CORAL also utilizes randomly sampled response targets and increases/decreases its probability of decoding based on $R_3$. The encoder part of the Seq2Seq architecture has been hidden here to focus on the relevant part only.}
    \label{fig:compare-losses}
\end{figure}

The overall approach presented in this section comprises a three-step design process. First, we train a reward function ($R_3$) using a response retrieval model architecture. Next, we apply reinforcement learning for training a Seq2Seq context-to-response generation model that maximizes this response retrievability reward. Applying the REINFORCE~\citep{williams1992simple} algorithm leads to the final differentiable objective function (CORAL Loss). Finally, we design Algorithm \ref{alg:coral} to train a Seq2Seq dialog model using minibatch gradient descent with the CORAL loss.

\subsection{The CORAL Loss Function}\label{a.-the-coral-loss-function}

The main idea of CORAL or Contextual Response Retrievability Loss is to
directly optimize some notion of compatibility between the context and
the response. Such a measure is learned by retrieval models like \citet{chen2019sequential,Henderson2020ConveRTEA}. \citet{santra2021representation} recently showed that such models
actually optimize an estimate of the mutual information between context
and response. Their experiments also prove that these representations
are more feature-rich than large-scale pretrained generative dialog
models (based on CE loss). That is why we consider contextual response retrievability, defined below,
as a suitable target to be optimized for training dialog
generation models.


\paragraph{Contextual Response Retrievability Reward ($R_3$)}\label{sec:response-retrievability-reward} We define the output score of an already
trained response retrieval model for a context-response pair as the Contextual Response Retrievability Reward. This score is usually normalized between $0$ and $1$ and indicates the probability that the response is a valid (coherent and on-topic) continuation to the context.
In general, retrieval models compute this score based on both the input context and the output response from a human annotator or a generative model.
As the reward function is
not differentiable, we first formulate response generation as an RL task. Then, we use the REINFORCE algorithm to
obtain a differentiable objective function as will be described next.

\paragraph{Formulation of the CORAL
Loss}\label{formulation-of-the-coral-loss}

For formulating the final expression for the CORAL loss function, we
first pose the response generation problem as a reinforcement
learning task. Each instance of the context-to-response generation task
is considered as an episode in the RL formulation. The episode consists
of several actions taken by the agent, in our case the decoder. Each action corresponds to
generation of an output token. When the agent generates an EOS
(end-of-sequence) token or has produced a max number of allowed tokens
(T), the episode ends and the environment (the response retrieval model)
generates the $R_3$ reward, for the $\langle$context $c$, generated response $r\rangle$ pair.

The updates to the Seq2Seq model weights are then determined by the Episodic REINFORCE
algorithm\footnote{See Section 5 in \cite{williams1992simple}} as:
\begin{align}
  \Delta W_{\text{CORAL}} = \eta R_3(\langle c,r\rangle) \sum_{t=1}^{T} \frac{\partial\log P(r|c)}{\partial W_{\text{CORAL}}}
  \label{eqn:coral-update}
\end{align}


Equivalently, the loss function to be minimized (for an auto-regressive decoder) can be written as follows.
\begin{align}
    L_{\text{CORAL}}=-\eta R_3(\langle c,r\rangle) \sum_{t=1}^{T} \log P(r_t|r_{<t},c)
    \label{eqn:loss-coral}
\end{align}

Fig.~\ref{fig:compare-losses} gives a sense of how the standard cross-entropy loss and the proposed CORAL loss differs from each other. Decoder using CORAL loss also incorporates quality of response predicted by a response retrieval model as a factor in computing the loss.

\subsection{Training Seq2Seq Models using CORAL Loss}

Next, we explain the algorithm used for training a Seq2Seq model for dialog generation using the CORAL loss and an existing dialog dataset $D$. Fig.~\ref{fig:s2s-coral} shows the architecture of our Seq2Seq model which generates a response given the context using a Transformer encoder-decoder model trained under CORAL loss.

\begin{algorithm}
\caption{Training Algorithm for Seq2Seq Models using CORAL Loss}\label{alg:coral}
\begin{algorithmic}
\State $D \gets \{\langle c,r^+\rangle\}$ \Comment{Positive pairs from training dataset}
\State $R \gets $ Pool of all utterances in dataset (train split)
\For{$\langle c,r^+\rangle \in D$} \Comment{Actual implementation uses batch gradient descent}
    \If{$rand() > p_+$}
        \State $r\leftarrow$ Sample $r^- \sim R$ \Comment{Uniform Negative Sampling}
    \Else
    \State $r\leftarrow$ $r^+$ \Comment{Use Positive Response}
    \EndIf
    \State $R_3(\langle c,r\rangle)=f_R(\langle c,r\rangle) - m$ \Comment{$f_R$: Response Retrieval Model Score Function}
    \State Compute $P_{S2S}(r_t|r_{<t},c) \forall t \in [1,T]$ \Comment{$P_{S2S}$: Decoder output token distribution}
    \State $L_{\text{CORAL}}=-R_3(\langle c,r\rangle)\sum_{t=1}^{T}P_{S2S}(r_t|r_{<t},c)$
    \State Update parameters of $P_{S2S}$ using gradient descent on $L_{\text{CORAL}}$
\EndFor
\end{algorithmic}
\end{algorithm}

\paragraph{Training Stability}
In the true spirit of RL, one would use responses sampled from the
Seq2Seq model for each context and train the model based on the reward
for these $\langle$context, decoded response$\rangle$ pairs. 
But, since a high positive reward is obtained only if the generated sequence is an appropriate enough response and only after all the tokens are generated (i.e. the full episode is over), it becomes very difficult to train the model just based on sampling.
Also, because of the combinatorial complexity of the space
of all possible token sequences, the reward will initially be very close to zero for almost all sampled sequences.

Hence, given a dataset $D$ of positive context response pairs, $\langle c, r^+\rangle$, with probability $p_+$, we use the response $r^+$ itself. With the remaining probability ($1-p_+$), we sample utterances randomly from the whole training set as
response candidates $r^-$ for each context $c$. A response retrieval model $R$ is then used to compute the retrieval score $f_R(\langle c,r\rangle)$. To make sure that the probability of good response candidates is increased, we
offset the retrieval score by a fixed margin to get $R_3(\langle c,r\rangle)$. The Seq2Seq decoder computes $P_{S2S}(r_t|r_{<t},c)$ for all tokens $t$ in the response. 
The two factors $R_3$ and $P_{S2S}$ are used to compute the CORAL loss. Then based on the CORAL loss, the
model will increase or decrease the probability of generating these candidate responses while decoding. We illustrate the algorithm to train our Seq2Seq dialog model using minibatch gradient descent with CORAL loss in Algorithm \ref{alg:coral}.




\begin{figure}
    \centering
    \includegraphics[width=0.8\textwidth]{images/coral-s2s-coral.drawio.pdf}
    \caption{This figure shows the model architecture used for training the Seq2Seq dialog model using the CORAL loss. The context input to the Transformer-encoder is denoted by a token sequence $[c_1, c_2, ..., c_L]$. The response input sequence $r = [ r_1,  r_2, ..., r_T]$ denotes the positive or a randomly sampled negative response candidate.}
    \label{fig:s2s-coral}
\end{figure}

\subsection{Hyperparameters of CORAL}\label{hyperparameters-of-coral}
One needs to tune the following hyperparameters when using CORAL for training

\begin{itemize}
    \item  \textbf{Probability of positive samples ($p_+$)} During training each context in a batch is paired up with a response candidate from the full utterance pool (from training set). With probability $p_+$, we select the true response, and with probability $(1-p_+)$, we randomly select a utterance as response.
    \item  \textbf{Margin ($m$)} To make sure that the probability of good response candidates is increased and that of negative responses is
    decreased, we offset the retrieval model score by  a margin\footnote{This is more commonly known as \textit{baseline} in RL literature.} $m$. This margin denotes the minimum score value that we expect for a good response. $R_3 = score-m$. This ensures that the $R_3$ value will be negative for candidates unsuitable for the input context. E.g., if score is based on a sigmoid activation at the output of the model, margin $m$ will be some positive number between 0 and 1.
    \item  \textbf{Retrieval model} There is also the choice of the retrieval model that is used for measuring $R_3$. Here we have chosen ESIM \citep{chen2019sequential}, BERT \citep{devlin2018bert} and DMI \citep{santra2021representation} as the base models for training the response retrieval (i.e., $R_3$ reward) models.
\end{itemize}

\subsection{Similarities and Differences between CORAL and CE Loss Functions}

In this section, we explore the similarities and differences between the proposed CORAL loss and the CE loss function. Although CORAL is derived from quite a different viewpoint, under certain hyperparameter settings CORAL approximates a weighted version of the CE loss.

\begin{enumerate}
    \item If we only consider positive samples as candidate responses and set the score range ($score\in[0,1]$) and margin m ($m=0$) such that $R_3$ is always greater than zero, CORAL is equivalent to a weighted version of CE.
    \item Cross-entropy loss has always relied \textit{strictly} on the positive responses in the dataset. CORAL utilizes both positive and negative response candidates. 
    \item
      Training of a dialog generation model using CE may over-weigh generic responses more than more informative ones as there is no mechanism for automatically assigning weight to different $\langle$context,response$\rangle$ pairs.
      CORAL has provision for assigning different weight for different $\langle$context, candidate response$\rangle$ pairs.
    \item
      CORAL uses randomly sampled response candidates for training which allows us to utilize more samples of $\langle$context,response$\rangle$ pairs during training.
      This provides a richer training signal from the same dataset.
    \item
    CE loss decomposes to a token level comparison between the predicted and the target token. Its main goal is to increase the probability of the tokens in ground truth response strictly in the given form and order.
    CORAL loss works quite differently as it treats responses as whole units. It will either increase or decrease probability of responses as a whole, based
    on their semantics and compatibility to the context.
\end{enumerate}



\section{Experimental Setup}\label{sec:experimental-setup}

\subsection{Model Setup}\label{sec:model-architecture}

We use a standard transformer-based Seq2Seq architecture \citep{vaswani2017attention} for training a
dialog generation model using the CORAL loss as shown in Fig.~\ref{fig:s2s-coral}. We use PyTorch's inbuilt
Transformer module to implement the model and did not change anything
else in the architecture. For tokenization and vocabulary, we used the
word-piece tokenizer from the BERT \cite{devlin2018bert} model\footnote{We used the implementation by huggingface \citep{wolf2019huggingface} library
(bert-base-uncased).}. It has a vocabulary comprising $30,~522$ subword tokens. We train our models using early-stopping, upto a maximum 50 epochs, based on validation-$R_3$ (average $R_3$ score of generated responses on the validation set). We train the model using Adam optimizer with a peak learning rate of $1e-4$ that is warmed up (first 1000 steps) and decayed linearly.

For the retrieval model, we did our main experiments with the LSTM-based ESIM model
proposed by \citep{chen2016enhanced} and transformer-based BERT \cite{devlin2018bert} and DMI \cite{santra2021representation} models. The ESIM model has two LSTM encoding layers (Encoding and Composition Layers) for individually encoding the context and a candidate response, interleaved by a cross-attention layer. The sigmoid output from the ESIM model is used as
the score for a context-response pair. Since both BERT and DMI are pretrained models, we added an MLP, with two hidden layers 
and sigmoid activation at the output, on top of the [CLS] token embedding and finetuned them on DailyDialog or DSTC7-Ubuntu for the response retrieval tasks.
We obtained the reward ($R_3$) subtracting the margin ($m$) (also between 0 and 1) from the output score (sigmoid output). 

\begin{itemize}
\item  \textbf{Configuration of ESIM}  For faster learning, we also initialized the ESIM model's token embeddings with Blenderbot-3B embeddings, which has a size of 2560. The encoding and composition layers both used 1 layer LSTMs.

\item \textbf{Configuration of BERT and DMI} For BERT and DMI based retrieval models, we used the ``bert-base-uncased'' and the ``DMI\_Medium'' checkpoints, respectively. 

\item \textbf{Configuration of S2S (CORAL)} The Transformer-Seq2Seq model had 4 self-attention layers for both the encoder and the decoder. We used 8 self-attention heads and 1024 as the size of hidden representations.

\item \textbf{Training Setup}
We used a single NVIDIA V100-16GB GPU-based system for training all our
models. We used early stopping based on a validation set for both R3 and S2S
training.
\end{itemize}




\subsection{Data}\label{sec:data-and-code}

We used the \textbf{DailyDialog} (DD) \citep{li2017dailydialog} and \textbf{DSTC7-Ubuntu}\footnote{\url{https://github.com/IBM/dstc-noesis}} \citep{yoshino2019dialog} datasets for all our
experiments. DD is an open-domain dialog dataset in English. For
training the retrieval model, we used randomly sampled utterances as
negative samples. DSTC7-Ubuntu is a domain-specific dataset based on
conversations from the Ubuntu IRC channel. DSTC7-Ubuntu contains majorly English conversations, with a small percentage instances from other languages.
This is available directly as
a dataset for training retrieval models with single positive and
multiple negative responses per context. A summary of the datasets'
stats is provided in Table~\ref{tab:data-stats}. We will make the code and trained model checkpoints publicly available upon publication of the paper.

\begin{table}[!htbp]
\centering
\caption{This table reports some basic stats about the two datasets. These represent the number of context-response pair that was extracted from the various splits of the datasets.}
\label{tab:data-stats}
\begin{tabular}{lrrr}
\toprule
Dataset & Train & Valid & Test \\ \midrule
DailyDialog & 76052 & 7069 & 5740 \\
DSTC7-Ubuntu & 470860 & 23478 & 3247 \\
\bottomrule
\end{tabular}
\end{table}

\subsection{Baselines}\label{baselines}

\begin{itemize}
  
  
  
  \item
  \textbf{Mirror} This is a recently proposed \citep{li2021improving} Seq2Seq model trained with a
  backward-reasoning loss function. For making the model generate more informative responses, authors of Mirror extended the CVAE model for dialog generation \citep{shen2017conditional,zhao2017learning} to optimize for generating final and pre-final utterances in a bidirectional fashion. This
  is also currently state-of-the-art loss function for training small-scale dialog generation models and has been shown to outperform \citep{shen2017conditional,zhao2017learning,saleh2020hierarchical}.

  \item
  \textbf{AdaLabel} 
  In the AdaLabel model \cite[]{wang2021diversifying}, the authors applied an adaptive label smoothing to prevent the model from being overconfident over a single choice. The main idea of this paper is to use a soft-target distribution depending on the context, instead of usual one-hot distribution.
  
\item
  \textbf{Blenderbot} This large-scale model is based on the Transformer-based Seq2Seq architecture-based. Blenderbot \citep{roller2020recipes} was first pretrained on a large dialog corpus
  based on Reddit and then further finetuned on Blended Skill Talk dataset \citep{smith2020together}. This final finetuning helps to improve the quality of responses by reducing unwanted response generation.
\item
  \textbf{DialoGPT} DialoGPT \citep{zhang2019dialogpt} is a GPT-2 \citep{radford2019language} based language model further finetuned on dialogs from Reddit.
\item 
    \textbf{DialogRPT} DialogRPT \citep{gao2020dialogrpt} is a response ranking model that was trained based on a dataset of upvote/downvote and number of replies on Reddit comments. For generation purposes it reranks sampled responses from DialoGPT and return the one with highest rank predicted by DialogRPT.
\end{itemize}

\subsection{Evaluation Metrics}\label{evaluation-metrics}

We use a standard set of referenced evaluation metrics and a recently proposed reference-free metric for automatic evaluation. These are described below.

BLEU \citep{papineni2002bleu} and METEOR \citep{banerjee2005meteor} measure lexical match upto $n=4$-grams between the predicted and ground truth response. Since a valid response generated by the model can be different from the ground truth, these two metrics cannot always capture the validity of a generation.

MAUDE \citep{Sinha2020LearningAU} is a recently proposed metric that can capture the suitability between a context and any response without a ground truth reference. MAUDE is also a metric based on a response retrieval model and thus our model effectively optimizes MAUDE through the CORAL loss function. \cite{yeh2021comprehensive} observed that MAUDE performs particularly well when the model has been trained on a dataset from a similar domain to its target application domain. Hence, we report results for two MAUDE variants based on ESIM \citep{chen2016enhanced,chen2019sequential} and BERT \citep{devlin2018bert} finetuned on the target dataset (DSTC7 or DailyDialog).

The Distinct-n \citep{liu2016not} metric measures the diversity of n-grams in the overall set of generated responses. This number should not be too low, and an ideal target for this is the Distinct-n of the ground truth responses.

\section{Results and Discussions}\label{sec:results}

\subsection{Automatic Evaluation}
In Table~\ref{tab:results-dd-new}, we present the results of automatic evaluation
metrics for response generation tasks on DD
dataset. For this experiment, we first trained the model on the DD and DSTC7 datasets and saved
the best checkpoint based on the average $R_3$ reward on the validation set. For our proposed CORAL models, we used a margin is 0.2 for DSTC7 and 0.45 for DailyDialog datasets respectively. Also, we set $p^+$ as 1 for these experiments. Fig.~\ref{fig:hparams} show sensitivity analysis for hyper-parameters $p_+$ and $m$ for both the datasets.

\begin{figure}
     \centering
     \begin{subfigure}[b]{0.48\textwidth}
         \centering
         \includegraphics[width=\textwidth]{plots/DailyDialog-ESIM-hparams.pdf}
         \caption{$\text{CORAL}_\text{ESIM}$ (DailyDialog)}
         \label{fig:dd1}
     \end{subfigure}
     \hfill
     \begin{subfigure}[b]{0.48\textwidth}
         \centering
         \includegraphics[width=\textwidth]{plots/DailyDialog-BERT-hparams.pdf}
         \caption{$\text{CORAL}_\text{BERT}$ (DailyDialog)}
         \label{fig:dd2}
     \end{subfigure}
     
     \begin{subfigure}[b]{0.48\textwidth}
         \centering
         \includegraphics[width=\textwidth]{plots/DailyDialog-DMI-hparams.pdf}
         \caption{$\text{CORAL}_\text{DMI}$ (DailyDialog)}
         \label{fig:dd3}
     \end{subfigure}
     \hfill
     \begin{subfigure}[b]{0.48\textwidth}
         \centering
         \includegraphics[width=\textwidth]{plots/DSTC7-Ubuntu-ESIM-hparams.pdf}
         \caption{$\text{CORAL}_\text{ESIM}$ (DSTC7-Ubuntu)}
         \label{fig:dstc7-1}
     \end{subfigure}
     
     \begin{subfigure}[b]{0.48\textwidth}
         \centering
         \includegraphics[width=\textwidth]{plots/DSTC7-Ubuntu-BERT-hparams.pdf}
         \caption{$\text{CORAL}_\text{BERT}$ (DSTC7-Ubuntu)}
         \label{fig:dstc7-2}
     \end{subfigure}
     \hfill
     \begin{subfigure}[b]{0.48\textwidth}
         \centering
         \includegraphics[width=\textwidth]{plots/DSTC7-Ubuntu-DMI-hparams.pdf}
         \caption{$\text{CORAL}_\text{DMI}$ (DSTC7-Ubuntu)}
         \label{fig:dstc7-3}
     \end{subfigure}
    \caption{Hyperparameter Sensitivity Analysis: These plots showcase the effect of $p^+$ and margin on the final validation-$R_3$ score obtained by the corresponding CORAL model. Each lineplot corresponds to a single $p^+$ value as indicated by the legend.}
    \label{fig:hparams}
\end{figure}


The \textbf{diversity} of the generated responses, in terms of number of unique unigrams and bigrams, is indicated by the dist-1 and dist-2 metrics. All baselines, except AdaLabel, have very low diversity for the generated responses. The RNN-based  Mirror model mostly generates the same bland response (e.g., \textit{``I don't know how to do that''}, \textit{``I don't know what to suggest''}) for most contexts leading to a low diversity score. Although DialogRPT and Blenderbot-3B produce relevant responses (as indicated by a high MAUDE score), the response produced by these models sometimes does not change after the initial few turns, and repeatedly generates the same utterance for all remaining turns which leads to a poor Distinct-n score. The diversity of most baselines (except AdaLabel) is quite low compared to that of CORAL models. 
From Table \ref{tab:results-dd-new}, it is quite clear that only the proposed CORAL-based models are able to produce responses which are both diverse (high dist-n) and coherent to the context (high MAUDE), while most baselines have failed to maintain this consistently.

Next, we also report the \textbf{BLEU} and \textbf{METEOR} scores which are two standard metrics used for evaluating NLG models. Though the reader should note that these metrics do not correlate strongly with human judgments on the quality of generated responses, as shown by several prior works \citep{liu2016not,Sinha2020LearningAU,yeh2021comprehensive}. In terms of these word overlap-based metrics, we outperform most of the baselines, except AdaLabel which beats $\text{CORAL}_\text{ESIM}$ and $\text{CORAL}_\text{DMI}$ on DailyDialog.

In terms of \textbf{average length} of the generated responses, the pretrained models outperform other baselines and also the CORAL models. Though it should be noted that high generation length does not necessarily mean higher quality responses, as also indicated by the MAUDE scores.

Finally, in terms of the \textbf{MAUDE} metric, our proposed method, CORAL, outperforms all baselines by a significant margin. Among all the metrics reported here, MAUDE has been shown to correlate well with human preferences for generated responses, along several important quality aspects (e.g., interestingness, specificity, relevance etc. \citealp{Sinha2020LearningAU}). This was the motivation behind the $R_3$ reward. The diverse and high quality responses justify the choice and design of the CORAL loss function. 









\begin{table}[]
  \caption{Results for DailyDialog and DSTC7 datasets: From the results, we can see that by optimizing the contextual response retrievability score directly, using REINFORCE, the S2S-CORAL model is able to produce coherent and diverse responses at the same time. The average length is reported to make sure that the model is not resorting to short utterances, such as ``I don't think I know about \textit{[topic\_word]}'', just to be coherent. Among the baselines, Blenderbot-3B, which is a large-scale pretrained model, produces the most human-like responses as per MAUDE but not as diverse as the CORAL-based models. \textbf{$\text{CORAL}_x$} denotes a Seq2Seq model trained with CORAL loss. `x' identifies the retrieval model used for the $R_3$ reward. Note that we used the DialoGPT-medium, DialogRPT-medium, Blenderbot-400M-distill and Blenderbot-3B checkpoints. \textit{We have kept these large-scale baselines as a separate group, in the table.} Each value reported for CORAL models is computed as average of 5 runs.} 
  \label{tab:results-dd-new}
  \centering
  \small
  \resizebox{\textwidth}{!}{%
   
    \begin{tabular}{clcrrrrrr}
    \toprule
    &\multicolumn{8}{c}{\textbf{DSTC7}}\\ \midrule
    &Model & Avg. Len & BLEU & METEOR & Dist-1 & Dist-2 & MAUDE-ESIM & MAUDE-BERT  \\
    \midrule
    &Ground Truth & 13.73 & NA & NA & 0.0922 & 0.5231 & 0.8583 & 0.7363\\
    \midrule
    \multirow{4}{*}{\rotatebox{90}{\makecell{Large\\ Models}}}&DialogRPT & 14.17 & 0.0714 & 0.0424 & 0.0351 & 0.1333 & 0.7328 & 0.6813 \\
    &DialoGPT & 8.59 & 0.0565 & 0.0369 & 0.0504 & 0.1723 & 0.7058 & 0.6557 \\
    &Blenderbot-400M-distill & 17.78 & 0.0869 & 0.0571 & 0.0266 & 0.0893 & 0.5683 & 0.5294  \\
    &Blenderbot-3B & 16.94 & 0.0947 & 0.0591 & 0.0320 & 0.1240 & 0.7300 & 0.7404 \\
    \midrule
   
    \multirow{5}{*}{\rotatebox{90}{\makecell{Small\\ Models}}}&Mirror & 6.19 & 0.0299 & 0.0387 & 0.0068 & 0.0145 & 0.5121 & 0.4926 \\
    &AdaLabel & 15.70 & 0.2022 & 0.1824 & 0.0539 & 0.3099 & 0.7929 & 0.7046  \\
    &$\text{CORAL}_\text{ESIM}$ & 10.28 & 0.1729 & 0.1581 & \textbf{0.0744} & \textbf{0.4242} & \textbf{0.8590} & 0.7600  \\
    &$\text{CORAL}_\text{BERT}$ & 10.53 & 0.0949 & 0.0724 & 0.0694 & 0.4093 & 0.8534 & \textbf{0.8108} \\
    &$\text{CORAL}_\text{DMI}$ & 10.39 & \textbf{0.2194} & \textbf{0.2111} & 0.0707 & 0.4151 & 0.8274 & 0.7309  \\
    \midrule
    \midrule
    &\multicolumn{8}{c}{\textbf{DailyDialog}}\\ \midrule
    & Model & Avg. Len & BLEU & METEOR & Dist-1 & Dist-2 & MAUDE-ESIM & MAUDE-BERT  \\
    \midrule
    &Ground Truth & 11.97 & NA & NA & 0.0681 & 0.4061 & 0.8180 & 0.8603\\
    \midrule
    \multirow{4}{*}{\rotatebox{90}{\makecell{Large\\ Models}}}&DialoGPT & 8.89 & 0.0667 & 0.0445 & 0.0415 & 0.1682 & 0.6327 & 0.6016  \\
    &DialoRPT & 15.94 & 0.0809 & 0.0530 & 0.0273 & 0.1192 & 0.6523 & 0.6055  \\
    &Blenderbot-400M-distill & 17.27 & 0.1049 & 0.0771 & 0.0223 & 0.0977 & 0.6363 & 0.6322  \\
    &Blenderbot-3B & 16.47 & 0.1063 & 0.0783 & 0.0302 & 0.1284 & 0.7137 & \textbf{0.6812} \\
    \midrule
   
    \multirow{5}{*}{\rotatebox{90}{\makecell{Small\\ Models}}}&Mirror & 7.93 & 0.0618 & 0.0538 & 0.0371 & 0.1485 & 0.6223 & 0.4995  \\
    &AdaLabel & 11.59 & 0.1208 & 0.0881 & 0.0381 & 0.2292 & 0.5814 & 0.4890 \\
    &$\text{CORAL}_\text{ESIM}$& 10.00 & \textbf{0.1910} & \textbf{0.1727} & 0.0459 & 0.2840 & \textbf{0.7402} & 0.6474  \\
    &$\text{CORAL}_\text{BERT}$& 9.68 & 0.1838 & 0.1656 & \textbf{0.0462} & \textbf{0.2902} & 0.7279 & 0.6526 \\
    &$\text{CORAL}_\text{DMI}$& 9.44 & 0.1727 & 0.1567 & 0.0461 & 0.2844 & 0.7229 & 0.6234  \\
    \bottomrule
    \end{tabular}
  }
\end{table}

\begin{table}[]
  \caption{This table shows sample generations from different baseline models and our $\text{CORAL}_\text{BERT}$ model. Compared to Mirror, responses from our model feels like a more natural continuation to the context. The AdaLabel baseline also has a higher tendency to generate a generic/dull response to the context. A common issue that we notice in all small-scale models is it's random behavior when it encounters a new concept unseen during training.}
  \label{tab:manual-samples}
  \centering
  \scriptsize
  \resizebox{\textwidth}{!}{%
  \begin{tabular}{llll}
  \toprule
  \multicolumn{4}{c}{\textbf{Example Generations from all Models}} \\ \midrule
  \multicolumn{4}{L{14cm}}{\textbf{Context}: I really need to start eating healthier. EOU  I have to start eating better too. EOU  What kind of food do you usually eat? EOU  I try my best to eat only fruits, vegetables, and chicken. EOU} \\ \midrule
  \multicolumn{1}{L{3.2cm}|}{\textbf{Mirror}: What kind of food do you want ?} & 
  \multicolumn{1}{L{3.2cm}|}{\textbf{Blender}: I try to eat a lot of veggies and lean meats.} & 
  \multicolumn{1}{L{3.2cm}|}{\textbf{AdaLabel} i have no idea what i'm supposed to eat .} & 
  \multicolumn{1}{L{3.2cm}}{\textbf{$\text{CORAL}_\text{BERT}$}: I usually eat a bowl of dumplings.}\\ \midrule

  \multicolumn{4}{L{14cm}}{\textbf{Context}: Room service, is there anything I can do for you? EOU  Yes, I am afraid there are not enough outlets. EOU} \\ \midrule
  \multicolumn{1}{L{3.2cm}|}{\textbf{Mirror}: Have you got any wine ?} &
  \multicolumn{1}{L{3.2cm}|}{\textbf{Blender}: I'm not sure what you mean by that, but I'm sure there's something you can do.} & 
  \multicolumn{1}{L{3.2cm}|}{\textbf{AdaLabel}: ok. thank you .} & 
  \multicolumn{1}{L{3.2cm}}{\textbf{$\text{CORAL}_\text{BERT}$}: OK, sir. Can I take your room key to the Room 20 minutes?} \\ \midrule

  \multicolumn{4}{L{14cm}}{\textbf{Context}: Good evening, madam. Can I help you? EOU Yes. I bought this sheet here yesterday. EOU} \\ \midrule
  \multicolumn{1}{L{3.2cm}|}{\textbf{Mirror}: Did it happen ?} &
  \multicolumn{1}{L{3.2cm}|}{\textbf{Blender}: How much did you pay for it?? I'm a madam} & 
  \multicolumn{1}{L{3.2cm}|}{\textbf{AdaLabel}: ok. when will you come back ?} & 
  \multicolumn{1}{L{3.2cm}}{\textbf{$\text{CORAL}_\text{BERT}$}: I'd like to buy a new dress.} \\ \midrule

  \multicolumn{4}{L{14cm}}{\textbf{Context}: Could I have my bill, please? EOU Certainly, sir. EOU I'm afraid there's been a mistake. EOU I'm sorry, sir. What seems to be the trouble? EOU} \\ \midrule
  \multicolumn{1}{L{3.2cm}|}{\textbf{Mirror}: Yes, I have been looking forward to it .} &
  \multicolumn{1}{L{3.2cm}|}{\textbf{Blender}: I don't know what to do with my life. I feel like I have no purpose.} & 
  \multicolumn{1}{L{3.2cm}|}{\textbf{AdaLabel} oh, i see. then, let me see. ... oh, this switch down to the shop tonight. i' ve got a lot} & 
  \multicolumn{1}{L{3.2cm}}{\textbf{$\text{CORAL}_\text{BERT}$} I'm sorry, sir. I've got you a wrong case. I've been waiting for the wrong medicine. I'm sorry.} \\ \midrule

  \multicolumn{4}{L{14cm}}{\textbf{Context}: Are you busy tomorrow morning? EOU I'm free. What's up? EOU Someone has to pick up the boss at the airport. EOU} \\ \midrule
  \multicolumn{1}{L{3.2cm}|}{\textbf{Mirror}: Oh, I see. Thank you .} &
  \multicolumn{1}{L{3.2cm}|}{\textbf{Blender}: Yes, I have to get up early to go to work. How about you?} & 
  \multicolumn{1}{L{3.2cm}|}{\textbf{AdaLabel}: i' m sorry to hear that. do you want to go to the hospital ?} & 
  \multicolumn{1}{L{3.2cm}}{\textbf{$\text{CORAL}_\text{BERT}$}: Yes. I'd like to. What time do you want to} \\ \bottomrule
  \end{tabular}
  }
\end{table}



\subsection{Case Study: Generation Quality of Models}\label{human-evaluation}
Aside from automatic evaluation, we also manually went through a random selections of generated outputs from all models and compared them. We wanted to find out the strengths and weaknesses of each of the approaches, both for baselines and CORAL. We summarize these points below. To help the reader to visualize these qualities, we provide samples generated from each of the models in Table \ref{tab:manual-samples}.\\
\textbf{Strengths} Since Blenderbot-3B \citep{roller2020recipes} model was trained with a much larger dialog corpus, it is able to generate responses in many cases where small-scale models failed to understand the context. In such cases, the small-scale models generate some random utterance. Among the small scale models, CORAL generally replies with more engaging and specific responses, whereas Mirror tends to produce generic responses most of the time.\\
\textbf{Weaknesses} Blenderbot fails to realize the switch in the turns sometimes and generates the same response repeatedly, only based on the context topic. This is the reason for low diversity scores of Blenderbot. For all small-scale models, because of limited training data, they lack knowledge of many entities that only appears in test set but not during training. In such cases, the small scale models just produces some unrelated utterance.

\section{Limitations}
 
While running the experiments, we discovered a few limitations for the specific implementation of the CORAL loss as described in this paper. First problem is regarding the difficulty of exploration in this current setup. The space of all possible sequences is combinatorially large and provides reward very sparsely. Because of this it is really difficult to do on-policy training. The second problem is with randomly sampled negative response candidates which are often too trivial (leading to zero reward) to help the model while training.

\section{Conclusion}
Natural Language Generation models in NLP have usually been
trained using the cross-entropy loss, and it has been reasonably
successful too. But dialog-response generation poses a few
unique challenges for the CE loss. 
First, cross-entropy loss assumes
that for any given input, the only possible output is the one available
as the ground truth through the dataset. 
Secondly, cross-entropy does not take the context into consideration
while processing the response. 
In this paper, we
propose a novel loss function, CORAL, that circumvents these
shortcomings of CE and is more effective for training dialog-generation
models. 
Using CORAL, we can train dialog generation models without
assuming a fixed ground-truth response and the value of the loss function is based on both the context and response. 
The CORAL loss is based on pretrained response retrieval models that, in prior literature, have been shown to correlate with human preferences.
The proposed loss function will make it possible to train future models focused on maximizing human preference. 
We also hope that our work will motivate the NLP
community to look for more suitable loss functions for training dialog
generation models and to rely less on the cross-entropy loss.

We plan to extend this framework for training larger scale models that can capture more patterns from larger training data. We are also looking into the possibility of designing a learning curriculum using the retrieval model. This will help make the training of the dialog generation model more efficient.

Research in loss functions also gives us insights into which attributes for a generative dialog system are most appreciated in the final model as we compare their outcomes.

\section*{Acknowledgments}
This work was partially supported by Microsoft Academic Partnership Grant (MAPG) 2021. The first author was also supported by Prime Minister's Research Fellowship (PMRF), India.

\bibliographystyle{abbrvnat}
