\section{Introduction}
Automatic evaluation measures, such as \textsc{\small bleu}{} \cite{papineni2002bleu}, are used with increasing frequency to evaluate Natural Language Generation (NLG) systems: Up to 60\% of  NLG research published between 2012--2015 relies on automatic metrics \cite{gkatzia:enlg2015}.
Automatic evaluation is popular because it is cheaper and faster to run than human evaluation, and it is needed for automatic benchmarking and tuning of algorithms. 
The use of such metrics is, however, only sensible if they are known to be sufficiently correlated with human preferences.
This is rarely the case, as shown by various studies in NLG  (\citealp{stent2005evaluating}; \citealp{Belz:EACL06,belz:CL2009}), as well as in  related fields, such as  dialogue systems \cite{Liu:EMNLP2016}, machine translation (MT) \cite{callison:eacl2006}, and image captioning \cite{keller:acl2014,erdem:eacl2017}.
This paper follows on from the above previous work and presents another evaluation study into automatic metrics with the aim to firmly establish the need for new metrics. 
We consider this paper to be the most complete study to date, across metrics, systems, datasets and domains, focusing on recent advances in data-driven NLG.
 In contrast to previous work, we are the first to:

\noindent $\bullet$  Target end-to-end data-driven NLG,  
where we compare 3 different approaches. 
In contrast to NLG methods evaluated in previous work, our  
systems can produce ungrammatical output by (a) generating word-by-word, and (b) learning from noisy data.

\noindent $\bullet$ Compare a large number of 21
automated metrics, including novel grammar-based ones. 

\noindent $\bullet$ Report results on two different domains and three different datasets, which allows us to draw more general conclusions.

\noindent $\bullet$ Conduct a detailed error analysis, which 
suggests that, while metrics can be  reasonable indicators at the system-level, they are not reliable at the sentence-level.

\noindent $\bullet$ Make all associated code and data publicly available, including detailed analysis results.\footnote{Available for download at: \url{https://github.com/jeknov/EMNLP_17_submission}}

\section{End-to-End NLG Systems}\label{sec:systems}

In this paper, we focus on recent end-to-end, data-driven NLG methods, which jointly learn sentence planning and surface realisation from non-aligned data (\citealp{jurcicek:2015:ACL,wen:emnlp2015,Mei:NAACL2015,Wen:NAACL16}; \citeauthor{SharmaHSSB16}, \citeyear{SharmaHSSB16}; \citealp{Dusek:ACL16}, \citealp{vlachos:coling2016}).
These approaches do not require costly semantic alignment between Meaning Representations (MR) and human references (also referred to as ``ground truth" or ``targets"), but are based on parallel datasets, which can be collected in sufficient quality and quantity using effective crowdsourcing techniques, e.g.\ \cite{novikova:INLG2016}, and as such, enable rapid development of NLG components in new domains. 
In particular, we compare the performance of the following systems:

\noindent $\bullet$ {\bf \textsc{rnnlg} :}{}\footnote{\url{https://github.com/shawnwun/RNNLG}}
The system by \citet{wen:emnlp2015} uses a Long Short-term Memory (LSTM) network to
jointly address sentence planning and surface realisation.
It augments each LSTM cell with a gate that conditions it on the input MR, which allows it to keep track of MR contents generated so far.


\noindent $\bullet$ {\bf \textsc{\small TGen}}%{Du\v{s}ek :}\footnote{\url{https://github.com/UFAL-DSG/tgen}}
The system by \citet{jurcicek:2015:ACL} learns to incrementally generate deep-syntax dependency trees of candidate sentence plans (i.e.\ which MR elements to mention and the overall sentence structure). Surface realisation is performed using a separate, domain-independent rule-based module.


\noindent $\bullet$ {\bf \textsc{lols}}%{Lampouras \& Vlachos  :}\footnote{\url{https://github.com/glampouras/JLOLS_NLG}}
The system by \citet{vlachos:coling2016} learns sentence planning and surface realisation using Locally Optimal Learning to Search (\textsc{lols}}%{Lampouras \& Vlachos ), an imitation learning framework which learns using \textsc{\small bleu}{} and \textsc{\small rouge}{} as non-decomposable loss functions.



\begin{table}[tb]
\begin{center}
\footnotesize
\begin{tabular}{c|r|r|r|r}
\multirow{2}{*}{\bf System} & \multicolumn{3}{c|}{\bf Dataset} & \multirow{2}{*}{Total} \\
 & \textsc{\small Bagel} & \textsc{\small SFRest} & \textsc{\small SFHotel} & \Bstrut \\\hline\hline
\textsc{lols}}%{Lampouras \& Vlachos  & 202   & 581    & 398   & 1,181\Tstrut \\
\textsc{rnnlg}     & -     & 600    & 477   & 1,077 \\
\textsc{\small TGen}}%{Du\v{s}ek   & 202   & -      & -     &   202\Bstrut \\\hdashline[0.5pt/2pt]
Total    & 404   & 1,181  & 875   & 2,460\Tstrut \\
\end{tabular}
\end{center}
\caption{Number of NLG system outputs from different datasets and systems used in this study.}\label{tab:overview}
\vspace{-0.5cm}
\end{table}

\section{Datasets}\label{sec:data}

We consider the following crowdsourced datasets, which target utterance generation for spoken dialogue systems. 
Table \ref{tab:overview} shows the number of system outputs for each dataset.
Each data instance consists of one MR and one
or more natural language references as produced by humans, such as the following example, taken from the \textsc{\small Bagel}{} dataset:%
\footnote{Note that we use lexicalised versions of \textsc{\small SFHotel}{} and \textsc{\small SFRest}{} and a partially lexicalised version of \textsc{\small Bagel}, where proper names and place names are replaced by placeholders (``X''), in correspondence with the outputs generated by the systems, as provided by the system authors.}
\vspace{0.2cm}

\noindent\begin{boxedminipage}{\columnwidth}
 \small {\bf MR:} inform(name=X, area=X, pricerange=moderate, type=restaurant)\\
 {\bf Reference:} ``{\em X is a moderately priced restaurant in X.}''
 \end{boxedminipage}   
\vspace{0.1cm}
 
\noindent  $\bullet$ {\bf \textsc{\small SFHotel}{} \& \textsc{\small SFRest}}  \cite{wen:emnlp2015} provide information about hotels and restaurants in San Francisco. There are 8 system dialogue act types, such as \emph{inform}, 
\emph{confirm}, 
\emph{goodbye}
 etc.
 Each domain contains 12 attributes, where some are common to both domains,
 such as \emph{name, type, pricerange, address, area,} etc.,
 and the others are domain-specific, e.g.\ \emph{food} and \emph{kids-allowed} for restaurants;
 \emph{hasinternet} and \emph{dogs-allowed} for hotels.

For each domain, around 5K human references were collected
with 2.3K unique human utterances for \textsc{\small SFHotel}{} and 1.6K for \textsc{\small SFRest}. The number of unique system outputs produced is 1181 for \textsc{\small SFRest}{} and 875 for \textsc{\small SFHotel}{}. 

\noindent $\bullet$ {\bf \textsc{\small Bagel}} \cite{mairesse:acl2010} provides information about restaurants in Cambridge. The dataset contains 202 aligned pairs of MRs and 2 corresponding references each. 
The domain is a subset of \textsc{\small SFRest}, including only the  {\em inform} act and 8 attributes.

\section{Metrics}\label{sec:metrics}

\subsection{Word-based Metrics (WBMs)}\label{sec:wbm}

NLG evaluation has borrowed a number of automatic metrics from related fields, such as MT, summarisation or image captioning,
which compare
output texts generated by systems to ground-truth references produced by humans. We refer to this group as 
 word-based metrics.
 In general, the higher these scores are, the better or more similar to the human references the output is.\footnote{Except for \textsc{\small ter}{} whose scale is reversed.}
 The following order reflects the degree these metrics move from simple $n$-gram overlap to also considering term frequency (TF-IDF) weighting and semantically similar words.


\noindent $\bullet$ {\bf Word-overlap Metrics (WOMs):} 
We consider frequently used metrics, including 
 {\bf \textsc{\small ter}} \cite{ter},
{\bf \textsc{\small bleu}} \cite{papineni2002bleu},
{\bf \textsc{\small rouge}} \cite{lin2004rouge},
{\bf \textsc{\small nist}} \cite{nist},
{\bf \textsc{\small lepor}} \cite{lepor},
{\bf \textsc{\small cide}r} \cite{cider}, and
{\bf \textsc{\small meteor}} \cite{meteor}

\noindent $\bullet$ {\bf Semantic Similarity (\textsc{\small sim})
: We calculate the Semantic Text Similarity measure designed by \citet{han2013umbc}.
This measure is based on distributional similarity and Latent Semantic Analysis (LSA) and is further complemented with semantic relations extracted from WordNet. 
\subsection{Grammar-based metrics (GBMs)}
\label{sec:gbm}
Grammar-based measures have been explored in related fields, such as MT  \cite{Gimenez:2008} or grammatical error correction \cite{napoles2016}, and, in contrast to WBMs, do not rely on ground-truth references. 
To our knowledge, we are the first to consider
 GBMs for sentence-level NLG evaluation. We focus on two important properties of texts here -- readability and grammaticality:
 
\noindent $\bullet$ {\bf Readability}
quantifies the difficulty with which a reader understands a text, as used for e.g.\ evaluating summarisation \cite{kan2001applying} or text simplification \cite{readabilitySI:2014}. 
We measure readability by the
Flesch Reading Ease score ({\bf \textsc{\small re}})~\cite{flesch1979write},
which  calculates
a ratio between the number of characters per sentence, the number of words per sentence,
and
the number of syllables per word. Higher \textsc{\small re}{} score indicates a less complex utterance that is easier to read and understand.
We also consider related measures, such as 
 characters per utterance ({\bf len}) and per word ({\bf cpw}), words per sentence ({\bf wps}), syllables per sentence ({\bf sps}) and per word ({\bf spw}), as well as polysyllabic words per utterance ({\bf pol}) and per word ({\bf ppw}). The higher these scores, the more complex the utterance.

\noindent $\bullet$ {\bf Grammaticality:} 
In contrast to previous NLG methods, our corpus-based end-to-end systems can produce ungrammatical output by (a) generating word-by-word, and (b) learning from noisy data.
As a first approximation of
grammaticality, we measure the number of misspellings (\textbf{msp}) and the parsing score as returned by the Stanford parser (\textbf{prs}). The lower the \textbf{msp}, the more grammatically correct an utterance is. 
The 
Stanford parser score is not designed to measure grammaticality, however, it will generally prefer a grammatical parse to a non-grammatical one.\footnote{\url{http://nlp.stanford.edu/software/parser-faq.shtml}} Thus, lower parser scores indicate less grammatically-correct utterances. 
In future work, we aim to use specifically designed grammar-scoring functions, e.g.\ \cite{napoles2016}, once they become publicly available.

\section{Human Data Collection}\label{sec:datacollection}

To collect human rankings, 
we presented the MR together with 2 utterances generated by different systems side-by-side to crowdworkers, which were asked to score each utterance on a 6-point Likert scale for:

\noindent $\bullet$ {\bf Informativeness:} {\em Does the utterance  provide all the useful information from the meaning representation?}

\noindent$\bullet$ {\bf Naturalness:} {\em Could the utterance have been produced by a native speaker?}

\noindent$\bullet$ {\bf Quality:} {\em How do you judge the overall quality of the utterance in terms of its grammatical correctness and fluency?}

Each system output (see~Table~\ref{tab:overview}) was scored by 3 different crowdworkers.
To reduce participants' bias, the order of appearance of utterances produced by each system was randomised and crowdworkers were restricted to evaluate a maximum of 20 utterances.
The crowdworkers were selected from  English-speaking countries only, based on their IP addresses, and asked to confirm
that English was their native language.

To assess the reliability of ratings, we calculated the intra-class correlation coefficient (ICC), which measures inter-observer reliability on ordinal data for more than two raters \cite{landis1977measurement}. The overall ICC across all three datasets is 0.45 ($p<0.001$), which corresponds to a moderate agreement. 
In general, we find consistent differences in inter-annotator agreement per system and dataset,
with lower agreements for \textsc{lols}}%{Lampouras \& Vlachos {} than for \textsc{rnnlg}{} and \textsc{\small TGen}}%{Du\v{s}ek. Agreement is highest for the \textsc{\small SFHotel}{} dataset, followed by \textsc{\small SFRest}{} and \textsc{\small Bagel}{} (details provided in supplementary material). 

\section{System Evaluation}\label{sec:evaluation}

\begin{table*}[h!]
\centering
\begin{adjustbox}{max width=1\textwidth}
\begin{tabular}{l|cc|cc|cc}
 & \multicolumn{2}{c|}{\textsc{\small Bagel}} & \multicolumn{2}{c|}{\textsc{\small SFHotel}} & \multicolumn{2}{c}{\textsc{\small SFRest}} \\
\bf metric & \multicolumn{1}{c}{\textsc{\small TGen}}%{Du\v{s}ek} & \multicolumn{1}{c|}{\textsc{lols}}%{Lampouras \& Vlachos } & \multicolumn{1}{c}{\textsc{rnnlg}} & \multicolumn{1}{c|}{\textsc{lols}}%{Lampouras \& Vlachos } & \multicolumn{1}{c}{\textsc{rnnlg}} & \multicolumn{1}{c}{\textsc{lols}}%{Lampouras \& Vlachos } \\
\hline \hline
WOMs &  & More overlap & More overlap* &  & More overlap* & \Tstrut \\
\textsc{\small sim} & More similar &  & More similar* &  &  & More similar \\
\hdashline[1pt/3pt]
GBMs & Better grammar(*) &  & Better grammar(*) &  & Better grammar &  \\
\textsc{\small re} &  & More complex* &  & More complex* &  & More complex*\Bstrut\\
\hline
inform & 4.77(Sd=1.09) & \textbf{4.91}(Sd=1.23) & \textbf{5.47*}(Sd=0.81) & 5.27(Sd=1.02) & \textbf{5.29*}(Sd=0.94) & 5.16(Sd=1.07)\Tstrut\\
natural & \textbf{4.76}(Sd=1.26) & 4.67(Sd=1.25) & \textbf{4.99*}(Sd=1.13) & 4.62(Sd=1.28) & \textbf{4.86}\phantom{*}(Sd=1.13) & 4.74(Sd=1.23) \\
quality & \textbf{4.77}(Sd=1.19) & 4.54(Sd=1.28) & \textbf{4.54}\phantom{*}(Sd=1.18) & 4.53(Sd=1.26) & 4.51\phantom{*}(Sd=1.14) & \textbf{4.58}(Sd=1.33)
\end{tabular}
\end{adjustbox}
\caption{System performance per dataset (summarised over metrics), where ``*'' denotes $p<0.05$ for all the metrics and ``(*)'' shows significance on $p<0.05$ level for the majority of the metrics.}
\label{tab:sys-eval}
\end{table*}


Table \ref{tab:sys-eval} summarises the

 individual systems' overall corpus-level performance in terms of automatic and human scores (details are provided in the supplementary material). 

All WOMs
produce similar results, with \textsc{\small sim}{}
showing different results for the restaurant domain (\textsc{\small Bagel}{} and \textsc{\small SFRest}).
Most GBMs show the same trend (with different levels of statistical significance), but \textsc{\small re}{} is showing inverse results.
 System performance is dataset-specific:
For WBMs,
 the \textsc{lols}}%{Lampouras \& Vlachos {} system consistently produces better results on \textsc{\small Bagel}{} compared to \textsc{\small TGen}}%{Du\v{s}ek,
 while for \textsc{\small SFRest}{} and \textsc{\small SFHotel}, \textsc{lols}}%{Lampouras \& Vlachos {} is outperformed by \textsc{rnnlg}{} in terms of WBMs.
We observe that human {\em informativeness} ratings follow the same pattern as WBMs, while the average similarity score (\textsc{\small sim}) seems to be related to human {\em quality} ratings.

Looking at GBMs, we observe that they seem to be related to {\em naturalness} and {\em quality} ratings.
Less complex utterances, as measured by readability (\textsc{\small re}) and word length (cpw), have higher {\em naturalness} ratings.
More complex utterances, as measured in terms of their length (len), number of words (wps), syllables (sps, spw) and polysyllables (pol, ppw), have lower {\em quality} evaluation.
 Utterances measured as more grammatical
 are on average evaluated higher in terms of {\em naturalness}.

These initial results suggest a relation between automatic metrics and human ratings at system level. However, average scores can be misleading, as they do not identify worst-case scenarios.
This leads us to inspect the correlation of human and automatic metrics for each MR-system output pair at utterance level.


\begin{table*}[ht]
\centering
\begin{adjustbox}{max width=1\textwidth}
\begin{tabular}{cl|ll|ll|ll}
 &  & \multicolumn{2}{c|}{\textsc{\small Bagel}} & \multicolumn{2}{c|}{\textsc{\small SFHotel}} & \multicolumn{2}{c}{\textsc{\small SFRest}} \\
 &  & \textsc{\small TGen}}%{Du\v{s}ek & \textsc{lols}}%{Lampouras \& Vlachos  & \textsc{rnnlg} & \textsc{lols}}%{Lampouras \& Vlachos  & \textsc{rnnlg} & \textsc{lols}}%{Lampouras \& Vlachos  \\
\hline \hline
Best & inform. & \p0.30* (\textsc{\small bleu}{\small-1}) & \p0.20* (\textsc{\small rouge}) & \p0.09 (\textsc{\small bleu}{\small-1}) & \p0.14* (\textsc{\small lepor}) & \p0.13* (\textsc{\small sim}) & \p0.28* (\textsc{\small lepor}) \\
WBM & natural. & -0.19* (\textsc{\small ter}) & -0.19* (\textsc{\small ter}) & \p0.10* (\textsc{\small meteor}) & -0.20* (\textsc{\small ter}) & \p0.17* (\textsc{\small rouge}) & \p0.19* (\textsc{\small meteor}) \\
 & quality & -0.16* (\textsc{\small ter}) & \p0.16* (\textsc{\small meteor}) & \p0.10* (\textsc{\small meteor}) & -0.12* (\textsc{\small ter}) & \p0.09* (\textsc{\small meteor}) & \p0.18* (\textsc{\small lepor}) \\
 \hline
Best & inform. & \p0.33* (wps) & \p0.16* (ppw) & -0.09 (ppw) & \p0.13* (cpw) & \p0.11* (len) & \p0.21* (len) \\
GBM & natural. & -0.25* (len) & -0.28* (wps) & -0.17* (len) & -0.18* (sps) & -0.19* (wps) & -0.21* (sps) \\
 & quality & -0.19* (cpw) & \p0.31* (prs) & -0.16* (ppw) & -0.17* (spw) & \p0.11* (prs) & -0.16* (sps) \\
\end{tabular}
\end{adjustbox}
\caption{Highest absolute Spearman correlation between metrics and human ratings, 
 with ``*'' denoting $p<0.05$ (metric with the highest absolute value of $\rho$ given in brackets).
\label{tab:corr_rf}
\end{table*}

\begin{figure*}[h!]
\centering
\vspace{-2mm}
\includegraphics[width=0.99\textwidth]{images/corr2.png}
\vspace{2mm}
\caption{Spearman correlation results for \textsc{\small TGen}}%{Du\v{s}ek{} on \textsc{\small Bagel}.
Bordered area shows correlations between human ratings and automatic metrics, the rest shows correlations among the metrics. Blue colour of circles indicates positive correlation, while red indicates negative correlation. The size of circles denotes the correlation strength.} 
\label{fig:corr}
\end{figure*}

\begin{figure}[h]
\centering
\includegraphics[width=0.49\textwidth]{images/wil_dusek.png}
\caption{Williams test results: X represents a {\em non}-significant difference between correlations ($p<0.05$; top: WBMs, bottom: GBMs).} 
\label{fig:wil}
\end{figure}

\section{Relation of Human and Automatic Metrics}\label{sec:correlation}



\subsection{Human Correlation Analysis}\label{ssec:correlation}

We
calculate the correlation between automatic metrics and human ratings
using the Spearman coefficient ($\rho$).
We split the data per dataset and system in order to make valid pairwise comparisons. 
To handle outliers within human ratings, we use the median score of the three human raters.%
\footnote{As an alternative to using the median human judgment for each item,
a more effective way to use all the human judgments could be to use
\citet{hovy_learning_2013}'s MACE tool for inferring the reliability of
judges.}
Following \citet{erdem:eacl2017}, we use the Williams' test \cite{williamstest} to determine significant differences between correlations.
Table~\ref{tab:corr_rf} summarises the utterance-level correlation results between automatic metrics and human ratings, listing the best (i.e.\ highest absolute $\rho$) results for each type of metric (details provided in supplementary material). 
 Our results  
suggest that:

\noindent $\bullet$ 
In sum, no metric produces
an even moderate correlation with human ratings, 
 independently of dataset, system, or aspect of human rating. This contrasts with our initially promising results on the system level (see Section \ref{sec:evaluation})
 and will be further discussed in Section \ref{sec:error}. Note that similar inconsistencies between document- and sentence-level evaluation results are observed in MT \cite{specia:MT2010}. 

\noindent $\bullet$ Similar to our
results in Section \ref{sec:evaluation}, we find that WBMs show better correlations to human ratings of {\em informativeness}
(which reflects content selection),
whereas GBMs show better correlations to {\em quality} and {\em naturalness}. 

\noindent $\bullet$ Human ratings for {\em informativeness}, {\em naturalness} and {\em quality} are highly correlated with each other, with the highest correlation between the latter two ($\rho=0.81$) reflecting that they both target surface realisation.

\noindent $\bullet$ All WBMs produce similar results (see Figure~\ref{fig:corr} and \ref{fig:wil}): They are strongly correlated with each other, and most of them produce correlations with human ratings which are {\em not} significantly different from each other. 
GBMs, on the other hand, show greater diversity. 

\noindent $\bullet$ Correlation results are system- and dataset-specific (details provided in supplementary material).
We observe the  highest correlation for \textsc{\small TGen}}%{Du\v{s}ek{} on \textsc{\small Bagel}{} (Figures~\ref{fig:corr} and \ref{fig:wil}) and \textsc{lols}}%{Lampouras \& Vlachos {} on \textsc{\small SFRest}, whereas \textsc{rnnlg}{} often shows low
correlation between metrics and human ratings. 
This lets us conclude that WBMs and GBMs are sensitive to different systems and datasets.


 \noindent $\bullet$ The highest positive correlation
  is observed between the number of words (wps) and {\em informativeness} for the \textsc{\small TGen}}%{Du\v{s}ek{} system on \textsc{\small Bagel}{} ($\rho=0.33$,\linebreak $p<0.01$, see Figure~\ref{fig:corr}){.
However, the wps metric (amongst most others) is not robust across systems and datasets: Its correlation on other datasets is very weak, ($\rho\le .18$) and its correlation with informativeness ratings of \textsc{lols}}%{Lampouras \& Vlachos {} outputs is insignificant.

\noindent $\bullet$ As a sanity check, we also measure a random score $[0.0,1.0]$ which proves to have a close-to-zero correlation with human ratings (highest $\rho=0.09$).


\subsection{Accuracy of Relative Rankings}\label{ssec:ranking}

We now evaluate a more coarse measure, namely the metrics' ability to predict relative human ratings. That is, we compute the score of each metric for
two system output sentences corresponding to the same MR. 
The prediction of a metric is correct if it orders the sentences in the same way as median human ratings (note that ties are allowed).
Following previous work \cite{cider,erdem:eacl2017}, we mainly concentrate on WBMs. 
 Results summarised in Table \ref{tab:accuracy}
 show that most metrics' performance is not significantly different from that of a random score
 (Wilcoxon signed rank test).
 While the random score fluctuates between 25.4--44.5\% prediction accuracy, the metrics achieve an accuracy of between 30.6--49.8\%.
 Again, the performance of the metrics is dataset-specific:
 Metrics perform best on \textsc{\small Bagel}{} data; for \textsc{\small SFHotel}, metrics show mixed performance while for \textsc{\small SFRest}, metrics perform worst.
 
 \ignore{
 The metrics perform best on \textsc{\small Bagel}{} data, with all metrics
 performing significantly better than random.
 For \textsc{\small SFHotel}, metrics show mixed performance: 
For informativeness, all but \textsc{\small nist}{} are better than the random metric, and for naturalness only \textsc{\small meteor}{} outperforms random.
None of the metrics can outperform random for quality (with \textsc{\small sim}{} and \textsc{\small re}{} being significantly worse than random).

For \textsc{\small SFRest}, metrics perform worst, with only \textsc{\small sim}{} for informativeness and \textsc{\small lepor}{} for naturalness outperforming random.
\OD{I'd shorten the talk about datasets and leave it to the table... just mention that the performance is dataset-specific?}


\begin{table}[h!]
\centering
\begin{adjustbox}{max width=0.5\textwidth}
\begin{tabular}{ll|l|l|l}
\multicolumn{2}{l|}{} & \bf informat. & \bf naturalness & \bf quality \\
\hline
\hline
\textsc{\small Bagel} & \begin{tabular}[c]{@{}l@{}}raw \\ data\end{tabular} & \begin{tabular}[c]{@{}l@{}}\textsc{\small ter}, \textsc{\small bleu}{\small 1-4}, \\ \textsc{\small rouge}, \textsc{\small nist}, \\ \textsc{\small lepor}, \textsc{\small cide}r, \\ \textsc{\small meteor}, \textsc{\small sim} \end{tabular} & \begin{tabular}[c]{@{}l@{}}\textsc{\small ter}, \textsc{\small bleu}{\small 1-4}, \\ \textsc{\small rouge}, \textsc{\small nist}, \\ \textsc{\small lepor}, \textsc{\small cide}r, \\ \textsc{\small meteor}, \textsc{\small sim} \end{tabular} & \begin{tabular}[c]{@{}l@{}}\textsc{\small ter}, \textsc{\small bleu}{\small 1-4}, \\ \textsc{\small rouge}, \textsc{\small nist}, \\ \textsc{\small lepor}, \textsc{\small cide}r, \\ \textsc{\small meteor}, \textsc{\small sim} \end{tabular} \\
\hline
\textsc{\small SFHotel} & \begin{tabular}[c]{@{}l@{}}raw \\ data\end{tabular} & \begin{tabular}[c]{@{}l@{}}\textsc{\small ter}, \textsc{\small bleu}{\small 1-4}, \\ \textsc{\small rouge}, \textsc{\small lepor}, \\ \textsc{\small cide}r, \textsc{\small meteor}, \\  \textsc{\small sim} \end{tabular} & \textsc{\small meteor} & N/A \\
\hline
\multirow{2}{*}{\textsc{\small SFRest}} & \begin{tabular}[c]{@{}l@{}}raw \\ data\end{tabular} &  \textsc{\small sim} & \textsc{\small lepor} & N/A \\
\cdashline{2-5}[1pt/3pt]
 & \begin{tabular}[c]{@{}l@{}}quant. \\ data\end{tabular} & \begin{tabular}[c]{@{}l@{}}\textsc{\small ter}, \textsc{\small bleu}{\small 1-4}, \\ \textsc{\small rouge}, \textsc{\small nist}, \\ \textsc{\small lepor}, \textsc{\small cide}r, \\ \textsc{\small meteor} \\ \textsc{\small sim} \end{tabular} & N/A & N/A
\end{tabular}
\end{adjustbox}
\caption{Metrics predicting relative human rating with significantly higher accuracy than a random baseline.}
\label{tab:accuracy}
\end{table}

\noindent {\bf Discussion:}
 Our data differs from the one used in previous work \cite{cider,erdem:eacl2017}, which uses
explicit relative rankings 
(``{\em Which output do you prefer?}"), whereas we compare two Likert-scale ratings. As such, we have 3 possible outcomes (allowing ties).
This way, we can account for equally valid system outputs, which is one of the main drawbacks of forced-choice approaches \cite{hodosh2016focused}.
Our results are akin to previous work:
\citet{erdem:eacl2017} report results between 60-74\% accuracy for binary classification on machine-machine data, which is comparable to our results for 3-way classification.

Still, we observe a mismatch between the ordinal human ratings and the continuous metrics. For example, humans might rate system A and system B both as a 6, whereas \textsc{\small bleu}, for example, might assign 0.98 and 1.0 respectively, meaning that \textsc{\small bleu}{} will declare system B as the winner.
In order to account for this mismatch, we quantise our metric data to the same scale as the median scores from our human ratings.\footnote{Note that this mismatch can also be accounted for by continuous rating scales, as suggested by \citet{Belz:ACL2011}.} Applied to \textsc{\small SFRest}, where we previously got our worst results, we can see an improvement for predicting {\em informativeness}, where all WBMs now perform significantly better than the random baseline (see Table~\ref{tab:accuracy}).
In the future, we will investigate related discriminative approaches, e.g.\  \cite{hodosh2016focused,vinyals:gans}, where the task is simplified to distinguishing correct from incorrect output.
 
\section{Error Analysis}\label{sec:error} 

In this section, we attempt to uncover why automatic metrics perform so poorly.

\subsection{Scales} 

We first explore the hypothesis that metrics are good in distinguishing extreme cases, i.e.\ system outputs which are rated as clearly good or bad by the human judges, but do not perform well for utterances rated in the middle of the Likert scale, as suggested by \citet{erdem:eacl2017}.  
 We  `bin' our data into three groups: {\em bad}, which comprises low ratings ($\leq$2); {\em good}, comprising  high ratings ($\geq$5); and finally a group comprising {\em average} ratings. 
 
We find that utterances with low human ratings of {\em informativeness} and {\em naturalness} correlate significantly better ($p<0.05$) with automatic metrics than those with average and good human ratings. For example,  as shown in Figure~\ref{fig:corr-bins}, the correlation between WBMs and human ratings for utterances with low {\em informativeness} scores ranges between $0.3 \le \rho \le 0.5$ (moderate correlation), while the highest correlation for utterances of average and high informativeness barely reaches $\rho \le 0.2$ (very weak correlation).
The same pattern can be observed for correlations with {\em quality} and {\em naturalness} ratings. 

This discrepancy in correlation results between low and other user ratings, together with the fact that the majority of system outputs are rated
``good" for informativeness (79\%), naturalness (64\%) and quality (58\%), whereas low ratings do not exceed 7\% in total, 
could explain why the overall correlations are low (Section \ref{sec:correlation})
despite the observed trends in relationship between average system-level performance scores (Section \ref{sec:evaluation}).
It also explains why the \textsc{rnnlg}{} system, which
contains very few instances of low user ratings, shows poor correlation between human ratings and automatic metrics. 

\begin{figure}[h]
\centering
\includegraphics[width=0.3\textwidth]{images/corr_bins3.png}
\caption{Correlation between automatic metrics (WBMs) and human ratings for utterances of bad informativeness (top), and average and good informativeness (bottom).  } 
\label{fig:corr-bins}
\vspace{-0.5cm}
\end{figure}

 
 \begin{table*}[htp]
\small
\centering
\begin{adjustbox}{max width=1\textwidth}
\begin{tabular}{c|c|p{3.7cm}|p{3.5cm}|p{3.7cm}|c|c|c }
\hspace{-1mm}\bf No.\hspace{-1mm} & \bf system & \multicolumn{1}{c|}{\bf MR} &  \multicolumn{1}{c|}{\bf system output} &  \multicolumn{1}{c|}{\bf human reference} & \bf WOMs   & \bf \textsc{\small sim} & \hspace{-1mm}\bf humans\Tstrut\hspace{-1mm} \\
\hline \hline
1& \textsc{lols}}%{Lampouras \& Vlachos  & \raggedright\hangindent=0.5em\hangafter=1 inform(name = the
donatello, hasinternet = yes) & well
there is a hotel with internet access called the donatello &  the donatello has internet & 1.4	&5  & 6\Tstrut \\
\hline
2& \textsc{lols}}%{Lampouras \& Vlachos  & \raggedright\hangindent=0.5em\hangafter=1 inform\_nomatch(area = embarcadero, kidsallowed\hspace{-0.2em}~=\hspace{-0.2em} yes, pricerange\hspace{-0.1em}~=\hspace{-0.1em}~expensive)\hspace{-0.2em} & i but i but i but i
but i but i but i but i but i but i & unfortunately i could not find any expensive restaurants in embarcadero that allow kids. & 1.1 & 1& 1\Tstrut \\
\hline
3& \textsc{\small TGen}}%{Du\v{s}ek{} & \raggedright\hangindent=0.5em\hangafter=1 inform(name = X, area
= riverside, eattype = restaurant, food = fastfood,
pricerange = cheap) & x is a restaurant on the
riverside called located at the riverside and at is   & x is a cheap fastfood
restaurant located near the riverside & 2.4 & 4& 1 \Tstrut\\
\hline
4& \textsc{rnnlg}{}  & \raggedright\hangindent=0.5em\hangafter=1 inform\_nomatch(kidsallowed = yes, food = moroccan) & i am sorry, i did
not find any restaurants that allows kids and serve
moroccan. & sorry,
there are no restaurants allowing kids and serving
moroccan food & 1.85  & 4 & 5 \Tstrut\\
\end{tabular}
\end{adjustbox}
\vspace{-0.1cm}
\caption
{Example pairs of MRs and system outputs from our data, contrasting the average of word-overlap metrics (normalised in the 1-6 range) and semantic similarity (\textsc{\small sim}) with human ratings (median of all measures).}
\label{tab:examples}
\vspace{-0.2cm}
\end{table*}%

\subsection{Impact of Target Data}
\label{ssec:impactData}

\noindent {\bf Characteristics of Data: 
In Section \ref{ssec:correlation}, we observed that datasets have a significant impact on how well automatic metrics reflect human ratings. A closer inspection shows that \textsc{\small Bagel}{} data differs significantly from \textsc{\small SFRest}{} and \textsc{\small SFHotel}, both in terms of grammatical and MR properties.
\textsc{\small Bagel}{} has significantly shorter references both in terms of number of characters and words compared to the other two datasets.
Although being shorter, the words in \textsc{\small Bagel}{} references are significantly more often polysyllabic.
Furthermore,
\textsc{\small Bagel}{} only consists of utterances generated from \emph{inform} MRs, while \textsc{\small SFRest}{} and \textsc{\small SFHotel}{} also have less complex MR types, such as {\em confirm}, {\em goodbye}, etc.
Utterances produced from \emph{inform} MRs are significantly longer and have a significantly higher correlation with human ratings of \emph{informativeness} and \emph{naturalness} than \emph{non-inform} utterance types. 
In other words, \textsc{\small Bagel}{} is the most complex dataset to generate from. 
Even though it is more complex, metrics perform most reliably on \textsc{\small Bagel}{} here (note that the correlation is still only weak).
One possible explanation
 is that \textsc{\small Bagel}{} only contains two human references per MR, whereas \textsc{\small SFHotel}{} and \textsc{\small SFRest}{} both contain 5.35 references per MR on average.
Having more references means that WBMs naturally will return higher scores (`anything goes'). This problem could possibly be solved by weighting multiple references according to their quality, as suggested by \cite{deltableu}, or following a reference-less approach \cite{specia:MT2010}.


 \noindent {\bf Quality of Data:}
 Our corpora contain crowdsourced human references that have grammatical errors, e.g.\ ``{\em Fifth Floor does not allow childs}'' (\textsc{\small SFRest}{} reference).
 Corpus-based methods may pick up these errors, and word-based metrics will rate these system utterances as correct, whereas we can expect human judges to be sensitive to ungrammatical utterances. Note that the parsing score (while being a crude approximation of grammaticality) achieves one of our highest correlation results against human ratings, with  $|\rho|=.31$.
 Grammatical errors raise questions about the quality of the training data, especially when being crowdsourced.
 For example, \citet{Belz:EACL06} find that human experts assign low rankings to their original corpus text. Again,  weighting \cite{deltableu} or reference-less approaches \cite{specia:MT2010} might remedy this issue.
 
 \ignore{
 Following this approach, we asked crowdworkers to rate
200 randomly sampled instances from each corpus, following the procedure outlined in Section \ref{sec:datacollection}. However, we find no significant correlation between rankings of the original references and the output generated by the systems. This suggests that data-driven approaches can cancel out noise.
}
 
 
\begin{table*}[htp!]
\centering
\begin{adjustbox}{max width=1\textwidth}
\begin{tabular}{c|c|c|c}
& \multicolumn{2}{c|}{\bf Dimension of human ratings} & \\
{\bf Study} & {\bf Sentence Planning}  & {\bf Surface Realisation} & {\bf Domain} \\
\hline \hline
this paper  &  weak positive ($\rho=0.33$, WPS) & weak negative ($\rho=0.-31$, parser)& NLG, restaurant/hotel search \Tstrut\\
 
 \hline
\cite{belz:CL2009} &  none & strong positive (Pearson's $r=0.96$, \textsc{\small nist}) & NLG, weather forecast \\
 
  \hline
  
\cite{stent2005evaluating}  &  weak positive ($\rho=0.47$, \textsc{\small LSA}) & negative ($\rho=-0.56$, \textsc{\small nist})& paraphrasing of news\\
   \hline
\cite{Liu:EMNLP2016}  & weak positive ($\rho=0.35$, \textsc{\small bleu}{\small -4}) &  N/A & dialogue/Twitter pairs\\

\hline
\cite{keller:acl2014}  &  positive ($\rho=0.53$, \textsc{\small meteor}) & N/A & image caption  \\

 \hline
\cite{erdem:eacl2017}   &  positive ($\rho=0.64$, \textsc{\small spice}) &  N/A & image caption\\

  \hline
\cite{cahill:acl2009} & N/A & negative ($\rho=-0.64$, \textsc{\small rouge}) & NLG, German news texts \\

\hline
\cite{espinosa_further_2010} & weak positive ($\rho=0.43$, \textsc{\small ter}) & positive ($\rho=0.62$, \textsc{\small bleu}{\small -4}) & NLG, news texts \\

\end{tabular}
\end{adjustbox}
\vspace{-0.1cm}
\caption{Best correlation results achieved by our and previous work. 
Dimensions targeted towards  Sentence Planning include `accuracy', `adequacy', `correctness', `informativeness'. Dimensions for Surface Realisation include `clarity', `fluency', `naturalness'.
}\label{tab:previous}
\vspace{-0.2cm}
\end{table*}%

\subsection{Example-based Analysis}
\label{ssec:lingExmpl}


As shown in previous sections, word-based metrics moderately agree with humans on bad  quality output, but cannot distinguish output of good or medium quality. Table \ref{tab:examples} provides  examples from our three systems.\footnote{Please note that WBMs 
tend to match against the reference that is closest to the generated output. Therefore, we only include the closest match in Table \ref{tab:examples} for simplicity.} Again, we observe different behaviour between WOMs and \textsc{\small sim}{} scores. 
In Example 1, \textsc{lols}}%{Lampouras \& Vlachos {} generates a grammatically correct English sentence, which represents the meaning of the MR well, and, as a result, this utterance received high human ratings (median = 6) for \textit{informativeness, naturalness} and \textit{quality}.
However, WOMs rate this utterance low, i.e.\ scores of \textsc{\small bleu}{}{\small 1-4}, \textsc{\small nist}, \textsc{\small lepor}, \textsc{\small cide}r, \textsc{\small rouge}{} and \textsc{\small meteor}{} normalised into the 1-6 range all stay below 1.5. This is because the system-generated utterance
has low overlap with the human/corpus references.  
Note that the \textsc{\small sim}{} score is high (5), as it ignores human references and
computes distributional semantic similarity between the MR and the system output.
Examples~2 and~3 show outputs which 
receive low scores from both automatic metrics and humans.
WOMs score these system outputs low due to little or no overlap  with human references, whereas humans are sensitive to ungrammatical output and missing information (the former is partially captured by GBMs). Examples~2 and~3 also illustrate inconsistencies in human ratings since system output~2 is clearly worse than {output~3 and both are rated by human with a median score of 1. 
Example~4 shows an output of the \textsc{rnnlg}{} system which is semantically very similar to the reference (\textsc{\small sim}=4) and rated high by humans, but WOMs fail to capture this similarity.
GBMs show more accurate results for this utterance, with mean of readability scores 4 and parsing score 3.5.

\section{Related Work}\label{sec:related}

Table \ref{tab:previous} summarises results published by previous studies in related fields which investigate the relation between human scores and automatic metrics.
These studies mainly considered WBMs, 
while we are the first study to consider GBMs. Some studies ask users to provide separate ratings for surface realisation (e.g.\ asking about `clarity' or `fluency'), whereas other studies focus only on sentence planning  (e.g.\ `accuracy', `adequacy', or `correctness').
In general, correlations reported by previous work range from weak to strong. 
The results confirm that metrics can be reliable indicators at system-level \cite{belz:CL2009}, while they perform less reliably at sentence-level \cite{stent2005evaluating}. Also, the results show that the metrics capture realization better than sentence planning.
There is a general trend showing that best-performing metrics tend to be the more complex ones, combining word-overlap, semantic similarity and term frequency weighting. Note, however, that the majority of previous works do not report whether any of the metric correlations are significantly different from each other.

\section{Conclusions}\label{sec:discussion}

This paper shows that state-of-the-art automatic evaluation metrics for NLG systems do not sufficiently reflect human ratings, which stresses the need for human evaluations. This result is opposed to the current trend of relying on automatic evaluation identified in \cite{gkatzia:enlg2015}.

A detailed error analysis suggests that automatic metrics are particularly weak in distinguishing  outputs of medium and good quality, which can be partially attributed to the fact that human judgements and metrics are given on different scales.
We also show that metric performance is data- and system-specific.

Nevertheless, our results also suggest that 
automatic metrics can be useful 
for error analysis by helping to find cases where the system is performing poorly. In addition, we find reliable results on system-level, which suggests that metrics can be useful for system development.%


\section{Future Directions}\label{sec:future}

Word-based metrics make two strong assumptions: They treat human-generated references as a gold standard, which is \textit{correct} and \textit{complete}. 
We argue that these assumptions are invalid for corpus-based NLG, especially when using crowdsourced datasets. 
Grammar-based metrics, on the other hand, do not rely on human-generated references and are not influenced by their quality. However, these metrics can be easily manipulated with grammatically correct and easily readable output that is unrelated to the input. We have experimented with combining WBMs and GBMs using ensemble-based learning. However, while our model achieved high correlation with humans within a single domain, its cross-domain performance is insufficient.

Our paper clearly demonstrates the need for
more advanced metrics, as used in related fields, 
 including:  
assessing output quality within the dialogue context, e.g.\ \cite{duvsek-jurcicek:2016:SIGDIAL}; extrinsic evaluation metrics, such as NLG's contribution to task success, e.g.\ \cite{Rieser:IEEE14,Gkatzia:acl16,hastie:iwsds2016}; 
building discriminative models, e.g.\ \cite{hodosh2016focused}, \cite{vinyals:gans}; or reference-less quality prediction as used in MT, e.g.\ \cite{specia:MT2010}.
We see our paper as a first step towards reference-less evaluation for NLG by introducing grammar-based metrics. In current work
 \cite{dusek:icml2017}, we investigate a reference-less quality estimation approach based on recurrent neural networks, which predicts a quality score for a NLG system output by comparing it to the source meaning representation only.
 
 

Finally, note that the datasets considered in this study are fairly small (between  404 and 2.3k human references per domain). To remedy this, systems train on de-lexicalised versions \cite{wen:emnlp2015}, which bears the danger of ungrammatical lexicalisation \cite{SharmaHSSB16} and a possible overlap between testing and training set \cite{vlachos:coling2016}. There are ongoing efforts to release larger and more diverse data sets, e.g.\ \cite{novikova:INLG2016, novikova2017e2e}.

\section*{Acknowledgements}
This research received funding from the EPSRC projects  DILiGENt (EP/M005429/1) and  MaDrIgAL (EP/N017536/1). The Titan Xp used for this research was donated by the NVIDIA Corporation.


