\section{Appendix: Ada Answers}
\label{sec:adareplies}
We present some of the replies from the Ada model on various prompts in \cref{tab:adareplies} to illustrate the frequent, invalid answers of this model. Similar behaviour was observed when querying GPT~2. Both models failed with respect to our translation quality assessment experiment.

\newcommand{\specialcell}[2][c]{%
  \begin{tabular}[#1]{@{}l@{}}#2\end{tabular}}

\begin{table*}[h]
\centering
\begin{tabular}{ll}
\toprule
DA & \specialcell{0 = no meaning preserved\\1 = perfect meaning and grammar\\2 = excellent meaning\\3 = very good meaning\\4 = excellent\\5 = very good\\6 = excellent\\7 = excellent\\8 = excellent\\9 = excellent\\10 = perfect} \\
\midrule
DA & \specialcell{No meaning preserved: 0 \\ Perfect meaning and grammar: 100} \\
\midrule
DA & \specialcell{0 – It does not exist in my language \\ 1 – It does not exist in my text \\2 – It does not exist in my text \\3 – It exists in my text \\4 – It exists in my text\\5 – It exists in my text\\6 – It exists in my text\\7 – It exists in my text\\8 – It exists in my text\\9 – It exists in my text\\10 – It exists in my text} \\
\midrule
DA & This sentence does not have a score.\\
\midrule
DA[noref]  & \specialcell{The source for this translation is English. The translation is 1 out of 100.} \\
\midrule
DA[noref]  & \specialcell{0 = no meaning preserved\\1 = perfect meaning and grammar\\2 = perfect meaning} \\
\midrule
SQM  & \specialcell{Some meaning preserved\\Most meaning preserved and few grammar mistakes} \\
\midrule
SQM[noref]  & \specialcell{No meaning preserved\\Some meaning preserved\\Most meaning preserved and few grammar mistakes} \\

\bottomrule
\end{tabular}
\caption{Answers by the Ada model for various prompts. We observe that SQM prompts are closer to expected outputs than answers to the corresponding DA prompts. Similar behaviour was observed when querying GPT~2.}
\label{tab:adareplies}
\end{table*}
\section{Appendix: Full Results}
\label{sec:fullresults}

Below table lists all GEMBA results we have obtained for this work. Any missing segment-level scores are due to a subset of segments for which we could not obtain a score even after adding randomness.

\begin{table*}[hbt!]
\small
\centering
\input{tables/wmt22_table11.tex}
\label{tab:all_results}
\caption{System-level pairwise accuracy and segment-level Kendall Tau scores.}
\end{table*}


\section{Appendix: Prompt Templates}
\label{sec:prompttemplates}
Below we provide our prompt templates which we use for the experiments described in this paper. Template \textbf{portions in bold face} are used only when a human reference translation is available.

\subsection{DA: Direct Assessment}
Output scores range from $0-100$.\\
 {\footnotesize
    \begin{Verbatim}[commandchars=+\[\]]
    Score the following translation from {source_lang} to {target_lang} +textbf[with respect to]
    +textbf[the human reference] on a continuous scale from 0 to 100, where a score of zero means
    "no meaning preserved" and score of one hundred means "perfect meaning and grammar".

    {source_lang} source: "{source_seg}"
    +textbf[{target_lang} human reference: {reference_seg}]
    {target_lang} translation: "{target_seg}"
    Score:
    \end{Verbatim}
}
\subsection{SQM: Scalar Quality Metrics}
Output scores range from $0-100$.\\
 {\footnotesize
    \begin{Verbatim}[commandchars=+\[\]]
    Score the following translation from {source_lang} to
    {target_lang} +textbf[with respect to the human reference] on a continuous
    scale from 0 to 100 that starts with "No meaning preserved", goes
    through "Some meaning preserved", then "Most meaning preserved and
    few grammar mistakes", up to "Perfect meaning and grammar".

    {source_lang} source: "{source_seg}"
    +textbf[{target_lang} human reference: "{reference_seg}"]
    {target_lang} translation: "{target_seg}"
    Score (0-100):
    \end{Verbatim}
}
\subsection{Stars: One to Five Stars Ranking}
Output scores range from $1-5$. Special care is taken for answers containing non-numerical answers, such as "Three stars", "****", or "1 star".\\
 {\footnotesize
    \begin{Verbatim}[commandchars=+\[\]]
    Score the following translation from {source_lang} to {target_lang} 
    +textbf[with respect to the human reference] with one to five stars. 
    Where one star means "Nonsense/No meaning preserved", 
    two stars mean "Some meaning preserved, but not understandable", 
    three stars mean "Some meaning preserved and understandable", 
    four stars mean "Most meaning preserved with possibly few grammar mistakes", 
    and five stars mean "Perfect meaning and grammar".
    
    {source_lang} source: "{source_seg}"
    +textbf[{target_lang} human reference: "{reference_seg}"]
    {target_lang} translation: "{target_seg}"
    Stars:
    \end{Verbatim}
}
\subsection{Classes: Quality Class Labels}
Output label one of {\tt "No meaning
    preserved"}, {\tt "Some meaning preserved, but not understandable"}, {\tt "Some meaning preserved and understandable"}, {\tt "Most meaning preserved, minor issues"}, {\tt "Perfect translation"}.\\
 {\footnotesize
    \begin{Verbatim}[commandchars=+\[\]]
    Classify the quality of translation from {source_lang} to {target_lang}
    +textbf[with respect to the human reference] into one of following classes: "No meaning
    preserved", "Some meaning preserved, but not understandable", "Some meaning
    preserved and understandable", "Most meaning preserved, minor issues", "Perfect
    translation".

   {source_lang} source: "{source_seg}"
   +textbf[{target_lang} human reference: "{reference_seg}"]
   {target_lang} translation: "{target_seg}"
   Class:
    \end{Verbatim}
}

\noindent Templates available from: {\small \url{https://github.com/MicrosoftTranslator/GEMBA/gemba/prompt.py}}
\section{Introduction}
One of the interesting properties of large language models (LLMs) such as GPT~\cite{GPT} is their (implicit) support for multilingual Q\&A. Prompting the model in the right way allows us to translate text between languages \parcite{vilar2022prompting}. This is surprising as GPT has not been fine-tuned for the translation task.

\perscite{hendy2023good} show that GPT-enabled translation achieves high quality when applied for the translation of high-resource languages, but still lacks in terms of translation quality for under-represented languages.
Building on this finding---\emph{if the model can translate, it may be able to differentiate good from bad translations}---we apply GPT for the task of translation quality assessment. 

In the remainder of this paper, inspired by recent progress on generative, pre-trained large language models (LLMs), we explore how these models can be applied for automated assessment of translation quality. The main research question for this work lies in the question: \emph{Can LLMs be used for effective quality assessment of translations?}

We propose GEMBA, which stands for \emph{GPT Estimation Metric Based Assessment}. The metric evaluates each segment translation in isolation and then averages across all obtained scores for the final, system-level score.

We define and evaluate several prompt variants for zero-shot assessment of translation quality in two modes, either with a human reference translation, as a quality metric, or without one, as a quality estimation task.

We design the main prompts based on the DA+SQM template used for human assessment of translation quality as implemented in the Appraise framework~\cite{federmann-2018-appraise} for WMT22~\cite{kocmi-EtAl:2022:WMT}, building on previous work conducted by \citet{freitag-etal-2021-experts}. 


The main contributions of this paper are:

\begin{itemize}
    \item[-] We demonstrate state-of-the-art capabilities of GPT-based translation quality assessment on the latest WMT22 metrics evaluation data (on the system level);
    \item[-] We experiment with four prompt templates, showing that the least constrained template achieves best performance;
    \item[-] We evaluate seven different models of GPT, showing that only GPT~3.5 and larger models are capable of translation quality assessment;
    \item[-] We show that GEMBA is not yet reliable enough on the segment level and, thus, should be applied for system-level evaluation only.
\end{itemize}

\section{The GEMBA Metric}
To assess translation quality via prompting an LLM, the following parameters are needed:

\begin{itemize}
\item[-] prompt variant (from a pre-defined set)\\[-2em]
\item[-] source language name, e.g., ``Chinese''\\[-2em]
\item[-] target language name, e.g., ``English''\\[-2em]
\item[-] source segments $src_{1..N}$\\[-2em]
\item[-] candidate translations $hyp_{1..N}$\\[-2em]
\item[-] optionally, reference translations $ref_{1..N}$
\end{itemize}
We generate a GPT request for every segment, querying as individual zero-shot problems, and then aggregate results. For this initial proof of concept, we leave improvements such as few-shot queries or document-level context to future work.

\subsection{Prompt variants}
\begin{figure*}[htb]
\input{prompt_da}
\caption{The best-performing prompt based on Direct Assessment expecting a score between 0--100. Template \textbf{portions in bold face} are used only when a human reference translation is available.}
\label{fig:prompt_example}
\end{figure*}

We experiment with four distinct prompt types: modeling two scoring and two classification tasks. For the scoring tasks, first, one based on \textbf{direct assessment} (\emph{GEMBA-DA}), second, another based on recent research efforts on \textbf{scalar quality metrics} (\emph{GEMBA-SQM}).\footnote{Although names are based on existing techniques for human assessment, they do not match perfectly.} 
As scoring of translation quality may be an unnatural task for an LLM, we also design two classification tasks. The first is based on \textbf{one-to-five stars ranking} (\emph{GEMBA-stars}), which is a style often used when users are asked to review various services or products. The second prompt asks the LLM to label translation quality as one of five discrete \textbf{quality classes} (\emph{GEMBA-classes}).

For each of these four prompt types, we experiment with two modes that differ with respect to the wording of the corresponding query templates which either have access to a human reference or not. 
As an example, we show the \emph{GEMBA-DA}{} prompt in \cref{fig:prompt_example}. Based on token count, this is the least constrained prompt template that we experiment with. The complete set of prompt templates is available in \cref{sec:prompttemplates}.
For naming convention, we mark quality estimation metrics (without reference) with the suffix "[noref]".

\subsection{Scoring process}
The expected scores are in $[0, 100]$ for \emph{GEMBA-DA}{} and \emph{GEMBA-SQM}{} prompts, same as for human assessment \parcite{graham-etal-2013-continuous}; for \emph{GEMBA-stars}{} the output ranges from $[1, 5]$ and \emph{GEMBA-classes}{} assigns one of five class labels.

We average segment-level scores to obtain system-level scores. For the \emph{GEMBA-classes}{} metric variant, we assign classes a numerical value $[0-4]$, based on the label, before averaging.

Depending on the GPT model we query, sometimes answers are returned outside these ranges, as text. When we observe such an \emph{invalid} answer, we add randomness and sample more responses, selecting the first answer matching the output range as the final result.



\subsection{GPT models}

We experiment with seven GPT models---\emph{ranging from GPT~2 up to the latest ChatGPT model}---that are described in \cref{tab:gpt_models}.\footnote{\scriptsize\url{https://learn.microsoft.com/en-us/azure/cognitive-services/openai/concepts/models}}
We use the Davinci-003 model as the default model for most experiments and compare the performance of other models in \cref{sec:various_models}.
Specifically, we use these models:

\begin{description}
\item[GPT~2] We use models provided by \citet{radford2019language}, assessing if GPT~2 may be useful for quality assessment---\emph{we find that it is not};\\[-7mm]

\item[Ada] GPT~3. Max request size of 2,048 tokens and training data up to October 2019;\\[-7mm]

\item[Babbage] GPT~3. More capable than Ada;\\[-7mm]

\item[Curie] GPT~3. More capable than Babbage;\\[-7mm]

\item[Davinci-002] GPT~3.5. Max request size of 4,000 tokens and training data up to June 2021. Uses \texttt{\small FeedME} training;\\[-7mm]

\item[Davinci-003] GPT~3.5.1. Uses \texttt{\small PPO} training;\\[-7mm]

\item[ChatGPT] Improved GPT~3.5 model, fine-tuned using Reinforcement Learning from Human Feedback (RLHF).
\end{description}

GPT~3 models are based on \citet{InstructGPT}. Due to API request limits not all combinations of prompt variants and GPT models may have been evaluated for this work.

\begin{table}[t]
\centering
\scriptsize
\begin{tabular}{llll}
\toprule
Model name & Abbrev. & Details \\
\midrule
GPT-2       & ---   & See \citet{radford2019language}; does not work\\
Ada         & ---   & We use \texttt{text-ada-001}; does not work\\
Babbage     & Bab   & We use \texttt{text-babbage-001}  \\
Curie       & Curie & We use \texttt{text-curie-001}  \\
Davinci-002 & Dav2  & We use \texttt{text-davinci-002} \\
Davinci-003 & Dav3  & We use \texttt{text-davinci-003}\\
ChatGPT     & Chat  & We use \texttt{text-chat-davinci-002}\\
\bottomrule
\end{tabular}
\caption{Details of all models used in this work. Models are sorted from oldest to newest which also reflects their respective power. GPT~2 and Ada do not work.}
\label{tab:gpt_models}
\end{table}

\section{Experiments}
To measure the performance of the proposed GEMBA metric, we follow the methodology and use test data provided by the WMT22 Metrics shared task \parcite{freitag-EtAl:2022:WMT} which hosts an annual evaluation of automatic metrics, benchmarking them against human gold labels. Effectively, we compare GEMBA against the best-performing automatic metrics: COMET \parcite{rei-etal-2020-comet, rei-EtAl:2022:WMT1}, BLEURT \parcite{sellam-etal-2020-bleurt}, or the non-public winner MetricX XXL.

\subsection{Test set}
We use the MQM 2022 test set which contains human judgments for the following three translation directions: English into German, English into Russian, and Chinese into English. The test set contains a total of 54 machine translation system outputs or human translations. It contains a total of 106k segments. Translation systems are mainly from participants of the WMT22 General MT shared task \parcite{kocmi-EtAl:2022:WMT}.

The source segments and human reference translations for each language pair contain around 2,000 sentences from four different texts domains: news, social, conversational, and e-commerce. 
The gold standard for scoring translation quality is based on human MQM ratings, annotated by professionals who mark individual errors in each translation, as described in \perscite{freitag-etal-2021-experts}.

\subsection{Evaluation methods}
To determine how well automatic metrics correlate with humans, we measure system-level, pairwise accuracy (\emph{accuracy}, \inparcite{kocmi-etal-2021-ship}). For segment-level evaluation, we use Kendall's Tau ($\tau$, \inparcite{freitag-etal-2022-results}). 

Here, accuracy is defined as the number of system pairs ranked correctly by the metric with respect to the human ranking divided by the total number of system pair comparisons.

Formally:

\begin{center}
\footnotesize
\[\mbox{Accuracy} = \frac{|\mbox{sign}(\mbox{metric} \Delta)~=~\mbox{sign}(\mbox{human} \Delta)|}{|\mbox{all system pairs}|}\]
\end{center}

The variant of Kendall's Tau used for metric evaluation changed over the years. Initially, \perscite{callison-burch-etal-2011-findings} proposed to use Kendall's Tau-a ignoring human rankings that tied, while penalising ties in automatic metrics.

\begin{center}
\footnotesize
\[\tau = \frac{\mbox{Concordant} - \mbox{Discordant}}{\mbox{Concordant} + \mbox{Discordant}}\]
\end{center}

where Concordant is the set of all human segment comparisons for which a given metric suggests the same order of systems and Discordant is the set of all human comparisons for which a given metric disagrees. 

\begin{center}
\footnotesize
\begin{tabular}{ll|ccc}
 & &   \multicolumn{3}{c}{Metric} \\
 & &     $s_1 < s_2$ & $s_1 = s_2$ & $s_1 > s_2$ \\
\midrule
\multirow{3}{*}{\rotatebox[origin=c]{90}{Human}} & $s_1 < s_2$ & Conc & Disc & Disc\\
& $s_1 = s_2$                        & -- & -- & -- \\
& $s_1 > s_2$                     & Disc & Disc & Conc \\
\end{tabular}
\vspace{1em}
\end{center}

This definition was later updated by \perscite{machacek-bojar-2014-results}, who handle ties as a separate group in contrast to Concordant and Discordant. 
Metrics shared tasks \perscite{mathur-etal-2020-results} and \perscite{freitag-etal-2021-results} changed this back to the 2011 version. Last year, \perscite{freitag-etal-2022-results} changed it to Kendall's Tau-b, which makes adjustments for ties.
Overall, ties in automatic metrics are rare for non-identical translations but are an issue when a method outputs only a discrete set of scores (as in our case).
Additionally, Kendall's Tau is susceptible to noise in gold pairwise rankings \cite{freitag-etal-2022-results}.





We reproduced all scores reported in the WMT22 Metrics shared task findings paper with the official WMT22 script.\footnote{\scriptsize\url{https://github.com/google-research/mt-metrics-eval}} Reported scores match Table 11 of the WMT22 metrics findings paper \parcite{freitag-EtAl:2022:WMT}.

\section{Results}
We investigate GEMBA's performance for two modes: with a reference translation and without reference translation (in a quality estimation setting).
\cref{tab:allmain} reports pairwise accuracy on the system level, comparing \emph{GEMBA-DA}{} against the best-performing metrics from the WMT22 Metrics shared task~\parcite{freitag-EtAl:2022:WMT}. 

\begin{table}[t]
\footnotesize
\centering
\input{tables/wmt22_table11_accuracy_allmain.tex}
\caption{Results for the system-level pairwise accuracy compared to the current automatic metric. Metrics marked as ``[noref]'' do not use a reference translation.}
\label{tab:allmain}
\end{table}

\subsection{Reference-based}
The results in \cref{tab:allmain} show that our reference-based \textbf{GEMBA-Dav3-DA} metric sets a new state of the art. It outperforms all of the other reference-based metrics from the WMT22 Metrics shared task.
The observed level of metric performance is unexpected, especially considering that human labels used as a gold standard in itself are noisy and therefore an accuracy of 100\% is impossible to obtain for an automatic metric.

\subsection{Quality estimation}
\cref{tab:allmain} shows that our reference-less metric \textbf{GEMBA-Dav3-DA[noref]} achieves the highest performance for the quality estimation mode, and strongly outperforms all of the other reference-less metrics. Moreover, it also outperforms all of the other reference-based metrics, performing only slightly worse than \textbf{GEMBA-Dav3-DA}. Again, the observed level of assessment quality is unexpectedly high, highlighting the potential of using LLMs for translation quality assessment tasks.

\subsection{Comparison of GPT models}
\label{sec:various_models}

We compare the performance of various GPT versions as an automatic metric. \cref{tab:all_models} shows results for all models we have experimented with and all prompt variants tested.

We do not show results for GPT-2 or Ada models. Neither of those have produced replies in the specific scoring range and neither seemed to be producing any meaningful replies. We list a couple of their answers in \cref{sec:adareplies}. Based on our experiments, we conclude that they are not powerful enough to understand the zero-shot prompts.

By contrast, Babbage and Curie models appear to understand what type of answer they should produce, but the quality of their scores seems to be close to random guessing. Thus, both Babbage and Curie are useless for translation quality assessment.

The main performance jump occurs for GPT 3.5 and larger models, i.e., Davinci-002, Davinci-003, and ChatGPT. Each of them achieves highly competitive results for all of the prompt variants we have tested. Interestingly, ChatGPT appears to have the lowest quality among these three models. In addition, ChatGPT often replies with a score followed by an explanation of why it has assigned that score.
One possible reason for this lower level of performance may be in the form of the prompt, which wasn't modified to instruct ChatGPT not to generate an explanation. Therefore, it is possible that different prompts may improve the performance of the ChatGPT model.

Unsurprisingly, the best performance is obtained by the most powerful LLM, Davinci-003. This also confirms the findings of \perscite{hendy2023good} who demonstrated superior translation capabilities with this model over all other GPT variants.

\begin{table}[t]
\scriptsize
\centering
\input{tables/all_models.tex}
\caption{Accuracy of the system-level pairwise accuracy for quality estimation methods for most combinations of prompts and different GPT models. The evaluation is based on three language pairs and MQM human labels.}
\label{tab:all_models}
\end{table}


\subsection{Segment-level performance}

All previous results are reported on the system level. We also investigate how well the GEMBA metric performs on the segment level, with respect to the human gold annotations. We present Kendall's Tau results for each language pair separately in \cref{tab:seglevel} (results for all metrics are in \cref{tab:all_results}).

Reference-based \textbf{GEMBA-Dav3-DA} is slightly behind the top-performing metrics but continues to have a high correlation with human judgment. On the other hand, quality estimation \textbf{GEMBA-Dav3-DA [noref]} has significantly lower segment-level performance in contrast to other top-performing metrics. Still, it outperforms string-based metrics (ChrF and BLEU).

The lower performance of a segment-level correlation could be attributed to Kendall's Tau, which penalises ties. Our metric in contrast to other automatic metrics returns a discrete value between 0--100. There is a high probability that two translations will obtain an equal score.


\begin{table}[t]
\scriptsize
\centering
\input{tables/wmt22_seg_kendal_table11}
\caption{Kendall's Tau ($\tau$) segment-level evaluation. Full results are in \cref{tab:all_results}.}
\label{tab:seglevel}
\end{table}

In order to investigate this further, we collect all answers across all systems and all three language pairs and then calculate the frequency of each distinct answer value.

We can notice several interesting observations in \cref{tab:score_distribution}. The DA reference-based prompt generates only multiples of five, reducing the scale to just 16 distinct values. Over three-quarters of all scores are either score 80, 90, or 95. This could reflect the actual quality of the system translations as the underlying systems are provably high-quality.

When we investigate the ``DA[noref]'', we notice that 79.9\% of all scores are of value "95". Despite this fact, the metric still manages to differentiate the systems from each other and outperform all other quality estimation metrics on the system level.

The SQM-based method uses the scale in a more fine-grained manner, producing several values that are not multiples of five, i.e., 68, 87, 92, and 93.

We conjecture that frequent ties and the discrete scale thus may be the reason behind the lower segment-level performance.

\begin{table}[t]
\scriptsize
\centering
\input{tables/score_count.tex}
\caption{Distribution of all distinct segment-level score values for MQM 2022 for model Davinci-003.}
\label{tab:score_distribution}
\end{table}

\subsection{Failure rate}

As we described earlier, LLMs may answer with an invalid answer, for example with a textual answer instead of a score, mostly explaining its decision. When such a situation happens, we iteratively increase the temperature---\emph{adding randomness to the model}---and take the first answer matching the expected score output range.

This adds non-determinism to our evaluation, therefore we investigate how frequently this phenomenon happens. \cref{tab:failrate} shows the number of invalid answers. For almost all combinations of models and prompts, LLMs understand the prompt and provide answers in a valid range with less than 1\% of the answers being invalid.\footnote{Roughly 1,000 answers equal to 1\% of the total volume.}


\begin{table*}[t]
\footnotesize
\centering
\input{tables/failrate.tex}
\caption{Number of invalid answers (full set size 106,758) that needed to be re-prompted with added randomness.}
\label{tab:failrate}
\end{table*}

Processing answers is straightforward as it is usually a stand-alone number. In some occasions, LLMs give a numerical score and continue with a textual explanation, for such cases, we parse only the first number.
A more complex approach needs to be taken for \emph{GEMBA-stars}{} prompts where the model provides different answers which we parse separately. Here are some examples of two-star answers: "2", "two", "**", "★★", "two stars", or "2 stars". For non-English target languages the answer may be produced in the target language, e.g., "一颗星", or "五".
We have not observed attempts to translate output for other prompts.

\section{Conclusion}
We have presented our work on GEMBA, a GPT-based estimation metric-based assessment method. Comparing our metrics to other automated metrics from the WMT22 Metrics shared task we report state-of-the-art performance on the MQM 2022 test set across three language pairs: English to German, English to Russian, and Chinese to English.

We intend to continue research on the application of GPT models for quality assessment. Further research will focus on the switch to few-shot (as opposed to our current zero-shot methodology) as well as model fine-tuning. Both of which promise to increase GEMBA accuracy. Furthermore, modifying prompts to support MQM error-based evaluation or post-editing efforts may lead to further improvements.



GPT-enhanced evaluation metrics may allow us to make progress with respect to document-level evaluation (due to their ability to use much larger context windows). This could be beneficial as there is little research into document-level metrics \parcite{vernikos-EtAl:2022:WMT}.

\section*{Limitations}
While preliminary results indicate that the GEMBA metric performs very well when compared to other automated metrics evaluated as part of the WMT22 Metrics shared task, it is important to note that these results are based on human labels for {\emph{only three language pairs}}. We expect that the metrics performance may suffer for other language pairs, mainly under-resourced languages similar to \perscite{hendy2023good} who show low translation quality for such languages. In addition, GEMBA's state-of-the-art performance only holds for the system level, while segment-level scores still have room for improvement.
Reported results are indicative of the potential performance LLMs could achieve for the translation quality assessment task in the long run.

\section*{Acknowledgments}
This work would not have been possible without the help and support from our friend and colleague, Olivier Nano, who provided access to GPT models via Microsoft Azure -- {\emph{Merci beaucoup, Olivier!}} 
The authors are also grateful to Matt Post, Vikas Raunak, Shabnam Sadegharmaki, and the Microsoft Translator research team for fruitful discussions and helpful feedback.








