\subsection{Glancing Transformers}

The Glancing Transformer (GLAT, \citealp{qian-etal-2021-glancing}) tackles the \emph{multimodality problem} by encouraging word interdependence through \emph{glancing sampling}. In that, they adaptively select a number of ground truth tokens by comparing them to the model predictions $\hat{e}_1^{\hat{I}}$ and feeding the embeddings of the selected tokens to the decoder during training. Formally, the probability distribution can be described as
\begin{equation}
    p_{\bm{\theta}}(e_1^I | f_1^J) = \prod_{i=1}^I p_{\bm{\theta}}(e_i|\Phi(e_1^I, \hat{e}_1^{\hat{I}}), f_1^J),
    \label{eq.glat}
\end{equation}
where $\Phi(e_1^I, \hat{e}_1^{\hat{I}})$ represents the subset of tokens selected by the glancing sampling strategy. This sampling process can be split into two parts. First, determining the number of sampled tokens $S$ by comparing $e_1^I$ and $\hat{e}_1^{\hat{I}}$, here defined by the function $\Gamma(e_1^I, \hat{e}_1^{\hat{I}})$. Secondly, selecting $S$ tokens from the ground truth that form the final output of the glancing sampling. \citet{qian-etal-2021-glancing} deploy a random function for this purpose which we adopt and define here as the function $\Omega(e_1^I, S)$. Combining all these, the sampling strategy can be formulated as
\begin{equation}
    \Phi(e_1^I, \hat{e}_1^{\hat{I}}) = \Omega(e_1^I, \Gamma(e_1^I, \hat{e}_1^{\hat{I}})),
    \label{eq.glat_sampling}
\end{equation}
with the number of sampled tokens $S$ being determined by $\Gamma(e_1^I, \hat{e}_1^{\hat{I}}) = \lambda \cdot d(e_1^I, \hat{e}_1^{\hat{I}})$ where $d(e_1^I, \hat{e}_1^{\hat{I}})$ is a distance measure between the ground truth $e_1^I$ and the predicted tokens $\hat{e}_1^{\hat{I}}$ while $\lambda$ is a hyperparameter. For this distance measure, there are many choices that can be deployed in theory such as the Levenshtein distance \citep{Levenshtein_SPD66} or, with slight adaptions, one of the Bregman divergences \citep{BanerjeeMDG05}. For their experiments, \citet{qian-etal-2021-glancing} choose the Hamming distance \citep{Hamming1950} defined as $d(e_1^I, \hat{e}_1^{\hat{I}}) = \sum_{i=1}^I e_i \neq \hat{e}_i$ which has the useful property that it takes into account the current prediction quality of the model, causing the number of sampled tokens $S$ to be high initially and decrease over time. In their implementation, they additionally deploy a linearly decreasing schedule with a lower bound i.e.\@\xspace $\lambda = \lambda_s - \lambda_m \cdot \frac{u}{U}$ where $u$ is the current and $U$ is the maximum number of update steps with $\lambda_s = 0.5$ and $\lambda_m = 0.2$ to further control the number of sampled tokens. We adopt their configuration.

\subsection{Connectionist Temporal Classification}

Traditional sequence-to-sequence models output target tokens autoregressively until an end-of-sentence token is encountered and are trained using the standard cross-entropy loss. In a non-autoregressive setting, though, the number of tokens in the output is not known \emph{a priori}. One solution is to train a length prediction module to determine the number of output tokens, but this approach still has drawbacks, as it doesn't properly address token repetition and the predicted length can easily be either too short or too long. The quality of the length prediction module is influenced by properties of the respective language pairs such as alignment patterns, word orders or intrinsic length ratios \citep{wang-etal-2021-length}. These fluctuations in accuracy are especially influential on the final translation quality \citep{wang-etal-2021-length}.

The Connectionist Temporal Classification (\textsc{CTC}\@\xspace, \citealp{GravesFGS06,libovicky-helcl-2018-end}) approach tries to improve this property by setting a large enough output length $\tilde{J}$ and giving the model the option to flexibly adjust the target offsets using an additional \texttt{<blank>} token. Specifically, by marginalizing all possible alignments using dynamic programming, for all aligned sequences $a_1^{\tilde{J}}$ that can be reduced to the target $e_1^I$, the conditional probability corresponds to
\begin{equation}
    \begin{split}
    p_{\bm{\theta}}(e_1^I | f_1^J) & = \sum_{a_1^{\tilde{J}} \in \mathcal{A}(e_1^I)} p_{\bm{\theta}}(a_1^{\tilde{J}}|f_1^J) \\
    & = \sum_{a_1^{\tilde{J}} \in \mathcal{A}(e_1^I)} \prod_{j=1}^{\tilde{J}} p_{\bm{\theta}}(a_j|f_1^J)
    \end{split}
    \label{eq.ctc}
\end{equation}
where $\mathcal{A}$ is a one-to-many map that produces all possible alignments $a_1^{\tilde{J}}$ and $\mathcal{A}^{-1}$ is a many-to-one map that collapses an alignment to recover the target sequence by first removing repeated and then \texttt{<blank>} tokens e.g.\@\xspace 
\begin{equation}
\mathcal{A}^{-1}(\underbrace{\bigstrut[b]a-abb}_{a_1^5}) = \mathcal{A}^{-1}(\underbrace{\bigstrut[b]aa-ab}_{a_1^5}) = \underbrace{\bigstrut[b]aab}_{e_1^3}
\label{eq.ctc_example}
\end{equation}
with $-$ representing the \texttt{<blank>} token for brevity. This approach requires setting an alignment length $\tilde{J}$ that is determined by a scaling factor $s$ commonly chosen to be $2$ -- $3$ times the source length $J$, i.e.~$\tilde{J} = J \cdot s$. In the example of \Cref{eq.ctc_example}, the alignment length is $\tilde{J} = 5$.

\paragraph{Combining with \textsc{GLAT}\@\xspace}
To effectively combine \textsc{CTC}\@\xspace with \textsc{GLAT}\@\xspace, an alignment for the target tokens is required to enable the glancing sampling correspondence with the decoder input tokens. For that, \citet{gu-kong-2021-fully} suggest using the Viterbi aligned tokens  
\begin{equation}
    \hat{a}_1^{\tilde{J}}(f_1^J) = \argmax_{a_1^{\tilde{J}} \in \mathcal{A}(e_1^I)}\ p_{\bm{\theta}}(a_1^{\tilde{J}}|f_1^J)
\end{equation}
which we adopt in our implementation and obtain leveraging the \texttt{best\_alignment} method from the Imputer\footnote{\url{https://github.com/rosinality/imputer-pytorch}} \citep{ChanSH0J20}. Similar to previous works, we use greedy decoding instead of beam search for all our experiments to keep a strict non-autoregressive property.

\subsection{Deep Supervision}
\label{sec.ds}
The Deeply Supervised Layer-wise Prediction-aware Transformer (DSLP, \citealp{huang2021nonautoregressive}) introduces three strategies to address the \emph{multimodality problem}. Namely, Deep Supervision, Layer-Wise Prediction-Awareness, and Mixed Training. In a nutshell, Deep Supervision refers to predicting the output tokens at every decoder layer, computing the layer-wise losses, and aggregating them to form the overall loss used for optimizing the model. Formally, we can maximize the log-likelihood
\begin{equation}
    \mathcal{L} = \frac{1}{L_D} \sum_{l=1}^{L_D} \sum_{i=1}^I \log p_{\bm{\theta}}^{(l)}(e_i^{(l)}| \bm{h}^{(l)}_i),
    \label{eq.deep_supervision}
\end{equation}
where $\bm{h}^{(l)}_i$ is the hidden state of decoder layer $l$ corresponding to position $i$ and $L_D$ is the overall number of decoder layers. Intuitively, this approach loses its value when moving to a deep encoder and shallow decoder \citep{Kasai0PCS21} with $L_D = 1$. 

While Deep Supervision keeps the model parameter count constant and only increases the used memory, the other two proposed techniques increase the number of trained parameters. For this work, we only adopt the Deep Supervision part due to the aforementioned drawback.













\section{Introduction}
Traditional sequence-to-sequence models aim to predict a target sequence $e_1^I=e_1, \ldots, e_i, \ldots, e_{I}$ of length $I$ given an input sequence $f_1^J=f_1, \ldots, f_j, \ldots, f_{J}$ of length $J$. In the autoregressive case, this is done token by token, and the probability distribution for the output at timestep $i$ is conditioned on the source sentence $f_1^J$ but also on the preceding outputs of the model $e_1^{i-1}$, and parameterized by $\bm{\theta}$:

\begin{equation}
    p_{\bm{\theta}}(e_1^I | f_1^J) = \prod_{i=1}^I p_{\bm{\theta}}(e_i|e_1^{i-1}, f_1^J).
    \label{eq.at}
\end{equation}
Even though these types of models are widely deployed, one of the major drawbacks is the inherent left-to-right factorization that requires iterative generation of output tokens, which is not efficiently parallelizable on modern hardware such as GPUs or TPUs. Non-autoregressive translation, on the other hand, assumes conditional independence between output tokens, allowing all tokens to be generated in parallel. Effectively, it removes the dependence on the decoding history for generation: 
\begin{equation}
    p_{\bm{\theta}}(e_1^I | f_1^J) = \prod_{i=1}^I p_{\bm{\theta}}(e_i|f_1^J).
    \label{eq.nat}
\end{equation}
This modeling assumption, despite its computational advantages, tends to be more restrictive and introduces the \emph{multimodality problem} \citep{Gu0XLS18} where consecutive output tokens are repeated or fail to correctly incorporate the preceeding information to form a meaningful translation. Overcoming this problem is one of the key challenges to achieving parity with autoregressive translation on a wide variety of tasks and architectures.

Closest to the presented work is a study by \citet{gu-kong-2021-fully} that also analyzes recent works regarding their effectiveness and successfully combines them to achieve remarkable translation quality. In this work, we additionally address several shortcomings in the literature: 1) standardized \textsc{Bleu}\xspace, \textsc{chrF++}\xspace, and \textsc{Ter}\xspace scores using \texttt{sacreBLEU}\xspace accompanied by open source \texttt{fairseq}\xspace code; 2) more realistic non-autoregressive speed-up expectations, by comparing against faster baselines; 3) a call for clarity in the community regarding data pre-processing and evaluation settings.

\section{Experimental setup}
\label{sec.datasets}
\paragraph{Datasets \& knowledge distillation}
We perform our experiments on two datasets: WMT'14 English$\leftrightarrow$German (\textsc{EN} $\leftrightarrow$\textsc{DE}{}, 4M sentence pairs), and WMT'16 English$\leftrightarrow$Romanian (\textsc{EN} $\leftrightarrow$\textsc{RO}{}, 610K sentence pairs), allowing us to evaluate on $4$ translation directions. As is common practice in the non-autoregressive literature, we train our models on knowledge distilled data (KD, \citealp{kim-rush-2016-sequence}), obtained from Transformer \emph{base} teacher models \citep{VaswaniSPUJGKP17} with a beam size of $5$. We tokenize all data using the Moses tokenizer and also apply the Moses scripts \citep{koehn-etal-2007-moses} for punctuation normalization. Byte-pair encoding (BPE, \citealp{sennrich-etal-2016-neural}) is used with $40,000$ merge operations.

For \lenglish$\leftrightarrow$\lromanian\@\xspace, we use WMT'16 provided scripts to normalize the \textsc{RO}{} side, and to remove diacritics for \lromanian$\rightarrow$\lenglish\@\xspace only. \lenglish$\rightarrow$\lromanian\@\xspace keeps diacritics for producing accurate translations which explains the gap between our numbers and those claimed in previous works \citep{gu-kong-2021-fully,qian-etal-2021-glancing}, who besides computing \textsc{Bleu}\xspace on tokenized text, compared on diacritic-free text. More details are summarized in \Cref{app:dataset_details}.

\paragraph{Models}
In this work, we focused our efforts on four models/techniques for training non-autoregressive transformer (NAT) models: Vanilla NAT (\citealp{Gu0XLS18}), Glancing Transformer (GLAT, \citealp{qian-etal-2021-glancing}), Connectionist Temporal Classification (\textsc{CTC}\@\xspace, \citealp{GravesFGS06,libovicky-helcl-2018-end}), and Deep Supervision (DS, \citealp{huang2021nonautoregressive}), see \Cref{app:related_work,app:methods} for a short overview. For all NAT models, we use learned encoder and decoder positional embeddings, shared word embeddings \citep{press-wolf-2017-using}, no label-smoothing, keep \textsc{Adam}\@\xspace betas and epsilon as defaults with $(0.9, 0.98)$ and $1\mathrm{e}{-8}$ respectively, train for $200$k (\lenglish$\leftrightarrow$\lgerman\@\xspace) / $30$k (\lenglish$\leftrightarrow$\lromanian\@\xspace) update steps with $10$k / $3$k warmup steps using an inverse square root schedule \citep{VaswaniSPUJGKP17} where the $5$ best checkpoints are averaged based on validation \textsc{Bleu}\xspace. For \textsc{CTC}\@\xspace-based models, the source upsampling factor is $2$. More details can be found in our open-sourced training procedures. We intentionally did not fine-tune hyperparameters for each translation direction and instead opted for choices that will most likely transfer across datasets.

For autoregressive transformer (AT) models, we focus on the \emph{base} \citep{VaswaniSPUJGKP17} architecture, as well as a deep encoder ($11$ layers) \& shallow decoder ($2$ layers) variant \citep{Kasai0PCS21}.

\section{A call for clarity}
From our experiments with different algorithms, we noticed a few problems that are currently not properly addressed in the literature. These issues make it harder to do a rigorous comparison, so here, we explain how we addressed them, and give recommendations for future work.

\begin{table*}[t]
\setlength\tabcolsep{4pt}
\centering
\scriptsize
\begin{tabular}{llLLLLLrS[table-format=3.1]@{${}\pm{}$}S[table-format=1.2]rS[table-format=4.1]@{${}\pm{}$}S[table-format=2.1]}
\toprule
 \multicolumn{1}{l}{\multirow{2}{*}{\textbf{Models}}} & \multicolumn{1}{c}{\multirow{2}{*}{$|\bm{\theta}|$}} & 
 \multicolumn{1}{c}{\textbf{WMT'13}} & \multicolumn{2}{c}{\textbf{WMT'14}} & \multicolumn{2}{c}{\textbf{WMT'16}} &
 \multicolumn{1}{c}{\multirow{1}{*}{\textbf{Speed}}} & \multicolumn{2}{c}{\multirow{1}{*}{\textbf{Latency GPU}}} & \multicolumn{1}{c}{\multirow{1}{*}{\textbf{Speed}}} & \multicolumn{2}{c}{\multirow{1}{*}{\textbf{Latency CPU}}} \\
 \multicolumn{1}{c}{} & & \multicolumn{1}{c}{\textbf{\lenglish$\rightarrow$\lgerman\@\xspace}} & \multicolumn{1}{c}{\textbf{\lenglish$\rightarrow$\lgerman\@\xspace}} & \multicolumn{1}{c}{\textbf{\lgerman$\rightarrow$\lenglish\@\xspace}} & \multicolumn{1}{c}{\textbf{\lenglish$\rightarrow$\lromanian\@\xspace}} & \multicolumn{1}{c}{\textbf{\lromanian$\rightarrow$\lenglish\@\xspace}} & \multicolumn{1}{c}{(GPU)} & \multicolumn{2}{c}{(\si{\ms}~/~sentence)} &
 \multicolumn{1}{c}{(CPU)} & 
 \multicolumn{2}{c}{(\si{\ms}~/~sentence)}\\
\midrule
\textbf{Autoregressive} & & & & & & & & \multicolumn{1}{c}{} & & & \multicolumn{1}{c}{} & \\
\midrule

 Transformer \textit{base} & 66M & 25.6 & 26.2 & 30.3 & 23.3 & 32.8 & 0.5$\times$ & 165.8 & 1.3  & 0.3$\times$ & 657.2 & 11.4 \\
 \quad\quad + Beam $5$ (teacher) & 66M & 26.4 & 26.8 & 31.5 & 24.0 & 33.4 & 0.4$\times$ & 212.8 & 8.9  & 0.1$\times$ & 1272.4 & 38.5 \\
 \quad\quad + KD & 66M & 26.2 & 26.5 & 30.6 & 23.8 & 32.6\ssymbol{2}\xspace & 0.5$\times$ & 173.4 & 1.0  & 0.3$\times$ & 646.3 & 8.9 \\
\cdashlinelr{1-13}
 Transformer \textit{base} (11-2) & 63M $\downarrow$ & 25.6\ssymbol{2}\xspace & 26.1\ssymbol{2}\xspace & 30.1\ssymbol{2}\xspace & 23.8 & 32.7\ssymbol{2}\xspace & 0.9$\times$ & 89.1 & 3.2  & 0.4$\times$ & 424.2 & 15.0 \\
 \quad\quad + KD & 63M $\downarrow$ & 26.3 & 26.7 & 30.8 & 23.8\ssymbol{2}\xspace & 32.7\ssymbol{2}\xspace & 0.9$\times$ & 88.1 & 1.5  & 0.5$\times$ & 408.0 & 3.2 \\
 \quad\quad + KD + AA & 62M $\downarrow$ & 26.2\ssymbol{2}\xspace & 26.7\ssymbol{2}\xspace & 30.7\ssymbol{2}\xspace & 23.9\ssymbol{2}\xspace & 32.6\ssymbol{2}\xspace & 1.0$\times$ & 79.4 & 2.2  & 0.5$\times$ & 396.9 & 5.4 \\
 \quad\quad + KD + AA + SL & 62M $\downarrow$ & 26.2\ssymbol{2}\xspace & 26.7\ssymbol{2}\xspace & 30.7 & 23.9\ssymbol{2}\xspace & 32.6 & 1.0$\times$ & 82.7 & 7.5  & 1.0$\times$ & 188.3 & 2.2 \\

\midrule
\textbf{Non-Autoregressive} & & & & & & & & \multicolumn{1}{c}{} & & & \multicolumn{1}{c}{} & \\
\midrule

 Vanilla-NAT & 66M & 21.8 & 21.2 & 26.7 & 19.3 & 27.4 & 6.1$\times$ & 13.1 & 0.3  & 2.1$\times$ & 90.1 & 0.4 \\
 \quad\quad + \textsc{GLAT}\@\xspace & 66M & 24.2 & 24.1 & 28.9 & 21.3 & 29.4 & 6.2$\times$ & 12.8 & 0.4  & 2.1$\times$ & 90.4 & 0.6 \\
\cdashlinelr{1-13}
 \textsc{CTC}\@\xspace & 66M & 25.1 & 25.5 & 29.8 & 23.0 & 32.3 & 6.4$\times$ & 12.4 & 0.4  & 1.5$\times$ & 122.9 & 1.7 \\
 \quad\quad + \textsc{DS}\@\xspace & 66M & 25.3\ssymbol{2}\xspace & 25.6\ssymbol{2}\xspace & 30.0\ssymbol{2}\xspace & 22.8\ssymbol{2}\xspace & 32.7 & 6.3$\times$ & 12.7 & 0.6  & 1.6$\times$ & 121.4 & 0.5 \\
 \quad\quad + \textsc{GLAT}\@\xspace  & 66M & 25.8 & 25.7 & 30.4 & 23.2\ssymbol{2}\xspace & 32.8 & 6.3$\times$ & 12.7 & 0.3  & 1.6$\times$ & 121.2 & 1.2 \\
 \quad\quad + \textsc{GLAT}\@\xspace + \textsc{DS}\@\xspace & 66M & 25.9\ssymbol{2}\xspace & 25.9\ssymbol{2}\xspace & 30.7 & 23.2\ssymbol{2}\xspace & 32.7\ssymbol{2}\xspace & 6.2$\times$ & 12.8 & 0.3  & 1.5$\times$ & 121.8 & 0.4 \\
\cdashlinelr{1-13}
 \textsc{CTC}\@\xspace (11-2) & 63M $\downarrow$ & 25.4 & 25.6 & 30.1 & 23.3 & 32.8 & 6.9$\times$ & 11.5 & 0.2  & 1.6$\times$ & 114.6 & 1.3 \\
\quad\quad + \textsc{GLAT}\@\xspace & 63M $\downarrow$ & 25.7 & 25.8\ssymbol{2}\xspace & 30.7 & 23.5\ssymbol{2}\xspace & 33.1\ssymbol{2}\xspace & 6.8$\times$ & 11.7 & 0.4  & 1.6$\times$ & 114.6 & 1.2 \\
 \quad\quad + \textsc{GLAT}\@\xspace + SL & 63M $\downarrow$ & 25.7\ssymbol{2}\xspace & 25.8\ssymbol{2}\xspace & 30.7 & 23.5\ssymbol{2}\xspace & 33.1\ssymbol{2}\xspace & 6.7$\times$ & 11.9 & 0.2  & 2.6$\times$ & 72.5 & 0.8 \\

\bottomrule
\end{tabular}
\caption{Main results using \texttt{sacreBLEU}\xspace on four different translation directions. Latencies are measured on the WMT'14 \lenglish$\rightarrow$\lgerman\@\xspace test set as the average processing time per sentence using batch size $1$. All AT models require $I$ decoding iterations at inference time while the NAT models only require $1$. WMT'13 \lenglish$\rightarrow$\lgerman\@\xspace serves as validation performance. No re-scoring is used for all results. Statistically \emph{non-significant}\xspace results using paired bootstrap resampling are marked with $\ssymbol{2}\xspace$ for $p \geq 0.05$, see \Cref{sec:stat_tests} for more details. \lenglish$\rightarrow$\lromanian\@\xspace keeps diacritics.}
\label{tb.results_full_bleu}
\end{table*}



\subsection{Evaluation: The return of \texttt{sacreBLEU}\xspace}
\label{sec.broken}
In many works, the method for comparing novel non-autoregressive translation models is \emph{tokenized} \textsc{Bleu}\xspace \citep{papineni-etal-2002-bleu}. However, this can cause incomparable translation quality scores as the reference processing is \emph{user-supplied} and not \emph{metric-internal}. Well known problems with user-supplied processing for \textsc{Bleu}\xspace include different tokenization and normalization schemes applied to the reference across papers that can cause \textsc{Bleu}\xspace deviations of up to 1.8 \textsc{Bleu}\xspace points \citep{post-2018-call}. For the NAT literature, in particular, the basis for comparison has been the distilled data released by \citet{ZhouGN20} where binarization is handled by the individual researchers. One instance for such deviations can be found for the \textsc{Bleu}\xspace computations on WMT'14 \lenglish$\rightarrow$\lgerman\@\xspace of \citet{huang2021nonautoregressive} using processed references instead of the original ones. Applying a vocabulary obtained from the distilled training data for processing the references results in \texttt{<unk>} tokens appearing in the references for the German left (\,\emph{„}\xspace) and right (\emph{“}\,) quotation marks because these tokens have not been generated during the distillation process. However, using such processed references leads to artificially inflated \textsc{Bleu}\xspace scores. We evaluated our models and open-sourced models of previous works using processed references, original references, and \texttt{sacreBLEU}\xspace and observe deviations of up to 1.7 \textsc{Bleu}\xspace points across multiple previous works. This fact, together with arbitrary optimization choices that can not be accredited to the presented method, hinders meaningful progress in the field by making results incomparable across papers without individual re-implementation by subsequent researchers. 

\paragraph{Our approach}
To alleviate these potential pitfalls, we advise the community to return to \texttt{sacreBLEU}\xspace\footnote{\url{https://github.com/mjpost/sacrebleu}} \citep{post-2018-call} for evaluation. We provide standardized \textsc{Bleu}\xspace \citep{papineni-etal-2002-bleu}, \textsc{chrF++}\xspace \citep{popovic-2017-chrf}, and \textsc{Ter}\xspace \citep{snover-etal-2006-study} scores in \Cref{tb.results_full_bleu,tb.results_full_chrf,tb.results_full_ter} for current state-of-the-art non-autoregressive models that are reproducible with our codebase, as well as easy to configure through flags, and can be used as baselines for novel model comparisons. No re-ranking is used for all numbers. See \Cref{app:signatures} for the used \texttt{sacreBLEU}\xspace evaluation signatures. Statistically \emph{non-significant}\xspace results ($p \geq 0.05$) are marked with $\ssymbol{2}\xspace$ for all main results using paired bootstrap resampling \citep{koehn-2004-statistical}. For more information on how we select base and reference systems for these statistical tests, please see \Cref{sec:stat_tests}.

\subsection{Establishing a realistic AT speed baseline}
Traditionally, speed comparisons in the NAT literature compare to weak baselines in terms of achievable autoregressive decoding speed which leads to heavily overestimated speed multipliers for non-autoregressive models which has been recently criticized by \citet{heafield-etal-2021-findings}.

\paragraph{Our approach}
We deploy a much more competitive deep encoder ($11$ layers) \& shallow decoder ($2$ layers) \citep{Kasai0PCS21} autoregressive baseline, with average attention (AA, \citealp{zhang-etal-2018-accelerating}) and shortlists (SL, \citealp{schwenk-etal-2006-continuous,junczys-dowmunt-etal-2018-marian-cost}). It achieves a similar accuracy with comparable number of parameters in comparison to the \emph{base} architecture (see \Cref{tb.results_full_bleu}).

Our latency measurements (mean processing time per sentence over $3$ runs) were collected using a single NVIDIA A100 GPU (Latency GPU) or a single-threaded Intel Xeon Processor (Skylake, IBRS) @ 2.3 GHz (Latency CPU), both with batch size of $1$ which faithfully captures the inference on a deployed neural machine translation model.

\subsection{Establishing an accurate NAT baseline}
Hyperparameters and smaller implementation details are often not properly ablated and only copied between open sourced training instructions, causing ambiguity for researchers on which parts of the proposed approach \emph{actually} make a difference. Such choices include the decoder inputs and many more optimization configurations such as dropout values, initialization procedures, and activation functions. Here, we provide our insights from experimenting with these variants.

\subsubsection{No source embeddings as decoder input}
It is common in NAT models to feed the source embeddings as decoder inputs using either \emph{UniformCopy}, \emph{SoftCopy} \citep{wei-etal-2019-imitation}, or sometimes even more exotic approaches without any details such as an ``attention version using positions'' \citep{qian-etal-2021-glancing}, in this paper referenced as \emph{PositionAttention}. While it seems to be common consensus that using any of these variants benefits the final translation quality of the model, our experiments with these have shown the clear opposite when compared to simply feeding the embedding of a special token such as \texttt{<unk>} (see \Cref{app:src_embeddings}). We confirmed this observation by disabling \emph{PositionAttention} for the open-sourced GLAT training procedure \citep{qian-etal-2021-glancing}, and observed no notable effect on \textsc{Bleu}\xspace. For us, either \emph{UniformCopy} or \emph{SoftCopy} lead to larger accuracy drops.

Interestingly, for a setup without guided decoder input, especially the token embedding as well as the self-attention component in the first decoder layer supposedly contain only little information. We investigate the performance contribution of these in \Cref{app:decoder_components} and observe that 1) the self-attention component in the first decoder layer is not beneficial and 2) solely relying on learned positional embeddings is sufficient for achieving comparable performance. However, we do keep both components in the main results for comparability reasons.

\subsubsection{Influence of dropout}
Dropout is one of the hyperparameters which either prior works did not mention or simply used the default values from previous studies. We performed a sweep for different architectures and language pairs, where we varied the dropout probability from $0$ to $0.5$, with increments of $0.1$ (see \Cref{app:dropout}). We observe that choosing an appropriate dropout rate is important for achieving high translation quality. In particular, for \lenglish$\leftrightarrow$\lgerman\@\xspace, $0.1$ achieves the best validation performance for all models, confirming the choice in prior works. For \lenglish$\leftrightarrow$\lromanian\@\xspace, we find that the value used in prior works, $0.3$, is optimal for non-\textsc{CTC}\@\xspace models, but, for \textsc{CTC}\@\xspace-based approaches, higher values seem to outperform.


\subsubsection{Miscellaneous optimization choices}
Many previous works use the Gaussian Error Linear Unit (GELU, \citealp{hendrycks2016gelu}) activation function, initialize the parameters with the procedure from \textsc{BERT}\@\xspace \citep{devlin-etal-2019-bert}, use length offset prediction, or a larger
\textsc{Adam}\@\xspace \citep{KingmaB14} $\epsilon$ value for optimization. In \Cref{app:adam_eps} we summarize the effect of these choices and find that for Vanilla NAT models these variations regularly have a positive impact on translation quality but for \textsc{CTC}\@\xspace-based models the picture is less clear and rarely leads to consistent \textsc{Bleu}\xspace improvements. Based on these results, we deploy all choices besides $\epsilon$ tuning for non-\textsc{CTC}\@\xspace variants but omit them for \textsc{CTC}\@\xspace-based models.

\subsection{Model comparison}
\label{sec.results}

Our results are consistent across the language pairs and metrics shown in \Cref{tb.results_full_bleu,tb.results_full_chrf,tb.results_full_ter}, and align with former work \citep{gu-kong-2021-fully}. The most significant translation quality improvement can be achieved by switching from the traditional length prediction module to a \textsc{CTC}\@\xspace-based variant, confirming the strong dependence on accurate length predictions \citep{wang-etal-2021-length}. This amounts to the largest accuracy increase that is achievable with current non-autoregressive methods but comes with a slight CPU latency increase due to the upsampling process. On top of this, \textsc{GLAT}\@\xspace consistently increases translation quality further by a significant margin. Deep Supervision can further increase the quality but, for the most part, not by a significant amount. Notably, though, it rarely hurts accuracy for our experiments across all three metrics and can sometimes even achieve a significant increase, i.e.\@\xspace \lromanian$\rightarrow$\lenglish\@\xspace on all three metrics and \lenglish$\leftrightarrow$\lgerman\@\xspace on \textsc{Ter}\xspace. For our analysis on the predicted translations, please see \Cref{app:analysis}. 

Comparing the strongest AT and NAT system in \Cref{tab:at_vs_nat}, we can see that despite \textsc{Bleu}\xspace parity on WMT'14 \lgerman$\rightarrow$\lenglish\@\xspace, the \textsc{Comet}\xspace scores \citep{rei-etal-2020-comet}\footnote{Obtained with \texttt{wmt20-comet-da} from version \texttt{1.1.0}} and our human evaluation results still indicate a quality gap between the two. Hence, metrics beyond \textsc{Bleu}\xspace are vital for rating NAT quality.

In terms of expected inference speed for non-autoregressive models, we believe that our latency numbers are more realistic than what is commonly reported in the literature. By considering additional speed-up techniques, improvements shrink to roughly $7.0\times$ on GPU and around $2.5\times$ on CPU.

\begin{table}[t]
    \centering
    \small
    \begin{tabular}{lLLLL}
    \toprule
    \multirow{3}{*}{\textbf{Metric}} & \multicolumn{4}{c}{\textbf{WMT'14}} \\
    & \multicolumn{2}{c}{\textbf{\lenglish$\rightarrow$\lgerman\@\xspace}} & \multicolumn{2}{c}{\textbf{\lgerman$\rightarrow$\lenglish\@\xspace}} \\
    & \multicolumn{1}{c}{$\Delta$} & \multicolumn{1}{c}{p-value} & \multicolumn{1}{c}{$\Delta$} & \multicolumn{1}{c}{p-value} \\
    \midrule
    \textsc{Bleu}\xspace $\uparrow$ & 0.9 & 0.001 & 0.0 & 0.397 \\
    \textsc{chrF++}\xspace $\uparrow$ & 0.3 & 0.009 & -0.3 & 0.002\\
    \textsc{Ter}\xspace $\downarrow$ & -1.0 & 0.001 & -1.0 & 0.001\\
    \textsc{Comet}\xspace $\uparrow$ & 0.3 & 0.000 & 0.1 & 0.000 \\
    Human Evaluation $\uparrow$ & 7.5 & 0.000 & 1.1 & 0.000 \\
    \bottomrule 
    \end{tabular}
    \caption{Score deltas (AT score $-$ NAT score) of best autoregressive vs.~non-autoregressive models on the WMT'14 \lenglish$\leftrightarrow$\lgerman\@\xspace test sets with p-values of the corresponding significance tests (paired bootstrap resampling for \textsc{Bleu}\xspace, \textsc{chrF++}\xspace, \textsc{Ter}\xspace, \textsc{Comet}\xspace; paired Student's t-test for Human Acceptability Evaluation).}
    \label{tab:at_vs_nat}
\end{table}

\section{Conclusion}
In this work, we provide multiple standardized \texttt{sacreBLEU}\xspace metrics, accompanied by reproducible and modular \texttt{fairseq}\xspace code. We hope that our work solves comparison problems in the literature, provides more realistic speed-up expectations, and accelerates research in this area to achieve parity with autoregressive models in the future.

\section*{Limitations}
While our work improves upon the state of translation quality and speed comparisons in the NAT literature, we acknowledge that there are many more low-level optimizations (e.g.\@\xspace porting the Python code to \texttt{C++} and writing dedicated CUDA kernels) that could be made to improve the presented work. This would allow for an even more realistic inference speed comparison and enable direct comparability to e.g.\@\xspace \texttt{Marian} \citep{junczys-dowmunt-etal-2018-marian} which is currently not given. There exists a concurrent work by \citet{helcl2022} that tries to address some of these shortcomings by providing a \textsc{CTC}\@\xspace implementation in \texttt{C++} and comparing the inference speed across different batching scenarios. 

Apart from that, current non-autoregressive models still have a strong dependency on appropriate knowledge distillation from an autoregressive teacher that is able to effectively reduce data modes and limit the multimodality problem. While this already shows to be an important factor for achieving good translation quality on benchmarking datasets such as WMT, it is even more relevant for larger and/or multilingual datasets \citep{agrawal2021} that tend to be of higher complexity. Even though non-autoregressive models can lead to improved inference speed, this strong dependency still requires additional effort for tuning effective teacher architectures, conducting multiple training procedures, and generating simplified data through one or more distillation rounds. Until these problems are solved, it seems unrealistic that the average practitioner, who does not have access to a lot of compute, can effectively deploy these in production settings as general-purpose translation models. Apart from that, there is still more work needed in reducing the multimodality problem through more suitable architectures, optimization objectives, or a better data selection process.




\section*{Acknowledgements}
We would like to thank Matthias Sperber for coordinating the publication process and Yi-Hsiu Liao for providing the Shortlist implementation. Further, we'd like to thank Sarthak Garg, Hendra Setiawan, Luke Carlson, Russ Webb, and Jesse Allardice for their helpful comments on the manuscript.  Many thanks to Sachin Mehta and David Harrison for suggesting interesting ideas and the rest of Apple's Machine Translation Team for their support.

\clearpage

