\section{Introduction}
\label{sec:intro}

These guidelines include complete descriptions of the fonts, spacing, and
related information for producing your proceedings manuscripts. Please follow
them and if you have any questions, direct them to Conference Management
Services, Inc.: Phone +1-979-846-6800 or email
to \\\texttt{papers@2021.ieeeicassp.org}.

\section{Formatting your paper}
\label{sec:format}

All printed material, including text, illustrations, and charts, must be kept
within a print area of 7 inches (178 mm) wide by 9 inches (229 mm) high. Do
not write or print anything outside the print area. The top margin must be 1
inch (25 mm), except for the title page, and the left margin must be 0.75 inch
(19 mm).  All {\it text} must be in a two-column format. Columns are to be 3.39
inches (86 mm) wide, with a 0.24 inch (6 mm) space between them. Text must be
fully justified.

\section{PAGE TITLE SECTION}
\label{sec:pagestyle}

The paper title (on the first page) should begin 1.38 inches (35 mm) from the
top edge of the page, centered, completely capitalized, and in Times 14-point,
boldface type.  The authors' name(s) and affiliation(s) appear below the title
in capital and lower case letters.  Papers with multiple authors and
affiliations may require two or more lines for this information. Please note
that papers should not be submitted blind; include the authors' names on the
PDF.

\section{TYPE-STYLE AND FONTS}
\label{sec:typestyle}

To achieve the best rendering both in printed proceedings and electronic proceedings, we
strongly encourage you to use Times-Roman font.  In addition, this will give
the proceedings a more uniform look.  Use a font that is no smaller than nine
point type throughout the paper, including figure captions.

In nine point type font, capital letters are 2 mm high.  {\bf If you use the
smallest point size, there should be no more than 3.2 lines/cm (8 lines/inch)
vertically.}  This is a minimum spacing; 2.75 lines/cm (7 lines/inch) will make
the paper much more readable.  Larger type sizes require correspondingly larger
vertical spacing.  Please do not double-space your paper.  TrueType or
Postscript Type 1 fonts are preferred.

The first paragraph in each section should not be indented, but all the
following paragraphs within the section should be indented as these paragraphs
demonstrate.

\section{MAJOR HEADINGS}
\label{sec:majhead}

Major headings, for example, "1. Introduction", should appear in all capital
letters, bold face if possible, centered in the column, with one blank line
before, and one blank line after. Use a period (".") after the heading number,
not a colon.

\subsection{Subheadings}
\label{ssec:subhead}

Subheadings should appear in lower case (initial word capitalized) in
boldface.  They should start at the left margin on a separate line.
 
\subsubsection{Sub-subheadings}
\label{sssec:subsubhead}

Sub-subheadings, as in this paragraph, are discouraged. However, if you
must use them, they should appear in lower case (initial word
capitalized) and start at the left margin on a separate line, with paragraph
text beginning on the following line.  They should be in italics.

\section{PRINTING YOUR PAPER}
\label{sec:print}

Print your properly formatted text on high-quality, 8.5 x 11-inch white printer
paper. A4 paper is also acceptable, but please leave the extra 0.5 inch (12 mm)
empty at the BOTTOM of the page and follow the top and left margins as
specified.  If the last page of your paper is only partially filled, arrange
the columns so that they are evenly balanced if possible, rather than having
one long column.

In LaTeX, to start a new column (but not a new page) and help balance the
last-page column lengths, you can use the command ``$\backslash$pagebreak'' as
demonstrated on this page (see the LaTeX source below).

\section{PAGE NUMBERING}
\label{sec:page}

Please do {\bf not} paginate your paper.  Page numbers, session numbers, and
conference identification will be inserted when the paper is included in the
proceedings.

\section{ILLUSTRATIONS, GRAPHS, AND PHOTOGRAPHS}
\label{sec:illust}

Illustrations must appear within the designated margins.  They may span the two
columns.  If possible, position illustrations at the top of columns, rather
than in the middle or at the bottom.  Caption and number every illustration.
All halftone illustrations must be clear black and white prints.  Colors may be
used, but they should be selected so as to be readable when printed on a
black-only printer.

Since there are many ways, often incompatible, of including images (e.g., with
experimental results) in a LaTeX document, below is an example of how to do
this \cite{Lamp86}.

\section{FOOTNOTES}
\label{sec:foot}

Use footnotes sparingly (or not at all!) and place them at the bottom of the
column on the page on which they are referenced. Use Times 9-point type,
single-spaced. To help your readers, avoid using footnotes altogether and
include necessary peripheral observations in the text (within parentheses, if
you prefer, as in this sentence).

\begin{figure}[htb]

\begin{minipage}[b]{1.0\linewidth}
  \centering
  \centerline{\includegraphics[width=8.5cm]{image1}}
  \centerline{(a) Result 1}\medskip
\end{minipage}
\begin{minipage}[b]{.48\linewidth}
  \centering
  \centerline{\includegraphics[width=4.0cm]{image3}}
  \centerline{(b) Results 3}\medskip
\end{minipage}
\hfill
\begin{minipage}[b]{0.48\linewidth}
  \centering
  \centerline{\includegraphics[width=4.0cm]{image4}}
  \centerline{(c) Result 4}\medskip
\end{minipage}
\caption{Example of placing a figure with experimental results.}
\label{fig:res}
\end{figure}



\section{COPYRIGHT FORMS}
\label{sec:copyright}

You must submit your fully completed, signed IEEE electronic copyright release
form when you submit your paper. We {\bf must} have this form before your paper
can be published in the proceedings.

\section{RELATION TO PRIOR WORK}
\label{sec:prior}

The text of the paper should contain discussions on how the paper's
contributions are related to prior work in the field. It is important
to put new work in  context, to give credit to foundational work, and
to provide details associated with the previous work that have appeared
in the literature. This discussion may be a separate, numbered section
or it may appear elsewhere in the body of the manuscript, but it must
be present.

You should differentiate what is new and how your work expands on
or takes a different path from the prior studies. An example might
read something to the effect: "The work presented here has focused
on the formulation of the ABC algorithm, which takes advantage of
non-uniform time-frequency domain analysis of data. The work by
Smith and Cohen \cite{Lamp86} considers only fixed time-domain analysis and
the work by Jones et al \cite{C2} takes a different approach based on
fixed frequency partitioning. While the present study is related
to recent approaches in time-frequency analysis [3-5], it capitalizes
on a new feature space, which was not considered in these earlier
studies."

\vfill\pagebreak

\section{REFERENCES}
\label{sec:refs}

List and number all bibliographical references at the end of the
paper. The references can be numbered in alphabetic order or in
order of appearance in the document. When referring to them in
the text, type the corresponding reference number in square
brackets as shown at the end of this sentence \cite{C2}. An
additional final page (the fifth page, in most cases) is
allowed, but must contain only references to the prior
literature.

\bibliographystyle{IEEEbib}

\section{Introduction}


Evaluation is a major bottleneck for speech synthesis tasks like text-to-speech (TTS) and speech-to-speech translation. In principle, there are infinitely many spoken renditions of a piece of text, and there is no universally agreed upon definition of what makes an utterance ``correct''. Thus, researchers rely heavily on human evaluation, more specifically listening tests, during their day-to-day development cycles. The most popular type of listening test is MOS (Mean Opinion Score), during which several annotators listen to audio segments and rate them on a Likert scale between 1 to 5 (examples of foundational studies that use it include Tacotron~\citet{wang2017tacotron}, Parallel WaveNet~\citet{oord2018parallel}, or FastSpeech 2~\citet{ren2020fastspeech}). Listening tests can produce reliable results \citep{lewis2001psychometric}, since humans usually excel at detecting speech quality, and the scheme can be adapted to the need of every specific task. But they are also impractical and expensive: recruiting and polling annotators increases the cost of running experiments, slows down model research, and makes it impossible to compare results across time and institutions. The problem gets exacerbated in the multilingual setup: it may be challenging for researchers to find speakers of languages that are neither spoken by many, nor geographically close to them. Ultimately, this hinders their progress, skews the literature towards high resource languages, and prevents them from engaging in heavily multilingual research. This comes in contrast to text-based generation tasks, such as Machine Translation, for which the research community has long adopted automatic metrics (such as BLEU~\cite{bleupaper} or more recently COMET~\cite{rei-etal-2020-comet} and BLEURT~\citet{sellam-etal-2020-bleurt}) as a complement to human assessment.

To address these issues, there has been a growing interest in developing automatic metrics for speech synthesis, spearheaded by systems such as AutoMOS~\citep{45744}, MOSNet~\citep{lo19_interspeech}, or LDNet~\citep{huang2022ldnet}. The idea is to cast quality evaluation as a regression or classification problem: these systems predict a quality score from an utterance, using past listening tests as a source of training data.  Predicting quality is a difficult task because the target domain is complex, even in a monolingual setup: synthesis artifacts come in many forms and can affect all levels of speech production, including pronunciation, prosody, voice, and audio quality. And the task is getting harder over time~\cite{lemaguer22_interspeech}: as voice synthesis systems progress, the focus has shifted from obvious artifacts (e.g., robotic voices) to more subtle errors, such as inappropriate prosody or mispronunciations. Yet, the same problem that motivates the task plagues its solution---data is expensive to collect, especially outside high-resource languages, and so existing studies tend to use relatively limited training and testing sets. Early MOS Predictors have been shown to be brittle when used out of domain~\citep{cooper22Generalization}, and there are few studies outside English and Chinese.






In this paper, we study MOS Prediction at scale, and in a massively multilingual setup. We introduce SQuId (Speech Quality Identifier) a speech quality detector based on mSLAM (multilingual Speech and LAnguage Model), a recently published pre-trained model~\cite{bapna2022mslam}. SQuId is trained on over a million ratings, an order of magnitude more than most recent studies~\cite{lo19_interspeech}. More importantly it is, to the best of our knowledge, the first massively multilingual model for MOS prediction: we trained the model on 42 locales\footnote{Compared to language, locales take regional variation into account. For instance English is covered by five locales in our dataset: US English, UK English, Indian English, Nigerian English, and Australian English. Each of these variants should be rendered differently by a TTS engine.} and tested it on 65. For comparison, VoiceMOS, the most comprehensive benchmark to date, covers two locales~\cite{huang2022voicemos}. We describe our dataset, our model, and show that SQuId outperforms a competitive baseline based on SSL and VoiceMOS by up to 50.0\%. Most improvements come from the additional supervised multilingual data, complemented by minor optimizations that target the multilingual case.  We conduct several studies to highlight the factors that contribute to MOS prediction quality in this massively multilingual, \emph{in the wild}, setup. The key insights from our work include:

(i) Training \emph{one} model on a diverse dataset consisting of data from \emph{many} locales consistently outperforms the monolingual approach as a result of \emph{cross-locale transfer}, an effect well known in NLP~\cite{zoph2016transfer,aharoni-etal-2019-massively, Arivazhagan2019MassivelyMN, conneau-etal-2020-unsupervised}, ASR~\cite{lin2009study,li2018multi,kannan2019large} and TTS literature~\cite{Zhang2019LearningTS,he2021multilingual}. The most spectacular manifestation of this phenomenon is the model's strong performance on \emph{zero-shot} locales, where there is no labelled MOS prediction data. Cross-locale transfer allows us to increase the model's language coverage dramatically and has significant implications on evaluation of multilingual speech synthesis.

(ii) We conduct analyses to understand the nature and mechanism of cross-locale transfer for MOS prediction. We demonstrate that locale diversity has a large influence on model's performance during fine-tuning, but transfer is driven less by language similarity and more by the presence of language-agnostic phenomena (possibly including diversity of audio quality, voices and TTS systems) in the dataset. We highlight the role of \emph{para-lingual transfer}, by which training data in one locale improves performance in another for reasons orthogonal to linguistics.

(iii) Through ablation studies, we highlight the importance of modeling choices, including \emph{multilingual} pre-training, model capacity, data re-balancing and locale tagging on SQuId performance.


\section{Background and Related Work}

\textbf{MOS Prediction.} Automatic MOS prediction has a long history in the TTS literature~\cite{ragano2022comparison}. In addition to the systems cited above, recently published work that use neural approaches include Williams et al.~\cite{williams2020}, MBNet~\cite{mbnet21}, and Tseng et al.~\cite{tseng2021utilizing}, which relies heavily on self-supervised representations. Choi et al.~\cite{choi2021neural} explores transfer from auxiliary tasks, as does Tian et al.~\cite{tian22d_interspeech} which combines a domain-independent encoder and a domain-specific decoder. None of these works tackle multilinguality.  Our work directly builds upon Cooper et al.\ \citet{cooper22Generalization}, which experiments with wide range of pre-trained models and show that they are a promising resource to make MOS prediction models that generalize. Our methodology is close, but we scale up the data (from about 30K to 1,3M samples), number of locales (from 2 to 65), model size (from 300M to 600M parameters), and contribute a novel analysis of multilinguality and cross-locale transfer.

\noindent\textbf{Neural Translation and Multilingual NLP.} There is a rich literature on cross-lingual transfer in NLP. Notable works include Zoph et al.\  \cite{zoph2016transfer}, which introduces the idea of pre-training an MT model on a high resource language pair to increase performance on a lower-resourced one. Aharoni et al. \cite{aharoni-etal-2019-massively}, then Arivazhagan et al.~\cite{Arivazhagan2019MassivelyMN} discuss transfer at a much larger scale, and describe a massively multilingual model that can cope with up to 103 languages. Outside translation, pre-trained models such as mBERT~\cite{devlin-etal-2019-bert}, XLM-R~\cite{conneau-etal-2020-unsupervised}, and mT5~\cite{xue-etal-2021-mt5} have allowed successful cross-lingual transfer in over a hundred languages.

\noindent\textbf{Multilingual ASR and TTS.}
In the speech recognition literature, authors have studied cross-lingual transfer for at least a decade~\cite{swietojanski2012unsupervised,huang2013cross}. More recently, Li et al.\ \cite{li2020universal} propose to jointly model the distributions of phones (which are language-agnostic) and phonemes (which are language-dependent).  {\.Z}elasko et al.\ \cite{zelasko2020sounds}, then Feng et al.\ \cite{feng2021phonotactics}, present an empirical study mono-, multi-, and cross-lingual phone recognition in 13 languages. Multilingual TTS has also been investigated for decades~\cite{javkin1989multilingual}, and the idea to share representations across languages has been studied in a number of papers including Zhang et al.~\cite{Zhang2019LearningTS} and Sanchez et al.~\cite{Sanchez2022}.


\section{Multilingual MOS Prediction in the Wild}
\label{sec:data}

\begin{table}[t!]
\centering
\footnotesize
\begin{tabularx}{\linewidth}{ l | X | X }
\toprule
Error category & Meaning & Example \\
\midrule
Pronunciation & One or more words are not pronounced correctly. & Reading ``tough" as ``toe" rather than ``tuff". \\
Text Normalization & Non-standard words are not resolved to the correct spoken form in a given context. & Reading ``7/7" as ``seven divided by seven" rather than ``July seventh". \\
Prosody & The pitch, duration, or energy of the utterance or part of the utterance is unnatural. & Questions without rising intonation at the end, incorrect emphasis. \\
Acoustic Artifacts & Issues with the audio that are not due to an improper linguistic analysis of the text. & Audible joins in concatenative TTS, signal-correlated noise, improperly-realized phonemes. \\
\bottomrule
\end{tabularx}
\caption{Error categories present in the dataset.}
\label{table:error_cat}
\vspace{-5mm}
\end{table}
 
 
We wish to predict \emph{MOS Naturalness} ratings for both human and synthetic speech. Broadly speaking, MOS Naturalness describes how human-like an utterance sounds. Our main resource is an in-house corpus, aggregating approximately 1.9 million ratings in 66 locales across 2,092 research and commercial annotation projects completed between January 2021 and March 2022.
Most of the audio is generated  by TTS systems, and the annotators are asked to select a rating based on how natural or unnatural the samples sound.



\noindent \textbf{Coverage.} The TTS systems in the dataset cover a wide range of technologies including both concatenative and neural systems. 
The test sets used for the evaluations are primarily focused on TTS applications such as virtual assistant responses, driving directions, book passages, and news articles; general text from web-crawled corpora is also used. Specific challenge sets are used to test particularly difficult inputs for TTS such as compound nouns, questions, and punctuation \citep{49375}. Sentences are typically rated in isolation (that is, outside of the context in which they originally appeared), though entire paragraphs are occasionally rated as well. This variety in test sets and TTS technologies means the stimuli contain a diverse set of errors, including pronunciation errors, text normalization errors, unnatural prosody, and acoustic artifacts such as discontinuities and signal-correlated noise. Table~\ref{table:error_cat} describes these phenomena in more detail.

\begin{figure}[b!]
\centering
\includegraphics[width=.3\textwidth]{figures/locales_distrib_compact}
\caption{Data distribution for each locale.}
\label{fig:locales_distrib_compact}
\end{figure}


\noindent \textbf{Splits and Locales.} We apply two splits to the dataset. First, we split the data by time. We use the ratings collected between January 1st and December 1st 2021 for training and development, and the rest for test. The motivation is to simulate a realistic use case, whereby a TTS engineer would want to use the past annotations to predict future ones. The second split is based on region: we hold out 24 locales for which have exceptionally few ratings (less than 8,000 each, adding up to about 5\% of the data) and use them for test. The rationale is that small tasks yield little improvements during training but are particularly useful for analysis. Since there is no training data, we refer to these locales as \emph{zero-shot locales}, as opposed to \emph{fine-tuned locales}. Figure~\ref{fig:locales_distrib_compact} presents an overview of the locales distributions (details in Appendix~\ref{sec:training_data_details}). The dataset is dominated by US English (18\%), followed by UK English (12\%), and ES Spanish (4.2\%). To build the development set, we sample 2.5\% of the training set without replacement. Table~\ref{tab:datastats} provides additional statistics.

 \begin{table}[h!]
 \center
 \footnotesize
\begin{tabular} {l | c }
\toprule
Num. training utt. / systems. / locales & 969,589 / 1,476 / 42\\
Num. dev utt. / systems / locales & 34,042 / 1,474 / 42\\
Num. test utt. / systems / locales & 381,323 / 605 / 65\\
Ave. utterance duration (seconds) & 4.5s\\
Avg. ratings per utt. & 1.4\\
\bottomrule
\end{tabular}
\caption{MOS Prediction dataset statistics.}
\label{tab:datastats}
\vspace{-4mm}
\end{table}



\noindent  \textbf{Challenges and Caveats.} Due to the nature of splits, we do not expect the data to be i.i.d.---the TTS systems and annotators used to produce and rate the utterances in February 2021 are usually not those of January 2022. The of number, listening conditions and quality of the annotations also vary across projects and locales, as do the input texts chosen to test the systems. Furthermore, it is generally understood that the term \textit{naturalness} is underspecified and may be interpreted differently be different raters \citep{dall2014rating,wester15c_interspeech}.
In short, these conditions reflect TTS evaluation ``in the wild''. 


\noindent  \textbf{Details of the Listening Test.} Listening tests were conducted using crowd-sourced raters on an internal ratings platform.  Listeners were first asked to read the instructions presented above. They were then presented with a speech sample and instructed to rate how natural the speech sounds on a 9 point Likert scale, with values ranging from 1 to 5 in 0.5 increments. The whole number points on the scale were labelled \textit{Bad}, \textit{Poor}, \textit{Fair}, \textit{Good}, and \textit{Excellent}, and the half-step points were unlabelled.

 
\section{The SQuId Model}

\textbf{One Model, Many languages.} The most important design decision behind our study is to fine-tune a single model on all locales rather than keeping separate models. Doing so offers convenience, since we have one model to maintain rather than 65. More importantly, the assumption is that if the model has enough capacity, \emph{positive transfer} will emerge between the locales. If the model is pre-trained on many languages, we can even hope for strong results on the locales for which we have no labelled data. We demonstrate the validity of all these claims in the following sections.

\noindent  \textbf{Architecture.} SQuId is based on mSLAM~\citep{bapna2022mslam}, a recently published multi-modal pre-trained model trained on unlabelled speech (429K hours in 51 languages), text (15TB in 101 languages), and speech-text pairs (2.3K hours). We chose this model because it produced state-of-the-art results in many languages at the time of writing. It is based on the Conformer architecture, with 600M parameters by default. SQuId's input is a 16KHz utterance's spectrogram, along with an optional locale tag. The output is a scalar. We fine-tune the model end-to-end with a simple regression loss. After optional resampling to 16KHz, we compute an 80-dimensional log Mel spectrogram and extract mSLAM embeddings for each time step:
$$
\mathrm{mSLAM}(\mathrm{MelSpec_{80}}(\mathrm{Resample_{16KHz}}(\mathbf{utt}^i))) = \mathbf{e}_1^i, ,...,\mathbf{e}_T^i
$$
\noindent We mean-pool across the time dimension, returning an embedding $\mathbf{e}_{*}^i$, and apply a fully connected layer to obtain the prediction $\hat{y}^i = \mathbf{M}.\mathbf{e}_{*}^i + \mathbf{b}$. By default we use $T=3,200$ time steps, and the embeddings have dimension 1,024. The target MOS ratings are linearly rescaled from [1, 5] to [0, 1].

\noindent \textbf{Optimizations.} Two optimizations lead to slight but consistent performance improvement. First, we embed the locale tags of each utterance into a 64-wide vector $\mathbf{e_\ell}^i$ and concatenate it to $\mathbf{e}_{*}^i$, forming vector $[\mathbf{e}_{*}^i, \mathbf{e_\ell}^i]$. For 5\% of the data, we use a wildcard identifier \textsc{Any-Loc}, which we use for inference on locales unseen during training. Second, we resample the data to rebalance the relative proportion of the training locales~\cite{Arivazhagan2019MassivelyMN}. If $l$ is a locale with frequency $p_l$, we resample it with probability $p_l^{1/\tau}$ with $\tau=10$ by default. We describe the method in detail in Apendix~\ref{sec:temperature}.

\noindent  \textbf{Details.}  By default we use batch size 32 and learning rate $10^{-5}$, obtained with hyper-parameter search during a preliminary set of experiments. We train the models for 100k steps, save a snapshot every 10k steps, and export the version that yields the best version on our development set. We run experiments on Cloud TPU v3, using the Adam optimizer~\tbo{to fact-check} with 1,500 warm-up steps.


\section{Performance}

Let us now present SQuId's overall performance. We first ensure that it performs correctly on VoiceMOS'22, currently the main benchmark on MOS prediction. We then scale up the test set and analyze its performance on 65 locales.

Throughout the section, we will compare SQuId to Big-SSL-MOS, a competitive baseline in the spirit of SSL-MOS~\citet{cooper22Generalization}. We fine-tune w2v-BERT on the main VoiceMOS dataset with a regression objective. We use a 600M parameters version of the pre-trained model to ensure a fair comparison with SQuId (Cooper et al.\ uses up to 317M~\citet{cooper22Generalization}). The model comes in two variants: English-only, and multilingual. The architecture and dataset of our w2v-BERT implementation are described in detail in Bapna et al.\ \citet{bapna2022mslam}.

For all experiments we report the correlation with human ratings. More specifically, we use the segment-level Kendall Tau --- segment-level because there are many locales for which we only have a handful of systems so reporting system-level correlations would be too brittle, and Kendall Tau because it is resistant to outliers (it is based on rank) and has a rich history in the meta-evaluation literature~\citep{huang2022voicemos, ma-etal-2019-results}. By default, all experiments are replicated three times and the results averaged. Confidence bars represent bootstrapping-based 95\% intervals.

\begin{table}[!t]
\centering
\footnotesize
\begin{tabular} {l  l | c }
\toprule
Model & & Kendall Tau\\
\midrule
VoiceMOS '22& SSL-MOS & 0.690\\
Baselines& MOSA-Net & 0.621\\
& LDNet & 0.599\\
Submissions& Top & \textbf{0.730} \\
& Median & 0.698 \\
& Last & \emph{0.562} \\
\midrule
Big-SSL-MOS & English & \textbf{0.702}\\
 & Multilingual & 0.693\\
\midrule
SQuId & SQuIdDS & 0.606\\
& VoiceMOS & 0.700\\
& SDS+VMOS & \textbf{0.701}\\
\bottomrule
\end{tabular}
\caption{Results on VoiceMOS'22, Main Track. Segment-level correlation with human ratings (Kendall Tau). The VoiceMOS'22 results were obtained from~\cite{huang2022voicemos}, except SSL-MOS which we recalculated to confirm the alignment.}
\label{table:voicemos}
\vspace{-4mm}
\end{table}


\noindent \textbf{Correctness Check: Performance on VoiceMOS'22.}
The VoiceMOS Challenge is currently the main public benchmark for MOS prediction.\footnote{\url{https://codalab.lisn.upsaclay.fr/competitions/695}} The challenge is now closed, and the full dataset has been released by the organizers. The main task contains 4,974, 1,066, and 1,066 utterances for training, development, and test respectively, all in English, as well as 1,352 utterances in Chinese. The submissions are aggressively optimized towards the task; for instance UT-MOS~\citep{UTMOS22}, a top performing system, ensembles several models and use contextual information such as Listener and Domain ID. Additionally, the organizers provide the results of three recently published baselines: SSL-MOS~\citep{cooper22Generalization}, MOSA-NET~\citep{zezario2021deep}, and LDNet~\citep{huang2022ldnet}. Since the dataset is very focused comparatively (we test on 30X more locales), we use it as a correctness check for our approach and baselines. We restrict ourselves to simple approaches: we use batch size 8 (instead of 32) and train for 10K steps (instead of a million), but do not introduce any additional optimization. We diligently set the test side aside during hyper-parameter tuning.

Table~\ref{table:voicemos} presents the results. We present three versions of SQuId: fine-tuned on the SQuId Dataset (SQuIdDS), VoiceMOS, and both sequentially. Although our approach is simple, we confirm that it is competitive. The SQuId models trained on VoiceMOS outpeform all the baselines, and would sit in the upper half of the benchmark. Our baseline Big-SSL-MOS performs even better, justifying that the comparison is fair.

VoiceMOS also includes an out-of-distribution dataset, based on Chinese---we provide the results in Appendix~\ref{voicemos:ood}. The trends are similar, except that SQuId on SQuIdDS is weaker and Big-SSL-MOS Multilingual outperforms Big-SSL-MOS English.

\begin{figure}[h!]
  \centering
    \includegraphics[width=0.4\textwidth]{figures/ft_smos_compact.pdf}
    \vspace{-3mm}
    \caption{Correlations with human ratings, all locale averaged.}
    \label{fig:squid_scores-compact}
    \vspace{-3mm}
\end{figure}


\noindent \textbf{Main Results.} Figure~\ref{fig:squid_scores-compact} reports SQuId's performance on the main dataset. We compute the correlation with human ratings on all 65 test locales and average the results. SQuId VoiceMOS is slightly better than Big-SSL-MOS English; the main difference is due to the switch from w2v-BERT to mSLAM, which performs better on this task. Using the SQuId Dataset incurs a dramatic ~38\% performance boost, which validates the usefulness of the dataset. The optimizations (re-sampling and locale identifier) add a more modest 3.75\% increase. Our final model outperforms the best baseline by ~50.0\%. Appendix~\ref{sec:training_data_details} presents the breakdown per locale, showing that SQuId outperforms Big-SSL-MOS on 60 locales out of 65.

\begin{figure}[h!]
  \centering
    \includegraphics[width=0.4\textwidth]{figures/locales_vs_perf.pdf}
    \vspace{-4mm}
    \caption{Distributions of the scores. Each dot represent a locale, the box plots report quartiles.}
    \label{fig:squid_scores_by_group}
\end{figure}

\noindent Figure~\ref{fig:squid_scores_by_group} presents SQuId's performance on each locale, as we increase the number of locales used for fine-tuning. Each addition improves the model, on both fine-tuned and zero-shot splits, demonstrating the existence of cross-locale transfer. SQuId's performance is comparable in average on both sets, this illustrates the viability of zero-shot inference for MOS prediction.

There are large disparities between the scores, which can range from below 0 to over 0.3. What explains these differences? The first, and possibly main factor, is that the test set is not homogeneous: the sub-tasks have various degrees of complexity and the annotations have various degrees of noisiness. Indirect evidence is that Big-SSL-MOS and SQuId tend to perform similarly on the same locales even though they were fine-tuned on very different datasets (Pearson correlation: 0.845). The training data is also heterogeneous, and so we cannot expect its performance to be constant, even on a perfect test set. We expose the relationship between training size and performance for each locale in Appendix~\ref{sec:data_vs_perf}.



\section{Understanding Cross-Locale Transfer during Fine-Tuning}
\label{sec:abl8-finetuning}


\noindent \textbf{Measuring Cross-Locale Transfer.} To understand cross-locale transfer further, we focus on 8 languages and compare three models: mSLAM fine-tuned on the target locale only, on all 8 locales, and on all the data we have (42). We picked languages from different families, with different levels of coverage (from 2.2K to 280K ratings), and for which we have ample test data to reduce the variance of the experiment.

\begin{figure}[ht!]
    \centering
    \includegraphics[width=0.45\textwidth]{figures/abl8_crosslingual_tuning.pdf}
    \caption{Performance on 8 target locales varying the number of fine-tuning languages.}
    \label{fig:vary_ft_locales}
    \vspace{-3mm}
\end{figure}
\noindent Figure~\ref{fig:vary_ft_locales} presents the results. For five out of eight setups, each addition increases the results. The most spectacular improvement happen between 1 and 8 on Thai (+244\%) and Japanese (+31.2\%). In contrast, adding data makes no difference or causes a slight degradation for three setups. We conclude that cross-locale ``works'' but it does not affect all locales uniformly.






\noindent \textbf{Cross-locale transfer and language.} Which locales are impacted by transfer? Intuitively, we may expect it to occur between tasks that are linguistically similar to each other~\cite{zoph2016transfer, li2018multi,kannan2019large}. To test this hypothesis, we focus on two languages for which we have different locales: Portuguese (the data set contains utterances in Brazilian and European Portuguese) and French (we have French and Canadian). We first fine-tune mSLAM on the locale with the lowest amount of data (i.e., European Portuguese and French, with about 9,700 examples each) and test it on the same. We then compare to mSLAM fine-tuned on locales pairs, by coupling the target locale with either a close one (e.g., European Portuguese and Brazilian Portuguese) or remote ones (e.g., European Portuguese and Japanese). If the hypothesis holds, then adding in-language data will help, and similar pairs will perform better than remote ones. We chose the baselines locales such that the amount of fine-tuning data is approximately similar in all setups.


\begin{figure}[t!]
     \centering
     \begin{subfigure}[b]{0.49\columnwidth}
         \centering
         \includegraphics[width=\textwidth]{figures/langfam_pt.pdf}
     \end{subfigure}
     \hfill
     \begin{subfigure}[b]{0.49\columnwidth}
         \centering
         \includegraphics[width=\textwidth]{figures/langfam_fr.pdf}
     \end{subfigure}
     \vspace{-4mm}
     \caption{Cross-lingual transfer to Pt-PT (left) and Fr-FR (right)}
     \label{fig:langfam}
     \vspace{-5mm}
\end{figure}

\noindent The results are in Figure~\ref{fig:langfam}. There is much transfer from Brazilian to European Portuguese, but the effect is approximately similar with German. Furthermore, Japanese and Arabic are not far behind. The results are more spectacular on French: adding French Canadian to European French actually \textit{decreases} its performance, while Icelandic improves it. Our conclusions are therefore negative: it seems that the influence of linguistic similarity on cross-lingual transfer is very limited in this application. A similar observation may be found in~\cite{Zhang2019LearningTS}, where improvements are observed in TTS systems transferring voices across highly dissimilar languages like English and Chinese.




\noindent \textbf{Cross-Lingual vs. Para-Lingual Transfer.}  If language similarity does not facilitate transfer, then what does? We hypothesize that the task is in fact dominated by sound and paralinguistics, i.e., artifacts such as robotic voices and flat prosody that do not depend on words. This does not mean that there is no cross-locale transfer: the performance on each task does indeed improve as the model is exposed to the others. But the phenomenon is distinct from the cross-lingual transfer that is commonly discussed in the NLP literature~\cite{zoph2016transfer}, and that motivates the design of multilingual pre-trained models such as mSLAM. We distinguish \emph{para-lingual} transfer from \emph{cross-lingual} transfer, and treat them as two orthogonal components of \emph{cross-locale transfer}.


 \begin{figure}[ht!]
    \centering
    \includegraphics[width=0.3\textwidth]{figures/abl8_crosslingual_inference.pdf}
    \vspace{-3mm}
    \caption{Cross-locale performance, Kendall Tau with human ratings.}
    \vspace{-4mm}
    \label{fig:cross-locale}
\end{figure}


To highlight the role of para-lingual transfer, let us now revisit a previous experiment. We fine-tune single-locale mSLAM models on the 8 locales discussed above and run them on each other's test set, i.e., we run each of the 8 models on each of the the 8 locales. We present the results in a matrix where the rows represent fine-tuning and columns test locales (Figure~\ref{fig:cross-locale}). We trained randomly initialized models to prevent any cross-lingual transfer, with a smaller architecture to stabilize the learning (we used the 42M parameters model discussed in the next section). If the tasks were truly independent, the diagonal would be salient. If they depended on language alone, there would be more affinity between languages of the same family than between very different ones, or at least, between languages that share phonological features.
Instead we find that the results are diffuse: the good results of Turkish on Portuguese could come from a shared phonemic inventory, but the fact that Japanese yields relatively good results on five out of eight tasks (doing better than French on French) confirms that MOS Naturalness is, to some extent, orthogonal to language.

\section{Cross-Task Transfer between Pre-Training and Fine-Tuning}
\label{sec:pretraining}

If the task is dominated by para-linguistics and sound artifacts, does it still make sense to pre-train models in multiple languages? And since the burden of memorization seems to be lesser than for, e.g., translation, do we still benefit from scale? To answer these questions, we rerun our experiments varying the sets of pre-training languages (from 1 to 51) and capacities (from 42M to 600M parameters). We use speech-only models to simplify the setup, we provide the detail of the architectures and pre-training datasets in  Appendix~\ref{sec:architectures}.

\begin{figure}[h]
     \centering
     \begin{subfigure}[b]{0.49\columnwidth}
         \centering
         \includegraphics[width=\textwidth]{figures/abl8_pt-en.pdf}
     \end{subfigure}
     \hfill
     \begin{subfigure}[b]{0.49\columnwidth}
         \centering
         \includegraphics[width=\textwidth]{figures/abl8_pt-all.pdf}
     \end{subfigure}
     \vspace{-3mm}
     \caption{Impact of model capacity and number of pre-training locales on SQuId's performance.}
     \label{fig:pretraining}
     \vspace{-3mm}
\end{figure}

\noindent Figure~\ref{fig:pretraining} illustrates the results. On US English (left), pre-training in English seems essential, but any addition causes little to no improvement. The smallest model actually gets worse, we hypothesize that it is saturated (see the curse of multilinguality in the NLP literature~\cite{conneau-etal-2020-unsupervised}). The model however improves consistently if we broaden the analysis to all locales. Here again, it is difficult to assess whether the improvement comes from in-locale, cross- or para-lingual transfer (the voices and recording conditions are not the same for all locales, which may help the model generalize), but it is clear that linguistic diversity improves performance, as does model size.

\begin{figure}[h]
     \centering
     \begin{subfigure}[b]{0.49\columnwidth}
         \centering
         \includegraphics[width=\textwidth]{figures/abl8_pt-all-ft.pdf}
     \end{subfigure}
     \hfill
     \begin{subfigure}[b]{0.49\columnwidth}
         \centering
         \includegraphics[width=\textwidth]{figures/abl8_pt-all-zs.pdf}
     \end{subfigure}
     \vspace{-3mm}
     \caption{Impact of model capacity and number of pre-training locales on fine-tuned (left) and zero-shot (right) locales.}
     \label{fig:pretraining-subsets}
     \vspace{-3mm}
\end{figure}

\noindent Figure~\ref{fig:pretraining-subsets} presents the results on fine-tuned and zero-shot locales separately. Pre-training diversity and larger models help in all cases. Mode size seems to have a particularly large impact on zero-shot locales.



\section{Conclusion}

We presented SQuId, a massively multi-locale MOS prediction model, trained on data from 42 locales and evaluated on 65 locales. We conducted several studies highlighting the role of multilingual pre-training, model capacity, locale-tagging and most importantly, the role of fine-tuning a single model on a diverse multilingual dataset on MOS quality prediction. Our results demonstrate significantly improved MOS prediction performance by multilingually fine-tuning a large, multilingual pre-trained SSL model, as compared against monolingual alternatives. Our results demonstrate that that multilingual, zero-shot MOS prediction is a viable path towards lowering the cost of massively multilingual speech synthesis evaluation, especially for low-resource locales. We observe that cross-locale transfer may in fact have little to do with language similarity, and is likely driven by the presence of diverse acoustic phenomena, locales and TTS systems.

The importance of non-linguistic phenomena in MOS Naturalness prediction holds lessons for both TTS evaluation and general speech SSL. For TTS evaluation, we see that naturalness is only one facet of speech quality, and reiterate the importance of using complementary evaluation methods, both human and automatic, to measure language-specific aspects such as prosody and pronunciation. Regarding SSL, it seems plausible that para-lingual transfer affects other tasks well (such TTS and ASR). We therefore call for model developers to enrich their pre-training corpora to include a diverse set of voice (natural and synthetic), prosody, and sound~\cite{hsu2021robust}, in addition to variations in domains and languages.~\tbo{Too weak?}

\section{Acknowledgements}

We would like to thank Erica Cooper, who provided extremely valuable help and support with the VoiceMOS dataset. We also thank Alexis Conneau, Min Ma, Dan Garrette, Heiga Zen, Yang Yu, Yu Zhang, Ye Jia, David Kao, RJ Skerry-Ryan, Fadi Haik, Siamak Tazari, Michael Herscovici, Sid Dalmia, Rob Clark, Shankar Kumar, Yonghui Wu, and more generally the TTS Team for their help and advise.

\bibliographystyle{IEEEbib}

