\section{Introduction}

\begin{figure}
    \centering
    \includegraphics[width=0.40\textwidth]{figures/SSL_framework.png}
	 \caption{Framework for using self-supervised representation learning in
	 downstream applications}
    \label{fig:SSL_framework}
\end{figure}








Over the past decade, deep learning approaches have revolutionized speech processing
through a giant leap in performance, enabling various real-world applications.
Supervised learning of deep neural networks has been the cornerstone of this
transformation, offering impressive gains for scenarios rich in labeled
data~\cite{lecun2015deep, hinton2012deep, bourlard2012connectionist}. 
Paradoxically, this heavy reliance on supervised learning has restricted progress in
languages and domains that do not attract the same level of labeling
investment. 

To overcome the need for labeled data, researchers have explored approaches that use
unpaired audio-only data to open up new industrial speech use-cases and
low-resource languages~\cite{Kemp1999, csl01_limsi, ma_bbn_06}. Inspired by how
children learn their first language through listening and interacting with
family and surroundings, scientists seek to use raw waveforms and
spectral signals to learn speech representations that capture low-level
acoustic events, lexical knowledge, all the way to syntactic and semantic
information. These learned representations are then used for target downstream
applications requiring a minimal number of labeled data~\cite{Hinton_2007,
LeCun06atutorial, bengio_representation_2013}. 
Formally, representation learning refers to algorithms for extracting latent
features that capture the underlying explanatory factors for the observed
input~\cite{bengio_representation_2013}. 

Representation learning approaches are generally considered examples of \textit{unsupervised
learning}, which refers to the family of machine learning methods that discover
naturally occurring patterns in training samples for which there are no pre-assigned
labels or scores~\cite{jordan_2015}. 
The term ``unsupervised'' is used to distinguish this family of methods from
``supervised'' approaches, which assign a label to each training sample, and
``semi-supervised'' approaches, which utilize a small number of training samples
with labels to guide learning using a larger volume of unlabeled samples.
Examples of unsupervised learning techniques include k-means{}
clustering~\cite{vq}, mixture models~\cite{MoE}, autoencoders~\cite{hinton_94},
and non-negative matrix factorization~\cite{nmf}. 
\textit{Self-supervised learning} (SSL) is a fast-growing subcategory of
unsupervised learning approaches, which are techniques that utilize
information extracted from the input data itself as the label to learn
representations useful for 
downstream tasks. In this review, we
focus on self-supervised learning approaches.

\Cref{fig:SSL_framework} outlines self-supervised representation learning in
relation to downstream applications. 
There are two stages in this framework.
In the first stage, we use SSL to pre-train a \textit{representation model},
also called an \textit{upstream model} or a \textit{foundation model}.
In the second stage, downstream tasks use either the learned
representation from the frozen model, or fine-tune the entire pre-trained model
in a supervised phase~\cite{hinton_2006}. 
Automatic speech recognition (ASR) and speaker identification (SID) are 
examples of downstream applications in \cref{fig:SSL_framework}.







It is considered desirable for learned speech representations to be
disentangled, invariant, and hierarchical.
Since spoken utterances contain much richer information than the corresponding text
transcriptions---e.g., speaker identity, style, emotion, surrounding noise, and
communication channel noise---it is important to learn representations that
disentangle these factors of variation. Furthermore, invariance of the learned
features to changes in background noise and in the communication channel ensures
stability with respect to downstream application scenarios. Learning feature
hierarchies at the acoustic, lexical, and semantic levels supports applications
with different requirements. For instance, whereas a speaker identification task
benefits from a low-level acoustic representation, a speech
translation task requires a more semantic representation of the input
utterance. 

Due to the popularity of SSL, reviews have been published about the
technology in
general~\cite{bommasani2021opportunities,ericsson2021selfsupervised,LiuSSLsurvey}
as well as its application to natural language processing
(NLP)~\cite{rogers-etal-2020-primer,liu2021pretrain,xia-etal-2020-bert,QiuSSLNLPsurvey}
and computer vision (CV)~\cite{JingSSLCVsurvey}.
However, none of these overviews focus on SSL for speech processing.
Since the speech signal differs greatly from image and text inputs, 
many  theories  and technologies have been developed to address the unique challenges of speech.
One review addresses speech representation learning based
on deep learning models~\cite{latif2021deep}, but does not address 
recent developments in self-supervised learning. 
This motivates this overview of speech SSL.



The structure of this paper is arranged as follows. \Cref{sec:thirdwave}
briefly reviews the history of speech representation learning, and
\cref{sec:approach} reviews current speech SSL models.
\Cref{section:benchmark} surveys SSL datasets and benchmarks, 
and discusses and compares results from different works. \Cref{analysis}
analyzes successful SSL approaches and offers insights into the
importance of technological innovations. \Cref{sec:zero} reviews 
zero-resource downstream tasks that utilize SSL. 
Finally, \cref{sec:conclusion} summarizes the paper and suggests 
future research directions.



\section{Historical Context of Representation Learning}

\label{sec:thirdwave}

In this section we present the historical background of the current surge in
self-supervised representation learning methods in the context of two previous
waves of research work in the 1990s and 2000s. The discussed approaches go
beyond speech to describe the overall landscape of ML development during the
past few decades.

\subsection{Clustering and mixture models}


Initial research in learning latent speech and audio representations involved
simple models in which the training data likelihood was optimized directly
or via the expectation--maximization (EM) algorithm.

Early work used simple clustering methods. For example, in work such
as \cite{Rabiner1979,Wilpon1985}, word patterns were clustered
semi-automatically using techniques such as k-means, after which isolated words
were recognized by finding the training cluster closest to 
  the test data.  

Through time, modeling techniques improved such that subword units were
represented by Gaussian mixture models (GMMs) \cite{Gauvain1994}, which facilitated
the modeling of more variability in the input data. GMMs were first built for
context-independent phonemes; state-clustering 
algorithms~\cite{Young1994} then resulted in GMMs for context-dependent phonemes. 
Each latent component of these mixture models acted as a template of a
prototypical speech frame, 
  making it difficult to handle            
large volumes of data with diverse characteristics. 
Furthermore, dynamical models like hidden Markov models (HMMs)~\cite{Bahl1986}
allowed for the processing of continuous speech rather than just isolated word
recognition. These generative GMM and HMM models were trained by maximizing the
likelihood of data given the model, which could be accomplished in either an
unsupervised or a supervised manner.

Another line of research focused on extracting speech features from generative
models. The main objective here was to render the knowledge learned by generative
models accessible to discriminative downstream classifiers, or to map
variable-length sequences to fixed-length representations. Feature vectors
were derived from the parameters of trained GMM models. In the case of {\em
Fisher vectors}, the features were the normalized gradients of the
log-likelihood with respect to the model parameters (mixture weights, means, and
variances) of the Gaussian mixtures. An extension of this approach (likelihood
ratio score space) used the derivative of the log-likelihood ratio of two
models, e.g., a background model and a foreground model. Examples of their use
in speech processing include speech recognition~\cite{smith01,venkata03} and
speaker recognition~\cite{wan03}. Subsequent techniques in speaker and language
verification~\cite{dehak11a,dehak11b} similarly extracted parameters
(concatenated means) from trained background GMMs as representations that were
then combined with low-rank projections of speaker/session- or language-specific
vectors.



\subsection{Stacked neural models}
\label{subsec:stack}

More recently, representation learning has seen a shift of focus towards neural models,
which, compared to GMMs and HMMs, offer distributed representations with more
capacity to model diverse input signals into efficient latent binary codes. 
Examples of early techniques include restricted Boltzmann machines
(RBM)~\cite{hinton_2006}, denoising autoencoders~\cite{DAE}, noise contrastive
estimation (NCE)~\cite{gutmann2012noise}, sparse coding~\cite{Olshausen1996,
sparse_lee, sivaram2010sparse}, and energy-based methods~\cite{Ranzato2007}.
Many of these techniques have also been applied to CV and NLP problems, which
provided inspiration for their application to speech.

Higher-capacity neural models were achieved by stacking several neural network
layers to build progressively higher-level concept representations.
However, these deeper networks also increased the training complexities. For
example, approximate training methods such as contrastive divergence
\cite{hinton_cd_2002} were a practical technique to 
  streamline RBM training.                  
Furthermore, deep networks had non-convex objective functions, which
often resulted in long training times compared to GMMs, which are trained
using full batches instead of mini-batch learning.



\subsection{Learning through pretext task optimization}



A more recent trend is learning networks that map the input to desired
representations by solving a \textit{pretext} task. Such studies have several
characteristics:
(1)~All layers are trained end-to-end to optimize a single pretext task instead
of relying on layer-wise pre-training
(2)~Past stacked networks typically had only a few layers, but 
very deep networks with more than ten layers are now common.
(3)~It is common to evaluate a representation model on a wide range of tasks.
For example, in NLP, a representation model is usually assessed on GLUE,
which comprises nine tasks~\cite{GLUE}, whereas in speech, a representation model can be
evaluated on SUPERB, which comprises ten tasks~\cite{yang21c_interspeech}, 
as described in detail in \cref{sec:benchmark}.

The cornerstone of this third wave is the design of a pretext task, which
allows the model to efficiently leverage knowledge from unlabeled data.
The pretext task should be challenging enough for the model to learn high-level
abstract representations and 
  not be so easy as to encourage the exploitation of low-level shortcuts. 
Early breakthroughs included end-to-end learning of deep neural architectures
via pretext tasks for restoring the true color of black-and-white
images~\cite{colorizing}, joint learning of latent representations and their
cluster assignments~\cite{deepcluster}, and the prediction of the relative positions of
image patches~\cite{context_pred}. Other popular approaches include variational
auto-encoders~\cite{vae, rezende2014stochastic}, which learn data
representations using unsupervised objectives by reconstructing the input
after passing it through an information bottleneck.

In the SSL context, a pretext task related to auto-encoding is to generate
an object from its partial information. Such tasks are widely used in NLP, for
example, using 
the previous tokens 
in a sentence to predict the next token such as in
ELMo~\cite{peters2018deep}, the GPT
series~\cite{alex2018GPT,alex2019GPT2,brown2020gpt3}, and
Megatron~\cite{shoeybi2020megatronlm}, or predicting the masked tokens in a
sentence such as with the bidirectional encoder representations from Transformers
(BERT) series~\cite{jacob2019BERT,Liu2019RoBERTa,albert}. 
Another common pretext task in the third wave is contrastive
learning~\cite{oord2018representation}, in which a model learns to identify a
target instance from a set of negative samples. 
This approach has become especially popular in the
CV context~\cite{pmlr-v119-chen20j,he2020momentum,chen2020improved,SwAV}.
In this survey, we will mainly focus on techniques for pretext task
optimization for speech processing, and discuss these techniques in detail
in \cref{sec:approach}.


\subsection{Other related work}
A closely related area of research that is not covered in this review is
semi-supervised pre-training methods such as pseudo-labeling (that is,
self-training). Pseudo-labeling (PL) relies on a supervised teacher model to
label a large volume of speech-only data, which is then used to augment the
initial labeled data to train a student model~\cite{Kemp1999, csl01_limsi,
ma_bbn_06, hari_1mhour}. PL has been successful and widely adopted in the
speech community since the 1990s. Other proposed variations of PL include
augmenting speech-only data with noise to improve robustness, iterating
over the PL process to improve teacher labeling quality, and training student
models with more parameters than their original teachers to capture the
complexities in vastly larger speech-only data~\cite{park2020improved,
xu2020iterative, xiao_scaling_2021}. 
Both SSL and PL leverage unlabeled speech-only data.
One distinguishing factor in PL is the utilization of supervised data for a
specific task during model pre-training, which limits the model's focus to
a single (or at best a few) downstream tasks. 
SSL, in turn, is an attempt to learn task-agnostic representations to benefit
a wide range of tasks. 

Transfer learning (TL) is another closely related area of research for
pre-training speech models. TL transfers knowledge captured by models
trained on one task to different but related tasks~\cite{caruana1997multitask}. 
The past few decades have seen active research on TL and its
extension to multitask learning for more general representations. 
Multilingual and cross-lingual supervised models have proven superior in
low-resource speech recognition tasks~\cite{Cui2015MultilingualRF}.
SSL can be regarded as a type of TL because knowledge learned from pre-training
is used for different downstream tasks.
This survey paper focuses on SSL, and not all TL technologies for speech. 
One survey indeed addresses TL for speech processing~\cite{survey_speech_TL}
but does not include current SSL technologies for speech.  










\section{Speech Representation Learning Paradigms} \label{sec:approach}
Due to the characteristics of speech, SSL pretext tasks developed for CV
and NLP may not directly apply to speech.
Below we summarize the characteristics of speech as compared to CV and NLP.
\begin{itemize}
\item \textit{Speech is a sequence.} 
Unlike CV, in which an image usually has a fixed size representation, it is
natural to represent a speech utterance as a variable-length sequence. 
Therefore, pretext tasks developed for CV cannot generally be directly
applied to speech.

\item \textit{Speech is a long sequence without segment boundaries.} 
Both text and speech can be represented as sequences. From this viewpoint, it
is natural to apply learning approaches developed for text directly to
speech. 
In NLP, morpheme-like tokens are widely used as sequence units in
pre-training. The standard BERT takes 512 morpheme-like tokens as
input, usually covering a paragraph including several sentences. 
However, speech signals consist of sound pressure measurements with thousands
of samples per second, resulting in sequences much longer than those for text. Even
spectral representations which reduce the sequence length can have hundreds of
frames per second.
Processing such sequences with typical neural network architectures like
Transformers can result in problems with running time and memory requirements. 
One could gather consecutive frames to form shorter segments,
but unlike text, there is no obvious segmentation for unlabeled
speech.
\item \textit{Speech is continuous.} 
In NLP, it is common to use a pretext task that models a categorical
distribution of masked or future inputs. Since text is easily broken down into
individual tokens such as words, subwords, or characters, it is
straightforward to define a finite vocabulary for such tasks.
However, this idea does not apply to speech modeling because speech
signals are continuous; 
  in this sense there is no such thing as a speech vocabulary.  
\item \textit{Speech processing tasks are diverse.}
Building generalizable self-supervised representation models for diverse speech
processing tasks is challenging.
Speech contains rich, hierarchical information, and different speech tasks
may require mutually orthogonal information.
For example, speech recognition requires a model that extracts content information
but ignores speaker information; in contrast, speaker recognition 
requires a model that extracts speaker information but removes content information.
Therefore, it is challenging to define a self-supervised model whose
representations are suitable for both speech recognition and speaker
recognition. Analogous considerations apply within CV and NLP.
\end{itemize}

In the sections below, we group modern SSL pretext tasks designed for speech
into three main categories: \textit{generative} approaches,
\textit{contrastive} approaches and \textit{predictive} approaches. For each
category, we provide a brief motivation and then describe prominent
models that belong to it. Lastly, we discuss a set of challenges related to the
category.
\Cref{fig:timeline} shows a timeline of the models covered in these sections
with each model colored according to our categorization. 
\Cref{table:pretext} summarizes model pretext tasks along within the categories.






\subsection{Notation}
\label{sec:notation}

To efficiently describe the different approaches, we use a simple
notation where models are taken to consist of functions $f(\cdot)$ and
$g(\cdot)$. Often, we will refer to $f(\cdot)$ as the encoder and $g(\cdot)$ as
the decoder. However, a more general view is that $f(\cdot)$ is used as the
representation model after pre-training, whereas $g(\cdot)$ is needed only to
support the pretext task. Given an acoustic input $X =\{x_1,x_2, ..., x_T\}$,
the encoder $f(\cdot)$ outputs a representation $H =\{h_1,h_2,...,h_T\}$. The
model can be fine-tuned for a downstream task directly or used to extract
features which are fed to another model, as visualized in 
\cref{fig:SSL_framework}. It is common to use the output representations~$H$,
but often representations from hidden layers of $f(\cdot)$ are more suited. The
input~$X$ may be either the raw waveform samples or a sequence of spectral
feature vectors. Below, we do not distinguish between the two;
in practice, however, both are viable options.


\begin{figure*}[t!]
    \centering
    \includegraphics[width=0.98\textwidth]{figures/model_timeline.pdf}
  
	 \caption{A selection of models listed according to first publication date
	 on arXiv or conference submission date when this clearly precedes the
	 former. The models are categorized as generative, contrastive, or predictive.
	 In addition, some models are characterized as embedding models or
	 multi-modal models, although most learn frame-level
	 representations from speech only. Some models use a mixture of generative
	 and contrastive tasks. For instance, PASE and PASE+ use a multi-task setup,
	 but find that generative tasks are the most important for downstream
	 task performance~\cite{pascual2019learning}.}
    \label{fig:timeline}
\end{figure*}


 

\subsection{Generative approaches}
\label{sec:generative}


\subsubsection{Motivation}



In this category, the pretext task is to generate, or reconstruct, the input
data based on some limited view. This includes predicting future inputs from
past inputs, masked from unmasked, or the original from some other corrupted view. 
``Generative'' as used in this paper refers to models that target the
original input in their pretext task. Note that this differs
from generative models, which learn to sample new data.


\subsubsection{Approaches}


\paragraph{Autoencoding} 
Since their introduction in the mid-1990s~\cite{hinton_94}, autoencoders (AE)
have played an essential role in learning distributed latent representations of
sensory data. 
As described above, AEs consist of an encoder and decoder; the pretext task
is to reconstruct the given input. The most common type of AE places an
information bottleneck on the latent representation simply by having fewer
hidden units available than input features. This forces the model to discard
low-level details and disccourages the learning of trivial solutions. Other models add
regularization to the latent space to further improve the quality of the
learned representations.
For instance, denoising autoencoders (DAE) learn latent representations by
reconstructing from input corrupted by noise~\cite{DAE}. 
The vector-quantized variational autoencoder (VQ-VAE) is another model in this
category~\cite{vqvae};
it extends the original variational autoencoder~\cite{vae} with a novel
parameterization of the posterior distribution for discrete latent
representations. 
Forcing the use of discrete latent representations encourages the model to
focus on salient input features, e.g., phonemes in speech, rather than using
the capacity to model channel noise and imperceptible local input details.


Specifically, the continuous representation vector $h_t$ at the output of the
encoder is quantized by mapping it to a codebook vector, which is then used as
the input to the decoder. This operation is non-differentiable and the
gradients of the encoder parameters with respect to the loss must be obtained by
approximation. In the VQ-VAE this is done using a straight-through 
estimator~\cite{bengio2013estimating}, i.e., the gradients of the encoder output are
taken to be equal to those of the decoder input. Given a learned codebook
$A\in\mathbb{R}^{K \times D}$, where $K$ is the codebook size and $D$ is the
dimensionality of each codebook vector $a_k$, the quantized representation
$q_t$ of $h_t$ is obtained as
\begin{align}
    q_t = a_k, \text{ where } k=\arg\min_j \norm{h_t - a_j}_2 .
\end{align}
The decoder $g(\cdot)$ then takes $q_t$ as input to generate $x_t$. 
Codebook learning is facilitated by a two-term auxiliary loss similar to
classical vector quantization dictionary 
learning~\cite{burton_generalization_1983, soong_vector_1985}. 
Gradients for the codebook vectors are given solely by a term that moves
codebook vectors $a_k$ closer to the non-quantized vectors $h_t$. A so-called
\emph{commitment term} is added to ensure that non-quantized vectors do not grow
unboundedly by enforcing the encoder to keep them close to a codebook vector.
This commitment term is an auxiliary loss optimized only by the encoder. The
total VQ-VAE loss is
\begin{align}
   
    \mathcal{L} = \underset{\text{encoder+decoder}}{\underbrace{\log p(x_t | q_t)}} +  \underset{\text{codebook}}{\underbrace{\norm{\mathrm{sg}\left[h_t\right] - A}_2^2}} + \beta\underset{\text{encoder}}{\underbrace{\norm{h_t - \mathrm{sg}\left[A\right]}_2^2}} \;,
    \label{eq: vector quantization losses}
\end{align}
where $\mathrm{sg}[x] = x$ is the so-called stop-gradient operator, which has
partial derivatives all equal to zero, and $\beta$ is a scalar hyperparameter.
We note that the subtraction of a vector and a matrix in \cref{eq: vector
quantization losses} implies the use of broadcasting.


These learned discrete representations have been shown to capture high-level
speech information closely related to phonemes, and are useful for other
applications such as speaker conversion~\cite{chorowski2019unsupervised}.
Vector quantization is exclusive to VQ-VAE but has seen
widespread application within SSL for regularization purposes and to define
targets for the pretext task. We will cover these applications below.


\begin{comment}
\paragraph{Regularization}
In the generative-based approaches, some regularization in the encoder $f(\cdot)$ or decoder $g(\cdot)$ is required to encourage the model to utilize global speech information and prevent naive copying input when reconstruction. 
The models below have been enhanced by regularization methods. 
\begin{itemize}
    \item DeCoAR 2.0~\cite{ling2020decoar} presents a deep contextualized acoustic representation learning approach with the addition of a vector quantization (VQ) layer.
    \item In VQ-APC~\cite{chung20e_interspeech}, a VQ layer is used with the APC objective, which imposes a bottleneck and forces the model to learn better representations.
    \item Two dropout regularization methods, attention dropout and layer dropout, are introduced to TERA~\cite{luo2021drop}. 
\end{itemize}
\end{comment}

\paragraph{Autoregressive prediction}

Autoregressive predictive coding (APC)~\cite{chung2019unsupervised,
chung2020generative} is inspired by autoregressive language models (LM) for
text, in which the model learns to use past information to predict future
information.
An encoder $f(\cdot)$ reads the input sequence $X_{[1,t]}$ and generates a
sequence of latent representations $H_{[1,t]}$.
The decoder $g(\cdot)$ is a prediction head which takes the last vector of
$H_{[1,t]}$ as input to generate $x_{t+c}$, where $c \geq 1$. Thus, $c$
indicates how many timesteps the model predicts ahead. The encoder $f(\cdot)$
and decoder $g(\cdot)$ are jointly learned to minimize the distance (e.g.,
\ensuremath{L_1}{} loss) between the decoder output $\hat{x}_{t+c} = g(h_{t})$ and $x_{t+c}$.
APC is formulated as 
   
   
   
\begin{align}
    H_{[1,t]} &= f(X_{[1,t]}) , \\
    \hat{x}_{t+c} &= g(h_{t}) \label{eq:c} , \\
    \mathcal{L}_t &= \lVert \hat{x}_{t+c} - x_{t+c} \rVert_1 .
\end{align}
In text-based autoregressive LMs, $c$ is set to $1$. However, due to the
smoothness of the speech signal, neighboring acoustic features are usually
similar. Depending on the downstream task, we are often interested in learning
so-called \emph{slow features} that typically span multiple input 
frames~\cite{wiskott2002slow}. Even the smallest linguistic units of 
speech---phonemes---span $0.07$~seconds on average in the English TIMIT 
  dataset~\cite{garofolo1993timit},   whereas spectrogram frames $\mathbf{x}_t$ are  
typically computed at $0.01$ second intervals. Thus, simply predicting the next
frame constitutes a trivial pretext task for APC; the original work finds
that $c=3$ performs well. 
 
In \cite{chung2020improved}, the APC objective is extended to multi-target
training. The new objective generates both past and future frames conditioned
on previous context. 
In VQ-APC~\cite{chung20e_interspeech}, quantization is used with the APC
objective, which imposes an information bottleneck serving as a regularizer.

A drawback of APC is that it encodes information only from previous timesteps
and not the entire input.
DeCoAR~\cite{ling2020deep} combines the bidirectionality of
ELMo~\cite{peters2018deep} and the reconstruction objective of APC to alleviate
this issue and allow encoding information from the entire input. 
It uses a forward LSTM $f_1(\cdot)$ to encode $X_{[1,t]}$ and a backward LSTM
$f_2(\cdot)$ to encode $X_{[t+k,T]}$, where $k>1$: 
\begin{gather}
    H_{[1,t]} = f_1(X_{[1,t]}), \\
    H^\prime_{[t+k,T]} = f_2(X_{[t+k,T]}), \\
    \hat{X}_{[t+1,t+k-1]} = g(h_t, h^\prime_{t+k}).
\end{gather}
The input feature vector used in the downstream tasks is the concatenation of
$h_{t}$ and $h^\prime_{t}$. 


\paragraph{Masked Reconstruction}

Masked reconstruction is largely inspired by the masked language model (MLM)
task from BERT~\cite{jacob2019BERT}. 
During BERT pre-training, some tokens in the input sentences are masked by
randomly replacing them by a learned masking token or another input token. The
model then learns to reconstruct the masked tokens from the non-masked tokens.
Recent work has explored similar pretext tasks to pre-train speech encoders.
Similar to the DeCoAR model described above, this allows a model to learn
contextualized representations that encode information from the entire input.
While we here focus on the models that reconstruct the masked input, it is
important to note that masking has also been used extensively for contrastive
(\cref{contrastive_approaches}) and predictive (\cref{predictive_approaches})
models. 

From a high-level perspective, the training phase of models using masked
reconstruction can be formulated as
\begin{align}
    H &= f(m(X)), \\
    \hat{x}_t &= g(h_{t}), \\
    \mathcal{L}_t &= \lVert \hat{x}_{t} - x_{t} \rVert_1 .
\end{align}
The function $m(\cdot)$ defines a stochastic masking policy which will be
discussed further below. The function $f(\cdot)$ is typically a Transformer
encoder~\cite{liu2020mockingjay,jiang2019improving,liu2020masked}, but
recurrent neural networks have also been used~\cite{wang2020unsupervised}. The
decoder is a prediction head. Finally, the loss $\mathcal{L}_t$ is commonly
computed only for masked timesteps in order to discourage the model from
learning an identity mapping.




The masking policies used in NLP can be adapted to speech by considering a
speech frame equivalent to a token in a sentence; indeed, the masking
strategy of BERT has also been used for speech
pre-training~\cite{liu2020mockingjay}.
However, taking inspiration from NLP is not sufficient, as speech data warrants
domain-specific considerations.
\begin{itemize}
	 \item In the standard BERT masking policy, each token is masked
	 independently at random. However, for speech, masking a single frame constitutes
	 a trivial reconstruction task. Due to the smoothness of audio signals,
	 models may learn to simply interpolate neighboring frames to
	 reconstruct the masked frame. Therefore it is common to mask chunks of
	 consecutive frames~\cite{liu2020mockingjay,jiang2021further}.
	 \item Instead of just masking a fixed number of consecutive frames,
	 pMPC~\cite{yue2021pMPC} selects masked speech frames according to the
	 phonetic segmentation in an utterance. We can bring the pretext task closer 
	 to the NLP equivalent by using a masking policy where the 	 masked regions 
	 of the input correspond to linguistic units.
	 \item Whereas most studies use masking along the temporal dimension of the
	 input, speech can also be masked along the frequency dimension when
	 spectral input features are used~\cite{wang2020unsupervised,liu2021tera}.
	 Frequency masking has been shown to improve representations used for
	 speaker classification~\cite{liu2021tera}. 
   
\end{itemize}



Some studies explore alternatives to masking the input directly. In
non-autoregressive predictive coding (NPC)~\cite{liu21l_interspeech}, time
masking is introduced through masked convolution blocks. Taking inspiration
from XLNet~\cite{XLNet}, it has also been suggested that the input be
reconstructed from a shuffled version~\cite{song20d_interspeech} to address the
discrepancy between pre-training and fine-tuning of masking-based approaches.
Regularization methods can further improve on masked reconstruction
approaches. DeCoAR 2.0~\cite{ling2020decoar} uses vector quantization,
which is shown to improve the learned representations. Furthermore, two dropout
regularization methods---attention dropout and layer dropout---are introduced
with the TERA model~\cite{luo2021drop}. 


\paragraph{More Generative Approaches}
Other than the autoregressive and masked reconstruction tasks discussed above,
various studies have explored the reconstruction of other targets derived from the
input. PASE and PASE+~\cite{pascual2019learning,ravanelli2020multi}
use multiple targets, including the waveform, log power spectrum, MFCCs, and
prosody features. Models that learn acoustic embeddings of small speech
segments have targeted future and past spectrogram
segments~\cite{chung2018speech2vec, tagliasacchi2019self, tagliasacchi2020pre},
phase information~\cite{quitry2019learning}, and the temporal gap between two
segments~\cite{tagliasacchi2019self, tagliasacchi2020pre}.





\subsubsection{Challenges} 

Although successful NLP models like BERT and GPT are based on generative pretext tasks, the progress have not been translated directly to the speech domain. A speech signal encodes more information than text, such as speaker identity and prosodic features, which makes it harder to generate. However, in order to generate all the details of the input, the model must encode all the information in the speech signal. 
Hence, a model that learns to perfectly reconstruct its input may not necessarily have learned to isolate the features of interest. 

There are many choices involved in designing a generative pretext task. For instance, the number of consecutive frames to mask. This choice influences what the model learns through the pretext task. However, there is little research on the relationship between task design and the information encoded in the learned representations.








\input{tables/pretext}




\subsection{Contrastive approaches}
\label{contrastive_approaches}
\subsubsection{Motivation}
As discussed above, speech contains many entangled features. Thus, directly modeling speech might not be the best way to discover contextualized latent factors of variations. 
Contrastive models learn self-supervised representations by distinguishing a target sample (positive) from distractor samples (negatives) given an anchor representation. The pretext task is to minimize latent space distances between anchor and positive samples while maximizing distance to negative samples. This approach has been extensively used in the machine learning and computer vision communities \cite{Schultz_Joachims_NeurIPS2004}. 

\subsubsection{Approaches}
\paragraph{CPC}
Contrastive Predictive Coding (CPC)~\cite{oord2018representation} is a prominent example of a constrastive learning model. In CPC, the anchor representation $h_t$ is contextualized by depending on the input up until timestep $t$ whereas the positives and negatives are sampled from a set of localized representations $z_t$ jointly learned by the model.
\begin{align}
    Z &= f_{1}(X) \label{eq: cpc local representation} \enspace ,\\
    h_{t} &= f_{2}(Z_{[1,t]}) \enspace , \\
    \hat{z}_{t,k} &= g_k(h_{t}) \enspace,
\end{align}
\noindent where $f_1(\cdot)$ is a convolutional neural network, such that each $z_{t}$ only encodes information from a limited receptive field in $X$, $f_2(\cdot)$ is limited to condition each $h_t$ only on previous timesteps $Z_{[1,t]}$ and $g_k$ is a step-wise transformation (e.g., a linear regression layer). 
The loss function measures the distance from projections of the contextualized representation, $\hat{z}_t$, to learned representations $z_t$.
The approach is similar to previous work on Noise-Contrastive Estimation (NCE)~\cite{gutmann2010noise}. The loss corresponds to maximizing a lower bound on the mutual information between $h_t$ and $z_t$ (and in turn $x_t$) and is hence called InfoNCE.
\begin{align}
    \mathcal{L}_{t,k} &= - \log \left(\frac{\exp(\hat{z}_{t,k}^{\text{\tiny T}}z_{t+k})}{\sum_{n \sim \mathcal{D}} \exp(\hat{z}_{t,k}^{\text{\tiny T}}z_{n})} \right)\enspace .
    \label{eq: cpc loss}
\end{align}
\noindent Here, $\mathcal{D}$ is a set of indices including the target index $t+k$ and negative samples drawn from a proposal distribution, which can be taken to be a uniform distribution over the set $\{1,\dots,T\}$. This corresponds to sampling negatives from the same sequence which has been show to give good performance for phoneme classification~\cite{oord2018representation}. The loss is indexed by $k$ to show that CPC targets multiple offsets using different projection layers. The authors find $k=12$ to work well for phoneme classification.

The wav2vec model \cite{schneider2019wav2vec} extends the CPC approach with different parameterizations for the functions $f_1(\cdot)$ and $f_2(\cdot)$. It also modifies the loss to consider a binary prediction task, such that we have
\begin{align}
   
    \mathcal{L}_{t,k} &= - \log(\sigma(\hat{z}_{t,k}^{\text{\tiny T}}z_{t+k})) - \sum_{n \sim \mathcal{D}} \log(\sigma(-\hat{z}_{t,k}^{\text{\tiny T}}z_{n})) \enspace ,
    \label{eq: wav2vec loss}
\end{align}
where $\sigma$ is the sigmoid function.

\paragraph{wav2vec 2.0} 
The wav2vec 2.0 model combines masking and contrastive learning. As the CPC model, it uses the InfoNCE loss \cite{oord2018representation} to minimize the distance between contextualized representations and quantized target vectors. 
It takes as input a waveform and uses a convolutional feature encoder followed by a transformer network. 
Masking is applied to the convolutional features at the transformer input: 
\begin{align}
    Z &= f_1(X) \label{w2v2 f_v} \enspace , \\
    H &= f_2(m(Z)) \label{w2v2 f_c} \enspace , \\
    q_t &= g(z_t) \enspace . \label{w2v2 qtz}
\end{align}

\noindent Here, $m(\cdot)$ again defines a masking policy, 
$f_1(\cdot)$ is a convolutional neural network with limited receptive field, $f_2(\cdot)$ is a transformer encoder, and $g(\cdot)$ is a quantization module which uses a Gumbel softmax~\cite{Gumbel-Softmax} with a straight-through estimator to form discrete targets from $Z$.

In wav2vec 2.0, anchors are taken to be $h_t$ at masked timesteps only, the positive sample is chosen as the quantized vector at the same timestep, $q_t$, and negatives are sampled from other masked timesteps. Notably, quantized targets allow excluding the class of the positive sample $q_t$ when sampling negatives. The loss is
\begin{align}
    \mathcal{L}_t &= - \log \left(\frac{\exp(S_{\text{c}}(h_{t}, q_{t}))}{\sum_{n \sim \mathcal{D}} \exp(S_{\text{c}}(h_{t}, q_{n}))} \right) \enspace , \label{w2v2 loss}
\end{align}
\noindent where $S_{\text{c}}(\cdot)$ is the cosine similarity and $\mathcal{D}$ contains $t$ and the indices sampled from other masked timesteps. 

Since the quality of the learned representations is contingent on the quality of the quantization, wav2vec 2.0 combines two techniques to learn high-quality codebooks. First, wav2vec 2.0 concatenates quantized representations from multiple codebooks at each timestep, so-called Product Quantization (PQ)~\cite{ProductQuantization}. Also, the training loss is augmented with an auxiliary term designed to encourage equal use of all codebook entries. The wav2vec 2.0 approach was the first to reach single-digit WER on all Librispeech and Libri-light subsets (see \cref{sec:datasets}). Its impressive performance inspired many follow-up research studies. The wav2vec-C~\cite{sadhu21_interspeech} approach extends wav2vec 2.0 with a consistency term in the loss that aims to reconstruct the input features from the learned quantized representations, similar to VQ-VAE~\cite{VQVAE2}. The wav2vec-C representations achieve, on average, twice the error reduction over baseline and a higher codebook utilization in comparison to wav2vec 2.0 for far-field English voice commands and voice queries. 


\subsubsection{Challenges}
Although representations learned using contrastive approaches have proved effective across a wide range of downstream applications, they face many challenges when applied to speech data. 
One challenging aspect is that the strategy used to define positive and negative samples can also indirectly impose invariances on the learned representations. For example, sampling negatives exclusively from the same utterance as the positive biases the features towards speaker invariance, which may or may not be desired for downstream applications. 
Another standing challenge is that since speech input does not have explicit segmentation of acoustic units, the negative and positive samples do not represent a whole unit of language but rather partial or multiple units, depending on the span covered by each sample. 
Finally, since speech input is smooth and lacks natural segmentation, it can be difficult to define a contrastive sampling strategy that is guaranteed to provide samples that always relate to the anchor as truly positives and negatives in a sound way.

\subsection{Predictive approaches}
\label{predictive_approaches}
\subsubsection{Motivation}
Unlike generative representation learning approaches, predictive methods output a distribution over a discrete vocabulary given a masked input utterance. The predictive loss is only applied over the masked regions, forcing the model to learn good high-level representations of unmasked inputs to infer the targets of masked ones correctly. These approaches are motivated by the success of BERT-like methods in NLP \cite{jacob2019BERT} where the model learns contextual representation aggregating evidence from distant timesteps. Predictive approaches for speech representation learning also draw inspiration from the DeepCluster method \cite{Caron_2018_ECCV} for self-supervised visual learning. 

\subsubsection{Approaches}

\paragraph{Discrete BERT} Applying BERT training for spoken input is challenged due to the continuous nature of the speech. The DiscreteBERT approach~\cite{baevski2019effectiveness} benefited from a fixed discrete vocabulary previously learned by a contrastive vq-wav2vec \cite{Baevski2020vq-wav2vec} model. For the Librispeech dataset, the discrete vocabulary contained about 13.5k unique codes. Concretely, given an input spoken utterance, the discrete unit sequence is first extracted from the vq-wav2vec model and then used for input and target sides of a standard BERT model. During learning, only the BERT model is trained using the masked-prediction loss -- with no gradient updates done for the vq-wav2vec model. 
Following the Libri-light near-zero evaluation policy \cite{kahn2020libri}, fine-tuning is done using 10 minute, 1 hour and 10 hour subsets of Librispeech balanced for gender and recording conditions, as well as the standard 100 hour-clean part of Librispeech. DiscreteBERT was the first model to show the effectiveness of self-supervised speech representation learning by achieving a WER of 25\% on the standard test-other subset using merely a 10-minute fine-tuning set, which opened the gate for more ambitious approaches to follow.

\paragraph{HuBERT} Rather than relying on an advanced representation learning model for discretizing continuous spoken inputs, the Hidden Unit BERT (HuBERT) approach \cite{hsu2021hubert} examines the effectiveness of using the classic k-means{} units trained on MFCC features. Also, different from the DiscreteBERT approach, the HuBERT model consumes continuous input waveforms (or log-mel features), rather than discrete units to ensure that the SSL model gets the complete input speech information needed for pre-training and fine-tuning.
Specifically, HuBERT predicts the predetermined k-means{} cluster assignment given the masked continuous speech features. Similar to the wav2vec 2.0, the HuBERT model consumes continuous waveform inputs using a convolutional encoder. Masking is then applied before the transformer network as in wav2vec 2.0 \namecrefs{w2v2 f_v} \ref{w2v2 f_v} and \ref{w2v2 f_c}.

Since the targets are pre-computed cluster identities, the HuBERT model can directly evaluate the regular cross-entropy loss between the correct k-means{} cluster and the predicted one. This is opposed to constrastive methods which require negative samples to avoid degenerating to trivial solutions. The loss is computed on both masked, $\mathcal{L}_m$, and unmasked $\mathcal{L}_u$ timesteps, 
\begin{align}

   
    \mathcal{L}_m &= \sum_{t\in M} - \log p(c_t \mid X) \enspace , \\
    \mathcal{L} &= \alpha
    \mathcal{L}_m + (1 - \alpha) \mathcal{L}_u \enspace ,
\end{align}
\noindent where $c_t$ is the k-means{} cluster of timestep $t$, $M$ is the set of all masked timesteps and $\mathcal{L}_u$ is computed as $\mathcal{L}_m$ but instead summing over $t\notin M$. 
Intuitively, the HuBERT model is forced to learn both acoustic and language models from continuous inputs. First, the model needs to model unmasked inputs into meaningful continuous latent representations, which maps to the classical frame-based acoustic modeling problem. Second, the model needs to capture the long-range temporal relations between learned representations to reduce the prediction error. One crucial insight motivating this work is the importance of consistency of the targets, not just their correctness, which enables the model to focus on modeling the sequential structure of input data. 

The HuBERT approach further refines the target units using its own learned representations. After training the initial transformer model, latent HuBERT features are quantized using k-means{} to provide target vocabulary for a second iteration of pre-training. Alternating between learning offline discrete labels using k-means{} and training a transformer model using masked prediction ensures training stability while preserving representation quality. Only two iterations were needed for the HuBERT approach to match or better the previous SOTA score on Librispeech and Libri-light evaluation subsets for high and ultra-low resource training conditions.

Getting the best of both worlds, the w2v-BERT approach~\cite{w2vbert} optimizes the wav2vec 2.0 contrastive loss and the HuBERT masked prediction loss simultaneously. It showed 5\% to 10\% relative WER reduction on the test-clean and test-other subsets of Librispeech, and
more than 30\% relative improvement for real-world in-house voice search compared to a wav2vec 2.0 baseline. 

\paragraph{WavLM} Based on the HuBERT framework for SSL, WavLM emphasizes spoken content modeling and speaker identity preservation~\cite{chen2021wavlm}. First, it extends the transformer self-attention with a content-based gated relative position bias which improves the model's capability on recognition tasks compared to the convolutional context embedding using in HuBERT and wav2vec 2.0 models. The gates adaptively adjust the relative position bias by conditioning on the current speech content. Intuitively, the same distance offset between two frames tends to play different roles if one frame is silent and the other belongs to a speech segment. WavLM uses 320 bucketed positions for the range of possible offsets. The relative positional embedding parameters are shared across all layers.

Most SSL methods are trained on single speaker data; therefore, they perform poorly on multispeaker tasks like separation and diarization. The WavLM framework proposed an utterance mixing strategy where partially overlapped signals from different speakers are constructed to augment the training data. Mixed utterances are selected from the same training minibatch, then randomly cropped and scaled by a sampled source energy ratio. The overall overlapping region per utterance is capped to be less than 50\%. The content information corresponding to the main speaker is used as the target during the masked prediction pre-training.

The pre-training data used for WavLM extends the sources used for HuBERT and wav2vec 2.0 to reach a total of 94k hours of public audio. On the SUPERB benchmark, WavLM achieved SOTA on speech separation, speaker verification, and diarization. It also improved upon HuBERT and wav2vec 2.0 models across all ten tasks of the SUPERB benchmark while maintaining a comparable performance on ASR. 

\paragraph{data2vec} Motivated by the success of an exponential moving average (EMA) teacher for self-supervised visual representations~\cite{grill2020byol, dino}, data2vec~\cite{data2vec} predicts contextualized latent input representations given the masked view of the input. The prediction targets $y_t$ are continuous representations generated by encoding unmasked inputs using an EMA of the same model. They are computed by averaging instance-normalized~\cite{instance_norm} representations of the top $k$ transformer blocks of the teacher network.  Data2vec uses the smoothed \ensuremath{L_1}{} loss, which is less sensitive to outliers and prevents exploding gradients~\cite{fastrcnn},
\begin{equation}
\mathcal{L}(y_t, h_t) =
    \begin{cases}
      \frac{1}{2}(y_t - h_t)^2 / \beta\text{,} & |y_t - h_t| \leq \beta\\
      |y_t - h_t| - \frac{1}{2}\beta\text{,} & \text{otherwise}
    \end{cases} \enspace ,
\end{equation}
\noindent where the contextual representations $h_t$ are computed similarly to wav2vec 2.0 and HuBERT. The $\beta$ hyperparameter controls the transition from a squared loss to an \ensuremath{L_1}{} loss. 

The data2vec approach was shown to work well for speech representation learning and visual and text representations. It is the first approach to achieve competitive results when trained on either one of the three modalities. 

\subsubsection{Challenges}
The iterative nature of pre-training for the HuBERT and wavLM predictive models could present a practical inconvenience when working with large volumes of data. High-dimensional latent representations are generated from the previous model iteration to produce targets for the next one. Another challenge centers around the quality of the initial vocabulary generated from MFCC features. Although masked prediction loss enables the SSL models not to be trapped by the quality of their initial target units, experiments show that initial teachers with higher quality, e.g., Gaussian Mixture Models (GMMs) instead of k-means{}, provide non-negligible performance gains. The data2vec approach improves over other predictive models since its continuously improving teacher provides progressively better targets; however, two-tower approaches present a computational challenge for enormous model sizes unless model sharding approaches are utilized.

\subsection{Learning from multi-modal data}
\subsubsection{Motivation}
Multiple modalities are useful in many settings, where each modality provides information that is complementary to other modalities.  Multi-modal work includes supervised settings, such as audio-visual ASR~\cite{petajan1984automatic,potamianos2003recent} and person identification~\cite{aleksic2006audio} which have been studied for decades.  In this section, we focus only on unsupervised representation learning from multi-modal data.

One of the motivations for learning from multiple modalities is that it can reduce the effect of noise, since noise in different modalities is likely to be largely independent or uncorrelated.  In addition, learning from speech data with accompanying signals such as images or video can help learn representations that encode more semantic information. Such ``grounding" signals can contain supplementary information that can be used by models to infer the content of the speech. Human language learning provides a proof of concept for this, as it is believed that infants benefit from the visual modality when learning language~\cite{legerstee1990infants}. Early computational models of multi-modal language learning were motivated by (and tried to emulate) human learning of language in the context of the visual surroundings~\cite{roy1999learning}.

\subsubsection{Approaches}
We define two broad classes of approaches in this area. Specifically, depending on what type of multi-modal data is involved we refer to ``intrinsic" or ``extrinsic" modalities.

\paragraph{Intrinsic modalities} refer to modalities produced directly by the speech source.  Examples of intrinsic modalities (besides the speech audio) include images or video of the speaker's face~\cite{lee2004avicar,chung2016lip}, lip-movement~\cite{shi2022learning}, articulatory flesh point measurements~\cite{westbury1990x,wrench2001new}, or simultaneous MRI scans~\cite{narayanan2011multimodal}.  Typically, learning from multiple intrinsic modalities is done so as to improve robustness to noise, since acoustic noise is likely to be uncorrelated with the other modality(s).  This type of representation learning is often referred to as ``multi-view learning" because the multiple intrinsic modalities can be seen as multiple views of the same content. Some typical approaches in this category include
\begin{itemize}
    \item Multi-view autoencoders and variations~\cite{ngiam2011multimodal,badino2016integrating},
    \item Multi-modal deep Boltzmann machines~\cite{srivastava2012multimodal},
    \item Canonical correlation analysis (CCA)~\cite{hotelling1936relations} and its nonlinear extensions~\cite{andrew2013deep,wang2015deep,wang2016deep,michaeli2016nonparametric,Melzer_01a,LaiFyfe00a,LaiFyfe99a,BachJordan05a,wang_15a},
    \item Multi-view contrastive losses~\cite{HermanBlunsom14a,Huang_13b},
    \item More recently, audio-visual extensions of masked prediction methods, specifically Audio-Visual HuBERT (AV-HuBERT)~\cite{shi2022learning, shi2022robust}.
\end{itemize}

\paragraph{Extrinsic modalities} refer to modalities that are not produced by the same source but still provide context for each other. A typical example is an image and its spoken caption: The image tells us what the speech is likely describing, so a representation model that takes both modalities into account will hopefully encode more of the meaning of the speech than a single-modality model. 
There has recently been an surge of datasets collected for this purpose, usually consisting of images and spoken captions, the audio and image frames in a video, or video clips with their spoken descriptions. A recent review of datasets, as well as methods, in this category is provided by Chrupala~\cite{chrupala2021visually}. 

Typical approaches involve learning a neural representation model for each modality, with a multi-modal contrastive loss that encourages paired examples in the two modalities to have similar representations while unpaired examples remain different~\cite{synnaeve+etal_nipsworkshop14,harwath+etal_nips16,harwath+glass_asru15,merkx2019language,rouditchenko2020avlnet,peng2022fast}. Other choices include training with a masked margin softmax loss~\cite{ilharco2019large,sanabria2021talk}.  Such models are typically evaluated on cross-modal retrieval, although some work has also used the models for other downstream tasks such as the ZeroSpeech and SUPERB benchmark tasks~\cite{peng2022self}. Analyses of such models have found that, despite the very high-level learning objective of matching speech with a corresponding image (or other contextual modality), such models often learn multiple levels of linguistic representations from the shallowest to the deepest model layers~\cite{harwath2019learning,chrupala2017representations,scharenborg+etal_icassp18}.  They are also able to learn word-like units~\cite{harwath2018jointly,peng2022word,wang2020dnn} and can be used for cross-lingual retrieval, by considering the visual signal as an ``interlingua"~\cite{harwath2018vision,havard2019models,kamper+roth_sltu18}. In some settings, even in the presence of some amount of textual supervision (i.e., the speech is transcribed), visual grounding still helps learn a better representation for retrieval~\cite{pasad2019contributions}. 

There has also been growing interest in learning joint speech and text representations using paired and unpaired data. The SLAM approach~\cite{bapna2021slam} is an example where speech and text are first represented using two separate pre-trained encoders followed by a multi-modal encoder to build the joint representations. The entire model is trained using a multi-task loss including two supervised and two self-supervised tasks. 
    


\subsubsection{Challenges}
One of the challenges of using multi-modal approaches is that the multi-modal data they rely on is often in shorter supply than single-modality data. In addition, multi-modal data is typically drawn from specific domains, for example domains involving descriptions of visual scenes. It is not clear how well the learned speech representations apply to other speech domains that are not necessarily describing or situated in a visual scene, and this question requires further study.

\subsection{Acoustic Word Embeddings}


Most of the representation learning techniques discussed in the preceding sections are aimed at learning frame-level representations. For some purposes, however, it may be useful to explicitly represent longer spans of speech audio of arbitrary duration, such as phone, word, or phrase-level segments.  For example, searching within a corpus of recorded speech for segments that match a given (written or spoken) query can be seen as finding segments whose representations are most similar to that of the query~\cite{levin+etal_icassp15,guoguo+etal_icassp15,Chung2016AudioWord2Vec,settle2017query}; word embeddings can be defined by pooling representations of instances of a given word~\cite{chung2018speech2vec}; unsupervised segmentation and spoken term discovery can be seen as a problem of detecting and clustering segments~\cite{kamper2017embedded,Kamper17BESGMM}; and even ASR can be viewed as the problem of matching written word representations to representations of audio spans~\cite{maas+etal_icmlwrl12,bengio+heigold_interspeech14,settle2019_a2w}.  

Several lines of work have begun to address the problem of learning representations of spans of speech, especially word segments, typically referred to as {\it acoustic word embeddings}.  Early work on unsupervised acoustic word embeddings defined them as vectors of distances from the target segment to a number of pre-defined ``template" segments~\cite{levin+etal_asru13}. Later work used variants of neural autoencoders~\cite{Chung2016AudioWord2Vec,holzenberger2018learning,kamper2019truly,pengcorrespondence}. These are often evaluated on word discrimination, that is the task of determining whether two word segments correspond to the same word or not~\cite{carlin2011rapid}. This task can be thought of as a proxy for query-by-example search, since the basic operation in search is to determine whether a segment in the search database matches a query segment, and has been used for evaluation of both frame-level (e.g.,~\cite{kamper2015unsupervised}) and word-level~\cite{levin+etal_asru13,kamper2016deep} representations.  


Since most work on acoustic word embeddings preceded the very recent wave of new self-supervised frame-level representations, one question is whether word (or more generally segment) embeddings could be derived more simply by pooling self-supervised frame-level representations, as has been done for text span embeddings by pooling over word embeddings~\cite{toshniwal2020cross,wang2021phrase}. Some initial results suggest that at least very simple pooling approaches like downsampling and mean or max pooling are not successful~\cite{van2021comparison,pengcorrespondence}, but more work is needed to reach conclusive results.









\section{Benchmarks for Self-supervised Learning Approaches}
\label{section:benchmark}





The previous sections presented various methodologies by which to learn speech
representations from unlabeled corpora. This section surveys the datasets
available to learn and evaluate these representations. We also summarize
several studies and their results to demonstrate the usefulness of the learned
representations for various downstream tasks. 

\subsection{Datasets only for pre-training} 
\Cref{table:datasets} summarizes datasets used for pre-training SSL techniques
in the literature. These datasets are usually large but with limited or no
labels. Libri-light (LL)~\cite{kahn2020libri}, one of these datasets, is
derived from audiobooks that are part of the
LibriVox\footnote{https://librivox.org/ \label{librivox}} project. LL contains
60k hours of spoken English audio tagged with SNR, speaker ID, and genre
descriptions. The speech examples in Audioset~\cite{gemmeke2017audio}, which
consists of over 2M 10-second YouTube video clips human-annotated with 632
audio events, have also been used for pre-training. Audioset has 2.5k hours
of audio of varying quality, different languages, and sometimes multiple sound
sources. AVSpeech~\cite{ephrat2018looking} is another large-scale audio-visual
dataset used in SSL research, comprising 4.7k hours of clips from a wide
variety of languages. 
Each clip contains a visible face and audible sound originating from a single
speaker without interfering background signals. The 3100-hour audio part of
AVSpeech has been used to learn audio-only 
representations~\cite{kawakami2020learning}. The Fisher corpus~\cite{cieri2004fisher} collects
over 2k hours of conversational telephone speech, 1k hours of which is utilized
for pre-training~\cite{jiang2021further}. Industrial researchers have also
begun to build large-scale datasets for learning speech representations.
For instance, 10k hours of real-world far-field English voice commands for
self-supervised pre-training have been collected at 
Amazon~\cite{sadhu21_interspeech}. 

In addition to these English and multilingual efforts, researchers have also
collected corpora for pre-training Chinese speech representations. Didi Dictation
and Didi Callcenter~\cite{jiang2019improving, jiang2021further} are two
internal datasets containing respectively 10k hours of read speech collected
from mobile dictation application and 10k hours of spontaneous phone calls
between users and customer service staff.

\subsection{Datasets for both pre-training and evaluation}\label{sec:datasets}
Several datasets that provide both speech and associated transcripts and
speaker labels have also been used to develop SSL techniques by enabling
in-domain pre-training and evaluation. Such datasets are also listed in
\cref{table:datasets}. One of the most commonly used datasets in this category
is LibriSpeech (LS)~\cite{panayotov2015librispeech}, a labeled corpus
containing 960 hours of
read English speech, which is also derived from an open-source audiobooks
project.\textsuperscript{\ref{librivox}} The corpus consists of subsets
\textit{train-clean-100}, \textit{train-clean-360}, \textit{train-other-500},
\textit{dev-clean}, \textit{dev-other}, \textit{test-clean}, and
\textit{test-other} used for training, development, and testing, respectively.
Subsets tagged with \textit{other} are more challenging utterances from
speakers that yield higher WER as measured with previously built models. LS is
used for unsupervised representation pre-training by ignoring its labels, and
can also be utilized to evaluate the performance of representation on ASR, PR,
PC, and SID tasks. Wall Street Journal (WSJ)~\cite{paul1992design} is another
widely adopted, labeled corpus for pre-training. Its labels can evaluate
performance for ASR, PR, PC, and SID. The original WSJ corpus contains 400 hours
of English read speech data, and today its \textit{si284} (81 hours),
\textit{dev93}, \textit{eval92} subsets are the most-used partitions for
unsupervised training, development, and test, respectively.  The \textit{si84}
(15 hours) partition is also used for training.

The speech community also utilizes multilingual corpora. These are often
large-scale, which facilitates pre-training, but are also partially labeled for ASR
evaluation (PC and PR can be enabled via phone-level forced alignment). These
corpora include Common Voice (CV)~\cite{ardila2020common}, Multilingual
LibriSpeech (MLS)~\cite{pratap20_interspeech}, VoxPopuli 
(VP)~\cite{wang-etal-2021-voxpopuli}, and BABEL (BBL)~\cite{gales2014speech}. CV is
an open-source, multi-language, growing dataset of voices containing 11k hours
of audio from 76 languages as of the date this review was written (Common Voice
corpus~7.0). Researchers usually use part of this for pre-training (e.g., 7k
hours/60 languages in \cite{babu2021xlsr} and 430 hours/29 languages in
\cite{kawakami2020learning}) or evaluation. 
MLS derives content from read audiobooks of LibriVox and contains data in eight
European languages for a total of 50k hours of audio. VP comprises a total of 400k
hours of parliamentary speech from the European parliament in 23 European
languages.
The entire dataset~\cite{babu2021xlsr} or a 24k-hour 
portion~\cite{chen2021unispeechsat, chen2021wavlm} thereof has been used for pre-training. BBL
consists of 1k hours of conversational telephone speech in 17 African and Asian
languages.

Several datasets, including GigaSpeech~\cite{chen21o_interspeech}, TED-LIUM~3
(TED3)~\cite{hernandez2018ted}, TED-LIUM~2 (TED2)~\cite{rousseau2012ted},
Switchboard (SWB)~\cite{godfrey1992switchboard}, 
TIMIT~\cite{garofolo1993timit}, and VoxLingua107~\cite{valk2021voxlingua107}, are
labeled and conventionally used for evaluation, while their audio streams are
also aggregated to build diversified and large-scale corpora for unsupervised
pre-training~\cite{kawakami2020learning, song20d_interspeech, babu2021xlsr}.
GigaSpeech is a multi-domain English ASR corpus with 33k hours of audio
collected from audiobooks, podcasts, and YouTube. A subset of 10k audio is
transcribed. TED2 comes with 118 hours of English speech extracted from
TED conference talks and its transcription for evaluating ASR. Its recordings
are clear but with some reverberation. TED3 is an extension of TED2 and
comprises 450 hours of talks. SWB is a 260-hour conversational speech
recognition dataset containing two-sided telephone conversations.
The TIMIT corpus was designed to provide read speech data and its word and
phone-level transcriptions for acoustic-phonetic studies. It contains recordings
in American English. Compared to the previous corpora labeled for ASR
evaluation, VoxLingua107 consists of 6.6k hours of audio in 107 languages and
is annotated for language identification. Beyond the original purpose of
evaluation, these corpora are also used in pre-training to improve the
generalizability of learned representations.


For the purpose of pre-training and evaluating Mandarin speech representations,
the authors of \cite{jiang2019improving, jiang2021further} also compiled Open Mandarin, 
an open-source Mandarin dataset of 1.5k hours of speech from
the Linguistic Data Consortium (LDC) and
OpenSLR.\footnote{\url{https://openslr.org}} Open Mandarin consists of the HKUST
Mandarin Telephone Speech Corpus (HKUST, 200 hours of spontaneous 
speech, of which  
168 hours of audio is used for pre-training; the development and test sets are
excluded.)~\cite{liu2006hkust}, AISHELL-1~\cite{bu2017aishell} (178 hours of
read speech),
aidatatang 200zh (200 hours, read speech)~\cite{aidatatang}, MAGICDATA
Mandarin Chinese Read Speech Corpus (755 hours, read speech)~\cite{magic_data},
Free ST Chinese Mandarin Corpus (ST-CMDS, 100 hours, read speech)~\cite{st_cmds},
and Primewords Chinese Corpus Set 1 (100 hours, read speech)~\cite{primewords_201801}. 
Both HKUST and AISHELL-1 are labeled and are suitable for ASR evaluation.


\subsection{Datasets for evaluation}

Besides the aforementioned datasets, conventional speech processing benchmarks
are also used to evaluate self-supervised representations. Studies leverage
  Hub5,    
DIRHA, and CHiME-5 to measure the efficacy of representations in ASR.
The Hub5 evaluation (LDC2002T43 and LDC2002S09, also referred to as the NIST 2000
Hub5 English evaluation set) contains 40 transcribed English telephone
conversations only for testing, where 20 are from conversations collected in
SWB studies but not released with the SWB dataset, and the rest are from CallHome
American English Speech (LDC97S42). DIRHA~\cite{ravanelli2015dirha}, short for
Distant-speech Interaction for Robust Home Applications, is a database composed
of utterances sampled from WSJ, speech of keywords and commands, and
phonetically-rich sentences. These utterances are read by UK and US English
speakers and recorded with microphone arrays. CHiME-5~\cite{barker2018fifth} is
a challenge that aims to advance robust ASR and presents a dataset of natural
conversational speech collected under a dinner party scenario with microphone
arrays. A team at Amazon Alexa also recorded and transcribed a corpus of 1k
hours of audio for model training and evaluation~\cite{sadhu21_interspeech}.

Researchers also evaluate representations for sentiment analysis with the
INTERFACE~\cite{hozjan2002interface} and MOSEI (CMU Multimodal Opinion
Sentiment and Emotion Intensity)~\cite{zadeh2018multimodal} datasets. INTERFACE
is an emotional speech database for Slovenian, English, Spanish, and French,
and contains six emotions: anger, sadness, joy, fear, disgust, and
surprise, plus neutral. MOSEI is comprised of sentence-level sentiment
annotations of 65 hours of YouTube videos
using emotion categories similar to INTERFACE, but replacing joy with
happiness.

In addition, datasets employed to demonstrate the benefit of SSL
representations on various tasks include SID/ASV 
(VCTK~\cite{veaux2016superseded} and VoxCeleb1~\cite{nagrani17_interspeech}), IC
(Fluent Speech Commands, or FSC \cite{lugosch19_interspeech}), QbE (QUESST
2014, or QUESST~\cite{anguera2015quesst2014}), ST (LS 
En-Fr~\cite{kocabiyikoglu2018augmenting} and CoVoST-2~\cite{wang2020covost}), and
multilingual ASR (ALFFA and OpenSLR-multi). The VCTK corpus includes speech
data with 109 English speakers of various accents, each reading out about 400
sentences sampled from newspapers. VoxCeleb1 is an audio-visual dataset
comprised of short YouTube clips containing human speech. It consists of 1251
unique speakers and 352 hours of audio. FSC contains utterances of spoken
English commands that one might use for a smart home or virtual assistant, and
is used to evaluate the performance of a spoken language understanding system.
The QUESST search dataset comprises spoken documents and queries in 6~languages
to measure the capability of models in spotting spoken keywords from documents.
LS En-Fr is a dataset augmenting existing LS monolingual utterances with
corresponding French translations to train and evaluate English-French machine
translators. CoVoST-2 is a multilingual speech translation benchmark based on
CV. It provides data for translating from English into 15 languages and from 21
languages into English, and has a total of 2.9k hours of speech. The ALFFA
project\footnote{\url{http://alffa.imag.fr}} collects speech of African
languages to 
  promote   
the development of speech technologies in Africa, and
\cite{kawakami2020learning} leverages four African languages collected in the
project for evaluation: Amharic~\cite{tachbelie2014using}, Fongbe~\cite{laleye2016first},
Swahili~\cite{gelas2012developments}, and Wolof~\cite{gauthier2016collecting}.
In the same work~\cite{kawakami2020learning}, the authors
further select 21 phonetically diverse languages from OpenSLR to evaluate the
generalizability of SSL representations across languages. We denote the
collection as OpenSLR-multi below.

Last, \cite{tagliasacchi2020pre} puts together five datasets 
(MUSAN~\cite{snyder2015musan}, Bird Audio Detection~\cite{stowell2019automatic},
Speech Commands~\cite{warden2018speech}, Spoken Language 
Identification~\cite{oponowiczspoken}, and TUT Urban Acoustic Scenes 2018~\cite{dcase2018}) plus
an SID task built with the LS \textit{train-clean-100} subset to evaluate the
capability of representations on audio event detection.
\cite{quitry2019learning} employs the NSynth dataset~\cite{engel2017neural} on top
of the six for benchmarking. As many of the datasets are built for research in
audio processing, we here provide only a list of these datasets for reference.



\begin{table*}[ht]
  \centering
  \scriptsize
  \caption{Summary of datasets used in pre-training (denoted as PT) or
  evaluating (denoted as EV) SSL techniques in the literature. The languages and
  sizes of the datasets are provided in columns 3 and 4. Column 5 lists the tasks each
  dataset is used to evaluate. We use the following abbreviations:
  \textbf{EN}: English; 
  \textbf{Multi}: multilingual;    
  \textbf{ZH}: Chinese;
  \textbf{Fr}: French; \textbf{ASR}: automatic speech recognition; \textbf{PR}:
  phoneme recognition; \textbf{PC}: phoneme classification; \textbf{SID}:
  speaker identification; \textbf{ASV}: automatic speaker verification;
  \textbf{Sentiment}: sentiment analysis; \textbf{ST}: speech translation;
  \textbf{QbE}: query by example or spoken term detection; \textbf{IC}: intent
  classification; \textbf{AED}: audio event detection; and \textbf{LID}: language
  identification. We distinguish \textbf{PR} from \textbf{PC} based on whether
  the inference is made at the phone level sequentially or the frame level separately.
  \textbf{SID} and \textbf{ASV} both evaluate model capability in encoding
  speaker information; \textbf{SID} classifies one utterance into a pre-defined
  set of speaker labels, whereas \textbf{ASV} infers whether a given pair of
  utterances was uttered by the same speaker.}
  \label{table:datasets}
  {\renewcommand*\arraystretch{1.2}
  \begin{tabular}{llllll}
    \toprule
    Dataset & Purpose & Lang. & Size (hrs) & Task & License \\
    \midrule
   
    LibriLight (LL) & PT & EN & 60k & - & MIT License \\ \hline
    AudioSet & PT & Multi & 2.5k & - & CC BY 4.0 \\ \hline
    AVSpeech & PT & Multi & 3.1k & - & CC BY 4.0 \\ \hline
    Fisher & PT & EN & 2k/1k \cite{jiang2021further}  & - & Linguistic Data Consortium (LDC) \\ \hline

    Alexa-10k & PT & EN & 10k & - & Not released \\ \hline
    Didi Callcenter & PT & ZH & 10k & - & Not released \\ \hline
    Didi Dictation & PT & ZH & 10k & - & Not released \\ \hline

    LibriSpeech (LS) & PT/EV & EN & 960 & ASR/PR/PC/SID & CC BY 4.0 \\ \hline
    Wall Street Journal (WSJ) & PT/EV & EN & 81 & ASR/PR/PC/SID & Linguistic Data Consortium (LDC)  \\ \hline

    Common Voice (CV-dataset) & PT/EV & Multi & 11k/7k \cite{babu2021xlsr}/430 \cite{kawakami2020learning} & ASR/PR/PC & CC0 \\ \hline
    Multilingual LS (MLS) & PT/EV & Multi & 50k hrs & ASR & CC BY 4.0 \\ \hline
    VoxPopuli (VP) & PT/EV & Multi & 400k \cite{babu2021xlsr}/24k \cite{chen2021unispeechsat, chen2021wavlm} & ASR & CC0 \\ \hline
    BABEL (BBL) & PT/EV & Multi & 1k & ASR & IARPA Babel Agreement  \\ \hline

    GigaSpeech & PT/EV & EN & 40k/10k \cite{chen2021unispeechsat, chen2021wavlm} & ASR & Apache-2.0 License \\ \hline
    TED-LIUM 3 (TED3) & PT/EV & EN & 450 & ASR & CC BY-NC-ND 3.0 \\ \hline
    TED-LIUM 2 (TED2) & PT/EV & EN & 118 & ASR & CC BY-NC-ND 3.0  \\ \hline
    Switchboard (SWB) & PT/EV & EN & 260 & ASR & Linguistic Data Consortium (LDC) \\ \hline
    TIMIT & PT/EV & EN & 4 & ASR/PR/PC & Linguistic Data Consortium (LDC) \\ \hline
    VoxLingua107 & PT/EV & Multi & 6.6k & LID & CC BY 4.0 \\ \hline

    Open Mandarin & PT/EV & ZH & 1.5k & ASR & \makecell[l]{CC BY-NC-ND 4.0, Apache License v.2.0,\\ Linguistic Data Consortium (LDC)} \\ \hline
    HKUST & PT/EV & ZH & 168/200 & ASR & Linguistic Data Consortium (LDC) \\ \hline
    AISHELL-1 & PT/EV & ZH & 178 & ASR & Apache License v.2.0 \\ \hline
    Hub5'00 & EV & EN & 13 & ASR & Linguistic Data Consortium (LDC) \\ \hline
    DIRHA & EV & EN & 11 & ASR & See link for details\textsuperscript{\ref{footnote:DIRHA_license}} \\ \hline
   
    CHiME-5 & EV & EN & 50 & ASR & See link for details\textsuperscript{\ref{footnote:CHiME_license}}\\ \hline
    Alexa-eval & EV & EN & 1k & ASR & Not released \\ \hline
    INTERFACE & EV & Multi & 16 & Sentiment & No information \\ \hline
    MOSEI & EV & EN & 65 & Sentiment & See link for details\textsuperscript{\ref{footnote:MOSEI_license}}\\ \hline
    VCTK & EV & EN & 44 & SID/ASV & CC BY 4.0 \\ \hline
    VoxCeleb1 & EV & Multi & 352 & SID/ASV & CC BY 4.0 \\ \hline
    Fluent Speech Commands (FSC) & EV & EN & 14.7 & IC & CC BY-NC-ND 4.0 \\ \hline
    QUESST 2014 (QUESST) & EV & Multi & 23 & QbE & No information \\ \hline
    LS En-Fr & EV & En-Fr & 236 & ST & CC BY 4.0 \\ \hline
    CoVoST-2 & EV & Multi & 2.9k & ST & CC0 \\ \hline

    ALFFA & EV & Multi & 5.2--18.3 & ASR-multi & MIT License \\ \hline
    OpenSLR-multi & EV & Multi & 4.4--265.9 & ASR-multi & \makecell[l]{CC BY-SA 3.0 US, CC BY-SA 4.0, CC BY 4.0, \\ CC BY-NC-ND 4.0, Apache License v.2.0}\\ \hline
    AED datasets & EV & - & - & AED & \makecell[l]{CC BY 4.0 (MUSAN, Speech Commands, NSynth, Bird Audio\\ Detection),  CC0 (Spoken Language Identification),\\ Non-Commercial (TUT)} \\
    \bottomrule
  \end{tabular}}
\end{table*}


\addtocounter{footnote}{1}
\footnotetext{\label{footnote:DIRHA_license}\url{https://dirha.fbk.eu/node/107}}
\addtocounter{footnote}{1}
\footnotetext{\label{footnote:CHiME_license}\url{https://chimechallenge.github.io/chime6/download.html}}
\addtocounter{footnote}{1}
\footnotetext{\label{footnote:MOSEI_license}\url{https://github.com/A2Zadeh/CMU-MultimodalSDK/blob/master/LICENSE.txt}}

\begin{table*}[ht]
  \centering
  \scriptsize
  \caption{A summary of common experiment settings for various SSL
  evaluations (Part 1). Networks are usually pre-trained with SSL
  techniques, augmented with prediction heads, and fine-tuned (or trained) with
  labeled data in downstream tasks for benchmarking. The 
  \textbf{Pre-training corpus}, \textbf{Training (fine-tuning)}, and
  \textbf{Test} columns list the datasets used in each work, and the \textbf{Task}
  column lists the tasks performed in the corresponding papers. We follow
  the abbreviation introduced in \cref{table:datasets}. The \textbf{Transfer}
  column indicates whether the SSL technique is evaluated by its capability for
  transfer learning, i.e., different datasets are utilized for pre-training and
  fine-tuning. The \textbf{Fine-tuning labels used} column summarizes the
  amount of labeled examples used in downstream fine-tuning.}
  \label{table:method_data_setting}
  
  \renewcommand*\arraystretch{1.4}
  \begin{tabular}{p{0.12\textwidth}p{0.12\textwidth}p{0.06\textwidth}p{0.13\textwidth}p{0.13\textwidth}p{0.06\textwidth}p{0.20\textwidth}}
    \toprule
    \multirow{2}{*}{Work} & \multirow{2}{*}{\makecell[l]{Pre-training\\ corpus}} & \multirow{2}{*}{Task} & \multicolumn{2}{c}{Dataset} & \multirow{2}{*}{Transfer} & \multirow{2}{*}{Fine-tuning labels used} \\ \cline{4-5}
     &  &  & Training (fine-tuning) & Test & & \\
    \midrule

   
    \multirow{2}{*}[0mm]{CPC \cite{oord2018representation}} & \multirow{2}{*}[0mm]{LS 100 hrs} & PC & LS 100 hrs\textsuperscript{\ref{footnote:CPC-split}} & LS 100 hrs\textsuperscript{\ref{footnote:CPC-split}} & - & 80\textsuperscript{\ref{footnote:low-resource}} hrs \\ \cline{3-7}
    & & SID & LS 100 hrs\textsuperscript{\ref{footnote:CPC-split}} & LS 100 hrs\textsuperscript{\ref{footnote:CPC-split}} & - & 80\textsuperscript{\ref{footnote:low-resource}} hrs \\ \hline
    
   
    \multirow{2}{*}[-5mm]{PASE\cite{pascual2019learning}} & \multirow{2}{*}[-5mm]{LS 50 hrs \cite{ravanelli2018learning}} & SID & VCTK\textsuperscript{\ref{footnote:official-split}} & VCTK\textsuperscript{\ref{footnote:official-split}} & \checkmark & 44 hrs \\ \cline{3-7}
    & & Sentiment & INTERFACE\textsuperscript{\ref{footnote:interface-split}} & INTERFACE\textsuperscript{\ref{footnote:interface-split}} & \checkmark & 3 hrs \\ \cline{3-7}
    & & PR & TIMIT\textsuperscript{\ref{footnote:official-split}} & TIMIT\textsuperscript{\ref{footnote:official-split}} & \checkmark & 4 hrs \\ \cline{3-7}
    & & ASR & DIRHA\textsuperscript{\ref{footnote:official-split}} & DIRHA\textsuperscript{\ref{footnote:official-split}} & \checkmark & 11 hrs \\ \hline
   
    Audio2Vec \cite{tagliasacchi2020pre} & AudioSet & AED & 6 AED datasets\textsuperscript{\ref{footnote:AED-datasets}} & 6 AED datasets\textsuperscript{\ref{footnote:AED-datasets}} & \checkmark & See \cite{tagliasacchi2020pre} for details \\ \hline
   
    \multirow{3}{*}[0mm]{APC \cite{chung2019unsupervised, chung2020generative}} & \multirow{3}{*}[0mm]{LS 360 hrs} & ASR & WSJ si284\textsuperscript{\ref{footnote:91-split-dev}} & WSJ dev93 & \checkmark & 72 hrs \\ \cline{3-7}
    & & ST & LS En-Fr\textsuperscript{\ref{footnote:official-split}} & LS En-Fr\textsuperscript{\ref{footnote:official-split}} & - & 236 hrs \\ \cline{3-7}
    & & SID & WSJ si284\textsuperscript{\ref{footnote:APC-split}} & WSJ si284\textsuperscript{\ref{footnote:APC-split}} & \checkmark & 65\textsuperscript{\ref{footnote:low-resource}} hrs \\ \hline
   
    \multirow{2}{*}[0mm]{wav2vec \cite{schneider2019wav2vec}} & \multirow{2}{*}[-0.5mm]{\makecell[l]{LS 80/960 hrs,\\ LS 960 hrs \\ \:+ WSJ si284}} & ASR & WSJ si284 & WSJ eval92 & \checkmark & 81 hrs\vspace{1mm} \\ \cline{3-7}
    & & PR & TIMIT\textsuperscript{\ref{footnote:official-split}} & TIMIT\textsuperscript{\ref{footnote:official-split}} & \checkmark & 4 hr\vspace{1mm} \\ \hline
   
    PhasePredict \cite{quitry2019learning} & AudioSet & AED & 7 AED datasets\textsuperscript{\ref{footnote:AED-datasets}} & 7 AED datasets\textsuperscript{\ref{footnote:AED-datasets}} & \checkmark & See \cite{quitry2019learning} for details \\ \hline
   
    \multirow{3}{*}{Bidir-CPC\cite{kawakami2020learning}} & \multirow{3}{*}{\makecell[l]{LS 960 hrs,\\ CPC-8k\textsuperscript{\ref{footnote:CPC-8k}}}} & ASR & \makecell[l]{WSJ si284,\\LS 960 hrs,\\TED3\textsuperscript{\ref{footnote:official-split}}} & \makecell[l]{WSJ eval92,\\LS test-clean,\\LS test-other,\\TED3\textsuperscript{\ref{footnote:official-split}}, SWB\textsuperscript{\ref{footnote:official-split}}}  & \makecell[l]{Different\\ datasets for\\ training\\ and test} & 81/960/450 hrs \\ \cline{3-7}
    & & \makecell[l]{ASR-multi} & ALFFA\textsuperscript{\ref{footnote:official-split}} & ALFFA\textsuperscript{\ref{footnote:official-split}} & \checkmark & 4 languages, 5.2--18.3 hrs\\ \cline{3-7}
    & & \makecell[l]{ASR-multi} & OpenSLR-multi\textsuperscript{\ref{footnote:official-split}} & OpenSLR-multi\textsuperscript{\ref{footnote:official-split}} & \checkmark & 21 languages, 4.4--265.9 hrs \\ \hline
   

   
    \multirow{3}{*}{MockingJay \cite{liu2020mockingjay}} & \multirow{3}{*}{\makecell[l]{LS 360 hrs}} & PC & LS 360 hrs & LS test-clean & - & 0.36\textsuperscript{\ref{footnote:low-resource}}/1.8\textsuperscript{\ref{footnote:low-resource}}/3.6\textsuperscript{\ref{footnote:low-resource}}/18\textsuperscript{\ref{footnote:low-resource}}/45\textsuperscript{\ref{footnote:low-resource}}/360  hrs \\ \cline{3-7}
    & & SID & LS 100 hrs\textsuperscript{\ref{footnote:91-split-test}} & LS 100 hrs\textsuperscript{\ref{footnote:91-split-test}} & - & 90\textsuperscript{\ref{footnote:low-resource}} hrs \\ \cline{3-7}
    & & Sentiment & MOSEI\textsuperscript{\ref{footnote:official-split}} & MOSEI\textsuperscript{\ref{footnote:official-split}} & \checkmark & 65 hrs \\ \hline

   
    \multirow{2}{*}{CPC modified \cite{riviere2020unsupervised}} & LS 100 hrs & PC & LS 100 hrs\textsuperscript{\ref{footnote:CPC-split}} & \makecell[l]{LS 100 hrs\textsuperscript{\ref{footnote:CPC-split}}} & - & \makecell[l]{80\textsuperscript{\ref{footnote:low-resource}} hrs} \\ \cline{2-7}
    & \makecell[l]{LS 100 hrs,\\LS 960 hrs,\\LL 60k hrs} & PC & CV-dataset\textsuperscript{\ref{footnote:official-split}} & CV-dataset\textsuperscript{\ref{footnote:official-split}} & \checkmark & 1 hrs \\ \hline
   
    \multirow{2}{*}[0mm]{vq-wav2vec \cite{Baevski2020vq-wav2vec}} & \multirow{2}{*}[0mm]{LS 960 hrs} & ASR & WSJ si284 & \makecell[l]{WSJ eval92} & \checkmark & \makecell[l]{81 hrs} \\ \cline{3-7}
    & & PR & TIMIT\textsuperscript{\ref{footnote:official-split}} & TIMIT\textsuperscript{\ref{footnote:official-split}} & \checkmark & 4 hrs \\ \hline
   
    \multirow{2}{*}[0mm]{DeCoAR \cite{ling2020deep}} & \multirow{2}{*}[0mm]{\makecell[l]{LS 100/360/460/\\\:\:\:960 hrs,\\ WSJ si284}} & ASR & WSJ si284 & WSJ eval92 & - & \makecell[l]{25\textsuperscript{\ref{footnote:low-resource}}/40\textsuperscript{\ref{footnote:low-resource}}/81 hrs} \\ \cline{3-7}
    & & ASR & \makecell[l]{LS 100/360/\\\:\:\:460/960 hrs} & \makecell[l]{LS test-clean,\\ LS test-other} & - & 100/360/460/960 hrs \\ \hline
   
    \multirow{2}{*}[0mm]{MT-APC\cite{chung2020improved}} & \multirow{2}{*}[0mm]{LS 360 hrs} & ASR & WSJ si284\textsuperscript{\ref{footnote:91-split-dev}} & WSJ dev93 & \checkmark & 72 hrs \\ \cline{3-7}
    & & ST & LS En-Fr\textsuperscript{\ref{footnote:official-split}} & LS En-Fr\textsuperscript{\ref{footnote:official-split}} & - & 236 hrs \\ \hline
   
    \multirow{3}{*}[0mm]{PASE+\cite{ravanelli2020multi}} & \multirow{3}{*}[0mm]{LS 50 hrs \cite{ravanelli2018learning}} & PR & TIMIT\textsuperscript{\ref{footnote:official-split}} & TIMIT\textsuperscript{\ref{footnote:official-split}} & \checkmark & 4 hrs \\ \cline{3-7}
    & & ASR & DIRHA\textsuperscript{\ref{footnote:official-split}} & DIRHA\textsuperscript{\ref{footnote:official-split}} & \checkmark & 11 hrs \\ \cline{3-7}
    & & ASR & CHiME-5\textsuperscript{\ref{footnote:official-split}} & CHiME-5\textsuperscript{\ref{footnote:official-split}} & \checkmark & 50 hrs \\ \hline

   
    \multirow{2}{*}[0mm]{AALBERT \cite{chi2020audio}} & \multirow{2}{*}[0mm]{\makecell[l]{LS 360 hrs}} & PC & LS 100 hrs\textsuperscript{\ref{footnote:APC-split}} & LS 100 hrs\textsuperscript{\ref{footnote:APC-split}} & - & 80\textsuperscript{\ref{footnote:low-resource}} hrs \\ \cline{3-7}
    & & SID & LS 360 hrs\textsuperscript{\ref{footnote:APC-split}} & LS 360 hrs\textsuperscript{\ref{footnote:APC-split}} & - & 288\textsuperscript{\ref{footnote:low-resource}} hrs \\
    \bottomrule
  \end{tabular}}
\end{table*}


\begin{table*}[ht]
  \centering
  \scriptsize
  \caption{A summary of common experiment settings for various SSL
  evaluations (Part 2). See the caption of \cref{table:method_data_setting} for a
  detailed description of all the abbreviations used in this table.}
  \label{table:method_data_setting_2}
  
  \renewcommand*\arraystretch{1.4}
  \begin{tabular}{p{0.12\textwidth}p{0.12\textwidth}p{0.05\textwidth}p{0.18\textwidth}p{0.18\textwidth}p{0.05\textwidth}p{0.15\textwidth}}
    \toprule
    \multirow{2}{*}{Work} & \multirow{2}{*}{\makecell[l]{Pre-training\\ corpus}} & \multirow{2}{*}{Task} & \multicolumn{2}{c}{Dataset} & \multirow{2}{*}{Transfer} & \multirow{2}{*}{Fine-tuning labels used} \\ \cline{4-5}
     &  &  & Training (fine-tuning) & Test & & \\
    \midrule

   
    \multirow{2}{*}[0mm]{BMR \cite{wang2020unsupervised}} & \multirow{2}{*}[0mm]{\makecell[l]{WSJ si284,\\LS 960 hrs}} & ASR & WSJ si284 & WSJ eval92 & - & 81 hrs \\ \cline{3-7}
    & & PR & WSJ si84/si284 & WSJ dev93 & - & 15/81 hrs \\ \hline

   
    \multirow{2}{*}[0mm]{vq-APC \cite{chung20e_interspeech}} & \multirow{2}{*}[0mm]{LS 360 hrs} & PC & WSJ si284\textsuperscript{\ref{footnote:91-split-dev}} & \makecell[l]{WSJ dev93} & \checkmark & \makecell[l]{81 hrs} \\ \cline{3-7}
    & & SID & WSJ si284\textsuperscript{\ref{footnote:APC-split}} & WSJ si284\textsuperscript{\ref{footnote:APC-split}} & \checkmark & 65\textsuperscript{\ref{footnote:low-resource}} hrs \\ \hline

   
    \makecell[l]{vq-wav2vec + \\\:DiscreteBERT \cite{baevski2019effectiveness}} & LS 960 hrs & ASR & LS 100 hrs & \makecell[l]{LS test-clean,\\ LS test-other} & - & \makecell[l]{10 mins\textsuperscript{\ref{footnote:low-resource}},\\ 1\textsuperscript{\ref{footnote:low-resource}}/10\textsuperscript{\ref{footnote:low-resource}}/100 hrs} \\ \hline

   
    \multirow{2}{*}[-1mm]{speech-XLNet \cite{song20d_interspeech}} & \multirow{2}{*}[-1mm]{\makecell[l]{LS 960 hrs\\\;\;\;+ WSJ si284\\\;\;\;+ TED2}} & PR & TIMIT\textsuperscript{\ref{footnote:official-split}} & TIMIT\textsuperscript{\ref{footnote:official-split}} & \checkmark & 4 hrs\vspace{1mm} \\ \cline{3-7}
    & & ASR & WSJ si284 & WSJ eval92 & - & \makecell[l]{7\textsuperscript{\ref{footnote:low-resource}}/14\textsuperscript{\ref{footnote:low-resource}}/30\textsuperscript{\ref{footnote:low-resource}}/81 hrs}\vspace{1mm} \\ \hline
    
   
    \multirow{3}{*}[0mm]{MPC \cite{jiang2019improving, jiang2021further}} & \multirow{2}{*}[0mm]{\makecell[l]{Didi Callcenter,\\Didi Dictation,\\Open Mandarin}} & ASR-zh & HKUST\textsuperscript{\ref{footnote:official-split}} & HKUST\textsuperscript{\ref{footnote:official-split}} & \checkmark & 168 hrs \\ \cline{3-7}
    & & ASR-zh & AISHELL-1\textsuperscript{\ref{footnote:official-split}} & AISHELL-1\textsuperscript{\ref{footnote:official-split}} & \checkmark & 178 hrs\vspace{1mm} \\ \cline{2-7}
    & \multirow{1}{*}[-0.5mm]{\makecell[l]{SWB, Fisher 1k,\\LS 960 hrs}} & ASR & SWB & Hub5'00 & - & 260 hrs \vspace{2.5mm} \\ \hline

   
    \multirow{2}{*}[0mm]{MPE \cite{liu2020masked}} & \multirow{2}{*}[0mm]{\makecell[l]{WSJ si284,\\LS 960 hrs}} & ASR & WSJ si284 & WSJ eval92 & - & \makecell[l]{25\textsuperscript{\ref{footnote:low-resource}}/40\textsuperscript{\ref{footnote:low-resource}}/81 hrs} \\ \cline{3-7}
    & & ASR & LS 100/360/960 hrs & LS test-clean & - & 100/360/960 hrs \\ \hline

   
    ConvDMM \cite{khurana20_interspeech} & \makecell[l]{LS 50\cite{ravanelli2018learning}/360/\\\:\:\:960 hrs} & PC/PR & WSJ si284 & WSJ eval92 & \checkmark & \makecell[l]{5\textsuperscript{\ref{footnote:low-resource}}/50\textsuperscript{\ref{footnote:low-resource}}/100\textsuperscript{\ref{footnote:low-resource}} mins,\\ 4\textsuperscript{\ref{footnote:low-resource}}/8\textsuperscript{\ref{footnote:low-resource}}/40\textsuperscript{\ref{footnote:low-resource}} hrs} \\ \hline
   
    \multirow{2}{*}{wav2vec 2.0 \cite{baevski2020wav2vec}} & \multirow{2}{*}{\makecell[l]{LS 960 hrs,\\ LL 60k hrs}} & ASR & LS 960 hrs & \makecell[l]{LS test-clean,\\ LS test-other} & - & \makecell[l]{10 mins\textsuperscript{\ref{footnote:low-resource}},\\ 1\textsuperscript{\ref{footnote:low-resource}}/10\textsuperscript{\ref{footnote:low-resource}}/100/960 hrs} \\ \cline{3-7}
    & & PR & TIMIT\textsuperscript{\ref{footnote:official-split}} & TIMIT\textsuperscript{\ref{footnote:official-split}} & \checkmark & 4 hrs \\ \hline
   
    \multirow{2}{*}[0mm]{NPC \cite{liu21l_interspeech}} & \multirow{2}{*}[0mm]{LS 360 hrs} & PC & WSJ si284\textsuperscript{\ref{footnote:91-split-dev}} & \makecell[l]{WSJ dev93} & \checkmark & \makecell[l]{81 hrs} \\ \cline{3-7}
    & & SID & WSJ si284\textsuperscript{\ref{footnote:APC-split}} & WSJ si284\textsuperscript{\ref{footnote:APC-split}} & \checkmark & 65\textsuperscript{\ref{footnote:low-resource}} hrs \\ \hline
   
    DeCoAR 2.0 \cite{ling2020decoar} & LS 960 hrs & ASR & \makecell[l]{LS 100 hrs} & \makecell[l]{LS test-clean,\\ LS test-other} & - & \makecell[l]{1\textsuperscript{\ref{footnote:low-resource}}/10\textsuperscript{\ref{footnote:low-resource}}/100 hrs} \\ \hline

   
    \multirow{4}{*}[0mm]{TERA \cite{liu2021tera}} & \multirow{4}{*}[0mm]{\makecell[l]{LS 100/360/\\\:\:\:960 hrs}} & PC & LS 100 hrs\textsuperscript{\ref{footnote:CPC-split}} & LS 100 hrs\textsuperscript{\ref{footnote:CPC-split}} & - & 80\textsuperscript{\ref{footnote:low-resource}} hrs \\ \cline{3-7}
    & & SID & LS 100 hrs\textsuperscript{\ref{footnote:CPC-split}} & LS 100 hrs\textsuperscript{\ref{footnote:CPC-split}} & - & 80\textsuperscript{\ref{footnote:low-resource}} hrs \\ \cline{3-7}
    & & PR & TIMIT\textsuperscript{\ref{footnote:official-split}} & TIMIT\textsuperscript{\ref{footnote:official-split}} & \checkmark & 4 hrs \\ \cline{3-7}
    & & ASR & LS 100 hrs & LS test-clean & - & 100 hrs \\ \hline
   
    HuBERT \cite{hsu2021hubert} & \makecell[l]{LS 960 hrs,\\ LL 60k hrs} & ASR & LS 960 hrs & \makecell[l]{LS test-clean\\ LS test-other} & - & \makecell[l]{10 mins\textsuperscript{\ref{footnote:low-resource}},\\ 1\textsuperscript{\ref{footnote:low-resource}}/10\textsuperscript{\ref{footnote:low-resource}}/100/960 hrs} \\ \hline
   
    wav2vec-c \cite{sadhu21_interspeech} & Alexa-10k & ASR & \makecell[l]{Alexa-eval} & \makecell[l]{Alexa-eval} & \checkmark & 1k hrs \\ \hline
    UniSpeech-SAT \cite{chen2021unispeechsat} & \makecell[l]{LL 60k hrs\\\;\;\;+ GigaSpeech-10k\\\;\;\;+ VP-24k} & Multi & SUPERB & SUPERB & \checkmark & See SUPERB \cite{yang21c_interspeech} paper for details \\ \hline
    WavLM \cite{chen2021wavlm} & \makecell[l]{LL 60k hrs\\\;\;\;+ GigaSpeech-10k\\\;\;\;+ VP-24k} & Multi & SUPERB & SUPERB & \checkmark & See \cite{yang21c_interspeech} for details \\ \hline
    \multirow{4}{*}[0mm]{XLS-R \cite{babu2021xlsr}} & \multirow{4}{*}[0mm]{\makecell[l]{VP-400k + MLS\\\;\;\;+ CV-dataset-7k + VL\\\;\;\;+ BBL}} & ASR & VP, MLS, CV-dataset, BBL, LS & VP, MLS, CV-dataset, BBL, LS & - & \multirow{4}{*}[0mm]{See \cite{babu2021xlsr} for details} \\ \cline{3-6}
    & & SID & VoxCeleb1 & VoxCeleb1 & \checkmark & \\ \cline{3-6}
    & & ST & CoVoST-2 & CoVoST-2 & \checkmark & \\ \cline{3-6}
    & & LID & VL & VL & - & \\

   
   
   
   
   
   
    \bottomrule
  \end{tabular}}
\end{table*}

\addtocounter{footnote}{1}
\footnotetext{\label{footnote:CPC-split}
Train/test split made available
by \cite{oord2018representation} on Google drive
https://drive.google.com/drive/folders/1BhJ2umKH3whguxMwifaKtSra0TgAbtfb.}
\addtocounter{footnote}{1}
\footnotetext{\label{footnote:official-split}
Utilizes official training or test split.}

\addtocounter{footnote}{1}
\footnotetext{\label{footnote:interface-split}
English utterances used in experiments. The utterances correspond to
approximately 3~hours for training, 40~minutes for development, and 30~minutes
for testing.}

\addtocounter{footnote}{1}
\footnotetext{\label{footnote:AED-datasets}
The 6 AED datasets used in
\cite{tagliasacchi2020pre} are MUSAN~\cite{snyder2015musan}, Bird Audio
Detection~\cite{stowell2019automatic}, Speech Commands~\cite{warden2018speech},
Spoken Language Identification~\cite{oponowiczspoken}, TUT Urban Acoustic
Scenes 2018~\cite{dcase2018} plus an SID task built with LS
\textit{train-clean-100}. In addition to the 6 datasets,
\cite{quitry2019learning} use the NSynth dataset~\cite{engel2017neural} for
evaluation.}
\addtocounter{footnote}{1}
\footnotetext{\label{footnote:CPC-8k}
A collection of AudioSet, AVSpeech,
CV-dataset, LS, WSJ, TIMIT, Speech Accent Archive 
(SSA)~\cite{weinberger2009towards}, TED3, and SWB. SSA is a growing annotated corpus of
English speech with various accents. Among the papers studied in this review, SSA
is used in \cite{kawakami2020learning} only for pre-training, and only 1 hour
of audio is utilized. Thus, we exclude it from our discussion in
\cref{section:benchmark}.}
\addtocounter{footnote}{1}
\footnotetext{\label{footnote:low-resource}
A subset of the official training
split is sampled, usually to mimic low-resource learning conditions or to
quickly evaluate for training and testing on the same split but disjoint
subsets.}
\addtocounter{footnote}{1}
\footnotetext{\label{footnote:APC-split}
Dataset split into training,
validation, and test subsets at a ratio of 8:1:1.}
\addtocounter{footnote}{1}
\footnotetext{\label{footnote:91-split-dev}
Dataset split into training
and validation subsets at a ratio of 9:1.}
\addtocounter{footnote}{1}
\footnotetext{\label{footnote:91-split-test}
The dataset split into training
and test subsets at a ratio of 9:1.}

\subsection{Experiment settings for evaluating SSL techniques}
A common way to benchmark SSL techniques and show their efficacy is to
fine-tune a pre-trained SSL model for a supervised downstream task.
Depending on the corpora used in pre-training and fine-tuning, techniques can
be benchmarked in terms of their capability to transfer knowledge across datasets
(i.e., using pre-training corpora that differ from the fine-tuning ones), their
benefit when training with limited labeled examples (i.e., sampling a subset of
labeled examples for fine-tuning), or their improvement over a fully supervised
baseline (i.e., using the entire training split of downstream datasets for
fine-tuning). Tables~\ref{table:method_data_setting} 
and~\ref{table:method_data_setting_2} summarize experiment settings used in the
SSL literature, including the pre-training corpora, downstream tasks and datasets,
and the amount of fine-tuning labels used, which indicates the targeted
benchmarking scenario as discussed above. Note that there are a variety of ways
to fine-tune pre-trained networks (e.g., fine-tune the entire network,
freeze certain layers during fine-tuning, and add various architectures of
prediction layers to pre-trained networks). We here omit
descriptions of these choices; readers can consult the
original publications for details.

As observed in Tables~\ref{table:method_data_setting} 
and~\ref{table:method_data_setting_2}, LS and WSJ are the most commonly used
pre-training corpora. At the same time, we observe a
growing industry investment in pre-training with larger datasets, e.g.,
CPC-8k (8k hours) for Bidir-CPC~\cite{kawakami2020learning}, LL (60k hours) for
CPC modified~\cite{riviere2020unsupervised}, 
wav2vec 2.0~\cite{baevski2020wav2vec}, and HuBERT~\cite{hsu2021hubert}, Alexa internal
datasets (10k hours) for wav2vec-c~\cite{sadhu21_interspeech}, Didi internal
datasets (10k hours) for MPC~\cite{jiang2019improving, jiang2021further}, the
combination of Gigaspeech, VP-24k, and LL (94k hours in total) for
UniSpeech-SAT~\cite{chen2021unispeechsat} and WavLM~\cite{chen2021wavlm}, and
the combination of VP-400K, MLS, CV-dataset, VL and BBL (436k hours in total)
for XLS-R~\cite{babu2021xlsr}. We expect this trend to continue with the growth
in available computing power. Most studies focuses on learning representations
for English, whereas Chinese~\cite{jiang2019improving, jiang2021further} and
multilinguality~\cite{kawakami2020learning, babu2021xlsr} are also gaining
attention. Compared to pre-training, datasets used for fine-tuning are more
diverse and cover downstream tasks as varied as ASR, PR, PC, SID, AED,
Sentiment, ST, and LID. For benchmarking training scenarios covering full
supervision as well as limited resources, the amount of labeled examples used for
fine-tuning also varies from several minutes up to 1k hours. Recent
benchmarks such as SUPERB~\cite{yang21c_interspeech} that consolidate multiple 
downstream tasks have gained attention for evaluating SSL
methodologies~\cite{chen2021unispeechsat, chen2021wavlm}. The goal of such benchmarks is
to provide a holistic evaluation of the performance of learned representations; 
we discuss these in detail in \cref{sec:benchmark}. With the increasing
popularity of SSL research, we expect future experiment settings to proliferate
and cover more languages, downstream tasks, and pre-training/fine-tuning
datasets.

\begin{figure*}
\centering
\includegraphics[width=1.0\textwidth]{figures/test-clean-all.pdf}
\caption{SSL performance on ASR WER (vertical axis) evaluated with
LS \textit{test-clean} split. Techniques are sorted based on the number of
model parameters along the horizontal axis. Markers and lines in red, green,
blue, purple, and pink correspond to models initialized with various SSL
techniques and then fine-tuned using 960, 100, 10, 1 hour(s), and 10 minutes
respectively. The 960-hour training set is the aggregation of
\textit{train-clean-100}, \textit{train-clean-360}, and
\textit{train-other-500} splits. The 100-, 10-, 1-hour, and 10-minute sets
leverage \textit{train-clean-100} or its sampling, except for Bidir-CPC, which
samples 10\% of the training examples from the entire 960-hour corpus. For simplicity,
several SSL techniques are appended with suffixes \textit{B}, \textit{L},
\textit{XL}, or \textit{XXL} indicating the \textit{Base}, \textit{Large},
\textit{X-Large}, or \textit{XX-Large} variants specified in the original
publication. We also include comparisons with semi-supervised, self-training
approaches (iterative pseudo labeling (IPL)~\cite{xu2020iterative}, 
slimIPL~\cite{likhomanenko2020slimipl}, noisy student~\cite{park2020improved}), as well
as the current state of the art---conformer XXL + noisy student~\cite{zhang2020pushing}---which 
combines self-training and SSL techniques. These approaches are
visualized in black, grey, or orange to reflect the number of labeled
utterances (960, 100, or 10 hours) used to train the systems. Given the
diversity of the listed methods in experiment settings (e.g., pre-training corpora
and objectives, whether a language model is used in decoding, whether model
parameters are frozen in fine-tuning), readers should note that the superiority
of methods cannot be decided only based on lower WER numbers.}
\label{fig:asr_result_no_limit}
\end{figure*}


\subsection{Benchmark results and discussion} \label{sec:benchmark} 
Given the diversity of datasets and downstream tasks used to evaluate SSL
techniques in the literature, it is infeasible to discuss all 
experiment settings in this survey. Hence, due to their wide adoption for
experiments conducted by studies in both SSL and the speech community in
general, we focus first on ASR on the LS dataset to understand the
efficacy of SSL. We examine SSL techniques which report ASR results on the LS
\textit{test-clean} split, and summarize the published WER in
\cref{fig:asr_result_no_limit}. The ASR models were obtained first by using
unlabeled speech to pre-train a model with each SSL technique. The model was
then fine-tuned on labeled data by utilizing a supervised training objective.
Respectively, 960, 100, 10, 1 hour(s), and 10 minutes of labeled LS training data
were used for training, as indicated in
\cref{fig:asr_result_no_limit} by different colors of markers and lines (see
the caption of \cref{fig:asr_result_no_limit} for more details).
Semi-supervised methods such as self-training, where a model is first trained
on labeled data to annotate unlabeled speech, and then subsequently trained on
combined golden and self-annotated label-speech pairs, are gaining popularity
in the speech community and have yielded competitive results. For comparison, we also
show performance from such methods (iterative pseudo labeling 
(IPL)~\cite{xu2020iterative}, slimIPL~\cite{likhomanenko2020slimipl}, noisy 
student~\cite{park2020improved}), as well as the current state of the art---conformer XXL + noisy
student~\cite{zhang2020pushing}---which augments SSL with various advanced
techniques including self-training. Results are visualized in black, grey, or
orange to reflect the number of labeled utterances used during training (960,
100, or 10 hours).
As observed in the figure, most SSL techniques outperform fBANK
features, and with the growing investment in model size, better performance is
achieved. The largest ones, such as wav2vec 2.0-L and HuBERT-L/XL, yield
  competitive results                        
when the entire 960-hour of labeled data is used in
training/fine-tuning. The benefit of SSL, especially models with more parameters
like wav2vec 2.0 and HuBERT, becomes more evident when the labeling resources
become scarce. Compared to popular semi-supervised methods such as IPL,
slimIPL, and noisy student using 100 hours of labels, wav2vec~2.0 and HuBERT
achieve lower or competitive WERs with 1 hour or even 10 minutes of labeled
examples. The results are highly favorable for low-resource use cases, for instance when
expanding systems to new domains or languages for which large amounts of unlabeled
audio are available, since collecting labels for new conditions is often prohibitively
slow or costly.

In addition to the ASR task, where the current state of the art is achieved by a method
combining SSL pre-training and self-training 
techniques~\cite{zhang2020pushing}, SSL models 
  are competitive               
in other tasks, including IC,
SID, ASV, and QbE. We summarize the performance of these models and previous
non-SSL methods in \cref{table:sota_performance}. The results suggest that the
benefit of SSL is generalizable among tasks that require encoding 
information such as content, speaker, and semantics. As SSL research
gains more attention, we expect that SSL pre-trained models will 
achieve state-of-the-art results on an increasing number of tasks.

\begin{table}[ht]
  \centering
  \footnotesize
  \caption{Tasks where the state of the art is models with SSL pre-training.}
  \label{table:sota_performance}
  \renewcommand*\arraystretch{1.2}
  \begin{tabular}{llllll}  
    \toprule
    Tasks & Dataset & non-SSL & SSL \\
    \midrule
    ASR (WER $\downarrow$) & LS test-clean/other & 2.1/4.0 \cite{xu2020iterative} & 1.4/2.6 \cite{zhang2020pushing} \\ \hline
    IC (Acc $\uparrow$) & FSC & 98.8 \cite{lugosch19_interspeech} & 99.3\cite{chen2021unispeechsat} \\ \hline
    SID (Acc $\uparrow$) & VoxCeleb1 & 94.8 \cite{hajibabaei2018unified} & 95.5 \cite{chen2021wavlm} \\ \hline
    ASV (EER $\downarrow$) & VoxCeleb1 & 3.1 \cite{hajavi2021siamese} & 2.4 \cite{wang2021fine} \\ \hline
    QbE (MTWV $\uparrow$) & QUESST (EN) & 10.6 \cite{rodriguez2014gtts} & 11.2\cite{chen2021unispeechsat} \\

   
   
   
   
   
   
   
   

    \bottomrule
  \end{tabular}
\end{table}

Despite the obvious trend of increasing performance as more parameters and SSL
pre-training data are being used, numbers in \cref{fig:asr_result_no_limit} 
and \cref{table:sota_performance} are less comparable than might be expected.
The task performance is obtained from the original papers and is often
achieved with different downstream fine-tuning recipes, including various
language models (used in the ASR system), prediction heads (networks added to
SSL for downstream inference), or choices between fine-tuning the whole
networks or freezing the SSL encoders. For example, in the ASR task, HuBERT-L
and wav2vec~2.0-L leverage Transformer as their language model, while a 4-gram
language model trained on LS is used in DeCoAR~2.0. The lack of common and
established mechanisms to evaluate SSL techniques in downstream applications
makes it difficult to compare techniques fairly and understand their
capabilities. To address this challenge, there are increasing efforts to establish
benchmarks with shared downstream tasks, datasets, and downstream recipes. Such
efforts include SUPERB~\cite{yang21c_interspeech}, 
LeBenchmark~\cite{evain21_interspeech}, ZeroSpeech~\cite{dunbar2020zero},
HEAR,\footnote{\url{https://neuralaudio.ai/hear2021-holistic-evaluation-of-audio-representations.html}\label{footnote:HEAR}}
NOSS~\cite{shor20_interspeech}, and HARES~\cite{wang2021towards}. 

SUPERB~\cite{yang21c_interspeech} is a benchmarking platform that allows the
SSL community to train, evaluate, and compare speech representations on
diverse downstream speech processing tasks, from acoustic and speaker identity
to paralinguistic and semantics. SUPERB consolidates downstream recipes to
focus on common and straightforward settings (e.g., prediction head
architectures, language models, hyperparameter spaces) to facilitate generalizable
and reproducible benchmarking of SSL techniques. SUPERB also encourages
researchers to innovate for efficient use of model parameters and computation
resources 
to democratize SSL beyond race among Big Tech.  
LeBenchmark~\cite{evain21_interspeech} shares a vision similar to SUPERB and provides a
reproducible framework for assessing SSL in French with ASR, spoken language
understanding, speech translation, and emotion recognition. 
ZeroSpeech~\cite{dunbar2020zero} (described in more detail in \cref{zero_speech})
challenges the scientific community to build speech and language understanding
systems using zero expert resources 
for millions of users of ``low-resource" languages.
SSL techniques are also benchmarked with the ZeroSpeech 
challenge~\cite{tjandra20_interspeech, niekerk20b_interspeech}. Apart from the speech
community, researchers have also established HEAR (holistic evaluation of audio
representations),\textsuperscript{\ref{footnote:HEAR}} NOSS (non-semantic
speech benchmark)~\cite{shor20_interspeech}, and HARES (holistic audio
representation evaluation suite)~\cite{wang2021towards} to benchmark audio
representations. These efforts promote the creation of an audio embedding
that is as holistic as the human ear in interpreting speech, environmental
sound, and music. Given the significant need to understand and compare SSL techniques
fairly and comprehensively, we expect SSL benchmarking to remain an
active research area.








\section{Analysis of Self-Supervised Representations}
\label{analysis}

The previous sections have shown how self-supervised learning can result in
powerful representations that provide a robust starting point for several
downstream tasks. It is natural to ask if we can gain an even deeper
understanding of the nature of these representations, in order to further
optimize them or apply them to different problems.
What is the information encoded in these representations? How robust are they
to distributional shifts, and how dependent are they on the size of the
training data? Do they generalize across languages? What are the key
ingredients for training powerful representations: input data, network
architecture, training criterion, or all three? Can we predict their
performance on downstream tasks from their training behavior? This section
tries to answer these questions by summarizing several studies that analyze
self-supervised representations.

\subsection{Information content}

In \cite{pasad2021} wav2vec~2.0 representations were analyzed with
respect to their acoustic-linguistic information content at different
network layers. Three different mechanisms were used for this
purpose. The first of these is canonical correlation analysis (CCA),
which computes similarity scores between two continuous vectors
based on the maximum correlation of their linear projections.  These
can be used to judge the similarity of embeddings at different layers
with each other, with standard acoustic representations such as mel
filterbank features, or word embeddings derived from text. The second
method clusters continuous representation vectors and computes the
discrete mutual information between cluster IDs and phone or word
labels. The third method involves probing tasks: representation
vectors extracted from the network are used to perform simple
downstream tasks, in particular determining whether two acoustic segments
correspond to the same word, and a standard benchmark of 11 word
similarity tasks~\cite{faruqi14}. These are mostly used to gauge the
amount of lexical information present in the embeddings.  Using this
battery of tests the authors compared pre-trained models of varying
sizes as well as models fine-tuned for ASR. They found that pre-trained
models show an autoencoder-style behavior, with early layers showing
strong similarity with input features, intermediate layers diverging
more, and final layers reverting back to higher similarity with input
features and early layers. Generally, the earlier layers in wav2vec~2.0 
models encode acoustic information. The next set of layers encodes
phonetic class information, followed by word meaning information,
before reverting back to encoding phonetic/acoustic information. Thus,
extracting representations from the last layers for tasks that require
phonetic or word-related information may not be the best
strategy. Indeed, the authors of \cite{baevski2021unsupervised} show that a
phone classifier trained on each of the 24 frozen layers of a wav2vec~2.0 model
showed the lowest phone error rates
for layers 10--21 and higher error rates for the other layers. 
\cite{pasad2021} further show that fine-tuning the pre-trained model with a
character-level CTC training criterion changes the behavior of the
last layers (especially the final two layers), breaking the
autoencoder-style behavior and focusing the information encoded in the
last layers on orthographic-phonetic and word information. 

The peaking of class-relevant information in intermediate layers seems to be
common across different self-supervised learners and different modalities. In
an analysis of text-based Transformers trained with a masked language model
criterion~\cite{voita2019} observed a similar compression plus reconstruction
pattern. Interestingly, similar network behavior was also recently described
for self-supervised learners in computer vision: using a contrastive
self-supervised learner (SimCLR) that optimizes for augmentation invariance,
\cite{grigg2021self} show that it is the intermediate representations that most
closely approximate information learned in a supervised way, i.e., they
provide more class information than the representations from final layers. This
is similar to the findings described above for wav2vec~2.0 without fine-tuning,
where intermediate layers provide more information about phone and word
classes.

Self-supervised representations may encode other information besides phonetic
classes or words, for example, channel, language, speaker, and sentiment
information. 
Location of information pertaining to speakers vs.\ language classes was
analyzed in \cite{ling20odyssey} for a 12-layer BERTphone model. This model
combines a self-supervised masked reconstruction loss with a phone-based CTC
loss to produce representations	for speaker recognition and language
identification. By	analyzing the weights	of a linear combination	of layer
representations for these two downstream tasks, it was	shown that language
recognition draws on representation	from higher layers (peaking at layer~10)
whereas speaker recognition benefited from layers at positions 6, 9, and 12.
This may indicate that language recognition relies more on higher-level
phonetic information whereas speaker recognition uses a combination of
acoustic and phonetic information. In a recent study~\cite{chen2021wavlm} the
same technique was used to identify layer contributions for the downstream
SUPERB benchmark tasks in the WavLM model. For a smaller model (95M parameters)
it was again confirmed that lower layers encode speaker-related information
necessary for speaker diarization and verification whereas higher layers encode
phonetic and semantic information. Another study~\cite{wang2021layersup} used
explicit self-supervised loss at the intermediate layers rather than just the
output layer of a HuBERT model in order to enforce better learning of phonetic
information. The resulting model was indeed better at downstream tasks
requiring information about phonetic content, such as phone recognition, ASR,
and keyword spotting, but worse at speaker-related tasks like speaker
diarization and verification. 

Most self-supervised learning approaches rely on a Transformer architecture for
the representation model. In \cite{AnalyzeAttention} the attention patterns in
generatively trained Transformer representation models were analyzed.
Self-attention heads were grouped into three categories: diagonal, vertical,
and global. It was found that the diagonal head focuses on neighbors and is
highly correlated with phoneme boundaries, whereas the vertical head focuses on
specific phonemes in the utterance. Global heads were found to be redundant as
removing them resulted in faster inference time and higher performance.

\subsection{Training criterion}
In \cite{chung2021}, representations based on different training criteria
(masked predictive coding, contrastive predictive coding, and autoregressive
predictive coding) were compared  and analyzed with respect to the correlation
between their training loss and performance on both phone discrimination and
speaker classification probing tasks. It was observed that the autoregressive
predictive coding loss showed the strongest correlation with downstream
performance on both tasks; however, models were not further analyzed
internally. An evaluation of the similarity of representations trained
according to the three criteria above (but with different architectures and
directionality of contextual information) also showed that it is the training
criterion that most influences the information encoded in the representations,
not the architecture of the learner or the directionality of the input. 

A similar insight was obtained in \cite{zhou2020}, which compared vq-vae and
vq-wav2vec with respect to their ability to discover phonetic units.
The vq-vae model extracts continuous features from the audio signal; a
quantizer then  maps them into a discrete space, and a decoder is trained to
reconstruct the original audio conditioned on the latent discrete
representation and the past acoustic observations. By contrast, vq-wav2vec
predicts future latent discrete representations based on contextualized
embeddings of past discrete representations, in a CPC-style way. The models
were evaluated according to their ability to discover phonetic units (as
measured by phone recognition error rate on TIMIT, and the ZeroSpeech ABX task
(see \cref{sec:zero} for more details)), and it was found that the predictive vq-wav2vec
model fared better than the autoencoder-like vq-vae model, most likely due to
its superior ability to model temporal dynamics.

\subsection{Effects of data and model size}
\label{subsec:modelsize}

How does the performance of self-supervised models change in relation to the
amount of training data, and in relation to the size (number of parameters) of
the model?
Several studies have demonstrated better downstream performance when using
larger datasets~\cite{rivi20, kawakami2020learning, chen2021wavlm}. For
example,
\cite{kawakami2020learning} compared representations learned by a bidirectional
CPC model from the standard 960 hour LS corpus and a corpus of 8,000 hours of
diverse speech from multiple sources.\textsuperscript{\ref{footnote:CPC-8k}}
Not surprisingly, an ASR model trained on top of these representations
performed better when representations were learned from the larger dataset. 
Although the precise relationship between data size and performance has not
been quantified, we can assume that it follows a law of diminishing returns (or
power law), 
similar to observations for most data-intensive machine learning tasks.
In addition to the size of the dataset, the diversity of the data also seems
to play a role, although this was not quantified in this study. However, recent
experiments with larger and more diverse data collections~\cite{chen2021wavlm}
confirm this assumption, as do 
explicit investigations of domain shift robustness (see
\cref{subsec:robustness} below). Using the Mockingjay 
model~\cite{liu2020mockingjay}, the authors in \cite{pue2021scaling} attempt to establish a relationship between
model size and self-supervised \ensuremath{L_1}{} loss and demonstrate that it approximately
follows a power law. Model size and accuracy on downstream phone
classification and speaker recognition tasks are positively correlated but do not
exactly follow a power law; rather, the accuracy saturates as models increase in
size, possibly due to the lack of a corresponding expansion in training data
size.

\subsection{Robustness and transferability}
\label{subsec:robustness}

It is well known that traditional speech features like MFCCs lack
robustness against environmental effects such as additive noise,
reverberation, accents, etc., that cause differences in the distributions of
speech features.
Do pre-trained representations offer greater robustness against distributional
shifts? 
One study~\cite{kawakami2020learning} compared pre-trained
representations from a CPC model against MFCCs and 
found pre-trained representations to be more robust to mismatches between
training and test data. The 
training data consisted of clean, read speech (LS)
whereas test data consisted of the Switchboard corpus and TED talks. The
distributional shifts here may stem from both the acoustics (microphone, room
reverberation) as well as
lexical effects related to topic and style, as well as differences in speaker
characteristics such as accent. 
Similar problems were also investigated using HuBERT and wav2vec~2.0 models in
\cite{chang2021exploration}.
In \cite{robustw2v2} domain effects were studied in greater detail using 
datasets from six different domains. In particular, the authors focused on the
usefulness of adding out-of-domain data to pre-training. The general
conclusions are that pre-training on more and diverse domains is preferable:
models pre-trained on more domains performed better than those pre-trained on
fewer when tested on held-out domains, regardless of which additional
labeled data was used for fine-tuning. Adding in-domain unlabeled data---if
available---to pre-training improves performance robustly; however, even
out-of-domain unlabeled data is helpful and closes  66--73\% of the performance
gap between the ideal setting of in-domain labeled data and a competitive
supervised out-of-domain model. 
 
In \cite{rivi20} the effectiveness of CPC-trained representations for
phone discrimination tasks was compared across several languages. It
was found that representations pre-trained only on English successfully
enabled phone discrimination in 10 other languages, rivaling
supervised methods in accuracy in low-data regimes (1h of labeled data
per language). Thus, self-supervised pre-training enables the model to
learn contextualized speech features that generalize across different
languages. In \cite{conneau20xling}, a wav2vec~2.0 model was trained on data
from multiple different languages and different corpora (Babel, Common Voice,
and multilingual LS)
jointly, followed by fine-tuning for each individual language. The largest
model covers 53 languages in total
and consists of 56,000 hours of speech. Compared to monolingual pre-training,
even smaller models trained on only ten languages improve performance
substantially on a downstream character-based ASR task. Low-resource languages
with little labeled data improve the most under this training regime.
Multilingual representations also resulted in competitive performance (lower
character error rate than monolingual representations) for languages not
present in the training dataset, again showing that unsupervised pre-trained
representations can learn generic features of the speech signal that generalize
across different languages. The study also found that sharing data from closely
related languages is more beneficial than combining distant languages. An
analysis of language clusters in the shared discrete latent representation
space revealed that similar languages do indeed show a higher degree of sharing
of discrete tokens. 
Finally, one might ask whether the interpretation of representations extracted
from different layers of a self-supervised models also generalizes to the
multilingual setting. Experiments in \cite{baevski2021unsupervised} on phone
recognition in eight languages based on the different layers of the
multilingual wav2vec~2.0 XLSR-53 model indicate that this is indeed the case:
phone error rates showed the same pattern as in the monolingual (English)
scenario, with lower phone error rates for middle layers as opposed to
earlier/later layers. 






\section{From Representation Learning to Zero Resources}
\label{sec:zero}



In the self-supervised learning framework, speech representations can be
learned and used in various downstream tasks to achieve competitive, robust,
and transferable performance, as shown in
\crefrange{section:benchmark}{analysis}. However, labeled data is still
required. For example, in ASR, utterances and their manual transcriptions are
needed to learn downstream models or fine-tune representation models. 
A natural question to ask is this: can a model learn without any labeled data? 
In \cref{secsec:unpaired}, we show how to learn ASR models without any paired
audio and text and how SSL improves the framework for learning from unpaired
data.
In addition, many languages have no writing system.
In \cref{subsec:zero}, the SSL representation is further used in scenarios
where text data is not available.

\subsection{Unpaired text and audio} \label{secsec:unpaired}




\begin{table*}[ht]
\centering
    {\renewcommand*\arraystretch{1.4}}
\caption{
Unsupervised ASR.
TIMIT numbers are phoneme error rates (PER), while the numbers for
LibriSpeech are word error rates (WER).
SWC $=$ spoken word classifier, ST $=$ speech translation.
All speech and text are in English if not specified. 
}
\label{fig:unsupervisedASR}
\resizebox{0.99\textwidth}{!}{
\begin{tabular}{cllllll}
    \toprule
Reference & Speech representation & Speech segmentation  & Text token/representation & Mapping approach & Refinement  & Results  \\ 
    \midrule
\cite{liu2018completely}& Audio word2vec~\cite{Wang2018SegAudioWord2Vec} & Oracle & Phoneme & GAN & - & TIMIT (PER): 63.6\%  \\
  \midrule
\cite{yeh2018unsupervised} & MFCC & GAS~\cite{wang2017gate} & Phoneme & Empirical-ODM~\cite{Liu2017ODM} & Self-training &  TIMIT (PER):  41.6\%  \\
  \midrule
\cite{chen2019completely} & MFCC & GAS  & Phoneme & GAN & Self-training &  TIMIT (PER):  33.1\%   \\
  \midrule
\cite{baevski2021unsupervised} & Wav2vec 2.0~\cite{baevski2020wav2vec} & k-means{} & Phoneme &GAN & Self-training & \tabincell{l}{TIMIT (PER):  18.6\%, \\ LibriSpeech (WER): 5.9\%}   \\
  \midrule
\cite{chung2018unsupervised} & Speech2vec~\cite{chung2018speech2vec} & BES-GMM~\cite{Kamper17BESGMM} &  Word2Vec & GAN & Self-training & SWC (Acc): 10.9\%  \\
  \midrule
\cite{chung2019towards} & \tabincell{c}{Speech2vec\\(English)} & Oracle & \tabincell{c}{Word2Vec\\(French)} & VecMap~\cite{artetxe-etal-2018-robust} & \tabincell{l}{LM rescore,\\sequence DAE} & ST (BLEU): 10.8\%   \\
    \bottomrule
\end{tabular}
}
\end{table*}



\subsubsection{Unsupervised ASR}
If only unpaired speech and text are available, that is, the text is not a
manual transcription of speech, can the machine learn how to transcribe speech
into text?
This scenario is called \textit{unsupervised ASR}, and the framework is as
below. 
Given a set of unlabeled utterances $\mathcal{S}=\{S_1, S_2, ..., S_N\}$  and a
set of sentences $\mathcal{Y}=\{Y_1, Y_2, ..., Y_M\}$,\footnote{Note that
the speech and text are not paired, that is, $Y_i$ is not the transcription of
$S_i$.} a mapping function~$F$, which can take an utterance~$S$ as input and
generate its transcription, is learned from data. 
\Cref{fig:unsupervisedASR} summarizes recent work on unsupervised ASR,
including the speech representation used, the algorithm used to learn the
mapping without supervision, and the results. Below, we will discuss these
methods in more detail.

Generative adversarial networks (GANs)~\cite{goodfellow2014generative,
arjovsky2017wasserstein, gulrajani2017improved} are one way to learn such a
mapping function.
The GAN framework includes a discriminator and a generator.
The mapping function~$F$ plays the role of the generator, which takes speech
utterances as input and outputs text.
The discriminator learns to distinguish real text from generated
output, and the generator learns to ``fool'' the discriminator.
The generator and the discriminator are trained in an iterative, 
interleaved way. 
After the training, the generator serves as the speech recognition model.
Another way to map speech and text is via segmental empirical output distribution
matching (segmental empirical-ODM)~\cite{yeh2018unsupervised}, in which the
generator learns to make n-grams of output tokens close to n-grams
calculated from $\mathcal{S}$. 

Success in unsupervised neural machine translation
(MT)~\cite{artetxe2017unsupervised, conneau2017word, lample2017unsupervised}
has inspired innovative exploration of various unsupervised ASR algorithms.
If learning a translation model from unaligned sentences in two languages is
possible, considering speech and text as two different languages, learning the
mapping relationship from speech space to text space without an alignment 
  should likewise be possible.  
However, there are differences between unsupervised MT and unsupervised
ASR.
In unsupervised MT, most discrete source tokens can be mapped to specific
target tokens representing the same meaning. 
However, because speech has segmental structures, in unsupervised ASR, each text
token maps to a segment of consecutive acoustic features of variable length in
an utterance.
The generator is supposed to learn the segmental structure of an utterance
because information like token boundaries is not directly available.
This makes unsupervised ASR more challenging than unsupervised MT.

For unsupervised ASR to be feasible, the common idea is to make the speech and
text units close to each other.
For the text side, word sequences can be transformed into phoneme sequences if
a lexicon is available. 
However, we must first convert the speech signal into something close to phonemes. 
To achieve that, most studies on unsupervised ASR use a phoneme
segmentation module before the generator to segment utterances into
phoneme-level
segments~\cite{liu2018completely,chen2019completely,yeh2018unsupervised}. 
A representation vector or a token then represents each phoneme-level segment. 
It is easier for the generator to map each segment-level representation 
or token to the correct phoneme when the representation or token is highly 
correlated to the phonemes.
Wav2vec-U~\cite{baevski2021unsupervised} selects the input feature from
different layers of wave2vec~2.0~\cite{baevski2020wav2vec}.
The selection criterion is based on analysis of the phonetic information in each layer.
Another series of work is to transform a word into a word embedding. 
\cite{chung2018unsupervised,chung2019towards} use a GAN to map the word-level
speech embedding space~\cite{chung2018speech2vec} to the word embedding space
and achieve promising performance on spoken word classification, speech
translation, and spoken word retrieval.
\Cref{fig:unsupervisedASR} summarizes the various ways to segment speech and represent
speech and text in each reference.  

\begin{comment}
\TNS{if we are limited on space,we can consider removing some of the detail of the various algorithms}
\TNS{split into 2 sentences, also have a connection to the previous para}
\cite{liu2018completely} is the first work to successfully achieve unsupervised
phoneme recognition by first clustering phoneme-level
embeddings~\cite{Chung2016AudioWord2Vec,Wang2018SegAudioWord2Vec} into a set of
tokens and then using GAN to learn the mapping relationship between tokens and
phonemes. 
However, in \cite{liu2018completely}, oracle phoneme segmentation is still
required to get reasonable performance. 
\cite{chen2019completely,yeh2018unsupervised} uses gate activation signals
(GAS)~\cite{wang2017gate} to get the phoneme-level segmentation, which is
unsupervised. 
Wav2vec-U~\cite{baevski2021unsupervised} selects the input feature from
different layers of wave2vec 2.0~\cite{baevski2020wav2vec}, a self-supervised
model.
The selecting criterion is the PER by training a linear model in a supervised manner. 
Wav2vec-U uses k-means{} to cluster the selected features, and the boundaries
are drawn whenever the clustered index changes.
While Wav2vec-U merges the neighboring segments containing the same predicted
labels in each step of GAN training, this design can be viewed as refining the
segmentation implicitly.  
\end{comment}


As shown in \cref{fig:unsupervisedASR}, after GAN training, most studies use
\textit{self-training} to refine the models.
In self-training, the generator serves as the first-version phoneme recognition model.
Inputting unpaired speech to the generator generates the corresponding
``pseudo transcription''. 
We then view the speech utterances and their pseudo transcriptions as paired
data which we use to train a model in a supervised manner.
Although the pseudo transcriptions have more errors than oracle
transcriptions, experiments show that training models on pseudo
transcriptions still significantly boosts performance compared to the
first-version model.

Wav2vec-U~\cite{baevski2021unsupervised} achieved state-of-the-art results
at the time, which suggests that representation learning is
essential for the success of unsupervised ASR.
It achieved an 11.3\% phoneme error rate on the TIMIT benchmark. 
On the LS benchmark, wav2vec-U achieved a 5.9\% word error rate on
\emph{test-other}, rivaling some of the best published systems trained on 960 hours of
labeled data from only two years earlier. 
The robustness of wav2vec-U was further analyzed with respect to 
domain-mismatch scenarios in which the domains of unpaired
speech and text were different~\cite{lin2021analyzing}.
Experimental results showed that domain mismatch leads to inferior performance,
but a representation model pre-trained on the targeted speech domain 
extracts better representations and reduces this drop in performance.

\begin{comment}
In unsupervised learning, how to determine the hyperparameters is an issue. 
In a typical supervised framework, hyperparameter tuning is done by a
development set, which includes labeled data.
In an unsupervised setting, the existence of this kind of development is tricky.
The existence of such kind of development set means the presence of the labeled
data.
Therefore, wav2vec-U~\cite{baevski2021unsupervised} develops a new approach to
determine the hyperparameters without a labeled development set. 
\end{comment}


\subsubsection{ASR-TTS}
Here we describe an alternative approach by which to train an ASR and TTS system
based on unpaired text and audio. The ASR-TTS framework, which combines the ASR and
TTS systems in a cascaded manner,
can be regarded as an autoencoder, where the encoder~$f$
corresponds to the ASR module and the decoder~$g$ corresponds to the TTS module.
In this framework, we consider the intermediate ASR output as a latent
representation; the framework as a whole can be regarded as a variant of
self-supervised learning.\footnote{However, to make this complicated
system work, we often require that data is paired. Therefore, in practice,
ASR-TTS and other methods described in this section are categorized as
semi-supervised learning.}


\subsubsection{ASR-TTS}
Here we describe an alternative approach by which to train an ASR and TTS system
based on unpaired text and audio. The ASR-TTS framework, which combines the ASR and
TTS systems in a cascaded manner,
can be regarded as an autoencoder, where the encoder~$f$
corresponds to the ASR module and the decoder~$g$ corresponds to the TTS module.
In this framework, we consider the intermediate ASR output as a latent
representation; the framework as a whole can be regarded as a variant of
self-supervised learning.\footnote{However, to make this complicated
system work, we often require that data is paired. Therefore, in practice,
ASR-TTS and other methods described in this section are categorized as
semi-supervised learning.}

The ASR-TTS framework can jointly optimize both ASR and TTS without using 
paired data~\cite{tjandra2017listening,hori2019cycle,wang2020improving}.
A speech chain~\cite{tjandra2017listening,tjandra2018machine} is one 
successful way to utilize audio-only and text-only data to train both
end-to-end ASR/TTS models.
This approach first prepares pre-trained ASR and TTS models
$f^{\mathrm{asr}}(X; \theta^{\mathrm{asr}})$ and $g^{\mathrm{tts}}(Y;
\theta^{\mathrm{tts}})$ with learnable parameters $\theta^{\mathrm{asr}}$ and
$\theta^{\mathrm{tts}}$, respectively.
By following the ASR system with a TTS system, we generate new 
text~$\hat{S}$, which must be close to each other.
Thus, we design a loss function $L(S, \hat{S})$ between the original text~$S$
and the generated text~$\hat{S}$:
\begin{equation}
    L(S, \hat{S}; \theta^{\mathrm{asr}}, \theta^{\mathrm{tts}}),
\end{equation}    
where
\begin{equation}
    \hat{S} = g^{\mathrm{tts}}(f^{\mathrm{asr}}(S; \theta^{\mathrm{asr}}); \theta^{\mathrm{tts}}).
    \label{eq:asr-tts}
\end{equation}
Thus, we train the ASR parameters (or both ASR and TTS parameters) using only
the text data as
\begin{equation}
\hat{\theta}^{\mathrm{asr}} = \arg \min _{\theta^{\mathrm{asr}}} L(S, \hat{S}; \theta^{\mathrm{asr}}, \theta^{\mathrm{tts}}).
\end{equation}
Note that this approach does not require the supervised data~$Y$.
As an analogy to the generative approach in \cref{sec:generative}, the
intermediate ASR output~$\hat{Y}$ can be regarded as the latent representation~$Z$.

The other cycle with the text-only data~$Y$ is also accomplished by the
concatenated TTS-ASR systems:
\begin{equation}
    \hat{Y} = f^{\mathrm{asr}}(g^{\mathrm{tts}}(Y; \theta^{\mathrm{tts}}); \theta^{\mathrm{asr}}).
    \label{eq:tts-asr}
\end{equation}
Similarly, this approach does not require the supervised data~$S$, and the
intermediate TTS output~$\hat{S}$ can be regarded as the latent representation~$Z$.
Although this approach initially freezes either the ASR or TTS model, 
extensions of this study~\cite{hori2019cycle,tjandra2019end,baskar2019semi}
implement the joint training of both ASR and TTS parameters using 
REINFORCE~\cite{williams1992simple} and straight-through estimators.

An emerging technique uses a well-trained TTS system to generate
speech and text data from text-only data.
This technique is a sub-problem of the TTS-ASR approach formulated in
\eqref{eq:tts-asr} in which we fix the TTS system part and estimate only the ASR
parameters. 
For example, a huge amount of text resources can be obtained from the web
and document archives without corresponding audio data.
The typical use case scenario of such a text resource for ASR is through the
language model $P(Y)$. 
We combine the ASR and language model via a noisy channel 
model~\cite{jelinek1997statistical}, a weighted finite state 
transducer~\cite{mohri2002weighted}, or shallow 
fusion~\cite{gulcehre2015using,chorowski2016towards}.
However, the progress of TTS systems boosted by deep 
learning~\cite{oord2016wavenet,shen2018natural} has inspired another interesting and
straightforward research direction: artificially creating paired text and audio
data $\{\hat{S}, Y\}$ with only text data~$Y$ by generating the corresponding
audio data~$\hat{S}$ with TTS as
\begin{equation}
    \{\hat{S}, Y\} \text{ where } \hat{S} = g^{\mathrm{tts}} (Y),
\end{equation}
where $g^{\mathrm{tts}} (\cdot)$ corresponds to a TTS function.
The most straightforward approach is to simply use multi-speaker TTS to
generate the waveform with various acoustic 
variations~\cite{li2018training,ueno2019multi,rosenberg2019speech,laptev2020you,huang2020using}.
The other approaches are based on the generation of high-level (more
linguistic) features instead of generating the waveform, e.g., encoder
features~\cite{hayashi2018back} and phoneme 
features~\cite{renduchintala18_interspeech,masumura20_interspeech}.
This approach is similar to the back-translation technique developed in 
NMT~\cite{sennrich2016improving}.
One benefit of the above data generation approaches is that it can be used to feed
unseen word or context phrases to end-to-end ASR.


\subsection{No text or lexicon} \label{subsec:zero}


\subsubsection{Zero-resource speech technologies and challenges}
\label{zero_speech}
Zero-resource speech technologies, which seek to discover linguistic concepts
from audio only (no text nor lexicon), are one of the most active applications
of unsupervised/self-supervised speech processing.
Zero-resource speech technologies were initially studied for acoustic and
linguistic unit discovery from speech data without linguistic resources,
e.g., transcriptions and other annotations~\cite{jansen2013summary}.
This study was motivated by unsupervised query-by-example, applications of
non-parametric Bayesian machine learning to speech processing, and low-resource
speech recognition, and was also inspired by the learning process of infants.
The goal of this type of work is to build spoken dialog systems in a zero-resource
setup for any language.
  To encourage      zero-resource research,                 
zero-resource speech challenges have been organized since 2015.

In this section, we describe the research directions of zero-resource speech
technologies by following the series of zero-resource speech challenges.
\begin{itemize}
	 \item Zero Resource Challenge 2015~\cite{versteegh2015zero}: 
	 In this first challenge, the main focus was to build an acoustic model without using any
	 linguistic annotations based on subword unit modeling and spoken term
	 discovery tracks.
	 For the subword unit modeling track, the ABX score for the within- and across-speaker
	 tasks was used as an evaluation metric.
	 The spoken term discovery track used the normalized edit distance and coverage
	 scores in addition to the precision, recall, and F1 scores for types,
	 tokens, and boundaries.
    Both tracks were based on the English and Xitsonga languages.
    (add technical trends)
	 \item The Zero Resource Challenge 2017~\cite{dunbar2017zero} focused on 
	 unseen language and speaker aspects from the previous challenge. For
	 example, to demonstrate the robustness against unseen languages, the systems
	 were developed with English, French, and Mandarin and tested on
	 two ``surprise'' languages: German and Wolof.
	 Similarly, robustness against unseen speakers was demonstrated by varying
	 the amount of speech available for each speaker.
	 \item The Zero Resource Challenge 2019~\cite{dunbar2019zero} extended a goal of
	 previous challenges by synthesizing speech without
	 text or phonetic labels but with acoustic units obtained using
	 zero-resource techniques.
	 The evaluation metrics were also extended to subjectively evaluate the
	 quality of synthesized speech, including its intelligibility, naturalness,
	 and speaker similarity.
	 \item The Zero Resource Challenge 2020~\cite{dunbar2020zero} was based on two
	 tracks, revisiting previous challenges with different evaluation metrics. 
	 The first task revisited the 2019 challenge with low bit-rate subword
	 representations that optimize the quality of speech synthesis. The second
	 task revisited the 2017 challenge by focusing on the discovery of word-like
	 units from unsegmented raw speech.
	 \item The Zero Resource Challenge 2021~\cite{nguyen2020zero}, the latest
	 challenge, expanded the scope to include language modeling tasks.
	 In addition to phoneme-level ABX, the challenge includes lexical,
	 semantic, and syntactic evaluation metrics computed via a language model of
	 pseudo-acoustic labels.
\end{itemize}

These challenges have facilitated the tracking of technical trends in
zero-resource speech technologies.
For example, research directions thereof 
have expanded to various speech processing components to cover the entire
spoken dialogue systems.
To keep up with this expansion, the challenge has continued to develop
appropriate evaluation metrics for zero-resource scenarios.
Following the success of representation learning, baseline and challenge
techniques have shifted from purely generative 
models~\cite{ondel2016variational,heck2017feature}, deep 
autoencoders~\cite{tjandra19_interspeech,chorowski2019unsupervised}, and incorporation of
neural-network-based TTS/VC techniques~\cite{niekerk20b_interspeech} to
self-supervised learning~\cite{maekaku21_interspeech}.
The latest challenge included the visual modality, continuing the
expansion to include more aspects of human interaction.

\subsubsection{Textless NLP}

Textless NLP is a new research direction that leverages the progress mentioned
above in self-supervised speech representation learning to model language
directly from audio, bypassing the need for text or 
labels~\cite{GSLM,SpeechResynthesis_IS21,pGSLM,emotion_conversion}. 
Not only does this open the gate for language and dialect modeling without
orthographic rules, but it also offers the opportunity to model other
non-lexical information about how speech is delivered, e.g., speaker identity,
emotion, hesitation, interruptions. 
The generative spoken language model (GSLM)~\cite{GSLM} utilizes discrete
representations from wav2vec~2.0, HuBERT, and CPC algorithms as inputs to an
autoregressive language model trained by cross-entropy to maximize the
probability of predicting the next discrete speech token. A synthesis module
follows the language model to produce speech waveforms given the generated
discrete speech units. The generated spoken continuations compete with
supervised generations and synthesis using a character language model in
subjective human evaluations. The model completes incomplete words
(pow[..] $\rightarrow$ POWER) and continues using words in the same general mood (dark $\rightarrow$
BLACKNESS)\footnote{https://speechbot.github.io/gslm/} and has been extended to
model and generate
dialogues~\cite{dialogue_GSLM}.\footnote{https://speechbot.github.io/dgslm/}
Given its flexibility in modeling spoken content, the GSLM has been further extended
to jointly model content and prosody~\cite{pGSLM}. This prosodic-GSLM model
introduced a multistream causal Transformer, where the input and output layers
use multiple heads to model three channels:  discrete speech units, duration,
and quantized pitch. The prosodic-GSLM model jointly generates novel content
and prosody congruently in the expressive 
style of the prompt.\footnote{https://speechbot.github.io/pgslm/}
Going one step further, \cite{emotion_conversion} used a speech emotion
conversion framework to modify the perceived emotion of a speech utterance
while preserving its lexical content and speaker identity. Other studies have
extended the idea of textless language processing or audio discrete representation to applications such as 
spoken question answering~\cite{spoken_qa_dual}, speech separation~\cite{shi2021discretization}, TTS~\cite{hayashi2020discretalk}, and speech-to-speech
translation~\cite{textless_translation}.




\section{Discussion and Conclusion}
\label{sec:conclusion}

In this overview, we review current speech SSL models, benchmark and
analyze results over SSL, and demonstrate zero-resource downstream tasks
that leverage SSL.
Below we recommend various research directions:
\begin{itemize}
\item \textbf{Using the representation model.} So far, there are two main ways to
use representation models: Freeze the representation models and
use them as feature extractors, or fine-tune the representation models
to downstream tasks. Are there other ways to use these models? In the NLP community,
the adapter-based
method~\cite{pmlr-v97-houlsby19a,zaken2021bitfit,guo-etal-2021-parameter} and
prompt/instruction learning~\cite{liu2021pretrain} yield competitive
performance compared with fine-tuning, but 
  these methods have yet to be fully exploited for speech.  

\item \textbf{Increasing the efficiency of the representation model.} As discussed in
\cref{subsec:modelsize}, larger representation models lead to better
downstream performance.
Despite the success of these huge models, they require large amounts of memory and high
pre-training costs, which makes them unsuitable for edge devices.
Preliminary studies have been conducted on compressing speech representation
models through network pruning~\cite{PARP} or knowledge
distillation~\cite{chang2021distilhubert}.
On the other hand, non-streaming architectures in models such as
the bidirectional Transformer have hindered the representation model used in streaming
scenarios, leading to studies that address these problems~\cite{StreamingW2v}.
We anticipate continuing research in these directions in the future.
\item \textbf{Creating robust models.} As discussed in \cref{subsec:robustness}, studies 
have been conducted on the robustness of representation models. However, the
failure modes thereof are still poorly understood. Does the model fail on some downstream tasks?
Are they robust to adversarial attacks? Preliminary studies investigate
adversarial attacks on speech represention models~\cite{wu2021characterizing}.
\item \textbf{Using text representation models to improve speech representation.}
The amount of content information in speech corpora used to train speech
representation models is far less than that of text representation
models. Noting that the BERT training corpus exceeds 3 billion
words~\cite{jacob2019BERT}, and assuming a typical speaking rate of 120 words per minute,
a speech corpus containing the same content as the BERT training
data would include 400,000 hours of audio, which exceeds than the training data of all
current speech representation models put together.
Therefore, to enable speech representation models to better learn human language,
for instance by extracting semantic information from acoustic signals, the use of text
models such as BERT and GPT is key: nevertheless, how to use these to improve 
speech representation model pre-training remains an open question.
\end{itemize}

We believe SSL representation models 
  have considerable room to grow.     
The relationship between representation models and downstream tasks can be
compared to the relationship between operating systems and applications.
Today, even individuals can build applications with desired functions 
on a smartphone because the smartphone's operating system handles the
complex communication with the hardware and provides a convenient developer interface.
Likewise, as SSL representation models learn general knowledge from human
speech, it is easy to develop new speech processing applications on this
basis.
From this viewpoint, representation speech models will play the role of
operating systems in speech processing and further facilitate the continued 
development of speech technology.
  Suppose, for instance, a speaker of an indigenous language seeks to            
  purchase an intelligent assistant, but discovers that it does not yet support   %
  the indigenous language.                                                        %
Because the smart assistant has a built-in representation model, though, it
can quickly learn a new language with little supervision.
We believe SSL representation models will bring the benefits of speech
technology to more people.

















