\subsection{Devices used in PA data recording}
The replay and recording devices used to create PA data is shown in Table.~\ref{tab:PA_device}. All setups are cited from the specifications available at the official websites.

\begin{comment}
GENELEC: https://www.genelec.jp/studio-monitors/classic-studio-monitors-subwoofers/8030-studio-monitor/
https://assets.ctfassets.net/4zjnzn055a4v/1my4dzvqUoe4oKYwS6KsUu/a8f964c3ce07eb802b88b8d861799a07/8030c_opman_gb_fi.pdf
NEUMMAN: https://www.neumann.com/homestudio/en/kh-80
SONY: https://www.sony.jp/active-speaker/products/SRS-XB43/
Marantz: MPM-1000 http://marantzpro.jp/mpm-1000/
http://www.marantzpro.jp/mpm-1000/data/mpm-1000_userguide_v1.0-jp.pdf
M-audio: https://www.m-audio.com/ubermic
\end{comment}


\subsection{Description of Top-Performing Systems}
The top-5 systems for each scenario are ranked and selected respectively based on minimum t-DCF. The architectures are presented in terms of block diagrams which were drawn based on the system descriptions and feedback from the challenge participants.

For a unified presentation, we break down each system into four cascaded processing blocks: 
\begin{itemize}
    \item \textbf{Audio pre-processor} includes the steps to partition data and process input wave files. Most of the methods in this step are applied only at training phase.
    \item \textbf{Input processor} contains the acoustic frontends used for further processing. Learnable acoustic frontends are also categorized to this part.
    \item \textbf{Classifier} covers the neural network as well as statistical models for the countermeasure system. Online augmentation methods and training strategies (if any) are also covered here.
    \item \textbf{Post-processor} contains score-domain operations, including
    score fusion and normalization.
\end{itemize}

\noindent In the following (starting from next page) we give short descriptions to the architecture of the illustrated systems.


\newpage

\begin{figure*}[t]
    \centering
    \includegraphics[width=1.0\textwidth]{figs/support_figs/LA.pdf}
        \caption{Illustration of top-5 systems for the LA scenario. The descending order is based on performance via minimum t-DCF values on evaluation set. Numbers in brackets are the values. Dash lines indicate steps/modules which are only applied for training, while solid lines indicate ones applied for both training and testing.}
    \label{fig:LA}
\end{figure*}

\subsubsection{Logical Access (LA)}




\textbf{T23}: Multiple sub-systems are trained based on the codec-augmented and trimmed audio. Most of them are based on various types spectral features and two parallel classifiers, LightCNN (LCNN) and ResNet. They are forwarded to LSTMs and the scores are summed up with weights. Further, another sub-system uses a SincNet frontend and an LSTM backend. The output score of this subsystem is further fused with the weighted averaged score above.

\textbf{T35}: The input audio is processed using pre-emphasis and a-law companding algorithm. LFCC features are extracted from the processed audio and fed into two ResNet-based classifiers. The output scores from the two classifiers are averaged.

\textbf{T19}: The training data is augmented via RIR and MUSAN datasets, followed by trimming and additive noise. The mel spectrograms are fed into a ResNet with squeeze-and-excitation in its building blocks. The output scores are averaged via pre-defined empirical weights.

\textbf{T36} is a combination of multiple classifiers based on RawNet and ECAPA-TDNN. The training data is augmented via additive noise and 3-fold speed perturbation with parameters of 0.9, 1.0 and 1.1. Three types of feature extractors are used --- raw waveform, mel spectrogram and a learnable LEAF frontend, whose outputs are fed into one RawNet and two ECAPA-TDNNs. The output scores are combined using equal weights.

\textbf{T20} uses a ResNet classifier trained with log filterbanks. The training data is augmented via RIR, MUSAN and media codecs such as mp3 and m4a. The log filterbanks are augmented via zero-value masking at frequency domain (FreqAugment). A fully-connected layer is added after the ResNet and trained with \emph{large margin cosine loss} (LMCL).


\newpage

\begin{figure*}[t]
    \centering
    \includegraphics[width=1.0\textwidth]{figs/support_figs/PA.pdf}
    \caption{Illustration of top-5 systems for the PA scenario. The descending order is based on performance via minimum t-DCF values on evaluation set. Numbers in brackets are the values.  Dash lines indicate steps/modules which are only applied for training, while solid lines indicate ones applied for both training and testing.}
    \label{fig:PA}
\end{figure*}

\subsubsection{Physical Access (PA)}
\textbf{T07} is an one-class learning framework based on \emph{Gaussian mixture models} (GMM) and \emph{variation autoencoder} (VAE). The training data is augmented via room simulation \footnote{via \emph{pyroomacoustics}: \href{https://github.com/LCAV/pyroomacoustics}{https://github.com/LCAV/pyroomacoustics}} and 2-fold speed perturbation with parameters of 0.9 and 1.1. The acoustic feature used is log spectrogram, which is fed into a neural vocoder to create filtered spectrograms. Applied candidates for the vocoder are WORLD, HifiGAN, and MelGAN. The original and processed spectrograms, which are time-frequency representations, are averaged via the temporal axis. One GMM and one VAE are trained via the temporal-averaged spectrograms and another GMM is trained via the original log spectrogram and the filtered spectrogram. The output scores are combined via equally weighting.

\textbf{T16} uses a combination of SEnet and GMM trained with various short-term spectral features. The data is augmented via in-house replay data. The scores are fused using empirically weighted averaging.

\textbf{T23}: A parallel combination of multiple sub-systems trained on trimmed speech segments, augmented via RIR, MUSAN and other additive noises. Mel spectrogram with various number of frequency bins and LEAF frontend are the feature extractors. Classifiers are implemented via ResNet, whose output are forwarded to scors via \emph{attentive statistical pooling} (ASP). Final score is obtained via empirically weight averaging.

\textbf{T01}: The pre-emphasized audio is input to produce three different types of acoustic features: LFCCs, magnitude spectrogram (MagSpec), and MFCCs. Each of them is fed into a LightCNN classifier and the output score is obtained by fusing the output of the three classifiers using equal weights.

\textbf{T04}: A TDNN network with statistical pooling. The pre-emphasized audio is used to produce another three types of features: MFCCs, CQCCs and CQT spectrogram.


\newpage
\begin{figure*}[t]
    \centering
    \includegraphics[width=1.0\textwidth]{figs/support_figs/DF.pdf}
    \caption{Illustration of top-5 systems for the DF scenario. The descending order is based on performance via equal error rates on evaluation set. Numbers in brackets are the error rates.  Dash lines indicate steps/modules which are only applied for training, while solid lines indicate ones applied for both training and testing.}
    \label{fig:DF}
\end{figure*}

\subsubsection{DeepFake (DF)}
\textbf{T23} is a parallel hybrid of three classifiers. Two acoustic frontends process the trimmed and codec-augmented audio: Mel spectrogram extractor and a convolutive frontend built via sinc function. The former is fed into a ResNet and a LightCNN and the latter is fed into a RawNet. The outputs of the three nets is feed into separate bi-directional LSTMs to produce scores, which are finally fused via empirically weight averaging.

\textbf{T20}: The feature extractor, classifier and scoring scheme is the same as one used by the same team in the LA scenario. The training data is augmented via media codecs such as mp3 and m4a, and other additive noises.

\textbf{T08} fuses multiple LightCNN classifiers using LFCCs and CQT spectrograms. The input data is augmented using multiple media and transmission codecs\footnote{G722, alaw, G723, G726, gsm, opus, SPEEX, mp3, m4a, ogg, mp2}. LFCCs are used to train one LightCNN and the remaining classifiers are trained on the other feature. The scores are processed via min-max normalization, before being averaged. 

\textbf{T06} is a combination of a TDNN classifier trained on LFCCs, four SEnet modules trained on LFCCs, two GMMs trained on \emph{product spectral cepstral coefficients} (PSCCs), and three TDNNs trained separately on the two mentioned features and \emph{log linear filterbank energies} (LLFBs).

\textbf{T22} is a convolutional network based on CondenseV2Net, trained on log power spectrum derived from \emph{long-term variable Q transform} (L-VQT). The training data is augmented via mp3, m4a, wma and trimmed to 4-second chunks.

\newpage 

\subsection{Hidden Track Results}
The comparison of EERs for evaluation set and hidden track are presented in Fig.~\ref{fig:hidden_track}. We can see that for LA and DF scenarios, while for the main track the EERs for the top systems are all lower than 10\%, for the hidden track the number raised to more than 20\%. Such pattern can also be found for the PA scenario, with narrower performance gap between the main and hidden tracks. 

The results indicate the effect of silence on the spoofing detection. Meanwhile, the different gap between PA and the other two scenarios may due to the artifact on the length of silence injected in the spoofed and bonafide training data --- for LA and DF, where synthetic speeches are produced via TTS systems, such difference is large. For PA where the attack comes from replaying the audio, such difference becomes smaller. 

\subsection{Challenge Overall Results}

We present full statistical results on progress and evaluation sets in Table \ref{tab:LA_results}, \ref{tab:PA_results} and \ref{tab:DF_results}, for LA, PA and DF scenarios respectively. Same as last section, the team IDs are kept consistent with ones used in the main paper. The IDs starting with `B' are IDs of the baseline systems provided by us, for comparability concern. For LA and PA we present both t-DCF and EER results, while for DF we report EERs.
 






\begin{figure*}[t]
\centering
\subfloat[LA]{
\includegraphics[height=4cm]{figs/support_figs/paper_fig_pre_hidden_track_LA.pdf}
}
\subfloat[PA (hidden track 1)]{
\includegraphics[height=4cm]{figs/support_figs/paper_fig_pre_hidden_track_new_3.pdf}
}
\subfloat[PA (hidden track 2)]{
\includegraphics[height=4cm]{figs/support_figs/paper_fig_pre_hidden_track_new_4.pdf}
}
\subfloat[DF]{
\includegraphics[height=4cm]{figs/support_figs/paper_fig_pre_hidden_track_DF.pdf}
}
\caption{Comparison of CMs EERs on evaluation set and hidden tracks. \textcolor{blue}{Blue dots} correspond to top 1-10 submissions. \textcolor{red}{Red dots} are other submissions. Best viewed in color.}
\label{fig:hidden_track}%
\end{figure*}



\newpage

\begin{table}[h]
    \centering
    \caption{ASVspoof 2021 progress and evaluation results for LA. Results shown in terms of pooled normalised minimum t-DCF and pooled EER [\%].}
    \begin{tabular}{cccccc}
    \toprule
    
           &          &    \multicolumn{2}{c}{Progress set} & \multicolumn{2}{c}{Evaluation set} \\
    \cmidrule(lr){3-4}\cmidrule(lr){5-6}
       \#  & ID       &     t-DCF   &  EER      & t-DCF   &  EER       \\ 
    \midrule
         1   &  T23 &    0.1816 &  0.89  &  0.2176 &  1.32 \\ 
         2   &  T35 &    0.2115 &  2.61  &  0.2480 &  2.77 \\ 
         3   &  T19 &    0.2175 &  2.69  &  0.2495 &  3.13 \\ 
         4   &      &    0.2119 &  2.51  &  0.2500 &  2.81 \\ 
         5   &  T36 &    0.2373 &  3.69  &  0.2531 &  3.10 \\ 
         6   &  T20 &    0.2137 &  2.39  &  0.2608 &  3.21 \\ 
         7   &  T08 &    0.2376 &  3.23  &  0.2672 &  3.62 \\ 
         8   &  T16 &    0.2394 &  3.39  &  0.2689 &  3.63 \\ 
         9   &      &    0.2436 &  3.11  &  0.2725 &  3.61 \\ 
        10   &  T04 &    0.2372 &  4.54  &  0.2747 &  5.58 \\ 
        11   &  T06 &    0.2476 &  5.61  &  0.2853 &  5.66 \\ 
        12   &      &    0.2556 &  4.22  &  0.2880 &  5.01 \\ 
        13   &  T03 &    0.2461 &  3.65  &  0.2882 &  4.66 \\ 
        14   &      &    0.2556 &  5.90  &  0.2893 &  5.70 \\ 
        15   &  T31 &    0.2622 &  5.00  &  0.3094 &  5.46 \\ 
        16   &  T17 &    0.2990 &  7.10  &  0.3279 &  7.19 \\ 
        17   &  T07 &    0.3029 &  7.88  &  0.3310 &  8.23 \\ 
        18   &  T30 &    0.2999 &  7.58  &  0.3362 &  8.89 \\ 
        19   &  B03 &    0.3152 &  8.90  &  0.3445 &  9.26 \\ 
        20   &  T02 &    0.3220 &  7.71  &  0.3445 &  7.79 \\ 
        21   &  T14 &    0.3167 &  8.82  &  0.3451 &  8.98 \\ 
        22   &  T11 &    0.3015 &  6.80  &  0.3666 &  7.19 \\ 
        23   &  T34 &    0.3688 &  13.25 &  0.4059 &  13.45\\ 
        24   &  B04 &    0.4152 &  9.49  &  0.4257 &  9.50 \\ 
        25   &  T15 &    0.2986 &  9.67  &  0.4890 &  14.68\\ 
        26   &  B01 &    0.4948 &  15.80 &  0.4974 &  15.62\\ 
        27   &  T25 &    0.2086 &  2.03  &  0.5148 &  13.75\\ 
        28   &  T32 &    0.4256 &  10.93 &  0.5270 &  12.90\\ 
        29   &      &    0.5887 &  19.04 &  0.5748 &  18.50\\ 
        30   &  B02 &    0.5836 &  21.13 &  0.5758 &  19.30\\ 
        31   &      &    0.2528 &  5.26  &  0.5775 &  14.28\\ 
        32   &  T01 &    0.2773 &  6.32  &  0.6204 &  15.95\\ 
        33   &      &    0.2792 &  7.87  &  0.6288 &  15.87\\ 
        34   &  T24 &    0.3108 &  10.85 &  0.6320 &  15.98\\ 
        35   &      &    0.3233 &  9.72  &  0.6370 &  16.27\\ 
        36   &  T29 &    0.3316 &  11.87 &  0.6741 &  17.41\\ 
        37   &      &    0.3679 &  12.54 &  0.6812 &  17.66\\ 
        38   &  T12 &    0.6900 &  26.26 &  0.7227 &  26.41\\ 
        39   &      &    0.4382 &  13.77 &  0.7233 &  19.19\\ 
        40   &      &    0.6123 &  23.16 &  0.8521 &  26.14\\ 
        41   &      &    0.9988 &  53.16 &  1.0000 &  53.81\\  
    \bottomrule
    \end{tabular}
    \label{tab:LA_results}
\end{table}


\begin{table}[h]
    \centering
    \caption{ASVspoof 2021 progress and evaluation results for PA. Results shown in terms of pooled normalised minimum t-DCF and pooled EER [\%].}
    \begin{tabular}{cccccc}
    \toprule
    
           &          &    \multicolumn{2}{c}{Progress set} & \multicolumn{2}{c}{Evaluation set} \\
    \cmidrule(lr){3-4}\cmidrule(lr){5-6}
       \#  & ID       &     t-DCF   &  EER      & t-DCF   &  EER       \\ 
    \midrule
    1   & T07 &     0.6736 &  23.60 &  0.6824 &  24.25\\ 
    2   & T16 &     0.6790 &  25.37 &  0.7122 &  27.59\\ 
    3   & T23 &     0.7019 &  24.22 &  0.7296 &  26.42\\ 
    4   & T01 &     0.6925 &  25.73 &  0.7446 &  28.36\\ 
    5   & T04 &     0.6676 &  24.80 &  0.7462 &  29.00\\ 
    6   &     &     0.7207 &  27.23 &  0.7469 &  29.22\\ 
    7   & T33 &     0.7263 &  27.00 &  0.7648 &  29.55\\ 
    8   & T08 &     0.7177 &  25.95 &  0.7670 &  29.02\\ 
    9   & T37 &     0.7912 &  32.39 &  0.8216 &  35.07\\ 
    10   & T27 &     0.8102 &  31.62 &  0.8307 &  32.00\\ 
    11   & T26 &     0.7837 &  26.91 &  0.8362 &  29.61\\ 
    12   & T28 &     0.8498 &  30.94 &  0.8879 &  32.96\\ 
    13   &     &     0.8934 &  35.16 &  0.9265 &  37.10\\ 
    14   & B01 &     0.9062 &  36.33 &  0.9434 &  38.07\\ 
    15   & T03 &     0.9084 &  36.37 &  0.9444 &  38.07\\ 
    16   &     &     0.9193 &  35.78 &  0.9530 &  38.50\\ 
    17   & T09 &     0.9531 &  34.39 &  0.9666 &  34.77\\ 
    18   & B02 &     0.9747 &  39.79 &  0.9724 &  39.54\\ 
    19   & T11 &     0.9836 &  40.91 &  0.9939 &  42.55\\ 
    20   &     &     0.9855 &  40.38 &  0.9945 &  42.98\\ 
    21   & B03 &     0.9827 &  42.16 &  0.9958 &  44.77\\ 
    22   & B04 &     0.9993 &  46.03 &  0.9997 &  48.60\\ 
    23   & T10 &     0.9996 &  45.10 &  1.0000 &  45.50\\ 
    \bottomrule
    \end{tabular}
    \label{tab:PA_results}
\end{table}


\begin{table}[h]
    \centering
    \caption{ASVspoof 2021 progress and evaluation results for DF. Results shown in terms of pooled normalised minimum t-DCF and pooled EER [\%].}
    \begin{tabular}{cccc}
    \toprule
    
           &          &    {Progress set} & {Evaluation set} \\
       \#  & ID       &      EER          &  EER       \\ 
    \midrule
        1   & T23 &     0.24 &  15.64\\ 
        2   & T20 &     1.79 &  16.05\\ 
        3   & T08 &     2.93 &  18.30\\ 
        4   &     &     1.65 &  18.80\\ 
        5   & T06 &     8.29 &  19.01\\ 
        6   & T22 &     2.17 &  19.22\\ 
        7   & T03 &     2.32 &  19.24\\ 
        8   & T01 &     4.72 &  19.70\\ 
        9   & T36 &     0.78 &  20.23\\ 
        10  & T19 &     3.00 &  20.33\\ 
        11  & T31 &     6.88 &  20.33\\ 
        12  &     &     2.25 &  20.63\\ 
        13  &     &     2.00 &  20.82\\ 
        14  & T16 &     1.98 &  20.84\\ 
        15  &     &     4.84 &  21.61\\ 
        16  &     &     5.06 &  21.67\\ 
        17  &     &     2.37 &  22.03\\ 
        18  &     &     6.10 &  22.38\\ 
        19  & B04 &     6.10 &  22.38\\ 
        20  & T25 &     4.18 &  22.62\\ 
        21  & B03 &     11.61 &  23.48\\ 
        22  &     &     0.10 &  23.57\\ 
        23  &     &     5.36 &  23.88\\ 
        24  & T35 &     0.42 &  24.12\\ 
        25  &     &     11.82 &  24.89\\ 
        26  & T30 &     4.66 &  25.21\\ 
        27  & B02 &     21.01 &  25.25\\ 
        28  & T26 &     16.40 &  25.41\\ 
        29  & B01 &     17.63 &  25.56\\ 
        30  &     &     3.40 &  26.67\\ 
        31  & T21 &     17.20 &  28.96\\ 
        32  &     &     9.07 &  29.25\\ 
        33  &     &     29.63 &  29.75\\ 
    \bottomrule
    \end{tabular}
    \label{tab:DF_results}
\end{table}


\subsection{Useful Resources}
We provide useful resources for the challenge participants as well as the research community to reproduce the baselines and conduct their own experiments. 

\begin{itemize}
    \item Link to the datasets, in a per-scenario manner: \\ LA -  \texttt{\url{https://zenodo.org/record/4837263}},  \\ PA - \texttt{\url{https://zenodo.org/record/4834716}}, \\ DF - \texttt{\url{https://zenodo.org/record/4835108}}.
    \item Link to the keys and meta-labels, in a per-scenario manner:  \\ LA -  \texttt{\url{https://www.asvspoof.org/asvspoof2021/LA-keys-full.tar.gz}},  \\ PA - \texttt{\url{https://www.asvspoof.org/asvspoof2021/PA-keys-full.tar.gz}}, \\ DF - \texttt{\url{https://www.asvspoof.org/asvspoof2021/DF-keys-full.tar.gz}}.
    \item The challenge baseline repository, which is open-sourced with ASV scores, baseline CM systems and the evaluation package. One can visit \texttt{\url{https://github.com/asvspoof-challenge/2021}} for more details.
\end{itemize}

\section{Introduction}


\IEEEPARstart{B}{iometric} systems implemented with automatic speaker verification (ASV) technology \cite{Bai2021-speaker-rec-deep-learning-overview} are vulnerable to \emph{spoofing attacks}~\cite{ISOpresentationAtack} whereby an adversary attempts to masquerade as another individual through the presentation of manipulated speech data~\cite{wu2015spoofing}. 
There are four principal means to generate attacks: \emph{impersonation}, \emph{voice conversion} (VC), \emph{text-to-speech synthesis} (TTS), and \emph{replay}. 
Impersonation attacks have received the least attention; successful attacks typical require special expertise, e.g.~of professional impersonators.
In contrast, VC, TTS and replay spoofing attacks can all be mounted using readily available software toolkits and consumer devices and have hence received greater attention~\cite{wu2015spoofing}.
The threat posed by such techniques is now well recognised, particularly in academia and increasingly in industry~\cite{VoiceID,Binsider,Pindrop,FacebookMicrosoft}.

The ASVspoof initiative and challenge series~\cite{Wu-ASVspoof2015} was conceived to foster the development of countermeasures (CMs) to protect against the manipulation of ASV systems from spoofing attacks.
ASVspoof has designed, collected and distributed substantial databases of both bona fide and spoofed data, with the latter being generated with a broad range of state-of-the-art VC and TTS technologies, recording and replay devices.
They have been used in the series of four biennial benchmarking challenges, the results of which show tremendous progress in spoofing detection reliability.

The first edition of ASVspoof in 2015~\cite{Wu-ASVspoof2015} focused on the development of CMs for the detection of VC and TTS attacks. 
For the subsequent edition in 2017, the focus switched to replay attacks. 
The third edition, ASVspoof 2019, was the first to address all three attack types through two separate tasks involving: (i)~a logical access (LA) scenario involving VC and TTS attacks generated using a set of different statistical and neural methods; 
(ii)~a physical access (PA) scenario involving replay attacks implemented in a large number of simulated acoustic environments. 

While the overall scope of ASVspoof challenges has been steadily expanding with the introduction of new attack algorithms, evaluation scenarios, and performance metrics~\cite{Kinnunen2020-tDCF-fundamentals}, the technical quality of the audio data to date has remained high; the data  
is relatively free of additive noise and encoding, compression and transmission artifacts. 
While experiments with clean data detached from the complexities of the real world have a legitimate role in basic research, the prolonged use of such data may promote the development of CM solutions which fail to generalize
to in-the-wild application scenarios where external distortions are commonplace.
Furthermore, the past ASVspoof training, development and evaluation sets all originate from the same VCTK source corpus~\cite{vctk}.
This may hinder the design of CMs which generalize to other domains with different speech characteristics.


\begin{figure*}[th!]
     \centering
    
    
     \includegraphics[width=0.99\textwidth]{figs/ASVspoof21_overview_v0.4.pdf}
     \vspace{-2mm}
     \caption{The ASVspoof 2021 challenge consists of three different tasks: logical access (LA); physical access (PA); deepfake (DF). 
     While the general theme of exposing spoofing countermeasures (CMs) to more realistic conditions is common to all three tasks, each was effectively run as a separate sub-challenge. The database and results for each task are described in the sections indicated.
     }
     \label{fig:ASVspoof21_overview}
\end{figure*}


As illustrated in Fig.~\ref{fig:ASVspoof21_overview}, the move towards an evaluation using speech data which is more representative of practical applications 
was the essence of the ASVspoof 2021 challenge.  
It sought to benchmark the latest CM solutions in more realistic conditions 
in which speech data undergoes coding, compression and transmission across telephony channels (LA task, left in Fig.~\ref{fig:ASVspoof21_overview}), or undergoes 
acoustic propagation in a real physical space (PA task, middle in Fig.~\ref{fig:ASVspoof21_overview}).
ASVspoof 2021 also introduced the new deepfake (DF) task (right in Fig.~\ref{fig:ASVspoof21_overview}) in which adversaries aim not to deceive an ASV system, but merely fabricate an utterance in the voice of a target speaker.  
The goal may be to harm the reputation of a well-known personality or to manipulate society by disseminating disinformation or fake news~\cite{DeepfakesNews1,DeepfakesNews2}.

The aim behind the new DF
task is to assess the robustness of spoofing detection solutions when used to detect compressed manipulated speech data of varying characteristics posted online.
The scenario is simulated by processing audio files from different sources with various codecs used in  social media. 



Described in this paper are the key advances made since previous editions of ASVspoof,
together with details of the dataset design, collection and protocol policies.  
Also described are the challenge results and principal techniques that are common to the top-performing systems for all three sub-tasks. 
The study substantially extends the preliminary workshop version~\cite{yamagishi21_asvspoof} with an in-depth analysis of influential data factors that impact upon CM performance. 
Additionally, we present a summary of system descriptions submitted by participating teams and identify the most promising detection algorithms and techniques.
The article also contains a new, detailed analysis of results for specially-designed \emph{hidden subsets} and includes a survey of \emph{post-challenge} studies and related work outside of the scope of ASVspoof. None of these were presented in~\cite{yamagishi21_asvspoof}.  

The new insights presented in this paper, together with newly released meta-data for the three ASVspoof 2021 task databases\footnote{The meta-data was used in the post-evaluation analysis presented herein and is publicly available via the challenge website: \url{https://www.asvspoof.org/index2021.html}.} 
should be helpful to those looking to participate in future ASVspoof challenges as well as all those working in the field. 
Toward the end of the paper we reflect upon the limitations and key lessons learned from ASVspoof 2021, with a discussion of current ideas and directions for future research.


\section{ASVspoof 2021 Challenge Outline}


In this section we outline the three challenge tasks,
each an independent sub-challenge with their own corresponding evaluation database.
All three are partitioned into progress and evaluation subsets in addition to a number of hidden subsets.  
Progress subsets of modest size were used for intermediate assessment prior to the evaluation submission deadline for which participants submitted scores for the full evaluation database comprising the three subsets.
Only progress and evaluation subsets are discussed in this section, with corresponding results being presented in Section~\ref{sec:results}.
Hidden subsets and results are presented in Section~\ref{sec:Hidden-Tracks}.

For all three tasks, all data was distributed in free lossless audio codec (FLAC) format with a sampling rate of 16~kHz and with 16-bit quantisation.
There were no new training data for ASVspoof 2021, though protocols for training and development using data released through previous ASVspoof editions were made available. 
Challenge rules and participant guidelines are available in the challenge evaluation plan~\cite{delgado2021asvspoof}. 







\input{table/All_trial_number}

\subsection{Logical Access (LA)}
\label{sec:LAdb}



The ASVspoof 2015 and 2019 editions both incorporated LA tasks in which all speech signals were clean, i.e.\ without either additive noise, reverberation or other sources of channel variation.
As a consequence, the research effort focused upon the identification of key techniques for the capturing and classification of processing artifacts which result from the generation of spoofed speech signals using either TTS or VC algorithms.
Such ideal conditions are unrealistic and other work has shown the likely degradation in CM performance when they are deployed in practical scenarios such as person authentication over telephony channels~\cite{Delgado_channel_variation2017,Lavrentyeva_phonespoof_2019}.

The ASVspoof 2021 LA task was hence designed to reduce the gap between ideal laboratory conditions and those to be expected in the wild. 
This was achieved with the transmission of both bona fide and spoofed speech across real telephony systems, including a voice-over-internet-protocol (VoIP) system and a public switched telephone network (PSTN).
The 2021 LA challenge hence focuses on the study of robustness to nuisance variation from compression, packet loss and other artifacts stemming from different bandwidths, transmission infrastructures and bitrates, etc.; the consideration of additive noise remains for a future challenge edition. 

\input{table/LA_condition_table.tex}

Speech data are sourced from the ASVspoof 2019 LA evaluation database, itself derived from the VCTK database~\cite{vctk}. 
Spoofed trials are generated using one of 13 different VC, TTS or hybrid spoofing attack algorithms (A07-A19 in~\cite{asvspoof2019database}).
For all but one condition, \emph{both} bona fide \emph{and} spoofed data undergo transmission across either a VoIP or PSTN+VoIP network using one of six  codecs giving the seven evaluation conditions listed in Table~\ref{tab:LA_conditions}.
The choice of codecs is broadly representative of traditional or legacy codecs (a-law, G.722, etc.) and modern IP streaming codecs (OPUS) in use today.

The reference condition C1 is identical to the ASVspoof 2019 LA scenario, i.e.\ with neither encoding nor transmission.
Conditions C2 and C4-C7 correspond to transmission across an Asterisk\footnote{\url{https://www.asterisk.org}} private branch exchange (PBX) system using codecs operating at sampling rates of either 8~kHz (C2: a-law, C4: $\mu$-law, C5: GSM) or 16~kHz (C4: G.722, C7: OPUS).
The bitrate of each data condition (last column of Table~\ref{tab:LA_conditions}) was set commensurate with the codec, with the lowest being 13 kbps for condition C6.
Each utterance is transmitted in its own individual VoIP channel, by automatically generated Asterisk call files, and between a pair of
session initiation protocol (SIP) endpoints
either within a local area network connection, or from an endpoint hosted in France to endpoints in Italy or Singapore. 
Condition C3 involves transmission in Spain over a PSTN system for which codec conditions are uncontrollable and unknown.
Calls are initiated from a mobile smartphone whereas data recordings are made at a SIP endpoint hosted on a professional VoIP system which uses a $\mu$-law codec operating with an 8~kHz sampling rate.  
Condition C3 may hence reflect the application of multiple, unknown intermediate transcodings and network transmissions.
No codec information nor any external metadata was provided in the audio file headers.
The number of trials in the LA progress and evaluation subsets is shown in Table~\ref{tab:All_trial_num}. 
The progress subset contains a modest number of C1-C4 utterances, while the evaluation subset contains the remaining C1-C4 utterances in addition to the full set of C5-C7 utterances. 


The distribution of speakers and spoofing attacks is balanced in each condition, implying that differences in detection performance can be attributed reliably to the variations in encoding and transmission.
While the total number of speakers is the same as that of the ASVspoof 2019 LA evaluation set, the 2021 database contains a substantial number of previously unexposed, new bona fide utterances collected from the same set of speakers in addition to new spoofed utterances generated with the same attack algorithms.

Last, with no new matched training nor development data being provided for the 2021 edition, challenge rules dictate the use for training purposes of \emph{only} ASVspoof 2019 LA training and development subsets which contain only clean data (no similar encoding and transmission). 
Use of the 2019 evaluation subset was strictly forbidden.
The challenge for the ASVspoof 2021 LA task is hence to design spoofing CMs which generalise well to unknown channel variation. 
While the use of external \emph{speech} data was not permitted, the challenge rules allow the use of external \emph{non-speech} resources such as noise samples, impulse responses, and audio compression software for training or data augmentation~\cite{delgado2021asvspoof}.





\subsection{Physical Access (PA)}
\label{sec:data_pa}


Post-challenge analysis of the ASVspoof 2019 PA results showed evidence of over-fitting to simulated attacks;
CM performance for attacks recorded in real physical spaces was generally found to be worse than for simulated attacks~\cite{Nautsch2021}. 
Even so, the use of simulation allows for the generation of plentiful data in an unlimited number of different, simulated physical spaces at little cost and hence remains an attractive alternative to recordings in real physical spaces.



The ASVspoof 2021 PA task was designed to encourage progress in generalisation, specifically CMs trained and developed using simulated attack data which reliably detect presentation attacks made in real physical spaces.  
The evaluation data hence comprises bona fide speech and replayed recordings both collected in a variety of \textit{real} physical spaces.
Bona fide data comprises 670 utterances from the VCTK corpus\cite{vctk}\footnote{This is the same set of VCTK utterances used to create the ASVspoof 2019 PA evaluation set~\cite[Fig.~2, \#15]{asvspoof2019database}.} 
which are presented to an ASV system using a high-quality loudspeaker with a reasonably flat frequency response~\cite{PreSonus}. 
Recordings are made in 162 ($=9 \times 3 \times 6$) acoustic environments comprising nine different rooms ${S}_{\text{asv}} \in \{\text{R1}, \text{R2}, \cdots, \text{R9}\}$, three different ASV microphones ${Q}_{\text{asv,m}} \in \{\text{M1}, \text{M2}, \text{M3}\}$, and six different talker-to-ASV distances (and angles) ${D}_{\text{s}}  \in \{\text{D1},\text{D2},\cdots, \text{D6}\}$ listed in Table~\ref{tab:PA_conditions}. The talker-to-ASV distances are illustrated in Fig.~\ref{fig:pa_d_s}.  


\input{table/PA_condition_table2}


Replays are made in the same set of rooms according
to different attacker factors including the room size ${S}_\text{a} \in \{\text{r1},\text{r2},\cdots, \text{r9}\}$\footnote{We use ${S}_\text{asv}$ and ${S}_\text{a}$ to denote the rooms in which utterances are presented to an ASV microphone and the rooms in which recording of the target speaker's speech are made by the attacker, respectively.}, the attacker microphone device ${Q}_{\text{a,m}} \in \{\text{m1}, \text{m2}, \text{m3}\}$, the attacker replay device ${Q}_{\text{a,s}} \in \{\text{s2},\text{s3},\text{s4}\}$\footnote{\text{s1} denotes the 
simulated bona fide talker.}, and the attacker-to-talker distance ${D}_{\text{a}} \in \{\text{c2},\text{c3},\text{c4}\}$. 
Attacker replay devices are situated at the same positions as bona fide talkers ${D}'_{\text{s}} \in \{\text{d1},\cdots,\text{d6}\}$\footnote{Different to the notations used in \cite{yamagishi21_asvspoof}, the attacker-to-ASV and the attacker-to-talker distances in this paper are denoted by d* and c*, respectively. c2, c3, and c4 are the same positions as d2, d3, and d4, respectively.}. 
Since the number of exhaustive attack factor combinations is large (1,458 $= 9\times 3 \times 3 \times 3 \times 6$), a non-exhaustive policy was adopted. 
First, the nine rooms are divided into three groups\footnote{$\{$R1/r1, R2/r2, R3/r3$\}$, $\{$R4/r4, R5/r5, R6/r6$\}$, $\{$R7/r7, R8/r8, R9/r9$\}$.} so that each group contains one large, one medium, and one small sized room. 
Replay attacks in each room $S_{\text{asv}}$ are presented to the ASV microphone using a single replay device and speech data recorded in three $S_\text{a}$ rooms in the same room group. 
Recordings are made using the full set of three attacker microphones ${Q}_{\text{a,m}}$, each set at one of the three attacker-to-talker distances ${D}_{\text{a}}$. 
Accordingly, replayed data presented in each room $S_{\text{asv}}$ cover nine different combinations of replay device, attacker room, and attacker microphone. 
The nine combinations are different for each room in which replays are made.

\begin{figure}[t!]
    \centering
    \includegraphics[trim=0 560 0 100, clip, width=1.0\columnwidth]{figs/fig_PA_d2.pdf}
    \caption{Illustration of talker-to-ASV distance ${D}_{\text{s}}\in\{\text{D1},\text{D2},\cdots, \text{D6}\}$. The full set of ASV microphones $\{\text{M1}, \text{M2}, \text{M3}\}$ are used in all positions. Attacker-to-ASV distance $\mathcal{D}_{\text{s}}'$ can be plotted similarly by replacing the talker with the attacker replay device.  Distances and angles depicted for D1 and D2 only; those for D3-6 are listed in Table~\ref{tab:PA_conditions}.}
    \label{fig:pa_d_s}
\end{figure}

Recordings and replays are made using VCTK source utterances at a sampling rate of 48~kHz. After collection, all bona fide and spoofed data are  downsampled to 16~kHz using a non-causal Hamming-windowed sinc filter truncated to a length of $2^{14}+1$ samples. 
This downsampling configuration is the same as that used in collecting the ASVspoof 2019 PA database.
Leading and trailing non-speech segments of greater than 0.3 seconds are truncated.
The set of trials are then divided into utterance-disjoint progress and evaluation subsets.
Both sets include data from the full set of speakers.
As a further test of generalisation, a subset of replay trials corresponding to the factors underlined in Table~\ref{tab:PA_conditions} were withheld from the progress subset and reserved exclusively for the evaluation subset. 
For other factors, 80\% of the bona fide and 70\% of the replay trials are assigned to the evaluation set, with the remaining data constituting the progress subset. 
The number of trials in each set are listed in Table~\ref{tab:All_trial_num}.



\subsection{DeepFake (DF)}

Evaluation data for the ASVspoof 2021 DF task is a collection of bona fide and spoofed speech utterances processed with different lossy codecs used typically for media storage. 
The data is encoded and then decoded to recover uncompressed audio. 
This process introduces distortions which depend on the codec and its configuration. 
Generic end-user applications are envisaged, namely we aim to promote solutions for the detection of deepfakes in compressed audio used in television and media hosted on news websites and social media platforms etc. 
In contrast to the LA and PA scenarios,
the DF task does not involve the use of an ASV system.
Accordingly, whereas the t-DCF~\cite{Kinnunen2020-tDCF-fundamentals} is used as the primary metric for LA and PA tasks,
that for the DF task is the CM equal error rate (EER).

While the evaluation database originates partly from the ASVspoof 2019 LA evaluation set,
there are two additional sets of source data.
They are
the 2018~\cite{LorenzoTrueba2018-VCC} and 2020~\cite{Yi2020-VCC-challenge} \emph{voice conversion challenge} (VCC) databases. 
Both 
are publicly available but neither has been used previously for ASVspoof challenges.
In contrast to LA and PA evaluation data, both of which are derived solely from the VCTK database~\cite{vctk}, the VCC 2018 and 2020 challenge data are derived from DAPS~\cite{Mysore2015-DAPS} and EMIME~\cite{Wester2010-EMIME} corpora, respectively.
We included all the audio data from the two VCC challenges, including source and target speaker training data and all data submitted by VCC participants. 
Combined, these two additional source datasets contain bona fide speech collected from a total of 26 additional speakers (12 in VCC 2018, 14 in VCC 2020) and a large number of spoofed utterances generated using VC attack algorithms not used in generating any of the past ASVspoof challenge databases. 
The aim was to address CM generalisation across different codecs as well as different source databases (domains) and spoofing attacks.

The full DF evaluation database is hence generated using in excess of 100 different spoofing attack algorithms generated by the large number teams or individuals who contributed to the ASVspoof 2019 database~\cite{asvspoof2019database}, or by VCC participants.  
It should still be noted that, since many of the VC approaches share similar voice coders (vocoders), we are most interested in analysing the dependence of CM performance upon broad vocoder categories (rather than individual VC systems). 
To this end, the set of vocoders are divided into five broad categories: \emph{traditional}, \emph{neural autoregressive}, \emph{neural non-autoregressive}, \emph{waveform concatenation} and \emph{unknown}. 
The unknown category covers missing or incomplete VC system descriptions.  
   
   
    
     
      
       
       
   
   


Audio data from the three sources (ASVspoof 2019, VCC 2018, VCC 2020) was processed with the set of codecs giving the nine evaluation conditions shown in Table~\ref{tab:DF_conditions}.
The `no codec' condition C1 corresponds to original audio samples. 
Conditions C2 and C3 both use an mp3 codec. 
C4 and C5 use an m4a codec with advanced audio coding,\footnote{\url{https://trac.ffmpeg.org/wiki/Encode/AAC} (referred \today).} whereas both C6 and C7 use an ogg Vorbis codec.\footnote{\url{https://xiph.org/vorbis/} (referred \today).} 
The differences between conditions with the same codec lie in the use of different, variable bit rate (VBR) configurations, one lower and one higher, as illustrated in the right-most column of   
Table~\ref{tab:DF_conditions}. 
Two additional conditions C8 and C9 involve the successive application of two different codecs, 
one with a lower VBR, the other with a higher VBR.
The goal for these two dual-codec conditions is to study whether spoofing artefacts persist in the case of distortion introduced through transcoding.
Such transcoding could take place, for instance, when an adversary acquires a sample of compressed bona fide speech for a given target speaker, e.g.\ a podcast posted online, uses it somehow to generate a manipulated or deepfake sample, 
and then uploads the result to a social media website which uses a different audio compression technique.
We used \texttt{ffmpeg}\footnote{\url{http://ffmpeg.org/}} and \texttt{sox}\footnote{\url{http://sox.sourceforge.net/}} toolkits in creating the DF evaluation data. 
The number of trials for the DF subset is shown in Table~\ref{tab:All_trial_num}.

\begin{table}[]
    \centering
    \caption{Summary of DF evaluation conditions. Each of these conditions also includes different vocoder types as sub-conditions. \underline{Underlined factors} only appear in evaluation subset but not in progress subset.}
    \begin{tabular}{|l|c|c|}
        \hline
        Cond. & Compression	& Bitrate    \\ 
        \hline\hline
        DF-C1 & -- & 256~kbps  \\ 
        DF-C2 & Low mp3 & $\sim$80-120~kbps  \\ 
        DF-C3 & High mp3 & $\sim$220-260~kbps  \\
        DF-C4 & Low m4a & $\sim$20-32~kbps \\ 
        DF-C5 & High m4a & $\sim$96-112~kbps \\ 
        \underline{DF-C6} & Low ogg & $\sim$80-96~kbps\\ 
        \underline{DF-C7} & High ogg & $\sim$256-320~kbps \\
        \underline{DF-C8} & mp3$\rightarrow $ m4a & $\sim$80-120~kbps, $\sim$96-112~kbps \\
       
        \underline{DF-C9} & ogg $\rightarrow$ m4a  &  $\sim$80-96~kbps, $\sim$96-112~kbps\\
       
        \hline
    \end{tabular}
    \label{tab:DF_conditions}
\end{table}




\begin{table*}[htbp]
  \footnotesize
  \renewcommand{\arraystretch}{1.1}
  \centering
  \caption{Empirical description of top-5 submissions for each task. Teams are presented in order according to results of pooled normalized min t-DCF. LFCC: Linear frequency cepstral coefficients \cite{lfcc_mfcc2011}. L-VQT: long-term variable Q transform \cite{l-vqt}. SEnet: Squeeze-and-excitation network \cite{senet}. SFR-CNN: sparse feature reactivation CNN block, from CondenseNetV2 \cite{condensenetv2}. }
  \begin{tabular}{|c|c|c|c|c|c|}
  \hline
    Task & Team ID & Data Augmentation & Acoustic Feature & Classifier & System Fusion \\ \hline
    \multirow{5}{4.25em}{\emph{LA}} & T23 & Trans codec & Raw waveform; Raw spec & LightCNN;ResNet;LSTM & Weighted score average \\ 
    & T35 & Additive noise & LFCC & ResNet & Score average \\ 
    & T19 & RIR;MUSAN & Mel spec & ResNet;SEnet & Weighted score average \\ 
    & T36 & Speed perturbation & Raw Waveform;Mel spec & TDNN;ResNet;MLP & Score average \\
    & T20 & RIR;MUSAN;Media codec & Linear Filter Bank & ResNet;MLP & Score average \\ \hline
    \multirow{5}{4.25em}{\emph{PA}} & T07 & Speed perturbation & Log spec & VAE;GMM & Score average \\
    & T16 & In-house replay & Linear spec;Raw spec;Phase spectrum & SEnet & Score average \\
    & T23 & RIR;MUSAN & Raw spec & ResNet & Weighted score average \\
    & T01 & - & linear spec;Mel spec & LightCNN & Score average \\
    & T04 & RIR;Speed perturbation & Mel spec; CQT spec & TDNN & Score average \\ \hline
    \multirow{5}{4.25em}{\emph{DF}} & T23 & Media codec & Raw Waveform; Raw spec & LightCNN;ResNet;LSTM & Weighted score average \\
    & T20 & RIR;MUSAN;Media codec & Linear filter bank & ResNet;MLP& Score average \\
    & T08 & Trans codec;Media codec & CQT spec & LightCNN & Score average \\
    & T06 & Trans codec;Media codec & Linear spec; Raw spec & SEnet;TDNN;GMM & Weighted score average \\
    & T22 & Media codec & L-VQT spec & SFR-CNN & - \\ \hline
  \end{tabular}
\label{tab:results_briefing}
\end{table*}






\section{ASVspoof 2021 Challenge Results}
\label{sec:results}

A summary of results for all three tasks is shown in Fig.~\ref{fig:all_results}. 
Baseline systems (B0$n$) and submissions ( T$nn$)\footnote{Submissions with no identifier correspond to teams that failed to submit a valid system description.} are ranked according their performance on the evaluation subset and primary metric.
For all three tasks, CM performance is illustrated in terms of the EER in red bars (bottom axes), which is the primary metric only for the new DF task.
For LA and PA tasks, the primary metric is the min t-DCF \cite{Kinnunen2020-tDCF-fundamentals}. 
These results are shown in blue bars (top axes).
The \emph{ASV floor}~\cite[Eq. (11) and Section V]{Kinnunen2020-tDCF-fundamentals}, which corresponds to the t-DCF in the case of a perfect CM but imperfect ASV system, is illustrated with a vertical blue line.
The gap between blue bars and blue lines give an indication of CM performance.
Table~\ref{tab:results_briefing} shows a description of the best performing submissions for each task. 




\subsection{Logical Access (LA)}

\subsubsection{Full challenge results}
Results for the LA task are shown in Fig.~\ref{fig:all_results}a. 
Many CMs outperform the best baseline B03, some by a large margin.
The top systems show t-DCFs close to the ASV floor,
indicating CMs with low error rates.
For the majority of systems the gap in performance for progress and evaluation subsets is modest.
The over-fitting to the progress subset seen for other systems is likely caused by differences between the known (C1-C4) and unknown (C5-C7) conditions, the latter of which appear only in the evaluation subset.
 

\subsubsection{Influential data factors}

The left-most pane of Fig.~\ref{fig:LA_Channel_Transmission_decomposed} shows the distribution in t-DCF for the top-10 systems for each of the seven conditions shown in Table~\ref{tab:LA_conditions}.
For reference, the distribution for pooled conditions is shown in the right-most pane.
The distribution in t-DCF for wideband conditions (no codec, G.722, OPUS) is lower than for narrowband conditions (a-law, PSTN, u-law and GSM), indicating the general importance of information at higher frequencies to CM performance.
Among the narrowband conditions, lower bit rates and uncontrolled transmission (GSM and unk.\ + $\mu$-law, respectively -- see Table~\ref{tab:LA_conditions}) lead to worse performance.





The middle pane of Fig.~\ref{fig:LA_Channel_Transmission_decomposed} shows the distribution in t-DCF for the three transmission routes detailed in Section~\ref{sec:LAdb}, namely routes across a LAN and from France to Italy (FR-IT) and from France to Singapore (FR-SG). 
The similarity in t-DCF for all three shows that transmission routes have little impact upon CM performance; they are not dissimilar to the distribution for pooled results.
This would suggest that future challenge editions could use simpler LAN routes only.

Fig.~\ref{fig:la_subcondition_analysis} depicts a condition/attack analysis, with attack algorithms on the vertical axis and evaluation conditions on the horizontal axis. 
The color of each circle encodes the median min t-DCF for the top-10 systems, while the radius represents the inter-quartile range. 
For some attacks and some conditions the min t-DCF is consistently high.  
For attack A18, and to a lesser extent also A17, there is greater variation across conditions, with the median t-DCF ranging from approximately 0.4 for condition C1 (no encoding) to over 0.8 for C3 (unk.\ + $\mu$-law) and approximately 0.7 for C6 (GSM).
These observations show that A18, an attack which is already difficult to detect even in the absence of encoding, becomes disproportionately more difficult to detect after low bandwidth encoding.



\begin{figure}
    \centering
    \includegraphics[trim=0 15 0 0, width=0.9\columnwidth]{figs/paper_fig_analysis_tdcf_LA.pdf}
    \caption{Statistics on tDCF values of top-10 LA submissions in sub-conditions. Dot size is decided by inter-quartile range of tDCF distribution, and color is decided by its median.}
    \label{fig:la_subcondition_analysis}
\end{figure}

\subsubsection{Top-performing systems}
A summary of the top-5
systems is presented
to the top of Table~\ref{tab:results_briefing}. 
All use some form of data augmentation, though no common form can be identified.
Most are ensemble systems,
and most operate upon short-term spectral features or raw waveforms.
Most use a ResNet classier or variant, with other types of convolutional networks also being popular. 
Fusion strategies include weighted averaging, using either uniformly or empirically set weights. 








\subsection{Physical Access (PA)}

\subsubsection{Full challenge results}
A summary of results for the PA task is shown in Fig.~\ref{fig:all_results}b.
While almost half of the systems outperform the best B01 baseline, the performance of all systems is substantially higher than the ASV floor of 0.12.
This would suggest that the PA task is more challenging than the LA task.
The difficulty is likely caused by the difference between simulated replay attacks (data used for training and development)~\cite[Section 2.2.3]{asvspoof2019database} and replay attacks recorded in real physical spaces (evaluation data), e.g., differences in room acoustics and noise conditions.
We also observe only modest gaps between performance for progress and evaluation subsets indicating that, while error rates remain high, system performance is stable across different evaluation conditions (rooms and devices, see Section~\ref{sec:data_pa}).


\begin{figure*}[t]
    \centering
    \subfloat[LA]{
    \includegraphics[trim=0 15 0 10, width=0.3\textwidth]{figs/paper_fig_pre_LA_eers.pdf}
    }
    \subfloat[PA]{
    \includegraphics[trim=0 15 0 10, width=0.3\textwidth]{figs/paper_fig_pre_PA_eers.pdf}
    }
    \subfloat[DF]{
    \includegraphics[trim=0 15 0 10, width=0.3\textwidth]{figs/paper_fig_pre_DF_eers.pdf}
    }
     \caption{ASVspoof 2021 results for LA, PA, DF tracks. Results shown in terms of \textcolor{colortDCF}{pooled normalised minimum t-DCF} and \textcolor{colorEER}{pooled EER (\%)}. \textcolor{colortDCF}{Blue line} marks ASV floor. Only teams that submitted system description are marked with ID. }
     \vspace{-2mm}
     \label{fig:all_results}
\end{figure*}


\begin{figure}[t!]
    \centering
    \includegraphics[trim=0 10 0 0, width=0.85\columnwidth]{figs/paper_fig_cross_factor_eer_DF.pdf}
    \caption{Break-down of the EER in the VCC 2018/2020 subset of the DF track, according to the codecs and the type of vocoder used by the VC system.
    }
    \label{fig:cross-factor-EER-DF}
\end{figure}


\begin{figure*}
 \centering
 \subfloat[LA]{
 \includegraphics[trim=0 15 0 5,  width=.9\textwidth]{figs/paper_fig_pre_tDCF_box_LA_trans_codec.pdf}

 \label{fig:LA_Channel_Transmission_decomposed}
 }
 \qquad





\subfloat[PA]{
\includegraphics[trim=0 10 0 0, width=.9\textwidth]{figs/paper_fig_pre_tDCF_box_PA_att_env.pdf}
}
\label{fig:PA_decomposed}%
\qquad
\subfloat[DF]{
 \includegraphics[trim=0 10 0 0, width=.9\textwidth]{figs/paper_fig_pre_eer_box_DF_codec_source.pdf}

 \label{fig:DF_subcondition_result_decomposition}}
 \vspace{-1mm}
 \caption{Boxplots of evaluation set min t-DCF or EERs of top-10 submissions decomposed over different factors. Markers are top-1 submission (\textcolor{teal}{$\diamond$}), B01 (\textcolor{blue}{o}), B02 (\textcolor{red}{$\triangleleft$}), B03 (\textcolor{orange}{$\medwhitestar$}), and B04 (\textcolor{purple}{$\times$}).}
    
     
    
    
    
    
\end{figure*}


\subsubsection{Influential data factors}

Fig.~\ref{fig:DF_subcondition_result_decomposition} shows min t-DCF results pooled over the top-10 and baseline systems for a selection of attacker and environment factors. 
The rooms used for replay acquisition and voice presentation are not included in the analysis because we observe no substantial correlation between the room size and the min t-DCF values. 
However, we observe expected impacts related to the quality of the attacker microphone. 
For example, higher-quality attacker microphones m1 and m2 lead to higher min t-DCFs than the lower-quality microphone m3. 
This is to be expected because spoofed data acquired using a higher-quality microphone $Q_{\text{a, m}}$ introduces less distortion and are hence more challenging to detect. 

The min t-DCF is also higher for a better quality attacker replay device $Q_{\text{a, s}}=\text{s3}$ or $\text{s4}$.
Similarly, shorter attacker-to-talker distances $D_\text{a}$ also lead to worse performance.
At the closest position $D_\text{a}=\text{c4}$, min t-DCF values are the highest.


The attacker-to-ASV distance $D'_{\text{s}}$ is observed to be inversely correlated with the min t-DCT. 
However, as expected, if the talker is closer to the ASV microphone (i.e., moving from $D_\text{s} = \text{D1}$ to D4), then the min t-DCF decreases.


\subsubsection{Top-performing systems}

A summary of the top-5 PA systems is presented in the middle of Table~\ref{tab:results_briefing}. 
Similar to the LA task, most use data augmentation, once again with various different approaches, and all are ensemble systems.
Linear, raw or mel scaled spectrogram features are most common,
whereas classifiers are diverse.
The top-1 system uses both frame-level and temporal-level features
and the parallel combination of variational autoencoder (VAE) and Gaussian mixture model (GMM) classifiers. 
Like for the LA task, all the top-5 systems use score averaging. 







\subsection{DeepFake (DF)}

\subsubsection{Full challenge results}
A summary of results for the DF task is shown in Fig.~\ref{fig:all_results}c.
The main observation is the striking difference in results for progress and evaluation subsets. 
While 23 (out of 33) systems have EERs of less than 10\% for the progress subset, and while the best performing system even has an EER of less than 1\%, \emph{all} have EERs exceeding 15\% for the evaluation set. 
This is a strong indication of over-fitting. 
Despite high EERs, however, 
18 systems outperformed the best B04 baseline system.



\subsubsection{Influential data factors}

The left-most pane of Fig.~\ref{fig:DF_subcondition_result_decomposition} shows the distribution in EER performance for each compression method, with the distribution for pooled conditions being shown in the right-most pane.
For mp3 compression, a higher VBR (C3) gives a lower median EER than for a lower VBR (C2). 
For m4a and ogg codecs, there is no similar effect between the low and high bit rates for either codec. 
EERs for the ogg codec are lower than those for 
both mp3 and m4a codecs, no matter what the VBR.  
Interestingly, EERs for the two double compression methods are only modestly higher than those for the single ogg compression, but below those for either mp3 or m4a compression, indicating some resilience to transcoding.

The middle pane of Fig.~\ref{fig:DF_subcondition_result_decomposition} shows the distribution in EER for each source database.
EERs are substantially higher for VCC datasets than for the ASVspoof 2019 LA database. 
This is likely due to the use of the latter in the progress set only and is hence a consequence over-fitting, or lack of generalisation to mis-matched VCC source data.
Further analysis of this observation is provided in Section~\ref{subsec:DF-high-EER}.

Finally, Fig.~\ref{fig:cross-factor-EER-DF} shows a breakdown in the EER for pooled VCC 2018 and 2020 source data according to the type of vocoder and codec. 
The results indicate that,
for any given codec, both types of neural vocoders yield higher EERs than more traditional vocoders or waveform concatenation.




\subsubsection{Top-performing systems}
A summary of the top-5 DF submissions is illustrated to the bottom of Table~\ref{tab:results_briefing}.
With one exception, all submissions are ensemble systems.
All use data augmentation.
While there are different approaches, all use some form of media codec augmentation.\footnote{We use \emph{media codec} to refer to generic codecs implemented in various software packages. The challenge participants used mp3, m4a, ogg, opus, mp4, aac, alac, vorbis, wma and sbc, along with other open-source toolkits for media codec augmentation.}
Acoustic features
are diverse. 
A range of different classifiers are also used, including mostly convolutional neural networks,
but with simpler 
MLP and GMM classifiers also featuring among the top-5 submissions.
Once again, 
all top-5 systems use
score averaging for fusion.






\section{Hidden Subsets and Additional Analyses}\label{sec:Hidden-Tracks}


We now turn attention to results for the hidden subsets first mentioned in Section~\ref{sec:data_pa} before presenting other analyses.
The purpose of hidden subsets is to assess the dependence of CM performance upon certain data characteristics that were withheld from challenge participants.
They were unaware of their inclusion and treated hidden subsets in the same manner as evaluation data.
Hidden subsets were not used in deriving challenge results or rankings.



\subsection{The role of non-speech}
The databases for all three tasks contain a hidden subset of utterances from which non-speech intervals are automatically removed.
The motivation is to assess performance when spoofed \emph{speech} detectors are constrained to operate only upon \emph{speech} segments; 
recent reports (e.g.~\cite{Chettri2020-dataset-artefacts, zhang21da_interspeech, muller21_asvspoof}) suggest that some CMs might rely upon information within non-speech segments. 
While non-speech information might serve as legitimate cues for spoofing detection, adversaries could easily remove non-speech segments to gain an advantage.
Alternatively, cues within non-speech intervals might correspond to unintended database design artifacts.
It is hence of interest to determine any differences in CM performance when they are restricted to using information from speech segments only. 





\begin{figure*}
\centering
\subfloat[LA]{
\includegraphics[trim=0 5 0 0, width=0.3\textwidth]{figs/paper_fig_pre_hidden_track_LA_decomposed_tDCF_1.pdf}
}
\subfloat[PA]{
\includegraphics[trim=0 5 0 0,width=0.3\textwidth]{figs/paper_fig_pre_hidden_track_PA_decomposed_tDCF.pdf}
}
\subfloat[DF]{
\includegraphics[trim=0 5 0 0,width=0.3\textwidth]{figs/paper_fig_pre_hidden_track_DF_decomposed_1.pdf}
}
\caption{Comparison of top-10 CMs performance on evaluation subset (with silence) and hidden subset (w/o silence). For LA and DF, min t-DCFs over VC- and TTS-based spoofing attacks are plotted separately. Markers are top-1 system (\textcolor{teal}{$\diamond$}), B01 (\textcolor{blue}{o}), B02 (\textcolor{red}{$\triangleleft$}), B03 (\textcolor{orange}{$\medwhitestar$}), and B04 (\textcolor{purple}{$\times$}).}
\label{fig:hidden_track_decomposed}%
\end{figure*}

Bona fide and spoofed utterances are selected at random from evaluation subsets giving the number of utterances shown in Table~\ref{tab:All_trial_num} for LA, PA and DF tasks.
An energy-based voice activity detector with the recommended threshold settings~\cite[Section 5.1]{kinnunen2010overview} is then used to remove segments labeled as non-speech in addition to any segments of less than 50ms duration.
For the LA task, the hidden subset contains C1-C7 utterances that are all treated by a VAD to remove non-speech segments.
So as to focus upon speech/non-speech effects only, rather than variations in room characteristics and replay configurations etc., the PA hidden subset contains data from the D4 talker-to-ASV distance only (90$^{\circ}$, 0.5~m from the ASV microphone as per Table~\ref{tab:DF_conditions}).
The hidden subset for the DF task comprises utterances for all conditions C1-9, but ASVspoof 2019 LA source data only.

Contrasting min t-DCF/EER results for evaluation subset (w/non-speech) and hidden subset (w/o non-speech segments) are plotted in Fig.~\ref{fig:hidden_track_decomposed} for the LA task (a), the PA task (b) and DF task (c).
Boxplots for the top-10 LA systems are shown
separately for VC-based (left), TTS-based (middle) and pooled (right) attacks.\footnote{Following the descriptions in \cite{asvspoof2019database}, A17, A18, and A19 are grouped as VC-based attacks.  All others are grouped as TTS-based attacks.} 
While the contrast is greatest for TTS attacks, min t-DCFs calculated from utterances containing non-speech segments (w/non-speech) are substantially lower than those calculated from utterances without non-speech segments (w/o).
LA source data contain long non-speech segments at the start and/or end of each utterance.  
While VC spoofing attack algorithms reproduce these characteristics, TTS algorithms produce utterances without such long non-speech segments. 
Their length may hence serve as a cue to distinguish between bona fide and spoofed utterances, accounting for why their absence leads to degraded performance.
The reliance upon such cues may not lead to reliable detection in the wild.
Observations for the DF task shown in Fig.~\ref{fig:hidden_track_decomposed}c are much the same as for the LA task.
Without the use of non-speech segments, performance is substantially degraded, more so for TTS attacks than VC attacks.

Boxplots for the top-10 PA systems are shown in Fig.~\ref{fig:hidden_track_decomposed}b for replay attacks recorded in real physical spaces, with (evaluation subset) and without (hidden subset) non-speech segments (two left-most boxes).
The length of non-speech intervals is not expected to have an impact since  recording and replaying do not alter utterance duration.
Nonetheless, without non-speech segments, performance is notably worse.  



Any asymmetry in the \emph{length} of non-speech segments in bona fide and spoofed speech should not serve as a detection cue.  
The \emph{contents} of non-speech segments, however, is a different issue.
Non-speech segments, while being information that is easily removed by the adversary, are an innate characteristics of recorded speech which might still provide legitimate spoofing detection cues. 

 





\subsection{Real v/s simulated replay attack}
The second hidden subset in the PA database contains \emph{simulated} replayed data. 
The motivation is to determine whether reliable estimates of performance can also be derived using evaluation data that is also simulated instead of being recorded in real physical spaces.
Simulated bonafide and spoofed data are created using the same tool as that used in generating the ASVspoof 2019 PA database~\cite[Section 4.3]{asvspoof2019database}. 
Room impulse responses and background noises are extracted from swept-sine signals recorded in the same rooms and with the same set of microphones. For the same reasons outlined above,
data is simulated for the D4 talker-to-ASV distance only.

Results shown in the right-most pane of Fig.~\ref{fig:hidden_track_decomposed}b depict the distribution in min t-DCFs for the top-10 PA systems for the hidden, simulated subset which contains non-speech intervals. 
The median min t-DCF for the simulated subset is much higher than that for the evaluation subset. 
Further analysis shows that four out of the top-10 PA systems' min t-DCFs for the simulated subset are higher than 0.99. 
These results discourage the use of simulated evaluation data to estimate CM performance in real physical environments.


\subsection{Performance gap for DF progress and evaluation subsets}
\label{subsec:DF-high-EER}

Fig.~\ref{fig:all_results} shows a substantial gap in performance for progress and evaluation subsets for the DF task. 
As evident from results shown in Fig.~\ref{fig:DF_subcondition_result_decomposition}, the gap relates in part to the inclusion of two previously unexposed source corpora. 
To explore the cause for this gap, we examined the class- and data source-conditional CM score distributions for the top-5 submissions.
Results are shown in Fig.~\ref{fig:DF-score-plot} along with distributions for the best B04 baseline. 
While similar findings were observed for compressed data conditions, the plots shown correspond to non-compressed data (condition C1 in Table~\ref{tab:DF_conditions}).

While, for any given system, score distributions for the spoofed class (bottom row) are reasonably well aligned, there are significant differences in the distributions for the bona fide class (top row).
Scores for the two VCC source corpora are consistently lower than those for the ASVspoof 2019 LA corpora.  
This shift leads to greater overlap in the distributions for bona fide and spoofed classes.
The use of only ASVspoof 2019 LA data for training leads to over-fitting and models which generalise poorly to VCC data in the evaluation subset, greater confusion between bona fide and spoofed trials, and hence degraded detection performance.

As a further numerical quantification, Table \ref{tab:DF-trials-from-different-source-subsets} displays the EERs with either bonafide or spoof trials constrained to specific subsets. 
We find that whenever either one of the two VCC subsets is included on bonafide side (the first three rows), the EERs are very high ($\sim 15\% \dots 34\%$). 
In stark contrast, however, when both VCC'18 and VCC'20 trials are \emph{excluded} from bonafide side (the last three rows), the EERs of the six CMs are substantially lower ($< 1\%$ for T23) -- regardless of whether the \emph{spoof} trials include VCC'18 or VCC'20 data. 

These findings, and given the performance mismatch between evaluation and hidden track which excludes silences, motivated a deeper exploration of the possible role of silence in the VCC bonafide score distribution shift. To this end, for each audio segment from C1 (pooled from the progress and evaluation subsets), the \emph{proportion of detected nonspeech}
was calculated and used as recognition score. EER computed from these scores reflects the degree of similarity of the corresponding positive and negative trial sets in terms of nonspeech proportion. These EERs, for different combination of positive and negative trials, are shown in Table~\ref{tab:DF-silence-proportion-as-score-from-different-source-subsets}.
The trends are similar to those shown in Table~\ref{tab:DF-trials-from-different-source-subsets}; EERs are substantially lower when ASVspoof 2019 bonafide data is used as positive set, particularly when the negative set contains VCC 2018 and VCC 2020 spoof data. The last two rows show an alternative experiment were both positive and negative sets consists of bona fide data only. EER values between 11\% and 14\% 10\% indicate that the nonspeech proportion distribution for ASVspoof and VCC bonafide data are reasonably well separated. It would not be a surprise that submitted CMs learned the bonafide class based (at least, partially) on the implicit different silence distribution of the ASVspoof bonafide data (the only data source included in the training set). This in turn makes them fail when an unseen bonafide dataset does not follow the same silence distribution.


\begin{figure*}
    \centering
   
    \includegraphics[scale=0.33]{figs/DF_score_plot_v0.2.pdf}
    \caption{Score distributions of bonafide (top) and spoof (bottom) for  top-5 DF subsystem and baseline B04, broken down according to three data sources used.}
    \label{fig:DF-score-plot}
\end{figure*}

\begin{table}[!t]
    \centering
    \caption{EERs (\%) for the DF track with all compressors pooled, but bonafide/spoof trials selected from different combinations of source datasets.}
    \setlength{\tabcolsep}{5pt}
    \begin{tabular}{|cc||cccccc|}
         \hline
         Bonafide & Spoof   & & & & & &\\
         trials   & trials & T23   & T20   & T08   & T06   & T22   & B04\\
         \hline\hline
         All    & All      & 15.63 & 16.04	& 18.29	& 19.01	& 19.22 & 22.38\\
         VCC18  & All      & 23.39 & 27.64	& 31.03 & 30.87 & 29.81 & 34.44\\
         VCC20  & All      & 16.98 & 19.77 & 21.55 & 22.88 & 25.00 & 31.48\\
         ASV19  & ASV19    & 0.33 &	 2.41 &	3.75 & 8.82 & 2.76 & 6.21\\
         ASV19  & VCC18+20 & 0.60 & 1.98 &	2.78    & 5.83  & 1.93  & 3.09\\
         ASV19  & All      & 0.55  & 2.05     & 2.92  & 6.32  & 2.11  & 3.56\\
         \hline
    \end{tabular}
    \label{tab:DF-trials-from-different-source-subsets}
\end{table}

\begin{table}[!t]
    \centering
    \caption{Similar to Table~\ref{tab:DF-trials-from-different-source-subsets}, but using the proportion of detected nonspeech frames by VAD as spoofing detection score. Only trials of C1 were considered.}
    \setlength{\tabcolsep}{5pt}
    \begin{tabular}{|ll||c|}
         \hline
         Positive set & Negative set & EER\\
         \hline\hline
         All bona    & All spoof      & 37.59 \\
         VCC18 bona  & All spoof     & 44.17 \\
         VCC20 bona  & All spoof     & 51.14 \\
         ASV19 bona  & ASV19 spoof    & 24.98 \\
         ASV19 bona  & VCC18+20 spoof & 9.19 \\
         ASV19 bona  & All spoof      & 10.36 \\
         \hline
         ASV19 bona  & VCC18 bona   & 11.90  \\
         ASV19 bona  & VCC20 bona   & 14.00  \\
         \hline
    \end{tabular}
    \label{tab:DF-silence-proportion-as-score-from-different-source-subsets}
\end{table}






\section{Post-Challenge Studies and Related Work}
Each edition of ASVspoof addresses new challenges but also leads to the identification of new challenges and concerns which have stimulated a number of post-challenge studies. The 2021 edition is no exception.
A selection of such studies is discussed below, with a summary of related results shown in Table~\ref{tab:post_challenge_results}.\footnote{The selection is based on searches of ``ASVspoof 2021" and ``Deepfake" 
in the ISCA archive and IEEExplore.} 
They pertain to LA and DF tasks and (1) the shift from the use of acoustic features toward \textbf{end-to-end} (E2E) architectures with raw waveform inputs and (2) the popularity of \textbf{data augmentation} (DA). 

Originating from~\cite{rawnet2_spoofing2021} and based on RawNet~\cite{rawnet2} and graph attention networks, AASIST~\cite{aasist2022} is representative of progress in the use of E2E architectures.
The RawNet in AASIST has further been replaced by various \emph{self-supervised learning} (SSL) front-ends, including wav2vec \cite{wav2vec_antispoofing2022, ssl_frontends_asvspoof2021, wav2vec_cnn_encoder_add2022}.
While the use of SSL brings substantial improvements in performance, the use of models pre-trained on external data is not compliant with the ASVspoof training protocol. 
A study of different SSL frontends and related techniques can be found in~\cite{ssl_frontends_asvspoof2021}. 
While conventional acoustic features remain popular~\cite{dataaug_study_asvspoof2021}, E2E architectures and SSL-based frontends have led to substantial improvements in performance for the LA task, even for single systems (not ensembles), with best t-DCF being 0.2066 (best EER 0.28\%) for the ASVspoof 2021 LA database~\cite{wav2vec_antispoofing2022}. Such result comes from the use of SSL with data augmentation. Post-challenge studies that employ ASVspoof 2021 PA data and work on replay attacks are rarely seen.


An investigation of DA for the E2E solution which operates directly upon raw waveform inputs, named \emph{RawBoost}, was reported in~\cite{rawboost2022}. The approach is based upon the combination of linear and non-linear filtering in addition to signal-dependent and signal-independent additive noise. 
Investigation on DA techniques applied on anti-spoofing can be found in~\cite{dataaug_study_asvspoof2021,donas22,fathan22}. In these studies a number of approaches to DA have been used. Compression, codec and channel effect, bandwidth difference augmentation, as well as time- frequency-masking~\cite{specaugment}.
All these studies confirmed that the robustness of spoofed audio detection is substantially improved when DA techniques are employed.  



Finally, other anti-spoofing / deepfake detection challenges and databases have emerged since 2021. The first \emph{audio deep synthesis detection} (ADD) challenge~\cite{add2022} was launched to evaluate the threat of low-quality and partially-spoofed audio~\cite{partialspoof} in order to address real-time and more challenging scenarios. The summary of the results indicates the fragility of synthesis detection model against lower quality and unseen fake audio, especially when partially-spoofed audio is presented.
The spoofing-aware speaker verification (SASV) challenge~\cite{Jung2022SASV}, which exploits the same ASVspoof 2019 LA source database as the ASVspoof 2021 LA evaluation, provides protocols for the design of not just CM solutions, as in the case of ASVspoof, but also ASV systems and alternative architectures for their combination.
Three different metrics support the assessment of spoofing robustness, automatic speaker verification and combined, spoofing aware speaker verification. 
A summary of SASV results~\cite{Jung2022SASV} anticipates the significance of improving the reliability of both ASV and CM sub-systems, including the joint optimization schemes. 

\input{table/PostChallenge}

\iffalse
\section{Lessons learned}
\textcolor{red}{Broader outlook: what did we learn? This can be be discussed from two major perspectives: attackers (data-quality, attack variations) and defender (spoofing countermeasures).}

Lessons:
\begin{itemize}
    \item channel variation not the problem we thought it was / little difference for results for synthesized/real channel telephony 
    \item New/previously not-used data (vcc18/20) handled poorly in the challenge, mostly attributed to problems with handling new bonafide data (rather than new spoofing attacks) --- overfit to bonafide of '19 / too few training speakers $\rightarrow$ larger speaker coverage needed in future (this could go under 'future directions')
\end{itemize}
\fi

\section{Limitations and future directions}

While we are hopeful that ASVspoof has made substantial contributions to open data and reproducible research in anti-spoofing, 
like any evaluation, it
is not without limitations. 
Some of the most relevant issues that are fundamental to the future ASVspoof road-map are discussed in the following.

\textbf{Realistic audio data} -- 
ASVspoof 2021 made important steps towards evaluation with data that is more representative of realistic conditions.  Other steps remain to be taken, especially the consideration of background noise for the LA task, the collection of recordings from human talkers as opposed to loudspeakers for the PA task, and the use of real social media for the DF task. 

\textbf{Role of non-speech} -- The results for hidden data sets confirm that non-speech intervals can play a role in the detection of spoofed speech, particularly for LA and DF tasks. 
While non-speech intervals are
an innate characteristic of natural speech, and while their contents can help to distinguish between bona fide and spoofed speech, their length is a database characteristic that should not serve as a cue for detection. 
It is of interest to investigate the generation or conversion of non-speech regions in the case of TTS and VC attacks as well as the difference between cues used for detection in non-speech and speech intervals.

\textbf{Training data policy} -- 
The use of a strict training protocol ensures the meaningful comparability of results generated by different teams and is fundamental to most benchmarking evaluations. 
Even so, post-evaluation results show the benefits of a relaxed training policy which allows the use of larger, more complex models trained using external data, e.g. via semi-supervised learning.  
To allow for their further exploration in a benchmarking evaluation setting, future editions of ASVspoof could hence allow a dual training policy, one fixed, the other relaxed.

\textbf{Diversity in attacks} -- 
The spoofing attacks in the 2021 LA and DF databases were generated with TTS and VC algorithms that were at the state of the art prior to 2020.
TTS and VC technology has advanced rapidly in recent years, especially 
in terms of requiring far less target speaker training data~\cite{cooper2020zero, huang2022meta, pmlr-v162-casanova22a}.
The next ASVspoof edition calls for a renewed spoofed data collection effort and the exploration of vulnerabilities to the latest attack techniques.

\textbf{Diversity in audio data} -- 
ASVspoof data are exclusively English-language, read-speech sourced from the VCTK corpus~\cite{vctk}. 
The inclusion of two unexposed datasets in the DF database exposed 
a lack of CM generalization. 
The cause is due mostly to differences in the characteristics of \emph{bona fide} speech and score distributions. 
This may suggest over-fitting to speakers in the training datasets.
In the future, ASVspoof should 
increase diversity not only in spoofing attacks, but also in bona fide source data, including collection in different recording environments, languages and speaking styles, among other factors.

\textbf{Extensible evaluation} -- 
The two ASV-related tasks, LA and PA, used a fixed ASV system, with participants 
developing only the spoofing CM. 
The literature shows an increasing interest in the joint optimisation of both CMs and ASV systems and alternative combination architectures. 
The spoofing-aware speaker verification (SASV) challenge~\cite{Jung2022SASV} was launched to support research in this direction. 
SASV challenge results~\cite{Jung2022SASV} have shown the need for larger labeled datasets for the learning of 
speaker embedding extractors and end-to-end solutions. 
ASVspoof and SASV will likely merge for a common evaluation in the future.
This will require not only a huge data collection effort but further thought to evaluation metrics; ASVspoof uses an ASV-constrained t-DCF, whereas the SASV challenge uses a set of three different EER-based metrics.

\textbf{Integration of multiple scenarios} --
There is no reason why replay attacks cannot be made in LA scenarios (variable microphone, telephony applications), nor why TTS and VC attacks cannot be replayed in PA scenarios (fixed microphone, access control applications).
Their exploration will increase the complexity of both protocol design and data collection, but remains an important future direction for ASVspoof.

\textbf{Partially spoofed audio} --
ASVspoof has considered only the scenario in which utterances are either bona fide or spoofed in \emph{full}.
\emph{Partial} spoofs, where only a potentially short interval or perhaps a single word is substituted or manipulated, are known to pose a greater detection challenge~\cite{partialspoof}.
The partially spoofed scenario is perhaps most relevant to the DF task in which there is no ASV system.
This is because partially-spoofed utterances of the shortest duration may be unlikely to provoke ASV false alarms, but can still fundamentally change the meaning of a given phrase\footnote{An example might be the manipulation of a phrase within a video posted to social media from \emph{`I won the election'} to \emph{`I lost the election'}.}.
Nonetheless, utterances in which a greater proportion of the utterance is spoofed still stand to fool the ASV system used in LA and PA tasks while also being more difficult to detect with a CM.
Depending of the application, given a partially spoofed utterance, it might then be of interest to estimate bona fide and spoofed labels at the segment level, e.g.\ to \emph{explain} the classifier decision.


\textbf{Adversarial attacks} -- 
Many examples in the literature show the threat to reliable ASV stemming from adversarial attacks~\cite{villalba20_interspeech,Joshi2021}.
These target the ASV system, but rarely the CM. 
With estimates of security being only as strong as the adversary model, future editions of ASVspoof should consider adversarial attacks that target \emph{both} the ASV \emph{and} the CM.  


\iffalse
Limitations:
\begin{itemize}
    \item PA bonafide segments are still `simulated' with flat frequence response loudspeaker (not real). Need to argue about trade-off in data collection vs. reasonable approximation of bonafide data. This challenge fixed the ASV'17 mess (real replay but no meaningful control) and advanced from ASV'19 (from fully-simulated to nearly-real)
    \item Silence
    \item The amount of training and dev data in general. It is reflected not only from the main challenge, but also from post-challenge works and my personal research that the amount of data is relatively small, especially if we are allowed to include in-the-wild data. This is also part of the problems to be solved in ASVSpoof5.
    \item More real-time data. This echoes some of the SASV people’s trials on creating a “spoofed” edition of voxceleb. Participants like to see purely real data as evaluation. Crowd-sourcing might be a good choice.
    \item Language independency. So far all data is collected in English. We may need more language variations.
    \item The amount of training data is small to try recent deep models.
    \item The bonafide and spoof data are collected from the same domain(s) but they could originate from different domains.
\end{itemize}


Future:
\begin{itemize}
    \item adversarial attacks against the CM in addition to ASV
    \item shorter test utterance duration, less data for TTS and VC training/adapation
    \item should we relax the training data policy, given SSL results etc. - how to incorporate in a benchmark
    \item An open-set track (especially for DF), where challengers can train their CM system using whatever data they would like to have. For instance, this recent work addresses a bit about the relationship between system reliability in general and CM detection efficiency, so the bonafide addition can be useful. 
    \item Integration of multiple scenarios (can start with LA and DF) into a more unified and challenging scenario. Joint optimization has been an interesting topic in the field of ASVspoof, but most of the works we have so far are approaching it from solutions’ perspective, instead of data and scenarios.
    \item Include partial spoofing and explainability (perhaps most relevant for the DF scenarios).
    \item Spoofed data are so far generated with clean training data and source data. Deepfake generation with publicly available (e.g., YouTube) audio-data could be investigated.
    \item To tackle the limited trainining data issue due to the nature of the problem, encourage exploring self-supervised learning frameworks with other publicly available audio-data.
\end{itemize}
\fi

\section{Conclusions}
\label{Section:Conclusion}

The ASVspoof 2021 challenge was designed to foster progress in reliable automatic speaker verification, spoofing and deepfake detection in more realistic and practical scenarios. 
The challenge attracted submissions from 37 teams across the three tasks, each of whom submitted valid challenge scores and system descriptions. 
This paper provides an overview of the challenge datasets and an analysis of results. 
We also report an analysis of various data-related issues, in addition to new results for hidden data subsets.
They reveal some challenge limitations, prompting ideas for future challenge editions.

Results show that the transmission of speech data across real telephony systems in a logical access task causes only modest degradation to spoofing detection reliability and that estimates of performance from transmission across a local area network are as reliable as estimates from transmission across more geographically distant endpoints.
Impacts on performance caused by compression effects in a deepfake task are also modest, but results show a lack of genearlisation to different source data.
Data augmentation is common to all top-performing systems for both logical access and deepfake tasks. 

The physical access task appears to be the most challenging of the three, likely due to the mismatch between training and evaluation data.
Results 
show the added difficulty when attacks are recorded with higher quality microphones and replayed with high-quality loudspeakers, both at a short distance from the talker and microphone respectively, or when the automatic speaker verification microphone is of lower quality.

Future editions of ASVspoof will continue the path towards
ever-more realistic conditions and will consider stronger adversaries, larger databases containing greater variation in acoustic conditions and spoofing attacks, data collected from a larger speaker population and an additional, relaxed training data policy.  
Finally, ASVspoof and SASV challenges will merge to allow greater flexibility in the form of jointly-optimised CM and ASV sub-systems and alternative combination architectures.









\section*{Acknowledgments}
The ASVspoof 2021 organising committee extends its sincere gratitude and appreciation to the 72 participants and teams. Since participants are assured of anonymity, we regret that we cannot acknowledge them here by name. This work was supported jointly by the French National Research Agency (grant No. ANR-18-JSTS-0001) and the Japan Science and Technology Agency (grant No. JPMJCR18A6) under project VoicePersonae. The work also received partial support from Academy of Finland (proj. 349605, SPEECHFAKES) and MEXT KAKENHI Grants (21K17775, 21H04906). The work also received partial support from Region Grand Est, France, and from the Science and Engineering Research Council, Agency for Science, Technology and Research (A$^\star$STAR), Singapore, through its Council Research Fund (Project No. CR-2021-005).


\iffalse
{













