

\section{Conclusion}
\label{sec:conclusion}


In this paper, we study learning lip-based audio-visual speaker embeddings from AV-HuBERT. We show that the general-purpose audio-visual speech representation learning framework, AV-HuBERT, is able to learn high-quality speaker embeddings that improves the performance on speaker related tasks including classification and verification using less labeled data. The proposed model also greatly improves robustness to a variety of noise while preserving better privacy as it requires only lip instead of whole face videos.
\section{Experiments}
\label{sec:exp}

\subsection{Setup}\label{sec:setup}

\noindent 
\textbf{Pre-training}\hspace{.5em} We pre-training a 12-layer BASE and 24-layer LARGE AV-HuBERT following~\cite{avhubert} and \cite{shi2022robust} with only one change. In~\cite{avhubert}, models are pre-trained with 433 hours of LRS3~\cite{afouras2018lrs3} and the English portion of VoxCeleb2 (VC2)~\cite{voxceleb2} from both the dev and test split. In this paper, since we evaluate the speaker embeddings on the VC2 test split and it contains both English and non-English speakers, we combine LRS3 with all VC2 data, English or non-English, but exclude the VC2 test split for pre-training, which sums up to roughly 2,800 hours. Both LRS3 and VC2 are sampled at a frame rate of 25Hz. The AV-HuBERT model also produces representations at this frame rate. To improve noise robustness, noise randomly sampled from MUSAN~\cite{Snyder2015MUSANAM} is added to the audio stream following~\cite{shi2022robust}. The audio and video preprocessing steps remain the same as~\cite{avhubert}.

\vspace{2pt}
\noindent 
\textbf{Fine-tuning}\hspace{.5em} 
For the frozen protocol, to compare with results reported in~\cite{Yang2021SUPERBSP}, we adopt the same prediction heads as~\cite{Yang2021SUPERBSP}, which is an average pooling layer for SC, and an x-vector model~\cite{snyder2018x} for SV. 
Two widely used audio-visual speech recognition datasets, VoxCeleb1 (VC1, 352 hours/1,251 speakers)~\cite{nagrani2017voxceleb}\footnote{We use the videos provided by~\cite{Nagrani18seeing} since the \cite{nagrani2017voxceleb} only releases processed audio files. As a few dozen video clips in VoxCeleb1 are missing in the data provided by~\cite{Nagrani18seeing}, we download raw videos from the URL and extract clips with ground-truth timestamps provided in~\cite{nagrani2017voxceleb}. The speaker face is then extracted with dlib face detector~\cite{david2009dlib}. The video files from~\cite{Nagrani18seeing} was down-sampled by a factor of six ($\sim$4.17Hz). To tackle this, we let the ResNet video feature extractor to process the downsampled video as is and upsample its output by a factor of six before passing it to the Transformer. Empirically, this leads to similar performance as upsampling the video at input to the ResNet while reducing the memory and the compute. 
} and VC2 (2,442 hours/6,112 speakers), are adopted for supervised fine-tuning. Noise-augmented fine-tuning with MUSAN following~\cite{shi2022robust} is explored in \S\ref{sec:noise_aug}.
All models are optimized with Adam~\cite{kingma2014adam} with learning rate warmed up to 0.001 for one third of training steps and then linearly decayed. The pre-trained parameters are frozen for certain steps before being updated. For \{5h, 50h, 500h, VC2\} setups in Table~\ref{tab:lab-eff}, we train the model for \{20, 30, 90, 75\} K-steps with a batch size of \{100, 100, 100, 400\} and \{60, 60, 120, 240\} for audio-only and audio-visual setting.
Whenever the model is used in audio-only setting, the visual feature (ResNet output) is replaced by an all-zero vector.


\vspace{2pt}
\noindent
\textbf{Evaluation}\hspace{.5em}
For SV, we follow the standard evaluation protocol used in~\cite{chung2020in}. We report SC accuracy (SC-ACC) and SV-EER on VC1 using the official test set and test trials. As for VC2-EER, since there does not exist official test trials, we follow~\cite{Sari2021AMA} to create one by sampling one positive trial and one negative trial for each test set utterance. To probe noise robustness, we follow \cite{shi2022robust} to create 20 noisy test sets for VC1 and VC2, where each clean set is mixed with \{Babble, Speech, Music, Other\} noise at a SNR in \{-10, -5, 0, 5, 10\} dB.

By default, we fine-tune all models on VC2 without noise augmentation and with the protocol where AV-HuBERT parameters are updated, and report EER on the clean set. We use AV-HuBERT Base for analysis purpose (sec~\ref{sec:eff_pt}-\ref{sec:face}).







\subsection{Effectiveness of AV-HuBERT pre-training}\label{sec:eff_pt}

\input{tables/lab-eff}

We first study label efficiency of AV-HuBERT pre-training for audio-only and audio-visual speaker verification, and evaluate their performance on both the clean and noisy test trials (Table~\ref{tab:lab-eff}). 
In the ``noisy'' columns, we report the average EER over the 20 testing configurations.
Three sizes of subsets are considered: 20\%/2\%/0.2\% of the labeled data. For each size, we generate subsets in two ways: one is by sampling x\% of the utterances, and the other is by sampling x\% of the speakers and select all their utterances. For a given size, the former would contain more speakers but less utterances per speaker. 

For each labeled subset, AV-HuBERT outperforms models trained from scratch (PT=None) when the same modalities are used as input. Comparing across subsets, we find AV-HuBERT often matches or outperforms a randomly initialized model that use 10 times more data
(e.g., AV-HuBERT is 16.7\% on VC2 clean set with VC2-5h data and AV input, while the baseline is 21.5\% with VC2-50h data and AV input).

Next, we observe that with pre-training, audio-visual models always outperform audio-only ones. The consistent gain on the clean set shows that lip videos bring complementary information for recognizing speakers, which is different from speech recognition where visual information helps very little in clean conditions~\cite{shi2022robust,ma2021conformer}. On the other hand, audio-visual based models bring substantial gains in noisy conditions compared to audio-based models (e.g., reducing EER from 20.0\% to 2.5\% on noisy VC1 test set when pre-trained and fine-tuned on the entire VC2). This demonstrates the enhancement in noise robustness by incorporating lip information.

Comparing the different subset sampling strategies for 0.2\% VC2 (VC2-5h versus VC2-15spk), it shows that pre-trained models benefit from having more speakers (w/ PT, 16.7\% for VC2-5h vs. 22.6\% for VC2-15spk on VC2 clean with AV input),\footnote{
The model achieves similar performance on noisy test sets for both subset sampling strategies, which is caused by the mismatch in training/testing audio conditions (clean vs. noisy). With noise-augmented fine-tuning, the same trend emerges again (w/ PT, 5h vs. 15spk: 26.1\%/13.8\% vs. 30.7\%/21.6\% with A/AV input on VC1 noisy).
} while the supervised audio-visual baselines struggle with too few examples per speaker (w/o PT 32.8\% vs. 29.8\% on VC2 clean with AV input). 
This implies that a visual encoder trained from scratch hardly benefits from increasing number of speakers given only few utterances per speaker.
In the 5h-setting (few utterances per speaker), the extra visual modality is only effective in case of pre-training whereas an audio-visual model trained from scratch is consistently worse than its audio-only counterpart regardless of auditory conditions (PT=None, A vs. AV). To conclude, we discover pre-training can lead to better generalization with few-shot learning.













\subsection{Noise-augmented fine-tuning}\label{sec:noise_aug}
Observing the gap between audio-based and audio-visual models on SV in the previous section, we study if applying noise augmentation during fine-tuning can bridge the gap. We fine-tune AV-HuBERT on VC2-500h audio and audio-visual data with and without noise augmentation described in \S\ref{sec:setup}

The performance of the four models (A or AV, with or without noise augmentation) on the 20 noisy test sets are presented in Table~\ref{tab:noise-aug}. Unsurprisingly, all models perform worse on lower SNR conditions. Nevertheless, we observe much bigger degradation for audio-based model, especially when corrupted with speech (S) or babble (B) noise, because audio models can not determine who the target speaker is from a mixture of speech.
In contrast, audio-visual based models suffer very minor performance degradation in noisier conditions, because lip videos can help identify the target to infer speaker embedding for.

We also see noise augmentation reduces the average EER at -10dB from 43.7\% to 30.8\% for audio-only models, and from 6.3\% to 3.3\% for audio-visual models. The results suggest that while noise-augmentation is beneficial, adopting it alone can not bridge the gap between audio and audio-visual models pre-trained and fine-tuned on the same amount of data.

\input{tables/noise-aug}









\subsection{Choice of visual input}\label{sec:face}
Using lip instead of face videos is one of the key features of the proposed model as it better addresses privacy concerns while exhibits excellent noise robustness. We quantify in this section how much our model degrades by comparing it with a variant of AV-HuBERT pre-trained and fine-tuned on face videos. As a baseline, we also evaluate a widely used face recognition model RetinaFace~\cite{deng2020retina} taking a still image as input, which only achieves an EER of 14.6\% on VC2. Table~\ref{tab:face-vs-mouth} shows that the face-based AV-HuBERT model indeed performs slightly better (0.9\% absolute EER reduction) , which is a trade-off to be considered.


\input{tables/face-lip}


\input{tables/overall-comparison}

\subsection{Comparison with prior work}\label{sec:prior}
We first compare AV-HuBERT with other self-supervised models using the SUPERB~\cite{Yang2021SUPERBSP} protocol that evaluates the quality of frozen representations (upper half of Table~\ref{tab:cmp_all}), where models are fine-tuned and evaluated on VC1 for SC and SV. 
With audio-only input, AV-HuBERT-L outperforms wav2vec2-L and HuBERT-L and is inferior to WavLM-L. 
Outperforming all audio-only pre-trained models, AV-HuBERT achieves significantly better results on the SC and SV tasks with audio-visual input even with an order of magnitude less pre-training data. 

The lower half of Table~\ref{tab:cmp_all} compares AV-HuBERT with other supervised and self-supervised models reporting results on VC1 or VC2 without following the SUPERB protocol.
Specifically, we fine-tune our pre-trained model using the whole VC2 labeled data. As is expected, fine-tuning with a large amount of labeled data improves performance. In audio-visual setting, our best model (0.84\%) outperforms~\cite{Sari2021AMA} (1.8\% and 1.4\%) with a single model and slightly falls behind its ensembled model (0.7\%). Note in contrast to the prior works~\cite{Nagrani2020DisentangledSE,Sari2021AMA,shon2019noise} which uses the whole face, our model only relies on the lip area of the speaker as visual input and achieves a better trade-off between privacy and performance. In addition, we acknowledge the gap between our best model and the current SOTA on VC1 (\cite{Chen2021WavLM}: 0.38\%). Such gap is attributed to the relative smaller amount of pre-training data we use as well as the simplicity of our training and evaluation pipeline (e.g., \cite{Chen2021WavLM} uses Inter-TopK penalty~\cite{zhao2021speakin}, ECAPA-TDNN~\cite{desplanques2020ecapa} as an upstream network, second stage large-margin fine-tuning, adaptive s-norm~\cite{Karam2011TowardsRF,Cumani2011ComparisonOS} and calibration~\cite{Thienpondt2021theidlab} in evaluation, etc). As the goal of this paper is to study the speaker embedding learned by AV-HuBERT, we stick to a simple and standard pipeline for speaker verification. 
How to incorporate extra techniques in AV-HuBERT framework is left as our future work. 






\section{Introduction}
\label{sec:intro}

Personalizing user experiences is essential in spoken language technology systems, e.g., smart speakers and personal banking applications. Robust speaker verification (SV) and recognition models are crucial for enabling authentication and conversational experiences, as well as many other tasks like speaker diarization~\cite{Wang2018SpeakerDW}, voice conversion~\cite{Zhang2020DurIANSCDI} and source separation~\cite{Wang2019VoiceFilterTV}.

Supervised speaker representation methods made significant progress over the past decade~\cite{Snyder2018xvectors,chung2020in, desplanques2020ecapa, Lee2020two}; however, they require a non-trivial amount of human annotations of speaker identity, which might not comply with the evolving privacy-preserving standards. Furthermore, it is challenging to provide speaker labels for multi-speaker dialogues or when speakers' voices alternate between whispering and shouting~\cite{hansen2015speaker}. Self-supervised speaker representation approaches, which work around these challenges, have recently gained popularity. One family of self-supervised speaker representation methods relies on contrastive learning, which constructs positive samples by either augmenting the same speech segment or assuming a single speaker is recorded per utterance~\cite{Inoue2020SemiSupervisedCL,tao2021self,Xia2021SelfSupervisedTS}. They present solid downstream performance, where the unsupervised state-of-the-art approach~\cite{tao2021self} achieves EER of 1.66\%, close to some of the SOTA supervised systems (e.g., 0.41\% from \cite{zhao2021speakin}). However, one downside of these approaches is that they are tailored solely for speaker embedding tasks. In contrast, general self-supervised speech representation learning approaches, e.g., wav2vec 2.0~\cite{Baevski2020wav2vec2A} and HuBERT~\cite{Hsu2021HuBERT}, were found to capture enough speaker information to be competitive on SV while excelling at many other downstream tasks~\cite{Yang2021SUPERBSP}. 

The Audio-Visual Hidden Unit BERT (AV-HuBERT) was recently introduced as a general audio-visual representation learning approach. It learns joint representations over speech and lip-movement streams by alternating between clustering representations using a small codebook mimicking broad phonetic units and learning latent contextual representations through the masked prediction loss. AV-HuBERT achieves SOTA results on lip-reading and audio-visual speech recognition (AVSR) under adverse noise conditions~\cite{shi2022robust,avhubert}, thanks to the noise-immune visual modality.

This paper goes beyond single modality speaker representations to work with audio and lip-movement information to learn noise-robust speaker embeddings. We extend the representation learned by the AV-HuBERT approach to study their effectiveness for speaker-based downstream tasks in multi-modal settings.
Compared to recent specialized unsupervised speaker representation methods~\cite{Inoue2020SemiSupervisedCL,tao2021self,Xia2021SelfSupervisedTS}, one advantage of utilizing a general approach like AV-HuBERT is its ability to simultaneously serve other downstream tasks beyond speaker embedding. Prior work on audio-visual speaker representation learning focused on the consistency between the audio and the visual information, either by learning speaker embedding via audio-visual synchronization and identity matching~\cite{Nagrani2020DisentangledSE} or by multi-way matching in a joint audio-visual embedding space~\cite{Chung2020PerfectMS}. AV-HuBERT offers a more stable training procedure than methods utilizing contrastive and consistency objectives since its masked prediction loss is computed over offline learned discrete units. 

In our experiments, AV-HuBERT representations are used either in an ELMo-style feature combination protocol~\cite{sarzynska2021detecting} or through fine-tuning the whole network for the target downstream task. We report our results on speaker classification and verification tasks under four types of interfering noise and five different signal-to-noise ratios (SNR). Our audio-visual models improve label efficiency by 10 folds from supervised models, and offer 38\% and 75\% relative equal error rate (EER) reduction for SV under clean and noisy conditions compared to audio-only pre-trained models.








\section{Method}
\label{sec:method}
\begin{figure}[t]
    \centering
        \caption{\label{fig:avsv-model}AV-HuBERT for learning speaker embedding. Dashed box: added during fine-tuning.}
    \includegraphics[width=\linewidth]{figure/avsv-model-v2.pdf}
\end{figure}

\subsection{Overview of AV-HuBERT}
Audio-Visual Hidden Unit BERT (AV-HuBERT) is a self-supervised model that learns from unlabeled audio-visual speech data. Similar to its audio counterpart --- HuBERT~\cite{Hsu2021HuBERT}, AV-HuBERT was initially benchmarked on speech recognition tasks and achieved state-of-the-art performances on uni-modal (audio-only and video-only)~\cite{avhubert} and multimodal (audio-visual)~\cite{shi2022robust} setups. As depicted in Figure~\ref{fig:avsv-model}, AV-HuBERT comprises four modules: a feed-forward network (FFN) audio feature extractor, a modified ResNet~\cite{Stafylakis2017CombiningRN,Martnez2020LipreadingUT} video feature extractor, a fusion module, and a Transformer~\cite{Vaswani2017attention} backend. The two feature extractors generate frame-level representation for the corresponding stream, which are frame-wise concatenated by the fusion module to form initial audio-visual features. The Transformer backend takes these features and produces the contextualized frame-level audio-visual representations. The entire model is optimized to perform masked prediction, where random segments are masked for each stream independently (denoted by the red crosses in Figure~\ref{fig:avsv-model}), and the model learns to predicts the cluster assignment of the masked frames (the middle three frames in the example). The cluster assignment are iteratively refined: it is produced by clustering MFCC features in the first iteration, and by clustering previous iteration's AV-HuBERT representations in the subsequent iterations. 

While the self-supervised learning objective of HuBERT and AV-HuBERT (masked prediction of cluster assignments) resembles the task of automatic speech recognition (ASR)~\cite{avhubert} because it could be understood as joint acoustic and language modeling, several analyses observed that such models still learn rich speaker information especially in earlier layers~\cite{chang2021distilhubert, Yang2021SUPERBSP, Chen2021WavLM}.
These observations can be linked to many studies in the ASR and SV literature. For example, researchers have found that speaker information can improve ASR performance~\cite{senior2014improving}. Hence, AV-HuBERT may learn to extract speaker information at early layers to better infer phonetic information in the subsequent layers. In addition, statistics of the acoustic features extracted from ASR models have been widely used for SV~\cite{lei2014novel}, which can also explain why HuBERT features benefits SV. Motivated by the strong results of adapting HuBERT for SV, we explore learning audio-visual speaker embeddings from a pre-trained AV-HuBERT model with emphasis on the noise robustness aspect strengthened by the addition of visual stream.

\begin{figure}[t]
    \centering
        \caption{Example lip images, from LRS3 dataset~\cite{afouras2018lrs3} \label{fig:mouth-grid}. Images in the same row are from the same speaker. }
    \includegraphics[width=0.8\linewidth]{figure/grid_gray_3row.png}
    \vspace{-0.2in}
\end{figure}




\subsection{Learning multimodal speaker embedding}


We describe in this section how to learn speaker embeddings with pairs of audio-visual speech $\mathbf{X}$ and speaker label $y$.
Given a pre-trained AV-HuBERT model $f_{\thetaavh}$ parameterized by $\theta_{avh}$, the goal is to leverage it to learn a speaker embedder $f_{\thetaspk}$ such that $f_{\thetaspk}(\mathbf{X}_i)$ are similar to $f_{\thetaspk}(\mathbf{X}_j)$ if $y_i = y_j$, and are dissimilar otherwise.

Two learning protocols are considered, which differs in whether $f_{\thetaavh}$ is frozen or not. 
The former is used to evaluate representation quality, since it treats the self-supervised model as a fixed feature extractor. 
Following~\cite{Yang2021SUPERBSP}, we consider ELMo~\cite{Peters2018DeepCW} style fine-tuning, where frame-level AV-HuBERT representations at each Transformer layer are weighted and summed using a learnable non-negative weight vector that sums to one. This representation is then passed to a prediction model to produce a fixed-sized speaker embedding, followed by a softmax layer to estimate the posterior over a closed set of training speakers. The weight vector, prediction model, and softmax layer are trained to minimize a cross entropy loss with respect to speaker labels.
The architecture of the prediction model depends on the downstream task but is generally much lighter compared to AV-HuBERT.

The other protocol is often adopted for pre-training~\cite{bert, Baevski2020wav2vec2A, Hsu2021HuBERT, baevski2022data2vec}, where a self-supervised model is updated along with new parameters during fine-tuning. Similar to how BERT does sequence classification tasks~\cite{bert}, we prepend an additional trainable \texttt{[cls]} vector to the Transformer input as shown in Figure~\ref{fig:avsv-model}, and take its contextualized representation at the Transformer model output as the speaker embedding. Pooling of information across frames is carried out by the self-attention module at each Transformer layer. Similar to the other protocol, the speaker embedding is passed to a softmax layer to predict the speaker label and the same loss function is used, but in this protocol the entire model is optimized.

Two tasks, SV and speaker classification (SC), are used for evaluating learned speaker embeddings. SC evaluates prediction accuracy of a closed set speaker seen during training. In contrast, SV considers an speaker-independent setup, where a set of testing trials is provided, each of which contains two utterances and a label indicating if the two are from the same speaker. The learned speaker embedder is used to compute a similarity score for each trial, and the equal error rate (EER) is reported, which is the error rate where the false positive rate is the same as the false negative rate.

We should note that unlike most of the prior work on learning audio-visual speaker embeddings~\cite{shon2019noise, Sari2021AMA}, we use lip video (see figure~\ref{fig:mouth-grid}) instead of whole-face video or image as input. This can significantly improve the robustness in noisy environments compared to audio-based systems while reducing the amount of biometric information required compared to face-based systems.








