\section{Introduction}
\label{section:introduction}

Over the years, we have always been fascinated with questions that can push the limits of computer vision. For instance, can we recognize actions~\cite{efros} or objects~\cite{torralba_2006} in videos/images of about $30\times30$ pixels? Or how small a face can we detect in an image? It turns out, the face can be as tiny as $3$ pixels tall~\cite{Hu_2017_CVPR}! Recovering extremely feeble signals has also led to remarkable achievements, such as the imaging of the black hole. In this vein, we explore whether we can upsample talking-face videos with resolutions as low as $8\times8$ pixels. Clearly, this is an exceptionally challenging task. Surprisingly, we reveal that one can obtain realistic high-resolution (HR) talking-faces ($256\times256$ pixels) when provided with the right set of additional information. We utilize a single target identity image and the accompanying audio to upsample $8\times8$ video to a full $256\times256$ dimensional video (a fascinating $32\times$ scale-factor) that far exceeds previous methods. In today's digitally connected world, where talking-face videos are among the most common forms of video content, our multi-modal system can have numerous potential applications. Some of them include: (i) video conferencing in low-bandwidth situations, (ii) recovering low-quality archival footage of public talks and speeches and (iii) enhancing videos captured from a distance with a high camera zoom.

\vspace{5pt}
\noindent
\textbf{Challenges:}
Although our task has promising applications, synthesizing HR videos from extremely low-resolution (LR) inputs (e.g., $8\times8$ pixels) is a very challenging task. Essential face attributes such as identity, age and gender are almost entirely lost at such low resolutions and cannot be directly recovered (see LR video in Figure~\ref{fig:teaser}). Apart from reconstructing these elementary identity details, the network must also learn to predict the original head poses and accurate lip shapes. Given the arduous nature of the task, it is evident that the model will struggle to achieve the desired quality if it relies solely on the LR input. Thus, in our work, we argue that considering appropriate priors is quintessential to obtain high-quality results. Specifically, we assess the importance of two kinds of priors: (i) audio signal, which can help to recover elemental facial attributes and can significantly improve lip shape generation; (ii) a single HR target identity image which can aid in restoring fine-grained details such as skin texture, color, hair, teeth and surrounding background. The target identity image can be any sample frame, either from the same video or any other image with similar characteristics in terms of the face identity, pose, clothing and background. To understand our task better, we will now explore how our work is connected to some of the existing problems in literature. 

\vspace{5pt}
\noindent
\textbf{Super-Resolution (SR) Perspective:}
The SR literature till date has focused on super-resolving inputs (either faces, generic images or videos)~\cite{progressive-face-sr, dbpn, RBPN2019, tecogan} where sufficient information is already available (e.g., $256\times256$ pixels input). None of these methods can handle extreme scale-factors like $32\times$. When the input resolution is very low (like $8\times8$), we observe that most of the current SR works~\cite{progressive-face-sr, dicgan, dbpn, SPARNet, tecogan, frvsr, RBPN2019} generate sub-optimal results. The essential visual attributes such as the identity, face texture and lip shape do not accurately match the original face. This is natural because the network is forced to speculate these details without adequate priors. Thus, the existing methods: (i) do not aim to preserve the specific identity details and (ii) do not explicitly deal with talking-face videos where specialized temporal information like lip synchronisation must be maintained throughout the video.

In our work, we propose to address these limitations by generating high-quality talking-face videos with accurate lip-sync. It is important to note that although we aim to generate HR talking-faces from LR inputs, the task we are attempting is very different compared to the typical SR problem. The use of a single HD target identity image and synthesizing from extremely LR talking-faces ($8\times8$ pixels) sets us considerably apart from the traditional SR.

\vspace{5pt}
\noindent
\textbf{Compression Perspective:}
Our task enables applications such as low-bandwidth video calling; thus making it closely related to the task of talking-face video compression. However, unlike the existing works like ``os-synth''~\cite{nvidia_compression_2021_cvpr} where 3D face keypoints are transmitted, we take a unique path in our work. We propose to transmit the LR frames to extract the face structure, motion and pose information. Transmitting keypoints have several limitations: (i) Keypoints can only be extracted if we have the HR video beforehand. While the availability of actual video might be a possibility in video conferencing, this poses severe constraints in various other applications where the actual HD video is not present; (ii) Keypoints do not encode adequate head pose information, thus requiring additional specialized head pose estimation models; (iii) Keypoints do not cover other information like background, lighting, accessories like glasses and beards that could be present in talking-faces. Thus, in our work, we demonstrate the advantages of using LR frames and achieve better compression ratio compared to the standard codecs, while also not compromising on the desired quality.

\vspace{5pt}
\noindent
\textbf{Talking-Face Animation Perspective:}
Our task also shares similarities with audio-driven talking-face generation (A2TF)~\cite{YST_ijcv_2019, lipgan_2019, wav2lip, Yang:2020:MakeItTalk, talkingfaceacmmm} and face re-enactment (FR)~\cite{fomm_nips_2019, PC_AVS_2021_cvpr} tasks. A2TF works aim to generate videos of a target identity conditioned on the audio signal (input: single target identity image + audio). FR methods ingest a single target identity image and an HD video of a different identity with an aim to animate the target image according to the motion of the driving video (input: single target identity image + HD video as pose prior). Although we agree that in terms of the problem space, our work resembles A2TF/FR, we want to point out that our focus is very different - upsampling extremely low-resolution videos. There are key differences as noted in Table~\ref{table:tfgen_differences}. We leverage positive aspects from these dimensions to solve an entirely new task - generating HD videos from extremely LR inputs while preserving the exact same facial features, e.g., pose and identity. Nevertheless, we also include a comparison with A2TF and FR works in Section~\ref{section:A2TH_comparison}.


\vspace{-12pt}
\begin{table}[ht]
    \centering
    \small
    \setlength{\tabcolsep}{3pt}
    \caption{Key differences between audio-driven talking-face generation (A2TF), face re-enactment (FR) and proposed task.}

    \vspace{-8pt}
    \begin{tabular}{l|c|c|c|c}
    \hline

    & Pose prior & Lips are in-sync & Matches GT & Can be used \\
    & used & with audio? & frames? & for SR?\\
     
    \hline
    
    A2TF & None & $\checkmark$ & $\times$ (changes pose) & $\times$\\
    
    FR & HD video & $\checkmark$ & $\checkmark$ (same id-recons.) & $\times$\\
    
    \hline
    
    \textbf{Ours} & LR frames & \textbf{$\checkmark$} & \textbf{$\checkmark$} & \textbf{$\checkmark$}\\

    \hline
    
    \end{tabular}
    \vspace{-10pt}
    \label{table:tfgen_differences}
\end{table}



\vspace{-5pt}
\noindent
\textbf{Overview of this Work:}
In this work, we propose a talking-face video upsampling framework, where the core idea is to utilize adequate priors to generate high-quality ($256\times256$) videos from extremely low-resolution inputs. A gallery showing our synthesized frames is displayed in Figure~\ref{fig:gallery}. We conduct extensive experiments and comparisons with state-of-the-art methods for the tasks of super-resolution and compression. To the best of our knowledge, we are the first to synthesize high-quality talking-faces at scale-factors of $32\times$ from an input as small as $8\times8$ pixels. We also show how our network can be utilized for low-bandwidth video conferencing, along with a demo video on our project page. 

\begin{figure}[t]
  \includegraphics[width=\linewidth]{images/gallery.png}
  \vspace{-20pt}
  \caption{Input: LR frames (top); Output: HR frames (bottom). Our approach effectively synthesizes a diverse set of HR talking-faces, handling different poses, gender, age and race. Our method also works for ``human-like'' synthetic faces even though it was trained only on real faces.}
  \label{fig:gallery}
  \vspace{-15pt}
\end{figure}

To summarize, the major contributions of our work are:

\begin{itemize}[topsep=0.8pt, noitemsep, leftmargin=3mm]

    \item We present a novel audio-visual network that can upsample very low-resolution talking-face videos at scale-factors previously unseen in video SR literature ($32\times$). 
    
    \item Our approach to make use of adequate priors: (i) audio signal and (ii) a single target identity, achieves significant improvements over the existing works for SR and compression tasks.
    
    \item Our system has strong real-world use-cases, owing to its ability to preserve true identity information. We specifically demonstrate the superior quality of our results for low-bandwidth video conferencing and achieve a significant reduction in bandwidth compared to H.264 standard.
    
\end{itemize}


\section{Related Work}
\label{section:related_works}

\noindent
\textbf{Image \& Video Super-Resolution:}
Single image super-resolution (SISR) has progressed tremendously with the advent of deep learning and CNNs~\cite{dong_first_sr}. A flurry of works~\cite{vdsr, edsr, ntire_report, dbpn, talkingfaceacmmm, depthSR, SRusingdual} have followed this initial method, thereby improving the performance by many folds. To account for specialized facial attributes which are not present in generic images, face super-resolution (FSR) methods~\cite{SPARNet, face_sr_somework, KIM202111} with specific loss functions like facial landmarks~\cite{dicgan} and facial heatmaps~\cite{progressive-face-sr} were proposed. However, these FSR approaches focus on scale-factors up to $8\times$, which is in stark contrast to our attempt of $32\times$ SR.  

Recently, PULSE~\cite{PULSE_CVPR_2020} proposed a StyleGAN-based approach to super-resolve static faces at high scale-factors of $64\times$. However, the model generates imaginary faces of people who do not exist, posing severe constraints on applications where the person's identity needs to be matched/recovered correctly. Conversely, in our work, we aim to generate HR faces of the specific (real) identity.

To capture the temporal aspect which is not present in the static faces, video SR approaches came into picture. From early recurrent architectures~\cite{video_sr_1} to more recent advancements~\cite{frvsr, RBPN2019, tecogan}, impressive results have been achieved for scale-factors up to $4\times$. However, these generic video SR methods produce blurred results with many artifacts when the input resolution is very low ($8\times8$) and the scale-factors are very high ($>8\times$). Moreover, ``talking-face videos'' have their own set of additional challenges, which are neither tackled in video SR nor in static FSR works. Thus, in our work, we overcome these limitations and design a novel network that specifically deals with talking-face videos. 


\vspace{5pt}
\noindent
\textbf{Talking-Face Video Generation:}
Audio-driven talking-face generation (A2TF) is an active research area. Various methods have been proposed~\cite{text-based-editing, Yao2020IterativeTE, wav2lip, talking_face_video_sig, Yang:2020:MakeItTalk} to accurately morph the lip movements of input target identity to be in-sync with the corresponding speech. Face re-enactment (FR) works~\cite{face2face, deep_video_portraits, fomm_nips_2019} have also shown impressive performance in transferring head motions and expressions based on guiding videos. While our task shares some similarities with A2TF/FR works, we differ in the fact that we are given very sparse information in the form of very low-resolution input. This is not the case in talking-face generation models where HR frames (or extracted landmarks) are used for conditional generation.

Few works~\cite{deep_video_portraits, facial_iccv_2021} use a 3D model as an intermediate step to recover high-quality videos. The major benefit of adapting a 3D model is the superior quality of the output generations. But, it comes with an additional overhead - the computational complexity, making such heavy models very impractical for mobile hardware deployments. In contrast, our method is (i) simple, (ii) generalised, since it can be applied for any in-the-wild identity (unlike some of the 3D models that require speaker-specific training) and (iii) does not require specialized large-scale 3D datasets to train the models.

\vspace{5pt}
\noindent
\textbf{Data Compression using Deep Learning:}
Deep learning has lead to profound improvements in image and video compression techniques. Starting from initial auto-encoder based methods~\cite{CompressionAE_icassp_2018, CompressionNN_nips_2014} to flow-based approaches~\cite{Agustsson_2020_CVPR, CompressionFlow_nips_2019}, there have been multiple efforts to obtain a compact image/video representation. To enable video calls at a reduced bandwidth, specific methods like~\cite{VideoConfMedical_2013_atnac, VideoChatCompression_2021_cvprw} have been designed. In ``SRVC''~\cite{SRVC_2021_ICCV} authors used video SR as a tool for compression, but at a scale-factor of merely $2\times$. Recently, ``os-synth''~\cite{nvidia_compression_2021_cvpr} demonstrated impressive results by transmitting a sequence of learned 3D facial keypoints. However, as discussed previously, transmitting keypoints has its own set of limitations. Thus, unlike the existing compression methods, we take a path of extreme-scale ($32\times$) SR for the first time and transmit very low-resolution ($8\times8$) videos. We show that our proposed approach of utilizing the audio and the visual modalities can enable video-conferencing in bandwidth-limited regions, while also achieving a better compression ratio over the existing works.


\section{Learning to Upsample LR Videos}
\label{section:methodology}

We start the discussion by highlighting the critical elements of our approach. We then present our framework with a detailed description of the modules involved.

\vspace{-5pt}
\subsection{Critical Elements of our Approach}

\noindent
\textbf{Audio Prior:}
\label{issue:ambiguous_nature}
As discussed previously, when the input resolution is a meager $8\times8$ pixel video, the ambiguity and the loss of information is so paramount that the person's original identity characteristics are barely discernible. In such situations, we show that audio can aid in the recovery of dominant facial traits of the person because the audio and the face share multiple common features~\cite{speech2face, SeeingVoices_cvpr_2018, FaceVoiceAssociations_accv_2018} like gender, age and ethnicity. We exploit the audio signal not only to disambiguate the LR input, but also to greatly improve the lip shape generation. Although precise lip shape is not a crucial necessity for static face SR, it is a very important aspect of video SR where the generated lip movements should sync with the given speech. In our work, we explore the natural correlation between speech and lips~\cite{wav2lip, conversation, Hegde_2021_WACV, perfect_match} to generate accurate lip movements. \\ 

\vspace{-5pt}
\noindent
\textbf{Visual Prior:}
\label{issue:inadequate_prior}
To be able to generate faces that replicate the actual identity, it is important to preserve sharp details like face texture, lip colour, hair, teeth and skin tone. Most of the current works hallucinate these details, leading to significant variations in fine-grained information. Such models are thus unusable for real-world use-cases, where videos of a specific identity needs to be generated. We argue that considering the adequate prior information is of utmost importance to: (i) generate a video of a specific identity and (ii) reconstruct the high-quality facial details. To achieve this, we provide our network with a single HR image of the target identity, which helps to transfer the identity-specific sharp details to the synthesized video. In most applications, a single HR identity image is easily accessible. For example, during video conferencing, the first frame can be transmitted in the original resolution.

\vspace{-5pt}
\subsection{Our Approach}

\begin{figure*}[t]
  \includegraphics[width=\textwidth]{images/architecture.jpeg}
  \vspace{-20pt}
  \caption{We propose a novel audio-visual talking-face upsampling framework. Our approach recovers the basic facial attributes like the identity structure, pose and motion using a backbone network. We then consider a single HR target identity image to capture the fine-grained details. Our end-to-end trainable animation network ingests the intermediate outputs (from backbone network) and the target identity and reconstructs realistic, high-quality videos that accurately match the actual identity.}
  \label{fig:architecture}
  \vspace{-10pt}
\end{figure*}

An overview of our proposed framework is depicted in Figure~\ref{fig:architecture}. The goal of our work is to generate a sequence of HR frames, $F_{hr} = (hr_1, hr_2, ..., hr_n)$ from the LR input, $F_{lr} = (lr_1, lr_2, ..., lr_n)$ that accurately match the ground-truth frames $F_{gt} = (gt_1, gt_2, ..., gt_n)$. We consider the corresponding audio signal, $A$ and a single HR target identity, $F_{id}$ as prior information. We detail the different components of our model and the network architecture below. 

\vspace{3pt}
\noindent
\textbf{Backbone Network:}
We pre-train a backbone network to generate a ``driving video'' for our face animation network. It extracts the relevant features of the identity, face structure, pose and motion information. We observe that pre-training the network before training the face animation network helps to improve the model's performance because the basic face details and motion information captured is essential to animate the HR face (details about the animation network is explained below). We empirically demonstrate the importance of this in the ablation study (in supplementary).


\noindent
\underline{\textit{Visual Encoder:}} We extract the visual features from the LR frames $F_{lr}$ using a visual encoder, which comprises a series of $3$D convolution layers with residual connections. Our visual encoder resembles multiple previous models~\cite{avobjects, Hegde_2021_WACV, perfect_match} designed specifically for processing talking-face videos. The input to the visual encoder is a contiguous window of $T$ LR frames $l: (N, T, 3, 8, 8)$ where $N$ refers to the batch size and $T$ refers to the window frames (here $T$=5). The encoder processes these input frames and generates the visual embeddings, $f_v: (N, T, 512, 8, 8)$.

\noindent
\underline{\textit{Audio Encoder:}} We consider the corresponding audio segment $A$ and extract the melspectrogram representation using a window length of $25$ms with a hop length of $10$ms sampled at $16$kHz. The melspectrograms $(T', 80)$ are given to the audio encoder, which is a stack of residual $1$D convolutions with appropriate strides to match the visual time-steps $T$. The generated features ($N, T, 512$) are then upsampled using transpose convolution layers to obtain the audio embeddings, $f_a: (N, T, 512, 8, 8)$. 

\noindent
\underline{\textit{Identity Structure $\&$ Motion Predictor:}} We concatenate the learned visual and audio embeddings in the latent space (along the channel dimension) to obtain $f_{cat} = (N, T, 1024, 8, 8)$. Inspired from Deep Back-Projection Network (DBPN)~\cite{dbpn}, we use iterative upsampling and downsampling layers in our module, where the primary idea is to effectively capture the mutual relationship between the LR and HR frames. We consider the fused representation $f_{cat}$ and stack the time steps along the batch dimension to obtain $f_{cat}: (N*T, 1024, 8, 8)$. As the visual and the audio encoders have already captured the temporal information, the stacking strategy improves the convergence speed and also gives us the desired performance. The output of this block is a sequence of frames ($F_{int}$) of resolution $256\times256$ pixels, which encapsulates the elemental facial details, face structure, pose and motion information. The network is trained to minimize the $L_1$ reconstruction loss:


\vspace{-5pt}   
\begin{equation}
L_{\mathrm{rec}} = \frac{1}{N} \sum^{N}_{i=1} ||F_{int} - F_{gt}||_1
\label{eqn:recon}
\end{equation}

\noindent
\textbf{Face Animation Network:}
To synthesize the HR videos of the target identity by preserving all the details, we consider a single HR image of the target identity as our input. This is in-line with the audio-driven talking-face generation works~\cite{wav2lip, PC_AVS_2021_cvpr, fomm_nips_2019, nvidia_compression_2021_cvpr} where a single target identity is considered to replicate the identity-specific details. The target identity image helps to capture the fine-grained features like face texture, skin tone, hair and lip color, which are otherwise not recovered in the existing FSR works~\cite{dicgan, dbpn, SPARNet}. These details are crucial, especially when our input is a mere $8\times8$ pixel video and make our model applicable in cases where we must match the actual identity to the maximum extent. We adopt one of the popular methods, FOMM~\cite{fomm_nips_2019} to animate the target identity based on the driving video obtained from our backbone network.

\noindent
\underline{\textit{Overview of First-Order Motion Model (FOMM):}} FOMM ingests a target identity image (to extract the appearance) and a driving video (to extract the pose and motion). The learned latent representation of motion in the driving video is combined with the target identity to synthesize the output video. During training, the model observes the target-driving image pairs and predicts a dense motion field which is later encoded using a keypoint detection network. The target image is then rendered according to the learned trajectories in the driving video. We refer the reader to~\cite{fomm_nips_2019} for more details about FOMM. Note that FOMM is designed to work even when the target image and the driving video are of different identities. However, this feature is not necessary in our work where the aim is to preserve the target identity, so we make appropriate modifications, as described below.

\noindent
\underline{\textit{Adapting FOMM to our Task:}} For our task at hand, the goal is to animate the HR target identity image $F_{id}$ in accordance with the motion of the driving video $F_{int}$. As described previously, our backbone network reconstructs the basic identity attributes like face structure, age and gender. We thus generate a residual mask as the output of the animation network and add the intermediate outputs $F_{int}$ (see Figure~\ref{fig:architecture}) to obtain a realistic HR talking-face video $F_{hr}$ of the target identity as the final output. We fine-tune the entire network (including the backbone network) end-to-end, by optimizing the FOMM loss $L_{fomm}$ and our task-specific losses, $L_{rec}$, $L_{region}$ and $L_{sync}$. The FOMM loss $L_{fomm}$ consists of: (i) a VGG-19 based perceptual loss at multiple resolutions and (ii) an equivariance constraint to enforce the model to predict consistent keypoints to known geometric transformations. We describe other task-specific losses that we use in out network below.

\noindent
\underline{\textit{Enforcing Local Correspondence:}} In our experiments, we observed that the model at times generates frames where the facial regions like eyes, eyebrows and lips are slightly off-position. This occurs if the target identity ingested by our animation network is significantly different from the driving video. Hence, to further improve the generation quality, we add a face landmark-based region loss, which penalizes the network for generating incorrect regions. We compute the face landmarks~\cite{fan_landmark_iccv_2017} for both GT $F_{gt}$ and generated frames $F_{hr}$ and extract the following $R$ face regions: lips, nose, eyes and eyebrows. We then add a patch-based local loss by minimizing the $L_2$ distance for all these $R$ regions (here $R$=4) to ensure that the predicted regions are as close as possible to the actual regions. 

\vspace{-10pt}
\begin{equation}
    L_\mathrm{region} = \frac{1}{N} \sum^{N}_{i=1}\sum^{R}_{r=1} ||F_{hr} - F_{gt}||_2
\label{eq:l2_region}
\end{equation}


\noindent
\underline{\textit{Enforcing Accurate Lip-sync:}} Prior works~\cite{wav2lip} involving speech and lip movements have shown that using a lip-sync discriminator can greatly benefit in enforcing strong audio-visual correspondences. This can also be observed in our task, as learning to synthesize HR frames from very low-resolution input might lead to generating lips that are out-of-sync with the audio segment. Thus, we pre-train a lip-sync discriminator, ``SyncNet'' adapted from~\cite{perfect_match}, trained to maximize the cosine similarity between the lip-speech pair $(F_{gt}, A)$ when they are in-sync (and minimise the similarity if they are out-of-sync). Once trained, we use this network as a frozen discriminator to penalize the generated frames $F_{hr}$ if they do not match the corresponding audio segment. In our end-to-end network, we minimize the sync loss:

\vspace{-10pt}
\begin{equation}
    L_\mathrm{sync} = - \frac{1}{N} \sum^{N}_{i=1} \log(\dfrac{f_{hr} \cdot a}{max(\lVert f_{hr} \rVert_{2} \cdot \lVert a \rVert_{2}, \epsilon)})
\label{eq:syncloss}
\end{equation}


\subsection{Training Settings and Datasets}


The final loss function is the combination of the above losses:

\vspace{-15pt}
\begin{equation}
    \resizebox{.9\hsize}{!}{$L_\mathrm{HR} = \lambda_{rec}L_\mathrm{rec} + L_{fomm} +  \lambda_{region}L_\mathrm{region} + \lambda_{sync}L_\mathrm{sync}$}
\label{eq:final_loss}
\end{equation}

\noindent
In our experiments, we set $\lambda_{rec}=50$, $\lambda_{region}=100$ and $\lambda_{sync}=0.05$. We provide the details regarding pre-processing and training settings in supplementary file on our project page.

\vspace{3pt}
\noindent
\textbf{Datasets:} 
We train our model using AVSpeech~\cite{avspeech_2018_tog} and VoxCeleb2~\cite{voxceleb2_2018_interspeech} datasets; both containing talking-face videos spanning a wide variety of identities, languages and poses. For AVSpeech data, we extract the face tracks using an off-the-shelf face detector~\cite{S3FDFaceDet_iccv_2017}. We curate a set of $50$ hours for training and $\sim3$ hours from the official test split for testing and verified it for accurate lip-sync using SyncNet~\cite{syncnet_2016}. We also benchmark our model on VoxCeleb2 data which comprises face tracks with a fair amount of background. Owing to computational limitations, we randomly sample a subset of $100$ hours for training and use the full official test split for testing. Note that there are no overlaps between the identities used in training and testing sets in both datasets. The code, models and file-lists are released on our website for reproducibility and future research.


\vspace{-3pt}
\section{Experiments}
\label{sec:results}


\subsection{Extreme-scale Super-Resolution}


\begin{table*}[ht]
    \centering
    \setlength{\tabcolsep}{8pt}
    \caption{Quantitative comparison for $32\times$ SR on AVSpeech~\cite{avspeech_2018_tog} and VoxCeleb2~\cite{voxceleb2_2018_interspeech} datasets. Our method outperforms the baselines by a significant margin across all metrics. Note that the baselines have also been trained with a single identity image.}

    \vspace{-10pt}
    \begin{tabular}{l|ccccc||ccccc}
    \hline

    \textbf{Dataset} & \multicolumn{5}{c||}{\textbf{AVSpeech}~\cite{avspeech_2018_tog}} & \multicolumn{5}{c}{\textbf{VoxCeleb2}~\cite{voxceleb2_2018_interspeech}}   \\
    \hline
    
    \textbf{Method} & 
    \textbf{PSNR}$\uparrow$ & \textbf{SSIM}$\uparrow$ & \textbf{FID}$\downarrow$ & \textbf{LMD}$\downarrow$ & \textbf{LSE-D}$\downarrow$ & 
    \textbf{PSNR}$\uparrow$ & \textbf{SSIM}$\uparrow$ & \textbf{FID}$\downarrow$ & \textbf{LMD}$\downarrow$ & \textbf{LSE-D}$\downarrow$ \\
     
    \hline
    
    Bicubic & 22.33 & 0.60 & 102.41 & 0.246 & 14.18 & 22.16 & 0.60 & 105.14 & 0.255 & 17.83\\
    
    SPARNet~\cite{SPARNet} & 23.17 & 0.68 & 92.14 & 0.201 & 12.87 & 22.98 & 0.67 & 83.01 & 0.228 & 14.07\\  
    
    TecoGAN~\cite{tecogan} & 19.26 & 0.62 & 84.73 & 0.213 & 13.01 & 16.91 & 0.54 & 82.19 & 0.234 & 14.12\\    

    \hline
    
    \textbf{Ours} & \textbf{25.06} & \textbf{0.73} & \textbf{11.54} & \textbf{0.162} & \textbf{12.43} & \textbf{24.95} & \textbf{0.71} & \textbf{14.10} & \textbf{0.196} & \textbf{13.91}\\
    
    \hline
    
    \end{tabular}
    \vspace{-5pt}
    \label{table:metrics}
\end{table*}


\begin{figure*}[ht]
  \includegraphics[width=\textwidth]{images/comparison_avspeech.png}
  \vspace{-20pt}
  \caption{Qualitative comparisons on AVSpeech dataset~\cite{avspeech_2018_tog}. Our method  captures the rich identity-specific attributes like eyeballs, hair strands, face texture and lip shape, far better compared to the existing approaches.}
  \label{fig:comparison}
  \vspace{-10pt}
\end{figure*}


\begin{figure*}[ht]
  \includegraphics[width=\textwidth]{images/comparison_vox2.png}
  \vspace{-20pt}
  \caption{Qualitative comparisons on the models trained on VoxCeleb2 dataset~\cite{voxceleb2_2018_interspeech}. Our method surpasses the existing baselines in generating the outputs that accurately match the ground-truth identity.}
  \label{fig:comparison_vox2}
  \vspace{-10pt}
\end{figure*}

\noindent
\textbf{Baselines:}
The state-of-the-art works in video SR literature super-resolve up to a scale-factor of $4\times$. We thus re-train the existing state-of-the-art video SR method, TecoGAN~\cite{tecogan} at a scale-factor of $32\times$ on the same training dataset as ours. We extend the existing face SR approach, SPARNet~\cite{SPARNet} to work for video SR by appropriately modifying the architecture (ingest a window of $5$ frames) and train using the same settings as mentioned above. These methods originally do not consider a HR target identity as input; thus, it would be unfair to compare them without the identity information. Hence we provide a HR target identity image to these models in a manner typically used in talking-face generation methods: concatenating it channel-wise with the input.


\noindent
\textbf{Metrics:} 
We evaluate our SR model on: (i) PSNR, (ii) SSIM, (iii) Fréchet Inception Distance (FID)~\cite{heusel2018gans}, (iv) Landmark Distance (LMD) \cite{fan_landmark_iccv_2017} and (v) Lip-Sync Error Distance (LSE-D)~\cite{wav2lip}. More details about the metrics can be found in the supplementary file.


\noindent
\textbf{Results:} 
We compare our results with existing SR approaches at extreme scale-factor of $32\times$ in Table~\ref{table:metrics}. As we can see from the table, our method outperforms the existing works by a significant margin on both AVSpeech~\cite{avspeech_2018_tog} and VoxCeleb2~\cite{voxceleb2_2018_interspeech} datasets. None of the current techniques match the ground-truth identity (measured using PSNR) and perceptual quality (measured using FID) of our generations. The LSE-D metric indicates that our method achieves accurate lip-synchronization with audio, thus validating our claim that the audio signal enables us to generate far more accurate lip shapes than the competing methods. Our method also surpasses the existing approaches in preserving the overall face structure (measured using SSIM and LMD). 

Figures~\ref{fig:comparison} and~\ref{fig:comparison_vox2} show the qualitative comparisons. We can clearly observe that our models generate results with far fewer artifacts and captures rich, fine-grained details. Although all the comparison methods consider the HR target identity as input, they do not match the quality of our generations. This shows that our overall network design is highly effective in making use of the available target identity image. In the examples, we can also see the diverse range of our models' generative capabilities: eyeballs with precise eye color (Fig.~\ref{fig:comparison}: row $1$), microphone (Fig.~\ref{fig:comparison}: row $2$), hair strands (Fig.~\ref{fig:comparison}: row $2$), lip shape (Fig.~\ref{fig:comparison}: rows $2$,$3$ and Fig.~\ref{fig:comparison_vox2}: rows $1$,$2$), face texture such as wrinkles (Fig.~\ref{fig:comparison}: row $3$), beard (Fig.~\ref{fig:comparison_vox2}: rows $1$,$2$). More visual examples can be found on our project page.

\vspace{3pt}
\noindent
\textbf{Ablation study:}
We validate the design choices of our network by analyzing the importance of audio signal, landmark-based region loss, the use of different target identity images and several other additional experiments, along with human evaluations in our supplementary. We also compare the performance of different models at multiple scale-factors like $4\times$, $8\times$, $16\times$ and $32\times$ in supplementary.


\subsection{Talking-Face Video Compression}

One of the major applications of our system is in compressing talking-face videos to reduce the bandwidth in video conferencing applications. We can transmit the LR frames ($8\times8$ pixels) along with the audio signal on the sender's side and the receiver can reconstruct the high-quality video ($256\times256$ pixels) using a single HR target identity image. A sample video calling demo is illustrated in Figure~\ref{fig:vidcall_demo}. We assume that a single target identity image can be sent at the beginning (e.g., $1^{st}$ frame) and hence does not consume additional bandwidth. Note that this is very different from the standard codecs, where full resolution I-frames are transmitted at regular intervals. Also, since the audio signal is always accompanied in a video call, we do not consider this as an extra overhead. 



\vspace{-2pt}
\begin{table}[t]
    \centering
    \setlength{\tabcolsep}{2pt}
    \caption{Quantitative comparison for talking-face video compression on VoxCeleb2 dataset~\cite{voxceleb2_2018_interspeech}. We achieve the best trade-off in terms of quality versus compression ratio. Our method achieves the lowest FID (indicates very high perceptual quality) and a very low/comparable BPP.} 
    
    \vspace{-10pt}
    \begin{tabular}{l|cccc}
    \hline

    \textbf{Method} & \textbf{BPP}$\downarrow$ & \textbf{PSNR}$\uparrow$ & \textbf{SSIM}$\uparrow$& \textbf{FID}$\downarrow$\\
    \hline
    H.264 (CRF=23) (min. compression) & 0.109 & 32.96 & 0.79 & 9.75 \\  
    \hline
    H.264 (CRF=36) & 0.027 & 19.24 & 0.67 & 30.12 \\
    H.266 & 0.0076 & 23.27 & 0.70 & 58.32 \\
    \hline

    fs-vid2vid~\cite{fsvid2vid_nips_2019} & n/a & 20.36 & 0.71 & 85.76\\
    
    os-synth~\cite{nvidia_compression_2021_cvpr} & 0.016 & 24.37 & \textbf{0.80} & 69.13\\
    
    \hline
    \textbf{Ours} & 0.023 & \textbf{24.95} & 0.71 & \textbf{14.10}\\
    \textbf{Ours (Frame-Interpolation)} & \textbf{0.0046} & 23.72 & 0.68 & 14.51 \\

    \hline
    \end{tabular}
    \vspace{-15pt}
    \label{table:compression}
\end{table}


\vspace{5pt}
\noindent
\textbf{Baselines:} 
We benchmark our model's capability using the existing talking-face video compression methods: few-shot vid2vid (fs-vid2vid)~\cite{fsvid2vid_nips_2019}, one-shot free-view synthesis (os-synth)~\cite{nvidia_compression_2021_cvpr} and the standard codecs: (i) H.264 (with CRF of $23$ and $36$) and (ii) H.266 (implemented using vvenc: \url{https://github.com/fraunhoferhhi/vvenc}). Since we train and evaluate on the same dataset (VoxCeleb2~\cite{voxceleb2_2018_interspeech}), we directly take the scores reported in os-synth~\cite{nvidia_compression_2021_cvpr} for comparison.   

\noindent
\textbf{Metrics:}
We compare the compression factor using the standard bits-per-pixels (BPP) metric and measure the reconstruction quality using PSNR, SSIM and FID metrics. 

\begin{figure}[t]
  \includegraphics[width=\linewidth]{images/demo_call.png}
  \vspace{-15pt}
  \caption{Illustration of low-bandwidth video calling enabled by our system. Note that HR frames shown at both the sender's end are taken from an actual video call recording (credits: \url{https://www.youtube.com/watch?v=lQJD8RAq3lY}). } 
  \label{fig:vidcall_demo}
  \vspace{-20pt}
\end{figure}

\begin{figure*}[ht]
  \includegraphics[width=\linewidth]{images/limitations.png}
  \vspace{-20pt}
  \caption{Failure cases: (a) Drastic color changes, (b) Extreme variations of facial expressions and (c) Rapid head motions.}
  \label{fig:limitations}
  \vspace{-10pt}
\end{figure*}


\noindent
\textbf{Results:}
Table~\ref{table:compression} shows the comparison of our approach with the competing methods. We calculate the average BPP across all test videos for H.264 and H.266 codecs. The number of bits required by the current state-of-the-art ``os-synth''~\cite{nvidia_compression_2021_cvpr} to represent a $256\times256$ image is $1056$ (20 keypoints: $(20\times6 + 12)\times8$). Our approach requires $1536$ bits ($8\times8\times3\times8$), with a BPP of $0.023$. The BPP obtained using our method beats H.264 and is comparable to os-synth~\cite{nvidia_compression_2021_cvpr}. In terms of perceptual quality, we can observe that there is a trade-off in compression factor v/s quality for H.264, (i) CRF of $23$: Higher quality, with very less compression and (ii) CRF of $36$: Better compression, but poor reconstruction. While H.266 achieves comparable PSNR and SSIM measures with a BPP of $0.0076$, it still lags behind perceptually as shown by the FID metric. In contrast, our method is able to generate higher quality videos (remarkably low FID), with a better/comparable compression factor. 

\noindent
\textbf{Computation Comparison:}
Our network has $\sim20\%$ fewer parameters ($143$M) while being $2\times$ faster ($50$FPS) than state-of-the-art compression method (os-synth) on an NVIDIA 2080Ti GPU. Previous talking-face generation works focus on obtaining plausible faces, whereas our aim is to develop the first architecture to reconstruct the specific-identity by preserving most of the actual identity details; future works can build upon this to achieve optimal design. Further, model optimization techniques can be applied to the current design to make it suitable for mobile hardware deployment. 


\vspace{3pt}
\noindent
\textbf{Frame-Interpolation Network:}
To further reduce the bandwidth consumption, we develop a baseline frame-interpolation network with an aim to upsample $5$FPS videos to $25$FPS. While such networks have been studied in literature~\cite{super-slowmo}, most of them restrict the amount of upsampling, mainly due to the computational resources involved in training the bulky networks (often consisting of $3$D CNNs). In contrast, we take the advantage of LR videos and design a model to increase the temporal resolution of frames ($5$FPS to $25$FPS) for the first time. Our encoder-decoder based model processes $5$FPS LR frames ($5,3,8,8$) and upsamples it $5\times$ to generate $25$FPS LR frames as output ($25,3,8,8$). We refer the reader to our supplementary for more details. The lower resolution of both the input and output allows us to train a very light $3$D CNN model with only $0.2M$ parameters. This strategy effectively permits us to transmit $5$FPS LR videos. On receiving these frames, frame-interpolation network initially upsamples them to $25$FPS LR videos, which can subsequently be ingested by our spatial talking-face video upsampling network to render the final HR video. As shown in Table~\ref{table:compression}, our frame-interpolation network achieves a further reduction in BPP, since only $1$ in $5$ LR frames needs to be transferred. We thus obtain $\sim6\times$ and $\sim25\times$ reduction in bandwidth compared to os-synth~\cite{nvidia_compression_2021_cvpr} and H.264 codec (with CRF $23$) respectively, without a significant compromise in the generated quality. 



\vspace{-5pt}
\subsection{Audio-driven Talking-Face Generation (A2TF) and Face Re-enactment (FR)}
\label{section:A2TH_comparison}


\noindent
\textbf{Baselines:}
For A2TF, we compare with Wav2Lip~\cite{wav2lip} and MakeItTalk \cite{Yang:2020:MakeItTalk}. Talking-face videos are generated using the audio segment and the first frame of the original HD video as inputs (same strategy as our model). For FR, the original models take the actual HD frames as the driving video input, however, we do not have access to the actual HD video in our model. Thus, we upsample the LR input using the existing video SR method, TecoGAN~\cite{tecogan} (trained on VoxCeleb2 dataset) and use it as our input driving video, along with the first frame of the original HD video as target identity input. 

\noindent
\textbf{Metrics:}
Along with the standard metrics, we specifically evaluate the ability of models to match the original identity using head pose estimation metrics~\cite{LwPosr_wacv_2022, 6DHeadPose_arxiv_2022}. We convert the rotation matrix\footnote{\url{https://learnopencv.com/head-pose-estimation-using-opencv-and-dlib/}} to Euler angles and report Mean Absolute Error (MAE) of these angles. 

\noindent
\textbf{Results:}
Table~\ref{table:a2tf_fr_comparison} shows the performance of different A2TF and FR methods in comparison to our approach. A2TF models generate accurate lip-sync (LSE-D), but they fail to match the pose of the original identity (as noted by Yaw, Pitch, Roll and MAE metrics). FR methods also face similar issues and do not match the identity, as they do not have the actual HD video as driving frames. Our model is clearly the most applicable for this task, is consistent across all metrics and produces noticeably better results. Most importantly, our model preserves the identity to the maximum extent. 

\vspace{-9pt}
\begin{table}[ht]
    \centering
    \footnotesize
    \setlength{\tabcolsep}{1.2pt}
    \caption{Quantitative comparison with A2TF~\cite{wav2lip, Yang:2020:MakeItTalk} and FR~\cite{fomm_nips_2019, PC_AVS_2021_cvpr} methods on VoxCeleb2 test set.}

    \vspace{-10pt}
    \begin{tabular}{l|ccccccccccc}
    \hline
    
    \textbf{Method} & 
    \textbf{PSNR}$\uparrow$ & \textbf{SSIM}$\uparrow$ & \textbf{FID}$\downarrow$ & \textbf{LMD}$\downarrow$ & \textbf{LSE-D}$\downarrow$ & \textbf{Yaw}$\downarrow$ & \textbf{Pitch}$\downarrow$ & \textbf{Roll}$\downarrow$ & \textbf{MAE}$\downarrow$\\
     
    \hline
    
    Wav2Lip~\cite{wav2lip} & 14.18 & 0.32 & \textbf{8.15} & 4.320 & 9.19 & 24.68 & 38.31 & 28.94 & 30.64 \\
    
    MakeItTalk~\cite{Yang:2020:MakeItTalk} & 18.88 & 0.49 & 31.19 & 2.012 & 11.91 & 26.29 & 40.13 & 31.42 & 32.61 \\
    
    \hline
    
    FOMM~\cite{fomm_nips_2019} & 20.14 & 0.56 & 21.18 & 0.864 & 14.03 & 19.14 & 30.57 & 22.76 & 24.35 \\
    
    PC-AVS~\cite{PC_AVS_2021_cvpr} & 15.68 & 0.37 & 33.38 & 1.063 & \textbf{8.42} & 22.27 & 31.89 & 25.80 & 26.65 \\
        
    \hline
    
    \textbf{Ours} & \textbf{24.95} & \textbf{0.71} & 14.10 & \textbf{0.196} & 13.91 & \textbf{13.55} & \textbf{21.01} & \textbf{15.48} & \textbf{16.68}\\

    \hline
    
    \end{tabular}
    \vspace{-10pt}
    \label{table:a2tf_fr_comparison}
\end{table} 




\vspace{-5pt}
\section{Limitations and Future Directions}

Although our method generates realistic results for a wide variety of inputs, there are certain situations as shown in Figure~\ref{fig:limitations} where our model results in sub-optimal generations. For example, if the color contrast changes drastically as the video progresses, the model fails to capture these details (Figure~\ref{fig:limitations} (a)). Significant variations in facial expressions is another case where our model struggles to replicate the details (Figure~\ref{fig:limitations} (b)). Explicitly handling the expressions is an interesting direction that can be investigated in the future, which we currently do not handle in our work. Finally, in the case of sudden/rapid changes in view, camera angle, or head movements, our model attempts to generate smooth transitions (Figure~\ref{fig:limitations} (c)). However, we found our method to be stable over a large variety of inputs and anticipate that our idea of utilizing extremely LR frames will be a basis for other domains and applications.


\vspace{-5pt}
\section{Conclusion}
\label{section:conclusion}

In this work, we present a novel framework for extreme-scale talking-face video super-resolution and compression. We show that by considering appropriate priors (audio signal and a single target identity image), we are able to generate realistic, high-quality talking-face videos from (very) low-resolution frames. Our method handles various inputs, including but not limited to people of different ages, gender and ethnicity. Most importantly, our framework is the first of its kind to produce photo-realistic lip-synced talking-face videos while also matching the actual identity. By dramatically reducing the bandwidth requirements, our approach can be utilized as a tool for a seamless video-conferencing experience. We believe our core idea of exploiting very low-resolution videos along with adequate priors will be an important step towards the future of super-resolution and low-bandwidth video-conferencing.

\bibliographystyle{ACM-Reference-Format}
\balance
