\section{Introduction}
\textit{\textbf{Motivation:}} With the ever-increasing demands for high-definition video streaming services, the needs for \emph{Video Quality Assessment}~(VQA) is growing potentially.
It plays an essential role in video processing from capturing to rendering, including compression, transmission, restoration, and display~\cite{VQ_survey}. With all the available encoding options and trade-offs to consider in \emph{HTTP Adaptive Streaming }~(HAS)~\cite{DASH_Survey}, having a lightweight and reliable VQA method is crucial. According to the degree of information available for the reference video signals, VQA is classified into \textit{full reference} (FR), \textit{reduced reference} (RR), and \textit{no reference} (NR) methods. NR-VQA methods are `blind' where the original video content is not used for the quality assessment, leading to an unreliable VQA~\cite{VQ_survey}. On the other hand, since RR-VQA methods use \textit{(i)} less overhead data compared to FR-based VQA approaches, and \textit{(ii)} are more reliable than NR-based VQA methods, they are employed in real-time scenarios~\cite{rr_vqa_survey}.

\textit{Peak Signal to Noise Ratio (PSNR)} continues to be the predominant industry benchmark for standardizing video codecs. PSNR is an effective method for generating a numeric value that compares an original input file and a coded output file. The limitations of PSNR are \textit{(i)} its failure to account for the temporal nature of compression artifacts and \textit{(ii)} lack of correlation between PSNR improvements and subjective quality, particularly in the presence of camera noise~\cite{psnr_ref1,psnr_ref2}. \textit{Structural Similarity (SSIM)} is a still image quality metric introduced in 2004~\cite{ssim_ref} which considers image degradation as perceived change in structural information while also incorporating critical perceptual phenomena, including both luminance masking and contrast masking terms. \textit{Video Multi-Method Assessment Fusion (VMAF)} was explicitly formulated by Netflix to correlate strongly with subjective Mean Opinion Scores (MOSs). Using machine learning techniques, a large sample of MOSs was used as ground truth to train a quality estimation model. Among the state-of-the-art VQA metrics, VMAF achieves the highest correlation with Difference Mean Opinion Score (DMOS); however, its computation time is very high compared to PSNR and SSIM metrics~\cite{vmaf_ref_madhu}. 

\textit{\textbf{Target:}} 
This paper targets a fast estimation of VMAF to facilitate low-latency VQA in video streaming applications. The expected computation time should be comparable with PSNR and SSIM, with the highest possible accuracy compared to the ground truth VMAF score. 

\begin{figure*}[t]
\centering
\begin{subfigure}{0.323\textwidth}
    \centering
    \includegraphics[width=\textwidth]{figures/Bitrate_PSNR_plot.pdf}
    \caption{}
        \label{fig:rd_psnr}
\end{subfigure}
\hfill
\begin{subfigure}{0.323\textwidth}
    \centering
    \includegraphics[width=\textwidth]{figures/Bitrate_SSIM_plot.pdf}
    \caption{}
        \label{fig:rd_ssim}    
\end{subfigure}
\hfill
\begin{subfigure}{0.323\textwidth}
    \centering
    \includegraphics[width=\textwidth]{figures/Bitrate_VMAF_plot.pdf}
    \caption{}
    \label{fig:rd_vmaf}    
\end{subfigure}
\vspace{-0.2cm}
\caption{Rate Distortion (RD) curves of selected segments of different spatiotemporal complexities -- \textit{Beauty\_s000} ($E=$59.90, $h$=17.49, $L$=89.25), \textit{Basketball\_s000} ($E=$15.30, $h$=12.59, $L$=108.18), \textit{Characters\_s000} ($E=$45.42, $h$=36.88, $L$=134.56), and \textit{Runners\_s000} ($E=$105.85, $h$=22.48, $L$=126.60) -- selected from the VCD dataset~\cite{VCD_ref}.  Distortion is represented by (a) PSNR, (b) SSIM, and (c) VMAF. The segments are encoded with the x264 AVC encoder using \textit{ultrafast} preset and CRF ratecontrol.}
\vspace{-0.2cm}
\label{fig:rd_plots}
\end{figure*}

\textit{\textbf{Contributions:}} 
The paper proposes a fast machine learning (ML)-based VQA model using texture information fusion (\texttt{VQ-TIF} ), which can be implemented in real-time to determine VMAF. The \texttt{VQ-TIF}  model determines VMAF by \emph{Discrete Cosine Transform }~(DCT)-energy-based texture information fusion and SSIM between the original and the reconstructed video. The model extracts video complexity features like brightness and spatio-temporal texture information from the luma channels of the videos. The extracted features are fused using a \emph{Long Short-Term Memory}  (LSTM)-based model to determine the VMAF score. \texttt{VQ-TIF} -based VMAF can be determined at a rate of 9.14 times faster than the state-of-the-art VMAF evaluation, with a \emph{Pearson Correlation Coefficient}  (PCC) of 0.96 with the ground truth VMAF values and 89.44\% reduction in energy consumption , assuming a UHD (2160p) display.

\section{Video quality metrics}
\textit{Peak Signal to Noise Ratio (PSNR)} is a conventional quality metric for signals and  expressed in decibels. PSNR is derived from the mean square error (MSE) or its square root (RMSE). The formula used is: $PSNR = 20 log_{10} \frac{Max}{RMSE}$, where the error is computed over all the pixels in the video to a reference video. 
SSIM computes a score for each pixel using a window of neighboring pixels~\cite{ssim_ref2}. These scores can then be averaged to produce a global score for the entire image to a reference image. The original metric~\cite{ssim_ref} produces scores ranging between zero and one; however, it is commonly expressed in a non-linear decibel scale, \emph{i.e.},  $-10 * log_{10} (1 - SSIM)$.

VMAF is a full-reference, perceptual video quality metric that aims to approximate human perception of video quality. This metric is focused on quality degradation due to compression and rescaling. VMAF estimates the perceived quality score by computing scores from multiple quality assessment algorithms and fusing them by a support vector machine (SVM). Currently, three image fidelity metrics and one temporal signal have been chosen as features to the SVM: \textit{(i)} Anti-noise SNR (AN-SNR), \textit{(ii)} Detail Loss Measure (DLM), \textit{(iii)} Visual Information Fidelity (VIF), and \textit{(iv)} Mean Co-Located Pixel Difference (MCPD). An essential feature is the MCPD of a frame to the previous frame (\emph{i.e.},  the temporal component). PSNR and SSIM metrics do not consider temporal information~\cite{ssim_ref3}. A VMAF score is simpler to understand because it operates in a linear range of 0 to 100, whereas PSNR is logarithmic. It considers scaling and compression artifacts and has a model trained for mobile video consumption~\cite{vmaf_comp}.

Pearson Correlation of PSNR, SSIM, and VMAF quality metrics is analyzed for five hundred video sequences of the Video Complexity Dataset (VCD)~\cite{VCD_ref} encoded at Ultra High Definition (2160p) resolution with the x264 AVC encoder -- using \textit{ultrafast} preset and CRF rate control, as shown in Table~\ref{tab:pcc_vqa}. CRF values ranging between 1 and 51 are used in the analysis. The correlation of the VMAF score with PSNR and SSIM scores are 0.83 and 0.88, respectively. The correlation can also be observed graphically in Fig.~\ref{fig:rd_plots}, which shows rate-distortion (RD) curves of selected video sequences based on their spatio-temporal complexity, where the distortion is measured using PSNR, SSIM, and VMAF.
\begin{table}[t]
\caption{Pearson Correlation of VQA metrics.}
\centering
\resizebox{0.615\linewidth}{!}{
\begin{tabular}{l|c|c|c}
\specialrule{.12em}{.05em}{.05em}
 & PSNR & SSIM & VMAF \\
\specialrule{.12em}{.05em}{.05em}
PSNR & 1.00 & 0.70 & 0.83 \\
SSIM & 0.70 & 1.00 & 0.88 \\
VMAF & 0.83 & 0.88 & 1.00 \\
\specialrule{.12em}{.05em}{.05em}
\end{tabular}}
\vspace{-1.25em}
\label{tab:pcc_vqa}
\end{table}

\begin{figure*}[t]
\centering
\includegraphics[width=1\linewidth]{figures/RRTIF_arch.pdf}
\caption{VMAF estimation for a video chunk using \texttt{VQ-TIF}  model envisioned in this paper.}
 \vspace{-0.8em}
\label{fig:rrtif_arch}
\end{figure*}
\vspace{-0.55em}
\section{\texttt{VQ-TIF}  model}
\label{sec:rrtif_framework}
The architecture of the proposed \texttt{VQ-TIF} -based VMAF estimation is illustrated in Fig.~\ref{fig:rrtif_arch}. Since the correlation between SSIM and VMAF is very high (\emph{cf.}\xspace Table~\ref{tab:pcc_vqa}), and the computation time of SSIM is significantly lower than VMAF, SSIM is used as a feature to compute VMAF. The input video segment is divided into multiple chunks. The first phase of the model is the frame-wise \textit{texture information extraction} for each chunk (explained in Section~\ref{sec:feature_extraction}) of the original and reconstructed video segment and SSIM calculation. The second phase of the model is the \textit{texture information fusion}, where the features and the computed SSIM are fused using an LSTM-based model to determine the VMAF score for each chunk (refer to Section~\ref{sec:tif}). The VMAF scores obtained for each chunk is averaged as the VMAF for the reconstructed video segment.

\subsection{Texture Information Extraction}
\label{sec:feature_extraction}
An intuitive feature extraction method would be utilizing Convolutional Neural Networks (CNNs)~\cite{3d_cnn_vqa_ref}. However, such models have several inherent disadvantages, such as higher training time, inference time, and storage requirements, which are impractical in streaming applications. Although CNN-based approaches could result in rich features, simpler models that yield a significant prediction performance are more suitable for video streaming. The popular state-of-the-art video complexity features are Spatial Information (SI) and Temporal Information (TI)~\cite{siti_itu_ref}. However, the correlation of SI and TI features with the encoding output features such as bitrate, encoding time, \emph{etc. } are very low, which is insufficient for encoding parameter prediction in streaming applications~\cite{vca_ref}.

In this paper, three DCT-energy-based features, the average luma texture energy $E$, the average gradient of the luma texture energy $h$, and the average luminescence $L$, which are extracted using VCA\footnote{\label{ref_vca}\href{https://vca.itec.aau.at}{https://vca.itec.aau.at}, last access: Feb 20, 2023.} open-source video complexity analyzer~\cite{vca_ref} are used as the texture information measures~\cite{coda_ref, opte_ref}.
\vspace{-0.5em}
\subsection{Texture Information Fusion}
\label{sec:tif}
The texture information fusion step of \texttt{VQ-TIF}  is accomplished using the following steps:

\textit{\textbf{Spatial pooling}:} The video segments are divided into $T$ chunks with a fixed number of frames (\emph{i.e.},  $f_c$) in each chunk. The averages of the $E$, $h$, and $L$ features of each frame in the chunk are computed to obtain the spatially pooled representation of the chunk, expressed as:
$X = \{x_{1}, x_{2},.., x_{f_c}\}$, and  $\hat{X} = \{\hat{x}_{1}, \hat{x}_{2},.., \hat{x}_{f_c}\}$, where, $x_{i}$ and $\hat{x}_{i}$ is the feature set of every $i^{th}$ frame of the original and reconstructed video chunks, respectively.
\begin{equation}
x_{i} = [E_{i}, h_{i}, L_{i}], \hat{x}_{i} = [\hat{E}_{i}, \hat{h}_{i}, \hat{L}_{i}] \hspace{0.4cm} \forall i \in [1,{f_c}]
\end{equation}

\textit{\textbf{Residual computation}:} Residual features are formed by subtracting the original video texture information features from the reconstructed video features. This difference is known as the error or residual feature, which is expressed as $r_{E_{i}} = E_{i} - \hat{E}_{i}$, $r_{h_{i}} = h_{i} - \hat{h}_{i}$, $r_{L_{i}} = L_{i} - \hat{L}_{i}$, respectively, where $i \in [1,{f_c}]$. The residual features usually have low information entropy since the original and reconstructed video frames are similar. 




\textit{\textbf{Fusion}:} The fusion of the texture information features is established using a \emph{Long Short-Term Memory}  (LSTM). LSTM is selected as a model for processing sequential data, making it suitable for combining information for temporally adjacent frames in a video. An advantage of LSTM models is that they can better handle long-term dependencies in long sequences. The videos are divided into several chunks, each of which consists of a fixed number of frames. Each chunk's feature averages are considered separate data points in the model training process.
Thus, the input data consist of the residuals of the spatially pooled luma texture information features extracted per frame of the video chunk. Additionally frame-wise SSIM values denoted by $S = \{s_{1}, s_{2},.., s_{f_c}\}$ are appended to the residual features. The prediction model is a function of the residual features and the SSIM values of the frames in a chunk, as shown in Eq.~\ref{eq:lstm_in}. This approach can fuse feature information from temporally adjacent frames to estimate VMAF. 
\begin{equation}
\label{eq:lstm_in}
\Tilde{x_{i}} = [r_{i} | s_{i}]^{T} \hspace{0.4cm} i \in [1, {f_c}]
\end{equation}
where $r_{i} = [r_{E_{i}}, r_{h_{i}}, r_{L_{i}}]$.
Estimated VMAF per chunk $\hat{v}$ can be presented as: $\hat{v} = f(\Tilde{x})$. The VMAF of the reconstructed video segment is the average of the $\hat{v}$ values estimated for every chunk.
\vspace{-0.9em}
\section{Evaluation}
\label{sec:evaluation}
\subsection{Test Methodology}

\begin{figure*}[t]
\centering
\begin{subfigure}{0.24\textwidth}
    \centering
    \includegraphics[width=\textwidth]{figures/rrtif_feature_importance.pdf}
    \caption{}
        \label{fig:rrtif_feat_imp}
\end{subfigure}
\hfill
\begin{subfigure}{0.24\textwidth}
    \centering
    \includegraphics[width=\textwidth]{figures/rrtif_pred_res_scatter.pdf}
    \caption{}
        \label{fig:rrtif_scatter_plot}
\end{subfigure}
\hfill
\begin{subfigure}{0.24\textwidth}
    \centering
    \includegraphics[width=\textwidth]{figures/rrtif_time_plot.pdf}
    \caption{}
        \label{fig:rrtif_time_plot}
\end{subfigure}
\hfill
\begin{subfigure}{0.24\textwidth}
    \centering
    \includegraphics[width=\textwidth]{figures/rrtif_energy_plot.pdf}
    \caption{}
    \label{fig:rrtif_energy_plot}    
\end{subfigure}
\caption{(a) Univariate feature importance of the LSTM model, (b) scatterplot of the ground truth and estimated VMAF scores, (c) total processing time, $\tau_T$ of VQA methods, and (d) total energy consumed, $J_T$ for VQA methods.}
\vspace{-0.4cm}
\label{fig:rrtif_time_energy_plot}
\end{figure*}


In this paper, 75\% of the five hundred UHD video sequences (segments) of the Video Complexity Dataset (VCD)~\cite{VCD_ref} are used as the training dataset, 5\% are set as the validation set and the remaining 20\%  as the test dataset. All experiments are run on a computer with Intel i7-11370H processor and 16GB RAM. The videos are encoded using the x264 AVC encoder with constant rate factor (CRF) values between 1 and 51 to induce different quality distortions, and the corresponding VMAF is evaluated. A chunk comprises eight frames, \emph{i.e.},  $f_{c}=8$. Thus, a video segment is divided into 15 chunks. The original and reconstructed video segments' luma texture features are extracted with the VCA v2.0 open-source video complexity analyzer running with eight CPU threads. The original and reconstructed video feature extraction process is implemented concurrently, with 4 CPU threads for each process. Keras~\cite{keras_ref} is used as the machine learning framework to implement the model.


The importance of the features input to the LSTM-based fusion model is analyzed using the univariate approach, wherein for each feature, all the other features are kept constant, and the MAE is computed, and this is subtracted from the MAE of the model with all features intact, which gives a measure of the decrease in accuracy (\emph{i.e.},  increase in error) when that feature is removed from the model~\cite{feature_imp}. Subsequently, the absolute value of the decrease in accuracy is computed and normalized to obtain the importance score, where higher absolute scores indicate more critical features.
Pearson’s correlation coefficient (PCC) and Mean Absolute Error (MAE) scores are analyzed between the \texttt{VQ-TIF} -based VMAF scores and the ground truth VMAF quality scores\footnote{\href{https://github.com/Netflix/vmaf}{https://github.com/Netflix/vmaf}, last access: Feb 20, 2023}.  Furthermore, $\tau_T$ and $J_T$, \emph{i.e.},  the total time taken and energy consumed to compute the quality metrics, are evaluated. The energy consumption is measured using \textit{codecarbon}\footnote{\href{https://codecarbon.io/}{https://codecarbon.io/}, last access: Feb 20, 2023}.
\subsection{Experimental Results}
\vspace{-0.3em}
\quad\textit{Relevance of features:}
The importance of the features is visualized in Fig.~\ref{fig:rrtif_feat_imp}. It is observed that the SSIM feature contributes the most to the \texttt{VQ-TIF} -based VMAF estimation, followed by $r_E$, $r_h$, and $r_L$ features.

\textit{Accuracy:} Fig~\ref{fig:rrtif_scatter_plot} shows the scatterplot of the \texttt{VQ-TIF} -based VMAF scores and the ground truth VMAF scores. A strong correlation between the scores is observed. Moreover, PCC and MAE of the \texttt{VQ-TIF} -based scores with the ground truth VMAF score are evaluated. The average PCC of the \texttt{VQ-TIF}  based scores to the ground truth VMAF score in the evaluation dataset is 0.96, while MAE is 2.71. In the current industry practice, \textit{Just Noticeable Difference} (JND) between quality levels is 6 VMAF points\footnote{\href{https://streaminglearningcenter.com/codecs/finding-the-just-noticeable-difference-with-netflix-vmaf.html}{https://streaminglearningcenter.com/codecs/finding-the-just-noticeable-difference-with-netflix-vmaf.html}, last access: Feb 20, 2023.}. In this light, \texttt{VQ-TIF} -based VMAF does not yield any significant error.

\textit{Processing time and energy:} The computation time and energy consumed for texture information extraction in \texttt{VQ-TIF}  is observed as 0.67 s (\emph{i.e.},  179.10 fps) and 17.18 $\mu$J, respectively (\emph{cf.}\xspace Fig.~\ref{fig:rrtif_time_plot} and ~\ref{fig:rrtif_energy_plot}). The time taken and energy consumed for SSIM computation are 0.85 s and 22.01 $\mu$J, respectively. The time taken and the energy consumed for the texture information fusion are 0.07 s and 1.83 $\mu$J, respectively. Hence, the total processing time, $\tau_{T}$ is 1.59 s (\emph{i.e.},  75.47 fps), while the processing time of state-of-the-art VMAF computation is 14.52 s (\emph{i.e.},  8.26 fps). The computation speed of \texttt{VQ-TIF}  is 9.14 times higher than the state-of-the-art VMAF evaluation. Furthermore, in terms of the total energy consumed, \texttt{VQ-TIF}  consumes 89.44\% less energy than the state-of-the-art VMAF implementation.
\vspace{-0.7em}
\section{Conclusions}
\label{sec:conclusion_future_dir}
\vspace{-0.4em}
This paper proposed \texttt{VQ-TIF} , a fast and accurate reduced-reference video quality assessment (RR-VQA) method based on texture information fusion. \texttt{VQ-TIF}  includes DCT-energy-based video complexity feature extraction where features representing the luma texture and temporal activity are extracted from the original and the reconstructed video segments. The extracted texture information is fused using an LSTM-based model to determine the \texttt{VQ-TIF} -based VMAF. It is observed that \texttt{VQ-TIF} -based VMAF is determined at a speed of 9.14 times faster than the state-of-the-art VMAF implementation for Ultra HD (2160p) videos, consuming 89.44\% less energy. At the same time, \texttt{VQ-TIF} -based VMAF scores yield a PCC of 0.96 and MAE of 2.71 compared to the ground truth VMAF.
\vspace{-0.3em}
\section{Acknowledgment}
\vspace{-0.3em}
The financial support of the Austrian Federal Ministry for Digital and Economic Affairs, the National Foundation for Research, Technology and Development, and the Christian Doppler Research Association is gratefully acknowledged. Christian Doppler Laboratory ATHENA: \small{\url{https://athena.itec.aau.at/}}.
\balance
\newpage
\bibliographystyle{IEEEtran}

\section{Introduction}
\textit{\textbf{Motivation:}} With the ever-increasing demands for high-definition video streaming services, the needs for \emph{Video Quality Assessment}~(VQA) is growing potentially.
It plays an essential role in video processing from capturing to rendering, including compression, transmission, restoration, and display~\cite{VQ_survey}. With all the available encoding options and trade-offs to consider in \emph{HTTP Adaptive Streaming }~(HAS)~\cite{DASH_Survey}, having a lightweight and reliable VQA method is crucial. According to the degree of information available for the reference video signals, VQA is classified into \textit{full reference} (FR), \textit{reduced reference} (RR), and \textit{no reference} (NR) methods. NR-VQA methods are `blind' where the original video content is not used for the quality assessment, leading to an unreliable VQA~\cite{VQ_survey}. On the other hand, since RR-VQA methods use \textit{(i)} less overhead data compared to FR-based VQA approaches, and \textit{(ii)} are more reliable than NR-based VQA methods, they are employed in real-time scenarios~\cite{rr_vqa_survey}.

\textit{Peak Signal to Noise Ratio (PSNR)} continues to be the predominant industry benchmark for standardizing video codecs. PSNR is an effective method for generating a numeric value that compares an original input file and a coded output file. The limitations of PSNR are \textit{(i)} its failure to account for the temporal nature of compression artifacts and \textit{(ii)} lack of correlation between PSNR improvements and subjective quality, particularly in the presence of camera noise~\cite{psnr_ref1,psnr_ref2}. \textit{Structural Similarity (SSIM)} is a still image quality metric introduced in 2004~\cite{ssim_ref} which considers image degradation as perceived change in structural information while also incorporating critical perceptual phenomena, including both luminance masking and contrast masking terms. \textit{Video Multi-Method Assessment Fusion (VMAF)} was explicitly formulated by Netflix to correlate strongly with subjective Mean Opinion Scores (MOSs). Using machine learning techniques, a large sample of MOSs was used as ground truth to train a quality estimation model. Among the state-of-the-art VQA metrics, VMAF achieves the highest correlation with Difference Mean Opinion Score (DMOS); however, its computation time is very high compared to PSNR and SSIM metrics~\cite{vmaf_ref_madhu}. 

\textit{\textbf{Target:}} 
This paper targets a fast estimation of VMAF to facilitate low-latency VQA in video streaming applications. The expected computation time should be comparable with PSNR and SSIM, with the highest possible accuracy compared to the ground truth VMAF score. 

\begin{figure*}[t]
\centering
\begin{subfigure}{0.323\textwidth}
    \centering
    \includegraphics[width=\textwidth]{figures/Bitrate_PSNR_plot.pdf}
    \caption{}
        \label{fig:rd_psnr}
\end{subfigure}
\hfill
\begin{subfigure}{0.323\textwidth}
    \centering
    \includegraphics[width=\textwidth]{figures/Bitrate_SSIM_plot.pdf}
    \caption{}
        \label{fig:rd_ssim}    
\end{subfigure}
\hfill
\begin{subfigure}{0.323\textwidth}
    \centering
    \includegraphics[width=\textwidth]{figures/Bitrate_VMAF_plot.pdf}
    \caption{}
    \label{fig:rd_vmaf}    
\end{subfigure}
\vspace{-0.2cm}
\caption{Rate Distortion (RD) curves of selected segments of different spatiotemporal complexities -- \textit{Beauty\_s000} ($E=$59.90, $h$=17.49, $L$=89.25), \textit{Basketball\_s000} ($E=$15.30, $h$=12.59, $L$=108.18), \textit{Characters\_s000} ($E=$45.42, $h$=36.88, $L$=134.56), and \textit{Runners\_s000} ($E=$105.85, $h$=22.48, $L$=126.60) -- selected from the VCD dataset~\cite{VCD_ref}.  Distortion is represented by (a) PSNR, (b) SSIM, and (c) VMAF. The segments are encoded with the x264 AVC encoder using \textit{ultrafast} preset and CRF ratecontrol.}
\vspace{-0.2cm}
\label{fig:rd_plots}
\end{figure*}

\textit{\textbf{Contributions:}} 
The paper proposes a fast machine learning (ML)-based VQA model using texture information fusion (\texttt{VQ-TIF} ), which can be implemented in real-time to determine VMAF. The \texttt{VQ-TIF}  model determines VMAF by \emph{Discrete Cosine Transform }~(DCT)-energy-based texture information fusion and SSIM between the original and the reconstructed video. The model extracts video complexity features like brightness and spatio-temporal texture information from the luma channels of the videos. The extracted features are fused using a \emph{Long Short-Term Memory}  (LSTM)-based model to determine the VMAF score. \texttt{VQ-TIF} -based VMAF can be determined at a rate of 9.14 times faster than the state-of-the-art VMAF evaluation, with a \emph{Pearson Correlation Coefficient}  (PCC) of 0.96 with the ground truth VMAF values and 89.44\% reduction in energy consumption , assuming a UHD (2160p) display.

\section{Video quality metrics}
\textit{Peak Signal to Noise Ratio (PSNR)} is a conventional quality metric for signals and  expressed in decibels. PSNR is derived from the mean square error (MSE) or its square root (RMSE). The formula used is: $PSNR = 20 log_{10} \frac{Max}{RMSE}$, where the error is computed over all the pixels in the video to a reference video. 
SSIM computes a score for each pixel using a window of neighboring pixels~\cite{ssim_ref2}. These scores can then be averaged to produce a global score for the entire image to a reference image. The original metric~\cite{ssim_ref} produces scores ranging between zero and one; however, it is commonly expressed in a non-linear decibel scale, \emph{i.e.},  $-10 * log_{10} (1 - SSIM)$.

VMAF is a full-reference, perceptual video quality metric that aims to approximate human perception of video quality. This metric is focused on quality degradation due to compression and rescaling. VMAF estimates the perceived quality score by computing scores from multiple quality assessment algorithms and fusing them by a support vector machine (SVM). Currently, three image fidelity metrics and one temporal signal have been chosen as features to the SVM: \textit{(i)} Anti-noise SNR (AN-SNR), \textit{(ii)} Detail Loss Measure (DLM), \textit{(iii)} Visual Information Fidelity (VIF), and \textit{(iv)} Mean Co-Located Pixel Difference (MCPD). An essential feature is the MCPD of a frame to the previous frame (\emph{i.e.},  the temporal component). PSNR and SSIM metrics do not consider temporal information~\cite{ssim_ref3}. A VMAF score is simpler to understand because it operates in a linear range of 0 to 100, whereas PSNR is logarithmic. It considers scaling and compression artifacts and has a model trained for mobile video consumption~\cite{vmaf_comp}.

Pearson Correlation of PSNR, SSIM, and VMAF quality metrics is analyzed for five hundred video sequences of the Video Complexity Dataset (VCD)~\cite{VCD_ref} encoded at Ultra High Definition (2160p) resolution with the x264 AVC encoder -- using \textit{ultrafast} preset and CRF rate control, as shown in Table~\ref{tab:pcc_vqa}. CRF values ranging between 1 and 51 are used in the analysis. The correlation of the VMAF score with PSNR and SSIM scores are 0.83 and 0.88, respectively. The correlation can also be observed graphically in Fig.~\ref{fig:rd_plots}, which shows rate-distortion (RD) curves of selected video sequences based on their spatio-temporal complexity, where the distortion is measured using PSNR, SSIM, and VMAF.
\begin{table}[t]
\caption{Pearson Correlation of VQA metrics.}
\centering
\resizebox{0.615\linewidth}{!}{
\begin{tabular}{l|c|c|c}
\specialrule{.12em}{.05em}{.05em}
 & PSNR & SSIM & VMAF \\
\specialrule{.12em}{.05em}{.05em}
PSNR & 1.00 & 0.70 & 0.83 \\
SSIM & 0.70 & 1.00 & 0.88 \\
VMAF & 0.83 & 0.88 & 1.00 \\
\specialrule{.12em}{.05em}{.05em}
\end{tabular}}
\vspace{-1.25em}
\label{tab:pcc_vqa}
\end{table}

\begin{figure*}[t]
\centering
\includegraphics[width=1\linewidth]{figures/RRTIF_arch.pdf}
\caption{VMAF estimation for a video chunk using \texttt{VQ-TIF}  model envisioned in this paper.}
 \vspace{-0.8em}
\label{fig:rrtif_arch}
\end{figure*}
\vspace{-0.55em}
\section{\texttt{VQ-TIF}  model}
\label{sec:rrtif_framework}
The architecture of the proposed \texttt{VQ-TIF} -based VMAF estimation is illustrated in Fig.~\ref{fig:rrtif_arch}. Since the correlation between SSIM and VMAF is very high (\emph{cf.}\xspace Table~\ref{tab:pcc_vqa}), and the computation time of SSIM is significantly lower than VMAF, SSIM is used as a feature to compute VMAF. The input video segment is divided into multiple chunks. The first phase of the model is the frame-wise \textit{texture information extraction} for each chunk (explained in Section~\ref{sec:feature_extraction}) of the original and reconstructed video segment and SSIM calculation. The second phase of the model is the \textit{texture information fusion}, where the features and the computed SSIM are fused using an LSTM-based model to determine the VMAF score for each chunk (refer to Section~\ref{sec:tif}). The VMAF scores obtained for each chunk is averaged as the VMAF for the reconstructed video segment.

\subsection{Texture Information Extraction}
\label{sec:feature_extraction}
An intuitive feature extraction method would be utilizing Convolutional Neural Networks (CNNs)~\cite{3d_cnn_vqa_ref}. However, such models have several inherent disadvantages, such as higher training time, inference time, and storage requirements, which are impractical in streaming applications. Although CNN-based approaches could result in rich features, simpler models that yield a significant prediction performance are more suitable for video streaming. The popular state-of-the-art video complexity features are Spatial Information (SI) and Temporal Information (TI)~\cite{siti_itu_ref}. However, the correlation of SI and TI features with the encoding output features such as bitrate, encoding time, \emph{etc. } are very low, which is insufficient for encoding parameter prediction in streaming applications~\cite{vca_ref}.

In this paper, three DCT-energy-based features, the average luma texture energy $E$, the average gradient of the luma texture energy $h$, and the average luminescence $L$, which are extracted using VCA\footnote{\label{ref_vca}\href{https://vca.itec.aau.at}{https://vca.itec.aau.at}, last access: Feb 20, 2023.} open-source video complexity analyzer~\cite{vca_ref} are used as the texture information measures~\cite{coda_ref, opte_ref}.
\vspace{-0.5em}
\subsection{Texture Information Fusion}
\label{sec:tif}
The texture information fusion step of \texttt{VQ-TIF}  is accomplished using the following steps:

\textit{\textbf{Spatial pooling}:} The video segments are divided into $T$ chunks with a fixed number of frames (\emph{i.e.},  $f_c$) in each chunk. The averages of the $E$, $h$, and $L$ features of each frame in the chunk are computed to obtain the spatially pooled representation of the chunk, expressed as:
$X = \{x_{1}, x_{2},.., x_{f_c}\}$, and  $\hat{X} = \{\hat{x}_{1}, \hat{x}_{2},.., \hat{x}_{f_c}\}$, where, $x_{i}$ and $\hat{x}_{i}$ is the feature set of every $i^{th}$ frame of the original and reconstructed video chunks, respectively.
\begin{equation}
x_{i} = [E_{i}, h_{i}, L_{i}], \hat{x}_{i} = [\hat{E}_{i}, \hat{h}_{i}, \hat{L}_{i}] \hspace{0.4cm} \forall i \in [1,{f_c}]
\end{equation}

\textit{\textbf{Residual computation}:} Residual features are formed by subtracting the original video texture information features from the reconstructed video features. This difference is known as the error or residual feature, which is expressed as $r_{E_{i}} = E_{i} - \hat{E}_{i}$, $r_{h_{i}} = h_{i} - \hat{h}_{i}$, $r_{L_{i}} = L_{i} - \hat{L}_{i}$, respectively, where $i \in [1,{f_c}]$. The residual features usually have low information entropy since the original and reconstructed video frames are similar. 




\textit{\textbf{Fusion}:} The fusion of the texture information features is established using a \emph{Long Short-Term Memory}  (LSTM). LSTM is selected as a model for processing sequential data, making it suitable for combining information for temporally adjacent frames in a video. An advantage of LSTM models is that they can better handle long-term dependencies in long sequences. The videos are divided into several chunks, each of which consists of a fixed number of frames. Each chunk's feature averages are considered separate data points in the model training process.
Thus, the input data consist of the residuals of the spatially pooled luma texture information features extracted per frame of the video chunk. Additionally frame-wise SSIM values denoted by $S = \{s_{1}, s_{2},.., s_{f_c}\}$ are appended to the residual features. The prediction model is a function of the residual features and the SSIM values of the frames in a chunk, as shown in Eq.~\ref{eq:lstm_in}. This approach can fuse feature information from temporally adjacent frames to estimate VMAF. 
\begin{equation}
\label{eq:lstm_in}
\Tilde{x_{i}} = [r_{i} | s_{i}]^{T} \hspace{0.4cm} i \in [1, {f_c}]
\end{equation}
where $r_{i} = [r_{E_{i}}, r_{h_{i}}, r_{L_{i}}]$.
Estimated VMAF per chunk $\hat{v}$ can be presented as: $\hat{v} = f(\Tilde{x})$. The VMAF of the reconstructed video segment is the average of the $\hat{v}$ values estimated for every chunk.
\vspace{-0.9em}
\section{Evaluation}
\label{sec:evaluation}
\subsection{Test Methodology}

\begin{figure*}[t]
\centering
\begin{subfigure}{0.24\textwidth}
    \centering
    \includegraphics[width=\textwidth]{figures/rrtif_feature_importance.pdf}
    \caption{}
        \label{fig:rrtif_feat_imp}
\end{subfigure}
\hfill
\begin{subfigure}{0.24\textwidth}
    \centering
    \includegraphics[width=\textwidth]{figures/rrtif_pred_res_scatter.pdf}
    \caption{}
        \label{fig:rrtif_scatter_plot}
\end{subfigure}
\hfill
\begin{subfigure}{0.24\textwidth}
    \centering
    \includegraphics[width=\textwidth]{figures/rrtif_time_plot.pdf}
    \caption{}
        \label{fig:rrtif_time_plot}
\end{subfigure}
\hfill
\begin{subfigure}{0.24\textwidth}
    \centering
    \includegraphics[width=\textwidth]{figures/rrtif_energy_plot.pdf}
    \caption{}
    \label{fig:rrtif_energy_plot}    
\end{subfigure}
\caption{(a) Univariate feature importance of the LSTM model, (b) scatterplot of the ground truth and estimated VMAF scores, (c) total processing time, $\tau_T$ of VQA methods, and (d) total energy consumed, $J_T$ for VQA methods.}
\vspace{-0.4cm}
\label{fig:rrtif_time_energy_plot}
\end{figure*}


In this paper, 75\% of the five hundred UHD video sequences (segments) of the Video Complexity Dataset (VCD)~\cite{VCD_ref} are used as the training dataset, 5\% are set as the validation set and the remaining 20\%  as the test dataset. All experiments are run on a computer with Intel i7-11370H processor and 16GB RAM. The videos are encoded using the x264 AVC encoder with constant rate factor (CRF) values between 1 and 51 to induce different quality distortions, and the corresponding VMAF is evaluated. A chunk comprises eight frames, \emph{i.e.},  $f_{c}=8$. Thus, a video segment is divided into 15 chunks. The original and reconstructed video segments' luma texture features are extracted with the VCA v2.0 open-source video complexity analyzer running with eight CPU threads. The original and reconstructed video feature extraction process is implemented concurrently, with 4 CPU threads for each process. Keras~\cite{keras_ref} is used as the machine learning framework to implement the model.


The importance of the features input to the LSTM-based fusion model is analyzed using the univariate approach, wherein for each feature, all the other features are kept constant, and the MAE is computed, and this is subtracted from the MAE of the model with all features intact, which gives a measure of the decrease in accuracy (\emph{i.e.},  increase in error) when that feature is removed from the model~\cite{feature_imp}. Subsequently, the absolute value of the decrease in accuracy is computed and normalized to obtain the importance score, where higher absolute scores indicate more critical features.
Pearson’s correlation coefficient (PCC) and Mean Absolute Error (MAE) scores are analyzed between the \texttt{VQ-TIF} -based VMAF scores and the ground truth VMAF quality scores\footnote{\href{https://github.com/Netflix/vmaf}{https://github.com/Netflix/vmaf}, last access: Feb 20, 2023}.  Furthermore, $\tau_T$ and $J_T$, \emph{i.e.},  the total time taken and energy consumed to compute the quality metrics, are evaluated. The energy consumption is measured using \textit{codecarbon}\footnote{\href{https://codecarbon.io/}{https://codecarbon.io/}, last access: Feb 20, 2023}.
\subsection{Experimental Results}
\vspace{-0.3em}
\quad\textit{Relevance of features:}
The importance of the features is visualized in Fig.~\ref{fig:rrtif_feat_imp}. It is observed that the SSIM feature contributes the most to the \texttt{VQ-TIF} -based VMAF estimation, followed by $r_E$, $r_h$, and $r_L$ features.

\textit{Accuracy:} Fig~\ref{fig:rrtif_scatter_plot} shows the scatterplot of the \texttt{VQ-TIF} -based VMAF scores and the ground truth VMAF scores. A strong correlation between the scores is observed. Moreover, PCC and MAE of the \texttt{VQ-TIF} -based scores with the ground truth VMAF score are evaluated. The average PCC of the \texttt{VQ-TIF}  based scores to the ground truth VMAF score in the evaluation dataset is 0.96, while MAE is 2.71. In the current industry practice, \textit{Just Noticeable Difference} (JND) between quality levels is 6 VMAF points\footnote{\href{https://streaminglearningcenter.com/codecs/finding-the-just-noticeable-difference-with-netflix-vmaf.html}{https://streaminglearningcenter.com/codecs/finding-the-just-noticeable-difference-with-netflix-vmaf.html}, last access: Feb 20, 2023.}. In this light, \texttt{VQ-TIF} -based VMAF does not yield any significant error.

\textit{Processing time and energy:} The computation time and energy consumed for texture information extraction in \texttt{VQ-TIF}  is observed as 0.67 s (\emph{i.e.},  179.10 fps) and 17.18 $\mu$J, respectively (\emph{cf.}\xspace Fig.~\ref{fig:rrtif_time_plot} and ~\ref{fig:rrtif_energy_plot}). The time taken and energy consumed for SSIM computation are 0.85 s and 22.01 $\mu$J, respectively. The time taken and the energy consumed for the texture information fusion are 0.07 s and 1.83 $\mu$J, respectively. Hence, the total processing time, $\tau_{T}$ is 1.59 s (\emph{i.e.},  75.47 fps), while the processing time of state-of-the-art VMAF computation is 14.52 s (\emph{i.e.},  8.26 fps). The computation speed of \texttt{VQ-TIF}  is 9.14 times higher than the state-of-the-art VMAF evaluation. Furthermore, in terms of the total energy consumed, \texttt{VQ-TIF}  consumes 89.44\% less energy than the state-of-the-art VMAF implementation.
\vspace{-0.7em}
\section{Conclusions}
\label{sec:conclusion_future_dir}
\vspace{-0.4em}
This paper proposed \texttt{VQ-TIF} , a fast and accurate reduced-reference video quality assessment (RR-VQA) method based on texture information fusion. \texttt{VQ-TIF}  includes DCT-energy-based video complexity feature extraction where features representing the luma texture and temporal activity are extracted from the original and the reconstructed video segments. The extracted texture information is fused using an LSTM-based model to determine the \texttt{VQ-TIF} -based VMAF. It is observed that \texttt{VQ-TIF} -based VMAF is determined at a speed of 9.14 times faster than the state-of-the-art VMAF implementation for Ultra HD (2160p) videos, consuming 89.44\% less energy. At the same time, \texttt{VQ-TIF} -based VMAF scores yield a PCC of 0.96 and MAE of 2.71 compared to the ground truth VMAF.
\vspace{-0.3em}
\section{Acknowledgment}
\vspace{-0.3em}
The financial support of the Austrian Federal Ministry for Digital and Economic Affairs, the National Foundation for Research, Technology and Development, and the Christian Doppler Research Association is gratefully acknowledged. Christian Doppler Laboratory ATHENA: \small{\url{https://athena.itec.aau.at/}}.
\balance
\newpage
\bibliographystyle{IEEEtran}
