\section{Introduction}

After receiving paper reviews, authors may optionally submit a rebuttal to address the reviewers' comments, which will be limited to a {\bf one page} PDF file.
Please follow the steps and style guidelines outlined below for submitting your author response.

The author rebuttal is optional and, following similar guidelines to previous CVPR conferences, is meant to provide you with an opportunity to rebut factual errors or to supply additional information requested by the reviewers.
It is NOT intended to add new contributions (theorems, algorithms, experiments) that were absent in the original submission and NOT specifically requested by the reviewers.
You may optionally add a figure, graph, or proof to your rebuttal to better illustrate your answer to the reviewers' comments.

Per a passed 2018 PAMI-TC motion, reviewers should refrain from requesting significant additional experiments for the rebuttal or penalize for lack of additional experiments.
Authors should refrain from including new experimental results in the rebuttal, especially when not specifically requested to do so by the reviewers.
Authors may include figures with illustrations or comparison tables of results reported in the submission/supplemental material or in other papers.

Just like the original submission, the rebuttal must maintain anonymity and cannot include external links that reveal the author identity or circumvent the length restriction.
The rebuttal must comply with this template (the use of sections is not required, though it is recommended to structure the rebuttal for ease of reading).


\subsection{Response length}
Author responses must be no longer than 1 page in length including any references and figures.
Overlength responses will simply not be reviewed.
This includes responses where the margins and formatting are deemed to have been significantly altered from those laid down by this style guide.
Note that this \LaTeX\ guide already sets figure captions and references in a smaller font.

\section{Formatting your Response}

{\bf Make sure to update the paper title and paper ID in the appropriate place in the tex file.}

All text must be in a two-column format.
The total allowable size of the text area is $6\frac78$ inches (17.46 cm) wide by $8\frac78$ inches (22.54 cm) high.
Columns are to be $3\frac14$ inches (8.25 cm) wide, with a $\frac{5}{16}$ inch (0.8 cm) space between them.
The top margin should begin 1 inch (2.54 cm) from the top edge of the page.
The bottom margin should be $1\frac{1}{8}$ inches (2.86 cm) from the bottom edge of the page for $8.5 \times 11$-inch paper;
for A4 paper, approximately $1\frac{5}{8}$ inches (4.13 cm) from the bottom edge of the page.

Please number any displayed equations.
It is important for readers to be able to refer to any particular equation.

Wherever Times is specified, Times Roman may also be used.
Main text should be in 10-point Times, single-spaced.
Section headings should be in 10 or 12 point Times.
All paragraphs should be indented 1 pica (approx.~$\frac{1}{6}$ inch or 0.422 cm).
Figure and table captions should be 9-point Roman type as in \cref{fig:onecol}.


List and number all bibliographical references in 9-point Times, single-spaced,
at the end of your response.
When referenced in the text, enclose the citation number in square brackets, for example~\cite{Alpher05}.
Where appropriate, include the name(s) of editors of referenced books.

\begin{figure}[t]
  \centering
  \fbox{\rule{0pt}{0.5in} \rule{0.9\linewidth}{0pt}}
 
   \caption{Example of caption.  It is set in Roman so that mathematics
   (always set in Roman: $B \sin A = A \sin B$) may be included without an
   ugly clash.}
   \label{fig:onecol}
\end{figure}

To avoid ambiguities, it is best if the numbering for equations, figures, tables, and references in the author response does not overlap with that in the main paper (the reviewer may wonder if you talk about \cref{fig:onecol} in the author response or in the paper).
See \LaTeX\ template for a workaround.

\subsection{Illustrations, graphs, and photographs}

All graphics should be centered.
Please ensure that any point you wish to make is resolvable in a printed copy of the response.
Resize fonts in figures to match the font in the body text, and choose line widths which render effectively in print.
Readers (and reviewers), even of an electronic copy, may choose to print your response in order to read it.
You cannot insist that they do otherwise, and therefore must not assume that they can zoom in to see tiny details on a graphic.

When placing figures in \LaTeX, it is almost always best to use \verb+\includegraphics+, and to specify the  figure width as a multiple of the line width as in the example below
{\small\begin{verbatim}
   \usepackage{graphicx} ...
   \includegraphics[width=0.8\linewidth]
                   {myfile.pdf}
\end{verbatim}
}


{\small
\bibliographystyle{ieee_fullname}

\section{Introduction} \label{sec:intro}
\input{paper/1_intro}

\section{Related Works} \label{sec:related}
\input{paper/2_related}

\section{Methods}
\input{paper/3_methods}

\section{Experiments}
\input{paper/4_experiments}

\section{NTIRE 2022 Challenge}
\input{paper/5_ntire}

\section{Conclusion}
\input{paper/6_conclusion}

\section{Acknowledgement}
This work was supported by Institute of Information \& communications Technology Planning \& Evaluation (IITP) grant funded by the Korea government(MSIT) [NO.2021-0-01343, Artificial Intelligence Graduate School Program (Seoul National University)]

{
\newpage
\small
\bibliographystyle{ieee_fullname}


\subsection{Single Image Super Resolution}
Super-resolution has been studied long in computer vision fields. Before deep learning-based methods have been applied, sparsed coding\cite{dai2015jointlyRegressedRegressors, sun2012sceneMatching, yang2008sparseRepr, yang2010sparseRepr} and local linear regression\cite{timofte2013anchNeighReg, timofte2014A+, yang2013simpleFuncSR} have been highly applied. Many deep learning-based methods have been approached for SISR, since SRCNN \cite{wang2014srcnn} which exploited CNN layers and L1 Loss. After SRCNN was proposed, many variations have been suggested including \cite{wang2018esrgan}. But as CNN-based methods have relied on L1 or L2 loss, they have generated blurry images. GAN-based methods, which were first suggested by SRGAN\cite{ledig2017srgan}, have shown improvements by employing adversarial loss and perceptual loss. Although GAN-based methods have generated images with good quality\cite{ledig2017srgan, wang2018esrgan}, their diversities were so limited, thereby generating only one image.


\subsection{Normalizing Flow}
Flow-based models have been first proposed by \cite{dinh2014nice} for modeling complex high dimensional density. As flow-based models learn the whole distribution, they have been widely used for mapping complex distributions given simple distribution, including Gaussian distribution. Invertible neural networks have been adopted to map complex distributions from simple distributions\cite{dinh2014nice, dinh2017realnvp, kingma2018glow}. Flow-based models in the early days have not shown great improvements relative to GAN-based models. However, SRFlow\cite{lugmayr2020srflow}, which adopts negative log-likelihood loss, showed improvements in image quality and diversities simultaneously. As SRFlow used negative log-likelihood loss, it could learn the whole distribution, which leads to generating much more diverse images than GAN-based methods. NCSR \cite{kim2021noise} has shown further improvements in terms of image quality and diversity, by providing networks with noises. \cite{kim2021noise} has proposed adding a conditional noise layer, which essentially resolves distribution discrepancy between simple data and complex data.


\subsection{Frequency Separation}
The study of frequency domain based on Fast Fourier Transform (FFT) algorithm \cite{cooley1965algorithm} played a crucial role in traditional signal processing. In this perspective, before the era of deep learning, studying the frequency information was important in image restoration research. In this light, it is readily known that high-frequency information of the given image contributes greatly to its sharpness and high-quality detail. Therefore, we can say that a recent huge success of deep learning-based approaches in realistic images generation is due to the success of synthesizing high-frequency information of the desired images.

Therefore, in recent image restoration research including super-resolution, there exist approaches \cite{whang2021deblurring} in which low-frequency and high-frequency are separated and treated by a separate neural network, and approaches \cite{suvorov2021resolution} in which an FFT-based layer is designed to better process information of the frequency domain. 
We observed that when the former approaches were combined with NCSR, instability of the NLL training of the flow-based model occurred. 
And in the case of the latter approaches, the existing FFT-based layers are not suitable for the flow-based approach due to their non-invertible nature.
\subsection{Background}
Various model frameworks (\eg Generative Adversarial Networks \cite{goodfellow2014generative}, Normalizing Flow \cite{rezende2015variational}, and Diffusion probabilistic models \cite{ho2020denoising}) have been proposed in recent deep learning-based generative model research. And they show their respective strengths and weaknesses along with excellent performance.
Among them, the flow-based model configures a mapping $f_{\theta}:X\rightarrow Z$ between the desired data distribution $X$ and latent space distribution $Z$ (\eg Gaussian) through a series of invertible transformations. Such an invertible mapping architecture enables an explicit computation of negative log-likelihood (NLL) by the change of variable formula as:
\begin{equation}\label{eq:eq_1}
    -\log{p_{X}(x)} = -\log{p_{Z}(f_{\theta}(x))}-\log{|\det{\frac{\partial{f_\theta}}{\partial{x}}}(x)|}.
\end{equation}
By minimizing NLL directly, it is widely known that the flow-based models show decent performance in mode coverage of the desired data distribution.

Based on this advantage of the flow-based approach, SRFlow \cite{lugmayr2020srflow} first showed that the flow-based modeling of the conditional distribution of the HR image can successfully learn super-resolution space corresponding to the given LR input. And one of its variants model, NCSR \cite{kim2021noise}, proposed an additional noise-conditional layer to SRFlow to generate more diverse super-resolution outputs.
Results of the previous works show that the ill-posedness of super-resolution can be solved from the perspective of super-resolution space learning. To take advantage of the flow-based model's good mode coverage performance, we propose a method to learn more diverse super-resolution space with NCSR architecture.


\subsection{High-Frequency Information}
There are various ways to configure a High-pass filter and Low-pass filter to separate high-frequency and low-frequency information. Without affecting the stability of NLL training of the flow-based model, we utilize the bicubic downsampling-upsampling process as the Low-pass filter, $L_{s}$, with a specific scale factor $s$. And the corresponding High-pass filter, $H_s$, computes the high-frequency information $x_{hf}$ of the given input by subtracting low-frequency information from the HR target $x$:
\begin{equation}\label{eq:eq_2}
    L_{s}(x) = ((x)_{s\downarrow})_{s\uparrow}, \quad
    H_{s}(x) = x_{hf} = x - L_{s}(x).
\end{equation}

There are also other frequency separation methods. Some can configure $L_{s}$ and $H_s$ based on FFT and others can utilize the known 3x3 (or 5x5) kernel.
In the former case, the filtering threshold level is an additional hyperparameter that is heavily dependent on an individual image. And in both cases, to match the low-frequency information of the LR input $y$ and $L_{s}(x)$, additional process such as the usage of a neural network is required leading to instability of NLL training.

By using this simple kind of High-pass filter, sparse high-frequency information can be efficiently obtained since we have the LR input as $y = (x)_{s\downarrow}$. And it leads to our proposed method which achieves efficient training without the need for additional memory or network compared to the previous flow-based approaches.

\subsection{Overall Method}
We propose FS-NCSR (Frequency Separating Noise-Conditioned Normalizing Flow for Super-Resolution), the generative model for super-resolution only produces the high-frequency information of the target HR image $x$ without redundant low-frequency information readily available from $y = (x)_{s\downarrow}$. Our overall model architecture is shown in Figure \ref{fig:overall}.

In the training process of the flow-based models, dequantization processes exist \cite{kingma2018glow, ho2019flow++} for better performance. As can be readily checked in Figure \ref{fig:gt_freq} and Table \ref{tab:Freq_space}, the high-frequency information is relatively sparse compared to HR images. And training the model with this kind of information is difficult. In the previous work of NCSR, the idea of Softflow \cite{kim2020softflow} was used by adding a different level of noise to the input instead of the dequantization process.
This can be interpreted as an attempt to expand the modality of the desired data distribution's sparse region in the perspective of score matching \cite{song2019generative, song2020score} which is in the spotlight of the generative model today.
Therefore, we applied the same idea of Softflow \cite{kim2020softflow} to deal with sparse information, and it was crucial in the training stability of the proposed method.

Now, with the same analog to the work of \cite{lugmayr2020srflow, kim2021noise}, we can formulate the training process of our method as follows:
\begin{equation}\label{eq:eq_3}
    \begin{aligned}
    x^{+}_{hf} = x_{hf} + v, \quad v \sim \mathcal{N}(0, \Sigma), \\
    y^{+} = y + w, \quad w \sim \mathcal{N}(0, \hat{\Sigma}),\\
    z = f_{\theta}(x^{+}_{hf}|y^{+}, v).
    \end{aligned}
\end{equation}
where $w$ indicates noise resized to the same size as the LR input $y$.
And also similar to \cite{lugmayr2020srflow, kim2021noise}, we formulate the loss function only NLL $\mathcal{L}_{nll}$ as below,
\begin{equation}\label{eq:eq_4}
    \begin{aligned}
    \mathcal{L}_{nll} &= -\log{ p_{X}(x|y^{+},v)} \\
    &= -\log{p_{Z}(f_{\theta}(x;y^{+},v))} - \log{|\det{\frac{\partial{f_\theta}}{\partial{y}}}(x;y^{+},v)|}.
    \end{aligned}
\end{equation}

The model trained in proposed method does not require additional cost in the inference stage compared to the previous approaches. Since the low-frequency information $L_{s}(x)$ is readily given by the LR input $y = (x)_{s\downarrow}$. The super-resolution output $\hat{x}$ is obtained by:
\begin{equation}\label{eq:inference}
\begin{aligned}
    \hat{x} = f^{-1}_{\theta}(z;y,v) + (y)_{s\uparrow} \\
    = f^{-1}_{\theta}(z;(x)_{s\downarrow},v) + L_{s}(x).
\end{aligned}
\end{equation}
where $v$ is the random noise from the latent space $Z$.

In this perspective of frequency domain, super-resolution is the process of generating the corresponding high-frequency information since we have $f^{-1}_{\theta}(\cdot;(x)_{s\downarrow}) \approx H_{s}(x)$.
\subsection{Datasets}
We utilize DF2K dataset, a merged dataset of DIV2K \cite{agustsson2017ntire} and Flickr2K\footnote{\url{https://github.com/limbee/NTIRE2017}}, for training and evaluation.
DIV2K dataset consists of 800, 100, and 100 high-resolution images of train, validation, and test split, respectively.
Flickr2K dataset comprises 2560 high-resolution images.
The train split from the DIV2K dataset and the whole Flickr2K dataset are merged and used for training.
We evaluate our model with the validation split of DIV2K dataset.

We try to increase the amount of training dataset by including crawled images from Unsplash website\footnote{\url{https://unsplash.com}}, but there are no performance improvements in the diversity and visual quality of super-resolved images.
Thus, we do not include our crawled dataset in this research.

During training, we randomly crop 160×160 patches from original HR images and use them as HR samples.
We obtain LR samples by downsampling HR patches and utilizing HR and LR patches as HR-LR pairs for training.
The LR samples are downsampled via bicubic kernel.
We train our model in RGB channels, and randomly apply horizontal flips and 90-degree rotation for data augmentation.
\begin{figure}[t]
    \centering
    \includegraphics[width=1.0\linewidth]{fig/high_freq.pdf}
    \caption{Full RGB vs. High-frequency information. High-frequency information is relatively sparse compared to its original RGB images. The high-frequency information was obtained based on $\times$128 scale for clear visual confirmation. For $\times$4 and $\times$8 cases, the high-frequency information is much sparser, making it difficult to see.}
    \label{fig:gt_freq}
\end{figure}


\subsection{Training}

We use the Adam optimizer \cite{kingma2014adam} with $\beta_1$ = 0.9, $\beta_2$ = 0.99, $\epsilon$ = $10 ^{-8}$, and set the initial learning rate as $2 \times 10 ^{-4}$.
Following \cite{kim2021noise}, the learning rate is halved at 50\%, 75\%, 90\%, and 95\% of the total training steps.
We train our network with a batch size of 16 on a V100 GPU.
The $\times$4 network was trained at 180k steps and the $\times$8 network at 220k steps.


\subsection{Evaluation}
We evaluate our model and other baselines based on three criteria: photo-realism, diversity of super-resolution space, and image consistency on LR.
We adopt LPIPS \cite{zhang2018unreasonable} to evaluate photo-realism, diversity score to evaluate diversity, and LR PSNR to evaluate LR consistency.

\noindent\textbf{LPIPS.}
LPIPS is the distance between the super-resolved and the ground-truth HR image.
The distance is measured on the feature space of AlexNet \cite{krizhevsky2012imagenet}.

\noindent\textbf{Diversity Score.}
To obtain meaningful diversity of models, Lugmayr \etal{} \cite{Lugmayr_2021_CVPR} proposed the diversity score.
Let the ground-truth HR image $y$ and $y_{k}$ be the $k$-th patch of $y$.
Generating $M$ samples from the super-resolution models, the $i$-th super-resolved images from the model is $\hat{y^{i}}$, and its $k$-th patch is $\hat{y_{k}^{i}}$, where $i \in \{1, 2, ..., M\}$.
Than the diversity score $S_{M}$ can be computed as follows:
\begin{equation}\label{eq:div_score}
    S_{M} = \cfrac{1}{\bar{d_{M}}} \left(\bar{d_{M}} - \cfrac{1}{K} \sum_{k=1}^{K}\min{\left\{d\left(y_{k}, \hat{y_{k}^{i}}\right)\right\}}^{M}_{i=1} \right),
\end{equation}
where minimum distance on a global sample, $\bar{d_{M}}$, defined as follows:
\begin{equation}\label{eq:avg_distance}
    \bar{d_{M}}=\min{\left\{ \cfrac{1}{K} \sum_{k=1}^{K}d\left(y_{k}, \hat{y_{k}^{i}}\right) \right\}}^{M}_{i=1}.
\end{equation}
We use LPIPS as distance function $d$, and set $M=10$.

\noindent\textbf{LR PSNR.}
In LR, the super-resolved output of the model must be consistent with the original LR input.
Thus, we measure PSNR (Peak Signal-to-Noise Ratio) between downsampled super-resolved image and given input LR image.

\begin{table}[t]
\begin{tabular}{@{}llll@{}}
\toprule
Model & Diversity$\uparrow$ & LPIPS$\downarrow$ & LR PSNR$\uparrow$ \\ \midrule
RRDB~\cite{wang2018esrgan}  & 0 & 0.253 & 49.20   \\
ESRGAN~\cite{wang2018esrgan}  & 0 & 0.124 & 39.03   \\
ESRGAN+~\cite{Rakotonirina_2020} & 22.13 & 0.279 & 35.45   \\
SRFlow~\cite{lugmayr2020srflow} & 25.26 & 0.120 & 49.97   \\
HCFlow~\cite{liang21hierarchical} & 22.73 & \textbf{0.116} & 49.46 \\
NCSR~\cite{kim2021noise}  & 26.72 & 0.119 & \textbf{50.75}   \\
\textbf{FS-NCSR (Ours)} & \textbf{29.44} & 0.127 & 49.31  \\ \bottomrule
\end{tabular}
\captionsetup{justification=raggedright,singlelinecheck=false}
\vspace{-0.3cm}
\caption{General image $\times$4 super-resolution results on the DIV2K validation set.  We measure all the metrics with $M=10$ samples for each HR image.}
\label{tab:X4_results}
\end{table}

\subsection{Quantitative Results} \label{sub:quantitative}
We compare our model, FS-NCSR, with diverse baseline models: RRDB \cite{wang2018esrgan}, ESRGAN \cite{wang2018esrgan}, ESRGAN+ \cite{Rakotonirina_2020}, SRFlow \cite{Lugmayr_2021_CVPR}, HCFlow \cite{liang21hierarchical}, and NCSR \cite{kim2021noise}.
RRDB is the model trained with $L_{1}$ loss with ground-truth HR image, consequently oriented to minimizing PSNR.
ESRGAN and ESRGAN+ are GAN-based methods that are the common baselines for photo-realistic super-resolution.
RRDB and ESRGAN are deterministic models, so their diversity scores are zero.
SRFlow, HCFlow, and NCSR are stochastic super-resolution models that can super-resolve diverse photo-realistic images from the given input LR image.
For all the flow-based super-resolution models, the temperature is set to 0.9. However, the temperature is 0.85 for NCSR $\times$8 model.

We measure the diversity score, LPIPS, LR PSNR of our model and compare them with the reported results of other baselines.
We evaluate all the models in $\times$4 super-resolution setting.
As shown in Table \ref{tab:X4_results}, our proposed model, FS-NCSR, achieves the highest diversity score in $\times$4 setting.
The diversity score of FS-NCSR is significantly higher than NCSR \cite{kim2021noise}, which indicates frequency separation plays a key role to improve diversity.
Although FS-NCSR achieves the lower LR PSNR and higher LPIPS than SRFlow \cite{lugmayr2020srflow}, HCFlow \cite{liang21hierarchical} and NCSR, diversity increase is significant compared to such performance degradation so can be compensated.
In addition, we observe that the number of artifacts and failure cases in the generated samples of FS-NCSR is less than that of NCSR.
We will discuss this qualitative comparison in \ref{sub:qualitative}.


We also evaluate all the models except ESRGAN+ \cite{Rakotonirina_2020} in $\times$8 super-resolution setting.
As presented in Table \ref{tab:X8_results}, FS-NCSR outperforms all the other methods in terms of diversity score and LPIPS.
Also, FS-NCSR achieves comparable LR PSNR with SRFlow \cite{lugmayr2020srflow}, the model which achieved the highest LR PSNR.
These results show that FS-NCSR outperforms all the other methods in terms of photo-realism and diversity, and frequency separation is a decisive factor.

\begin{table}[t]
\begin{tabular}{@{}llll@{}}
\toprule
Model & Diversity$\uparrow$ & LPIPS$\downarrow$ & LR PSNR$\uparrow$ \\ \midrule
RRDB~\cite{wang2018esrgan} & 0 & 0.419 & 45.43   \\
ESRGAN~\cite{wang2018esrgan} & 0 & 0.277 & 31.35   \\
SRFlow~\cite{lugmayr2020srflow} & 25.31 & 0.272 & \textbf{50.00}   \\
NCSR~\cite{kim2021noise} & 26.8 & 0.278 & 44.55   \\
\textbf{FS-NCSR (Ours)} & \textbf{26.9} & \textbf{0.257} & 48.90 \\ \bottomrule
\end{tabular}
\captionsetup{justification=raggedright,singlelinecheck=false}
\vspace{-0.3cm}
\caption{General image $\times$8 super-resolution results on the DIV2K validation set. We measure all the metrics with $M=10$ samples for each HR image.}
\label{tab:X8_results}
\end{table}

To clearly demonstrate the effect of frequency separation, we additionally report the metric trajectories during the training process of FS-NCSR and NCSR \cite{kim2021noise}.
We measure LPIPS and diversity score in 150k, 160k, 170k, 180k steps for each model.
The results of such models during the training process are presented in Figure \ref{fig:trajectory}.
For trained weights of FS-NCSR, higher diversity and lower LPIPS than NCSR weights of the same iteration are measured.
These results show that frequency separation consistently improves the diversity and photo-realism of the model output during the training process.

\begin{figure*}
\begin{center}
\centering
    \begin{subfigure}{.24\textwidth}
    \centering
    \includegraphics[width=\linewidth]{fig/crop/4x_crop1.png}
    \caption{$\times$4 LR}
    \end{subfigure}
    \begin{subfigure}{.24\textwidth}
    \centering
    \includegraphics[width=\linewidth]{fig/crop/ncsr_crop1.png}
    \caption{NCSR \cite{kim2021noise}}
    \end{subfigure}
    \begin{subfigure}{.24\textwidth}
    \centering
    \includegraphics[width=\linewidth]{fig/crop/fs_ncsr_crop1.png}
    \caption{FS-NCSR (Ours)}
    \end{subfigure}
    \begin{subfigure}{.24\textwidth}
    \centering
    \includegraphics[width=\linewidth]{fig/crop/gt_crop1.png}
    \caption{Ground Truth}
    \end{subfigure}
    \vspace{-0.3cm}
\caption{Qualitative results with comparison to NCSR on the DIV2K validation set for SR $\times$4 results. The cropped part of the ground truth is from 0850 from DIV2K. Each output of NCSR and FS-NCSR was chosen randomly from 10 generated outputs respectively.}
\label{fig:compare_crop}
\end{center}
\end{figure*}

\begin{figure*}
\begin{center}
\centering
    \begin{subfigure}{.16\textwidth}
    \centering
    \includegraphics[width=\linewidth]{fig/diverse/diverse_crop0_edit.png} 
    \caption{Output 1}
    \end{subfigure}
    \begin{subfigure}{.16\textwidth}
    \centering
    \includegraphics[width=\linewidth]{fig/diverse/diverse_crop1_edit.png}
    \caption{Output 2}
    \end{subfigure}
    \begin{subfigure}{.16\textwidth}
    \centering
    \includegraphics[width=\linewidth]{fig/diverse/diverse_crop2_edit.png}
    \caption{Output 3}
    \end{subfigure}
    \begin{subfigure}{.16\textwidth}
    \centering
    \includegraphics[width=\linewidth]{fig/diverse/diverse_crop3_edit.png}
    \caption{Output 4}
    \end{subfigure}
    \begin{subfigure}{.16\textwidth}
    \centering
    \includegraphics[width=\linewidth]{fig/diverse/diverse_crop4_edit.png}
    \caption{Output 5}
    \end{subfigure}
    \begin{subfigure}{.16\textwidth}
    \centering
    \includegraphics[width=\linewidth]{fig/diverse/gt_crop_edit.png}
    \caption{Ground Truth}
    \end{subfigure}
    \vspace{-0.3cm}
\caption{Qualitative result to check the FS-NCSR's diversity of generated details on the DIV2K validation set for $\times$4 super-resolved results. The ground truth is a cropped part of 0875 from DIV2K. 5 outputs were chosen from 10 generated FS-NCSR outputs.}
\label{fig:diversity}
\end{center}
\end{figure*}

\begin{figure*}
\begin{center}
\centering
    \begin{subfigure}{.45\textwidth}
    \centering
    \includegraphics[width=.95\linewidth]{fig/artifact_eg/ncsr_fail_edit.png}
    \caption{Failure case of NCSR}
    \end{subfigure}
    \begin{subfigure}{.45\textwidth}
    \centering
    \includegraphics[width=.95\linewidth]{fig/artifact_eg/fs_ncsr_fail_edit.png}
    \caption{Failure case of FS-NCSR}
    \end{subfigure}
    
    \begin{subfigure}{.45\textwidth}
    \centering
    \includegraphics[width=.95\linewidth]{fig/artifact_eg/fs_ncsr_ok.png}
    \caption{FS-NCSR Ouput without any artifact}
    \end{subfigure}
    \begin{subfigure}{.45\textwidth}
    \centering
    \includegraphics[width=.95\linewidth]{fig/artifact_eg/0807_gt.png}
    \caption{Ground Truth: 0807 from DIV2K}
    \end{subfigure}
    
    \begin{subfigure}{.45\textwidth}
    \centering
    \includegraphics[width=.95\linewidth]{fig/artifact_eg2/ncsr_fail_edit.png}
    \caption{Failure case of NCSR}
    \end{subfigure}
    \begin{subfigure}{.45\textwidth}
    \centering
    \includegraphics[width=.95\linewidth]{fig/artifact_eg2/fs_ncsr_fail_edit.png}
    \caption{Failure case of FS-NCSR}
    \end{subfigure}
    
    \begin{subfigure}{.45\textwidth}
    \centering
    \includegraphics[width=.95\linewidth]{fig/artifact_eg2/fs_ncsr_ok.png}
    \caption{FS-NCSR Ouput without any artifact}
    \end{subfigure}
    \begin{subfigure}{.45\textwidth}
    \centering
    \includegraphics[width=.95\linewidth]{fig/artifact_eg2/0828_gt.png}
    \caption{Ground Truth: 0828 from DIV2K}
    \end{subfigure}
\caption{Visual comparison of failure cases on SR $\times$4 results from NCSR and FS-NCSR. The degree of the artifact is relatively less in the result of FS-NCSR than that of NCSR. And FS-NCSR could generate clear SR output without any artifact while NCSR couldn't. Each output was chosen randomly.}
\label{fig:artifact}
\end{center}
\end{figure*}

\begin{figure}[t]
    \centering
    \includegraphics[width=0.8\linewidth]{fig/comparison.png}
    \vspace{-1.\baselineskip}
    \caption{%
    LPIPS and diversity scores of multiple checkpoints of FS-NCSR and NCSR \cite{kim2021noise}. The super-resolution ratio is $\times$4 setting. We measure LPIPS and diversity at 150k, 160k, 170k, 180k steps of training procedures. The arrows in the figure indicate the change in both metrics as the iteration increases by 10k.
    }
    \label{fig:trajectory}
\end{figure}


\subsection{Qualitative Results} \label{sub:qualitative}

The qualitative result in Figure \ref{fig:diversity} shows that the direction and the degree of density of the leaves are slightly different for every 5 outputs. 
Thus, we can say that the proposed method not only shows a higher diversity score than previous approaches but also can generate outputs with diverse details that are distinguishable visually. It means that the frequency separation can enhance the high mode coverage performance of the flow-based model.

We now qualitatively compare our result with the output of NCSR to verify the effect of the frequency separation. As discussed in \ref{sub:quantitative}, the FS-NCSR's LPIPS was lower than the existing approaches. But Figure \ref{fig:compare_crop} shows that FS-NCSR can reproduce the characters more clearly than NCSR. This qualitatively confirmed that although the training focused on high-frequency information performs slightly lower on LPIPS, actual outputs do not suffer a degradation of image quality than the existing methodologies.

The existing SRFlow and NCSR models show repeated failure cases where artifacts appear in a specific image (\eg 0807, 0828 from DIV2K validation set). In the case of 0807 from DIV2K, for instance, when both SRFlow and NCSR generated the corresponding $\times$4 super-resolved outputs, all outputs were failure cases since some artifacts appeared. 
On the other hand, when FS-NCSR generate the $\times$4 super-resolved outputs of the given image, 4 out of 10 outputs were made without any artifact, and even for 6 failure cases, the degree of the artifact was relatively less than that of NCSR. Figure \ref{fig:artifact} presents the degree of the artifact differs between NCSR and FS-NCSR output and the FS-NCSR's artifact-free results compared to the ground truth image.


\subsection{Ablation: Comparison of Generated High-Frequency Information}

So far, we have discussed the results both quantitatively and qualitatively with the super-resolved outputs only. But we tried to compare the results from the perspective of frequency information additionally. 
Since the sparse high-frequency information plays a key role in the proposed method, we investigated how the proposed method affects the sparsity of the generated high-frequency information.
For this purpose, the generated high-frequency information in $[0,1]$ range is first quantized to the uint8 $[0,255]$ range. And then \textbf{Sparsity} and \textbf{Relative Sparsity (RS)} is computed as follows:
\begin{equation}\label{eq:sparsity}
\begin{aligned}
    \textbf{Sparsity} = 1 - \frac{\textbf{Number of non-zero pixels}}{H\times W\times C} \\
    \textbf{RS} = 1 - \frac{\textbf{Number of non-zero pixels}}{\textbf{Number of non-zero gt pixels}}
\end{aligned}
\end{equation}
where $(H, W, C)$ is the shape of a given image.
Since the ground truth high-frequency information is already sparse, the RS reflects the sparsity of each ground truth image for a more fair comparison.

\begin{table}[ht]
\begin{tabular}{@{}lll@{}}
\toprule
Model & Average Sparsity & Average RS \\ \midrule
NCSR~\cite{kim2021noise}  & 66.2\% & 1.123   \\
\textbf{FS-NCSR (Ours)} & \textbf{66.0\%} & \textbf{1.120}  \\ 
\bottomrule
\end{tabular}
\captionsetup{justification=raggedright,singlelinecheck=false}
\vspace{-0.3cm}
\caption{As a $\times$4 super-resolution results on DIV2K validation set, \textbf{Average Sparsity of GT high frequency information is 59.0\%}.}
\label{tab:Freq_space}
\end{table}

See Table \ref{tab:Freq_space}. Although our proposed method shows less average sparsity and average RS slightly, the average sparsity of the ground truth high-frequency information and the that of generated output from both NCSR and FS-NCSR was about 10\% more sparse, resulting in a lack of information compared to the ground truth.
This margin of difference with the ground truth verifies that a significant loss of information still exists from the perspective of the frequency domain. Therefore, it seems that it needs to be addressed in future studies.

