\section{Introduction} 
Generative adversarial networks (GAN)~\cite{goodfellow2014generative} is a machine learning framework to generate realistic fake data. GAN learns the probabilistic distribution of the training (real) data using two adversarial networks: the generator that is trained to create realistic fake data from a random seed called the latent vector, and the discriminator that is dedicated to distinguish the reals with fake data. GAN has been studied exhaustively in the several past years and currently is an essential tool in a wide variety of applications.
\par
However, the stability of optimization is a crucial issue of GANs~\cite{Nagarajan17gdGANstable,Mescheder2018which,berard2019closer,kurach2019large,Wang2020LSVGDforGAN}.  Since it is a two-player game between the generator and discriminator~\cite{berard2019closer}, GAN optimization can fall into a local minima where the discriminator reaches a perfect solution first and the generator cannot be trained anymore. This failure of GAN optimization is known as mode collapse that severely decreases the quality of fake data and, in the worst case, forces the generator to output a single data.
The mode collapse occurs when there is almost no overlap between the distributions of real data and fake data~\cite{arjovsky2017towards}, in particular, in the beginning of training when the discriminator can reject fake data with high confidence~\cite{goodfellow2014generative}. Thus GANs require regularization in order to stabilize the optimization process and obtain satisfying fake data.
Moreover, the regularization methods of GAN are different with those for the feed-forward deep networks, e.g., weight-decay~\cite{krogh1992simple} and the momentum~\cite{sutton1986two}, due to the dynamics of discriminator with generator.
The explicit regularization of GAN modifies the loss of GAN so as to provide gradients to the generator even with a small overlap of the two distributions using such as Wasserstein metric in GAN loss and the gradient regularization.
However, the Wasserstein loss is inferior to the original loss in quality of fake data in practice~\cite{Mescheder2018which,kurach2019large} and also the gradient regularization depends on the model that varies during the training~\cite{kurach2018gan}.
\par
The implicit regularization is obtained through the learning procedure and the data manipulation. We focus on the data-driven regularization for GAN optimization.
The repetition of noise injection, or namely noise-space, to both real and fake data enlarges the overlap between probability distributions~\cite{jenni2019noiseGAN}. However, the noise increases the variance of data essentially.
The repetition of data smoothing, or scale-space, is a general technique in machine learning that suppresses the low-level information such as textures and details in the image so as to make each data more simple to learn by algorithms. However, we have observed that the smoothing shrinks the capability of the generator to learn high-frequency information.
\par
We propose an implicit regularization method for stabilizing GAN optimization in the image generation tasks using \textit{noisy scale-space} that continuously removes high-frequency information in image while imposing noise.
The proposed noisy scale-space aims a coarse-to-fine training of GANs 
in which we can train the model using low-level information in data with noise without the increment of data variance, while keeping the current model capable of high-level information.
We also present a synthetic data-set using the Hadamard bases~\cite{townsend2001walsh} that enables us to visualize the true distributions of real and fake data in order to demonstrate the drawback of the conventional scale-space in GAN optimization.
We then implement the proposed data representation using the baseline DCGAN and present Noisy Scale-Space (NSS) GAN and present experimental results based on the major image data-sets in which the proposed NSS-GAN overtook other methods in most cases irrespective of the image generation tasks.
Moreover it is shown that the regularization effect by our method is not the simple summation of the noise-space with the scale-space but due to the use of their mutually complementary relationship.
\par
We relate our method to prior works in Section~\ref{sec:related_works}.
Then we consider GAN with the data-driven regularization in Section~\ref{sec:preliminary}, 
followed by the proposed noisy scale-space in Section~\ref{sec:method}.
The effectiveness of our method is demonstrated experimentally in Section~\ref{sec:results} and we conclude in Section~\ref{sec:conclusion}.
\section{Related works}   \label{sec:related_works}
\par
\vspace{3pt}
\noindent {\bf GAN loss:} 
The original GAN~\cite{goodfellow2014generative} uses the KL-divergence to measure the difference between the real and fake distributions.
Since the mode collapse arises from the use of the KL-divergence, a variety of the divergence have been studied for GAN optimization, e.g., ~\cite{mao2017least,Nowozin16fGAN,cai2020utilizing}, including Wasserstein distance~\cite{arjovsky2017towards,arjovsky2017wasserstein}.
In our experiments, we employ the non-saturating loss~\cite{goodfellow2014generative} based on the KL-divergence that is known to be the best choice in practice~\cite{Mescheder2018which,kurach2019large}. 
\par
\vspace{3pt}
\noindent {\bf GAN architecture:} 
There are two foundations of the GAN architecture: One is the basic GAN~\cite{goodfellow2014generative} of which the discriminator and generator consist of the fully-connected neural networks. The basic GAN has been extended to early GANs, e.g., Conditional GAN~\cite{mirza2014cGAN} and  WGAN~\cite{arjovsky2017wasserstein}.
The other one is DCGAN~\cite{radford2015dcgan} that utilizes convolution networks. The convolution network is the essential architecture of the modern GANs, e.g.,~\cite{chen2016infogan,pathakCVPR16context,odena2017conditional,zhang2017stackgan,CycleGAN2017,brock2018large,karras2019style}. We thus use DCGAN as the baseline architecture.
\par
\vspace{3pt}
\noindent {\bf Explicit regularization for GAN optimization:} 
Wasserstain GAN requires an additional loss term to guarantee the Lipschitz constraint. To this aim, the original WGAN uses weight clipping~\cite{arjovsky2017wasserstein}. WGAN-gp uses gradient penalty~\cite{Sailmans16improved} that can improve the quality of fakes in practice. The spectral normalization~\cite{Miyato17spectral} is more computational efficient than the gradient-penalty. Dragan~\cite{kodali2017convergence} penalizes the sharp gradient of discriminator to real data. These constraints act as an explicit regularizer that helps the generator to be trained.
GANs with the non-saturating loss can be improved by the gradient-based regularization~\cite{Mescheder2018which}. The gradient-based regularization adds the regularization term into the GAN loss using the gradient norm of discriminator in order to restrict the training of discriminator.
It is shown that the penalty based on the gradient-norm of the discriminator is equivalent to adding input noise in GAN using $f$-divergence~\cite{Kevin2017regularization}. 
The drawback of gradient-based regularization is that it depends on the distribution of fakes determined by the generator which changes during training~\cite{kurach2018gan}.
Lens-GAN~\cite{sajjadi2018tempered} introduces a filtering network that transforms the real data against the discriminator, and thus depends on the networks.
\par
\vspace{3pt}
\noindent {\bf Implicit regularization for GAN optimization} 
is obtained through the learning procedure and the data manipulation. An example of the former is the two time-scale update rule~\cite{Heusel17TTUR} that decreases the learning-rate of the discriminator in order to slow the convergence of the discriminator. The implicit regularization effect by the simultaneous update of generator and discriminator is studied in~\cite{schafer2019implicit}.
The label noise is an implicit regularization technique in the classic deep learning~\cite{zhang2016understanding} and is also shown to be valuable in the GAN framework.
Progressive Augmentation of GANs~\cite{Zhang19progressive_aug} perturbs the real and fake labels.
The one-sided label smoothing replaces the 0 and 1 target labels for the discriminator with smoothed values, like 0.9 or 0.1~\cite{Sailmans16improved,hazan2017adversarial}. 
\par
Our method is closely related to the noise injection and the smoothing to data.
On one hand, the noise injection to data flattens the probability distribution of data.
Thus imposing noise to the real data~\cite{sonderby2016amortised,arjovsky2017towards} or both the real and fake data~\cite{jenni2019noiseGAN} enlarges the overlap of the two distributions.
However, adding high-dimensional noise introduces significant variance in the parameter estimation, slowing down the training and requiring multiple samples for counteraction~\cite{Kevin2017regularization,Zhang19progressive_aug}.
On the other hand, the scale-space decreases the dimensionality of data by removing high-frequency information and makes data easy to learn by algorithms~\cite{witkin1987scale,lindeberg2013scale}.
However, the conventional scale-space has a critical side-effect in GAN optimization.
In contrast, we consider the general framework of the data-driven regularization and present a data representation that mitigates the side effects of the noise injection and the data smoothing.
\par
The multi-resolution training of GAN
is a recent topic studied in, e.g., Progressive-GAN~\cite{karras2017progressive}, Style-GAN~\cite{karras2019style}, and MSG-GAN~\cite{Karnewar20MSGGAN}, for generating high-resolution images.
It trains a shallow network first using low-resolution images, and gradually increases both the number of layers in the model and the resolution of example images.
The drawback of the multi-resolution training is that it strongly restricts the architecture.
In contrast, we consider an implicit regularization independent to the model networks.




\section{Preliminary} \label{sec:preliminary}
\subsection{Generative adversarial networks}
Let us start with a review of the generative adversarial networks (GAN).
Given a set of real data ($x$), GAN aims to generate new data with similar statistics as the real data. GAN consists of the two networks: the generator (\generator) that creates fake data ($G(\latent)$) from a random latent vector ($z$), and the discriminator (D) that distinguishes the real data with the fake data.
To this aim, a min-max loss ($F$) is defined~\cite{goodfellow2014generative} as 
\begin{equation} \label{eq:GAN-loss-original}
F_{D,G} \bigl( x,G(z) \bigr) \quad \coloneqq \quad
\mathbb{E}_{x \sim p_\text{data}(x)} \bigl[ \log D(x) \bigr] + 
\mathbb{E}_{z \sim p_\text{z}(z)} \bigl[ \log (  1 - D(G(z)) ) \bigr], 
\end{equation}
where $p_\text{data}(x)$ is the true distribution of the reals, and $p_\text{z}(z)$ is a random variable.
GANs are optimized using a stochastic gradient descent, e.g., Adam~\cite{kingma2014adam}, in an alternative way such that the generator is updated to minimize $F$ while the discriminator is trained to maximize $F$.
\subsection{Data-driven regularization of GAN}
We consider the general framework of date-driven implicit regularization for stabilizing the GAN optimization. The data-driven method projects the reals and fakes to another data space and feed into the GAN loss as
\begin{flalign}\label{eq:GAN-loss-filtered}
& F_{D,G} \bigl( \Phi_t(x),\Phi_t(G(z)) \bigr),
\end{flalign}
where $\Phi_t(x)$ and $\Phi_t(G(z))$ are the projected data of reals and fakes using a function $\Phi$ with the time parameter $t$.
Using Eq.(\ref{eq:GAN-loss-filtered}), the generator is trained to create fake data such that the projected fakes $\Phi_t(G(z))$ are similar to the projected reals $\Phi_t(x)$.  
The data-driven regularization will be imposed via a discrete representation of data
$\{\Phi_T(y),\Phi_{T-1}(y),...,\Phi_2(y),\Phi_1(y),\Phi_0(y)\}$ 
where $\Phi_t(y)$ with larger $t$ is expected to have larger regularization effect and $\Phi_0(y) = y$ is the original real or fake data.
To this aim, the time parameter ($t$) should start with a large value $T$ and shrink to zero with the optimization process, and the choice of $\Phi$ determines regularization effect.







\def 80pt {55pt}
\def 61pt {45pt}
\begin{figure} [htb]
\centering
\includegraphics[width=225pt]{concept}
\caption{An illustration of real and fake distributions ($P,Q$) in (left) the original space, 
(top) the noise-space,
(bottom) the scale-space,
and (right) the proposed noisy scale-space.
}
\label{fig:concept}
\end{figure}


\def 80pt {55pt}
\def 61pt {45pt}
\begin{figure} [htb]
\centering
\small
\begin{tabular}{m{6pt} m{61pt}m{61pt}m{61pt}m{61pt}m{61pt}m{61pt}}
(a) &
\includegraphics[height=80pt]{img,noise,cameraman,T,000} &
\includegraphics[height=80pt]{img,noise,cameraman,T,004} &
\includegraphics[height=80pt]{img,noise,cameraman,T,016} &
\includegraphics[height=80pt]{img,noise,cameraman,T,064} &
\includegraphics[height=80pt]{img,noise,cameraman,T,256} \\
(b) & 
\includegraphics[height=80pt]{img,smooth,cameraman,T,000} &
\includegraphics[height=80pt]{img,smooth,cameraman,T,004} &
\includegraphics[height=80pt]{img,smooth,cameraman,T,016} &
\includegraphics[height=80pt]{img,smooth,cameraman,T,064} &
\includegraphics[height=80pt]{img,smooth,cameraman,T,256} \\
(c) &
\includegraphics[height=80pt]{img,sn,cameraman,T,000} &
\includegraphics[height=80pt]{img,sn,cameraman,T,004} &
\includegraphics[height=80pt]{img,sn,cameraman,T,016} &
\includegraphics[height=80pt]{img,sn,cameraman,T,064} &
\includegraphics[height=80pt]{img,sn,cameraman,T,256} \\
& 
\multicolumn{1}{c}{\quad $t=0$} & 
\multicolumn{1}{c}{\quad $t=4$} & 
\multicolumn{1}{c}{\quad $t=16$} & 
\multicolumn{1}{c}{\quad $t=64$} & 
\multicolumn{1}{c}{\quad $t=256$} & 
\\
\end{tabular}
\caption{A real image in (a) the conventional noise-space or the repetition of the Gaussian noise with $\sigma=0.15$, 
(b) the conventional scale-space or the repetition of smoothing, and (c) the proposed noisy scale-space over (columns) the time ($t$).}
\label{fig:real_images_with_filters}
\end{figure}




\def \rh {0.8}
\def 80pt {100pt}
\begin{figure} [h!]
\centering
\small
\renewcommand{\arraystretch}{\rh}
\begin{tabular}{cccc}
\includegraphics[width=80pt]{image_variance_celeba} &
\includegraphics[width=80pt]{image_variance_church} \\
CelebA & Church \\
\includegraphics[width=80pt]{image_variance_conference} &
\includegraphics[width=80pt]{image_variance_oxflower} \\
 Conference & Flowers \\
\end{tabular}
\renewcommand{\arraystretch}{1}
\caption{(y-axis) The variance of pixel intensities in the noise-space (red line), the scale-space (blue dashed-line), and the proposed noisy scale-space (magenta dotted-line) over (x-axis) the time $t$ based on 128 images in CelebA~\cite{liu2015faceattributes}, LSUN-Church, LSUN-Conference~\cite{yu15lsun}, and Oxford-Flowers~\cite{Nilsback08flowers} using the noise of $\sigma=0.15$.}
\label{fig:image_variance}
\end{figure}





\section{Regularization via noisy scale-space} \label{sec:method}
\subsection{Proposed noisy scale-space}
We propose a data representation, namely noisy scale-space, that is valuable for stabilizing the GAN optimization.
Figure~\ref{fig:concept} illustrates the conceptual difference among the noisy scale-space, the conventional noise-space, and the scale-space. The conventional noise-space (top) flattens the real and fake distributions and enlarges their overlap at the cost of significant data variance. The scale-space (bottom) reduces the dimensionality of data by removing high-frequency information but suppresses the capability of GAN for those dimensions. Our idea is to use a balanced composition of the noise injection with the smoothing (right) such that the noise is imposed as the high-frequency information is removed by smoothing.
\par
Formally, the noisy scale-space is given as
\begin{equation} \label{eq:noisy_scale-space}
\Phi_t (y) \coloneqq k \odot \Phi_{t-1} (y) + \epsilon_t,
\end{equation}
where $k$ is typically the $3 \times 3$ Gaussian kernel, the symbol $\odot$ denotes the convolution, $\epsilon_t \sim N(0,\sigma)$, $\Phi_0 (y)=y$, and $t \in \mathbb{N}_0$, i.e., we apply the kernel and the noise to data simultaneously $t$-times.
Figure~\ref{fig:real_images_with_filters} presents an example of image in the conventional noise-space, the conventional scale-space, and the proposed noisy scale-space.
Figure~\ref{fig:image_variance} shows the image variance of the three data-representations using large data-sets.
\par
The std. of noise ($\sigma$) balances the smoothing with the noise in the proposed data representation.
The scale-space removes high-frequency information in image and decreases the image variance as in Figure~\ref{fig:image_variance}.
We therefore tune $\sigma$ in the noisy scale-space such that the image variance is almost preserved over $t$.
The noisy scale-space is robust to $\sigma$ for natural images as shown in Figure~\ref{fig:image_variance} and we employ $\sigma=0.15$ for all of the natural image data-sets in our experiments.
Moreover $\sigma$ can be determined for any data-set using only the original data.






\subsection{Comparison to conventional noise-space}
Given a constant $\sigma$, we can consider a noise-space as
\begin{equation} \label{eq:noise-space}
\Phi^\text{n}_t (y) \coloneqq \Phi^\text{n}_{t-1} (y)  + \epsilon_t, 
\end{equation}
with $\epsilon_{t} \sim  N(0,\sigma), t \in \mathbb{N}_0$, and $\Phi^\text{n}_0 (y) = y$.
Eq.(\ref{eq:noise-space}) provides a discrete representation of data with multiple noises that we call noise-space.
Figure~\ref{fig:real_images_with_filters} (a) shows an example of noise-space using a real image. We refer to GAN using the noise-space as Noise-Space (NS) GAN.
Considering the sum of normal distributions, the noise-space defined by Eq.(\ref{eq:noise-space}) is equivalent to
\begin{eqnarray} \label{eq:noise-space:closed}
\Phi^\text{n}_t (y) &=& y  + \epsilon_1+ \epsilon_2 + ... + \epsilon_t, \nonumber \\
&=& y  + \hat{\epsilon}(t), 
\end{eqnarray}
where $\hat{\epsilon}(t) \sim N(0, \sigma \cdot t)$, $t \in \mathbb{N}$. Therefore, the drawback of the noise-space is that it increases the variance of data, and makes the original data harder to be learned.
\par
The noisy scale-space using Eq.(\ref{eq:noisy_scale-space}) also can be rewritten in a closed-form as
\begin{eqnarray} \label{eq:noisy_scale-space:closed}
\Phi_t (y) &=& 
\underbrace{k \odot^{(t)} y}_{\textrm{smoothed data}} +  
\underbrace{
k \odot^{(t-1)} \epsilon_{1} + k \odot^{(t-2)} \epsilon_{2} + ... + k \odot^{(1)} \epsilon_{t-1} + k \odot^{(0)} \epsilon_{t} 
}_{\textrm{low-to high-frequency noises}}, \nonumber \\
\end{eqnarray}
where $k \odot^{(t)}$ denotes the $t$-times convolution with kernel $k$, and $k\odot^{(0)}\epsilon_{t}=\epsilon_{t}$.
Thus, our noisy scale-space balances the variances of the data-term and the noise-terms using the smoothing. A finding is that we can keep the data variance using a constant $\sigma$ as demonstrated in Figure~\ref{fig:image_variance}.
Eq.(\ref{eq:noisy_scale-space:closed}) also shows that the noisy scale-space replaces the high-frequency information in data by a set of noises with the corresponding frequencies. 
In the following section, we demonstrate this property using a synthetic data-set.












\subsection{Comparison to conventional scale-space} 
The scale-space is a multi-scale representation of the image in which the smoothing is applied to data recursively as
\begin{equation} \label{eq:scale-space}
\Phi_t^\text{s} (y) \coloneqq k \odot \Phi^\text{s}_{t-1}(y) , 
\end{equation}
with $\Phi^\text{s}_0 (y)=y$ and $t \in \mathbb{N}_0$.
As shown in Figure~\ref{fig:real_images_with_filters} (b), the scale-space continuously removes the low-level information, e.g., textures and details,
and provides the high-level information that is invariant to the scale. 
The scale-space will enable a coarse-to-fine training of GANs such that we train networks using the simplified data first and then train the networks using data with details.
\par
However, we observe that the scale-space has a side effect in the GAN optimization.
To demonstrate this side effect, we propose to visualize the true probability distributions of both reals and fakes using a synthetic data-set that is created based on eight of vertical Hadamard bases ($\{B_i\}^8$) shown in Figure~\ref{fig:Hadamard_coefficients} (top).  
Given coefficients $\{\alpha_i\}^8$ for the eight basis,  $\sum_i \alpha_i B_i$ produces an $8\times8$ pixel image as real data that looks a white/grey vertical stripes.
The coefficients are the true probability distribution of real data.
Given fake data, we can fit the bases to the fake image and compute the coefficients of eight bases that reflect the probability distribution of fakes with the fitting residuals.


\def 80pt {70pt}
\def 150pt {150pt}
\begin{figure}[h!]
\centering
\small
\includegraphics[width=300pt]{bases} \\
\vspace{8pt}
\begin{tabular}{cccc}
\includegraphics[width=80pt]{real_raw,hadamard_h-0-200000,Adam-0.4-0.999,0-8.fix.1,origan_woAct,reg,0,lr,2e-05.fixedLR,ep,200,tr,1} &
\includegraphics[width=80pt]{real,hadamard_h-0-200000,Adam-0.4-0.999,0-8.fix.1,origan_woAct,reg,0,lr,2e-05.fixedLR,ep,200,tr,1} & 
\includegraphics[width=80pt]{fake,hadamard_h-0-200000,Adam-0.4-0.999,0-8.fix.1,origan_woAct,reg,0,lr,2e-05.fixedLR,ep,200,tr,1} &
\includegraphics[width=80pt]{fake_raw,hadamard_h-0-200000,Adam-0.4-0.999,0-8.fix.1,origan_woAct,reg,0,lr,2e-05.fixedLR,ep,200,tr,1} \\
$x$ & 
$\Phi^\text{s}(x)$ &
$\Phi^\text{s}(G(\latent))$ & 
$G(\latent)$ \\
\end{tabular} \\
\begin{tabular}{cc}
\includegraphics[width=150pt]{hada_coef,hadamard_h-0-200000,Adam-0.4-0.999,0-8.fix.1,origan_woAct,reg,0,lr,2e-05.fixedLR,ep,200,tr,1} &
\includegraphics[width=150pt]{hada_coef_raw,hadamard_h-0-200000,Adam-0.4-0.999,0-8.fix.1,origan_woAct,reg,0,lr,2e-05.fixedLR,ep,200,tr,1}  \\
$\{\alpha_i\}$ of $\Phi^\text{s}(x), \Phi^\text{s}(G(\latent))$ &  
$\{\alpha_i\}$ of $x, G(\latent)$ \\
\end{tabular} \\
\caption{GAN with synthetic data using the scale-space:
(top) The Hadamard bases ($B_1, ..., B_8$), 
(middle part) 
nine examples of the original reals ($x$), 
the smoothed reals ($\Phi^\text{s}(x)$) with fixed $t=8$, 
the smoothed fakes ($\Phi^\text{s}(G(\latent))$) with fixed $t=8$, 
and the original fakes ($G(\latent)$), 
(bottom left) (y-axis) the mean and std. of the coefficients for (x-axis) the Hadamard bases $B_1,...,B_8$ with low to high frequencies within the smoothed reals (green) and the smoothed fakes (orange), 
and (bottom right) those within the original reals and fakes.
$200$K of the synthetic images were created using $\alpha_i \sim U(-1,1), \forall i$ with the uniform distribution ($U$). 
We applied the smoothing to all data in the batch.
The basic GAN~\cite{goodfellow2014generative} was trained using Adam with the learning-rate scale of $\eta=2 \times 10^{-5}$ for 200 epochs.
}
\label{fig:Hadamard_coefficients}
\vspace{4pt}
\end{figure}
\def 150pt {150pt}
\begin{figure}[h!]
\centering
\small
\begin{tabular}{cc}
\includegraphics[width=150pt]{hada_coef,hadamard_h-0-200000,Adam-0.4-0.999,0-8.smoothnoise_0.02_0,origan_woAct,reg,0,lr,2e-05.fixedLR,ep,200,tr,1} &
\includegraphics[width=150pt]{hada_coef_raw,hadamard_h-0-200000,Adam-0.4-0.999,0-8.smoothnoise_0.02_0,origan_woAct,reg,0,lr,2e-05.fixedLR,ep,200,tr,1} \\
$\{\alpha_i\}$ of $\Phi(x), \Phi(G(\latent))$ & $\{\alpha_i\}$ of $x, G(\latent)$ \\
\end{tabular} \\
\caption{GAN with synthetic data using the proposed noisy scale-space: (left) The mean and std. of  the Hadamard coefficients (y-axis) with low to high frequencies (x-axis) within the projected reals (green) and the projected fakes (orange), and (right) those within the original reals and fakes using the noisy scale-space with fixed $t=8$.}
\label{fig:Hadamard_coefficients_smoothing-with-noise}
\end{figure}


\par
We trained the basic GAN~\cite{goodfellow2014generative} based on the synthetic data using the smoothing with fixed $t=8$.
Figure~\ref{fig:Hadamard_coefficients} illustrates 
(top) the Hadamard bases,
(middle) example of the original reals, 
smoothed-reals, smoothed-fakes, and the original fake data, and 
(bottom) the distribution of their Hadamard coefficients.
There are two observations:
(bottom-left) The coefficients of smoothed fakes $\Phi(G(\latent))$ follow the those of $\Phi(x)$ of which high-frequency coefficients are suppressed by the smoothing, i.e., the model learned the probability distribution of the smoothed data.
(bottom-right) However, the smoothing also decreased the diversity of high-frequency coefficients of fakes ($G(\latent)$).
This means that the smoothing reduces the overlap between the fake distribution with the distribution of reals with fine details that will be given in future optimization process.
\par
We visualize the effect of the noisy scale-space using the synthetic data-set in Figure~\ref{fig:Hadamard_coefficients_smoothing-with-noise} 
in which we used a fixed $t=8$.
Figure~\ref{fig:Hadamard_coefficients_smoothing-with-noise} shows that 
(left) the projected data are smoothed yet (right) the diversity of high-frequency coefficients of fakes are preserved as expected.














\subsection{Implementation of noisy scale-space}
The data-driven regularization methods can be embedded into GANs as depicted in Figure~\ref{fig:architecture_of_SSN-GAN} (a).
We implement our data representation into DCGAN~\cite{radford2015dcgan} that is the foundation of recent extensive studies, and call it as Noisy Scale-Space (NSS) GAN.
The function $\Phi$ consists of the repetition of the smoothing and the noise-injection layers (Figure~\ref{fig:architecture_of_SSN-GAN}b) that can be computed efficiently in parallel.
Regarding the annealing of the time parameter ($t$) that determines the magnitude of regularization effect, we use an exponential function as
\begin{equation} \label{eq:time-annealing}
t^{(i)} \coloneqq T \cdot \exp \left( i/\beta \right),
\end{equation}
where $i \in [0,1]$ is the relative iteration in the optimization process that starts at $i=0$, $\beta$ is the power of decay, and $T$ is the initial time $t^{(0)}$. 
We use $\beta=20$ in order to apply the implicit regularization in the early stage of GAN optimization.
Figure~\ref{fig:architecture_of_SSN-GAN} (c) shows the annealing curve of $t$ with $T=256$.
We have observed that the exponential function achieves better accuracy than others including the step function and the decaying-wave function.
Also we employ an implementation technique~\cite{jenni2019noiseGAN} in which we apply the filtering function $\Phi$ to only half of data in real-and fake-batches for obtaining a stable and accurate generator.






\def 160pt{160pt}
\def 75pt{75pt}
\begin{figure} [htb]
\centering
\small
\begin{tabular}{m{160pt} m{75pt} m{75pt}}

\includegraphics[width=160pt]{architecture_GAN}&
\includegraphics[width=75pt]{architecture_filter}&
\includegraphics[width=75pt]{schedule} \\

\multicolumn{1}{c}{(a)} &
\multicolumn{1}{c}{(b)} &
\multicolumn{1}{c}{\qquad (c)} \\
\end{tabular}
\caption{(a) The pipeline of GAN using the data-based regularization ($\Phi$). The function $\Phi$ is applied to the half of both real data ($x$) and fake data ($G(\latent)$); 
(b) The module of the noise scale-space that consists of $t$-times repetition of the convolution-layer using the Gaussian kernel with the random noise-layer; and (c) The exponential annealing of $t$ using the power of $\beta=20$ with $T = 256$.}
\label{fig:architecture_of_SSN-GAN}
\vspace{-12pt}
\end{figure}




\section{Experimental results} \label{sec:results}
\subsection{Experimental set-up}
We empirically demonstrate the expected stabilization effect of the noisy scale-space for GAN optimization.
In the preliminary experiment, we compare our NSS-GAN with potential competitors:
DCGAN (GAN) as the baseline,  GAN using the scale-space (SS-GAN), DCGAN using the noise-space (NS-GAN) where SS-GAN, NS-GAN, and NSS-GAN share the same architecture and the annealing function of $t$ except for the filtering function $\Phi$.
Moreover, we involve GAN with the gradient regularization~\cite{Mescheder2018which} (GAN-gr), LSGAN~\cite{mao2017LSGAN}, WGAN-gp~\cite{Sailmans16improved}, and Dragan~\cite{kodali2017convergence} as the state-of-the-arts in the final experiment.
\par%
\vspace{6pt}
\noindent {\bf Data-sets:} 
We use four of the major data-sets: 
CelebA~\cite{liu2015faceattributes}, 
LSUN-Church, LSUN-Conference~\cite{yu15lsun}, 
and Oxford-Flowers~\cite{Nilsback08flowers}
as the image generation tasks of faces, outdoor scenes, indoor scenes, and plants, respectively.
CelebA consists of about 200K of celebrity face images. LSUN-Church and LSUN-Conference have about 126K images of outdoor scene of churches, 
and about 224K images of indoor scene of conferences, respectively. Oxford-Flowers has about 8K images of flowers.
The images are resized to $64\times64$ pixels to meet the public implementation of DCGAN.
\par
\vspace{3pt}
\noindent {\bf Loss and hyper-parameters:} 
We use the non-saturating loss that is known to be superior to the Wasserstein loss and others~\cite{Mescheder2018which}, and employ Adam~\cite{kingma2014adam} as one of the popular optimizers in GAN studies.
For all the experiments, the mini-batch size is set to 128, and the number of training epochs is set to 10 epochs for CelebA, LSUN-Church and LSUN-Conference, and 100 epochs for Oxford-Flowers, respectively, based on their example sizes.
The std. of noise is set to $\sigma=0.15$.
The hyper-parameters that determine the quality of generated data are 
the initial time $T$ of SS, NS, and NSS-GANs,
the learning-rate scale ($\eta$) and the 1st momentum coefficient ($b_1$) of Adam,
and the regularization coefficient ($\lambda$) of GAN-gr.
We conduct our experiments in two steps: a preliminary experiment on $T$ with fixed $\eta$ and $b_1$, and the final experiment using the tuned $\eta$, $b_1$, and $\lambda$ with fixed $T$.
We perform each condition 20 individual times.
\par
\vspace{3pt}
\noindent {\bf Evaluation metrics:} 
For quantitative evaluation of generated fake images, we use the Fr\'echet inception distance (FID)~\cite{Heusel17TTUR} 
with the inception score (IS)~\cite{Sailmans16improved} that are widely accepted in GAN studies.
FID measures the distance between the real data with fake data in feature space defined using a pre-trained network.
IS measures the diversity of fakes in the feature space.
Lower FID values with higher IS indicate better quality and diversity of fake data, respectively.
We use FID as the primal metric that reflects the objective of GAN.
\subsection{Effect of initial time}
We first examine the initial time $T$ in Eq.(\ref{eq:time-annealing}) that determines the degree of the scale-space, the noise-space, and the proposed noisy scale-space.
In order to observe the relative stabilization effect of the three data representations in comparison to the baseline GAN, we purposely use a condition that can make the baseline GAN unstable for each data-set. Concretely, we employ $(\eta,b_1)=(0.0002,0.4)$ for CelebA and LSUN-Church, $(\eta,b_1)=(0.0005,0.3)$ for LSUN-Conference, and $(\eta,b_1)=(0.0005,0.4)$  for Oxford-Flowers data-sets.
\par
Table~\ref{tab:tuning-T} shows the mean FID within the 20 trials over the initial time of $T = 0,32,64,128,256$ where $T = 0$ is the baseline DCGAN.
Table~\ref{tab:tuning-T} demonstrates that both the scale-space and the noise-space improved the quality of fakes compared to the baseline GAN; and the proposed NSS-GAN has achieved a better stabilization effect than SS-GAN and NS-GAN independent of the data-sets and the initial time $T$.
Although $T$ depends on the context and the scale of images essentially, we recommend $T = 256$ and use it in the final experiment since the time $t$ should start with a large value for any type of images.







\def \rh {1.2}
\def 61pt {12pt}
\def 38pt {16pt}
\begin{table}[htb]
\caption{The Fr\'echet inception distance (FID) over the initial time $T=0,32,64,128,256$ for (column parts) CelebA, LSUN-Church, LSUN-Conference, and Oxford-Flowers by DCGAN using the scale-space (SS), the noise-space (NS), and our noisy scale-space (NSS): In order to demonstrate the stabilization effect over $T$, the learning-rate scale and the 1st momentum were fixed for each data-set such that the baseline DCGAN ($T=0$) can be unstable.
}
\label{tab:tuning-T}
\centering
\scriptsize
\renewcommand{\arraystretch}{\rh}
\begin{tabular}{p{20pt} | P{61pt}P{61pt}P{38pt} | P{61pt}P{61pt}P{38pt} | P{38pt}P{61pt}P{38pt} | P{38pt}P{61pt}P{38pt}}
\hline
& \multicolumn{3}{c|}{CelebA}  & \multicolumn{3}{c|}{Church}   & \multicolumn{3}{c|}{Conference} &  \multicolumn{3}{c}{Flowers} \\ 
& SS	& NS	& NSS	& SS	& NS	& NSS	 & SS	& NS	& NSS	 & SS	& NS	& NSS	 \\ \hline
$T$=0     & 24.60	& -	 & -	& 70.94	& -	 & -	& 127.32	& -	& -	& 201.02	& -	  & -	\\ \hline
$T$=32    & 23.14	& 22.82& 22.32	& 69.80	& 70.20	& 69.88	& 119.06	& 71.16	& 67.37 & 162.24	& 89.46	& 87.30 \\ \hline
$T$=64    & 23.16	& 23.01	& 22.43	& 69.05	& 69.60	& 69.67	& 98.73	 & 72.21	& 70.33	& 120.57	& 93.46	& 88.68	\\ \hline
$T$=128   & 23.11	& 24.02	& 21.18 & 69.38	& 69.51	& 68.59	& 84.31	 & 71.83	& 68.90	& 116.29	& 95.66	& 92.57	\\ \hline
$T$=256   & 22.31	& 24.29	& 21.79	& 70.58	& 69.61	& 67.38 & 87.46	 & 73.11	& 70.47	& 100.84	& 95.47	& 94.24	\\ 
\hline
\end{tabular}
\renewcommand{\arraystretch}{1}
\end{table}





\def \rh {0.8}
\def 61pt {22pt}
\def 38pt {38pt}
\begin{table}[htb]
\caption{The tuned hyper-parameters of (columns) the experimented GANs for (rows) CelebA, LSUN-Church, LSUN-Conference, and Oxford-Flowers data-sets: the learning-rate scale ($\eta \times 10^{-4}$ ), the 1st momentum ($b_1$) of Adam, and the regularization coefficient ($\lambda$) were selected using grid search based on the mean FID. GAN with the scale-space (SS), GAN with the noise-space (NS), and GAN with the noisy scale-space (NSS) share the fixed $T=256$.}
\label{tab:tuned_condition}
\scriptsize
\centering
\renewcommand{\arraystretch}{\rh}
\begin{tabular}{l l P{61pt}P{38pt}P{61pt}P{38pt}P{61pt}P{61pt}P{61pt}P{61pt}}          
\hline
 &  & GAN  & GAN-gr  & LSGAN  & WGAN-gp  & Dragan  & SS  & NS  & NSS \\
\hline
\parbox[t]{2mm}{\multirow{3}{*}{\rotatebox[origin=c]{90}{CelebA}}}  & $\eta$  & 5  & 5  & 1  & 10  & 1  & 5  & 5  & 5 \\
  & $b_1$  & 0.3  & 0.3  & 0.4  & 0.3  & 0.4  & 0.4  & 0.3  & 0.4 \\
  & $\lambda$  & -  & 1  & -  & 20  & 10  & -  & -  & - \\
\hline
\parbox[t]{2mm}{\multirow{3}{*}{\rotatebox[origin=c]{90}{Church}}}  & $\eta$  & 2  & 5  & 0.5  & 10  & 1  & 5  & 10  & 5 \\
  & $b_1$  & 0.3  & 0.3  & 0.5  & 0.3  & 0.4  & 0.3  & 0.3  & 0.3 \\
  & $\lambda$  & -  & 10  & -  & 20  & 10  & -  & -  & - \\
\hline
\parbox[t]{2mm}{\multirow{3}{*}{\rotatebox[origin=c]{90}{Confer.}}}  & $\eta$  & 2  & 2  & 0.5  & 10  & 2  & 2  & 5  & 5 \\
  & $b_1$  & 0.3  & 0.3  & 0.5  & 0.3  & 0.4  & 0.3  & 0.3  & 0.3 \\
  & $\lambda$  & -  & 5  & -  & 20  & 10  & -  & -  & - \\
\hline
\parbox[t]{2mm}{\multirow{3}{*}{\rotatebox[origin=c]{90}{Flowers}}}  & $\eta$  & 2  & 5  & 0.5  & 10  & 2  & 5  & 5  & 5 \\
  & $b_1$  & 0.4  & 0.4  & 0.5  & 0.3  & 0.5  & 0.3  & 0.3  & 0.3 \\
  & $\lambda$  & -  & 5  & -  & 20  & 10  & -  & -  & - \\
\hline
\end{tabular}          
\renewcommand{\arraystretch}{1}
\end{table}









\def 80pt {80pt}
\def 61pt {75pt}
\begin{figure} [h!]
\centering
\hspace*{-20pt}
\def\arraystretch{0.70}
\vspace{24pt}
\begin{tabular}{P{61pt}P{61pt}P{61pt}P{61pt}}
\includegraphics[width=80pt]{fake,celeba,Adam-0.3-0.999,std,dcgan_gr,reg,0.0,lr,0.0005.fixedLR,ep,10,tr,4} &
\includegraphics[width=80pt]{fake,celeba,Adam-0.3-0.999,std,dcgan_gr,reg,1.0,lr,0.0005.fixedLR,ep,10,tr,6} &
\includegraphics[width=80pt]{fake,celeba,lsgan,lr,0.001,0.4,reg,0,ep,10,tr,8} &
\includegraphics[width=80pt]{fake,celeba,wgan_gp,lr,0.01,0.3,reg,20.0,ep,10,tr,6} \\
\quad GAN & \quad GAN-gr & \quad LSGAN & \quad WGAN-gp\\ 
\includegraphics[width=80pt]{fake,celeba,dragan,lr,0.001,0.4,reg,10.0,ep,10,tr,2} &
\includegraphics[width=80pt]{fake,celeba,Adam-0.4-0.999,0-256.halfbatch.decay.20,dcgan_gr,reg,0.0,lr,0.0005.fixedLR,ep,10,tr,19} &
\includegraphics[width=80pt]{fake,celeba,Adam-0.3-0.999,0-256.noisebatch_0.1_20,dcgan_gr,reg,0.0,lr,0.0005.fixedLR,ep,10,tr,1} &
\includegraphics[width=80pt]{fake,celeba,Adam-0.4-0.999,0-256.sn2batch_0.1_20,dcgan_gr,reg,0.0,lr,0.0005.fixedLR,ep,10,tr,21} \\
\quad Dragan & \quad SS-GAN & \quad NS-GAN & \quad NSS-GAN \\ 
\multicolumn{4}{c}{(a) CelebA} \vspace{6pt} \\

\includegraphics[width=80pt]{fake,lsun-church,Adam-0.3-0.999,std,dcgan_gr,reg,0.0,lr,0.0002.fixedLR,ep,10,tr,13} &
\includegraphics[width=80pt]{fake,lsun-church,Adam-0.3-0.999,std,dcgan_gr,reg,10.0,lr,0.0005.fixedLR,ep,10,tr,18} &
\includegraphics[width=80pt]{fake,lsun-church,lsgan,lr,0.0005,0.5,reg,0,ep,10,tr,3} &
\includegraphics[width=80pt]{fake,lsun-church,wgan_gp,lr,0.01,0.3,reg,20.0,ep,10,tr,3} \\
\quad GAN & \quad GAN-gr & \quad LSGAN & \quad WGAN-gp\\ 
\includegraphics[width=80pt]{fake,lsun-church,dragan,lr,0.001,0.4,reg,10.0,ep,10,tr,6} &
\includegraphics[width=80pt]{fake,lsun-church,Adam-0.3-0.999,0-256.halfbatch.decay.20,dcgan_gr,reg,0.0,lr,0.0005.fixedLR,ep,10,tr,5} &
\includegraphics[width=80pt]{fake,lsun-church,Adam-0.3-0.999,0-256.noisebatch_0.1_20,dcgan_gr,reg,0.0,lr,0.001.fixedLR,ep,10,tr,24} &
\includegraphics[width=80pt]{fake,lsun-church,Adam-0.3-0.999,0-256.sn2batch_0.1_20,dcgan_gr,reg,0.0,lr,0.0005.fixedLR,ep,10,tr,3} \\
\quad Dragan & \quad SS-GAN & \quad NS-GAN & \quad NSS-GAN \\ 
\multicolumn{4}{c}{(b) Church}
\end{tabular}
\caption{Fake images created by the generator of which FID is the closest to the mean of the individual trials for 
(top part) CelebA and (bottom part) LSUN-Church data-sets using
the baseline DCGAN (GAN), 
GAN with gradient regularization (GAN-gr)~\cite{Mescheder2018which}, 
LSGAN~\cite{mao2017LSGAN},
WGAN-gp~\cite{Sailmans16improved},
(bottom)
Dragan~\cite{kodali2017convergence}, 
and GANs using the scale-space (SS), 
the noise-space (NS), 
and the proposed noisy scale-space (NSS) with $T=256$.}
\label{fig:final-comparison-images-part1}
\end{figure}
\def 80pt {80pt}
\def 61pt {75pt}
\begin{figure} [h!]
\centering
\hspace*{-20pt}
\def\arraystretch{0.70}
\vspace{24pt}
\begin{tabular}{P{61pt}P{61pt}P{61pt}P{61pt}}
\includegraphics[width=80pt]{fake,lsun-conference,Adam-0.3-0.999,std,dcgan_gr,reg,0.0,lr,0.0002.fixedLR,ep,10,tr,13} &
\includegraphics[width=80pt]{fake,lsun-conference,Adam-0.3-0.999,std,dcgan_gr,reg,5.0,lr,0.0002.fixedLR,ep,10,tr,20} &
\includegraphics[width=80pt]{fake,lsun-conference,lsgan,lr,0.0005,0.5,reg,0,ep,10,tr,10} &
\includegraphics[width=80pt]{fake,lsun-conference,wgan_gp,lr,0.01,0.3,reg,20.0,ep,10,tr,2} \\
\quad GAN & \quad GAN-gr & \quad LSGAN & \quad WGAN-gp\\ 
\includegraphics[width=80pt]{fake,lsun-conference,dragan,lr,0.002,0.4,reg,10.0,ep,10,tr,1} &
\includegraphics[width=80pt]{fake,lsun-conference,Adam-0.3-0.999,0-256.halfbatch.decay.20,dcgan_gr,reg,0.0,lr,0.0002.fixedLR,ep,10,tr,13} &
\includegraphics[width=80pt]{fake,lsun-conference,Adam-0.3-0.999,0-256.noisebatch_0.1_20,dcgan_gr,reg,0.0,lr,0.0005.fixedLR,ep,10,tr,12} &
\includegraphics[width=80pt]{fake,lsun-conference,Adam-0.3-0.999,0-256.sn2batch_0.1_20,dcgan_gr,reg,0.0,lr,0.0005.fixedLR,ep,10,tr,2} \\
\quad Dragan & \quad SS-GAN & \quad NS-GAN & \quad NSS-GAN \\ 
\multicolumn{4}{c}{(a) Conference} \vspace{6pt} \\

\includegraphics[width=80pt]{fake,oxflower,Adam-0.4-0.999,std,dcgan_gr,reg,0.0,lr,0.0002.fixedLR,ep,100,tr,4} &
\includegraphics[width=80pt]{fake,oxflower,Adam-0.4-0.999,std,dcgan_gr,reg,5.0,lr,0.0005.fixedLR,ep,100,tr,11} &
\includegraphics[width=80pt]{fake,oxflower,lsgan,lr,0.0005,0.5,reg,0,ep,100,tr,3} &
\includegraphics[width=80pt]{fake,oxflower,wgan_gp,lr,0.01,0.3,reg,20.0,ep,100,tr,6} \\
\quad GAN & \quad GAN-gr & \quad LSGAN & \quad WGAN-gp\\ 
\includegraphics[width=80pt]{fake,oxflower,dragan,lr,0.002,0.5,reg,10.0,ep,100,tr,6} &
\includegraphics[width=80pt]{fake,oxflower,Adam-0.3-0.999,0-256.halfbatch.decay.20,dcgan_gr,reg,0.0,lr,0.0005.fixedLR,ep,100,tr,15} &
\includegraphics[width=80pt]{fake,oxflower,Adam-0.3-0.999,0-256.noisebatch_0.1_20,dcgan_gr,reg,0.0,lr,0.0005.fixedLR,ep,100,tr,7} &
\includegraphics[width=80pt]{fake,oxflower,Adam-0.3-0.999,0-256.sn2batch_0.1_20,dcgan_gr,reg,0.0,lr,0.0005.fixedLR,ep,100,tr,17} \\
\quad Dragan & \quad SS-GAN & \quad NS-GAN & \quad NSS-GAN \\ 
\multicolumn{4}{c}{(b) Flowers}
\end{tabular}
\caption{Fake images created by the generator of which FID is the closest to the mean of the individual trials for 
(top part) LSUN-Conference and (bottom part) Oxford-Flowers data-sets using
the baseline DCGAN (GAN), 
GAN with gradient regularization (GAN-gr)~\cite{Mescheder2018which}, 
LSGAN~\cite{mao2017LSGAN},
WGAN-gp~\cite{Sailmans16improved},
(bottom)
Dragan~\cite{kodali2017convergence}, 
and GANs using the scale-space (SS), 
the noise-space (NS), 
and the proposed noisy scale-space (NSS) with $T=256$.}
\label{fig:final-comparison-images-part2}
\end{figure}
\def \rh {1.2}
\def 61pt {61pt}
\begin{table}[h!]         
\caption{The mean and std. of Fr\'echet inception distance (FID) and inception score (IS) within 20 individual trials using CelebA, LSUN-Church, LSUN-Conference, and Oxford-Flowers data-sets by (rows in each part) 
the baseline DCGAN (GAN),
GAN with gradient regularization (GAN-gr)~\cite{Mescheder2018which},
LSGAN~\cite{mao2017LSGAN},
WGAN-gp~\cite{Sailmans16improved},
Dragan~\cite{kodali2017convergence}, 
and GANs using the conventional scale-space (SS), the noise-space (NS), and the proposed noisy scale-space (NSS) with
$T=256$: The learning-rate and 1st momentum were tuned for each condition based on the mean FID.}         
\label{tab:final-comparison-metrics}
\centering         
\small
\renewcommand{\arraystretch}{\rh}
\begin{tabular}{l |P{61pt}P{61pt}  |P{61pt}P{61pt}}      
\hline      
 & \multicolumn{2}{c|}{CelebA}  & \multicolumn{2}{c}{Church}   \\
 & FID ($\downarrow$) & IS ($\uparrow$)  & FID & IS  \\
 \hline
GAN	 & 21.36 $\pm$ 1.71	 & 2.39 $\pm$ 0.06	 & 69.05 $\pm$ 4.65	 & 2.98 $\pm$ 0.07		\\
GAN-gr	 & 19.87 $\pm$ 1.19	 & 2.35 $\pm$ 0.05	 & 61.67 $\pm$ 5.07	 & 2.97 $\pm$ 0.11		\\
LSGAN	 & 33.38 $\pm$ 5.10	 & 2.28 $\pm$ 0.06	 & 75.88 $\pm$ 9.56	 & 3.02 $\pm$ 0.09		\\
WGAN-gp	 & 42.16 $\pm$ 2.77	 & 2.45 $\pm$ 0.07	 & 102.61 $\pm$ 18.48 & 2.90 $\pm$ 0.14		\\
Dragan	 & 20.36 $\pm$ 1.25	 & 2.35 $\pm$ 0.05	 & \textbf{50.41} $\pm$ 2.68	 & 2.91 $\pm$ 0.04		\\
SS-GAN	 & 21.49 $\pm$ 1.49	 & 2.37 $\pm$ 0.05	 & 61.20 $\pm$ 3.53	 & 2.97 $\pm$ 0.08		\\
NS-GAN	 & 20.50 $\pm$ 1.14	 & 2.37 $\pm$ 0.04	 & 60.02 $\pm$ 4.47	 & 2.98 $\pm$ 0.08		\\
NSS-GAN	 & \textbf{19.45} $\pm$ 1.15	 & 2.42 $\pm$ 0.05	 & 58.86 $\pm$ 3.92	 & 2.97 $\pm$ 0.08		\\
\hline      
 & \multicolumn{2}{c|}{Conference}  & \multicolumn{2}{c}{Flowers}   \\
 & FID & IS & FID & IS  \\
  \hline
GAN	 & 77.96 $\pm$ 5.49	 & 4.19 $\pm$ 0.11	 & 97.71 $\pm$ 4.91	 & 2.98 $\pm$ 0.09		\\
GAN-gr	 & 70.82 $\pm$ 3.80	 & 4.08 $\pm$ 0.11	 & 99.25 $\pm$ 5.59	 & 3.15 $\pm$ 0.09		\\
LSGAN	 & 81.90 $\pm$ 9.49	 & 4.01 $\pm$ 0.11	 & 121.81 $\pm$ 8.50	 & 2.72 $\pm$ 0.15		\\
WGAN-gp	 & 121.05 $\pm$ 7.60	 & 3.51 $\pm$ 0.13	 & 129.05 $\pm$ 7.70	 & 2.89 $\pm$ 0.08		\\
Dragan	 & 67.07 $\pm$ 5.45	 & 4.18 $\pm$ 0.16	 & 98.37 $\pm$ 6.63	 & 3.00 $\pm$ 0.09		\\
SS-GAN	 & 80.07 $\pm$ 5.44	 & 4.02 $\pm$ 0.10	 & 91.93 $\pm$ 5.12	 & 3.11 $\pm$ 0.09		\\
NS-GAN	 & 73.11 $\pm$ 4.07	 & 4.02 $\pm$ 0.08	 & 91.44 $\pm$ 4.47	 & 3.07 $\pm$ 0.08		\\
NSS-GAN	 & \textbf{64.61} $\pm$ 3.20	 & 4.21 $\pm$ 0.14	 & \textbf{86.28} $\pm$ 6.57	 & 3.09 $\pm$ 0.10		\\
\hline      
\end{tabular}      
\renewcommand{\arraystretch}{1}
\vspace{12pt}
\end{table}         



\subsection{Comparison to state-of-the-arts}
We now compare the proposed NSS-GAN with the state-of-the-arts:
the baseline GAN (GAN),
GAN with the gradient regularization~\cite{Mescheder2018which},
LSGAN~\cite{mao2017LSGAN},
WGAN with gradient penalty (WGAN-gp)~\cite{Sailmans16improved},
Dragan~\cite{kodali2017convergence}, 
GANs using the scale-space (SS-GAN),
and the noise-space (NS-GAN) using the four of data-sets.
In order to make our result independent to the hyper parameters of Adam, we employ grid search and tuned the learning-rate scale ($\eta$) in combination with the 1st momentum coefficient ($b_1$) for each pair of model and data-set, while the 2nd momentum of Adam was fixed as $b_2=0.999$ based on our pretest.
For each model, we then choose the best condition using the mean FID within the 20 trials. Table~\ref{tab:tuned_condition} summarizes the tuned parameters in which the baseline GAN prefers a stable condition compared to the GANs using explicit and implicit stabilization methods.
\par
Figure~\ref{fig:final-comparison-images-part1} and Figure~\ref{fig:final-comparison-images-part2} present fake images by the tested GANs with their tuned hyper-parameters, where we use the generator of which FID is the closest to the mean of the independent trials.
Table~\ref{tab:final-comparison-metrics} summarizes the mean and std. of FID and IS of the experimented GANs for the four data-sets within the 20 trials, where the proposed NSS-GAN with the constant parameter of $T=256$ has achieved better and comparable results than the state-of-the-arts, demonstrating the effectiveness of the presented NSS-GAN.
\par
More importantly, Table~\ref{tab:final-comparison-metrics} shows that the conventional scale-space (SS) GAN and the noise-space (NS) GAN were not consistently better than the baseline GAN.
In contrast, the proposed NSS-GAN has consistently outperformed the baseline GAN, SS-GAN, and NS-GAN in FID. This indicates that the stabilization effect of the presented NSS-GAN is not the simple summation of those by the scale-space with the noise-space but due to the better use of their mutually complementary relationship in the GAN optimization.
\section{Conclusion} \label{sec:conclusion}
We have considered the general framework of implicit regularization for GAN optimization based on data representation.
Then we have proposed a discrete representation of data, called noisy scale-space, that gradually removes high-frequency information in image while imposing noise, leading to a coarse-to-fine training of GANs.
In order to observe the side-effect of the conventional scale-space in GAN optimization, we have also proposed the synthetic data-set based on the Hadamard bases that visualizes the true distribution of the real and fake data.
We have experimented with a DCGAN with the noisy scale-space (NSS-GAN) using the major data-sets for natural image generation tasks.
The experimental results have successfully demonstrated that: NSS-GAN overtook the potential competitors and the state-of-the-arts in most cases; and the regularization effect by NSS-GAN is not the simple summation of the noise-space with the scale-space.
\par
A limitation of our method is that we assume the smoothing can simplify (the real) data. Concretely, we found that our NSS-GAN is inferior to the original DCGAN with the tuned hyper-parameters when using MNIST~\cite{lecun1998gradient} of which images consist of 0/1 binary values. Applying the smoothing kernel to the binary pixels produces pixels with other values, e.g. 0.5, and increases the diversity of pixel values. Fortunately, this assumption holds for natural images and our method yields the sufficient stabilization effect for GAN optimization irrespective of the content of images.




