\section{Introduction}\label{sec:introduction}}
	
	\textit{Conditional generative adversarial networks} (cGANs), first proposed in \cite{mirza2014conditional}, aim to estimate the distribution of images conditioning on some auxiliary information, especially class labels. Subsequent studies \cite{odena2017conditional, miyato2018cgans, brock2018large, SAGAN-zhang19d} confirm the feasibility of generating diverse, high-quality (even photo-realistic), and class-label consistent fake images from class-conditional GANs. Unfortunately, these cGANs are not applicable for conditional generative modeling with continuous, scalar conditions, termed \textit{regression labels}, due to two problems:
	
	{\setlength{\parindent}{0cm}\textbf{(P1)}} cGANs are often trained to minimize the empirical versions of their losses (a.k.a. empirical cGAN losses) on some training data, a principle also known as \textit{empirical risk minimization} (ERM) \cite{vapnik2000nature, mohri2018foundations, shalevshwartz2014understanding}. The success of ERM relies on a large sample size for each distinct condition. Unfortunately, we usually have only a few real images for some regression labels. Moreover, since regression labels are continuous, some values may not even appear in the training set. Consequently, a cGAN cannot accurately estimate the image distribution conditional on such missing labels.
	
	{\setlength{\parindent}{0cm}\textbf{(P2)}} In the class-conditional generative modeling, class labels are often encoded by one-hot vectors or label embedding and then fed into the generator and discriminator by hidden concatenation \cite{mirza2014conditional}, an auxiliary classifier \cite{odena2017conditional} or label projection \cite{miyato2018cgans}. A precondition for such label encoding is that the number of distinct labels (e.g., the number of classes) is finite and known. Unfortunately, in the continuous scenario, we may have infinitely many distinct regression labels. 
	
	A naive approach denoted by \textbf{cGAN ($\bm{K}$ classes)} to solve \textbf{(P1)-(P2)} is to ``bin'' the regression labels into $K$ disjoint intervals and still train a cGAN in the class-conditional manner (these intervals are treated as independent classes) \cite{Greg2019}. Another naive approach denoted by \textbf{cGAN (concat)} for solving \textbf{(P2)} directly combines a regression label with the input or a hidden map of the generator and discriminator. However, both approaches fails in our empirical studies in Sections ~\ref{sec:experiment} and \ref{sec:experiment_hd}, representing two types of failures of conventional cGANs: (1) cGAN ($\bm{K}$ classes) cannot generate visually realistic and diverse images; (2) cGAN (concat) fails to control the image generation by setting regression labels.
	
	In machine learning, \textit{vicinal risk minimization} (VRM) \cite{vapnik2000nature, chapelle2001vicinal} is an alternative rule to ERM. VRM assumes that a sample point shares the same label with other samples in its vicinity. Motivated by VRM, in generative modeling conditional on regression labels where we estimate a conditional distribution $p(\bm{x}|y)$ ($\bm{x}$ is an image and $y$ is a regression label), it is natural to assume that a small perturbation to $y$ results in a negligible change to $p(\bm{x}|y)$. This assumption is consistent with our perception of the world. For example, the image distribution of facial features for a population of 15-year-old teenagers should be close to that of 16-year olds. 
	
	We therefore introduce the \textit{continuous conditional GAN} (CcGAN) to tackle \textbf{(P1)} and \textbf{(P2)}. To our best knowledge, this is the first generative model for image data conditional on regression labels. It is noted that \cite{semiGAN} and \cite{2018reg} train GANs in an unsupervised manner and synthesize unlabeled fake images for a subsequent image regression task. \cite{olmschenk2019dense} proposes a semi-supervised GAN for dense crowd counting. \cite{dosovitskiy2016learning} uses a \textit{convolutional neural network} (CNN) to generate images of objects in terms of some high-level parameters such as object style, viewpoint, color, brightness, etc. \cite{infogan2016} proposes InfoGAN, which can control some continuous/discrete attributes of generated images. Some text-to-image generation methods \cite{yan2016attribute2image, NEURIPS2019_1d72310e, tao2020df} train generative models conditional on high-dimensional attribute vectors with continuous or discrete elements. The objectives of these works are entirely different from ours since they do not aim to estimate the image distribution conditional on regression labels. Moreover, some recent works \cite{zhao2020differentiable, karras2020training, tran2020data, zhao2020image} propose several novel schemes to train GANs when training data are limited, which seems to be relevant to \textbf{(P1)}. However, they are also fundamentally different from CcGAN, since they are designed for unconditional and class-conditional scenarios rather than continuous ones. Our contributions can be summarized as follows:
	
	\begin{itemize}[leftmargin=0.4cm]
		
		\item We propose in Section \ref{sec:solution_1} a solution to address \textbf{(P1)}, which consists of two novel empirical discriminator losses, termed the \textit{hard vicinal discriminator loss} (HVDL) and the \textit{soft vicinal discriminator loss} (SVDL), and a novel empirical generator loss. We take the vanilla cGAN loss as an example to show how to derive HVDL, SVDL, and the novel empirical generator loss by reformulating existing empirical cGAN losses.
		
		\item In Section \ref{sec:solution_2}, we propose two novel label input mechanisms, consisting of a \textit{naive label input} (NLI) mechanism and an \textit{improved label input} (ILI) mechanism, as solutions to address \textbf{(P2)}.
		
		\item We derive in Section \ref{sec:convergence_rate} the error bounds of a discriminator trained with HVDL and SVDL. These error bounds not only help us understand how HVDL and SVDL influence the discriminator training but also guide our implementation in practice (especially the selection of hyper-parameters).
		
		\item We propose in Section \ref{sec:sfid} a novel evaluation metric, termed \textit{Sliding Fr\'echet Inception Distance} (SFID), to evaluate the generative image modeling conditional on regression labels when there are insufficient real images to compute Intra-FID \cite{miyato2018cgans}. 
		
		\item In Sections \ref{sec:experiment} and \ref{sec:experiment_hd}, we propose two new benchmark datasets, RC-49 and Cell-200, for generative image modeling conditional on regression labels, since very few benchmark datasets are suitable for the studied continuous scenario. We conduct extensive experiments on four benchmark datasets with various resolutions (from $64\times 64$ to $256\times 256$) to demonstrate that CcGAN not only generates diverse, high-quality, and label consistent images, but also substantially outperforms cGAN both visually and quantitatively. The effectiveness of SFID is also studied on the RC-49 dataset at the end of Section \ref{sec:experiment}.
	\end{itemize}
	
	A preliminary version of this paper has been presented in International Conference on Learning Representations \cite{ding2021ccgan}. This paper improves the initial version from several aspects to strengthen our method. (1) We propose an improved label input (ILI) mechanism to better incorporate regression labels into CcGAN. Our experiments in this paper demonstrate the superiority of ILI to the naive label input method in \cite{ding2021ccgan}. (2) We introduce Lemmas~\ref{lem:conditional_mean_bound_HVDL} and \ref{lem:conditional_mean_bound_SVDL}, which are used to derive the error bounds of HVDL and SVDL but are omitted in \cite{ding2021ccgan}. The motivation of deriving these error bounds are also better illustrated, and an improved proof for the derivation is provided in Appendix. (3) We propose SFID to replace Intra-FID when Intra-FID is not applicable. (4) We create a new benchmark dataset called Cell-200 for generaive modeling conditional on regression labels. (5) We conduct a more extensive empirical study to demonstrate the effectiveness of CcGAN. This extensive study includes more datasets (e.g., Cell-200 and Steering Angle) and more complicated settings (e.g., various image resolutions). We also add a new baseline, cGAN (concat), to the comparison to better demonstrate that conventional cGANs are inapplicable in our task.
	
	
	
	


	\section{From cGAN to CcGAN}\label{sec:cGAN2CcGAN}
	In this section, we introduce the \textit{continuous conditional GAN} (CcGAN), consisting of solutions to \textbf{(P1)} and \textbf{(P2)}. The combinations of two vicinal discriminator losses (HVDL and SVDL) proposed in Section \ref{sec:solution_1} and two novel label input mechanisms (NLI and ILI) proposed in Section \ref{sec:solution_2} result in four CcGAN methods denoted by HVDL+NLI, SVDL+NLI, HVDL+ILI, and SVDL+ILI, respectively.  {The overall workflow of CcGAN is visualized in Fig.\ \ref{fig:workflow_CcGAN}.} 
	
	\begin{figure}[!htbp]
		\centering
		\includegraphics[width=0.8\textwidth]{./workflow_CcGAN.png}
		\caption{{  A typical workflow of the proposed CcGAN framework. A regression label $y$ is input into the generator ($G$) and the discriminator ($D$) by novel label input mechanisms proposed in Section \ref{sec:solution_2}. Novel empirical losses proposed in Section \ref{sec:solution_1} are adopted to optimize $G$ and $D$, respectively. The CcGAN framework is compatible with modern GAN architectures (e.g., SNGAN \cite{miyato2018spectral}, SAGAN \cite{SAGAN-zhang19d}) and advanced GAN training techniques (e.g., DiffAugment \cite{zhao2020differentiable}).}}
		\label{fig:workflow_CcGAN}
	\end{figure}
	
	

	\subsection{Solution to (P1): Reformulated Empirical Losses }\label{sec:solution_1}
	Theoretically, cGAN losses (e.g., the vanilla cGAN loss \cite{mirza2014conditional}, the Wasserstein loss \cite{pmlr-v70-arjovsky17a, gulrajani2017improved}, and the hinge loss \cite{miyato2018spectral}) are suitable for both class labels and regression labels; however, their empirical versions fail in the continuous scenario (i.e., \textbf{(P1)}). Our first solution \textbf{(S1)} focuses on reformulating these empirical cGAN losses for continuous labels. Without loss of generality, we only take the vanilla cGAN loss as an example to show such reformulation (the empirical versions of the Wasserstein loss and the hinge loss can be reformulated similarly). 
	
	The vanilla discriminator loss and generator loss \cite{mirza2014conditional} are defined as:
	\begin{align}
		\mathcal{L}(D) = & -\mathbb{E}_{y\sim p_r(y)}\left[\mathbb{E}_{\bm{x}\sim p_r(\bm{x}|y)}\left[ \log \left( D(\bm{x},y) \right) \right] \right] - \mathbb{E}_{y\sim p_g(y)}\left[\mathbb{E}_{\bm{x}\sim p_g(\bm{x}|y)}\left[ \log \left( 1 - D(\bm{x},y) \right) \right] \right] \nonumber \\
		= & -\int \log(D(\bm{x},y))p_r(\bm{x},y) d\bm{x}dy - \int \log(1-D(\bm{x},y))p_g(\bm{x},y)d\bm{x}dy, \label{eq:loss_cD} \\
	
		\mathcal{L}(G) = &-\mathbb{E}_{y\sim p_g(y)}\left[\mathbb{E}_{\bm{z}\sim q(\bm{z})}\left[ \log \left( D(G(\bm{z},y),y) \right) \right] \right] \nonumber \\
		= & -\int \log(D(G(\bm{z},y),y)) q(\bm{z})p_g(y) d\bm{z}dy, \label{eq:loss_cG}
	\end{align}
	where $\bm{x}\in\mathcal{X}$ is an image, $y\in\mathcal{Y}$ is a label, $p_r(y)$ and $p_g(y)$ are respectively the true and fake label marginal distributions, $p_r(\bm{x}|y)$ and $p_g(\bm{x}|y)$ are respectively the true and fake image distributions conditional on $y$, $p_r(\bm{x}, y)$ and $p_g(\bm{x}, y)$ are respectively the true and fake joint distributions of $\bm{x}$ and $y$, and $q(\bm{z})$ is the probability density function of $\mathcal{N}(\bm{0},\bm{I})$.
	
	Since the distributions in the losses of Eqs.\ \eqref{eq:loss_cD} and \eqref{eq:loss_cG} are unknown, for class-conditional  {generative modeling}, \cite{mirza2014conditional} follows ERM and minimizes the empirical losses:
	\begin{equation}
		\label{eq:loss_cD_emp}
		\widehat{\mathcal{L}}^{\delta}(D)  = -\frac{1}{N^r}\sum_{c=1}^C \sum_{j=1}^{N_c^r}\log(D(\bm{x}_{c,j}^r, c))  - \frac{1}{N^g}\sum_{c=1}^C\sum_{j=1}^{N_c^g}\log(1-D(\bm{x}_{c,j}^g, c)),
	\end{equation}
	
	\begin{equation}
		\label{eq:loss_cG_emp}
		\widehat{\mathcal{L}}^{\delta}(G) = - \frac{1}{N^g}\sum_{c=1}^C\sum_{j=1}^{N_c^g}\log (D(G(\bm{z}_{c,j}, c), c)),
	\end{equation}
	where $C$ is the number of classes, $N^r$ and $N^g$ are respectively the number of real and fake images, $N_c^r$ and $N_c^g$ are respectively the number of real and fake images with label $c$, $\bm{x}_{c,j}^r$ and $\bm{x}_{c,j}^g$ are respectively the $j$-th real image and the $j$-th fake image with label $c$,  and the $\bm{z}_{c,j}$ are independently and identically sampled from $q(\bm{z})$. Eq.\ \eqref{eq:loss_cD_emp} implies we estimate $p_r(\bm{x},y)$ and $p_g(\bm{x}, y)$ by their empirical probability density functions as follows:
	\begin{equation}
		\label{eq:joint_emp_esti}
		\begin{aligned}
			\hat{p}_r^{\delta}(\bm{x},y) & = \frac{1}{N^r}\sum_{c=1}^C\sum_{j=1}^{N_c^r}\delta(\bm{x}-\bm{x}_{c,j}^r)\delta(y-c),\\
			\hat{p}_g^{\delta}(\bm{x},y) & = \frac{1}{N^g}\sum_{c=1}^C\sum_{j=1}^{N_c^g}\delta(\bm{x}-\bm{x}_{c,j}^g)\delta(y-c),
		\end{aligned}
	\end{equation}
	where $\delta(\cdot)$ is a Dirac delta function (Appendix A of \cite{salasnich2014quantum}) centered at 0. However, $\hat{p}_r^{\delta}(\bm{x},y)$ and $\hat{p}_g^{\delta}(\bm{x},y)$ in Eq.\ \eqref{eq:joint_emp_esti} are not good estimates in the continuous scenario because of \textbf{(P1)}.
	
	To overcome \textbf{(P1)}, we propose a novel estimate for each of $p_r(\bm{x},y)$ and $p_g(\bm{x},y)$, termed the \textit{hard vicinal estimate\/} (HVE). We also provide an intuitive alternative to HVE, named the \textit{soft vicinal estimate\/} (SVE). The HVEs of $p_r(\bm{x},y)$ and $p_g(\bm{x},y)$ are:
	\begin{align}
		\hat{p}_r^{\text{HVE}}(\bm{x},y) = & C_1\cdot\left[\frac{1}{N^r}\sum_{j=1}^{N^r}\exp{\left(-\frac{(y-y_j^r)^2}{2\sigma^2}\right)} \right]  \cdot \left[ \frac{1}{N^r_{y,\kappa}}\sum_{i=1}^{N^r}\mathbbm{1}_{\{|y-y_i^r|\leq\kappa\}}\delta(\bm{x}-\bm{x}_i^r) \right], \label{eq:HVE_joint_estiamte_dist_real} \\
		\hat{p}_g^{\text{HVE}}(\bm{x},y) = & C_2\cdot\left[\frac{1}{N^g}\sum_{j=1}^{N^g}\exp{\left(-\frac{(y-y_j^g)^2}{2\sigma^2}\right)} \right] \cdot \left[ \frac{1}{N^g_{y,\kappa}}\sum_{i=1}^{N^g}\mathbbm{1}_{\{|y-y_i^g|\leq\kappa\}}\delta(\bm{x}-\bm{x}_i^g) \right], \label{eq:HVE_joint_estiamte_dist_fake}
	\end{align}
	where $\bm{x}_i^r$ and $\bm{x}_i^g$ are respectively real image $i$ and fake image $i$, $y_i^r$ and $y_i^g$are respectively the labels of $\bm{x}_i^r$ and $\bm{x}_i^g$, $\kappa$ and $\sigma$ are two positive hyper-parameters, $C_1$ and $C_2$ are two constants making these two estimates valid probability density functions, $N^r_{y,\kappa}$ is the number of the $y_i^r$ satisfying $|y-y_i^r|\leq\kappa$, $N^g_{y,\kappa}$ is the number of the $y_i^g$ satisfying $|y-y_i^g|\leq\kappa$, and $\mathbbm{1}$ is an indicator function with support in the subscript. The terms in the first square brackets of $\hat{p}_r^{\text{HVE}}$ and $\hat{p}_g^{\text{HVE}}$ imply we estimate the marginal label distributions $p_r(y)$ and $p_g(y)$ by \textit{kernel density estimates} (KDEs) \cite{davis2011remarks, parzen1962estimation, silverman1986density, hastie2009elements}. The terms in the second square brackets are designed based on the assumption that a small perturbation to $y$ results in negligible changes to $p_r(\bm{x}|y)$ and $p_g(\bm{x}|y)$. If this assumption holds, we can use images with labels in a small vicinity of $y$ to estimate $p_r(\bm{x}|y)$ and $p_g(\bm{x}|y)$. The SVEs of $p_r(\bm{x},y)$ and $p_g(\bm{x},y)$ are:
	\begin{align}
		\hat{p}_r^{\text{SVE}}(\bm{x},y) = & C_3\cdot\left[\frac{1}{N^r}\sum_{j=1}^{N^r}\exp{\left(-\frac{(y-y_j^r)^2}{2\sigma^2}\right)} \right] \cdot \left[\frac{\sum_{i=1}^{N^r}w^r(y_i^r,y)\delta(\bm{x}-\bm{x}_i^r)}{\sum_{i=1}^{N^r}w^r(y_i^r,y)} \right], \label{eq:SVE_joint_estiamte_dist_real}\\
		\hat{p}_g^{\text{SVE}}(\bm{x},y) = & C_4\cdot\left[\frac{1}{N^g}\sum_{j=1}^{N^g}\exp{\left(-\frac{(y-y_j^g)^2}{2\sigma^2}\right)} \right] \cdot \left[ \frac{\sum_{i=1}^{N^g}w^g(y_i^g,y)\delta(\bm{x}-\bm{x}_i^g)}{\sum_{i=1}^{N^g}w^g(y_i^g,y)} \right], \label{eq:SVE_joint_estiamte_dist_fake} 
	\end{align}
	where $C_3$ and $C_4$ are two constants making these two estimates valid probability density functions, 
	\begin{equation}
		\label{eq:SVE_weights}
		w^r(y_i^r, y) =  e^{-\nu(y_i^r-y)^2} \quad \text{and} \quad  w^g(y_i^g, y)  =  e^{-\nu(y_i^g-y)^2},
	\end{equation}
	and the hyper-parameter $\nu>0$. In Eqs.\ \eqref{eq:SVE_joint_estiamte_dist_real} and \eqref{eq:SVE_joint_estiamte_dist_fake}, similar to the HVEs, we estimate $p_r(y)$ and $p_g(y)$ by KDEs. Instead of using samples in a hard vicinity, the SVEs use all respective samples to estimate $p_r(\bm{x}|y)$ and $p_g(\bm{x}|y)$ but each sample is assigned with a weight based on the distance of its label from $y$. Two diagrams in Fig.\ \ref{fig:visualization_HVE_and_SVE} visualize the process of using hard/soft vicinal samples to estimate the Gaussian distribution $p(\bm{x}|y)$ conditional on $y$, for univariate $\bm{x}$.
	
	
	\begin{figure}[!htbp]
		\captionsetup[subfloat]{farskip=2pt,captionskip=1pt}
		\centering
		\subfloat[Hard Vicinity]{\includegraphics[width=0.49\textwidth]{./visualization_HVE.png}\label{fig:visualization_HVE}}  
		\hspace{2mm}
		\subfloat[Soft Vicinity]{\includegraphics[width=0.49\textwidth]{./visualization_SVE.png}\label{fig:visualization_SVE}}   
		\caption{ HVE (Eqs.\ \eqref{eq:HVE_joint_estiamte_dist_real} and \eqref{eq:HVE_joint_estiamte_dist_fake}) and SVE (Eqs.\ \eqref{eq:SVE_joint_estiamte_dist_real} and \eqref{eq:SVE_joint_estiamte_dist_fake}) estimate $p(\bm{x}|y)$ (a univariate Gaussian conditional on $y$) by using two samples in hard and soft vicinities, respectively, of $y$. To estimate $p(\bm{x}|y)$ (the red Gaussian curve) only from samples drawn from $p(\bm{x}|y_1)$ and $p(\bm{x}|y_2)$ (the blue Gaussian curves), estimation is based on the samples (red dots) in a hard vicinity (defined by $y\pm \kappa$) or a soft vicinity (defined by the weight decay curve) around $y$. The histograms in blue are samples in the hard or soft vicinity. The labels $y_1$, $y$, and $y_2$ on the $x$-axis denote the means of $\bm{x}$ conditional on $y_1$, $y$, and $y_2$, respectively. }
		\label{fig:visualization_HVE_and_SVE}
	\end{figure}
	
	
	By plugging Eq.\ \eqref{eq:HVE_joint_estiamte_dist_real}, \eqref{eq:HVE_joint_estiamte_dist_fake}, \eqref{eq:SVE_joint_estiamte_dist_real}, and \eqref{eq:SVE_joint_estiamte_dist_fake} into Eq.\ \eqref{eq:loss_cD}, we derive the \textit{hard vicinal discriminator loss} (HVDL) and the \textit{soft vicinal discriminator loss} (SVDL) as follows:
	\begin{equation}
		\label{eq:HVDL}
		\begin{aligned}
			& \widehat{\mathcal{L}}^{\text{HVDL}}(D) \\
			= & - \frac{C_5}{N^r}\sum_{j=1}^{N^r}\sum_{i=1}^{N^r}\mathbb{E}_{\epsilon^r\sim\mathcal{N}(0,\sigma^2)}\left[ W_1\log(D(\bm{x}_i^r, y_j^r+\epsilon^r)) \right]\\
			&- \frac{C_6}{N^g}\sum_{j=1}^{N^g}\sum_{i=1}^{N^g}\mathbb{E}_{\epsilon^g\sim\mathcal{N}(0,\sigma^2)}\left[ W_2\log(1-D(\bm{x}_i^g, y_j^g+\epsilon^g)) \right],
		\end{aligned}
	\end{equation}
	\begin{equation}
		\label{eq:SVDL}
		\begin{aligned}
			& \widehat{\mathcal{L}}^{\text{SVDL}}(D) \\
			= & - \frac{C_7}{N^r}\sum_{j=1}^{N^r}\sum_{i=1}^{N^r}\mathbb{E}_{\epsilon^r\sim\mathcal{N}(0,\sigma^2)}\left[W_3\log(D(\bm{x}_i^r, y_j^r+\epsilon^r)) \right]\\
			&- \frac{C_8}{N^g}\sum_{j=1}^{N^g}\sum_{i=1}^{N^g}\mathbb{E}_{\epsilon^g\sim\mathcal{N}(0,\sigma^2)}\left[ W_4\log(1-D(\bm{x}_i^g, y_j^g+\epsilon^g)) \right],
		\end{aligned}
	\end{equation}
	where $\epsilon^r\triangleq y-y_j^r$, $\epsilon^g\triangleq y-y_j^g$,
	\begin{align}
		& W_1=\frac{\mathbbm{1}_{\{|y_j^r+\epsilon^r-y_i^r|\leq\kappa\}}}{N_{y^r_j+\epsilon^r,\kappa}^r},\quad W_2=\frac{\mathbbm{1}_{\{|y_j^g+\epsilon^g-y_i^g|\leq\kappa\}}}{N_{y^g_j+\epsilon^g,\kappa}^g} \nonumber \\ 
		& 
		W_3=\frac{w^r(y_i^r,y_j^r+\epsilon^r)}{\sum_{i=1}^{N^r}w^r(y_i^r,y_j^r+\epsilon^r)},\quad
		W_4=\frac{w^g(y_i^g,y_j^g+\epsilon^g)}{\sum_{i=1}^{N^g}w^g(y_i^g,y_j^g+\epsilon^g)}, \nonumber
	\end{align}
	and $C_5$, $C_6$, $C_7$, and $C_8$ are some constants. 
	
	\textbf{Generator training:} The generator of CcGAN is trained by minimizing Eq. \eqref{eq:loss_G_CcGAN},
	\begin{equation}
		\label{eq:loss_G_CcGAN}
		\widehat{\mathcal{L}}^{\epsilon}(G) = - \frac{1}{N^g}\sum_{i=1}^{N^g}\mathbb{E}_{\epsilon^g\sim\mathcal{N}(0,\sigma^2)}\log (D(G(\bm{z}_{i}, y_i^g+\epsilon^g), y_i^g+\epsilon^g)).
	\end{equation}
	
	\textbf{How do HVDL, SVDL, and Eq.\ \eqref{eq:loss_G_CcGAN} overcome \textbf{(P1)}?}
	
	{\setlength{\parindent}{0cm}\textbf{(i)}} Given a label $y$ as the condition, we use images in a hard/soft vicinity of $y$ to train the discriminator instead of just using images with label $y$. It enables us to estimate $p_r(\bm{x}|y)$ when there are not enough real images with label $y$.
	
	{\setlength{\parindent}{0cm}\textbf{(ii)}} From Eqs.\ \eqref{eq:HVDL} and \eqref{eq:SVDL}, we can see that the KDEs in Eqs.\ \eqref{eq:HVE_joint_estiamte_dist_real}, \eqref{eq:HVE_joint_estiamte_dist_fake}, \eqref{eq:SVE_joint_estiamte_dist_real}, and \eqref{eq:SVE_joint_estiamte_dist_fake} are adjusted by adding Gaussian noise to the labels. Moreover, in Eq.\ \eqref{eq:loss_G_CcGAN}, we add Gaussian noise to seen labels (assume the $y_i^g$ are seen) to train the generator to generate images at unseen labels. This enables estimation of $p_r(\bm{x}|y^\prime)$ when $y^\prime$ is not in the training set.
	
	
	\begin{remark}
		Based on the kernel density estimation \cite{davis2011remarks, parzen1962estimation, silverman1986density, hastie2009elements} and the property of the Dirac delta function (Appendix A of \cite{salasnich2014quantum}), $\int\delta(\bm{x}-\bm{x}^r_i)d\bm{x}=\int\delta(\bm{x}-\bm{x}^g_i)d\bm{x}=1$ and $C_1=C_2=C_3=C_4=1/\sigma$. Therefore, $C_5=C_6$ and $C_7=C_8$, which implies these constants $C_1,\ldots, C_8$ can be ignored when minimizing $\widehat{\mathcal{L}}^{\text{HVDL}}(D)$ and $\widehat{\mathcal{L}}^{\text{SVDL}}(D)$.
	\end{remark}
	
	\begin{remark}
		An algorithm is proposed in Supp. \ref{supp:algorithm_train} for training CcGAN in practice. Moreover, CcGAN does not require any specific network architecture, so it can use modern GAN architectures such as SNGAN \cite{miyato2018spectral}, SAGAN \cite{SAGAN-zhang19d} and BigGAN \cite{brock2018large}. CcGAN is also compatible with modern GAN training techniques such as DiffAugment \cite{zhao2020differentiable}. 
	\end{remark}
	
	\begin{remark}[A rule of thumb for hyper-parameter selection]
		\label{rmk:rule_of_thumb}
		 {In our experiments, we normalize labels to real numbers in $[0,1]$ and the hyper-parameter selection is conducted based on the normalized labels. To be more specific, the hyper-parameter $\sigma$ is computed based on a rule of thumb formula for the bandwidth selection of KDE \cite{silverman1986density}, i.e., $\sigma=\left({4\hat{\sigma}^5_{y^r}}/{3N^r}\right)^{1/5}$, where $\hat{\sigma}_{y^r}$ is the sample standard deviation of normalized labels in the training set. Let $\kappa_{\text{base}}=\max\left(y^r_{[2]}-y^r_{[1]}, y^r_{[3]}-y^r_{[2]}, \dots, y^r_{[N^r_{\text{uy}}]}-y^r_{[N^r_{\text{uy}}-1]} \right)$, where $y^r_{[l]}$ is the $l$-th smallest normalized distinct real label and $N^r_{\text{uy}}$ is the number of normalized distinct labels in the training set. The $\kappa$ is set as a multiple of $\kappa_{\text{base}}$ (i.e., $\kappa=m_{\kappa}\kappa_{\text{base}}$) where the multiplier $m_{\kappa}$ stands for 50\% of the minimum number of neighboring labels used for estimating $p_r(\bm{x}|y)$ given a label $y$. For example, $m_{\kappa}=1$ implies using 2 neighboring labels (one on the left while the other one on the right). In our experiments, $m_{\kappa}$ is generally set as 1 or 2. In some extreme case when many distinct labels have too few real samples, we may consider increasing $m_{\kappa}$. We also found $\nu=1/\kappa^2$ works well in our experiments.}
	\end{remark}
	
	

	\subsection{Solutions to (P2): Novel Label Input Mechanisms}\label{sec:solution_2} 
	In this section, we propose two solutions, consisting of a naive and an improved label input mechanism, to solve \textbf{(P2)}. 
	
	{\setlength{\parindent}{0cm}\textbf{A naive label input (NLI) mechanism:}} We first propose a naive approach to incorporate the regression labels into the cGANs. For $G$, we add the label $y$ element-wisely to the output of its first linear layer. For $D$, the label $y$ is first projected to the latent space learned by an extra linear layer. Then, we incorporate the embedded label into the discriminator by label projection \cite{miyato2018cgans}. Fig.\ \ref{fig:naive_label_input} visualizes the naive label input mechanism.   
		
	\begin{figure}[!htbp]
		\centering
		\includegraphics[width=0.8\textwidth]{./naive_label_input.png}
		\caption{{  The workflow of the naive label input (NLI) mechanism. NLI inputs a regression label $y$ into $G$ by adding $y$ element-wisely to the output of the first linear layer. NLI inputs $y$ into $D$ by label projection \cite{miyato2018cgans}.}}
		\label{fig:naive_label_input}
	\end{figure}
	
	{\setlength{\parindent}{0cm}\textbf{An improved label input (ILI) mechanism:}} Empirical studies in Section \ref{sec:experiment} show that CcGAN with the naive label input mechanism already substantially outperforms cGAN. Nevertheless, it still suffers from severe label inconsistency on some datasets (e.g., Cell-200 and Steering Angle). To improve the label consistency of CcGAN, we propose an improved label input (ILI) mechanism. The ILI approach consists of a pre-trained CNN and a label embedding network. The pre-trained CNN, as shown in Fig. \ref{fig:pre_trained_CNN_for_label_embedding}, includes two subnetworks, $T_1$ and $T_2$, where $T_1$ maps an image $\bm{x}$ to a feature space and $T_2$ maps the extracted feature $\bm{h}$ to a regression label $y$. The dimension of the feature space is set to 128 in our experiments. The label embedding network $T_3$, as shown in Fig. \ref{fig:label_embedding_network}, is a multilayer perceptron (MLP) \cite{hastie2009elements} mapping a regression label $y$ back to its hidden representation $\bm{h}$ in the feature space defined by $T_1$. Assume that there are $m$ distinct regression labels in the training set, i.e., $y_1^u, y_2^u, \dots, y_m^u$, then the label embedding network $T_3$ is trained by:
	\begin{equation}
		\label{eq:label_embedding_loss}
		\min_{T_3}\frac{1}{m}\mathbb{E}_{\gamma\sim\mathcal{N}(0,\sigma_{\gamma}^2)}\left[ (T_2(T_3(y_i^u+\gamma)) - (y_i^u+\gamma))^2 \right],
	\end{equation}
	where $\sigma_{\gamma}$ is often a small value and is set at 0.2 in this paper. Then, given a regression label $y$, we can evaluate $T_3(y)$ to get a unique hidden representation of $y$ which will be incorporated into CcGAN as the condition (visualized in Fig. \ref{fig:improved_label_input}). Specifically, for $G$, we input the embedded label by using conditional batch normalization \cite{de2017modulating}. For $D$, similar to the naive approach, we input the embedded label into $D$ by label projection \cite{miyato2018cgans}. 
	
	\begin{figure}[!htbp]
		\centering
		\includegraphics[width=0.6\textwidth]{./pre-trained_CNN_for_label_embedding.png}
		\caption{The pre-trained CNN $T_1+T_2$ for label embedding. The first subnetwork $T_1$ consists of some convolutional layers (Conv.) and some linear layers. The second subnetwork $T_2$ includes one linear layer.}
		\label{fig:pre_trained_CNN_for_label_embedding}
	\end{figure}
	
	\begin{figure}[!htbp]
		\centering
		\includegraphics[width=0.4\textwidth]{./label_embedding_network.png}
		\caption{The label embedding network is a multilayer perceptron (MLP).}
		\label{fig:label_embedding_network}
	\end{figure}
	
	\begin{figure}[!htbp]
		\centering
		\includegraphics[width=0.8\textwidth]{./improved_label_input.png}
		\caption{{  The workflow of the improved label input (ILI) mechanism. ILI first uses an embedding network to convert $y$ into its high-dimensional representation $h$. Then, $h$ is input into $G$ and $D$ by conditional batch normalization \cite{de2017modulating} and label projection \cite{miyato2018cgans}, respectively.}}
		\label{fig:improved_label_input}
	\end{figure}
	


	\section{Error Bounds of $D$ Trained With HVDL and SVDL} \label{sec:convergence_rate}  %
	
	 {In this section, we derive the error bounds of a discriminator $D$ trained with $\widehat{\mathcal{L}}^{\text{HVDL}}(D)$ and $\widehat{\mathcal{L}}^{\text{SVDL}}(D)$ under the theoretical discriminator loss $\mathcal{L}(D)$. Denote by $D^*$ the optimal discriminator \cite{goodfellow2014generative} which minimizes $\mathcal{L}(D)$. Let $\widehat{D}^{\text{HVDL}}\triangleq{\arg\min}_{D\in\mathcal{D}}\widehat{\mathcal{L}}^{\text{HVDL}}(D)$; similarly, we define $\widehat{D}^{\text{SVDL}}$. We are interested in an reasonable bound (i.e, error bound) of the distance of $\widehat{D}^{\text{HVDL}}$ and $\widehat{D}^{\text{SVDL}}$ from $D^*$ under $\mathcal{L}(D)$, i.e., $\mathcal{L}(\widehat{D}^{\text{HVDL}})-\mathcal{L}(D^*)$ and $\mathcal{L}(\widehat{D}^{\text{SVDL}})-\mathcal{L}(D^*)$. These error bounds theoretically illustrate how HVDL and SVDL influence the discriminator training, which can guide our implementation of HVDL and SVDL in practice such as the selection of $\kappa$ and $\nu$. }
	
	Before we move to the derivation, without loss of generality, we first assume $y \in [0,1]$. Then, we introduce some notations. Let $\mathcal{D}$ stand for the \textit{Hypothesis Space} of $D$. Please note that $\mathcal{D}$ may not cover $D^*$. Let $\hat{p}_r^{\text{KDE}}(y)$ and $\hat{p}_g^{\text{KDE}}(y)$ stand for the KDEs of $p_r(y)$ and $p_g(y)$ respectively. Let $p^r_w(y^\prime|y)\triangleq \frac{w^r(y^\prime,y)p^r(y^\prime)}{W^r(y)}$, $p^g_w(y^\prime|y)\triangleq \frac{w^g(y^\prime,y)p^g(y^\prime)}{W^g(y)}$, $W^r(y)\triangleq\int w^r(y^\prime,y)p_r(y^\prime)dy^\prime$ and $W^g(y)\triangleq\int w^g(y^\prime,y)p_g(y^\prime)dy^\prime$.
	
	\begin{definition} 
		\label{def:holder_class}
		(H\"older Class) Define the H\"older class of functions as:
		\begin{equation}
			\Sigma(L)\triangleq\left\{p: \forall t_1, t_2\in\mathcal{Y}, \exists L >0, s.t. \frac{|p^\prime(t_1)-p^\prime(t_2)|}{|t_1-t_2|}\leq L \right\}.
		\end{equation}
	\end{definition}
	
	Please see Supp. \ref{supp:CcGAN_convergence_rate} for more details of these notations. Moreover, we will also work with the following assumptions:
	
	{\setlength{\parindent}{0cm}\textbf{(A1)}} All $D$'s in $\mathcal{D}$ are measurable and uniformly bounded. Let $U \triangleq  \max \{  \sup_{D\in \mathcal{D}} \left[-\log D\right], \allowbreak \sup_{D\in \mathcal{D}} \left[ -\log (1-D) \right] \}$ and $U < \infty$;\\ {\setlength{\parindent}{0cm}\textbf{(A2)}} For $\forall \bm{x}\in\mathcal{X}$ and $y, y^\prime\in\mathcal{Y}$, $\exists g^r(\bm{x}) >0 $ and $ M^r>0$, s.t. $|p_r(\bm{x}|y^\prime)-p_r(\bm{x}|y)|\leq g^r(\bm{x})|y^\prime-y|$ with $\int g^r(\bm{x}) d\bm{x} = M^r$; \\
	{\setlength{\parindent}{0cm}\textbf{(A3)}} For $\forall \bm{x}\in\mathcal{X}$ and $y, y^\prime\in\mathcal{Y}$, $\exists g^g(\bm{x}) >0 $ and $ M^g>0$, s.t. $|p_g(\bm{x}|y^\prime)-p_g(\bm{x}|y)|\leq g^g(\bm{x})|y^\prime-y|$ with $\int g^g(\bm{x}) d\bm{x} = M^g$; \\
	{\setlength{\parindent}{0cm}\textbf{(A4)}} $p_r(y)\in\Sigma(L^r)$ and $p_g(y)\in\Sigma(L^g)$.
	
	With these definitions and assumptions, we derive two lemmas based on which we derive the error bounds of a discriminator trained by using HVDL and SVDL in Theorems \ref{thm:convergence_rate_HVDL} and \ref{thm:convergence_rate_SVDL}.
	


	\begin{lemma}[Lemma for HVDL]
		\label{lem:conditional_mean_bound_HVDL}
		 {Suppose that (A1)-(A2) and (A4) hold, then $\forall \delta\in(0,1)$, with probability at least $1-\delta$, 
			\begin{align}
				&\sup_{D\in\mathcal{D}}\left| \frac{1}{N_{y,\kappa}}\sum_{i=1}^{N}\mathbbm{1}_{\{ |y-y_i|\leq \kappa \}} \left[ -\log D(\bm{x}_i, y) \right] - \mathbb{E}_{\bm{x}\sim p(\bm{x}|y)} \left[ -\log D(\bm{x}, y) \right] \right| \nonumber\\
				& \leq  U\sqrt{\frac{1}{2 N_{y,\kappa}}\log\left( \frac{2}{\delta} \right)} + \kappa U M,
				\label{eq:conditional_mean_bound_HVDL}
			\end{align}
			for a fixed $y$. If image-label pairs are real, then  $N=N^r$, $N_{y,\kappa}=N^r_{y,\kappa}$, $p=p_r$, and $M=M^r$. Similarly, we have $N=N^g$, $N_{y,\kappa}=N^g_{y,\kappa}$, $p=p_g$, and $M=M^g$ for fake image-label pairs.}
	\end{lemma}
	


	\begin{lemma}[Lemma for SVDL]
		\label{lem:conditional_mean_bound_SVDL}
		 {Suppose that (A1), (A2) and (A4) hold, then $\forall\delta\in (0,1)$, with probability at least $1-\delta$,
			\begin{align}
				&\sup_{D\in\mathcal{D}}\left| \frac{\frac{1}{N}\sum_{i=1}^{N}w(y_i,y)\left[ -\log D(\bm{x}_i, y) \right]}{\frac{1}{N}\sum_{i=1}^{N}w(y_i,y)} - \mathbb{E}_{\bm{x}\sim p(\bm{x}|y)} \left[ -\log D(\bm{x}, y) \right] \right| \nonumber\\
				& \leq  \frac{2U}{W(y)}\sqrt{\frac{1}{2N} \log\left( \frac{4}{\delta}\right) } + UM \mathbb{E}_{y^\prime \sim p_w(y^\prime|y)} \left[|y^\prime-y| \right] ,
				\label{eq:conditional_mean_bound_SVDL}
			\end{align}
			for a fixed $y$. If image-label pairs are real, then  $N=N^r$, $N_{y,\kappa}=N^r_{y,\kappa}$, $p=p_r$, $p_w=p_w^r$, $w=w^r$, $W=W^r$, and $M=M^r$. Similarly, we have $N=N^g$, $N_{y,\kappa}=N^g_{y,\kappa}$, $p=p_g$, $p_w=p_w^g$, $w=w^g$, $W=W^g$, and $M=M^g$ for fake image-label pairs.}
	\end{lemma}
	


	\begin{theorem}[Error Bound of $D$ trained with HVDL]
		\label{thm:convergence_rate_HVDL} 
		Assume that (A1)-(A4) hold, then $\forall \delta\in(0,1)$, with probability at least $1-\delta$,
		\begin{align}
			&\mathcal{L}(\widehat{D}^{\text{HVDL}})-\mathcal{L}(D^*) \leq   2U\left(\sqrt{\frac{C_{1,\delta}^{\text{KDE}}\log N^r}{N^r\sigma}} +  L^r\sigma^2 \right)\nonumber\\
			& + 2U\left(\sqrt{\frac{C_{2,\delta}^{\text{KDE}}\log N^g}{N^g\sigma}} + L^g\sigma^2 \right) +  2\kappa U (M^r+M^g) \nonumber\\
			& +  2U\sqrt{\frac{1}{2} \log \left( \frac{8}{\delta}\right)} \left(  \mathbb{E}_{y\sim \hat{p}_r^{\text{KDE}}(y)} \left[ \sqrt{\frac{1}{N_{y, \kappa}^r}} \, \right] \right. \nonumber\\
			&\quad\left. + \mathbb{E}_{y\sim \hat{p}_g^{\text{KDE}}(y)} \left[ \sqrt{\frac{1}{N_{y, \kappa}^g}}\, \right] \right) \nonumber\\
			&+ \mathcal{L}(\widetilde{D})-\mathcal{L}(D^*), \label{eq:convergence_rate_HVDL}
		\end{align}
		for some constants $C_{1,\delta}^{\text{KDE}}, C_{2,\delta}^{\text{KDE}}$ depending on $\delta$. 
	\end{theorem}
	


	\begin{theorem}[Error Bound of $D$ trained with SVDL]
		\label{thm:convergence_rate_SVDL}
		Assume that (A1)-(A4) hold, then $\forall \delta\in(0,1)$, with probability at least $1-\delta$,
		\begin{align}
			&\mathcal{L}(\widehat{D}^{\text{SVDL}})-\mathcal{L}(D^*) \leq 2U\left(\sqrt{\frac{C_{1,\delta}^{\text{KDE}}\log N^r}{N^r\sigma}} + L^r\sigma^2 \right) \nonumber\\
			& + 2U\left(\sqrt{\frac{C_{2,\delta}^{\text{KDE}}\log N^g}{N^g\sigma}} + L^g\sigma^2 \right) \nonumber\\
			& + 4U\sqrt{\frac{1}{2}\log \left( \frac{16}{\delta}\right)} \left(  \frac{1}{\sqrt{N^r}}\mathbb{E}_{y\sim \hat{p}_r^{\text{KDE}}(y)}\left[  \frac{1}{W^r(y)}\right] \right.\nonumber\\
			& \quad \left. + \frac{1}{\sqrt{N^g}}\mathbb{E}_{y\sim \hat{p}_g^{\text{KDE}}(y)} \left[  \frac{1}{W^g(y)}\right] \right) \nonumber\\
			& + 2U \left( M^r\mathbb{E}_{y\sim \hat{p}_r^{\text{KDE}}(y)}\left[ \mathbb{E}_{y^\prime \sim \hat{p}^r_{w}(y^\prime|y)}\left|y^\prime-y\right|  \right] \right. \nonumber\\
			& \quad \left. + M^g\mathbb{E}_{y\sim \hat{p}_g^{\text{KDE}}(y)}\left[  \mathbb{E}_{y^\prime \sim \hat{p}^g_{w}(y^\prime|y)}\left|y^\prime-y\right| \right] \right) \nonumber\\
			& + \mathcal{L}(\widetilde{D})-\mathcal{L}(D^*), 
			\label{eq:convergence_rate_SVDL}
		\end{align}
		for some constant $C_{1,\delta}^{\text{KDE}}, \; C_{2,\delta}^{\text{KDE}}$ depending on $\delta$. 
	\end{theorem}
	
	\begin{remark}
		Please see Supp.\ \ref{supp:CcGAN_convergence_rate} for the proofs to these lemmas and theorems.
	\end{remark}
		
	\begin{remark}[Illustration of Theorems \ref{thm:convergence_rate_HVDL} and \ref{thm:convergence_rate_SVDL}]
		\label{rmk:illustration_error_bounds}
		 {Both theorems imply HVDL and SVDL perform well if the output of $D$ is not too close to 0 or 1 (i.e., favor small $U$). The first two terms in both upper bounds control the quality of KDE, which implies KDE works better if we have a large $N^r$ and a large $N^g$ but a small $\sigma$. The rest terms of the two bounds are different. In the HVDL case, we favor small $\kappa$, $M^r$, and $M^g$. However, we should avoid setting $\kappa$ for a too small value because we prefer large $N^r_{y,\kappa}$ and $N^g_{y,\kappa}$. In the SVDL case, we prefer small $M^r$ and $M^g$ but large $W^r(y)$ and $W^g(y)$. Large $W^r(y)$ and $W^g(y)$ imply that the weight function decays slowly (i.e., small $\nu$). However, we should avoid setting $\nu$ too small because a small $\nu$ leads to large $\mathbb{E}_{y^\prime \sim \hat{p}^r_{w}(y^\prime|y)}\left|y^\prime-y\right|$ and $\mathbb{E}_{y^\prime \sim \hat{p}^g_{w}(y^\prime|y)}\left|y^\prime-y\right|$ (i.e., large weights for $y^\prime$'s which are far away from $y$). The rule of thumb formulae to select $\kappa$ and $\nu$ in Remark \ref{rmk:rule_of_thumb} are consistent with our analysis here. Besides the rule of thumb formulae, as a future work, a more refined hyper-parameter selection method should be proposed. }
	\end{remark}
	
	
	


	\section{Sliding Fr\'echet Inception Distance} \label{sec:sfid}
	
	A conditional GAN (no matter the type of the condition) needs to be evaluated from three perspectives \cite{devries2019evaluation}: (1) the visual quality, (2) the intra-label diversity (the diversity of fake images with the same label), and (3) the label consistency (whether assigned labels of fake images are consistent with their true labels). Measuring the performance of cGANs from these three perspectives is often conducted by using a popular overall metric, termed the Intra-FID \cite{miyato2018cgans,SAGAN-zhang19d,devries2019evaluation}. Intra-FID computes the \textit{Fr\'echet inception distance} (FID) \cite{heusel2017gans} separately at each of the distinct labels and reports the average FID score. Intra-FID is also used in our experiments on RC-49, UTKFace, and Cell-200 in Section \ref{sec:experiment}; however, Intra-FID is not reliable or is even inapplicable when we have very few (even zero) real images for some distinct regression labels, e.g., the experiment on the Steering Angle dataset in Section \ref{sec:steeringangle}. We therefore propose a novel metric, termed the \textit{Sliding Fr\'echet Inception Distance} (SFID), to replace Intra-FID in this scenario. SFID computes FID within an interval sliding on the range of the regression label $y$, and then reports the average of these FIDs. Specifically, we first prespecify a finite set of \textit{SFID centers}  $c_{\text{SFID}}$  evenly over the range of $y$ and a constant \textit{SFID radius} $r_{\text{SFID}}$. Then, based on the $c_{\text{SFID}}$ and $r_{\text{SFID}}$, we can define many joint \textit{SFID intervals} of the form  $[c_{\text{SFID}}-r_{\text{SFID}}, c_{\text{SFID}}+r_{\text{SFID}}]$. For each SFID interval, we compute FID between real and fake images with labels within this interval. Finally, SFID reports the average of these FIDs. Usually, we also report the standard deviation of these FIDs. We visualize the procedure for computing SFID in Fig. \ref{fig:SFID}. A pseudo code for computing SFID is shown in Alg. \ref{alg:compute_SFID}. Similar to Intra-FID, a small SFID is preferred.
	
	\begin{figure}[!htbp]
		\centering
		\includegraphics[width=0.6\textwidth]{./SFID_diagram.png}
		\caption{Sliding Fr\'echet Inception Distance (SFID). We preset finite centers (blue dots) on $y$'s range evenly and a radius $r_{\text{SFID}}$. Given an interval $[c_{\text{SFID}}-r_{\text{SFID}}, c_{\text{SFID}}+r_{\text{SFID}}]$, we compute FID between fake and real images with labels in this interval. SFID is the average of these FIDs.}
		\label{fig:SFID}
	\end{figure}
	
	\begin{algorithm}[!htbp]
		\footnotesize
		\SetAlgoLined
		\KwData{Real image-label pairs $D^r=\{ (\bm{x}^r_1, y_{1}^r), \cdots, (\bm{x}^r_{N^r}, y_{N^r}^r) \}$; fake image-label pairs $D^g=\{ (\bm{x}^g_1, y_{1}^g), \cdots, (\bm{x}^g_{N^g}, y_{N^g}^g)\}$; preset SFID centers $\{ y_{1}^c, \cdots, y_{N^c}^c \}$; preset window radius $r_{SFID}$.}
		\KwResult{$SFID$.} 
		Initialize real and fake image sets in $l$-th sliding-window, i.e., $D_{l}^r=\phi$ and $D_{l}^g=\phi$, $l=1,2,\cdots, N^c$\;
		Initialize a FID array $FID(l)=\infty, l=1,2,\cdots, N^c$\; 
		
		\For{$l=1$ \KwTo $N^c$}{
			Update real image set at $y_{l}^c$: $D_l^r = \bigcup_{|y_{i}^r - y_{l}^c|\leq r_{{SFID}}} \{\bm{x}_i^r\}, i=1,2,\cdots, N^r$\;
			Update fake image set at $y_{l}^c$: $D_l^g = \bigcup_{|y_{i}^g - y_{l}^c|\leq r_{{SFID}}} \{\bm{x}_i^g\}, i=1,2,\cdots, N^g$\;
			Compute current FID: $FID(l)=FID(D_l^r, D_l^g)$ \; 
		}
		Compute $SFID=\frac{1}{N^c} \sum_{l=1}^{N^c} FID(l)$.
		\caption{An algorithm to compute the Sliding Fr\'echet Inception Distance (SFID).}
		\label{alg:compute_SFID}
	\end{algorithm}
	


	\section{Low-resolution Experiments}\label{sec:experiment}
	
	In this section, we study the effectiveness of CcGAN on four image datasets with resolution $64\times 64$ where cGAN ($K$ classes) and cGAN (concat) cannot generate realistic samples. For a fair comparison, all candidate methods use the same network architecture (a customized DCGAN \cite{radford2015unsupervised} architecture for Cell-200 and the SNGAN \cite{miyato2018spectral} architecture for the remaining datasets) except for the label input modules. The four CcGAN methods (i.e., HVDL+NLI, SVDL+NLI, HVDL+ILI, and SVDL+ILI) are tested in all our experiments below. For stability, regression labels in all datasets are normalized to $[0,1]$ during training.
	
	The proposed SFID (see Section\ \ref{sec:sfid}) is only used in the experiment conducted on the Steering Angle dataset in Section\ \ref{sec:steeringangle}, where there are not enough real images to compute Intra-FID. The effectiveness of SFID is studied on the RC-49 dataset in Section\ \ref{sec:effectiveness_SFID}, where we can control the sample size of real images.
	
	
	

	\subsection{RC-49}\label{sec:rc49}
	
	Since most benchmark datasets in the GAN literature do not have continuous, scalar regression labels, we propose a new benchmark dataset---RC-49, a synthetic dataset created by rendering 49 3-D chair models at different yaw angles. Each of 49 chair models is rendered at 899 yaw angles ranging from 0.1$^{\circ}$ to 89.9$^{\circ}$ with step size 0.1$^{\circ}$. Therefore, RC-49 consists of 44,051 $64\times 64$ rendered RGB images and 899 distinct angles. Please see Supp. \ref{supp:details_of_rc49} for more details of the data generation. Example images are shown in Fig.\  \ref{fig:rc49_visual_results} in Appendix. 
	
	{\setlength{\parindent}{0cm}\textbf{Experimental setup:}} Not all images are used for the GAN training. A yaw angle is selected for training if its last digit is odd. Moreover, at each selected angle, only 25 images are randomly chosen for training. Thus, the training set includes 11250 images and 450 distinct angles. The remaining images are held out for evaluation. 
	
	When training cGAN ($K$ classes), we divide $[0.1^{\circ}, 89.9^{\circ}]$ into 150 equal intervals where each interval is treated as a class. When training CcGAN, we use the rule of thumb formulae in Remark \ref{rmk:rule_of_thumb} to select the three hyper-parameters of HVDL and SVDL, i.e., $\sigma\approx 0.047$, $\kappa\approx 0.004$ and $\nu=50625$. The two novel label input mechanisms for CcGAN (NLI and ILI) are implemented in this experiment. For ILI, we pre-train a modified ResNet-34 \cite{he2016deep} with 3 linear layers after the average pooling layer and we only keep the last linear layer for label embedding (i.e., the $T_1+T_2$ in Fig.\ \ref{fig:pre_trained_CNN_for_label_embedding}). We use a five-layer MLP with 128 nodes in each layer to convert an angle into its hidden representation (i.e., the $T_3$ in Fig.\ \ref{fig:label_embedding_network}). All candidates are trained for 30,000 iterations with batch size 256. Afterwards, we evaluate the trained GANs on all 899 angles by generating 200 fake images for each angle.  Please see Supp.\ \ref{supp:details_of_rc49} for the network architectures and more details about the training/testing setup. 
	
	{\setlength{\parindent}{0cm}\textbf{Quantitative and visual results:}} To evaluate (1) the visual quality, (2) the intra-label diversity, and (3) the label consistency of fake images, we study an overall metric and three separate metrics here. (i) \textbf{Intra-FID} \cite{miyato2018cgans} is utilized as the overall metric. It computes FID \cite{heusel2017gans} separately at each of the 899 evaluation angles and reports the average FID score along with the standard deviation of these 899 FIDs. (ii) \textbf{Naturalness Image Quality Evaluator (NIQE)} \cite{mittal2012making} measures the visual quality only.  (iii) \textbf{Diversity} is the average entropy of predicted chair types of fake images over evaluation angles.
	(iv) \textbf{Label Score} is the average absolute error between assigned angles and predicted angles. Please see Supp. \ref{supp:rc49_performance} for details of these metrics.
	
	We report in Table \ref{tab:rc49_main_results} the performances of each GAN. The example fake images in Fig.\ \ref{fig:rc49_visual_results} in Appendix and line graphs in Fig.\ \ref{fig:rc49_line_graphs} support the quantitative results. cGAN (150 classes) often generates unrealistic, identical images for a target angle (i.e., low visual quality and low intra-label diversity). ``Binning'' $[0.1^{\circ}, 89.9^{\circ}]$ into other number of classes (e.g., 90 classes and 210 classes) is also tried but does not improve cGAN's performance. cGAN (concat) has good visual quality and high intra-label diversity but terrible label consistency. In contrast, four CcGAN methods performs well in all three perspectives, i.e., good visual quality, high intra-label diversity, and high label consistency. Moreover, both ILI-based CcGANs outperform the two NLI-based CcGANs in terms of all four metrics, especially the label score. 
	
	\begin{table*}[!htbp]
		\centering
	
		\caption{Average quality of 179,800 fake RC-49 images from cGAN and CcGAN with standard deviations after the ``$\pm$" symbol. ``$\downarrow$" (``$\uparrow$") indicates lower (higher) values are preferred.}
		\begin{adjustbox}{width=0.8\textwidth}
			\begin{tabular}{ccccc}
				\toprule
				Method & Intra-FID $\downarrow$ & NIQE $\downarrow$  & Diversity $\uparrow$ & Label Score $\downarrow$ \\
				\midrule
				cGAN (150 classes)  & $1.720 \pm 0.384$ & $2.731 \pm 0.162$ & $0.779 \pm 0.199$ & $4.815 \pm 5.152$ \\
				cGAN (concat) & $1.141 \pm 0.108$ & $1.819 \pm 0.111$ & $2.459 \pm 0.049$ & $30.212 \pm 21.391$ \\
				CcGAN (HVDL+NLI) & $0.612 \pm 0.145$ & $1.869 \pm 0.181$ & $2.353 \pm 0.121$ & $5.617 \pm 4.452$ \\
				CcGAN (SVDL+NLI) & $0.515 \pm 0.181$ & $1.853 \pm 0.159$ & $2.610 \pm 0.113$ & $4.982 \pm 4.439$ \\
				CcGAN (HVDL+ILI) & $\bm{0.424 \pm 0.081}$ & $\bm{1.805 \pm 0.179}$ & $\bm{2.814 \pm 0.052}$ & $\bm{1.816 \pm 1.481}$ \\
				CcGAN (SVDL+ILI) & $\bm{0.389 \pm 0.095}$ & $\bm{1.783 \pm 0.173}$ & $\bm{2.949 \pm 0.069}$ & $\bm{1.940 \pm 1.489}$ \\
				\bottomrule
			\end{tabular}%
		\end{adjustbox}
		\label{tab:rc49_main_results}%
	\end{table*}%
	
	\begin{figure}[!htbp]
		\centering
		\includegraphics[width=0.8\textwidth]{./RC-49_line_graphs.png}
		\caption{Line graphs of FID/NIQE/Diversity/Lable Score versus yaw angle for RC-49. Figs. (a) to (c) show that four CcGAN methods consistently outperform cGAN (150 classes) across all angles. All graphs of CcGANs appear much smoother than those of cGAN (150 classes) because of HVDL and SVDL. Figs. (a) and (d) show that four CcGAN methods consistently outperform cGAN (concat) across all angles. Moreover, in most graphs, we can clearly see ILI-based CcGANs perform better than NLI-based CcGANs.}
		\label{fig:rc49_line_graphs}
	\end{figure}
	
	\begin{figure}[!htbp] 
		\centering
		\includegraphics[width=0.5\textwidth]{./Extra_Exp2_RC-49_vary_sample_size_line_graphs.png}
		\caption{ Line graphs of Intra-FID versus the sample size for each distinct training angle of RC-49. The grey vertical dashed line stands for the sample size used in the main study of the RC-49 experiment. Four CcGAN methods substantially outperform two cGANs and ILI performs better than NLI no matter what the sample size for each distinct angle in the training set. The overall trend in this figure shows that a smaller sample size deteriorates the performance of both cGAN and CcGAN.}
		\label{fig:Extra_Exp2_RC49}
	\end{figure}
	
	{\setlength{\parindent}{0cm}\textbf{Extra experimental results:}} To test cGAN and CcGAN under more challenging scenarios, we vary the sample size for each distinct angle in the training set from 45 to 5. We visualize the line graphs of Intra-FID versus the sample size for each distinct training angle in Fig.\ \ref{fig:Extra_Exp2_RC49}. From this figure, we can see the four CcGAN methods substantially outperform two cGANs and ILI performs better than NLI no matter what is the sample size for each distinct angle in the training set. The overall trend in this figure also shows that smaller sample size reduces the performance of both cGAN and CcGAN.
	
	

	\subsection{UTKFace}\label{sec:utkface}
	
	In this section, we compare CcGAN and cGAN on UTKFace \cite{utkface}, a dataset consisting of RGB images of human faces which are labeled by age.
	
	{\setlength{\parindent}{0cm}\textbf{Experimental setup:}} In this experiment, we only use images with age in $[1, 60]$. Some images with bad visual quality and watermarks are also discarded. After the preprocessing, 14,760 images are left. The number of images for each age ranges from 50 to 1051. We resize all selected images to $64\times 64$. Some example UTKFace images are shown in the first image array in Fig.\ \ref{fig:UTKFace_visual_results}. 
	
	When implementing cGAN ($K$ classes), each age is treated as a class. For CcGAN we use the rule of thumb formulae in Remark\ \ref{rmk:rule_of_thumb} to select the three hyper-parameters of HVDL and SVDL, i.e., $\sigma\approx 0.041$, $\kappa\approx 0.017$ and $\nu=3600$. Similar to the RC-49 experiment, we use NLI and ILI to incorporate ages into CcGAN. All GANs are trained for 40,000 iterations with batch size 512. In testing, we generate 1,000 fake images from each trained GAN for each age.  Please see Supp. \ref{supp:details_of_utkface} for more details of the data preprocessing, network architectures and training/testing setup.
	
	{\setlength{\parindent}{0cm}\textbf{Quantitative and visual results:}} Similar to the RC-49 experiment, we evaluate the quality of fake images by Intra-FID, NIQE, Diversity (entropy of predicted races), and Label Score. We report in Table \ref{tab:UTKFace_main_results} the average quality of 60,000 fake images. From this table, we can see the four CcGAN methods substantially outperform both cGANs and ILI performs better than NLI. Notably, although cGAN (concat) has the highest Diversity score, the huge label score reveals that cGAN (concat) cannot control the image generation with respective to ages. Thus, cGAN (concat) fails in this experiment. We also show in Fig.\ \ref{fig:UTKFace_visual_results} some example fake images from candidate models and line graphs of FID/NIQE/Diversity/Lable Score versus Age in Fig.\ \ref{fig:utkface_line_graphs}. Analogous to the quantitative comparisons, we can see that CcGAN performs much better than cGAN.
	
	\begin{table*}[!htbp]
		\centering
	
		\caption{Average quality of 60,000 fake UTKFace images from cGAN and CcGAN with standard deviations after the ``$\pm$" symbol. ``$\downarrow$" (``$\uparrow$") indicates lower (higher) values are preferred.}
		\begin{adjustbox}{width=0.8\textwidth}
			\begin{tabular}{ccccc}
				\toprule
				Method & Intra-FID $\downarrow$ & NIQE $\downarrow$  & Diversity $\uparrow$ & Label Score $\downarrow$ \\
				\midrule
				cGAN (60 classes)  & $4.516 \pm 0.965$ & $2.315 \pm 0.306$ & $0.254 \pm 0.353$ & $11.087 \pm 8.119$ \\
				cGAN (concat)  & $0.834 \pm 0.199$ & $2.051 \pm 0.227$ & $\bm{1.394 \pm 0.026}$ & $17.291 \pm 11.717$ \\
				CcGAN (HVDL+NLI) & ${0.572 \pm 0.167}$ & ${1.739 \pm 0.145}$ & $1.338 \pm 0.178$ & ${9.782 \pm 7.166}$ \\
				CcGAN (SVDL+NLI) & ${0.547 \pm 0.181}$ & ${1.753 \pm 0.196}$ & $1.326 \pm 0.198$ & ${10.739 \pm 8.340}$ \\
				CcGAN (HVDL+ILI) & $\bm{0.480 \pm 0.145}$ & $\bm{1.709 \pm 0.169}$ & ${1.280 \pm 0.203}$ & $\bm{7.505 \pm 5.857}$ \\
				CcGAN (SVDL+ILI) & $\bm{0.425 \pm 0.157}$ & $\bm{1.725 \pm 0.171}$ & ${1.298 \pm 0.176}$ & $\bm{7.452 \pm 6.022}$ \\
				\bottomrule
			\end{tabular}%
		\end{adjustbox}
		\label{tab:UTKFace_main_results}%
	\end{table*}%
	
	
	
	\begin{figure}[!htbp]
		\centering
		\includegraphics[width=0.8\textwidth]{./UTKFace_line_graphs.png}
		\caption{Line graphs of FID/NIQE/Diversity/Lable Score versus Age for UTKFace.  The four CcGAN methods significantly outperform cGAN (60 classes) in Figs. (a) to (c). All graphs of CcGANs appear much smoother than those of cGAN (60 classes) because of HVDL and SVDL. Figs. (a) and (d) show that four CcGAN methods consistently outperform cGAN (concat) across all ages. Fig. (d) also shows that the ILI-based CcGANs have higher label consistency than the NLI-based CcGANs.}
		\label{fig:utkface_line_graphs}
	\end{figure}
	
	
	\begin{figure}[!htbp]
		\centering
		\includegraphics[width=0.5\textwidth]{./Extra_Exp7_UTKFace_vary_sample_size_line_graphs.png}
		\caption{ Line graphs of Intra-FID versus the maximum sample size for each distinct age in the training set of UTKFace. Fig.\ \ref{fig:Extra_Exp7_UTKFace} shows that four CcGAN methods perform much better than cGAN and ILI is better than NLI. Moreover, a smaller sample size deteriorates the performance of both cGAN and CcGAN. }
		\label{fig:Extra_Exp7_UTKFace}
	\end{figure}
	
	
	{\setlength{\parindent}{0cm}\textbf{Extra experimental results:}} The histogram in Fig.\ \ref{fig:utkface_histogram} shows that the UTKFace dataset is highly imbalanced. To balance the training data and also test the performance of cGAN and CcGAN under smaller sample sizes, we vary the maximum sample size for each distinct age in the training from 200 to 50. Note that, we do not restrict the maximum sample size in the main study. Since we have a much smaller sample size, we reduce the number of iterations for the GAN training from 40,000 to 20,000 and slightly increase $m_{\kappa}$ in Remark\ \ref{rmk:rule_of_thumb} from 1 to 2 (we therefore use a wider hard/soft vicinity). We visualize the line graphs of Intra-FID versus the maximum sample size for each age of cGAN and CcGAN in Fig.\ \ref{fig:Extra_Exp7_UTKFace}. From the figure, we can clearly see that a smaller sample size worsens the performance of both cGAN and CcGAN. Moreover, the Intra-FID scores of two cGANs often stay at a high level and are larger than those of the four CcGAN methods. The ILI-based CcGANs are also better than the NLI-based CcGANs.
	
	

	\subsection{Cell-200}\label{sec:cell200}
	
	In addition to RC-49, we propose another benchmark dataset--Cell-200, a dataset of synthetic fluorescence microscopy images with cell populations generated by SIMCEP \cite{lehmussola2007computational}. Please see Supp. \ref{supp:cell200_data} for more details about the data generation. Some example images are shown in Fig.\  \ref{fig:cell200_visual_results}. 
	
	{\setlength{\parindent}{0cm}\textbf{Experimental setup:}} The Cell-200 dataset consists of 200,000 $64\times 64$ grayscale images. The number of cells per image ranges from 1 to 200 and there are 1,000 images for each cell count. However, only a subset of Cell-200 with only odd cell counts and 10 images per count (1,000 training images in total) is used for the GAN training.
	
	When training cGAN ($K$ classes), we divide $[1, 200]$ into 100 equal intervals where each interval is treated as a class (i.e., $K=100$). We use the rule of thumb formulae in Remark \ref{rmk:rule_of_thumb} to select the three hyperparameters of HVDL and SVDL, i.e., $\sigma\approx 0.077$, $\kappa\approx 0.020$ and $\nu=2500$.  Both cGAN and CcGAN are trained for 5,000 iterations. Afterwards, we evaluate the trained GANs on all 200 cell counts by generating 1,000 fake images for each count.  Please see Supp.\ \ref{supp:details_of_cell200} for the network architectures and more details about the training/testing setup. 
	
	{\setlength{\parindent}{0cm}\textbf{Quantitative and visual results:}} We evaluate the quality of fake images by Intra-FID, NIQE, and Label Score. Please note that the Diversity score is not available in this experiment because there is no class label in Cell-200. We report in Table \ref{tab:cell200_main_results} the average quality of 200,000 fake images from cGAN and CcGAN. We also show in Fig.\ \ref{fig:cell200_visual_results} some example fake images from cGAN and CcGAN and line graphs of FID/NIQE/Label Score versus Cell Count in Fig.\ \ref{fig:cell200_line_graphs}. Unlike the experimental results on RC-49 and UTKFace, although NLI-based CcGANs outperform cGAN (100 classes) in terms of Intra-FID and NIQE, their label scores are very high, which implies low label consistency. Fortunately, two ILI-based CcGANs still perform very well and substantially outperform two cGANs and two NLI-based CcGANs. 
	
	
	\begin{table*}[!htbp]
		\centering
	
		\caption{Average quality of 200,000 fake Cell-200 images from cGAN and CcGAN with standard deviations after the ``$\pm$" symbol. ``$\downarrow$" (``$\uparrow$") indicates lower (higher) values are preferred.}
		\begin{adjustbox}{width=0.7\textwidth}
			\begin{tabular}{cccc}
				\toprule
				Method & Intra-FID $\downarrow$ & NIQE $\downarrow$  & Label Score $\downarrow$ \\
				\midrule
				cGAN (100 classes)  & $90.255 \pm 64.595$ & $2.130 \pm 2.440$ & $66.748 \pm 51.711$ \\
				cGAN (concat)  & $41.599 \pm 21.430$ & $3.250 \pm 0.646$ & $73.187 \pm 51.133$ \\
				CcGAN (HVDL+NLI) & ${50.052 \pm 20.584}$ & ${1.488 \pm 0.153}$ & ${72.599 \pm 37.425}$ \\
				CcGAN (SVDL+NLI) & ${56.078 \pm 19.334}$ & ${1.829 \pm 0.386}$ & ${83.367 \pm 49.577}$ \\
				CcGAN (HVDL+ILI) & $\bm{8.759 \pm 6.652}$ & $\bm{1.283 \pm 0.534}$ & $\bm{5.861 \pm 4.900}$ \\
				CcGAN (SVDL+ILI) & $\bm{7.266 \pm 2.305}$ & $\bm{1.220 \pm 0.515}$ & $\bm{5.905 \pm 5.020}$ \\
				\bottomrule
			\end{tabular}%
		\end{adjustbox}
		\label{tab:cell200_main_results}%
	\end{table*}%
	
	
	\begin{figure}[!htbp]
		\centering
		\includegraphics[width=1\textwidth]{./Cell200_line_graphs.png}
		\caption{Line graphs of FID/NIQE/Lable Score versus Cell Count for Cell-200. Figs. (a) to (c) show that, although the NLI-based CcGANs do not perform well, the ILI-based CcGANs outperform both cGANs across most cell counts. All graphs of CcGANs also appear much smoother than those of cGAN (100 classes) because of HVDL and SVDL. Moreover, in all figures, we can see ILI-based CcGANs perform better than NLI-based CcGANs especially in Fig. (c).}
		\label{fig:cell200_line_graphs}
	\end{figure}
	
	


	\subsection{Steering Angle}\label{sec:steeringangle}
	
	In this section, we demonstrate the effectiveness of the proposed CcGAN on the Steering Angle dataset, a subset of an autonomous driving dataset \cite{steeringangle}. The complete autonomous driving dataset consists of 109,231 RGB images. Each image is taken by using a dash camera mounted on a car and, at the same moment, the angle of the steering wheel rotation of the same car (i.e., steering angle) is recorded by a device attached to the steering wheel. Thus, each image in this autonomous driving dataset is paired with a steering angle ranging from $-338.82^\circ$ to $501.78^\circ$. 
	
	{\setlength{\parindent}{0cm}\textbf{Experimental setup:}} To make the training and evaluation easier, we remove many images in this autonomous driving dataset where an image is removed due to at least one of the following reasons:
	\begin{itemize}
		\item The image is incorrectly labeled (e.g., some images show that the car was turning left/right but the corresponding steering angles are zero).
		\item The image has very bad visual quality due to overexposure or underexposure.
		\item There is no reference object (e.g., double yellow lines or side roads) in the image to let a human visually determine whether the car was turning left/right.
		\item The corresponding steering angle is outside $[-80^\circ, 80^\circ]$.
	\end{itemize}
	Eventually, there are 12,271 images left with 1,904 distinct steering angles in $[-80^\circ, 80^\circ]$. These images are then resized to $64\times 64$ and they form a subset of the autonomous driving dataset \cite{steeringangle}, termed \textit{Steering Angle} in this paper. Please note that the Steering Angle dataset is highly imbalanced and a histogram of steering angles is shown in Fig.\ \ref{fig:steeringangle_histogram}.
	
	When training cGAN ($K$ classes), we divide $[-80^\circ, 80^\circ]$ into 210 equal intervals where each interval is taken as a class (i.e., $K=210$). When implementing CcGAN, the three hyper-parameters of HVDL and SVDL are selected by the rule of thumb formulae in Remark \ref{rmk:rule_of_thumb}, i.e., $\sigma\approx 0.029$, $\kappa\approx 0.032$ and $\nu\approx1000.438$. All GANs are trained for 20,000 iterations. To evaluate the candidate models, we choose 2,000 evenly spaced angles in $[-80^\circ, 80^\circ]$ and generate 50 images from each candidate GAN model for each of these angles. Please see Supp. \ref{supp:details_of_steeringangle} for the network architectures and more details about the training/testing setup.
	
	
	{\setlength{\parindent}{0cm}\textbf{Quantitative and visual results:}} To evaluate the quality of fake images, we use the proposed \textit{Sliding Fr\'echet Inception Distance} (SFID) as the overall metric instead of Intra-FID, since we have very few real images for many angles (e.g., angles close to the two end points of $[-80^\circ, 80^\circ]$). We preset 1,000 SFID centers in $[-80^\circ, 80^\circ]$ and let the SFID radius be $2^\circ$. NIQE, Diversity (entropy of predicted types of scenes), and Label Score are also reported. Please see Supp. \ref{supp:steeringangle_performance} for more details of these performance measures. 
	
	We report in Table\ \ref{tab:SteeringAngle_main_results} the average quality of 100,000 fake images from each candidate method. Some example fake images are also shown in Fig.\ \ref{fig:SteeringAngle_visual_results} in Appendix. We also compute FID, NIQE, Diversity, and Label Score in each SFID interval and plot the line graphs of FID/NIQE/Diversity/Label Score versus SFID Center in Fig.\ \ref{fig:SteeringAngle_line_graphs}. Based on these quantitative and visual results, we can conclude:
	\begin{itemize}
		\item The two ILI-based CcGAN methods are better than cGAN (210 classes) in terms of all four metrics; however, the two NLI-based CcGAN methods have lower label consistency than cGAN (210 classes). Although cGAN (concat) has the highest Diversity score, four CcGAN methods outperform cGAN (concat) in terms of other three metrics. 
		\item Although the NIQE score and Label Score of cGAN (210 classes) are not grossly uncompetitive, cGAN (210 classes) has a very low Diversity score and Fig.\ \ref{fig:SteeringAngle_line_graphs}(c) shows that the Diversity scores are almost zero at some angles. Example fake images in Fig.\ \ref{fig:SteeringAngle_visual_results} also show that cGAN (210 classes) has the mode collapse problem \cite{srivastava2017veegan, pmlr-v70-arjovsky17a, chang2018escaping} (i.e., it always generates the same image for some angles).
		\item Although cGAN (concat) has the highest Diversity score, its NIQE score and Label Score are terrible, implying bad visual quality and low label consistency. Example fake images in Fig.\ \ref{fig:SteeringAngle_visual_results} support these quantitative results. 
		\item The line graphs in Fig.\ \ref{fig:SteeringAngle_line_graphs} show that the performance of cGAN (210 classes) is very unstable across all SFID intervals. Differently, cGAN (concat) has very smooth graphs but its graph for Label Score is above all other graphs.
		\item The two ILI-based CcGANs perform better than the two NLI-based CcGANs in terms of all metrics except NIQE. 
	\end{itemize}
	
	
	\begin{table*}[!htbp]
		\centering
	
		\caption{Average quality of 100,000 fake Steering Angle images from cGAN and CcGAN with standard deviations after the ``$\pm$" symbol. ``$\downarrow$" (``$\uparrow$") indicates lower (higher) values are preferred.}
		\begin{adjustbox}{width=0.8\textwidth}
			\begin{tabular}{ccccc}
				\toprule
				Method & SFID $\downarrow$ & NIQE $\downarrow$  & Diversity $\uparrow$ & Label Score $\downarrow$ \\
				\midrule
				cGAN (210 classes)  & $3.285 \pm 0.647$ & $1.296 \pm 0.095$ & $0.603 \pm 0.396$ & $14.596 \pm 15.402$ \\
				cGAN (concat)  & $2.446 \pm 1.122$ & $1.717 \pm 0.003$ & $\bm{1.255 \pm 0.015}$ & $41.686 \pm 25.864$ \\
				CcGAN (HVDL+NLI) & ${1.969 \pm 0.676}$ & $\bm{1.093 \pm 0.024}$ & ${0.991 \pm 0.361}$ & ${22.322 \pm 18.758}$ \\
				CcGAN (SVDL+NLI) & ${1.866 \pm 0.649}$ & $\bm{1.098 \pm 0.038}$ & ${1.007 \pm 0.248}$ & ${19.678 \pm 18.281}$ \\
				CcGAN (HVDL+ILI) & $\bm{1.635 \pm 0.699}$ & ${1.152 \pm 0.047}$ & ${1.153 \pm 0.153}$ & $\bm{10.868 \pm 9.644}$ \\
				CcGAN (SVDL+ILI) & $\bm{1.546 \pm 0.626}$ & ${1.130 \pm 0.078}$ & ${1.156\pm 0.189}$ & $\bm{10.933 \pm 8.978}$ \\
				\bottomrule
			\end{tabular}%
		\end{adjustbox}
		\label{tab:SteeringAngle_main_results}%
	\end{table*}%
	
	\begin{figure}[!htbp]
		\centering
		\includegraphics[width=0.8\textwidth]{./SteeringAngle_line_graphs.png}
		\caption{Line graphs of FID/NIQE/Diversity/Lable Score versus SFID Center for the Steering Angle dataset. To plot these line graphs, we compute these metrics within each SFID interval defined by the corresponding SFID center. Figs. (a) to (d) show that, although the NLI-based CcGANs do not have good label consistency, the ILI-based CcGANs substantially outperform cGAN (210 classes) in most SFID intervals. All graphs of CcGANs also appear much smoother than those of cGAN (210 classes) because of HVDL and SVDL. Figs. (b) to (c) show that athough cGAN (concat) has the highest Diversity score, it also has the worst NIQE score and Label Score.}
		\label{fig:SteeringAngle_line_graphs}
	\end{figure}
	
	

	\subsection{Effectiveness of SFID}\label{sec:effectiveness_SFID} 
	In this section, we study the effectiveness of SFID on RC-49. Since RC-49 has a large enough sample size of real images, we can get a reliable Intra-FID. At the same time, we can also deliberately reduce the sample size of real images to mimic the scenario where a reliable Intra-FID is not applicable but SFID still works well. The experiment in this section can also be conducted on Cell-200 but is omitted in this paper. We study 11 combinations of the SFID radius ($r_\text{SFID}$) and the number of SFID centers (\# $c_\text{SFID}$) in this experiment where we use SFID to evaluate cGAN and two CcGAN methods (i.e., HVDL+NLI and HVDL+ILI) pre-trained in Section \ref{sec:rc49}. In Setting 1, we let $r_\text{SFID}=0$ so SFID degenerates to Intra-FID. In the same setting, we evaluate the three GANs on all 899 distinct angles and all real images in RC-49 are used to compute Intra-FID, so Setting 1 is taken as the oracle in this experiment. In Setting 2, we also let $r_\text{SFID}=0$ so SFID degenerates to Intra-FID again. In Setting 2, however, we only evaluate GANs on the 450 distinct angles which are seen in the training set of the experiment in Section \ref{sec:rc49}. Moreover, to simulate the scenario where we have very few real images to compute Intra-FID, we deliberately reduce the number of real images at each distinct angle from 49 to 10. Therefore, in Setting 2, there are 10 real images for each angle seen in the training set and 0 real image for each angle unseen in the training set. Setting 2 is treated as the baseline in this experiment. Settings 3 to 11 are designed to show the effectiveness of SFID so we let $r_\text{SFID}>0$. Similar to Setting 2, from Settings 3 to 11, real images are available only for those 450 distinct angles seen in the training set and only 10 real images are available for each angle. We consider three values for $r_\text{SFID}$ (i.e., 0.5, 1, and 2) and three values for the number of $c_\text{SFID}$'s (i.e., 400, 600, and 800). 
	
	For all settings, we compute one FID in each SFID interval (in Settings 1 and 2, the SFID interval degenerates to an angle) and report in Table\ \ref{tab:effectiveness_of_SFID} the mean of these FIDs along with their standard deviation after the ``$\pm$" symbol. Setting 1 is the oracle setting whose evaluation results can be seen as the ground truth, and we hope the evaluation results of SFID are close to Setting 1. In Setting 2, when we have very few real images (even zero) for each angle, Intra-FID overestimates the average FID of each GAN (e.g., from 1.7201 to 1.9664 for cGAN) and underestimates the quantitative difference between cGAN and CcGANs (e.g., from $(1.7201-0.6119)/1.7201\approx 64.4\%$ to $(1.9664-1.2102)/1.9664\approx 38.5\%$ for cGAN and HVDL+NLI). However, the performance of our proposed SFID in Settings 3 to 11 is very close to the oracle setting. If we compare within Settings 3 to 11, we can see $r_\text{SFID}$ is inversely proportional to the SFID score while \# $c_{SFID}$ does not have obvious influence on SFID. From Table\ \ref{tab:effectiveness_of_SFID}, we may conclude that as long as $r_\text{SFID}$ is set at a moderate level, SFID is a valid proxy to the oracle Intra-FID when there are insufficient real images to compute Intra-FID.
	
	\begin{table*}[!htbp]
		\centering
		\caption{Evaluation results of SFID on RC-49 under different setups of $r_\text{SFID}$ and number of $c_\text{SFID}$'s. In the first two settings, SFID degenerates to Intra-FID since $r_\text{SFID}=0$. In Setting 1, we evaluate GANs on all 899 distinct angles and all real images are used to compute Intra-FID, so Setting 1 is the oracle setting. In Setting 2, we evaluate GANs on the 450 angles seen in the training set and, for each angle, 10 real images are used to compute Intra-FID, so Setting 2 is treated as the baseline. The performance of our proposed SFID (Settings 3 to 11) is close to the oracle setting while Intra-FID (Setting 2) tends to overestimate the average FID and underestimate the quantitative difference between cGAN and CcGANs.}
		\begin{adjustbox}{width=0.8\textwidth}
			\begin{tabular}{cccccc}
				\toprule
				Settings & $r_\text{SFID}$ & \# $c_\text{SFID}$ & cGAN (150 classes) & HVDL+NLI & HVDL+ILI \\
				\midrule
				1 {\tiny (Oracle)} & 0     & 899   & $1.7201 \pm 0.3843$ & $0.6119\pm 0.1452$ & $0.4241\pm 0.0811$ \\
				2 {\tiny (Baseline)} & 0     & 450   & $1.9664\pm 0.4970$ & $1.2102\pm 0.2976$ & $1.0318\pm 0.3035$ \\ \hline
				3     & 0.5   & 400   & $1.7642 \pm 0.4010$ & $0.6748 \pm 0.1445$ & $0.4257 \pm 0.0867$ \\
				4     & 1     & 400   & $1.7258\pm 0.3790$ & $0.6182\pm 0.1233$ & $0.4121\pm 0.0785$ \\
				5     & 2     & 400   & $1.6845\pm 0.3560$ & $0.5814\pm 0.1040$ & $0.4012\pm 0.0729$ \\
				6     & 0.5   & 600   & $1.7641\pm 0.3993$ & $0.6747\pm 0.1435$ & $0.4247\pm 0.0865$ \\
				7     & 1     & 600   & $1.7266\pm 0.3781$ & $0.6176\pm 0.1228$ & $0.4120\pm 0.0783$ \\
				8     & 2     & 600   & $1.6854\pm 0.3566$ & $0.5816\pm 0.1040$ & $0.4014\pm 0.0726$ \\
				9     & 0.5   & 800   & $1.7649\pm 0.3994$ & $0.6748\pm 0.1436$ & $0.4254\pm 0.0869$ \\
				10    & 1     & 800   & $1.7257\pm 0.3782$ & $0.6175\pm 0.1232$ & $0.4120\pm 0.0783$ \\
				11    & 2     & 800   & $1.6855\pm 0.3566$ & $0.5817\pm 0.1039$ & $0.4014\pm 0.0727$ \\
				\bottomrule
			\end{tabular}%
		\end{adjustbox}
		\label{tab:effectiveness_of_SFID}%
	\end{table*}%
	
	


	\section{High-resolution Experiments}\label{sec:experiment_hd}
	
	 {Besides the low-resolution experiments in Section \ref{sec:experiment}, we also compare CcGAN (SVDL+ILI) with cGAN ($K$ classes) and cGAN (concat) on high-resolution images. We consider three datasets with different resolutions, i.e., RC-49 ($128\times 128$ and $256\times 256$), UTKFace ($128\times 128$ and $192\times 192$), and Steering Angle ($128\times 128$). Since high-resolution experiments are more challenging than low-resolution ones, we make some changes to the setups in Section \ref{sec:experiment} to improve GANs' performance. First, all candidates use the more advanced SAGAN \cite{SAGAN-zhang19d} architecture. Second, cGAN ($K$ classes) and cGAN (concat) are trained with the hinge loss \cite{lim2017geometric}. CcGAN (SVDL+ILI) is trained with the reformulated hinge loss (see Eq.\ \eqref{eq:SVDL_hinge} in Supp.\ \ref{supp:reformulated_hinge_loss}). Third, DiffAugment \cite{zhao2020differentiable}, a recently proposed technique for training GANs with few samples, is also applied to improve the performance of the candidate methods.}
	
	 {Both visual and quantitative results demonstrate that the high-resolution fake images generated from CcGAN are visually realistic, diverse, and label consistent. These results also show that CcGAN is compatible with state-of-the-art GAN architectures and training techniques. Furthermore, the failure patterns of cGAN ($K$ classes) and cGAN (concat) in this experiment are consistent with those in Section \ref{sec:experiment}. cGAN ($K$ classes) tends to have high label consistency but bad visual quality and low diversity. Oppositely, cGAN (concat) often has high diversity but bad/fair visual quality and terrible label consistency. }
	
	

	\subsection{High-resolution RC-49}\label{sec:rc49_hd}
	 {In this experiment, we test three candidate methods on RC-49 with two resolutions, i.e., $128\times 128$ and $256\times 256$. Most training setups are consistent with Section \ref{sec:rc49} and please see Supp.\ \ref{supp:rc-49_hd_setups} for details. The quantitative and visual results are shown in Table~\ref{tab:rc49_hd_main_results} and Fig.~\ref{fig:rc49_hd_visual_results}. We can see CcGAN can generate high-quality images and the example fake images (Fig.~\ref{fig:rc49_hd_visual_results}) are indistinguishable from real images. However, cGAN (150 classes) and cGAN (concat) fail again. Fake images generated from cGAN (150 classes) are visually unrealistic and less diverse. Differently, cGAN (concat) can generate images with fair visual quality and high diversity, but it cannot control the image generation via conditioning angles. }
	
	
	\begin{table*}[!htbp]
		\centering
		\caption{{  Average quality of 179,800 fake high-resolution RC-49 images from cGAN (150 classes), cGAN (concat), and CcGAN with standard deviations after the ``$\pm$" symbol. ``$\downarrow$" (``$\uparrow$") indicates lower (higher) values are preferred.}}
		\begin{adjustbox}{width=0.9\textwidth}
			\begin{tabular}{cccccc}
				\toprule
				Resolution & Method & Intra-FID $\downarrow$ & NIQE $\downarrow$ & Diversity $\uparrow$ & Label Score $\downarrow$ \\
				\midrule
				\multicolumn{1}{c}{\multirow{3}[0]{*}{$128\times 128$}} & cGAN (150 classes) & $1.250 \pm 0.492$ & $2.293 \pm 0.133$ & $2.341 \pm 0.224$ & $\bm{2.032 \pm 1.653}$ \\
				\multicolumn{1}{c}{} & cGAN (concat) & $1.128 \pm 0.166$ & $2.104 \pm 0.104$ & $3.431 \pm 0.039$ & $29.414 \pm 7.052$ \\
				\multicolumn{1}{c}{} & CcGAN (SVDL+ILI) & $\bm{0.111 \pm 0.033}$ & $\bm{1.775 \pm 0.051}$ & $\bm{3.552 \pm 0.047}$ & $2.643 \pm 2.077$ \\
				\hline
				\multicolumn{1}{c}{\multirow{3}[0]{*}{$256\times 256$}} & cGAN (150 classes) & $1.224 \pm 0.336$ & $2.147 \pm 0.085$ & $2.462 \pm 0.095$ & $\bm{2.790 \pm 2.852}$ \\
				\multicolumn{1}{c}{} & cGAN (concat) & $1.325 \pm 0.227$ & $3.153 \pm 0.122$ & $\bm{3.120 \pm 0.043}$ & $28.776 \pm 20.273$ \\
				\multicolumn{1}{c}{} & CcGAN (SVDL+ILI) & $\bm{0.495 \pm 0.139}$ & $\bm{1.655 \pm 0.070}$ & $2.844 \pm 0.101$ & $3.260 \pm 2.641$ \\
				\bottomrule
			\end{tabular}%
		\end{adjustbox}
		\label{tab:rc49_hd_main_results}%
	\end{table*}%
	
	\begin{figure}[!htbp]
		\centering
		\includegraphics[width=1\textwidth]{./RC49_HD_comparison_3x6.png}
		\caption{{  Six example RC-49 images in $128\times 128$ resolution for each of 3 rotation angles: real images and example fake images from CcGAN (SVDL+ILI), cGAN (150 classes), and cGAN (concat), respectively. The fake images from CcGAN are almost indistinguishable from real images. Oppositely, cGAN (150 classes) has bad visual quality and low diversity. cGAN (concat) has fair visual quality and terrible label consistency.}}
		\label{fig:rc49_hd_visual_results}
	\end{figure}
	
	

	\subsection{High-resolution UTKFace}\label{sec:utkface_hd}
	 {In this experiment, we test three candidate methods on UTKFace with two resolutions, i.e., $128\times 128$ and $192\times 192$. Most training setups are consistent with Section \ref{sec:utkface} except that we let $\nu=900$ when implementing CcGAN (SVDL+ILI). Please see Supp.\ \ref{supp:utkface_hd_setups} for details. The quantitative and visual results are shown in Table~\ref{tab:utkface_hd_main_results} and Fig.~\ref{fig:utkface_hd_visual_results}. We can see CcGAN substantially outperforms two cGANs. Fake images generated from cGAN (150 classes) are visually unrealistic and less diverse. cGAN (concat) cannot control the image generation via conditioning ages.} 
	
	\begin{table*}[!htbp]
		\centering
		\caption{{  Average quality of 60,000 fake high-resolution UTKFace images from cGAN (60 classes), cGAN (concat), and CcGAN with standard deviations after the ``$\pm$" symbol. ``$\downarrow$" (``$\uparrow$") indicates lower (higher) values are preferred.}}
		\begin{adjustbox}{width=0.9\textwidth}
			\begin{tabular}{cccccc}
				\toprule
				Resolution & Method & Intra-FID $\downarrow$ & NIQE $\downarrow$ & Diversity $\uparrow$  & Label Score $\downarrow$ \\
				\midrule
				\multicolumn{1}{c}{\multirow{3}[0]{*}{$128\times 128$}} & cGAN (60 classes) & $1.195 \pm 0.356$ & $1.381 \pm 0.208$ & $0.788 \pm 0.425$ & $\bm{6.150 \pm 5.268}$ \\
				\multicolumn{1}{c}{} & cGAN (concat) & $0.408 \pm 0.144$ & $1.377 \pm 0.079$ & $\bm{1.332 \pm 0.026}$ & $18.064 \pm 12.550$ \\
				\multicolumn{1}{c}{} & CcGAN (SVDL+ILI) & $\bm{0.367 \pm 0.123}$ & $\bm{1.113 \pm 0.033}$ & $1.199 \pm 0.232$ & $7.747 \pm 6.580$ \\
				\hline
				\multicolumn{1}{c}{\multirow{3}[0]{*}{$192\times 192$}} & cGAN (60 classes) & $0.908 \pm 0.327$ & $1.755 \pm 0.215$ & $1.047 \pm 0.381$ & $\bm{6.639 \pm 5.686}$ \\
				\multicolumn{1}{c}{} & cGAN (concat) & $0.591 \pm 0.214$ & $2.352 \pm 0.126$ & $\bm{1.358 \pm 0.019}$ & $17.116 \pm 11.652$ \\
				\multicolumn{1}{c}{} & CcGAN (SVDL+ILI) & $\bm{0.499 \pm 0.186}$ & $\bm{1.661 \pm 0.047}$ & $1.207 \pm 0.260$ & $7.885 \pm 6.272$ \\
				\bottomrule
			\end{tabular}%
		\end{adjustbox}
		\label{tab:utkface_hd_main_results}%
	\end{table*}%
	
	\begin{figure}[!htbp]
		\centering
		\includegraphics[width=1\textwidth]{./UTKFace_HD_comparison_3x6.png}
		\caption{{  Six example UTKFace images in $192\times 192$ resolution for each of 3 ages: real images and example fake images from CcGAN (SVDL+ILI), cGAN (150 classes), and cGAN (concat), respectively. CcGAN can generate visually realistic, diverse, and label consistent images. Fake images from cGAN (150 classes) are visually bad and do not have enough diversity (e.g., last row only has while male). cGAN (concat) fails to control the image generation via conditioning ages (e.g., first row has many adults).}}
		\label{fig:utkface_hd_visual_results}
	\end{figure}
	

	\subsection{High-resolution Steering Angle}\label{sec:steering_angle_hd}
	 {Although the low-resolution Steering Angle experiment is already challenging enough due to the highly imbalanced problem, We further increase the image resolution to $128\times 128$, making the generative modeling more difficult. Most training setups are consistent with Section \ref{sec:steeringangle} and please see Supp.\ \ref{supp:steering_angle_hd_setups} for details. The quantitative and visual results are shown in Table~\ref{tab:steering_angle_hd_main_results} and Fig.~\ref{fig:steeringangle_hd_visual_results}. Both visual and quantitative results of cGAN (210 classes) imply severe mode collapse problem~\cite{srivastava2017veegan, pmlr-v70-arjovsky17a, chang2018escaping}. Similar to previous experiments, cGAN (concat) has a very high Diversity score, but its label consistency is terrible. On the contrary, the proposed CcGAN performs well in all three evaluation perspectives. }
	
	
	\begin{table*}[!htbp]
		\centering
		\caption{{  Average quality of 100,000 fake $128\times 128$ Steering Angle images from cGAN (210 classes), cGAN (concat), and CcGAN with standard deviations after the ``$\pm$" symbol. ``$\downarrow$" (``$\uparrow$") indicates lower (higher) values are preferred.}}
		\begin{adjustbox}{width=0.8\textwidth}
			\begin{tabular}{ccccc}
				\toprule
				Method & SFID $\downarrow$ & NIQE $\downarrow$  & Diversity $\uparrow$ & Label Score $\downarrow$ \\
				\midrule
				cGAN (210 classes) & $4.963 \pm 0.916$ & $2.520 \pm 0.282$ & $0.564 \pm 0.401$ & $15.903 \pm 11.522$ \\
				cGAN (concat) & $2.140 \pm 0.821$ & $2.542 \pm 0.006$ & $\bm{1.292 \pm 0.014}$ & $21.383 \pm 13.697$ \\
				CcGAN (SVDL+ILI) & $\bm{1.689 \pm 0.443}$ & $\bm{2.411 \pm 0.100}$ & $1.088 \pm 0.243$ & $\bm{9.193 \pm 8.005}$ \\
				\bottomrule
			\end{tabular}%
		\end{adjustbox}
		\label{tab:steering_angle_hd_main_results}%
	\end{table*}%
	
	\begin{figure}[!htbp]
		\centering
		\includegraphics[width=1\textwidth]{./SteeringAngle_HD_comparison_3x6.png}
		\caption{{  Six example Steering Angle images in $128\times 128$ resolution for each of 3 steering angles: real images and example fake images from CcGAN (SVDL+ILI), cGAN (150 classes), and cGAN (concat), respectively. Obviously, CcGAN outperforms both cGANs. Notably, cGAN (210 classes) has the mode collapse problem \cite{srivastava2017veegan, pmlr-v70-arjovsky17a, chang2018escaping} on this dataset.}}
		\label{fig:steeringangle_hd_visual_results}
	\end{figure}
	
	
	

	%

	\section{Conclusion}\label{sec:discussion}
	
	We propose the CcGAN in this paper for generative image modeling conditional on regression labels. In CcGAN, two novel empirical discriminator losses (HVDL and SVDL), a novel empirical generator loss and two novel label input mechanisms (NLI and ILI) are proposed to overcome the two problems of existing cGANs. The error bounds of a discriminator trained under HVDL and SVDL are studied in this work. Two new benchmark datasets, RC-49 and Cell-200, are created for the continuous scenario. A new evaluation metric, termed SFID, is also proposed to replace Intra-FID when there are insufficient real images. Finally we demonstrate the superiority of the proposed CcGAN to cGAN on RC-49, UTKFace, Cell-200, and Steering Angle datasets. 
	
	
	



	\section*{Acknowledgments}
	This work was supported by UBC ARC Sockeye, Compute Canada, and the Natural Sciences and Engineering Research Council of Canada (NSERC) under Grants CRDPJ 476594-14, RGPIN-2019-05019, and RGPAS2017-507965.
	
	
   



	\bibliographystyle{IEEEtran}
	