\section{Introduction}
\label{introduction}
\vspace{-2mm}

Knowledge Distillation (KD) is a widely-accepted approach to the problem of model compression and acceleration, which has received sustained attention from both the academic and industrial research communities \cite{gou2021knowledge,ozkara2021quped,stanton2021does,he2020group}. The goal of KD is to extract knowledge from a cumbersome model or an ensemble of models, known as the teacher, and use it as supervision to guide the training of lightweight models, known as the student. KD has been widely applied to various scenarios \cite{bergmann2020uninformed,pan2020spatio,aguilar2020knowledge}.
Most recent research regards feature maps from intermediate layers as knowledge to train the student. Although the performance of feature-based and relation-based KD is higher than the response-based methods, they require access to the source model as well as extra computation and storage.



Differently, in federated-learning where the source model is blind (\emph{source-blind KD}), the server hosts a teacher model whose internal structure and composition, connections between layers, and parameters are all invisible to edge devices. 
As shown in Fig. \ref{fig:intro}, they only host a lightweight student model and send query samples to the teacher model for heavy-duty inferences. 
Since data from edge devices do not have labeled targets, it is difficult to assign ground-truth labels. 
Response-based KD methods \cite{hinton2015distilling,zhao2022decoupled} have the natural property of hiding models. However, existing approaches are built on similar paradigms such as logits mimicking \cite{ba2014deep}, logit-distribution alignment \cite{xu2017training}, \emph{etc}. They only utilize logit-level knowledge that will reduce the robustness and generalizability \cite{xu2017training,fang2021mosaicking}.

In this paper, we propose a response-based approach to source-blind KD  by \emph{mapping emulation}. Our motivation is in accordance with the fact that a neural network can be considered a mapping function from the input distribution to the output one. Being blind to what structure and parameters are combined into the teacher network, it is still theoretically-feasible to use only the output responses as the supervision to generate the inverse function of the teacher. According to the Kolmogorov Theorem, a sufficiently-complex generator is capable of representing an arbitrary multivariate continuous function from any dimension to another one. Thus, a well-trained generative model can not only emulate the inverse mapping of the teacher but also help update the low-dimensional logits by minimizing the distance between data points in the high-dimensional space. The student learning is guided by reducing the discrepancy between generated samples from student's and teacher's outputs.

\begin{figure} 
  \centering 
  \includegraphics[scale=0.46]{./intro.pdf}
  \vspace{-5mm}
  \caption{The setting of source-blind knowledge distillation. 
  The teacher model in the cloud provides the query interface to the caller and distills a lightweight student model deployed on edge devices.
  }
  \label{fig:intro}
\end{figure}

A natural fit for the generative model is the generative adversarial network (GAN), which uses low-dimension logits as inputs to generate images and emulates the inverse mapping of the teacher. GAN is expected to fool the discriminator and blur the deviation of generative distribution from a genuine one. The generator, which is seen as an \emph{assistant teacher}, receives the knowledge transferred by the teacher and then passes it on to the student. 
The output vectors of the teacher and student are treated as a subset of noise samples, so that they have equivalent generated images after freezing the well-trained GAN.
The same image is fed into both the pre-trained teacher and the student to be trained. Two identical images are generated,  inducing the student to mimic the teacher's behavior. 

In the following, we will present related work, our proposal, and experiments. Our contribution are:
{\bf 1)} we theoretically analyze source-blind KD in a geometric view and state that it works for any GAN that satisfies the condition;
{\bf 2)} we propose a new approach which transfers knowledge via mapping emulation, outperforms the state of the art, and reaches superior performance on challenging datasets.

\section{Related Work}

\textbf{Knowledge Distillation} (KD) aims to extract knowledge from a cumbersome model or an ensemble of models and use it as supervision to help the training of lightweight models. Hinton \emph{et al.} \cite{hinton2015distilling} propose original teacher-student architecture that uses the logits of the teacher model as the knowledge. Since then, KD methods regard knowledge as final responses to input samples \cite{ba2014deep,meng2019conditional,zhao2022decoupled}, some regard knowledge as features extracted from different layers of neural networks \cite{komodakis2017paying,kim2018paraphrasing,passban2021alp}, and some regard knowledge as relations between such layers \cite{yim2017gift,passalis2020heterogeneous,chen2020learning,xiang2021coarse}. The purpose of defining different types of knowledge is to efficiently extract the underlying representation learned by the teacher model from the large-scale data. If we consider a network as a mapping function of input distribution to output, then different knowledge types just approximate such functions.
Based on the type of knowledge transferred, KD can be divided into response-based, feature-based, and relation-based \cite{gou2021knowledge}. The first two aim to derive the student to mimic the response of the output layer or the feature map of the hidden layers of the teacher, and the last approach uses the relationships between the teacher's different layers to guide the training of the student model. Feature-based and relation-based methods \cite{komodakis2017paying,yim2017gift}, depending on the model utilized, may leak the information of structure and parameters through the intermediate layers' data. For example, we can reconstruct a ResNet \cite{he2016deep} based on the feature map dimensions
of different layers, and calculate each neuron's parameter using specific images and their responses in the feature maps. 
An active line of work involves KD with adversarial training. Inspired by the generator-discriminator architecture, many methods have been proposed to enable the teacher-student model to have a better modeling of the real data manifold \cite{xu2017training,wang2018adversarial,belagiannis2018adversarial,shen2019meal}. 


\textbf{Source-blind Knowledge Distillation.} Response-based KD methods have the natural property of solving source-blind knowledge distillation tasks. Ba and Caruana propose to drive the output logits of models to be consistent to achieve model compression \cite{ba2014deep}. Hinton \emph{et al.} \cite{hinton2015distilling} improve such an approach by using the softened logits as imitation targets. Many subsequent approaches are based on logits mimicking \cite{meng2019conditional,zhao2022decoupled,zhang2018deep} aiming to reduce the discrepancy between final outputs of the teacher and student model. However, simply forcing the logits to mimic is unnecessary and difficult. It also impairs the diversified nature of the output distribution to some extent. Xu \emph{et al.} \cite{xu2017training} suggests using a discriminator to align the logits inspired by adversarial learning. Such a logits-distribution alignment method is also widely studied with various deformations \cite{fang2021mosaicking,wang2018adversarial,shen2019meal} yielding superior results.

\textbf{Generative Adversarial Network} (GAN). Compared to other generative models, GANs \cite{goodfellow2014generative} have the capacity to handle sharp estimated density functions and generate realistic-looking images efficiently. A typical GAN comprises a discriminator distinguishing between the real images and  generated images, and a generator generating images to fool the discriminator. GANs are divided into architecture-variant and loss-variant. The former focuses on network architectures \cite{radford2015unsupervised,brock2018large} or latent space \cite{mirza2014conditional,chen2016infogan}, \emph{e.g.}, some specific architectures are proposed according to specific tasks \cite{zhu2017unpaired,karras2019style}. The latter utilizes different loss types and regularization tools \cite{arjovsky2017wasserstein,gulrajani2017improved} to enable more stable learning.

\vspace{-1mm}
\section{Proposed Theory and Approach}
\vspace{-2mm}

In this section, 
we first theoretically analyze the source-blind KD in Section \ref{3.2}, then propose a new architecture in Section \ref{3.1}, and elaborate on how to train a lightweight student model in Section \ref{3.3}.




\subsection{A Geometric View of Source-blind Knowledge Distillation}
\label{3.2}
\vspace{-1mm}

The success of deep learning can be attributed to the discovery of intrinsic structures of data, which is defined as the manifold distribution hypothesis \cite{tenenbaum2000global}. The data is concentrated on a manifold $\Sigma\in \mathbb{R}^n$, which is embedded in the image space $\mathcal{X}$, and data distribution can be abstracted as a probability distribution $\mu$ over the data manifold. The encoding-map $\varphi:\Sigma\rightarrow\Omega$ maps the data manifold $\Sigma$ to the label manifold $\Omega\in \mathbb{R}^C$ in a label space $\mathcal{Y}$ which is also called latent space, while mapping the data distribution $\mu$ to label distribution $\upsilon=\varphi_\#\mu$. Each sample $x$ is mapped from the image space into the label space, and its result $\varphi(x)$ is called a latent code. The decoding-map $\varphi^{-1}$ remaps latent codes to the data manifold. Both $\varphi$ and $\varphi^{-1}$ are strongly nonlinear functions, which can be simulated with different neural networks \cite{lei2019geometric,lei2020geometric}.
Meanwhile, the well-known Kolmogorov Theorem \cite{koppen2002training,braun2009constructive} indicates that any multivariate continuous function can be represented as the sum of continuous real-valued functions with continuous one-dimensional outer and inner functions $\Phi_q$ and $\Psi_{q,p}$.

The source-blind KD task is essentially training a lightweight model from a cumbersome one that is trained on a large-scale dataset. 
The teacher function $f_T\in\varphi$ can be considered as a kind of encoding-map, and the generator function $f_G\in\varphi^{-1}$ can be considered as a kind of decoding-map.
We assume that the training and testing data for the teacher and student models are independently sampled from the identical distribution (\emph{i.i.d}). Let $\mathcal{X}\in \mathbb{R}^n$ be the image space, where data $x$ is sampled from. For a $C$-way classification task, let $\mathcal{Y}\in\mathbb{R}^C$ be the label space, where $|\mathcal{Y}|=C$. Defining the model as a complex mapping function of the image distribution to the label distribution, we can consider the teacher model as $f_T:\mathcal{X}\rightarrow\mathcal{Y}$ parameterized by $\theta_T\in\Theta_T$, whose outputs indicate the probabilities (\emph{e.g.}, Softmax of logits) of what category the samples belong to. The same for the student model $f_S:\mathcal{X}\rightarrow\mathcal{Y}$ parameterized by $\theta_S\in\Theta_S$. 
To guarantee the feasibility of MEKD in solving source-blind KD tasks, we propose three corollaries from a geometric perspective. 








\begin{lemma}
\label{lemma1}
Given the data $x$ and the network $f$ with $L$ layers. The mapping function $f_{l\rightarrow l+1}$ between any two adjacent layers is a superposition of one-dimensional functions, i.e.,

\vspace{-6mm}
\begin{equation}
    x^{l+1}=[\mathbf{x}^{l+1}_1,\ldots,\mathbf{x}^{l+1}_i,\ldots,\mathbf{x}^{l+1}_n]^T=f_{l\rightarrow l+1}(x^l),\ \text{with}\ \mathbf{x}^{l+1}_i=\sum_{q=0}^{2n}\Phi_q\left(\sum_{p=1}^n\Psi_{q,p}(\mathbf{x}^l_p)\right),
\end{equation}

\vspace{-4mm}
for $l=1,\ldots,L-1$, and $x^l$ refers to the feature map of the $l$-th layer.
\end{lemma}






Lemma \ref{lemma1} tells us that the mapping between any two adjacent layers in a network satisfies Kolmogorov Theorem. For any $i,j\in \{1,\ldots,L\}$ and $i<j$, we have $f_{i\rightarrow j}=f_{j-1\rightarrow j} \circ \cdots \circ f_{i\rightarrow i+1}$ that can also be viewed as a superposition of one-dimensional functions so that $f_{i\rightarrow j}$ represents a multivariate continuous mapping from any space $\mathbb{R}^i$ to another one $\mathbb{R}^j$. The same for the whole network $f_{1\rightarrow L}$.

\begin{corollary}
\label{corollary1}
Assuming that the structure of network $f$ is large and complex enough, its function $f:\mathcal{A}\rightarrow\mathcal{B}$ can simulate any mapping from an arbitrary dimensional space $\mathcal{A}$ to another space $\mathcal{B}$.
\end{corollary}


We further denote $z$ as a random variable (\emph{e.g.}, Gaussian) over noise space $\mathcal{Z}\in\mathbb{R}^C$, and $p$ a prior over $z$ such that $\mathbb{E}_{z\sim p}[||z||]<\infty$ (\emph{e.g.}, Gaussian, Uniform, \emph{etc.}). Arjovsky \emph{et al.} \cite{arjovsky2017wasserstein} shows that the generator function $f_G : \mathcal{Z}\rightarrow\mathcal{X}$ can approximate the mapping from noise distribution $p$ to any data distribution $\mu$ if variable $z$ is continuous and the function $f_G$ satisfies the assumption that it is continuous in $\theta_G\in\Theta_G$. Then, the following lemma holds.

\begin{lemma}
\label{lemma2}
Let $Z$ be a set of random variable $z$ sampled from noise distribution $p$, and denote the set of the teacher model's output logits as $Y$ in label distribution $\upsilon$. Then,

\begin{enumerate}
\vspace{-2mm}
    \item The generator $f_G$ cannot (but can) be well-trained only using $Y$ (using $Z$) as its input set.
\vspace{-2mm}
    \item A well-trained $f_G$ can map $Y$ to a subset of images sampled from any data distribution $\mu$.
\end{enumerate}

\end{lemma}










Lemma \ref{lemma2} guarantees that the well-trained generator $f_G$ maps the label distribution $\upsilon$ output by the teacher model to the real image distribution $\mu$, as shown in Fig. \ref{fig:mapping_function}. In another sense, it has the ability to emulate the inverse mapping of teacher function $f_T^{-1}$. See Supplementary Material for all proofs.

\begin{figure} 
\vspace{-10mm}
  \centering 
  \includegraphics[scale=0.6]{./mapping_function.pdf}
  \vspace{-8mm}
  \caption{Mapping process illustrating Lemma 2. 
  The $f_G$ maps the noise distribution $p$ to the data manifold $\Sigma$ and the label distribution $\upsilon$ to the sub-distribution $\mu$. Meanwhile, $f_T$ maps $\mu$ to $\upsilon$.
  }
  \label{fig:mapping_function}
\end{figure}

\begin{figure}
\vspace{-7mm}
  \centering 
  \includegraphics[scale=0.45]{./cell_decomposition.pdf}
  \vspace{-6mm}
  \caption{Cell decomposition illustrating Lemma 3. 
  The cell $U_{\alpha}$ in the noise space is mapped to exact image $x^{(i)}$ with the same color by $f_G$. The movement of point $x_S'$ leads to the alignment of logits $y_S$.
  }
  \label{fig:cell_decomposition}
\end{figure}


\begin{corollary}
\label{corollary2}
Whatever structure the generator $f_G$ is, as long as it is well-trained with the utilization of noise $Z$ and real data $X$, then it can emulate the inverse mapping of teacher function, i.e. $f_G=f_T^{-1}$.
\end{corollary}

Fixing a decoding map $f_G\in\varphi^{-1}$ for a well-trained generator, the noise space $\mathcal{Z}$ is partitioned as

\vspace{-3mm}
\begin{equation}
    \mathcal{D}(f_G):\mathcal{Z}=\bigcup_{\alpha}U_{\alpha},
\end{equation}

where $\mathcal{D}(f_G)$ is called the decomposition induced by the decoding map $f_G$ \cite{lei2020geometric}. As shown in Fig. \ref{fig:cell_decomposition}, $f_G$ maps cell decomposition in the noise space $\mathcal{D}(f_G)$ to a cell decomposition in the image space, and each cell $U_{\alpha}$ is mapped to a sample $x_i$ by the decoding map $f_G$ \cite{lei2019geometric}. In another word, $f_G$ pushes the noise distribution $p$ to the exact empirical distribution:

\vspace{-3mm}
\begin{equation}
    ({f_G})_{\#}p=\frac{1}{n}\sum_{i}x^{(i)}.
\end{equation}

\begin{lemma}
\label{lemma3}
Let $\mu$ be any data distribution, $f_T, f_S, f_G$ be the mapping function of the teacher, student, and generator, respectively. Then, it holds that $f_S=f_T$ when achieving the optimal solution of

\vspace{-3mm}
\begin{equation}
    \min \mathbb{E}_{x\sim\mu}\left[\ \mathbbm{d}\left(f_G\circ f_S(x),\ f_G\circ f_T(x)\right)\ \right],
\end{equation}

where $\mathbbm{d}(\cdot)$ is a distance metric formula.
\end{lemma}







\begin{corollary}
\label{corollary3}
By reducing the empirical distance $\sum_{i}\mathbbm{d}(\cdot)$ between generated images $f_G\circ f_S(x)$ and $f_G\circ f_T(x)$ of the student and teacher via a well-trained generator, $f_S=f_T$ can be achieved for KD.
\end{corollary}




 



\subsection{MEKD: Mapping-Emulation Knowledge Distillation}
\label{3.1}
\vspace{-1mm}



Hinton \emph{et al.} \cite{hinton2015distilling} proposes a simple but effective KD method that uses the soften logits of the teacher model as supervision to guide the training of the student. They use the Kullback-Leibler Divergence (KLD) to measure the discrepancy between the logits of the two models and the student model is trained to minimize that in hope of achieving the same output. Specifically, the loss is defined as

\vspace{-3mm}
\begin{equation}
\label{equation:KLD}
    \mathcal{L}_{KD}=\mathcal{KL}[p(c|\mathbf{x}_i;\theta_T)||p(c|\mathbf{x}_i;\theta_S)]=-\frac{1}{N}\sum_i^N\sum_c^C p(c|\mathbf{x}_i;\theta_T)\log\left[\frac{p(c|\mathbf{x}_i;\theta_T)}{p(c|\mathbf{x}_i;\theta_S)}\right],
\end{equation}
\vspace{-2mm}

where $i$ is the sample index and $N$ is the number of samples.
Regardless of the approaches, the essence of KD is learning the mapping function of the teacher model from input to output, \emph{i.e.}, $f_T$. However, we are blind to the mapping function from the existing teacher-model parameters. One can only guess the mapping process by using the responses of different network layers to the input samples or the relations between layers and treat them as \emph{knowledge} to guide the training of the student model \cite{yim2017gift}. Meanwhile, in some cases (\emph{e.g.}, see Sec. \ref{introduction}), the internal structure and parameters of the teacher network are not available, which makes effective distillation more challenging. 


We first train a generative adversarial network using random noise variable $z$ as input. Note that the dimensionality of $z$ is the same with output logits of the teacher model, \emph{i.e.} $|z|=C$. The generator $G$ generates fake images using noise $z$, and the discriminator $D$ minimizes the Wasserstein distance between generated distribution and the genuine one. If the discriminator is capable of completely blurring the discrepancy of fake and real images, then the resulting generator, represents a function from label space to image space, defined as $f_G:\mathcal{Y}\rightarrow\mathcal{X}$, has the inverse mapping of the teacher. 

\begin{figure}
\vspace{-7mm}
  \centering 
  \includegraphics[scale=0.56]{./pipeline.pdf}
  \vspace{-5mm}
  \caption{The proposed mapping-emulation knowledge distillation (MEKD). 
  The overall process is divided into two stages, first training the GAN to emulate the inverse mapping of the teacher model, and then guiding the training of the student model utilizing the generator for knowledge distillation.
  }
  \vspace{-3mm}
  \label{fig:pipeline}
\end{figure}


The well-trained generator contains all the knowledge that the teacher model used to make inferences. It is equivalent to an assistant teacher transferring the teacher's knowledge to the student. Fig.~\ref{fig:pipeline} illustrates the holistic architecture and training sequence of MEKD. We freeze the generator and graft it on the teacher and student model in the same way, that is using the output logits of two models as its input to generate images of the same size as each other. A batch of images $X$ from the dataset is fed into the embedded network to obtain the generated images at a time. The generated samples from teacher's outputs $X_T'=f_G\circ f_T(X)$ and another generated ones from student's outputs $X_S'=f_G\circ f_S(X)$ are measured by the distance metric formula $\ell=\mathbbm{d}(X_S',X_T')$. Minimizing the distance $\ell$ to drive the student mimicking the behavior of the student. 


In practice, we place the first stage of mapping emulation (\emph{i.e.} the training of GAN) in the edge devices using their stored data. They send query samples to the teacher model in the server, then use the corresponding inference results to conduct the second stage of KD and model deployment.







\subsection{Learning and Optimization}
\label{3.3}
\vspace{-1mm}

Having the architecture, we propose an algorithm, summarized in Alg. \ref{alg:mekd}, to train the student model.
 
\begin{algorithm}
\caption{MEKD optimization algorithm.}
\label{alg:A}  
\hspace*{0.02in}{\bf Input:}
Pre-trained teacher $T(x;\theta_T)$, random initialized student $S(x;\theta_S)$ and dataset $X$.\\
\hspace*{0.02in}\vspace*{0.03in}{\bf Output:}
An optimized student $S(x;\theta_S)$.
\begin{algorithmic}[1]
\label{alg:mekd}
\STATE {Initialize a generator $G(z;\theta_G)$ and a discriminator $D(x;\theta_D)$}
\STATE {\textbf{repeat}}
\STATE \hspace*{0.2in}{\textcolor{GRAY}{$\triangleright$ Mapping Emulation}}
\STATE \hspace*{0.2in}{\textbf{for} k steps \textbf{do}:}
\STATE \hspace*{0.4in}{Sample a mini-batch of $m$ noise samples $\{z^{(1)},\ldots,z^{(m)}\}$ from noise prior $p$}
\STATE \hspace*{0.4in}{Sample a mini-batch of $m$ real examples $\{x^{(1)},\ldots,x^{(m)}\}$ from dataset $X$}
\STATE \hspace*{0.4in}{Update discriminator $D$ to distinguish fake data from real ones using $\mathcal{L}_D$ from Eqn. \ref{equation:loss_of_D}}
\STATE \hspace*{0.2in}{\textbf{end for}}
\STATE \hspace*{0.2in}{Sample a mini-batch of $m$ noise samples $\{z^{(1)},\ldots,z^{(m)}\}$ from noise prior $p$}
\STATE \hspace*{0.2in}{Update generator $G$ to fool the discriminator $D$ using $\mathcal{L}_G$ from Eqn.
\ref{equation:loss_of_G}}
\STATE {\textbf{until} converge}
\STATE {GAN can be in any structure, only to ensure the dimensionality of $z$ equals to category count $C$.}
\STATE {Initialize the student $S(x;\theta_S)$}
\STATE {\textbf{repeat}}
\STATE \hspace*{0.2in}{\textcolor{GRAY}{$\triangleright$ Knowledge Distillation}}
\STATE \hspace*{0.2in}{Sample a mini-batch of $m$ examples $\{x^{(1)},\ldots,x^{(m)}\}$ from dataset $X$}
\STATE \hspace*{0.2in}{Update student $S$ through knowledge distillation using $\mathcal{L}_S$ from Eqn. \ref{equation:loss_of_S}}
\STATE {\textbf{until} converge}
\end{algorithmic}

\end{algorithm}



\textbf{Mapping Emulation}. For emulating the inverse mapping of teacher function $f_G=f_T^{-1}$, the input size of the generator is designed to have the same as the category count $C$. This also means the dimensionality of the random variable $z$ is equal to the category count, \emph{i.e.} $|z|=C$. The theoretical analysis shows that we can use different structures of GAN as emulators. The only thing needs to be ensured is that the generator should be trained to generate samples as close as possible to the real data distribution. We choose some GANs as our emulators, such as DCGAN \cite{radford2015unsupervised}, WGAN-gp \cite{gulrajani2017improved}, InfoGAN \cite{chen2016infogan}, \emph{etc}. 
Note that the generator and discriminator are trained simultaneously: we adjust parameters for the generator to minimize $\log (1-D(G(z)))$ and adjust parameters for the discriminator to minimize $\log D(x)$. And their loss functions are defined as follows:

\vspace{-3mm}
\begin{equation}
\label{equation:loss_of_D}
    \mathcal{L}_D=-\frac{1}{m}\sum_{i=1}^m\left[ \log D\left(x^{(i)}\right)+\log \left(1-D\left(G\left(z^{(i)}\right)\right)\right) \right],
\end{equation}

\vspace{-5mm}
\begin{equation}
\label{equation:loss_of_G}
    \mathcal{L}_G=-\frac{1}{m}\sum_{i=1}^m\log \left(1-D\left(G\left(z^{(i)}\right)\right)\right).
\end{equation}

\textbf{Knowledge Distillation}. After grafting the well-trained generator from the precious step on the teacher and student model, the same image examples $X$ are input to the embedded model and get the generated images $X_S'=G(S(X))$ from the student and $X_T'=G(T(X))$ from the teacher. We use the distance metric formula $L_1$ norm ($p=1$) of $X_S'$ and $X_T'$ as the loss function of student model:

\vspace{-3mm}
\begin{equation}
\label{equation:loss_of_S}
    \mathcal{L}_S=-\frac{1}{m}\sum_{i=1}^m \left[ \left\Vert G\left(S\left(x^{(i)}\right)\right) - G\left(T\left(x^{(i)}\right)\right) \right\Vert_{p} + T\left(x^{(i)}\right) \log \left( \frac{T\left(x^{(i)}\right)}{S\left(x^{(i)}\right)}\right) \right].
\end{equation}
\vspace{-2mm}

$\mathcal{L}_2$-norm has similar effect with $\mathcal{L}_1$. We also add logit-level knowledge (Eqn. \ref{equation:KLD}) to induce distillation.






\vspace{-1mm}
\section{Experiments}
\label{experiments}
\vspace{-2mm}



In this section, we compare our methods with response-based KD methods including KD \cite{hinton2015distilling}, CDL \cite{kim2017transferring}, CAN \cite{xu2017training} and DKD \cite{zhao2022decoupled} in the \emph{unsupervised} setting that the student is blind to instance-level knowledge of labels, due to no access of labels of data stored in edge devices in federated learning.

\textbf{Implementation}. We use ResNet \cite{he2016deep} as backbone, and use standard data augmentation techniques (random crop and horizontal flip) and SGD optimizer for all experiments. We train the teacher model and student model for 240 epochs, except 24 epochs for MNIST  \cite{lecun1998gradient}, with SGD optimizer and multi-step LR scheduler following \cite{kim2018paraphrasing}. After the training of teacher model, we use Gaussian noise with the same dimension as the category count of homologous dataset as the input to train GAN. 

\vspace{-1mm}
\setlength{\tabcolsep}{4pt}
\renewcommand\arraystretch{1.2}
\begin{table}[h!]
\begin{center}
\caption{Top-1 student classification accuracy on CIFAR-10 and MNIST.}
\label{table:CIFAR10}
\setlength{\tabcolsep}{4mm}{
\vspace{-1mm}
\begin{tabular}{ccc|cc}
    \toprule
    Dataset & \multicolumn{2}{c|}{CIFAR-10} & \multicolumn{2}{c}{MNIST}\\
    \hline
    \multirow{2}{*}{Teacher} & ResNet56 & ResNet110 & ResNet56 & ResNet32\\
    &93.86 & 93.98  & 99.45 & 99.43 \\
    Student & ResNet20 & ResNet32 & ResNet8 & ResNet8\\ 
   
	\hline
	 KD \cite{hinton2015distilling} & 93.06 & 93.40 & 99.48 & 99.41\\
	 CDL \cite{kim2017transferring} & 93.12 & 93.51 & 99.43 & 99.45\\
	 CAN \cite{xu2017training} & 92.71 & 93.27 & 99.06 & 99.24\\
     DKD \cite{zhao2022decoupled}& 92.78 & 93.37 &  99.52 & 99.49\\
	 MEKD & \textbf{93.39} & \textbf{93.67} & \textbf{99.56} & \textbf{99.53} \\
    \bottomrule
\end{tabular}}
\end{center}
\end{table}
\setlength{\tabcolsep}{1.4pt}

\vspace{-4mm}

\setlength{\tabcolsep}{4pt}
\renewcommand\arraystretch{1.2}
\begin{table}[h!]
\begin{center}
\caption{Top-1 student classification accuracy on CIFAR-100 and Tiny ImageNet.}
\vspace{-1mm}
\label{table:CIFAR100}
\setlength{\tabcolsep}{4mm}{
\begin{tabular}{ccc|cc}
    \toprule
    Dataset & \multicolumn{2}{c|}{CIFAR-100} & \multicolumn{2}{c}{Tiny ImageNet}\\
    \hline
    \multirow{2}{*}{Teacher} & ResNet56 & ResNet110 & ResNet56 & ResNet110\\
    &73.48 & 73.84  & 57.70 & 60.47\\
    Student & ResNet20 & ResNet32 & ResNet20 & ResNet32\\ 
   
	\hline
	 KD \cite{hinton2015distilling} & 70.29 & 72.71 & 50.85 & 56.04\\
	 CDL \cite{kim2017transferring} & 69.82 & 72.63 & 50.73 & 55.57\\
	 CAN \cite{xu2017training} & 68.81 & 72.03 & 48.55 & 53.47\\
 	 DKD \cite{zhao2022decoupled} & 69.99 & 72.38 &  \textbf{51.72} & 56.07 \\
	 MEKD & \textbf{70.91} & \textbf{73.14}& 51.56 & \textbf{56.55}\\
    \bottomrule
\end{tabular}}
\end{center}
\end{table}
\setlength{\tabcolsep}{1.4pt}
\vspace{-2mm}

\textbf{Performance on MNIST} \textbf{and CIFAR-10} \cite{krizhevsky2009learning}.  Table~\ref{table:CIFAR10} presents the results on these two small datasets. For MNIST, we use ResNet32/ResNet56 as teacher model and ResNet8 as student model, and use WGAN-gp as the generative model due to the low dimensional inputs (for other datasets, we use DCGAN). All baseline methods and MEKD are trained for 24 epochs for MNIST and 240 epochs for CIFAR-10 with the identical setting. On both datasets, MEKD achieves SOTA performance compared with all baseline methods. For example, MEKD is higher than the SOTA method DKD, by 0.04$\sim$0.61 on both datasets. For CIFAR-10, we use two teacher-student pairs, MEKD achieves consistent improvements on both pairs by 0.33 and 0.27 compared with the baseline methods. 




\textbf{Performance on CIFAR-100} \cite{krizhevsky2009learning} \textbf{and Tiny ImageNet} \cite{deng2009imagenet}.  Table~\ref{table:CIFAR100} presents results on larger datasets. We test MEKD on two teacher-student pairs including ResNet56-ResNet20 and ResNet110-ResNet32. Compared with baseline, MEKD outperforms by 0.41 $\sim$ 0.62. On Tiny ImageNet, the performance of the SOTA methods, DKD, is slightly better than our MEKD with ResNet56-ResNet20 pair. This may be due to the difference between KLD loss and decoupled loss in DKD \cite{zhao2022decoupled}. The results verify the effectiveness of MEKD on challenging datasets with the \emph{unsupervised} setting. 












\textbf{Ablation study of loss function for generated images.} \ As explained, we use an additional loss to enhance the distillation. Table~\ref{MNIST-loss-ablation},\ref{cifar-loss-ablation} display the performance of different distance metric computed on generated images of the teacher and student model. Compared to $\mathcal{L}_1$ loss, the $\mathcal{L}_2$ loss achieves comparable performance on the MNIST and CIFAR-100 dataset, respectively. 

\vspace{-2mm}
\begin{minipage}{\textwidth}
\begin{minipage}[t]{0.48\textwidth}
\centering
\makeatletter\def\@captype{table}
\caption{The loss computed on generated images of the teacher and student model. Shown for CIFAR-100. ACC: Top-1 accuracy of the student model. R-110 / R-32: ResNet110 / ResNet32.}
\vspace{2mm}
\begin{tabular}{cc|cc}
    \toprule
   
   
   
    \makebox[0.1\textwidth][l]{\quad MNIST} & \makebox[0.1 \textwidth][c]{} & \makebox[0.2 \textwidth][c]{Methods} & \makebox[0.25\textwidth][c]{ACC}\\
    \midrule
    Teacher & Student & KD \cite{hinton2015distilling}  & 99.41\\
    R-32 & R-8 & MEKD ($\mathcal{L}_1$ loss)  & 99.53    \\
    99.43 & 98.91 & MEKD ($\mathcal{L}_2$ loss) & 99.48   \\
    \bottomrule
\end{tabular}
\label{MNIST-loss-ablation}
\end{minipage}
\hspace{4mm}
\begin{minipage}[t]{0.48\textwidth}
\centering
\makeatletter\def\@captype{table}
\caption{The loss computed on generated images of the teacher and student model. Shown for MNIST. ACC: Top-1 accuracy of the student model. R-32 / R-8: ResNet32 / ResNet8.}
\vspace{2mm}
\begin{tabular}{cc|cc}
    \toprule
   
   
   
    \makebox[0.1\textwidth][l]{CIFAR-100} & \makebox[0.1 \textwidth][c]{} & \makebox[0.2 \textwidth][c]{Methods} & \makebox[0.25\textwidth][c]{ACC}\\
    \midrule
    Teacher & Student & KD \cite{hinton2015distilling}  & 72.71\\
    R-110 & R-32 & MEKD ($\mathcal{L}_1$ loss)  & 73.14     \\
    73.84 & 71.05 & MEKD ($\mathcal{L}_2$ loss) & 73.22    \\
    \bottomrule
\end{tabular}
\label{cifar-loss-ablation}
\end{minipage}
\end{minipage}

\vspace{3mm}


\textbf{Ablation study of the generative model's architecture.} \ We verify if MEKD is a generic method for different structures of GANs. On MNIST dataset, we use three kinds of GAN models including WGAN-gp, InfoGAN and DCGAN in MEKD to test the performance of distillation. As illustrated in Table~\ref{GAN ablation}, it demonstrates that WGAN-gp could facilitate the training of student model best and the FID score of WGAN-gp is the lowest. The result also supports the conclusion that the better FID score of the generative model the better student model. All variants of MEKD with different generative model outperforms baseline KD by 0.05 $\sim$ 0.12 on MNIST, which demonstrate MEKD's advantages.

\textbf{Ablation study of the quality of generated images.} \ As the mapping of the generator and teacher model are inverse functions of each other, GAN needs to be trained to capture the underlying mapping function of teacher model. We study how the performance of GAN affects the distillation performance. FID score \cite{lucic2018gans} is a metric to measure the similarity of generated image distribution and genuine one. We use the same GAN architecture with different FID score to test our model. As shown in Table~\ref{FID ablation}, we observe that the lower FID score, the higher accuracy of student model. Note that all MEKD methods outperform baseline KD by 0.1$\sim$0.62 which shows the effectiveness of MEKD. The results also show that MEKD with a better GAN model can achieve higher distillation performance.



\vspace{-2mm}
\begin{minipage}{\textwidth}
\begin{minipage}[t]{0.55\textwidth}
\centering
   
    \makeatletter\def\@captype{table}
    \caption{MEKD with different GAN models. ACC: Top-1 accuracy of the student model. R is short for ResNet.}
    \vspace{2mm}
	\begin{tabular}{cc|cccc}
	\toprule
    \makebox[0.04\textwidth][l]{\quad MNIST} & \makebox[0.04\textwidth][c]{} & \makebox[0.09 \textwidth][c]{Method} & \makebox[0.06\textwidth][c]{GANs} &\makebox[0.08 \textwidth][c]{FID$\downarrow$}& \makebox[0.08\textwidth][c]{ACC$\uparrow$}\\
    \midrule
    Teacher & Student & KD \cite{hinton2015distilling}  & N/A & N/A & 99.41\\
    \multirow{2}{*}{R-32} &\multirow{2}{*}{R-8} & MEKD & WGAN-gp & 22.64 & 99.53 \\
    && MEKD  & InfoGAN & 34.39 & 99.49 \\
    73.48 & 69.54 & MEKD & DCGAN & 62.95 & 99.46\\
    \bottomrule
	\end{tabular}
	\label{GAN ablation}
\end{minipage}
\hspace{1mm}
\begin{minipage}[t]{0.43\textwidth}
\centering
	\makeatletter\def\@captype{table}
    \caption{The relationship between distillation quality and FID score of the generator. }
    \vspace{2mm}
    \begin{tabular}{cc|ccc}
        \toprule
        \makebox[0.05\textwidth][l]{CIFAR-100} & \makebox[0.05 \textwidth][c]{} & \makebox[0.08 \textwidth][c]{Method} & \makebox[0.1 \textwidth][c]{FID$\downarrow$} & \makebox[0.1\textwidth][c]{ACC$\uparrow$}\\
        \midrule
        Teacher & Student & KD \cite{hinton2015distilling}  & N/A & 70.29\\
        \multirow{2}{*}{R-56} &\multirow{2}{*}{R-20} & MEKD  & 135.06 & 70.91 \\
        && MEKD  & 239.83 & 70.49 \\
        73.48 & 69.54 & MEKD & 332.65 & 70.39\\
        \bottomrule
    \end{tabular}
   
    \label{FID ablation}
\end{minipage}
\end{minipage}

\vspace{3mm}
\textbf{Ablation study of correlations between KLD loss and $\mathcal{L}_1$/$\mathcal{L}_2$ loss.} \ Compared to the baseline KD, we add a generative model to emulate the inverse mapping of the teacher model. The $\mathcal{L}_1$/$\mathcal{L}_2$ loss is used to measure the similarity between the generated images of the teacher model and the student model. 
We assume that the $\mathcal{L}_1$/$\mathcal{L}_2$ loss positively correlates with the KLD loss, which helps to justify why MEKD outperforms the baseline. Fig.~\ref{fig:gradient} on the next page visualizes the gradients of student model's final last classification layer output during the half of total training process. The x-axis is the category and the gradient belonging to the label is put in the first place. The image is the current sample in the training of student model. We compare gradients in Standard Supervised Learning, baseline KD \cite{hinton2015distilling} and $\mathcal{L}_1$/$\mathcal{L}_2$ Loss in MEKD using the same sample.
The student model learns with MEKD using $\mathcal{L}_1$/$\mathcal{L}_2$ loss gives higher positive updates for the target class and higher negative updates for the non-target class, which implies that MEKD can facilitate the training of student model.




\vspace{-1mm}
\section{Conclusion}
\label{conclusion}
\vspace{-1mm}

\begin{figure}
  \centering 
  \includegraphics[scale=0.34]{./gradient_88.pdf}
  \vspace{-8mm}
  \caption{Visualizing logit-gradients of student models trained with different loss on various datasets.
 
  }
  \label{fig:gradient}
\end{figure}

In this paper, we propose Mapping-Emulation Knowledge Distillation (MEKD) for source-blind KD, which is essential for federated learning. 
A new geometric perspective is proposed to view such a deep-learning problem using the function approximation theory and optimal transport theory. Recent network-interpretability research has seen some theoretical analyses of GANs from a geometric view \cite{lei2019geometric,lei2020geometric,arjovsky2017towards}, while none of them analyze GANs particularly for KD.
Unlike most approaches using intermediate-level supervision, learning inverse functions from classifier-logits is model-blind, but has a geometric intuition and theoretical guarantee. We validate MEKD on various image-classification benchmarks, showing that it effectively pass knowledge from the teacher to students, outperforms the state of the art, and still achieves superior performance on challenging datasets.

{\bf Limitation}. MEKD has limited generalizability for out-of-distribution (OOD) data, because response-based method cannot capture uniform features of OOD data, which will be examined in future work.  










\bibliographystyle{IEEEtran}
