

\section{Introduction}\label{introduction}
The advent of automatic feature engineering fuels deep neural networks to achieve remarkable success in a plethora of computer vision tasks, such as image classification \cite{simonyan2014very, howard2017mobilenets, zhang2018shufflenet, you2020greedynas, huang2022dyrep}, object detection \cite{lin2017feature, Cai_2019}, and semantic segmentation \cite{zhao2017pyramid, chen2017rethinking}. In the path of pursuing better performance, current deep learning models generally grow deeper and wider \cite{he2016deep, xie2017aggregated}. However, such heavy models are clumsy to deploy in practice due to the limitations of computational and memory resources. For an efficient model with competitive performance to those larger models, knowledge distillation (KD)~\cite{hinton2015distilling} has been proposed to boost the performance of the efficient model (student) by distilling the knowledge of a larger model (teacher) during training.

The essence of knowledge distillation relies on how to formulate and transfer the knowledge from teacher to student. The most intuitive yet effective approach is to match the probabilistic prediction (response) scores between the teacher and student via Kullback–Leibler (KL) divergence \cite{hinton2015distilling}. In this way, the student can be guided with more informative signals during training, and is thus expected to have more promising performance than that being trained stand-alone. 
Besides this vanilla prediction match, other works \cite{heo2019comprehensive, peng2019correlation, tian2019contrastive, du2020agree} also investigate the knowledge within intermediate representations to further boost the distillation performance, but this usually induces additional training cost as a consequence. For example, OFD~\cite{heo2019comprehensive} proposes to distill the information via multiple intermediate layers, but requires additional convolutions for feature alignments; CRD~\cite{tian2019contrastive} introduces a contrastive loss to transfer pair-wise relationships, but it needs to hold a memory bank for all 128-d features of ImageNet images, and produces additional 260M FLOPs of computation cost.

\begin{figure}
    \centering
    \begin{minipage}[t]{0.49\textwidth}
        \centering
        \subfigure[Stronger model sizes]
        {\includegraphics[width=0.50\linewidth]{figures/f1_stronger_teachers.pdf}}
        \hspace{-2mm}
        \subfigure[Stronger strategies]
        {\includegraphics[width=0.50\linewidth]{figures/f1_stronger_strategies.pdf}}
        \vspace{-2mm}
        \caption{Comparisons of KD and our proposed DIST on ImageNet with different teachers. (a) The ResNet-18 students are trained using baseline strategy with different model sizes of the teacher. (b) The ResNet-18 students are trained using different strategies with ResNet-50 teachers.
        }
        \label{fig:f1}
    \end{minipage}\hfill
    \begin{minipage}[t]{0.49\textwidth}
        \centering
        \subfigure[KL div. ($\tau=1$)]
        {\includegraphics[width=0.50\linewidth]{figures/error_kldiv_t1.pdf}}
        \hspace{-2mm}
        \subfigure[KL div. ($\tau=4$)]
        {\includegraphics[width=0.50\linewidth]{figures/error_kldiv_t4.pdf}}
        \vspace{-2mm}
        \caption{Discrepancy between the predictions of models trained standalone with different strategies on ImageNet validation set. \textit{R18B1} represents ResNet-18 trained with strategy B1 for instance. Details of training strategies B1 and B2 refer to Table~\ref{tab:strategies}.
       
        }
        \label{fig:error}
    \end{minipage}
\end{figure}

Recently, a few studies have been performed to address the poor learning issue of the student network when the student and teacher model sizes significantly differ. For example, TAKD~\cite{mirzadeh2020improved} proposes to reduce the discrepancy of teacher and student by resorting to an additional teaching assistant of moderate model size; DGKD~\cite{son2021densely} further improves TAKD by densely gathering all the assistant models to guide the student. However, increasing the model size is only one of the popular approaches to have a stronger teacher. There lacks a thorough analysis on the training strategies to derive a stronger teacher and their effect on KD. Most importantly, a generic enough solution is preferred to address the difficulty of KD brought by stronger teachers, rather than struggling to deal with different types of stronger teachers (with larger model size or stronger training strategy) individually. 

To understand what makes a stronger teacher and their effect on KD, we systematically study the prevalent strategies for designing and training deep neural networks, and show that:
\begin{itemize}
    \item Beyond scaling up the model size, a stronger teacher can also be derived through advanced training strategies, \emph{e.g.}} \def\Eg{\emph{E.g.}, label smoothing and data augmentation~\cite{zhang2018mixup}. However, given a stronger teacher, the student's performance on the vanilla KD could be dropped, even worse than training from scratch without KD, as shown in Figure~\ref{fig:f1}.
    
    \item The discrepancy between teacher and student tends to get fairly larger when we switch their training strategy to a stronger one (see Figure~\ref{fig:error}). In this case, an exact recovery of predictions via KL divergence could be challenging and lead to the failure of vanilla KD.
    
    \item Preserving the \textit{relation of predictions} between teacher and student is sufficient and effective. When transferring the knowledge from teacher to student, what we really care about is preserving the preference (relative ranks of predictions) by the teacher, instead of recovering the absolute values accurately. Correlation between teacher and student predictions could be favored to relax the exact match of KL divergence and distill the intrinsic relations. 
\end{itemize}

In this paper, we thus leverage the Pearson correlation coefficient \cite{pearson1896vii} as a new match manner to replace the KL divergence.
In addition, besides the \textit{inter-class relations} in prediction vector (see Figure~\ref{fig:f3}), with the intuition that different instances have different spectrum of similarities with respect to each class, we also propose to distill the \textit{intra-class relations} for further boosting the performance as Figure~\ref{fig:f3}. Concretely, for each class, we gather its corresponding predicted probabilities of all instances in a batch, then transfer this relation from teacher to student. Our proposed method (dubbed DIST) is super simple, efficient, and practical, which can be implemented with only several lines of code (see Appendix~\ref{sec:impl}) and has almost the same training cost as the vanilla KD. As a result, the student can be liberated from the burden of matching the exact output of a strong teacher, but only be guided appropriately to distill those truly informative relations. 

Extensive experiments are conducted on benchmark datasets to verify our effectiveness on various tasks, including image classification, object detection, and semantic segmentation. Experimental results show that our DIST significantly outperforms vanilla KD and those sophisticatedly-designed state-of-the-art KD methods. For example, with the same baseline settings on ImageNet, our DIST achieves the highest 72.07\% accuracy on ResNet-18. With the stronger strategy, our method obtains 82.3\% accuracy on the recent transformer Swin-T~\cite{liu2021swin}, improving KD by 1\%. 

\begin{figure}[t]
    \centering
    \includegraphics[width=\textwidth]{figures/f3.pdf}
    \vspace{-2mm}
    \caption{\textbf{Difference between our DIST and existing KD methods.} Conventional KD matches the outputs of student ($s\in \mathbb{R}^5$) to teacher ($t\in \mathbb{R}^5$) point-wisely; instance relation methods operate on the feature level and measure the internal correlations (corr.) between instances in student and teacher separately, then transfer the teacher's correlations to student. Our DIST proposes to maintain the inter-class and intra-class relations between student and teacher. Inter-class relation: correlation between the predicted probabilistic distributions on each instance of teacher and student. Intra-class relation: correlation of the probabilities of all the instances on each classes.}
    \label{fig:f3}
\end{figure}

\section{Revisiting Prediction Match of KD}

In vanilla knowledge distillation~\cite{hinton2015distilling}, the knowledge is transferred from a pre-trained teacher model to a student model by minimizing the discrepancy between the prediction scores of the teacher and student models.

Formally, with the logits $\bm{Z}^{(\mathrm{s})} \in \mathbb{R}^{B\times C}$ and $\bm{Z}^{(\mathrm{t})} \in \mathbb{R}^{B\times C}$ of student and teacher networks, where $B$ and $C$ denote batch size and the number of classes, respectively, the vanilla KD loss~\cite{hinton2015distilling} is represented as
\begin{align} \label{eq:kd_loss}
    \begin{split}
    \L_\mathrm{KD} &:= \frac{\tau^2}{B}\sum_{i=1}^B\mbox{KL}(\bm{Y}^{(\mathrm{t})}_{i,:}, \bm{Y}^{(\mathrm{s})}_{i,:}) = \frac{\tau^2}{B}\sum_{i=1}^B\sum_{j=1}^{C} Y^{(\mathrm{t})}_{i,j} \mbox{log} \left(\frac{Y^{(\mathrm{t})}_{i,j}}{Y^{(\mathrm{s})}_{i,j}}\right) ,
    \end{split}
\end{align}
where KL refers to Kullback–Leibler divergence with
\begin{align}
\bm{Y}^{(\mathrm{s})}_{i,:} = softmax(\bm{Z}^{(\mathrm{s})}_{i,:} / \tau), \quad
\bm{Y}^{(\mathrm{t})}_{i,:} = softmax(\bm{Z}^{(\mathrm{t})}_{i,:} / \tau), 
\end{align}
being the probabilistic prediction vectors, and $\tau$ is the temperature factor to control the softness of logits.

In addition to the teacher's soft targets in Eq.(\ref{eq:kd_loss}), KD~\cite{hinton2015distilling} stated that it is beneficial to train the student together with ground-truth labels, and the overall training loss is composed of the original classification loss $\L_\mathrm{cls}$ and KD loss $\L_\mathrm{KD}$, \emph{i.e.}} \def\Ie{\emph{I.e.},
\begin{equation} \label{eq:kd_total_loss}
    \L_\mathrm{tr} = \alpha \L_\mathrm{cls} + \beta \L_\mathrm{KD} ,
\end{equation}
where $\L_\mathrm{cls}$ is usually the cross-entropy loss between the predictions of student network and ground-truth labels, $\alpha$ and $\beta$ are factors for balancing the losses.

\subsection{Catastrophic discrepancy with a stronger teacher}
As illustrated in Section \ref{introduction}, the effect of a teacher on KD has not been sufficiently investigated, especially when the performance of pre-trained teacher grows stronger, such as with larger model size or being trained with more advanced and competing strategies, \emph{e.g.}} \def\Eg{\emph{E.g.}, label smoothing, mix-up~\cite{zhang2018mixup}, auto augmentations~\cite{cubuk2020randaugment}, \textit{etc}. With this regard,  
as Figure~\ref{fig:error}, we train ResNet-18 and ResNet-50 standalone with strategy B1 and strategy B2\footnote{Training with B2 obtains higher accuracy compared to B1, \emph{e.g.}} \def\Eg{\emph{E.g.}, 73.4\% (B2) vs. 69.8\% (B1) on ResNet-18.}, and obtain 4 trained models (\textit{R18B1}, \textit{R18B2}, \textit{R50B1}, and \textit{R50B2} with accuracies 69.76\%, 73.4\%, 76.13\%, and 78.5\%, respectively), then compare their discrepancy using KL divergence ($\tau=1$ and $\tau=4$) on the predicted probabilities $\bm{Y}$. We have the following observations:
\begin{itemize}
    \item The outputs of ResNet-18 do not change much with the stronger strategy compared to ResNet-50. This implies that the representational capacity limits the student's performance, and it tends to be fairly challenging for the student to exactly match the teacher's outputs as their discrepancy becomes larger.
    
    \item When the teacher and student models are trained with a stronger strategy, the discrepancy between teacher and student would be larger. This indicates that when we adopt KD with a stronger training strategy, the misalignment between KD loss and classification loss would be severer, thus disturbing the student's training.
\end{itemize}

As a result, the exact match (\emph{i.e.}} \def\Ie{\emph{I.e.}, the loss reaches the minimal if and only if the teacher and student outputs are exactly identical) with KL divergence seems way too overambitious and demanding since the discrepancy between student and teacher can be considerably huge. Since the exact match can be detrimental with a stronger teacher, our intuition is to develop a relaxed manner for matching the predictions between the teacher and student. 


\section{DIST: Distillation from A Stronger Teacher}


\subsection{Relaxed match with relations}
The prediction scores indicate the teacher's confidence (or preference) over all classes. For a relaxed match of predictions between the teacher and student, we are motivated to consider what we really care about for the teacher's output. Instead of the exact probabilistic values, actually, during inference, we are only concerned about
their \textbf{relations}, \emph{i.e.}} \def\Ie{\emph{I.e.}, relative ranks of predictions of teacher. 

In this way, for some metric $d(\cdot,\cdot)$ with $\mathbb{R}^C \times \mathbb{R}^C \rightarrow \mathbb{R}^+$, the exact match can be formulated that $d(\a,\b)=0$ if $\a=\b$ for any two prediction vector as $\bm{Y}^{(\mathrm{s})}_{i,:}$ and  $\bm{Y}^{(\mathrm{t})}_{i,:}$ in the KL divergence of Eq.(\ref{eq:kd_loss}). Then as a relaxed match, we can introduce additional mappings $\phi(\cdot)$ and $\psi(\cdot)$ with $ \mathbb{R}^C \rightarrow \mathbb{R}^C$ such that 
\begin{equation}\label{relax}
d(\phi(\a),\psi(\b)) =  d(\a,\b), \forall \a,\b  
\end{equation}
Therefore, $d(\a,\b)=0$ does not necessarily require $\a$ and $\b$ should be exactly the same. Nevertheless, since we care about the relation within $\a$ or $\b$, the mappings $\phi$ and $\psi$ should be isotone and do not affect the semantic information and inference result of the prediction vector. 

With this regard, a simple yet effective choice for the isotone mapping is the positive linear transformation, namely, 
\begin{equation}\label{linear}
d(m_1\a+n_1, m_2\b+n_2) =  d(\a,\b), 
\end{equation}
where $m_1$, $m_2$, $n_1$, and $n_2$ are constants with $m_1\times m_2>0$. As a result, this match could be invariant under separate changes in scale and shift for the predictions. Actually, to satisfy the property Eq.(\ref{linear}), we can thus adopt the widely-used Pearson’s distance as the metric, \emph{i.e.}} \def\Ie{\emph{I.e.}, 
\begin{equation} \label{eq:pearson_distance}
    d_\mathrm{p} (\u, \v) := 1 - \rho_\mathrm{p} (\u, \v).
\end{equation}
$\rho_\mathrm{p} (\u, \v)$ is the Pearson correlation coefficient between two random variables $\u$ and $\v$, 
\begin{align} \label{eq:pearson}
    \begin{split}
        \rho_\mathrm{p} (\u, \v) &:= \frac{\mbox{Cov}(\u, \v)}{\mbox{Std}(\u) \mbox{Std}(\v)} 
       
        = \frac{\sum_{i=1}^C (u_i - \bar{u})(v_i - \bar{v})}{\sqrt{\sum_{i=1}^C(u_i - \bar{u})^2 \sum_{i=1}^C(v_i - \bar{v})^2}} \\
       
    \end{split}
\end{align}
where $\mbox{Cov}(\u, \v)$ is the covariance of $\u$ and $\v$, $\bar{u}$ and $\mbox{Std}(\u)$ denote the mean and standard derivation of $\u$, respectively.

In this way, we can define the \textbf{relation as correlation}. More specifically, and the original exact match in vanilla KD~\cite{hinton2015distilling} can thus be relaxed and replaced by maximizing the linear correlation to preserve the relation of teacher and student on the probabilistic distribution of each instance, which we call \textit{inter-class relation}. Formally, for each pair of prediction vector $\bm{Y}^{(\mathrm{s})}_{i,:}$ and  $\bm{Y}^{(\mathrm{t})}_{i,:}$, the inter-relation loss can be formulated as 
\begin{equation}
    \L_\mathrm{inter} := \frac{1}{B} \sum_{i=1}^{B} d_\mathrm{p}(\bm{Y}^{(\mathrm{s})}_{i,:}, \bm{Y}^{(\mathrm{t})}_{i,:}) .
\end{equation}
Some isotone mappings or metrics can also be used to relax the match as Eq.(\ref{relax}), such as cosine similarity investigated empirically in Section~\ref{sec:ablation}; other more advanced and delicate choices could be left as future work.






\subsection{Better distillation with intra-relations}

Besides the inter-class relation, for each class, the prediction scores of different instances could also be informative and useful. 
For instance, suppose we have three input images containing ``plane'', ``dog'', and ``cat'', respectively. Regarding the prediction scores in the ``cat'' class, the values of ``plane'' should be the smallest, ``dog'' has a moderate magnitude, and the ``cat'' should have the highest confidence. Besides, even for the images from the same class, the intrinsic intra-class variance of the semantic similarities is actually also informative. It indicates the prior from the teacher that \textit{which one is more reliable to cast in this class}.  

Therefore, we also encourage to distill this \textit{intra-relation} for better performance. Actually, define prediction matrix $\bm{Y}^{(\mathrm{s})}$ and $\bm{Y}^{(\mathrm{t})}$ with each row as $\bm{Y}^{(\mathrm{s})}_{i,:}$ and  $\bm{Y}^{(\mathrm{t})}_{i,:}$, then the above inter-relation is to maximize the correlation row-wisely (see Figure \ref{fig:f3}). In contrast, for intra-relation, the corresponding loss is thus to maximize the correlation column-wisely, \emph{i.e.}} \def\Ie{\emph{I.e.}, 
\begin{equation}
    \L_\mathrm{intra} := \frac{1}{C} \sum_{j=1}^{C} d_\mathrm{p}(\bm{Y}^{(\mathrm{s})}_{:,j}, \bm{Y}^{(\mathrm{t})}_{:,j}) .
\end{equation}

As a result, the overall training loss $\L_\mathrm{tr}$ can be composed of the classification loss, inter-class KD loss, and intra-class KD loss, \emph{i.e.}} \def\Ie{\emph{I.e.},
\begin{equation} \label{eq:rekd_total_loss}
    \L_\mathrm{tr} = \alpha\L_\mathrm{cls} + \beta\L_\mathrm{inter} + \gamma\L_\mathrm{intra} ,
\end{equation}
where $\alpha$, $\beta$, and $\gamma$ are factors for balancing the losses. In this way, via the relation loss, we have endowed the student with freedom more or less to match the teacher network's output adaptively, thus boosting the distillation performance to a great extent. 
\input{tables/training_strategies.tex}

\section{Experiments}

\subsection{Experimental settings}
\textbf{Training strategies.} The training strategies of image classification task are summarized in Table~\ref{tab:strategies}.
\textbf{CIFAR-100.} For fair comparisons, we use the same training strategies (referred to \textit{A1} in Table~\ref{tab:strategies}) and pretrained models following CRD~\cite{tian2019contrastive}. 
\textbf{ImageNet.} B1: for comparisons with previous KD methods, we train our baselines with the same simple training strategy as CRD~\cite{tian2019contrastive}. B2: to validate the effectiveness of KD methods on modern training strategies, we follow EfficientNet~\cite{tan2019efficientnet} and design a training strategy B2, which can significantly improve the performance compared to B1. B3: the strategy B3 is used for training Swin-Transformers~\cite{liu2021swin}, and contains even more stronger data augmentations and regularization.

\textbf{Loss weights.} On CIFAR-100 and ImageNet, we set $\alpha=1$, $\beta=2$, and $\gamma=2$ in Eq.(\ref{eq:rekd_total_loss}). 
On object detection and semantic segmentation, these three factors are all equal to $1$. For KD~\cite{hinton2015distilling}, we set $\alpha=0.9$, $\beta=1$ in Eq.(\ref{eq:kd_total_loss}), and use a default temperature $\tau=4$.

\subsection{Image Classification}

\input{tables/baseline.tex}

\textbf{Baseline results on ImageNet.} We first compare our method with prior works using the baseline settings. As shown in Table~\ref{tab:baseline}, our DIST significantly outperforms prior KD methods. Note that our method is only conducted on the outputs of models, and has a similar computational cost as KD~\cite{hinton2015distilling}. Nevertheless, it even achieves better performance compared to those sophisticatedly-designed methods. For example, CRD~\cite{tian2019contrastive} needs to preserve a memory bank for all 128-d features of ImageNet images, and produces additional 260M FLOPs of computation cost; SRRL~\cite{yang2020knowledge} and Review~\cite{chen2021distilling} require additional convolutions for feature alignments. The implementation of DIST can be found in Appendix~\ref{sec:impl}, which is quite simple compared to these methods.

\textbf{Distillation from stronger teacher models.} As the stronger teachers come from larger model sizes and stronger strategies, we here first conduct experiments to compare our DIST with the vanilla KD on different scales (model sizes) of ResNets with baseline strategy B1. As shown in Table~\ref{tab:teachers}, when the teacher goes larger, the ResNet-18 students perform even worse than that with a medium-sized ResNet-50 teacher. Nevertheless, our DIST shows an upward trend with larger teachers, and the improvements compared to KD also become more significant, indicating that our DIST tackles better on the large discrepancy between the student and larger teacher.

\input{tables/baseline_teachers.tex}
\input{tables/sota_strategy.tex}

\textbf{Distillation from stronger training strategies.} Recently, the performance of models on ImageNet has been significantly improved by the sophisticated training strategies and strong data augmentations (\emph{e.g.}} \def\Eg{\emph{E.g.}, TIMM~\cite{wightman2021resnet} achieves 80.4\% accuracy on ResNet-50 while the baseline strategy B1 only obtains 76.1\%). However, most of the KD methods still conduct experiments with simple training settings. It is seldomly investigated whether the KD methods are suitable to the advanced strategies. 
In this way, we conduct experiments with advanced training strategies and compare our method with vanilla KD, instance relation-based RKD~\cite{park2019relational}, and SRRL~\cite{yang2020knowledge}.

We first train traditional CNNs with strong strategies, and also use a strong ResNet-50 with 80.1\% accuracy trained by \cite{wightman2021resnet} as the teacher. As results shown in Table~\ref{tab:sota}, on both similar architectures (ResNet-18, ResNet-34) and dissimilar architectures (MobileNetV2, EfficientNet-B0), our DIST can achieve the best performance. Note that RKD and SRRL can perform worse than training from scratch, especially when the students are small (ResNet-18 and MobileNet) or the architectures of teacher and student are fairly different (ResNet-50 and Swin-L), this might be because they focus on the intermediate features, which can be more challenging for the student to recover teacher's features compared to predictions.

Furthermore, we experiment on the recent state-of-the-art Swin-Transformer~\cite{liu2021swin}. The results show that our DIST gains improvements on even more stronger models and strategies. For example, with Swin-L teacher, our method improves ResNet-50 and Swin-T by 1.7\% and 1.0\%, respectively.


\textbf{CIFAR-100.}
The results on CIFAR-100 dataset in Table~\ref{tab:cifar100} show that, by distilling on the predicted logits, our method even outperforms those sophisticatedly-designed feature distillation methods.

\input{tables/cifar100.tex}

\subsection{Object Detection}

\input{tables/det_seg.tex}

We further investigate the effectiveness of DIST on downstream tasks. We conduct experiments on MS COCO object detection dataset~\cite{lin2014microsoft}, and simply leverage our DIST as an additional supervision on the final predictions of classes. Following \cite{zhang2020improve, shu2021channel}, we use the same standard training strategies and utilize Cascade Mask R-CNN~\cite{Cai_2019} with ResNeXt-101 backbone as the teacher for two-stage student of Faster R-CNN~\cite{lin2017feature} with ResNet-50 backbone; while for one-stage RetinaNet~\cite{lin2017focal} with ResNet-50 backbone, the RetinaNet with ResNeXt-101 backbone is utilized as the teacher.

As shown in Table~\ref{tab:det}, our DIST achieves competitive results on COCO validation set. For comparisons, we train the vanilla KD under the same settings as our DIST, the results show that our DIST significantly outperforms vanilla KD by simply replacing the loss functions. Moreover, by combining DIST with mimic, which minimizes the mean square error between FPN features of teacher and student, we can even outperform the state-of-the-art KD methods designed for object detection.

\subsection{Semantic Segmentation}
We also perform experiments on semantic segmentation, a challenging dense prediction task. Following \cite{wang2020intra, shu2021channel, yang2022cross}, we train DeepLabV3~\cite{chen2018encoder} and PSPNet~\cite{zhao2017pyramid} with ResNet-18 backbone on Cityscapes dataset, and adopt our DIST on the predictions of classification head using a teacher with ResNet-101 backbone of DeepLabV3. As the results summarized in Table~\ref{tab:seg}, with only the supervision of class predictions, our DIST can significantly outperform existing knowledge distillation methods on semantic segmentation task. For example, our DIST outperforms recent state-of-the-art method CIRKD~\cite{yang2022cross} by 1.58\% on PSPNet-R18. This demonstrate our effectiveness on relation modeling.


\subsection{Ablation studies} \label{sec:ablation}

\textbf{Effects of inter-class and intra-class correlations.} This paper proposes two types of relations: inter-class and intra-class relations. To validate the effectiveness of each relation, we conduct experiments to train students with these relations separately. The results on Table~\ref{tab:ab_logit_class} verify that, both inter-class and intra-class relations can outperform the vanilla KD; also, the performance could be further boosted by combining them together.

\input{tables/ab_logit_class.tex}
\textbf{Effect of intra-class relation in vanilla KD.} To investigate the effectiveness of intra-class relation in vanilla KD, we adopt experiments to train our DIST using KL divergence as the relation metric, denoted as \textit{DIST (KL div.)}\footnote{Specifically, the vanilla KD is the same as DIST (KL div.) with inter-class relation only.}. As the results summarized in Table~\ref{tab:ab_logit_class}, adding intra-class relation in the vanilla KD can also improve the performance (from 71.21\% to 71.62\%). However, when the student is trained with intra-class relation only, the improvement of using KL divergence is less significant than using Pearson correlation (70.61\% vs. 71.55\%), since the means and variances of intra-class distributions could be varied.


\begin{wrapfigure}{r}{0.5\textwidth}
    \vspace{-4mm}
    \centering
    \subfigure[Inter-class correlation]
    {\includegraphics[width=0.48\linewidth]{figures/logit_correlation.pdf}}
    \subfigure[Intra-class correlation]
    {\includegraphics[width=0.48\linewidth]{figures/class_correlation.pdf}}
    \vspace{-2mm}
    \caption{\textbf{Correlations between ResNet-18 student and ResNet-34 teacher.} We train the methods on ImageNet with B1 strategy.}
    \label{fig:correlation}
    \vspace{-2mm}
\end{wrapfigure}
\textbf{Correlations between teacher and student.} To validate the effectiveness of our correlation-based loss, we measure the correlations between teacher and student models, where the student models are trained by plain classification loss, KD, and our DIST. We choose commonly used Pearson correlation coefficient, Spearman's~\cite{dodge2008concise} and Kendall's Tau~\cite{kendall1938new} rank correlation coefficients as the metrics of correlation. As summarized in Figure~\ref{fig:correlation}, our DIST obtains higher inter-class and intra-class correlations compared to baselines.

\textbf{Effect of training students with KD loss only.} Training student with only the KD loss can better reflect the distillation ability and the information richness of supervision signals. As results in Table~\ref{tab:ab_kd_wo_ce} show that, when the student is trained with only the KD loss, our DIST significantly outperforms the vanilla KD. Without using the ground-truth labels, it can even outperform the standalone training accuracy, which indicates the effectiveness of our DIST in distilling those truly-beneficial relations. 

\input{tables/ab_kd_wo_ce.tex}


\section{Conclusion}
This paper presents a new knowledge distillation (KD) method named DIST to implement better distillation from a stronger teacher. We empirically study the catastrophic discrepancy problem between the student and a stronger teacher, and propose a 
relation-based loss to relax the exact match of KL divergence in a linear sense. Our method DIST is simple yet effective in handling strong teachers. Extensive experiments show our superiority in various benchmark tasks. For example, DIST even outperforms state-of-the-art KD methods designed specifically for object detection and semantic segmentation.

\nocite{gou2021knowledge}
\nocite{kong2020learning}
\nocite{you2017learning}
\nocite{zagoruyko2016paying}
\nocite{heo2019comprehensive}
\nocite{yang2020knowledge}
\nocite{chen2021distilling}
\nocite{romero2014fitnets}
\nocite{tung2019similarity}
\nocite{peng2019correlation}
\nocite{ahn2019variational}
\nocite{huang2021relational}
\nocite{park2019relational}
\nocite{passalis2020probabilistic}
\nocite{heo2019knowledge}
\nocite{kim2018paraphrasing}



{
\small















