\section{Introduction}
\label{sec:intro}

Please follow the steps outlined below when submitting your manuscript to the IEEE Computer Society Press.
This style guide now has several important modifications (for example, you are no longer warned against the use of sticky tape to attach your artwork to the paper), so all authors should read this new version.

\subsection{Language}

All manuscripts must be in English.

\subsection{Dual submission}

Please refer to the author guidelines on the CVPR\ 2023\ web page for a
discussion of the policy on dual submissions.

\subsection{Paper length}
Papers, excluding the references section, must be no longer than eight pages in length.
The references section will not be included in the page count, and there is no limit on the length of the references section.
For example, a paper of eight pages with two pages of references would have a total length of 10 pages.
{\bf There will be no extra page charges for CVPR\ 2023.}

Overlength papers will simply not be reviewed.
This includes papers where the margins and formatting are deemed to have been significantly altered from those laid down by this style guide.
Note that this \LaTeX\ guide already sets figure captions and references in a smaller font.
The reason such papers will not be reviewed is that there is no provision for supervised revisions of manuscripts.
The reviewing process cannot determine the suitability of the paper for presentation in eight pages if it is reviewed in eleven.

\subsection{The ruler}
The \LaTeX\ style defines a printed ruler which should be present in the version submitted for review.
The ruler is provided in order that reviewers may comment on particular lines in the paper without circumlocution.
If you are preparing a document using a non-\LaTeX\ document preparation system, please arrange for an equivalent ruler to appear on the final output pages.
The presence or absence of the ruler should not change the appearance of any other content on the page.
The camera-ready copy should not contain a ruler.
(\LaTeX\ users may use options of cvpr.sty to switch between different versions.)

Reviewers:
note that the ruler measurements do not align well with lines in the paper --- this turns out to be very difficult to do well when the paper contains many figures and equations, and, when done, looks ugly.
Just use fractional references (\eg, this line is $087.5$), although in most cases one would expect that the approximate location will be adequate.


\subsection{Paper ID}
Make sure that the Paper ID from the submission system is visible in the version submitted for review (replacing the ``*****'' you see in this document).
If you are using the \LaTeX\ template, \textbf{make sure to update paper ID in the appropriate place in the tex file}.


\subsection{Mathematics}

Please number all of your sections and displayed equations as in these examples:
\begin{equation}
  E = m\cdot c^2
  \label{eq:important}
\end{equation}
and
\begin{equation}
  v = a\cdot t.
  \label{eq:also-important}
\end{equation}
It is important for readers to be able to refer to any particular equation.
Just because you did not refer to it in the text does not mean some future reader might not need to refer to it.
It is cumbersome to have to use circumlocutions like ``the equation second from the top of page 3 column 1''.
(Note that the ruler will not be present in the final copy, so is not an alternative to equation numbers).
All authors will benefit from reading Mermin's description of how to write mathematics:
\url{http://www.pamitc.org/documents/mermin.pdf}.

\subsection{Blind review}

Many authors misunderstand the concept of anonymizing for blind review.
Blind review does not mean that one must remove citations to one's own work---in fact it is often impossible to review a paper unless the previous citations are known and available.

Blind review means that you do not use the words ``my'' or ``our'' when citing previous work.
That is all.
(But see below for tech reports.)

Saying ``this builds on the work of Lucy Smith [1]'' does not say that you are Lucy Smith;
it says that you are building on her work.
If you are Smith and Jones, do not say ``as we show in [7]'', say ``as Smith and Jones show in [7]'' and at the end of the paper, include reference 7 as you would any other cited work.

An example of a bad paper just asking to be rejected:
\begin{quote}
\begin{center}
    An analysis of the frobnicatable foo filter.
\end{center}

   In this paper we present a performance analysis of our previous paper [1], and show it to be inferior to all previously known methods.
   Why the previous paper was accepted without this analysis is beyond me.

   [1] Removed for blind review
\end{quote}


An example of an acceptable paper:
\begin{quote}
\begin{center}
     An analysis of the frobnicatable foo filter.
\end{center}

   In this paper we present a performance analysis of the  paper of Smith \etal [1], and show it to be inferior to all previously known methods.
   Why the previous paper was accepted without this analysis is beyond me.

   [1] Smith, L and Jones, C. ``The frobnicatable foo filter, a fundamental contribution to human knowledge''. Nature 381(12), 1-213.
\end{quote}

If you are making a submission to another conference at the same time, which covers similar or overlapping material, you may need to refer to that submission in order to explain the differences, just as you would if you had previously published related work.
In such cases, include the anonymized parallel submission~\cite{Authors14} as supplemental material and cite it as
\begin{quote}
[1] Authors. ``The frobnicatable foo filter'', F\&G 2014 Submission ID 324, Supplied as supplemental material {\tt fg324.pdf}.
\end{quote}

Finally, you may feel you need to tell the reader that more details can be found elsewhere, and refer them to a technical report.
For conference submissions, the paper must stand on its own, and not {\em require} the reviewer to go to a tech report for further details.
Thus, you may say in the body of the paper ``further details may be found in~\cite{Authors14b}''.
Then submit the tech report as supplemental material.
Again, you may not assume the reviewers will read this material.

Sometimes your paper is about a problem which you tested using a tool that is widely known to be restricted to a single institution.
For example, let's say it's 1969, you have solved a key problem on the Apollo lander, and you believe that the CVPR70 audience would like to hear about your
solution.
The work is a development of your celebrated 1968 paper entitled ``Zero-g frobnication: How being the only people in the world with access to the Apollo lander source code makes us a wow at parties'', by Zeus \etal.

You can handle this paper like any other.
Do not write ``We show how to improve our previous work [Anonymous, 1968].
This time we tested the algorithm on a lunar lander [name of lander removed for blind review]''.
That would be silly, and would immediately identify the authors.
Instead write the following:
\begin{quotation}
\noindent
   We describe a system for zero-g frobnication.
   This system is new because it handles the following cases:
   A, B.  Previous systems [Zeus et al. 1968] did not  handle case B properly.
   Ours handles it by including a foo term in the bar integral.

   ...

   The proposed system was integrated with the Apollo lunar lander, and went all the way to the moon, don't you know.
   It displayed the following behaviours, which show how well we solved cases A and B: ...
\end{quotation}
As you can see, the above text follows standard scientific convention, reads better than the first version, and does not explicitly name you as the authors.
A reviewer might think it likely that the new paper was written by Zeus \etal, but cannot make any decision based on that guess.
He or she would have to be sure that no other authors could have been contracted to solve problem B.
\medskip

\noindent
FAQ\medskip\\
{\bf Q:} Are acknowledgements OK?\\
{\bf A:} No.  Leave them for the final copy.\medskip\\
{\bf Q:} How do I cite my results reported in open challenges?
{\bf A:} To conform with the double-blind review policy, you can report results of other challenge participants together with your results in your paper.
For your results, however, you should not identify yourself and should not mention your participation in the challenge.
Instead present your results referring to the method proposed in your paper and draw conclusions based on the experimental comparison to other results.\medskip\\

\begin{figure}[t]
  \centering
  \fbox{\rule{0pt}{2in} \rule{0.9\linewidth}{0pt}}
  

   \caption{Example of caption.
   It is set in Roman so that mathematics (always set in Roman: $B \sin A = A \sin B$) may be included without an ugly clash.}
   \label{fig:onecol}
\end{figure}

\subsection{Miscellaneous}

\noindent
Compare the following:\\
\begin{tabular}{ll}
 \verb'$conf_a$' &  $conf_a$ \\
 \verb'$\mathit{conf}_a$' & $\mathit{conf}_a$
\end{tabular}\\
See The \TeX book, p165.

The space after \eg, meaning ``for example'', should not be a sentence-ending space.
So \eg is correct, {\em e.g.} is not.
The provided \verb'\eg' macro takes care of this.

When citing a multi-author paper, you may save space by using ``et alia'', shortened to ``\etal'' (not ``{\em et.\ al.}'' as ``{\em et}'' is a complete word).
If you use the \verb'\etal' macro provided, then you need not worry about double periods when used at the end of a sentence as in Alpher \etal.
However, use it only when there are three or more authors.
Thus, the following is correct:
   ``Frobnication has been trendy lately.
   It was introduced by Alpher~\cite{Alpher02}, and subsequently developed by
   Alpher and Fotheringham-Smythe~\cite{Alpher03}, and Alpher \etal~\cite{Alpher04}.''

This is incorrect: ``... subsequently developed by Alpher \etal~\cite{Alpher03} ...'' because reference~\cite{Alpher03} has just two authors.




\begin{figure*}
  \centering
  \begin{subfigure}{0.68\linewidth}
    \fbox{\rule{0pt}{2in} \rule{.9\linewidth}{0pt}}
    \caption{An example of a subfigure.}
    \label{fig:short-a}
  \end{subfigure}
  \hfill
  \begin{subfigure}{0.28\linewidth}
    \fbox{\rule{0pt}{2in} \rule{.9\linewidth}{0pt}}
    \caption{Another example of a subfigure.}
    \label{fig:short-b}
  \end{subfigure}
  \caption{Example of a short caption, which should be centered.}
  \label{fig:short}
\end{figure*}

\section{Formatting your paper}
\label{sec:formatting}

All text must be in a two-column format.
The total allowable size of the text area is $6\frac78$ inches (17.46 cm) wide by $8\frac78$ inches (22.54 cm) high.
Columns are to be $3\frac14$ inches (8.25 cm) wide, with a $\frac{5}{16}$ inch (0.8 cm) space between them.
The main title (on the first page) should begin 1 inch (2.54 cm) from the top edge of the page.
The second and following pages should begin 1 inch (2.54 cm) from the top edge.
On all pages, the bottom margin should be $1\frac{1}{8}$ inches (2.86 cm) from the bottom edge of the page for $8.5 \times 11$-inch paper;
for A4 paper, approximately $1\frac{5}{8}$ inches (4.13 cm) from the bottom edge of the
page.

\subsection{Margins and page numbering}

All printed material, including text, illustrations, and charts, must be kept
within a print area $6\frac{7}{8}$ inches (17.46 cm) wide by $8\frac{7}{8}$ inches (22.54 cm)
high.
Page numbers should be in the footer, centered and $\frac{3}{4}$ inches from the bottom of the page.
The review version should have page numbers, yet the final version submitted as camera ready should not show any page numbers.
The \LaTeX\ template takes care of this when used properly.



\subsection{Type style and fonts}

Wherever Times is specified, Times Roman may also be used.
If neither is available on your word processor, please use the font closest in
appearance to Times to which you have access.

MAIN TITLE.
Center the title $1\frac{3}{8}$ inches (3.49 cm) from the top edge of the first page.
The title should be in Times 14-point, boldface type.
Capitalize the first letter of nouns, pronouns, verbs, adjectives, and adverbs;
do not capitalize articles, coordinate conjunctions, or prepositions (unless the title begins with such a word).
Leave two blank lines after the title.

AUTHOR NAME(s) and AFFILIATION(s) are to be centered beneath the title
and printed in Times 12-point, non-boldface type.
This information is to be followed by two blank lines.

The ABSTRACT and MAIN TEXT are to be in a two-column format.

MAIN TEXT.
Type main text in 10-point Times, single-spaced.
Do NOT use double-spacing.
All paragraphs should be indented 1 pica (approx.~$\frac{1}{6}$ inch or 0.422 cm).
Make sure your text is fully justified---that is, flush left and flush right.
Please do not place any additional blank lines between paragraphs.

Figure and table captions should be 9-point Roman type as in \cref{fig:onecol,fig:short}.
Short captions should be centred.

\noindent Callouts should be 9-point Helvetica, non-boldface type.
Initially capitalize only the first word of section titles and first-, second-, and third-order headings.

FIRST-ORDER HEADINGS.
(For example, {\large \bf 1. Introduction}) should be Times 12-point boldface, initially capitalized, flush left, with one blank line before, and one blank line after.

SECOND-ORDER HEADINGS.
(For example, { \bf 1.1. Database elements}) should be Times 11-point boldface, initially capitalized, flush left, with one blank line before, and one after.
If you require a third-order heading (we discourage it), use 10-point Times, boldface, initially capitalized, flush left, preceded by one blank line, followed by a period and your text on the same line.

\subsection{Footnotes}

Please use footnotes\footnote{This is what a footnote looks like.
It often distracts the reader from the main flow of the argument.} sparingly.
Indeed, try to avoid footnotes altogether and include necessary peripheral observations in the text (within parentheses, if you prefer, as in this sentence).
If you wish to use a footnote, place it at the bottom of the column on the page on which it is referenced.
Use Times 8-point type, single-spaced.


\subsection{Cross-references}

For the benefit of author(s) and readers, please use the
{\small\begin{verbatim}
  \cref{...}
\end{verbatim}}  command for cross-referencing to figures, tables, equations, or sections.
This will automatically insert the appropriate label alongside the cross-reference as in this example:
\begin{quotation}
  To see how our method outperforms previous work, please see \cref{fig:onecol} and \cref{tab:example}.
  It is also possible to refer to multiple targets as once, \eg~to \cref{fig:onecol,fig:short-a}.
  You may also return to \cref{sec:formatting} or look at \cref{eq:also-important}.
\end{quotation}
If you do not wish to abbreviate the label, for example at the beginning of the sentence, you can use the
{\small\begin{verbatim}
  \Cref{...}
\end{verbatim}}
command. Here is an example:
\begin{quotation}
  \Cref{fig:onecol} is also quite important.
\end{quotation}

\subsection{References}

List and number all bibliographical references in 9-point Times, single-spaced, at the end of your paper.
When referenced in the text, enclose the citation number in square brackets, for
example~\cite{Authors14}.
Where appropriate, include page numbers and the name(s) of editors of referenced books.
When you cite multiple papers at once, please make sure that you cite them in numerical order like this \cite{Alpher02,Alpher03,Alpher05,Authors14b,Authors14}.
If you use the template as advised, this will be taken care of automatically.

\begin{table}
  \centering
  \begin{tabular}{@{}lc@{}}
    \toprule
    Method & Frobnability \\
    \midrule
    Theirs & Frumpy \\
    Yours & Frobbly \\
    Ours & Makes one's heart Frob\\
    \bottomrule
  \end{tabular}
  \caption{Results.   Ours is better.}
  \label{tab:example}
\end{table}

\subsection{Illustrations, graphs, and photographs}

All graphics should be centered.
In \LaTeX, avoid using the \texttt{center} environment for this purpose, as this adds potentially unwanted whitespace.
Instead use
{\small\begin{verbatim}
  \centering
\end{verbatim}}
at the beginning of your figure.
Please ensure that any point you wish to make is resolvable in a printed copy of the paper.
Resize fonts in figures to match the font in the body text, and choose line widths that render effectively in print.
Readers (and reviewers), even of an electronic copy, may choose to print your paper in order to read it.
You cannot insist that they do otherwise, and therefore must not assume that they can zoom in to see tiny details on a graphic.

When placing figures in \LaTeX, it's almost always best to use \verb+\includegraphics+, and to specify the figure width as a multiple of the line width as in the example below
{\small\begin{verbatim}
   \usepackage{graphicx} ...
   \includegraphics[width=0.8\linewidth]
                   {myfile.pdf}
\end{verbatim}
}


\subsection{Color}

Please refer to the author guidelines on the CVPR\ 2023\ web page for a discussion of the use of color in your document.

If you use color in your plots, please keep in mind that a significant subset of reviewers and readers may have a color vision deficiency; red-green blindness is the most frequent kind.
Hence avoid relying only on color as the discriminative feature in plots (such as red \vs green lines), but add a second discriminative feature to ease disambiguation.

\section{Final copy}

You must include your signed IEEE copyright release form when you submit your finished paper.
We MUST have this form before your paper can be published in the proceedings.

Please direct any questions to the production editor in charge of these proceedings at the IEEE Computer Society Press:
\url{https://www.computer.org/about/contact}.


{\small
\bibliographystyle{ieee_fullname}

\section{Introduction}

After receiving paper reviews, authors may optionally submit a rebuttal to address the reviewers' comments, which will be limited to a {\bf one page} PDF file.
Please follow the steps and style guidelines outlined below for submitting your author response.

The author rebuttal is optional and, following similar guidelines to previous CVPR conferences, is meant to provide you with an opportunity to rebut factual errors or to supply additional information requested by the reviewers.
It is NOT intended to add new contributions (theorems, algorithms, experiments) that were absent in the original submission and NOT specifically requested by the reviewers.
You may optionally add a figure, graph, or proof to your rebuttal to better illustrate your answer to the reviewers' comments.

Per a passed 2018 PAMI-TC motion, reviewers should refrain from requesting significant additional experiments for the rebuttal or penalize for lack of additional experiments.
Authors should refrain from including new experimental results in the rebuttal, especially when not specifically requested to do so by the reviewers.
Authors may include figures with illustrations or comparison tables of results reported in the submission/supplemental material or in other papers.

Just like the original submission, the rebuttal must maintain anonymity and cannot include external links that reveal the author identity or circumvent the length restriction.
The rebuttal must comply with this template (the use of sections is not required, though it is recommended to structure the rebuttal for ease of reading).


\subsection{Response length}
Author responses must be no longer than 1 page in length including any references and figures.
Overlength responses will simply not be reviewed.
This includes responses where the margins and formatting are deemed to have been significantly altered from those laid down by this style guide.
Note that this \LaTeX\ guide already sets figure captions and references in a smaller font.

\section{Formatting your Response}

{\bf Make sure to update the paper title and paper ID in the appropriate place in the tex file.}

All text must be in a two-column format.
The total allowable size of the text area is $6\frac78$ inches (17.46 cm) wide by $8\frac78$ inches (22.54 cm) high.
Columns are to be $3\frac14$ inches (8.25 cm) wide, with a $\frac{5}{16}$ inch (0.8 cm) space between them.
The top margin should begin 1 inch (2.54 cm) from the top edge of the page.
The bottom margin should be $1\frac{1}{8}$ inches (2.86 cm) from the bottom edge of the page for $8.5 \times 11$-inch paper;
for A4 paper, approximately $1\frac{5}{8}$ inches (4.13 cm) from the bottom edge of the page.

Please number any displayed equations.
It is important for readers to be able to refer to any particular equation.

Wherever Times is specified, Times Roman may also be used.
Main text should be in 10-point Times, single-spaced.
Section headings should be in 10 or 12 point Times.
All paragraphs should be indented 1 pica (approx.~$\frac{1}{6}$ inch or 0.422 cm).
Figure and table captions should be 9-point Roman type as in \cref{fig:onecol}.


List and number all bibliographical references in 9-point Times, single-spaced,
at the end of your response.
When referenced in the text, enclose the citation number in square brackets, for example~\cite{Alpher05}.
Where appropriate, include the name(s) of editors of referenced books.

\begin{figure}[t]
  \centering
  \fbox{\rule{0pt}{0.5in} \rule{0.9\linewidth}{0pt}}
 
   \caption{Example of caption.  It is set in Roman so that mathematics
   (always set in Roman: $B \sin A = A \sin B$) may be included without an
   ugly clash.}
   \label{fig:onecol}
\end{figure}

To avoid ambiguities, it is best if the numbering for equations, figures, tables, and references in the author response does not overlap with that in the main paper (the reviewer may wonder if you talk about \cref{fig:onecol} in the author response or in the paper).
See \LaTeX\ template for a workaround.

\subsection{Illustrations, graphs, and photographs}

All graphics should be centered.
Please ensure that any point you wish to make is resolvable in a printed copy of the response.
Resize fonts in figures to match the font in the body text, and choose line widths which render effectively in print.
Readers (and reviewers), even of an electronic copy, may choose to print your response in order to read it.
You cannot insist that they do otherwise, and therefore must not assume that they can zoom in to see tiny details on a graphic.

When placing figures in \LaTeX, it is almost always best to use \verb+\includegraphics+, and to specify the  figure width as a multiple of the line width as in the example below
{\small\begin{verbatim}
   \usepackage{graphicx} ...
   \includegraphics[width=0.8\linewidth]
                   {myfile.pdf}
\end{verbatim}
}


{\small
\bibliographystyle{ieee_fullname}


\section{Introduction}
\label{sec:intro}

\begin{figure}[h]
    \centering
   
   
    \includegraphics[width=\linewidth]{figures/pull_mobile_2.pdf}
    \caption{\textbf{Top row}: when averaging performance on 6 tasks (ImageNet-1k and 5 transfer tasks) self-supervised distillation surpasses supervised pretraining and is competitive with supervised distillation. \textbf{Bottom left}: self-supervised distillation of MSN produces students with low-shot performance competitive with DINO or iBOT on larger architectures (measured on ImageNet-1k). \textbf{Bottom right}: RoB compared to previous works on SSL for low-compute network, leveraging knowledge distillation or not.}
    \label{fig:pull_figure}
\end{figure}

\begin{figure*}[h]
    \centering
    \includegraphics[width=\linewidth]{figures/combi_rob_claims_3.pdf}
    \caption{\textbf{Left}: on average over 6 tasks, RoB self-supervised distillation is advantageous over supervised distillation and SSL pretraining without distillation. \textbf{Middle}: Breakdown of transfer performance of RoB against supervised pretraining. \textbf{Right}: RoB ViT-Ti student low-shot performance against DINO ViT-Ti and the self-supervised distillation approach BINGO~\cite{xu2022bag}, SOTA on IN1k $1\%$ for RN18.}
    \label{fig:pull_figure_radar}
\end{figure*}




Self-Supervised Learning (SSL) has demonstrated promising performance for transfer learning~\cite{chen2020simple,caron2021emerging,zhou2021ibotyes,goyal2022vision}. Most methods focus on large neural network architectures with at least  $20$M parameters, ResNet50 architecture and bigger~\cite{he2019moco,caron2020unsupervised,chen2020simple}
or ViT-Small and bigger~\cite{caron2021emerging,zhou2021ibotyes,assran2022masked}
, with the most interesting results obtained with architectures such as ResNet50x4 ($64$GFlops) for SimCLR~\cite{chen2020simple}, or ViT-S/8 ($22$GFlops) and ViT-B/8 ($78$GFlops) for DINO~\cite{caron2021emerging}. At these scales, self-supervised methods have shown stronger transfer and semi-supervised performance than supervised pretraining.
At smaller scales however, ResNet50 and smaller, like ViT-Tiny ($1.3$GFlops) or ResNet18 ($1.8$GFlops) or MobileNetV3~\cite{mobilenet} ($0.24$GFlops), SSL approaches struggle to be competitive with their supervised counterparts~\cite{compress,seed}. This limits the scale of application of SSL methods, as ResNet50 and smaller architectures are the most commonly used architectures by practitioners \cite{https://doi.org/10.48550/arxiv.2106.05237}. Additionally, large models require significantly higher memory, compute, and storage requirements, limiting their use on low compute or low power devices, where lightweight models are usually preferable.




In this paper, we bring the benefits of self-supervised learning to neural network architectures with lower computational requirements.
In particular, we focus on strong off-the-shelf performance on transfer tasks, \eg, linear probe, few-shot, and semi-supervised settings.
We show that Knowledge Distillation~\cite{hinton_knowledge_distillation, compress, seed, xu2022bag, DBLP:journals/corr/abs-2201-05131, disco}, originally developed for supervised learning, can be easily adapted to transfer knowledge from a large self-supervised model (the teacher) to a compact smaller model (the student).



Our main insight is that the underlying training in many existing SSL joint-embedding methods~\cite{chen2020simple, caron2020unsupervised, caron2021emerging, zhou2021ibotyes} can be easily repurposed to a knowledge-distillation framework.
We propose a simple and generic self-supervised distillation method, named Replace one Branch (RoB), that adapts the joint-embedding approaches into distillation methods by simply replacing one of the branches with a pretrained self-supervised teacher.
We instantiate our approach for $4$ such methods: DINO \cite{caron2021emerging}, SwAV \cite{caron2020unsupervised} and iBOT \cite{zhou2021ibotyes} which are currently state of the art in the linear probe transfer setting, and MSN \cite{assran2022masked} as the current state of the start for low shot in-distribution benchmarks (semi-supervised setting).



When pretrained on ImageNet, RoB surpasses previous work in self-supervised distillation on ViT-Tiny and low compute ResNets (ResNet18 and ResNet34). More importantly, we demonstrate that the distilled students significantly surpass their supervised pre-trained counterparts on transfer tasks. More importantly, the self-supervised models are even competitive with low-compute neural nets obtained via supervised distillation (see Figures~\ref{fig:pull_figure} and~\ref{fig:pull_figure_radar}).

\par \noindent \textbf{Contributions}:
\begin{itemize}[leftmargin=*]
    \setlength\itemsep{-0.4em}
  \item We focus on SSL performance using small models and show that existing joint embedding based SSL methods can be repurposed to a knowledge distillation framework.
  \item We propose, RoB, a simple and generic self-supervised distillation approach for joint embedding self-supervised methods. RoB is applicable to a wide variety of methods and architectures.
  \item We demonstrate state of the art self-supervised ViT-Tiny, ResNet18 and ResNet34, improving upon previous work on self-supervision for low-compute networks.
  \item We demonstrate that self-supervision is competitive on linear transfer with supervised pre-training and supervised distillation.
\end{itemize}

\begin{figure*}[h]
    \centering
    \includegraphics[width=\linewidth]{figures/ssl_kd_method.pdf}
    \caption{\textbf{Overview of our method.} RoB adapts existing joint-embedding approaches (left) by replacing the branch that produces the target with a frozen teacher pre-trained with the same SSL method (right). Importantly, RoB removes the regularisation terms that aim at preventing collapse from the loss, and use identical-view predictions instead of cross-view predictions in the loss.}
    \label{fig:method}
\end{figure*}

\section{Related work}

\paragraph{Supervised distillation.}
The work of \cite{hinton_knowledge_distillation} introduced the concept of distillation of knowledge from a strong teacher or an ensemble of teacher to a lower-compute student by using a Kullback Leibler (KL) divergence term in the loss. Since then, several works have tried to push the performance of distillation by introducing additional loss terms \cite{https://doi.org/10.48550/arxiv.1412.6550, Tian2020Contrastive, https://doi.org/10.48550/arxiv.2205.10536} or by refining training recipes relying on KL divergence \cite{https://doi.org/10.48550/arxiv.2106.05237}.

Of particular interest to us is DIST \cite{Tian2020Contrastive}, a recent work that produced low compute ResNet18 and ResNet34 from ResNet Strikes Back teachers \cite{https://doi.org/10.48550/arxiv.2110.00476}, reaching state of the art ImageNet-1k top-1 accuracy of $74.5\%$ and $77.8\%$ respectively, and DeiT \cite{touvron2021training} for Vision Transformer. We will use those works as baselines.

\paragraph{Self-supervised Learning.}
While approaches based on reconstruction \cite{bao2021beit, he2021masked} have had a resurgence in the last years, the current state of the art in transfer linear probe is still held by joint-embedding approaches such as DINO \cite{caron2021emerging} or iBOT \cite{zhou2021ibotyes}. Those methods learn to maximize agreement between the representations of two different views of the same image or concept, while making sure that representations do not collapse to the trivial constant solution. Different solutions have been proposed to deal with the collapse issue, contrastive losses \cite{chen2020simple, he2019moco}, soft clustering \cite{caron2020unsupervised, zhou2021ibotyes, caron2021emerging, assran2022masked} and more \cite{https://doi.org/10.48550/arxiv.2006.07733, zbontar2021barlow, bardes2021vicreg}. Our work will concentrate on soft-clustering methods as they currently provide the state of the art linear probe performance.

\paragraph{Self-supervision on low-compute networks.}
Multiple works, such as CompRess \cite{compress}, SEED \cite{seed}, BINGO \cite{xu2022bag}, DisCo \cite{disco} or SimReg \cite{DBLP:journals/corr/abs-2201-05131}, have noticed that joint-embedding self-supervised learning methods such as SwAV \cite{caron2020unsupervised}, MoCo \cite{he2019moco, chen2020mocov2} or DINO \cite{caron2021emerging} suffer from a drop in performance when applied on low compute neural nets. These works have proposed to use Knowledge Distillation \cite{hinton_knowledge_distillation} to circumvent those difficulties. CompRess \cite{compress} and SEED \cite{seed} use a memory queue like MoCo \cite{he2019moco} to distill the knowledge of the teacher by minimizing the cross-entropy between the probability distribution of the teacher and student obtained by comparing a sample to each point in the queue. DisCo \cite{disco} and BINGO \cite{xu2022bag} makes use of contrastive learning, with BINGO additionally grouping samples into cluster of related samples. Finally, SimReg \cite{DBLP:journals/corr/abs-2201-05131} proposes regression as a generic way to transfer feature representation from a teacher to a student.

Another recent line of work \cite{anonymous2023effective, https://doi.org/10.48550/arxiv.2107.14762} has started to explore the reasons behind the performance drop on low compute architectures, with the work~\cite{anonymous2023effective} addressing some of the gap without the recourse to knowledge distillation

In this work, we will introduce a self-supervised distillation method and show it outperforms those previous approaches for training low-compute networks. Additionally, we will demonstrate that our approach is competitive with supervised training and supervised distillation.




\section{Replace one Branch}
\label{sec:background}
\label{sec:approach}

\paragraph{Background} Joint-embedding architectures~\cite{bromley1993signature} (Figure~\ref{fig:method}, left) aim at learning encoders that produce similar embeddings for different views extracted from a same image.

We consider two views $\mathbf{x_1}$ and $\mathbf{x_2}$ of a given image ${\bf x}$. A student  encoder $f_\theta(\cdot)$ and a teacher encoder $g_\phi(.)$ independently process one view and output the representations $z^S_1$ and $z^T_2$.
Training occurs by pushing the representations $z^S_1$ and $z^T_2$ to be close in order to learn view-invariant teacher/student encoders. The encoder architecture is usually shared across the branches. Teacher weights, however, are not necessary learned and might be updated through of exponential moving average of the student weights~\cite{grill2020bootstrap, caron2021emerging}.


One difficulty with joint-embedding architectures is to prevent representation collapse in which the encoder ignores its inputs and produces a constant image embedding. Several approaches have been investigated in the literature to avoid collapse~\cite{chen2020simple,bardes2021vicreg,caron2020unsupervised}. In this work, we apply our approach on cluster-based methods that use variants of  entropy maximization / prediction sharpening to avoid collapse~\cite{caron2021emerging,assran2022masked}.


\paragraph{RoB Overview}
Replace one Branch (Figure~\ref{fig:method}, left) is a method that explores self-supervised distillation to train low-compute neural network in an unsupervised fashion. 

Replace one Branch (RoB) simple recipe follows two principles: 1) we  replace the teacher encoder $g_\phi$ by an already trained self-supervised teacher $f^T_{\theta^T}$, which is kept frozen during distillation 2) we remove the mechanism that avoids collapse from the self-supervised loss. Representation collapse is indeed not an issue with RoB as the self-supervised teacher is fixed. As a final twist, RoB uses identical-view predictions, where the student representation $z^S_i$ is pushed toward the teacher representation of the same view $z^T_i$.


In the experiment section, we will demonstrate that despite its simplicity, RoB obtains state-of-art SSL performances for low-compute neural network.

We now show how to instantiate RoB for different SSL approaches: DINO~\cite{caron2021emerging}, IBOT~\cite{zhou2021ibotyes}, SwAV~\cite{caron2020unsupervised} and MSN~\cite{assran2022masked}.

\paragraph{RoB-DINO}
To distill from DINO, we replace its  online teacher with an pretrained teacher and remove the centering and sharpening terms in the loss, used to avoid representation collapse.
The RoB loss for DINO becomes:
\begin{equation}
    \mathcal{L_\text{DINO}} = \frac{1}{N} \big( \sum_{i=0}^1 H(z_i^T, z_i^S) + \sum_{i=2}^{N-1} \sum_{j=0}^1 H(z_j^T, z_i^S) \big),
\end{equation}
\label{eq:dino}where $z_i^T$ and  $z_i^S$  are the $i$th teacher and student representations of the $i$-view, and $H$ is the cross entropy. DINO uses  $N$ views from a given images, two large views and $N-2$ small views following muti-crop~\cite{caron2020unsupervised}. RoB uses the same number of views, keeping the same multi-crop mechanism.

\paragraph{RoB-MSN}
MSN~\cite{assran2022masked} tries to match the representation of a masked view, e.g. where some of the patches are randomly dropped,  to an unmasked view. The distillation of MSN is identical to the one of DINO and  only differs in the masking augmentation used in the student encoder.


\paragraph{RoB-iBOT}
iBOT\cite{zhou2021ibotyes} adds a patch-based loss to the DINO loss. The student predicts the representation of masked patches and matches them to the corresponding teacher outputs. We denote by $z_i,p$ the representation of a given patch computed from the view $x_i$. The RoB loss becomes:
\begin{equation}
    \mathcal{L_\text{iBOT}} = \lambda_1 \mathcal{L_\text{DINO}} + \frac{\lambda_2}{2 N_\text{mask}} \sum_{i=0}^1 \sum_{p=1}^{N_\text{mask}} H(z_{i,p}^T, z_{i,p}^S),
\end{equation}
where $z_{i,p}^T$ and $z_{i,p}^S$ denotes respectively the teacher representation and student prediction of the masked patch $p$.


\paragraph{RoB-SwAV}

SwAV relies on Sinkhorn-Knopp (SHK) regularization to avoid representation collapse. With the removal of Sinkhorn-Knopp, the SwAV self-supervised distillation loss $\mathcal{L}_\text{SwAV}$ is the same as the DINO distillation loss.


\section{Experiments}
\label{sec:experiments}

\input{tables/vit_tiny}

\subsection{Experimental setup}

\paragraph{Evaluation datasets.}
We evaluate performance of all of our models with linear evaluation on a range of downstream tasks requiring different levels of abstraction, i.e., classification with CIFAR100~\cite{krizhevsky2009learning}, Places205~\cite{zhou2014learning}, and iNat18~\cite{van2018inaturalist}, with respectively $100$, $205$ and $8142$ categories; object counting with Clevr/Count~\cite{johnson2017clevr, https://doi.org/10.48550/arxiv.1910.04867}; and depth prediction with Clevr/Dist~\cite{johnson2017clevr, https://doi.org/10.48550/arxiv.1910.04867}.
We also evaluate the in-distribution top-1 accuracy on ImageNet-1k ~\cite{russakovsky2015imagenet} (IN1k).

\paragraph{Evaluation protocol.}
For ViT-Tiny, the linear evaluation is performed using either the concatenation of  class tokens extracted from the last 4 layers, as done in DINO \cite{caron2021emerging}, or using only  the last class token, depending on what work best. For ResNets, linear evaluation is performed using the SEER recipes \cite{https://doi.org/10.48550/arxiv.2103.01988} that are available in the VISSL library \cite{goyal2021vissl}. Additional details on those evaluations are provided in the appendix \ref{appendix:eval_details}.




\paragraph{Supervised distillation baselines.}
To show that self-supervised distillation is of any practical interest, we compare it to supervised distillation.

In our experiments, we use DeiT \cite{touvron2021training} and DIST \cite{https://doi.org/10.48550/arxiv.2205.10536} as our supervised distillation baseline. DeiT \cite{touvron2021training} trains a Vision Transformer with an additional distillation token to learn from a RegNet16Gf teacher (with $82.9\%$ top-1 accuracy on ImageNet). DIST \cite{https://doi.org/10.48550/arxiv.2205.10536} distills ResNet Strikes Back \cite{https://doi.org/10.48550/arxiv.2110.00476} to ResNet18 and ResNet34, reaching $74.3\%$ and $77.8\%$ top-1 accuracy on ImageNet respectively.




\paragraph{Teachers.}
Our list of teachers for supervised distillation and self-supervised distillation include the RegNet16Gf trained in DEIT \cite{touvron2021training}, the ResNet50 trained in ResNet Strikes Back \cite{https://doi.org/10.48550/arxiv.2110.00476}, the SSL  ResNet50 and RegNet128Gf\cite{caron2020unsupervised, https://doi.org/10.48550/arxiv.2103.01988} and the iBOT \cite{zhou2021ibotyes}, DINO \cite{caron2021emerging} and MSN \cite{assran2022masked} ViT models. All teachers and their top-1 accuracy on ImageNet are listed in Appendix \ref{appendix:teachers_list}.

\subsection{Transfer performance of SSL distillation}
\label{sec:vit_tiny}

In this section, we investigate the transfer performance of self-supervised distillation and demonstrate RoB is a viable alternative to supervised distillation.


\paragraph{Vision Transformer.} We distill DINO \cite{caron2021emerging} and iBOT \cite{zhou2021ibotyes} teachers to a vanilla ViT-Tiny with 16x16 patches. Specifically, we rely on three different teachers: a ViT-S/8 pretained with DINO, a ViT-B/16 and a ViT-L/16 both pretrained with iBOT. Our students are trained using the recipe described in section \ref{sec:approach} for 300 epochs. For both method, the student uses a projector head with a similar architecture than the teacher head, but changing its input dimension to $192$ to match the student encoder output dimension.

We compare our results with DeiT~\cite{touvron2021training} . Specifically, we use two supervised DeiT ViT-Tiny baselines, trained with and without knowledge distillation and reaching respectively  $72.2\%$ and $74.5\%$ ImageNet-1k top-1 accuracy.
From the results in Table~\ref{tab:vit_tiny}, we observe that 1) self-supervised distillation has better linear probe performances on transfer tasks compared to supervised pre-training. RoB is also competitive on ImageNet-1k 2) RoB outperforms DeiT with supervised distillation on most transfer tasks 3) self-supervised distillation achieves better linear probe performances with stronger teachers. An iBOT-L/16 teacher leading to strictly superior performance on all benchmarks when compared to an iBOT-B/16.

RoB achieves $71.8\%$ with a ViT-Tiny on ImageNet-1k which outperforms by a significant $2.3\%$ the best reported performance by self-supervised method on this architecture~\cite{anonymous2023effective}.


\paragraph{ConvNets.} We also reports result on ResNet-18 and ResNet-34 architecture in Table~\ref{tab:swav_mean} to demonstrate the versability of RoB. We compare RoB with two supervised pre-trained baselines: torchvision \cite{marcel2010torchvision} and ResNet Strikes Back (RNSB) \cite{DBLP:journals/corr/abs-2110-00476}. For RNSB, we use the $A1$ training recipe. We also use DIST \cite{https://doi.org/10.48550/arxiv.2205.10536} to train supervised distilled ResNet18 and ResNet34.

Similar to our previous results with ViT-Tiny, RoB models show better transfer performance than their supervised counterparts. RoB is also competitive with supervised distillation on all benchmarks.
We finally explore the distillation of DINO/S8 teacher with a ResNet student. Using a ViT as teacher does improve the average performance on our set of 6 tasks, especially on the ResNet34 where the gap is the most pronounced.
We include the detailed results for ResNets  and additional experimental details in Apppendix~\ref{appendix:training_details} and ~\ref{appendix:full_results}. Results for MobileNetV3 are reported in Appendix~\ref{experiment:mobilenet}.


\subsection{Semi-supervised setting}

In this section, we show that RoB can produce low-compute students with competitive low-shot classification performances.

We explore low-shot classification on Imagenet-1K~\cite{assran2022masked} using 1, 5 or 12/13 labeled images per class. 12/13 images per class corresponds to $1\%$ of  training data. 
We follow a linear evaluation protocol and fit a linear layer on top of a frozen encoder with  logistic regression~\cite{caron2021emerging}. Results reported for $1$ image per class and $5$ images per class settings are averaged over 5 and 3 ImageNet splits respectively. Additional details on evaluations are available in Appendix \ref{appendix:eval_details}.

We use RoB to distill a ViT-B/16 MSN teacher \cite{assran2022masked}, a method achieving state-of-art performances in low-shot classification, to a ViT-Tiny student. We also the use a DINO and a iBOT teachers. We train our students for 300 epochs. For MSN, we use an input masking ratio of  of $5\%$ which we found empirically that this ratio achieves better results than $10\%$ or $0\%$. 



Results are shown in Table \ref{tab:msn_low_shot}. We compare our results with the self-supervised distillation approach BINGO \cite{xu2022bag}, DINO and iBOT self-supervised models.
Our MSN student shows competitive low-shot performance and achieves better low-shot performances than BINGO and vanilla ViT-Tiny model trained with DINO. RoB even outperforms ViT-S/16 baselines trained with DINO or iBOT in the case of 1 image per class, an architecture with $4$ times more parameters and $4$ times more FLOPS.

\input{tables/msn_low_shot}

\subsection{Comparison with the state of the art}

We compare RoB to the self-supervised distillation methods CompRess \cite{compress}, SEED \cite{seed}, BINGO \cite{xu2022bag} SimReg \cite{DBLP:journals/corr/abs-2201-05131} and DisCo \cite{disco} on ResNet18, ResNet34 and MobileNetV3. We focus on the ImageNet-1k linear probing, the common benchmark reported across previous works. Resnet results are available in Table \ref{tab:sota}, MobileNetV3 in Appendix~\ref{experiment:mobilenet}.

We train ResNet18 and ResNet34 students with RoB, first using ResNet50 \cite{caron2020unsupervised} teacher and a  RegNet128Gf teacher \cite{https://doi.org/10.48550/arxiv.2103.01988} both pretrained with SwAV~\cite{caron2020unsupervised}. We also explore the use of a DINO/S8 pretrained teacher with RoB on ResNets. Refer to Appendix \ref{appendix:cross_arch_distillation} for more experimental details.

 Our method outperforms all previous methods on the ResNet18 and ResNet34 architectures, pushing the in-distribution linear probe results by $1.3\%$ for a ResNet18 and by $1.8\%$  on a ResNet34. 
\input{tables/sota}

\input{tables/swav_mean}

















\subsection{Beyond low-compute networks}

We demonstrate the RoB can also improve the performance of self-supervised ResNet50 using a Vision Transformers  teacher such as DINO \cite{caron2021emerging}.

We use RoB on a DINO/S8 teacher and distill it to a ResNet50 student. We train our student  for 100 epochs. Results are reported in Table \ref{tab:swav_mean}. The student ResNet50 outperforms the same model trained using DINO  ($+0.65\%$ across our 6 tasks) as well a SwAV ResNet50 ($+0.46\%$ across our 6 tasks) and supervised pretraining ($+6.18\%$ across our 6 tasks). The detailed results are available in Appendix \ref{appendix:full_results} while additional training details on cross-architecture distillation are available in Appendix \ref{appendix:cross_arch_distillation}.

\subsection{Visualizing the transferred knowledge}

\begin{figure*}[t]
    \centering
    \includegraphics[width=\linewidth]{figures/rcdm.pdf}
    \caption{\textbf{RCDM \cite{bordes2022high} visualisations} applied on iBOT (on the left) and MSN (on the right) both teacher (top row) and student (bottom row). We use RCDM to enable visualization of the learned SSL representation. Samples are generated by conditioning on a the left image representation and with various random seeds. Features that remain constant across samples depict information contained in the SSL representation, whereas features that vary depict information that is not contained. We qualitatively observe that the iBOT conserves more information about the context around the bird while MSN abstracts those details away. The RoB-iBOT and RoB-MSN students visually inherits these characteristics from their respective teachers.}
    \label{fig:rcdm}
\end{figure*}

\begin{figure}[h]
    \centering
    \includegraphics[width=\linewidth]{figures/RoB_Heads_3.pdf}
    \caption{\textbf{Student head designs.} We explore three different kinds of student head designs of joint-embedding prototype based methods. First is RoB default design: student and teacher have different heads. Second is replacing the student head with a MLP. Third is to use the teacher prototypes in the student head.}
    \label{fig:head_designs}
\end{figure}

We now use the RCDM \cite{bordes2022high} conditional diffusion model to visualize the prototypes of the prototypes learned with our RoB students. As highlighted by \cite{bordes2022high}, conditional diffusion models are useful to get a better understanding of what is learned by neural networks. We qualitatively compare the results obtained in the student against the results of the teacher to assess how much knowledge has been transferred.

We compare the representations of both iBOT and MSN teacher against the representations of both RoB-iBOT and RoB-MSN student in Figure \ref{fig:rcdm}. Qualities that remain constant across samples depict information contained in the SSL representation, whereas qualities that vary depict information that is not contained in the representation.
MSN retains less details about background and poses than iBOT, and this difference is kept when we observe the RoB-MSN and RoB-iBOT students. We hypothesize that these difference are connected to the superior performance of RoB-MSN on low-shot in-distribution, where removing background details lessens the risk of over-fitting, while also explaining why RoB-iBOT representations lead to stronger transfer performance where more labels are available.

\subsection{Ablations}


\paragraph{Student head designs.}
Because RoB replaces one branch of the distilled joint-embedding approach with a pre-trained teacher, the design of the projection head of the teacher and the student are identical (Teacher head and RoB head in Figure \ref{fig:head_designs}), only differing on their input dimension. In the case of joint-embedding approaches based on soft clustering \cite{caron2020unsupervised, caron2021emerging,zhou2021ibotyes, assran2022masked}, the projection head is composed of a projection to a low dimension space ($128$ for SwAV), normalized, followed by a dot product with a set of learned prototypes used to compute a distribution over the prototypes. We investigate whether the method can be simplified and the performance improved by using a different projection head for the student.

Our first experiment is to replace the SSL projection head of the student with a simpler multi-layer perceptron head (MLP head in Figure \ref{fig:head_designs}). The output dimension of this prediction head is set to be the number of prototypes of the teacher, $3000$ in the case of SwAV, to directly predict the distribution given by the teacher. We try MLPs of depth $1$, $2$ or $3$ and report the best result (obtained with depth $2$). 


Our second experiment consists in keeping the student projection head intact but using the prototypes of the teacher in the student head instead of the student learning its own set of prototypes. We dubbed this approach Partial head (see Figure \ref{fig:head_designs}) as the projection head is partially learned.





The results are available in Table \ref{tab:ablation_head}. Distilling with a MLP head for the student does improve performance up to $0.3\%$ improvement on ImageNet, but suffers from a up to $4.9\%$ drop in terms of transfer performance on tasks such as CIFAR100 or iNaturalist18, while distilling using the partial head design for the student significantly degrades performance on both on ImageNet and all transfer tasks.

\paragraph{Importance of multi-crop.}
As RoB distills the joint-embedding approach by mimicking it, only replacing one branch with a pretrained teacher and adapting the loss, RoB will make use of multi-crops \cite{caron2020unsupervised} if the method being distilled does use multi-crops. We investigate here if this is necessary, or if the student can be distilled using large crops alone.

We use RoB to distill a DINO/S8 \cite{caron2021emerging} teacher and train for 300 epochs once with and once without multi-crops and compare the results. Table \ref{tab:ablation_views} shows that without multi-crop, performance is decreased significantly on ImageNet with a $1\%$ decrease in top-1 accuracy, as well as on transfer tasks with up to $4.4\%$ accuracy drop on iNaturalist18, the transfer task the most sensitive to multi-crop. This shows the critical importance of multi-crop in the context of self-supervised distillation. Additional detailed results are provided in \ref{appendix:full_results}.

\paragraph{Identical-view vs cross-view predictions.}
As described in Section \ref{sec:approach}, RoB distills joint-embedding approaches by using identical-view predictions, where the student learns to match the teacher representation of the same view, instead of using cross-view predictions, where the student would learn to predict the teacher representation of a different view of the same original image.

Cross-view predictions are used in joint embedding methods to learn invariance to augmentations. However, as the desired invariances are already learned by the teacher, our hypothesis is that identical view predictions will suffice to learn the useful invariances encoded by the teacher while making training more stable, leading to equivalent or slightly better performance on all tasks.

To verify our hypothesis, we distill both a DINO S/8 \cite{caron2021emerging} teacher and a iBOT L/16 \cite{zhou2021ibotyes} teacher with either cross-view predictions or identical-view predictions and compare the results obtained. As shown in Table \ref{tab:ablation_views}, distilling with identical-view prediction yields overall equal or better linear probe results, with differences being more pronounced on iBOT than on DINO. RoB therefore adopts identical-view predictions as part of the method.

\input{tables/ablation_views}
\input{tables/ablation_head}

\section{Limitations and Conclusion}

We explored a simple and generic approach for self-supervised distillation that consists in "Replacing one Branch" (RoB) in joint-embedding SSL methods by a pretrained teacher, while adapting the loss of the method to remove mechanisms preventing representation collapse.

We instantiated RoB on 4 different joint embedding approaches and demonstrated that RoB produces state of the art SSL students with properties that follow those of their respective teachers. RoB applied on SwAV, DINO or iBOT leads to transfer performance that competes with supervised distillation, fixing the performance gap of SSL on low-compute architectures, while RoB-MSN produces students with strong semi-supervised capabilities.


\pagebreak

\section*{Acknowledgements}

We would like to thank Shagun Sodhani for his helpful feedback all along the project and his help in proof reading the paper. We also would like to thank Olivier Delalleau and Mike Rabbat for their helpful feedback and discussions around the subject of SSL and knowledge distillation.

{\small
\bibliographystyle{cvprtemplate/ieee_fullname}






















