\section{Introduction}

Currently, deep learning is criticized because it is
data hungry, has limited capacity for transfer,
insufficiently integrates prior knowledge, and
presumes a largely stable world \citep{Marcus:18}.
In particular, these problems appear after a domain shift,
that is, a change of the input-target distribution.
A domain shift forces deep learning models to adapt.
The goal is to exploit models
that were trained on the typically rich original data
for solving tasks from the new domain with much less data.
Examples for domain shifts are
new users or customers, new products and product lines,
new diseases (e.g.\ adapting from SARS to COVID19),
new images from another field (e.g.\ from cats to dogs
or from cats to bicycles),
new social behaviors after societal change (e.g.\ introduction of cell phones, pandemic),
self-driving cars in new cities or countries (e.g.\ from European countries 
to Arabic countries), and robot manipulation of new objects.


Domain shifts are often tackled by 
meta-learning \citep{schmidhuber87, bengio90, hochreiter01}, 
since it exploits already acquired knowledge to adapt to new data.
One prominent application of meta-learning 
dealing with domain shifts is few-shot learning,
since, typically, from
the new domain much less data is available than from the original domain.
Meta-learning methods perform well on small domain shifts like
new target classes with similar inputs.
However, larger domain shifts are still challenging for current approaches.
Large domain shifts lead to inputs, which are considerably different 
from the original inputs and possess different high-level concepts.
Nonetheless, low-level concepts are often still shared between the inputs of
the original domain and the inputs of the new domain.
For images, such shared low-level concepts
can be edges, textures, small shapes, etc.
One way of obtaining low level concepts
is to train a new deep learning model from scratch,
where the new data is merged with the original data.
However, although models of the original domain are 
often available, the original data, 
which the models were trained on, often are not. 
This might have several reasons,
e.g.\ the data owner does no longer grant access to the data, 
General Data Protection Regulation (GDPR)
does no longer allow access to the data,
IP restrictions prevent access to the data,
sensitive data items must not be touched anymore (e.g.\ phase III drug candidates),
or data is difficult to extract again.
We therefore suggest to effectively exploit original data models directly
by accessing not only high level but also low level abstractions.
In this context, we propose a cross-domain few-shot learning method
extracting information from different levels of 
abstraction in a deep neural network.

\textbf{Representation fusion.} 
Deep Learning constructs neural network models 
that represent the data at multiple levels of abstraction \citep{lecun2015deep}.
We introduce \textit{representation fusion},
which is the concept of unifying and merging information from different levels of abstraction.
Representation fusion uses 
a fast and adaptive system for detecting relevant information
at different abstraction levels of a deep neural network, 
which we will show allows solving versatile and complex cross-domain tasks.

\textbf{CHEF.} We propose cross-domain ensemble
few-shot learning (CHEF) that achieves representation fusion
by an ensemble of Hebbian learners, which are built upon a trained network.
CHEF naturally addresses the problem of domain shifts
which occur in a wide range of real-world applications.
Furthermore, since CHEF only builds on 
representation fusion,
it can adapt to new characteristics
of tasks like 
unbalanced data sets, 
classes with few examples,
change of the measurement method, 
new measurements in unseen ranges,
new kind of labeling errors,
and more. 
The usage of simple Hebbian learners 
allows the application of CHEF without needing to backpropagate information through the backbone network.

The main contributions of this paper are:
\begin{itemize}
    \item We introduce representation fusion as the concept of unifying and merging information from different layers of abstraction. An extensive ablation study shows that representation fusion is a decisive factor to boost cross-domain few-shot learning.
    \item We introduce CHEF\footnote{Our implementation is available at \href{https://github.com/ml-jku/chef}{github.com/ml-jku/chef}.} as our new cross-domain few-shot learning method that builds on representation fusion.
    We show that using different layers of abstraction allows one to successfully tackle various few-shot learning tasks across a wide range of different domains. CHEF does not need to backpropagate information through the backbone network.
    \item We apply CHEF to various cross-domain few-shot tasks and obtain several state-of-the-art results. We further apply CHEF to cross-domain real-world applications from drug discovery, where we outperform all competitors. 
\end{itemize}


\textbf{Related work. }
Representation fusion builds on learning a meaningful
representation \citep{bengio2013representation, girshick2014rich} at multiple levels of abstraction \citep{lecun2015deep, schmidhuber2015deep}.
The concept of using representations from different
layers of abstraction has been used in CNN architectures~\citep{lecun1998gradient}
such as \citet{huang2017densely, rumetshofer2018human, hofmarcher2019accurate}, 
in CNNs for semantic segmentation
in the form of multi-scale context pooling \citep{yu2015multi,chen2018encoder}, 
and in the form of context capturing
and symmetric upsampling \citep{ronneberger2015u}.
Learning representations from different domains has been explored 
by \citet{federici2020learning, tschannen2020on} under the viewpoint 
of mutual information optimization. 
Work on domain shifts discusses the problem that new inputs are considerably 
different from the original inputs \citep{kouw2019, kouw2018, webb2018, gama2014, widmer1996}. 
Domain adaptation \citep{pan2009, bendavid2010} overcomes this problem 
by e.g.\ reweighting the original samples \citep{huang2007}, 
learning features that are invariant to a 
domain shift \citep{ganin2016jmlr, xu2019cvpr}
or learning a classifier in the new domain. 
Domain adaptation where only few data
is available in the new domain \citep{bendavid2010, lu20} 
is called cross-domain few-shot learning \citep{guo19, lu20, tseng2020cross},
which is an instance of the general few-shot learning setting \citep{fei2006one}.
Few-shot learning can be roughly divided into three approaches \citep{lu20, Hospedales:20}:
(i) augmentation, (ii) metric learning, and (iii) meta-learning.
For (i), where the idea is to learn an augmentation to produce more than the few samples available, supervised \citep{dixit2017aga,kwitt2016one} and 
unsupervised \citep{hariharan2017low, pahde2019low, gao2018low} methods are considered. For (ii), approaches aim to learn a pairwise similarity metric under which similar samples obtain high similarity scores \citep{koch2015siamese, ye2018deep, hertz2006learning}.
For (iii), methods comprise \textit{embedding and nearest-neighbor} approaches
\citep{snell2017prototypical, sung18, vinyals16}, \textit{finetuning} approaches
\citep{finn17, rajeswaran19, ravi17, andrychowicz2016learning}, and \textit{parametrized} approaches \citep{gidaris18,ye20,lee19,yoon19,mishra18,hou19,rusu18}.
Few-shot classification under domain shifts for metric-based methods has been discussed in \citet{tseng2020cross}.
Ensemble methods for few-shot learning have been applied in \citet{dvornik2019diversity},
where an ensemble of distance-based classifiers is designed
from different networks. In contrast, our method builds an ensemble of different layers from the same network. Hebbian learning as part of a few-shot learning method has been implemented in \citet{munkhdalai2018metalearning},
where fast weights that are used for binding labels to representations
are generated by a Hebbian learning rule.


\section{Cross-domain few-shot learning} \label{sec:cross-domain}


\textbf{Domain shifts.}
We assume to have data $(\Bx,\By)$, where $\Bx \in \BX$ is the input data
and $\By \in \BY$ is the target data.
A domain is a distribution $p$ over $\BX\times \BY$
assigning each pair $(\Bx,\By)$ a probability $p(\Bx,\By)$.
A domain shift is a change from $p(\Bx,\By)$ to $\tilde{p}(\Bx,\By)$.
We measure the magnitude of the domain shift by
a distance $d(p,\tilde{p})$ between the distributions $p$ and $\tilde{p}$.
We consider four types of domain shifts \citep{kouw2019, kouw2018, webb2018, gama2014, widmer1996}:
\begin{itemize}
    \item \textbf{Prior shift (small domain shift)}: $p(\By)$ is changed to $\tilde{p}(\By)$, while $p(\Bx \mid \By)$ stays the same. For example, when new classes are considered (typical case in few-shot learning):
        $p(\Bx,\By) = p(\By)p(\Bx \mid \By)$ and $\tilde{p}(\Bx,\By) = \tilde{p}(\By)p(\Bx \mid \By)$.
    \item \textbf{Covariate shift (large domain shift)}: $p(\Bx)$ is changed to $\tilde{p}(\Bx)$, while $p(\By \mid \Bx)$ stays the same. For example, when new inputs are considered, which occurs when going from color to grayscale images, using a new measurement device, or looking at traffic data from different continents: 
        $p(\Bx,\By) = p(\Bx)p(\By \mid \Bx)$ and $\tilde{p}(\Bx,\By) = \tilde{p}(\Bx)p(\By \mid \Bx)$.
    \item \textbf{Concept shift}:  $p(\By \mid \Bx)$ is changed to $\tilde{p}(\By \mid \Bx)$, while $p(\Bx)$ stays the same. For example, when including new aspects changes the decision boundaries:
        $p(\Bx,\By) = p(\Bx)p(\By \mid \Bx)$ and $\tilde{p}(\Bx,\By) = p(\Bx)\tilde{p}(\By \mid \Bx)$.
    \item \textbf{General domain shift}: domain shift between $p(\Bx,\By)$ and $\tilde{p}(\Bx,\By)$. For example, going from Imagenet data to grayscale X-ray images (typical case in cross-domain datasets).
\end{itemize}

\textbf{Domain shift for images.}
We consider the special case that the input $\Bx$ is an image.
In general, domain shifts can be measured on the raw image distributions e.g.\ 
by using the $\mathcal{H}$-divergence \citep{bendavid2010}. 
However, distances between raw image distributions were shown to be 
less meaningful in computer vision tasks than abstract representations 
of deep neural networks \citep{heusel2017, salismans2016}. We approximate the 
distance between the joint distributions $d(p(\Bx,\By),\tilde{p}(\Bx,\By))$ 
by the distance between the marginals $d(p(\Bx),\tilde{p}(\Bx))$, 
which is exact in the case of the covariate shift for certain choices of $d(\cdot,\cdot)$, 
like e.g.\ the Jensen-Shannon divergence.
To measure the distance between the marginals $d(p(\Bx),\tilde{p}(\Bx))$ we use the 
Fr\'{e}chet Inception Distance (FID; \citealp{heusel2017}), i.e.\ the Wasserstein-2
distance of the features of the respective images activating an Inception v3 network 
\citep{szegedy2016rethinking} under a Gaussian assumption. The FID has proven 
reliable for measuring performance of Generative Adversarial Networks~\citep{goodfellow2014generative}.


\textbf{Cross-domain few-shot learning.}
Large domain shifts lead to inputs, which are considerably 
different from the original inputs.
As a result, the model trained on the original domain
will not work anymore on the new domain.
To overcome this problem, domain adaptation 
techniques are applied \citep{pan2009, bendavid2010}.
Domain adaption can be achieved
in several ways, e.g.\ by reweighting the original
samples \citep{huang2007}.
Another possibility is to learn a classifier in the new domain.
Domain adaptation where in the new domain 
only few data is available \citep{bendavid2010} which can be used for learning
is called cross-domain few-shot learning \citep{guo19, lu20, tseng2020cross}.
In an $N$-shot $K$-way few-shot learning setting, 
the training set (in meta learning also called one episode)
consists of $N$ samples for each of the $K$ classes.





\section{Cross-domain Hebbian Ensemble Few-shot learning (CHEF)} \label{sec:chef}

We propose a new cross-domain few-shot learning method, CHEF, 
that consists of an ensemble of Hebbian learners built on representation fusion (see Fig.~\ref{fig:sketch_system1system}). 
A schematic view of the CHEF algorithm is outlined in Alg.~\ref{alg:chef}.

We conducted an extensive ablation study on the cross-domain 
few-shot learning challenge datasets~\citep{guo19} to test (i) the influence of 
different levels of abstraction 
(Tab.~\ref{tab:ablation2} in the appendix)
and (ii) the influence of different learning algorithms 
(Tab.~\ref{tab:cross_domain_res18} in the appendix).
For (ii), we compared support vector machines (SVMs) with different kernels, 
random forest (RF) with different numbers of trees, 
$k$-nearest neighbors with different numbers of neighbors, 
and a Hebbian learning rule 
(Tab.~\ref{tab:cross_domain_res18} in the appendix).
In principle, any learning algorithm can be used for representation fusion.
We chose Hebbian learning since it is a simple and fast learning rule
that outperformed or performed on par with the other methods 
(Tab.~\ref{tab:cross_domain_res18} in the appendix).


\begin{figure*}[t]
\centering
\includegraphics[width=.77\linewidth]{figures/sketch_v4.pdf}
\caption[]{Working principle of CHEF. 
An ensemble of Hebbian learners is applied to the upper layers 
of a trained neural network. Distilling information from 
different layers of abstraction is called representation fusion.
Each Hebbian learner is iteratively
optimized and the results are combined. 
CHEF does not require backpropagation of error signals through the entire
backbone network, only the parameters of the individual learners need adjustment.
Hebbian learners can easily be exchanged with other supervised techniques.}
\label{fig:sketch_system1system}
\end{figure*}


\textbf{Representation fusion. }
CHEF builds its ensemble of learners
using representation fusion.
Deep learning models \citep{lecun2015deep} provide hierarchical 
representations that allow to fuse information from different
layers of abstraction.
The closer a layer is to the network output the more specific are its features. 
Conversely, the closer a layer is to the input of the network, 
the more general are the features. 
In cross-domain few-shot learning, 
it is not a priori clear how specific or general the features should be 
because this depends on how close the target domain is to the training domain. Therefore, we design our few-shot learning algorithm 
such that it can flexibly choose the specificity of the features depending on 
the current episode. 
We achieve this by representation fusion,
where the learners are  applied to 
several layers at different levels of the backbone network in parallel.
This yields a separate prediction for each level of abstraction. 
The final classification result is then obtained from the sum of logits 
arising from the respective learners.
Ablation studies revealed that combining the last six layers via representation fusion achieves the best results.

\textbf{Computation time. }
In contrast to many other methods, CHEF does not require
backpropagation of error signals through the entire backbone network.
Only the parameters of the learners that are obtained for the 
uppermost layers need adjustment. This makes CHEF extremely fast and versatile.
A detailed study of the computation time can be found in the appendix. 


\iffalse
\textbf{Hebbian Learning. } \label{hebbian_learning}
Consider a feature vector $\Bz_i \in \dR^d$, which is obtained by 
activating the backbone network with sample $\Bx_i$ up to a certain 
hidden layer. 
We construct a linear regression model with $K$ output units. 
Using this representation, we can also tackle classification problems by 
regressing label vectors which indicate class membership. Given a label 
vector $\Bv_i \in \dR^K$, we are concerned with choosing a weight matrix 
$\BW \in \dR^{K \times d}$ such that the quadratic error
\begin{equation}
    \cL_{\text{Hebb}} = (\Bv_i - \BW \Bz_i)^\top (\Bv_i - \BW \Bz_i)
   
   
\end{equation}
becomes minimal. This leads to the solution $\BW = \BV^\top \BZ (\BZ^\top \BZ)^{-1}$
where $\BZ \in \dR^{NK \times d}$ is the matrix of features
and $\BV \in \dR^{NK \times K}$ is the label matrix. 
The inverse $(\BZ^\top \BZ)^{-1} \in \dR^{d \times d}$ is intractable for many 
practical settings as the number of features $d$ can get very large.
Moreover, the number of samples $NK$ is usually very small in the few-shot 
setting and in case $NK < d$, the matrix $\BZ^\top \BZ$ cannot even have full rank 
and its inverse does not exist. 
In this case, we approximate the desired solution by letting $\BZ^\top \BZ = \BI$ 
which simplifies it to a sum of outer products of label and feature vectors.
The quadratic error 
function $\cL_{\text{Hebb}}$ is usually used for regression while 
the few-shot setting usually consists of classification problems, 
where error functions like the cross-entropy loss are used. 
Therefore, if we choose $\Bv_i \propto \nabla_{\hat{\By}} \cL(\By_i, \hat{\By}) |_{\hat{\By}=\BW\Bz_i}$, 
we cast a gradient descent step 
as a simple Hebbian learning rule. 
Note that $\cL$ is the loss function of 
the few-shot learning task at hand
and is not to be confused with 
$\cL_{\text{Hebb}}$, which we use only to derive the Hebbian learning rule.
Since $\BV^\top \BZ$ is only an approximation,
we apply it as an iterative learning rule
\begin{equation}
    \BW \gets \BW - \alpha \BV^\top \BZ, \label{eqn:hebb}
\end{equation}
where $\alpha$ is the Hebbian learning rate. We design the number of steps for which 
to run \eqref{eqn:hebb} as a hyperparameter of our method. 
\fi

\textbf{Obtaining one Hebbian Learner. } \label{hebbian_learning}
We consider an $N$-shot $K$-way few-shot learning setting. 
Let $\Bz_i \in \dR^D$ be a feature 
vector obtained from activating a pre-trained backbone network with a sample 
$\Bx_i$ up to a certain layer, where $D$ is the number of 
units in that layer. 
We combine the $NK$ feature 
vectors into a matrix $\BZ \in \dR^{NK \times D}$ and 
initialize a weight matrix 
$\BW \in \dR^{K \times D}$. 
The Hebbian learning rule~\citep{hebb1949} reads:
\begin{equation}
    \Delta w_{ij} = \alpha \cdot a_i \cdot a_j \ ,
\end{equation}
where $\Delta w_{ij}$ is the weight change from neuron $i$ to neuron $j$,
$a_i$ and $a_j$ are the activations of neuron $i$ and neuron $j$,
respectively, and $\alpha$ is the Hebbian learning rate.
In accordance with \citet{fregnac2002hebbian}, 
we use the update rule
\begin{equation}
\BW \gets \BW - \alpha \BV^\top \BZ \label{eqn:hebb}
\end{equation}
for a given number of steps, where
$\BV \in \dR^{NK \times K}$
is the matrix of postsynaptic responses $\Bv_i$. 
We design the Hebbian learning rate and the number of steps for which 
to run the update rule \ref{eqn:hebb} as hyperparameters of our method. 
Given a loss function $\cL(\cdot, \cdot)$ and few-shot labels 
$\By_i$, we choose the postsynaptic response
\begin{equation}
\Bv_i = \nabla_{\hat{\By}} \cL(\By_i, \hat{\By})|_{\hat{\By} = \BW \Bz_i},
\end{equation}
casting the update rule \ref{eqn:hebb} effectively as a gradient descent step. 
However, since we use only a few steps of rather strong updates we prefer to 
view this as a Hebbian learning rule. 
We initialize the weight matrix $\BW$ with zeros. 
In principle, any other initialization is possible but due to strong updates 
the initialization scheme is of minor importance. 

\begin{algorithm}
\caption{CHEF algorithm. The data matrix $\BX$ consists of 
input vectors $\Bx_i$ and the label matrix $\BY$ consists of the corresponding 
label vectors $\By_i$. 
The function BB activates the backbone network up
to a certain layer specified by an index $l$. 
$L$ is the set of indices specifying 
the layers used in the ensemble,
$\cL$ is the loss function of the few-shot learning task at hand.
The function \textsc{HebbRule} executes $M$ steps of the Hebbian 
learning rule and yields a weight matrix $\BW$ 
that maps the feature vectors in $\BZ$ to vectors of length $K$, 
which are then used for $K$-fold classification. 
$\alpha$ is the Hebbian learning rate.}
\begin{algorithmic}
\REQUIRE $\alpha, M, \operatorname{Softmax}(\cdot), \cL(\cdot, \cdot), \operatorname{BB}(\cdot,\cdot)$
\FUNCTION{$\operatorname{HebbRule}(\BX, \BY, l)$
  \STATE $\BW \gets \BZe, \BZ \gets \operatorname{BB}(\BX, l)$
  \FOR{$m \in \{1, \dots, M\}$} 
    \STATE $\BV \gets \nabla_{(\BZ \BW^\top)} \cL(\BY, \operatorname{Softmax}(\BZ \BW^\top))$
    \STATE $\BW \gets \BW - \alpha \BV^\top \BZ$
  \ENDFOR
  \STATE {\bfseries return} $\BW$
\ENDFUNCTION
\FUNCTION{$\operatorname{Ensemble}(\BX, \BY, L)$
  \STATE $\BU = \sum_{l \in L} \operatorname{BB}(\BX, l)\,\operatorname{HebbRule}(\BX, \BY, l)^\top$
  \STATE {\bfseries return} $\operatorname{Softmax}(\BU)$
\ENDFUNCTION
\end{algorithmic}
\label{alg:chef}
\end{algorithm}


\section{Experiments}\label{sec:experiments}

We apply CHEF to four cross-domain few-shot challenges,
where we obtain state-of-the-art results in all categories.
The four cross-domain few-shot challenges are 
characterized by domain shifts of different magnitude,
which we measure using the Fr\'{e}chet-Inception-Distance (FID).
We conduct ablation studies showing the influence of the 
different layer representations on the results.
Further, we test CHEF on two standardized image-based few-shot classification 
benchmark datasets established in the field,
which are characterized by a prior domain shift: \textit{mini}Imagenet \citep{vinyals16} 
and \textit{tiered}Imagenet \citep{ren18}.
Finally, we illustrate the impact of our CHEF approach on two real-world 
applications in the field of drug discovery,
which are characterized first by a small domain shift and 
second by a large domain shift.


\subsection{Cross-domain few-shot learning} \label{sec:cross_domain}

\textbf{Dataset and evaluation. }
The cross-domain few-shot learning challenge \citep{guo19} uses 
\textit{mini}Imagenet as training 
domain and then evaluates the trained models on four different test domains
with increasing distance to the training domain: 
1)~CropDisease \citep{mohanty2016} consisting of 
plant disease images, 2)~EuroSAT \citep{helber2019eurosat}, 
a collection of satellite images, 
3)~ISIC2018 \citep{tschandl2018ham10000, codella2019skin} containing
dermoscopic images of skin lesions, and 
4)~ChestX \citep{wang2017chestx} containing a set of X-ray images. 
For evaluation, we measure the accuracy
drawing 800 tasks (five test instances per class) 
from the cross-domain test set.
Following prior
work, we focus on 5-way/5-shot, 5-way/20-shot, and 5-way/50-shot tasks.
We report the average accuracy and a 95\,\% confidence 
interval across all test images and tasks.

Recent work \citep{ericsson2020selfsupervised,cai2020sbmtl,wang2020comparison} 
showed impressive results in the meta-transfer learning setting,
where pre-trained models are optimized over specific few-shot learning tasks
via transfer learning. In contrast, our experiments focus on the case where only one 
episode of the 5-way/\{5,20,50\}-shot data of the new domain is available for 
adjusting the model. 
That is, we relinquish episodic training completely, which is in 
accordance with the findings of \citet{laenen2020episodes}. 

\textbf{Measuring the domain shifts via FID. } 
In \citet{guo19}, the four datasets of the new domain are characterized 
by their distance to the original domain using three criteria:
whether images contain perspective distortion, 
the semantic content of images, 
and color depth.
In Table~\ref{tab:fid_result}, we provide measurements of
the domain shift of these four datasets 
with respect to the original \textit{mini}Imagenet dataset
using the FID. 
The FID measurements confirm the characterization in \citet{guo19},
except that the EuroSAT dataset is closer to the original domain
than the CropDisease dataset. The difference in both FID measurements
is mostly driven by the mean terms.
This can be explained by the fact that the FID does not measure perspective distortion 
and satellite images might have a higher variety of shapes and colors than plant images.
Recent associative techniques, e.g.~\citep{ramsauer2020hopfield} are not tested so far,
but are a potentially promising filed of future research.

\begin{table*}[t]
    \centering
    \begin{tabular}{llc}
         \toprule
         Dataset & Conceptual difference to original domain (\textit{mini}Imagenet) & FID \\ 
         \midrule
         CropDisease & None & 257.58  \\
         EuroSAT & No perspective distortion & 151.64  \\
         ISIC2018 & No perspective distortion, unnatural content & 294.05  \\
         ChestX & No perspective distortion, unnatural content, different color depth & 312.52  \\
         \bottomrule
    \end{tabular}
    \caption{Conceptual difference and domain shift between \textit{mini}Imagenet and the four cross-domain datasets CropDisease, EuroSAT, ISIC2018, and ChestX. The domain shift is measured using the FID.}
    \label{tab:fid_result}
   
\end{table*}


\textbf{CHEF implementation. }
We perform pre-training on the \textit{mini}Imagenet dataset
similar but not identical to that in \citet{ye20}. 
We utilize a softmax output layer with as many units as classes 
are contained in the meta-training and the meta-validation sets combined. 
We make a validation split on the combination of these two sets for 
supervised learning, i.e.\ instead of separating whole classes into 
the validation set (vertical split) we move a randomly selected fraction of samples 
of each class into the validation set (horizontal split) as it is standard in 
supervised learning. We evaluate CHEF using the same ResNet-10 backbone architecture 
as in \citet{guo19}.
For better representation fusion, we place two fully 
connected layers after the last convolutional layer.
We perform model selection during training using 
the cross-entropy loss function on the horizontal data split,
and perform hyperparameter selection for CHEF on the vertical data split.

\textbf{Results and ablation study. }CHEF achieves state-of-the-art performance in all 12 categories. 
Results are provided in Table~\ref{tab:meta}.
An ablation of the influence of representation fusion
on 5-way 5-shot and 50-shot results
is shown in Fig.~\ref{fig:res18} where we use a pre-trained PyTorch \citep{paszke2019:pytorch} ResNet-18, which consists of 8 convolutional blocks. 5-way 20-shot results can be found in Fig.~\ref{fig:res18_20shot} in the appendix.
Results are obtained by applying our Hebbian learning 
rule to the logits of the output layer, to the pre-activations of the 
blocks 4 through 8 individually, and to an ensemble of them.
The results are considerably better than the above reported ResNet-10 results, which presumably arises from the fact that the power of representation fusion is larger since the ResNet-18 is pretrained on the whole Imagenet dataset,
and, thus, has stronger representations in the individual layers.
This illustrates the power of CHEF considering better feature abstractions. Another interesting insight is that for the ChestX dataset, the dataset with the largest domain shift, the lower level features gain importance.
In contrast, in CropDiseases and EuroSAT, the datasets with the smallest domain shifts, the differences between block 4 and block 8 are large. 
In general, the farther the domain is away from the original domain the more 
important are features from lower layers, i.e.\ features that are less specific 
to the original domain. Since CHEF combines features of different specificity to the training domain, it is particularly powerful in cross-domain settings. 


\begin{figure}[!htp]
 \centering
 \includegraphics[width=0.88\linewidth]{figures/res18_5-shot_arxiv.pdf}
 \includegraphics[width=0.88\linewidth]{figures/res18_50-shot_arxiv.pdf}
 \caption{Ablation study of 5-shot and 50-shot top-1 accuracies 
     (along with $95\%$ confidence intervals) of different 
     residual blocks and the output layer of 
     a PyTorch Imagenet-pretrained ResNet-18 and the ensemble result (orange, ``ens'') 
     on the four different datasets of the cross-domain few-shot learning 
     benchmark. For comparison, also the ResNet-10 ensemble results (green) 
     are included. The ResNet-18 is pretrained on the whole Imagenet dataset.}
 \label{fig:res18}
\end{figure}

\begin{table*}[!htb]
\centering
\scalebox{0.82}{
\begin{tabular}{l c c c c c c}
    \hline
   
&\multicolumn{3}{c}{\textbf{CropDiseases 5-way}}& \multicolumn{3}{c}{\textbf{EuroSAT 5-way}}  \\
   
    \textbf{Method} & \textbf{5-shot} &\textbf{20-shot}&\textbf{50-shot}&\textbf{5-shot} &\textbf{20-shot} &\textbf{50-shot} \\
    \hlin
MatchingNet$^{\dagger}$   &66.39 $\pm$ 0.78  & 76.38 $\pm$ 0.67& 58.53 $\pm$ 0.73& 64.45 $\pm$ 0.63&77.10 $\pm$ 0.57&54.44 $\pm$ 0.67 \\

MatchingNet+FWT$^{\dagger}$    & 62.74 $\pm$ 0.90 & 74.90 $\pm$ 0.71 & 75.68 $\pm$ 0.78& 56.04 $\pm$ 0.65&  63.38 $\pm$ 0.69 & 62.75 $\pm$ 0.76 \\

MAML$^{\dagger}$   & 78.05 $\pm$ 0.68 & \underline{89.75} $\pm$ 0.42 &-& 71.70 $\pm$ 0.72&81.95 $\pm$ 0.55& -\\
  
ProtoNet$^{\dagger}$    & \underline{79.72} $\pm$ 0.67 &88.15 $\pm$ 0.51 &90.81 $\pm$ 0.43 &\underline{73.29} $\pm$ 0.71 & \underline{82.27} $\pm$ 0.57 & 80.48 $\pm$ 0.57 \\

ProtoNet+FWT$^{\dagger}$    & 72.72 $\pm$ 0.70 &85.82 $\pm$ 0.51&87.17 $\pm$ 0.50& 67.34 $\pm$ 0.76& 75.74 $\pm$ 0.70  & 78.64 $\pm$ 0.57\\

RelationNet$^{\dagger}$    &  68.99 $\pm$ 0.75 &80.45 $\pm$ 0.64&85.08 $\pm$ 0.53& 61.31 $\pm$ 0.72  & 74.43 $\pm$ 0.66& 74.91 $\pm$ 0.58 \\

RelationNet+FWT$^{\dagger}$    & 64.91 $\pm$ 0.79  & 78.43 $\pm$ 0.59& 81.14 $\pm$ 0.56& 61.16 $\pm$ 0.70 & 69.40 $\pm$ 0.64 & 73.84 $\pm$ 0.60 \\

MetaOpt$^{\dagger}$   & 68.41 $\pm$ 0.73&82.89 $\pm$ 0.54 & \underline{91.76} $\pm$ 0.38& 64.44 $\pm$ 0.73 &79.19 $\pm$ 0.62 & \underline{83.62} $\pm$ 0.58  \\

CHEF (Ours)  &\textbf{86.87} $\pm$ 0.27 &\textbf{94.78} $\pm$ 0.12&\textbf{96.77} $\pm$ 0.08 &\textbf{74.15} $\pm$ 0.27 & \textbf{83.31} $\pm$ 0.14 & \textbf{86.55} $\pm$ 0.15\\



    \hline
    \hline
    \hline
 &\multicolumn{3}{c}{\textbf{ISIC 5-way}}& \multicolumn{3}{c}{\textbf{ChestX 5-way}}  \\
   
    \textbf{Method} & \textbf{5-shot} &\textbf{20-shot}&\textbf{50-shot}&\textbf{5-shot} &\textbf{20-shot} &\textbf{50-shot} \\
    \hlin
MatchingNet$^{\dagger}$    & 36.74 $\pm$ 0.53& 45.72 $\pm$ 0.53&  54.58 $\pm$ 0.65& 22.40 $\pm$ 0.7 & 23.61 $\pm$ 0.86 &22.12 $\pm$ 0.88 \\


MatchingNet+FWT$^{\dagger}$    &  30.40 $\pm$ 0.48 & 32.01 $\pm$ 0.48 & 33.17 $\pm$ 0.43 & 21.26 $\pm$ 0.31& 23.23 $\pm$ 0.37  & 23.01 $\pm$ 0.34 \\

MAML$^{\dagger}$   &\underline{40.13} $\pm$ 0.58 &\underline{52.36} $\pm$ 0.57 & - & 23.48 $\pm$ 0.96 & 27.53 $\pm$ 0.43  & - \\
  
ProtoNet$^{\dagger}$    & 39.57 $\pm$ 0.57&49.50 $\pm$ 0.55&51.99 $\pm$ 0.52& \underline{24.05} $\pm$ 1.01& \underline{28.21} $\pm$ 1.15 &29.32 $\pm$ 1.12 \\

ProtoNet+FWT$^{\dagger}$    &  38.87 $\pm$ 0.52 & 43.78 $\pm$ 0.47 & 49.84 $\pm$ 0.51& 23.77 $\pm$ 0.42 & 26.87 $\pm$ 0.43 & \underline{30.12} $\pm$ 0.46 \\

RelationNet$^{\dagger}$    & 39.41 $\pm$ 0.58 & 41.77 $\pm$ 0.49 &49.32 $\pm$ 0.51 & 22.96 $\pm$ 0.88&26.63 $\pm$ 0.92&28.45 $\pm$ 1.20 \\

RelationNet+FWT$^{\dagger}$    &35.54 $\pm$ 0.55 & 43.31 $\pm$ 0.51& 46.38 $\pm$ 0.53&22.74 $\pm$ 0.40 & 26.75 $\pm$ 0.41& 27.56 $\pm$ 0.40 \\

MetaOpt$^{\dagger}$   &36.28 $\pm$ 0.50 &49.42 $\pm$ 0.60& \underline{54.80} $\pm$ 0.54 & 22.53 $\pm$ 0.91 &25.53 $\pm$ 1.02& 29.35 $\pm$ 0.99\\

CHEF (Ours)  &\textbf{41.26} $\pm$ 0.34 &\textbf{54.30} $\pm$ 0.34 & \textbf{60.86} $\pm$ 0.18 &\textbf{24.72} $\pm$ 0.14 & \textbf{29.71} $\pm$ 0.27& \textbf{31.25} $\pm$ 0.20\\

    \hline
    \multicolumn{4}{l}{\footnotesize{$^\dagger$ Results reported in \citet{guo19}}} \\
\end{tabular}}
\caption{Comparative results of few-shot learning methods on four proposed cross-domain few-shot challenges CropDiseases, EuroSAT, ISIC, and ChestX. The average 5-way few-shot classification accuracies ($\%$, top-1) along with $95\%$ confidence intervals are reported on the test split of each dataset. Runner-up method is underlined.}
\label{tab:meta}
\end{table*}





\subsection{\textit{mini}Imagenet and \textit{tiered}Imagenet}

\textbf{Datasets and evaluation.}
The \textit{mini}Imagenet dataset \citep{vinyals16}
consists of 100 randomly chosen classes from the ILSVRC-2012 dataset~\citep{russakovsky15}.
We use the commonly-used class split proposed in~\citet{ravi17}. 
The \textit{tiered}Imagenet dataset \citep{ren18} is a subset
of ILSVRC-2012 \citep{russakovsky15}, composed of 608 classes grouped in 34 high-level categories. 
For evaluation, we measure the accuracy
drawing 800 tasks (five test instances per class) 
from the meta-test set.
Following prior
work, we focus on 5-way/1-shot and 5-way/5-shot tasks.
We report the average accuracy and a 95\,\% confidence 
interval across all test images and tasks.

\textbf{CHEF implementation and results.} 
We perform pre-training of the respective backbone networks
on the \textit{mini}Imagenet
and the \textit{tiered}Imagenet
dataset in the same way as described in 
Sec.~\ref{sec:cross_domain}.
We evaluate CHEF 
using two different backbone architectures: 
a Conv-4 and a ResNet-12 network. We use the Conv-4 network described 
by \citet{vinyals16}.
Following \citet{lee19}, we 
configure the ResNet-12 backbone as 4 residual blocks, 
which contain a 
max-pooling and a batch-norm layer and are regularized by 
DropBlock \citep{ghiasi18}.
Again, model selection and hyper-parameter tuning is performed as 
described in Sec.~\ref{sec:cross_domain}.
CHEF achieves state-of-the-art performance in 5 out of 8 categories.
Results are provided in Table~\ref{tab:main}.
An ablation study of the \textit{mini}Imagenet and \textit{tiered}Imagenet results can be found in the appendix.




\begin{table*}[!htb]
\centering
\scalebox{0.78}{
\begin{tabular}{lccccc}
\hline 
 & & \multicolumn{2}{c}{\textbf{\textit{mini}Imagenet 5-way}} & \multicolumn{2}{c}{\textbf{\textit{tiered}Imagenet 5-way}}  \\
\textbf{Method} & \textbf{Backbone} & \textbf{1-shot} & \textbf{5-shot} & \textbf{1-shot} & \textbf{5-shot}  \\
\hline 
MatchingNet \citep{vinyals16} & Conv-4 & $43.56\pm0.84$ & $55.31\pm0.73$ & - & - \\
Meta-LSTM \citep{ravi17} & Conv-4 & $43.44\pm0.77$ & $60.60\pm0.71$ & - & - \\
MAML \citep{finn17} & Conv-4 & $48.70\pm1.84$ & $63.11\pm0.92$ & $51.67\pm1.81^{\dagger}$ & $70.30\pm1.75^{\dagger}$  \\
ProtoNets \citep{snell17} & Conv-4 & $49.42\pm0.78$ & $68.20\pm0.66$ & $48.58\pm0.87^{\dagger}$ & $69.57\pm0.75^{\dagger}$ \\
Reptile \citep{nichol18} & Conv-4 & $47.07\pm0.26$ & $62.74\pm0.37$ & $48.97\pm0.21^{\dagger}$ & $66.47\pm0.21^{\dagger}$ \\
RelationNet \citep{sung18} & Conv-4 & $50.44\pm0.82$ & $65.32\pm0.70$ & $54.48\pm0.93^{\dagger}$ & $71.32\pm0.78^{\dagger}$ \\
IMP \citep{allen19} & Conv-4 & $49.60\pm0.80$ & $68.10\pm0.80$ & - & - \\
FEAT \citep{ye20} & Conv-4 & $55.15\pm0.20$ & $71.61\pm0.16$ & - & - \\
Dynamic FS \citep{gidaris18} & Conv-4 & $56.20\pm0.86$ & $\underline{72.81}\pm0.62$ & - & - \\
MetaQCD \citep{metaQDA2021} & Conv-4 & ${\underline{56.32}\pm0.79}$ & ${72.54\pm0.23}$ & ${\underline{58.11}\pm0.58}$ & ${\underline{74.26}\pm0.69}$ \\
CHEF (Ours) & Conv-4 & $\mathbf{57.60}\pm0.29$ & $\mathbf{73.26}\pm0.23$ & $\mathbf{61.10}\pm0.21$ & $\mathbf{75.83}\pm0.25$ \\
\hline
\hline
\hline
SNAIL \citep{mishra18} & ResNet-12 & $55.71\pm0.99$ & $68.88\pm0.92$ & - & - \\
Dynamic FS \citep{gidaris18} & ResNet-12 & $55.45\pm0.89$ & $70.13\pm0.68$ & - & - \\
TADAM \citep{oreshkin18} & ResNet-12 & $58.50\pm0.30$ & $76.70\pm0.30$ & - & - \\
MTL \citep{sun19} & ResNet-12 & $61.20\pm1.80$ & $75.50\pm0.80$ & - & - \\
VariationalFSL \citep{zhang19} & ResNet-12 & $61.23\pm0.26$ & $77.69\pm0.17$ & - & - \\
TapNet \citep{yoon19} & ResNet-12 & $61.65\pm0.15$ & $76.36\pm0.10$ & $63.08\pm0.15$ & $80.26\pm0.12$ \\
MetaOptNet \citep{lee19} & ResNet-12 & $62.64\pm0.61$ & $78.63\pm0.46$ & $65.81\pm0.74$ & $81.75\pm0.53$ \\
CTM \citep{li19} & ResNet-12 & $62.05\pm0.55$ & $78.63\pm0.06$ & $64.78\pm0.11$ & $81.05\pm0.52$ \\
CAN \citep{hou19} & ResNet-12 & $63.85\pm0.48$ & $79.44\pm0.34$ & $69.89\pm0.51$ & $84.23\pm0.37$ \\%NEED ALL TRAINING DATA
FEAT \citep{ye20} & ResNet-12 & $\mathbf{66.78}\pm0.20$ & $\mathbf{82.05}\pm0.14$ & $\mathbf{70.80}\pm0.23$ & $\underline{84.79}\pm0.16$ \\%NEED ALL TRAINING DATA\\
CHEF (Ours) & ResNet-12 & $\underline{64.11}\pm0.32$ & $\underline{79.99}\pm0.21$ & $\mathbf{70.70}\pm0.35$ & $\mathbf{85.97}\pm0.09$ \\
\hline
\multicolumn{4}{l}{\footnotesize{$^\dagger$ Results reported in~\citep{liu19}}} \\
\end{tabular}}
\caption{Comparative results of few-shot learning methods on the two benchmark datasets \textit{mini}Imagenet and \textit{tiered}Imagenet. The average 5-way few-shot classification accuracies ($\%$, top-1) along with $95\%$ confidence intervals are reported on the test split of each dataset. Runner-up method is underlined.}
\label{tab:main}
\vspace{-0.16in}
\end{table*}

\newpage
\subsection{Example Application: Drug Discovery}\label{sec:drugdiscovery}

In drug discovery, it is essential to know properties of drug candidates, 
such as biological activities
or toxicity \citep{sturm2020industry,mayr2018large}. 
Since the measurements of these properties
require time- and cost-intensive laboratory experiments, 
machine learning models are used to substitute such
measurements \citep{hochreiter2018machine}.
However, due to the high experimental effort 
often only few high-quality measurements are available
for training. 
Thus, few-shot learning is highly relevant 
for computational drug discovery.
We introduce a new perspective to the problem
by viewing the few high-quality measurements as a domain shift
of a much larger dataset.

\textbf{Problem setting. }
We consider a 50-shot cross-domain few-shot learning setting
in the field of toxicity prediction, utilizing
the \emph{Tox21 Data Challenge dataset} (Tox21) 
with twelve different toxic effects \citep{huang2017editorial, mayr2016deeptox}.
Around 50 available measurements is a typical scenario 
when introducing a new high-quality
assay in drug design. 
So far, the standard approach
to deal with such few data points
was to use machine learning methods like
Support Vector Machines (SVMs; \citealp{cortes1995support}) 
or Random Forests (RFs; \citealp{breiman2001random}).
However, these methods do not exploit
the rich data available, like the \emph{ChEMBL20 
drug discovery benchmark} (ChEMBL20) \citep{mayr2018large,gaulton2017chembl}.
Viewing the Tox21 data as a domain shift of the 
ChEMBL20 allows the application 
of cross-domain few-shot learning methods.
In this setting,
a domain shift can be observed
both in the input data and in the 
target data.
The molecules (input domain) are 
strongly shifted towards a specialized 
chemical space, with a Jaccard index of $0.01$
between the two datasets, and 
the biological effects (output domain) 
are shifted towards toxicity without any overlap in this domain. 
A further shift is in the distribution of 
the target labels, which are now much more imbalanced in comparison
to ChEMBL20.
In order to mirror this distribution shift correctly,
the number of toxic vs.\ non-toxic molecules 
in the training sets
for each of the twelve few-shot tasks (twelve different toxic effects) 
are sub-sampled accordingly.
For example, the 50-shot few-shot scenario
(50-50 toxic/non-toxic)
is adjusted to
a 10-90 scenario.  
For the twelve few-shot learning tasks, 
training samples are drawn from the 
training set and 
test samples are drawn from the
test set of the Tox21 data, respectively.
We sample individually
for each of the twelve tasks.

\textbf{CHEF implementation for molecules.}
We first train
a fully-connected deep neural network (FCN)
for the prediction of bioactivities from the 
ChEMBL20 database (original domain).
The network is trained in a massive multi-task setting,
where 1,830 
tasks are predicted at once, such that the network
is forced to learn proficient representations that can be shared for multiple tasks \citep{ma2015deep,unterthiner2014multi}.
The total number of 892,480 features of the ChEMBL20 database
was reduced by a sparseness criterion on the molecules
to 1,866 features.
The neurons in the input layer of the FCN 
represent one of 1,866 ECFP6 \citep{rogers2010extended} features, 
which are used as a feature representation for describing 
the raw structure of the molecules.
Each neuron of the output layer represents one of the
1,830 prediction tasks.
We use the pre-trained network and apply CHEF
by representation fusion of the 
three bottleneck layers of the network
for predicting the twelve different 
toxic effects of the new domain of the Tox21 Data Challenge.


\begin{table}[t]
    \centering
   
    \begin{tabular}{lc}
         \toprule
         Method & ROC-AUC \\ 
         \midrule
         CHEF & {\bf 0.76} $\pm$ 0.02   \\
         SVM & 0.66  $\pm$ 0.03 \\
         RF & 0.64$\pm$ 0.03 \\
         \bottomrule
    \end{tabular}
    \caption{
    ROC-AUC performance for few-shot drug discovery.
    CHEF is compared to conventional methods (SVM, RF)
    for the prediction of toxic effects.
    Mean and standard deviation are computed across twelve
    different effects and across 100 differently sampled
    training and test sets.}
    \label{tab:result_dd}
   
\end{table}

\textbf{Experimental evaluation.} 
We evaluate the performance of CHEF 
on the twelve tasks of the Tox21 Data Challenge
and compare it to conventional methods, like SVMs and RFs, 
that are used in drug design when few data is available.
We use SVMs with a MinMax kernel since it previously yielded
the best results \citep{mayr2018large}.
For CHEF, only the 1,866 ECFP input features
of ChEMBL20 pre-training network database are used
where features with only few occurrences in the training set are discarded
since they do not give enough learning signal for neural network training.
For SVMs and RFs, all encountered ECFP features are used.
ROC-AUC values are computed across the twelve
tasks of the 
Tox21 Data Challenge and across 100
differently sampled training and test sets.
CHEF achieves significantly better
ROC-AUC values than SVMs and RFs.
Table~\ref{tab:result_dd} shows the results 
($p$-value~$< 10^{-17}$ for both SVM and RF 
when using a paired Wilcoxon test).
Results for the twelve individual tasks and
a more detailed description is given in the appendix.
CHEF significantly outperforms traditional methods in drug design,
which demonstrates the great potential of 
cross-domain few-shot learning in this field.

\section{Conclusion} 

We have introduced CHEF as new cross-domain few-shot learning method.
CHEF builds on the concept of representation fusion,
which unifies information from different levels of abstraction. 
An extensive ablation study shows that representation fusion is a decisive factor to boost cross-domain few-shot learning.
Representation fusion allows one to successfully tackle
various few-shot learning problems with large domain shifts across a wide range of different tasks
CHEF obtains new state-of-the-art results in all categories 
of the broader study of cross-domain few-shot learning benchmarks.
Finally, we have tested the performance of CHEF
in a real-world cross-domain application in drug discovery, 
i.e.\ toxicity prediction when a domain shift appears.
CHEF significantly outperforms all traditional approaches 
demonstrating great potential for applications in 
computational drug discovery.

\iffalse
\section*{Acknowledgments}
The ELLIS Unit Linz, the LIT AI Lab, the 
Institute for Machine Learning,
are supported by
the Federal State Upper Austria.
IARAI is supported by Here Technologies.
We thank the projects
AI-MOTION (LIT-2018-6-YOU-212),
DeepToxGen (LIT-2017-3-YOU-003),
AI-SNN (LIT-2018-6-YOU-214), 	
DeepFlood (LIT-2019-8-YOU-213),
Medical Cognitive Computing Center (MC3),
PRIMAL (FFG-873979),
S3AI (FFG-872172),
DL for granular flow (FFG-871302),
ELISE (H2020-ICT-2019-3 ID: 951847),
AIDD (MSCA-ITN-2020 ID: 956832).
We thank
Janssen Pharmaceutica,
UCB Biopharma SRL,
Merck Healthcare KGaA,
Audi.JKU Deep Learning Center,
TGW LOGISTICS GROUP GMBH,
Silicon Austria Labs (SAL),
FILL Gesellschaft mbH,
Anyline GmbH,
Google Brain,
ZF Friedrichshafen AG,
Robert Bosch GmbH,
T\"{U}V Austria,
and the NVIDIA Corporation.
\fi



