\section*{Appendix}
\label{appendix: Appendix}
\input{A0_supp}

\end{document}

\section{Introduction}
\label{sec:introduction}

Deep neural networks 
usually do not generalize well to domains that are not distributed identically to the training data.
Domain adaptation~\cite{csurka2017domain, wang2018deep} addresses this problem by transferring knowledge from a label-rich source domain to a target domain where labels are scarce or unavailable.
Despite impressive results on commonly used benchmark datasets, domain adaptation algorithms often assume that the source and target domains share the same label space~\cite{ganin2015unsupervised, gretton2012kernel, long2015learning, long2018conditional, long2016unsupervised, shen2017wasserstein, sun2016deep}. However, since large-scale labelled datasets are readily accessible as source domain data, a more realistic scenario is partial domain adaptation (PDA), which assumes that the target label space is a subset of the source label space, that has received increasing research attention recently~\cite{bucci2019tackling, chen2019domain, chen2020selective, hu2019multi, xu2019larger, zhang2018importance}. 


Several methods have been proposed to solve partial domain adaptation by reweighting source samples~\cite{bucci2019tackling, chen2019domain, chen2020selective, hu2019multi, xu2019larger, zhang2018importance}. 
However, (1) most of the existing methods still suffer from negative transfer due to the presence of outlier source domain classes, 
which cripples domain-wise transfer with untransferable knowledge; (2) in absence of labels, they often neglect the class-aware information in the target domain which fails to guarantee the discriminability of the latent space; and (3) given filtering of the outliers, samples from the source and the target domain are not alone sufficient to learn domain invariant features for such a complex problem. As a result, the domain classifier may falsely align unlabeled target samples with samples of a different class in the source domain, leading to inconsistent predictions. 

To address the above mentioned challenges, we propose a novel end-to-end \textbf{Select, Label, and Mix (\texttt{\textbf{SLM}}\xspace)} framework for learning discriminative invariant feature representation while preventing negative transfer in partial domain adaptation. Our framework consists of three unique modules working in concert, \textit{i.e.}, select, label and mix, as shown in Figure~\ref{fig:teaser}. First, the select module facilitates the identification of relevant source samples while preventing the negative transfer of untransferrable knowledge. To be specific, our main idea is to learn a model (referred to as the selector network) that outputs the posterior probabilities of all the binary decisions for selecting or discarding each source domain sample before aligning source and target distributions using an adversarial discriminator~\cite{ganin2016domain,long2018conditional}. As these decision functions are discrete and non-differentiable, we rely on a recent Gumbel Softmax sampling approach~\cite{jang2016categorical} to learn the policy jointly with the network parameters through standard back-propagation, without resorting to complex reinforcement learning settings, as in~\cite{chen2019domain, chen2020selective}. Second, we develop an efficient self-labeling strategy that iteratively trains the classifier using both the labeled source domain data and the generated soft pseudo-labels for the target domain to enhance the discriminabilty of the latent space. Finally, the mix module utilizes both intra-domain and inter-domain mixup regularization~\cite{zhang2017mixup} to generate convex combinations of pairs of training samples and their corresponding labels in both domains. Our proposed mix strategy not only helps to explore more intrinsic structures across domains leading to an invariant latent space, but also helps to stabilize the domain discriminator while bridging the distribution shift across domains. In each mini-batch, our unified framework simultaneously eliminates negative transfer by removing outlier source samples and learns discriminative invariant features by labeling and mixing samples respectively for partial domain adaptation. 
Experiments on several datasets illustrate the effectiveness of our proposed framework with state-of-the-art performance (e.g., our approach outperforms DRCN~\cite{li2020deep} (TPAMI'20) by 18.91\% on the challenging VisDA-2017~\cite{peng2017visda} benchmark).
To summarize, our key contributions include:

\begin{itemize}
\setlength{\itemsep}{-1pt}
    \item We propose a novel Select, Label, and Mix (\texttt{\textbf{SLM}}\xspace) framework for learning discriminative and invariant feature representation while preventing intrinsic negative transfer in partial domain adaptation. 
    \item We develop a simple and efficient source sample selection strategy where the selector network is jointly trained with the domain adaptation model using backpropagation through Gumbel Softmax sampling.
    \item We conduct extensive experiments on several PDA benchmark datasets, including Office31~\cite{saenko2010adapting}, Office-Home~\cite{venkateswara2017deep}, ImageNet-Caltech, and VisDA-2017~\cite{peng2017visda} to demonstrate the superiority of our proposed approach over state-of-the-art methods.
\end{itemize}

\section{Related Work}
\label{sec:relatedwork}

\noindent\textbf{Unsupervised Domain Adaptation.} Unsupervised domain adaptation which aims to leverage labeled source domain data to learn to classify unlabeled target domain data has been studied from multiple perspectives (see reviews~\cite{csurka2017domain, wang2018deep}). Various strategies have been developed, including methods for reducing the cross-domain divergence~\cite{gretton2012kernel, long2015learning, shen2017wasserstein, sun2016deep}, adding domain discriminators for adversarial training~\cite{chen2019joint, ganin2015unsupervised, ganin2016domain, long2015learning, long2018conditional, long2016unsupervised, pei2018multi, tzeng2017adversarial}, and image-to-image translation techniques~\cite{hoffman2018cycada,hu2018duplex,murez2018image}. Despite remarkable progress, UDA methods assume that label spaces across source and target domains are identical unlike the practical problem we consider in this work. 

\vspace{1mm}
\noindent\textbf{Partial Domain Adaptation.} 
Representative PDA methods train domain discriminators~\cite{cao2018partialsan, cao2018partialpada, zhang2018importance} with weighting, or use residual correction blocks~\cite{li2020deep}, or weight source examples based on their similarities to target domain~\cite{cao2019learning}. Most relevant to our approach is the work in~\cite{chen2019domain, chen2020selective} which uses Reinforcement Learning (RL) for source data selection in partial domain adaptation. RL policy gradients are
often complex, unwieldy to train and require techniques to reduce variance during training.
By contrast, our approach utilizes a gradient based optimization for relevant source sample selection which is extremely fast and computationally efficient.  
Moreover, while prior PDA methods try to reweigh source samples in some form or other, they often do not take class-aware information in target domain into consideration. Our approach instead, ensures discriminability and invariance of the latent space by considering both pseudo-labeling and cross-domain mixup with sample selection in an unified framework for partial domain adaptation.

\vspace{1mm}
\noindent\textbf{Self-training with Pseudo-Labels.} Deep self-training methods that focus on iteratively training the model by using both labeled source data and generated target pseudo-labels have been proposed for aligning both domains~\cite{inoue2018cross, mei2020instance, saito2017asymmetric, zhang2020label}. While majority of the methods directly choose hard pseudo-labels with high prediction confidence, the works in~\cite{zou2019confidence,zou2018unsupervised} use class-balanced confidence regularizers to generate soft pseudo-labels for unsupervised domain adaptation that share same label space across domains. Our work on the other hand iteratively utilizes soft pseudo-labels within a batch by smoothing one-hot pseudo-label to a conservative target distribution for partial domain adaptation.

\vspace{1mm}
\noindent\textbf{Mixup Regularization.} Mixup regularization~\cite{zhang2017mixup} or its variants~\cite{berthelot2019mixmatch, verma2019manifold} that train models on virtual examples constructed as convex combinations of pairs of inputs and labels are recently used in many computer vision tasks to improve the generalization of neural networks. A few very recent methods apply Mixup, but mainly for UDA to stabilize the domain discriminator~\cite{wu2020dual, xu2019adversarial, yan2020improve} or to smoothen the predictions~\cite{mao2019virtual}. Our proposed SLM strategy can be regarded as an extension of this line of research by introducing both intra-domain and inter-domain mixup not only to stabilize the discriminator but also to guide the classifier in enriching the intrinsic structure of the latent space to solve the more challenging partial domain adaptation task.


\section{Proposed Method}
\label{sec:proposedmethod}

Partial domain adaptation aims to mitigate the domain shift and generalize the model to an unlabelled target domain with a label space which is a subset of that of the labelled source domain. Formally, given a set of labelled source domain samples $\mathcal{D}_{source}\!=\!\{(\textbf{x}_{i}^{s}, y_{i})\}_{i=1}^{N_{S}}$ and unlabelled target domain samples $\mathcal{D}_{target}\!=\!\{\textbf{x}_{i}^{t}\}_{i=1}^{N_{T}}$, with label spaces $\mathcal{L}_{source}$ and $\mathcal{L}_{target}$, respectively, where $\mathcal{L}_{source}\!\subsetneq\!\mathcal{L}_{target}$. Let $p$ and $q$ represent the probability distribution of data in source and target domain respectively. In case of PDA, we further have $p\!\neq\!q$ and $p_{\mathcal{L}_{target}}\!\neq\!q$, where $p_{\mathcal{L}_{target}}$ is the distribution of source domain data in $\mathcal{L}_{target}$. Our goal is to develop an approach with the above given data to improve the performance of a model on $\mathcal{D}_{target}$.

\subsection{Approach Overview}
Figure~\ref{fig:main_figure} illustrates an overview of our proposed approach. Our framework consists of a feature extractor $\mathcal{G}$, a classifier network $\mathcal{F}$, a domain discriminator $\mathcal{D}$ and a selector network $\mathcal{H}$. Our goal is to improve the classification performance of the combined network $\mathcal{F(G(.))}$ on $\mathcal{D}_{target}$. While the feature extractor $\mathcal{G}$ maps the images to a common latent space, the task of the classifier $\mathcal{F}$ is to output a probability distribution over the classes for a given feature from $\mathcal{G}$. Given a feature from $\mathcal{G}$, the discriminator $\mathcal{D}$ helps in minimizing the domain discrepancy by identifying the domain (either source or target) to which it belongs. The selector network $\mathcal{H}$ helps in reducing negative transfer by identifying outlier source samples from $\mathcal{D}_{source}$. On the other hand, label module utilizes the predictions of $\mathcal{F(G(.))}$ to obtain the soft pseudo-labels for the target domain samples. Finally, the mix module leverages both pseudo-labeled target samples and source samples to generate augmented images for achieving a domain invariance in the latent space. During training, for a mini-batch of images, all the components are trained jointly in an iterative manner and during testing, we evaluate the performance using classification accuracy of the network $\mathcal{F(G(.))}$ on target domain data $\mathcal{D}_{target}$.


\subsection{Select, Label, and Mix Framework} Our approach adopts three unique modules namely Select, Label and Mix in a unified framework for learning discriminative invariant feature representation while eliminating negative transfer in partial domain adaptation.
The individual modules are discussed below.


\vspace{1mm}
\noindent\textbf{Select Module.}
This module aims to identify and get rid of the outlier class samples in the source domain to minimize negative transfer in partial domain adaptation. Instead of using different heuristically designed criteria for weighting source samples, we develop a novel selector network $\mathcal{H}$, that takes images from the source domain as input and decides the relevant source samples to align with the target samples. 
Specifically, the selector network $\mathcal{H}$ performs robust selection by providing a discrete output of either a 0 (discard) or 1 (select) for each source sample, \textit{i.e.}, $\mathcal{H}: \mathcal{D}_{source}\!\rightarrow\!\{0, 1\}$. 
However,  the fact that the decision policy is discrete makes the network non-differentiable and therefore difficult to optimize via standard backpropagation.
To resolve the non-differentiability and enable gradient descent optimization for the selector, we adopt Gumbel-Softmax distribution~\cite{jang2016categorical, maddison2016concrete} that uses the Gumbel-Max trick with softmax as a continuous relaxation to $arg max$. For a binary decision space, as in our case (select or discard), the formulation is as follows: 
\begin{align}
    \mathcal{Y}_{i} = \frac{\exp((\log\alpha_{i}\! +\! G_{i})/\tau)}{\sum_{j\in\{0, 1\}} \exp((\log\alpha_{j}\! +\! G_{j})/\tau)} \quad \textnormal{for } i\! \in\!\{0, 1\}
    \label{eq:gumbel_softmax}    
\end{align}
where $G_{i}$'s are i.i.d samples from standard Gumbel distribution $Gumbel(0, 1) = -log(-log(U))$, where $U\sim\text{Uniform}[0,1]$,
$\alpha_{0}$ and $\alpha_{1}$ are output probability of the selector network for a sample to be selected and discarded respectively. $\tau$ denotes temperature of the softmax. Clearly, when $\tau>$ 0, the Gumbel-Softmax distribution is smooth and hence gradients can be computed with respect to  logits $\alpha_{i}$'s to train the selector network using backpropagation. As $\tau$ approaches 0, the decision $\mathcal{Y}_{i}$ becomes one-hot and discrete.
Note that to avoid any interference from the backbone feature extractor $\mathcal{G}$, we use a separate feature extractor for the select module, while making these decisions.   

For a given batch of source domain images $\mathcal{D}_{source}^{b}$ and target domain images $\mathcal{D}_{target}^{b}$, of size $b$, the selector results in two subsets of source samples $\mathcal{D}_{sel}^{b}\! =\! \{ x\! \in\! \mathcal{D}_{source}^{b}\! :\! \mathcal{H}(x)\! =\! 1\}$ and $\mathcal{D}_{dis}^{b}\! =\! \{ x\! \in\! \mathcal{D}_{source}^{b}\! :\! \mathcal{H}(x)\! =\! 0\}$. For training the selector, we propose a triplet loss $\mathcal{L}_{select}$ that tries to bring $\mathcal{D}_{sel}^{b}$ \& $\mathcal{D}_{target}^{b}$ closer while pushing apart $\mathcal{D}_{dis}^{b}$ \& $\mathcal{D}_{target}^{b}$ in the latent feature space of $\mathcal{G}$ as follows:
\begin{align}
    d_{sel} &= d_{H}(\mathcal{G}(\mathcal{D}_{sel}^{b}), \mathcal{G}(\mathcal{D}_{target}^{b})) \nonumber \\
    d_{dis} &= d_{H}(\mathcal{G}(\mathcal{D}_{dis}^{b}), \mathcal{G}(\mathcal{D}_{target}^{b})) \nonumber \\
    \mathcal{L}_{select} &= \lambda_{s}max(d_{sel} - d_{dis} + margin, 0) + \mathcal{L}_{reg}
    \label{eq:loss_select}  
\end{align}
where $d_{H}(X, Y)$ represents the average Hausdorff distance between set of features $X$ and $Y$. $\mathcal{L}_{reg} = \lambda_{reg1}\sum_{x\in\mathcal{D}_{source}^{b}} \mathcal{H}(x)\log(\mathcal{H}(x)) + \lambda_{reg2}\{\sum_{\hat{p}}l_{ent}(\hat{p}) - l_{ent}(\hat{p}_{m})\}$, with $l_{ent}$ being the entropy loss, $\hat{p}$ is the Softmax prediction of $\mathcal{F(G(D}_{target}))$ and $\hat{p}_{m}$ is mean prediction of output $\hat{p}$. $\mathcal{L}_{reg}$ is a regularization to restrict $\mathcal{H}$ from producing trivial all-0 or all-1 outputs as well as ensuring confident and diverse predictions by $\mathcal{F(G(.))}$ for $\mathcal{D}_{target}$. During forward pass, we obtain discrete decisions for source images to discard the outliers, and during backward pass, we learn the parameters of selector network using Gumbel-Softmax while jointly training it with other components.

\vspace{1mm}
\noindent\textbf{Label Module.} While our select module helps in removing source domain outliers, it fails to guarantee the discriminability of the latent space due to the absence of class-aware information in the target domain. Specifically, given our main objective is to improve the classification performance on target domain samples, it becomes essential for the classifier to learn confident decision boundaries in the latent feature space. To this end, we propose a label module that provides additional self-supervision for target domain samples. Motivated by the effectiveness of confidence guided self-training~\cite{zou2019confidence}, we generate soft pseudo-labels for the target domain samples that efficiently attenuates the unwanted deviations caused by false and noisy one-hot pseudo-labels. For a target domain sample $\textbf{x}_{k}^{t} \in \mathcal{D}_{target}$, the soft-pseudo-label $\hat{y}_{k}^{t}$ is computed as follows:
\begin{align}
    \hat{y}_{k}^{t(i)} = \frac{p(i|\textbf{x}_{i}^{t})^{\frac{1}{\alpha}}}{\sum_{j=1}^{|\mathcal{L}_{source}|}p(j|\textbf{x}_{i}^{t})^{\frac{1}{\alpha}}}
    \label{eq:soft_pseudo_label}  
\end{align}
where $p(j|\textbf{x}_{i}^{t}) = \mathcal{F(G(}\textbf{x}_{i}^{t}))^{(j)}$ is the softmax probability of the classifier for class $j$ given $\textbf{x}_{i}^{t}$ as input, and $\alpha$ is a hyper-parameter that controls the softness of the label. 
The soft pseudo-label $\hat{y}_{i}^{t}$ is then used to compute the loss $\mathcal{L}_{label}$ for a given batch of target samples $\mathcal{D}_{target}^{b}$ as follows:
\begin{align}
    \mathcal{L}_{label} = \mathbb{E}_{\textbf{x}_{i}^{t} \in \mathcal{D}_{target}^{b}} l_{ce}(\mathcal{F(G(}\textbf{x}_{i}^{t})), \hat{y}_{i}^{t})
    \label{eq:loss_label}
\end{align}
where $l_{ce}(.)$ represents the cross-entropy loss.

\vspace{1mm}
\noindent\textbf{Mix Module.}
Learning a domain-invariant latent space is crucial for effective adaptation of a classifier from source to target domain. However, with limited samples per batch and after discarding the outlier samples, it becomes even more challenging in preventing over-fitting and learning domain invariant representation in partial domain adaptation. To mitigate this problem, we apply MixUp~\cite{zhang2017mixup} on the relevant source samples and the target samples for discovering ingrained structures in establishing domain invariance. Given $\mathcal{D}_{sel}^{b}$ from the select module and $\mathcal{D}_{target}^{b}$ with corresponding labels $\hat{y}^{t}$ from the label module, we perform convex combinations of the images belonging to these two sets on pixel-level in three different ways namely, inter-domain, intra-source domain and intra-target domain to obtain the following sets of augmented data:
\begin{align}
    \mathcal{D}_{inter\_mix}^{b} = \{(\lambda\textbf{x}_{i}^{s} + (1\!-\!\lambda)\textbf{x}_{j}^{t}, \lambda y_{i} + (1\!-\!\lambda)\hat{y}_{j}^{t})\} \nonumber\\
    \mathcal{D}_{intra\_mix\_s}^{b} = \{(\lambda\textbf{x}_{i}^{s} + (1\!-\!\lambda)\textbf{x}_{j}^{s}, \lambda y_{i} + (1\!-\!\lambda)y_{j})\} \nonumber\\
    \mathcal{D}_{intra\_mix\_t}^{b} = \{(\lambda\textbf{x}_{i}^{t} + (1\!-\!\lambda)\textbf{x}_{j}^{t}, \lambda\hat{y}_{i}^{t} + (1\!-\!\lambda)\hat{y}_{j}^{t})\} \nonumber\\
    \mathcal{D}_{mix}^{b} = \mathcal{D}_{inter\_mix}^{b} \cup \mathcal{D}_{intra\_mix\_s}^{b} \cup \mathcal{D}_{intra\_mix\_t}^{b}
    \label{eq:mixed_data}
\end{align}
where $(\textbf{x}_{i/j}^{s}, y_{i/j}) \in \mathcal{D}_{sel}^{b}$, while $\textbf{x}_{i/j}^{t} \in \mathcal{D}_{target}^{b}$ with  $\hat{y}_{i/j}^{t}$ being the corresponding soft-pseudo-labels. $\lambda$ is the mix-ratio randomly sampled from a beta distribution $Beta(\alpha, \alpha)$ for $\alpha \in (0, \infty)$. We use $\alpha=2.0$ in all our experiments. 
We utilize the new augmented images in training both the classifier $\mathcal{F}$ and the domain discriminator $\mathcal{D}$ as follows:
\begin{align}
    \mathcal{L}_{mix\_cls} &= \mathbb{E}_{(\textbf{x}_{i}, y_{i}) \in \mathcal{D}_{mix}^{b}} l_{ce}(\mathcal{F(G(}\textbf{x}_{i})), y_{i}) \nonumber\\
    \mathcal{L}_{mix\_dom} &= \mathbb{E}_{\textbf{x}_{i} \sim \mathcal{D}_{inter\_mix}^{b}} \{\lambda\log(\mathcal{D(G(}\textbf{x}_{i}))) \nonumber\\
    &+ (1\!-\!\lambda)\log(1\!-\!\mathcal{D(G(}\textbf{x}_{i})))\} \nonumber\\
    &+ \mathbb{E}_{\textbf{x}_{i} \sim \mathcal{D}_{intra\_mix\_s}^{b}} \log(\mathcal{D(G(}\textbf{x}_{i}))) \nonumber\\
    &+ \mathbb{E}_{\textbf{x}_{i} \sim \mathcal{D}_{intra\_mix\_t}^{b}} \log(1\!-\!\mathcal{D(G(}\textbf{x}_{i}))) \nonumber\\
    \mathcal{L}_{mix} &= \mathcal{L}_{mix\_cls} + \mathcal{L}_{mix\_dom}
    \label{eq:loss_mix}
\end{align}
where $\mathcal{L}_{mix\_cls}$ and $\mathcal{L}_{mix\_dom}$ represent loss for classifier and domain discriminator respectively. Our mix strategy with the combined loss $\mathcal{L}_{mix}$ not only helps to explore more intrinsic structures across domains leading to an invariant latent space, but also helps to stabilize the domain discriminator while bridging the distribution shift across domains.

\subsection{Optimization}
Besides the above three unique modules that are tailored for PDA, we use the standard supervised loss on the labeled source data and domain adversarial loss as follows:
\begin{align}
    \mathcal{L}_{sup} &= \mathbb{E}_{(\textbf{x}_{i}, y_{i}) \in \mathcal{D}_{sel}^{b}} l_{ce}(\mathcal{F(G(}\textbf{x}_{i})), y_{i})\nonumber\\
    \mathcal{L}_{adv} &= \mathbb{E}_{\textbf{x}^{s} \sim \mathcal{D}_{sel}^{b}} w^{s}\log(\mathcal{D(G(}\textbf{x}^{s}))) \nonumber\\
    &+ \mathbb{E}_{\textbf{x}^{t} \sim \mathcal{D}_{target}^{b}} w^{t}\log(1\!-\!\mathcal{D(G(}\textbf{x}^{t})))
    \label{eq:loss_standard}
\end{align}
where $\mathcal{L}_{adv}$ is standard entropy-conditioned domain adversarial loss with weights  $w^{s}$ and $w^{t}$ for source and target domain respectively~\cite{long2018conditional}. The overall loss $\mathcal{L}_{total}$ is
\begin{align}
    \mathcal{L}_{total} = \mathcal{L}_{sup} + \mathcal{L}_{adv} + \mathcal{L}_{select} + \mathcal{L}_{label} + \mathcal{L}_{mix}
    \label{eq:loss_total}
\end{align}
where $\mathcal{L}_{select}$, $\mathcal{L}_{label}$, and $\mathcal{L}_{mix}$ are given by Equations (\ref{eq:loss_select}), (\ref{eq:loss_label}), and (\ref{eq:loss_mix}) respectively. We integrate all the modules into one framework, as shown in the Figure~\ref{fig:main_figure} and train the network jointly for partial domain adaptation.   

\section{Experiments}
\label{sec:experiments}

We conduct extensive experiments to show that our \texttt{\textbf{SLM}}\xspace framework outperforms many competing approaches to achieve the new state-of-the-art on several PDA benchmark datasets. We also perform comprehensive ablation experiments and feature visualizations to verify the effectiveness of different components in detail.

\subsection{Experimental Setup}

\noindent\textbf{Datasets.} We evaluate the performance of our approach using several standard domain adaptation datasets under PDA setting, namely Office31~\cite{saenko2010adapting}, Office-Home~\cite{venkateswara2017deep} ImageNet-Caltech and VisDA-2017~\cite{peng2017visda}. 
Office31 contains 4,110 images of 31 classes from three distinct domains, namely Amazon (A), Webcam (W) and DSLR (D). We follow the setting in~\cite{chen2020selective,li2020deep} and select 10 classes shared by Office31 and Caltech256~\cite{griffin2007caltech} as target categories. Office-Home is a challenging dataset that contains images of everyday objects from four domains: Artistic images (Ar), Clipart images (Cl), Product images (Pr) and Real-World images (Rw). We follow~\cite{chen2020selective} to select the first 25 categories (in alphabetic order) in each domain as the target classes. 
ImageNet-Caltech is another challenging dataset that consists of two subsets, ImageNet1K (I)~\cite{russakovsky2015imagenet} and Caltech256 (C)~\cite{griffin2007caltech}. While source domain contains 1,000 and 256 classes for ImageNet and Caltech respectively, each target domain contains only 84 classes that are common across both domains.
VisDA-2017 is a large-scale dataset with 12 categories across 2 domains: one consists photo-realistic images or real images (R) and the other comprises of synthetic 2D renderings of 3D models (S). Following~\cite{li2020deep}, we select the first 6 categories (in alphabetical order) in each of the domain as the target categories.

\vspace{1mm}
\noindent\textbf{Baselines.} We compare our approach with several methods that fall into two main categories: (1) popular unsupervised domain adaptation methods like DAN~\cite{long2015learning}, DANN~\cite{ganin2016domain}, and CORAL~\cite{sun2016deep}, (2) existing partial domain adaptation methods including PADA~\cite{cao2018partialpada}, SAN~\cite{cao2018partialsan}, ETN~\cite{cao2019learning}, and DRCN~\cite{li2020deep}. We also compare with the recent state-of-the-art method, RTNet~\cite{chen2020selective} (CVPR'20) that uses reinforcement learning for source dataset selection in partial domain adaptation. We directly quote the numbers reported in published papers~\cite{chen2020selective,li2020deep} and use the same backbone network in our approach to make a fair comparison with different baselines.

\vspace{1mm}
\noindent\textbf{Implementation Details.} 
We use ResNet-50~\cite{he2016deep} as the backbone network for the feature extractor while we use ResNet-18 for the selector network, initialized with ImageNet~\cite{russakovsky2015imagenet} pretrained weights. 
In Eqn.~\ref{eq:loss_select} we set $\lambda_{s}, \lambda_{reg1}$ and $\lambda_{reg2}$ as $0.01, 10.0$ and $0.1$, respectively. We use gradient reversal layer (GRL) for adversarially training the discriminator. We set $\tau=1.0$ in Eqn.~\ref{eq:gumbel_softmax}, $\alpha=0.1$ in Eqn.~\ref{eq:soft_pseudo_label}, and $\lambda=0.0$ for the GRL as initial values and gradually anneal $\tau$ and $\alpha$ down to 0 while increase $\lambda$ to 1.0 during the training, as in~\cite{jang2016categorical}. We set a threshold of $0.3$ while obtaining the soft pseudo-labels for all the datasets except for the case when ImageNet was used as the source domain where we did not use any threshold value. Additionally, we use label-smoothing for all the losses for the feature extractor involving source domain images as in~\cite{liang2020we, muller2019does}, with $\epsilon\!=\!0.2$.
We use SGD for optimization with momentum=0.9 while a weight decay of 1e-3 and 5e-4 for the selector network and the other networks respectively. We use an initial learning rate of 5e-3 for the selector and the classifier, while 5e-4 for the rest of the networks and decay it following a cosine annealing strategy. A batch size of 64 is used for all datasets except ImageNet-Caltech for which a batch size of 128 is used. We report average classification accuracy and standard deviation over 3 random trials. More implementation details are included in Appendix and we will make our source codes publicly available at \url{https://github.com/CVIR/Select-Label-Mix-SLM-PDA}.


\input{tables/OfficeHome}

\subsection{Results and Analysis}

Table~\ref{table:cls-office-31} shows the results of our method and other competing approaches on Office31 dataset. We have the following key observations.
(1) As expected, the popular UDA methods fail to outperform the simple no adaptation model (ResNet-50), which implies that they suffer from negative transfer due to the presence of outlier source samples.
(2) Overall, our \texttt{\textbf{SLM}}\xspace framework outperforms all the existing PDA methods by achieving the best results on \textbf{5 out of 6} transfer tasks. Among PDA methods, $\text{RTNet}_{adv}$~\cite{chen2020selective} is the most competitive.
However, the gap is still significant (96.9\% vs 98.3\%) due to our two novel components working in concert with the removal of outliers: enhancing discriminability of the latent space via iterative pseudo-labeling of target domain samples and learning domain-invariance through mixup regularizations.
(3) Our approach performed remarkably well on transfer tasks from a large source domain to small target domains, e.g., on A $\!\rightarrow\!$ W, \texttt{\textbf{SLM}}\xspace outperforms $\text{RTNet}_{adv}$ by \textbf{3.5\%}.
Likewise, our approach obtains \textbf{3.8\%} improvement over $\text{RTNet}_{adv}$ on D $\!\rightarrow\!$ A task where the source domain is very small compared to the target domain. These results well demonstrate that our method improves the generalization ability of the source classifier in the target domain while reducing the negative transfer.


On the challenging Office-Home dataset, our proposed approach significantly outperforms all the compared methods to obtain the best performance of 75.3\% which is about \textbf{3\%} more than the previous state-of-the-art performance on this dataset (Table~\ref{table:cls-office-home}).
Our method obtains the best on \textbf{9 out of 12} transfer tasks.
Table~\ref{table:image} summarizes the results of different baselines on ImageNet-Caltech and VisDA-2017 datasets. Our approach achieves new state-of-the-art result, outperforming the next competitive method by a margin of about \textbf{2.8\%} and \textbf{18.9\%} on ImageNet-Caltech and VisDA-2017 datasets respectively. Especially for task S $\rightarrow$ R on VisDA-2017, our approach significantly outperforms SAFN~\cite{xu2019larger} and DRCN~\cite{li2020deep} by an increase of \textbf{24.1\%} and \textbf{33.5\%} respectively. Note that on the most challenging VisDA-2017 dataset, our approach is still able to distill more positive knowledge from the synthetic to the real domain despite significant domain gap across them. In summary, our \texttt{\textbf{SLM}}\xspace framework clearly outperforms the existing PDA methods on all four datasets, showing the effectiveness of our approach in not only identifying the most relevant source classes but also learning more transferable features for partial domain adaptation.

\input{tables/ImageNetCaltech_VisDA17}

\input{tables/OfficeHome_Ablation}

\subsection{Ablation Studies} We perform the following experiments to test the effectiveness of the proposed modules including the effect of number of target classes on different datasets.

\vspace{1mm}
\noindent\textbf{Effectiveness of Individual Modules.} We conduct experiments to investigate the importance of our three unique modules on Office-Home dataset. 
As seen from Table~\ref{table:cls-office-home-ablation}, while the Select only module improves the vanilla performance by 8\%, addition of Label and Mix modules progressively improves the result to obtain the best performance of 75.29\% on Office-Home dataset. This
corroborates the fact that both discriminability and invariance of the latent space plays a crucial role in partial domain adaptation in addition to the removal of source domain outlier samples. 

\vspace{1mm}
\noindent\textbf{Comparison with Varying Number of Target Classes.} We compare different methods by varying the number of target classes. Figure~\ref{fig:var_tgt_cls} shows that our \texttt{\textbf{SLM}}\xspace framework consistently obtains the best results indicating its advantage in alleviating negative transfer by removing outlier source samples. Moreover, our approach outperforms all the compared methods even in the case of completely shared space (A31 $\rightarrow$ W31), which shows that it does not discard relevant samples incorrectly when there are no outlier classes. 

\vspace{1mm}
\noindent\textbf{Effectiveness of Hausdorff Distance.} We investigate the effect of Hausdorff distance (Eqn.~\ref{eq:loss_select}) in training the selector network and find that by removing it from the total loss lowers down the performance from 75.29\% to 73.67\% on Office-Home dataset, showing its importance in guiding the selector network to discard the outlier source samples for effective reduction in negative transfer.

\input{figures/tSNEPlot}
\input{figures/PlotVaryTargetClass}

\vspace{1mm}
\noindent\textbf{Distance between Domains.} Following~\cite{chen2020selective}, we compute the Wasserstein distance between the probability distribution of the target samples (T) with that of the selected ($\text{S}_{sel}$) and discarded samples ($\text{S}_{dis}$) by the selector network. Table~\ref{table:wasser} shows that $\texttt{dist} (\text{S}_{sel}, \text{T})$ is smaller than $\texttt{dist} (\text{S}_{all}, \text{T})$, while $\texttt{dist} (\text{S}_{dis}, \text{T})$ is greater than $\texttt{dist} (\text{S}_{all}, \text{T})$ on two randomly sampled adaptation tasks from Office31 and Office-Home datasets. This results indicate that the samples selected by our selector network is closer to the target domain while  the discarded samples are very dissimilar to the target domain. 

\vspace{1mm}
\noindent\textbf{Effectiveness of Soft Pseudo-Labels.} We also test the effectiveness of soft pseudo-labels by replacing them with hard pseudo-labels for the target samples and observe that hard pseudo-labels decreases the average performance from 75.29\% to 71.96\% on Office-Home dataset. This confirms that soft pseudo-labels are critical for good performance as it efficiently attenuates the unwanted deviations caused by the false and noisy hard pseudo-labels. 

\vspace{1mm}
\noindent\textbf{Effectiveness of Different Mixup.} We examine the effect of mixup regularization on both domain discriminator and classifier on Office-Home dataset. With mixup regularizations working for both discriminator and classifier, the average performance on Office-Home dataset is 75.29\%. By removing mixup regularization from the training of domain discriminator, the performance decreases to 73.58\%. Similarly, by removing mixup regularization from the classifier training, the average performance becomes 73.85\%. This corroborates the fact that our Mix strategy not only helps to explore intrinsic structures across domains, but also helps to stabilize the domain discriminator.  

\subsection{Feature Visualizations}

We use t-SNE~\cite{maaten2008visualizing} to visualize the features learned using different components of our \texttt{\textbf{SLM}}\xspace framework. We choose an UDA setup (similar to DANN~\cite{ganin2016domain}) as a vanilla method and add different modules one-by-one to visualize the contribution of each of the modules in learning discriminative features for partial domain adaptation. As can be seen from Figure~\ref{fig:tsne}, the feature space for the vanilla setup lacks dicriminability for both source and target features. The discriminability improves for both source as well as the target features as we add ``Select" and ``Label" to the Vanilla setup. The best results are obtained when all the three modules ``Select", ``Label" and ``Mix" i.e.~\texttt{\textbf{SLM}}\xspace are added and trained jointly in an end-to-end manner. 
  
Additional results, discussions and more visualization are included in the Appendix.


\input{tables/WassersteinDistance}

\section{Conclusion}
\label{sec:conclusions}

In this paper, we propose an end-to-end framework for learning discriminative invariant feature representation while preventing negative transfer in partial domain adaptation. While our select module facilitates the identification of relevant source samples for adaptation, the label module enhances the discriminability of the latent space by utilizing pseudo-labels for the target domain samples. The mix module uses mixup regularizations jointly with the other two strategies to enforce domain invariance in latent space.     
We demonstrate the effectiveness of our approach on four standard datasets, outperforming several competing methods.
\section{Appendix Overview}


\input{A1_dataset}

\input{A2_implementation}

\input{A3_experiments}

\input{A4_qualitative}


\section{Dataset Details}
We evaluate the performance of our approach on several benchmark datasets for partial domain adaptation, namely Office31~\cite{saenko2010adapting}, Office-Home~\cite{venkateswara2017deep}, ImageNet-Caltech and VisDA-2017~\cite{peng2017visda}. The following are the detailed descriptions of the above datasets: \\

\vspace{-3mm}
\noindent \textbf{Office31.} This dataset contains 4,110 images distributed among 31 different classes and collected from three different domains: Amazon (A), Webcam (W) and DSLR (D), resulting in 6 transfer tasks. The dataset is imbalanced across domains with 2,817 images belonging to Amazon, 795 images to Webcam, and 498 images to DSLR, making Amazon a larger domain as compared to Webcam and DSLR. For all our experiments, we select the 10 classes shared by Office31 and Caltech256~\cite{griffin2007caltech} as the target categories and obtain the following label spaces: \\
$\mathcal{L}_{source} = \{0, 1, 2, ... , 30\}$.\\
$\mathcal{L}_{target} = \{0, 1, 5, 10, 11, 12, 15, 16, 17, 22\}$.\\
Number of Outlier Classes = $21$.\\
Figure~\ref{fig:office-31-gallery} shows few randomly sampled images from this dataset. The dataset is publicly available to download at: \url{https://people.eecs.berkeley.edu/~jhoffman/domainadapt/#datasets_code}. 

\input{figures/Office31Gallery}

\vspace{2mm}
\noindent \textbf{Office-Home.} This dataset contains 15,588 images distributed among 65 different classes and collected from four different domains: Art (Ar), Clipart (Cl), Product (Pr), and RealWorld (Rw), resulting in 12 transfer tasks. The dataset is split across domains with 2427 images belonging to Art, 4365 images to Clipart, 4439 images to Product, and 4347 images to RealWorld. We select the first 25 categories (in alphabetic order) in each domain as the target classes and obtain the following label spaces:\\
$\mathcal{L}_{source} = \{0, 1, 2, ... , 64\}$.\\
$\mathcal{L}_{target} = \{0, 1, 2, ... , 24\}$.\\
Number of Outlier Classes = $40$.\\
Figure~\ref{fig:office-home-gallery} displays a gallery of sample images for this dataset. The dataset is publicly available to download at: \url{http://hemanthdv.org/OfficeHome-Dataset/}. \\

\input{figures/OfficeHomeGallery}

\vspace{2mm}
\noindent
\textbf{ImageNet-Caltech.} This large-scale dataset consists of two datasets (ImageNet1K~\cite{russakovsky2015imagenet} (I) \& Caltech256~\cite{griffin2007caltech} (C)) as two separate domains and consist of over 14 million images combined. 2 transfer tasks are formed for this dataset. While source domain contains 1,000 and 256 classes for ImageNet and Caltech respectively, each target domain contains only 84 classes that are common across both domains. As it is a general practice to use ImageNet pretrained weights for network initialization, we use the validation set images when using ImageNet as the target domain.
\noindent
Number of Outlier Classes = $172$ for C$\rightarrow$I, $916$ for I$\rightarrow$C.
Figure~\ref{fig:imagenet-caltech-gallery} displays a gallery of sample images for this dataset. The datasets are publicly available to download at: \url{http://www.image-net.org/} \\ \url{http://www.vision.caltech.edu/Image_Datasets/Caltech256/}.

\input{figures/ImageNetCaltechGallery}

\vspace{2mm}
\noindent
\textbf{VisDA-2017.} This dataset contains 280,157 images distributed among 12 different classes and two domains. The dataset contains three sets of images: training, validation and testing. The training set contains 152,397 synthetic (S) images, the validation set contains 55,388 real-world (R) images, while the test set contains 72,372 real-world images. For the experiments, the training set is considered as the Synthetic (S) domain, while the validation set as the Real (R) domain, following~\cite{li2020deep}. This results in 2 transfer tasks. The first 6 categories (in alphabetical order) are selected in each of the domains as the target classes, and the following label spaces are obtained: \\
$\mathcal{L}_{source} = \{0, 1, 2, ... , 11\}$.\\
$\mathcal{L}_{target} = \{0, 1, 2, ... , 5\}$.\\
Number of Outlier Classes = $6$.\\
Figure~\ref{fig:visda-gallery} displays a gallery of sample images for this dataset. The dataset is publicly available to download at:\\ \url{http://ai.bu.edu/visda-2017/#download}.

\input{figures/VisDAGallery}
\section{Implementation Details}
Following are the detailed description of the implementation we follow for various components of the framework: 

\vspace{1mm}
\noindent
\textbf{Feature Extractor ($\mathcal{G}$).} We use ResNet-50~\cite{he2016deep} backbone for the feature extractor. The overall structure of ResNet-50 is \texttt{Initial Layers, Layer-1, Layer-2, Layer-3, Layer-4, AvgPool, Fc}. The model is initialized with ImageNet~\cite{russakovsky2015imagenet} pretrained weights. Additionally, we add a bottleneck layer of width 256 just after the \texttt{AvgPool} layer to obtain the features and replace all the BatchNorm layers with Domain-Specific Batch-Normalization~\cite{chang2019domain} layers. All the layers till \texttt{Layer-3} are frozen and only the rest of the layers are fine-tuned.

\vspace{1mm}
\noindent\textbf{Selector Network ($\mathcal{H}$).} We use a ResNet-18~\cite{he2016deep} network with the \texttt{Fc} layer replaced with a binary-length fully connected layer as the selector network in our framework. The network is initialized with ImageNet pretrained weights and all the layers are trained while optimization. 

\vspace{1mm}
\noindent\textbf{Classifier ($\mathcal{F}$).} The final \texttt{Fc} layer of ResNet-50 described above is replaced with a task-specific fully-connected layer to form the classifier network of our framework.

\vspace{1mm}
\noindent\textbf{Domain Discriminator ($\mathcal{D}$).} A three-layer fully-connected network is used as the domain discriminator network. It takes the 256-length features obtained from the feature extractor as input. The adversarial training is incorporated using a gradient reversal layer (GRL).

\vspace{1mm}
\noindent\textbf{Hyperparameters.}
All the networks are optimised with using mini-batch stochastic gradient descent with a momentum of 0.9. A batch size of 64 is used for Office31, Office-Home and VisDA-2017, while a batch size of 128 is used for ImageNet-Caltech. For feature extractor an initial learning rate of 5e-5 for the convolutional layers while an initial learning rate of 5e-4 for all the fully-connected layers is used. For the selector network and the domain discriminator an initial learning rate of 5e-3 and 5e-4 are used respectively. The learning rates are decayed following a cosine-annealing strategy as the training progresses. The best models are captured by obtaining the performance on a validation set. We do not follow the ten-crop technique~\cite{cao2018partialpada, cao2019learning}, to improve the performance in the inference phase.
\section{Additional Experimental Results}

\input{tables/Office31_VGG16}
\input{tables/Office31_VisDA17_Ablation}
\input{tables/OfficeHome_HausdorffAblation}


\noindent
\textbf{Effectiveness on Different Backbone Networks.}
To show that the proposed framework is backbone-agnostic, i.e. it provides the best performance irrespective of the architecture of the feature extractor, we conduct experiments using a VGG-16~\cite{Simonyan15} backbone for the feature extractor. We report the results on the transfer tasks from the Office31 dataset in Table~\ref{table:cls-office-31-vgg16} and compare it with the current state-of-the-art methods. Our method outperforms the previously best results by a margin of \textbf{3.05\%} on average and achieves new state-of-the-art results. This confirms that our proposed framework for partial domain adaptation is robust with respect to the change of backbone network. 


\vspace{1mm}
\noindent
\textbf{Effectiveness of Individual Modules.}
In Section 4.3 of the main paper, we discussed the importance of the proposed three unique modules on Office-Home dataset. Here, we extend the experiments to Office31 and VisDA-2017 and provide the performance on the transfer tasks in Table~\ref{table:cls-office31-visda17-ablation}. Similar to the results on Office-Home dataset, our approach with all the three modules (Select, Label and Mix) working jointly, works the best on both datasets.

\input{tables/OfficeHome_HardPLAblation}
\input{tables/OfficeHome_MixAblation}

\vspace{1mm}
\noindent
\textbf{Effectiveness of Hausdorff Distance.} 
In Section 4.3 of the main paper, we discussed the importance of Hausdorff Distance loss in guiding the selector network to discard the outlier source samples. Here we provide the individual performance of all the transfer tasks on Office-Home dataset in Table~\ref{table:cls-office-home-hausdorff}, which shows that our approach with Hausdorff distance loss works the best in all cases.

\vspace{1mm}
\noindent
\textbf{Effectiveness of Soft Pseudo-Labels.}
As discussed in the Section 4.3, we confirmed the importance of soft pseudo-labels for our framework as it attenuates the unwanted deviations because of noisy and false hard pseudo-labels. Here, we provide the performance on each of the transfer tasks from Office-Home in Table~\ref{table:cls-office-home-ablation-hardpl}.

\vspace{1mm}
\noindent
\textbf{Effectiveness of Different MixUp.} 
We examined the effect of mixup regularization on both domain discriminator and classifier separately in Section 4.3 of the main paper. We concluded that our Mix strategy not only helps to explore intrinsic structures across domains, but also helps
to stabilize the domain discriminator. Here, we provide the corresponding performance on each of the transfer tasks of Office-Home in Table~\ref{table:cls-office-home-ablation-mix}.

\section{Qualitative Results}
\vspace{-0.8mm}
\input{figures/tSNEPlot_Supp}

\noindent
\textbf{Feature Visualizations.}
We provide some additional feature visualizations using t-SNE~\cite{maaten2008visualizing} in Figure~\ref{fig:tsne_supp}. Similar to Section 4.4 in the main paper, we choose an UDA setup as a vanilla method and add the proposed modules one-by-one to visualize the contribution of each of the modules in learning discriminative features for partial domain adaptation.























