




\section{Introduction}\label{sec:1_introduction}
Many machine learning studies assume that training and test data are independent and identically distributed (\textit{i.i.d}).
However, this \textit{i.i.d} assumption does not always hold in real-world scenarios where distribution shifts between training and test data occur frequently.
Thus, traditional machine learning models often show poor performance on unseen domains shifted from source (training) domains~\citep{quinonero2008dataset,torralba2011unbiased}.
To tackle this problem, \textit{domain generalization} has attracted much attention recently.

The main goal of domain generalization is to build generalized models that also perform the target task (\textit{e.g.,} classification) well on unseen domains (\textit{e.g.,} painted images) when only source domains (\textit{e.g.,} realistic images) are accessible during model optimization.
Early domain generalization studies~\citep{muandet2013domain,ganin2016domain,li2018domain} have focused on learning domain-invariant representations across the source domains.
However, \citet{gulrajani2021in} have recently shown that simple empirical risk minimization (ERM)~\citep{vapnik1999nature} outperforms the previous methods on \textsc{DomainBed}, a benchmark for domain generalization, with pre-trained ResNet-50~\citep{he2016deep}.
Moreover, \citet{yu2021empirical} provide empirical evidence that large-scale pre-trained models could play an important role in domain generalization by providing their generalization power.

Motivated by this, several studies have begun to leverage the generalization power of large-scale pre-trained models.
\citet{cha2022domain} employ a pre-trained model for regularization, considering it as an approximation of the oracle model on any domain, and \citet{li2022domain} utilize a frozen pre-trained model as a feature extractor.
These studies have proven the usefulness of pre-trained models in domain generalization.
However, the pre-trained models used in those studies cannot learn task-specific knowledge further since they are frozen during model optimization to preserve their generalization ability.
To learn the task-specific knowledge, one can choose \textit{fine-tuning} that updates all the parameters of pre-trained models by optimizing the models on the source domains.
However, \citet{kumar2022finetuning} demonstrate that fine-tuning distorts generalized representations of the pre-trained models. 
Namely, fine-tuning hurts the generalization ability of pre-trained models.
\input{Figures/1_gradient_bias}

In this paper, we interpret the above issue in terms of \textit{gradient bias} during model optimization.
As shown in Figure~\ref{fig:1_a_biased_gradients}, the gradient of naive fine-tuning is biased toward the source domains because it is computed by only the source domains, disregarding unseen domains.
Although this biased gradient reduces empirical risks in the source domains with the learning of task-specific knowledge, it probably increases risks in the unseen domains.
We argue that the gradient bias would be relieved if gradients that lower the risks in the unseen domains are observable.

To this end, we propose a new domain generalization method, called \textit{GESTUR}, which estimates the unobservable gradients with a large-scale pre-trained model.
GESTUR consists of two key components: a task expert (TE) and a generalization expert (GE).
Based on ERM where gradients tend to be biased to the source domains, TE learns task-specific knowledge from source domains directly to transfer the knowledge to GE.
Meanwhile, GE learns the task-specific knowledge from TE indirectly via exponential moving average (EMA) while preserving the generalization ability of a large-scale pre-trained model.
Still, the gradient bias of TE might impair the generalization ability of GE.
To mitigate this, GE is utilized to estimate the unobservable gradient that minimizes risks in unseen domains for TE based on the assumption that large-scale pre-trained models could act as a loose approximation of the oracle model of unseen domains (\S~\ref{sec:2_methodology}).
As shown in Figure~\ref{fig:1_b_gradient_conflict}, the biased gradient of TE is relieved by simply adding the estimated unobservable gradient to the biased gradient, improving domain generalization performance (\S~\ref{sec:3_experiments}).
Extensive experiments and analyses demonstrate that GESTUR outperforms baseline methods by learning the task-specific knowledge appropriately from source domains while preserving the generalization ability of large-scale pre-trained models.

\paragraph{Contributions:} 
(1) We propose a simple yet effective domain generalization method that learns task-specific knowledge while preserving the generalization ability of large-scale pre-trained models.
Our proposed method estimates the unobservable gradients that reduce potential risks in unseen domains to relieve the gradient bias toward source domains, based on the two experts, TE and GE.
(2) We conduct extensive experiments to show the effectiveness of our proposed method in domain generalization. 
By providing careful analyses, we demonstrate that the unobservable gradients could be estimated with a large-scale pre-trained model, and it relieves the gradient bias.
We also demonstrate that our proposed method learns task-specific knowledge without sacrificing the generalization ability of the large-scale pre-trained model.
\section{Methodology}\label{sec:2_methodology}
\subsection{Preliminaries}\label{subsec:2_1_preliminaries}
\paragraph{Problem formulation.}
Let ${\mathcal{D}}_{s}$ and ${\mathcal{D}}_{u}$ be sets of source domains and unseen domains, respectively.
Each domain ${\mathcal{D}}$ contains the total number of $n_{\mathcal{D}}$ data samples, $\{(x_i,y_i)\}_{i=1}^{n_{\mathcal{D}}}\sim{\mathcal{D}}$, where each data sample $(x_i,y_i)$ consists of an input $x_i$ and its target label $y_i$.
The $n_{\mathcal{D}}$ data samples are \textit{i.i.d} over some probability distribution.
The main goal of domain generalization is to build a model ${\mathbf{\theta}}$ that performs well on the unseen domains ${\mathcal{D}}_u$ when the source domains ${\mathcal{D}}_s$ are only available:
\begin{equation}
\label{eq:2_1_dg_formulation}
    \min_{{\mathbf{\theta}}} \mathbb{E}_{{\mathcal{D}} \sim {\mathcal{D}}_u}
    \mathbb{E}_{(x,y) \sim {\mathcal{D}}} [\ell((x,y);{\mathbf{\theta}})],
\end{equation}
where $\ell((x,y);{\mathbf{\theta}})$ is the loss function defined for the model ${\mathbf{\theta}}$ on the data sample $(x,y)$.
Note that this study focuses on solving classification tasks.
Hence, we denote the model in detail as ${\mathbf{\theta}}=\{{\mathbf{\theta}}^{f};{\mathbf{\theta}}^{c}\}$ consisting of its feature extractor ${\mathbf{\theta}}^{f}$ and classifier ${\mathbf{\theta}}^{c}$.

\paragraph{Motivation.}
With success in many downstream tasks, it has become a convention to initialize the feature extractor ${\mathbf{\theta}}^f$ with a large-scale pre-trained model.
Although pre-trained models provide better feature representations than randomly initialized parameters, they do not fully equip task-specific knowledge yet.
It is because there is a discrepancy between the pre-training objective and the target task.
For example, CLIP~\citep{radford2021learning} is pre-trained to match web-crawled image-caption pairs, whereas the target task is to classify data into seven classes (\textit{e.g.,} horse and dog), in the case of PACS~\citep{li2017deeper}.
Therefore, many studies have adopted fine-tuning that updates all the parameters of the feature extractor ${\mathbf{\theta}}^f$ to learn the task-specific knowledge by optimizing the model on source domains ${\mathcal{D}}_s$.
However, \citet{kumar2022finetuning} observe that fine-tuning impairs generalization ability of pre-trained models during the learning of task-specific knowledge.

We try to interpret this issue at the gradient level.
Based on ERM~\citep{vapnik1999nature}, the gradient ${\mathbf{g}}$ of fine-tuning is computed for the model ${\mathbf{\theta}}$ on the source domains ${\mathcal{D}}_s$, as follows:
\begin{equation}
\label{eq:2_2_fine_tuning_gradients}
    {\mathbf{g}} = \nabla_{{\mathbf{\theta}}}
    \mathbb{E}_{(x, y) \sim {\mathcal{B}}} [\ell((x, y);{\mathbf{\theta}})],
\end{equation}
where ${\mathcal{B}}$ is a mini-batch sampled from the source domains ${\mathcal{D}}_s$.
The gradient ${\mathbf{g}}$ is influenced by only the source domains ${\mathcal{D}}_s$ because the unseen domains ${\mathcal{D}}_u$ are not accessible.
Namely, the gradient is biased toward the source domains.
We presume that this gradient bias degrades generalization performance in the unseen domains.

\subsection{GESTUR: \underline{G}rdient \underline{Est}imation for \underline{U}nseen Domain \underline{R}isk Minimization with Pre-Trained Models}\label{subsec:2_2_gestur}
We hypothesize that the gradient bias mentioned above could be relieved if the unobservable gradient ${\mathbf{g}}_u$ minimizing risks in the unseen domains is computable.
To achieve this, we borrow the assumption of \citet{cha2022domain} that large-scale pre-trained models are the approximation of the oracle model ${\mathbf{\theta}}^{*}$ which is optimally generalized for any domain ${\mathcal{D}}$.
Since the unobservable gradient ${\mathbf{g}}_u$ cannot be computed from the unseen domains ${\mathcal{D}}_u$ directly, we consider the direction from the current model ${\mathbf{\theta}}$ to the oracle model ${\mathbf{\theta}}^{*}$ as the unobservable gradient ${\mathbf{g}}_u$.
However, the oracle model is inaccessible in practice.
Hence, we estimate the unobservable gradient using a large-scale pre-trained model as the approximation of the oracle model.

Note that we aim to estimate the unobservable gradient ${\mathbf{g}}_u$ for the unseen domains ${\mathcal{D}}_u$ to alleviate the gradient bias, so the above assumption needs to be more elaborate due to the following reasons.
First, we intend to design the unobservable gradient for the unseen domains only rather than any domain.
Second, pre-trained models do not have task-specific knowledge yet, as described in $\S$~\ref{subsec:2_1_preliminaries}.
Therefore, we slightly modify the assumption as follows: pre-trained models are the \textit{loose} approximation of the oracle model ${\mathbf{\theta}}^*_u$ of \textit{the unseen domains} ${\mathcal{D}}_u$, and they could get \textit{closer} to the oracle model by learning task-specific knowledge.
Based on this assumption, we propose a simple yet effective domain generalization method, GESTUR, which estimates the unobservable gradient ${\mathbf{g}}_u$ for unseen domain risk minimization with a large-scale pre-trained model.

\paragraph{Task expert and generalization expert.}
GESTUR consists of two classification models: a task expert (TE, ${\mathbf{\theta}}_\text{TE}$) and a generalization expert (GE, ${\mathbf{\theta}}_\text{GE}$), which are complementary to each other.
Their feature extractors are both initialized with a large-scale pre-trained model ${\mathbf{\theta}}_0$, respectively.
TE learns task-specific knowledge from the source domains ${\mathcal{D}}_s$ directly to transfer the knowledge to GE.
Meanwhile, GE also learns task-specific knowledge from TE via EMA, but it preserves the generalization ability of the pre-trained model deliberately.
Here, the gradient bias of TE might hurt the generalization ability of GE because the knowledge of TE is injected into GE.
To relieve the gradient bias, GE is used to estimate the unobservable gradient ${\mathbf{g}}_u$ as the loose approximation of the oracle model ${\mathbf{\theta}}^*_u$ for the unseen domains ${\mathcal{D}}_u$.
Our proposed GESTUR is summarized in Algorithm~\ref{alg:1_gestur}.

\input{Algorithms/1_gradg_algorithm}
\paragraph{Gradient estimation.}
Using Equation~\ref{eq:2_2_fine_tuning_gradients}, the gradient ${\mathbf{g}}$ for TE is computed as ${\mathbf{g}}$ = $\nabla_{{\mathbf{\theta}}}\mathbb{E}_{(x, y) \sim {\mathcal{B}}} [\ell((x, y);{\mathbf{\theta}}_\text{TE})]$ while learning task-specific knowledge.
The gradient ${\mathbf{g}}$ is biased toward the source domains ${\mathcal{D}}_s$.
A gradient that minimizes risks in the unseen domains could relieve the gradient bias, but it is unobservable.
When we have access to the oracle model ${\mathbf{\theta}}^*_u$ of the unseen domains ${\mathcal{D}}_u$, we can direct the current model to head to the oracle model instead of empirically calculating the unobservable gradient from the unseen domains.
Hence, we treat the direction from the current model ${\mathbf{\theta}}_\text{TE}$ to the oracle model ${\mathbf{\theta}}^*_u$ as the unobservable gradient ${\mathbf{g}}_u$:
\begin{equation}
\label{eq:2_3_unobservable_gradient}
    {\mathbf{g}}_u = {\mathbf{\theta}}^*_u - {\mathbf{\theta}}_\text{TE}.
\end{equation}
In fact, it is infeasible to access the oracle model.
Thus, we estimate the unobservable gradient using GE that approximates the oracle model loosely, as follows:
\begin{equation}
\label{eq:2_4_gradient_estimation}
    \Tilde{{\mathbf{g}}}_u = {\mathbf{\theta}}_\text{GE} - {\mathbf{\theta}}_\text{TE}.
\end{equation}
This estimated gradient $\Tilde{{\mathbf{g}}}_u$ is used to relieve the gradient bias during the parameter optimization.

\paragraph{Parameter optimization.}
We want to emphasize again that GESTUR leverages the generalization power of large-scale pre-trained models to relieve the gradient bias which distorts the generalized feature representations of the feature extractor ${\mathbf{\theta}}^f$.
Hence, we limit the scope of usage of the estimated unobservable gradient $\Tilde{{\mathbf{g}}}_u$ only to the feature extractor ${\mathbf{\theta}}^f$, not the classifier ${\mathbf{\theta}}^c$.

For TE, the estimated gradient $\Tilde{{\mathbf{g}}}^f_u$ for the feature extractor ${\mathbf{\theta}}^f_\text{TE}$ is added to the biased gradient ${\mathbf{g}}^f$ for the same feature extractor, as follows:
\begin{equation}
\label{eq:2_5_gradient_addition}
    {\mathbf{g}}^f = \frac{1}{2}\Big({\mathbf{g}}^f + \lambda \|{\mathbf{g}}^{f}\|_2 \cdot\dfrac{\Tilde{{\mathbf{g}}}^{f}_u}{\|\Tilde{{\mathbf{g}}}^{f}_u\|_2}\Big),
\end{equation}
where $\lambda$ is a gradient scale factor that controls the influence of the normalized $\Tilde{{\mathbf{g}}}^f_u$.
The feature extractor ${\mathbf{\theta}}^f_\text{TE}$ is updated with the gradient ${\mathbf{g}}^f$ adjusted by $\Tilde{{\mathbf{g}}}^f_u$.
On the other hand, the classifier ${\mathbf{\theta}}^c_\text{TE}$ of TE is updated with its original gradient ${\mathbf{g}}^c$.

As our assumption, GE can get closer to the oracle model ${\mathbf{\theta}}^*$ by learning task-specific knowledge.
However, the generalization ability of GE decreases when we optimize GE on the source domains ${\mathcal{D}}_s$ directly to learn the task-specific knowledge.
Therefore, we inject the learned task-specific knowledge of TE into GE delicately via EMA:
\begin{equation}
\label{eq:2_6_ema}
    {\mathbf{\theta}}_{\text{GE}} = m{\mathbf{\theta}}_{\text{GE}} + (1-m) {\mathbf{\theta}}_{\text{TE}},
\end{equation}
where $m$ is the moving average coefficient.
By encouraging the parameters of GE to change slowly, EMA is helpful in preserving the generalization ability of GE.
Since the goal of domain generalization is to build a model that minimizes the risk of the unseen domains, we choose GE, designed to approximate the oracle model of the domains, as our final model ${\mathbf{\theta}}$.
\section{Experiments}\label{sec:3_experiments}
\subsection{Experimental Setup}\label{subsec:3_1_setup}
\paragraph{Datasets.}
We conduct experiments using five popular domain generalization benchmark datasets:
PACS~\citep{li2017deeper} (4 domains \& 7 classes), VLCS~\citep{fang2013unbiased} (4 domains \& 5 classes), OfficeHome~\citep{venkateswara2017deep} (4 domains \& 65 classes), TerraIncognita~\citep{beery2018recognition} (4 domains \& 10 classes), and DomainNet~\citep{peng2019moment} (6 domains \& 345 classes).

\paragraph{Pre-trained models.}
GESTUR heavily relies on pre-trained models.
Therefore, we employ three pre-trained models of different sizes to verify that the proposed method performs well with various pre-trained models generally:
ResNet-50~\citep{he2016deep} pre-trained on ImageNet~\citep{5206848} (\texttt{RN50}), ViT-B/16~\citep{dosovitskiy2021an} with CLIP~\citep{radford2021learning} (\texttt{CLIP}), and RegNetY-16GF~\citep{radosavovic2020designing} with SWAG~\citep{Singh_2022_CVPR} (\texttt{SWAG}).

\paragraph{Evaluation protocol.}
We adopt the experimental protocol of \textsc{DomainBed}, which enforces fair and realistic evaluations (\textit{e.g.,} same model selection criterion) across competitors.
We divide the data from each domain into 80\% and 20\% splits and follow \textit{training-domain validation set} strategy for the model selection and the hyperparameter search in every experiment.
We also repeat every experiment three times to reduce the randomness in dataset splits and parameter initialization, similar to \citet{gulrajani2021in}, and report the mean and standard error of the experimental results.

\input{Tables/1_main_table}
\paragraph{Implementation details.}
Our implementation is built on the codebase of \citet{cha2022domain}.
We use Adam optimizer~\citep{kingma2015adam} for parameter optimization.
GESTUR has two hyperparameters, the gradient scale factor ($\lambda$) and the moving average coefficient ($m$).
In every experiment, we search the optimal $\lambda$ from \{0.01, 0.05, 0.1, 0.5\} and fix $m$ as $0.999$.
Other hyperparameters such as learning rate, weight decay, and dropout are searched in the same way as ~\citet{cha2022domain}.
We explain more details of implementation in Appendix~\ref{appedix:a_implementation_details}.

\paragraph{Baselines.}
We exhaustively compare our proposed method with various baseline methods in the experiment.
For simplicity, we report only the experimental results of baseline methods that show the higher performance than ERM~\citep{vapnik1999nature}, the simplest baseline method.
We describe the baseline methods and report the full version of the results in Appendix~\ref{appendix:b_1_exp}.

\subsection{Main results}\label{subsec:3_2_results}
\paragraph{Results on \texttt{RN50}.}
The first part of Table~\ref{tab:1_main_table} shows the experimental results where \texttt{RN50} is used to initialize the feature extractor.
GESTUR achieves the best performance for all the datasets except DomainNet.
In detail, the proposed method outperforms ERM by an average of 3.2\textit{\%p}.
Furthermore, our proposed method improves the runner-up by: 1.0\textit{\%p} in VLCS, 0.5\textit{\%p} in OfficeHome, and 1.3\textit{\%p} in TerraIncognita.
Especially, the proposed method outperforms the state-of-the-art method (SWAD~\citep{cha2021swad}) by an average of 0.5\textit{\%p}.

\paragraph{Results on \texttt{CLIP} and \texttt{SWAG}.}
In the second and third parts of Table~\ref{tab:1_main_table}, we show the experimental results where the larger pre-trained models, \texttt{CLIP} and \texttt{SWAG}, are used to initialize the feature extractor, respectively.
In summary, GESTUR achieves the best performance in all the datasets.
In detail, the proposed method outperforms MIRO that also leverages generalization power of large-scale pre-trained models by 1.8\textit{\%p} and 2.8\textit{\%p} on \texttt{CLIP} and \texttt{SWAG}, respectively.
From this, we verify that the proposed method successfully leverages the generalization ability of pre-trained models compared to other baseline methods.
Interestingly, we observe that the performance gap between the proposed method and ERM increases as the size of the pre-trained model increases.

\subsection{Comparison between the task expert and the generalization expert}\label{subsec:3_3_te}
\input{Tables/2_analysis_expert}
\paragraph{Setup.}
GESTUR consists of two essential components: the task expert (TE) and the generalization expert (GE).
In this paper, we use GE as the final model based on the assumption that GE is set as the approximation of the oracle model of unseen domains.
Nevertheless, TE is also designed to preserve the generalization ability of pre-trained models since it also considers the estimated unobservable gradient in every update to relieve its gradient bias.
Therefore, we compare the performance of ERM, GESTUR w/ GE, and its variant GESTUR w/ TE based on the hyperparameters searched in \S~\ref{subsec:3_2_results} to show that they preserve the generalization ability of large-scale pre-trained models.

\paragraph{Results.}
As shown in Table~\ref{tab:2_analysis_expert}, GESTUR w/ GE achieves the best performance in all experiments.
Also, GESTUR w/ TE outperforms ERM by averages of 9.8\textit{\%p} and 4.5\textit{\%p} when using \texttt{CLIP} and \texttt{SWAG}, respectively.
The performance of GESTUR w/ TE is higher when the larger pre-trained models are given, similar to the observation in \S~\ref{subsec:3_2_results}.
These observations demonstrate that GESTUR w/ TE could preserve the generalization ability of the pre-trained models with the estimated unobservable gradient, \textit{i.e.,} the gradient bias of TE is relieved.
Moreover, GESTUR w/ GE shows a higher performance than GESTUR w/ TE, which indicates that EMA ensures the model preserves generalization ability during the learning of task-specific knowledge stable.
From these, we reaffirm the justification for our choice of GE as the final model.

\subsection{Comparison with ERM in terms of gradient bias}\label{subsec:3_4_gb}
\input{Tables/3_analysis_conflict}
\paragraph{Setup.}
As described in $\S$~\ref{sec:1_introduction}, we suspect that the gradient bias degrades the domain generalization performance.
We further conduct analysis to check how much gradient bias occurs during the fine-tuning and how much gradient bias is alleviated by our proposed method.
To quantify the gradient bias, we borrow the concept of \textit{gradient conflict}~\citep{yu2020gradient,mansilla2021domain}: there is a conflict between two gradients ${\mathbf{g}}_i$ and ${\mathbf{g}}_j$ if ${\mathbf{g}}_i\cdot{\mathbf{g}}_j < 0$.
For every iteration, we first sample two mini-batches from both source domains and an unseen domain, respectively.
We then compute losses of the mini-batches, and calculate gradients ${\mathbf{g}}$ and ${\mathbf{g}}_u$ from the losses, respectively.
Finally, we count the number of iterations where the gradient conflict (${\mathbf{g}}\cdot{\mathbf{g}}_u < 0$) occurs, for ERM and GESTUR.
Here, we update the model using only the gradient ${\mathbf{g}}$ since unseen domains are inaccessible in practice.

\paragraph{Results.}
As shown in Table~\ref{tab:3_analysis_conflict}, GESTUR reduces the gradient conflicts of ERM by around 11.5\%, 21\%, and 28.2\% for the pre-trained models, respectively.
From this, we verify that our proposed method relieves gradient bias by estimating unobservable gradients with the pre-trained model.
We observe that gradient conflicts occur more often in GESTUR than ERM on only the experimental setup (OfficeHome w/ \texttt{RN50}), which is consistent with the performance in Table~\ref{tab:2_analysis_expert} where ERM outperforms GESTUR w/ TE.
This observation indicates that the domain generalization performance is affected by the gradient bias represented as the gradient conflicts in this analysis.
Additional analysis on the similarity of the true and estimated unobservable gradients is provided in Appendix~\ref{appendix:c_3_similarity}.

\subsection{Task-specific knowledge learned by the generalization expert}\label{subsec:3_5_tsk}
\input{Tables/4_analysis_task_specific_knowledge}
\paragraph{Setup.}
We conduct additional experiments to show that the feature extractor ${\mathbf{\theta}}^f_\text{GE}$ of GE learns task-specific knowledge successfully.
Linear probing that updates parameters of only the classifier while freezing those of the feature extractor is common practice for assessing representation quality.
We assume that the more task-specific knowledge the feature extractor learns, the better linear probing performance it exhibits in unseen domains targeting the same task.
In detail, we first train GE on source domains and then evaluate linear probing performance on an unseen domain with the trained feature extractor of GE.
We compare it with the case that a \textit{frozen} pre-trained model is used as the feature extractor.
For linear probing, we simply train a logistic regression classifier on the output feature representations of each feature extractor using the unseen domain only.
Note that, in this analysis, we use \texttt{CLIP} and \texttt{SWAG} which are pre-trained with objectives significantly different from the target task to demonstrate the effectiveness of the newly learned task-specific knowledge clearly.

\paragraph{Results.} As shown in Table~\ref{tab:4_analysis_task_specific_knowledge}, GE outperforms \textit{frozen} in all benchmark datasets except the one case where the two models reach the near 99\% performance. 
This shows that GE is learning task-specific knowledge further during training, which makes it a better approximation of the oracle model.
The result supports our claim that pre-trained models are not fully equipped with target task-specific knowledge, and injecting the knowledge further increases performance.

\subsection{Relationship between $\lambda$ and the size of the pre-trained model}\label{subsec:3_6_lambda}
\paragraph{Setup.}
\input{Tables/5_analysis_lambda}
Our proposed GESTUR controls the scale of the estimated unobservable gradients that reduce risks in unseen domains using the gradient scale factor $\lambda$.
To verify the effect of the scale factor, we observe the performance change varying the scale factor.

\paragraph{Results.}
In Table~\ref{tab:5_analysis_lambda}, \texttt{RN50} achieves the best performance with $\lambda = 0.01$.
On the other hand, the larger pre-trained models, \texttt{CLIP} and \texttt{SWAG} achieve the best performance with the relatively larger $\lambda = 0.1$ and $\lambda = 0.5$, respectively.
We summarize more results on other datasets (\textit{i.e.,} VLCS, OfficeHome, and TerraIncognita) in Appendix~\ref{appendix:c_1_lambda}, and they show the similar pattern as in PACS.

Intuitively, the larger pre-trained models act as a better approximation of the oracle model than the small one because they are likely to encounter various domains from the huge web-crawled datasets during pre-training.
They help to estimate unobservable gradients more accurately. 
The larger gradient scale factor, gradients ${\mathbf{g}}$ of TE is more affected by the estimated unobservable gradients $\Tilde{{\mathbf{g}}}_u$ while optimizing the model on source domains.
From this, we can conclude that the larger scale factor improves the generalization performance when larger pre-trained models are given.
\section{Related Work}\label{sec:4_related_work}
\subsection{Domain generalization}\label{subsec:4_1_dg}
\paragraph{Domain alignment.} 
Domain alignment is to learn domain-invariant feature representations by removing domain-specific knowledge in the representations.
Adversarial training is widely adopted to learn domain invariant features through a min-max game between a feature extractor and a domain discriminator~\citep{ganin2016domain, li2018deep, matsuura2020domain, zhu2022localized}.
On the other hand, several studies~\citep{muandet2013domain, sun2016deep, li2018domain} aim to minimize feature divergence across source domains.
Recently, contrastive learning-based algorithms~\citep{kim2021selfreg, yao2022pcl} have been proposed to minimize distances between feature representations of samples in the same class, regardless their domains.

\paragraph{Data augmentation.} Many studies have employed data augmentation techniques to improve domain generalization performance.
For example, \citet{gulrajani2021in} apply simple data augmentation techniques as a default setup in \textsc{DomainBed} and some studies~\citep{wang2020heterogeneous, xu2020adversarial, yan2020improve} utilize Mixup~\citep{zhang2017mixup}. 
Recently, a few works~\citep{zhou2021domain,nam2021reducing,kang2022style} focus on image style, based on the idea that domain gap is closely related to image style.
On the other side, some works on single domain generalization introduce adversarial data augmentation~\citep{volpi2018generalizing,fan2021adversarially, qiao2020learning} to generate hard samples adversarially while assuring their reliability.


\paragraph{Gradient-based.} Recently, several studies utilize gradients to build generalized models, especially by aligning gradients from different domains. 
\citet{mansilla2021domain} exploit gradient agreement for gradient surgery, based on the hypothesis that conflicting gradients contain domain-specific information.
\citet{shi2022gradient} propose a training method that maximizes inner product between source domain gradients to match optimization paths across domains.
Similarly, \citet{rame2022fishr} try to match domain-level Hessian to align loss landscapes across domains.
As another line of work, \citet{huang2020self} introduce the self-challenging algorithm that iteratively masks dominant features, which are selected by the scale of the gradients.

\paragraph{Meta-learning-based.} 
Since simulating domain shift by dividing source domains into meta-train and meta-test domains was first introduced in MLDG~\citep{li2018learning}, several approaches have been proposed in a similar setting.
For example, \citet{balaji2018metareg} propose to learn a regularizer for classifier weights and \citet{2112.12329} bring the idea of Reptile~\citep{nichol2018first} to MLDG to further increase performance with a multi-view framework.
On the other hand, \citet{zhang2021adaptive} employ meta-learning to adaptively predict model parameters from a batch of inputs.

\paragraph{Others.}
Some of the works bring concepts of causality~\citep{lv2022causality}, optimize the worst-case performance~\citep{sagawa2019distributionally, krueger2021out}, utilize text labels~\citep{min2022grounding}, or average model weights from different epochs~\citep{cha2021swad, arpit2022ensemble}.

Our work differs from aforementioned approaches in that we mainly concentrate on effectively utilizing large-scale pre-trained models.

\subsection{Domain generalization with pre-trained models}\label{subsec:4_2_dg_pt}
Recently, \citet{gulrajani2021in} empirically show that simple ERM~\citep{vapnik1999nature} outperforms most of early methods with pre-trained ResNet-50~\citep{he2016deep}.
\citet{yu2021empirical} show that using large-scale models pre-trained on massive datasets improves out-of-distribution performance.
\citet{kumar2022finetuning} find that fine-tuning distorts pre-trained features and propose the \textit{linear-probing then fine-tuning} to mitigate the feature distortion.
\citet{Wortsman_2022_CVPR} find that linearly interpolating the zero-shot and fine-tuned parameters of a pre-trained model improves performance in both source and unseen domains.
Although GESTUR's EMA (Equation~\ref{eq:2_6_ema}) looks similar to their interpolation, GESTUR updates the pre-trained model to inject task-specific knowledge.
\citet{li2022domain} propose a method to efficiently leverage a pool of large-scale pre-trained models through specialty-aware ensemble learning.
\citet{cha2022domain} propose MIRO, a regularization method that targets to minimize mutual information with pre-trained models which approximate the oracle model.
In this work, we share similar motivation with MIRO in that we initially approximate the oracle model with a large-scale pre-trained model.
However, we iteratively inject task-specific knowledge into the approximation of the oracle model, resulting in a better approximation.
\section{Conclusion and Future Work}\label{sec:5_conclusion}
In this paper, we propose a new domain generalization method that learns task-specific knowledge while preserving the generalization ability of large-scale pre-trained models.
We point out that gradient bias toward source domains hurts the generalization ability of pre-trained models during fine-tuning.
To alleviate the gradient bias, our proposed method estimates unobservable gradients that minimize risk in unseen domains based on two key components: a task expert and a generalization expert.
Experimental results on \textsc{DomainBed} show that our proposed method outperforms baseline methods in domain generalization.
Through extensive analyses, we also demonstrate that the estimated unobservable gradients effectively reduce gradient bias, thereby helping to learn task-specific knowledge without hurting the generalization power of large-scale pre-trained models.

Although we verify the effectiveness of our proposed method, it heavily relies on the capability of pre-trained models.
When unseen domains that pre-trained models did not encounter are given (\textit{e.g.} ResNet trained on ImageNet does not see medical images), the pre-trained models might not act as an approximation of the oracle model of the domains.
We will address this issue in future work.
\section*{Reproducibility Statement}\label{sec:6_repro}
We provide the source code for reproduction in the supplementary materials.
See Appendix~\ref{appedix:a_implementation_details} for the hyperparameters used for the experiments.
\section{Implementation Details}\label{appedix:a_implementation_details}
\paragraph{Hyperparameter search strategy.}
Similar to ~\citet{cha2022domain}, the hyperparameter tuning strategy differs depending on which pre-trained model is used.
In the experiments of this work, we use three different pre-trained models:
ResNet-50~\citep{he2016deep} pre-trained on ImageNet~\citep{5206848} (\texttt{RN50}), ViT-B/16~\citep{dosovitskiy2021an} with CLIP~\citep{radford2021learning} (\texttt{CLIP}), and RegNetY-16GF~\citep{radosavovic2020designing} with SWAG~\citep{Singh_2022_CVPR} (\texttt{SWAG}).

\input{Tables/appendix_hparams_resnet}
A two-stage hyperparameter search strategy is used for experiments with \texttt{RN50}.
Here, the batch size and the moving average coefficient ($m$) are fixed as 32 and 0.999 in the entire search procedure, respectively.
In the first stage, we search the gradient scale factor ($\lambda$) from \{0.01, 0.05, 0.1, 0.5\}.
In this stage, we fix the learning rate as 5e-5 and do not use weight decay and dropout (\textit{i.e.,} weight decay and dropout are equal to 0).
In the second stage, we fix $\lambda$ as the one searched in the first stage.
Then, we search the learning rate from \{1e-5, 3e-5, 5e-5\}, weight decay from \{0, 1e-6, 1e-4\}, and dropout from \{0, 0.1, 0.5\}.
We provide the hyperparameters we use for \texttt{RN50} in Table~\ref{tab:appendix_hparams_resnet}.

\input{Tables/appendix_hparams_clip_swag}
Unlike the experiments with \texttt{RN50}, we apply single-stage hyperparameter search strategy to \texttt{CLIP} and \texttt{SWAG}.
Here, we only search $\lambda$ from \{0.01, 0.05, 0.1, 0.5\} with hyperparameters such as the batch size, learning rate, weight decay, and dropout fixed.
In particular, we fix the learning rate, weight decay, and dropout as the first stage of the hyperparameter search of \texttt{RN50}.
For the batch size, we fix the batch size as 32 except for the experiments with DomainNet~\citep{peng2019moment} where the batch size is fixed as 24 for the experiments with \texttt{CLIP}. 
For \texttt{SWAG}, we fix the batch size as 16 for all experiments.
In Table~\ref{tab:appendix_hparams_clip_swag}, we show what $\lambda$ we use for \texttt{CLIP} and \texttt{SWAG}.
Similar to \citet{cha2022domain}, we fix the number of iterations as 15,000 for DomainNet and 5,000 for the others regardless of pre-trained models.
\section{Additional Results}\label{appedix:b_additional_results}
\subsection{Main results}\label{appendix:b_1_exp}
\input{Tables/appendix_additional_main_table}
In \S~\ref{subsec:3_2_results}, we only compare baselines superior to ERM~\citep{vapnik1999nature} with GESTUR for simplicity.
Here, we provide the entire results of the main experiment in Table~\ref{tab:appendix_main_table}.
\paragraph{Baselines.}
In the main experiment, we compare GESTUR against a number of baselines:
MMD~\citep{li2018domain}, MixStyle~\citep{zhou2021domain}, GroupDRO~\citep{sagawa2019distributionally}, IRM~\citep{arjovsky2019invariant}, ARM~\citep{zhang2021adaptive}, VREx~\citep{krueger2021out}, CDANN~\citep{li2018deep}, DANN~\citep{ganin2016domain}, RSC~\citep{huang2020self}, MTL~\citep{blanchard2021domain}, Mixup~\citep{wang2020heterogeneous,xu2020adversarial,yan2020improve}, MLDG~\citep{li2018learning}, Fish~\citep{shi2022gradient}, Fishr~\citep{rame2022fishr}, ERM~\citep{vapnik1999nature}, SagNet~\citep{nam2021reducing}, SelfReg~\citep{kim2021selfreg}, CORAL~\citep{sun2016deep}, mDSDI~\citep{bui2021exploiting}, GVRT~\citep{min2022grounding}, MIRO~\citep{cha2022domain},  SMA~\citep{arpit2022ensemble}, and SWAD~\citep{cha2021swad}.



\subsection{Relationship between $\lambda$ and the types of the pre-trained model}\label{appendix:c_1_lambda}
\input{Tables/appendix_lambda_vlcs}
\input{Tables/appendix_lambda_officehome}
\input{Tables/appendix_lambda_terrainc}
In \S~\ref{subsec:3_6_lambda}, we analyze the relationship between $\lambda$ and the size of the pre-trained model.
However, we only present the results from PACS~\citep{li2017deeper} in Table~\ref{tab:5_analysis_lambda} for simplicity.
Here, we provide the additional results from VLCS~\citep{fang2013unbiased}, OfficeHome~\citep{venkateswara2017deep}, and TerraIncognita~\citep{beery2018recognition} in Table~\ref{tab:appendix_lambda_vlcs}, Table~\ref{tab:appendix_lambda_officehome}, and Table~\ref{tab:appendix_lambda_terrainc}, respectively.






\section{Further Analysis}\label{appedix:c_further_analysis}



\subsection{Performance on source domains}\label{appendix:c_2_source_domain}
\input{Tables/appendix_source_domain.tex}
\paragraph{Setup.}
Domain generalization aims to improve the generalization performance on unseen domains shifted from source domains. 
Thus, domain generalization studies often do not consider situations where the source domains and the target domains are similar. 
To verify whether estimated unobservable gradients are useful when the unseen domains are similar to the source domains, we report the performance on the training-domain validation set, simulating the situations when the testing domains are exactly the same as the training domains. 

\paragraph{Results.}
The evaluation results are summarized in Table~\ref{tab:appendix_source_domain}. GESTUR w/ TE shows worse performance than ERM, indicating that the estimated unobservable gradients act as noisy gradients. 
Namely, gradients biased toward the source domains are more helpful than estimated unobservable gradients when the source domains and the target domains are similar. 
Nonetheless, GESTUR w/ GE performs better than ERM, demonstrating that our two-expert architecture is robust to various situations even when source domains and unseen domains are similar or not.



\subsection{Comparison with CLIP-based baselines}\label{appendix:b_3_zero_shot}
\input{Tables/appendix_clip}
\paragraph{Setup.}
CLIP~\citep{radford2021learning} is pre-trained on the huge web-crawled image-caption pair dataset and has been widely adopted in various computer vision tasks due to its generalization ability.
CLIP-based methods could be strong baselines in domain generalization because the text content they used in pre-training could act as a robust anchor to the domain shift of images.
Therefore, we conduct additional experiments using CLIP-based methods, CLIP Zero-shot and WiSE-FT~\citep{Wortsman_2022_CVPR}.
The CLIP-based methods require text-based queries to output text-based representations of target classes.
Following the previous study, we obtain the $80$ text-based queries from the official repository\footnote{\url{https://github.com/openai/CLIP/blob/main/notebooks/Prompt_Engineering_for_ImageNet.ipynb}} of CLIP and compute the final text-based representation of each target class by averaging the text-based representations of the queries.
Finally, the model predictions are computed with the text-based representations and the representations of input images.
For WiSE-FT, an ensemble of the fine-tuned and zero-shot models, we set the balance factor $\alpha$ as $0.5$ following its original paper since target unseen domains are inaccessible in the domain generalization setting.

\paragraph{Results.}
Table~\ref{tab:appendix_clip} shows the evaluation results where GESTUR achieves the best averaged performance.
In detail, GESTUR outperforms CLIP Zero-shot on VLCS, OfficeHome, and TerraIncognita, and shows comparable performance on PACS.
Likewise, GESTUR achieves better performance on PACS, OfficeHome, and TerraIncognita than WiSE-FT and comparable performance on VLCS.

Interestingly, the CLIP-based methods exhibit severe performance degradation on TerraIncognita.
We conjecture that their performance is sensitive to pre-defined text-based queries.
For example, the query ``a sketch of a \{\}" is helpful for the ``\underline{S}ketch" domain of PAC\underline{S}.
On the other hand, the queries ``a plastic \{\}" and ``a \{\} in a video game" are not helpful for TerraIncognita, which is composed of animal images taken from the wild.
These observations indicate that the CLIP-based methods require hard prompt engineering for each target dataset.
Moreover, the CLIP-based methods depend on language modality, which cannot be extended to other architecture or learning methods trained on only visual modality, such as \texttt{RN50} and \texttt{SWAG}.
Considering these, our GESTUR achieves a meaningful performance.



\subsection{Applicability of SWAD~\citep{cha2021swad} to GESTUR}\label{subsec:b_2_swad}
\input{Tables/appendix_swad.tex}
\paragraph{Setup.}
The recent studies~\citep{cha2021swad,cha2022domain} have observed that SWAD~\citep{cha2021swad} that seeks the flat minima is a good optimizer for domain generalization,  improving the generalization performance of several baselines by applying it to the baselines as a optimizer. 
Motivated by this observation, we evaluate the performance of our GESTUR applied with SWAD as a optimizer to verify whether GESTUR and SWAD are orthogonal directions to each other.
\paragraph{Results.}
Table~\ref{tab:appendix_swad} shows that SWAD does not improve the performance of GESTUR. 
We conjecture that it is because EMA used to transfer the knowledge of TE to GE has a similar effect as SWAD to find a flat minima by averaging the model’s weights.






\subsection{Similarity between true unobservable gradients ${\mathbf{g}}_u$ and estimated unobservable gradients $\tilde{{\mathbf{g}}}_u$ of GESTUR}\label{appendix:c_3_similarity}
\input{Figures/2_cosine_similarity}
\paragraph{Setup.}
In this paper, we argue that gradient bias is a major culprit in degrading domain generalization performance (Figure~\ref{fig:1_a_biased_gradients}) and our proposed method relieves the gradient bias by estimating unobservable gradients. 
To support this argument, we reported the number of iterations where gradient conflicts exist in Figure~\ref{fig:1_b_gradient_conflict} and Table~\ref{tab:3_analysis_conflict}. 
To examine whether the estimated unobservable gradients $\tilde{{\mathbf{g}}}_u$ are similar to the true unobservable gradients ${\mathbf{g}}_u$, we add the analysis calculating the cosine similarity of the true and estimated unobservable gradients.
Note that the true unobservable gradients are computed by cross-entropy loss using true labels of unseen domain datasets $\mathcal{D}_u$. 
On the other hand, the estimated unobservable gradients are just computed as the parameter difference between GE and TE ($\theta_{\text{GE}} - \theta_{\text{TE}})$.

\paragraph{Results.}
Figure~\ref{fig:2_cosine_similarity} shows that our estimated gradients display positive similarity scores with the true gradients. 
This trend demonstrates that the estimated gradients reduce the number of gradient conflicts, leading models to reduce the risks of unseen domains without accessing unseen domain data. 















\section*{Appendix}
\input{Sections/A_implementation_details}
\newpage
\input{Sections/B_additional_results}
\newpage
\input{Sections/C_further_analysis.tex}

\end{document}

