\section{Introduction}
\label{seq:intro}

In a continuous quest for better pre-training strategies, models based on image and language supervision have set impressive milestones, with CLIP~\citep{clip_icml21}, ALIGN~\citep{align_icml21} and Flamingo~\citep{flamingo_arxiv22} being leading examples.
Contrastively trained vision-language models consist of image and text encoders that align semantically-related concepts in a joint embedding space. 
Such models offer impressive zero-shot image classification by using the text encoder to generate classifier weights from arbitrarily newly defined category classes without relying on any visual data. In particular, the class name is used within a handcrafted \textit{prompt} template and then tokenized and encoded into the shared embedding space to generate new classifier weights. Rather than manually defining prompts, 
\citet{zhou2022learning} and \citet{lester2021power} proposed that prompts can be instead optimized in a data-driven manner through back-propagation by maximizing a cross-entropy loss on the downstream task. 
However, despite the performance improvement on downstream tasks, prompt learning negatively affects the generalization capability of the vision-language model.  

While subsequent works have focused on how to bridge the generalization gap, e.g. \citet{zhou2022conditional,zhu2022prompt}, in practice the generalization power of the foundation model is significantly degraded. Our work tackles this same problem, as it seeks to improve downstream performance but without degrading the generalization capability of the original model. 

To do so, we propose a data-driven method for directly learning the underlying distribution within the prompt space associated to the target concept. 
In particular, we frame prompt tuning as a variational inference problem, where a base learned prompt is combined with a residual vector sampled from the instance-specific underlying distribution. This formulation provides two advantages. First, it investigates the prompt space more thoroughly and results in more informative use of the language space, leading to better generalization. Second, it enables us to boost performance by capturing the uncertainty information in fine-grained classification problems.
The resulting approach is orthogonal to standard prompt learning approaches, being effective when combined with both standard~\citep{zhou2022learning} and conditional~\citep{zhou2022conditional} approaches. In fact, when combined with the conditional approach, our method maintains the gains on the seen classes provided by the conditional method while simultaneously matching or even surpassing the generalization capability on unseen classes of the original vision-language model.

In summary, our contributions in this paper are as follows:
\begin{enumerate}
\item We propose a variational framework that is capable of capturing the general or instance specific distribution within the prompt space. Since generalization is obtained through transfer from the language space, we obtain better generalization capability.
\item We show that the proposed approach is orthogonal to recent developments, and can be successfully combined with both standard and conditional prompt learning variants.
\item We empirically show that our proposed method improves performance and provides better generalization, leading to state-of-the-art accuracy in 24 out of 28 standard benchmarks set forth by prior work, surpassing \cocoop by 1.6\% average Top-1 accuracy.
\end{enumerate}

\section{Related works}
\label{sec:related}


\textbf{Prompt learning in NLP.} Prompt learning was originally proposed within the NLP domain, following the appearance of foundation models such as GPT-3~\citep{brown2020language}. 
Early prompt learning methods constructed prompts by combining words in the language space such that the model would perform better on downstream evaluation \citep{shin2020autoprompt,jiang2020can}.
Subsequent methods, e.g. \cite{li2021prefix,lester2021power}, prepend a set of learnable prompts to the input of a frozen model and optimize through back-propagation, which allows better flexibility than using existing words, at the cost of leading to prompts that do not correspond to an actual phrase.
Instead, \citet{he2022hyperprompt} focus on a multi-task scenario and use a HyperNetwork to conditionally generate task-specific and layer-specific prompts that are pre-pended to the values and keys inside the self-attention layers of a the frozen model. Within the NLP domain, prompt learning has also been shown to work better than in-context learning~\citep{liu2022few}.



\textbf{Prompting in Vision and Language models.} Research on prompt learning for vision-language models have been largely inspired by prior work within NLP. Similar to e.g. \citet{li2021prefix}, \coop \citep{zhou2022learning} proposes a prompt learning method that optimizes unified or class specific prompts in the continuous space through back-propagation. While \coop obtains good accuracy on downstream tasks, it negatively affects the generalization ability to new unseen classes. \cocoop \citep{zhou2022conditional} extends \coop and partially bridges the generalization gap by generating instance-specific prompt residuals through a conditioning mechanism dependent on the visual data. ProGrad~\citep{zhu2022prompt} shares the same goal as \cocoop of bridging the generalization gap, but instead proposes to match the gradient of the prompt to the general knowledge of the CLIP model to prevent prompt tuning from forgetting the general knowledge learned from the foundation model. Alternative directions consist of test-time prompt tuning~\citep{testtimept_neurips22}, where consistency across multiple views is used as the supervisory signal, and unsupervised prompt learning~\citep{unsup_prompt22}, where a pseudo-labelling strategy is proposed instead to obtain the labels needed to drive the prompt learning.
Perhaps the most similar work to ours is~\citet{lu2022prompt}. 
In this work, 
the authors use an ensemble of prompts and model their distribution within the language embedding space, 
with optimization seeking to minimize the negative log-likelihood with respect to the corresponding visual embedding. 
Unlike ours, 
their method relies on hand-crafted rules 
to define the prompt ensemble, thus still relying on the effectiveness of hand-crafted designs. The number of learnable prompts is also pre-defined, potentially offering sub-optimal coverage of an NLP concept.
Finally, 
it is not clear how to apply their strategy within the context of conditional prompt learning. 
We believe that modelling the input prompt space rather than relying on a fixed number of templates is a more powerful and flexible approach.
We provide empirical evidence of the superiority of our approach in the experiments.

While beyond our current scope, it is worth noting that prompt learning has been applied to a wider range of problems and scenarios, which highlights its power and flexibility.
Among them are important topics such as unsupervised domain adaptation~\citep{ge2022domain}, multi-label classification~\citep{sun2022dualcoop}, video classification~\citep{ju2021prompting}, object detection~\citep{detpro_cvpr22,promptdet_eccv22} or pixel-level labelling~\citep{rao2022denseclip}. Finally, prompt learning as a means to adapt pre-trained models has also been applied to purely vision models~\citep{jia2022visual,sandler2022fine} providing similar performance to fine tuning the whole model but with great parameter efficiency.







\section{Method}

\begin{figure*}[t!]
\centering
\includegraphics[width=.9\textwidth,clip,trim=0cm 0cm 0cm 0.4cm]{images/variational_prompt.pdf}
\caption{\textbf{Overview of variational prompt tuning}. For each input image $\mathbf{x}$, we use image features $f(\mathbf{x})$ to infer the mean $\mu(\mathbf{x})$ and standard deviation $\Sigma(\mathbf{x})$ of the residual distribution using metanet $\pi_\phi$. The prompts to generate the classifier weights are constructed by summing up learnable prompts $\textbf{p}$ and residual samples from the residual distribution. The obtained prompts are fed through a text encoder $g(\textbf{t})$, and the classifier weights are estimated. Finally, the cosine similarity scores are computed between the image features $f(\mathbf{x})$ and the classifier weights.}
\vspace{-3mm}
\label{fig:overall-idea}
\end{figure*}

\subsection{Background}

\textbf{Contrastive Language-Image Pretraining (CLIP)}~\citep{clip_icml21} consists of an \textit{image encoder} $f(\mathbf{x})$ and \textit{text encoder} $g(\mathbf{t})$, each producing a $d$-dimensional ($L_2$ normalized) embedding from an arbitrary image $\mathbf{x} \in \mathbb{R}^{3 \times H \times W}$, and word embeddings $\mathbf{t} \in \mathbb{R}^{L \times e}$, with $L$ representing the text length and $e$ the embedding dimension\footnote{In CLIP the word embedding is learned together with the text encoder. A \textit{tokenizer} is used to convert the text into one-hot vectors, or tokens, that can be directly mapped into the word embeddings. For the sake of clarity we refer indistinctly to words and word embeddings.}. Both encoders are trained together using a contrastive loss from a large-scale dataset composed of paired images and captions. Once trained, CLIP can be used for zero-shot $C$-class image classification by generating each of the $c$ classifier weights $\mathbf{w}_c$ as the $d$-dimensional text encoding $g(\mathbf{t}_c)$. Here $\mathbf{t}_c$ results from adding the class-specific word embedding $\mathbf{e}_c$ to a pre-defined prompt $\mathbf{p} \in \mathbb{R}^{L-1 \times e}$, i.e., $\mathbf{w}_c {=} g(\mathbf{t}_c)$ with $\mathbf{t}_c {=} \{\mathbf{p},\mathbf{e}_c\}$. The prompt $\mathbf{p}$ is manually crafted to capture the semantic meaning of the downstream task, e.g., $\mathbf{t}_c = ``\texttt{An image of a \{class\}}"$. The probability of image $\mathbf{x}$ being classified as $y \in \{1...C\}$ is thus defined as 
$p(y|\mathbf{x}) {=} \frac{e^{f(\mathbf{x})^T\mathbf{w}_y}}{\sum_{c}^C e^{f(\mathbf{x})^T \mathbf{w}_{c}}}$. 

\textbf{Context Optimization (CoOp)}~\citep{zhou2022learning} provides a learned alternative to manually defining prompts. CoOp learns a fixed prompt from a few annotated samples. The prompt is designed as a learnable embedding matrix $\mathbf{p} \in \mathbb{R}^{L \times e}$ which is updated via back-propagating the classification error through the frozen CLIP model. Specifically, for a set of $N$ annotated meta-training samples $\{\mathbf{x}_i, y_i\}_{i=1}^N$, the prompt $\mathbf{p}$ is obtained by minimizing the cross-entropy loss, as:
\begin{equation}
\label{eq:coop}
   \mathbf{p}^* = \arg \min_\mathbf{p} \mathbb{E}_{\mathbf{x}_i,y_i} [-\log p(y_i | \mathbf{x}_i, \mathbf{p})].
\end{equation}
Note that this approach, while resembling that of common meta-learning approaches, can still be deployed in a zero-shot scenario provided that for new classes the classification weights will be given by the text encoder. Although this approach allows to generalize to new tasks with few training iterations, learning a fixed prompt is sensitive to domain shifts between the annotated samples and the test set. 


\textbf{Conditional Prompt Learning (CoCoOp)}~\citep{zhou2022conditional} attempts to overcome domain shifts by learning an instance-specific continuous prompt that is conditioned on the input image. To ease the training of a conditional prompt generator, \cocoop defines each conditional token in a residual way, with a task-specific, learnable set of tokens $\mathbf{p}$ and a residual vector that is conditioned on the input image. Assuming $\mathbf{p}$ to be composed of $L$ learnable tokens $\mathbf{p} {=} [\mathbf{p}_1, \mathbf{p}_2, ... , \mathbf{p}_L]$, the residual vector ${\mathbf{r}}(\mathbf{x}) {=} \pi_\phi(f(\mathbf{x})) \in \mathbb{R}^e$ is produced by a small neural network $\pi_\phi$ with as input the image features $f(\mathbf{x})$. The new prompt is then computed as $\mathbf{p}(\mathbf{x}) {=} [\mathbf{p}_1 + {\mathbf{r}}(\mathbf{x}), \mathbf{p}_2 + {\mathbf{r}}(\mathbf{x}) , ... , \mathbf{p}_L + {\mathbf{r}}(\mathbf{x})]$. The training now comprises learning the task-specific prompt $\mathbf{p}$ and the parameters $\phi$ of the neural network $\pi_\phi$. Defining the context-specific text embedding $\mathbf{t}_c(\mathbf{x}) {=} \{\mathbf{p}(\mathbf{x}), \mathbf{e}_c \}$, and $p(y | \mathbf{x})$ as :
\begin{equation}
\label{eq:prob_y}
    p(y | \mathbf{x}) = \frac{ e^{ f(\mathbf{x})^T g( \mathbf{t}_c(\mathbf{x}) )} }{ \sum_{c}^C e^{ f(\mathbf{x})^T g( \mathbf{t}_{c}(\mathbf{x}) ) } },
\end{equation}
the learning is formulated as:
\begin{equation}
    \label{eq:cocoop}
   \mathbf{p}^*, \phi^* = \arg \min_{\mathbf{p}, \phi} \mathbb{E}_{\mathbf{x}_i,y_i} [-\log p(y_i | \mathbf{x}_i, \mathbf{p}, \phi)].
\end{equation}
While \cocoop achieves state-of-the-art results in a large variety of downstream tasks, it is still prone to the domain shift problem, considering that $\pi_\phi$ provides a deterministic residual vector from the image features $f(\mathbf{x})$ which are expected to be domain-specific.


\textbf{Prompt Distribution Learning (ProDA)}~\citep{lu2022prompt} is work concurrent to \cocoop that focuses learning a distribution of prompts that generalize to a broader set of tasks. ProDA proposes to learn a collection of prompts $\mathbf{P}{=}\{\mathbf{p}_k\}_{k=1}^\mathcal{K}$ that can be used to subsequently generate an \textit{a posteriori} distribution of the classifier weights for each of the target classes. For a given mini-batch of $K$ sampled prompts $\mathbf{p}_k \sim \mathbf{P}$, the classifier weights $\mathbf{w}_c$ are sampled from the posterior distribution $\mathcal{N}(\mu_{\mathbf{w}_{1:C}}, \Sigma_{\mathbf{w}_{1:C}})$, with mean $\mu_{\mathbf{w}_{1:C}}$ and covariance $\Sigma_{\mathbf{w}_{1:C}}$ computed from the collection $\{\mathbf{w}_{k,c} = g(\mathbf{t}_{k,c})\}_{c=1:C, k=1:K}$, with $\mathbf{t}_{k,c} = \{\mathbf{p}_k, \mathbf{e}_c\}$. The objective is now formulated as: 
\begin{equation}
\label{eq:variational_coop}
   \mathbf{P}^* = \arg \min_{\mathbf{P}} \mathbb{E}_{\mathbf{x}_i,y_i} [-\log \mathbb{E}_{\mathbf{w}_l \sim \mathcal{N}(\mu_{\mathbf{w}_{1:C}},\Sigma_{\mathbf{w}_{1:C}})}\, p(y_i | \mathbf{x}_i, \mathbf{w}_l)].
\end{equation}
Computing $\mathbb{E}_{\mathbf{w}_l} \, p(y_i | \mathbf{x}_i, \mathbf{w}_l)]$ is intractable and an upper bound to Eq.~\ref{eq:variational_coop} is derived. During inference, the classifier weights are set to those given by the predictive mean $\mathbf{w}_c = \mu_{\mathbf{w}_{1:C}}$, computed across the collection of learned prompts $\mathbf{P}$. While showing promising results compared to CoOp, how to combine it with the conditional prompt learning framework of CoCoOp is unclear.



\subsection{Variational Prompt Tuning}
\label{ssec:method}

In this paper, we propose to model the input prompt space in a probabilistic manner, as an \textit{a priori, instance-specific} distribution. In particular, we define a distribution $p_\gamma$ over the prompts $\mathbf{p}$ that is instance-specific, i.e. $\mathbf{p} \sim p_\gamma(\mathbf{x})$. To this end, we assume that $\mathbf{p}$ can be split into a fixed set of prompts $\mathbf{p}_i$ and an instance specific residual vector ${\mathbf{r}}$ that act as a latent variable over $\mathbf{p}$. The instance-specific prompt is then defined as:
 \begin{equation}
\label{eq:new_prompt}
\mathbf{p}_\gamma(\mathbf{x}) = [\mathbf{p}_1 + {\mathbf{r}}_\gamma, \mathbf{p}_2 + {\mathbf{r}}_\gamma, \cdots, \mathbf{p}_L + {\mathbf{r}}_\gamma], \, \, {\mathbf{r}}_\gamma \sim p_\gamma(\mathbf{x}),
\end{equation}
 where $p_\gamma(\mathbf{x})$ refers to the real posterior distribution over ${\mathbf{r}}$ conditioned on the observed features $\mathbf{x}$. Denoting the class-specific input as $\mathbf{t}_{c,\gamma}(\mathbf{x}) {=} \{\mathbf{p}_\gamma(\mathbf{x}), \mathbf{e}_c\}$, the marginal likelihood $p(y | \mathbf{x})$ is now defined as:
\begin{equation}
     \label{eq:probabilistic}
 p(y | \mathbf{x}) = \int_\gamma \frac{ e^{ f(\mathbf{x})^T g( \mathbf{t}_{c,\gamma}(\mathbf{x}) )} }{ \sum_{c'} e^{ f(\mathbf{x})^T g( \mathbf{t}_{c',\gamma}(\mathbf{x}) ) } } p(\mathbf{p}_\gamma(\mathbf{x})) d \gamma.
\end{equation}
Solving Eq.~\ref{eq:cocoop} with the marginal likelihood defined as in Eq.~\ref{eq:probabilistic} is intractable, as it requires computing $p_\gamma({\mathbf{r}} | \mathbf{x}) p_\gamma(\mathbf{x})$. Instead, we resort to deriving a lower bound, by introducing a variational posterior distribution $\pi_\phi(\mathbf{x})$ from which the residual ${\mathbf{r}}_\gamma$ can be sampled. The variational bound is defined as:
\begin{equation}
\label{eq:elbo}
\log p(y | \mathbf{x}) \geq \mathbb{E}_{\pi_\phi({\mathbf{r}} | \mathbf{x})} [\log p(y | \mathbf{x}, {\mathbf{r}})] - D_{\mathrm{KL}}\big[\pi_{\phi}(\mathbf{r} | \mathbf{z}) \| p_{\gamma}(\mathbf{r})\big],
\end{equation}
with $p(y | \mathbf{x}, {\mathbf{r}}) \propto e^{ f(\mathbf{x})^T g( \mathbf{t}_{c,\gamma}(\mathbf{x}) )}$, where the dependency on ${\mathbf{r}}$ comes through the definition of $\mathbf{t}_{c,\gamma}$. The variational posterior distribution $\pi_\phi$ plays a role akin to the metanet in CoCoOp. We thus refer to it as metanet in the following to align terminology. Following standard variational optimization practices~\citep{kingma2013auto,gordon2018metalearning}, we define $\pi_\phi$ as a Gaussian distribution conditioned on the input image features $\mathbf{x}$, as ${\mathbf{r}}(\mathbf{x}) \sim \mathcal{N}( \mu(\mathbf{x}), \Sigma(\mathbf{x} ))$, with $\mu$ and $\Sigma$ parameterized by two linear layers placed on top of the metanet $\pi_\phi$ (see Figure~\ref{fig:overall-idea}). The prior $p_\gamma({\mathbf{r}})$ is defined as $\mathcal{N}(\mathbf{0},\mathbf{I})$, and we make use of the reparameterization trick to generate Monte-Carlo samples from $\pi_\phi$ to maximize the right side of Eq.~\ref{eq:elbo}. 
The optimization of Eq.~\ref{eq:elbo} comprises learning the prompt embeddings $\{\mathbf{p}_i\}_{i=1}^L$ as well as the parameters of the metanet $\pi_\phi$ and the linear layers parameterizing $\mu$ and $\Sigma$. Note that this adds little complexity as it requires learning $\mathbf{p}$ and $\pi_\phi$, given that $\mu$ and $\Sigma$ are defined as two linear layers on top of $\pi_\phi$.


\textbf{Inference.} At test time, $K$ residuals are sampled from the conditional distribution $\pi_\phi(\mathbf{x})$, which are used to generate $K$ different prompts per class $\mathbf{p}_{k} {=} [\mathbf{p}_1 + {\mathbf{r}}_k, \mathbf{p}_2 + {\mathbf{r}}_k, \cdots, \mathbf{p}_L + {\mathbf{r}}_k]$. Each prompt is prepended to the class-specific embedding to generate a series of $K$ separate classifier weights $\mathbf{w}_{k,c}$. We then compute $p(y {=} c \,| \, \mathbf{x}) {=} (1/K) \sum_{k=1}^K p(y {=} c \, | \, \mathbf{x}, \mathbf{w}_{k,c})$ and select $\hat{c} {=} \arg \max_{c} p(y {=} c \, | \, \mathbf{x})$ as the predicted class. It is worth noting that because the posterior distribution is generated by the text encoder, it is not expected that for $K \rightarrow \infty$, $(1/K) \sum_{k} p( y {=} c \, | \, \mathbf{x}, \mathbf{w}_{c,k}) \rightarrow p( y {=} c | \mathbf{x}, g(\{\mu(\mathbf{x}), \mathbf{e}_c\})$, meaning that sampling at inference time remains relevant. We study the dependency on the number of samples in the ablations. 

Notably, our framework can be also used with the unconditional setting of CoOp, by simply removing the dependency of the input image from the latent distribution. In such scenario, we keep a fixed set of prompt embeddings and learn a global latent distribution $p_\gamma$ over the residual vectors ${\mathbf{r}}$. In this case, $p_\gamma$ is a general distribution learned during training with no dependency on the input sample $\mathbf{x}$. We show that CoOp with our proposed approach improves generalization on new classes. 
















\section{Experiments and Results}
\label{sec:experiments}

\subsection{Experimental setup}

We follow the exact experimental setup of CoCoOp~\citep{zhou2022conditional}, which is currently the state-of-the-art approach for prompt tuning for vision-language models. We describe the setup in the following for completeness. 

\textbf{Three tasks and fifteen datasets.} We evaluate variational prompt tuning for three different tasks: \textit{base-to-new generalization}, \textit{cross-dataset transfer}, and \textit{domain generalization}. For base-to-new generalization and cross-dataset transfer tasks,
we rely on the same 11 image recognition datasets as \citet{zhou2022learning,zhou2022conditional}. These include generic image classification datasets (ImageNet by~\citep{deng2009imagenet} and Caltech101 by~\citep{fei2004learning}), fine-grained classification datasets (OxfordPets by~\citep{parkhi2012cats}, StanfordCars by~\citep{krause20133d}, Flowers102 by~\citep{nilsback2008automated}, Food101 by~\citep{bossard2014food} and FGVCAircraft by~\citep{maji2013fine}), scene recognition (SUN397 by~\citep{xiao2010sun}), action recognition (UCF101 by~\citep{soomro2012dataset}), texture classification (DTD by~\citep{cimpoi2014describing}), and satellite imagery recognition (EuroSAT by~\citep{helber2019eurosat}). For the domain generalization task, we first train our model on  ImageNet and report our evaluation metric on ImageNetV2~\citep{recht2019imagenet}, ImageNet-Sketch~\citep{wang2019learning}, ImageNet-A~\citep{hendrycks2021natural}, and ImageNet-R~\citep{hendrycks2021many}. 

\textbf{Evaluation metrics.} We report average accuracy and harmonic mean $H{=}2\times(\text{base}\times\text{new})/(\text{base}+\text{new})$ \citep{xian2017zero} for the base-to-new generalization tasks. For cross-dataset transfer learning and domain adaptation, we provide average accuracy results.

\textbf{Baselines.} We compare against zero-shot CLIP~\citep{clip_icml21}, \coop \citep{zhou2022learning}, \cocoop \citep{zhou2022conditional}, and ProDA~\citep{lu2022prompt}. 
For zero-shot CLIP, CoOp, CoCoOp, all results are adopted from~\citep{zhou2022conditional}, and we reproduce all results for ProDA.

\textbf{Implementation details.}
Our variational prompt tuning contains three sub-networks: an image encoder $f(\mathbf{x})$, a text encoder $g(\mathbf{t})$, and a metanet $\pi_{\phi}$. The image encoder $f(\mathbf{x})$ and text encoder $g(\mathbf{t})$ are a ViT-B/16 \citep{dosovitskiy2020image} and transformer \citep{vaswani2017attention}, which are initialized with CLIP's pre-trained weights and kept frozen during training, as in \citet{zhou2022learning, zhou2022conditional}. 
The metanet $\pi_{\phi}$ consists of two linear layers followed by ELU activation function as trunk and two linear heads on top to estimate the $\mu$ and $\Sigma$ of the residual distribution. For each task and dataset,
we optimize the number of samples $K$ and epochs. Other hyper-parameters as well as the training pipeline in terms of few-shot task definitions are identical to \citet{zhou2022learning, zhou2022conditional} (see table~\ref{tab:shared_hparams} and~\ref{tab:dataset_hparams} in the appendix). Implementation code will be released.


\subsection{Base-to-new generalization} 
\textbf{Setup.} We report the few-shot generalization of our method on $11$ datasets for three different random seeds. Each dataset is divided into two disjoint subsets: base classes and new classes. We train our method on base classes and evaluate it on both base and new classes. For a fair comparison, we follow \citet{zhou2022conditional, zhou2022conditional} in terms of dataset split and number of shots (see Table~\ref{tab:shared_hparams} appendix.)

\textbf{Results.} From the results in Table~\ref{tab:base-to-new},
the lack of generalization capability of the \coop approach is evidenced by the considerable discrepancy between the base and new classes' accuracy. This is expected as \coop only observes a small number of training samples to adapt the CLIP model for downstream tasks, resulting in overfitting. Adding variational prompt tuning to \coop reduces overfitting to base classes, improving the performance of new classes by $11.54\%$ and the harmonic mean by $1.7\%$ across all datasets, at the expense of a base classes accuracy drop by $10.7\%$. Figure~\ref{fig:delta-harmonic-mean}(a) depicts the relative improvement of our proposed strategy when compared to \coop in terms of the harmonic mean, where we observe an improvement in 10 out of 11 datasets. The limited generalization capability of \coop is mitigated by \cocoop by exploiting instance-conditional prompts, which improves the accuracy on new classes from $63.22\%$ to $72.23\%$. Nonetheless, augmenting \cocoop with variational prompts still improves performance on the new classes and the harmonic mean by $3.25\%$ and $1.6\%$, respectively, with a mere decrease of $0.37\%$ in base accuracy. Figure~\ref{fig:delta-harmonic-mean}(b) shows per-dataset relative harmonic mean improvement. We observe an improvement in 10 out of 11 datasets. Note that the drop in the average base accuracy for \cocoopvpt is negligible compared with \coopvpt. Moreover, our best performing model \cocoopvpt performs better than ProDA~\citep{lu2022prompt} in terms of new classes accuracy and harmonic mean by $\%2.64$ and $\%0.78$. It is also worth noting that \coopvpt and \cocoopvpt excel at CLIP zero-shot performance in base, new and harmonic accuracies. 


\begin{table*}[t!]
\centering
\caption{\textbf{Base-to-new generalization} comparison between the state-of-the-art and variational prompt tuning. We average our accuracy over three random seeds. Our proposed model is trained on a few-shot training set (base) and then evaluated on held-out classes (new). As shown, \coop and \cocoop overfit on base classes and do not provide good generalization on new classes. However, our model provides better generalization performance on new classes as well as harmonic mean.}
\vspace{-2mm}
\resizebox{\linewidth}{!}{%
\begin{tabular}{lc|c|ccc|ccc|c}
\toprule
\textbf{Dataset} & 
 & 
\textbf{CLIP} &
\textbf{\coop} & 
\textbf{+VPT} & 
\textbf{$\Delta$} & 
\textbf{\cocoop} & 
\textbf{+VPT} &
\textbf{$\Delta$} & 
\textbf{ProDA} \\ \midrule
\multirow{3}{*}{Average}       & Base      & 69.34  & 82.66   & 71.98  &  \color{red}{$-$10.71}            & 80.47  & 80.10  &  \color{red}{$-$00.37} & 81.56\\
                               & New       & 74.22  & 63.22   & 74.76  &  \color{blue}{$+$11.54}           & 71.69  & 74.94  &  \color{blue}{$+$03.25} & 72.30\\
                               & H         & 71.69  & 71.65   & 73.34  &  \textbf{\color{blue}{$+$01.69}}  & 75.83  & 77.43  &  \textbf{\color{blue}{$+$01.60}} & 76.65\\
\midrule
\multirow{3}{*}{ImageNet}      & Base      & 72.43  & 76.14   &  74.73 &  \color{red}{$-$01.41}            & 75.98  & 76.00 &  \color{blue}{$+$00.02} & 75.40\\ 
                               & New       & 68.14  & 67.88   &  70.60 &  \color{blue}{$+$02.72}           & 70.43  & 70.93 &  \color{blue}{$+$00.50} & 70.23\\ 
                               & H         & 70.21  & 71.77   &  72.60 &  \textbf{\color{blue}{$+$00.83}}  & 73.10  & 73.37 &  \textbf{\color{blue}{$+$00.27}} & 72.72\\
\midrule
\multirow{3}{*}{Caltech101}    & Base      & 96.84  & 98.00   &  95.47 &  \color{red}{$-$02.53}            &  97.96 & 98.00 &  \color{blue}{$+$00.04} & 98.27\\
                               & New       & 94.00  & 89.81   &  93.80 &  \color{blue}{$+$03.99}           &  93.81 & 94.93 &  \color{blue}{$+$01.12} & 93.23\\
                               & H         & 95.39  & 93.72   &  94.62 &  \textbf{\color{blue}{$+$00.90}}  &  95.84 & 96.44 &  \textbf{\color{blue}{$+$00.60}} & 95.68\\
\midrule
\multirow{3}{*}{OxfordPets}    & Base      & 91.17  & 93.67   & 90.77  &  \color{red}{$-$02.90}            & 95.20  & 95.67 &  \color{blue}{$+$00.47} & 95.43\\
                               & New       & 97.26  & 95.29   & 97.83  &  \color{blue}{$+$02.54}           & 97.69  & 98.00 &  \color{blue}{$+$00.31} & 97.83\\
                               & H         & 94.11  & 94.47   & 94.16  &  \color{red}{$-$00.31}            & 96.43  & 96.82 &  \textbf{\color{blue}{$+$00.39}} & 96.62\\
\midrule
\multirow{3}{*}{StanfordCars}  & Base      & 63.37 & 78.12    & 65.27  &  \color{red}{$-$12.85}            & 70.49  & 72.93 &  \color{blue}{$+$02.44} & 74.70\\
                               & New       & 74.89 & 60.40    & 75.97  &  \color{blue}{$+$15.57}           & 73.59  & 73.23 &  \color{red}{$-$00.36} & 71.20\\
                               & H         & 68.65 & 68.12    & 70.21  &  \textbf{\color{blue}{$+$02.09}}  & 72.01  & 73.07 &  \textbf{\color{blue}{$+$01.06}} & 72.91\\
\midrule
\multirow{3}{*}{Flowers102}    & Base      & 72.08 & 97.60    & 72.97  &  \color{red}{$-$24.63}            & 94.87  & 95.70 &  \color{blue}{$+$00.83} & 97.70\\
                               & New       & 77.80 & 59.67    & 75.90  &  \color{blue}{$+$16.23}           & 71.75  & 70.40 &  \color{red}{$-$01.35} & 68.68\\
                               & H         & 74.83 & 74.06    & 74.40  &  \textbf{\color{blue}{$+$00.34}}  & 81.71  & 81.12 &  \textbf{\color{red}{$-$00.59}} & 80.66\\
\midrule
\multirow{3}{*}{Food101}       & Base      & 90.10 & 88.33    & 90.37  &  \color{blue}{$+$02.04}           & 90.70  & 91.03 &  \color{blue}{$+$00.33} & 90.30\\
                               & New       & 91.22 & 82.26    & 91.67  &  \color{blue}{$+$09.41}           & 91.29  & 92.13 &  \color{blue}{$+$00.84} & 88.57\\
                               & H         & 90.65 & 85.18    & 91.01  &  \textbf{\color{blue}{$+$05.83}}  & 90.99  & 91.57 &  \textbf{\color{blue}{$+$00.58}} & 89.43\\
\midrule
\multirow{3}{*}{FGVCAircraft}  & Base      & 27.19 & 40.44    & 29.57  &  \color{red}{$-$10.87}            & 33.41  & 34.40 &  \color{red}{$-$00.99} & 36.90\\
                               & New       & 36.29 & 22.30    & 33.80  &  \color{blue}{$+$11.50}           & 23.71  & 35.00 &  \color{blue}{$+$11.29} & 34.13\\
                               & H         & 31.08 & 28.74    & 31.54  &  \textbf{\color{blue}{$+$02.80}}  & 27.74  & 34.69 &  \textbf{\color{blue}{$+$06.95}} & 35.46\\
\midrule
\multirow{3}{*}{SUN397}        & Base      & 69.36 & 80.60    &  73.77  & \color{red}{$-$06.83}            & 79.74  & 79.17 &  \color{red}{$-$00.57} & 78.67\\
                               & New       & 75.35 & 65.89    &  77.90  & \color{blue}{$+$12.01}           & 76.86  & 77.87 &  \color{blue}{$+$01.01} & 76.93\\
                               & H         & 72.23 & 72.50    &  75.77  & \textbf{\color{blue}{$+$03.27}}  & 78.27  & 78.51 &  \textbf{\color{blue}{$+$00.24}} & 77.79\\
\midrule
\multirow{3}{*}{DTD}           & Base      & 53.24 & 79.44    & 57.67   & \color{red}{$-$21.77}            & 77.01  & 75.30 &  \color{red}{$-$01.71} & 80.67\\
                               & New       & 59.90 & 41.18    & 58.70   & \color{blue}{$+$17.52}           & 56.00  & 60.80 &  \color{blue}{$+$04.80} & 56.48\\
                               & H         & 56.37 & 54.24    & 58.18   & \textbf{\color{blue}{$+$03.94}}  & 64.85  & 67.27 &  \textbf{\color{blue}{$+$02.42}} & 66.44\\
\midrule
\multirow{3}{*}{EuroSAT}       & Base      & 56.48 & 92.19    & 67.97   & \color{red}{$-$24.22}            & 87.49  & 80.30 &  \color{red}{$-$07.19} & 83.90\\
                               & New       & 64.05 & 54.74    & 71.63   & \color{blue}{$+$16.89}           & 60.04  & 75.30 &  \color{blue}{$+$15.26} & 66.00\\
                               & H         & 60.02 & 68.69    & 69.75   & \textbf{\color{blue}{$+$01.06}}  & 71.21  & 77.71 &  \textbf{\color{blue}{$+$06.50}} & 73.88\\
\midrule
\multirow{3}{*}{UCF101}        & Base      & 70.53  & 84.69   & 73.23   & \color{red}{$-$11.46}            & 82.33  & 82.53 &  \color{blue}{$+$00.20} & 85.23\\
                               & New       & 77.50  & 56.05   & 74.63   & \color{blue}{$+$18.58}           & 73.45  & 75.77 &  \color{blue}{$+$02.32} & 71.97\\
                               & H         & 73.85  & 67.45   & 73.92   & \textbf{\color{blue}{$+$06.47}}  & 77.64  & 79.00 &  \textbf{\color{blue}{$+$01.36}} & 78.04\\
\bottomrule
\end{tabular}%
}
\label{tab:base-to-new}
\vspace{-5mm}
\end{table*}

\begin{figure}[t!]
\centering
\begin{subfigure}{.5\linewidth}
  \centering
  \includegraphics[width=1\linewidth]{images/coop_delta.pdf}
  \caption{$\Delta(\text{H})=\text{H}(\text{CoOp+VPT}) - \text{H}(\text{CoOp})$}
  \label{fig:delta-harmonic-mean-coop}
\end{subfigure}%
\begin{subfigure}{.5\linewidth}
  \centering
  \includegraphics[width=1\linewidth]{images/cocoop_delta.pdf}
  \caption{$\Delta(\text{H})=\text{H}(\text{CoCoOp+VPT}) - \text{H}(\text{CoCoOp})$}
  \label{fig:delta-harmonic-mean-cocoop}
\end{subfigure}
\caption{\textbf{Relative enhancement of variational prompt tuning} over \coop and \cocoop in terms of harmonic mean over 11 datasets for 3 distinct random seeds. Variational prompt tuning improves the harmonic mean for all baselines other than OxfordPets for \coop and Flowers102 for CoCoOp.}
\vspace{-3mm}
\label{fig:delta-harmonic-mean}
\end{figure}



\subsection{Cross-dataset transfer learning}

\textbf{Setup.} For cross-dataset transfer learning, the model is trained on a source dataset (ImageNet) and then assessed on 10 distinct target datasets. This experiment tries to determine how effectively our methods generalizes transfer beyond the scope of a single dataset. 

\textbf{Results.} As reported in Table~\ref{tab:cross-dataset-transfer}, \coopvpt has a drop in performance in ImageNet by $1.76\%$ while outperforming the target dataset on average by $1.63\%$. Moreover, our proposed method leads to an increase on 9 out of 10 target datasets, with a small drop in accuracy of $0.03\%$ in Caltech101. Note that on target datasets such as FGVCAircraft and DTD, our proposed method achieves an improvement of more than $3\%$. 
Similarly to CoOp, augmenting \cocoop with our method still leads to an overall performance enhancement of $0.16\%$ on 7 out of 10 target datasets, showing its effectiveness for cross-dataset transfer learning. In addition, unlike \cocoop, which has better performance in ImageNet-like datasets such as Caltech101 and OxfordPets, our proposed method exhibits improvement on dissimilar datasets (e.g. FGVCAircraft, DTD, and EuroSAT), demonstrating its capacity to capture the unique characteristics of each dataset.

\begin{table*}[htbp]
\centering
\caption{\textbf{Cross-dataset transfer learning} comparison between the state-of-the-art and our variational prompt tuning in terms of average accuracy on three different random seeds. Following~\citep{zhou2022conditional}, our proposed model is trained on a source dataset and evaluated on target datasets. As shown, variational prompt tuning performs better than other baselines 16 out of 20 datasets, although it loses performance on the source dataset.}
\vspace{-2mm}
\resizebox{1\linewidth}{!}{%
\begin{tabular}{lcccccccccccc}
\toprule
 & 
\multicolumn{1}{c}{Source} & 
\multicolumn{10}{c}{Target}\\  \cmidrule(lr){2-2} \cmidrule(lr){3-13} 
\textbf{Methods} & \textbf{\rot{ImageNet}} & \textbf{\rot{Caltech101}} & \textbf{\rot{OxfordPets}} & \textbf{\rot{StanfordCars}} & \textbf{\rot{Flowers102}} & \textbf{\rot{Food101}} & \textbf{\rot{FGVCAircraft}} & \textbf{\rot{SUN397}} & \textbf{\rot{DTD}} & \textbf{\rot{EuroSAT}} & \textbf{\rot{UCF101}} & \textbf{\rot{Average}} \\ \midrule
\coop       & 71.51  & 93.70 & 89.14  & 64.51   & 68.71  & 85.30  & 18.47  & 64.15  & 41.92  & 46.39  & 66.55  & 63.88\\
+VPT        & 69.73  & 93.67 & 89.27  & 65.50   & 70.20  & 86.27  & 22.13  & 66.57  & 46.93  & 47.43  & 67.2  & 65.51  \\
\rowcolor{Gray}
$\Delta$   &\color{red}{$-$1.78} & \color{red}{$-$0.03} & \color{blue}{$+$0.13} & \color{blue}{$+$0.99} & \color{blue}{$+$1.49} & \color{blue}{$+$0.97} & \color{blue}{$+$3.66} & \color{blue}{$+$2.42} & \color{blue}{$+$5.01} & \color{blue}{$+$1.04} & \color{blue}{$+$0.65} & \textbf{\color{blue}{$+$1.63}} 
\\\midrule
\cocoop     & 71.02  & 94.43 & 90.14  & 65.32   & 71.88  & 86.06  & 22.94  & 67.36  & 45.73  & 45.37  & 68.21  & 65.74\\
+VPT        & 70.70   & 93.67 & 90.63  & 65.00   & 70.90  & 86.30  & 24.93  & 67.47  & 46.10  & 45.87  & 68.67  & 65.95\\
\rowcolor{Gray}
$\Delta$    & \color{red}{$-$0.32} & \color{red}{$-$0.76} & \color{blue}{$+$0.49} & \color{red}{$-$0.32} & \color{red}{$-$0.98} & \color{blue}{$+$0.24} & \color{blue}{$+$1.99} & \color{blue}{$+$0.11} & \color{blue}{$+$0.37} & \color{blue}{$+$0.50} & \color{blue}{$+$0.46} & \textbf{\color{blue}{$+$0.16}}\\
\bottomrule
\end{tabular}%
}
\label{tab:cross-dataset-transfer}
\vspace{-3mm}
\end{table*}

\subsection{Domain generalization}
\textbf{Setup.} Lastly, We examine variational prompt tuning through the lens of distribution shift and robustness. We train our proposed model on the source dataset (ImageNet) for three different random seeds, and assess it on ImageNetV2, ImageNet-Sketch, ImageNet-A, and ImageNet-R. Prior work such as \coop \citep{zhou2022learning} and \cocoop \citep{zhou2022conditional} demonstrate empirically that learning a soft-prompt improves the model's resilience against distribution shift and adversarial attack. Following their experiments, we are also interested in determining if treating prompts in a variational manner maintain or improve the performance. 

\textbf{Results.} As reported in Table~\ref{tab:generalization} our method enhances the accuracy of \coop on ImageNet-Sketch, ImageNet-A, and ImageNet-R by $0.88\%$, $0.70\%$, and $2.19\%$ while degrading the performance on ImageNet and ImageNetV2 by $1.78\%$, by $1.03\%$. However, on CoCoOp, adding VPT, while losing the performance on source dataset similar to CoOp, consistently improves the accuracy on all target datasets which highlight the effectiveness of our proposed method. 



\begin{table*}[htbp]
\centering
\caption{\textbf{Domain generalization} comparison between the state-of-the-art and variational prompt tuning in terms of average accuracy on three different random seeds. Our proposed model is trained on a source dataset and evaluated on target classes. Variational prompt tuning outperforms alternative baselines on the target datasets while losing performance on the source dataset.}
\vspace{-2mm}
\resizebox{1\linewidth}{!}{%
\begin{tabular}{lcccccc}
\toprule
 & 
 &
\multicolumn{1}{c}{Source} & 
\multicolumn{4}{c}{Target}\\  \cmidrule(lr){3-3} \cmidrule(lr){4-7} 
 \textbf{Methods} & \textbf{Learnable} & \textbf{ImageNet} & \textbf{ImageNetV2} & \textbf{ImageNet-Sketch} & \textbf{ImageNet-A} & \textbf{ImageNet-R} \\ \midrule
CLIP       & \xmark & 66.73 & 60.83 & 46.15 & 47.77 & 73.96 \\\midrule
\coop       & \cmark & 71.51 & 64.20	& 47.99	& 49.71	& 75.21 \\
\rowcolor{Gray}
+VPT       & \cmark & 69.73 & 63.17 & 48.87 & 50.77 & 77.40 \\
$\Delta$   &    -   & \color{red}{$-$1.78} & \color{red}{$-$1.03} & \color{blue}{$+$0.88}  & \color{blue}{$+$0.70}  & \color{blue}{$+$2.19}  \\\midrule
\cocoop       & \cmark & 71.02 & 64.07 & 48.75 & 50.63 & 76.18 \\
\rowcolor{Gray}
+VPT       & \cmark & 70.70	& 64.23	& 49.20	& 51.33	& 77.00 \\
$\Delta$   &    -   & \color{red}{$-$0.32} & \color{blue}{$+$0.16}  & \color{blue}{$+$0.45}  & \color{blue}{$+$0.70}  & \color{blue}{$+$0.82}  \\
\bottomrule
\end{tabular}%
}

\label{tab:generalization}
\vspace{-3mm}
\end{table*}



\subsection{Ablations}
\label{ablation}



\textbf{Effectiveness of the posterior distribution $q_\phi$.}
We first ablate the effectiveness of the variational posterior distribution. To do this, we consider sampling one residual vector from the uniform distribution 
$\mathcal{U}(\text{0},\text{1})$, normal distribution $\mathcal{N}(\text{0},\text{I})$, normal distribution $\mathcal{N}( \mu(\mathbf{x}), \text{0})$, normal distribution $\mathcal{N}( \mu(\mathbf{x}), \Sigma(\mathbf{x} ))$, and report the new class accuracy for \cocoopvpt for one random seed in Table~\ref{tab:ablation-posterior}. 
Except for the EuroSAT dataset, a sample from the normal distribution $\mathcal{N}( \mu(\mathbf{x}), \text{0})$ obtains the best performance in comparison with alternatives, showing that the mean of the normal distribution $\mu(\mathbf{x})$ is the most effective sample. 
In addition, we find that drawing one sample from $\mathcal{N}( \mu(\mathbf{x}), \Sigma(\mathbf{x} ))$ yields superior results compared to drawing one sample from uniform distribution $\mathcal{U}(\text{0},\text{1})$ and normal distribution $\mathcal{N}(\text{0},\text{I})$, further demonstrating the efficacy of our proposed method in capturing the underlying distribution of the prompt space.
We also ablate increasing the number of samples from the normal distribution $\mathcal{N}( \mu(\mathbf{x}), \Sigma(\mathbf{x} ))$ to understand the informativeness of the learned variational distribution. It is shown that enlarging the number of samples further improves the model performance as they capture the prompt space appropriately. 

\begin{table*}[htbp]
\centering
\caption{\textbf{Effectiveness of the posterior distribution}.  The informative posterior distribution $\mathcal{N}( \mu(\mathbf{x}), \Sigma(\mathbf{x} ))$ outperforms the two uninformative distributions  $\mathcal{U}(\text{0},\text{1})$ and $\mathcal{N}(\text{0},\text{I})$ by a large margin for all datasets. Increasing the number of samples further improves results.
}
\vspace{-2mm}
\resizebox{0.95\linewidth}{!}{%
\begin{tabular}{lcccccc}
\toprule
 \textbf{Methods} & \textbf{Samples} & \textbf{DTD} & \textbf{Flowers102} & \textbf{EuroSAT} & \textbf{FGVCAircraft} & \textbf{UCF101} \\ \midrule
$\mathcal{U}(\text{0},\text{1})$                       & 1   & 33.20   & 45.30    & 54.20 & 10.50 & 55.70 \\
$\mathcal{N}(\text{0},\text{I})$                        & 1   & 26.60   & 36.10	& 50.00	&  07.70	& 48.80 \\
$\mathcal{N}( \mu(\mathbf{x}), 0)$               & 1   & 59.80   & 73.50    & 59.90 & 34.10 & 76.50 \\
$\mathcal{N}( \mu(\mathbf{x}), \Sigma(\mathbf{x} ))$     & 1   & 56.40   & 72.30	& 64.50	& 33.00	& 75.60 \\
\midrule
$\mathcal{N}( \mu(\mathbf{x}), \Sigma(\mathbf{x} ))$     & 2   & 60.00   & 73.90	& 67.40	& 33.90	& 76.20 \\
$\mathcal{N}( \mu(\mathbf{x}), \Sigma(\mathbf{x} ))$     & 5   & 62.20   & 74.00	& 71.00	& 34.20	& 76.60 \\
$\mathcal{N}( \mu(\mathbf{x}), \Sigma(\mathbf{x} ))$     & 10  & 61.60   & 73.50	& 73.60	& 34.40	& 77.00 \\

\bottomrule
\end{tabular}%
}
\label{tab:ablation-posterior}
\vspace{-2mm}
\end{table*}



\textbf{Prompt initialization.}
We investigate the effectiveness of the prompt initialization on the new class accuracy for \cocoopvpt for one random seed. We consider two variants. In the first one, we initialize the context tokens randomly using a normal distribution, whereas in second one we initialize the context tokens with ``An image of a $\{\texttt{class}\}$''. Table~\ref{tab:ablation-prompt-init} summarizes this ablation. Comparing the two variants demonstrates that an appropriately initialized prompt consistently outperforms a randomly initialized prompt, highlighting the necessity for further research of the prompt space.

\begin{table*}[htbp]
\centering
\caption{\textbf{Prompt initialization.} Initializing the context tokens with an appropriate prompt ``An image of a $\{\texttt{class}\}$'' improves the performance compared to random tokens.}
\vspace{-2mm}
\resizebox{0.9\linewidth}{!}{%
\begin{tabular}{lccccc}
\toprule
\textbf{Method} & \textbf{DTD} & \textbf{Flowers102} & \textbf{EuroSAT} & \textbf{FGVCAircraft} & \textbf{UCF101} \\ 
\midrule
Random                             &  74.10& 67.23& 75.10 & 34.30 & 81.30 \\
``An image of a $\{\texttt{class}\}$''     &  74.80& 70.05& 77.90 & 35.50 & 82.50 \\
\bottomrule
\end{tabular}%
}
\label{tab:ablation-prompt-init}
\end{table*}



\begin{wrapfigure}{r}{0.4\textwidth}
   
  \begin{center}
    \vspace{-0.7cm}
    \includegraphics[width=0.4\textwidth]{images/iclr-context-MC-new.pdf}
    \vspace{-0.7cm}
  \end{center}
  \caption{\textbf{Effectiveness of Monte Carlo samples} on new accuracy. As demonstrated, increasing the number of Monte Carlo samples boosts performance initially but reaches a plateau after 10 samples for all datasets.\vspace{-1cm}}
  \label{fig:MC-new}
  
\end{wrapfigure}
\textbf{Number of Monte-Carlo samples.}
When approximating the log-likelihood of input data, the number of Monte Carlo samples is an important hyperparameter. Generally, a large number of samples should lead to a better approximation and better classification accuracy. We ablate this hyperparameter on new accuracy for \cocoopvpt by varying the number of Monte Carlo samples at inference time. We show results for a varying number of samples in Figure~\ref{fig:MC-new} for DTD, Flowers102, EuroSAT, FGVCAircraft, and UCF101. Increasing the Monte Carlo samples from $1$ to $10$  consistently improves the new accuracy,  afterwards the accuracy saturates. Hence, we recommend evaluating variational prompt tuning on a larger number of Monte Carlo samples for better model accuracy. 

For more ablations, see Section \ref{more_ablation} in the appendix.

\section{Conclusion}
In this paper, we introduce variational prompt tuning, a probabilistic modeling of the underlying distribution of prompts, allowing prompts within the support of an associated concept to be derived through stochastic sampling. 
By doing so, we are able to generate adaptive context tokens for each data point. 
This formulation leads to better generalization capabilities in terms of the new accuracy and harmonic mean for downstream tasks.  
We show that it can be seamlessly integrated into both standard and conditional prompt learning frameworks, considerably improving the performance in both cases.
We conduct extensive experiments on 15 datasets and demonstrate the benefits of a variational formulation in learning data-driven prompts. Our method provides the current state-of-the-art for prompt learning, and constitutes, to the best of our knowledge, the first method for CLIP adaptation that fully maintains the generalization capability to new classes of the original model.






\section*{Ethics Statement}
Our method has the potential to affect applications that often need rapid adaptation, such as medical imaging, astronomical imaging, and autonomous driving.
Because of this, there may be negative societal consequences associated with the adoption of our technology. For example, a lack of fairness with models trained on insufficient data, regulatory compliance, patient privacy in medical imaging, and more general biases encoded into large vision-language models such as CLIP.

\section*{Reproducibility Statement}
Details on benchmarks, metrics, and the implementation of variational prompt tuning in terms of architecture and training are contained in Section \ref{sec:experiments}. 
We also list of all hyperparameters in Table~\ref{tab:shared_hparams} and~\ref{tab:dataset_hparams}.
Lastly, we will release the source code, scripts to reproduce the results and evaluating the performance of the model at: \url{https://github.com/<redacted>/<redacted>}.
      

