\section{Introduction}
\glsresetall

\begin{figure*}[!ht]
     \centering
     \begin{subfigure}[c]{0.05\textwidth}
         \centering
         \caption{}
         \label{fig:effective_number}
     \end{subfigure}\hfill
     \begin{subfigure}[c]{0.95\textwidth}
         \centering
         \includegraphics[height=0.14\paperheight]{.//effective_number}
     \end{subfigure}
     \begin{subfigure}[c]{0.05\textwidth}
         \centering
         \caption{}
         \label{fig:nonlinear}
     \end{subfigure}\hfill
     \begin{subfigure}[c]{0.95\textwidth}
         \centering
         \includegraphics[height=0.14\paperheight]{.//nonlinear}
     \end{subfigure}
     \begin{subfigure}[c]{0.05\textwidth}
         \centering
         \caption{}
         \label{fig:correlations}
     \end{subfigure}\hfill
     \begin{subfigure}[c]{0.95\textwidth}
         \centering
         \includegraphics[height=0.14\paperheight]{.//correlations}
     \end{subfigure}
     \caption{
       (a) The Vendi Score, \acrshort{VS} in the figure, can be interpreted as the effective number of unique elements in a sample. It increases linearly with the number of modes in the dataset. IntDiv, the expected dissimilarity, becomes less sensitive as the number of modes increases, converging to 1.
      (b) Combining distinct similarity functions can increase the Vendi Score, as should be expected of a diversity metric, while leaving IntDiv unchanged.
      (c) IntDiv does not take into account correlations between features, but the Vendi Score does. The Vendi Score is highest when the items in the sample differ in many attributes, and the attributes are not correlated with each other.
     }
        \label{fig:properties}
\end{figure*}
 
Diversity is a criterion that is sought after in many areas of \gls{ML}, from dataset curation 
and generative modeling to reinforcement learning, active learning, and decoding algorithms. A lack of diversity in datasets and 
models can hinder the usefulness of \gls{ML} in many critical applications, e.g. scientific discovery. 
It is therefore important to be able to measure diversity.
 
\parhead{Are we measuring diversity correctly in \gls{ML}?} Several existing metrics for diversity rely on a reference distribution or dataset, which is limiting. 
These reference-based metrics define diversity in terms of coverage of the reference.
They assume access to an embedding function--such as a pretrained Inception model~\citep{szegedy2016rethinking}--that maps samples to real-valued vectors.
One example of a reference-based metric is \gls{FID}~\citep{heusel2017gans}, which measures the Wasserstein-2 distance between two Gaussian distributions, one Gaussian fit to the embeddings of the reference sample and another one fit to the embeddings of the sample to be evaluated for diversity.
\gls{FID} was originally proposed for evaluating image \glspl{GAN} but has since been applied to text~\citep{cifka2018eval} and molecules~\citep{preuer2018frechet} using domain-specific neural network encoders.
~\citet{sajjadi2018assessing} proposed a two-metric evaluation paradigm using precision and recall, with precision measuring quality and recall measuring diversity in terms of coverage of the reference distribution. Several other variations of precision and recall have been proposed~\citep{kynkaanniemi2019improved,simon2019revisiting,naeem2020reliable}. Compared to these approaches, the Vendi Score can measure diversity without relying on a reference distribution or dataset and is therefore more general. 

Some other existing metrics evaluate diversity using a pre-trained classifier, therefore requiring labeled datasets.
For example, the \gls{IS}~\citep{Salimans2016}, which is mainly used to evaluate the perceptual quality of image generative models, 
evaluates diversity using the entropy of the marginal distribution of class labels predicted by an ImageNet classifier.
Another example is \gls{NOM}~\citep{Srivastava2017}, a metric used to evaluate the diversity of \glspl{GAN}. 
\gls{NOM} is calculated by using a classifier trained on a labeled dataset and then counting the number of unique labels predicted by the classifier when 
using samples from a \gls{GAN} as input. 
Both \gls{IS} and \gls{NOM} define diversity in terms of predefined labels, and therefore require knowledge of the ground truth labels and a separate classifier, which limits their applicability.

In some discrete domains, diversity is often evaluated in terms of the distribution of unique features. 
For example in \gls{NLP}, a standard metric is n-gram diversity, which is defined as the number of distinct n-grams divided by the total number of n-grams~\citep[e.g.][]{li2016diversity}.
These metrics require an explicit, discrete feature representation.

There are proposed metrics that use similarity scores to define diversity. The most widely used metric of this form is the average pairwise similarity score or the complement, the average dissimilarity.
In text, variants of this metric include \acrshort{PAIRWISE-BLEU}~\citep{shen2019mixture} and \acrshort{D-LEX-SIM}~\citep{fomicheva2020unsupervised}, in which the similarity function is an n-gram overlap metric such as \acrshort{BLEU}~\citep{papineni2002bleu}.
In biology, average dissimilarity is known as IntDiv~\citep{benhenda2017chemgan}, with similarity defined as the Jaccard (Tanimoto) similarity between molecular fingerprints.
Average similarity has some shortcomings, which we highlight in \Cref{fig:properties}. 
The figure shows the similarity matrices induced by a shape similarity function and/or a color similarity function. Each entry in the matrices shown is defined to be $1$ if the column and the row share both 
shape and color, $0.5$ if they share either shape or color, and $0$ otherwise. As shown in \Cref{fig:properties}, the average similarity--IntDiv in the figure--becomes less sensitive as diversity increases and does not account for correlations between features. This is not the case for the Vendi Score, which accounts for correlations between features and is able to capture the increased diversity resulting from composing two orthogonal similarity functions. 

Some other diversity metrics in the \gls{ML} literature fall outside of these categories.
The Birthday Paradox Test~\citep{Arora2018} aims to estimate the size of the support of a generative model, but requires some manual inspection of samples.
\acrshort{GILBO}~\citep{alemi2018gilbo} is a reference-free metric but is only applicable to latent variable generative models.

As discussed above, several attempts have been made to measure diversity in \gls{ML}. However, the proposed metrics can be limited in their applicability in that they require a reference dataset or predefined labels, or are domain-specific and applicable to one class of models. Besides, these metrics don't give their user the agency to decide which type of diversity they want to measure. The existing metrics that don't have those applicability limitations have serious shortcomings when it comes to capturing diversity that we've illustrated in \Cref{fig:properties} and will further demonstrate in our experiments in \Cref{sec:empirical}.   

\parhead{Contributions.} We make several contributions in this paper, which we summarize as follows:
\begin{itemize}
	\item We extend ecological diversity to \gls{ML}, and propose the Vendi Score, a metric for evaluating diversity in \gls{ML}. 
	We study the properties of the Vendi Score, which provides us with a more formal understanding of desiderata for diversity.  
	\item We showcase the flexibility and wide applicability of the Vendi Score and highlight the shortcomings of existing metrics used to measure diversity in different domains. 
	\item With the Vendi Score, we are now able to measure the diversity of datasets in \gls{ML}, which opens the door for diversity-informed data augmentation.  
\end{itemize}
\parhead{Outline.} In \Cref{sec:method} we define the Vendi Score, state its properties, and discuss its computational complexity. We then apply 
it to several datasets, models, and decoding algorithms, in different domains, in \Cref{sec:empirical}. Finally, we conclude and discuss future work in \Cref{sec:discussion}

\section{Measuring Diversity with the Vendi Score}
\label{sec:method}

We now define the Vendi Score, state its properties, and study its computational complexity. (We relegate all proofs of lemmas and theorems to the appendix.)

\subsection{Defining the Vendi Score}

To define a diversity metric in \gls{ML} we look to ecology, the field that centers diversity in its work.
In ecology, one main way diversity is defined is as the exponential of the entropy of the distribution of the species under study~\citep{jost2006entropy}. This is a reasonable index for diversity. Consider a population with a uniform distribution over $N$ species, with entropy $\log(N)$. This population has maximal ecological diversity $N$, the same diversity as a population with $N$ members, each belonging to a different species. The ecological diversity decreases as the distribution over the species becomes less uniform, and is minimized and equal to one when all members of the population belong to the same species. 
 
How can we extend this way of thinking about diversity to \gls{ML}? One naive approach is to define diversity as the exponential of the Shannon entropy of the probability distribution defined by a machine learning model or dataset. However, this approach is limiting in that it requires a probability distribution for which entropy is tractable, which is not possible in many \gls{ML} settings. We would like to define a diversity metric that only relies on the samples being evaluated for diversity. And we would like for such a metric to achieve its maximum value when all samples are dissimilar and its minimum value when all samples are the same. This implies the need to define a similarity function over the samples. Endowed with such a similarity function, we can define a form of entropy that only relies on the samples to be evaluated for diversity. This leads us to the Vendi Score: 

\begin{definition}[Vendi Score]\label{def:vendi_score}
  Let $x_1, \ldots, x_n \in {\mathcal{X}}$ denote a collection of samples, let $k: {\mathcal{X}} \times {\mathcal{X}} \to {\mathbb{R}}$ be a positive semidefinite similarity function, with $k(x, x) = 1$ for all $x$, and let ${\bm{K}} \in {\mathbb{R}}^{n \times n}$ denote the kernel matrix with entry $K_{i, j} = k(x_i, x_j)$. Denote by $\lambda_1, \ldots, \lambda_n$ the eigenvalues of ${\bm{K}}/n$. The Vendi Score (${{\text{VS}}}$) is defined as the exponential of the Shannon entropy of the eigenvalues of ${\bm{K}}/n$:
  \begin{align}\label{eq:vendi-score}
    {{\text{VS}}}_k(x_1, \ldots, x_n) =  \exp\left( - \sum_{i=1}^S \lambda_i \log \lambda_i \right), 
  \end{align}
where we use the convention $0 \log 0 = 0$.
\end{definition}
To understand the validity of the Vendi Score as a mathematical object, consider the following lemma:
\begin{lemma}\label{lemma:eigen}
  Let ${\bm{K}} \in {\mathbb{R}}^{n \times n}$ denote a positive semi-definite kernel matrix with $K_{i, i} = 1$ for $i \in \{1, \ldots, n\}$. 
  Denote by $\Delta_n$ the $(n-1)$-dimensional simplex; its elements are vectors of dimension $n$ whose entries are nonnegative and sum to one. Let ${\bm{\lambda}} = (\lambda_1, \ldots, \lambda_n)$ denote the vector of eigenvalues of ${\bm{K}}/n$. 
  Then ${\bm{\lambda}} \in \Delta_n$.
\end{lemma}

The lemma implies that the eigenvalues of ${\bm{K}}/n$ as defined in \Cref{def:vendi_score} are nonnegative and sum to one. The Shannon entropy is therefore well-defined and the Vendi Score is well-defined. 
In this form, the Vendi Score can also be seen as the \emph{effective rank} of the kernel matrix ${\bm{K}}/n$. Effective rank was introduced by~\citet{roy2007effective} in the context of signal processing; the effective rank of a matrix is defined as the exponential of the entropy of the normalized singular values.
Effective rank has also been used in machine learning, for example, to evaluate word embeddings~\citep{torregrossa2020correlation} and to study the implicit bias of gradient descent for low-rank solutions~\citep{arora2019implicit}.

The Vendi Score can be expressed directly as a function of the kernel similarity matrix ${\bm{K}}$:
\begin{lemma}\label{lemma:von_neumann}
Consider the same setting as Definition~\ref{def:vendi_score}. Then 
  \begin{align}
    {{\text{VS}}}_k(x_1, \ldots, x_n) =  \exp \left (-\tr \left(\frac{{\bm{K}}}{n} \log \frac{{\bm{K}}}{n} \right) \right )
    .
  \end{align}
\end{lemma}
The lemma makes explicit the connection of the Vendi Score to quantum statistical mechanics: the Vendi Score is equal to the exponential of the von Neumann entropy associated with ${\bm{K}}/n$~\citep{bach2022information}.
In quantum statistical mechanics, the state of a quantum system is described by a \emph{density matrix}, often denoted $\rho$.
The von Neumann entropy of $\rho$ quantifies the uncertainty in the state of the system~\citep{wilde2013quantum}. 
The normalized similarity matrix ${\bm{K}}/n$ here plays the role of the density matrix. 

Our formulation of the Vendi Score assumes that $x_1, \ldots, x_n$ were sampled independently, and so $p(x_i) \approx \frac{1}{n}$ for all $i$. 
This is the usual i.i.d setting in \gls{ML} and the setting we study in our experiments. However, we can generalize the Vendi Score to a setting in which 
we have an explicit probability distribution over the sample space. 
\begin{definition}[Probability-Weighted Vendi Score]\label{def:weighted_vendi_score}
  Let ${\bm{p}} \in \Delta_n$ denote a probability distribution on a discrete space ${\mathcal{X}} = \{x_1, \ldots, x_n\}$, where $\Delta_n$ denotes the $(n-1)$-dimensional simplex. 
  Let $k: {\mathcal{X}} \times {\mathcal{X}} \to {\mathbb{R}}$ be a positive semidefinite similarity function, with $k(x, x) = 1$ for all $x$. Let ${\bm{K}} \in {\mathbb{R}}^{n \times n}$ denote the kernel matrix with $K_{i, j} = k(x_i, x_j)$.
Let $\tilde{{\bm{K}}}_{{\bm{p}}} = \mathrm{diag}(\sqrt{{\bm{p}}}){\bm{K}}\mathrm{diag}(\sqrt{{\bm{p}}})$
denote the probability-weighted kernel matrix. Let $\lambda_1, \ldots, \lambda_n$ denote the eigenvalues of $\tilde{{\bm{K}}}_{{\bm{p}}}$. 
  The Vendi Score (${{\text{VS}}}$) is defined as the exponential of the Shannon entropy of the eigenvalues of $\tilde{{\bm{K}}}_{{\bm{p}}}$:
  \begin{align}\label{eq:def-general}
    {{\text{VS}}}_k(x_1, \ldots, x_n, {\bm{p}}) =  \exp\left( - \sum_{i=1}^S \lambda_i \log \lambda_i \right).
  \end{align}
\end{definition}
This is the same definition as \Cref{eq:vendi-score}, with the uniform normalization of the similarity matrix ${\bm{K}}$ replaced by normalization with the probability distribution ${\bm{p}}$. 
When all elements in the sample are completely dissimilar, the probability-weighted Vendi Score defined in~\Cref{def:weighted_vendi_score} reduces to the exponential of the Shannon entropy of the weighting distribution:
\begin{lemma}\label{lemma:dissimilar}
Let ${\bm{p}} \in \Delta_n$ be a probability distribution over $x_1, \ldots, x_n$ and suppose $k(x_i, x_j) = 0$ for all $i \neq j$. Then the Vendi Score is the exponential of the Shannon entropy of ${\bm{p}}$, 
   \begin{itemize}
      	\item[] \centering ${{\text{VS}}}_k(x_1, \ldots, x_n, {\bm{p}}) = \exp\left(H({\bm{p}})\right)$.
      \end{itemize}
\end{lemma}

\subsection{Understanding the Vendi Score}
Figure~\ref{fig:properties} illustrates the behavior of the Vendi Score on simple toy datasets in which each element is defined by a shape and a color, and similarity is 
defined to be $1$ if elements share both shape and color, $0.5$ if they share either shape or color, and $0$ otherwise.

First, Figure~\ref{fig:effective_number} illustrates that the Vendi Score is an \textit{effective number}, and can be understood as the effective number of dissimilar elements in a sample.
The value of measuring diversity with effective numbers has been argued in ecology~\citep[e.g.][]{hill1973diversity,patil1982diversity,jost2006entropy} and economics~\citep{adelman1969comment}.
Effective numbers provide a consistent basis for interpreting diversity scores, and make it possible to compare diversity scores using ratios and percentages.
For example, in Figure~\ref{fig:effective_number}, when the number of modes doubles from two to four, the Vendi Score doubles as well. 
If we doubled the number of modes from four to eight, the Vendi Score would double once again. 

Figures~\ref{fig:nonlinear} and~\ref{fig:correlations} illustrate another strength of the Vendi Score, which is that it accounts for correlations between features. 
Given distinct similarity functions $k$ and $k'$, the Vendi Score calculated using the combined similarity function $\frac{1}{2}k(x) + \frac{1}{2}k'(x)$ can be 
greater than the average of the individual Vendi Scores if the two similarity functions describe distinct dimensions of variation. 
Furthermore, the Vendi Score increases when the items in the sample differ in more attributes, and the attributes become less correlated with each other.

The Vendi Score has several desirable properties as a diversity metric. We summarize them in the following theorem.
\begin{theorem}[Properties of the Vendi Score]\label{thm:properties}
  Consider the same definitions in \Cref{def:vendi_score} and \Cref{def:weighted_vendi_score}.
  \begin{enumerate}
    \item \textbf{Effective number.} If $k(x_i, x_j) = 0$ for all $i \neq j$, then ${{\text{VS}}}_k(x_1, \ldots, x_n)$ is maximized and equal to $n$. If $k(x_i, x_j) = 1$ for all $i, j$, then ${{\text{VS}}}_k(x_1, \ldots, x_n)$ is minimized and equal to $1$.
    \item \textbf{Identical elements.} Suppose $k(x_i, x_j) = 1$ for some $i \neq j$.
      Let ${\bm{p}}'$ denote the probability distribution created by combining $i$ and $j$, i.e. $p'_i = p_i + p_j$ and $p'_j = 0$. Then the Vendi Score is unchanged,
      \begin{itemize}
      	\item[] \centering ${{\text{VS}}}_k(x_1, \ldots, x_n, {\bm{p}}) = {{\text{VS}}}_{k}(x_1, \ldots, x_n, {\bm{p}}')$.
      \end{itemize}
      \item \textbf{Partitioning.} Suppose $S_1, \ldots, S_m$ are collections of samples such that, for any $i \neq j$, for all $x \in S_i, x' \in S_j$, $k(x, x') = 0$.
      Then the diversity of the combined samples depends only on the diversities of $S_1, \ldots, S_m$ and their relative sizes.
      In particular, if $p_i = |S_i|/\sum_{j}|S_j|$ is the relative size of $S_i$ and $H(p_1, \ldots, p_m)$ denotes the Shannon entropy, then the Vendi Score is the geometric mean,
      \begin{itemize}
      	\item[] \centering ${{\text{VS}}}_k(S_1, \ldots, S_m) = \exp(H(p_1, \ldots, p_m))\prod_{i=1}^m {{\text{VS}}}_k(S_i)^{p_i}.$
      \end{itemize}
    \item \textbf{Symmetry.} If $\pi_1, \ldots, \pi_n$ is a permutation of $1, \ldots, n$, then 
     \begin{itemize}
      	\item[] \centering ${{\text{VS}}}_k(x_1, \ldots, x_n) = {{\text{VS}}}_k(x_{\pi_1}, \ldots, x_{\pi_n})$.
      \end{itemize}
  \end{enumerate}
\end{theorem}

The effective number property provides a consistent frame of reference for interpreting the Vendi Score: a sample with a Vendi Score of $m$ can be understood to be as diverse as a sample consisting of $m$ completely dissimilar elements.
The identical elements property provides some justification for our use of a sampling approximation: for example, calculating the empirical Vendi Score of a sample of $90$ blue diamonds and $10$ yellow squares is equivalent to calculating the probability-weighted Vendi Score of a sample of one blue spade and one yellow square, with ${\bm{p}} = (0.9, 0.1)$.
The partitioning property is analogous to the partitioning property of the Shannon entropy and means that if two samples are completely dissimilar we can calculate the diversity of the union of the samples using only the diversity of each sample independently and their relative sizes.
The symmetry property means that the Vendi Score will be the same regardless of how we order the rows and columns in the similarity matrix.

\subsection{Calculating the Vendi Score}
Calculating the Vendi Score for a sample of $n$ elements requires finding the eigenvalues of an $n \times n$ matrix, which has a time complexity of $O(n^3)$.
The Vendi Score can be approximated using column sampling methods~\citep[i.e. the Nystr\"{o}m method; ][]{williams2000using}.
However, in many of the applications we consider, the similarity functions we use are inner products between explicit feature vectors $\phi(x) \in {\mathbb{R}}^d$, with $d \ll n$. That is, ${\bm{K}} = {\bm{X}}^{\top}{\bm{X}}$, where ${\bm{X}} \in {\mathbb{R}}^{n \times d}$ is the feature matrix with row ${\bm{X}}_{i, :} = \phi(x_i)$.
The eigenvalues of ${\bm{K}}/n$ are the same as the eigenvalues of the covariance matrix ${\bm{X}}\mX^{\top}/n$, therefore we can calculate the Vendi Score exactly in a time of $O(d^2n + d^3) = O(d^2n)$.
This is the same complexity as existing metrics such as \gls{FID}~\citep{heusel2017gans}, which require calculating the covariance matrix of Inception embeddings.

 

\section{Experiments}
\label{sec:empirical}
\begin{table}[t]
\centering
\begin{tabular}{lcr}
  \toprule
  Model & IntDiv & \acrshort{VS}\\
  \midrule
  Original & 0.855 & 403.9\\
  \midrule
  \acrshort{AAE} & 0.859 & 501.1\\
  \acrshort{Char-RNN} & 0.856 & 482.4\\
  Combinatorial & 0.873 & 536.9\\
  \acrshort{HMM} & 0.871 & 250.9\\
  \acrshort{JTN} & 0.856 & 489.5\\
  Latent \acrshort{GAN} & 0.857 & 486.4\\
  N-gram & 0.874 & 479.8\\
  \acrshort{VAE} & 0.856 & 475.3\\
  \bottomrule
\end{tabular}
\caption{
\label{tab:molecule_metrics}
IntDiv and \acrshort{VS} for generative models of molecules. The \acrshort{HMM} has one of the highest IntDiv scores, but scores much lower on \acrshort{VS} . 
An analysis of $250$ molecules from the \acrshort{HMM} reveals \acrshort{VS} is more accurate in this case. (See \Cref{fig:molecule_kernels}.)
}
\end{table}


 We used the Vendi Score, which we now denote by \acrshort{VS} for the rest of this section, to evaluate the diversity of generative models of molecules, an application where diversity plays an important role in enabling discovery. 
We compare \acrshort{VS} to IntDiv, a function of the average similarity:
\begin{align*}
  \mathrm{IntDiv}(x_1, \ldots, x_n) = 1 - \frac{1}{n^2} \sum_{i, j} k(x_i, x_j).
\end{align*}
We found that unlike IntDiv, 
\acrshort{VS} is effective at capturing diversity. 
We also applied \acrshort{VS} to generative models of images, and decoding algorithms of text, where we found it confirms what we know about diversity in those applications. 
We also used \acrshort{VS} to measure mode collapse in \glspl{GAN} and datasets. Unlike the usual metrics for measuring mode collapse, \acrshort{VS} makes it possible to evaluate mode collapse without needing labels. 
Furthermore, \acrshort{VS} revealed that current metrics for measuring mode collapse are insufficient in that, unlike \acrshort{VS}, they do not capture fine-grained notions of diversity.
Finally, we used \acrshort{VS} to analyze the diversity of several image, text, and molecule datasets, gaining insights into the diversity profile of those datasets. 
(Implementation details are provided in Appendix~\ref{app:implementation_details}.)

\subsection{Evaluating molecular generative models for diversity}
First, we evaluate the diversity of samples from generative models of molecules. For generative models to be 
useful for the discovery of novel molecules, they ought to be diverse. The standard diversity metric in this setting is IntDiv.
We evaluate samples from generative models provided in the \acrshort{MOSES} benchmark~\citep{polykovskiy2020moses}, using the first $2500$ valid molecules in each sample.
Following prior work, our similarity function is the Morgan fingerprint similarity (radius 2), implemented in RDKit.\footnote{RDKit: Open-source Cheminformatics. https://www.rdkit.org.}

In Table~\ref{tab:molecule_metrics}, we highlight an instance where IntDiv and \acrshort{VS} disagree: IntDiv ranks the \acrshort{HMM} among the most diverse models, while \acrshort{VS} ranks it as the least diverse.
The reason is illustrated in Figure~\ref{fig:molecule_kernels}: the \acrshort{HMM} has a high IntDiv score because, on average, the \acrshort{HMM} molecules have low pairwise similarity scores,
but there are a number of clusters of identical or nearly identical molecules.
\begin{figure}[t]
  \centering
  \includegraphics[width=\textwidth]{.//molecule_kernels}
  \caption{The kernel matrices for $250$ molecules sampled 
    from the \acrshort{HMM}, \acrshort{AAE}, and the original dataset, sorted
    lexicographically by \acrshort{SMILES} string representation.
    The samples have similar IntDiv scores,
    but the \acrshort{HMM} samples score much lower on \acrshort{VS}.
    The figure shows that the \acrshort{HMM} generates a number of exact duplicates. 
    \acrshort{VS} is able to capture the \acrshort{HMM}'s lack of diversity while IntDiv cannot.}
\label{fig:molecule_kernels}
\end{figure}

\subsection{Assessing mode collapse in \glspl{GAN} and datasets}
\begin{figure}[h]
  \centering
\includegraphics[width=0.49\textwidth]{.//mnist_modes}\includegraphics[width=0.49\textwidth]{.//mnli_modes}\caption{Detecting mode dropping in image and text datasets. 
    We evaluate \acrshort{VS} and IntDiv on datasets containing $500$ examples drawn uniformly from between one and ten classes: 
    digits in \acrshort{MNIST} and sentences genres in Multi\acrshort{NLI}.
Compared to IntDiv, \acrshort{VS} increases more consistently with the number of classes.
} \label{fig:mode_dropping}
\end{figure}

 First we examine whether \acrshort{VS} captures mode collapse in a controlled setting, where we have information about the ground truth class distribution.
We simulate mode collapse by sampling equally-sized subsets of two classification datasets, with each subset ${\mathcal{S}}_i$ containing examples sampled uniformly from the first $i$ categories. 
We perform this experiment on one image dataset (\acrshort{MNIST}) and one text dataset (Multi\acrshort{NLI};~\citealp{williams2018broad}), using simple similarity functions.

\acrshort{MNIST} consists of 28$\times$28-pixel images of hand-written digits, divided into ten classes.
The similarity score we use is the cosine similarity between pixel vectors: $k(x, x') = \langle x, x' \rangle / \|x\| \|x'\|$,
where $x, x'$ are $28^2$-dimensional vectors with entries specifying pixel intensities between 0 and 1.
Multi\acrshort{NLI}~\citep{williams2018broad} is a multi-genre sentence-pair classification dataset.
We use the premise sentences from the validation split (mismatched), which are drawn from one of ten genres.
We define similarity using the n-gram overlap kernel:
for a given $n$, the n-gram kernel $k_n$ can be expressed as the cosine similarity between feature vectors $\phi^n(x)$, where $\phi_i^n(x)$ is equal to the number of times n-gram $i$ appears in $x$.
We use the average of $k(x, x') = \frac{1}{4}\sum_{n=1}^4 k_n(x, x')$.

The results (Figure~\ref{fig:mode_dropping}) show that \acrshort{VS} generally increases with the number of classes, even using these simple similarity scores.
In \acrshort{MNIST} (left), \acrshort{VS} increases roughly linearly for the first six digits (0-5) and then fluctuates. This could occur if the new modes are similar to the other modes in the sample, or have low internal diversity.
In Multi\acrshort{NLI} (right), \acrshort{VS} increases monotonically with the number of genres represented in the sample.
In both cases, \acrshort{VS} has a stronger correlation with the number of modes compared to IntDiv.

Mode collapse is a failure mode of \glspl{GAN} that has received a lot of attention from the \gls{ML} community~\citep{Metz2017, dieng2019prescribed}. 
The main metric for measuring mode collapse, called \acrlong{NOM}(\gls{NOM}), can only be used to assess mode collapse for \glspl{GAN} trained on a labelled dataset. 
\gls{NOM} is computed by training a classifier on the labeled training data and counting the number of unique classes that are predicted by the trained classifier for the generated samples. 
In Table~\ref{tab:mode_dropping}, we evaluate two models that were trained on the Stacked\acrshort{MNIST} dataset, a standard setting for evaluating mode collapse in \glspl{GAN}. 
Stacked\acrshort{MNIST} is created by stacking three \acrshort{MNIST} images along the color channel, creating $1000$ classes corresponding to $1000$ number of modes.

\begin{table*}[ht]
\small
\centering
\begin{tabular}{l c c c c}
  \toprule
Model & \acrshort{NOM} & Mode Div. & \acrshort{VS}\\
\midrule
Self-cond. \gls{GAN} & 1000 & 921.0 & 746.7 \\
Pres\gls{GAN} & 1000 & 948.7 & 866.6 \\
\midrule
Original & 1000 & 950.8 & 943.7 \\
\bottomrule
\end{tabular}
\caption{
\label{tab:mode_dropping}
\acrshort{VS} captures a more fine-grained notion of diversity than \acrlong{NOM}(\acrshort{NOM}). 
Although Pres\gls{GAN} and Self-cond.\gls{GAN} both capture all the $1000$ modes, \acrshort{VS} reveals that Pres\gls{GAN} is more diverse than Self-cond.\gls{GAN} 
and that they both are less diverse than the original dataset. }
\end{table*}

In prior work, mode collapse is evaluated by training an \acrshort{MNIST} classifier
and counting the number of unique classes that are predicted for the generated samples.
We adapt this approach and 
we calculate \acrshort{VS} using the probability product kernel~\citep{jebara2004probability}: $k(x, x') = \sum_{y} p(y \mid x)^{\frac{1}{2}} p(y \mid x')^{\frac{1}{2}}$, where the class likelihoods are given by the classifier.
We compare Pres\acrshort{GAN}~\citep{dieng2019prescribed} and Self-conditioned \gls{GAN}~\citep{liu2020selfconditioned}, two \glspl{GAN} that are known to capture all the modes.
Table~\ref{tab:mode_dropping} shows that Pres\acrshort{GAN} and Self-conditioned \gls{GAN} have the same diversity according to number of modes, they capture all $1000$ modes. 
However, \acrshort{VS} reveals a more fine-grained notion of diversity, indicating that Pres\acrshort{GAN} is more diverse than Self-conditioned \gls{GAN} and that both are less diverse than the original dataset. 
One possibility is that \acrshort{VS} is capturing imbalances in the mode distribution.
To see whether this is the case, we also calculate what we call \emph{Mode Diversity}, the exponential entropy of the predicted mode distribution: $\exp H(\hat{p}(y))$, where $\hat{p}(y) = \frac{1}{n} \sum_{i=1}^n p(y \mid x_i)$. The generative models score lower on \acrshort{VS} than Mode Diversity, indicating that low scores cannot be entirely attributed to imbalances in the mode distribution. 
Therefore \acrshort{VS} captures more aspects of diversity, even when we are using the same representations as existing methods. 

\subsection{Evaluating image generative models for diversity}
\label{sec:images}

\begin{table*}[!t]
  \small
  \setlength{\tabcolsep}{4pt}
    \begin{minipage}{0.49\linewidth}
      \centering
        \begin{tabular}{l r r r r r}
          \toprule
          Model & \acrshort{IS}$_\uparrow$ & \acrshort{FID}$_\downarrow$ & Prec$_\uparrow$ & Rec$_\uparrow$ & \acrshort{VS}$_\uparrow$ \\
          \\
          \multicolumn{6}{l}{\textbf{\acrshort{CIFAR}-10}} \\
          \midrule
          Original &  &  &  &  & 19.50 \\
          \acrshort{VDVAE} & 5.82 & 40.05 & 0.63 & 0.35 & 12.87 \\
          DenseFlow & 6.01 & 34.54 & 0.62 & 0.38 & 13.55 \\
          \acrshort{IDDPM} & 9.24 & 4.39 & 0.66 & 0.60 & 16.86 \\
          \\
           \multicolumn{5}{l}{\textbf{LSUN Cat 256$\times$256}} \\
          \midrule
          Original &  &  &  &  & 15.12 \\
          Style\acrshort{GAN}2 & 4.84 & 7.25 & 0.58 & 0.43 & 13.55 \\
          \acrshort{ADM} & 5.19 & 5.57 & 0.63 & 0.52 & 13.09 \\
          \acrshort{RQ-VT} & 5.76 & 10.69 & 0.53 & 0.48 & 14.91 \\
        \end{tabular}
\end{minipage}\hfill
    \begin{minipage}{.49\linewidth}
      \centering
        \begin{tabular}{ r r r r r}
          \toprule
         \acrshort{IS}$_\uparrow$ &\acrshort{FID}$_\downarrow$ & Prec$_\uparrow$ & Rec$_\uparrow$ & \acrshort{VS}$_\uparrow$ \\
          \\
          \multicolumn{5}{l}{\textbf{ImageNet 64$\times$64}} \\
          \midrule
            &  &  &  & 43.93 \\
           9.68 & 57.57 & 0.47 & 0.37 & 18.04 \\
          5.62 & 102.90 & 0.36 & 0.17 & 12.71 \\
         15.59 & 19.24 & 0.59 & 0.58 & 24.28 \\
          \\
           \multicolumn{5}{l}{\textbf{LSUN Bedroom 256$\times$256}} \\
          \midrule
            &  &  &  & 8.99 \\
          2.55 & 2.35 & 0.59 & 0.48 & 8.76 \\
           2.38 & 1.90 & 0.66 & 0.51 & 7.97 \\
           2.56 & 3.16 & 0.60 & 0.50 & 8.48 \\
        \end{tabular}
\end{minipage} 
\caption{
\label{tab:comparing_image_models}
\acrshort{VS} generally agrees with the existing metrics. On low-resolution datasets (top left and top right) the diffusion model performs better on all of the metrics. 
On the \acrshort{LSUN} datasets (bottom left and bottom right), the diffusion model gets the highest quality scores as measured by \acrshort{IS}, but scores lower on \acrshort{VS}. 
No model matches the diversity score of the original dataset they were trained on.
}
\end{table*}

\begin{figure*}[t!]
     \centering
     \begin{subfigure}[c]{0.5\linewidth}
         \centering
         \includegraphics[width=\linewidth]{.//lsun_cat_components_inception}
     \end{subfigure}\hfill
     \begin{subfigure}[c]{0.5\linewidth}
         \centering
         \includegraphics[width=\linewidth]{.//lsun_cat_components_pixels}
     \end{subfigure}
\caption{
\label{fig:lsun_cat_comparing_kernels}
The choice of similarity function provides a way of specifying the notion of diversity that is relevant for a given application.
We project \acrshort{LSUN} Cat images along the top eigenvectors of the kernel matrix, using either Inception features or pixels to define similarity.
Inception similarity provides a form of semantic similarity, with components corresponding to particular cat breeds, while the pixel kernel captures visual similarity.
For each eigenvector ${\bm{u}}$, we show the four images with the highest and lowest entries in ${\bm{u}}$. 
For both kernels, every similarity score is positive, so all entries in the top eigenvector have the same sign; the images with the 
highest weights in this component have the highest expected similarity scores.
The remaining eigenvectors partition the images along different dimensions of variation.
}
\end{figure*}
 We now evaluate several recent models for unconditional image generation, comparing the diversity scores with standard evaluation metrics, \gls{IS}~\citep{Salimans2016}, \gls{FID}~\citep{heusel2017gans}, Precision~\citep{sajjadi2018assessing}, and Recall~\citep{sajjadi2018assessing}.
The models we evaluate represent popular classes of generative models, including a variational autoencoder~\citep[\acrshort{VDVAE}; ][]{child2020very}, a flow model~\citep[DenseFlow; ][]{grcic2021densely}, diffusion models (\acrshort{IDDPM}, \citealp{nichol2021improved}; \acrshort{ADM}~\citealp{dhariwal2021diffusion}), \gls{GAN}-based models~\citep{karras2019style,karras2020analyzing}, and an auto-regressive model~\citep[\acrshort{RQ-VT}; ][]{lee2022autoregressive}.
The models are trained on \acrshort{CIFAR}-10~\citep{Krizhevsky2009}, ImageNet~\citep{russakovsky2015imagenet}, or two categories from the \acrshort{LSUN} dataset~\citep{yu2015lsun}.
We either select models that provide precomputed samples, or download publicly available model checkpoints and sample new images using the default hyperparameters. (More details are in Appendix~\ref{app:implementation_details}.)

The standard metrics in this setting use a pre-trained Inception ImageNet classifier to map images to real vectors.
Therefore, we calculate \acrshort{VS} using the cosine similarity between Inception embeddings, using the same 2048-dimensional representations used for evaluating \acrshort{FID} and Precision/Recall.
As a result, the highest possible similarity score is 2048.
The baseline metrics are reference-based, with the exception of \acrshort{IS}. 
\acrshort{FID} and \acrshort{IS} capture diversity implicitly. Recall was introduced to capture diversity explicitly, with diversity defined as coverage of the reference distribution.

The results of this comparison are in Table~\ref{tab:comparing_image_models}.
On the lower resolution datasets (top left and top right), \acrshort{VS} generally agrees with the existing metrics.
On those datasets the diffusion model performs better on all of the metrics.
On the \acrshort{LSUN} datasets (bottom left and bottom right), the diffusion model gets the highest quality scores as measured by \acrshort{IS}, but scores lower on \acrshort{VS}.
No model matches the diversity score of the original dataset they were trained on.
In addition to comparing the diversity of the models, we can also compare the diversity scores between datasets:
as a function of Inception similarity, the most diverse dataset is ImageNet 64$\times$64, followed by \acrshort{CIFAR}-10, 
followed by \acrshort{LSUN} Cat, and then \acrshort{LSUN} Bedroom.
This agrees with the intuition that the Inception network will induce the highest-rank embedding space for datasets that most resemble the training distribution.

\acrshort{VS} should be understood as the diversity with respect to a specific similarity function, in this case, the Inception ImageNet similarity.
We illustrate this point in Figure~\ref{fig:lsun_cat_comparing_kernels} by comparing the top eigenvalues of the kernel matrices corresponding to the 
Inception similarity and the pixel similarity, which we calculate by resizing the images to 32$\times$32 pixels and taking the cosine similarity between pixel vectors.
Inception similarity provides a form of semantic similarity, with components corresponding to particular cat breeds, while the pixel kernel provides a 
simple form of visual similarity, with components corresponding to broad differences in lightness, darkness, and color.

\subsection{Evaluating decoding algorithms for text for diversity}
\label{sec:text}
We evaluate diversity on the \acrshort{MS COCO} image-captioning dataset~\citep{lin2014microsoft}, following prior work on diverse text generation~\citep{vijayakumar2018diverse}.
In this setting, the subjects of evaluation are diverse decoding algorithms rather than parametric models.
Given a fixed conditional model of text $p(x \mid c)$, where $c$ is some conditioning context, the aim is to identify a ``Diverse N-Bet List'', a list of sentences that have high likelihood but are mutually distinct.
The baseline metric we compare to is n-gram diversity~\citep{li2016diversity},
which is the proportion of unique n-grams divided by the total number of n-grams.
We define similarity using the n-gram overlap kernel:
for a given $n$, the n-gram kernel $k_n$ is the cosine similarity between bag-of-n-gram feature vectors.
We use the average of $k_1, \ldots, k_4$.
This ensures that \acrshort{VS} and n-gram diversity are calculated using the same feature representation.

Each image in the validation split has five captions written by different human annotators, and we compare these with captions generated by a 
publicly available captioning model trained on this dataset~\footnote{https://huggingface.co/ydshieh/vit-gpt2-coco-en-ckpts}.
For each image, we generate five captions using either beam search or \gls{DBS}~\citep{vijayakumar2018diverse}.
\gls{DBS} takes a parameter, $\gamma$, called the diversity penalty, and we vary this between $0.2$, $0.6$, and $0.8$.

Table~\ref{tab:caption_diversity} shows that all diversity metrics increase as expected, ranking beam search the lowest, the human 
captions the highest, and \gls{DBS} in between, increasing with the diversity penalty.
The human diversity score of $4.88$ can be interpreted as meaning that, on average, all five human-written captions 
are almost completely dissimilar from each other, while beam search effectively returns only three distinct responses for every five that it generates.

\begin{table}[t]
  \centering
\begin{tabular}{l c c c}
  \toprule
Source & \acrshort{BLEU}-4 & N-gram div. & \acrshort{VS} \\
\midrule
Human &  & 0.82 & 4.88\\
  \midrule
Beam Search & 0.27 & 0.42 & 3.00\\
\acrshort{DBS} $\gamma=0.2$ & 0.25 & 0.49 & 3.16\\
\acrshort{DBS}  $\gamma=0.5$ & 0.22 & 0.63 & 4.14\\
\acrshort{DBS}  $\gamma=0.8$ & 0.21 & 0.68 & 4.37\\
\bottomrule
\end{tabular}
\captionof{table}{
\label{tab:caption_diversity}
Quality and diversity scores for an image captioning model using Beam Search or \acrlong{DBS}(\acrshort{DBS}), varying the diversity penalty $\gamma$.
\acrshort{BLEU}-4 measures n-gram overlap with the human-written reference captions, a proxy for quality.
Increasing $\gamma$ leads to higher diversity scores, as expected and as captured by both N-gram div. and \acrshort{VS}, but leads to a lower quality score. 
}
\end{table}

\begin{figure}[t]
\centering
\resizebox{0.5\linewidth}{!}{\includegraphics{.//caption_diversity.pdf}
}
\caption{
\label{fig:caption_diversity}
\acrshort{VS} is correlated with N-gram diversity. Each point represents a group of five captions for a particular image.
}
\end{figure}

 \begin{figure}[h!]
  \small
\begin{minipage}[t]{.48\textwidth}
  High Vendi Score, low n-gram diversity:
  \it{}
  \begin{itemize}[leftmargin=*]
\item two men in bow ties standing next to steel rafter.
\item several men in suits talking together in a room.
\item an older \textbf{man in a tuxedo} standing next to a younger \textbf{man in a tuxedo} wearing glasses.
\item two men wearing tuxedos glance at each other.
\item older \textbf{man in tuxedo} sitting next to another younger \textbf{man in tuxedo}.
  \end{itemize}
\end{minipage}\hfill
\begin{minipage}[t]{.48\textwidth}
  Low Vendi Score, high n-gram diversity:
  \it{}
  \begin{itemize}[leftmargin=*]
\item a man and woman cutting a slice of cake by trees.
\item a couple of people standing cutting a cake.
\item \textbf{the dork with the earring stands next to the asian beauty who is way out of his league.}
\item a newly married couple cutting a cake in a park.
\item a bride and groom are cutting a cake as they smile.
\end{itemize}
\end{minipage}
\caption{
\label{fig:caption_diversity_example}
Two sets of captions that receive different ranks according to \acrshort{VS} and n-gram diversity.
We manually highlight some features contributing to the different scores. 
On the left, a sentence contains repeated n-grams, which is penalized by n-gram diversity even though the sentences in the bunch are diverse.
On the right, one long outlier sentence contributes most of the n-grams for this group of sentences, greatly increasing the n-gram diversity even though the 
other sentences in the group are not diverse. \acrshort{VS} doesn't have those limitations. 
}
\end{figure}
 

\begin{figure}[!hbpt]
\begin{minipage}{\linewidth}
     \centering
      \includegraphics{.//goodscents_categories}
\captionof{figure}{
\label{fig:goodscents_categories}
The Vendi Scores of samples containing $500$ molecules with different scent labels, calculating diversity using two similarity functions:
Morgan molecular fingerprint similarity, and the similarity between odor descriptors. Each molecule is associated 
with one or more human-written tags (e.g. ``floral, fruity, green, sweet''), and the odor-descriptor similarity is the cosine similarity between binary tag indicator vectors.
}
\end{minipage}
\end{figure}
 

\begin{figure}[ht!]
\begin{minipage}{\linewidth}
     \centering
      \includegraphics[width=\linewidth]{.//molecule_categories}
\captionof{figure}{
\label{fig:molecule_categories}
The scent categories in Goodscents dataset with the highest (top) and lowest (bottom) \gls{VS}, using the molecular fingerprint similarity.
We show $100$ examples from each category, in decreasing order of average similarity, with the image at the top left having the highest average similarity scores.
}
\end{minipage}
\end{figure}
 

\begin{figure}[!hbpt]
     \centering
\includegraphics[width=\linewidth]{.//celeba_categories_attr}
\caption{
\label{fig:celeba_categories}
The attributes in \acrshort{CelebA} with the lowest (left) and highest (right) \acrshort{VS}, defining similarity as the cosine similarity 
between either Inception embeddings (top), pixel vectors (middle), or binary attribute vectors (bottom).
We show $100$ examples from each category, in decreasing order of average similarity, with the image at the top left having the highest average similarity scores according to the corresponding kernel.
These examples illustrate the importance of the choice of similarity function for defining the notion of diversity that is relevant for a given application. However, almost all choices of similarity functions show that the \acrshort{CelebA} dataset is more diverse for men than for women. 
}
\end{figure}
 In Figure~\ref{fig:caption_diversity}, we plot the relationship between \acrshort{VS} and n-gram diversity using the \acrshort{MS COCO} captioning data 
and the n-gram overlap kernel described in Section~\ref{sec:text}.
The figure shows that \acrshort{VS} is highly correlated with n-gram diversity, which is expected given that our similarity function is based on n-gram overlap.
Nonetheless, there are some data points that the metrics rank differently.
This is because n-gram diversity conflates two properties: the diversity of n-grams within a single sentences and the n-gram overlap between sentences.
We highlight two examples in Figure~\ref{fig:caption_diversity_example}.
In general, the instances that n-gram diversity ranks lower compared to \acrshort{VS} contain individual sentences that repeat phrases.
On the other hand, n-gram diversity can be inflated in cases when one sentence in the sample is much longer than the others, even if the other sentences are not diverse.

\subsection{Diagnosing datasets for diversity}
\label{sec:diagnosing_datasets}
\paragraph{Molecules.}
We evaluate the diversity of molecules in the GoodScents database of perfume materials\footnote{http://www.thegoodscentscompany.com/}, which 
has been used in prior \gls{ML} research on odor modeling~\citep{sanchez2019machine}.
We use the standardized version of the data provided by the Pyrfume library~\footnote{https://pyrfume.org/}. 
Each molecule in the dataset is labeled with one or more odor descriptors (for example, ``clean, oily, waxy'' or ``floral, fruity, green'').
We form groups of molecules corresponding to the seven most common odor descriptors, with each group consisting of $500$ randomly sampled molecules.
We evaluate \acrshort{VS} using two similarity functions: the Morgan fingerprint similarity (radius $2$), and the 
similarity between odor descriptors, defined as the cosine similarity between descriptor indicator vectors $\phi(x)$, where $\phi_i(x)$ is equal to one 
if descriptor $i$ is associated with molecule $x$ and zero otherwise.
The diversity scores are plotted in Figure~\ref{fig:goodscents_categories}.
The molecular diversity score and the odor-descriptor diversity scores are correlated, meaning that words like ``woody'' and ``green'' are used to 
describe molecules that vary in molecular structure and also elicit diverse odor descriptions, 
while words like ``waxy'' and ``fatty'' are used for molecules that are similar to each other and elicit similar odor descriptions.
For example, the word ``green'' appears in tag sets such as
``aldehydic, citrus, cortex, green, herbal, tart'' and ``floral, green, terpenic, tropical, vegetable, woody'', whereas the word ``waxy'' tends to co-occur with the same tags
(``fresh, waxy'';
``fresh, green, melon rind, mushroom, tropical, waxy'';
``fruity, green, musty, waxy'').
Molecules from the categories with the highest and lowest scores are illustrated in Figure~\ref{fig:molecule_categories}.
\paragraph{Images.} We now assess both \acrshort{CIFAR-100} and \acrshort{CelebA} for diversity. 
In Figure~\ref{fig:celeba_categories} we calculate \acrshort{VS} for samples from \acrshort{CelebA}. 
Images in \acrshort{CelebA} are associated with $40$-dimensional binary attribute vectors.
We use these attributes as an additional similarity score, defining the attribute similarity as the cosine similarity between attribute vectors.
Figure~\ref{fig:celeba_categories} shows that the \acrshort{CelebA} dataset is more diverse for men than for women. 
This result illustrates the potential usefulness of the Vendi Score in mitigating biases in datasets. 

In Figure~\ref{fig:image_categories}, we calculate \acrshort{VS} for samples from different categories in \acrshort{CIFAR}-100, 
using the cosine similarity between either Inception embeddings or pixel vectors.
The pixel diversity is highest for categories like ``aquarium fish'', which vary in color, brightness, and orientation, and lowest for 
categories like ``cockroach'' in which images have similar regions of high pixel intensity (like white backgrounds).
The Inception diversity is less straightforward to interpret, but might correspond to some form of semantic diversity---for example, the 
Inception diversity might be lower for classes like ``castle,'' that correspond to distinct ImageNet categories, and 
higher for categories like ``clock'' and ``keyboard'' that are more difficult to classify.
In Appendix~\ref{app:diagnosing_datasets}, we show additional examples from text and other image datasets.


\begin{figure*}[t]
     \centering
      \includegraphics[width=\linewidth]{.//cifar100_categories}
\caption{
\label{fig:image_categories}
The categories in \acrshort{CIFAR}-100 with the lowest and highest \acrshort{VS}, defining similarity as the cosine similarity between either Inception embeddings or pixel vectors.
We show $100$ examples from each category, in decreasing order of average similarity, with the image at the top left having the highest average similarity scores according to the corresponding kernel.
}
\end{figure*}
 
 \glsresetall

\section{Discussion}
\label{sec:discussion}
We introduced the Vendi Score, a metric for evaluating diversity in \gls{ML}. 
The Vendi Score is defined as a function of the pairwise similarity scores between elements of a sample and can be interpreted as the effective number of unique elements in the sample. The Vendi Score is interpretable, general, and applicable to any domain where similarity can be defined. It is unsupervised, in that it doesn't require labels or a reference probability distribution or dataset. Importantly, the Vendi Score allows its user to specify the form of diversity they want to measure via the similarity function. We showed the Vendi Score can be computed efficiently exactly and showcased its usefulness in several \gls{ML} applications, different datasets, and different domains. In future work, we'll leverage the Vendi Score to improve data augmentation, an important \gls{ML} approach in settings with limited data.

 
\subsection*{Acknowledgements}
Adji Bousso Dieng is supported by the National Science Foundation, Office of Advanced Cyberinfrastructure (OAC): \#2118201.

\clearpage
\bibliographystyle{apa}
