\subsection{Bayesian Adaptive Design}
\label{sec:BAD}

BED can be extended to adaptive settings---which we will refer to as \emph{Bayesian adaptive design} (BAD)---by iteratively incorporating information gathered during the experiment.
In particular, if our designs and outcomes can be broken down into a sequence of steps, denoted $\xi=\{\xi_1,\dots,\xi_T\}$ and $y=\{y_1,\dots,y_T\}$ respectively, then we can consider the \emph{incremental} EIG for each experiment step given the history, $h_{t-1}=\{(\xi_k,y_k)\}_{k=1}^{t-1}$:
\begin{align}
	\label{eq:cond-eig}
 \begin{split}
	&\text{EIG}_{\theta}(\xi_t | h_{t-1}) =\\
 &\quad\E_{p(\theta|h_{t-1}) p(y_t|\theta, \xi_t, h_{t -1})} \left[ \log \frac{p(y_t|\theta, \xi_t, h_{t -1})}{p(y_t|\xi_t, h_{t-1})}\right],
 \end{split}
\end{align}
with $h_0=\varnothing$.
This can also be viewed as a conventional EIG with an updated prior, $p(\theta|h_{t-1})$.


The traditional BAD approach now chooses designs 
by optimizing $\text{EIG}_{\theta}(\xi_t | h_{t-1})$ at each experiment step~\citep{myung2013,watson2017quest+,chaloner1995,sebastiani2000maximum}, as per~\Cref{fig:boed_approaches_traditional}.
This now provides a quantitative mechanism for adaptive experimentation.
It is generally more effective than \emph{static} design approaches---wherein all the $\xi_1,\dots,\xi_T$ are simultaneously chosen upfront---as information from previous observations is incorporated in each design decision through the iterative model updates.


\input{bad_fig.tex}


Note here that
updates of the model between steps of BAD do not need to be restricted to updating our beliefs about $\theta$.
In particular, one typically makes Bayesian updates to all the parameters of the underlying model (i.e.~$\psi$) if these are distinct to $\theta$, with $p(\theta|h_{t-1})$ and $p(y_t|\theta, \xi_t, h_{t -1})$ implicitly defined through this update.





\subsection{Nested Estimation}

Perhaps the most immediate computational challenge is that, except in special cases, the EIG is a doubly-intractable quantity that is difficult to estimate, let alone optimize.
In particular, it cannot be directly estimated using a conventional Monte Carlo estimator.


The origin of this issue is that, in general, both $p(y|\xi)$ and $p(\latent|y, \xi)$ are themselves intractable.
Thus, regardless of whether we use the form of the EIG from~\eqref{eq:eig-post} or~\eqref{eq:eig}, we have a term inside the expectation that is both intractable and varies between realizations of $y$.
For example, suppose we try to construct a Monte Carlo estimator of~\eqref{eq:eig}:
\begin{align}
\label{eq:eig_mc}
    \text{EIG}_{\theta}(\xi) \approx \frac{1}{N}\sum_{n=1}^N \log p(y_n|\theta_n,\xi) - \log p(y_n|\xi),
\end{align}
where $\theta_n,y_n \overset{\scriptstyle{\text{i.i.d.}}}{\sim} p(\theta)p(y|\theta,\xi)$.
We now find that our estimator cannot itself be directly evaluated as the individual $p(y_n|\xi)$ terms are intractable.


If $y$ can only take on a small number of discrete values, this does not form too much of an issue as we can simply enumerate them, estimate $p(y|\xi)$ for each, and use a Rao-Blackwellized estimator~\citep{vincent2017darc,rainforth2018nesting,gal2017deep,foster2021variational}, such as
\begin{align}
\label{eq:eig_finite}
\begin{split}
     \hat{\mu}_N :=& \frac{1}{N}\sum\nolimits_{n=1}^N \sum\nolimits_{y\in\mathcal{Y}} p(y|\theta_n,\xi) \log p(y|\theta_n,\xi) \\
     &-\sum\nolimits_{y\in\mathcal{Y}} \hat{p}(y|\xi) \log \hat{p}(y|\xi),
\end{split}
\end{align}
where $\hat{p}(y|\xi) = \frac{1}{N}\sum_{n=1}^N p(y|\theta_n,\xi)$.
However, if $y$ is continuous or the set $\mathcal{Y}$ is large, this is not feasible and we must fall back to sampling $y$, as per~\eqref{eq:eig_mc}.
Here estimation of the EIG for a given $\xi$ requires an additional \emph{nested} mechanism for estimating $p(y_n|\xi)$ for each $y_n$ \citep{hamada2001finding}, or, alternatively, approximating $p(\theta_n|y_n,\xi)$ for each pair $(\theta_n,y_n)$.


\newcommandx{\munmc}[2][1=N,2=M]{\ensuremath{\hat{\mu}_{#1,#2}}}
\newcommandx{\munmcq}[2][1=N,2=M]{\ensuremath{\hat{\mu}_{#1,#2,q}}}
Traditional approaches for this nested estimation have typically either been based on nested Laplace approximations~\citep{lewi2009sequential,cavagnaro2010adaptive,long2013,ryan2015fully,long2022multimodal} or \emph{nested Monte Carlo} (NMC) estimators~\citep{rainforth2017thesis,rainforth2018nesting,hamada2001finding,ryan2003estimating,cook2008optimal,ryan2014towards,myung2013,beck2018fast,zheng2020sequential}.


While the former is prone to introducing substantial biases, the latter can provide asymptotically consistent estimates.
Specifically,
the NMC estimator
\begin{align}
\label{eq:nmc}
    \munmc:= \frac{1}{N}\sum_{n=1}^N \log \frac{p(y_n|\theta_n,\xi)}{\frac{1}{M}\sum_{m=1}^M p(y_n|\theta_m',\xi)},
\end{align}
where $\theta_n,y_n \overset{\scriptstyle{\text{i.i.d.}}}{\sim} p(\theta)p(y|\theta,\xi)$ and $\theta_m' \overset{\scriptstyle{\text{i.i.d.}}}{\sim} p(\theta)$, has an asymptotic mean squared error (MSE) to $\text{EIG}_{\theta}(\xi)$ of $\mathcal{O}(a/N+b/M^2)$ for constants $a$ and $b$ (see~\citep{rainforth2018nesting} for a precise characterization), and thus converges as $N,M \to \infty$ (subject to some weak assumptions).


One can improve this estimator by replacing the simple Monte Carlo estimates for $p(y_n|\xi)$ with ones based on importance sampling, yielding the more general estimator
\begin{align}
\label{eq:nmcq}
    \munmcq:= \frac{1}{N}\sum_{n=1}^N \log \frac{p(y_n|\theta_n,\xi)}{\frac{1}{M}\sum_{m=1}^M \frac{p(y_n|\theta_m',\xi)p(\theta_m')}{q(\theta_m'|y_n,\xi)}},
\end{align}
where we now instead draw samples from our proposal: $\theta_m' \overset{\scriptstyle{\text{i.i.d.}}}{\sim} q(\theta_m'|y_n,\xi)$.
This can help reduce the constants in the convergence rate (i.e.~$a$ and $b$) compared with $\munmc$, along with the finite sample bias and variance.


Unfortunately, NMC estimators have a number of undesirable properties.
First, they are, in general, biased for any finite value of $M$, due to the fact that nonlinear mappings of unbiased estimators are themselves biased estimators.
Second, and perhaps more importantly, they can be very expensive to evaluate as their cost scales as $C=NM$.
In turn, this causes them to have a slow convergence rate: for most problems, they cannot achieve a better MSE rate than $\mathcal{O}(C^{-2/3})$, with this occurring when $M\propto \sqrt{N}$~\citep{rainforth2018nesting}.


\subsection{Debiasing Schemes}

An important recent advancement in BED has been the development of mechanisms for reducing~\citep{goda2020multilevel,goda2022unbiased}, or even eliminating~\citep{goda2022unbiased}, the bias of EIG estimators and their gradients.
While debiasing techniques for (nested) Monte Carlo estimators date back to at least~\citep{mcleish2011general,rhee2015unbiased,lyne2015russian} (with some aspects even having much earlier origins in the physics literature~\citep{kennedy1985noise,bhanot1985bosonic}), their application to the specific problem of EIG estimation is a more recent development.


The basis for these debiased EIG estimators is the framework of multi-level Monte Carlo (MLMC)~\citep{giles2008multilevel}.
Perhaps most notably, Goda et al. (2022)~\citep{goda2022unbiased} recently showed that the MLMC variant of~\citep{rhee2015unbiased} can be used to generate fully \emph{unbiased} estimates of the EIG and its gradients.
Their approach works by first expressing the EIG as the expectation of the NMC in~\eqref{eq:nmcq} with $N=1$ and $M=\infty$, then noting that this can be written as a telescopic sum.
Namely,
\begin{align}
    \text{EIG}_{\theta}(\xi) &= \E[\munmcq[1][\infty]] =  \E\left[\sum\nolimits_{\ell=0}^\infty \Delta_\ell \right],
\end{align}
where $\Delta_0 = \munmcq[1][M_0]$ for a user-chosen integer $M_0$, and 
\begin{align}
    \Delta_\ell = \munmcq[1][M_0 2^{\ell}]-\frac{1}{2}\left(\munmcq[1][M_0 2^{\ell-1}]^{(a)}+\munmcq[1][M_0 2^{\ell-1}]^{(b)}\right)
\end{align}
for $\ell>0$, where the samples of $\theta'$ used in $\munmcq[1][M_0 2^{\ell}]$ are split equally between 
$\munmcq[1][M_0 2^{\ell-1}]^{(a)}$ and $\munmcq[1][M_0 2^{\ell-1}]^{(b)}$ to form an antithetic coupling~\citep{giles2014antithetic} through the sharing.
They then introduce an importance sampler to produce an unbiased estimate of the infinite sum $\sum_{\ell=0}^\infty \Delta_\ell$ with a single sampled term, such that we have 
\begin{align}
\label{eq:mlmc_est}
    \text{EIG}_{\theta}(\xi)
    &= \E\left[\Delta_{\ell} / r(\ell) \right],
\end{align}
where the expectation is now over both $\ell \sim r(\ell)$ and $\Delta_{\ell}$.
The EIG can now be unbiasedly estimated through direct Monte Carlo estimation of~\eqref{eq:mlmc_est}.


Critically, the antithetic coupling used means that it is possible, under weak assumptions, to construct $r(\ell)$ such that the resulting estimators (and their gradients with respect to $\xi$) have both finite expected variance and cost.
Specifically,~\citep{goda2022unbiased} show that these properties hold for most problems when we take $r(\ell)\propto 2^{-\tau \ell}$ for any $1<\tau<2$.
While the cost of the resulting estimators can still be significant (as each sample of $\Delta_{\ell}$ itself requires $M_0 2^{\ell}+1$ likelihood evaluations), it can provide noticeable theoretical and practical gains.
In particular, it allows convergence rates $\mathcal{O}(C^{-1/2})$ in the total cost, $C$, to be achieved, thereby recovering the standard (unnested) Monte Carlo rate.

\subsection{Functional and Variational Approaches}

Another important area of recent progress is in the development of approaches that utilize some kind of functional approximation for an intractable nested term (e.g. $p(y|\theta)$ or $p(\theta|y,\xi)$)~\citep{kleinegesse2018efficient,foster2019variational,foster2020unified,foster2021dad,ivanova2021implicit,huan2013simulation,overstall2020bayesian,kennamer2022design}, rather than relying solely on sampling for estimation.
The aforementioned Laplace approximations provide a simple classical example of this, but a number of more advanced and theoretically-grounded approaches have recently been introduced that can provide consistent estimates, substantially increase efficiency compared to naive NMC estimators, improve synergy with the design optimization process itself, and accommodate implicit likelihood models.


To explain the motivations for these approaches, consider the form of the EIG in~\eqref{eq:eig}, where the nested term we need to estimate is $p(y|\xi)$.
Rather than estimating it from scratch each time we sample a new $y_n$, we can note that it will usually have a degree of smoothness that we can exploit to transfer information between estimates.
In fact, we can view the problem as a regression from $y$ to $p(y|\xi)$, where our estimates of $p(y|\xi)$ provide unbiased noisy evaluations of the target.
Alternatively, we can also think about estimating $p(y|\xi)$ using a simple density estimator trained on samples of $y$ (utilizing the fact we can draw from the joint, $p(\theta)p(y|\theta,\xi)$).
Either way, our learned approximation, $q(y|\xi)$, can then be used to directly replace the intractable $p(y|\xi)$ term in~\eqref{eq:eig_mc}, after which we can estimate the EIG via standard Monte Carlo.


This two-stage process can provide substantial computational savings, as the costs from approximating $p(y|\xi)$ and the subsequent Monte Carlo estimation are now additive, not multiplicative.
Indeed,~\citep{foster2019variational} shows that such an approach can theoretically recover convergence rates of $\mathcal{O}(C^{-1/2})$ in total cost $C$ (though, in practice, there are usually limits on how perfect a functional approximation we can learn).


\subsubsection{Variational Approaches} An intriguing property of this framework is that it turns out that approximating $p(y|\xi)$ with $q(y|\xi)$ actually produces a \emph{variational bound} on the EIG whenever the latter forms a valid normalized density~\citep{foster2019variational}, such that
\begin{align}
 \text{EIG}_{\theta}(\xi) 
 &\le \E_{p(\latent)p(y|\latent, \xi)}\left[ \log p(y|\latent, \xi) - \log q(y|\xi)\right],
 	\label{eq:var_marg}
\end{align}
with equality holding when $\forall y: q(y|\xi)=p(y|\xi)$.
This variational bound is distinct from those on the log evidence commonly used by the approximate inference~\citep{blei2016variational,hoffman2013stochastic} and variational auto-encoder communities~\citep{kingma2014adam,rezende2014stochastic,burda2015importance}, but it behaves in very much the same way and can be used as an objective for optimizing our approximation $q(y|\xi)$.


In fact, one can construct a whole host of different variational bounds for the EIG by approximating different terms in the expectation.
For example, if we introduce an amortized inference network~\citep{stuhlmuller2013learning,kingma2014auto}, $q(\theta|y,\xi)$, that maps from outcomes to approximations of
the corresponding posterior, $p(\theta|y,\xi)$, then we instead get the variational lower bound~\citep{foster2019variational,pacheco2019variational,barber2003imalgorithm}
\begin{align}
 \text{EIG}_{\theta}(\xi) 
 &\ge \E_{p(\latent)p(y|\latent, \xi)}\left[ \log q(\theta|y,\xi) - \log p(\theta)\right],

 	\label{eq:var_post}
\end{align}
with equality again holding when the approximation is exact.
The expectation of the NMC estimator in~\eqref{eq:nmcq} even turns out to itself be a variational upper bound~\citep{foster2019variational},
\begin{align}
    \label{eq:vnmc}
   
    \text{EIG}_{\theta}(\xi) & \le \E_{p(\latent)p(y|\latent, \xi)}\left[\munmcq[1][M]\right],
\end{align}
which can be arbitrarily tightened by increasing $M$.
In both these cases we can also use the learned $q(\theta|y,\xi)$ as a proposal for an efficient NMC estimator using~\eqref{eq:nmcq}, allowing for further reduction of the estimator bias if needed.


Note that these are not the only variational bounds one can construct (see e.g.~\citep{foster2019variational,foster2020unified,kleinegesse2020mine,ivanova2021implicit,foster2021variational}).
Furthermore, as the EIG is equivalent to the mutual information between $y$ and $\theta$, a number of methods for optimizing mutual information outside of the experimental design setting can also be applied~\citep{poole2018variational,belghazi2018mutual,oord2018representation,guo2021tight}.
Indeed,~\eqref{eq:var_post} is exactly equivalent to the classical mutual information bound of~\citep{barber2003imalgorithm}.

\subsubsection{Estimation for Implicit Models}
\label{sec:implicit}

Another important class of functional-approximation approaches for EIG estimation are based on accommodating implicit likelihood models, wherein we can draw samples for $y|\theta,\xi$, but the density $p(y|\theta,\xi)$ is intractable.
This introduces an additional intractable term that must be approximated.


Some approaches have been introduced that approximate this density in isolation~\citep{huan2013simulation,overstall2020bayesian}, while~\citep{kleinegesse2018efficient} use the approach of~\citep{thomas2022likelihood} to estimate the ratio $p(y|\theta,\xi)/p(y|\xi)$ using logistic regression.
Variational bounds that allow for implicit likelihood models (e.g.~\eqref{eq:var_post}) have been also developed~\citep{kleinegesse2020mine,ivanova2021implicit,foster2019variational,foster2020unified}, inspired by analogous bounds used in the representation learning community~\citep{belghazi2018mutual,oord2018representation,nguyen2010nwj}.
Historically, approximate Bayesian computation techniques have also been used to deal with implicit likelihoods in a BED context~\citep{drovandi2013bayesian,Hainy2016likelihoodfree,Price2016,dehideniya2018}.


Dealing with implicit priors is typically easier than implicit likelihoods, as one can use formulations based on~\eqref{eq:eig} that avoid calculating the prior density (e.g.~\eqref{eq:nmc}, \eqref{eq:var_marg}).
While some of the recently developed variational bounds reintroduce a need to calculate the prior density (e.g.~\eqref{eq:var_post}, \eqref{eq:vnmc}), there are some that do not, critically including some that also allow implicit likelihood models as well~\citep{kleinegesse2020mine,ivanova2021implicit}.


\subsection{Optimization}

The ultimate aim in BED is to produce the best possible designs: the EIG is an objective that we wish to optimize, rather than simply estimate.
Historically, many of the most commonly used optimization schemes for this have acted as wrappers around the estimator.
Examples of this include simple discretization and enumeration schemes~\citep{carlin1998approaches,ryan2003estimating,watson2017quest+,van2003optimal,lyu2019ultra,han2004bayesian,gillespie2019efficient} (sometimes coupled with dynamic resource allocation~\citep{rainforth2017thesis,vincent2017darc}), Bayesian optimization~\citep{foster2019variational,kleinegesse2018efficient,von2019optimal}, evolutionary algorithms~\citep{hamada2001finding,price2018induced}, and co-ordinate exchange methods~\citep{meyer1995coordinate,overstall2017bayesian}.
Unfortunately, all these approaches can be extremely slow to run, as they require many individual EIG estimations to be performed during the optimization.
They also often scale poorly to high-dimensional design spaces, as they have limited ability to exploit problem-specific information like gradients.


Others have suggested sampling-based approaches that couple the outer sampling from $p(\theta)p(y|\theta,\xi)$ with the optimization of $\xi$~\citep{ryan2014towards,clyde1996exploring,drovandi2013bayesian}, often employing extended spaces to concentrate sampling around the optimum of the EIG~\citep{amzal2006bayesian,muller2004optimal,muller2005simulation,cook2008optimal,kuck2006smc,stroud2001optimal}.
These approaches can sometimes provide efficiency and scaling benefits over more black-box optimizers, but are still typically quite expensive procedures.
They further tend to require customized proposals to be constructed, which can be challenging, especially for high-dimensional problems.


A significantly more scalable approach (when applicable) is to use some sort of stochastic gradient ascent (SGA) scheme~\citep{robbins1951stochastic}, wherein we side-step direct approximation of the EIG and instead leverage estimates of its derivative with respect to $\xi$.
Analogously to deep learning contexts~\citep{lecun2015deep}, it is typically possible here to effectively utilize high-variance, small-sample gradient estimates in the optimization, thereby also providing substantial efficiency gains.
Such an SGA approach was first suggested in~\citep{huan2014gradient}, who use gradients of the basic NMC estimator given in~\eqref{eq:nmc}.
Their method was more recently revisited by~\citep{carlon2020nesterov}, who also consider using SGA with nested Laplace approximations and the more general NMC estimator given in~\eqref{eq:nmcq}.


Unfortunately, both these approaches inevitably introduce bias into the optimization.
This, though, is where the full potential of the aforementioned recent developments in debiasing schemes and variational bounds really becomes apparent: both can provide a consistent mechanism for optimizing the EIG with SGA.


Foster et al.~(2020)~\cite{foster2020unified} first demonstrated such a unified stochastic gradient approach to BED, by simultaneously maximizing variational \emph{lower} bounds to the EIG (e.g.~\eqref{eq:var_post} but not~\eqref{eq:var_marg} or~\eqref{eq:vnmc}) with respect to both the variational and design parameters.
They also show that the small adjustment to~\eqref{eq:vnmc} of including the outer $\theta$ sample in the estimate of $p(y|\xi)$ changes it from an upper to a lower bound, so that it can be used for SGA:
\begin{align}
\label{eq:ace}
    \text{EIG}_{\theta}(\xi) \ge \E\left[\log \frac{p(y|\theta_0,\xi)}{\frac{1}{M+1}\sum_{m=0}^M \frac{p(y|\theta_m,\xi)p(\theta_m)}{q(\theta_m|y,\xi)}}\right],
\end{align}
where $\theta_0,y \sim p(\theta)p(y|\theta,\xi)$ and each $\theta_m | y \overset{\scriptstyle{\text{i.i.d.}}}{\sim} q(\theta_m|y,\xi)$ for $m\ge 1$.
This contrastive bound again recovers the true EIG when $q(\theta|y,\xi)$ matches the true posterior, while allowing the bound tightness to be controlled through $M$.
It is also useful for evaluation, as it can be used in conjunction with~\eqref{eq:nmcq} to provide upper and lower limits on the true EIG.
Variations on this unified variational SGA approach to BED have since been developed by~\citep{kleinegesse2021gradientbased,zhang2021sagabed,foster2021dad,ivanova2021implicit}.


The aforementioned MLMC approach of~\citep{goda2022unbiased} alternatively provides unbiased, finite-variance estimates of the EIG gradient, $\nabla_{\xi} \text{EIG}_{\theta}(\xi)$, itself, which can be directly used to perform SGA.
This avoids the need for any variational distribution---and thus any limitations originating from the variational family---but tends to result in higher per-sample costs and variances.


An inevitable problem with all these gradient-based approaches occurs when the design space or likelihood function is not continuous, for example because some design decisions are discrete.
Relaxation schemes can, in principle, allow such cases to be dealt with~\citep{ivanova2022efficient,zhang2021sagabed}, but further work is still required to fully address this problem.
\subsection{Policy-Based BAD}

The emergence of policy-based BAD methods has led to a significant improvement in state-of-the-art performance in terms of both deployment speed and design quality~\citep{foster2021dad,ivanova2021implicit,blau2022optimizing}.
However, they still represent a fledgling approach and we believe their potential stretches far beyond what has been achieved so far.
We thus anticipate substantial opportunities in developing new policy-based approaches to improve and generalize current methods.


Perhaps the biggest challenge here, and indeed a key challenge for BED more generally, will be scaling to larger and more complex problems.
Though significant progress on scalability has already been made in recent years, current methods can still be limited by a number of factors, such as the dimensionality and smoothness of both the model and the design space, and the length of the experiment sequence, $T$.
Dealing with discrete design components can also be difficult.
We believe that it should be possible to make substantial improvements in our ability to deal with such challenges by making advancements on a number of different fronts, such as network architectures, objectives and estimators, and training mechanisms.  


\subsection{Linking with Related Areas}

The BED literature is spread across a wide variety of different fields, with papers appearing in, for example, statistics~\citep{prangle2022bayesian,overstall2022properties}, machine learning~\citep{foster2021dad,blau2022optimizing}, engineering~\citep{carlon2020nesterov,beck2018fast}, physics~\citep{huan2013simulation}, and social science~\citep{myung2013,kasy2021adaptive} venues.
This somewhat scattered nature has sometimes led to a, potentially problematic, detachment between research threads.
Moreover, some important links between BED and related areas have recently started to become apparent, with these connections ready to be exploited.


Perhaps the most obvious related literature is that of Bayesian active learning (BAL)~\citep{houlsby2011bayesian,gal2017deep,golovin2010near,mackay1992information,kossen_active_2020,kapoor2007active,kossen2022active}.
BAL can be viewed as an important special case of BAD, wherein our aim is to adaptively train a machine learning model in a data-efficient manner by iteratively acquiring the required labels.
Here the designs correspond to the inputs we acquire labels for, which are typically chosen from a finite pool of unlabeled datapoints.
This is most commonly done by following the traditional BAD approach outlined in Section~\ref{sec:BAD} (often with datapoints chosen in batches~\citep{kirsch2019batchbald,pinsler19bayesian}), with the EIG for the model parameters typically referred to as the \emph{Bayesian active learning by disagreement} (BALD) score in the BAL literature~\citep{houlsby2011bayesian}.


We believe that there is much to be gained in reestablishing links and transferring ideas between the intrinsically connected BAL and general BED literatures.
For example, the BAL community has had notable success in dealing with much larger and higher-dimensional datasets---like those seen in computer vision~\citep{gal2017deep,beluch2018power} and natural language processing~\citep{siddhant2018deep,margatina2021bayesian}---than have typically been considered elsewhere.
On the other hand, BAL has seen limited application outside of classification problems with a modest number of classes, relying on enumeration of $y$ and estimators like~\eqref{eq:eig_finite} to keep costs manageable.
Even for such cases, it can still experience computational difficulties when performing the large batch acquisitions often used in practice~\citep{kirsch2019batchbald,pinsler19bayesian}.
The recent computational developments discussed earlier are well-placed to address these shortfalls, while there is also obvious potential in applying policy-BAD approaches in the BAL setting.
More fundamentally, it has also recently been suggested that the BALD score is itself a sub-optimal application of BED for BAL~\citep{bickfordsmith2023epig}, as we care about maximizing information gain for downstream \emph{predictions}, not the model parameters.


Another under-explored area of overlap is between BAD and model-based approaches to reinforcement learning (RL)~\citep{sutton2018reinforcement,szepesvari2010algorithms}, especially Bayesian RL methods~\citep{igl2018deep,ghavamzadeh2015bayesian}.
Though few links have historically been drawn between the fields, the concept of a design policy has a clear analogy to the policies used in RL.
Indeed,~\citep{huan2016sequential} explicitly first introduced the idea of a BAD policy through an RL formulation, while others have also recently directly used RL techniques to learn design policies~\citep{blau2022optimizing,lim2022policy,shen2021bayesian}.


Foster (2021)~\citep{foster2021variational} recently showed that BAD can be viewed as a Bayes adaptive Markov decision process~\citep{guez2012efficient,ross2007bayes,duff2002optimal} with information-based rewards.
Thus, while there are clear differences in how policies are trained and utilized between BAD and Bayesian RL, there is a concrete overlap in problem formulation.
There are also strong links between the utility functions used by BED and the concept of exploration in RL~\citep{foster2021variational,jorke2022simple}, where information-based reward signals have also been used~\citep{sun2011planning,schmidhuber2010formal}.
With these links now established, but largely unexplored, we believe that further cross-dissemination of ideas between the fields will be of substantial mutual benefit.


\subsection{Model Misspecification and Downstream Analysis}

Despite its theoretical elegance and asymptotic guarantees~\citep{paninski2005asymptotic}, BED, and in particular BAD, can suffer from serious pathologies in practice if our model is misspecified~\citep{kennedy2001bayesian,brynjarsdottir2014learning,grunwald2017inconsistency,kleijn2012bernstein,feng2015optimal}, that is there is no $\theta^*$ such that $p(y|\theta=\theta^*,\xi)$ matches the true distribution for $y|\xi$.
While such misspecification is an inevitable issue in any Bayesian setting, BED approaches can be particularly sensitive because we are not only using our model to fit the data, but also to gather new data as well.
Most problematically, because Bayesian approaches only capture uncertainty in the model parameters, and not any uncertainty about whether the model itself is correct, one can occasionally see catastrophic failure cases where BAD becomes stuck querying similar designs, producing poor-quality datasets~\citep{vincent2017darc}.
An extreme example of this is given by linear regression models, where the EIG of the regression coefficients is always maximized at the extrema of the inputs, regardless of the chosen prior.




While misspecification is not something we can realistically hope to eliminate completely, at least not in a general BED context, there is still much to be done to better understand and manage it.
At present, there is only limited literature on both its theoretical~\citep{farquhar2020statistical,fudenberg2017active,overstall2021bayesian} and empirical implications~\citep{sloman2022characterizing,feng2015optimal}, with little done to investigate the precise failure mechanisms that can occur.
Similarly, only a small number of practical methods for mitigating the effects of such misspecification have been suggested~\citep{overstall2021bayesian,go2022robust,ouyang2016,etzioni1993optimal}.
Given the potential dangers of model misspecification in BED, it is clear that more work is needed on both of these fronts.


An important related issue can occur when the gathered data is utilized for purposes other than updating the model used in the BED formulation.
The likelihood principle~\citep{berger1988likelihood,barnard1962likelihood} theoretically provides protection against downstream Bayesian analysis of the gathered data yielding erroneous conclusions.
However, in practice, the data-gathering process can still significantly influence not only the informativeness of the data for other downstream problems, but also the model selection and checking processes.
Moreover, gathered data is sometimes used downstream for entirely non-Bayesian purposes, such as training a machine learning model via empirical risk minimization.
This is a scenario that regularly occurs in a Bayesian active learning context~\citep{farquhar2020statistical}, wherein accurate inference is often infeasible and one resorts to low-fidelity posterior approximations~\citep{pinsler19bayesian,kirsch2019batchbald,siddhant2018deep}, or even entirely non-Bayesian updates~\citep{beluch2018power,gal2017deep,margatina2021bayesian}.
More work is required to understand the influence of using BED to gather data in such contexts.




\subsection{Models and Applications}

As a model-based approach, the performance of BED is only ever as good as the underlying model.
Improvements in models themselves will therefore always be one of, if not the, most important vectors for advancing BED.
Though most such advancements will inevitably be problem-specific, there are still two important general points worth noting.


First, recent developments in BED for implicit likelihoods~\citep{kleinegesse2020mine,ivanova2021implicit,overstall2020bayesian,foster2019variational} have significant potential implications for model development.
It is often much easier to construct an accurate \emph{simulator} for experiment outcomes than a classical closed-form likelihood.
Moreover, many such simulators already exist in most scientific fields, from quantum logic gates~\citep{wittler2021integrated}, to large-scale climate simulations~\citep{fanourgakis2019evaluation}.
Such simulators can be interpreted as implicit likelihood models~\citep{rainforth2017thesis}, and therefore modern approaches allow them to be directly exploited for BED.


Second, the aforementioned substantial recent computational advancements are potentially transformative in the complexity of the models that we can consider using with BED.
As such, practitioners should look to embrace this freedom and develop models that are accurate to their beliefs, rather than restricting themselves to restricted models which allow easy estimation, as has often been the case in the past.
Further, we should now consider BED as a viable approach for larger and more complex problems than those it has most commonly previously been utilized for.
Indeed, finding the limits of current approaches is an important area for future work in its own right.






\section{Introduction}
\label{sec:introduction}

\input{intro}

\section{Formalizing Information Gain}
\label{sec:background}

\input{background}

\section{Estimation and Optimization}
\label{sec:estimation}

\input{estimation}

\vspace{-2pt}
\section{From Designs to Policies}
\label{sec:policy}
\vspace{-2pt}

\input{policy}

\section{Future Directions}


\input{future.tex}






\begin{acks}[Acknowledgments]
We would like to thank Dennis Prangle and Christian Robert for inviting us to write this paper.
\end{acks}

\begin{funding}
Desi R Ivanova was supported by the EPSRC Centre for Doctoral Training in Modern Statistics and Statistical Machine Learning (EP/S023151/1).
Freddie Bickford Smith was supported by the EPSRC Centre for Doctoral Training in Autonomous Intelligent Machines and Systems (EP/S024050/1).
\end{funding}




\bibliographystyle{imsart-number}


