\section{Introduction}
\input{1_intro}

\section{Preliminaries}
\label{sec:model}
\input{2_model}

\section{On the efficacy of the per-channel target ROIs and budgets as levers in solving the global problem}
\label{sec:futil}
\input{3_futil}


\section{Optimization algorithm for per-channel budgets under bandit feedback}
\label{sec:online}
\input{4_online}


\section{Generalizing to autobidding in multi-item  auctions}
\label{sec:multi}
\input{5_multislot}

\section{Additional discussions}
\label{sec:extend}
\input{6_extensions}


\bibliographystyle{informs2014}

\subsection{Main contributions}
\paragraph{1. Modelling ad procurement through per-channel ROI and budget levers.} In Section \ref{sec:model} we develop a novel model for online advertisers to optimize over the per-channel ROI and budget levers to maximize total conversion over channels while respecting a global ROI and budget constraint. This multi-channel optimization model closely imitates real-world practices (see Figure \ref{fig:manager} for evidence), and to the best of our knowledge is  the first of its kind to characterize advertisers' interactions with channels to run ad campaigns. 
\paragraph{2. Solely optimizing per-channel budgets  are sufficient to maximize conversion.} In Theorem \ref{thm:roifail} of Section \ref{sec:futil}, we show that solely optimizing for per-channel ROIs is inadequate to optimize conversion across all channels, possibly resulting in arbitrary worse total conversions compared to the hypothetical global optimal where advertisers can optimize over individual auctions. In contrast, in Theorem \ref{thm:budgonlysuff} and Corollary \ref{cor:roiredund} we show that solely optimizing for per-channel budgets allows an advertiser to achieve the global optimal.
\paragraph{3. Algorithm to optimize per-channel budget levers.} Under a realistic bandit feedback structure where advertisers can only observe the total conversion and spend in each channel after making a per-channel budget decision, in Section \ref{sec:online} we develop an algorithm that augments stochastic gradient descent (SGD) with the upper-confidence bound (UCB) algorithm, and eventually outputs within $T$ iterations a per-channel budget profile with which advertisers can achieve $\mathcal{O}(T^{-1/3})$ approximation accuracy in total conversion to that of the optimal per-channel budget profile, and a $\mathcal{O}(T^{-1/2})$ violation in both global budget and ROI constraints.  Our algorithm relates to constrained convex optimization with uncertain constraints and bandit feedback under a ``one point estimation'' regime, and to the best of our knowledge, our proposed algorithm is the first to handle such a setting; see more discussions in Section \ref{subsec:litrev} and Remark \ref{rmk:onepoint} of Section \ref{sec:online}. Finally, we also present an extended version of our algorithm that achieves the same $\mathcal{O}(T^{-1/3})$ conversion accuracy, while respecting both  constraints exactly.

\paragraph{4. Extensions to general advertiser objectives and mutli-impression auctions.} In Sections \ref{sec:multi} and \ref{sec:extend}, we shed light on the applicability of our results in Section \ref{sec:futil} and \ref{sec:online} to more general settings  when auctions correspond to the sale of multiple auctions, or when advertisers aim to optimize a private cost model instead of conversion.  

\subsection{Related works.}
\label{subsec:litrev}
Generally speaking, our work focuses on advertisers' impression procurement process or the interactions between advertisers and impression sellers, which has been addressed in a vast amount of literature in mechanism design and online learning; see e.g. \cite{braverman2018selling,deng2019prior,golrezaei2019dynamic,golrezaei2019incentive,balseiro2019black, golrezaei2021bidding} to name a few. Here, we review literature that relate to key themes of this work, namely autobidding, budget and ROI management, and constrained optimization with bandit feedback. 
\paragraph{Autobidding. } There has been a rich line of research that model the autobidding setup as well as budget and ROI management strategies. The autobidding model has been formally developed in \cite{aggarwal2019autobidding}, and has been analyzed through the lens of welfare efficiency or price of anarchy in \cite{deng2021towards,balseiro2021robust,deng2022efficiency,mehta2022auction}, as well as individual advertiser fairness in \cite{deng2022fairness}. The autobidding model has also been compared to classic quasi-linear utility models in \cite{balseiro2021landscape}. The autobidding model considered in these papers assume advertisers can directly optimize over individual auctions, whereas in this work we address a more realistic setting that mimics practice where advertisers can only use levers provided by channels, and let channels procure ads on their behalf. 
\paragraph{Budget and ROI management.} 
Budget and ROI management strategies have been widely studied in the context of mechanism design and online learning. \cite{balseiro2017budget} studies the ``system equilibria'' of a range of budget management strategies in terms of the platforms’ profits and advertisers’ utility; \cite{balseiro2019learning,balseiro2022best} study online bidding algorithms (called pacing) that help advertisers achieve high utility in repeated second-price auctions while maintaining a budget constraint, whereas \cite{feng2022online} studies similar algorithms but considers respecting a long term ROI constraint in addition to a fixed budget. All of these works on budget and ROI management focus on bidding strategies in a single repeated auction where advertisers' decisions are bid values submitted directly to the auctions. In contrast, this work focuses on the setting where advertisers procure ads from multiple auctions through channels, and make decisions on how to adjust the per-channel ROI and budget levers while leaving the bidding to channels' blackbox algorithms. 

\paragraph{Online optimization. }
Section \ref{sec:online} where we develop an  algorithm to optimize over per-channel target ROI and budgets relates to the area of convex constrained optimization with bandit feedback (also referred to as zero-order or gradient-less feedback) since in light of Lemma \ref{lem:Vstruct} in Section \ref{sec:online} our problem of interest is also constrained and convex. First, there has been a plenitude of algorithms developed for deterministic constrained convex optimization under a bandit feedback structures where function evaluations for the objective and constraints are non-stochastic. Such algorithms include filter methods
\cite{audet2004pattern,pourmohamad2020statistical}, barrier-type methods \cite{fasano2014linesearch, dzahini2022constrained}, as well as Nelder-Mead type algorithms \cite{bHurmen2006grid,audet2018mesh}; see \cite{nguyen2022stochastic} and references therein for a comprehensive survey. In contrast to these works, our optimization algorithm developed in Section \ref{sec:online} handles 
noisy bandit feedback. 


Regarding works that also address stochastic settings,  \cite{flaxman2004online} presents online optimization algorithms under the \textit{known constraint} regime, which assumes the optimizer can evaluate whether all constraints are satisfied, i.e. constraints are analytically available. Further, the algorithm achieves a $\mathcal{O}(T^{-1/4})$ accuracy. In this work, our setting is more complex as the optimizer (i.e. the advertiser) cannot tell whether the ROI constrained is satisfied (due to unknown value and cost distributions in each channels' auctions). Yet our proposed algorithm can still achieve a more superior $\mathcal{O}(T^{-1/3})$ accuracy. Most relevant to this paper is the very recent works \cite{usmanova2019safe,nguyen2022stochastic}, which considers a similar setting to ours that optimizes for a constrained optimization problem where the objective and constraints are only available through noisy function value evaluations (i.e. unkownn constraints). \cite{usmanova2019safe} focuses on a special (unknown) linear constraint setting, and  \cite{nguyen2022stochastic} extends to general convex constraints. Although \cite{usmanova2019safe} and \cite{nguyen2022stochastic} achieve  $\mathcal{O}(T^{-1})$ and $\mathcal{O}(T^{-1/2})$ approximation accuracy to the optimal solution which contrasts our $\mathcal{O}(T^{-1/3})$ accuracy, these works imposes several assumptions that are stronger than the ones that we consider. First, the objective and constraint functions are strongly smooth (i.e. the gradients are Lipschitz continuous) and \cite{nguyen2022stochastic} further assume strong convexity. But in our work,  our objectives and constraints are piece-wise linear and do not satisfy such salient properties.  Second, and most importantly, both works consider a setting with ``two point estimations'' that allows the optimizer to access the objective and constraint function values twice in each iteration, enabling more efficient estimations. This work, however, lies in the one-point setting where we can only access function values once per iteration. Finally, we remark that the optimal accuracy/oracle complexity for the one-point setting for constrained (non-smooth) convex optimization with bandit feedback and unknown constraints remains an open question; see  Remark \ref{rmk:onepoint} in Section \ref{sec:online} for more details. We refer readers to Table 4.1 in \cite{larson2019derivative} for a survey on best known bounds under different one-point bandit feedback settings.
\subsection{Sequential multi-channel campaigns}





\subsection{Sequential multi-channel campaigns}






\subsection{The SGD-UCB algorithm to optimize per-channel budgets}
\label{subsec:algo}
Here, we describe our algorithm to solve for optimal per-channel budgets w.r.t.  $\textsc{CH-OPT}(\mathcal{I}_{B})$. Similar to most algorithms for constrained optimization, we take a dual stochastic gradient descent (SGD) approach; see a comprehensive survey on dual descent methods in \cite{bertsekas2014constrained}. First, we consider the Lagrangian functions w.r.t. $\textsc{CH-OPT}(\mathcal{I}_{B})$ where we let  $\bm{c} = (\lambda, \mu) \in \mathbb{R}_{+}^{2}$ be the dual variables corresponding to the ROI and budget constraints, respectively:
\begin{align}
    \label{eq:Lagrangian}
    \mathcal{L}_{j}(\rho_{j}, \bm{c}; \bm{z}_{j}) =  (1+\lambda)V_{j}(\rho_{j};\bm{z}_{j}) - (\lambda \gamma  + \mu) \rho_{j} ~\text{ and }~\mathcal{L}_{j}(\rho_{j}, \bm{c}) = \mathbb{E}\left[  \mathcal{L}_{j}(\rho_{j}, \bm{c}; \bm{z}_{j})\right]\,.
\end{align}
Then, in each period $t\in [T]$ given dual  variables $\bm{c}_{t} = (\lambda_{t},\gamma_{t})$, SGD decides on a primal decision, i.e. per-channel budget $(\rho_{j,t})_{j\in[M]}$ by optimizing the following:
\begin{align}
\label{eq:SGDprimal}
    \rho_{j,t} ~=~ \arg\max_{\rho_{j}\in [0,\rho]}\mathcal{L}_{j}(\rho_{j}, \bm{c}_{t};\bm{z}_{t})\,.
\end{align}
Having observed the realized values $\left(V_{j}(\rho_{j,t};\bm{z}_{t})\right)_{j\in[M]}$ (note that spend is $(\rho_{j,t})_{j\in [M]}$ in light of Assumption \ref{ass:budgetbind}),  we calculate the current period violation in budget and ROI constraints, namely  $g_{1,t}:= \sum_{j \in M} \left(V_{j}(\rho_{j,t};\bm{z}_{t}) -  \gamma \rho_{j,t}\right)$ and $g_{2,t} = \rho - \sum_{j\in [M]} \rho_{j,t}$. Next, we update dual variables via $ 
\lambda_{t+1} = \left(\lambda_{t} - \eta g_{1,t}\right)_{+} \text{ and } \mu_{t+1}  = \left(\mu_{t} - \eta g_{2,t}\right)_{+}$, where $\eta$ is some pre-specified step size.\footnote{Here, the dual updates follow the vanilla gradient descent approach, and one can also employ more general mirror descent updates; see e.g. \cite{balseiro2022best}.}

However, the above SGD approach faces a fatal drawback, namely we cannot realistically find the primal decisions by solving Eq. \eqref{eq:SGDprimal} since the function $\mathcal{L}_{j}(\cdot, \bm{c}_{t};\bm{z}_{t})$ is unknown due to the  bandit feedback  structure. Therefore, we provide a modification to SGD to handle this issue. Before we present our approach, we briefly note that although bandit feedback prevents the naive application of SGD for our problem of interest, this may not be the case in other online advertising scenarios that involve relevant learning tasks, underlining the challenges of our problem; see following Remark \ref{rmk:banditfeedback} for details.
\begin{remark}
\label{rmk:banditfeedback}
Our problem of interest under bandit feedback is more difficult than similar problems in related works that study online bidding strategies under budget and ROI constraints; see e.g. \cite{balseiro2017budget,balseiro2022best,feng2022online}. To illustrate, consider for instance  \cite{balseiro2017budget} in which a budget constrained advertiser's primal decision at period $t$ is to submit a bid value  $b_{t}$ after observing her value $v_{t}$. The advertiser competes with some unknown  highest competing bid $d_{t}$ in the market, and after submitting bid $b_{t}$, does not observe $d_{t}$ if she does not win the competition, which involves a semi-bandit feedback structure. Nevertheless, the corresponding Lagrangian under SGD takes the special form $\mathcal{L}_{j}(b, \mu_{t};\bm{z}_{t}) =\left(v_{t} - (1+\mu_{t})d_{t}\right)\mathbb{I}\{b_{t} \geq d_{t}\}$ where $\mu_{t}$ is the dual variable w.r.t. the budget constraint.  This simply allows an advertiser to optimize for her primal decision by bidding $\arg\max_{b\geq 0} \mathcal{L}_{j}(b, \bm{c}_{t};\bm{z}_{t}) = \frac{v_{t}}{1+\mu_{t}}$. So even though \cite{balseiro2017budget,balseiro2022best,feng2022online} study dual SGD under bandit feedback, the special structures of their problem instances permits SGD to effectively optimize for primal decisions in each period, as opposed to Eq. \eqref{eq:SGDprimal} in our setting which can not be  solved.
\end{remark}
To resolve challenges that arise with bandit feedback in our model, we take a natural approach to augment SGD with the \textit{upper-confidence bound (UCB)} algorithm, which is well celebrated for solving learning problems under bandit feedback such as multi-arm bandits; see an introduction to bandits in \cite{slivkins2019introduction}. In particular, we first discretize our per-channel buudget decision set $[0,\rho]$ into granular ``arms'' that are separated by some distance $\delta > 0$, so that the discretized per-channel budget decisions become
\begin{align}
\label{eq:budgetdisc}
    \mathcal{A}(\delta) = \{a_{k}\}_{k \in [K]} \text{ where } a_{k} = (k-1)\delta \text{ and }K := \lceil{\rho}/{\delta}\rceil + 1\,.
\end{align}
In the following we will use the terms ``per-channel budget'' and ``arm'' interchangeably. In the spirit of UCB, in each period $t$ we maintain some estimate $\left(\Hat{V}_{j,t}(a_{k})\right)_{j\in[M]}$ of the conversions $\left(V_{j}(a_{k})\right)_{j\in [M]}$ as well as an upper confidence bound $\texttt{UCB}_{j,t}(a_{k})$ for each arm $a_{k}$ using historical payoffs from periods in which arm $a_{k}$ is pulled. Finally, we update primal decisions for each channel $j\in[M]$ using the ``best arm'' $\rho_{j,t} = \arg\max_{a_{k}\in \mathcal{A}(\delta)}~(1+\lambda_{t})\left(\Hat{V}_{j,t}(a_{k}) +  \texttt{UCB}_{j,t}(a_{k})\right) - \left(\lambda_{t} \gamma + \mu_{t} \right)a_{k}$. We summarize our algorithm, called SGD-UCB, in the following Algorithm \ref{alg:omducb}.

\begin{algorithm}[ht]
    \centering
    \caption{SGD-UCB}\label{alg:omducb}
    \footnotesize
    \begin{algorithmic}[1]
    \Require Budget discretization decision set  $\mathcal{A}(\delta)$ defined in Eq.\eqref{eq:budgetdisc}. Step size $\eta > 0$. Initialize $ N_{j,1}(a_{k}) = \Hat{V}_{j,1}(a_{k})  = 0$ for all $j \in [M]$ and $k\in [K]$, and dual variables $\lambda_{1} = \mu_{1}= 0$.
   
   
   
    \\
    \textbf{Output:} Per channel budget.   
	\For{$t = 1 \dots T$ }
	 \State \textbf{Update (primal) per-channel budget.} For each channel $j\in [M]$ set (primal) per-channel budget:
  \begin{itemize}
      \item If $t \leq K$, set $\rho_{j,t} = a_{t}$.
      \item If $t > K$, set
      \begin{align}
        \begin{aligned}
        \label{eq:choosearm}
          & \rho_{j,t} = \arg\max_{a_{k}\in \mathcal{A}(\delta)}~\Hat{V}_{j,t}(a_{k}) +  \texttt{UCB}_{j,t}(a_{k}) - \frac{\lambda_{t} \gamma + \mu_{t}}{1+\lambda_{t}} a_{k}, ~\text{ where } \texttt{UCB}_{j,t}(a_{k}) = \sqrt{\frac{2\log(T)}{N_{j,t}(a_{k})}} 
           \end{aligned}
          \end{align}
  \end{itemize}
    \State  Observe realized values $\left\{V_{j}(\rho_{j,t};\bm{z}_{t}) \right\}_{j \in [M]}$, and update for each arm $k \in [K]$:
    \begin{align}
    \begin{aligned}
        & N_{j,t+1}(a_{k}) =  N_{j,t}(a_{k}) + \mathbb{I}\{\rho_{j,t} = a_{k}\}\\
        & \Hat{V}_{j,t+1}(a_{k}) = \frac{1}{N_{j,t+1}(a_{k})} \left(N_{j,t}(a_{k}) \Hat{V}_{j,t}(a_{k}) + V_{j}(\rho_{j,t};\bm{z}_{t}) \mathbb{I}\{\rho_{j,t} = a_{k}\}\right) \quad, \text{ for } j = 1 \dots M
        \end{aligned}
    \end{align}
    \State \textbf{Update dual variables.} Update dual variables with $g_{1,t}:= \sum_{j \in M} \left(V_{j}(\rho_{j,t};\bm{z}_{t}) -  \gamma \rho_{j,t}\right)$ and $g_{2,t} = \rho - \sum_{j\in [M]} \rho_{j,t}$:
   
   
       
       
       
       
       
       
        \begin{align}
        \label{eq:dualupdate}
        \begin{aligned}
     \lambda_{t+1} = \left(\lambda_{t} - \eta g_{1,t}\right)_{+} \text{ and } \mu_{t+1}  = \left(\mu_{t} - \eta g_{2,t}\right)_{+}
        \end{aligned}
        \end{align}
	\EndFor
 \State Output $\overline{\bm{\rho}}_{T} = \left(\frac{1}{T}\sum_{t\in[T]} \rho_{j,t}\right)_{j\in [M]}$.
    \end{algorithmic}
\end{algorithm}
We remark that there has been very recent works that combine SGD with Thompson sampling which is another well-known algortithm for solving bandit problems (e.g. \cite{ding2021efficient} and references therein), and works that employ SGD in bandit problems (e.g. \cite{han2021generalized}). Yet to the best of our knowledge, approach to augment SGD with UCB is novel.

\subsection{Analyzing the SGD-UCB algorithm}
\label{subsec:analyzealgo}

In this subsection, we analyze the performance of SGD-UCB in Algorithm \ref{alg:omducb}, and present accuracy guarantees on the final output $\overline{\bm{\rho}}_{T} = \left(\frac{1}{T}\sum_{t\in[T]} \rho_{j,t}\right)_{j\in [M]}$ of the algorithm. The backbone of our analysis strategy is to show the cumulative conversion loss over $T$ periods, namely $T\cdot  \textsc{GL-OPT} - \mathbb{E}\left[\sum_{t\in [T]}\sum_{j\in[M]}V_{j}(\rho_{j,t})\right]$ consists of two main parts, namely the error induced by the UCB in our algorithm, and the error due to SGD (or what is typically viewed as the deviations from  complementary slackness), as shows in the following Proposition \ref{prop:ubbenchlag}. Then we further bound each part, respectively.
\begin{proposition}
\label{prop:ubbenchlag}
For any channel $ j\in [M]$ define $\rho_{j}^{*}(t) = \arg\max_{\rho_{j\in[M]}}\mathcal{L}_{j}(\rho_{j};\bm{c}_{t})$ to be the optimal per-channel budget w.r.t. dual variables $\bm{c}_{t}=(\lambda_{t},\mu_{t})_{t\in [T]}$ during period $t\in[T]$. Then we have 
 \begin{align}
 \begin{aligned}
 \label{eq:decompregret}
     & T \cdot\textsc{GL-OPT} - \sum_{t\in [T]}\sum_{j\in[M]}\mathbb{E}\left[V_{j}(\rho_{j,t})\right] \\ ~\leq~&  M\Bar{V}K +
     \sum_{j\in [M]}\underbrace{\sum_{t> K} 
     \mathbb{E}\left[\mathcal{L}_{j}(\rho_{j}^{*}(t), \lambda_{t}, \mu_{t})  - \mathcal{L}_{j}(\rho_{j,t}, \lambda_{t}, \mu_{t})\right]}_{\text{UCB error}}+ \underbrace{\sum_{t >K}\mathbb{E}\left[\lambda_{t}g_{1,t} + \mu_{t}g_{2,t}\right]}_{\text{SGD complementary slackness deviations}}\,,
      \end{aligned}
 \end{align}
\end{proposition}
 where we recall the definitions of $g_{1,t}$ and $g_{2,t}$ in step 4 of Algorithm \ref{alg:omducb}, and the fact that the conversion $ V_{j}(\rho_{j};\bm{z}_{j})$ is bounded above by absolute constant $\Bar{V} \in (0,\infty)$ almost surely for any channel $j\in[M]$, $(\gamma_{j},\rho_{j})$ and realization $\bm{z}_{j}$.

The bound on SGD complementary slackness violation is presented in the following Lemma \ref{lem:boundcs}, and follows a standard analyses for SGD; we refer readers to the proof in Appendix \ref{pf:lem:boundcs}. 
\begin{lemma}[Bounding complementary slackness deviations]
\label{lem:boundcs}
Recall $g_{1,t} = \sum_{j\in [M]}\left(V_{j}(\rho_{j,t};\bm{z}_{t}) - \gamma \rho_{j,t}\right)$, $g_{2,t} = \rho - \sum_{j\in [M]}\rho_{j,t}$ and $\eta > 0$ the step size defined in Algorithm \ref{alg:omducb}. Then we have
    \begin{align}
    \sum_{t>K} \mathbb{E}\left[\lambda_{t}g_{1,t} + \mu_{t}g_{2,t}  \right] ~\leq~ 
    \mathcal{O}\left( \eta  T + \frac{1}{\eta}\right)\,.
    \end{align}
\end{lemma}
\textbf{Challenges in bounding UCB error due to adversarial contexts and continuum-arm dicretization.}
Bounding our UCB error is much more challenging than doing so in classic stochastic multi-arm bandit settings: first, our setup involves discretizing a continuum of arms i.e. our discretization in  Eq.\eqref{eq:budgetdisc} for $[0,\rho]$; second, and more importantly, the dual variables $\{\bm{c}_{t}\}_{t\in[T]}$ are effectively \textit{adversarial contexts} since they are updated via SGD instead of being stochastically sampled from some nice distribution, and correspondingly the Lagrangian function $\mathcal{L}_{j}(a_{k},\bm{c}_{t};\bm{z}_{t})$ can be viewed as a reward function that maps any arm-context pair $(a_{k},\bm{c}_{t})$ to (stochastic) payoffs.  Moreover, the UCB error in Eq.\eqref{eq:decompregret}
concerns a dynamic benchmark $\mathcal{L}_{j}(\rho_{j}^{*}(t),\bm{c}_{t};\bm{z}_{t})$ instead of a single-arm benchmark across all contexts, namely $\max_{a\in\mathcal{A}(\delta)}\mathcal{L}_{j}(a,\bm{c}_{t};\bm{z}_{t})$ . Both continuum-arms and dynamic benchmarks under adversarial contexts have been notorious in making reward function estimations highly inefficient; see e.g. discussions in \cite{agrawal1995continuum,agarwal2014taming}. We further elaborate on additional challenges that adversarial contexts bring about:
\begin{itemize}
    \item \textbf{Boundedness of rewards.} In classic stochastic multi-arm bandtis and UCB, losses in total rewards grow linearly with the magnitude of rewards. In our setting, the reward function, i.e. the Lagrangian function $\mathcal{L}_{j}(a_{k},\bm{c}_{t};\bm{z}_{t})$, scales linearly with the magnitude of contexts (see Eq. \eqref{eq:Lagrangian}, so large contexts (i.e. large dual variables) may lead to large losses.
    \item \textbf{Context-dependent exploration-exploitation tradeoffs.} The typical trade-off for arm exploration and exploitation in our setting depends on the particular values of the contexts (i.e. the dual variables), which means there may exist ``bad'' contexts that lead to poor tradeoffs that require significantly more explorations to achieve accurate estimates of arm rewards than  other ``good'' contexts. We elaborate more in Lemma \ref{lem:ucbregret} and discussions thereof.
\end{itemize}
 In the following, we first handle continuum arm discretization and dynamic contextual benchmarks via analyzing structural properties of the reward (i.e. Lagrangian) functions. Fortunately,  the specific form of conversion functions $V(\rho_{j};\bm{z})$ defined in Eq. \eqref{eq:budperchannelsol} imposes a salient structure on the Lagrangian for pulling an arm. Specifically, the following lemma shows that the Lagrangian is continuous, piece-wise linear, concave, and unimodal\footnote{We say a real-valued function $f:\mathbb{R}\to \mathbb{R}$ is unimodal if there exists some $y^{*}$ such that $f(y)$ strictly increases when $y \leq y^{*}$ and  strictly decreases when $y \geq y^{*}$.}; we present the proof in Appendix \ref{pf:lem:Vstruct}
\begin{lemma}[Structural properties of conversion  and Lagrangian functions]
\label{lem:Vstruct}
\begin{itemize}
\item
For any channel $j\in [M]$ and per-channel budget $\rho_{j}$, the conversion function $V_{j}(\rho_{j})$ is continuous, piece-wise linear, strictly increasing, and concave. In particular,  $V_{j}(\rho_{j})$ takes the form
\begin{align}
\label{eq:Vstruct}
        V_{j}(\rho_{j}) = \sum_{n\in [S_{j}]} \left(s_{j,n} \rho_{j} + b_{j,n}\right)\mathbb{I}\{r_{j,n-1}\leq \rho_{j} \leq r_{j,n}\}\,,
\end{align}
where the parameters $S_{j} \in \mathbb{N}$ and $\{(s_{j,n},b_{j,n},r_{j,n})\}_{n\in [S_{j}]}$ only depend on the support $F_{j}$ and distribution $\bm{p}_{j}$ from which value-to-cost pairs are sampled.  These parameters satisfy
$s_{j,1} > s_{j,2} > \dots > s_{j,S_{j}} > 0$ and $0=r_{j,0} < r_{j,1} < r_{j,2} < \dots < r_{j,S_{j}} =\rho$, as well as $b_{j,n} > 0$ s.t. $s_{j,n} r_{j,n} + b_{j,n} = s_{j,n+1} r_{j,n} + b_{j,n+1}$ for all $n \in [S_{j}-1]$, implying $V_{j}(\rho_{j}) $ is continuous  in $\rho_j$.
\item For any dual variables $\bm{c} = (\lambda,\mu) \in \mathbb{R}_{+}^{2}$, the Lagrangian function $ \mathcal{L}_{j}(\rho_{j},\bm{c})$ defined in Eq. \eqref{eq:Lagrangian} is continuous, piece-wise linear, concave, and unimodal in $\rho_{j}$. In particular,
\begin{align}
\label{eq:lagrrep}
        \mathcal{L}_{j}(\rho_{j}, \bm{c}) =  \sum_{n\in [S_{j}]} \left( \sigma_{j,n}(\bm{c}) \rho_{j} + (1+\lambda)b_{j,n}\right)\mathbb{I}\{r_{j,n-1}\leq \rho_{j} \leq r_{j,n}\}\,,
\end{align}
where the slope $\sigma_{j,n}(\bm{c}) = (1+\lambda) s_{j,n} -(\mu + \gamma \lambda)$. 
Then, this also implies $\arg\max_{\rho_{j}\geq 0} \mathcal{L}_{j}(\rho_{j}, \bm{c}) = \max\{r_{j,n}:n=0,1\dots, S_{j}~,\sigma_{j,n}(\bm{c}) \geq 0\}$.
\end{itemize}
\end{lemma}
In fact,  for any realized value-cost pairs $\bm{z}$, the ``realization 
versions'' of the conversion and Lagrangians functions, namely $V_{j}(\rho_{j};\bm{z}) $ and $\mathcal{L}_{j}(\rho_{j}, \bm{c};\bm{z})$,  also satisfy the same properties as those of $V_{j}(\rho_{j})$ and $\mathcal{L}_{j}(\rho_{j}, \bm{c})$, respectively. We provide a visual illustration for these structural properties in Figure \ref{fig:lagr}. 

\begin{figure}[ht]
\begin{center}
\includegraphics[width=0.8\textwidth]{lagr.png}
\caption{Illustration of Lagrangian functions defined in Eq. \eqref{eq:Lagrangian} with $M_{j} = 2$ auctions in channel $j$, and support $F_{j}$ that contains 3 elements,  $\bm{z}_{(1)} = (\bm{v}_{(1)},\bm{d}_{(1)}) = ((8,2), (2,3))$, $\bm{z}_{(2)} = ((3,4), (1,4))$, $\bm{z}_{(3)} = ((8,1), (4,2))$, and context $\bm{c} = (\lambda,\mu) = (4,2)$. In light of Lemma \ref{lem:Vstruct}, $S_{j} = 5$, where the ``turning points'' $r_{j,0} \dots r_{j,S_{j}}$ are indicated on the x-axis, and the optimal budget w.r.t. $\bm{c}$ is $\arg\max_{\rho_{j}\in [0,\rho]}\mathcal{L}_{j}(\rho_{j};\bm{c}) = r_{j,2}$. The adjacent slopes in Eq. \eqref{eq:adj} are $\sigma_{j}^{-}(\bm{c}) = \sigma_{j,2}(\bm{c})$, and  $\sigma_{j}^{+}(\bm{c}) = \sigma_{j,3}(\bm{c})$, respectively.}
\label{fig:lagr}
\end{center}
\end{figure}




We now handle the reward boundedness issue in the Lagrangian functions defined in Eq. \eqref{eq:Lagrangian} that arise from adversarial contexts. In the following Lemma \ref{lem:dualvarub}, we show that 
the Lagrangian functions, as well as dual variables, are indeed bounded by some absolute constants under a mild feasibility Assumption \ref{ass:feas} stated below:
\begin{assumption}[Strictly feasible global ROI constraints]
\label{ass:feas}
For any realization of value-cost pairs  $\bm{z} =(\bm{v}_{j},\bm{d}_{j})_{j\in[M]} \in F_{1}\times \dots F_{M}$, the realized version of the ROI constraint in \textsc{GL-OPT} defined in Eq. \ref{eq:glopt} is strictly feasible, i.e. the set $\left\{\bm{x}= (\bm{x}_{j})_{j\in [M]}: \bm{x}_{j} \in [0,1]^{m_{j}} \text{ for }\forall j\in [M],~\sum_{j\in[M]}\bm{v}_{j}^{\top} \bm{x}_{j} > \gamma\sum_{j\in[M]}\bm{d}_{j}^{\top}\bm{x}_{j}\right\}$ is nonempty.\footnote{Equivalently, for any realization of value-cost pairs  $\bm{z} = (\bm{v}_{j},\bm{d}_{j})_{j\in[M]}$ there always exists a channel $j\in [M]$ and an auction  $n\in [m_{j}]$ in this channel  whose value-to-cost ratio is at least $\gamma$, i.e. $v_{j,n} >  \gamma d_{j,n}$.}
\end{assumption}
\begin{lemma}[Bounding dual variables and Lagrangian functions]
\label{lem:dualvarub}
Let $(\lambda_{t},\mu_{t})_{t\in [T]}$ be the variables generated from Algorithm \ref{alg:omducb}. Under Assumption \ref{ass:feas}, and assuming the step size $\eta > 0$ satisfies $\eta < \frac{1}{M \sqrt{(\Bar{V}+\gamma\rho)^{2} + \rho^{2}}}$, there exists some absolute constant $C_{F} > 0$ that depends only on the support of value-cost pairs  $F = F_{1} \times \dots \times F_{M}$, as well as aggregate target ROI and budget $(\gamma,\rho)$,  such that $\lambda_{t},\mu_{t} \leq C_{F}$ for all $t \in [T]$. Moreover, for any for any $t\in [T]$, $j\in [M]$ and $\rho_{j} \in [0,\rho]$ we have 
\begin{align}
   -\left(1+\gamma \right)\rhoC_{F}
   ~\leq~ \mathcal{L}_{j}(\rho_{j},\bm{c}_{t})~\leq~(1+C_{F})\Bar{V}\,.
\end{align} 
\end{lemma}
See the proof in Appendix \ref{pf:lem:dualvarub}.

Finally, we address the context-dependent exploration-exploitation tradeoff. We remark that
this tradeoff is embodied in the ``flatness'' of the reward function that depends on adversarial contexts. To illustrate (see e.g. Figure \ref{fig:lagr}), we define the slopes that are adjacent to the optimal budget $\arg\max_{\rho_{j}\in[0,\rho]} \mathcal{L}_{j}(\rho_{j},\bm{c})$ for any $\bm{c} = (\lambda,\mu)$ as followed: assuming the optimal budget $\arg\max_{\rho_{j}\in[0,\rho]} \mathcal{L}_{j}(\rho_{j},\bm{c})$ is located at the $n$th ``turning point'' $r_{j,n}$ we have
\begin{align}
\label{eq:adj}
\begin{aligned}
    & \sigma_{j}^{-}(\bm{c}) = \sigma_{j,n}(\bm{c}) 
    ~\text{ and }~ \sigma_{j}^{+}(\bm{c}) = \sigma_{j,n+1}(\bm{c})
  \end{aligned}  
\end{align}
Similar to standard multi-arm bandits exploration-exploitation tradeoffs, the flatter the slope (e.g. $\sigma_{j}^{-}(\bm{c})$ is close to 0), the more pulls required to accurately estimate rewards for sub-optimal arms on the slope, but the lower the loss in conversion for pulling sub-optimal arms. Our setting is challenging because the magnitude of this tradeoff depends on the adversarial contexts $\bm{c}_{t}$, i.e., the dual variables, which requires delicate treatments. In the following Lemma  \ref{lem:ucbregret} where we bound the UCB error, we handle this context-dependent tradeoff by separately analyzing periods during which the adjacent slopes $\sigma_{j}^{-}(\bm{c})$ and $\sigma_{j}^{+}(\bm{c})$ are less or greater than some parameter $\underline{\sigma}$, and characterize the context-dependent tradeoff w.r.t. flatness of adjacent slopes using $\underline{\sigma}$.
\begin{lemma}[Bounding the UCB error in  per-channel budgets]
\label{lem:ucbregret}
Assume the discretization width $\delta$ satisfies $\delta < \underline{r}_{j}:= \min_{n \in [S_{j}]}r_{j,n} - r_{j,n-1}$, where $S_{j}$ and $\{r_{j,n}\}_{n=0}^{S_{j}}$ are defined in Lemma \ref{lem:Vstruct}. Then we have
\begin{align}
        \sum_{t > K} 
     \mathbb{E}\left[\mathcal{L}_{j}(\rho_{j}^{*}(t), \bm{c}_{t}) - \mathcal{L}_{j}(\rho_{j,t}, \bm{c}_{t})\right]~\leq~ \widetilde{\mathcal{O}}\left(\delta T + \underline{\sigma} T + \frac{1}{\underline{\sigma}\delta} + \frac{1}{\delta T}\right)\,,
    \end{align} 
    where $\underline{\sigma} > 0$ is any small positive absolute constant that we can specify later, and recall $\rho_{j}^{*}(t) = \arg\max_{\rho_{j\in[M]}}\mathcal{L}_{j}(\rho_{j};\bm{c}_{t})$ is the optimal per-channel budget w.r.t. dual variables $\bm{c}_{t}$. $\widetilde{\mathcal{O}}$ hides all logarithmic factors.
\end{lemma}
 We refer the readers to Appendix \ref{pf:lem:ucbregret} for the proof. Note that the parameter $\underline{\sigma}$ will be chosen later. Finally, returning to bounding the UCB error in Proposition \ref{prop:ubbenchlag}, we put together Lemmas \ref{lem:boundcs} and \ref{lem:ucbregret}, and obtain the main result of this section in the following Theorem \ref{thm:regret} whose proof we detail in Appendix \ref{pf:thm:regret}.

\begin{theorem}[Putting everything together]
\label{thm:regret}
Assume assumptions  \ref{ass:budgetbind} and \ref{ass:feas} hold. Let $(\rho_{j,t})_{j\in [M], t\in [T]}$ be the per-channel budgets generated from Algorithm \ref{alg:omducb} and assume we take step size $\eta = \Theta(1/\sqrt{T})$, discretization width $\delta = \Theta(T^{-1/3}) $, and $\underline{\sigma} = \Theta(T^{-1/3})$ in Lemma \ref{lem:ucbregret}. Then for large enough $T$ we have $ T\cdot  \textsc{GL-OPT} - \mathbb{E}\left[\sum_{t\in [T]}\sum_{j\in[M]}V_{j}(\rho_{j,t})\right] \leq \mathcal{O}(T^{2/3})$.
Recalling $\overline{\bm{\rho}}_{T} = \left(\frac{1}{T}\sum_{t\in[T]} \rho_{j,t}\right)_{j\in [M]}$ is the vector of time-averaged per-channel budgets, this implies
\begin{align*}
    \textsc{GL-OPT} - \sum_{j \in [M]}\mathbb{E}\left[V_{j}(\overline{\rho}_{T,j})\right] \leq \mathcal{O}(T^{-1/3})\,,
\end{align*}
as well as approximate constraint satisfaction
\begin{align*}
     \sum_{j \in [M]}\mathbb{E}\left[V_{j}(\overline{\rho}_{T,j}) -\gamma \overline{\rho}_{T,j}\right] \geq -\mathcal{O}(T^{-1/2}),~\text{ and }\rho - \sum_{j \in [M]}\mathbb{E}[\overline{\rho}_{T,j}] \geq -\mathcal{O}(T^{-1/2})
\end{align*}
\end{theorem}

Here, we note that the above approxamate constraint satisfaction is in expectation, similar to our definition of $\textsc{CH-OPT}(\mathcal{I}_{B})$
defined in Eq. \eqref{eq:chopt}. To conclude, we make an important remark that distinguishes our result in Theorem \ref{thm:regret} with related literature on convex optimization.
 

\begin{remark}
\label{rmk:onepoint}
In light of Lemma \ref{lem:Vstruct}, the 
advertiser's optimization problem $\textsc{CH-OPT}(\mathcal{I}_{B})$ in Eq. \eqref{eq:chopt} effectively becomes a convex problem (see Proposition \ref{prop:convexprob} in Appendix \ref{app:onlineadd}). Hence it may be tempting for one to directly employ off-the-shelf convex optimization algorithms. However, our problem involves stochastic bandit feedback, and more importantly, \textit{uncertain constraints}, meaning that we cannot analytically determine whether a primal decision satisfies the constraints of the problem. For example, in $\textsc{CH-OPT}(\mathcal{I}_{B})$, for some primal decision $(\rho_{j})_{j\in[M]}$, we cannot determine whether the ROI constraint  $\sum_{j \in M} \mathbb{E}\left[V_{j}(\gamma_{j},\rho_{j};\bm{z}_{j}) - \gamma D_{j}(\gamma_{j},\rho_{j};\bm{z}_{j})\right]\geq 0$ holds because the distribution $(\bm{p}_{j})_{j\in[M]}$ from which $\bm{z}$ is sampled is unknown. To the best of our knowledge, there are only two recent works that handle a similar stochastic bandit feedback, and uncertain constraint setting, namely \cite{usmanova2019safe} and \cite{nguyen2022stochastic}. Nevertheless, our setting is more challenging because these works consider a ``two-point estimation'' regime where one can make function evaluations to the objective and constraints twice each period, whereas our setting involves ``one-point estimation'' such that we can only make function calls once per period. We note the optimal oracle complexities for unknown constraint convex optimization with one-point bandit feedback, remains an open problem.\footnote{See Table 4.1 in \cite{larson2019derivative} for best known  complexity bounds for one-point bandit feedback setups.}
\end{remark}



\subsection{Extension to strict constraint satisfaction: UCB-SGD-II}
In Theorem \ref{thm:regret}, we showed that the final output of the UCB-SGD Algorithm \ref{alg:omducb} outputs a per-channel budget profile $\overline{\bm{\rho}}_{T} = \left(\frac{1}{T}\sum_{t\in[T]} \rho_{j,t}\right)_{j\in [M]}$ that satisfies both ROI and budget constraints in $\textsc{CH-OPT}(\mathcal{I}_{B})$ approximately, i.e. there can be at most violations in the magnitude of $\mathcal{O}(T^{-1/2})$ for both constraints. In this subsection, we present a modification to UCB-SGD that enables us to achieve no-constraint violations, while still retaining the $\mathcal{O}(T^{-1/3})$ accuracy in total conversion. Similar modification techniques have been introduced in \cite{balseiro2022best,feng2022online}. 

Our modification strategy handles ROI and budget constraint satisfactions differently. For budget constraints, we simply maintain a spend balance $B_{t}$ in each period starting from $B_{1} = 0$, and increase the balance by the expenditure in each period. When the balance nears $\rho T$, i.e. total spend comes close to $\rho T$, we simply terminate the algorithm. Regarding the ROI constraint, we develop two phases. Phase 1 is a ``safety buffer phase'' where we conservatively set per-channel budgets to accumulate a positive ``ROI balance'', i.e. in this phase (assume ending in period $T_{1}$) we hope to achieve $\sum_{t\in [T_{1}]}g_{1,t} \geq \Theta(\sqrt{T})$, where we recall $-g_{1,t}$ defined in step 4 of Algorithm \ref{alg:omducb} can be viewed as the ROI constraint violations in period $t$. For phase 2, we then naively run SGD-UCB. The motivation for this two-phase design is that we aim to have a buffer, i.e. positive ROI balance, in phase 1 that can compensate for possible constraint violations in phase 2 when we run SGD-UCB (see Theorem \ref{thm:regret}). We call our algorithm SGD-UCB-II which we present in Algorithm \ref{alg:omducb2}.

We remark that in order to implement the buffer phase 1 to attain a positive ROI balance, we rely on the following Assumption \ref{ass:channelfeas} which is a strengthened version of Assumption \ref{ass:feas} that states in each channel there is always an auction that has a value-to-cost ratio above the global target ROI $\gamma$. Then by setting a small budget  we denote as $\beta$, the channel will only procure impressions with high value-to-cost ratios (due to the structure of conversion functions in Lemma \ref{lem:Vstruct}), and thus ensuring that the ROI balance increases.

\begin{algorithm}[ht]
    \centering
    \caption{SGD-UCB-II}\label{alg:omducb2}
    \footnotesize
    \begin{algorithmic}[1]
    \Require{Set spend balance $B_{1}=0$, and $g_{1,0} = 0$, $\beta > 0$} 
    \phase{Accumulate ROI balance buffer}
    \While{$\sum_{t' \in [t-1]}g_{1,t'} \leq \sqrt{T}\log(T)$}
    \If{$B_{t} + M\rho  > \rho T$}
         \State Terminate  algorithm and output $\overline{\bm{\rho}} = \left(\frac{1}{T}\sum_{t'\in[t]} \rho_{j,t'}\right)_{j\in [M]}$.
    \EndIf
    \State  Set per-channel budget to be $\beta$ for all channels.  Observe conversion $V_{j}(\beta;\bm{z}_{t})$ for all $j \in [M]$, and calculate
    \begin{align*}
        g_{1,t} = \sum_{j \in M} \left(V_{j}(\beta;\bm{z}_{t}) -  \gamma  \beta\right)\,.
    \end{align*}
    \State Calculate $B_{t+1} = B_{t} + M\beta$.
    \State Increment $t \leftarrow t+1$
	\EndWhile
 \State Denote end period of Phase 1 as $T_{1} = t-1$.
 \phase{Run SGD-UCB}
 \State For remaining $T - T_{1}$ periods, run SGD-UCB  in Algorithm \ref{alg:omducb} with a spend balance check during each period $t$:
 \If{$B_{t} + M\rho  > \rho T$}
 \State Terminate and output 
 $\overline{\bm{\rho}} = \left(\frac{1}{T}\sum_{t'\in[t]} \rho_{j,t'}\right)_{j\in [M]}$.
 \Else 
 \State Set per-channel budgets $\{\rho_{j,t}\}_{j\in[M]}$ according to SGD-UCB for all channels. 
 \State Update spend balance $B_{t+1} = B_{t} + \sum_{j\in [M]}\rho_{j,t}$.
 \EndIf
    \end{algorithmic}
\end{algorithm}


\begin{assumption}[Strictly feasible per-channel ROI constraints]
\label{ass:channelfeas}
Fix any channel $j\in [M]$ and any realization of value-cost pairs  $\bm{z}_{j} =(\bm{v}_{j},\bm{d}_{j}) \in F_{j}$, the channel's optimization problem in Eq. \ref{eq:budperchannelsol} is strictly feasible, i.e. the set $\left\{\bm{x}_{j} \in [0,1]^{m_{j}}: \bm{v}_{j}^{\top} \bm{x}_{j} > \gamma\bm{d}_{j}^{\top}\bm{x}_{j}\right\}$ is nonempty.\footnote{Equivalently, for any realization of value-cost pairs  $\bm{z}_{j} = (\bm{v}_{j},\bm{d}_{j})$ there always exists an auction $n\in [m_{j}]$ in this channel  whose value-to-cost ratio is at least $\gamma$, i.e. $v_{j,n} >  \gamma d_{j,n}$.}
\end{assumption}

Our strategy to bound the performance of SGD-UCB-II is as followed: In the first phase, we show that we acquire sufficient ROI balance buffers to compensate for global ROI constraint violation in the second phase. On the other hand, conversion loss in the second phase is solely due to SGD-UCB (Algorithm \ref{alg:omducb}), and thus our proof to bound such loss follows from similar ideas in Theorem \ref{thm:regret} (but here we have to additionally handle  ``early stopping'' of UCB-SGD-II due to the spend balance check.

Note that in the  first ROI balance buffer phase, we are not optimizing for per-channel budgets which may lead to significant per-period conversion loss. Nevertheless, in the following lemma, we first show that the first ROI balance buffer phase does not last too long; We refer readers to Appendix \ref{pf:lem:boundphase1} for the proof.
\begin{lemma}[Bounding length of Phase 1]
\label{lem:boundphase1}
Recall $T_{1}\in [T]$ is the end period of Phase 1 in the SGD-UCB-II algorithm (see step 10). Denote the event $\mathcal{E} = \{T_{1}\geq 2\sqrt{T}\log^{3}(T)\}$, and take small  budget $\beta = \frac{1}{\log(T)}$ in the SGD-UCB-II algorithm.  Then, under Assumption \ref{ass:feas}, for large enough $T$ we have $\mathbb{P}\left(\mathcal{E}\right)\leq \frac{1}{T}$. 
\end{lemma}

Our main result in this subsection is the following Theorem \ref{thm:regret2} which bounds the conversion loss of SGD-UCB-II. The proof is detailed in Appendix \ref{pf:thm:regret2}.
\begin{theorem}
\label{thm:regret2} 
Suppose that Assumptions \ref{ass:budgetbind} and \ref{ass:channelfeas}  hold. Let $T_{2} \in [T]$ be the termination period of SGD-UCB-II (Algorithm \ref{alg:omducb2}), and recall
 $\overline{\bm{\rho}} = \left(\frac{1}{T}\sum_{t\in[T_{2}]}\rho_{j,t}\right)_{j\in [M]}$ are the final outputs of the algorithm. Further, we input $\beta =\frac{1}{
 \log(T)} $ to the algorithm that is used for Phase 1, and in Phase 2 where we run SGD-UCB, we use the same parameters as those in Theorem \ref{thm:regret}. Then,  for large enough $T$ we have 
    $ \textsc{GL-OPT} - \sum_{j \in [M]}\mathbb{E}\left[V_{j}(\overline{\rho}_{j})\right]\leq \mathcal{O}(T^{-1/3})$, and further $\sum_{j \in [M]}\mathbb{E}\left[V_{j}(\overline{\rho}_{j}) -\gamma \overline{\rho}_{j}\right]\geq 0 $ and $\sum_{j \in [M]} \mathbb{E}\left[\overline{\rho}_{j}\right]\leq \rho$. 
\end{theorem}

\subsection{Extension to strict constraint satisfaction: UCB-SGD-II}
In Theorem \ref{thm:regret}, we showed that the final output of the UCB-SGD Algorithm \ref{alg:omducb} outputs a per-channel budget profile $\overline{\bm{\rho}}_{T} = \left(\frac{1}{T}\sum_{t\in[T]} \rho_{j,t}\right)_{j\in [M]}$ that satisfies both ROI and budget constraints in $\textsc{CH-OPT}(\mathcal{I}_{B})$ approximately, i.e. there can be at most violations in the magnitude of $\mathcal{O}(T^{-1/2})$ for both constraints. In this subsection, we present a modification to UCB-SGD that enables us to achieve no-constraint violations, while still maintaining the $\mathcal{O}(T^{-1/3})$ accuracy in total conversion. Similar modification techniques have been introduced in \cite{balseiro2022best,feng2022online}. 

Our modification strategy handles ROI and budget constraint satisfactions differently. For budget constraints, we simply maintain a budget balance $B_{t}$ in each period starting from $B_{1} = \rho T$, and decrease the balance by the expenditure in each period. When the balance nears 0, i.e. total spend comes close to $\rho T$, we simply terminate the algorithm. Regarding the ROI constraint, we develop two phases. Phase 1 is a ``safety phase'' where we conservatively utilize channel levers to accumulate a positive ``ROI balance'', i.e. in this phase (assume ending in period $T'$) we hope to achieve $\sum_{t\in [T']}g_{1,t} \geq \Theta(\sqrt{T})$, where we recall $g_{1,t}$ defined in step 4 of Algorithm \ref{alg:omducb} can be viewed as the ROI constraint violations in period $t$. For phase 2, we then naively run SGD-UCB. Here, we aim to have a positive ROI balance in phase 1 that will be used to  compensate possible constraint violations in phase 2 when we run SGD-UCB. 

We remark that in order to implement the safety phase to attain a positive ROI balance, we resort to per-channel ROIs $(\gamma_{j})_{j\in[M]}$, since setting $\gamma_{j} = \gamma$ for all channels will ensure that our ROI balance increases.\footnote{ Setting $\gamma_{j} = \gamma$ for all channels implies total realized conversion as at least $\gamma$ times spend, i.e. $V_{j}(\gamma, \rho_{j,t};\bm{z}_{j,t}) \geq \gamma V_{j}(\gamma, \rho_{j,t};\bm{z}_{j,t})$ due to Eq. \eqref{eq:budperchannelsol}. Hence the ROI balance, namely $g_{1,t} = V_{j}(\gamma, \rho_{j,t};\bm{z}_{j,t})  - \gamma V_{j}(\gamma, \rho_{j,t};\bm{z}_{j,t})$ is nonnegative, and thus in phase 1 the cumulative ROI balance increases in each period.} From a high level,  SGD-UCB attempts to approximate the optimal solution for $\textsc{CH-OPT}(\mathcal{I}_{B})$ by solely using the per-channel budget lever, whereas in the above modification with the usage of per-channel ROIs hopes to approximate an optimal solution to 
$\textsc{CH-OPT}(\mathcal{I}_{G})$ 
for the general option (see Eq. \eqref{eq:option}) which is inspired by the fact that  $\textsc{CH-OPT}(\mathcal{I}_{G}) = \textsc{CH-OPT}(\mathcal{I}_{B})=\textsc{GL-OPT}$ according to Lemma \ref{cor:roiredund} and Corollary \ref{cor:roiredund}. We call this modified algorithm SGD-UCB-II, and summarize details in Algorithm \ref{alg:omducb2}. 

\begin{algorithm}[H]
    \centering
    \caption{SGD-UCB-II}\label{alg:omducb2}
    \footnotesize
    \begin{algorithmic}[1]
    \Require{Set $B_{1}=0$, and $g_{1,0} = 0$, ROI buffer $\beta > 0$} 
    \phase{Accumulate ROI buffer}
    \While{$\sum_{t' \in [t-1]}g_{1,t'} \leq \sqrt{T}\log(T)$}
    \If{$B_{t} + M\rho  > \rho T$}
         \State Terminate  algorithm and output $\overline{\bm{\rho}} = \left(\frac{1}{T}\sum_{t'\in[t]} \rho_{j,t'}\right)_{j\in [M]}$ and $\overline{\bm{\lambda}}  =  \left(\frac{1}{T}\sum_{t'\in[t]} \lambda_{j,t'}\right)_{j\in [M]}$.
    \EndIf
    \State  Set $\gamma_{j,t} = \gamma + \beta$ and $\rho_{j,t} = \beta$ for all $j \in [M]$ 
    \State Observe conversion and costs $V_{j}(\gamma+\beta, \rho;\bm{z}_{t}),D_{j}(\gamma+\beta, \beta;\bm{z}_{t}) $ for all channels $j \in [M]$. Calculate
    \begin{align*}
        g_{1,t} = \sum_{j \in M} \left(V_{j}(\gamma_{j,t}, \rho_{j,t};\bm{z}_{t}) -  \gamma \cdot D_{j}(\gamma_{j,t}, \rho_{j,t};\bm{z}_{t})\right)\,.
    \end{align*}
    \State Calculate $B_{t+1} = B_{t} + \sum_{j\in [M]}\rho_{j,t}$.
    \State Increment $t \leftarrow t+1$
	\EndWhile
 \State Denote end period of Phase 1 as $T_{1} = t-1$.
 \phase{Run SGD-UCB}
 \State For remaining $T - T_{1}$ periods, run SGD-UCB  in Algorithm \ref{alg:omducb} with a spend balance check during each period $t$:
 \If{$B_{t} + M\rho  > \rho T$}
 \State Terminate and output 
 $\overline{\bm{\rho}} = \left(\frac{1}{T}\sum_{t'\in[t]} \rho_{j,t'}\right)_{j\in [M]}$ and $\overline{\bm{\lambda}}  =  \left(\frac{1}{T}\sum_{t'\in[t]} \lambda_{j,t'}\right)_{j\in [M]}$ where  $\lambda_{j,t'} = 0$ for any $t'> T_{1}$.
 \Else 
 \State Set per-channel budgets $\{\rho_{j,t}\}_{j\in[M]}$ according to SGD-UCB, and set per-channel ROI $\gamma_{j,t} = 0$ for all channels. 
 \State Update spend balance $B_{t+1} = B_{t} + \sum_{j\in [M]}\rho_{j,t}$.
 \EndIf
    \end{algorithmic}
\end{algorithm}



\begin{lemma}[Bounding length of Phase 1]
\label{lem:boundphase1}
Recall $T_{1}$ is the end period of Phase 1 in the SGD-UCB-II algorithm (see step 10). Denote the event $\mathcal{E} = \{T_{1}\geq 2\sqrt{T}\log^{3}(T)\}$, and take ROI buffer $\beta = \frac{1}{\log(T)}$ in the SGD-UCB-II algorithm.  Then, under Assumption \ref{ass:feas}, for large enough $T$ we have $\mathbb{P}\left(\mathcal{E}\right)\leq \frac{1}{T}$. 
\end{lemma}

\textit{Proof.}

\begin{claim}
\label{clm:smallbudgetbinding}
Assume Assumption \ref{ass:channelfeas} holds. Let $\beta = \frac{1}{\log(T)}$ and assume the global ROI target $\gamma > 0$ is independent of $T$. For large enough $T$, the per-channel optimization problem in Eq. \eqref{eq:budperchannelsol} w.r.t. per-channel ROI $\gamma + \beta$ and per-channel budget $\beta$ is budget binding, i.e. $D_{j}(\gamma + \beta, \beta; \bm{z}_{j}) = \beta$
\end{claim}


Consider the hypothetical version of SGD-UCB-II where we ignore the budget balance condition $B_{t}< M\rho$ in step 2 of the algorithm, and terminate phase 1 only when 
 the condition $\sum_{t' \in [t-1]}g_{1,t'} \leq \sqrt{T}\log(T)$ in Step 1 holds. Denote this hypothetical stopping time 
as $\widetilde{T}_{1}$, defined to be 
\begin{align}
\label{eq:fakeT1}
\begin{aligned}
    \widetilde{T}_{1} = \min\Big\{t\in [T]:\sum_{t' \in [t]}  \underbrace{\sum_{j \in M} \left(V_{j}(\gamma+\beta, \rho;\bm{z}_{t'}) -  \gamma \cdot D_{j}(\gamma+\beta, \rho;\bm{z}_{t'})\right)}_{:=h_{t'}} > \sqrt{T}\log(T)\Big\}
    \end{aligned}
\end{align}
It is easy to see that with probability 1,  we have $\widetilde{T}_{1} \geq T_{1}$ where we recall  $T_{1}$ is the real stopping time of Phase 1 in SGD-UCB-II. Hence we have 
\begin{align}
\label{eq:usefakeT1}
    \mathbb{P}\left(T_{1}\geq R\right) \leq  \mathbb{P}\left(\widetilde{T}_{1}\geq R\right)
\end{align}
Now, it is not difficult to see $\{h_{t}\}_{t}$ where $h_{t}$ defined in  Eq. \eqref{eq:fakeT1} are i.i.d. random variables, since the only randomness comes from the realization of value-cost pairs $\{\bm{z}_{t}\}_{t\in[T]}$ which are i.i.d.. Further, by definition of conversion and spend functions $V_{j},D_{j}$ defined in Eq. \eqref{eq:budperchannelsol},  and \eqref{eq:budperchannel}, for any $t \in [T]$ we have 
\begin{align}
\label{eq:Elower}
\begin{aligned}
    & \sum_{j \in M} \left(V_{j}(\gamma+\beta, \rho;\bm{z}_{t}) -  (\gamma + \beta) \cdot D_{j}(\gamma+\beta, \rho;\bm{z}_{t})\right)\overset{(i)}{\geq} 0 \\
    ~\Longrightarrow~& h_{t} :=  \sum_{j \in M} \left(V_{j}(\gamma+\beta, \rho;\bm{z}_{t}) -  \gamma \cdot D_{j}(\gamma+\beta, \rho;\bm{z}_{t})\right) \\
    & \quad \geq \beta \sum_{j \in M}D_{j}(\gamma+\beta, \rho;\bm{z}_{t})\overset{(ii)}{\geq} \beta M \underline{D}\\ 
    ~\Longrightarrow~& \Bar{h}:= \mathbb{E}[h_{t}] \geq \beta M \underline{D} > 0
    \end{aligned}
\end{align}
where (i) follows from the fact that when we input per-channel target ROI $\gamma + \beta$ to each channel, the realized ROI for that channel will be at least $\gamma + \beta$; in (ii) we defined 
$ \underline{D} = \min_{j\in [M]} \min_{\bm{z}_{j}\in F_{j}} D_{j}(\gamma+\beta, \rho;\bm{z}_{j}) > 0 $ where the inequality follows from Assumption \ref{ass:feas} such that there is at least 1 channel's optimization problem is feasible with per-channel ROI $\gamma + \beta = \gamma + \frac{1}{\log(T)}$ for large enough $T$; in the final inequality we let $ \Bar{h}$ be the mean of the i.i.d. random variables $\{h_{t}\}_{t\in [T]}$.

We note that $\widetilde{T}_{1}> R$ implies that the sum of the first $R$ $h_{t}$'s do not exceed $\sqrt{T}\log(T)$ (see definition of $\widetilde{T}_{1}$ in Eq. \eqref{eq:fakeT1}), hence we have 
\begin{align}
\label{eq:boundfakeT1}
\begin{aligned}
    \mathbb{P}\left(\widetilde{T}_{1} > R \right) ~\leq~ &   
   \mathbb{P}\left(\sum_{t =1}^{\lceil R\rceil} h_{t} \leq  \sqrt{T}\log(T)\right)\\
   ~=~ &   
   \mathbb{P}\left(\sum_{t =1}^{\lceil R\rceil} (h_{t} -\Bar{h})\leq  \sqrt{T}\log(T) - \lceil R\rceil \cdot \Bar{h}\right)\\
    ~\overset{(i)}{\leq}~ &   \mathbb{P}\left(\sum_{t =1}^{\lceil R\rceil} (h_{t} -\Bar{h})\leq  \sqrt{T}\log(T) - R\beta M\underline{D}\right) \\
     ~\overset{(ii)}{=}~ &   \mathbb{P}\left(\sum_{t =1}^{\lceil R\rceil} (h_{t} -\Bar{h})\leq  -\sqrt{T}\log(T) \right) \\
      ~\overset{(iii)}{\leq}~ &   \exp\left(-\frac{T\log^{2}(T)}{2\lceil R\rceil M^{2}\Bar{V}^{2}}\right)\\
       ~\overset{(iv)}{\leq}~ &   \frac{1}{T}
\end{aligned}
\end{align}
Here (i) follows from $\Bar{h}\geq \beta M \underline{D}$ in Eq. \eqref{eq:Elower}; (ii) follows from the definition that $R = 2\sqrt{T}\log^{3}(T)$ and $\beta = \frac{1}{\log(T)}$ so  $R\beta M \underline{D} = 
2M \underline{D} \sqrt{T}\log^{2}(T) 
\geq 2\sqrt{T}\log(T) $ for large enough $T$ such that $\log(T) > \frac{1}{M\underline{D}}$; (iii) follows from Azuma Hoefding's inequality given that $h_{t} \in [0, M\Bar{V}]$ for any $t\in [T]$; and finally (iv) follows from $T\geq \lceil R\rceil$, and $\log(T) >  M^{2}\Bar{V}^{2}$ for large enough $T$. 
Hence, combining Eq. \eqref{eq:boundfakeT1} and \eqref{eq:usefakeT1} yields the desired statement of the lemma. 
\halmos

\begin{assumption}[Strictly feasible per-channel ROI constraints]
\label{ass:channelfeas}
Fix any channel $j\in [M]$ and any realization of value-cost pairs  $\bm{z}_{j} =(\bm{v}_{j},\bm{d}_{j}) \in F_{j}$, the channel's optimization problem in Eq. \ref{eq:budperchannelsol} is strictly feasible, i.e. the set $\left\{\bm{x}_{j} \in [0,1]^{m_{j}}: \bm{v}_{j}^{\top} \bm{x}_{j} > \gamma\bm{d}_{j}^{\top}\bm{x}_{j}\right\}$ is nonempty.\footnote{Equivalently, for any realization of value-cost pairs  $\bm{z}_{j} = (\bm{v}_{j},\bm{d}_{j})$ there always exists an auction $n\in [m_{j}]$ in this channel  whose value-to-cost ratio is at least $\gamma$, i.e. $v_{j,n} >  \gamma d_{j,n}$.}
\end{assumption}

Our main result in this subsection is the following Theorem \ref{thm:regret2}:
\begin{theorem}
\label{thm:regret2}
    Let $T_{2} \in [T]$ be the termination period of SGD-UCB-II, and recall
    $\overline{\bm{\rho}} = \left(\frac{1}{T}\sum_{t\in[T_{2}]}\rho_{j,t}\right)_{j\in [M]}$ and  $\overline{\bm{\lambda}} = \left(\frac{1}{T}\sum_{t\in[T_{2}]}\rho_{j,t}\right)_{j \in [M]}$ are the final outputs of  SGD-UCB-II in Algorithm \ref{alg:omducb2}. Then under Assumptions \ref{ass:budgetbind} and \ref{ass:channelfeas} and the same SGD-UCB parameter choices in Theorem \ref{thm:regret}, for large enough $T$ we have 
    $ \textsc{GL-OPT} - \sum_{j \in [M]}\mathbb{E}\left[V_{j}(\overline{\gamma}_{j},\overline{\rho}_{j})\right]\leq \mathcal{O}(T^{-1/3})$, and further $\sum_{j \in [M]}\mathbb{E}\left[V_{j}(\overline{\gamma}_{j},\overline{\rho}_{j}) -\gamma D_{j}(\overline{\gamma}_{j},\overline{\rho}_{j})\right]\geq 0 $ and $ \sum_{j \in [M]} \mathbb{E}\left[D_{j}(\overline{\gamma}_{j},\overline{\rho}_{j})\right]\leq \rho$. 
\end{theorem}

\textit{Proof.}


\textbf{Part I. Bounding global budget constraint violation.}

\begin{align}
    \sum_{j\in [M]}D_{j}(\Bar{\gamma}_{j},\Bar{\rho}_{j}) \overset{(i)}{\leq}\sum_{j}D_{j}(0,\Bar{\rho}_{j}) \leq \Bar{\rho}_{j} = \frac{1}{T}\sum_{t\in [T_{2}]}\sum_{j\in [M]} \rho_{j,t} \overset{(ii)}{<}\rho \,.
\end{align}
Here (i) is due to the fact that for any value-cost realization $\bm{z}_{j}$, per-channel budget $\rho_{j}\in [0,\rho]$ and per-channel ROI $\gamma_{j} \geq 0$, we have
$D_{j}(\gamma_{j},\rho_{j};\bm{z}_{j}) \leq D_{j}(0,\rho_{j};\bm{z}_{j}) $, since we are effectively ignoring the per-channel ROI constraint in $ D_{j}(0,\rho_{j};\bm{z}_{j})$, so this also holds in expectation w.r.t. randomness in $\bm{z}_{j}$;  (ii) is due to the design of SGD-UCB so that the per-channel budget decisions never sum up to exceed $\rho T - M\rho $.

\textbf{Part II. Bounding global ROI constraint violation.}

Recall the event $\mathcal{E} = \{T_{1}\geq 2\sqrt{T}\log^{3}(T)\}$ defined in Lemma \ref{lem:boundphase1} where $T_{1}$ is the end period of Phase 1 in the SGD-UCB-II algorithm (see step 10). We consider two scenarios, namely when event $\mathcal{E}$ holds and doesn't hold. Recall $g_{1,t} = \sum_{j\in[M]}\left(V_{j}(\gamma_{j,t},\rho_{j,t};\bm{z}_{t}) - \gamma D_{j}(\gamma_{j,t},\rho_{j,t};\bm{z}_{t})\right)$

When event $\mathcal{E} $ holds, then 
\begin{align}
\begin{aligned}
\label{eq:badeventroi}
   
    & \mathbb{E}\left[\sum_{t\in [T_{2}]}g_{1,t} ~\Big |~ \mathcal{E}\right] 
 \geq - T\gamma M\rho



 \end{aligned}
\end{align}

When event  $\mathcal{E} $ does not hold, Phase 1 terminates within  the first $2\sqrt{T}\log^{3}(T)$ periods, i.e.
$T_{1} < 2\sqrt{T}\log^{3}(T)$, and the total spend balance in Phase 1 is at most 
\begin{align*}
    \sum_{t\in [T_{1}]} \sum_{j\in [M]}\rho = M\rho T_{1} <  2M\rho\sqrt{T}\log^{3}(T) < \rho T - \rho M
\end{align*}
where the final inequality holds for large enough $T$. This implies that Phase 1 terminates because the ROI buffer condition in step 1 is met, i.e. 
\begin{align}
\label{eq:goodeventroi1}
    \sum_{t\in [T_{1}]} g_{1,t}> \sqrt{T}\log(T)
\end{align}
Now in periods $t = T_{1}+1 \dots T_{2}$, following the proof of Lemma \ref{lem:approxconst}, we have
\begin{align}
\label{eq:goodeventroi2}
    \sum_{t = T_{1}+1}^{T_{2}}g_{1,t} \geq \frac{1}{\eta} \left(\lambda_{T_{1}} - \lambda_{T_{2}}\right) \overset{(i)}{\geq} -\frac{C_{F}}{\eta} = C_{F} \sqrt{T}
\end{align}
we (i)  follows from $ 0\leq \lambda_{t} \leq C_{F}$ for any $t\in [T]$ for some absolute constant $C_{F} >0$ that only depends on the support of value-cost pairs $F = F_{1}\times \ldots \times F_{M}$. Hence, combining Eq. \eqref{eq:badeventroi}, \eqref{eq:goodeventroi1}, \eqref{eq:goodeventroi2}, we have
\begin{align}
\begin{aligned}
      & \sum_{j\in [M]}\left(V_{j}(\Bar{\gamma}_{j},\Bar{\rho}_{j};\bm{z}_{t})  - \gamma D_{j}(\Bar{\gamma}_{j},\Bar{\rho}_{j};\bm{z}_{t}) \right) \\
      ~\geq~ & \sum_{j\in [M]}\left(V_{j}(\Bar{\gamma}_{j},\Bar{\rho}_{j};\bm{z}_{t})  - \gamma D_{j}(0,\Bar{\rho}_{j};\bm{z}_{t}) \right) \\
      ~\overset{(i)}{=}~& \sum_{j\in [M]}\left(V_{j}(\Bar{\gamma}_{j},\Bar{\rho}_{j};\bm{z}_{t})  - \gamma \Bar{\rho}_{j}) \right) \\
       ~\overset{(ii)}{\geq}~& \frac{1}{T}\sum_{t\in [T_{2}]}\sum_{j\in [M]}\left(V_{j}(\gamma_{j,t},\rho_{j,t};\bm{z}_{t})  - \gamma \rho_{j,t}) \right) \\
        ~=~& \frac{1}{T}\sum_{j\in [M]}\left(\sum_{t\in [T_{1}]}\left(V_{j}(\gamma +\beta ,\beta;\bm{z}_{t})  - \gamma \beta) \right) +  \sum_{t=T_{1}+1}^{T_{2}}\left(V_{j}(0,\rho_{j,t};\bm{z}_{t})  - \gamma \rho_{j,t}) \right)\right)\\
         ~\overset{(iii)}{=}~& \frac{1}{T}\sum_{j\in [M]}\left(\sum_{t\in [T_{1}]}\left(V_{j}(\gamma +\beta ,\beta;\bm{z}_{t})  - \gamma D_{j}(\gamma +\beta ,\beta;\bm{z}_{t})) \right) +  \sum_{t=T_{1}+1}^{T_{2}}\left(V_{j}(0,\rho_{j,t};\bm{z}_{t})  - \gamma \rho_{j,t}) \right)\right)
\end{aligned}
\end{align}

\textbf{Part III. Bounding estimation error.}
Similar to the proof of Theorem \ref{thm:regret}, we only need to bound
$T_{2}\cdot  \textsc{GL-OPT} - \mathbb{E}\left[\sum_{t\in [T_{2}]}\sum_{j\in[M]}V_{j}(\gamma_{j,t},\rho_{j,t})\right] $


We first show that $T - T_{2}\leq $


\begin{align}
    \sum_{t\in [T_{2}]}\sum_{j\in[M]}V_{j}(\gamma_{j,t},\rho_{j,t})
\end{align}
\halmos


We conclude with the insight that if satisfying ROI and budget constraints approximately as in Theorem \ref{thm:regret} for the UCB-SGD algorithm is acceptable, then considering the per-channel budget lever along would suffice. Yet, when advertisers hope to respect constraints strictly, advertisers may need resort to the per-channel ROI lever unde the UCB-SGD-II framework.
\subsection{The SGD-UCB algorithm to optimize per-channel budgets}
\label{subsec:algo}
Here, we describe our algorithm to solve for optimal per-channel budgets w.r.t.  $\textsc{CH-OPT}(\mathcal{I}_{B})$. Similar to most algorithms for constrained optimization, we take a dual stochastic gradient descent (SGD) approach; see a comprehensive survey on dual descent methods in \cite{bertsekas2014constrained}. First, we consider the Lagrangian functions w.r.t. $\textsc{CH-OPT}(\mathcal{I}_{B})$ where we let  $\bm{c} = (\lambda, \mu) \in \mathbb{R}_{+}^{2}$ be the dual variables corresponding to the ROI and budget constraints, respectively:
\begin{align}
    \label{eq:Lagrangian}
    \mathcal{L}_{j}(\rho_{j}, \bm{c}; \bm{z}_{j}) =  (1+\lambda)V_{j}(\rho_{j};\bm{z}_{j}) - (\lambda \gamma  + \mu) \rho_{j} ~\text{ and }~\mathcal{L}_{j}(\rho_{j}, \bm{c}) = \mathbb{E}\left[  \mathcal{L}_{j}(\rho_{j}, \bm{c}; \bm{z}_{j})\right]\,.
\end{align}
With this lagrangification, SGD decouples the aggregate constraints over channels, and further operates on both primal and dual domains to update per-channel budgets for each channel independently, as well dual variables, respectively:
In each period $t\in [T]$ given dual  variables $\bm{c}_{t} = (\lambda_{t},\gamma_{t})$, SGD decides on a primal decision, i.e. per-channel budget $(\rho_{j,t})_{j\in[M]}$ by solving the following optimization problem 
\begin{align}
\label{eq:SGDprimal}
    \rho_{j,t} ~=~ \arg\max_{\rho_{j}\in [0,\rho]}\mathcal{L}_{j}(\rho_{j}, \bm{c}_{t};\bm{z}_{t})\,.
\end{align}
 Then, having observed the realized values $\left(V_{j}(\rho_{j,t};\bm{z}_{t})\right)_{j\in[M]}$ (note that spend is $(\rho_{j,t})_{j\in [M]}$ in light of Proposition \ref{prop:budgetbind}),  we calculate the current period violation in budget and ROI constraints, namely  $g_{1,t}:= \sum_{j \in M} \left(V_{j}(\rho_{j,t};\bm{z}_{t}) -  \gamma \rho_{j,t}\right)$ and $g_{2,t} = \rho - \sum_{j\in [M]} \rho_{j,t}$. Next, we update dual variables via $ 
\lambda_{t+1} = \left(\lambda_{t} - \eta g_{1,t}\right)_{+} \text{ and } \mu_{t+1}  = \left(\mu_{t} - \eta g_{2,t}\right)_{+}$, where $\eta$ is some pre-specified step size.\footnote{Here, the dual updates follow the vanilla gradient descent approach, and one can also employ more general mirror descent updates; see e.g. \cite{balseiro2022best}.}

{\color{blue} However, the above SGD approach faces a fatal drawback, namely we cannot realistically find the primal decisions by solving Eq. \eqref{eq:SGDprimal} since the function $\mathcal{L}_{j}(\cdot, \bm{c}_{t};\bm{z}_{t})$ is unknown due to the  bandit feedback  structure. Therefore, we provide a modification to SGD to handle this issue. Before we present our approach, we briefly note that although bandit feedback prevents the naive application of SGD for our problem of interest, this may not be the case in other online advertising scenarios that involve relevant learning tasks, underlining the challenges of our problem; see following Remark \ref{rmk:banditfeedback} for details.
\begin{remark}
\label{rmk:banditfeedback}
Our problem of interest under bandit feedback is more difficult than similar problems in related works that study online bidding strategies under budget and ROI constraints; see e.g. \cite{balseiro2017budget,balseiro2022best,feng2022online}. To illustrate, consider for instance  \cite{balseiro2017budget} in which a budget constrained advertiser's primal decision at period $t$ is to submit a bid value  $b_{t}$ after observing her value $v_{t}$. The advertiser competes with some unknown  highest competing bid $d_{t}$ in the market, and after submitting bid $b_{t}$, does not observe $d_{t}$ if she does not win the competition, which involves a semi-bandit feedback structure. Nevertheless, the corresponding Lagrangian under SGD takes the special form $\mathcal{L}_{j}(b, \mu_{t};\bm{z}_{t}) =\left(v_{t} - (1+\mu_{t})d_{t}\right)\mathbb{I}\{b_{t} \geq d_{t}\}$ where $\mu_{t}$ is the dual variable w.r.t. the budget constraint.  This simply allows an advertiser to optimize for her primal decision by bidding $\arg\max_{b\geq 0} \mathcal{L}_{j}(b, \bm{c}_{t};\bm{z}_{t}) = \frac{v_{t}}{1+\mu_{t}}$. So even though \cite{balseiro2017budget,balseiro2022best,feng2022online} study dual SGD under bandit feedback, the special structures of their problem instances permits SGD to effectively optimize for primal decisions in each period, as opposed to Eq. \eqref{eq:SGDprimal} in our setting which can not be  solved.
\end{remark}}
To resolve challenges that arise with bandit feedback in our model, we take a natural approach to augment SGD with the \textit{upper-confidence bound (UCB)} algorithm, which is arguably the first resort to solve learning problems under bandit feedback such as the famous multi-arm bandit problem;\yuan{Are we the first to combine SGD and UCB? If yes, could you highlight this? if not, could you add some citations for comparison?} \negin{agree with this comment. also, when  you say this is ``the first resort" implies that there are prior works that use this technique. If this is not the case, please paraphrase that sentence. } see an introduction to multi-arm bandits in \cite{slivkins2019introduction}. In particular, we first discretize our per-channel buudget decision set $[0,\rho]$ into granular ``arms'' that are separated by some distance $\delta > 0$, so that the discretized per-channel budget decisions become
\begin{align}
\label{eq:budgetdisc}
    \mathcal{A}(\delta) = \{a_{k}\}_{k \in [K]} \text{ where } a_{k} = (k-1)\delta \text{ and }K := \lceil{\rho}/{\delta}\rceil + 1\,.
\end{align}
In the following we will use the terms ``per-channel budget'' and ``arm'' interchangeably. {\color{red} I suggest this we discuss these (rewards and contexts) when we present the proof in the next section. Here, we can mainly focus on describing the novelty of the design.} Then, we perceive dual variables $\{\bm{c}_{t}\}_{t\in[T]}$ to be \textit{adversarial contexts}, and the Lagrangian $\mathcal{L}_{j}(a_{k},\bm{c}_{t};\bm{z}_{t})$ to be the reward function that maps any arm-context pair $(a_{k},\bm{c}_{t})$ to a (stochastic) payoff. Then following UCB, in each period $t$ we maintain some estimate $\left(\Bar{V}_{j}(a_{k})\right)_{j\in[M]}$ of the conversions $\left(V_{j}(a_{k})\right)_{j\in [M]}$ as well as an upper confidence bound $\texttt{UCB}_{j,t}(a_{k})$ for each arm $a_{k}$ using historical payoffs from periods in which arm $a_{k}$ is pulled. Finally, we update primal decisions for each channel $j\in[M]$ using the ``best arm'' $\rho_{j,t} = \arg\max_{a_{k}\in \mathcal{A}(\delta)}~(1+\lambda_{t})\left(\Bar{V}_{j,t}(a_{k}) +  \texttt{UCB}_{j,t}(a_{k})\right) - \left(\lambda_{t} \gamma + \mu_{t} \right)a_{k}$. We summarize our algorithm, called SGD-UCB, in the following Algorithm \ref{alg:omducb}.

\begin{algorithm}[H]
    \centering
    \caption{SGD-UCB$(\gamma,\rho)$}\label{alg:omducb}
    \footnotesize
    \begin{algorithmic}[1]
    \Require Global target ROI $\gamma$ and global budget parameter $\rho$. Budget discretization $\delta > 0$ and set of arms $\mathcal{A}(\delta)$ defined in Eq.\eqref{eq:budgetdisc} with $|\mathcal{A}(\delta)| =K$. Step size $\eta > 0$ and strictly convex regularizer $h:\mathbb{R}\to\mathbb{R}$. Initialize $ N_{j,1}(a_{k}) = \Bar{V}_{j,1}(a_{k})  = 0$ for all $j \in [M]$ and $k\in [K]$.
    Initialize dual variables $\lambda_{1} = \mu_{1}= 0$.
   
   
   
    \\
    \textbf{Output:} Per channel budget.   
	\For{$t = 1 \dots T$ }
	 \State \textbf{Update (primal) per-channel budget.} For each channel $j\in [M]$ set (primal) per-channel budget:
  \begin{itemize}
      \item If $t \leq K$, set $\rho_{j,t} = a_{t}$.
      \item If $t > K$, set
      \begin{align}
        \begin{aligned}
          & \rho_{j,t} = \arg\max_{a_{k}\in \mathcal{A}(\delta)}~(1+\lambda_{t})\left(\Bar{V}_{j,t}(a_{k}) +  \texttt{UCB}_{j,t}(a_{k})\right) - \left(\lambda_{t} \gamma + \mu_{t} \right)a_{k}, ~\text{ where } \texttt{UCB}_{j,t}(a_{k}) = \sqrt{\frac{2\log(T)}{N_{j,t}(a_{k})}} 
           \end{aligned}
          \end{align}
  \end{itemize}
    \State  Observe realized values $\left\{V_{j}(\rho_{j,t};\bm{z}_{t}) \right\}_{j \in [M]}$, and update for each arm $k \in [K]$:{\color{red}NG: $\rho_{j,t} = a_{k}$ instead of $\rho_{j,\tau} = a_{k}$?}
    \begin{align}
    \begin{aligned}
        & N_{j,t+1}(a_{k}) =  N_{j,t}(a_{k}) + \mathbb{I}\{\rho_{j,\tau} = a_{k}\}\\
        & \Bar{V}_{j,t+1}(a_{k}) = \frac{1}{N_{j,t+1}(a_{k})} \left(N_{j,t}(a_{k}) \Bar{V}_{j,t}(a_{k}) + V_{j}(\rho_{j,t};\bm{z}_{t}) \mathbb{I}\{\rho_{j,\tau} = a_{k}\}\right) \quad, \text{ for } j = 1 \dots M
        \end{aligned}
    \end{align}
    \State \textbf{Update dual variables.} Update dual variables with $g_{1,t}:= \sum_{j \in M} \left(V_{j}(\rho_{j,t};\bm{z}_{t}) -  \gamma \rho_{j,t}\right)$ and $g_{2,t} = \rho - \sum_{j\in [M]} \rho_{j,t}$:
   
   
       
       
       
       
       
       
        \begin{align}
        \label{eq:dualupdate}
        \begin{aligned}
     \lambda_{t+1} = \left(\lambda_{t} - \eta g_{1,t}\right)_{+} \text{ and } \mu_{t+1}  = \left(\mu_{t} - \eta g_{2,t}\right)_{+}
        \end{aligned}
        \end{align}
	\EndFor
 \State Output $\overline{\bm{\rho}}_{T} = \left(\frac{1}{T}\sum_{t\in[T]} \rho_{j,t}\right)_{j\in [M]}$.
    \end{algorithmic}
\end{algorithm}


\subsection{Analyzing the SGD-UCB algorithm}
\label{subsec:algo}
In this subsection, we analyze the performance of SGD-UCB in Algorithm \ref{alg:omducb}. At first sight, our algorithm faces a setup that involves discretizing a continuum of arms (i.e. the continuous per-channel budget decision set $[0,\rho]$), as well as contextual bandit rewards {\color{red} as I said before, let's define the contextual bandit setting here and please elaborate more on this using equation (10) so that readers can better understand this. }, both of which have been notoriously difficult to handle without special structures in the reward function (in our case the Lagrangian $\mathcal{L}_{j}(\rho_{j},\bm{c};\bm{z})$ defined in Eq. \ref{eq:Lagrangian}; see e.g. discussions in \cite{agrawal1995continuum} and \cite{agarwal2014taming} for continuum-arm bandits and contextual bandits, respectively. Fortunately, the specific form of conversion functions $V(\rho_{j};\bm{z})$ {\color{red} NG: not sure if we have done it already. But, please make sure that in the model section we define the term ``conversion functions."} defined in Eq. \eqref{eq:budperchannelsol} imposes a salient structure on the Lagrangian, or equivalently the reward function for pulling an arm. Specifically, the following lemma shows that the Lagrangian is continuous, piecewise linear, concave, and unimodal\footnote{We say a real-valued function $f:\mathbb{R}\to \mathbb{R}$ is unimodal if there exists some $y^{*}$ such that $f(y)$ strictly increases when $y \leq y^{*}$ and  strictly decreases when $y \geq y^{*}$. {\color{red} NG: why strictly?}}; we present the proof in Appendix \ref{pf:lem:Vstruct}
\begin{lemma}[Structural properties of conversion  and Lagrangian functions]
\label{lem:Vstruct}
\begin{itemize}
\item
For any channel $j\in [M]$ and per-channel budget $\rho_{j}$ the conversion function $V_{j}(\rho_{j})$ is continuous, piece-wise linear, strictly increasing, and concave. In particular,  $V_{j}(\rho_{j})$ takes the form
\begin{align}
\label{eq:Vstruct}
        V_{j}(\rho_{j}) = \sum_{n\in [S_{j}]} \left(s_{j,n} \rho_{j} + b_{j,n}\right)\mathbb{I}\{r_{j,n-1}\leq \rho_{j} \leq r_{j,n}\}\,,
\end{align}
where the parameters $S_{j} \in \mathbb{N}$ and $\{(s_{j,n},b_{j,n},r_{j,n})\}_{n\in [S_{j}]}$ only depend on the support $F_{j}$ and distribution $\bm{p}_{j}$ from which value-to-cost pairs are sampled.  These parameters satisfy
$s_{j,1} > s_{j,2} > \dots > s_{j,S_{j}} > 0$ and $0=r_{j,0} < r_{j,1} < r_{j,2} < \dots < r_{j,S_{j}} =\rho$, as well as $b_{j,n} > 0$ s.t. $s_{j,n} r_{j,n} + b_{j,n} = s_{j,n+1} r_{j,n} + b_{j,n+1}$ for all $n \in [S_{j}-1]$, implying $V_{j}(\rho_{j}) $ is continous  in $\rho_j$.
\item For any dual variables $\bm{c} = (\lambda,\mu) \in \mathbb{R}_{+}^{2}$, the Lagrangian function $ \mathcal{L}_{j}(\rho_{j},\bm{c})$ defined in Eq. \eqref{eq:Lagrangian} is continuous, piece-wise linear, concave, and unimodal in $\rho_{j}$. In particular,
\begin{align}
\label{eq:lagrrep}
        \mathcal{L}_{j}(\rho_{j}, \bm{c}) =  \sum_{n\in [S_{j}]} \left( \sigma_{j,n}(\bm{c}) \rho_{j} + (1+\lambda)b_{j,n}\right)\mathbb{I}\{r_{j,n-1}\leq \rho_{j} \leq r_{j,n}\}\,,
\end{align}
where the slope $\sigma_{j,n}(\bm{c}) = (1+\lambda) s_{j,n} -(\mu + \gamma \lambda)$. 
Then, this also implies $\arg\max_{\rho_{j}\geq 0} \mathcal{L}_{j}(\rho_{j}, \bm{c}) = \max\{r_{j,n}:n=0,1\dots, S_{j}~,\sigma_{j,n}(\bm{c}) \geq 0\}$.
\end{itemize}
\end{lemma}
In fact,  for any realized value-cost pairs $\bm{z}$, the ``realization 
versions'' of the conversion functions and Lagrangians {\color{red}NG: Lagrangians or Lagrangians? We have both. Let's be consistent. Also, let's call it Lagrangian function as it is a function that maps the dual variables to some (virtual) reward. Using this terminology, you can better define the contextual bandit setting.}, namely $V_{j}(\rho_{j};\bm{z}) $ and $\mathcal{L}_{j}(\rho_{j}, \bm{c};\bm{z})$,  also satisfy the same properties as those of $V_{j}(\rho_{j})$ and $\mathcal{L}_{j}(\rho_{j}, \bm{c})$, respectively. See  Lemma \ref{lem:Vstruct}. We provide a visual illustration for these structural properties in Figure \ref{fig:lagr}. 

In light of concavity for the conversion functions $\left(V_{j}(\rho_{j})\right)_{j \in [M]}$, the 
advertiser's optimization problem $\textsc{CH-OPT}(\mathcal{I}_{B})$ in Eq. \eqref{eq:chopt} effectively becomes a convex problem (as we maximize a concave function in $\textsc{CH-OPT}(\mathcal{I}_{B})$, and it is apparent {\color{red} NG: it is not very apprent to me. Do we show this formally somewhere?} that the feasible region for $\textsc{CH-OPT}(\mathcal{I}_{B})$ is convex).  Here, we make an important remark that distinguishes our work with  related literature on convex optimization.

{\color{red}NG: I suggest we present this remark at the end of this section as it is a distraction and is hurting the flow. the point of the remark would be  better received after presenting our regret bound. Another option is to put a short version of this in the intro or related work section and then  quickly revisit that discussion here. }
\begin{remark}
\label{rmk:onepoint}
Given the advertiser's optimization problem  $\textsc{CH-OPT}(\mathcal{I}_{B})$ is convex, it may be tempting for one to directly employ off-the-shelf convex optimization algorithms. However, our problem instance involves stochastic bandit feedback, and more importantly, \textit{uncertain constraints}, meaning that during each period a primal decision is made, we cannot analytically determine whether the primal decision satisfies the constraints of the problem. For example, in $\textsc{CH-OPT}(\mathcal{I}_{B})$, for some primal decision $(\rho_{j})_{j\in[M]}$, we cannot determine whether the ROI and budget constraints  $\sum_{j \in M} \mathbb{E}\left[V_{j}(\gamma_{j},\rho_{j};\bm{z}_{j})\right] \geq \gamma \sum_{j \in M} \mathbb{E}\left[D_{j}(\gamma_{j},\rho_{j};\bm{z}_{j})\right]$ and $\sum_{j\in [M]}\mathbb{E}\left[D_{j}(\gamma_{j},\rho_{j};\bm{z}_{j})\right]\leq \rho$ hold because the distributions $(\bm{p}_{j})_{j\in[M]}$ from which $\bm{z}$ is sampled are unknown. In other words, we cannot evaluate the expectations. To the best of our knowledge, there are only two recent works that handle a similar stochastic bandit feedback, and uncertain constraint setting, namely \cite{usmanova2019safe} and \cite{nguyen2022stochastic}. Nevertheless, our setting is more challenging because these works consider a ``two-point estimation'' regime where one can make function evaluations to the objective and constraints twice each period, whereas our setting lies in the ``one-point estimation'' such that we can only make function calls once per period. Finally, we also point out that optimal oracle optimization complexities for unknown constrained (non-smooth) convex optimization with one-point bandit feedback, remains an open problem. We refer readers to Table 4.1 in \cite{larson2019derivative} for a survey on best known bounds under different one-point bandit feedback settings.
\end{remark}


\begin{figure}[h]
\includegraphics[width=\textwidth]{lagr.png}
\caption{Illustration of Lagrangian functions defined in Eq. \eqref{eq:Lagrangian} with $M_{j} = 2$ auctions in channel $j$, and support $F_{j}$ that contains 3 elements,  $\bm{z}_{(1)} = (\bm{v}_{(1)},\bm{d}_{(1)}) = ((8,2), (2,3))$, $\bm{z}_{(2)} = ((3,4), (1,4))$, $\bm{z}_{(3)} = ((8,1), (4,2))$, and context $\bm{c} = (\lambda,\mu) = (4,2)$. In the context of Lemma \ref{lem:Vstruct}, $S_{j} = 5$, where the ``turning points'' $r_{j,0} \dots r_{j,S_{j}}$ are indicated on the x-axis, and the optimal budget w.r.t. $\bm{c}$ is $\arg\max_{\rho_{j}\in [0,\rho]}\mathcal{L}_{j}(\rho_{j};\bm{c}) = r_{j,2}$. The adjacent slopes in Eq. \eqref{eq:adj} are $\sigma_{j}^{-}(\bm{c}) = \sigma_{j,2}(\bm{c})$, and  $\sigma_{j}^{+}(\bm{c}) = \sigma_{j,3}(\bm{c})$, respectively.}
\label{fig:lagr}
\end{figure}










With the salient structural properties of the Lagrangian functions illustrated in Lemma \ref{lem:Vstruct}, we now present results to analyze the performance of UCB-SGD Algorithm \ref{alg:omducb}. Our strategy is to show $T\cdot  \textsc{GL-OPT} - \mathbb{E}\left[\sum_{t\in [T]}\sum_{j\in[M]}V_{j}(\rho_{j,t})\right] = \mathcal{O}(T^{\zeta})$ for some $\zeta \in (0,1)$, because using concavity of the $V_{j}(\cdot)$ functions directly yields an optimization accuracy $\textsc{GL-OPT} - \mathbb{E}\left[\sum_{j\in[M]}V_{j}\left(\frac{\sum_{t\in [T]\rho_{j,t}}}{T}\right)\right]\leq \mathcal{O}(T^{-(1-\zeta})$. Now, the following Proposition \ref{prop:ubbenchlag} allows us to decompose $T\cdot  \textsc{GL-OPT} - \mathbb{E}\left[\sum_{t\in [T]}\sum_{j\in[M]}V_{j}(\rho_{j,t})\right]$ into two main parts, namely the error induced by the UCB in our algorithm, and the error due to SGD (or what is typically viewed as the deviations from  complementary slackness):

\begin{proposition}
\label{prop:ubbenchlag}
For any channel $ j\in [M]$ define $\rho_{j}^{*}(t) = \arg\max_{\rho_{j\in[M]}}\mathcal{L}_{j}(\rho_{j};\bm{c})$ to be the optimal per-channel budget w.r.t. dual variables $\bm{c}_{t}=(\lambda_{t},\mu_{t})_{t\in [T]}$ during period $t\in[T]$. Then we have 
 \begin{align}
 \begin{aligned}
 \label{eq:decompregret}
     & T \cdot\textsc{GL-OPT} - \mathbb{E}\left[\sum_{t\in [T]}\sum_{j\in[M]}V_{j}(\rho_{j,t})\right] \\ ~\leq~&  M\Bar{V}K +
     \sum_{j\in [M]}\underbrace{\sum_{t> K} 
     \mathbb{E}\left[\mathcal{L}_{j}(\rho_{j}^{*}(t), \lambda_{t}, \mu_{t})  - \mathcal{L}_{j}(\rho_{j,t}, \lambda_{t}, \mu_{t})\right]}_{\text{UCB error}}+ \underbrace{\sum_{t >K}\mathbb{E}\left[\lambda_{t}g_{1,t} + \mu_{t}g_{2,t}\right]}_{\text{SGD complementary slackness deviations}}\,,
      \end{aligned}
 \end{align}
\end{proposition}
 where we recall the definitions of $g_{1,t}$ and $g_{2,t}$ in step 4 of Algorithm \ref{alg:omducb}, and the fact that the conversion $ V_{j}(\rho_{j};\bm{z}_{j})$ is bounded above by absolute constant $\Bar{V} \in (0,\infty)$ almost surely for any channel $j\in[M]$, $(\gamma_{j},\rho_{j})$ and realization $\bm{z}_{j}$.
 
 Before we move on to bound the UCB error, we first need to show that the UCB rewards are bounded under all arms and contexts, or in other words, the Lagrangian function is bounded for for all arms and dual variables. Boundedness on rewards is typically assumed to hold in general multi-arm bandit setups, but in our case, the rewards, i.e. the Lagrangian depends scales linearly with dual variables, which may seemingly violate boundedness conditions that can make UCB fail.  {\color{red} NG: recall the definition of the Lagrangian function by referring to its equation.} Fortunately, we show that 
  the Lagrangian functions, as well as dual variables, are indeed bounded by some absolute constants in the following Lemma \ref{lem:dualvarub} under a mild feasibility Assumption \ref{ass:feas} stated below:
\begin{assumption}[Strictly feasible global ROI constraints]
\label{ass:feas}
For any realization of value-cost pairs  $\bm{z} =(\bm{v}_{j},\bm{d}_{j})_{j\in[M]} \in F_{1}\times \dots F_{M}$, the realized version of the ROI constraint in \textsc{GL-OPT} defined in Eq. \ref{eq:glopt} is strictly feasible, i.e. the set $\left\{\bm{x}= (\bm{x}_{j})_{j\in [M]}: \bm{x}_{j} \in [0,1]^{m_{j}} \text{ for }\forall j\in [M],~\sum_{j\in[M]}\bm{v}_{j}^{\top} \bm{x}_{j} > \gamma\sum_{j\in[M]}\bm{d}_{j}^{\top}\bm{x}_{j}\right\}$ is nonempty.\footnote{Equivalently, for any realization of value-cost pairs  $\bm{z} = (\bm{v}_{j},\bm{d}_{j})_{j\in[M]}$ there always exists a channel $j\in [M]$ and an auction  $n\in [m_{j}]$ in this channel  whose value-to-cost ratio is at least $\gamma$, i.e. $v_{j,n} >  \gamma d_{j,n}$.}
\end{assumption}
\begin{lemma}[Bounding dual variables and Lagrangian functions]
\label{lem:dualvarub}
Let $(\lambda_{t},\mu_{t})_{t\in [T]}$ be the variables generated from Algorithm \ref{alg:omducb}. Under Assumption \ref{ass:feas}, and assuming the step size $\eta > 0$ satisfies $\eta < \frac{1}{M \cdot \max\left\{\bar{V}, \rho, \bar{V}^{2}\right\}}$, there exists some absolute constant $C_{F} > 0$ that depends only on the support of value-cost pairs  $F = F_{1} \times \dots \times F_{M}$ as well as aggregate target ROI and budget $(\gamma,\rho)$ such that $\lambda_{t},\mu_{t} \leq C_{F}$ for all $t \in [T]$. Moreover, for any for any $t\in [T]$, $j\in [M]$ and $\rho_{j} \in [0,\rho]$ we have 
\begin{align}
   -\left(1+\gamma \right)\rhoC_{F}
   ~\leq~ \mathcal{L}_{j}(\rho_{j},\lambda_{t},\mu_{t})~\leq~(1+C_{F})\Bar{V}\,.
\end{align}
\end{lemma}
See proof in Appendix \ref{pf:lem:dualvarub}.

We now move on to bound the UCB error, which is the main technical challenge in this section. As illustrated before,
 the salient structures of the Lagrangian function as illustrated in Lemma \ref{lem:Vstruct} allows us to handle the contextual, and discretized continuum-arm aspects of the problem. However, there is an additional caveat that remains unhandled: namely the ``flatness'' of the reward function that depends on adversarial contexts, and makes arm exploration difficult. To illustrate (see e.g. Figure \ref{fig:lagr}), we define the slopes that are adjacent to the optimal budget $\arg\max_{\rho_{j}\in[0,\rho]} \mathcal{L}_{j}(\rho_{j},\bm{c})$ for any $\bm{c} = (\lambda,\mu)$ as followed: assuming the optimal budget $\arg\max_{\rho_{j}\in[0,\rho]} \mathcal{L}_{j}(\rho_{j},\bm{c})$ is located at the $n$th ``turning point'' $r_{j,n}$ we have
\begin{align}
\label{eq:adj}
\begin{aligned}
    & \sigma_{j}^{-}(\bm{c}) = \sigma_{j,n}(\bm{c}) 
    ~\text{ and }~ \sigma_{j}^{+}(\bm{c}) = \sigma_{j,n+1}(\bm{c})
  \end{aligned}  
\end{align}
Similar to standard multi-arm bandits, the flatter the slope (e.g. $\sigma_{j}^{-}(\bm{c})$ is close to 0), the more pulls required to accurately estimate rewards for sub-optimal arms on the slope, but the lower the loss in conversion for pulling sub-optimal arms. Our setting is challenging because the magnitude of this tradeoff depends on the adversarial contexts $\bm{c}_{t}$, i.e., the dual variables, which requires delicate treatments. In the following Lemma  \ref{lem:ucbregret} where we bound the UCB error, we handle this context-dependent tradeoff by separately analyzing periods during which the adjacent slopes $\sigma_{j}^{-}(\bm{c})$ and $\sigma_{j}^{+}(\bm{c})$ are less or greater than some parameter $\underline{\sigma}$, and characterize the context-dependent tradeoff w.r.t. flatness of adjacent slopes using $\underline{\sigma}$.
\begin{lemma}[Bounding UCB error in primal per-channel budgets]
\label{lem:ucbregret}
Assume the discretization width $\delta$ satisfies $\delta < \underline{r}_{j}:= \min_{n \in [S_{j}]}r_{j,n} - r_{j,n-1}$, where $S_{j}$ and $\{r_{j,n}\}_{n=0}^{S_{j}}$ are defined in Lemma \ref{lem:Vstruct}. Then we have
\begin{align}
        \sum_{t\in[T]} 
     \mathbb{E}\left[\mathcal{L}_{j}(\rho_{j,t}^{*}, \lambda_{t}, \mu_{t}) - \mathcal{L}_{j}(\rho_{j,t}, \lambda_{t}, \mu_{t})\right]~\leq~ \mathcal{O}\left(\delta T + \underline{\sigma} T + \frac{1}{\underline{\sigma}\delta}\right)
    \end{align}
where $\underline{\sigma} > 0$ is any small positive number. {\color{red} why is this a good bound? Can we tune $\underline{\sigma}$ later?}
\end{lemma}
We refer the readers to Appendix \ref{pf:lem:ucbregret} for the proof. On the other hand,  the bound on SGD complementary slackness violation is presented in the following Lemma \ref{lem:boundcs}, and follows a standard argument for analyses on SGD; we refer readers to the proof in Appendix \ref{pf:lem:boundcs}. 
\begin{lemma}[Bounding complementary slackness deviations]
\label{lem:boundcs}
Recall $g_{1,t} = \sum_{j\in [M]}\left(V_{j}(\rho_{j,t};\bm{z}_{t}) - \gamma \rho_{j,t}\right)$, $g_{2,t} = \rho - \sum_{j\in [M]}\rho_{j,t}$ and $\eta > 0$ the step size defined in Algorithm \ref{alg:omducb}. Then we have
    \begin{align}
    \sum_{t\in[T]} \mathbb{E}\left[\lambda_{t}g_{1,t} + \mu_{t}g_{2,t}  \right] ~\leq~ 
    \mathcal{O}\left( \eta  T + \frac{1}{\eta}\right)
    \end{align}
\end{lemma}
Putting Lemmas \ref{lem:ucbregret} and \ref{lem:boundcs}  together into Eq. \eqref{eq:decompregret}, we obtain the main result of this section in the following Theorem \ref{thm:regret} whose proof we detail in Appendix \ref{pf:thm:regret}
\begin{theorem}[Putting everything together]
\label{thm:regret}
Assume assumptions  \ref{ass:budgetbind} and \ref{ass:feas} hold. Let $(\rho_{j,t})_{j\in [M], t\in [T]}$ be the per-channel budgets generated from Algorithm \ref{alg:omducb} and assume we take step size $\eta = \Theta(1/\sqrt{T})$,discretization width $\delta = \Theta(T^{-1/3}) $, and $\underline{\sigma} = \Theta(T^{-1/3})$ in Lemma \ref{lem:ucbregret}. Then we have 
\begin{align}
\label{eq:finalregret}
    T\cdot  \textsc{GL-OPT} - \mathbb{E}\left[\sum_{t\in [T]}\sum_{j\in[M]}V_{j}(\rho_{j,t})\right] \leq \mathcal{O}(T^{2/3}).
\end{align}
Recalling $\overline{\bm{\rho}}_{T} = \left(\frac{1}{T}\sum_{t\in[T]} \rho_{j,t}\right)_{j\in [M]}$ is the vector of time-averaged per-channel budgets, we have 
$ \textsc{GL-OPT} - \sum_{j \in [M]}\mathbb{E}\left[V_{j}(\overline{\rho}_{T,j})\right] \leq \mathcal{O}(T^{-1/3})$, and further $ \sum_{j \in [M]}\mathbb{E}\left[V_{j}(\overline{\rho}_{T,j}) -\gamma \overline{\rho}_{T,j}\right] \geq -\mathcal{O}(T^{-1/2})$ and $\rho - \sum_{j \in [M]}\mathbb{E}[\overline{\rho}_{T,j}] \geq -\mathcal{O}(T^{-1/2})$. {\color{red} in words, say this means the constraint of the problem are satisfied in expectation. }
\end{theorem}

{\color{red} NG: could we say sth about why this is a good bound?}
 









\subsection{Extension to strict constraint satisfaction: UCB-SGD-II}
In Theorem \ref{thm:regret}, we showed that the final output of the UCB-SGD Algorithm \ref{alg:omducb} outputs a per-channel budget profile $\overline{\bm{\rho}}_{T} = \left(\frac{1}{T}\sum_{t\in[T]} \rho_{j,t}\right)_{j\in [M]}$ that satisfies both ROI and budget constraints in $\textsc{CH-OPT}(\mathcal{I}_{B})$ approximately, i.e. there can be at most violations in the magnitude of $\mathcal{O}(T^{-1/2})$ for both constraints. In this subsection, we present a modification to UCB-SGD that enables us to achieve no-constraint violations, while still maintaining the $\mathcal{O}(T^{-1/3})$ accuracy in total conversion. Similar modification techniques have been introduced in \cite{balseiro2022best,feng2022online}. 

Our modification strategy handles ROI and budget constraint satisfactions differently. For budget constraints, we simply terminate the entire algorithm once the total spend comes close to $\rho T$. Regarding the ROI constraint, we develop two phases. Phase 1 is a ``safety phase'' where we conservatively utilize channel levers to accumulate a positive ``ROI balance'', i.e. in this phase (assume ending in period $T'$) we hope to achieve $\sum_{t\in [T']}g_{1,t} \geq \Theta(\sqrt{T})$ {\color{red}NG: $\Theta(\sqrt{T})$ or $\Theta(\sqrt{T'})$?}, where we recall $g_{1,t}$ defined in step 4 of Algorithm \ref{alg:omducb} can be viewed as the ROI constraint violations in period $t$. For phase 2, we then naively run SGD-UCB. Here, we aim to have a positive ROI balance in phase 1 that will be used to  compensate possible constraint violations in phase 2 when we run SGD-UCB. 

We remark that in order to implement the safety phase to attain a positive ROI balance, we resort to per-channel ROIs $(\gamma_{j})_{j\in[M]}$, since setting $\gamma_{j} = \gamma$ for all channels will ensure that our ROI balance increases. {\color{red}NG: did not understand  ``since setting $\gamma_{j} = \gamma$ for all channels will ensure that our ROI balance increases"} From a high level,  SGD-UCB attempts to approximate the optimal solution for $\textsc{CH-OPT}(\mathcal{I}_{B})$ by solely using the per-channel budget lever, whereas in the above modification with the usage of per-channel ROIs hopes to approximate an optimal solution to 
$\textsc{CH-OPT}(\mathcal{I}_{G})$ 
for the general option (see Eq. \eqref{eq:option}) which is inspired by the fact that  $\textsc{CH-OPT}(\mathcal{I}_{G}) = \textsc{CH-OPT}(\mathcal{I}_{B})=\textsc{GL-OPT}$ according to Lemma \ref{cor:roiredund} and Corollary \ref{cor:roiredund}. We call this modified algorithm SGD-UCB-II, and summarize details in Algorithm \ref{alg:omducb2}. {\color{red} NG: this is a nice insight. We show that if we are OK with satisfying the constraint with some small error, considering per-channel ROI constraint is not required. But, this is no longer the case where we cannot tolerate any constraint violations.}

\begin{algorithm}[H]
    \centering
    \caption{SGD-UCB-II}\label{alg:omducb2}
    \footnotesize
    \begin{algorithmic}[1]
    \Require{Set $B_{1}=\rho T$, and $g_{1,0} = 0$.}
    \While{$\sum_{t' \in [t-1]}g_{1,t'} \leq \sqrt{T}\log(T)$}
    \If{$B_{t} < M\rho $ {\color{red} NG: what is $B_t$? I know what it is. But, I don't think you defined it anywhere. }}
         \State Terminate  algorithm and output $\overline{\bm{\rho}}_{t} = \left(\frac{1}{t}\sum_{t'\in[t]} \rho_{j,t'}\right)_{j\in [M]}$.
    \EndIf
    \State  Set $\gamma_{j,t} = \gamma$ and $\rho_{j,t} = \rho$ for all $j \in [M]$ 
    \State Observe $\left\{V_{j}(\rho_{j,t};\bm{z}_{t}) \right\}_{j \in [M]}$, and calculate $g_{1,t} = \sum_{j \in M} \left(V_{j}(\rho_{j,t};\bm{z}_{t}) -  \gamma \rho_{j,t}\right)$
    \State Calculate $B_{t+1} = B_{t} - \sum_{j\in [M]}\rho_{j,t}$.
    \State Increment $t \leftarrow t+1$
	\EndWhile
 \State For remaining $T - t$ periods, set $\hat{\rho} = B_{t}/(T-t)$, and run SGD-UCB$(\gamma,\hat{\rho})$  in Algorithm \ref{alg:omducb}.
 \State Output $\overline{\bm{\rho}}_{T} = \left(\frac{1}{T}\sum_{t\in[T]} \rho_{j,t}\right)_{j\in [M]}$.
    \end{algorithmic}
\end{algorithm}
Our main result in this subsection is the following Theorem \ref{thm:regret2}:
\begin{theorem}
\label{thm:regret2}
    Let $\overline{\bm{\rho}} \in [0,\rho]^{M}$ be the final output of SGD-UCB-II in Algorithm \ref{alg:omducb2}. Then under the same assumptions and SGD-UCB parameter choices in Theorem \ref{thm:regret}, we have
$ \textsc{GL-OPT} - \sum_{j \in [M]}\mathbb{E}\left[V_{j}(\overline{\rho}_{j})\right]\leq \mathcal{O}(T^{-1/3})$, and further $\sum_{j \in [M]}\mathbb{E}\left[V_{j}(\overline{\rho}_{j}) -\gamma \overline{\rho}_{j}\right]\geq 0 $ and $ \sum_{j \in [M]} \mathbb{E}\left[\overline{\rho}_{j}\right]\leq \rho$. 
\end{theorem}
\subsection{Multi-item autobidding setup} 
\label{subsec:multisetup}

We first formalize our multi-item setup as followed. For each auction $n\in [m_{j}]$ of channel $j\in [M]$, assume $L_{j,n} \in \mathbb{N}$ impressions are sold, and channel $j$ is only allowed to procure at most 1 impression in auction $n$ on the advertiser's behalf. The value acquired and cost incurred by the advertiser when procuring impression $\ell \in [L_{j,n}]$ are $v_{j,n}(\ell)$ and $d_{j,n}(\ell)$, respectively. With a slight abuse of notation from previous sections, we write
 $\bm{v}_{j,n} = (v_{j,n}(1),\ldots v_{j,n}(L_{j,n})) \in \mathbb{R}_{+}^{L_{j,n}}$ as the $L_{j,n}$-dimensional vector that includes all impression values of auction $n$ in channel $j$, and further write  $\bm{v}_{j} = (\bm{v}_{j,1} \ldots \bm{v}_{j,m_{j}}) \in \mathbb{R}_{+}^{\sum_{n \in [m_{j}]}L_{j,n}}$ 
as the vector that concatenates all value vectors across auctions in channel $j$. 
We also define $\bm{d}_{j,n}\in \mathbb{R}_{+}^{L_{j,n}}$ and $\bm{d}_{j} \in \mathbb{R}_{+}^{\sum_{n \in [m_{j}]}L_{j,n}}$ accordingly for costs. Similar to Section \ref{sec:model}, we assume $\bm{z}_{j} = (\bm{v}_{j},\bm{d}_{j})$ is sampled from finite support $F_{j}$ according to discrete distribution $\bm{p}_{j}$ for any channel $j\in [M]$, and without loss of generality,  we assume that for any element $\bm{z}_{j} \in F_{j}$,  the value and costs for individual impressions in any auction $n \in [m_{j}]$ satisfy 
$v_{j,n}(1) > \ldots > v_{j,n}(L_{j,n}) > 0$  and $d_{j,n}(1) > \ldots > d_{j,n}(L_{j,n}) > 0$.

Under the above multi-item setup, an advertiser's global optimization problem (analogous to \textsc{GL-OPT}
in Eq. \eqref{eq:glopt} for the single-item auction setup in previous sections), can be written as the following problem called $\textsc{GL-OPT}^{+}$:
\begin{align}
\label{eq:gloptmulti}
\begin{aligned}
\textsc{GL-OPT}^{+}
~=~ & \\
\max_{ \left(\bm{x}_{j} = (\bm{x}_{j,1}\ldots, \bm{x}_{j,m_{j}})\right)_{j\in [M]}} ~&~ \sum_{j\in [M]}\mathbb{E}\left[\bm{v}_{j}^{\top} \bm{x}_{j}\right] \\
     \text{s.t. } & \sum_{j\in [M]}\mathbb{E}\left[\bm{v}_{j}^{\top} \bm{x}_{j}\right]~\geq~ \gamma \sum_{j\in [M]} \mathbb{E}\left[\bm{d}_{j}^{\top} \bm{x}_{j}\right]\\
     & \sum_{j\in [M]} \mathbb{E}\left[\bm{d}_{j}^{\top} \bm{x}_{j}\right] ~\leq~\rho \\
     & \bm{x}_{j,n}\in [0,1]^{\sum_{n \in [m_{j}]}L_{j,n}} \text{ and } \sum_{\ell \in [L_{j,n}]}x_{j,n}(\ell) \leq 1,  \quad \forall j\in [M],~ n\in [m_{j}]
    \end{aligned}
\end{align} 
Here
$\bm{x}_{j,n} = (x_{j,n}(\ell))_{\ell \in [L_{j,n}]}$ denotes the indicator vector for procuring impressions $\ell \in L_{j,n}$ in auction $n \in [m_{j}]$ of channel $j\in [M]$. Compared to \textsc{GL-OPT}, the key difference for $\textsc{GL-OPT}^{+}$ is that we introduced additional constraints which states ``at most 1 impression is procured in every multi-item auction''.

On the other hand, analogous to a channel's autobidding problem for the single-item auction setup in previous sections (Eq. \eqref{eq:budperchannelsol}), in the multi-item setting each channel $j$'s autobidding problem can be written as
\begin{align}
\begin{aligned}
\label{eq:budperchannelsolmulti}
    \bm{x}_{j}^{*,+}(\gamma_{j},\rho_{j};\bm{z}_{j}) ~=~  \arg\max_{\bm{x} = (\bm{x}_{1} \ldots \bm{x}_{m_{j}})} & ~ \bm{v}_{j}^{\top} \bm{x} \\
    \quad 
     \text{s.t. }  \quad  & \bm{v}_{j}^{\top} \bm{x}\geq \gamma_{j}\bm{d}_{j}^{\top}\bm{x} ,\text{ and }  \quad  \bm{d}_{j}^{\top} \bm{x}\leq \rho_{j}\\
    & \bm{x}_{n}\in [0,1]^{L_{j,n}} \text{ and } \sum_{\ell \in [L_{j,n}]}x_{n}(\ell) \leq 1,  \quad \forall  n\in [m_{j}]
    \end{aligned}
\end{align} 
where $\bm{x}_{n} =(x_{n}(\ell))_{\ell\in [L_{j,n}]}\in [0,1]^{m_{j}}$ denotes the (possibly random) vector of indicators to win each impression of auction $n$ in channel $j$. With respect to this per-channel multi-item auction optimization problem in Eq. \eqref{eq:budperchannelsolmulti}, we can further define $V_{j}^{+}(\gamma_{j},\rho_{j};\bm{z}_{j})$, $D_{j}^{+}(\gamma_{j},\rho_{j};\bm{z}_{j}) $, $V_{j}^{+}(\gamma_{j},\rho_{j})$, $D_{j}^{+}(\gamma_{j},\rho_{j})$ as in Eq.\eqref{eq:budperchannel}, and $\textsc{CH-OPT}^{+}(\mathcal{I})$ as in Eq.\eqref{eq:chopt} for any advertiser lever option $\mathcal{I}$ in Eq.\eqref{eq:option}. 


\subsection{Optimizing per-channel budgets is sufficient to achieve global optimal}
\label{subsec:multifutil}

Our first main result for the multi-item setting is the following Theorem \ref{thm:roiredundmulti} which again shows an advertiser can achieve the global optimal conversion $\textsc{GL-OPT}^{+}$ via solely optimizing over per-channel budgets (analogous to Theorem \ref{thm:budgonlysuff} and Corollary \ref{cor:roiredund}).
\begin{theorem}[Redundancy of per-channel ROIs in multi-slot auctions]
\label{thm:roiredundmulti}
For the per-channel budget option $\mathcal{I}_{B}$ and general options $\mathcal{I}_{G}$ defined in Eq.\eqref{eq:option},  we have  $\textsc{GL-OPT}^{+}=\textsc{CH-OPT}^{+}(\mathcal{I}_{B})= \textsc{CH-OPT}^{+}(\mathcal{I}_{G})$ for any aggregate ROI $\gamma > 0$ and total budget $\rho > 0$, even for $\rho= \infty$. Further, there exists and optimal solution $(\gamma_{j},\rho_{j})_{j\in [M]}$ to $\textsc{CH-OPT}^{+}(\mathcal{I}_{G})$, s.t. $\gamma_{j} = 0$ for all $j \in [M]$.
\end{theorem}
It is easy to see that the proof of Lemma \ref{lem:glbest}, Theorem \ref{thm:budgonlysuff}, and Corollary \ref{cor:roiredund} w.r.t. the single item setting in Section \ref{sec:futil} can be directly applied to Theorem \ref{thm:roiredundmulti} since we did not rely on specific structures of the solutions to \textsc{GL-OPT} and \textsc{CH-OPT} other than the presence of the respective ROI and budget constraints
(which are still present in 
$\textsc{GL-OPT}^{+}$ and $\textsc{CH-OPT}^{+}$). Thereby we will omit the proof of Theorem \ref{thm:roiredundmulti}.  In light of Theorem \ref{thm:roiredundmulti}, we again conclude that the per-channel ROI lever is redundant, and hence omit per-channel ROI $\gamma_{j}$ when the context is clear.



\subsection{Applying UCB-SGD to the multi-item setting } 
\label{subsec:multionline}
We now turn to our second main focus of the multi-item setting, which is to understand whether our proposed UCB-SGD algorithm can achieve accurate approximations to the optimal per-channel budgets, similar to Theorem \ref{thm:regret} for the single-item setting. A key observation is that the only difference between bounding the error of UCB-SGD in the single and multi-item settings is the structure of the conversion and corresponding Lagrangian functions (see Lemma \ref{lem:Vstruct}), since the only change in the multi-item setting compared to the single-item setting is how a given per-channel budget translates into a certain conversion. 
Therefore, in this section we introduce a broad class of multi-item auction formats that induce the same conversion function structural properties as those illustrated in Lemma \ref{lem:Vstruct}, which will allows us to directly apply the proof for bounding the error of UCB-SGD (Theorem \ref{thm:regret}) to the multi-item setting of interest. 

To begin with, we introduce the following notion of increasing marginal values, which is a characteristic that preserves the 
structural properties for conversion and Lagrangian functions from the single-item setting (in Lemma \ref{lem:Vstruct}), as shown later in Lemma \ref{lem:Vstructmulti}.
\begin{definition}[Multi-item auctions with increasing marginal values]
\label{def:incrmarginal}
 We say an auction $n \in [m_{j}]$ in channel $j\in [M]$ has increasing marginal values if for any realization $\bm{z}_{j} = (\bm{v}_{j},\bm{d}_{j})$, 
we have 
\begin{align*}
    \frac{v_{j,n}(1)-v_{j,n}(2)}{d_{j,n}(1)-d_{j,n}(2)} > \ldots > \frac{v_{j,n}(L_{j,n}-1)-v_{j,n}(L_{j,n})}{d_{j,n}(L_{j,n}-1)-d_{j,n}(L_{j,n})} > \frac{v_{j,n}(L_{j,n})}{d_{j,n}(L_{j,n})} > 0\,,
\end{align*}
where we recall $v_{j,n}(1) > \ldots > v_{j,n}(L_{j,n}) > 0$  and $d_{j,n}(1) > \ldots > d_{j,n}(L_{j,n}) > 0$. 
\end{definition}
Increasing marginal values intuitively means that in some multi-item auction, the marginal value per cost gained increases with procuring impressions of greater values. Many classic position auction formats satisfy increasing marginal gains, such as the Vickrey–Clarke–Groves (VCG) auction; see \cite{varian2007position,edelman2007internet} for more details on position auctions.

\begin{example}[VCG auctions have increasing marginal values]
Let auction $n \in [m_{j}]$ in channel $j \in [M]$ be a VCG auction, where  
for any realization of $(\bm{v}_{j,n},\bm{d}_{j,n}) = (v_{j,n}(\ell), d_{j,n}(\ell))_{\ell\in [L_{j,n}]}$
there exists some $\widetilde{v}_{n,j} > 0$, position discounts $1 \geq \theta_{n,j}(1) >\theta_{n,j}(2) \ldots \theta_{n,j}(L_{n,j}) > 0$, and $L_{n,j}$-highest competing bids from competitors in the market  $\widetilde{d}_{n,j}(1) > \widetilde{d}_{n,j}(2)\ldots > \widetilde{d}_{n,j}(L_{n,j}) > 0$, such that the acquired value for procuring impression $\ell\in L_{n,j}$ is $v_{n,j}(\ell) = \theta_{n,j}(\ell)\cdot \widetilde{v}_{n,j}$, and the corresponding cost  is $d_{j,n}(\ell) = \sum_{\ell'=\ell}^{L_{j,n}}(\theta_{n,j}(\ell')- \theta_{n,j}(\ell'+1))\widetilde{d}_{n,j}(\ell')$ where we denote $\theta_{n,j}(L_{j,n}+1)= 0$.\footnote{Here, the distribution over $(\bm{v}_{j,n},\bm{d}_{j,n})$ can be viewed as the joint distribution over $\widetilde{v}_{n,j}$, $(\theta_{n,j}(\ell))_{\ell\in [L_{j,n}]}$ and $(\widetilde{d}_{n,j}(\ell))_{\ell\in [L_{j,n}]}$.} 
Thereby, under VCG the marginal values are
\begin{align*}
    \frac{v_{j,n}(\ell)-v_{j,n}(\ell+1)}{d_{j,n}(\ell)-d_{j,n}(\ell+1)} =  \frac{\left(\theta_{n,j}(\ell)-\theta_{n,j}(\ell+1)\right)\widetilde{v}_{j,n}}{\left(\theta_{n,j}(\ell)-\theta_{n,j}(\ell+1)\right)\widetilde{d}_{j,n}(\ell)} = \frac{\widetilde{v}_{j,n}}{\widetilde{d}_{j,n}(\ell)}\,,
\end{align*}
which decreases in $\ell$ since $\widetilde{d}_{n,j}(1) > \widetilde{d}_{n,j}(2)\ldots > \widetilde{d}_{n,j}(L_{n,j}) > 0$. Hence  VCG auctions admit increasing marginal values. 
\end{example}
We remark that the generalized second price auction (GSP) does not necessarily have increasing marginal values. Now, if all auctions in a channel have increasing marginal values, then we can show the conversion function $V_{j}^{+}(\rho_{j})$ and the corresponding Lagrangian function for multi-item auctions admits the same structural properties as those in Lemma \ref{lem:Vstruct}:
\begin{lemma}[Structural properties for multi-item auctions]
\label{lem:Vstructmulti}
For any channel $j\in [M]$ whose auctions have increasing marginal values (see Definition \ref{def:incrmarginal}), the conversion function $V_{j}^{+}(\rho_{j})= \mathbb{E}\left[\bm{v}_{j}^{\top} \bm{x}_{j}^{*,+}(\rho_{j};\bm{z}_{j})\right]$  is continuous, piece-wise linear, strictly increasing, and concave. Here recall $\bm{x}_{j}^{*,+}(\rho_{j};\bm{z}_{j})$ is the optimal solution to the channel's optimization problem in Eq. \eqref{eq:budperchannelsolmulti}. Further, for any dual variables $\bm{c} = (\lambda,\theta) \in \mathbb{R}_{+}^{2}$, the Lagrangian function $\mathcal{L}_{j}^{+}(\rho_{j},\bm{c}):= (1+\lambda)V_{j}^{+}(\rho_{j}) - (\theta+\gamma\lambda)\rho_{j}$ is continuous, piece-wise linear, concave, and unimodal in $\rho_{j}$. 
\end{lemma}
See the proof in Appendix \ref{pf:lem:Vstructmulti}. In light of Lemma \ref{lem:Vstructmulti}, we can argue that UCB-SGD produces per-channel budgets that yield the same accuracy as in Theorem \ref{thm:regret} for the single-item setting,

\begin{theorem}[UCB-SGD applied to channel procurement for multi-item auctions]
Assume multi-item auctions in any channel $j\in [M]$ has increasing marginal values (per Definition \ref{def:incrmarginal}), and assume Assumptions \ref{ass:budgetbind} and  \ref{ass:feas} hold for the multi-item setting.\footnote{Assumption \ref{ass:budgetbind} in the multi-item setting again implies the spend in any channel is exactly the input per-channel budget; Assumption \ref{ass:feas} in the multi-item setting states that for any realization of value-cost pairs  $\bm{z} =(\bm{v}_{j},\bm{d}_{j})_{j\in[M]} \in F_{1}\times \dots F_{M}$, the realized version of the ROI constraint in $\textsc{GL-OPT}^{+}$ defined in Eq. \eqref{eq:gloptmulti} is strictly feasible.} Then with the same parameter choices as in Theorem \ref{thm:regret}, and recalling $\overline{\bm{\rho}}_{T} = \left(\frac{1}{T}\sum_{t\in[T]} \rho_{j,t}\right)_{j\in [M]}$ is the vector of time-averaged per-channel budgets produced by UCB-SGD, we have
\begin{align*}
    \textsc{GL-OPT}^{+} - \sum_{j \in [M]}\mathbb{E}\left[V_{j}^{+}(\overline{\rho}_{T,j})\right] \leq \mathcal{O}(T^{-1/3})\,,
\end{align*}
as well as approximate constraint satisfaction
\begin{align*}
     \sum_{j \in [M]}\mathbb{E}\left[V_{j}^{+}(\overline{\rho}_{T,j}) -\gamma \overline{\rho}_{T,j}\right] \geq -\mathcal{O}(T^{-1/2}),~\text{ and }\rho - \sum_{j \in [M]}\mathbb{E}[\overline{\rho}_{T,j}] \geq -\mathcal{O}(T^{-1/2})\,,
\end{align*}
where we recall $\textsc{GL-OPT}^{+}$ is defined in Eq. \eqref{eq:gloptmulti}, $V_{j}^{+}(\rho_{j})  = \mathbb{E}\left[\bm{v}_{j}^{\top} \bm{x}_{j}^{*,+}(\rho_{j};\bm{z}_{j})\right]$ and $\bm{x}_{j}^{*,+}(\rho_{j};\bm{z}_{j})$ is defined in Eq. \eqref{eq:budperchannelsolmulti}.
\end{theorem}
The proof for this theorem is identical to that of Theorem \ref{thm:regret} given the same structural properties of the conversion and Lagrangian functions in Lemma \ref{lem:Vstructmulti} and Lemma \ref{lem:Vstruct}. Hence we will omit the proof. Finally, we remark that UCB-SGD-II (Algorithm \ref{alg:omducb2}) can also be applied to the multi-item setting and yield per-channel budget estimates that achieve the same performance as illustrated in Theorem \ref{thm:regret2} while satisfying both global budget and ROI constraints exactly. 
\subsection{More general advertiser objectives}
\label{subsec:genutil}
In \textsc{GL-OPT}  and $\textsc{CH-OPT}(\mathcal{I})$  defined Section  \ref{sec:model}
(or similarly $\textsc{GL-OPT}^{+}$ and $\textsc{CH-OPT}^{+}(\mathcal{I})$ defined in the multi-item setting in Section \ref{sec:multi}),  we can also consider more general objectives, namely $\max_{\bm{x}_{1},\ldots, \bm{x}_{M}} ~ \sum_{j\in [M]}\mathbb{E}\left[\bm{v}_{j}^{\top} \bm{x}_{j} - \alpha\bm{d}_{j}^{\top} \bm{x}_{j} \right] $ and $\max_{(\gamma_{j},\rho_{j})_{j\in [M]}\in \mathcal{I}} ~ \sum_{j \in M} \mathbb{E}\left[V_{j}(\gamma_{j},\rho_{j};\bm{z}_{j})-\alpha V_{j}(\gamma_{j},\rho_{j};\bm{z}_{j})\right]$ for some private cost $\alpha\in [0,\gamma]$\footnote{If $\alpha > \gamma$ the ROI constraints in \textsc{GL-OPT} as well as $\textsc{CH-OPT}(\mathcal{I})$ become redundant.} in \textsc{GL-OPT} and $\textsc{CH-OPT}(\mathcal{I})$, respectively. When $\alpha = 0$, we recover our considered models in the previous section, whereas in when $\alpha = 1$, we obtain the classic quasi-linear utility. We remark that this private cost model has been introduced and studied in related literature; see \cite{balseiro2019black} and references therein. Nevertheless, when each channel's autobidding problem remains as is in  Eq.\eqref{eq:budperchannelsol}, i.e. channels still aim to maximize conversion which causes a misalignment between advertiser objectives and channel behavior, it is not difficult to see in our proofs that all our results still hold in Section \ref{sec:futil}, and our UCB-SGD algorithm still produces estimates of the same order of accuracy via introducing $\alpha$ into the Lagrangian. 
In other words, even if channels aim to maximize total conversion for advertisers, advertisers can optimize for \textsc{GL-OPT} with a private cost $\alpha$ through optimizing  $\textsc{CH-OPT}(\mathcal{I})$ that also incorporates the same private cost. 



\subsection{Non-optimal autobidding in channels.}
\label{subsec:nonoptautobid}
We recall in previous sections we assumed that each channel adopt ``optimal autobidding'' that solves Eq. \eqref{eq:budperchannelsol} to optimality. This raises the natural question that whether our findings will still hold when channels do not procure ads optimally, perhaps because of non-stationary environments \cite{besbes2014stochastic,luo2018efficient,cheung2019learning}, or the presence of strategic market participants who aim to manipulate the market \cite{golrezaei2019incentive,drutsa2020optimal,golrezaei2021learning,golrezaei2021bidding}. In such a scenario, an advertiser's  (bandit) conversion feedback in a channel $j$ would be $V(\gamma_{j},\rho_{j};\bm{z}_{j}) - \epsilon_{j}$ for some channel-specific and possibly adversarial loss $\epsilon_{j} > 0$. One potential resolution is to treat such $\epsilon_{j}$ as adversarial corruptions to bandit rewards, and instead of integrating vanilla UCB with SGD as in Algorithm \ref{alg:omducb}, augment SGD with bandit algorithms that are robust to robust corruptions; see e.g. \cite{lykouris2018stochastic,gupta2019better}. Nevertheless, it remains an open question to prove how such augmentation would perform in our specific bandit-feedback constrained optimization setup. This leads to potential research directions of both practical and theoretical significance.
\section{Extended Literature Review}






\section{Proofs for Section \ref{sec:futil}}
\input{A2_futil}




\section{Proofs for Section \ref{sec:online}}
\input{A3_online}


\section{Proofs for Section \ref{sec:multi}}
\input{A4_multislot}




\end{APPENDICES}
\subsection{Proof of Lemma \ref{lem:glbest}}
\label{pf:lem:glbest}

Fix any option $\mathcal{I} \in \{ \mathcal{I}_{B}, \mathcal{I}_{R},\mathcal{I}_{G}\}$ defined in Eq. \eqref{eq:option}, and let $(\widetilde{\bm{\gamma}},\widetilde{\bm{\rho}} )\in \mathcal{I}$ be the optimal solution to $\textsc{CH-OPT}(\mathcal{I})$. Note that for the per-channel ROI only option $\mathcal{I}_{R}$, we have $\widetilde{\rho}_{j} = \infty$ and for the per-channel budget only we have $\widetilde{\gamma}_{j} = 0$ for all $j\in[M]$. Further, for any realization of value-cost pairs over all auctions 
$\bm{z} = (\bm{v}_{j},\bm{d}_{j})_{j\in [M]}$, recall the optimal solution $\bm{x}_{j}^{*}(\widetilde{\gamma}_{j}, \widetilde{\rho}_{j};\bm{z}_{j}) $
to $V_{j}(\widetilde{\gamma}_{j}, \widetilde{\rho}_{j};\bm{z}_{j})$ for each channel $j \in [M]$ as defined in Eq. \eqref{eq:budperchannelsol}. 

Due to feasibility of  $(\widetilde{\bm{\gamma}},\widetilde{\bm{\rho}} )\in \mathcal{I}$ for $\textsc{CH-OPT}(\mathcal{I})$, we have
\begin{align*}
    \sum_{j \in M} \mathbb{E}\left[V_{j}(\widetilde{\gamma}_{j},\widetilde{\rho}_{j}; \bm{z}_{j})\right] \geq \gamma \sum_{j \in M} \mathbb{E}\left[D_{j}(\widetilde{\gamma}_{j},\widetilde{\rho}_{j}; \bm{z}_{j})\right] \Longrightarrow \sum_{j\in [M]} \mathbb{E}\left[\bm{v}_{j}^{\top} \bm{x}_{j}^{*}(\widetilde{\gamma}_{j},\widetilde{\rho}_{j}; \bm{z}_{j})\right] \geq \gamma \sum_{j\in [M]} \left[\bm{d}_{j}^{\top}\bm{x}_{j}^{*}(\widetilde{\gamma}_{j},\widetilde{\rho}_{j}; \bm{z}_{j})\right]
\end{align*}
where we used the definitions $V_{j}(\widetilde{\gamma}_{j},\widetilde{\rho}_{j}; \bm{z}_{j}) = \bm{v}_{j}^{\top} \bm{x}_{j}^{*}(\widetilde{\gamma}_{j},\widetilde{\rho}_{j}; \bm{z}_{j})$ and $D_{j}(\widetilde{\gamma}_{j},\widetilde{\rho}_{j}; \bm{z}_{j}) = \bm{d}_{j}^{\top} \bm{x}_{j}^{*}(\widetilde{\gamma}_{j},\widetilde{\rho}_{j}; \bm{z}_{j})$ in Eq. \eqref{eq:budperchannel}.
This implies $\left(\bm{x}_{j}^{*}(\widetilde{\gamma}_{j},\widetilde{\rho}_{j}; \bm{z}_{j})\right)_{j\in[M]}$ satisfies the ROI constraint in \textsc{GL-OPT}. A similar analysis implies $\left(\bm{x}_{j}^{*}(\widetilde{\gamma}_{j},\widetilde{\rho}_{j}; \bm{z}_{j})\right)_{j\in[M]}$ also satisfies the budget constraint in \textsc{GL-OPT}. Therefore, 
$\left(\bm{x}_{j}^{*}(\widetilde{\gamma}_{j},\widetilde{\rho}_{j}; \bm{z}_{j})\right)_{j\in[M]}$ is feasible to \textsc{GL-OPT}. So
\begin{align}
    \textsc{GL-OPT}\geq  \sum_{j\in [M]} \mathbb{E}\left[\bm{v}_{j}^{\top} \bm{x}_{j}^{*}(\widetilde{\gamma}_{j},\widetilde{\rho}_{j}; \bm{z}_{j})\right] =  \sum_{j \in M}\left[ V_{j}(\widetilde{\gamma}_{j},\widetilde{\rho}_{j}; \bm{z}_{j})\right] = \textsc{CH-OPT}(\mathcal{I})\,.
\end{align}
where the final equality follows from the assumption that $(\widetilde{\bm{\gamma}},\widetilde{\bm{\rho}} )\in \mathcal{I}$ is the optimal solution to $\textsc{CH-OPT}(\mathcal{I})$.
\halmos



\subsection{Proof of Theorem \ref{thm:budgonlysuff}}
\label{pf:thm:budgonlysuff}
In light of Lemma \ref{lem:glbest}, we only
need to show $\textsc{CH-OPT}(\mathcal{I}_{B}) \geq \textsc{GL-OPT}$.  Let $\widetilde{\bm{x}}(\bm{z}) = \{\widetilde{\bm{x}}_{j}(\bm{z}_{j}))\}_{j\in [N]}$
 be the optimal solution to \textsc{GL-OPT}, and define $\widetilde{\gamma}_{j} = 0$ and $\widetilde{\rho}_{j}= \mathbb{E}\left[\bm{d}_{j}^{\top}\widetilde{\bm{x}}_{j}(\bm{z}_{j}))\right]$ 
to be the corresponding expected spend for each channel $j$ under the optimal solution $\widetilde{\bm{x}}(\bm{z})$ to \textsc{GL-OPT}, respectively. 


We first argue that $(\widetilde{\gamma}_{j}, \widetilde{\rho}_{j})_{j\in[M]}$ is feasible to  $\textsc{CH-OPT}(\mathcal{I}_{B})$. Recall the optimal solution $\bm{x}_{j}^{*}(\widetilde{\gamma}_{j}, \widetilde{\rho}_{j};\bm{z}_{j}) $ to $V_{j}(\widetilde{\gamma}_{j}, \widetilde{\rho}_{j};\bm{z}_{j})$
for each channel $j \in [M]$ as defined in Eq. \eqref{eq:budperchannelsol}, as well as the definitions $V_{j}(\widetilde{\gamma}_{j},\widetilde{\rho}_{j}; \bm{z}_{j}) = \bm{v}_{j}^{\top} \bm{x}_{j}^{*}(\widetilde{\gamma}_{j},\widetilde{\rho}_{j}; \bm{z}_{j})$ and $D_{j}(\widetilde{\gamma}_{j},\widetilde{\rho}_{j}; \bm{z}_{j}) = \bm{d}_{j}^{\top} \bm{x}_{j}^{*}(\widetilde{\gamma}_{j},\widetilde{\rho}_{j}; \bm{z}_{j})$ in Eq. \eqref{eq:budperchannel}.
Then,
we have 
\begin{align}
\label{eq:budgsuffbound1}
    \mathbb{E}\left[D_{j}(\widetilde{\gamma}_{j},\widetilde{\rho}_{j}; \bm{z}_{j})\right] ~=~ \mathbb{E}\left[\bm{d}_{j}^{\top}\bm{x}_{j}^{*}(\widetilde{\gamma}_{j},\widetilde{\rho}_{j});\bm{z}_{j}\right]~\overset{(i)}{\leq}~ \widetilde{\rho}_{j} ~=~ \mathbb{E}\left[\bm{d}_{j}^{\top}\widetilde{\bm{x}}_{j}(\bm{z}_{j})\right]\,,
\end{align}
where (i) follows from feasibility of  $\bm{x}_{j}^{*}(\widetilde{\gamma}_{j}, \widetilde{\rho}_{j};\bm{z}_{j}) $ to $V_{j}(\widetilde{\gamma}_{j},\widetilde{\rho}_{j}; \bm{z}_{j})$. Summing over $j \in [M]$ we conclude that $(\bm{\gamma}_{j}, \bm{\rho}_{j})_{j\in[M]}$ satisfies the budget constraint in 
$\textsc{CH-OPT}(\mathcal{I}_{B})$:
\begin{align}
\label{eq:budgsuffbound0}
    \sum_{j \in [M]} \mathbb{E}\left[D_{j}(\widetilde{\gamma}_{j},\widetilde{\rho}_{j}; \bm{z}_{j})\right] \leq \sum_{j \in [M]} \mathbb{E}\left[\bm{d}_{j}^{\top}\widetilde{\bm{x}}_{j}(\bm{z}_{j})\right] \overset{(i)}{\leq} \rho\,.
\end{align}
Here (i) follows from feasibility of $\widetilde{\bm{x}}(\bm{z}) = \{\widetilde{\bm{x}}_{j}(\bm{z}_{j}))\}_{j\in [N]}$
 to \textsc{GL-OPT} since it is the optimal solution.
 
On the other hand, we have
\begin{align}
     V_{j}(\widetilde{\gamma}_{j},\widetilde{\rho}_{j};\bm{z}_{j})~=~ \bm{v}_{j}^{\top}\bm{x}_{j}^{*}(\widetilde{\gamma}_{j},\widetilde{\rho}_{j};\bm{z}_{j})~\overset{(i)}{\geq}~ \bm{v}_{j}^{\top}\widetilde{\bm{x}}_{j}(\bm{z}_{j})
\end{align}
where (i) follows from optimality of $\bm{x}_{j}^{*}(\widetilde{\gamma}_{j},\widetilde{\rho}_{j};\bm{z}_{j})$ to $ V_{j}(\widetilde{\gamma}_{j},\widetilde{\rho}_{j};\bm{z}_{j})$. Hence, we have
\begin{align}
\label{eq:budgsuffbound2}
     \sum_{j \in M} \mathbb{E}\left[V_{j}(\widetilde{\gamma}_{j},\widetilde{\rho}_{j};\bm{z}_{j})\right]~\geq~  \sum_{j \in M}\mathbb{E}\left[\bm{v}_{j}^{\top}\widetilde{\bm{x}}_{j}(\bm{z}_{j})\right]
     ~\overset{(i)}{\geq}~ \gamma \sum_{j \in M} \mathbb{E}\left[\bm{d}_{j}^{\top}\widetilde{\bm{x}}_{j}(\bm{z}_{j})\right] ~\overset{(ii)}{\geq}~ \gamma  \sum_{j \in [M]} \mathbb{E}\left[D_{j}(\widetilde{\gamma}_{j},\widetilde{\rho}_{j}; \bm{z}_{j})\right] 
\end{align}
where (i) follows from feasibility of $\widetilde{\bm{x}}(\bm{z}) = \{\widetilde{\bm{x}}_{j}(\bm{z}_{j}))\}_{j\in [N]}$
 to \textsc{GL-OPT} since it is the optimal solution; (ii) follows from Eq. \eqref{eq:budgsuffbound1}.  Hence combining Eq. \eqref{eq:budgsuffbound0} \eqref{eq:budgsuffbound2} we can conclude that $(\widetilde{\gamma}_{j}, \widetilde{\rho}_{j})_{j\in[M]}$  is feasible to  $\textsc{CH-OPT}(\mathcal{I}_{B})$. 
 
Finally, we have $ \textsc{CH-OPT}(\mathcal{I}_{B}) \geq  \sum_{j \in M} \mathbb{E}\left[V_{j}(\widetilde{\gamma}_{j},\widetilde{\rho}_{j}; \bm{z}_{j})\right]  \geq \sum_{j \in M} \mathbb{E}\left[\bm{v}_{j}^{\top}\widetilde{\bm{x}}_{j}(\bm{z}_{j})\right] = \textsc{GL-OPT}$, where the last inequality follows from \eqref{eq:budgsuffbound2}, and the final equality is because we assumed $\widetilde{\bm{x}}(\bm{z}) = \{\widetilde{\bm{x}}_{j}(\bm{z}_{j}))\}_{j\in [N]}$
 is the optimal solution to \textsc{GL-OPT}.
\halmos


\subsection{Proof of Corollary \ref{cor:roiredund}}
\label{pf:cor:roiredund}
In light of Lemma \ref{lem:glbest}, we only
need to show $\textsc{CH-OPT}(\mathcal{I}_{G}) \geq \textsc{GL-OPT}$. Let $(\widetilde{\bm{\gamma}},\widetilde{\bm{\rho}})\in \mathcal{I}_{B}$, and by definition of $\mathcal{I}_{B}$ in Eq. \eqref{eq:option} we have $\widetilde{\gamma}_{j} = 0$ for all $j\in [M]$. Since $(\widetilde{\bm{\gamma}},\widetilde{\bm{\rho}})$ is feasible to $\textsc{CH-OPT}(\mathcal{I}_{B})$, it is also feasible to $\textsc{CH-OPT}(\mathcal{I}_{G})$ since these two problems share the same ROI and budget constraints. Because they also share the same objectives, we have
\begin{align}
   \textsc{CH-OPT}(\mathcal{I}_{G}) \geq \textsc{CH-OPT}(\mathcal{I}_{B}) = \textsc{GL-OPT}
\end{align}
where the final equality follows from Theorem \ref{thm:budgonlysuff}. 
\halmos
\subsection{Proof of Proposition \ref{prop:budgetbind}}




\subsection{Proof of Proposition \ref{prop:ubbenchlag}}
\label{pf:prop:ubbenchlag}
Define $\Bar{\mu}_{T} = \frac{1}{T}\sum_{t\in[T]}\mu_{t}$ as well as $\Bar{\lambda}_{T} = \frac{1}{T}\sum_{t\in[T]}\lambda_{t}$ . Let $(\rho_{j}^{*})_{j\in[M]}$ be the optimal per-channel budgets to $ \textsc{CH-OPT}(\mathcal{I}_{B})$, then we have the following weak duality statement for any $\lambda, \mu \geq 0$:
\begin{align}
\begin{aligned}
\label{eq:weakduality}
    \textsc{CH-OPT}(\mathcal{I}_{B}) = \sum_{j\in [M]}V_{j}(\rho_{j}^{*}) ~\overset{(i)}{\leq}~ & \sum_{j\in [M]}V_{j}(\rho_{j}^{*}) + \lambda\sum_{j\in[M]} \left(V_{j}(\rho_{j}^{*}) - \gamma \rho_{j}^{*}\right) + \mu\left(\rho - \sum_{j\in[M]} \rho_{j}^{*}\right)\\
    ~\overset{(ii)}{\leq}~ & \sum_{j\in[M]}\mathcal{L}(\rho_{j}^{*},\lambda,\mu) + \rho \mu 
\end{aligned}
\end{align}
where (i) follows from feasibility of $(\rho_{j}^{*})_{j\in[M]}$ to $ \textsc{CH-OPT}(\mathcal{I}_{B})$; (ii) follows from the definition of the Lagrangian function in Eq. \eqref{eq:Lagrangian}. We can further bound the advertiser's regret as followed:
\begin{align}
\begin{aligned}
     & T\cdot \textsc{GL-OPT} - \mathbb{E}\left[\sum_{t\in [T]}\sum_{j\in[M]}V_{j}(\rho_{j,t})\right] \\
     
    ~\overset{(i)}{=}~ & M\Bar{V}K + (T-K)\cdot \textsc{CH-OPT}(\mathcal{I}_{B}) - \mathbb{E}\left[\sum_{t > K}\sum_{j\in[M]}V_{j}(\rho_{j,t})\right]\\
     ~\overset{(ii)}{\leq}~ & M\Bar{V}K + (T-K)\cdot \sum_{j\in [M]}\mathbb{E}\left[\mathcal{L}_{j}(\rho_{j}^{*}, \Bar{\lambda}_{T} , \Bar{\mu}_{T})  + \rho \Bar{\mu}_{T}\right] - \mathbb{E}\left[\sum_{t > K}\sum_{j\in[M]}V_{j}(\rho_{j,t})\right]\\
     ~\overset{(iii)}{<}~ & M\Bar{V}K + \rho\sum_{t > K}\mathbb{E}\left[\mu_{t}\right] + \sum_{t>K}\sum_{j\in [M]} \mathbb{E}\left[\mathcal{L}_{j}(\rho_{j}^{*}, \lambda_{t}, \mu_{t})\right] -  \sum_{t > K}\sum_{j\in [M]}\mathbb{E}\left[\mathcal{L}_{j}(\rho_{j,t}, \bm{c}_{t}) - \lambda_{t}\left(V_{j}(\rho_{j,t}) - \gamma\rho_{j,t}\right) + \mu_{t}\rho_{j,t}\right]\\
      ~\overset{(iv)}{\leq}~ & M\Bar{V}K +  \sum_{j\in [M]} \sum_{t > K} 
     \mathbb{E}\left[\mathcal{L}_{j}(\rho_{j}^{*}(t), \bm{c}_{t}) - \mathcal{L}_{j}(\rho_{j,t}, \bm{c}_{t})\right] +  \sum_{t > K}\left(\lambda_{t}g_{1,t} + \mu_{t}g_{2,t}\right)\,.
    \end{aligned}
\end{align}
 In (i), we first used the trivial upper bound $\Bar{V}$ on conversion functions in each channel for the first $K$ periods where we choose each  of the arms once and pull that arm across all channels. Then we applied $\textsc{GL-OPT} =   \textsc{CH-OPT}(\mathcal{I}_{B})$ from Theorem \ref{thm:budgonlysuff};  (ii) follows from the weak duality statement in Eq. \eqref{eq:weakduality} by taking $\lambda = \Bar{\lambda}_{T} = \frac{1}{T}\sum_{t\in[T]}\lambda_{t}$ and $\mu = \Bar{\mu}_{T} = \frac{1}{T}\sum_{t\in[T]}\mu_{t}$; (iii)
follows from the definition of the Lagrangian function in Eq. \eqref{eq:Lagrangian} where we have 
 $\mathcal{L}_{j}(\rho_{j}, \bm{c}; \bm{z}_{j}) =  (1+\lambda)V_{j}(\rho_{j};\bm{z}_{j}) - (\lambda \gamma  + \mu) \rho_{j}$, and $\mathcal{L}_{j}(\rho_{j}, \bm{c}) = \mathbb{E}\left[  \mathcal{L}_{j}(\rho_{j}, \bm{c}; \bm{z}_{j})\right]$; 
in (iv)  we used the definition that $\rho_{j}^{*}(t) = \arg\max_{\rho_{j}\geq 0} \mathcal{L}_{j}(\rho_{j}, \bm{c}_{t})$ to be the optimal budget that maximizes the Lagrangian w.r.t. the dual variables $\bm{c}_{t}=(\lambda_{t},\mu_{t})$, and the definition $g_{1,t} = \sum_{j\in[M]}V_{j}(\rho_{j,t};\bm{z}_{t}) - \gamma \rho_{j,t}$ and $g_{2,t} = \rho -\sum_{j\in[M]}\rho_{j,t}$ as defined in Algorithm \ref{alg:omducb}.
\halmos


\subsection{Proof for Lemma \ref{lem:boundcs}}
\label{pf:lem:boundcs}
Here, we would like to show that   $\sum_{t>K} \mathbb{E}\left[\lambda_{t}g_{1,t} + \mu_{t}g_{2,t}  \right] ~\leq~ 
 \mathcal{O}\left( \eta  T + \frac{1}{\eta}\right)$, where we recall that 
 $g_{1,t} = \sum_{j\in [M]}\left(V_{j}(\rho_{j,t};\bm{z}_{t}) - \gamma \rho_{j,t}\right)$, $g_{2,t} = \rho - \sum_{j\in [M]}\rho_{j,t}$ and $\eta > 0$ the step size defined in Algorithm \ref{alg:omducb}. 

Now from lemma \ref{lem:omdtelescope}, we have for any $\lambda\geq 0$ and $\mu\geq 0$,
\begin{align*}
     &\left(\lambda_{t} - \lambda\right) g_{1,t} ~\leq~ 
   \frac{\eta M^{2}(\Bar{V}+\gamma\rho)^{2}}{2}  +  \frac{1}{2\eta}\left((\lambda- \lambda_{t})^{2} - (\lambda- \lambda_{t+1})^{2}\right)\\
   &\left(\mu_{t} - \mu\right) g_{2,t} ~\leq~
   \frac{\eta M^{2}\rho^{2}}{2}  +  \frac{1}{2\eta}\left((\mu- \mu_{t})^{2} - (\mu- \mu_{t+1})^{2}\right)
\end{align*}
Telescoping the above from $t = K+1$ to $t=T$, we get
\begin{align*}
\begin{aligned}
    & \sum_{t>K} \left(\lambda_{t} - \lambda\right) g_{1,t} \leq  \frac{\eta M^{2}(\Bar{V}+\gamma\rho)^{2}}{2} \cdot T +  \frac{1}{2\eta}(\lambda- \lambda_{1})^{2}\\
    & \sum_{t>K} \left(\mu_{t} - \mu\right) g_{2,t} \leq  \frac{\eta M^{2}\rho^{2}}{2} \cdot T +  \frac{1}{2\eta}(\mu- \mu_{1})^{2}\,.
  \end{aligned}  
\end{align*}
By taking $\lambda = \mu = 0$ and recalling $\lambda_{1} = \mu_{1} = 0$, we get the desired bound in the statement of the lemma. 
\halmos



\subsection{Proof of Lemma \ref{lem:Vstruct}}
\label{pf:lem:Vstruct}
We first show for any realization $\bm{z} = (\bm{z}_{j})_{j\in [M]} = (\bm{v}_{j},\bm{d}_{j})_{j\in [M]} $, the conversion function $V_{j}(\rho_{j};\bm{z}_{j})$
is piecewise linear, strictly inreasing, and concave for any $j \in [M]$. 

Fix any channel $j$ which consists of $m_{j}$ parallel auctions, and recall that we assumed the orderding
$\frac{v_{j,1}}{d_{j,1}} > \frac{v_{j,2}}{d_{j,2}} > \dots  > \frac{v_{j,m_{j}}}{d_{j,m_{j}}}$ for any realization $\bm{z}_{j}$. Then, with the option where the per-channel ROI is set to 0 (i.e. omitted) $V_{j}(\rho_{j};\bm{z}_{j})$ is exactly the LP relaxation of a 0-1 knapsack, whose optimal solution  $\bm{x}_{j}^{*}(\rho_{j};\bm{z}_{j})$ is well known to be unique \cite{dantzig1957discrete}, and takes the form for any auction index $n\in [m_{j}]$:
\begin{align}
\label{eq:knapsack}
    x_{j,n}^{*}(\rho_{j};\bm{z}_{j}) = \begin{cases}
    1 & \text{if } \sum_{n' \in [n]} d_{j,n'} \leq \rho_{j}\\
    \frac{\rho_{j} - \sum_{n' \in [n-1]} d_{j,n'}}{d_{j,n}} & \text{if } \sum_{n' \in [n]} d_{j,n'} > \rho_{j}\\
    0 & \text{otherwise}
    \end{cases}
\end{align}
where we denote $d_{j,0} =0$. With this form, we can see that when  $\rho_{j}$  lies in between  $\sum_{n' \in [n-1]} d_{j,n'}$  and $\sum_{n' \in [n]} d_{j,n'}$, the conversion functions is
\begin{align*}
     V_{j}(\rho_{j};\bm{z}_{j}) = \bm{v}_{j}^{\top} \bm{x}_{j}^{*}(\rho_{j};\bm{z}_{j}) = \sum_{n'\in[m_{j}]} v_{j,n'}\cdot x_{j,n'}^{*}(\rho_{j};\bm{z}_{j}) =\sum_{n'\in [n-1]}v_{j,n'}\cdot 1 + v_{j,n}\cdot   \frac{\rho_{j} - \sum_{n' \in [n-1]} d_{j,n'}}{d_{j,n}} = \frac{v_{j,n}}{d_{j,n}}\rho_{j} + b_{j,n}
\end{align*}
where we denote $d_{j,0} =v_{j,0}=0$ and also $b_{j,n} = \sum_{n' \in [n-1]} v_{j,n'} - \frac{v_{j,n}}{d_{j,n}}\cdot \left(\sum_{n' \in [n-1]} d_{j,n'}\right)$. Hence, by using indicators to specify where $\rho_{j}$ lies, we have

\begin{align}
\label{eq:pieceline}
   V_{j}(\rho_{j};\bm{z}_{j}) = \bm{v}_{j}^{\top} \bm{x}_{j}^{*}(\rho_{j};\bm{z}_{j})=  \sum_{n \in [m_{j}]}\left(\frac{v_{j,n}}{d_{j,n}}\rho_{j}+ b_{j,n}\right)\mathbb{I}\left\{d_{j,0}+ \dots +d_{j,n-1}\leq \rho_{j}\leq d_{j,0}+ \dots +d_{j,n} \right\}\,,
   \end{align}
Further, it is easy to check that any two line segments, say $[X_{n-1},X_{n}]$ and $[X_{n},X_{n+1}]$ where we write $X_{n} =  d_{j,0}+ \dots +d_{j,n} $, intersect at $\rho_{j} =X_{n}$, because $\frac{v_{j,n}}{d_{j,n}}\rho_{j}+ b_{j,n} = \frac{v_{j,n+1}}{d_{j,n+1}}\rho_{j}+ b_{j,n+1}$
at $\rho_{j} =X_{n}$. Hence, from Eq. \eqref{eq:pieceline} we can conclude $ V_{j}(\rho_{j};\bm{z}_{j})$ is continuous, which further implies it is piece-wise linear and strictly increasing. Further, the ordering
$\frac{v_{j,1}}{d_{j,1}} > \frac{v_{j,2}}{d_{j,2}} > \dots  > \frac{v_{j,m_{j}}}{d_{j,m_{j}}}$ implies that the slopes on each segment $[X_{n},X_{n+1}]$ decreases as $n$ increases, which implies  $ V_{j}(\rho_{j};\bm{z}_{j})$  is concave.

Since $V_{j}(\rho_{j})= \mathbb{E}\left[V_{j}(\rho_{j};\bm{z}_{j})\right]$, where the expectation is taken w.r.t. randomness in $\bm{z}_{j}$, and since the $\bm{z}_{j}$ is sampled from some discrete distribution $\bm{p}_{j}$ on finite support $F_{j}$, $V_{j}(\rho_{j})$ is simply a weighted average over all $\left(V_{j}(\rho_{j};\bm{z}_{j})\right)_{\bm{z}_{j} \in F_{j}}$ with weights in $\bm{p}_{j}$, so $V_{j}(\rho_{j})$ is also continuous, piece-wise linear, strictly increasing, and concave, and thus can be written as in Lemma \ref{lem:Vstruct}:
\begin{align*}
        V_{j}(\rho_{j}) = \sum_{n\in [S_{j}]} \left(s_{j,n} \rho_{j} + b_{j,n}\right)\mathbb{I}\{r_{j,n-1}\leq \rho_{j} \leq r_{j,n}\}\,,
\end{align*}
where the parameters $S_{j} \in \mathbb{N}$ and $\{(s_{j,n},b_{j,n},r_{j,n})\}_{n\in [S_{j}]}$ only depend on the support $F_{j}$ and distribution $\bm{p}_{j}$ from which value-to-cost pairs are sampled.  These parameters satisfy
$s_{j,1} > s_{j,2} > \dots > s_{j,S_{j}} > 0$ and $0=r_{j,0} < r_{j,1} < r_{j,2} < \dots < r_{j,S_{j}} =\rho$, as well as $b_{j,n} > 0$ s.t. $s_{j,n} r_{j,n} + b_{j,n} = s_{j,n+1} r_{j,n} + b_{j,n+1}$ for all $n \in [S_{j}-1]$, implying $V_{j}(\rho_{j}) $ is continuous  in $\rho_j$.

Finally, according to the definition of $\mathcal{L}_{j}(\rho_{j},\bm{c})= \mathbb{E}\left[\mathcal{L}_{j}(\rho_{j},\bm{c};\bm{z}_{j})\right]$ and $  \mathcal{L}_{j}(\rho_{j}, \bm{c}; \bm{z}_{j}) =  (1+\lambda)V_{j}(\rho_{j};\bm{z}_{j}) - (\lambda \gamma  + \mu) \rho_{j} $
as defined in Eq. \eqref{eq:Lagrangian}, we have
\begin{align}
\label{eq:Elagr}
    \mathcal{L}_{j}(\rho_{j},\bm{c}) = (1+\lambda)V_{j}(\rho_{j}) - (\lambda \gamma  + \mu) \rho_{j}\,,
\end{align}
which implies $ \mathcal{L}_{j}(\rho_{j},\bm{c})$ is continuous, piece-wise linear, and concave in $\rho_j$ because $V_{j}(\rho_{j})$ is continuous, piece-wise linear, and concave as shown above. Combining Eq. \eqref{eq:Elagr} and the representation of $V_{j}(\rho_{j})$ in Lemma \eqref{lem:Vstruct}, we have
\begin{align}
        \mathcal{L}_{j}(\rho_{j}, \bm{c}) =  \sum_{n\in [S_{j}]} \left( \sigma_{j,n}(\bm{c}) \rho_{j} + (1+\lambda)b_{j,n}\right)\mathbb{I}\{r_{j,n-1}\leq \rho_{j} \leq r_{j,n}\}\,.
\end{align}
where the slope $\sigma_{j,n}(\bm{c}) = (1+\lambda) s_{j,n} -(\mu + \gamma \lambda)$ decreases in $n$. Thus at the point $r_{j,n^{*}} = \max\{r_{j,n}:n=0,1\dots, S_{j}~,\sigma_{j,n}(\bm{c}) \geq 0\}$ in which the slope to the right turns negative for the first time, $ \mathcal{L}_{j}(\rho_{j}, \bm{c}) $ takes its maximum value
$\max_{\rho_{j}\geq 0} \mathcal{L}_{j}(\rho_{j}, \bm{c})$. This is because to the left of  $r_{j,n^{*}}$, namely the region
$[0,r_{j,n^{*}}]$, $\mathcal{L}_{j}(\rho_{j}, \bm{c}) $ strictly increases as the slopes are positive; and to the right of  $r_{j,n^{*}}$, namely the region $[r_{j,n^{*}},\rho]$, $\mathcal{L}_{j}(\rho_{j}, \bm{c}) $ strictly decreases. 
\halmos 






\subsection{Proof for Lemma \ref{lem:dualvarub}}
\label{pf:lem:dualvarub}


We first present some definitions for convenience: denote $\bm{c} = (\lambda, \mu)$, $\bm{c}_{t} = (\lambda_{t}, \mu_{t})$, and $\bm{z}_{t} = (\bm{v}_{t},\bm{d}_{t})$. For any realization $\bm{z} = (\bm{v},\bm{d}) = (\bm{v}_{j},\bm{d}_{j})_{j\in[M]} \in F_{1}\times \dots F_{M}$ of values and costs across channels, define the dual function 
\begin{align}
\label{eq:dualfunc}
\begin{aligned}
    \mathcal{L}_{d}(\bm{c};\bm{z}) ~=~& \max_{\bm{\rho}\in [0,\rho]^{M}} \sum_{j\in[M]}V_{j}(\rho_{j};\bm{z}) + 
\lambda \sum_{j\in [M]}\left(V_{j}(\rho_{j};\bm{z}) - \gamma \rho_{j}\right) + \mu\left(\rho - \sum_{j\in [M]}\rho_{j}\right)\\
~=~& \mu \rho + \max_{(\rho_{j})_{j\in[M]}\in [0,\rho]^{M}}\sum_{j\in [M]}\mathcal{L}_{j}(\rho_{j},\bm{c};\bm{z}_{j})\,.
\end{aligned}
\end{align} 
From standard convex analysis we know that $ \mathcal{L}_{d}$ is convex in $(\lambda,\mu)$, which implies for any $\bm{c}' = (\lambda',\mu')$
\begin{align}
\label{eq:convex}
    (\bm{c}' -\bm{c})^{\top} \nabla  \mathcal{L}_{d}(\bm{c};\bm{z}) \leq \mathcal{L}_{d}(\bm{c}';\bm{z})-\mathcal{L}_{d}(\bm{c};\bm{z})\,.
\end{align}


We prove the lemma by induction that 
\begin{align}
   \norm{(\lambda_{t},\mu_{t})} =   \norm{\bm{c}_{t}}\leq C_{F} := 1 + \frac{\max_{\bm{z}\in F_{1}\times \dots F_{M}}\bm{e}^{\top}\bm{v} + 1}{\min_{\bm{z}\in F_{1}\times \dots F_{M}}\left\{\sum_{j\in [M]}\left(V_{j}(\rho_{(\bm{z}),j};\bm{z}) - \gamma \rho_{j}\right), \rho - \sum_{j\in [M]}\rho_{(\bm{z}),j} \right\}}\,. 
\end{align}
Here, we utilized the existence of some $(\rho_{(\bm{z}),j})_{j\in [M]}\in [0,\rho]^{M}$ for any  $\bm{z}$ according  to  Proposition \ref{prop:slater} which states at per-channel budget profile $(\rho_{(\bm{z}),j})_{j\in [M]}$ Slater's condition holds, i.e. $\sum_{j\in [M]}\left(V_{j}(\rho_{(\bm{z}),j};\bm{z}) - \gamma \rho_{j}\right) > 0$ and 
$\rho - \sum_{j\in [M]}\rho_{(\bm{z}),j}  > 0$. Slater's condition also implies $C_{F}> 1$. 

The base case for $t = 1$ is satisfied trivially since we take $\lambda_{1}=\gamma_{1} = 0$ and thus 
$\norm{\bm{c}_{1}}= 0 < 1 < C_{F}$. Now assume $\norm{\bm{c}_{t}}\leq C_{F}$ for some $t\geq 1$, and we will show $\norm{\bm{c}_{t+1}}\leq C_{F}$ by considering two different case:


\textbf{Case (A):}  $\mathcal{L}_{d}(\bm{c}_{t};\bm{z}_{t}) > \mathcal{L}_{d}(\bm{0};\bm{z}_{t}) + \eta^{2}M^{2}\left((\Bar{V}+\gamma\rho)^{2} + \rho^{2}\right)$. Define $\bm{g}_{t} = (g_{1,t},g_{2,t})$ where we recall $g_{1,t} = \sum_{j\in [M]}\left(V_{j,t}(\rho_{j,t}) - \gamma \rho_{j,t}\right)$ and $g_{2,t} = \rho - \sum_{j\in [M]}\rho_{j,t}$. 
From Eq. \eqref{eq:CSbound0} in the statement of Lemma \ref{lem:omdtelescope}, we have for any $\lambda\geq 0$ and $\mu\geq 0$
\begin{align*}
\begin{aligned}
    & (\lambda- \lambda_{t+1})^{2} ~\leq~
  (\lambda- \lambda_{t})^{2} + 2\eta\left(\lambda-\lambda_{t}\right) g_{1,t} + \eta^{2} M^{2}(\Bar{V}+\gamma\rho)^{2}\\
  & (\mu- \mu_{t+1})^{2} ~\leq~
  (\mu- \mu_{t})^{2} + 2\eta\left(\mu-\mu_{t}\right) g_{1,t} + \eta^{2} M^{2}\Bar{\rho}^{2}\,.
  \end{aligned}  
\end{align*}
By summing the two inequalities and taking $\lambda = \mu = 0$ we get
\begin{align}
\label{eq:dualvarubtelescope}
\begin{aligned}
\norm{\bm{c}_{t+1}}^{2} ~\leq~& \norm{\bm{c}_{t}}^{2} +  2\eta\left(\bm{c}_{t} \right)^{\top} \bm{g}_{t}  +  \eta^{2}M^{2}\left((\Bar{V}+\gamma\rho)^{2} + \rho^{2}\right)\\
~\overset{(i)}{\leq}~ & \norm{ \bm{c}_{t}}^{2} +  \eta\left(\mathcal{L}_{d}(\bm{0};\bm{z}_{t})-\mathcal{L}_{d}(\bm{c}_{t};\bm{z}_{t})\right)+  \eta^{2}M^{2}\left((\Bar{V}+\gamma\rho)^{2} + \rho^{2}\right)
\,,
\end{aligned}
\end{align}
where in (i) we applied Eq. \eqref{eq:convex} with $\bm{z} =\bm{z}_{t}$ and used the fact that $\bm{g}_{t} = (g_{1,t},g_{2,t})= \nabla \mathcal{L}_{d}(\bm{c}_{t};\bm{z}_{t})$.

Then from Eq. \eqref{eq:dualvarubtelescope} and the assumption $\mathcal{L}_{d}(\bm{c}_{t};\bm{z}_{t}) > \mathcal{L}_{d}(\bm{0};\bm{z}_{t}) + \eta^{2}M^{2}\left((\Bar{V}+\gamma\rho)^{2} + \rho^{2}\right)$  we have
\begin{align*}
    \max\{\lambda_{t},\mu_{t}\}\leq \norm{\bm{c}_{t+1}} < \norm{\bm{c}_{t}}\leq C_{F}\,,
\end{align*}
where the final inequality follows from the induction hypothesis.


\textbf{Case (B):}   $\mathcal{L}_{d}(\bm{c}_{t};\bm{z}_{t}) \leq \mathcal{L}_{d}(\bm{0};\bm{z}_{t}) + \eta^{2}M^{2}\left((\Bar{V}+\gamma\rho)^{2} + \rho^{2}\right)$. Under this case, we  have 
\begin{align}
\label{eq:bounddualvar0}
\begin{aligned}
 \mathcal{L}_{d}(\bm{0};\bm{z}_{t}) + \eta M^{2}\Bar{V}^{2} 
 ~\geq~ &\mathcal{L}_{d}(\bm{c}_{t};\bm{z}_{t}) \\
 ~\overset{(i)}{=}~& 
\mu_{t} \rho + \max_{\bm{\rho}\in [0,\rho]^{M}}\sum_{j\in [M]}\mathcal{L}_{j}(\rho,\bm{c}_{t};\bm{z}_{t})\\
~\overset{(ii)}{\geq}~& 
\mu_{t} \rho + \mathcal{L}(\bm{\rho}_{(\bm{z}_{t})},\bm{c}_{t};\bm{z}_{t})\\
~=~& \sum_{j\in[M]}V_{j}(\rho_{(\bm{z}_{t}),j};\bm{z}_{t}) + 
\lambda_{t} \sum_{j\in [M]}\left(V_{j}(\rho_{(\bm{z}_{t}),j};\bm{z}_{t}) - \gamma \rho_{j}\right) + \mu_{t}\left(\rho - \sum_{j\in [M]}\rho_{(\bm{z}_{t}),j}\right)\\
~\geq~& \lambda_{t} \sum_{j\in [M]}\left(V_{j}(\rho_{(\bm{z}_{t}),j};\bm{z}_{t}) - \gamma \rho_{j}\right) + \mu_{t}\left(\rho - \sum_{j\in [M]}\rho_{(\bm{z}_{t}),j}\right)\\
~\overset{(iii)}{\geq}~& \left(\lambda_{t} +\mu_{t} \right)
\cdot \min_{\bm{z}\in F_{1}\times \dots F_{M}}\left\{\sum_{j\in [M]}\left(V_{j}(\rho_{(\bm{z}),j};\bm{z}) - \gamma \rho_{j}\right), \rho - \sum_{j\in [M]}\rho_{(\bm{z}),j} \right\}\,.
\end{aligned}
\end{align}
Here, (i) follows from the definition  of the dual function in Eq. \eqref{eq:dualfunc};  in (ii) we recall the definition of $\bm{\rho}_{(\bm{z})}\in [0,\rho]^{M}$ that satisfies the Slater's condition in Proposition \ref{prop:slater}; in (iii), we used the fact that $\lambda_{t},\mu_{t}\geq 0$ and also under Slater's condition we have for any $\bm{z}$, $\sum_{j\in [M]}\left(V_{j}(\rho_{(\bm{z}),j};\bm{z}) - \gamma \rho_{j}\right) > 0$ and 
$\rho - \sum_{j\in [M]}\rho_{(\bm{z}),j}  > 0$. 


 On the other hand, we have $\eta^{2}M^{2}\left((\Bar{V}+\gamma\rho)^{2} + \rho^{2}\right)< 1$ since in the statement of the lemma we assumed $\eta < \frac{1}{M \sqrt{(\Bar{V}+\gamma\rho)^{2} + \rho^{2}}}$. Also, $\mathcal{L}_{d}(\bm{0};\bm{z}_{t}) = \max_{\bm{\rho}\in [0,\rho]^{M}}\sum_{j\in [M]}V_{j}(\rho_{j};\bm{z}_{t}) =\max_{\bm{z}\in F_{1}\times \dots F_{M}}\bm{e}^{\top}\bm{v}$. Hence combining this with Eq. \eqref{eq:bounddualvar0} we get 
\begin{align}
    \lambda_{t} +\mu_{t} < \frac{\max_{\bm{z}\in F_{1}\times \dots F_{M}}\bm{e}^{\top}\bm{v} + 1}{\min_{\bm{z}\in F_{1}\times \dots F_{M}}\left\{\sum_{j\in [M]}\left(V_{j}(\rho_{(\bm{z}),j};\bm{z}) - \gamma \rho_{j}\right), \rho - \sum_{j\in [M]}\rho_{(\bm{z}),j} \right\}} := C_{F} - 1\,,
\end{align}
which further implies $ \lambda_{t},\mu_{t}\leq C_{F} - 1$. Hence, we can finally conclude
\begin{align}
    \lambda_{t+1} = \left(\lambda_{t} - \eta g_{1,t} \right)_{+} \leq \lambda_{t} + \eta |g_{1,t}|\leq C_{F} - 1 + \eta M(\bar{V} + \gamma\rho)< C_{F} 
\end{align}
where in the final inequality we used the fact that $\eta M(\bar{V} + \gamma\rho) < 1$ since in the statement of the lemma we assumed $\eta < \frac{1}{M \sqrt{(\Bar{V}+\gamma\rho)^{2} + \rho^{2}}}$. A similar bound holds for $\mu_{t+1}$.
\halmos





\subsection{Proof for Lemma \ref{lem:ucbregret}}
\label{pf:lem:ucbregret}


 We fix some channel $j\in [M]$ and omit the subscript $j$ when the context is clear. Also, we first introduce some definitions that will be used throughout our proof. Fix some positive constant $\underline{\sigma}>0$ whose value we choose later, and 
recall $a_{k}$ denotes the $k$th arm in the discretized budget set $\mathcal{A}(\delta)$ as we defined in Eq. \eqref{eq:budgetdisc}. Then we define the following
\begin{align}
\label{eq:UCBdefs}
\begin{aligned}
    &  \Delta_{k}(\bm{c}) =  \max_{\rho_{j}\in [0,\rho]}\mathcal{L}_{j}(\rho_{j}, \bm{c}) -  \mathcal{L}_{j}(a_{k}, \bm{c}) \\
    & \mathcal{C}_{n} = \Big\{\bm{c} \in \{\bm{c}_{t}\}_{t\in [T]}: r_{j,n} = \arg\max_{\rho_{j}\geq 0} \mathcal{L}_{j}(\rho_{j},\bm{c})\Big\} \text{ for } n = 0 \dots S_{j} \\
    & \mathcal{C}(\underline{\sigma}) = \left\{\bm{c} \in \{\bm{c}_{t}\}_{t\in [T]}: \sigma_{j}^{-}(\bm{c}) > \underline{\sigma},~ |\sigma_{j}^{+}(\bm{c})| > \underline{\sigma}\right\} \text{ for } n = 0, \dots,  S_{j} \\
    & m_{k}(\bm{c}) = \frac{8\log(T)}{\Delta_{k}^{2}(\bm{c})} \text{ for } \forall (k,\bm{c}) \text{ s.t. }\Delta_{k}(\bm{c}) > 0 \,.
    \end{aligned}
\end{align}
Here, the ``adjacent slopes'' $\sigma_{j}^{-}(\bm{c})$ and $\sigma_{j}^{+}(\bm{c})$, which are defined in Eq.\eqref{eq:adj}, 
 represent the slopes that are adjacent to the optimal budget $\arg\max_{\rho_{j}\in[0,\rho]}\mathcal{L}_{j}(\rho_{j}, \bm{c}) $ for any context $\bm{c} = (\lambda,\mu) $. Further, $S_{j}$ and $\{r_{j,n}\}_{j\in [S_{j}]}$ are defined in Lemma \ref{lem:Vstruct}.  Here we state in words the meanings of $\Delta_{k}(\bm{c})$, $\mathcal{C}(\underline{\sigma}) $ and $\mathcal{C}_{n} $, respectively.
\begin{itemize}
    \item $\Delta_{k}(\bm{c})$ denotes the loss in contextual bandit rewards when pulling arm $a_{k}$ under context $\bm{c}$.  
    \item $\mathcal{C}_{n} $ is the set including all context $\bm{c}_{t}$ under which the optimal per-channel budget $\arg\max_{\rho_{j}\geq 0} \mathcal{L}_{j}(\rho_{j},\bm{c}_{t})$ is taken at the $n$th ``turning point'' $r_{j,n}$ (see Lemma \ref{lem:Vstruct}).
     \item $\mathcal{C}(\underline{\sigma}) $ is the set of all contexts,  in which the adjacent slopes to the optimal point w.r.t. the context $\bm{c}$, namely $\arg\max_{\rho_{j}\geq 0} \mathcal{L}_{j}(\rho_{j},\bm{c})$, have magnitude greater than $\underline{\sigma}$, or in other words, the adjacent slopes are steep.
\end{itemize}
On a related note, for any context $\bm{c}$, we define the following ``adjacent regions''  that sandwich  the optimal budget w.r.t.$\bm{c}$
\begin{align}
\label{eq:adjregion}
    \mathcal{U}_{j}^{-}(\bm{c}) = [r_{j,n-1},r_{j,n}]    ~\text{ and }~ \mathcal{U}_{j}^{+}(\bm{c}) = [r_{j,n},r_{j,n+1}] \quad \text{ if }\quad \bm{c} \in \mathcal{C}_{n}\,.
\end{align}
In other words, if $\bm{c}\in \mathcal{C}_{n}$, per the definition of $\mathcal{C}_{n}$ above, $\arg\max_{\rho_{j}\in[0,\rho]} \mathcal{L}_{j}(\rho_{j},\bm{c})$ is located at the $n$th ``turning point'' $r_{j,n}$, then $\mathcal{U}_{j}^{-}(\bm{c})$ and $\mathcal{U}_{j}^{-}(\bm{c})$ are respectively  the left and right regions surrounding $r_{j,n}$.


With the above definitions, we demonstrate how to bound the UCB-error. Define $N_{k,t} = \sum_{\tau\leq t-1}\mathbb{I}\{\rho_{j,\tau} = a_{k}\}$ to be the number of times arm $k$ is pulled up to time $t$, then we can decompose the UCB error as followed
\begin{align}
\begin{aligned}
    &  \sum_{ t > K}\mathcal{L}_{j}(\rho_{j}^{*}(t), \bm{c}_{t}) - \mathcal{L}_{j}(\rho_{j,t}, \bm{c}_{t}) = X_{1}+X_{2} + X_{3} \quad \text{ where }\\
    & X_{1} = \sum_{t > K:\bm{c}_{t} \notin \mathcal{C}(\underline{\sigma})}  ~ \sum_{k\in [K]} \Delta_{k}(\bm{c}_{t}) \mathbb{I}\{\rho_{j,t} = a_{k}, N_{k,t} \leq m_{k}(\bm{c}_{t})\}\\
    & X_{2} = \sum_{t > K:\bm{c}_{t} \in \mathcal{C}(\underline{\sigma})}   ~ \sum_{k\in [K]}\Delta_{k}(\bm{c}_{t}) \mathbb{I}\{\rho_{j,t} = a_{k}, N_{k,t} \leq m_{k}(\bm{c}_{t})\}\\
    & X_{3} = \sum_{k\in [K]}~ \sum_{t > K} \Delta_{k}(\bm{c}_{t}) \mathbb{I}\{\rho_{j,t} = a_{k}, N_{k,t} > m_{k}(\bm{c}_{t})\}\,.
    \end{aligned}
\end{align}
In Section \ref{subsubsec:X1}, we show that $X_{1} \leq \widetilde{\mathcal{O}}(\delta T + \underline{\sigma}T + \frac{1}{\delta})$; in Section \ref{subsubsec:X2}  we show that $X_{2} \leq \widetilde{\mathcal{O}}(\delta T + \frac{1}{\delta \underline{\sigma}})$;
 in Section \ref{subsubsec:X3}  we show that $X_{3} \leq \widetilde{\mathcal{O}}(\frac{1}{\delta T})$. 

 \begin{remark}
 \label{rmk:bestarm}
 In the following sections  \ref{subsubsec:X1}, \ref{subsubsec:X2} and \ref{subsubsec:X3} where we bound $X_{1}$, $X_{2}$, and $X_{3}$, respectively,  we assume the optimal per-channel $\rho_{j}^{*}(t) = \arg\max_{\rho_{j}\in [0,\rho]}\mathcal{L}_{j}(\rho_{j}, \bm{c}_{t})$ lies in the arm set $\mathcal{A}(\delta)$ for all $t$. This is because otherwise, we can consider the following decomposition of the UCB error in period $t$ as followed:
  \begin{align*}
     \mathcal{L}_{j}(\rho_{j}^{*}(t), \bm{c}_{t}) - \mathcal{L}_{j}(\rho_{j,t}, \bm{c}_{t}) = \mathcal{L}_{j}(\rho_{j}^{*}(t), \bm{c}_{t}) - \mathcal{L}_{j}(a_{t}^{*}, \bm{c}_{t}) + \mathcal{L}_{j}(a_{t}^{*}, \bm{c}_{t}) - \mathcal{L}_{j}(\rho_{j,t}, \bm{c}_{t}) \quad \text{where }a_{t}^{*}=\arg\max_{a_{k}\in \mathcal{A}(\delta)}\mathcal{L}_{j}(a_{k}, \bm{c}_{t})
  \end{align*}
 The first term will yield an error in the order of $\mathcal{O}(\delta)$ due to the Lagrangian function being unimodal, piecewise linear liner, which implies 
$|a_{t}^{*} - \rho_{j}^{*}(t)|\leq \delta$ so that $\mathcal{L}_{j}(\rho_{j}^{*}(t), \bm{c}_{t}) - \mathcal{L}_{j}(a_{t}^{*}, \bm{c}_{t}) = \mathcal{O}(\delta)$. Hence, this ``discretization error'' will accumulate to a magnitude of $\mathcal{O}(\delta T)$ over $T$ periods, which leads to an additional error that is already accounted for in the statement of the lemma.
 \end{remark}
 
 
\subsubsection{Bounding $X_{1}$.}
\label{subsubsec:X1} 
 Our strategy to bound $X_{1}$ consists of 4 steps, namely bounding the 
loss of arm $a_{k}$ at each context $\bm{c}\notin \mathcal{C}(\underline{\sigma})$ when $a_{k} \in \mathcal{U}_{j}^{-}(\bm{c})$ lies on the left adjacent region of the optimal budget; $a_{k} < \min~ \mathcal{U}_{j}^{-}(\bm{c})$
lies to the left of the left adjacent region; $a_{k} \in \mathcal{U}_{j}^{+}(\bm{c})$ lies on the right adjacent region of the optimal budget; and $a_{k} > \max~ \mathcal{U}_{j}^{+}(\bm{c})$
lies to the right of the right adjacent region. Here we recall the adjacent regions are defined in Eq.\eqref{eq:adjregion}.


\textbf{Step 1: $a_{k} \in \mathcal{U}_{j}^{-}(\bm{c}_{t})$. } For arm $k$ such that $a_{k} \in \mathcal{U}_{j}^{-}(\bm{c}_{t})$, recall Lemma \ref{lem:Vstruct} that $\mathcal{L}_{j}(a,\bm{c}_{t})$ is linear in $a$ for $a \in \mathcal{U}_{j}^{-}(\bm{c}_{t})$, so $\Delta_{k}(\bm{c}_{t}) = \sigma_{j}^{-}(\bm{c}_{t})\cdot (\rho_{j}^{*}(t) - a_{k}) \leq  \underline{\sigma} \rho$
 where we used the condition
that $\bm{c}_{t}\notin \mathcal{C}(\underline{\sigma})$ so the adjacent slopes have magnitude at most $\underline{\sigma}$, and $\rho_{j}^{*}(t)\leq \rho$.
Thus, summing over all such $k$  we get 
\begin{align}
\label{eq:X1near}
\begin{aligned}
   &  \sum_{t > K:\bm{c}_{t} \notin \mathcal{C}(\underline{\sigma})}  ~\sum_{k\in[K]:a_{k} \in \mathcal{U}_{j}^{-}(\bm{c}_{t})}  \Delta_{k}(\bm{c}_{t}) \mathbb{I}\{\rho_{j,t} = a_{k}, N_{k,t} \leq m_{k}(\bm{c}_{t})\} \\
   ~\leq~& \sum_{t > K:\bm{c}_{t} \notin \mathcal{C}(\underline{\sigma})}   ~\sum_{k\in[K]:a_{k} \in \mathcal{U}_{j}^{-}(\bm{c}_{t})}  \underline{\sigma} \rho \cdot  \mathbb{I}\{\rho_{j,t} = a_{k}, N_{k,t} \leq m_{k}(\bm{c}_{t})\}  ~\leq~ \underline{\sigma} \rho T = \mathcal{O}(\underline{\sigma}T )\,.
   \end{aligned}
\end{align}

\textbf{Step 2: $a_{k} < \min~ \mathcal{U}_{j}^{-}(\bm{c}_{t})$. } For arm $k$ such that $a_{k} < \min ~ \mathcal{U}_{j}^{-}(\bm{c}_{t})$, we further split contexts into groups $\mathcal{C}_{n}$ for $n = 0 \dots S_{j}$ (defined in Eq. \eqref{eq:UCBdefs}) based on whether the corresponding optimal budget w.r.t. the Lagrangian at the context is taken at the $n$th ``turning point'' (see Figure \ref{fig:lagr} of illustration). Then, for each context group $n$ by defining  $k':=\max\{k: a_{k} < r_{j,n-1}\}$ to be the arm closest to and less than $r_{j,n-1}$, we have
\begin{align}
\label{eq:nbadslope}
\begin{aligned}
   & \sum_{t > K:\bm{c}_{t} \in \mathcal{C}_{n}/\mathcal{C}(\underline{\sigma})} ~ \sum_{k\in[K]:a_{k} < \min \mathcal{U}_{j}^{-}(\bm{c}_{t})} ~  \Delta_{k}(\bm{c}_{t}) \mathbb{I}\{\rho_{j,t} = a_{k}, N_{k,t} \leq m_{k}(\bm{c}_{t})\} \\
    ~\overset{(i)}{=}~ & \sum_{t > K:\bm{c}_{t} \in \mathcal{C}_{n}/\mathcal{C}(\underline{\sigma})} ~ \sum_{k\in[K]:a_{k} < r_{j,n-1}} ~  \Delta_{k}(\bm{c}_{t}) \mathbb{I}\{\rho_{j,t} = a_{k}, N_{k,t} \leq m_{k}(\bm{c}_{t})\} \\
    ~=~ &  \sum_{t > K} \sum_{\bm{c} \in \mathcal{C}_{n}/\mathcal{C}(\underline{\sigma})} ~ \sum_{k\in[K]:a_{k} < r_{j,n-1}} \Delta_{k}(\bm{c}) \mathbb{I}\{\bm{c}_{t} = \bm{c}, \rho_{j,t} = a_{k}, N_{k,t} \leq m_{k}(\bm{c})\}\\
    ~\overset{(ii)}{\leq}~ &  \sum_{t > K} \sum_{\bm{c} \in \mathcal{C}_{n}/\mathcal{C}(\underline{\sigma})} ~ \left(\Delta_{k'}(\bm{c}) \mathbb{I}\{\bm{c}_{t}=\bm{c}\} + \sum_{k\in[K]:a_{k} < r_{j,n-1}-\delta} \Delta_{k}(\bm{c}) \mathbb{I}\{\bm{c}_{t} = \bm{c}, \rho_{j,t} = a_{k}, N_{k,t} \leq m_{k}(\bm{c})\}\right)\\
   ~\overset{(iii)}{\leq}~ & \left((1+C_{F})s_{j,n-1} \delta + \rho \underline{\sigma}\right) T + \sum_{k\in[K]:a_{k} < r_{j,n-1}-\delta} ~ \sum_{\bm{c} \in \mathcal{C}_{n}/\mathcal{C}(\underline{\sigma})} \Delta_{k}(\bm{c}) Y_{k}(\bm{c})
\end{aligned}
\end{align}
where in the final equality we defined $ Y_{k}(\bm{c})=  \sum_{t > K} \mathbb{I}\{\bm{c}_{t} = \bm{c}, \rho_{j,t} = a_{k}, N_{k,t} \leq m_{k}(\bm{c})\}$. In (i) we used the fact that the left end of the left adjacent region, i.e. $
\min \mathcal{U}_{j}^{-}(\bm{c}_{t})$ is exactly $r_{j,n-1}$ because for context $\bm{c}_{t} \in \mathcal{C}_{n}$ the optimal budget $\arg\max_{\rho_{j}\in[0,\rho]} \mathcal{L}_{j}(\rho_{j},\bm{c}_{t})$ is at the $n$th turning point; in (ii) we used the definition $k':=\max\{k: a_{k} < r_{j,n-1}\}$  where we recall arms are indexed such that $a_{1} < a_{2}<\dots < a_{K}$. Note that in (ii) we separate out the arm $a_{k'}$ because its distance to
the optimal per-channel may be less than $\delta$ since it is the closest arm, and thus we ensure  all other arms indexed by $k\in[K]:a_{k} < r_{j,n-1}-\delta$, 
 are at least $\delta$ away from the optimal per-channel budget; (iii) follows from the fact that under a context $\bm{c} \in \mathcal{C}_{n}/\mathcal{C}(\underline{\sigma})$,  we have $\arg\max_{\rho_{j}\in [0,\rho]} \mathcal{L}_{j}(\rho_{j}, \bm{c}) = r_{j,n}$ so
\begin{align*}
    \Delta_{k'}(\bm{c}) ~=~& \mathcal{L}_{j}(r_{j,n},\bm{c}) -   \mathcal{L}_{j}(r_{j,n-1},\bm{c}) + \mathcal{L}_{j}(r_{j,n-1},\bm{c}) - \mathcal{L}_{j}(a_{k'},\bm{c})\\
     ~=~&  \sigma_{j}^{-}(\bm{c}) (r_{j,n} - r_{j,n-1}) + \sigma_{j,n-1}(\bm{c})(r_{j,n-1} - a_{k'}) \\
     ~\overset{(iv)}{\leq}~ & \underline{\sigma}\rho + \sigma_{j,n-1}(\bm{c}) \delta\\
       ~\overset{(v)}{\leq}~ & \underline{\sigma}\rho + (1+C_{F})s_{j,n-1}\delta
     \,,
\end{align*}
where in (iv) we used  $\bm{c} \in \mathcal{C}_{n}/\mathcal{C}(\underline{\sigma})$ implies $ \sigma_{j}^{-}(\bm{c}) \leq \underline{\sigma}$,   as well as all $r_{j,n}\leq \rho$ for any $n$ and the fact that $k'$ lies on the line segment between points $r_{j,n-2}$ and $r_{j,n-1}$ since $\delta < \min_{n' \in [S_{j}]} r_{j,n'} - r_{j,n'-1}$; in (v) we recall $\sigma_{j,n-1}(\bm{c}) = (1+\lambda)s_{j,n-1} - (\mu + \gamma\lambda) \leq (1+C_{F})s_{j,n-1}$ where $C_{F}$ is defined in Lemma \ref{lem:dualvarub}.

We now bound $\sum_{\bm{c} \in \mathcal{C}_{n}/\mathcal{C}(\underline{\sigma})} \Delta_{k}(\bm{c}) Y_{k}(\bm{c})$ in Eq. \eqref{eq:nbadslope}.  It is easy to see the following inequality for any sequence of context $\bm{c}_{(1)}, \dots , \bm{c}_{(\ell)} \in \{\bm{c}_{t}\}_{t\in[T]}$ (This is a slight generalization of an inequality result shown in \cite{balseiro2019contextual}):
\begin{align}
\label{eq:numbadarms}
    Y_{k}(\bm{c}_{(1)}) + \dots +  Y_{k}(\bm{c}_{(\ell)}) \leq \max_{\ell'=1\dots \ell} m_{k}(\bm{c}_{(\ell')})\,. 
\end{align}
This is because 
\begin{align*}
   \sum_{\ell'\in [\ell]}  Y_{k}(\bm{c}_{(\ell')}) ~=~ &\sum_{t > K} \sum_{\ell'\in [\ell]}  \mathbb{I}\{\bm{c}_{t} =\bm{c}_{(\ell')}, \rho_{j,t} = a_{k}, N_{k,t} \leq m_{k}(\bm{c}_{(\ell')})\}\\
     ~\leq~& \sum_{t > K}  \sum_{\ell'\in [\ell]}  \mathbb{I}\{\bm{c}_{t}= \bm{c}_{(\ell')}, \rho_{j,t} = a_{k}, N_{k,t} \leq \max_{\ell'=1\dots \ell} m_{k}(\bm{c}_{(\ell')})\}\\
      ~=~& \sum_{t > K} \mathbb{I}\{\bm{c}_{t} \in \{\bm{c}_{(\ell')}\}_{\ell'\in[\ell]}, \rho_{j,t} = a_{k}, N_{k,t} \leq \max_{\ell'=1\dots \ell} m_{k}(\bm{c}_{(\ell')})\}\\
        ~\leq~& \max_{\ell'=1\dots \ell} m_{k}(\bm{c}_{(\ell')})\,.
\end{align*}
For simplicity denote $L = |\mathcal{C}_{n}/\mathcal{C}(\underline{\sigma})|$, and order contexts in $\bm{c} \in \mathcal{C}_{n}/\mathcal{C}(\underline{\sigma})$ as $\{\bm{c}_{(\ell)}\}_{\ell \in [L]}$ s.t. 
$\Delta_{k}(\bm{c}_{(1)}) > \Delta_{k}(\bm{c}_{(2)}) > \dots >\Delta_{k}(\bm{c}_{(L)})$, or equivalently 
$m_{k}(\bm{c}_{(1)}) < m_{k}(\bm{c}_{(2)}) < \dots  < m_{k}(\bm{c}_{(L)})$ according to Eq.\eqref{eq:UCBdefs}. Then multiplying Eq. \eqref{eq:numbadarms} by 
by $\Delta_{k}(\bm{c}_{(\ell)}) - \Delta_{k}(\bm{c}_{(\ell+1)})$ (which is strictly positive due to the ordering of contexts), and summing $\ell = 1 \dots L$  we get 
\begin{align}
\begin{aligned}
\label{eq:slope}
    \sum_{\bm{c} \in \mathcal{C}_{n}/\mathcal{C}(\underline{\sigma})} \Delta_{k}(\bm{c}) Y_{k}(\bm{c}) &= \sum_{\ell \in [L]} \Delta_{k}(\bm{c}_{(\ell)}) Y_{k}(\bm{c}_{(\ell)})
    ~\leq~  \sum_{\ell \in [L]} m_{k}(\bm{c}_{(\ell)}) \left( \Delta_{k}(\bm{c}_{(\ell)}) -  \Delta_{k}(\bm{c}_{(\ell+1)})\right)\\
    ~\overset{(i)}{=}~& 8\log(T)\sum_{\ell \in [L-1]} \frac{ \Delta_{k}(\bm{c}_{(\ell)}) -  \Delta_{k}(\bm{c}_{(\ell+1)})}{\Delta_{k}^{2}(\bm{c}_{(\ell)})}
    ~\overset{(ii)}{\leq}~8\log(T) \int_{\Delta_{k}(\bm{c}_{(L)})}^{\infty}\frac{dz}{z^{2}}\\
     ~=~& \frac{8\log(T)}{\Delta_{k}(\bm{c}_{(L)})} ~\overset{(iii)}{=} \frac{8\log(T)}{\min_{\bm{c} \in \mathcal{C}_{n}/\mathcal{C}(\underline{\sigma})} \Delta_{k}(\bm{c})}\,.
     \end{aligned}
\end{align}
Here  (i) follows from the definition of $m_k(\bm{c})$ in Eq. \eqref{eq:UCBdefs} where $m_{k}(\bm{c}) = \frac{8\log(T)}{\Delta_{k}^{2}(\bm{c})}$;  both (ii) and (iii) follow from the ordering of contexts so that $\Delta_{k}(\bm{c}_{(1)}) > \Delta_{k}(\bm{c}_{(2)}) > \dots >\Delta_{k}(\bm{c}_{(L)})$. 
Note that for any $\bm{c} \in \mathcal{C}_{n}/\mathcal{C}(\underline{\sigma})$ and arm $k$ such that $a_{k} < r_{j,n-1}$, we have
\begin{align}
\label{eq:slope2}
\begin{aligned}
  \Delta_{k}(\bm{c}) ~=~& \mathcal{L}_{j}(r_{j,n},\bm{c}) -   \mathcal{L}_{j}(r_{j,n-1},\bm{c}) + \mathcal{L}_{j}(r_{j,n-1},\bm{c}) - \mathcal{L}_{j}(a_{k},\bm{c})\\
  ~>~&  \mathcal{L}_{j}(r_{j,n-1},\bm{c}) - \mathcal{L}_{j}(a_{k},\bm{c})\\
  ~\overset{(i)}{\geq} ~&  \sigma_{j,n-1}(\bm{c}) (r_{j,n-1} - a_{k})\\
   ~\overset{(ii)}{\geq} ~& \left( \sigma_{j,n-1}(\bm{c}) -  \sigma_{j,n}(\bm{c})\right) (r_{j,n-1} - a_{k})\\
    ~\overset{(iii)}{=} ~& (1+\lambda)\left(s_{j,n-1}-s_{j,n}\right) (r_{j,n-1} - a_{k})\\
     ~> ~& \left(s_{j,n-1}-s_{j,n}\right) (r_{j,n-1} - a_{k})\,,
\end{aligned}
\end{align}
where in (i) we recall the slope $\sigma_{j,n-1}(\bm{c})$ is defined in Lemma \ref{lem:Vstruct} and further (i) follows from concavity of $ \mathcal{L}_{j}(\rho{j},\bm{c})$ in the first argument $\rho_{j}$; in (ii) we used the fact that $ \sigma_{j,n}(\bm{c})\geq 0$ since the optimal budget 
$\arg\max_{\rho_{j}\in [0,\rho]} \mathcal{L}_{j}(\rho_{j}, \bm{c})$ is taken at the $n$th turning point, and is the largest turning point whose left slope is non-negative from  Lemma \ref{lem:Vstruct}; (iii) follows from the definition $\sigma_{j,n'}(\bm{c}) = (1+\lambda)s_{j,n'} - (\mu+\gamma\lambda)$ for any $n'$. 


Finally combining Eqs. \eqref{eq:nbadslope}, \eqref{eq:slope} and \eqref{eq:slope2}, and summing over $n = 1 \dots S_{j}$ we get
\begin{align}
\begin{aligned}
\label{eq:X1secbound}
  & \sum_{t > K:\bm{c}_{t} \notin \mathcal{C}(\underline{\sigma})} ~ \sum_{k\in[K]:a_{k} < \min \mathcal{U}_{j}^{-}(\bm{c}_{t})} ~  \Delta_{k}(\bm{c}_{t}) \mathbb{I}\{\rho_{j,t} = a_{k}, N_{k,t} \leq m_{k}(\bm{c}_{t})\}   \\
 ~=~& \sum_{n\in [S_{j}]} \sum_{t > K:\bm{c}_{t} \in \mathcal{C}_{n}/\mathcal{C}(\underline{\sigma})} ~ \sum_{k\in[K]:a_{k} < \min \mathcal{U}_{j}^{-}(\bm{c}_{t})} ~  \Delta_{k}(\bm{c}_{t}) \mathbb{I}\{\rho_{j,t} = a_{k}, N_{k,t} \leq m_{k}(\bm{c}_{t})\} \\
 ~\leq~& \sum_{n\in [S_{j}]}\left((1+C_{F})s_{j,n-1} \delta + \rho \underline{\sigma}\right) T  + \sum_{n\in [S_{j}]}~\sum_{k\in[K]:a_{k} < r_{j,n-1}-\delta} \frac{8\log(T)}{\left(s_{j,n-1}-s_{j,n}\right) (r_{j,n-1} - a_{k})}\\
 ~\overset{(i)}{\leq}~& \sum_{n\in [S_{j}]}\left((1+C_{F})s_{j,n-1} \delta + \rho \underline{\sigma}\right) T  + \sum_{n\in [S_{j}]}~\sum_{\ell=1}^{K} \frac{8\log(T)}{\left(s_{j,n-1}-s_{j,n}\right) \ell \delta}\\
 ~\leq~& \sum_{n\in [S_{j}]}\left((1+C_{F})s_{j,n-1} \delta + \rho \underline{\sigma}\right) T  + \frac{8\log(T)\log(K)}{\delta} \sum_{n\in [S_{j}]}~\frac{1}{\left(s_{j,n-1}-s_{j,n}\right)}\\
 ~=~& \widetilde{\mathcal{O}}(\delta T + \underline{\sigma}T + \frac{1}{\delta})\,.
 \end{aligned}
\end{align}
Note that (i) follows because for all $a_{k} < r_{j,n-1} - \delta$, the $a_{k}$'s distances from $r_{j,n-1}$ are at least $\delta, 3\delta, 2\delta \dots $.  In the last equation, we hide all logarithmic factors using the notation $\widetilde{\mathcal{O}}$, and note that the constants $C_{F}$, $(s_{j,n})_{n\in S_{j}}$, $S_{j}$ are all absolute constants that depend only on the support $F_{j}$ and corresponding sampling distribution $\bm{p}_{j}$ for value-cost pairs; see definitions of these absolute constants in Lemmas \ref{lem:Vstruct} and \ref{lem:dualvarub}.



\textbf{Step 3 and 4: $a_{k} \in \mathcal{U}_{j}^{+}(\bm{c}_{t})$ or $a_{k} > \max~ \mathcal{U}_{j}^{+}(\bm{c}_{t})$. } The cases where arm $a_k \in \mathcal{U}_{j}^{+}(\bm{c}_{t})$ and $a_{k} > \max \mathcal{U}_{j}^{+}(\bm{c}_{t})$
are symmetric to $a_k \in \mathcal{U}_{j}^{-}(\bm{c}_{t})$ and $a_{k} < \min \mathcal{U}_{j}^{+}(\bm{c}_{t})$, respectively, and we omit from this paper. 

Therefore, combining Eqs. \eqref{eq:X1near} and \eqref{eq:X1secbound} we  can conclude 
\begin{align}
    X_{1} \leq  \widetilde{\mathcal{O}}(\delta T + \underline{\sigma}T + \frac{1}{\delta})\,.
\end{align}



\subsubsection{Bounding $X_{2}$.}
\label{subsubsec:X2}
We first rewrite $X_{2}$ as followed
\begin{align}
\label{eq:X2all}
\begin{aligned}
    X_{2} ~=~& \sum_{t > K:\bm{c}_{t} \in \mathcal{C}(\underline{\sigma})} \sum_{k\in [K]} \Delta_{k}(\bm{c}_{t}) \mathbb{I}\{\rho_{j,t} = a_{k}, N_{k,t} \leq m_{k}(\bm{c}_{t})\} \\
     ~=~& \sum_{t > K}\sum_{n\in[S_{j}]}\sum_{k\in [K]}\sum_{\bm{c} \in \mathcal{C}_{n}\cap \mathcal{C}(\underline{\sigma})}  \Delta_{k}(\bm{c}) \mathbb{I}\{\bm{c}_{t}=\bm{c}, \rho_{j,t} = a_{k}, N_{k,t} \leq m_{k}(\bm{c})\} \\
     ~\overset{(i)}{=}~& 
   \sum_{n\in[S_{j}]}\sum_{k\in [K]}\sum_{\bm{c} \in \mathcal{C}_{n}\cap \mathcal{C}(\underline{\sigma})} \Delta_{k}(\bm{c}) Y_{k}(\bm{c})\\
    ~\overset{(ii)}{=}~&  \sum_{n\in[S_{j}]}~\sum_{\bm{c} \in \mathcal{C}_{n}\cap \mathcal{C}(\underline{\sigma})}~\sum_{k\in \{k_{n}^{-},k_{n}^{+}\}} \Delta_{k}(\bm{c}) Y_{k}(\bm{c})+ \sum_{n\in[S_{j}]}~\sum_{\bm{c} \in \mathcal{C}_{n}\cap \mathcal{C}(\underline{\sigma})}~\sum_{k\in [K]/\{k_{n}^{-},k_{n}^{+}\}}\Delta_{k}(\bm{c}) Y_{k}(\bm{c})\\
    ~\overset{(iii)}{\leq}~&  T \delta (1+C_{F}) \sum_{n\in[S_{j}]} \left(s_{j,n} + s_{j,n+1}\right) + 
   \sum_{n\in[S_{j}]}~\sum_{\bm{c} \in \mathcal{C}_{n}\cap \mathcal{C}(\underline{\sigma})}~\sum_{k\in [K]/\{k_{n}^{-},k_{n}^{+}\}} \Delta_{k}(\bm{c}) Y_{k}(\bm{c})\,.
    \end{aligned}
\end{align}
where in (i) we define 
$Y_{k}(\bm{c}) =  \sum_{t > K}\mathbb{I}\{\bm{c}_{t} = \bm{c}, \rho_{j,t} = a_{k}, N_{k,t} \leq m_{k}(\bm{c})\}$;  in (ii)
we separate out two arms  $k_{n}^{-}$ and $k_{n}^{+}$ defined as followed: recall for context $\bm{c} \in \mathcal{C}_{n}\cap \mathcal{C}(\underline{\sigma})$, the optimal budget $\arg\max_{\rho_{j}\in [0,\rho]} \mathcal{L}_{j}(\rho_{j}, \bm{c}) = r_{j,n}$ is taken at the $n$th turning point per the definition of $\mathcal{C}_{n}$ in Eq. \eqref{eq:UCBdefs}, and thereby we defined $k_{n}^{-}:= \max\{k\in [K]: a_{k} < r_{j,n}\}$ to be the arm closest to and no greater than  $r_{j,n}$, whereas  $k_{n}^{+}:= \min\{k\in [K]: a_{k} > r_{j,n}\}$ to be the arm closest to and no less than  $r_{j,n}$; in (iii), for small enough $\delta < \min_{n'\in[S_{j}]}r_{j,n'} - r_{j,n'-1} $, we know that $k_{n}^{-}$ lies on the line segment between $r_{j,n-1}$ and $r_{j,n}$, so $\Delta_{k_{n}^{-}}(\bm{c}) = \sigma_{j}^{-}(\bm{c})(r_{j,n} - a_{k_{n}^{-}}) \leq \sigma_{j}^{-}(\bm{c}) \delta \leq (1+C_{F})s_{j,n-1} \delta$, where in the final inequality follows from the definition of  $\sigma_{j}^{-}(\bm{c}) = \sigma_{j,n}(\bm{c}) = (1+\lambda)s_{j,n} - (\mu + \gamma\lambda) \leq (1+\lambda)s_{j,n} \leq (1+C_{F})s_{j,n}$ where $C_{F}$ is defined in Eq. \eqref{lem:dualvarub}. A similar bound holds for 
$\Delta_{k_{n}^{+}}(\bm{c})$. 


Then, following the same logic as Eqs. \eqref{eq:numbadarms}, \eqref{eq:slope}, \eqref{eq:slope2} in Section \ref{subsubsec:X1} where we bound $X_{1}$, we can bound $\sum_{\bm{c} \in \mathcal{C}_{n}\cap \mathcal{C}(\underline{\sigma})} \Delta_{k}(\bm{c}) Y_{k}(\bm{c})$ as followed for any arm $k\in [K]/\{k_{n}^{-},k_{n}^{+}\}$, i.e. arms who are at least $\delta$ away from the optimal per-channel budget w.r.t. $\bm{c}$:
\begin{align}
\label{eq:X2slope}
    \sum_{\bm{c} \in \mathcal{C}_{n}\cap \mathcal{C}(\underline{\sigma})} \Delta_{k}(\bm{c}) Y_{k}(\bm{c}) \leq \frac{8\log(T)}{\min_{\bm{c} \in \mathcal{C}_{n}\cap \mathcal{C}(\underline{\sigma})}\Delta_{k}(\bm{c})}\,.
\end{align}

Now, the set $k\in [K]/\{k_{n}^{-},k_{n}^{+}\}$ in Eq. \eqref{eq:X2all} can be further split into two subsets, namely $\{k\in [K]: a_{k} < r_{j,n} - \delta\}$ and $\{k\in [K]: a_{k} > r_{j,n} + \delta\}$ due to the definitions $k_{n}^{-}:= \max\{k\in [K]: a_{k} < r_{j,n}\}$ and $k_{n}^{+}:= \min\{k\in [K]: a_{k} > r_{j,n}\}$. Therefore, for any $k$ s.t. $ a_{k} < r_{j,n} - \delta$ and any $\bm{c} \in  \mathcal{C}_{n}\cap \mathcal{C}(\underline{\sigma})$, 
\begin{align*}
    \Delta_{k}(\bm{c}) = \mathcal{L}_{j}(r_{j,n},\bm{c}) - \mathcal{L}_{j}(a_{k},\bm{c}) \geq \sigma_{j}^{-}(\bm{c}) (r_{j,n} - a_{k}) \geq \underline{\sigma}  (r_{j,n} - a_{k})\,,
\end{align*}
where the final inequality follows from the definition of $\mathcal{C}(\underline{\sigma})$ in Eq. \eqref{eq:UCBdefs} such that $ \sigma_{j}^{-}(\bm{c}) \geq \underline(\sigma) $ for $\bm{c} \in \mathcal{C}(\underline{\sigma})$. Hence combining this with Eq. \eqref{eq:X2slope} we have
\begin{align}
\label{eq:X2last}
    \sum_{k\in [K]:a_{k} < r_{j,n}- \delta}~ \sum_{\bm{c} \in \mathcal{C}_{n}\cap \mathcal{C}(\underline{\sigma})} \Delta_{k}(\bm{c}) Y_{k}(\bm{c}) \leq  \sum_{k\in [K]:a_{k} < r_{j,n}- \delta}\frac{8\log(T)}{ \underline{\sigma}  (r_{j,n} - a_{k})} \overset{(i)}{\leq} \sum_{\ell=1}^{K}\frac{8\log(T)}{ \underline{\sigma} \ell \delta} \leq\frac{8\log(T)\log(K)}{\underline{\sigma} \delta}\,, 
\end{align}
where (i) follows because for all $a_{k} < r_{j,n} - \delta$, the $a_{k}$'s distances from $r_{j,n-1}$ are at least $\delta, 3\delta, 2\delta \dots $. Symmetrically, we can show an identical bound for the set  $\{k\in [K]: a_{k} > r_{j,n} + \delta\}$. Hence, combining Eqs. \eqref{eq:X2all} and \eqref{eq:X2last} we can conclude 
\begin{align}
    X_{2}\leq \widetilde{\mathcal{O}}\left(\delta T + \frac{1}{\delta \underline{\sigma}}\right).
\end{align}
Here, similar to our bound in Eq. \eqref{eq:X1secbound} for bounding $X_{1}$, we hide all logarithmic factors using the notation $\widetilde{\mathcal{O}}$, and note that the constants $C_{F}$, $(s_{j,n})_{n\in S_{j}}$, $S_{j}$ are all absolute constants that depend only on the support $F_{j}$ and corresponding sampling distribution $\bm{p}_{j}$ for value-cost pairs; see definitions of these absolute constants in Lemma \ref{lem:Vstruct} and \ref{lem:dualvarub}.

\subsubsection{Bounding $X_{3}$.}
\label{subsubsec:X3}

We first define 
\begin{align}
    \Bar{\mathcal{L}} = \left(1+\gamma \right)\rhoC_{F} + (1+C_{F})\Bar{V}
\end{align}
 where $C_{F}$ is specified in Lemma \ref{lem:dualvarub}. Recalling the definition  $\Delta_{k}(\bm{c}) =  \max_{\rho_{j}\in [0,\rho]}\mathcal{L}_{j}(\rho_{j}, \bm{c}) -  \mathcal{L}_{j}(a_{k}, \bm{c}) $ in Eq. \eqref{eq:UCBdefs}, and $-(1+\gamma)\rho C_{F}\leq \mathcal{L}_{j}(\rho_{j},\bm{c})\leq (1+C_{F})\Bar{V}$ for any $\rho_{j}\in[0,\rho]$ and context $\bm{c}$
 (see Lemma \ref{lem:dualvarub}), it is easy to see 
 \begin{align}
 \label{eq:boundsuboptarmDelta}
     \Delta_{k}(\bm{c})\leq \Bar{\mathcal{L}}  \quad \forall k \in [K] , \forall \bm{c}\,.
 \end{align}
 
 Then we bound $X_{3}$ as followed 
\begin{align}
\label{eq:X3first}
\begin{aligned}
   X_{3} ~=~&  \sum_{k\in [K]}\sum_{t > K} \mathbb{E}\left[\Delta_{k}(\bm{c}) \mathbb{I}\{ \rho_{j,t} = a_{k}, N_{k,t} > m_{k}(\bm{c})\}\right] \\
    ~\overset{(i)}{\leq}~& \Bar{\mathcal{L}}\cdot \sum_{k\in[K]}\sum_{t > K} \mathbb{P}\left(\rho_{j,t} = a_{k}, N_{k,t} > m_{k}(\bm{c}_{t})\right)\\
    ~\overset{(ii)}{\leq}~& \Bar{\mathcal{L}} \cdot \sum_{k\in[K]} \sum_{t > K} \mathbb{P}\Big(\Hat{V}_{j,t}(a_{k}) -\frac{\lambda_{t}\gamma + \mu_{t}}{1+\lambda_{t}} a_{k} + \texttt{UCB}_{j,t}(a_{k}) \geq \Hat{V}_{j,t}(\rho_{j}^{*}(t))-\frac{\lambda_{t}\gamma + \mu_{t}}{1+\lambda_{t}} \rho_{j}^{*}(t)+ \texttt{UCB}_{j,t}(\rho_{j}^{*}(t)),\\
    & \quad \quad N_{k,t} > m_{k}(\bm{c}_{t})\Big)\,,
\end{aligned}
\end{align}
 where (i) follows from Eq. \eqref{eq:boundsuboptarmDelta}; in (ii), recall that we choose arm $\rho_{j,t} = a_{k}$ because the estimated UCB rewards of arm $a_{k}$ is greater than that of any other arm including $\rho_{j}^{*}(t)$ according to Eq. \eqref{eq:choosearm} in UCB-SGD (Algorithm \ref{alg:omducb2}), or mathematically,
 $\Hat{V}_{j,t}(a_{k}) -\frac{\lambda_{t}\gamma + \mu_{t}}{1+\lambda_{t}} a_{k} + \texttt{UCB}_{j,t}(a_{k}) \geq \Hat{V}_{j,t}(\rho_{j}^{*}(t))-\frac{\lambda_{t}\gamma + \mu_{t}}{1+\lambda_{t}} \rho_{j}^{*}(t)+ \texttt{UCB}_{j,t}(\rho_{j}^{*}(t))$.  Here we also used the fact that $\rho_{j}^{*}(t)$ lies in the arm set $\mathcal{A}(\delta)$ for all $t$ (see Remark \ref{rmk:bestarm}).
 
Now let $\Hat{R}_{n}(a_{k})$ denote the average conversion of arm $k$ over its first $n$ pulls, i.e. 
\begin{align}
    \Hat{R}_{n}(a_{k}) =\Hat{V}_{j,\tau}(a_{k}) \text{ for }\tau = \min\{t\in [T]:N_{k,t} = n\}
\end{align}
 where we recall $\Hat{V}_{j,\tau}(a_{k})$ is the estimated conversion for arm $a_{k}$ in channel $j$ during period $\tau$ as defined in Algorithm \ref{alg:omducb}.
 In other words, $\tau$ is the period during which arm $a_{k}$ is pulled for the $n$th time so $ \Hat{R}_{n}(a_{k})  = \Hat{V}_{j,\tau}(a_{k}) $.
 
 Hence, we continue with Eq. \eqref{eq:X3first} as followed:
\begin{align}
\label{eq:X3sec}
\begin{aligned}
&\mathbb{P}\Big(\Hat{V}_{j,t}(a_{k}) -\frac{\lambda_{t}\gamma + \mu_{t}}{1+\lambda_{t}} a_{k} + \texttt{UCB}_{j,t}(a_{k}) \geq \Hat{V}_{j,t}(\rho_{j}^{*}(t))- \frac{\lambda_{t}\gamma + \mu_{t}}{1+\lambda_{t}} \rho_{j}^{*}(t)+ \texttt{UCB}_{j,t}(\rho_{j}^{*}(t)),~ N_{k,t} > m_{k}(\bm{c}_{t})\Big)\\
    ~\leq~&  \mathbb{P} \left(\max_{n: m_{k}(\bm{c}_{t}) < n \leq t} \Big\{\Hat{R}_{n}(a_{k}) + \texttt{UCB}_{n}(a_{k}) - \frac{\lambda_{t} \gamma + \mu_{t}}{1+\lambda_{t}}a_{k}\Big\} \geq \min_{n': 1\leq n'\leq t} \Big\{\Hat{R}_{n'}(\rho_{j}^{*}(t)) + \texttt{UCB}_{n'}(\rho_{j}^{*}(t)) - \frac{\lambda_{t} \gamma + \mu_{t} }{1+\lambda_{t}}\rho_{j}^{*}(t)\Big\}
    \right)\\
    ~\leq~& \sum_{n = \left\lceil m_{k}(\bm{c}_{t}) \right\rceil+1}^{t}\sum_{n'=1}^{t}\mathbb{P}\left(\Hat{R}_{n}(a_{k}) + \texttt{UCB}_{n}(a_{k}) - \frac{\lambda_{t} \gamma + \mu_{t} }{1+\lambda_{t}}a_{k} > \Hat{R}_{n'}(\rho_{j}^{*}(t)) + \texttt{UCB}_{n'}(\rho_{j}^{*}(t)) - \frac{\lambda_{t} \gamma + \mu_{t} }{1+\lambda_{t}}\rho_{j}^{*}(t)
    \right)
    \end{aligned}
\end{align}
Now, when the event $\left\{\Hat{R}_{n}(a_{k}) + \texttt{UCB}_{n}(a_{k}) - \frac{\lambda_{t} \gamma + \mu_{t} }{1+\lambda_{t}}a_{k} > \Hat{R}_{n'}(\rho_{j}^{*}(t)) + \texttt{UCB}_{n'}(\rho_{j}^{*}(t)) - \frac{\lambda_{t} \gamma + \mu_{t} }{1+\lambda_{t}}\rho_{j}^{*}(t)
    \right\}$ occurs, it is easy to see that one of the following events must also occur: 
\begin{align}
\label{eq:event}
\begin{aligned}
    & \mathcal{G}_{1,n} =\left\{  \Bar{R}_{n}(a_{k})  \geq V(a_{k}) + \texttt{UCB}_{n}(a_{k})\right\} \quad \text{ for } n \text{ s.t. } m_{k}(\bm{c}_{t}) < n \leq t\\
    & \mathcal{G}_{2,n'} =\left\{  \Bar{R}_{n'}(\rho_{j}^{*}(t))  \leq V(\rho_{j}^{*}(t)) -\texttt{UCB}_{n}(\rho_{j}^{*}(t)) \right\}\quad  \text{ for } n' \text{ s.t. } 1\leq n' \leq t\\
    & \mathcal{G}_{3} =\left\{ V_{j}(\rho_{j}^{*}(t)) -  \frac{\lambda_{t} \gamma + \mu_{t} }{1+\lambda_{t}}\rho_{j}^{*}(t)~<~ V_{j}(a_{k}) - \frac{\lambda_{t} \gamma + \mu_{t} }{1+\lambda_{t}}a_{k}  + 2 \cdot \texttt{UCB}_{n}(a_{k}) \right\}\\
    \end{aligned}
\end{align}
Note that for $n > m_{k}(\bm{c}_{t})$, we have 
$\texttt{UCB}_{n}(a_{k}) = \sqrt{\frac{2\log(T)}{n}} < \sqrt{\frac{2\log(T)}{m_{k}(\bm{c}_{t})}} = \frac{\Delta_{k}(\bm{c}_{t})}{2}$ since we defined $m_{k}(\bm{c}) = \frac{8\log(T)}{\Delta_{k}^{2}(\bm{c})}$ in Eq. \eqref{eq:UCBdefs}. Therefore
\begin{align*}
    V_{j}(a_{k}) - \frac{\lambda_{t} \gamma + \mu_{t}}{1+\lambda_{t}}a_{k} + 2 \cdot \texttt{UCB}_{n}(a_{k}) <  \underbrace{V_{j}(a_{k}) - \frac{\lambda_{t} \gamma + \mu_{t}}{1+\lambda_{t}}a_{k}}_{=\mathcal{L}(a_{k},\bm{c}_{t})} + \Delta_{k}(\bm{c}_{t}) ~\overset{(i)}{=}~ \underbrace{V_{j}(\rho_{j}^{*}(t)) -  \frac{\lambda_{t} \gamma + \mu_{t} }{1+\lambda_{t}}\rho_{j}^{*}(t)}_{=\mathcal{L}(\rho_{j}^{*}(t),\bm{c}_{t}) = \max_{a\in \mathcal{A}(\delta)}\mathcal{L}(a,\bm{c}_{t})}
\end{align*}
where (i) follows from the definition of $\Delta_{k}(\bm{c}) = \max_{a\in \mathcal{A}(\delta)}\mathcal{L}(a,\bm{c}) - \mathcal{L}(a_{k},\bm{c})$ in Eq. \eqref{eq:UCBdefs} for any context $\bm{c}$. This implies that event $\mathcal{G}_{3}$ in Eq. \eqref{eq:event} cannot hold for $n >  m_{k}(\bm{c}_{t})$. Therefore
\begin{align}
\label{eq:X3third}
    \mathbb{P}\left(\Hat{R}_{n}(a_{k}) + \texttt{UCB}_{n}(a_{k}) - \frac{\lambda_{t} \gamma + \mu_{t} }{1+\lambda_{t}}a_{k} > \Hat{R}_{n'}(\rho_{j}^{*}(t)) + \texttt{UCB}_{n'}(\rho_{j}^{*}(t)) - \frac{\lambda_{t} \gamma + \mu_{t} }{1+\lambda_{t}}\rho_{j}^{*}(t)\right)~\leq~\mathbb{P}\left(\mathcal{G}_{1,n} \cup \mathcal{G}_{2,n'}
    \right)\,.
\end{align}

From the standard UCB analysis and the Azuma Hoeffding's inequality, we have $\mathbb{P}(\mathcal{G}_{1,n} )\leq \frac{\Bar{V}}{T^{4}}$ and  $\mathbb{P}(\mathcal{G}_{2,n'} )\leq \frac{\Bar{V}}{T^{4}}$. Hence combining Eqs. \eqref{eq:X3first} \eqref{eq:X3sec}, \eqref{eq:X3third} we can conclude 

\begin{align}
\begin{aligned}
 X_{3} ~\leq~& \sum_{k\in[K]}\sum_{t > K}\sum_{n = \left\lceil m_{k}(\bm{c}_{t}) \right\rceil+1}^{t}\sum_{n'=1}^{t}\left(\mathbb{P}\left(\mathcal{G}_{1,n}\right) + \mathbb{P}\left(\mathcal{G}_{2,n'}
    \right)\right)\\
    ~\leq~& \sum_{k\in[K]}\sum_{t > K}\ \sum_{n = \left\lceil m_{k}(\bm{c}_{t}) \right\rceil+1}^{t}\sum_{n'=1}^{t} \frac{2\Bar{V}}{T^{4}}\\
     ~\leq~&  \frac{2K\Bar{V}}{T} = \mathcal{O}\left(\frac{1}{\delta T}\right).
    \end{aligned}
\end{align}
\halmos



\subsection{Proof for Theorem \ref{thm:regret}}
\label{pf:thm:regret}
Starting from Proposition \ref{prop:ubbenchlag}, we get
\begin{align}
\begin{aligned}
    T\cdot \textsc{GL-OPT} - \mathbb{E}\left[\sum_{t\in [T]}\sum_{j\in[M]}V_{j}(\rho_{j,t})\right]~\leq~& M\Bar{V}K +  \sum_{j\in [M]} \sum_{t > K} 
     \mathbb{E}\left[\mathcal{L}_{j}(\rho_{j}^{*}(t), \bm{c}_{t}) - \mathcal{L}_{j}(\rho_{j,t}, \bm{c}_{t})\right] +  \sum_{t > K}\left(\lambda_{t}g_{1,t} + \mu_{t}g_{2,t}\right)\\
     ~\overset{(i)}{\leq}~&M\Bar{V}K  + \mathcal{O}\left(\underline{\sigma}T + \delta T + \frac{1}{\underline{\sigma} \delta}\right) + \mathcal{O}\left(\eta T + \frac{1}{\eta}\right)\,,
     \end{aligned}
\end{align}
where in (i) we applied Lemmas \ref{lem:ucbregret} and \ref{lem:boundcs}. Taking $\eta = 1/\sqrt{T}$, $\delta = \underline{\sigma} = T^{-1/3}$ (i.e. $K = \mathcal{O}(T^{1/3})$ yields $ T\cdot \textsc{GL-OPT} - \mathbb{E}\left[\sum_{t\in [T]}\sum_{j\in[M]}V_{j}(\rho_{j,t})\right] \leq \mathcal{O}(T^{2/3})$. According to Lemma \ref{lem:Vstruct}, $V_{j}(\rho_{j})$ is concave for all $j \in [M]$, so
\begin{align*}
\begin{aligned}
    \mathcal{O}(T^{-1/3})~\geq~& \textsc{GL-OPT} -  \frac{1}{T}\sum_{t\in [T]} \mathbb{E}\left[\sum_{j\in[M]}V_{j}(\rho_{j,t})\right] \\
    ~\geq~&  \textsc{GL-OPT}  - \mathbb{E}\left[ \sum_{j\in[M]}  V_{j}\left(\frac{1}{T}\sum_{t\in [T]} \rho_{j,t}\right)\right]\\
     ~\geq~&  \textsc{GL-OPT}  - \mathbb{E}\left[ \sum_{j\in[M]}V_{j}(\Bar{\rho}_{T,j})\right]\,,
     \end{aligned}
\end{align*}
where in the final equality we used the definition $\Bar{\bm{\rho}}_{T}$ as defined in Algorithm \ref{alg:omducb}. 


 On the other hand, Lemma \ref{lem:approxconst} shows  that
\begin{align*}
\begin{aligned}
    -\mathcal{O}(1/\sqrt{T}) ~\leq~& \frac{1}{T}\sum_{t\in[T]} \mathbb{E}\left[g_{1,t}\right]
    ~=~ \frac{1}{T}\sum_{t\in[T]}\sum_{j\in [M]}\mathbb{E}\left[\left(V_{j}(\rho_{j,t};\bm{z}_{t}) - \gamma \rho_{j,t}\right)\right]\\
     ~=~& \frac{1}{T}\sum_{t\in[T]}\sum_{j\in [M]}\mathbb{E}\left[\left(V_{j}(\rho_{j,t}) - \gamma \rho_{j,t}\right)\right]\\
    ~\overset{(i)}{\leq}~&\sum_{j\in [M]}\left(V_{j}\left( \frac{1}{T}\sum_{t\in[T]}\rho_{j,t}\right) - \gamma \cdot \frac{1}{T}\sum_{t\in[T]}\rho_{j,t}\right)\\
     ~=~&\sum_{j\in [M]}\left(V_{j}\left(\Bar{\rho}_{T,j}\right) - \gamma \Bar{\rho}_{T,j}\right)\,.
     \end{aligned}
\end{align*}
where in (i) we again applied concavity of $V_{j}(\rho_{j})$.

For the budget constraint, again Lemma \ref{lem:approxconst} shows 
\begin{align*}
\begin{aligned}
    -\mathcal{O}(1/\sqrt{T}) ~\leq~& \frac{1}{T}\sum_{t\in[T]} \mathbb{E}\left[g_{2,t}\right]
    ~=~ \frac{1}{T}\sum_{t\in[T]}\left(\rho - \sum_{j\in [M]}\mathbb{E}\left[\rho_{j,t}\right]\right)
     ~=~ \rho - \sum_{j\in [M]}\mathbb{E}[\Bar{\rho}_{T,j}]\,.
     \end{aligned}
\end{align*}

\halmos





\subsection{Proof of Lemma \ref{lem:boundphase1}}
\label{pf:lem:boundphase1}
Recall in SGD-UCB-II for all periods $t\in T_{1}$ in Phase 1, we set the per-channel budget for each channel as some constant $\beta$ which we input to the algorithm, so that  
 $g_{1,t} = \sum_{j \in M} \left(V_{j}(\beta;\bm{z}_{t}) -  \gamma \beta\right)$ for all  $t\in T_{1}$.  Consider the hypothetical version of SGD-UCB-II where we ignore the budget balance condition $B_{t} + M\rho  > \rho T$ in step 2 of the algorithm, and terminate phase 1 only when 
 the condition $\sum_{t' \in [t-1]}g_{1,t'} > \sqrt{T}\log(T)$ in Step 1 holds. Denote this hypothetical stopping time 
as $\widetilde{T}_{1}$, defined to be 
\begin{align}
\label{eq:fakeT1}
\begin{aligned}
    \widetilde{T}_{1} = \min\Big\{t\in [T]:\sum_{t' \in [t]}  \underbrace{\sum_{j \in M} \left(V_{j}(\beta;\bm{z}_{t'}) -  \gamma \beta \right)}_{:=h_{t'}} > \sqrt{T}\log(T)\Big\}
    \end{aligned}
\end{align}
It is easy to see that with probability 1,  we have $\widetilde{T}_{1} \geq T_{1}$ where we recall  $T_{1}$ is the real stopping time of Phase 1 in SGD-UCB-II. Hence we have 
\begin{align}
\label{eq:usefakeT1}
    \mathbb{P}\left(T_{1}\geq R\right) \leq  \mathbb{P}\left(\widetilde{T}_{1}\geq R\right)
\end{align}
 Now, it is not difficult to see $\{h_{t}\}_{t}$ where $h_{t}$ defined in  Eq. \eqref{eq:fakeT1} are i.i.d. random variables, since the only randomness comes from the realization of value-cost pairs $\{\bm{z}_{t}\}_{t\in[T]}$ which are i.i.d.. Further, for any $t \in [T]$ we have
\begin{align}
\label{eq:Elower}
\begin{aligned}
     h_{t} :=\sum_{j \in M} \left(V_{j}(\beta;\bm{z}_{t}) -  \gamma \beta \right)\overset{(i)}{\geq} (\xi - \gamma)\beta 
    ~\Longrightarrow~ \Bar{h}:= \mathbb{E}[h_{t}] \geq (\xi - \gamma)\beta > 0
    \end{aligned}
\end{align}
where (i) follows from Claim \ref{clm:smallbudgetbinding} where we also defined $\xi:=\min_{j\in [M]}\min_{\bm{z}_{j} \in F_{j}} \frac{v_{j,1}}{d_{j,1}} > \gamma$ under Assumption \ref{ass:channelfeas}; in the final inequality we let $ \Bar{h}$ be the mean of the i.i.d. random variables $\{h_{t}\}_{t\in [T]}$.

We note that $\widetilde{T}_{1}> R$ implies that the sum of the first $R$ $h_{t}$'s do not exceed $\sqrt{T}\log(T)$ (see definition of $\widetilde{T}_{1}$ in Eq. \eqref{eq:fakeT1}). Hence, we have 
\begin{align}
\label{eq:boundfakeT1}
\begin{aligned}
    \mathbb{P}\left(\widetilde{T}_{1} > R \right) ~\leq~ &   
   \mathbb{P}\left(\sum_{t =1}^{\lceil R\rceil} h_{t} \leq  \sqrt{T}\log(T)\right)\\
   ~=~ &   
   \mathbb{P}\left(\sum_{t =1}^{\lceil R\rceil} (h_{t} -\Bar{h})\leq  \sqrt{T}\log(T) - \lceil R\rceil \cdot \Bar{h}\right)\\
    ~\overset{(i)}{\leq}~ &   \mathbb{P}\left(\sum_{t =1}^{\lceil R\rceil} (h_{t} -\Bar{h})\leq  \sqrt{T}\log(T) -  R(\xi - \gamma)\beta \right) \\
     ~\overset{(ii)}{=}~ &   \mathbb{P}\left(\sum_{t =1}^{\lceil R\rceil} (h_{t} -\Bar{h})\leq  -\sqrt{T}\log(T) \right) \\
      ~\overset{(iii)}{\leq}~ &   \exp\left(-\frac{T\log^{2}(T)}{2\lceil R\rceil M^{2}\Bar{V}^{2}}\right)\\
       ~\overset{(iv)}{\leq}~ &   \frac{1}{T}\,.
\end{aligned}
\end{align}
Here (i) follows from $\Bar{h}\geq (\xi - \gamma)\beta$ in Eq. \eqref{eq:Elower}; (ii) follows from the definition that $R = 2\sqrt{T}\log^{3}(T)$ and $\beta = \frac{1}{\log(T)}$  so  $R (\xi - \gamma)\beta  = 
2(\xi - \gamma)\sqrt{T}\log^{2}(T) 
\geq 2\sqrt{T}\log(T) $ for large enough $T$ such that $\log(T) > \frac{1}{\xi - \gamma}$; (iii) follows from Azuma Hoefding's inequality given that $h_{t} \in [0, M\Bar{V}]$ for any $t\in [T]$; and finally (iv) follows from $T\geq \lceil R\rceil$, and $\log(T) >  M^{2}\Bar{V}^{2}$ for large enough $T$. 
Hence, combining Eqs. \eqref{eq:boundfakeT1} and \eqref{eq:usefakeT1} yields the desired statement of the lemma. 
\halmos



\subsection{Proof of Theorem \ref{thm:regret2}}
\label{pf:thm:regret2}
The proof of this theorem consists of 3 parts. In Part I, we bound the global budget constraint violation; in Part II, we bound the global ROI constraint violation; in Part III, we bound the conversion error.

\textbf{Part I. Bounding global budget constraint violation.} The design of the 
SGD-UCB-II algorithm ensures that the per-channel budget decisions never sum up to exceed $\rho T - M\rho $, so 
\begin{align*}
 \frac{1}{T}\sum_{t\in [T_{2}]}\sum_{j\in [M]} \rho_{j,t} \leq \frac{1}{T}(\rho T - M\rho) < \rho   \,.
\end{align*}

\textbf{Part II. Bounding global ROI constraint violation.} Recall the event $\mathcal{E} = \{T_{1}\geq 2\sqrt{T}\log^{3}(T)\}$ defined in Lemma \ref{lem:boundphase1} where $T_{1}$ is the end period of Phase 1 in the SGD-UCB-II algorithm (see step 10). We consider two scenarios, namely when event $\mathcal{E}$ holds and doesn't hold. Recall $g_{1,t} = \sum_{j\in[M]}\left(V_{j}(\rho_{j,t};\bm{z}_{t}) - \gamma \rho_{j,t}\right) \in [-\gamma M\rho, MV]$. 

When event $\mathcal{E} $ holds,  we consider the following naive bound that is independent of $\mathcal{E} $
\begin{align}
\begin{aligned}
\label{eq:badeventroi}
     \mathbb{E}\left[\sum_{t\in [T_{2}]}g_{1,t} ~\Big |~ \mathcal{E}\right] 
 \geq - T\gamma M\rho\,.
 \end{aligned}
\end{align}

When event  $\mathcal{E} $ does not hold, Phase 1 terminates within the first $2\sqrt{T}\log^{3}(T)$ periods, i.e.
$T_{1} < 2\sqrt{T}\log^{3}(T)$, and the total spend balance at the end of Phase 1, namely $ B_{T_{1}}$ is at most 
\begin{align*}
    B_{T_{1}}=\sum_{t\in [T_{1}]} \sum_{j\in [M]}\beta = M\beta T_{1} <  2M \sqrt{T}\log^{2}(T) < \rho T - M\rho \,,
\end{align*}
 where we recall in Phase 1, i.e. the first $T_{1}$ periods we set each per-channel budget to be $\beta$,  and the final inequality holds for large enough $T$. This implies that the budget balance termination condition in step 2 of Algorithm \ref{alg:omducb2} is not met, so Phase 1 terminates because the ROI buffer condition in step 1 of Algorithm \ref{alg:omducb2} is met, i.e. 
\begin{align}
\label{eq:goodeventroi1}
    \sum_{t\in [T_{1}]} g_{1,t}> \sqrt{T}\log(T)\,.
\end{align}
Now in periods $t = T_{1}+1 \dots T_{2}$, following the proof of Lemma \ref{lem:approxconst}, we have
\begin{align}
\label{eq:goodeventroi2}
    \sum_{t = T_{1}+1}^{T_{2}}g_{1,t} \geq \frac{1}{\eta} \left(\lambda_{T_{1}} - \lambda_{T_{2}}\right) \overset{(i)}{\geq} -\frac{C_{F}}{\eta} = -C_{F} \sqrt{T}
\,.\end{align}
Here, (i)  follows from Lemma \ref{lem:dualvarub} such that $ 0\leq \lambda_{t} \leq C_{F}$ for any $t\in [T]$ for some absolute constant $C_{F} >0$ that only depends on the support of value-cost pairs $F = F_{1}\times \ldots \times F_{M}$, and recall the SGD step size $\eta = 1/\sqrt{T}$. Hence, combining Eqs. \eqref{eq:badeventroi}, \eqref{eq:goodeventroi1}, \eqref{eq:goodeventroi2}, we have
\begin{align}
\label{eq:exactboundroi}
\begin{aligned}
\mathbb{E}\left[ \sum_{t\in[T_{2}]}g_{1,t}  \right]
~=~& \mathbb{E}\left[ \sum_{t\in[T_{2}]}g_{1,t}  ~\Big |~\mathcal{E}\right]\mathbb{P}\left(\mathcal{E}\right) +  \mathbb{E}\left[ \sum_{t\in[T_{2}]}g_{1,t}  ~\Big |~ \mathcal{E}^{c}\right]\mathbb{P}\left(\mathcal{E}^{c}\right)\\
~\geq~ & -T\gamma M\rho \cdot \mathbb{P}\left(\mathcal{E}\right) +  \mathbb{E}\left[ \sum_{t\in[T_{2}]} g_{1,t}  ~\Big|~ \mathcal{E}^{c}\right]\mathbb{P}\left(\mathcal{E}^{c}\right)
\\
~\overset{(i)}{\geq}~ & -\gamma M\rho +  \left(\sqrt{T} \log(T) -C_{F} \sqrt{T}\right) \mathbb{P}\left(\mathcal{E}^{c}\right)\\
~\overset{(ii)}{\geq}~ & 0\,,
\end{aligned}
\end{align}
where (i) follows from Lemma \ref{lem:boundphase1} which states $\mathbb{P}\left(\mathcal{E}\right) \leq \frac{1}{T}$; in (ii) we used the fact that for large enough $T$, we have $\log(T) > -C_{F}$, and $\mathbb{P}\left(\mathcal{E}^{c}\right) \geq 1-1/T \geq 1/2$ so $-\gamma M\rho +  \frac{1}{2}\left(\sqrt{T} \log(T) -C_{F} \sqrt{T}\right) > 0$ since $\log(T) > C_{F} +\frac{2\gamma M \rho}{\sqrt{T}}$ for large $T$. Therefore Eq. \eqref{eq:exactboundroi} implies 
\begin{align}
    \mathbb{E}\left[\sum_{t\in [T_{2}]} \sum_{j\in [M]}\left(V_{j} (\rho_{j,t}) - \gamma \rho_{j,t}\right)\right]\geq 0\,.
\end{align}
Finally, we have
\begin{align*}
    \sum_{j \in [M]}\mathbb{E}\left[V_{j}(\overline{\rho}_{j}) -\gamma \overline{\rho}_{j}\right]~\overset{(i)}{\geq}~ \frac{1}{T} \mathbb{E}\left[\sum_{t\in [T_{2}]} \sum_{j\in [M]}\left(V_{j} (\rho_{j,t}) - \gamma \rho_{j,t}\right)\right]\geq 0
\end{align*}
where (i) follows from concavity of $V_{j}(\rho_{j})$ according to Lemma \ref{lem:Vstruct}.
 
\textbf{Part III. Bounding conversion error.}
We first show that $T - T_{2}\leq  M + \frac{C_{F}}{\rho} \sqrt{T}$ where $C_{F}$ is an absolute constant independent of $T$ defined in Lemma \ref{lem:dualvarub} and $T_2$ is the last period of the algorithm.

If $T_{2} = T$, the inequality $T - T_{2}\leq  M + \frac{C_{F}}{\rho} \sqrt{T}$ holds trivially. Assume $T_{2} < T$, the by the algorithm's termination criteria we have 
$\rho T \leq B_{T_{2}} + M\rho = M\rho  + \sum_{t \in [T_{2}]} \sum_{j\in [M]}\rho_{j,t} $, where we recall the definition of the spend balance 
$B_{t} = \sum_{t' \in [t]} \sum_{j\in [M]}\rho_{j,t'} $. Now, recalling $g_{2,t} = \rho - \sum_{j\in [M]}\rho_{j,t} $, we have
\begin{align}
\label{eq:boundT2stop}
\begin{aligned}
   &  \rho T ~\leq~  T_{2} \rho - \sum_{t\in [T_{2}]}g_{2,t} + M\rho 
~\overset{(i)}{\leq}~  T_{2} \rho +\frac{C_{F}}{\eta} + M\rho \\
\Longrightarrow &  T - T_{2} \leq  M + \frac{C_{F}}{\rho} \sqrt{T}\,.
\end{aligned}
\end{align}
where (i) follows from the proof of Lemma \ref{lem:approxconst} such that $ \sum_{t = 1}^{T_{2}}g_{2,t} \geq \frac{1}{\eta} \left(\lambda_{1} - \lambda_{T_{2}}\right) \overset{(ii)}{\geq} -\frac{C_{F}}{\eta} = -C_{F} \sqrt{T}$, and (ii) follows from Lemma \ref{lem:dualvarub} such that $ 0\leq \lambda_{t} \leq C_{F}$ for any $t\in [T]$ for some absolute constant $C_{F} >0$ that only depends on the support of value-cost pairs $F = F_{1}\times \ldots \times F_{M}$, and recall the SGD step size $\eta = 1/\sqrt{T}$.

So far, we have shown that $T - T_{2}\leq  M + \frac{C_{F}}{\rho} \sqrt{T}$.  Next, we bound $T\cdot  \textsc{GL-OPT} - \mathbb{E}\left[\sum_{t\in [T_{2}]}\sum_{j\in[M]}V_{j}(\rho_{j,t})\right]$ using this result. Recalling the event $\mathcal{E} = \{T_{1} > 2\sqrt{T}\log^{3}(T)\}$ defined in Lemma \ref{lem:boundphase1}, we have
\begin{align*}
\begin{aligned}
     & T\cdot  \textsc{GL-OPT} - \mathbb{E}\left[\sum_{t\in [T_{2}]}\sum_{j\in[M]}V_{j}(\rho_{j,t})\right]\\
    ~\leq~ &  \mathbb{E}\left[T\cdot  \textsc{GL-OPT}~\Big|~ \mathcal{E}\right]\mathbb{P}\left(\mathcal{E}\right) + 
    \mathbb{E}\left[ T\cdot  \textsc{GL-OPT} - \sum_{t\in [T_{2}]}\sum_{j\in[M]}V_{j}(\rho_{j,t})\Big|~ \mathcal{E}^{c}\right] \\
    ~\overset{(i)}{\leq}~& M\Bar{V} +   \mathbb{E}\left[\left(T_{1} + T-T_{2}\right)\cdot  \textsc{GL-OPT} + (T_{2} - T_{1})\cdot\textsc{GL-OPT} - \sum_{t=T_{1}+1}^{T_{2}}\sum_{j\in[M]}V_{j}(\rho_{j,t}) \Big|~ \mathcal{E}^{c} \right] \\
     ~\overset{(ii)}{\leq}~& M\Bar{V} + M\Bar{V}\left(2\sqrt{T}\log^{3}(T) + M + \frac{C_{F}}{\rho} \sqrt{T}\right)+
     \mathbb{E}\left[ (T_{2} - T_{1})\cdot\textsc{GL-OPT} - \sum_{t=T_{1}+1}^{T_{2}}\sum_{j\in[M]}V_{j}(\rho_{j,t}) \Big|~ \mathcal{E}^{c} \right] \\
    ~\overset{(iii)}{\leq}~& \mathcal{O}(T^{2/3}) \,.
     \end{aligned}
\end{align*}
Here, (i) follows from Lemma \ref{lem:boundphase1} s.t. $\mathbb{P}(\mathcal{E})\leq \frac{1}{T}$; in (ii) we used the fact that under event $\mathcal{E}^{c}$ we have $T_{1} \leq 2\sqrt{T}\log^{3}(T)$ and also Eq. \eqref{eq:boundT2stop} that bounds $T- T_{2}$; in (iii), the term $(T_{2} - T_{1})\cdot\textsc{GL-OPT} - \sum_{t=T_{1}+1}^{T_{2}}\sum_{j\in[M]}V_{j}(\rho_{j,t})$ represents the convergence loss for UCB-SGD in Algorithm \ref{alg:omducb}, which is in the order of $\mathcal{O}(T^{2/3})$ according to Theorem \ref{thm:regret}. 

Finally, using concavity of $V_{j}(\rho_{j})$ as illustrated in Lemma \ref{lem:Vstruct}, we have $\textsc{GL-OPT} - \mathbb{E}\left[\sum_{j\in[M]}V_{j}(\Bar{\rho}_{j})\right]\leq   \textsc{GL-OPT} -  \frac{1}{T}\mathbb{E}\left[\sum_{t\in [T_{2}]}\sum_{j\in[M]}V_{j}(\rho_{j,t})\right]\leq\mathcal{O}(-T^{1/3}) $
\halmos




\subsection{Additional Results for Section \ref{sec:online}}
\label{app:onlineadd}

Assumption \ref{ass:feas} ensures that for any realization $\bm{z}=(\bm{v},\bm{d})$ there must be some per-channel budget allocation that allows the advertiser to satisfy her ROI constraints as illustrated in the following proposition
\begin{proposition}[Slater's condition]
\label{prop:slater}
Assume Assumption \ref{ass:feas} holds. Let  $\bm{z}=(\bm{v},\bm{d}) = (\bm{v}_{j},\bm{d}_{j})_{j\in[M]} \in F_{1}\times \dots F_{M}$ be any realization of values and costs across all channels, then there exists some per-channel budget allocation $(\rho_{(\bm{z}),j})_{j\in[M]} \in [0,\rho]^{M}$ s.t. $ \sum_{j \in M} V_{j}(\rho_{(\bm{z}),j};\bm{z} ) > \gamma \sum_{j \in M} \rho_{(\bm{z}),j}$ and $ \sum_{j\in [M]}\rho_{(\bm{z}),j} < \rho$.
\end{proposition}
\textit{Proof.} Under Assumption \ref{ass:feas}, it is easy to see for any realization of value-cost pairs  $\bm{z} = (\bm{v}_{j},\bm{d}_{j})_{j\in[M]}$ there always exists a channel $j\in [M]$ in which there is an auction $n\in [m_{j}]$ whose value-to-cost ration is at least $\gamma$, i.e. $v_{j,n} >  \gamma d_{j,n}$. Since we assumed the ordering $\frac{v_{j,1}}{d_{j,1}} > \frac{v_{j,2}}{d_{j,2}} > \dots  > \frac{v_{j,m_{j}}}{d_{j,m_{j}}}$, we know that 
$\frac{v_{j,1}}{d_{j,1}}\geq \frac{v_{j,n}}{d_{j,n}} > \gamma$. Now, in Eq. \eqref{eq:pieceline} within the proof of Lemma \ref{lem:Vstruct}, we showed 
\begin{align*}
   V_{j}(\rho_{j};\bm{z}_{j}) = \bm{v}_{j}^{\top} \bm{x}_{j}^{*}(\rho_{j};\bm{z}_{j})=  \sum_{n \in [m_{j}]}\left(\frac{v_{j,n}}{d_{j,n}}\rho_{j}+ b_{j,n}\right)\mathbb{I}\left\{d_{j,0}+ \dots +d_{j,n-1}\leq \rho_{j}\leq d_{j,0}+ \dots +d_{j,n} \right\}
\end{align*}
where $b_{j,n} = \sum_{n' \in [n-1]} v_{j,n'} - \frac{v_{j,n}}{d_{j,n}}\cdot \left(\sum_{n' \in [n-1]} d_{j,n'}\right)$ and $  d_{j,0} = v_{j,0} = 0$. Hence by taking $\rho_{j} = \underline{\rho}$ for some $ \underline{\rho} < \min\{d_{j,1},\rho\}$, we have $V_{j}(\underline{\rho}; \bm{z}_{j}) = \frac{v_{j,1}}{d_{j,1}}\underline{\rho} > \gamma \underline{\rho}$. Therefore, constructing $\rho_{(\bm{z}),j} = (0,\dots,0,\rho_{j},0\dots 0)$ for $\rho_{j} = \underline{\rho}$ satisfies $ \sum_{j \in M} V_{j}(\rho_{(\bm{z}),j};\bm{z} ) > \gamma \sum_{j \in M} \rho_{(\bm{z}),j}$ and $ \sum_{j\in [M]}\rho_{(\bm{z}),j} < \rho$.
\halmos

\begin{lemma}[Approximate constraint satisfaction]
\label{lem:approxconst}
Assume Assumption \ref{ass:feas} holds. If Algorithm \ref{alg:omducb} is run with stepsize $\eta = 1/\sqrt{T}$ such that $\eta < \frac{1}{M \sqrt{(\Bar{V}+\gamma\rho)^{2} + \rho^{2}}}$, then we have 
\begin{align*}
     \frac{1}{T}\sum_{t\in[T]}g_{1,t} \geq -C_{F}/\sqrt{T} \text{ and }
  \frac{1}{T}\sum_{t\in[T]}g_{2,t} \geq -C_{F}/\sqrt{T} 
\end{align*}
where we recall $g_{1,t} = \sum_{j\in [M]}\left(V_{j}(\rho_{j,t};\bm{z}_{t}) - \gamma \rho_{j,t}\right)$, $g_{2,t} = \rho - \sum_{j\in [M]}\rho_{j,t}$, and $C_{F} >0 $ is an absolute constant defined in Lemma \ref{lem:dualvarub} that depends only on the support $F = F_{1} \times \dots F_{M}$.
\end{lemma}

\textit{Proof.}
According to the dual update step in Algorithm \ref{alg:omducb} wee have
\begin{align*}
    \lambda_{t+1} = \left(\lambda_{t} - \eta g_{1,t}\right)_{+} \geq \lambda_{t} - \eta g_{1,t}
\end{align*}
Summing from $t = 1 \dots T-1$ and telescoping, we get
\begin{align*}
    \lambda_{T} \geq \lambda_{1} - \eta \sum_{t\in[T]}g_{1,t} = - \eta \sum_{t\in[T]}g_{1,t} 
\end{align*}
From Lemma \ref{lem:dualvarub}, we have $ \lambda_{T} \leq C_{F}$ for some absolute constant $C_{F} >0$, therefore we have
\begin{align*}
    \sum_{t\in[T]}g_{1,t} \geq -\frac{C_{F}}{\eta} 
\end{align*}
since we take $\eta = 1/\sqrt{T}$. A similar bound holds for $ \sum_{t\in[T]}g_{2,t}$.
\halmos

\begin{lemma}
\label{lem:omdtelescope}

    Let $(\lambda_{t},\mu_{t})_{t\in[T]}$ be the dual variables generated by Algorithm \ref{alg:omducb}. Then for any $\lambda,\mu\geq 0$ we have
\begin{align}
\label{eq:CSbound0}
\begin{aligned}
    & (\lambda- \lambda_{t+1})^{2} ~\leq~
  (\lambda- \lambda_{t})^{2} + 2\eta\left(\lambda-\lambda_{t}\right) g_{1,t} + \eta^{2} M^{2}(\Bar{V}+\gamma\rho)^{2}\\
  & (\mu- \mu_{t+1})^{2} ~\leq~
  (\mu- \mu_{t})^{2} + 2\eta\left(\mu-\mu_{t}\right) g_{1,t} + \eta^{2} M^{2}\rho^{2}\Bar{V}^{2}\,,
  \end{aligned}  
\end{align} where we  recall $g_{1,t} = \sum_{j\in [M]}\left(V_{j,t}(\rho_{j,t}) - \gamma \rho_{j,t}\right)$ and $g_{2,t} = \rho - \sum_{j\in [M]}\rho_{j,t}$. 
\end{lemma}
\textit{Proof.}
We  start with the first inequality w.r.t. $\lambda_{t}$'s. We have
\begin{align}
\label{eq:decompCS}
    \left(\lambda_{t} - \lambda\right) g_{1,t} =  \left(\lambda_{t+1} - \lambda\right) g_{1,t}  +  \left(\lambda_{t} - \lambda_{t+1}\right) g_{1,t}\,. 
\end{align}
Since $\lambda_{t+1} = \left(\lambda_{t} - \eta g_{1,t}\right)_{+} = \arg\min_{\lambda\geq0}\left(\lambda - \left(\lambda_{t} - \eta g_{1,t}\right)\right)^{2}$ (see Algorithm \ref{alg:omducb}), for any $\lambda\geq 0$ we have
\begin{align*}
    \left(\lambda_{t+1} - \left(\lambda_{t} - \eta g_{1,t}\right)\right)\cdot (\lambda - \lambda_{t+1}) \geq 0\,.
\end{align*}
 by rearranging terms in $\left(\lambda_{t+1} - \left(\lambda_{t} - \eta g_{1,t}\right)\right)\cdot (\lambda - \lambda_{t+1}) \geq 0$, we get
\begin{align*}
\begin{aligned}
     \left(\lambda_{t+1} - \lambda\right) g_{1,t}~\leq~ & \frac{1}{\eta}(\lambda_{t+1} - \lambda_{t})\cdot(\lambda - \lambda_{t+1})\\
     ~=~& \frac{1}{2\eta}\left((\lambda- \lambda_{t})^{2} - (\lambda- \lambda_{t+1})^{2}-(\lambda_{t+1}- \lambda_{t})^{2}\right)\,.
     \end{aligned}
\end{align*}
 where the equality can be checked easily by expending terms apart.
Plugging the above back into Eq. \eqref{eq:decompCS} we get
\begin{align}
\label{eq:omdtelescope}
\begin{aligned}
     \left(\lambda_{t} - \lambda\right) g_{1,t} ~\leq~&  \left(\lambda_{t} - \lambda_{t+1}\right) g_{1,t}  +  \frac{1}{2\eta}\left((\lambda- \lambda_{t})^{2} - (\lambda- \lambda_{t+1})^{2}-(\lambda_{t+1}- \lambda_{t})^{2}\right)\\
     ~\overset{(i)}{\leq}~&  
   \frac{\eta}{2}g_{1,t}^{2}  +  \frac{1}{2\eta}\left((\lambda- \lambda_{t})^{2} - (\lambda- \lambda_{t+1})^{2}\right)\\
    ~\overset{(ii)}{\leq}~&
   \frac{\eta M^{2}(\Bar{V}+\gamma\rho)^{2}}{2}  +  \frac{1}{2\eta}\left((\lambda- \lambda_{t})^{2} - (\lambda- \lambda_{t+1})^{2}\right)\,,
   \end{aligned}
\end{align}
 where (i) follows from $\left(\lambda_{t} - \lambda_{t+1}\right) g_{1,t} - \frac{1}{2\eta} (\lambda_{t+1}- \lambda_{t})^{2}\leq  \frac{\eta}{2}g_{1,t}^{2}$ using the basic inequality $ab\leq \frac{1}{2}a^{2} + \frac{1}{2}b^{2}$ with $a^{2} =\frac{1}{\eta}(\lambda_{t+1}- \lambda_{t})^{2}$ and $b^{2} = \eta g_{1,t}^{2}$; (ii) follows from the fact that $V_{j,t}(\rho_{j,t})\leq \Bar{V}$ for any $j\in [M]$ and $t\in [T]$ so $-M\gamma\rho\leq g_{1,t}\leq M\Bar{V}$ so $g_{1,t}^{2}\leq M^{2}(\Bar{V}+\gamma\rho)^{2}$. Rearranging terms yields the first inequality in Eq. \eqref{eq:CSbound0}.

Following the same arguments above we can show for any $\mu\geq 0$ we have
\begin{align*}
     \left(\mu_{t} - \mu\right) g_{1,t}\leq
 \frac{\eta M^{2}\rho^{2}}{2}  +  \frac{1}{2\eta}\left((\mu- \mu_{t})^{2} - (\mu- \mu_{t+1})^{2}\right)\,.
\end{align*}
\halmos



\begin{proposition}
\label{prop:convexprob}
    Under Assumption \ref{ass:feas}, the advertiser's per-channel only budget optimization problem, namely $\textsc{CH-OPT}(\mathcal{I}_{B})$ is a convex problem.
\end{proposition}
\text{Proof.}
Recalling the $\textsc{CH-OPT}(\mathcal{I}_{B})$ in Eq. \eqref{eq:chopt} and the definition of $\mathcal{I}_{B}$ in Eq. \eqref{eq:option}, we can write $\textsc{CH-OPT}(\mathcal{I}_{B})$ as
\begin{align}
\begin{aligned}
   \textsc{CH-OPT}(\mathcal{I}_{B})=  \max_{(\rho_{j})_{j\in [M]}\in \mathcal{I}} ~&~ \sum_{j \in M} V_{j}(\rho_{j})\\
    \text{s.t. } &  \sum_{j \in M} V_{j}(\rho_{j}) \geq \gamma \sum_{j \in M} \rho_{j}\\
   &   \sum_{j\in [M]}\rho_{j}\leq \rho\,.
    \end{aligned}
\end{align}
Here, we used the definition $V_{j}(\rho_{j}) = \mathbb{E}\left[V_{j}(\rho_{j};\bm{z}_{j})\right]$ in Eq. \eqref{eq:budperchannel}, and $D_{j}(\rho_{j};\bm{z}_{j}) = \rho_{j}$ for any $\bm{z}_{j}$ under the budget depletion Assumption \ref{ass:budgetbind}. According to Lemma \ref{lem:Vstruct}, $V_{j}(\rho_{j})$ is concave in $\rho_{j}$ for any $j$, so the objective of $\textsc{CH-OPT}(\mathcal{I}_{B})$ maximizes a concave function. For the feasibility region, assume $\rho_{j}$ and $\rho_{j}'$ are feasible, then defining $\rho_{j}'' = \theta \rho_{j} +  (1-\theta)\rho_{j}'$ for any $\theta \in [0,1]$, we know that 
\begin{align*}
\begin{aligned}
     \sum_{j \in M} \left(V_{j}(\rho_{j}'') - \gamma  \rho_{j}''\right) 
     ~\overset{(i)}{\geq}~ &  \sum_{j \in M} \left(\thetaV_{j}(\rho_{j})  +(1-\theta)V_{j}(\rho_{j}')  - \gamma  \rho_{j}''\right)\\
      ~=~ &  \theta\sum_{j \in M} \left(V_{j}(\rho_{j}) - \gamma  \rho_{j}\right) + (1-\theta)
     \sum_{j \in M} \left(V_{j}(\rho_{j}') - \gamma  \rho_{j}'\right)\\
       ~\overset{(ii)}{\geq}~ & 0
       \end{aligned}
\end{align*}
where (i) follows from concavity of $V_{j}(\rho_{j})$ and (ii) follows from feasiblity of $\rho_{j}$ and $\rho_{j}'$. On the other hand it is apparent that $\sum_{j\in [M]}\rho_{j}'' \leq\rho$. Hence we conclude that for any $\rho_{j}$ and $\rho_{j}'$  feasible,  $\rho_{j}'' = \theta \rho_{j} +  (1-\theta)\rho_{j}'$  is also feasible, so the feasible region of 
$\textsc{CH-OPT}(\mathcal{I}_{B})$ is convex. This concludes the statement of the proposition. \halmos




\begin{claim}
\label{clm:smallbudgetbinding}
Assume Assumption \ref{ass:channelfeas} holds, then for  any channel $j\in [M]$  and value-cost realization $\bm{z}_{j} = (\bm{v}_{j},\bm{d}_{j}) \in F_{j}$, we have $\frac{v_{j,1}}{d_{j,1}} > \gamma$. This further implies that $\xi:= \min_{j\in [M]}\min_{\bm{z}_{j} \in F_{j}} \frac{v_{j,1}}{d_{j,1}} > \gamma$. Further, let $\beta = \frac{1}{\log(T)}$. 
Then, for large enough $T$ we have  $V_{j}(\beta;\bm{z}_{j}) = \frac{v_{j,1}}{d_{j,1}} \beta \geq \xi\beta$ for any realization $\bm{z}_{j} \in F_{j}$.
\end{claim}


\textit{Proof.}
 Under Assumption \ref{ass:channelfeas}, we know that for any realization of value-cost pairs  $\bm{z}_{j} = (\bm{v}_{j},\bm{d}_{j})$ there always exists an auction $n\in [m_{j}]$ in this channel  whose value-to-cost ratio is at least $\gamma$, i.e. $v_{j,n} >  \gamma d_{j,n}$. Under the ordering $\frac{v_{j,1}}{d_{j,1}} > \frac{v_{j,2}}{d_{j,2}} > \dots  > \frac{v_{j,m_{j}}}{d_{j,m_{j}}}$, we have 
 $\frac{v_{j,1}}{d_{j,1}} > \gamma$. In Eq. \eqref{eq:pieceline} within the proof of Lemma \ref{lem:Vstruct}, we showed that for any realization $\bm{z}_{j}\in F_{j}$, $V_{j}(\rho_{j};\bm{z}_{j}) = \frac{v_{j,1}}{d_{j,1}} \rho_{j}$ for all $\rho_{j} \leq d_{j,1}$. Hence we know that  when $T$ is large enough such that $\beta = \frac{1}{\log(T)} < \min_{j\in [M]}\min_{\bm{z}_{j}\in F_{j}}d_{j,1}$, we always have $V_{j}(\beta;\bm{z}_{j}) = \frac{v_{j,1}}{d_{j,1}}\beta \geq \xi\beta$.
\halmos
\subsection{Proof of Proposition \ref{prop:budgetbind}}




\subsection{Proof of Proposition \ref{prop:ubbenchlag}}
\label{pf:prop:ubbenchlag}
Let $(\rho_{j}^{*})_{j\in[M]}$ be the optimal per-channel budgets to $ \textsc{CH-OPT}(\mathcal{I}_{B})$. Then
\begin{align}
\begin{aligned}
     & T\cdot \textsc{GL-OPT} - \mathbb{E}\left[\sum_{t\in [T]}\sum_{j\in[M]}V_{j}(\rho_{j,t})\right] \\
    ~\overset{(i)}{\leq}~ & M\Bar{V}K + (T-K)\cdot \textsc{CH-OPT}(\mathcal{I}_{B}) - \mathbb{E}\left[\sum_{t > K}\sum_{j\in[M]}V_{j}(\rho_{j,t})\right]\\
     ~\overset{(ii)}{=}~ & M\Bar{V}K + \rho\sum_{t > K}\mathbb{E}\left[\mu_{t}\right] + \sum_{t>K}\sum_{j\in [M]} \mathbb{E}\left[\mathcal{L}_{j}(\rho_{j}^{*}, \lambda_{t}, \mu_{t})\right] -  \sum_{t > K}\sum_{j\in [M]}\mathbb{E}\left[\mathcal{L}_{j}(\rho_{j,t}, \bm{c}_{t}) - \lambda_{t}\left(V_{j}(\rho_{j,t}) - \gamma\rho_{j,t}\right) + \mu_{t}\rho_{j,t}\right]\\
      ~\overset{(iii)}{\leq}~ & M\Bar{V}K +  \sum_{j\in [M]} \sum_{t > K} 
     \mathbb{E}\left[\mathcal{L}_{j}(\rho_{j}^{*}(t), \bm{c}_{t}) - \mathcal{L}_{j}(\rho_{j,t}, \bm{c}_{t})\right] +  \sum_{t > K}\left(\lambda_{t}g_{1,t} + \mu_{t}g_{2,t}\right)\,.
    \end{aligned}
\end{align}
Here, (i) follows from Theorem \ref{thm:budgonlysuff} that states $\textsc{GL-OPT} =   \textsc{CH-OPT}(\mathcal{I}_{B})$; (ii) follows from the $ \textsc{CH-OPT}(\mathcal{I}_{B}) = \sum_{j\in[M]}V_{j}(\rho_{j}^{*})$ and the definition of the Lagrangian in Eq. \eqref{eq:Lagrangian};  
in (ii) we define $\rho_{j}^{*}(t) = \arg\max_{\rho_{j}\geq 0} \mathcal{L}_{j}(\rho_{j}, \bm{c}_{t})$ to be the optimal budget that maximizes the Lagrangian w.r.t. the dual variables $\bm{c}_{t}=(\lambda_{t},\mu_{t})$.
\halmos





\subsection{Proof of Lemma \ref{lem:Vstruct}}
\label{pf:lem:Vstruct}
We first show for any realization $\bm{z} = (\bm{z}_{j})_{j\in [M]} = (\bm{v}_{j},\bm{d}_{j})_{j\in [M]} $, the conversion function $V_{j}(\rho_{j};\bm{z}_{j})$
is piecewise linear, strictly increasing, and concave for any $j \in [M]$. 

Fix any channel $j$ which consists of $m_{j}$ parallel auctions, and recall that we assumed the orderding
$\frac{v_{j,1}}{d_{j,1}} > \frac{v_{j,2}}{d_{j,2}} > \dots  > \frac{v_{j,m_{j}}}{d_{j,m_{j}}}$ for any realization $\bm{z}_{j}$. Then, with the option where the per-channel ROI is set to 0 (i.e. omitted) $V_{j}(\rho_{j};\bm{z}_{j})$ is exactly the LP relaxation of a 0-1 knapsack, whose optimal solution  $\bm{x}_{j}^{*}(\rho_{j};\bm{z}_{j})$ is well known to be unique, and takes the form for any auction index $n\in [m_{j}]$:
\begin{align}
\label{eq:knapsack}
    x_{j,n}^{*}(\rho_{j};\bm{z}_{j}) = \begin{cases}
    1 & \text{if } \sum_{n' \in [n]} d_{j,n'} \leq \rho_{j}\\
    \frac{\rho_{j} - \sum_{n' \in [n-1]} d_{j,n'}}{d_{j,n}} & \text{if } \sum_{n' \in [n]} d_{j,n'} > \rho_{j}\\
    0 & \text{otherwise}
    \end{cases}
\end{align}
where we denote $d_{j,0} =0$. With this form, it is easy to see
\begin{align}
\label{eq:pieceline}
   V_{j}(\rho_{j};\bm{z}_{j}) = \bm{v}_{j}^{\top} \bm{x}_{j}^{*}(\rho_{j};\bm{z}_{j})=  \sum_{n \in [m_{j}]}\left(\frac{v_{j,n}}{d_{j,n}}\rho_{j}+ \widetilde{b}_{j,n}\right)\mathbb{I}\left\{d_{j,0}+ \dots +d_{j,n-1}\leq \rho_{j}\leq d_{j,0}+ \dots +d_{j,n} \right\}
   \end{align}
where we denote $d_{j,0} =0$ and also $\widetilde{b}_{j,n} = \sum_{n' \in [n-1]} v_{j,n'} - \frac{v_{j,n}}{d_{j,n}}\cdot \left(\sum_{n' \in [n-1]} d_{j,n'}\right)$ and $ v_{j,0} = 0$. It is easy to check that any two line segments, say $[X_{n-1},X_{n}]$ and $[X_{n},X_{n+1}]$ where we write $X_{n} =  d_{j,0}+ \dots +d_{j,n} $, intersect at $\rho_{j} =X_{n}$, because $\frac{v_{j,n}}{d_{j,n}}\rho_{j}+ \widetilde{b}_{j,n} = \frac{v_{j,n+1}}{d_{j,n+1}}\rho_{j}+ \widetilde{b}_{j,n+1}$
at $\rho_{j} =X_{n}$. Hence, from Eq. \eqref{eq:pieceline} we can conclude $ V_{j}(\rho_{j};\bm{z}_{j})$ is continuous, which further implies it is piecewise linear and strictly increasing. Further, the ordering
$\frac{v_{j,1}}{d_{j,1}} > \frac{v_{j,2}}{d_{j,2}} > \dots  > \frac{v_{j,m_{j}}}{d_{j,m_{j}}}$ implies that the slopes on each segment $[X_{n},X_{n+1}]$ decreases as $n$ increases, which implies  $ V_{j}(\rho_{j};\bm{z}_{j})$  is concave.

Since $V_{j}(\rho_{j})= \mathbb{E}\left[V_{j}(\rho_{j};\bm{z}_{j})\right]$, where the expectation is taken w.r.t. randomness in $\bm{z}_{j}$, and since the $\bm{z}_{j}$ is sampled from some discrete distribution $\bm{p}_{j}$ on finite support $F_{j}$, $V_{j}(\rho_{j})$ is simply a weighted average over all $\left(V_{j}(\rho_{j};\bm{z}_{j})\right)_{\bm{z}_{j} \in F_{j}}$ with weights in $\bm{p}_{j}$, so $V_{j}(\rho_{j})$ is also continuous, piecewise linear, strictly increasing, and concave, and thus can be written as Eq. \eqref{eq:Vstruct} with parameters $\{(s_{j,n},b_{j,n},r_{j,n})\}_{n\in [S_{j}]}$ that only depend on the support $F_{j}$ and distribution $\bm{p}_{j}$. 

Finally, according to the definition of $\mathcal{L}_{j}(\rho_{j},\bm{c})= \mathbb{E}\left[\mathcal{L}_{j}(\rho_{j},\bm{c};\bm{z}_{j})\right]$ and $  \mathcal{L}_{j}(\rho_{j}, \bm{c}; \bm{z}_{j}) =  (1+\lambda)V_{j}(\rho_{j};\bm{z}_{j}) - (\lambda \gamma  + \mu) \rho_{j} $
as defined in Eq. \eqref{eq:Lagrangian}, we have
\begin{align}
\label{eq:Elagr}
    \mathcal{L}_{j}(\rho_{j},\bm{c}) = (1+\lambda)V_{j}(\rho_{j}) - (\lambda \gamma  + \mu) \rho_{j}
\end{align}
which implies $ \mathcal{L}_{j}(\rho_{j},\bm{c})$ is continuous, piecewise linear, and concave because $V_{j}(\rho_{j})$ is continuous, piecewise linear, and concave as shown above. Combining Eq. \eqref{eq:Elagr} and the representation of $V_{j}(\rho_{j})$ in Eq. \eqref{eq:Vstruct}, we have
\begin{align}
        \mathcal{L}_{j}(\rho_{j}, \bm{c}) =  \sum_{n\in [S_{j}]} \left( \sigma_{j,n}(\bm{c}) \rho_{j} + (1+\lambda)b_{j,n}\right)\mathbb{I}\{r_{j,n-1}\leq \rho_{j} \leq r_{j,n}\}\,.
\end{align}
where the slope $\sigma_{j,n}(\bm{c}) = (1+\lambda) s{j,n} -(\mu + \gamma \lambda)$ decreases in $n$. Thus at the point $r_{j,n^{*}} = \max\{r_{j,n}:n=0,1\dots, S_{j}~,\sigma_{j,n}(\bm{c}) \geq 0\}$ in which the slope to the right turns negative for the first time, $ \mathcal{L}_{j}(\rho_{j}, \bm{c}) $ takes its maximum value
$\max_{\rho_{j}\geq 0} \mathcal{L}_{j}(\rho_{j}, \bm{c})$, because to the left of  $r_{j,n^{*}}$, namely the region
$[0,r_{j,n^{*}}]$, $\mathcal{L}_{j}(\rho_{j}, \bm{c}) $ strictly increases because slopes are positive; and to the right of  $r_{j,n^{*}}$, namely the region $[r_{j,n^{*}},\rho]$, $\mathcal{L}_{j}(\rho_{j}, \bm{c}) $ strictly decreases because slopes are negative.
\halmos 






\subsection{Proof for Lemma \ref{lem:dualvarub}}
\label{pf:lem:dualvarub}


We first present some definitions for convenience: denote $\bm{c} = (\lambda, \mu)$, $\bm{c}_{t} = (\lambda_{t}, \mu_{t})$, and $\bm{z}_{t} = (\bm{v}_{t},\bm{d}_{t})$. For any realization $\bm{z} = (\bm{v},\bm{d}) = (\bm{v}_{j},\bm{d}_{j})_{j\in[M]} \in F_{1}\times \dots F_{M}$ of values and costs across channels, define the dual function 
\begin{align}
\label{eq:dualfunc}
\begin{aligned}
    \mathcal{L}_{d}(\bm{c};\bm{z}) ~=~& \max_{\bm{\rho}\in [0,\rho]^{M}} \sum_{j\in[M]}V_{j}(\rho_{j};\bm{z}) + 
\lambda \sum_{j\in [M]}\left(V_{j}(\rho_{j};\bm{z}) - \gamma \rho_{j}\right) + \mu\left(\rho - \sum_{j\in [M]}\rho_{j}\right)\\
~=~& \mu \rho + \max_{\bm{\rho}\in [0,\rho]^{M}}\mathcal{L}(\bm{\rho},\bm{c};\bm{z})\,.
\end{aligned}
\end{align}
From standard convex analysis we know that $ \mathcal{L}_{d}$ is convex in $(\lambda,\mu)$, which implies for any $\bm{c}' = (\lambda',\mu')$
\begin{align}
\label{eq:convex}
    (\bm{c}' -\bm{c})^{\top} \nabla  \mathcal{L}_{d}(\bm{c};\bm{z}) \leq \mathcal{L}_{d}(\bm{c}';\bm{z})-\mathcal{L}_{d}(\bm{c};\bm{z})
\end{align}


We prove the lemma by induction that 
\begin{align}
    \lambda_{t},\mu_{t} \leq C_{F} := 1 + \frac{\max_{\bm{z}\in F_{1}\times \dots F_{M}}\bm{e}^{\top}\bm{v} + 1}{\min_{\bm{z}\in F_{1}\times \dots F_{M}}\left\{\sum_{j\in [M]}\left(V_{j}(\rho_{(\bm{z}),j};\bm{z}) - \gamma \rho_{j}\right), \rho - \sum_{j\in [M]}\rho_{(\bm{z}),j} \right\}} 
\end{align}
Here, in Proposition \ref{prop:slater} we defined $\bm{\rho}_{(\bm{z})}$ for any  $\bm{z}$ such that Slater's condition holds, i.e. $\sum_{j\in [M]}\left(V_{j}(\rho_{(\bm{z}),j};\bm{z}) - \gamma \rho_{j}\right) > 0$ and 
$\rho - \sum_{j\in [M]}\rho_{(\bm{z}),j}  > 0$. 

The base case for $t = 1$ is satisfied trivially since we take $\bm{\bm{c}}_{t} = 0$. Now assume $\norm{\bm{c}_{t}}\leq C_{F}$, and we will show $\norm{\bm{c}_{t+1}}\leq C_{F}$ by considering two different case:


\textbf{Case (A):} If $\mathcal{L}_{d}(\bm{c}_{t};\bm{z}_{t}) > \mathcal{L}_{d}(\bm{0};\bm{z}_{t}) + \eta M^{2}\Bar{V}^{2}$.

Define $\bm{g}_{t} = (g_{1,t},g_{2,t})$ where we recall $g_{1,t} = \sum_{j\in [M]}\left(V_{j,t}(\rho_{j,t}) - \gamma \rho_{j,t}\right)$ and $g_{2,t} = \rho - \sum_{j\in [M]}\rho_{j,t}$. 
From Eq. \eqref{eq:omdtelescope} in the proof of Lemma \ref{lem:omdtelescope}, we have for any $\lambda\geq 0$ and $\mu\geq 0$,
\begin{align*}
\begin{aligned}
    & (\lambda- \lambda_{t+1})^{2} ~\leq~
  (\lambda- \lambda_{t})^{2} + 2\eta\left(\lambda-\lambda_{t}\right) g_{1,t} + \eta^{2} M^{2}\Bar{V}^{2}\\
  & (\mu- \mu_{t+1})^{2} ~\leq~
  (\mu- \mu_{t})^{2} + 2\eta\left(\mu-\mu_{t}\right) g_{1,t} + \eta^{2} M^{2}\Bar{V}^{2}\,.
  \end{aligned}  
\end{align*}
By summing the two inequalities and taking $\lambda = \mu = 0$ we get
\begin{align}
\label{eq:dualvarubtelescope}
\begin{aligned}
\norm{\bm{c}_{t+1}}^{2} ~\leq~& \norm{\bm{c}_{t}}^{2} +  2\eta\left(\bm{c}_{t} \right)^{\top} \bm{g}_{t}  +  2\eta^{2}M^{2}\Bar{V}^{2}\\
~\overset{(i)}{\leq}~ & \norm{ \bm{c}_{t}}^{2} +  2\eta\left(\mathcal{L}_{d}(\bm{0};\bm{z}_{t})-\mathcal{L}_{d}(\bm{c}_{t};\bm{z}_{t})\right)+  2\eta^{2} M^{2}\Bar{V}^{2}& 
\,,
\end{aligned}
\end{align}
where in (i) we applied Eq. \eqref{eq:convex} with $\bm{z} =\bm{z}_{t}$ and used the fact that $\bm{g}_{t} = (g_{1,t},g_{r,t})= \nabla \mathcal{L}_{d}(\bm{c}_{t};\bm{z}_{t})$.

Then from Eq. \eqref{eq:dualvarubtelescope} and the assumption $\mathcal{L}_{d}(\bm{c}_{t};\bm{z}_{t}) > \mathcal{L}_{d}(\bm{0};\bm{z}_{t}) + \eta M^{2}\Bar{V}^{2}$  we have
\begin{align*}
    \max\{\lambda_{t},\mu_{t}\}\leq \norm{\bm{c}_{t+1}} < \norm{\bm{c}_{t}}\leq C_{F}
\end{align*}
where the final inequality follows from the induction hypothesis.


\textbf{Case (B):}  If $\mathcal{L}_{d}(\bm{c}_{t};\bm{z}_{t}) \leq \mathcal{L}_{d}(\bm{0};\bm{z}_{t}) + \eta M^{2}\Bar{V}^{2}$, then 
\begin{align}
\label{eq:bounddualvar0}
\begin{aligned}
 & \mathcal{L}_{d}(\bm{0};\bm{z}_{t}) + \eta M^{2}\Bar{V}^{2} \\
 ~\geq~ &\mathcal{L}_{d}(\bm{c}_{t};\bm{z}_{t}) \\
 ~\overset{(i)}{=}~& 
\mu_{t} \rho + \max_{\bm{\rho}\in [0,\rho]^{M}}\mathcal{L}(\bm{\rho},\bm{c}_{t};\bm{z}_{t})\\
~\overset{(ii)}{\geq}~& 
\mu_{t} \rho + \mathcal{L}(\bm{\rho}_{(\bm{z}_{t})},\bm{c}_{t};\bm{z}_{t})\\
~=~& \sum_{j\in[M]}V_{j}(\rho_{(\bm{z}_{t}),j};\bm{z}_{t}) + 
\lambda_{t} \sum_{j\in [M]}\left(V_{j}(\rho_{(\bm{z}_{t}),j};\bm{z}_{t}) - \gamma \rho_{j}\right) + \mu_{t}\left(\rho - \sum_{j\in [M]}\rho_{(\bm{z}_{t}),j}\right)\\
~\geq~& \lambda_{t} \sum_{j\in [M]}\left(V_{j}(\rho_{(\bm{z}_{t}),j};\bm{z}_{t}) - \gamma \rho_{j}\right) + \mu_{t}\left(\rho - \sum_{j\in [M]}\rho_{(\bm{z}_{t}),j}\right)\\
~\overset{(iii)}{\geq}~& \left(\lambda_{t} +\mu_{t} \right)
\cdot \min_{\bm{z}\in F_{1}\times \dots F_{M}}\left\{\sum_{j\in [M]}\left(V_{j}(\rho_{(\bm{z}),j};\bm{z}) - \gamma \rho_{j}\right), \rho - \sum_{j\in [M]}\rho_{(\bm{z}),j} \right\}\,.
\end{aligned}
\end{align}
Here, (i) follows from the definition  of the dual function in Eq. \eqref{eq:dualfunc}; in (ii) we recall the definition of $\bm{\rho}_{(\bm{z})}$ that satisfies the Slater's condition in Lemma \ref{prop:slater}; in (iii), we used the fact that $\lambda_{t},\mu_{t}\geq 0$ and also under Slater's condition we have for any $\bm{z}$, $\sum_{j\in [M]}\left(V_{j}(\rho_{(\bm{z}),j};\bm{z}) - \gamma \rho_{j}\right) > 0$ and 
$\rho - \sum_{j\in [M]}\rho_{(\bm{z}),j}  > 0$. 


On the other hand, we have $\eta M^{2}\bar{V}^{2} < 1$ and also $\mathcal{L}_{d}(\bm{0};\bm{z}_{t}) = \max_{\bm{\rho}\in [0,\rho]^{M}}\sum_{j\in [M]}V_{j}(\rho_{j};\bm{z}_{t}) =\max_{\bm{z}\in F_{1}\times \dots F_{M}}\bm{e}^{\top}\bm{v}$. Hence combining this with Eq. \eqref{eq:bounddualvar0} we get 
\begin{align}
    \lambda_{t} +\mu_{t} < \frac{\max_{\bm{z}\in F_{1}\times \dots F_{M}}\bm{e}^{\top}\bm{v} + 1}{\min_{\bm{z}\in F_{1}\times \dots F_{M}}\left\{\sum_{j\in [M]}\left(V_{j}(\rho_{(\bm{z}),j};\bm{z}) - \gamma \rho_{j}\right), \rho - \sum_{j\in [M]}\rho_{(\bm{z}),j} \right\}} := C_{F} - 1
\end{align}
which further implies $ \lambda_{t},\mu_{t}\leq C_{F} - 1$. Hence, we can finally conclude
\begin{align}
    \lambda_{t+1} = \left(\lambda_{t} - \eta g_{1,t} \right)_{+} \leq \lambda_{t} + \eta |g_{1,t}|\leq \norm{\bm{c}_{t}} - 1 + \eta M\bar{V} < C_{F} 
\end{align}
where in the final inequality we used the fact that $\eta M \bar{V} < 1$. A similar bound holds for $\mu_{t+1}$.
\halmos





\subsection{Proof for Lemma \ref{lem:ucbregret}}
\label{pf:lem:ucbregret}

For simplicity, for any context $\bm{c}$, assuming the optimal budget $\arg\max_{\rho_{j}\in[0,\rho]} \mathcal{L}_{j}(\rho_{j},\bm{c})$ is located at the $n$th ``turning point'' $r_{j,n}$ we have
\begin{align}
\label{eq:adj2}
\begin{aligned}
    & \sigma_{j}^{-}(\bm{c}) = \sigma_{j,n}(\bm{c}) 
    ~\text{ and }~ \sigma_{j}^{+}(\bm{c}) = \sigma_{j,n}(\bm{c}) \\
    & \mathcal{U}_{j}^{-}(\bm{c}) = [r_{j,n-1},r_{j,n}]    ~\text{ and }~ \mathcal{U}_{j}^{+}(\bm{c}) = [r_{j,n},r_{j,n+1}]  \,.
  \end{aligned}  
\end{align}
Further, define the following 
\begin{align}
\label{eq:UCBdefs}
\begin{aligned}
    &  \Delta_{k}(\bm{c}) =  \max_{\rho_{j}\in [0,\rho]}\mathcal{L}_{j}(\rho_{j}, \bm{c}) -  \mathcal{L}_{j}(a_{k}, \bm{c}) \\
    & \mathcal{C}(\underline{\sigma}) = \left\{\bm{c} \in \{\bm{c}_{t}\}_{t\in [T]}: \sigma_{j}^{-}(\bm{c}) > \underline{\sigma},~ |\sigma_{j}^{+}(\bm{c})| > \underline{\sigma}\right\} \text{ for } n = 0 \dots S_{j} \\
    & \mathcal{C}_{n} = \Big\{\bm{c} \in \{\bm{c}_{t}\}_{t\in [T]}: r_{j,n} = \arg\max_{\rho_{j}\geq 0} \mathcal{L}_{j}(\rho_{j},\bm{c})\Big\} \text{ for } n = 0 \dots S_{j} \\
    & m_{k}(\bm{c}) = \frac{8\log(T)}{\Delta_{k}^{2}(\bm{c})} \text{ for } \forall (k,\bm{c}) \text{ s.t. }\Delta_{k}(\bm{c}) > 0 \,.
    \end{aligned}
\end{align}
where $S_{j}$ and $\{r_{j,n}\}_{j\in [S_{j}]}$ are defined in Lemma \ref{lem:Vstruct}. The set $\mathcal{C}_{k,n} $ represents all context $\bm{c}_{t}$ under which the optimal $\arg\max_{\rho_{j}\geq 0} \mathcal{L}_{j}(\rho_{j},\bm{c}_{t})$ is taken at the $n$th ``turning point'' $r_{j,n}$ of $\mathcal{L}_{j}(\rho_{j},\bm{c}_{t})$ which is piecewiese linear in $\rho_{j}$ and also the two slopes adjacent to 
$r_{j,n}$, namely $\sigma_{j,n}(\bm{c})$ and $\sigma_{j,n+1}(\bm{c})$ satisfy $\sigma_{j,n}(\bm{c}) > \underline{\sigma}$ and $|\sigma_{j,n+1}(\bm{c})| > \underline{\sigma}$. Further it is easy to see that $\mathcal{C} = \cup_{n=0}^{S_{j}}\mathcal{C}_{n}$.

Further, define $N_{k,t} = \sum_{\tau\leq t-1}\mathbb{I}\{\rho_{j,\tau} = a_{k}\}$ to be the number of times arm $k$ is pulled up to time $t$. we get
\begin{align}
\begin{aligned}
    &  \sum_{ t \in [T]}\mathcal{L}_{j}(\rho_{j,t}^{*}, \bm{c}_{t}) - \mathcal{L}_{j}(\rho_{j,t}, \bm{c}_{t}) = X_{1}+X_{2} + X_{3} \quad \text{ where }\\
    & X_{1} = \sum_{t\in[T]:\bm{c}_{t} \notin \mathcal{C}(\underline{\sigma})}  \sum_{k\in [K]} \Delta_{k}(\bm{c}_{t}) \mathbb{I}\{\rho_{j,t} = a_{k}, N_{k,t} \leq m_{k}(\bm{c}_{t})\}\\
    & X_{2} = \sum_{t\in[T]:\bm{c}_{t} \in \mathcal{C}}   \sum_{k\in [K]}\Delta_{k}(\bm{c}_{t}) \mathbb{I}\{\rho_{j,t} = a_{k}, N_{k,t} \leq m_{k}(\bm{c}_{t})\}\\
    & X_{3} = \sum_{k\in [K]}\sum_{ t \in [T]} \Delta_{k}(\bm{c}_{t}) \mathbb{I}\{\rho_{j,t} = a_{k}, N_{k,t} > m_{k}(\bm{c}_{t})\}
    \end{aligned}
\end{align}
In Section \ref{subsubsec:X1}, we show that $X_{1} \leq \mathcal{O}(\delta T + \underline{\sigma}T + \frac{1}{\delta})$; in Section \ref{subsubsec:X2}  we show that $X_{2} \leq \mathcal{O}(\delta T + \frac{1}{\delta \underline{\sigma}})$;
 in Section \ref{subsubsec:X3}  we show that $X_{3} \leq \mathcal{O}(1)$.

\subsubsection{Bounding $X_{1}$.}
\label{subsubsec:X1}
At each context $\bm{c}_{t} \notin \mathcal{C}(\underline{\sigma})$, we separate the set of arms $[K]$ into two groups, namely arms that lie at the left and right adjacent edges of optimal, namely $a_{k}\in \mathcal{U}_{j}^{-}(\bm{c}_{t}) \cup  \mathcal{U}_{j}^{+}(\bm{c}_{t})$ and all other arms (recall the sets $\mathcal{U}_{j}^{-}$ and $\mathcal{U}_{j}^{+}$ are defined in Eq. \eqref{eq:adj2}).

(1) For arm $k$ such that $a_{k} \in \mathcal{U}_{j}^{-}(\bm{c}_{t})$, recall Lemma \ref{lem:Vstruct} that $\mathcal{L}_{j}(a,\bm{c}_{t})$ is linear in $a$ for $a \in \mathcal{U}_{j}^{-}(\bm{c}_{t})$, so $\Delta_{k}(\bm{c}_{t}) = \sigma_{j}^{-}(\bm{c}_{t})\cdot (\rho_{j,t}^{*} - a_{k}) \leq \underline{\sigma} (r_{j,n} - r_{j,n}) \leq  \underline{\sigma} \rho$, so summing over all such $k$  we get 
\begin{align}
\label{eq:X1near}
\begin{aligned}
   &  \sum_{t\in[T]:\bm{c}_{t} \notin \mathcal{C}(\underline{\sigma})}  ~\sum_{k\in[K]:a_{k} \in \mathcal{U}_{j}^{-}(\bm{c}_{t})}  \Delta_{k}(\bm{c}_{t}) \mathbb{I}\{\rho_{j,t} = a_{k}, N_{k,t} \leq m_{k}(\bm{c}_{t})\} \\
   ~\leq~& \sum_{t\in[T]:\bm{c}_{t} \notin \mathcal{C}(\underline{\sigma})}   ~\sum_{k\in[K]:a_{k} \in \mathcal{U}_{j}^{-}(\bm{c}_{t})}  \underline{\sigma} \rho \cdot  \mathbb{I}\{\rho_{j,t} = a_{k}, N_{k,t} \leq m_{k}(\bm{c}_{t})\}  ~\leq~ \underline{\sigma} \rho T = \mathcal{O}(\underline{\sigma}T )\,.
   \end{aligned}
\end{align}

(2) For arm $k$ such that $a_{k} < \min \mathcal{U}_{j}^{-}(\bm{c}_{t})$, we further split contexts into groups $\mathcal{C}_{n}$ for $n = 0 \dots S_{j}$ (defined in Eq. \eqref{eq:UCBdefs}) based on whether the corresponding optimal budget w.r.t. the Lagrangian at the context is taken at the $n$th ``turning point'' (see Figure \ref{fig:lagr} of illustration). Then, for each context group $n$ by defining  $k':=\max\{k: a_{k} < r_{j,n-1}\}$ to be the arm closest to and less than $r_{j,n-1}$, we have
\begin{align}
\label{eq:nbadslope}
\begin{aligned}
   & \sum_{t\in[T]:\bm{c}_{t} \in \mathcal{C}_{n}/\mathcal{C}(\underline{\sigma})} ~ \sum_{k\in[K]:a_{k} < \min \mathcal{U}_{j}^{-}(\bm{c}_{t})} ~  \Delta_{k}(\bm{c}_{t}) \mathbb{I}\{\rho_{j,t} = a_{k}, N_{k,t} \leq m_{k}(\bm{c}_{t})\} \\
    ~\overset{(i)}{=}~ & \sum_{t\in[T]:\bm{c}_{t} \in \mathcal{C}_{n}/\mathcal{C}(\underline{\sigma})} ~ \sum_{k\in[K]:a_{k} < r_{j,n-1}} ~  \Delta_{k}(\bm{c}_{t}) \mathbb{I}\{\rho_{j,t} = a_{k}, N_{k,t} \leq m_{k}(\bm{c}_{t})\} \\
    ~=~ &  \sum_{t\in[T]} \sum_{\bm{c} \in \mathcal{C}_{n}/\mathcal{C}(\underline{\sigma})} ~ \sum_{k\in[K]:a_{k} < r_{j,n-1}} \Delta_{k}(\bm{c}) \mathbb{I}\{\bm{c}_{t} = \bm{c}, \rho_{j,t} = a_{k}, N_{k,t} \leq m_{k}(\bm{c})\}\\
    ~\overset{(ii)}{\leq}~ &  \sum_{t\in[T]} \sum_{\bm{c} \in \mathcal{C}_{n}/\mathcal{C}(\underline{\sigma})} ~ \left(\Delta_{k'}(\bm{c}) \mathbb{I}\{\bm{c}_{t}=\bm{c}\} + \sum_{k\in[K]:a_{k} < r_{j,n-1}-\delta} \Delta_{k}(\bm{c}) \mathbb{I}\{\bm{c}_{t} = \bm{c}, \rho_{j,t} = a_{k}, N_{k,t} \leq m_{k}(\bm{c})\}\right)\\
   ~\overset{(iii)}{\leq}~ & \left((1+C_{F})s_{j,n-1} \delta + \rho \underline{\sigma}\right) T + \sum_{k\in[K]:a_{k} < r_{j,n-1}-\delta} ~ \sum_{\bm{c} \in \mathcal{C}_{n}/\mathcal{C}(\underline{\sigma})} \Delta_{k}(\bm{c}) Y_{k}(\bm{c})
\end{aligned}
\end{align}
where in the final equality we defined $ Y_{k}(\bm{c})=  \sum_{t\in [T]} \mathbb{I}\{\bm{c}_{t} = \bm{c}, \rho_{j,t} = a_{k}, N_{k,t} \leq m_{k}(\bm{c})\}$. In (i) we used the fact that the left end of the left adjacent region, i.e. $
\min \mathcal{U}_{j}^{-}(\bm{c}_{t})$ is exactly $r_{j,n-1}$ because for context $\bm{c}_{t} \in \mathcal{C}_{n}$ the optimal budget $\arg\max_{\rho_{j}\in[0,\rho]} \mathcal{L}_{j}(\rho_{j},\bm{c}_{t})$ is at the $n$th turning point; in (ii) we used the definition $k':=\max\{k: a_{k} < r_{j,n-1}\}$; (iii) follows the fact that under a context $\bm{c} \in \mathcal{C}_{n}/\mathcal{C}(\underline{\sigma})$,  we have $\arg\max_{\rho_{j}\in [0,\rho]} \mathcal{L}_{j}(\rho_{j}, \bm{c}) = r_{j,n}$ so
\begin{align*}
    \Delta_{k'}(\bm{c}) ~=~& \mathcal{L}_{j}(r_{j,n},\bm{c}) -   \mathcal{L}_{j}(r_{j,n-1},\bm{c}) + \mathcal{L}_{j}(r_{j,n-1},\bm{c}) - \mathcal{L}_{j}(a_{k'},\bm{c})\\
     ~=~&  \sigma_{j}^{-}(\bm{c}) (r_{j,n} - r_{j,n-1}) + \sigma_{j,n-1}(\bm{c})(r_{j,n-1} - a_{k'}) \\
     ~\overset{(iv)}{\leq}~ & \underline{\sigma}\rho + \sigma_{j,n-1}(\bm{c}) \delta\\
       ~\overset{(v)}{\leq}~ & \underline{\sigma}\rho + (1+C_{F})s_{j,n-1}\delta
     \,,
\end{align*}
where in (iv) we used the fact that $\bm{c} \in \mathcal{C}_{n}/\mathcal{C}(\underline{\sigma})$ so $ \sigma_{j}^{-}(\bm{c}) \leq \underline{\sigma}$, as well as $k'$ lies on the line segment between points $r_{j,n-2}$ and $r_{j,n-1}$ since $\delta < \min_{n' \in [S_{j}]} r_{j,n'} - r_{j,n'-1}$; in (v) we recall $\sigma_{j,n-1}(\bm{c}) = (1+\lambda)s_{j,n-1} - (\mu + \gamma\lambda) \leq (1+C_{F})s_{j,n-1}$ where $C_{F}$ is defined in Lemma \ref{lem:dualvarub}.

We now bound $\sum_{\bm{c} \in \mathcal{C}_{n}/\mathcal{C}(\underline{\sigma})} \Delta_{k}(\bm{c}) Y_{k}(\bm{c})$ in Eq. \eqref{eq:nbadslope}.  It is easy to see the following inequality for any sequence of context $\bm{c}_{(1)}, \dots , \bm{c}_{(\ell)} \in \{\bm{c}_{t}\}_{t\in[T]}$:
\begin{align}
\label{eq:numbadarms}
    Y_{k}(\bm{c}_{(1)}) + \dots +  Y_{k}(\bm{c}_{(\ell)}) \leq \max_{\ell'=1\dots \ell} m_{k}(\bm{c}_{(\ell')})\,. 
\end{align}
This is because 
\begin{align*}
   \sum_{\ell'\in [\ell]}  Y_{k}(\bm{c}_{(\ell')}) ~=~ &\sum_{t\in[T]} \sum_{\ell'\in [\ell]}  \mathbb{I}\{\bm{c}_{t} =\bm{c}_{(\ell')}, \rho_{j,t} = a_{k}, N_{k,t} \leq m_{k}(\bm{c}_{(\ell')})\}\\
     ~\leq~& \sum_{t\in[T]}  \sum_{\ell'\in [\ell]}  \mathbb{I}\{\bm{c}_{t}= \bm{c}_{(\ell')}, \rho_{j,t} = a_{k}, N_{k,t} \leq \max_{\ell'=1\dots \ell} m_{k}(\bm{c}_{(\ell')})\}\\
      ~=~& \sum_{t\in[T]} \mathbb{I}\{\bm{c}_{t} \in \{\bm{c}_{(\ell')}\}_{\ell'\in[\ell]}, \rho_{j,t} = a_{k}, N_{k,t} \leq \max_{\ell'=1\dots \ell} m_{k}(\bm{c}_{(\ell')})\}\\
        ~\leq~& \max_{\ell'=1\dots \ell} m_{k}(\bm{c}_{(\ell')})\,.
\end{align*}
For simplicity denote $L = |\mathcal{C}_{n}/\mathcal{C}(\underline{\sigma})|$, and order contexts in $\bm{c} \in \mathcal{C}_{n}/\mathcal{C}(\underline{\sigma})$ as $\{\bm{c}_{(\ell)}\}_{\ell \in [L]}$ s.t. 
$\Delta_{k}(\bm{c}_{(1)}) > \Delta_{k}(\bm{c}_{(2)}) > \dots >\Delta_{k}(\bm{c}_{(L)})$, or equivalently 
$m_{k}(\bm{c}_{(1)}) < m_{k}(\bm{c}_{(2)}) < \dots  < m_{k}(\bm{c}_{(L)})$ according to Eq.\eqref{eq:UCBdefs}. Then multiplying Eq. \eqref{eq:numbadarms} by 
by $\Delta_{k}(\bm{c}_{(\ell)}) - \Delta_{k}(\bm{c}_{(\ell+1)})$ (which is strictly positive due to the ordering of contexts), and summing $\ell = 1 \dots L$  we get 
\begin{align}
\begin{aligned}
\label{eq:slope}
    &\sum_{\bm{c} \in \mathcal{C}_{n}/\mathcal{C}(\underline{\sigma})} \Delta_{k}(\bm{c}) Y_{k}(\bm{c}) = \sum_{\ell \in [L]} \Delta_{k}(\bm{c}_{(\ell)}) Y_{k}(\bm{c}_{(\ell)})\\
    ~\leq~&  \sum_{\ell \in [L]} m_{k}(\bm{c}_{(\ell)}) \left( \Delta_{k}(\bm{c}_{(\ell)}) -  \Delta_{k}(\bm{c}_{(\ell+1)})\right)\\
    ~\overset{(i)}{=}~& 8\log(T)\sum_{\ell \in [L-1]} \frac{ \Delta_{k}(\bm{c}_{(\ell)}) -  \Delta_{k}(\bm{c}_{(\ell+1)})}{\Delta_{k}^{2}(\bm{c}_{(\ell)})}\\
    ~\overset{(ii)}{\leq}~& 8\log(T) \int_{z = \Delta_{k}(\bm{c}_{(L)})}\frac{dz}{z^{2}}\\
     ~=~& \frac{8\log(T)}{\Delta_{k}(\bm{c}_{(L)})} ~\overset{(iii)}{=} \frac{8\log(T)}{\min_{\bm{c} \in \mathcal{C}_{n}/\mathcal{C}(\underline{\sigma})} \Delta_{k}(\bm{c})}\,.
     \end{aligned}
\end{align}
Here  (i) follows from the definition in Eq. \eqref{eq:UCBdefs} where we defined $m_{k}(\bm{c}) = \frac{8\log(T)}{\Delta_{k}^{2}(\bm{c})}$; both (ii) and (iii) follow from the ordering of contexts so that $\Delta_{k}(\bm{c}_{(1)}) > \Delta_{k}(\bm{c}_{(2)}) > \dots >\Delta_{k}(\bm{c}_{(L)})$. 
Note that for any $\bm{c} \in \mathcal{C}_{n}/\mathcal{C}(\underline{\sigma})$ and arm $k$ such that $a_{k} < r_{j,n-1}$, we have
\begin{align}
\label{eq:slope2}
\begin{aligned}
  \Delta_{k}(\bm{c}) ~=~& \mathcal{L}_{j}(r_{j,n},\bm{c}) -   \mathcal{L}_{j}(r_{j,n-1},\bm{c}) + \mathcal{L}_{j}(r_{j,n-1},\bm{c}) - \mathcal{L}_{j}(a_{k},\bm{c})\\
  ~>~&  \mathcal{L}_{j}(r_{j,n-1},\bm{c}) - \mathcal{L}_{j}(a_{k},\bm{c})\\
  ~\overset{(i)}{\geq} ~&  \sigma_{j,n-1}(\bm{c}) (r_{j,n-1} - a_{k})\\
   ~\overset{(ii)}{\geq} ~& \left( \sigma_{j,n-1}(\bm{c}) -  \sigma_{j,n}(\bm{c})\right) (r_{j,n-1} - a_{k})\\
    ~\overset{(iii)}{=} ~& (1+\lambda)\left(s_{j,n-1}-s_{j,n}\right) (r_{j,n-1} - a_{k})\\
     ~> ~& \left(s_{j,n-1}-s_{j,n}\right) (r_{j,n-1} - a_{k})
\end{aligned}
\end{align}
where in (i) we recall the slope $\sigma_{j,n-1}(\bm{c})$ is defined in Lemma \ref{lem:Vstruct} and further (i) follows from concavity of $ \mathcal{L}_{j}(\rho{j},\bm{c})$ in the first argument $\rho_{j}$; in (ii) we used the fact that $ \sigma_{j,n}(\bm{c})\geq 0$ since the optimal budget 
$\arg\max_{\rho_{j}\in [0,\rho]} \mathcal{L}_{j}(\rho_{j}, \bm{c})$ is taken at the $n$th turning point, and is the largest turning point whose left slope is non-negative from  Lemma \ref{lem:Vstruct}; (iii) follows from the definition $\sigma_{j,n'}(\bm{c}) = (1+\lambda)s_{j,n'} - (\mu+\gamma\lambda)$ for any $n'$. 


Finally combining Eq. \eqref{eq:nbadslope}, \eqref{eq:slope} and \eqref{eq:slope2}, and summing over $n = 1 \dots S_{j}$ we get
\begin{align}
\begin{aligned}
\label{eq:X1secbound}
  & \sum_{t\in[T]:\bm{c}_{t} \notin \mathcal{C}(\underline{\sigma})} ~ \sum_{k\in[K]:a_{k} < \min \mathcal{U}_{j}^{-}(\bm{c}_{t})} ~  \Delta_{k}(\bm{c}_{t}) \mathbb{I}\{\rho_{j,t} = a_{k}, N_{k,t} \leq m_{k}(\bm{c}_{t})\}   \\
 ~=~& \sum_{n\in [S_{j}]} \sum_{t\in[T]:\bm{c}_{t} \in \mathcal{C}_{n}/\mathcal{C}(\underline{\sigma})} ~ \sum_{k\in[K]:a_{k} < \min \mathcal{U}_{j}^{-}(\bm{c}_{t})} ~  \Delta_{k}(\bm{c}_{t}) \mathbb{I}\{\rho_{j,t} = a_{k}, N_{k,t} \leq m_{k}(\bm{c}_{t})\} \\
 ~\leq~& \sum_{n\in [S_{j}]}\left((1+C_{F})s_{j,n-1} \delta + \rho \underline{\sigma}\right) T  + \sum_{n\in [S_{j}]}~\sum_{k\in[K]:a_{k} < r_{j,n-1}-\delta} \frac{8\log(T)}{\left(s_{j,n-1}-s_{j,n}\right) (r_{j,n-1} - a_{k})}\\
 ~\overset{(i)}{\leq}~& \sum_{n\in [S_{j}]}\left((1+C_{F})s_{j,n-1} \delta + \rho \underline{\sigma}\right) T  + \sum_{n\in [S_{j}]}~\sum_{\ell=1}^{K} \frac{8\log(T)}{\left(s_{j,n-1}-s_{j,n}\right) \ell \delta}\\
 ~\leq~& \sum_{n\in [S_{j}]}\left((1+C_{F})s_{j,n-1} \delta + \rho \underline{\sigma}\right) T  + \frac{8\log(T)\log(K)}{\delta} \sum_{n\in [S_{j}]}~\frac{1}{\left(s_{j,n-1}-s_{j,n}\right)}\\
 ~=~& \mathcal{O}(\delta T + \underline{\sigma}T + \frac{1}{\delta})\,.
 \end{aligned}
\end{align}
Note that (i) follows because for all $a_{k} < r_{j,n-1} - \delta$, the $a_{k}$'s distances from $r_{j,n-1}$ are at least $\delta, 3\delta, 2\delta \dots $. 

(3) The cases where arm $a_k \in \mathcal{U}_{j}^{+}(\bm{c}_{t})$ and $a_{k} > \max \mathcal{U}_{j}^{+}(\bm{c}_{t})$
are symmetric to $a_k \in \mathcal{U}_{j}^{-}(\bm{c}_{t})$ and $a_{k} < \min \mathcal{U}_{j}^{+}(\bm{c}_{t})$, respectively, and we omit from this paper. 

Therefore, combining Eq. \eqref{eq:X1near} and \eqref{eq:X1secbound} we  can conclude 
\begin{align}
    X_{1} \leq  \mathcal{O}(\delta T + \underline{\sigma}T + \frac{1}{\delta})
\end{align}



\subsubsection{Bounding $X_{2}$.}
\label{subsubsec:X2}
We first rewrite $X_{2}$ as followed
\begin{align}
\label{eq:X2all}
\begin{aligned}
    X_{2} ~=~& \sum_{t\in[T]:\bm{c}_{t} \in \mathcal{C}(\underline{\sigma})} \sum_{k\in [K]} \Delta_{k}(\bm{c}_{t}) \mathbb{I}\{\rho_{j,t} = a_{k}, N_{k,t} \leq m_{k}(\bm{c}_{t})\} \\
     ~=~& \sum_{t\in[T]}\sum_{n\in[S_{j}]}\sum_{k\in [K]}\sum_{\bm{c} \in \mathcal{C}_{n}\cap \mathcal{C}(\underline{\sigma})}  \Delta_{k}(\bm{c}) \mathbb{I}\{\bm{c}_{t}=\bm{c}, \rho_{j,t} = a_{k}, N_{k,t} \leq m_{k}(\bm{c})\} \\
     ~\overset{(i)}{=}~& 
   \sum_{n\in[S_{j}]}\sum_{k\in [K]}\sum_{\bm{c} \in \mathcal{C}_{n}\cap \mathcal{C}(\underline{\sigma})} \Delta_{k}(\bm{c}) Y_{k}(\bm{c})\\
    ~\overset{(ii)}{=}~&  T \delta (1+C_{F}) \sum_{n\in[S_{j}]} \left(s_{j,n} + s_{j,n+1}\right) + 
   \sum_{n\in[S_{j}]}~\sum_{k\in [K]/\{k_{n}^{-},k_{n}^{+}\}}~\sum_{\bm{c} \in \mathcal{C}_{n}\cap \mathcal{C}(\underline{\sigma})} \Delta_{k}(\bm{c}) Y_{k}(\bm{c})\,.
    \end{aligned}
\end{align}
where in (i) we define 
$Y_{k}(\bm{c}) =  \sum_{t\in[T]}\mathbb{I}\{\bm{c}_{t} = \bm{c}, \rho_{j,t} = a_{k}, N_{k,t} \leq m_{k}(\bm{c})\}$;  in (ii) for any context $\bm{c} \in \mathcal{C}_{n}\cap \mathcal{C}(\underline{\sigma})$, the optimal budget $\arg\max_{\rho_{j}\in [0,\rho]} \mathcal{L}_{j}(\rho_{j}, \bm{c}) = r_{j,n}$ is taken at the $n$th turning point per the definition of $\mathcal{C}_{n}$ in Eq. \eqref{eq:UCBdefs}, and further we defined $k_{n}^{-}:= \max\{k\in [K]: a_{k} < r_{j,n}\}$ to be the arm closest to and no greater than  $r_{j,n}$, whereas  $k_{n}^{+}:= \min\{k\in [K]: a_{k} > r_{j,n}\}$ to be the arm closest to and no less than  $r_{j,n}$. Thus for small enough $\delta < \min_{n'\in[S_{j}]}r_{j,n'} - r_{j,n'-1} $,  $k_{n}^{-}$ lies on the line segment between $r_{j,n-1}$ and $r_{j,n}$, so $\Delta_{k_{n}^{-}}(\bm{c}) = \sigma_{j}^{-}(\bm{c})(r_{j,n} - a_{k_{n}^{-}}) \leq \sigma_{j}^{-}(\bm{c}) \delta \leq (1+C_{F})s_{j,n-1} \delta$, where in the final inequality follows from the definition of  $\sigma_{j}^{-}(\bm{c}) = \sigma_{j,n}(\bm{c}) = (1+\lambda)s_{j,n} - (\mu + \gamma\lambda) \leq (1+\lambda)s_{j,n} \leq (1+C_{F})s_{j,n}$ where $C_{F}$ is defined in Eq. \eqref{lem:dualvarub}. A similar bound holds for 
$\Delta_{k_{n}^{+}}(\bm{c})$. 


Then, following the same logic as Eq. \eqref{eq:numbadarms}, \eqref{eq:slope}, \eqref{eq:slope2} in Section \ref{subsubsec:X1} where we bound $X_{1}$, we can bound $\sum_{\bm{c} \in \mathcal{C}_{n}\cap \mathcal{C}(\underline{\sigma})} \Delta_{k}(\bm{c}) Y_{k}(\bm{c})$ as followed:
\begin{align}
\label{eq:X2slope}
    \sum_{\bm{c} \in \mathcal{C}_{n}\cap \mathcal{C}(\underline{\sigma})} \Delta_{k}(\bm{c}) Y_{k}(\bm{c}) \leq \frac{8\log(T)}{\min_{\bm{c} \in \mathcal{C}_{n}\cap \mathcal{C}(\underline{\sigma})}\Delta_{k}(\bm{c})}\,.
\end{align}

Now, the set $k\in [K]/\{k_{n}^{-},k_{n}^{+}\}$ in Eq. \eqref{eq:X2all} can be split into two sets, namely $\{k\in [K]: a_{k} < r_{j,n} - \delta\}$ and $\{k\in [K]: a_{k} > r_{j,n} + \delta\}$ due to the definitions $k_{n}^{-}:= \max\{k\in [K]: a_{k} < r_{j,n}\}$ and $k_{n}^{+}:= \min\{k\in [K]: a_{k} > r_{j,n}\}$. Therefore, for any $k$ s.t. $ a_{k} < r_{j,n} - \delta$ and any $\bm{c} \in  \mathcal{C}_{n}\cap \mathcal{C}(\underline{\sigma})$, 
\begin{align*}
    \Delta_{k}(\bm{c}) = \mathcal{L}_{j}(r_{j,n},\bm{c}) - \mathcal{L}_{j}(a_{k},\bm{c}) \geq \sigma_{j}^{-}(\bm{c}) (r_{j,n} - a_{k}) \geq \underline{\sigma}  (r_{j,n} - a_{k}),,
\end{align*}
where the final inequality follows from the definition of $\mathcal{C}(\underline{\sigma})$ in Eq. \eqref{eq:UCBdefs} such that $ \sigma_{j}^{-}(\bm{c}) \geq \underline(\sigma) $ for $\bm{c} \mathcal{C}(\underline{\sigma})$. Hence combining this with Eq. \eqref{eq:X2slope} we have
\begin{align}
\label{eq:X2last}
    \sum_{k\in [K]:a_{k} < r_{j,n}- \delta}~ \sum_{\bm{c} \in \mathcal{C}_{n}\cap \mathcal{C}(\underline{\sigma})} \Delta_{k}(\bm{c}) Y_{k}(\bm{c}) \leq  \sum_{k\in [K]:a_{k} < r_{j,n}- \delta}\frac{8\log(T)}{ \underline{\sigma}  (r_{j,n} - a_{k})} \overset{(i)}{\leq} \sum_{\ell=1}^{K}\frac{8\log(T)}{ \underline{\sigma} \ell \delta} \leq\frac{8\log(T)\log(K)}{\underline{\sigma} \delta} 
\end{align}
where (i) follows because for all $a_{k} < r_{j,n} - \delta$, the $a_{k}$'s distances from $r_{j,n-1}$ are at least $\delta, 3\delta, 2\delta \dots $. Symmetrically, we can show an identical bound for the set  $\{k\in [K]: a_{k} > r_{j,n} + \delta\}$. Hence, combining Eq. \eqref{eq:X2all} and \eqref{eq:X2last} we can conclude
\begin{align}
    X_{2}\leq \mathcal{O}\left(\delta T + \frac{1}{\delta \underline{\sigma}}\right).
\end{align}

\subsubsection{Bounding $X_{3}$.}
\label{subsubsec:X3}
For any arm $k \in [K]$, denote $ R_{t}(a_{k}) =\Bar{V}_{t}(a_{k})  - \frac{\lambda_{t} \gamma + \mu_{t} }{1+\lambda_{t}}a_{k}$. Also, define $\Bar{\mathcal{L}} = \left(1+\gamma \right)\rhoC_{F} + (1+C_{F})\Bar{V}$, where $C_{F}$ is specified in Lemma \ref{lem:dualvarub}, and $a_{t}^{*} = \arg\max_{a_{k'}\in \mathcal{A}(\delta)} \mathcal{L}_{j}(a_{k'}, \bm{c}_{t})$ for convenience. Then we bound $X_{3}$ as followed 
\begin{align}
\label{eq:X3first}
\begin{aligned}
   X_{3} ~=~&  \sum_{t\in[T]} \mathbb{E}\left[\Delta_{k}(\bm{c}) \mathbb{I}\{ \rho_{j,t} = a_{k}, N_{k,t} > m_{k}(\bm{c})\}\right] \\
    ~\overset{(i)}{\leq}~& \Bar{\mathcal{L}}\cdot \mathbb{P}\left(\rho_{j,t} = a_{k}, N_{k,t} > m_{k}(\bm{c}_{t})\right)\\
    ~\leq~& \Bar{\mathcal{L}} \cdot  \sum_{t\in[T]} \mathbb{P}\left(R_{t}(a_{k}) +  \texttt{UCB}_{j,t}(a_{k}) \geq R_{t}(a_{t}^{*})+  \texttt{UCB}_{j,t}(a_{t}^{*}), N_{k,t} > m_{k}(\bm{c}_{t})\right)\,,
\end{aligned}
\end{align}
where (i) follows from Lemma \ref{lem:dualvarub} such that $  -\left(1+\gamma \right)\rhoC_{F}
 ~\leq~ \mathcal{L}_{j}(\rho_{j},\bm{c}_{t})~\leq~(1+C_{F})\Bar{V}$ for all $j \in [M]$, $t\ in [T]$, $\rho_{j} \in [0,\rho]$ and $\bm{c}_{t}$.

Now let $\Bar{R}_{n}(a_{k})$ denote the average conversion of arm $k$ over its first $n$ pulls, i.e. letting $\tau = \min\{t\in [T]:N_{k,t} = n\}$ be the period during which arm $k$ is pulled for the $n$th time, then 
$\Bar{R}_{n}(a_{k}) =\Bar{V}_{j,\tau}(a_{k})$, where $\Bar{V}_{j,\tau}$ is defined in Algorithm \ref{alg:omducb}. Hence, we continue as follwoed 
\begin{align}
\label{eq:X3sec}
\begin{aligned}
 &\mathbb{P}\left(R_{t}(a_{k}) +  \texttt{UCB}_{j,t}(a_{k}) \geq R_{t}(a_{t}^{*})+  \texttt{UCB}_{j,t}(a_{t}^{*}), N_{k,t} > m_{k}(\bm{c}_{t})\right)\\
    ~\leq~&  \mathbb{P}\left(\max_{m_{k}(\bm{c}_{t}) < n \leq t} \Bar{R}_{n}(a_{k}) + \texttt{UCB}_{t}(a_{k}) - \frac{\lambda_{t} \gamma + \mu_{t} }{1+\lambda_{t}}a_{k} \geq \min_{1\leq n'\leq t}\Bar{R}_{n}(a_{t}^{*}) + \texttt{UCB}_{t}(a_{t}^{*}) - \frac{\lambda_{t} \gamma + \mu_{t} }{1+\lambda_{t}}a_{t}^{*}
    \right)\\
    ~\leq~& \sum_{ n = \left\lceil m_{k}(\bm{c}_{t}) \right\rceil+1}^{t}\sum_{n'=1}^{t}\mathbb{P}\left(\Bar{R}_{n}(a_{k}) + \texttt{UCB}_{n}(a_{k}) - \frac{\lambda_{t} \gamma + \mu_{t} }{1+\lambda_{t}}a_{k} > \Bar{R}_{n'}(a_{t}^{*}) + \texttt{UCB}_{n'}(a_{t}^{*}) - \frac{\lambda_{t} \gamma + \mu_{t} }{1+\lambda_{t}}a_{t}^{*}
    \right)
    \end{aligned}
\end{align}
Now, when the event $\left\{\Bar{R}_{n}(a_{k}) + \texttt{UCB}_{n}(a_{k}) - \frac{\lambda_{t} \gamma + \mu_{t} }{1+\lambda_{t}}a_{k} > \Bar{R}_{n}(a_{t}^{*}) + \texttt{UCB}_{n'}(a_{t}^{*}) - \frac{\lambda_{t} \gamma + \mu_{t} }{1+\lambda_{t}}a_{t}^{*}
    \right\}$ occurs, it is easy to see that one of the following events must also occur:
\begin{align}
\label{eq:event}
\begin{aligned}
    & \mathcal{G}_{1,n} =\left\{  \Bar{R}_{n}(a_{k})  \geq V(a_{k}) + \texttt{UCB}_{n}(a_{k})\right\} \\
    & \mathcal{G}_{2,n'} =\left\{  \Bar{R}_{n'}(a_{t}^{*})  \leq V(a_{t}^{*}) -\texttt{UCB}_{n}(a_{t}^{*}) \right\}\\
    & \mathcal{G}_{3} =\left\{   V(a_{t}^{*}) -  \frac{\lambda_{t} \gamma + \mu_{t} }{1+\lambda_{t}}a_{t}^{*}~<~ V(a_{k}) - \frac{\lambda_{t} \gamma + \mu_{t} }{1+\lambda_{t}}a_{k}  + 2 \cdot \texttt{UCB}_{n}(a_{k}) \right\}\\
    \end{aligned}
\end{align}
Note that for $n > m_{k}(\bm{c}_{t})$, we have 
$\texttt{UCB}_{n}(a_{k}) = \sqrt{\frac{2\log(T)}{n}} < \sqrt{\frac{2\log(T)}{m_{k}(\bm{c}_{t})}} = \frac{\Delta_{k}(\bm{c}_{t})}{2}$ since we defined $m_{k}(\bm{c}) = \frac{8\log(T)}{\Delta_{k}^{2}(\bm{c})}$ in Eq. \eqref{eq:UCBdefs}. Therefore
\begin{align*}
    V(a_{k}) - \frac{\lambda_{t} \gamma + \mu_{t}}{1+\lambda_{t}}a_{k} + 2 \cdot \texttt{UCB}_{n}(a_{k}) <  \underbrace{V(a_{k}) - \frac{\lambda_{t} \gamma + \mu_{t}}{1+\lambda_{t}}a_{k}}_{=\mathcal{L}(a_{k},\bm{c}_{t})} + \Delta_{k}(\bm{c}_{t}) ~\overset{(i)}{=}~ \underbrace{V(a_{t}^{*}) -  \frac{\lambda_{t} \gamma + \mu_{t} }{1+\lambda_{t}}a_{t}^{*}}_{=\mathcal{L}(a_{t}^{*},\bm{c}_{t}) = \max_{a\in \mathcal{A}(\delta)}\mathcal{L}(a,\bm{c}_{t})}
\end{align*}
where (i) follows from the definition of $\Delta_{k}(\bm{c}) = \max_{a\in \mathcal{A}(\delta)}\mathcal{L}(a,\bm{c}) - \mathcal{L}(a_{k},\bm{c})$ in Eq. \eqref{eq:UCBdefs} for any context $\bm{c}$. This implies that event $\mathcal{G}_{3}$ in Eq. \eqref{eq:event} cannot hold for $n >  m_{k}(\bm{c}_{t})$. Therefore
\begin{align}
\label{eq:X3third}
    \mathbb{P}\left(\Bar{R}_{n}(a_{k}) + \texttt{UCB}_{n}(a_{k}) - \frac{\lambda_{t} \gamma + \mu_{t} }{1+\lambda_{t}}a_{k} > \Bar{R}_{n'}(a_{t}^{*}) + \texttt{UCB}_{n'}(a_{t}^{*}) - \frac{\lambda_{t} \gamma + \mu_{t} }{1+\lambda_{t}}a_{t}^{*}
    \right)~\leq~\mathbb{P}\left(\mathcal{G}_{1,n} \cup \mathcal{G}_{2,n'}
    \right)
\end{align}

From standard UCB analysis and the Azuma Hoeffding's inequality, we have $\mathbb{P}(\mathcal{G}_{1,n} )\leq \frac{\Bar{V}}{T^{4}}$ and  $\mathbb{P}(\mathcal{G}_{2,n'} )\leq \frac{\Bar{V}}{T^{4}}$. Hence combining Eq. \eqref{eq:X3first} \eqref{eq:X3sec}, \eqref{eq:X3third} we can conclude

\begin{align}
\begin{aligned}
 X_{3} ~\leq~& \sum_{t\in[T]}\sum_{n = \left\lceil m_{k}(\bm{c}_{t}) \right\rceil+1}^{t}\sum_{n'=1}^{t}\left(\mathbb{P}\left(\mathcal{G}_{1,n}\right) + \mathbb{P}\left(\mathcal{G}_{2,n'}
    \right)\right)\\
    ~\leq~& \sum_{t\in[T]}\ \sum_{n = \left\lceil m_{k}(\bm{c}_{t}) \right\rceil+1}^{t}\sum_{n'=1}^{t} \frac{2\Bar{V}}{T^{4}}\\
     ~\leq~&  \frac{2\Bar{V}}{T}
    \end{aligned}
\end{align}
\halmos


\subsection{Proof for Lemma \ref{lem:boundcs}}
\label{pf:lem:boundcs}
Recall $g_{1,t} = \sum_{j\in [M]}\left(V_{j,t}(\rho_{j,t}) - \gamma \rho_{j,t}\right)$ and $g_{2,t} = \rho - \sum_{j\in [M]}\rho_{j,t}$ defined in Algorithm \ref{alg:omducb}.

Now from lemma \ref{lem:omdtelescope}, we have for any $\lambda\geq 0$ and $\mu\geq 0$,
\begin{align*}
\begin{aligned}
    & \sum_{t} \left(\lambda_{t} - \lambda\right) g_{1,t} \leq  \frac{\eta M^{2}\Bar{V}^{2}}{2} \cdot T +  \frac{1}{2\eta}(\lambda- \lambda_{1})^{2}\\
    & \sum_{t} \left(\mu_{t} - \mu\right) g_{2,t} \leq  \frac{\eta \rho^{2}}{2} \cdot T +  \frac{1}{2\eta}(\mu- \mu_{1})^{2}\,.
  \end{aligned}  
\end{align*}
By taking $\lambda = \mu = 0$ and recalling $\lambda_{1} = \mu_{1} = 0$, we get the desired bound in the statement of the lemma. 
\halmos


\subsection{Proof for Theorem \ref{thm:regret}}
\label{pf:thm:regret}
Starting from Proposition \ref{prop:ubbenchlag}, we get
\begin{align}
\begin{aligned}
    T\cdot \textsc{GL-OPT} - \mathbb{E}\left[\sum_{t\in [T]}\sum_{j\in[M]}V_{j}(\rho_{j,t})\right]~\leq~& M\Bar{V}K +  \sum_{j\in [M]} \sum_{t > K} 
     \mathbb{E}\left[\mathcal{L}_{j}(\rho_{j}^{*}(t), \bm{c}_{t}) - \mathcal{L}_{j}(\rho_{j,t}, \bm{c}_{t})\right] +  \sum_{t > K}\left(\lambda_{t}g_{1,t} + \mu_{t}g_{2,t}\right)\\
     ~\overset{(i)}{\leq}~&M\Bar{V}K  + \mathcal{O}\left(\underline{\sigma}T + \delta T + \frac{1}{\underline{\sigma} \delta}\right) + \mathcal{O}\left(\eta T + \frac{1}{\eta}\right)
     \end{aligned}
\end{align}
where in (i) we applied Lemma \ref{lem:ucbregret} and \ref{lem:boundcs}. Taking $\eta = 1/\sqrt{T}$, $\delta = \underline{\sigma} = T^{-1/3}$ (i.e. $K = t^{1/3}$ yields $ T\cdot \textsc{GL-OPT} - \mathbb{E}\left[\sum_{t\in [T]}\sum_{j\in[M]}V_{j}(\rho_{j,t})\right] \leq \mathcal{O}(T^{2/3})$. According to Lemma \ref{lem:Vstruct}, $V_{j}(\rho_{j})$ is concave for all $j \in [M]$, so
\begin{align}
\begin{aligned}
    \mathcal{O}(T^{-1/3})~\geq~& \textsc{GL-OPT} -  \frac{1}{T}\sum_{t\in [T]} \mathbb{E}\left[\sum_{j\in[M]}V_{j}(\rho_{j,t})\right] \\
    ~\geq~&  \textsc{GL-OPT}  - \mathbb{E}\left[ \sum_{j\in[M]}  V_{j}\left(\frac{1}{T}\sum_{t\in [T]} \rho_{j,t}\right)\right]\\
     ~\geq~&  \textsc{GL-OPT}  - \mathbb{E}\left[ \sum_{j\in[M]}V_{j}(\Bar{\rho}_{T,j})\right]
     \end{aligned}
\end{align}
where in the final equality we used the definition $\Bar{\bm{\rho}}_{T}$ as defined in Algorithm \ref{alg:omducb}. 


On the other hand, Lemma \ref{lem:boundcs} shows 
\begin{align}
\begin{aligned}
    -\mathcal{O}(1/\sqrt{T}) ~\leq~& \frac{1}{T}\sum_{t\in[T]} \mathbb{E}\left[g_{1,t}\right]\\
    ~=~& \frac{1}{T}\sum_{t\in[T]}\sum_{j\in [M]}\mathbb{E}\left[\left(V_{j}(\rho_{j,t};\bm{z}_{t}) - \gamma \rho_{j,t}\right)\right]\\
     ~=~& \frac{1}{T}\sum_{t\in[T]}\sum_{j\in [M]}\mathbb{E}\left[\left(V_{j}(\rho_{j,t}) - \gamma \rho_{j,t}\right)\right]\\
    ~\overset{(i)}{\leq}~&\sum_{j\in [M]}\left(V_{j}\left( \frac{1}{T}\sum_{t\in[T]}\rho_{j,t}\right) - \gamma \cdot \frac{1}{T}\sum_{t\in[T]}\rho_{j,t}\right)\\
     ~=~&\sum_{j\in [M]}\left(V_{j}\left(\Bar{\rho}_{T,j}\right) - \gamma \Bar{\rho}_{T,j}\right)\,.
     \end{aligned}
\end{align}
where in (i) we again applied concavity of $V_{j}(\rho_{j})$. We omit the analysis for the budget constraint as it is similar to the above.
\halmos




\subsection{Additional Results for Section \ref{sec:online}}
\label{app:onlineadd}

Assumption \ref{ass:feas} ensures that for any realization $\bm{z}=(\bm{v},\bm{d})$ there must be some per-channel budget allocation that allows the advertiser to satisfy her ROI constraints as illustrated in the following proposition
\begin{proposition}[Slater's condition]
\label{prop:slater}
Assume Assumption \ref{ass:feas} holds. Let  $\bm{z}=(\bm{v},\bm{d}) = (\bm{v}_{j},\bm{d}_{j})_{j\in[M]} \in F_{1}\times \dots F_{M}$ be any realization of values and costs across all channels, then there exists some per-channel budget allocation $\rho_{(\bm{z})} \in [0,\rho]^{M}$ s.t. $ \sum_{j \in M} V_{j}(\rho_{(\bm{z}),j};\bm{z} ) > \gamma \sum_{j \in M} \rho_{(\bm{z}),j}$ and $ \sum_{j\in [M]}\rho_{(\bm{z}),j} < \rho$.
\end{proposition}
\textit{Proof.} Under Assumption \ref{ass:feas}, it is easy to see for any realization of value-cost pairs  $\bm{z} = (\bm{v}_{j},\bm{d}_{j})_{j\in[M]}$ there always exists a channel $j\in [M]$ in which there is an auction $n\in [m_{j}]$ whose value-to-cost ration is at least $\gamma$, i.e. $v_{j,n} >  \gamma d_{j,n}$. Since we assumed the ordering $\frac{v_{j,1}}{d_{j,1}} > \frac{v_{j,2}}{d_{j,2}} > \dots  > \frac{v_{j,m_{j}}}{d_{j,m_{j}}}$, we know that 
$\frac{v_{j,1}}{d_{j,1}}\geq \frac{v_{j,n}}{d_{j,n}} > \gamma$. Now, in Eq. \eqref{eq:pieceline} within the proof of Lemma \ref{lem:Vstruct}, we showed 
\begin{align*}
   V_{j}(\rho_{j};\bm{z}_{j}) = \bm{v}_{j}^{\top} \bm{x}_{j}^{*}(\rho_{j};\bm{z}_{j})=  \sum_{n \in [m_{j}]}\left(\frac{v_{j,n}}{d_{j,n}}\rho_{j}+ \widetilde{b}_{j,n}\right)\mathbb{I}\left\{d_{j,0}+ \dots +d_{j,n-1}\leq \rho_{j}\leq d_{j,0}+ \dots +d_{j,n} \right\}
\end{align*}
where $\widetilde{b}_{j,n} = \sum_{n' \in [n-1]} v_{j,n'} - \frac{v_{j,n}}{d_{j,n}}\cdot \left(\sum_{n' \in [n-1]} d_{j,n'}\right)$ and $  d_{j,0} = v_{j,0} = 0$. Hence by taking $\rho_{j} = \underline{\rho}$ for some $ \underline{\rho} < \min\{d_{j,1},\rho\}$, we have $V_{j}(\underline{\rho}; \bm{z}_{j}) = \frac{v_{j,1}}{d_{j,1}}\underline{\rho} > \gamma \underline{\rho}$. Therefore, constructing $\rho_{(\bm{z}),j} = (0,\dots,0,\rho_{j},0\dots 0)$ for $\rho_{j} = \underline{\rho}$ satisfies $ \sum_{j \in M} V_{j}(\rho_{(\bm{z}),j};\bm{z} ) > \gamma \sum_{j \in M} \rho_{(\bm{z}),j}$ and $ \sum_{j\in [M]}\rho_{(\bm{z}),j} < \rho$.
\halmos

\begin{lemma}[Approximate constraint satisfaction]
\label{lem:approxconst}
Assume Assumption \ref{ass:feas} holds. If Algorithm \ref{alg:omducb} is run with stepsize $\eta = 1/\sqrt{T}$ such that $\eta < \frac{1}{M \max\left\{V, \rho, V^{2}\right\} }$, then we have 
\begin{align*}
     \frac{1}{T}\sum_{t\in[T]}g_{1,t} \geq -C_{F}/\sqrt{T} \text{ and }
  \frac{1}{T}\sum_{t\in[T]}g_{2,t} \geq -C_{F}/\sqrt{T} 
\end{align*}
where we recall $g_{1,t} = \sum_{j\in [M]}\left(V_{j}(\rho_{j,t};\bm{z}_{t}) - \gamma \rho_{j,t}\right)$, $g_{2,t} = \rho - \sum_{j\in [M]}\rho_{j,t}$, and $C_{F} >0 $ is an absolute constant defined in Lemma \ref{lem:dualvarub} that depends only on the support $F = F_{1} \times \dots F_{M}$.
\end{lemma}

\textit{Proof.}
According to the dual update step in Algorithm \ref{alg:omducb} wee have
\begin{align}
    \lambda_{t+1} = \left(\lambda_{t} - \eta g_{1,t}\right)_{+} \geq \lambda_{t} - \eta g_{1,t}
\end{align}
Summing from $t = 1 \dots T-1$ and telescoping, we get
\begin{align}
    \lambda_{T} \geq \lambda_{1} - \eta \sum_{t\in[T]}g_{1,t} = - \eta \sum_{t\in[T]}g_{1,t} 
\end{align}
From Lemma \ref{lem:dualvarub}, we have $ \lambda_{T} \leq C_{F}$ for some absolute constant $C_{F} >0$, therefore we have
\begin{align}
    \sum_{t\in[T]}g_{1,t} \geq -\frac{C_{F}}{\eta} 
\end{align}
since we take $\eta = 1/\sqrt{T}$. A similar bound holds for $ \sum_{t\in[T]}g_{2,t}$.
\halmos

\begin{lemma}
\label{lem:omdtelescope}
    Let $(\lambda_{t},\mu_{t})_{t\in[T]}$ be the dual variables generated by Algorithm \ref{alg:omducb}. Then for any $\lambda,\mu\geq 0$ we have
\begin{align}
\label{eq:CSbound}
\begin{aligned}
    & \sum_{t} \left(\lambda_{t} - \lambda\right) g_{1,t} \leq  \frac{\eta M^{2}\Bar{V}^{2}}{2} \cdot T +  \frac{1}{2\eta}(\lambda- \lambda_{1})^{2}\\
    & \sum_{t} \left(\mu_{t} - \mu\right) g_{2,t} \leq  \frac{\eta \rho^{2}}{2} \cdot T +  \frac{1}{2\eta}(\mu- \mu_{1})^{2}\,.
  \end{aligned}  
\end{align}
where we  recall $g_{1,t} = \sum_{j\in [M]}\left(V_{j,t}(\rho_{j,t}) - \gamma \rho_{j,t}\right)$ and $g_{2,t} = \rho - \sum_{j\in [M]}\rho_{j,t}$. 
\end{lemma}
\textit{Proof.}
It only remains to prove Eq. \eqref{eq:CSbound}. Starting with the first inequality w.r.t. $\lambda_{t}$'s, we have
\begin{align}
\label{eq:decompCS}
    \left(\lambda_{t} - \lambda\right) g_{1,t} =  \left(\lambda_{t+1} - \lambda\right) g_{1,t}  +  \left(\lambda_{t} - \lambda_{t+1}\right) g_{1,t} 
\end{align}
Since $\lambda_{t+1} = \left(\lambda_{t} - \eta g_{1,t}\right)_{+} = \arg\min_{\lambda\geq0}\left(\lambda - \left(\lambda_{t} - \eta g_{1,t}\right)\right)^{2}$, we have
\begin{align}
    \left(\lambda_{t+1} - \left(\lambda_{t} - \eta g_{1,t}\right)\right)\cdot (\lambda - \lambda_{t+1}) \geq 0\,.
\end{align}
So we have
\begin{align}
\begin{aligned}
     \left(\lambda_{t+1} - \lambda\right) g_{1,t}~\leq~ & \frac{1}{\eta}(\lambda_{t+1} - \lambda_{t})\cdot(\lambda - \lambda_{t+1})\\
     ~=~& \frac{1}{2\eta}\left((\lambda- \lambda_{t})^{2} - (\lambda- \lambda_{t+1})^{2}-(\lambda_{t+1}- \lambda_{t})^{2}\right)
     \end{aligned}
\end{align}
Plugging the above back into Eq. \eqref{eq:decompCS} we get
\begin{align}
\label{eq:omdtelescope}
\begin{aligned}
     \left(\lambda_{t} - \lambda\right) g_{1,t} ~\leq~&  \left(\lambda_{t} - \lambda_{t+1}\right) g_{1,t}  +  \frac{1}{2\eta}\left((\lambda- \lambda_{t})^{2} - (\lambda- \lambda_{t+1})^{2}-(\lambda_{t+1}- \lambda_{t})^{2}\right)\\
     ~\leq~&  
   \frac{\eta}{2}g_{1,t}^{2}  +  \frac{1}{2\eta}\left((\lambda- \lambda_{t})^{2} - (\lambda- \lambda_{t+1})^{2}\right)\\
    ~\leq~&  
   \frac{\eta M^{2}\Bar{V}^{2}}{2}  +  \frac{1}{2\eta}\left((\lambda- \lambda_{t})^{2} - (\lambda- \lambda_{t+1})^{2}\right)\,,
   \end{aligned}
\end{align}
where the final inequality follows from the fact that $V_{j,t}(\rho_{j,t})\leq \Bar{V}$ for any $j\in [M]$ and $t\in [T]$ so $g_{1,t}\leq M\Bar{V}$. Summing the above over $t =1\dots T$ and telescoping we get
\begin{align*}
    \sum_{t} \left(\lambda_{t} - \lambda\right) g_{1,t} \leq  \frac{\eta M^{2}\Bar{V}^{2}}{2} \cdot T +  \frac{1}{2\eta}(\lambda- \lambda_{1})^{2}
\end{align*}
Following the same arguments above we can show 
\begin{align*}
   \sum_{t} \left(\mu_{t} - \mu\right) g_{2,t} \leq  \frac{\eta \rho^{2} }{2} \cdot T +  \frac{1}{2\eta}(\mu- \mu_{1})^{2}
\end{align*} 
\halmos



\begin{proposition}
\label{prop:convexprob}
    Under Assumption \ref{ass:feas}, the advertiser's per-channel only budget optimization problem, namely $\textsc{CH-OPT}(\mathcal{I}_{B})$ is a convex problem.
\end{proposition}
\text{Proof.}
Recalling the $\textsc{CH-OPT}(\mathcal{I}_{B})$ in Eq. \eqref{eq:chopt} and the definition of $\mathcal{I}_{B}$ in Eq. \eqref{eq:option}, we can write $\textsc{CH-OPT}(\mathcal{I}_{B})$ as
\begin{align}
\begin{aligned}
   \textsc{CH-OPT}(\mathcal{I}_{B})=  \max_{(\gamma_{j})_{j\in [M]}\in \mathcal{I}} ~&~ \sum_{j \in M} V_{j}(\rho_{j})\\
    \text{s.t. } &  \sum_{j \in M} V_{j}(\rho_{j}) \geq \gamma \sum_{j \in M} \rho_{j}\\
   &   \sum_{j\in [M]}\rho_{j}\leq \rho\,,
    \end{aligned}
\end{align}
Here we used the definition $V_{j}(\rho_{j}) = \mathbb{E}\left[V_{j}(\rho_{j};\bm{z}_{j})\right]$ in Eq. \eqref{eq:budperchannel}, and $D_{j}(\rho_{j};\bm{z}_{j}) = \rho_{j}$ for any $\bm{z}_{j}$ under Assumption \ref{ass:feas}. According to Lemma \ref{lem:Vstruct}, $V_{j}(\rho_{j})$ is concave in $\rho_{j}$ for any $j$, so the objective of $\textsc{CH-OPT}(\mathcal{I}_{B})$ maximizes a concave function. For the feasibility region, assume $\rho_{j}$ and $\rho_{j}'$ are feasible, then defining $\rho_{j}'' = \theta \rho_{j} +  (1-\theta)\rho_{j}'$ for any $\theta \in [0,1]$, we know that 
\begin{align*}
\begin{aligned}
     \sum_{j \in M} \left(V_{j}(\rho_{j}'') - \gamma  \rho_{j}''\right) 
     ~\overset{(i)}{\geq}~ &  \sum_{j \in M} \left(\thetaV_{j}(\rho_{j})  +(1-\theta)V_{j}(\rho_{j}')  - \gamma  \rho_{j}''\right)\\
      ~=~ &  \theta\sum_{j \in M} \left(V_{j}(\rho_{j}) - \gamma  \rho_{j}\right) + (1-\theta)
     \sum_{j \in M} \left(V_{j}(\rho_{j}') - \gamma  \rho_{j}'\right)\\
       ~\overset{(ii)}{\geq}~ & 0
       \end{aligned}
\end{align*}
where (i) follows from concavity of $V_{j}(\rho_{j})$ and (ii) follows from feasiblity of $\rho_{j}$ and $\rho_{j}'$. On the other hand it is apparent that $\sum_{j\in [M]}\rho_{j}'' \leq\rho$. Hence we conclude that for any $\rho_{j}$ and $\rho_{j}'$  feasible,  $\rho_{j}'' = \theta \rho_{j} +  (1-\theta)\rho_{j}'$  is also feasible, so the feasible region of 
$\textsc{CH-OPT}(\mathcal{I}_{B})$ is convex. This concludes the statement of the proposition. \halmos
\subsection{Proof of Lemma \ref{lem:Vstructmulti}}
\label{pf:lem:Vstructmulti}
Before we show the lemma, we first show the following claim is true:

\begin{claim}
\label{cl:ordering}
 Recall $v_{j,n}(1) > \ldots > v_{j,n}(L_{j,n}) > 0$  and $d_{j,n}(1) > \ldots > d_{j,n}(L_{j,n}) > 0$ for any channel $j\in [M]$ and auction $n\in [m_{j}]$. If auction $n$ in channel $j$ has increasing marginal values, i.e. for any realization $\bm{z}_{j} = (\bm{v}_{j},\bm{d}_{j})$,  for any $n\in [m_{j}]$ we have $\frac{v_{j,n}(\ell-1) - v_{j,n}(\ell)}{d_{j,n}(\ell-1) - d_{j,n}(\ell)}$ decreases in $\ell$ then $\frac{v_{j,n}(\ell)}{d_{j,n}(\ell)}$ also decreases in $\ell$.
\end{claim}
\textit{Proof.}
We prove this claim by induction. The base case is $\ell = L_{j,n}$:
it is easy to see 
\begin{align*}
    \frac{v_{j,n}(L_{j,n}-1) - v_{j,n}(L_{j,n})}{d_{j,n}(L_{j,n}-1) - d_{j,n}(L_{j,n})} > \frac{v_{j,n}(L_{j,n})}{d_{j,n}(L_{j,n})} \Longrightarrow  \frac{v_{j,n}(L_{j,n}-1)}{d_{j,n}(L_{j,n}-1)} > \frac{v_{j,n}(L_{j,n})}{d_{j,n}(L_{j,n})}\,. 
\end{align*}
Now assume the induction hypothesis $\frac{v_{j,n}(\ell)}{d_{j,n}(\ell)} > \frac{v_{j,n}(\ell+1)}{d_{j,n}(\ell+1)} > \dots > \frac{v_{j,n}(L_{j,n})}{d_{j,n}(L_{j,n})} $. Then, we have 
\begin{align}
\label{eq:ordering}
\begin{aligned}
   \frac{v_{j,n}(\ell)}{d_{j,n}(\ell)} > \frac{v_{j,n}(\ell+1)}{d_{j,n}(\ell+1)} \Longrightarrow  & \frac{d_{j,n}(\ell+1) - d_{j,n}(\ell)}{d_{j,n}(\ell)} > \frac{v_{j,n}(\ell+1)-v_{j,n}(\ell)}{v_{j,n}(\ell)}\\
   \Longrightarrow  & \frac{d_{j,n}(\ell) - d_{j,n}(\ell+1)}{d_{j,n}(\ell)} < \frac{v_{j,n}(\ell)-v_{j,n}(\ell+1)}{v_{j,n}(\ell)}\\
    \Longrightarrow  & \frac{v_{j,n}(\ell)}{d_{j,n}(\ell)} < \frac{v_{j,n}(\ell)-v_{j,n}(\ell+1)}{d_{j,n}(\ell) - d_{j,n}(\ell+1) }\,.
    \end{aligned}
\end{align}
Since $\frac{v_{j,n}(\ell-1) - v_{j,n}(\ell)}{d_{j,n}(\ell-1) - d_{j,n}(\ell)}$ decreases in $\ell$  we have 
\begin{align*}
    & \frac{v_{j,n}(\ell-1) - v_{j,n}(\ell)}{d_{j,n}(\ell-1) - d_{j,n}(\ell)} > \frac{v_{j,n}(\ell) - v_{j,n}(\ell+1)}{d_{j,n}(\ell) - d_{j,n}(\ell+1)} \overset{(i)}{>} \frac{v_{j,n}(\ell)}{d_{j,n}(\ell)}\\
      \Longrightarrow  & \frac{v_{j,n}(\ell-1)}{d_{j,n}(\ell-1)} \overset{(ii)}{>} \frac{v_{j,n}(\ell)}{d_{j,n}(\ell)}\,,
\end{align*}
where (i) follows from Eq. \eqref{eq:ordering}, and (ii) follows from the fact that $\frac{A}{B} > \frac{C}{D}$ for $A,B,C,D > 0$ implies $\frac{A+C}{B+D} > \frac{C}{D}$ where we let $A = v_{j,n}(\ell-1) - v_{j,n}(\ell)$, $B = d_{j,n}(\ell-1) - d_{j,n}(\ell)$, $C = v_{j,n}(\ell)$ and $D = d_{j,n}(\ell)$. This concludes the proof.
\halmos

Now we prove Lemma \ref{lem:Vstructmulti}. Similar to the proof of Lemma \ref{lem:Vstruct}, we only need to show for any realization $\bm{z}_{j} = (\bm{v}_{j},\bm{d}_{j})_{j\in [M]}$, the conversion function $V_{j}^{+}(\rho_{j};\bm{z}_{j}) = \bm{v}_{j}^{\top} \bm{x}_{j}^{*,+}(\rho_{j};\bm{z}_{j})$ where $\bm{x}_{j}^{*,+}(\rho_{j};\bm{z}_{j})$ is defined as Eq. \eqref{eq:budperchannelsolmulti} is piecewise linear, continuous, strictly increasing and concave.  


For simplicity we use the shorthand notation $\bm{x}_{j}^{*} = \bm{x}_{j}^{*,+}(\rho_{j};\bm{z}_{j})\in [0,1]^{\sum_{n\in [m_{j}]}L_{j,n}}$ as the optimal solution to $V_{j}^{+}(\rho_{j};\bm{z}_{j})$, defined in Eq. \eqref{eq:budperchannelsolmulti}. By re-labeling the auction indices in channel $j\in [M]$ such that
$\frac{v_{j,1}(1)}{d_{j,1}(1)} > \frac{v_{j,2}(1)}{d_{j,2}(1)} > \dots  > \frac{v_{j,m_{j}}(1)}{d_{j,m_{j}}(1)}$, we claim that $\bm{x}_{j}^{*} $ takes the following form:
\begin{align}
\label{eq:multilprelax}
    x_{j,n}^{*}(\ell) = 
    \begin{cases}
    1 & \text{if } \ell = 1 \text{ and } \sum_{n' \in [n]} d_{j,n'}(1) \leq \rho_{j}\\
    \frac{\rho_{j} - \sum_{n' \in [n-1]} d_{j,n'}(1)}{d_{j,n}(1)} & \text{if } \ell = 1 \text{ and } \sum_{n' \in [n]} d_{j,n'}(1) > \rho_{j}\\
    0 & \text{otherwise}
    \end{cases}
\end{align}
which is analogous to that of Eq. \eqref{eq:knapsack} in the proof of  Lemma \ref{lem:Vstruct}. In other words, in the optimal solution, an advertiser would only procure impressions who are in the first position in each auction, and also those with high value-to-cost ratios.
With the above representation of $\bm{x}_{j}^{*} $, the rest of the proof follows exactly from that for Lemma \ref{lem:Vstruct}. 


It now remains to show that Eq. \eqref{eq:multilprelax} holds. 
We first argue by contradiction that in any auction, no impression other than the first would get procured, i.e. $x_{j,n}^{*}(\ell) = 0$  for any $\ell \in 2 \ldots L_{j,n}$.  Assume there exists some auction $n \in [m_{j}]$ and impression slot $\ell' \in 2 \ldots L_{j,n}$ such that
$x_{j,n}^{*}(\ell') > 0$, then by the constraint that at most 1 impression can be procured, i.e. $\sum_{\ell \in [L_{j,n}]}x_{j,n}^{*}(\ell) \leq 1$ in Eq. \eqref{eq:budperchannelsolmulti}, we know that $x_{j,n}^{*}(1) < 1$. Also, note that $x_{j,n}^{*}(\ell')$ incurs a cost of $d_{j,n}(\ell') \cdot x_{j,n}^{*}(\ell')$ amongst the total per-channel budget $\rho_{j}$. If we instead use this 
cost on the first impression,  then we will obtain a value increase of 
\begin{align*}
    v_{j,n}(1)\cdot \frac{d_{j,n}(\ell') \cdot x_{j,n}^{*}(\ell')}{d_{j,n}(1)} - v_{j,n}(\ell') \cdot x_{j,n}^{*}(\ell') = d_{j,n}(\ell')\cdot x_{j,n}^{*}(\ell') \cdot \left( \frac{ v_{j,n}(1)}{d_{j,n}(1)} - \frac{ v_{j,n}(\ell')}{d_{j,n}(\ell')}\right)
> 0,
\end{align*}
where the final inequality follows from the assumption that $x_{j,n}^{*}(\ell') > 0$, and the multi-item auction has increasing marginal values (see Definition \ref{def:incrmarginal}) so Claim \ref{cl:ordering} holds. This contradicts the optimality of $\bm{x}_{j}^{*}$, and hence $x_{j,n}^{*}(\ell) = 0$  for any $\ell \in 2 \ldots L_{j,n}$, or in other words, a channel will only procure impressions ranked first. Hence, a channel's procurement problem in Eq. \eqref{eq:budperchannelsolmulti} can be restricted to the first impression in each auction, and thus similar to the proof of Lemma \ref{lem:Vstruct}, is an LP-relaxation to the 0-1 knapsack with budget $\rho_{j}$, and $m_{j}$ items whose values are $v_{j,1}(1)\ldots v_{j,m_{j}}(1) $ with costs $d_{j,1}(1)\ldots d_{j,m_{j}}(1)$.  
\halmos


\section{Introduction}
\input{1_intro}

\section{Preliminaries}
\label{sec:model}
\input{2_model}

\section{Fairness guarantees for VCG with ML advice}
\label{sec:VCG}
\input{3_fairnessLB}


\section{Impossibility result: VCG is the fairest }
\label{sec:imposs}
\input{4_impossible}

\section{Extensions: fairness guarantees for GSP and GFP with ML advice}
\label{sec:extensions}
\input{5_extensions}

\bibliographystyle{informs2014}
