\section{\label{app:sc}Main results on generalized self-concordant functions}

In this section, we start by introducing a few notations. We define the key notion of generalized self-concordance in \cref{app:sc_2}, and present the main results concerning generalized self-concordant functions. In \cref{app:sc_3}, we describe how generalized self-concordance behaves with respect to an expectation or to certain relaxations.

\paragraph{Notations}

Let $\lambda \geq 0$ and $\mathbf{A}$ be a bounded positive semidefinite Hermitian operator on ${\mathcal{H}}$. We denote with $\mathbf{I}$ the identity operator, and 
\eqal{\|x\|_{\mathbf{A}} &:= \|\mathbf{A}^{1/2} x\|,\\
\mathbf{A}_\lambda &:= \mathbf{A} + \lambda \mathbf{I}.}



Let $f$ be a twice differentiable convex function on a Hilbert space ${\mathcal{H}}$. We adopt the following notation for the Hessian of $f$: 

\[\forall x \in {\mathcal{H}},~\mathbf{H}_f(x):= \nabla^2 f(x) \in \mathcal{L}({\mathcal{H}}).\]

For any $\lambda > 0$, we define the $\lambda$-regularization of $f$:
\[f_\lambda:= f + \frac{\lambda}{2}\|\cdot\|^2.\]
 $f_\lambda$ is $\lambda$-strongly convex and has a unique minimizer which we denote with $x^{f,\lambda}_{\star}$. Moreover, define
 \[\forall x \in {\mathcal{H}},~\mathbf{H}_{f,\lambda}(x):= \nabla^2 f_\lambda(x) = \mathbf{H}_f(x) + \lambda \mathbf{I},\qquad \nu_{f,\lambda}(x):= \|\nabla f_\lambda(x)\|_{\mathbf{H}_{f,\lambda}^{-1}(x)}.\]
The quantity $\nu_{f,\lambda}(x)$ is called the \textbf{Newton decrement} at point $x$ and will play a significant role. 


When the function $f$ is clear from the context, we will omit the subscripts with $f$ and use $\mathbf{H},\mathbf{H}_\lambda,\nu_\lambda ...$. 




\subsection{Definitions and results on generalized self-concordant functions \label{app:sc_2}}

In this section, we introduce the main definitions and results for self-concordant functions. These results are mainly the same as in appendix B of \cite{marteau2019}. 



\begin{definition}[generalized self-concordant function]\label{df:genscc}
Let ${\mathcal{H}}$ be a Hilbert space. Formally, a generalized self-concordant function on ${\mathcal{H}}$ is a couple $(f,\mathcal{G})$  where:
\begin{enumerate}[label = \roman*]
\item $\mathcal{G}$ is a bounded subset of ${\mathcal{H}}$; we will usually denote $\|\mathcal{G}\|$ or $R$ the quantity $\sup_{g \in \mathcal{G}}\|g\| < \infty$;
\item $f$ is a convex and three times differentiable mapping on ${\mathcal{H}}$ such that
\[\forall x \in {\mathcal{H}},~ \forall h,k \in {\mathcal{H}},~ \nabla^{(3)}f(x)[h,k,k] \leq \sup_{g \in \mathcal{G}}|g \cdot h| ~ \nabla^2 f(x)[k,k].\]


\end{enumerate}
\end{definition}

To make notations lighter, we will often omit $\mathcal{G}$ from the notations and simply say that $f$ stands both for the mapping and the couple $(f,\mathcal{G})$. 




\begin{definition}[Definitions]\label{df:first_definitions_sc}
Let $f$ be a generalized self-concordant function. We define the following quantities. 
\begin{itemize}
    \item $\forall h \in {\mathcal{H}},~\mathsf{t}_f(h):= \sup_{g \in \mathcal{G}}|h \cdot g|$;
    \item $\forall x \in {\mathcal{H}},~ \forall \lambda > 0,~ \mathsf{r}_{f,\lambda}(x):= \frac{1}{\sup_{g \in \mathcal{G}}\|g\|_{\mathbf{H}^{-1}_{f,\lambda}(x)}}$;
    \item $\forall \mathsf{c} \geq 0,~\forall \lambda > 0,~ \mathsf{D}_{f,\lambda}(\mathsf{c}):= \left\{x~:~\nu_{f,\lambda}(x) \leq \mathsf{c} \mathsf{r}_{f,\lambda}(x)\right\}$.
\end{itemize}
We also define the following functions: 
\eqal{
\label{eq:def_fun}
\psi(t) = \frac{e^t - t  -1 }{t^2},\quad \underline{\phi}(t) = \frac{1 - e^{-t}}{t},\quad \overline{\phi}(t) = \frac{e^t -1}{t}.
}
\end{definition} 

Note that $\psi$, $\overline{\phi}$ are increasing functions and that $\underline{\phi}$ is a decreasing function. Moreover, $\frac{\overline{\phi}(t)}{\underline{\phi}(t)} = e^t$. Once again, if $f$ is clear, we will often omit the reference to $f$ in the quantities above, keeping only $\mathsf{t},\mathsf{r}_\lambda,\mathsf{D}_\lambda...$


We condense results obtained in \cite{marteau2019} under a slightly different form. The proofs, however, are exactly the same. 

While in \cite{marteau2019}, only the regularized case is dealt with, the proof techniques are exactly the same to obtain \cref{prp:thmcool1}. \cref{prp:thmcool2} is proved explicitly in Proposition 4 of \cite{marteau2019} and \cref{lm:loc} is proved in Proposition 5. \\\
Omitting the subscript $f$, we get the following results.

\begin{proposition}[Bounds for the non-regularized function $f$]\label{prp:thmcool1} Let $f$ be a generalized self-concordant function. Then the following bounds hold (we omit $f$ in the subscripts):
\eqal{
\label{eq:h}
\forall x \in {\mathcal{H}},~ \forall h \in {\mathcal{H}},~ e^{-\mathsf{t}(h)}\mathbf{H}(x) \preceq \mathbf{H}(x + h) \preceq e^{\mathsf{t}(h)}\mathbf{H}(x),
}
\eqal{
\label{eq:g}
\forall x,h \in {\mathcal{H}},~\forall \lambda > 0,~ \|\nabla f(x + h) - \nabla f(x)\|_{\mathbf{H}^{-1}_\lambda(x)} \leq \overline{\phi}(\mathsf{t}(h)) \| h \|_{\mathbf{H}_\lambda(x)},
}
\eqal{
\label{eq:fv}
\forall x,h \in {\mathcal{H}},~ \psi(-\mathsf{t}(h)) \|h\|^2_{\mathbf{H}(x)} \leq f(x + h) - f(x) - \nabla f(x).h \leq \psi(\mathsf{t}(h)) \|h\|^2_{\mathbf{H}(x)}.
}

\end{proposition}


We get the analoguous bounds in the regularized case. 


\begin{proposition}[Bounds for the regularized function $f_\lambda$] \label{prp:thmcool2}
Let $f$ be a generalized self-concordant function and $\lambda >0$ be a regularizer. Then the following bounds hold:
\eqal{
\label{eq:hl}
\forall x,h \in {\mathcal{H}},~ e^{-\mathsf{t}(h)}\mathbf{H}_\lambda(x) \preceq \mathbf{H}_{\lambda}(x + h) \preceq e^{\mathsf{t}(h)}\mathbf{H}_\lambda(x),
}
\eqal{
\label{eq:gl}
\forall x,h \in {\mathcal{H}},~ \underline{\phi}(\mathsf{t}(h)) \| h \|_{\mathbf{H}_\lambda(x)} \leq  \|\nabla f_\lambda(x + h) - \nabla f_\lambda(x)\|_{\mathbf{H}^{-1}_\lambda(x)} \leq \overline{\phi}(\mathsf{t}(h)) \| h \|_{\mathbf{H}_\lambda(x)},
}
\eqal{
\label{eq:fvl}
\forall x,h \in {\mathcal{H}},~ \psi(-\mathsf{t}(h)) \|h\|^2_{\mathbf{H}_\lambda(x)} \leq f_{\lambda}(x + h) - f_\lambda(x) - \nabla f_\lambda(x).h \leq \psi(\mathsf{t}(h)) \|h\|^2_{\mathbf{H}_\lambda(x)}.
}

\end{proposition} 

\begin{corollary}
Let $f$ be a $\mathcal{G}$ generalized self-concordant function and $\lambda >0$ be a regularizer, and $x_\lambda^{\star}$ the unique minimizer of $f_\lambda$. Then the following bounds hold for any $x \in {\mathcal{H}}$:
\eqal{
\label{eq:go}
\underline{\phi}(\mathsf{t}(x - x_\lambda^{\star})) \| x - x_\lambda^{\star} \|_{\mathbf{H}_\lambda(x)} \leq  \underbrace{\|\nabla f_\lambda(x)\|_{\mathbf{H}^{-1}_\lambda(x)}}_{\nu_\lambda(x)} \leq \overline{\phi}(\mathsf{t}(x - x_\lambda^{\star})) \| x - x_\lambda^{\star} \|_{\mathbf{H}_\lambda(x)}
,}
\eqal{
\label{eq:fvo}
 \psi(-\mathsf{t}(x - x_\lambda^{\star})) \|x - x_\lambda^{\star}\|^2_{\mathbf{H}_\lambda(x_\lambda^{\star})} \leq f_{\lambda}(x) - f_\lambda(x_\lambda^{\star}) \leq \psi(\mathsf{t}(x - x_\lambda^{\star})) \|x - x_\lambda^{\star}\|^2_{\mathbf{H}_\lambda(x_\lambda^{\star})}
.}
\end{corollary}


Moreover, the following localization lemma holds. 


\begin{lemma}[localization]\label{lm:loc}
Let $\lambda > 0$ be fixed. If $\frac{\nu_\lambda(x)}{\mathsf{r}_\lambda(x)} <1$, then 
\eqal{
\label{eq:loc}
\mathsf{t}(x - x_\lambda^{\star}) \leq -\log\left(1-\frac{\nu_\lambda(x)}{\mathsf{r}_\lambda(x)} \right).
}
In particular, this shows:
\[\forall \mathsf{c} <1,~\forall \lambda >0,~ x \in \mathsf{D}_\lambda(\mathsf{c}) \implies \mathsf{t}(x - x_\lambda^{\star}) \leq - \log(1 - \mathsf{c}). \]
\end{lemma}

We now state a Lemma which shows that the difference to the optimum in function values is equivalent to the squared newton decrement in a small Dikin ellipsoid. We will use this result in the main paper.

\begin{lemma}[Equivalence of norms]\label{lm:equiv_norms}
Let $\lambda >0$ and $x \in \mathsf{D}_\lambda(\frac{1}{7})$. Then the following holds:
\[\frac{1}{4}\nu_\lambda(x)^2 \leq f_\lambda(x) - f_\lambda(x_\lambda^{\star})\leq \nu_\lambda(x)^2.\]
\end{lemma} 

\begin{proof} 
Apply \cref{lm:loc} knowing $x \in \mathsf{D}_\lambda(\frac{1}{7})$ to get $\nm{x - x_\lambda^{\star}} \leq \log(7/6)$. Then apply \cref{eq:fvl} and \cref{eq:gl} to get:
\eqals{
 f_\lambda(x) - f_\lambda(x_\lambda^{\star}) &\leq \psi(\nm{x - x_\lambda^{\star}})\|x-x_\lambda^{\star}\|_{\mathbf{H}_\lambda(x_\lambda^{\star})}^2\\
&\leq e^{\nm{x - x_\lambda^{\star}}}\psi(\nm{x - x_\lambda^{\star}})\|x-x_\lambda^{\star}\|_{\mathbf{H}_\lambda(x)}^2 \\
&\leq  \frac{e^{\nm{x - x_\lambda^{\star}}}\psi(\nm{x - x_\lambda^{\star}})}{\underline{\phi}(\nm{x-x_\lambda^{\star}})^2}\nu_\lambda(x)^2.  }
Replacing with the bound above, we get
\[\forall \lambda >0,~ \forall x \in \mathsf{D}_\lambda(\frac{1}{7}),~  f_\lambda(x) - f_\lambda(x_\lambda^{\star}) \leq \nu_\lambda(x)^2. \]
For the lower bound, proceed in exactly the same way.
\end{proof} 

\subsection{Comparison between generalized self-concordant functions\label{app:sc_3}}

The following result is straightforward. 

\begin{lemma}[Comparison between generalized self-concordant functions]
Let $\mathcal{G}_1 \subset \mathcal{G}_2 \subset {\mathcal{H}}$ be two bounded subsets. If $(f,\mathcal{G}_1)$ is  generalized self-concordant, then $(f,\mathcal{G}_2)$ is also generalized self-concordant. Moreover, 
\[\forall x \in {\mathcal{H}},~ \forall \lambda > 0,~ \mathsf{r}_{(f,\mathcal{G}_1),\lambda}(x) \geq \mathsf{r}_{(f,\mathcal{G}_2),\lambda}(x).\]
\end{lemma}

In particular, we will often use the following fact. If $(f,\mathcal{G})$ is generalized self-concordant, and $\mathcal{G}$ is bounded by $R$, then $(f,B_{{\mathcal{H}}}(R))$ is also generalized self-concordant. Moreover,  
\[\mathsf{r}_{(f,B_{{\mathcal{H}}}(R)),\lambda}(x) = \frac{\sqrt{\lambda + \lambda_{\min}(\mathbf{H}_f(x))}}{R} \geq \frac{\sqrt{\lambda}}{R}.\]


We now state a result which shows that, given a family of generalized self-concordant functions, the expectancy of that family is also generalized self-concordant. This can be seen as a reformulation of Proposition 2 of \cite{marteau2019}. 

\begin{proposition}[Expectation]\label{prp:expectancy}
Let $\mathcal{Z}$ be a polish space equipped with its Borel sigma-algebra, and ${\mathcal{H}}$ be a Hilbert space. Let $((f_z,\mathcal{G}_z))_{z \in \mathcal{Z}}$ be a family of generalized self-concordant functions such that the mapping $(z,x) \mapsto f_z(x)$ is measurable.

Assume we are given a random variable $Z$ on $\mathcal{Z}$, whose support we denote with $\supp(Z)$, such that 

\begin{itemize}
    \item the random variables $\|f_Z(0)\|, \|\nabla f_Z(0)\|, \Tr(\nabla^2 f_Z(0))$ are are bounded;
    \item $\mathcal{G}:= \bigcup_{z \in \supp(Z)}\mathcal{G}_z$ is a bounded subset of ${\mathcal{H}}$.
\end{itemize}

Then the mapping $f: x \in {\mathcal{H}} \mapsto \Exp{f_Z(x)}$ is well defined, $(f,\mathcal{G})$ is  generalized self-concordant, and we can differentiate under the expectation.

\end{proposition}


\begin{corollary}
Let $n \in \mathbb{N}$ and $(f_i,\mathcal{G}_i)_{1\leq i\leq n}$ be a family of generalized self-concordant functions. Define
\[f(x) = \frac{1}{n}\sum_{i=1}^n{f_i(x)},~\mathcal{G} = \bigcup_{i=1}^n{\mathcal{G}_i}.\]
Then $(f,\mathcal{G})$ is generalized self-concordant.

\end{corollary} 

\newpage





\section{\label{app:res-APN}Results on approximate Newton methods}

In this section, we assume we are given a generalized self-concordant function $f$ in the sense of \cref{app:sc}. As $f$ will be fixed throughout this part, we will omit it from the notations. Recall the definitions from \cref{df:first_definitions_sc}:
\[\nu_\lambda(x):= \|\nabla f_\lambda(x)\|_{\mathbf{H}_\lambda^{-1}(x)},~\frac{1}{\mathsf{r}_\lambda(x)}:= \sup_{g\in \mathcal{G}}{\|g\|_{\mathbf{H}_\lambda^{-1}(x)}},~\mathsf{D}_\lambda(\mathsf{c}):= \left\{x~:~\frac{\nu_\lambda(x)}{\mathsf{r}_\lambda(x)}\leq \mathsf{c} \right\}. \]

Define the following quantities: 

\begin{itemize}
    \item the true Newton step at point $x$ for the $\lambda$-regularized problem:
    $$\Delta_\lambda(x):= \mathbf{H}_\lambda^{-1}(x)\nabla f_\lambda(x).$$
    \item the renormalized Newton decrement $\widetilde{\nu}_\lambda(x)$:
    \[\widetilde{\nu}_\lambda(x):= \frac{\nu_\lambda(x)}{\mathsf{r}_\lambda(x)}.\]
\end{itemize}

Moreover, note that a direct application of \cref{eq:hl} yields the following equation which relates the radii at different points:

\eqal{
\label{eq:radii relation} 
\forall \lambda >0,~ \forall x \in {\mathcal{H}},~ \forall h \in {\mathcal{H}},~ e^{-\nm{h}} \mathsf{r}_\lambda(x) \leq \mathsf{r}_\lambda(x+h) \leq e^{\nm{h}}\mathsf{r}_\lambda(x).
}

In this appendix, we develop a complete analysis of so-called approximate Newton methods in the case of generalized self-concordant losses. By "approximate Newton method", we mean that instead of performing the classical update $x_{t+1} = x_t - \Delta_\lambda(x_t)$, we perform an update of the form $x_{t+1 } = x_t - \widetilde{\Delta}_t$ where $\widetilde{\Delta}_t$ is an approximation of the real Newton step. We will characterize this approximation by measuring its distance to the real Newton step using two parameters $\rho$ and $\epsilon_0$:
\[\|\widetilde{\Delta}_t - \Delta_\lambda(x_t)\| \leq \rho \nu_\lambda(x_t) + \epsilon_0.\]
We start by presenting a few technical results in \cref{app:newton_tech}. We continue by proving that an approximate Newton method has linear convergence guarantees in the right Dikin ellipsoid in \cref{app:newton_gen}. In \cref{app:newton_particular}, we adapt these results to a certain way of computing approximate Newton steps, which will be the one we use in the core of the paper. In \cref{app:newton_better}, we mention ways to reduce the computational burden of these methods by showing that since all Hessians are equivalent in Dikin ellipsoids, one can actually sketch the Hessian at one given point in that ellipsoid instead of re-sketching it at each Newton step. For the sake of simplicity, this is not mentioned in the core paper, but works very well in practice. 

\subsection{Main technical results \label{app:newton_tech}}

We start with a technical decomposition of the Newton decrement at point $x - \widetilde{\Delta}$ for a given $\widetilde{\Delta} \in {\mathcal{H}}$. 

\begin{lemma}[Technical decomposition] Let $\lambda >0$, $x \in {\mathcal{H}}$ be fixed.
Assume we perform a step of the form $x - \widetilde{\Delta}$ for a certain $\widetilde{\Delta} \in {\mathcal{H}}$. 
Define

$$\delta := \|\widetilde{\Delta} - \Delta_\lambda(x)\|_{\mathbf{H}_\lambda(x)},\qquad \widetilde{\delta}:= \frac{\delta}{\mathsf{r}_\lambda(x)}.$$ 
The following holds:
\eqal{\widetilde{\nu}_\lambda(x - \widetilde{\Delta}) &\leq e^{\widetilde{\nu}_\lambda(x) + \widetilde{\delta}}\left[\psi(\widetilde{\nu}_\lambda(x) + \widetilde{\delta})(\widetilde{\nu}_\lambda(x) + \widetilde{\delta})^2 + \widetilde{\delta}\right]; \\
\nu_\lambda(x - \widetilde{\Delta}_\lambda(x)) &\leq e^{\widetilde{\nu}_\lambda(x) + \widetilde{\delta}}\left[\psi(\widetilde{\nu}_\lambda(x) + \widetilde{\delta})(\widetilde{\nu}_\lambda(x) + \widetilde{\delta})(\nu_\lambda(x) + \delta) + \delta\right].}
\end{lemma}

\begin{proof}

Note that by definition, $\nabla f_\lambda(x) = \mathbf{H}_\lambda(x) \Delta_\lambda(x)$. Hence

\eqals{\|\nabla f^\lambda(x - \widetilde{\Delta})\|_{\mathbf{H}_\lambda^{-1}(x)} & = \|\nabla f^\lambda(x - \widetilde{\Delta}) - \nabla f^\lambda(x) + \mathbf{H}_\lambda(x)\Delta_\lambda(x)\|_{\mathbf{H}_\lambda^{-1}(x)}\\
&\leq \|\nabla f^\lambda(x - \widetilde{\Delta}) - \nabla f^\lambda(x) + \mathbf{H}_\lambda(x)\widetilde{\Delta}\|_{\mathbf{H}_\lambda^{-1}(x)} \\
&+ \|\mathbf{H}_\lambda(x)(\Delta_\lambda(x)-\widetilde{\Delta})\|_{\mathbf{H}_\lambda^{-1}(x)} \\
&= \|\int_{0}^1{[\mathbf{H}_\lambda(x - s\widetilde{\Delta}) -\mathbf{H}_\lambda(x)]\widetilde{\Delta} ds} \|_{\mathbf{H}_\lambda^{-1}(x)} + \delta \\
& \leq \int_0^1{\|\mathbf{H}_\lambda^{-1/2}(x)\mathbf{H}_\lambda(x - s\widetilde{\Delta})\mathbf{H}_\lambda^{-1/2}(x) - \mathbf{I}\|ds}~\|\widetilde{\Delta}\|_{\mathbf{H}_\lambda(x)} + \delta.
}

Now using \cref{eq:hl}, one has $\|\mathbf{H}_\lambda^{-1/2}(x)\mathbf{H}_\lambda(x - s\widetilde{\Delta})\mathbf{H}_\lambda^{-1/2}(x) - \mathbf{I}\| \leq e^{s\nm{\widetilde{\Delta}}}-1$, whose integral on $s$ is $\psi(\nm{\widetilde{\Delta}})\nm{\widetilde{\Delta}}$ where $\psi$ is defined in \cref{df:first_definitions_sc}. Morever, bounding 
\[\|\widetilde{\Delta}\|_{\mathbf{H}_\lambda(x)} \leq \|\widetilde{\Delta} - \Delta_\lambda(x)\|_{\mathbf{H}_\lambda(x)} + \|\Delta_\lambda(x)\|_{\mathbf{H}_\lambda(x)} = \delta + \nu_\lambda(x),\]
it holds
\[\|\nabla f^\lambda(x - \widetilde{\Delta})\|_{\mathbf{H}_\lambda^{-1}(x)} \leq  \psi(\nm{\widetilde{\Delta}}) \nm{\widetilde{\Delta}}~(\nu_\lambda(x) + \delta) + \delta .\]

\paragraph{1. }
Now note that using \cref{eq:hl}, it holds:
$\nu_\lambda(x-\widetilde{\Delta}) \leq e^{\nm{\widetilde{\Delta}}/2}\|\nabla f^\lambda(x - \widetilde{\Delta})\|_{\mathbf{H}_\lambda^{-1}(x)}$ and hence: 
\eqal{\label{eq:lm11}\nu_\lambda(x-\widetilde{\Delta}) \leq e^{\nm{\widetilde{\Delta}}/2}\left(\psi(\nm{\widetilde{\Delta}}) \nm{\widetilde{\Delta}}~(\nu_\lambda(x) + \delta) + \delta \right) .}

\paragraph{2. } Moreover, using \cref{eq:radii relation},

\eqal{\label{eq:lm12}
\widetilde{\nu}_\lambda(x-\widetilde{\Delta}) \leq e^{\nm{\widetilde{\Delta}}}\left(\psi(\nm{\widetilde{\Delta}}) \nm{\widetilde{\Delta}}~(\widetilde{\nu}_\lambda(x) + \widetilde{\delta}) + \widetilde{\delta} \right)
.}

Noting that 
$$\nm{\widetilde{\Delta}} \leq \frac{\|\widetilde{\Delta}\|_{\mathbf{H}_\lambda(x)}}{\mathsf{r}_\lambda(x)} \leq  \widetilde{\nu}_\lambda(x) + \widetilde{\delta},$$
and bounding \cref{eq:lm11} simply by taking $e^{\nm{\widetilde{\Delta}}/2} \leq e^{\nm{\widetilde{\Delta}}}$, we get the two bounds in the lemma.


\end{proof}

We now place ourselves in the case where we are given an approximation of the Newton step of the following form. Assume $\lambda$ and $x$ are fixed, and that we approximate $\Delta_\lambda(x)$ with $\widetilde{\Delta}$ such that there exists $\rho \geq 0$ and $\epsilon_0 \geq 0$ such that it holds:
\[\|\widetilde{\Delta} - \Delta_\lambda(x)\|_{\mathbf{H}_\lambda(x)} \leq \rho \nu_\lambda(x) + \epsilon_0.\]
We define/prove the three different following regimes. 

\begin{lemma}[3 regimes]\label{lm:3regimes}
Let $x \in \mathsf{D}_\lambda\left(\frac{1}{7}\right)$ and $\lambda >0$ be fixed. Let
\[ 0 \leq \rho \leq \frac{1}{7},~\epsilon_0 \geq 0 \text{ s.t. } \tilde{\varepsilon}_0:= \frac{\epsilon_0}{\mathsf{r}_\lambda(x)} \leq \frac{1}{21}.\]
Let $\widetilde{\Delta}$ be an approximation of the Newton steps satisfying $\|\widetilde{\Delta} - \Delta_\lambda(x)\|_{\mathbf{H}_\lambda(x)} \leq \rho \nu_\lambda(x) + \epsilon_0$. The three following regimes appear.
\begin{itemize}
    \item If $\widetilde{\nu}_\lambda(x) \geq \rho$ and $\widetilde{\nu}_\lambda(x)^2 \geq \tilde{\varepsilon}_0$, then we are in the \textbf{quadratic regime}, i.e.
 \[\frac{10\widetilde{\nu}_\lambda(x - \widetilde{\Delta}_\lambda(x))}{3} \leq \left(\frac{10\widetilde{\nu}_\lambda(x)}{3}\right) ^2 ,~ \nu_\lambda(x- \widetilde{\Delta}_\lambda(x)) \leq \frac{10}{3} \widetilde{\nu}_\lambda(x) \nu_\lambda(x).\]
 \item If $ \rho \geq \widetilde{\nu}_\lambda(x) $ and $\rho \widetilde{\nu}_\lambda(x) \geq \widetilde{\epsilon}_0$, then we are in the \textbf{linear regime}, i.e.  
  \[\frac{10}{3}\widetilde{\nu}_\lambda(x - \widetilde{\Delta}_\lambda(x)) \leq \left(\frac{10\rho}{3}\right)\left(\frac{10}{3} \widetilde{\nu}_\lambda(x)\right) ,~ \nu_\lambda(x- \widetilde{\Delta}_\lambda(x)) \leq \frac{10}{3} \widetilde{\nu}_\lambda(x) \nu_\lambda(x).\]
  \item If $\widetilde{\epsilon}_0 \geq \widetilde{\nu}_\lambda(x)^2, \rho ~ \widetilde{\nu}_\lambda(x) $, then the \textbf{maximal precision} of the approximation is reached, and it holds: 
    \[\widetilde{\nu}_\lambda(x - \widetilde{\Delta}_\lambda(x)) \leq 3 \widetilde{\epsilon}_0 \leq \frac{1}{7} ,~ \nu_\lambda(x- \widetilde{\Delta}_\lambda(x)) \leq 3 \epsilon_0 .\]
\end{itemize}
\end{lemma}

\begin{proof}
Using the previous lemma, 
\eqals{\widetilde{\nu}_\lambda(x - \widetilde{\Delta}_\lambda(x)) &\leq e^{(1 + \rho)\widetilde{\nu}_\lambda(x) + \widetilde{\epsilon}_0}\left[\psi((1+\rho)\widetilde{\nu}_\lambda(x) + \widetilde{\epsilon}_0)((1 + \rho)\widetilde{\nu}_\lambda(x) + \widetilde{\epsilon}_0)^2 + \rho \widetilde{\nu}_\lambda(x) + \widetilde{\epsilon}_0\right]\\
& \leq \square_1(\widetilde{\nu}_\lambda(x),\rho,\widetilde{\epsilon}_0)~\widetilde{\nu}_\lambda(x)^2 + \square_2(\widetilde{\nu}_\lambda(x),\rho,\widetilde{\epsilon}_0)~\rho \widetilde{\nu}_\lambda(x) + \square_3(\widetilde{\nu}_\lambda(x),\rho,\widetilde{\epsilon}_0)~\widetilde{\epsilon}_0
,}
and 
\[\nu_\lambda(x - \widetilde{\Delta}_\lambda(x)) \leq \square_1(\widetilde{\nu}_\lambda(x),\rho,\widetilde{\epsilon}_0)~\widetilde{\nu}_\lambda(x) \nu_\lambda(x) + \square_2(\widetilde{\nu}_\lambda(x),\rho,\widetilde{\epsilon}_0)~\rho \nu_\lambda(x) + \square_3(\widetilde{\nu}_\lambda(x),\rho,\widetilde{\epsilon}_0)~\epsilon_0,\]
where the following defintions are used: 
\eqals{\square_1(\widetilde{\nu},\rho,\widetilde{\epsilon}_0) &:= e^{(1 + \rho)\widetilde{\nu} + \widetilde{\epsilon}_0}\psi((1 + \rho)\widetilde{\nu} + \widetilde{\epsilon}_0)(1 + \rho)^2,\\
\square_2(\widetilde{\nu},\rho,\widetilde{\epsilon}_0) &:= e^{(1 + \rho)\widetilde{\nu} + \widetilde{\epsilon}_0},\\
\square_3(\widetilde{\nu},\rho,\widetilde{\epsilon}_0) &:= e^{(1 + \rho)\widetilde{\nu} + \widetilde{\epsilon}_0}\left[2 \psi((1 + \rho)\widetilde{\nu} + \widetilde{\epsilon}_0)(1 + \rho)\widetilde{\nu} + 1\right].}
Now assume $\widetilde{\epsilon}_0 \leq \frac{1}{21}$, $\widetilde{\nu}_\lambda(x), \rho \leq \frac{1}{7}$.
Replacing these values in the functions above bounds $\square_1,\square_2$ and $\square_3$, and using the case distinction, we get the result.
\end{proof}


\subsection{General analysis of an approximate Newton method \label{app:newton_gen}}



The following proposition describes the behavior of an approximate newton method where $\rho$ and $\epsilon_0$ are fixed a priori. 

\begin{proposition}[General approximate Newton scheme results]\label{prp:gen_anm}
Let $\mathsf{c} \leq \frac{1}{7}$ be fixed and $x_0 \in \mathsf{D}_\lambda(\mathsf{c})$ be a given starting point. \\
Let $\rho \leq \frac{1}{7}$ and $\epsilon_0$ such that $\epsilon_0 \leq \frac{\mathsf{c}}{4}~\mathsf{r}_\lambda(x_0)$. \\
Define the following approximate Newton scheme: 

\[\forall t \geq 0,~ x_{t+1} = x_t - \widetilde{\Delta}_t,~\qquad\|\widetilde{\Delta}_t - \Delta_\lambda(x_t)\|_{\mathbf{H}_\lambda(x_t)} \leq \rho \nu_\lambda(x_t) + \epsilon_0.\]
The following guarantees hold.
\begin{itemize}
    \item $\forall t \geq 0,~ x_t \in \mathsf{D}_\lambda(\mathsf{c})$.
    \item Let $t_c = \left\lfloor \log_2 \log_2 \frac{3}{10 \rho}\right\rfloor + 1$.
    \[\forall t \leq t_c,~ \frac{10 \widetilde{\nu}_\lambda(x_t)}{3} \leq \max\left(\frac{12 \epsilon_0}{\mathsf{r}_\lambda(x_0)}, 2^{-2^t}\right),\]
    \[\forall t \geq t_c,~ \frac{10 \widetilde{\nu}_\lambda(x_t)}{3} \leq \max\left(\frac{12 \epsilon_0}{\mathsf{r}_\lambda(x_0)}, \left(\frac{10\rho}{3}\right)^{t-t_c  +1 }\right).\]
    \item We can bound the relative decrease for both the Newton decrement and the renormalized Newton decrement:
    \begin{align*}
        &\forall t \leq t_c,~  &&\nu_\lambda(x_t) \leq \max\left(3 \epsilon_0, \left(\frac{1}{2}\right)^{2^t-1}\nu_\lambda(x_0)\right),\\
        & &&\widetilde{\nu}_\lambda(x_t) \leq \max\left(\frac{18 \epsilon_0}{5\mathsf{r}_\lambda(x_0)}, \left(\frac{1}{2}\right)^{2^t-1}\widetilde{\nu}_\lambda(x_0)\right).\\
    &\forall t \geq t_c,~ &&\nu_\lambda(x_t) \leq \max\left(3 \epsilon_0, \left(\frac{10\rho}{3}\right)^{t-t_c  +1 }\nu_\lambda(x_0)\right),\\
    &&&\widetilde{\nu}_\lambda(x_t) \leq \max\left(\frac{18 \epsilon_0}{5 \mathsf{r}_\lambda(x_0)}, \left(\frac{10\rho}{3}\right)^{t-t_c  +1 }\widetilde{\nu}_\lambda(x_0)\right).
        \end{align*}
\end{itemize}
\end{proposition} 





\begin{proof}
Start by noting, using \cref{eq:radii relation},

\eqal{\label{eq:cond_eps}\forall x \in \mathsf{D}_\lambda\left(\frac{1}{7}\right),~ \varepsilon \leq \frac{\mathsf{r}_\lambda(x)}{21},~ \frac{6}{7}\mathsf{r}_\lambda(x_0)\leq  \mathsf{r}_\lambda(x) \leq \frac{7}{6}\mathsf{r}_\lambda(x_0).}
In particular, this holds for any $x \in \mathsf{D}_\lambda(\mathsf{c}),~ \mathsf{c} \leq \frac{1}{7}$.
Thus,
\[\forall \mathsf{c} \leq \frac{1}{7},~ \forall x_0 \in \mathsf{D}_\lambda(\mathsf{c}),~\frac{\epsilon_0}{\mathsf{r}_\lambda(x_0)} \leq \frac{\mathsf{c}}{4} \implies \forall x \in \mathsf{D}_\lambda(\mathsf{c}),~\frac{\epsilon_0}{\mathsf{r}_\lambda(x)} \leq \frac{\mathsf{c}}{3}.\]
\paragraph{1.} Proving the first point is simple by induction.
Indeed, assume $\widetilde{\nu}_\lambda(x_t) \leq \mathsf{c}$. We can apply \cref{lm:3regimes} since the conditions on $\varepsilon$ and $\rho$ guarantee that the conditions of this lemma are satisfied. 

If we are in either the linear or quadratic regime, the fact that $\frac{10\rho}{3},\frac{10 \widetilde{\nu}_\lambda(x_t)}{3} \leq \frac{10}{21}$ show that $\widetilde{\nu}_\lambda(x_{t+1}) \leq \frac{10}{21} \widetilde{\nu}_\lambda(x_t) \leq \mathsf{c}$.

If we are in the last case, $\widetilde{\nu}_\lambda(x_{t+1}) \leq \frac{3 \epsilon_0}{\mathsf{r}_\lambda(x_t)} \leq \mathsf{c}$.

\paragraph{2.} Let us prove the second bullet point by induction. 
Start by assuming the property holds at $t$. By the previous point, the hypothesis of \cref{lm:3regimes} are satisfied at $x_t$ with $\rho$ and $\varepsilon$. Assume we are in the limiting case; we easily show that in this case, 
\[\frac{10 \widetilde{\nu}_\lambda(x_{t+1})}{3} \leq \frac{10}{3} ~3 \frac{\epsilon_0}{\mathsf{r}_\lambda(x_t)} \leq \frac{35 \epsilon_0}{3\mathsf{r}_\lambda(x_0)}.\]
Here, the last inequality comes from \cref{eq:cond_eps}. 
If we are not in the limiting case, let us distinguish between the two following cases. \\\

If  $t \leq t_c - 1$,
\begin{align*}\frac{10 \widetilde{\nu}_\lambda(x_{t+1})}{3} &\leq \frac{10 \widetilde{\nu}_\lambda(x_{t})}{3} \max\left(\frac{10 \widetilde{\nu}_\lambda(x_{t})}{3},\frac{10 \rho}{3}\right) \\
&\leq \max \left(\frac{35 \epsilon_0}{3\mathsf{r}_\lambda(x_0)},\frac{10 \widetilde{\nu}_\lambda(x_{t})}{3} \max\left(\left(\frac{1}{2}\right)^{2^t},\frac{10 \rho}{3}\right)\right),\end{align*}
where the last inequality comes from using the induction hypothesis and the fact that $\frac{10 \widetilde{\nu}_\lambda(x_{t})}{3} \leq 1$. Using once again the induction hypotheses and the fact that $t \leq \left\lfloor \log_2 \log_{2} \frac{3}{10 \rho}\right\rfloor$ which implies $\frac{10 \rho }{3} \leq \left(\frac{1}{2}\right)^{2^t}$, we finally get 
\[\frac{10 \widetilde{\nu}_\lambda(x_{t+1})}{3} \leq \max \left(\frac{35 \epsilon_0}{3 \mathsf{r}_\lambda(x_0)},\left(\frac{1}{2}\right)^{2^{t+1}}\right).\]


The fact that the second property holds for $t = t_c$ is trivial
Now consider the case where $t \geq t_c$. Using the same technique as before but noting that in this case
\[\frac{10 \widetilde{\nu}_\lambda(x_t)}{3} \leq \max\left(\frac{35 \epsilon_0}{3 \mathsf{r}_\lambda(x_0)},\left(\frac{10 \rho}{3}\right)^{t-t_c+1}\right) \leq \max\left(\frac{35 \epsilon_0}{3 \mathsf{r}_\lambda(x_0)},\frac{10 \rho}{3}\right) ,\]
We easily use \cref{lm:3regimes} to reach the desired conclusion. 

\paragraph{3.} Let $t < t_c$. Then using \cref{lm:3regimes}:
\[\forall s \leq t,~ \nu_\lambda(x_{s+1}) \leq \max\left(3 \epsilon_0,\max(\frac{10\rho}{3},\frac{10\widetilde{\nu}_\lambda(x_s)}{3}) \nu_\lambda(x_s)\right).\]
Using the fact that for any $s \leq t$, $\frac{10\widetilde{\nu}_\lambda(x_s)}{3} \leq \max(\frac{35 \epsilon_0}{3\mathsf{r}_\lambda(x_0)},\left(\frac{1}{2}\right)^{2^s})$:
\[\forall s \leq t,~ \nu_\lambda(x_{s+1}) \leq \max\left(3 \epsilon_0,\frac{35 \epsilon_0}{3}\frac{\nu_\lambda(x_s)}{\mathsf{r}_\lambda(x_0)},\max(\frac{10\rho}{3},\left(\frac{1}{2}\right)^{2^s}) \nu_\lambda(x_s)\right).\]
Now using the fact that for any $s \leq t,~ \widetilde{\nu}_\lambda(x_s) \leq \frac{1}{7}$, we see that $\frac{\nu_\lambda(x_s)}{\mathsf{r}_\lambda(x_0)} \leq \frac{7}{6}\widetilde{\nu}_\lambda(x_s) \leq \frac{1}{6}$ and hence $\frac{35 \epsilon_0}{3}\frac{\nu_\lambda(x_s)}{\mathsf{r}_\lambda(x_0)} \leq 3 \epsilon_0$. Moreover, since $s \leq t < t_c$, $\max(\frac{10\rho}{3},\left(\frac{1}{2}\right)^{2^s}) = \left(\frac{1}{2}\right)^{2^s}$. Thus:
\[\forall s \leq t,~ \nu_\lambda(x_{s+1}) \leq \max\left(3 \epsilon_0,\left(\frac{1}{2}\right)^{2^s} \nu_\lambda(x_s)\right).\]
Combining these results yields: 
\[\nu_\lambda(x_{t+1}) \leq \max\left(3 \epsilon_0, \left(\frac{1}{2}\right)^{2^{t+1}-1} \nu_\lambda(x_0)\right).\]
This shows the first equation, that is: 

\[\forall t \leq t_c,~ \nu_\lambda(x_{t}) \leq \max\left(3 \epsilon_0, \left(\frac{1}{2}\right)^{2^{t}-1} \nu_\lambda(x_0)\right).\]
The case for $t \geq t_c$ is completely analogous. We can also reproduce the same proof to get the same bounds for $\widetilde{\nu}$, since the bounds in \cref{lm:3regimes} are the same for both. 


\end{proof}




\subsection{Main results in the paper \label{app:newton_particular}}

In the main paper, we mention two types of Newton method. First, we present a result of convergence on the full Newton method: 



\begin{lemma}[Quadratic convergence of the full Newton method]\label{lm:full_newton}
Let $\mathsf{c} \leq \frac{1}{7}$ and $x_0 \in \mathsf{D}_\lambda(\mathsf{c})$. Define 
\[x_{t+1} =  x_t - \Delta_\lambda(x_t).\]
Then this scheme converges quadratically, i.e.:
\[\forall t \in \mathbb{N},~ \frac{\nu_\lambda(x_t)}{\nu_\lambda(x_0)},\frac{\widetilde{\nu}_\lambda(x_t)}{\widetilde{\nu}_\lambda(x_0)}\leq 2^{-(2^{t} - 1)}.\]
Thus :
\begin{itemize}
    \item $\forall t \in \mathbb{N},~ x_t \in \mathsf{D}_\lambda(\mathsf{c})$.
    \item For any $\tilde{\mathsf{c}} \leq \mathsf{c}$ then $\forall t \geq  \left\lceil \log_2 \left(1  + \log_2 \frac{\mathsf{c}}{\tilde{\mathsf{c}}}\right) \right\rceil,~ x_t \in \mathsf{D}_\lambda(\tilde{\mathsf{c}})$.
    \item For any $\varepsilon > 0$, $\forall t \geq  \left\lceil \log_2 \left(1  + \log_2 \frac{\nu_\lambda(x_0)}{\sqrt{\varepsilon}}\right) \right\rceil,~ \nu_\lambda(x_t) \leq \sqrt{\varepsilon},~ f_\lambda(x)-f_\lambda(x_\lambda^{\star}) \leq \varepsilon$.
    \item If we perform the Newton method and return  the first $x_t$ such that $\nu_\lambda(x_t) \leq \sqrt{\varepsilon}$, then the number of Newton steps computations is at most 
    $1 + \left\lceil \log_2 \left(1  + \log_2 \frac{\nu_\lambda(x_0)}{\sqrt{\varepsilon}}\right) \right\rceil$. 
\end{itemize}
\end{lemma} 

\begin{proof} 

A full Newton method is an approximate Newton method where $\rho,\epsilon_0 = 0$. Thus apply \cref{prp:gen_anm}; note that in this case $t_c = +\infty$. The last point shows that if $\mathsf{c} \leq \frac{1}{7}$, and if we perform the Newton method with a full Newton step, then 
\[\forall t \geq 0,~ \widetilde{\nu}_\lambda(x_t) \leq 2^{-(2^t -1)} \nu_\lambda(x_0),~\widetilde{\nu}_\lambda(x_t) \leq 2^{-(2^t -1)} \nu_\lambda(x_0).\]
This shows the quadratic convergence, and the first two points directly follow. For the third point, the result for $\nu_\lambda(x_t)$ directly follows from the previous equation, and the one on function values is a direct consequence of \cref{lm:equiv_norms} and the fact that $x_t \in \mathsf{D}_\lambda(1/7)$.\\\

For the last point, note that $\nu_t(x_t) = \nabla f_\lambda(x_t)\cdot \Delta_\lambda(x_t)$ is accessible. Moreover, the bound on $t$ is given in the point before, and since one has to compute $\Delta_\lambda(x_s)$ for $0 \leq s \leq t$, there are at most $t+1$ computations.
\end{proof} 

In the main paper, we compute approximate Newton steps by considering methods which naturally yield only a relative error $\rho$ and no absolute error $\epsilon_0$. Indeed, we take the following notation.


\paragraph{Approximate solutions to linear problems.}
Let $\mathbf{A}$ be a positive definite Hermitian operator on ${\mathcal{H}}$, $b$ in ${\mathcal{H}}$, and a wanted relative precision $\rho$.

We say that $x$ is a $\rho$-relative approximation to the linear problem $\mathbf{A} x = b$ and write $x \in \lso(\mathbf{A},b,\rho)$ if the following holds:
\[\|\mathbf{A}^{-1}b - x\|_{\mathbf{A}} \leq \rho \|b\|_{\mathbf{A}^{-1}} = \rho \|\mathbf{A}^{-1} b\|_{\mathbf{A}}.\]
Note that if $x \in \lso(\mathbf{A},b,\rho)$ for $\rho < 1$, then $$ (1-\rho) \|b\|_{\mathbf{A}^{-1}} \leq x \cdot b \leq (1+\rho) \|b\|_{\mathbf{A}^{-1}} . $$



The following lemma shows that if, instead of computing the exact Newton step, we compute a relative approximation of the Newton step belonging to $\lso(\mathbf{H}_\lambda(x),\nabla f_\lambda(x),\rho)$ for a given $\rho < 1$, then one has linear convergence. Moreover, we show that we can still perform a method which automatically stops. 



\begin{proposition}[relative approximate Newton method] \label{prp:approximate_newton_scheme}
Let $\lambda > 0$, $\rho \leq \frac{1}{7}$, $\mathsf{c} \leq \frac{1}{7}$ and a starting point $x_0 \in \mathsf{D}_\lambda(\mathsf{c})$. 
Assume we perform the following Newton scheme:
\[\forall t \geq 0,~ x_{t+1} = x_t - \widetilde{\Delta}_t,~\qquad \widetilde{\Delta}_t \in \lso(\mathbf{H}_\lambda(x_t),\nabla f_\lambda(x_t),\rho) .\]
Then the scheme converges linearly, i.e. 
\[\forall t \in \mathbb{N},~ \frac{\nu_\lambda(x_t)}{\nu_\lambda(x_0)},\frac{\widetilde{\nu}_\lambda(x_t)}{\widetilde{\nu}_\lambda(x_0)} \leq 2^{-t}.\]
Thus,
\begin{itemize}
   \item $\forall t \in \mathbb{N},~ x_t \in \mathsf{D}_\lambda(\mathsf{c})$.
    \item For any $\tilde{\mathsf{c}} \leq \mathsf{c}$ then $\forall t \geq  \left\lceil \log_2  \frac{\mathsf{c}}{\tilde{\mathsf{c}}} \right\rceil,~ x_t \in \mathsf{D}_\lambda(\tilde{\mathsf{c}})$.
    \item For any $\varepsilon > 0$, $\forall t \geq  \left\lceil \log_2  \frac{\nu_\lambda(x_0)}{\sqrt{\varepsilon}} \right\rceil,~ \nu_\lambda(x_t) \leq \sqrt{\varepsilon},~ f_\lambda(x)-f_\lambda(x_\lambda^{\star}) \leq \varepsilon$
    \item If the method is performed and returns  the first $x_t$ such that $x_t \cdot \widetilde{\Delta}_t \leq \frac{6}{7}\varepsilon$, then at most 
    $2 + \left\lfloor \log_2\left( \sqrt{\frac{4}{3}}\frac{\nu_\lambda(x_0)}{\sqrt{ \varepsilon}}\right) \right\rfloor$ approximate Newton steps computations have been performed, and $\nu_\lambda(x_t) \leq \sqrt{\varepsilon},~ f_\lambda(x)-f_\lambda(x_\lambda^{\star}) \leq \varepsilon$.
\end{itemize}
\end{proposition}

\begin{proof} 
Apply \cref{prp:gen_anm} with $\epsilon_0 = 0$ and $\rho = \frac{1}{7}$, since if $\rho \leq \frac{1}{7}$, then a fortiori the approximation satisfies the condition for $\rho = \frac{1}{7}$. The last point clearly states that
\[\forall t \in \mathbb{N},~ \frac{\nu_\lambda(x_t)}{\nu_\lambda(x_0)},\frac{\widetilde{\nu}_\lambda(x_t)}{\widetilde{\nu}_\lambda(x_0)} \leq \left(\frac{10}{21}\right)^{t} \leq 2^{-t}.\]
From this, using \cref{lm:equiv_norms} for the third point, the first three points are easily proven. \\\
For the last point, note that since $\widetilde{\Delta}_t \in \lso(\mathbf{H}_\lambda(x_t),\nabla f_\lambda(x_t),\rho)$, the following holds: $\nabla f_\lambda(x_t) \cdot \widetilde{\Delta}_t = \nu_\lambda(x_t)^2 + \nabla f_\lambda(x_t) \cdot \left(\widetilde{\Delta}_t - \mathbf{H}_\lambda^{-1}(x_t) \nabla f_\lambda(x_t)  \right)$. 
Now bound 
\[|\nabla f_\lambda(x_t) \cdot\left( \widetilde{\Delta}_t - \mathbf{H}_\lambda^{-1}(x_t) \nabla f_\lambda(x_t) \right)| \leq \nu_\lambda(x_t) ~\|\widetilde{\Delta}_t - \mathbf{H}_\lambda^{-1}(x_t) \nabla f_\lambda(x_t)\|_{\mathbf{H}_\lambda(x_t)} \leq \rho \nu_\lambda(x_t)^2.\]
Thus: 
\[(1-\rho)\nu_\lambda(x_t)^2 \leq \nabla f_\lambda(x_t) \cdot \widetilde{\Delta}_t \leq (1+\rho)\nu_\lambda(x_t)^2.\]
Since $\rho = \frac{1}{7}$, we see that if $\nabla f_\lambda(x_t) \cdot \widetilde{\Delta}_t \leq \frac{6}{7}\varepsilon$, then $\nu_\lambda(x_t)^2 \leq \varepsilon$. 
Moreover, since we stop at the first $t$ where $\nabla f_\lambda(x_t) \cdot \widetilde{\Delta}_t \leq \frac{6}{7}\varepsilon$, then if $t$ denotes the time at which we stop,
\[ \frac{6}{7}\varepsilon < \nabla f_\lambda(x_{t-1}) \cdot \widetilde{\Delta}_{t-1}  \leq \frac{8}{7} \nu_\lambda(x_{t-1})^2 \implies \nu_\lambda(x_{t-1}) ^2\geq \frac{3}{4}\varepsilon.\]
Since $\nu_\lambda(x_{t-1})^2 \leq 2^{-2(t-1)}\nu_\lambda(x_0)^2$, this implies in turn that $t-1 \leq \log_2\left( \sqrt{\frac{4}{3}}\frac{\nu_\lambda(x_0)}{\sqrt{ \varepsilon}}\right)$.
Thus, necessarily, $t \leq 1 + \left\lfloor \log_2\left( \sqrt{\frac{4}{3}}\frac{\nu_\lambda(x_0)}{\sqrt{ \varepsilon}}\right) \right\rfloor$, and since we compute approximate Newton steps for $s = 0,...,t$, we finally have that the number of approximate Newton steps is bounded by 
\[2 + \left\lfloor \log_2\left( \sqrt{\frac{4}{3}}\frac{\nu_\lambda(x_0)}{\sqrt{ \varepsilon}}\right) \right\rfloor.\]
\end{proof} 

Last but not least, we summarize all these theorem in the following simple result.

\begin{lemma}\label{lm:dikin-is-cool_2}
Let $\lambda > 0, \mathsf{c} \leq 1/7$, let $f_\lambda$ be generalized self-concordant and $x \in \mathsf{D}_\lambda(\mathsf{c})$. It holds:
   $
    \frac{1}{4}\nu_\lambda(x)^2 \leq f_\lambda(x) - f_\lambda(x_\lambda^{\star}) \leq \nu_\lambda(x)^2$.
Moreover, the full Newton method starting from $x_0$ has quadratic convergence, i.e. if $x_t$ is obtained via $t \in \mathbb{N}$ steps of the Newton method \cref{eq:newton-intro}, then
$ \nu_\lambda(x_t) \leq 2^{-(2^{t} - 1)} \nu_\lambda(x_0).$
Finally, the approximate Newton method starting from $x_0$ has linear convergence, i.e. if $x_t$ is obtained via $t \in \mathbb{N}$ steps of  \cref{eq:approx-newton-intro}, with $\widetilde{\Delta}_t \in \lso(\mathbf{H}_\lambda(x_t),\nabla f_\lambda(x_t),\rho)$ and $\rho \leq 1/7$, then
$\nu_\lambda(x_t) \leq 2^{-t} \nu_\lambda(x_0).$
\end{lemma}

\begin{proof} 
 The three points are obtained in the following lemmas, assuming $x \in \mathsf{D}_\lambda(1/7)$.
 \begin{itemize}
 \item  For $\frac{1}{4}\nu_\lambda(x)^2 \leq f_\lambda(x) - f_\lambda(x_\lambda^{\star}) \leq \nu_\lambda(x)^2$, see \cref{lm:equiv_norms} in \cref{app:sc_2}.
 \item The convergence rate of the full Newton method starting in $\mathsf{D}_\lambda(1/7)$ is obtained in \cref{lm:full_newton}.
 \item The convergence rate of the approximate Newton method starting in $\mathsf{D}_\lambda(1/7)$ is obtained in \cref{prp:approximate_newton_scheme}. 
 \end{itemize}
\end{proof}

\subsection{Sketching the Hessian only once in each Dikin ellispoid \label{app:newton_better}}

In this section, we provide a lemma which shows in essence that if we are in a small Dikin ellipsoid, then we can keep the Hessian of the starting point and compute approximations of $\mathbf{H}_\lambda^{-1}(x_0)\nabla f_\lambda(x_t)$; they will be good approximations to $\mathbf{H}_\lambda^{-1}(x_t)\nabla f_\lambda(x_t)$ as well.


\begin{lemma} 
Let $\mathsf{c} < 1$ and $x_0 \in \mathsf{D}_\lambda(\mathsf{c})$ be fixed.

Let $\widetilde{\mathbf{H}}$ be an approximation of the Hessian at $x_0$, approximation wich we quantify with 
\[t:= \|\mathbf{H}^{-1/2}_\lambda(x_0)\left(\mathbf{H}_\lambda(x_0) - \widetilde{\mathbf{H}}\right)\mathbf{H}^{-1/2}_\lambda(x_0)\|.\]
Assume 
\[1+t < 2(1 - \mathsf{c})^2.\]
Let $b \in {\mathcal{H}}$. If $\widetilde{\Delta} \in \lso(\widetilde{\mathbf{H}}_\lambda,b,\tilde{\rho})$, then
$$\forall x \in \mathsf{D}_\lambda(\mathsf{c}),~ \widetilde{\Delta} \in \lso(\mathbf{H}_\lambda(x),b,\rho),~ \rho = \frac{(\tilde{\rho} -1)(1-\mathsf{c})^2 + (1+t)}{2(1 - \mathsf{c})^2 - (1+t)}.$$
In particular, if $\mathsf{c} \leq \frac{1}{30}$, $x_0 \in \mathsf{D}_\lambda(\mathsf{c})$, 
\[\forall x \in \mathsf{D}_\lambda(\mathsf{c}),~\forall b \in {\mathcal{H}},~ \widetilde{\Delta} \in \lso(\mathbf{H}_\lambda(x_0),b,\frac{1}{20}) \implies \widetilde{\Delta} \in \lso(\mathbf{H}_\lambda(x),b,\frac{1}{7}).\]

\end{lemma} 

\begin{proof} 
First, start with a general theoretical result. 
\paragraph{1. } Let $\mathbf{A}$ and $\mathbf{B}$ be two positive semi-definite hermitian operators. Let $\lambda > 0$, $b \in {\mathcal{H}}$ and $\widetilde{\Delta} \in \lso(\mathbf{B}_\lambda,b,\tilde{\rho})$. 
Decompose 
\eqals{
\|\mathbf{A}_\lambda^{-1}b - \widetilde{\Delta}\|_{\mathbf{A}_\lambda} &\leq \|\mathbf{A}_\lambda^{-1}b - \mathbf{B}_\lambda^{-1}b\|_{\mathbf{A}_\lambda}  + \|\mathbf{B}_\lambda^{-1}b - \widetilde{\Delta}\|_{\mathbf{A}_\lambda} \\
&\leq \|\mathbf{A}_\lambda^{1/2}(\mathbf{A}_\lambda^{-1} - \mathbf{B}_\lambda^{-1})\mathbf{A}_\lambda^{1/2}\|~ \|b\|_{\mathbf{A}_\lambda^{-1}} + \|\mathbf{A}_\lambda^{1/2}\mathbf{B}_\lambda^{-1/2}\| ~\|\mathbf{B}_\lambda^{-1}b - \widetilde{\Delta}\|_{\mathbf{B}_\lambda}.}

Now using the fact that $\mathbf{A}_\lambda^{-1} - \mathbf{B}_\lambda^{-1} = \mathbf{B}_\lambda^{-1}(\mathbf{B} - \mathbf{A})\mathbf{A}_\lambda^{-1}$, 

\eqals{
\|\mathbf{A}_\lambda^{1/2}(\mathbf{A}_\lambda^{-1} - \mathbf{B}_\lambda^{-1})\mathbf{A}_\lambda^{1/2}\| &\leq \|\mathbf{A}_\lambda^{-1/2}(\mathbf{B}-\mathbf{A})\mathbf{A}_\lambda^{-1/2}\|~\|\mathbf{A}_\lambda^{1/2}\mathbf{B}_\lambda^{-1} \mathbf{A}_\lambda^{1/2}\| \\
&=  \|\mathbf{A}_\lambda^{-1/2}(\mathbf{B}-\mathbf{A})\mathbf{A}_\lambda^{-1/2}\|~\|\mathbf{A}_\lambda^{1/2}\mathbf{B}_\lambda^{-1/2} \|^2.
}
Moreover, 
\[\|\mathbf{B}_\lambda^{-1}b - \widetilde{\Delta}\|_{\mathbf{B}_\lambda} \leq \tilde{\rho} \|b\|_{\mathbf{B}_\lambda^{-1}} \leq \|\mathbf{A}^{1/2}\mathbf{B}^{-1/2}\|~\|b\|_{\mathbf{A}_\lambda^{-1}}.\]
Putting things together, and noting that from \cref{lm:equiv_operators}, $\|\mathbf{A}^{1/2}\mathbf{B}^{-1/2}\|^2 \leq \frac{1}{1-\|\mathbf{A}_\lambda^{-1/2}(\mathbf{B} - \mathbf{A})\mathbf{A}_\lambda^{-1/2}\|}$ as soon as $\|\mathbf{A}_\lambda^{-1/2}(\mathbf{B} - \mathbf{A})\mathbf{A}_\lambda^{-1/2}\| < 1$, it holds: 
$$\widetilde{\Delta} \in \lso(\mathbf{A}_\lambda,b,\rho),~\rho = \frac{\tilde{\rho} + \|\mathbf{A}_\lambda^{-1/2}(\mathbf{B} - \mathbf{A})\mathbf{A}_\lambda^{-1/2}\|}{1 - \|\mathbf{A}_\lambda^{-1/2}(\mathbf{B} - \mathbf{A})\mathbf{A}_\lambda^{-1/2}\|} .$$

The aim is now to apply this lemma to $\mathbf{A} = \mathbf{H}(x)$ and $\mathbf{B} = \widetilde{\mathbf{H}}$.

\paragraph{2. } Let $x,x_0 \in \mathsf{D}_\lambda(\mathsf{c})$. Using \cref{lm:comb_approx}, we see that 
\[1 + \|\mathbf{H}_\lambda^{-1/2}(x)(\widetilde{\mathbf{H}} - \mathbf{H}(x))\mathbf{H}_\lambda^{-1/2}(x)\| \leq (1 + t)(1 + \|\mathbf{H}_\lambda^{-1/2}(x)(\mathbf{H}(x_0) - \mathbf{H}(x))\mathbf{H}_\lambda^{-1/2}(x)\| ).\]
Using \cref{eq:hl}, it holds:
\[ (e^{-\nm{x-x_0}}- 1)\mathbf{I} \preceq \mathbf{H}_\lambda^{-1/2}(x)(\mathbf{H}(x_0) - \mathbf{H}(x))\mathbf{H}_\lambda^{-1/2}(x) \preceq (e^{\nm{x_0 - x}} - 1)\mathbf{I} .\]
Thus, 
\[\|\mathbf{H}_\lambda^{-1/2}(x)(\mathbf{H}(x_0) - \mathbf{H}(x))\mathbf{H}_\lambda^{-1/2}(x)\| \leq \max(1-e^{-\nm{x-x_0}},e^{\nm{x-x_0}} - 1) = e^{\nm{x-x_0}} - 1 .\]
Finally, using the fact that $x_0,x \in \mathsf{D}_\lambda(\mathsf{c})$ for $\mathsf{c} < 1$ yields $\nm{x-x_0} \leq 2 \log \frac{1}{1-\mathsf{c}}$. Hence
\[1 + \|\mathbf{H}_\lambda^{-1/2}(x)(\mathbf{H}(x_0) - \mathbf{H}(x))\mathbf{H}_\lambda^{-1/2}(x)\| \leq \frac{1}{(1-\mathsf{c})^2}.\]
Thus,
\[\|\mathbf{H}_\lambda^{-1/2}(x)(\widetilde{\mathbf{H}} - \mathbf{H}(x))\mathbf{H}_\lambda^{-1/2}(x)\| \leq \frac{1 + t}{(1 - \mathsf{c})^2} - 1.\]
The result then follows.
\end{proof} 









\newpage


\section{Proof of bounds for the globalization scheme}\label{app:dec-lambda}

In this section, we prove that the scheme of decreasing $\mu$ towards $\lambda$ converges. 

\subsection{Main technical lemmas \label{app:dec_la_tech}}

\begin{lemma}[Next $\mu$]\label{lm:next_la} Let $\mu >0$, $\mathsf{c} <1$.
$$\nu_\mu(x) \leq \frac{\mathsf{c}}{3}~\frac{\sqrt{\mu}}{R} \implies \nu_{\widetilde{\mu}}(x) \leq \mathsf{c} ~ \frac{\sqrt{\widetilde{\mu}}}{R}, \qquad \widetilde{\mu}:=  q~\mu,\qquad q \geq \frac{\frac{1}{3} + \frac{R \sqrt{\mu} \|x\|_{\mathbf{H}_\mu^{-1}(x)}}{\mathsf{c}}}{1 + \frac{R \sqrt{\mu} \|x\|_{\mathbf{H}_\mu^{-1}(x)}}{\mathsf{c}}} .$$

$$ x\in \mathsf{D}_\mu\left( \frac{\mathsf{c}}{3}\right)  \implies x\in \mathsf{D}_{\widetilde{\mu}}\left( \mathsf{c}\right), \qquad \widetilde{\mu}:=  q~\mu,\qquad q \geq \frac{\frac{1}{3} + \frac{\mu \|x\|_{\mathbf{H}_\mu^{-1}(x)}}{\mathsf{c}~ \mathsf{r}_\mu(x)}}{1 + \frac{\mu \|x\|_{\mathbf{H}_\mu^{-1}(x)}}{\mathsf{c}~ \mathsf{r}_\mu(x)}} .$$

\end{lemma}

\begin{proof}
For any $\widetilde{\mu} < \mu$, note that 
$$ \forall x \in {\mathcal{H}},~ \|\mathbf{H}_{\widetilde{\mu}}^{-1/2}(x)\mathbf{H}_{\mu}^{1/2}(x)\| = \sqrt{\frac{\lambda_{\min}(\mathbf{H}(x)) + \mu}{\lambda_{\min}(\mathbf{H}(x)) + \widetilde{\mu}}}\leq \sqrt{\mu/\widetilde{\mu}}.$$
 This shows that $\|\cdot\|_{\mathbf{H}_{\widetilde{\mu}}^{-1}(x)} \leq  \sqrt{\frac{\mu}{\widetilde{\mu}}}~\|\cdot\|_{\mathbf{H}_{\mu}^{-1}(x)}$, and in particular that $\frac{1}{\mathsf{r}_{\widetilde{\mu}}(x)} \leq \sqrt{\mu/\widetilde{\mu}}\frac{1}{\mathsf{r}_{\mu}(x)}$.
 
 Using this fact, it holds: 
\eqals{\widetilde{\nu}_{\widetilde{\mu}}(x) &= \frac{\|\nabla f_{\widetilde{\mu}}(x)\|_{\mathbf{H}^{-1}_{\widetilde{\mu}}(x)}}{\mathsf{r}_{\widetilde{\mu}}(x)} \\
&= \frac{\|\nabla f_\mu(x) -(\mu - \widetilde{\mu})x\|_{\mathbf{H}^{-1}_{\widetilde{\mu}}(x)}}{\mathsf{r}_{\widetilde{\mu}}(x)} \\
& \leq \frac{\mu}{\widetilde{\mu}}~\frac{\|\nabla f_\mu(x)\|_{\mathbf{H}_{\mu}^{-1}(x)}}{\mathsf{r}_\mu(x)}
 + \left(\frac{\mu}{\widetilde{\mu}}-1\right) \frac{\|\mu x\|_{\mathbf{H}^{-1}_\mu(x)}}{\mathsf{r}_\mu(x)}.}
 
 Hence, if $\widetilde{\nu}_\mu(x) \leq \frac{\mathsf{c}}{3}$, a condition to obtain $\widetilde{\nu}_{\widetilde{\mu}}(x) \leq \mathsf{c}$ is the following:
 
 \[\frac{\mu}{\widetilde{\mu}} \left(\frac{\mathsf{c}}{3} + t\right) \leq \mathsf{c} + t \Leftrightarrow \widetilde{\mu} \geq \mu \frac{\mathsf{c}/3 +  t}{ \mathsf{c} +  t}\qquad t =  \frac{\|\mu x\|_{\mathbf{H}^{-1}_\mu(x)}}{\mathsf{r}_\mu(x)}.\]
 This yields the second point of the lemma. The analysis is completely analoguous for the first.
\end{proof}



\begin{lemma}[Useful bounds for $q$]\label{lm:bounds_q}
Let $\mu >0$. Then the following hold:
\[ \forall x \in {\mathcal{H}},~ \frac{\mu \|x\|_{\mathbf{H}_\mu^{-1}(x)}}{\mathsf{r}_\mu(x)} \leq R \sqrt{\mu} \|x\|_{\mathbf{H}_\mu^{-1}(x)} \leq R \|x\|
.\]
Moreover, we can bound all of these quantities using $x_\mu^\star$: 
\begin{itemize}
\item For any $\mathsf{c} < 1$, $x \in {\mathcal{H}}$, if $ x \in \mathsf{D}_\mu(\mathsf{c}/3)$, then the following holds:
\[\frac{\mu \|x\|_{\mathbf{H}^{-1}_\mu(x)}}{\mathsf{c} ~ \mathsf{r}_\mu(x)} \leq \frac{1}{3}\left(1 + \frac{1}{1-\mathsf{c}/3}\right) + \frac{1}{1-\mathsf{c}/3} \frac{\|\mu x_\mu^\star\|_{\mathbf{H}_\mu^{-1}(x_\mu^\star)}}{ \mathsf{c} ~ \mathsf{r}_\mu(x_\mu^\star)}. \]
\item For any $\mathsf{c} < 1$, $x \in {\mathcal{H}}$, if $\frac{R \nu_\mu(x)}{\sqrt{\mu}} \leq \frac{\mathsf{c}}{3}$, then the following holds:
\[\frac{R \sqrt{\mu} \|x\|_{\mathbf{H}_\mu^{-1}(x)}}{\mathsf{c}} \leq \left(1 + \frac{1}{1-\mathsf{c}/3}\right)\frac{1}{3} + \sqrt{\frac{1}{1-\mathsf{c}/3}} \frac{R \sqrt{\mu} \|x_\mu^\star\|_{\mathbf{H}_\mu^{-1}(x_\mu^\star)}}{\mathsf{c}}.\]
Likewise, it can be shown that under the same conditions: 
\[\frac{R \|x\|}{\mathsf{c}} \leq \frac{R \|x_\mu^\star\|}{\mathsf{c}} + \frac{1}{3}\overline{\phi}(-\log(1 - \mathsf{c}/3)).\]
\end{itemize}
\end{lemma}

\begin{proof} 
The first bound is obvious. Moreover, the fact that $\widetilde{\nu}_\mu(x) \leq \frac{\mathsf{c}}{3}$ implies that $\nm{x - x_\mu^\star}  \leq \log\frac{1}{1-\mathsf{c}/3}$. Thus, we get the classical bounds on the Hessian using \cref{eq:h}: 
\[e^{-\nm{x - x_\mu^\star}} \mathbf{H}(x) \preceq \mathbf{H}(x_\mu^\star) \preceq e^{\nm{x - x_\mu^\star}} \mathbf{H}(x).\]

\paragraph{1. Bound on $\mu \|x\|_{\mathbf{H}_\mu^{-1}(x)}$.}
 Using \cref{eq:hl,eq:gl},

\eqals{
\mu \|x\|_{\mathbf{H}^{-1}_\mu(x)} & = \|\nabla f_\mu(x) - \nabla f(x) + \nabla f(x_\mu^\star) - \nabla f(x_\mu^\star)\|_{\mathbf{H}^{-1}_\mu(x)}  \\
&\leq \nu_\mu(x) + \int_{0}^1{\|\mathbf{H}_\mu(x)^{-1/2}\mathbf{H}(x_t)(x - x_\mu^\star)\|~dt} + \|\nabla f(x_\mu^\star)\|_{\mathbf{H}_\mu(x)},~x_t = t x + (1-t)x_\mu^\star.}

Now bound $\|\mathbf{H}_\mu(x)^{-1/2}\mathbf{H}(x_t)(x - x_\mu^\star)\| \leq \|\mathbf{H}_\mu(x)^{-1/2}~\mathbf{H}_\mu(x_t)^{1/2}\|~\|x - x_\mu^\star\|_{\mathbf{H}(x_t)}$ and use \cref{eq:hl} and \cref{eq:h} to get:
\[\|\mathbf{H}_\mu(x)^{-1/2}\mathbf{H}(x_t)(x - x_\mu^\star)\| \leq e^{t~\nm{x-x_\mu^\star}} \|x - x_\mu^\star\|_{\mathbf{H}(x)}.\]
Integrating this yields:
\[\int_{0}^1{\|\mathbf{H}_\mu(x)^{-1/2}\mathbf{H}(x_t)(x - x_\mu^\star)\|~dt} \leq \overline{\phi}(\nm{x -x_\mu^\star})~ \|x - x_\mu^\star\|_{\mathbf{H}(x)} \leq e^{\nm{x-x_\mu^\star}}~\nu_\mu(x).\]
Where the last inequality is obtained using the bounds between gradient and hessian distance \cref{eq:gl}. Finally, using the bound on $\nm{x-x_\mu^\star}$,
\[\mu \|x\|_{\mathbf{H}_\mu^{-1}(x)} \leq \left(1 + \frac{1}{1-\mathsf{c}/3}\right)\nu_\mu(x) + \sqrt{\frac{1}{1-\mathsf{c}/3}}\|\nabla f(x_\mu^\star)\|_{\mathbf{H}_\mu^{-1}(x_\mu^\star)}.\]

\paragraph{2. Bound on $R  \|x\|$.}
Start by decomposing 
\[R \|x\| \leq R \|x_\mu^\star\| + R  \|x - x_\mu^\star\|.\]
Now bound 
\[R  \|x - x_\mu^\star\| \leq \frac{R}{\sqrt{\mu}} \|x - x_\mu^\star\|_{\mathbf{H}_\mu(x)}.\]
Using \cref{eq:hl}, $ \|x - x_\mu^\star\|_{\mathbf{H}_\mu(x)} \leq \overline{\phi}(-\log(1-\mathsf{c}/3))\nu_\mu(x).$ Hence:
\[R \|x\| \leq R \|x_\mu^\star\| + \overline{\phi}(-\log(1-\mathsf{c}/3)) \frac{R \nu_\mu(x)}{\sqrt{\mu}} .\]

\paragraph{3. Now assume $x \in \mathsf{D}_\mu(\mathsf{c}/3)$.} Using the bound on $\mu \|x\|_{\mathbf{H}^{-1}_\mu(x)}$, and noting that 
\[\frac{1}{\mathsf{r}_\mu(x)} \leq e^{\nm{x-x_\mu^\star}/2} \frac{1}{\mathsf{r}_\mu(x_\mu^\star)},\]
it holds:
\[\frac{\mu \|x\|_{\mathbf{H}^{-1}_\mu(x)}}{\mathsf{c} ~ \mathsf{r}_\mu(x)} \leq \frac{1}{3}\left(1 + \frac{1}{1-\mathsf{c}/3}\right) + \frac{1}{1-\mathsf{c}/3} \frac{\|\mu x_\mu^\star\|_{\mathbf{H}_\mu^{-1}(x_\mu^\star)}}{ \mathsf{c} ~ \mathsf{r}_\mu(x_\mu^\star)}.\]
\paragraph{4. Now assume $\frac{R \nu_\mu(x)}{\sqrt{\mu}} \leq \frac{\mathsf{c}}{3}$.}. We know that in particular, $x \in \mathsf{D}_\mu(\mathsf{c}/3)$ and hence:
\eqals{R \sqrt{\mu} \|x\|_{\mathbf{H}_\mu^{-1}(x)} &\leq \left(1 + \frac{1}{1-\mathsf{c}/3}\right)\frac{R \nu_\mu(x)}{\sqrt{\mu}} + \sqrt{\frac{1}{1-\mathsf{c}/3}}\frac{R \mu \|x_\mu^\star\|_{\mathbf{H}_\mu^{-1}(x_\mu^\star)}}{\sqrt{\mu}}\\
&\leq \left(1 + \frac{1}{1-\mathsf{c}/3}\right)\frac{\mathsf{c}}{3} + \sqrt{\frac{1}{1-\mathsf{c}/3}} R \sqrt{\mu} \|x_\mu^\star\|_{\mathbf{H}_\mu^{-1}(x_\mu^\star)}.}
Hence 
\[\frac{R \sqrt{\mu} \|x\|_{\mathbf{H}_\mu^{-1}(x)}}{\mathsf{c}} \leq \left(1 + \frac{1}{1-\mathsf{c}/3}\right)\frac{1}{3} + \sqrt{\frac{1}{1-\mathsf{c}/3}} \frac{R \sqrt{\mu} \|x_\mu^\star\|_{\mathbf{H}_\mu^{-1}(x_\mu^\star)}}{\mathsf{c}}.\]
Likewise:  
\[\frac{R \|x\|}{\mathsf{c}} \leq \frac{R \|x_\mu^\star\|}{\mathsf{c}} + \frac{1}{3}\overline{\phi}(-\log(1 - \mathsf{c}/3)).\]
\end{proof}

We can get the following simpler bounds.


\begin{corollary}[Application to $\mathsf{c} = \frac{1}{7}$]\label{cor:bounds_q} Applying \cref{lm:bounds_q} to $\mathsf{c} = \frac{1}{7}$, we get the following bounds.
Let $\mu >0$.
\begin{itemize}
\item For any $x \in {\mathcal{H}}$, if $ x \in \mathsf{D}_\mu(\mathsf{c}/3)$, then the following holds:
\[\frac{7 \mu \|x\|_{\mathbf{H}^{-1}_\mu(x)}}{ \mathsf{r}_\mu(x)} \leq 1 + \frac{8\|\mu x_\mu^\star\|_{\mathbf{H}_\mu^{-1}(x_\mu^\star)}}{  \mathsf{r}_\mu(x_\mu^\star)}. \]
\item For any $\mathsf{c} < 1$, $x \in {\mathcal{H}}$, if $\frac{R \nu_\mu(x)}{\sqrt{\mu}} \leq \frac{\mathsf{c}}{3}$, then the following hold:
\[ 7R \sqrt{\mu} \|x\|_{\mathbf{H}_\mu^{-1}(x)} \leq 1 +  8 R \sqrt{\mu} \|x_\mu^\star\|_{\mathbf{H}_\mu^{-1}(x_\mu^\star)}.\]
\[7R \|x\| \leq 7R \|x_\mu^\star\| + 1.\]
\end{itemize}
\end{corollary}




\subsection{Proof of main theorems \label{app:maintheorems}}

In this section, we bound the number of iterations of our scheme in different cases.

Recall the proposed globalization scheme in the paper, where $\texttt{ANM}_\rho(f,x,t)$ is a method performing $t$ successive $\rho$-relative approximate Newton steps of $f$ starting at $x$.  


\fbox{
\begin{minipage}[t]{0.9 \textwidth}
\textbf{Proposed Globalization Scheme}
\begin{center}
\textit{Phase I: Getting in the Dikin ellispoid of $f_\lambda$}
\end{center}
Start with $x_0 \in {\mathcal{H}}, \mu_0 > 0$, $t, T \in \mathbb{N}$ and $(q_k)_{k \in \mathbb{N}} \in (0,1]$.\\
For $k \in \mathbb{N}$\\
${}\qquad x_{k+1} \leftarrow \texttt{ANM}_\rho(f_{\mu_k},x_{k},t)$\\
${}\qquad \mu_{k+1} \leftarrow q_{k+1}\mu_{k}$\\
Stop when $\mu_{k+1} < \lambda$ and set $x_{last}\leftarrow x_k$.
$K \leftarrow k$
\begin{center}
\textit{Phase II: reach a certain precision starting from  inside the Dikin ellipsoid}
\end{center}
Return $\widehat{x} \leftarrow \texttt{ANM}_\rho(f_{\lambda},x_{last},T)$
\end{minipage}
}

\vspace*{0.3cm}

Throughout this section, we will denote with $K$ the value of $k$ when the scheme stops, i.e. the first value of $k$ such that $\mu_{k+1} < \lambda$. 

\paragraph{Adaptive methods } We start by presenting an adaptive way to select $\mu_{k+1}$ from $\mu_k$, with theoretical guarantees. The main result is the following.

\begin{proposition}[Adaptive, simple version]\label{prp:adapt}
Assume that we perform phase I starting at $x_0$ such that 
\[\frac{R \nu_{\mu_0}(x_0)}{\sqrt{\mu_0}} \leq \frac{1}{7}.\]
Assume that at each step $k$, we compute $x_{k+1}$ using $t = 2$ iterations of the $\rho$-relative approximate Newton method. 
Then if at each iteration, we set: 
\[\mu_{k+1} = q_{k+1} ~\mu_{k},\qquad q_{k+1}:= \frac{\frac{1}{3} + 7R \|x_{k+1}\|}{1 + 7R \|x_{k+1}\|} .\]
Then the following hold:
\paragraph{1.} $\forall k \leq K+1,~\frac{R \nu_{\mu_k}(x_k)}{\sqrt{\mu_k}} \leq \frac{1}{7}$ .
\paragraph{2.} The decreasing parameter $q_{k+1}$ is bounded above before reaching $K$:
$$\forall k \leq K,~ q_{k+1} \leq  \frac{\frac{4}{3} + 7R \|x^\star_{\mu_k}\|}{2 + 7R \|x^\star_{\mu_k}\|} \leq  \frac{\frac{4}{3} + 7R \|x^\star_{\lambda}\|}{2 + 7R \|x^\star_{\lambda}\|}.$$
\paragraph{3. } $K$ is finite, 
$$ K \leq \left\lfloor \frac{\log \frac{\mu_0}{\lambda}}{\log \frac{2 + 7R \|x^\star_{\lambda}\|}{\frac{4}{3} + 7R \|x^\star_{\lambda}\|}}\right\rfloor \leq \left\lfloor \left(3 + 11R \|x_\lambda^{\star}\|\right)\log \frac{\mu_0}{\lambda}\right\rfloor,$$ 
and $\frac{R \nu_\lambda(x_{K+1})}{\sqrt{\lambda}} \leq \frac{1}{7} $.
\end{proposition} 

\begin{proof} 

Let us prove the three points one by one.
\paragraph{1.} This is easily proved by induction, the keys to the induction hypothesis being:
\begin{itemize}
    \item  Using the induction hypothesis, $x_k \in \mathsf{D}_{\mu_k}(\mathsf{c})$ and hence, using \cref{prp:approximate_newton_scheme} shows that after two iterations of the approximate Newton scheme, $\frac{\nu_{\mu_k}(x_{k+1})}{\nu_{\mu_k}(x_k)}\leq \frac{1}{3}$ which implies $\frac{R \nu_{\mu_k}(x_{k+1})}{\sqrt{\mu_k}} \leq \frac{\mathsf{c}}{3}$.
    \item Now using \cref{lm:next_la}, we see that that since 
    $$7 R \|x_{k+1}\| = \frac{R \|x_{k+1}\|}{\mathsf{c}} \geq  \frac{R \sqrt{\mu_k} \|x_{k+1}\|_{\mathbf{H}^{-1}_{\mu_k}(x_{k+1})}}{\mathsf{c} },$$
    the hypotheses to guarantee the bound for $q_{k+1}$ hold, hence 
    \[\frac{R \nu_{\mu_{k+1}}(x_{k+1})}{\sqrt{\mu_{k+1}}} \leq \mathsf{c}.\]
\end{itemize}
\paragraph{2.} Using the second bullet point of \cref{cor:bounds_q}, we see that the previous point implies
\[\forall k \leq K,~ 7R \|x_{k+1}\| \leq 7 R \|x^\star_{\mu_k}\| + 1 \implies q_{k+1} \leq \frac{4/3 + 7 R \|x^\star_{\mu_k}\|}{2 + 7 R \|x^\star_{\mu_k}\|}.\]
Now using the fact that for any $k \leq K$, $\mu_k > \lambda$, we can use the simple fact that $\|x^\star_\lambda\| \geq \|x^\star_{\mu_k}\|$ to get the desired bound for $q_{k+1}$. 
\paragraph{3.} Using the previous point clearly shows the following bound: 
\[\forall k \leq K+1,~\mu_{k} \leq  \left(\frac{\frac{4}{3} + 7R \|x^\star_{\lambda}\|}{2 + 7R \|x^\star_{\lambda}\|}\right)^{k}\mu_0.\]
As this clearly converges to $0$ when $k$ goes to infinity, $K$ is necessarily finite. Applying this for $k = K$, we see that:
\[\lambda \leq \mu_K \leq \left(\frac{\frac{4}{3} + 7R \|x^\star_{\lambda}\|}{2 + 7R \|x^\star_{\lambda}\|}\right)^{K}\mu_0.\]
This shows that $K \leq \frac{\log \frac{\mu_0}{\lambda}}{\log\frac{2 + 7R \|x^\star_{\lambda}\|}{\frac{4}{3} + 7R \|x^\star_{\lambda}\|} }$.

The final bound is obtained noting that 
\[\frac{2 + 7R \|x^\star_{\lambda}\|}{\frac{4}{3} + 7R \|x^\star_{\lambda}\|} = 1 + \frac{1}{t},\qquad t = 2 + \frac{21}{2}R \|x^\star_{\lambda}\|,\]
and using the classical bound:
\[\frac{1}{\log(1 + \frac{1}{t})} \leq t+1.\]
Finally, the fact that $\frac{R\nu_\lambda(x_{K+1})}{\sqrt{\lambda}} \leq \mathsf{c}$ is just a consequence of the fact that $\mu_{K+1} \leq \lambda \leq \mu_K$ and thus that $\lambda = q \mu_K$ with $q \geq q_{K+1}$, which is shown to satisfy the condition in \cref{lm:next_la}. Hence, the lemma holds not only for $\mu_{K+1}$ but also for $\lambda$.
\end{proof}


\begin{remark}[$\mu_0$] \label{rmk:mu0}
In the previous proposition, we assume start at $x_0,\mu_0$ such that 
\[\frac{R \nu_{\mu_0}(x_0)}{\sqrt{\mu_0}} \leq \frac{1}{7}.\]
A simple way to have such a pair is simply to select:
\[x_0 = 0,~ \mu_0 = 7R \| \nabla f(0) \|,\]
since $\frac{R \nu_{\mu_0}(x_0)}{\sqrt{\mu_0}}  =\frac{R \|\nabla f(0)\|_{\mathbf{H}_{\mu_0}^{-1}(0)}}{\sqrt{\mu_0}} \leq \frac{R \|\nabla f(0)\|}{\mu_0} $.
\end{remark}


Alternatively, if one can approximately compute $\|x\|_{\mathbf{H}^{-1}_\mu(x)}$, one can propose the following variant, whose proof is completely analogous. 


\begin{proposition}[Adaptive, small variant version]\label{prp:adap_var}
Assume that we perform phase I starting at $x_0$ such that 
\[\frac{R \nu_{\mu_0}(x)}{\sqrt{\mu_0}} \leq \frac{1}{7}.\]
Then if at each iteration, we set: 
\[t_{k+1} = 7\sqrt{\frac{7}{6}}R \sqrt{\mu_k }\sqrt{x_{k+1}\cdot s_{k+1}},s_{k+1} \in \lso(\mathbf{H}_{\mu_k}(x_{k+1}),x_{k+1},\frac{1}{7}),\]
and 
\[\mu_{k+1} = q_{k+1} ~\mu_{k},\qquad q_{k+1}:= \frac{\frac{1}{3} + t_{k+1} }{1 + t_{k+1}} .\]
Then the following hold:
\paragraph{1.} $\forall k \leq K,~\frac{R \nu_{\mu_k}(x_k)}{\sqrt{\mu_k}} \leq \frac{1}{7}$.
\paragraph{2.} The decreasing parameter $q_{k+1}$ is bounded above before reaching $K$:
\[\forall k \leq K,~q_{k+1} \leq \sup_{\mu_0 \geq \mu \geq \lambda}\frac{\frac{7}{3} + 10 R \sqrt{\mu}\|x^\star_{\mu}\|_{\mathbf{H}^{-1}_{\mu}(x^\star_{\mu})}}{3 + 10R \sqrt{\mu}\|x^\star_{\mu}\|_{\mathbf{H}^{-1}_{\mu}(x^\star_{\mu})}} \leq \frac{\frac{7}{3} + 10 R \|x_\lambda^{\star}\|}{3 + 10R \|x_\lambda^{\star}\|}.\]
\paragraph{3. } $K$ is finite, 
$$ K   \leq \left(\frac{9}{2} + 15\sup_{\lambda \leq \mu \leq \mu_0}R \sqrt{\mu}\|x^\star_{\mu}\|_{\mathbf{H}^{-1}_{\mu}(x^\star_{\mu})}\right)\log \frac{\mu_0}{\lambda},$$ 
and $\frac{R \nu_\lambda(x_{K+1})}{\sqrt{\lambda}} \leq \frac{1}{7} $.
\end{proposition} 

\begin{proof} 
The main thing to note is that because of the properties of $\frac{1}{7}$-approximations, if $s_{k+1} \in \lso(\mathbf{H}_{\mu_k}(x_{k+1}),x_{k+1},\frac{1}{7})$,
\[(1-\frac{1}{7})\|x_{k+1}\|_{\mathbf{H}^{-1}_{\mu_k}(x_{k+1})}^2 \leq x_{k+1}\cdot s_{k+1} \leq (1+\frac{1}{7})\|x_{k+1}\|_{\mathbf{H}^{-1}_{\mu_k}(x_{k+1})}^2 .\]
Hence,
\[\|x_{k+1}\|_{\mathbf{H}^{-1}_{\mu_k}(x_{k+1})} \leq \sqrt{\frac{7}{6}}\sqrt{x_{k+1}\cdot s_{k+1}} \leq \sqrt{\frac{4}{3}}\|x_{k+1}\|_{\mathbf{H}^{-1}_{\mu_k}(x_{k+1})}.\]
Hence, $t_{k+1} \geq 7 R \sqrt{\mu_k}\|x_{k+1}\|_{\mathbf{H}^{-1}_{\mu_k}(x_{k+1})}$, and we can apply \cref{lm:next_la} to get the first point.\\\
To get the second point, we bound $t_{k+1}$ above: 
\[t_{k+1} \leq 7 \sqrt{\frac{4}{3}} R \sqrt{\mu_k}\|x_{k+1}\|_{\mathbf{H}^{-1}_{\mu_k}(x_{k+1})}.\]
Now use \cref{cor:bounds_q} to find:
\[t_{k+1} \leq  \sqrt{\frac{4}{3}}\left( 1 + 8R \sqrt{\mu_k}\|x^\star_{\mu_k}\|_{\mathbf{H}^{-1}_{\mu_k}(x^\star_{\mu_k})}\right) \leq 2 + 10 R \sqrt{\mu_k}\|x^\star_{\mu_k}\|_{\mathbf{H}^{-1}_{\mu_k}(x^\star_{\mu_k})}.\]
Thus, 
\[q_{k+1} \leq \frac{\frac{7}{3} + 10 R \sqrt{\mu_k}\|x^\star_{\mu_k}\|_{\mathbf{H}^{-1}_{\mu_k}(x^\star_{\mu_k})}}{3 + 10R \sqrt{\mu_k}\|x^\star_{\mu_k}\|_{\mathbf{H}^{-1}_{\mu_k}(x^\star_{\mu_k})}}.\]
Note that as long as $k \geq K$, 
\[q_{k+1} \leq \sup_{\mu \geq \lambda}\frac{\frac{7}{3} + 10 R \sqrt{\mu}\|x^\star_{\mu}\|_{\mathbf{H}^{-1}_{\mu}(x^\star_{\mu})}}{3 + 10R \sqrt{\mu}\|x^\star_{\mu}\|_{\mathbf{H}^{-1}_{\mu}(x^\star_{\mu})}} \leq \frac{\frac{7}{3} + 10 R \|x_\lambda^{\star}\|}{3 + 10R \|x_\lambda^{\star}\|}.\]
This guarantees convergence. \\\

For the last point, the proof is exactly the same as in the previous proposition. 

\end{proof}

\paragraph{General non-adaptive result.}

As mentioned in the core of the article, in practice, we do not select $q_{k+1}$ at each iteration using a safe adaptative value, but rather decrease $\mu_{k+1} = q \mu_k$ with a constant $q$, which we see as a parameter to tune. The following result shows that for $q$ large enough, this is justified, and that the lower bound we get for $q$ depends on the radius of the Dikin ellipsoid $\mathsf{r}_\mu(x)$, instead of $\frac{\sqrt{\mu}}{R}$ in the previous bounds, which is somewhat finer and shows that if the data is structured such that this radius is very big, then $q$ might actually be very small.

\begin{proposition}[Fixed $q$]\label{prp:fixed_q}
Assume that we perform phase I starting at $x_0$ such that 
\[x_0 \in \mathsf{D}_{\mu_0}(\frac{1}{7}).\]
Assume we perform the method with a fixed $q_{k+1} = q$, satisfying
\[q \geq \sup_{\lambda \leq \mu \leq \mu_0}\frac{\frac{4}{3} + 8\frac{\mu \|x^\star_{\mu}\|_{\mathbf{H}_{\mu}^{-1}(x^\star_{\mu})}}{\mathsf{r}_{\mu}(x^\star_{\mu})}}{2 + 8\frac{\mu \|x^\star_{\mu}\|_{\mathbf{H}_{\mu}^{-1}(x^\star_{\mu})}}{\mathsf{r}_{\mu}(x^\star_{\mu})}}.\]
Then the following hold:
\paragraph{1.} $\forall k \leq K+1,~ x_k \in \mathsf{D}_{\mu_k}(\frac{1}{7})$.
\paragraph{2. } $K$ is finite, 
$$ K   \leq \frac{1}{1-q}\log \frac{\mu_0}{\lambda}, $$ 
and $x_{K+1} \in \mathsf{D}_{\lambda}(\frac{1}{7}) $.
\end{proposition} 

\begin{proof} 
Let us prove the two points.

\paragraph{1.} Let us prove the result by induction. The initialization is trivial. Now assume $x_k \in \mathsf{D}_{\mu_k}(\frac{1}{7})$. Performing two iterations of the approximate Newton method guarantees that 
\[x_{k+1} \in \mathsf{D}_{\mu_k}(\frac{1}{21}),\]
as show in \cref{prp:approximate_newton_scheme}.
Now using \cref{lm:next_la}, we see that $x_{k+1} \in \mathsf{D}_{q \mu_k}(\frac{1}{7})$, provided that 

\[q \geq \frac{\frac{1}{3} + \frac{7 \mu_k \|x_{k+1}\|_{\mathbf{H}_{\mu_k}^{-1}(x_{k+1})}}{\mathsf{r}_{\mu_k}(x_{k+1})}}{1 + \frac{7 \mu_k \|x_{k+1}\|_{\mathbf{H}_{\mu_k}^{-1}(x_{k+1})}}{\mathsf{r}_{\mu_k}(x_{k+1})}} .\]

Now using \cref{cor:bounds_q}, we get that 
\[\frac{7 \mu_k \|x_{k+1}\|_{\mathbf{H}_{\mu_k}^{-1}(x_{k+1})}}{\mathsf{r}_{\mu_k}(x_{k+1})} \leq 1 + \frac{8\mu_k \|x^\star_{\mu_k}\|_{\mathbf{H}_{\mu_k}^{-1}(x^\star_{\mu_k})}}{\mathsf{r}_{\mu_k}(x^\star_{\mu_k})} \leq 1 + 8 \sup_{\lambda \leq \mu \leq \mu_0} \frac{\mu \|x^\star_{\mu}\|_{\mathbf{H}_{\mu}^{-1}(x^\star_{\mu})}}{\mathsf{r}_{\mu}(x^\star_{\mu})}.\]
Hence the result. 

\paragraph{2.} This point just follows, using the bound $\frac{1}{\log \frac{1}{q}} \leq \frac{1}{1-q}$.


\end{proof} 

\subsection{Proof of \cref{thm:easy_first_phase} \label{app:proof_main_thm}}

Using \cref{rmk:mu0}, the fact that $x_0 = 0$ and  $\mu_0 = 7 R \|\nabla f(0)\|$, as well as the hypotheses of the theorem, we can apply \cref{prp:adapt}, and show that the number of steps $K$ performed in the first phase is bounded: 

\[K \leq \left\lfloor (3 + 11 R \|x_\lambda^{\star}\|)\log(7R \|\nabla f(0)\|/\lambda)\right\rfloor.\]
Moreover, this proposition also shows that $R \nu_\lambda(x_{last})/\sqrt{\lambda} \leq \frac{1}{7}$. Hence, we can use \cref{prp:approximate_newton_scheme}: if 
\[t\geq T = \left\lceil \log_2\sqrt{\frac{\lambda \varepsilon^{-1}}{R^2}} \right\rceil  \geq \left\lceil \log_2\frac{\nu_\lambda(x_{last})}{\sqrt{\varepsilon}} \right\rceil, \] 
then it holds $\nu_\lambda(\hat{x}) \leq \sqrt{\varepsilon}$ and $f_\lambda(\hat{x}) - f_\lambda(x_\lambda^{\star}) \leq \varepsilon$. 
\qed{}



\newpage

\section{Non-parametric learning with generalized self-concordant functions\label{app:kernels}}
In this section, the aim is to provide a fast algorithm in the case of Kernel methods which achieves the optimal statistical guarantees. 

\subsection{General setting and assumptions, statistical result for regularized ERM.\label{app:kern-setting}}


In this section, we consider the supervised learning problem of learning a predictor $f:\mathcal{X} \rightarrow \mathcal{Y}$ from training samples $(x_i,y_i)_{1\leq i \leq n}$ which we assume to be realisations from a certain random variable $ Z = (X,Y) \in \mathcal{Z} = \mathcal{X}\times \mathcal{Y} $ whose distribution is $\rho$.  In what follows, for simplification purposes, we assume $\mathcal{Y} = \mathbb{R}$; however, this analysis can easily be adapted (although with heavier notations) to the setting where $\mathcal{Y} = \mathbb{R}^p$. Our aim is to compute the predictor of minimal generalization error 
\eqal{\label{pb:learning}\inf_{f \in {\mathcal{H}}}{L(f):= \Expb{z \sim \rho}{\ell_z(f(x))}},}
where ${\mathcal{H}}$ is a space of candidate solutions and $\ell_z:  \mathbb{R} \rightarrow \mathbb{R}$ is a loss function comparing the prediction $f(x)$ to the objective $y$.  

\paragraph{Kernel methods.}

Kernel methods consider a space of functions ${\mathcal{H}}_K$ implicitly constructed from a symmetric positive semi-definite Kernel  $K: \mathcal{X} \times \mathcal{X} \rightarrow$ and whose basic functions are the $K_x: t \in \mathcal{X} \mapsto K(x,t)$ and the linear combinations of such functions 
$f = \sum_{j=1}^m{\alpha_j K_{x_j}}$. \\

It is endowed with a scalar product such that: $\forall x_1,x_2 \in \mathcal{X},~ K_{x_1}\cdot K_{x_2} = K(x_1,x_2)$, and as a consequence, ${\mathcal{H}}_K$ satisfies the self-reprocucing property:
 $$ \forall x \in \mathcal{X},~ \forall f \in {\mathcal{H}},~f(x) = \langle f,K_x\rangle_{{\mathcal{H}}}.$$  In order to find a good predictor for \cref{pb:learning}, the following estimator, called the regularized ERM estimator, is often computed:
$$\widehat{f}_\lambda:= \argmin_{f \in {\mathcal{H}}} \widehat{L}_\lambda(f):= \frac{1}{n}\sum_{i=1}^n{\ell_{z_i}(f(x_i))} + \frac{\lambda}{2}\|f\|_{{\mathcal{H}}}^2.$$

The properties of this estimator have been studied in \cite{devito} for the square loss and \cite{marteau2019} for generalized self-concordant functions. In \cref{app:proj_stat}, we recall the full setting of \cite{marteau2019}, and extend it to include the statistical properties of the projected problem. 



\paragraph{Assumptions}

In this section, we will make the following assumptions, which are reformulations of the assumptions of \cite{marteau2019}, which we recall in \cref{app:proj_stat}, in order to have the statistical properties of the regularized ERM. First, we assume that the $(x_i,y_i)$ are i.i.d. samples.



\begin{restatable}[i.i.d. data]{ass}{asmiid1}
\label{asm:iid1}
The samples $(z_i)_{1 \leq i \leq n}=(x_i,y_i)_{1\leq i \leq n}\in\mathcal{Z}^n$ are independently and identically distributed according to $\rho$. 
\end{restatable}
In the case where $\mathcal{Y} = \mathbb{R}$, we make the following assumptions on the loss, which leads to the self concordance of the mappings $f \mapsto \ell_z(f(x))$ and that of $L$, $\widehat{L}$...

\begin{restatable}[Technical assumptions]{ass}{asmtech1}
\label{asm:asmtech1}
The mapping $(z,t) \in \mathcal{Z} \times \mathbb{R} \mapsto \ell_z(t)$ is measurable. Moreover, 
\begin{itemize}
    \item there exists $\mathsf{R}_{\ell} < \infty$ such that for all $z \in \supp(Z)$, 
    \[\forall t \in \mathbb{R},~ |\ell_z^{(3)}(t)|\leq \mathsf{R}_{\ell} \ell_z^{\prime \prime}(t),\]
    \item the random variables $|\ell_Z(0)|, |\ell^{\prime}_Z(0)|, | \ell^{\prime \prime}_Z(0)|$ are are bounded;
    \
    \item The kernel is bounded, i.e. $\forall x \in \supp(X),~K(x,x) \leq \kappa^2$ for a certain $\kappa$. 
\end{itemize}
\end{restatable}

Using these assumptions, we see that the following properties are satisfied. Define $L_z(f): = \ell_z(f(x))$. Then the $L_z$ satisfy the following properties:
\begin{itemize}
    \item For any $z \in \mathcal{Z}$, $(L_z,\left\{\mathsf{R}_{\ell} K_{x}\right\})$ is a generalized self-concordant function in the sense of \cref{df:first_definitions_sc}.
    \item The mapping $(z,f) \in \mathcal{Z} \times {\mathcal{H}} \mapsto L_z(f)$ is measurable;
    \item the random variables $\|L_Z(0)\|, \|\nabla L_Z(0)\|, \Tr(\nabla^2 L_Z(0))$ are bounded by $|\ell_Z(0)|$, $\kappa|\ell^{\prime}_Z(0)|$, $\kappa^2 |\ell_Z^{\prime \prime}(0)|$;
    \item $\mathcal{G}:= \left\{\mathsf{R}_{\ell} K_{x}~:~z \in \supp(Z)\right\}$ is a bounded subset of ${\mathcal{H}}$, bounded by $R = \mathsf{R}_{\ell}\kappa$.
\end{itemize}

This shows that \cref{asm:gen_sc} and \cref{asm:asmtech} are satisfied by the $L_z$ and hence, using \cref{prp:scProblems} in the next appendix, $L$ is well-defined, generalized self-concordant with $\mathcal{G}$. Moreover, the empirical loss 
\[\widehat{L} = \frac{1}{n}\sum_{i=1}^n{L_{z_i}},\]
is also generalized self-concordant with $\widehat{\mathcal{G}}:= \left\{\mathsf{R}_{\ell} K_{x_i}~:~1 \leq i \leq n\right\}$.\\\

Finally, as in \cref{app:proj_stat}, we make an assumption on the regularity of the problem; namely, we assume that a solution to the learning problem exists in ${\mathcal{H}}$.

\begin{restatable}[Existence of a minimizer]{ass}{asmminimizer1}
\label{asm:minimizer1}
There exists $f^\star \in {\mathcal{H}}$ such that $ L(f^\star) = \inf_{f \in {\mathcal{H}}} L(f).$
\end{restatable}







We adopt all the notations from \cref{app:proj_stat}, doing the distinction between expected an empirical problems by adding a $\widehat{\cdot}$ over the quantities related to the empirical problem. We continue using the standard notations for $L$: for any $f \in {\mathcal{H}}$ and $\lambda >0$,
\[L_\lambda(f) = L(f) + \frac{\lambda}{2}\|f\|^2,\qquad \widehat{L}_\lambda(f) = \widehat{L}(f) + \frac{\lambda}{2}\|f\|^2  \]
\[\mathbf{H}(f) = \nabla^2 L(f),\qquad \mathbf{H}_\lambda(f) = \nabla^2 L_\lambda(f) = \mathbf{H}(f) + \lambda \mathbf{I} \]
\[\widehat{\mathbf{H}}(f) = \nabla^2 \widehat{L}(f),\qquad \widehat{\mathbf{H}}_\lambda(f) = \nabla^2 \widehat{L}_\lambda(f) = \widehat{\mathbf{H}}(f) + \lambda \mathbf{I} \]
Recall that $\widehat{f}_\lambda$ is defined as the minimizer of $\widehat{L}_\lambda$. 

Define the following bounds on the second order derivatives: 
\[\forall f \in {\mathcal{H}},~ \mathsf{b}_2(f) = \sup_{z \in \supp(Z)}\ell^{\prime \prime}_z(f(x)).\]
\paragraph{Statistical properties of the estimator}

The statistical properties of the estimator $\widehat{f}_\lambda$ have been studied in \cite{marteau2019} in the case of generalized self concordance, an are reported in the main lines in \cref{app:proj_stat}. The statistical rates of this estimator and the optimal choice of $\lambda$ is determined by two parameters, defined in \cref{df:main_quant_stat} and which we adapt to the Kernel problem here. 


\begin{itemize}
    \item the \emph{bias} $\mathsf{b}_\lambda = \|\mathbf{H}_\lambda(f^\star)^{-1/2} \nabla L_\lambda(f^\star)\| =\lambda \|f^\star\|_{\mathbf{H}^{-1}_\lambda(f^\star)}$, which characterizes the regularity of the optimum. The faster $\mathsf{b}_\lambda$ decreases to zero, the more regular $f^\star$ is.
    \item the \emph{effective dimension} 
    \eqal{\label{eq:deff-statistics}
    \mathsf{df}_\lambda = \Exp{\|\mathbf{H}_\lambda(f^\star)^{-1/2}\nabla L_Z(f^\star)\|^2}.
    }
    This quantity characterizes the size of the space ${\mathcal{H}}$ with respect to the problem; the slower it explodes as $\lambda$ goes to zero, the smaller the size of ${\mathcal{H}}$. 
\end{itemize}

For more complete explanations on the meaning of these quantities, we refer to \cite{marteau2019}. \\\


Moreover, as mentioned in \cref{df:main_quant_stat}, one can define
\eqal{\label{eq:bb}\mathsf{B}_1^\star:= \sup_{z \in \supp(Z)}{\|\nabla L_z(f^\star)\|},~ \mathsf{B}_2^\star:= \sup_{z \in \supp(Z)}{\Tr(\nabla^2 L_z(f^\star))},~ \mathsf{Q}^\star = \frac{\mathsf{B}_1^\star}{\sqrt{\mathsf{B}_2^\star}}, ~ \mathsf{b}_2^\star = \mathsf{b}_2(f^\star).}



We assume the following regularity condition on the minimizer $f^\star$, in order to get statistical bounds.  

\begin{restatable}[Source condition]{ass}{asmsource1}
\label{asm:source}
There exists $r >0 $ and $g \in {\mathcal{H}}$ such that $f^\star = \mathbf{H}^r(f^\star)g$. This implies the following decrease rate of the bias:
\[\mathsf{b}_\lambda \leq \mathsf{L} \lambda^{1/2+r},\qquad \mathsf{L} = \|g\|_{{\mathcal{H}}}.\]
\end{restatable}

This is a stronger assumption than the existence of the minimizer as $r > 0$ is crucial for our analysis. 

We also quantify the effective dimension $\mathsf{df}_\lambda$: (however, since it always holds for $\alpha  =1$, this is not, strictly speaking, an additional assumption). 

\begin{restatable}[Effective dimension]{ass}{asmeffdim}
\label{asm:deff}
The effective dimension decreases as $\mathsf{df}_\lambda \leq \mathsf{Q} \lambda^{-1/\alpha}$. 
\end{restatable}


If these two assumptions hold, define: 

\[\beta = \frac{\alpha}{1 + \alpha(1+2r)},\qquad \gamma = \frac{(1+2r)\alpha}{1 + \alpha(1+2r)}.\]





Under these assumptions, one can obtain the following statistical rates (which can be found in \cite{marteau2019} or in \cref{cor:bounds_source_capacity}).


 \begin{proposition}\label{prp:bounds_source_capacity_K}
Let $\delta \in (0,1/2]$. Under \cref{asm:iid1,asm:deff,asm:asmtech1,asm:minimizer1,asm:source}, when $n \geq N$ and $\lambda = (C_0/n)^{\beta}$, then with probability at least $1-2\delta$,
\[L(\widehat{f}_\lambda) - L(f^\star) \leq C_1 n^{-\gamma}\log \frac{2}{\delta},\]
with $C_0 = 256 (\mathsf{Q}/\mathsf{L})^2,~C_1 = 8 (256)^{\gamma}~(\mathsf{Q}^{\gamma} ~ 
    \mathsf{L}^{1-\gamma})^2$ and $N$ defined in \cite{marteau2019}, and satisfying
$N = O(\poly(\mathsf{B}_1^\star,\mathsf{B}_2^\star,\mathsf{L},\mathsf{Q},R,\log(1/\delta)))$. 
\end{proposition} 





\subsection{Reducing the dimension: projecting on a subspace using Nystr\"om sub-sampling.\label{app:kern-rand-proj}}

\paragraph{Computations}

Using a representer theorem, one of the key properties of Kernel spaces is that, owing to the reproducing property, 
$$\widehat{f}_\lambda \in {\mathcal{H}}_n:= \left\{ \sum_{i=1}^n{\alpha_i K_{x_i}}~:~(\alpha_i) \in \mathbb{R}^n\right\}.$$
This means that solving the regularized empirical problem can be turned into a finite dimensional problem in $\alpha$. Indeed
 $\widehat{f}_\lambda = \sum_{i=1}^n{\alpha_i K_{x_i}}$ where $\alpha =  (\alpha_i)_{1\leq i\leq n}$ is the solution to the following problem:
\[\alpha = \argmin_{\alpha \in \mathbb{R}^n}{\frac{1}{n}\sum_{i=1}^n{\ell_{z_i}(\alpha^\top {\bf K}_{nn}e_i)} + \frac{\lambda}{2}\alpha^\top {\bf K}_{nn} \alpha},\qquad {\bf K}_{nn} = (K(x_i,x_j))_{1 \leq i,j\leq n}\in\mathbb{R}^{n \times n}.\]

The previous problem is usually too costly to solve directly for large values of $n$, both in time and memory, because of the operations involving ${\bf K}_{nn}$.  A solution consists in looking for a solution in a smaller dimensional sub-space ${\mathcal{H}}_M$ constructed from sub-samples of the data $\left\{ \tilde{x}_1,...,\tilde{x}_M\right\} \subset \left\{x_1,...,x_n\right\}$:
\[{\mathcal{H}}_M:= \left\{\sum_{j=1}^M{\tilde{\alpha}_j K_{\tilde{x}_j}}~:~ \tilde{\alpha} \in \mathbb{R}^M\right\}.\]

In this case, the minimizer $\widehat{f}_{M,\lambda}  = \argmin_{f \in {\mathcal{H}}_M}{\widehat{L}_\lambda(f)}$ can be written $\widehat{f}_{M,\lambda} = \sum_{j=1}^M{\tilde{\alpha}_j K_{\tilde{x}_j}}$, where $\tilde{\alpha}$ is the solution to the following problem:
\[\tilde{\alpha} = \argmin_{\alpha \in \mathbb{R}^M}{\frac{1}{n}\sum_{i=1}^n{\ell_{z_i}(\alpha^\top {\bf K}_{Mn}e_i)} + \frac{\lambda}{2}\alpha^\top {\bf K}_{MM} \alpha}, \]
where 
\[{\bf K}_{nM} = (K(x_i,\tilde{x}_j))_{\substack{1 \leq i \leq n \\ 1 \leq j \leq M}},~{\bf K}_{Mn} = {\bf K}_{nM}^\top,~ {\bf K}_{MM}:= (K(\tilde{x}_i,\tilde{x}_j))_{1 \leq i,j \leq M}.\]


Let $\bf T$ be an upper triangular matrix such that ${\bf T}^\top {\bf T} = {\bf K}_{MM}$. One can re-parametrize the previous problem in the following way. For any $\beta \in \mathbb{R}^M$, define $f_\beta = \sum_{j=1}^M{[{\bf T}^{\dagger}\beta]_j ~K_{\tilde{x}_j}}$. This implies in particular that $\|f_\beta\|_{{\mathcal{H}}} = \|\beta\|_{\mathbb{R}^M}$. Then $\widehat{f}_{M,\lambda} = f_{\beta_{M,\lambda}}$, where 

\[\beta_{M,\lambda} = \argmin_{\beta \in \mathbb{R}^M}{\widehat{L}_{M,\lambda}(\beta):= \frac{1}{n}\sum_{i=1}^n{\ell_{z_i}(e_i^\top {\bf K}_{nM}{\bf T}^{\dagger}\beta)} + \frac{\lambda}{2}\|\beta\|^2}.\]

Using the properties the $\ell_z$, one easily shows that 
$ \beta \mapsto \ell_{z_i}(e_i^\top {\bf K}_{nM}{\bf T}^{\dagger}\beta)$ is $\left\{\mathsf{R}_{\ell} {\bf T}^{-\top}{\bf K}_{Mn}e_i\right\}$ generalized self-concordant, and $\|\mathsf{R}_{\ell} {\bf T}^{-\top}{\bf K}_{Mn}e_i\| \leq \mathsf{R}_{\ell} \sqrt{K(x_i,x_i)}$. Thus, $\widehat{L}_M$ is also generalized self-concordant, and the associated $\widehat{\mathcal{G}}_M$ is bounded by $R= \mathsf{R}_{\ell} \kappa$. It will therefore be possible to apply the second order scheme presented in this paper to approximately compute $\beta_{M,\lambda}$. 


\paragraph{Statistics}

Let $\widehat{\nu}_{\lambda,M}(\beta)$ denote the Newton decrement of $\widehat{L}_{\lambda,M}$ at point $\beta$ and $\mathbf{P}_M$ denote the orthogonal projection on ${\mathcal{H}}_M$. Then the following statistical result shows that provided $\beta$ is a good enough approximation of the optimum, and provided ${\mathcal{H}}_M$ is large enough, then $f_\beta$ has the same generalization error as the empirical risk minimizer $\widehat{f}_\lambda$. 

Recall the following result proved in \cref{prp:proj_pb_stat_gen_part} in \cref{app:bounds_proj}. 
\begin{proposition}[Behavior of an approximation to the projected problem] \label{prp:proj_pb_stat_gen_kernels} Suppose that 
\cref{asm:iid1,asm:asmtech1,asm:minimizer1} are satisfied.
Let $n \in \mathbb{N}$, $\delta \in (0,1/2]$, $0 < \lambda \leq \mathsf{B}_2^\star$. Whenever
\[n \geq \triangle_1 \frac{\mathsf{B}_2^\star}{\lambda}\log \frac{8 \square_1^2\mathsf{B}_2^\star}{\lambda \delta},\qquad \mathsf{C}_1 \sqrt{ \frac{ \mathsf{df}_\lambda \vee (\mathsf{Q}^\star)^2}{n} ~\log \frac{2}{\delta}}\leq \frac{\lambda^{1/2}}{R}, \qquad \mathsf{C}_1 \mathsf{b}_\lambda \leq \frac{\lambda^{1/2}}{R},\]
if
\[\|\mathbf{H}^{1/2}(f^\star)(\mathbf{I} - \mathbf{P}_M)\|^2 \leq \lambda \frac{\sqrt{2}}{480},~ 126 \widehat{\nu}_{M,\lambda}(\beta) \leq \frac{\lambda^{1/2}}{R}, \]

the following holds, with probability at least $1 - 2 \delta$.

\[L(f_{\beta}) - L(f^\star) \leq \mathsf{K}_1~ \mathsf{b}_\lambda^2 + \mathsf{K}_2~ \frac{\mathsf{df}_\lambda \vee (\mathsf{Q}^\star)^2}{n}~\log \frac{2}{\delta} + \mathsf{K}_3~ \widehat{\nu}^2_{M,\lambda}(\beta), \qquad R \|f_{\beta} - f^\star\|_{{\mathcal{H}}} \leq 10, \]

where $\mathsf{K}_1 \leq 6.0\mathrm{e}{4}$, $\mathsf{K}_2 \leq 6.0\mathrm{e}{6} $ and $\mathsf{K}_3 \leq 810$, $\mathsf{C}_1$ is defined in \cref{lm:technical_before_proj}, and the other constants are defined in \cref{thm:general-result}. 
\end{proposition} 

In particular, if we apply the previous result for a fixed $\lambda$, the following theorem holds (for a proof, see \cref{app:quantibounds}).
\begin{theorem}[Quantitative result with source $r > 0$]\label[theorem]{thm:quantisource_kernels} Suppose that 
\cref{asm:iid1,asm:deff,asm:asmtech1,asm:minimizer1,asm:source} are satisfied.
Let $n \geq N$ and $\delta \in (0,\frac{1}{2}]$. If $\lambda = \left( \left(\frac{\mathsf{Q}}{\mathsf{L}}\right)^{2}~\frac{1}{n}\right)^{\frac{\alpha}{\alpha(1+2r) + 1}}$, and if 
\[\|\mathbf{H}^{1/2}(f^\star)(\mathbf{I} - \mathbf{P}_M)\|^2 \leq \lambda \frac{\sqrt{2}}{480},~\widehat{\nu}_{M,\lambda}(\beta) \leq \mathsf{Q}^{\gamma} ~ 
    \mathsf{L}^{1-\gamma} n^{-\gamma/2},\]
    then with probability at least $1-2 \delta$,
    \[L(f_\beta) - L(f^\star) \leq \mathsf{K} \left(\mathsf{Q}^{\gamma} ~ 
    \mathsf{L}^{1-\gamma}\right)^2 ~~\frac{1}{n^{\gamma}} \log \frac{2}{\delta}, \qquad R\|f_\beta - f^\star\| \leq 10,\]
    where $N$ is defined in \cref{eq:Nvalue} and $\mathsf{K} \leq 7.0e6$.
\end{theorem}

The proof of the previous result is quite technical and can be found in \cref{app:proj_stat}, in \cref{thm:quantisource}.

\subsection{A note on sub-sampling techniques\label{app:kern-ny-examples}}
Let $Z$ be a random variable on a Polish space $\mathcal{Z}$ and $(v_z)_{z \in \mathcal{Z}}$ be a family of vectors in ${\mathcal{H}}$ such that $||v||_{L^\infty(Z)}:= \sup_{z \in \supp(Z)}\|v_z\| < \infty $ is bounded. Assume that $z_1,...,z_n$ are i.i.d. samples from $Z$.

Define the following trace class Hermitian operators: 

$$\mathbf{A} = \Exp{v_Z \otimes v_Z},~\widehat{\Ab} = \frac{1}{n}\sum_{i=1}^n{v_{z_i} \otimes v_{z_i}}.$$ 

Define 
\eqal{\label{eq:deg_freed}\mathcal{N}^{\mathbf{A}}(\lambda):= \Tr(\mathbf{A}_\lambda^{-1}\mathbf{A}),\qquad \mathcal{N}_{\infty}^{\mathbf{A}}(\lambda):= \sup_{z \in \supp(Z)}\|\mathbf{A}_\lambda^{-1/2}v_z\|^2 .}
We typically have:
\[\mathcal{N}^{\mathbf{A}}(\lambda) \leq \mathcal{N}_{\infty}^{\mathbf{A}}(\lambda) \leq \frac{\|v\|_{L^\infty(Z)}^2}{\lambda}.\]

We define the leverage scores associated to the points $z_i$ and $\mathbf{A}$: 
\eqal{\label{eq:lev_scores} \forall 1 \leq i \leq n,~\forall t > 0,~ l^{\mathbf{A}}_i(t) = \|\widehat{\Ab}^{-1/2}_{t}v_{z_i}\|^2 = n\left((\mathbf{G}_{nn} + t n \mathbf{I})^{-1}\mathbf{G}_{nn}\right)_{ii},}
where $\mathbf{G}_{nn} = (v_{z_i}\cdot v_{z_j})_{1 \leq i,j \leq n}$ denotes the Gram matrix associated to the family $v_{z_i}$. 

As in \cite{Rudi15}, definition 1, we give the following definition for leverage scores.
\begin{definition}[$q$-approximate leverage scores]
given $t_0$, a family $(\tilde{l}^{\mathbf{A}}_i(t))_{1\leq i \leq n}$ is said to be a family of $q$-approximate leverage scores with respect to $\mathbf{A}$ if  
\[\forall 1 \leq i \leq n,~ \forall t \geq t_0,~ \frac{1}{q}~ l^{\mathbf{A}}_i(t) \leq \tilde{l}^{\mathbf{A}}_i(t) \leq q~l^{\mathbf{A}}_i(t).\]
\end{definition}

We say that a subset of $m$ points $\left\{\tilde{z}_1,...,\tilde{z}_m\right\} \subset \left\{z_i ~:~1 \leq i\leq n\right\}$ is:
\begin{itemize}
    \item \textbf{Sampled using $q$-approximate leverage scores for $t$} if the $\tilde{z}_j  = z_{i_j}$ where the $i_j$ are $m$ i.i.d. samples from $\left\{1,...,n\right\}$ using the probability vector $p_i = \frac{\tilde{l}^{\mathbf{A}}_i(t)}{\sum_{\tilde{i}=1}^n{\tilde{l}^{\mathbf{A}}_{\tilde{i}}(t)}}$.
    In that case, we define $\widehat{\Ab}_{m}:= \frac{1}{m}\sum_{j=1}^m{\frac{1}{np_{i_j}} v_{\tilde{z}_j} \otimes v_{\tilde{z}_j}}$.
    \item \textbf{Sampled uniformly} if the $\left\{i_j~:~1 \leq j \leq m\right\}$ is a uniformly chosen subset of  $\left\{1,...,n\right\}$ of size $m$. 
    In this case, we define 
    $\widehat{\Ab}_{m}:= \frac{1}{m}\sum_{j=1}^m{ v_{\tilde{z}_j} \otimes v_{\tilde{z}_j}}$.
\end{itemize}

In \cref{app:sub_nystrom}, we present technical lemmas which allow us to show that if $m$ is large enough, the following hold:
\begin{itemize}
    \item $\|\mathbf{A}_\eta(\mathbf{I} - \mathbf{P}_m)\|^2 \leq 3 \eta$, where $\mathbf{P}_m$ is the orthogonal projection on the subspace induced by the $v_{\tilde{z}_j}$;
    \item $\widehat{\Ab}_{m,\lambda}$ is equivalent to $\widehat{\Ab}_\lambda$.
\end{itemize}


\begin{remark}[cost of computing $q$-approximate leverage scores]\label{rmk:cost_lev_scores}
In \cite{rudi2018fast}, one can show that the complexity of computing $q$-approximate leverage scores can be achieved in: 
$\mathsf{c}_{\texttt{samp}} = O(q^2 \mathcal{N}^{\mathbf{A}}(\lambda)^2 \min(n,1/\lambda))$ time (where a unit of time is a scalar product evaluation) and $O(\mathcal{N}^{\mathbf{A}}(\lambda)^2 + n)$ in memory.
\end{remark}

\subsection{Selecting the $M$ Nystr\"om points\label{app:selecting_nys}}

In order for \cref{thm:quantisource_kernels} to hold, we must subsample the $M$ points such as to guarantee $\|\mathbf{H}^{1/2}(f^\star)(\mathbf{I} - \mathbf{P}_M)\|^2  \leq \frac{\sqrt{2}\lambda}{480}$. 


Since we must sub-sample the $M$ points a priori, i.e. before performing the method, it is necessary to have sub-sampling schemes which do not depend heavily on the point. Define the covariance operator:
$$\mathbf{\Sigma} = \Exp{K_X \otimes K_X}.$$
Since $\mathbf{H}(f^\star) = \Exp{\ell^{\prime \prime}_Z(f(X)) ~K_X \otimes K_X }$, it is easy to see that $\mathbf{H}(f^\star)  \preceq \mathsf{b}_2^\star \mathbf{\Sigma}$. Note that for $\mathbf{\Sigma}$, since $\widehat{\mathbf{\Sigma}} = \frac{1}{n}\sum_{i=1}^n{K_{x_i} \otimes K_{x_i}}$, the leverage scores have the following form:
\[\forall 1 \leq i \leq n,~ l_i^{\mathbf{\Sigma}}(t) = n\left(({\bf K}_{nn} + \lambda n \mathbf{I})^{-1} {\bf K}_{nn}\right)_{ii}.\]


\begin{proposition}[Selecting Nystr\"om points]\label{prp:nystrom_select}
Let $\delta >0$. Let $\eta = \min(\|\mathbf{\Sigma}\|,\frac{\lambda \sqrt{2}}{1440 (\mathsf{b}_2^\star\vee 1)})$. Assume the samples  $\left\{\tilde{x}_1,...,\tilde{x}_M\right\}$ are obtained with one of the following.\\
\noindent{\bf 1.} $n \geq  M \geq \left(10 + 160 \mathcal{N}_{\infty}^\mathbf{\Sigma}(\eta)\right)\log \frac{8 \kappa^2}{\eta \delta}$ using uniform sampling;\\
\noindent{\bf 2.} $  M \geq \left(6 + 486 q^2 \mathcal{N}^\mathbf{\Sigma}(\eta)\right)\log \frac{8 \kappa^2}{\eta \delta}$ using $q$-approximate leverage scores with respect to $\mathbf{\Sigma}$ for $t = \eta$, $t_0 \vee \frac{19 \kappa^2}{n}\log \frac{n}{2 \delta} < \eta$, $n \geq 405 \kappa^2 \vee 67 \kappa^2 \log \frac{12 \kappa^2}{\delta}$.\\
Then it holds, with probability at least $1-\delta$:
\[\|\mathbf{\Sigma}_{\eta}^{1/2}(\mathbf{I} - \mathbf{P}_M)\| \leq 3 \eta \implies \|\mathbf{H}^{1/2}(f^\star)(\mathbf{I} - \mathbf{P}_M)\|^2 \leq \lambda \frac{\sqrt{2}}{480}.\]

\end{proposition} 

\begin{proof} 
The proof is a direct application of the lemmas in \cref{app:sub_nystrom}. Indeed, note that since $\mathbf{\Sigma} = \Exp{K_X \otimes K_X}$, then the results can be applied with $Z \leftarrow X$ and $v_z \leftarrow K_x$. Indeed, from \cref{asm:asmtech1}, it holds:
\[\sup_{x\in \supp(X)}\|K_x\|^2 \leq \kappa^2.\]
\end{proof} 

We can now combine \cref{prp:nystrom_select} and \cref{prp:proj_pb_stat_gen_kernels} to obtain the following statistical bounds for the optimizer of the projected Nystr\"om problem $\beta_{M,\lambda}$.

\begin{theorem} \label{thm:nystrom_estimator_bounds}
Suppose that 
\cref{asm:iid1,asm:asmtech1,asm:minimizer1} are satisfied.
Let $n \in \mathbb{N}$, $\delta \in (0,1/2]$, $0 < \lambda \leq \mathsf{B}_2^\star \wedge 720\sqrt{2} (\mathsf{b}_2^\star\vee 1)\|\mathbf{\Sigma}\| $. Assume
\[n \geq \triangle_1 \frac{\mathsf{B}_2^\star}{\lambda}\log \frac{8 \square_1^2\mathsf{B}_2^\star}{\lambda \delta},\qquad \mathsf{C}_1 \sqrt{  \frac{\mathsf{df}_\lambda \vee (\mathsf{Q}^\star)^2}{n} ~\log \frac{2}{\delta}} \leq \frac{\lambda^{1/2}}{R},\qquad  \mathsf{C}_1 \mathsf{b}_\lambda \leq \frac{\lambda^{1/2}}{R},\]
Let $\eta = \frac{\lambda \sqrt{2}}{1440 (\mathsf{b}_2^\star\vee 1)}$. Assume the samples  $\left\{\tilde{x}_1,...,\tilde{x}_M\right\}$ are obtained with one of the following.\\
\noindent{\bf 1.} $n \geq  M \geq \left(10 + 160 \mathcal{N}_{\infty}^\mathbf{\Sigma}(\eta)\right)\log \frac{8 \kappa^2}{\eta \delta}$ using uniform sampling;\\
\noindent{\bf 2.} $  M \geq \left(6 + 486 q^2 \mathcal{N}^\mathbf{\Sigma}(\eta)\right)\log \frac{8 \kappa^2}{\eta \delta}$ using $q$-approximate leverage scores with respect to $\mathbf{\Sigma}$ for $t = \eta$, $t_0 \vee \frac{19 \kappa^2}{n}\log \frac{n}{2 \delta} < \eta$, $n \geq 405 \kappa^2 \vee 67 \kappa^2 \log \frac{12 \kappa^2}{\delta}$.\\
The following holds, with probability at least $1 - 3 \delta$.
\[L(f_{\beta_{M,\lambda}}) - L(f^\star) \leq \mathsf{K}_1~ \mathsf{b}_\lambda^2 + \mathsf{K}_2~ \frac{\mathsf{df}_\lambda \vee (\mathsf{Q}^\star)^2}{n}~\log \frac{2}{\delta} ,\qquad R \|\beta_{M,\lambda}\| \leq R \|f^\star\|  + 10,\]
where $\mathsf{K}_1 \leq 6.0\mathrm{e}{4}$, $\mathsf{K}_2 \leq 6.0\mathrm{e}{6} $ and $\mathsf{K}_3 \leq 810$, $\mathsf{C}_1$ is defined in \cref{lm:technical_before_proj}, and the other constants are defined in \cref{thm:general-result}. 

\end{theorem}

\begin{proof} 
This is simply a reformulation of \cref{prp:proj_pb_stat_gen_kernels}, noting that $\widehat{\nu}_{M,\lambda}(\beta_{M,\lambda}) = 0$ and that \cref{prp:nystrom_select} implies the condition on the Hessian at the optimum.
\end{proof} 


Provided source condition holds with $r>0$, the conditions of this theorem are not void.


\subsection{\label{app:approximate_b}Performing the globalization scheme to approximate $\beta_{M,\lambda}$}

In order to apply \cref{prp:proj_pb_stat_gen_kernels}, one needs to control $\widehat{\nu}_{M,\lambda}(\beta)$.

We will apply our general scheme to $\widehat{L}_{M,\lambda}$ in order to obtain such a control. 

\subsubsection{Performing approximate Newton steps \label{app:ans_kernels}}

The key element in the globalization scheme is to be able to compute $\frac{1}{7}$-approximate Newton steps.





 Note that at a given point $\beta$ and for a given $\mu > 0$ the Hessian is of the form: 

\[\widehat{\mathbf{H}}_{M,\mu}(\beta) = \frac{1}{n} {\bf T}^{-\top}{\bf K}_{Mn} {\bf D}_n(\beta) {\bf K}_{nM}{\bf T}^{-1} + \mu \mathbf{I}_M ,\]
where ${\bf D}_n(\beta) =  \diag((d_i(\beta))_{1 \leq i\leq n})$ is a diagonal matrix whose elements are given by $d_i(\beta) = \ell_{z_i}^{\prime\prime}(e_i^\top {\bf K}_{nM} {\bf T}^{-1} \beta)$.

Note that we can always write
\[\widehat{\mathbf{H}}_{M,\mu}(\beta) = \frac{1}{n} \sum_{i=1}^n{u_i(\beta) u_i(\beta)^\top}+ \mu \mathbf{I},\qquad u_i(\beta) = \sqrt{d_i(\beta)}{\bf T}^{-\top}{\bf K}_{Mn} e_i\]



The gradient can be put in the following form:

\[\nabla \widehat{L}_{M,\mu}(\beta) = \frac{1}{n}{\bf T}^{-\top}{\bf K}_{Mn}  v + \mu \beta,\qquad v = (\ell_{z_i}^{\prime}(e_i^\top {\bf K}_{nM}{\bf T}^{-1}\beta))_{1 \leq i \leq n}.\]

Computing the gradient at one point therefore costs $O(nM + M^2)$, this being the cost of computing ${\bf K}_{nM}$ times a vector costs $O(nM)$ and computing ${\bf T}^{-1}$ times a vector takes $O(M^2)$ since ${\bf T}$ is triangular. Moreover, the cost in memory is $O(M^2 + n)$, $M^2$ being needed for the saving of ${\bf T}$ and $n$ for the saving of the gradient; ${\bf K}_{nM}$ times a vector can also be done in $O(n)$ memory, provided we compute it by blocks. \\\

On the other hand, computing the full Hessian matrix would cost $nM^2$ operations, which is un-tractable. However, computing a Hessian vector product can be done in $O(nM + M^2)$ time, as for the gradient, which suggest using an iterative solver with preconditioning. 


\paragraph{Computing $x \in \lso(\mathbf{A},b,\rho)$ through pre-conditioned conjugate gradient descent.}

Assume we wish to solve the problem 
$\mathbf{A} x = b$ where $\mathbf{A} \in \mathbb{R}^{M \times M}$ is a positive definite matrix and $b$ is a vector of $\mathbb{R}^M$. If one uses the conjugate gradient method starting from zero, then if $x_k$ denotes the $k$-the iterate of the conjugate gradient algorithm, Theorem 6.6 in \cite{Saad03} shows that  
\[x_k \in \lso(\mathbf{A},b,\rho),\qquad \rho = 2\left(\frac{\sqrt{\cond(\mathbf{A})} -1 }{\sqrt{\cond(\mathbf{A})} + 1}\right)^k.\]

where $\cond(\mathbf{A})$ is the condition number of the matrix $\mathbf{A}$, namely the ratio $\frac{\lambda_{\max}(\mathbf{A})}{\lambda_{\min}(\mathbf{A})}$. If $\cond(\mathbf{A})$ is large, this convergence can be very slow. The idea of preconditioning is to compute an approximation matrix $\widetilde{\mathbf{A}}$ such that 
\eqal{\label{eq:tab}\frac{1}{2}\widetilde{\mathbf{A}} \preceq \mathbf{A} \preceq \frac{3}{2} \widetilde{\mathbf{A}}.}

We then compute ${\bf B}$ a triangular matrix such that ${\bf B}^\top {\bf B} = \widetilde{\mathbf{A}}$ using a cholesky decomposition, which can be done in $O(M^3)$, and note that ${\bf B}^{-\top} \mathbf{A} {\bf B}^{-1}$ is very well conditioned; indeed, its condition number is bounded by $3$. 

Perform a conjugate gradient method to solve the pre-conditioned problem ${\bf B}^{-\top} \mathbf{A} {\bf B}^{-1} z = {\bf B}^{-\top} b$, and denote with $z_\tau$ the $\tau$-th iteration of this method. Then 
using the bound on the condition number, we find 

\[z_\tau \in \lso({\bf B}^{-\top}\mathbf{A} {\bf B}^{-1},{\bf B}^{-\top}b,\rho),\qquad \rho = 2 \left(\frac{\sqrt{3}-1}{\sqrt{3}+1}\right)^{\tau},\]
which in turn implies that by setting $x_\tau:= {\bf B}^{-1}z_\tau$, 
\[x_\tau \in \lso(\mathbf{A},b,\rho),~ \rho = 2 \left(\frac{\sqrt{3}-1}{\sqrt{3}+1}\right)^{\tau}.\]

This shows that after at most $\tau = 3$  iterations, provided $\widetilde{\mathbf{A}}$ satisfies \cref{eq:tab}, $x_\tau \in \lso(\mathbf{A},b,\frac{1}{7})$. The cost of this method is therefore $ O(M^3 + nM) $ in time, and $O(n + M^2)$ due to the computing of the preconditioner and computing matrix vector products by block. This does not include the cost of finding a suitable $\widetilde{\mathbf{A}}$.

\paragraph{Computing a suitable approximation of $\widehat{\mathbf{H}}_{M,\mu}(\beta)$}



To compute a good pre-conditioner, we will subsample $Q$ points $i_1,...,i_Q$ points from $\left\{1,...,n\right\}$,  and sketch the Hessian using these $Q$ points. 




\begin{proposition}[Computing approximate newton steps]\label{prp:approx_ns_kernels} Let $\delta >0$. Let $\beta \in \mathbb{R}^M$ and $\mu \geq \lambda $, and assume $\frac{19 \mathsf{b}_2(f_\beta)\kappa^2}{n}\log \frac{n}{2 \delta} < \lambda$ and $n \geq 405 \mathsf{b}_2(f_\beta)\kappa^2 \vee 67 \mathsf{b}_2(f_\beta)\kappa^2 \log \frac{12 \mathsf{b}_2(f_\beta)\kappa^2}{\delta}$. Let $\tilde{\mu} = \min(\mu,\|\mathbf{H}(f_\beta)\|)$. Assume one of the following properties is satisfied


{\noindent \bf 1.} $Q \geq \left(10 + 160 \mathcal{N}_{\infty}^{\mathbf{H}(f_\beta)}(\tilde{\mu})\right)\log \frac{8 \mathsf{b}_2(f_\beta)\kappa^2}{\tilde{\mu} \delta}$ with uniform sampling of the  $\left\{i_1,...,i_Q\right\}$. We set ${\bf D}_Q = \diag(\ell^{\prime \prime}_{z_{i_j}}(f_\beta(x_{i_j})))_{1 \leq j \leq Q}$\\
{\noindent \bf 2.} $Q \geq \left(6 + 486 q^2 \mathcal{N}^{\mathbf{H}(f_\beta)}(\tilde{\mu})\right)\log \frac{8 \mathsf{b}_2(f_\beta)\kappa^2}{\tilde{\mu} \delta}$ using $q$-approximate leverage scores associated to $\mathbf{H}(f_{\beta})$ for $t = \tilde{\mu}$.
We set ${\bf D}_Q = \diag\left(\frac{\ell^{\prime \prime}_{z_{i_j}}(f_\beta(x_{i_j}))}{p_{i_j}}\right),$
where the $p_{i_j}$ are the probabilities computed from the leverage scores.\\

Assume we use a pre-conditioner ${\bf B}$ such that 
$${\bf B}^\top {\bf B} =  \frac{1}{Q}{\bf T}^{-\top} {\bf K}_{MQ} {\bf D}_Q {\bf K}_{QM} {\bf T}^{-1} + \mu \mathbf{I}_M,\qquad {\bf K}_{QM} = (K(x_{i_j},\tilde{x}_k))_{\substack{1 \leq j \leq Q \\ 1 \leq k \leq M}}.$$



If we perform $\tau = \log(\rho/2) / \log((\sqrt{3} +1)/{\sqrt{3} -1})$ iterations of the conjugate gradient descent on the pre-conditioned Newton system using ${\bf B}$ as a preconditioner, then with probability at least $1-\delta$, this procedure is returns $\widetilde{\Delta} \in \lso(\widehat{\mathbf{H}}_{M,\lambda}(\beta),\nabla \widehat{L}_{M,\lambda}(\beta),\rho)$, and the computational time is of order $O(\tau(Mn + M^2 Q + M^3 + \mathsf{c}_{\texttt{samp}}))$, and the memory requirements can be reduced to $O(M^2 + n)$. Here $\mathsf{c}_{\texttt{samp}}$ stands for the complexity of computing Nystrom leverage scores, and using \cref{rmk:cost_lev_scores} or \cite{rudi2018fast}, $\mathsf{c}_{\texttt{samp}} = O(1)$ if uniform sampling is used, and $\mathsf{c}_{\texttt{samp}} = O(\mathcal{N}^{\mathbf{H}(f_\beta)}(\tilde{\mu})^2/\lambda)$ if Nystrom sub-sampling is used. Note that for $\tau = 3$, $\rho = \frac{1}{7}$.

\end{proposition} 




\begin{proof}  Start by defining the following operators:

\begin{itemize}
    \item  $K_n: f\in {\mathcal{H}} \rightarrow  (f(x_i))_{1\leq i\leq n} \in \mathbb{R}^n$;
    \item  $K_M: f \in {\mathcal{H}} \rightarrow  (f(\tilde{x}_j))_{1\leq j\leq M} \in \mathbb{R}^M$;
    \item $V = K_M^* {\bf T}^{-1}$, where ${\bf T}$ is an upper triangular matrix such that ${\bf T}^\top {\bf T} = {\bf K}_{MM} = K_M K_M^*$. 
\end{itemize}

Note that $K_n V = {\bf K}_{nM} {\bf T}^{-1}$. 

Now note that 
\[\forall f \in {\mathcal{H}},~ \mathbf{H}(f) = \Exp{v_z \otimes v_z},\qquad \widehat{\mathbf{H}}(f) = \frac{1}{n} \sum_{i=1}^n{v_{z_i} \otimes v_{z_i}} ,\qquad v_z = \sqrt{\ell^{\prime \prime}_z(f(x))}K_x. \]

Since for any $f \in {\mathcal{H}}$, $\widehat{\mathbf{H}}(f) = \frac{1}{n} K_n^* {\bf D}_n(f) K_n $, where ${\bf D}_n(f) = \diag(\ell^{\prime \prime}_{z_i}(f(x_i)))$, we see that 

\[\widehat{\mathbf{H}}_{M,\mu}(\beta) = V^* \widehat{\mathbf{H}}(f_{\beta}) V + \mu \mathbf{I}_M. \]

Thus, the last lemma of \cref{app:sub_nystrom} can be applied, using the fact that $\|v_z\|^2 \leq \mathsf{b}_2(f)\kappa^2$, to get that in both cases of the proposition, under the corresponding assumptions: 

\[ \frac{1}{2}\left(\frac{1}{Q}{\bf T}^{-\top} {\bf K}_{MQ} {\bf D}_Q {\bf K}_{QM} {\bf T}^{-1} + \mu \mathbf{I}_M\right) \preceq \widehat{\mathbf{H}}_{M,\mu}(\beta) \preceq \frac{3}{2}\left(\frac{1}{Q}{\bf T}^{-\top} {\bf K}_{MQ} {\bf D}_Q {\bf K}_{QM} {\bf T}^{-1} + \mu \mathbf{I}_M\right).\]

The rest of the proposition follows from the previous discussion. 


\end{proof} 



















\subsubsection{Applying the globalization scheme to control $\widehat{\nu}_{M,\lambda}(\beta)$ \label{sssec:gen_scheme_optim} }



In order to apply 
\cref{prp:approx_ns_kernels} to each point $\beta$ in our method, we need to have a globalized version of the condition of this proposition.

First, we start by localizing the different values of $\beta$ we will visit throughout the algorithm.

\begin{definition}[path of regularized solutions]
Let $\lambda > 0$, $\varepsilon >0$. Define the path of regularized solutions
\eqal{
\label{df:path_M}
 \widehat{\Gamma}^M_{\lambda}:= \left\{\beta_{M,\mu}~:~ \mu \geq \lambda\right\}
.}
And the $\varepsilon$ approximation of this path: 
\eqal{
\label{df:path_M_eps}
 \widehat{\Gamma}^M_{\lambda,\varepsilon}:= \left\{\beta \in \mathbb{R}^M~:~ d(\beta, \widehat{\Gamma}^M_{\lambda}) \leq \varepsilon \right\}.
}

\end{definition} 

Note that we always have $ \widehat{\Gamma}^M_\lambda \subset \mathcal{B}_{\mathbb{R}^M}(\|\beta_{M,\lambda}\|)$. 
We now state a lemma proving that all the values visited during the algorithm will lie in an approximation of this path.

\begin{lemma} Define
Let $\beta \in \mathbb{R}^M$ such that $\widehat{\nu}_{M,\mu}(\beta) \leq \frac{\mu^{1/2}}{7R}$ for some $\mu \geq \lambda$. Then the following holds:
\[\beta \in  \widehat{\Gamma}^M_{\lambda,\frac{1}{6R}}.\]
\end{lemma} 




\begin{proof}
Bound 
\[R \|\beta - \beta_{M,\mu}\| \leq \frac{R}{\mu^{1/2}} \|\beta - \beta_{M,\mu}\|_{\widehat{\mathbf{H}}_{M,\mu}(\beta)} \leq \frac{1}{\underline{\phi}(\mathsf{t}_{M}(\beta - \beta_{M,\mu}))} \frac{R \widehat{\nu}_{M,\mu}(\beta)}{\mu^{1/2}}.\]
Just apply \cref{eq:gl} to obtain $R \|\beta - \beta_{M,\mu}\| \leq \frac{1}{6}$. 
\end{proof}

We now introduce the following quantities which will allow to control the number of sub-samples throughout the whole algorithm.

\begin{definition} Define 
\begin{itemize}
  \item   
$\overline{\mathsf{b}}_2:= \sup_{\beta \in  \widehat{\Gamma}^M_{\lambda,1/6R}}{\mathsf{b}_2(f_\beta)}$.
\item $\overline{\mathcal{N}}^{\mathbf{H}}(\lambda) = \sup_{\beta \in  \widehat{\Gamma}^M_{\lambda,1/6R}}\mathcal{N}^{\mathbf{H}(f_\beta)}(\lambda)$.
\item $\overline{\mathcal{N}}_{\infty}^{\mathbf{H}}(\lambda) = \sup_{\beta \in  \widehat{\Gamma}^M_{\lambda,1/6R}}\mathcal{N}_{\infty}^{\mathbf{H}(f_\beta)}(\lambda)$.
\item $\overline{\|\mathbf{H}\|} = \min_{\beta \in  \widehat{\Gamma}^M_{\lambda,1/6R}}\|\mathbf{H}(f_\beta)\|$.
\end{itemize}

\end{definition} 


\begin{proposition}[Performance of the globalization scheme]\label{prp:gen_scheme_optim}
Let $\varepsilon > 0$, $\delta >0$, $\tilde{\lambda} = \min(\lambda,\overline{\|\mathbf{H}\|})$. Assume $\frac{19 \overline{\mathsf{b}}_2 \kappa^2}{n}\log \frac{n}{2 \delta} < \tilde{\lambda}$ and $n \geq 405 \overline{\mathsf{b}}_2 \kappa^2 \vee 67 \overline{\mathsf{b}}_2 \kappa^2 \log \frac{12 \overline{\mathsf{b}}_2 \kappa^2}{\delta}$.

Assume we perform the globalization scheme with the parameters in \cref{thm:easy_first_phase},  where in order to compute any $\rho$ approximation of a regularized Newton step, we use a conjugate gradient descent on the pre-conditioned system, where the pre-conditioner is computed as in \cref{prp:approx_ns_kernels} using \\
\noindent{\bf 1.} $Q \geq \left(10 + 160 \overline{\mathcal{N}}_{\infty}^{\mathbf{H}}(\tilde{\lambda})\right)\log \frac{8 \overline{\mathsf{b}}_2\kappa^2}{\tilde{\lambda} \delta}$ if using uniform sampling\\
\noindent{\bf 2.} $Q \geq \left(6 + 486 q^2 \overline{\mathcal{N}}^{\mathbf{H}}(\tilde{\lambda})\right)\log \frac{8 \overline{\mathsf{b}}_2\kappa^2}{\tilde{\lambda} \delta}$ if using Nystr\"om leverage scores\\

Recall that $t$ denotes the number of approximate Newton steps performed at for each $\mu$ in Phase I and $T$ denotes the number of approximate Newton steps performed in Phase II, and that using \cref{thm:easy_first_phase}, $t = 2$ and $T = \lceil \log_2 \sqrt{1 \vee (\lambda \varepsilon^{-1}/R^2)}   \rceil$. Moreover, recall that $K$ denotes the number of steps performed in Phase I. Define 
\[N_{ns}:= 2  \left\lfloor \left(3 + 11 R \|\beta_{M,\lambda}\|\right)\log_2 (7 R \|\nabla \widehat{L}_M(0)\|/\lambda)\right\rfloor +  \lceil \log_2 \sqrt{1 \vee (\lambda \varepsilon^{-1}/R^2)}   \rceil.\]

Then with probability at least $(1-\delta)^{N_{ns}}$: 

\begin{itemize}
    \item The method presented in \cref{prp:approx_ns_kernels} returns a $1/7$- approximate Newton step at each time it is called in the algorithm.
    \item If $\beta$ denotes the result of the method, $\widehat{\nu}_{M,\lambda}(\beta) \leq \sqrt{\varepsilon}$.
    \item The number of approximate Newton steps computed during the algorithm is bounded by $N_{ns}$; the complexity of the method is therefore of order $O(N_{ns}(M^2\max(M,Q)+ nM + \mathsf{c}_{\texttt{samp}}(\lambda)))$ in time and $O(M Q + M^2 +n)$ in memory, where $\mathsf{c}_{\texttt{samp}}(\lambda)$ is a bound on the complexity associated to the computing of leverage scores (see \cite{rudi2018fast} for details).
\end{itemize}

The algorithm is detailed in \cref{sec:algorithm}, in  \cref{alg:algo}. Note however that the notations are those of the main paper, which are slightly different from the ones used here.
\end{proposition} 
\begin{proof} 
If we take the globalization scheme, using the parameters of \cref{thm:easy_first_phase}. Assume that all previous approximate Newton steps have been computed in a good way. Then the $\beta$ at which we are belongs to $ \widehat{\Gamma}^M_{\lambda,1/6R}$. Thus, the hypotheses of this proposition imply that the hypothesis of \cref{prp:approx_ns_kernels} are satisfied; and hence, up to a $(1-\delta)$ probability factor, we can assume that the next approximate Newton step is performed correctly, continuing the globalization scheme in the right way. Thus, the globalization scheme converges as in \cref{thm:easy_first_phase}. 
\end{proof} 


\subsection{Statistical properties of the algorithm \label{app:main_kernels}}

The following theorem describes the computational and statistical behavior of our algorithm.



\begin{proposition}[Behavior of an approximation to the projected problem] \label{prp:algo_stat_gen} Suppose that 
\cref{asm:iid1,asm:asmtech1,asm:minimizer1} are satisfied.\\
Let $n \in \mathbb{N}$, $\varepsilon > 0$, $\delta \in (0,1/2]$, $0 < \lambda \leq \mathsf{B}_2^\star$. \\
Define $\tilde{\lambda} = \min(\lambda,\overline{\|\mathbf{H}\|})$ and assume $\frac{19 \overline{\mathsf{b}}_2 \kappa^2}{n}\log \frac{n}{2 \delta} < \tilde{\lambda}$, $n \geq 405 \overline{\mathsf{b}}_2 \kappa^2 \vee 67 \overline{\mathsf{b}}_2 \kappa^2 \log \frac{12 \overline{\mathsf{b}}_2 \kappa^2}{\delta}$, and $n \geq \triangle_1 \frac{\mathsf{B}_2^\star}{\lambda}\log \frac{8 \square_1^2\mathsf{B}_2^\star}{\lambda \delta}$. Assume
\[ \mathsf{C}_1 \sqrt{ \frac{ \mathsf{df}_\lambda \vee (\mathsf{Q}^\star)^2}{n} ~\log \frac{2}{\delta}}\leq \frac{\lambda^{1/2}}{R}, \qquad \mathsf{C}_1 \mathsf{b}_\lambda \leq \frac{\lambda^{1/2}}{R}, \qquad 126 \sqrt{\varepsilon} \leq \frac{\lambda^{1/2}}{R}.\]
Assume that the $M$ points $\tilde{x}_1,...,\tilde{x}_M$ are obtained through Nystr\"om sub-sampling using $\eta = \|\mathbf{\Sigma}\| \wedge \frac{\lambda \sqrt{2}}{1440 (\mathsf{b}_2^\star\vee 1)}$, with either\\
\noindent{\bf 1.} $M \geq \left(10 + 160 \mathcal{N}_{\infty}^\mathbf{\Sigma}(\eta)\right)\log \frac{8 \kappa^2}{\eta \delta}$ if using uniform sampling;\\
\noindent{ \bf 2.} 
$M \geq \left(6 + 486 q^2 \mathcal{N}^\mathbf{\Sigma}(\eta)\right)\log \frac{8 \kappa^2}{\eta \delta}$ if using $q$-approximate leverage scores for $ \eta$, associated to the co-variance operator $\mathbf{\Sigma}$.


Assume we perform the globalization scheme as in \cref{prp:gen_scheme_optim}, i.e. with the parameters in \cref{thm:easy_first_phase},  where in order to compute any $\rho$ approximation of a regularized Newton step, we use a conjugate gradient descent on the pre-conditioned system, where the pre-conditioner is computed as in \cref{prp:approx_ns_kernels} using \\
\noindent{\bf 1.} $Q \geq \left(10 + 160 \overline{\mathcal{N}}_{\infty}^{\mathbf{H}}(\tilde{\lambda})\right)\log \frac{8 \overline{\mathsf{b}}_2\kappa^2}{\tilde{\lambda} \delta}$ if using uniform sampling\\
\noindent{\bf 2.} $Q \geq \left(6 + 486 q^2 \overline{\mathcal{N}}^{\mathbf{H}}(\tilde{\lambda})\right)\log \frac{8 \overline{\mathsf{b}}_2\kappa^2}{\tilde{\lambda} \delta}$ if using Nystr\"om leverage scores\\
Let $N_{ns}$ be defined as in \cref{prp:gen_scheme_optim}. Recall $N_{ns}$ is an upper bound for the number of approximate Newton steps performed in the algorithm. One can bound 
\[N_{ns} \leq 2  \left\lfloor \left(113 + 11 R \|f^\star\|\right)\log_2 \frac{7 R \|\nabla \widehat{L}_M(0)\|}{\lambda}\right\rfloor +  \left\lceil \log_2 \frac{\lambda^{1/2}}{R \varepsilon}   \right\rceil. \]
Moreover, with probability at least $1 - (N_{ns} +2)\delta$, the following holds:


\[L(f_{\beta}) - L(f^\star) \leq \mathsf{K}_1~ \mathsf{b}_\lambda^2 + \mathsf{K}_2~ \frac{\mathsf{df}_\lambda \vee (\mathsf{Q}^\star)^2}{n}~\log \frac{2}{\delta} + \mathsf{K}_3~ \varepsilon .\]

where $\mathsf{K}_1 \leq 6.0\mathrm{e}{4}$, $\mathsf{K}_2 \leq 6.0\mathrm{e}{6} $ and $\mathsf{K}_3 \leq 810$, $\mathsf{C}_1$ is defined in \cref{lm:technical_before_proj}, and the other constants are defined in \cref{thm:general-result}. 
\end{proposition} 

\begin{proof} 
This is a simple combination between \cref{prp:gen_scheme_optim,prp:nystrom_select,prp:proj_pb_stat_gen_kernels}. To bound the number of Newton steps $N_{ns}$, one simply uses the fact that under the conditions of the theorem, $R\|\beta_{M,\lambda} \| \leq 10 + R \|f^\star\|$. 
\end{proof} 


\begin{remark}[Complexity] \label{rmk:complexity}
Let $L = \overline{\mathsf{b}}_2 \kappa^2$.  The complexity of the previous method using leverage scores computed for $\mathbf{\Sigma}$ for the Nystrom projections and for $\mathbf{H}(f_\beta)$ for choosing the $Q$ points at the different stages is the following. The total complexity in time will be of order:
\[O\left(N_{ns}\left(  n  \mathcal{N}^{\overline{\mathbf{H}}}(\lambda)\log(L \lambda^{-1}\delta^{-1})  + \overline{\mathsf{b}}_2^3 \mathcal{N}^{\mathbf{\Sigma}}(\lambda)^3 \log^3(L \lambda^{-1}\delta^{-1}) +  L / \lambda ~\overline{\mathsf{b}}_2^2 \mathcal{N}^{\mathbf{\Sigma}}(\lambda)^2 \right)\right).\]

The memory complexity can be bounded by
\[ O(\overline{\mathsf{b}}_2^2 \mathcal{N}^{\mathbf{\Sigma}}(\lambda)^2\log^2(L\lambda^{-1}\delta^{-1}) +n).\]
Here, we use the fact that $\mathbf{H} \leq \overline{\mathsf{b}}_2 \mathbf{\Sigma}$.
\end{remark} 
We can now write down the previous proposition by classifying problems using \cref{asm:source,asm:deff} and in order to get optimal rates.


\begin{theorem}[Performance of the scheme using pre-conditioning]\label{thm:main_kernels}
Let $\delta > 0$. 
Assume \cref{asm:iid1,asm:deff,asm:asmtech1,asm:minimizer1,asm:source} are satisfied. Let $n \geq \tilde{N}$, where $\tilde{N}$ is characterized in the proof,  $\lambda = \left( \left(\frac{\mathsf{Q}}{\mathsf{L}}\right)^{2}~\frac{1}{n}\right)^{\frac{\alpha}{\alpha(1+2r) + 1}}$. \\\
Assume that the $M$ points $\tilde{x}_1,...,\tilde{x}_M$ are obtained through Nystr\"om sub-sampling using $\eta = \frac{\lambda \sqrt{2}}{1440 (\mathsf{b}_2^\star\vee 1)}$, with either\\
\noindent{\bf 1.} $M \geq \left(10 + 160 \mathcal{N}_{\infty}^\mathbf{\Sigma}(\eta)\right)\log \frac{8 \kappa^2}{\eta \delta}$ if using uniform sampling;\\
\noindent{ \bf 2.} 
$M \geq \left(6 + 486 q^2 \mathcal{N}^\mathbf{\Sigma}(\eta)\right)\log \frac{8 \kappa^2}{\eta \delta}$ if using $q$-approximate leverage scores for $ \eta$, associated to the co-variance operator $\mathbf{\Sigma}$.


Assume we perform the globalization scheme as in \cref{prp:gen_scheme_optim}, i.e. with the parameters in \cref{thm:easy_first_phase},  where in order to compute any $\rho$ approximation of a regularized Newton step, we use a conjugate gradient descent on the pre-conditioned system, where the pre-conditioner is computed as in \cref{prp:approx_ns_kernels} using \\
\noindent{\bf 1.} $Q \geq \left(10 + 160 \overline{\mathcal{N}}_{\infty}^{\mathbf{H}}(\lambda)\right)\log \frac{8 \overline{\mathsf{b}}_2\kappa^2}{\lambda\delta}$ if using uniform sampling\\
\noindent{\bf 2.} $Q \geq \left(6 + 486 q^2 \overline{\mathcal{N}}^{\mathbf{H}}(\lambda)\right)\log \frac{8 \overline{\mathsf{b}}_2\kappa^2}{\lambda \delta}$ if using Nystr\"om leverage scores\\
Let $N_{ns}$ be defined as in \cref{prp:gen_scheme_optim}. Recall $N_{ns}$ is an upper bound for the number of approximate Newton steps performed in the algorithm. One can bound 
 \[ N_{ns} \leq  \left(227 + 22 R \|f^\star\|\right)\left( \left\lceil \log_2 \left(7 R \|\nabla \widehat{L}_M(0)\| \right)\right\rceil + \left\lceil \log_2\frac{n \mathsf{L}^2}{\mathsf{Q}^2} \right\rceil +  \left\lceil \log_2 \frac{1}{R \mathsf{L}} \right\rceil\right).\]
Moreover, with probability at least $1 - (N_{ns} +2)\delta$, the following holds:
\begin{itemize}
    \item all of the approximate Newton methods yield $\frac{1}{7}$-approximate Newton steps
    \item The scheme finishes, and the number of approximate Newton steps is bounded by $N_{ns}$. The total complexity of the method is therefore 
\[O((nM + M^3 + M^2Q + \mathsf{c}_{\texttt{samp}})N_{ns}) \text{ in time },\qquad O(n + M^2) \text{ in memory} .\]
\item The returned $\beta$ is statistically optimal: 
\[L(f_\beta) - L(f^\star) \leq \mathsf{K} \left(\mathsf{Q}^{\gamma} ~ 
    \mathsf{L}^{1-\gamma}\right)^2 ~~\frac{1}{n^{\gamma}} \log \frac{2}{\delta},\]
    where $\mathsf{K}$ is defined in \cref{thm:quantisource_kernels}.
\end{itemize}
\end{theorem} 


\begin{proof} 
The proof consists mainly of combining \cref{prp:nystrom_select,prp:gen_scheme_optim,thm:quantisource_kernels}. 

Recall that we set $\lambda = \left(\frac{\mathsf{Q}^2}{\mathsf{L}^2}\frac{1}{n}\right)^{\frac{\alpha}{\alpha(1+2r) + 1}}$. 
\paragraph{1.} Start by defining $\tilde{N}$ such that:
\begin{itemize}
    \item $\tilde{N} \geq N$ where $N$ is defined in \cref{thm:quantisource_kernels};
    \item $\forall n \geq \tilde{N},~ \lambda \leq \overline{\|\mathbf{H}\|}$. This is possible as $\frac{\alpha}{\alpha(1+2r) + 1}$ is a strictly positive exponent. 
    \item 
    $\forall n \geq \tilde{N},~\frac{19 \overline{\mathsf{b}}_2 \vee 1 ~\kappa^2}{n}\log \frac{n}{2 \delta} < \lambda$; this is possible as soon as $\frac{\alpha}{\alpha(1+2r) + 1} < 1$, i.e. this is satisfied since $r>0$;  
    \item $\tilde{N} \geq 405 \overline{\mathsf{b}}_2 \vee 1 ~\kappa^2 \vee 67 \overline{\mathsf{b}}_2 \vee 1 ~\kappa^2 \log \frac{12 \overline{\mathsf{b}}_2 \vee 1~~\kappa^2}{\delta}$;
    \item $\forall n \geq \tilde{N},~ \frac{\lambda \sqrt{2}}{1440 (\mathsf{b}_2^\star\vee 1)} \leq \|\mathbf{\Sigma}\|$.
\end{itemize}
We see that such a $\tilde{N}$ can be defined explicitly. 
\paragraph{2.} Combining the assumptions on $\tilde{N}$ with the ones on $M$, we see that all the assumptions of \cref{prp:nystrom_select} are satisfied and thus that with probability at least $1 - \delta$, all the hypotheses for \cref{thm:quantisource_kernels} are satisfied except the bound on $\widehat{\nu}_{M,\lambda}(\beta)$.



\paragraph{3.} Applying \cref{prp:gen_scheme_optim}, taking $\sqrt{\varepsilon} =\mathsf{Q}^{\gamma} ~ 
    \mathsf{L}^{1-\gamma} n^{-\gamma/2} $ and $\lambda = \left(\frac{\mathsf{Q}^2}{\mathsf{L}^2}\frac{1}{n}\right)^{\frac{\alpha}{\alpha(1+2r) + 1}}$, we see that under these hypotheses, 
    \begin{align*}
    N_{ns}:= &2  \left\lfloor \left(3 + 11 R \|\beta_{M,\lambda}\|\right)\log_2 \left(7 R \|\nabla \widehat{L}_M(0)\|\left(\frac{n \mathsf{L}^2}{\mathsf{Q}^2}\right)^{\frac{\alpha}{\alpha(1+2r) + 1}}\right)\right\rfloor \\
    &+  \left\lceil \log_2 \left(\frac{1}{R \mathsf{L}}\left(\frac{n \mathsf{L}^2}{\mathsf{Q}^2}\right)^{\frac{r \alpha}{\alpha(1+2r) + 1}}\right)  \right\rceil.
    \end{align*}
    Now we can bound this harshly:
    \[ N_{ns} \leq  \left(7 + 22 R \|\beta_{M,\lambda}\|\right)\left( \left\lceil \log_2 \left(7 R \|\nabla \widehat{L}_M(0)\| \right)\right\rceil + \left\lceil \log_2\frac{n \mathsf{L}^2}{\mathsf{Q}^2} \right\rceil +  \left\lceil \log_2 \frac{1}{R \mathsf{L}} \right\rceil\right).\]
    Now bounding $R\|\beta_{M,\lambda}\| \leq 10 + R \|f^\star\|$, we get
    \[ N_{ns} \leq  \left(227 + 22 R \|f^\star\|\right)\left( \left\lceil \log_2 \left(7 R \|\nabla \widehat{L}_M(0)\| \right)\right\rceil + \left\lceil \log_2\frac{n \mathsf{L}^2}{\mathsf{Q}^2} \right\rceil +  \left\lceil \log_2 \frac{1}{R \mathsf{L}} \right\rceil\right).\]

\paragraph{4. } Finally, we use a union bound to conclude.
\end{proof} 

\newpage 

\section{Algorithm\label{sec:algorithm}}
\begin{algorithm}
\footnotesize
\fbox{
\begin{minipage}[t]{0.97\textwidth}
{\bf Input:} $(x_i,y_i)_{i=1}^n$, $n \in \mathbb{N}$, $\ell$ loss function, $k$ kernel function and $\lambda > 0$.\\
{\bf Return:} estimated function $\widehat{g}:\mathcal{X} \to \mathbb{R}$\\
Parameters: $Q, M, T \in \mathbb{N}$, $\mu_0 > 0$, $(q_k)_{k \in \mathbb{N}}$.\\
Fixed parameters: $t = 2$ from \cref{thm:main-thm}, $\tau = 3$ from \cref{prp:approx_ns_kernels} in \cref{app:ans_kernels}.\\
$(\bar{x}_j)_{j=1}^M \leftarrow \texttt{leverage-scores-sampling}((x_i)_{i=1}^n, M, \lambda, k)$\\
${\bf K} \leftarrow \texttt{kernel-matrix}((\bar{x}_j)_{j=1}^M,(\bar{x}_j)_{j=1}^M)$\\
${\bf T} \leftarrow \texttt{cholesky-upper-triangular}({\bf K})$\\
define the function $v(\cdot) = (k(\bar{x}_1,\cdot), \dots, k(\bar{x}_M, \cdot)) \in \mathbb{R}^M$\\
\fbox{
\begin{minipage}[t]{0.95\textwidth}
\textbf{define \texttt{compute-preconditioner}:}\\
{\bf Input:} $\alpha \in \mathbb{R}^M, \lambda > 0$\\
$c_i \leftarrow \sqrt{\ell^{(2)}(v(x_i)^\top {\bf T}^{-1} \alpha,y_i)}$ for all $i=1, \dots, n$\\
define the function $k'(\circ, \bullet)$ as 
$k'(\circ, \bullet):=  c_\circ \times c_\bullet \times k(x_\circ, x_\bullet) $ for $\circ, \bullet \in \{1,\dots,n\}$\\
$(h_s)_{s=1}^Q \leftarrow \texttt{leverage-scores-sampling}((i)_{i=1}^n, Q, \lambda, k')$\\
${\bf G} \leftarrow \texttt{kernel-matrix}((\bar{x}_j)_{i=1}^M, (x_{h_s})_{s=1}^Q, k)$\\
${\bf H} \leftarrow {\bf T}^{-\top} \times {\bf G} \times \text{diag}((c_{l_h}^2)_{h=1}^Q) \times {\bf G}^\top \times {\bf T}^{-1}$\\
${\bf B} \leftarrow \texttt{cholesky-upper-triangular}(\frac{1}{Q}{\bf H} + \lambda I)$\\
return $\bf B$
\end{minipage}
}
\fbox{
\begin{minipage}[t]{0.95\textwidth}
\textbf{define \texttt{preconditioned-conj-grad}:}\\
{\bf Input:} $\alpha \in \mathbb{R}^M, \mu >0, r \in \mathbb{R}^M, \tau \in \mathbb{N}, {\bf B} \in \mathbb{R}^{M \times M}$\\
$p \leftarrow r, s_0 \leftarrow \|r\|^2, \beta \leftarrow 0$\\
For $i=1,\dots, \tau$\\
${}\qquad z \leftarrow \mu {\bf B}^{-\top}{\bf B}^{-1}p + \frac{1}{n}\sum_{i=1}^n \ell^{(2)}(v(x_i)^\top {\bf T}^{-1} \alpha, y_i) ~ (v(x_i)^\top {\bf T}^{-1} {\bf B}^{-1} p) ~{\bf B}^{-\top}{\bf T}^{-\top} v(x_i)$ \\ 
${}\qquad a \leftarrow s_0/(p^\top z)$\\
${}\qquad \beta \leftarrow \beta + a p$\\
${}\qquad r \leftarrow r - a z,~~s_1 \leftarrow \|r\|^2$\\
${}\qquad p \leftarrow r + (s_1/s_0) p$\\
${}\qquad s_0 \leftarrow s_1$\\
return $\beta$
\end{minipage}
}
\fbox{
\begin{minipage}[t]{0.95\textwidth}
\textbf{define \texttt{appr-linear-solver}:}\\
{\bf Input:} $\alpha \in \mathbb{R}^M, \mu > 0, g \in \mathbb{R}^M$ \\
${\bf B} \leftarrow \texttt{compute-preconditioner}(\alpha, \mu)$\\
$u \leftarrow \texttt{preconditioned-conjugate-gradient}(\alpha, \mu, {\bf B}^{-\top}g, \tau = 3, {\bf B})$\\
return ${\bf B}^{-1} u$
\end{minipage}
}
\fbox{
\begin{minipage}[t]{0.95\textwidth}
\textbf{define \texttt{approximate-Newton}:}\\
Input: $\alpha_0 \in \mathbb{R}^M, \mu > 0, t\in \mathbb{N}$\\
For $j = 1,\dots, t$\\
${}\qquad g \leftarrow \mu \alpha_{j-1} + \frac{1}{n}\sum_{i=1}^n \ell^{(1)}(v(x_i)^\top {\bf T}^{-1}\alpha_{j-1}, y_i)~{\bf T}^{-\top} v(x_i)$\\
${}\qquad \alpha_j \leftarrow \alpha_{j-1} - \texttt{appr-linear-solver}(\alpha_{j-1}, \mu, g)$\\
return $\alpha_t$
\end{minipage}
}\\
$\alpha_0 \leftarrow 0$\\
For $k \in \mathbb{N}$\\
${}\qquad \alpha_{k+1} \leftarrow \texttt{approximate-Newton}(\alpha_k,\mu_k,t=2)$\\
${}\qquad \mu_{k+1} \leftarrow q_{k+1}\mu_{k}$\\
Stop when $\mu_{k+1} < \lambda$ and set $\alpha_{last}\leftarrow \alpha_k$\\
$\widehat{\alpha} \leftarrow \texttt{approximate-Newton}(\alpha_{last}, \lambda, T)$\\
return $\widehat{g}(\cdot):= v(\cdot)^\top {\bf T}^{-1} \widehat{\alpha}$
\end{minipage}
}
\caption{Algorithm efficient non-parametric learning for generalized self-concordant losses with optimal statistical guarantees discussed in \cref{sec:kernels} of the main paper.\label{alg:algo}}
\end{algorithm}
Let $N, M \in \mathbb{N}$ with 
$M \leq N$. In Alg.~\ref{alg:algo}, $\texttt{leverage-scores-sampling}((z_i)_{i=1}^N, M, k, \lambda)$ returns a subset of $(z_i)_{i=1}^N$ of cardinality $M$ sampled by using (approximate) leverage scores at scale $\lambda > 0$ and computed using the kernel $k$. An explicit example of an algorithm computing $\texttt{leverage-scores-sampling}$ is in \cite{rudi2018fast}. Moreover $\texttt{kernel-matrix}((x_i)_{i=1}^N, (x'_i)_{i=1}^M, k)$ computes the kernel matrix $K \in \mathbb{R}^{N\times M}$ where $K_{ij} = k(x_i, x'_j)$, with $N, M \in \mathbb{N}$.


\newpage

\section{Experiments\label{sec:exp-appendix}}
 
 
We present our algorithm's performance for logistic regression on two large scale data sets: Higgs and Susy. We have implemented our method using pytorch, and performed computations on one node of a Tesla P100-PCIE-16GB GPU. Recall that in the case of logistic regression, $\ell_{(x,y)}(t) = \log(1 + e^{-yt})$. 

In what follows, denote with $n$ the cardinality of the data set and $d$ the number of features of this data set. The error is measured in terms of classification error for both data sets. In both cases, we pre-process the data by substracting the mean and dividing by the standard deviation for each feature. The data sets are the following. 

\paragraph{Susy}($n = 5 \times 10^{6}$, $d = 18$, binary classification). We always use a Gaussian Kernel with $\sigma = 5$ for logistic loss (obtained through a grid search; note that in \cite{Rudi17}, $\sigma = 4$ is used for the square loss), and will always use $10^4$ Nystrom points.  

\paragraph{Higgs}($n = 1.1 \times 10^7$, $d = 28$, binary classification). We then apply a Gaussian Kernel with $\sigma = 5$, as in \cite{Rudi17} (we have also performed a grid search). \\\


For these data sets, we do not have a fixed test set, and thus set apart $20 \%$ of the data set at random to be the test set, and use the rest of the $80\%$ to train the classifier.\\\

In practice, we perform our globally convergent scheme with the following parameters.
\begin{itemize}
    \item We use $Q = M$ uniform random features to compute the pre-conditioner for each approximate Newton step;
    \item In the first phase, we decrease $\mu$ in a very fast way to $\lambda$ by starting at $\mu =1$ and dividing $\mu$ by 1000 after performing only a single approximate Newton step (using $2$ iterations of conjugate gradient descent);
    \item In the second phase, we perform $10$ approximate Newton steps (each ANS is computed using $8$ iterations of conjugate gradient descent).
\end{itemize}


\paragraph{Selection of $\lambda$}

In the introduction, we claim that in many a learning problem, the parameter $\lambda$ obtained through cross validation is often much smaller than the ones obtained in statistical bounds which are usually of order $\frac{1}{\sqrt{n}}$. This leads to very ill conditioned problems. 

For both data sets, we select $\lambda$ (and $\sigma$, but we omit the double tables from this paper) by computing the test loss and classification errors for different values of $\lambda$, and report the evolution of these losses as a function of the parameter $\lambda$ in \cref{fig:test_higgs} for the Higgs data set, and \cref{fig:test_susy} for the Susy data set. 
We see that the optimal $\lambda$ yield strongly ill-conditioned problems. 

\begin{figure}[ht]
\includegraphics[width=0.48\textwidth]{class_error_higgs_arxiv.pdf}
\hspace{0.25cm}%
\includegraphics[width=0.48\textwidth]{test_loss_higgs_arxiv.pdf} 
\caption{ \textbf{(Left)}  Classification error as a function of the regularization parameter and \textbf{(Right)} test loss as a function of the regularization parameter, when performing a logistic regression with $M = 2\times10^4$ Nystr\"om features on the entire Higgs data set; we select $\lambda = 10^{-9}$.}
\label{fig:test_higgs}
\end{figure}

\begin{figure}[ht]
\includegraphics[width=0.48\textwidth]{class_error_susy_arxiv.pdf}
\hspace{0.25cm}%
\includegraphics[width=0.48\textwidth]{test_loss_susy_arxiv.pdf} 
\caption{ \textbf{(Left)}  Classification error as a function of the regularization parameter and \textbf{(Right)} test loss as a function of the regularization parameter, when performing a logistic regression with $M = 10^4$ Nystr\"om features on the entire Susy data set; we select $\lambda = 10^{-10}$.}
\label{fig:test_susy}
\end{figure}

\paragraph{Comparison with accelerated methods}

Given the $M$ Nystrom points, our aims to minimize $\widehat{L}_{M,\lambda}$. 
From an optimization point of view, i.e. from a point of view where the aim is to minimize $\widehat{L}_{M,\lambda}$, we compare our method with a large mini-batch version of Katyusha accelerated SVRG (see \cite{Allen2017}). \\
Indeed, we perform this method using batch sizes of size $M$; the theoretical bounds provided in \cite{Allen2017} show that the algorithm has linear convergence, with a time complexity of order $O(nM + M^3 + M^2 \sqrt{\frac{L}{\lambda}})\log \frac{1}{\varepsilon}$ to reach precision $\varepsilon$. In the following plots, we compare both methods in terms of passes and time. \\\

By pass, we mean the following.
\begin{itemize}
    \item In the case of our second-order scheme, we define a pass on the data to be one step of the conjugate gradient descent used to compute approximate newton steps.
    \item In the case of Katyusha SVRG, we define a pass on the data to be either a full gradient computation or $n / M$ computations of the type $K_{\tau M}T^{-1} \beta $ where $T$ is an upper triangular matrix, and $K_{\tau M}$ is a $M \times M$ kernel matrix, associated to one batch gradient. 
\end{itemize}

We use this notion to measure the speed of our method as they both correspond to natural $O(nM)$ operations, and incorporate the essential of the computing time. However, the second point is often much slower to compute than the first, due to the solving of the triangular system. Thus, the notion of passes is to take with precaution, as a pass for the accelerated SVRG algorithm takes much longer to run that a pass for our method. This is confirmed by the time plots (see \cref{fig:susy_compare_small} for in instance).

\textit{Comparison between the two methods} - Due to the running time of K-SVRG, we compare both methods for $M = 10000$ Nystr\"om points for both data sets. We compare the performance of these two algorithm with respect to the distance to the optimum in function values as well as classification error \cref{fig:higgs_compare} for the Higgs data set, and in \cref{fig:susy_compare_small} for the Susy data set.




\begin{figure}[ht]
\includegraphics[width=0.48\textwidth]{higgs_distance_arxiv.pdf}
\hspace{0.25cm}%
\includegraphics[width=0.48\textwidth]{higgs_distance_class_arxiv.pdf} 
\caption{\textbf{(Left)} Distance to optimum as a function of  time and  \textbf{(Right)} distance to optimum and classification error as a function of the number of passes on the data when performing our second order scheme and K-SVRG to minimize the train loss on Higgs, with $1.0\times10^4$ Nystr\"om points and $\lambda = 10^{-9}$.}
\label{fig:higgs_compare}
\end{figure}

\begin{figure}[ht]
\includegraphics[width=0.48\textwidth]{susy_distance_small_arxiv.pdf}
\hspace{0.25cm}%
\includegraphics[width=0.48\textwidth]{susy_distance_class_small_arxiv.pdf} 
\caption{\textbf{(Left)} Distance to optimum as a function of  time and  \textbf{(Right)} distance to optimum and classification error as a function of the number of passes on the data when performing our second order scheme and K-SVRG to minimize the train loss on Susy, with $1.0\times10^4$ Nystr\"om points and $\lambda = 10^{-10}$.}
\label{fig:susy_compare_small}
\end{figure}

\textit{Note on the need for precise optimization} - As noted in the introduction, we see in both \cref{fig:susy_compare_small} and \cref{fig:higgs_compare} that precise optimization of the objective function is needed in order to get a good classification error. This justifies a posteriori the use of a second order method. In particular, in \cref{fig:susy_compare_small}, one notes the difference in behavior between the two methods : the second order method converges linearly in a fast way while the first order method slows down because of the condition number. 


\textit{Note on ill-conditioning} - First note that in order to optimize test error, one gets very poorly conditioned problems. As predicted by the rates, we observe that K-SVRG is more sensible to ill-conditioning than our second order scheme. Indeed, in \cref{fig:susy_compare}, we have plotted the results for Susy for a smaller condition number with $\lambda = 10^{-8}$, compared to $\lambda = 10^{-10}$ to get optimal test error in \cref{fig:susy_compare_small}. We see that the difference in number of passes needed to reach a certain precision is much lower when $\lambda = 10^{-8}$ in \cref{fig:susy_compare}, confirming that K-SVRG behaves better when the condition number is smaller.


\begin{figure}[ht]
\includegraphics[width=0.48\textwidth]{susy_distance_arxiv.pdf}
\hspace{0.25cm}%
\includegraphics[width=0.48\textwidth]{susy_distance_arxiv.pdf} 
\caption{\textbf{(Left)} Distance to optimum as a function of  time and  \textbf{(Right)} distance to optimum and classification error as a function of the number of passes on the data when performing our second order scheme and K-SVRG to minimize the train loss on Susy, with $1.0\times10^4$ Nystr\"om points and $\lambda = 10^{-8}$.}
\label{fig:susy_compare}
\end{figure}


\paragraph{Performance of our method.}

In \cref{tab:performance},  we record the performance of the following methods, taking the $\lambda$ values we have obtained previously for the different data sets.  

For FALKON (see \cite{Rudi17}), we take the parameters suggested in the paper (except for the number of Nystr\"om points needed for Higgs, as our computational capacity is limited). 

\begin{table}[h!]
    \centering
    \begin{tabular}{|c|c|c|c|c|c|c|}
 \hline
    \multirow{2}{*}{Method} & \multicolumn{3}{c|}{Susy} &  \multicolumn{3}{c|}{Higgs}\\
 \cline{2-7}
 & c-error & $M$ & time(m) &c-error & $M$ & time(m)\\
 \hline
 Logistic regression with K-SVRG & 19.64\%  & $10^4$ &  230& 27.82 \% & $ 10^4$& 500 \\ 
 \hline 
 
 Logistic regression with our scheme & 19.5\% & $10^4$ & 15 & 26.9 \% & $2.5\times10^4$ & 65\\
 \hline
 Ridge Regression with FALKON (\cite{Rudi17}) & 19.7\% & $10^4$ & 5  & 27.16 \% & $2.5\times10^4$ & 60\\
 \hline
 
\end{tabular}
    \caption{Classification error of different methods}
    \label{tab:performance}
\end{table}



\newpage

\section{\label{app:proj}Solving a projected problem to reduce dimension}

\subsection{Introduction and notations}

In this section, we give ourselves a generalized self-concordant function $f$ whose associated subset we denote with $\mathcal{G}$. Once again, we will always omit the subscript $f$ in the notations associated to $f$.\\\

The aim of this section is the following. Given $f$ and $\lambda > 0$, computing an approximate solution to 
\[x_\lambda^{\star}  =\argmin_{x \in {\mathcal{H}}}{f_\lambda(x)},\]
is often too costly. Instead, we look for a solution in a small subset of ${\mathcal{H}}$ which we see as the image of a certain orthogonal projector $\mathbf{P}$ and which we denote ${\mathcal{H}}_\mathbf{P}$. Usually, this subset will be finite dimensional and admit an easy parametrization. Thus we will compare an approximation of $x_\lambda^{\star}$ to an approximation of

\[x_{\Pj,\lambda}^* = \argmin_{x \in {\mathcal{H}}_\mathbf{P}}{f_\lambda(x)} = \argmin_{x \in {\mathcal{H}}}{f(\mathbf{P} x) + \frac{\lambda}{2}\|x\|^2}.\]

Denote with $f_\mathbf{P}$ the mapping $x \in {\mathcal{H}} \mapsto f(\mathbf{P} x)$. It is easy to see that, as $f$ is a generalized self-concordant function with $\mathcal{G}$, $f_\mathbf{P}$ is naturally a generalized self-concordant with $\mathcal{G}_\mathbf{P}:= \mathbf{P} \mathcal{G} = \left\{\mathbf{P} g ~:~ g \in \mathcal{G}\right\}$. Moreover, $x_{\Pj,\lambda}^* = x^\star_{f_\mathbf{P},\lambda}$. \\\

We will adopt the following notations for the quantities related to the generalized self-concordant function $f_\mathbf{P}$. Essentially, we always replace $f_\mathbf{P}$ simply by $\mathbf{P}$ from our definitions in appendix.
\begin{itemize}
    \item For the regularized function :
    \[\forall x \in {\mathcal{H}},~ \forall \lambda > 0,~ f_{\mathbf{P},\lambda}(x) = f_\mathbf{P}(x) + \frac{\lambda}{2}\|x\|^2.\]
    \item For the Hessians 
    \[\forall x \in {\mathcal{H}}, ~ \lambda >0,~ \mathbf{H}_{\mathbf{P},\lambda}(x) = \mathbf{H}_{f_{\mathbf{P}},\lambda}(x) = \mathbf{P} \mathbf{H}(\mathbf{P} x  ) \mathbf{P} + \lambda \mathbf{I}.\]
    \item $\forall h \in {\mathcal{H}},~ \mathsf{t}_\mathbf{P}(h):= \mathsf{t}_{f_\mathbf{P}}(h) = \mathsf{t}(\mathbf{P} h)$.
    \item For the Newton decrement:
    \[\forall x \in {\mathcal{H}}, ~ \lambda >0,~  \nu_{\mathbf{P},\lambda}(x) = \nu_{f_\mathbf{P},\lambda}(x) = \|\nabla f_{\mathbf{P},\lambda}\|_{\mathbf{H}^{-1}_{\mathbf{P},\lambda}(x)} = \|\mathbf{P} \nabla f(\mathbf{P} x) + \lambda x\|_{\mathbf{H}^{-1}_{\mathbf{P},\lambda}(x)}.\]
    \item For the Dikin ellipsoid radius:
    $$\forall \lambda>0,~\forall x \in {\mathcal{H}},~ \mathsf{r}_{\mathbf{P},\lambda}(x):= \mathsf{r}_{f_\mathbf{P},\lambda}(x) = \frac{1}{\sup_{g \in  \mathcal{G}}\|\mathbf{P} g\|_{\mathbf{H}_{\lambda,\mathbf{P}}^{-1}(x)}};$$
    \item For the Dikin ellipsoid:
    \[\forall \lambda > 0,~ \forall \mathsf{c} \geq 0,~ \mathsf{D}_{\mathbf{P},\lambda}(\mathsf{c}):= \mathsf{D}_{f_\mathbf{P},\lambda}(\mathsf{c}).\]
    
\end{itemize}


Note that for any $x \in {\mathcal{H}}_\mathbf{P}$, $\mathsf{r}_{\Pj,\lambda}( x) \geq \mathsf{r}_\lambda(x)$.

We will now introduce the key quantities in order to compare an approximation of $x_{\Pj,\lambda}^*$ to an approximation of $x_\lambda^{\star}$. 


\begin{definition}[key quantities]
Define the following quantities
\begin{itemize}
    \item For any $\lambda >0$, the source term $\mathsf{s}_\lambda:= \lambda\|x_\lambda^{\star}\|_{\mathbf{H}_\lambda^{-1}(x_\lambda^{\star})} = \|\nabla f(x_\lambda^{\star})\|_{\mathbf{H}_\lambda^{-1}(x_\lambda^{\star})} $;
    \item Given an orthogonal projector $\mathbf{P}$, $\lambda >0$, and $x \in {\mathcal{H}}$, the capacity of the projector $\mathsf{C}_{\Pj}(x,\lambda):= \frac{\|\mathbf{H}(x)^{1/2}(\mathbf{I} - \mathbf{P})\|^2}{\lambda}$.
\end{itemize}
\end{definition} 



\subsection{Relating the projected to the original problem\label{app:proj-to-ideal}}

Given $x \in {\mathcal{H}}_\mathbf{P}$, our aim is to bound $\nu_\lambda(x)$ given $\nu_{\lambda,\mathbf{P}}(x)$ and $\mathsf{s}_\lambda$. 

\begin{proposition} \label{prp:link_proj}
Let $x \in {\mathcal{H}}_\mathbf{P}$. If
\[\frac{\mathsf{s}_\lambda}{\mathsf{r}_\lambda(x_\lambda^{\star})} \leq \frac{1}{4},~ \mathsf{C}_{\Pj}(x_\lambda^{\star},\lambda) \leq \frac{1}{120},~ \nu_{\mathbf{P},\lambda}(x) \leq \frac{\mathsf{r}_{\Pj,\lambda}(x)}{2}, \]
Then it holds: 
\[\nu_\lambda(x) \leq 3(\nu_{\mathbf{P},\lambda}(x) + \mathsf{s}_\lambda).\]
Moreover, under these conditions, 
\begin{itemize}
    \item $  \|x - x_\lambda^{\star}\| \leq 7 \lambda^{-1/2}(\nu_{\mathbf{P},\lambda}(x) + \mathsf{s}_\lambda) $;
    \item $\lambda \|x\|_{\mathbf{H}^{-1}_{\mathbf{P},\lambda}(x)} \leq 7 \nu_{\mathbf{P},\lambda}(x) + 9 \mathsf{s}_\lambda$.
\end{itemize}

\end{proposition} 

\begin{proof}
In this proof, introduce the following auxiliary quantity:

\[\gamma_\lambda:= \frac{\mathsf{s}_\lambda}{\mathsf{r}_\lambda(x_\lambda^{\star})}.\]
\paragraph{1) Start by bounding $\mathsf{t}(\mathbf{P}x_\lambda^{\star} - x_\lambda^{\star})$.}

It holds:

\eqals{\mathsf{t}(\mathbf{P} x - x_\lambda^{\star}) &= \sup_{g \in \mathcal{G}}|g \cdot (\mathbf{I} - \mathbf{P}) x_\lambda^{\star}| \\
& \leq \frac{1}{\mathsf{r}_\lambda(x_\lambda^{\star})} ~ \|(\mathbf{I} - \mathbf{P})x_\lambda^{\star}\|_{\mathbf{H}_\lambda(x_\lambda^{\star})}\\
&\leq \frac{1}{\mathsf{r}_\lambda(x_\lambda^{\star})}~\|\mathbf{H}_\lambda(x_\lambda^{\star})^{1/2}(\mathbf{I} - \mathbf{P}) \mathbf{H}_\lambda(x_\lambda^{\star})^{1/2}\|~\|\mathbf{H}^{-1/2}_\lambda(x_\lambda^{\star})x_\lambda^{\star}\|\\
& = \left(1 + \mathsf{C}_{\Pj}(x_\lambda^{\star},\lambda)\right) ~\frac{\lambda \|\mathbf{H}^{-1/2}_\lambda(x_\lambda^{\star})x_\lambda^{\star}\|}{\mathsf{r}_\lambda(x_\lambda^{\star})}\\
& = (1 + \mathsf{C}_{\Pj}(x_\lambda^{\star},\lambda)) ~\gamma_\lambda
.}

\paragraph{2) Then bound $\mathsf{t}(x_{\Pj,\lambda}^* - \mathbf{P} x_\lambda^{\star})$}
First, bound $\nu_{\mathbf{P},\lambda}(\mathbf{P} x_\lambda^{\star})$: 

\eqals{\nu_{\mathbf{P},\lambda}(\mathbf{P} x_\lambda^{\star}) & = \|\mathbf{P} \nabla f_\lambda(\mathbf{P} x_\lambda^{\star}) \|_{\mathbf{H}_{\lambda,\mathbf{P}}(\mathbf{P} x_\lambda^{\star})^{-1}} \\
&\leq \| \nabla f_\lambda(\mathbf{P} x_\lambda^{\star}) \|_{\mathbf{H}_{\lambda}(\mathbf{P} x_\lambda^{\star})^{-1}} 
.}

Using \cref{eq:hl}, we get $\| \nabla f_\lambda(\mathbf{P} x_\lambda^{\star}) \|_{\mathbf{H}_{\lambda}(\mathbf{P} x_\lambda^{\star})^{-1}} \leq e^{\mathsf{t}((\mathbf{I} - \mathbf{P})x_\lambda^{\star})/2} \nu_\lambda(\mathbf{P} x_\lambda^{\star})$. Using \cref{eq:go}, we can bound 
$$\nu_\lambda(\mathbf{P} x_\lambda^{\star}) \leq \overline{\phi}(\mathsf{t}((\mathbf{I} - \mathbf{P})x_\lambda^{\star}))~\|(\mathbf{I} - \mathbf{P})x_\lambda^{\star}\|_{\mathbf{H}_\lambda(x_\lambda^{\star})} \leq  \overline{\phi}(\mathsf{t}((\mathbf{I} - \mathbf{P})x_\lambda^{\star}))~(1 + \mathsf{C}_{\Pj}(x_\lambda^{\star},\lambda))\mathsf{s}_\lambda.$$ 
Putting things together,

\[\nu_{\mathbf{P},\lambda}(\mathbf{P} x_\lambda^{\star}) \leq e^{\mathsf{t}((\mathbf{I} - \mathbf{P})x_\lambda^{\star})/2} \overline{\phi}(\mathsf{t}((\mathbf{I} - \mathbf{P})x_\lambda^{\star}))~(1 + \mathsf{C}_{\Pj}(x_\lambda^{\star},\lambda))\mathsf{s}_\lambda.\]
Now 
\[\frac{1}{\mathsf{r}_{\Pj,\lambda}(\mathbf{P} x_\lambda^{\star})}\leq \frac{1}{\mathsf{r}_\lambda(\mathbf{P} x_\lambda^{\star})} \leq  e^{\mathsf{t}((\mathbf{I} - \mathbf{P})x_\lambda^{\star})/2} \frac{1}{\mathsf{r}_\lambda(x_\lambda^{\star})}.\]
Hence, 

\[\frac{\nu_{\mathbf{P},\lambda}(\mathbf{P} x_\lambda^{\star}) }{\mathsf{r}_{\Pj,\lambda}(\mathbf{P} x_\lambda^{\star})}\leq e^{\tilde{t}_\lambda} \overline{\phi}(\tilde{t}_\lambda)~\tilde{t}_\lambda,\qquad\tilde{t}_\lambda = (1 + \mathsf{C}_{\Pj}(x_\lambda^{\star},\lambda))\gamma_\lambda. \]

Since $t \mapsto e^{t} \overline{\phi}(t)~t$ is an increasing function whose value in $0$ is $0$, we find numerically that for $t = \frac{3}{10}$, $e^{t} \overline{\phi}(t)~t \leq \frac{1}{2}$. Hence, if $(1 + \mathsf{C}_{\Pj}(x_\lambda^{\star},\lambda))\gamma_\lambda \leq \frac{3}{10}$, then $\frac{\nu_{\mathbf{P},\lambda}(\mathbf{P} x_\lambda^{\star}) }{\mathsf{r}_{\Pj,\lambda}(\mathbf{P} x_\lambda^{\star})} \leq \frac{1}{2}$. Using \cref{lm:loc}, this shows that 
$$\mathsf{t}_\mathbf{P}(\mathbf{P} x_\lambda^{\star} - x_{\Pj,\lambda}^*) = \mathsf{t}(\mathbf{P} x_\lambda^{\star} - x_{\Pj,\lambda}^*) \leq \log 2 .$$ 

\paragraph{3) Getting a bound for $\mathsf{t}(x - x_\lambda^{\star})$.} To do so, combine the two previous bounds with the fact that if $\nu_{\mathbf{P},\lambda}(x) \leq \frac{\mathsf{r}_{\Pj,\lambda}(x)}{2}$, then using \cref{lm:loc} with $f_{\mathbf{P}}$, $\mathsf{t}_\mathbf{P}(x - x_{\Pj,\lambda}^*) = \mathsf{t}(x-x_{\Pj,\lambda}^*) \leq \log 2$. Thus, if 

\[(1 + \mathsf{C}_{\Pj}(x_\lambda^{\star},\lambda))\gamma_\lambda \leq \frac{3}{10},~\nu_{\mathbf{P},\lambda}(x) \leq \frac{\mathsf{r}_{\Pj,\lambda}(x)}{2},\]
then it holds
\[\mathsf{t}(x - x_\lambda^{\star}) \leq \frac{3}{10} + 2 \log 2.\]
\paragraph{4) A technical result to bound $\|\mathbf{H}_\lambda(x)^{-1/2} \mathbf{H}_{\mathbf{P},\lambda}(x)^{1/2}\|$}. Using the fact that $\mathbf{P} x = x$, and \cref{lm:proj_hermit}, applied to $\mathbf{A} = \mathbf{H}(x)$, we get 
\[\|\mathbf{H}_\lambda(x)^{-1/2} \mathbf{H}_{\mathbf{P},\lambda}(x)^{1/2}\| \leq  1 + \sqrt{\mathsf{C}_{\Pj}(x,\lambda)}.\]

Then, one can easily bound $\mathsf{C}_{\Pj}(x,\lambda) \leq e^{\mathsf{t}(x-x_\lambda^{\star})}\mathsf{C}_{\Pj}(x_\lambda^{\star},\lambda)$. 
\paragraph{5) Let us now bound $\nu_\lambda(x)$.}

First, decompose the term 
\[\nu_\lambda(x) = \|\nabla f_\lambda(x)\|_{\mathbf{H}^{-1}_\lambda(x)} \leq  \|\mathbf{P} \nabla f_\lambda(x)\|_{\mathbf{H}^{-1}_\lambda(x)} + \|(\mathbf{I} - \mathbf{P})\nabla f(x)\|_{\mathbf{H}^{-1}_\lambda(x)} .\]

Since $x \in {\mathcal{H}}_\mathbf{P}$,  $\|\mathbf{P} \nabla f_\lambda(x)\|_{\mathbf{H}^{-1}_\lambda(x)} = \|\nabla f_{\mathbf{P},\lambda}(x)\|_{\mathbf{H}^{-1}_\lambda(x)}$, and using the previous point, we get

\[\|\mathbf{P} \nabla f_\lambda(x)\|_{\mathbf{H}^{-1}_\lambda(x)} \leq  \left(1 + e^{\mathsf{t}(x - x_\lambda^{\star})/2}\sqrt{\mathsf{C}_{\Pj}(x_\lambda^{\star},\lambda)}\right) \nu_{\mathbf{P},\lambda}(x).\]

Let us now bound the second term. We divide it into two terms: 
\[\|(\mathbf{I} - \mathbf{P})\nabla f(x)\|_{\mathbf{H}^{-1}_\lambda(x)} \leq \|(\mathbf{I} - \mathbf{P})\left(\nabla f(x) - \nabla f(x_\lambda^{\star})\right)\|_{\mathbf{H}^{-1}_\lambda(x)} + \|(\mathbf{I} - \mathbf{P})\nabla f(x_\lambda^{\star})\|_{\mathbf{H}^{-1}_\lambda(x)}.\]
The second term can be bounded in the following way: 
\[\|(\mathbf{I} - \mathbf{P})\nabla f(x_\lambda^{\star})\|_{\mathbf{H}^{-1}_\lambda(x)} \leq \frac{1}{\sqrt{\lambda}} \|(\mathbf{I} - \mathbf{P})\mathbf{H}^{1/2}_\lambda(x_\lambda^{\star})\|~\|\nabla f(x_\lambda^{\star})\|_{\mathbf{H}^{-1}_\lambda(x_\lambda^{\star})} \leq  \sqrt{1 + \mathsf{C}_{\Pj}(x_\lambda^{\star},\lambda)} ~\mathsf{s}_\lambda.\]
For the first term, we proceed in the following way. 
\eqals{
\|(\mathbf{I} - \mathbf{P})\left(\nabla f(x) - \nabla f(x_\lambda^{\star})\right)\|_{\mathbf{H}^{-1}_\lambda(x)} & = \| \int_{0}^1{\mathbf{H}_\lambda^{-1/2}(x)(\mathbf{I} - \mathbf{P}) \mathbf{H}(x_t)(x-x_\lambda^{\star})~dt} \| \\
& \leq \frac{1}{\sqrt{\lambda}}~\int_{0}^1{\|(\mathbf{I} - \mathbf{P}) \mathbf{H}^{1/2}(x_t)\|~ \|\mathbf{H}^{1/2}(x_t)(x-x_\lambda^{\star})\|~dt}\\
& \leq \sqrt{ \mathsf{C}_{\Pj}(x_\lambda^{\star},\lambda)}~\overline{\phi}(\mathsf{t}(x - x_\lambda^{\star}))~\|x - x_\lambda^{\star}\|_{\mathbf{H}(x_\lambda^{\star})}\\
& \leq \sqrt{ \mathsf{C}_{\Pj}(x_\lambda^{\star},\lambda)} ~e^{\mathsf{t}(x - x_\lambda^{\star})} \nu_\lambda(x)
.}
Hence the final bound: 

\[\left(1 - \sqrt{ \mathsf{C}_{\Pj}(x_\lambda^{\star},\lambda)} ~e^{\mathsf{t}(x - x_\lambda^{\star})}\right)\nu_\lambda(x) \leq \left(1 + e^{\mathsf{t}(x - x_\lambda^{\star})/2}\sqrt{\mathsf{C}_{\Pj}(x_\lambda^{\star},\lambda)}\right) \nu_{\mathbf{P},\lambda}(x) +  \sqrt{1 + \mathsf{C}_{\Pj}(x_\lambda^{\star},\lambda)} ~\mathsf{s}_\lambda.\]

Now if $\mathsf{C}_{\Pj}(x_\lambda^{\star},\lambda) \leq \frac{1}{120}$, we see that $\sqrt{ \mathsf{C}_{\Pj}(x_\lambda^{\star},\lambda)} ~e^{\mathsf{t}(x - x_\lambda^{\star})} \leq \frac{1}{2}$, and hence, using the bound on $\mathsf{t}(x - x_\lambda^{\star})$, 

\[\nu_\lambda(x) \leq 3(\nu_{\mathbf{P},\lambda}(x) + \mathsf{s}_\lambda).\]

\paragraph{6) Showing the last two points}. We leverage the fact that $\nu_\lambda(x) \leq 3(\nu_{\mathbf{P},\lambda}(x) + \mathsf{s}_\lambda)$ and $\mathsf{t}(x - x_\lambda^{\star}) \leq \frac{3}{10} + 2 \log 2$.\\
To show the first bound, we plug in the previous results in the following equation: 
\[\|x - x_\lambda^{\star}\| \leq \lambda^{-1/2}\|x - x_\lambda^{\star}\|_{\mathbf{H}_\lambda(x)} \leq \frac{1}{\underline{\phi}(\mathsf{t}(x - x_\lambda^{\star}))} ~ \lambda^{-1/2} \nu_\lambda(x).\]
The last inequality is obtained using \cref{eq:gl}.\\

To show the second point, we use the fact that $x \in {\mathcal{H}}_\mathbf{P}$ to show that 
\[\lambda \|x\|_{\mathbf{H}_{\mathbf{P},\lambda}^{-1}(x)} \leq  \lambda \|x\|_{\mathbf{H}^{-1}_{\lambda}(x)} \leq \lambda \|x - x_\lambda^{\star}\|_{\mathbf{H}_\lambda(x)} + \lambda\|x_\lambda^{\star}\|_{\mathbf{H}^{-1}_\lambda(x)}.\]
Then applying \cref{eq:hl} and \cref{eq:gl}:
\[\lambda \|x\|_{\mathbf{H}_{\mathbf{P},\lambda}^{-1}(x)} \leq \frac{1}{\underline{\phi}(\mathsf{t}(x - x_\lambda^{\star}))} ~ \nu_\lambda(x) + e^{\mathsf{t}(x-x_\lambda^{\star})/2} \mathsf{s}_\lambda.\]
We then use the previous results to conclude.
\end{proof}

\subsection{Finding a good projector}

\begin{lemma} 
If for a certain $\eta \leq \lambda$ and for a certain constant $C$, $\|\mathbf{H}_\eta^{1/2}(x)(\mathbf{I}-\mathbf{P})\|^2 \leq C \eta$, then 
$$\mathsf{C}_{\Pj}(x,\lambda) \leq \frac{C\eta}{\lambda}.$$
\end{lemma}

\begin{proof} 
This is completely direct, using the fact that $\mathbf{H}^{1/2}(x)\preceq \mathbf{H}_\eta^{1/2}(x)$.
\end{proof} 




\newpage 

\section{Relations between statistical problems and empirical problem.\label{app:proj_stat}}

In this section, we recall and reformulate the framework from \cite{marteau2019}.

\subsection{Statistical problem and ERM estimator}


Let $\mathcal{Z}$ be a Polish space and $Z$ be a random variable on $\mathcal{Z}$ with distribution $\rho$. 
Let ${\mathcal{H}}$ be a separable Hilbert space, with norm $\|\cdot\|$, and let $(f_z)_{z \in \mathcal{Z}}$ be a family of functions on ${\mathcal{H}}$.
Our goal is to minimize the \textit{expected risk} with respect to $x \in {\mathcal{H}}$:

$$ \inf_{x \in {\mathcal{H}}}~f(x):= \Exp{f_Z(x)}.$$ 
Given $(z_i)_{i=1}^n \in \mathcal{Z}^n$, we define the \textit{empirical risk}:

\[\widehat{f}(x):= \frac{1}{n}\sum_{i=1}^n f_{z_i}(x)  ,\]
and consider the following estimator based on regularized empirical risk minimization given $\lambda>0$ (note that the minimizer is unique in this case):

\[
\widehat{x}_\lambda^\star = \argmin_{x \in {\mathcal{H}}} \widehat{f}_\lambda(x):= \widehat{f}(x)  + \frac{\lambda}{2} \|x\|^2,
\]
where we assume the following.
\begin{restatable}[i.i.d. data]{ass}{asmiid}
\label{asm:iid}
The samples $(z_i)_{1 \leq i \leq n}$ are independently and identically distributed according to $\rho$. 
\end{restatable}
We make the following assumption on the family $(f_z)$ (this is a reformulation of Assumption 8 in \cite{marteau2019})
\begin{restatable}[Generalized self-concordance]{ass}{asmgensc}
\label{asm:gen_sc}
For any $z \in \mathcal{Z}$, there exists an associated subset $\mathcal{G}_z \subset {\mathcal{H}}$ such that $(f_z,\mathcal{G}_z)$ is generalized self-concordant in the sense of \cref{df:genscc}. 
\end{restatable}


Moreover we require the following technical assumption to guarantee that $f$ and and its derivatives are well defined for any $x \in {\mathcal{H}}$ (this is a reformulation of Assumptions 3 and 4 in \cite{marteau2019}, and the necessary conditions to obtain \cref{prp:expectancy}).

\begin{restatable}[Technical assumptions]{ass}{asmtech}
\label{asm:asmtech}
The mapping $(z,x) \in \mathcal{Z} \times {\mathcal{H}} \mapsto f_z(x)$ is measurable. Moreover, 
\begin{itemize}
    \item the random variables $\|f_Z(0)\|, \|\nabla f_Z(0)\|, \Tr(\nabla^2 f_Z(0))$ are are bounded;
    \item $\mathcal{G}:= \bigcup_{z \in \supp(Z)}\mathcal{G}_z$ is a bounded subset of ${\mathcal{H}}$.
\end{itemize}
\end{restatable}
The assumptions above are usually easy to check in practice. In particular, if the support of~$\rho$ is bounded, the mappings $z \mapsto \ell_z(0),\nabla \ell_z(0), \Tr(\nabla^2\ell_z(0))$ are continuous, and $z \mapsto \mathcal{G}_z$ is uniformly bounded on bounded sets, then they hold. \\\

\begin{proposition} \label{prp:scProblems}

Under \cref{asm:gen_sc,asm:asmtech}, the function $(f,\mathcal{G})$ (or simply $f$) is generalized self-concordant. 

Moreover, under \cref{asm:iid}, define 
\[\widehat{\mathcal{G}}:= \bigcup_{i=1}^n{\mathcal{G}_{z_i}}.\]
Then $(\widehat{f},\widehat{\mathcal{G}})$ (or simply $\widehat{f}$) is generalized self-concordant. Moreover, note that $\widehat{\mathcal{G}} \subset \mathcal{G}$. 

\end{proposition}





The main regularity assumption we make on our statistical problems follows (see Assumption 5 in \cite{marteau2019}). 



\begin{restatable}[Existence of a minimizer]{ass}{asmminimizer}
\label{asm:minimizer}
There exists $x^\star \in {\mathcal{H}}$ such that $ f(x^\star) = \inf_{x \in {\mathcal{H}}} f(x).$
\end{restatable}



\paragraph{Notations}


We adopt all the notations from \cref{app:sc} for $f$ and $\widehat{f}$, which are generalized self-concordant functions with associated subsets given in \cref{prp:scProblems} with the following conventions:

\begin{itemize}
    \item For all quantities relating to $f$, we omit the subscript $f$ as usual;
    \item For all quantities relating to $\widehat{f}$, we omit the subscript $\widehat{f}$ and instead put a hat over all these quantities. For instance:
    \[\widehat{\mathbf{H}}(x):= \mathbf{H}_{\widehat{f}} (x) = \frac{1}{n}\sum_{i=1}^n{\nabla^2f_{z_i}(x)},~ \widehat{\mathsf{r}}_{\lambda}(x):= \mathsf{r}_{\widehat{f},\lambda}(x) = \frac{1}{\sup_{g \in \widehat{\mathcal{G}}}{\|g\|_{\widehat{\mathbf{H}}^{-1}_\lambda(x)}}}, \text{ etc...}\]
\end{itemize}

Recall the two main quantities introduced in \cite{marteau2019} to establish the quality of our estimator $\widehat{x}_\lambda^\star$ (in \cite{marteau2019}, this is a mix between Proposition 2 and Definition 3).
\begin{proposition}[Bias, degrees of freedom] \label{df:main_quant_stat} Suppose  \cref{asm:gen_sc,asm:asmtech,asm:minimizer} are satisfied. The following key quantities are well defined: 
\begin{itemize}
    \item the \emph{bias} $\mathsf{b}_\lambda = \|\mathbf{H}_\lambda(x^\star)^{-1/2} \nabla f_\lambda(x^\star)\|$;
    \item the \emph{effective dimension} $\mathsf{df}_\lambda = \Exp{\|\mathbf{H}_\lambda(x^\star)^{-1/2}\nabla f_Z(x^\star)\|^2}$.
\end{itemize}
Moreover, we also introduce the following quantities:
\[\mathsf{B}_1^\star:= \sup_{z \in \supp(Z)}{\|\nabla f_z(x^\star)\|},\qquad \mathsf{B}_2^\star:= \sup_{z \in \supp(Z)}{\Tr(\nabla^2 f_z(x^\star))},\qquad \mathsf{Q}^\star = \frac{\mathsf{B}_1^\star}{\sqrt{\mathsf{B}_2^\star}}.\]
\end{proposition}

We can now recall the main theorem of \cite{marteau2019} (Theorem 4), which quantifies the behavior of the ERM estimator: 

\begin{theorem}[Bound for the ERM estimator]\label[theorem]{thm:general-result}
Let $n \in \mathbb{N}$, $\delta \in (0,1/2]$, $0 < \lambda \leq \mathsf{B}_2^\star$. Whenever
\[n \geq \triangle_1 \frac{\mathsf{B}_2^\star}{\lambda}\log \frac{8 \square_1^2\mathsf{B}_2^\star}{\lambda \delta},\qquad \sqrt{\triangle_2~  \frac{\mathsf{df}_\lambda \vee (\mathsf{Q}^\star)^2}{n} ~\log \frac{2}{\delta}}\leq \mathsf{r}_\lambda(x^\star) ,\qquad 2\mathsf{b}_\lambda \leq \mathsf{r}_\lambda(x^\star),\]
 then with probability at least $1-2 \delta$, it holds
\eqal{\label{eq:general-result-bound}
f(\widehat{x}_\lambda^\star) - f(x^\star) \leq \mathsf{C}_{\textup{bias}}~\mathsf{b}_\lambda^2 + \mathsf{C}_{\textup{var}}~\frac{\mathsf{df}_\lambda \vee (\mathsf{Q}^\star)^2}{n}~\log \frac{2}{\delta},
}
where $\mathsf{C}_{\textup{bias}},\mathsf{C}_{\textup{var}}, \square_1 \leq 414,~\triangle_1,~\triangle_2 \leq 5184$.
\end{theorem}


\subsection{Link between a good approximation of $\widehat{x}_\lambda^\star$ and $x^\star$}

In this paper, we provide an algorithm which can effectively compute a good approximation of $\widehat{x}_\lambda^\star$ (as it is a finite sum problem which can be solved). This algorithm will return a certain $x \in {\mathcal{H}}$, whose precision with respect to the empirical problem will be characterized by $\widehat{\nu}_\lambda(x)$. The aim of the following lemma is to see how this approximation $x$ behaves with respect to the statistical problem. 

\begin{lemma}\label{lm:approx_emp_pb} Suppose the conditions for \cref{thm:general-result} are satisfied, i.e.
let $n \in \mathbb{N}$, $\delta \in (0,1/2]$, $0 < \lambda \leq \mathsf{B}_2^\star$ and suppose
\[n \geq \triangle_1 \frac{\mathsf{B}_2^\star}{\lambda}\log \frac{8 \square_1^2\mathsf{B}_2^\star}{\lambda \delta},\qquad  \sqrt{\triangle_2~  \frac{\mathsf{df}_\lambda \vee (\mathsf{Q}^\star)^2}{n} ~\log \frac{2}{\delta}} \leq \mathsf{r}_\lambda(x^\star), \qquad 2\mathsf{b}_\lambda \leq \mathsf{r}_\lambda(x^\star).\]
Let $x$ be an approximation of $\widehat{x}_\lambda^\star$ characterized by its Newton decrement $\widehat{\nu}_\lambda(x)$. If 
\[\widehat{\nu}_\lambda(x) \leq \frac{\widehat{\mathsf{r}}_{\lambda}(x)}{2},~ \widehat{\nu}_\lambda(x) \leq \frac{\mathsf{r}_\lambda(x^\star)}{2},\]
then with probability at least $1-2 \delta$, it holds

\[f(x) - f(x^\star) \leq 14(f(\widehat{x}_\lambda^\star) - f(x^\star)) + 30\widehat{\nu}_\lambda(x)^2.\]


\end{lemma}
\begin{proof} 
Using \cref{eq:fv}, 
\eqals{
f(x) - f(\widehat{x}_\lambda^\star) &\leq \langle \nabla f(\widehat{x}_\lambda^\star),x - \widehat{x}_\lambda^\star\rangle_{{\mathcal{H}}} + \psi(\nm{x - \widehat{x}_\lambda^\star})\|x - \widehat{x}_\lambda^\star\|_{\mathbf{H}_\lambda(\widehat{x}_\lambda^\star)}^2\\
& \leq \frac{1}{2}\| \nabla f(\widehat{x}_\lambda^\star)\|^2_{\mathbf{H}^{-1}_\lambda(\widehat{x}_\lambda^\star)} + \left(\psi(\nm{x - \widehat{x}_\lambda^\star}) + \frac{1}{2}\right)\|x - \widehat{x}_\lambda^\star\|^2_{\mathbf{H}_\lambda(\widehat{x}_\lambda^\star)}
.}

\paragraph{1. Let us bound $\|\nabla f(\widehat{x}_\lambda^\star)\|_{\mathbf{H}^{-1}_\lambda(\widehat{x}_\lambda^\star)}$}

\eqals{
\|\nabla f(\widehat{x}_\lambda^\star)\|_{\mathbf{H}^{-1}_\lambda(\widehat{x}_\lambda^\star)} & \leq \int_{0}^1{\|\mathbf{H}^{-1/2}_\lambda(\widehat{x}_\lambda^\star)\mathbf{H}(x_t)(\widehat{x}_\lambda^\star - x^\star)\|~dt}, &x_t = (1-t)\widehat{x}_\lambda^\star + tx^\star \\
&\leq \int_{0}^1{\|\mathbf{H}^{-1/2}_\lambda(\widehat{x}_\lambda^\star)\mathbf{H}^{1/2}(x_t)\| ~\|\mathbf{H}^{1/2}(x_t)(\widehat{x}_\lambda^\star - x^\star)\|~dt}.}
Now using equation \cref{eq:h}
\[\mathbf{H}(x_t) \preceq e^{t \nm{\widehat{x}_\lambda^\star - x^\star}}\mathbf{H}(\widehat{x}_\lambda^\star),\qquad \mathbf{H}(x_t) \preceq e^{(1-t)\nm{\widehat{x}_\lambda^\star - x^\star}} .\]
Thus:

\[\|\nabla f(\widehat{x}_\lambda^\star)\|_{\mathbf{H}^{-1}_\lambda(\widehat{x}_\lambda^\star)} \leq e^{\nm{\widehat{x}_\lambda^\star - x^\star}/2} ~ \|\widehat{x}_\lambda^\star - x^\star\|_{\mathbf{H}(x^\star)} .\]

Finally, using equation \cref{eq:fv}
\[ \|\nabla f(\widehat{x}_\lambda^\star)\|_{\mathbf{H}^{-1}_\lambda(\widehat{x}_\lambda^\star)} \leq \frac{e^{\nm{\widehat{x}_\lambda^\star - x^\star}/2} }{\psi(-\nm{\widehat{x}_\lambda^\star - x^\star})^{1/2}}\left( f(\widehat{x}_\lambda^\star) - f(x^\star)\right)^{1/2}.
\]

\paragraph{2. Let us bound the terms involving $ \|x - \widehat{x}_\lambda^\star\|_{\mathbf{H}_\lambda(\widehat{x}_\lambda^\star)} $}

Note that using \cref{eq:gl} and \cref{eq:hl} applied to $\widehat{f}$, 

\[\|x - \widehat{x}_\lambda^\star\|_{\mathbf{H}_\lambda(\widehat{x}_\lambda^\star)}  \leq \|\mathbf{H}^{1/2}_\lambda(\widehat{x}_\lambda^\star) \widehat{\mathbf{H}}^{-1/2}_\lambda(\widehat{x}_\lambda^\star)\|~\frac{e^{\hnm{x - \widehat{x}_\lambda^\star}/2}}{\underline{\phi}(\hnm{x - \widehat{x}_\lambda^\star})} \widehat{\nu}_\lambda(x).\]

This also leads to:


\eqals{
\nm{x - \widehat{x}_\lambda^\star} &\leq \frac{1}{\mathsf{r}_\lambda(\widehat{x}_\lambda^\star)}~\|\mathbf{H}^{1/2}_\lambda(\widehat{x}_\lambda^\star) \widehat{\mathbf{H}}^{-1/2}_\lambda(\widehat{x}_\lambda^\star)\|~\|x - \widehat{x}_\lambda^\star\|_{\widehat{\mathbf{H}}_\lambda(\widehat{x}_\lambda^\star)}\\
& \leq \frac{1}{\mathsf{r}_\lambda(\widehat{x}_\lambda^\star)}~\|\mathbf{H}^{1/2}_\lambda(\widehat{x}_\lambda^\star) \widehat{\mathbf{H}}^{-1/2}_\lambda(\widehat{x}_\lambda^\star)\|~\frac{e^{\hnm{x - \widehat{x}_\lambda^\star}/2}}{\underline{\phi}(\hnm{x - \widehat{x}_\lambda^\star})} \widehat{\nu}_\lambda(x).
}


\paragraph{3. Putting things together}

In the end, we get 

\eqals{f(x) - f(x^\star) &\leq \left(1 + \frac{e^{\nm{\widehat{x}_\lambda^\star - x^\star}} }{\psi(-\nm{\widehat{x}_\lambda^\star - x^\star}) }\right)(f(\widehat{x}_\lambda^\star)  - f(x^\star))\\
& + \left(\psi(\nm{x - \widehat{x}_\lambda^\star}) + \frac{1}{2}\right)\left(e^{\nm{\widehat{x}_\lambda^\star - x^\star_\lambda}/2}\|\mathbf{H}^{1/2}_\lambda(x^\star_\lambda) \widehat{\mathbf{H}}^{-1/2}_\lambda(x^\star_\lambda)\|\frac{e^{\hnm{x - \widehat{x}_\lambda^\star}/2}}{\underline{\phi}(\hnm{x - \widehat{x}_\lambda^\star})}\right)~\widehat{\nu}_\lambda(x)^2
.}

Moreover, we bound 
\[\nm{x - \widehat{x}_\lambda^\star} \leq e^{(\nm{x^\star - \widehat{x}_\lambda^\star} + \nm{\widehat{x}_\lambda^\star - x^\star_\lambda})/2}~\|\mathbf{H}^{1/2}_\lambda(x^\star_\lambda) \widehat{\mathbf{H}}^{-1/2}_\lambda(x^\star_\lambda)\|~\frac{e^{\hnm{x - \widehat{x}_\lambda^\star}/2}}{\underline{\phi}(\hnm{x - \widehat{x}_\lambda^\star})} \frac{\widehat{\nu}_\lambda(x)}{\mathsf{r}_\lambda(x^\star)}.\]

\paragraph{4. Plugging in previous results}

Under the assumptions of this lemma, which include the assumptions of Theorem 4. in \cite{marteau2019}, we get the following bounds.
\begin{itemize}
    \item In \cite{marteau2019},the assumptions of Theorem 4 imply that we can use Lemma 9, which uses Lemma 8 in which we show that with probability at least $1-\delta$, 
    \[\|\widehat{\mathbf{H}}_\lambda^{-1/2}(x_\lambda^{\star})\mathbf{H}_\lambda(x_\lambda^{\star})^{1/2}\|^2 \leq 2.\]
    \item Still using the assumptions of Theorem 4, we see in the proof of this theorem that the assumptions of Theorem 7 of \cite{marteau2019} are satisfied in the case where $\mathsf{b}_\lambda \leq \frac{\mathsf{r}_\lambda(x^\star)}{2}$, and thus that 
    $$\mathsf{t}(\widehat{x}_\lambda^\star - x^\star_\lambda) \leq \log 2,~ \mathsf{t}(x^\star_\lambda - x^\star) \leq \log 2.$$
    
\end{itemize}

Plugging in all these bounds, we get

\[\left(1 + \frac{e^{\nm{\widehat{x}_\lambda^\star - x^\star}} }{\psi(-\nm{\widehat{x}_\lambda^\star - x^\star}) }\right) \leq 14,~\nm{x - \widehat{x}_\lambda^\star} \leq 6,\]
\[\left(\psi(\nm{x - \widehat{x}_\lambda^\star}) + \frac{1}{2}\right)\left(e^{\nm{\widehat{x}_\lambda^\star - x^\star_\lambda}/2}\|\mathbf{H}^{1/2}_\lambda(x^\star_\lambda) \widehat{\mathbf{H}}^{-1/2}_\lambda(x^\star_\lambda)\|\frac{e^{\hnm{x - \widehat{x}_\lambda^\star}/2}}{\underline{\phi}(\hnm{x - \widehat{x}_\lambda^\star})}\right) \leq 30.\]
\end{proof}


\subsection{Bounds when we solve a projected empirical problem \label{app:bounds_proj}}

In this section, we place ourselves in the setting of \cref{app:proj}. In this section, we had argued that for computational purposes, it was less costly to compute an approximate solution to a projected problem.\\\

In this section, we assume that we are going to project the regularized empirical problem, that is solve approximately

\[ x \approx \argmin_{x \in {\mathcal{H}}}{\widehat{f}_{\mathbf{P},\lambda}(x)} = \widehat{f}(\mathbf{P} x) + \frac{\lambda}{2}\|x\|^2.\]

for a given orthogonal projection $\mathbf{P}$. Recall from \cref{app:proj} that there is a natural way of seeing $\widehat{f}_\mathbf{P}$ as a generalized self-concordant function. We import all the notations from this section, keeping a $\widehat{\cdot}$ over all notations to mark the fact that we are projecting $\widehat{f}$ and not $f$. 

To quantify the quality of the approximation $x$, we will use the Newton decrement for the empirical projected problem $\widehat{\nu}_{\mathbf{P},\lambda}(x):=\nu_{\widehat{f}_\mathbf{P},\lambda}(x)$.\\\

As we see in \cref{prp:link_proj}, under certain conditions, bounding $\widehat{\nu}_\lambda(x)$ amounts to bounding two terms:

\begin{itemize}
    \item The empirical source $\widehat{s}_\lambda:= \lambda \|\widehat{x}_\lambda^\star\|_{\widehat{\mathbf{H}}_\lambda^{-1}(\widehat{x}_\lambda^\star)}$,
    \item The projected empirical Newton decrement $\widehat{\nu}_{\mathbf{P},\lambda}(x)$.
\end{itemize}

\paragraph{1. Bounding the empirical source term $\widehat{s}_\lambda$} Start by bounding the source empirical source term using quantities we know.





\begin{lemma}[Empirical source]\label{lm:emp_source}
Let $n \in \mathbb{N}$, $\delta \in (0,1/2]$, $0 < \lambda \leq \mathsf{B}_2^\star$. Whenever
\[n \geq \triangle_1 \frac{\mathsf{B}_2^\star}{\lambda}\log \frac{8 \square_1^2\mathsf{B}_2^\star}{\lambda \delta},\qquad  \sqrt{\triangle_2~  \frac{\mathsf{df}_\lambda \vee (\mathsf{Q}^\star)^2}{n} ~\log \frac{2}{\delta}} \leq \mathsf{r}_\lambda(x^\star), \qquad 2\mathsf{b}_\lambda \leq \mathsf{r}_\lambda(x^\star).\]
The following holds, with probability at least $1 - 2 \delta$.


\[\widehat{s}_\lambda \leq 8~ \mathsf{b}_\lambda + 80~ \sqrt{\frac{\mathsf{df}_\lambda \vee (\mathsf{Q}^\star)^2~\log \frac{2}{\delta}}{n}}.\]

Moreover, we also have the following bound :
\[\|\widehat{x}_\lambda^\star - x^\star\| \leq 3~\lambda^{-1/2}~\mathsf{b}_\lambda + 8~\lambda^{-1/2} ~\sqrt{ \frac{\mathsf{df}_\lambda \vee (\mathsf{Q}^\star)^2~\log \frac{2}{\delta}}{n}}.\]
\end{lemma}


\begin{proof}

We first decompose the source term into two terms, and then apply different bounds from \cite{marteau2019} to effectively bound it. We will use the following quantity:
\[\widehat{\mathsf{v}}_\lambda:=  \|\mathbf{H}_\lambda^{1/2}(x_\lambda^{\star}) \widehat{\mathbf{H}}_\lambda^{-1/2}(x_\lambda^{\star})\|^2~\|\nabla \widehat{f}_\lambda(x_\lambda^{\star})\|_{\mathbf{H}_\lambda^{-1}(x_\lambda^{\star})}.\]

It is also defined in equation (23) in \cite{marteau2019}. 


\paragraph{1. Dividing $\widehat{s}_\lambda$ into two controllable terms}. Decompose
\eqals{\widehat{s}_\lambda = \|\lambda \widehat{x}_\lambda^\star\|_{\widehat{\mathbf{H}}_\lambda^{-1}(\widehat{x}_\lambda^\star)} & \leq \|\widehat{\mathbf{H}}_\lambda^{-1/2}(\widehat{x}_\lambda^\star)\mathbf{H}_\lambda^{1/2}(\widehat{x}_\lambda^\star)\| ~ \|\lambda \widehat{x}_\lambda^\star\|_{\mathbf{H}_\lambda^{-1}(\widehat{x}_\lambda^\star)}\\
& \leq 
\|\widehat{\mathbf{H}}_\lambda^{-1/2}(\widehat{x}_\lambda^\star)\mathbf{H}_\lambda^{1/2}(\widehat{x}_\lambda^\star)\| ~  \left(\|\nabla f_\lambda(\widehat{x}_\lambda^\star)\|_{\mathbf{H}^{-1}_\lambda(\widehat{x}_\lambda^\star)} + \|\nabla f(\widehat{x}_\lambda^\star)\|_{\mathbf{H}^{-1}_\lambda(\widehat{x}_\lambda^\star)}\right)
.}

On the one hand, from the previous proof, we get

\eqals{\|\nabla f(\widehat{x}_\lambda^\star)\|_{\mathbf{H}^{-1}_\lambda(\widehat{x}_\lambda^\star)} &\leq e^{\nm{\widehat{x}_\lambda^\star - x^\star}/2} ~ \|\widehat{x}_\lambda^\star - x^\star\|_{\mathbf{H}(x^\star)} \\
& \leq e^{\nm{\widehat{x}_\lambda^\star - x^\star}/2} ~\left( e^{\nm{x^\star_\lambda - x^\star}}\|\widehat{x}_\lambda^\star - x^\star_\lambda\|_{\mathbf{H}_\lambda(x^\star_\lambda)} + \|x^\star_\lambda - x^\star \|_{\mathbf{H}_\lambda(x^\star)}\right)\\
&\leq e^{\nm{\widehat{x}_\lambda^\star - x^\star}/2} ~\left( \frac{e^{\nm{x^\star_\lambda - x^\star}}}{\underline{\phi}(\nm{\widehat{x}_\lambda^\star - x^\star_\lambda})}\widehat{\mathsf{v}}_\lambda + \frac{1}{\underline{\phi}(\nm{x^\star_\lambda - x^\star})}\mathsf{b}_\lambda\right)
.}

In the last line, we use the fact that 
$\|\widehat{x}_\lambda^\star - x^\star_\lambda\|_{\mathbf{H}_\lambda(x^\star_\lambda)} \leq \| \mathbf{H}^{1/2}_\lambda(x^\star_\lambda)\widehat{\mathbf{H}}^{-1/2}_\lambda(x^\star_\lambda)\| ~\|\widehat{x}_\lambda^\star - x^\star_\lambda\|_{\widehat{\mathbf{H}}_\lambda(x^\star_\lambda)}$ and then bound it using \cref{eq:gl} applied to $\widehat{f}$ to get 
\eqals{\|\widehat{x}_\lambda^\star - x^\star_\lambda\|_{\widehat{\mathbf{H}}_\lambda(x^\star_\lambda)} &\leq 
\frac{1}{\underline{\phi}(\widehat{\mathsf{t}}(x_\lambda^{\star} - \widehat{x}_\lambda^\star))}\|\nabla \widehat{f}_\lambda(x_\lambda^{\star})\|_{\widehat{\mathbf{H}}_\lambda^{-1}(x_\lambda^{\star})} \\
& \leq \frac{1}{\underline{\phi}(\mathsf{t}(x_\lambda^{\star} - \widehat{x}_\lambda^\star))} \| \mathbf{H}^{1/2}_\lambda(x^\star_\lambda)\widehat{\mathbf{H}}^{-1/2}_\lambda(x^\star_\lambda)\|~\|\nabla \widehat{f}_\lambda(x_\lambda^{\star})\|_{\mathbf{H}_\lambda^{-1}(x_\lambda^{\star})}
.}

On the other hand, apply successively \cref{eq:gl} to $f$ and $\widehat{f}$ using the fact that $\widehat{\mathsf{t}} \leq \mathsf{t}$ to get 

\eqals{\|\nabla f_\lambda(\widehat{x}_\lambda^\star)\|_{\mathbf{H}^{-1}_\lambda(\widehat{x}_\lambda^\star)} &= \|\nabla f_\lambda(\widehat{x}_\lambda^\star) - \nabla f_\lambda(x^\star_\lambda)\|_{\mathbf{H}^{-1}_\lambda(\widehat{x}_\lambda^\star)}\\
&\leq e^{\nm{\widehat{x}_\lambda^\star - x^\star_\lambda}/2} \overline{\phi}(\nm{\widehat{x}_\lambda^\star - x^\star_\lambda})~\|\widehat{x}_\lambda^\star - x^\star_\lambda\|_{\mathbf{H}_\lambda(x^\star_\lambda)} \\
 &\leq e^{\nm{\widehat{x}_\lambda^\star - x^\star_\lambda}/2} \overline{\phi}(\nm{\widehat{x}_\lambda^\star - x^\star_\lambda})~\|\mathbf{H}^{1/2}_\lambda(x^\star_\lambda)\widehat{\mathbf{H}}^{-1/2}_\lambda(x^\star_\lambda)\|~\|\widehat{x}_\lambda^\star - x^\star_\lambda\|_{\widehat{\mathbf{H}}_\lambda(x^\star_\lambda)}\\
 & \leq \frac{e^{\nm{\widehat{x}_\lambda^\star - x^\star_\lambda}/2}\overline{\phi}(\nm{\widehat{x}_\lambda^\star - x^\star_\lambda})}{\underline{\phi}(\nm{\widehat{x}_\lambda^\star - x^\star_\lambda})}~\|\mathbf{H}^{1/2}_\lambda(x^\star_\lambda)\widehat{\mathbf{H}}^{-1/2}_\lambda(x^\star_\lambda)\|^2~\|\nabla \widehat{f}_\lambda(x^\star_\lambda)\|_{\mathbf{H}_\lambda(x^\star_\lambda)}\\
 & = e^{3\nm{\widehat{x}_\lambda^\star - x^\star_\lambda}/2} \widehat{\mathsf{v}}_\lambda .
}

Putting things together: 

\[\widehat{s}_\lambda \leq \|\widehat{\mathbf{H}}_\lambda^{-1/2}(\widehat{x}_\lambda^\star)\mathbf{H}_\lambda^{1/2}(\widehat{x}_\lambda^\star)\| ~  \left(e^{3 \mathsf{t}(x_\lambda^{\star} - \widehat{x}_\lambda^\star)/2}\left(1 + \frac{1}{\underline{\phi}(\mathsf{t}(x_\lambda^{\star} - \widehat{x}_\lambda^\star))}\right) \widehat{\mathsf{v}}_\lambda + \frac{e^{\mathsf{t}(x_\lambda^{\star} - \widehat{x}_\lambda^\star)/2}}{\underline{\phi}(\mathsf{t}(x_\lambda^{\star} - x^\star))}\mathsf{b}_\lambda \right). \]


\paragraph{2. We now import the results from \cite{marteau2019}}. The following hypotheses imply those of Thms 4 and 7 in \cite{marteau2019}:

Let $n \in \mathbb{N}$, $\delta \in (0,1/2]$, $0 < \lambda \leq \mathsf{B}_2^\star$. Whenever
\[n \geq \triangle_1 \frac{\mathsf{B}_2^\star}{\lambda}\log \frac{8 \square_1^2\mathsf{B}_2^\star}{\lambda \delta},~~~n \geq \triangle_2~  \frac{\mathsf{df}_\lambda \vee (\mathsf{Q}^\star)^2}{\mathsf{r}_\lambda(x^\star)^2} ~\log \frac{2}{\delta}, \mathsf{b}_\lambda \leq \frac{\mathsf{r}_\lambda(x^\star)}{2}.\]

In particular, they imply that with probability at least $1 - 2 \delta$: 

\begin{itemize}
    \item $\widehat{\mathsf{v}}_\lambda \leq \frac{1}{2}\mathsf{b}_\lambda + 4 \square_1~\sqrt{ \frac{\mathsf{df}_\lambda \vee (\mathsf{Q}^\star)^2~\log \frac{2}{\delta}}{n}}$;
    \item $\|\mathbf{H}^{1/2}_\lambda(x^\star_\lambda)\widehat{\mathbf{H}}^{-1/2}_\lambda(x^\star_\lambda)\| \leq \sqrt{2}$;
    \item $\nm{x^\star - x^\star_\lambda} \leq \log 2$;
    \item $\nm{\widehat{x}_\lambda^\star - x^\star_\lambda} \leq \log 2$.
\end{itemize}

Hence, plugging these bounds in the previous equation, we get 

\[\widehat{s}_\lambda \leq 8 \mathsf{b}_\lambda + 80 \sqrt{\frac{\mathsf{df}_\lambda \vee (\mathsf{Q}^\star)^2~\log \frac{2}{\delta}}{n}}.\]

\paragraph{3.} Note that in what has been done previously, we can bound: 

\[\|\widehat{x}_\lambda^\star - x_\lambda^{\star}\|_{\mathbf{H}_\lambda(x_\lambda^{\star})} \leq \frac{1}{\underline{\phi}(\mathsf{t}(x_\lambda^{\star} - \widehat{x}_\lambda^\star))} \widehat{\mathsf{v}}_\lambda \leq \mathsf{b}_\lambda + 8 ~\sqrt{ \frac{\mathsf{df}_\lambda \vee (\mathsf{Q}^\star)^2~\log \frac{2}{\delta}}{n}}.\]

Moreover,

\[\|x_\lambda^{\star} - x^\star\|_{\mathbf{H}_\lambda(x^\star)} \leq \frac{1}{\underline{\phi}(\mathsf{t}(x_\lambda^{\star} - x^\star))} \|\nabla f_\lambda(x^\star)\|_{\mathbf{H}^{-1}_\lambda(x^\star)} \leq 2 \mathsf{b}_\lambda.\]
Hence: 
\[\|\widehat{x}_\lambda^\star - x^\star\| \leq 3~\lambda^{-1/2}~\mathsf{b}_\lambda + 8~\lambda^{-1/2} ~\sqrt{ \frac{\mathsf{df}_\lambda \vee (\mathsf{Q}^\star)^2~\log \frac{2}{\delta}}{n}}.\]
\end{proof}


\paragraph{2. Final bound for the projected ERM approximation}

In this paragraph, denote with $\mathsf{C}_{\Pj}(x,\lambda)$ the quantity $\frac{\|\mathbf{H}^{1/2}(x)(\mathbf{I} -\mathbf{P})\|^2}{\lambda}$ and $\widehat{\mathsf{C}}_{\Pj}(x,\lambda)$ the quantity $\frac{\|\widehat{\mathbf{H}}^{1/2}(x)(\mathbf{I} -\mathbf{P})\|^2}{\lambda}$


\begin{lemma} \label{lm:technical_before_proj} 
Let $n \in \mathbb{N}$, $\delta \in (0,1/2]$, $0 < \lambda \leq \mathsf{B}_2^\star$. Whenever
\[n \geq \triangle_1 \frac{\mathsf{B}_2^\star}{\lambda}\log \frac{8 \square_1^2\mathsf{B}_2^\star}{\lambda \delta},\qquad  \mathsf{C}_1 \sqrt{ \frac{\mathsf{df}_\lambda \vee (\mathsf{Q}^\star)^2}{n} ~\log \frac{2}{\delta}} \leq \mathsf{r}_\lambda(x^\star), \qquad \mathsf{C}_1 \mathsf{b}_\lambda \leq \mathsf{r}_\lambda(x^\star),\]
if
\[\mathsf{C}_{\Pj}(x^\star,\lambda) \leq \frac{\sqrt{2}}{480},~ \widehat{\nu}_{\mathbf{P},\lambda}(x) \leq \frac{\widehat{\mathsf{r}}_{\mathbf{P},\lambda}(x)}{2} ~\wedge~\frac{\mathsf{r}_\lambda(x^\star)}{126},\]

the following holds, with probability at least $1 - 2 \delta$.

\[\widehat{\nu}_\lambda(x) \leq \frac{\widehat{\mathsf{r}}_{\lambda}(x)}{2},~ \widehat{\nu}_\lambda(x) \leq \frac{\mathsf{r}_\lambda(x^\star)}{2}.\]
Here, $\mathsf{C}_1 = 1008$.
\end{lemma}

\begin{proof}

Proceed in the following way.
\paragraph{1. } It is easy to see that the conditions of this lemma imply the conditions of \cref{thm:general-result}. Hence, as in the previous proofs, the following hold:

\begin{itemize}
    \item $\|\mathbf{H}^{1/2}_\lambda(x^\star_\lambda)\widehat{\mathbf{H}}^{-1/2}_\lambda(x^\star_\lambda)\| \leq \sqrt{2}$;
    \item $\nm{x^\star - x^\star_\lambda} \leq \log 2$;
    \item $\nm{\widehat{x}_\lambda^\star - x^\star_\lambda} \leq \log 2$.
\end{itemize}

\paragraph{2. } Let us now apply \cref{prp:link_proj} to $\widehat{f}$. 
If 
\[\frac{\widehat{s}_\lambda}{\widehat{\mathsf{r}}_{\lambda}(\widehat{x}_\lambda^\star)} \leq \frac{1}{4},~ \widehat{\mathsf{C}}_{\Pj}(\widehat{x}_\lambda^\star,\lambda) \leq \frac{1}{120},~ \widehat{\nu}_{\mathbf{P},\lambda}(x) \leq \frac{\widehat{\mathsf{r}}_{\mathbf{P},\lambda}(x)}{2}, \]
Then it holds: 
\eqal{\label{eq:res_proj_lemma}\widehat{\nu}_\lambda(x) \leq 3(\widehat{\nu}_{\mathbf{P},\lambda}(x) + \widehat{s}_\lambda),\qquad \widehat{\mathsf{t}}(x-\widehat{x}_\lambda^\star) \leq \frac{3}{10} +2 \log 2.}
where the second bound is obtained in the proof of this proposition.
Now since 

\eqals{
\frac{1}{\widehat{\mathsf{r}}_{\lambda}(\widehat{x}_\lambda^\star)} &\leq e^{\widehat{\mathsf{t}}(\widehat{x}_\lambda^\star - x_\lambda^{\star})/2}\frac{1}{\widehat{\mathsf{r}}_\lambda(x_\lambda^{\star})} & \cref{eq:hl}\\
& \leq e^{\widehat{\mathsf{t}}(\widehat{x}_\lambda^\star - x_\lambda^{\star})/2}~\|\mathbf{H}^{1/2}_\lambda(x^\star_\lambda)\widehat{\mathbf{H}}^{-1/2}_\lambda(x^\star_\lambda)\|~\sup_{g \in \widehat{\mathcal{G}}}{\|g\|_{\mathbf{H}^{-1}_\lambda(x_\lambda^{\star})}} & \text{Def}\\
&\leq e^{\mathsf{t}(\widehat{x}_\lambda^\star - x_\lambda^{\star})/2}~\|\mathbf{H}^{1/2}_\lambda(x^\star_\lambda)\widehat{\mathbf{H}}^{-1/2}_\lambda(x^\star_\lambda)\|~\sup_{g \in \mathcal{G}}{\|g\|_{\mathbf{H}^{-1}_\lambda(x_\lambda^{\star})}} & \widehat{\mathcal{G}}\subset \mathcal{G}\\
& =  e^{\mathsf{t}(\widehat{x}_\lambda^\star - x_\lambda^{\star})/2}~\|\mathbf{H}^{1/2}_\lambda(x^\star_\lambda)\widehat{\mathbf{H}}^{-1/2}_\lambda(x^\star_\lambda)\|~\frac{1}{\mathsf{r}_\lambda(x_\lambda^{\star})}& \text{Def}\\
& \leq e^{(\mathsf{t}(\widehat{x}_\lambda^\star - x_\lambda^{\star}) + \mathsf{t}(x_\lambda^{\star} - x^\star))/2}~\|\mathbf{H}^{1/2}_\lambda(x^\star_\lambda)\widehat{\mathbf{H}}^{-1/2}_\lambda(x^\star_\lambda)\|~\frac{1}{\mathsf{r}_\lambda(x^\star)} & \cref{eq:hl} \\
& \leq \frac{2\sqrt{2}}{\mathsf{r}_\lambda(x^\star)}. & \text{previous bounds}
}
In a similar way, we get $\widehat{\mathsf{C}}_{\Pj}(\widehat{x}_\lambda^\star,\lambda) \leq 2 \sqrt{2} \mathsf{C}_{\Pj}(x^\star,\lambda)$. Thus, the conditions above are satisfied if the following conditions are satisfied:
\[\frac{\widehat{s}_\lambda}{\mathsf{r}_\lambda(x^\star)} \leq \frac{\sqrt{2}}{16},~ \mathsf{C}_{\Pj}(x^\star,\lambda) \leq \frac{\sqrt{2}}{480},~ \widehat{\nu}_{\mathbf{P},\lambda}(x) \leq \frac{\widehat{\mathsf{r}}_{\mathbf{P},\lambda}(x)}{2}.\]
Finally, note that under these conditions,
\eqal{\label{eq:techh}\frac{1}{\widehat{\mathsf{r}}_{\lambda}(x)} \leq \frac{e^{\widehat{\mathsf{t}}(x - \widehat{x}_\lambda^\star)/2}}{\widehat{\mathsf{r}}_{\lambda}(x)} \leq \frac{7}{\mathsf{r}_\lambda(x^\star)}.}
using the previous bound and the bound on $\widehat{\mathsf{t}}(x -\widehat{x}_\lambda^\star)$. 
\paragraph{3.} Let us assume 
\[\frac{\widehat{s}_\lambda}{\mathsf{r}_\lambda(x^\star)} \leq \frac{\sqrt{2}}{16},~ \mathsf{C}_{\Pj}(x^\star,\lambda) \leq \frac{\sqrt{2}}{480},~ \widehat{\nu}_{\mathbf{P},\lambda}(x) \leq \frac{\widehat{\mathsf{r}}_{\mathbf{P},\lambda}(x)}{2}.\]
According to \cref{eq:techh}, and to \cref{eq:res_proj_lemma}, if
\[\widehat{\nu}_{\mathbf{P},\lambda}(x) + \widehat{s}_\lambda \leq \frac{\mathsf{r}_\lambda(x^\star)}{42},\]
then it holds
\[\widehat{\nu}_\lambda(x) \leq \frac{\widehat{\mathsf{r}}_{\lambda}(x)}{2},~ \widehat{\nu}_\lambda(x) \leq \frac{\mathsf{r}_\lambda(x^\star)}{2}.\]
We simplify this condition as: 
\[\widehat{\nu}_{\mathbf{P},\lambda}(x) \leq \frac{\mathsf{r}_\lambda(x^\star)}{126},\qquad \widehat{s}_\lambda \leq \frac{2 \mathsf{r}_\lambda(x^\star)}{126}.\]
\paragraph{4.}
Now using the fact that under the conditions of this lemma, those of \cref{lm:emp_source} are satisfied:  
\[\widehat{s}_\lambda \leq 8~ \mathsf{b}_\lambda + 80~ \sqrt{\frac{\mathsf{df}_\lambda \vee (\mathsf{Q}^\star)^2~\log \frac{2}{\delta}}{n}}.\]
Thus, $\widehat{s}_\lambda \leq \frac{2 \mathsf{r}_\lambda(x^\star)}{126}$ holds, provided
\[\mathsf{b}_\lambda \leq \frac{\mathsf{r}_\lambda(x^\star)}{\mathsf{C}_1},~ n \geq \mathsf{C}_1^2 \frac{\mathsf{df}_\lambda \vee (\mathsf{Q}^\star)^2~\log \frac{2}{\delta}}{\mathsf{r}_\lambda(x^\star)^2} ,\]
where $\mathsf{C}_1 = 1008 $.
\end{proof}


\begin{proposition}[Behavior of an approximation to the projected problem] \label{prp:proj_pb_stat_gen}
Let $n \in \mathbb{N}$, $\delta \in (0,1/2]$, $0 < \lambda \leq \mathsf{B}_2^\star$. Let $x \in {\mathcal{H}}_\mathbf{P}$. Whenever
\[n \geq \triangle_1 \frac{\mathsf{B}_2^\star}{\lambda}\log \frac{8 \square_1^2\mathsf{B}_2^\star}{\lambda \delta},\qquad  \mathsf{C}_1 \sqrt{ \frac{\mathsf{df}_\lambda \vee (\mathsf{Q}^\star)^2}{n} ~\log \frac{2}{\delta}} \leq \mathsf{r}_\lambda(x^\star), \qquad \mathsf{C}_1 \mathsf{b}_\lambda \leq \mathsf{r}_\lambda(x^\star),\]
if
\[\mathsf{C}_{\Pj}(x^\star,\lambda) \leq \frac{\sqrt{2}}{480},~ \widehat{\nu}_{\mathbf{P},\lambda}(x) \leq \frac{\widehat{\mathsf{r}}_{\mathbf{P},\lambda}(x)}{2} ~\wedge~\frac{\mathsf{r}_\lambda(x^\star)}{126}. \]

The following holds, with probability at least $1 - 2 \delta$.

\[f(x) - f(x^\star) \leq \mathsf{K}_1~ \mathsf{b}_\lambda^2 + \mathsf{K}_2~ \frac{\mathsf{df}_\lambda \vee (\mathsf{Q}^\star)^2}{n}~\log \frac{2}{\delta} + \mathsf{K}_3~ \widehat{\nu}^2_{\mathbf{P},\lambda}(x), \]

where $\mathsf{K}_1 \leq 6.0\mathrm{e}{4}$, $\mathsf{K}_2 \leq 6.0\mathrm{e}{6} $ and $\mathsf{K}_3 \leq 810$, $\mathsf{C}_1$ are defined in \cref{lm:technical_before_proj}, and the other constants are defined in \cref{thm:general-result}. 
\end{proposition} 

\begin{remark}[Constants]
In this result, absolutely huge constants are obtained. They are (of course) totally sub-optimal. Indeed, this analysis has been simplified by dividing the bound into blocks: error of the empirical risk minimization with regularization, error of the projection compared to this empirical risk minimizer. Going back and forth from empirical to statistical, from projected to non projected induces exponential explosion of the constants. There is a way of doing the analysis directly by projecting the statistical problem. However, in order to relate to our previous work \cite{marteau2019} and avoid re-doing all of our work we discarded this. If we were to perform this more direct analysis, we could keep the constants to a reasonable level, of order $10^2$. 

\end{remark}

\begin{proof} 
We apply \cref{lm:approx_emp_pb}, using the previous lemma to guarantee the conditions.
\paragraph{1.} Under the conditions of this proposition, applying \cref{lm:technical_before_proj}, the conditions of \cref{lm:approx_emp_pb} are satisfied. Thus, 
\[f(x) - f(x^\star) \leq 14(f(\widehat{x}_\lambda^\star) - f(x^\star)) + 30\widehat{\nu}_\lambda(x)^2.\]
Moreover, from the previous proof, 
\[\widehat{\nu}_\lambda(x) \leq 3(\widehat{\nu}_{\mathbf{P},\lambda}(x) + \widehat{s}_\lambda),\]
and seeing as \cref{lm:emp_source} is satisfied,
\[\widehat{s}_\lambda \leq 8~ \mathsf{b}_\lambda + 80~ \sqrt{\frac{\mathsf{df}_\lambda \vee (\mathsf{Q}^\star)^2~\log \frac{2}{\delta}}{n}}.\]
This therefore yields:
\[\widehat{\nu}_\lambda(x)^2 \leq 27 \widehat{\nu}_{\mathbf{P},\lambda}(x)^2 + 1726 \mathsf{b}_\lambda^2 + 172600 \frac{\mathsf{df}_\lambda \vee (\mathsf{Q}^\star)^2~\log \frac{2}{\delta}}{n}.\]
\paragraph{2.} Moreover, from \cref{thm:general-result}, it holds:
\[f(\widehat{x}_\lambda^\star) - f(x^\star) \leq 414~\mathsf{b}_\lambda^2 + 414~\frac{\mathsf{df}_\lambda \vee (\mathsf{Q}^\star)^2}{n}~\log \frac{2}{\delta}.\]
\paragraph{3. }
Putting things together:
\[f(x) - f(x^\star) \leq \mathsf{K}_1~ \mathsf{b}_\lambda^2 + \mathsf{K}_2~ \frac{\mathsf{df}_\lambda \vee (\mathsf{Q}^\star)^2}{n}~\log \frac{2}{\delta} + \mathsf{K}_3~ \widehat{\nu}^2_{\mathbf{P},\lambda}(x).\]
We bound the constants in the theorem.

\end{proof}


\begin{lemma}

Under the conditions of the previous theorem, the following hold: 
\begin{itemize}
    \item $\frac{1}{\widehat{\mathsf{r}}_{\mathbf{P},\lambda}(x)} \leq \frac{8}{\mathsf{r}_\lambda(x^\star)}$;
    \item $\lambda^{1/2}\|x - x^\star\| \leq 7 \widehat{\nu}_{\mathbf{P},\lambda}(x) + 59 \mathsf{b}_\lambda + 568\sqrt{\frac{\mathsf{df}_\lambda \vee (\mathsf{Q}^\star)^2~\log \frac{2}{\delta}}{n}}$;
    \item $\lambda \|x\|_{\widehat{\mathbf{H}}_{\mathbf{P},\lambda}^{-1}(x)}  \leq 7 \widehat{\nu}_{\mathbf{P},\lambda}(x) + 72 \mathsf{b}_\lambda + 720\sqrt{\frac{\mathsf{df}_\lambda \vee (\mathsf{Q}^\star)^2~\log \frac{2}{\delta}}{n}}$.
\end{itemize}
In particular,
$\frac{\lambda \|x\|_{\widehat{\mathbf{H}}_{\mathbf{P},\lambda}^{-1}(x)}}{\widehat{\mathsf{r}}_{\mathbf{P},\lambda}(x)} \leq 11$.
\end{lemma} 

\begin{proof} Let us prove the three statements. 

\paragraph{1.} Write $\frac{1}{\widehat{\mathsf{r}}_{\mathbf{P},\lambda}(x)} = \sup_{g \in \widehat{\mathcal{G}}}{\|\mathbf{P} g\|_{\widehat{\mathbf{H}}^{-1}_{\mathbf{P},\lambda}(x)}}$. Now
\[\sup_{g \in \widehat{\mathcal{G}}}{\|\mathbf{P} g\|_{\widehat{\mathbf{H}}^{-1}_{\mathbf{P},\lambda}(x)}} \leq \sup_{g \in \widehat{\mathcal{G}}}{\| g\|_{\widehat{\mathbf{H}}^{-1}_{\lambda}(x)}} \leq e^{\widehat{\mathsf{t}}(x - \widehat{x}_\lambda^\star)/2}\sup_{g \in \widehat{\mathcal{G}}}{\| g\|_{\widehat{\mathbf{H}}^{-1}_{\lambda}(\widehat{x}_\lambda^\star)}}.\]

Now bound

\[\sup_{g \in \widehat{\mathcal{G}}}{\| g\|_{\widehat{\mathbf{H}}^{-1}_{\lambda}(\widehat{x}_\lambda^\star)}} \leq e^{\widehat{\mathsf{t}}(x_\lambda^{\star} - \widehat{x}_\lambda^\star)/2}\sup_{g \in \widehat{\mathcal{G}}}{\| g\|_{\widehat{\mathbf{H}}^{-1}_{\lambda}(x_\lambda^{\star})}} \leq e^{\widehat{\mathsf{t}}(x_\lambda^{\star} - \widehat{x}_\lambda^\star)/2}~\|\mathbf{H}^{1/2}_\lambda(x_\lambda^{\star})\widehat{\mathbf{H}}^{-1/2}_\lambda(x_\lambda^{\star})\|~\sup_{g \in \widehat{\mathcal{G}}}{\| g\|_{\mathbf{H}^{-1}_{\lambda}(x_\lambda^{\star})}}. \]

Finally bound 
\[\sup_{g \in \widehat{\mathcal{G}}}{\| g\|_{\mathbf{H}^{-1}_{\lambda}(x_\lambda^{\star})}} \leq e^{\mathsf{t}(x^\star - x_\lambda^{\star})/2} \frac{1}{\mathsf{r}_\lambda(x^\star)}.\]

Now using the fact that under the previous assumptions $\mathsf{t}(x^\star - x_\lambda^{\star} ), \mathsf{t}(x_\lambda^{\star} - \widehat{x}_\lambda^\star) \leq \log 2$, $\widehat{\mathsf{t}}(x - \widehat{x}_\lambda^\star) \leq \frac{3}{10} + 2 \log 2$ and $\|\mathbf{H}^{1/2}_\lambda(x_\lambda^{\star})\widehat{\mathbf{H}}^{-1/2}_\lambda(x_\lambda^{\star})\| \leq \sqrt{2}$, we get the first equation.

\paragraph{2. } In order to bound $\lambda^{1/2}\|x - x^\star\|$, decompose
\[\lambda^{1/2}\|x - x^\star\| \leq \lambda^{1/2}\|x - \widehat{x}_\lambda^\star\| + \lambda^{1/2}\|\widehat{x}_\lambda^\star - x^\star\|.\]
Now use \cref{prp:link_proj} to bound
$  \lambda^{1/2}\|x - \widehat{x}_\lambda^\star\| \leq 7 (\widehat{\nu}_{\mathbf{P},\lambda}(x) + \widehat{s}_\lambda) $. Using \cref{lm:emp_source}, under the conditions above, 

\[\widehat{s}_\lambda \leq 8~ \mathsf{b}_\lambda + 80~ \sqrt{\frac{\mathsf{df}_\lambda \vee (\mathsf{Q}^\star)^2~\log \frac{2}{\delta}}{n}}.\]

Hence

\[ \lambda^{1/2}\|x - \widehat{x}_\lambda^\star\| \leq 7 \widehat{\nu}_{\mathbf{P},\lambda}(x) + 56 \mathsf{b}_\lambda + 560\sqrt{\frac{\mathsf{df}_\lambda \vee (\mathsf{Q}^\star)^2~\log \frac{2}{\delta}}{n}}. \]
Moreover, using again \cref{lm:emp_source}

\[\lambda^{1/2}\|\widehat{x}_\lambda^\star - x^\star\| \leq 3~\mathsf{b}_\lambda + 8~ ~\sqrt{ \frac{\mathsf{df}_\lambda \vee (\mathsf{Q}^\star)^2~\log \frac{2}{\delta}}{n}}.\]
Combining these two inequalities, we get:

\[ \lambda^{1/2}\|x - x^\star\| \leq 7 \widehat{\nu}_{\mathbf{P},\lambda}(x) + 59 \mathsf{b}_\lambda + 568\sqrt{\frac{\mathsf{df}_\lambda \vee (\mathsf{Q}^\star)^2~\log \frac{2}{\delta}}{n}}.\]

\paragraph{3. } In order to bound $\lambda \|x\|_{\widehat{\mathbf{H}}_{\mathbf{P},\lambda}^{-1}(x)}$, use \cref{prp:link_proj} to get  $\lambda \|x\|_{\widehat{\mathbf{H}}^{-1}_{\mathbf{P},\lambda}(x)} \leq 7 \widehat{\nu}_{\mathbf{P},\lambda}(x) + 9 \widehat{s}_\lambda$.

Now using \cref{lm:emp_source}, the following bound holds:

\[ \lambda \|x\|_{\widehat{\mathbf{H}}_{\mathbf{P},\lambda}^{-1}(x)}  \leq 7 \widehat{\nu}_{\mathbf{P},\lambda}(x) + 72 \mathsf{b}_\lambda + 720\sqrt{\frac{\mathsf{df}_\lambda \vee (\mathsf{Q}^\star)^2~\log \frac{2}{\delta}}{n}}.\]


\end{proof} 



\begin{proposition}[Simplification] 

\label{prp:proj_pb_stat_gen_part}
Let $n \in \mathbb{N}$, $\delta \in (0,1/2]$, $0 < \lambda \leq \mathsf{B}_2^\star$. Let $x \in {\mathcal{H}}_\mathbf{P}$. Whenever
\[n \geq \triangle_1 \frac{\mathsf{B}_2^\star}{\lambda}\log \frac{8 \square_1^2\mathsf{B}_2^\star}{\lambda \delta},\qquad  \mathsf{C}_1 \sqrt{ \frac{\mathsf{df}_\lambda \vee (\mathsf{Q}^\star)^2}{n} ~\log \frac{2}{\delta}} \leq \frac{\sqrt{\lambda}}{R}, \qquad \mathsf{C}_1 \mathsf{b}_\lambda \leq \frac{\sqrt{\lambda}}{R},\]
if
\[\mathsf{C}_{\Pj}(x^\star,\lambda) \leq \frac{\sqrt{2}}{480},~ \widehat{\nu}_{\mathbf{P},\lambda}(x) \leq \frac{\sqrt{\lambda}}{126R},\]

then the following holds, with probability at least $1 - 2 \delta$.

\[f(x) - f(x^\star) \leq \mathsf{K}_1~ \mathsf{b}_\lambda^2 + \mathsf{K}_2~ \frac{\mathsf{df}_\lambda \vee (\mathsf{Q}^\star)^2}{n}~\log \frac{2}{\delta} + \mathsf{K}_3~ \widehat{\nu}^2_{\mathbf{P},\lambda}(x), \]

where $\mathsf{K}_1 \leq 6.0\mathrm{e}{4}$, $\mathsf{K}_2 \leq 6.0\mathrm{e}{6} $ and $\mathsf{K}_3 \leq 810$, $\mathsf{C}_1$ are defined in \cref{lm:technical_before_proj}, and the other constants are defined in \cref{thm:general-result}. 

Moreover, in that case, $R \|x - x^\star\| \leq 10$. 

\end{proposition} 









\subsection{Optimal choice of $\lambda$, specific source conditions \label{app:quantibounds}}



In this part, we continue to assume \cref{asm:gen_sc,asm:asmtech,asm:iid,asm:minimizer}. We present a classification of distributions $\rho$ and show that we can achieve better rates than the classical slow rates, as presented in Appendix F of \cite{marteau2019}. 

\subsubsection{Classification of distributions and statistical bounds for the ERM}

We use the following classification for distributions.

\begin{definition}[class of distributions]
Let $\alpha \in [1,+\infty]$ and $r \in [0,1/2]$. \\\
We denote with $ \mathcal{P}_{\alpha,r}$ the set of probability distributions $\rho$ such that there exists $\mathsf{L},\mathsf{Q} \geq 0$, 
\begin{itemize}
    \item $\mathsf{b}_\lambda \leq \mathsf{L} ~\lambda^{\frac{1 + 2r}{2}}$;
    \item $\mathsf{df}_\lambda \leq \mathsf{Q}^2~ \lambda^{-1/\alpha}$;
\end{itemize}
where this holds for any $0<\lambda \leq 1$. 
For simplicity, if $\alpha = +\infty$, we assume that $\mathsf{Q} \geq \mathsf{Q}^\star$.
\end{definition}

 Note that given our assumptions, we always have 

\eqal{\label{eq:basic_class}\rho \in \mathcal{P}_{1,0},~~~  \mathsf{L} = \|x^\star\|,~\mathsf{Q} = \mathsf{B}_1^\star.}

We also define 
\eqal{ \label{eq:lambda1}\lambda_1 = \left(\frac{\mathsf{Q}}{\mathsf{Q}^\star}\right)^{2\alpha} \wedge 1 , }
such that 
$$ \forall \lambda \leq \lambda_1,~ \mathsf{df}_\lambda \vee (\mathsf{Q}^\star)^2 \leq \frac{\mathsf{Q}^2}{\lambda^{1/\alpha}} .$$


\paragraph{Interpretation of the classes}

\begin{itemize}
    \item The bias term $\mathsf{b}_\lambda$ characterizes the regularity of the objective $x^\star$. In a sense, if $r$ is big, then this means $x^\star$ is very regular and will be easier to estimate. The following results reformulates this intuition.
   
    
\begin{remark}[source condition]\label[remark]{rmk:source_cond}
Assume there exists $0 \leq r \leq 1/2$ and $v \in {\mathcal{H}}$ such that 
\[\mathbf{P}_{\mathbf{H}(x^\star)}x^\star = \mathbf{H}(x^\star)^r v.\]
Then it holds:  
\[\forall \lambda >0,~\mathsf{b}_\lambda \leq \mathsf{L}~\lambda^{\frac{1 + 2r}{2}},~~~~\mathsf{L} = \|\mathbf{H}(x^\star)^{-r}x^\star\|.\]
\end{remark}

 \item The effective dimension $\mathsf{df}_\lambda$ characterizes the size of the space ${\mathcal{H}}$ with respect to the problem. The higher $\alpha$, the smaller the space. If ${\mathcal{H}}$ is finite dimensional for instance, $\alpha = +\infty$.
 
 \end{itemize}
 
 In this section, for any given pair $(\alpha,r)$ characterizing the regularity and size of the problem, we associate 
\[\beta = \frac{1}{1 + 2r + 1/\alpha},~~~~\gamma = \frac{\alpha(1+2r)}{\alpha(1+2r) + 1}.\]
 
 
 In \cite{marteau2019} (see corollary 3), explicit bounds are given for the performance of the regularized expected risk minimizer $\widehat{x}_\lambda^\star$ depending on which class $\rho$ belongs to, i.e., as a function of $\alpha,r$.
 
 \begin{corollary}\label[corollary]{cor:bounds_source_capacity}
Let $\delta \in (0,1/2]$. Under \cref{asm:gen_sc,asm:asmtech,asm:iid,asm:minimizer}, if $\rho \in \mathcal{P}_{\alpha,r}$ with $r>0$ , when $n \geq N$ and $\lambda = (C_0/n)^{\beta}$, then with probability at least $1-2\delta$,
\[f(\widehat{x}_\lambda^\star) - f(x^\star) \leq C_1 n^{-\gamma}\log \frac{2}{\delta},\]
with $C_0 = 256 (\mathsf{Q}/\mathsf{L})^2,~C_1 = 8 (256)^{\gamma}~(\mathsf{Q}^{\gamma} ~ 
    \mathsf{L}^{1-\gamma})^2$ and $N$ defined in \cite{marteau2019}, and satisfying
$N = O(\poly(\mathsf{B}_1^\star,\mathsf{B}_2^\star,\mathsf{L},\mathsf{Q},R,\log(1/\delta)))$. 
\end{corollary}
 




\subsubsection{Quantitative bounds for the projected problem}
In this part,  the aim is to show that if we approximately solve the projected problem up to a certain precision, then this approximation has the same statistical rates as the regularized ERM with the good choice of $\lambda$. For the sake of simplicity, we will assume that $r >0$.




In what follows, we define 

\eqal{\label{eq:Nvalue}N = \frac{ \mathsf{Q}^2}{\mathsf{L}^2}\left(\mathsf{B}_2^\star \wedge \lambda_0 \wedge \lambda_1\right)^{-1/\beta} ~~~\vee~~~\left(2.1e4 \frac{1}{1-\beta}  A \log  \left(1.4e6   \frac{1}{1-\beta}A^2 \frac{1}{\delta}\right)\right)^{1/(1-\beta)},}
where $A = \frac{\mathsf{B}_2^\star \mathsf{L}^{2\beta} }{ \mathsf{Q}^{2\beta} }$, $\lambda_0 = (\mathsf{C}_1 \mathsf{L} R \log \frac{2}{\delta})^{-1/r} \wedge 1$ and $\lambda_1 = \frac{\mathsf{Q}^{2\alpha}}{(\mathsf{Q}^\star)^{2\alpha}}$. 


\begin{theorem}[Quantitative result with source $r > 0$]\label[theorem]{thm:quantisource}
Let $\rho \in \mathcal{P}_{\alpha,r}$  and assume $r>0$. Let $\delta \in (0,\frac{1}{2}]$.\\\
Let $\mathbf{P}$ be an orthogonal projection, $x \in {\mathcal{H}}$.
If 
    \[n \geq N,~~~\lambda = \left( \left(\frac{\mathsf{Q}}{\mathsf{L}}\right)^{2}~\frac{1}{n}\right)^{\beta},~~~~\mathsf{C}_{\Pj}(x^\star,\lambda) \leq \frac{\sqrt{2}}{480},~ \widehat{\nu}_{\mathbf{P},\lambda}(x) \leq \mathsf{Q}^{\gamma} ~ 
    \mathsf{L}^{1-\gamma} n^{-\gamma/2}\]
    then with probability at least $1-2 \delta$,
    \[f(x) - f(x^\star) \leq \mathsf{K} \left(\mathsf{Q}^{\gamma} ~ 
    \mathsf{L}^{1-\gamma}\right)^2 ~~\frac{1}{n^{\gamma}} \log \frac{2}{\delta},\]
    where $N$ is defined in \cref{eq:Nvalue} and $\mathsf{K} \leq 7.0e6$. Moreover, $R \|x - x^\star\| \leq 10$.

\end{theorem}

\begin{proof} 

Using the definition of $\lambda_1$, as soon as $\lambda \leq \lambda_1$ ,it holds:  $\mathsf{df}_\lambda \vee (\mathsf{Q}^\star)^2 \leq \mathsf{Q}^2 \lambda^{-1/\alpha}$.\\\

Let us formulate  \cref{prp:proj_pb_stat_gen_part} using the fact that $\rho \in \mathcal{P}_{\alpha,r}$.\\\

Let $n \in \mathbb{N}$, $\delta \in (0,1/2]$, $0 < \lambda \leq \mathsf{B}_2^\star$, $x \in {\mathcal{H}}_\mathbf{P}$. Whenever
\[n \geq \triangle_1 \frac{\mathsf{B}_2^\star}{\lambda}\log \frac{8 \square_1^2\mathsf{B}_2^\star}{\lambda \delta},~~~\mathsf{C}_1 \sqrt{ \frac{\mathsf{Q}^2}{\lambda^{1/\alpha}n} ~\log \frac{2}{\delta}}\leq \frac{\lambda^{1/2}}{R}, \mathsf{C}_1 ~\mathsf{L} \lambda^{1/2 + r} \leq \frac{\lambda^{1/2}}{R},\]
if
\[\mathsf{C}_{\Pj}(x^\star,\lambda) \leq \frac{\sqrt{2}}{480},~ \widehat{\nu}_{\mathbf{P},\lambda}(x) \leq \mathsf{L} \lambda^{1/2  + r}, \]

The following holds, with probability at least $1 - 2 \delta$.

\[f(x) - f(x^\star) \leq (\mathsf{K}_1 + \mathsf{K}_3) \mathsf{L}^2 \lambda^{1 + 2r} + \mathsf{K}_2~ \frac{\mathsf{Q}^2}{\lambda^{1/\alpha} n}~\log \frac{2}{\delta}, \qquad R \|x - x^\star\|\leq 10,\]

where all constants are defined in \cref{prp:proj_pb_stat_gen_part}.


\paragraph{Assume that $r > 0$}.
Define 
\[\lambda_0 = (\mathsf{C}_1 \mathsf{L} R \log \frac{2}{\delta})^{-1/r} \wedge 1.\]
Then for any $\lambda \leq \lambda_0$:
\[ \mathsf{L} \lambda^{1/2 + r} \leq \frac{1}{\mathsf{C}_1}\frac{\sqrt{\lambda}}{R}.\]

\noindent{\normalfont \bfseries 1)} First, we find a simple condition to guarantee 
$$\mathsf{r}_\lambda(x^\star)^2\lambda^{ 1/\alpha} \geq \mathsf{C}_2~  \mathsf{Q}^2
\frac{1}{n} ~\log \frac{2}{\delta}.$$

We see that if $\lambda \leq  \lambda_0$, then $\mathsf{r}_\lambda \geq \mathsf{C}_1 \mathsf{L} \lambda^{1/2 + r} \log \frac{2}{\delta}$. Hence, this condition is satisfied if 
$$ \lambda \leq \lambda_0,~~~~\mathsf{C}_1^2 \mathsf{L}^2 \lambda^{1+ 2r + 1/\alpha} \geq \mathsf{C}_2~  \mathsf{Q}^2
\frac{1}{n} .$$ 
Using the fact that $\mathsf{C}_2 = \mathsf{C}_1^2$, we reformulate:
$$ \lambda \leq \lambda_0,~~~~\mathsf{L}^2 \lambda^{1+ 2r + 1/\alpha} \geq   \mathsf{Q}^2
\frac{1}{n} .$$ 

\noindent{\normalfont \bfseries 2) } Now fix  
$$\lambda^{1 + 2r + 1/\alpha} =\frac{\mathsf{Q}^2}{ \mathsf{L}^2} ~\frac{1}{n} \Longleftrightarrow \lambda = \left(\frac{\mathsf{Q}^2}{ \mathsf{L}^2} ~\frac{1}{n}\right)^{\beta} .$$
where $\beta = 1/(1 + 2r + 1/\lambda) \in [1/2,1)$.


Using our restatement of \cref{prp:proj_pb_stat_gen}, with probability at least $1-2 \delta$,
$$\L(x) - \L(x^\star) \leq 
  \left(\mathsf{K}_1 + \mathsf{K}_3 + \mathsf{K}_2 \log \frac{2}{\delta}\right)~\mathsf{L}^2 \lambda^{1 + 2r} \leq    \mathsf{K}~\log \frac{2}{\delta}~\mathsf{L}^2 \lambda^{1 + 2r},$$
  where $\mathsf{K} = \mathsf{K}_1 + \mathsf{K}_3 + \mathsf{K}_2 \leq 7.0e6$ (see \cref{prp:proj_pb_stat_gen}). \\\
  This result holds provided
     \eqal{\label{eq:cond_refor}0 < \lambda \leq \mathsf{B}_2^\star \wedge \lambda_0 \wedge \lambda_1,~n \geq \triangle_1 \frac{\mathsf{B}_2^\star}{\lambda}\log \frac{8 \square_1^2\mathsf{B}_2^\star}{\lambda \delta}.}
     Indeed, it is shown in the previous point that the other conditions are satisfied.

 
  
\noindent{\normalfont \bfseries 3) }  Let us now work to guarantee the conditions in \cref{eq:cond_refor}.  \\\
First, to guarantee $n \geq \triangle_1 \frac{\mathsf{B}_2^\star}{\lambda}\log \frac{8 \square_1^2\mathsf{B}_2^\star}{\lambda \delta}$, bound
\[\frac{\mathsf{B}_2^\star}{\lambda} = \frac{\mathsf{B}_2^\star \mathsf{L}^{2\beta} n^{\beta}}{ \mathsf{Q}^{2\beta} ~\log^{\beta} \frac{2}{\delta}} \leq 2~\frac{\mathsf{B}_2^\star \mathsf{L}^{2\beta} }{ \mathsf{Q}^{2\beta} } n^{\beta}.\]

Then apply lemma 15 from \cite{marteau2019} with $a_1 = 2\triangle_1$, $a_2 = 16 \square_1^2$, $A = \frac{\mathsf{B}_2^\star \mathsf{L}^{2\beta} }{ \mathsf{Q}^{2\beta} }$. Since $\beta \geq 1/2$, using the bounds in \cref{thm:general-result}, we find $a_1 \leq 10400$ and $a_2 \leq 64,$ hence the following sufficient condition:

\[ n \geq  \left(2.1e4 \frac{1}{1-\beta}  A \log  \left(1.4e6   \frac{1}{1-\beta}A^2 \frac{1}{\delta}\right)\right)^{1/(1-\beta)}.\]
  
 Then, to guarantee the condition 
\[ \lambda \leq \mathsf{B}_2^\star \wedge \lambda_0 \wedge \lambda_1, \]
we simply need 
\[n \geq \frac{\mathsf{Q}^2}{\mathsf{L}^2}\left(\mathsf{B}_2^\star \wedge \lambda_0 \wedge \lambda_1\right)^{-1/\beta}.\]
Hence, defining 
$$N = \frac{ \mathsf{Q}^2}{\mathsf{L}^2}\left(\mathsf{B}_2^\star \wedge \lambda_0 \wedge \lambda_1\right)^{-1/\beta} ~~~\vee~~~\left(2.1e4 \frac{1}{1-\beta}  A \log  \left(1.4e6   \frac{1}{1-\beta}A^2 \frac{1}{\delta}\right)\right)^{1/(1-\beta)},$$ we see that as soon as $n \geq N$, \cref{eq:cond_refor} holds. 



     


\end{proof}



\newpage 

\section{Multiplicative approximations for Hermitian operators\label{sec:lemmas-operators}}


In this section, we put together useful tools for approximating linear operators and solving linear systems with regularization. 

In this section, $\mathbf{A}$ and $\mathbf{B}$ will always denote positive semi-definite Hermitian operators on a Hilbert space ${\mathcal{H}}$, and $\mathbf{P}$ will denote an orthogonal projection operator. Moreover, given a positive semi-definite operator $\mathbf{A}$, and $\lambda >0$, $\mathbf{A}_\lambda$ will stand for the regularized operator $\mathbf{A} + \lambda \mathbf{I}$. 


\begin{lemma}[Equivalence of Hermitian operators]\label{lm:equiv_operators}
Let $\mathbf{A}$ and $\mathbf{B}$ be two semi-definite Hermitian operators. Let $\lambda > 0$. Assume you have access to 
\[t:= \|\mathbf{A}_\lambda^{-1/2}(\mathbf{B} - \mathbf{A})\mathbf{A}^{-1/2}_\lambda\|.\]
It holds: 
\[\|\mathbf{A}_\lambda^{-1/2} \mathbf{B}_\lambda^{1/2}\|^2 \leq 1 + t \Leftrightarrow \mathbf{B}_\lambda \preceq (1+t)\mathbf{A}_\lambda.\]
Moreover, if $t<1$, 
\[\|\mathbf{B}_\lambda^{-1/2} \mathbf{A}_\lambda^{1/2}\|^2 \leq \frac{1}{1-t} \Leftrightarrow (1-t)\mathbf{A}_\lambda \preceq \mathbf{B}_\lambda .\]
\end{lemma}


\begin{proof} 
For the first point, simply note that: 
\[\|\mathbf{A}_\lambda^{-1/2}\mathbf{B}_\lambda^{1/2}\|^2 = \|\mathbf{A}_\lambda^{-1/2} \mathbf{B}_\lambda \mathbf{A}_\lambda^{-1/2}\| = \|\mathbf{I} +\mathbf{A}_\lambda^{-1/2} \left(\mathbf{B} - \mathbf{A}\right) \mathbf{A}_\lambda^{-1/2} \| \leq 1 + t.\]
For the second point,
\[\|\mathbf{B}_\lambda^{-1/2}\mathbf{A}_\lambda^{1/2}\|^2 = \|\left(\mathbf{A}_\lambda^{-1/2} \mathbf{B}_\lambda \mathbf{A}_\lambda^{-1/2}\right)^{-1}\| = \|\left(\mathbf{I} +\mathbf{A}_\lambda^{-1/2} \left(\mathbf{B} - \mathbf{A}\right) \mathbf{A}_\lambda^{-1/2} \right)^{-1}\| .\]
Moreover, we know that if $\|\mathbf{H}\| < 1$ with $\mathbf{H}$ a Hermitian operator, then $\|(\mathbf{I} + \mathbf{H})^{-1}\| \leq \frac{1}{1-\|\mathbf{H}\|}$. The result follows. 
\end{proof} 


We will now state a technical lemma which describes how combining approximation behaves. 

\begin{lemma}[Combination of approximations]\label{lm:comb_approx}
Let $N \geq 1$. Let $(\mathbf{A}_i)_{1 \leq i \leq N+1}$ be a sequence of positive semi-definite Hermitian operators. Define
\[t_i:= \|\mathbf{A}_{i,\lambda}^{-1/2}(\mathbf{A}_{i+1} - \mathbf{A}_i)\mathbf{A}_{i,\lambda}^{-1/2}\|.\]
For any $1 \leq i,j \leq N+1$, define 
$$t_{i:j}:= \|\mathbf{A}_{i,\lambda}^{-1/2}(\mathbf{A}_{j} - \mathbf{A}_i)\mathbf{A}_{i,\lambda}^{-1/2}\|.$$
In particular, $t_i = t_{i:i+1}$. Then the following holds:

\[\forall 1 \leq i \leq j \leq N,~1 + t_{i:j} \leq \prod_{k=i}^{j-1}(1 + t_k)\]

Moreover, if $t_i < 1$, then it holds: 
\[\|\mathbf{A}_{i+1,\lambda}^{-1/2}(\mathbf{A}_{i} - \mathbf{A}_{i+1})\mathbf{A}_{i+1,\lambda}^{-1/2}\| \leq \frac{t_i}{1-t_i}\]
Hence, in that case
\[\forall 1 \leq j \leq i \leq N,~ 1 + t_{j:i} \leq \prod_{k=i}^{j-1}{\frac{1}{1-t_k}} \]


\end{lemma} 

\begin{proof}
Let us prove everything for a sequence of three operators; the rest follows by induction. Let $\mathbf{A}_1,\mathbf{A}_2,\mathbf{A}_3$ be three positive semi-definite operators. 
\paragraph{1.} Bound
\eqals{t_{1:3} &= \|\mathbf{A}_{1,\lambda}^{-1/2}\left(\mathbf{A}_1 - \mathbf{A}_3\right)\mathbf{A}_{1,\lambda}^{-1/2}\| \\
&\leq \|\mathbf{A}_{1,\lambda}^{-1/2}\left(\mathbf{A}_1 - \mathbf{A}_2\right)\mathbf{A}_{1,\lambda}^{-1/2}\|  + \|\mathbf{A}_{1,\lambda}^{-1/2}\left(\mathbf{A}_2 - \mathbf{A}_3\right)\mathbf{A}_{1,\lambda}^{-1/2}\|\\
&\leq t_{1:2} + \|\mathbf{A}_{1,\lambda}^{-1/2}\mathbf{A}_{2,\lambda}^{1/2}\|^2 t_{2:3}\\
& \leq t_{1:2} + (1 + t_{1:2})t_{2:3}
.}
The last line comes from \cref{lm:equiv_operators}. Thus 
\[1 + t_{1:3} \leq 1 + t_{1:2} + t_{2:3} + t_{1:2} t_{2:3} = (1 + t_{1:2})(1 + t_{2:3}).\]
\paragraph{2.} Let us now bound $t_{2:1}$ knowing $t_{1:2}$. This will imply the rest of the lemma. 

Indeed, simply note that 
\[t_{2:1} = \|\mathbf{A}^{-1/2}_{2,\lambda}(\mathbf{A}_2 - \mathbf{A}_1)\mathbf{A}^{-1/2}_{2,\lambda} \| \leq \|\mathbf{A}^{-1/2}_{2,\lambda}\mathbf{A}^{1/2}_{1,\lambda}\|^2 ~t_{1:2} .\]
Using \cref{lm:equiv_operators}, if $t_{1:2} < 1$, $\|\mathbf{A}^{-1/2}_{2,\lambda}\mathbf{A}^{1/2}_{1,\lambda}\|^2 \leq \frac{1}{1-t_{1:2}}$, hence
\[t_{2:1} \leq \frac{t_{1:2}}{1-t_{1:2}}.\]
\end{proof} 






\begin{lemma}[Projection of Hermitian operators]\label{lm:proj_hermit}
For any Hermitian operator $\mathbf{A}$ and orthogonal projection $\mathbf{P}$, the following holds:
\[\|\mathbf{A}_\lambda^{-1/2}(\mathbf{A} - \mathbf{P} \mathbf{A} \mathbf{P} )\mathbf{A}_\lambda^{-1/2}\| \leq \left(1 + \frac{\|\mathbf{A}^{1/2}(\mathbf{I} - \mathbf{P})\|}{\sqrt{\lambda}}\right)^2 - 1.\]
In particular, 
\[\|\mathbf{A}^{-1/2}_\lambda \left(\mathbf{P} \mathbf{A} \mathbf{P} + \lambda \mathbf{I}\right)^{1/2}\| \leq 1 + \frac{\|\mathbf{A}^{1/2}(\mathbf{I} - \mathbf{P})\|}{\sqrt{\lambda}}.\]
Moreover, if 
\[\frac{\|\mathbf{A}^{1/2}(\mathbf{I} - \mathbf{P})\|}{\sqrt{\lambda}} < \sqrt{2}-1,\]
then it holds 
\[\|\mathbf{A}^{1/2}_\lambda \left(\mathbf{P} \mathbf{A} \mathbf{P} + \lambda \mathbf{I}\right)^{-1/2}\|^2 \leq \frac{1}{2 - \left(1 + \frac{\|\mathbf{A}^{1/2}(\mathbf{I} - \mathbf{P})\|}{\sqrt{\lambda}}\right)^2}.\]
We also always have:
\[\| \left(\mathbf{P} \mathbf{A} \mathbf{P} + \lambda \mathbf{I}\right)^{-1/2} \mathbf{P} \mathbf{A}^{1/2}_\lambda\|^2 \leq 1 .\]
\end{lemma}

\begin{proof} 
For any Hermitian operator $\mathbf{A}$, the following computation holds:
\eqals{\|\mathbf{A}_\lambda^{-1/2}(\mathbf{A} - \mathbf{P} \mathbf{A} \mathbf{P} )\mathbf{A}_\lambda^{-1/2}\| &=
\|\mathbf{A}_\lambda^{-1/2}(\mathbf{A} - (\mathbf{I} - (\mathbf{I} - \mathbf{P})) \mathbf{A} (\mathbf{I} - (\mathbf{I} - \mathbf{P} ))\mathbf{A}_\lambda^{-1/2}\|\\
& \leq 2\|\mathbf{A}_\lambda^{-1/2}(\mathbf{I}-\mathbf{P})\mathbf{A} \mathbf{A}_\lambda^{-1/2}\| + \|\mathbf{A}_\lambda^{-1/2}(\mathbf{I}-\mathbf{P})\mathbf{A} (\mathbf{I} - \mathbf{P})\mathbf{A}_\lambda^{-1/2}\|\\
& \leq \frac{2\|\mathbf{A}^{1/2}(\mathbf{I} - \mathbf{P})\|}{\sqrt{\lambda}} + \frac{\|\mathbf{A}^{1/2}(\mathbf{I}-\mathbf{P})\|^2}{\lambda}\\
& = \left(1 + \frac{\|\mathbf{A}^{1/2}(\mathbf{I} - \mathbf{P})\|}{\sqrt{\lambda}}\right)^2 - 1
.}
\end{proof}



\begin{lemma}[Relationship between approximations]\label{lm:relation_ship}
Let $\mathbf{A}$ and $\mathbf{B}$ be two positive semi-definite hermitian operators. Let $\lambda > 0$, $b \in {\mathcal{H}}$ and $\rho > 0$.  If
\[\|\mathbf{A}_\lambda^{-1/2}(\mathbf{B} - \mathbf{A})\mathbf{A}_\lambda^{-1/2}\| \leq \frac{1}{2} \wedge \frac{\rho}{4},\qquad \widetilde{\Delta} \in \lso(\mathbf{B}_\lambda,b,\rho/4) ,\]
then it holds:
$$\widetilde{\Delta} \in \lso(\mathbf{A}_\lambda,b,\rho).$$

\end{lemma}

\begin{proof}  Assume $\widetilde{\Delta} \in \lso(\mathbf{B}_\lambda,b,\tilde{\rho}/4)$ for a certain $\tilde{\rho}$. 
Decompose 
\eqals{
\|\mathbf{A}_\lambda^{-1}b - \widetilde{\Delta}\|_{\mathbf{A}_\lambda} &\leq \|\mathbf{A}_\lambda^{-1}b - \mathbf{B}_\lambda^{-1}b\|_{\mathbf{A}_\lambda}  + \|\mathbf{B}_\lambda^{-1}b - \widetilde{\Delta}\|_{\mathbf{A}_\lambda} \\
&\leq \|\mathbf{A}_\lambda^{1/2}(\mathbf{A}_\lambda^{-1} - \mathbf{B}_\lambda^{-1})\mathbf{A}_\lambda^{1/2}\|~ \|b\|_{\mathbf{A}_\lambda^{-1}} + \|\mathbf{A}_\lambda^{1/2}\mathbf{B}_\lambda^{-1/2}\| ~\|\mathbf{B}_\lambda^{-1}b - \widetilde{\Delta}\|_{\mathbf{B}_\lambda}.}

Now using the fact that $\mathbf{A}_\lambda^{-1} - \mathbf{B}_\lambda^{-1} = \mathbf{B}_\lambda^{-1}(\mathbf{B} - \mathbf{A})\mathbf{A}_\lambda^{-1}$, 

\begin{align*}\|\mathbf{A}_\lambda^{1/2}(\mathbf{A}_\lambda^{-1} - \mathbf{B}_\lambda^{-1})\mathbf{A}_\lambda^{1/2}\| &\leq \|\mathbf{A}_\lambda^{-1/2}(\mathbf{B}-\mathbf{A})\mathbf{A}_\lambda^{-1/2}\|~\|\mathbf{A}_\lambda^{1/2}\mathbf{B}_\lambda^{-1} \mathbf{A}_\lambda^{1/2}\| \\
&=  \|\mathbf{A}_\lambda^{-1/2}(\mathbf{B}-\mathbf{A})\mathbf{A}_\lambda^{-1/2}\|~\|\mathbf{A}_\lambda^{1/2}\mathbf{B}_\lambda^{-1/2} \|^2.\end{align*}
Moreover, 
\[\|\mathbf{B}_\lambda^{-1}b - \widetilde{\Delta}\|_{\mathbf{B}_\lambda} \leq \tilde{\rho} \|b\|_{\mathbf{B}_\lambda^{-1}} \leq \|\mathbf{A}^{1/2}\mathbf{B}^{-1/2}\|~\|b\|_{\mathbf{A}_\lambda^{-1}}.\]
Putting things together, and noting that from \cref{lm:equiv_operators}, $\|\mathbf{A}^{1/2}\mathbf{B}^{-1/2}\|^2 \leq \frac{1}{1-\|\mathbf{A}_\lambda^{-1/2}(\mathbf{B} - \mathbf{A})\mathbf{A}_\lambda^{-1/2}\|}$ as soon as $\|\mathbf{A}_\lambda^{-1/2}(\mathbf{B} - \mathbf{A})\mathbf{A}_\lambda^{-1/2}\| < 1$, it holds: 
$$\widetilde{\Delta} \in \lso(\mathbf{A}_\lambda,b,\rho),~\rho = \frac{\tilde{\rho} + \|\mathbf{A}_\lambda^{-1/2}(\mathbf{B} - \mathbf{A})\mathbf{A}_\lambda^{-1/2}\|}{1 - \|\mathbf{A}_\lambda^{-1/2}(\mathbf{B} - \mathbf{A})\mathbf{A}_\lambda^{-1/2}\|} .$$

Choosing the right values for $\tilde{\rho}$ and $\|\mathbf{A}_\lambda^{-1/2}(\mathbf{B} - \mathbf{A})\mathbf{A}_\lambda^{-1/2}\|$ yields the result.
\end{proof} 
\subsection{Results for Nystrom sub-sampling\label{app:sub_nystrom}}

Recall the notations from \cref{app:kernels}. 

We write without proof the following lemmas, which are just restatements of lemmas 9 and 10 of \cite{Rudi17}. 

\begin{lemma}[Uniform sampling]
Let $\delta >0$. If $\left\{\tilde{z}_1,...,\tilde{z}_m\right\}$ are sampled uniformly, then if $0 < \lambda < \|\mathbf{A}\|$, $m \leq n$ and 
\[m \geq \left(10 + 160 \mathcal{N}_{\infty}^\mathbf{A}(\lambda)\right)\log \frac{8 \|v\|_{L^{\infty}(Z)}^2}{\lambda \delta}.\]
Then it holds, with probability at least $1-\delta$:
\[\|\mathbf{A}_\lambda^{-1/2}(\widehat{\Ab} - \mathbf{A})\mathbf{A}_\lambda^{-1/2}\| \leq \frac{1}{2},\qquad\|\widehat{\Ab}_{m,\lambda}^{-1/2}(\widehat{\Ab} - \widehat{\Ab}_m)\widehat{\Ab}_{m,\lambda}^{-1/2}\| \leq \frac{1}{2} .\]
\end{lemma}


\begin{lemma}[Nystrom sampling]
Let $\delta >0$. If $\left\{\tilde{z}_1,...,\tilde{z}_m\right\}$ are sampled using $q$-approximate leverage scores for $t = \lambda$, then if $t_0 \vee \frac{19 \|v\|_{L^{\infty}(Z)}^2}{n}\log \frac{n}{2 \delta} < \lambda < \|\mathbf{A}\|$, and $n \geq 405 \|v\|_{L^{\infty}(Z)}^2 \vee 67 \|v\|_{L^{\infty}(Z)}^2 \log \frac{12 \|v\|_{L^{\infty}(Z)}^2}{\delta}$, if
\[m \geq \left(6 + 486 q^2 \mathcal{N}^\mathbf{A}(\lambda)\right)\log \frac{8 \|v\|_{L^{\infty}(Z)}^2}{\lambda \delta}.\]
Then it holds, with probability at least $1-\delta$:
\[\|\mathbf{A}_\lambda^{-1/2}(\widehat{\Ab} - \mathbf{A})\mathbf{A}_\lambda^{-1/2}\| \leq \frac{1}{2},\qquad\|\widehat{\Ab}_{m,\lambda}^{-1/2}(\widehat{\Ab} - \widehat{\Ab}_m)\widehat{\Ab}_{m,\lambda}^{-1/2}\| \leq \frac{1}{2} .\]
\end{lemma}


\begin{lemma} Let $\lambda >0$. 
Assume: 
\[\|\mathbf{A}_\lambda^{-1/2}(\widehat{\Ab} - \mathbf{A})\mathbf{A}_\lambda^{-1/2}\| \leq \frac{1}{2},\qquad\|\widehat{\Ab}_{m,\lambda}^{-1/2}(\widehat{\Ab} - \widehat{\Ab}_m)\widehat{\Ab}_{m,\lambda}^{-1/2}\| \leq \frac{1}{2} .\]
Denote with $P_m$ the projection on $\lspan(v_{\tilde{z}_j})_{1\leq j \leq m}$. Then the following holds:
\[\|\mathbf{A}^{1/2}_\lambda(\mathbf{I} - \mathbf{P}_m)\|^2 \leq 3 \lambda,\]
and for any partial isometry $V$,
\[\frac{1}{2}\left(V^* \widehat{\Ab}_m V + \lambda \mathbf{I} \right)\preceq  V^* \widehat{\Ab} V + \lambda \mathbf{I} \preceq \frac{3}{2}\left( V^* \widehat{\Ab}_m V + \lambda \mathbf{I}\right).\]

\end{lemma} 


\begin{proof} 
For the first point, use the well known fact that 
\[\mathbf{I} - \mathbf{P}_m \leq \lambda \widehat{\Ab}_{m,\lambda}^{-1},\]
since the range of $\mathbf{P}_m$ contains that of $\widehat{\Ab}_m$. 
Thus, 
\[\|\mathbf{A}_\lambda^{1/2}(\mathbf{I} - \mathbf{P}_m)\|^2 \leq \lambda \|\mathbf{A}_\lambda^{1/2} \widehat{\Ab}_{m,\lambda}^{-1/2}\|^2.\]
Now using \cref{lm:comb_approx},
\[\|\mathbf{A}_\lambda^{-1/2}(\widehat{\Ab} - \mathbf{A})\mathbf{A}_\lambda^{-1/2}\| \leq \frac{1}{2} \implies \|\widehat{\Ab}_\lambda^{-1/2}(\widehat{\Ab} - \mathbf{A})\widehat{\Ab}_\lambda^{-1/2}\| \leq 1.\]
Hence, again using \cref{lm:comb_approx}, 
\[\|\widehat{\Ab}_{m,\lambda}^{-1/2}(\widehat{\Ab}_m - \mathbf{A})\widehat{\Ab}_{m,\lambda}^{-1/2}\| \leq 2,\]
and therefore, using \cref{lm:equiv_operators},
\[\|\mathbf{A}_\lambda^{1/2} \widehat{\Ab}_{m,\lambda}^{-1/2}\|^2 \leq 3.\]
For the second point, this is only a consequence of \cref{lm:equiv_operators}. 
\end{proof} 
Now state two results which show that 

\begin{lemma}[Uniform sampling yielding $\rho$-approximation]\label{lm:sub_unif}
Let $0 < \rho \leq 1$ and $\delta >0$. Let $b \in {\mathcal{H}}$. If $\left\{\tilde{z}_1,...,\tilde{z}_m\right\}$ are sampled uniformly, $0 < \lambda < \|\mathbf{A}\|$, $m \leq n$ and 
\[m \geq \left(2 + \frac{48}{\rho} + \frac{5000}{\rho^2} \mathcal{N}_{\infty}^\mathbf{A}(\lambda)\right)\log \frac{8 \|v\|_{L^{\infty}(Z)}^2}{\lambda \delta}.\]
Then it holds, with probability at least $1-\delta$:
\[x \in \lso(\widehat{\Ab}_{m,\lambda},b,\rho/4) \implies x \in \lso(\mathbf{A}_\lambda,b,\rho)  .\]
In particular, with probability $1-\delta$, 
\[\widehat{\Ab}_{m,\lambda}^{-1}b \in \lso(\mathbf{A}_\lambda,b,\rho) . \]
\end{lemma}

\begin{proof} 
Apply Lemma 9 from \cite{Rudi17} with $\eta = \frac{\rho}{12} < \frac{1}{2}$. We find that under the conditions above, with probability at least $1-\delta$, 
\[\|\mathbf{A}_\lambda^{-1/2}(\widehat{\Ab} - \mathbf{A})\mathbf{A}_\lambda^{-1/2}\| \leq \eta ,\qquad\|\widehat{\Ab}_{m,\lambda}^{-1/2}(\widehat{\Ab} - \widehat{\Ab}_m)\widehat{\Ab}_{m,\lambda}^{-1/2}\| \leq \eta .\]
Now use \cref{lm:comb_approx} to see that
\[\|\mathbf{A}_\lambda^{-1/2}(\widehat{\Ab}_m - \mathbf{A})\mathbf{A}_\lambda^{-1/2}\| \leq (1 + \eta^2) - 1 \leq 3 \eta \leq \rho/4.\]
Thus, we can apply \cref{lm:relation_ship} to get the desired result. 
\end{proof} 


\begin{lemma}[Leverage scores Nystrom sampling yielding $\rho$-approximation]\label{lm:sub_ny}
Let $\delta >0$. If $\left\{\tilde{z}_1,...,\tilde{z}_m\right\}$ are sampled using $q$-approximate leverage scores for $t = \lambda$, then if $t_0 \vee \frac{19 \|v\|_{L^{\infty}(Z)}^2}{n}\log \frac{n}{2 \delta} < \lambda < \|\mathbf{A}\|$, and $n \geq 405 \|v\|_{L^{\infty}(Z)}^2 \vee 67 \|v\|_{L^{\infty}(Z)}^2 \log \frac{12 \|v\|_{L^{\infty}(Z)}^2}{\delta}$, if
\[m \geq \left(2 + \frac{24}{\rho} + \frac{13000 q^2}{\rho^2} \mathcal{N}^\mathbf{A}(\lambda)\right)\log \frac{8 \|v\|_{L^{\infty}(Z)}^2}{\lambda \delta}.\]
Then it holds, with probability at least $1-\delta$:
\[x \in \lso(\widehat{\Ab}_{m,\lambda},b,\rho/4) \implies x \in \lso(\mathbf{A}_\lambda,b,\rho)  .\]
In particular, with probability $1-\delta$, 
\[\widehat{\Ab}_{m,\lambda}^{-1}b \in \lso(\mathbf{A}_\lambda,b,\rho) . \]
\end{lemma}

\begin{proof} 
The proof is exactly the same as that of the previous lemma, using Lemma 10 instead of Lemma 9 in \cite{Rudi17}. 
\end{proof} 

\subsubsection*{\bibname}}

\usepackage[top=3cm, bottom=3cm, left=3cm, right=3cm]{geometry}

\usepackage{booktabs}
\usepackage{color}
\usepackage{latexsym} 
\usepackage{amsmath}              
\usepackage{amssymb}              
\usepackage{amsfonts}             
\usepackage{amsthm}               
\usepackage{tcolorbox}
\usepackage{times}
\usepackage[utf8]{inputenc}
\usepackage[T1]{fontenc}
\usepackage{enumitem}
\usepackage{xcolor}
\usepackage{thmtools}
\usepackage{thm-restate}
\usepackage[square,numbers]{natbib}
\usepackage[colorlinks=true,allcolors=blue]{hyperref}
\usepackage{cleveref}
\usepackage{url}           
\usepackage{booktabs}
\usepackage{nicefrac}      
\usepackage{microtype}     
\usepackage[american]{babel}
\usepackage{thmtools}
\usepackage{booktabs}
\usepackage{algorithm}
\usepackage{algorithmic}
\usepackage[tikz]{bclogo}
\usepackage{multicol,tabularx,capt-of}
\usepackage{hhline}
\usepackage{multirow}


\renewenvironment{proof}[1][\proofname]{{\bfseries #1.}}{}

\makeatother








\bibliographystyle{plainnat}






  
\input{macro_arxiv.tex}

\renewcommand{\leq}{\leqslant}
\renewcommand{\geq}{\geqslant}


\usepackage{authblk}
\renewcommand\Authands{, }
\author{\bfseries
Ulysse Marteau-Ferey}
\author{\bfseries
Francis Bach}
\author{\bfseries
Alessandro Rudi
}
\affil{
INRIA - D{\'e}partement d'Informatique de l'{\'E}cole Normale Sup{\'e}rieure \\
PSL Research University\\
Paris, France
}



\begin{document}

\title{Globally Convergent Newton Methods for Ill-conditioned Generalized Self-concordant Losses}





\date{}




\maketitle




\begin{abstract}
In this paper, we study large-scale convex optimization algorithms based on the Newton method applied to regularized generalized self-concordant losses, which include logistic regression and softmax regression.  We first prove that our new simple scheme based on a sequence of problems with decreasing regularization parameters is provably globally convergent, that this convergence is linear with a constant factor which scales only logarithmically with the condition number. In the parametric setting, we obtain an algorithm with the same scaling than regular first-order methods but with an improved behavior, in particular in ill-conditioned problems. Second, in the non-parametric machine learning setting, we provide an explicit algorithm combining  the previous scheme with Nystr\"om projection techniques, and prove that it achieves optimal generalization bounds with a time complexity of order $O(n\mathsf{df}_\lambda)$, a memory complexity of order $O(\mathsf{df}_\lambda^2)$ and {\em no dependence on the condition number}, generalizing the results known for least-squares regression. Here $n$ is the number of observations and $\mathsf{df}_\lambda$ is the associated degrees of freedom. In particular, this is the first large-scale algorithm to solve logistic and softmax regressions in the non-parametric setting with large condition numbers and theoretical guarantees.
\end{abstract}





\section{Introduction}
Minimization algorithms constitute a crucial algorithmic part of many machine learning methods, with   algorithms available for a variety of situations~\cite{bottou2018optimization}. In this paper, we focus on \emph{finite sum} problems of the form
\[\min_{x \in {\mathcal{H}}}{f_\lambda(x) = f(x) + \frac{\lambda}{2}\|x\|^2}, \mbox{ with }  f(x) = \frac{1}{n}\sum_{i=1}^n{f_i(x)},\] 
where ${\mathcal{H}}$ is a Euclidean or a Hilbert space, and each function is convex and smooth. The running-time of minimization algorithms classically depends on the number of functions $n$, the explicit (for Euclidean spaces) or implicit (for Hilbert spaces) dimension $d$ of the search space, and the condition number of the problem, which is upper bounded by   $\kappa = L/\lambda$, where $L$ characterizes the smoothness of the functions~$f_i$, and $\lambda$ the regularization parameter.
    
In the last few years, there has been a strong focus on problems with large $n$ and $d$, leading to \emph{first-order} (i.e., gradient-based) stochastic algorithms, culminating in a sequence of linearly convergent algorithms whose running time is favorable in $n$ and $d$, but scale at best in $\sqrt{\kappa}$~\cite{defazio2014saga,lin2015universal,defazio2016simple,Allen2017}.  
However, modern problems lead to objective functions with very large condition numbers, i.e., in many learning problems, the regularization parameter that is optimal for test predictive performance may be so small that the scaling above in $\sqrt{\kappa}$ is not practical anymore (see examples in Sect.~\ref{sec:exp}).
    
These ill-conditioned problems are good candidates for \emph{second-order methods}  (i.e., that use the Hessians of the objective functions) such as Newton method. These methods are traditionally discarded within machine learning for several reasons: (1) they are usually adapted to high precision results which are not necessary for generalization to unseen data for machine learning problems~\cite{bottou2008tradeoffs}, (2) computing the Newton step $\Delta_\lambda(x) = \nabla^2 f_\lambda(x)^{-1}\nabla f_\lambda(x)$ requires to form the Hessian and solve the associated linear system, leading to complexity which is at least quadratic in $d$, and thus prohibitive for large $d$, and (3) the global convergence properties are not applicable, unless the function is very special, i.e., self-concordant~\cite{nemirovskii1994interior} (which includes only few classical learning problems),  so they often are only shown to converge in a  small area around the optimal $x$.

In this paper, we argue that the three reasons above for not using Newton method can be circumvented to obtain competitive algorithms: (1) high {absolute} precisions are indeed not needed for machine learning, but faced with strongly ill-conditioned problems, even a low-precision solution requires second-order schemes; (2) many approximate Newton steps have been designed for approximating the solution of the associated large linear system \cite{Monta15,RoostaKhorasaniMahoney19,pilanci2017newton,bollapragada2018exact}; (3) we propose a novel second-order method which is globally convergent and which is based on performing approximate Newton methods for a certain class of so-called \emph{generalized self-concordant functions} which includes logistic regression~\cite{bach2010self}. For these functions, the conditioning of the problem is also characterized by a more \emph{local} quantity: $\kappa_\ell = R^2/\lambda$, where $R$ characterizes the local evolution of Hessians. This leads to second-order algorithms which are competitive with first-order algorithms for well-conditioned problems, while being superior for ill-conditioned problems which are common in practice.

    










\paragraph{Contributions.} We make the following contributions:
\begin{itemize}

\item[$(a)$] We build a global second-order method for the minimization of $f_\lambda$, which relies only on computing approximate Newton steps of the functions $f_\mu,\mu \geq \lambda$. The number of such steps will be of order $O(c \log \kappa_\ell + \log \frac{1}{\epsilon})$ where $\epsilon$ is the desired precision, and $c$ is an explicit constant. In the parametric setting (${\mathcal{H}} = \mathbb{R}^d$), $c$ can be as bad as $\sqrt{\kappa_\ell}$ in the worst-case but much smaller in theory and practice. Moreover in the non-parametric/kernel machine learning setting (${\mathcal{H}}$ infinite dimensional), $c$ does not depend on the local condition number~$\kappa_\ell$.

\item[$(b)$] Together with the appropriate quadratic solver to compute approximate Newton steps, we obtain an algorithm with the same scaling as regular first-order methods but with an improved behavior, in particular in ill-conditioned problems. Indeed, this algorithm matches the performance of the best quadratic solvers but covers any generalized self-concordant function, up to logarithmic terms.


\item[$(c)$] In the non-parametric/kernel machine learning setting we provide an explicit algorithm combining the previous scheme with Nystr\"om projections techniques. We prove that it achieves optimal generalization bounds with $O(n\mathsf{df}_\lambda)$ in time and $O(\mathsf{df}_\lambda^2)$ in memory, where~$n$ is the number of observations and $\mathsf{df}_\lambda$ is the associated degrees of freedom. In particular, this is the first large-scale algorithm to solve logistic and softmax regression in the non-parametric setting with large condition numbers and theoretical guarantees. 
\end{itemize}

\subsection{Comparison to related work}\label{sec:comparison}
We consider two cases for ${\mathcal{H}}$ and the functions $f_i$ that are common  in machine learning:  ${\mathcal{H}} = \mathbb{R}^d$ with linear (in the parameter) models with explicit feature maps, and  ${\mathcal{H}}$ infinite-dimensional, corresponding in machine learning to learning with kernels~\cite{shawe2004kernel}. Moreover in this section we first consider the quadratic case, for example the squared loss in machine learning (i.e., $f_i(x) = \frac{1}{2}(x^\top z_i - y_i)^2$ for some $z_i \in {\mathcal{H}}, y_i \in \mathbb{R}$). We first need to introduce the Hessian of the problem, for any $\lambda >0$, define 
\[ \mathbf{H}(x):= \nabla^2 f(x) , \qquad \mathbf{H}_\lambda(x):= \nabla^2 f_\lambda(x) = \mathbf{H}(x) + \lambda \mathbf{I},\]
in particular we denote by $\mathbf{H}$ (and analogously $\mathbf{H}_\lambda$) the Hessian at optimum (which in case of squared loss corresponds to the covariance matrix of the inputs). 

\paragraph{Quadratic problems and ${\mathcal{H}} = \mathbb{R}^d$ (ridge regression).}
The problem then consists in solving a (ill-conditioned) positive semi-definite symmetric linear system of dimension $d \times d$. Methods based on {\em randomized linear algebra}, {\em sketching} and suitable {\em subsampling} \cite{drineas2011faster,drineas2012fast,boutsidis2013improved} are able to find the solution with precision $\epsilon$ in time that is $O((n d + \min(n,d)^3)\log(L/\lambda \epsilon))$, so  essentially independently of the condition number, because of the logarithmic complexity in $\lambda$.

\paragraph{Quadratic problems and ${\mathcal{H}}$ infinite-dimensional (kernel ridge regression).} Here the problem corresponds to solving a (ill-conditioned) infinite-dimensional linear system in a reproducing kernel Hilbert space \cite{shawe2004kernel}. Since however the sum defining $f$ is finite, the problem can be projected on a subspace of dimension at most $n$~\cite{aronszajn1950theory}, leading to a linear system of dimension $n \times n$. Solving it with the techniques above would lead to a complexity of the order $O(n^2)$, which is not feasible on massive learning problems (e.g., $n \approx 10^7$). Interestingly these problems are usually approximately low-rank, with the rank represented by the so called {\em effective-dimension} $\mathsf{df}_\lambda$ \cite{devito}, counting essentially the eigenvalues of the problem larger than $\lambda$, 
\eqal{\label{eq:eff-dim-text}
\mathsf{df}_\lambda = \Tr(\mathbf{H} \mathbf{H}_\lambda^{-1}).
}
Note that $\mathsf{df}_\lambda$ is bounded by $\min\{ n, L/\lambda\}$ and in many cases $\mathsf{df}_\lambda \ll \min(n, L/\lambda)$.  Using suitable projection techniques, like {\em Nystr\"om}~\cite{williams2001using} or {\em random features}~\cite{rahimi2008random} it is possible to further reduce the problem to dimension $\mathsf{df}_\lambda$, for a total cost to find the solution of $O(n \mathsf{df}_\lambda^2)$. Finally recent methods~\cite{Rudi17}, combining suitable projection methods with refined preconditioning techniques, are able to find the solution with precision compatible with the optimal statistical learning error \cite{devito} in time that is $O(n \mathsf{df}_\lambda \log(L/\lambda))$, so being essentially independent of the condition number of the problem.

\paragraph{Convex problems and explicit features (logistic regression).} When the loss function is {\em self-concordant} it is possible to leverage the fast techniques for linear systems in approximate Newton algorithms \cite{pilanci2017newton} (see more in \cref{sec:background}), to achieve the solution in essentially $O(n d + \min(n,d)^3)$ time, modulo logarithmic terms. However only few loss functions of interest are self-concordant, in particular the widely used logistic and soft-max losses are not self-concordant, but {\em generalized-self-concordant}~\cite{bach2010self}. In such cases we need to use (accelerated/stochastic) first order optimization methods to enter in the quadratic convergence region of Newton methods \cite{Agarwal2017}, which leads to a solution in $O(d n + d \sqrt{ n L/\lambda} + \min(n,d)^3)$ time, which does not present any improvement on a simple accelerated first-order method.  Globally convergent second-order methods have also been proposed to solve such problems \cite{jaggi18}, but the number of Newton steps needed being bounded only by $L/\lambda$, they lead to a solution in $O(L/\lambda~(nd +\min(n,d)^3))$. With $\lambda$ that could be as small as $10^{-12}$ in modern machine learning problems, this makes both these kind of approaches expensive from a computational viewpoint for ill-conditioned problems. For such problems, with our new global second-order scheme, the algorithm we propose achieves instead a complexity of essentially $O((n d + \min(n,d)^3)\log(R^2/\lambda\epsilon))$ (see \cref{thm:main-thm}).

\paragraph{Convex problems and ${\mathcal{H}}$ infinite-dimensional (kernel logistic regression).}
Analogously to the case above, it is not possible to use Newton methods profitably as global optimizers on losses that are not self-concordant as we see in \cref{sec:global_cv_scheme}. In such cases by combining projecting techniques developped in \cref{sec:kernels} and accelerated first-order optimization methods, it is possible to find a solution in $O(n\mathsf{df}_\lambda + \mathsf{df}_\lambda \sqrt{ n  L/\lambda})$ time. This can still be prohibitive in the very small regularization scenario, since it strongly depends on the condition number $L/\lambda$. In \cref{sec:kernels} we suitably combine our optimization algorithm with projection techniques achieving optimal statistical learning error~\cite{marteau2019} in essentially $O(n \mathsf{df}_\lambda \log(R^2/\lambda))$.


\paragraph{First-order algorithms for finite sums.}
In dimension $d$, accelerated algorithms for strongly-convex smooth (not necessarily self-concordant) finite sums, such as K-SVRG~\cite{Allen2017}, have a running time proportional $O( (n+ \sqrt{n L / \lambda})d)$. This can be improved with preconditioning to $O( (n+ \sqrt{d L / \lambda})d)$ for large $n$~\cite{Agarwal2017}. Quasi-Newton methods can also be used~\cite{gower2018accelerated},  but typically without the guarantees that we provide in this paper (which are logarithmic in the condition number in natural scenarios).




 


\section{Background: Newton methods and generalized self concordance}\label{sec:background}
In this section we start by recalling the definition of generalized self concordant functions and motivate it with examples. We then recall basic facts about Newton and approximate Newton methods, and present existing techniques to efficiently compute approximate Newton steps. We start by introducing the definition of generalized self-concordance, that here is an extension of the one in~\cite{bach2010self}.
\begin{definition}[generalized self-concordant (GSC) function]\label{df:gensc}
Let ${\mathcal{H}}$ be a Hilbert space. We say that $f$ is a generalized self-concordant function on $\mathcal{G} \subset {\mathcal{H}}$, when $\mathcal{G}$ is a bounded subset of ${\mathcal{H}}$ and $f$ is a convex and three times differentiable mapping on ${\mathcal{H}}$ such that
\[ \textstyle \forall x \in {\mathcal{H}},~ \forall h,k \in {\mathcal{H}},~ \nabla^{(3)}f(x)[h,k,k] \leq \sup_{g \in \mathcal{G}}|g \cdot h| ~ \nabla^2 f(x)[k,k].\]
\end{definition}
We will usually denote by $R$ the quantity $\sup_{g \in \mathcal{G}}\|g\| < \infty$ and often omit $\mathcal{G}$ when it is clear from the context (for simplicity think of $\mathcal{G}$ as the ball in ${\mathcal{H}}$ centered in zero and with radius $R > 0$, then $\sup_{g \in \mathcal{G}}|g \cdot h| = R\|h\|$).
The globally convergent second-order scheme we present in \cref{sec:global_cv_scheme} is specific to losses which satisfy this generalized self-concordance property. The following loss functions, which are widely used in machine learning, are generalized-self-concordant, and motivate this work.
\begin{example}[Application to finite-sum minimization]\label{ex:loss-text}
The following loss functions are generalized self-concordant functions, but not self-concordant: 
\\[.05cm]
(a) Logistic  regression: $f_i(x)= \log(1 + \exp(  - y_i w_i^\top x))$, where $x, w_i \in \mathbb{R}^d$ and $y_i \in \{-1,1\}$.
\\[.05cm]
(b) Softmax regression: $f_i(x) = \log \big( \sum_{j=1}^k \exp(x_j^\top w_i)\big) - x_{y_i}^\top w_i$, where now $x \in \mathbb{R}^{d \times k}$ and $y_i \in \{1,\dots,k\}$ and $x_j$ denotes the $j$-th column of $x$.
 \\[.05cm]
    (c) Generalized linear models with bounded features (see details in \cite[Sec.~2.1]{bach2014adaptivity}), which include conditional random fields~\cite{sutton2012introduction}.
\\[.05cm]
    (d) Robust regression: $f_i(x) = \varphi(y_i -w_i^\top x)$ with $\varphi(u) = \log(e^u+e^{-u})$.
\end{example}

Note that these losses are not \textit{self-concordant} in the sense of \cite{pilanci2017newton}. Moreover, even if the losses $f_i$ are self-concordant, the objective function $f$ is not necessarily self-concordant, making any attempt to prove the self-concordance of the objective function $f$ almost impossible.  

\paragraph{Newton method (NM).}
Given $x_0 \in {\mathcal{H}}$, the Newton method consists in doing the following update:
\eqal{\label{eq:newton-intro}
x_{t+1} = x_t - \Delta_\lambda(x_t),\qquad \Delta_\lambda(x_t):= \mathbf{H}^{-1}_\lambda(x_t)\nabla f_\lambda(x_t).
}
The quantity $\Delta_\lambda(x):= \mathbf{H}^{-1}_\lambda(x)\nabla f_\lambda(x)$ is called the Newton step at point $x$, and $x - \Delta_\lambda(x)$ is the minimizer of the second order approximation of $f_\lambda$ around $x$. Newton methods enjoy the following key property: if $x_0$ is close enough to the optimum, the convergence to the optimum is quadratic and the number of iterations required to a given precision is independent of the condition number of the problem \cite{boyd2004convex}.  

However Newton methods have two main limitations:  (a) the region of quadratic convergence can be quite small and reaching the region can be computationally expensive, since it is usually done via first order methods \cite{Agarwal2017} that converge linearly depending on the condition number of the problem, (b) the cost of computing the Hessian can be really expensive when $n, d$ are large, and also (c) the cost of computing $\Delta_\lambda(x_t)$ can be really prohibitive. In the rest of the section we recall some ways to deal with (b) and (c). Our main result of \cref{sec:global_cv_scheme} is to provide globalization scheme for the Newton method to tackle  problem (a), which is easily integrable with approximate techniques to deal with (b) ans (c), to make second-order technique competitive. 

\paragraph{Approximate Newton methods (ANM) and approximate solutions to linear systems.}
Computing exactly the Newton increment $\Delta_\lambda(x_t)$, which corresponds essentially to the solution of a linear system, can be too expensive when $n, d$ are large. A natural idea is to approximate the Newton iteration, leading to {\em approximate Newton methods},
\eqal{\label{eq:approx-newton-intro}
x_{t+1} = x_t - \widetilde{\Delta}_\lambda(x_t), \qquad \widetilde{\Delta}_\lambda \approx \Delta_\lambda(x_t).
}
In this paper, more generally we consider any technique to compute $\widetilde{\Delta}_\lambda(x_t)$ that provides a {\em relative approximation} \cite{Deuflhard2011} of $\Delta_\lambda(x_t)$ defined as follows.
\begin{definition}[relative approximation]
Let $\rho < 1$, let $\mathbf{A}$ be an invertible positive definite Hermitian operator on ${\mathcal{H}}$ and $b$ in ${\mathcal{H}}$.
We denote by $\lso(\mathbf{A},b,\rho)$ the set of all  $\rho$-relative approximations of $z^* = \mathbf{A}^{-1}b$, i.e., $\lso(\mathbf{A},b,\rho) = \{z \in {\mathcal{H}} ~|~ \|z - z^*\|_{\mathbf{A}} \leq \rho \|z^*\|_{\mathbf{A}}\}$.
\end{definition}

\paragraph{Sketching and subsampling for approximate Newton methods.}
Many techniques for approximating linear systems have been used to compute $\widetilde{\Delta}_\lambda$, in particular {\em sketching} of the Hessian matrix via fast transforms and {\em subsampling} (see \cite{pilanci2017newton,bollapragada2018exact,Agarwal2017} and references therein). Assuming for simplicity that $f_i = \ell_i(w_i^\top x)$, with $\ell_i: \mathbb{R} \to \mathbb{R}$ and $w_i \in {\mathcal{H}}$, it holds:
\eqal{\label{eq:hessian-dec}
 \mathbf{H}(x) = \frac{1}{n} \sum_{i=1}^n \ell_i^{(2)}(w_i^\top x) w_i w_i^\top = V_x^\top V_x,
}
    with $V_x \in \mathbb{R}^{n\times d} = D_x W$, where $D_x \in \mathbb{R}^{n\times n}$ is a diagonal matrix defined as $(D_x)_{ii} = (\ell_i^{(2)}(w_i^\top x))^{1/2}$ and $W \in \mathbb{R}^{n \times d}$ defined as $W = (w_1, \dots, w_n)^\top$. 
    
    Both sketching and subsampling methods approximate $z^* = \mathbf{H}_\lambda(x)^{-1} \nabla f_\lambda(x)$ with $\tilde{z} = \widetilde{\mathbf{H}}_\lambda(x)^{-1} \nabla f_\lambda(x)$, in particular, in the case of subsampling $\widetilde{\mathbf{H}}(x) = \sum_{j=1}^Q p_{j} w_{i_j} w_{i_j}^\top$ where $Q \ll \min(n,d)$, $(p_j)_{j=1}^n$ are suitable weights and $(i_j)_{j=1}^Q$ are indices selected at random from $\{1,\dots,n\}$ with suitable probabilities. Sketching methods instead use $\widetilde{\mathbf{H}}(x) = \widetilde{V}_x^\top \widetilde{V}_x$, with $\widetilde{V}_x =  \Omega V_x$ with $\Omega \in \mathbb{R}^{Q \times n}$ a structured matrix such that computing $\widetilde{V}_x$ has a cost in the order of $O(n d \log n)$; to this end usually $\Omega$ is based on fast Fourier or Hadamard transforms~\cite{pilanci2017newton}.
Note that essentially all the techniques used in approximate Newton methods guarantee relative approximation. In particular the following results can be found in the literature (see \cref{lm:sub_unif,lm:sub_ny} in \cref{sec:lemmas-operators} and \cite{pilanci2017newton}, Lemma~2 for more details).

\begin{lemma}\label{lm:appr-linear-system}
Let $x, b \in {\mathcal{H}}$ and assume that $\ell_i^{(2)} \leq a$ for $a > 0$. With probability $1-\delta$ the following methods output an element in $\lso(\mathbf{H}_\lambda(x), b,\rho)$, in $O(Q^2 d + Q^3 + c)$ time, $O(Q^2+d)$ space:
\\[.05cm]
(a) Subsampling with uniform sampling (see \cite{RoostaKhorasaniMahoney19,Rudi15}), where $Q = O(\rho^{-2}a/\lambda\log \frac{1}{\lambda\delta})$ and $c = O(1)$. 
\\[.05cm]
(b) Subsampling with approximate leverage scores \cite{RoostaKhorasaniMahoney19,alaoui2015fast,Rudi15}), where $Q = O(\rho^{-2} \bar{\mathsf{df}_\lambda}\log 1/\lambda\delta), c = O(\min(n,a/\lambda)\bar{\mathsf{df}_\lambda}^2)$ and $\bar{\mathsf{df}_\lambda} = \Tr(W^\top W(W^\top W + \lambda/a I)^{-1})$ \cite{rudi2018fast}. Note that $\bar{\mathsf{df}_\lambda} \leq \min(n, d)$.\\
(c) Sketching with {\em fast Hadamard transform} \cite{pilanci2017newton}, where $Q = O(\rho^{-2} \bar{\mathsf{df}_\lambda}\log a/\lambda\delta), c = O(nd \log n)$.
\end{lemma}

\section{Globally convergent scheme for ANM algorithms on GSC functions\label{sec:global_cv_scheme}}
The algorithm is based on the observation that when $f_\lambda$ is generalized self concordant, there exists a region where $t$ steps of ANM converge as fast as $2^{-t}$. Our idea is to start from a very large regularization parameter $\lambda_0$, such that we are sure that $x_0$ is in the convergence region and perform some steps of ANM such that the solution enters in the convergence region of $f_{\lambda_1}$, with $\lambda_1 = q \lambda_0$ with $q < 1$, and to iterate this procedure until we enter the convergence region of $f_\lambda$. First we define the region of interest and characterize the behavior of NM and ANM in the region, then we analyze the globalization scheme.  

\paragraph{Preliminary results: the Dikin ellipsoid.}
We consider the following region that we prove to be contained in the region of quadratic convergence for the Newton method and that will be useful to build the globalization scheme. Let $c,R > 0$ and $f_\lambda$ be generalized self-concordant with coefficient~$R$, we call {\em Dikin ellipsoid} and denote by $\mathsf{D}_\lambda(\mathsf{c})$ the region
\[\mathsf{D}_\lambda(\mathsf{c}):= \big\{x ~|~ \nu_\lambda(x) \leq \mathsf{c} \sqrt{\lambda}/R  \big\}, \quad \textrm{with} \quad \nu_\lambda(x):= \|\nabla f_\lambda(x)\|_{\mathbf{H}^{-1}_\lambda(x)},\]
where $\nu_\lambda(x)$ is usually called the {\em Newton decrement} and $\|x\|_{\mathbf{A}}$ stands for $\|\mathbf{A}^{1/2} x\|$.
\begin{lemma}\label{lm:dikin-is-cool}
Let $\lambda > 0, \mathsf{c} \leq 1/7$, let $f_\lambda$ be generalized self-concordant and $x \in \mathsf{D}_\lambda(\mathsf{c})$. Then it holds:
   $
    \frac{1}{4}\nu_\lambda(x)^2 \leq f_\lambda(x) - f_\lambda(x_\lambda^{\star}) \leq \nu_\lambda(x)^2$.
Moreover Newton method starting from $x_0$ has quadratic convergence, i.e., let $x_t$ be obtained via $t \in \mathbb{N}$ steps of Newton method in \cref{eq:newton-intro}, then
$ \nu_\lambda(x_t) \leq 2^{-(2^{t} - 1)} \nu_\lambda(x_0).$
Finally, approximate Newton methods starting from $x_0$ have a linear convergence rate, i.e., let $x_t$ given by \cref{eq:approx-newton-intro}, with $\widetilde{\Delta}_t \in \lso(\mathbf{H}_\lambda(x_t),\nabla f_\lambda(x_t),\rho)$ and $\rho \leq 1/7$, then
$\nu_\lambda(x_t) \leq 2^{-t} \nu_\lambda(x_0).$
\end{lemma}

This result is proved in \cref{lm:dikin-is-cool_2} in \cref{app:newton_particular}. 
The crucial aspect of the result above is that when $x_0 \in \mathsf{D}_\lambda(\mathsf{c})$, the convergence of the approximate Newton method is linear and does not depend on the condition number of the problem. However $\mathsf{D}_\lambda(\mathsf{c})$ itself can be very small depending on $\sqrt{\lambda}/R$. In the next subsection we see how to enter in $\mathsf{D}_\lambda(\mathsf{c})$ in an efficient way.


\paragraph{Entering the Dikin ellipsoid using a second-order scheme.}
The lemma above shows that $\mathsf{D}_\lambda(\mathsf{c})$ is a good region where to use the approximate Newton algorithm on GSC functions. However the region itself is quite small, since it depends on $\sqrt{\lambda}/R$. Some other globalization schemes arrive to regions of interest by first-order methods or back-tracking schemes \cite{Agarwal2017,Monta15}. However such approaches require a number of steps that is  usually proportional to $\sqrt{L/\lambda}$ making them non-beneficial in machine learning contexts. Here instead we consider the following simple scheme where $\texttt{ANM}_\rho(f_\lambda,x,t)$ is the result of a $\rho$-relative approximate Newton method performing $t$ steps of optimization starting from $x$.

The main ingredient to guarantee the scheme to work is the following lemma (see \cref{lm:next_la} in \cref{app:dec_la_tech} for a proof).

\begin{lemma}\label{lm:dec_la}
Let $\mu > 0$, $\mathsf{c} < 1$ and $x \in {\mathcal{H}}$.  Let $s = 1 + R\|x\|/\mathsf{c}$, then for $q \in [1 - 2/(3s), 1)$
\[\mathsf{D}_\mu(\mathsf{c}/3) \subseteq \mathsf{D}_{q\mu}(\mathsf{c}).\]
\end{lemma} 
Now we are ready to show that we can guarantee the loop invariant $x_k \in \mathsf{D}_{\mu_k}(\mathsf{c})$.
Indeed assume that $x_{k-1} \in \mathsf{D}_{\mu_{k-1}}(\mathsf{c})$. Then $\nu_{\mu_{k-1}}(x_{k-1}) \leq \mathsf{c}\sqrt{\mu_{k-1}}/R$. By taking $t = 2, \rho = 1/7$, and performing $x_{k} = \texttt{ANM}_\rho(f_{\mu_{k-1}},x_{k-1},t)$, by \cref{lm:dikin-is-cool}, 
$\nu_{\mu_{k-1}}(x_{k}) \leq 1/4 \nu_{\mu_{k-1}}(x_{k-1}) \leq \mathsf{c}/4~\sqrt{\mu_{k-1}}/R$,
i.e., $x_{k} \in \mathsf{D}_{\mu_{k-1}}(\mathsf{c}/4)$. If $q_{k}$ is large enough, this implies that $x_{k} \in \mathsf{D}_{q_{k}\mu_{k-1}}(\mathsf{c}) = \mathsf{D}_{\mu_{k}}(\mathsf{c})$, by \cref{lm:dec_la}. Now we are ready to state our main theorem of this section.

\vspace*{.3cm}

\fbox{
\begin{minipage}[t]{0.9\textwidth}
\textbf{Proposed Globalization Scheme}
\begin{center}
\textit{Phase I: Getting in the Dikin ellispoid of $f_\lambda$}
\end{center}

\vspace*{-.1cm}

Start with $x_0 \in {\mathcal{H}}, \mu_0 > 0$, $t, T \in \mathbb{N}$ and $(q_k)_{k \in \mathbb{N}} \in (0,1]$.\\
For $k \in \mathbb{N}$\\
${}\qquad x_{k+1} \leftarrow \texttt{ANM}_\rho(f_{\mu_k},x_{k},t)$\\
${}\qquad \mu_{k+1} \leftarrow q_{k+1}\mu_{k}$\\
Stop when $\mu_{k+1} < \lambda$ and set $x_{last}\leftarrow x_k$.
\begin{center}
\textit{Phase II: reach a certain precision starting from  inside the Dikin ellipsoid}
\end{center}

\vspace*{-.1cm}

Return $\widehat{x} \leftarrow \texttt{ANM}_\rho(f_{\lambda},x_{last},T)$
\end{minipage}
}



\paragraph{Fully adaptive method.} The scheme presented above converges with the following parameters. 

\begin{theorem} \label{thm:easy_first_phase}\label{thm:main-thm}
Let $\epsilon >0$. Set  $\mu_0 = 7 R \|\nabla f (0)\|$, $x_0 = 0$, and perform the globalization scheme above for $\rho \leq 1/7, t = 2$, and
$ \textstyle q_k = \frac{1/3 + 7 R \|x_{k}\|}{1 + 7 R \|x_{k}\|}$, $ T = \lceil \log_2 \sqrt{1 ~\vee ~ (\lambda \epsilon^{-1} / R^2)}\rceil.$
Then denoting by $K$ the number of steps performed in the Phase I, it holds:
\[f_\lambda(\widehat{x}) - f_\lambda(x_\lambda^\star) \leq \epsilon, \qquad K \leq \left\lfloor \left(3 + 11 R \|x_\lambda^{\star}\|\right)\log (7 R \|\nabla f(0)\|/\lambda)\right\rfloor.\]
\end{theorem} 
 Note that the theorem above (proven in \cref{app:proof_main_thm}) guarantees a solution with error $\epsilon$ with $K$ steps of ANM each performing 2 iterations of approximate linear system solving, plus a final step of ANM which performs $T$ iterations of approximate linear system solving. In case of $f_i(x) = \ell_i(w_i^\top x)$, with $\ell_i:\mathbb{R}\to\mathbb{R}$, $w_i \in {\mathcal{H}}$ with $\ell_i^{(2)} \leq a$, for $a > 0$, the final runtime cost of the proposed scheme to achieve precision $\epsilon$, when combined with of the methods for approximate linear system solving from \cref{lm:appr-linear-system} (i.e. sketching), is $O(Q^2 + d)$ in memory and
\[O\Big((nd\log n + dQ^2 + Q^3)\Big(R \|x_\lambda^{\star}\|\log\frac{R}{\lambda} + \log\frac{\lambda}{R\epsilon}\Big)\Big)~~\textrm{in time},\quad Q = O\Big(\bar{\mathsf{df}_\lambda}\log\frac{1}{\lambda\delta}\Big),\]
where $\bar{\mathsf{df}_\lambda}$, defined in \cref{lm:appr-linear-system}, measures the {\em effective dimension} of the correlation matrix $W^\top W$ with $W = (w_1,\dots, w_n)^\top \in \mathbb{R}^{n \times d}$, corresponding essentially to the number of eigenvalues of $W^\top W$ larger than $\lambda/a$. In particular note that $\bar{\mathsf{df}_\lambda} \leq \min(n, d, \text{rank}(W), a b^2/\lambda)$, with  $b:= \max_i\|w_i\| $, and usually way smaller than such quantities.
\begin{remark} \label{rm:cond-dependence}
The proposed method does not depend on the condition number of the problem $L/\lambda$, but on the term $R \|x_\lambda^{\star}\|$ which can be in the order of $R/\sqrt{\lambda}$ in the worst case, but usually way smaller. For example, it is possible to prove that this term is bounded by an absolute constant not depending on $\lambda$, if at least one minimum for $f$ exists. In the appendix (see \cref{prp:adap_var}), we show a variant of this adaptive method which can leverage the regularity of the solution with respect to the Hessian, i.e., depending on the smaller quantity $R \sqrt{\lambda}\|x^\star_{\lambda}\|_{\mathbf{H}^{-1}_{\lambda}(x^\star_{\lambda})}$ instead of $R\|x_\lambda^{\star}\|$.
\end{remark}
Finally note that it is possible to use $q_k = q$ fixed for all the iterations and way smaller than the one in \cref{thm:main-thm}, depending on some regularity properties of $\mathbf{H}$ (see \cref{prp:fixed_q} in \cref{app:maintheorems}).

\section{Application to the non-parametric setting: Kernel methods}\label{sec:kernels}
In supervised learning the goal is to predict well on future data, given the observed training dataset. Let $\mathcal{X}$ be the input space and $\mathcal{Y} \subseteq \mathbb{R}^p$ be the output space. We consider a probability distribution $P$ over $\mathcal{X} \times \mathcal{Y}$ generating the data and the goal is to estimate $g^*:\mathcal{X} \to \mathcal{Y}$ solving the problem 
\eqal{\label{eq:ideal-problem}
g^* = \argmin_{g:\mathcal{X} \to \mathcal{Y}} \mathcal{L}(g), \quad \mathcal{L}(g) = \mathbb{E}[\ell(g(x),y)],
}
for a given loss function $\ell:\mathcal{Y}\times \mathcal{Y} \to \mathbb{R}$. Note that $P$ is not known, and accessible only via the dataset $(x_i,y_i)_{i=1}^n$, with $n \in \mathbb{N}$, independently sampled from $P$. A prototypical estimator for $g^*$ is the regularized minimizer of the empirical risk $\widehat{\mathcal{L}}(g) = \frac{1}{n}\sum_{i=1}^n \ell(g(x_i), y_i)$ over a suitable space of functions $\cal G$. Given $\phi: \mathcal{X} \to {\mathcal{H}}$ a common choice is to select ${\cal G}$ as the set of linear functions of $\phi(x)$, that is, ${\cal G} = \{ w^\top \phi(\cdot) ~|~ w \in {\mathcal{H}}\}$. Then the regularized minimizer of $\widehat{\mathcal{L}}$, denoted by $\widehat{g}_\lambda$, corresponds to  
\eqal{\label{eq:ERM-text}
\widehat{g}_\lambda(x) = \widehat{w}_\lambda^\top \phi(x), \quad   \widehat{w}_\lambda = \argmin_{w \in {\mathcal{H}}} \textstyle \frac{1}{n} \sum_{i=1}^n f_i(w) + \lambda\|w\|^2, \quad f_i(w) = \ell(w^\top \phi(x_i), y_i).
}
Learning theory guarantees how fast $\widehat{g}_\lambda$ converges to the best possible estimator $g^*$ with respect to    the number of observed examples, in terms of the so called {\em excess risk} $\mathcal{L}(\widehat{g}_\lambda) - \mathcal{L}(g^*)$. The following theorem recovers the minimax optimal learning rates for squared loss and extend them to any generalized self-concordant loss function.

{\em Note on $\mathsf{df}_\lambda$.} In this section, we always denote with $\mathsf{df}_\lambda$ the effective dimension of the problem in \cref{eq:ideal-problem}. 
When the loss belongs to the family of generalized linear models (see \cref{ex:loss-text}) and if the model is well-specified, then $\mathsf{df}_\lambda$ is defined exactly as in \cref{eq:eff-dim-text} otherwise we need a more refined definition (see \cite{marteau2019} or  \cref{eq:deff-statistics} in \cref{app:kernels}).

\begin{theorem}[from \cite{marteau2019}, Thm. 4]\label{thm:opt-rates-ERM-text}
Let $\lambda > 0, \delta \in (0,1]$. Let $\ell$ be generalized self-concordant with parameter $R > 0$ and $\sup_{x \in X} \|\phi(x)\| \leq C < \infty$. Assume that there exists $g^*$ minimizing $\mathcal{L}$. Then there exists $c_0$ not depending on $n, \lambda, \delta, \mathsf{df}_\lambda, C, g^*$, such that if $\sqrt{\mathsf{df}_\lambda/n},\mathsf{b}_\lambda \leq \lambda^{1/2}/R$, and $n \geq C/\lambda \log(\delta^{-1}C/\lambda)$ the following holds with probability $1-\delta$:
\begin{equation}
\label{eq:opt-rates-ERM-text}
\mathcal{L}(\widehat{g}_\lambda) - \mathcal{L}(g^*) \leq c_0\Big(\frac{\mathsf{df}_\lambda}{n} + \mathsf{b}^2_\lambda\Big)\log (1/\delta), \qquad \mathsf{b}_\lambda:=  \lambda \|g^*\|_{\mathbf{H}_\lambda^{-1}(g^*)}.
\end{equation}
\end{theorem}
Under standard regularity assumptions of the learning problems \cite{marteau2019}, i.e., (a) the {\em capacity condition} $\sigma_j(\mathbf{H}(g^*)) \leq C j^{-\alpha}$, for $\alpha \geq 1,  C > 0$ (i.e., a decay of eigenvalues $\sigma_j(\mathbf{H}(g^*))$ of the Hessian at the optimum), and (b) the {\em source condition}  $g^* = \mathbf{H}(g^*)^r v$, with $v \in {\mathcal{H}}$ and $r > 0$ (i.e., the control of the optimal $g^\ast$ for a specific Hessian-dependent norm),   $\mathsf{df}_\lambda \leq C' \lambda^{-1/\alpha}$ and $\mathsf{b}^2_\lambda \leq C^{\prime \prime} \lambda^{1+2r}$, leading to the following optimal learning rate,
\eqal{\label{eq:rates-text}
\mathcal{L}(\widehat{g}_\lambda) - \mathcal{L}(g^*)  \leq c_1 n^{-\frac{1+2r \alpha}{1 + \alpha + 2r\alpha}} \log (1/\delta), \quad \textrm{when} \quad \lambda = n^{-\frac{\alpha}{1+\alpha+2r\alpha}}.
}
Now we propose an algorithmic scheme to compute efficiently an approximation of $\widehat{g}_\lambda$   that achieves the same optimal learning rates. First we need to introduce the technique we are going to use. 

\paragraph{Nystr\"om projection.} It consists in suitably selecting $\{\bar{x}_1, \dots, \bar{x}_M\} \subset \{x_1,\dots, x_n\}$, with $M \ll n$ and computing $\bar{g}_{M,\lambda},$ i.e., the solution of \cref{eq:ERM-text} over ${\mathcal{H}}_M = \text{span}\{\phi(\bar{x}_1),\dots, \phi(\bar{x}_M)\}$ instead of ${\mathcal{H}}$. In this case the problem can be reformulated as a problem in $\mathbb{R}^M$ as
\eqal{\label{eq:prob-ny-char-text}
\bar{g}_{M,\lambda} = \bar{\alpha}_{M,\lambda}^\top {\bf T}^{-1} v(x), \qquad \bar{\alpha}_{M,\lambda} = \argmin_{\alpha \in \mathbb{R}^M} \bar{f}_\lambda(\alpha), \qquad \bar{f}(\alpha) = \frac{1}{n} \sum_{i=1}^n \bar{f}_i(\alpha) + \lambda\|\alpha\|^2,
}
where $\bar{f}_i(\alpha) = \ell( v(x_i)^\top {\bf T}^{-1}\alpha,~y_i)$ and $v(x) \in \mathbb{R}^M$, $v(x) = (k(x,\bar{x}_1), \dots, k(x,\bar{x}_M))$ with $k(x,x') = \phi(x)^\top \phi(x')$ the associated positive-definite kernel~\cite{shawe2004kernel}, while ${\bf T}$ is the upper triangular matrix such that ${\bf K} = {\bf T}^\top {\bf T} $, with ${\bf K} \in \mathbb{R}^{M \times M}$ with ${\bf K}_{ij} = k(\bar{x}_i,\bar{x}_j)$. In the next theorem we characterize the sufficient $M$ to achieve minimax optimal rates, for two standard techniques of choosing the Nystr\"om points $\{\bar{x}_1, \dots, \bar{x}_M\}$.

\begin{theorem}[Optimal rates for learning with Nystr\"om]\label{thm:opt-rates-ny-text}
Let $\lambda > 0, \delta \in (0,1]$. Assume the conditions of \cref{thm:opt-rates-ERM-text}. Then the excess risk of $\bar{g}_{M,\lambda}$ is bounded with prob. $1-2\delta$ as in \cref{eq:opt-rates-ERM-text} (with $c_1' \propto c_1$), when\\
${}\qquad(1)~$ Uniform Nystr\"om method \cite{Rudi15,Rudi17} is used and $~M ~\geq~ C_1/\lambda~\log(C_2/\lambda\delta)$.\\
${}\qquad(2)~$ Approximate leverage score method \cite{alaoui2015fast,Rudi15,Rudi17} is used and $~M ~\geq~ C_3~\mathsf{df}_\lambda~\log(C_4/\lambda\delta)$.\\
Here $C, C_1, C_2, C_4$ do not depend on $\lambda, n, M, \mathsf{df}_\lambda, \delta$.
\end{theorem}
\cref{thm:opt-rates-ny-text} generalizes results for learning with Nystr\"om and squared loss \cite{Rudi15}, to GSC losses. It is proved in \cref{thm:nystrom_estimator_bounds}, in \cref{app:selecting_nys}. As in \cite{Rudi15}, \cref{thm:opt-rates-ny-text} shows that Nystr\"om is a valid technique for dimensionality reduction. Indeed it is essentially possible to project the learning problem on a subspace ${\mathcal{H}}_M$ of dimension $M = O(c/\lambda)$ or even as small as $M = O(\mathsf{df}_\lambda)$ and still achieve the optimal rates of \cref{thm:opt-rates-ERM-text}. Now we are ready to introduce our algorithm.

\paragraph{Proposed algorithm.}
The algorithm conceptually consists in (a) performing a projection step with Nystr\"om, and  (b) solving the resulting optimization problem with the globalization scheme proposed in \cref{sec:global_cv_scheme} based on ANM in \cref{eq:approx-newton-intro}. 
In particular, we want to avoid to apply explicitly ${\bf T}^{-1}$ to each $v(x_i)$ in \cref{eq:prob-ny-char-text}, which would require $O(n M^2)$ time. Then we will use the following approximation technique based only on matrix vector products, so we can just apply ${\bf T}^{-1}$ to $\alpha$ at each iteration, with a total cost proportional only to $O(n M  + M^2)$ per iteration.
Given $\alpha, \nabla \bar{f}_\lambda(\alpha)$, we approximate $z^* =  \bar{\mathbf{H}}_\lambda(\alpha)^{-1} \nabla \bar{f}_\lambda(\alpha)$, where $\bar{\mathbf{H}}_\lambda$ is the Hessian of $\bar{f}_\lambda(\alpha)$, with $\tilde{z}$ defined as
\[ \tilde{z} = \texttt{prec-conj-grad}_t(\bar{\mathbf{H}}_\lambda(\alpha), \nabla \bar{f}_\lambda(\alpha)),\]
where $\texttt{prec-conj-grad}_t$ corresponds to performing $t$ steps of preconditioned conjugate gradient~\cite{golub2012matrix} with preconditioner computed using a subsampling approach for the Hessian among the ones presented in \cref{sec:background}, in the paragraph starting with \cref{eq:hessian-dec}. The pseudocode for the whole procedure is presented in Alg.~\ref{alg:algo}, \cref{sec:algorithm}.
This technique of approximate linear system solving has been studied in \cite{Rudi17} in the context of empirical risk minimization for squared loss.

\begin{lemma}[\cite{Rudi17}]\label{lm:pcg-appr-tech}
Let $\lambda > 0, \alpha, b \in \mathbb{R}^M$. The previous method, applied with $t = O(\log 1/\rho)$, outputs an element of $\lso(\bar{\mathbf{H}}_{\lambda}(\alpha), b,\rho)$, with probability $1-\delta$ with complexity $O((nM + M^2 Q + M^3 + c)t)$ in time and $O(M^2 + n)$ in space, with $Q = O(C_1/\lambda \log (C_1/\lambda\delta)), c = O(1)$ if uniform sub-sampling is used or $Q = O(C_2\mathsf{df}_\lambda \log (C_1/\lambda\delta)), c = O(\mathsf{df}_\lambda^2\min(n,\frac{1}{\lambda}))$ if sub-sampling with leverage scores is used \cite{rudi2018fast}.
\end{lemma}

A more complete version of this lemma is shown in \cref{prp:approx_ns_kernels} in \cref{app:ans_kernels}. 
We conclude this section with a result proving the learning properties of the proposed algorithm.

\begin{theorem}[Optimal rates for the proposed algorithms]\label{thm:final-alg-kernel-text}
Let $\lambda > 0$ and $\epsilon < \lambda/R^2$. Under the hypotheses of \cref{thm:opt-rates-ny-text}, if we set $M$ as in \cref{thm:opt-rates-ny-text}, $Q$ as in \cref{lm:pcg-appr-tech} and setting the globalization scheme as in \cref{thm:main-thm}, then the proposed algorithm (Alg.~\ref{alg:algo}, \cref{sec:algorithm}) finishes in a finite number of newton steps $N_{ns} = O(R \|g^{*}\|\log(C/\lambda) +\log(C/\epsilon))$ and returns a predictor $g_{Q,M,\lambda}$ of the form $g_{Q,M,\lambda} = \alpha^\top {\bf T}^{-1}v(x)$. With probability at least $1-\delta$, this predictor satisfies: 
\begin{equation}
\label{eq:opt-alg}
\mathcal{L}(g_{Q,M,\lambda}) - \mathcal{L}(g^*) \leq c_0\Big(\frac{\mathsf{df}_\lambda}{n} + \mathsf{b}^2_\lambda + \epsilon\Big)\log (1/\delta), \qquad \mathsf{b}_\lambda:=  \lambda \|g^*\|_{\mathbf{H}_\lambda^{-1}(g^*)}.
\end{equation}
\end{theorem}
The theorem above (see \cref{prp:algo_stat_gen}, \cref{app:main_kernels} for exacts quantifications) shows that the proposed algorithm is able to achieve the same learning rates of plain empirical risk minimization as in \cref{thm:opt-rates-ERM-text}. The total complexity of the procedure, including the cost of computing the preconditioner,  the selection of the Nystr\"om points via approximate leverage scores and also the computation of the leverage scores \cite{rudi2018fast} is then
$$O\left(R \|g^*\| \log (R^2/\lambda)\left(n~\mathsf{df}_\lambda \log(C\lambda^{-1}\delta^{-1})~c_X +  ~+~ \mathsf{df}_\lambda^3 \log^3(C\lambda^{-1} \delta^{-1}) + \min(n,C/\lambda)~\mathsf{df}_\lambda^2 \right)\right)$$ in time and $O(\mathsf{df}_\lambda^2\log^2(C\lambda^{-1}\delta^{-1}))$ in space, where $c_X$ is the cost of computing the inner product $k(x,x')$ (in the kernel setting assumed when the input space $X$ is $X=\mathbb{R}^p$ it is $c=O(p)$). As noted in \cite{rudi2018fast}, under the standard regularity assumptions on the learning problem seen above, $\mathsf{df}_\lambda^2 \leq \mathsf{df}_\lambda/\lambda \leq n$ when the optimal $\lambda$ is chosen. So the total computational complexity  is
\[ O\left(R  \log (R^2/\lambda)~ \log^3(C\lambda^{-1} \delta^{-1})~\|g^*\|\cdot  n \cdot\mathsf{df}_\lambda \cdot c_X \right) ~~\textrm{in time}, \quad O(\mathsf{df}_\lambda^2\cdot \log^2(C \lambda^{-1}\delta^{-1})) ~~\textrm{in space}.\]
First note, the fact that due to the statistical properties of the problem the complexity does not depend even implicitly on $\sqrt{C/\lambda}$, but only on $\log(C/\lambda)$, so the algorithm runs in essentially $O(n \mathsf{df}_\lambda)$, compared to $O(\mathsf{df}_\lambda \sqrt{n C/\lambda})$ of the accelerated first-order methods we develop in \cref{sec:exp-appendix} and the $O(n \mathsf{df}_\lambda \sqrt{ C/\lambda})$ of other Newton schemes (see \cref{sec:comparison}). To our knowledge,
this is the first algorithm to achieve optimal statistical learning rates for generalized self-concordant losses and with complexity only $\widetilde{O}(n \mathsf{df}_\lambda)$. This generalizes similar results for squared loss \cite{Rudi17,rudi2018fast}.

\section{Experiments}
\label{sec:exp}

The code necessary to reproduce the following experiments is available on GitHub at \url{https://github.com/umarteau/Newton-Method-for-GSC-losses-}.

We compared the performances of our algorithm for kernel logistic regression on two large scale classification datasets ($n\approx 10^7$), Higgs and Susy, pre-processed as in \cite{Rudi17}. We implemented the algorithm in pytorch and performed the computations on $1$ Tesla P100-PCIE-16GB GPU. For Susy ($n=5 \times 10^6, p=18$): we used Gaussian kernel with $k(x,x') = e^{-\|x-x'\|^2/(2\sigma^2)}$, with $\sigma=5$, which we obtained through a grid search (in \cite{Rudi17}, $\sigma = 4$ is taken for the ridge regression); $M=10^4$ Nystr\"om centers and a subsampling $Q = M$ for the preconditioner, both obtained with uniform sampling. Analogously for Higgs ($n=1.1 \times 10^7, p=28$): , we used a Gaussian kernel with $\sigma=5$ and $M=2.5\times10^4$ and $Q=M$, using again uniform sampling.
\begin{figure}
    \centering
    \includegraphics[width=0.44\textwidth]{susy_distance_class_small_arxiv.pdf}~~
    \includegraphics[width=0.44\textwidth]{higgs_distance_class_arxiv.pdf}
   
    \caption{Training loss and test error as as function of the number of passes on the data for our algorithm vs. K-SVRG. on the \textbf{(left)} Susy and \textbf{(right)} Higgs data sets.}
    \label{fig:time-comparison}
\end{figure}
To find reasonable $\lambda$ for supervised learning applications, we cross-validated $\lambda$ finding the minimum test error at $\lambda = 10^{-10}$ for Susy and $\lambda = 10^{-9}$ for Higgs (see \cref{fig:test_susy,fig:test_higgs} in \cref{sec:exp-appendix}) for such values our algorithm and the competitor achieve an error of 19.5\% on the test set for Susy, comparable to the state of the art (19.6\% \cite{Rudi17}) and analogously for Higgs (see \cref{sec:exp-appendix}). We then used such $\lambda$'s as regularization parameters and compared our algorithm with a well known accelerated stochastic gradient technique {\em Katyusha SVRG} (K-SVRG)~\cite{Allen2017}, tailored to our problem using mini batches. In \cref{fig:time-comparison} we show the convergence of the training loss and classification error with respect to the number of passes on the data, of our algorithm compared to K-SVRG. It is possible to note our algorithm is order of magnitude faster in achieving convergence, validating empirically the fact that the proposed algorithm scales as $O(n \mathsf{df}_\lambda)$ in learning settings, while accelerated first order methods go as $O( (n+ \sqrt{n L / \lambda})\mathsf{df}_\lambda)$. Moreover, as mentioned in the introduction, this highlights the fact that precise optimization is necessary to achieve a good performance in terms of test error. Finally, note that since a pass on the data is much more expensive for K-SVRG than for our second order method (see \cref{sec:exp-appendix} for details), the difference in computing time between the second order scheme and K-SVRG is even more in favour of our second order method (see \cref{fig:susy_compare_small,fig:higgs_compare} in \cref{sec:exp-appendix}). 


\subsection*{Acknowledgments}

We acknowledge support from the European Research Council (grant SEQUOIA 724063).


