\section{Introduction}
\subsection{Blind Deconvolution}
Suppose we are interested in an underlying time series ${\bf x} = (x(t))$ which we cannot observe directly. What we can observe is  ${\bf y} = {\bf a} * {\bf x}$ where ${\bf a}$ is an unknown `blurring' filter. Blind deconvolution is the problem of recovering ${\bf x}$ merely from the observed ${\bf y}$ without knowing either ${\bf a}$ or ${\bf x}$. 

This problem occurs naturally in seismology and
digital communications
as well as astronomy, satellite imaging, and computer vision.

In its most ambitious form, the problem is literally impossible; there are simply too few data and too many unknowns. Indeed, imagine that ${\bf x}$, ${\bf y}$ and ${\bf a}$ all have $N$ entries; we observe only $N$ pieces of information (${\bf y}$) but there are $2N$ unknowns (${\bf x}$ and ${\bf a}$). 
Nevertheless, in \sout{various} \redit{some (not all)} fields, heuristic \redit{approaches}
\sout{methods showed occasional} \redit{have occasionally 
led to consistent } success \redit{in isolated applications};
\redit{presumably such success stories exploit} \sout{under various} specialized assumptions \redit{-- although not always in
an explicit or recognized way, and not with rigorous understanding}.

\redit{This paper, in contrast,  will exhibit a set of 
assumptions enabling practical algorithms for blind deconvolution,
backed by rigorous theoretical analysis}.

\subsection{The Promise of Sparsity}

\redit{Central to our approach
is an assumption about the}
{\it sparsity of the signal ${\bf x}$ to be recovered}, 
in which case the problem \sout{becomes}
\redit{can be called} \emph{sparse blind deconvolution}. Sparse signals\sout{,} \redit{-- i.e. signals} having relatively few nonzero entries, arise frequently in many fields, including seismology, microscopy, astronomy, neuroscience spike identification. Even in more abstract settings such as representation learning for computer vision, 
it surfaces in recently popular research trends, 
such as single-channel convolutional dictionary learning   \cite{bristow2013fast, heide2015fast,zhang2017global,zhang2018structured}. 

The sparsity of ${\bf x}$ \-- if it holds \-- would constrain the recovery problem significantly; and so possibly, sparsity can play a role
in enabling useful solutions to an otherwise hopeless problem.

An inspiring precedent can be found in modern commercial medical imaging, 
where sparsity of an image's wavelet coefficients enables 
MRIs from fewer observations than unknowns.
Taking fewer observations speeds up data collection,
a principle known as compressed sensing, which
already benefits tens of millions of patients yearly.

\subsection{Translating Heuristics into Effective Algorithms}

For sparsity to reliably enable blind deconvolution, there are two
apparent hurdles. First, develop an objective function which promotes
sparsity of the solution. Second, develop an algorithm 
which can reliably optimize the objective.

Many sparsity-promoting objectives have 
been proposed over the years; typically they 
imply non-convex optimization problems. Indeed
sparsity is quantified by the $\ell_0$
pseudo norm $\|{\bf x}\|_0 = \#\{t: x(t) \neq 0\}$, which is the limit of
$\| {\bf x} \|_p^p$ as $p \rightarrow 0$
of concave $\ell_p$ pseudo-norms where $p <1$.

Traditionally, non-convex problems have been viewed
by mathematical scientists with skepticism;
for such problems, gradient descent and its various refinements
lack any guarantee of effectiveness. 
Still, the lack of guarantees has not stopped engineers from trying!

A noticeable success in blind signal processing
was scored in digital communications,
where blind equalization today 
benefits billions of smartphone users.
Blind equalization is a form of blind deconvolution 
where one exploits the known discrete-valued 
nature of the signal ${\bf x}$ (for example the signal 
entries might take only two values $\{-1,1\}$).
Practitioners found that if an initial guess of 
the equalizer (i.e. our inverse filter ${\bf a}^{-1}$) is `fairly good' 
(in engineer-speak `opening the eye' so that a `hint' 
of the `digital constellation` becomes
`visible'), then certain `discreteness-promoting' on-line gradient methods
can reliably `focus' the result better and better 
and allow reliable recovery.

Our work identifies an 
analogous phenomenon in the sparsity-promoting blind deconvolution
setting, however it exposes and crystallizes the phenomenon in a rigorous 
and dependably exploitable form.

Namely, we show that if sparsity of ${\bf x}$ holds,
and if an initial guess of the filter ${\bf a}$
is `fairly good' in a precise sense, 
then a specific convex optimization
algorithm will accurately recover 
both the filter and the original signal\footnote{As we explain below, recover means: recover up to rescaling and time shift.}.

In retrospect, our insights on sparsity-promoting blind deconvolution
can be cross-applied to explain the major successes of discreteness-promoting
blind equalization in modern digital communications.
Namely, a direct variation of our arguments 
provide a related convex optimization problem 
for discrete-valued signals which rigorously converts a 
a rough initial approximation into precise recovery.

In our view these new arguments clear away some persistent
fog, mystery and misunderstandings in blind signal processing; and pave
the way for future success stories.


\subsection{Prior Work}
\begin{comment}
\paragraph{Non-convexity for blind deconvolution problems}
As both the sparse signal and the forward filter ${\bf a}$ are unknown, the problem is bilinear in ${\bf a}$ and $X$. And the expression ${\bf a}*{\bf x}$ has scaling and shift symmetry for $({\bf a},{\bf x})$.
Therefore, blind deconvolution is often formulated as a non-convex problem. Lots of previous works \cite{kuo2019geometry, kuo2020geometry, lau2019short} studied the that problem of recovering short $a$ and sparse $x$ as an non-convex optimization problem.

\Dave{The previous paragraph is jumbled and also out of order. If it has any value or content it should come later.}
\end{comment}

\paragraph{Searching for an inverse filter that promotes desired output properties}
Instead of trying to recover ${\bf a}$ and ${\bf x}$ together from ${\bf a}*{\bf x}$, 
we could formulate this problem as looking for an approximate inverse  filter ${\bf w}$ so that the output ${\bf w}*{\bf y}$ exhibits \sout{desired} \redit{extremal} properties. 
\sout{Namely} \redit{Under this formulation, 
our goal is to find} ${\bf w}\neq 0$, so that  
\begin{equation}
 \begin{array}{ll}
 \underset{{\bf w}\neq 0}{\mbox{optimize}}   & J({\bf w} \star {\bf y}) \label{JoptBD}
\end{array}
\end{equation}
where the functional $J$ quantifies the properties we seek to promote.
(Depending on $J$, we might \redit{either prefer its maximum or minimum} \sout{it}).
 
 \newcommand{{\bf z}}{{\bf z}}
\sout{Motivated by blind deconvolution} \redit{Working} in
exploration seismology, 
Wiggins \cite{wiggins1978minimum} adopted this approach
with the normalized $4$-norm, $J({\bf z}) = J_{4,2} = {\|{\bf z}\|_4}/{\|{\bf z}\|_2}$
and gave a few successful data-processing case studies. 
His \redit{successful} examples all
clearly exhibit sparsity, although this was not discussed at the time.
Other objectives 
considered at that time included $J_{2,1}({\bf z}) = \|{\bf z}\|_2/\|{\bf z}\|_1$
and $J_{\infty,2}({\bf z}) = \|{\bf z}\|_\infty/\|{\bf z}\|_2$ \cite{cabrelli1985minimum}.

It was fully understood at that time
that output property optimization could succeed in principle,
if one did not have to worry about an effective algorithm.
\cite{donoho1981minimum} showed that 
if the signal ${\bf x}$ is a realization of
independent and identically distributed
entries from any nonGaussian distribution,  optimizing
$J$ of the output ${\bf w} \star {\bf y}$ is successful in the large-$N$ setting \-- as long as the \redit{functional} \sout{property} $J$ belongs to a \redit{certain} large family of non-Gaussianity measures,
for example including $J_{4,2}$ and $J_{2,1}$ as well as many others. 
\footnote{Sparsity is of course a form of non-Gaussianity,
this very explicitly in the Bernoulli-Gaussian mixture model considered below.}

The issue left unresolved in those days was how to solve such
optimization problems. Indeed optimizations like $J_{4,2}$ are badly nonconvex,
as we see clearly by rewriting the $J_{4,2}$ problem as
\[
 \begin{array}{ll}
 \underset{{\bf w}}{\mbox{maximize}}   &  
 \|{\bf w} * {\bf y}\|_4 
 \\
\mbox{subject to} 
& \|{\bf w}* {\bf y}\|_2 = 1.
\end{array}
  \]
The theory cited above derived favorable properties of a would-be procedure which truly finds the optimum of a badly nonconvex objective. It clarifies that  blind deconvolution is possible in principle but does not by itself help us algorithmically, i.e. in practice. In the intellectual climate of the time, solving badly nonconvex optimization problems was considered a pipe dream, a time-wasting charade for non-serious people.

Even \sout{up to} today, blind deconvolution 
continues to be studied as an non-convex optimization problem;
see recent work
\cite{kuo2019geometry, kuo2020geometry, lau2019short}, who study the \sout{that} problem of recovering short $a$ and sparse $x$ 
  
  \paragraph{Blind equalization}
  In digital communications, 
  the transmitted signal ${\bf x}$ can be viewed as 
  \redit{discrete-alphabet valued;}
  for example, in PAM signaling, where $x(t) \in \{\pm 1\}$, and QAM signaling, where the signal \redit{alphabet} has equally spaced points on unit sphere in complex space \cite{kennedy1992blind,ding2000fast}.  
  
  \cite{vembu1994convex} considered the non-convex problem
   \[
 \begin{array}{ll}
 \underset{{\bf w}}{\mbox{maximize}}   &  
 \|{\bf w} * {\bf y}\|_8 
 \\
\mbox{subject to} 
& \|{\bf w}* {\bf y}\|_2 = 1
\end{array}
  \]
If the data ${\bf y}$ were preprocessed to be serially uncorrelated,
this optimization is effectively of the earlier form $J_{8,2}$.

The authors attack this nonconvex problem
using projected gradient descent and give suggestive 
experimental results. They apparently view the $\ell_8$ norm objective as an approximation of the  $\ell_\infty$ norm objective $\|{\bf w} * {\bf y}\|_\infty $.
Later, \cite{ding2000fast} used linear programming to solve the $\infty$ norm problem directly.


\begin{comment}

\paragraph{Sparse projection pursuit and dictionary learning}
Sparse projection pursuit is the problem of finding  a sparse projection $d^T Y$ time from a high dimensional data matrix $Y\in {\mathbb R}^{n\times M}$. Here $Y= D X$ is a matrix product of dictionary $D\in {\mathbb R}^{n\times d}$ and sparse coefficient $X\in {\mathbb R}^{d\times M}$. 

\Dave{I can't follow either what you are trying to do in the paragraph below
or what you are trying to say. You mention many different things, with no references definitions or discussion and all very telegraphically.  Why is it even here? What does it add? I don't see it.}

\Dave{In general it's really a huge mistake to put 
inconclusive aimless material at the beginning of a document.
Readers very quickly abandon reading and decide that you're at least confusing
and possibly confused.

In your writing you don't want to put in a lot of extraneous references that 
only distract from your central thread.
}
The projection pursuit problem was first studied by Friedman and collaborators in a series of important works \cite{friedman1974projection}, \cite{friedman1981projection} and \cite{friedman1987exploratory} as a tool for exploratory high dimensional data analysis. Then several statisticians like Huber \cite{huber1985projection}  and Jones \cite{jones1987projection}  also studied this problem. 

The projection pursuit problem is also related to blind source separation, independent component analysis (ICA) and the cocktail-party problem.
In the case of an orthogonal dictionary $D$, we could simply transform the problem into finding its inverse $D^{-1} = D^T$, such that $D^T Y = X$ sparse. A work by Daniel Spielman and John Wright\cite{spielman2012exact} used sparse projection pursuit formulation to find the orthogonal basis of $D$ one-by-one, by proposing a set of rule to select the initial guess $v$ and give theoretical guarantee of the convex method. In this work, due to this common population problem, the phase transition phenomenon would also be applicable to dictionary learning, as the first phase transition result for dictionary learning.

The sparse projection pursuit formulation is to solve the following linear-constrained convex optimization problem for some vector $u$:
\[
 \begin{array}{ll}
 \underset{q }{\mbox{minimize}}   & \frac{1}{M}\|q^T Y\|_{1}\\
\mbox{subject to}  & u^T q =1.
\end{array}
  \]
  This convex problem is also used in \cite{spielman2012exact} to solve sparse dictionary learning problem.
  
\end{comment}

 
 \paragraph{Searching for a projection with desired output properties} 
Here is another setting for property-promoting
output optimization.
We have a data matrix $Y \in {\bf R}^{n,p}$ \-- which we think of as
$n$ points in ${\bf R}^p$,
and we have a unit vector $w \in {\bf R}^p$
called the projection direction.  Our
output vector $Y\cdot w$ contains the 
projection of the $n$-points on the projection
direction q; it has $n$ entries.
We seek `interesting' projections; i.e. directions where the 
projection displays some structure. We
adopt a functional $J$ which measures properties we seek to promote
in the output, and we seek to solve:
\begin{equation}
 \begin{array}{ll}
 \underset{\|w\|=1}{\mbox{optimize}}   & J(Y \cdot w). \label{JoptPP}
\end{array}
\end{equation}

This was implemented by  [Friedman and Tukey, 1974], who proposed a
functional that promotes `clumping' or 'clustering' of the output.
They called it {\it projection pursuit} and
were motivated by exploratory  high  dimensional  data  analysis;
for $p$-dimensional data involve $p>2$ and we can't 
easily get a visual sense of what's in the data. 
It was hoped at the time that looking at selected
low-dimensional projections might lead to better insights.
For the most part, such hopes for exploration of high-dimensional data
never materialized.

However, output optimization of this type has proven to be
useful in important problems in blind signal processing,
where $Y$ has known structure that can be exploited systematically.

In {\it blind source separation}
we observe $Y = XA$, $Y \in {\bf R}^{n,p}$, $X \in {\bf R}^{n,p}$ and
$A \in {\bf R}^{p,p}$ is an invertible matrix. 
(We don't observe $X$ or $A$ separately).

Think of the $X$ matrix as containing columns giving
successive time samples of $p$ clean source signals, for example $p$
individual acoustic signals. These signals arrive
at array of $p$ spatially distributed acoustic sensors, each
of which records the acoustic information it receives.
The matrix $Y$ contains in its
columns what was obtained by each of the $p$ different sensors.
In general, each sensor receives information from each of the sources.
This is colorfully called the {\it cocktail party problem}, referring to
the setting where $X$ records the sources are human speakers at a cocktail party,
and $Y$ records what is heard at various locations in a room.
Each column of $Y$ then contains a
superposition of different speakers; while
we would prefer to separate these and pay attention to 
just the ones of most immediate interest to us.

In this separation problem, sparsity of the signal might be valuable.
Suppose that each individual speaker is listening quite a bit
and so not speaking much of the time. Then the columns of $X$
are each sparse. On the other hand, if there are many people
in the room, the room
as a whole may still always be noisy, and so each
column of $Y$ may be fully dense.
Assuming $A$ is invertible, and the vector $w$
obeys $Aw= e_j$, then $Y \cdot w$ extracts column $j$ of $X$,
which will be sparse. Hence, we may hope that the projection
pursuit principle, with an appropriate measure $J$, might
identify the `sparse' projections we seek.  

Ju Sun, Qu Qing, Yu Bai, John Wright, Zibulevsky and Pearlmutter \cite{sun2015complete,sun2016complete,bai2018subgradient,zibulevsky2000blind} proposed the optimization
problem
 \[
 \begin{array}{ll}
 \underset{w }{\mbox{minimize}}   & \cancel{\frac{1}{n}}\| Y w\|_{1}\\
\mbox{subject to}  & \|w\|_2 =1.
\end{array}
  \]
Assuming the $Y$ data pre-processed so that $\frac{1}{n} Y'Y=I_p$ 
this is equivalent to projection pursuit (\ref{JoptPP}) 
applied to the objective $J_{2,1}$. Notably, this is 
again a highly non-convex optimization problem..
The authors proposed projected gradient descent
and recited some favorable empirical results.
 
There is a close relation between the
 blind deconvolution optimization (\ref{JoptBD}) 
 and the projection pursuit optimization (\ref{JoptPP}). Indeed, if the $Y$ matrix 
 is filled in from an observed time series appropriately, then 
 output optimization in \sout{the} 
 blind deconvolution and in projection pursuit are
 essentially identical. Namely, let ${\bf y}^{ser} = (y^{ser}(t))_{t=1}^N$ denote a time series of interest to us,
 and  $Y^{mat} = (Y^{mat}_{i,j})$ 
 denote an $n \times p$ matrix where $n = N-p$ constructed 
 using ${\bf y}^{ser}$ like so:
 \[
  Y^{mat}_{i,j} = y^{ser}(p+i-(j-1)) , \qquad 1 \leq i \leq n = N-p; \; 1 \leq j \leq p.
  \]
  Now suppose the filter vector 
  ${\bf w}$ in the blind deconvolution optimization 
  and the projection direction $w$ in the projection pursuit optimization
  are chosen identically. Then the blind deconvolution output objective 
  $J({\bf w} \star {\bf y}^{ser})$ is identical to the projection pursuit objective
  $J(Y^{mat} w)$, except for
  possibly different treatment of the first $p$ entries of ${\bf y}^{ser}$.
 
 \paragraph{Convex Projection Pursuit}
 
 In view of the connection between blind deconvolution and
 projection pursuit, and in view of our results in this paper,
 it is quite interesting to consider the
 work of \cite{spielman2012exact} and \cite{gottlieb2010matrix}.
\begin{comment}
They call their problem sparsity-promoting projection pursuit,
 but they depart from the PP optimization framework (\ref{JoptPP}).
 \Sun{Their paper did not mention the name "projection pursuit", so I comment out this sentence.}
\end{comment}
They propose to solve the following 
linear-constrained convex optimization problem.
Given an $n \times p$ 
data matrix $Y$ and a constraint vector $u$,
they propose to solve:
\[
 \begin{array}{ll}
 \underset{w }{\mbox{minimize}}   & \cancel{\frac{1}{n}}\|Y w\|_{1}\\
\mbox{subject to}  & u^T w =1.
\end{array}
  \]
As it turns out, \redit{when the matrix $Y$ in this problem and
${\bf y}$ in the time series deconvolution problem
are related by the} \sout{under the same} $Y^{mat}$-${\bf y}^{ser}$
construction just mentioned, the objective we propose in this paper
is essentially identical, when $\tilde{{\bf a}} = u$. \sout{However} \redit{As we will show},
our setting permits much more thorough studies and more penetrating analyses, \redit{and we find that success in 
sparse blind deconvolution is more
broadly prevalent than one might have expected, based on
earlier analyses such as \cite{spielman2012exact}
or \cite{sun2015complete,sun2016complete,bai2018subgradient,zibulevsky2000blind} }.


 


\subsection{Mathematical setup}
\paragraph{Sequence Space, and Filtering }
To make our results concrete, let's discuss things formally.
Let ${\mathbf X}$ denote the collection of {\it bilaterally infinite sequences}
${\bf x} = (x(t): t = 0, \pm 1, \pm 2, \dots)$; for short
we call such objects {\it bisequences}. Then $\ell_1({\mathbb Z}) \subset {\mathbf X}$
denotes the collection of bisequences 
obeying $\| {\bf x} \|_1 = \sum_t |x(t)| < \infty.$
For a bisequence ${\bf x}$ we denote time reversal operator
$t \leftrightarrow -t$ by ${\bf x}^{\dagger}$.
For whole number $k > 0$ let ${\mathbf X}_k$ denote the subspace
of bisequences supported in $-k \leq t \leq k$. We sometimes
abuse notation: for a bisequence ${\bf x}$ we might write
${\bf x} = (1, .3)$ when we really mean ${\bf x} = (\ldots,0,0,1,.3,0,0,\ldots)$.

Let $\star$ denote the convolution 
product on pairs of bisequences in $\ell_1({\mathbb Z})$
\-- $( {\bf x} \star {\bf y} ) = \sum_u x(t-u)y(u)$. 
Let ${\bf e}_0$ denote the  `delta'
or `Kronecker' bisequence: ${\bf e}_0(t) = 1_{\{t=0\}}$;
${\bf e}_0$ is the unit of convolution. The convolution inverse
of ${\bf x}$ $\--$ ${\bf x}^{-1}$ $\--$ is a bisequence
obeying $ {\bf x} \star {\bf x}^{-1}  = {\bf e}_0$. For example,
the filter ${\bf x} = (\dots,0,1,1/2,1/4,1/8,\ldots)$ 
anchored at the time origin so ${\bf x}(0)=1$, has inverse
${\bf x}^{-1} = (\dots,0,1,-1/2,0,\dots)$, again anchored at the time origin.
Abusing notation we may simply write ${\bf x}^{-1} = (1,-1/2)$.


Our approach to blind deconvolution searches 
among candidates for a filter ($\equiv$ bisequence) that extremizes a certain objective function. We then
show that the extremal is in fact the desired inverse filter
to our (unknown) true underlying filter. Hence, it helps know
conditions under which an inverse filter actually exists! 

For a bisequence ${\bf x} \in \ell_1({\mathbb Z})$,
we define the Fourier transform 
$\hat{{\bf x}}(w) = \mathcal{F} {\bf x} (w) \equiv \sum_t x(t) \exp\{ i 2\pi w t \}$.
For the inverse transform, we use
$ x(t) = (\mathcal{F}^{-1} \hat{{\bf x}} ) (t) =  \frac{1}{2\pi} \int_0^{2\pi} \hat{{\bf x}}(w) \exp\{ -i 2\pi w t \} dw$.


\begin{lemma}[Wiener's lemma]
If ${\bf a}\in \ell_1({\mathbb Z})$, and also  
$(\mathcal{F} {\bf a})(\widetilde{w}) \neq 0$, $\forall \widetilde{w}\in T$,
then an inverse filter exists in $\ell_1({\mathbb Z})$. 
The bilaterally infinite sequence defined formally by
$${\bf a}^{-1} := \mathcal{F}^{-1}(\frac{1}{\mathcal{F} {\bf a}})$$
exists as an element of $\ell_1({\mathbb Z})$
and obeys ${\bf a}^{-1}*{\bf a} = {\bf e}_0.$
\end{lemma}
In the engineering literature, we
say that the bisequence ${\bf a}$ has so-called $Z$-transform 
$A(z)$, defined by:
\[
 \begin{array}{ll}
A(z) =  \sum_{t=-\infty}^{\infty} a_{t} z^{-t}.
\end{array}
  \]

Evaluating $A$ on the unit circle in the complex plane, at $z$ of the form $z=\exp\{ -i 2\pi w \}$, we see that  the 
$Z$-transform is effectively the Fourier transform $A = {\cal F} {\bf a}$. 
Applying Wiener's lemma, we see that, if 
${\bf a} \in \ell_1$ and $ \min_w |A(\exp\{ - 2\pi i w \}) | > 0$,
i.e. $A$ is never zero on the unit circle, then ${\bf a}^{-1} \in \ell_1$ and $1/A$ is the $Z$-transform of ${\bf a}^{-1}$.


\paragraph{Finite-sample observation model and finite-length inverse filter }
In searching for an inverse filter, our \sout{early} initial  results assume
existence of a {\it finite-length} inverse. Namely, we assume that ${\bf a}\in \ell_1({\mathbb Z})$ is a forward filter with an inverse filter ${\bf a}^{-1} \in \ell_1({\mathbb Z})$ supported in a centered window of radius $k$.

In practice we only have a finite dataset! Suppose that 
there is an underlying bisequence ${\bf y} \in {\mathbf X}$
of the form ${\bf y} = {\bf a}* {\bf x}$, and let ${\bf y}^{[N]}$ denote the
restriction to an $N$-long centered window 
${\cal T} = \{ -T,\dots, T\}$ of radius $T$ and size $N=2T+1$.

Our goal is seemingly to 
recover ${\bf x}$ or ${\bf a}$ from the observed data ${\bf y}^{[N]}$.
However, in statistical theory we generally 
don't expect to exactly recover the true underlying 
representation (i.e. the generating ${\bf a}$ and ${\bf x}$) exactly.
Our goal is instead to find an inverse filter 
${\bf w} \in \ell_1^k$ such that the convolution ${\bf w}*{\bf y}$ 
would be close\footnote{modulo time shift and rescaling}  to ${\bf x}$, and where the closeness improves with
increasing data size $N \rightarrow \infty$.

\paragraph{Finite-length filtering and practical algorithms}

While our analysis framework concerns bisequences (bilaterally infinite sequences), our data have finite length (as just mentioned). The algorithms we discuss are often {\it motivated by} convolutions on bisequences; however, they reduce {\it in practice} to truncated convolutions involving finite data windows. A certain ambiguity is helpful for efficient communication. Suppose we have ${\bf y}^{[N]}$, an N-long observed window of ${\bf y}$, and we also have ${\bf w}$, a k-long filter, by which we mean a bisequence nonzero only within a 
fixed $k$-long window. We might encounter discussion both of 
${\bf w} \star {\bf y}^{[N]} $ as well as
${\bf w} \star {\bf y}$, both using the same filter ${\bf w}$. In the first case, we would actually be thinking of ${\bf y}^{[N]}$ as zero-padded out to a bisequence, so that both situations involve bisequence convolutions. 

Finite $N$ effects are important in practice but tedious to discuss. It can be important to account for {\it end effects} in truncated convolution. In a setting where we initially
think to consider a norm $\| {\bf w} \star {\bf y}^{[N]} \|_{\ell_p({\mathbb Z})}$, we might instead next think to rather consider the windowed norm
$\| {\bf w} \star {\bf y}^{[N]} \|_{\ell_p(\{-T,\dots,T\})}$,
while finally we realize $\| {\bf w} \star {\bf y}^{[N]} \|_{\ell_p(\{-T+k,\dots,T-k\})}$ is more correct for our purposes, as it includes only the terms which do not suffer from truncation of the convolution.



\paragraph{Algorithmic formulation}

Under assumptions we will be making, the sequence ${\bf x}$ underlying our observed data will be either exactly sparse -- having few nonzeros -- or approximately so. Moreover, there will be either an exact length $k$ inverse filter ${\bf w}$, or approximately such. It follows that the filter output ${\bf w} \star {\bf y}$ is sparse.
This suggests the would-be optimization principle 
\[
 \begin{array}{ll}
 \underset{{\bf w}\neq 0}{\mbox{minimize}}   & \|{\bf w} \star {\bf y}^{[N]}\|_{\ell_0(\{-T+k,\dots,T-k\}})
\end{array}
  \]
where the $\ell_0$ quasi-norm simply 
counts the number of nonzero entries. Unfortunately, this objective, though well-motivated, is not suitable for numerical optimization.

Inspired by this, we perform convex relaxation of the $\ell_0$ norm, replacing it with the $\ell_1$ norm, 
which is convex.

We also need to fix the scale to get a unique output. One might think to constrain $\|{\bf w}\|_2 =1$, however, this would give a non-convex constraint and again is not suitable for effective algorithms.



We instead suppose given a rough initial approximation  
$\widetilde{{\bf a}}$ of the forward filter ${\bf a}$, and impose 
an $\ell_\infty$ constraint on the 
`pseudo-delta' ${\bf w}\star\widetilde{{\bf a}}$,
forcing it to `peak' at target entry $t$.
\begin{comment}
\begin{equation}
 \begin{array}{ll}
& \|\widetilde{{\bf a}}*{\bf w}\|_\infty = 1.
\end{array}
\end{equation} 

This intuition for the choice is based on the fact that $a^{-1}*a = {\bf e}_0$, we know $(a^{-1}*a)_0 = 1$ is a constraint on the largest entry. Therefore, $\|\widetilde{{\bf a}}*a^{-1}\|_\infty = 1$ would be true if  $\widetilde{{\bf a}} = a$. 

Based on shift and scaling symmetry, without loss of generality, we could assume that   $\widetilde{{\bf a}}* a^{-1}$ 
has $0$-th entry being the largest absolute value entry by shifting $\widetilde{{\bf a}}$, and set its value to $1$ by scaling $\widetilde{{\bf a}}$, then up to shift and scaling of $\widetilde{{\bf a}}$, 
\[
(\widetilde{{\bf a}}*a^{-1})_0 = \|\widetilde{{\bf a}}*a^{-1}\|_\infty =1
\]


Although this transformed $\ell_\infty$ sphere constraint is still a non-convex constraint, we can focus on one surface by fixing the $t$-th coordinate of entry with largest magnitude:
\end{comment}
\begin{equation}
 \begin{array}{ll}
&  (\widetilde{{\bf a}}*{\bf w})_t =1, \quad \|\widetilde{{\bf a}}*{\bf w}\|_\infty \leq 1.
\end{array}
\end{equation}
Combining these steps, we obtain a convex optimization problem 
associated to each possible target coordinate $t$:
\begin{equation}
 \begin{array}{ll}
 \underset{{\bf w} \in \ell_1^k}{ \mbox{minimize} }   & \frac{1}{N-2k} \| {\bf w} \star {\bf y}^{[N]}\|_{\ell_1(\{-T+k,T-k\})}\\
\mbox{subject to}  
&  (\widetilde{{\bf a}}*{\bf w})_t =1, \quad \|\widetilde{{\bf a}}*{\bf w}\|_\infty \leq 1.
\end{array}
\end{equation}

\begin{comment}

\end{comment}In the case of perfect shift symmetry of convolution ( linear convolution on bisequence, or circular convolution for finite observation ), we could design the problem to fully utilize the shift symmetry of objective  $\frac{1}{N} \| w*Y\|_{\ell_1({\cal T})}$, so that the convex problem is equivalent to a convex optimization problem with a simple linear constraint as the $\ell_\infty$ box constraint 
with a fixed location of largest entry at $0$ coordinate under shift:
\begin{equation}
 \begin{array}{ll}
\widetilde{{\bf a}}^{T} w^{\dagger} = \langle \widetilde{{\bf a}}, w^{\dagger}\rangle = (\widetilde{{\bf a}}*w)_0  =1, 
\end{array}
\end{equation}
where $w^{\dagger}$ is the time reversal of $w$.
\end{comment}

It will be convenient to reformulate slightly,  
hide consideration of end effects,
and force the peak to occur at target coordinate $t=0$.
Abusing notation somewhat, we then write:
\begin{equation}
 \begin{array}{ll}
 \underset{{\bf w} \in \ell_1^k}{ \mbox{minimize} }   & \frac{1}{N} \| {\bf w}* {\bf y} \|_{\ell_1^N}\\
\mbox{subject to}  
& \langle \widetilde{{\bf a}},  {\bf w}^{\dagger} \rangle =1, 
\end{array}
\tag{${C}_{\widetilde{{\bf a}}}$ }
\end{equation}

(Again ${\bf w}^\dagger$ denotes time-reversal of ${\bf w}$). In practice we
might truncate the convolution due to end effects, or truncate the window over which we take the norm, but we will hide such practical details in the coming material, for ease of exposition; they would not change our results. 

 
\paragraph{Stochastic models for sparse signals}
Although our algorithms make sense in the absence of any theory,
our theoretical results concern properties of our algorithm for data generated under a probabilistic generative model i.e. a {\it stochastic signal} model. 

Let $X = (X_t)$ be a bisequence
of independent identically distributed random variables indexed by $t \in {\mathbb Z}$, having a common marginal CDF $F = F_X$, such that $F(x) = 1 - F(-x)$. One realization is then a sequence ${\bf x}$ of the type discussed in earlier paragraphs.

We assume that $F$ has an atom at $0$ \-- $F = (1-p) H +p G$, where $H$ is the standard Heaviside distribution and $G$ is the standard Gaussian distribution.  We say that $F$ follows the \emph{Bernoulli$(p)$-Gaussian} model. 
Equivalently, $X_t$ is sampled IID from the \emph{Bernoulli$(p)$-Gaussian} distribution
$p N(0,1) +(1-p) \delta_0$. 

The iid process $X$ is of course ergodic. 
If ${\bf x}$ denotes one realization of $X$, then in a window $(x_t)_{-T}^T$ of length $N$,  $\approx p N$ nonzero values  will occur,
for large $N$. Consequently, if $p \ll 1$,
realizations from $X$
will empirically be sparse.

\newcommand{{\bf b}}{{\bf b}}
Let $Y = {\bf a} \star X$ denote the random bisequence produced as the output of convolution of the random signal $X$
with deterministic filter ${\bf a} \in \ell_1({\mathbb Z})$. More explicitly,
\[
    Y_t = \sum_u a(u) X_{t-u}.
\]
This defines formally a so-called {\it stationary linear process},  a classical object for which careful foundational results are well established.\footnote{In our case, we assume that $X$ is iid Bernoulli-Gaussian, so
$E |X_0| = p \cdot E |N(0,1)|$ is finite. Using this,
we can see that the sum in the display above, 
even though possibly containing an 
infinite number of terms, converges in various natural senses.}
Consider now filtering $Y$, by a length-$k$ filter 
${\bf w}$, producing the random bisequence $V = {\bf w} \star Y$.
The {\it end-to-end} filter ${\bf b} = {\bf w} \star {\bf a}$ 
is a well-defined element of $\ell_1({\mathbb Z})$; 
using it, we can represent 
the filtered output in terms of the underlying iid process $X$:
$V  = {\bf b} \star X$. 
This representation shows that the filtered output series $V$ is itself a well-defined stationary linear process, and moreover, since $E |X_0| < 1$ and $\| {\bf b} \|_{\ell_1} \leq \|{\bf w} \|_{\ell_1} \cdot  \| {\bf a} \|_{\ell_1} < \infty$, we have $E | V_0| < \| {\bf b} \|_{\ell_1} < \infty$.

Any such stationary linear process is ergodic.
By the ergodic theorem, the
large-$N$ limit of the objective 
will \redit{almost surely}  be an expectation over $X$:
\begin{equation}
 \begin{array}{ll}
\lim_{N\rightarrow \infty}\frac{1}{N} \| {\bf w}*Y\|_{\ell_1^N}  = \lim_{N\rightarrow \infty}\frac{1}{N} \sum_{t\in {\cal T}^N}|({\bf b} \star X)_t|  =  {\mathbb E}_X|({\bf b} \star X)_0|={\mathbb E}|({\bf w}*Y)_0|.
\end{array}
\end{equation}
\sout{Evidently} \redit{Consequently}, 
the large-$N$ properties of our proposed algorithm
are driven by properties of the following optimization problem
{\it in the population}:
 \[
 \begin{array}{ll}
 \underset{{\bf w} }{\mbox{minimize}}   & {\mathbb E}|({\bf w}*Y)_0|\\
\mbox{subject to}  &  \langle \widetilde{{\bf a}},  {\bf w}^{\dagger} \rangle =1. 
\end{array}
  \]

  
  




\section{Main Results Overview}











\subsection{Main Result $1$: Phase Transition Phenomenon for Sparse Blind Deconvolution}

We have at last defined a convex optimization problem 
at the population level, which we will now want to study in detail.
In our studies, we can make various choices of the sparsity parameter $p$, of the underlying forward filter ${\bf a}$, and of the guess $\tilde{{\bf a}}$. The tuple $(p,{\bf a},\tilde{{\bf a}})$ defines in this way a kind of phase space. We can then study performance of the algorithm at different points in phase space. 

\newcommand{{\sc Exact}{\sc Recovery}}{{\sc Exact}{\sc Recovery}}
Consider this performance property: 
\begin{quotation}
{\sc Exact}{\sc Recovery} \, $\equiv$ \,  {\sl ``There is an unique solution of the population-based optimization problem --- modulo time shift and output rescaling --- and
this solution {\bf exactly} solves the blind deconvolution problem correctly. ''}
\end{quotation}
It probably seems too much to ask that such a property could {\sl ever} be true, i.e. could ever be true for even one 
choice of phase space tuple. After all the optimization problem doesn't have any apparent connection to blind deconvolution -- instead only to some sort of relaxation of the search for sparse output filters.


We will see that this phase space can be partitioned into two regions: one where the exact recovery property holds and its complement where the exact recovery property fails. Surprisingly the region where {\sc Exact}{\sc Recovery} \  holds is nonempty and can be appreciable. And it can be described in a clear and insightful way.

\paragraph{Surprising phase transition for a special case of sparse blind deconvolution}



 \sout{To demonstrate the flavor of the} \redit{We first illustrate the phase transition phenomenon in possibly the most elementary situation}. \sout{we present the population phase transition theorem for} \redit{Consider the special case when} the forward filter ${\bf a}$ is the exponential decay filter:  ${\bf a} = (1, s, s^2, s^3, \ldots )$ with $|s|\leq 1$; then the inverse filter is a basic short filter  ${\bf a}^{-1} = ( 1, -s )$. 
\begin{theorem}[Population phase transition for exponential decay filter]
\label{thm: single root PT}
Consider a linear process $Y = {\bf a} * X$ with
\begin{itemize}
\item
${\bf a} = (1, s, s^2, s^3, \ldots )$ where $|s|\leq 1$, so ${\bf a}^{-1} = ( 1, -s )$;
\item 
$X_t$ is IID \emph{Bernoulli$(p)$-Gaussian}
$p N(0,1) +(1-p) \delta_0$.
\end{itemize}
Consider as initial approximation 
$\widetilde{{\bf a}} = {\bf e}_0 = ( 1, 0, 0,0, \ldots)$,
and the resulting 
fully specified population
optimization problem, 
with parameter tuple $(p,{\bf a},\tilde{{\bf a}})$:
 \[
 \begin{array}{ll}
 \underset{{\bf w} }{\mbox{minimize}}   & {\mathbb E}|({\bf w}*Y)_0|\\
\mbox{subject to}  & {\bf w}_0 =1.
\end{array}
\tag{${P}_1(e_0)$ }
\label{bd:population}
  \]
Let ${\bf w}^\star$ denote the (or simply some) solution.
Define the threshold
\[
p^\star  = 1-|s| .
\]
The property {\sc Exact}{\sc Recovery} experiences a phase transition at $p = p^\star$:
\begin{itemize}
    \item provided $p<p^\star$, then ${\bf w}^\star$ is uniquely defined and equal to ${\bf a}^{-1}$ up to shift and scaling ; and
    \item provided $p> p^\star$, then ${\bf w}^\star$ is not ${\bf a}^{-1}$ up to shift and scaling .
\end{itemize}
  \end{theorem}
  
  
  \begin{figure}[!tb]
\centering
\includegraphics[width=.8\textwidth]{./figures_BD/PT_BD_1root_none.png}
\caption{Finite-sample phase transition diagram; ground truth filter is ${\bf w}^\star =  ( 0, 1, -s )$ with $T=200$. Horizontal axis:  sparsity level $p$, ranging from $0.01$ to $0.99$; Vertical axis:  filter parameter $|s|$, ranging from $0$ to $0.99$.  The \redit{shaded attribute indicates the observed fraction of  
successful experiments at the given parameter combination from deep blue for 1.0 down to deep red for 0.0. Thus,} the red region indicates failure of recovery and the blue region indicates success. \redit{The transition from success to failure is quite abrupt.}}
\label{fig-PT-finite}
\end{figure}

We can empirically \sout{verify} observe that the population phase transition described in Theorem 1 
describes accurately the situation with finite-length signals.  \sout{by discretizing the} \redit{The setting of Theorem 1,
defines a phase space by the numbers $(p,s)$. We can
sample the phase space according to} a grid and then at each grid point, conducting a sequence of experiments like so:
\begin{itemize}
\item sample a realization of synthetic data $Y = {\bf a} \star X$ according to the stochastic signal;
\item extract a window  ${\bf y}^{[N]}$ of size $N$ from within 
each generated $Y$; and
\item solve the resulting finite-$N$ optimization problem:
\[
 \begin{array}{ll}
 \underset{{\bf w} }{\mbox{minimize}}   & \frac{1}{N}\|{\bf w}*Y\|_1\\
\mbox{subject to}  & {\bf w}_0 =1.
\end{array}
\tag{${P}^N_1(e_0)$ }
\label{bd:finite}
  \]
\end{itemize}
Tabulating the fraction of instances with numerically precise recovery of the correct underlying inverse filter ${\bf a}^{-1}$ 
and sparse signal $X$ across
grid points, we can make a heatmap of empirical success probability.

Specifically, we numerically run $20$ independent experiments, and choose an accuracy $\epsilon = 10^{-3}$ and count the number of successfully accurate recovery where the convex optimization solution $w$ satisfy $\|w^\star -w\|_1/\|w^\star\|_\infty<\epsilon$.

We do this in Figure~\ref{fig-PT-finite}; the reader will see there an empirical phase transition curve, produced by a logistic-regression calculation of the location in $p$ where $50\%$ success probability is achieved. 
We observe empirical behavior entirely consistent 
with $p^\star  = 1-|s|$.






\paragraph{Phase transition for sparse blind deconvolution with general filter}
Now we present the main phase transition theorem for blind deconvolution of general inverse filter.

To state the theorem,
define the operator $'$ that 
sets the largest entry in a bisequence to zero,
while preserving the other entries. Operating on bisequence ${\bf v}$
it yields
 \[
 {\bf v}' \equiv {\bf v} - v_{t_m}{\bf e}_{i_m}
 \]
 where $t_m = \arg\max_t |v(t)|$;  if there are multiple largest entries, $t_m$ is the smallest such index.
Also, for, $k \in \{ 1, 2, 3 , \dots \}$, 
let $|{\bf v}|_{(k)}$ denote
the $k$-th largest amplitude entry in bisequence 
${\bf v}$.

In addition, let $I$ denote an iid Bernoulli(p) bisequence.
For a bisequence ${\bf w}$ let $ {\bf w} \cdot I$ denote the 
elementwise multiplication
of ${\bf w}$ by $I$.
Define the optimization problem
\[
 \begin{array}{ll}
 \underset{{\bf w}}{\mbox{minimize}}   & {\mathbb E}_I \|{\bf w} \cdot I\|_2\\
\mbox{subject to}  
& \langle   {\bf v}, {\bf w}\rangle =1
\end{array}
\tag{$Q_1({\bf v})$ }
  \]


	\begin{theorem}[Population (large-$N$) phase transition]
	\label{thm:Population phase transition}
Consider  a linear process $Y = {\bf a} \star X$ where
\begin{itemize}
\item
 $X = (X_t)_{t \in {\mathbb Z}}$ is IID with marginal distribution  $pN(0,1) + (1-p) \delta_0$; and
 \item
${\bf a}\in l_1({\mathbb Z})$ is invertible:  $(\mathcal{F} {\bf a})(\widetilde{w}) \neq 0$, $\forall \widetilde{w}\in T$; thus ${\bf a}^{-1}$ exists in $\ell_1({\mathbb Z})$.
\end{itemize}

Consider the convex optimization problem 
 \[
 \begin{array}{ll}
 \underset{{\bf w} \in l_1({\mathbb Z})}{\mbox{minimize}}   & {\mathbb E}|({\bf w}*Y)_0|\\
\mbox{subject to}  
& \langle \widetilde{{\bf a}},  {\bf w}^{\dagger} \rangle =1,
\end{array}
\tag{${P}_1(\widetilde{{\bf a}})$ }
\label{eqn:P1}
  \]
and let ${\bf w}^\star$ denote any solution of the optimization problem. 

There is a threshold $p^\star > 0$,
\begin{itemize}
\item  ${\bf w}^\star$ is ${\bf a}^{-1}$ up to time shift and rescaling provided $p<p^\star$; and
\item   ${\bf w}^\star$ is not ${\bf a}^{-1}$ up to time shift and rescaling, provided $p> p^\star$.
\end{itemize}
Define $\widetilde{{\bf e}}:= \widetilde{{\bf a}}*{\bf a}^{-1}$.
The threshold $p^\star$ obeys
 \[
\frac{p}{1-p}= val(Q_1(\widetilde{{\bf e}}')).
\] 
We have an upper bound and lower bound
\[
 \frac{p}{\|\widetilde{{\bf e}}'\|_\infty}  \geq val(Q_1(\widetilde{{\bf e}}')) \geq  p \cot\angle(\widetilde{{\bf e}}, {\bf e}_0) 
\]
explicitly, the upper bound can be expressed as
\[
1- \cot\angle(\widetilde{{\bf e}}, {\bf e}_0)   \leq p^\star \leq 1- \frac{|\widetilde{{\bf e}}|_{(2)}}{|\widetilde{{\bf e}}|_{(1)}}
\]
Additionally, the upper bound 
takes equality if
\[
	\frac{|\widetilde{{\bf e}}|_{(3)}}{|\widetilde{{\bf e}}|_{(2)}} \leq \frac{|\widetilde{{\bf e}}|_{(2)}}{|\widetilde{{\bf e}}|_{(1)}}
	\]
Here \redit{the multiplicative gap} $\frac{|\widetilde{{\bf e}}|_{(2)}}{|\widetilde{{\bf e}}|_{(1)}}$ denotes 
the ratio of the  first and second largest entries 
of $|\widetilde{{\bf e}}|$,and is a natural measure of closeness 
between our approximate Kronecker delta $\widetilde{{\bf e}}=\widetilde{{\bf a}} * {\bf a}^{-1}$ 
and true Kronecker delta  ${\bf e}_0$. In words {\it the less accurate the initial approximation $\widetilde{{\bf a}} \approx {\bf a}$, the greater we rely on sparsity of $X$ \redit{to allow exact recovery}.}

	\end{theorem}

\Dave{This result fails to deliver on the reader's investment of time to this point. We are on page 14. The reader gets something quite non-tangible (value of an optimization problem which itself requires deciphering) and then all sorts of ancillary details that only add to the deciphering. It would be better to state the most insightful and easily appreciated result first, otherwise essentially no-one will continue reading beyond this moment.}


\Dave{Also spotlighting this way of looking at the problem pigeon holes the paper as providing detailed insights of very limited generality and unlikely connection to problems most readers could care about.}










\paragraph{Alternate: Phase transition for sparse blind deconvolution with general filter}
\redit{Consider now a more general situation where ${\bf a}$ is a quite general filter, and $X$ is Bernoulli Gaussian. 
Our formal assumptions are:}

\redit{\begin{enumerate}
 \item  [{\bf A1}]
${\bf a}\in l_1({\mathbb Z})$ is invertible:  $(\mathcal{F} {\bf a})(\widetilde{w}) \neq 0$, $\forall \widetilde{w}\in T$; thus ${\bf a}^{-1}$ exists in $\ell_1({\mathbb Z})$.
\item [{\bf A2}] $X = (X_t)_{t \in {\mathbb Z}}$ is IID with marginal distribution  $pN(0,1) + (1-p) \delta_0$; and
\item [{\bf A3}] $Y = (Y_t)_{t \in {\mathbb Z}}$ is a linear process
obeying $Y = {\bf a} \star X$.
\end{enumerate}}

\redit{Consider the convex optimization problem 
 \[
 \begin{array}{ll}
 \underset{{\bf w} \in l_1({\mathbb Z})}{\mbox{minimize}}   & {\mathbb E}|({\bf w}*Y)_0|\\
\mbox{subject to}  
& \langle \widetilde{{\bf a}},  {\bf w}^{\dagger} \rangle =1,
\end{array}
\tag{${P}_1(\widetilde{{\bf a}})$ }
\label{eqn:P1}
  \]
and let ${\bf w}^\star$ denote any solution of the optimization problem. }

\newcommand{\widetilde{{\bf a}}}{\widetilde{{\bf a}}}
\newcommand{\mbox{Gap}}{\mbox{Gap}}
\newcommand{l_1({\mathbb Z})}{l_1({\mathbb Z})}
\newcommand{l_\infty({\mathbb Z})}{l_\infty({\mathbb Z})}
\redit{Our results establish the existence of a general
phase transition phenomenon and a precise quantification of it,
in terms of a {\it phase transition functional} .}

\redit{Define the following {\bf deltaness discrepancy} $\Delta({\bf v})$:
\[
 \Delta({\bf v}) =  \inf_{\alpha,k} \frac{\| {\bf v} - \alpha \delta_k \|_\infty}{\|{\bf v}\|_\infty}.
\]
}

\redit{This discrepancy  indeed obeys
$\Delta({\bf e}_0) = 0$. However, it is not restricted only to
deltaness concentrating at the origin; in fact,
$\Delta( \alpha \cdot {\bf e}_k) = 0$ for
each $\alpha > 0$, $k \in {\mathbb Z}$.
Also, $0 \leq  \Delta({\bf v}) \leq 1$.
At the other extreme,
$\Delta([...,0,0,1/2,1/2,0,0,...]) = 1$. 
Finally,
$\Delta$ is a continuous function on $l_1({\mathbb Z})$:
so that, 
setting ${\varepsilon}({\bf v},{\bf w}) \equiv \frac{\|v-w\|_\infty}{\|w\|_\infty}$,
\[
   | \Delta({\bf v}) - \Delta({\bf w}) | \leq \frac{2 \cdot {\varepsilon}}{1-{\varepsilon}}.
\]
}
	\begin{coro}
	[\redit{Alternative Population (large-$N$) phase transition}]
	\label{coro:AltPopulationPhaseTransition}

\redit{Under assumptions (A1)-(A3),}
\begin{itemize}
\item \redit{There is a functional
$\Pi^*$ defined on tuples $({\bf a},\widetilde{{\bf a}})$ in 
$\ell_1({\mathbb Z}) \times \ell_1({\mathbb Z})$
with the property that, for $p > \Pi^*({\bf a},\widetilde{{\bf a}})$,
every solution of the optimization problem ${P}_1(\widetilde{{\bf a}})$
is exactly the correct answer ${\bf a}$, up to lateral shift and scaling.}

\item \redit{We have the upper bound:
\begin{equation}
    \label{eq:coro-upperbnd}
\Pi^*({\bf a},\widetilde{{\bf a}}) \leq 1-\Delta(\widetilde{{\bf a}}*{\bf a}^{-1}), \qquad ({\bf a},\widetilde{{\bf a}}) \in \ell_1({\mathbb Z}) \times \ell_1({\mathbb Z}).
\end{equation}
}


\end{itemize}
\end{coro}


\redit{The reader might compare this with our earlier example
in Theorem \ref{thm: single root PT}, the special case of
geometrically decaying filters. 
In that example ${\bf a} = (\dots,0,0,1,s,s^2,\dots)$ while
$\widetilde{{\bf a}} = (\dots,0,0,1,0,0,\dots)$. $\Delta(\widetilde{{\bf a}}*{\bf a}^{-1})= \Delta((\dots,0,0,1,s,s^2,\dots)) = |s|$. In short, the upper bound $\Pi^* = 1- |s|$ deriving from this general viewpoint agrees precisely with the the exact answer $p^* = 1- |s|$ given in that earlier Theorem. }

\redit{According to this corollary that there is a substantial
region in the space of $({\bf a},\widetilde{{\bf a}})$ pairs where a meaningful 
sparsity-accuracy of initialization tradeoff exists, such that
 any sufficiently accurate initial guess results in
exact recovery \-- provided that the sparsity of the underlying object $X$ exceeds a threshold.}

\begin{coro}
\redit{Let $r \in (0,1/2)$ and suppose 
that assumptions (A1)-(A3)
hold with $p > \frac{r}{1-r}$. 
Normalize the problem so that 
$\| {\bf a}^{-1} \|_\infty =1$.}

\redit{Consider $\widetilde{{\bf a}}$ in the $\ell_1$ metric ball
(\ref{eq:lonemetricball}) of radius $r$
about ${\bf a}$:
\begin{equation} \label{eq:lonemetricball}
    \| \widetilde{{\bf a}} - {\bf a} \|_1 \leq r .
\end{equation}
}
\redit{Every solution of the optimization problem $P_1(\widetilde{{\bf a}})$ 
achieves exact recovery of ${\bf a}$ up to shift and rescaling.}
\end{coro}

\redit{{\bf Proof:} Indeed, we can give a more explicit form
for $\Delta$; 
\[
 \Delta(\widetilde{{\bf e}}) = \frac{|\widetilde{{\bf e}}|_{(2)}}{|\widetilde{{\bf e}}|_{(1)}},
\]
where $|\widetilde{{\bf e}}|_{(1)}$ denotes the largest entry
in $\widetilde{{\bf e}}$ and $|\widetilde{{\bf e}}|_{(2)}$ the second largest.
$\frac{|\widetilde{{\bf e}}|_{(2)}}{|\widetilde{{\bf e}}|_{(1)}}$ 
is a kind of multiplicative gap functional,
measuring the extent to which the second-largest entry in $\widetilde{{\bf e}}$
is small compared to the largest entry.}

\redit{Since 
\[
  \| \widetilde{{\bf e}} - {\bf e}_0\|_\infty \leq \| {\bf a}^{-1} \|_\infty \cdot \| \widetilde{{\bf a}}-{\bf a}\|_1 = r,
\]
we have $|\widetilde{{\bf e}}|_{(1)} \geq 1-r$ and $|\widetilde{{\bf e}}|_{(2)} \leq r.$ This 
implies that $\Pi^* \leq \frac{r}{1-r}$. Now apply Corollary
\ref{coro:AltPopulationPhaseTransition}.
\QED
}


When does equality hold in (\ref{eq:coro-upperbnd})?
For a vector ${\bf v}$, let ${\bf v}'$ denote the same vector,
except the largest-amplitude  entry is replaced by $0$. We will see
that a sufficient condition for equality is:
\[
   \Delta(\widetilde{{\bf e}}') \geq \Delta(\widetilde{{\bf e}}).
\]
Put another way, equality holds if 
\begin{equation} \label{eq:suffcondsharp}
|\widetilde{{\bf e}}|_{(3)} \leq  (1-\Delta(\widetilde{{\bf e}})) \cdot |\widetilde{{\bf e}}|_{(2)}.
\end{equation}
The set of situations where this occurs is ample but not overwhelming.
It has relative
Lebesgue measure at least  $1-\Delta$. In the situation covered by  Theorem \ref{thm: single root PT},
$|\widetilde{{\bf e}}|_{(3)}  = s^2$, $ |\widetilde{{\bf e}}|_{(2)} = s$,
$1- \Delta = s$, and so equality holds in (\ref{eq:suffcondsharp}).
Hence, the general-filter result of Corollary 
\ref{coro:AltPopulationPhaseTransition} along with the sufficient condition (\ref{eq:suffcondsharp})
imply as a special case
Theorem \ref{thm: single root PT}.






\subsection{Main Result $2$: Finite Observation Window, Finite-length Inverse}

With a finite observation window of length $N$ we (surprisingly) still can have exact recovery, starting as soon as $N \geq \Omega(k\log(k))$.

\paragraph{Finite-observation phase transition}


Let 
$\ell_1^k$
denote the collection of bisequences vanishing outside a centered window of radius $k$.

	\begin{theorem}[Finite observation window, finite-length inverse filter]
	\label{thm:fin_guarantee}
	Suppose $Y = {\bf a} \star X$, where:
	\begin{enumerate}
 \item  [{\bf A1}]
${\bf a}\in l_1({\mathbb Z})$ is invertible and the inverse has finite-length:  $(\mathcal{F} {\bf a})(\widetilde{w}) \neq 0$, $\forall \widetilde{w}\in T$; thus ${\bf a}^{-1}$ exists in $\ell_1({\mathbb Z})$; furthermore, ${\bf a}^{-1} \in \ell_1^k$ vanishes off a centered window of length $k$.
\item [{\bf A2}] $X = (X_t)_{t \in {\mathbb Z}}$ is IID with marginal distribution  $pN(0,1) + (1-p) \delta_0$; 
\item [{\bf A3}] $Y = (Y_t)_{t \in {\mathbb Z}}$ is a linear process
obeying $Y = {\bf a} \star X$. Suppose we observe a window $(Y_t)_{t\in {\cal T}}$ of length $N = |{\cal T}|$. 
\end{enumerate}


Consider the convex optimization problem
\begin{equation}
 \begin{array}{ll}
 \underset{{\bf w}\in \ell_1^k}{  \mbox{minimize} }   & \frac{1}{N}\|{\bf w}*Y\|_{\ell_1^N}\\
 \mbox{subject to}  
& (\widetilde{{\bf a}}* {\bf w})_0 =1.
\end{array}
\tag{${P}_{1}^{N,k}(\widetilde{{\bf a}})$}
\label{P1_finite}
\end{equation}
and let ${\bf w}^\star$  denote any solution of the optimization problem.

	Our result establishes that there exist $\epsilon>0, \delta>0$, so that when the number of observations $N$ satisfies
		\[
N \geq  k\log(\frac{k}{\delta}) ( \frac{C\kappa_{\bf a}}{\epsilon}  )^2,
\]
and the sparsity level $p$ obeys $\frac{1}{N}\leq p < p^\star - \delta_p(N,\epsilon)$,
		then 	with probability exceeding $1-\delta$, ${\bf w}^\star$   is ${\bf a}^{-1}$ up to rescaling and time shift. In the statement, $\kappa_{\bf a}$ is the condition number of the circular matrix with its first column being ${\bf a}$; $C$ is a positive constant independent of $N$ and $k$.
	\end{theorem}
	
 


\subsection{Main Result $3$: Stability Guarantee with Finite Length Inverse Filter}
The results so far concern the ideal setting when true inverse filter has a known length $k$ and we use $k$ to set up a correctly matched optimization problem. In practice we do not know $k$ and $k$ might even be infinite.

We can provide practical guarantees even when the inverse filter is an infinite length inverse. To develop these, we must be more technical about the situation. We assume that ${\bf a}$ has a Z-transform having $N_{-}$ roots and $N_{+}$ poles $(s_{i})$ inside the unit circle and we construct a finite length approximation ${\bf w}$ to  ${\bf a}^{-1}$, in fact of length
$r(N_{-}+N_{+})$.
This approximation has error 
$\|{\bf w}*{\bf a}-{\bf e}_0\|_2= O(\max_i |s|_{i}^{r}  )$. 

Since the objective value ${\mathbb E}_I\|{\bf w}\cdot I\|_2$ of this approximation ${\bf w}$ is an upper bound of the optimal value of the optimization solution ${\bf w}^\star$, we could use the objective value upper bound  to derive a upper bound for $\|{\bf w}^\star*{\bf a}-{\bf e}_0\|_2= O(
\max_i |s|_{i}^{r}  )$ when $p<p^\star$, where the constant of this upper bound is determined by the Bi-Lipschitz constant of the finite difference of objective ${\mathbb E}_I\|{\bf w}\cdot I\|_2-{\mathbb E}_I\|({\bf e}_0)\cdot I\|_2$.

\paragraph{Approximation theory for infinite length inverse filter}
\begin{theorem}[Approximation theory for infinite length inverse filter based on roots of Z-transform]
\label{thm:RootNorm}
Let the finite-length forward filter ${\bf a}$ have a Z-transform with roots inside the unit circle, namely $s_{k}:= e^{-\rho_k + i \varphi_k }$ with $|s_k|<1$ and $\rho_k>0$
for $k \in \{-N_{-}, \ldots -1, 1, \ldots N_{+}\}$.  Let ${\cal I} = \{-N_{-},\ldots, -1, 1, \ldots, N_{+} \}$ as the set of all the possible indexes.
\[
 \begin{array}{ll}
A(z) &=  \sum_{i=-N_{-}}^{N_{+}} a_{i} z^{-i}
\\
&= c_0\prod_{j=1}^{N_{-}} (1- s_{-j} z) \prod_{i=1}^{N_{+}} (1- s_{i} z^{-1});
\end{array}
  \]
here $c_0$ is a constant ensuring $a_0=1$.

Then for a scalar $r$
, we construct an approximate inverse filter ${\bf w}^r$ with Z-transform
\[
 \begin{array}{ll}
W(z) 
&=  \frac{1}{c_0}\prod_{j=1}^{N_{-}} (\sum_{\ell_j=0}^{r-1} s^{\ell_j}_{-j} z^{-\ell_j} ) \prod_{i=1}^{N_{+}} (\sum_{\ell_i=0}^{r-1} s^{\ell_i}_{i} z^{\ell_i} )\\
&=  \frac{1}{c_0}\prod_{j=1}^{N_{-}} (1- (s_{-j} z)^{r})(1- s_{-j} z)^{-1} \prod_{i=1}^{N_{+}}(1- (s_{i}  z^{-1})^{r}) (1- s_{i} z^{-1})^{-1}.
\end{array}
\]
We have
\[
 \begin{array}{ll}
 \|{\bf w}^r*{\bf a}-{\bf e}_0\|_2^2  
 &= \sum_{n\in \{1,2,3,\ldots,|{\cal I}| \} }\sum_{ k_1, \ldots, k_n  \in {\cal I} } \prod_{i\in [n]}|s_{k_i}|^{2r }
\end{array}
  \]
as $r \rightarrow \infty$, this converges to zero at an exponential rate, determined by the slowest decaying term,
\[
 \begin{array}{ll}
\|{\bf w}^r*{\bf a}-{\bf e}_0\|_2
&=  O( |s|_{(1)}^{r}  ) , \quad r \rightarrow \infty.
\end{array}
  \]
  where $|s|_{(1)}$ is the largest absolute value root.
 \end{theorem}
 
 
 \paragraph{Stability for infinite length inverse filter}
 \begin{theorem}[Stability for infinite length inverse filter]
 \label{thm:bd_stab}
 Let ${\bf a}\in V_{N_-, N_+} $ be a forward filter with all the roots of Z-transform strictly in the unit circle. Let ${\bf w}^\star \in V_{(r-1)N_-, (r-1)N_+} $ be the solution of the convex optimization problem. Let ${\bf w}^r$ be the constructed filter in previous theorem with a uniform vector index $(r, \ldots, r, r,\ldots, r)$.
  Then provided $p<p^\star$, as 
  $r \rightarrow \infty$, 
  \[
 \begin{array}{ll}
  \| {\bf w}^\star*{\bf a} -{\bf e}_0\|_2 
   \leq
    O(
 |s|_{(1)}^{r}  ), \quad r \rightarrow \infty.
\end{array}
  \]
  where $|s|_{(1)}$ is the largest absolute value root.
In words, \textbf{the Euclidean distance between ${\bf w}^\star*{\bf a}$ and ${\bf e}_0$ converges to zero at an exponential rate as the approximation length is allowed to increase.} 
 \end{theorem} 
 
 
 
\subsection{Main Result $4$: Robustness Against Stochastic Noise and Adversarial Noise}
We now extend the previous analysis of an exactly sparse model of $X$, exactly observed, to the more practical setting of approximate sparsity and observation noise. We consider two cases: first we add stochastic noise as an independent Gaussian linear process, and second adversarial noise with bounded $\ell_\infty$ norm.  

In each scenario, since the noisy objective value at ${\bf e}_0$ is an upper bound on the optimal value of the optimization solution ${\bf w}^\star$, we use this upper bound on the objective value  to derive an upper bound of $\|{\bf w}^\star*{\bf a}-{\bf e}_0\|_2$ when $p<p^\star$.
The upper bound shows that the distance $\|{\bf w}^\star*{\bf a}-{\bf e}_0\|_2$ is bounded by the (appropriately measured)
magnitude of input noise in both cases.


\paragraph{Robustness under stochastic noise }
 \begin{theorem}[Robustness against Gaussian Linear Process noise]
 \label{thm:rand_exact_rob}
 We consider a Gaussian linear process $Z = \sigma \cdot {\bf b} \star G$ where: $\sigma > 0$ denotes the noise level;   ${\bf b}$ is
 a bisequence having  unit $\ell_2$ norm $\|{\bf b}\|_2=1$; and $G$ is a standard Normal iid bisequence:
 \[
\begin{array}{ll}
Y&= {\bf a}* (X+ Z).
\end{array}
\]
 
 \newcommand{{\bf w}}{{\bf w}}
 Let ${\bf w}^\star$ be the solution of the convex optimization problem in Eq.(4.1);


\label{thm:Bound_Normal_Noise}
\[
 {\mathbb E}_I  \| ({\bf w}^\star)\cdot I \|_2 -{\mathbb E}_I  \| ({\bf e}_0)\cdot I \|_2 \leq  (1-p) \sigma +
 p(\sqrt{1+ \sigma^2 }-1).
\]
When $p<p^\star$, $\sigma\leq 1$, there exists a constant $C$,
\[
  \| {\bf w}^\star*{\bf a} -{\bf e}_0\|_2 = \|{\bf w}^\star -{\bf e}_0\|_2 \leq C \sigma.
\]
In words, \textbf{the Euclidean distance between ${\bf w}^\star \star {\bf a}$ and ${\bf e}_0$ is bounded linearly by the magnitude of stochastic noise}.
\end{theorem}  

\newcommand{{\bf c}}{{\bf c}}
\paragraph{Robustness under adversarial noise }
 \begin{theorem}[Robustness under adversarial noise with $\ell_p$ norm bound]
\label{thm:adv_exact_rob}
Suppose that an adversary chooses a disturbance bisequence
${\bf c}$ subject to the constraint:	
\[
	\| {\bf c}\|_\infty \leq \eta;
\]
and perturbs the observation process $Y$ via:
	\[
	\begin{array}{ll}
	Y&= {\bf a}* (X+  {\bf c}) .
	\end{array}
	\]
\newcommand{{\bf v}}{{\bf v}}
Let ${\bf w}^\star$ denote the solution of the convex optimization problem~\ref{eqn:P1}.
Define ${\bf v}^\star = ({\bf a}*{\bf w}^\star)^{\dagger}$, then ${\bf v}^\star$ satisfies the following bound on objective difference:
 \[
 \begin{array}{ll}
 {\mathbb E}_I  \| ({\bf v}^\star)\cdot I \|_2 -{\mathbb E}_I  \| ({\bf e}_0)\cdot I \|_2 \leq 
 p\sqrt{\frac{2}{\pi}}(\mathcal{R}(\eta) - 1)+(1-p)\eta.
\end{array}
  \]  
 
Therefore, when $p<p^\star$, there exists a constant $C'$, so that
 \[
 \| {\bf w}^\star*{\bf a} -{\bf e}_0\|_2 = \|{\bf v}^\star -{\bf e}_0\|_2  \leq C'  \eta, \forall \eta > 0.
 \]
 In words, \textbf{the Euclidean distance between ${\bf w}^\star \star {\bf a}$ and ${\bf e}_0$ is at most proportional to the magnitude of the adversarial noise}.
 
 
\end{theorem}  

\textbf{Remark}: Here $\mathcal{R}(\eta)$
is the folded Gaussian mean, for standard Gaussian $G$:
\[
\mathcal{R}(\eta) :=  \sqrt{\frac{\pi}{2}} E_G |\eta+  G| = \exp\{-\eta^2/2\} + \sqrt{\frac{\pi}{2}}\eta \left(1 - 2\Phi\left(-\eta\right) \right).
\]  
$\mathcal{R}(\eta)-1$ is an even function that is monotonically non-decreasing for $\eta \geq 0$ with quadratic upper and lower bound: there exists constants $C_1\leq C_2$,
  \[
C_1 \eta^2 \leq \mathcal{R}(\eta)-1 \leq C_2 \eta^2, \forall \eta.
\]

\newcommand{{\bf u}}{{\bf u}}
	





\section{Rest of Old Introduction}

	

















\paragraph{Sketch of proof ideas for Theorem~\ref{thm:Population phase transition}}
The full proof is in section $6$. Here we highlight some of the key ideas in the proof:
\begin{itemize}
\item
\textbf{Change of variable.}
Rewrite the population version of our convex sparse blind deconvolution problem, with the population objective 
${\mathbb E}\frac{1}{N} \| {\bf w}*Y\|_{\ell_1({\cal T})} ={\mathbb E}|({\bf w}*Y)_0|={\mathbb E}|({\bf w}*{\bf a}*X)_0|={\mathbb E}_X|\langle X, ({\bf w}*{\bf a})^{\dagger} \rangle|$ due to  the ergodic property of stationary process and shift invariance, and $(\widetilde{{\bf a}}*{\bf w})_0=((\widetilde{{\bf a}}*{\bf a}^{-1})*({\bf a}*{\bf w}))_0  = \langle (\widetilde{{\bf a}}*{\bf a}^{-1})^{\dagger}, {\bf w}*{\bf a}\rangle$, the convex problem becomes 
\[
 \begin{array}{ll}
 \underset{{\bf w}}{\mbox{minimize}}   & {\mathbb E}_X|\langle X ,  ({\bf a}*{\bf w})^{\dagger} \rangle|\\
\mbox{subject to}  
& \langle \widetilde{{\bf a}}*{\bf a}^{-1} , ({\bf a}*{\bf w})^{\dagger}\rangle =1,
\end{array}
 \]
 \newcommand{{\bf v}}{{\bf v}}
Let ${\bf v}$ denote  the time reversed version of ${\bf a}*{\bf w}$: ${\bf v} := ({\bf a}*{\bf w})^{\dagger}$, and
  let $\widetilde{{\bf e}}:= \widetilde{{\bf a}}*{\bf a}^{-1}$, then by previous assumptions, 
  $\widetilde{{\bf e}}(0)=1$, $\widetilde{{\bf e}}' = \widetilde{{\bf e}}-{\bf e}_0$.
  
 Now we arrive at a simple and fundamental population convex problem: 
 \[
 \begin{array}{ll}
 \underset{{\bf v}}{\mbox{minimize}}   &{\mathbb E}_X|\langle X ,  {\bf v}  \rangle|\\
\mbox{subject to}  
& \langle \widetilde{{\bf e}} ,  {\bf v}  \rangle  =1.
\end{array}
  \]
  \item
\textbf{Expectation using Gaussian.  }
Since $X$ follows Bernoulli-Gaussian IID probability model $X_t = I_t G_t$, we nest the expectation over $I_t$ outside the expectation over Gaussian $G_t$, for which we use $E |N(0,1)| = \sqrt{\frac{2}{\pi}}$:
\[
{\mathbb E}_X|\langle X ,  {\bf v}  \rangle|
= {\mathbb E}_I {\mathbb E}_G {|\sum_{t\in {\mathbb Z}}  I_t G_t {\bf v}(t)| }
= \sqrt{\frac{2}{\pi}} \cdot{\mathbb E}_I \|{\bf v}\cdot I\|_2
\]
\item
\textbf{KKT condition for ${\bf e}_0$.}
Let ${\bf v}^\star$ denote the solution of the optimization problem:
\[
 \begin{array}{ll}
 \underset{\psi}{\mbox{minimize}}   & \sqrt{\frac{2}{\pi}} \cdot{\mathbb E}_I \|{\bf v}\cdot I\|_2\\
\mbox{subject to}  
& \langle   \widetilde{{\bf e}}, {\bf v}\rangle =1
\end{array}
\tag{$Q_1(\widetilde{{\bf e}})$ }
  \]
  
To prove that ${\bf v}^\star={\bf e}_0$, i.e. ${\bf e}_0$ solves ($Q_1(\widetilde{{\bf e}})$), we calculate the directional finite difference at ${\bf e}_0$. Then ${\bf e}_0$ solves this convex problem if the directional finite difference at ${\bf v} = {\bf e}_0$ is non-negative at every direction $\beta$ on unit sphere where $\widetilde{{\bf e}}^T \beta =0$:
	 \[
	 {\mathbb E}\|({\bf e}_0+\beta)\cdot I\|_2 - {\mathbb E}\|({\bf e}_0)\cdot I\|_2 \geq 0.
	 \]


\item
\textbf{Conditional expectation at one sparse element $X_0$.} 
We decompose the objective into a sum of terms, 
conditioning on whether $I_0 = 1_{\{X_0\neq 0\}}$ is zero or not:
	 \[
	 \begin{array}{ll}
	    {\mathbb E}\|({\bf e}_0+\beta)\cdot I\|_2 - {\mathbb E}\|({\bf e}(0))\cdot I\|_2
	    &= p (1+\beta_0)  + (1-p) \nabla_{\beta}{{\mathbb E}_I[\|({\bf e}_0+\beta)'\|_{\ell_2(I-\{0\})} \mid I_0 =0]} -p\\
	    &= p \beta_0  + (1-p) {\mathbb E}_{I'}[\|\beta'\|_{\ell_2(I')} ].
	 \end{array}
	\]
This will be non-negative in case either $\beta(0) > 0$, or else $\beta(0) < 0$ but
	\[
	\frac{p}{1-p} \leq  \frac{{\mathbb E}_{I'}\|\beta'\|_{\ell_2(I')}}{ |\beta_0|}
	\]
	for all $\beta $ that satisfy $ \widetilde{{\bf e}}^T \beta =0$.
\item
\textbf{Reduction to $val(Q_1 (\widetilde{{\bf e}}')).$}	
	We normalize the direction sequence $\beta$ so that $\beta(0)= -1$; 
	using $\tilde{e}(0)=1$, we obtain a lowerbound:
\[
 \inf_{\beta(0)=-1, \langle  \widetilde{{\bf e}}, \beta\rangle  =0} {\mathbb E}_{I'}\|\beta'\|_{\ell_2(I')}
 =\inf_{ \beta(0)=-1, \beta(0) \tilde{e}(0)-\langle  \widetilde{{\bf e}}', \beta'\rangle  =0} {\mathbb E}_{I'}\|\beta'\|_{\ell_2(I')}
= \inf_{\langle   \widetilde{{\bf e}}', \beta'\rangle  =1} {\mathbb E}_{I'} \|\beta'\|_{\ell_2(I')} = val(Q_1 (\widetilde{{\bf e}}'))
\]	
Here $Q_1(\widetilde{{\bf e}}')$ is the optimization problem:
\[
 \begin{array}{ll}
 \underset{\beta \in l_1({\mathbb Z})}{\mbox{minimize}}   &{\mathbb E}_{I'} \|\beta'\|_{\ell_2(I')}\\
\mbox{subject to}  
& \langle  \widetilde{{\bf e}}', \beta'\rangle  =1
\end{array}
  \]

\item 
\textbf{The explicit phase transition condition with upper and lower bound.}
We have shown the existence of  $p^\star$ so that for all $p<p^\star$, the KKT condition is satisfied. And we have represented $p^\star$ as the optimal value of a derived optimization problem $val(Q_1 (\widetilde{{\bf e}}'))$. The following lemma finds simple upper and lower bounds for $p^\star=val(Q_1 (\widetilde{{\bf e}}'))$. 
	\begin{lemma}[Explicit phase transition condition with upper and lower bound]
	\label{lemma:Condition phase transition}
The threshold $p^\star$ determined by
 \[
\frac{p}{1-p}= val(Q_1(\widetilde{{\bf e}}')).
\]
obeys an upper bound and lower
\[
 \frac{p}{\|\widetilde{{\bf e}}'\|_\infty}  \geq val(Q_1(\widetilde{{\bf e}}')) \geq  {\mathbb E}_I\|\widetilde{{\bf e}}'\cdot I\|_{2}^{-1}
\]
where $\|\widetilde{{\bf e}}'\|_\infty  = \frac{|\widetilde{{\bf e}}|_{(2)}}{|\widetilde{{\bf e}}|_{(1)}}  $.
	Additionally, the upper bound 
is sharp if and only if
	\[
	\frac{p}{1-p} \leq val(Q_1(\frac{\widetilde{{\bf e}}'}{\|\widetilde{{\bf e}}'\|_\infty}))=val(Q_1(\widetilde{{\bf e}}''))/\|\widetilde{{\bf e}}'\|_\infty
	\]
	therefore, the upper bound 
holds with equality 
	\[
	p^\star= 1- \frac{|\widetilde{{\bf e}}|_{(2)}}{|\widetilde{{\bf e}}|_{(1)}}
	\]
if 
		\[
	p\leq 1- \frac{|\widetilde{{\bf e}}|_{(3)}}{|\widetilde{{\bf e}}|_{(2)}}
	\]
	Therefore, if 
		\[
	\frac{|\widetilde{{\bf e}}|_{(3)}}{|\widetilde{{\bf e}}|_{(2)}} \leq \frac{|\widetilde{{\bf e}}|_{(2)}}{|\widetilde{{\bf e}}|_{(1)}}
	\]
	then
	\[
	p^\star= 1- \frac{|\widetilde{{\bf e}}|_{(2)}}{|\widetilde{{\bf e}}|_{(1)}}.
	\]
	
	\end{lemma}

The above narrative gives a sketch of our result and its proof. The upper bound $ 1- \frac{|\widetilde{{\bf e}}|_{(2)}}{|\widetilde{{\bf e}}|_{(1)}}$ of $p^\star$ generalized the previous special case of exponential decay filter in theorem~\ref{thm: single root PT} with $p^\star = 1-|s|$.
\item
\begin{lemma}[Geometric lower bound $val(Q_1(\widetilde{{\bf e}}'))$ for the phase transition condition]
\label{lemma: geo lower}
    \[
val(Q_1(\widetilde{{\bf e}}'))
\geq p \cot\angle(\widetilde{{\bf e}}, {\bf e}_0)
\]
where
\[
\cot\angle(\widetilde{{\bf e}}, {\bf e}_0) = \frac{1}{\|\widetilde{{\bf e}}'\|_2} 
\]
\end{lemma}
\item
Using the technical background provided in theorem~\ref{thm:BGPropLemmaNorm}, we can further provide a tighter upper bound to compute phase transition $p^\star$.  
\begin{lemma}[Tighter upper bound $val(Q_1(\widetilde{{\bf e}}'))$ for the phase transition condition]
\label{lemma: tigher upper}
Let $|\widetilde{{\bf e}}|$ be the entry-wise absolute value of $\widetilde{{\bf e}}$. We can rank the entries of $|\widetilde{{\bf e}}|$ to be $|\widetilde{{\bf e}}|_{(1)}, |\widetilde{{\bf e}}|_{(2)}, |\widetilde{{\bf e}}|_{(3)},\ldots$, then the entries of $\widetilde{{\bf e}}'$ will be ranked as $|\widetilde{{\bf e}}|_{(2)}, |\widetilde{{\bf e}}|_{(3)},\ldots$, we define $|\widetilde{{\bf e}}|^{(m)}$ as the vector that only keep the entries $|\widetilde{{\bf e}}|_{(2)}, |\widetilde{{\bf e}}|_{(3)},\ldots, |\widetilde{{\bf e}}|_{(m+1)}$ of $|\widetilde{{\bf e}}|$, and send the rest of entries to zero. 
Then
    \[
val(Q_1(\widetilde{{\bf e}}'))
\leq \inf \{ \frac{p}{|\widetilde{{\bf e}}|_{(2)}}, \frac{V_1(|\widetilde{{\bf e}}|^{(2)})}{\||\widetilde{{\bf e}}|^{(2)}\|_2}, \frac{V_1(|\widetilde{{\bf e}}|^{(3)})}{\||\widetilde{{\bf e}}|^{(3)}\|_2}, \ldots, \frac{V_1(\widetilde{{\bf e}}')}{\|\widetilde{{\bf e}}'\|_2}   \}
\]
where $V_1$ function takes value in $[p, \sqrt{p})$.
Therefore, intuitively,
  \[
val(Q_1(\widetilde{{\bf e}}'))
\leq 
\inf \{ \frac{p}{|\widetilde{{\bf e}}|_{(2)}},  \sqrt{p} \cot\angle(\widetilde{{\bf e}}, {\bf e}_0) \}
\]
\end{lemma}

\end{itemize}

		
		
	
	



\subsection{Main Result $2$: Finite Observation Window, Finite-length Inverse}

With a finite observation window of length $N$ we (surprisingly) still can have exact recovery, starting as soon as $N \geq \Omega(k\log(k))$.

\paragraph{Finite-observation phase transition}


Let 
$\ell_1^k$
denote the collection of bisequences vanishing outside a centered window of radius $k$.

	\begin{theorem}[Finite observation window, finite-length inverse filter]
	\label{thm:fin_guarantee}
	Suppose $Y = {\bf a} \star X$, where:
	\begin{itemize}
\item
$(X_t)_{t \in {\mathbb Z}}$ is IID  $pN(0,1) + (1-p) \delta_0$. 
 \item
${\bf a}\in l_1({\mathbb Z})$ has a finite-length inverse: ${\bf a}^{-1} \in \ell_1^k$ vanishes off a centered window of length $k$.
\end{itemize}
\item
Suppose we are given a window $(Y_t)_{t\in {\cal T}}$ of length $N = |{\cal T}|$. 
Solve the optimization problem:
\begin{equation}
 \begin{array}{ll}
 \underset{{\bf w}\in \ell_1^k}{  \mbox{minimize} }   & \frac{1}{N}\|{\bf w}*Y\|_{\ell_1^N}\\
 \mbox{subject to}  
& (\widetilde{{\bf a}}* {\bf w})_0 =1.
\end{array}
\tag{${P}_{1}^{N,k}(\widetilde{{\bf a}})$}
\label{P1_finite}
\end{equation}

	There exist $\epsilon>0, \delta>0$, so that when the number of observations $N$ satisfies
		\[
N \geq  k\log(\frac{k}{\delta}) ( \frac{C\kappa_{\bf a}}{\epsilon}  )^2,
\]
and the sparsity level $p$ obeys $\frac{1}{N}\leq p \leq p^\star - \delta_p(N,\epsilon)$,
		then 		with probability exceeding $1-\delta$ the solution ${\bf w}^\star$  of  $({P}_{1}^{N,k}(\widetilde{{\bf a}}))$ is ${\bf a}^{-1}$ up to rescaling and time shift.
	\end{theorem}
	
 


	







	

\subsection{Main Result $3$: Stability Guarantee with Finite Length Inverse Filter}
The results so far concern the ideal setting when true inverse filter has a known length $k$ and we use $k$ to set up a correctly matched optimization problem. In practice we do not know $k$ and $k$ might even be infinite.

We can provide practical guarantees even when the inverse filter is an infinite length inverse. To develop these, we must be more technical about the situation. We assume that ${\bf a}$ has a Z-transform having $N_{-}$ roots and $N_{+}$ poles $(s_{i})$ inside the unit circle and we construct a finite length approximation ${\bf w}$ to  ${\bf a}^{-1}$, in fact of length
$r(N_{-}+N_{+})$.
This approximation has error 
$\|{\bf w}*{\bf a}-{\bf e}_0\|_2= O(\max_i |s|_{i}^{r}  )$. 

Since the objective value ${\mathbb E}_I\|{\bf w}\cdot I\|_2$ of this approximation ${\bf w}$ is an upper bound of the optimal value of the optimization solution ${\bf w}^\star$, we could use the objective value upper bound  to derive a upper bound for $\|{\bf w}^\star*{\bf a}-{\bf e}_0\|_2= O(
\max_i |s|_{i}^{r}  )$ when $p<p^\star$, where the constant of this upper bound is determined by the Bi-Lipschitz constant of the finite difference of objective ${\mathbb E}_I\|{\bf w}\cdot I\|_2-{\mathbb E}_I\|({\bf e}_0)\cdot I\|_2$.

\paragraph{Approximation theory for infinite length inverse filter}
\begin{theorem}[Approximation theory for infinite length inverse filter based on roots of Z-transform]
\label{thm:RootNorm}
Let the finite-length forward filter ${\bf a}$ have a Z-transform with roots inside the unit circle, namely $s_{k}:= e^{-\rho_k + i \varphi_k }$ with $|s_k|<1$ and $\rho_k>0$
for $k \in \{-N_{-}, \ldots -1, 1, \ldots N_{+}\}$.  Let ${\cal I} = \{-N_{-},\ldots, -1, 1, \ldots, N_{+} \}$ as the set of all the possible indexes.
\[
 \begin{array}{ll}
A(z) &=  \sum_{i=-N_{-}}^{N_{+}} a_{i} z^{-i}
\\
&= c_0\prod_{j=1}^{N_{-}} (1- s_{-j} z) \prod_{i=1}^{N_{+}} (1- s_{i} z^{-1});
\end{array}
  \]
here $c_0$ is a constant ensuring $a_0=1$.

Then for a scalar $r$
, we construct an approximate inverse filter ${\bf w}^r$ with Z-transform
\[
 \begin{array}{ll}
W(z) 
&=  \frac{1}{c_0}\prod_{j=1}^{N_{-}} (\sum_{\ell_j=0}^{r-1} s^{\ell_j}_{-j} z^{-\ell_j} ) \prod_{i=1}^{N_{+}} (\sum_{\ell_i=0}^{r-1} s^{\ell_i}_{i} z^{\ell_i} )\\
&=  \frac{1}{c_0}\prod_{j=1}^{N_{-}} (1- (s_{-j} z)^{r})(1- s_{-j} z)^{-1} \prod_{i=1}^{N_{+}}(1- (s_{i}  z^{-1})^{r}) (1- s_{i} z^{-1})^{-1}.
\end{array}
\]
We have
\[
 \begin{array}{ll}
 \|{\bf w}^r*{\bf a}-{\bf e}_0\|_2^2  
 &= \sum_{n\in \{1,2,3,\ldots,|{\cal I}| \} }\sum_{ k_1, \ldots, k_n  \in {\cal I} } \prod_{i\in [n]}|s_{k_i}|^{2r }
\end{array}
  \]
as $r \rightarrow \infty$, this converges to zero at an exponential rate, determined by the slowest decaying term,
\[
 \begin{array}{ll}
\|{\bf w}^r*{\bf a}-{\bf e}_0\|_2
&=  O( |s|_{(1)}^{r}  ) , \quad r \rightarrow \infty.
\end{array}
  \]
  where $|s|_{(1)}$ is the largest absolute value root.
 \end{theorem}
 
 
 \paragraph{Stability for infinite length inverse filter}
 \begin{theorem}[Stability for infinite length inverse filter]
 \label{thm:bd_stab}
 Let ${\bf a}\in V_{N_-, N_+} $ be a forward filter with all the roots of Z-transform strictly in the unit circle. Let ${\bf w}^\star \in V_{(r-1)N_-, (r-1)N_+} $ be the solution of the convex optimization problem. Let ${\bf w}^r$ be the constructed filter in previous theorem with a uniform vector index $(r, \ldots, r, r,\ldots, r)$.
  Then provided $p<p^\star$, as 
  $r \rightarrow \infty$, 
  \[
 \begin{array}{ll}
  \| {\bf w}^\star*{\bf a} -{\bf e}_0\|_2 
   \leq
    O(
 |s|_{(1)}^{r}  ), \quad r \rightarrow \infty.
\end{array}
  \]
  where $|s|_{(1)}$ is the largest absolute value root.
In words, \textbf{the Euclidean distance between ${\bf w}^\star*{\bf a}$ and ${\bf e}_0$ converges to zero at an exponential rate as the approximation length is allowed to increase.} 
 \end{theorem} 
 
 
 
\subsection{Main Result $4$: Robustness Against Stochastic Noise and Adversarial Noise}
We now extend the previous analysis of an exactly sparse model of $X$, exactly observed, to the more practical setting of approximate sparsity and observation noise. We consider two cases: first we add stochastic noise as an independent Gaussian linear process, and second adversarial noise with bounded $\ell_\infty$ norm.  

In each scenario, since the noisy objective value at ${\bf e}_0$ is an upper bound on the optimal value of the optimization solution ${\bf w}^\star$, we use this upper bound on the objective value  to derive an upper bound of $\|{\bf w}^\star*{\bf a}-{\bf e}_0\|_2$ when $p<p^\star$.
The upper bound shows that the distance $\|{\bf w}^\star*{\bf a}-{\bf e}_0\|_2$ is bounded by the (appropriately measured)
magnitude of input noise in both cases.


\paragraph{Robustness under stochastic noise }
 \begin{theorem}[Robustness against Gaussian Linear Process noise]
 \label{thm:rand_exact_rob}
 We consider a Gaussian linear process $Z = \sigma \cdot {\bf b} \star G$ where: $\sigma > 0$ denotes the noise level;   ${\bf b}$ is
 a bisequence having  unit $\ell_2$ norm $\|{\bf b}\|_2=1$; and $G$ is a standard Normal iid bisequence:
 \[
\begin{array}{ll}
Y&= {\bf a}* (X+ Z).
\end{array}
\]
 
 \newcommand{{\bf w}}{{\bf w}}
 Let ${\bf w}^\star$ be the solution of the convex optimization problem in Eq.(4.1);


\label{thm:Bound_Normal_Noise}
\[
 {\mathbb E}_I  \| ({\bf w}^\star)\cdot I \|_2 -{\mathbb E}_I  \| ({\bf e}_0)\cdot I \|_2 \leq  (1-p) \sigma +
 p(\sqrt{1+ \sigma^2 }-1).
\]
When $p<p^\star$, $\sigma\leq 1$, there exists a constant $C$,
\[
  \| {\bf w}^\star*{\bf a} -{\bf e}_0\|_2 = \|{\bf w}^\star -{\bf e}_0\|_2 \leq C \sigma.
\]
In words, \textbf{the Euclidean distance between ${\bf w}^\star \star {\bf a}$ and ${\bf e}_0$ is bounded linearly by the magnitude of stochastic noise}.
\end{theorem}  

\newcommand{{\bf c}}{{\bf c}}
\paragraph{Robustness under adversarial noise }
 \begin{theorem}[Robustness under adversarial noise with $\ell_p$ norm bound]
\label{thm:adv_exact_rob}
Suppose that an adversary chooses a disturbance bisequence
${\bf c}$ subject to the constraint:	
\[
	\| {\bf c}\|_\infty \leq \eta;
\]
and perturbs the observation process $Y$ via:
	\[
	\begin{array}{ll}
	Y&= {\bf a}* (X+  {\bf c}) .
	\end{array}
	\]
\newcommand{{\bf v}}{{\bf v}}
Let ${\bf w}^\star$ denote the solution of the convex optimization problem~\ref{eqn:P1}.
Define ${\bf v}^\star = ({\bf a}*{\bf w}^\star)^{\dagger}$, then ${\bf v}^\star$ satisfies the following bound on objective difference:
 \[
 \begin{array}{ll}
 {\mathbb E}_I  \| ({\bf v}^\star)\cdot I \|_2 -{\mathbb E}_I  \| ({\bf e}_0)\cdot I \|_2 \leq 
 p\sqrt{\frac{2}{\pi}}(\mathcal{R}(\eta) - 1)+(1-p)\eta.
\end{array}
  \]  
 
Therefore, when $p<p^\star$, there exists a constant $C'$, so that
 \[
 \| {\bf w}^\star*{\bf a} -{\bf e}_0\|_2 = \|{\bf v}^\star -{\bf e}_0\|_2  \leq C'  \eta, \forall \eta > 0.
 \]
 In words, \textbf{the Euclidean distance between ${\bf w}^\star \star {\bf a}$ and ${\bf e}_0$ is at most proportional to the magnitude of the adversarial noise}.
 
 
\end{theorem}  

\textbf{Remark}: Here $\mathcal{R}(\eta)$
is the folded Gaussian mean, for standard Gaussian $G$:
\[
\mathcal{R}(\eta) :=  \sqrt{\frac{\pi}{2}} E_G |\eta+  G| = \exp\{-\eta^2/2\} + \sqrt{\frac{\pi}{2}}\eta \left(1 - 2\Phi\left(-\eta\right) \right).
\]  
$\mathcal{R}(\eta)-1$ is an even function that is monotonically non-decreasing for $\eta \geq 0$ with quadratic upper and lower bound: there exists constants $C_1\leq C_2$,
  \[
C_1 \eta^2 \leq \mathcal{R}(\eta)-1 \leq C_2 \eta^2, \forall \eta.
\]

\newcommand{{\bf u}}{{\bf u}}

\subsection{Technical Preparation:  Landscape of  Expected Homogeneous Function over Bernoulli Support on Sphere}

Let 
\[
V_k({\bf u}) := \frac{{\mathbb E}_J\|{\bf u}\cdot J\|_2^k}{\|{\bf u}\|^k_{2}} 
\]
where ${\bf u}\in {\mathbb R}^N$, $J$ is a Bernoulli sequence indexed from $1$ to $N$.


\paragraph{Expectation $V_1({\bf u}):= {\mathbb E}_J \|{\bf u}\cdot J\|_2^{1}$ on sphere.}





\begin{theorem} \label{thm:BGPropLemmaNorm}
Let ${\bf u} \in {\mathbb R}^N$, then
\[
p \leq V_{1}({\bf u}) \leq V_{1}(\frac{\sum_{j\in [N]}\pm {\bf e}_j}{\sqrt{N}} ) \leq \sqrt{p} .
\]
the lower bound is approached by on-sparse vectors ${\bf u} \in \{ \pm {\bf e}_i, i\in [N]\}$, and the upper bound is approached by 
${\bf u}  \in \{  \frac{1}{\sqrt{N}}\sum_{j\in [N]}\pm {\bf e}_j\}$.

Furthermore, all the stationary points of $V_{1}({\bf u})$ are $\{\frac{\sum_{i\in J} \pm {\bf e}_i}{\sqrt{N_J}}\}$ for different support $J\subset {\cal T}$, where $\{ \pm {\bf e}_i, i\in [N]\}$ are the global minimizers, and $\{  \frac{1}{\sqrt{N}}\sum_{j\in [N]}\pm {\bf e}_j\}$ are the global maximizers. And for $J$ with $1<N_J<N$, $\{\frac{\sum_{i\in J} \pm {\bf e}_i}{\sqrt{N_J}}\}$ are saddle points with value
\[
V_1(\frac{\sum_{i\in J} \pm {\bf e}_i}{\sqrt{N_J}}) = {\mathbb E}_I \sqrt{\frac{\sum_{i\in J} 1_{I_i}}{N_J}} = \sum_{j=0}^{N_J} (1-p)^{N_J-j} p^{j} {N_J \choose j}\sqrt{\frac{j}{N_J}}.
\]
\end{theorem}
To support geometric intuition, Figure~\ref{fig-landscape-V1}
visualizes $V_1$ on the $2-$dimensional sphere.

\begin{figure} 
\includegraphics[width=.7\textwidth]{./figures_BD/Ju_landscape.png}
\caption{The value of $V_1$ on a two-dimensional sphere in ${\mathbb R}^3$, normalized by the affine transform to send the value in $[0,1].$  
}
\label{fig-landscape-V1}
\end{figure}


\paragraph{Expectation $V_{2k}({\bf u}):= {\mathbb E}_J \|{\bf u}\cdot J\|_2^{2k}$ on sphere.}
\begin{theorem} \label{thm:BGPropLemmaSub_k}
For $k\geq 2$, let ${\bf u}$ denote a vector in ${\mathbb R}^N$, then
\[
p^k \leq V_{2k}(\frac{\sum_{j\in [N]}\pm {\bf e}_j}{\sqrt{N}} ) \leq V_{2k}({\bf u}) \leq p.
\]
the upper bound is approached by one-sparse vectors ${\bf u} \in \{ \pm {\bf e}_i, i\in [N]\}$, and the lower bound is approached by 
${\bf u}  \in \{  \frac{1}{\sqrt{N}}\sum_{j\in [N]}\pm {\bf e}_j\}$.

Furthermore, all the stationary points of $V_{2k}(\psi)$ are $\{\frac{\sum_{i\in J} \pm {\bf e}_i}{\sqrt{N_J}}\}$ for different support $J\subset {\cal T}$, where $\{ \pm {\bf e}_i, i\in [N]\}$ are the set of global maximizers, and $\{  \frac{1}{\sqrt{N}}\sum_{j\in [N]}\pm {\bf e}_j\}$ are the set of all the global minimizers. And for $J$ with $1<N_J<N$, $\{\frac{\sum_{i\in J} \pm {\bf e}_i}{\sqrt{N_J}}\}$ are saddle points with value
\[
V_{2k}(\frac{\sum_{i\in J} \pm {\bf e}_i}{\sqrt{N_J}}) = {\mathbb E}_J (\frac{\sum_{i\in J} 1_{I_i}}{N_J})^k = \sum_{j=0}^{N_J} (1-p)^{N_J-j} p^{j} {N_J \choose j}(\frac{j}{N_J})^k.
\]
\end{theorem}

\paragraph{Expectation $V_{-1}({\bf u}):= {\mathbb E}_J \|{\bf u}\cdot J\|_2^{-1}$ on sphere.}
\begin{theorem}
\label{thm:TaylorSqrtInvExpect}
\begin{equation} 
  V_{-1}({\bf u}) 
  \geq p^{-1/2} .
\end{equation}
\end{theorem}



\subsection{Technical Tool: Relation between Finite Difference of Objective and Euclidean Distance }
 \paragraph{Bi-Lipschitzness of finite difference of objective}
As an important proof tool, we study functional $B$ that allows us to bound the Euclidean distance $d_2(\psi, {\bf e}_0) := \| \psi - {\bf e}_0\|_2$ by the objective difference ${\mathbb E}_I  \| \psi\cdot I \|_2- {\mathbb E}_I  \| ({\bf e}_0)\cdot I \|_2$  : 
\[
B({\bf e}_0,\phi) := \frac{{\mathbb E}_I  \| ({\bf e}_0+\phi)\cdot I \|_2 - {\mathbb E}_I  \| ({\bf e}_0)\cdot I \|_2}{\|\phi\|_2} 
\]
When we rescale $\phi$ to $\beta:= \frac{\phi}{\|\phi \|_2}$, we have
\[
B({\bf e}_0,t \beta) = \frac{{\mathbb E}_I  \| ({\bf e}_0+t \beta)\cdot I \|_2 - {\mathbb E}_I  \| ({\bf e}_0)\cdot I \|_2}{t} 
\]
From the definition of directional derivative,
\[
\nabla_{\phi}{{\mathbb E}_I  \| ({\bf e}_0+\phi)\cdot I \|_2}\mid_{\phi=0}=\lim_{t\rightarrow 0^+}\frac{{\mathbb E}_I  \| ({\bf e}_0+t \beta)\cdot I \|_2 - {\mathbb E}_I  \| ({\bf e}_0)\cdot I \|_2}{t}=\lim_{t\rightarrow 0^+}B({\bf e}_0,t \beta).
\]

We have upper and lower bound on their difference.
This upper and lower bound allows us to connect objective ${\mathbb E}_I  \| \psi\cdot I \|_2$ and the $2-$norm of $ \psi - {\bf e}_0$:
\begin{theorem}[Bi-Lipschitzness of finite difference of objective near ${\bf e}_0$ for linear constraint]
\label{thm:bound_diff}
We have upper and lower bound of $B({\bf e}_0, \phi)$ :
\begin{eqnarray*}
0\leq B({\bf e}_0,t \beta) -\lim_{t\rightarrow 0^+}B({\bf e}_0,t \beta) 
\leq 
  \frac{t}{2}p\Big( 
  \beta_0^2 
 +
 p  (1-\beta_0^2) \Big) \leq \frac{pt}{2}.
\end{eqnarray*}
This leads to
\[
0 \leq B({\bf e}_0, \phi) -	 \nabla_{\phi}{{\mathbb E}_I  \| ({\bf e}_0+\phi)\cdot I \|_2}\mid_{\phi=0}\leq \frac{p}{2}\|\phi \|_2.
\]
Therefore, when $p<p^\star$, $\nabla_{\phi}{{\mathbb E}_I  \| ({\bf e}_0+\phi)\cdot I \|_2}\geq \epsilon(p,p^\star)>0$, 
we have $B({\bf e}_0, \phi) \geq \epsilon(p,p^\star)>0$, which allows us to bound difference of objective by Euclidean distance .
Reversely, 
$1/B({\bf e}_0, \phi) \leq  1/\epsilon(p,p^\star)$, which allows us to bound Euclidean distance by difference of objective.

\end{theorem}



\subsection{Initialization of forward filter}
Now we study how to get the initial guess $\widetilde{{\bf a}}$.


First, let $C_Y$ denote the circular embedding of $Y$;  we can define $\Bar{Y} = (C_Y C_Y^T)^{-1/2}Y$, then from $Y={\bf a}*X$, we get $C_Y = C_a C_X$, then
\[
C_{\Bar{Y}} =  (C_Y C_Y^T)^{-1/2}C_Y = (C_a C_X C_X^T  C_a^T)^{-1/2}C_a C_X
\]
as $X$ are IID Bernoulli-Gaussian, 
we know approximately 
\[
C_{\Bar{Y}} =  (C_Y C_Y^T)^{-1/2}C_Y = (C_a C_X C_X^T  C_a^T)^{-1/2}C_a C_X \approx (C_a C_a^T)^{-1/2}C_a C_X
\]
We can also define $C_{\Bar{a}}=(C_a C_a^T)^{-1/2}C_a$.

One approach is to find initialization filter one root by another. The first step is to look for a single root inverse filter $\widetilde{{\bf w}}^{[1]} = (1, -\widetilde{s_1}) $
\[
 \begin{array}{ll}
 \underset{\widetilde{s_1} \in {\mathbb R} }{\mbox{minimize}}   & -\frac{1}{N} \| (1, -\widetilde{s_1})*\Bar{Y}\|_{\ell_4({\cal T}_N)}^4
\end{array}
  \]
then iterate by assuming $\Bar{Y}^{[1]} = (1, -\widetilde{s_1})*\Bar{Y}$, then try to look for $\widetilde{{\bf w}}^{[2]} = (1, -\widetilde{s_2})$ by solving
\[
 \begin{array}{ll}
 \underset{\widetilde{s_2} \in {\mathbb R} }{\mbox{minimize}}   & -\frac{1}{N} \| (1, -\widetilde{s_2})*\Bar{Y}\|_{\ell_4({\cal T}_N)}^4
\end{array}
  \]
  
We could find the initialization by running the projected gradient method on sphere with random initialization:
\[
 \begin{array}{ll}
 \underset{\widetilde{{\bf w}} \in l_1^k}{\mbox{minimize}}   & -\frac{1}{N} \| \widetilde{{\bf w}}*\Bar{Y}\|_{\ell_4({\cal T}_N)}^4\\
\mbox{subject to}  
& \|\widetilde{{\bf w}}\|_2 =1,
\end{array}
  \]
  We could also approximate this objective via re-weighted PCA.

With the solution of one of the non-convex optimization as $\widetilde{{\bf w}}^\star$, we have a initial guess of sparse signal:
\[
C_{\widetilde{{\bf w}}^\star}\Bar{Y}
\]
and we can set the initial guess of forward filter $\widetilde{{\bf a}}$ via least square solution
\[
\widetilde{{\bf a}} = C_{\Bar{Y}}^{+} C_{\widetilde{{\bf w}}^\star}^{+} Y 
\]

Another popular non-convex method is alternating minimization, which finds a good approximation of the forward filter ${\bf a}$:
\[
 \begin{array}{ll}
 \underset{\widetilde{{\bf a}}, {\bf x} \in l_1^N}{\mbox{minimize}}   & \frac{1}{N} (\lambda \| {\bf x}\|_{\ell_1({\cal T}_N)} + \| \Bar{Y}-\widetilde{{\bf a}}*  {\bf x}\|_{\ell_2({\cal T}_N)}^2)\\
\mbox{subject to}  
& \|\widetilde{{\bf a}}\|_2 =1,
\end{array}
  \]

\subsection{Non-convex method initialization of forward filter}

Our method is an iterative alternating minimization algorithm on convolutional dictionary learning objective.
\[
 \begin{array}{ll}
 \underset{A \in {\mathbb R}^{N_1\times N times K}, X \in {\mathbb R}^{N}}{\mbox{minimize}}   & \frac{1}{N} (\lambda \| {\bf x}\|_{\ell_1({\cal T}_N)} + \| \Bar{Y}-A  {\bf x}\|_{\ell_2({\cal T}_N)}^2)\\
\mbox{subject to}  
& A \in \mbox{Riemann manifold},
\end{array}
  \]

\subsection{Dave: using full $4$ order moments tensor to solve a system of linear equation to initialization}

Iteration of full $4$-th order momentum matching equation.


\subsection{Paper Organization}
Now we have stated the main results of our paper.
The paper is organized as follows. 
\begin{itemize}
\item
In section $1$, we introduce the blind deconvolution problems and its related works.
\item
In Section $2$, we have presented the main results of this paper.
\item
In Section $3$, we provide a systematic study of the landscape of the expected projection pursuit loss on sphere for linear combination of IID Bernoulli Gaussian $X$.
\item
In Section $4$, we introduce the technical backgrounds for convex blind deconvolution problem, including the existence of inverse filter, the change of variable, the definition of directional derivative and projected sub-gradient. 
\item
In Section $5$, we prove the main theorem of phase transition for convex blind deconvolution problem. 
\item
In Section $6$, we provide tight upper and lower bound of $val(Q_1)$ to compute the phase transition threshold $p^\star$. 
\item
In Section $7$. we provide upper and lower bound on the directional finite difference near ${\bf e}_0$ as a tool for the study of stability and robustness in the section $9$ and $10$.
\item
In Section $8$, we prove the finite sample observation version of the phase transition theorem and give an observation sample complexity bound $N \geq O(k\log(k))$ for a length $k$ inverse filter $a^{-1}$.
\item
In Section $9$, we prove the stability of the convex algorithm with infinite length inverse filter approximation.
\item
In Section $10$, we prove the robustness of the convex algorithm under random and adversarial  noise.
\item
In Section $11$, we give a general phase transition condition for other probabilistic models on $X$ beyond IID Bernoulli-Gaussian.
\item
In Section $12$, we conclude the paper.
\end{itemize}





\section{Discrete signal structure}
\paragraph{Random model for discrete digital signals.}
We define a random model for discrete digital signals. 

Let $(X_t)$ be an independent
sequence of random variable indexed by  $t \in Z$, with a symmetric distribution with CDF $F$. 
\begin{itemize}
\item The two point symmetric distribution on real number, $F = 1/2 \delta_{+1} + 1/2 \delta_{-1}.$ 
\item The four point symmetric distribution on complex number, 
$F = 1/4 \delta_{1+i} + 1/4 \delta_{-1+i}+1/4 \delta_{1-i}
+1/4 \delta_{-1-i}.$ 
\item A symmetric distribution on ${\mathbb R}^s$,
$F = 1/2^s( \sum_{c\in \{\pm 1\}^s} \delta_{\sum_{i\in[s]} c_i e_i } ).$ 

\item Constallantion in unit sphere.
The $2k$ point symmetric distribution on complex number, 
$F_p = \sum_{j=0}^{2k-1} p_j \delta_{\exp{(i\pi\frac{j}{k})}},$
where $\sum_{j=0}^{2k-1} p_j =1, p_j>0.$
\end{itemize}



\paragraph{Convex loss for discrete digital structure.}
The convex loss  we choose could be a relaxation of the convex indictor function of the points.

\begin{itemize}
\item For the two point symmetric distribution on real number, 
\[
g(x) = (|x|-1)_+
\]
\item The four point symmetric distribution on complex number, 
\[
g(z) = (|z|-1)_+,
\]
or more accurately,
\[
g(z) = (\|z\|_{\infty}-1)_+.
\]
\item For $s$-dimensional symmetric distribution on $\{\pm 1\}^s$,
\[
g(x) = (\|x\|_{\infty}-1)_+.
\] 
\end{itemize}


\paragraph{Blind recovery problem.}
For observation $Y$, assume that there exist inverse filter $\mu^* (e_0 + f^*)$, where $f^*_0 = 0$. We try to find it by solving a convex problem,
\begin{eqnarray*}
    \min_{f_0 = 0} &    
    E \sum g(\mu (Y+ f*Y)).
\end{eqnarray*}
First, when we know the correct scale $\mu^*$.

Let 
\[
e_0+h = ( e_0 + f)(e_0 + f^*)^{-1},
\]
we have
\begin{eqnarray*}
    \min_{f_0 = 0} &    
    E g( x_0+ h*x)).
\end{eqnarray*}
For $g(x) =(|x|-1)_+$, then the sub-gradient is
\[
g'(x) =  \left.
  \begin{cases}
    1, & \text{for } x > 1 \\
    0, & \text{for } -1 < x < 1 \\
    -1, & \text{for }-1 > x \\
    [0,1], & \text{for }x = 1  \\
    [-1,0], & \text{for }x = -1
  \end{cases}
  \right\}
\]

And the proximal operator $\mbox{prox}_{\rho g}$ is
\[
\mbox{prox}_{\rho g}(x) =  \left.
  \begin{cases}
    x-\rho, & \text{for } x > 1+\rho \\
    x, & \text{for } -1 < x < 1 \\
    x+\rho, & \text{for }-1-\rho > x \\
    1, & \text{for }1 \leq x \leq 1 +\rho \\
    -1, & \text{for } -1 -\rho\leq x \leq -1
  \end{cases}
  \right\}
\]


Assume $(X_t)$ are independent
sequence of random variable with CDF $F_{p}^t = \sum_{j=0}^{1} p^t_j \delta_{(-1)^j},$
where $\sum_{j=0}^{1} p^t_j =1, p^t_j>0.$ We choose the objective
\[
g(z) = (|z|-1)_+.
\] 
To show that $h=0$ is the global minimizer of the convex problem, we only need to show that for any $h\neq 0$, there exists small $\epsilon>0$, 
\begin{eqnarray*}    
    E g( x_0+ \epsilon h*x)-E g( x_0)>0.
\end{eqnarray*}
We use sub-gradient information,
\begin{eqnarray*}    
    E g( x_0+ \epsilon h*x)-E g( x_0)
    &\approx& 
    E g'( x_0)(\epsilon h*x)\\
    &=&
    \epsilon( p^0 _1 g'( +1)(h_0)+p^0_{-1}g'( -1)(-h_0)  )
    \geq 0.
\end{eqnarray*}

\paragraph{Polar form and extension to complex plane}
We can rewrite a point on the complex plane as its polar form
\[
z = r \exp{i\theta},
\]
where $r\geq 0$.

Under the polar form, the loss function is 
\[
g(z) = (|z|-1)_+ = (r-1)_+,
\]

We can reparametrize

Using this polar form, we are able to generalize our result to the complex setting.

\paragraph{Non-convex loss for discrete digital structure.}
If we are allowed to use non-convex loss, then we can fix the scale by the loss function.

Instead of convex loss
\[
g(x) = (|x|-1)_+,
\]
or
\[
g(x) = (|x|^2-1)_+,
\]
we can choose
\begin{itemize}
\item The smooth loss 
\[
g(x) = (|x|^2-1)^2,
\]
\item The composition of convex non-smooth function with smooth function,
\[
g(x) = (|x|-1)^2,
\]
\item The composition of  smooth function with convex non-smooth function,
\[
g(x) = ||x|^2-1|,
\]
\item The composition of  two convex non-smooth functions,
\[
g(x) = ||x|-1|.
\]
\end{itemize}


And instead of
\[
g(x) = (\|x\|_{\infty}-1)_+,
\]
we can choose
\begin{itemize}
\item The composition of convex non-smooth function with smooth function,
\[
g(x) = (\|x\|_{\infty}-1)^2,
\]
\item The composition of  two convex non-smooth functions,
\[
g(x) = |\|x\|_{\infty}-1|.
\]
\end{itemize}

\subsection{Noisy observation}
We consider the noisy recovery problem, there are two noise model, one is
\[
y = w_0 * (x+  n),
\]
the other is 
\[
y = w_0 * x+  u = w_0 * (x+ w_0^{-1} *u).
\]
When $u$ is Gaussian $N(0,\Sigma) $ , $w_0^{-1} *u$ is Gaussian $N(0,C_{w_0}^{-1}\Sigma C_{w_0}^{-1,T}) $. Therefore, we could reduce this problem as a colored noise $n$. 

\paragraph{Post-process}
For noisy observation $y$, we apply the algorithm on $y$ to get a solution $\hat x$, then we apply our prior knowledge to project the solution to the constellation points by taking $\title{x} = \mbox{sign}(\hat x).$

Then we measure the ratio of correct recovery by mean of
$1_{\title{x} = x}$ over multiple MC samples.
\paragraph{Population analysis}
Assume that the observation model are $y = w_0 * (x+  \sigma n),$ where $n$ is IID $N(0,1)$. Then we analyze the population risk,
 \begin{eqnarray*}
    \min_{h_0 = 0} &    
     E_x E_n g( x_0+ \sigma n_0+ h^T (x+\sigma n) ).
\end{eqnarray*}
Here $g(x) =(|x|-1)_+$.

Assume that the optimal solution is $h^*_\sigma$, when $\sigma = 0$ we proved before that $h^*=0.$ Here we get
$\hat x_0 = x_0+ \sigma n_0+ h^{*,T} (x+\sigma n) $
want to prove
\[
\tilde x_0 = \mbox{sign}(x_0+ \sigma n_0+ h_\sigma^{*,T} (x+\sigma n) ) = x_0,
\]
when $\sigma<\sigma_{m}$.
Without loss of generality, we assume $x_0=1$, we want to ask that when would $1+ \sigma n_0+ h_\sigma^{*,T} (x+\sigma n)\leq 0$, in general, the condition we need to charaterize is when 
$ |\sigma n_0+ h_\sigma^{*,T} (x+\sigma n)|\leq 1$.

\paragraph{Huber version of loss}
Instead of using $g(x) =(|x|-1)_+$, when there are Gaussian noise on $x$, we can use the Huber version 
\[
g_{\mu}(x) =\min_z  \frac{\mu}{2}\|x-z\|^2_2+ (|z|-1)_+  .
\]



























\section{Expectation calculation}
\subsection{Expectation over Bernoulli support}

We previously considered the identity
\begin{equation} \label{eq:startingPoint}
    {\mathbb E} | \sum_{i \in {\cal T}}  \psi_i X_i | = \sqrt{\frac{2}{\pi}} \cdot {\mathbb E}_I \| \psi \|_{\ell_2(I)},
\end{equation}
where $X$ is a Bernoulli-Gaussian RV $BG(p;0,1)$,
and where $I$ is a random subset of the domain ${\cal T}$
determined by Bernoulli-$p$ coin tossing.




This leads us to consider
\[
  {\mathbb E}_I \| \psi \|_{\ell_2(I)}^{2k}
 \]
 where $I$ is again a random subset, and $k \geq 1$ is an integer.
 The following lemma shows that ${\mathbb E}_I \| \psi \|_{\ell_2(I)}^{2k} \approx \| \psi \|_{\ell_2({\cal T})}^{2k}$.

\begin{lemma} \label{lem:BGPropLemmaSub}
Let $\psi$ denote a vector in ${\mathbb R}^N$.
\[
 \| \psi\|_2^{2k} \cdot \bar{B}(k-1;n,p)    \leq {\mathbb E}_I  \| \psi \|_{\ell_2(I)}^{2k}  \leq     \| \psi\|_2^{2k} ,
\]
where $\bar{B}(k-1;N,p) \equiv  \sum_{n=k}^{N} (1-p)^n p^{N-n} {N \choose n} $
is the survivor function (Complementary CDF) of the Binomial.
\end{lemma}

In particular, if $k \leq n \cdot p - 3 \cdot \sqrt{np (1-p)}$ for large $n$, then $\bar{B}(k-1;n,p) > 0.99$.
 The following lemma infers from this that ${\mathbb E}_I \| \psi \|_{\ell_2(I)} \approx \| \psi \|_{\ell_2({\cal T})}$.
 
\begin{lemma} \label{lem:BGPropLemmaNorm}
{\bf Qingyun: Numerically, I found this is not true even for fully dense $\psi$ as ${\varepsilon}_N$ could be quite large!}


There exists a sequence $({\varepsilon}_N)$ depending only on $N$ and $A$ so that  ${\varepsilon}_N \rightarrow 0$ 
and so that for each $N$ and for each 
vector $\psi$  in ${\mathbb R}^N$, with $\|\psi\|_{\ell_2} \leq A$ and $\|\psi\|_{\ell_0} = N$,
\begin{equation} \label{eq:LemmConclu}
 (1-N^{-1}) \cdot(1 - {\varepsilon}_N)  \leq  \frac{{\mathbb E}_I  \| \psi \|_{\ell_2(I)} }{\| \psi \|_{\ell_2({\cal T})}} \leq 1 + {\varepsilon}_N .
\end{equation}
\end{lemma}
Applying (\ref{eq:startingPoint}), we get:
\begin{coro} \label{lem:BGPropLemma}
{\bf Qingyun: Numerically, I found this is not true even for fully dense $\psi$!}
There exists a sequence $({\varepsilon}_N)$ depending only on $N$, so that  ${\varepsilon}_N \rightarrow 0$ and
and so that for each $N$ and for each 
vector $\psi$in ${\mathbb R}^N$, with $\|\psi\|_{\ell_2} \leq A$ and $\|\psi\|_{\ell_0} = N$.
\[
(1 - {\varepsilon}_N) \cdot (1-N^{-1}) \leq  \frac{E | \sum_{i \in {\cal T}}  \psi_i X_i |}{\sqrt{\frac{2}{\pi}}  \| \psi \|_{\ell_2({\cal T})}}  \leq   ( 1 + {\varepsilon}_N).
\]
\end{coro}


\subsection{Proof of Lemma \ref{lem:BGPropLemmaSub}}

\begin{proof}[Proof of Lemma \ref{lem:BGPropLemmaSub}]
Let's first suppose that $|\psi_t| \geq {\varepsilon} > 0$ for $-T \leq t \leq T$. Let $N = 2T+1$, and
let $I$ denote a subset of indices in $ {\cal T} = \{-T,\dots,T\}$ and
let $ \pi(I) = (1-p)^{|I|} p^{|{\cal T} \Delta I|}$. 
\begin{eqnarray*}
{\mathbb E}_I  \| \psi \|_{\ell_2(I)}^{2k}  &=&  \sum_{n=1}^{N}  \sum_{ |I |=n} \pi(I)   \| \psi \|_{\ell_2(I)}^{2k} \\
&=&   \sum_{n=1}^{N}  (1-p)^n p^{N-n}  \cdot  \sum_{|I| =n} \| \psi \|_{\ell_2(I)}^{2k}  \\
   \end{eqnarray*}
where we used $ \pi(I) = (1-p)^{|I|} p^{|{\cal T} \Delta I|} $.
We partition the sum  $\sum_{n=1}^{N} = \sum_{n=1}^{k-1} + \sum_{k}^N =  \sum_{n<k} + \sum_{n\geq k}$, say.
Now from $\#\{ |I|\leq n\} = {N \choose n}$ and $ \| \psi \|_{\ell_2(I)} \leq \| \psi \|_{\ell_2({\cal T})}$,
\begin{eqnarray*}
 \sum_{n<k}  (1-p)^n p^{N-n}  \cdot  \sum_{|I| =n} \| \psi \|_{\ell_2(I)}^{2k} 
     &\leq&   \left( \sum_{n=1}^{k-1} (1-p)^n p^{N-n} {N \choose n} \right) \cdot  \left( \| \psi \|_{\ell_2}^{2k} \right) \\
     &=& B(k-1;n,p) \cdot   \| \psi \|_{\ell_2}^{2k} ,
\end{eqnarray*}
where $B(x;n,p)$ denotes the cumulative distribution function of the binomial distribution.
For the sum $\sum_{n\geq k}$, we may write $\sum_{|I|=n} = {N \choose n}  \cdot Ave_{|I| = n}$
and $\sum_{t_1,\dots, t_k \in I} =  {n^k} \cdot  Ave_{t_1,\dots, t_k \in I}$, so
\begin{eqnarray*}
 \sum_{|I| = n}  \| \psi \|_{\ell_2(I)}^{2k} &=& \sum_{|I| = n} \sum_{t_1,\dots, t_k \in I} \prod_{i=1}^k \psi_{t_i}^2 \\
& =& {N \choose n}  \cdot Ave_{|I| = n}  \cdot {n^k} \cdot  Ave_{t_1,\dots, t_k \in I} \; \prod_{i=1}^k \psi_{t_i}^2 \\
&=& {N \choose n}  \cdot {n^k} \cdot Ave_{t_1,\dots, t_k \in {\cal T}} \; \prod_{i=1}^k \psi_{t_i}^2,\\
&=& {N \choose n}  \cdot \sum_{t_1,\dots, t_k \in {\cal T}} \; \prod_{i=1}^k \psi_{t_i}^2,\\
&=& {N \choose n}  \cdot  \| \psi \|_{\ell_2}^{2k} ,
\end{eqnarray*}
because the inner two averages  cover the $ {N \choose n} n^k$ terms in a uniform way.
We arrive at 
 \begin{eqnarray*}
 \sum_{n\geq k}  (1-p)^n p^{N-n}  \cdot  \sum_{|I| =n} \| \psi \|_{\ell_2(I)}^{2k} 
  &=&     \| \psi \|_{\ell_2}^{2k} \cdot  \left( \sum_{n=k}^{N} (1-p)^n p^{N-n} {N \choose n} \right) \\
  &=&    \| \psi \|_{\ell_2}^{2k} \cdot \bar{B}(k-1;n,p) .
 \end{eqnarray*} 
Here $\bar{B}(x;n,p) = 1- B(x;n,p)$ denotes the 
survivor function (complementary distribution function) of the binomial distribution.
We obtain:
 \begin{eqnarray*}
  \sum_{I \subset {\cal T}} \pi(I)   ( \sum_{t \in I} \psi_t^2)^{k}  
  & = &  \left( \sum_{n \geq k} + \sum_{n<k}   \right) \sum_{|I|=n}  \pi(I)   ( \sum_{t \in I} \psi_t^2)^{k} \\
 &\leq &    \|\psi\|_2^{2k} \cdot  \left( \bar{B}(k-1;p,n)  + B(k-1;p,n) \right) \\
 & = &  \|\psi\|_2^{2k} .
 \end{eqnarray*}
 while
 \begin{eqnarray*}
  \sum_{I \subset {\cal T}} \pi(I)   ( \sum_{t \in I} \psi_t^2)^{k}   & \geq & \sum_{n \geq k} \pi(I)   ( \sum_{t \in I} \psi_t^2)^{k}  \\
 &= &    \|\psi\|_2^{2k} \cdot \bar{B}(k-1;p,n) .
  \end{eqnarray*}
\end{proof}

\subsection{Proof of Lemma \ref{lem:BGPropLemmaNorm}}

\begin{proof}
Fix $A > 0$ and $p \in (0,1)$
By the Weierstrass approximation theorem, we may find a sequence of polynomials $S_K(t) = \sum_{k=0}^{K} c_k t^k$
so that
\[
    | \sqrt{x} - S_K(x) |_{L^\infty([0,A])} \leq {\varepsilon}_K, \quad \mbox { and } {\varepsilon}_K \rightarrow 0 \mbox{ as } K \rightarrow \infty.
\]
Pick a sequence $(K_N)$ so that $K_N \rightarrow \infty$ while
$K_N < N \cdot p -  \sqrt{2 \log(N) \sqrt{N} p (1-p)} $ and
\[
\bar{B}(K_N-1; n,p) > 1 - N^{-1}.
\]
Relabel $S_N  \equiv S_{K_N}$ and ${\varepsilon}_N \equiv {\varepsilon}_{K_N} $.
Then ${\varepsilon}_N \rightarrow 0$ as $N \rightarrow \infty$.

Fix $N$. For each $I \subset {\cal T}$,
let  $x_I \equiv {\mathbb E}_I \| \psi \|_{\ell_2(I)}^{2}$.  Then
\[
     E _I \| \psi \|_{\ell_2(I)} = {\mathbb E}_I \sqrt{\| \psi \|_{\ell_2(I)}^2} = {\mathbb E}_I \sqrt{ x_I }.
\]
while
\[
     E _I \| \psi \|_{\ell_2(I)}^{2k} = {\mathbb E}_I  x_I^k.
\]
Invoking our earlier lemma,
\[
 \bar{B}(k-1;n,p) \leq  \frac{{\mathbb E}_I  x_I^k}{\| \psi \|_{\ell_2({\cal T})}^{2k}} \leq 1.
\]

Armed with  polynomial $S_N$ of degree $K_N$, consider
\begin{eqnarray*}
     {\mathbb E}_I S_{N}( x_I ) &=& {\mathbb E}_I \sum_{k=1}^{K_N} c_k x_I^{k}\\
     &=&  \sum_{k=1}^{K_N} c_k  {\mathbb E}_I x_I^{k} \\
     &\leq& \sum_{k=1}^{K_N} c_k   \| \psi \|_{\ell_2({\cal T})}^{2k}\\
     &=& S_{N}( \| \psi \|_{\ell_2({\cal T})}^2)  \\
     &\leq& ( \| \psi \|_{\ell_2({\cal T})}^2)^{1/2} + {\varepsilon}_{N} . 
\end{eqnarray*}
where we used ${\mathbb E}_I x_I^{k} \leq  \| \psi \|_{\ell_2({\cal T})}^{2k}$ from the previous Lemma.
Similarly,
\begin{eqnarray*}
     {\mathbb E}_I S_N( x_I )   &=&  \sum_{k=1}^{K_N} c_k  {\mathbb E}_I x_I^{k} \\
     &\geq& \sum_{k=1}^{K_N} c_k   \cdot \bar{B}(k-1; n,p) \cdot \| \psi \|_{\ell_2({\cal T})}^{2k}\\
     &\geq& \bar{B}(K_N-1; n,p) \cdot \sum_{k=1}^{K} c_k   \cdot \| \psi \|_{\ell_2({\cal T})}^{2k}\\
     &=& \bar{B}(K_N-1; n,p)  \cdot S_K( \| \psi \|_{\ell_2({\cal T})}^2)  \\
     &\geq&  \bar{B}(K_N-1; n,p)  \cdot \left( ( \| \psi \|_{\ell_2({\cal T})}^2)^{1/2} - {\varepsilon}_N \right)  \\
   &\geq&  (1- N^{-1}) \cdot \left( ( \| \psi \|_{\ell_2({\cal T})}^2)^{1/2} - {\varepsilon}_N \right)  . 
\end{eqnarray*}
Here  we used ${\mathbb E}_I x_I^{k} \geq  \| \psi \|_{\ell_2({\cal T})}^{2k} \cdot \bar{B}(k-1;n,p)$ from the previous Lemma.
We also used the monotonicity 
of the survivor function $\bar{B}(x) \geq \bar{B}(x+1)$.
We obtain (\ref{eq:LemmConclu}):
\[
 (1-N^{-1}) \cdot(1 - {\varepsilon}_N)  \leq  \frac{{\mathbb E}_I  \| \psi \|_{\ell_2(I)} }{\| \psi \|_{\ell_2({\cal T})}} \leq 1 + {\varepsilon}_N .
\]
\end{proof}


We define the functional  $\Gamma: \psi  \mapsto \frac{{\mathbb E}_I  \| \psi \|_{\ell_2(I)} }{\| \psi \|_{\ell_2({\cal T})}}$,
from the previous calculation we know $\Gamma(\psi) $ is only a function of $\| \psi\|_0$. 
So we can identify  $\Gamma(\psi) $ as  $\Gamma(\| \psi\|_0)$. 





\subsection{Sharper bounds on $\frac{{\mathbb E}_I  \| \psi \|_{\ell_2(I)} }{\| \psi \|_{\ell_2({\cal T})}}$}




\begin{lemma}
\label{TaylorSqrt}
For $ 0 \leq x\leq 1,$
\begin{align}
(1-x)^{1/2}
=1-\sum_{\ell=0}^\infty\frac2{\ell+1}\binom{2\ell}{\ell}\left(\frac x4\right)^{\ell+1}
\end{align}
\end{lemma}

\begin{proof}
From generalized binomial theorem, we know that for $ 0 \leq x\leq 1,$
\begin{equation}
(1-x)^{1/2}
=\sum_{\ell=0}^\infty\binom{1/2}{\ell}(-x)^\ell
\end{equation}

\begin{align}
\binom{1/2}{\ell}
&=\frac{\frac12(\frac12-1)(\frac12-2)\cdots(\frac12-\ell+1)}{\ell!}\\
&=\frac{(-1)^{\ell-1}}{2^\ell\ell!}1\cdot3\cdot5\cdots(2\ell-3)\\
&=\frac{(-1)^{\ell-1}}{2^\ell\ell!}\frac{(2\ell-2)!}{2^{\ell-1}(\ell-1)!}\\
&=\frac{(-1)^{\ell-1}}{\ell2^{2\ell-1}}\binom{2\ell-2}{\ell-1}
\end{align}
then
\begin{align}
(1-x)^{1/2}
&=1-\sum_{\ell=1}^\infty\frac2\ell\binom{2\ell-2}{\ell-1}\left(\frac x4\right)^\ell\\
&=1-\sum_{\ell=0}^\infty\frac2{\ell+1}\binom{2\ell}{\ell}\left(\frac x4\right)^{\ell+1}
\end{align}






\end{proof}

\begin{lemma}
\begin{equation} 
0 \leq 1- Q(N, p)\leq  \frac{{\mathbb E}_I  \| \psi \|_{\ell_2(I)} }{\| \psi \|_{\ell_2({\cal T})}} \leq 1 .
\end{equation}
where 
\[
Q(N, p) := \sum_{\ell=0}^N\frac{2^{-(2\ell+1)}}{\ell+1}\binom{2\ell}{\ell} \bar{B}(\ell;N,1-p),
\]
\[
\bar{B}(\ell;N,1-p) \equiv  \sum_{n=0}^{N-\ell-1} (1-p)^n p^{N-n} {N \choose n}.
\]
\end{lemma}
\begin{proof}
For any fixed support $J\subset {\cal T},$ 
\begin{equation} 
 \frac{ \| \psi \|_{\ell_2(J)} }{\| \psi \|_{\ell_2({\cal T})}} 
 = \sqrt{ \frac{ \| \psi \|^2_{\ell_2(J)} }{\| \psi \|^2_{\ell_2({\cal T})}}} \leq 1 .
\end{equation}
Therefore, 
\begin{equation} 
{\mathbb E}_I \frac{ \| \psi \|_{\ell_2(I)} }{\| \psi \|_{\ell_2({\cal T})}} 
 = {\mathbb E}_I \sqrt{ \frac{ \| \psi \|^2_{\ell_2(I)} }{\| \psi \|^2_{\ell_2({\cal T})}}} \leq {\mathbb E}_I 1 = 1 .
\end{equation}

On the other hand, let 
\begin{equation} 
{\mathbb E}_I \frac{ \| \psi \|_{\ell_2(I)} }{\| \psi \|_{\ell_2({\cal T})}} 
 = {\mathbb E}_I \sqrt{ 1- \frac{ \| \psi \|^2_{\ell_2({\cal T}-I)} }{\| \psi \|^2_{\ell_2({\cal T})}}}  .
\end{equation}

For a fixed subset $J \subset [n]$ define 
the ratio $\rho_J \equiv  \frac{ \| \psi \|^2_{\ell_2({\cal T}-J)} }{\| \psi \|^2_{\ell_2({\cal T})}}$; it 
obeys $0 \leq \rho_J \leq 1$.

Applying Lemma \ref{TaylorSqrt},
\begin{align}
(1-\rho_J)^{1/2}
=1-\sum_{\ell=0}^\infty\frac{2^{-(2\ell+1)}}{\ell+1}\binom{2\ell}{\ell}
\rho_J^{\ell+1}
\end{align}
Now with $I$ a random subset as earlier, we
induce a random variable $\rho_I$.
\begin{align}
{\mathbb E}_I(1-\rho_I)^{1/2}
=1-\sum_{\ell=0}^\infty\frac{2^{-(2\ell+1)}}{\ell+1}\binom{2\ell}{\ell}
{\mathbb E}_I \rho_I^{\ell+1}
\end{align}
Modify the argument behind Lemma \ref{lem:BGPropLemmaSub} so it applies to ${\mathbb E}_I \rho_I^{\ell+1}$. In the new argument the set $I$ is everywhere replaced by its complement $T-I$. The conclusion is correspondingly modified so that $p$ in the final formula is replaced by $q \equiv 1-p$.
We obtain:
\[
 \bar{B}(\ell;n,1-p)    \leq {\mathbb E}_I \rho_I^{\ell+1} \leq    1,
\]
where, if $k \in \{0,1,\dots,N\}$,
$\bar{B}(k-1;N,p) \equiv  \sum_{n=k}^{N} (1-p)^n p^{N-n} {N \choose n}  = 1- \sum_{n=0}^{k-1} (1-p)^n p^{N-n} {N \choose n}$, while if $k \geq N$, $\bar{B}(k) = 0$, and if $k < 0$,
$\bar{B}(k) =1$.
Then
\begin{align}
{\mathbb E}_I(1-\rho_I)^{1/2}
&=1-\sum_{\ell=0}^\infty\frac{2^{-(2\ell+1)}}{\ell+1}\binom{2\ell}{\ell}
{\mathbb E}_I \rho_I^{\ell+1}\\
&\geq  1-\sum_{\ell=0}^\infty\frac{2^{-(2\ell+1)}}{\ell+1}\binom{2\ell}{\ell} \bar{B}(\ell;N,1-p)\\
&=1- \sum_{\ell=0}^N\frac{2^{-(2\ell+1)}}{\ell+1}\binom{2\ell}{\ell} \bar{B}(\ell;N,1-p)\\
&= 1- Q(N, p)
\end{align}
{\bf Qingyun NOTE: I observe numerically that $Q(N,p)$ could be between $0$ and $1$ in a wide range, and $Q(N,p)$ is monotonically increasing with $N$, and monotonically decreasing with $p$, with $Q(N,1) = 0$, $\lim_{N\rightarrow \infty}\lim_{p\rightarrow 0} Q(N,p) = 1.$
.}
\end{proof}

\paragraph{Beta function and $Q(N,p)$}
If we define the complete beta function as 
\[
\mathcal{B}(a,b) := \int_0^1 t^{a-1}\,(1-t)^{b-1}\,dt.
\]
then
\[
\binom{n}{k} = \frac{1}{(n+1) \mathcal{B}(n-k+1, k+1)}
\]
the incomplete beta function as
\[
\mathcal{B}(x;\,a,b) = \int_0^x t^{a-1}\,(1-t)^{b-1}\,dt.
\]
then the regularized incomplete beta function is defined in terms of the incomplete beta function and the complete beta function:
\[
I_x(a,b) = \frac{\mathcal{B}(x;\,a,b)}{\mathcal{B}(a,b)}.
\]
then
\[
I_p(k,N-k-1)= \bar{B}(k-1;N,p) = \sum_{n=k}^{N} (1-p)^n p^{N-n} {N \choose n}
\]
and
\[
I_{1-p}(k+1, N-k) = 1 - I_p(N-k, k+1) = \bar{B}(k;N,1-p).
\]
Let $\mu = N \cdot (1-p)$ and 
$\sigma = \sqrt{N \cdot p \cdot (1-p)}$.

We note that, for large $N$, from the classical De Moivre–Laplace theorem on normal approximation to the binomial, 
\[
{N \choose k}\, p^k q^{N-k} \approx \frac{1}{\sqrt{2 \pi Np(1-p)}}\,e^{-\frac{(k-Np)^2}{2Np(1-p)}}
\]
therefore, 
\[
I_{1-p}(\ell+1, N-\ell) = \bar{B}(\ell;N,1-p)  \approx \bar{\Phi}( (\ell - \mu)/\sigma ) ,
\]
where $\Phi$ denotes the standard Normal $N(0,1)$ CDF
and $\bar{\Phi} \equiv 1-\Phi$ 
the corresponding survivor function;

\begin{eqnarray*}
Q(N,p) 
&\equiv& \sum_{\ell=0}^N\frac{2^{-(2\ell+1)}}{\ell+1}\binom{2\ell}{\ell} \bar{B}(\ell;N,1-p) \\
&=& \sum_{\ell=0}^N\frac{2^{-(2\ell+1)}}{\ell+1} \frac{1}{(2\ell+1) \mathcal{B}(\ell+1, \ell+1)}  I_{1-p}(\ell+1, N-\ell)\\
&\approx& \sum_{\ell=0}^N\frac{1}{2(\ell+1)} \frac{1}{\sqrt{\frac{\pi}{2} N}}\,e^{-\frac{(\ell-\frac{N}{2})^2}{\frac{N}{2}}}  \bar{\Phi}( (\ell - N \cdot (1-p))/\sqrt{N \cdot p \cdot (1-p)} )\\
&\approx& \int_0^1 \frac{1}{2(Nt+1)} \frac{1}{\sqrt{\frac{\pi}{2} N}}\,e^{- 2N(t-\frac{1}{2})^2}  \bar{\Phi}( \frac{\sqrt{N}(t - (1-p))}{\sqrt{ p \cdot (1-p)}} ) \,dt
\end{eqnarray*}

Numerically, we can see that the first term in the sum is always $0.5.$
Also we know that by definition $Q(N, 1) = 1$ for any $N$.
From numerically calculation we guess that
$Q(N,p)$ is monotonically increasing with $N$, and monotonically decreasing with $p$.
\[
Q(N,p') \geq Q(N,p), \quad, p>p'
\]
and 
$\lim_{N\rightarrow \infty}\lim_{p\rightarrow 0} Q(N,p) = 1.$


\begin{lemma}
Define:
\[
Q(N,p) \equiv \sum_{\ell=0}^N\frac{2^{-(2\ell+1)}}{\ell+1}\binom{2\ell}{\ell} \bar{B}(\ell;N,1-p).
\]
Then, 
\[
Q(N,p') \geq Q(N,p), \quad, p>p'
\]
and 
\[
\lim_{N\rightarrow \infty}\lim_{p\rightarrow 0} Q(N,p) = 1.
\]
\end{lemma}


\begin{proof}
The first term in the sum is $0.5$.

(Sketch). 
{\bf The sum has the lower bound $Q \geq 1/10$ just by inspecting
the first 10 terms (say), which are all positive and all greater
than $0.01$ (say). The following argument shows that later terms
add up to at least $\Omega(N^{-1/2})$.}
First, we know that
\[
\bar{B}(\ell;N,1-p) \equiv  \sum_{n=0}^{N-\ell-1} (1-p)^n p^{N-n} {N \choose n}.
\]

Let $\mu = N \cdot (1-p)$ and 
$\sigma = \sqrt{N \cdot p \cdot (1-p)}$.

We note that, for large $N$ 
\[
\bar{B}(\ell;N,1-p)  \approx \bar{\Phi}( (\ell - \mu)/\sigma ) ,
\]
where $\Phi$ denotes the standard Normal $N(0,1)$ CDF
and $\bar{\Phi} \equiv 1-\Phi$ 
the corresponding survivor function;
this is the classical De Moivre theorem on normal approximation to the binomial. It can be made precise in many ways
that have been well documented over the centuries.
Roughly speaking, in terms of variable $z = (\ell - \mu)/\sigma$,
we have
\[
\bar{B}(\ell;N,1-p)  \approx \left \{ \begin{array}{ll}
1 & z < 3 \\
0 & z > 3 .
\end{array}
\right .
\]

Hence, defining for $m = \mu - 3 \sigma$
\[
\bar{B}(\ell;N,1-p) \approx 1, \mbox{ for } \ell \leq m.
\]
Now of course
\begin{eqnarray*}
Q(N,p) &\geq& \sum_{\ell=0}^m\frac{2^{-(2\ell+1)}}{\ell+1}\binom{2\ell}{\ell} \bar{B}(\ell;N,1-p) \\
&\geq& \sum_{\ell=0}^m\frac{2^{-(2\ell+1)}}{\ell+1}\binom{2\ell}{\ell} .
\end{eqnarray*}
Now from DeMoivre's theorem (case $p=1/2$, $N=2\ell$) we also know that
\[
    2^{-2\ell} { 2 \ell \choose \ell} = bin(\ell; 2 \ell, 1/2) \sim \sqrt{\frac{1}{\ell \cdot \pi}},
\]
where $bin(k;N,p)$ is the usual binomial probability mass function.
We obtain
\[
  \sum_{\ell=0}^m\frac{2^{-(1)}}{\ell+1} \ell^{-1/2} \geq 
  \sum_{\ell=m/2}^m\frac{2^{-(1)}}{\ell+1} \ell^{-1/2}  \geq 
  Const  \cdot m^{-1/2}
\]

\end{proof}





\subsection{Bounds on $\frac{{\mathbb E}_I  \| \psi \|^{-1}_{\ell_2(I)} }{\| \psi \|^{-1}_{\ell_2({\cal T})}}$}
\begin{lemma}
\label{TaylorSqrtInv}
For $ 0 \leq x\leq 1,$
\begin{align}
(1-x)^{-1/2}
=\sum_{\ell=0}^\infty\frac{1}{2^{2\ell}}\binom{2\ell}{\ell}\left(x\right)^{\ell}
\end{align}
\end{lemma}

\begin{proof}
From generalized binomial theorem, we know that for $ 0 \leq x\leq 1,$
\begin{equation}
(1-x)^{-1/2}
=\sum_{\ell=0}^\infty\binom{-1/2}{\ell}(-x)^\ell
\end{equation}

\begin{align}
\binom{-1/2}{\ell}(-1)^\ell
&=\frac{-\frac12(-\frac12-1)(-\frac12-2)\cdots(-\frac12-\ell+1)}{\ell!}\\
&=\frac{1}{2^\ell\ell!}1\cdot3\cdot5\cdots(2\ell-1)\\
&=\frac{1}{2^\ell\ell!}\frac{(2\ell)!}{2^{\ell}(\ell)!}\\
&=\frac{1}{2^{2\ell}}\binom{2\ell}{\ell}
\end{align}
\end{proof}

\begin{lemma}
\label{TaylorSqrtInvExpect}
\begin{equation} 
1+\sum_{\ell=1}^\infty\frac{1}{2^{2\ell}}\binom{2\ell}{\ell}\bar{B}(\ell-1;N,1-p) \geq  \frac{{\mathbb E}_I  \| \psi \|^{-1}_{\ell_2(I)} }{\| \psi \|^{-1}_{\ell_2({\cal T})}} \geq 1 .
\end{equation}
where 
\[
\bar{B}(\ell-1;N,1-p) \equiv  \sum_{n=0}^{N-\ell} (1-p)^n p^{N-n} {N \choose n} 
\]
\end{lemma}
\begin{proof}
For any fixed support $J\subset {\cal T},$ 
\begin{equation} 
 \frac{ \| \psi \|^{-1}_{\ell_2(J)} }{\| \psi \|^{-1}_{\ell_2({\cal T})}} 
 = \sqrt{ \frac{ \| \psi \|^2_{\ell_2({\cal T})} }{\| \psi \|^2_{\ell_2(J)}}} \geq 1 .
\end{equation}
Therefore, 
\begin{equation} 
\frac{{\mathbb E}_I  \| \psi \|^{-1}_{\ell_2(I)} }{\| \psi \|^{-1}_{\ell_2({\cal T})}}  
 \geq   1 .
\end{equation}

On the other hand, let 
\begin{equation} 
{\mathbb E}_I (\frac{ \| \psi \|_{\ell_2(I)} }{\| \psi \|_{\ell_2({\cal T})}} )^{-1}
 = {\mathbb E}_I ( 1- \frac{ \| \psi \|^2_{\ell_2({\cal T}-I)} }{\| \psi \|^2_{\ell_2({\cal T})}})^{-1/2}.
\end{equation}

For a fixed subset $J \subset [n]$ define 
the ratio $\rho_J \equiv  \frac{ \| \psi \|^2_{\ell_2({\cal T}-J)} }{\| \psi \|^2_{\ell_2({\cal T})}}$; it 
obeys $0 \leq \rho_J \leq 1$.


\begin{align}
(1-\rho_J)^{-1/2}
=\sum_{\ell=0}^\infty\frac{1}{2^{2\ell}}\binom{2\ell}{\ell}\left(\rho_J\right)^{\ell}
\end{align}
Now with $I$ a random subset as earlier, we
induce a random variable $\rho_I$.
\begin{align}
{\mathbb E}_I(1-\rho_J)^{-1/2}
=\sum_{\ell=0}^\infty\frac{1}{2^{2\ell}}\binom{2\ell}{\ell}{\mathbb E}_I\left(\rho_J\right)^{\ell}
\end{align}
Modify the argument behind Lemma \ref{lem:BGPropLemmaSub} so it applies to ${\mathbb E}_I \rho_I^{\ell+1}$. In the new argument the set $I$ is everywhere replaced by its complement $T-I$. The conclusion is correspondingly modified so that $p$ in the final formula is replaced by $q \equiv 1-p$.
We obtain:
\[
 \bar{B}(\ell-1;n,1-p)    \leq {\mathbb E}_I \rho_I^{\ell} \leq    1,
\]
where, if $k \in \{0,1,\dots,N\}$,
$\bar{B}(k-1;N,p) \equiv  \sum_{n=k}^{N} (1-p)^n p^{N-n} {N \choose n}  = 1- \sum_{n=0}^{k-1} (1-p)^n p^{N-n} {N \choose n}$, while if $k \geq N$, $\bar{B}(k) = 0$, and if $k < 0$,
$\bar{B}(k) =1$.
Then
\[
\bar{B}(\ell-1;N,1-p) \equiv  \sum_{n=0}^{N-\ell} (1-p)^n p^{N-n} {N \choose n} 
\]
\begin{align}
{\mathbb E}_I(1-\rho_I)^{1/2}
&=\sum_{\ell=0}^\infty\frac{1}{2^{2\ell}}\binom{2\ell}{\ell}{\mathbb E}_I\left(\rho_J\right)^{\ell}
\\
&\geq  \sum_{\ell=0}^\infty\frac{1}{2^{2\ell}}\binom{2\ell}{\ell}\bar{B}(\ell-1;N,1-p)\\
&=1+\sum_{\ell=1}^\infty\frac{1}{2^{2\ell}}\binom{2\ell}{\ell}\bar{B}(\ell-1;N,1-p)
\end{align}

\end{proof}





\subsection{Numerically calculate ${\mathbb E}_I  \| \psi_I \|_2/\| \psi \|_2$}

We numerically compute $Q(N,p)$, the result is in the following table. Compare the table with the tables of MC approximation, we see that our previous bound is still too loose.
\begin{table}[]
\begin{tabular}{|p{3cm}||llllllll}
\hline
\multicolumn{9}{|c|}{$Q(N,p)$} \\
\hline
  $p=$ &0   & 0.1   & 0.2       &   0.5     &   0.9     &   0.99     &  0.999 &1      \\
 \hline\hline
$N=100$ &0.9439 & 0.9406 & 0.9370 & 0.920  & 0.8172 & 0.3358 & 0.0482 & 0 \\
\hline
$N=500$ & 0.9748  & 0.9734 & 0.9718 & 0.9643 & 0.9199 & 0.7302 & 0.209 & 0 \\
\hline      
\end{tabular}
\label{Q_table}
\end{table}

Then we use MC approximation to calculate ${\mathbb E}_I  \| \psi_I \|_2/\|  \psi \|_2$ for a few different $\psi.$
We will know that
\begin{itemize}
\item 
When $\psi = e_0$,  ${\mathbb E}_I  \| (e_0)_I \|_2=p$ 

\item
When $\psi = e_0+e_1$,  
\[
{\mathbb E}_I  \|(e_0+e_1)_I \|_2 /\| e_0+e_1 \|_2 = \sqrt{\frac12} {\mathbb E}_I  \| (e_0+e_1)_I \|_2 = 
{\mathbb E}_I \sqrt{\frac{\sum_{i=\{0,1\}} I_i}{2}}=p^2+ 2p(1-p)\sqrt{\frac12}
\]
\item
We know that when $\psi = a =(1,\ldots,1)$, then
\[
{\mathbb E}_I  \| a_I \|_2/\| a \|_2={\mathbb E}_I \sqrt{\frac{\sum_{i=[N]} I_i}{N}} 
\]
Using law of large numbers, 
\[
\lim_{N\rightarrow \infty }\sqrt{\frac{\sum_{i=[N]} I_i}{N}}  = \sqrt{p}.
\]
\item
For fixed support $J$, we can choose $\psi = \sum_{i\in J} e_i,$ then $\| \psi_J \|_2/\|  \psi \|_2 =1.$
However, the expectation is not going to be one. Let $M := |J|,$
\[
{\mathbb E}_I \sqrt{\frac{\sum_{i\in J} I_i}{|J|}} = \sum_{k=0}^{M} (1-p)^{M-k} p^{k} {M \choose k}\sqrt{\frac{k}{M}}\leq \sqrt{p}
\]
The inequality can be seeing using central limit theorem, we have normal approximation  $\frac{\sum_{i\in J} I_i}{|J|} \sim N(p,  \frac{p(1-p)}{\sqrt{|J|}})$, then using Jensen's inequality, since square root fucntion is concave,
\[
{\mathbb E}_I \sqrt{\frac{\sum_{i\in J} I_i}{|J|}} \approx {\mathbb E}_G\sqrt{p+ \frac{p(1-p)}{\sqrt{|J|}}G}  \leq \sqrt{{\mathbb E}_G[p+ \frac{p(1-p)}{\sqrt{|J|}}G]} =\sqrt{p} 
\]
\end{itemize}
Using symmetry, we guess that the maximum value of ${\mathbb E}_I  \| \psi_I \|_2/\|  \psi \|_2$  is $\sqrt{p}$ instead of $1$, and the maximum value is achieved when $\psi$ is dense.

\begin{table}[]
\begin{tabular}{|p{3cm}||llllllll}
\hline
\multicolumn{9}{|c|}{MC approximation ($M=10^4$) of ${\mathbb E}_I  \| a_I \|_2/\| a \|_2$ where $a=(1,\ldots,1)$} \\
\hline
  $p=$ & 0   & 0.1   & 0.2       &   0.5     &   0.9     &   0.99     &  0.999 &1      \\
 \hline\hline
$N=100$ &0 & 0.3123 & 0.4460 & 0.7063  & 0.94865 & 0.9950 & 0.9995 & 1 \\
\hline
$N=500$ &0 &0.3153  & 0.4464 & 0.7068 & 0.94874 & 0.9950 & 0.9995 & 1\\
\hline      
\end{tabular}
\label{allone_table}
\end{table}


\begin{table}[]
\begin{tabular}{|p{3cm}||llllllll}
\hline
\multicolumn{9}{|c|}{MC approximation ($M=10^4$) of ${\mathbb E}_I  \|(e_0+e_1)_I \|_2 /\| e_0+e_1 \|_2$} \\
\hline
  $p=$ & 0   & 0.1   & 0.2       &   0.5     &   0.9     &   0.99     &  0.999 &1      \\
 \hline\hline
$N=500$ & 0 &0.1375 &	0.2667 &	0.6034 &	0.9365 &	0.9946 &	0.9994 & 1 \\
\hline      
\end{tabular}
\label{coordinatevector_table}
\end{table}



\subsection{Tight bound and stationary points of ${\mathbb E}_I  \| \psi_I \|_2/\| \psi \|_2$}
\begin{lemma}
\begin{equation} 
p \leq  \frac{{\mathbb E}_I  \| \psi \|_{\ell_2(I)} }{\| \psi \|_{\ell_2({\cal T})}} \leq \sqrt{p}  .
\end{equation}
where the lower bound is achieved by $1-$sparse vectors $\pm e_i$, the upper bound is achieve by $\{\frac{\sum_{i\in {\cal T}} \pm e_i}{\sqrt{N}}\}$.
Furthermore, all the stationary points of this objective are $\{\frac{\sum_{i\in J} \pm e_i}{\sqrt{|J|}}\}$ for different support $J\subset {\cal T}$. 
\end{lemma}
\begin{proof}
This problem can be reformulate as the projection pursuit with sphere constraint,
 \begin{eqnarray*}
    \min&   {\mathbb E}_I  \| \psi_I \|_2\\
    \mbox{ subject to }&   \| \psi \|_2 = 1.
\end{eqnarray*}
Then from \cite{bai2018subgradient} Proposition $3.3$, we get the result. 

The main idea of the proof is to calculate the projected gradient
for $q\in \{\frac{\sum_{i\in J} \pm e_i}{\sqrt{|J|}}\}$, when $|J| = M,$
\[
e_j^T  {\mathbb E}_I  \partial\| q_I \|_2 = q_j [\sum_{k=0}^{M} (1-p)^{M-k} p^{k} {M \choose k}\sqrt{\frac{k}{M}}] 
\]
so
\[
(I-q q^T)  {\mathbb E}_I  \partial\| q_I \|_2 = [\sum_{k=0}^{M} (1-p)^{M-k} p^{k} {M \choose k}\sqrt{\frac{k}{M}}]q-[\sum_{k=0}^{M} (1-p)^{M-k} p^{k} {M \choose k}\sqrt{\frac{k}{M}}]q = 0.
\]
then $q\in \{\frac{\sum_{i\in J} \pm e_i}{\sqrt{|J|}}\}$ is stationary points.
 
The other direction (all other points are not stationary) is implied by (the proof of) theorem $3.4$ in  \cite{bai2018subgradient}.
\end{proof}
\section{Expectation calculation}
\subsection{Expectation over Bernoulli support}

We previously considered the identity
\begin{equation} \label{eq:startingPoint}
    {\mathbb E} | \sum_{i \in {\cal T}}  \psi_i X_i | = \sqrt{\frac{2}{\pi}} \cdot {\mathbb E}_I \| \psi \|_{\ell_2(I)},
\end{equation}
where $X$ is a Bernoulli-Gaussian RV $BG(p;0,1)$,
and where $I$ is a random subset of the domain ${\cal T}$
determined by Bernoulli-$p$ coin tossing.

Let $J \subset {\cal T}$ be the support of $X$, where $  N:=|{\cal T}|$, $N_J:=|J|$. Let $\psi_J : = \psi\cdot 1_J$ denote the entrywise product of vector $\psi$ with the indicator vector of subset $J$. Then  we can consider our problem on the space ${\cal T}$ by default, and rewrite for simplicity
\[
\| \psi_J \|_{2} : = \| \psi_J \|_{\ell_2({\cal T})} = \| \psi \|_{\ell_2(J)}
\]

This leads us to consider the following ratio:
\[
  V^J_k(\psi) := \frac{\| \psi \|_{\ell_2(J)}^{k}}{\| \psi \|_{\ell_2({\cal T})}^{k}}
 \]
 and its expectation over all Bernoulli random subset,
\[
  V_k(\psi) := \frac{{\mathbb E}_I \| \psi \|_{\ell_2(I)}^{k}}{\| \psi \|_{\ell_2({\cal T})}^{k}}
 \]
 where $I$ is again a random subset.
We remark that lower bound of $V_k(\psi)$ is the optimal value of the optimization problem:
 \begin{eqnarray*}
    \min&   {\mathbb E}_I  \| \psi_I \|^k_2\\
    \mbox{ subject to }&   \| \psi \|_2 = 1.
\end{eqnarray*}
and the upper bound of $V_k(\psi)$ is the optimal value of the optimization problem
 \begin{eqnarray*}
    \max&   {\mathbb E}_I  \| \psi_I \|^k_2\\
    \mbox{ subject to }&   \| \psi \|_2 = 1.
\end{eqnarray*}




\begin{lemma} \label{lem:BGPropLemmaSub_fix}
Let $\psi$ denote a vector in ${\mathbb R}^N$. For fixed support $J\subset [N]$ then for any $k\neq 0$, the deterministic quantity $V^J_{k}(\psi)$ satistfies
\[
0 \leq V^J_{k}(\psi) \leq 1.
\]
The upper bound is approached by the vectors $\psi  \in \{  \frac{1}{\sqrt{N}}\sum_{j\in J}\pm e_j\}$, where  $\| \psi_J \|_2^k/\|  \psi \|_2^k =1$,
 and the lower bound is approached by 
any $\psi$ that has all of its entries on $J$ to be zero.
\end{lemma}

\begin{lemma} \label{lem:BGPropLemmaSub_2}
Let $\psi$ denote a vector in ${\mathbb R}^N$.
\[
V_2(\psi) =   p.
\]
\end{lemma}



\begin{theorem} \label{thm:BGPropLemmaNorm}
Let $\psi$ denote a vector in ${\mathbb R}^N$, then
\[
p \leq V_{1}(\psi) \leq V_{1}(\frac{\sum_{j\in [N]}\pm e_j}{\sqrt{N}} ) \leq \sqrt{p} .
\]
the lower bound is approached by on-sparse vectors $\psi \in \{ \pm e_i, i\in [N]\}$, and the upper bound is approached by 
$\psi  \in \{  \frac{1}{\sqrt{N}}\sum_{j\in [N]}\pm e_j\}$.

Furthermore, all the stationary points of $V_{1}(\psi)$ are $\{\frac{\sum_{i\in J} \pm e_i}{\sqrt{N_J}}\}$ for different support $J\subset {\cal T}$, where $\{ \pm e_i, i\in [N]\}$ are the set of global minimizers, and $\{  \frac{1}{\sqrt{N}}\sum_{j\in [N]}\pm e_j\}$ are the set of all the global maximizers. And for $J$ with $1<N_J<N$, $\{\frac{\sum_{i\in J} \pm e_i}{\sqrt{N_J}}\}$ are saddles points with value
\[
V_1(\frac{\sum_{i\in J} \pm e_i}{\sqrt{N_J}}) = {\mathbb E}_I \sqrt{\frac{\sum_{i\in J} 1_{I_i}}{N_J}} = \sum_{j=0}^{N_J} (1-p)^{N_J-j} p^{j} {N_J \choose j}\sqrt{\frac{j}{N_J}}.
\]
\end{theorem}
Remark:
Specifically,
\[
V_1( e_i) = p.
\]
\[
V_1(\frac{ e_i+e_j}{\sqrt{2}}) = p(\sqrt{2}+(1-\sqrt{2}p)).
\]



\begin{theorem} \label{thm:BGPropLemmaSub_k}
For $k\geq 2$, let $\psi$ denote a vector in ${\mathbb R}^N$, then
\[
p^k \leq V_{2k}(\frac{\sum_{j\in [N]}\pm e_j}{\sqrt{N}} ) \leq V_{2k}(\psi) \leq p.
\]
the upper bound is approached by on-sparse vectors $\psi \in \{ \pm e_i, i\in [N]\}$, and the lower bound is approached by 
$\psi  \in \{  \frac{1}{\sqrt{N}}\sum_{j\in [N]}\pm e_j\}$.

Furthermore, all the stationary points of $V_{2k}(\psi)$ are $\{\frac{\sum_{i\in J} \pm e_i}{\sqrt{N_J}}\}$ for different support $J\subset {\cal T}$, where $\{ \pm e_i, i\in [N]\}$ are the set of global maximizers, and $\{  \frac{1}{\sqrt{N}}\sum_{j\in [N]}\pm e_j\}$ are the set of all the global minimizers. And for $J$ with $1<N_J<N$, $\{\frac{\sum_{i\in J} \pm e_i}{\sqrt{N_J}}\}$ are saddles points with value
\[
V_{2k}(\frac{\sum_{i\in J} \pm e_i}{\sqrt{N_J}}) = {\mathbb E}_I (\frac{\sum_{i\in J} 1_{I_i}}{N_J})^k = \sum_{j=0}^{N_J} (1-p)^{N_J-j} p^{j} {N_J \choose j}(\frac{j}{N_J})^k.
\]
\end{theorem}







\paragraph{Tight bound on $V_1$ and $V_{2k}$ using mean and variance of $\frac{\sum_{i\in J} 1_{I_i}}{N_J}$}

Now notice that 
\[
\frac{\sum_{i\in J} 1_{I_i}}{N_J} 
\]
has mean $p$ variance $\frac{p(1-p)}{N_J}$.
We could define a mean zero unit variance random variable $g^J$ as a function of $\{1_{I_i}\mid i\in J\}$ 
\[
g^J := \left(\frac{\sum_{i\in J} 1_{I_i}}{N_J}-p\right)/ \left( \sqrt{\frac{p(1-p)}{N_J}} \right)
\]
namely, ${\mathbb E}_I \left( g^I\right)=0, {\mathbb E}_I \left( g^I\right)^{2}=1$.
then
\[
\frac{\sum_{i\in J} 1_{I_i}}{N_J} := p + \sqrt{\frac{p(1-p)}{N_J}} g^J = p(1 + \sqrt{\frac{1-p}{p N_J}} g^J)
\]
then
\[
{\mathbb E}_I \sqrt{\frac{\sum_{i\in J} 1_{I_i}}{N_J}} = {\mathbb E}_I\sqrt{p- \sqrt{\frac{p(1-p)}{N_J}}g^I} 
= \sqrt{p} {\mathbb E}_I\sqrt{1+ \sqrt{\frac{1-p}{pN_J}}g^I}  
\]
\[
{\mathbb E}_I \left(\frac{\sum_{i\in J} 1_{I_i}}{N_J}\right)^k 
= p^k {\mathbb E}_I\left({1+ \sqrt{\frac{1-p}{pN_J}}g^I}\right)^k 
\]
\begin{lemma}
\label{TaylorSqrt}
For $ 0 \leq x\leq 1,$
\begin{align}
(1-x)^{1/2}
=1-\sum_{\ell=0}^\infty\frac2{\ell+1}\binom{2\ell}{\ell}\left(\frac x4\right)^{\ell+1}
\end{align}
\end{lemma}

\begin{lemma}[Taylor expansion for $V_1$]
\label{TaylorSqrt_Expect}
and using Taylor expansion, asymptotically when $N_J\rightarrow \infty$, 
\begin{eqnarray*}
\frac{1}{\sqrt{p}}{\mathbb E}_I \sqrt{\frac{\sum_{i\in J} 1_{I_i}}{N_J}} 
&=& {\mathbb E}_I\left({1+ \sqrt{\frac{1-p}{pN_J}}g^I}\right)^{1/2} \\
&=& 1-\sum_{\ell=0}^\infty\frac2{\ell+1}\binom{2\ell}{\ell}2^{-2(\ell+1)}\left( \sqrt{\frac{1-p}{pN_J}}\right)^{\ell+1}{\mathbb E}_I \left( g^I\right)^{\ell+1}\\
&=& 1- 2^{-3}({\frac{1-p}{pN_J}}) +O\left( ({\frac{1-p}{pN_J}})^2 \right)
\end{eqnarray*}
\end{lemma}

\begin{lemma}[Taylor expansion for $V_{2k}$]
\label{TaylorSqrt_Expect_k}
and using Taylor expansion, asymptotically when $N_J\rightarrow \infty$, 
\begin{eqnarray*}
\frac{1}{p^k}{\mathbb E}_I ({\frac{\sum_{i\in J} 1_{I_i}}{N_J}})^k  
&=& {\mathbb E}_I\left({1+ \sqrt{\frac{1-p}{pN_J}}g^I}\right)^{k} \\
&=& 1 +\sum_{\ell=1}^{k} \binom{k}{\ell} ({\frac{1-p}{pN_J}})^{\ell/2} {\mathbb E}_I (g^I)^{\ell}\\
&=& 1 + \frac{k(k-1)}{2} ({\frac{1-p}{pN_J}}) +O\left( ({\frac{1-p}{pN_J}})^2 \right)
\end{eqnarray*}
\end{lemma}
Therefore, we know what when $N_J>>1$, 
$V_1(\frac{\sum_{i\in J} 1_{I_i}}{N_J})$ is close to its upper bound $\sqrt{p}$, 
and for $k\geq 2$, 
$V_k(\frac{\sum_{i\in J} 1_{I_i}}{N_J})$ is close to its lower bound $p^k$ namely 
\[
{\mathbb E}_I \sqrt{\frac{\sum_{i\in J} 1_{I_i}}{N_J}} \approx \sqrt{p}
\]
\[
{\mathbb E}_I (\frac{\sum_{i\in J} 1_{I_i}}{N_J})^k \approx p^k
\]





\begin{figure} 
\centering
\includegraphics[width=.7\textwidth]{./figures_BD/ratio/Full_compare_with_baseline.png}
\caption{The value of $V_1(\psi)$ for different $\psi$, $x-$axis is the sparsity parameter $p$, $y-$axis is the $V_1(\psi)$. }
\label{fig-V1}
\end{figure}



\begin{figure} 
\centering
\includegraphics[width=.7\textwidth]{./figures_BD/landscape/L1_conv_Landscape_Sparsity0p1_.png}
\caption{The value of $V_1(\psi)$ for $\psi$ on two dimensional sphere in ${\mathbb R}^3$, normalized by the affine transform to send the value in $[0,1].$ . }
\label{fig-landscape}
\end{figure}



\begin{figure} 
\centering
\includegraphics[width=.7\textwidth]{./figures_BD/ratio/L4_plot.png}
\caption{The value of $V_4(\psi)$ for different $\psi$, $x-$axis is the sparsity parameter $p$, $y-$axis is the $V_4(\psi)$. }
\label{fig-V4}
\end{figure}



\begin{figure} 
\centering
\includegraphics[width=.7\textwidth]{./figures_BD/ratio/L8_plot.png}
\caption{The value of $V_8(\psi)$ for different $\psi$, $x-$axis is the sparsity parameter $p$, $y-$axis is the $V_8(\psi)$. }
\label{fig-V4}
\end{figure}



\subsection{Proofs}


\begin{proof}[Proof of theorem~\ref{thm:BGPropLemmaNorm}]
First, we show that the upper and lower bound value we give is achievable.
\begin{itemize}
\item 
When $\psi = e_0$,  
\[
{\mathbb E}_I  \| (e_0)_I \|_2=p\| (e_0)\|_2+(1-p)\| 0\|_2 = p
\]
\item
When $\psi = \frac{1}{\sqrt{N}}\sum_{j\in [N]}\pm e_j,$ then  
\[
V_1(\psi) = {\mathbb E}_I \sqrt{\frac{\sum_{i\in [N]} 1_{I_i}}{N}} \leq   \sqrt{{\mathbb E}_I \frac{\sum_{i\in [N]} 1_{I_i}}{N}} = \sqrt{p}
\]
The inequality comes from Jensen's inequality, since square root function is concave.
\item
For fixed support $J$, when $\psi = \frac{1}{\sqrt{N_J}}\sum_{i\in J} e_i,$  then $\| \psi_J \|_2/\|  \psi \|_2 =1.$
However, the expectation is going to be smaller for $p<1$. 
\[
V_1(\psi) = {\mathbb E}_I \sqrt{\frac{\sum_{i\in J} 1_{I_i}}{N_J}} = \sum_{k=0}^{N_J} (1-p)^{N_J-k} p^{k} {N_J \choose k}\sqrt{\frac{k}{N_J}}.
\]
\end{itemize}

This problem can be reformulate as the projection pursuit with sphere constraint,
 \begin{eqnarray*}
    \min&   {\mathbb E}_I  \| \psi_I \|_2\\
    \mbox{ subject to }&   \| \psi \|_2 = 1.
\end{eqnarray*}
Then from \cite{bai2018subgradient} Proposition $3.3$, we get the result. 

The main idea of the proof is to calculate the projected gradient
for $q  \in \{\frac{\sum_{i\in J} \pm e_i}{\sqrt{N_J}}\}$, when $N_J = M,$
\[
e_j^T  {\mathbb E}_I  \partial\| (q)_I \|_2 = q_j [\sum_{k=0}^{M} (1-p)^{M-k} p^{k} {M \choose k}\sqrt{\frac{k}{M}}] 
\]
therefore,
\[
  {\mathbb E}_I  \partial\| (q)_I \|_2 =  [\sum_{k=0}^{M} (1-p)^{M-k} p^{k} {M \choose k}\sqrt{\frac{k}{M}}] q
\]
so
\[
(I-q q^T)  {\mathbb E}_I  \partial\| q_I \|_2 = [\sum_{k=0}^{M} (1-p)^{M-k} p^{k} {M \choose k}\sqrt{\frac{k}{M}}]q -[\sum_{k=0}^{M} (1-p)^{M-k} p^{k} {M \choose k}\sqrt{\frac{k}{M}}]q  = 0.
\]
then $q \in \{\frac{\sum_{i\in J} \pm e_i}{\sqrt{N_J}}\}$ is stationary points.
 
The other direction (all other points are not stationary) is implied by (the proof of) theorem $3.4$ in  \cite{bai2018subgradient}.
\end{proof}





\begin{proof}[Proof of lemma~\ref{TaylorSqrt}]
From generalized binomial theorem, we know that for $ 0 \leq x\leq 1,$
\begin{equation}
(1-x)^{1/2}
=\sum_{\ell=0}^\infty\binom{1/2}{\ell}(-x)^\ell
\end{equation}

\begin{align}
\binom{1/2}{\ell}
&=\frac{\frac12(\frac12-1)(\frac12-2)\cdots(\frac12-\ell+1)}{\ell!}\\
&=\frac{(-1)^{\ell-1}}{2^\ell\ell!}1\cdot3\cdot5\cdots(2\ell-3)\\
&=\frac{(-1)^{\ell-1}}{2^\ell\ell!}\frac{(2\ell-2)!}{2^{\ell-1}(\ell-1)!}\\
&=\frac{(-1)^{\ell-1}}{\ell2^{2\ell-1}}\binom{2\ell-2}{\ell-1}
\end{align}
then
\begin{align}
(1-x)^{1/2}
&=1-\sum_{\ell=1}^\infty\frac2\ell\binom{2\ell-2}{\ell-1}\left(\frac x4\right)^\ell\\
&=1-\sum_{\ell=0}^\infty\frac2{\ell+1}\binom{2\ell}{\ell}\left(\frac x4\right)^{\ell+1}
\end{align}
\end{proof}







\begin{proof}[Proof of lemma~\ref{TaylorSqrt_Expect}]
\begin{eqnarray*}
{\mathbb E}_I\sqrt{1- \sqrt{\frac{1-p}{pN_J}}g^I}  
&=& 1-\sum_{\ell=0}^\infty\frac2{\ell+1}\binom{2\ell}{\ell}2^{-2(\ell+1)}\left( \sqrt{\frac{1-p}{pN_J}}\right)^{\ell+1}{\mathbb E}_I \left( g^I\right)^{\ell+1}\\
&=& 1- 2^{-3}({\frac{1-p}{pN_J}}) - O(({\frac{1-p}{pN_J}})^2)
\end{eqnarray*}
Using central limit theorem, when $N_J\rightarrow \infty$, we have normal approximation for $g^J$ so that
\[
\frac{\sum_{i\in J} 1_{I_i}}{N_J} \sim N(p,  \sqrt{\frac{p(1-p)}{N_J}})
\]
then for $G\sim N(0,1),$ asymptotically when $N_J\rightarrow \infty$,
\[
{\mathbb E}_I \sqrt{\frac{\sum_{i\in J} 1_{I_i}}{N_J}} \approx  \sqrt{p} {\mathbb E}_G\sqrt{1- \sqrt{\frac{1-p}{pN_J}}G} =
\sqrt{p} \left( 1- 2^{-3}(1-p)^{2}  N_J^{-1} + O((1-p)^{4}  N_J^{-2})\right)
\]
where
\begin{eqnarray*}
{\mathbb E}_I\sqrt{1- \sqrt{\frac{1-p}{pN_J}}g^I}  
&\approx &{\mathbb E}_G\sqrt{1- \sqrt{\frac{1-p}{pN_J}}G}\\
&=& 1-\sum_{\ell=0}^\infty\frac2{\ell+1}\binom{2\ell}{\ell}2^{-2(\ell+1)}\left( \sqrt{\frac{1-p}{pN_J}}\right)^{\ell+1}{\mathbb E}_G \left( G\right)^{\ell+1}\\
&=& 1-\sum_{\ell \mbox{ is odd}}^\infty\frac2{\ell+1}\binom{2\ell}{\ell}2^{-2(\ell+1)}\left( \sqrt{\frac{1-p}{pN_J}}\right)^{\ell+1} (\ell)!!\\
&=& 1-\sum_{s=1}^\infty\frac1{s}\binom{4s-2}{2s-1}2^{-4s} (2s-1)!! (p^{-1}-1)^{s}  N_J^{-s}\\
&=& 1-\sum_{s=1}^\infty\frac1{s}\frac{(4s-2)!}{(2s-1)!(2s-1)!}2^{-4s} \frac{(2s-1)!}{2^{s-1}(s-1)!} (p^{-1}-1)^{s}  N_J^{-s}\\
&=& 1-\sum_{s=1}^\infty\frac{(4s-2)!}{(2s-1)!(s)!}2^{-5s+1}  (p^{-1}-1)^{s}  N_J^{-s}
\end{eqnarray*}
\end{proof}





\subsection{General sparse signal }
we study in detail for the quantity  
\[
E{| \psi^T X| } = E{|\sum_{i} \psi_i X_i| }.
\]
We know when $X_i$ have variance $\sigma^2$,
\[
   E (\sum_i \psi_i X_i)^2 =  \sum_i   \psi_i^2 E(X_i)^2 = \sigma^2 \|\psi\|_2^2.
\]
The ratio indicate the sparsity level of random variable $ \sum_i \psi_i X_i$
\[
\frac{ E{|\sum_{i} \psi_i X_i| }}{\sqrt{E (\sum_i \psi_i X_i)^2}} = \frac{ E{|\sum_{i} \psi_i X_i| }}{\sigma^2 \|\psi\|_2} 
\]
Now, we consider $E{|\sum_{i} \psi_i X_i| }$ for general symmetric setting when lower bound can be derived.

\paragraph{Exact calculation for Bernoulli Gaussian}
Let $X_i = B_i Z_i$ be IID Bernoulli Gaussian, where $B_i$ is sampled from Bernoulli with parameter $p$, $Z_i$ is sampled from $N(0,1)$. Let $I$ be the support where $B_i =1$.
\[
{\mathbb E}_X{|\sum_{i} \psi_i X_i| }  
= {\mathbb E}_B {\mathbb E}_G {|\sum_{i} \psi_i B_i G_i| }
= {\mathbb E}_I {\mathbb E}_G {|\sum_{i\in I} \psi_i G_i| }
\]
Leveraging the invariance property for linear transform of Gaussian, we get
\[
{\mathbb E}_G {|\sum_{i\in I} \psi_i G_i| } = 
\sqrt{\frac{2}{\pi}} \cdot\|\psi_I\|_2.
\]


\paragraph{Symmetric distribution}

Dave Comment:
Put the generalization in the end.
Focus on Bernoulli Gaussian for the main part, and then talk about extension.


\begin{lemma} \label{lem:sym_Khintchine_inequality}
For we only know $\Xi_i$ are IID sampled from a symmetric distribution $F$ with unit variance, then we already have a lower bound,  
\[
     1/\sqrt{2}  \| a\|_2 \leq {\mathbb E} |\sum_i a_i \Xi_i| \leq \|a\|_2.
\] 
\end{lemma}
 \begin{proof}[Proof of Lemma \ref{lem:sym_Khintchine_inequality}]
 The upper bound come from Cauchy inequality.
 
Now we derive the lower bound. We write $\Xi_i = \sigma_i |\Xi_i|$, where $s_i =\pm 1$ with equal probability since $\Xi_i$ are symmetric RV, and $\{s_i,|\Xi_i|\}$  are all independent random variables.  
we could use Khintchine inequality,
here is the Wiki link for more detail:
\begin{quote}
\href{https://en.wikipedia.org/wiki/Khintchine_inequality}{Khintchine inequality}.
\end{quote}




\begin{eqnarray*}
{\mathbb E}_\Xi |\sum_i a_i \Xi_i| &=& E_{|\Xi_i|}E_{s_i} |\sum_i a_i s_i |\Xi_i||\\
&\geq&  
  1/\sqrt{2}  \sum_i a_i^2  E_{|\Xi_i|} |\Xi_i|^2 \\
  &=&   1/\sqrt{2}  \|a\|_2^2.
\end{eqnarray*}
\end{proof}

Let $\psi_I = a$, we have a corollary for Bernoulli Symmetric case,
   \[
 {\mathbb E}_I \|\psi_I\|_2  \geq {\mathbb E}_X{|\sum_{i} \psi_i X_i| }  
\geq \sqrt{\frac{1}{2}} \cdot {\mathbb E}_I \|\psi_I\|_2.
\]


\begin{theorem}
Let $X_i = B_i \Xi_i$ be IID, where $B_i$ is sampled from Bernoulli with parameter $p$, let $I$ be the support where $B_i =1$. And $\Xi_i$ is variance $1$, sampled from: 
\begin{itemize}
    \item $N(0,1)$
   
\item general symmetric distribution with variance $1$.
\end{itemize}

Then 
\begin{itemize}
    \item(for Bernoulli Gaussian:)
    \[
{\mathbb E}_X{|\sum_{i} \psi_i X_i| }  
= \sqrt{\frac{2}{\pi}} \cdot {\mathbb E}_I \|\psi_I\|_2
\]
\item (for Bernoulli symmetric R.V. :) 
    \[
{\mathbb E}_X{|\sum_{i} \psi_i X_i| }  
\geq \sqrt{\frac{1}{2}} \cdot {\mathbb E}_I \|\psi_I\|_2.
\]
\end{itemize}

\end{theorem}








\section{Introduction}

Blind deconvolution is the problem of recovering a kernel and a signal from 
their (noisy or corrupted) convolution.
This is a ubiquitous problem in fields including signal processing, 
machine learning, computer vision, seismology, time series and neuro-science.
Blind deconvolution is also a crucial problem at the core of digital communication that powers our modern society by enabling cell phone communication.

One important example of blind deconvolution is when the signal is sparse, as the sparse blind deconvolution setting. We call this problem \emph{sparse blind deconvolution}. The sparse signal setting of blind deconvolution appears in seismology, microscopy, astronomy, neuro-science spike identification. In representation learning for computer vision, computational imaging and speech analysis, sparse blind deconvolution is also interpreted as single-channel convolutional dictionary learning.



\paragraph{First suprise: convex formulation for blind deconvolution}
Despite the importance of blind deconvolution, until recently there has been little understanding of when, how and why these non-convex algorithms succeed.
Practitioners have accumulated poetic yet vague observations like "when the initialization is close, gradient-based methods find the true signal", but there is no clear understanding of this statement.

In this paper, we proposed a scalable convex optimization framework for blind deconvolution.
Suppose there is a sparse signal $x$ and a forward filter $a$, the observation is their convolution $y$ defined as $y = a * x$. The blind deconvolution is a problem of finding the sparse signal $x$ from the observation $y$ without knowing the forward filter $a$. 
As both the sparse signal and the forward filter $a$ are unknown, the problem is often formulated as a non-convex problem. For example, in large collection of papers \cite{kuo2019geometry, bristow2013fast, heide2015fast,zhang2017global,zhang2018structured}, people designed alternating minimization type algorithms that jointly find a short $a$ on sphere and sparse $x$ from $y$ by solving the following non-convex problem: 
  \[
 \begin{array}{ll}
 \underset{a, x}{\mbox{minimize}}   & \|y-a*x\|_2^2+\lambda \|x\|_1 \\
\mbox{subject to}  & \|a\|_2 = 1.
\end{array}
  \]  

In this work, we formulate this problem as looking for a sparsifying filter $w$ so that $\Tilde{x} = w*y$ is sparse. This formulation is adapted in the earlier work of Wiggins \cite{wiggins1978minimum} and Donoho \cite{donoho1981minimum} for minimum entropy deconvolution, or maximum cumulant deconvolution. The core idea is that  Find $w\approx a^{-1}$, so that $w*y$ is the most non-Gaussian.
\item Find $w\neq 0$  
\[
 \begin{array}{ll}
 \underset{w\neq 0}{\mbox{maximize}}   & J(w*y)
\end{array}
  \]
\item Non-Gaussianity: 
\begin{itemize}
\item
$J(z) = -\mbox{Entropy}(p_z)$ 
\item
$J(z) = \frac{\|z\|_4^4}{\|z\|_2^4}$, normalized $4$-th order cumulant 

by solving the convex optimization problem for a chosen initial guess  $\Tilde{a}$ of the forward filter:
\begin{equation}
 \begin{array}{ll}
 \underset{w}{ \mbox{minimize} }   & \|w*y\|_{1}\\
\mbox{subject to}  
& (\Tilde{a}*w)_0 =1, 
\end{array}
\end{equation}


My work demonstrates that the hidden convex formulation behind the scene leads to the success of these heuristic algorithms. This mathematical formulation gives a crisp, clean view that clears all the fog in the claims of past practitioners. 

\paragraph{Random model for sparse signals}
We define a random model for sparse signals. Let $(X_t)$ be an independent
sequence of random variable indexed by  $t \in Z$, with a symmetric distribution with CDF $F_t$, such that $F_t(x) = 1 - F_t(-x)$. 
And we assume $F_0$ has an atom at $0$, $F_0 = (1-p) H +p G$, where $H$ is Heaviside distribution and $G$ is standard Gaussian distribution, we called it \emph{Bernoulli$(p)$-Gaussian} model. 


For a process $Y = a * X$, where $a\in l_1({\mathbb Z}) ,  
(\mathcal{F} a)(\omega) \neq 0, \forall \omega\in T,$ additionally, $a^{-1}$ is zero outside of a centered window of radius $2k+1$. 

Now given a series of observation $(Y_t)_{t\in [-(T+k),T+k]}$ from a centered window of radius $T+k$, we want to find an inverse filter 
$w = (w_{-k}, \ldots, w_0, \ldots, w_k) \in V$ such that the convolution $w*Y$ is sparse, where $V$ is a $2k+1$ dimensional subspace $V$ of the double infinite sequence. Let $\|\cdot\|_{1,{\cal T}} =\|\cdot\|_{\ell_1({\cal T})}= \|\cdot\|_{1,[-T,T]}  $ be the $1$ norm over the centered window of radius $T$. Let $N:= 2T+1$ be the total length of $X$.



\paragraph{Convex sparse blind deconvolution}
 In this work, we consider a convex formulation of the problem as
\begin{equation}
 \begin{array}{ll}
 \underset{w\in V}{ \mbox{minimize} }   & \frac{1}{N}\|w*Y\|_{\ell_1({\cal T})}\\
\mbox{subject to}  
& v^T w =1, 
\end{array}
\tag{$P_{v}^T$ }
\end{equation}

In the following, we first study the population problem of convex blind deconvolution, which coincident with the population problem of convex sparse dictionary learning, which is also called projection pursuit or convex sparse ICA.

We first prove the existence of population phase transition by analyzing projected sub-gradient and give a characterization of the population phase transition. Then we study finite sample guarantee, noisy perturbation stability, finite length inverter stability.

\paragraph{Application}
Image deblur, neural spike measurement, time series sparse feature transform learning, Seismic application with oil discovery, signal demixing. Audio deconvolution (dereverberation) is a reverberation reduction in audio mixtures. It is part of audio processing of recordings in ill-posed cases such as the cocktail party effect.






\paragraph{Ratio of norm formulation for blind deconvolution}
First, from the paper \cite{donoho1981minimum}, we know that we could minimize the ratio of norm of $X$ to recover $X$.

For example, \cite{donoho1981minimum,wiggins1978minimum} propose to study $(\ell_{2k}/\ell_2)$ problem for $k=2, 3, 4,\ldots$.
\begin{eqnarray*}
    (\ell_{2k}/\ell_2) \qquad  \max&    \frac{1}{N} \| w*Y\|^{2k}_{\ell_{2k}({\cal T})}\\
    \mbox{ subject to }&   \frac{1}{N} \| w*Y\|^2_{\ell_2({\cal T})} \geq 1.
\end{eqnarray*}


For $\ell_1/\ell_2$ ratio, we propose to use a non-convex (convex objective concave constraint) problem for blind deconvultion:
\begin{eqnarray*}
    (\ell_1/\ell_2) \qquad  \min&    \frac{1}{N} \| w*Y\|_{\ell_1({\cal T})}\\
    \mbox{ subject to }&   \frac{1}{N} \| w*Y\|_{\ell_2({\cal T})} \geq 1.
\end{eqnarray*}
This can be further reduced to sphere-constrained optimization
\begin{eqnarray*}
      \min&    \frac{1}{N} \| w*Y\|_{\ell_1({\cal T})}\\
    \mbox{ subject to }&   \| w\|_{2}^2 = 1.
\end{eqnarray*}
If we use iterative linearization of the nonconvex sphere-constrained, on every iteration we could use convex form:
Let $w^0 = e_0$, iteratively find $w^{k}$ by
\begin{eqnarray*}
     \min&    \frac{1}{N} \| w*Y\|_{\ell_1({\cal T})}\\
    \mbox{ subject to }&   w^{k-1,T} w = 1.
\end{eqnarray*}
This is analogous to use power method to solve PCA problem.


\paragraph{Infinity norm ball constraint}


Before stating the main theorem and its proof, we make some comments about the formulation.

If we study the population level, 
\begin{equation}
 \begin{array}{ll}
  \underset{w\in V}{ \mbox{minimize} }  &    {\mathbb E} |\tilde{w}^T Y|\\
    \mbox{ subject to }
    & \|w\|_{\infty}= 1.
\end{array}
\tag{$P_{\infty}$ }
\end{equation}
then we only need solve one convex problems
\begin{equation}
 \begin{array}{ll}
  \underset{w\in V}{ \mbox{minimize} }  &  {\mathbb E} |\tilde{w}^T Y|\\
 \mbox{subject to}  
&  w_0 =1.
\end{array}
\tag{$P_{e_0}$ }
\end{equation}
Since the entry with largest magnitude will automatically be shifted to coordinate $0$ due to shift invariance.


If we want to solve the following non-convex problem with infinity norm ball constraint for finite sample,
\begin{equation}
 \begin{array}{ll}
  \underset{w\in V}{ \mbox{minimize} }  &    \frac{1}{2T+1}\|w*Y\|_{1,[-T,T]}\\
    \mbox{ subject to }
    & \|w\|_{\infty}= 1.
\end{array}
\tag{$P_{\infty}^T$}
\end{equation}
We can solve the following convex problems, and select the one with minimal objective value. This is equivalent to solve the problem with fixed infinity norm. 
\begin{equation}
 \begin{array}{ll}
  \underset{w\in V}{ \mbox{minimize} }  &  \frac{1}{2T+1}\|w*Y\|_{1,[-T,T]}\\
 \mbox{subject to}  
&  w_j =1.
\end{array}
\tag{$P_{e_j}^T$ }
\EEQ
\section{Analysis of the inverse kernel method}
\subsection{Inverse kernel approximation}
Now we consider a setting where there is a kernel whose corresponding the inverse kernel is infinite, and we give a finite length approximation. 
We assume that $a =  (a_{-t},\ldots, a_{-1}, 1, a_1, a_2, \ldots, a_t) = \delta_0 + a_L + a_R$ 
is a length $ l= 2t+1$ filter, 
  and $\sum_{i\neq 0} |a_i|\leq M < 1$.    
 \begin{theorem} 
For a kernel $a = (a_{-s},\ldots, a_{-1}, 1, a_1, a_2, \ldots, a_t)$ of length $l=1+s+t$, if 
\[
\sum_{i\neq 0} |a_i|\leq M < 1,
\]
 then there exists an approximate inverse kernel $w^r$ of length $rl$, such that 
  \[
 \begin{array}{ll}
\|\mathcal{F}w-\mathcal{F}w^r \|_{\infty} 
&\leq M^{r+1}/(1-M),\\
\end{array}
  \]
\end{theorem}
\begin{proof}
Consider the $z$ transform of $a$ to be 
\[
 \begin{array}{ll}
A(z) =  \sum_{i=-s}^t a_{i} z^{-i},
\end{array}
  \]
let $\bar A(z) = 1-A(z)$, then 
from the condition 
$\sum_{i\neq 0} |a_i|\leq M < 1$,
we define $M' $ to be the maximum absolute value of $\bar A(z)$ on unit circle, as
\[
 \begin{array}{ll}
\mbox{maximize}   & |\bar A(z)| \\
\mbox{subject to}  & z\in {\mathbb C},|z|=1.
\end{array}
  \]
Then 
\[
 \begin{array}{ll}
M' \leq \sum_{i\neq 0} |a_i|\leq M<1.
\end{array}
  \]
The Z-transform of the inverse kernel of $a$ is defined by
\[
 \begin{array}{ll}
1/A(z) &=  (1-\bar A(z))^{-1},\\
&= 1+ \sum_{i=1}^{\infty} (\bar A(z))^i
\end{array}
  \]
 The infinite series absolutely converge on $|z| = 1$ because $|\bar A(z)|\leq M<1$.
Let the finite approximation of level $r$ of $w = a^{-1}$ be $w^r$, 
 its Z-transform is  $\sum_{i=0}^{r} (\bar A(z))^i$,
its length is at most $r(s+t+1)$, 
the $j$-th term of $w^r$ is $\sum_{b}bj_i=j \prod_{i} a_i^{j_i}$.
We have that for all  $|z| = 1$,
 \[
 \begin{array}{ll}
|1/A(z) - \sum_{i=0}^{r} (\bar A(z))^i | 
&= |\sum_{i=r+1}^{\infty} (\bar A(z))^i|\\
&\leq M^{r+1}/(1-M).
\end{array}
  \]
Therefore, 
   \[
 \begin{array}{ll}
\|\mathcal{F}w-\mathcal{F}w^r \|_{\infty} 
&\leq M^{r+1}/(1-M).
\end{array}
  \]
 
\end{proof}

Using this inverse approximation, we know for any kernel $a$ of length $l$ with possibly infinite inverse kernel $w$, there exists a length-$rl$ approximate inverse kernel $w^r$, such that 
  \[
 \begin{array}{ll}
\|\mathcal{F}w-\mathcal{F}w^r \|_{\infty} 
&\leq M^{r+1}/(1-M).
\end{array}
  \]

Therefore, we can prove the following theorem on sparse blind deconvolution algorithm. 


\paragraph{Simple example.}
We consider a simple example, when $a = (0,\ldots, 0, 1, r, 0, \ldots, 0) = \delta_0 - r\delta_1$, $|r|<1$. Then $a^{-1} = \delta_0 + \sum_{j=1}^\infty  r^j \delta_j.$ Now our data generation model is $y = a*x.$ 

We analysis the optimization problem 
 \begin{eqnarray*}
    \min_{f_0 = 0} &    
    E_x\| (\delta_0+f)*a*x\|_1.
\end{eqnarray*}
Here $f = (0, 0, f_R)$ is in a $K$ dimensional subspace of the double infinite sequence.


For the objective, using IID property of $x$, we only need to analyze
 \begin{eqnarray*} 
    E_x | ((\delta_0+f)*(\delta_0 - r\delta_1)*x)_0| .
\end{eqnarray*}
Let $f^*_K$ be the best length $K$ approximation, as the solution of 
\begin{eqnarray*}
    \min_{f_0 = 0} &    
    \| (\delta_0+f)*a\|_2,
\end{eqnarray*}

In this specific example, $f^*_K = \sum_{j=1}^K  r^j \delta_j.$ And $(\delta_0+f^*_K)*a = (\delta_0 - r^{K+1}\delta_{K+1})*(\delta_0 - r^{1}\delta_{1})^{-1}$.
Let $\tilde{x} = (\delta_0 - r^{K+1}\delta_{K+1})*x$, as one approximation of $x$.
Since $x$ is IID symmetric, $ \tilde{x}$ is identically-distributed, symmetric and weakly correlated for $|r^{K+1}|$ small enough, and $\tilde{x}_0, \ldots, \tilde{x}_K$ are independent.

Let 
\begin{eqnarray*} 
\delta_0+h &=& (\delta_0+f)*(\delta_0+f^*_K)^{-1}\\
&=& (\delta_0+f)*(\delta_0 - r^{K+1}\delta_{K+1})
\end{eqnarray*}
then 
 \begin{eqnarray*} 
    E_x | ((\delta_0+f)*(\delta_0 - r\delta_1)*x)_0| 
    &=& E_x | ((\delta_0+h)*\tilde{x})_0|\\
&=& E_x | \tilde{x}_0|+\mbox{sign}(\tilde{x}_0)(h*\tilde{x})_0\\
&\approx& 
 |1+  h_{0}| \cdot E |\tilde{x}_0| \cdot (1-\eta)+   \cdot  E{| (h*\tilde{x})_0| } \cdot \eta.
\end{eqnarray*}

Following the same analysis as the phase transition analysis, we have
\begin{eqnarray*}
     \min_\delta &E | ((\delta_0+f)*a*\tilde{x})_0 | \\
    \mbox{ subject to }  &f_{0} = 0.
\end{eqnarray*}

\subsection{Distance metrics}
We care about three types of metrics, first, distance  in $w$ space, between the computed composed filter $w$ and ground truth $a^{-1}$; second, distance in $w$ space, between the computed inverse filter $w$ and ground truth inverse filter $a^{-1}$, second, distance  in $\psi = w*a$ space, between the computed composed filter $w*a$ and ground truth $e_0$; third,  distance  in $x$ space, between recovered signal sequence and true sparse signal sequence. 

First, the MSE in space of $\psi = w*a$,
  \[
 \begin{array}{ll}
d_{\psi,2}^2 := \|\phi\|_2^2 =  \|\psi-e_0\|_2^2= \|a*(w-a^{-1})\|_2^2
\end{array}
  \] 
 The MSE in space of $w$. 
   \[
 \begin{array}{ll}
d_{w,2}^2  = \|w-a^{-1}\|_2^2 = \|a^{-1}*(\psi-e_0))\|_2^2 = \|a^{-1}*\phi\|_2^2
\end{array}
  \]
Last, $\ell_1$ and $\ell_2$ distance in $x$ space:
\[
 \begin{array}{ll}
d_{x,1} &:= \frac{1}{2T+1}\|w*Y-X\|_{1,[-T,T]} =  \frac{1}{2T+1}\|\tilde \phi* X\|_{1,[-T,T]} \\
d_{x,2}^2 &:= \frac{1}{2T+1}\|w*Y-X\|_{2,[-T,T]}^2 = \frac{1}{2T+1}\|\tilde \phi* X\|_{2,[-T,T]}^2
\end{array}
  \]
We compute their expectation: 
\[
 \begin{array}{ll}
{\mathbb E} d_{x,1}  &=  {\mathbb E}|\phi^T X| = \sqrt{\frac{2}{\pi}} \cdot {\mathbb E}_I \|\phi \cdot I\|_2\\
{\mathbb E} d_{x,2}^2 &=  {\mathbb E}|\phi^T X|^2 = p \|\phi\|^2  = p \|\psi-e_0\|^2 
\end{array}
  \]  
We remark that in the following we focus on distance in $x$ space to recover the signal. 



\section{Stability for noisy observation}


\subsection{Constrained Convex algorithms }
 \paragraph{Constrained noisy convex problems}
Then we define the constrained convex program that allows noise,
\[
 \begin{array}{ll}
 \mbox{minimize}_{w,z}   & |\alpha| \\
\mbox{subject to}  
& v^T w =1,\\
& {\mathbb E}(\alpha -\tilde{w}^T Y)^2 \leq \epsilon^2
\end{array}
\tag{$P^c_{1,\epsilon}$}
\label{eq:ConstrainProb}
  \]
  If we write $\alpha =\tilde{w}^T Y - \epsilon \zeta $ , then
\[
 \begin{array}{ll}
 \mbox{minimize}_{w,\zeta}   & {\mathbb E}|\tilde{w}^T Y - \epsilon \zeta| \\
\mbox{subject to}  
& v^T w =1,\\
& \zeta^2 \leq 1
\end{array}
\tag{$P^c_{1,\epsilon}$}
  \]
If we set $\epsilon = 0,$ then we recover 
the original convex problems for noiseless setting
\[
 \begin{array}{ll}
 \mbox{minimize}_{w}   & {\mathbb E}|\tilde{w}^T Y|\\
\mbox{subject to}  
& v^T w =1,
\end{array}
\tag{$P_1$}
  \]
\subsection{Stability for adversarial perturbation of sparse signal }
\paragraph{Mathematical formulation: sparse signal plus adversarial perturbation}

\begin{itemize}
\item
Let $(X_t)_{t \in {\mathbb Z}}$ be IID sampled from  $pN(0,1) + (1-p) \delta_0$, write $X_t = I_t \cdot G_t$ where $I_t$ is IID sampled from Bernoulli, $G_t$ is IID sampled from standard Gaussian.
 \item
The filter $a\in l_1({\mathbb Z}), (\mathcal{F} a)(\omega) \neq 0, \forall \omega\in T.$
\item
$Z$ is a unit vector in $2$ sphere. 
\item
\[
\begin{array}{ll}
Y&= a* (X+\eta Z ) 
\end{array}
\]
\end{itemize}
Let $\tilde{\psi} := w*a$,   $\phi: = \psi - e_0$, then
\[
 \begin{array}{ll}
{\mathbb E}|(w*Y)_0|
&= {\mathbb E}|\psi^T (X+\eta Z )| \\
\end{array}
  \]

  
  
  
 \paragraph{Study the noisy problem}
Under the data generative model, 
\[
\begin{array}{ll}
Y&= a* (X+\eta Z ) 
\end{array}
\]
    the constrained problem becomes
  \[
 \begin{array}{ll}
 \mbox{minimize}_{\psi,\zeta}   & {\mathbb E}|\psi^T (X+\eta Z ) - \epsilon \zeta| \\
\mbox{subject to}  
&u^T \psi =1,\\
& \zeta^2 \leq 1
\end{array}
\tag{$P^c_{1,\epsilon}$}
  \]
  
\paragraph{Worst case objective over $Z$}  
For the constrained problem, we optimize over the worst case objective for all $Z$ that $\|Z\|_2\leq 1$,
    \[
 \begin{array}{ll}
 \mbox{minimize}_{\psi,\zeta} \quad \sup_{Z: \|Z\|_2\leq 1}  &  {\mathbb E}|\psi^T X+\eta  \psi^T Z  - \epsilon \zeta| \\
\mbox{subject to}  
&u^T \psi =1,\\
& \zeta^2 \leq 1
\end{array}
  \]
Which is equivalent to 
  \[
 \begin{array}{ll}
 \mbox{minimize}_{\psi} \quad \inf_{\zeta: \zeta^2 \leq 1} \quad \sup_{Z: \|Z\|_2\leq 1}  &  {\mathbb E}|\psi^T X+\eta  \psi^T Z  - \epsilon \zeta| \\
\mbox{subject to}  
&u^T \psi =1
\end{array}
  \]
In the following we study the min-max objective
  \[
 \begin{array}{ll}
L_{\eta, \epsilon}(\psi) : = \inf_{\zeta: \zeta^2 \leq 1} \quad \sup_{Z: \|Z\|_2\leq 1}  &  {\mathbb E}|\psi^T X+\eta  \psi^T Z  - \epsilon \zeta| 
\end{array}
  \]
  
\paragraph{Robust optimization}
Let the deterministic residual function of $\psi, Z, \zeta$ be
\[
\mu_{\eta, \epsilon}(\psi, Z, \zeta): = \eta  \psi^T Z  - \epsilon \zeta
\]
We reparametrize $\epsilon$ as by defining 
\[
\tau :=  \frac{\epsilon}{\eta}
\]
then 
\[
\mu_{\eta, \epsilon} = \eta ( \psi^T Z  - \tau \zeta)
\]






\begin{lemma}
For any given sign $s =\pm 1$,
\[
\sup_{Z: \|Z\|_p\leq 1}(s \psi^T Z) = \|\psi\|_q,
\]
specifically, $Z$ is along the direction of $s\psi$ so that $s\psi^T Z = \|\psi\|_2$.

\end{lemma}  


\begin{lemma}[Hölder's inequality]
For any given sign $s =\pm 1$, for $p\geq 1, \frac{1}{p}+\frac{1}{q} =1,$
\[
\sup_{Z: \|Z\|_p\leq 1}(s \psi^T Z) = \|\psi\|_q,
\]
specifically, when $|Z|^p$ is along the direction of $s|\psi|^q$ so that $s\psi^T Z = \|\psi\|_q$.

\end{lemma}  


\begin{lemma}
\label{min-maxError}
The min-max quantity  
\[
 \begin{array}{ll}
\inf_{\zeta: \zeta^2 \leq 1} \sup_{Z: \|Z\|_2\leq 1} (\eta  \psi^T Z  - \epsilon \zeta)^2 = (\eta \|\psi\|_2  - \epsilon )_+^2 = \eta^2 ( \|\psi\|_2  - \tau )_+^2
\end{array}
\]
where $(\cdot)_+: = \max{(0, \cdot)}$ is the ReLU function.
\end{lemma}
\begin{proof}
\[
 \begin{array}{ll}
\inf_{\zeta: \zeta^2 \leq 1} \sup_{Z: \|Z\|_2\leq 1} \mu_{\eta, \epsilon}^2 
&=\inf_{\zeta: \zeta^2 \leq 1}\sup_{Z: \|Z\|_2\leq 1} (\eta  \psi^T Z  - \epsilon \zeta)^2\\
&=\inf_{\zeta: \zeta^2 \leq 1}\sup_{Z: \|Z\|_2\leq 1} [(\eta  \psi^T Z)^2+  (\epsilon \zeta)^2 - 2(\eta  \psi^T Z)(\epsilon \zeta)]\\
&=\inf_{\zeta: \zeta^2 \leq 1}\{   \epsilon^2( \zeta)^2+ \eta^2 \sup_{Z: \|Z\|_2\leq 1} [ (\psi^T Z)^2] + 2\eta \epsilon |\zeta|   \sup_{Z: \|Z\|_2\leq 1}[ (-{\rm sign}(\zeta) \psi^T Z)]\}\\
&=\inf_{\zeta: \zeta^2 \leq 1}\{   \epsilon^2( \zeta)^2+ \eta^2 \|\psi\|_2^2 + 2\eta \epsilon |\zeta|   \|\psi\|_2\}\\
&=\inf_{\zeta: \zeta^2 \leq 1}\{  (\eta \|\psi\|_2  - \epsilon |\zeta| )^2\}\\
&= (\eta \|\psi\|_2  - \epsilon )_+^2
\end{array}
\]
The fourth equation is because for any given sign $s =\pm 1$,
\[
\sup_{Z: \|Z\|_2\leq 1}(s \psi^T Z) = \|\psi\|_2.
\]
\[
\sup_{Z: \|Z\|_2\leq 1}( \psi^T Z)^2 = \|\psi\|_2^2.
\]



\end{proof}

We have the following result, generalizing $\|Z\|_2\leq 1$ to $\|Z\|_p\leq 1, \quad, p\geq 1$, and then to $Z\in K$ for a closed compact convex set $K\in {\mathbb R}^N$.



\begin{lemma}
\label{min-maxErrorL_p}
The min-max quantity 
for $p\geq 1, \frac{1}{p}+\frac{1}{q} =1,$
for a one-dimensional convex even function $F$ that is monotonically non-decreasing for positive coordinates,
\[
 \begin{array}{ll}
\inf_{\zeta: \zeta^2 \leq 1} \sup_{Z: \|Z\|_p\leq 1} F( \eta (\psi^T Z  - \tau \zeta)) = 
F(\eta( \|\psi\|_q  - \tau )_+)
\end{array}
\]
where $(\cdot)_+: = \max{(0, \cdot)}$ is the ReLU function.
\end{lemma}

\begin{coro}
 \[
 \begin{array}{ll}
 \inf_{\zeta: \zeta^2 \leq 1} \quad \sup_{Z: \|Z\|_2\leq 1}  &  {\mathbb E}|\psi^T X+\eta (\psi^T Z  - \tau \zeta)| ={\mathbb E}|\psi^T X+\eta( \|\psi\|_2  - \tau )_+|
\end{array}
  \]
  in general, for $p\geq 1, \frac{1}{p}+\frac{1}{q} =1,$
   \[
 \begin{array}{ll}
 \inf_{\zeta: \zeta^2 \leq 1} \quad \sup_{Z: \|Z\|_p\leq 1}  &  {\mathbb E}|\psi^T X+\eta (\psi^T Z  - \tau \zeta)| ={\mathbb E}|\psi^T X+\eta( \|\psi\|_q  - \tau )_+|
\end{array}
  \]
\end{coro}


\subsection{Folded Gaussian}
\paragraph{Folded Gaussian}
For Bernoulli Gaussian $X$ with support $I$, if we condition on $I$ fixed, then for any constant $\mu$, using the rotation invariance of Gaussian, $\psi^T X +\mu$ is  Gaussian with mean $\mu$ and standard deviation $\|\psi_I\|_2$, therefore,
\[
|\psi^T X +\mu |
\]
following the folded Gaussian distribution.
\footnote{see: \url{ https://en.wikipedia.org/wiki/Folded_normal_distribution}.}

From general theory of folded Gaussian $ |\mu+ \sigma G |, \quad  G \sim \mathcal{N}(0, 1).$
Then its mean is
\begin{equation}
\label{FoldedGaussianMean}
E_G |\mu+ \sigma G| =  \sqrt{\frac{2}{\pi}} \sigma \exp\{-\mu^2/(2\sigma^2)\} + \mu \left(1 - 2\Phi\left(\frac{-\mu}{\sigma}\right) \right)
\end{equation}
where $\Phi$  is the normal cumulative distribution function:
\[
\Phi(x) = \frac 1 {\sqrt{2\pi}} \int_{-\infty}^x e^{-t^2/2} \, dt.
\]
Its variance is
\begin{equation}
\label{FoldedGaussianVar}
{\rm Var}_G |\mu+ \sigma G| = E_G( |\mu+ \sigma G|-E_G |\mu+ \sigma G|)^2 =\sigma^2+ \mu^2 -(E_G |\mu+ \sigma G|)^2
\end{equation}  
The variance will be useful for concentration with finite sample.  

\paragraph{Asymptotic expansion for mean of folded Gaussian}
Notice that if $\sigma =0$, then
\[
E_G |\mu+ 0\cdot G| =   |\mu|.
\]
If $\mu =0$, then we go back to mean of half-normal,
\[
E_G |\sigma G| =   \sqrt{\frac{2}{\pi}} \sigma
\]
Let 
\[
\gamma:= \frac{\mu}{\sigma}
\] 

\begin{lemma}
\label{ExpansionMeanFoldedGaussian}
We define the ratio of folded gaussian mean as 
\[
\mathcal{R}^f(\gamma) := \frac{E_G |\mu+ \sigma G|}{E_G |\sigma G|} = \frac{E_G |\mu+ \sigma G|}{\sqrt{\frac{2}{\pi}}\sigma} = \sqrt{\frac{\pi}{2}} E_G |\gamma+  G|
\]
We have three different expressions (asymptotic expansion around  $\gamma$) for $\mathcal{R}^f(\gamma)$:
\begin{eqnarray}
\label{FoldedGaussianMean_ratio}
\mathcal{R}^f(\gamma)
&=&  \exp\{-\gamma^2/2\} + \sqrt{\frac{\pi}{2}}\gamma \left(1 - 2\Phi\left(-\gamma\right) \right)\\
&=&  \exp\{-\gamma^2/2\}\{1 + \frac{1}{2}\gamma \left[ \gamma + \frac{ \gamma^3}{3} + \frac{ \gamma^5}{3\cdot 5} + \cdots + \frac{ \gamma^{2n+1}}{(2n+1)!!} + \cdots\right]\}\\
&=& 1 + \frac{3}{2}\gamma^2 + \frac{1}{24} \gamma^4   - \frac{ \gamma^6}{120}  + O(\gamma^8) 
\end{eqnarray}
 \end{lemma}
 
 \begin{lemma}
 \label{ExpansionMeanFoldedGaussian_LB}
when $\gamma<2,$ 
\[
\mathcal{R}^f(\gamma) \geq 1 + \frac{3}{2}\gamma^2
\]
hence
\[
 \begin{array}{ll}
E_G |\mu+ \sigma G|
&\geq \sqrt{\frac{2}{\pi}}\left[\sigma + \frac{3}{2}\mu^2/\sigma \right]
\end{array}
\]

\end{lemma}
\begin{proof}[Proof of lemma~\ref{ExpansionMeanFoldedGaussian}]
Plug in the formula for mean of folded Gaussian \ref{FoldedGaussianMean}, 
we have
\[
E_G |\mu+ \sigma G| = \sqrt{\frac{2}{\pi}} \sigma \exp\{-\gamma^2/2\} + \mu \left(1 - 2\Phi\left(-\gamma\right) \right)
\]
Divide it by 
\[
E_G |\sigma G| =   \sqrt{\frac{2}{\pi}} \sigma
\]
we get the first equality.

Since the CDF of the standard normal distribution can be expanded by integration by parts into a series:
\[
\Phi( \gamma)=\frac{1}{2} + \frac{1}{\sqrt{2\pi}}\cdot e^{- \gamma^2/2} \left[ \gamma + \frac{ \gamma^3}{3} + \frac{ \gamma^5}{3\cdot 5} + \cdots + \frac{ \gamma^{2n+1}}{(2n+1)!!} + \cdots\right]
\]
where $!!$ denotes the double factorial. 

\[
1 - 2\Phi(-\gamma)  = \frac{1}{\sqrt{2\pi}}\cdot e^{- \gamma^2/2}\left[ \gamma + \frac{ \gamma^3}{3} + \frac{ \gamma^5}{3\cdot 5} + \cdots + \frac{ \gamma^{2n+1}}{(2n+1)!!} + \cdots\right]
\]
This gives the second inequality.

Additionally,
 \[
 \begin{array}{ll}
  e^{- \gamma^2/2} &=\sum_{n=0}^{\infty} \frac{ \gamma^{2n}}{(-2)^n (n)!}\\
  &=  1- \frac{ \gamma^2}{2} + \sum_{n=2}^{\infty} \frac{ \gamma^{2n}}{(-2)^n (n)!}
\end{array}
  \] 
then
 \[
 \begin{array}{ll}
& \exp\{-\gamma^2/2\}\{1 + \frac{1}{2}\gamma \left[ \gamma + \frac{ \gamma^3}{3} + \frac{ \gamma^5}{3\cdot 5} + \cdots + \frac{ \gamma^{2n+1}}{(2n+1)!!} + \cdots\right]\}\\
=& (\sum_{n=0}^{\infty} \frac{ \gamma^{2n}}{(-2)^n (n)!})\{1 + \frac{1}{2}\gamma \left[ \sum_{n=0}^{\infty} \frac{ \gamma^{2n+1}}{(2n+1)!!} \right]\}\\

=& 1 + \frac{3}{2}\gamma^2 + \frac{1}{24} \gamma^4   - \frac{ \gamma^6}{120}  + O(\gamma^8) 
\end{array}
  \] 
\end{proof}


\begin{proof}[Proof of lemma~\ref{ExpansionMeanFoldedGaussian_LB}]
Let $C_\Phi( \gamma^2)$ be a function of $ \gamma$, then it is a composition function with the inner funciton being $ \gamma^2,$ defined as follows: 
\[
 \begin{array}{ll}
C_\Phi( \gamma^2) 
&:= (2\Phi( \gamma)-1)/\left(\frac{ \gamma}{\sqrt{2\pi}}\right) \\
&= e^{- \gamma^2/2} \left[1 + \frac{ \gamma^2}{3} + \frac{ \gamma^4}{3\cdot 5} + \cdots + \frac{ \gamma^{2n}}{(2n+1)!!} + \cdots\right]\\
&= [\sum_{n=0}^{\infty} \frac{ \gamma^{2n}}{(2n+1)!!} ]/[\sum_{n=0}^{\infty} \frac{ \gamma^{2n}}{2^n (n)!}] \\
&= [\sum_{n=0}^{\infty} \frac{ \gamma^{2n}}{(2n+1)!!} ]/[\sum_{n=0}^{\infty} \frac{ \gamma^{2n}}{(2n)!!}] \\
&\leq 1
\end{array}
\]
We remark further that
\[
 \begin{array}{ll}
C_\Phi( \gamma^2) 
&= [\sum_{n=0}^{\infty} \frac{ \gamma^{2n}}{(2n+1)!!} ]/[\sum_{n=0}^{\infty} \frac{ \gamma^{2n}}{(2n)!!}] \\
&= [1 + \frac{ \gamma^2}{3}+ +  \frac{ \gamma^4}{15}+ O( \gamma^6) ]/[1 + \frac{ \gamma^2}{2}+ +  \frac{ \gamma^4}{8}+O( \gamma^6)] \\
&= 1 -\frac{ \gamma^2}{6}+  \frac{ \gamma^4}{40}+ O( \gamma^6)
\end{array}
\]
Then
 \[
 \begin{array}{ll}
\mu \left(1 - 2\Phi\left(-\gamma\right) \right)
&= \frac{1}{\sqrt{2\pi}}\cdot \mu^2 \frac{1}{\sigma} C_\Phi(\gamma^2)\\
&= \frac{1}{\sqrt{2\pi}}\cdot \mu^2 \frac{1}{\sigma} (1-\gamma^2/6 +O(\gamma^4))\\
\end{array}
\] 
Plug in all the fact together
\[
 \begin{array}{ll}
E_G |\mu+ \sigma G|
&=   \left[ \sqrt{\frac{2}{\pi}} \sigma \exp\{-\gamma^2/2\} + \frac{1}{\sqrt{2\pi}}\cdot \mu^2 \frac{1}{\sigma} C_\Phi(\gamma^2)\right]\\
&=   \left[ \sqrt{\frac{2}{\pi}} \sigma \{ 1- \frac{\gamma^2}{2} + \sum_{n=2}^{\infty} \frac{\gamma^{2n}}{(-2)^n (n)!}\} + \frac{1}{\sqrt{2\pi}}\cdot \mu^2 \frac{1}{\sigma} C_\Phi(\gamma^2)\right]\\
&= \sqrt{\frac{2}{\pi}}   \left[ \sigma 
\{ 1 + (1+\frac{1}{2}\cdot C_\Phi(\gamma^2))\gamma^2 +   \sum_{n=2}^{\infty} \frac{\gamma^{2n}}{(-2)^n (n)!}\} \}\right]\\
\end{array}
\] 
Define the residual of this expansion as
\[
 \begin{array}{ll}
\mbox{Res}(\gamma^2) 
&:= \frac{ (2\Phi( \gamma)-1)/\left(\frac{ \gamma}{\sqrt{2\pi}}\right)-1}{2}\cdot \gamma^2 +  (\exp\{-\gamma^2/2\} - 1 + \gamma^2/2)
\\
&= \frac{ C_\Phi(\gamma^2)-1}{2}\cdot \gamma^2 +  (\exp\{-\gamma^2/2\} - 1 + \gamma^2/2) \\
&=    \frac{ C_\Phi(\gamma^2)-1}{2}\cdot \gamma^2  +   \{ \sum_{n=2}^{\infty} \frac{\gamma^{2n}}{(-2)^n (n)!}\} \} \\
&=    \frac{-\gamma^4 }{12}  +    \frac{ \gamma^6}{80} +\frac{\gamma^{4}}{(-2)^2 (2)!} +\frac{\gamma^{6}}{(-2)^3 (3)!}+
O(\gamma^8)\\
&=    \frac{\gamma^4 }{24}  - \frac{ \gamma^6}{120}  + O(\gamma^8)
\end{array}
\] 
We know that if $\gamma^2<1,$ 
\[
 \begin{array}{ll}
\mbox{Res}(\gamma^2) 
&=    \frac{\gamma^4 }{24}  - \frac{ \gamma^6}{120}  + O(\gamma^8)\\
&\geq 0
\end{array}
\] 
(We comment that we could push the range for $\mbox{Res}(\gamma^2) 
\geq 0$ further to at least $\gamma^2<5$ if needed.)
Then we give an lower bound:
\[
 \begin{array}{ll}
E_G |\mu+ \sigma G| 
&= \sqrt{\frac{2}{\pi}} \sigma\left[ 1 + \frac{3}{2}\gamma^2+   \mbox{Res}\right]\\
&\geq \sqrt{\frac{2}{\pi}}\left[  \sigma + \frac{3}{2}\mu^2  \sigma^{-1}\right]
\end{array}
\] 


\end{proof}






\subsection{Other setting on $Z$}
\paragraph{ Adversarial outer noise} 
If our generative model is
\[
\begin{array}{ll}
Y&= a* X + \eta_2 Z_2\\
&= a* (X +\eta_2 a^{-1}* Z_2)
\end{array}
\] 
And we consider the worst case objective over $\|Z_2\|_p\leq 1, \quad p\geq 1.$

Let $C_a^{-1}$ be the linear operator represents $a^{-1}*$, then following previous calculation,
\begin{coro}
For $p\geq 1, \frac{1}{p}+\frac{1}{q} =1,$
   \[
 \begin{array}{ll}
 \inf_{\zeta: \zeta^2 \leq 1} \quad \sup_{Z: \|Z_2\|_p \leq 1}  &  {\mathbb E}|\psi^T X+\eta_2  (\psi^T C_a^{-1} Z  - \tau \zeta)| ={\mathbb E}|\psi^T X+\eta( \|C_{a^{-1}}^T \psi\|_q  - \tau )_+|
\end{array}
  \]
\end{coro}

Let $C_{a^{-1}}^T \psi =\bar{a}^{-1}*\psi$, where $\tilde{a}^{-1}$ is defined by flipping the coordinate of $a^{-1}$ (reflecting the coordinates around zero by map $i$ to $-i$ for $i\in {\mathbb Z}$.)
 
 We can replace the definition of $\gamma$ and $\mu_0$  by:
 \[
 \gamma^{a}_{\eta, \epsilon}(\psi, I): = \inf_{\zeta: \zeta^2 \leq 1} \sup_{Z: \|Z\|_2\leq 1} \eta_2  (\psi^T C_a^{-1} Z  - \tau \zeta)/\|\psi_I\|_2 = \eta \frac{ (\|\bar{a}^{-1}*\psi\|_q  - \tau)_+}{\|\psi_I\|_2}
\]
\[
\mu_0^{a} := \eta( \|\tilde{a}^{-1}\|_q-\tau)_+
\]
We can get the same result as theorem~\ref{Thm:StabilityMSE}.

  

  

\subsection{ Gaussian additional noise}
\paragraph{ Gaussian additional observation inner and outer noise} 
If $G_1$ and $G_2$ are two independent standard normal $N(0,I_N)$. Our generative model is
\[
\begin{array}{ll}
Y&= a* (X+  \sigma_1 G_1) +  \sigma_2 G_2\\
&= a* (X+\sigma_1 G_1+\sigma_2 a^{-1}* G_2)
\end{array}
\] 
let $C_{a^{-1}}$ be the linear operator represents $a^{-1}*$, now using the property for sum of independent Gaussian, then $C_{a^{-1}} = C_a^{-1}$ and 
\[
\begin{array}{ll}
Z: = \sigma_1 G_1+\sigma_2 a^{-1}* G_2
\sim N(0, \sigma_1^2 I_N + \sigma_2^2 C_{a^{-1}} C_{a^{-1}}^T)
\end{array}
\] 
then
\[
\begin{array}{ll}
\psi^T Z
\sim N(0, \sigma_1^2\|\psi\|_2^2 + \sigma_2^2\|C_{a^{-1}}^T \psi\|_2^2 )
\end{array}
\] 
 Let $G$ be another standard normal $N(0,I_N)$, for any fixed vector $\mu \in {\mathbb R}^N, $
 \[
 \begin{array}{ll}
  {\mathbb E}_{G_1} {\mathbb E}_{G_2} {\mathbb E}_X|\psi^T X+\sigma_1 \psi^T G_1+\sigma_2 \psi^T (a^{-1}* G_2)  | = 
   {\mathbb E}_I{\mathbb E}_{G_3}  | \sqrt{ \|\psi_I\|_2^2 + \sigma_1^2\|\psi\|_2^2 + \sigma_2^2\|C_{a^{-1}}^T \psi\|_2^2}G_3  |
\end{array}
  \]
In this case, it seems the previous constrained problem no longer can help us find exact solution $\psi^\star =e_0.$ 



We first study $\sigma_2 = 0$ case.
\[
{\mathbb E}_I{\mathbb E}_{G_3}  | \sqrt{ \|\psi_I\|_2^2 + \sigma_1^2\|\psi\|_2^2}G_3 | = 
\sqrt{\frac{2}{\pi}} \cdot {\mathbb E}_I(\sqrt{ \|\psi_I\|_2^2 + \sigma_1^2\|\psi\|_2^2}) =  \sqrt{\frac{2}{\pi}} \cdot \|\psi\|_2 {\mathbb E}_I(\sqrt{ \frac{\|\psi_I\|_2^2}{\|\psi\|_2^2} + \sigma_1^2})
\]
Define worst case objective
\[
F_{\sigma}(\psi):= \sup_{\sigma_1 \leq \sigma} {\mathbb E}_I(\sqrt{ \|\psi_I\|_2^2 + \sigma_1^2\|\psi\|_2^2}) = {\mathbb E}_I(\sqrt{ \|\psi_I\|_2^2 + \sigma^2\|\psi\|_2^2})
\]
The second equality comes from monotonicity.

Again, we assume the minimizer is $\psi^\star $. Since $e_0$ is  minimizer of $L_{0}$ and $\psi^\star $ is  minimizer of $L_{\sigma}$, we have the chain of inequality:
    \[
 \begin{array}{ll}
F_{\sigma}(e_0)
 \geq F_{\sigma}(\psi^\star ) 
 \geq F_{0}(\psi^\star )
 \geq F_{0}(e_0) 
\end{array}
  \] 
Then
    \[
 \begin{array}{ll}
F_{\eta}(e_0)/F_{0}(e_0)
 \geq (F_{\eta}(\psi^\star )/F_{0}(\psi^\star ))(F_{0}(\psi^\star )/F_{0}(e_0) )
\end{array}
  \] 

   \[
 \begin{array}{ll}
 F_{\eta}(\psi)/F_{0}(\psi) = {\mathbb E}_I(\sqrt{ \frac{\|\psi_I\|_2^2}{\|\psi\|_2^2} + \sigma^2})/{\mathbb E}_I(\frac{\|\psi_I\|_2}{\|\psi\|_2} )
\end{array}
  \] 
  
     \[
 \begin{array}{ll}
 F_{\eta}(e_0)/F_{0}(e_0) = {\mathbb E}_I(\sqrt{ \frac{\|(e_0)_I\|_2^2}{\|e_0\|_2^2} + \sigma^2})/{\mathbb E}_I(\frac{\|(e_0)_I\|_2}{\|e_0\|_2} )= (p\sqrt{ 1 + \sigma^2}+ (1-p) \sqrt{ \sigma^2})/p
\end{array}
  \] 
  
    \[
 \begin{array}{ll}
 F_{0}(\psi)/F_{0}(e_0) 
 &= {\mathbb E}_I\|\psi_I\|_2/{\mathbb E}_I\|(e_0)_I\|_2 \\
 &= 1+({\mathbb E}_I\|\psi_I\|_2-{\mathbb E}_I\|(e_0)_I\|_2)/{\mathbb E}_I\|(e_0)_I\|_2 \\
 &= 1+ (B(p,N)/{\mathbb E}_I\|(e_0)_I\|_2) \|\psi -e_0\|_2\\
 &= 1+ (B(p,N)/p) \|\phi\|_2
\end{array}
  \] 

   \[
 \begin{array}{ll}
1+ (B(p,N)/p) \|\phi\|_2 &\leq (F_{\eta}(e_0)/F_{0}(e_0))/ (F_{\eta}(\psi)/F_{0}(\psi)) \\
&\leq \{(p\sqrt{ 1 + \sigma^2}+ (1-p) \sqrt{ \sigma^2})/p\}/\{{\mathbb E}_I(\sqrt{ \frac{\|\psi_I\|_2^2}{\|\psi\|_2^2} + \sigma^2})/{\mathbb E}_I(\frac{\|\psi_I\|_2}{\|\psi\|_2} )\}
\end{array}
  \] 
  First, a loose bound on $\|\phi^\star\|_2$:
  \[
 \begin{array}{ll}
1+ (B(p,N)/p) \|\phi^\star\|_2 
&\leq \{(p\sqrt{ 1 + \sigma^2}+ (1-p) \sqrt{ \sigma^2})/p\}
\end{array}
  \] 
We found that
 \[
 \begin{array}{ll}
 \|\phi^\star\|_2 
&\leq (\{(p\sqrt{ 1 + \sigma^2}+ (1-p) \sqrt{ \sigma^2})/p\}-1)/(B(p,N)/p) \\
&\leq ((p\sqrt{ 1 + \sigma^2}+ (1-p) \sqrt{ \sigma^2})-p)/B(p,N)
\end{array}
  \] 

In general, when $\sigma_1, \sigma_2$ both non-zero, using the same argument,
\[
 \begin{array}{ll}
 \|\phi^\star\|_2 
&\leq \frac{(p[\sqrt{ 1 + \sigma_1^2+ \sigma_2^2\|\tilde a^{-1}\|_2^2}-1]+ (1-p) \sqrt{ \sigma_1^2+ \sigma_2^2\|\tilde a^{-1}\|_2^2})}{B(p,N)}
\end{array}
  \] 

\subsection{Inner and outer Gaussian and adversarial noise}
\paragraph{Inner and outer Gaussian and adversarial noise together}
When 
If $G_1$ and $G_2$ are two independent standard normal $N(0,I_N)$. 
$Z_1$ and $Z_2$ are inner and outer adversarial noise with $p_1$ and $p_2$ norm bounded by $1$.

Our generative model is
\[
\begin{array}{ll}
Y&= a* (X+ \eta_1 Z_1+  \sigma_1 G_1) + \eta_2 Z_2+ \sigma_2 G_2\\
&= a* (X +\eta_1 Z_1 +\eta_2 a^{-1}* Z_2+  \sigma_1 G_1+\sigma_2 a^{-1}* G_2)
\end{array}
\]
We define our robust objective as   
\[
 \begin{array}{ll}
& F_{\eta_1,\eta_2, \sigma_1,\sigma_2}(\psi)\\
:=& 
\inf_{\zeta^2\leq 1}\sup_{Z_1, Z_2: \|Z_1\|_{p_1} \leq 1; \|Z_2\|_{p_2} \leq 1}
{\mathbb E}_{G_1} {\mathbb E}_{G_2} {\mathbb E}_X|\psi^T X+\sigma_1 \psi^T G_1+\sigma_2 \psi^T (a^{-1}* G_2) + \eta_1 Z_1 +\eta_2 a^{-1}* Z_2-\epsilon \zeta |
\end{array}
  \]  
\begin{lemma}
\label{lemma:Obj_Folded_FullNoise}
Let 
 \[
 \begin{array}{ll}
 \mu(\psi) &=(\eta_1\|\psi\|_{q_1}+ \eta_2\|C_{a^{-1}}^T \psi\|_{q_2}-\epsilon)_+\\
 \sigma_n^2(\psi) &= \sqrt{ \sigma_1^2\|\psi\|_2^2 + \sigma_2^2\|C_{a^{-1}}^T \psi\|_2^2}
 \end{array}
  \]
then the objective is a mean of a folded Gaussian random variable:
\[
 \begin{array}{ll}
 F_{\eta_1,\eta_2, \sigma_1,\sigma_2}(\psi)
&= {\mathbb E}_I{\mathbb E}_{G}  | \sqrt{\|\psi_I\|_2^2 + \sigma_n^2(\psi)} G  + \mu(\psi) |\\
&= \sqrt{\frac{2}{\pi}} {\mathbb E}_I[ \sqrt{\|\psi_I\|_2^2 + \sigma_n^2(\psi)} 
\mathcal{R}^f(\frac{\mu(\psi)}{\sqrt{\|\psi_I\|_2^2 + \sigma_n^2(\psi)}})]
\end{array}
  \] 
  where $G$ is scalar standard Gaussian.
\end{lemma}
From a simple inequality:
   \[
 \begin{array}{ll}
 F_{0,0,0,0}(\psi^\star ) &\leq F_{\eta_1,\eta_2, \sigma_1,\sigma_2}(e_0)
\end{array}
  \]  
we have  
\begin{theorem}
\label{thm:Bound_FullNoise}
\[
 \begin{array}{ll}
 B(p,N) \|\psi^\star -e_0\|_2\leq (1-p) \sigma_n(e_0)^2 \mathcal{R}^f(\frac{\mu(e_0)}{\sigma_n(e_0)})+  
 p\sqrt{1+ \sigma^2_n(e_0)} \mathcal{R}^f(\frac{\mu(e_0)}{\sqrt{1+ \sigma^2_n(e_0)}}) - p
\end{array}
  \]  
where
\[
 \begin{array}{ll}
 \mu(e_0) &= (\eta_1 + \eta_2 \| a^{-1} \|_{q_2}-\epsilon)_+\\
 \sigma_n(e_0) &=\sqrt{ \sigma_1^2 + \sigma_2^2\|a^{-1}\|_2^2} 
 \end{array}
  \]
\end{theorem}  

    
\begin{proof}[Proof of lemma~\ref{lemma:Obj_Folded_FullNoise}]
 First, for any fixed vector $\mu$,
\[
 \begin{array}{ll}
  {\mathbb E}_{G_1} {\mathbb E}_{G_2} {\mathbb E}_X|\psi^T X+\sigma_1 \psi^T G_1+\sigma_2 \psi^T (a^{-1}* G_2) + \mu | = 
   {\mathbb E}_I{\mathbb E}_{G_3}  | \sqrt{ \|\psi_I\|_2^2 + \sigma_1^2\|\psi\|_2^2 + \sigma_2^2\|C_{a^{-1}}^T \psi\|_2^2}G_3  + \mu|
\end{array}
  \]
  Let 
 \[
 \begin{array}{ll}
 \mu(\psi) &= (\eta_1\|\psi\|_{q_1}+ \eta_2\|C_{a^{-1}}^T \psi\|_{q_2}-\epsilon)_+\\
 \sigma_n^2(\psi) &= \sqrt{ \sigma_1^2\|\psi\|_2^2 + \sigma_2^2\|C_{a^{-1}}^T \psi\|_2^2}\\
  \sigma_s^2(\psi_I) &=  \|\psi_I\|_2^2 \\
 \sigma^2 (\psi, \psi_I) &= \sigma_s^2(\psi_I) + \sigma_n^2(\psi) 
 \end{array}
  \]
 Using lemma~\ref{min-maxErrorL_p}, we have
\[
 \begin{array}{ll}
F_{\eta_1,\eta_2, \sigma_1,\sigma_2}(\psi)
&={\mathbb E}_I{\mathbb E}_{G}  | \sigma (\psi, \psi_I) G  + \mu(\psi) | \\
&={\mathbb E}_I{\mathbb E}_{G}  | \sqrt{\sigma_s^2(\psi_I) + \sigma_n^2(\psi)} G  + \mu(\psi) | \\
\end{array}
  \]    

\end{proof}

\begin{proof}[Proof of theorem~\ref{thm:Bound_FullNoise}]  
Additionally, for $\tilde a^{-1} $, define its norm as
\[
M_q: = \|\tilde a^{-1} \|_{q} = \| a^{-1} \|_{q}
\]

 When $\psi = e_0$, then
 \[
 \begin{array}{ll}
 \mu(e_0) &= \eta_1 + \eta_2 M_{q_2}\\
  \sigma_n(e_0) &=\sqrt{ \sigma_1^2 + \sigma_2^2 M_{2}^2} \\
 \sigma_s((e_0)_I) &= I_0 
 \end{array}
  \]

  
We have 
\[
 \begin{array}{ll}
 F_{\eta_1,\eta_2, \sigma_1,\sigma_2}(e_0)
 &= 
  {\mathbb E}_{G} {\mathbb E}_{I_0} |  \sigma(e_0, (e_0)_{I_0})  G  +  \mu(e_0)| \\
  &= 
 (1-p){\mathbb E}_{G} |  \sigma_n(e_0) G  +  \mu(e_0)|+ p{\mathbb E}_{G} | \sqrt{1+ \sigma^2_n(e_0)}  G  +  \mu(e_0)|\\
   &=
 \sqrt{\frac{2}{\pi}}[ (1-p) \sigma_n(e_0)^2 \mathcal{R}^f(\frac{\mu(e_0)}{\sigma_n(e_0)})+  
 p\sqrt{1+ \sigma^2_n(e_0)} \mathcal{R}^f(\frac{\mu(e_0)}{\sqrt{1+ \sigma^2_n(e_0)}}) ]
\end{array}
  \]
  Using the chain of inequality:
    \[
 \begin{array}{ll}
F_{\eta_1,\eta_2, \sigma_1,\sigma_2}((e_0)
 \geq F_{\eta_1,\eta_2, \sigma_1,\sigma_2}((\psi^\star ) 
 \geq F_{0,0,0,0}(\psi^\star )
 \geq F_{0,0,0,0}(e_0) 
\end{array}
  \] 
and the fact that
  \[
 \begin{array}{ll}
 F_{0,0,0,0}(\psi^\star )
 &= {\mathbb E}_I\|\psi_I\|_2 \\
 &= {\mathbb E}_I\|(e_0)_I\|_2 + ({\mathbb E}_I\|\psi_I\|_2-{\mathbb E}_I\|(e_0)_I\|_2) \\
 &= \sqrt{\frac{2}{\pi}}[p+ B(p,N) \|\psi -e_0\|_2]
\end{array}
  \]   
  From
   \[
 \begin{array}{ll}
 F_{0,0,0,0}(\psi^\star ) &\leq F_{\eta_1,\eta_2, \sigma_1,\sigma_2}(e_0)
\end{array}
  \]   
we have
  \[
 \begin{array}{ll}
p+ B(p,N) \|\psi^\star -e_0\|_2\leq (1-p) \sigma_n(e_0)^2 \mathcal{R}^f(\frac{\mu(e_0)}{\sigma_n(e_0)})+  
 p\sqrt{1+ \sigma^2_n(e_0)} \mathcal{R}^f(\frac{\mu(e_0)}{\sqrt{1+ \sigma^2_n(e_0)}}) 
\end{array}
  \]   

\end{proof}  
 
  
  
  
  
  
  
  
  
  
  
  
  
  





  
  
  

  
  
  
  
  
  
  
  
  
  
   
   





 
 
 
  
  
  









    
    
    
  
    
   
   
 
 
 


 
 

























\section{Stability for noisy observation}
\subsection{Inner and outer Gaussian and adversarial noise}
\paragraph{Mathematical formulation: sparse signal plus inner and outer adversarial perturbation and Gaussian noise}
\begin{itemize}
\item
Let $(X_t)_{t \in {\mathbb Z}}$ be IID sampled from  $pN(0,1) + (1-p) \delta_0$, write $X_t = I_t \cdot G_t$ where $I_t$ is IID sampled from Bernoulli, $G_t$ is IID sampled from standard Gaussian.
 \item
The filter $a\in l_1({\mathbb Z}), (\mathcal{F} a)(\omega) \neq 0, \forall \omega\in T.$
\item
Let $G_1$ and $G_2$ are two independent standard normal $N(0,I_N)$. 
\item
Let $\zeta_1$ and $\zeta_2$ are inner and outer adversarial noise with $p_1$ and $p_2$ norm bounded by $1$, where for $p_1, p_2\geq 1, $ $q_1$ and $q_2$ is defined by $\frac{1}{p_i}+\frac{1}{q_i} =1.$
\item
Our data generative model is
\[
\begin{array}{ll}
Y&= a* (X+ \eta_1 \zeta_1+  \sigma_1 G_1) + \eta_2 \zeta_2+ \sigma_2 G_2\\
&= a* (X +\eta_1 \zeta_1 +\eta_2 a^{-1}* \zeta_2+  \sigma_1 G_1+\sigma_2 a^{-1}* G_2)
\end{array}
\]
\item
Let the total perturbation be
\[
\begin{array}{ll}
Z= \eta_1 \zeta_1 +\eta_2 a^{-1}* \zeta_2+  \sigma_1 G_1+\sigma_2 a^{-1}* G_2
\end{array}
\]
We can write data generative model  simply as
\[
\begin{array}{ll}
Y&= a* (X+ Z ) 
\end{array}
\]
\end{itemize}
Let $\tilde{\psi} := w*a$,   $\phi: = \psi - e_0$, then
\[
 \begin{array}{ll}
(w*Y)_0
&= \psi^T (X+  Z ) 
\end{array}
  \]

 \paragraph{Perturbation analysis of the robust convex problems}
The original convex problems is
\[
 \begin{array}{ll}
 \mbox{minimize}_{w}   & {\mathbb E}|\tilde{w}^T Y|\\
\mbox{subject to}  
& v^T w =1,
\end{array}
\tag{$P_1$}
  \]  
Our objective is
\[
|\psi^T (X+  Z )|  = |\psi^T X+\psi^T[\sigma_1  G_1+\sigma_2 (a^{-1}* G_2) + \eta_1 \zeta_1 +\eta_2 a^{-1}* \zeta_2] |
\]
\paragraph{Worst case robust objective over adversarial perturbation}  
For the constrained problem, we first take expectation over Gaussian noise and over the Bernoulli Gaussian signal, then we optimize over the worst case expected objective for all adversarial perturbation, we define our robust objective as   
\[
 \begin{array}{ll}
 F_{\eta_1,\eta_2, \sigma_1,\sigma_2}(\psi)
:=
\sup_{\zeta_1, \zeta_2: \|\zeta_1\|_{p_1} \leq 1; \|\zeta_2\|_{p_2} \leq 1}
{\mathbb E}_{G_1} {\mathbb E}_{G_2} {\mathbb E}_X|\psi^T (X+  Z )|
\end{array}
  \]  
then our robust constrained problem becomes
  \[
 \begin{array}{ll}
 \mbox{minimize}_{\psi} \quad &F_{\eta_1,\eta_2, \sigma_1,\sigma_2}(\psi) \\
\mbox{subject to}  
&u^T \psi =1
\end{array}
  \]
Assume that the minimizer of the robust problem is $\psi^\star$, then we have the chain of inequality:
    \[
 \begin{array}{ll}
F_{\eta_1,\eta_2, \sigma_1,\sigma_2}(e_0)
 \geq F_{\eta_1,\eta_2, \sigma_1,\sigma_2}(\psi^\star ) 
 \geq F_{0,0,0,0}(\psi^\star )
 \geq F_{0,0,0,0}(e_0) =p
\end{array}
  \] 

  
In the following,
let 
\[
D_R( \psi) :=  F_{\eta_1,\eta_2, \sigma_1,\sigma_2}(\psi)-F_{0,0,0,0}(\psi).
\]
We use this fact and the bi-lipschitz property of noiseless objective around $e_0$ to bound the $\ell_2$ norm deviation $\|\psi^\star-e_0\|_2.$

Let $\phi^\star := \psi^\star-e_0$,
we have
   \[
 \begin{array}{ll}
D_R( e_0) &= F_{\eta_1,\eta_2, \sigma_1,\sigma_2}(e_0)-F_{0,0,0,0}(e_0)\\
 &\geq F_{0,0,0,0}(\psi^\star )-F_{0,0,0,0}(e_0) \\
 &\geq B(p,\phi^\star) \|\psi^\star-e_0\|_2\\
 &\geq p (
(1-p)- \tan(\theta) )\cos{\theta} \|\phi^\star\|_2
\end{array}
  \] 

This bound is not tight but it is tight in order and it is easy to compute since we only need to compute  $F_{\eta_1,\eta_2, \sigma_1,\sigma_2}(e_0)$, the robust objective at $e_0$.
 
\subsection{Folded Gaussian}
\paragraph{Folded Gaussian}
For Bernoulli Gaussian $X$ with support $I$, if we condition on $I$ fixed, then  using the rotation invariance of Gaussian, $\psi^T X$ is  Gaussian with mean $0$ and standard deviation $\|\psi_I\|_2$.

From general theory of folded Gaussian,
\footnote{see: \url{ https://en.wikipedia.org/wiki/Folded_normal_distribution}.}
 $ |\mu+ \sigma G |, \quad  G \sim \mathcal{N}(0, 1).$
Then its mean is
\begin{equation}
\label{FoldedGaussianMean}
E_G |\mu+ \sigma G| =  \sqrt{\frac{2}{\pi}} \sigma \exp\{-\mu^2/(2\sigma^2)\} + \mu \left(1 - 2\Phi\left(\frac{-\mu}{\sigma}\right) \right)
\end{equation}
where $\Phi$  is the normal cumulative distribution function:
\[
\Phi(x) = \frac 1 {\sqrt{2\pi}} \int_{-\infty}^x e^{-t^2/2} \, dt.
\]
Its variance is
\begin{equation}
\label{FoldedGaussianVar}
{\rm Var}_G |\mu+ \sigma G| = E_G( |\mu+ \sigma G|-E_G |\mu+ \sigma G|)^2 =\sigma^2+ \mu^2 -(E_G |\mu+ \sigma G|)^2
\end{equation}  
The variance will be useful for concentration with finite sample.  

\paragraph{Asymptotic expansion for mean of folded Gaussian}
Notice that if $\sigma =0$, then
\[
E_G |\mu+ 0\cdot G| =   |\mu|.
\]
If $\mu =0$, then we go back to mean of half-normal,
\[
E_G |\sigma G| =   \sqrt{\frac{2}{\pi}} \sigma
\]
Let 
\[
\gamma:= \frac{\mu}{\sigma}
\] 

\begin{lemma}
\label{ExpansionMeanFoldedGaussian}
We define the ratio of folded gaussian mean as 
\[
\mathcal{R}(\gamma) := \frac{E_G |\mu+ \sigma G|}{E_G |\sigma G|} = \frac{E_G |\mu+ \sigma G|}{\sqrt{\frac{2}{\pi}}\sigma} = \sqrt{\frac{\pi}{2}} E_G |\gamma+  G| = \exp\{-\gamma^2/2\} + \sqrt{\frac{\pi}{2}}\gamma \left(1 - 2\Phi\left(-\gamma\right) \right)
\]
We have three different expressions (asymptotic expansion around  $\gamma$) for $\mathcal{R}(\gamma)$:
\begin{eqnarray}
\label{FoldedGaussianMean_ratio}
\mathcal{R}(\gamma)
&=&  \exp\{-\gamma^2/2\} + \sqrt{\frac{\pi}{2}}\gamma \left(1 - 2\Phi\left(-\gamma\right) \right)\\
&=&  \exp\{-\gamma^2/2\}\{1 + \frac{1}{2}\gamma \left[ \gamma + \frac{ \gamma^3}{3} + \frac{ \gamma^5}{3\cdot 5} + \cdots + \frac{ \gamma^{2n+1}}{(2n+1)!!} + \cdots\right]\}\\
&=& 1 + \frac{3}{2}\gamma^2 + \frac{1}{24} \gamma^4   - \frac{ \gamma^6}{120}  + O(\gamma^8) 
\end{eqnarray}
 \end{lemma}
 
  \begin{lemma}
 \label{ExpansionMeanFoldedGaussian_Monot}
$\mathcal{R}(\gamma)$ is an even function that is monotonically non-decreasing for $\gamma \geq 0$.
\end{lemma}
 
 \begin{lemma}
 \label{ExpansionMeanFoldedGaussian_LB}
when $\gamma<2,$ 
\[
\mathcal{R}(\gamma) \geq 1 + \frac{3}{2}\gamma^2
\]
\end{lemma}


\paragraph{Robust optimization}
\begin{lemma}
From Hölder's inequality, for any vector $a \in {\mathbb R}^N$, for $p\geq 1, \frac{1}{p}+\frac{1}{q} =1,$
\[
\sup_{Z: \|Z\|_p\leq 1}(a^T Z) = \|a\|_q,
\]
specifically, when $|Z|^p$ is along the direction of $\mbox{sign}(a)|a|^q$ so that $a^T Z = \|a\|_q$.

\end{lemma}  



\paragraph{Bound on the noisy solution using robust objective}  
\begin{lemma}
\label{lemma:Obj_Folded_FullNoise}
Let 
 \[
 \begin{array}{ll}
 \mu(\psi) &=\eta_1\|\psi\|_{q_1}+ \eta_2\|C_{a^{-1}}^T \psi\|_{q_2}\\
 \sigma_n^2(\psi) &=  \sigma_1^2\|\psi\|_2^2 + \sigma_2^2\|C_{a^{-1}}^T \psi\|_2^2
 \end{array}
  \]
then the objective is a mean of a folded Gaussian random variable:
\[
 \begin{array}{ll}
 F_{\eta_1,\eta_2, \sigma_1,\sigma_2}(\psi)
&= \sup_{\zeta_1, \zeta_2: \|\zeta_1\|_{p_1} \leq 1; \|\zeta_2\|_{p_2} \leq 1}{\mathbb E}_I{\mathbb E}_{G}  | \sqrt{\|\psi_I\|_2^2 + \sigma_n^2(\psi)} G  + \eta_1 \zeta_1 +\eta_2 a^{-1}* \zeta_2 |\\
&= {\mathbb E}_I{\mathbb E}_{G}  | \sqrt{\|\psi_I\|_2^2 + \sigma_n^2(\psi)} G  + \sup_{\zeta_1, \zeta_2: \|\zeta_1\|_{p_1} \leq 1; \|\zeta_2\|_{p_2} \leq 1}[\eta_1 \zeta_1 +\eta_2 a^{-1}* \zeta_2] |\\
&= {\mathbb E}_I{\mathbb E}_{G}  | \sqrt{\|\psi_I\|_2^2 + \sigma_n^2(\psi)} G  + \mu(\psi) |\\
&= \sqrt{\frac{2}{\pi}} {\mathbb E}_I[ \sqrt{\|\psi_I\|_2^2 + \sigma_n^2(\psi)} 
\mathcal{R}(\frac{\mu(\psi)}{\sqrt{\|\psi_I\|_2^2 + \sigma_n^2(\psi)}})]
\end{array}
  \] 
  where $G$ is scalar standard Gaussian.
 
 The second inequality comes from the fact that $\mathcal{R}(\gamma)$ is an even function that is monotonically non-decreasing for $\gamma \geq 0$.
  
\end{lemma}
From a simple inequality:
   \[
 \begin{array}{ll}
 F_{0,0,0,0}(\psi^\star ) &\leq F_{\eta_1,\eta_2, \sigma_1,\sigma_2}(e_0)
\end{array}
  \]  
we have  
\begin{theorem}
\label{thm:Bound_FullNoise}
For $p<p^\star_1 = 1-\tan(\theta),$
 \[
 \begin{array}{ll}
 \|\phi^\star\|_2\leq 
  \frac{(1-p) \sigma_n(e_0)^2 \mathcal{R}(\frac{\mu(e_0)}{\sigma_n(e_0)})+  
 p\sqrt{1+ \sigma^2_n(e_0)} \mathcal{R}(\frac{\mu(e_0)}{\sqrt{1+ \sigma^2_n(e_0)}}) - p}{p (
(1-p)- \tan(\theta) )\cos{\theta} }
\end{array}
  \]  

 
\end{theorem}  
\begin{proof}
 Since
 \[
 \begin{array}{ll}
  B(p,\phi^\star) \|\phi^\star\|_2\leq 
 D_R(e_0) = F_{\eta_1,\eta_2, \sigma_1,\sigma_2}(e_0)-F_{0,0,0,0}(e_0) 
\end{array}
  \] 
  and
   \[
 \begin{array}{ll}
 p (
(1-p)- \tan(\theta) )\cos{\theta} \|\phi^\star\|_2\leq  B(p,\phi^\star) \|\phi^\star\|_2 
\end{array}
  \]  
  where
\[
 \begin{array}{ll}
 D_R(e_0) = (1-p) \sigma_n(e_0)^2 \mathcal{R}(\frac{\mu(e_0)}{\sigma_n(e_0)})+  
 p\sqrt{1+ \sigma^2_n(e_0)} \mathcal{R}(\frac{\mu(e_0)}{\sqrt{1+ \sigma^2_n(e_0)}}) - p
\end{array}
  \]  
where
\[
 \begin{array}{ll}
 \mu(e_0) &= (\eta_1 + \eta_2 \| a^{-1} \|_{q_2}-\epsilon)_+\\
 \sigma_n(e_0) &=\sqrt{ \sigma_1^2 + \sigma_2^2\|a^{-1}\|_2^2} 
 \end{array}
  \]
\end{proof}
\begin{coro}
When $\sigma_1 = \sigma_2 = 0,$
 \[
 \begin{array}{ll}
\|\psi^\star -e_0\|_2\leq 
 \frac{p\mathcal{R}(\frac{\mu(e_0)}{\sqrt{1+ \sigma^2_n(e_0)}}) - p}{p (
(1-p)- \tan(\theta) )\cos{\theta} }
\end{array}
  \]  

\end{coro}

\begin{coro}
When $\eta_1 = \eta_2 = 0,$
 \[
 \begin{array}{ll}
 \|\psi^\star -e_0\|_2\leq 
 \frac{ (1-p) \sigma_n(e_0)^2 +  
 p\sqrt{1+ \sigma^2_n(e_0)}  - p}{p (
(1-p)- \tan(\theta) )\cos{\theta} }
\end{array}
  \]  

\end{coro}








\subsection{All the proofs}    
\begin{proof}[Proof of lemma~\ref{lemma:Obj_Folded_FullNoise}]
 First, for any fixed vector $\mu$,
\[
 \begin{array}{ll}
  {\mathbb E}_{G_1} {\mathbb E}_{G_2} {\mathbb E}_X|\psi^T X+\sigma_1 \psi^T G_1+\sigma_2 \psi^T (a^{-1}* G_2) + \mu | = 
   {\mathbb E}_I{\mathbb E}_{G_3}  | \sqrt{ \|\psi_I\|_2^2 + \sigma_1^2\|\psi\|_2^2 + \sigma_2^2\|C_{a^{-1}}^T \psi\|_2^2}G_3  + \mu|
\end{array}
  \]
  Let 
 \[
 \begin{array}{ll}
 \mu(\psi) &= (\eta_1\|\psi\|_{q_1}+ \eta_2\|C_{a^{-1}}^T \psi\|_{q_2}-\epsilon)_+\\
 \sigma_n^2(\psi) &= \sqrt{ \sigma_1^2\|\psi\|_2^2 + \sigma_2^2\|C_{a^{-1}}^T \psi\|_2^2}\\
  \sigma_s^2(\psi_I) &=  \|\psi_I\|_2^2 \\
 \sigma^2 (\psi, \psi_I) &= \sigma_s^2(\psi_I) + \sigma_n^2(\psi) 
 \end{array}
  \]
 Using lemma~\ref{min-maxErrorL_p}, we have
\[
 \begin{array}{ll}
F_{\eta_1,\eta_2, \sigma_1,\sigma_2}(\psi)
&={\mathbb E}_I{\mathbb E}_{G}  | \sigma (\psi, \psi_I) G  + \mu(\psi) | \\
&={\mathbb E}_I{\mathbb E}_{G}  | \sqrt{\sigma_s^2(\psi_I) + \sigma_n^2(\psi)} G  + \mu(\psi) | \\
\end{array}
  \]    

\end{proof}

\begin{proof}[Proof of theorem~\ref{thm:Bound_FullNoise}]  
Additionally, for $\tilde a^{-1} $, define its norm as
\[
M_q: = \|\tilde a^{-1} \|_{q} = \| a^{-1} \|_{q}
\]

 When $\psi = e_0$, then
 \[
 \begin{array}{ll}
 \mu(e_0) &= \eta_1 + \eta_2 M_{q_2}\\
  \sigma_n(e_0) &=\sqrt{ \sigma_1^2 + \sigma_2^2 M_{2}^2} \\
 \sigma_s((e_0)_I) &= I_0 
 \end{array}
  \]

  
We have 
\[
 \begin{array}{ll}
 F_{\eta_1,\eta_2, \sigma_1,\sigma_2}(e_0)
 &= 
  {\mathbb E}_{G} {\mathbb E}_{I_0} |  \sigma(e_0, (e_0)_{I_0})  G  +  \mu(e_0)| \\
  &= 
 (1-p){\mathbb E}_{G} |  \sigma_n(e_0) G  +  \mu(e_0)|+ p{\mathbb E}_{G} | \sqrt{1+ \sigma^2_n(e_0)}  G  +  \mu(e_0)|\\
   &=
 \sqrt{\frac{2}{\pi}}[ (1-p) \sigma_n(e_0)^2 \mathcal{R}(\frac{\mu(e_0)}{\sigma_n(e_0)})+  
 p\sqrt{1+ \sigma^2_n(e_0)} \mathcal{R}(\frac{\mu(e_0)}{\sqrt{1+ \sigma^2_n(e_0)}}) ]
\end{array}
  \]
  Using the chain of inequality:
    \[
 \begin{array}{ll}
F_{\eta_1,\eta_2, \sigma_1,\sigma_2}((e_0)
 \geq F_{\eta_1,\eta_2, \sigma_1,\sigma_2}((\psi^\star ) 
 \geq F_{0,0,0,0}(\psi^\star )
 \geq F_{0,0,0,0}(e_0) 
\end{array}
  \] 
and the fact that
  \[
 \begin{array}{ll}
 F_{0,0,0,0}(\psi^\star )
 &= {\mathbb E}_I\|\psi_I\|_2 \\
 &= {\mathbb E}_I\|(e_0)_I\|_2 + ({\mathbb E}_I\|\psi_I\|_2-{\mathbb E}_I\|(e_0)_I\|_2) \\
 &= \sqrt{\frac{2}{\pi}}[p+ B(p,N) \|\psi -e_0\|_2]
\end{array}
  \]   
  From
   \[
 \begin{array}{ll}
 F_{0,0,0,0}(\psi^\star ) &\leq F_{\eta_1,\eta_2, \sigma_1,\sigma_2}(e_0)
\end{array}
  \]   
we have
  \[
 \begin{array}{ll}
p+ B(p,N) \|\psi^\star -e_0\|_2\leq (1-p) \sigma_n(e_0)^2 \mathcal{R}(\frac{\mu(e_0)}{\sigma_n(e_0)})+  
 p\sqrt{1+ \sigma^2_n(e_0)} \mathcal{R}(\frac{\mu(e_0)}{\sqrt{1+ \sigma^2_n(e_0)}}) 
\end{array}
  \]   

\end{proof}  







\begin{proof}[Proof of lemma~\ref{ExpansionMeanFoldedGaussian}]
Plug in the formula for mean of folded Gaussian \ref{FoldedGaussianMean}, 
we have
\[
E_G |\mu+ \sigma G| = \sqrt{\frac{2}{\pi}} \sigma \exp\{-\gamma^2/2\} + \mu \left(1 - 2\Phi\left(-\gamma\right) \right)
\]
Divide it by 
\[
E_G |\sigma G| =   \sqrt{\frac{2}{\pi}} \sigma
\]
we get the first equality.

Since the CDF of the standard normal distribution can be expanded by integration by parts into a series:
\[
\Phi( \gamma)=\frac{1}{2} + \frac{1}{\sqrt{2\pi}}\cdot e^{- \gamma^2/2} \left[ \gamma + \frac{ \gamma^3}{3} + \frac{ \gamma^5}{3\cdot 5} + \cdots + \frac{ \gamma^{2n+1}}{(2n+1)!!} + \cdots\right]
\]
where $!!$ denotes the double factorial. 

\[
1 - 2\Phi(-\gamma)  = \frac{1}{\sqrt{2\pi}}\cdot e^{- \gamma^2/2}\left[ \gamma + \frac{ \gamma^3}{3} + \frac{ \gamma^5}{3\cdot 5} + \cdots + \frac{ \gamma^{2n+1}}{(2n+1)!!} + \cdots\right]
\]
This gives the second inequality.

Additionally,
 \[
 \begin{array}{ll}
  e^{- \gamma^2/2} &=\sum_{n=0}^{\infty} \frac{ \gamma^{2n}}{(-2)^n (n)!}\\
  &=  1- \frac{ \gamma^2}{2} + \sum_{n=2}^{\infty} \frac{ \gamma^{2n}}{(-2)^n (n)!}
\end{array}
  \] 
then
 \[
 \begin{array}{ll}
& \exp\{-\gamma^2/2\}\{1 + \frac{1}{2}\gamma \left[ \gamma + \frac{ \gamma^3}{3} + \frac{ \gamma^5}{3\cdot 5} + \cdots + \frac{ \gamma^{2n+1}}{(2n+1)!!} + \cdots\right]\}\\
=& (\sum_{n=0}^{\infty} \frac{ \gamma^{2n}}{(-2)^n (n)!})\{1 + \frac{1}{2}\gamma \left[ \sum_{n=0}^{\infty} \frac{ \gamma^{2n+1}}{(2n+1)!!} \right]\}\\

=& 1 + \frac{3}{2}\gamma^2 + \frac{1}{24} \gamma^4   - \frac{ \gamma^6}{120}  + O(\gamma^8) 
\end{array}
  \] 
\end{proof}


\begin{proof}[Proof of lemma~\ref{ExpansionMeanFoldedGaussian_LB}]
Let $C_\Phi( \gamma^2)$ be a function of $ \gamma$, then it is a composition function with the inner funciton being $ \gamma^2,$ defined as follows: 
\[
 \begin{array}{ll}
C_\Phi( \gamma^2) 
&:= (2\Phi( \gamma)-1)/\left(\frac{ \gamma}{\sqrt{2\pi}}\right) \\
&= e^{- \gamma^2/2} \left[1 + \frac{ \gamma^2}{3} + \frac{ \gamma^4}{3\cdot 5} + \cdots + \frac{ \gamma^{2n}}{(2n+1)!!} + \cdots\right]\\
&= [\sum_{n=0}^{\infty} \frac{ \gamma^{2n}}{(2n+1)!!} ]/[\sum_{n=0}^{\infty} \frac{ \gamma^{2n}}{2^n (n)!}] \\
&= [\sum_{n=0}^{\infty} \frac{ \gamma^{2n}}{(2n+1)!!} ]/[\sum_{n=0}^{\infty} \frac{ \gamma^{2n}}{(2n)!!}] \\
&\leq 1
\end{array}
\]
We remark further that
\[
 \begin{array}{ll}
C_\Phi( \gamma^2) 
&= [\sum_{n=0}^{\infty} \frac{ \gamma^{2n}}{(2n+1)!!} ]/[\sum_{n=0}^{\infty} \frac{ \gamma^{2n}}{(2n)!!}] \\
&= [1 + \frac{ \gamma^2}{3}+ +  \frac{ \gamma^4}{15}+ O( \gamma^6) ]/[1 + \frac{ \gamma^2}{2}+ +  \frac{ \gamma^4}{8}+O( \gamma^6)] \\
&= 1 -\frac{ \gamma^2}{6}+  \frac{ \gamma^4}{40}+ O( \gamma^6)
\end{array}
\]
Then
 \[
 \begin{array}{ll}
\mu \left(1 - 2\Phi\left(-\gamma\right) \right)
&= \frac{1}{\sqrt{2\pi}}\cdot \mu^2 \frac{1}{\sigma} C_\Phi(\gamma^2)\\
&= \frac{1}{\sqrt{2\pi}}\cdot \mu^2 \frac{1}{\sigma} (1-\gamma^2/6 +O(\gamma^4))\\
\end{array}
\] 
Plug in all the fact together
\[
 \begin{array}{ll}
E_G |\mu+ \sigma G|
&=   \left[ \sqrt{\frac{2}{\pi}} \sigma \exp\{-\gamma^2/2\} + \frac{1}{\sqrt{2\pi}}\cdot \mu^2 \frac{1}{\sigma} C_\Phi(\gamma^2)\right]\\
&=   \left[ \sqrt{\frac{2}{\pi}} \sigma \{ 1- \frac{\gamma^2}{2} + \sum_{n=2}^{\infty} \frac{\gamma^{2n}}{(-2)^n (n)!}\} + \frac{1}{\sqrt{2\pi}}\cdot \mu^2 \frac{1}{\sigma} C_\Phi(\gamma^2)\right]\\
&= \sqrt{\frac{2}{\pi}}   \left[ \sigma 
\{ 1 + (1+\frac{1}{2}\cdot C_\Phi(\gamma^2))\gamma^2 +   \sum_{n=2}^{\infty} \frac{\gamma^{2n}}{(-2)^n (n)!}\} \}\right]\\
\end{array}
\] 
Define the residual of this expansion as
\[
 \begin{array}{ll}
\mbox{Res}(\gamma^2) 
&:= \frac{ (2\Phi( \gamma)-1)/\left(\frac{ \gamma}{\sqrt{2\pi}}\right)-1}{2}\cdot \gamma^2 +  (\exp\{-\gamma^2/2\} - 1 + \gamma^2/2)
\\
&= \frac{ C_\Phi(\gamma^2)-1}{2}\cdot \gamma^2 +  (\exp\{-\gamma^2/2\} - 1 + \gamma^2/2) \\
&=    \frac{ C_\Phi(\gamma^2)-1}{2}\cdot \gamma^2  +   \{ \sum_{n=2}^{\infty} \frac{\gamma^{2n}}{(-2)^n (n)!}\} \} \\
&=    \frac{-\gamma^4 }{12}  +    \frac{ \gamma^6}{80} +\frac{\gamma^{4}}{(-2)^2 (2)!} +\frac{\gamma^{6}}{(-2)^3 (3)!}+
O(\gamma^8)\\
&=    \frac{\gamma^4 }{24}  - \frac{ \gamma^6}{120}  + O(\gamma^8)
\end{array}
\] 
We know that if $\gamma^2<1,$ 
\[
 \begin{array}{ll}
\mbox{Res}(\gamma^2) 
&=    \frac{\gamma^4 }{24}  - \frac{ \gamma^6}{120}  + O(\gamma^8)\\
&\geq 0
\end{array}
\] 
(We comment that we could push the range for $\mbox{Res}(\gamma^2) 
\geq 0$ further to at least $\gamma^2<5$ if needed.)
Then we give an lower bound:
\[
 \begin{array}{ll}
E_G |\mu+ \sigma G| 
&= \sqrt{\frac{2}{\pi}} \sigma\left[ 1 + \frac{3}{2}\gamma^2+   \mbox{Res}\right]\\
&\geq \sqrt{\frac{2}{\pi}}\left[  \sigma + \frac{3}{2}\mu^2  \sigma^{-1}\right]
\end{array}
\] 


\end{proof}




\section{Stability for noisy observation}
\subsection{Inner and outer Gaussian and adversarial noise}
\paragraph{Mathematical formulation: sparse signal plus inner and outer adversarial perturbation and Gaussian noise}
\begin{itemize}
\item
Let $(X_t)_{t \in {\mathbb Z}}$ be IID sampled from  $pN(0,1) + (1-p) \delta_0$, write $X_t = I_t \cdot G_t$ where $I_t$ is IID sampled from Bernoulli, $G_t$ is IID sampled from standard Gaussian.
 \item
The filter $a\in l_1({\mathbb Z}), (\mathcal{F} a)(\omega) \neq 0, \forall \omega\in T.$
\item
Let $G_1$ and $G_2$ are two independent standard normal $N(0,I_N)$. 
\item
Let $Z_1$ and $Z_2$ are inner and outer adversarial noise with $p_1$ and $p_2$ norm bounded by $1$, where for $p_1, p_2\geq 1, $ $q_1$ and $q_2$ is defined by $\frac{1}{p_i}+\frac{1}{q_i} =1.$
\item
Our data generative model is
\[
\begin{array}{ll}
Y&= a* (X+ \eta_1 Z_1+  \sigma_1 G_1) + \eta_2 Z_2+ \sigma_2 G_2\\
&= a* (X +\eta_1 Z_1 +\eta_2 a^{-1}* Z_2+  \sigma_1 G_1+\sigma_2 a^{-1}* G_2)
\end{array}
\]
\item
Let the total perturbation be
\[
\begin{array}{ll}
Z= \eta_1 Z_1 +\eta_2 a^{-1}* Z_2+  \sigma_1 G_1+\sigma_2 a^{-1}* G_2
\end{array}
\]
We can write data generative model  simply as
\[
\begin{array}{ll}
Y&= a* (X+ Z ) 
\end{array}
\]
\end{itemize}
Let $\tilde{\psi} := w*a$,   $\phi: = \psi - e_0$, then
\[
 \begin{array}{ll}
(w*Y)_0
&= \psi^T (X+  Z ) 
\end{array}
  \]

 \paragraph{Perturbation analysis of the robust convex problems}
The original convex problems is
\[
 \begin{array}{ll}
 \mbox{minimize}_{w}   & {\mathbb E}|\tilde{w}^T Y|\\
\mbox{subject to}  
& v^T w =1,
\end{array}
\tag{$P_1$}
  \]  
Our objective is
\[
|\psi^T (X+  Z )-\epsilon \zeta|  = |\psi^T X+\psi^T[\sigma_1  G_1+\sigma_2 (a^{-1}* G_2) + \eta_1 Z_1 +\eta_2 a^{-1}* Z_2]-\epsilon \zeta |
\]
\paragraph{Worst case robust objective over adversarial perturbation}  
For the constrained problem, we first take expectation over Gaussian noise and over the Bernoulli Gaussian signal, then we optimize over the worst case expected objective for all adversarial perturbation, we define our robust objective as   
\[
 \begin{array}{ll}
 F_{\eta_1,\eta_2, \sigma_1,\sigma_2}(\psi)
:=
\inf_{\zeta^2\leq 1}\sup_{Z_1, Z_2: \|Z_1\|_{p_1} \leq 1; \|Z_2\|_{p_2} \leq 1}
{\mathbb E}_{G_1} {\mathbb E}_{G_2} {\mathbb E}_X|\psi^T (X+  Z )-\epsilon \zeta|
\end{array}
  \]  
then our robust constrained problem becomes
  \[
 \begin{array}{ll}
 \mbox{minimize}_{\psi} \quad &F_{\eta_1,\eta_2, \sigma_1,\sigma_2}(\psi) \\
\mbox{subject to}  
&u^T \psi =1
\end{array}
  \]
Assume that the minimizer of the robust problem is $\psi^\star$, then we have the chain of inequality:
    \[
 \begin{array}{ll}
F_{\eta_1,\eta_2, \sigma_1,\sigma_2}(e_0)
 \geq F_{\eta_1,\eta_2, \sigma_1,\sigma_2}(\psi^\star ) 
 \geq F_{0,0,0,0}(\psi^\star )
 \geq F_{0,0,0,0}(e_0) =p
\end{array}
  \] 
In the following, we use this fact and the bi-lipschitz property of noiseless objective around $e_0$ to bound the $\ell_2$ norm deviation $\|\psi^\star-e_0\|_2.$
   \[
 \begin{array}{ll}
F_{\eta_1,\eta_2, \sigma_1,\sigma_2}(e_0)-F_{0,0,0,0}(e_0)
 &\geq F_{0,0,0,0}(\psi^\star )-F_{0,0,0,0}(e_0) \\
 &\geq B(p,N) \|\psi^\star-e_0\|_2
\end{array}
  \] 

This bound is not tight but it is tight in order and it is easy to compute since we only need to compute  $F_{\eta_1,\eta_2, \sigma_1,\sigma_2}(e_0)$, the robust objective at $e_0$.
 
\subsection{Folded Gaussian}
\paragraph{Folded Gaussian}
For Bernoulli Gaussian $X$ with support $I$, if we condition on $I$ fixed, then  using the rotation invariance of Gaussian, $\psi^T X$ is  Gaussian with mean $0$ and standard deviation $\|\psi_I\|_2$.

From general theory of folded Gaussian,
\footnote{see: \url{ https://en.wikipedia.org/wiki/Folded_normal_distribution}.}
 $ |\mu+ \sigma G |, \quad  G \sim \mathcal{N}(0, 1).$
Then its mean is
\begin{equation}
\label{FoldedGaussianMean}
E_G |\mu+ \sigma G| =  \sqrt{\frac{2}{\pi}} \sigma \exp\{-\mu^2/(2\sigma^2)\} + \mu \left(1 - 2\Phi\left(\frac{-\mu}{\sigma}\right) \right)
\end{equation}
where $\Phi$  is the normal cumulative distribution function:
\[
\Phi(x) = \frac 1 {\sqrt{2\pi}} \int_{-\infty}^x e^{-t^2/2} \, dt.
\]
Its variance is
\begin{equation}
\label{FoldedGaussianVar}
{\rm Var}_G |\mu+ \sigma G| = E_G( |\mu+ \sigma G|-E_G |\mu+ \sigma G|)^2 =\sigma^2+ \mu^2 -(E_G |\mu+ \sigma G|)^2
\end{equation}  
The variance will be useful for concentration with finite sample.  

\paragraph{Asymptotic expansion for mean of folded Gaussian}
Notice that if $\sigma =0$, then
\[
E_G |\mu+ 0\cdot G| =   |\mu|.
\]
If $\mu =0$, then we go back to mean of half-normal,
\[
E_G |\sigma G| =   \sqrt{\frac{2}{\pi}} \sigma
\]
Let 
\[
\gamma:= \frac{\mu}{\sigma}
\] 

\begin{lemma}
\label{ExpansionMeanFoldedGaussian}
We define the ratio of folded gaussian mean as 
\[
\mathcal{R}^f(\gamma) := \frac{E_G |\mu+ \sigma G|}{E_G |\sigma G|} = \frac{E_G |\mu+ \sigma G|}{\sqrt{\frac{2}{\pi}}\sigma} = \sqrt{\frac{\pi}{2}} E_G |\gamma+  G|
\]
We have three different expressions (asymptotic expansion around  $\gamma$) for $\mathcal{R}^f(\gamma)$:
\begin{eqnarray}
\label{FoldedGaussianMean_ratio}
\mathcal{R}^f(\gamma)
&=&  \exp\{-\gamma^2/2\} + \sqrt{\frac{\pi}{2}}\gamma \left(1 - 2\Phi\left(-\gamma\right) \right)\\
&=&  \exp\{-\gamma^2/2\}\{1 + \frac{1}{2}\gamma \left[ \gamma + \frac{ \gamma^3}{3} + \frac{ \gamma^5}{3\cdot 5} + \cdots + \frac{ \gamma^{2n+1}}{(2n+1)!!} + \cdots\right]\}\\
&=& 1 + \frac{3}{2}\gamma^2 + \frac{1}{24} \gamma^4   - \frac{ \gamma^6}{120}  + O(\gamma^8) 
\end{eqnarray}
 \end{lemma}
 
 \begin{lemma}
 \label{ExpansionMeanFoldedGaussian_LB}
when $\gamma<2,$ 
\[
\mathcal{R}^f(\gamma) \geq 1 + \frac{3}{2}\gamma^2
\]
\end{lemma}


\paragraph{Robust optimization}
We want to find the min-max value 
\[
\inf_{\zeta^2\leq 1}\sup_{Z_1, Z_2: \|Z_1\|_{p_1} \leq 1; \|Z_2\|_{p_2} \leq 1} H( \psi^T (\eta_1 Z_1 +\eta_2 a^{-1}* Z_2)  - \epsilon \zeta)
\]
for a function $H$,
\[
H(t) := {\mathbb E}_{G_1} {\mathbb E}_{G_2} {\mathbb E}_X|t+ \psi^T (X+  \sigma_1  G_1+\sigma_2 (a^{-1}* G_2) )|
\]
\begin{lemma}
From Hölder's inequality, for any vector $a \in {\mathbb R}^N$, for $p\geq 1, \frac{1}{p}+\frac{1}{q} =1,$
\[
\sup_{Z: \|Z\|_p\leq 1}(a^T Z) = \|a\|_q,
\]
specifically, when $|Z|^p$ is along the direction of $\mbox{sign}(a)|a|^q$ so that $a^T Z = \|a\|_q$.

\end{lemma}  


\begin{lemma}
\label{min-maxError}
For $A>0$, 
\[
 \begin{array}{ll}
\inf_{\zeta: \zeta^2 \leq 1} (A - \epsilon \zeta) = (A - \epsilon )_+
\end{array}
\]
where $(\cdot)_+: = \max{(0, \cdot)}$ is the ReLU function.
\end{lemma}
\begin{lemma}
\label{min-maxErrorL_p}
Let
 \[
 \begin{array}{ll}
 \mu(\psi) &=(\eta_1\|\psi\|_{q_1}+ \eta_2\|C_{a^{-1}}^T \psi\|_{q_2}-\epsilon)_+
 \end{array}
  \]
  where $(\cdot)_+: = \max{(0, \cdot)}$ is the ReLU function,
  then
\[
 \begin{array}{ll}
\inf_{\zeta^2\leq 1}\sup_{Z_1, Z_2: \|Z_1\|_{p_1} \leq 1; \|Z_2\|_{p_2} \leq 1} H( \psi^T (\eta_1 Z_1 +\eta_2 a^{-1}* Z_2)  - \epsilon \zeta) = 
H( \mu(\psi))
\end{array}
\]

\end{lemma}
\paragraph{Bound on the noisy solution using robust objective}  
\begin{lemma}
\label{lemma:Obj_Folded_FullNoise}
Let 
 \[
 \begin{array}{ll}
 \mu(\psi) &=(\eta_1\|\psi\|_{q_1}+ \eta_2\|C_{a^{-1}}^T \psi\|_{q_2}-\epsilon)_+\\
 \sigma_n^2(\psi) &= \sqrt{ \sigma_1^2\|\psi\|_2^2 + \sigma_2^2\|C_{a^{-1}}^T \psi\|_2^2}
 \end{array}
  \]
then the objective is a mean of a folded Gaussian random variable:
\[
 \begin{array}{ll}
 F_{\eta_1,\eta_2, \sigma_1,\sigma_2}(\psi)
&= {\mathbb E}_I{\mathbb E}_{G}  | \sqrt{\|\psi_I\|_2^2 + \sigma_n^2(\psi)} G  + \mu(\psi) |\\
&= \sqrt{\frac{2}{\pi}} {\mathbb E}_I[ \sqrt{\|\psi_I\|_2^2 + \sigma_n^2(\psi)} 
\mathcal{R}^f(\frac{\mu(\psi)}{\sqrt{\|\psi_I\|_2^2 + \sigma_n^2(\psi)}})]
\end{array}
  \] 
  where $G$ is scalar standard Gaussian.
\end{lemma}
From a simple inequality:
   \[
 \begin{array}{ll}
 F_{0,0,0,0}(\psi^\star ) &\leq F_{\eta_1,\eta_2, \sigma_1,\sigma_2}(e_0)
\end{array}
  \]  
we have  
\begin{theorem}
\label{thm:Bound_FullNoise}
\[
 \begin{array}{ll}
 B(p,N) \|\psi^\star -e_0\|_2\leq (1-p) \sigma_n(e_0)^2 \mathcal{R}^f(\frac{\mu(e_0)}{\sigma_n(e_0)})+  
 p\sqrt{1+ \sigma^2_n(e_0)} \mathcal{R}^f(\frac{\mu(e_0)}{\sqrt{1+ \sigma^2_n(e_0)}}) - p
\end{array}
  \]  
where
\[
 \begin{array}{ll}
 \mu(e_0) &= (\eta_1 + \eta_2 \| a^{-1} \|_{q_2}-\epsilon)_+\\
 \sigma_n(e_0) &=\sqrt{ \sigma_1^2 + \sigma_2^2\|a^{-1}\|_2^2} 
 \end{array}
  \]
\end{theorem}  

\subsection{Proofs}    
\begin{proof}[Proof of lemma~\ref{lemma:Obj_Folded_FullNoise}]
 First, for any fixed vector $\mu$,
\[
 \begin{array}{ll}
  {\mathbb E}_{G_1} {\mathbb E}_{G_2} {\mathbb E}_X|\psi^T X+\sigma_1 \psi^T G_1+\sigma_2 \psi^T (a^{-1}* G_2) + \mu | = 
   {\mathbb E}_I{\mathbb E}_{G_3}  | \sqrt{ \|\psi_I\|_2^2 + \sigma_1^2\|\psi\|_2^2 + \sigma_2^2\|C_{a^{-1}}^T \psi\|_2^2}G_3  + \mu|
\end{array}
  \]
  Let 
 \[
 \begin{array}{ll}
 \mu(\psi) &= (\eta_1\|\psi\|_{q_1}+ \eta_2\|C_{a^{-1}}^T \psi\|_{q_2}-\epsilon)_+\\
 \sigma_n^2(\psi) &= \sqrt{ \sigma_1^2\|\psi\|_2^2 + \sigma_2^2\|C_{a^{-1}}^T \psi\|_2^2}\\
  \sigma_s^2(\psi_I) &=  \|\psi_I\|_2^2 \\
 \sigma^2 (\psi, \psi_I) &= \sigma_s^2(\psi_I) + \sigma_n^2(\psi) 
 \end{array}
  \]
 Using lemma~\ref{min-maxErrorL_p}, we have
\[
 \begin{array}{ll}
F_{\eta_1,\eta_2, \sigma_1,\sigma_2}(\psi)
&={\mathbb E}_I{\mathbb E}_{G}  | \sigma (\psi, \psi_I) G  + \mu(\psi) | \\
&={\mathbb E}_I{\mathbb E}_{G}  | \sqrt{\sigma_s^2(\psi_I) + \sigma_n^2(\psi)} G  + \mu(\psi) | \\
\end{array}
  \]    

\end{proof}

\begin{proof}[Proof of theorem~\ref{thm:Bound_FullNoise}]  
Additionally, for $\tilde a^{-1} $, define its norm as
\[
M_q: = \|\tilde a^{-1} \|_{q} = \| a^{-1} \|_{q}
\]

 When $\psi = e_0$, then
 \[
 \begin{array}{ll}
 \mu(e_0) &= \eta_1 + \eta_2 M_{q_2}\\
  \sigma_n(e_0) &=\sqrt{ \sigma_1^2 + \sigma_2^2 M_{2}^2} \\
 \sigma_s((e_0)_I) &= I_0 
 \end{array}
  \]

  
We have 
\[
 \begin{array}{ll}
 F_{\eta_1,\eta_2, \sigma_1,\sigma_2}(e_0)
 &= 
  {\mathbb E}_{G} {\mathbb E}_{I_0} |  \sigma(e_0, (e_0)_{I_0})  G  +  \mu(e_0)| \\
  &= 
 (1-p){\mathbb E}_{G} |  \sigma_n(e_0) G  +  \mu(e_0)|+ p{\mathbb E}_{G} | \sqrt{1+ \sigma^2_n(e_0)}  G  +  \mu(e_0)|\\
   &=
 \sqrt{\frac{2}{\pi}}[ (1-p) \sigma_n(e_0)^2 \mathcal{R}^f(\frac{\mu(e_0)}{\sigma_n(e_0)})+  
 p\sqrt{1+ \sigma^2_n(e_0)} \mathcal{R}^f(\frac{\mu(e_0)}{\sqrt{1+ \sigma^2_n(e_0)}}) ]
\end{array}
  \]
  Using the chain of inequality:
    \[
 \begin{array}{ll}
F_{\eta_1,\eta_2, \sigma_1,\sigma_2}((e_0)
 \geq F_{\eta_1,\eta_2, \sigma_1,\sigma_2}((\psi^\star ) 
 \geq F_{0,0,0,0}(\psi^\star )
 \geq F_{0,0,0,0}(e_0) 
\end{array}
  \] 
and the fact that
  \[
 \begin{array}{ll}
 F_{0,0,0,0}(\psi^\star )
 &= {\mathbb E}_I\|\psi_I\|_2 \\
 &= {\mathbb E}_I\|(e_0)_I\|_2 + ({\mathbb E}_I\|\psi_I\|_2-{\mathbb E}_I\|(e_0)_I\|_2) \\
 &= \sqrt{\frac{2}{\pi}}[p+ B(p,N) \|\psi -e_0\|_2]
\end{array}
  \]   
  From
   \[
 \begin{array}{ll}
 F_{0,0,0,0}(\psi^\star ) &\leq F_{\eta_1,\eta_2, \sigma_1,\sigma_2}(e_0)
\end{array}
  \]   
we have
  \[
 \begin{array}{ll}
p+ B(p,N) \|\psi^\star -e_0\|_2\leq (1-p) \sigma_n(e_0)^2 \mathcal{R}^f(\frac{\mu(e_0)}{\sigma_n(e_0)})+  
 p\sqrt{1+ \sigma^2_n(e_0)} \mathcal{R}^f(\frac{\mu(e_0)}{\sqrt{1+ \sigma^2_n(e_0)}}) 
\end{array}
  \]   

\end{proof}  







\begin{proof}[Proof of lemma~\ref{ExpansionMeanFoldedGaussian}]
Plug in the formula for mean of folded Gaussian \ref{FoldedGaussianMean}, 
we have
\[
E_G |\mu+ \sigma G| = \sqrt{\frac{2}{\pi}} \sigma \exp\{-\gamma^2/2\} + \mu \left(1 - 2\Phi\left(-\gamma\right) \right)
\]
Divide it by 
\[
E_G |\sigma G| =   \sqrt{\frac{2}{\pi}} \sigma
\]
we get the first equality.

Since the CDF of the standard normal distribution can be expanded by integration by parts into a series:
\[
\Phi( \gamma)=\frac{1}{2} + \frac{1}{\sqrt{2\pi}}\cdot e^{- \gamma^2/2} \left[ \gamma + \frac{ \gamma^3}{3} + \frac{ \gamma^5}{3\cdot 5} + \cdots + \frac{ \gamma^{2n+1}}{(2n+1)!!} + \cdots\right]
\]
where $!!$ denotes the double factorial. 

\[
1 - 2\Phi(-\gamma)  = \frac{1}{\sqrt{2\pi}}\cdot e^{- \gamma^2/2}\left[ \gamma + \frac{ \gamma^3}{3} + \frac{ \gamma^5}{3\cdot 5} + \cdots + \frac{ \gamma^{2n+1}}{(2n+1)!!} + \cdots\right]
\]
This gives the second inequality.

Additionally,
 \[
 \begin{array}{ll}
  e^{- \gamma^2/2} &=\sum_{n=0}^{\infty} \frac{ \gamma^{2n}}{(-2)^n (n)!}\\
  &=  1- \frac{ \gamma^2}{2} + \sum_{n=2}^{\infty} \frac{ \gamma^{2n}}{(-2)^n (n)!}
\end{array}
  \] 
then
 \[
 \begin{array}{ll}
& \exp\{-\gamma^2/2\}\{1 + \frac{1}{2}\gamma \left[ \gamma + \frac{ \gamma^3}{3} + \frac{ \gamma^5}{3\cdot 5} + \cdots + \frac{ \gamma^{2n+1}}{(2n+1)!!} + \cdots\right]\}\\
=& (\sum_{n=0}^{\infty} \frac{ \gamma^{2n}}{(-2)^n (n)!})\{1 + \frac{1}{2}\gamma \left[ \sum_{n=0}^{\infty} \frac{ \gamma^{2n+1}}{(2n+1)!!} \right]\}\\

=& 1 + \frac{3}{2}\gamma^2 + \frac{1}{24} \gamma^4   - \frac{ \gamma^6}{120}  + O(\gamma^8) 
\end{array}
  \] 
\end{proof}


\begin{proof}[Proof of lemma~\ref{ExpansionMeanFoldedGaussian_LB}]
Let $C_\Phi( \gamma^2)$ be a function of $ \gamma$, then it is a composition function with the inner funciton being $ \gamma^2,$ defined as follows: 
\[
 \begin{array}{ll}
C_\Phi( \gamma^2) 
&:= (2\Phi( \gamma)-1)/\left(\frac{ \gamma}{\sqrt{2\pi}}\right) \\
&= e^{- \gamma^2/2} \left[1 + \frac{ \gamma^2}{3} + \frac{ \gamma^4}{3\cdot 5} + \cdots + \frac{ \gamma^{2n}}{(2n+1)!!} + \cdots\right]\\
&= [\sum_{n=0}^{\infty} \frac{ \gamma^{2n}}{(2n+1)!!} ]/[\sum_{n=0}^{\infty} \frac{ \gamma^{2n}}{2^n (n)!}] \\
&= [\sum_{n=0}^{\infty} \frac{ \gamma^{2n}}{(2n+1)!!} ]/[\sum_{n=0}^{\infty} \frac{ \gamma^{2n}}{(2n)!!}] \\
&\leq 1
\end{array}
\]
We remark further that
\[
 \begin{array}{ll}
C_\Phi( \gamma^2) 
&= [\sum_{n=0}^{\infty} \frac{ \gamma^{2n}}{(2n+1)!!} ]/[\sum_{n=0}^{\infty} \frac{ \gamma^{2n}}{(2n)!!}] \\
&= [1 + \frac{ \gamma^2}{3}+ +  \frac{ \gamma^4}{15}+ O( \gamma^6) ]/[1 + \frac{ \gamma^2}{2}+ +  \frac{ \gamma^4}{8}+O( \gamma^6)] \\
&= 1 -\frac{ \gamma^2}{6}+  \frac{ \gamma^4}{40}+ O( \gamma^6)
\end{array}
\]
Then
 \[
 \begin{array}{ll}
\mu \left(1 - 2\Phi\left(-\gamma\right) \right)
&= \frac{1}{\sqrt{2\pi}}\cdot \mu^2 \frac{1}{\sigma} C_\Phi(\gamma^2)\\
&= \frac{1}{\sqrt{2\pi}}\cdot \mu^2 \frac{1}{\sigma} (1-\gamma^2/6 +O(\gamma^4))\\
\end{array}
\] 
Plug in all the fact together
\[
 \begin{array}{ll}
E_G |\mu+ \sigma G|
&=   \left[ \sqrt{\frac{2}{\pi}} \sigma \exp\{-\gamma^2/2\} + \frac{1}{\sqrt{2\pi}}\cdot \mu^2 \frac{1}{\sigma} C_\Phi(\gamma^2)\right]\\
&=   \left[ \sqrt{\frac{2}{\pi}} \sigma \{ 1- \frac{\gamma^2}{2} + \sum_{n=2}^{\infty} \frac{\gamma^{2n}}{(-2)^n (n)!}\} + \frac{1}{\sqrt{2\pi}}\cdot \mu^2 \frac{1}{\sigma} C_\Phi(\gamma^2)\right]\\
&= \sqrt{\frac{2}{\pi}}   \left[ \sigma 
\{ 1 + (1+\frac{1}{2}\cdot C_\Phi(\gamma^2))\gamma^2 +   \sum_{n=2}^{\infty} \frac{\gamma^{2n}}{(-2)^n (n)!}\} \}\right]\\
\end{array}
\] 
Define the residual of this expansion as
\[
 \begin{array}{ll}
\mbox{Res}(\gamma^2) 
&:= \frac{ (2\Phi( \gamma)-1)/\left(\frac{ \gamma}{\sqrt{2\pi}}\right)-1}{2}\cdot \gamma^2 +  (\exp\{-\gamma^2/2\} - 1 + \gamma^2/2)
\\
&= \frac{ C_\Phi(\gamma^2)-1}{2}\cdot \gamma^2 +  (\exp\{-\gamma^2/2\} - 1 + \gamma^2/2) \\
&=    \frac{ C_\Phi(\gamma^2)-1}{2}\cdot \gamma^2  +   \{ \sum_{n=2}^{\infty} \frac{\gamma^{2n}}{(-2)^n (n)!}\} \} \\
&=    \frac{-\gamma^4 }{12}  +    \frac{ \gamma^6}{80} +\frac{\gamma^{4}}{(-2)^2 (2)!} +\frac{\gamma^{6}}{(-2)^3 (3)!}+
O(\gamma^8)\\
&=    \frac{\gamma^4 }{24}  - \frac{ \gamma^6}{120}  + O(\gamma^8)
\end{array}
\] 
We know that if $\gamma^2<1,$ 
\[
 \begin{array}{ll}
\mbox{Res}(\gamma^2) 
&=    \frac{\gamma^4 }{24}  - \frac{ \gamma^6}{120}  + O(\gamma^8)\\
&\geq 0
\end{array}
\] 
(We comment that we could push the range for $\mbox{Res}(\gamma^2) 
\geq 0$ further to at least $\gamma^2<5$ if needed.)
Then we give an lower bound:
\[
 \begin{array}{ll}
E_G |\mu+ \sigma G| 
&= \sqrt{\frac{2}{\pi}} \sigma\left[ 1 + \frac{3}{2}\gamma^2+   \mbox{Res}\right]\\
&\geq \sqrt{\frac{2}{\pi}}\left[  \sigma + \frac{3}{2}\mu^2  \sigma^{-1}\right]
\end{array}
\] 


\end{proof}



\subsection{Constrained Convex algorithms }
 \paragraph{Constrained noisy convex problems}
Then we define the constrained convex program that allows noise,
\[
 \begin{array}{ll}
 \mbox{minimize}_{w,z}   & |\alpha| \\
\mbox{subject to}  
& v^T w =1,\\
& {\mathbb E}(\alpha -\tilde{w}^T Y)^2 \leq \epsilon^2
\end{array}
\tag{$P^c_{1,\epsilon}$}
\label{eq:ConstrainProb}
  \]
  If we write $\alpha =\tilde{w}^T Y - \epsilon \zeta $ , then
\[
 \begin{array}{ll}
 \mbox{minimize}_{w,\zeta}   & {\mathbb E}|\tilde{w}^T Y - \epsilon \zeta| \\
\mbox{subject to}  
& v^T w =1,\\
& \zeta^2 \leq 1
\end{array}
\tag{$P^c_{1,\epsilon}$}
  \]
If we set $\epsilon = 0,$ then we recover 
the original convex problems for noiseless setting
\[
 \begin{array}{ll}
 \mbox{minimize}_{w}   & {\mathbb E}|\tilde{w}^T Y|\\
\mbox{subject to}  
& v^T w =1,
\end{array}
\tag{$P_1$}
  \]
\section{Numerical experiments}
We set up a ground truth inverse filter as $a^{-1} = (0.5, 1, 0.5)$, and numerically run $10$ independent experiments of finding the inverse filter by the convex problem 
\begin{equation}
 \begin{array}{ll}
  \underset{w\in V}{ \mbox{minimize} }  &  \frac{1}{2T+1}\|w*Y\|_{1,[-T,T]}\\
 \mbox{subject to}  
&  w_0 =1.
\end{array}
\tag{$P_{e_0}^T$ }
\end{equation}
We choose an accuracy $\epsilon = 10^{-3}$ and count the number of successfully accurate recovery of $a^{-1}$ where $\|w^\star -a^{-1}\|_1/\|a^{-1}\|_\infty<\epsilon$. We see a consistent sharp transition of success rate from $0$ to $1$ in a narrow parameter region.
\begin{figure} 
\centering
\includegraphics[width=.7\textwidth]{./figures_BD/phase_transition_detail.png}
\caption{The finite sample phase transition phenomenon, where $a^{-1} = (0.5, 1, 0.5)$,  $x-$axis is the sparsity level $p$, $y-$axis is the empirical probability $P(\|w^\star -a^{-1}\|_1/\|a^{-1}\|_\infty<10^{-3})$. Red part is failure, Blue part is success.}
\label{fig-PT}
\end{figure}
\subsection{Population Phase transition theorem}
Qingyun: Old proof without subgradient language:

\paragraph{Population phase transition theorem}
\begin{theorem}
Under the previous assumption,
$
         p^\star   =\frac{\sin{\theta}}{\sin{\theta}+ \cos{\theta}} .
$
 The solution of  $P_{1,v}$ is $w^\star$, then 
 $w^\star$ is $a^{-1}$ up to shift and scaling provided $p<p^\star$,
and  $w^\star$ is not $a^{-1}$ up to shift and scaling provided $p> p^\star$.
  \end{theorem}
The population phase transition diagram is in figure~\ref{fig-PT}.

\begin{figure} 
\centering
\includegraphics[width=.7\textwidth]{./figures_BD/Phase_T_sine.png}
\caption{The population phase transition diagram derived from theory for planted sparsity projection pursuit, $x-$axis is $\sin{\theta}$, $y-$axis is the sparsity level $p$ . Red part is failure, Blue part is success.}
\label{fig-PT}
\end{figure}

\begin{figure} 
\centering
\includegraphics[width=.7\textwidth]{./figures_BD/PT_BD_1root_pure.png}
\caption{The finite sample phase transition diagram when ground truth filter is $w^\star = (0,1,s)$ with $T=200$.$x-$axis is sparsity level $p$ ranging from $0.01$ to $0.99$ with step size $0.01$, $y-$axis is $\tan(\theta)=\| w_{(0)}^\star \|_2$, ranging from $0$ to $0.99$ with step size $0.02$.  Red part is failure, Blue part is success. The success probability is calculated over $10$ independent repeated experiments solved by cvx in matlab. }
\label{fig-PT-finite}
\end{figure}






\paragraph{Proof of the population phase transition theorem}
\begin{itemize}
 \item
  Let $\phi  := \psi -e_0 = \tilde w*\tilde a -e_0,$  then $|[w*Y]_0| = |[(e_0 +\tilde \phi)*X]_0| =  |[(e_0 + \phi)^T X| $, we define
  \[
  L(\phi) = |[(e_0 + \phi)^T X|.
  \] 
  WLOG, we can assume the ground truth $a^{-1}$ is scaled so that $v^T \tilde{a}^{-1} =1$

  We want to find a threshold $p^\star$ so that for all $p<p^\star$, the directional derivative  $ {\mathbb E} \nabla_{\phi}{L}|_{\phi = 0}>0$ for all directions $\phi$ that satisfy $u^T \phi = 0$.

 We define the finite difference
  \[
  D = L(\phi)-L(0) =  |[(e_0 + \phi)^T X| -|X_0| 
  \]
  Let $\epsilon : = \|\phi\|_2 $ and taking limit of $D$ as $\epsilon  \rightarrow 0,$ we have
    \begin{equation}
  \begin{array}{ll}
   {\mathbb E} D &=  p{\mathbb E}[D\mid X_0 \neq 0 ] + 
   (1-p){\mathbb E}[D \mid X_0 = 0 ] \\
   &=  p{\mathbb E}[\mbox{sign}(X_0) ( \phi_0 X_0 + \sum_{ j\neq 0}\phi_{j} X_{j} )\mid X_0 \neq 0 ] + (1-p){\mathbb E}\mid \sum_{ j\neq 0}\phi_{j} X_{j}\mid   \\
   &= p \phi_0 {\mathbb E} [|X_0|\mid X_0 \neq 0 ] + (1-p){\mathbb E}\mid \sum_{ j\neq 0}\phi_{j} X_{j}\mid  
   \end{array}
 \end{equation}
The last equality is because 
 $
 {\mathbb E}[\mbox{sign}(X_0) (\sum_{ j\neq 0}\phi_{j} X_{j} )\mid X_0 \neq 0 ] = 0 
$
 as $X_j$ are sampled independently. Notice that for Bernoulli Gaussian $X$,
$
{\mathbb E} [|X_0|\mid X_0 \neq 0 ]=\sqrt{\frac{2}{\pi}}.
$
 
 Then  $ {\mathbb E} \nabla_{\phi}{L}|_{\phi = 0} = {\mathbb E} \lim_{\epsilon \rightarrow 0}\frac{D}{\epsilon}>0$ 
 in case either $\phi_{0} > 0$, or in case $\phi_0 < 0$ and
\[
          \frac{p}{1-p} < \frac{ {\mathbb E}\mid \sum_{ j\neq 0}\phi_{j} X_{j}\mid  }{|\phi_0| 
          {\mathbb E} [|X_0|\mid X_0 \neq 0 ]} = \frac{\|\phi_{(0)}\|_2}{| \phi_{0}|} \frac{ {\mathbb E} | \phi_{(0)}^T X_{(0)} |}{\|\phi_{(0)}\|_2 \sqrt{\frac{2}{\pi}}}  .
\]
Here  $X_{(0)} = (X_i)_{i\neq 0}$
\item (Geometric lower bound:)
Using the fact that $u^T \phi = 0$. We define $v = \cos{(\theta)}\frac{\tilde{a}}{\|\tilde{a}\|_2} + \sin{(\theta)} (\frac{\tilde{a}}{\|\tilde{a}\|_2} )^{\bot}$, then $u =\tilde{a}^{-1}*v = \tilde{a}^{-1}*(\cos{(\theta)}\frac{\tilde{a}}{\|\tilde{a}\|_2} + \sin{(\theta)} (\frac{\tilde{a}}{\|\tilde{a}\|_2})^{\bot}) =\cos{(\theta)} \frac{e_0}{\|\tilde{a}\|_2}+\sin{(\theta)} \tilde{a}^{-1}*(\frac{\tilde{a}}{\|\tilde{a}\|_2} )^{\bot} $, 
where $\tilde{a}^{-1}*(\frac{\tilde{a}}{\|\tilde{a}\|_2} )^{\bot}$ is perpendicular to $e_0$, 
therefore, 
$0= u^T \phi = \frac{1}{\|\tilde{a}\|_2}[\cos{(\theta)} \phi_0+\sin{(\theta)} \phi_{(0)}]$, therefore,  
$\frac{\|\phi_{(0)}\|_2}{| \phi_{0}|} = \frac{\cos(\theta)}{\sin(\theta)} =\cot(\theta).$
\[
 \frac{\|\phi_{(0)}\|_2}{| \phi_{0}|}\geq  \cot(\theta)
 \]

 
\item
We showed that for IID Bernoulli Gaussian $X$,
\[
\frac{ {\mathbb E} | \phi_{(0)}^T X_{(0)} |}{\sqrt{\frac{2}{\pi}} \|\phi_{(0)}\|_2} = \frac{ {\mathbb E}_{I_{(0)}}\| \phi_{I_{(0)}}\|_2}{ \|\phi_{(0)}\|_2} \in [p,1)
\]
Using semicontinuous extension, we can choose sequence of $\phi$ that is dense on doubly infinite sequence space $V_{-\infty, \infty}$, then we know that $\frac{ {\mathbb E}_{I_{(0)}}\| \phi_{I_{(0)}}\|_2}{ \|\phi_{(0)}\|_2}$ should be replaced by its semicontinuous extension on $V_{-\infty, \infty}$, which is $1$.

The phase transition $p^\star$ is the smallest $p$ so that 
\[
          \frac{p}{1-p} \leq  \cot(\theta) \frac{ {\mathbb E}_{I_{(0)}}\| \phi_{I_{(0)}}\|_2}{ \|\phi_{(0)}\|_2} \leq  \cot(\theta).
\]
Therefore,  generically, $ \frac{p^\star}{1-p^\star}  = \cot(\theta) $.
\EIT
\section{Projection pursuit for planted sparsity}



 
 
 
 
 









 


\section{Phase transition for sub-Gaussian}
From the Khinthine’s inequality for sub-Gaussian and $p = 1$, we have 
\[
E{|h_{(0)}^T X_{(0)}| }\geq  3^{-3/2}C^{-3} K^{-3}  \|h_{(0)}\|_2,
\]
for Bernoulli-SubGaussian, we have,
\[
E{|h_{(0)}^T X_{(0)}| }\geq  (1-(1-p)^M)^{1/2} 3^{-3/2}C^{-3} K^{-3}  \|h_{(0)}\|_2.
\]
In general, we define
\[
\gamma(F,p) = E{|h_{(0)}^T X_{(0)}| }/\|h_{(0)}\|_2 E |X_0|,
\]
as the ratio between the linear combination $E{|h_{(0)}^T X_{(0)}| }$ for $\|h_{(0)}\|_2=1$ and $ E |X_0|$.

Then we need
\[
          \frac{1-p}{p} < \frac{ \gamma(F,p)  \|h_{(0)}\|_2}{| h_{0}|}
\]
for all $h$ such that $   u^T h = 0$ .

Also, using the fact that
 
\[
         \frac{  \|h_{(0)}\|_2} {| h_{0}|} \geq \cot(\theta).
\]
we have 
\[
          \frac{1-p}{p} <\gamma(F,p) \cot(\theta).
\]

\section{Noisy  signal}
 \subsection{Folder Gaussian}
 For a Gaussian RV $Z \sim N(\mu, \sigma)$, then for RV $|Z|$,  
 \[
 \mu_{|Z|} = E|Z|=\sqrt{\frac{2}{\pi}}\sigma e^{-\frac{\mu^2}{2\sigma^2}}+\mu\left[1-2\Phi\left(-\frac{\mu}{\sigma}\right) \right] 
 \]
  \[
 \sigma_{|Z|}^2 = \mu^2 + \sigma^2 - \mu_{|Z|}^2.,
 \]
 
 \subsection{Noisy perturbation analysis}
Consider a noisy perturbation $\bar X_i  =  X_i + \sigma  E_i $, where $E_i$ is standard Gaussian $N(0,1)$, $ X_i = B_i Z_i$ are iid product of Bernolli RV with parameter $p$ and subGaussian unit variance RV with c.d.f $F$. 

Then we let $b^{\sigma}$ be the solution of the optimization problem 
\[
E D_\sigma = E_X E_E( |(1+ {\varepsilon} h_0)(X_0+\sigma  E_0) +  {\varepsilon} h_{(0)}^T (X_{(0)}+\sigma  E_{(0)})| -|X_0+\sigma  E_0|)
\]
for sufficiently small ${\varepsilon}$, and fixed $\sigma<1$, we consider the asymptotic analysis when ${\varepsilon}  << 1 $， $ \sigma<< 1 $. We know $\sigma ((1+ {\varepsilon} h_0)(  E_0)+{\varepsilon} h_{(0)}^T  E_{(0)}))$ is a centered Gaussian with standard deviation 
\[\sigma\sqrt{(1+ {\varepsilon} h_0)^2+({\varepsilon} h_{(0)})^2}\approx \sigma(1+ {\varepsilon} h_0)
\]
therefore,
\begin{eqnarray*}
 E D_\sigma &\approx&   E|(1+ {\varepsilon} h_0)X_0 +  {\varepsilon} h_{(0)}^T X_{(0)}+\sigma (1+ {\varepsilon} h_0) E_0| -E|X_0+\sigma  E_0| \\
 &\approx& 
  p \cdot(|1+{\varepsilon} \cdot  h_{0}|-1) \cdot E |Z_0+\sigma  E_0| + (1-p) \cdot ( E{|{\varepsilon}  \cdot h_{(0)}^T X_{(0)}+\sigma  E_0| -E|\sigma  E_0|} ),
\end{eqnarray*}
 where $p = P(X_0\neq 0)$.
 
 First we derive lower bounds,
 \[
 E |Z_0+\sigma  E_0|
 \geq E |Z_0| - E|\sigma  E_0| =   E |Z_0| - \sqrt{\frac{2}{\pi}}\sigma,
\]
 \begin{eqnarray*}
 E{|{\varepsilon}  \cdot h_{(0)}^T X_{(0)}+\sigma  E_0| -E|\sigma  E_0|}&\geq& E{|{\varepsilon}  \cdot h_{(0)}^T X_{(0)} | -2 E|\sigma  E_0|} \\
 &=&  E|{\varepsilon}  \cdot h_{(0)}^T X_{(0)} | -2 \sqrt{\frac{2}{\pi}}\sigma
\end{eqnarray*}
Then we have 
\begin{eqnarray*}
 E D_\sigma 
\geq  E D_0 - (p+2(1-p)) \cdot \sqrt{\frac{2}{\pi}}\sigma.
\end{eqnarray*}

Therefore, in the existence of noise, we don't have exact recovery result, but we would have a lower bound. 
 
 Second, we use the mean of folded Gaussian to compute the condition expectation.
 
 Here, for a fixed $\mu$, we rescale the variance $\sigma' = \sigma/{\varepsilon}$. Under the assumption ${\varepsilon}/\sigma<< 1 $, using asymptotic expansion, we can compute the condition expectation 
 \begin{eqnarray*}
   E({|\mu+\sigma'  E_0| -|\sigma'  E_0|}\mid \mu)
 &=& \sqrt{\frac{2}{\pi}}\sigma' e^{-\frac{\mu^2}{2\sigma'^2}}+\mu\left[1-2\Phi\left(-\frac{\mu}{\sigma'}\right) \right] - \sqrt{\frac{2}{\pi}}\sigma'
 \\
 &\approx& -\sqrt{\frac{2}{\pi}}\sigma' {\frac{\mu^2}{2\sigma'^2}}+\mu\sqrt{\frac{2}{\pi}}(-\frac{\mu}{\sigma'}  )
 \\
  &=&- \sqrt{\frac{2}{\pi}}  {\frac{3\mu^2}{2\sigma'}}  
\end{eqnarray*}
The first equality is the formula for the mean of folded Gaussian.
The second equality uses the Taylor expansion of Gaussian c.d.f
\[
\Phi (z)  = \frac{1}{2}+  \frac{1}{\sqrt{\pi}} \left(z-\frac{z^3}{3}+\frac{z^5}{10}-\frac{z^7}{42}+\frac{z^9}{216}-\cdots\right)
\]

 Therefore, for $Z_0$ standard Gaussian,
  \begin{eqnarray*}
 E_X E_{E_0}(|Z_0+\sigma  E_0|) 
  &=&  \sqrt{\frac{2}{\pi}}   \sqrt{1+\sigma^2}  
\end{eqnarray*}
 \begin{eqnarray*}
 E_X E_{E_0}({|{\varepsilon}  \cdot h_{(0)}^T X_{(0)} +\sigma  E_0| -|\sigma  E_0|}) 
  &=&-{\varepsilon} \cdot 3\sqrt{\frac{2}{\pi}}  {\frac{E_X( h_{(0)}^T X_{(0)})^2 }{2\sigma'}}  
   \\
  &=&-{\varepsilon} \cdot 3\sqrt{\frac{2}{\pi}}  {\frac{\|h_{(0)}\|^2 }{2\sigma'}}  
\end{eqnarray*}

Then 
\begin{eqnarray*}
 E D_\sigma 
 &\approx& 
 {\varepsilon} \left[ p \cdot h_{0} \cdot \sqrt{\frac{2}{\pi}}   \sqrt{1+\sigma^2}    - (1-p) \cdot 3\sqrt{\frac{2}{\pi}}  {\frac{\|h_{(0)}\|^2 }{2(\sigma/{\varepsilon})}} \right] .
\end{eqnarray*}



 
\section{Population problem}
\subsection{Population problem setting}
\paragraph{Doubly infinite sequence setting}
First, we study the population objective under doubly infinite sequence setting:
\begin{itemize}
\item
Let $(X_t)_{t \in {\mathbb Z}}$ be IID sampled from  $pN(0,1) + (1-p) \delta_0$.
 \item
The filter $a\in l_1({\mathbb Z}), (\mathcal{F} a)(\omega) \neq 0, \forall \omega\in T.$
\item
Let  $(Y_t)$ defined by $Y_t \equiv \sum_{h\in {\mathbb Z}} a_h X_{t-h}$ be a linear process, i.e.
\[
Y= a* X,
\]
\end{itemize}

\paragraph{Population problem objective}
Due to shift invariance, the population loss is
 \[
 {\mathbb E} |[w*Y]_0| = {\mathbb E} |\tilde{w}^T Y| 
 \]
 where $\tilde{w}_i = w_{-i}.$

From the ergodic property of stationary process,
 \[
{\mathbb E}\frac{1}{N} \| w*Y\|_{\ell_1({\cal T})} = {\mathbb E} |[w*Y]_0|
  \]
 Due to shift invariance, we only need to study 
 \[
 {\mathbb E} |[w*Y]_0| = {\mathbb E} |\tilde{w}^T Y| 
 \]
 where $\tilde{w}_i = w_{-i}.$
\paragraph{Population problem formulation}
Now we state the theorem for a general doubly infinite sequence $v$ with unit $2$ norm, we assume that $v$ and $a$ are not orthogonal so that the angle $\theta$ between $v$ and $a$ satisfies $\cos{\theta} = a^T v /\|a\|_2>0$, $\sin{\theta}\geq 0$. Then we define the optimization problem $P_{1,v}$ over  $w \in l_1({\mathbb Z})$  as
\begin{equation}
 \begin{array}{ll}
  \underset{w\in V}{ \mbox{minimize} }  &  {\mathbb E} |\tilde{w}^T Y|\\
 \mbox{subject to}  
&  v^T w =1.
\end{array}
\tag{$P_{1,v}$}
\end{equation}
\paragraph{Change of variable}
We define global change of variable over the paper:
\[
\psi := \tilde{w}*\tilde{a}
\]
then
\[
 \tilde{\psi} :=w*a
\]
\[
 \tilde{\psi}*Y :=w*a*X
\]
also let $u$ be a unit vector along the direction of $C_{\tilde{a}}^{-1} \tilde{v}$
\[
u:= C_{\tilde{a}}^{-1} \tilde{v}/\|C_{\tilde{a}}^{-1} \tilde{v}\|_2 = \tilde{a}^{-1}* \tilde{v} /\|\tilde{a}^{-1}* \tilde{v}\|_2.
\]
then $ \theta =\angle({v, a}) = \angle({u, e_0})$.

Without loss of generality, we could rescale $v$ so that
\[
\|\tilde{a}^{-1}* \tilde{v}\|_2 = \| a* v\|_2 =1.
\]



\paragraph{Projection pursuit with linear constraint}
The population problem becomes the \textbf{convex projection pursuit problem with linear constraint}
\begin{equation}
 \begin{array}{ll}
  \underset{\psi \in V}{ \mbox{minimize} }  &  {\mathbb E} |\psi^T X|\\
 \mbox{subject to}  
&  u^T \psi =1.
\end{array}
\tag{$P_{1,v}$}
\end{equation}
\paragraph{Other objectives}

After the change of variable, $\tilde{\psi} :=w*a$, we have
\begin{eqnarray*}
    (\ell_1/\ell_2) \qquad  \min&    \frac{1}{N} \| \tilde{\psi}*X\|_{\ell_1({\cal T})}\\
    \mbox{ subject to }&   \frac{1}{N} \| \tilde{\psi}*X\|_{\ell_2({\cal T})} =1 .
\end{eqnarray*}

In population, since ${\mathbb E} | \psi^T X|^2 = p\|\psi\|_{\ell_2({\cal T})}$, we have
\begin{eqnarray*}
    (\ell_1/\ell_2) \qquad  \min&    {\mathbb E} | \psi^T X|\\
    \mbox{ subject to }&   {\mathbb E} | \psi^T X|^2 =p\|\psi\|_{\ell_2({\cal T})} =1 .
\end{eqnarray*}





\paragraph{Projection pursuit with sphere constraint}
We comment that the results we derive in the following sections are also applicable to the projection pursuit (orthogonal dictionary learning) problem.
\begin{lemma}
The problem
\[
 \begin{array}{ll}
 \mbox{minimize}_{\psi} \quad & {\mathbb E} |\psi^T X|  \\
\mbox{subject to}  
&{\mathbb E} |\psi^T X|^2 \geq {\mathbb E} (X_0)^2
\end{array}
  \]
is equivalent to the following convex-concave problem
\[
 \begin{array}{ll}
 \mbox{minimize}_{\psi} \quad &\sqrt{\frac{2}{\pi}}\left[{\mathbb E}_I \|\psi_I\|_2 \right] \\
\mbox{subject to}  
&\|\psi\|_2^2 \geq 1
\end{array}
  \]
We claim that  it has the same minimizer as the \textbf{ projection pursuit (orthogonal dictionary learning) problem with sphere constraint}.
  \[
 \begin{array}{ll}
 \mbox{minimize}_{\psi} \quad & {\mathbb E}_I \|\psi_I\|_2  \\
\mbox{subject to}  
&\|\psi\|_2 = 1
\end{array}
  \]
  which is also equivalent to
   \[
 \begin{array}{ll}
 \mbox{minimize}_{\psi\neq 0} \quad & {\mathbb E}_I \|\psi_I\|_2/\|\psi\|_2 
\end{array}
  \]
  And  also equivalent to
    \[
 \begin{array}{ll}
 \mbox{minimize}_{\psi\neq 0} \quad & {\mathbb E} |\psi^T X|/\sqrt{{\mathbb E} |\psi^T X|^2} 
\end{array}
  \]
\end{lemma}


\subsection{Inverse filter}
\paragraph{Fourier transform and inverse filter}
The discrete-time Fourier transform $\mathcal{F}$ is defined by $\mathcal{F} a = \hat{a}$ where
\[
\hat{a}(\omega) : = \sum_{n=-\infty}^\infty a_n\, e^{2\pi i n \omega}, \quad  \omega\in T = [-\frac{1}{2}, \frac{1}{2}];
\]
the inverse Fourier transform $\mathcal{F}^{-1}$ is defined by
\[
(\mathcal{F}^{-1} f)(n) =  \int_{-\frac{1}{2}}^\frac{1}{2} f(\omega) e^{-2\pi i n \omega}  d\omega, \quad n\in {\mathbb Z}.
\]
Now, our condition on the filter $a$ is:
\[a\in l_1({\mathbb Z}) ,  
(\mathcal{F} a)(\omega) \neq 0, \forall \omega\in T.
\]
In the following theorem, we show that this condition would provide the existence of an inverse filter in  $l_1({\mathbb Z})$.
\begin{theorem}
If $a\in l_1({\mathbb Z}) ,  
(\mathcal{F} a)(\omega) \neq 0, \forall \omega\in T,$ we could define the inverse filter of $a$ as
$$a^{-1} := \mathcal{F}^{-1}(\frac{1}{\mathcal{F} a})$$
so that $a^{-1}*a = e_0.$ Here $e_0$ is the sequence with $1$ at $0$ coordinate and $0$ elsewhere.
From Wiener's lemma on $\mathcal{F} a$,   $a^{-1} \in l_1({\mathbb Z})$ .
\end{theorem}

\begin{lemma}{\textbf{Wiener's lemma:}}
Assume that a function $f$ on unit circle has an absolutely converging Fourier series, and $f(t)\neq 0$ for all $t \in T$, then $1/f$ also has an absolutely convergent Fourier series.
\end{lemma}




\section{Population loss and phase transition}
We first study population loss of the sparse deconvolution problem
 \begin{eqnarray*}
    (\mbox{Population sparse deconvolution}) \qquad  \min_{w\in {\mathbb R}^k} &    
    E\| w * Y\|_1\\
    \mbox{ subject to }&  v^T w = 1,\\
    &  w\in {\cal M}.
\end{eqnarray*}
\subsection{Population loss of projection pursuit}
Let's first consider the population objective $E|w^T \Phi X|$.

Consider a vector  $w \in {\cal M}$ which is a small perturbation of $w^\star  \in {\cal M}$, where
$w^T \Phi =e_0$. Let $w = w^\star + {\varepsilon} \delta$ where $\delta \in T_{w^\star} {\cal M}$, the tangent space at $w^\star$. And since $\delta \in T_{w^\star} {\cal M}_1$ is a subspace, we assume the constraint is given by $M \delta = 0$.
Let $h = \Phi^T \delta$, then $   u^T \delta = 0$, where $ v = \Phi u$. We have
\[
    b^T Y = X_0 + {\varepsilon} \delta^T \Phi X=X_0 + {\varepsilon} h^T  X = (1+ {\varepsilon} h_0)X_0 +  + {\varepsilon} h_{(0)}^T X_{(0)} .
\]
 where $h_{(0)}$, $X_{(0)}$ are $h$, $X$ without $0$-th coordinate.
 \subsection{Population phase transition}
 We want to study $E D =  E|(1+ {\varepsilon} h_0)X_0 +  + {\varepsilon} h_{(0)}^T X_{(0)}| -E|X_0|$.
 
 Then for sufficiently small ${\varepsilon}$
\begin{eqnarray*}
 E|(1+ {\varepsilon} h_0)X_0 +  {\varepsilon} h_{(0)}^T X_{(0)}| -E|X_0| &\approx& 
 (|1+{\varepsilon} \cdot  h_{0}|-1) \cdot E |X_0| \cdot (1-\eta)+ {\varepsilon} \cdot  E{|h_{(0)}^T X_{(0)}| } \cdot \eta,
\end{eqnarray*}
 where $p = P(X_0=0)$.
 
 The perturbation increases the objective value 
just in case either $h_{0} > 0$, or in case $h_{0} < 0$ and
\[
          \frac{\eta}{1-\eta} > \frac{| h_{0}| E |X_0|}{ E | h_{(0)}^T X_{(0)} |. }
\]

Now we consider the optimization problem
\begin{eqnarray*}
    (Q_1) \qquad  \min_h&   E | h_{(0)}^T X_{(0)} |\\
    \mbox{ subject to }&  h_{0} = -1, \\
    & h\in \Phi^T T_\beta {\cal M}_1.
\end{eqnarray*}


We know $h = \Phi^T \delta$, then 
\begin{eqnarray*}
    (Q_1) \qquad  \min_\delta &E | \delta^T \Phi_{(0)} X_{(0)} | \\
    \mbox{ subject to }  &\Phi_0^T \delta = -1, \\
     & \delta \in T_\beta {\cal M}_1.
\end{eqnarray*}



Following the quadratic problem reduction, we can related $Q_1$ to the deterministic quadratic problem.

After the study of the general population model, we remark that for convolutional model, $A$ ad $B$ are both generated by the circular or Topelitz matrices from a single vector.




\subsection{Regularized convex problems}
\paragraph{Regularized convex problems}
For any $\epsilon>0$, we can find the Lagrangian form  of the constrained problem to define the regularized convex program, where we can find a corresponding $\lambda = \Lambda(\epsilon)$ so that $\frac{1}{2\lambda}$ is the dual variable.
\[
 \begin{array}{ll}
 \mbox{minimize}_{w,\zeta}   & |\tilde{w}^T Y -\zeta |+ \frac{1}{2\lambda}((\zeta)^2-\epsilon^2) \\
\mbox{subject to}  
& v^T w =1,
\end{array}
\tag{$P_{1,\lambda}$}
  \]
The convention to use for optimization is to ignorne the constant $-\epsilon^2$, and write the problem as
\[
 \begin{array}{ll}
 \mbox{minimize}_{w,\alpha}   & [|\alpha|+ \frac{1}{2\lambda}(\alpha -\tilde{w}^T Y)^2] \\
\mbox{subject to}  
& v^T w =1,
\end{array}
\tag{$P_{1,\lambda}$}
  \]
  Define soft-threshold operator as 
  \[
  S_{\mu}(x): = {\rm sign} (x) (|x| -\mu)_+
  \]
  and define its complement
   \[
  \bar{S}_{\mu}(x): = x-S_{\mu}(x) =  {\rm sign} (x)(x- (|x| -\mu)_+) =  {\rm sign} (x)\max(|x| ,\mu).
  \]
 \begin{lemma}
 If we partially minimize over $\alpha,$ the minimizer is $\alpha = S_{\lambda}(\tilde{w}^T Y)$, 
    \[
 \begin{array}{ll}
\mbox{inf}_{\alpha}   & [|\alpha|+ \frac{1}{2\lambda}(\alpha -\tilde{w}^T Y)^2]   = [(|\tilde{w}^T Y|-\lambda)_+ + \frac{1}{2\lambda}(\max(|\tilde{w}^T Y| ,\lambda) )^2] 

\end{array}
  \]
  \end{lemma} 
  
  
 
\begin{proof}
Due to the proximal operator of $\ell_1$ norm,  the minimizer is $\alpha = S_{\lambda}(\tilde{w}^T Y)$, 
The objective can be further simplified to
  \[
 \begin{array}{ll}
&{\mathbb E}[|\alpha|+ \frac{1}{2\lambda}(\alpha -\tilde{w}^T Y)^2]\\
=&{\mathbb E}[|S_{\lambda}(\tilde{w}^T Y)|+ \frac{1}{2\lambda}(\bar{S}_{\lambda}(\tilde{w}^T Y) )^2]\\
=& [(|\tilde{w}^T Y|-\lambda)_+ + \frac{1}{2\lambda}(\max(|\tilde{w}^T Y| ,\lambda) )^2] 
\end{array}
  \]
 \end{proof} 

Using the data generative model and do change of variable, 
the partially minimized regularized problem is
 \[
 \begin{array}{ll}
 \mbox{minimize}_{\psi}   & {\mathbb E}[|(|\psi^T (X+\eta Z )|-\lambda)_+|+ \frac{1}{2\lambda}(\max(|\psi^T (X+\eta Z )| ,\lambda) )^2] \\
\mbox{subject to}  
& u^T \psi =1,
\end{array}
\tag{$P_{1,\lambda}$}
  \]

\section{Related work}

Blind source seperation, cocktail party problem

ICA, sparse ICA

Projection pursuit (exploratory)

AR, non-causal AR (non-Gaussian)

dictionary learning


\paragraph{Prior work: analysis approach, non-Gaussianity, Ratio formulation}
  \begin{itemize}
  \item
  Key idea $1$: Find inverse filter $a^{-1}$ so that
$ a^{-1}*y = x $.
\item Key idea $2$: 
If $x$ is non-Gaussian, then $a*x$ is closer to Gaussian. 
\item Find $w\neq 0$  
\[
 \begin{array}{ll}
 \underset{w}{\mbox{maximize}}   & J(w*y)
\end{array}
  \]
\item Non-Gaussianity: $J$ is negative entropy function or normalized $4$-th order cumulant $J(z) = \frac{\|z\|_4^4}{\|z\|_2^4}$ \cite{wiggins1978minimum, donoho1981minimum}
  \item Sparsity: 
  $J(z) = - \| z\|_1/\|z\|_2$,  \cite{gray1979variable,donoho1981minimum,kijko1994seismological}.
  \item varimax norm: $\max_i \frac{|z_i|}{\|z\|_2}$ \cite{cabrelli1985minimum}
\end{itemize}









\paragraph{Prior work: blind equalization for digital signal}
  
  
There are a series of impactful digital communication work on blind deconvolution when the digital signal is discrete, which is called Blind Equalization. Examples of digital signal are PAM signaling, which is $\pm 1$ binary signal, and QAM signaling, where signal are equaly spaced points on unit sphere in complex space, as $|x|=1, x\in {\mathbb C}$ \cite{kennedy1992blind,ding2000fast}.  In the work \cite{vembu1994convex}, they propose an ideal convex cost function for blind equalization:
  find $w$ so that $w*a = e_0 = (\ldots,0,1,0, \ldots)$ as minimizer of the ideal convex problem (
 Not computable without knowing $a$!)
  \[
 \begin{array}{ll}
 \underset{w}{\mbox{minimize}}   & \| w*a\|_1 
 \\
\mbox{subject to} 
& (w*a)_0 = 1.
\end{array}
  \]  
   Using the condition that $x$ are IID and bounded by $M = \mbox{max}(x_i)$, using duality, they use surrogate objective for $\| w*a\|_1$ as
   \[
 \begin{array}{ll}
 \underset{w}{\mbox{minimize}}   &  \frac{1}{M} E \|w * y\|_\infty 
 \\
\mbox{subject to} 
& \|w*y\|_2 = 1.
\end{array}
  \]
   In \cite{vembu1994convex}, they use projected SGD on $E \|w * y\|_8 $ to approximate $E \|w * y\|_\infty$.
   Later, use linear programming to solve $\infty$ norm problem directly \cite{ding2000fast}: 
    \[
 \begin{array}{ll}
 \underset{w}{\mbox{minimize}}   &  E \|w * y\|_\infty
 \\
\mbox{subject to} 
& w_0 = 1.
\end{array}
  \]
  


\paragraph{Prior work: synthesis approach, sparsity}
\begin{itemize}
\item
  Convolutional dictionary learning formulation \cite{bristow2013fast, heide2015fast}, etc.
  \item
  Synthesis blind deconvolution with $2$ norm ball constraint  \cite{zhang2017global,zhang2018structured, kuo2019geometry}, etc.
  \item
  Short $a$ and sparse $x$: 
  \[
 \begin{array}{ll}
 \mbox{minimize}   & \|y-a*x\|_2^2+\lambda \|x\|_1 \\
\mbox{subject to}  & \|a\|_2 = 1.
\end{array}
  \]  
\end{itemize}

\paragraph{Prior work: synthesis approach, subspace assumption}
\[
 \begin{array}{ll}
 y &= a^0 * x^0\\
 \hat y  &= \hat a^0 \circ \hat x^0
\end{array}
  \]
   \begin{itemize}
  \item  $\hat a^0= D_a c_a$.
  \item $\hat x^0 = D_X c_x$.
  \item $D_X$ is often a complex Gaussian matrix.  
\end{itemize}
No translation symmetry in this assumption!


Lifting:
 \[
 \begin{array}{ll}
 \hat y &= (D_a c_a) \circ (D_X c_x)\\
 &= \mathcal{D}(c_a c_x^T)
\end{array}
  \]
 $\mathcal{D}$ is a linear operator defined as 
 \[
 \begin{array}{ll}
\mathcal{D}(c_a c_x^T)_i = \mbox{trace}((D_{a,i}D_{X,i}^T)(c_a c_x^T))
\end{array}
  \]
  Rank one matrix $Z=c_a c_x^T$.
   \[
      \hat y =  \mathcal{D}(Z)
  \]
  \begin{itemize}  
  \item Convex optimization  \cite{ahmed2014blind},\cite{ling2015self}, \cite{chi2016guaranteed},\cite{romberg2013multichannel},\cite{lee2017blind} etc.
  \item Non-convex optimization  \cite{li2016rapid}.
\end{itemize}

\paragraph{Projection pursuit  with linear constraint}
We point out that the population problem formulation for sparse blind deconvolution is also the population problem formulation for sparse projection pursuit, which might also be called convex sparse ICA, or convex sparse dictionary learning.
\begin{equation}
 \begin{array}{ll}
  \underset{\psi \in V}{ \mbox{minimize} }  &  {\mathbb E} |\psi^T X|\\
 \mbox{subject to}  
&  u^T \psi =1.
\end{array}
\tag{$P_{1,v}$}
\end{equation}


In sparse projection pursuit with linear constraint, we observe a matrix $Y_M\in {\mathbb R}^{n\times M }$, and we know that there exists a ground-truth filter $w^0$, such that $w^{0,T} Y_M = X_M$, $X_M$ satisfy Bernoulli$(p)$-Gaussian model.  

We consider the "projection pursuit" problem.
 \begin{eqnarray*}
    (\mbox{Projection pursuit}) \qquad  \min&    \frac{1}{M} \| \tilde{w}^T Y_M\|_1\\
    \mbox{ subject to }&  v^T w = 1.
\end{eqnarray*}
Here $w$ is in a $k$ dimensional subspace of the double infinite sequence. 


\paragraph{Projection pursuit with sphere constraint}
We comment that the results we derive in the following sections are also applicable to the projection pursuit (orthogonal dictionary learning) problem \cite{bai2018subgradient, sun2016complete, sun2015complete}.
 \begin{eqnarray*}
    (\mbox{Projection pursuit with sphere constraint}) \qquad  \min&    \frac{1}{M} \| \tilde{w}^T Y_M\|_1\\
    \mbox{ subject to }&   \frac{1}{M} \| \tilde{w}^T Y_M\|_2 = 1.
\end{eqnarray*}
or
 \begin{eqnarray*}
    (\mbox{Projection pursuit with sphere constraint}) \qquad  \min&    \frac{1}{M} \| \tilde{w}^T Y_M\|_1\\
    \mbox{ subject to }&   \frac{1}{M} \| \tilde{w}^T Y_M\|_2 \geq 1.
\end{eqnarray*}
\begin{lemma}
The population problem is:
\[
 \begin{array}{ll}
 \mbox{minimize}_{\psi} \quad & {\mathbb E} |\psi^T X|  \\
\mbox{subject to}  
&{\mathbb E} |\psi^T X|^2 \geq {\mathbb E} (X_0)^2
\end{array}
  \]
is equivalent to the following convex-concave problem
\[
 \begin{array}{ll}
 \mbox{minimize}_{\psi} \quad &\sqrt{\frac{2}{\pi}}\left[{\mathbb E}_I \|\psi_I\|_2 \right] \\
\mbox{subject to}  
&\|\psi\|_2^2 \geq 1
\end{array}
  \]
We claim that  it has the same minimizer as the \textbf{ projection pursuit (orthogonal dictionary learning) problem with sphere constraint}.
  \[
 \begin{array}{ll}
 \mbox{minimize}_{\psi} \quad & {\mathbb E}_I \|\psi_I\|_2  \\
\mbox{subject to}  
&\|\psi\|_2 = 1
\end{array}
  \]
  which is also equivalent to
   \[
 \begin{array}{ll}
 \mbox{minimize}_{\psi\neq 0} \quad & {\mathbb E}_I \|\psi_I\|_2/\|\psi\|_2 
\end{array}
  \]
  And  also equivalent to
    \[
 \begin{array}{ll}
 \mbox{minimize}_{\psi\neq 0} \quad & {\mathbb E} |\psi^T X|/\sqrt{{\mathbb E} |\psi^T X|^2} 
\end{array}
  \]
\end{lemma}


Inspired by this formulation, we propose a non-convex (convex objective concave constraint) problem for blind deconvultion:
\begin{eqnarray*}
    (\mbox{Projection pursuit with sphere constraint}) \qquad  \min&    \frac{1}{M} \| w*Y_M\|_1\\
    \mbox{ subject to }&   \frac{1}{M} \| w*Y_M\|_2 \geq 1.
\end{eqnarray*}

Another choice is to use infinity norm ball constraint, which allows us to translate it to a convex problem.
\begin{eqnarray*}
    (\mbox{Projection pursuit with sphere constraint}) \qquad  \min&    \frac{1}{M} \| w*Y_M\|_1\\
    \mbox{ subject to }&   \frac{1}{M} \| w*Y_M\|_\infty \geq 1.
\end{eqnarray*}



\paragraph{Dirac re-parametrization and connection to residual CNN}
When $v = e_0$, we are imposing a constraint that the largest entry of $w$ is $w_0 =1$.  
We can re-parametrize our filter, as $w = \delta_0 + f$, $f_0 = 0$. Then our problem became 
 \begin{eqnarray*}
    \min_{f, f_0 = 0} &    
    \frac{1}{2T+1}\|(e_0 +f)*Y\|_{1,[-T,T]}.
\end{eqnarray*}
Here $f = (f_L, 0, f_R)$, $f_R$ and $f_L$ are in $k$ dimensional subspace of the double infinite sequence with positive and negative support.

Allowing Noise:
\[
 \begin{array}{ll}
 \mbox{minimize}   & \|x\|_ 1+\lambda/2\|Y+f*Y-x\|_2^2\\
\mbox{subject to}  
& f_0 =0,
\end{array}
  \]
  with variables $f\in {\mathbb R}^k$, $x\in {\mathbb R}^n$. 
  
 \texbf{General multi-channel approximate version}
\[
 \begin{array}{ll}
 \mbox{minimize}   & \sum_i(\|x^i\|_ 1+\lambda^i/2\|Y+f^i*Y-x^i\|_2^2)\\
\mbox{subject to}  
& f^i_0 =0,
\end{array}
  \]
  with variables $f^i\in {\mathbb R}^k$, $x^i\in {\mathbb R}^n$. 
  
\paragraph{Link to CNN}
Partial minimization:(similar to one-layer Residual CNN)
\[
 \begin{array}{ll}
 x^i = S_{1/\lambda^i}(Y+f^i*Y)
\end{array}
  \]
  with $S_{b}(z) :=\mbox{sign}(z)(|z|-b)_+ $. 


\texbf{Method $1$: plug-in}
\[
 \begin{array}{ll}
 \mbox{minimize}   & \sum_i(\|S_{1/\lambda^i}(Y+f^i*Y)\|_ 1+\\
 &\lambda^i/2\|Y+f^i*Y-S_{1/\lambda^i}(Y+f^i*Y)\|_2^2)\\
\mbox{subject to}  
& f^i_0 =0,
\end{array}
  \]
  with variables $f^i\in {\mathbb R}^k$, $x^i\in {\mathbb R}^n$.   
  

\section{Stability for approximation of infinite length inverse filter}

 
 
 

\paragraph{Reduction}
 \begin{lemma} 
 Let $w^\star \in V_{rs, rt} $ be the solution of the convex optimization problem
 \[
 \begin{array}{ll}
  \underset{w}{ \mbox{minimize} }  &  {\mathbb E}|(w*Y)_0|\\
 \mbox{subject to}  
&  w_0 =1, \quad w \in V_{rs, rt}.
\end{array}
  \]
 Then the previous convex optimization problem can be reduced to 
  \[
 \begin{array}{ll}
  \underset{w}{ \mbox{minimize} }  & B(p, \phi) \|\phi \|_2\\
 \mbox{subject to}  
&  \phi = a*w -e_0, \quad w_0 =1, \quad w \in V_{rs, rt}.
\end{array}
  \]
  where
  \[
B(p, \phi) := \frac{{\mathbb E}_I  \| (e_0+\phi)_I \|_2 - {\mathbb E}_I  \| (e_0)_I \|_2}{\|\phi\|_2} 
\]
 \end{lemma} 
  
  
\begin{proof}
Following the previous phase transition analysis, 
let the directional finite difference of the objective at $e_0$ be $D(w) = |(w*Y)_0| - |X_0|$, $I$ be the support of $\tilde{X}$, let $\phi = w*a-e_0$,  then we have 
\[
 \begin{array}{ll}
{\mathbb E} D(w) &= \sqrt{\frac{2}{\pi}}{\mathbb E}_I [\|(w*a)_I\|_2 - \|(e_0)_I\|_2]\\
&= \sqrt{\frac{2}{\pi}}{\mathbb E}_I [\|(e_0+\phi)_I\|_2 - \|(e_0)_I\|_2]\\
&= \sqrt{\frac{2}{\pi}}\|{\phi}\|_2 B(p, \phi)  \\
\end{array}
  \]
The last inequality comes from the bi-Lipschitz property of ${\mathbb E}_I \|(e_0+\phi)_I\|_2$ around $e_0$. 








Now using the optimality of $w^\star$ in $V_{rs, rt}$, for all $w in V_{s,t}, w\neq w^\star$,
we have
 \[
 \begin{array}{ll}
{\mathbb E} D(w^\star) \leq {\mathbb E} D(w)
\end{array}
  \]
We could directly reduce the problem to 
  \[
 \begin{array}{ll}
B(p, \phi^\star) \|{\phi^\star}\|_2 \leq B(p, \phi)\|{\phi}\|_2 
\end{array}
  \]
\end{proof}   
 


\begin{coro}
If there exists a $\phi$ with $\|{\phi}\|_2<1$ such that \[
 \begin{array}{ll}
B(p, \phi^\star) \|{\phi^\star}\|_2 \leq B(p, \phi)\|{\phi}\|_2
\end{array}
  \]
 for  $\epsilon> 0$ when $p\leq p^\star(\epsilon)$, then 
  \[
  B(p, \phi) \geq \epsilon.
  \]
  for all $\phi$, 
  then
    \[
 \begin{array}{ll}
\|{\phi^\star}\|_2 \leq \frac{(1-p)\sqrt{p}  + 
p\Big(1  +
  \frac{\|{\phi}\|_2}{2} 
 +
 p\frac{\|{\phi}\|_2}{2}   \Big)}{\epsilon}\|{\phi}\|_2
\end{array}
  \]
  is also bounded.
\end{coro}
\begin{proof}
From
 \[
 \begin{array}{ll}
B(p, \phi^\star) \|{\phi^\star}\|_2 \leq B(p, \phi)\|{\phi}\|_2
\end{array}
  \]
  We have
  \[
 \begin{array}{ll}
\|{\phi^\star}\|_2 
&\leq \frac{B(p, \phi)}{B(p, \phi^\star)}
\|{\phi}\|_2 \\
&\leq \frac{(1-p)\Big( V_1(\phi_{{(0)}}) \sqrt{1-\phi_0^2}\Big) + 
p\Big(\phi_0  +
  \frac{\|{\phi}\|_2}{2} \phi_0^2 
 +
 p\frac{\|{\phi}\|_2}{2}  (1-\phi_0^2) \Big)}{(1-p)\Big( V_1(\phi^\star_{{(0)}}) \sqrt{1-(\phi^\star_0)^2}\Big) + 
p\Big(\frac{1}{\|{\phi^\star}\|_2} (\left| 1 + \|{\phi^\star}\|_2 \phi^\star_0 \right|-1 \Big)}\|{\phi}\|_2 \\
&\leq \frac{(1-p)\sqrt{p}  + 
p\Big(1  +
  \frac{\|{\phi}\|_2}{2} 
 +
 p\frac{\|{\phi}\|_2}{2}   \Big)}{\epsilon}\|{\phi}\|_2
\end{array}
  \]
  \end{proof}
  In the following we constructed a filter to give an upper bound on $\|{\phi}\|_2$. 


\paragraph{Setting for finite length approximation}
Now we consider a setting where there is a kernel whose corresponding inverse kernel has infinite support, and we give a finite-length approximation. 

Within the space $V_{-\infty,\infty}$ of bilaterally infinite real-valued sequences $(h(i))_{i \in {\mathbb Z}}$, consider the affine subspace $V_{N_{-}, N_{+}} = \{ (...,0,0,h_{-N_{-}},\ldots, h_{-1}, 1, h_1, h_2, \ldots, h_{N_{+}},0,0,...)\}$, an $N_{-}+N_{+}$-dimensional subspace of bilateral sequences with support at most $N_{-}+N_{+}+1$.
The coordinate that is fixed to one is 
located at index $i=0$.
Each coordinate is zero outside of a window
of size $N_{-}$ on the left of zero 
and  size $N_{+}$ on the right of zero.
The special sequence $e_0 = (\dots,0,0,1,0,0,\dots)$,
vanishing everywhere except the origin,
belongs to $V_{-\infty,\infty}$ and to every
$V_{N_-,N_+}$.

Let $a \in V_{N_-,N_+}$.
Then $a =  (...,0,0,a_{-N_{-}},\ldots, a_{-1}, 1, a_1, a_2, \ldots, a_{N_{+}},0,0,...)$.
We also write $a = e_0 + a_{(0)}$,
where $a_{(0)} \equiv (1-e_0) \cdot a$ 
denotes the `part of $a$ supported away from location
$i=0$'. We also write $a = e_0 + a_L + a_R$,
where $a_L = (a_L(i))_{i \in {\mathbb Z}} 
= (a(i) 1_{\{i < 0\}})_{i \in {\mathbb Z}}$ 
denotes the `part of $a$ supported to
the left of $i=0$' and 
where $a_R = (a_R(i))_{i \in {\mathbb Z}} 
= (a(i) 1_{\{i > 0\}})_{i \in {\mathbb Z}}$
denotes the `part of $a$
supported to the right of $i=0$'.
Finally, we say that $a \in V_{N_{-}, N_{+}}$ 
is a length $ L= N_{-}+N_{+}+1$ filter.

An interesting, if not implementable, objective
function for the blind deconvolution problem
is to minimize
${\mathbb E} [|(w*a*X)_0| - |X_0|] $,
particularly when combined with constraints 
somehow ensuring ${{\mathbb E}} |(w*a*X)_0|^2 \geq {{\mathbb E}}|X_0|^2$.
Defining $\phi \equiv w*a-e_0$, 
\[
 \begin{array}{lll}
{\mathbb E} [|(w*a*X)_0| - |X_0|] &=&{\mathbb E} [|((e_0 +\phi)*X)_0| - |X_0|].
\end{array}
  \]
  Below we establish the relation
  \[
   \begin{array}{lll}
{\mathbb E} [|((e_0 +\phi)*X)_0| - |X_0|] 
&\approx&  C \|{\phi}\|_2 ; \\
\end{array}
\]
This suggests that we should  seek to
minimize ${\mathbb E} |(w*Y)_0| $,
while constraining $w$ in some way that
ensures ${{\mathbb E}} |(w*a*X)_0|^2 \geq {{\mathbb E}}|X_0|^2$.
This  the deterministic objective as 
$$
d_a(w, a^{-1}) :=\|{\phi}\|_2 = \|a*w -e_0 \|_2 =  \|a*(w -a^{-1}) \|_2.
$$

Therefore, we want to construct a finite length approximation of $a^{-1}$ to bound $d_a(w^\star, a^{-1}) = \|a*w -e_0 \|_2$ for the optimal point $w^\star$.








We construct the finite length approximation filter explicitly by truncation of Z-transform.

Let the Z-transform of $a$ be 
\[
 \begin{array}{ll}
A(z) =  \sum_{i=-s}^t a_{i} z^{-i},
\end{array}
  \]
Then Z-transform of the inverse kernel $a^{-1}$ is $1/A(z)$.






 
\subsection{A different approach to prove stability for factorized filter}
\subsection{Examples}
In the following, we study the simple case when there is only one root.
Let $s\in (-1,1)$. For $a=(\ldots, 0, 1, -s, 0, \ldots)$, $a^{-1} = (\ldots, 0, 1, s, s^2, s^3, \ldots)$, we could construct 
\[
w^{r,1}:=  (\ldots, 0, 1, s, s^2, s^3, \ldots, s^r)
\]

If we choose this approximation $w^{r,1},$ then
\[
\|a*w^{r,1}-e_0\|_2^2 = \|*_{r+1} (\bar a^{\gamma})\|_2^2 = s^{2(r+1)} 
\]

In the following, we show that this is indeed the best 

For $\gamma\in (0,1]$,
\[
\bar a^{\gamma} :=e_0-\gamma a= (\ldots, 0, 1-\gamma,  \gamma s, 0, \ldots)
\]
\[
 \begin{array}{ll}
w^{r,\gamma} 
&= a^{-1} - a^{-1}*_{r+1} (\bar a^{\gamma})
\end{array}
\]
\[
\begin{array}{ll}
a*(a^{-1}-w^{r,\gamma}) 
&= *_{r+1} (\bar a^{\gamma})\\
&= *_{r+1} (\ldots, 0, 1-\gamma,  \gamma s, 0, \ldots)
\end{array}
\]

\[
\mathcal{F}[(\ldots, 0, 1-\gamma, \gamma s, 0, \ldots)](t)  = (1-\gamma)+   \gamma s  \cdot e^{2\pi i  t}, \quad  t\in T = [\frac{-1}{2}, \frac{1}{2}];
\]

\[
|\mathcal{F}[(\ldots, 0, 1-\gamma, \gamma s, 0, \ldots)] |^2(t) = (1-\gamma)^2+ (\gamma s )^2+ 2 {(1-\gamma)} {\gamma}s \cos{(2\pi  t)} , \quad t\in T = [\frac{-1}{2}, \frac{1}{2}];
\]

\[
 \begin{array}{ll}
\|*_{r+1} (\bar a^{\gamma})\|_2^2 
&= \| (\hat{\bar a}^{\gamma})^{r+1}\|_2^2\\
&=  \int_{\frac{-1}{2}}^\frac{1}{2}  |(1-\gamma)+   \gamma s  \cdot e^{2\pi i  t}|^{2r+2} dt\\
&= \int_{\frac{-1}{2}}^\frac{1}{2}  [(1-\gamma)^2+ (\gamma s )^2+ 2 {(1-\gamma)} {\gamma}s \cos{(2\pi  t)} ]^{r+1} dt
\end{array} \]
Notice the following lemma, 
\begin{lemma}
For odd number, $\int_{\frac{-1}{2}}^\frac{1}{2}  \cos^{2k-1}{(2\pi  t)}  dt = 0$ for any $k\in {\mathbb Z}_+$; for even number, 
\[
\int_{\frac{-1}{2}}^\frac{1}{2}  \cos^{2k}{(2\pi  t)}  dt =  {2k\choose k} 2^{-2k},
\]
\end{lemma}
\begin{proof}
\[
 \begin{array}{ll}
2^n\cos^n (2\pi  t)=(e^{i \cdot 2\pi  t}+e^{-i \cdot 2\pi  t})^n=\sum_{k=0}^n {n\choose k} (e^{i\cdot 2\pi  t})^k\ (e^{-i\cdot 2\pi  t})^{n-k}=\sum_{k=0}^n {n\choose k} e^{(2k-n)i\cdot 2\pi  t},
\end{array} \]

\[
 \begin{array}{ll}
\int_{\frac{-1}{2}}^\frac{1}{2} e^{(2k-n)i\cdot 2\pi  t} dt= \delta_{2k-n}
\end{array} \]
\[
 \begin{array}{ll}
\int_{\frac{-1}{2}}^\frac{1}{2} 2^n\cos^n (2\pi  t) dt =\sum_{k=0}^n {n\choose k} \delta_{2k-n},
\end{array} \]
therefore, for odd number $n=2k-1$, $\int_{\frac{-1}{2}}^\frac{1}{2}  \cos^{2k-1}{(2\pi  t)}  dt = 0$ for any $k\in {\mathbb Z}_+$; for even number $n=2k$, 
\[
\int_{\frac{-1}{2}}^\frac{1}{2} 2^{2k} \cos^{2k}{(2\pi  t)}  dt =  {2k\choose k}.
\]
\end{proof}

\begin{lemma}
Now
$R_1 :=  (1-\gamma)^2+ (\gamma s )^2,$
$R_2 := (1-\gamma) \gamma s,$
\[
 \begin{array}{ll}
 \| (\hat{\bar a}^{\gamma})^{r+1}\|_2^2
=\sum_{k=0}^{\frac{r+1}{2}} [(1-\gamma)^2+ (\gamma s )^2]^{r+1-2k} [ (1-\gamma) \gamma s]^{2k}{2k\choose k}{r+1\choose 2k} 
\end{array} \]
\end{lemma}
\begin{proof}
Now define
$R_1 :=  (1-\gamma)^2+ (\gamma s )^2,$
$R_2 := {(1-\gamma)} {\gamma}s,$
\[
 \begin{array}{ll}
 \| (\hat{\bar a}^{\gamma})^{r+1}\|_2^2
&= \int_{\frac{-1}{2}}^\frac{1}{2}  [(1-\gamma)^2+ (\gamma s )^2+ 2 {(1-\gamma)} {\gamma}s \cos{(2\pi  t)} ]^{r+1} dt\\
&= \int_{\frac{-1}{2}}^\frac{1}{2}  
[R_1+ R_2 \cdot 2 \cos{(2\pi  t)} ]^{r+1} dt\\
&=\sum_{k=0}^{\frac{r+1}{2}} R_1^{r+1-2k} R_2^{2k}{2k\choose k}{r+1\choose 2k} 
\end{array} \]
\end{proof}

Use the Laplace's method theorem 
We could instead directly optimize  $\gamma$ over the $\ell_\infty$ norm of $|\hat{\bar a}^{\gamma}|^2$,
\begin{lemma}
\[
 \||\hat{\bar a}^{\gamma}|^2\|_\infty = \max_{ t\in [\frac{-1}{2}, \frac{1}{2}]}\{ (1-\gamma)^2 + (\gamma s)^2+ 2 (1-\gamma) \gamma s \cos{(2\pi  t)} \}  = (1-\gamma)^2 + (\gamma s)^2+ 2 |(1-\gamma) \gamma s| = (|1-\gamma|+|\gamma s|)^2 
\]
then

 \[
  \begin{array}{ll}
\gamma_\infty &= \arg\min_{\gamma>0}\||\hat{\bar a}^{\gamma}|^2\|_\infty \\
&= \arg\min_{\gamma>0}(|1-\gamma|+|\gamma s|)^2 \\
&= 1
 \end{array} \]
\[
\min_{\gamma\in (0,1)}\||\hat{\bar a}^{\gamma}|^2\|_\infty
= (|1-\gamma_\infty|+|\gamma_\infty s|)^2 
= |s|^2
\]
\end{lemma}
And
 \[
 \begin{array}{ll}
\|*_{r+1} (\bar a^{\gamma_\infty  })\|_2^2 
&=  s^{2r+2} 
\end{array} \]

 


\paragraph{Finite length approximation constructing using Z-transform truncation on factorization form }
Let the Z-transform of $a$ be 
\[
 \begin{array}{ll}
A(z) =  \sum_{i=-N_{-}}^{N_{+}} a_{i} z^{-i},
\end{array}
  \]
We can factorize the Z-transform as product of factors with roots $s_{-j}$ for $j = 1, \ldots N_{-}$ and  $i = 1, \ldots N_{+}.$
\[
 \begin{array}{ll}
A(z) =  c_0\prod_{j=1}^{N_{-}} (1- s_{-j} z) \prod_{i=1}^{N_{+}} (1- s_{i} z^{-1}),
\end{array}
  \]
  Where $c_0$ is a constant to make sure that the coefficient $a_0=1$.
  
 We assume that $|s_k|<1,$ for any $k = -N_{-}, \ldots , N_+.$  
  
Then Z-transform of the inverse kernel $a^{-1}$ is $1/A(z)$.
\[
 \begin{array}{ll}
1/A(z) &=  \frac{1}{c_0}\prod_{j=1}^{N_{-}} (1- s_{-j} z)^{-1} \prod_{i=1}^{N_{+}} (1- s_{i} z^{-1})^{-1}\\
&=  \frac{1}{c_0}\prod_{j=1}^{N_{-}} (\sum_{\ell_j=0}^{\infty} s^{\ell_j}_{-j} z^{-\ell_j} ) \prod_{i=1}^{N_{+}} (\sum_{\ell_i=0}^{\infty} s^{\ell_i}_{i} z^{\ell_i} )
\end{array}
  \]

From previous single root study, we get a hint to study a single truncation of length $ 1 + \sum_{j=1}^{N_{-}} r_{-j}  +  \sum_{j=1}^{N_{+}}  r_i$,  we construct
\[
 \begin{array}{ll}
W(z) 
&=  \frac{1}{c_0}\prod_{j=1}^{N_{-}} (\sum_{\ell_j=0}^{r_{-j}} s^{\ell_j}_{-j} z^{-\ell_j} ) \prod_{i=1}^{N_{+}} (\sum_{\ell_i=0}^{r_{-j}} s^{\ell_i}_{i} z^{\ell_i} )\\
&=  \frac{1}{c_0}\prod_{j=1}^{N_{-}} (1- (s_{-j} z)^{r_{-j}+1})(1- s_{-j} z)^{-1} \prod_{i=1}^{N_{+}}(1- (s_{i}  z^{-1})^{r_i+1}) (1- s_{i} z^{-1})^{-1}
\end{array}
  \]
Let the corresponding filter being $w$.
  
  
Now let $\psi =  a*w$ with Z-transform $\Psi(z)$, $\phi = a*w-e_0$ with Z-transform $\Phi(z)$,we have
\[
 \begin{array}{ll}
\Psi(z): = A(z)W(z) 
&=   \prod_{j=1}^{N_{-}} (1- (s_{-j} z)^{r_{-j}+1}) \prod_{i=1}^{N_{+}}(1- (s_{i} z^{-1})^{r_i+1}) 
\end{array}
  \]
  therefore,
\[
 \begin{array}{ll}
\Phi(z): = A(z)W(z) -1 
&=  \prod_{j=1}^{N_{-}} (1- (s_{-j} z)^{r_{-j}+1}) \prod_{i=1}^{N_{+}}(1- (s_{i} z^{-1})^{r_i+1}) -1
\end{array}
  \]
\paragraph{Upper Bound on  $\|\phi\|_2^2$ for constructed finite length approximation}
In the following,  we want to give upper bound for
  \[
\|a*w -e_0\|_2^2 =\|\phi\|_2^2 
 \]  
  
Redefine the index $\tau_{-j} := r_{-j}+1; \tau_{i} = r_i+1.$
Let ${\cal I} = \{-N_{-},\ldots, -1, 1, \ldots, N_{+} \}$ as the set of all the possible index.
And we define polar represenation of complex roots:
for any $k \in {\cal I} = \{-N_{-},\ldots, -1, 1, \ldots, N_{+} \}$
\[
s_{k} := |s_{k}| e^{i \varphi_k } = e^{\rho_k + i \varphi_k }
\]
where 
\[
 \begin{array}{ll}
\varphi_k := \operatorname{Im}(\log( s_{k} )) = -i \log(\frac{s_{k}}{|s_{k}|}) 
\end{array}
\]
and since $|s_{k}|<1$ we have 
\[
\rho_k = \operatorname{Re}(\log( s_{k} ))<0
\]

Now we consider all $z = e^{2 \pi i t}$ on the unit circle for $t\in [\frac{-1}{2},\frac{1}{2}]$,
then for any $k \in {\cal I}$
\[
 \begin{array}{ll}
- (s_{k} z^{-\mbox{sign}(k)})^{\tau_{k}} 
&= - |s_{k}|^{\tau_{k}} \exp \big(i [\varphi_k-  2 \pi t \mbox{sign}(k)]\cdot \tau_{k} \big)\\
&=   \exp \Big( \tau_{k} \rho_k+ i \cdot \big( \tau_{k} (\varphi_k- 2 \pi t \mbox{sign}(k) ) + \pi  \big) \Big) 
\end{array}
  \]
 Now we simplify the notation by defining 
 \[
 \upsilon_k(t) :=\tau_{k} (\varphi_k- 2 \pi t \mbox{sign}(k) ) + \pi
 \]
then
 \[
 \begin{array}{ll}
\Phi(z)
&=  \prod_{j=1}^{N_{-}} (1- (s_{-j} z)^{\tau_{-j}}) \prod_{i=1}^{N_{+}}(1- (s_{i} z^{-1})^{\tau_{i}}) -1\\
&=  \prod_{k \in {\cal I}} \Big(1- |s_{k}|^{\tau_{k}} \exp \big(i [\varphi_k-  2 \pi t \mbox{sign}(k)]\cdot \tau_{k} \big)\Big)  -1\\
&=  \prod_{k \in {\cal I}} \bigg( 1+ \exp \Big( \tau_{k} \rho_k+ i \cdot \big( \tau_{k} (\varphi_k- 2 \pi t \mbox{sign}(k) ) + \pi  \big) \Big) \bigg)     -1\\
&=  \prod_{k \in {\cal I}} \bigg( 1+ \exp \Big( \tau_{k} \rho_k+ i \cdot \upsilon_k(t) \Big) \bigg)     -1
\end{array}
  \]
  
\begin{lemma}
 For any $z = \exp{(2 \pi i t )}$ on the unit circle,
 \[
 \begin{array}{ll}
\Phi\big(\exp{(2 \pi i t )}\big)
&=  \prod_{k \in {\cal I}} \bigg( 1+ \exp \Big( \tau_{k} \rho_k+ i \cdot \upsilon_k(t) \Big) \bigg)     -1\\
&= \sum_{n=1}^{|{\cal I}|}\sum_{ k_1, \ldots, k_n  \in {\cal I} } 
\exp \Big( \big(\tau_{k_1} \rho_{k_1}+ \ldots+  \tau_{k_n} \rho_{k_n} \big)
+ i \cdot \big( \upsilon_{k_1}(t) + \ldots+  \upsilon_{k_n}(t)  \big) \Big)  
\end{array}
  \]
 \end{lemma}
 
 \begin{lemma}
Additionally, for any $t \in [\frac{-1}{2},\frac{1}{2}]$,
\[
 \begin{array}{ll}
 |\Phi\big(\exp{(2 \pi i t )}\big)|^2  
= 
\sum_{n=1}^{|{\cal I}|}\sum_{ k_1, \ldots, k_n  \in {\cal I} } 
\exp \Big( 2 \big(\tau_{k_1} \rho_{k_1}+ \ldots+  \tau_{k_n} \rho_{k_n} \big) \Big)   
\end{array}
  \]
is independent of $t$, since the oscillation integral over the imaginary part is 
  \[
 \begin{array}{ll}
\int_{\frac{-1}{2}}^\frac{1}{2}
\exp  \bigg( i \cdot \Big(  \big(\upsilon_{k_1}(t) - \upsilon_{k_1}(t) \big) + \ldots+  \big(\upsilon_{k_n}(t) -\upsilon_{k_n}(-t)\big)  \Big) \bigg)  dt
= 1
\end{array}
  \]
Therefore, using Fourier isometry,
 \[
 \begin{array}{ll}
 \|\phi\|_2^2 &= \int_{\frac{-1}{2}}^\frac{1}{2} |\Phi\big(\exp{(2 \pi i t )}\big)|^2 dt \\
&= \sum_{n=1}^{|{\cal I}|}\sum_{ k_1, \ldots, k_n  \in {\cal I} } 
\exp \Big( 2 \big(\tau_{k_1} \rho_{k_1}+ \ldots+  \tau_{k_n} \rho_{k_n} \big) \Big)   
\end{array}
  \]
  
 \end{lemma}

\begin{theorem}
\label{thm: Root_2_norm}
Since $|s_{k}|<1$, i.e. $\rho_{k}<0$ for all $k\in {\cal I}$.
 \[
 \begin{array}{ll}
 \|\phi\|_2^2 
&= \sum_{n=1}^{|{\cal I}|}\sum_{ k_1, \ldots, k_n  \in {\cal I} } 
\exp \Big( 2 \big(\tau_{k_1} \rho_{k_1}+ \ldots+  \tau_{k_n} \rho_{k_n} \big) \Big) 
\end{array}
  \]
  Let $\gamma^m := \min_{i}\big(\tau_{i} |\rho_{i}| \big) =  \min\{\tau_{-N_-}|\rho_{-N_-}|, \ldots, \tau_{-1}|\rho_{-1}|, \tau_{1}|\rho_{1}|, \ldots, \tau_{N_+}|\rho_{N_+}|\}.$
When $\gamma^m \rightarrow \infty$, we have a finite sum of exponential rate  converging terms: the convergence rate is determined by the exponential term as a function of $\gamma^m$.
   \[
 \begin{array}{ll}
 \|\phi\|_2^2 
=    O(\exp \Big( -2 \min_{i}\big(\tau_{i} |\rho_{i}| \big) \Big))
= O(\exp \Big( -2 \gamma^m  \big) \Big))
\rightarrow 0
\end{array}
  \]

 \end{theorem}

 







\subsection{Examples}
In the following, we study the simple case when there is only one root.
Let $s\in (-1,1)$. For $a=(\ldots, 0, 1, -s, 0, \ldots)$, $a^{-1} = (\ldots, 0, 1, s, s^2, s^3, \ldots)$, we could construct 
\[
w^{r,1}:=  (\ldots, 0, 1, s, s^2, s^3, \ldots, s^r)
\]

For $\gamma\in (0,1)$,
\[
\bar a^{\gamma} :=e_0-\gamma a= (\ldots, 0, 1-\gamma,  \gamma s, 0, \ldots)
\]
\[
 \begin{array}{ll}
w^{r,\gamma} 
&= a^{-1} - a^{-1}*_{r+1} (\bar a^{\gamma})
\end{array}
\]
\[
\begin{array}{ll}
a*(a^{-1}-w^{r,\gamma}) 
&= *_{r+1} (\bar a^{\gamma})\\
&= *_{r+1} (\ldots, 0, 1-\gamma,  \gamma s, 0, \ldots)
\end{array}
\]

\[
\mathcal{F}[(\ldots, 0, 1-\gamma, \gamma s, 0, \ldots)](t)  = (1-\gamma)+   \gamma s  \cdot e^{2\pi i  t}, \quad  t\in T = [\frac{-1}{2}, \frac{1}{2}];
\]

\[
|\mathcal{F}[(\ldots, 0, 1-\gamma, \gamma s, 0, \ldots)] |^2(t) = (1-\gamma)^2+ (\gamma s )^2+ 2 {(1-\gamma)} {\gamma}s \cos{(2\pi  t)} , \quad t\in T = [\frac{-1}{2}, \frac{1}{2}];
\]

\[
 \begin{array}{ll}
\|*_{r+1} (\bar a^{\gamma})\|_2^2 
&= \| (\hat{\bar a}^{\gamma})^{r+1}\|_2^2\\
&=  \int_{\frac{-1}{2}}^\frac{1}{2}  |(1-\gamma)+   \gamma s  \cdot e^{2\pi i  t}|^{2r+2} dt\\
&= \int_{\frac{-1}{2}}^\frac{1}{2}  [(1-\gamma)^2+ (\gamma s )^2+ 2 {(1-\gamma)} {\gamma}s \cos{(2\pi  t)} ]^{r+1} dt
\end{array} \]
Notice the following lemma, 
\begin{lemma}
For odd number, $\int_{\frac{-1}{2}}^\frac{1}{2}  \cos^{2k-1}{(2\pi  t)}  dt = 0$ for any $k\in {\mathbb Z}_+$; for even number, 
\[
\int_{\frac{-1}{2}}^\frac{1}{2}  \cos^{2k}{(2\pi  t)}  dt =  {2k\choose k} 2^{-2k},
\]
\end{lemma}
\begin{proof}
\[
 \begin{array}{ll}
2^n\cos^n (2\pi  t)=(e^{i \cdot 2\pi  t}+e^{-i \cdot 2\pi  t})^n=\sum_{k=0}^n {n\choose k} (e^{i\cdot 2\pi  t})^k\ (e^{-i\cdot 2\pi  t})^{n-k}=\sum_{k=0}^n {n\choose k} e^{(2k-n)i\cdot 2\pi  t},
\end{array} \]

\[
 \begin{array}{ll}
\int_{\frac{-1}{2}}^\frac{1}{2} e^{(2k-n)i\cdot 2\pi  t} dt= \delta_{2k-n}
\end{array} \]
\[
 \begin{array}{ll}
\int_{\frac{-1}{2}}^\frac{1}{2} 2^n\cos^n (2\pi  t) dt =\sum_{k=0}^n {n\choose k} \delta_{2k-n},
\end{array} \]
therefore, for odd number $n=2k-1$, $\int_{\frac{-1}{2}}^\frac{1}{2}  \cos^{2k-1}{(2\pi  t)}  dt = 0$ for any $k\in {\mathbb Z}_+$; for even number $n=2k$, 
\[
\int_{\frac{-1}{2}}^\frac{1}{2} 2^{2k} \cos^{2k}{(2\pi  t)}  dt =  {2k\choose k}.
\]
\end{proof}

\begin{lemma}
Now
$R_1 :=  (1-\gamma)^2+ (\gamma s )^2,$
$R_2 := (1-\gamma) \gamma s,$
\[
 \begin{array}{ll}
 \| (\hat{\bar a}^{\gamma})^{r+1}\|_2^2
=\sum_{k=0}^{\frac{r+1}{2}} [(1-\gamma)^2+ (\gamma s )^2]^{r+1-2k} [ (1-\gamma) \gamma s]^{2k}{2k\choose k}{r+1\choose 2k} 
\end{array} \]
\end{lemma}
\begin{proof}
Now define
$R_1 :=  (1-\gamma)^2+ (\gamma s )^2,$
$R_2 := {(1-\gamma)} {\gamma}s,$
\[
 \begin{array}{ll}
 \| (\hat{\bar a}^{\gamma})^{r+1}\|_2^2
&= \int_{\frac{-1}{2}}^\frac{1}{2}  [(1-\gamma)^2+ (\gamma s )^2+ 2 {(1-\gamma)} {\gamma}s \cos{(2\pi  t)} ]^{r+1} dt\\
&= \int_{\frac{-1}{2}}^\frac{1}{2}  
[R_1+ R_2 \cdot 2 \cos{(2\pi  t)} ]^{r+1} dt\\
&=\sum_{k=0}^{\frac{r+1}{2}} R_1^{r+1-2k} R_2^{2k}{2k\choose k}{r+1\choose 2k} 
\end{array} \]
\end{proof}

Use the Laplace's method theorem \ref{Laplace},
We could instead directly optimize  $\gamma$ over the $\ell_\infty$ norm of $|\hat{\bar a}^{\gamma}|^2$,
\begin{lemma}
\[
 \||\hat{\bar a}^{\gamma}|^2\|_\infty = \max_{ t\in [\frac{-1}{2}, \frac{1}{2}]}\{ (1-\gamma)^2 + (\gamma s)^2+ 2 (1-\gamma) \gamma s \cos{(2\pi  t)} \}  = (1-\gamma)^2 + (\gamma s)^2+ 2 |(1-\gamma) \gamma s| = (|1-\gamma|+|\gamma s|)^2 
\]
then

 \[
  \begin{array}{ll}
\gamma_\infty &= \arg\min_{\gamma>0}\||\hat{\bar a}^{\gamma}|^2\|_\infty \\
&= \arg\min_{\gamma>0}(|1-\gamma|+|\gamma s|)^2 \\
&= 1
 \end{array} \]
\[
\min_{\gamma\in (0,1)}\||\hat{\bar a}^{\gamma}|^2\|_\infty
= (|1-\gamma_\infty|+|\gamma_\infty s|)^2 
= |s|^2
\]
\end{lemma}
And
 \[
 \begin{array}{ll}
\|*_{r+1} (\bar a^{\gamma_\infty  })\|_2^2 
&=  s^{2r+2} 
\end{array} \]

If we instead choose $\gamma_2$ as $ \cos^2{\theta} = \frac{1}{1+s^2}$ to minimize the $\ell_2$ norm instead, then
\[
\|\hat{\bar a}^{\gamma_2}\|_2^2  = \min_{\gamma\in (0,1)}\|\hat{\bar a}^{\gamma}\|_2^2 
=  (1-\gamma_2)^2 + (\gamma_2 s)^2 
= \frac{s^2}{1+s^2}
\]
And in this case,
 \[
  \begin{array}{ll}
\||\hat{\bar a}^{\gamma_2}|^2\|_\infty
&= (|1- \frac{1}{1+s^2}|+| \frac{1}{1+s^2} s|)^2 \\
&=  s^2 (\frac{1+|s|}{1+s^2})^2 \in (s^2, 1)
 \end{array} \]
 
 Therefore, 
 \[
  \begin{array}{ll}
\||\hat{\bar a}^{\gamma_2}|^2\|_\infty \in (s^2, 1)
 \end{array} \]
\section{Subgradient KKT condition and phase transition}
\paragraph{Exact calculation of subgradient for phase transition condition}

Here I try to give a clean proof of phase transition directly using sub-gradient and directional derivative.

\begin{lemma}
Let $J \subset [n]$ be the support of $X$, let $\psi_J : = \psi\cdot 1_J$ denote the entrywise product of vector $\psi$ with the indicator vector of subset $J$.  Let $B_J$ denote
the central section of the euclidean  ball $B({\mathbb R}^n)$,
where the slice is produced the linear space $\mbox{span} \{e_i : i\in J \}$.
Alternatively, we may write
$B_J: = \{ v_J: \| v_J\|_2 \leq 1\}$.  The set-valued subgradient
operator applied to $\|\psi_J\|$
evaluates as follows:
 \[
 \begin{array}{ll}
\partial_{\psi}    \| \psi_J \|
&=  N(\psi_J).
\end{array}
  \]
Here $N$ maps ${\mathbb R}^n$ into  subsets of ${\mathbb R}^n$,
and is given by:
\[
N(\psi_J):= 
\begin{cases}
    \psi_J/\|\psi_J\|_2, &  \psi_J\neq 0 \\
    B_J, &  \psi_J = 0
  \end{cases},
\]  
\end{lemma}

\begin{definition}
Consider a probability space containing just the possible  
outcomes  $J \subset [n]$.
Let $S_J$, $J \subset [n]$,  denote a closed compact subset of ${\mathbb R}^n$.
Let $I$ be a random subset of
$[n]$ drawn at random from this probability space with probability $\pi_J = P\{ I = J \}$
of elementary event $J$.
Consider the set-valued random variable $S \equiv S_J$. We define its expectation as
\[
 {{\mathbb E}} S  := \sum_J \pi_J \cdot S_J.
\]
On the right side, we mean the closure of the 
compact set produced by all sums of the form
\[
 \sum_J \pi_J \cdot s_J
\]
where $s_J \in S_J$.
\end{definition}

\begin{lemma}
For any $\psi \in {\mathbb R}^n$, and $X = (X_i)_{i=1}^n$ with $X_i \sim_{iid} BG(p,0,1)$.
Let now $I \subset [n]$ be the random support of $X$.
Let $\partial_\psi $ denote the set-valued subgradient
operator.
  \[
 \begin{array}{ll}
\partial_\psi [E{|\sum_{i} \psi_i X_i| }  ]
&= \sqrt{\frac{2}{\pi}} \cdot {{\mathbb E}}_I [N(\psi_I)].
\end{array}
  \]
\end{lemma}

\paragraph{Subgradient and directional derivative at $e_0$}
Now we focus on $e_0$. Note that for any subset $J \subset [n]$, $(e_0)_J$ is either the zero
vector or else $e_0$. Hence $N((e_0)_J)$
is either $B_{J}$ or $\{e_0\}$. What drives
this dichotomy is whether $0 \in J$ or not.

\begin{lemma}  
\[
 \begin{array}{ll}
{\mathbb E}_I [N((e_0)_I) ]
&= p e_0 + \sum_{J,0\not\in J } \pi_{J} B_{J} \\
& = p e_0 + (1-p) {\cal B}_0
\end{array}
  \]
where
\[
 \begin{array}{ll}
{\cal B}_0 = {\mathbb E}_{I}[ B_{I} | 0 \not \in I ] = 
(1-p)^{-1} \cdot \sum_{J,0\not\in J } {\pi_{J}} B_{J}.
\end{array}
\]
 
\end{lemma}


\begin{proof}
\begin{eqnarray*}
{\mathbb E}_I [N((e_0)_I) ] &=& \sum_{J } \pi_{J} B_{J} \\
&=&  \sum_{J, 0 \in J}   \pi_{J} e_0 +  \sum_{J,0\not\in J }  \pi_{J} B_{J}\\
&=& p \cdot e_0 + (1-p) \cdot {{\cal B}}_0 .
\end{eqnarray*}
 \end{proof}
To compute with ${\mathbb E}_I [N((e_0)_I) ]$,
we need:
\begin{lemma} \label{eq:ComputeExpect}
For each fixed $\beta \in {\mathbb R}^n$:
\[
  \sup_{b \in {\cal B}_0} \langle \beta, b \rangle
   = {\mathbb E}_{I}[ \|\beta_I\|_2 | 0 \not \in I  ].
  \]
\end{lemma}

\begin{proof}
Note that in the definition of the set 
\[
{\cal B}_0 = {\mathbb E}_{I}[ B_{I} | 0 \not \in I ],
\]
 each term $B_J$ obeys the bound
$\sup_{g_J \in B_J} \|g_J\|_2 = 1 $.
Now, given the fixed vector $\beta \in {\mathbb R}^n$, define:
\begin{eqnarray*}
    b^* &=& {\mathbb E}_{I}[ \frac{\beta_I}{\| \beta_I \|_2} | 0 \not \in I  ]\\
     &=&  \sum_{0 \not \in J} \pi_J \frac{\beta_J }{\|\beta_J\|_2}.
 \end{eqnarray*}
 Since each term in this sum has euclidean norm at most $1$, $b^* \in {{\cal B}}_0$.
Now
\begin{eqnarray*}
\langle  \beta, b^* \rangle &=&
\langle \beta , \sum_{0 \not \in J} \pi_J \frac{\beta_J }{\|\beta_J\|_2} \rangle \\
&=& \sum_{0 \not \in J} \pi_J \langle \beta , \frac{\beta_J }{\|\beta_J\|_2} \rangle \\
&=& \sum_{0 \not \in J} \pi_J \|\beta_J\|_2 .
 \end{eqnarray*}
On the other hand, for any fixed 
vector $g = \sum \pi_J g_J $,
with each $g_J \in B_J$,
we have
\begin{eqnarray*}
\langle  \beta, g \rangle &=&  \sum_{0 \not \in J} \pi_J \langle \beta , g_J  \rangle \\
&\leq& \sum_{0 \not \in J} \pi_J  \|\beta_J \|_2 \|g_J\|_2 \\
& \leq & \sum_{0 \not \in J} \pi_J  \|\beta_J \|_2 
= \langle  \beta, b^* \rangle.
 \end{eqnarray*}
\end{proof}

Lemma (\ref{eq:ComputeExpect}) 
can be viewed as a special instance of
Theorem \textbf{A.15} from \cite{bai2018subgradient}:
\begin{lemma}[Interchangeability of set expectation and support function] 
Suppose a random compact set $S \subset {\mathbb R}^n$ is integrably bounded and the underlying probability space is non-atomic, then ${\mathbb E}[S]$ is a convex set and for any fixed vector $\beta \in {\mathbb R}^n,$
 \begin{equation} \label{eq:SetExpectEquality}
 \sup_{g\in {\mathbb E} S} \langle\beta, g\rangle =  \sup_{g\in  S} {\mathbb E}\langle\beta, g\rangle .
 \end{equation}
\end{lemma}

Define $\beta_{(0)} = \beta \cdot 1_{\{0\}^c}$ as the part of $\beta$ supported away from $0$.
Further define 
\[
V_1(\beta_{(0)}):= \frac{{\mathbb E}_{I}[ \|\beta_{I}\|_2 | 0 \not \in I ]}{\|\beta_{(0)}\|_2}.
\]
Let $N = \|\beta_{(0)}\|_0$; then 
 $V_1(\beta_{(0)})$ satisfies
\begin{equation} 
 p  \leq  V_1(\beta_{(0)}) \leq \sqrt{p} .
\end{equation}

\begin{lemma}


 The directional derivative at $e_0$ along $\beta$ 
 evaluates to the following:
 \begin{eqnarray*}
 \lim_{t\rightarrow 0^+}\frac{1}{t}({\mathbb E}_I [\|(e_0+t\beta)_I\|_2 - \|(e_0)_I\|_2]) 
 &=&  p \cdot \langle\beta, e_0\rangle + (1-p) \cdot {\mathbb E}_{I}[ \|\beta_{I}\|_2 | 0 \not \in I ]\\
 &=& [p \cos{\angle(\beta, e_0)} + (1-p)   \sin{\angle(\beta, e_0)} V_1(\beta_{(0)})] \|\beta\|_2
 \end{eqnarray*}
\end{lemma}
 \begin{proof}

Let $\langle b, v \rangle  \equiv b^T v$.

Applying Lemma \ref{eq:ComputeExpect}, we proceed as follows:
\[
 \begin{array}{ll}
\lim_{t\rightarrow 0^+}\frac{1}{t}({\mathbb E}_I [\|(e_0+t\beta)_I\|_2 - \|(e_0)_I\|_2])
&= \sup_{g\in {\mathbb E}_I [N((e_0)_I)]} \langle\beta, g\rangle\\
&= p \cdot \langle\beta, e_0\rangle + (1-p) \cdot  \sup_{g\in {\cal B}_0 } \langle\beta, g\rangle\\

&= p \cdot \langle\beta, e_0\rangle + (1-p) \cdot {\mathbb E}_{I}[ \|\beta_{I}\|_2 | 0 \not \in I ] \\
&= [p \cdot \cos{\angle(\beta, e_0)} + (1-p) \cdot  \sin{\angle(\beta, e_0)} V_1(\beta_{(0)})] \|\beta\|_2
\end{array}
  \]
  \end{proof}


\paragraph{Projected subgradient}
\begin{lemma} 
If $\beta$ need to satisfy a constraint $u^T \beta =0$, then we only consider directional derivative at $e_0$ along $\beta$ for those $\beta$ that $u^T \beta =0$.
Therefore, let $P_{u}^{\perp}$ be the projection onto the hyperplane as the orthogonal complement of $u$, then the following three equivalent conditions all make sure $e_0$ is the stationary point.
\begin{itemize}
\item
  \[
 \begin{array}{ll}
0\in P_{u}^{\perp} {\mathbb E}_I [N((e_0)_I) ]
\end{array}
  \]
  \item
Equivalently, there exists a subgradient $g\in  P_{u}^{\perp} {\mathbb E}_I [N((e_0)_I) ]$ such that for all $\beta$ satisfy $u^T \beta = 0$,
\[
\beta^T g \geq 0
\]
\item
Also equivalently, for all $\beta$,
\[
 \begin{array}{ll}
\sup_{g\in  [N((e_0)_I)]}{\mathbb E}_I[ \langle   P_{u}^{\perp}  \beta, g\rangle] \quad \geq 0
\end{array}
  \]
  \end{itemize}
\end{lemma}
 
 Let $\theta = \angle({u, e_0})$, for $\beta$ need to satisfy a constraint $u^T \beta =0$,
we get
\[
 \begin{array}{ll}
\lim_{t\rightarrow 0^+}\frac{1}{t}({\mathbb E}_I [\|(e_0+t\beta)_I\|_2 - \|(e_0)_I\|_2])
&= [p \cos{\angle(\beta, e_0)} + (1-p)  V_1(\beta_{(0)}) \sin{\angle(\beta, e_0)}] \|\beta\|_2
\end{array}
  \]
  Here $\angle(\beta, e_0)\in [0, \pi], \sin{\angle(\beta, e_0)}\in [0,1]$. 
  
\begin{figure} 
\centering
\includegraphics[width=.7\textwidth]{./figures_BD/Angle_Diagram_label.png}
\caption{Demonstration of relation between
  $\angle( \beta, -e_0)$ and ${\angle(u, e_0)}$}
\label{fig-Angle}
\end{figure}
  
  \begin{lemma}
  \label{Angle_Inequality}
   \[
  \angle(  \beta, - e_0) \geq  \frac{\pi}{2} - \theta
  \]
  
    \[
  \tan(\angle(  \beta, - e_0)) \geq \cot{\theta}
  \]
  \end{lemma}
  
  \begin{proof}
   
we know that geometrically, using the property that projection of $e_0$ on the hyperplane with normal vector $u$ has the smallest angle among all the $\beta$ in that hyperplane, we have
   \[
  \angle(  \beta, - e_0) \geq \frac{\pi}{2} - \angle(u, e_0) = \frac{\pi}{2} - \theta
  \]
then
  \[
  \tan(\angle(  \beta, - e_0)) \geq \cot{\theta}
  \]
 
  \end{proof}
  
  
 \begin{lemma}
 Let $H_u=\{\beta : \quad u^T \beta =0 \}$. For all $p$ that 
   \[
      \frac{1-p}{ p}  \inf_{\beta\in H_u} [V_1(\beta_{(0)}) \tan{(\angle(\beta, -e_0))}]>1
  \]
 the convex program finds the correct inverse filter $a^{-1}$ and hence the sparse signal $X$.
  \end{lemma}
  


  \begin{proof}
 
We know that the KKT condition of the convex program is
  \[
  p \cos{\angle(\beta, e_0)} + (1-p)  V_1(\beta_{(0)}) \sin{\angle(\beta, e_0)}>0
  \]
  if and only if 
  $\cos{\angle(\beta, e_0)}\geq 0,$ or $\cos{\angle(\beta, e_0)}<0$ and 
  \[
      \frac{1-p}{ p}  V_1(\beta_{(0)}) \tan{(\angle(\beta, -e_0))}>1
  \]
  If $\cos{\angle(\beta, e_0)}<0, \cos{\angle(\beta, -e_0)}>0, \angle(\beta, -e_0) \in [0, \frac{\pi}{2}).$ 
  
  Therefore,
  \[
      \frac{1-p}{ p}  \inf_{\beta\in H_u} [V_1(\beta_{(0)}) \tan{(\angle(\beta, -e_0))}]>1
  \]
  
   \end{proof}
 
  
   \begin{theorem}
 \label{thm:PT_lower}
 Let $p^\star_1 = 1- \tan{\theta},$ then if $p< p^\star_1 $, then the convex program finds the correct inverse filter $a^{-1}$ and hence the sparse signal $X$.
 \end{theorem}
 

 \begin{proof}
  Now we study the relation between
  `
  As demonstrated in figure~\ref{fig-Angle}. Since $u^T \beta =0, P_{u}^{\perp}  \beta = \beta.$ Without loss of generality we could assume $\angle(u, e_0) \in [0, \frac{\pi}{2}]$ (otherwise we could flip the sign of the final solution.)


  
  Therefore, we could find a lower bound
    \begin{eqnarray*}
      \frac{1-p}{ p}  \inf_{\beta\in H_u} [V_1(\beta_{(0)}) \tan{(\angle(\beta, -e_0))}]
      &\geq& \frac{1-p}{ p}  \inf_{\beta\in H_u} V_1(\beta_{(0)}) \inf_{\beta\in H_u}\tan{(\angle(\beta, -e_0))}]\\
      &=& \frac{1-p}{ p} \inf_{\beta\in H_u} V_1(\beta_{(0)}) \cot{\theta}
  \end{eqnarray*}
 
Moreover,
\[
\inf_{\beta\in H_u} V_1(\beta_{(0)}) \geq p
\]
where the lower bound is achieved when $\beta_{(0)}$ is one-sparse.
So we have a lower bound $p^\star_1 $ (on the actual phase transition sparsity $p^\star$) defined as,
  \[
 \frac{1-p_1^\star}{ p_1^\star} p_1^\star \cot{\theta} =1
  \]
  Then $p^\star_1 = 1- \tan{\theta},$ then if $p< p^*$, then the convex program finds the correct inverse filter $a^{-1}$ and hence the sparse signal $X$.
  \end{proof}
  
  
  
  
   
  
 
  


\paragraph{Detailed characterization of phase transition threshold}
  We define 
  \[
  b:= \beta_{(0)}/\|\beta\|_2
  \]
   \[
  \nu:= u_{(0)}/\|u_{(0)}\|_2=u_{(0)}/\sin{\theta}
  \]
  so that $\|\nu\|_2=1$.
  Then using the fact that $u_0= \cos{\theta}$, $\|u\|_2=1$ and $\beta_0 = -\sqrt{1-\| \beta_{(0)}\|_2^2}$,
  the condition $u^T \beta =0$ can be translated into
  \[
  -\cos{\theta}\sqrt{1-\| b\|_2^2} + \sin{\theta} \nu^T b =0
  \]
  Then
   \[
  \nu^T b /\sqrt{1-\| b\|_2^2} = \cot{\theta}
  \]
  Therefore, using Cauchy inequality, we recover the geometric conclusion
    \[
  \|b\|_2/\sqrt{1-\| b\|_2^2}  \geq \nu^T b /\sqrt{1-\| b\|_2^2} = \cot{\theta}
  \]
   \begin{lemma}
  We define the phase transition threshold as
  \[
 p^\star(u):= 
 \sup_{p\in [0,1]}\{\inf_{\|\beta\|_2=1, u^T \beta =0 }(-p \cdot \sqrt{1-\| \beta_{(0)}\|_2^2} + (1-p) \cdot  \| \beta_{(0)}\|_2 V_1(\beta_{(0)}))> 0\}
 \]
then
 \[
 p^\star(u):= 
 \sup_{p\in [0,1]}\{\inf_{\| P_{u}^{\perp}\beta\|_2=1 }(-p \cdot \sqrt{1-\| P_{e_0}^{\perp}  P_{u}^{\perp}\beta\|_2^2} + (1-p) \cdot  {\mathbb E}_I\| ( P_{e_0}^{\perp} P_{u}^{\perp} \beta)_I\|_2 ) > 0\}
 \]

 Additionally, for a given $\epsilon>0$, we can define $p^\star(\epsilon),$ as 
 \[
 p^\star(u,\epsilon):= 
 \sup_{p}\{\inf_{\|\beta\|_2=1, u^T \beta =0 }(-p \cdot \sqrt{1-\| \beta_{(0)}\|_2^2} + (1-p) \cdot  \| \beta_{(0)}\|_2 V_1(\beta_{(0)}))> \epsilon\}
 \]
 Then $p^\star(u,\epsilon)<p^\star(u).$
 \end{lemma}
 
 
    \begin{theorem}
 \label{thm:PT_upper}
 Let the $\mu$ dependent upper bound $p^\star_2(u)$ be the solution of the equation
      \[
  \tan{\theta}  = 
 \frac{1-p^\star}{p^\star}V_1(\nu)
 \] 
then for a given initial choice $v$ and true inverse filter $a^{-1}$, we define $u$ by $\title{u}= a^{-1} * v$, and reparemetrize $u = \cos{\theta} e_0 +\sin{\theta}\nu$, then the phase transition point has an upper bound given by $p^\star_2(u)$.
If $p> p^\star_2$, then the solution of the convex program is not the correct inverse filter $a^{-1}$.

And there is a $\mu$ independent upper bound $p^\star_3$, as the solution of the equation
    \[
  \tan{\theta}  = 
 \frac{1-p^\star}{p^\star}\sqrt{p^\star}
 \] 
 then $p^\star_1 \leq p^\star(u)\leq p^\star_2(u)\leq p^\star_3$.
 \end{theorem}
 \begin{coro}
If $u$ is two sparse, there exists $i$ such that $u_i = \pm \sin{\theta},$ then $u$ is also two sparse and we could get a two sparse $\beta = (0, \ldots,0, \beta_0, 0, \ldots,0, \beta_i, 0, \ldots,0). $ Then $ p_2^\star(u) = p^\star_1$, therefore, $p^\star(u) = p^\star_1 = 1-\tan{\theta} $. 
 \end{coro}
  
   \begin{proof}
  We want to calculate 
   \[
 B(p)= \inf_{\|\beta\|_2=1, u^T \beta =0 }(-p \cdot \sqrt{1-\| \beta_{(0)}\|_2^2} + (1-p) \cdot  \| \beta_{(0)}\|_2 V_1(\beta_{(0)}))
  \]
 After the change of variable,
   \[
  B(p)= \inf_{\nu^T b /\sqrt{1-\| b\|_2^2} = \cot{\theta} }(-p \cdot \sqrt{1-\|b\|_2^2} + (1-p) \cdot  \| b\|_2 V_1(b))
  \]
  $\frac{b}{\|b\|_2}=\nu$, $\|b\|_2 =\cos{\theta}$ is one feasible point to the equation $\nu^T b /\sqrt{1-\| b\|_2^2} = \cot{\theta}$, it correspond to $\beta = P_u (-e_0)$. 
  Therefore, plug in this specific $\beta$,
  \[
 \cot{\theta} }(-p \cdot \sin{\theta} + (1-p) \cdot  \cos{\theta} V_1(\nu)) \geq \inf_{\nu^T b /\sqrt{1-\| b\|_2^2} = \cot{\theta} }(-p \cdot \sqrt{1-\|b\|_2^2} + (1-p) \cdot  \| b\|_2 V_1(b))
  \]
  we find an upper bound on phase transition point  $p^\star_2(u) \geq p^\star(u)$. 
  Here $p^\star_2(u)$ is defined by
   \[
 p_2^\star(u):= 
 \sup_{p\in [0,1]}\{\cot{\theta} }(-p \cdot \sin{\theta} + (1-p) \cdot  \cos{\theta} V_1(\nu)) > 0\}
 \]
 then $p^\star_2(u)$ is the solution of the equation
   \[
  \tan{\theta}  = 
 \frac{1-p^\star}{p^\star}V_1(\nu)
 \] 
 Since $V_1(\nu)\leq \sqrt{p}$,
 the $\mu$ independent upper bound $p^\star_3$ is the solution of the equation
    \[
  \tan{\theta}  = 
 \frac{1-p^\star}{p^\star}\sqrt{p^\star}
 \] 
  \end{proof}
  
  

  \begin{lemma}
 



  \end{lemma}
  
  
  
 
 


  
  
  



 
  







































\subsection{Bi-Lipschitz property of finite difference of objective near $e_0$}
We study the upper and lower bound of: 
\[
B(p, \phi) := \frac{{\mathbb E}_I  \| (e_0+\phi)_I \|_2 - {\mathbb E}_I  \| (e_0)_I \|_2}{\|\phi\|_2} 
\]


\paragraph{Bi-Lipschitzness of finite difference of objective for sphere constraint}
\begin{lemma}
For the sphere constraint $\|e_0 + \phi\|_2 =1$ for any $\phi$, 
  \[
 \begin{array}{ll}
\frac{1}{16}p(1-p) \|\phi\|_2 \leq  
V_1(e_0+\phi) - V_1(e_0) \leq 2\sqrt{N} \|\phi\|_2
\end{array}
  \]
\end{lemma}
This lemma is based on Proposition $C.7$(upper bound) and $C.8$(lower bound) of section $C.6$ of the paper \cite{bai2018subgradient}, proof can be found in there.






 
  


\paragraph{Bi-Lipschitzness of finite difference of objective for linear constraint}
We normalize the problem by defining $t:=\|\phi\|_2, \beta:=\frac{\phi}{\|\phi\|_2}$.
After normalization, and taking into account of the linear constraint $u^T \phi = 0$, we will study that  for $\beta\in \mathcal{B}_t :=\{\|\beta\|_2=1, u^T \beta = 0,  \|e_0+t\beta\|_\infty\leq 1 \}$ for finite $t$, what is the upper and lower bound of 
\[
\frac{{\mathbb E}_I  \| (e_0+t\beta)_I \|_2 - {\mathbb E}_I  \| (e_0)_I \|_2}{t} 
\]
First, if $t\rightarrow 0^+$, then we get the directional derivative along direction of $\beta$. 
\begin{eqnarray*}
\lim_{t\rightarrow 0^+}\frac{{\mathbb E}_I  \| (e_0+t \beta)_I \|_2 - {\mathbb E}_I  \| (e_0)_I \|_2}{t} 
&=&(1-p) {\mathbb E}_{I}[ \|\beta_{I}\|_2 | 0 \not \in I ]  + 
p\beta_0  
\\
&=&
(1-p) V_1(\beta_{{(0)}}) \sqrt{1-\beta_0^2} + 
p\beta_0  
\end{eqnarray*}
where  $\|\beta_{(0)}\|_2 = \sqrt{1-\beta_0^2} $ since  $\|\beta\|_2 = 1$, and
\[
V_1(\beta_{{(0)}})= \frac{{\mathbb E}_{I}[ \|\beta_{I}\|_2 | 0 \not \in I ]}{\|\beta_{(0)}\|_2}.
\]
We know that for 
\begin{eqnarray*}
p \leq V_1(\beta_{{(0)}})
\leq \sqrt{p}
\end{eqnarray*}
Where  
\[
\beta^a = ( \beta_0,\sqrt{1-\beta_0^2}, 0,\ldots,0 )
\]
achieves lower bound,
and 
\[
\beta^b = (\beta_0,\frac{1}{\sqrt{N-1}}\sqrt{1-\beta_0^2},\ldots, \frac{1}{\sqrt{N-1}}\sqrt{1-\beta_0^2} ) = \beta_0 e_0 + \frac{1}{\sqrt{N-1}}\sqrt{1-\beta_0^2} (\sum_{i\neq 0} e_i)
\]
achieves upper bound.


For finite $t$, we calculate the finite difference condition on $I_0$:
\begin{eqnarray*}
\frac{{\mathbb E}_I  \| (e_0+t \beta)_I \|_2 - {\mathbb E}_I  \| (e_0)_I \|_2}{t}
&=&
(1-p)  {\mathbb E}_{I}[ \|\beta_{I}\|_2 | 0 \not \in I ] + p\big(\frac{{\mathbb E}_I [ \| e_0+t\beta_I \|_2 -   1 | 0 \in I ]}{t} \big)
\end{eqnarray*}

We look into the following quantity for finite $t$,
\begin{eqnarray*}
\frac{{\mathbb E}_I [ \| e_0+t\beta_I \|_2 -   1 | 0 \in I ]}{t}
&=&
{\mathbb E}_{I}[ \left( (\frac{1}{t}+ \beta_0)^2+\| \beta_{I_{(0)}} \|_2^2 \right)^{1/2} -\frac{1}{t}| 0 \in I ]\\
&=& \frac{1}{t}{\mathbb E}_{I}[ \left( 1+ 2t\beta_0 + t^2 \beta_0^2+t^2\| \beta_{I_{(0)}} \|_2^2 \right)^{1/2} -1| 0 \in I ]
\end{eqnarray*}

\begin{lemma}
We have upper bound 
\begin{eqnarray*}
 \frac{1}{t}{\mathbb E}_{I}[ \left( 1+ 2t\beta_0 + t^2 \beta_0^2+t^2\| \beta_{I_{(0)}} \|_2^2 \right)^{1/2} -1| 0 \in I ] 
\leq  \beta_0+ \frac{t}{2} p \| \beta_{{(0)}} \|_2^2  + \frac{t}{2} \beta_0^2
\end{eqnarray*}
and a lower bound
\begin{eqnarray*}
\frac{1}{t} (\left| 1 + t \beta_0 \right|-1 ) \leq \frac{1}{t}{\mathbb E}_{I}[ \left( 1+ 2t\beta_0 + t^2 \beta_0^2+t^2\| \beta_{I_{(0)}} \|_2^2 \right)^{1/2} -1| 0 \in I ] 
\end{eqnarray*}
we comment that if $\beta_0 \geq -\frac{1}{t}$, the lower bound reduce to
\begin{eqnarray*}
\beta_0 \leq \frac{1}{t}{\mathbb E}_{I}[ \left( 1+ 2t\beta_0 + t^2 \beta_0^2+t^2\| \beta_{I_{(0)}} \|_2^2 \right)^{1/2} -1| 0 \in I ] 
\end{eqnarray*}
\end{lemma}

\begin{proof}

Now we use the expansion
\[
\sqrt{1+x}-1= \frac{x}{2}-\frac{1}{8}x^2+1/16x^3-5/128x^4+O(x^5)
\]
first we have upper bound 
for any $x\geq -1$:
\[
\sqrt{1+x}-1 - \frac{x}{2}\leq 0
\]
Additionally, using concavity of function $\sqrt{1+x}$, we have
\begin{eqnarray*}
 && \frac{1}{t}{\mathbb E}_{I}[ \left( 1+ 2t\beta_0 + t^2 \beta_0^2+t^2\| \beta_{I_{(0)}} \|_2^2 \right)^{1/2} -1| 0 \in I ]\\
&\leq&  \frac{1}{t}[ \left( 1+ 2t\beta_0 + t^2 \beta_0^2+t^2{\mathbb E}_{I}\| \beta_{I_{(0)}} \|_2^2| 0 \in I ] \right)^{1/2} -1]\\
&=&  \frac{1}{t}[ \left( 1+ 2t\beta_0 + t^2 \beta_0^2+t^2 p \| \beta_{{(0)}} \|_2^2 \right)^{1/2} -1]\\
&=&  \frac{1}{t}[ \left( 1+ 2t\beta_0 + t^2 \beta_0^2+t^2 p (1-\beta_0^2) \right)^{1/2} -1]
\end{eqnarray*}
Now, apply the inequality above, we have
\begin{eqnarray*}
 \frac{1}{t}[ \left( 1+ 2t\beta_0 + t^2 \beta_0^2+t^2 p \| \beta_{I_{(0)}} \|_2^2 \right)^{1/2} -1]
&\leq&  \beta_0 + \frac{t}{2} (p \| \beta_{{(0)}} \|_2^2  +  \beta_0^2)
\end{eqnarray*}
On the other hand,
\begin{eqnarray*}
\frac{1}{t} {\mathbb E}_{I}[ \left( (1+ t\beta_0)^2+t^2\| \beta_{I_{(0)}} \|_2^2 \right)^{1/2} -\frac{1}{t}| 0 \in I ]
&\geq&  \frac{1}{t} (\left| 1 + t \beta_0 \right|-1 )
\end{eqnarray*}
when $\beta_0 \geq -\frac{1}{t}$,
\begin{eqnarray*}
\frac{1}{t} (\left| 1 + t \beta_0 \right|-1 )
= \beta_0 
\end{eqnarray*}

\end{proof}

\begin{theorem}
\label{thm:bound_diff}
We have upper bound and lower bound 
\begin{eqnarray*}
\frac{{\mathbb E}_I  \| (e_0+t \beta)_I \|_2 - {\mathbb E}_I  \| (e_0)_I \|_2}{t} 
\leq 
(1-p)\Big( V_1(\beta_{{(0)}}) \sqrt{1-\beta_0^2}\Big) + 
p\Big(\beta_0  +
  \frac{t}{2} \beta_0^2 
 +
 p\frac{t}{2}  (1-\beta_0^2) \Big)
\end{eqnarray*}
and lower bound 
\begin{eqnarray*}
\frac{{\mathbb E}_I  \| (e_0+t \beta)_I \|_2 - {\mathbb E}_I  \| (e_0)_I \|_2}{t} 
\geq 
(1-p)\Big( V_1(\beta_{{(0)}}) \sqrt{1-\beta_0^2}\Big) + 
p\Big(\frac{1}{t} (\left| 1 + t \beta_0 \right|-1 \Big)
\end{eqnarray*}
Specially, when $t\leq 1$, $\frac{1}{t} (\left| 1 + t \beta_0 \right|-1 = \beta_0$, therefore,
\begin{eqnarray*}
\frac{{\mathbb E}_I  \| (e_0+t \beta)_I \|_2 - {\mathbb E}_I  \| (e_0)_I \|_2}{t} 
\geq 
(1-p)\Big( V_1(\beta_{{(0)}}) \sqrt{1-\beta_0^2}\Big) + 
p\beta_0 = \lim_{t\rightarrow 0^+}\frac{{\mathbb E}_I  \| (e_0+t \beta)_I \|_2 - {\mathbb E}_I  \| (e_0)_I \|_2}{t} 
\end{eqnarray*}
\end{theorem}

Qingyun's Remark: this bound is numerically verified for $t\leq 5$ with Matlab. 
Where we use the fact that $p\leq V_1(\beta_{{(0)}}) \leq \sqrt{p},$ for both $\beta_0 = 1, -0.5, -1/\sqrt{N}, 0,1/\sqrt{N}, 0.5, 1 $, we tested the two extreme points  
\[
\beta^a = ( \beta_0,\sqrt{1-\beta_0^2}, 0,\ldots,0 )
\]
and 
\[
\beta^b = (\beta_0,\frac{1}{\sqrt{N-1}}\sqrt{1-\beta_0^2},\ldots, \frac{1}{\sqrt{N-1}}\sqrt{1-\beta_0^2} ) = \beta_0 e_0 + \frac{1}{\sqrt{N-1}}\sqrt{1-\beta_0^2} (\sum_{i\neq 0} e_i)
\]

\begin{coro}
\label{coro:bound_diff}
We have upper bound 
\begin{eqnarray*}
\frac{{\mathbb E}_I  \| (e_0+t \beta)_I \|_2 - {\mathbb E}_I  \| (e_0)_I \|_2}{t} 
&\leq& 
(1-p)\Big( \sqrt{p} \sqrt{1-\beta_0^2}\Big) + 
p\Big(\beta_0  +
  \frac{t}{2} \beta_0^2 
 +
 p\frac{t}{2}  (1-\beta_0^2) \Big) \\
 &\leq& 
 (1-p)\sqrt{p}  + 
p\Big(1  +
  \frac{t}{2} 
 +
 p\frac{t}{2}   \Big)
\end{eqnarray*}



\end{coro}


\begin{coro}
\label{coro:bound_diff}
We have  lower bound for $p<p^\star(\epsilon)$ and $t\leq 1$:
\begin{eqnarray*}
\frac{{\mathbb E}_I  \| (e_0+t \beta)_I \|_2 - {\mathbb E}_I  \| (e_0)_I \|_2}{t} 
&\geq&
\lim_{t\rightarrow 0^+}\frac{{\mathbb E}_I  \| (e_0+t \beta)_I \|_2 - {\mathbb E}_I  \| (e_0)_I \|_2}{t} 
\geq
\epsilon
\end{eqnarray*}


\end{coro}




























































 

 


\section{Old concentration results}

 
 

  
 
  
  
  



 





  









 





  
  



 






Now we need to derive a concentration inequality on $D$. 
We focus on deriving a concentration inequality on $\|h*x\|_1$, and the concentration inequality 
on  $\|(h*x)_S\|_1$ can be derived similarly.

We remark that $ \|h*x\|_1 =\sum_{j=1}^n |h^{(j),T} x|= \sum_{j=1}^n |\langle C(a^{(j)}) v, x\rangle|$, 
if they are independent random variables, we can bound the deviation of $ \|h*x\|_1$ from its mean
using tail bounds via Bernstein’s inequality, by using the previous two bound on
variance and absolute value. 
This is the strategy to derive concentration inequality in dictionary learning\cite{luh2016dictionary,blasiok2016improved}.
This is also the strategy to derive concentration inequality for multiple observation blind deconvolution setting, since there are independence across different observations.

However, here due to the nature of the convolution, we cannot directly apply Bernstein’s inequality. There are additional dependencies that cause technical difficulties. 

Therefore, we use the dual representation of the $1$-norm and use Talagrand-Bernstein’s inequality on supreme of random function to derive concentration.
We put this part of the proof in the following lemma, since they are technical and they could be of independent interest for other problems with dependency structure. 




From the lemma, if we let $Z = \|h*x\|_1$, then we have 
 \[
  \begin{array}{ll}
& {\mathbb P} \left[  |\|h*x\|_1- {\mathbb E} \|h*x\|_1 | > t   \right] \\
&= {\mathbb P} \left[  |Z- {\mathbb E} Z | > t   \right] \\
&\leq   \exp\left(  \frac{-t^2/2   }{\sigma^2+K(2{\mathbb E} Z+t/3)}\right)
\end{array}
  \]  
where $\sigma^2\leq  {\mathbb E} Z^2$, $K\leq \|Z\|_\infty$.

From the previous calculation, we recall that we have a lower bound on expectation 
$ {\mathbb E} \|h*x\|_1$,
\[
{\mathbb E} \|h*x\|_1\geq    \sqrt{\frac{p}{2}}  n \| h\|_2 
\] 
\[
{\mathbb E} Z = {\mathbb E} \|h*x\|_1 \geq  \sqrt{\frac{p}{2}}  n \| h\|_2
\]
  









Now we calculate $\sigma^2, K$:
 \[
  \begin{array}{ll}
\sigma^2 &={\mathbb E} \langle w, C(h)x \rangle^2,\\
&={\mathbb E} (C(h)^* w)^T xx^* C(h)^* w  \\
&\leq  p \|C(h)^* w\|_2^2\\
&=p \langle w, C(h)C(h)^* w \rangle 
\end{array}
  \]  
Let $C(h)=UD(\hat{h})U^*$ be the eigen-decomposition, where $U$ is the unitary Fourier matrix, $D(\hat{h})$ is the diagonal matrix with $\hat{h}$ as its diagonal entries. Then using $h=a*v$, $\hat{h}=\hat{a}\hat{v}$, we have
      \[
  \begin{array}{ll}
(C(h)C(h)*) 
&=   UD(\hat{h})U^* U D(\hat{h})^* U^* \\
&=  UD(|\hat{h}|^2) U^* \\
&=  UD(|\hat{a}\hat{v}|^2) U^*. \\
\end{array}
  \] 
Now we have
 \[
  \begin{array}{ll}
\sigma^2 &\leq 
 p \langle w, C(h)C(h)^* w \rangle \\
&= p  \langle w,  UD(|\hat{a}\hat{v}|^2) U^* w \rangle \\
&= p  \langle U^* w,  D(|\hat{a}\hat{v}|^2) U^* w \rangle 
\end{array}
  \] 
Let $\tilde{w} = \langle U^* w $, using the fact that $\|w\|_{\infty}\leq 1$, we have  $\|\tilde{w}\|_{\infty}\leq 1$, therefore,
 \[
  \begin{array}{ll}
\sigma^2 &\leq  p  \langle \tilde{w},  D(|\hat{a}\hat{v}|^2) \tilde{w} \rangle \\
&=  p  \sum_{i=1}^n \tilde{w}_i^2 |\hat{a}_i\hat{v}_i|^2)  \\
&\leq  p  \sum_{i=1}^n  |\hat{a}_i\hat{v}_i|^2 \\
&\leq  p  \max_{i=1}^n |\hat{a}_i|^2\sum_{i=1}^n  |\hat{v}_i|^2 \\
&=  p  \max_{i=1}^n |\hat{a}_i|^2 \|v\|_2^2 \\
&=  p  \max_{i=1}^n |\hat{a}_i|^2 
\end{array}
  \] 
  
  Now we derive the absolute value bound. First, since $x_i$ are iid sampled from the product of Bernoulli random variable and standard Gaussian, the supreme of the absolute value $\tau = \max_i |x_i|$ is upper bounded by the  supreme of standard Gaussian, which is less than $  2\log^{1/2} n$ with probability $1-\exp(-Cn^2)$. In the following, we only consider this case and assume $\tau \leq 2\log^{1/2} n$ and add the tail probability into the tail probability in the concentration inequality.
  
 Using other side of Khintchine inequality, we have
  \[
 \begin{array}{ll}
&  |(h*x)_i|\\
 =&  |\sum_{j=1}^n h_{i-j} \Omega_{j} |G_{j}|\mbox{sign}(G_{j}) |\\
\leq&  
  \sqrt{\sum_{j=1}^{n} (h_{i-j} \Omega_{j}|G_{j}|)^2}\\
 \leq&  \tau  \sum_{i=1}^n \sqrt{\sum_{j=1}^{n}  h_{i-j}^2}\\
   =&   \tau  \| h\|_2 
\end{array}
  \]  
Therefore, we can take $K=  \tau  \| h\|_2  .$

 












We point out that the eigenvalues of $C(a)$ is the Fourier transform of $\hat a$ of $a$, which is the 
the entry-wise inverse $1/\hat w^{0}$. And the singular values of $C(a)$ are the absolute values $|\hat a|$. 
Therefore, using the fact that $\|v\|_2=1$,
\[
 \begin{array}{ll}
 \min_i |\hat a_i| \leq \|h\|_2 =  \|C(a)v\|_2 \leq \max_i |\hat a_i| 
\end{array}
  \]
  
Then the condition number of $C(a)$ can be represented as $\delta_a =   \max_i |\hat a_i|/\min_i |\hat a_i|$.



Overall, for $Z = \|h*x\|_1$, we have ${\mathbb E} Z \geq \sqrt{\frac{p}{2}}  n \| h\|_2$, we choose $t = \sqrt{\frac{p}{2}}  n \| h\|_2$. Since 
 \[
  \begin{array}{ll}
 {\mathbb P} \left[  |Z- {\mathbb E} Z | > t   \right] 
\leq   \exp\left(  \frac{-t^2/2   }{\sigma^2+K(2{\mathbb E} Z+t/3)}\right),
\end{array}
  \]  
we calculate
\[
  \begin{array}{ll}
 &  \frac{-t^2/2   }{\sigma^2+K(2{\mathbb E} Z+t/3)}\\
=&  -\frac{1   }{2(\sigma^2/t^2+K(2{\mathbb E} Z+t/3)/t^2)}\\
\leq &  -\frac{1   }{2(\sigma^2/t^2+K(2+1/3)/t)}\\
 \end{array}
  \]  
Now, we calculate both terms on the denominator,
  \[
  \begin{array}{ll}
   \sigma^2/t^2 &\leq (p  \max_{i=1}^n |\hat{a}_i|^2)/( \sqrt{\frac{p}{2}}  n \| h\|_2)^2 \\
  &\leq 2\max_{i=1}^n |\hat{a}_i|^2)/(\min_{i=1}^n |\hat{a}_i|^2 n^2 \\
   &\leq 2\delta_a^2/ n^2 ,
 \end{array}
  \]  
   \[
  \begin{array}{ll}
  (2+1/3)K/t 
 &\leq  \frac{7}{3}\tau  \| h\|_2 /\sqrt{\frac{p}{2}}  n \| h\|_2\\
  &\leq \frac{7\sqrt{2}}{3}\tau  /\sqrt{p}  n \\
   &\leq \frac{7}{2}\tau  /\sqrt{p}  n 
 \end{array}
  \]  
  
 As we discussed before, since $\tau$ is the maximum of IID Gaussian random variable, 
 $\tau \leq C_1 \log^{1/2}(n)$ with probability $1 - \exp(-C_2 n^2)$ for $C_1$ large enough.
 
And from the assumption we have $ \delta_a \leq C_0 n^{1/2}/ \log^{1/4}(n)$.
  
Therefore, 
\[
  \begin{array}{ll}
& {\mathbb P} \left[  |\|h*x\|_1- {\mathbb E} \|h*x\|_1 | > t   \right] \\
&\leq   \exp\left( \frac{-1   }{ 4\delta_a^2/ n^2  + 7\tau  /\sqrt{p}  n  } \right)\\
&\leq   \exp\left( \frac{-1   }{ 4\delta_a^2/ n^2  + 7 C_1 \log^{1/2}(n)  /\sqrt{p}  n  } -C_2 n^2\right)\\
&\leq   \exp\left( \frac{-1   }{ 4\delta_a^2/ n^2  + 7 C_1 \log^{1/2}(n)  /\sqrt{p}  n  } -C_2 n^2\right)\\
&\leq   \exp\left( \frac{-n   }{  \log^{1/2}(n) (4C_0+  C_3 \sqrt{p})} -C_2 n^2 \right)\\
&\leq   \exp\left( \frac{- C n   }{  (p\log(n))^{1/2}}  \right)
 \end{array}
  \]  
  
  

  
  
  
  
  
  
  
  


In the lemma below, we use a covering argument to give a union bound for $v\in B_k=\{v
\in{\mathbb R}^k\mid \|v\|_2=1\}$. 
We want to cover $B_k$ with $\epsilon$ balls. From classical covering argument,
we know that we need at most $|N| = (3/\epsilon)^k$ $\epsilon$ balls to cover $B_k$, and we use symmetrization trick to find the proper $\epsilon$.

Let the deviation be
\[
\tilde{\mathcal{X}}_v =   \sum_{i=1}^n \left( | \langle C(a^{(i)}) v, x\rangle| - {\mathbb E} | \langle C(a^{(i)}) v, x\rangle|\right).
\]  
We will choose $1/\epsilon$ to be the Lipschitz constant of  $\tilde{\mathcal{X}}_v$ as a function of $v$. Let's compute the Lipschitz constant.

 
 

We introduce additional independent random sign $s_j$ , which are IID sampled from Rademacher variable by defining 
\[
\mathcal{X}_v =   \sum_{i=1}^n s_j \left( |\langle C(a^{(i)}) v, x\rangle| \right),
\]  
 then by lemma $24$ of \cite{blasiok2016improved}, for every integer $p$ we have 
 \[
 \| \sup_{v\in B_k} |\tilde{\mathcal{X}}_v | \|_p \lesssim \| \sup_{v\in B_k}  |\mathcal{X}_v| \|_p
 \]
 This is called symmetrization trick. Using this trick, we only need to analyze the tail behavior of the random variable $\mathcal{X}_v$. To obtain tail bounds for maximum  of  $\sup_{v\in B_k}  |\mathcal{X}_v|$, we need to bound tails of the random variables $\mathcal{X}_v-\mathcal{X}_{v'}$ for $v, v'\in B_k$.
 
Since  
\[
\mathcal{X}_v-\mathcal{X}_{v'} =  
\sum_{i=1}^n s_i \left( | \langle C(a^{(i)}) v, x\rangle| -   | \langle C(a^{(i)}) v', x\rangle|\right).
\]

Then use the fact that 
\[
| \langle C(a^{(i)}) v, x\rangle| -   | \langle C(a^{(i)}) v', x\rangle| \leq   \left( | \langle C(a^{(i)}) (v-v'), x\rangle|  \right),
\]
and  both $s_i | \langle C(a^{(i)}) v, x\rangle| -   | \langle C(a^{(i)}) v', x\rangle| $ and  
$s_i \left( | \langle C(a^{(i)}) (v-v'), x\rangle|  \right)$ are symmetric, if we have a tail bound for 
$\sum_{i=1}^n s_i \left( | \langle C(a^{(i)}) (v-v'), x\rangle|  \right)$ , 
we also have tail bounds on $\mathcal{X}_v-\mathcal{X}_{v'}$.

From these calculation, we saw that the Lipschitz constant of  $\tilde{\mathcal{X}}_v$ is bounded by 
 \[
  \begin{array}{ll}
\sqrt{\sum_{j=1}^n (\sum_{i=1}^n s_i |\sum_{t=1}^n a^{i-j+t} x_t|)^2}
&\leq   \sqrt{n (\|a\|_1 \|x\|_1)^2}\\
&\leq  \tau \sqrt{n}np \|a\|_1
\end{array}
  \]  
  
Therefore, we can choose $1/\epsilon = \tau \sqrt{n}np \|a\|_1$, then
at most $|N| = (3\tau \sqrt{n}np \|a\|_1)^k$ balls can cover $B_k$.
We have a concentration inequality in each ball,
We define the tail event 
\[
\mathcal{E}_v = \{|\tilde{\mathcal{X}}_v | > t\}
\]
we can take a union bound,
 \[
  \begin{array}{ll}
 {\mathbb P} \left(\cup_{v \in B_k} \mathcal{E}^c_v \right) &\leq |N| {\mathbb P} \left[  |\|h*x\|_1- {\mathbb E} \|h*x\|_1 | > t   \right] \\
&\leq   (3\tau \sqrt{n}np \|a\|_1)^k {\mathbb P} \left[  |\|h*x\|_1- {\mathbb E} \|h*x\|_1 | > t   \right] 
\end{array}
  \] 
  
Since  $n>  k \log(k/p)$, for large enough constant $C$, we showed in the lemma that there exist a positive constant $C_0$, 
 \[
  \begin{array}{ll}
 {\mathbb P} \left(\cup_{v \in B_k} \mathcal{E}^c_v \right) &\leq  \exp\left(  k\log(3\tau \sqrt{n}np \|a\|_1) - \frac{ C_6 n   }{  (p\log(n))^{1/2}}\right) \\
&\leq    \exp\left( k(\log(3\tau)+\log(\|a\|_1)+\log(p)+3/2 \log(n)) -\frac{ C n   }{  (p\log(n))^{1/2}}\right) 
\end{array}
  \] 
  

Then bound on $\sup_{v \in B_k} |\mathcal{X}_v|$ would imply the same bound on $\sup_{v \in B_k} |\tilde{\mathcal{X}}_v|$.
And with this bound on $\sup_{v \in B_k} |\tilde{\mathcal{X}}_v|$, we proved that 
for all $v\neq 0, v \in B_k$, $D>0$ with probability $1-\delta$, where
\[
\delta \leq  \exp\left(  -C_0 \delta_a^{-2} k\log(kp) \right)
\]
Therefore, when $n>C_1k^2\log(kp)$, 
$
\delta \leq   \exp\left(  -C_0 \delta_a^{-2} k\log(kp) \right),
$
then $\delta$ converges to zero as $k$ goes to infinity.
\end{proof}





\begin{lemma}
Let $Z = \|h*x\|_1$ where $h \in \mathcal{H}$, then 
 \[
  \begin{array}{ll}
& {\mathbb P} \left[  Z - {\mathbb E} Z  > t   \right] \\
&\leq   \exp\left(  \frac{-t^2/2   }{\sigma^2+K(2{\mathbb E} Z+t/3)}\right)
\end{array}
  \]  
where $\sigma^2\leq  {\mathbb E} Z^2$, $K\leq \|Z\|_\infty$.
\end{lemma}
\begin{proof}
Using duality of $1$-norm, we know that
\[
\|h*x\|_1 = \sup_{\|w\|_\infty\leq 1}\langle w, C(h)x \rangle
\]
 From the theorem $8.42$ of \cite{foucart2013mathematical}, we have the following inequality:
 for the supreme of functions $\langle w, C(h)x \rangle$, since ${\mathbb E} \langle w, C(h)x \rangle=0$,
 $ \langle w, C(h)x \rangle\leq K$, 
 \[
 \sigma^2 ={\mathbb E} \langle w, C(h)x \rangle^2,
 \]
 we have
 \[
  \begin{array}{ll}
 {\mathbb P} \left[  Z - {\mathbb E} Z  > t   \right] \leq   \exp\left(  \frac{-t^2/2   }{\sigma^2+K(2{\mathbb E} Z+t/3)}\right)
\end{array}
  \]  
\end{proof}
We also remark that the essential concentration inequality that we are trying to derive is the following:
let's consider the case when $x$ is IID standard Gaussian, we define $z = h*x = C(h)x$,
then $z$ is a non-isotropic Gaussian vector \[\mathcal{N}(0,\Sigma(h)),\] where 
\[  \Sigma(h)_{ij} = \langle h^{(i)},  h^{(j)}\rangle\]
And we are trying to derive a concentration inequality for $\|z\|_1$.






\begin{lemma}
For all $v\in B_k$, the probability 
\[
  \begin{array}{ll}
& {\mathbb P} \left[  |\|v*a*x\|_1- {\mathbb E} \|v*a*x\|_1 | > t, v \in B_k  \right]   \\ 
\leq&  (3\tau \sqrt{n}np \|a\|_1)^k {\mathbb P} \left[  |\|h*x\|_1- {\mathbb E} \|h*x\|_1 | > t   \right] 
\end{array}
  \] 
\end{lemma}
\begin{proof}
Let the deviation be
\[
\tilde{\mathcal{X}}_v =   \sum_{i=1}^n \left( | \langle C(a^{(i)}) v, x\rangle| - {\mathbb E} | \langle C(a^{(i)}) v, x\rangle|\right).
\]  
We will choose $1/\epsilon$ to be the Lipschitz constant of  $\tilde{\mathcal{X}}_v$ as a function of $v$. Let's compute the Lipschitz constant.

 
 

We introduce additional independent random sign $s_j$ , which are IID sampled from Rademacher variable by defining 
\[
\mathcal{X}_v =   \sum_{i=1}^n s_j \left( |\langle C(a^{(i)}) v, x\rangle| \right),
\]  
 then by lemma $24$ of \cite{blasiok2016improved}, for every integer $p$ we have 
 \[
 \| \sup_{v\in B_k} |\tilde{\mathcal{X}}_v | \|_p \lesssim \| \sup_{v\in B_k}  |\mathcal{X}_v| \|_p
 \]
 This is called symmetrization trick. Using this trick, we only need to analyze the tail behavior of the random variable $\mathcal{X}_v$. To obtain tail bounds for maximum  of  $\sup_{v\in B_k}  |\mathcal{X}_v|$, we need to bound tails of the random variables $\mathcal{X}_v-\mathcal{X}_{v'}$ for $v, v'\in B_k$.
 
Since  
\[
\mathcal{X}_v-\mathcal{X}_{v'} =  
\sum_{i=1}^n s_i \left( | \langle C(a^{(i)}) v, x\rangle| -   | \langle C(a^{(i)}) v', x\rangle|\right).
\]

Then use the fact that 
\[
| \langle C(a^{(i)}) v, x\rangle| -   | \langle C(a^{(i)}) v', x\rangle| \leq   \left( | \langle C(a^{(i)}) (v-v'), x\rangle|  \right),
\]
and  both $s_i | \langle C(a^{(i)}) v, x\rangle| -   | \langle C(a^{(i)}) v', x\rangle| $ and  
$s_i \left( | \langle C(a^{(i)}) (v-v'), x\rangle|  \right)$ are symmetric, if we have a tail bound for 
$\sum_{i=1}^n s_i \left( | \langle C(a^{(i)}) (v-v'), x\rangle|  \right)$ , 
we also have tail bounds on $\mathcal{X}_v-\mathcal{X}_{v'}$.

From these calculation, we saw that the Lipschitz constant of  $\tilde{\mathcal{X}}_v$ is bounded by 
 \[
  \begin{array}{ll}
\sqrt{\sum_{j=1}^n (\sum_{i=1}^n s_i |\sum_{t=1}^n a^{i-j+t} x_t|)^2}
&\leq   \sqrt{n (\|a\|_1 \|x\|_1)^2}\\
&\leq  \tau \sqrt{n}np \|a\|_1
\end{array}
  \]  
  
Therefore, we can choose $1/\epsilon = \tau \sqrt{n}np \|a\|_1$, then
at most $|N| = (3\tau \sqrt{n}np \|a\|_1)^k$ balls can cover $B_k$.
We have a concentration inequality in each ball,
We define the tail event 
\[
\mathcal{E}_v = \{|\tilde{\mathcal{X}}_v | > t\}
\]
we can take a union bound,
 \[
  \begin{array}{ll}
 {\mathbb P} \left(\cup_{v \in B_k} \mathcal{E}^c_v \right) &\leq |N| {\mathbb P} \left[  |\|h*x\|_1- {\mathbb E} \|h*x\|_1 | > t   \right] \\
&\leq   (3\tau \sqrt{n}np \|a\|_1)^k {\mathbb P} \left[  |\|h*x\|_1- {\mathbb E} \|h*x\|_1 | > t   \right] 
\end{array}
  \] 
\end{proof}
\section{Finite sample analysis}
\paragraph{Finite sample setting}
\begin{itemize}
\item
Let $(X_t)_{t \in {\mathbb Z}}$ be IID sampled from  $pN(0,1) + (1-p) \delta_0$.
 \item
The filter $a\in l_1({\mathbb Z}), (\mathcal{F} a)(\omega) \neq 0, \forall \omega\in T.$ Additionally, in finite sample setting, we assume $a^{-1}$ is zero outside of a centered window of radius $2k+1$, namely, the non-zero part can be written as $(\beta_{-k}, \ldots, \beta_0, \ldots, \beta_k)\in V$. 

We fix the shift and scaling effect by assuming $v^T a^{-1} = 1$ for a generic $v$ that has one single largest entry. 
\item
Let $Y= a* X$ be a linear process, we are given a series of observation $(Y_t)_{t\in [-(T+k),T+k]}$ from a centered window of radius $T+k$.
\item
 We denote $X^{(i)}$ as the sequence being flipped and shifted for $i$ step from $X$ s that $X^{(i)}_k = X_{-k +i}$. Let $J_N$ be the support of $X$.
 We define the length $N=2T+1$ window as ${\cal T} := [-T, T].$
\end{itemize}
Now we want to find an inverse filter 
$w = (w_{-k}, \ldots, w_0, \ldots, w_k) \in V$ such that the convolution $w*Y$ is sparse,
\begin{equation}
 \begin{array}{ll}
 \underset{w\in V}{  \mbox{minimize} }   & \frac{1}{N}\|w*Y\|_{1,{\cal T}}\\
 \mbox{subject to}  
& v^T w =1.
\end{array}
\tag{$P_{v}^T$}
\end{equation}




First, after change of variable,
\[
 \frac{1}{N}\|w*Y\|_{1,{\cal T}} = \frac{1}{N}\|\psi*X\|_{1,{\cal T}}
\]
Then the empirical average can be represented as sum of shifted and flipped inner products.
\[
\frac{1}{N}\|\psi*X\|_{1,{\cal T}}  = \frac{1}{N}\sum_{|i|\in {\cal T}} |[\psi*X]_i| = \frac{1}{N}\sum_{|i|\in {\cal T}}|\psi^T X^{(i)}|
\]
We define 
\[
p_N = \frac{1}{N}\sum_{i\in {\cal T}} I_i
\]
The population average is being calculated,  as
\[
{\mathbb E}  \|\psi*X \|_1:= {\mathbb E}|C_i(\psi)^T X| = {\mathbb E}|\tilde{\psi}^T X|
\]
We know the mean of $p_N$ is $p$, the variance of $p_N$ as
\[
{\mathbb E}[p_N-p]^2=\frac{1}{N}p(1-p)
\]






 
 
 
 \paragraph{Finite sample directional derivative}
 \begin{eqnarray*}
& & \lim_{t\rightarrow 0^+}\frac{1}{t}[(\hat{{\mathbb E}}  \| (\psi+t\beta)*X \|_1 - \hat{{\mathbb E}}  \| \psi*X \|_1)
\\
  &=& 
 \langle  \beta, {\frac{1}{N}\sum_{|i|\in {\cal T}} \nabla_{\psi}L_i}|_{\psi} \rangle \\
 &=& 
 \frac{1}{N}\sum_{|i|\in {\cal T}}  \mbox{sign}(\psi^T X^{(i)})X^{(i),T}\beta
 \end{eqnarray*}
Now to study the finite sample convex problem, we need to calculate the directional derivative at $ \psi = e_0$.
Then
\begin{theorem}
For any $X$ sequence with support  $J_N$,
\begin{eqnarray*}
& & \lim_{t\rightarrow 0^+}\frac{1}{t}[(\hat{{\mathbb E}}  \| (e_0+t\beta)*X \|_1 - \hat{{\mathbb E}}  \| e_0*X \|_1)
\\
  &=& 
 \langle  \beta, {\frac{1}{N}\sum_{|i|\in {\cal T}} \nabla_{\psi}L_i}|_{e_0} \rangle \\
 &=& 
 \frac{1}{N}\sum_{|i|\in {\cal T}}  \mbox{sign}( X_{-i})X^{(i),T}\beta
 \\
  &=& 
 \frac{1}{N} \left(\|X\|_{1,J_N} \beta_0  +
 \|X*\beta_{(0)}\|_{1,{\cal T}-J_N}+ 
\sum_{i\in J_N} (1_{X_{-i}> 0}-1_{X_{-i}< 0})  \cdot X^{(i),T}\beta_{(0)} \right) 
 \end{eqnarray*}
 
\end{theorem}
\begin{proof}
 \begin{eqnarray*}
& & \lim_{t\rightarrow 0^+}\frac{1}{t}[(\hat{{\mathbb E}}  \| (e_0+t\beta)*X \|_1 - \hat{{\mathbb E}}  \| e_0*X \|_1)
\\
  &=& 
 \langle  \beta, {\frac{1}{N}\sum_{|i|\in {\cal T}} \nabla_{\psi}L_i}|_{e_0} \rangle \\
 &=& 
 \frac{1}{N}\sum_{|i|\in {\cal T}}  \mbox{sign}( X_0^{(i)})X^{(i),T}\beta
 \\
 &=& 
 \frac{1}{N}\sum_{|i|\in {\cal T}}  \mbox{sign}( X_{-i})X^{(i),T}\beta
 \\
 &=& 
\frac{1}{N}\sum_{|i|\in {\cal T}} 1_{X_{-i}> 0}  \cdot X^{(i),T}\beta  +
\frac{1}{N}\sum_{|i|\in {\cal T}} 1_{X_{-i}< 0}  \cdot (-X^{(i),T}\beta)  +
\frac{1}{N}\sum_{|i|\in {\cal T}} 1_{X_{-i}= 0} \cdot |X^{(i),T}\beta|
 \\
 &=& 
\frac{1}{N}\sum_{|i|\in {\cal T}} 1_{X_{-i}> 0}  \cdot X^{(i),T}\beta  +
\frac{1}{N}\sum_{|i|\in {\cal T}} 1_{X_{-i}< 0}  \cdot (-X^{(i),T}\beta)  +
\frac{1}{N}\sum_{|i|\in {\cal T}}  |X^{(i),T}\beta_{(0)}|
 \\
 &=& 
\frac{1}{N}\sum_{|i|\in {\cal T}} 1_{X_{-i}> 0}  \cdot X_{-i}\beta_0  +
\frac{1}{N}\sum_{|i|\in {\cal T}} 1_{X_{-i}< 0}  \cdot (-X_{-i}\beta_0)  + 
\frac{1}{N}  \|X*\beta_{(0)}\|_{1,{\cal T}-J_N}+ 
\\
& &
\frac{1}{N}\sum_{|i|\in {\cal T}} 1_{X_{-i}> 0}  \cdot X^{(i),T}\beta_{(0)}  +
\frac{1}{N}\sum_{|i|\in {\cal T}} 1_{X_{-i}< 0}  \cdot (-X^{(i),T}\beta_{(0)})  
 \\
  &=& 
\frac{1}{N}\sum_{|i|\in {\cal T}} 1_{X_{-i}\neq 0}  \cdot |X_{-i}|\beta_0  +
\frac{1}{N}  \|X*\beta_{(0)}\|_{1,{\cal T}-J_N}+ 
\frac{1}{N}\sum_{i\in J_N} (1_{X_{-i}> 0}-1_{X_{-i}< 0})  \cdot X^{(i),T}\beta_{(0)}   
 \\
  &=& 
 \frac{1}{N} \left(\|X\|_{1,J_N} \beta_0  +
 \|X*\beta_{(0)}\|_{1,{\cal T}-J_N}+ 
\sum_{i\in J_N} (1_{X_{-i}> 0}-1_{X_{-i}< 0})  \cdot X^{(i),T}\beta_{(0)} \right) 
 \end{eqnarray*}
\end{proof}
\begin{coro}
For Gaussian IID sequence $X$, 
 the random variable $ \frac{\|X\|_{1,J_N}}{N} = p_N   ( \frac{\|X\|_{1,J_N}}{|J_N|})$ has mean $p\sqrt{\frac{2}{\pi}}$. 
 And the random expression
\begin{eqnarray*}
& &  \frac{1}{N} \left(\|X\|_{1,J_N} \beta_0  +
 \|X*\beta_{(0)}\|_{1,{\cal T}-J_N} \right) 
\\
  &=& 
p_N   ( \frac{\|X\|_{1,J_N}}{|J_N|}) \beta_0  +
(1-p_N ) \frac{1}{|{\cal T}-J_N|}  \|X*\beta_{(0)}\|_{1,{\cal T}-J_N}
 \end{eqnarray*}
 has mean $\sqrt{\frac{2}{\pi}}(p \cdot \beta_0 + (1-p) \cdot {\mathbb E}_{I}[ \|\beta_{I}\|_2 | 0 \not \in I ])$.
\end{coro}
\begin{proof}
 \begin{eqnarray*}
& & 
\frac{1}{N} \left(\|X\|_{1,J_N} \beta_0  +
 \|X*\beta_{(0)}\|_{1,{\cal T}-J_N} \right) 
 \\
  &=& 
p_N ( \frac{\|X\|_{1,J_N}}{|J_N|})\beta_0  +
(1-p_N ) \frac{1}{|{\cal T}-J_N|}  \|X*\beta_{(0)}\|_{1,{\cal T}-J_N}
 \end{eqnarray*}
where $p_N = \frac{|J_N|}{N} = \frac{1}{N}\sum_{|i|\in {\cal T}}  1_{X_{-i}\neq 0}$ has mean $p$ and variance $\frac{1}{N} p(1-p)$.
And notice that for the Gaussian IID case 
\[
{\mathbb E} [ p_N\frac{\|X\|_{1,J_N}}{|J_N|}] = {\mathbb E}[p_N|X_{0}|\mid X_{0}\neq 0]  = p\sqrt{\frac{2}{\pi}},
\]
\[
{\mathbb E} [(1-p_N ) \frac{1}{|{\cal T}-J_N|}  \|X*\beta_{(0)}\|_{1,{\cal T}-J_N} ] = \sqrt{\frac{2}{\pi}}{\mathbb E}_{I}[ \|\beta_{I}\|_2 | 0 \not \in I ]).
\]
\end{proof}

\begin{coro}
 For symmetric IID sequence $X$, 
  \[
  \frac{1}{N}\sum_{i\in J_N} (1_{X_{-i}> 0}-1_{X_{-i}< 0})  \cdot X^{(i),T}\beta_{(0)}  = \frac{1}{N}\sum_{i\in J_N} (1_{X_{-i}> 0}-1_{X_{-i}< 0})  \cdot (\sum_{j\in J_N-\{i\} } X_j \beta_{i-j} ) 
  \]
  where 
  $\sum_{j\in J_N-\{i\} } X_j \beta_{i-j} $ is independent of $X_i$, therefore independent of $(1_{X_{-i}> 0}-1_{X_{-i}< 0}) $.
  Specifically, it has mean $0$
   \[
 {\mathbb E} \frac{1}{N}\sum_{i\in J_N} (1_{X_{-i}> 0}-1_{X_{-i}< 0})  \cdot X^{(i),T}\beta_{(0)}  = 0
  \]
  for $q\leq 2$,
    \[
  \frac{1}{N}{\mathbb E}|\sum_{i\in J_N} (1_{X_{-i}> 0}-1_{X_{-i}< 0})  \cdot X^{(i),T}\beta_{(0)} |^q \leq 
  \frac{1}{N} {\mathbb E}[]|\sum_{i\in J_N}|1_{X_{-i}> 0}-1_{X_{-i}< 0}||^q\cdot  \max_j |X^{(j),T}\beta_{(0)} |^q]
  \]
  
\end{coro}



 \paragraph{Concerntration of $\|\psi*X\|_{1}$ }
The next step is to provide an uniform bound of the finite sample derivation of $\|X\|_{1,J_N}$ and 
$\|X*\beta_{(0)}\|_{1,{\cal T}-J_N}$.

In general, for any random subset $I$ of ${\cal T}$, we study the uniform concentration of 
\[
\|\psi*X\|_{1,I} 
\]
for $\psi = a*w \in a*B_1(V_k),$ by providing moments and tail bound on
\[
U_I:= \sup_{\psi \in a*B_1(V_k)}
| \|\psi*X\|_{1,I}    - {\mathbb E}  \|\psi*X\|_{1,I}  |
\]

For clarity of the argument, we only state the concentration for $\|\psi*X\|_{1,{\cal T}}.$ Also the result about ${\cal T}$ would easy carry to a random subset $I$.

Let the minimal and maximal eigenvalue of the (convolutional) linear operator $a*$ be
\[
\lambda_{m}(a) := \inf_{y: \|y\|_2\leq 1} \frac{\|a*y\|_2}{\|y\|_2}
\]
\[
\lambda_{M}(a) := \sup_{y: \|y\|_2\leq 1} \frac{\|a*y\|_2}{\|y\|_2}
\]
\begin{lemma}
Using circluar embedding of Topelitz matrix, 
\[
\lambda_{M}(a) \leq \max| \hat{a}|
\]
\end{lemma}

Following the idea of lemma~$4$ of the paper of Adamczak on sparse dictionary learning\cite{adamczak2016note} and lemma~$16$ of the paper \cite{spielman2012exact}, we derive a similar bound for convolutional case.
\begin{lemma}
In the following, we define 
\[
Z :=  \frac{1}{N}\|w*a*X\|_{1,{\cal T}}
\]
We find the lower bound of ${\mathbb E} Z$ as a function of the $1-$norm  of $w$:
\[
{\mathbb E} Z  \geq \frac{1}{\sqrt{2}\pi}\lambda_{m}(a)\sqrt{\frac{p}{k}} \|w\|_{1} 
\]
For convenience, we define
\[
\mu_{\mbox{min}}:=\frac{1}{\sqrt{2}\pi}\lambda_{m}(a)\sqrt{\frac{p}{k}} \|w\|_{1}
\]

\end{lemma}
\begin{proof}
WOLG, we rescale $w$ so that $\|w\|_1 =1$, now
\begin{eqnarray*}
{\mathbb E}_X Z 
&=& \frac{1}{N}\sum_{j\in [-T,T] }{\mathbb E}_X |\sum_{i=1}^k w_i (a*X)_{j-i}|\\
&\geq& \inf_{\|w\|_{1} = 1}\frac{1}{N}\sum_{j\in [-T,T] }{\mathbb E}_X |\sum_{i=1}^k w_i (a*X)_{j-i}|
\end{eqnarray*}
By symmetry, we know the minimizer is $w = (\frac{1}{k}, \ldots,\frac{1}{k}),$
then we use the symmetrization trick to insert a sequence of $k$ IID Rademacher ($\pm 1$)  random variables 
$\varepsilon_1, \ldots, \varepsilon_k$ in the second inequality, and then apply Khintchine inequality as the third inequality,
\begin{eqnarray*}
{\mathbb E}_X Z 
&\geq&  \frac{1}{k} \frac{1}{N}\sum_{j\in [-T,T] }{\mathbb E}_X |\sum_{i=1}^k (a*X)_{j-i}|\\
&\geq& \frac{1}{2} \frac{1}{k} \frac{1}{N}\sum_{j\in [-T,T] }{\mathbb E}_X |\sum_{i=1}^k \varepsilon_i (a*X)_{j-i}|\\
&\geq&  \frac{1}{2} \frac{1}{k} \frac{1}{N}\sum_{j\in [-T,T] }\frac{1}{\sqrt{2}}{\mathbb E}_X \sqrt{ \sum_{i=1}^k  (a*X)^2_{j-i}}\\
&\geq&  \frac{1}{2} \frac{1}{k} \frac{1}{N}\sum_{j\in [-T,T] }\lambda_{m}(a)\frac{1}{\sqrt{2}}{\mathbb E}_X \sqrt{ \sum_{i=1}^k  X^2_{j-i}}\\
&\geq& \frac{1}{2}  \frac{1}{k} \frac{1}{\sqrt{2}}\frac{1}{N}\sum_{j\in [-T,T] }\lambda_{m}(a)\frac{2}{\pi}\sqrt{pk}\\
&=&  \frac{1}{\sqrt{2}\pi}\lambda_{m}(a)\sqrt{\frac{p}{k}}\\
\end{eqnarray*}
\end{proof}


Now we define the intersection of $1-$norm ball of the $k-$dimensional subspace $V_k$ as $B_1(V_k)$, then we consider $w\in B_1(V_k).$ 
\begin{lemma}
Hence we consider $\psi = a*w \in a*B_1(V_k),$ and we define
\[
W:= \sup_{\psi \in a*B_1(V_k)}
|\frac{1}{N}\sum_{j\in [-T,T] } (|(\psi*X)_{j}| - {\mathbb E} |(\psi*X)_{j}|) |
\]

For all $q\geq \max(2, \log(k)),$ there exists constant $C$,
\[
{\mathbb E} W^q\leq 2^{2q}(\frac{\lambda_{M}(a)}{N})^q   k C^q(\sqrt{Npq}+q)^q
\]
so
\begin{eqnarray*}
 \|W\|_q \leq \frac{4eC}{N} \lambda_{M}(a)  
 (\sqrt{Np(q+\log(k))}+q +\log(k))
\end{eqnarray*}
and
\begin{eqnarray*}
P( W > \frac{4eC\lambda_{M}(a) }{N}  
 (\sqrt{Np\log(\frac{k}{\delta})}+\log(\frac{k}{\delta})))\leq \delta
\end{eqnarray*}
\end{lemma}

\begin{proof}
Let $\varepsilon_1, \ldots, \varepsilon_N$ be a sequence of IID Rademacher ($\pm 1$)  random variables indepedent of $X$,  By the symmetrization inequality, see e.g. lemma~$6.3$ of book \cite{ledoux2013probability}, we have the first inequality. Then since the function $t \rightarrow |t| $ is a contraction, an application of Talagrand’s contraction principle (see lemma~$8$ of \cite{adamczak2016note}) with $F (x) = |x|^q$ 
conditionally on $X$ gives the second inequality.
\begin{eqnarray*}
{\mathbb E} W^q 
&\leq& 2^q {\mathbb E} \sup_{\psi \in a*B_1(V_k)} 
|\frac{1}{N}\sum_{j\in [-T,T] } \varepsilon_j |(\psi*X)_{j}| |^q\\
&\leq&
2^{2q}{\mathbb E} \sup_{\psi \in a*B_1(V_k)} |\frac{1}{N}\sum_{j\in [-T,T] } \varepsilon_j (\psi*X)_{j} |^q\\
&=&
2^{2q}\frac{1}{N^q}{\mathbb E} \sup_{\psi \in a*B_1(V_k)} |\sum_{i\in [-T,T] }\psi_{i}  \sum_{j\in [-T,T] } \varepsilon_j X_{j-i} |^q\\
&=&
2^{2q}\frac{1}{N^q}{\mathbb E} \sup_{w \in B_1(V_k)} |\sum_{i\in [-T,T] }(w*a)_{i}  \sum_{j\in [-T,T] } \varepsilon_j X_{j-i} |^q\\
&\leq&
2^{2q}\frac{1}{N^q}{\mathbb E} \max_{s\in [k]}|\sum_{i\in [-T,T] }a_{i-s}( \sum_{j\in [-T,T] } \varepsilon_j X_{j-i}) |^q\\
&\leq&
2^{2q}(\frac{\lambda_{M}(a)}{N})^q   \sum_{s\in [k] }{\mathbb E}| \sum_{j\in [-T,T] } \varepsilon_j X_{j-s} |^q
\end{eqnarray*}
Now by the moment version of Bernstein’s inequality (see lemma~$7$, equation $(18)$ of\cite{adamczak2016note}), we know that there exists a universal constant $C$,
\begin{eqnarray*}
{\mathbb E}| \sum_{j\in [-T,T] } \varepsilon_j X_{j-i} |^q \leq C^q(\sqrt{Npq}+q)^q
\end{eqnarray*}
Therefore, 
\begin{eqnarray*}
{\mathbb E} W^q &\leq& 2^{2q}(\frac{\lambda_{M}(a)}{N})^q    
\sum_{s\in [k] }{\mathbb E}| \sum_{j\in [-T,T] } \varepsilon_j X_{j-s} |^q \\
&\leq& 2^{2q}(\frac{\lambda_{M}(a)}{N})^q   k C^q(\sqrt{Npq}+q)^q
\end{eqnarray*}
Then
\begin{eqnarray*}
 \|W\|_q \leq \frac{4eC}{N} \lambda_{M}(a)  
 (\sqrt{Np(q+\log(k))}+q +\log(k))
\end{eqnarray*}
therefore, we could get the tail bound using the Chebyshev inequality for the moments,
\begin{eqnarray*}
P( W > \frac{4eC\lambda_{M}(a) }{N}  
 (\sqrt{Np(q+\log(k))}+q +\log(k)))\leq e^{-q}
\end{eqnarray*}
We set $q = \log(\frac{1}{\delta})$, then we will get an upper bound for $W$ which is satisfied with probability at least $1-\delta$:
\begin{eqnarray*}
P( W > \frac{4eC\lambda_{M}(a) }{N}  
 (\sqrt{Np\log(\frac{k}{\delta})}+\log(\frac{k}{\delta})))\leq \delta
\end{eqnarray*}
\end{proof}

\begin{lemma}
Let the condition number of the linear operator $a*$ be
\[
\kappa_a := \frac{\lambda_M(a)}{\lambda_m(a)}.
\]
If
\[
N\geq k\log(\frac{k}{\delta}) (2^7 \pi e^2 \frac{C^2}{\epsilon^2} \kappa_a^2 )
\]
and  $p\geq \frac{1}{N}$, 
then
\begin{eqnarray*}
P( W > \epsilon \mu_{\mbox{min}})\leq \delta
\end{eqnarray*}
\end{lemma}
\begin{proof}
If
\[
N\geq k\log(\frac{k}{\delta}) (2^7 \pi e^2 \frac{C^2}{\epsilon^2} \kappa_a^2 )
\]
then 
\[
\frac{4eC\lambda_{M}(a) }{N}  
 (\sqrt{Np\log(\frac{k}{\delta})} \leq \frac{1}{2}\epsilon (\frac{1}{\sqrt{2}\pi}\lambda_{m}(a)\sqrt{\frac{p}{k}}) \leq \frac{1}{2}\epsilon \mu_{\mbox{min}}
\]
additionally, if $p\geq \frac{1}{N}$, 
then a weaker inequality is also true:
\[
\frac{4eC\lambda_{M}(a) }{N}  
 \log(\frac{k}{\delta}) \leq \frac{1}{2}\epsilon (\frac{1}{\sqrt{2}\pi}\lambda_{m}(a)\sqrt{\frac{p}{k}}) \leq \frac{1}{2}\epsilon \mu_{\mbox{min}}
\]
therefore, combining the two parts and the previous inequality,
\begin{eqnarray*}
P( W > \frac{4eC\lambda_{M}(a) }{N}  
 (\sqrt{Np\log(\frac{k}{\delta})}+\log(\frac{k}{\delta})))\leq \delta
\end{eqnarray*}
we have
\begin{eqnarray*}
P( W > \epsilon \mu_{\mbox{min}})\leq \delta
\end{eqnarray*}
\end{proof}


\begin{theorem}
Let the solution of finite sample convex optimization be $w^\star$, 
for small $\epsilon>0$ and concentration level $\delta>0$, 
there exists universal constant $C$, if there are 
\[
N \geq k\log(\frac{k}{\delta}) ( \frac{C\kappa_a}{\epsilon}  )^2
\]
and  $ \frac{1}{N}\leq p< p^\star(\epsilon)$, we have 
\begin{eqnarray*}
P( W > \epsilon \mu_{\mbox{min}})\leq \delta.
\end{eqnarray*}

  \end{theorem}
\begin{proof}
From previous lemma, we know that with the conditions, 
\begin{eqnarray*}
P( W > \epsilon \mu_{\mbox{min}})\leq \delta.
\end{eqnarray*}
Therefore, with probability at least $1-\delta$,
\begin{eqnarray*}
W = \sup_{\psi \in a*B_1(V_k)}
|\frac{1}{N}\sum_{j\in [-T,T] } (|(\psi*X)_{j}| - {\mathbb E} |(\psi*X)_{j}|) | \leq \epsilon \mu_{\mbox{min}}
\end{eqnarray*}
When this is true, for all $\psi \in a*B_1(V_k)$, we have a uniform bound 
\[
 {\mathbb E} |(\psi*X)_{j}|-\epsilon\mu_{\mbox{min}} \leq \frac{1}{N}\sum_{j\in [-T,T] } (|(\psi*X)_{j}| \leq {\mathbb E} |(\psi*X)_{j}|+\epsilon\mu_{\mbox{min}}
\]
\end{proof}


\begin{coro}
For $\psi = a*w \in a*B_1(V_k),$ 
\[
U_I:= \sup_{\psi \in a*B_1(V_k)}
| \|\psi*X\|_{1,I}    - {\mathbb E}  \|\psi*X\|_{1,I}  |
\]
For all $q\geq \max(2, \log(k)),$ there exists constant $C$,
\[
{\mathbb E} U_I^q\leq 2^{2q}(\lambda_{M}(a))^q   k C^q(\sqrt{Npq}+q)^q
\]
so
\begin{eqnarray*}
 \|U_I\|_q \leq 4eC \lambda_{M}(a)  
 (\sqrt{Np(q+\log(k))}+q +\log(k))
\end{eqnarray*}
and
\begin{eqnarray*}
P( U_I > 4eC\lambda_{M}(a)  
 (\sqrt{Np\log(\frac{k}{\delta})}+\log(\frac{k}{\delta})))\leq \delta
\end{eqnarray*}
\end{coro}

\paragraph{Uniform bound of directional derivative}
We  define the uniform bound of directional derivative
\[
W_D  =\sup_{\|\beta\|_2=1, u^T \beta =0, \beta +e_0 \in a*B_1(V_k)}|\langle  \beta, {\frac{1}{N}\sum_{|i|\in {\cal T}} \nabla_{\phi}L_i}|_{\phi = 0} \rangle  -   \langle  \beta, {\mathbb E} \nabla_{\phi}L|_{\phi = 0} \rangle  | 
\]
Notice that
\begin{eqnarray*}
& &\langle  \beta, {\frac{1}{N}\sum_{|i|\in {\cal T}} \nabla_{\phi}L_i}|_{\phi = 0} \rangle  -   \langle  \beta, {\mathbb E} \nabla_{\phi}L|_{\phi = 0} \rangle  
\\
&=& 
\left[\frac{1}{N} \left(\|X\|_{1,J_N} \beta_0  +
 \|X*\beta_{(0)}\|_{1,{\cal T}-J_N}+ 
\sum_{i\in J_N} (1_{X_{-i}> 0}-1_{X_{-i}< 0})  \cdot X^{(i),T}\beta_{(0)}\right)\right]-
\\
& &
\left[\sqrt{\frac{2}{\pi}}(p \cdot \beta_0 + (1-p) \cdot {\mathbb E}_{I}[ \|\beta_{I}\|_2 | 0 \not \in I ])\right]
\end{eqnarray*}
From previous uniform bound, with high probability, we have the following three uniform bounds on
\[
|\frac{1}{N}\sum_{i\in J_N} (1_{X_{-i}> 0}-1_{X_{-i}< 0})  \cdot X^{(i),T}\beta_{(0)}|
\]
\[
|(\frac{1}{N} \|X\|_{1,J_N}-\sqrt{\frac{2}{\pi}}p)|
\]
\[
|\frac{1}{N}\|X*\beta_{(0)}\|_{1,{\cal T}-J_N}-(1-p) \cdot {\mathbb E}_{I}[ \|\beta_{I}\|_2 | 0 \not \in I ]|
\]
If $p<p^\star(u, \epsilon)$, we know from population result that
\[
\langle  \beta, {\mathbb E} \nabla_{\phi}L|_{\phi = 0} \rangle >\epsilon.
\]
If with high probability,  
\[
W_D  \leq \epsilon/2
\]
then using the convexity argument, 
for any direction $\beta \in V,$ the directional derivative at $\phi = 0$
\[
\langle  \beta, {\frac{1}{N}\sum_{|i|\in {\cal T}} \nabla_{\phi}L_i}|_{\phi = 0} \rangle  >0,
\]
then $a*w^\star-e_0=0$. 





















 








                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           


 
 

 


  







\subsection{$E{| \psi^T X| }$ for IID Bernoulli Gaussian $X$ }
we study in detail for the quantity  
\[
E{| \psi^T X| } = E{|\sum_{i} \psi_i X_i| }.
\]
We know when $X_i$ are unit variance,
\[
   E (\sum_i \psi_i X_i)^2 =  \sum_i   \psi_i^2 E(X_i)^2 = \|a\|_2^2.
\]
The ratio indicate the sparsity level of random variable $Z = \sum_i \psi_i X_i$
\[
\frac{ E{|\sum_{i} \psi_i X_i| }}{\sqrt{E (\sum_i \psi_i X_i)^2}} = \frac{ E{|\sum_{i} \psi_i X_i| }}{\|a\|_2} 
\]
Now, we consider some settings where exact calculation of $E{|\sum_{i} \psi_i X_i| }$ are possible.

\paragraph{Exact calculation for Bernoulli Gaussian}
Let $X_i = B_i G_i$ be IID Bernoulli Gaussian, where $B_i$ is sampled from Bernoulli with parameter $p$, $G_i$ is sampled from $N(0,1)$. Let $I$ be the support where $B_i =1$.
\[
E_X{|\sum_{i} \psi_i X_i| }  
= E_B E_G {|\sum_{i} \psi_i B_i G_i| }
= E_I E_G {|\sum_{i\in I} \psi_i G_i| }
\]
Leveraging the invariance property for linear transform of Gaussian, we get
\[
E_G {|\sum_{i\in I} \psi_i G_i| } = 
\sqrt{\frac{2}{\pi}} \cdot\|\psi_I\|_2.
\]
then
\[
E_X{|\sum_{i} \psi_i X_i| }  
= \sqrt{\frac{2}{\pi}} \cdot E_I \|\psi_I\|_2
= \sqrt{\frac{2}{\pi}} \cdot (1-(1-p)^{\|a\|_0})^{1/2}   \cdot  \|a\|_2.
\]
Here the last equality use the lemma below, and we define that $\|a\|_0= \#\{ t : \psi_t \neq 0\}$.
\begin{lemma} (Expectation of $2-$norm over Bernoulli support)
\label{lem:BLemma}
\begin{equation}
E_I \|\psi_I\|_2 = (1-(1-p)^{\|a\|_0})^{1/2}   \cdot  \|a\|_2
\end{equation}
\end{lemma}
\begin{proof}[Proof of Lemma \ref{lem:BLemma}]
We have the power series representation $\sqrt{x} = \sum_k c_k x^k$ valid on $(1/A,A)$
for large $A \gg 1$. 
Let $N = |{\cal T}|$. Fix $A > 1/{\varepsilon}^2$; then every sum $ \sum_{t \in I} \psi_t^2 \in (1/A,A)$;
using this:
\begin{eqnarray*}
 \sum_{I \subset {\cal T}} \pi(I)   ( \sum_{t \in I} \psi_t^2)^{1/2}  &=&  
 \sum_{I \subset {\cal T}} \pi(I)   \sum_k  c_k \cdot ( \sum_{t \in I} \psi_t^2  )^k \\
 &=& \sum_{I \subset {\cal T}} \pi(I)   \sum_k  c_k \sum_{t_1,\dots, t_k \in I} \prod_{i=1}^k \psi_{t_i}^2  \\
 &=&   \sum_k  c_k \sum_{I \subset {\cal T}} \pi(I)   \sum_{t_1,\dots, t_k \in I} \prod_{i=1}^k \psi_{t_i}^2  \\
 \end{eqnarray*}
 Hence from $ \pi(I) = (p)^{|I|} (1-p)^{|{\cal T} \Delta I|}$
 \begin{eqnarray*}
  \sum_{I \subset {\cal T}} \pi(I)   ( \sum_{t \in I} \psi_t^2)^{1/2}  
 &=&  \sum_k c_k   \sum_{n=1}^{N}  \sum_{|I| = n} (p)^n (1-p)^{N-n}  \cdot  \sum_{t_1,\dots, t_k \in I} \prod_{i=1}^k \psi_{t_i}^2  \\
&=&  \sum_k c_k   \sum_{n=1}^{N}  (p)^n (1-p)^{N-n}  \cdot \sum_{|I| = n} \sum_{t_1,\dots, t_k \in I} \prod_{i=1}^k \psi_{t_i}^2 
 \end{eqnarray*}
 Since the inner two sums  cover $ {N \choose n} n^k$ terms in a uniform way,
 we can write
 \begin{eqnarray*}
  \sum_{I \subset {\cal T}} \pi(I)   ( \sum_{t \in I} \psi_t^2)^{1/2}  
&=&  \sum_k c_k   \sum_{n=1}^{N}  (p)^n (1-p)^{N-n}  \cdot \sum_{|I| = n} \sum_{t_1,\dots, t_k \in I} \prod_{i=1}^k \psi_{t_i}^2  \\
 &=&  \sum_k c_k   \sum_{n=1}^{N}  (p)^n (1-p)^{N-n}  \cdot {N \choose n}  \cdot Ave_{|I| = n}  \cdot {n^k} \cdot  Ave_{t_1,\dots, t_k \in I} \; \prod_{i=1}^k \psi_{t_i}^2  \\
&=&  \sum_k c_k   \sum_{n=1}^{N}  (p)^n (1-p)^{N-n}  \cdot {N \choose n}  \cdot {n^k} \cdot Ave_{t_1,\dots, t_k \in {\cal T}} \; \prod_{i=1}^k \psi_{t_i}^2  \\
 &=&  \sum_k c_k   \cdot \left( \sum_{n=1}^{N}  (p)^n (1-p)^{N-n}  \cdot {N \choose n} \right)  \cdot ( \sum_{t \in {\cal T}} \psi_t^2  )^k \\
 \end{eqnarray*}
Now use the identity $1 = \sum_{n=0}^{N}  (p)^n (1-p)^{N-n}  \cdot {N \choose n}$  for the complete binomial sum, 
hence the incomplete sum $\sum_{n=1}^{N}  (p)^n (1-p)^{N-n}  \cdot {N \choose n}= 1-(1-p)^N$. We get
 \begin{eqnarray*}
  \sum_{I \subset {\cal T}} \pi(I)   ( \sum_{t \in I} \psi_t^2)^{1/2}  
 &=&  \sum_k c_k   \cdot \left( 1-(1-p)^N \right)  \cdot ( \sum_{t \in {\cal T}} \psi_t^2  )^k \\
 &=& \left( 1-(1-p)^N \right) \cdot \sum_k c_k  \left ( \sum_{t \in {\cal T}} \psi_t^2  \right )^k \\
 &=&  \left( 1-(1-p)^N \right) \cdot  \sqrt{\sum_{t \in {\cal T}} \psi_t^2 } .
\end{eqnarray*}
\end{proof}
\section{Discrete signal structure}
\paragraph{Random model for discrete digital signals.}
We define a random model for discrete digital signals. 

Let $(X_t)$ be an independent
sequence of random variable indexed by  $t \in Z$, with a symmetric distribution with CDF $F$. 
\begin{itemize}
\item The two point symmetric distribution on real number, $F = 1/2 \delta_{+1} + 1/2 \delta_{-1}.$ 
\item The four point symmetric distribution on complex number, 
$F = 1/4 \delta_{1+i} + 1/4 \delta_{-1+i}+1/4 \delta_{1-i}
+1/4 \delta_{-1-i}.$ 
\item A symmetric distribution on ${\mathbb R}^s$,
$F = 1/2^s( \sum_{c\in \{\pm 1\}^s} \delta_{\sum_{i\in[s]} c_i e_i } ).$ 

\item Constallantion in unit sphere.
The $2k$ point symmetric distribution on complex number, 
$F_p = \sum_{j=0}^{2k-1} p_j \delta_{\exp{(i\pi\frac{j}{k})}},$
where $\sum_{j=0}^{2k-1} p_j =1, p_j>0.$
\end{itemize}



\paragraph{Convex loss for discrete digital structure.}
The convex loss  we choose could be a relaxation of the convex indictor function of the points.

\begin{itemize}
\item For the two point symmetric distribution on real number, 
\[
g(x) = (|x|-1)_+
\]
\item The four point symmetric distribution on complex number, 
\[
g(z) = (|z|-1)_+,
\]
or more accurately,
\[
g(z) = (\|z\|_{\infty}-1)_+.
\]
\item For $s$-dimensional symmetric distribution on $\{\pm 1\}^s$,
\[
g(x) = (\|x\|_{\infty}-1)_+.
\] 
\end{itemize}


\paragraph{Blind recovery problem.}
For observation $Y$, assume that there exist inverse filter $\mu^* (e_0 + f^*)$, where $f^*_0 = 0$. We try to find it by solving a convex problem,
\begin{eqnarray*}
    \min_{f_0 = 0} &    
    E \sum g(\mu (Y+ f*Y)).
\end{eqnarray*}
First, when we know the correct scale $\mu^*$.

Let 
\[
e_0+h = ( e_0 + f)(e_0 + f^*)^{-1},
\]
we have
\begin{eqnarray*}
    \min_{f_0 = 0} &    
    E g( x_0+ h*x)).
\end{eqnarray*}
For $g(x) =(|x|-1)_+$, then the sub-gradient is
\[
g'(x) =  \left.
  \begin{cases}
    1, & \text{for } x > 1 \\
    0, & \text{for } -1 < x < 1 \\
    -1, & \text{for }-1 > x \\
    [0,1], & \text{for }x = 1  \\
    [-1,0], & \text{for }x = -1
  \end{cases}
  \right\}
\]

And the proximal operator $\mbox{prox}_{\rho g}$ is
\[
\mbox{prox}_{\rho g}(x) =  \left.
  \begin{cases}
    x-\rho, & \text{for } x > 1+\rho \\
    x, & \text{for } -1 < x < 1 \\
    x+\rho, & \text{for }-1-\rho > x \\
    1, & \text{for }1 \leq x \leq 1 +\rho \\
    -1, & \text{for } -1 -\rho\leq x \leq -1
  \end{cases}
  \right\}
\]


Assume $(X_t)$ are independent
sequence of random variable with CDF $F_{p}^t = \sum_{j=0}^{1} p^t_j \delta_{(-1)^j},$
where $\sum_{j=0}^{1} p^t_j =1, p^t_j>0.$ We choose the objective
\[
g(z) = (|z|-1)_+.
\] 
To show that $h=0$ is the global minimizer of the convex problem, we only need to show that for any $h\neq 0$, there exists small $\epsilon>0$, 
\begin{eqnarray*}    
    E g( x_0+ \epsilon h*x)-E g( x_0)>0.
\end{eqnarray*}
We use sub-gradient information,
\begin{eqnarray*}    
    E g( x_0+ \epsilon h*x)-E g( x_0)
    &\approx& 
    E g'( x_0)(\epsilon h*x)\\
    &=&
    \epsilon( p^0 _1 g'( +1)(h_0)+p^0_{-1}g'( -1)(-h_0)  )
    \geq 0.
\end{eqnarray*}

\paragraph{Polar form and extension to complex plane}
We can rewrite a point on the complex plane as its polar form
\[
z = r \exp{i\theta},
\]
where $r\geq 0$.

Under the polar form, the loss function is 
\[
g(z) = (|z|-1)_+ = (r-1)_+,
\]

We can reparametrize

Using this polar form, we are able to generalize our result to the complex setting.

\paragraph{Non-convex loss for discrete digital structure.}
If we are allowed to use non-convex loss, then we can fix the scale by the loss function.

Instead of convex loss
\[
g(x) = (|x|-1)_+,
\]
or
\[
g(x) = (|x|^2-1)_+,
\]
we can choose
\begin{itemize}
\item The smooth loss 
\[
g(x) = (|x|^2-1)^2,
\]
\item The composition of convex non-smooth function with smooth function,
\[
g(x) = (|x|-1)^2,
\]
\item The composition of  smooth function with convex non-smooth function,
\[
g(x) = ||x|^2-1|,
\]
\item The composition of  two convex non-smooth functions,
\[
g(x) = ||x|-1|.
\]
\end{itemize}


And instead of
\[
g(x) = (\|x\|_{\infty}-1)_+,
\]
we can choose
\begin{itemize}
\item The composition of convex non-smooth function with smooth function,
\[
g(x) = (\|x\|_{\infty}-1)^2,
\]
\item The composition of  two convex non-smooth functions,
\[
g(x) = |\|x\|_{\infty}-1|.
\]
\end{itemize}

\subsection{Noisy observation}
We consider the noisy recovery problem, there are two noise model, one is
\[
y = w_0 * (x+  n),
\]
the other is 
\[
y = w_0 * x+  u = w_0 * (x+ w_0^{-1} *u).
\]
When $u$ is Gaussian $N(0,\Sigma) $ , $w_0^{-1} *u$ is Gaussian $N(0,C_{w_0}^{-1}\Sigma C_{w_0}^{-1,T}) $. Therefore, we could reduce this problem as a colored noise $n$. 

\paragraph{Post-process}
For noisy observation $y$, we apply the algorithm on $y$ to get a solution $\hat x$, then we apply our prior knowledge to project the solution to the constellation points by taking $\title{x} = \mbox{sign}(\hat x).$

Then we measure the ratio of correct recovery by mean of
$1_{\title{x} = x}$ over multiple MC samples.
\paragraph{Population analysis}
Assume that the observation model are $y = w_0 * (x+  \sigma n),$ where $n$ is IID $N(0,1)$. Then we analyze the population risk,
 \begin{eqnarray*}
    \min_{h_0 = 0} &    
     E_x E_n g( x_0+ \sigma n_0+ h^T (x+\sigma n) ).
\end{eqnarray*}
Here $g(x) =(|x|-1)_+$.

Assume that the optimal solution is $h^*_\sigma$, when $\sigma = 0$ we proved before that $h^*=0.$ Here we get
$\hat x_0 = x_0+ \sigma n_0+ h^{*,T} (x+\sigma n) $
want to prove
\[
\tilde x_0 = \mbox{sign}(x_0+ \sigma n_0+ h_\sigma^{*,T} (x+\sigma n) ) = x_0,
\]
when $\sigma<\sigma_{m}$.
Without loss of generality, we assume $x_0=1$, we want to ask that when would $1+ \sigma n_0+ h_\sigma^{*,T} (x+\sigma n)\leq 0$, in general, the condition we need to charaterize is when 
$ |\sigma n_0+ h_\sigma^{*,T} (x+\sigma n)|\leq 1$.

\paragraph{Huber version of loss}
Instead of using $g(x) =(|x|-1)_+$, when there are Gaussian noise on $x$, we can use the Huber version 
\[
g_{\mu}(x) =\min_z  \frac{\mu}{2}\|x-z\|^2_2+ (|z|-1)_+  .
\]



























\section{Expectation calculation}
\subsection{Expectation over Bernoulli support}

We previously considered the identity
\begin{equation} \label{eq:startingPoint}
    {\mathbb E} | \sum_{i \in {\cal T}}  \psi_i X_i | = \sqrt{\frac{2}{\pi}} \cdot {\mathbb E}_I \| \psi \|_{\ell_2(I)},
\end{equation}
where $X$ is a Bernoulli-Gaussian RV $BG(p;0,1)$,
and where $I$ is a random subset of the domain ${\cal T}$
determined by Bernoulli-$p$ coin tossing.




This leads us to consider
\[
  {\mathbb E}_I \| \psi \|_{\ell_2(I)}^{2k}
 \]
 where $I$ is again a random subset, and $k \geq 1$ is an integer.
 The following lemma shows that ${\mathbb E}_I \| \psi \|_{\ell_2(I)}^{2k} \approx \| \psi \|_{\ell_2({\cal T})}^{2k}$.

\begin{lemma} \label{lem:BGPropLemmaSub}
Let $\psi$ denote a vector in ${\mathbb R}^N$.
\[
 \| \psi\|_2^{2k} \cdot \bar{B}(k-1;n,p)    \leq {\mathbb E}_I  \| \psi \|_{\ell_2(I)}^{2k}  \leq     \| \psi\|_2^{2k} ,
\]
where $\bar{B}(k-1;N,p) \equiv  \sum_{n=k}^{N} (1-p)^n p^{N-n} {N \choose n} $
is the survivor function (Complementary CDF) of the Binomial.
\end{lemma}

In particular, if $k \leq n \cdot p - 3 \cdot \sqrt{np (1-p)}$ for large $n$, then $\bar{B}(k-1;n,p) > 0.99$.
 The following lemma infers from this that ${\mathbb E}_I \| \psi \|_{\ell_2(I)} \approx \| \psi \|_{\ell_2({\cal T})}$.
 
\begin{lemma} \label{lem:BGPropLemmaNorm}
{\bf Qingyun: Numerically, I found this is not true even for fully dense $\psi$ as ${\varepsilon}_N$ could be quite large!}


There exists a sequence $({\varepsilon}_N)$ depending only on $N$ and $A$ so that  ${\varepsilon}_N \rightarrow 0$ 
and so that for each $N$ and for each 
vector $\psi$  in ${\mathbb R}^N$, with $\|\psi\|_{\ell_2} \leq A$ and $\|\psi\|_{\ell_0} = N$,
\begin{equation} \label{eq:LemmConclu}
 (1-N^{-1}) \cdot(1 - {\varepsilon}_N)  \leq  \frac{{\mathbb E}_I  \| \psi \|_{\ell_2(I)} }{\| \psi \|_{\ell_2({\cal T})}} \leq 1 + {\varepsilon}_N .
\end{equation}
\end{lemma}
Applying (\ref{eq:startingPoint}), we get:
\begin{coro} \label{lem:BGPropLemma}
{\bf Qingyun: Numerically, I found this is not true even for fully dense $\psi$!}
There exists a sequence $({\varepsilon}_N)$ depending only on $N$, so that  ${\varepsilon}_N \rightarrow 0$ and
and so that for each $N$ and for each 
vector $\psi$in ${\mathbb R}^N$, with $\|\psi\|_{\ell_2} \leq A$ and $\|\psi\|_{\ell_0} = N$.
\[
(1 - {\varepsilon}_N) \cdot (1-N^{-1}) \leq  \frac{E | \sum_{i \in {\cal T}}  \psi_i X_i |}{\sqrt{\frac{2}{\pi}}  \| \psi \|_{\ell_2({\cal T})}}  \leq   ( 1 + {\varepsilon}_N).
\]
\end{coro}


\subsection{Proof of Lemma \ref{lem:BGPropLemmaSub}}

\begin{proof}[Proof of Lemma \ref{lem:BGPropLemmaSub}]
Let's first suppose that $|\psi_t| \geq {\varepsilon} > 0$ for $-T \leq t \leq T$. Let $N = 2T+1$, and
let $I$ denote a subset of indices in $ {\cal T} = \{-T,\dots,T\}$ and
let $ \pi(I) = (1-p)^{|I|} p^{|{\cal T} \Delta I|}$. 
\begin{eqnarray*}
{\mathbb E}_I  \| \psi \|_{\ell_2(I)}^{2k}  &=&  \sum_{n=1}^{N}  \sum_{ |I |=n} \pi(I)   \| \psi \|_{\ell_2(I)}^{2k} \\
&=&   \sum_{n=1}^{N}  (1-p)^n p^{N-n}  \cdot  \sum_{|I| =n} \| \psi \|_{\ell_2(I)}^{2k}  \\
   \end{eqnarray*}
where we used $ \pi(I) = (1-p)^{|I|} p^{|{\cal T} \Delta I|} $.
We partition the sum  $\sum_{n=1}^{N} = \sum_{n=1}^{k-1} + \sum_{k}^N =  \sum_{n<k} + \sum_{n\geq k}$, say.
Now from $\#\{ |I|\leq n\} = {N \choose n}$ and $ \| \psi \|_{\ell_2(I)} \leq \| \psi \|_{\ell_2({\cal T})}$,
\begin{eqnarray*}
 \sum_{n<k}  (1-p)^n p^{N-n}  \cdot  \sum_{|I| =n} \| \psi \|_{\ell_2(I)}^{2k} 
     &\leq&   \left( \sum_{n=1}^{k-1} (1-p)^n p^{N-n} {N \choose n} \right) \cdot  \left( \| \psi \|_{\ell_2}^{2k} \right) \\
     &=& B(k-1;n,p) \cdot   \| \psi \|_{\ell_2}^{2k} ,
\end{eqnarray*}
where $B(x;n,p)$ denotes the cumulative distribution function of the binomial distribution.
For the sum $\sum_{n\geq k}$, we may write $\sum_{|I|=n} = {N \choose n}  \cdot Ave_{|I| = n}$
and $\sum_{t_1,\dots, t_k \in I} =  {n^k} \cdot  Ave_{t_1,\dots, t_k \in I}$, so
\begin{eqnarray*}
 \sum_{|I| = n}  \| \psi \|_{\ell_2(I)}^{2k} &=& \sum_{|I| = n} \sum_{t_1,\dots, t_k \in I} \prod_{i=1}^k \psi_{t_i}^2 \\
& =& {N \choose n}  \cdot Ave_{|I| = n}  \cdot {n^k} \cdot  Ave_{t_1,\dots, t_k \in I} \; \prod_{i=1}^k \psi_{t_i}^2 \\
&=& {N \choose n}  \cdot {n^k} \cdot Ave_{t_1,\dots, t_k \in {\cal T}} \; \prod_{i=1}^k \psi_{t_i}^2,\\
&=& {N \choose n}  \cdot \sum_{t_1,\dots, t_k \in {\cal T}} \; \prod_{i=1}^k \psi_{t_i}^2,\\
&=& {N \choose n}  \cdot  \| \psi \|_{\ell_2}^{2k} ,
\end{eqnarray*}
because the inner two averages  cover the $ {N \choose n} n^k$ terms in a uniform way.
We arrive at 
 \begin{eqnarray*}
 \sum_{n\geq k}  (1-p)^n p^{N-n}  \cdot  \sum_{|I| =n} \| \psi \|_{\ell_2(I)}^{2k} 
  &=&     \| \psi \|_{\ell_2}^{2k} \cdot  \left( \sum_{n=k}^{N} (1-p)^n p^{N-n} {N \choose n} \right) \\
  &=&    \| \psi \|_{\ell_2}^{2k} \cdot \bar{B}(k-1;n,p) .
 \end{eqnarray*} 
Here $\bar{B}(x;n,p) = 1- B(x;n,p)$ denotes the 
survivor function (complementary distribution function) of the binomial distribution.
We obtain:
 \begin{eqnarray*}
  \sum_{I \subset {\cal T}} \pi(I)   ( \sum_{t \in I} \psi_t^2)^{k}  
  & = &  \left( \sum_{n \geq k} + \sum_{n<k}   \right) \sum_{|I|=n}  \pi(I)   ( \sum_{t \in I} \psi_t^2)^{k} \\
 &\leq &    \|\psi\|_2^{2k} \cdot  \left( \bar{B}(k-1;p,n)  + B(k-1;p,n) \right) \\
 & = &  \|\psi\|_2^{2k} .
 \end{eqnarray*}
 while
 \begin{eqnarray*}
  \sum_{I \subset {\cal T}} \pi(I)   ( \sum_{t \in I} \psi_t^2)^{k}   & \geq & \sum_{n \geq k} \pi(I)   ( \sum_{t \in I} \psi_t^2)^{k}  \\
 &= &    \|\psi\|_2^{2k} \cdot \bar{B}(k-1;p,n) .
  \end{eqnarray*}
\end{proof}

\subsection{Proof of Lemma \ref{lem:BGPropLemmaNorm}}

\begin{proof}
Fix $A > 0$ and $p \in (0,1)$
By the Weierstrass approximation theorem, we may find a sequence of polynomials $S_K(t) = \sum_{k=0}^{K} c_k t^k$
so that
\[
    | \sqrt{x} - S_K(x) |_{L^\infty([0,A])} \leq {\varepsilon}_K, \quad \mbox { and } {\varepsilon}_K \rightarrow 0 \mbox{ as } K \rightarrow \infty.
\]
Pick a sequence $(K_N)$ so that $K_N \rightarrow \infty$ while
$K_N < N \cdot p -  \sqrt{2 \log(N) \sqrt{N} p (1-p)} $ and
\[
\bar{B}(K_N-1; n,p) > 1 - N^{-1}.
\]
Relabel $S_N  \equiv S_{K_N}$ and ${\varepsilon}_N \equiv {\varepsilon}_{K_N} $.
Then ${\varepsilon}_N \rightarrow 0$ as $N \rightarrow \infty$.

Fix $N$. For each $I \subset {\cal T}$,
let  $x_I \equiv {\mathbb E}_I \| \psi \|_{\ell_2(I)}^{2}$.  Then
\[
     E _I \| \psi \|_{\ell_2(I)} = {\mathbb E}_I \sqrt{\| \psi \|_{\ell_2(I)}^2} = {\mathbb E}_I \sqrt{ x_I }.
\]
while
\[
     E _I \| \psi \|_{\ell_2(I)}^{2k} = {\mathbb E}_I  x_I^k.
\]
Invoking our earlier lemma,
\[
 \bar{B}(k-1;n,p) \leq  \frac{{\mathbb E}_I  x_I^k}{\| \psi \|_{\ell_2({\cal T})}^{2k}} \leq 1.
\]

Armed with  polynomial $S_N$ of degree $K_N$, consider
\begin{eqnarray*}
     {\mathbb E}_I S_{N}( x_I ) &=& {\mathbb E}_I \sum_{k=1}^{K_N} c_k x_I^{k}\\
     &=&  \sum_{k=1}^{K_N} c_k  {\mathbb E}_I x_I^{k} \\
     &\leq& \sum_{k=1}^{K_N} c_k   \| \psi \|_{\ell_2({\cal T})}^{2k}\\
     &=& S_{N}( \| \psi \|_{\ell_2({\cal T})}^2)  \\
     &\leq& ( \| \psi \|_{\ell_2({\cal T})}^2)^{1/2} + {\varepsilon}_{N} . 
\end{eqnarray*}
where we used ${\mathbb E}_I x_I^{k} \leq  \| \psi \|_{\ell_2({\cal T})}^{2k}$ from the previous Lemma.
Similarly,
\begin{eqnarray*}
     {\mathbb E}_I S_N( x_I )   &=&  \sum_{k=1}^{K_N} c_k  {\mathbb E}_I x_I^{k} \\
     &\geq& \sum_{k=1}^{K_N} c_k   \cdot \bar{B}(k-1; n,p) \cdot \| \psi \|_{\ell_2({\cal T})}^{2k}\\
     &\geq& \bar{B}(K_N-1; n,p) \cdot \sum_{k=1}^{K} c_k   \cdot \| \psi \|_{\ell_2({\cal T})}^{2k}\\
     &=& \bar{B}(K_N-1; n,p)  \cdot S_K( \| \psi \|_{\ell_2({\cal T})}^2)  \\
     &\geq&  \bar{B}(K_N-1; n,p)  \cdot \left( ( \| \psi \|_{\ell_2({\cal T})}^2)^{1/2} - {\varepsilon}_N \right)  \\
   &\geq&  (1- N^{-1}) \cdot \left( ( \| \psi \|_{\ell_2({\cal T})}^2)^{1/2} - {\varepsilon}_N \right)  . 
\end{eqnarray*}
Here  we used ${\mathbb E}_I x_I^{k} \geq  \| \psi \|_{\ell_2({\cal T})}^{2k} \cdot \bar{B}(k-1;n,p)$ from the previous Lemma.
We also used the monotonicity 
of the survivor function $\bar{B}(x) \geq \bar{B}(x+1)$.
We obtain (\ref{eq:LemmConclu}):
\[
 (1-N^{-1}) \cdot(1 - {\varepsilon}_N)  \leq  \frac{{\mathbb E}_I  \| \psi \|_{\ell_2(I)} }{\| \psi \|_{\ell_2({\cal T})}} \leq 1 + {\varepsilon}_N .
\]
\end{proof}


We define the functional  $\Gamma: \psi  \mapsto \frac{{\mathbb E}_I  \| \psi \|_{\ell_2(I)} }{\| \psi \|_{\ell_2({\cal T})}}$,
from the previous calculation we know $\Gamma(\psi) $ is only a function of $\| \psi\|_0$. 
So we can identify  $\Gamma(\psi) $ as  $\Gamma(\| \psi\|_0)$. 





\subsection{Sharper bounds on $\frac{{\mathbb E}_I  \| \psi \|_{\ell_2(I)} }{\| \psi \|_{\ell_2({\cal T})}}$}




\begin{lemma}
\label{TaylorSqrt}
For $ 0 \leq x\leq 1,$
\begin{align}
(1-x)^{1/2}
=1-\sum_{\ell=0}^\infty\frac2{\ell+1}\binom{2\ell}{\ell}\left(\frac x4\right)^{\ell+1}
\end{align}
\end{lemma}

\begin{proof}
From generalized binomial theorem, we know that for $ 0 \leq x\leq 1,$
\begin{equation}
(1-x)^{1/2}
=\sum_{\ell=0}^\infty\binom{1/2}{\ell}(-x)^\ell
\end{equation}

\begin{align}
\binom{1/2}{\ell}
&=\frac{\frac12(\frac12-1)(\frac12-2)\cdots(\frac12-\ell+1)}{\ell!}\\
&=\frac{(-1)^{\ell-1}}{2^\ell\ell!}1\cdot3\cdot5\cdots(2\ell-3)\\
&=\frac{(-1)^{\ell-1}}{2^\ell\ell!}\frac{(2\ell-2)!}{2^{\ell-1}(\ell-1)!}\\
&=\frac{(-1)^{\ell-1}}{\ell2^{2\ell-1}}\binom{2\ell-2}{\ell-1}
\end{align}
then
\begin{align}
(1-x)^{1/2}
&=1-\sum_{\ell=1}^\infty\frac2\ell\binom{2\ell-2}{\ell-1}\left(\frac x4\right)^\ell\\
&=1-\sum_{\ell=0}^\infty\frac2{\ell+1}\binom{2\ell}{\ell}\left(\frac x4\right)^{\ell+1}
\end{align}






\end{proof}

\begin{lemma}
\begin{equation} 
0 \leq 1- Q(N, p)\leq  \frac{{\mathbb E}_I  \| \psi \|_{\ell_2(I)} }{\| \psi \|_{\ell_2({\cal T})}} \leq 1 .
\end{equation}
where 
\[
Q(N, p) := \sum_{\ell=0}^N\frac{2^{-(2\ell+1)}}{\ell+1}\binom{2\ell}{\ell} \bar{B}(\ell;N,1-p),
\]
\[
\bar{B}(\ell;N,1-p) \equiv  \sum_{n=0}^{N-\ell-1} (1-p)^n p^{N-n} {N \choose n}.
\]
\end{lemma}
\begin{proof}
For any fixed support $J\subset {\cal T},$ 
\begin{equation} 
 \frac{ \| \psi \|_{\ell_2(J)} }{\| \psi \|_{\ell_2({\cal T})}} 
 = \sqrt{ \frac{ \| \psi \|^2_{\ell_2(J)} }{\| \psi \|^2_{\ell_2({\cal T})}}} \leq 1 .
\end{equation}
Therefore, 
\begin{equation} 
{\mathbb E}_I \frac{ \| \psi \|_{\ell_2(I)} }{\| \psi \|_{\ell_2({\cal T})}} 
 = {\mathbb E}_I \sqrt{ \frac{ \| \psi \|^2_{\ell_2(I)} }{\| \psi \|^2_{\ell_2({\cal T})}}} \leq {\mathbb E}_I 1 = 1 .
\end{equation}

On the other hand, let 
\begin{equation} 
{\mathbb E}_I \frac{ \| \psi \|_{\ell_2(I)} }{\| \psi \|_{\ell_2({\cal T})}} 
 = {\mathbb E}_I \sqrt{ 1- \frac{ \| \psi \|^2_{\ell_2({\cal T}-I)} }{\| \psi \|^2_{\ell_2({\cal T})}}}  .
\end{equation}

For a fixed subset $J \subset [n]$ define 
the ratio $\rho_J \equiv  \frac{ \| \psi \|^2_{\ell_2({\cal T}-J)} }{\| \psi \|^2_{\ell_2({\cal T})}}$; it 
obeys $0 \leq \rho_J \leq 1$.

Applying Lemma \ref{TaylorSqrt},
\begin{align}
(1-\rho_J)^{1/2}
=1-\sum_{\ell=0}^\infty\frac{2^{-(2\ell+1)}}{\ell+1}\binom{2\ell}{\ell}
\rho_J^{\ell+1}
\end{align}
Now with $I$ a random subset as earlier, we
induce a random variable $\rho_I$.
\begin{align}
{\mathbb E}_I(1-\rho_I)^{1/2}
=1-\sum_{\ell=0}^\infty\frac{2^{-(2\ell+1)}}{\ell+1}\binom{2\ell}{\ell}
{\mathbb E}_I \rho_I^{\ell+1}
\end{align}
Modify the argument behind Lemma \ref{lem:BGPropLemmaSub} so it applies to ${\mathbb E}_I \rho_I^{\ell+1}$. In the new argument the set $I$ is everywhere replaced by its complement $T-I$. The conclusion is correspondingly modified so that $p$ in the final formula is replaced by $q \equiv 1-p$.
We obtain:
\[
 \bar{B}(\ell;n,1-p)    \leq {\mathbb E}_I \rho_I^{\ell+1} \leq    1,
\]
where, if $k \in \{0,1,\dots,N\}$,
$\bar{B}(k-1;N,p) \equiv  \sum_{n=k}^{N} (1-p)^n p^{N-n} {N \choose n}  = 1- \sum_{n=0}^{k-1} (1-p)^n p^{N-n} {N \choose n}$, while if $k \geq N$, $\bar{B}(k) = 0$, and if $k < 0$,
$\bar{B}(k) =1$.
Then
\begin{align}
{\mathbb E}_I(1-\rho_I)^{1/2}
&=1-\sum_{\ell=0}^\infty\frac{2^{-(2\ell+1)}}{\ell+1}\binom{2\ell}{\ell}
{\mathbb E}_I \rho_I^{\ell+1}\\
&\geq  1-\sum_{\ell=0}^\infty\frac{2^{-(2\ell+1)}}{\ell+1}\binom{2\ell}{\ell} \bar{B}(\ell;N,1-p)\\
&=1- \sum_{\ell=0}^N\frac{2^{-(2\ell+1)}}{\ell+1}\binom{2\ell}{\ell} \bar{B}(\ell;N,1-p)\\
&= 1- Q(N, p)
\end{align}
{\bf Qingyun NOTE: I observe numerically that $Q(N,p)$ could be between $0$ and $1$ in a wide range, and $Q(N,p)$ is monotonically increasing with $N$, and monotonically decreasing with $p$, with $Q(N,1) = 0$, $\lim_{N\rightarrow \infty}\lim_{p\rightarrow 0} Q(N,p) = 1.$
.}
\end{proof}

\paragraph{Beta function and $Q(N,p)$}
If we define the complete beta function as 
\[
\mathcal{B}(a,b) := \int_0^1 t^{a-1}\,(1-t)^{b-1}\,dt.
\]
then
\[
\binom{n}{k} = \frac{1}{(n+1) \mathcal{B}(n-k+1, k+1)}
\]
the incomplete beta function as
\[
\mathcal{B}(x;\,a,b) = \int_0^x t^{a-1}\,(1-t)^{b-1}\,dt.
\]
then the regularized incomplete beta function is defined in terms of the incomplete beta function and the complete beta function:
\[
I_x(a,b) = \frac{\mathcal{B}(x;\,a,b)}{\mathcal{B}(a,b)}.
\]
then
\[
I_p(k,N-k-1)= \bar{B}(k-1;N,p) = \sum_{n=k}^{N} (1-p)^n p^{N-n} {N \choose n}
\]
and
\[
I_{1-p}(k+1, N-k) = 1 - I_p(N-k, k+1) = \bar{B}(k;N,1-p).
\]
Let $\mu = N \cdot (1-p)$ and 
$\sigma = \sqrt{N \cdot p \cdot (1-p)}$.

We note that, for large $N$, from the classical De Moivre–Laplace theorem on normal approximation to the binomial, 
\[
{N \choose k}\, p^k q^{N-k} \approx \frac{1}{\sqrt{2 \pi Np(1-p)}}\,e^{-\frac{(k-Np)^2}{2Np(1-p)}}
\]
therefore, 
\[
I_{1-p}(\ell+1, N-\ell) = \bar{B}(\ell;N,1-p)  \approx \bar{\Phi}( (\ell - \mu)/\sigma ) ,
\]
where $\Phi$ denotes the standard Normal $N(0,1)$ CDF
and $\bar{\Phi} \equiv 1-\Phi$ 
the corresponding survivor function;

\begin{eqnarray*}
Q(N,p) 
&\equiv& \sum_{\ell=0}^N\frac{2^{-(2\ell+1)}}{\ell+1}\binom{2\ell}{\ell} \bar{B}(\ell;N,1-p) \\
&=& \sum_{\ell=0}^N\frac{2^{-(2\ell+1)}}{\ell+1} \frac{1}{(2\ell+1) \mathcal{B}(\ell+1, \ell+1)}  I_{1-p}(\ell+1, N-\ell)\\
&\approx& \sum_{\ell=0}^N\frac{1}{2(\ell+1)} \frac{1}{\sqrt{\frac{\pi}{2} N}}\,e^{-\frac{(\ell-\frac{N}{2})^2}{\frac{N}{2}}}  \bar{\Phi}( (\ell - N \cdot (1-p))/\sqrt{N \cdot p \cdot (1-p)} )\\
&\approx& \int_0^1 \frac{1}{2(Nt+1)} \frac{1}{\sqrt{\frac{\pi}{2} N}}\,e^{- 2N(t-\frac{1}{2})^2}  \bar{\Phi}( \frac{\sqrt{N}(t - (1-p))}{\sqrt{ p \cdot (1-p)}} ) \,dt
\end{eqnarray*}

Numerically, we can see that the first term in the sum is always $0.5.$
Also we know that by definition $Q(N, 1) = 1$ for any $N$.
From numerically calculation we guess that
$Q(N,p)$ is monotonically increasing with $N$, and monotonically decreasing with $p$.
\[
Q(N,p') \geq Q(N,p), \quad, p>p'
\]
and 
$\lim_{N\rightarrow \infty}\lim_{p\rightarrow 0} Q(N,p) = 1.$


\begin{lemma}
Define:
\[
Q(N,p) \equiv \sum_{\ell=0}^N\frac{2^{-(2\ell+1)}}{\ell+1}\binom{2\ell}{\ell} \bar{B}(\ell;N,1-p).
\]
Then, 
\[
Q(N,p') \geq Q(N,p), \quad, p>p'
\]
and 
\[
\lim_{N\rightarrow \infty}\lim_{p\rightarrow 0} Q(N,p) = 1.
\]
\end{lemma}


\begin{proof}
The first term in the sum is $0.5$.

(Sketch). 
{\bf The sum has the lower bound $Q \geq 1/10$ just by inspecting
the first 10 terms (say), which are all positive and all greater
than $0.01$ (say). The following argument shows that later terms
add up to at least $\Omega(N^{-1/2})$.}
First, we know that
\[
\bar{B}(\ell;N,1-p) \equiv  \sum_{n=0}^{N-\ell-1} (1-p)^n p^{N-n} {N \choose n}.
\]

Let $\mu = N \cdot (1-p)$ and 
$\sigma = \sqrt{N \cdot p \cdot (1-p)}$.

We note that, for large $N$ 
\[
\bar{B}(\ell;N,1-p)  \approx \bar{\Phi}( (\ell - \mu)/\sigma ) ,
\]
where $\Phi$ denotes the standard Normal $N(0,1)$ CDF
and $\bar{\Phi} \equiv 1-\Phi$ 
the corresponding survivor function;
this is the classical De Moivre theorem on normal approximation to the binomial. It can be made precise in many ways
that have been well documented over the centuries.
Roughly speaking, in terms of variable $z = (\ell - \mu)/\sigma$,
we have
\[
\bar{B}(\ell;N,1-p)  \approx \left \{ \begin{array}{ll}
1 & z < 3 \\
0 & z > 3 .
\end{array}
\right .
\]

Hence, defining for $m = \mu - 3 \sigma$
\[
\bar{B}(\ell;N,1-p) \approx 1, \mbox{ for } \ell \leq m.
\]
Now of course
\begin{eqnarray*}
Q(N,p) &\geq& \sum_{\ell=0}^m\frac{2^{-(2\ell+1)}}{\ell+1}\binom{2\ell}{\ell} \bar{B}(\ell;N,1-p) \\
&\geq& \sum_{\ell=0}^m\frac{2^{-(2\ell+1)}}{\ell+1}\binom{2\ell}{\ell} .
\end{eqnarray*}
Now from DeMoivre's theorem (case $p=1/2$, $N=2\ell$) we also know that
\[
    2^{-2\ell} { 2 \ell \choose \ell} = bin(\ell; 2 \ell, 1/2) \sim \sqrt{\frac{1}{\ell \cdot \pi}},
\]
where $bin(k;N,p)$ is the usual binomial probability mass function.
We obtain
\[
  \sum_{\ell=0}^m\frac{2^{-(1)}}{\ell+1} \ell^{-1/2} \geq 
  \sum_{\ell=m/2}^m\frac{2^{-(1)}}{\ell+1} \ell^{-1/2}  \geq 
  Const  \cdot m^{-1/2}
\]

\end{proof}





\subsection{Bounds on $\frac{{\mathbb E}_I  \| \psi \|^{-1}_{\ell_2(I)} }{\| \psi \|^{-1}_{\ell_2({\cal T})}}$}
\begin{lemma}
\label{TaylorSqrtInv}
For $ 0 \leq x\leq 1,$
\begin{align}
(1-x)^{-1/2}
=\sum_{\ell=0}^\infty\frac{1}{2^{2\ell}}\binom{2\ell}{\ell}\left(x\right)^{\ell}
\end{align}
\end{lemma}

\begin{proof}
From generalized binomial theorem, we know that for $ 0 \leq x\leq 1,$
\begin{equation}
(1-x)^{-1/2}
=\sum_{\ell=0}^\infty\binom{-1/2}{\ell}(-x)^\ell
\end{equation}

\begin{align}
\binom{-1/2}{\ell}(-1)^\ell
&=\frac{-\frac12(-\frac12-1)(-\frac12-2)\cdots(-\frac12-\ell+1)}{\ell!}\\
&=\frac{1}{2^\ell\ell!}1\cdot3\cdot5\cdots(2\ell-1)\\
&=\frac{1}{2^\ell\ell!}\frac{(2\ell)!}{2^{\ell}(\ell)!}\\
&=\frac{1}{2^{2\ell}}\binom{2\ell}{\ell}
\end{align}
\end{proof}

\begin{lemma}
\label{TaylorSqrtInvExpect}
\begin{equation} 
1+\sum_{\ell=1}^\infty\frac{1}{2^{2\ell}}\binom{2\ell}{\ell}\bar{B}(\ell-1;N,1-p) \geq  \frac{{\mathbb E}_I  \| \psi \|^{-1}_{\ell_2(I)} }{\| \psi \|^{-1}_{\ell_2({\cal T})}} \geq 1 .
\end{equation}
where 
\[
\bar{B}(\ell-1;N,1-p) \equiv  \sum_{n=0}^{N-\ell} (1-p)^n p^{N-n} {N \choose n} 
\]
\end{lemma}
\begin{proof}
For any fixed support $J\subset {\cal T},$ 
\begin{equation} 
 \frac{ \| \psi \|^{-1}_{\ell_2(J)} }{\| \psi \|^{-1}_{\ell_2({\cal T})}} 
 = \sqrt{ \frac{ \| \psi \|^2_{\ell_2({\cal T})} }{\| \psi \|^2_{\ell_2(J)}}} \geq 1 .
\end{equation}
Therefore, 
\begin{equation} 
\frac{{\mathbb E}_I  \| \psi \|^{-1}_{\ell_2(I)} }{\| \psi \|^{-1}_{\ell_2({\cal T})}}  
 \geq   1 .
\end{equation}

On the other hand, let 
\begin{equation} 
{\mathbb E}_I (\frac{ \| \psi \|_{\ell_2(I)} }{\| \psi \|_{\ell_2({\cal T})}} )^{-1}
 = {\mathbb E}_I ( 1- \frac{ \| \psi \|^2_{\ell_2({\cal T}-I)} }{\| \psi \|^2_{\ell_2({\cal T})}})^{-1/2}.
\end{equation}

For a fixed subset $J \subset [n]$ define 
the ratio $\rho_J \equiv  \frac{ \| \psi \|^2_{\ell_2({\cal T}-J)} }{\| \psi \|^2_{\ell_2({\cal T})}}$; it 
obeys $0 \leq \rho_J \leq 1$.


\begin{align}
(1-\rho_J)^{-1/2}
=\sum_{\ell=0}^\infty\frac{1}{2^{2\ell}}\binom{2\ell}{\ell}\left(\rho_J\right)^{\ell}
\end{align}
Now with $I$ a random subset as earlier, we
induce a random variable $\rho_I$.
\begin{align}
{\mathbb E}_I(1-\rho_J)^{-1/2}
=\sum_{\ell=0}^\infty\frac{1}{2^{2\ell}}\binom{2\ell}{\ell}{\mathbb E}_I\left(\rho_J\right)^{\ell}
\end{align}
Modify the argument behind Lemma \ref{lem:BGPropLemmaSub} so it applies to ${\mathbb E}_I \rho_I^{\ell+1}$. In the new argument the set $I$ is everywhere replaced by its complement $T-I$. The conclusion is correspondingly modified so that $p$ in the final formula is replaced by $q \equiv 1-p$.
We obtain:
\[
 \bar{B}(\ell-1;n,1-p)    \leq {\mathbb E}_I \rho_I^{\ell} \leq    1,
\]
where, if $k \in \{0,1,\dots,N\}$,
$\bar{B}(k-1;N,p) \equiv  \sum_{n=k}^{N} (1-p)^n p^{N-n} {N \choose n}  = 1- \sum_{n=0}^{k-1} (1-p)^n p^{N-n} {N \choose n}$, while if $k \geq N$, $\bar{B}(k) = 0$, and if $k < 0$,
$\bar{B}(k) =1$.
Then
\[
\bar{B}(\ell-1;N,1-p) \equiv  \sum_{n=0}^{N-\ell} (1-p)^n p^{N-n} {N \choose n} 
\]
\begin{align}
{\mathbb E}_I(1-\rho_I)^{1/2}
&=\sum_{\ell=0}^\infty\frac{1}{2^{2\ell}}\binom{2\ell}{\ell}{\mathbb E}_I\left(\rho_J\right)^{\ell}
\\
&\geq  \sum_{\ell=0}^\infty\frac{1}{2^{2\ell}}\binom{2\ell}{\ell}\bar{B}(\ell-1;N,1-p)\\
&=1+\sum_{\ell=1}^\infty\frac{1}{2^{2\ell}}\binom{2\ell}{\ell}\bar{B}(\ell-1;N,1-p)
\end{align}

\end{proof}





\subsection{Numerically calculate ${\mathbb E}_I  \| \psi_I \|_2/\| \psi \|_2$}

We numerically compute $Q(N,p)$, the result is in the following table. Compare the table with the tables of MC approximation, we see that our previous bound is still too loose.
\begin{table}[]
\begin{tabular}{|p{3cm}||llllllll}
\hline
\multicolumn{9}{|c|}{$Q(N,p)$} \\
\hline
  $p=$ &0   & 0.1   & 0.2       &   0.5     &   0.9     &   0.99     &  0.999 &1      \\
 \hline\hline
$N=100$ &0.9439 & 0.9406 & 0.9370 & 0.920  & 0.8172 & 0.3358 & 0.0482 & 0 \\
\hline
$N=500$ & 0.9748  & 0.9734 & 0.9718 & 0.9643 & 0.9199 & 0.7302 & 0.209 & 0 \\
\hline      
\end{tabular}
\label{Q_table}
\end{table}

Then we use MC approximation to calculate ${\mathbb E}_I  \| \psi_I \|_2/\|  \psi \|_2$ for a few different $\psi.$
We will know that
\begin{itemize}
\item 
When $\psi = e_0$,  ${\mathbb E}_I  \| (e_0)_I \|_2=p$ 

\item
When $\psi = e_0+e_1$,  
\[
{\mathbb E}_I  \|(e_0+e_1)_I \|_2 /\| e_0+e_1 \|_2 = \sqrt{\frac12} {\mathbb E}_I  \| (e_0+e_1)_I \|_2 = 
{\mathbb E}_I \sqrt{\frac{\sum_{i=\{0,1\}} I_i}{2}}=p^2+ 2p(1-p)\sqrt{\frac12}
\]
\item
We know that when $\psi = a =(1,\ldots,1)$, then
\[
{\mathbb E}_I  \| a_I \|_2/\| a \|_2={\mathbb E}_I \sqrt{\frac{\sum_{i=[N]} I_i}{N}} 
\]
Using law of large numbers, 
\[
\lim_{N\rightarrow \infty }\sqrt{\frac{\sum_{i=[N]} I_i}{N}}  = \sqrt{p}.
\]
\item
For fixed support $J$, we can choose $\psi = \sum_{i\in J} e_i,$ then $\| \psi_J \|_2/\|  \psi \|_2 =1.$
However, the expectation is not going to be one. Let $M := |J|,$
\[
{\mathbb E}_I \sqrt{\frac{\sum_{i\in J} I_i}{|J|}} = \sum_{k=0}^{M} (1-p)^{M-k} p^{k} {M \choose k}\sqrt{\frac{k}{M}}\leq \sqrt{p}
\]
The inequality can be seeing using central limit theorem, we have normal approximation  $\frac{\sum_{i\in J} I_i}{|J|} \sim N(p,  \frac{p(1-p)}{\sqrt{|J|}})$, then using Jensen's inequality, since square root fucntion is concave,
\[
{\mathbb E}_I \sqrt{\frac{\sum_{i\in J} I_i}{|J|}} \approx {\mathbb E}_G\sqrt{p+ \frac{p(1-p)}{\sqrt{|J|}}G}  \leq \sqrt{{\mathbb E}_G[p+ \frac{p(1-p)}{\sqrt{|J|}}G]} =\sqrt{p} 
\]
\end{itemize}
Using symmetry, we guess that the maximum value of ${\mathbb E}_I  \| \psi_I \|_2/\|  \psi \|_2$  is $\sqrt{p}$ instead of $1$, and the maximum value is achieved when $\psi$ is dense.

\begin{table}[]
\begin{tabular}{|p{3cm}||llllllll}
\hline
\multicolumn{9}{|c|}{MC approximation ($M=10^4$) of ${\mathbb E}_I  \| a_I \|_2/\| a \|_2$ where $a=(1,\ldots,1)$} \\
\hline
  $p=$ & 0   & 0.1   & 0.2       &   0.5     &   0.9     &   0.99     &  0.999 &1      \\
 \hline\hline
$N=100$ &0 & 0.3123 & 0.4460 & 0.7063  & 0.94865 & 0.9950 & 0.9995 & 1 \\
\hline
$N=500$ &0 &0.3153  & 0.4464 & 0.7068 & 0.94874 & 0.9950 & 0.9995 & 1\\
\hline      
\end{tabular}
\label{allone_table}
\end{table}


\begin{table}[]
\begin{tabular}{|p{3cm}||llllllll}
\hline
\multicolumn{9}{|c|}{MC approximation ($M=10^4$) of ${\mathbb E}_I  \|(e_0+e_1)_I \|_2 /\| e_0+e_1 \|_2$} \\
\hline
  $p=$ & 0   & 0.1   & 0.2       &   0.5     &   0.9     &   0.99     &  0.999 &1      \\
 \hline\hline
$N=500$ & 0 &0.1375 &	0.2667 &	0.6034 &	0.9365 &	0.9946 &	0.9994 & 1 \\
\hline      
\end{tabular}
\label{coordinatevector_table}
\end{table}



\subsection{Tight bound and stationary points of ${\mathbb E}_I  \| \psi_I \|_2/\| \psi \|_2$}
\begin{lemma}
\begin{equation} 
p \leq  \frac{{\mathbb E}_I  \| \psi \|_{\ell_2(I)} }{\| \psi \|_{\ell_2({\cal T})}} \leq \sqrt{p}  .
\end{equation}
where the lower bound is achieved by $1-$sparse vectors $\pm e_i$, the upper bound is achieve by $\{\frac{\sum_{i\in {\cal T}} \pm e_i}{\sqrt{N}}\}$.
Furthermore, all the stationary points of this objective are $\{\frac{\sum_{i\in J} \pm e_i}{\sqrt{|J|}}\}$ for different support $J\subset {\cal T}$. 
\end{lemma}
\begin{proof}
This problem can be reformulate as the projection pursuit with sphere constraint,
 \begin{eqnarray*}
    \min&   {\mathbb E}_I  \| \psi_I \|_2\\
    \mbox{ subject to }&   \| \psi \|_2 = 1.
\end{eqnarray*}
Then from \cite{bai2018subgradient} Proposition $3.3$, we get the result. 

The main idea of the proof is to calculate the projected gradient
for $q\in \{\frac{\sum_{i\in J} \pm e_i}{\sqrt{|J|}}\}$, when $|J| = M,$
\[
e_j^T  {\mathbb E}_I  \partial\| q_I \|_2 = q_j [\sum_{k=0}^{M} (1-p)^{M-k} p^{k} {M \choose k}\sqrt{\frac{k}{M}}] 
\]
so
\[
(I-q q^T)  {\mathbb E}_I  \partial\| q_I \|_2 = [\sum_{k=0}^{M} (1-p)^{M-k} p^{k} {M \choose k}\sqrt{\frac{k}{M}}]q-[\sum_{k=0}^{M} (1-p)^{M-k} p^{k} {M \choose k}\sqrt{\frac{k}{M}}]q = 0.
\]
then $q\in \{\frac{\sum_{i\in J} \pm e_i}{\sqrt{|J|}}\}$ is stationary points.
 
The other direction (all other points are not stationary) is implied by (the proof of) theorem $3.4$ in  \cite{bai2018subgradient}.
\end{proof}
\section{Expectation calculation}
In the following, we calculate ${\mathbb E}| \psi^T X| $ for $X_t$ IID sampled from Bernoulli Gaussian $p N(0,1) +(1-p) \delta_0.$
\subsection{Expectation of inner product for sparse $X$}
In the following, we study in detail the quantity  
\[
E{| \psi^T X| } = E{|\sum_{i} \psi_i X_i| }.
\]
We know when $X_i$ have variance $\sigma^2$,
\[
   E (\sum_i \psi_i X_i)^2 =  \sum_i   \psi_i^2 E(X_i)^2 = \sigma^2 \|\psi\|_2^2.
\]
The ratio indicates the sparsity level of the random variable $ \sum_i \psi_i X_i$
\[
\frac{ E{|\sum_{i} \psi_i X_i| }}{\sqrt{E (\sum_i \psi_i X_i)^2}} = \frac{ E{|\sum_{i} \psi_i X_i| }}{\sigma^2 \|\psi\|_2} 
\]
Now, we consider $E{|\sum_{i} \psi_i X_i| }$ in the general symmetric setting, where a lower bound can be derived.

\paragraph{Exact calculation for Bernoulli Gaussian}
Let $X_i = B_i Z_i$ be IID Bernoulli Gaussian, where $B_i$ is sampled from Bernoulli with parameter $p$, $Z_i$ is sampled from $N(0,1)$. Let $I$ be the support where $B_i =1$.
\[
{\mathbb E}_X{|\sum_{i} \psi_i X_i| }  
= {\mathbb E}_B {\mathbb E}_G {|\sum_{i} \psi_i B_i G_i| }
= {\mathbb E}_I {\mathbb E}_G {|\sum_{i\in I} \psi_i G_i| }
\]
Leveraging the fact that linear transforms of Gaussians are also Gaussian, we get
\[
{\mathbb E}_G {|\sum_{i\in I} \psi_i G_i| } = 
\sqrt{\frac{2}{\pi}} \cdot\|\psi_I\|_2.
\]


\paragraph{Upper and lower bound for symmetric distribution}
It is worth commenting that we could still calculate the upper and lower bound of ${\mathbb E}{| \psi^T X| }$ in terms of ${\mathbb E}_I\| \psi_I\|_2$ for $X_t$ IID sampled from any Bernoulli symmetric $p G +(1-p) \delta_0$, where $G$ is a symmetric distribution. 


\begin{lemma} \label{lem:sym_Khintchine_inequality}
If we only know $\Xi_i$ are IID sampled from a symmetric distribution $F$ with unit variance, then we already have a lower bound,  
\[
     1/\sqrt{2}  \| a\|_2 \leq {\mathbb E} |\sum_i a_i \Xi_i| \leq \|a\|_2.
\] 
\end{lemma}
\begin{proof}[Proof of Lemma \ref{lem:sym_Khintchine_inequality}]
The upper bound come from Cauchy inequality.
 
Now we derive the lower bound. We write $\Xi_i = \sigma_i |\Xi_i|$, where $s_i =\pm 1$ with equal probability since $\Xi_i$ are symmetric RV, and $\{s_i,|\Xi_i|\}$  are all independent random variables. We could use Khintchine inequality,


\begin{eqnarray*}
{\mathbb E}_\Xi |\sum_i a_i \Xi_i| &=& E_{|\Xi_i|}E_{s_i} |\sum_i a_i s_i |\Xi_i||\\
&\geq&  
  1/\sqrt{2}  \sum_i a_i^2  E_{|\Xi_i|} |\Xi_i|^2 \\
  &=&   1/\sqrt{2}  \|a\|_2^2.
\end{eqnarray*}
\end{proof}

Let $\psi_I = a$, we have a corollary for Bernoulli Symmetric case,
\[
 {\mathbb E}_I \|\psi_I\|_2  \geq {\mathbb E}_X{|\sum_{i} \psi_i X_i| }  
\geq \sqrt{\frac{1}{2}} \cdot {\mathbb E}_I \|\psi_I\|_2.
\]


\begin{theorem}
Let $X_i = B_i \Xi_i$ be IID, where $B_i$ is sampled from Bernoulli with parameter $p$, let $I$ be the support where $B_i =1$. And $\Xi_i$ is variance $1$, sampled from: 
\begin{itemize}
    \item $N(0,1)$
   
    \item general symmetric distribution with variance $1$.
\end{itemize}

Then 
\begin{itemize}
    \item(for Bernoulli Gaussian:)
    \[
{\mathbb E}_X{|\sum_{i} \psi_i X_i| }  
= \sqrt{\frac{2}{\pi}} \cdot {\mathbb E}_I \|\psi_I\|_2
\]
    \item (for Bernoulli symmetric R.V. :) 
    \[
{\mathbb E}_I \|\psi_I\|_2 \geq {\mathbb E}_X{|\sum_{i} \psi_i X_i| }  
\geq \sqrt{\frac{1}{2}} \cdot {\mathbb E}_I \|\psi_I\|_2.
\]
\end{itemize}

\end{theorem}

\subsection{Expectation over Bernoulli support}

We previously considered the identity
\begin{equation} \label{eq:startingPoint}
    {\mathbb E} | \sum_{i \in {\cal T}}  \psi_i X_i | = \sqrt{\frac{2}{\pi}} \cdot {\mathbb E}_I \| \psi \|_{\ell_2(I)},
\end{equation}
where $X$ is a Bernoulli-Gaussian RV $BG(p;0,1)$,
and where $I$ is a random subset of the domain ${\cal T}$
determined by Bernoulli-$p$ coin tossing.

Let $J \subset {\cal T}$ be the support of $X$, where $  N:=|{\cal T}|$, $N_J:=|J|$. Let $\psi_J : = \psi\cdot 1_J$ denote the elementwise product of vector $\psi$ with the indicator vector of subset $J$. Then we can consider our problem on the space ${\cal T}$ by default, and rewrite for simplicity
\[
\| \psi_J \|_{2} : = \| \psi_J \|_{\ell_2({\cal T})} = \| \psi \|_{\ell_2(J)}.
\]

This leads us to consider the following ratio:
\[
  V^J_k(\psi) := \frac{\| \psi \|_{\ell_2(J)}^{k}}{\| \psi \|_{\ell_2({\cal T})}^{k}},
 \]
 and its expectation over all Bernoulli random subset,
\[
  V_k(\psi) := \frac{{\mathbb E}_I \| \psi \|_{\ell_2(I)}^{k}}{\| \psi \|_{\ell_2({\cal T})}^{k}},
 \]
 where $I$ is again a random subset.
We remark that the lower bound of $V_k(\psi)$ is the optimal value of the optimization problem:
 \begin{eqnarray*}
    \min&   {\mathbb E}_I  \| \psi_I \|^k_2\\
    \mbox{ subject to }&   \| \psi \|_2 = 1,
\end{eqnarray*}
and the upper bound of $V_k(\psi)$ is the optimal value of the optimization problem
 \begin{eqnarray*}
    \max&   {\mathbb E}_I  \| \psi_I \|^k_2\\
    \mbox{ subject to }&   \| \psi \|_2 = 1.
\end{eqnarray*}




\begin{lemma} \label{lem:BGPropLemmaSub_fix}
Let $\psi$ denote a vector in ${\mathbb R}^N$. For fixed support $J\subset [N]$ then for any $k\neq 0$, the deterministic quantity $V^J_{k}(\psi)$ satisfies:
\[
0 \leq V^J_{k}(\psi) \leq 1.
\]
The upper bound is approached by the vectors $\psi  \in \{  \frac{1}{\sqrt{N}}\sum_{j\in J}\pm e_j\}$, where  $\| \psi_J \|_2^k/\|  \psi \|_2^k =1$,
 and the lower bound is approached by 
any $\psi$ that has all of its entries on $J$ to be zero.
\end{lemma}

\begin{lemma} \label{lem:BGPropLemmaSub_2}
Let $\psi$ denote a vector in ${\mathbb R}^N$.
\[
V_2(\psi) =   p.
\]
\end{lemma}



\begin{theorem} \label{thm:BGPropLemmaNorm}
Let $\psi$ denote a vector in ${\mathbb R}^N$, then
\[
p \leq V_{1}(\psi) \leq V_{1}(\frac{\sum_{j\in [N]}\pm e_j}{\sqrt{N}} ) \leq \sqrt{p} .
\]
the lower bound is approached by on-sparse vectors $\psi \in \{ \pm e_i, i\in [N]\}$, and the upper bound is approached by 
$\psi  \in \{  \frac{1}{\sqrt{N}}\sum_{j\in [N]}\pm e_j\}$.

Furthermore, all the stationary points of $V_{1}(\psi)$ are $\{\frac{\sum_{i\in J} \pm e_i}{\sqrt{N_J}}\}$ for different support $J\subset {\cal T}$, where $\{ \pm e_i, i\in [N]\}$ are the global minimizers, and $\{  \frac{1}{\sqrt{N}}\sum_{j\in [N]}\pm e_j\}$ are the global maximizers. And for $J$ with $1<N_J<N$, $\{\frac{\sum_{i\in J} \pm e_i}{\sqrt{N_J}}\}$ are saddle points with value
\[
V_1(\frac{\sum_{i\in J} \pm e_i}{\sqrt{N_J}}) = {\mathbb E}_I \sqrt{\frac{\sum_{i\in J} 1_{I_i}}{N_J}} = \sum_{j=0}^{N_J} (1-p)^{N_J-j} p^{j} {N_J \choose j}\sqrt{\frac{j}{N_J}}.
\]
\end{theorem}
Remark:
Specifically,
\[
V_1( e_i) = p.
\]
\[
V_1(\frac{ e_i+e_j}{\sqrt{2}}) = p(\sqrt{2}+(1-\sqrt{2}p)).
\]



\begin{theorem} \label{thm:BGPropLemmaSub_k}
For $k\geq 2$, let $\psi$ denote a vector in ${\mathbb R}^N$, then
\[
p^k \leq V_{2k}(\frac{\sum_{j\in [N]}\pm e_j}{\sqrt{N}} ) \leq V_{2k}(\psi) \leq p.
\]
the upper bound is approached by on-sparse vectors $\psi \in \{ \pm e_i, i\in [N]\}$, and the lower bound is approached by 
$\psi  \in \{  \frac{1}{\sqrt{N}}\sum_{j\in [N]}\pm e_j\}$.

Furthermore, all the stationary points of $V_{2k}(\psi)$ are $\{\frac{\sum_{i\in J} \pm e_i}{\sqrt{N_J}}\}$ for different support $J\subset {\cal T}$, where $\{ \pm e_i, i\in [N]\}$ are the set of global maximizers, and $\{  \frac{1}{\sqrt{N}}\sum_{j\in [N]}\pm e_j\}$ are the set of all the global minimizers. And for $J$ with $1<N_J<N$, $\{\frac{\sum_{i\in J} \pm e_i}{\sqrt{N_J}}\}$ are saddle points with value
\[
V_{2k}(\frac{\sum_{i\in J} \pm e_i}{\sqrt{N_J}}) = {\mathbb E}_I (\frac{\sum_{i\in J} 1_{I_i}}{N_J})^k = \sum_{j=0}^{N_J} (1-p)^{N_J-j} p^{j} {N_J \choose j}(\frac{j}{N_J})^k.
\]
\end{theorem}







\paragraph{Tight bound on $V_1$ and $V_{2k}$ using mean and variance of $\frac{\sum_{i\in J} 1_{I_i}}{N_J}$}

Now notice that 
\[
\frac{\sum_{i\in J} 1_{I_i}}{N_J} 
\]
has mean $p$ and variance $\frac{p(1-p)}{N_J}$.
We could define a zero mean unit variance random variable $g^J$ as a function of $\{1_{I_i}\mid i\in J\}$ 
\[
g^J := \left(\frac{\sum_{i\in J} 1_{I_i}}{N_J}-p\right)/ \left( \sqrt{\frac{p(1-p)}{N_J}} \right)
\]
namely, ${\mathbb E}_I \left( g^I\right)=0, {\mathbb E}_I \left( g^I\right)^{2}=1$.
then
\[
\frac{\sum_{i\in J} 1_{I_i}}{N_J} := p + \sqrt{\frac{p(1-p)}{N_J}} g^J = p(1 + \sqrt{\frac{1-p}{p N_J}} g^J)
\]
then
\[
{\mathbb E}_I \sqrt{\frac{\sum_{i\in J} 1_{I_i}}{N_J}} = {\mathbb E}_I\sqrt{p- \sqrt{\frac{p(1-p)}{N_J}}g^I} 
= \sqrt{p} {\mathbb E}_I\sqrt{1+ \sqrt{\frac{1-p}{pN_J}}g^I}  
\]
\[
{\mathbb E}_I \left(\frac{\sum_{i\in J} 1_{I_i}}{N_J}\right)^k 
= p^k {\mathbb E}_I\left({1+ \sqrt{\frac{1-p}{pN_J}}g^I}\right)^k 
\]
\begin{lemma}
\label{TaylorSqrt}
For $ 0 \leq x\leq 1,$
\begin{align}
(1-x)^{1/2}
=1-\sum_{\ell=0}^\infty\frac2{\ell+1}\binom{2\ell}{\ell}\left(\frac x4\right)^{\ell+1}
\end{align}
\end{lemma}

\begin{lemma}[Taylor expansion for $V_1$]
\label{TaylorSqrt_Expect}
Using Taylor expansion, asymptotically when $N_J\rightarrow \infty$, 
\begin{eqnarray*}
\frac{1}{\sqrt{p}}{\mathbb E}_I \sqrt{\frac{\sum_{i\in J} 1_{I_i}}{N_J}} 
&=& {\mathbb E}_I\left({1+ \sqrt{\frac{1-p}{pN_J}}g^I}\right)^{1/2} \\
&=& 1-\sum_{\ell=0}^\infty\frac2{\ell+1}\binom{2\ell}{\ell}2^{-2(\ell+1)}\left( \sqrt{\frac{1-p}{pN_J}}\right)^{\ell+1}{\mathbb E}_I \left( g^I\right)^{\ell+1}\\
&=& 1- 2^{-3}({\frac{1-p}{pN_J}}) +O\left( ({\frac{1-p}{pN_J}})^2 \right).
\end{eqnarray*}
\end{lemma}

\begin{lemma}[Taylor expansion for $V_{2k}$]
\label{TaylorSqrt_Expect_k}
Using Taylor expansion, asymptotically when $N_J\rightarrow \infty$, 
\begin{eqnarray*}
\frac{1}{p^k}{\mathbb E}_I ({\frac{\sum_{i\in J} 1_{I_i}}{N_J}})^k  
&=& {\mathbb E}_I\left({1+ \sqrt{\frac{1-p}{pN_J}}g^I}\right)^{k} \\
&=& 1 +\sum_{\ell=1}^{k} \binom{k}{\ell} ({\frac{1-p}{pN_J}})^{\ell/2} {\mathbb E}_I (g^I)^{\ell}\\
&=& 1 + \frac{k(k-1)}{2} ({\frac{1-p}{pN_J}}) +O\left( ({\frac{1-p}{pN_J}})^2 \right).
\end{eqnarray*}
\end{lemma}
Therefore, we know what when $N_J>>1$, 
$V_1(\frac{\sum_{i\in J} 1_{I_i}}{N_J})$ is close to its upper bound $\sqrt{p}$, 
and for $k\geq 2$, 
$V_k(\frac{\sum_{i\in J} 1_{I_i}}{N_J})$ is close to its lower bound $p^k$. Namely 
\[
{\mathbb E}_I \sqrt{\frac{\sum_{i\in J} 1_{I_i}}{N_J}} \approx \sqrt{p},
\]
\[
{\mathbb E}_I (\frac{\sum_{i\in J} 1_{I_i}}{N_J})^k \approx p^k.
\]






\begin{figure} 
\includegraphics[width=.7\textwidth]{./figures_BD/Ju_landscape.png}
\caption{The value of $V_1$ on a two-dimensional sphere in ${\mathbb R}^3$, normalized by the affine transform to send the value in $[0,1].$  
Figure credit to Ju Sun's talk based on the paper \cite{bai2018subgradient}
}
\label{fig-landscape-V1}
\end{figure}





\begin{figure} 
\centering
\includegraphics[width=.7\textwidth]{./figures_BD/ratio/Full_compare_with_baseline.png}
\caption{The value of $V_1(\psi)$ for different $\psi$, $x-$axis is the sparsity parameter $p$, $y-$axis is $V_1(\psi)$. }
\label{fig-V1}
\end{figure}

\begin{figure} 
\centering
\includegraphics[width=.7\textwidth]{./figures_BD/ratio/L4_plot.png}
\caption{The value of $V_4(\psi)$ for different $\psi$, $x-$axis is the sparsity parameter $p$, $y-$axis is $V_4(\psi)$. }
\label{fig-V4}
\end{figure}



\begin{figure} 
\centering
\includegraphics[width=.7\textwidth]{./figures_BD/ratio/L8_plot.png}
\caption{The value of $V_8(\psi)$ for different $\psi$, $x-$axis is the sparsity parameter $p$, $y-$axis is $V_8(\psi)$. }
\label{fig-V4}
\end{figure}



\subsection{Proofs}


\begin{proof}[Proof of Theorem~\ref{thm:BGPropLemmaNorm}]
First, we show that the upper and lower bound value we give is achievable.
\begin{itemize}
\item 
When $\psi = e_0$,  
\[
{\mathbb E}_I  \| (e_0)_I \|_2=p\| (e_0)\|_2+(1-p)\| 0\|_2 = p.
\]
\item
When $\psi = \frac{1}{\sqrt{N}}\sum_{j\in [N]}\pm e_j,$ then  
\[
V_1(\psi) = {\mathbb E}_I \sqrt{\frac{\sum_{i\in [N]} 1_{I_i}}{N}} \leq   \sqrt{{\mathbb E}_I \frac{\sum_{i\in [N]} 1_{I_i}}{N}} = \sqrt{p}.
\]
The inequality comes from Jensen's inequality, since square root function is concave.
\item
For fixed support $J$, when $\psi = \frac{1}{\sqrt{N_J}}\sum_{i\in J} e_i,$  then $\| \psi_J \|_2/\|  \psi \|_2 =1.$
However, the expectation is going to be smaller for $p<1$. 
\[
V_1(\psi) = {\mathbb E}_I \sqrt{\frac{\sum_{i\in J} 1_{I_i}}{N_J}} = \sum_{k=0}^{N_J} (1-p)^{N_J-k} p^{k} {N_J \choose k}\sqrt{\frac{k}{N_J}}.
\]
\end{itemize}

This problem can be reformulated as projection pursuit with sphere constraint,
 \begin{eqnarray*}
    \min&   {\mathbb E}_I  \| \psi_I \|_2\\
    \mbox{ subject to }&   \| \psi \|_2 = 1.
\end{eqnarray*}
Then from \cite{bai2018subgradient} Proposition $3.3$, we get the result. 

The main idea of the proof is to calculate the projected gradient
for $q  \in \{\frac{\sum_{i\in J} \pm e_i}{\sqrt{N_J}}\}$, when $N_J = M,$
\[
e_j^T  {\mathbb E}_I  \partial\| (q)_I \|_2 = q_j [\sum_{k=0}^{M} (1-p)^{M-k} p^{k} {M \choose k}\sqrt{\frac{k}{M}}] ,
\]
therefore,
\[
  {\mathbb E}_I  \partial\| (q)_I \|_2 =  [\sum_{k=0}^{M} (1-p)^{M-k} p^{k} {M \choose k}\sqrt{\frac{k}{M}}] q,
\]
so
\[
(I-q q^T)  {\mathbb E}_I  \partial\| q_I \|_2 = [\sum_{k=0}^{M} (1-p)^{M-k} p^{k} {M \choose k}\sqrt{\frac{k}{M}}]q -[\sum_{k=0}^{M} (1-p)^{M-k} p^{k} {M \choose k}\sqrt{\frac{k}{M}}]q  = 0.
\]
then $q \in \{\frac{\sum_{i\in J} \pm e_i}{\sqrt{N_J}}\}$ are stationary points.
 
The other direction (all other points are not stationary) is implied by (the proof of) Theorem $3.4$ in  \cite{bai2018subgradient}.
\end{proof}





\begin{proof}[Proof of Lemma~\ref{TaylorSqrt}]
From generalized binomial theorem, we know that for $ 0 \leq x\leq 1,$
\begin{equation}
(1-x)^{1/2}
=\sum_{\ell=0}^\infty\binom{1/2}{\ell}(-x)^\ell.
\end{equation}

\begin{align}
\binom{1/2}{\ell}
&=\frac{\frac12(\frac12-1)(\frac12-2)\cdots(\frac12-\ell+1)}{\ell!}\\
&=\frac{(-1)^{\ell-1}}{2^\ell\ell!}1\cdot3\cdot5\cdots(2\ell-3)\\
&=\frac{(-1)^{\ell-1}}{2^\ell\ell!}\frac{(2\ell-2)!}{2^{\ell-1}(\ell-1)!}\\
&=\frac{(-1)^{\ell-1}}{\ell2^{2\ell-1}}\binom{2\ell-2}{\ell-1},
\end{align}
then
\begin{align}
(1-x)^{1/2}
&=1-\sum_{\ell=1}^\infty\frac2\ell\binom{2\ell-2}{\ell-1}\left(\frac x4\right)^\ell\\
&=1-\sum_{\ell=0}^\infty\frac2{\ell+1}\binom{2\ell}{\ell}\left(\frac x4\right)^{\ell+1}.
\end{align}
\end{proof}







\begin{proof}[Proof of Lemma~\ref{TaylorSqrt_Expect}]
\begin{eqnarray*}
{\mathbb E}_I\sqrt{1- \sqrt{\frac{1-p}{pN_J}}g^I}  
&=& 1-\sum_{\ell=0}^\infty\frac2{\ell+1}\binom{2\ell}{\ell}2^{-2(\ell+1)}\left( \sqrt{\frac{1-p}{pN_J}}\right)^{\ell+1}{\mathbb E}_I \left( g^I\right)^{\ell+1}\\
&=& 1- 2^{-3}({\frac{1-p}{pN_J}}) - O(({\frac{1-p}{pN_J}})^2)
\end{eqnarray*}
Using the central limit theorem, when $N_J\rightarrow \infty$, we have normal approximation for $g^J$ so that
\[
\frac{\sum_{i\in J} 1_{I_i}}{N_J} \sim N(p,  \sqrt{\frac{p(1-p)}{N_J}}),
\]
then for $G\sim N(0,1),$ asymptotically when $N_J\rightarrow \infty$,
\[
{\mathbb E}_I \sqrt{\frac{\sum_{i\in J} 1_{I_i}}{N_J}} \approx  \sqrt{p} {\mathbb E}_G\sqrt{1- \sqrt{\frac{1-p}{pN_J}}G} =
\sqrt{p} \left( 1- 2^{-3}(1-p)^{2}  N_J^{-1} + O((1-p)^{4}  N_J^{-2})\right),
\]
where
\begin{eqnarray*}
{\mathbb E}_I\sqrt{1- \sqrt{\frac{1-p}{pN_J}}g^I}  
&\approx &{\mathbb E}_G\sqrt{1- \sqrt{\frac{1-p}{pN_J}}G}\\
&=& 1-\sum_{\ell=0}^\infty\frac2{\ell+1}\binom{2\ell}{\ell}2^{-2(\ell+1)}\left( \sqrt{\frac{1-p}{pN_J}}\right)^{\ell+1}{\mathbb E}_G \left( G\right)^{\ell+1}\\
&=& 1-\sum_{\ell \mbox{ is odd}}^\infty\frac2{\ell+1}\binom{2\ell}{\ell}2^{-2(\ell+1)}\left( \sqrt{\frac{1-p}{pN_J}}\right)^{\ell+1} (\ell)!!\\
&=& 1-\sum_{s=1}^\infty\frac1{s}\binom{4s-2}{2s-1}2^{-4s} (2s-1)!! (p^{-1}-1)^{s}  N_J^{-s}\\
&=& 1-\sum_{s=1}^\infty\frac1{s}\frac{(4s-2)!}{(2s-1)!(2s-1)!}2^{-4s} \frac{(2s-1)!}{2^{s-1}(s-1)!} (p^{-1}-1)^{s}  N_J^{-s}\\
&=& 1-\sum_{s=1}^\infty\frac{(4s-2)!}{(2s-1)!(s)!}2^{-5s+1}  (p^{-1}-1)^{s}  N_J^{-s}.
\end{eqnarray*}
\end{proof}





\subsection{$E{| \psi^T X| }$ for IID Bernoulli Symmetric $X$ }
we study in detail for the quantity  
\[
E{| \psi^T X| } = E{|\sum_{i} \psi_i X_i| }.
\]
We know when $X_i$ have variance $\sigma^2$,
\[
   E (\sum_i \psi_i X_i)^2 =  \sum_i   \psi_i^2 E(X_i)^2 = \sigma^2 \|\psi\|_2^2.
\]
The ratio indicate the sparsity level of random variable $ \sum_i \psi_i X_i$
\[
\frac{ E{|\sum_{i} \psi_i X_i| }}{\sqrt{E (\sum_i \psi_i X_i)^2}} = \frac{ E{|\sum_{i} \psi_i X_i| }}{\sigma^2 \|\psi\|_2} 
\]
Now, we consider $E{|\sum_{i} \psi_i X_i| }$ for general symmetric setting when lower bound can be derived.

\paragraph{Exact calculation for Bernoulli Gaussian}
Let $X_i = B_i Z_i$ be IID Bernoulli Gaussian, where $B_i$ is sampled from Bernoulli with parameter $p$, $Z_i$ is sampled from $N(0,1)$. Let $I$ be the support where $B_i =1$.
\[
{\mathbb E}_X{|\sum_{i} \psi_i X_i| }  
= {\mathbb E}_B {\mathbb E}_G {|\sum_{i} \psi_i B_i G_i| }
= {\mathbb E}_I {\mathbb E}_G {|\sum_{i\in I} \psi_i G_i| }
\]
Leveraging the invariance property for linear transform of Gaussian, we get
\[
{\mathbb E}_G {|\sum_{i\in I} \psi_i G_i| } = 
\sqrt{\frac{2}{\pi}} \cdot\|\psi_I\|_2.
\]


\paragraph{Symmetric distribution}

Dave Comment:
Put the generalization in the end.
Focus on Bernoulli Gaussian for the main part, and then talk about extension.


\begin{lemma} \label{lem:sym_Khintchine_inequality}
For we only know $\Xi_i$ are IID sampled from a symmetric distribution $F$ with unit variance, then we already have a lower bound,  
\[
     1/\sqrt{2}  \| a\|_2 \leq {\mathbb E} |\sum_i a_i \Xi_i| \leq \|a\|_2.
\] 
\end{lemma}
 \begin{proof}[Proof of Lemma \ref{lem:sym_Khintchine_inequality}]
 The upper bound come from Cauchy inequality.
 
Now we derive the lower bound. We write $\Xi_i = \sigma_i |\Xi_i|$, where $s_i =\pm 1$ with equal probability since $\Xi_i$ are symmetric RV, and $\{s_i,|\Xi_i|\}$  are all independent random variables.  
we could use Khintchine inequality,
here is the Wiki link for more detail:
\begin{quote}
\href{https://en.wikipedia.org/wiki/Khintchine_inequality}{Khintchine inequality}.
\end{quote}




\begin{eqnarray*}
{\mathbb E}_\Xi |\sum_i a_i \Xi_i| &=& E_{|\Xi_i|}E_{s_i} |\sum_i a_i s_i |\Xi_i||\\
&\geq&  
  1/\sqrt{2}  \sum_i a_i^2  E_{|\Xi_i|} |\Xi_i|^2 \\
  &=&   1/\sqrt{2}  \|a\|_2^2.
\end{eqnarray*}
\end{proof}

Let $\psi_I = a$, we have a corollary for Bernoulli Symmetric case,
   \[
 {\mathbb E}_I \|\psi_I\|_2  \geq {\mathbb E}_X{|\sum_{i} \psi_i X_i| }  
\geq \sqrt{\frac{1}{2}} \cdot {\mathbb E}_I \|\psi_I\|_2.
\]


\begin{theorem}
Let $X_i = B_i \Xi_i$ be IID, where $B_i$ is sampled from Bernoulli with parameter $p$, let $I$ be the support where $B_i =1$. And $\Xi_i$ is variance $1$, sampled from: 
\begin{itemize}
    \item $N(0,1)$
   
\item general symmetric distribution with variance $1$.
\end{itemize}

Then 
\begin{itemize}
    \item(for Bernoulli Gaussian:)
    \[
{\mathbb E}_X{|\sum_{i} \psi_i X_i| }  
= \sqrt{\frac{2}{\pi}} \cdot {\mathbb E}_I \|\psi_I\|_2
\]
\item (for Bernoulli symmetric R.V. :) 
    \[
{\mathbb E}_X{|\sum_{i} \psi_i X_i| }  
\geq \sqrt{\frac{1}{2}} \cdot {\mathbb E}_I \|\psi_I\|_2.
\]
\end{itemize}

\end{theorem}








\section{Introduction}
\subsection{Introducing the blind deconvolution problem}
Blind deconvolution is the problem of recovering a filter and a signal from their (noisy or corrupted) convolution.
This is a ubiquitous problem, occuring in fields across signal processing, machine learning, computer vision, seismology, and neuro-science.
Blind deconvolution is also a crucial problem at the core of digital communication that powers our modern society by enabling cell phone communication.

One important example of blind deconvolution occurs when the signal is sparse. We call this problem \emph{sparse blind deconvolution}. The sparse signal setting of blind deconvolution appears in seismology, microscopy, astronomy, neuro-science spike identification. In representation learning for computer vision, computational imaging and speech analysis, sparse blind deconvolution is also interpreted as single-channel convolutional dictionary learning.

Despite the importance of blind deconvolution, until recently there has been little understanding of when, how and why these non-convex algorithms succeed.
Practitioners have accumulated poetic yet vague observations like "when the initialization is close, gradient-based methods find the true signal", but there is no clear understanding of this statement.

This work demonstrates that a convex formulation hiding behind the scene can explain the success of this poetic heuristic. Our mathematical formulation gives a crisp, clean view that clears all the fog in the claims of past practitioners. 

\subsection{Previous works on the blind deconvolution problem}
\paragraph{Non-convexity for blind deconvolution problems}
Suppose there is a sparse signal $X$ and a forward filter $a$, the observation is their convolution $Y$, defined as $Y = a * X$. Blind deconvolution is the problem of finding the sparse signal $X$ from the observation $Y$ without knowing the forward filter $a$. 

As both the sparse signal and the forward filter $a$ are unknown, the problem is bilinear in $a$ and $X$. And the expression $a*x$ has scaling and shift symmetry for $(a,x)$.
Therefore, blind deconvolution is often formulated as a non-convex problem. 

\paragraph{Searching the inverse filter}
Instead of trying to recover $a$ and $X$ together from $a*X$, 
we could formulate this problem as looking for an approximate inverse  filter $w$ so that $w*Y$ is special or \textit{simple}. 

 To make our discussion concrete, we define inverse filter formally.
  
\paragraph{Inverse filter }
Let $e_0$ be the sequence with $1$ at $0$ coordinate and $0$ elsewhere, then it is the unit of convolution.  For the forward filter $a\in l_1({\mathbb Z}) $, we want to give a condition that guarantees there exists an inverse filter $a^{-1}\in l_1({\mathbb Z})$ so that $a^{-1}*a = e_0.$

This condition is that $a\in l_1({\mathbb Z}) ,  
(\mathcal{F} a)(\omega) \neq 0, \forall \omega\in T.$ 
From Wiener's lemma, we know that this condition on $a$ would ensure that $a$ is invertible and the inverse filter $a^{-1} \in l_1({\mathbb Z})$.
\begin{lemma}
If $a\in l_1({\mathbb Z}) ,  
(\mathcal{F} a)(\omega) \neq 0, \forall \omega\in T,$ we could define the inverse filter of $a$ as:
$$a^{-1} := \mathcal{F}^{-1}(\frac{1}{\mathcal{F} a})$$
so that $a^{-1}*a = e_0.$
From Wiener's lemma on $\mathcal{F} a$,   $a^{-1} \in l_1({\mathbb Z})$ .
\end{lemma}
 
\paragraph{Minimal entropy deconvolution}
This formulation of searching inverse filter is adapted in the earlier work of Wiggins \cite{wiggins1978minimum} and Donoho \cite{donoho1981minimum} for minimum entropy deconvolution, or maximum high order cumulant deconvolution, with seismology as the motivating application. The core idea is to find an approximate inverse filter $w\approx a^{-1}$, so that $w*Y$ is the most non-Gaussian.
Namely, we want to find $w\neq 0$, so that  
\[
 \begin{array}{ll}
 \underset{w\neq 0}{\mbox{maximize}}   & J(w*Y)
\end{array}
 \]
  Here the non-Gaussianity objective $J$ could be the negative entropy $J(z) = -\mbox{Entropy}(p_z)$, the normalized $4$-th order cumulant, $J(z) = {\|z\|_4^4}/{\|z\|_2^4}$, or the varimax norm: $J(z) = \|z\|_\infty/\|z\|_2$ (proposed in the paper \cite{cabrelli1985minimum,wiggins1978minimum,donoho1981minimum}).
  \paragraph{Blind equalization}
  There are a series of digital communication work on blind deconvolution when the digital signal is discrete, which is called \textbf{blind equalization}. Examples of digital signals are PAM signaling, which is a $\pm 1$ binary signal, and QAM signaling, where the signal has equally spaced points on unit sphere in complex space \cite{kennedy1992blind,ding2000fast}.  
  In the work \cite{vembu1994convex}, using the condition that $X$ are IID and bounded by a known constant, for example, $ \|X\|_\infty =1$ after rescaling, the authors propose to solve a non-convex problem
   \[
 \begin{array}{ll}
 \underset{w}{\mbox{maximize}}   &  
 \|w * Y\|_8 
 \\
\mbox{subject to} 
& \|w*Y\|_2 = 1
\end{array}
  \]
using projected gradient descent, where the $\ell_8$ norm objective is used as an approximation of the  $\ell_\infty$ norm objective $\|w * Y\|_\infty $.
Later, the work \cite{ding2000fast} uses linear programming to solve the $\infty$ norm problem directly.

 

\section{Main results overview}
\subsection{Convex formulation of sparse blind deconvolution}






\paragraph{Scalable convex formulation for finding sparsing transform}
Now we propose a scalable convex optimization framework for blind deconvolution.

First, we set up the finite-sample observation model. Given a series of observations $(Y_t)_{t\in [-(T+k),T+k]}$ from a centered window of radius $T+k$, we want to find an inverse filter 
$w = (w_{-k}, \ldots, w_0, \ldots, w_k) \in V$ such that the convolution $w*Y$ is sparse, where $V$ is a $2k+1$ dimensional subspace of the doubly infinite sequences. Let $\|\cdot\|_{1,[-T,T]} =\|\cdot\|_{\ell_1({\cal T})} $ be the $1$-norm over the centered window of radius $T$. Let $N:= 2T+1$ be the size of the window ${\cal T}$.

In the searching for inverse filter framework, if we look for an approximate inverse filter $w$ so that $w*Y$ is sparse, then we could formulate the problem as 
\[
 \begin{array}{ll}
 \underset{w\neq 0}{\mbox{maximize}}   & \|w*Y\|_{\ell_0({\cal T})}
\end{array}
  \]
  
First, we could do convex relaxation of the $\ell_0$ norm, replacing it with the $\ell_1$ norm we have a convex objective.

Second, we need to fix the scale of $w$ to break the scale symmetry.

For a given matrix $A$, we set up the following optimization problem to fix the scale of $w$:
\begin{equation}
 \begin{array}{ll}
 \underset{w}{ \mbox{minimize} }   & \frac{1}{N} \| w*Y\|_{\ell_1({\cal T})}\\
\mbox{subject to}  
& \|Aw\|_\infty =1, 
\end{array}
\end{equation}

This is equivalent to finding the minimum objective solution out of $k$ convex optimization problems:

\begin{equation}
 \begin{array}{ll}
 \underset{w}{ \mbox{minimize} }   & \frac{1}{N} \| w*Y\|_{\ell_1({\cal T})}\\
\mbox{subject to}  
& A_i w =1, \|Aw\|_\infty \leq 1, 
\end{array}
\end{equation}

In the case of perfect shift symmetry of convolution (circular convolution for finite observation or linear convolution on double infinite sequence), when $A$ is a circular matrix with the first column vector  
$\tilde{a}$
, then we only need to
solve one convex problem.
 $w$:
\begin{equation}
 \begin{array}{ll}
 \underset{w}{ \mbox{minimize} }   & \frac{1}{N} \| w*Y\|_{\ell_1({\cal T})}\\
\mbox{subject to}  
& (\Tilde{a}*w)_0 =1, 
\end{array}
\end{equation}



 This formulation is a convex formulation for the $\ell_\infty$ box constraint with a fixed location of largest entry under shift:
Based on the fact that $a^{-1}*a = e_0$, we know $(a^{-1}*a)_0 = 1$ is a constraint on the largest entry. It would be an oracle constraint if we know $a$. In practice, we could choose an initial guess  $\Tilde{a}$ of the forward filter.





\subsection{Phase transition theorem for blind deconvolution
}
\paragraph{Random model for sparse signals}
For better theoretical analysis, we define a random model for sparse signals. Let $(X_t)$ be an independent
sequence of random variables indexed by $t \in Z$, with a symmetric distribution with CDF $F_t$, such that $F_t(x) = 1 - F_t(-x)$. 
And we assume $F_0$ has an atom at $0$, $F_0 = (1-p) H +p G$, where $H$ is the Heaviside distribution and $G$ is the standard Gaussian distribution, we called it the \emph{Bernoulli$(p)$-Gaussian} model. 

\paragraph{Surprising phase transition for sparse blind deconvolution with convex optimization}

The convex optimization problem exhibits a phase transition phenomenon for a ideal sparse signal modeled by random sparse process with independent entries. 

To demonstrate the flavor of the phase transition, we present the population phase transition theorem for the special case when the forward filter is $(\ldots,s^3, s^2,s, 1 )$ with $|s|\leq 1$, then the inverse filter is a basic short filter  $a^{-1} = ( -s, 1 )$. 
\begin{theorem}
For a process $Y = a * X$ with
\begin{itemize}
\item
$a = (\ldots,s^3, s^2,s, 1 )$ then $a^{-1} = ( -s, 1 )$ where $|s|\leq 1$; and
\item 
$X_t$ IID sampled from the \emph{Bernoulli$(p)$-Gaussian} distribution 
$p N(0,1) +(1-p) \delta_0$.
\end{itemize}
 Let $w^\star$ be the solution of the convex optimization problem 
 \[
 \begin{array}{ll}
 \underset{w }{\mbox{minimize}}   & {\mathbb E}|(w*Y)_0|\\
\mbox{subject to}  & w_0 =1;
\end{array}
  \]
 then  there exists a threshold
\[
p^\star  = 1-|s| ,
\]
$w^\star$ is $a^{-1}$ up to shift and scaling provided $p<p^\star$, and $w^\star$ is not $a^{-1}$ up to shift and scaling provided $p> p^\star$.
  \end{theorem}
  
Then we could empirically verify the phase transition, by repeatedly running the finite-sample experiments with independent randomness and count the probability of successful recovery of the true inverse filter $a^{-1}$ and sparse signal $X$.
The resulting plot of success probability is in figure~\ref{fig-PT-finite}, where we get a phase transition line (logistic-regressed $50\%$ success probability line). Indeed $p^\star  = 1-|s|$.
\begin{figure} 
\centering
\includegraphics[width=.8\textwidth]{./figures_BD/PT_BD_1root_none.png}
\caption{ The finite sample phase transition diagram when ground truth filter is $w^\star = (-s,1,0)$ with $T=200$. Here the horizontal axis is sparsity level $p$ ranging from $0.01$ to $0.99$, the vertical axis is $|s|$, ranging from $0$ to $0.99$.  The red region is failure and the blue region is success. }
\label{fig-PT-finite}
\end{figure}


\paragraph{Phase transition for sparse blind deconvolution with general filter}
Now we calculate the phase transition condition of blind deconvolution of general inverse filter.
	\begin{theorem}
	For a process $Y = a * X$ with
$X_t$ IID sampled from the \emph{Bernoulli$(p)$-Gaussian} distribution 
$p N(0,1) +(1-p) \delta_0$.
 Let $w^\star$ be the solution of the convex optimization problem 
 \[
 \begin{array}{ll}
 \underset{w }{\mbox{minimize}}   & {\mathbb E}|(w*Y)_0|\\
\mbox{subject to}  
& w_0 =1,
\end{array}
  \]
 then  there exists threshold $p^\star$ determined by
\[
\frac{p^\star}{1-p^\star}=  \inf_{\| \beta\|_2=1, u^T \beta =0} \tan(\angle( \beta_{(0)}, -e_0)) V_1(\beta_{(0)})
\]
 $w^\star$ is $a^{-1}$ up to shift and scaling provided $p<p^\star$,
and  $w^\star$ is not $a^{-1}$ up to shift and scaling provided $p> p^\star$.

The threshold $p^\star$ satisfies 
		\[ 
\tan(\angle( P_u^\perp (-e_0), -e_0)) p^\star \leq \frac{p^\star}{1-p^\star} \leq	\tan(\angle( P_u^\perp (-e_0), -e_0)) V_1(P_u^\perp (-e_0))
		\]
	\end{theorem}

The proof of this theorem is given in the later section. 
To understand the phase transition, we will look at  the population version of the problem.



We could reformulate this theorem into a theorem above stochastic angle:

	\begin{theorem}
 Let the true inverse filter be $a^{-1}$ of length $k$, the angle distance between initialization and ground truth be $\theta$.
For $N$ samples of $X \overset{iid}{\sim} p N(0,1) +(1-p) \delta_0$, if $N \geq  O(k\log(k)),$ \\ then  with probability $1-e^{-CN}$, the solution $w^\star$ of $C_{A}^{\infty}$ satisfies 
$$
  \begin{cases}
w^\star =a^{-1}& \theta \geq \theta^\star(p) \\
w^\star \neq a^{-1}& \theta < \theta^\star(p)
\end{cases}
$$
	\end{theorem}

\section{A fundamental phase transition for blind deconvolution and dictionary learning (also called sparse projection pursuit) :  }
In this section, we will show that after change of variables, the population version of blind deconvolution is the same as the population version of sparse projection pursuit, which is a convex formulation used for dictionary learning or sparse blind source separation. 

Therefore, the phase transition phenomenon presented above is a fundamental phase transition that works for a simple but fundamental population convex problem.


\paragraph{Change of variable for  convex sparse blind deconvolution}
We could rewrite the population version of our convex sparse blind deconvolution problem, with the population objective 
${\mathbb E}\frac{1}{N} \| w*Y\|_{\ell_1({\cal T})} ={\mathbb E}|(w*Y)_0|={\mathbb E}|(w*a*X)_0|$ due to  the ergodic property of stationary process and shift invariance. Then the convex problem becomes 
\[
 \begin{array}{ll}
 \underset{\psi}{\mbox{minimize}}   & {\mathbb E}|(w*a*X)_0|\\
\mbox{subject to}  
& ((\alpha*a^{-1})*(a*w))_0 =1,
\end{array}
 \]
Let $\psi$ be the flipped version of $a*w$, such that $\tilde{\psi} := a*w$, then 
 $(w*Y)_0 = (w*a*X)_0 = (\tilde{\psi}*X)_0 = \psi^T X.$  
  Let $u:= \alpha*a^{-1}$. 
 Now we arrive at a simple and fundamental  population convex problem: 
 \[
 \begin{array}{ll}
 \underset{\psi}{\mbox{minimize}}   & {\mathbb E}|\psi^T X|\\
\mbox{subject to}  
& u^T \psi =1.
\end{array}
\tag{$P_{u}$ }
  \]
In the following, we show that this population convex problem is also the population convex problem of sparse projection pursuit after change of variables.

\paragraph{Sparse projection pursuit and dictionary learning}
Sparse projection pursuit is the problem of finding  one sparse element $d^T Y$ each time from a high dimensional data matrix $Y\in {\mathbb R}^{n\times M}$. Here $Y= D X$ is a matrix product of dictionary $D\in {\mathbb R}^{n\times d}$ and sparse coefficient $X\in {\mathbb R}^{d\times M}$. 

The projection pursuit problem was studied in \cite{friedman1974projection,friedman1981projection,friedman1987exploratory,huber1985projection,jones1987projection}. It is also related to blind source separation, independent component analysis (ICA) and the cocktail-party problem.
In the case of an orthogonal dictionary $D$, we could simply transform the problem into finding its inverse $D^{-1} = D^T$, such that $D^T Y = X$ sparse. A work by Daniel Spielman and John Wright\cite{spielman2012exact} used sparse projection pursuit formulation to find the orthogonal basis of $D$ one-by-one, by proposing a set of rule to select the initial guess $v$ and give theoretical guarantee of the convex method. In this work, due to this common population problem, the phase transition phenomenon would also be applicable to dictionary learning, as the first phase transition result for dictionary learning.

The sparse projection pursuit formulation is to solve the following linear-constrained convex optimization problem for some vector $v$:
\[
 \begin{array}{ll}
 \underset{q }{\mbox{minimize}}   & \frac{1}{M}\|q^T Y\|_{1}\\
\mbox{subject to}  & v^T q =1.
\end{array}
  \]
 
  
The population version is 
\[
 \begin{array}{ll}
 \underset{d }{\mbox{minimize}}   & {\mathbb E}|q^T D X|\\
\mbox{subject to}  & v^T q =1.
\end{array}
  \]
We could also do a change of variable by assuming $\psi:= D^T q$, $u:=D^T q$, then the population sparse projection pursuit also becomes
   \[
 \begin{array}{ll}
 \underset{\psi}{\mbox{minimize}}   & {\mathbb E}|\psi^T X|\\
\mbox{subject to}  & u^T \psi =1.
\end{array}
\tag{$P_{u}$ }
  \]
Without loss of generality, we could assume $\|u\|_2=1$ by rescaling.
  
  
  
  
  
  








\paragraph{General phase transition for projection pursuit}
Now we state the general phase transition with light assumption. Here we don't need to even assume that $X$ are independent, we just need to know the existence of one sparse element $X_0$.

\begin{theorem}[Projection pursuit to find a sparse element]
Given a sequence $\{X_t\}_{t\in {\mathbb Z}}$ where one element $X_0$ is a sparse element with $p G + (1-p) \delta_0$ for a symmetric distribution $G$.
Let $\psi^\star$ be the solution of  $P_{u}$:
\[
 \begin{array}{ll}
 \underset{\psi}{\mbox{minimize}}   & {\mathbb E}|\psi^T X|\\
\mbox{subject to}  
& u^T \psi =1
\end{array}
\tag{$P_{u}$ }
  \]
then there exists a threshold
$
p^\star
$,
$\psi^\star = e_0$ provided $p<p^\star$,
and  $\psi^\star \neq e_0$ provided $p> p^\star$.

Here $p^\star$ is determined by $G$, the distribution of $\{X_t, t\neq 0\}$ and the direction of $u$:
  \[
  \frac{p^\star}{1-p^\star} =  \inf_{\| \beta\|_2=1, u^T \beta =0} \frac{{\mathbb E}[|X_{(0)}^T \beta_{(0)}| \mid X_0 =0]}{{\mathbb E}[|G|]\|\beta_{(0)}\|_2} \tan(\angle( \beta, -e_0))
  \]
 where $\beta_{(0)}$ is $\beta$ with the $0-$th entry deleted.
   

  \end{theorem}

\begin{proof}
The phase transition condition proof is based on  calculating the directional derivative.

First, we want to find a threshold $p^\star$ such that for all $p<p^\star$, the directional derivative at $\psi = e_0$ is non-negative along all direction $\beta$ on unit sphere such that $u^T \beta =0$:
	 \[
	 \nabla_{\beta}{{\mathbb E}|(e_0+\beta)^T X|}\geq 0
	 \]
Second, we calculate $\nabla_{\beta}{{\mathbb E}|(e_0+\beta)^T X|} $ based on whether $X_0$ is zero or not:
	 \[
	 \begin{array}{ll}
	     \nabla_{\beta}{{\mathbb E}|(e_0+\beta)^T X|} = p \beta_0 {\mathbb E}[|X_{0}| \mid X_0 \neq 0]  + (1-p) {\mathbb E}[|X_{(0)}^T \beta_{(0)}| \mid X_0 =0]
	 \end{array}
	\]
It is non-negative in case either $\beta_{0} > 0$, or in case $\beta_0 < 0$ and
	\[
	\frac{p}{1-p} < \frac{{\mathbb E}[|X_{(0)}^T \beta_{(0)}| \mid X_0 =0]}{{\mathbb E}[|G|]||\beta_0|}
	\]
	for all $\beta \in \{\| \beta\|_2=1, u^T \beta =0\}$.
$p^\star$ is the least upper bound of all $p$ satisfying this inequality. 
Therefore,  
	\[
	\frac{p^\star}{1-p^\star} =  \inf_{\| \beta\|_2=1, u^T \beta =0} \frac{{\mathbb E}[|X_{(0)}^T \beta_{(0)}| \mid X_0 =0]}{{\mathbb E}[|G|]|\beta_0|} =  \inf_{\| \beta\|_2=1, u^T \beta =0} \frac{{\mathbb E}[|X_{(0)}^T \beta_{(0)}| \mid X_0 =0]}{{\mathbb E}[|G|]\|\beta_{(0)}\|_2} \tan(\angle( \beta, -e_0))
	\]
\end{proof}



\paragraph{Special case: geometric phase transition condition for sparse Bernoulli Gaussian $X_0$ and Gaussian $\{X_t, t\neq 0\}$}

Now we study the implication of the general phase transition problem in a simplest projection pursuit case, where $X_t$ are independent, $X_0$ is sampled from Bernoulli Gaussian $p N(0,1) +(1-p) \delta_0,$ and the rest $\{X_t, t\neq 0\}$ are sampled from Gaussian $N(0,1)$.

\begin{theorem}
Let $X_t$ be independent, $X_0$ is sampled from Bernoulli Gaussian $p N(0,1) +(1-p) \delta_0,$ the rest $\{X_t, t\neq 0\}$ are sampled from Gaussian $N(0,1)$.
Let $\psi^\star$ be the solution of  $P_{u}$:
\[
 \begin{array}{ll}
 \underset{\psi}{\mbox{minimize}}   & {\mathbb E}|\psi^T X|\\
\mbox{subject to}  
& u^T \psi =1
\end{array}
\tag{$P_{u}$ }
  \]
then there exists a threshold
$
p^\star
$,
$\psi^\star = e_0$ provided $p<p^\star$,
and  $\psi^\star \neq e_0$ provided $p> p^\star$.

Here $p^\star$ is determined by the direction of $u$:
  \[
  \frac{p^\star}{1-p^\star} = \inf_{\| \beta\|_2=1, u^T \beta =0}\tan(\angle( \beta, -e_0)) = \cot(\angle(u, e_0))
  \]
  \end{theorem}
\begin{proof}
First, when the rest of $X_t, t\neq 0$ is independent of $X_0$, and IID sampled from $ N(0,1),$
	\[
 \frac{{\mathbb E}[|X_{(0)}^T \beta_{(0)}| \mid X_0 =0]}{\|\beta_{(0)}\|_2} = 
\sqrt{\frac{2}{\pi}}
\]
Since  
\[
	 {\mathbb E}[|G|] = \sqrt{\frac{2}{\pi}},
	 \]
we have a simple phase transition condition that is completely geometric.
	 \[
  \frac{p^\star}{1-p^\star} = \inf_{\| \beta\|_2=1, u^T \beta =0}\tan(\angle( \beta, -e_0))
  \]
 
Second, we know that there is a geometric structure:
\[
\inf_{\| \beta\|_2=1, u^T \beta =0} \frac{\|\beta_{(0)}\|_2}{|\beta_0|}
= \inf_{\| \beta\|_2=1, u^T \beta =0}\tan(\angle( \beta, -e_0)) = \cot(\angle(u, e_0)),
\]
as indicated by figure~\ref{fig-Angle}. 
When $\beta = P_u^\perp (-e_0)$,
	\[
	\begin{array}{ll}
	\inf_{\| \beta\|_2=1, u^T \beta =0} 
		\tan(\angle( \beta, -e_0)) 
		&=
		\tan(\angle( P_u^\perp (-e_0), -e_0)) \\
		&= \cot(\angle(u, e_0)).
	\end{array}
	\]
	
Therefore, $ \frac{p^\star}{1-p^\star}  = \cot(\angle(u, e_0)) $,
	

\end{proof}
    
\begin{figure} 
\centering
\includegraphics[width=.8\textwidth]{./figures_BD/Phase_T_sine_p.png}
\caption{The population phase transition diagram of $ \frac{p^\star}{1-p^\star}  = \cot(\angle(u, e_0)) $ for the special case when $X_t$ is independent, $X_0$ is sampled from Bernoulli Gaussian $p N(0,1) +(1-p) \delta_0,$ and the rest $\{X_t, t\neq 0\}$ are sampled from Gaussian $N(0,1)$. The red region is failure, and the blue region is success. }
\label{fig-PT-project}
\end{figure}



\section{Analysis of the inverse kernel method}
\subsection{Inverse kernel approximation}
Now we consider a setting where there is a kernel whose corresponding the inverse kernel is infinite, and we give a finite length approximation. 
We assume that $a =  (a_{-t},\ldots, a_{-1}, 1, a_1, a_2, \ldots, a_t) = \delta_0 + a_L + a_R$ 
is a length $ l= 2t+1$ filter, 
  and $\sum_{i\neq 0} |a_i|\leq M < 1$.    
 \begin{theorem} 
For a kernel $a = (a_{-s},\ldots, a_{-1}, 1, a_1, a_2, \ldots, a_t)$ of length $l=1+s+t$, if 
\[
\sum_{i\neq 0} |a_i|\leq M < 1,
\]
 then there exists an approximate inverse kernel $w^r$ of length $rl$, such that 
  \[
 \begin{array}{ll}
\|\mathcal{F}w-\mathcal{F}w^r \|_{\infty} 
&\leq M^{r+1}/(1-M),\\
\end{array}
  \]
\end{theorem}
\begin{proof}
Consider the $z$ transform of $a$ to be 
\[
 \begin{array}{ll}
A(z) =  \sum_{i=-s}^t a_{i} z^{-i},
\end{array}
  \]
let $\bar A(z) = 1-A(z)$, then 
from the condition 
$\sum_{i\neq 0} |a_i|\leq M < 1$,
we define $M' $ to be the maximum absolute value of $\bar A(z)$ on unit circle, as
\[
 \begin{array}{ll}
\mbox{maximize}   & |\bar A(z)| \\
\mbox{subject to}  & z\in {\mathbb C},|z|=1.
\end{array}
  \]
Then 
\[
 \begin{array}{ll}
M' \leq \sum_{i\neq 0} |a_i|\leq M<1.
\end{array}
  \]
The Z-transform of the inverse kernel of $a$ is defined by
\[
 \begin{array}{ll}
1/A(z) &=  (1-\bar A(z))^{-1},\\
&= 1+ \sum_{i=1}^{\infty} (\bar A(z))^i
\end{array}
  \]
 The infinite series absolutely converge on $|z| = 1$ because $|\bar A(z)|\leq M<1$.
Let the finite approximation of level $r$ of $w = a^{-1}$ be $w^r$, 
 its Z-transform is  $\sum_{i=0}^{r} (\bar A(z))^i$,
its length is at most $r(s+t+1)$, 
the $j$-th term of $w^r$ is $\sum_{b}bj_i=j \prod_{i} a_i^{j_i}$.
We have that for all  $|z| = 1$,
 \[
 \begin{array}{ll}
|1/A(z) - \sum_{i=0}^{r} (\bar A(z))^i | 
&= |\sum_{i=r+1}^{\infty} (\bar A(z))^i|\\
&\leq M^{r+1}/(1-M).
\end{array}
  \]
Therefore, 
   \[
 \begin{array}{ll}
\|\mathcal{F}w-\mathcal{F}w^r \|_{\infty} 
&\leq M^{r+1}/(1-M).
\end{array}
  \]
 
\end{proof}

Using this inverse approximation, we know for any kernel $a$ of length $l$ with possibly infinite inverse kernel $w$, there exists a length-$rl$ approximate inverse kernel $w^r$, such that 
  \[
 \begin{array}{ll}
\|\mathcal{F}w-\mathcal{F}w^r \|_{\infty} 
&\leq M^{r+1}/(1-M).
\end{array}
  \]

Therefore, we can prove the following theorem on sparse blind deconvolution algorithm. 


\paragraph{Simple example.}
We consider a simple example, when $a = (0,\ldots, 0, 1, r, 0, \ldots, 0) = \delta_0 - r\delta_1$, $|r|<1$. Then $a^{-1} = \delta_0 + \sum_{j=1}^\infty  r^j \delta_j.$ Now our data generation model is $y = a*x.$ 

We analysis the optimization problem 
 \begin{eqnarray*}
    \min_{f_0 = 0} &    
    E_x\| (\delta_0+f)*a*x\|_1.
\end{eqnarray*}
Here $f = (0, 0, f_R)$ is in a $K$ dimensional subspace of the double infinite sequence.


For the objective, using IID property of $x$, we only need to analyze
 \begin{eqnarray*} 
    E_x | ((\delta_0+f)*(\delta_0 - r\delta_1)*x)_0| .
\end{eqnarray*}
Let $f^*_K$ be the best length $K$ approximation, as the solution of 
\begin{eqnarray*}
    \min_{f_0 = 0} &    
    \| (\delta_0+f)*a\|_2,
\end{eqnarray*}

In this specific example, $f^*_K = \sum_{j=1}^K  r^j \delta_j.$ And $(\delta_0+f^*_K)*a = (\delta_0 - r^{K+1}\delta_{K+1})*(\delta_0 - r^{1}\delta_{1})^{-1}$.
Let $\tilde{x} = (\delta_0 - r^{K+1}\delta_{K+1})*x$, as one approximation of $x$.
Since $x$ is IID symmetric, $ \tilde{x}$ is identically-distributed, symmetric and weakly correlated for $|r^{K+1}|$ small enough, and $\tilde{x}_0, \ldots, \tilde{x}_K$ are independent.

Let 
\begin{eqnarray*} 
\delta_0+h &=& (\delta_0+f)*(\delta_0+f^*_K)^{-1}\\
&=& (\delta_0+f)*(\delta_0 - r^{K+1}\delta_{K+1})
\end{eqnarray*}
then 
 \begin{eqnarray*} 
    E_x | ((\delta_0+f)*(\delta_0 - r\delta_1)*x)_0| 
    &=& E_x | ((\delta_0+h)*\tilde{x})_0|\\
&=& E_x | \tilde{x}_0|+\mbox{sign}(\tilde{x}_0)(h*\tilde{x})_0\\
&\approx& 
 |1+  h_{0}| \cdot E |\tilde{x}_0| \cdot (1-\eta)+   \cdot  E{| (h*\tilde{x})_0| } \cdot \eta.
\end{eqnarray*}

Following the same analysis as the phase transition analysis, we have
\begin{eqnarray*}
     \min_\delta &E | ((\delta_0+f)*a*\tilde{x})_0 | \\
    \mbox{ subject to }  &f_{0} = 0.
\end{eqnarray*}

\subsection{Distance metrics}
We care about three types of metrics, first, distance  in $w$ space, between the computed composed filter $w$ and ground truth $a^{-1}$; second, distance in $w$ space, between the computed inverse filter $w$ and ground truth inverse filter $a^{-1}$, second, distance  in $\psi = w*a$ space, between the computed composed filter $w*a$ and ground truth $e_0$; third,  distance  in $x$ space, between recovered signal sequence and true sparse signal sequence. 

First, the MSE in space of $\psi = w*a$,
  \[
 \begin{array}{ll}
d_{\psi,2}^2 := \|\phi\|_2^2 =  \|\psi-e_0\|_2^2= \|a*(w-a^{-1})\|_2^2
\end{array}
  \] 
 The MSE in space of $w$. 
   \[
 \begin{array}{ll}
d_{w,2}^2  = \|w-a^{-1}\|_2^2 = \|a^{-1}*(\psi-e_0))\|_2^2 = \|a^{-1}*\phi\|_2^2
\end{array}
  \]
Last, $\ell_1$ and $\ell_2$ distance in $x$ space:
\[
 \begin{array}{ll}
d_{x,1} &:= \frac{1}{2T+1}\|w*Y-X\|_{1,[-T,T]} =  \frac{1}{2T+1}\|\tilde \phi* X\|_{1,[-T,T]} \\
d_{x,2}^2 &:= \frac{1}{2T+1}\|w*Y-X\|_{2,[-T,T]}^2 = \frac{1}{2T+1}\|\tilde \phi* X\|_{2,[-T,T]}^2
\end{array}
  \]
We compute their expectation: 
\[
 \begin{array}{ll}
{\mathbb E} d_{x,1}  &=  {\mathbb E}|\phi^T X| = \sqrt{\frac{2}{\pi}} \cdot {\mathbb E}_I \|\phi \cdot I\|_2\\
{\mathbb E} d_{x,2}^2 &=  {\mathbb E}|\phi^T X|^2 = p \|\phi\|^2  = p \|\psi-e_0\|^2 
\end{array}
  \]  
We remark that in the following we focus on distance in $x$ space to recover the signal. 



\section{Stability for noisy observation}


\subsection{Constrained Convex algorithms }
 \paragraph{Constrained noisy convex problems}
Then we define the constrained convex program that allows noise,
\[
 \begin{array}{ll}
 \mbox{minimize}_{w,z}   & |\alpha| \\
\mbox{subject to}  
& v^T w =1,\\
& {\mathbb E}(\alpha -\tilde{w}^T Y)^2 \leq \epsilon^2
\end{array}
\tag{$P^c_{1,\epsilon}$}
\label{eq:ConstrainProb}
  \]
  If we write $\alpha =\tilde{w}^T Y - \epsilon \zeta $ , then
\[
 \begin{array}{ll}
 \mbox{minimize}_{w,\zeta}   & {\mathbb E}|\tilde{w}^T Y - \epsilon \zeta| \\
\mbox{subject to}  
& v^T w =1,\\
& \zeta^2 \leq 1
\end{array}
\tag{$P^c_{1,\epsilon}$}
  \]
If we set $\epsilon = 0,$ then we recover 
the original convex problems for noiseless setting
\[
 \begin{array}{ll}
 \mbox{minimize}_{w}   & {\mathbb E}|\tilde{w}^T Y|\\
\mbox{subject to}  
& v^T w =1,
\end{array}
\tag{$P_1$}
  \]
\subsection{Stability for adversarial perturbation of sparse signal }
\paragraph{Mathematical formulation: sparse signal plus adversarial perturbation}

\begin{itemize}
\item
Let $(X_t)_{t \in {\mathbb Z}}$ be IID sampled from  $pN(0,1) + (1-p) \delta_0$, write $X_t = I_t \cdot G_t$ where $I_t$ is IID sampled from Bernoulli, $G_t$ is IID sampled from standard Gaussian.
 \item
The filter $a\in l_1({\mathbb Z}), (\mathcal{F} a)(\omega) \neq 0, \forall \omega\in T.$
\item
$Z$ is a unit vector in $2$ sphere. 
\item
\[
\begin{array}{ll}
Y&= a* (X+\eta Z ) 
\end{array}
\]
\end{itemize}
Let $\tilde{\psi} := w*a$,   $\phi: = \psi - e_0$, then
\[
 \begin{array}{ll}
{\mathbb E}|(w*Y)_0|
&= {\mathbb E}|\psi^T (X+\eta Z )| \\
\end{array}
  \]

  
  
  
 \paragraph{Study the noisy problem}
Under the data generative model, 
\[
\begin{array}{ll}
Y&= a* (X+\eta Z ) 
\end{array}
\]
    the constrained problem becomes
  \[
 \begin{array}{ll}
 \mbox{minimize}_{\psi,\zeta}   & {\mathbb E}|\psi^T (X+\eta Z ) - \epsilon \zeta| \\
\mbox{subject to}  
&u^T \psi =1,\\
& \zeta^2 \leq 1
\end{array}
\tag{$P^c_{1,\epsilon}$}
  \]
  
\paragraph{Worst case objective over $Z$}  
For the constrained problem, we optimize over the worst case objective for all $Z$ that $\|Z\|_2\leq 1$,
    \[
 \begin{array}{ll}
 \mbox{minimize}_{\psi,\zeta} \quad \sup_{Z: \|Z\|_2\leq 1}  &  {\mathbb E}|\psi^T X+\eta  \psi^T Z  - \epsilon \zeta| \\
\mbox{subject to}  
&u^T \psi =1,\\
& \zeta^2 \leq 1
\end{array}
  \]
Which is equivalent to 
  \[
 \begin{array}{ll}
 \mbox{minimize}_{\psi} \quad \inf_{\zeta: \zeta^2 \leq 1} \quad \sup_{Z: \|Z\|_2\leq 1}  &  {\mathbb E}|\psi^T X+\eta  \psi^T Z  - \epsilon \zeta| \\
\mbox{subject to}  
&u^T \psi =1
\end{array}
  \]
In the following we study the min-max objective
  \[
 \begin{array}{ll}
L_{\eta, \epsilon}(\psi) : = \inf_{\zeta: \zeta^2 \leq 1} \quad \sup_{Z: \|Z\|_2\leq 1}  &  {\mathbb E}|\psi^T X+\eta  \psi^T Z  - \epsilon \zeta| 
\end{array}
  \]
  
\paragraph{Robust optimization}
Let the deterministic residual function of $\psi, Z, \zeta$ be
\[
\mu_{\eta, \epsilon}(\psi, Z, \zeta): = \eta  \psi^T Z  - \epsilon \zeta
\]
We reparametrize $\epsilon$ as by defining 
\[
\tau :=  \frac{\epsilon}{\eta}
\]
then 
\[
\mu_{\eta, \epsilon} = \eta ( \psi^T Z  - \tau \zeta)
\]






\begin{lemma}
For any given sign $s =\pm 1$,
\[
\sup_{Z: \|Z\|_p\leq 1}(s \psi^T Z) = \|\psi\|_q,
\]
specifically, $Z$ is along the direction of $s\psi$ so that $s\psi^T Z = \|\psi\|_2$.

\end{lemma}  


\begin{lemma}[Hölder's inequality]
For any given sign $s =\pm 1$, for $p\geq 1, \frac{1}{p}+\frac{1}{q} =1,$
\[
\sup_{Z: \|Z\|_p\leq 1}(s \psi^T Z) = \|\psi\|_q,
\]
specifically, when $|Z|^p$ is along the direction of $s|\psi|^q$ so that $s\psi^T Z = \|\psi\|_q$.

\end{lemma}  


\begin{lemma}
\label{min-maxError}
The min-max quantity  
\[
 \begin{array}{ll}
\inf_{\zeta: \zeta^2 \leq 1} \sup_{Z: \|Z\|_2\leq 1} (\eta  \psi^T Z  - \epsilon \zeta)^2 = (\eta \|\psi\|_2  - \epsilon )_+^2 = \eta^2 ( \|\psi\|_2  - \tau )_+^2
\end{array}
\]
where $(\cdot)_+: = \max{(0, \cdot)}$ is the ReLU function.
\end{lemma}
\begin{proof}
\[
 \begin{array}{ll}
\inf_{\zeta: \zeta^2 \leq 1} \sup_{Z: \|Z\|_2\leq 1} \mu_{\eta, \epsilon}^2 
&=\inf_{\zeta: \zeta^2 \leq 1}\sup_{Z: \|Z\|_2\leq 1} (\eta  \psi^T Z  - \epsilon \zeta)^2\\
&=\inf_{\zeta: \zeta^2 \leq 1}\sup_{Z: \|Z\|_2\leq 1} [(\eta  \psi^T Z)^2+  (\epsilon \zeta)^2 - 2(\eta  \psi^T Z)(\epsilon \zeta)]\\
&=\inf_{\zeta: \zeta^2 \leq 1}\{   \epsilon^2( \zeta)^2+ \eta^2 \sup_{Z: \|Z\|_2\leq 1} [ (\psi^T Z)^2] + 2\eta \epsilon |\zeta|   \sup_{Z: \|Z\|_2\leq 1}[ (-{\rm sign}(\zeta) \psi^T Z)]\}\\
&=\inf_{\zeta: \zeta^2 \leq 1}\{   \epsilon^2( \zeta)^2+ \eta^2 \|\psi\|_2^2 + 2\eta \epsilon |\zeta|   \|\psi\|_2\}\\
&=\inf_{\zeta: \zeta^2 \leq 1}\{  (\eta \|\psi\|_2  - \epsilon |\zeta| )^2\}\\
&= (\eta \|\psi\|_2  - \epsilon )_+^2
\end{array}
\]
The fourth equation is because for any given sign $s =\pm 1$,
\[
\sup_{Z: \|Z\|_2\leq 1}(s \psi^T Z) = \|\psi\|_2.
\]
\[
\sup_{Z: \|Z\|_2\leq 1}( \psi^T Z)^2 = \|\psi\|_2^2.
\]



\end{proof}

We have the following result, generalizing $\|Z\|_2\leq 1$ to $\|Z\|_p\leq 1, \quad, p\geq 1$, and then to $Z\in K$ for a closed compact convex set $K\in {\mathbb R}^N$.



\begin{lemma}
\label{min-maxErrorL_p}
The min-max quantity 
for $p\geq 1, \frac{1}{p}+\frac{1}{q} =1,$
for a one-dimensional convex even function $F$ that is monotonically non-decreasing for positive coordinates,
\[
 \begin{array}{ll}
\inf_{\zeta: \zeta^2 \leq 1} \sup_{Z: \|Z\|_p\leq 1} F( \eta (\psi^T Z  - \tau \zeta)) = 
F(\eta( \|\psi\|_q  - \tau )_+)
\end{array}
\]
where $(\cdot)_+: = \max{(0, \cdot)}$ is the ReLU function.
\end{lemma}

\begin{coro}
 \[
 \begin{array}{ll}
 \inf_{\zeta: \zeta^2 \leq 1} \quad \sup_{Z: \|Z\|_2\leq 1}  &  {\mathbb E}|\psi^T X+\eta (\psi^T Z  - \tau \zeta)| ={\mathbb E}|\psi^T X+\eta( \|\psi\|_2  - \tau )_+|
\end{array}
  \]
  in general, for $p\geq 1, \frac{1}{p}+\frac{1}{q} =1,$
   \[
 \begin{array}{ll}
 \inf_{\zeta: \zeta^2 \leq 1} \quad \sup_{Z: \|Z\|_p\leq 1}  &  {\mathbb E}|\psi^T X+\eta (\psi^T Z  - \tau \zeta)| ={\mathbb E}|\psi^T X+\eta( \|\psi\|_q  - \tau )_+|
\end{array}
  \]
\end{coro}


\subsection{Folded Gaussian}
\paragraph{Folded Gaussian}
For Bernoulli Gaussian $X$ with support $I$, if we condition on $I$ fixed, then for any constant $\mu$, using the rotation invariance of Gaussian, $\psi^T X +\mu$ is  Gaussian with mean $\mu$ and standard deviation $\|\psi_I\|_2$, therefore,
\[
|\psi^T X +\mu |
\]
following the folded Gaussian distribution.
\footnote{see: \url{ https://en.wikipedia.org/wiki/Folded_normal_distribution}.}

From general theory of folded Gaussian $ |\mu+ \sigma G |, \quad  G \sim \mathcal{N}(0, 1).$
Then its mean is
\begin{equation}
\label{FoldedGaussianMean}
E_G |\mu+ \sigma G| =  \sqrt{\frac{2}{\pi}} \sigma \exp\{-\mu^2/(2\sigma^2)\} + \mu \left(1 - 2\Phi\left(\frac{-\mu}{\sigma}\right) \right)
\end{equation}
where $\Phi$  is the normal cumulative distribution function:
\[
\Phi(x) = \frac 1 {\sqrt{2\pi}} \int_{-\infty}^x e^{-t^2/2} \, dt.
\]
Its variance is
\begin{equation}
\label{FoldedGaussianVar}
{\rm Var}_G |\mu+ \sigma G| = E_G( |\mu+ \sigma G|-E_G |\mu+ \sigma G|)^2 =\sigma^2+ \mu^2 -(E_G |\mu+ \sigma G|)^2
\end{equation}  
The variance will be useful for concentration with finite sample.  

\paragraph{Asymptotic expansion for mean of folded Gaussian}
Notice that if $\sigma =0$, then
\[
E_G |\mu+ 0\cdot G| =   |\mu|.
\]
If $\mu =0$, then we go back to mean of half-normal,
\[
E_G |\sigma G| =   \sqrt{\frac{2}{\pi}} \sigma
\]
Let 
\[
\gamma:= \frac{\mu}{\sigma}
\] 

\begin{lemma}
\label{ExpansionMeanFoldedGaussian}
We define the ratio of folded gaussian mean as 
\[
\mathcal{R}^f(\gamma) := \frac{E_G |\mu+ \sigma G|}{E_G |\sigma G|} = \frac{E_G |\mu+ \sigma G|}{\sqrt{\frac{2}{\pi}}\sigma} = \sqrt{\frac{\pi}{2}} E_G |\gamma+  G|
\]
We have three different expressions (asymptotic expansion around  $\gamma$) for $\mathcal{R}^f(\gamma)$:
\begin{eqnarray}
\label{FoldedGaussianMean_ratio}
\mathcal{R}^f(\gamma)
&=&  \exp\{-\gamma^2/2\} + \sqrt{\frac{\pi}{2}}\gamma \left(1 - 2\Phi\left(-\gamma\right) \right)\\
&=&  \exp\{-\gamma^2/2\}\{1 + \frac{1}{2}\gamma \left[ \gamma + \frac{ \gamma^3}{3} + \frac{ \gamma^5}{3\cdot 5} + \cdots + \frac{ \gamma^{2n+1}}{(2n+1)!!} + \cdots\right]\}\\
&=& 1 + \frac{3}{2}\gamma^2 + \frac{1}{24} \gamma^4   - \frac{ \gamma^6}{120}  + O(\gamma^8) 
\end{eqnarray}
 \end{lemma}
 
 \begin{lemma}
 \label{ExpansionMeanFoldedGaussian_LB}
when $\gamma<2,$ 
\[
\mathcal{R}^f(\gamma) \geq 1 + \frac{3}{2}\gamma^2
\]
hence
\[
 \begin{array}{ll}
E_G |\mu+ \sigma G|
&\geq \sqrt{\frac{2}{\pi}}\left[\sigma + \frac{3}{2}\mu^2/\sigma \right]
\end{array}
\]

\end{lemma}
\begin{proof}[Proof of lemma~\ref{ExpansionMeanFoldedGaussian}]
Plug in the formula for mean of folded Gaussian \ref{FoldedGaussianMean}, 
we have
\[
E_G |\mu+ \sigma G| = \sqrt{\frac{2}{\pi}} \sigma \exp\{-\gamma^2/2\} + \mu \left(1 - 2\Phi\left(-\gamma\right) \right)
\]
Divide it by 
\[
E_G |\sigma G| =   \sqrt{\frac{2}{\pi}} \sigma
\]
we get the first equality.

Since the CDF of the standard normal distribution can be expanded by integration by parts into a series:
\[
\Phi( \gamma)=\frac{1}{2} + \frac{1}{\sqrt{2\pi}}\cdot e^{- \gamma^2/2} \left[ \gamma + \frac{ \gamma^3}{3} + \frac{ \gamma^5}{3\cdot 5} + \cdots + \frac{ \gamma^{2n+1}}{(2n+1)!!} + \cdots\right]
\]
where $!!$ denotes the double factorial. 

\[
1 - 2\Phi(-\gamma)  = \frac{1}{\sqrt{2\pi}}\cdot e^{- \gamma^2/2}\left[ \gamma + \frac{ \gamma^3}{3} + \frac{ \gamma^5}{3\cdot 5} + \cdots + \frac{ \gamma^{2n+1}}{(2n+1)!!} + \cdots\right]
\]
This gives the second inequality.

Additionally,
 \[
 \begin{array}{ll}
  e^{- \gamma^2/2} &=\sum_{n=0}^{\infty} \frac{ \gamma^{2n}}{(-2)^n (n)!}\\
  &=  1- \frac{ \gamma^2}{2} + \sum_{n=2}^{\infty} \frac{ \gamma^{2n}}{(-2)^n (n)!}
\end{array}
  \] 
then
 \[
 \begin{array}{ll}
& \exp\{-\gamma^2/2\}\{1 + \frac{1}{2}\gamma \left[ \gamma + \frac{ \gamma^3}{3} + \frac{ \gamma^5}{3\cdot 5} + \cdots + \frac{ \gamma^{2n+1}}{(2n+1)!!} + \cdots\right]\}\\
=& (\sum_{n=0}^{\infty} \frac{ \gamma^{2n}}{(-2)^n (n)!})\{1 + \frac{1}{2}\gamma \left[ \sum_{n=0}^{\infty} \frac{ \gamma^{2n+1}}{(2n+1)!!} \right]\}\\

=& 1 + \frac{3}{2}\gamma^2 + \frac{1}{24} \gamma^4   - \frac{ \gamma^6}{120}  + O(\gamma^8) 
\end{array}
  \] 
\end{proof}


\begin{proof}[Proof of lemma~\ref{ExpansionMeanFoldedGaussian_LB}]
Let $C_\Phi( \gamma^2)$ be a function of $ \gamma$, then it is a composition function with the inner funciton being $ \gamma^2,$ defined as follows: 
\[
 \begin{array}{ll}
C_\Phi( \gamma^2) 
&:= (2\Phi( \gamma)-1)/\left(\frac{ \gamma}{\sqrt{2\pi}}\right) \\
&= e^{- \gamma^2/2} \left[1 + \frac{ \gamma^2}{3} + \frac{ \gamma^4}{3\cdot 5} + \cdots + \frac{ \gamma^{2n}}{(2n+1)!!} + \cdots\right]\\
&= [\sum_{n=0}^{\infty} \frac{ \gamma^{2n}}{(2n+1)!!} ]/[\sum_{n=0}^{\infty} \frac{ \gamma^{2n}}{2^n (n)!}] \\
&= [\sum_{n=0}^{\infty} \frac{ \gamma^{2n}}{(2n+1)!!} ]/[\sum_{n=0}^{\infty} \frac{ \gamma^{2n}}{(2n)!!}] \\
&\leq 1
\end{array}
\]
We remark further that
\[
 \begin{array}{ll}
C_\Phi( \gamma^2) 
&= [\sum_{n=0}^{\infty} \frac{ \gamma^{2n}}{(2n+1)!!} ]/[\sum_{n=0}^{\infty} \frac{ \gamma^{2n}}{(2n)!!}] \\
&= [1 + \frac{ \gamma^2}{3}+ +  \frac{ \gamma^4}{15}+ O( \gamma^6) ]/[1 + \frac{ \gamma^2}{2}+ +  \frac{ \gamma^4}{8}+O( \gamma^6)] \\
&= 1 -\frac{ \gamma^2}{6}+  \frac{ \gamma^4}{40}+ O( \gamma^6)
\end{array}
\]
Then
 \[
 \begin{array}{ll}
\mu \left(1 - 2\Phi\left(-\gamma\right) \right)
&= \frac{1}{\sqrt{2\pi}}\cdot \mu^2 \frac{1}{\sigma} C_\Phi(\gamma^2)\\
&= \frac{1}{\sqrt{2\pi}}\cdot \mu^2 \frac{1}{\sigma} (1-\gamma^2/6 +O(\gamma^4))\\
\end{array}
\] 
Plug in all the fact together
\[
 \begin{array}{ll}
E_G |\mu+ \sigma G|
&=   \left[ \sqrt{\frac{2}{\pi}} \sigma \exp\{-\gamma^2/2\} + \frac{1}{\sqrt{2\pi}}\cdot \mu^2 \frac{1}{\sigma} C_\Phi(\gamma^2)\right]\\
&=   \left[ \sqrt{\frac{2}{\pi}} \sigma \{ 1- \frac{\gamma^2}{2} + \sum_{n=2}^{\infty} \frac{\gamma^{2n}}{(-2)^n (n)!}\} + \frac{1}{\sqrt{2\pi}}\cdot \mu^2 \frac{1}{\sigma} C_\Phi(\gamma^2)\right]\\
&= \sqrt{\frac{2}{\pi}}   \left[ \sigma 
\{ 1 + (1+\frac{1}{2}\cdot C_\Phi(\gamma^2))\gamma^2 +   \sum_{n=2}^{\infty} \frac{\gamma^{2n}}{(-2)^n (n)!}\} \}\right]\\
\end{array}
\] 
Define the residual of this expansion as
\[
 \begin{array}{ll}
\mbox{Res}(\gamma^2) 
&:= \frac{ (2\Phi( \gamma)-1)/\left(\frac{ \gamma}{\sqrt{2\pi}}\right)-1}{2}\cdot \gamma^2 +  (\exp\{-\gamma^2/2\} - 1 + \gamma^2/2)
\\
&= \frac{ C_\Phi(\gamma^2)-1}{2}\cdot \gamma^2 +  (\exp\{-\gamma^2/2\} - 1 + \gamma^2/2) \\
&=    \frac{ C_\Phi(\gamma^2)-1}{2}\cdot \gamma^2  +   \{ \sum_{n=2}^{\infty} \frac{\gamma^{2n}}{(-2)^n (n)!}\} \} \\
&=    \frac{-\gamma^4 }{12}  +    \frac{ \gamma^6}{80} +\frac{\gamma^{4}}{(-2)^2 (2)!} +\frac{\gamma^{6}}{(-2)^3 (3)!}+
O(\gamma^8)\\
&=    \frac{\gamma^4 }{24}  - \frac{ \gamma^6}{120}  + O(\gamma^8)
\end{array}
\] 
We know that if $\gamma^2<1,$ 
\[
 \begin{array}{ll}
\mbox{Res}(\gamma^2) 
&=    \frac{\gamma^4 }{24}  - \frac{ \gamma^6}{120}  + O(\gamma^8)\\
&\geq 0
\end{array}
\] 
(We comment that we could push the range for $\mbox{Res}(\gamma^2) 
\geq 0$ further to at least $\gamma^2<5$ if needed.)
Then we give an lower bound:
\[
 \begin{array}{ll}
E_G |\mu+ \sigma G| 
&= \sqrt{\frac{2}{\pi}} \sigma\left[ 1 + \frac{3}{2}\gamma^2+   \mbox{Res}\right]\\
&\geq \sqrt{\frac{2}{\pi}}\left[  \sigma + \frac{3}{2}\mu^2  \sigma^{-1}\right]
\end{array}
\] 


\end{proof}






\subsection{Other setting on $Z$}
\paragraph{ Adversarial outer noise} 
If our generative model is
\[
\begin{array}{ll}
Y&= a* X + \eta_2 Z_2\\
&= a* (X +\eta_2 a^{-1}* Z_2)
\end{array}
\] 
And we consider the worst case objective over $\|Z_2\|_p\leq 1, \quad p\geq 1.$

Let $C_a^{-1}$ be the linear operator represents $a^{-1}*$, then following previous calculation,
\begin{coro}
For $p\geq 1, \frac{1}{p}+\frac{1}{q} =1,$
   \[
 \begin{array}{ll}
 \inf_{\zeta: \zeta^2 \leq 1} \quad \sup_{Z: \|Z_2\|_p \leq 1}  &  {\mathbb E}|\psi^T X+\eta_2  (\psi^T C_a^{-1} Z  - \tau \zeta)| ={\mathbb E}|\psi^T X+\eta( \|C_{a^{-1}}^T \psi\|_q  - \tau )_+|
\end{array}
  \]
\end{coro}

Let $C_{a^{-1}}^T \psi =\bar{a}^{-1}*\psi$, where $\tilde{a}^{-1}$ is defined by flipping the coordinate of $a^{-1}$ (reflecting the coordinates around zero by map $i$ to $-i$ for $i\in {\mathbb Z}$.)
 
 We can replace the definition of $\gamma$ and $\mu_0$  by:
 \[
 \gamma^{a}_{\eta, \epsilon}(\psi, I): = \inf_{\zeta: \zeta^2 \leq 1} \sup_{Z: \|Z\|_2\leq 1} \eta_2  (\psi^T C_a^{-1} Z  - \tau \zeta)/\|\psi_I\|_2 = \eta \frac{ (\|\bar{a}^{-1}*\psi\|_q  - \tau)_+}{\|\psi_I\|_2}
\]
\[
\mu_0^{a} := \eta( \|\tilde{a}^{-1}\|_q-\tau)_+
\]
We can get the same result as theorem~\ref{Thm:StabilityMSE}.

  

  

\subsection{ Gaussian additional noise}
\paragraph{ Gaussian additional observation inner and outer noise} 
If $G_1$ and $G_2$ are two independent standard normal $N(0,I_N)$. Our generative model is
\[
\begin{array}{ll}
Y&= a* (X+  \sigma_1 G_1) +  \sigma_2 G_2\\
&= a* (X+\sigma_1 G_1+\sigma_2 a^{-1}* G_2)
\end{array}
\] 
let $C_{a^{-1}}$ be the linear operator represents $a^{-1}*$, now using the property for sum of independent Gaussian, then $C_{a^{-1}} = C_a^{-1}$ and 
\[
\begin{array}{ll}
Z: = \sigma_1 G_1+\sigma_2 a^{-1}* G_2
\sim N(0, \sigma_1^2 I_N + \sigma_2^2 C_{a^{-1}} C_{a^{-1}}^T)
\end{array}
\] 
then
\[
\begin{array}{ll}
\psi^T Z
\sim N(0, \sigma_1^2\|\psi\|_2^2 + \sigma_2^2\|C_{a^{-1}}^T \psi\|_2^2 )
\end{array}
\] 
 Let $G$ be another standard normal $N(0,I_N)$, for any fixed vector $\mu \in {\mathbb R}^N, $
 \[
 \begin{array}{ll}
  {\mathbb E}_{G_1} {\mathbb E}_{G_2} {\mathbb E}_X|\psi^T X+\sigma_1 \psi^T G_1+\sigma_2 \psi^T (a^{-1}* G_2)  | = 
   {\mathbb E}_I{\mathbb E}_{G_3}  | \sqrt{ \|\psi_I\|_2^2 + \sigma_1^2\|\psi\|_2^2 + \sigma_2^2\|C_{a^{-1}}^T \psi\|_2^2}G_3  |
\end{array}
  \]
In this case, it seems the previous constrained problem no longer can help us find exact solution $\psi^\star =e_0.$ 



We first study $\sigma_2 = 0$ case.
\[
{\mathbb E}_I{\mathbb E}_{G_3}  | \sqrt{ \|\psi_I\|_2^2 + \sigma_1^2\|\psi\|_2^2}G_3 | = 
\sqrt{\frac{2}{\pi}} \cdot {\mathbb E}_I(\sqrt{ \|\psi_I\|_2^2 + \sigma_1^2\|\psi\|_2^2}) =  \sqrt{\frac{2}{\pi}} \cdot \|\psi\|_2 {\mathbb E}_I(\sqrt{ \frac{\|\psi_I\|_2^2}{\|\psi\|_2^2} + \sigma_1^2})
\]
Define worst case objective
\[
F_{\sigma}(\psi):= \sup_{\sigma_1 \leq \sigma} {\mathbb E}_I(\sqrt{ \|\psi_I\|_2^2 + \sigma_1^2\|\psi\|_2^2}) = {\mathbb E}_I(\sqrt{ \|\psi_I\|_2^2 + \sigma^2\|\psi\|_2^2})
\]
The second equality comes from monotonicity.

Again, we assume the minimizer is $\psi^\star $. Since $e_0$ is  minimizer of $L_{0}$ and $\psi^\star $ is  minimizer of $L_{\sigma}$, we have the chain of inequality:
    \[
 \begin{array}{ll}
F_{\sigma}(e_0)
 \geq F_{\sigma}(\psi^\star ) 
 \geq F_{0}(\psi^\star )
 \geq F_{0}(e_0) 
\end{array}
  \] 
Then
    \[
 \begin{array}{ll}
F_{\eta}(e_0)/F_{0}(e_0)
 \geq (F_{\eta}(\psi^\star )/F_{0}(\psi^\star ))(F_{0}(\psi^\star )/F_{0}(e_0) )
\end{array}
  \] 

   \[
 \begin{array}{ll}
 F_{\eta}(\psi)/F_{0}(\psi) = {\mathbb E}_I(\sqrt{ \frac{\|\psi_I\|_2^2}{\|\psi\|_2^2} + \sigma^2})/{\mathbb E}_I(\frac{\|\psi_I\|_2}{\|\psi\|_2} )
\end{array}
  \] 
  
     \[
 \begin{array}{ll}
 F_{\eta}(e_0)/F_{0}(e_0) = {\mathbb E}_I(\sqrt{ \frac{\|(e_0)_I\|_2^2}{\|e_0\|_2^2} + \sigma^2})/{\mathbb E}_I(\frac{\|(e_0)_I\|_2}{\|e_0\|_2} )= (p\sqrt{ 1 + \sigma^2}+ (1-p) \sqrt{ \sigma^2})/p
\end{array}
  \] 
  
    \[
 \begin{array}{ll}
 F_{0}(\psi)/F_{0}(e_0) 
 &= {\mathbb E}_I\|\psi_I\|_2/{\mathbb E}_I\|(e_0)_I\|_2 \\
 &= 1+({\mathbb E}_I\|\psi_I\|_2-{\mathbb E}_I\|(e_0)_I\|_2)/{\mathbb E}_I\|(e_0)_I\|_2 \\
 &= 1+ (B(p,N)/{\mathbb E}_I\|(e_0)_I\|_2) \|\psi -e_0\|_2\\
 &= 1+ (B(p,N)/p) \|\phi\|_2
\end{array}
  \] 

   \[
 \begin{array}{ll}
1+ (B(p,N)/p) \|\phi\|_2 &\leq (F_{\eta}(e_0)/F_{0}(e_0))/ (F_{\eta}(\psi)/F_{0}(\psi)) \\
&\leq \{(p\sqrt{ 1 + \sigma^2}+ (1-p) \sqrt{ \sigma^2})/p\}/\{{\mathbb E}_I(\sqrt{ \frac{\|\psi_I\|_2^2}{\|\psi\|_2^2} + \sigma^2})/{\mathbb E}_I(\frac{\|\psi_I\|_2}{\|\psi\|_2} )\}
\end{array}
  \] 
  First, a loose bound on $\|\phi^\star\|_2$:
  \[
 \begin{array}{ll}
1+ (B(p,N)/p) \|\phi^\star\|_2 
&\leq \{(p\sqrt{ 1 + \sigma^2}+ (1-p) \sqrt{ \sigma^2})/p\}
\end{array}
  \] 
We found that
 \[
 \begin{array}{ll}
 \|\phi^\star\|_2 
&\leq (\{(p\sqrt{ 1 + \sigma^2}+ (1-p) \sqrt{ \sigma^2})/p\}-1)/(B(p,N)/p) \\
&\leq ((p\sqrt{ 1 + \sigma^2}+ (1-p) \sqrt{ \sigma^2})-p)/B(p,N)
\end{array}
  \] 

In general, when $\sigma_1, \sigma_2$ both non-zero, using the same argument,
\[
 \begin{array}{ll}
 \|\phi^\star\|_2 
&\leq \frac{(p[\sqrt{ 1 + \sigma_1^2+ \sigma_2^2\|\tilde a^{-1}\|_2^2}-1]+ (1-p) \sqrt{ \sigma_1^2+ \sigma_2^2\|\tilde a^{-1}\|_2^2})}{B(p,N)}
\end{array}
  \] 

\subsection{Inner and outer Gaussian and adversarial noise}
\paragraph{Inner and outer Gaussian and adversarial noise together}
When 
If $G_1$ and $G_2$ are two independent standard normal $N(0,I_N)$. 
$Z_1$ and $Z_2$ are inner and outer adversarial noise with $p_1$ and $p_2$ norm bounded by $1$.

Our generative model is
\[
\begin{array}{ll}
Y&= a* (X+ \eta_1 Z_1+  \sigma_1 G_1) + \eta_2 Z_2+ \sigma_2 G_2\\
&= a* (X +\eta_1 Z_1 +\eta_2 a^{-1}* Z_2+  \sigma_1 G_1+\sigma_2 a^{-1}* G_2)
\end{array}
\]
We define our robust objective as   
\[
 \begin{array}{ll}
& F_{\eta_1,\eta_2, \sigma_1,\sigma_2}(\psi)\\
:=& 
\inf_{\zeta^2\leq 1}\sup_{Z_1, Z_2: \|Z_1\|_{p_1} \leq 1; \|Z_2\|_{p_2} \leq 1}
{\mathbb E}_{G_1} {\mathbb E}_{G_2} {\mathbb E}_X|\psi^T X+\sigma_1 \psi^T G_1+\sigma_2 \psi^T (a^{-1}* G_2) + \eta_1 Z_1 +\eta_2 a^{-1}* Z_2-\epsilon \zeta |
\end{array}
  \]  
\begin{lemma}
\label{lemma:Obj_Folded_FullNoise}
Let 
 \[
 \begin{array}{ll}
 \mu(\psi) &=(\eta_1\|\psi\|_{q_1}+ \eta_2\|C_{a^{-1}}^T \psi\|_{q_2}-\epsilon)_+\\
 \sigma_n^2(\psi) &= \sqrt{ \sigma_1^2\|\psi\|_2^2 + \sigma_2^2\|C_{a^{-1}}^T \psi\|_2^2}
 \end{array}
  \]
then the objective is a mean of a folded Gaussian random variable:
\[
 \begin{array}{ll}
 F_{\eta_1,\eta_2, \sigma_1,\sigma_2}(\psi)
&= {\mathbb E}_I{\mathbb E}_{G}  | \sqrt{\|\psi_I\|_2^2 + \sigma_n^2(\psi)} G  + \mu(\psi) |\\
&= \sqrt{\frac{2}{\pi}} {\mathbb E}_I[ \sqrt{\|\psi_I\|_2^2 + \sigma_n^2(\psi)} 
\mathcal{R}^f(\frac{\mu(\psi)}{\sqrt{\|\psi_I\|_2^2 + \sigma_n^2(\psi)}})]
\end{array}
  \] 
  where $G$ is scalar standard Gaussian.
\end{lemma}
From a simple inequality:
   \[
 \begin{array}{ll}
 F_{0,0,0,0}(\psi^\star ) &\leq F_{\eta_1,\eta_2, \sigma_1,\sigma_2}(e_0)
\end{array}
  \]  
we have  
\begin{theorem}
\label{thm:Bound_FullNoise}
\[
 \begin{array}{ll}
 B(p,N) \|\psi^\star -e_0\|_2\leq (1-p) \sigma_n(e_0)^2 \mathcal{R}^f(\frac{\mu(e_0)}{\sigma_n(e_0)})+  
 p\sqrt{1+ \sigma^2_n(e_0)} \mathcal{R}^f(\frac{\mu(e_0)}{\sqrt{1+ \sigma^2_n(e_0)}}) - p
\end{array}
  \]  
where
\[
 \begin{array}{ll}
 \mu(e_0) &= (\eta_1 + \eta_2 \| a^{-1} \|_{q_2}-\epsilon)_+\\
 \sigma_n(e_0) &=\sqrt{ \sigma_1^2 + \sigma_2^2\|a^{-1}\|_2^2} 
 \end{array}
  \]
\end{theorem}  

    
\begin{proof}[Proof of lemma~\ref{lemma:Obj_Folded_FullNoise}]
 First, for any fixed vector $\mu$,
\[
 \begin{array}{ll}
  {\mathbb E}_{G_1} {\mathbb E}_{G_2} {\mathbb E}_X|\psi^T X+\sigma_1 \psi^T G_1+\sigma_2 \psi^T (a^{-1}* G_2) + \mu | = 
   {\mathbb E}_I{\mathbb E}_{G_3}  | \sqrt{ \|\psi_I\|_2^2 + \sigma_1^2\|\psi\|_2^2 + \sigma_2^2\|C_{a^{-1}}^T \psi\|_2^2}G_3  + \mu|
\end{array}
  \]
  Let 
 \[
 \begin{array}{ll}
 \mu(\psi) &= (\eta_1\|\psi\|_{q_1}+ \eta_2\|C_{a^{-1}}^T \psi\|_{q_2}-\epsilon)_+\\
 \sigma_n^2(\psi) &= \sqrt{ \sigma_1^2\|\psi\|_2^2 + \sigma_2^2\|C_{a^{-1}}^T \psi\|_2^2}\\
  \sigma_s^2(\psi_I) &=  \|\psi_I\|_2^2 \\
 \sigma^2 (\psi, \psi_I) &= \sigma_s^2(\psi_I) + \sigma_n^2(\psi) 
 \end{array}
  \]
 Using lemma~\ref{min-maxErrorL_p}, we have
\[
 \begin{array}{ll}
F_{\eta_1,\eta_2, \sigma_1,\sigma_2}(\psi)
&={\mathbb E}_I{\mathbb E}_{G}  | \sigma (\psi, \psi_I) G  + \mu(\psi) | \\
&={\mathbb E}_I{\mathbb E}_{G}  | \sqrt{\sigma_s^2(\psi_I) + \sigma_n^2(\psi)} G  + \mu(\psi) | \\
\end{array}
  \]    

\end{proof}

\begin{proof}[Proof of theorem~\ref{thm:Bound_FullNoise}]  
Additionally, for $\tilde a^{-1} $, define its norm as
\[
M_q: = \|\tilde a^{-1} \|_{q} = \| a^{-1} \|_{q}
\]

 When $\psi = e_0$, then
 \[
 \begin{array}{ll}
 \mu(e_0) &= \eta_1 + \eta_2 M_{q_2}\\
  \sigma_n(e_0) &=\sqrt{ \sigma_1^2 + \sigma_2^2 M_{2}^2} \\
 \sigma_s((e_0)_I) &= I_0 
 \end{array}
  \]

  
We have 
\[
 \begin{array}{ll}
 F_{\eta_1,\eta_2, \sigma_1,\sigma_2}(e_0)
 &= 
  {\mathbb E}_{G} {\mathbb E}_{I_0} |  \sigma(e_0, (e_0)_{I_0})  G  +  \mu(e_0)| \\
  &= 
 (1-p){\mathbb E}_{G} |  \sigma_n(e_0) G  +  \mu(e_0)|+ p{\mathbb E}_{G} | \sqrt{1+ \sigma^2_n(e_0)}  G  +  \mu(e_0)|\\
   &=
 \sqrt{\frac{2}{\pi}}[ (1-p) \sigma_n(e_0)^2 \mathcal{R}^f(\frac{\mu(e_0)}{\sigma_n(e_0)})+  
 p\sqrt{1+ \sigma^2_n(e_0)} \mathcal{R}^f(\frac{\mu(e_0)}{\sqrt{1+ \sigma^2_n(e_0)}}) ]
\end{array}
  \]
  Using the chain of inequality:
    \[
 \begin{array}{ll}
F_{\eta_1,\eta_2, \sigma_1,\sigma_2}((e_0)
 \geq F_{\eta_1,\eta_2, \sigma_1,\sigma_2}((\psi^\star ) 
 \geq F_{0,0,0,0}(\psi^\star )
 \geq F_{0,0,0,0}(e_0) 
\end{array}
  \] 
and the fact that
  \[
 \begin{array}{ll}
 F_{0,0,0,0}(\psi^\star )
 &= {\mathbb E}_I\|\psi_I\|_2 \\
 &= {\mathbb E}_I\|(e_0)_I\|_2 + ({\mathbb E}_I\|\psi_I\|_2-{\mathbb E}_I\|(e_0)_I\|_2) \\
 &= \sqrt{\frac{2}{\pi}}[p+ B(p,N) \|\psi -e_0\|_2]
\end{array}
  \]   
  From
   \[
 \begin{array}{ll}
 F_{0,0,0,0}(\psi^\star ) &\leq F_{\eta_1,\eta_2, \sigma_1,\sigma_2}(e_0)
\end{array}
  \]   
we have
  \[
 \begin{array}{ll}
p+ B(p,N) \|\psi^\star -e_0\|_2\leq (1-p) \sigma_n(e_0)^2 \mathcal{R}^f(\frac{\mu(e_0)}{\sigma_n(e_0)})+  
 p\sqrt{1+ \sigma^2_n(e_0)} \mathcal{R}^f(\frac{\mu(e_0)}{\sqrt{1+ \sigma^2_n(e_0)}}) 
\end{array}
  \]   

\end{proof}  
 
  
  
  
  
  
  
  
  
  
  
  
  
  





  
  
  

  
  
  
  
  
  
  
  
  
  
   
   





 
 
 
  
  
  









    
    
    
  
    
   
   
 
 
 


 
 

























\section{Robustness for noisy observation}
\subsection{Inner and outer Gaussian and adversarial noise}
\paragraph{Mathematical formulation: sparse signal plus inner and outer adversarial perturbation and Gaussian noise}
\begin{itemize}
\item
Let $(X_t)_{t \in {\mathbb Z}}$ be IID sampled from  $pN(0,1) + (1-p) \delta_0$, write $X_t = I_t \cdot G_t$ where $I_t$ is IID sampled from Bernoulli, $G_t$ is IID sampled from standard Gaussian.
 \item
The filter $a\in l_1({\mathbb Z}), (\mathcal{F} a)(\omega) \neq 0, \forall \omega\in T.$
\item
Let $G_1$ and $G_2$ are two independent standard normal $N(0,I_N)$. 
\item
Let $\zeta_1$ and $\zeta_2$ are inner and outer adversarial noise with $p_1$ and $p_2$ norm bounded by $1$, where for $p_1, p_2\geq 1, $ $q_1$ and $q_2$ is defined by $\frac{1}{p_i}+\frac{1}{q_i} =1.$
\item
Our data generative model is
\[
\begin{array}{ll}
Y&= a* (X+ \eta_1 \zeta_1+  \sigma_1 G_1) + \eta_2 \zeta_2+ \sigma_2 G_2\\
&= a* (X +\eta_1 \zeta_1 +\eta_2 a^{-1}* \zeta_2+  \sigma_1 G_1+\sigma_2 a^{-1}* G_2)
\end{array}
\]
\item
Let the total perturbation be
\[
\begin{array}{ll}
Z= \eta_1 \zeta_1 +\eta_2 a^{-1}* \zeta_2+  \sigma_1 G_1+\sigma_2 a^{-1}* G_2
\end{array}
\]
We can write data generative model  simply as
\[
\begin{array}{ll}
Y&= a* (X+ Z ) 
\end{array}
\]
\end{itemize}
Let $\tilde{\psi} := w*a$,   $\phi: = \psi - e_0$, then
\[
 \begin{array}{ll}
(w*Y)_0
&= \psi^T (X+  Z ) 
\end{array}
  \]

 \paragraph{Perturbation analysis of the robust convex problems}
The original convex problems is
\[
 \begin{array}{ll}
 \mbox{minimize}_{w}   & {\mathbb E}|\tilde{w}^T Y|\\
\mbox{subject to}  
& v^T w =1,
\end{array}
\tag{$P_1$}
  \]  
Our objective is
\[
|\psi^T (X+  Z )|  = |\psi^T X+\psi^T[\sigma_1  G_1+\sigma_2 (a^{-1}* G_2) + \eta_1 \zeta_1 +\eta_2 a^{-1}* \zeta_2] |
\]
\paragraph{Worst case robust objective over adversarial perturbation}  
For the constrained problem, we first take expectation over Gaussian noise and over the Bernoulli Gaussian signal, then we optimize over the worst case expected objective for all adversarial perturbation, we define our robust objective as   
\[
 \begin{array}{ll}
 F_{\eta_1,\eta_2, \sigma_1,\sigma_2}(\psi)
:=
\sup_{\zeta_1, \zeta_2: \|\zeta_1\|_{p_1} \leq 1; \|\zeta_2\|_{p_2} \leq 1}
{\mathbb E}_{G_1} {\mathbb E}_{G_2} {\mathbb E}_X|\psi^T (X+  Z )|
\end{array}
  \]  
then our robust constrained problem becomes
  \[
 \begin{array}{ll}
 \mbox{minimize}_{\psi} \quad &F_{\eta_1,\eta_2, \sigma_1,\sigma_2}(\psi) \\
\mbox{subject to}  
&u^T \psi =1
\end{array}
  \]
Assume that the minimizer of the robust problem is $\psi^\star$, then we have the chain of inequality:
    \[
 \begin{array}{ll}
F_{\eta_1,\eta_2, \sigma_1,\sigma_2}(e_0)
 \geq F_{\eta_1,\eta_2, \sigma_1,\sigma_2}(\psi^\star ) 
 \geq F_{0,0,0,0}(\psi^\star )
 \geq F_{0,0,0,0}(e_0) =p
\end{array}
  \] 

  
In the following,
let 
\[
D_R( \psi) :=  F_{\eta_1,\eta_2, \sigma_1,\sigma_2}(\psi)-F_{0,0,0,0}(\psi).
\]
We use this fact and the bi-lipschitz property of noiseless objective around $e_0$ to bound the $\ell_2$ norm deviation $\|\psi^\star-e_0\|_2.$

Let $\phi^\star := \psi^\star-e_0$,
we have
   \[
 \begin{array}{ll}
D_R( e_0) &= F_{\eta_1,\eta_2, \sigma_1,\sigma_2}(e_0)-F_{0,0,0,0}(e_0)\\
 &\geq F_{0,0,0,0}(\psi^\star )-F_{0,0,0,0}(e_0) \\
 &\geq B(p,\phi^\star) \|\psi^\star-e_0\|_2\\
 &\geq p (
(1-p)- \tan(\theta) )\cos{\theta} \|\phi^\star\|_2
\end{array}
  \] 

This bound is not tight but it is tight in order and it is easy to compute since we only need to compute  $F_{\eta_1,\eta_2, \sigma_1,\sigma_2}(e_0)$, the robust objective at $e_0$.
 
\subsection{Folded Gaussian}
\paragraph{Folded Gaussian}
For Bernoulli Gaussian $X$ with support $I$, if we condition on $I$ fixed, then  using the rotation invariance of Gaussian, $\psi^T X$ is  Gaussian with mean $0$ and standard deviation $\|\psi_I\|_2$.

From general theory of folded Gaussian,
\footnote{see: \url{ https://en.wikipedia.org/wiki/Folded_normal_distribution}.}
 $ |\mu+ \sigma G |, \quad  G \sim N(0, 1).$
Then its mean is
\begin{equation}
\label{FoldedGaussianMean}
E_G |\mu+ \sigma G| =  \sqrt{\frac{2}{\pi}} \sigma \exp\{-\mu^2/(2\sigma^2)\} + \mu \left(1 - 2\Phi\left(\frac{-\mu}{\sigma}\right) \right)
\end{equation}
where $\Phi$  is the normal cumulative distribution function:
\[
\Phi(x) = \frac 1 {\sqrt{2\pi}} \int_{-\infty}^x e^{-t^2/2} \, dt.
\]
Its variance is
\begin{equation}
\label{FoldedGaussianVar}
{\rm Var}_G |\mu+ \sigma G| = E_G( |\mu+ \sigma G|-E_G |\mu+ \sigma G|)^2 =\sigma^2+ \mu^2 -(E_G |\mu+ \sigma G|)^2
\end{equation}  
The variance will be useful for concentration with finite sample.  

\paragraph{Asymptotic expansion for mean of folded Gaussian}
Notice that if $\sigma =0$, then
\[
E_G |\mu+ 0\cdot G| =   |\mu|.
\]
If $\mu =0$, then we go back to mean of half-normal,
\[
E_G |\sigma G| =   \sqrt{\frac{2}{\pi}} \sigma
\]
Let 
\[
\gamma:= \frac{\mu}{\sigma}
\] 

\begin{lemma}
\label{ExpansionMeanFoldedGaussian}
We define the ratio of folded gaussian mean as 
\[
\mathcal{R}(\gamma) := \frac{E_G |\mu+ \sigma G|}{E_G |\sigma G|} = \frac{E_G |\mu+ \sigma G|}{\sqrt{\frac{2}{\pi}}\sigma} = \sqrt{\frac{\pi}{2}} E_G |\gamma+  G| = \exp\{-\gamma^2/2\} + \sqrt{\frac{\pi}{2}}\gamma \left(1 - 2\Phi\left(-\gamma\right) \right)
\]
We have three different expressions (asymptotic expansion around  $\gamma$) for $\mathcal{R}(\gamma)$:
\begin{eqnarray}
\label{FoldedGaussianMean_ratio}
\mathcal{R}(\gamma)
&=&  \exp\{-\gamma^2/2\} + \sqrt{\frac{\pi}{2}}\gamma \left(1 - 2\Phi\left(-\gamma\right) \right)\\
&=&  \exp\{-\gamma^2/2\}\{1 + \frac{1}{2}\gamma \left[ \gamma + \frac{ \gamma^3}{3} + \frac{ \gamma^5}{3\cdot 5} + \cdots + \frac{ \gamma^{2n+1}}{(2n+1)!!} + \cdots\right]\}\\
&=& 1 + \frac{3}{2}\gamma^2 + \frac{1}{24} \gamma^4   - \frac{ \gamma^6}{120}  + O(\gamma^8) 
\end{eqnarray}
 \end{lemma}
 
  \begin{lemma}
 \label{ExpansionMeanFoldedGaussian_Monot}
$\mathcal{R}(\gamma)$ is an even function that is monotonically non-decreasing for $\gamma \geq 0$.
\end{lemma}
 
 \begin{lemma}
 \label{ExpansionMeanFoldedGaussian_LB}
when $\gamma<2,$ 
\[
\mathcal{R}(\gamma) \geq 1 + \frac{3}{2}\gamma^2
\]
\end{lemma}


\paragraph{Robust optimization}
\begin{lemma}
From Hölder's inequality, for any vector $a \in {\mathbb R}^N$, for $p\geq 1, \frac{1}{p}+\frac{1}{q} =1,$
\[
\sup_{Z: \|Z\|_p\leq 1}(a^T Z) = \|a\|_q,
\]
specifically, when $|Z|^p$ is along the direction of $\mbox{sign}(a)|a|^q$ so that $a^T Z = \|a\|_q$.

\end{lemma}  



\paragraph{Bound on the noisy solution using robust objective}  
\begin{lemma}
\label{lemma:Obj_Folded_FullNoise}
Let 
 \[
 \begin{array}{ll}
 \mu(\psi) &=\eta_1\|\psi\|_{q_1}+ \eta_2\|C_{a^{-1}}^T \psi\|_{q_2}\\
 \sigma_n^2(\psi) &=  \sigma_1^2\|\psi\|_2^2 + \sigma_2^2\|C_{a^{-1}}^T \psi\|_2^2
 \end{array}
  \]
then the objective is a mean of a folded Gaussian random variable:
\[
 \begin{array}{ll}
 F_{\eta_1,\eta_2, \sigma_1,\sigma_2}(\psi)
&= \sup_{\zeta_1, \zeta_2: \|\zeta_1\|_{p_1} \leq 1; \|\zeta_2\|_{p_2} \leq 1}{\mathbb E}_I{\mathbb E}_{G}  | \sqrt{\|\psi_I\|_2^2 + \sigma_n^2(\psi)} G  + \eta_1 \zeta_1 +\eta_2 a^{-1}* \zeta_2 |\\
&= {\mathbb E}_I{\mathbb E}_{G}  | \sqrt{\|\psi_I\|_2^2 + \sigma_n^2(\psi)} G  + \sup_{\zeta_1, \zeta_2: \|\zeta_1\|_{p_1} \leq 1; \|\zeta_2\|_{p_2} \leq 1}[\eta_1 \zeta_1 +\eta_2 a^{-1}* \zeta_2] |\\
&= {\mathbb E}_I{\mathbb E}_{G}  | \sqrt{\|\psi_I\|_2^2 + \sigma_n^2(\psi)} G  + \mu(\psi) |\\
&= \sqrt{\frac{2}{\pi}} {\mathbb E}_I[ \sqrt{\|\psi_I\|_2^2 + \sigma_n^2(\psi)} 
\mathcal{R}(\frac{\mu(\psi)}{\sqrt{\|\psi_I\|_2^2 + \sigma_n^2(\psi)}})]
\end{array}
  \] 
  where $G$ is scalar standard Gaussian.
 
 The second inequality comes from the fact that $\mathcal{R}(\gamma)$ is an even function that is monotonically non-decreasing for $\gamma \geq 0$.
  
\end{lemma}
From a simple inequality:
   \[
 \begin{array}{ll}
 F_{0,0,0,0}(\psi^\star ) &\leq F_{\eta_1,\eta_2, \sigma_1,\sigma_2}(e_0)
\end{array}
  \]  
we have  
\begin{theorem}
\label{thm:Bound_FullNoise}
For $p<p^\star_1 = 1-\tan(\theta),$
 \[
 \begin{array}{ll}
 \|\phi^\star\|_2\leq 
  \frac{(1-p) \sigma_n(e_0)^2 \mathcal{R}(\frac{\mu(e_0)}{\sigma_n(e_0)})+  
 p\sqrt{1+ \sigma^2_n(e_0)} \mathcal{R}(\frac{\mu(e_0)}{\sqrt{1+ \sigma^2_n(e_0)}}) - p}{p (
(1-p)- \tan(\theta) )\cos{\theta} }
\end{array}
  \]  

 
\end{theorem}  
\begin{proof}
 Since
 \[
 \begin{array}{ll}
  B(p,\phi^\star) \|\phi^\star\|_2\leq 
 D_R(e_0) = F_{\eta_1,\eta_2, \sigma_1,\sigma_2}(e_0)-F_{0,0,0,0}(e_0) 
\end{array}
  \] 
  and
   \[
 \begin{array}{ll}
 p (
(1-p)- \tan(\theta) )\cos{\theta} \|\phi^\star\|_2\leq  B(p,\phi^\star) \|\phi^\star\|_2 
\end{array}
  \]  
  where
\[
 \begin{array}{ll}
 D_R(e_0) = (1-p) \sigma_n(e_0)^2 \mathcal{R}(\frac{\mu(e_0)}{\sigma_n(e_0)})+  
 p\sqrt{1+ \sigma^2_n(e_0)} \mathcal{R}(\frac{\mu(e_0)}{\sqrt{1+ \sigma^2_n(e_0)}}) - p
\end{array}
  \]  
where
\[
 \begin{array}{ll}
 \mu(e_0) &= (\eta_1 + \eta_2 \| a^{-1} \|_{q_2}-\epsilon)_+\\
 \sigma_n(e_0) &=\sqrt{ \sigma_1^2 + \sigma_2^2\|a^{-1}\|_2^2} 
 \end{array}
  \]
\end{proof}
\begin{coro}
When $\sigma_1 = \sigma_2 = 0,$
 \[
 \begin{array}{ll}
\|\psi^\star -e_0\|_2\leq 
 \frac{p\mathcal{R}(\frac{\mu(e_0)}{\sqrt{1+ \sigma^2_n(e_0)}}) - p}{p (
(1-p)- \tan(\theta) )\cos{\theta} }
\end{array}
  \]  

\end{coro}

\begin{coro}
When $\eta_1 = \eta_2 = 0,$
 \[
 \begin{array}{ll}
 \|\psi^\star -e_0\|_2\leq 
 \frac{ (1-p) \sigma_n(e_0)^2 +  
 p\sqrt{1+ \sigma^2_n(e_0)}  - p}{p (
(1-p)- \tan(\theta) )\cos{\theta} }
\end{array}
  \]  

\end{coro}








\subsection{All the proofs}    
\begin{proof}[Proof of lemma~\ref{lemma:Obj_Folded_FullNoise}]
 First, for any fixed vector $\mu$,
\[
 \begin{array}{ll}
  {\mathbb E}_{G_1} {\mathbb E}_{G_2} {\mathbb E}_X|\psi^T X+\sigma_1 \psi^T G_1+\sigma_2 \psi^T (a^{-1}* G_2) + \mu | = 
   {\mathbb E}_I{\mathbb E}_{G_3}  | \sqrt{ \|\psi_I\|_2^2 + \sigma_1^2\|\psi\|_2^2 + \sigma_2^2\|C_{a^{-1}}^T \psi\|_2^2}G_3  + \mu|
\end{array}
  \]
  Let 
 \[
 \begin{array}{ll}
 \mu(\psi) &= (\eta_1\|\psi\|_{q_1}+ \eta_2\|C_{a^{-1}}^T \psi\|_{q_2}-\epsilon)_+\\
 \sigma_n^2(\psi) &= \sqrt{ \sigma_1^2\|\psi\|_2^2 + \sigma_2^2\|C_{a^{-1}}^T \psi\|_2^2}\\
  \sigma_s^2(\psi_I) &=  \|\psi_I\|_2^2 \\
 \sigma^2 (\psi, \psi_I) &= \sigma_s^2(\psi_I) + \sigma_n^2(\psi) 
 \end{array}
  \]
 Using lemma~\ref{min-maxErrorL_p}, we have
\[
 \begin{array}{ll}
F_{\eta_1,\eta_2, \sigma_1,\sigma_2}(\psi)
&={\mathbb E}_I{\mathbb E}_{G}  | \sigma (\psi, \psi_I) G  + \mu(\psi) | \\
&={\mathbb E}_I{\mathbb E}_{G}  | \sqrt{\sigma_s^2(\psi_I) + \sigma_n^2(\psi)} G  + \mu(\psi) | \\
\end{array}
  \]    

\end{proof}

\begin{proof}[Proof of theorem~\ref{thm:Bound_FullNoise}]  
Additionally, for $\tilde a^{-1} $, define its norm as
\[
M_q: = \|\tilde a^{-1} \|_{q} = \| a^{-1} \|_{q}
\]

 When $\psi = e_0$, then
 \[
 \begin{array}{ll}
 \mu(e_0) &= \eta_1 + \eta_2 M_{q_2}\\
  \sigma_n(e_0) &=\sqrt{ \sigma_1^2 + \sigma_2^2 M_{2}^2} \\
 \sigma_s((e_0)_I) &= I_0 
 \end{array}
  \]

  
We have 
\[
 \begin{array}{ll}
 F_{\eta_1,\eta_2, \sigma_1,\sigma_2}(e_0)
 &= 
  {\mathbb E}_{G} {\mathbb E}_{I_0} |  \sigma(e_0, (e_0)_{I_0})  G  +  \mu(e_0)| \\
  &= 
 (1-p){\mathbb E}_{G} |  \sigma_n(e_0) G  +  \mu(e_0)|+ p{\mathbb E}_{G} | \sqrt{1+ \sigma^2_n(e_0)}  G  +  \mu(e_0)|\\
   &=
 \sqrt{\frac{2}{\pi}}[ (1-p) \sigma_n(e_0)^2 \mathcal{R}(\frac{\mu(e_0)}{\sigma_n(e_0)})+  
 p\sqrt{1+ \sigma^2_n(e_0)} \mathcal{R}(\frac{\mu(e_0)}{\sqrt{1+ \sigma^2_n(e_0)}}) ]
\end{array}
  \]
  Using the chain of inequality:
    \[
 \begin{array}{ll}
F_{\eta_1,\eta_2, \sigma_1,\sigma_2}((e_0)
 \geq F_{\eta_1,\eta_2, \sigma_1,\sigma_2}((\psi^\star ) 
 \geq F_{0,0,0,0}(\psi^\star )
 \geq F_{0,0,0,0}(e_0) 
\end{array}
  \] 
and the fact that
  \[
 \begin{array}{ll}
 F_{0,0,0,0}(\psi^\star )
 &= {\mathbb E}_I\|\psi_I\|_2 \\
 &= {\mathbb E}_I\|(e_0)_I\|_2 + ({\mathbb E}_I\|\psi_I\|_2-{\mathbb E}_I\|(e_0)_I\|_2) \\
 &= \sqrt{\frac{2}{\pi}}[p+ B(p,N) \|\psi -e_0\|_2]
\end{array}
  \]   
  From
   \[
 \begin{array}{ll}
 F_{0,0,0,0}(\psi^\star ) &\leq F_{\eta_1,\eta_2, \sigma_1,\sigma_2}(e_0)
\end{array}
  \]   
we have
  \[
 \begin{array}{ll}
p+ B(p,N) \|\psi^\star -e_0\|_2\leq (1-p) \sigma_n(e_0)^2 \mathcal{R}(\frac{\mu(e_0)}{\sigma_n(e_0)})+  
 p\sqrt{1+ \sigma^2_n(e_0)} \mathcal{R}(\frac{\mu(e_0)}{\sqrt{1+ \sigma^2_n(e_0)}}) 
\end{array}
  \]   

\end{proof}  







\begin{proof}[Proof of lemma~\ref{ExpansionMeanFoldedGaussian}]
Plug in the formula for mean of folded Gaussian \ref{FoldedGaussianMean}, 
we have
\[
E_G |\mu+ \sigma G| = \sqrt{\frac{2}{\pi}} \sigma \exp\{-\gamma^2/2\} + \mu \left(1 - 2\Phi\left(-\gamma\right) \right)
\]
Divide it by 
\[
E_G |\sigma G| =   \sqrt{\frac{2}{\pi}} \sigma
\]
we get the first equality.

Since the CDF of the standard normal distribution can be expanded by integration by parts into a series:
\[
\Phi( \gamma)=\frac{1}{2} + \frac{1}{\sqrt{2\pi}}\cdot e^{- \gamma^2/2} \left[ \gamma + \frac{ \gamma^3}{3} + \frac{ \gamma^5}{3\cdot 5} + \cdots + \frac{ \gamma^{2n+1}}{(2n+1)!!} + \cdots\right]
\]
where $!!$ denotes the double factorial. 

\[
1 - 2\Phi(-\gamma)  = \frac{1}{\sqrt{2\pi}}\cdot e^{- \gamma^2/2}\left[ \gamma + \frac{ \gamma^3}{3} + \frac{ \gamma^5}{3\cdot 5} + \cdots + \frac{ \gamma^{2n+1}}{(2n+1)!!} + \cdots\right]
\]
This gives the second inequality.

Additionally,
 \[
 \begin{array}{ll}
  e^{- \gamma^2/2} &=\sum_{n=0}^{\infty} \frac{ \gamma^{2n}}{(-2)^n (n)!}\\
  &=  1- \frac{ \gamma^2}{2} + \sum_{n=2}^{\infty} \frac{ \gamma^{2n}}{(-2)^n (n)!}
\end{array}
  \] 
then
 \[
 \begin{array}{ll}
& \exp\{-\gamma^2/2\}\{1 + \frac{1}{2}\gamma \left[ \gamma + \frac{ \gamma^3}{3} + \frac{ \gamma^5}{3\cdot 5} + \cdots + \frac{ \gamma^{2n+1}}{(2n+1)!!} + \cdots\right]\}\\
=& (\sum_{n=0}^{\infty} \frac{ \gamma^{2n}}{(-2)^n (n)!})\{1 + \frac{1}{2}\gamma \left[ \sum_{n=0}^{\infty} \frac{ \gamma^{2n+1}}{(2n+1)!!} \right]\}\\

=& 1 + \frac{3}{2}\gamma^2 + \frac{1}{24} \gamma^4   - \frac{ \gamma^6}{120}  + O(\gamma^8) 
\end{array}
  \] 
\end{proof}


\begin{proof}[Proof of lemma~\ref{ExpansionMeanFoldedGaussian_LB}]
Let $C_\Phi( \gamma^2)$ be a function of $ \gamma$, then it is a composition function with the inner funciton being $ \gamma^2,$ defined as follows: 
\[
 \begin{array}{ll}
C_\Phi( \gamma^2) 
&:= (2\Phi( \gamma)-1)/\left(\frac{ \gamma}{\sqrt{2\pi}}\right) \\
&= e^{- \gamma^2/2} \left[1 + \frac{ \gamma^2}{3} + \frac{ \gamma^4}{3\cdot 5} + \cdots + \frac{ \gamma^{2n}}{(2n+1)!!} + \cdots\right]\\
&= [\sum_{n=0}^{\infty} \frac{ \gamma^{2n}}{(2n+1)!!} ]/[\sum_{n=0}^{\infty} \frac{ \gamma^{2n}}{2^n (n)!}] \\
&= [\sum_{n=0}^{\infty} \frac{ \gamma^{2n}}{(2n+1)!!} ]/[\sum_{n=0}^{\infty} \frac{ \gamma^{2n}}{(2n)!!}] \\
&\leq 1
\end{array}
\]
We remark further that
\[
 \begin{array}{ll}
C_\Phi( \gamma^2) 
&= [\sum_{n=0}^{\infty} \frac{ \gamma^{2n}}{(2n+1)!!} ]/[\sum_{n=0}^{\infty} \frac{ \gamma^{2n}}{(2n)!!}] \\
&= [1 + \frac{ \gamma^2}{3}+ +  \frac{ \gamma^4}{15}+ O( \gamma^6) ]/[1 + \frac{ \gamma^2}{2}+ +  \frac{ \gamma^4}{8}+O( \gamma^6)] \\
&= 1 -\frac{ \gamma^2}{6}+  \frac{ \gamma^4}{40}+ O( \gamma^6)
\end{array}
\]
Then
 \[
 \begin{array}{ll}
\mu \left(1 - 2\Phi\left(-\gamma\right) \right)
&= \frac{1}{\sqrt{2\pi}}\cdot \mu^2 \frac{1}{\sigma} C_\Phi(\gamma^2)\\
&= \frac{1}{\sqrt{2\pi}}\cdot \mu^2 \frac{1}{\sigma} (1-\gamma^2/6 +O(\gamma^4))\\
\end{array}
\] 
Plug in all the fact together
\[
 \begin{array}{ll}
E_G |\mu+ \sigma G|
&=   \left[ \sqrt{\frac{2}{\pi}} \sigma \exp\{-\gamma^2/2\} + \frac{1}{\sqrt{2\pi}}\cdot \mu^2 \frac{1}{\sigma} C_\Phi(\gamma^2)\right]\\
&=   \left[ \sqrt{\frac{2}{\pi}} \sigma \{ 1- \frac{\gamma^2}{2} + \sum_{n=2}^{\infty} \frac{\gamma^{2n}}{(-2)^n (n)!}\} + \frac{1}{\sqrt{2\pi}}\cdot \mu^2 \frac{1}{\sigma} C_\Phi(\gamma^2)\right]\\
&= \sqrt{\frac{2}{\pi}}   \left[ \sigma 
\{ 1 + (1+\frac{1}{2}\cdot C_\Phi(\gamma^2))\gamma^2 +   \sum_{n=2}^{\infty} \frac{\gamma^{2n}}{(-2)^n (n)!}\} \}\right]\\
\end{array}
\] 
Define the residual of this expansion as
\[
 \begin{array}{ll}
\mbox{Res}(\gamma^2) 
&:= \frac{ (2\Phi( \gamma)-1)/\left(\frac{ \gamma}{\sqrt{2\pi}}\right)-1}{2}\cdot \gamma^2 +  (\exp\{-\gamma^2/2\} - 1 + \gamma^2/2)
\\
&= \frac{ C_\Phi(\gamma^2)-1}{2}\cdot \gamma^2 +  (\exp\{-\gamma^2/2\} - 1 + \gamma^2/2) \\
&=    \frac{ C_\Phi(\gamma^2)-1}{2}\cdot \gamma^2  +   \{ \sum_{n=2}^{\infty} \frac{\gamma^{2n}}{(-2)^n (n)!}\} \} \\
&=    \frac{-\gamma^4 }{12}  +    \frac{ \gamma^6}{80} +\frac{\gamma^{4}}{(-2)^2 (2)!} +\frac{\gamma^{6}}{(-2)^3 (3)!}+
O(\gamma^8)\\
&=    \frac{\gamma^4 }{24}  - \frac{ \gamma^6}{120}  + O(\gamma^8)
\end{array}
\] 
We know that if $\gamma^2<1,$ 
\[
 \begin{array}{ll}
\mbox{Res}(\gamma^2) 
&=    \frac{\gamma^4 }{24}  - \frac{ \gamma^6}{120}  + O(\gamma^8)\\
&\geq 0
\end{array}
\] 
(We comment that we could push the range for $\mbox{Res}(\gamma^2) 
\geq 0$ further to at least $\gamma^2<5$ if needed.)
Then we give an lower bound:
\[
 \begin{array}{ll}
E_G |\mu+ \sigma G| 
&= \sqrt{\frac{2}{\pi}} \sigma\left[ 1 + \frac{3}{2}\gamma^2+   \mbox{Res}\right]\\
&\geq \sqrt{\frac{2}{\pi}}\left[  \sigma + \frac{3}{2}\mu^2  \sigma^{-1}\right]
\end{array}
\] 


\end{proof}




\section{Stability for noisy observation}
\subsection{Inner and outer Gaussian and adversarial noise}
\paragraph{Mathematical formulation: sparse signal plus inner and outer adversarial perturbation and Gaussian noise}
\begin{itemize}
\item
Let $(X_t)_{t \in {\mathbb Z}}$ be IID sampled from  $pN(0,1) + (1-p) \delta_0$, write $X_t = I_t \cdot G_t$ where $I_t$ is IID sampled from Bernoulli, $G_t$ is IID sampled from standard Gaussian.
 \item
The filter $a\in l_1({\mathbb Z}), (\mathcal{F} a)(\omega) \neq 0, \forall \omega\in T.$
\item
Let $G_1$ and $G_2$ are two independent standard normal $N(0,I_N)$. 
\item
Let $Z_1$ and $Z_2$ are inner and outer adversarial noise with $p_1$ and $p_2$ norm bounded by $1$, where for $p_1, p_2\geq 1, $ $q_1$ and $q_2$ is defined by $\frac{1}{p_i}+\frac{1}{q_i} =1.$
\item
Our data generative model is
\[
\begin{array}{ll}
Y&= a* (X+ \eta_1 Z_1+  \sigma_1 G_1) + \eta_2 Z_2+ \sigma_2 G_2\\
&= a* (X +\eta_1 Z_1 +\eta_2 a^{-1}* Z_2+  \sigma_1 G_1+\sigma_2 a^{-1}* G_2)
\end{array}
\]
\item
Let the total perturbation be
\[
\begin{array}{ll}
Z= \eta_1 Z_1 +\eta_2 a^{-1}* Z_2+  \sigma_1 G_1+\sigma_2 a^{-1}* G_2
\end{array}
\]
We can write data generative model  simply as
\[
\begin{array}{ll}
Y&= a* (X+ Z ) 
\end{array}
\]
\end{itemize}
Let $\tilde{\psi} := w*a$,   $\phi: = \psi - e_0$, then
\[
 \begin{array}{ll}
(w*Y)_0
&= \psi^T (X+  Z ) 
\end{array}
  \]

 \paragraph{Perturbation analysis of the robust convex problems}
The original convex problems is
\[
 \begin{array}{ll}
 \mbox{minimize}_{w}   & {\mathbb E}|\tilde{w}^T Y|\\
\mbox{subject to}  
& v^T w =1,
\end{array}
\tag{$P_1$}
  \]  
Our objective is
\[
|\psi^T (X+  Z )-\epsilon \zeta|  = |\psi^T X+\psi^T[\sigma_1  G_1+\sigma_2 (a^{-1}* G_2) + \eta_1 Z_1 +\eta_2 a^{-1}* Z_2]-\epsilon \zeta |
\]
\paragraph{Worst case robust objective over adversarial perturbation}  
For the constrained problem, we first take expectation over Gaussian noise and over the Bernoulli Gaussian signal, then we optimize over the worst case expected objective for all adversarial perturbation, we define our robust objective as   
\[
 \begin{array}{ll}
 F_{\eta_1,\eta_2, \sigma_1,\sigma_2}(\psi)
:=
\inf_{\zeta^2\leq 1}\sup_{Z_1, Z_2: \|Z_1\|_{p_1} \leq 1; \|Z_2\|_{p_2} \leq 1}
{\mathbb E}_{G_1} {\mathbb E}_{G_2} {\mathbb E}_X|\psi^T (X+  Z )-\epsilon \zeta|
\end{array}
  \]  
then our robust constrained problem becomes
  \[
 \begin{array}{ll}
 \mbox{minimize}_{\psi} \quad &F_{\eta_1,\eta_2, \sigma_1,\sigma_2}(\psi) \\
\mbox{subject to}  
&u^T \psi =1
\end{array}
  \]
Assume that the minimizer of the robust problem is $\psi^\star$, then we have the chain of inequality:
    \[
 \begin{array}{ll}
F_{\eta_1,\eta_2, \sigma_1,\sigma_2}(e_0)
 \geq F_{\eta_1,\eta_2, \sigma_1,\sigma_2}(\psi^\star ) 
 \geq F_{0,0,0,0}(\psi^\star )
 \geq F_{0,0,0,0}(e_0) =p
\end{array}
  \] 
In the following, we use this fact and the bi-lipschitz property of noiseless objective around $e_0$ to bound the $\ell_2$ norm deviation $\|\psi^\star-e_0\|_2.$
   \[
 \begin{array}{ll}
F_{\eta_1,\eta_2, \sigma_1,\sigma_2}(e_0)-F_{0,0,0,0}(e_0)
 &\geq F_{0,0,0,0}(\psi^\star )-F_{0,0,0,0}(e_0) \\
 &\geq B(p,N) \|\psi^\star-e_0\|_2
\end{array}
  \] 

This bound is not tight but it is tight in order and it is easy to compute since we only need to compute  $F_{\eta_1,\eta_2, \sigma_1,\sigma_2}(e_0)$, the robust objective at $e_0$.
 
\subsection{Folded Gaussian}
\paragraph{Folded Gaussian}
For Bernoulli Gaussian $X$ with support $I$, if we condition on $I$ fixed, then  using the rotation invariance of Gaussian, $\psi^T X$ is  Gaussian with mean $0$ and standard deviation $\|\psi_I\|_2$.

From general theory of folded Gaussian,
\footnote{see: \url{ https://en.wikipedia.org/wiki/Folded_normal_distribution}.}
 $ |\mu+ \sigma G |, \quad  G \sim \mathcal{N}(0, 1).$
Then its mean is
\begin{equation}
\label{FoldedGaussianMean}
E_G |\mu+ \sigma G| =  \sqrt{\frac{2}{\pi}} \sigma \exp\{-\mu^2/(2\sigma^2)\} + \mu \left(1 - 2\Phi\left(\frac{-\mu}{\sigma}\right) \right)
\end{equation}
where $\Phi$  is the normal cumulative distribution function:
\[
\Phi(x) = \frac 1 {\sqrt{2\pi}} \int_{-\infty}^x e^{-t^2/2} \, dt.
\]
Its variance is
\begin{equation}
\label{FoldedGaussianVar}
{\rm Var}_G |\mu+ \sigma G| = E_G( |\mu+ \sigma G|-E_G |\mu+ \sigma G|)^2 =\sigma^2+ \mu^2 -(E_G |\mu+ \sigma G|)^2
\end{equation}  
The variance will be useful for concentration with finite sample.  

\paragraph{Asymptotic expansion for mean of folded Gaussian}
Notice that if $\sigma =0$, then
\[
E_G |\mu+ 0\cdot G| =   |\mu|.
\]
If $\mu =0$, then we go back to mean of half-normal,
\[
E_G |\sigma G| =   \sqrt{\frac{2}{\pi}} \sigma
\]
Let 
\[
\gamma:= \frac{\mu}{\sigma}
\] 

\begin{lemma}
\label{ExpansionMeanFoldedGaussian}
We define the ratio of folded gaussian mean as 
\[
\mathcal{R}^f(\gamma) := \frac{E_G |\mu+ \sigma G|}{E_G |\sigma G|} = \frac{E_G |\mu+ \sigma G|}{\sqrt{\frac{2}{\pi}}\sigma} = \sqrt{\frac{\pi}{2}} E_G |\gamma+  G|
\]
We have three different expressions (asymptotic expansion around  $\gamma$) for $\mathcal{R}^f(\gamma)$:
\begin{eqnarray}
\label{FoldedGaussianMean_ratio}
\mathcal{R}^f(\gamma)
&=&  \exp\{-\gamma^2/2\} + \sqrt{\frac{\pi}{2}}\gamma \left(1 - 2\Phi\left(-\gamma\right) \right)\\
&=&  \exp\{-\gamma^2/2\}\{1 + \frac{1}{2}\gamma \left[ \gamma + \frac{ \gamma^3}{3} + \frac{ \gamma^5}{3\cdot 5} + \cdots + \frac{ \gamma^{2n+1}}{(2n+1)!!} + \cdots\right]\}\\
&=& 1 + \frac{3}{2}\gamma^2 + \frac{1}{24} \gamma^4   - \frac{ \gamma^6}{120}  + O(\gamma^8) 
\end{eqnarray}
 \end{lemma}
 
 \begin{lemma}
 \label{ExpansionMeanFoldedGaussian_LB}
when $\gamma<2,$ 
\[
\mathcal{R}^f(\gamma) \geq 1 + \frac{3}{2}\gamma^2
\]
\end{lemma}


\paragraph{Robust optimization}
We want to find the min-max value 
\[
\inf_{\zeta^2\leq 1}\sup_{Z_1, Z_2: \|Z_1\|_{p_1} \leq 1; \|Z_2\|_{p_2} \leq 1} H( \psi^T (\eta_1 Z_1 +\eta_2 a^{-1}* Z_2)  - \epsilon \zeta)
\]
for a function $H$,
\[
H(t) := {\mathbb E}_{G_1} {\mathbb E}_{G_2} {\mathbb E}_X|t+ \psi^T (X+  \sigma_1  G_1+\sigma_2 (a^{-1}* G_2) )|
\]
\begin{lemma}
From Hölder's inequality, for any vector $a \in {\mathbb R}^N$, for $p\geq 1, \frac{1}{p}+\frac{1}{q} =1,$
\[
\sup_{Z: \|Z\|_p\leq 1}(a^T Z) = \|a\|_q,
\]
specifically, when $|Z|^p$ is along the direction of $\mbox{sign}(a)|a|^q$ so that $a^T Z = \|a\|_q$.

\end{lemma}  


\begin{lemma}
\label{min-maxError}
For $A>0$, 
\[
 \begin{array}{ll}
\inf_{\zeta: \zeta^2 \leq 1} (A - \epsilon \zeta) = (A - \epsilon )_+
\end{array}
\]
where $(\cdot)_+: = \max{(0, \cdot)}$ is the ReLU function.
\end{lemma}
\begin{lemma}
\label{min-maxErrorL_p}
Let
 \[
 \begin{array}{ll}
 \mu(\psi) &=(\eta_1\|\psi\|_{q_1}+ \eta_2\|C_{a^{-1}}^T \psi\|_{q_2}-\epsilon)_+
 \end{array}
  \]
  where $(\cdot)_+: = \max{(0, \cdot)}$ is the ReLU function,
  then
\[
 \begin{array}{ll}
\inf_{\zeta^2\leq 1}\sup_{Z_1, Z_2: \|Z_1\|_{p_1} \leq 1; \|Z_2\|_{p_2} \leq 1} H( \psi^T (\eta_1 Z_1 +\eta_2 a^{-1}* Z_2)  - \epsilon \zeta) = 
H( \mu(\psi))
\end{array}
\]

\end{lemma}
\paragraph{Bound on the noisy solution using robust objective}  
\begin{lemma}
\label{lemma:Obj_Folded_FullNoise}
Let 
 \[
 \begin{array}{ll}
 \mu(\psi) &=(\eta_1\|\psi\|_{q_1}+ \eta_2\|C_{a^{-1}}^T \psi\|_{q_2}-\epsilon)_+\\
 \sigma_n^2(\psi) &= \sqrt{ \sigma_1^2\|\psi\|_2^2 + \sigma_2^2\|C_{a^{-1}}^T \psi\|_2^2}
 \end{array}
  \]
then the objective is a mean of a folded Gaussian random variable:
\[
 \begin{array}{ll}
 F_{\eta_1,\eta_2, \sigma_1,\sigma_2}(\psi)
&= {\mathbb E}_I{\mathbb E}_{G}  | \sqrt{\|\psi_I\|_2^2 + \sigma_n^2(\psi)} G  + \mu(\psi) |\\
&= \sqrt{\frac{2}{\pi}} {\mathbb E}_I[ \sqrt{\|\psi_I\|_2^2 + \sigma_n^2(\psi)} 
\mathcal{R}^f(\frac{\mu(\psi)}{\sqrt{\|\psi_I\|_2^2 + \sigma_n^2(\psi)}})]
\end{array}
  \] 
  where $G$ is scalar standard Gaussian.
\end{lemma}
From a simple inequality:
   \[
 \begin{array}{ll}
 F_{0,0,0,0}(\psi^\star ) &\leq F_{\eta_1,\eta_2, \sigma_1,\sigma_2}(e_0)
\end{array}
  \]  
we have  
\begin{theorem}
\label{thm:Bound_FullNoise}
\[
 \begin{array}{ll}
 B(p,N) \|\psi^\star -e_0\|_2\leq (1-p) \sigma_n(e_0)^2 \mathcal{R}^f(\frac{\mu(e_0)}{\sigma_n(e_0)})+  
 p\sqrt{1+ \sigma^2_n(e_0)} \mathcal{R}^f(\frac{\mu(e_0)}{\sqrt{1+ \sigma^2_n(e_0)}}) - p
\end{array}
  \]  
where
\[
 \begin{array}{ll}
 \mu(e_0) &= (\eta_1 + \eta_2 \| a^{-1} \|_{q_2}-\epsilon)_+\\
 \sigma_n(e_0) &=\sqrt{ \sigma_1^2 + \sigma_2^2\|a^{-1}\|_2^2} 
 \end{array}
  \]
\end{theorem}  

\subsection{Proofs}    
\begin{proof}[Proof of lemma~\ref{lemma:Obj_Folded_FullNoise}]
 First, for any fixed vector $\mu$,
\[
 \begin{array}{ll}
  {\mathbb E}_{G_1} {\mathbb E}_{G_2} {\mathbb E}_X|\psi^T X+\sigma_1 \psi^T G_1+\sigma_2 \psi^T (a^{-1}* G_2) + \mu | = 
   {\mathbb E}_I{\mathbb E}_{G_3}  | \sqrt{ \|\psi_I\|_2^2 + \sigma_1^2\|\psi\|_2^2 + \sigma_2^2\|C_{a^{-1}}^T \psi\|_2^2}G_3  + \mu|
\end{array}
  \]
  Let 
 \[
 \begin{array}{ll}
 \mu(\psi) &= (\eta_1\|\psi\|_{q_1}+ \eta_2\|C_{a^{-1}}^T \psi\|_{q_2}-\epsilon)_+\\
 \sigma_n^2(\psi) &= \sqrt{ \sigma_1^2\|\psi\|_2^2 + \sigma_2^2\|C_{a^{-1}}^T \psi\|_2^2}\\
  \sigma_s^2(\psi_I) &=  \|\psi_I\|_2^2 \\
 \sigma^2 (\psi, \psi_I) &= \sigma_s^2(\psi_I) + \sigma_n^2(\psi) 
 \end{array}
  \]
 Using lemma~\ref{min-maxErrorL_p}, we have
\[
 \begin{array}{ll}
F_{\eta_1,\eta_2, \sigma_1,\sigma_2}(\psi)
&={\mathbb E}_I{\mathbb E}_{G}  | \sigma (\psi, \psi_I) G  + \mu(\psi) | \\
&={\mathbb E}_I{\mathbb E}_{G}  | \sqrt{\sigma_s^2(\psi_I) + \sigma_n^2(\psi)} G  + \mu(\psi) | \\
\end{array}
  \]    

\end{proof}

\begin{proof}[Proof of theorem~\ref{thm:Bound_FullNoise}]  
Additionally, for $\tilde a^{-1} $, define its norm as
\[
M_q: = \|\tilde a^{-1} \|_{q} = \| a^{-1} \|_{q}
\]

 When $\psi = e_0$, then
 \[
 \begin{array}{ll}
 \mu(e_0) &= \eta_1 + \eta_2 M_{q_2}\\
  \sigma_n(e_0) &=\sqrt{ \sigma_1^2 + \sigma_2^2 M_{2}^2} \\
 \sigma_s((e_0)_I) &= I_0 
 \end{array}
  \]

  
We have 
\[
 \begin{array}{ll}
 F_{\eta_1,\eta_2, \sigma_1,\sigma_2}(e_0)
 &= 
  {\mathbb E}_{G} {\mathbb E}_{I_0} |  \sigma(e_0, (e_0)_{I_0})  G  +  \mu(e_0)| \\
  &= 
 (1-p){\mathbb E}_{G} |  \sigma_n(e_0) G  +  \mu(e_0)|+ p{\mathbb E}_{G} | \sqrt{1+ \sigma^2_n(e_0)}  G  +  \mu(e_0)|\\
   &=
 \sqrt{\frac{2}{\pi}}[ (1-p) \sigma_n(e_0)^2 \mathcal{R}^f(\frac{\mu(e_0)}{\sigma_n(e_0)})+  
 p\sqrt{1+ \sigma^2_n(e_0)} \mathcal{R}^f(\frac{\mu(e_0)}{\sqrt{1+ \sigma^2_n(e_0)}}) ]
\end{array}
  \]
  Using the chain of inequality:
    \[
 \begin{array}{ll}
F_{\eta_1,\eta_2, \sigma_1,\sigma_2}((e_0)
 \geq F_{\eta_1,\eta_2, \sigma_1,\sigma_2}((\psi^\star ) 
 \geq F_{0,0,0,0}(\psi^\star )
 \geq F_{0,0,0,0}(e_0) 
\end{array}
  \] 
and the fact that
  \[
 \begin{array}{ll}
 F_{0,0,0,0}(\psi^\star )
 &= {\mathbb E}_I\|\psi_I\|_2 \\
 &= {\mathbb E}_I\|(e_0)_I\|_2 + ({\mathbb E}_I\|\psi_I\|_2-{\mathbb E}_I\|(e_0)_I\|_2) \\
 &= \sqrt{\frac{2}{\pi}}[p+ B(p,N) \|\psi -e_0\|_2]
\end{array}
  \]   
  From
   \[
 \begin{array}{ll}
 F_{0,0,0,0}(\psi^\star ) &\leq F_{\eta_1,\eta_2, \sigma_1,\sigma_2}(e_0)
\end{array}
  \]   
we have
  \[
 \begin{array}{ll}
p+ B(p,N) \|\psi^\star -e_0\|_2\leq (1-p) \sigma_n(e_0)^2 \mathcal{R}^f(\frac{\mu(e_0)}{\sigma_n(e_0)})+  
 p\sqrt{1+ \sigma^2_n(e_0)} \mathcal{R}^f(\frac{\mu(e_0)}{\sqrt{1+ \sigma^2_n(e_0)}}) 
\end{array}
  \]   

\end{proof}  







\begin{proof}[Proof of lemma~\ref{ExpansionMeanFoldedGaussian}]
Plug in the formula for mean of folded Gaussian \ref{FoldedGaussianMean}, 
we have
\[
E_G |\mu+ \sigma G| = \sqrt{\frac{2}{\pi}} \sigma \exp\{-\gamma^2/2\} + \mu \left(1 - 2\Phi\left(-\gamma\right) \right)
\]
Divide it by 
\[
E_G |\sigma G| =   \sqrt{\frac{2}{\pi}} \sigma
\]
we get the first equality.

Since the CDF of the standard normal distribution can be expanded by integration by parts into a series:
\[
\Phi( \gamma)=\frac{1}{2} + \frac{1}{\sqrt{2\pi}}\cdot e^{- \gamma^2/2} \left[ \gamma + \frac{ \gamma^3}{3} + \frac{ \gamma^5}{3\cdot 5} + \cdots + \frac{ \gamma^{2n+1}}{(2n+1)!!} + \cdots\right]
\]
where $!!$ denotes the double factorial. 

\[
1 - 2\Phi(-\gamma)  = \frac{1}{\sqrt{2\pi}}\cdot e^{- \gamma^2/2}\left[ \gamma + \frac{ \gamma^3}{3} + \frac{ \gamma^5}{3\cdot 5} + \cdots + \frac{ \gamma^{2n+1}}{(2n+1)!!} + \cdots\right]
\]
This gives the second inequality.

Additionally,
 \[
 \begin{array}{ll}
  e^{- \gamma^2/2} &=\sum_{n=0}^{\infty} \frac{ \gamma^{2n}}{(-2)^n (n)!}\\
  &=  1- \frac{ \gamma^2}{2} + \sum_{n=2}^{\infty} \frac{ \gamma^{2n}}{(-2)^n (n)!}
\end{array}
  \] 
then
 \[
 \begin{array}{ll}
& \exp\{-\gamma^2/2\}\{1 + \frac{1}{2}\gamma \left[ \gamma + \frac{ \gamma^3}{3} + \frac{ \gamma^5}{3\cdot 5} + \cdots + \frac{ \gamma^{2n+1}}{(2n+1)!!} + \cdots\right]\}\\
=& (\sum_{n=0}^{\infty} \frac{ \gamma^{2n}}{(-2)^n (n)!})\{1 + \frac{1}{2}\gamma \left[ \sum_{n=0}^{\infty} \frac{ \gamma^{2n+1}}{(2n+1)!!} \right]\}\\

=& 1 + \frac{3}{2}\gamma^2 + \frac{1}{24} \gamma^4   - \frac{ \gamma^6}{120}  + O(\gamma^8) 
\end{array}
  \] 
\end{proof}


\begin{proof}[Proof of lemma~\ref{ExpansionMeanFoldedGaussian_LB}]
Let $C_\Phi( \gamma^2)$ be a function of $ \gamma$, then it is a composition function with the inner funciton being $ \gamma^2,$ defined as follows: 
\[
 \begin{array}{ll}
C_\Phi( \gamma^2) 
&:= (2\Phi( \gamma)-1)/\left(\frac{ \gamma}{\sqrt{2\pi}}\right) \\
&= e^{- \gamma^2/2} \left[1 + \frac{ \gamma^2}{3} + \frac{ \gamma^4}{3\cdot 5} + \cdots + \frac{ \gamma^{2n}}{(2n+1)!!} + \cdots\right]\\
&= [\sum_{n=0}^{\infty} \frac{ \gamma^{2n}}{(2n+1)!!} ]/[\sum_{n=0}^{\infty} \frac{ \gamma^{2n}}{2^n (n)!}] \\
&= [\sum_{n=0}^{\infty} \frac{ \gamma^{2n}}{(2n+1)!!} ]/[\sum_{n=0}^{\infty} \frac{ \gamma^{2n}}{(2n)!!}] \\
&\leq 1
\end{array}
\]
We remark further that
\[
 \begin{array}{ll}
C_\Phi( \gamma^2) 
&= [\sum_{n=0}^{\infty} \frac{ \gamma^{2n}}{(2n+1)!!} ]/[\sum_{n=0}^{\infty} \frac{ \gamma^{2n}}{(2n)!!}] \\
&= [1 + \frac{ \gamma^2}{3}+ +  \frac{ \gamma^4}{15}+ O( \gamma^6) ]/[1 + \frac{ \gamma^2}{2}+ +  \frac{ \gamma^4}{8}+O( \gamma^6)] \\
&= 1 -\frac{ \gamma^2}{6}+  \frac{ \gamma^4}{40}+ O( \gamma^6)
\end{array}
\]
Then
 \[
 \begin{array}{ll}
\mu \left(1 - 2\Phi\left(-\gamma\right) \right)
&= \frac{1}{\sqrt{2\pi}}\cdot \mu^2 \frac{1}{\sigma} C_\Phi(\gamma^2)\\
&= \frac{1}{\sqrt{2\pi}}\cdot \mu^2 \frac{1}{\sigma} (1-\gamma^2/6 +O(\gamma^4))\\
\end{array}
\] 
Plug in all the fact together
\[
 \begin{array}{ll}
E_G |\mu+ \sigma G|
&=   \left[ \sqrt{\frac{2}{\pi}} \sigma \exp\{-\gamma^2/2\} + \frac{1}{\sqrt{2\pi}}\cdot \mu^2 \frac{1}{\sigma} C_\Phi(\gamma^2)\right]\\
&=   \left[ \sqrt{\frac{2}{\pi}} \sigma \{ 1- \frac{\gamma^2}{2} + \sum_{n=2}^{\infty} \frac{\gamma^{2n}}{(-2)^n (n)!}\} + \frac{1}{\sqrt{2\pi}}\cdot \mu^2 \frac{1}{\sigma} C_\Phi(\gamma^2)\right]\\
&= \sqrt{\frac{2}{\pi}}   \left[ \sigma 
\{ 1 + (1+\frac{1}{2}\cdot C_\Phi(\gamma^2))\gamma^2 +   \sum_{n=2}^{\infty} \frac{\gamma^{2n}}{(-2)^n (n)!}\} \}\right]\\
\end{array}
\] 
Define the residual of this expansion as
\[
 \begin{array}{ll}
\mbox{Res}(\gamma^2) 
&:= \frac{ (2\Phi( \gamma)-1)/\left(\frac{ \gamma}{\sqrt{2\pi}}\right)-1}{2}\cdot \gamma^2 +  (\exp\{-\gamma^2/2\} - 1 + \gamma^2/2)
\\
&= \frac{ C_\Phi(\gamma^2)-1}{2}\cdot \gamma^2 +  (\exp\{-\gamma^2/2\} - 1 + \gamma^2/2) \\
&=    \frac{ C_\Phi(\gamma^2)-1}{2}\cdot \gamma^2  +   \{ \sum_{n=2}^{\infty} \frac{\gamma^{2n}}{(-2)^n (n)!}\} \} \\
&=    \frac{-\gamma^4 }{12}  +    \frac{ \gamma^6}{80} +\frac{\gamma^{4}}{(-2)^2 (2)!} +\frac{\gamma^{6}}{(-2)^3 (3)!}+
O(\gamma^8)\\
&=    \frac{\gamma^4 }{24}  - \frac{ \gamma^6}{120}  + O(\gamma^8)
\end{array}
\] 
We know that if $\gamma^2<1,$ 
\[
 \begin{array}{ll}
\mbox{Res}(\gamma^2) 
&=    \frac{\gamma^4 }{24}  - \frac{ \gamma^6}{120}  + O(\gamma^8)\\
&\geq 0
\end{array}
\] 
(We comment that we could push the range for $\mbox{Res}(\gamma^2) 
\geq 0$ further to at least $\gamma^2<5$ if needed.)
Then we give an lower bound:
\[
 \begin{array}{ll}
E_G |\mu+ \sigma G| 
&= \sqrt{\frac{2}{\pi}} \sigma\left[ 1 + \frac{3}{2}\gamma^2+   \mbox{Res}\right]\\
&\geq \sqrt{\frac{2}{\pi}}\left[  \sigma + \frac{3}{2}\mu^2  \sigma^{-1}\right]
\end{array}
\] 


\end{proof}



\subsection{Constrained Convex algorithms }
 \paragraph{Constrained noisy convex problems}
Then we define the constrained convex program that allows noise,
\[
 \begin{array}{ll}
 \mbox{minimize}_{w,z}   & |\alpha| \\
\mbox{subject to}  
& v^T w =1,\\
& {\mathbb E}(\alpha -\tilde{w}^T Y)^2 \leq \epsilon^2
\end{array}
\tag{$P^c_{1,\epsilon}$}
\label{eq:ConstrainProb}
  \]
  If we write $\alpha =\tilde{w}^T Y - \epsilon \zeta $ , then
\[
 \begin{array}{ll}
 \mbox{minimize}_{w,\zeta}   & {\mathbb E}|\tilde{w}^T Y - \epsilon \zeta| \\
\mbox{subject to}  
& v^T w =1,\\
& \zeta^2 \leq 1
\end{array}
\tag{$P^c_{1,\epsilon}$}
  \]
If we set $\epsilon = 0,$ then we recover 
the original convex problems for noiseless setting
\[
 \begin{array}{ll}
 \mbox{minimize}_{w}   & {\mathbb E}|\tilde{w}^T Y|\\
\mbox{subject to}  
& v^T w =1,
\end{array}
\tag{$P_1$}
  \]
\section{Numerical experiments}
We set up a ground truth inverse filter as $a^{-1} = (0.5, 1, 0.5)$, and numerically run $10$ independent experiments of finding the inverse filter by the convex problem 
\begin{equation}
 \begin{array}{ll}
  \underset{w\in V}{ \mbox{minimize} }  &  \frac{1}{2T+1}\|w*Y\|_{1,[-T,T]}\\
 \mbox{subject to}  
&  w_0 =1.
\end{array}
\tag{$P_{e_0}^T$ }
\end{equation}
We choose an accuracy $\epsilon = 10^{-3}$ and count the number of successfully accurate recovery of $a^{-1}$ where $\|w^\star -a^{-1}\|_1/\|a^{-1}\|_\infty<\epsilon$. We see a consistent sharp transition of success rate from $0$ to $1$ in a narrow parameter region.
\begin{figure} 
\centering
\includegraphics[width=.7\textwidth]{./figures_BD/phase_transition_detail.png}
\caption{The finite sample phase transition phenomenon, where $a^{-1} = (0.5, 1, 0.5)$,  $x-$axis is the sparsity level $p$, $y-$axis is the empirical probability $P(\|w^\star -a^{-1}\|_1/\|a^{-1}\|_\infty<10^{-3})$. Red part is failure, Blue part is success.}
\label{fig-PT}
\end{figure}
\subsection{Population Phase transition theorem}
Qingyun: Old proof without subgradient language:

\paragraph{Population phase transition theorem}
\begin{theorem}
Under the previous assumption,
$
         p^\star   =\frac{\sin{\theta}}{\sin{\theta}+ \cos{\theta}} .
$
 The solution of  $P_{1,v}$ is $w^\star$, then 
 $w^\star$ is $a^{-1}$ up to shift and scaling provided $p<p^\star$,
and  $w^\star$ is not $a^{-1}$ up to shift and scaling provided $p> p^\star$.
  \end{theorem}
The population phase transition diagram is in figure~\ref{fig-PT}.

\begin{figure} 
\centering
\includegraphics[width=.7\textwidth]{./figures_BD/Phase_T_sine.png}
\caption{The population phase transition diagram, $x-$axis is $\sin{\theta}$, $y-$axis is the sparsity level $p$ . Red part is failure, Blue part is success.}
\label{fig-PT}
\end{figure}






\paragraph{Proof of the population phase transition theorem}
\begin{itemize}
 \item
  Let $\phi  := \psi -e_0 = \tilde w*\tilde a -e_0,$  then $|[w*Y]_0| = |[(e_0 +\tilde \phi)*X]_0| =  |[(e_0 + \phi)^T X| $, we define
  \[
  L(\phi) = |[(e_0 + \phi)^T X|.
  \] 
  WLOG, we can assume the ground truth $a^{-1}$ is scaled so that $v^T \tilde{a}^{-1} =1$

  We want to find a threshold $p^\star$ so that for all $p<p^\star$, the directional derivative  $ {\mathbb E} \nabla_{\phi}{L}|_{\phi = 0}>0$ for all directions $\phi$ that satisfy $u^T \phi = 0$.

 We define the finite difference
  \[
  D = L(\phi)-L(0) =  |[(e_0 + \phi)^T X| -|X_0| 
  \]
  Let $\epsilon : = \|\phi\|_2 $ and taking limit of $D$ as $\epsilon  \rightarrow 0,$ we have
    \begin{equation}
  \begin{array}{ll}
   {\mathbb E} D &=  p{\mathbb E}[D\mid X_0 \neq 0 ] + 
   (1-p){\mathbb E}[D \mid X_0 = 0 ] \\
   &=  p{\mathbb E}[\mbox{sign}(X_0) ( \phi_0 X_0 + \sum_{ j\neq 0}\phi_{j} X_{j} )\mid X_0 \neq 0 ] + (1-p){\mathbb E}\mid \sum_{ j\neq 0}\phi_{j} X_{j}\mid   \\
   &= p \phi_0 {\mathbb E} [|X_0|\mid X_0 \neq 0 ] + (1-p){\mathbb E}\mid \sum_{ j\neq 0}\phi_{j} X_{j}\mid  
   \end{array}
 \end{equation}
The last equality is because 
 $
 {\mathbb E}[\mbox{sign}(X_0) (\sum_{ j\neq 0}\phi_{j} X_{j} )\mid X_0 \neq 0 ] = 0 
$
 as $X_j$ are sampled independently. Notice that for Bernoulli Gaussian $X$,
$
{\mathbb E} [|X_0|\mid X_0 \neq 0 ]=\sqrt{\frac{2}{\pi}}.
$
 
 Then  $ {\mathbb E} \nabla_{\phi}{L}|_{\phi = 0} = {\mathbb E} \lim_{\epsilon \rightarrow 0}\frac{D}{\epsilon}>0$ 
 in case either $\phi_{0} > 0$, or in case $\phi_0 < 0$ and
\[
          \frac{p}{1-p} < \frac{ {\mathbb E}\mid \sum_{ j\neq 0}\phi_{j} X_{j}\mid  }{|\phi_0| 
          {\mathbb E} [|X_0|\mid X_0 \neq 0 ]} = \frac{\|\phi_{(0)}\|_2}{| \phi_{0}|} \frac{ {\mathbb E} | \phi_{(0)}^T X_{(0)} |}{\|\phi_{(0)}\|_2 \sqrt{\frac{2}{\pi}}}  .
\]
Here  $X_{(0)} = (X_i)_{i\neq 0}$
\item (Geometric lower bound:)
Using the fact that $u^T \phi = 0$. We define $v = \cos{(\theta)}\frac{\tilde{a}}{\|\tilde{a}\|_2} + \sin{(\theta)} (\frac{\tilde{a}}{\|\tilde{a}\|_2} )^{\bot}$, then $u =\tilde{a}^{-1}*v = \tilde{a}^{-1}*(\cos{(\theta)}\frac{\tilde{a}}{\|\tilde{a}\|_2} + \sin{(\theta)} (\frac{\tilde{a}}{\|\tilde{a}\|_2})^{\bot}) =\cos{(\theta)} \frac{e_0}{\|\tilde{a}\|_2}+\sin{(\theta)} \tilde{a}^{-1}*(\frac{\tilde{a}}{\|\tilde{a}\|_2} )^{\bot} $, 
where $\tilde{a}^{-1}*(\frac{\tilde{a}}{\|\tilde{a}\|_2} )^{\bot}$ is perpendicular to $e_0$, 
therefore, 
$0= u^T \phi = \frac{1}{\|\tilde{a}\|_2}[\cos{(\theta)} \phi_0+\sin{(\theta)} \phi_{(0)}]$, therefore,  
$\frac{\|\phi_{(0)}\|_2}{| \phi_{0}|} = \frac{\cos(\theta)}{\sin(\theta)} =\cot(\theta).$
\[
 \frac{\|\phi_{(0)}\|_2}{| \phi_{0}|}\geq  \cot(\theta)
 \]

 
\item
We showed that for IID Bernoulli Gaussian $X$,
\[
\frac{ {\mathbb E} | \phi_{(0)}^T X_{(0)} |}{\sqrt{\frac{2}{\pi}} \|\phi_{(0)}\|_2} = \frac{ {\mathbb E}_{I_{(0)}}\| \phi_{I_{(0)}}\|_2}{ \|\phi_{(0)}\|_2} \in [p,1)
\]
Using semicontinuous extension, we can choose sequence of $\phi$ that is dense on doubly infinite sequence space $V_{-\infty, \infty}$, then we know that $\frac{ {\mathbb E}_{I_{(0)}}\| \phi_{I_{(0)}}\|_2}{ \|\phi_{(0)}\|_2}$ should be replaced by its semicontinuous extension on $V_{-\infty, \infty}$, which is $1$.

The phase transition $p^\star$ is the smallest $p$ so that 
\[
          \frac{p}{1-p} \leq  \cot(\theta) \frac{ {\mathbb E}_{I_{(0)}}\| \phi_{I_{(0)}}\|_2}{ \|\phi_{(0)}\|_2} \leq  \cot(\theta).
\]
Therefore,  generically, $ \frac{p^\star}{1-p^\star}  = \cot(\theta) $.
\EIT
\section{Projection pursuit for planted sparsity}



 
 
 
 
 









 


\section{Phase transition for sub-Gaussian}
From the Khinthine’s inequality for sub-Gaussian and $p = 1$, we have 
\[
E{|h_{(0)}^T X_{(0)}| }\geq  3^{-3/2}C^{-3} K^{-3}  \|h_{(0)}\|_2,
\]
for Bernoulli-SubGaussian, we have,
\[
E{|h_{(0)}^T X_{(0)}| }\geq  (1-(1-p)^M)^{1/2} 3^{-3/2}C^{-3} K^{-3}  \|h_{(0)}\|_2.
\]
In general, we define
\[
\gamma(F,p) = E{|h_{(0)}^T X_{(0)}| }/\|h_{(0)}\|_2 E |X_0|,
\]
as the ratio between the linear combination $E{|h_{(0)}^T X_{(0)}| }$ for $\|h_{(0)}\|_2=1$ and $ E |X_0|$.

Then we need
\[
          \frac{1-p}{p} < \frac{ \gamma(F,p)  \|h_{(0)}\|_2}{| h_{0}|}
\]
for all $h$ such that $   u^T h = 0$ .

Also, using the fact that
 
\[
         \frac{  \|h_{(0)}\|_2} {| h_{0}|} \geq \cot(\theta).
\]
we have 
\[
          \frac{1-p}{p} <\gamma(F,p) \cot(\theta).
\]

\section{Noisy  signal}
 \subsection{Folder Gaussian}
 For a Gaussian RV $Z \sim N(\mu, \sigma)$, then for RV $|Z|$,  
 \[
 \mu_{|Z|} = E|Z|=\sqrt{\frac{2}{\pi}}\sigma e^{-\frac{\mu^2}{2\sigma^2}}+\mu\left[1-2\Phi\left(-\frac{\mu}{\sigma}\right) \right] 
 \]
  \[
 \sigma_{|Z|}^2 = \mu^2 + \sigma^2 - \mu_{|Z|}^2.,
 \]
 
 \subsection{Noisy perturbation analysis}
Consider a noisy perturbation $\bar X_i  =  X_i + \sigma  E_i $, where $E_i$ is standard Gaussian $N(0,1)$, $ X_i = B_i Z_i$ are iid product of Bernolli RV with parameter $p$ and subGaussian unit variance RV with c.d.f $F$. 

Then we let $b^{\sigma}$ be the solution of the optimization problem 
\[
E D_\sigma = E_X E_E( |(1+ {\varepsilon} h_0)(X_0+\sigma  E_0) +  {\varepsilon} h_{(0)}^T (X_{(0)}+\sigma  E_{(0)})| -|X_0+\sigma  E_0|)
\]
for sufficiently small ${\varepsilon}$, and fixed $\sigma<1$, we consider the asymptotic analysis when ${\varepsilon}  << 1 $， $ \sigma<< 1 $. We know $\sigma ((1+ {\varepsilon} h_0)(  E_0)+{\varepsilon} h_{(0)}^T  E_{(0)}))$ is a centered Gaussian with standard deviation 
\[\sigma\sqrt{(1+ {\varepsilon} h_0)^2+({\varepsilon} h_{(0)})^2}\approx \sigma(1+ {\varepsilon} h_0)
\]
therefore,
\begin{eqnarray*}
 E D_\sigma &\approx&   E|(1+ {\varepsilon} h_0)X_0 +  {\varepsilon} h_{(0)}^T X_{(0)}+\sigma (1+ {\varepsilon} h_0) E_0| -E|X_0+\sigma  E_0| \\
 &\approx& 
  p \cdot(|1+{\varepsilon} \cdot  h_{0}|-1) \cdot E |Z_0+\sigma  E_0| + (1-p) \cdot ( E{|{\varepsilon}  \cdot h_{(0)}^T X_{(0)}+\sigma  E_0| -E|\sigma  E_0|} ),
\end{eqnarray*}
 where $p = P(X_0\neq 0)$.
 
 First we derive lower bounds,
 \[
 E |Z_0+\sigma  E_0|
 \geq E |Z_0| - E|\sigma  E_0| =   E |Z_0| - \sqrt{\frac{2}{\pi}}\sigma,
\]
 \begin{eqnarray*}
 E{|{\varepsilon}  \cdot h_{(0)}^T X_{(0)}+\sigma  E_0| -E|\sigma  E_0|}&\geq& E{|{\varepsilon}  \cdot h_{(0)}^T X_{(0)} | -2 E|\sigma  E_0|} \\
 &=&  E|{\varepsilon}  \cdot h_{(0)}^T X_{(0)} | -2 \sqrt{\frac{2}{\pi}}\sigma
\end{eqnarray*}
Then we have 
\begin{eqnarray*}
 E D_\sigma 
\geq  E D_0 - (p+2(1-p)) \cdot \sqrt{\frac{2}{\pi}}\sigma.
\end{eqnarray*}

Therefore, in the existence of noise, we don't have exact recovery result, but we would have a lower bound. 
 
 Second, we use the mean of folded Gaussian to compute the condition expectation.
 
 Here, for a fixed $\mu$, we rescale the variance $\sigma' = \sigma/{\varepsilon}$. Under the assumption ${\varepsilon}/\sigma<< 1 $, using asymptotic expansion, we can compute the condition expectation 
 \begin{eqnarray*}
   E({|\mu+\sigma'  E_0| -|\sigma'  E_0|}\mid \mu)
 &=& \sqrt{\frac{2}{\pi}}\sigma' e^{-\frac{\mu^2}{2\sigma'^2}}+\mu\left[1-2\Phi\left(-\frac{\mu}{\sigma'}\right) \right] - \sqrt{\frac{2}{\pi}}\sigma'
 \\
 &\approx& -\sqrt{\frac{2}{\pi}}\sigma' {\frac{\mu^2}{2\sigma'^2}}+\mu\sqrt{\frac{2}{\pi}}(-\frac{\mu}{\sigma'}  )
 \\
  &=&- \sqrt{\frac{2}{\pi}}  {\frac{3\mu^2}{2\sigma'}}  
\end{eqnarray*}
The first equality is the formula for the mean of folded Gaussian.
The second equality uses the Taylor expansion of Gaussian c.d.f
\[
\Phi (z)  = \frac{1}{2}+  \frac{1}{\sqrt{\pi}} \left(z-\frac{z^3}{3}+\frac{z^5}{10}-\frac{z^7}{42}+\frac{z^9}{216}-\cdots\right)
\]

 Therefore, for $Z_0$ standard Gaussian,
  \begin{eqnarray*}
 E_X E_{E_0}(|Z_0+\sigma  E_0|) 
  &=&  \sqrt{\frac{2}{\pi}}   \sqrt{1+\sigma^2}  
\end{eqnarray*}
 \begin{eqnarray*}
 E_X E_{E_0}({|{\varepsilon}  \cdot h_{(0)}^T X_{(0)} +\sigma  E_0| -|\sigma  E_0|}) 
  &=&-{\varepsilon} \cdot 3\sqrt{\frac{2}{\pi}}  {\frac{E_X( h_{(0)}^T X_{(0)})^2 }{2\sigma'}}  
   \\
  &=&-{\varepsilon} \cdot 3\sqrt{\frac{2}{\pi}}  {\frac{\|h_{(0)}\|^2 }{2\sigma'}}  
\end{eqnarray*}

Then 
\begin{eqnarray*}
 E D_\sigma 
 &\approx& 
 {\varepsilon} \left[ p \cdot h_{0} \cdot \sqrt{\frac{2}{\pi}}   \sqrt{1+\sigma^2}    - (1-p) \cdot 3\sqrt{\frac{2}{\pi}}  {\frac{\|h_{(0)}\|^2 }{2(\sigma/{\varepsilon})}} \right] .
\end{eqnarray*}



 
\section{Population loss and phase transition}
We first study population loss of the sparse deconvolution problem
 \begin{eqnarray*}
    (\mbox{Population sparse deconvolution}) \qquad  \min_{w\in {\mathbb R}^k} &    
    E\| w * Y\|_1\\
    \mbox{ subject to }&  v^T w = 1,\\
    &  w\in {\cal M}.
\end{eqnarray*}
\subsection{Population loss of projection pursuit}
Let's first consider the population objective $E|w^T \Phi X|$.

Consider a vector  $w \in {\cal M}$ which is a small perturbation of $w^\star  \in {\cal M}$, where
$w^T \Phi =e_0$. Let $w = w^\star + {\varepsilon} \delta$ where $\delta \in T_{w^\star} {\cal M}$, the tangent space at $w^\star$. And since $\delta \in T_{w^\star} {\cal M}_1$ is a subspace, we assume the constraint is given by $M \delta = 0$.
Let $h = \Phi^T \delta$, then $   u^T \delta = 0$, where $ v = \Phi u$. We have
\[
    b^T Y = X_0 + {\varepsilon} \delta^T \Phi X=X_0 + {\varepsilon} h^T  X = (1+ {\varepsilon} h_0)X_0 +  + {\varepsilon} h_{(0)}^T X_{(0)} .
\]
 where $h_{(0)}$, $X_{(0)}$ are $h$, $X$ without $0$-th coordinate.
 \subsection{Population phase transition}
 We want to study $E D =  E|(1+ {\varepsilon} h_0)X_0 +  + {\varepsilon} h_{(0)}^T X_{(0)}| -E|X_0|$.
 
 Then for sufficiently small ${\varepsilon}$
\begin{eqnarray*}
 E|(1+ {\varepsilon} h_0)X_0 +  {\varepsilon} h_{(0)}^T X_{(0)}| -E|X_0| &\approx& 
 (|1+{\varepsilon} \cdot  h_{0}|-1) \cdot E |X_0| \cdot (1-\eta)+ {\varepsilon} \cdot  E{|h_{(0)}^T X_{(0)}| } \cdot \eta,
\end{eqnarray*}
 where $p = P(X_0=0)$.
 
 The perturbation increases the objective value 
just in case either $h_{0} > 0$, or in case $h_{0} < 0$ and
\[
          \frac{\eta}{1-\eta} > \frac{| h_{0}| E |X_0|}{ E | h_{(0)}^T X_{(0)} |. }
\]

Now we consider the optimization problem
\begin{eqnarray*}
    (Q_1) \qquad  \min_h&   E | h_{(0)}^T X_{(0)} |\\
    \mbox{ subject to }&  h_{0} = -1, \\
    & h\in \Phi^T T_\beta {\cal M}_1.
\end{eqnarray*}


We know $h = \Phi^T \delta$, then 
\begin{eqnarray*}
    (Q_1) \qquad  \min_\delta &E | \delta^T \Phi_{(0)} X_{(0)} | \\
    \mbox{ subject to }  &\Phi_0^T \delta = -1, \\
     & \delta \in T_\beta {\cal M}_1.
\end{eqnarray*}



Following the quadratic problem reduction, we can related $Q_1$ to the deterministic quadratic problem.

After the study of the general population model, we remark that for convolutional model, $A$ ad $B$ are both generated by the circular or Topelitz matrices from a single vector.





\subsection{Regularized convex problems}
\paragraph{Regularized convex problems}
For any $\epsilon>0$, we can find the Lagrangian form  of the constrained problem to define the regularized convex program, where we can find a corresponding $\lambda = \Lambda(\epsilon)$ so that $\frac{1}{2\lambda}$ is the dual variable.
\[
 \begin{array}{ll}
 \mbox{minimize}_{w,\zeta}   & |\tilde{w}^T Y -\zeta |+ \frac{1}{2\lambda}((\zeta)^2-\epsilon^2) \\
\mbox{subject to}  
& v^T w =1,
\end{array}
\tag{$P_{1,\lambda}$}
  \]
The convention to use for optimization is to ignorne the constant $-\epsilon^2$, and write the problem as
\[
 \begin{array}{ll}
 \mbox{minimize}_{w,\alpha}   & [|\alpha|+ \frac{1}{2\lambda}(\alpha -\tilde{w}^T Y)^2] \\
\mbox{subject to}  
& v^T w =1,
\end{array}
\tag{$P_{1,\lambda}$}
  \]
  Define soft-threshold operator as 
  \[
  S_{\mu}(x): = {\rm sign} (x) (|x| -\mu)_+
  \]
  and define its complement
   \[
  \bar{S}_{\mu}(x): = x-S_{\mu}(x) =  {\rm sign} (x)(x- (|x| -\mu)_+) =  {\rm sign} (x)\max(|x| ,\mu).
  \]
 \begin{lemma}
 If we partially minimize over $\alpha,$ the minimizer is $\alpha = S_{\lambda}(\tilde{w}^T Y)$, 
    \[
 \begin{array}{ll}
\mbox{inf}_{\alpha}   & [|\alpha|+ \frac{1}{2\lambda}(\alpha -\tilde{w}^T Y)^2]   = [(|\tilde{w}^T Y|-\lambda)_+ + \frac{1}{2\lambda}(\max(|\tilde{w}^T Y| ,\lambda) )^2] 

\end{array}
  \]
  \end{lemma} 
  
  
 
\begin{proof}
Due to the proximal operator of $\ell_1$ norm,  the minimizer is $\alpha = S_{\lambda}(\tilde{w}^T Y)$, 
The objective can be further simplified to
  \[
 \begin{array}{ll}
&{\mathbb E}[|\alpha|+ \frac{1}{2\lambda}(\alpha -\tilde{w}^T Y)^2]\\
=&{\mathbb E}[|S_{\lambda}(\tilde{w}^T Y)|+ \frac{1}{2\lambda}(\bar{S}_{\lambda}(\tilde{w}^T Y) )^2]\\
=& [(|\tilde{w}^T Y|-\lambda)_+ + \frac{1}{2\lambda}(\max(|\tilde{w}^T Y| ,\lambda) )^2] 
\end{array}
  \]
 \end{proof} 

Using the data generative model and do change of variable, 
the partially minimized regularized problem is
 \[
 \begin{array}{ll}
 \mbox{minimize}_{\psi}   & {\mathbb E}[|(|\psi^T (X+\eta Z )|-\lambda)_+|+ \frac{1}{2\lambda}(\max(|\psi^T (X+\eta Z )| ,\lambda) )^2] \\
\mbox{subject to}  
& u^T \psi =1,
\end{array}
\tag{$P_{1,\lambda}$}
  \]

\section{Related Work}
\paragraph{Prior work: analysis approach, non-Gaussianity}
  \begin{itemize}
  \item
  Key idea $1$: Find inverse filter $a^{-1}$ so that
$ a^{-1}*y = x $.
\item Key idea $2$: 
If $x$ is non-Gaussian, then $a*x$ is closer to Gaussian. 
\item Find $w\neq 0$  
\[
 \begin{array}{ll}
 \underset{w}{\mbox{maximize}}   & J(w*y)
\end{array}
  \]
\item Non-Gaussianity: $J$ is negative entropy function or normalized $4$-th order cumulant $J(z) = \frac{\|z\|_4^4}{\|z\|_2^4}$ \cite{wiggins1978minimum, donoho1981minimum}
  \item Sparsity: 
  $J(z) = - \| z\|_1/\|z\|_2$,  \cite{gray1979variable,donoho1981minimum,kijko1994seismological}.
  \item varimax norm: $\max_i \frac{|z_i|}{\|z\|_2}$ \cite{cabrelli1985minimum}
\end{itemize}

\paragraph{Prior work: blind equalization for digital signal}
  \begin{itemize}
\item
Digital signal: PAM signaling $\pm 1$ binary signal or QAM signaling with $|x|=1, x\in {\mathbb C}$ \cite{kennedy1992blind,ding2000fast}. 
\item More general,  $x$ are IID and bounded by $M = \mbox{max}(x_i)$ \cite{vembu1994convex,li2001blind}.
\item 
  Convex Cost Functions in Blind Equalization \cite{vembu1994convex}:
  find $w$ so that $w*a = e_0 = (\ldots,0,1,0, \ldots)$ as minimizer of the ideal convex problem (
 Not computable without knowing $a$!)
  \[
 \begin{array}{ll}
 \underset{w}{\mbox{minimize}}   & \| w*a\|_1 
 \\
\mbox{subject to} 
& (w*a)_0 = 1.
\end{array}
  \]  
  
    \item
  When $x$ are IID and bounded by $M = \mbox{max}(x_i)$, using duality, use surrogate objective for $\| w*a\|_1$ as
   \[
 \begin{array}{ll}
 \underset{w}{\mbox{minimize}}   &  \frac{1}{M} E \|w * y\|_\infty 
 \\
\mbox{subject to} 
& \|w*y\|_2 = 1.
\end{array}
  \]
   In \cite{vembu1994convex}, they use projected SGD on $E \|w * y\|_8 $ to approximate $E \|w * y\|_\infty$ .
  
  \item Later, use linear programming to solve $\infty$ norm problem directly \cite{ding2000fast}: 
    \[
 \begin{array}{ll}
 \underset{w}{\mbox{minimize}}   &  E \|w * y\|_\infty
 \\
\mbox{subject to} 
& w_0 = 1.
\end{array}
  \]
\end{itemize}

\paragraph{Prior work: synthesis approach, sparsity}
In many papers \cite{kuo2019geometry, bristow2013fast, heide2015fast,zhang2017global,zhang2018structured}, researchers designed alternating-minimization algorithms that jointly find a short $a$ on sphere and a sparse $X$ from $Y$ by solving the following non-convex problem: 
  \[
 \begin{array}{ll}
 \underset{a, x}{\mbox{minimize}}   & \|Y-a*x\|_2^2+\lambda \|x\|_1 \\
\mbox{subject to}  & \|a\|_2 = 1.
\end{array}
  \]  
\begin{itemize}
\item
  Convolutional dictionary learning formulation \cite{bristow2013fast, heide2015fast}, etc.
  \item
  Synthesis blind deconvolution with $2$ norm ball constraint  \cite{zhang2017global,zhang2018structured, kuo2019geometry}, etc.
  \item
  Short $a$ and sparse $x$: 
  \[
 \begin{array}{ll}
 \mbox{minimize}   & \|y-a*x\|_2^2+\lambda \|x\|_1 \\
\mbox{subject to}  & \|a\|_2 = 1.
\end{array}
  \]  
\end{itemize}

\paragraph{Prior work: synthesis approach, subspace assumption}
\[
 \begin{array}{ll}
 y &= a^0 * x^0\\
 \hat y  &= \hat a^0 \circ \hat x^0
\end{array}
  \]
   \begin{itemize}
  \item  $\hat a^0= D_a c_a$.
  \item $\hat x^0 = D_X c_x$.
  \item $D_X$ is often a complex Gaussian matrix.  
\end{itemize}
No translation symmetry in this assumption!


Lifting:
 \[
 \begin{array}{ll}
 \hat y &= (D_a c_a) \circ (D_X c_x)\\
 &= \mathcal{D}(c_a c_x^T)
\end{array}
  \]
 $\mathcal{D}$ is a linear operator defined as 
 \[
 \begin{array}{ll}
\mathcal{D}(c_a c_x^T)_i = \mbox{trace}((D_{a,i}D_{X,i}^T)(c_a c_x^T))
\end{array}
  \]
  Rank one matrix $Z=c_a c_x^T$.
   \[
      \hat y =  \mathcal{D}(Z)
  \]
  \begin{itemize}  
  \item Convex optimization  \cite{ahmed2014blind},\cite{ling2015self}, \cite{chi2016guaranteed},\cite{romberg2013multichannel},\cite{lee2017blind} etc.
  \item Non-convex optimization  \cite{li2016rapid}.
\end{itemize}

\paragraph{Projection pursuit  with linear constraint}
We point out that the population problem formulation for sparse blind deconvolution is also the population problem formulation for sparse projection pursuit, which might also be called convex sparse ICA, or convex sparse dictionary learning.
\begin{equation}
 \begin{array}{ll}
  \underset{\psi \in V}{ \mbox{minimize} }  &  {\mathbb E} |\psi^T X|\\
 \mbox{subject to}  
&  u^T \psi =1.
\end{array}
\tag{$P_{1,v}$}
\end{equation}


In sparse projection pursuit with linear constraint, we observe a matrix $Y_M\in {\mathbb R}^{n\times M }$, and we know that there exists a ground-truth filter $w^0$, such that $w^{0,T} Y_M = X_M$, $X_M$ satisfy Bernoulli$(p)$-Gaussian model.  

We consider the "projection pursuit" problem.
 \begin{eqnarray*}
    (\mbox{Projection pursuit}) \qquad  \min&    \frac{1}{M} \| \tilde{w}^T Y_M\|_1\\
    \mbox{ subject to }&  v^T w = 1.
\end{eqnarray*}
Here $w$ is in a $k$ dimensional subspace of the double infinite sequence. 


\paragraph{Projection pursuit with sphere constraint}
We comment that the results we derive in the following sections are also applicable to the projection pursuit (orthogonal dictionary learning) problem \cite{bai2018subgradient, sun2016complete, sun2015complete}.
 \begin{eqnarray*}
    (\mbox{Projection pursuit with sphere constraint}) \qquad  \min&    \frac{1}{M} \| \tilde{w}^T Y_M\|_1\\
    \mbox{ subject to }&   \frac{1}{M} \| \tilde{w}^T Y_M\|_2 = 1.
\end{eqnarray*}
or
 \begin{eqnarray*}
    (\mbox{Projection pursuit with sphere constraint}) \qquad  \min&    \frac{1}{M} \| \tilde{w}^T Y_M\|_1\\
    \mbox{ subject to }&   \frac{1}{M} \| \tilde{w}^T Y_M\|_2 \geq 1.
\end{eqnarray*}
\begin{lemma}
The population problem is:
\[
 \begin{array}{ll}
 \mbox{minimize}_{\psi} \quad & {\mathbb E} |\psi^T X|  \\
\mbox{subject to}  
&{\mathbb E} |\psi^T X|^2 \geq {\mathbb E} (X_0)^2
\end{array}
  \]
is equivalent to the following convex-concave problem
\[
 \begin{array}{ll}
 \mbox{minimize}_{\psi} \quad &\sqrt{\frac{2}{\pi}}\left[{\mathbb E}_I \|\psi_I\|_2 \right] \\
\mbox{subject to}  
&\|\psi\|_2^2 \geq 1
\end{array}
  \]
We claim that  it has the same minimizer as the \textbf{ projection pursuit (orthogonal dictionary learning) problem with sphere constraint}.
  \[
 \begin{array}{ll}
 \mbox{minimize}_{\psi} \quad & {\mathbb E}_I \|\psi_I\|_2  \\
\mbox{subject to}  
&\|\psi\|_2 = 1
\end{array}
  \]
  which is also equivalent to
   \[
 \begin{array}{ll}
 \mbox{minimize}_{\psi\neq 0} \quad & {\mathbb E}_I \|\psi_I\|_2/\|\psi\|_2 
\end{array}
  \]
  And  also equivalent to
    \[
 \begin{array}{ll}
 \mbox{minimize}_{\psi\neq 0} \quad & {\mathbb E} |\psi^T X|/\sqrt{{\mathbb E} |\psi^T X|^2} 
\end{array}
  \]
\end{lemma}


Inspired by this formulation, we propose a non-convex (convex objective concave constraint) problem for blind deconvultion:
\begin{eqnarray*}
    (\mbox{Projection pursuit with sphere constraint}) \qquad  \min&    \frac{1}{M} \| w*Y_M\|_1\\
    \mbox{ subject to }&   \frac{1}{M} \| w*Y_M\|_2 \geq 1.
\end{eqnarray*}

Another choice is to use infinity norm ball constraint, which allows us to translate it to a convex problem.
\begin{eqnarray*}
    (\mbox{Projection pursuit with sphere constraint}) \qquad  \min&    \frac{1}{M} \| w*Y_M\|_1\\
    \mbox{ subject to }&   \frac{1}{M} \| w*Y_M\|_\infty \geq 1.
\end{eqnarray*}


\paragraph{Infinity norm ball constraint}


Before stating the main theorem and its proof, we make some comments about the formulation.

If we study the population level, 
\begin{equation}
 \begin{array}{ll}
  \underset{w\in V}{ \mbox{minimize} }  &    {\mathbb E} |\tilde{w}^T Y|\\
    \mbox{ subject to }
    & \|w\|_{\infty}= 1.
\end{array}
\tag{$P_{\infty}$ }
\end{equation}
then we only need solve one convex problems
\begin{equation}
 \begin{array}{ll}
  \underset{w\in V}{ \mbox{minimize} }  &  {\mathbb E} |\tilde{w}^T Y|\\
 \mbox{subject to}  
&  w_0 =1.
\end{array}
\tag{$P_{e_0}$ }
\end{equation}
Since the entry with largest magnitude will automatically be shifted to coordinate $0$ due to shift invariance.


If we want to solve the following non-convex problem with infinity norm ball constraint for finite sample,
\begin{equation}
 \begin{array}{ll}
  \underset{w\in V}{ \mbox{minimize} }  &    \frac{1}{2T+1}\|w*Y\|_{1,[-T,T]}\\
    \mbox{ subject to }
    & \|w\|_{\infty}= 1.
\end{array}
\tag{$P_{\infty}^T$}
\end{equation}
We can solve the following convex problems, and select the one with minimal objective value. This is equivalent to solve the problem with fixed infinity norm. 
\begin{equation}
 \begin{array}{ll}
  \underset{w\in V}{ \mbox{minimize} }  &  \frac{1}{2T+1}\|w*Y\|_{1,[-T,T]}\\
 \mbox{subject to}  
&  w_j =1.
\end{array}
\tag{$P_{e_j}^T$ }
\end{equation}


  
  


  

\section{Main results for finite sample guarantee}
\paragraph{Finite sample setting}
\begin{itemize}
\item
Let $(X_t)_{t \in {\mathbb Z}}$ be IID sampled from  $pN(0,1) + (1-p) \delta_0$.
 \item
The filter $a\in l_1({\mathbb Z}), (\mathcal{F} a)(\omega) \neq 0, \forall \omega\in T.$ Additionally, in finite sample setting, we assume $a^{-1}$ is zero outside of a centered window of radius $k$.
\item
Let $Y= a* X$ be a linear process, we are given a series of observations $(Y_t)_{t\in [-(T+k),T+k]}$ from a centered window of radius $T+k$.
\item
 We denote $X^{(i)}$ as the sequence being flipped and shifted for $i$ step from $X$ so that $X^{(i)}_k = X_{-k +i}$. Let $J_N$ be the support of $X$.
 We define the length $N=2T+1$ window as ${\cal T} := [-T, T].$
\end{itemize}
Now we want to find an inverse filter 
$w = (w_{-k}, \ldots, w_0, \ldots, w_k) \in V$ such that the convolution $w*Y$ is sparse,
\begin{equation}
 \begin{array}{ll}
 \underset{w\in V}{  \mbox{minimize} }   & \frac{1}{N}\|w*Y\|_{1,{\cal T}}\\
 \mbox{subject to}  
& (\alpha* w)_0 =1.
\end{array}
\end{equation}
\paragraph{Finite sample directional derivative}
Now to study the finite sample convex problem, we need to calculate the directional derivative at $ \psi = e_0$.
Then
\begin{theorem}
\label{thm:fin_deriv}
For any $X$ sequence with support  $J_N$,
\begin{eqnarray*}
& & \lim_{t\rightarrow 0^+}\frac{1}{t}[(\hat{{\mathbb E}}  \| (e_0+t\beta)*X \|_1 - \hat{{\mathbb E}}  \| e_0*X \|_1)
\\
  &=& 
 \langle  \beta, {\frac{1}{N}\sum_{|i|\in {\cal T}} \nabla_{\psi}L_i}|_{e_0} \rangle \\
 &=& 
 \frac{1}{N}\sum_{|i|\in {\cal T}}  \mbox{sign}( X_{-i})X^{(i),T}\beta
 \\
  &=& 
 \frac{1}{N} \left(\|X\|_{1,J_N} \beta_0  +
 \|X*\beta_{(0)}\|_{1,{\cal T}-J_N}+ 
\sum_{i\in J_N} (1_{X_{-i}> 0}-1_{X_{-i}< 0})  \cdot X^{(i),T}\beta_{(0)} \right).
 \end{eqnarray*}
 
\end{theorem}











 \paragraph{Concentration of $\|\psi*X\|_{1}$ }
 \begin{lemma}
 \label{lem:mu_min}
 Let 
 \[
\mu_{\mbox{min}}:=\frac{1}{\sqrt{2}\pi}\lambda_{m}(a)\sqrt{\frac{p}{k}} \|w\|_{1},
\]
then it is the lower bound of ${\mathbb E} \frac{1}{N}\|w*a*X\|_{1,{\cal T}}$:
\[
{\mathbb E} \frac{1}{N}\|w*a*X\|_{1,{\cal T}}  \geq \mu_{\mbox{min}}.
\]
 \end{lemma}

Now we define the intersection of $1-$norm ball of the $k-$dimensional subspace $V_k$ as $B_1(V_k)$, then we consider $w\in B_1(V_k).$ 
 
\begin{lemma}
\label{lem:tail_W}
Hence we consider $\psi = a*w \in a*B_1(V_k),$ and we define
\[
W:= \sup_{\psi \in a*B_1(V_k)}
|\frac{1}{N}\sum_{j\in [-T,T] } (|(\psi*X)_{j}| - {\mathbb E} |(\psi*X)_{j}|) |.
\]

For all $q\geq \max(2, \log(k)),$ there exists constant $C$,
\[
{\mathbb E} W^q\leq 2^{2q}(\frac{\lambda_{M}(a)}{N})^q   k C^q(\sqrt{Npq}+q)^q,
\]
so
\begin{eqnarray*}
 \|W\|_q \leq \frac{4eC}{N} \lambda_{M}(a)  
 (\sqrt{Np(q+\log(k))}+q +\log(k)),
\end{eqnarray*}
and
\begin{eqnarray*}
P( W > \frac{4eC\lambda_{M}(a) }{N}  
 (\sqrt{Np\log(\frac{k}{\delta})}+\log(\frac{k}{\delta})))\leq \delta.
\end{eqnarray*}
\end{lemma}

\begin{theorem}
\label{thm:concen_W}
Let the solution of finite sample convex optimization be $w^\star$, 
for small $\epsilon>0$ and concentration level $\delta>0$, 
there exists universal constant $C$, if there are 
\[
N \geq k\log(\frac{k}{\delta}) ( \frac{C\kappa_a}{\epsilon}  )^2,
\]
and  $ \frac{1}{N}\leq p$, we have:
\begin{eqnarray*}
P( W > \epsilon \mu_{\mbox{min}})\leq \delta.
\end{eqnarray*}

  \end{theorem}
 
\paragraph{Uniform bound of directional derivative}
We  define the uniform bound of directional derivative:
\[
W_D  =\sup_{\|\beta\|_2=1, u^T \beta =0, \beta +e_0 \in a*B_1(V_k)}|\langle  \beta, {\frac{1}{N}\sum_{|i|\in {\cal T}} \nabla_{\phi}L_i}|_{\phi = 0} \rangle  -   \langle  \beta, {\mathbb E} \nabla_{\phi}L|_{\phi = 0} \rangle  |.
\]
Notice that
\begin{eqnarray*}
& &\langle  \beta, {\frac{1}{N}\sum_{|i|\in {\cal T}} \nabla_{\phi}L_i}|_{\phi = 0} \rangle  -   \langle  \beta, {\mathbb E} \nabla_{\phi}L|_{\phi = 0} \rangle  
\\
&=& 
\left[\frac{1}{N} \left(\|X\|_{1,J_N} \beta_0  +
 \|X*\beta_{(0)}\|_{1,{\cal T}-J_N}+ 
\sum_{i\in J_N} (1_{X_{-i}> 0}-1_{X_{-i}< 0})  \cdot X^{(i),T}\beta_{(0)}\right)\right]-
\\
& &
\left[\sqrt{\frac{2}{\pi}}(p \cdot \beta_0 + (1-p) \cdot {\mathbb E}_{I}[ \|\beta_{I}\|_2 | 0 \not \in I ])\right].
\end{eqnarray*}

Combining the following three uniform bounds on
\[
|(\frac{1}{N} \|X\|_{1,J_N}-\sqrt{\frac{2}{\pi}}p)|,
\]
\[
|\frac{1}{N}\|X*\beta_{(0)}\|_{1,{\cal T}-J_N}-\sqrt{\frac{2}{\pi}}(1-p) \cdot {\mathbb E}_{I}[ \|\beta_{I}\|_2 | 0 \not \in I ]|,
\]
\[
|\frac{1}{N}\sum_{i\in J_N} (1_{X_{-i}> 0}-1_{X_{-i}< 0})  \cdot X^{(i),T}\beta_{(0)}|,
\]
with high probability, we have  
\[
W_D  < \epsilon \mu_{\mbox{min}}.
\]

We define $\delta_p(N,\epsilon)$ the one side finite $N$ band such that for all $p<p^\star -\delta_p(N,\epsilon)$, 
\[
\langle  \beta, {\mathbb E} \nabla_{\phi}L|_{\phi = 0} \rangle >\epsilon \mu_{\mbox{min}}.
\]
for any direction $\beta \in V,$ the directional derivative at $\phi = 0$
\[
\langle  \beta, {\frac{1}{N}\sum_{|i|\in {\cal T}} \nabla_{\phi}L_i}|_{\phi = 0} \rangle  >0,
\]
then using the convexity argument, $a*w^\star-e_0=0$. 

\paragraph{Finite sample guarantee}
From previous concentration of finite sample directional derivative, we have the following theorem.
	\begin{theorem}
	\label{thm:fin_guarantee}
		When $a^{-1}$ is length $k$, for small constant $\epsilon>0, \delta>0$, when the number of observation $N$ satisfies
		\[
N \geq  k\log(\frac{k}{\delta}) ( \frac{C\kappa_a}{\epsilon}  )^2,
\]
then 
\[
W_D  < \epsilon \mu_{\mbox{min}}.
\]
For $\frac{1}{N}\leq p \leq p^\star - \delta_p(N,\epsilon)$,
\[
\langle  \beta, {\mathbb E} \nabla_{\phi}L|_{\phi = 0} \rangle >\epsilon \mu_{\mbox{min}}.
\]
therefore, 	with probability $1-\delta$, for any direction $\beta \in V,$ the directional derivative at $\phi = 0$
\[
\langle  \beta, {\frac{1}{N}\sum_{|i|\in {\cal T}} \nabla_{\phi}L_i}|_{\phi = 0} \rangle  >0,
\]

		the solution of finite sample convex optimization $w^\star$ is $a^{-1}$ up to scaling and shift 
		with probability $1-\delta$.
	\end{theorem}
	
\section{Proofs for finite sample guarantee}	
	\begin{proof}[Proof of theorem~\ref{thm:fin_deriv}]
 \begin{eqnarray*}
& & \lim_{t\rightarrow 0^+}\frac{1}{t}[(\hat{{\mathbb E}}  \| (e_0+t\beta)*X \|_1 - \hat{{\mathbb E}}  \| e_0*X \|_1)
\\
  &=& 
 \langle  \beta, {\frac{1}{N}\sum_{|i|\in {\cal T}} \nabla_{\psi}L_i}|_{e_0} \rangle \\
 &=& 
 \frac{1}{N}\sum_{|i|\in {\cal T}}  \mbox{sign}( X_0^{(i)})X^{(i),T}\beta
 \\
 &=& 
 \frac{1}{N}\sum_{|i|\in {\cal T}}  \mbox{sign}( X_{-i})X^{(i),T}\beta
 \\
 &=& 
\frac{1}{N}\sum_{|i|\in {\cal T}} 1_{X_{-i}> 0}  \cdot X^{(i),T}\beta  +
\frac{1}{N}\sum_{|i|\in {\cal T}} 1_{X_{-i}< 0}  \cdot (-X^{(i),T}\beta)  +
\frac{1}{N}\sum_{|i|\in {\cal T}} 1_{X_{-i}= 0} \cdot |X^{(i),T}\beta|
 \\
 &=& 
\frac{1}{N}\sum_{|i|\in {\cal T}} 1_{X_{-i}> 0}  \cdot X^{(i),T}\beta  +
\frac{1}{N}\sum_{|i|\in {\cal T}} 1_{X_{-i}< 0}  \cdot (-X^{(i),T}\beta)  +
\frac{1}{N}\sum_{|i|\in {\cal T}}  |X^{(i),T}\beta_{(0)}|
 \\
 &=& 
\frac{1}{N}\sum_{|i|\in {\cal T}} 1_{X_{-i}> 0}  \cdot X_{-i}\beta_0  +
\frac{1}{N}\sum_{|i|\in {\cal T}} 1_{X_{-i}< 0}  \cdot (-X_{-i}\beta_0)  + 
\frac{1}{N}  \|X*\beta_{(0)}\|_{1,{\cal T}-J_N}+ 
\\
& &
\frac{1}{N}\sum_{|i|\in {\cal T}} 1_{X_{-i}> 0}  \cdot X^{(i),T}\beta_{(0)}  +
\frac{1}{N}\sum_{|i|\in {\cal T}} 1_{X_{-i}< 0}  \cdot (-X^{(i),T}\beta_{(0)})  
 \\
  &=& 
\frac{1}{N}\sum_{|i|\in {\cal T}} 1_{X_{-i}\neq 0}  \cdot |X_{-i}|\beta_0  +
\frac{1}{N}  \|X*\beta_{(0)}\|_{1,{\cal T}-J_N}+ 
\frac{1}{N}\sum_{i\in J_N} (1_{X_{-i}> 0}-1_{X_{-i}< 0})  \cdot X^{(i),T}\beta_{(0)}   
 \\
  &=& 
 \frac{1}{N} \left(\|X\|_{1,J_N} \beta_0  +
 \|X*\beta_{(0)}\|_{1,{\cal T}-J_N}+ 
\sum_{i\in J_N} (1_{X_{-i}> 0}-1_{X_{-i}< 0})  \cdot X^{(i),T}\beta_{(0)} \right).
 \end{eqnarray*}
\end{proof}
	
	
\begin{proof}[Proof of lemma~\ref{lem:mu_min}]
Without loss of generality, we rescale $w$ so that $\|w\|_1 =1$, now
\begin{eqnarray*}
{\mathbb E}_X Z 
&=& \frac{1}{N}\sum_{j\in [-T,T] }{\mathbb E}_X |\sum_{i=1}^k w_i (a*X)_{j-i}|\\
&\geq& \inf_{\|w\|_{1} = 1}\frac{1}{N}\sum_{j\in [-T,T] }{\mathbb E}_X |\sum_{i=1}^k w_i (a*X)_{j-i}|.
\end{eqnarray*}
By symmetry, we know the minimizer is $w = (\frac{1}{k}, \ldots,\frac{1}{k}),$
then we use the symmetrization trick to insert a sequence of $k$ IID Rademacher ($\pm 1$)  random variables 
$\varepsilon_1, \ldots, \varepsilon_k$ in the second inequality, and then apply Khintchine inequality as the third inequality,
\begin{eqnarray*}
{\mathbb E}_X Z 
&\geq&  \frac{1}{k} \frac{1}{N}\sum_{j\in [-T,T] }{\mathbb E}_X |\sum_{i=1}^k (a*X)_{j-i}|\\
&\geq& \frac{1}{2} \frac{1}{k} \frac{1}{N}\sum_{j\in [-T,T] }{\mathbb E}_X |\sum_{i=1}^k \varepsilon_i (a*X)_{j-i}|\\
&\geq&  \frac{1}{2} \frac{1}{k} \frac{1}{N}\sum_{j\in [-T,T] }\frac{1}{\sqrt{2}}{\mathbb E}_X \sqrt{ \sum_{i=1}^k  (a*X)^2_{j-i}}\\
&\geq&  \frac{1}{2} \frac{1}{k} \frac{1}{N}\sum_{j\in [-T,T] }\lambda_{m}(a)\frac{1}{\sqrt{2}}{\mathbb E}_X \sqrt{ \sum_{i=1}^k  X^2_{j-i}}\\
&\geq& \frac{1}{2}  \frac{1}{k} \frac{1}{\sqrt{2}}\frac{1}{N}\sum_{j\in [-T,T] }\lambda_{m}(a)\frac{2}{\pi}\sqrt{pk}\\
&=&  \frac{1}{\sqrt{2}\pi}\lambda_{m}(a)\sqrt{\frac{p}{k}}.\\
\end{eqnarray*}
\end{proof}
 
\begin{proof}[Proof of Lemma~\ref{lem:tail_W}]
Let $\varepsilon_1, \ldots, \varepsilon_N$ be a sequence of IID Rademacher ($\pm 1$)  random variables independent of $X$.  By the symmetrization inequality, see e.g. Lemma~$6.3$ of book \cite{ledoux2013probability}, we have the first inequality. Then since the function $t \rightarrow |t| $ is a contraction, an application of Talagrand’s contraction principle (see Lemma~$8$ of \cite{adamczak2016note}) with $F (x) = |x|^q$ 
conditionally on $X$ gives the second inequality.
\begin{eqnarray*}
{\mathbb E} W^q 
&\leq& 2^q {\mathbb E} \sup_{\psi \in a*B_1(V_k)} 
|\frac{1}{N}\sum_{j\in [-T,T] } \varepsilon_j |(\psi*X)_{j}| |^q\\
&\leq&
2^{2q}{\mathbb E} \sup_{\psi \in a*B_1(V_k)} |\frac{1}{N}\sum_{j\in [-T,T] } \varepsilon_j (\psi*X)_{j} |^q\\
&=&
2^{2q}\frac{1}{N^q}{\mathbb E} \sup_{\psi \in a*B_1(V_k)} |\sum_{i\in [-T,T] }\psi_{i}  \sum_{j\in [-T,T] } \varepsilon_j X_{j-i} |^q\\
&=&
2^{2q}\frac{1}{N^q}{\mathbb E} \sup_{w \in B_1(V_k)} |\sum_{i\in [-T,T] }(w*a)_{i}  \sum_{j\in [-T,T] } \varepsilon_j X_{j-i} |^q\\
&\leq&
2^{2q}\frac{1}{N^q}{\mathbb E} \max_{s\in [k]}|\sum_{i\in [-T,T] }a_{i-s}( \sum_{j\in [-T,T] } \varepsilon_j X_{j-i}) |^q\\
&\leq&
2^{2q}(\frac{\lambda_{M}(a)}{N})^q   \sum_{s\in [k] }{\mathbb E}| \sum_{j\in [-T,T] } \varepsilon_j X_{j-s} |^q.
\end{eqnarray*}
Now by the moment version of Bernstein’s inequality (see Lemma~$7$, equation $(18)$ of~\cite{adamczak2016note}), we know that there exists a universal constant $C$,
\begin{eqnarray*}
{\mathbb E}| \sum_{j\in [-T,T] } \varepsilon_j X_{j-i} |^q \leq C^q(\sqrt{Npq}+q)^q.
\end{eqnarray*}
Therefore, 
\begin{eqnarray*}
{\mathbb E} W^q &\leq& 2^{2q}(\frac{\lambda_{M}(a)}{N})^q    
\sum_{s\in [k] }{\mathbb E}| \sum_{j\in [-T,T] } \varepsilon_j X_{j-s} |^q \\
&\leq& 2^{2q}(\frac{\lambda_{M}(a)}{N})^q   k C^q(\sqrt{Npq}+q)^q.
\end{eqnarray*}
Then
\begin{eqnarray*}
 \|W\|_q \leq \frac{4eC}{N} \lambda_{M}(a)  
 (\sqrt{Np(q+\log(k))}+q +\log(k)).
\end{eqnarray*}
Therefore, we could get the tail bound using the Chebyshev inequality for the moments,
\begin{eqnarray*}
P( W > \frac{4eC\lambda_{M}(a) }{N}  
 (\sqrt{Np(q+\log(k))}+q +\log(k)))\leq e^{-q}.
\end{eqnarray*}
We set $q = \log(\frac{1}{\delta})$, obtaining an upper bound for $W$ which is satisfied with probability at least $1-\delta$:
\begin{eqnarray*}
P( W > \frac{4eC\lambda_{M}(a) }{N}  
 (\sqrt{Np\log(\frac{k}{\delta})}+\log(\frac{k}{\delta})))\leq \delta.
\end{eqnarray*}
\end{proof}	
	
	
	
\begin{proof}[Proof of Theorem~\ref{thm:concen_W}]
From previous lemma, we know that with the conditions, 
\begin{eqnarray*}
P( W > \epsilon \mu_{\mbox{min}})\leq \delta.
\end{eqnarray*}
Therefore, with probability at least $1-\delta$,
\begin{eqnarray*}
W = \sup_{\psi \in a*B_1(V_k)}
|\frac{1}{N}\sum_{j\in [-T,T] } (|(\psi*X)_{j}| - {\mathbb E} |(\psi*X)_{j}|) | \leq \epsilon \mu_{\mbox{min}}.
\end{eqnarray*}
When this is true, for all $\psi \in a*B_1(V_k)$, we have a uniform bound:
\[
 {\mathbb E} |(\psi*X)_{j}|-\epsilon\mu_{\mbox{min}} \leq \frac{1}{N}\sum_{j\in [-T,T] } (|(\psi*X)_{j}| \leq {\mathbb E} |(\psi*X)_{j}|+\epsilon\mu_{\mbox{min}}.
\]
\end{proof}
	
	
	
	
\begin{proof}[Proof of Theorem~\ref{thm:fin_guarantee}]
Notice that
\begin{eqnarray*}
& &\langle  \beta, {\frac{1}{N}\sum_{|i|\in {\cal T}} \nabla_{\phi}L_i}|_{\phi = 0} \rangle  -   \langle  \beta, {\mathbb E} \nabla_{\phi}L|_{\phi = 0} \rangle  
\\
&=& 
\left[\frac{1}{N} \left(\|X\|_{1,J_N} \beta_0  +
 \|X*\beta_{(0)}\|_{1,{\cal T}-J_N}+ 
\sum_{i\in J_N} (1_{X_{-i}> 0}-1_{X_{-i}< 0})  \cdot X^{(i),T}\beta_{(0)}\right)\right]-
\\
& &
\left[\sqrt{\frac{2}{\pi}}(p \cdot \beta_0 + (1-p) \cdot {\mathbb E}_{I}[ \|\beta_{I}\|_2 | 0 \not \in I ])\right].
\end{eqnarray*}
From the previous uniform bound, with high probability, we have the following three uniform bounds on
\[
|(\frac{1}{N} \|X\|_{1,J_N}-\sqrt{\frac{2}{\pi}}p)|,
\]
\[
|\frac{1}{N}\|X*\beta_{(0)}\|_{1,{\cal T}-J_N}-\sqrt{\frac{2}{\pi}}(1-p) \cdot {\mathbb E}_{I}[ \|\beta_{I}\|_2 | 0 \not \in I ]|.
\]
The uniform bound on 
\[
|\frac{1}{N}\sum_{i\in J_N} (1_{X_{-i}> 0}-1_{X_{-i}< 0})  \cdot X^{(i),T}\beta_{(0)}|
\]
comes from symmetric distribution assumption.

Combining all three uniform bounds, with high probability, we have:
\[
W_D  < \epsilon \mu_{\mbox{min}}.
\]

We define $\delta_p(N,\epsilon)$ the one-side finite $N$ band such that for all $p<p^\star -\delta_p(N,\epsilon)$, 
\[
\langle  \beta, {\mathbb E} \nabla_{\phi}L|_{\phi = 0} \rangle >\epsilon \mu_{\mbox{min}}.
\]
for any direction $\beta \in V,$ the directional derivative at $\phi = 0$
\[
\langle  \beta, {\frac{1}{N}\sum_{|i|\in {\cal T}} \nabla_{\phi}L_i}|_{\phi = 0} \rangle  >0,
\]
then using the convexity argument, $a*w^\star-e_0=0$. 

	
\end{proof}

	
	
	
	
	
	
	
	





\section{Stability of convex blind deconvolution with infinite length inverse filter approximation }
Now we consider a setting where there is a kernel whose corresponding inverse kernel has infinite support, and we give a finite-length approximation. 

Within the space $V_{-\infty,\infty}$ of bilaterally infinite real-valued sequences $(h(i))_{i \in {\mathbb Z}}$, consider the affine subspace $V_{N_{-}, N_{+}} = \{ (...,0,0,h_{-N_{-}},\ldots, h_{-1}, 1, h_1, h_2, \ldots, h_{N_{+}},0,0,...)\}$, an $N_{-}+N_{+}$-dimensional subspace of bilateral sequences with support at most $N_{-}+N_{+}+1$.
The coordinate that is fixed to one is 
located at index $i=0$.
Each coordinate is zero outside of a window
of size $N_{-}$ on the left of zero 
and  size $N_{+}$ on the right of zero.
The special sequence $e_0 = (\dots,0,0,1,0,0,\dots)$,
vanishing everywhere except the origin,
belongs to $V_{-\infty,\infty}$ and to every
$V_{N_-,N_+}$.

Let $a \in V_{N_-,N_+}$.
Then $a =  (...,0,0,a_{-N_{-}},\ldots, a_{-1}, 1, a_1, a_2, \ldots, a_{N_{+}},0,0,...)$.
We also write $a = e_0 + a_{(0)}$,
where $a_{(0)} \equiv (1-e_0) \cdot a$ 
denotes the `part of $a$ supported away from location
$i=0$'. We also write $a = e_0 + a_L + a_R$,
where $a_L = (a_L(i))_{i \in {\mathbb Z}} 
= (a(i) 1_{\{i < 0\}})_{i \in {\mathbb Z}}$ 
denotes the `part of $a$ supported to
the left of $i=0$' and 
where $a_R = (a_R(i))_{i \in {\mathbb Z}} 
= (a(i) 1_{\{i > 0\}})_{i \in {\mathbb Z}}$
denotes the `part of $a$
supported to the right of $i=0$'.
Finally, we say that $a \in V_{N_{-}, N_{+}}$ 
is a length $ L= N_{-}+N_{+}+1$ filter.

\textbf{First example}: let's look at a simple example of infinite length inverse filter approximation.
Let $s\in (-1,1)$. For $a=( 0, 1, -s)$, then $a^{-1} = (\ldots, 0, 1, s, s^2, s^3, \ldots)$ is an infinite length inverse filter. 

If we choose a length $r$ approximation $w^{r}= (\ldots, 0, 1, s, s^2, s^3, \ldots, s^{r-1})$ then
\[
\|a*w^{r}-e_0\|_2^2 = |s|^{2r}.
\]


Now we consider general finite-length forward filter with infinite-length inverse filter.

We construct the finite-length approximation filter explicitly by truncation of Z-transform.

Let the Z-transform of $a$ be 
\[
 \begin{array}{ll}
A(z) =  \sum_{i=-s}^t a_{i} z^{-i}.
\end{array}
  \]
Then Z-transform of the inverse kernel $a^{-1}$ is $1/A(z)$.

\begin{theorem}
\label{thm:RootNorm}
Assuming we have a finite length forward filter $a$ with its Z-transform having roots inside the unit circle, namely $s_{k}:= e^{-\rho_k + i \varphi_k }$ with $|s_k|<1$ and $\rho_k>0$
for $k \in \{-N_{-}, \ldots -1, 1, \ldots N_{+}\}$.
\[
 \begin{array}{ll}
A(z) &=  \sum_{i=-N_{-}}^{N_{+}} a_{i} z^{-i}
\\
&= c_0\prod_{j=1}^{N_{-}} (1- s_{-j} z) \prod_{i=1}^{N_{+}} (1- s_{i} z^{-1}),
\end{array}
  \]
Where $c_0$ is a constant to make sure that the coefficient $a_0=1$.

Then for a vector index $r=(r_{-N_-}, \ldots, r_{-1}, r_{1},\ldots, r_{N_+} ) $, we could construct an approximate inverse filter $w^r$ with Z-transform
\[
 \begin{array}{ll}
W(z) 
&=  \frac{1}{c_0}\prod_{j=1}^{N_{-}} (\sum_{\ell_j=0}^{r_{-j}-1} s^{\ell_j}_{-j} z^{-\ell_j} ) \prod_{i=1}^{N_{+}} (\sum_{\ell_i=0}^{r_{-j}-1} s^{\ell_i}_{i} z^{\ell_i} )\\
&=  \frac{1}{c_0}\prod_{j=1}^{N_{-}} (1- (s_{-j} z)^{r_{-j}})(1- s_{-j} z)^{-1} \prod_{i=1}^{N_{+}}(1- (s_{i}  z^{-1})^{r_i}) (1- s_{i} z^{-1})^{-1}.
\end{array}
\]
Let $\phi^r = w^r*a-e_0$, then
\[
 \begin{array}{ll}
\|\phi^r \|_2^2 = \|a*w^r-e_0\|_2^2
&= \sum_{n=1}^{|{\cal I}|}\sum_{ k_1, \ldots, k_n  \in {\cal I} } 
\exp \Big(- 2 \big(r_{k_1} \rho_{k_1}+ \ldots+  r_{k_n} \rho_{k_n} \big) \Big).
\end{array}
  \]
When $\min_j r_j \rightarrow \infty$, it converges to zero at an exponential rate. The convergence rate is determined by the slowest decaying exponential term as a function of $\min_i (r_i \rho_i)$.
\[
 \begin{array}{ll}
\|\phi^r \|_2
&= O(
\exp \Big(-  \min_i (r_i \rho_i) \Big) ), \quad \min_i r_i \rightarrow \infty.
\end{array}
  \]
 \end{theorem}

 \begin{theorem} 
 \label{thm:bd_stab}
 Let $a\in V_{N_-, N_+} $ be a forward filter with all the roots of Z-transform strictly in the unit circle. Let $w^\star \in V_{(r-1)N_-, (r-1)N_+} $ be the solution of the convex optimization problem. Let $w^r$ be the constructed filter in previous theorem with a uniform vector index $(r, \ldots, r, r,\ldots, r)$.
  Let $\phi^\star = w^\star*a-e_0$, 
 then the solution satisfy
  \[
 \begin{array}{ll}
  B(e_0,\phi^\star) \|\phi^\star \|_2 \leq B(e_0,\phi^r) \|\phi^r \|_2.
\end{array}
  \]
  Additionally, as $p<p^\star$, $B(e_0,\phi^\star)$ and $B(e_0,\phi^r)$ are both upper and lower bounded. 
  Therefore, using the previous asymptotic exponential convergence bound on $\|\phi^r \|_2$ as 
  $r \rightarrow \infty$, it converges to zero at an exponential rate
  \[
 \begin{array}{ll}
   \|\phi^\star \|_2 \leq \frac{B(e_0,\phi^r)}{B(e_0,\phi^\star)} \|\phi^r \|_2
   \leq
   O(
\exp \Big(-  r \min_i (\rho_i) \Big) ), \quad r \rightarrow \infty.
\end{array}
  \]
 \end{theorem} 
 

\section{Proofs for stability results}	

\begin{proof}[Proof of Theorem~\ref{thm:RootNorm}]
Now let $\psi^r$ have Z-transform $\Psi(z)$, $\phi = a*w-e_0$ with Z-transform $\Phi(z)$,
\[
 \begin{array}{ll}
\Psi(z): = A(z)W(z) 
&=   \prod_{j=1}^{N_{-}} (1- (s_{-j} z)^{r_{-j}}) \prod_{i=1}^{N_{+}}(1- (s_{i} z^{-1})^{r_i}).
\end{array}
\]
Therefore,
\[
 \begin{array}{ll}
\Phi(z): = A(z)W(z) -1 
&=  \prod_{j=1}^{N_{-}} (1- (s_{-j} z)^{r_{-j}}) \prod_{i=1}^{N_{+}}(1- (s_{i} z^{-1})^{r_i}) -1.
\end{array}
  \]
Let ${\cal I} = \{-N_{-},\ldots, -1, 1, \ldots, N_{+} \}$ as the set of all the possible indicies.
And use polar representation of complex roots:
for any $k \in {\cal I} = \{-N_{-},\ldots, -1, 1, \ldots, N_{+} \}$
\[
s_{k} := |s_{k}| e^{i \varphi_k } = e^{-\rho_k + i \varphi_k },
\]
where 
\[
 \begin{array}{ll}
\varphi_k := \operatorname{Im}(\log( s_{k} )) = -i \log(\frac{s_{k}}{|s_{k}|}),
\end{array}
\]
and since $|s_{k}|<1$ we have 
\[
\rho_k = -\operatorname{Re}(\log( s_{k} ))>0.
\]

Now we consider all $z = e^{2 \pi i t}$ on the unit circle for $t\in [\frac{-1}{2},\frac{1}{2}]$,
then for any $k \in {\cal I}$
\[
 \begin{array}{ll}
- (s_{k} z^{-\mbox{sign}(k)})^{r_{k}} 
&= - |s_{k}|^{r_{k}} \exp \big(i [\varphi_k-  2 \pi t \mbox{sign}(k)]\cdot r_{k} \big)\\
&=   \exp \Big( -r_{k} \rho_k+ i \cdot \big( r_{k} (\varphi_k- 2 \pi t \mbox{sign}(k) ) + \pi  \big) \Big).
\end{array}
  \]
 Now we simplify the notation by defining 
 \[
 \upsilon_k(t) :=r_{k} (\varphi_k- 2 \pi t \mbox{sign}(k) ) + \pi,
 \]
then
 \[
 \begin{array}{ll}
\Phi(z)
&=  \prod_{j=1}^{N_{-}} (1- (s_{-j} z)^{r_{-j}}) \prod_{i=1}^{N_{+}}(1- (s_{i} z^{-1})^{r_{i}}) -1\\
&=  \prod_{k \in {\cal I}} \Big(1- |s_{k}|^{r_{k}} \exp \big(i [\varphi_k-  2 \pi t \mbox{sign}(k)]\cdot r_{k} \big)\Big)  -1\\
&=  \prod_{k \in {\cal I}} \bigg( 1+ \exp \Big( -r_{k} \rho_k+ i \cdot \big( r_{k} (\varphi_k- 2 \pi t \mbox{sign}(k) ) + \pi  \big) \Big) \bigg)     -1\\
&=  \prod_{k \in {\cal I}} \bigg( 1+ \exp \Big( -r_{k} \rho_k+ i \cdot \upsilon_k(t) \Big) \bigg) -1.
\end{array}
  \]
  
 For any $z = \exp{(2 \pi i t )}$ on the unit circle,
 \[
 \begin{array}{ll}
\Phi\big(\exp{(2 \pi i t )}\big)
&=  \prod_{k \in {\cal I}} \bigg( 1+ \exp \Big(- r_{k} \rho_k+ i \cdot \upsilon_k(t) \Big) \bigg)     -1\\
&= \sum_{n=1}^{|{\cal I}|}\sum_{ k_1, \ldots, k_n  \in {\cal I} } 
\exp \Big( -\big(r_{k_1} \rho_{k_1}+ \ldots+  r_{k_n} \rho_{k_n} \big)
+ i \cdot \big( \upsilon_{k_1}(t) + \ldots+  \upsilon_{k_n}(t)  \big) \Big).
\end{array}
  \]
 
Additionally, for any $t \in [\frac{-1}{2},\frac{1}{2}]$,
\[
 \begin{array}{ll}
 |\Phi\big(\exp{(2 \pi i t )}\big)|^2  
= 
\sum_{n=1}^{|{\cal I}|}\sum_{ k_1, \ldots, k_n  \in {\cal I} } 
\exp \Big( -2 \big(r_{k_1} \rho_{k_1}+ \ldots+  r_{k_n} \rho_{k_n} \big) \Big)
\end{array}
  \]
is independent of $t$, since the oscillation integral over the imaginary part is 
  \[
 \begin{array}{ll}
\int_{\frac{-1}{2}}^\frac{1}{2}
\exp  \bigg( i \cdot \Big(  \big(\upsilon_{k_1}(t) - \upsilon_{k_1}(t) \big) + \ldots+  \big(\upsilon_{k_n}(t) -\upsilon_{k_n}(-t)\big)  \Big) \bigg)  dt
= 1.
\end{array}
  \]
Therefore, using Fourier isometry,
 \[
 \begin{array}{ll}
 \|\phi\|_2^2 &= \int_{\frac{-1}{2}}^\frac{1}{2} |\Phi\big(\exp{(2 \pi i t )}\big)|^2 dt \\
&= \sum_{n=1}^{|{\cal I}|}\sum_{ k_1, \ldots, k_n  \in {\cal I} } 
\exp \Big( -2 \big(r_{k_1} \rho_{k_1}+ \ldots+  r_{k_n} \rho_{k_n} \big) \Big).
\end{array}
  \]
When $\min_j r_j \rightarrow \infty$, it converges to zero at an exponential rate. The convergence rate is determined by the slowest decay exponential term as a function of $\min_i (r_i \rho_i)$.
\[
 \begin{array}{ll}
\|\phi^r \|_2
&= O(
\exp \Big(-  \min_i (r_i \rho_i) \Big) ), \quad \min_i r_i \rightarrow \infty.
\end{array}
  \]
  
\end{proof}


\begin{proof}[Proof of Theorem~\ref{thm:bd_stab}]
Following the previous phase transition analysis, 
let the directional finite difference of the objective at $e_0$ be $D(w) = |(w*Y)_0| - |X_0|$, $I$ be the support of $\tilde{X}$, let $\phi = w*a-e_0$,  then we have 
\[
 \begin{array}{ll}
{\mathbb E} D(w) &= \sqrt{\frac{2}{\pi}}{\mathbb E}_I [\|(w*a)_I\|_2 - \|(e_0)_I\|_2]\\
&= \sqrt{\frac{2}{\pi}}{\mathbb E}_I [\|(e_0+\phi)_I\|_2 - \|(e_0)_I\|_2]\\
&= \sqrt{\frac{2}{\pi}}\|{\phi}\|_2 B(p, \phi) . \\
\end{array}
  \]
Now using the optimality of $w^\star \in V_{(r-1)N_-, (r-1)N_+} $, for all $w^r in V_{(r-1)N_-, (r-1)N_+}$,
we have
 \[
 \begin{array}{ll}
{\mathbb E}_I [\|(w^\star*a)_I\|_2 \leq {\mathbb E}_I [\|(w^r*a)_I\|_2,
\end{array}
  \]
  therefore,
  \[
 \begin{array}{ll}
B(p, \phi^\star) \|{\phi^\star}\|_2 \leq B(p, \phi)\|{\phi}\|_2.
\end{array}
  \]
Due to the bi-Lipschitz property of ${\mathbb E}_I \|(e_0+\phi)_I\|_2$ around $e_0$, we know when $p<p^\star$, $B(p, \phi)$ is positive and bounded by a constant.

Therefore, as $p<p^\star$, $B(e_0,\phi^\star)$ and $B(e_0,\phi^r)$ are both upper and lower bounded. 
 \[
 \begin{array}{ll}
   \|\phi^\star \|_2 \leq \frac{B(e_0,\phi^r)}{B(e_0,\phi^\star)} \|\phi^r \|_2.
\end{array}
  \]
 Using the previous theorem, 
 \[
 \begin{array}{ll}
   \|\phi^\star \|_2 \leq \frac{B(e_0,\phi^r)}{B(e_0,\phi^\star)} \sum_{n=1}^{|{\cal I}|}\sum_{ k_1, \ldots, k_n  \in {\cal I} } 
\exp \Big( -2 r\big(\rho_{k_1}+ \ldots+ \rho_{k_n} \big) \Big),
\end{array}
  \]
 using asymptotic exponential convergence bound on $\|\phi^r \|_2$ as 
  $r \rightarrow \infty$, it converges to zero at an exponential rate
  \[
 \begin{array}{ll}
   \|\phi^\star \|_2 
   \leq
   O(
\exp \Big(-  r \min_i (\rho_i) \Big) ), \quad r \rightarrow \infty.
\end{array}
  \]
\end{proof}

\section{Robustness under moving average Gaussian noise }
Consider a convoluted Gaussian noise with an average standard deviation $\sigma$ and a forward moving average filter $b$ with unit norm $\|b\|_2=1$, then
 \[
\begin{array}{ll}
Y&= a* (X+ \sigma b*G).
\end{array}
\]


This model would include the case of IID mixture of sparse Gaussian and small Gaussian $X_t \sim p N(0,1) +(1-p) N(0,\sigma_1)$ with $\sigma = \sigma_1$ and $b=e_0 $.
It also includes the Gaussian observation noise model
\[
\begin{array}{ll}
Y&= a* X+  \sigma_2 G_2 = a* (X+  \sigma_2 a^{-1}*G_2),
\end{array}
\]
with $b = a^{-1}/\|a^{-1}\|_2$, and $\sigma = \sigma_2 \|a^{-1}\|_2$.

 \begin{theorem}
 \label{thm:rand_exact_rob}
 Let $\psi^\star$ be the solution of the convex optimization problem in Eq.(4.1) for the moving average random noisy model
  \[
\begin{array}{ll}
Y&= a* (X+ \sigma b*G),
\end{array}
\]
 then
\label{thm:Bound_Normal_Noise}
\[
B(e_0, \psi^\star-e_0) \|\psi^\star -e_0\|_2 = {\mathbb E}_I  \| (\psi^\star)_I \|_2 -{\mathbb E}_I  \| (e_0)_I \|_2 \leq  (1-p) \sigma +
 p(\sqrt{1+ \sigma^2 }-1).
\]
When $\sigma\leq 1,$
\[
(1-p) \sigma +
 p(\sqrt{1+ \sigma^2 }-1)  \leq \sigma,
\]
therefore, when $p<p^\star$, $\sigma\leq 1,$ there exists a constant $C$,
\[
 \|\psi^\star -e_0\|_2 \leq C \sigma.
\]
\end{theorem}  

\section{Robustness under adversarial noise }
In the adversarial noise setting, we observe
\[
	\begin{array}{ll}
	Y&= a* (X+  \zeta) 
	\end{array}
	\]
where $\zeta$ is a sequence chosen by an adversary under constraint:	
\[
	\| \zeta\|_\infty \leq \eta.
	\]
 \begin{theorem}
\label{thm:adv_exact_rob}
	Under adversarial noise, 
Let $w^\star$ be the solution of the population convex optimization
	\[
 \begin{array}{ll}
 \underset{w }{\mbox{minimize}}   &{\mathbb E} \frac{1}{N}\|w*Y\|_{\ell_1({\cal T}_N)}\\
\mbox{subject to}  
& (\alpha*w)_0 =1.
\end{array}
  \]
Define $\psi^\star = a*w^\star$; then $\psi^\star$ satisfies the following bound:
 \[
 \begin{array}{ll}
B(e_0, \psi^\star-e_0) \|\psi^\star -e_0\|_2 = {\mathbb E}_I  \| (\psi^\star)_I \|_2 -{\mathbb E}_I  \| (e_0)_I \|_2 \leq 
 p\sqrt{\frac{2}{\pi}}(\mathcal{R}(\eta) - 1)+(1-p)\eta.
\end{array}
  \]  
  Here $\mathcal{R}(\eta)$
is the folded Gaussian mean, for standard Gaussian $G$:
\[
\mathcal{R}(\eta) :=  \sqrt{\frac{\pi}{2}} E_G |\eta+  G| = \exp\{-\eta^2/2\} + \sqrt{\frac{\pi}{2}}\eta \left(1 - 2\Phi\left(-\eta\right) \right).
\]  
$\mathcal{R}(\eta)-1$ is an even function that is monotonically non-decreasing for $\eta \geq 0$ with quadratic upper and lower bound: there exists constants $C_1\leq C_2$,
  \[
C_1 \eta^2 \leq \mathcal{R}(\eta)-1 \leq C_2 \eta^2, \forall \eta.
\]
  
Therefore, when $p<p^\star$, $B(e_0, \psi^\star-e_0)$ is a bounded positive constant, there exists a constant $C$, so that
 \[
 \|\psi^\star -e_0\|_2  \leq C  \eta, \forall \eta > 0.
 \]
\end{theorem}  





\section{Proofs}
\subsection{Gaussian noise robustness proof}
\begin{proof}[Proof of Theorem~\ref{thm:rand_exact_rob}]
Consider the noisy model
 \[
\begin{array}{ll}
Y&= a* (X+  \sigma b*G).
\end{array}
\]
 First, 
\[
 \begin{array}{ll}
 F_{\sigma}(\psi)= {\mathbb E}_{G} {\mathbb E}_X|\psi^T X+\sigma \psi^T (b* G) | = 
   {\mathbb E}_{G} {\mathbb E}_I | \sqrt{ \|\psi_I\|_2^2  + \sigma^2\|C_{b}^T \psi\|_2^2}G |,
\end{array}
  \]
where $C_{b}$ is the Topelitz matrix with the first column being $b$.
  
Let $\psi^\star $ be the optimization solution, when $p<p^\star$,  we have  a chain of inequality:
    \[
 \begin{array}{ll}
F_{\sigma}(e_0)
 \geq F_{\sigma}(\psi^\star ) 
 \geq F_{0}(\psi^\star )
 \geq F_{0}(e_0).
\end{array}
  \]
  We have 
\[
 \begin{array}{ll}
 F_{\sigma}(e_0)
 &= 
{\mathbb E}_{G} {\mathbb E}_I | \sqrt{ \|(e_0)_I\|_2^2  + \sigma^2\|b\|_2^2}G | \\
&= 
{\mathbb E}_{G} {\mathbb E}_I | \sqrt{ \|(e_0)_I\|_2^2  + \sigma^2}G | \\
  &=
 \sqrt{\frac{2}{\pi}}[ 
 (1-p) \sigma +
 p(\sqrt{1+ \sigma^2 }) ].
\end{array}
  \]
  
Therefore, when $p<p^\star$
   \[
 \begin{array}{ll}
F_{\sigma}(e_0)-F_{0}(e_0) 
 \geq F_{0}(\psi^\star )-F_{0}(e_0) \geq 0.
\end{array}
  \]
  \[
B(e_0, \psi^\star-e_0) \|\psi^\star -e_0\|_2 = {\mathbb E}_I  \| (\psi^\star)_I \|_2 -{\mathbb E}_I  \| (e_0)_I \|_2 \leq  (1-p) \sigma +
 p(\sqrt{1+ \sigma^2 }-1). 
\]
When $\sigma\leq 1,$
\[
(1-p) \sigma +
 p(\sqrt{1+ \sigma^2 }-1)  \leq \sigma,
\]
therefore, when $p<p^\star$, $\sigma\leq 1,$ there exists a constant $C$,
\[
 \|\psi^\star -e_0\|_2 \leq C \sigma.
\]
\end{proof}



\subsection{Adversarial noise robustness proof}
Our data generative model is
\[
	\begin{array}{ll}
	Y&= a* (X+  \zeta),
	\end{array}
	\]
where 	
\[
	\| \zeta\|_\infty \leq \eta.
	\]


\begin{proof}[Proof of Theorem~\ref{thm:adv_exact_rob}]
The population convex optimization is
	\[
 \begin{array}{ll}
 \underset{w, z }{\mbox{minimize}}   & {\mathbb E} \frac{1}{N}\|w*Y\|_{\ell_1({\cal T}_N)}\\
\mbox{subject to}  
& (\alpha*w)_0 =1.
\end{array}
  \]
 By change of variable $\tilde{\psi} = w*a$, it could be reduced to a simpler problem 
  \[
 \begin{array}{ll}
 \mbox{minimize}_{w,s}   & {\mathbb E}|\psi^T (X+  \zeta)| \\
\mbox{subject to}  
& u^T \psi =1.
\end{array}
  \]
 Let $\psi^\star $ be the optimization solution of the worst-case objective  over all possible $\zeta$, defined as $F_{\eta}(\psi)$:
 \[
 \begin{array}{ll}
 F_{\eta}(\psi)
 &= \sup_{\zeta: \| \zeta\|_\infty \leq \eta } {\mathbb E}|\psi^T (X+ \zeta)| \\
 &= \sup_{\zeta: \| \zeta\|_\infty \leq \eta } {\mathbb E}_I  | \|\psi_I\|_2 G  + \psi^T \zeta  |,
\end{array}
  \] 
   when $p<p^\star$,  we have a chain of inequality:
    \[
 \begin{array}{ll}
F_{\eta}(e_0)
 \geq F_{\eta}(\psi^\star ) 
 \geq F_{0}(\psi^\star )
 \geq F_{0}(e_0).
\end{array}
  \]
   Therefore, when $p<p^\star$
   \[
 \begin{array}{ll}
F_{\eta}(e_0)-F_{0}(e_0) 
\geq F(\psi(\zeta),\zeta)-F_{0}(e_0)
 \geq F_{0}(\psi^\star )-F_{0}(e_0) \geq 0.
\end{array}
  \]
  Therefore,
 \[
 \begin{array}{ll}
B(e_0, \psi^\star-e_0) \|\psi^\star -e_0\|_2 = {\mathbb E}_I  \| (\psi^\star)_I \|_2 -{\mathbb E}_I  \| (e_0)_I \|_2 .
\end{array}
  \]   
  
From the folded Gaussian mean formula, let $G$ be scalar standard Gaussian, we have 
\[
 \begin{array}{ll}
 F_{\eta}(e_0)
  &= \sup_{\zeta: \| \zeta\|_\infty \leq \eta } {\mathbb E}_I  {\mathbb E}_G| \|(e_0)_I\|_2 G  +  \zeta_0  |\\
&= 
\sup_{\zeta: \| \zeta\|_\infty \leq \eta} [p{\mathbb E}_G | G  +  \zeta_0  |+ (1-p)|\zeta_0|]\\
  &=
\sup_{\zeta: \| \zeta\|_\infty \leq \eta}[  p \sqrt{\frac{2}{\pi}} \mathcal{R}(\zeta_0) + (1-p)|\zeta_0|]
\\
 &=
p \sqrt{\frac{2}{\pi}} \mathcal{R}(\eta) + (1-p)\eta.
\end{array}
  \]
 The last inequality comes from the fact that $\mathcal{R}(\zeta_0)$ is an even function that is monotonically non-decreasing for $\zeta_0 \geq 0$. 
 We will prove this conclusion below:
 \[
 \begin{array}{ll}
B(e_0, \psi^\star-e_0) \|\psi^\star -e_0\|_2 = {\mathbb E}_I  \| (\psi^\star)_I \|_2 -{\mathbb E}_I  \| (e_0)_I \|_2 \leq 
 (1-p)\eta + p \sqrt{\frac{2}{\pi}} (\mathcal{R}(\eta) - 1).
\end{array}
  \] 



\end{proof}


\paragraph{Tool: folded Gaussian mean formula}
From general theory of folded Gaussian,
\footnote{see: \url{ https://en.wikipedia.org/wiki/Folded_normal_distribution}.}
 $ |\mu+ \sigma G |, \quad  G \sim N(0, 1).$
Then its mean is
\begin{equation}
\label{FoldedGaussianMean}
E_G |\mu+ \sigma G| =  \sqrt{\frac{2}{\pi}} \sigma \exp\{-\mu^2/(2\sigma^2)\} + \mu \left(1 - 2\Phi\left(\frac{-\mu}{\sigma}\right) \right),
\end{equation}
where $\Phi$  is the normal cumulative distribution function:
\[
\Phi(x) = \frac 1 {\sqrt{2\pi}} \int_{-\infty}^x e^{-t^2/2} \, dt.
\]
As a related remark, its variance is
\begin{equation}
\label{FoldedGaussianVar}
{\rm Var}_G |\mu+ \sigma G| = E_G( |\mu+ \sigma G|-E_G |\mu+ \sigma G|)^2 =\sigma^2+ \mu^2 -(E_G |\mu+ \sigma G|)^2.
\end{equation}  

 \begin{lemma}
\label{lem:FoldedGaussian}
We define the ratio of folded Gaussian mean as 
\[
\mathcal{R}(\gamma) := \frac{E_G |\mu+ \sigma G|}{E_G |\sigma G|} = \frac{E_G |\mu+ \sigma G|}{\sqrt{\frac{2}{\pi}}\sigma} = \sqrt{\frac{\pi}{2}} E_G |\gamma+  G| = \exp\{-\gamma^2/2\} + \sqrt{\frac{\pi}{2}}\gamma \left(1 - 2\Phi\left(-\gamma\right) \right).
\]

$\mathcal{R}(\gamma)$ is an even function that is monotonically non-decreasing for $\gamma \geq 0$.

We have three different expressions (asymptotic expansion around  $\gamma$) for $\mathcal{R}(\gamma)$:
\begin{eqnarray}
\label{FoldedGaussianMean_ratio}
\mathcal{R}(\gamma)
&=&  \exp\{-\gamma^2/2\} + \sqrt{\frac{\pi}{2}}\gamma \left(1 - 2\Phi\left(-\gamma\right) \right)\\
&=&  \exp\{-\gamma^2/2\}\{1 + \frac{1}{2}\gamma \left[ \gamma + \frac{ \gamma^3}{3} + \frac{ \gamma^5}{3\cdot 5} + \cdots + \frac{ \gamma^{2n+1}}{(2n+1)!!} + \cdots\right]\}\\
&=& 1 + \frac{3}{2}\gamma^2 + \frac{1}{24} \gamma^4   - \frac{ \gamma^6}{120}  + O(\gamma^8).
\end{eqnarray}
When $\gamma<1,$ 
\[
 (\frac{3}{2} + \frac{1}{24}) \gamma^2 \geq  \mathcal{R}(\gamma) -1\geq  \frac{3}{2}\gamma^2.
\]
When $\gamma<M$ for $M>1$, this lemma can be generalized:
there exists constants $C'\leq C$,
\[
C' \gamma^2 \leq \mathcal{R}(\gamma)-1 \leq C \gamma^2.
\]
 \end{lemma}
\paragraph{Proof for folded Gaussian mean}
\begin{proof}[Proof of lemma~\ref{lem:FoldedGaussian}]
Plug in the formula for mean of folded Gaussian \ref{FoldedGaussianMean}, 
we have
\[
E_G |\mu+ \sigma G| = \sqrt{\frac{2}{\pi}} \sigma \exp\{-\gamma^2/2\} + \mu \left(1 - 2\Phi\left(-\gamma\right) \right).
\]
Divide it by 
\[
E_G |\sigma G| =   \sqrt{\frac{2}{\pi}} \sigma,
\]
we get the first equality.

Since the CDF of the standard normal distribution can be expanded by integration by parts into a series:
\[
\Phi( \gamma)=\frac{1}{2} + \frac{1}{\sqrt{2\pi}}\cdot e^{- \gamma^2/2} \left[ \gamma + \frac{ \gamma^3}{3} + \frac{ \gamma^5}{3\cdot 5} + \cdots + \frac{ \gamma^{2n+1}}{(2n+1)!!} + \cdots\right],
\]
where $!!$ denotes the double factorial. 

\[
1 - 2\Phi(-\gamma)  = \frac{1}{\sqrt{2\pi}}\cdot e^{- \gamma^2/2}\left[ \gamma + \frac{ \gamma^3}{3} + \frac{ \gamma^5}{3\cdot 5} + \cdots + \frac{ \gamma^{2n+1}}{(2n+1)!!} + \cdots\right].
\]
This gives the second inequality.

Additionally,
 \[
 \begin{array}{ll}
  e^{- \gamma^2/2} &=\sum_{n=0}^{\infty} \frac{ \gamma^{2n}}{(-2)^n (n)!}\\
  &=  1- \frac{ \gamma^2}{2} + \sum_{n=2}^{\infty} \frac{ \gamma^{2n}}{(-2)^n (n)!},
\end{array}
  \] 
then
 \[
 \begin{array}{ll}
& \exp\{-\gamma^2/2\}\{1 + \frac{1}{2}\gamma \left[ \gamma + \frac{ \gamma^3}{3} + \frac{ \gamma^5}{3\cdot 5} + \cdots + \frac{ \gamma^{2n+1}}{(2n+1)!!} + \cdots\right]\}\\
=& (\sum_{n=0}^{\infty} \frac{ \gamma^{2n}}{(-2)^n (n)!})\{1 + \frac{1}{2}\gamma \left[ \sum_{n=0}^{\infty} \frac{ \gamma^{2n+1}}{(2n+1)!!} \right]\}\\

=& 1 + \frac{3}{2}\gamma^2 + \frac{1}{24} \gamma^4   - \frac{ \gamma^6}{120}  + O(\gamma^8). 
\end{array}
  \] 
  
Let $C_\Phi( \gamma^2)$ be a function of $ \gamma$, then it is a composition with the inner function being $ \gamma^2,$ defined as follows: 
\[
 \begin{array}{ll}
C_\Phi( \gamma^2) 
&:= (2\Phi( \gamma)-1)/\left(\frac{ \gamma}{\sqrt{2\pi}}\right) \\
&= e^{- \gamma^2/2} \left[1 + \frac{ \gamma^2}{3} + \frac{ \gamma^4}{3\cdot 5} + \cdots + \frac{ \gamma^{2n}}{(2n+1)!!} + \cdots\right]\\
&= [\sum_{n=0}^{\infty} \frac{ \gamma^{2n}}{(2n+1)!!} ]/[\sum_{n=0}^{\infty} \frac{ \gamma^{2n}}{2^n (n)!}] \\
&= [\sum_{n=0}^{\infty} \frac{ \gamma^{2n}}{(2n+1)!!} ]/[\sum_{n=0}^{\infty} \frac{ \gamma^{2n}}{(2n)!!}] \\
&\leq 1.
\end{array}
\]
We remark further that
\[
 \begin{array}{ll}
C_\Phi( \gamma^2) 
&= [\sum_{n=0}^{\infty} \frac{ \gamma^{2n}}{(2n+1)!!} ]/[\sum_{n=0}^{\infty} \frac{ \gamma^{2n}}{(2n)!!}] \\
&= [1 + \frac{ \gamma^2}{3}+ +  \frac{ \gamma^4}{15}+ O( \gamma^6) ]/[1 + \frac{ \gamma^2}{2}+ +  \frac{ \gamma^4}{8}+O( \gamma^6)] \\
&= 1 -\frac{ \gamma^2}{6}+  \frac{ \gamma^4}{40}+ O( \gamma^6).
\end{array}
\]
Then
 \[
 \begin{array}{ll}
\mu \left(1 - 2\Phi\left(-\gamma\right) \right)
&= \frac{1}{\sqrt{2\pi}}\cdot \mu^2 \frac{1}{\sigma} C_\Phi(\gamma^2)\\
&= \frac{1}{\sqrt{2\pi}}\cdot \mu^2 \frac{1}{\sigma} (1-\gamma^2/6 +O(\gamma^4)).\\
\end{array}
\] 
Combining the above:
\[
 \begin{array}{ll}
E_G |\mu+ \sigma G|
&=   \left[ \sqrt{\frac{2}{\pi}} \sigma \exp\{-\gamma^2/2\} + \frac{1}{\sqrt{2\pi}}\cdot \mu^2 \frac{1}{\sigma} C_\Phi(\gamma^2)\right]\\
&=   \left[ \sqrt{\frac{2}{\pi}} \sigma \{ 1- \frac{\gamma^2}{2} + \sum_{n=2}^{\infty} \frac{\gamma^{2n}}{(-2)^n (n)!}\} + \frac{1}{\sqrt{2\pi}}\cdot \mu^2 \frac{1}{\sigma} C_\Phi(\gamma^2)\right]\\
&= \sqrt{\frac{2}{\pi}}   \left[ \sigma 
\{ 1 + (1+\frac{1}{2}\cdot C_\Phi(\gamma^2))\gamma^2 +   \sum_{n=2}^{\infty} \frac{\gamma^{2n}}{(-2)^n (n)!}\} \}\right].\\
\end{array}
\] 
Define the residual of this expansion as
\[
 \begin{array}{ll}
\mbox{Res}(\gamma^2) 
&:= \frac{ (2\Phi( \gamma)-1)/\left(\frac{ \gamma}{\sqrt{2\pi}}\right)-1}{2}\cdot \gamma^2 +  (\exp\{-\gamma^2/2\} - 1 + \gamma^2/2)
\\
&= \frac{ C_\Phi(\gamma^2)-1}{2}\cdot \gamma^2 +  (\exp\{-\gamma^2/2\} - 1 + \gamma^2/2) \\
&=    \frac{ C_\Phi(\gamma^2)-1}{2}\cdot \gamma^2  +   \{ \sum_{n=2}^{\infty} \frac{\gamma^{2n}}{(-2)^n (n)!}\} \} \\
&=    \frac{-\gamma^4 }{12}  +    \frac{ \gamma^6}{80} +\frac{\gamma^{4}}{(-2)^2 (2)!} +\frac{\gamma^{6}}{(-2)^3 (3)!}+
O(\gamma^8)\\
&=    \frac{\gamma^4 }{24}  - \frac{ \gamma^6}{120}  + O(\gamma^8).
\end{array}
\] 
We know that if $\gamma^2<1,$ 
\[
 \begin{array}{ll}
\mbox{Res}(\gamma^2) 
&=    \frac{\gamma^4 }{24}  - \frac{ \gamma^6}{120}  + O(\gamma^8)\\
&\geq 0.
\end{array}
\] 

When $\gamma<1,$ 
\[
 (\frac{3}{2} + \frac{1}{24}) \gamma^2 \geq  \mathcal{R}(\gamma) -1\geq  \frac{3}{2}\gamma^2.
\]
When $\gamma<M$ for $M>1$, this lemma can be generalized:
there exist constants $C'\leq C$,
\[
C' \gamma^2 \leq \mathcal{R}(\gamma)-1 \leq C \gamma^2.
\]
\end{proof}



  
  
  
 





\subsection{Examples}
In the following, we study the simple case when there is only one root.
Let $s\in (-1,1)$. For $a=(\ldots, 0, 1, -s, 0, \ldots)$, $a^{-1} = (\ldots, 0, 1, s, s^2, s^3, \ldots)$, we could construct 
\[
w^{r,1}:=  (\ldots, 0, 1, s, s^2, s^3, \ldots, s^r)
\]

For $\gamma\in (0,1)$,
\[
\bar a^{\gamma} :=e_0-\gamma a= (\ldots, 0, 1-\gamma,  \gamma s, 0, \ldots)
\]
\[
 \begin{array}{ll}
w^{r,\gamma} 
&= a^{-1} - a^{-1}*_{r+1} (\bar a^{\gamma})
\end{array}
\]
\[
\begin{array}{ll}
a*(a^{-1}-w^{r,\gamma}) 
&= *_{r+1} (\bar a^{\gamma})\\
&= *_{r+1} (\ldots, 0, 1-\gamma,  \gamma s, 0, \ldots)
\end{array}
\]

\[
\mathcal{F}[(\ldots, 0, 1-\gamma, \gamma s, 0, \ldots)](t)  = (1-\gamma)+   \gamma s  \cdot e^{2\pi i  t}, \quad  t\in T = [\frac{-1}{2}, \frac{1}{2}];
\]

\[
|\mathcal{F}[(\ldots, 0, 1-\gamma, \gamma s, 0, \ldots)] |^2(t) = (1-\gamma)^2+ (\gamma s )^2+ 2 {(1-\gamma)} {\gamma}s \cos{(2\pi  t)} , \quad t\in T = [\frac{-1}{2}, \frac{1}{2}];
\]

\[
 \begin{array}{ll}
\|*_{r+1} (\bar a^{\gamma})\|_2^2 
&= \| (\hat{\bar a}^{\gamma})^{r+1}\|_2^2\\
&=  \int_{\frac{-1}{2}}^\frac{1}{2}  |(1-\gamma)+   \gamma s  \cdot e^{2\pi i  t}|^{2r+2} dt\\
&= \int_{\frac{-1}{2}}^\frac{1}{2}  [(1-\gamma)^2+ (\gamma s )^2+ 2 {(1-\gamma)} {\gamma}s \cos{(2\pi  t)} ]^{r+1} dt
\end{array} \]
Notice the following lemma, 
\begin{lemma}
For odd number, $\int_{\frac{-1}{2}}^\frac{1}{2}  \cos^{2k-1}{(2\pi  t)}  dt = 0$ for any $k\in {\mathbb Z}_+$; for even number, 
\[
\int_{\frac{-1}{2}}^\frac{1}{2}  \cos^{2k}{(2\pi  t)}  dt =  {2k\choose k} 2^{-2k},
\]
\end{lemma}
\begin{proof}
\[
 \begin{array}{ll}
2^n\cos^n (2\pi  t)=(e^{i \cdot 2\pi  t}+e^{-i \cdot 2\pi  t})^n=\sum_{k=0}^n {n\choose k} (e^{i\cdot 2\pi  t})^k\ (e^{-i\cdot 2\pi  t})^{n-k}=\sum_{k=0}^n {n\choose k} e^{(2k-n)i\cdot 2\pi  t},
\end{array} \]

\[
 \begin{array}{ll}
\int_{\frac{-1}{2}}^\frac{1}{2} e^{(2k-n)i\cdot 2\pi  t} dt= \delta_{2k-n}
\end{array} \]
\[
 \begin{array}{ll}
\int_{\frac{-1}{2}}^\frac{1}{2} 2^n\cos^n (2\pi  t) dt =\sum_{k=0}^n {n\choose k} \delta_{2k-n},
\end{array} \]
therefore, for odd number $n=2k-1$, $\int_{\frac{-1}{2}}^\frac{1}{2}  \cos^{2k-1}{(2\pi  t)}  dt = 0$ for any $k\in {\mathbb Z}_+$; for even number $n=2k$, 
\[
\int_{\frac{-1}{2}}^\frac{1}{2} 2^{2k} \cos^{2k}{(2\pi  t)}  dt =  {2k\choose k}.
\]
\end{proof}

\begin{lemma}
Now
$R_1 :=  (1-\gamma)^2+ (\gamma s )^2,$
$R_2 := (1-\gamma) \gamma s,$
\[
 \begin{array}{ll}
 \| (\hat{\bar a}^{\gamma})^{r+1}\|_2^2
=\sum_{k=0}^{\frac{r+1}{2}} [(1-\gamma)^2+ (\gamma s )^2]^{r+1-2k} [ (1-\gamma) \gamma s]^{2k}{2k\choose k}{r+1\choose 2k} 
\end{array} \]
\end{lemma}
\begin{proof}
Now define
$R_1 :=  (1-\gamma)^2+ (\gamma s )^2,$
$R_2 := {(1-\gamma)} {\gamma}s,$
\[
 \begin{array}{ll}
 \| (\hat{\bar a}^{\gamma})^{r+1}\|_2^2
&= \int_{\frac{-1}{2}}^\frac{1}{2}  [(1-\gamma)^2+ (\gamma s )^2+ 2 {(1-\gamma)} {\gamma}s \cos{(2\pi  t)} ]^{r+1} dt\\
&= \int_{\frac{-1}{2}}^\frac{1}{2}  
[R_1+ R_2 \cdot 2 \cos{(2\pi  t)} ]^{r+1} dt\\
&=\sum_{k=0}^{\frac{r+1}{2}} R_1^{r+1-2k} R_2^{2k}{2k\choose k}{r+1\choose 2k} 
\end{array} \]
\end{proof}

Use the Laplace's method theorem \ref{Laplace},
We could instead directly optimize  $\gamma$ over the $\ell_\infty$ norm of $|\hat{\bar a}^{\gamma}|^2$,
\begin{lemma}
\[
 \||\hat{\bar a}^{\gamma}|^2\|_\infty = \max_{ t\in [\frac{-1}{2}, \frac{1}{2}]}\{ (1-\gamma)^2 + (\gamma s)^2+ 2 (1-\gamma) \gamma s \cos{(2\pi  t)} \}  = (1-\gamma)^2 + (\gamma s)^2+ 2 |(1-\gamma) \gamma s| = (|1-\gamma|+|\gamma s|)^2 
\]
then

 \[
  \begin{array}{ll}
\gamma_\infty &= \arg\min_{\gamma>0}\||\hat{\bar a}^{\gamma}|^2\|_\infty \\
&= \arg\min_{\gamma>0}(|1-\gamma|+|\gamma s|)^2 \\
&= 1
 \end{array} \]
\[
\min_{\gamma\in (0,1)}\||\hat{\bar a}^{\gamma}|^2\|_\infty
= (|1-\gamma_\infty|+|\gamma_\infty s|)^2 
= |s|^2
\]
\end{lemma}
And
 \[
 \begin{array}{ll}
\|*_{r+1} (\bar a^{\gamma_\infty  })\|_2^2 
&=  s^{2r+2} 
\end{array} \]

If we instead choose $\gamma_2$ as $ \cos^2{\theta} = \frac{1}{1+s^2}$ to minimize the $\ell_2$ norm instead, then
\[
\|\hat{\bar a}^{\gamma_2}\|_2^2  = \min_{\gamma\in (0,1)}\|\hat{\bar a}^{\gamma}\|_2^2 
=  (1-\gamma_2)^2 + (\gamma_2 s)^2 
= \frac{s^2}{1+s^2}
\]
And in this case,
 \[
  \begin{array}{ll}
\||\hat{\bar a}^{\gamma_2}|^2\|_\infty
&= (|1- \frac{1}{1+s^2}|+| \frac{1}{1+s^2} s|)^2 \\
&=  s^2 (\frac{1+|s|}{1+s^2})^2 \in (s^2, 1)
 \end{array} \]
 
 Therefore, 
 \[
  \begin{array}{ll}
\||\hat{\bar a}^{\gamma_2}|^2\|_\infty \in (s^2, 1)
 \end{array} \]
\section{Phase transition based on directional derivative and subgradient condition}
\paragraph{Exact calculation of subgradient for phase transition }
In this section, we give a clean proof of phase transition theorems presented before using sub-gradient and directional derivative.

\begin{lemma}
Let $J \subset [n]$ be the support of $X$, let $\psi_J : = \psi\cdot 1_J$ denote the elementwise product of vector $\psi$ with the indicator vector of subset $J$.  Let $B_J$ denote the central section of the euclidean  ball $B({\mathbb R}^n)$,
where the slice is produced the linear space $\mbox{span} \{e_i : i\in J \}$.
Alternatively, we may write
$B_J: = \{ v_J: \| v_J\|_2 \leq 1\}$.  The set-valued subgradient
operator applied to $\|\psi_J\|$
evaluates as follows:
 \[
 \begin{array}{ll}
\partial_{\psi}    \| \psi_J \|
&=  N(\psi_J).
\end{array}
  \]
Here $N$ maps ${\mathbb R}^n$ into  subsets of ${\mathbb R}^n$,
and is given by:
\[
N(\psi_J):= 
\begin{cases}
    \psi_J/\|\psi_J\|_2, &  \psi_J\neq 0 \\
    B_J, &  \psi_J = 0
  \end{cases}.
\]  
\end{lemma}

\begin{definition}
Consider a probability space containing just the possible  
outcomes  $J \subset [n]$.
Let $S_J$, $J \subset [n]$,  denote a closed compact subset of ${\mathbb R}^n$.
Let $I$ be a random subset of
$[n]$ drawn at random from this probability space with probability $\pi_J = P\{ I = J \}$
of elementary event $J$.
Consider the set-valued random variable $S \equiv S_J$. We define its expectation as
\[
 {{\mathbb E}} S  := \sum_J \pi_J \cdot S_J.
\]
On the right side, we mean the closure of the 
compact set produced by all sums of the form
\[
 \sum_J \pi_J \cdot s_J
\]
where each $s_J \in S_J$.
\end{definition}

\begin{lemma}
For any $\psi \in {\mathbb R}^n$, and $X = (X_i)_{i=1}^n$ with $X_i \sim_{iid} BG(p,0,1)$.
Let now $I \subset [n]$ be the random support of $X$.
Let $\partial_\psi $ denote the set-valued subgradient
operator.
  \[
 \begin{array}{ll}
\partial_\psi [E{|\sum_{i} \psi_i X_i| }  ]
&= \sqrt{\frac{2}{\pi}} \cdot {{\mathbb E}}_I [N(\psi_I)].
\end{array}
  \]
\end{lemma}

\paragraph{Subgradient and directional derivative at $e_0$}
Now we focus on $e_0$. Note that for any subset $J \subset [n]$, $(e_0)_J$ is either the zero
vector or else $e_0$. Hence $N((e_0)_J)$
is either $B_{J}$ or $\{e_0\}$. What drives
this dichotomy is whether $0 \in J$ or not.

\begin{lemma}  
\[
 \begin{array}{ll}
{\mathbb E}_I [N((e_0)_I) ]
&= p e_0 + \sum_{J,0\not\in J } \pi_{J} B_{J} \\
& = p e_0 + (1-p) {\cal B}_0,
\end{array}
  \]
where
\[
 \begin{array}{ll}
{\cal B}_0 = {\mathbb E}_{I}[ B_{I} | 0 \not \in I ] = 
(1-p)^{-1} \cdot \sum_{J,0\not\in J } {\pi_{J}} B_{J}.
\end{array}
\]
 
\end{lemma}


\begin{proof}
\begin{eqnarray*}
{\mathbb E}_I [N((e_0)_I) ] &=& \sum_{J } \pi_{J} B_{J} \\
&=&  \sum_{J, 0 \in J}   \pi_{J} e_0 +  \sum_{J,0\not\in J }  \pi_{J} B_{J}\\
&=& p \cdot e_0 + (1-p) \cdot {{\cal B}}_0 .
\end{eqnarray*}
 \end{proof}
To compute with ${\mathbb E}_I [N((e_0)_I) ]$,
we need:
\begin{lemma} \label{eq:ComputeExpect}
For each fixed $\beta \in {\mathbb R}^n$:
\[
  \sup_{b \in {\cal B}_0} \langle \beta, b \rangle
   = {\mathbb E}_{I}[ \|\beta_I\|_2 | 0 \not \in I  ].
  \]
\end{lemma}

\begin{proof}
Note that in the definition of the set 
\[
{\cal B}_0 = {\mathbb E}_{I}[ B_{I} | 0 \not \in I ],
\]
 each term $B_J$ obeys the bound
$\sup_{g_J \in B_J} \|g_J\|_2 = 1 $.
Now, given the fixed vector $\beta \in {\mathbb R}^n$, define:
\begin{eqnarray*}
    b^* &=& {\mathbb E}_{I}[ \frac{\beta_I}{\| \beta_I \|_2} | 0 \not \in I  ]\\
     &=&  \sum_{0 \not \in J} \pi_J \frac{\beta_J }{\|\beta_J\|_2}.
 \end{eqnarray*}
 Since each term in this sum has Euclidean norm at most $1$, $b^* \in {{\cal B}}_0$.
Now
\begin{eqnarray*}
\langle  \beta, b^* \rangle &=&
\langle \beta , \sum_{0 \not \in J} \pi_J \frac{\beta_J }{\|\beta_J\|_2} \rangle \\
&=& \sum_{0 \not \in J} \pi_J \langle \beta , \frac{\beta_J }{\|\beta_J\|_2} \rangle \\
&=& \sum_{0 \not \in J} \pi_J \|\beta_J\|_2 .
 \end{eqnarray*}
On the other hand, for any fixed 
vector $g = \sum \pi_J g_J $,
with each $g_J \in B_J$,
we have
\begin{eqnarray*}
\langle  \beta, g \rangle &=&  \sum_{0 \not \in J} \pi_J \langle \beta , g_J  \rangle \\
&\leq& \sum_{0 \not \in J} \pi_J  \|\beta_J \|_2 \|g_J\|_2 \\
& \leq & \sum_{0 \not \in J} \pi_J  \|\beta_J \|_2 
= \langle  \beta, b^* \rangle.
 \end{eqnarray*}
\end{proof}

Lemma (\ref{eq:ComputeExpect}) 
can be viewed as a special instance of
Theorem \textbf{A.15} from \cite{bai2018subgradient}:
\begin{lemma}[Interchangeability of set expectation and support function] 
Suppose a random compact set $S \subset {\mathbb R}^n$ is integrably bounded and the underlying probability space is non-atomic, then ${\mathbb E}[S]$ is a convex set and for any fixed vector $\beta \in {\mathbb R}^n,$
 \begin{equation} \label{eq:SetExpectEquality}
 \sup_{g\in {\mathbb E} S} \langle\beta, g\rangle =  \sup_{g\in  S} {\mathbb E}\langle\beta, g\rangle .
 \end{equation}
\end{lemma}

Define $\beta_{(0)} = \beta \cdot 1_{\{0\}^c}$ as the part of $\beta$ supported away from $0$.
Further define 
\[
V_1(\beta_{(0)}):= \frac{{\mathbb E}_{I}[ \|\beta_{I}\|_2 | 0 \not \in I ]}{\|\beta_{(0)}\|_2}.
\]
Let $N = \|\beta_{(0)}\|_0$; then 
 $V_1(\beta_{(0)})$ satisfies
\begin{equation} 
 p  \leq  V_1(\beta_{(0)}) \leq \sqrt{p} .
\end{equation}

\begin{lemma}


 The directional derivative at $e_0$ along $\beta$ 
 evaluates to the following:
 \begin{eqnarray*}
 \lim_{t\rightarrow 0^+}\frac{1}{t}({\mathbb E}_I [\|(e_0+t\beta)_I\|_2 - \|(e_0)_I\|_2]) 
 &=&  p \cdot \langle\beta, e_0\rangle + (1-p) \cdot {\mathbb E}_{I}[ \|\beta_{I}\|_2 | 0 \not \in I ]\\
 &=& [p \cos{\angle(\beta, e_0)} + (1-p)   \sin{\angle(\beta, e_0)} V_1(\beta_{(0)})] \|\beta\|_2.
 \end{eqnarray*}
\end{lemma}
 \begin{proof}

Let $\langle b, v \rangle  \equiv b^T v$.

Applying Lemma \ref{eq:ComputeExpect}, we proceed as follows:
\[
 \begin{array}{ll}
\lim_{t\rightarrow 0^+}\frac{1}{t}({\mathbb E}_I [\|(e_0+t\beta)_I\|_2 - \|(e_0)_I\|_2])
&= \sup_{g\in {\mathbb E}_I [N((e_0)_I)]} \langle\beta, g\rangle\\
&= p \cdot \langle\beta, e_0\rangle + (1-p) \cdot  \sup_{g\in {\cal B}_0 } \langle\beta, g\rangle\\

&= p \cdot \langle\beta, e_0\rangle + (1-p) \cdot {\mathbb E}_{I}[ \|\beta_{I}\|_2 | 0 \not \in I ] \\
&= [p \cdot \cos{\angle(\beta, e_0)} + (1-p) \cdot  \sin{\angle(\beta, e_0)} V_1(\beta_{(0)})] \|\beta\|_2.
\end{array}
  \]
  \end{proof}


\paragraph{Projected subgradient}
\begin{lemma} 
If $\beta$ must satisfy a constraint $u^T \beta =0$, then we only consider directional derivative at $e_0$ along $\beta$ for those $\beta$ that $u^T \beta =0$.
Therefore, let $P_{u}^{\perp}$ be the projection onto the hyperplane as the orthogonal complement of $u$, then the following three equivalent conditions all make sure $e_0$ is the stationary point.
\begin{itemize}
\item
  \[
 \begin{array}{ll}
0\in P_{u}^{\perp} {\mathbb E}_I [N((e_0)_I) ].
\end{array}
  \]
  \item
Equivalently, there exists a subgradient $g\in  P_{u}^{\perp} {\mathbb E}_I [N((e_0)_I) ]$ such that for all $\beta$ satisfying $u^T \beta = 0$,
\[
\beta^T g \geq 0.
\]
\item
Also equivalently, for all $\beta$,
\[
 \begin{array}{ll}
\sup_{g\in  [N((e_0)_I)]}{\mathbb E}_I[ \langle   P_{u}^{\perp}  \beta, g\rangle] \quad \geq 0.
\end{array}
  \]
  \end{itemize}
\end{lemma}
 
Let $\theta = \angle({u, e_0})$, for $\beta$ need to satisfy a constraint $u^T \beta =0$, we get
\[
 \begin{array}{ll}
\lim_{t\rightarrow 0^+}\frac{1}{t}({\mathbb E}_I [\|(e_0+t\beta)_I\|_2 - \|(e_0)_I\|_2])
&= [p \cos{\angle(\beta, e_0)} + (1-p)  V_1(\beta_{(0)}) \sin{\angle(\beta, e_0)}] \|\beta\|_2.
\end{array}
  \]
Here $\angle(\beta, e_0)\in [0, \pi], \sin{\angle(\beta, e_0)}\in [0,1]$. 
  
\begin{figure} 
\centering
\includegraphics[width=.55\textwidth]{./figures_BD/Angle_Diagram_label.png}
\caption{Demonstration of the relation between
  $\angle( \beta, -e_0)$ and ${\angle(u, e_0)}$.}
\label{fig-Angle}
\end{figure}
  
\begin{lemma}
  \label{Angle_Inequality}
   \[
  \angle(  \beta, - e_0) \geq \angle(P_u^\perp (-e_0), -e_0)= \frac{\pi}{2} - \theta
  \]
  
    \[
  \tan(\angle(  \beta, - e_0)) \geq \tan\angle(P_u^\perp (-e_0), -e_0) =\cot{\theta}
  \]
\end{lemma}
  
\begin{proof}
   
We know that geometrically, using the property that projection of $e_0$ on the hyperplane with normal vector $u$ has the smallest angle among all the $\beta$ in that hyperplane, we have that if
   \[
  \angle(  \beta, - e_0) \geq \angle(P_u^\perp (-e_0), -e_0) = \frac{\pi}{2} - \angle(u, e_0) = \frac{\pi}{2} - \theta,
  \]
then
  \[
  \tan\angle(  \beta, - e_0) \geq \tan\angle(P_u^\perp (-e_0), -e_0)= \cot{\theta}.
  \]
 
\end{proof}
  
Let $H_u=\{\beta : \quad u^T \beta =0 \}$. We define the phase transition threshold as
  \[
 p^\star:= 
 \sup_{p\in [0,1]}\{\inf_{\|\beta\|_2=1, u^T \beta =0 }(-p \cdot \sqrt{1-\| \beta_{(0)}\|_2^2} + (1-p) \cdot  \| \beta_{(0)}\|_2 V_1(\beta_{(0)}))> 0\}.
 \]
 \begin{lemma}
Then 
 \[
\frac{p^\star}{1-p^\star}=  \inf_{\| \beta\|_2=1, u^T \beta =0} \tan(\angle( \beta_{(0)}, -e_0)) V_1(\beta_{(0)}).
\]
For all $p<p^\star$ the convex program finds the correct inverse filter $a^{-1}$ and hence the sparse signal $X$.
\end{lemma}
  


\begin{proof}
 
We know that the KKT condition of the convex program is
  \[
  p \cos{\angle(\beta, e_0)} + (1-p)  V_1(\beta_{(0)}) \sin{\angle(\beta, e_0)}>0,
  \]
  if and only if 
  $\cos{\angle(\beta, e_0)}\geq 0,$ or $\cos{\angle(\beta, e_0)}<0$ and 
  \[
      \frac{1-p}{ p}  V_1(\beta_{(0)}) \tan{(\angle(\beta, -e_0))}>1.
  \]
  If $\cos{\angle(\beta, e_0)}<0, \cos{\angle(\beta, -e_0)}>0, \angle(\beta, -e_0) \in [0, \frac{\pi}{2}).$ 
  
  Therefore,
  \[
      \frac{1-p}{ p}  \inf_{\beta\in H_u} [V_1(\beta_{(0)}) \tan\angle(  \beta, - e_0)>1,
  \]
  
  \[
\frac{p^\star}{1-p^\star}=  \inf_{\| \beta\|_2=1, u^T \beta =0} \tan(\angle( \beta_{(0)}, -e_0)) V_1(\beta_{(0)}).
\]
   \end{proof}
 
  
\begin{theorem}
 \label{thm:PT_lower}
The threshold $p^\star$ satisfies 
		\[ 
\tan(\angle( P_u^\perp (-e_0), -e_0)) p^\star \leq \frac{p^\star}{1-p^\star} \leq	\tan(\angle( P_u^\perp (-e_0), -e_0)) V_1(P_u^\perp (-e_0)).
		\]
\end{theorem}
 

\begin{proof}
  First, we prove the lower bound
    \begin{eqnarray*}
      \frac{p^\star}{1-p^\star} &=&  \inf_{\beta\in H_u} [V_1(\beta_{(0)}) \tan\angle(  \beta, - e_0)\\
      &\geq& \inf_{\beta\in H_u} V_1(\beta_{(0)}) \inf_{\beta\in H_u}\tan(\angle(\beta, -e_0))\\
      &=& p \tan{(\angle(P_u^\perp (-e_0), -e_0))}.
  \end{eqnarray*}
 
Moreover, the lower bound is achieved when $\beta_{(0)}$ is one-sparse. Since $\tan{(\angle(P_u^\perp (-e_0), -e_0))}=\cot\theta$, the lower bound of $p^\star$ is $1- \tan\theta$.
 
For the upper bound, we plug in $\beta = P_u^\perp (-e_0)$, then
	\[ 
	\inf_{\| \beta\|_2=1, u^T \beta =0} 
	\tan(\angle( \beta, -e_0))V_1(\beta_{(0)}) \leq
	\tan(\angle( P_u^\perp (-e_0), -e_0))
	V_1(P_u^\perp (-e_0)).
	\]
It is worth commenting that since $V_1(P_u^\perp (-e_0))\leq \sqrt{p}$, and $\tan(\angle( P_u^\perp (-e_0), -e_0))=\cot(\angle(u, e_0))=\cot\theta$, we have
	\[ 
	 \tan(\angle( P_u^\perp (-e_0), -e_0))
 V_1(P_u^\perp (-e_0)) 
	\leq \cot(\angle(u, e_0)) \sqrt{p}.
	\]
 
  \end{proof}
  
  
  
  
  

 
 

  
  
  

 



  \end{lemma}
  
  
  
 
 


  
  
  



 
  







































\section{Finite difference of objective}
We study the upper and lower bound of: 
\[
B(e_0, \phi) := \frac{{\mathbb E}_I  \| (e_0+\phi)_I \|_2 - {\mathbb E}_I  \| (e_0)_I \|_2}{\|\phi\|_2} 
\]

This upper and lower bound allows us to connect objective ${\mathbb E}_I  \| \psi_I \|_2$ and the $2-$norm of $ \psi - e_0$:
\[
{\mathbb E}_I  \| \psi_I \|_2 - {\mathbb E}_I  \| (e_0)_I \|_2 = {\mathbb E}_I  \| \psi_I \|_2 - p=B(e_0, \psi - e_0) \| \psi - e_0\|_2
\]














 
  


\paragraph{Bi-Lipschitzness of finite difference of objective near $e_0$ for linear constraint}
We normalize the problem by defining $t:=\|\phi\|_2, \beta:=\frac{\phi}{\|\phi\|_2}$.
After normalization, and taking into account the linear constraint $u^T \phi = 0$, we will study, for $\beta\in \mathcal{B}_t :=\{\|\beta\|_2=1, u^T \beta = 0,  \|e_0+t\beta\|_\infty\leq 1 \}$ for finite $t$, the upper and lower bound of 
\[
\frac{{\mathbb E}_I  \| (e_0+t\beta)_I \|_2 - {\mathbb E}_I  \| (e_0)_I \|_2}{t}.
\]

First, if $t\rightarrow 0^+$, then we get the directional derivative along direction of $\beta$. 
\begin{eqnarray*}
\lim_{t\rightarrow 0^+}\frac{{\mathbb E}_I  \| (e_0+t \beta)_I \|_2 - {\mathbb E}_I  \| (e_0)_I \|_2}{t}.
\end{eqnarray*}

Due to convexity of the function $\beta \rightarrow {\mathbb E}_I  \| (e_0+t \beta)_I \|_2$, we have the finite difference lower bounded by directional derivative:
\begin{eqnarray*}
\frac{{\mathbb E}_I  \| (e_0+t \beta)_I \|_2 - {\mathbb E}_I  \| (e_0)_I \|_2}{t} 
\geq 
 \lim_{t\rightarrow 0^+}\frac{{\mathbb E}_I  \| (e_0+t \beta)_I \|_2 - {\mathbb E}_I  \| (e_0)_I \|_2}{t} .
\end{eqnarray*}





\begin{theorem}
\label{thm:bound_diff}
We have upper and lower bound
\begin{eqnarray*}
0 \leq \frac{{\mathbb E}_I  \| (e_0+t \beta)_I \|_2 - {\mathbb E}_I  \| (e_0)_I \|_2}{t} -\lim_{t\rightarrow 0^+}\frac{{\mathbb E}_I  \| (e_0+t \beta)_I \|_2 - {\mathbb E}_I  \| (e_0)_I \|_2}{t}
\leq 
  \frac{t}{2}p\Big( 
   \beta_0^2 
 +
 p  (1-\beta_0^2) \Big) \leq \frac{pt}{2}.
\end{eqnarray*}
This leads to
\[
0\leq B(e_0, \phi) -	 \nabla_{\phi}{{\mathbb E}_I  \| (e_0+\phi)_I \|_2}\mid_{\phi=0}\leq \frac{p}{2}\|\phi \|_2.
\]
\end{theorem}


\begin{proof}
As mentioned before, the lower bound is derived from convexity.

In the proof, for finite $t$, we calculate the finite difference condition on $I_0$:
\begin{eqnarray*}
\frac{{\mathbb E}_I  \| (e_0+t \beta)_I \|_2 - {\mathbb E}_I  \| (e_0)_I \|_2}{t}
&=&
(1-p)  {\mathbb E}_{I}[ \|\beta_{I}\|_2 | 0 \not \in I ] + p\big(\frac{{\mathbb E}_I [ \| e_0+t\beta_I \|_2 -   1 | 0 \in I ]}{t} \big).
\end{eqnarray*}
And
\begin{eqnarray*}
\lim_{t\rightarrow 0^+}\frac{{\mathbb E}_I  \| (e_0+t \beta)_I \|_2 - {\mathbb E}_I  \| (e_0)_I \|_2}{t}
&=&
(1-p)  {\mathbb E}_{I}[ \|\beta_{I}\|_2 | 0 \not \in I ] + p\beta_0.
\end{eqnarray*}
In the following, we prove an upper bound 
\begin{eqnarray*}
 \frac{{\mathbb E}_I [ \| e_0+t\beta_I \|_2 -   1 | 0 \in I ]}{t}- \beta_0
\leq   \frac{t}{2} p \| \beta_{{(0)}} \|_2^2  + \frac{t}{2} \beta_0^2.
\end{eqnarray*}

For finite $t$,
\begin{eqnarray*}
\frac{{\mathbb E}_I [ \| e_0+t\beta_I \|_2 -   1 | 0 \in I ]}{t}
&=&
{\mathbb E}_{I}[ \left( (\frac{1}{t}+ \beta_0)^2+\| \beta_{I_{(0)}} \|_2^2 \right)^{1/2} -\frac{1}{t}| 0 \in I ]\\
&=& \frac{1}{t}{\mathbb E}_{I}[ \left( 1+ 2t\beta_0 + t^2 \beta_0^2+t^2\| \beta_{I_{(0)}} \|_2^2 \right)^{1/2} -1| 0 \in I ].
\end{eqnarray*}



Additionally, using concavity of the function $\sqrt{1+x}$, we have
\begin{eqnarray*}
 && \frac{1}{t}{\mathbb E}_{I}[ \left( 1+ 2t\beta_0 + t^2 \beta_0^2+t^2\| \beta_{I_{(0)}} \|_2^2 \right)^{1/2} -1| 0 \in I ]\\
&\leq&  \frac{1}{t}[ \left( 1+ 2t\beta_0 + t^2 \beta_0^2+t^2{\mathbb E}_{I}\| \beta_{I_{(0)}} \|_2^2| 0 \in I ] \right)^{1/2} -1]\\
&=&  \frac{1}{t}[ \left( 1+ 2t\beta_0 + t^2 \beta_0^2+t^2 p \| \beta_{{(0)}} \|_2^2 \right)^{1/2} -1]\\
&=&  \frac{1}{t}[ \left( 1+ 2t\beta_0 + t^2 \beta_0^2+t^2 p (1-\beta_0^2) \right)^{1/2} -1].
\end{eqnarray*}
Now, apply the inequality: 
for any $x\geq -1$:
\[
\sqrt{1+x} -1 \leq \frac{x}{2},
\]
we have
\begin{eqnarray*}
 \frac{1}{t}[ \left( 1+ 2t\beta_0 + t^2 \beta_0^2+t^2 p \| \beta_{I_{(0)}} \|_2^2 \right)^{1/2} -1]
&\leq&  \beta_0 + \frac{t}{2} (p \| \beta_{{(0)}} \|_2^2  +  \beta_0^2).
\end{eqnarray*}
Therefore, 
\begin{eqnarray*}
\frac{{\mathbb E}_I  \| (e_0+t \beta)_I \|_2 - {\mathbb E}_I  \| (e_0)_I \|_2}{t} -\lim_{t\rightarrow 0^+}\frac{{\mathbb E}_I  \| (e_0+t \beta)_I \|_2 - {\mathbb E}_I  \| (e_0)_I \|_2}{t}
\leq 
  \frac{t}{2}p\Big( 
   \beta_0^2 
 +
 p  (1-\beta_0^2) \Big) \leq \frac{pt}{2}.
\end{eqnarray*}
The last inequality is due to $\beta_0^2 \leq 1$.

\end{proof}












































































 

 


\section{Old concentration results}

 
 

  
 
  
  
  



 





  









 





  
  



 






Now we need to derive a concentration inequality on $D$. 
We focus on deriving a concentration inequality on $\|h*x\|_1$, and the concentration inequality 
on  $\|(h*x)_S\|_1$ can be derived similarly.

We remark that $ \|h*x\|_1 =\sum_{j=1}^n |h^{(j),T} x|= \sum_{j=1}^n |\langle C(a^{(j)}) v, x\rangle|$, 
if they are independent random variables, we can bound the deviation of $ \|h*x\|_1$ from its mean
using tail bounds via Bernstein’s inequality, by using the previous two bound on
variance and absolute value. 
This is the strategy to derive concentration inequality in dictionary learning\cite{luh2016dictionary,blasiok2016improved}.
This is also the strategy to derive concentration inequality for multiple observation blind deconvolution setting, since there are independence across different observations.

However, here due to the nature of the convolution, we cannot directly apply Bernstein’s inequality. There are additional dependencies that cause technical difficulties. 

Therefore, we use the dual representation of the $1$-norm and use Talagrand-Bernstein’s inequality on supreme of random function to derive concentration.
We put this part of the proof in the following lemma, since they are technical and they could be of independent interest for other problems with dependency structure. 




From the lemma, if we let $Z = \|h*x\|_1$, then we have 
 \[
  \begin{array}{ll}
& {\mathbb P} \left[  |\|h*x\|_1- {\mathbb E} \|h*x\|_1 | > t   \right] \\
&= {\mathbb P} \left[  |Z- {\mathbb E} Z | > t   \right] \\
&\leq   \exp\left(  \frac{-t^2/2   }{\sigma^2+K(2{\mathbb E} Z+t/3)}\right)
\end{array}
  \]  
where $\sigma^2\leq  {\mathbb E} Z^2$, $K\leq \|Z\|_\infty$.

From the previous calculation, we recall that we have a lower bound on expectation 
$ {\mathbb E} \|h*x\|_1$,
\[
{\mathbb E} \|h*x\|_1\geq    \sqrt{\frac{p}{2}}  n \| h\|_2 
\] 
\[
{\mathbb E} Z = {\mathbb E} \|h*x\|_1 \geq  \sqrt{\frac{p}{2}}  n \| h\|_2
\]
  









Now we calculate $\sigma^2, K$:
 \[
  \begin{array}{ll}
\sigma^2 &={\mathbb E} \langle w, C(h)x \rangle^2,\\
&={\mathbb E} (C(h)^* w)^T xx^* C(h)^* w  \\
&\leq  p \|C(h)^* w\|_2^2\\
&=p \langle w, C(h)C(h)^* w \rangle 
\end{array}
  \]  
Let $C(h)=UD(\hat{h})U^*$ be the eigen-decomposition, where $U$ is the unitary Fourier matrix, $D(\hat{h})$ is the diagonal matrix with $\hat{h}$ as its diagonal entries. Then using $h=a*v$, $\hat{h}=\hat{a}\hat{v}$, we have
      \[
  \begin{array}{ll}
(C(h)C(h)*) 
&=   UD(\hat{h})U^* U D(\hat{h})^* U^* \\
&=  UD(|\hat{h}|^2) U^* \\
&=  UD(|\hat{a}\hat{v}|^2) U^*. \\
\end{array}
  \] 
Now we have
 \[
  \begin{array}{ll}
\sigma^2 &\leq 
 p \langle w, C(h)C(h)^* w \rangle \\
&= p  \langle w,  UD(|\hat{a}\hat{v}|^2) U^* w \rangle \\
&= p  \langle U^* w,  D(|\hat{a}\hat{v}|^2) U^* w \rangle 
\end{array}
  \] 
Let $\tilde{w} = \langle U^* w $, using the fact that $\|w\|_{\infty}\leq 1$, we have  $\|\tilde{w}\|_{\infty}\leq 1$, therefore,
 \[
  \begin{array}{ll}
\sigma^2 &\leq  p  \langle \tilde{w},  D(|\hat{a}\hat{v}|^2) \tilde{w} \rangle \\
&=  p  \sum_{i=1}^n \tilde{w}_i^2 |\hat{a}_i\hat{v}_i|^2)  \\
&\leq  p  \sum_{i=1}^n  |\hat{a}_i\hat{v}_i|^2 \\
&\leq  p  \max_{i=1}^n |\hat{a}_i|^2\sum_{i=1}^n  |\hat{v}_i|^2 \\
&=  p  \max_{i=1}^n |\hat{a}_i|^2 \|v\|_2^2 \\
&=  p  \max_{i=1}^n |\hat{a}_i|^2 
\end{array}
  \] 
  
  Now we derive the absolute value bound. First, since $x_i$ are iid sampled from the product of Bernoulli random variable and standard Gaussian, the supreme of the absolute value $\tau = \max_i |x_i|$ is upper bounded by the  supreme of standard Gaussian, which is less than $  2\log^{1/2} n$ with probability $1-\exp(-Cn^2)$. In the following, we only consider this case and assume $\tau \leq 2\log^{1/2} n$ and add the tail probability into the tail probability in the concentration inequality.
  
 Using other side of Khintchine inequality, we have
  \[
 \begin{array}{ll}
&  |(h*x)_i|\\
 =&  |\sum_{j=1}^n h_{i-j} \Omega_{j} |G_{j}|\mbox{sign}(G_{j}) |\\
\leq&  
  \sqrt{\sum_{j=1}^{n} (h_{i-j} \Omega_{j}|G_{j}|)^2}\\
 \leq&  \tau  \sum_{i=1}^n \sqrt{\sum_{j=1}^{n}  h_{i-j}^2}\\
   =&   \tau  \| h\|_2 
\end{array}
  \]  
Therefore, we can take $K=  \tau  \| h\|_2  .$

 












We point out that the eigenvalues of $C(a)$ is the Fourier transform of $\hat a$ of $a$, which is the 
the entry-wise inverse $1/\hat w^{0}$. And the singular values of $C(a)$ are the absolute values $|\hat a|$. 
Therefore, using the fact that $\|v\|_2=1$,
\[
 \begin{array}{ll}
 \min_i |\hat a_i| \leq \|h\|_2 =  \|C(a)v\|_2 \leq \max_i |\hat a_i| 
\end{array}
  \]
  
Then the condition number of $C(a)$ can be represented as $\delta_a =   \max_i |\hat a_i|/\min_i |\hat a_i|$.



Overall, for $Z = \|h*x\|_1$, we have ${\mathbb E} Z \geq \sqrt{\frac{p}{2}}  n \| h\|_2$, we choose $t = \sqrt{\frac{p}{2}}  n \| h\|_2$. Since 
 \[
  \begin{array}{ll}
 {\mathbb P} \left[  |Z- {\mathbb E} Z | > t   \right] 
\leq   \exp\left(  \frac{-t^2/2   }{\sigma^2+K(2{\mathbb E} Z+t/3)}\right),
\end{array}
  \]  
we calculate
\[
  \begin{array}{ll}
 &  \frac{-t^2/2   }{\sigma^2+K(2{\mathbb E} Z+t/3)}\\
=&  -\frac{1   }{2(\sigma^2/t^2+K(2{\mathbb E} Z+t/3)/t^2)}\\
\leq &  -\frac{1   }{2(\sigma^2/t^2+K(2+1/3)/t)}\\
 \end{array}
  \]  
Now, we calculate both terms on the denominator,
  \[
  \begin{array}{ll}
   \sigma^2/t^2 &\leq (p  \max_{i=1}^n |\hat{a}_i|^2)/( \sqrt{\frac{p}{2}}  n \| h\|_2)^2 \\
  &\leq 2\max_{i=1}^n |\hat{a}_i|^2)/(\min_{i=1}^n |\hat{a}_i|^2 n^2 \\
   &\leq 2\delta_a^2/ n^2 ,
 \end{array}
  \]  
   \[
  \begin{array}{ll}
  (2+1/3)K/t 
 &\leq  \frac{7}{3}\tau  \| h\|_2 /\sqrt{\frac{p}{2}}  n \| h\|_2\\
  &\leq \frac{7\sqrt{2}}{3}\tau  /\sqrt{p}  n \\
   &\leq \frac{7}{2}\tau  /\sqrt{p}  n 
 \end{array}
  \]  
  
 As we discussed before, since $\tau$ is the maximum of IID Gaussian random variable, 
 $\tau \leq C_1 \log^{1/2}(n)$ with probability $1 - \exp(-C_2 n^2)$ for $C_1$ large enough.
 
And from the assumption we have $ \delta_a \leq C_0 n^{1/2}/ \log^{1/4}(n)$.
  
Therefore, 
\[
  \begin{array}{ll}
& {\mathbb P} \left[  |\|h*x\|_1- {\mathbb E} \|h*x\|_1 | > t   \right] \\
&\leq   \exp\left( \frac{-1   }{ 4\delta_a^2/ n^2  + 7\tau  /\sqrt{p}  n  } \right)\\
&\leq   \exp\left( \frac{-1   }{ 4\delta_a^2/ n^2  + 7 C_1 \log^{1/2}(n)  /\sqrt{p}  n  } -C_2 n^2\right)\\
&\leq   \exp\left( \frac{-1   }{ 4\delta_a^2/ n^2  + 7 C_1 \log^{1/2}(n)  /\sqrt{p}  n  } -C_2 n^2\right)\\
&\leq   \exp\left( \frac{-n   }{  \log^{1/2}(n) (4C_0+  C_3 \sqrt{p})} -C_2 n^2 \right)\\
&\leq   \exp\left( \frac{- C n   }{  (p\log(n))^{1/2}}  \right)
 \end{array}
  \]  
  
  

  
  
  
  
  
  
  
  

\begin{proof}
In the lemma below, we use a covering argument to give a union bound for $v\in B_k=\{v
\in{\mathbb R}^k\mid \|v\|_2=1\}$. 
We want to cover $B_k$ with $\epsilon$ balls. From classical covering argument,
we know that we need at most $|N| = (3/\epsilon)^k$ $\epsilon$ balls to cover $B_k$, and we use symmetrization trick to find the proper $\epsilon$.

Let the deviation be
\[
\tilde{\mathcal{X}}_v =   \sum_{i=1}^n \left( | \langle C(a^{(i)}) v, x\rangle| - {\mathbb E} | \langle C(a^{(i)}) v, x\rangle|\right).
\]  
We will choose $1/\epsilon$ to be the Lipschitz constant of  $\tilde{\mathcal{X}}_v$ as a function of $v$. Let's compute the Lipschitz constant.

 
 

We introduce additional independent random sign $s_j$ , which are IID sampled from Rademacher variable by defining 
\[
\mathcal{X}_v =   \sum_{i=1}^n s_j \left( |\langle C(a^{(i)}) v, x\rangle| \right),
\]  
 then by lemma $24$ of \cite{blasiok2016improved}, for every integer $p$ we have 
 \[
 \| \sup_{v\in B_k} |\tilde{\mathcal{X}}_v | \|_p \lesssim \| \sup_{v\in B_k}  |\mathcal{X}_v| \|_p
 \]
 This is called symmetrization trick. Using this trick, we only need to analyze the tail behavior of the random variable $\mathcal{X}_v$. To obtain tail bounds for maximum  of  $\sup_{v\in B_k}  |\mathcal{X}_v|$, we need to bound tails of the random variables $\mathcal{X}_v-\mathcal{X}_{v'}$ for $v, v'\in B_k$.
 
Since  
\[
\mathcal{X}_v-\mathcal{X}_{v'} =  
\sum_{i=1}^n s_i \left( | \langle C(a^{(i)}) v, x\rangle| -   | \langle C(a^{(i)}) v', x\rangle|\right).
\]

Then use the fact that 
\[
| \langle C(a^{(i)}) v, x\rangle| -   | \langle C(a^{(i)}) v', x\rangle| \leq   \left( | \langle C(a^{(i)}) (v-v'), x\rangle|  \right),
\]
and  both $s_i | \langle C(a^{(i)}) v, x\rangle| -   | \langle C(a^{(i)}) v', x\rangle| $ and  
$s_i \left( | \langle C(a^{(i)}) (v-v'), x\rangle|  \right)$ are symmetric, if we have a tail bound for 
$\sum_{i=1}^n s_i \left( | \langle C(a^{(i)}) (v-v'), x\rangle|  \right)$ , 
we also have tail bounds on $\mathcal{X}_v-\mathcal{X}_{v'}$.

From these calculation, we saw that the Lipschitz constant of  $\tilde{\mathcal{X}}_v$ is bounded by 
 \[
  \begin{array}{ll}
\sqrt{\sum_{j=1}^n (\sum_{i=1}^n s_i |\sum_{t=1}^n a^{i-j+t} x_t|)^2}
&\leq   \sqrt{n (\|a\|_1 \|x\|_1)^2}\\
&\leq  \tau \sqrt{n}np \|a\|_1
\end{array}
  \]  
  
Therefore, we can choose $1/\epsilon = \tau \sqrt{n}np \|a\|_1$, then
at most $|N| = (3\tau \sqrt{n}np \|a\|_1)^k$ balls can cover $B_k$.
We have a concentration inequality in each ball,
We define the tail event 
\[
\mathcal{E}_v = \{|\tilde{\mathcal{X}}_v | > t\}
\]
we can take a union bound,
 \[
  \begin{array}{ll}
 {\mathbb P} \left(\cup_{v \in B_k} \mathcal{E}^c_v \right) &\leq |N| {\mathbb P} \left[  |\|h*x\|_1- {\mathbb E} \|h*x\|_1 | > t   \right] \\
&\leq   (3\tau \sqrt{n}np \|a\|_1)^k {\mathbb P} \left[  |\|h*x\|_1- {\mathbb E} \|h*x\|_1 | > t   \right] 
\end{array}
  \] 
  
Since  $n>  k \log(k/p)$, for large enough constant $C$, we showed in the lemma that there exist a positive constant $C_0$, 
 \[
  \begin{array}{ll}
 {\mathbb P} \left(\cup_{v \in B_k} \mathcal{E}^c_v \right) &\leq  \exp\left(  k\log(3\tau \sqrt{n}np \|a\|_1) - \frac{ C_6 n   }{  (p\log(n))^{1/2}}\right) \\
&\leq    \exp\left( k(\log(3\tau)+\log(\|a\|_1)+\log(p)+3/2 \log(n)) -\frac{ C n   }{  (p\log(n))^{1/2}}\right) 
\end{array}
  \] 
  

Then bound on $\sup_{v \in B_k} |\mathcal{X}_v|$ would imply the same bound on $\sup_{v \in B_k} |\tilde{\mathcal{X}}_v|$.
And with this bound on $\sup_{v \in B_k} |\tilde{\mathcal{X}}_v|$, we proved that 
for all $v\neq 0, v \in B_k$, $D>0$ with probability $1-\delta$, where
\[
\delta \leq  \exp\left(  -C_0 \delta_a^{-2} k\log(kp) \right)
\]
Therefore, when $n>C_1k^2\log(kp)$, 
$
\delta \leq   \exp\left(  -C_0 \delta_a^{-2} k\log(kp) \right),
$
then $\delta$ converges to zero as $k$ goes to infinity.
\end{proof}





\begin{lemma}
Let $Z = \|h*x\|_1$ where $h \in \mathcal{H}$, then 
 \[
  \begin{array}{ll}
& {\mathbb P} \left[  Z - {\mathbb E} Z  > t   \right] \\
&\leq   \exp\left(  \frac{-t^2/2   }{\sigma^2+K(2{\mathbb E} Z+t/3)}\right)
\end{array}
  \]  
where $\sigma^2\leq  {\mathbb E} Z^2$, $K\leq \|Z\|_\infty$.
\end{lemma}
\begin{proof}
Using duality of $1$-norm, we know that
\[
\|h*x\|_1 = \sup_{\|w\|_\infty\leq 1}\langle w, C(h)x \rangle
\]
 From the theorem $8.42$ of \cite{foucart2013mathematical}, we have the following inequality:
 for the supreme of functions $\langle w, C(h)x \rangle$, since ${\mathbb E} \langle w, C(h)x \rangle=0$,
 $ \langle w, C(h)x \rangle\leq K$, 
 \[
 \sigma^2 ={\mathbb E} \langle w, C(h)x \rangle^2,
 \]
 we have
 \[
  \begin{array}{ll}
 {\mathbb P} \left[  Z - {\mathbb E} Z  > t   \right] \leq   \exp\left(  \frac{-t^2/2   }{\sigma^2+K(2{\mathbb E} Z+t/3)}\right)
\end{array}
  \]  
\end{proof}
We also remark that the essential concentration inequality that we are trying to derive is the following:
let's consider the case when $x$ is IID standard Gaussian, we define $z = h*x = C(h)x$,
then $z$ is a non-isotropic Gaussian vector \[\mathcal{N}(0,\Sigma(h)),\] where 
\[  \Sigma(h)_{ij} = \langle h^{(i)},  h^{(j)}\rangle\]
And we are trying to derive a concentration inequality for $\|z\|_1$.






\begin{lemma}
For all $v\in B_k$, the probability 
\[
  \begin{array}{ll}
& {\mathbb P} \left[  |\|v*a*x\|_1- {\mathbb E} \|v*a*x\|_1 | > t, v \in B_k  \right]   \\ 
\leq&  (3\tau \sqrt{n}np \|a\|_1)^k {\mathbb P} \left[  |\|h*x\|_1- {\mathbb E} \|h*x\|_1 | > t   \right] 
\end{array}
  \] 
\end{lemma}
\begin{proof}
Let the deviation be
\[
\tilde{\mathcal{X}}_v =   \sum_{i=1}^n \left( | \langle C(a^{(i)}) v, x\rangle| - {\mathbb E} | \langle C(a^{(i)}) v, x\rangle|\right).
\]  
We will choose $1/\epsilon$ to be the Lipschitz constant of  $\tilde{\mathcal{X}}_v$ as a function of $v$. Let's compute the Lipschitz constant.

 
 

We introduce additional independent random sign $s_j$ , which are IID sampled from Rademacher variable by defining 
\[
\mathcal{X}_v =   \sum_{i=1}^n s_j \left( |\langle C(a^{(i)}) v, x\rangle| \right),
\]  
 then by lemma $24$ of \cite{blasiok2016improved}, for every integer $p$ we have 
 \[
 \| \sup_{v\in B_k} |\tilde{\mathcal{X}}_v | \|_p \lesssim \| \sup_{v\in B_k}  |\mathcal{X}_v| \|_p
 \]
 This is called symmetrization trick. Using this trick, we only need to analyze the tail behavior of the random variable $\mathcal{X}_v$. To obtain tail bounds for maximum  of  $\sup_{v\in B_k}  |\mathcal{X}_v|$, we need to bound tails of the random variables $\mathcal{X}_v-\mathcal{X}_{v'}$ for $v, v'\in B_k$.
 
Since  
\[
\mathcal{X}_v-\mathcal{X}_{v'} =  
\sum_{i=1}^n s_i \left( | \langle C(a^{(i)}) v, x\rangle| -   | \langle C(a^{(i)}) v', x\rangle|\right).
\]

Then use the fact that 
\[
| \langle C(a^{(i)}) v, x\rangle| -   | \langle C(a^{(i)}) v', x\rangle| \leq   \left( | \langle C(a^{(i)}) (v-v'), x\rangle|  \right),
\]
and  both $s_i | \langle C(a^{(i)}) v, x\rangle| -   | \langle C(a^{(i)}) v', x\rangle| $ and  
$s_i \left( | \langle C(a^{(i)}) (v-v'), x\rangle|  \right)$ are symmetric, if we have a tail bound for 
$\sum_{i=1}^n s_i \left( | \langle C(a^{(i)}) (v-v'), x\rangle|  \right)$ , 
we also have tail bounds on $\mathcal{X}_v-\mathcal{X}_{v'}$.

From these calculation, we saw that the Lipschitz constant of  $\tilde{\mathcal{X}}_v$ is bounded by 
 \[
  \begin{array}{ll}
\sqrt{\sum_{j=1}^n (\sum_{i=1}^n s_i |\sum_{t=1}^n a^{i-j+t} x_t|)^2}
&\leq   \sqrt{n (\|a\|_1 \|x\|_1)^2}\\
&\leq  \tau \sqrt{n}np \|a\|_1
\end{array}
  \]  
  
Therefore, we can choose $1/\epsilon = \tau \sqrt{n}np \|a\|_1$, then
at most $|N| = (3\tau \sqrt{n}np \|a\|_1)^k$ balls can cover $B_k$.
We have a concentration inequality in each ball,
We define the tail event 
\[
\mathcal{E}_v = \{|\tilde{\mathcal{X}}_v | > t\}
\]
we can take a union bound,
 \[
  \begin{array}{ll}
 {\mathbb P} \left(\cup_{v \in B_k} \mathcal{E}^c_v \right) &\leq |N| {\mathbb P} \left[  |\|h*x\|_1- {\mathbb E} \|h*x\|_1 | > t   \right] \\
&\leq   (3\tau \sqrt{n}np \|a\|_1)^k {\mathbb P} \left[  |\|h*x\|_1- {\mathbb E} \|h*x\|_1 | > t   \right] 
\end{array}
  \] 
\end{proof}

\section{Finite sample analysis}
\paragraph{Finite sample setting}
\begin{itemize}
\item
Let $(X_t)_{t \in {\mathbb Z}}$ be IID sampled from  $pN(0,1) + (1-p) \delta_0$.
 \item
The filter $a\in l_1({\mathbb Z}), (\mathcal{F} a)(\omega) \neq 0, \forall \omega\in T.$ Additionally, in finite sample setting, we assume $a^{-1}$ is zero outside of a centered window of radius $2k+1$, namely, the non-zero part can be written as $(\beta_{-k}, \ldots, \beta_0, \ldots, \beta_k)\in V$. 

We fix the shift and scaling effect by assuming $v^T a^{-1} = 1$ for a generic $v$ that has one single largest entry. 
\item
Let $Y= a* X$ be a linear process, we are given a series of observation $(Y_t)_{t\in [-(T+k),T+k]}$ from a centered window of radius $T+k$.
\item
 We denote $X^{(i)}$ as the sequence being flipped and shifted for $i$ step from $X$ s that $X^{(i)}_k = X_{-k +i}$. Let $J_N$ be the support of $X$.
 We define the length $N=2T+1$ window as ${\cal T} := [-T, T].$
\end{itemize}
Now we want to find an inverse filter 
$w = (w_{-k}, \ldots, w_0, \ldots, w_k) \in V$ such that the convolution $w*Y$ is sparse,
\begin{equation}
 \begin{array}{ll}
 \underset{w\in V}{  \mbox{minimize} }   & \frac{1}{N}\|w*Y\|_{1,{\cal T}}\\
 \mbox{subject to}  
& v^T w =1.
\end{array}
\tag{$P_{v}^T$}
\end{equation}




First, after change of variable,
\[
 \frac{1}{N}\|w*Y\|_{1,{\cal T}} = \frac{1}{N}\|\psi*X\|_{1,{\cal T}}
\]
Then the empirical average can be represented as sum of shifted and flipped inner products.
\[
\frac{1}{N}\|\psi*X\|_{1,{\cal T}}  = \frac{1}{N}\sum_{|i|\in {\cal T}} |[\psi*X]_i| = \frac{1}{N}\sum_{|i|\in {\cal T}}|\psi^T X^{(i)}|
\]
We define 
\[
p_N = \frac{1}{N}\sum_{i\in {\cal T}} I_i
\]
The population average is being calculated,  as
\[
{\mathbb E}  \|\psi*X \|_1:= {\mathbb E}|C_i(\psi)^T X| = {\mathbb E}|\tilde{\psi}^T X|
\]
We know the mean of $p_N$ is $p$, the variance of $p_N$ as
\[
{\mathbb E}[p_N-p]^2=\frac{1}{N}p(1-p)
\]






 
 
 
 \paragraph{Finite sample directional derivative}
 \begin{eqnarray*}
& & \lim_{t\rightarrow 0^+}\frac{1}{t}[(\hat{{\mathbb E}}  \| (\psi+t\beta)*X \|_1 - \hat{{\mathbb E}}  \| \psi*X \|_1)
\\
  &=& 
 \langle  \beta, {\frac{1}{N}\sum_{|i|\in {\cal T}} \nabla_{\psi}L_i}|_{\psi} \rangle \\
 &=& 
 \frac{1}{N}\sum_{|i|\in {\cal T}}  \mbox{sign}(\psi^T X^{(i)})X^{(i),T}\beta
 \end{eqnarray*}
Now to study the finite sample convex problem, we need to calculate the directional derivative at $ \psi = e_0$.
Then
\begin{theorem}
For any $X$ sequence with support  $J_N$,
\begin{eqnarray*}
& & \lim_{t\rightarrow 0^+}\frac{1}{t}[(\hat{{\mathbb E}}  \| (e_0+t\beta)*X \|_1 - \hat{{\mathbb E}}  \| e_0*X \|_1)
\\
  &=& 
 \langle  \beta, {\frac{1}{N}\sum_{|i|\in {\cal T}} \nabla_{\psi}L_i}|_{e_0} \rangle \\
 &=& 
 \frac{1}{N}\sum_{|i|\in {\cal T}}  \mbox{sign}( X_{-i})X^{(i),T}\beta
 \\
  &=& 
 \frac{1}{N} \left(\|X\|_{1,J_N} \beta_0  +
 \|X*\beta_{(0)}\|_{1,{\cal T}-J_N}+ 
\sum_{i\in J_N} (1_{X_{-i}> 0}-1_{X_{-i}< 0})  \cdot X^{(i),T}\beta_{(0)} \right) 
 \end{eqnarray*}
 
\end{theorem}
\begin{proof}
 \begin{eqnarray*}
& & \lim_{t\rightarrow 0^+}\frac{1}{t}[(\hat{{\mathbb E}}  \| (e_0+t\beta)*X \|_1 - \hat{{\mathbb E}}  \| e_0*X \|_1)
\\
  &=& 
 \langle  \beta, {\frac{1}{N}\sum_{|i|\in {\cal T}} \nabla_{\psi}L_i}|_{e_0} \rangle \\
 &=& 
 \frac{1}{N}\sum_{|i|\in {\cal T}}  \mbox{sign}( X_0^{(i)})X^{(i),T}\beta
 \\
 &=& 
 \frac{1}{N}\sum_{|i|\in {\cal T}}  \mbox{sign}( X_{-i})X^{(i),T}\beta
 \\
 &=& 
\frac{1}{N}\sum_{|i|\in {\cal T}} 1_{X_{-i}> 0}  \cdot X^{(i),T}\beta  +
\frac{1}{N}\sum_{|i|\in {\cal T}} 1_{X_{-i}< 0}  \cdot (-X^{(i),T}\beta)  +
\frac{1}{N}\sum_{|i|\in {\cal T}} 1_{X_{-i}= 0} \cdot |X^{(i),T}\beta|
 \\
 &=& 
\frac{1}{N}\sum_{|i|\in {\cal T}} 1_{X_{-i}> 0}  \cdot X^{(i),T}\beta  +
\frac{1}{N}\sum_{|i|\in {\cal T}} 1_{X_{-i}< 0}  \cdot (-X^{(i),T}\beta)  +
\frac{1}{N}\sum_{|i|\in {\cal T}}  |X^{(i),T}\beta_{(0)}|
 \\
 &=& 
\frac{1}{N}\sum_{|i|\in {\cal T}} 1_{X_{-i}> 0}  \cdot X_{-i}\beta_0  +
\frac{1}{N}\sum_{|i|\in {\cal T}} 1_{X_{-i}< 0}  \cdot (-X_{-i}\beta_0)  + 
\frac{1}{N}  \|X*\beta_{(0)}\|_{1,{\cal T}-J_N}+ 
\\
& &
\frac{1}{N}\sum_{|i|\in {\cal T}} 1_{X_{-i}> 0}  \cdot X^{(i),T}\beta_{(0)}  +
\frac{1}{N}\sum_{|i|\in {\cal T}} 1_{X_{-i}< 0}  \cdot (-X^{(i),T}\beta_{(0)})  
 \\
  &=& 
\frac{1}{N}\sum_{|i|\in {\cal T}} 1_{X_{-i}\neq 0}  \cdot |X_{-i}|\beta_0  +
\frac{1}{N}  \|X*\beta_{(0)}\|_{1,{\cal T}-J_N}+ 
\frac{1}{N}\sum_{i\in J_N} (1_{X_{-i}> 0}-1_{X_{-i}< 0})  \cdot X^{(i),T}\beta_{(0)}   
 \\
  &=& 
 \frac{1}{N} \left(\|X\|_{1,J_N} \beta_0  +
 \|X*\beta_{(0)}\|_{1,{\cal T}-J_N}+ 
\sum_{i\in J_N} (1_{X_{-i}> 0}-1_{X_{-i}< 0})  \cdot X^{(i),T}\beta_{(0)} \right) 
 \end{eqnarray*}
\end{proof}
\begin{coro}
For Gaussian IID sequence $X$, 
 the random variable $ \frac{\|X\|_{1,J_N}}{N} = p_N   ( \frac{\|X\|_{1,J_N}}{|J_N|})$ has mean $p\sqrt{\frac{2}{\pi}}$. 
 And the random expression
\begin{eqnarray*}
& &  \frac{1}{N} \left(\|X\|_{1,J_N} \beta_0  +
 \|X*\beta_{(0)}\|_{1,{\cal T}-J_N} \right) 
\\
  &=& 
p_N   ( \frac{\|X\|_{1,J_N}}{|J_N|}) \beta_0  +
(1-p_N ) \frac{1}{|{\cal T}-J_N|}  \|X*\beta_{(0)}\|_{1,{\cal T}-J_N}
 \end{eqnarray*}
 has mean $\sqrt{\frac{2}{\pi}}(p \cdot \beta_0 + (1-p) \cdot {\mathbb E}_{I}[ \|\beta_{I}\|_2 | 0 \not \in I ])$.
\end{coro}
\begin{proof}
 \begin{eqnarray*}
& & 
\frac{1}{N} \left(\|X\|_{1,J_N} \beta_0  +
 \|X*\beta_{(0)}\|_{1,{\cal T}-J_N} \right) 
 \\
  &=& 
p_N ( \frac{\|X\|_{1,J_N}}{|J_N|})\beta_0  +
(1-p_N ) \frac{1}{|{\cal T}-J_N|}  \|X*\beta_{(0)}\|_{1,{\cal T}-J_N}
 \end{eqnarray*}
where $p_N = \frac{|J_N|}{N} = \frac{1}{N}\sum_{|i|\in {\cal T}}  1_{X_{-i}\neq 0}$ has mean $p$ and variance $\frac{1}{N} p(1-p)$.
And notice that for the Gaussian IID case 
\[
{\mathbb E} [ p_N\frac{\|X\|_{1,J_N}}{|J_N|}] = {\mathbb E}[p_N|X_{0}|\mid X_{0}\neq 0]  = p\sqrt{\frac{2}{\pi}},
\]
\[
{\mathbb E} [(1-p_N ) \frac{1}{|{\cal T}-J_N|}  \|X*\beta_{(0)}\|_{1,{\cal T}-J_N} ] = \sqrt{\frac{2}{\pi}}{\mathbb E}_{I}[ \|\beta_{I}\|_2 | 0 \not \in I ]).
\]
\end{proof}

\begin{coro}
 For symmetric IID sequence $X$, 
  \[
  \frac{1}{N}\sum_{i\in J_N} (1_{X_{-i}> 0}-1_{X_{-i}< 0})  \cdot X^{(i),T}\beta_{(0)}  = \frac{1}{N}\sum_{i\in J_N} (1_{X_{-i}> 0}-1_{X_{-i}< 0})  \cdot (\sum_{j\in J_N-\{i\} } X_j \beta_{i-j} ) 
  \]
  where 
  $\sum_{j\in J_N-\{i\} } X_j \beta_{i-j} $ is independent of $X_i$, therefore independent of $(1_{X_{-i}> 0}-1_{X_{-i}< 0}) $.
  Specifically, it has mean $0$
   \[
 {\mathbb E} \frac{1}{N}\sum_{i\in J_N} (1_{X_{-i}> 0}-1_{X_{-i}< 0})  \cdot X^{(i),T}\beta_{(0)}  = 0
  \]
  for $q\leq 2$,
    \[
  \frac{1}{N}{\mathbb E}|\sum_{i\in J_N} (1_{X_{-i}> 0}-1_{X_{-i}< 0})  \cdot X^{(i),T}\beta_{(0)} |^q \leq 
  \frac{1}{N} {\mathbb E}[]|\sum_{i\in J_N}|1_{X_{-i}> 0}-1_{X_{-i}< 0}||^q\cdot  \max_j |X^{(j),T}\beta_{(0)} |^q]
  \]
  
\end{coro}



 \paragraph{Concerntration of $\|\psi*X\|_{1}$ }
The next step is to provide an uniform bound of the finite sample derivation of $\|X\|_{1,J_N}$ and 
$\|X*\beta_{(0)}\|_{1,{\cal T}-J_N}$.

In general, for any random subset $I$ of ${\cal T}$, we study the uniform concentration of 
\[
\|\psi*X\|_{1,I} 
\]
for $\psi = a*w \in a*B_1(V_k),$ by providing moments and tail bound on
\[
U_I:= \sup_{\psi \in a*B_1(V_k)}
| \|\psi*X\|_{1,I}    - {\mathbb E}  \|\psi*X\|_{1,I}  |
\]

For clarity of the argument, we only state the concentration for $\|\psi*X\|_{1,{\cal T}}.$ Also the result about ${\cal T}$ would easy carry to a random subset $I$.

Let the minimal and maximal eigenvalue of the (convolutional) linear operator $a*$ be
\[
\lambda_{m}(a) := \inf_{y: \|y\|_2\leq 1} \frac{\|a*y\|_2}{\|y\|_2}
\]
\[
\lambda_{M}(a) := \sup_{y: \|y\|_2\leq 1} \frac{\|a*y\|_2}{\|y\|_2}
\]
\begin{lemma}
Using circluar embedding of Topelitz matrix, 
\[
\lambda_{M}(a) \leq \max| \hat{a}|
\]
\end{lemma}

Following the idea of lemma~$4$ of the paper of Adamczak on sparse dictionary learning\cite{adamczak2016note} and lemma~$16$ of the paper \cite{spielman2012exact}, we derive a similar bound for convolutional case.
\begin{lemma}
In the following, we define 
\[
Z :=  \frac{1}{N}\|w*a*X\|_{1,{\cal T}}
\]
We find the lower bound of ${\mathbb E} Z$ as a function of the $1-$norm  of $w$:
\[
{\mathbb E} Z  \geq \frac{1}{\sqrt{2}\pi}\lambda_{m}(a)\sqrt{\frac{p}{k}} \|w\|_{1} 
\]
For convenience, we define
\[
\mu_{\mbox{min}}:=\frac{1}{\sqrt{2}\pi}\lambda_{m}(a)\sqrt{\frac{p}{k}} \|w\|_{1}
\]

\end{lemma}
\begin{proof}
WOLG, we rescale $w$ so that $\|w\|_1 =1$, now
\begin{eqnarray*}
{\mathbb E}_X Z 
&=& \frac{1}{N}\sum_{j\in [-T,T] }{\mathbb E}_X |\sum_{i=1}^k w_i (a*X)_{j-i}|\\
&\geq& \inf_{\|w\|_{1} = 1}\frac{1}{N}\sum_{j\in [-T,T] }{\mathbb E}_X |\sum_{i=1}^k w_i (a*X)_{j-i}|
\end{eqnarray*}
By symmetry, we know the minimizer is $w = (\frac{1}{k}, \ldots,\frac{1}{k}),$
then we use the symmetrization trick to insert a sequence of $k$ IID Rademacher ($\pm 1$)  random variables 
$\varepsilon_1, \ldots, \varepsilon_k$ in the second inequality, and then apply Khintchine inequality as the third inequality,
\begin{eqnarray*}
{\mathbb E}_X Z 
&\geq&  \frac{1}{k} \frac{1}{N}\sum_{j\in [-T,T] }{\mathbb E}_X |\sum_{i=1}^k (a*X)_{j-i}|\\
&\geq& \frac{1}{2} \frac{1}{k} \frac{1}{N}\sum_{j\in [-T,T] }{\mathbb E}_X |\sum_{i=1}^k \varepsilon_i (a*X)_{j-i}|\\
&\geq&  \frac{1}{2} \frac{1}{k} \frac{1}{N}\sum_{j\in [-T,T] }\frac{1}{\sqrt{2}}{\mathbb E}_X \sqrt{ \sum_{i=1}^k  (a*X)^2_{j-i}}\\
&\geq&  \frac{1}{2} \frac{1}{k} \frac{1}{N}\sum_{j\in [-T,T] }\lambda_{m}(a)\frac{1}{\sqrt{2}}{\mathbb E}_X \sqrt{ \sum_{i=1}^k  X^2_{j-i}}\\
&\geq& \frac{1}{2}  \frac{1}{k} \frac{1}{\sqrt{2}}\frac{1}{N}\sum_{j\in [-T,T] }\lambda_{m}(a)\frac{2}{\pi}\sqrt{pk}\\
&=&  \frac{1}{\sqrt{2}\pi}\lambda_{m}(a)\sqrt{\frac{p}{k}}\\
\end{eqnarray*}
\end{proof}


Now we define the intersection of $1-$norm ball of the $k-$dimensional subspace $V_k$ as $B_1(V_k)$, then we consider $w\in B_1(V_k).$ 
\begin{lemma}
Hence we consider $\psi = a*w \in a*B_1(V_k),$ and we define
\[
W:= \sup_{\psi \in a*B_1(V_k)}
|\frac{1}{N}\sum_{j\in [-T,T] } (|(\psi*X)_{j}| - {\mathbb E} |(\psi*X)_{j}|) |
\]

For all $q\geq \max(2, \log(k)),$ there exists constant $C$,
\[
{\mathbb E} W^q\leq 2^{2q}(\frac{\lambda_{M}(a)}{N})^q   k C^q(\sqrt{Npq}+q)^q
\]
so
\begin{eqnarray*}
 \|W\|_q \leq \frac{4eC}{N} \lambda_{M}(a)  
 (\sqrt{Np(q+\log(k))}+q +\log(k))
\end{eqnarray*}
and
\begin{eqnarray*}
P( W > \frac{4eC\lambda_{M}(a) }{N}  
 (\sqrt{Np\log(\frac{k}{\delta})}+\log(\frac{k}{\delta})))\leq \delta
\end{eqnarray*}
\end{lemma}

\begin{proof}
Let $\varepsilon_1, \ldots, \varepsilon_N$ be a sequence of IID Rademacher ($\pm 1$)  random variables independent of $X$,  By the symmetrization inequality, see e.g. lemma~$6.3$ of book \cite{ledoux2013probability}, we have the first inequality. Then since the function $t \rightarrow |t| $ is a contraction, an application of Talagrand’s contraction principle (see lemma~$8$ of \cite{adamczak2016note}) with $F (x) = |x|^q$ 
conditionally on $X$ gives the second inequality.
\begin{eqnarray*}
{\mathbb E} W^q 
&\leq& 2^q {\mathbb E} \sup_{\psi \in a*B_1(V_k)} 
|\frac{1}{N}\sum_{j\in [-T,T] } \varepsilon_j |(\psi*X)_{j}| |^q\\
&\leq&
2^{2q}{\mathbb E} \sup_{\psi \in a*B_1(V_k)} |\frac{1}{N}\sum_{j\in [-T,T] } \varepsilon_j (\psi*X)_{j} |^q\\
&=&
2^{2q}\frac{1}{N^q}{\mathbb E} \sup_{\psi \in a*B_1(V_k)} |\sum_{i\in [-T,T] }\psi_{i}  \sum_{j\in [-T,T] } \varepsilon_j X_{j-i} |^q\\
&=&
2^{2q}\frac{1}{N^q}{\mathbb E} \sup_{w \in B_1(V_k)} |\sum_{i\in [-T,T] }(w*a)_{i}  \sum_{j\in [-T,T] } \varepsilon_j X_{j-i} |^q\\
&\leq&
2^{2q}\frac{1}{N^q}{\mathbb E} \max_{s\in [k]}|\sum_{i\in [-T,T] }a_{i-s}( \sum_{j\in [-T,T] } \varepsilon_j X_{j-i}) |^q\\
&\leq&
2^{2q}(\frac{\lambda_{M}(a)}{N})^q   \sum_{s\in [k] }{\mathbb E}| \sum_{j\in [-T,T] } \varepsilon_j X_{j-s} |^q
\end{eqnarray*}
Now by the moment version of Bernstein’s inequality (see lemma~$7$, equation $(18)$ of\cite{adamczak2016note}), we know that there exists a universal constant $C$,
\begin{eqnarray*}
{\mathbb E}| \sum_{j\in [-T,T] } \varepsilon_j X_{j-i} |^q \leq C^q(\sqrt{Npq}+q)^q
\end{eqnarray*}
Therefore, 
\begin{eqnarray*}
{\mathbb E} W^q &\leq& 2^{2q}(\frac{\lambda_{M}(a)}{N})^q    
\sum_{s\in [k] }{\mathbb E}| \sum_{j\in [-T,T] } \varepsilon_j X_{j-s} |^q \\
&\leq& 2^{2q}(\frac{\lambda_{M}(a)}{N})^q   k C^q(\sqrt{Npq}+q)^q
\end{eqnarray*}
Then
\begin{eqnarray*}
 \|W\|_q \leq \frac{4eC}{N} \lambda_{M}(a)  
 (\sqrt{Np(q+\log(k))}+q +\log(k))
\end{eqnarray*}
therefore, we could get the tail bound using the Chebyshev inequality for the moments,
\begin{eqnarray*}
P( W > \frac{4eC\lambda_{M}(a) }{N}  
 (\sqrt{Np(q+\log(k))}+q +\log(k)))\leq e^{-q}
\end{eqnarray*}
We set $q = \log(\frac{1}{\delta})$, then we will get an upper bound for $W$ which is satisfied with probability at least $1-\delta$:
\begin{eqnarray*}
P( W > \frac{4eC\lambda_{M}(a) }{N}  
 (\sqrt{Np\log(\frac{k}{\delta})}+\log(\frac{k}{\delta})))\leq \delta
\end{eqnarray*}
\end{proof}

\begin{lemma}
Let the condition number of the linear operator $a*$ be
\[
\kappa_a := \frac{\lambda_M(a)}{\lambda_m(a)}.
\]
If
\[
N\geq k\log(\frac{k}{\delta}) (2^7 \pi e^2 \frac{C^2}{\epsilon^2} \kappa_a^2 )
\]
and  $p\geq \frac{1}{N}$, 
then
\begin{eqnarray*}
P( W > \epsilon \mu_{\mbox{min}})\leq \delta
\end{eqnarray*}
\end{lemma}
\begin{proof}
If
\[
N\geq k\log(\frac{k}{\delta}) (2^7 \pi e^2 \frac{C^2}{\epsilon^2} \kappa_a^2 )
\]
then 
\[
\frac{4eC\lambda_{M}(a) }{N}  
 (\sqrt{Np\log(\frac{k}{\delta})} \leq \frac{1}{2}\epsilon (\frac{1}{\sqrt{2}\pi}\lambda_{m}(a)\sqrt{\frac{p}{k}}) \leq \frac{1}{2}\epsilon \mu_{\mbox{min}}
\]
additionally, if $p\geq \frac{1}{N}$, 
then a weaker inequality is also true:
\[
\frac{4eC\lambda_{M}(a) }{N}  
 \log(\frac{k}{\delta}) \leq \frac{1}{2}\epsilon (\frac{1}{\sqrt{2}\pi}\lambda_{m}(a)\sqrt{\frac{p}{k}}) \leq \frac{1}{2}\epsilon \mu_{\mbox{min}}
\]
therefore, combining the two parts and the previous inequality,
\begin{eqnarray*}
P( W > \frac{4eC\lambda_{M}(a) }{N}  
 (\sqrt{Np\log(\frac{k}{\delta})}+\log(\frac{k}{\delta})))\leq \delta
\end{eqnarray*}
we have
\begin{eqnarray*}
P( W > \epsilon \mu_{\mbox{min}})\leq \delta
\end{eqnarray*}
\end{proof}


\begin{theorem}
Let the solution of finite sample convex optimization be $w^\star$, 
for small $\epsilon>0$ and concentration level $\delta>0$, 
there exists universal constant $C$, if there are 
\[
N \geq k\log(\frac{k}{\delta}) ( \frac{C\kappa_a}{\epsilon}  )^2
\]
and  $ \frac{1}{N}\leq p< p^\star(\epsilon)$, we have 
\begin{eqnarray*}
P( W > \epsilon \mu_{\mbox{min}})\leq \delta.
\end{eqnarray*}

  \end{theorem}
\begin{proof}
From previous lemma, we know that with the conditions, 
\begin{eqnarray*}
P( W > \epsilon \mu_{\mbox{min}})\leq \delta.
\end{eqnarray*}
Therefore, with probability at least $1-\delta$,
\begin{eqnarray*}
W = \sup_{\psi \in a*B_1(V_k)}
|\frac{1}{N}\sum_{j\in [-T,T] } (|(\psi*X)_{j}| - {\mathbb E} |(\psi*X)_{j}|) | \leq \epsilon \mu_{\mbox{min}}
\end{eqnarray*}
When this is true, for all $\psi \in a*B_1(V_k)$, we have a uniform bound 
\[
 {\mathbb E} |(\psi*X)_{j}|-\epsilon\mu_{\mbox{min}} \leq \frac{1}{N}\sum_{j\in [-T,T] } (|(\psi*X)_{j}| \leq {\mathbb E} |(\psi*X)_{j}|+\epsilon\mu_{\mbox{min}}
\]
\end{proof}


\begin{coro}
For $\psi = a*w \in a*B_1(V_k),$ 
\[
U_I:= \sup_{\psi \in a*B_1(V_k)}
| \|\psi*X\|_{1,I}    - {\mathbb E}  \|\psi*X\|_{1,I}  |
\]
For all $q\geq \max(2, \log(k)),$ there exists constant $C$,
\[
{\mathbb E} U_I^q\leq 2^{2q}(\lambda_{M}(a))^q   k C^q(\sqrt{Npq}+q)^q
\]
so
\begin{eqnarray*}
 \|U_I\|_q \leq 4eC \lambda_{M}(a)  
 (\sqrt{Np(q+\log(k))}+q +\log(k))
\end{eqnarray*}
and
\begin{eqnarray*}
P( U_I > 4eC\lambda_{M}(a)  
 (\sqrt{Np\log(\frac{k}{\delta})}+\log(\frac{k}{\delta})))\leq \delta
\end{eqnarray*}
\end{coro}

\paragraph{Uniform bound of directional derivative}
We  define the uniform bound of directional derivative
\[
W_D  =\sup_{\|\beta\|_2=1, u^T \beta =0, \beta +e_0 \in a*B_1(V_k)}|\langle  \beta, {\frac{1}{N}\sum_{|i|\in {\cal T}} \nabla_{\phi}L_i}|_{\phi = 0} \rangle  -   \langle  \beta, {\mathbb E} \nabla_{\phi}L|_{\phi = 0} \rangle  | 
\]
Notice that
\begin{eqnarray*}
& &\langle  \beta, {\frac{1}{N}\sum_{|i|\in {\cal T}} \nabla_{\phi}L_i}|_{\phi = 0} \rangle  -   \langle  \beta, {\mathbb E} \nabla_{\phi}L|_{\phi = 0} \rangle  
\\
&=& 
\left[\frac{1}{N} \left(\|X\|_{1,J_N} \beta_0  +
 \|X*\beta_{(0)}\|_{1,{\cal T}-J_N}+ 
\sum_{i\in J_N} (1_{X_{-i}> 0}-1_{X_{-i}< 0})  \cdot X^{(i),T}\beta_{(0)}\right)\right]-
\\
& &
\left[\sqrt{\frac{2}{\pi}}(p \cdot \beta_0 + (1-p) \cdot {\mathbb E}_{I}[ \|\beta_{I}\|_2 | 0 \not \in I ])\right]
\end{eqnarray*}
From previous uniform bound, with high probability, we have the following three uniform bounds on
\[
|\frac{1}{N}\sum_{i\in J_N} (1_{X_{-i}> 0}-1_{X_{-i}< 0})  \cdot X^{(i),T}\beta_{(0)}|
\]
\[
|(\frac{1}{N} \|X\|_{1,J_N}-\sqrt{\frac{2}{\pi}}p)|
\]
\[
|\frac{1}{N}\|X*\beta_{(0)}\|_{1,{\cal T}-J_N}-(1-p) \cdot {\mathbb E}_{I}[ \|\beta_{I}\|_2 | 0 \not \in I ]|
\]
If $p<p^\star(u, \epsilon)$, we know from population result that
\[
\langle  \beta, {\mathbb E} \nabla_{\phi}L|_{\phi = 0} \rangle >\epsilon.
\]
If with high probability,  
\[
W_D  \leq \epsilon/2
\]
then using the convexity argument, 
for any direction $\beta \in V,$ the directional derivative at $\phi = 0$
\[
\langle  \beta, {\frac{1}{N}\sum_{|i|\in {\cal T}} \nabla_{\phi}L_i}|_{\phi = 0} \rangle  >0,
\]
then $a*w^\star-e_0=0$. 





















 








                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           


 
 

 


  







\subsection{$E{| \psi^T X| }$ for IID Bernoulli Gaussian $X$ }
we study in detail for the quantity  
\[
E{| \psi^T X| } = E{|\sum_{i} \psi_i X_i| }.
\]
We know when $X_i$ are unit variance,
\[
   E (\sum_i \psi_i X_i)^2 =  \sum_i   \psi_i^2 E(X_i)^2 = \|a\|_2^2.
\]
The ratio indicate the sparsity level of random variable $Z = \sum_i \psi_i X_i$
\[
\frac{ E{|\sum_{i} \psi_i X_i| }}{\sqrt{E (\sum_i \psi_i X_i)^2}} = \frac{ E{|\sum_{i} \psi_i X_i| }}{\|a\|_2} 
\]
Now, we consider some settings where exact calculation of $E{|\sum_{i} \psi_i X_i| }$ are possible.

\paragraph{Exact calculation for Bernoulli Gaussian}
Let $X_i = B_i G_i$ be IID Bernoulli Gaussian, where $B_i$ is sampled from Bernoulli with parameter $p$, $G_i$ is sampled from $N(0,1)$. Let $I$ be the support where $B_i =1$.
\[
E_X{|\sum_{i} \psi_i X_i| }  
= E_B E_G {|\sum_{i} \psi_i B_i G_i| }
= E_I E_G {|\sum_{i\in I} \psi_i G_i| }
\]
Leveraging the invariance property for linear transform of Gaussian, we get
\[
E_G {|\sum_{i\in I} \psi_i G_i| } = 
\sqrt{\frac{2}{\pi}} \cdot\|\psi_I\|_2.
\]
then
\[
E_X{|\sum_{i} \psi_i X_i| }  
= \sqrt{\frac{2}{\pi}} \cdot E_I \|\psi_I\|_2
= \sqrt{\frac{2}{\pi}} \cdot (1-(1-p)^{\|a\|_0})^{1/2}   \cdot  \|a\|_2.
\]
Here the last equality use the lemma below, and we define that $\|a\|_0= \#\{ t : \psi_t \neq 0\}$.
\begin{lemma} (Expectation of $2-$norm over Bernoulli support)
\label{lem:BLemma}
\begin{equation}
E_I \|\psi_I\|_2 = (1-(1-p)^{\|a\|_0})^{1/2}   \cdot  \|a\|_2
\end{equation}
\end{lemma}
\begin{proof}[Proof of Lemma \ref{lem:BLemma}]
We have the power series representation $\sqrt{x} = \sum_k c_k x^k$ valid on $(1/A,A)$
for large $A \gg 1$. 
Let $N = |{\cal T}|$. Fix $A > 1/{\varepsilon}^2$; then every sum $ \sum_{t \in I} \psi_t^2 \in (1/A,A)$;
using this:
\begin{eqnarray*}
 \sum_{I \subset {\cal T}} \pi(I)   ( \sum_{t \in I} \psi_t^2)^{1/2}  &=&  
 \sum_{I \subset {\cal T}} \pi(I)   \sum_k  c_k \cdot ( \sum_{t \in I} \psi_t^2  )^k \\
 &=& \sum_{I \subset {\cal T}} \pi(I)   \sum_k  c_k \sum_{t_1,\dots, t_k \in I} \prod_{i=1}^k \psi_{t_i}^2  \\
 &=&   \sum_k  c_k \sum_{I \subset {\cal T}} \pi(I)   \sum_{t_1,\dots, t_k \in I} \prod_{i=1}^k \psi_{t_i}^2  \\
 \end{eqnarray*}
 Hence from $ \pi(I) = (p)^{|I|} (1-p)^{|{\cal T} \Delta I|}$
 \begin{eqnarray*}
  \sum_{I \subset {\cal T}} \pi(I)   ( \sum_{t \in I} \psi_t^2)^{1/2}  
 &=&  \sum_k c_k   \sum_{n=1}^{N}  \sum_{|I| = n} (p)^n (1-p)^{N-n}  \cdot  \sum_{t_1,\dots, t_k \in I} \prod_{i=1}^k \psi_{t_i}^2  \\
&=&  \sum_k c_k   \sum_{n=1}^{N}  (p)^n (1-p)^{N-n}  \cdot \sum_{|I| = n} \sum_{t_1,\dots, t_k \in I} \prod_{i=1}^k \psi_{t_i}^2 
 \end{eqnarray*}
 Since the inner two sums  cover $ {N \choose n} n^k$ terms in a uniform way,
 we can write
 \begin{eqnarray*}
  \sum_{I \subset {\cal T}} \pi(I)   ( \sum_{t \in I} \psi_t^2)^{1/2}  
&=&  \sum_k c_k   \sum_{n=1}^{N}  (p)^n (1-p)^{N-n}  \cdot \sum_{|I| = n} \sum_{t_1,\dots, t_k \in I} \prod_{i=1}^k \psi_{t_i}^2  \\
 &=&  \sum_k c_k   \sum_{n=1}^{N}  (p)^n (1-p)^{N-n}  \cdot {N \choose n}  \cdot Ave_{|I| = n}  \cdot {n^k} \cdot  Ave_{t_1,\dots, t_k \in I} \; \prod_{i=1}^k \psi_{t_i}^2  \\
&=&  \sum_k c_k   \sum_{n=1}^{N}  (p)^n (1-p)^{N-n}  \cdot {N \choose n}  \cdot {n^k} \cdot Ave_{t_1,\dots, t_k \in {\cal T}} \; \prod_{i=1}^k \psi_{t_i}^2  \\
 &=&  \sum_k c_k   \cdot \left( \sum_{n=1}^{N}  (p)^n (1-p)^{N-n}  \cdot {N \choose n} \right)  \cdot ( \sum_{t \in {\cal T}} \psi_t^2  )^k \\
 \end{eqnarray*}
Now use the identity $1 = \sum_{n=0}^{N}  (p)^n (1-p)^{N-n}  \cdot {N \choose n}$  for the complete binomial sum, 
hence the incomplete sum $\sum_{n=1}^{N}  (p)^n (1-p)^{N-n}  \cdot {N \choose n}= 1-(1-p)^N$. We get
 \begin{eqnarray*}
  \sum_{I \subset {\cal T}} \pi(I)   ( \sum_{t \in I} \psi_t^2)^{1/2}  
 &=&  \sum_k c_k   \cdot \left( 1-(1-p)^N \right)  \cdot ( \sum_{t \in {\cal T}} \psi_t^2  )^k \\
 &=& \left( 1-(1-p)^N \right) \cdot \sum_k c_k  \left ( \sum_{t \in {\cal T}} \psi_t^2  \right )^k \\
 &=&  \left( 1-(1-p)^N \right) \cdot  \sqrt{\sum_{t \in {\cal T}} \psi_t^2 } .
\end{eqnarray*}
\end{proof}
\section{Problem setting for blind deconvolution}
\paragraph{Doubly infinite sequence setting}
First, we study the population objective under doubly infinite sequence setting:
\begin{itemize}
\item
Let $(X_t)_{t \in {\mathbb Z}}$ be IID sampled from  $pN(0,1) + (1-p) \delta_0$.
 \item
The filter $a\in l_1({\mathbb Z}), (\mathcal{F} a)(\omega) \neq 0, \forall \omega\in T.$
\item
Let $(Y_t)$, defined by $Y_t \equiv \sum_{h\in {\mathbb Z}} a_h X_{t-h}$, be a linear process, i.e.
\[
Y= a* X,
\]

\subsection{Inverse filter}
\paragraph{Fourier transform and inverse filter}
The discrete-time Fourier transform $\mathcal{F}$ is defined by $\mathcal{F} a = \hat{a}$ where
\[
\hat{a}(\omega) : = \sum_{n=-\infty}^\infty a_n\, e^{2\pi i n \omega}, \quad  \omega\in T = [-\frac{1}{2}, \frac{1}{2}];
\]
the inverse Fourier transform $\mathcal{F}^{-1}$ is defined by
\[
(\mathcal{F}^{-1} f)(n) =  \int_{-\frac{1}{2}}^\frac{1}{2} f(\omega) e^{-2\pi i n \omega}  d\omega, \quad n\in {\mathbb Z}.
\]
Now, our condition on the filter $a$ is:
\[a\in l_1({\mathbb Z}) ,  
(\mathcal{F} a)(\omega) \neq 0, \forall \omega\in T.
\]
In the following theorem, we show that this condition would provide the existence of an inverse filter in  $l_1({\mathbb Z})$.
\begin{theorem}
If $a\in l_1({\mathbb Z}) ,  
(\mathcal{F} a)(\omega) \neq 0, \forall \omega\in T,$ we could define the inverse filter of $a$ as
$$a^{-1} := \mathcal{F}^{-1}(\frac{1}{\mathcal{F} a})$$
so that $a^{-1}*a = e_0.$ Here $e_0$ is the sequence with $1$ at $0$ coordinate and $0$ elsewhere.
From Wiener's lemma on $\mathcal{F} a$,   $a^{-1} \in l_1({\mathbb Z})$ .
\end{theorem}

\begin{lemma}{\textbf{Wiener's lemma:}}
Assume that a function $f$ on unit circle has an absolutely converging Fourier series, and $f(t)\neq 0$ for all $t \in T$, then $1/f$ also has an absolutely convergent Fourier series.
\end{lemma}


\section{Stability for approximation of infinite length inverse filter}

 
 
 

\paragraph{Reduction}
 \begin{lemma} 
 Let $w^\star \in V_{rs, rt} $ be the solution of the convex optimization problem
 \[
 \begin{array}{ll}
  \underset{w}{ \mbox{minimize} }  &  {\mathbb E}|(w*Y)_0|\\
 \mbox{subject to}  
&  w_0 =1, \quad w \in V_{rs, rt}.
\end{array}
  \]
 Then the previous convex optimization problem can be reduced to 
  \[
 \begin{array}{ll}
  \underset{w}{ \mbox{minimize} }  & B(p, \phi) \|\phi \|_2\\
 \mbox{subject to}  
&  \phi = a*w -e_0, \quad w_0 =1, \quad w \in V_{rs, rt}.
\end{array}
  \]
  where
  \[
B(p, \phi) := \frac{{\mathbb E}_I  \| (e_0+\phi)_I \|_2 - {\mathbb E}_I  \| (e_0)_I \|_2}{\|\phi\|_2} 
\]
 \end{lemma} 
  
  
\begin{proof}
Following the previous phase transition analysis, 
let the directional finite difference of the objective at $e_0$ be $D(w) = |(w*Y)_0| - |X_0|$, $I$ be the support of $\tilde{X}$, let $\phi = w*a-e_0$,  then we have 
\[
 \begin{array}{ll}
{\mathbb E} D(w) &= \sqrt{\frac{2}{\pi}}{\mathbb E}_I [\|(w*a)_I\|_2 - \|(e_0)_I\|_2]\\
&= \sqrt{\frac{2}{\pi}}{\mathbb E}_I [\|(e_0+\phi)_I\|_2 - \|(e_0)_I\|_2]\\
&= \sqrt{\frac{2}{\pi}}\|{\phi}\|_2 B(p, \phi)  \\
\end{array}
  \]
The last inequality comes from the bi-Lipschitz property of ${\mathbb E}_I \|(e_0+\phi)_I\|_2$ around $e_0$. 








Now using the optimality of $w^\star$ in $V_{rs, rt}$, for all $w in V_{s,t}, w\neq w^\star$,
we have
 \[
 \begin{array}{ll}
{\mathbb E} D(w^\star) \leq {\mathbb E} D(w)
\end{array}
  \]
We could directly reduce the problem to 
  \[
 \begin{array}{ll}
B(p, \phi^\star) \|{\phi^\star}\|_2 \leq B(p, \phi)\|{\phi}\|_2 
\end{array}
  \]
\end{proof}   
 


\begin{coro}
If there exists a $\phi$ with $\|{\phi}\|_2<1$ such that \[
 \begin{array}{ll}
B(p, \phi^\star) \|{\phi^\star}\|_2 \leq B(p, \phi)\|{\phi}\|_2
\end{array}
  \]
 for  $\epsilon> 0$ when $p\leq p^\star(\epsilon)$, then 
  \[
  B(p, \phi) \geq \epsilon.
  \]
  for all $\phi$, 
  then
    \[
 \begin{array}{ll}
\|{\phi^\star}\|_2 \leq \frac{(1-p)\sqrt{p}  + 
p\Big(1  +
  \frac{\|{\phi}\|_2}{2} 
 +
 p\frac{\|{\phi}\|_2}{2}   \Big)}{\epsilon}\|{\phi}\|_2
\end{array}
  \]
  is also bounded.
\end{coro}
\begin{proof}
From
 \[
 \begin{array}{ll}
B(p, \phi^\star) \|{\phi^\star}\|_2 \leq B(p, \phi)\|{\phi}\|_2
\end{array}
  \]
  We have
  \[
 \begin{array}{ll}
\|{\phi^\star}\|_2 
&\leq \frac{B(p, \phi)}{B(p, \phi^\star)}
\|{\phi}\|_2 \\
&\leq \frac{(1-p)\Big( V_1(\phi_{{(0)}}) \sqrt{1-\phi_0^2}\Big) + 
p\Big(\phi_0  +
  \frac{\|{\phi}\|_2}{2} \phi_0^2 
 +
 p\frac{\|{\phi}\|_2}{2}  (1-\phi_0^2) \Big)}{(1-p)\Big( V_1(\phi^\star_{{(0)}}) \sqrt{1-(\phi^\star_0)^2}\Big) + 
p\Big(\frac{1}{\|{\phi^\star}\|_2} (\left| 1 + \|{\phi^\star}\|_2 \phi^\star_0 \right|-1 \Big)}\|{\phi}\|_2 \\
&\leq \frac{(1-p)\sqrt{p}  + 
p\Big(1  +
  \frac{\|{\phi}\|_2}{2} 
 +
 p\frac{\|{\phi}\|_2}{2}   \Big)}{\epsilon}\|{\phi}\|_2
\end{array}
  \]
  \end{proof}
  In the following we constructed a filter to give an upper bound on $\|{\phi}\|_2$. 


\paragraph{Setting for finite length approximation}
Now we consider a setting where there is a kernel whose corresponding inverse kernel has infinite support, and we give a finite-length approximation. 

Within the space $V_{-\infty,\infty}$ of bilaterally infinite real-valued sequences $(h(i))_{i \in {\mathbb Z}}$, consider the affine subspace $V_{N_{-}, N_{+}} = \{ (...,0,0,h_{-N_{-}},\ldots, h_{-1}, 1, h_1, h_2, \ldots, h_{N_{+}},0,0,...)\}$, an $N_{-}+N_{+}$-dimensional subspace of bilateral sequences with support at most $N_{-}+N_{+}+1$.
The coordinate that is fixed to one is 
located at index $i=0$.
Each coordinate is zero outside of a window
of size $N_{-}$ on the left of zero 
and  size $N_{+}$ on the right of zero.
The special sequence $e_0 = (\dots,0,0,1,0,0,\dots)$,
vanishing everywhere except the origin,
belongs to $V_{-\infty,\infty}$ and to every
$V_{N_-,N_+}$.

Let $a \in V_{N_-,N_+}$.
Then $a =  (...,0,0,a_{-N_{-}},\ldots, a_{-1}, 1, a_1, a_2, \ldots, a_{N_{+}},0,0,...)$.
We also write $a = e_0 + a_{(0)}$,
where $a_{(0)} \equiv (1-e_0) \cdot a$ 
denotes the `part of $a$ supported away from location
$i=0$'. We also write $a = e_0 + a_L + a_R$,
where $a_L = (a_L(i))_{i \in {\mathbb Z}} 
= (a(i) 1_{\{i < 0\}})_{i \in {\mathbb Z}}$ 
denotes the `part of $a$ supported to
the left of $i=0$' and 
where $a_R = (a_R(i))_{i \in {\mathbb Z}} 
= (a(i) 1_{\{i > 0\}})_{i \in {\mathbb Z}}$
denotes the `part of $a$
supported to the right of $i=0$'.
Finally, we say that $a \in V_{N_{-}, N_{+}}$ 
is a length $ L= N_{-}+N_{+}+1$ filter.

An interesting, if not implementable, objective
function for the blind deconvolution problem
is to minimize
${\mathbb E} [|(w*a*X)_0| - |X_0|] $,
particularly when combined with constraints 
somehow ensuring ${{\mathbb E}} |(w*a*X)_0|^2 \geq {{\mathbb E}}|X_0|^2$.
Defining $\phi \equiv w*a-e_0$, 
\[
 \begin{array}{lll}
{\mathbb E} [|(w*a*X)_0| - |X_0|] &=&{\mathbb E} [|((e_0 +\phi)*X)_0| - |X_0|].
\end{array}
  \]
  Below we establish the relation
  \[
   \begin{array}{lll}
{\mathbb E} [|((e_0 +\phi)*X)_0| - |X_0|] 
&\approx&  C \|{\phi}\|_2 ; \\
\end{array}
\]
This suggests that we should  seek to
minimize ${\mathbb E} |(w*Y)_0| $,
while constraining $w$ in some way that
ensures ${{\mathbb E}} |(w*a*X)_0|^2 \geq {{\mathbb E}}|X_0|^2$.
This  the deterministic objective as 
$$
d_a(w, a^{-1}) :=\|{\phi}\|_2 = \|a*w -e_0 \|_2 =  \|a*(w -a^{-1}) \|_2.
$$

Therefore, we want to construct a finite length approximation of $a^{-1}$ to bound $d_a(w^\star, a^{-1}) = \|a*w -e_0 \|_2$ for the optimal point $w^\star$.








We construct the finite length approximation filter explicitly by truncation of Z-transform.

Let the Z-transform of $a$ be 
\[
 \begin{array}{ll}
A(z) =  \sum_{i=-s}^t a_{i} z^{-i},
\end{array}
  \]
Then Z-transform of the inverse kernel $a^{-1}$ is $1/A(z)$.






 
\subsection{ Stability for factorized filter}
\subsection{Examples}
In the following, we study the simple case when there is only one root.
Let $s\in (-1,1)$. For $a=(\ldots, 0, 1, -s, 0, \ldots)$, $a^{-1} = (\ldots, 0, 1, s, s^2, s^3, \ldots)$, we could construct 
\[
w^{r,1}:=  (\ldots, 0, 1, s, s^2, s^3, \ldots, s^r)
\]

If we choose this approximation $w^{r,1},$ then
\[
\|a*w^{r,1}-e_0\|_2^2 = \|*_{r+1} (\bar a^{\gamma})\|_2^2 = s^{2(r+1)} 
\]

In the following, we show that this is indeed the best 

For $\gamma\in (0,1]$,
\[
\bar a^{\gamma} :=e_0-\gamma a= (\ldots, 0, 1-\gamma,  \gamma s, 0, \ldots)
\]
\[
 \begin{array}{ll}
w^{r,\gamma} 
&= a^{-1} - a^{-1}*_{r+1} (\bar a^{\gamma})
\end{array}
\]
\[
\begin{array}{ll}
a*(a^{-1}-w^{r,\gamma}) 
&= *_{r+1} (\bar a^{\gamma})\\
&= *_{r+1} (\ldots, 0, 1-\gamma,  \gamma s, 0, \ldots)
\end{array}
\]

\[
\mathcal{F}[(\ldots, 0, 1-\gamma, \gamma s, 0, \ldots)](t)  = (1-\gamma)+   \gamma s  \cdot e^{2\pi i  t}, \quad  t\in T = [\frac{-1}{2}, \frac{1}{2}];
\]

\[
|\mathcal{F}[(\ldots, 0, 1-\gamma, \gamma s, 0, \ldots)] |^2(t) = (1-\gamma)^2+ (\gamma s )^2+ 2 {(1-\gamma)} {\gamma}s \cos{(2\pi  t)} , \quad t\in T = [\frac{-1}{2}, \frac{1}{2}];
\]

\[
 \begin{array}{ll}
\|*_{r+1} (\bar a^{\gamma})\|_2^2 
&= \| (\hat{\bar a}^{\gamma})^{r+1}\|_2^2\\
&=  \int_{\frac{-1}{2}}^\frac{1}{2}  |(1-\gamma)+   \gamma s  \cdot e^{2\pi i  t}|^{2r+2} dt\\
&= \int_{\frac{-1}{2}}^\frac{1}{2}  [(1-\gamma)^2+ (\gamma s )^2+ 2 {(1-\gamma)} {\gamma}s \cos{(2\pi  t)} ]^{r+1} dt
\end{array} \]
Notice the following lemma, 
\begin{lemma}
For odd number, $\int_{\frac{-1}{2}}^\frac{1}{2}  \cos^{2k-1}{(2\pi  t)}  dt = 0$ for any $k\in {\mathbb Z}_+$; for even number, 
\[
\int_{\frac{-1}{2}}^\frac{1}{2}  \cos^{2k}{(2\pi  t)}  dt =  {2k\choose k} 2^{-2k},
\]
\end{lemma}
\begin{proof}
\[
 \begin{array}{ll}
2^n\cos^n (2\pi  t)=(e^{i \cdot 2\pi  t}+e^{-i \cdot 2\pi  t})^n=\sum_{k=0}^n {n\choose k} (e^{i\cdot 2\pi  t})^k\ (e^{-i\cdot 2\pi  t})^{n-k}=\sum_{k=0}^n {n\choose k} e^{(2k-n)i\cdot 2\pi  t},
\end{array} \]

\[
 \begin{array}{ll}
\int_{\frac{-1}{2}}^\frac{1}{2} e^{(2k-n)i\cdot 2\pi  t} dt= \delta_{2k-n}
\end{array} \]
\[
 \begin{array}{ll}
\int_{\frac{-1}{2}}^\frac{1}{2} 2^n\cos^n (2\pi  t) dt =\sum_{k=0}^n {n\choose k} \delta_{2k-n},
\end{array} \]
therefore, for odd number $n=2k-1$, $\int_{\frac{-1}{2}}^\frac{1}{2}  \cos^{2k-1}{(2\pi  t)}  dt = 0$ for any $k\in {\mathbb Z}_+$; for even number $n=2k$, 
\[
\int_{\frac{-1}{2}}^\frac{1}{2} 2^{2k} \cos^{2k}{(2\pi  t)}  dt =  {2k\choose k}.
\]
\end{proof}

\begin{lemma}
Now
$R_1 :=  (1-\gamma)^2+ (\gamma s )^2,$
$R_2 := (1-\gamma) \gamma s,$
\[
 \begin{array}{ll}
 \| (\hat{\bar a}^{\gamma})^{r+1}\|_2^2
=\sum_{k=0}^{\frac{r+1}{2}} [(1-\gamma)^2+ (\gamma s )^2]^{r+1-2k} [ (1-\gamma) \gamma s]^{2k}{2k\choose k}{r+1\choose 2k} 
\end{array} \]
\end{lemma}
\begin{proof}
Now define
$R_1 :=  (1-\gamma)^2+ (\gamma s )^2,$
$R_2 := {(1-\gamma)} {\gamma}s,$
\[
 \begin{array}{ll}
 \| (\hat{\bar a}^{\gamma})^{r+1}\|_2^2
&= \int_{\frac{-1}{2}}^\frac{1}{2}  [(1-\gamma)^2+ (\gamma s )^2+ 2 {(1-\gamma)} {\gamma}s \cos{(2\pi  t)} ]^{r+1} dt\\
&= \int_{\frac{-1}{2}}^\frac{1}{2}  
[R_1+ R_2 \cdot 2 \cos{(2\pi  t)} ]^{r+1} dt\\
&=\sum_{k=0}^{\frac{r+1}{2}} R_1^{r+1-2k} R_2^{2k}{2k\choose k}{r+1\choose 2k} 
\end{array} \]
\end{proof}

Use the Laplace's method theorem 
We could instead directly optimize  $\gamma$ over the $\ell_\infty$ norm of $|\hat{\bar a}^{\gamma}|^2$,
\begin{lemma}
\[
 \||\hat{\bar a}^{\gamma}|^2\|_\infty = \max_{ t\in [\frac{-1}{2}, \frac{1}{2}]}\{ (1-\gamma)^2 + (\gamma s)^2+ 2 (1-\gamma) \gamma s \cos{(2\pi  t)} \}  = (1-\gamma)^2 + (\gamma s)^2+ 2 |(1-\gamma) \gamma s| = (|1-\gamma|+|\gamma s|)^2 
\]
then

 \[
  \begin{array}{ll}
\gamma_\infty &= \arg\min_{\gamma>0}\||\hat{\bar a}^{\gamma}|^2\|_\infty \\
&= \arg\min_{\gamma>0}(|1-\gamma|+|\gamma s|)^2 \\
&= 1
 \end{array} \]
\[
\min_{\gamma\in (0,1)}\||\hat{\bar a}^{\gamma}|^2\|_\infty
= (|1-\gamma_\infty|+|\gamma_\infty s|)^2 
= |s|^2
\]
\end{lemma}
And
 \[
 \begin{array}{ll}
\|*_{r+1} (\bar a^{\gamma_\infty  })\|_2^2 
&=  s^{2r+2} 
\end{array} \]

 


\paragraph{Finite length approximation constructing using Z-transform truncation on factorization form }
Let the Z-transform of $a$ be 
\[
 \begin{array}{ll}
A(z) =  \sum_{i=-N_{-}}^{N_{+}} a_{i} z^{-i},
\end{array}
  \]
We can factorize the Z-transform as product of factors with roots $s_{-j}$ for $j = 1, \ldots N_{-}$ and  $i = 1, \ldots N_{+}.$
\[
 \begin{array}{ll}
A(z) =  c_0\prod_{j=1}^{N_{-}} (1- s_{-j} z) \prod_{i=1}^{N_{+}} (1- s_{i} z^{-1}),
\end{array}
  \]
  Where $c_0$ is a constant to make sure that the coefficient $a_0=1$.
  
 We assume that $|s_k|<1,$ for any $k = -N_{-}, \ldots , N_+.$  
  
Then Z-transform of the inverse kernel $a^{-1}$ is $1/A(z)$.
\[
 \begin{array}{ll}
1/A(z) &=  \frac{1}{c_0}\prod_{j=1}^{N_{-}} (1- s_{-j} z)^{-1} \prod_{i=1}^{N_{+}} (1- s_{i} z^{-1})^{-1}\\
&=  \frac{1}{c_0}\prod_{j=1}^{N_{-}} (\sum_{\ell_j=0}^{\infty} s^{\ell_j}_{-j} z^{-\ell_j} ) \prod_{i=1}^{N_{+}} (\sum_{\ell_i=0}^{\infty} s^{\ell_i}_{i} z^{\ell_i} )
\end{array}
  \]

From previous single root study, we get a hint to study a single truncation of length $ 1 + \sum_{j=1}^{N_{-}} r_{-j}  +  \sum_{j=1}^{N_{+}}  r_i$,  we construct
\[
 \begin{array}{ll}
W(z) 
&=  \frac{1}{c_0}\prod_{j=1}^{N_{-}} (\sum_{\ell_j=0}^{r_{-j}} s^{\ell_j}_{-j} z^{-\ell_j} ) \prod_{i=1}^{N_{+}} (\sum_{\ell_i=0}^{r_{-j}} s^{\ell_i}_{i} z^{\ell_i} )\\
&=  \frac{1}{c_0}\prod_{j=1}^{N_{-}} (1- (s_{-j} z)^{r_{-j}+1})(1- s_{-j} z)^{-1} \prod_{i=1}^{N_{+}}(1- (s_{i}  z^{-1})^{r_i+1}) (1- s_{i} z^{-1})^{-1}
\end{array}
  \]
Let the corresponding filter being $w$.
  
  
Now let $\psi =  a*w$ with Z-transform $\Psi(z)$, $\phi = a*w-e_0$ with Z-transform $\Phi(z)$,we have
\[
 \begin{array}{ll}
\Psi(z): = A(z)W(z) 
&=   \prod_{j=1}^{N_{-}} (1- (s_{-j} z)^{r_{-j}+1}) \prod_{i=1}^{N_{+}}(1- (s_{i} z^{-1})^{r_i+1}) 
\end{array}
  \]
  therefore,
\[
 \begin{array}{ll}
\Phi(z): = A(z)W(z) -1 
&=  \prod_{j=1}^{N_{-}} (1- (s_{-j} z)^{r_{-j}+1}) \prod_{i=1}^{N_{+}}(1- (s_{i} z^{-1})^{r_i+1}) -1
\end{array}
  \]
\paragraph{Upper Bound on  $\|\phi\|_2^2$ for constructed finite length approximation}
In the following,  we want to give upper bound for
  \[
\|a*w -e_0\|_2^2 =\|\phi\|_2^2 
 \]  
  
Redefine the index $\tau_{-j} := r_{-j}+1; \tau_{i} = r_i+1.$
Let ${\cal I} = \{-N_{-},\ldots, -1, 1, \ldots, N_{+} \}$ as the set of all the possible index.
And we define polar representation of complex roots:
for any $k \in {\cal I} = \{-N_{-},\ldots, -1, 1, \ldots, N_{+} \}$
\[
s_{k} := |s_{k}| e^{i \varphi_k } = e^{\rho_k + i \varphi_k }
\]
where 
\[
 \begin{array}{ll}
\varphi_k := \operatorname{Im}(\log( s_{k} )) = -i \log(\frac{s_{k}}{|s_{k}|}) 
\end{array}
\]
and since $|s_{k}|<1$ we have 
\[
\rho_k = -\operatorname{Re}(\log( s_{k} ))>0
\]

Now we consider all $z = e^{2 \pi i t}$ on the unit circle for $t\in [\frac{-1}{2},\frac{1}{2}]$,
then for any $k \in {\cal I}$
\[
 \begin{array}{ll}
- (s_{k} z^{-\mbox{sign}(k)})^{\tau_{k}} 
&= - |s_{k}|^{\tau_{k}} \exp \big(i [\varphi_k-  2 \pi t \mbox{sign}(k)]\cdot \tau_{k} \big)\\
&=   \exp \Big( \tau_{k} \rho_k+ i \cdot \big( \tau_{k} (\varphi_k- 2 \pi t \mbox{sign}(k) ) + \pi  \big) \Big) 
\end{array}
  \]
 Now we simplify the notation by defining 
 \[
 \upsilon_k(t) :=\tau_{k} (\varphi_k- 2 \pi t \mbox{sign}(k) ) + \pi
 \]
then
 \[
 \begin{array}{ll}
\Phi(z)
&=  \prod_{j=1}^{N_{-}} (1- (s_{-j} z)^{\tau_{-j}}) \prod_{i=1}^{N_{+}}(1- (s_{i} z^{-1})^{\tau_{i}}) -1\\
&=  \prod_{k \in {\cal I}} \Big(1- |s_{k}|^{\tau_{k}} \exp \big(i [\varphi_k-  2 \pi t \mbox{sign}(k)]\cdot \tau_{k} \big)\Big)  -1\\
&=  \prod_{k \in {\cal I}} \bigg( 1+ \exp \Big( \tau_{k} \rho_k+ i \cdot \big( \tau_{k} (\varphi_k- 2 \pi t \mbox{sign}(k) ) + \pi  \big) \Big) \bigg)     -1\\
&=  \prod_{k \in {\cal I}} \bigg( 1+ \exp \Big( \tau_{k} \rho_k+ i \cdot \upsilon_k(t) \Big) \bigg)     -1
\end{array}
  \]
  
\begin{lemma}
 For any $z = \exp{(2 \pi i t )}$ on the unit circle,
 \[
 \begin{array}{ll}
\Phi\big(\exp{(2 \pi i t )}\big)
&=  \prod_{k \in {\cal I}} \bigg( 1+ \exp \Big( \tau_{k} \rho_k+ i \cdot \upsilon_k(t) \Big) \bigg)     -1\\
&= \sum_{n=1}^{|{\cal I}|}\sum_{ k_1, \ldots, k_n  \in {\cal I} } 
\exp \Big( \big(\tau_{k_1} \rho_{k_1}+ \ldots+  \tau_{k_n} \rho_{k_n} \big)
+ i \cdot \big( \upsilon_{k_1}(t) + \ldots+  \upsilon_{k_n}(t)  \big) \Big)  
\end{array}
  \]
 \end{lemma}
 
 \begin{lemma}
Additionally, for any $t \in [\frac{-1}{2},\frac{1}{2}]$,
\[
 \begin{array}{ll}
 |\Phi\big(\exp{(2 \pi i t )}\big)|^2  
= 
\sum_{n=1}^{|{\cal I}|}\sum_{ k_1, \ldots, k_n  \in {\cal I} } 
\exp \Big( 2 \big(\tau_{k_1} \rho_{k_1}+ \ldots+  \tau_{k_n} \rho_{k_n} \big) \Big)   
\end{array}
  \]
is independent of $t$, since the oscillation integral over the imaginary part is 
  \[
 \begin{array}{ll}
\int_{\frac{-1}{2}}^\frac{1}{2}
\exp  \bigg( i \cdot \Big(  \big(\upsilon_{k_1}(t) - \upsilon_{k_1}(t) \big) + \ldots+  \big(\upsilon_{k_n}(t) -\upsilon_{k_n}(-t)\big)  \Big) \bigg)  dt
= 1
\end{array}
  \]
Therefore, using Fourier isometry,
 \[
 \begin{array}{ll}
 \|\phi\|_2^2 &= \int_{\frac{-1}{2}}^\frac{1}{2} |\Phi\big(\exp{(2 \pi i t )}\big)|^2 dt \\
&= \sum_{n=1}^{|{\cal I}|}\sum_{ k_1, \ldots, k_n  \in {\cal I} } 
\exp \Big( 2 \big(\tau_{k_1} \rho_{k_1}+ \ldots+  \tau_{k_n} \rho_{k_n} \big) \Big)   
\end{array}
  \]
  
 \end{lemma}

\begin{theorem}
\label{thm: Root_2_norm}
Since $|s_{k}|<1$, i.e. $\rho_{k}<0$ for all $k\in {\cal I}$.
 \[
 \begin{array}{ll}
 \|\phi\|_2^2 
&= \sum_{n=1}^{|{\cal I}|}\sum_{ k_1, \ldots, k_n  \in {\cal I} } 
\exp \Big( 2 \big(\tau_{k_1} \rho_{k_1}+ \ldots+  \tau_{k_n} \rho_{k_n} \big) \Big) 
\end{array}
  \]
  Let $\gamma^m := \min_{i}\big(\tau_{i} |\rho_{i}| \big) =  \min\{\tau_{-N_-}|\rho_{-N_-}|, \ldots, \tau_{-1}|\rho_{-1}|, \tau_{1}|\rho_{1}|, \ldots, \tau_{N_+}|\rho_{N_+}|\}.$
When $\gamma^m \rightarrow \infty$, we have a finite sum of exponential rate  converging terms: the convergence rate is determined by the exponential term as a function of $\gamma^m$.
   \[
 \begin{array}{ll}
 \|\phi\|_2^2 
=    O(\exp \Big( -2 \min_{i}\big(\tau_{i} |\rho_{i}| \big) \Big))
= O(\exp \Big( -2 \gamma^m  \big) \Big))
\rightarrow 0
\end{array}
  \]

 \end{theorem}

 







\section{Conclusion}
In this paper, we proposed a novel {\it convex} optimization problem for sparse {\it blind deconvolution problem} based on $\ell^1$
minimization of inverse filter outputs:
\[
 \begin{array}{ll}
 \underset{{\bf w} \in l_1^k}{\mbox{minimize}}   & \frac{1}{N} \| {\bf w} \star {\bf y}\|_{\ell_1^N}\\
\mbox{subject to}  
& \langle \widetilde{{\bf a}},  {\bf w}^{\dagger} \rangle =1.
\end{array}
  \]
Assuming the signal to be recovered is {\it sufficiently sparse}, the algorithm can convert a crude approximation to the filter into a high-accuracy recovery of the true filter.   
  
We present four main results. 

First, in a large-$N$ analysis where ${\bf x}$ is a realization of an IID Bernoulli-Gaussian signal with expected sparsity level $p$, we measure the approximation quality of $\widetilde{{\bf a}}$ 
by considering $\widetilde{{\bf e}}= \widetilde{{\bf a}} \star {\bf a}^{-1}$, 
which would be a Kronecker sequence if our approximation were perfect.
Under the condition
\[
\frac{|\widetilde{{\bf e}}|_{(2)}}{|\widetilde{{\bf e}}|_{(1)}} \leq 1-p,
\]
we show that, in the large-$N$ limit,
the $\ell^1$ minimizer ${\bf w}^*$
perfectly recovers ${\bf a}^{-1}$ to shift and scaling. 

In words {\it the less accurate the initial approximation $\widetilde{{\bf a}} \approx {\bf a}$, the greater we rely on sparsity of ${\bf x}$.}

Second, we develop finite-$N$ guarantees of the form $N\geq O(k\log(k))$,
for highly accurate reconstruction with high probability.

Third, we further show stable approximation when the true inverse filter is infinitely long
(rather than length $k$), we show that the approximation error decrease exponentially as the approximation length growth.

Last, we extend our guarantees to the case where
the observation contain stochastic or adversarial noise, we show that in both stochastic or adversarial noise cases, the approximation error growth linearly as a function of noise magnitude.

\section{Discrete signal structure}
\paragraph{Random model for discrete digital signals.}
We define a random model for discrete digital signals. 

Let $(X_t)$ be an independent
sequence of random variable indexed by  $t \in Z$, with a symmetric distribution with CDF $F$. 
\begin{itemize}
\item The two point symmetric distribution on real number, $F = 1/2 \delta_{+1} + 1/2 \delta_{-1}.$ 
\item The four point symmetric distribution on complex number, 
$F = 1/4 \delta_{1+i} + 1/4 \delta_{-1+i}+1/4 \delta_{1-i}
+1/4 \delta_{-1-i}.$ 
\item A symmetric distribution on ${\mathbb R}^s$,
$F = 1/2^s( \sum_{c\in \{\pm 1\}^s} \delta_{\sum_{i\in[s]} c_i e_i } ).$ 

\item Constallantion in unit sphere.
The $2k$ point symmetric distribution on complex number, 
$F_p = \sum_{j=0}^{2k-1} p_j \delta_{\exp{(i\pi\frac{j}{k})}},$
where $\sum_{j=0}^{2k-1} p_j =1, p_j>0.$
\end{itemize}



\paragraph{Convex loss for discrete digital structure.}
The convex loss  we choose could be a relaxation of the convex indictor function of the points.

\begin{itemize}
\item For the two point symmetric distribution on real number, 
\[
g(x) = (|x|-1)_+
\]
\item The four point symmetric distribution on complex number, 
\[
g(z) = (|z|-1)_+,
\]
or more accurately,
\[
g(z) = (\|z\|_{\infty}-1)_+.
\]
\item For $s$-dimensional symmetric distribution on $\{\pm 1\}^s$,
\[
g(x) = (\|x\|_{\infty}-1)_+.
\] 
\end{itemize}


\paragraph{Blind recovery problem.}
For observation $Y$, assume that there exist inverse filter $\mu^* (e_0 + f^*)$, where $f^*_0 = 0$. We try to find it by solving a convex problem,
\begin{eqnarray*}
    \min_{f_0 = 0} &    
    E \sum g(\mu (Y+ f*Y)).
\end{eqnarray*}
First, when we know the correct scale $\mu^*$.

Let 
\[
e_0+h = ( e_0 + f)(e_0 + f^*)^{-1},
\]
we have
\begin{eqnarray*}
    \min_{f_0 = 0} &    
    E g( x_0+ h*x)).
\end{eqnarray*}
For $g(x) =(|x|-1)_+$, then the sub-gradient is
\[
g'(x) =  \left.
  \begin{cases}
    1, & \text{for } x > 1 \\
    0, & \text{for } -1 < x < 1 \\
    -1, & \text{for }-1 > x \\
    [0,1], & \text{for }x = 1  \\
    [-1,0], & \text{for }x = -1
  \end{cases}
  \right\}
\]

And the proximal operator $\mbox{prox}_{\rho g}$ is
\[
\mbox{prox}_{\rho g}(x) =  \left.
  \begin{cases}
    x-\rho, & \text{for } x > 1+\rho \\
    x, & \text{for } -1 < x < 1 \\
    x+\rho, & \text{for }-1-\rho > x \\
    1, & \text{for }1 \leq x \leq 1 +\rho \\
    -1, & \text{for } -1 -\rho\leq x \leq -1
  \end{cases}
  \right\}
\]


Assume $(X_t)$ are independent
sequence of random variable with CDF $F_{p}^t = \sum_{j=0}^{1} p^t_j \delta_{(-1)^j},$
where $\sum_{j=0}^{1} p^t_j =1, p^t_j>0.$ We choose the objective
\[
g(z) = (|z|-1)_+.
\] 
To show that $h=0$ is the global minimizer of the convex problem, we only need to show that for any $h\neq 0$, there exists small $\epsilon>0$, 
\begin{eqnarray*}    
    E g( x_0+ \epsilon h*x)-E g( x_0)>0.
\end{eqnarray*}
We use sub-gradient information,
\begin{eqnarray*}    
    E g( x_0+ \epsilon h*x)-E g( x_0)
    &\approx& 
    E g'( x_0)(\epsilon h*x)\\
    &=&
    \epsilon( p^0 _1 g'( +1)(h_0)+p^0_{-1}g'( -1)(-h_0)  )
    \geq 0.
\end{eqnarray*}

\paragraph{Polar form and extension to complex plane}
We can rewrite a point on the complex plane as its polar form
\[
z = r \exp{i\theta},
\]
where $r\geq 0$.

Under the polar form, the loss function is 
\[
g(z) = (|z|-1)_+ = (r-1)_+,
\]

We can reparametrize

Using this polar form, we are able to generalize our result to the complex setting.

\paragraph{Non-convex loss for discrete digital structure.}
If we are allowed to use non-convex loss, then we can fix the scale by the loss function.

Instead of convex loss
\[
g(x) = (|x|-1)_+,
\]
or
\[
g(x) = (|x|^2-1)_+,
\]
we can choose
\begin{itemize}
\item The smooth loss 
\[
g(x) = (|x|^2-1)^2,
\]
\item The composition of convex non-smooth function with smooth function,
\[
g(x) = (|x|-1)^2,
\]
\item The composition of  smooth function with convex non-smooth function,
\[
g(x) = ||x|^2-1|,
\]
\item The composition of  two convex non-smooth functions,
\[
g(x) = ||x|-1|.
\]
\end{itemize}


And instead of
\[
g(x) = (\|x\|_{\infty}-1)_+,
\]
we can choose
\begin{itemize}
\item The composition of convex non-smooth function with smooth function,
\[
g(x) = (\|x\|_{\infty}-1)^2,
\]
\item The composition of  two convex non-smooth functions,
\[
g(x) = |\|x\|_{\infty}-1|.
\]
\end{itemize}

\subsection{Noisy observation}
We consider the noisy recovery problem, there are two noise model, one is
\[
y = w_0 * (x+  n),
\]
the other is 
\[
y = w_0 * x+  u = w_0 * (x+ w_0^{-1} *u).
\]
When $u$ is Gaussian $N(0,\Sigma) $ , $w_0^{-1} *u$ is Gaussian $N(0,C_{w_0}^{-1}\Sigma C_{w_0}^{-1,T}) $. Therefore, we could reduce this problem as a colored noise $n$. 

\paragraph{Post-process}
For noisy observation $y$, we apply the algorithm on $y$ to get a solution $\hat x$, then we apply our prior knowledge to project the solution to the constellation points by taking $\title{x} = \mbox{sign}(\hat x).$

Then we measure the ratio of correct recovery by mean of
$1_{\title{x} = x}$ over multiple MC samples.
\paragraph{Population analysis}
Assume that the observation model are $y = w_0 * (x+  \sigma n),$ where $n$ is IID $N(0,1)$. Then we analyze the population risk,
 \begin{eqnarray*}
    \min_{h_0 = 0} &    
     E_x E_n g( x_0+ \sigma n_0+ h^T (x+\sigma n) ).
\end{eqnarray*}
Here $g(x) =(|x|-1)_+$.

Assume that the optimal solution is $h^*_\sigma$, when $\sigma = 0$ we proved before that $h^*=0.$ Here we get
$\hat x_0 = x_0+ \sigma n_0+ h^{*,T} (x+\sigma n) $
want to prove
\[
\tilde x_0 = \mbox{sign}(x_0+ \sigma n_0+ h_\sigma^{*,T} (x+\sigma n) ) = x_0,
\]
when $\sigma<\sigma_{m}$.
Without loss of generality, we assume $x_0=1$, we want to ask that when would $1+ \sigma n_0+ h_\sigma^{*,T} (x+\sigma n)\leq 0$, in general, the condition we need to charaterize is when 
$ |\sigma n_0+ h_\sigma^{*,T} (x+\sigma n)|\leq 1$.

\paragraph{Huber version of loss}
Instead of using $g(x) =(|x|-1)_+$, when there are Gaussian noise on $x$, we can use the Huber version 
\[
g_{\mu}(x) =\min_z  \frac{\mu}{2}\|x-z\|^2_2+ (|z|-1)_+  .
\]



























\section{Expectation calculation}
\subsection{Expectation over Bernoulli support}

We previously considered the identity
\begin{equation} \label{eq:startingPoint}
    {\mathbb E} | \sum_{i \in {\cal T}}  \psi_i X_i | = \sqrt{\frac{2}{\pi}} \cdot {\mathbb E}_I \| \psi \|_{\ell_2(I)},
\end{equation}
where $X$ is a Bernoulli-Gaussian RV $BG(p;0,1)$,
and where $I$ is a random subset of the domain ${\cal T}$
determined by Bernoulli-$p$ coin tossing.




This leads us to consider
\[
  {\mathbb E}_I \| \psi \|_{\ell_2(I)}^{2k}
 \]
 where $I$ is again a random subset, and $k \geq 1$ is an integer.
 The following lemma shows that ${\mathbb E}_I \| \psi \|_{\ell_2(I)}^{2k} \approx \| \psi \|_{\ell_2({\cal T})}^{2k}$.

\begin{lemma} \label{lem:BGPropLemmaSub}
Let $\psi$ denote a vector in ${\mathbb R}^N$.
\[
 \| \psi\|_2^{2k} \cdot \bar{B}(k-1;n,p)    \leq {\mathbb E}_I  \| \psi \|_{\ell_2(I)}^{2k}  \leq     \| \psi\|_2^{2k} ,
\]
where $\bar{B}(k-1;N,p) \equiv  \sum_{n=k}^{N} (1-p)^n p^{N-n} {N \choose n} $
is the survivor function (Complementary CDF) of the Binomial.
\end{lemma}

In particular, if $k \leq n \cdot p - 3 \cdot \sqrt{np (1-p)}$ for large $n$, then $\bar{B}(k-1;n,p) > 0.99$.
 The following lemma infers from this that ${\mathbb E}_I \| \psi \|_{\ell_2(I)} \approx \| \psi \|_{\ell_2({\cal T})}$.
 
\begin{lemma} \label{lem:BGPropLemmaNorm}
{\bf Qingyun: Numerically, I found this is not true even for fully dense $\psi$ as ${\varepsilon}_N$ could be quite large!}


There exists a sequence $({\varepsilon}_N)$ depending only on $N$ and $A$ so that  ${\varepsilon}_N \rightarrow 0$ 
and so that for each $N$ and for each 
vector $\psi$  in ${\mathbb R}^N$, with $\|\psi\|_{\ell_2} \leq A$ and $\|\psi\|_{\ell_0} = N$,
\begin{equation} \label{eq:LemmConclu}
 (1-N^{-1}) \cdot(1 - {\varepsilon}_N)  \leq  \frac{{\mathbb E}_I  \| \psi \|_{\ell_2(I)} }{\| \psi \|_{\ell_2({\cal T})}} \leq 1 + {\varepsilon}_N .
\end{equation}
\end{lemma}
Applying (\ref{eq:startingPoint}), we get:
\begin{coro} \label{lem:BGPropLemma}
{\bf Qingyun: Numerically, I found this is not true even for fully dense $\psi$!}
There exists a sequence $({\varepsilon}_N)$ depending only on $N$, so that  ${\varepsilon}_N \rightarrow 0$ and
and so that for each $N$ and for each 
vector $\psi$in ${\mathbb R}^N$, with $\|\psi\|_{\ell_2} \leq A$ and $\|\psi\|_{\ell_0} = N$.
\[
(1 - {\varepsilon}_N) \cdot (1-N^{-1}) \leq  \frac{E | \sum_{i \in {\cal T}}  \psi_i X_i |}{\sqrt{\frac{2}{\pi}}  \| \psi \|_{\ell_2({\cal T})}}  \leq   ( 1 + {\varepsilon}_N).
\]
\end{coro}


\subsection{Proof of Lemma \ref{lem:BGPropLemmaSub}}

\begin{proof}[Proof of Lemma \ref{lem:BGPropLemmaSub}]
Let's first suppose that $|\psi_t| \geq {\varepsilon} > 0$ for $-T \leq t \leq T$. Let $N = 2T+1$, and
let $I$ denote a subset of indices in $ {\cal T} = \{-T,\dots,T\}$ and
let $ \pi(I) = (1-p)^{|I|} p^{|{\cal T} \Delta I|}$. 
\begin{eqnarray*}
{\mathbb E}_I  \| \psi \|_{\ell_2(I)}^{2k}  &=&  \sum_{n=1}^{N}  \sum_{ |I |=n} \pi(I)   \| \psi \|_{\ell_2(I)}^{2k} \\
&=&   \sum_{n=1}^{N}  (1-p)^n p^{N-n}  \cdot  \sum_{|I| =n} \| \psi \|_{\ell_2(I)}^{2k}  \\
   \end{eqnarray*}
where we used $ \pi(I) = (1-p)^{|I|} p^{|{\cal T} \Delta I|} $.
We partition the sum  $\sum_{n=1}^{N} = \sum_{n=1}^{k-1} + \sum_{k}^N =  \sum_{n<k} + \sum_{n\geq k}$, say.
Now from $\#\{ |I|\leq n\} = {N \choose n}$ and $ \| \psi \|_{\ell_2(I)} \leq \| \psi \|_{\ell_2({\cal T})}$,
\begin{eqnarray*}
 \sum_{n<k}  (1-p)^n p^{N-n}  \cdot  \sum_{|I| =n} \| \psi \|_{\ell_2(I)}^{2k} 
     &\leq&   \left( \sum_{n=1}^{k-1} (1-p)^n p^{N-n} {N \choose n} \right) \cdot  \left( \| \psi \|_{\ell_2}^{2k} \right) \\
     &=& B(k-1;n,p) \cdot   \| \psi \|_{\ell_2}^{2k} ,
\end{eqnarray*}
where $B(x;n,p)$ denotes the cumulative distribution function of the binomial distribution.
For the sum $\sum_{n\geq k}$, we may write $\sum_{|I|=n} = {N \choose n}  \cdot Ave_{|I| = n}$
and $\sum_{t_1,\dots, t_k \in I} =  {n^k} \cdot  Ave_{t_1,\dots, t_k \in I}$, so
\begin{eqnarray*}
 \sum_{|I| = n}  \| \psi \|_{\ell_2(I)}^{2k} &=& \sum_{|I| = n} \sum_{t_1,\dots, t_k \in I} \prod_{i=1}^k \psi_{t_i}^2 \\
& =& {N \choose n}  \cdot Ave_{|I| = n}  \cdot {n^k} \cdot  Ave_{t_1,\dots, t_k \in I} \; \prod_{i=1}^k \psi_{t_i}^2 \\
&=& {N \choose n}  \cdot {n^k} \cdot Ave_{t_1,\dots, t_k \in {\cal T}} \; \prod_{i=1}^k \psi_{t_i}^2,\\
&=& {N \choose n}  \cdot \sum_{t_1,\dots, t_k \in {\cal T}} \; \prod_{i=1}^k \psi_{t_i}^2,\\
&=& {N \choose n}  \cdot  \| \psi \|_{\ell_2}^{2k} ,
\end{eqnarray*}
because the inner two averages  cover the $ {N \choose n} n^k$ terms in a uniform way.
We arrive at 
 \begin{eqnarray*}
 \sum_{n\geq k}  (1-p)^n p^{N-n}  \cdot  \sum_{|I| =n} \| \psi \|_{\ell_2(I)}^{2k} 
  &=&     \| \psi \|_{\ell_2}^{2k} \cdot  \left( \sum_{n=k}^{N} (1-p)^n p^{N-n} {N \choose n} \right) \\
  &=&    \| \psi \|_{\ell_2}^{2k} \cdot \bar{B}(k-1;n,p) .
 \end{eqnarray*} 
Here $\bar{B}(x;n,p) = 1- B(x;n,p)$ denotes the 
survivor function (complementary distribution function) of the binomial distribution.
We obtain:
 \begin{eqnarray*}
  \sum_{I \subset {\cal T}} \pi(I)   ( \sum_{t \in I} \psi_t^2)^{k}  
  & = &  \left( \sum_{n \geq k} + \sum_{n<k}   \right) \sum_{|I|=n}  \pi(I)   ( \sum_{t \in I} \psi_t^2)^{k} \\
 &\leq &    \|\psi\|_2^{2k} \cdot  \left( \bar{B}(k-1;p,n)  + B(k-1;p,n) \right) \\
 & = &  \|\psi\|_2^{2k} .
 \end{eqnarray*}
 while
 \begin{eqnarray*}
  \sum_{I \subset {\cal T}} \pi(I)   ( \sum_{t \in I} \psi_t^2)^{k}   & \geq & \sum_{n \geq k} \pi(I)   ( \sum_{t \in I} \psi_t^2)^{k}  \\
 &= &    \|\psi\|_2^{2k} \cdot \bar{B}(k-1;p,n) .
  \end{eqnarray*}
\end{proof}

\subsection{Proof of Lemma \ref{lem:BGPropLemmaNorm}}

\begin{proof}
Fix $A > 0$ and $p \in (0,1)$
By the Weierstrass approximation theorem, we may find a sequence of polynomials $S_K(t) = \sum_{k=0}^{K} c_k t^k$
so that
\[
    | \sqrt{x} - S_K(x) |_{L^\infty([0,A])} \leq {\varepsilon}_K, \quad \mbox { and } {\varepsilon}_K \rightarrow 0 \mbox{ as } K \rightarrow \infty.
\]
Pick a sequence $(K_N)$ so that $K_N \rightarrow \infty$ while
$K_N < N \cdot p -  \sqrt{2 \log(N) \sqrt{N} p (1-p)} $ and
\[
\bar{B}(K_N-1; n,p) > 1 - N^{-1}.
\]
Relabel $S_N  \equiv S_{K_N}$ and ${\varepsilon}_N \equiv {\varepsilon}_{K_N} $.
Then ${\varepsilon}_N \rightarrow 0$ as $N \rightarrow \infty$.

Fix $N$. For each $I \subset {\cal T}$,
let  $x_I \equiv {\mathbb E}_I \| \psi \|_{\ell_2(I)}^{2}$.  Then
\[
     E _I \| \psi \|_{\ell_2(I)} = {\mathbb E}_I \sqrt{\| \psi \|_{\ell_2(I)}^2} = {\mathbb E}_I \sqrt{ x_I }.
\]
while
\[
     E _I \| \psi \|_{\ell_2(I)}^{2k} = {\mathbb E}_I  x_I^k.
\]
Invoking our earlier lemma,
\[
 \bar{B}(k-1;n,p) \leq  \frac{{\mathbb E}_I  x_I^k}{\| \psi \|_{\ell_2({\cal T})}^{2k}} \leq 1.
\]

Armed with  polynomial $S_N$ of degree $K_N$, consider
\begin{eqnarray*}
     {\mathbb E}_I S_{N}( x_I ) &=& {\mathbb E}_I \sum_{k=1}^{K_N} c_k x_I^{k}\\
     &=&  \sum_{k=1}^{K_N} c_k  {\mathbb E}_I x_I^{k} \\
     &\leq& \sum_{k=1}^{K_N} c_k   \| \psi \|_{\ell_2({\cal T})}^{2k}\\
     &=& S_{N}( \| \psi \|_{\ell_2({\cal T})}^2)  \\
     &\leq& ( \| \psi \|_{\ell_2({\cal T})}^2)^{1/2} + {\varepsilon}_{N} . 
\end{eqnarray*}
where we used ${\mathbb E}_I x_I^{k} \leq  \| \psi \|_{\ell_2({\cal T})}^{2k}$ from the previous Lemma.
Similarly,
\begin{eqnarray*}
     {\mathbb E}_I S_N( x_I )   &=&  \sum_{k=1}^{K_N} c_k  {\mathbb E}_I x_I^{k} \\
     &\geq& \sum_{k=1}^{K_N} c_k   \cdot \bar{B}(k-1; n,p) \cdot \| \psi \|_{\ell_2({\cal T})}^{2k}\\
     &\geq& \bar{B}(K_N-1; n,p) \cdot \sum_{k=1}^{K} c_k   \cdot \| \psi \|_{\ell_2({\cal T})}^{2k}\\
     &=& \bar{B}(K_N-1; n,p)  \cdot S_K( \| \psi \|_{\ell_2({\cal T})}^2)  \\
     &\geq&  \bar{B}(K_N-1; n,p)  \cdot \left( ( \| \psi \|_{\ell_2({\cal T})}^2)^{1/2} - {\varepsilon}_N \right)  \\
   &\geq&  (1- N^{-1}) \cdot \left( ( \| \psi \|_{\ell_2({\cal T})}^2)^{1/2} - {\varepsilon}_N \right)  . 
\end{eqnarray*}
Here  we used ${\mathbb E}_I x_I^{k} \geq  \| \psi \|_{\ell_2({\cal T})}^{2k} \cdot \bar{B}(k-1;n,p)$ from the previous Lemma.
We also used the monotonicity 
of the survivor function $\bar{B}(x) \geq \bar{B}(x+1)$.
We obtain (\ref{eq:LemmConclu}):
\[
 (1-N^{-1}) \cdot(1 - {\varepsilon}_N)  \leq  \frac{{\mathbb E}_I  \| \psi \|_{\ell_2(I)} }{\| \psi \|_{\ell_2({\cal T})}} \leq 1 + {\varepsilon}_N .
\]
\end{proof}


We define the functional  $\Gamma: \psi  \mapsto \frac{{\mathbb E}_I  \| \psi \|_{\ell_2(I)} }{\| \psi \|_{\ell_2({\cal T})}}$,
from the previous calculation we know $\Gamma(\psi) $ is only a function of $\| \psi\|_0$. 
So we can identify  $\Gamma(\psi) $ as  $\Gamma(\| \psi\|_0)$. 





\subsection{Sharper bounds on $\frac{{\mathbb E}_I  \| \psi \|_{\ell_2(I)} }{\| \psi \|_{\ell_2({\cal T})}}$}




\begin{lemma}
\label{TaylorSqrt}
For $ 0 \leq x\leq 1,$
\begin{align}
(1-x)^{1/2}
=1-\sum_{\ell=0}^\infty\frac2{\ell+1}\binom{2\ell}{\ell}\left(\frac x4\right)^{\ell+1}
\end{align}
\end{lemma}

\begin{proof}
From generalized binomial theorem, we know that for $ 0 \leq x\leq 1,$
\begin{equation}
(1-x)^{1/2}
=\sum_{\ell=0}^\infty\binom{1/2}{\ell}(-x)^\ell
\end{equation}

\begin{align}
\binom{1/2}{\ell}
&=\frac{\frac12(\frac12-1)(\frac12-2)\cdots(\frac12-\ell+1)}{\ell!}\\
&=\frac{(-1)^{\ell-1}}{2^\ell\ell!}1\cdot3\cdot5\cdots(2\ell-3)\\
&=\frac{(-1)^{\ell-1}}{2^\ell\ell!}\frac{(2\ell-2)!}{2^{\ell-1}(\ell-1)!}\\
&=\frac{(-1)^{\ell-1}}{\ell2^{2\ell-1}}\binom{2\ell-2}{\ell-1}
\end{align}
then
\begin{align}
(1-x)^{1/2}
&=1-\sum_{\ell=1}^\infty\frac2\ell\binom{2\ell-2}{\ell-1}\left(\frac x4\right)^\ell\\
&=1-\sum_{\ell=0}^\infty\frac2{\ell+1}\binom{2\ell}{\ell}\left(\frac x4\right)^{\ell+1}
\end{align}






\end{proof}

\begin{lemma}
\begin{equation} 
0 \leq 1- Q(N, p)\leq  \frac{{\mathbb E}_I  \| \psi \|_{\ell_2(I)} }{\| \psi \|_{\ell_2({\cal T})}} \leq 1 .
\end{equation}
where 
\[
Q(N, p) := \sum_{\ell=0}^N\frac{2^{-(2\ell+1)}}{\ell+1}\binom{2\ell}{\ell} \bar{B}(\ell;N,1-p),
\]
\[
\bar{B}(\ell;N,1-p) \equiv  \sum_{n=0}^{N-\ell-1} (1-p)^n p^{N-n} {N \choose n}.
\]
\end{lemma}
\begin{proof}
For any fixed support $J\subset {\cal T},$ 
\begin{equation} 
 \frac{ \| \psi \|_{\ell_2(J)} }{\| \psi \|_{\ell_2({\cal T})}} 
 = \sqrt{ \frac{ \| \psi \|^2_{\ell_2(J)} }{\| \psi \|^2_{\ell_2({\cal T})}}} \leq 1 .
\end{equation}
Therefore, 
\begin{equation} 
{\mathbb E}_I \frac{ \| \psi \|_{\ell_2(I)} }{\| \psi \|_{\ell_2({\cal T})}} 
 = {\mathbb E}_I \sqrt{ \frac{ \| \psi \|^2_{\ell_2(I)} }{\| \psi \|^2_{\ell_2({\cal T})}}} \leq {\mathbb E}_I 1 = 1 .
\end{equation}

On the other hand, let 
\begin{equation} 
{\mathbb E}_I \frac{ \| \psi \|_{\ell_2(I)} }{\| \psi \|_{\ell_2({\cal T})}} 
 = {\mathbb E}_I \sqrt{ 1- \frac{ \| \psi \|^2_{\ell_2({\cal T}-I)} }{\| \psi \|^2_{\ell_2({\cal T})}}}  .
\end{equation}

For a fixed subset $J \subset [n]$ define 
the ratio $\rho_J \equiv  \frac{ \| \psi \|^2_{\ell_2({\cal T}-J)} }{\| \psi \|^2_{\ell_2({\cal T})}}$; it 
obeys $0 \leq \rho_J \leq 1$.

Applying Lemma \ref{TaylorSqrt},
\begin{align}
(1-\rho_J)^{1/2}
=1-\sum_{\ell=0}^\infty\frac{2^{-(2\ell+1)}}{\ell+1}\binom{2\ell}{\ell}
\rho_J^{\ell+1}
\end{align}
Now with $I$ a random subset as earlier, we
induce a random variable $\rho_I$.
\begin{align}
{\mathbb E}_I(1-\rho_I)^{1/2}
=1-\sum_{\ell=0}^\infty\frac{2^{-(2\ell+1)}}{\ell+1}\binom{2\ell}{\ell}
{\mathbb E}_I \rho_I^{\ell+1}
\end{align}
Modify the argument behind Lemma \ref{lem:BGPropLemmaSub} so it applies to ${\mathbb E}_I \rho_I^{\ell+1}$. In the new argument the set $I$ is everywhere replaced by its complement $T-I$. The conclusion is correspondingly modified so that $p$ in the final formula is replaced by $q \equiv 1-p$.
We obtain:
\[
 \bar{B}(\ell;n,1-p)    \leq {\mathbb E}_I \rho_I^{\ell+1} \leq    1,
\]
where, if $k \in \{0,1,\dots,N\}$,
$\bar{B}(k-1;N,p) \equiv  \sum_{n=k}^{N} (1-p)^n p^{N-n} {N \choose n}  = 1- \sum_{n=0}^{k-1} (1-p)^n p^{N-n} {N \choose n}$, while if $k \geq N$, $\bar{B}(k) = 0$, and if $k < 0$,
$\bar{B}(k) =1$.
Then
\begin{align}
{\mathbb E}_I(1-\rho_I)^{1/2}
&=1-\sum_{\ell=0}^\infty\frac{2^{-(2\ell+1)}}{\ell+1}\binom{2\ell}{\ell}
{\mathbb E}_I \rho_I^{\ell+1}\\
&\geq  1-\sum_{\ell=0}^\infty\frac{2^{-(2\ell+1)}}{\ell+1}\binom{2\ell}{\ell} \bar{B}(\ell;N,1-p)\\
&=1- \sum_{\ell=0}^N\frac{2^{-(2\ell+1)}}{\ell+1}\binom{2\ell}{\ell} \bar{B}(\ell;N,1-p)\\
&= 1- Q(N, p)
\end{align}
{\bf Qingyun NOTE: I observe numerically that $Q(N,p)$ could be between $0$ and $1$ in a wide range, and $Q(N,p)$ is monotonically increasing with $N$, and monotonically decreasing with $p$, with $Q(N,1) = 0$, $\lim_{N\rightarrow \infty}\lim_{p\rightarrow 0} Q(N,p) = 1.$
.}
\end{proof}

\paragraph{Beta function and $Q(N,p)$}
If we define the complete beta function as 
\[
\mathcal{B}(a,b) := \int_0^1 t^{a-1}\,(1-t)^{b-1}\,dt.
\]
then
\[
\binom{n}{k} = \frac{1}{(n+1) \mathcal{B}(n-k+1, k+1)}
\]
the incomplete beta function as
\[
\mathcal{B}(x;\,a,b) = \int_0^x t^{a-1}\,(1-t)^{b-1}\,dt.
\]
then the regularized incomplete beta function is defined in terms of the incomplete beta function and the complete beta function:
\[
I_x(a,b) = \frac{\mathcal{B}(x;\,a,b)}{\mathcal{B}(a,b)}.
\]
then
\[
I_p(k,N-k-1)= \bar{B}(k-1;N,p) = \sum_{n=k}^{N} (1-p)^n p^{N-n} {N \choose n}
\]
and
\[
I_{1-p}(k+1, N-k) = 1 - I_p(N-k, k+1) = \bar{B}(k;N,1-p).
\]
Let $\mu = N \cdot (1-p)$ and 
$\sigma = \sqrt{N \cdot p \cdot (1-p)}$.

We note that, for large $N$, from the classical De Moivre–Laplace theorem on normal approximation to the binomial, 
\[
{N \choose k}\, p^k q^{N-k} \approx \frac{1}{\sqrt{2 \pi Np(1-p)}}\,e^{-\frac{(k-Np)^2}{2Np(1-p)}}
\]
therefore, 
\[
I_{1-p}(\ell+1, N-\ell) = \bar{B}(\ell;N,1-p)  \approx \bar{\Phi}( (\ell - \mu)/\sigma ) ,
\]
where $\Phi$ denotes the standard Normal $N(0,1)$ CDF
and $\bar{\Phi} \equiv 1-\Phi$ 
the corresponding survivor function;

\begin{eqnarray*}
Q(N,p) 
&\equiv& \sum_{\ell=0}^N\frac{2^{-(2\ell+1)}}{\ell+1}\binom{2\ell}{\ell} \bar{B}(\ell;N,1-p) \\
&=& \sum_{\ell=0}^N\frac{2^{-(2\ell+1)}}{\ell+1} \frac{1}{(2\ell+1) \mathcal{B}(\ell+1, \ell+1)}  I_{1-p}(\ell+1, N-\ell)\\
&\approx& \sum_{\ell=0}^N\frac{1}{2(\ell+1)} \frac{1}{\sqrt{\frac{\pi}{2} N}}\,e^{-\frac{(\ell-\frac{N}{2})^2}{\frac{N}{2}}}  \bar{\Phi}( (\ell - N \cdot (1-p))/\sqrt{N \cdot p \cdot (1-p)} )\\
&\approx& \int_0^1 \frac{1}{2(Nt+1)} \frac{1}{\sqrt{\frac{\pi}{2} N}}\,e^{- 2N(t-\frac{1}{2})^2}  \bar{\Phi}( \frac{\sqrt{N}(t - (1-p))}{\sqrt{ p \cdot (1-p)}} ) \,dt
\end{eqnarray*}

Numerically, we can see that the first term in the sum is always $0.5.$
Also we know that by definition $Q(N, 1) = 1$ for any $N$.
From numerically calculation we guess that
$Q(N,p)$ is monotonically increasing with $N$, and monotonically decreasing with $p$.
\[
Q(N,p') \geq Q(N,p), \quad, p>p'
\]
and 
$\lim_{N\rightarrow \infty}\lim_{p\rightarrow 0} Q(N,p) = 1.$


\begin{lemma}
Define:
\[
Q(N,p) \equiv \sum_{\ell=0}^N\frac{2^{-(2\ell+1)}}{\ell+1}\binom{2\ell}{\ell} \bar{B}(\ell;N,1-p).
\]
Then, 
\[
Q(N,p') \geq Q(N,p), \quad, p>p'
\]
and 
\[
\lim_{N\rightarrow \infty}\lim_{p\rightarrow 0} Q(N,p) = 1.
\]
\end{lemma}


\begin{proof}
The first term in the sum is $0.5$.

(Sketch). 
{\bf The sum has the lower bound $Q \geq 1/10$ just by inspecting
the first 10 terms (say), which are all positive and all greater
than $0.01$ (say). The following argument shows that later terms
add up to at least $\Omega(N^{-1/2})$.}
First, we know that
\[
\bar{B}(\ell;N,1-p) \equiv  \sum_{n=0}^{N-\ell-1} (1-p)^n p^{N-n} {N \choose n}.
\]

Let $\mu = N \cdot (1-p)$ and 
$\sigma = \sqrt{N \cdot p \cdot (1-p)}$.

We note that, for large $N$ 
\[
\bar{B}(\ell;N,1-p)  \approx \bar{\Phi}( (\ell - \mu)/\sigma ) ,
\]
where $\Phi$ denotes the standard Normal $N(0,1)$ CDF
and $\bar{\Phi} \equiv 1-\Phi$ 
the corresponding survivor function;
this is the classical De Moivre theorem on normal approximation to the binomial. It can be made precise in many ways
that have been well documented over the centuries.
Roughly speaking, in terms of variable $z = (\ell - \mu)/\sigma$,
we have
\[
\bar{B}(\ell;N,1-p)  \approx \left \{ \begin{array}{ll}
1 & z < 3 \\
0 & z > 3 .
\end{array}
\right .
\]

Hence, defining for $m = \mu - 3 \sigma$
\[
\bar{B}(\ell;N,1-p) \approx 1, \mbox{ for } \ell \leq m.
\]
Now of course
\begin{eqnarray*}
Q(N,p) &\geq& \sum_{\ell=0}^m\frac{2^{-(2\ell+1)}}{\ell+1}\binom{2\ell}{\ell} \bar{B}(\ell;N,1-p) \\
&\geq& \sum_{\ell=0}^m\frac{2^{-(2\ell+1)}}{\ell+1}\binom{2\ell}{\ell} .
\end{eqnarray*}
Now from DeMoivre's theorem (case $p=1/2$, $N=2\ell$) we also know that
\[
    2^{-2\ell} { 2 \ell \choose \ell} = bin(\ell; 2 \ell, 1/2) \sim \sqrt{\frac{1}{\ell \cdot \pi}},
\]
where $bin(k;N,p)$ is the usual binomial probability mass function.
We obtain
\[
  \sum_{\ell=0}^m\frac{2^{-(1)}}{\ell+1} \ell^{-1/2} \geq 
  \sum_{\ell=m/2}^m\frac{2^{-(1)}}{\ell+1} \ell^{-1/2}  \geq 
  Const  \cdot m^{-1/2}
\]

\end{proof}





\subsection{Bounds on $\frac{{\mathbb E}_I  \| \psi \|^{-1}_{\ell_2(I)} }{\| \psi \|^{-1}_{\ell_2({\cal T})}}$}
\begin{lemma}
\label{TaylorSqrtInv}
For $ 0 \leq x\leq 1,$
\begin{align}
(1-x)^{-1/2}
=\sum_{\ell=0}^\infty\frac{1}{2^{2\ell}}\binom{2\ell}{\ell}\left(x\right)^{\ell}
\end{align}
\end{lemma}

\begin{proof}
From generalized binomial theorem, we know that for $ 0 \leq x\leq 1,$
\begin{equation}
(1-x)^{-1/2}
=\sum_{\ell=0}^\infty\binom{-1/2}{\ell}(-x)^\ell
\end{equation}

\begin{align}
\binom{-1/2}{\ell}(-1)^\ell
&=\frac{-\frac12(-\frac12-1)(-\frac12-2)\cdots(-\frac12-\ell+1)}{\ell!}\\
&=\frac{1}{2^\ell\ell!}1\cdot3\cdot5\cdots(2\ell-1)\\
&=\frac{1}{2^\ell\ell!}\frac{(2\ell)!}{2^{\ell}(\ell)!}\\
&=\frac{1}{2^{2\ell}}\binom{2\ell}{\ell}
\end{align}
\end{proof}

\begin{lemma}
\label{TaylorSqrtInvExpect}
\begin{equation} 
1+\sum_{\ell=1}^\infty\frac{1}{2^{2\ell}}\binom{2\ell}{\ell}\bar{B}(\ell-1;N,1-p) \geq  \frac{{\mathbb E}_I  \| \psi \|^{-1}_{\ell_2(I)} }{\| \psi \|^{-1}_{\ell_2({\cal T})}} \geq 1 .
\end{equation}
where 
\[
\bar{B}(\ell-1;N,1-p) \equiv  \sum_{n=0}^{N-\ell} (1-p)^n p^{N-n} {N \choose n} 
\]
\end{lemma}
\begin{proof}
For any fixed support $J\subset {\cal T},$ 
\begin{equation} 
 \frac{ \| \psi \|^{-1}_{\ell_2(J)} }{\| \psi \|^{-1}_{\ell_2({\cal T})}} 
 = \sqrt{ \frac{ \| \psi \|^2_{\ell_2({\cal T})} }{\| \psi \|^2_{\ell_2(J)}}} \geq 1 .
\end{equation}
Therefore, 
\begin{equation} 
\frac{{\mathbb E}_I  \| \psi \|^{-1}_{\ell_2(I)} }{\| \psi \|^{-1}_{\ell_2({\cal T})}}  
 \geq   1 .
\end{equation}

On the other hand, let 
\begin{equation} 
{\mathbb E}_I (\frac{ \| \psi \|_{\ell_2(I)} }{\| \psi \|_{\ell_2({\cal T})}} )^{-1}
 = {\mathbb E}_I ( 1- \frac{ \| \psi \|^2_{\ell_2({\cal T}-I)} }{\| \psi \|^2_{\ell_2({\cal T})}})^{-1/2}.
\end{equation}

For a fixed subset $J \subset [n]$ define 
the ratio $\rho_J \equiv  \frac{ \| \psi \|^2_{\ell_2({\cal T}-J)} }{\| \psi \|^2_{\ell_2({\cal T})}}$; it 
obeys $0 \leq \rho_J \leq 1$.


\begin{align}
(1-\rho_J)^{-1/2}
=\sum_{\ell=0}^\infty\frac{1}{2^{2\ell}}\binom{2\ell}{\ell}\left(\rho_J\right)^{\ell}
\end{align}
Now with $I$ a random subset as earlier, we
induce a random variable $\rho_I$.
\begin{align}
{\mathbb E}_I(1-\rho_J)^{-1/2}
=\sum_{\ell=0}^\infty\frac{1}{2^{2\ell}}\binom{2\ell}{\ell}{\mathbb E}_I\left(\rho_J\right)^{\ell}
\end{align}
Modify the argument behind Lemma \ref{lem:BGPropLemmaSub} so it applies to ${\mathbb E}_I \rho_I^{\ell+1}$. In the new argument the set $I$ is everywhere replaced by its complement $T-I$. The conclusion is correspondingly modified so that $p$ in the final formula is replaced by $q \equiv 1-p$.
We obtain:
\[
 \bar{B}(\ell-1;n,1-p)    \leq {\mathbb E}_I \rho_I^{\ell} \leq    1,
\]
where, if $k \in \{0,1,\dots,N\}$,
$\bar{B}(k-1;N,p) \equiv  \sum_{n=k}^{N} (1-p)^n p^{N-n} {N \choose n}  = 1- \sum_{n=0}^{k-1} (1-p)^n p^{N-n} {N \choose n}$, while if $k \geq N$, $\bar{B}(k) = 0$, and if $k < 0$,
$\bar{B}(k) =1$.
Then
\[
\bar{B}(\ell-1;N,1-p) \equiv  \sum_{n=0}^{N-\ell} (1-p)^n p^{N-n} {N \choose n} 
\]
\begin{align}
{\mathbb E}_I(1-\rho_I)^{1/2}
&=\sum_{\ell=0}^\infty\frac{1}{2^{2\ell}}\binom{2\ell}{\ell}{\mathbb E}_I\left(\rho_J\right)^{\ell}
\\
&\geq  \sum_{\ell=0}^\infty\frac{1}{2^{2\ell}}\binom{2\ell}{\ell}\bar{B}(\ell-1;N,1-p)\\
&=1+\sum_{\ell=1}^\infty\frac{1}{2^{2\ell}}\binom{2\ell}{\ell}\bar{B}(\ell-1;N,1-p)
\end{align}

\end{proof}





\subsection{Numerically calculate ${\mathbb E}_I  \| \psi_I \|_2/\| \psi \|_2$}

We numerically compute $Q(N,p)$, the result is in the following table. Compare the table with the tables of MC approximation, we see that our previous bound is still too loose.
\begin{table}[]
\begin{tabular}{|p{3cm}||llllllll}
\hline
\multicolumn{9}{|c|}{$Q(N,p)$} \\
\hline
  $p=$ &0   & 0.1   & 0.2       &   0.5     &   0.9     &   0.99     &  0.999 &1      \\
 \hline\hline
$N=100$ &0.9439 & 0.9406 & 0.9370 & 0.920  & 0.8172 & 0.3358 & 0.0482 & 0 \\
\hline
$N=500$ & 0.9748  & 0.9734 & 0.9718 & 0.9643 & 0.9199 & 0.7302 & 0.209 & 0 \\
\hline      
\end{tabular}
\label{Q_table}
\end{table}

Then we use MC approximation to calculate ${\mathbb E}_I  \| \psi_I \|_2/\|  \psi \|_2$ for a few different $\psi.$
We will know that
\begin{itemize}
\item 
When $\psi = e_0$,  ${\mathbb E}_I  \| (e_0)_I \|_2=p$ 

\item
When $\psi = e_0+e_1$,  
\[
{\mathbb E}_I  \|(e_0+e_1)_I \|_2 /\| e_0+e_1 \|_2 = \sqrt{\frac12} {\mathbb E}_I  \| (e_0+e_1)_I \|_2 = 
{\mathbb E}_I \sqrt{\frac{\sum_{i=\{0,1\}} I_i}{2}}=p^2+ 2p(1-p)\sqrt{\frac12}
\]
\item
We know that when $\psi = a =(1,\ldots,1)$, then
\[
{\mathbb E}_I  \| a_I \|_2/\| a \|_2={\mathbb E}_I \sqrt{\frac{\sum_{i=[N]} I_i}{N}} 
\]
Using law of large numbers, 
\[
\lim_{N\rightarrow \infty }\sqrt{\frac{\sum_{i=[N]} I_i}{N}}  = \sqrt{p}.
\]
\item
For fixed support $J$, we can choose $\psi = \sum_{i\in J} e_i,$ then $\| \psi_J \|_2/\|  \psi \|_2 =1.$
However, the expectation is not going to be one. Let $M := |J|,$
\[
{\mathbb E}_I \sqrt{\frac{\sum_{i\in J} I_i}{|J|}} = \sum_{k=0}^{M} (1-p)^{M-k} p^{k} {M \choose k}\sqrt{\frac{k}{M}}\leq \sqrt{p}
\]
The inequality can be seeing using central limit theorem, we have normal approximation  $\frac{\sum_{i\in J} I_i}{|J|} \sim N(p,  \frac{p(1-p)}{\sqrt{|J|}})$, then using Jensen's inequality, since square root fucntion is concave,
\[
{\mathbb E}_I \sqrt{\frac{\sum_{i\in J} I_i}{|J|}} \approx {\mathbb E}_G\sqrt{p+ \frac{p(1-p)}{\sqrt{|J|}}G}  \leq \sqrt{{\mathbb E}_G[p+ \frac{p(1-p)}{\sqrt{|J|}}G]} =\sqrt{p} 
\]
\end{itemize}
Using symmetry, we guess that the maximum value of ${\mathbb E}_I  \| \psi_I \|_2/\|  \psi \|_2$  is $\sqrt{p}$ instead of $1$, and the maximum value is achieved when $\psi$ is dense.

\begin{table}[]
\begin{tabular}{|p{3cm}||llllllll}
\hline
\multicolumn{9}{|c|}{MC approximation ($M=10^4$) of ${\mathbb E}_I  \| a_I \|_2/\| a \|_2$ where $a=(1,\ldots,1)$} \\
\hline
  $p=$ & 0   & 0.1   & 0.2       &   0.5     &   0.9     &   0.99     &  0.999 &1      \\
 \hline\hline
$N=100$ &0 & 0.3123 & 0.4460 & 0.7063  & 0.94865 & 0.9950 & 0.9995 & 1 \\
\hline
$N=500$ &0 &0.3153  & 0.4464 & 0.7068 & 0.94874 & 0.9950 & 0.9995 & 1\\
\hline      
\end{tabular}
\label{allone_table}
\end{table}


\begin{table}[]
\begin{tabular}{|p{3cm}||llllllll}
\hline
\multicolumn{9}{|c|}{MC approximation ($M=10^4$) of ${\mathbb E}_I  \|(e_0+e_1)_I \|_2 /\| e_0+e_1 \|_2$} \\
\hline
  $p=$ & 0   & 0.1   & 0.2       &   0.5     &   0.9     &   0.99     &  0.999 &1      \\
 \hline\hline
$N=500$ & 0 &0.1375 &	0.2667 &	0.6034 &	0.9365 &	0.9946 &	0.9994 & 1 \\
\hline      
\end{tabular}
\label{coordinatevector_table}
\end{table}



\subsection{Tight bound and stationary points of ${\mathbb E}_I  \| \psi_I \|_2/\| \psi \|_2$}
\begin{lemma}
\begin{equation} 
p \leq  \frac{{\mathbb E}_I  \| \psi \|_{\ell_2(I)} }{\| \psi \|_{\ell_2({\cal T})}} \leq \sqrt{p}  .
\end{equation}
where the lower bound is achieved by $1-$sparse vectors $\pm e_i$, the upper bound is achieve by $\{\frac{\sum_{i\in {\cal T}} \pm e_i}{\sqrt{N}}\}$.
Furthermore, all the stationary points of this objective are $\{\frac{\sum_{i\in J} \pm e_i}{\sqrt{|J|}}\}$ for different support $J\subset {\cal T}$. 
\end{lemma}
\begin{proof}
This problem can be reformulate as the projection pursuit with sphere constraint,
 \begin{eqnarray*}
    \min&   {\mathbb E}_I  \| \psi_I \|_2\\
    \mbox{ subject to }&   \| \psi \|_2 = 1.
\end{eqnarray*}
Then from \cite{bai2018subgradient} Proposition $3.3$, we get the result. 

The main idea of the proof is to calculate the projected gradient
for $q\in \{\frac{\sum_{i\in J} \pm e_i}{\sqrt{|J|}}\}$, when $|J| = M,$
\[
e_j^T  {\mathbb E}_I  \partial\| q_I \|_2 = q_j [\sum_{k=0}^{M} (1-p)^{M-k} p^{k} {M \choose k}\sqrt{\frac{k}{M}}] 
\]
so
\[
(I-q q^T)  {\mathbb E}_I  \partial\| q_I \|_2 = [\sum_{k=0}^{M} (1-p)^{M-k} p^{k} {M \choose k}\sqrt{\frac{k}{M}}]q-[\sum_{k=0}^{M} (1-p)^{M-k} p^{k} {M \choose k}\sqrt{\frac{k}{M}}]q = 0.
\]
then $q\in \{\frac{\sum_{i\in J} \pm e_i}{\sqrt{|J|}}\}$ is stationary points.
 
The other direction (all other points are not stationary) is implied by (the proof of) theorem $3.4$ in  \cite{bai2018subgradient}.
\end{proof}
\section{Supplementary: Landscape of  Expected Homogeneous Function over Bernoulli Support on Sphere}
In the following, we study the expected landscape for projection pursuit on sphere: $V_1$, $V_{2k}$, $V_{-1}$ 

 First, we calculate ${\mathbb E}| \psi^T X| $ for $X_t$ IID sampled from Bernoulli Gaussian $p N(0,1) +(1-p) \delta_0.$ Let $X_i = B_i Z_i$ be IID Bernoulli Gaussian, where $B_i$ is sampled from Bernoulli with parameter $p$, $Z_i$ is sampled from $N(0,1)$. Let $I$ be the support where $B_i =1$.

From now on, we denote three equivalent notation, and interchange them for the convenience of each context:
\[
\|\psi\|_{\ell_2(I)} = \|\psi_I\|_2 = \|\psi\cdot I\|_2
\]
\subsection{Expectation of Inner Product for Sparse Signal}
In the following, we study in detail the quantity  
\[
E{| \psi^T X| } = E{|\sum_{i} \psi_i X_i| }.
\]
We know when $X_i$ have variance $\sigma^2$,
\[
   E (\sum_i \psi_i X_i)^2 =  \sum_i   \psi_i^2 E(X_i)^2 = \sigma^2 \|\psi\|_2^2.
\]
The ratio indicates the sparsity level of the random variable $ \sum_i \psi_i X_i$
\[
\frac{ E{|\sum_{i} \psi_i X_i| }}{\sqrt{E (\sum_i \psi_i X_i)^2}} = \frac{ E{|\sum_{i} \psi_i X_i| }}{\sigma^2 \|\psi\|_2} 
\]
Now, we consider $E{|\sum_{i} \psi_i X_i| }$ in the general symmetric setting, where a lower bound can be derived.

\paragraph{Exact calculation for Bernoulli Gaussian}
\[
{\mathbb E}_X{|\sum_{i} \psi_i X_i| }  
= {\mathbb E}_B {\mathbb E}_G {|\sum_{i} \psi_i B_i G_i| }
= {\mathbb E}_I {\mathbb E}_G {|\sum_{i\in I} \psi_i G_i| }
\]
Leveraging the fact that linear transforms of Gaussians are also Gaussian, we get
\[
{\mathbb E}_G {|\sum_{i\in I} \psi_i G_i| } = 
\sqrt{\frac{2}{\pi}} \cdot\|\psi_I\|_2.
\]


\paragraph{Upper and lower bound for symmetric distribution}
It is worth commenting that we could still calculate the upper and lower bound of ${\mathbb E}{| \psi^T X| }$ in terms of ${\mathbb E}_I\| \psi_I\|_2$ for $X_t$ IID sampled from any Bernoulli symmetric $p G +(1-p) \delta_0$, where $G$ is a symmetric distribution. 


\begin{lemma} \label{lem:sym_Khintchine_inequality}
If we only know $\Xi_i$ are IID sampled from a symmetric distribution $F$ with unit variance, then we already have a lower bound,  
\[
     1/\sqrt{2}  \| a\|_2 \leq {\mathbb E} |\sum_i a_i \Xi_i| \leq \|a\|_2.
\] 
\end{lemma}
\begin{proof}[Proof of Lemma \ref{lem:sym_Khintchine_inequality}]
The upper bound come from Cauchy inequality.
 
Now we derive the lower bound. We write $\Xi_i = \sigma_i |\Xi_i|$, where $s_i =\pm 1$ with equal probability since $\Xi_i$ are symmetric RV, and $\{s_i,|\Xi_i|\}$  are all independent random variables. We could use Khintchine inequality,


\begin{eqnarray*}
{\mathbb E}_\Xi |\sum_i a_i \Xi_i| &=& E_{|\Xi_i|}E_{s_i} |\sum_i a_i s_i |\Xi_i||\\
&\geq&  
  1/\sqrt{2}  \sum_i a_i^2  E_{|\Xi_i|} |\Xi_i|^2 \\
  &=&   1/\sqrt{2}  \|a\|_2^2.
\end{eqnarray*}
\end{proof}

Let $\psi_I = a$, we have a corollary for Bernoulli Symmetric case,
\[
 {\mathbb E}_I \|\psi_I\|_2  \geq {\mathbb E}_X{|\sum_{i} \psi_i X_i| }  
\geq \sqrt{\frac{1}{2}} \cdot {\mathbb E}_I \|\psi_I\|_2.
\]


\begin{theorem}
Let $X_i = B_i \Xi_i$ be IID, where $B_i$ is sampled from Bernoulli with parameter $p$, let $I$ be the support where $B_i =1$. And $\Xi_i$ is variance $1$, sampled from: 
\begin{itemize}
    \item $N(0,1)$
   
    \item general symmetric distribution with variance $1$.
\end{itemize}

Then 
\begin{itemize}
    \item(for Bernoulli Gaussian:)
    \[
{\mathbb E}_X{|\sum_{i} \psi_i X_i| }  
= \sqrt{\frac{2}{\pi}} \cdot {\mathbb E}_I \|\psi_I\|_2
\]
    \item (for Bernoulli symmetric R.V. :) 
    \[
{\mathbb E}_I \|\psi_I\|_2 \geq {\mathbb E}_X{|\sum_{i} \psi_i X_i| }  
\geq \sqrt{\frac{1}{2}} \cdot {\mathbb E}_I \|\psi_I\|_2.
\]
\end{itemize}

\end{theorem}

\subsection{Expectation over Bernoulli Support}

We previously considered the identity
\begin{equation} \label{eq:startingPoint}
    {\mathbb E} | \sum_{i \in {\cal T}}  \psi_i X_i | = \sqrt{\frac{2}{\pi}} \cdot {\mathbb E}_I \| \psi \|_{\ell_2(I)},
\end{equation}
where $X$ is a Bernoulli-Gaussian RV $BG(p;0,1)$,
and where $I$ is a random subset of the domain ${\cal T}$
determined by Bernoulli-$p$ coin tossing.

Let $J \subset {\cal T}$ be the support of $X$, where $  N:=|{\cal T}|$, $N_J:=|J|$. Let $\psi_J : = \psi\cdot 1_J$ denote the elementwise product of vector $\psi$ with the indicator vector of subset $J$. Then we can consider our problem on the space ${\cal T}$ by default, and rewrite for simplicity
\[
\| \psi_J \|_{2} : = \| \psi_J \|_{\ell_2({\cal T})} = \| \psi \|_{\ell_2(J)}.
\]

This leads us to consider the following ratio:
\[
  V^J_k(\psi) := \frac{\| \psi \|_{\ell_2(J)}^{k}}{\| \psi \|_{\ell_2({\cal T})}^{k}},
 \]
 and its expectation over all Bernoulli random subset,
\[
  V_k(\psi) := \frac{{\mathbb E}_I \| \psi \|_{\ell_2(I)}^{k}}{\| \psi \|_{\ell_2({\cal T})}^{k}},
 \]
 where $I$ is again a random subset.
We remark that the lower bound of $V_k(\psi)$ is the optimal value of the optimization problem:
 \begin{eqnarray*}
    \min&   {\mathbb E}_I  \| \psi_I \|^k_2\\
    \mbox{ subject to }&   \| \psi \|_2 = 1,
\end{eqnarray*}
and the upper bound of $V_k(\psi)$ is the optimal value of the optimization problem
 \begin{eqnarray*}
    \max&   {\mathbb E}_I  \| \psi_I \|^k_2\\
    \mbox{ subject to }&   \| \psi \|_2 = 1.
\end{eqnarray*}




\begin{lemma} \label{lem:BGPropLemmaSub_fix}
Let $\psi$ denote a vector in ${\mathbb R}^N$. For fixed support $J\subset [N]$ then for any $k\neq 0$, the deterministic quantity $V^J_{k}(\psi)$ satisfies:
\[
0 \leq V^J_{k}(\psi) \leq 1.
\]
The upper bound is approached by the vectors $\psi  \in \{  \frac{1}{\sqrt{N}}\sum_{j\in J}\pm e_j\}$, where  $\| \psi_J \|_2^k/\|  \psi \|_2^k =1$,
 and the lower bound is approached by 
any $\psi$ that has all of its entries on $J$ to be zero.
\end{lemma}

\begin{lemma} \label{lem:BGPropLemmaSub_2}
Let $\psi$ denote a vector in ${\mathbb R}^N$.
\[
V_2(\psi) =   p.
\]
\end{lemma}



\begin{theorem} 
Let $\psi$ denote a vector in ${\mathbb R}^N$, then
\[
p \leq V_{1}(\psi) \leq V_{1}(\frac{\sum_{j\in [N]}\pm e_j}{\sqrt{N}} ) \leq \sqrt{p} .
\]
the lower bound is approached by on-sparse vectors $\psi \in \{ \pm e_i, i\in [N]\}$, and the upper bound is approached by 
$\psi  \in \{  \frac{1}{\sqrt{N}}\sum_{j\in [N]}\pm e_j\}$.

Furthermore, all the stationary points of $V_{1}(\psi)$ are $\{\frac{\sum_{i\in J} \pm e_i}{\sqrt{N_J}}\}$ for different support $J\subset {\cal T}$, where $\{ \pm e_i, i\in [N]\}$ are the global minimizers, and $\{  \frac{1}{\sqrt{N}}\sum_{j\in [N]}\pm e_j\}$ are the global maximizers. And for $J$ with $1<N_J<N$, $\{\frac{\sum_{i\in J} \pm e_i}{\sqrt{N_J}}\}$ are saddle points with value
\[
V_1(\frac{\sum_{i\in J} \pm e_i}{\sqrt{N_J}}) = {\mathbb E}_I \sqrt{\frac{\sum_{i\in J} 1_{I_i}}{N_J}} = \sum_{j=0}^{N_J} (1-p)^{N_J-j} p^{j} {N_J \choose j}\sqrt{\frac{j}{N_J}}.
\]
\end{theorem}
Remark:
Specifically,
\[
V_1( e_i) = p.
\]
\[
V_1(\frac{ e_i+e_j}{\sqrt{2}}) = p(\sqrt{2}+(1-\sqrt{2}p)).
\]



\begin{theorem} 
For $k\geq 2$, let $\psi$ denote a vector in ${\mathbb R}^N$, then
\[
p^k \leq V_{2k}(\frac{\sum_{j\in [N]}\pm e_j}{\sqrt{N}} ) \leq V_{2k}(\psi) \leq p.
\]
the upper bound is approached by on-sparse vectors $\psi \in \{ \pm e_i, i\in [N]\}$, and the lower bound is approached by 
$\psi  \in \{  \frac{1}{\sqrt{N}}\sum_{j\in [N]}\pm e_j\}$.

Furthermore, all the stationary points of $V_{2k}(\psi)$ are $\{\frac{\sum_{i\in J} \pm e_i}{\sqrt{N_J}}\}$ for different support $J\subset {\cal T}$, where $\{ \pm e_i, i\in [N]\}$ are the set of global maximizers, and $\{  \frac{1}{\sqrt{N}}\sum_{j\in [N]}\pm e_j\}$ are the set of all the global minimizers. And for $J$ with $1<N_J<N$, $\{\frac{\sum_{i\in J} \pm e_i}{\sqrt{N_J}}\}$ are saddle points with value
\[
V_{2k}(\frac{\sum_{i\in J} \pm e_i}{\sqrt{N_J}}) = {\mathbb E}_I (\frac{\sum_{i\in J} 1_{I_i}}{N_J})^k = \sum_{j=0}^{N_J} (1-p)^{N_J-j} p^{j} {N_J \choose j}(\frac{j}{N_J})^k.
\]
\end{theorem}






\subsection{Upper and Lower bound on Expectation of Norm over Bernoulli Support}
\paragraph{Tight bound on $V_1$ and $V_{2k}$ using mean and variance of $\frac{\sum_{i\in J} 1_{I_i}}{N_J}$}

Now notice that 
\[
\frac{\sum_{i\in J} 1_{I_i}}{N_J} 
\]
has mean $p$ and variance $\frac{p(1-p)}{N_J}$.
We could define a zero mean unit variance random variable $g^J$ as a function of $\{1_{I_i}\mid i\in J\}$ 
\[
g^J := \left(\frac{\sum_{i\in J} 1_{I_i}}{N_J}-p\right)/ \left( \sqrt{\frac{p(1-p)}{N_J}} \right)
\]
namely, ${\mathbb E}_I \left( g^I\right)=0, {\mathbb E}_I \left( g^I\right)^{2}=1$.
then
\[
\frac{\sum_{i\in J} 1_{I_i}}{N_J} := p + \sqrt{\frac{p(1-p)}{N_J}} g^J = p(1 + \sqrt{\frac{1-p}{p N_J}} g^J)
\]
then
\[
{\mathbb E}_I \sqrt{\frac{\sum_{i\in J} 1_{I_i}}{N_J}} = {\mathbb E}_I\sqrt{p- \sqrt{\frac{p(1-p)}{N_J}}g^I} 
= \sqrt{p} {\mathbb E}_I\sqrt{1+ \sqrt{\frac{1-p}{pN_J}}g^I}  
\]
\[
{\mathbb E}_I \left(\frac{\sum_{i\in J} 1_{I_i}}{N_J}\right)^k 
= p^k {\mathbb E}_I\left({1+ \sqrt{\frac{1-p}{pN_J}}g^I}\right)^k 
\]
\begin{lemma}
\label{TaylorSqrt}
For $ 0 \leq x\leq 1,$
\begin{align}
(1-x)^{1/2}
=1-\sum_{\ell=0}^\infty\frac2{\ell+1}\binom{2\ell}{\ell}\left(\frac x4\right)^{\ell+1}
\end{align}
\end{lemma}

\begin{lemma}[Taylor expansion for $V_1$]
\label{TaylorSqrt_Expect}
Using Taylor expansion, asymptotically when $N_J\rightarrow \infty$, 
\begin{eqnarray*}
\frac{1}{\sqrt{p}}{\mathbb E}_I \sqrt{\frac{\sum_{i\in J} 1_{I_i}}{N_J}} 
&=& {\mathbb E}_I\left({1+ \sqrt{\frac{1-p}{pN_J}}g^I}\right)^{1/2} \\
&=& 1-\sum_{\ell=0}^\infty\frac2{\ell+1}\binom{2\ell}{\ell}2^{-2(\ell+1)}\left( \sqrt{\frac{1-p}{pN_J}}\right)^{\ell+1}{\mathbb E}_I \left( g^I\right)^{\ell+1}\\
&=& 1- 2^{-3}({\frac{1-p}{pN_J}}) +O\left( ({\frac{1-p}{pN_J}})^2 \right).
\end{eqnarray*}
\end{lemma}

\begin{lemma}[Taylor expansion for $V_{2k}$]
\label{TaylorSqrt_Expect_k}
Using Taylor expansion, asymptotically when $N_J\rightarrow \infty$, 
\begin{eqnarray*}
\frac{1}{p^k}{\mathbb E}_I ({\frac{\sum_{i\in J} 1_{I_i}}{N_J}})^k  
&=& {\mathbb E}_I\left({1+ \sqrt{\frac{1-p}{pN_J}}g^I}\right)^{k} \\
&=& 1 +\sum_{\ell=1}^{k} \binom{k}{\ell} ({\frac{1-p}{pN_J}})^{\ell/2} {\mathbb E}_I (g^I)^{\ell}\\
&=& 1 + \frac{k(k-1)}{2} ({\frac{1-p}{pN_J}}) +O\left( ({\frac{1-p}{pN_J}})^2 \right).
\end{eqnarray*}
\end{lemma}
Therefore, we know what when $N_J>>1$, 
$V_1(\frac{\sum_{i\in J} 1_{I_i}}{N_J})$ is close to its upper bound $\sqrt{p}$, 
and for $k\geq 2$, 
$V_k(\frac{\sum_{i\in J} 1_{I_i}}{N_J})$ is close to its lower bound $p^k$. Namely 
\[
{\mathbb E}_I \sqrt{\frac{\sum_{i\in J} 1_{I_i}}{N_J}} \approx \sqrt{p},
\]
\[
{\mathbb E}_I (\frac{\sum_{i\in J} 1_{I_i}}{N_J})^k \approx p^k.
\]





To gain geometric intuition, we visualize $V_1$ in $2-$dimensional sphere in figure~\ref{fig-landscape-V1}.






\begin{figure} 
\centering
\includegraphics[width=.7\textwidth]{./figures_BD/ratio/Full_compare_with_baseline.png}
\caption{The value of $V_1(\psi)$ for different $\psi$, $x-$axis is the sparsity parameter $p$, $y-$axis is $V_1(\psi)$. }
\end{figure}

\begin{figure} 
\centering
\includegraphics[width=.7\textwidth]{./figures_BD/ratio/L4_plot.png}
\caption{The value of $V_4(\psi)$ for different $\psi$, $x-$axis is the sparsity parameter $p$, $y-$axis is $V_4(\psi)$. }
\end{figure}



\begin{figure} 
\centering
\includegraphics[width=.7\textwidth]{./figures_BD/ratio/L8_plot.png}
\caption{The value of $V_8(\psi)$ for different $\psi$, $x-$axis is the sparsity parameter $p$, $y-$axis is $V_8(\psi)$. }
\label{fig-V4}
\end{figure}



\subsection{Proofs of Upper and Lower Bounds}


\begin{proof}[Proof of Theorem~\ref{thm:BGPropLemmaNorm}]
First, we show that the upper and lower bound value we give is achievable.
\begin{itemize}
\item 
When $\psi = e_0$,  
\[
{\mathbb E}_I  \| (e_0)_I \|_2=p\| (e_0)\|_2+(1-p)\| 0\|_2 = p.
\]
\item
When $\psi = \frac{1}{\sqrt{N}}\sum_{j\in [N]}\pm e_j,$ then  
\[
V_1(\psi) = {\mathbb E}_I \sqrt{\frac{\sum_{i\in [N]} 1_{I_i}}{N}} \leq   \sqrt{{\mathbb E}_I \frac{\sum_{i\in [N]} 1_{I_i}}{N}} = \sqrt{p}.
\]
The inequality comes from Jensen's inequality, since square root function is concave.
\item
For fixed support $J$, when $\psi = \frac{1}{\sqrt{N_J}}\sum_{i\in J} e_i,$  then $\| \psi_J \|_2/\|  \psi \|_2 =1.$
However, the expectation is going to be smaller for $p<1$. 
\[
V_1(\psi) = {\mathbb E}_I \sqrt{\frac{\sum_{i\in J} 1_{I_i}}{N_J}} = \sum_{k=0}^{N_J} (1-p)^{N_J-k} p^{k} {N_J \choose k}\sqrt{\frac{k}{N_J}}.
\]
\end{itemize}

This problem can be reformulated as projection pursuit with sphere constraint,
 \begin{eqnarray*}
    \min&   {\mathbb E}_I  \| \psi_I \|_2\\
    \mbox{ subject to }&   \| \psi \|_2 = 1.
\end{eqnarray*}
Then from \cite{bai2018subgradient} Proposition $3.3$, we get the result. 

The main idea of the proof is to calculate the projected gradient
for $q  \in \{\frac{\sum_{i\in J} \pm e_i}{\sqrt{N_J}}\}$, when $N_J = M,$
\[
e_j^T  {\mathbb E}_I  \partial\| (q)_I \|_2 = q_j [\sum_{k=0}^{M} (1-p)^{M-k} p^{k} {M \choose k}\sqrt{\frac{k}{M}}] ,
\]
therefore,
\[
  {\mathbb E}_I  \partial\| (q)_I \|_2 =  [\sum_{k=0}^{M} (1-p)^{M-k} p^{k} {M \choose k}\sqrt{\frac{k}{M}}] q,
\]
so
\[
(I-q q^T)  {\mathbb E}_I  \partial\| q_I \|_2 = [\sum_{k=0}^{M} (1-p)^{M-k} p^{k} {M \choose k}\sqrt{\frac{k}{M}}]q -[\sum_{k=0}^{M} (1-p)^{M-k} p^{k} {M \choose k}\sqrt{\frac{k}{M}}]q  = 0.
\]
then $q \in \{\frac{\sum_{i\in J} \pm e_i}{\sqrt{N_J}}\}$ are stationary points.
 
The other direction (all other points are not stationary) is implied by (the proof of) Theorem $3.4$ in  \cite{bai2018subgradient}.
\end{proof}





\begin{proof}[Proof of Lemma~\ref{TaylorSqrt}]
From generalized binomial theorem, we know that for $ 0 \leq x\leq 1,$
\begin{equation}
(1-x)^{1/2}
=\sum_{\ell=0}^\infty\binom{1/2}{\ell}(-x)^\ell.
\end{equation}

\begin{align}
\binom{1/2}{\ell}
&=\frac{\frac12(\frac12-1)(\frac12-2)\cdots(\frac12-\ell+1)}{\ell!}\\
&=\frac{(-1)^{\ell-1}}{2^\ell\ell!}1\cdot3\cdot5\cdots(2\ell-3)\\
&=\frac{(-1)^{\ell-1}}{2^\ell\ell!}\frac{(2\ell-2)!}{2^{\ell-1}(\ell-1)!}\\
&=\frac{(-1)^{\ell-1}}{\ell2^{2\ell-1}}\binom{2\ell-2}{\ell-1},
\end{align}
then
\begin{align}
(1-x)^{1/2}
&=1-\sum_{\ell=1}^\infty\frac2\ell\binom{2\ell-2}{\ell-1}\left(\frac x4\right)^\ell\\
&=1-\sum_{\ell=0}^\infty\frac2{\ell+1}\binom{2\ell}{\ell}\left(\frac x4\right)^{\ell+1}.
\end{align}
\end{proof}







\begin{proof}[Proof of Lemma~\ref{TaylorSqrt_Expect}]
\begin{eqnarray*}
{\mathbb E}_I\sqrt{1- \sqrt{\frac{1-p}{pN_J}}g^I}  
&=& 1-\sum_{\ell=0}^\infty\frac2{\ell+1}\binom{2\ell}{\ell}2^{-2(\ell+1)}\left( \sqrt{\frac{1-p}{pN_J}}\right)^{\ell+1}{\mathbb E}_I \left( g^I\right)^{\ell+1}\\
&=& 1- 2^{-3}({\frac{1-p}{pN_J}}) - O(({\frac{1-p}{pN_J}})^2)
\end{eqnarray*}
Using the central limit theorem, when $N_J\rightarrow \infty$, we have normal approximation for $g^J$ so that
\[
\frac{\sum_{i\in J} 1_{I_i}}{N_J} \sim N(p,  \sqrt{\frac{p(1-p)}{N_J}}),
\]
then for $G\sim N(0,1),$ asymptotically when $N_J\rightarrow \infty$,
\[
{\mathbb E}_I \sqrt{\frac{\sum_{i\in J} 1_{I_i}}{N_J}} \approx  \sqrt{p} {\mathbb E}_G\sqrt{1- \sqrt{\frac{1-p}{pN_J}}G} =
\sqrt{p} \left( 1- 2^{-3}(1-p)^{2}  N_J^{-1} + O((1-p)^{4}  N_J^{-2})\right),
\]
where
\begin{eqnarray*}
{\mathbb E}_I\sqrt{1- \sqrt{\frac{1-p}{pN_J}}g^I}  
&\approx &{\mathbb E}_G\sqrt{1- \sqrt{\frac{1-p}{pN_J}}G}\\
&=& 1-\sum_{\ell=0}^\infty\frac2{\ell+1}\binom{2\ell}{\ell}2^{-2(\ell+1)}\left( \sqrt{\frac{1-p}{pN_J}}\right)^{\ell+1}{\mathbb E}_G \left( G\right)^{\ell+1}\\
&=& 1-\sum_{\ell \mbox{ is odd}}^\infty\frac2{\ell+1}\binom{2\ell}{\ell}2^{-2(\ell+1)}\left( \sqrt{\frac{1-p}{pN_J}}\right)^{\ell+1} (\ell)!!\\
&=& 1-\sum_{s=1}^\infty\frac1{s}\binom{4s-2}{2s-1}2^{-4s} (2s-1)!! (p^{-1}-1)^{s}  N_J^{-s}\\
&=& 1-\sum_{s=1}^\infty\frac1{s}\frac{(4s-2)!}{(2s-1)!(2s-1)!}2^{-4s} \frac{(2s-1)!}{2^{s-1}(s-1)!} (p^{-1}-1)^{s}  N_J^{-s}\\
&=& 1-\sum_{s=1}^\infty\frac{(4s-2)!}{(2s-1)!(s)!}2^{-5s+1}  (p^{-1}-1)^{s}  N_J^{-s}.
\end{eqnarray*}
\end{proof}



\subsection{Bound on Harmonic Expectation }
\paragraph{Bounds on  $V_{-1}(\psi)=\frac{{\mathbb E}_I  \| \psi \|^{-1}_{\ell_2(I)} }{\| \psi \|^{-1}_{\ell_2({\cal T})}}$}
\begin{lemma}
\label{TaylorSqrtInv}
For $ 0 \leq x\leq 1,$
\begin{align}
(1-x)^{-1/2}
=\sum_{\ell=0}^\infty\frac{1}{2^{2\ell}}\binom{2\ell}{\ell}\left(x\right)^{\ell}
\end{align}
\end{lemma}

\begin{proof}
From generalized binomial theorem, we know that for $ 0 \leq x\leq 1,$
\begin{equation}
(1-x)^{-1/2}
=\sum_{\ell=0}^\infty\binom{-1/2}{\ell}(-x)^\ell
\end{equation}

\begin{align}
\binom{-1/2}{\ell}(-1)^\ell
&=\frac{-\frac12(-\frac12-1)(-\frac12-2)\cdots(-\frac12-\ell+1)}{\ell!}\\
&=\frac{1}{2^\ell\ell!}1\cdot3\cdot5\cdots(2\ell-1)\\
&=\frac{1}{2^\ell\ell!}\frac{(2\ell)!}{2^{\ell}(\ell)!}\\
&=\frac{1}{2^{2\ell}}\binom{2\ell}{\ell}
\end{align}
\end{proof}

\begin{theorem}
\label{thm: TaylorSqrtInvExpect}
\begin{equation} 
  V_{-1}(\psi) = \frac{{\mathbb E}_I  \| \psi \|^{-1}_{\ell_2(I)} }{\| \psi \|^{-1}_{\ell_2({\cal T})}} \geq p^{-1/2} \geq 1+\frac{1}{2}(1-p) \geq 1 .
\end{equation}
\end{theorem}
\begin{proof}
For any fixed support $J\subset {\cal T},$ 
\begin{equation} 
 \frac{ \| \psi \|^{-1}_{\ell_2(J)} }{\| \psi \|^{-1}_{\ell_2({\cal T})}} 
 = \sqrt{ \frac{ \| \psi \|^2_{\ell_2({\cal T})} }{\| \psi \|^2_{\ell_2(J)}}} \geq 1 .
\end{equation}
Therefore, 
\begin{equation} 
\frac{{\mathbb E}_I  \| \psi \|^{-1}_{\ell_2(I)} }{\| \psi \|^{-1}_{\ell_2({\cal T})}}  
 \geq   1 .
\end{equation}

On the other hand, let 
\begin{equation} 
{\mathbb E}_I (\frac{ \| \psi \|_{\ell_2(I)} }{\| \psi \|_{\ell_2({\cal T})}} )^{-1}
 = {\mathbb E}_I ( 1- \frac{ \| \psi \|^2_{\ell_2({\cal T}-I)} }{\| \psi \|^2_{\ell_2({\cal T})}})^{-1/2}.
\end{equation}

For a fixed true subset $J \subset {\cal T}$ define 
the ratio $\rho_{{\cal T} -J} \equiv  \frac{ \| \psi \|^2_{\ell_2({\cal T}-J)} }{\| \psi \|^2_{\ell_2({\cal T})}}$; it 
obeys $0 \leq \rho_J \leq 1$. Assume $\rho_J < 1$


\begin{align}
(1-\rho_{{\cal T} -J})^{-1/2}
=\sum_{\ell=0}^\infty\frac{1}{2^{2\ell}}\binom{2\ell}{\ell}\left(\rho_{{\cal T} -J}\right)^{\ell}
\end{align}
Now with $J$ a random subset as earlier, we
induce a random variable $\rho_I$.
\begin{align}
{\mathbb E}_J(1-\rho_{{\cal T} -J})^{-1/2}
=\sum_{\ell=0}^\infty\frac{1}{2^{2\ell}}\binom{2\ell}{\ell}{\mathbb E}_J\left(\rho_{{\cal T} -J}\right)^{\ell}
\end{align}
We apply the bound on $V_{2\ell}$ for $\ell\geq 2$, where $p$ in the final formula is replaced by $ 1-p$.

We obtain for $\ell\geq 2$
\[
 (1-p)^\ell    \leq {\mathbb E}_J \rho_{{\cal T} -J}^{\ell} \leq    (1-p),
\]
\begin{align}
{\mathbb E}_J(1-\rho_{{\cal T} -J})^{-1/2}
&=\sum_{\ell=0}^\infty\frac{1}{2^{2\ell}}\binom{2\ell}{\ell}{\mathbb E}_J\left(\rho_{{\cal T} -J}\right)^{\ell}
\\
&\geq  \sum_{\ell=0}^\infty\frac{1}{2^{2\ell}}\binom{2\ell}{\ell} (1-p)^\ell\\
&=(1-(1-p))^{-1/2}\\
&= p^{-1/2}\\
\geq 1+\frac{1}{2}(1-p) 
\end{align}

\end{proof}



\section{Image deblur experiments}
We set up a ground truth inverse filter as $a^{-1} = (0.5, 1, 0.5)$, and numerically run $10$ independent experiments of finding the inverse filter by the convex problem 
\begin{equation}
 \begin{array}{ll}
  \underset{w\in V}{ \mbox{minimize} }  &  \frac{1}{2T+1}\|w*Y\|_{1,[-T,T]}\\
 \mbox{subject to}  
&  w_0 =1.
\end{array}
\tag{$P_{e_0}^T$ }
\end{equation}
We choose an accuracy $\epsilon = 10^{-3}$ and count the number of successfully accurate recovery of $a^{-1}$ where $\|w^\star -a^{-1}\|_1/\|a^{-1}\|_\infty<\epsilon$. We see a consistent sharp transition of success rate from $0$ to $1$ in a narrow parameter region.
\begin{figure} 
\centering
\includegraphics[width=.7\textwidth]{./figures_BD/phase_transition_detail.png}
\caption{The finite sample phase transition phenomenon, where $a^{-1} = (0.5, 1, 0.5)$,  $x-$axis is the sparsity level $p$, $y-$axis is the empirical probability $P(\|w^\star -a^{-1}\|_1/\|a^{-1}\|_\infty<10^{-3})$. Red part is failure, Blue part is success.}
\label{fig-PT}
\end{figure}
\subsection{$E{| \psi^T X| }$ for IID Bernoulli Symmetric $X$ }
we study in detail for the quantity  
\[
E{| \psi^T X| } = E{|\sum_{i} \psi_i X_i| }.
\]
We know when $X_i$ have variance $\sigma^2$,
\[
   E (\sum_i \psi_i X_i)^2 =  \sum_i   \psi_i^2 E(X_i)^2 = \sigma^2 \|\psi\|_2^2.
\]
The ratio indicate the sparsity level of random variable $ \sum_i \psi_i X_i$
\[
\frac{ E{|\sum_{i} \psi_i X_i| }}{\sqrt{E (\sum_i \psi_i X_i)^2}} = \frac{ E{|\sum_{i} \psi_i X_i| }}{\sigma^2 \|\psi\|_2} 
\]
Now, we consider $E{|\sum_{i} \psi_i X_i| }$ for general symmetric setting when lower bound can be derived.

\paragraph{Exact calculation for Bernoulli Gaussian}
Let $X_i = B_i Z_i$ be IID Bernoulli Gaussian, where $B_i$ is sampled from Bernoulli with parameter $p$, $Z_i$ is sampled from $N(0,1)$. Let $I$ be the support where $B_i =1$.
\[
{\mathbb E}_X{|\sum_{i} \psi_i X_i| }  
= {\mathbb E}_B {\mathbb E}_G {|\sum_{i} \psi_i B_i G_i| }
= {\mathbb E}_I {\mathbb E}_G {|\sum_{i\in I} \psi_i G_i| }
\]
Leveraging the invariance property for linear transform of Gaussian, we get
\[
{\mathbb E}_G {|\sum_{i\in I} \psi_i G_i| } = 
\sqrt{\frac{2}{\pi}} \cdot\|\psi_I\|_2.
\]


\paragraph{Symmetric distribution}

Dave Comment:
Put the generalization in the end.
Focus on Bernoulli Gaussian for the main part, and then talk about extension.


\begin{lemma} \label{lem:sym_Khintchine_inequality}
For we only know $\Xi_i$ are IID sampled from a symmetric distribution $F$ with unit variance, then we already have a lower bound,  
\[
     1/\sqrt{2}  \| a\|_2 \leq {\mathbb E} |\sum_i a_i \Xi_i| \leq \|a\|_2.
\] 
\end{lemma}
 \begin{proof}[Proof of Lemma \ref{lem:sym_Khintchine_inequality}]
 The upper bound come from Cauchy inequality.
 
Now we derive the lower bound. We write $\Xi_i = \sigma_i |\Xi_i|$, where $s_i =\pm 1$ with equal probability since $\Xi_i$ are symmetric RV, and $\{s_i,|\Xi_i|\}$  are all independent random variables.  
we could use Khintchine inequality,
here is the Wiki link for more detail:
\begin{quote}
\href{https://en.wikipedia.org/wiki/Khintchine_inequality}{Khintchine inequality}.
\end{quote}




\begin{eqnarray*}
{\mathbb E}_\Xi |\sum_i a_i \Xi_i| &=& E_{|\Xi_i|}E_{s_i} |\sum_i a_i s_i |\Xi_i||\\
&\geq&  
  1/\sqrt{2}  \sum_i a_i^2  E_{|\Xi_i|} |\Xi_i|^2 \\
  &=&   1/\sqrt{2}  \|a\|_2^2.
\end{eqnarray*}
\end{proof}

Let $\psi_I = a$, we have a corollary for Bernoulli Symmetric case,
   \[
 {\mathbb E}_I \|\psi_I\|_2  \geq {\mathbb E}_X{|\sum_{i} \psi_i X_i| }  
\geq \sqrt{\frac{1}{2}} \cdot {\mathbb E}_I \|\psi_I\|_2.
\]


\begin{theorem}
Let $X_i = B_i \Xi_i$ be IID, where $B_i$ is sampled from Bernoulli with parameter $p$, let $I$ be the support where $B_i =1$. And $\Xi_i$ is variance $1$, sampled from: 
\begin{itemize}
    \item $N(0,1)$
   
\item general symmetric distribution with variance $1$.
\end{itemize}

Then 
\begin{itemize}
    \item(for Bernoulli Gaussian:)
    \[
{\mathbb E}_X{|\sum_{i} \psi_i X_i| }  
= \sqrt{\frac{2}{\pi}} \cdot {\mathbb E}_I \|\psi_I\|_2
\]
\item (for Bernoulli symmetric R.V. :) 
    \[
{\mathbb E}_X{|\sum_{i} \psi_i X_i| }  
\geq \sqrt{\frac{1}{2}} \cdot {\mathbb E}_I \|\psi_I\|_2.
\]
\end{itemize}

\end{theorem}








\section{Introduction}
\subsection{Blind Deconvolution}
Suppose we are interested in an underlying time series ${\bf x} = (x(t))$ which we cannot observe directly. What we can observe is  ${\bf y} = {\bf a} * {\bf x}$ where ${\bf a}$ is an unknown `blurring' filter. Blind deconvolution is the problem of recovering ${\bf x}$ merely from the observed ${\bf y}$ without knowing either ${\bf a}$ or ${\bf x}$. 

This problem occurs naturally in seismology and
digital communications
as well as astronomy, satellite imaging, and computer vision.

In its most ambitious form, the problem is literally impossible; there are simply too few data and too many unknowns. Indeed, imagine that ${\bf x}$, ${\bf y}$ and ${\bf a}$ all have $N$ entries; we observe only $N$ pieces of information (${\bf y}$) but there are $2N$ unknowns (${\bf x}$ and ${\bf a}$). 
Nevertheless, in  some (not all) fields, heuristic approaches
have occasionally 
led to consistent  success in isolated applications;
presumably such success stories exploit specialized assumptions -- although not always in
an explicit or recognized way, and not with rigorous understanding.

 This paper, in contrast,  will exhibit a set of 
assumptions enabling practical algorithms for blind deconvolution,
backed by rigorous theoretical analysis.

\subsection{The Promise of Sparsity}

{Central to our approach
is an assumption about the}
{\it sparsity of the signal ${\bf x}$ to be recovered}, 
in which case the problem 
{can be called} \emph{sparse blind deconvolution}. Sparse signals {-- i.e. signals} having relatively few nonzero entries, arise frequently in many fields, including seismology, microscopy, astronomy, neuroscience spike identification. Even in more abstract settings such as representation learning for computer vision, 
it surfaces in recently popular research trends, 
such as single-channel convolutional dictionary learning   \cite{bristow2013fast, heide2015fast,zhang2017global,zhang2018structured}. 

The sparsity of ${\bf x}$ \-- if it holds \-- would constrain the recovery problem significantly; and so possibly, sparsity can play a role
in enabling useful solutions to an otherwise hopeless problem.

An inspiring precedent can be found in modern commercial medical imaging, 
where sparsity of an image's wavelet coefficients enables 
MRIs from fewer observations than unknowns.
Taking fewer observations speeds up data collection,
a principle known as compressed sensing, which
already benefits tens of millions of patients yearly.

\subsection{Translating Heuristics into Effective Algorithms}

For sparsity to reliably enable blind deconvolution, there are two
apparent hurdles. First, develop an objective function which promotes
sparsity of the solution. Second, develop an algorithm 
which can reliably optimize the objective.

Many sparsity-promoting objectives have 
been proposed over the years; typically they 
imply non-convex optimization problems. Indeed
sparsity is quantified by the $\ell_0$
pseudo norm $\|{\bf x}\|_0 = \#\{t: x(t) \neq 0\}$, which is the limit of
$\| {\bf x} \|_p^p$ as $p \rightarrow 0$
of concave $\ell_p$ pseudo-norms where $p <1$.

Traditionally, non-convex problems have been viewed
by mathematical scientists with skepticism;
for such problems, gradient descent and its various refinements
lack any guarantee of effectiveness. 
Still, the lack of guarantees has not stopped engineers from trying!

A noticeable success in blind signal processing
was scored in digital communications,
where blind equalization today 
benefits billions of smartphone users.
Blind equalization is a form of blind deconvolution 
where one exploits the known discrete-valued 
nature of the signal ${\bf x}$ (for example the signal 
entries might take only two values $\{-1,1\}$).
Practitioners found that if an initial guess of 
the equalizer (i.e. our inverse filter ${\bf a}^{-1}$) is `fairly good' 
(in engineer-speak `opening the eye' so that a `hint' 
of the `digital constellation` becomes
`visible'), then certain `discreteness-promoting' on-line gradient methods
can reliably `focus' the result better and better 
and allow reliable recovery.

Our work identifies an 
analogous phenomenon in the sparsity-promoting blind deconvolution
setting, however it exposes and crystallizes the phenomenon in a rigorous 
and dependably exploitable form.

Namely, we show that if sparsity of ${\bf x}$ holds,
and if an initial guess of the filter ${\bf a}$
is `fairly good' in a precise sense, 
then a specific convex optimization
algorithm will accurately recover 
both the filter and the original signal\footnote{As we explain below, recover means: recover up to rescaling and time shift.}.

In retrospect, our insights on sparsity-promoting blind deconvolution
can be cross-applied to explain the major successes of discreteness-promoting
blind equalization in modern digital communications.
Namely, a direct variation of our arguments 
provide a related convex optimization problem 
for discrete-valued signals which rigorously converts a 
a rough initial approximation into precise recovery.

In our view these new arguments clear away some persistent
fog, mystery and misunderstandings in blind signal processing; and pave
the way for future success stories.


\subsection{Prior Work}
\begin{comment}
\paragraph{Non-convexity for blind deconvolution problems}
As both the sparse signal and the forward filter ${\bf a}$ are unknown, the problem is bilinear in ${\bf a}$ and $X$. And the expression ${\bf a}*{\bf x}$ has scaling and shift symmetry for $({\bf a},{\bf x})$.
Therefore, blind deconvolution is often formulated as a non-convex problem. Lots of previous works \cite{kuo2019geometry, kuo2020geometry, lau2019short} studied the that problem of recovering short $a$ and sparse $x$ as an non-convex optimization problem.

\Dave{The previous paragraph is jumbled and also out of order. If it has any value or content it should come later.}
\end{comment}

\paragraph{Searching for an inverse filter that promotes desired output properties}
Instead of trying to recover ${\bf a}$ and ${\bf x}$ together from ${\bf a}*{\bf x}$, 
we could formulate this problem as looking for an approximate inverse  filter ${\bf w}$ so that the output ${\bf w}*{\bf y}$ exhibits {extremal} properties. 
 {Under this formulation, 
our goal is to find} ${\bf w}\neq 0$, so that  
\begin{equation}
 \begin{array}{ll}
 \underset{{\bf w}\neq 0}{\mbox{optimize}}   & J({\bf w} \star {\bf y}) \label{JoptBD}
\end{array}
\end{equation}
where the functional $J$ quantifies the properties we seek to promote.
(Depending on $J$, we might  {either prefer its maximum or minimum}  ).
 
 \newcommand{{\bf z}}{{\bf z}}
 {Working} in
exploration seismology, 
Wiggins \cite{wiggins1978minimum} adopted this approach
with the normalized $4$-norm, $J({\bf z}) = J_{4,2} = {\|{\bf z}\|_4}/{\|{\bf z}\|_2}$
and gave a few successful data-processing case studies. 
His {successful} examples all
clearly exhibit sparsity, although this was not discussed at the time.
Other objectives 
considered at that time included $J_{2,1}({\bf z}) = \|{\bf z}\|_2/\|{\bf z}\|_1$
and $J_{\infty,2}({\bf z}) = \|{\bf z}\|_\infty/\|{\bf z}\|_2$ \cite{cabrelli1985minimum}.

It was fully understood at that time
that output property optimization could succeed in principle,
if one did not have to worry about an effective algorithm.
\cite{donoho1981minimum} showed that 
if the signal ${\bf x}$ is a realization of
independent and identically distributed
entries from any nonGaussian distribution,  optimizing
$J$ of the output ${\bf w} \star {\bf y}$ is successful in the large-$N$ setting \-- as long as the  functional  $J$ belongs to a certain large family of non-Gaussianity measures,
for example including $J_{4,2}$ and $J_{2,1}$ as well as many others. 
\footnote{Sparsity is of course a form of non-Gaussianity,
this very explicitly in the Bernoulli-Gaussian mixture model considered below.}

The issue left unresolved in those days was how to solve such
optimization problems. Indeed optimizations like $J_{4,2}$ are badly nonconvex,
as we see clearly by rewriting the $J_{4,2}$ problem as
\[
 \begin{array}{ll}
 \underset{{\bf w}}{\mbox{maximize}}   &  
 \|{\bf w} * {\bf y}\|_4 
 \\
\mbox{subject to} 
& \|{\bf w}* {\bf y}\|_2 = 1.
\end{array}
  \]
The theory cited above derived favorable properties of a would-be procedure which truly finds the optimum of a badly nonconvex objective. It clarifies that  blind deconvolution is possible in principle but does not by itself help us algorithmically, i.e. in practice. In the intellectual climate of the time, solving badly nonconvex optimization problems was considered a pipe dream, a time-wasting charade for non-serious people.

Even  today, blind deconvolution 
continues to be studied as an non-convex optimization problem;
see recent work
\cite{kuo2019geometry, kuo2020geometry, lau2019short}, who study the  problem of recovering short $a$ and sparse $x$ 
  
  \paragraph{Blind equalization}
  In digital communications, 
  the transmitted signal ${\bf x}$ can be viewed as 
  {\it discrete-alphabet valued;}
  for example, in PAM signaling, where $x(t) \in \{\pm 1\}$, and QAM signaling, where the signal alphabet has equally spaced points on unit sphere in complex space \cite{kennedy1992blind,ding2000fast}.  
  
  \cite{vembu1994convex} considered the non-convex problem
   \[
 \begin{array}{ll}
 \underset{{\bf w}}{\mbox{maximize}}   &  
 \|{\bf w} * {\bf y}\|_8 
 \\
\mbox{subject to} 
& \|{\bf w}* {\bf y}\|_2 = 1
\end{array}
  \]
If the data ${\bf y}$ were preprocessed to be serially uncorrelated,
this optimization is effectively of the earlier form $J_{8,2}$.

The authors attack this nonconvex problem
using projected gradient descent and give suggestive 
experimental results. They apparently view the $\ell_8$ norm objective as an approximation of the  $\ell_\infty$ norm objective $\|{\bf w} * {\bf y}\|_\infty $.
Later, \cite{ding2000fast} used linear programming to solve the $\infty$ norm problem directly.


\begin{comment}

\paragraph{Sparse projection pursuit and dictionary learning}
Sparse projection pursuit is the problem of finding  a sparse projection $d^T Y$ time from a high dimensional data matrix $Y\in {\mathbb R}^{n\times M}$. Here $Y= D X$ is a matrix product of dictionary $D\in {\mathbb R}^{n\times d}$ and sparse coefficient $X\in {\mathbb R}^{d\times M}$. 

\Dave{I can't follow either what you are trying to do in the paragraph below
or what you are trying to say. You mention many different things, with no references definitions or discussion and all very telegraphically.  Why is it even here? What does it add? I don't see it.}

\Dave{In general it's really a huge mistake to put 
inconclusive aimless material at the beginning of a document.
Readers very quickly abandon reading and decide that you're at least confusing
and possibly confused.

In your writing you don't want to put in a lot of extraneous references that 
only distract from your central thread.
}
The projection pursuit problem was first studied by Friedman and collaborators in a series of important works \cite{friedman1974projection}, \cite{friedman1981projection} and \cite{friedman1987exploratory} as a tool for exploratory high dimensional data analysis. Then several statisticians like Huber \cite{huber1985projection}  and Jones \cite{jones1987projection}  also studied this problem. 

The projection pursuit problem is also related to blind source separation, independent component analysis (ICA) and the cocktail-party problem.
In the case of an orthogonal dictionary $D$, we could simply transform the problem into finding its inverse $D^{-1} = D^T$, such that $D^T Y = X$ sparse. A work by Daniel Spielman and John Wright\cite{spielman2012exact} used sparse projection pursuit formulation to find the orthogonal basis of $D$ one-by-one, by proposing a set of rule to select the initial guess $v$ and give theoretical guarantee of the convex method. In this work, due to this common population problem, the phase transition phenomenon would also be applicable to dictionary learning, as the first phase transition result for dictionary learning.

The sparse projection pursuit formulation is to solve the following linear-constrained convex optimization problem for some vector $u$:
\[
 \begin{array}{ll}
 \underset{q }{\mbox{minimize}}   & \frac{1}{M}\|q^T Y\|_{1}\\
\mbox{subject to}  & u^T q =1.
\end{array}
  \]
  This convex problem is also used in \cite{spielman2012exact} to solve sparse dictionary learning problem.
  
\end{comment}

 
 \paragraph{Searching for a projection with desired output properties} 
Here is another setting for property-promoting
output optimization.
We have a data matrix $Y \in {\bf R}^{n,p}$ \-- which we think of as
$n$ points in ${\bf R}^p$,
and we have a unit vector $w \in {\bf R}^p$
called the projection direction.  Our
output vector $Y\cdot w$ contains the 
projection of the $n$-points on the projection
direction q; it has $n$ entries.
We seek `interesting' projections; i.e. directions where the 
projection displays some structure. We
adopt a functional $J$ which measures properties we seek to promote
in the output, and we seek to solve:
\begin{equation}
 \begin{array}{ll}
 \underset{\|w\|=1}{\mbox{optimize}}   & J(Y \cdot w). \label{JoptPP}
\end{array}
\end{equation}

This was implemented by  [Friedman and Tukey, 1974], who proposed a
functional that promotes `clumping' or 'clustering' of the output.
They called it {\it projection pursuit} and
were motivated by exploratory  high  dimensional  data  analysis;
for $p$-dimensional data involve $p>2$ and we can't 
easily get a visual sense of what's in the data. 
It was hoped at the time that looking at selected
low-dimensional projections might lead to better insights.
For the most part, such hopes for exploration of high-dimensional data
never materialized.

However, output optimization of this type has proven to be
useful in important problems in blind signal processing,
where $Y$ has known structure that can be exploited systematically.

In {\it blind source separation}
we observe $Y = XA$, $Y \in {\bf R}^{n,p}$, $X \in {\bf R}^{n,p}$ and
$A \in {\bf R}^{p,p}$ is an invertible matrix. 
(We don't observe $X$ or $A$ separately).

Think of the $X$ matrix as containing columns giving
successive time samples of $p$ clean source signals, for example $p$
individual acoustic signals. These signals arrive
at array of $p$ spatially distributed acoustic sensors, each
of which records the acoustic information it receives.
The matrix $Y$ contains in its
columns what was obtained by each of the $p$ different sensors.
In general, each sensor receives information from each of the sources.
This is colorfully called the {\it cocktail party problem}, referring to
the setting where $X$ records the sources are human speakers at a cocktail party,
and $Y$ records what is heard at various locations in a room.
Each column of $Y$ then contains a
superposition of different speakers; while
we would prefer to separate these and pay attention to 
just the ones of most immediate interest to us.

In this separation problem, sparsity of the signal might be valuable.
Suppose that each individual speaker is listening quite a bit
and so not speaking much of the time. Then the columns of $X$
are each sparse. On the other hand, if there are many people
in the room, the room
as a whole may still always be noisy, and so each
column of $Y$ may be fully dense.
Assuming $A$ is invertible, and the vector $w$
obeys $Aw= e_j$, then $Y \cdot w$ extracts column $j$ of $X$,
which will be sparse. Hence, we may hope that the projection
pursuit principle, with an appropriate measure $J$, might
identify the `sparse' projections we seek.  

Ju Sun, Qu Qing, Yu Bai, John Wright, Zibulevsky and Pearlmutter \cite{sun2015complete,sun2016complete,bai2018subgradient,zibulevsky2000blind} proposed the optimization
problem
 \[
 \begin{array}{ll}
 \underset{w }{\mbox{minimize}}   & \| Y w\|_{1}\\
\mbox{subject to}  & \|w\|_2 =1.
\end{array}
  \]
Assuming the $Y$ data pre-processed so that $\frac{1}{n} Y'Y=I_p$ 
this is equivalent to projection pursuit (\ref{JoptPP}) 
applied to the objective $J_{2,1}$. Notably, this is 
again a highly non-convex optimization problem..
The authors proposed projected gradient descent
and recited some favorable empirical results.
 
There is a close relation between the
 blind deconvolution optimization (\ref{JoptBD}) 
 and the projection pursuit optimization (\ref{JoptPP}). Indeed, if the $Y$ matrix 
 is filled in from an observed time series appropriately, then 
 output optimization in 
 blind deconvolution and in projection pursuit are
 essentially identical. Namely, let ${\bf y}^{ser} = (y^{ser}(t))_{t=1}^N$ denote a time series of interest to us,
 and  $Y^{mat} = (Y^{mat}_{i,j})$ 
 denote an $n \times p$ matrix where $n = N-p$ constructed 
 using ${\bf y}^{ser}$ like so:
 \[
  Y^{mat}_{i,j} = y^{ser}(p+i-(j-1)) , \qquad 1 \leq i \leq n = N-p; \; 1 \leq j \leq p.
  \]
  Now suppose the filter vector 
  ${\bf w}$ in the blind deconvolution optimization 
  and the projection direction $w$ in the projection pursuit optimization
  are chosen identically. Then the blind deconvolution output objective 
  $J({\bf w} \star {\bf y}^{ser})$ is identical to the projection pursuit objective
  $J(Y^{mat} w)$, except for
  possibly different treatment of the first $p$ entries of ${\bf y}^{ser}$.
 
 \paragraph{Convex Projection Pursuit}
 
 In view of the connection between blind deconvolution and
 projection pursuit, and in view of our results in this paper,
 it is quite interesting to consider the
 work of \cite{spielman2012exact} and \cite{gottlieb2010matrix}.
\begin{comment}
They call their problem sparsity-promoting projection pursuit,
 but they depart from the PP optimization framework (\ref{JoptPP}).
 \Sun{Their paper did not mention the name "projection pursuit", so I comment out this sentence.}
\end{comment}
They propose to solve the following 
linear-constrained convex optimization problem.
Given an $n \times p$ 
data matrix $Y$ and a constraint vector $u$,
they propose to solve:
\[
 \begin{array}{ll}
 \underset{w }{\mbox{minimize}}   & \|Y w\|_{1}\\
\mbox{subject to}  & u^T w =1.
\end{array}
  \]
As it turns out,  when the matrix $Y$ in this problem and
${\bf y}$ in the time series deconvolution problem
are related by the  $Y^{mat}$-${\bf y}^{ser}$
construction just mentioned, the objective we propose in this paper
is essentially identical, when $\tilde{{\bf a}} = u$. As we will show,
our setting permits much more thorough studies and more penetrating analyses, {and we find that success in 
sparse blind deconvolution is more
broadly prevalent than one might have expected, based on
earlier analyses such as \cite{spielman2012exact}
or \cite{sun2015complete,sun2016complete,bai2018subgradient,zibulevsky2000blind} }.


 


\subsection{Mathematical setup}
\paragraph{Sequence Space, and Filtering }
To make our results concrete, let's discuss things formally.
Let ${\mathbf X}$ denote the collection of {\it bilaterally infinite sequences}
${\bf x} = (x(t): t = 0, \pm 1, \pm 2, \dots)$; for short
we call such objects {\it bisequences}. Then $\ell_1({\mathbb Z}) \subset {\mathbf X}$
denotes the collection of bisequences 
obeying $\| {\bf x} \|_1 = \sum_t |x(t)| < \infty.$
For a bisequence ${\bf x}$ we denote time reversal operator
$t \leftrightarrow -t$ by ${\bf x}^{\dagger}$.
For whole number $k > 0$ let ${\mathbf X}_k$ denote the subspace
of bisequences supported in $-k \leq t \leq k$. We sometimes
abuse notation: for a bisequence ${\bf x}$ we might write
${\bf x} = (1, .3)$ when we really mean ${\bf x} = (\ldots,0,0,1,.3,0,0,\ldots)$.

Let $\star$ denote the convolution 
product on pairs of bisequences in $\ell_1({\mathbb Z})$
\-- $( {\bf x} \star {\bf y} ) = \sum_u x(t-u)y(u)$. 
Let ${\bf e}_0$ denote the  `delta'
or `Kronecker' bisequence: ${\bf e}_0(t) = 1_{\{t=0\}}$;
${\bf e}_0$ is the unit of convolution. The convolution inverse
of ${\bf x}$ $\--$ ${\bf x}^{-1}$ $\--$ is a bisequence
obeying $ {\bf x} \star {\bf x}^{-1}  = {\bf e}_0$. For example,
the filter ${\bf x} = (\dots,0,1,1/2,1/4,1/8,\ldots)$ 
anchored at the time origin so ${\bf x}(0)=1$, has inverse
${\bf x}^{-1} = (\dots,0,1,-1/2,0,\dots)$, again anchored at the time origin.
Abusing notation we may simply write ${\bf x}^{-1} = (1,-1/2)$.


Our approach to blind deconvolution searches 
among candidates for a filter ($\equiv$ bisequence) that extremizes a certain objective function. We then
show that the extremal is in fact the desired inverse filter
to our (unknown) true underlying filter. Hence, it helps know
conditions under which an inverse filter actually exists! 

For a bisequence ${\bf x} \in \ell_1({\mathbb Z})$,
we define the Fourier transform 
$\hat{{\bf x}}(w) = \mathcal{F} {\bf x} (w) \equiv \sum_t x(t) \exp\{ i 2\pi w t \}$.
For the inverse transform, we use
$ x(t) = (\mathcal{F}^{-1} \hat{{\bf x}} ) (t) =  \frac{1}{2\pi} \int_0^{2\pi} \hat{{\bf x}}(w) \exp\{ -i 2\pi w t \} dw$.


\begin{lemma}[Wiener's lemma]
If ${\bf a}\in \ell_1({\mathbb Z})$, and also  
$(\mathcal{F} {\bf a})(\widetilde{w}) \neq 0$, $\forall \widetilde{w}\in T$,
then an inverse filter exists in $\ell_1({\mathbb Z})$. 
The bilaterally infinite sequence defined formally by
$${\bf a}^{-1} := \mathcal{F}^{-1}(\frac{1}{\mathcal{F} {\bf a}})$$
exists as an element of $\ell_1({\mathbb Z})$
and obeys ${\bf a}^{-1}*{\bf a} = {\bf e}_0.$
\end{lemma}
In the engineering literature, we
say that the bisequence ${\bf a}$ has so-called $Z$-transform 
$A(z)$, defined by:
\[
 \begin{array}{ll}
A(z) =  \sum_{t=-\infty}^{\infty} a_{t} z^{-t}.
\end{array}
  \]

Evaluating $A$ on the unit circle in the complex plane, at $z$ of the form $z=\exp\{ -i 2\pi w \}$, we see that  the 
$Z$-transform is effectively the Fourier transform $A = {\cal F} {\bf a}$. 
Applying Wiener's lemma, we see that, if 
${\bf a} \in \ell_1$ and $ \min_w |A(\exp\{ - 2\pi i w \}) | > 0$,
i.e. $A$ is never zero on the unit circle, then ${\bf a}^{-1} \in \ell_1$ and $1/A$ is the $Z$-transform of ${\bf a}^{-1}$.


\paragraph{Finite-sample observation model and finite-length inverse filter }
In searching for an inverse filter, our  initial  results assume
existence of a {\it finite-length} inverse. Namely, we assume that ${\bf a}\in \ell_1({\mathbb Z})$ is a forward filter with an inverse filter ${\bf a}^{-1} \in \ell_1({\mathbb Z})$ supported in a centered window of radius $k$.

In practice we only have a finite dataset! Suppose that 
there is an underlying bisequence ${\bf y} \in {\mathbf X}$
of the form ${\bf y} = {\bf a}* {\bf x}$, and let ${\bf y}^{[N]}$ denote the
restriction to an $N$-long centered window 
${\cal T} = \{ -T,\dots, T\}$ of radius $T$ and size $N=2T+1$.

Our goal is seemingly to 
recover ${\bf x}$ or ${\bf a}$ from the observed data ${\bf y}^{[N]}$.
However, in statistical theory we generally 
don't expect to exactly recover the true underlying 
representation (i.e. the generating ${\bf a}$ and ${\bf x}$) exactly.
Our goal is instead to find an inverse filter 
${\bf w} \in \ell_1^k$ such that the convolution ${\bf w}*{\bf y}$ 
would be close\footnote{modulo time shift and rescaling}  to ${\bf x}$, and where the closeness improves with
increasing data size $N \rightarrow \infty$.

\paragraph{Finite-length filtering and practical algorithms}

While our analysis framework concerns bisequences (bilaterally infinite sequences), our data have finite length (as just mentioned). The algorithms we discuss are often {\it motivated by} convolutions on bisequences; however, they reduce {\it in practice} to truncated convolutions involving finite data windows. A certain ambiguity is helpful for efficient communication. Suppose we have ${\bf y}^{[N]}$, an N-long observed window of ${\bf y}$, and we also have ${\bf w}$, a k-long filter, by which we mean a bisequence nonzero only within a 
fixed $k$-long window. We might encounter discussion both of 
${\bf w} \star {\bf y}^{[N]} $ as well as
${\bf w} \star {\bf y}$, both using the same filter ${\bf w}$. In the first case, we would actually be thinking of ${\bf y}^{[N]}$ as zero-padded out to a bisequence, so that both situations involve bisequence convolutions. 

Finite $N$ effects are important in practice but tedious to discuss. It can be important to account for {\it end effects} in truncated convolution. In a setting where we initially
think to consider a norm $\| {\bf w} \star {\bf y}^{[N]} \|_{\ell_p({\mathbb Z})}$, we might instead next think to rather consider the windowed norm
$\| {\bf w} \star {\bf y}^{[N]} \|_{\ell_p(\{-T,\dots,T\})}$,
while finally we realize $\| {\bf w} \star {\bf y}^{[N]} \|_{\ell_p(\{-T+k,\dots,T-k\})}$ is more correct for our purposes, as it includes only the terms which do not suffer from truncation of the convolution.



\paragraph{Algorithmic formulation}

Under assumptions we will be making, the sequence ${\bf x}$ underlying our observed data will be either exactly sparse -- having few nonzeros -- or approximately so. Moreover, there will be either an exact length $k$ inverse filter ${\bf w}$, or approximately such. It follows that the filter output ${\bf w} \star {\bf y}$ is sparse.
This suggests the would-be optimization principle 
\[
 \begin{array}{ll}
 \underset{{\bf w}\neq 0}{\mbox{minimize}}   & \|{\bf w} \star {\bf y}^{[N]}\|_{\ell_0(\{-T+k,\dots,T-k\}})
\end{array}
  \]
where the $\ell_0$ quasi-norm simply 
counts the number of nonzero entries. Unfortunately, this objective, though well-motivated, is not suitable for numerical optimization.

Inspired by this, we perform convex relaxation of the $\ell_0$ norm, replacing it with the $\ell_1$ norm, 
which is convex.

We also need to fix the scale to get a unique output. One might think to constrain $\|{\bf w}\|_2 =1$, however, this would give a non-convex constraint and again is not suitable for effective algorithms.



We instead suppose given a rough initial approximation  
$\widetilde{{\bf a}}$ of the forward filter ${\bf a}$, and impose 
an $\ell_\infty$ constraint on the 
`pseudo-delta' ${\bf w}\star\widetilde{{\bf a}}$,
forcing it to `peak' at target entry $t$.
\begin{comment}
\begin{equation}
 \begin{array}{ll}
& \|\widetilde{{\bf a}}*{\bf w}\|_\infty = 1.
\end{array}
\end{equation} 

This intuition for the choice is based on the fact that $a^{-1}*a = {\bf e}_0$, we know $(a^{-1}*a)_0 = 1$ is a constraint on the largest entry. Therefore, $\|\widetilde{{\bf a}}*a^{-1}\|_\infty = 1$ would be true if  $\widetilde{{\bf a}} = a$. 

Based on shift and scaling symmetry, without loss of generality, we could assume that   $\widetilde{{\bf a}}* a^{-1}$ 
has $0$-th entry being the largest absolute value entry by shifting $\widetilde{{\bf a}}$, and set its value to $1$ by scaling $\widetilde{{\bf a}}$, then up to shift and scaling of $\widetilde{{\bf a}}$, 
\[
(\widetilde{{\bf a}}*a^{-1})_0 = \|\widetilde{{\bf a}}*a^{-1}\|_\infty =1
\]


Although this transformed $\ell_\infty$ sphere constraint is still a non-convex constraint, we can focus on one surface by fixing the $t$-th coordinate of entry with largest magnitude:
\end{comment}
\begin{equation}
 \begin{array}{ll}
&  (\widetilde{{\bf a}}*{\bf w})_t =1, \quad \|\widetilde{{\bf a}}*{\bf w}\|_\infty \leq 1.
\end{array}
\end{equation}
Combining these steps, we obtain a convex optimization problem 
associated to each possible target coordinate $t$:
\begin{equation}
 \begin{array}{ll}
 \underset{{\bf w} \in \ell_1^k}{ \mbox{minimize} }   & \frac{1}{N-2k} \| {\bf w} \star {\bf y}^{[N]}\|_{\ell_1(\{-T+k,T-k\})}\\
\mbox{subject to}  
&  (\widetilde{{\bf a}}*{\bf w})_t =1, \quad \|\widetilde{{\bf a}}*{\bf w}\|_\infty \leq 1.
\end{array}
\end{equation}

\begin{comment}

\end{comment}In the case of perfect shift symmetry of convolution ( linear convolution on bisequence, or circular convolution for finite observation ), we could design the problem to fully utilize the shift symmetry of objective  $\frac{1}{N} \| w*Y\|_{\ell_1({\cal T})}$, so that the convex problem is equivalent to a convex optimization problem with a simple linear constraint as the $\ell_\infty$ box constraint 
with a fixed location of largest entry at $0$ coordinate under shift:
\begin{equation}
 \begin{array}{ll}
\widetilde{{\bf a}}^{T} w^{\dagger} = \langle \widetilde{{\bf a}}, w^{\dagger}\rangle = (\widetilde{{\bf a}}*w)_0  =1, 
\end{array}
\end{equation}
where $w^{\dagger}$ is the time reversal of $w$.
\end{comment}

It will be convenient to reformulate slightly,  
hide consideration of end effects,
and force the peak to occur at target coordinate $t=0$.
Abusing notation somewhat, we then write:
\begin{equation}
 \begin{array}{ll}
 \underset{{\bf w} \in \ell_1^k}{ \mbox{minimize} }   & \frac{1}{N} \| {\bf w}* {\bf y} \|_{\ell_1^N}\\
\mbox{subject to}  
& \langle \widetilde{{\bf a}},  {\bf w}^{\dagger} \rangle =1, 
\end{array}
\tag{${C}_{\widetilde{{\bf a}}}$ }
\end{equation}

(Again ${\bf w}^\dagger$ denotes time-reversal of ${\bf w}$). In practice we
might truncate the convolution due to end effects, or truncate the window over which we take the norm, but we will hide such practical details in the coming material, for ease of exposition; they would not change our results. 

 
\paragraph{Stochastic models for sparse signals}
Although our algorithms make sense in the absence of any theory,
our theoretical results concern properties of our algorithm for data generated under a probabilistic generative model i.e. a {\it stochastic signal} model. 

Let $X = (X_t)$ be a bisequence
of independent identically distributed random variables indexed by $t \in {\mathbb Z}$, having a common marginal CDF $F = F_X$, such that $F(x) = 1 - F(-x)$. One realization is then a sequence ${\bf x}$ of the type discussed in earlier paragraphs.

We assume that $F$ has an atom at $0$ \-- $F = (1-p) H +p G$, where $H$ is the standard Heaviside distribution and $G$ is the standard Gaussian distribution.  We say that $F$ follows the \emph{Bernoulli$(p)$-Gaussian} model. 
Equivalently, $X_t$ is sampled IID from the \emph{Bernoulli$(p)$-Gaussian} distribution
$p N(0,1) +(1-p) \delta_0$. 

The iid process $X$ is of course ergodic. 
If ${\bf x}$ denotes one realization of $X$, then in a window $(x_t)_{-T}^T$ of length $N$,  $\approx p N$ nonzero values  will occur,
for large $N$. Consequently, if $p \ll 1$,
realizations from $X$
will empirically be sparse.

\newcommand{{\bf b}}{{\bf b}}
Let $Y = {\bf a} \star X$ denote the random bisequence produced as the output of convolution of the random signal $X$
with deterministic filter ${\bf a} \in \ell_1({\mathbb Z})$. More explicitly,
\[
    Y_t = \sum_u a(u) X_{t-u}.
\]
This defines formally a so-called {\it stationary linear process},  a classical object for which careful foundational results are well established.\footnote{In our case, we assume that $X$ is iid Bernoulli-Gaussian, so
$E |X_0| = p \cdot E |N(0,1)|$ is finite. Using this,
we can see that the sum in the display above, 
even though possibly containing an 
infinite number of terms, converges in various natural senses.}
Consider now filtering $Y$, by a length-$k$ filter 
${\bf w}$, producing the random bisequence $V = {\bf w} \star Y$.
The {\it end-to-end} filter ${\bf b} = {\bf w} \star {\bf a}$ 
is a well-defined element of $\ell_1({\mathbb Z})$; 
using it, we can represent 
the filtered output in terms of the underlying iid process $X$:
$V  = {\bf b} \star X$. 
This representation shows that the filtered output series $V$ is itself a well-defined stationary linear process, and moreover, since $E |X_0| < 1$ and $\| {\bf b} \|_{\ell_1} \leq \|{\bf w} \|_{\ell_1} \cdot  \| {\bf a} \|_{\ell_1} < \infty$, we have $E | V_0| < \| {\bf b} \|_{\ell_1} < \infty$.

Any such stationary linear process is ergodic.
By the ergodic theorem, the
large-$N$ limit of the objective 
will almost surely  be an expectation over $X$:
\begin{equation}
 \begin{array}{ll}
\lim_{N\rightarrow \infty}\frac{1}{N} \| {\bf w}*Y\|_{\ell_1^N}  = \lim_{N\rightarrow \infty}\frac{1}{N} \sum_{t\in {\cal T}^N}|({\bf b} \star X)_t|  =  {\mathbb E}_X|({\bf b} \star X)_0|={\mathbb E}|({\bf w}*Y)_0|.
\end{array}
\end{equation}
Consequently, 
the large-$N$ properties of our proposed algorithm
are driven by properties of the following optimization problem
{\it in the population}:
 \[
 \begin{array}{ll}
 \underset{{\bf w} }{\mbox{minimize}}   & {\mathbb E}|({\bf w}*Y)_0|\\
\mbox{subject to}  &  \langle \widetilde{{\bf a}},  {\bf w}^{\dagger} \rangle =1. 
\end{array}
  \]

  
  




\section{Main Results Overview}











\subsection{Main Result $1$: Phase Transition Phenomenon for Sparse Blind Deconvolution}

We have at last defined a convex optimization problem 
at the population level, which we will now want to study in detail.
In our studies, we can make various choices of the sparsity parameter $p$, of the underlying forward filter ${\bf a}$, and of the guess $\tilde{{\bf a}}$. The tuple $(p,{\bf a},\tilde{{\bf a}})$ defines in this way a kind of phase space. We can then study performance of the algorithm at different points in phase space. 

\newcommand{{\sc Exact}{\sc Recovery}}{{\sc Exact}{\sc Recovery}}
Consider this performance property: 
\begin{quotation}
{\sc Exact}{\sc Recovery} \, $\equiv$ \,  {\sl ``There is an unique solution of the population-based optimization problem --- modulo time shift and output rescaling --- and
this solution {\bf exactly} solves the blind deconvolution problem correctly. ''}
\end{quotation}
It probably seems too much to ask that such a property could {\sl ever} be true, i.e. could ever be true for even one 
choice of phase space tuple. After all the optimization problem doesn't have any apparent connection to blind deconvolution -- instead only to some sort of relaxation of the search for sparse output filters.


We will see that this phase space can be partitioned into two regions: one where the exact recovery property holds and its complement where the exact recovery property fails. Surprisingly the region where {\sc Exact}{\sc Recovery} \  holds is nonempty and can be appreciable. And it can be described in a clear and insightful way.

\paragraph{Surprising phase transition for a special case of sparse blind deconvolution}



We first illustrate the phase transition phenomenon in possibly the most elementary situation. Consider the special case when the forward filter ${\bf a}$ is the exponential decay filter:  ${\bf a} = (1, s, s^2, s^3, \ldots )$ with $|s|\leq 1$; then the inverse filter is a basic short filter  ${\bf a}^{-1} = ( 1, -s )$. 
\begin{theorem}[Population phase transition for exponential decay filter]
\label{thm: single root PT}
Consider a linear process $Y = {\bf a} * X$ with
\begin{itemize}
\item
${\bf a} = (1, s, s^2, s^3, \ldots )$ where $|s|\leq 1$, so ${\bf a}^{-1} = ( 1, -s )$;
\item 
$X_t$ is IID \emph{Bernoulli$(p)$-Gaussian}
$p N(0,1) +(1-p) \delta_0$.
\end{itemize}
Consider as initial approximation 
$\widetilde{{\bf a}} = {\bf e}_0 = ( 1, 0, 0,0, \ldots)$,
and the resulting 
fully specified population
optimization problem, 
with parameter tuple $(p,{\bf a},\tilde{{\bf a}})$:
 \[
 \begin{array}{ll}
 \underset{{\bf w} }{\mbox{minimize}}   & {\mathbb E}|({\bf w}*Y)_0|\\
\mbox{subject to}  & {\bf w}_0 =1.
\end{array}
\tag{${P}_1(e_0)$ }
\label{bd:population}
  \]
Let ${\bf w}^\star$ denote the (or simply some) solution.
Define the threshold
\[
p^\star  = 1-|s| .
\]
The property {\sc Exact}{\sc Recovery} experiences a phase transition at $p = p^\star$:
\begin{itemize}
    \item provided $p<p^\star$, then ${\bf w}^\star$ is uniquely defined and equal to ${\bf a}^{-1}$ up to shift and scaling ; and
    \item provided $p> p^\star$, then ${\bf w}^\star$ is not ${\bf a}^{-1}$ up to shift and scaling .
\end{itemize}
  \end{theorem}
  
  
  \begin{figure}[!tb]
\centering
\includegraphics[width=.8\textwidth]{./figures_BD/PT_BD_1root_none.png}
\caption{Finite-sample phase transition diagram; ground truth filter is ${\bf w}^\star =  ( 0, 1, -s )$ with $T=200$. Horizontal axis:  sparsity level $p$, ranging from $0.01$ to $0.99$; Vertical axis:  filter parameter $|s|$, ranging from $0$ to $0.99$.  The shaded attribute indicates the observed fraction of  
successful experiments at the given parameter combination from deep blue for 1.0 down to deep red for 0.0. Thus, the red region indicates failure of recovery and the blue region indicates success. The transition from success to failure is quite abrupt.}
\label{fig-PT-finite}
\end{figure}

We can empirically  observe that the population phase transition described in Theorem 1 
describes accurately the situation with finite-length signals.   The setting of Theorem 1,
defines a phase space by the numbers $(p,s)$. We can
sample the phase space according to a grid and then at each grid point, conducting a sequence of experiments like so:
\begin{itemize}
\item sample a realization of synthetic data $Y = {\bf a} \star X$ according to the stochastic signal;
\item extract a window  ${\bf y}^{[N]}$ of size $N$ from within 
each generated $Y$; and
\item solve the resulting finite-$N$ optimization problem:
\[
 \begin{array}{ll}
 \underset{{\bf w} }{\mbox{minimize}}   & \frac{1}{N}\|{\bf w}*Y\|_1\\
\mbox{subject to}  & {\bf w}_0 =1.
\end{array}
\tag{${P}^N_1(e_0)$ }
\label{bd:finite}
  \]
\end{itemize}
Tabulating the fraction of instances with numerically precise recovery of the correct underlying inverse filter ${\bf a}^{-1}$ 
and sparse signal $X$ across
grid points, we can make a heatmap of empirical success probability.

Specifically, we numerically run $20$ independent experiments, and choose an accuracy $\epsilon = 10^{-3}$ and count the number of successfully accurate recovery where the convex optimization solution $w$ satisfy $\|w^\star -w\|_1/\|w^\star\|_\infty<\epsilon$.

We do this in Figure~\ref{fig-PT-finite}; the reader will see there an empirical phase transition curve, produced by a logistic-regression calculation of the location in $p$ where $50\%$ success probability is achieved. 
We observe empirical behavior entirely consistent 
with $p^\star  = 1-|s|$.
















\paragraph{Phase transition for sparse blind deconvolution with general filter: upper bound from deltaness discrepancy}
 {Consider now a more general situation where ${\bf a}$ is a quite general filter, and $X$ is Bernoulli Gaussian. 
Our formal assumptions are:}

{\begin{enumerate}
 \item  [{\bf A1}]
${\bf a}\in l_1({\mathbb Z})$ is invertible:  $(\mathcal{F} {\bf a})(\widetilde{w}) \neq 0$, $\forall \widetilde{w}\in T$; thus ${\bf a}^{-1}$ exists in $\ell_1({\mathbb Z})$.
\item [{\bf A2}] $X = (X_t)_{t \in {\mathbb Z}}$ is IID with marginal distribution  $pN(0,1) + (1-p) \delta_0$; and
\item [{\bf A3}] $Y = (Y_t)_{t \in {\mathbb Z}}$ is a linear process
obeying $Y = {\bf a} \star X$.
\end{enumerate}}

{Consider the convex optimization problem 
 \[
 \begin{array}{ll}
 \underset{{\bf w} \in l_1({\mathbb Z})}{\mbox{minimize}}   & {\mathbb E}|({\bf w}*Y)_0|\\
\mbox{subject to}  
& \langle \widetilde{{\bf a}},  {\bf w}^{\dagger} \rangle =1,
\end{array}
\tag{${P}_1(\widetilde{{\bf a}})$ }
  \]
and let ${\bf w}^\star$ denote any solution of the optimization problem. }

\newcommand{\widetilde{{\bf a}}}{\widetilde{{\bf a}}}
\newcommand{\mbox{Gap}}{\mbox{Gap}}
\newcommand{l_1({\mathbb Z})}{l_1({\mathbb Z})}
\newcommand{l_\infty({\mathbb Z})}{l_\infty({\mathbb Z})}
 {Our results establish the existence of a general
phase transition phenomenon and a precise quantification of it,
in terms of a {\it phase transition functional} .}

 {Define the following {\bf deltaness discrepancy} $\Delta({\bf v})$:
\[
 \Delta({\bf v}) =  \inf_{\alpha,k} \frac{\| {\bf v} - \alpha \delta_k \|_\infty}{\|{\bf v}\|_\infty}.
\]
}

 {This discrepancy  indeed obeys
$\Delta({\bf e}_0) = 0$. However, it is not restricted only to
deltaness concentrating at the origin; in fact,
$\Delta( \alpha \cdot {\bf e}_k) = 0$ for
each $\alpha > 0$, $k \in {\mathbb Z}$.
Also, $0 \leq  \Delta({\bf v}) \leq 1$.
At the other extreme,
$\Delta([...,0,0,1/2,1/2,0,0,...]) = 1$. 
Finally,
$\Delta$ is a continuous function on $l_1({\mathbb Z})$:
so that, 
setting ${\varepsilon}({\bf v},{\bf w}) \equiv \frac{\|v-w\|_\infty}{\|w\|_\infty}$,
\[
   | \Delta({\bf v}) - \Delta({\bf w}) | \leq \frac{2 \cdot {\varepsilon}}{1-{\varepsilon}}.
\]
}
Indeed, we can give a more explicit form
for $\Delta$;  let $\widetilde{{\bf e}}:= \widetilde{{\bf a}}*{\bf a}^{-1}$,
\[
 \Delta(\widetilde{{\bf e}}) = \frac{|\widetilde{{\bf e}}|_{(2)}}{|\widetilde{{\bf e}}|_{(1)}},
\]
where $|\widetilde{{\bf e}}|_{(1)}$ denotes the largest entry
in $\widetilde{{\bf e}}$ and $|\widetilde{{\bf e}}|_{(2)}$ the second largest.
$\frac{|\widetilde{{\bf e}}|_{(2)}}{|\widetilde{{\bf e}}|_{(1)}}$ 
is a kind of multiplicative gap functional,
measuring the extent to which the second-largest entry in $\widetilde{{\bf e}}$
is small compared to the largest entry. It is a natural measure of closeness 
between our approximate Kronecker delta $\widetilde{{\bf e}}=\widetilde{{\bf a}} * {\bf a}^{-1}$ 
and true Kronecker delta  ${\bf e}_0$. 

	\begin{theorem}
	[ { Population (large-$N$) phase transition}: upper bound from {\bf deltaness discrepancy}]
	\label{theorem:AltPopulationPhaseTransition}

 {Under assumptions (A1)-(A3),}
\begin{itemize}
\item  {There is a functional
$\Pi^*$ defined on tuples $({\bf a},\widetilde{{\bf a}})$ in 
$\ell_1({\mathbb Z}) \times \ell_1({\mathbb Z})$
with the property that, for $p > \Pi^*({\bf a},\widetilde{{\bf a}})$,
every solution of the optimization problem ${P}_1(\widetilde{{\bf a}})$
is exactly the correct answer ${\bf a}$, up to lateral shift and scaling.}

\item  {We have the upper bound:
\begin{equation}
    \label{eq:coro-upperbnd}
\Pi^*({\bf a},\widetilde{{\bf a}}) \leq 1-\Delta(\widetilde{{\bf a}}*{\bf a}^{-1}), \qquad ({\bf a},\widetilde{{\bf a}}) \in \ell_1({\mathbb Z}) \times \ell_1({\mathbb Z}).
\end{equation}
}
In words, the theorem is stating that {\it the less accurate the initial approximation $\widetilde{{\bf a}} \approx {\bf a}$, the greater we rely on sparsity of $X$ to allow exact recovery.}


\end{itemize}
\end{theorem}


 {The reader might compare this with our earlier example
in Theorem \ref{thm: single root PT}, the special case of
geometrically decaying filters. 
In that example ${\bf a} = (\dots,0,0,1,s,s^2,\dots)$ while
$\widetilde{{\bf a}} = (\dots,0,0,1,0,0,\dots)$. $\Delta(\widetilde{{\bf a}}*{\bf a}^{-1})= \Delta((\dots,0,0,1,s,s^2,\dots)) = |s|$. In short, the upper bound $\Pi^* = 1- |s|$ deriving from this general viewpoint agrees precisely with the the exact answer $p^* = 1- |s|$ given in that earlier Theorem. }


When does equality hold in (\ref{eq:coro-upperbnd})?
For a vector ${\bf v}$, let ${\bf v}'$ denote the same vector,
except the largest-amplitude  entry is replaced by $0$. We will see
that a sufficient condition for equality is:
\[
   \Delta(\widetilde{{\bf e}}') \leq \Delta(\widetilde{{\bf e}}).
\]
Put another way, equality holds if 
\begin{equation} \label{eq:suffcondsharp}
|\widetilde{{\bf e}}|_{(3)} \leq  \Delta(\widetilde{{\bf e}}) \cdot |\widetilde{{\bf e}}|_{(2)}.
\end{equation}
The set of situations where this occurs is ample but not overwhelming.
It has relative
Lebesgue measure at least  $\Delta$. In the situation covered by  Theorem \ref{thm: single root PT},
$|\widetilde{{\bf e}}|_{(3)}  = s^2$, $ |\widetilde{{\bf e}}|_{(2)} = s$,
$\Delta = s$, and so equality holds in (\ref{eq:suffcondsharp}).
Hence, the general-filter result of Theorem 
\ref{theorem:AltPopulationPhaseTransition} along with the sufficient condition (\ref{eq:suffcondsharp})
imply as a special case
Theorem \ref{thm: single root PT}.


 {Now we state the following corollary to show that there is a substantial
region in the space of $({\bf a},\widetilde{{\bf a}})$ pairs where a meaningful 
sparsity-accuracy of initialization tradeoff exists, such that
 any sufficiently accurate initial guess results in
exact recovery \-- provided that the sparsity of the underlying object $X$ exceeds a threshold.}

\begin{coro}
 {Let $r \in (0,1/2)$ and suppose 
that assumptions (A1)-(A3)
hold with $p > \frac{r}{1-r}$. 
Normalize the problem so that 
$\| {\bf a}^{-1} \|_\infty =1$.}

 {Consider $\widetilde{{\bf a}}$ in the $\ell_1$ metric ball
(\ref{eq:lonemetricball}) of radius $r$
about ${\bf a}$:
\begin{equation} \label{eq:lonemetricball}
    \| \widetilde{{\bf a}} - {\bf a} \|_1 \leq r .
\end{equation}
}
 {Every solution of the optimization problem $P_1(\widetilde{{\bf a}})$ 
achieves exact recovery of ${\bf a}$ up to shift and rescaling.}
\end{coro}

 {\textbf{Proof}:
Since 
\[
  \| \widetilde{{\bf e}} - {\bf e}_0\|_\infty \leq \| {\bf a}^{-1} \|_\infty \cdot \| \widetilde{{\bf a}}-{\bf a}\|_1 = r,
\]
we have $|\widetilde{{\bf e}}|_{(1)} \geq 1-r$ and $|\widetilde{{\bf e}}|_{(2)} \leq r.$ This 
implies that $\Pi^* \leq \frac{r}{1-r}$. Now apply theorem
\ref{theorem:AltPopulationPhaseTransition}.
\QED
}





\paragraph{Phase transition for sparse blind deconvolution with general filter from optimization point of view}
Theorem 
\ref{theorem:AltPopulationPhaseTransition} has provided a upper bound of the threshold $p^\star$ for phase transition with clear mathematical meaning.   Now we present the main phase transition theorem with the exact $p^\star$ for blind deconvolution of general inverse filter. This statement is from optimization point of view that connects blind deconvolution problem to a classical projection pursuit problem. 


Let $I$ denote an iid Bernoulli(p) bisequence.
For a bisequence ${\bf w}$ let $ {\bf w} \cdot I$ denote the 
elementwise multiplication
of ${\bf w}$ by $I$.
Define the optimization problem
\[
 \begin{array}{ll}
 \underset{{\bf w}}{\mbox{minimize}}   & {\mathbb E}_I \|{\bf w} \cdot I\|_2\\
\mbox{subject to}  
& \langle   {\bf v}, {\bf w}\rangle =1
\end{array}
\tag{$Q_1({\bf v})$ }
  \]


	\begin{theorem}[Population (large-$N$) phase transition: upper and lower bound]
	\label{thm:Population phase transition}


	Under assumptions (A1)-(A3),consider the solution ${\bf w}^\star$ of the convex optimization problem ${P}_1(\widetilde{{\bf a}})$. There is a threshold $p^\star > 0$,
\begin{itemize}
\item  ${\bf w}^\star$ is ${\bf a}^{-1}$ up to time shift and rescaling provided $p<p^\star$; and
\item   ${\bf w}^\star$ is not ${\bf a}^{-1}$ up to time shift and rescaling, provided $p> p^\star$.
\end{itemize}

The threshold $p^\star$ obeys
 \[
\frac{p}{1-p}= val(Q_1(\widetilde{{\bf e}}')).
\] 
where $\widetilde{{\bf e}}:= \widetilde{{\bf a}}*{\bf a}^{-1}$.
As defined in theorem \ref{theorem:AltPopulationPhaseTransition}, $p^\star = \Pi^*({\bf a},\widetilde{{\bf a}}) $.

We have an upper bound and lower bound of $val(Q_1(\widetilde{{\bf e}}'))$,
\[
 \frac{p}{\Delta(\widetilde{{\bf e}})}  \geq val(Q_1(\widetilde{{\bf e}}')) \geq  p \cot\angle(\widetilde{{\bf e}}, {\bf e}_0) 
\]
explicitly, the upper bound can be expressed as
\[
1- \tan\angle(\widetilde{{\bf e}}, {\bf e}_0)   \leq p^\star \leq 1- \frac{|\widetilde{{\bf e}}|_{(2)}}{|\widetilde{{\bf e}}|_{(1)}} = 1-\Delta(\widetilde{{\bf e}})
\]
Namely,
\begin{equation}
    \label{eq:coro-upperbnd-lowerbnds}
1- \tan\angle(\widetilde{{\bf a}}*{\bf a}^{-1}, {\bf e}_0) \leq \Pi^*({\bf a},\widetilde{{\bf a}}) \leq 1-\Delta(\widetilde{{\bf a}}*{\bf a}^{-1}), \qquad ({\bf a},\widetilde{{\bf a}}) \in \ell_1({\mathbb Z}) \times \ell_1({\mathbb Z}).
\end{equation}
Additionally, the upper bound 
takes equality if
	\[
   \Delta(\widetilde{{\bf e}}') \leq \Delta(\widetilde{{\bf e}}).
\]




	\end{theorem}
	
We comment that theorem 	\ref{thm:Population phase transition} imply previous two theorems. Clearly theorem \ref{thm: single root PT} is a special case for exponential decay forward filter, and theorem \ref{theorem:AltPopulationPhaseTransition} can also be viewed as the natural upper bound side of the functional
$\Pi^*$  in theorem 	\ref{thm:Population phase transition}.







\subsection{Main Result $2$: Finite Observation Window, Finite-length Inverse}

With a finite observation window of length $N$ we (surprisingly) still can have exact recovery, starting as soon as $N \geq \Omega(k\log(k))$.

\paragraph{Finite-observation phase transition}


Let 
$\ell_1^k$
denote the collection of bisequences vanishing outside a centered window of radius $k$.

	\begin{theorem}[Finite observation window, finite-length inverse filter]
	\label{thm:fin_guarantee}
	Suppose $Y = {\bf a} \star X$, where:
	\begin{enumerate}
 \item  [{\bf A1}]
${\bf a}\in l_1({\mathbb Z})$ is invertible and the inverse has finite-length:  $(\mathcal{F} {\bf a})(\widetilde{w}) \neq 0$, $\forall \widetilde{w}\in T$; thus ${\bf a}^{-1}$ exists in $\ell_1({\mathbb Z})$; furthermore, ${\bf a}^{-1} \in \ell_1^k$ vanishes off a centered window of length $k$.
\item [{\bf A2}] $X = (X_t)_{t \in {\mathbb Z}}$ is IID with marginal distribution  $pN(0,1) + (1-p) \delta_0$; 
\item [{\bf A3}] $Y = (Y_t)_{t \in {\mathbb Z}}$ is a linear process
obeying $Y = {\bf a} \star X$. Suppose we observe a window $(Y_t)_{t\in {\cal T}}$ of length $N = |{\cal T}|$. 
\end{enumerate}


Consider the convex optimization problem
\begin{equation}
 \begin{array}{ll}
 \underset{{\bf w}\in \ell_1^k}{  \mbox{minimize} }   & \frac{1}{N}\|{\bf w}*Y\|_{\ell_1^N}\\
 \mbox{subject to}  
& (\widetilde{{\bf a}}* {\bf w})_0 =1.
\end{array}
\tag{${P}_{1}^{N,k}(\widetilde{{\bf a}})$}
\label{P1_finite}
\end{equation}
and let ${\bf w}^\star$  denote any solution of the optimization problem.

	Our result establishes that there exist $\epsilon>0, \delta>0$, so that when the number of observations $N$ satisfies
		\[
N \geq  k\log(\frac{k}{\delta}) ( \frac{C\kappa_{\bf a}}{\epsilon}  )^2,
\]
and the sparsity level $p$ obeys $\frac{1}{N}\leq p < p^\star - \delta_p(N,\epsilon)$,
		then 	with probability exceeding $1-\delta$, ${\bf w}^\star$   is ${\bf a}^{-1}$ up to rescaling and time shift. In the statement, $\kappa_{\bf a}$ is the condition number of the circular matrix with its first column being ${\bf a}$; $C$ is a positive constant independent of $N$ and $k$.
	\end{theorem}
	
 


\subsection{Main Result $3$: Stability Guarantee with Finite Length Inverse Filter}
The results so far concern the ideal setting when true inverse filter has a known length $k$ and we use $k$ to set up a correctly matched optimization problem. In practice we do not know $k$ and $k$ might even be infinite.

We can provide practical guarantees even when the inverse filter is an infinite length inverse. To develop these, we must be more technical about the situation. We assume that ${\bf a}$ has a Z-transform having $N_{-}$ roots and $N_{+}$ poles $(s_{i})$ inside the unit circle and we construct a finite length approximation ${\bf w}$ to  ${\bf a}^{-1}$, in fact of length
$r(N_{-}+N_{+})$.
This approximation has error 
$\|{\bf w}*{\bf a}-{\bf e}_0\|_2= O(\max_i |s|_{i}^{r}  )$. 

Since the objective value ${\mathbb E}_I\|{\bf w}\cdot I\|_2$ of this approximation ${\bf w}$ is an upper bound of the optimal value of the optimization solution ${\bf w}^\star$, we could use the objective value upper bound  to derive a upper bound for $\|{\bf w}^\star*{\bf a}-{\bf e}_0\|_2= O(
\max_i |s|_{i}^{r}  )$ when $p<p^\star$, where the constant of this upper bound is determined by the Bi-Lipschitz constant of the finite difference of objective ${\mathbb E}_I\|{\bf w}\cdot I\|_2-{\mathbb E}_I\|({\bf e}_0)\cdot I\|_2$.

\paragraph{Approximation theory for infinite length inverse filter}
\begin{theorem}[Approximation theory for infinite length inverse filter based on roots of Z-transform]
\label{thm:RootNorm}
Let the finite-length forward filter ${\bf a}$ have a Z-transform with roots inside the unit circle, namely $s_{k}:= e^{-\rho_k + i \varphi_k }$ with $|s_k|<1$ and $\rho_k>0$
for $k \in \{-N_{-}, \ldots -1, 1, \ldots N_{+}\}$.  Let ${\cal I} = \{-N_{-},\ldots, -1, 1, \ldots, N_{+} \}$ as the set of all the possible indexes.
\[
 \begin{array}{ll}
A(z) &=  \sum_{i=-N_{-}}^{N_{+}} a_{i} z^{-i}
\\
&= c_0\prod_{j=1}^{N_{-}} (1- s_{-j} z) \prod_{i=1}^{N_{+}} (1- s_{i} z^{-1});
\end{array}
  \]
here $c_0$ is a constant ensuring $a_0=1$.

Then for a scalar $r$
, we construct an approximate inverse filter ${\bf w}^r$ with Z-transform
\[
 \begin{array}{ll}
W(z) 
&=  \frac{1}{c_0}\prod_{j=1}^{N_{-}} (\sum_{\ell_j=0}^{r-1} s^{\ell_j}_{-j} z^{-\ell_j} ) \prod_{i=1}^{N_{+}} (\sum_{\ell_i=0}^{r-1} s^{\ell_i}_{i} z^{\ell_i} )\\
&=  \frac{1}{c_0}\prod_{j=1}^{N_{-}} (1- (s_{-j} z)^{r})(1- s_{-j} z)^{-1} \prod_{i=1}^{N_{+}}(1- (s_{i}  z^{-1})^{r}) (1- s_{i} z^{-1})^{-1}.
\end{array}
\]
We have
\[
 \begin{array}{ll}
 \|{\bf w}^r*{\bf a}-{\bf e}_0\|_2^2  
 &= \sum_{n\in \{1,2,3,\ldots,|{\cal I}| \} }\sum_{ k_1, \ldots, k_n  \in {\cal I} } \prod_{i\in [n]}|s_{k_i}|^{2r }
\end{array}
  \]
as $r \rightarrow \infty$, this converges to zero at an exponential rate, determined by the slowest decaying term,
\[
 \begin{array}{ll}
\|{\bf w}^r*{\bf a}-{\bf e}_0\|_2
&=  O( |s|_{(1)}^{r}  ) , \quad r \rightarrow \infty.
\end{array}
  \]
  where $|s|_{(1)}$ is the largest absolute value root.
 \end{theorem}
 
 
 \paragraph{Stability for infinite length inverse filter}
 \begin{theorem}[Stability for infinite length inverse filter]
 \label{thm:bd_stab}
 Let ${\bf a}\in V_{N_-, N_+} $ be a forward filter with all the roots of Z-transform strictly in the unit circle. Let ${\bf w}^\star \in V_{(r-1)N_-, (r-1)N_+} $ be the solution of the convex optimization problem. Let ${\bf w}^r$ be the constructed filter in previous theorem with a uniform vector index $(r, \ldots, r, r,\ldots, r)$.
  Then provided $p<p^\star$, as 
  $r \rightarrow \infty$, 
  \[
 \begin{array}{ll}
  \| {\bf w}^\star*{\bf a} -{\bf e}_0\|_2 
   \leq
    O(
 |s|_{(1)}^{r}  ), \quad r \rightarrow \infty.
\end{array}
  \]
  where $|s|_{(1)}$ is the largest absolute value root.
In words, \textbf{the Euclidean distance between ${\bf w}^\star*{\bf a}$ and ${\bf e}_0$ converges to zero at an exponential rate as the approximation length is allowed to increase.} 
 \end{theorem} 
 
 
 
\subsection{Main Result $4$: Robustness Against Stochastic Noise and Adversarial Noise}
We now extend the previous analysis of an exactly sparse model of $X$, exactly observed, to the more practical setting of approximate sparsity and observation noise. We consider two cases: first, we add stochastic noise as an independent Gaussian linear process, and second, we add adversarial noise with bounded $\ell_\infty$ norm.  

In each scenario, since the noisy objective value at ${\bf e}_0$ is an upper bound on the optimal value of the optimization solution ${\bf w}^\star$, we use this upper bound on the objective value  to derive an upper bound of $\|{\bf w}^\star*{\bf a}-{\bf e}_0\|_2$ when $p<p^\star$.
The upper bound shows that the distance $\|{\bf w}^\star*{\bf a}-{\bf e}_0\|_2$ is bounded by the (appropriately measured)
magnitude of input noise in both cases.


\paragraph{Robustness under stochastic noise }
 \begin{theorem}[Robustness against Gaussian Linear Process noise]
 \label{thm:rand_exact_rob}
 We consider a Gaussian linear process $Z = \sigma \cdot {\bf b} \star G$ where: $\sigma > 0$ denotes the noise level;   ${\bf b}$ is
 a bisequence having  unit $\ell_2$ norm $\|{\bf b}\|_2=1$; and $G$ is a standard Normal iid bisequence. Suppose that we observe:
 \[
\begin{array}{ll}
Y&= {\bf a}* (X+ Z).
\end{array}
\]
 
 \newcommand{{\bf w}}{{\bf w}}
 Let ${\bf w}^\star$ be the solution of the convex optimization problem in Eq.(4.1);


\label{thm:Bound_Normal_Noise}
\[
 {\mathbb E}_I  \| ({\bf w}^\star)\cdot I \|_2 -{\mathbb E}_I  \| ({\bf e}_0)\cdot I \|_2 \leq  (1-p) \sigma +
 p(\sqrt{1+ \sigma^2 }-1).
\]
When $p<p^\star$, $\sigma\leq 1$, there exists a constant $C$,
\[
  \| {\bf w}^\star*{\bf a} -{\bf e}_0\|_2 = \|{\bf w}^\star -{\bf e}_0\|_2 \leq C \sigma.
\]
In words, \textbf{the Euclidean distance between ${\bf w}^\star \star {\bf a}$ and ${\bf e}_0$ is bounded linearly by the magnitude of stochastic noise}.
\end{theorem}  

\newcommand{{\bf c}}{{\bf c}}
\paragraph{Robustness under adversarial noise }
 \begin{theorem}[Robustness under adversarial noise with $\ell_p$ norm bound]
\label{thm:adv_exact_rob}
Suppose that an adversary chooses a disturbance bisequence
${\bf c}$ subject to the constraint:	
\[
	\| {\bf c}\|_\infty \leq \eta;
\]
and perturbs the observation process $Y$ via:
	\[
	\begin{array}{ll}
	Y&= {\bf a}* (X+  {\bf c}) .
	\end{array}
	\]
\newcommand{{\bf v}}{{\bf v}}
Let ${\bf w}^\star$ denote the solution of the convex optimization problem~\ref{eqn:P1}.
Define ${\bf v}^\star = ({\bf a}*{\bf w}^\star)^{\dagger}$, then ${\bf v}^\star$ satisfies the following bound on objective difference:
 \[
 \begin{array}{ll}
 {\mathbb E}_I  \| ({\bf v}^\star)\cdot I \|_2 -{\mathbb E}_I  \| ({\bf e}_0)\cdot I \|_2 \leq 
 p\sqrt{\frac{2}{\pi}}(\mathcal{R}(\eta) - 1)+(1-p)\eta.
\end{array}
  \]  
 
Therefore, when $p<p^\star$, there exists a constant $C'$, so that
 \[
 \| {\bf w}^\star*{\bf a} -{\bf e}_0\|_2 = \|{\bf v}^\star -{\bf e}_0\|_2  \leq C'  \eta, \forall \eta > 0.
 \]
 In words, \textbf{the Euclidean distance between ${\bf w}^\star \star {\bf a}$ and ${\bf e}_0$ is at most proportional to the magnitude of the adversarial noise}.
 
 
\end{theorem}  

\textbf{Remark}: Here $\mathcal{R}(\eta)$
is the folded Gaussian mean, for standard Gaussian $G$:
\[
\mathcal{R}(\eta) :=  \sqrt{\frac{\pi}{2}} E_G |\eta+  G| = \exp\{-\eta^2/2\} + \sqrt{\frac{\pi}{2}}\eta \left(1 - 2\Phi\left(-\eta\right) \right).
\]  
$\mathcal{R}(\eta)-1$ is an even function that is monotonically non-decreasing for $\eta \geq 0$ with quadratic upper and lower bound: there exists constants $C_1\leq C_2$,
  \[
C_1 \eta^2 \leq \mathcal{R}(\eta)-1 \leq C_2 \eta^2, \forall \eta.
\]

\newcommand{{\bf u}}{{\bf u}}
	






  




  
  
\subsection{Non-convex Blind deconvolution initialization method}
Now we study how to get the initial guess $\widetilde{{\bf a}}$.

\paragraph{}
First, let $C_Y$ denote the circular embedding of $Y$;  we can define $\Bar{Y} = (C_Y C_Y^T)^{-1/2}Y$, then from $Y={\bf a}*X$, we get $C_Y = C_a C_X$, then
\[
C_{\Bar{Y}} =  (C_Y C_Y^T)^{-1/2}C_Y = (C_a C_X C_X^T  C_a^T)^{-1/2}C_a C_X
\]
as $X$ are IID Bernoulli-Gaussian, 
we know approximately 
\[
C_{\Bar{Y}} =  (C_Y C_Y^T)^{-1/2}C_Y = (C_a C_X C_X^T  C_a^T)^{-1/2}C_a C_X \approx (C_a C_a^T)^{-1/2}C_a C_X
\]
We can also define $C_{\Bar{a}}=(C_a C_a^T)^{-1/2}C_a$.

Let $S_{\kappa_1 \times \kappa_2, K}$ be Riemannian manifold of $K$-channel $\kappa_1 \times \kappa_2$ convolution dictionary with row norm $1$. 

Our non-convex algorithm objective is
\[
 \begin{array}{ll}
 \underset{A, X}{\mbox{minimize}}   & \|Y-(A*X+b) \|_2^2+ \lambda \|X\|_1\\
\mbox{subject to}  
& A \in S_{\kappa_1 \times \kappa_2, K},
\end{array}
\label{eqn:NC}
  \]
  And the algorithm is alternating between proximal gradient method on solving $X$ given $A$ with Backtracking line search for updating step size of proximal gradient on $X$, and  projected gradient method on solving $A$ given $X$ with line search for updating the step size of Riemannian gradient on $A$.
\begin{algorithm}[ht]
\LinesNumbered
\SetAlgoLined
Input: $Y$ of size $n_1 \times n_2$  estimated upper bound of kernel size of forward filter $a$: $\kappa_1 \times \kappa_2$, estimated number of kernel channels $K$.\;

Create zero tensor
$A^{\mbox{init}}$ be of size $3\kappa_1 \times 3\kappa_2  \times K$, \;

for $k \in 1, \ldots, K:$\;

random uniformly choose index $i_{1,k},i_{2,k}$; assign
\[
A^{\mbox{init}}[\kappa_1: 2*\kappa_1, \kappa_2: 2*\kappa_2, k] = 
Y[i_{1,k}:i_{1,k}+\kappa_1,i_{2,k}: i_{2,k}+\kappa_2]/\| Y[i_{1,k}:i_{1,k}+\kappa_1,i_{2,k}: i_{2,k}+\kappa_2]\|_2
\]
  \;
  
Let $X^{\mbox{init}}$ be zeros of shape $n_1 \times n_2 \times K$; 
$b^{\mbox{init}}$ be $\mbox{mean}(\Vec{Y})$.  \;

for $\mbox{iter} \in 1, \ldots,  \mbox{MaxIter}:$\;

Alnternating the two steps:\;

Given $A$ fixed, take a descent step on $X$ via proximal gradient descent;
Backtracking for update $X$ and update stepsize $t$;\;

Given $X$ fixed, take a Riemannian gradient step on $A$, use line search for updating the stepsize of Riemannian gradient on $A$. \;

Given $A, X$ fixed, update the bias $b$ \;

if $\|A -A^{\mbox{prev}\|_2}<\epsilon$ and $\|X -X^{\mbox{prev}\|_2}<\epsilon$, stop.
 \caption{Non-convex Blind deconvolution initialization method}
\end{algorithm}

\begin{algorithm}[ht]
\LinesNumbered
\SetAlgoLined
Input: $Y$ of size $n_1 \times n_2$, estimated upper bound of kernel size of forward filter $a$: $\kappa_1 \times \kappa_2$.\;

\eIf{Initialization of forward filter $\Tilde{a}$ not provided}{
Let $\Tilde{a}$ be the solution of the previous algorithm with estimated number of kernel channels $K=1$ and forward kernel size $\kappa_1 \times \kappa_2$. 
}
{Use initialization guess of forward filter $\Tilde{a}$ \;}
Find inverse filter ${\bf w}$ by
\[
 \begin{array}{ll}
 \underset{{\bf w} \in l_1({\mathbb Z})}{\mbox{minimize}}   & \|{\bf w}*Y\|_1\\
\mbox{subject to}  
& \langle \widetilde{{\bf a}},  {\bf w}^{\dagger} \rangle =1,
\end{array}
\tag{${P}_1(\widetilde{{\bf a}})$ }
\label{eqn:P1}
  \]
  \;
 \caption{Full blind deconvolution algorithm}
\end{algorithm}







\section{Analysis of the inverse kernel method}
\subsection{Inverse kernel approximation}
Now we consider a setting where there is a kernel whose corresponding the inverse kernel is infinite, and we give a finite length approximation. 
We assume that $a =  (a_{-t},\ldots, a_{-1}, 1, a_1, a_2, \ldots, a_t) = \delta_0 + a_L + a_R$ 
is a length $ l= 2t+1$ filter, 
  and $\sum_{i\neq 0} |a_i|\leq M < 1$.    
 \begin{theorem} 
For a kernel $a = (a_{-s},\ldots, a_{-1}, 1, a_1, a_2, \ldots, a_t)$ of length $l=1+s+t$, if 
\[
\sum_{i\neq 0} |a_i|\leq M < 1,
\]
 then there exists an approximate inverse kernel $w^r$ of length $rl$, such that 
  \[
 \begin{array}{ll}
\|\mathcal{F}w-\mathcal{F}w^r \|_{\infty} 
&\leq M^{r+1}/(1-M),\\
\end{array}
  \]
\end{theorem}
\begin{proof}
Consider the $z$ transform of $a$ to be 
\[
 \begin{array}{ll}
A(z) =  \sum_{i=-s}^t a_{i} z^{-i},
\end{array}
  \]
let $\bar A(z) = 1-A(z)$, then 
from the condition 
$\sum_{i\neq 0} |a_i|\leq M < 1$,
we define $M' $ to be the maximum absolute value of $\bar A(z)$ on unit circle, as
\[
 \begin{array}{ll}
\mbox{maximize}   & |\bar A(z)| \\
\mbox{subject to}  & z\in {\mathbb C},|z|=1.
\end{array}
  \]
Then 
\[
 \begin{array}{ll}
M' \leq \sum_{i\neq 0} |a_i|\leq M<1.
\end{array}
  \]
The Z-transform of the inverse kernel of $a$ is defined by
\[
 \begin{array}{ll}
1/A(z) &=  (1-\bar A(z))^{-1},\\
&= 1+ \sum_{i=1}^{\infty} (\bar A(z))^i
\end{array}
  \]
 The infinite series absolutely converge on $|z| = 1$ because $|\bar A(z)|\leq M<1$.
Let the finite approximation of level $r$ of $w = a^{-1}$ be $w^r$, 
 its Z-transform is  $\sum_{i=0}^{r} (\bar A(z))^i$,
its length is at most $r(s+t+1)$, 
the $j$-th term of $w^r$ is $\sum_{b}bj_i=j \prod_{i} a_i^{j_i}$.
We have that for all  $|z| = 1$,
 \[
 \begin{array}{ll}
|1/A(z) - \sum_{i=0}^{r} (\bar A(z))^i | 
&= |\sum_{i=r+1}^{\infty} (\bar A(z))^i|\\
&\leq M^{r+1}/(1-M).
\end{array}
  \]
Therefore, 
   \[
 \begin{array}{ll}
\|\mathcal{F}w-\mathcal{F}w^r \|_{\infty} 
&\leq M^{r+1}/(1-M).
\end{array}
  \]
 
\end{proof}

Using this inverse approximation, we know for any kernel $a$ of length $l$ with possibly infinite inverse kernel $w$, there exists a length-$rl$ approximate inverse kernel $w^r$, such that 
  \[
 \begin{array}{ll}
\|\mathcal{F}w-\mathcal{F}w^r \|_{\infty} 
&\leq M^{r+1}/(1-M).
\end{array}
  \]

Therefore, we can prove the following theorem on sparse blind deconvolution algorithm. 


\paragraph{Simple example.}
We consider a simple example, when $a = (0,\ldots, 0, 1, r, 0, \ldots, 0) = \delta_0 - r\delta_1$, $|r|<1$. Then $a^{-1} = \delta_0 + \sum_{j=1}^\infty  r^j \delta_j.$ Now our data generation model is $y = a*x.$ 

We analysis the optimization problem 
 \begin{eqnarray*}
    \min_{f_0 = 0} &    
    E_x\| (\delta_0+f)*a*x\|_1.
\end{eqnarray*}
Here $f = (0, 0, f_R)$ is in a $K$ dimensional subspace of the double infinite sequence.


For the objective, using IID property of $x$, we only need to analyze
 \begin{eqnarray*} 
    E_x | ((\delta_0+f)*(\delta_0 - r\delta_1)*x)_0| .
\end{eqnarray*}
Let $f^*_K$ be the best length $K$ approximation, as the solution of 
\begin{eqnarray*}
    \min_{f_0 = 0} &    
    \| (\delta_0+f)*a\|_2,
\end{eqnarray*}

In this specific example, $f^*_K = \sum_{j=1}^K  r^j \delta_j.$ And $(\delta_0+f^*_K)*a = (\delta_0 - r^{K+1}\delta_{K+1})*(\delta_0 - r^{1}\delta_{1})^{-1}$.
Let $\tilde{x} = (\delta_0 - r^{K+1}\delta_{K+1})*x$, as one approximation of $x$.
Since $x$ is IID symmetric, $ \tilde{x}$ is identically-distributed, symmetric and weakly correlated for $|r^{K+1}|$ small enough, and $\tilde{x}_0, \ldots, \tilde{x}_K$ are independent.

Let 
\begin{eqnarray*} 
\delta_0+h &=& (\delta_0+f)*(\delta_0+f^*_K)^{-1}\\
&=& (\delta_0+f)*(\delta_0 - r^{K+1}\delta_{K+1})
\end{eqnarray*}
then 
 \begin{eqnarray*} 
    E_x | ((\delta_0+f)*(\delta_0 - r\delta_1)*x)_0| 
    &=& E_x | ((\delta_0+h)*\tilde{x})_0|\\
&=& E_x | \tilde{x}_0|+\mbox{sign}(\tilde{x}_0)(h*\tilde{x})_0\\
&\approx& 
 |1+  h_{0}| \cdot E |\tilde{x}_0| \cdot (1-\eta)+   \cdot  E{| (h*\tilde{x})_0| } \cdot \eta.
\end{eqnarray*}

Following the same analysis as the phase transition analysis, we have
\begin{eqnarray*}
     \min_\delta &E | ((\delta_0+f)*a*\tilde{x})_0 | \\
    \mbox{ subject to }  &f_{0} = 0.
\end{eqnarray*}

\subsection{Distance metrics}
We care about three types of metrics, first, distance  in $w$ space, between the computed composed filter $w$ and ground truth $a^{-1}$; second, distance in $w$ space, between the computed inverse filter $w$ and ground truth inverse filter $a^{-1}$, second, distance  in $\psi = w*a$ space, between the computed composed filter $w*a$ and ground truth $e_0$; third,  distance  in $x$ space, between recovered signal sequence and true sparse signal sequence. 

First, the MSE in space of $\psi = w*a$,
  \[
 \begin{array}{ll}
d_{\psi,2}^2 := \|\phi\|_2^2 =  \|\psi-e_0\|_2^2= \|a*(w-a^{-1})\|_2^2
\end{array}
  \] 
 The MSE in space of $w$. 
   \[
 \begin{array}{ll}
d_{w,2}^2  = \|w-a^{-1}\|_2^2 = \|a^{-1}*(\psi-e_0))\|_2^2 = \|a^{-1}*\phi\|_2^2
\end{array}
  \]
Last, $\ell_1$ and $\ell_2$ distance in $x$ space:
\[
 \begin{array}{ll}
d_{x,1} &:= \frac{1}{2T+1}\|w*Y-X\|_{1,[-T,T]} =  \frac{1}{2T+1}\|\tilde \phi* X\|_{1,[-T,T]} \\
d_{x,2}^2 &:= \frac{1}{2T+1}\|w*Y-X\|_{2,[-T,T]}^2 = \frac{1}{2T+1}\|\tilde \phi* X\|_{2,[-T,T]}^2
\end{array}
  \]
We compute their expectation: 
\[
 \begin{array}{ll}
{\mathbb E} d_{x,1}  &=  {\mathbb E}|\phi^T X| = \sqrt{\frac{2}{\pi}} \cdot {\mathbb E}_I \|\phi \cdot I\|_2\\
{\mathbb E} d_{x,2}^2 &=  {\mathbb E}|\phi^T X|^2 = p \|\phi\|^2  = p \|\psi-e_0\|^2 
\end{array}
  \]  
We remark that in the following we focus on distance in $x$ space to recover the signal. 



\section{Stability for noisy observation}


\subsection{Constrained Convex algorithms }
 \paragraph{Constrained noisy convex problems}
Then we define the constrained convex program that allows noise,
\[
 \begin{array}{ll}
 \mbox{minimize}_{w,z}   & |\alpha| \\
\mbox{subject to}  
& v^T w =1,\\
& {\mathbb E}(\alpha -\tilde{w}^T Y)^2 \leq \epsilon^2
\end{array}
\tag{$P^c_{1,\epsilon}$}
\label{eq:ConstrainProb}
  \]
  If we write $\alpha =\tilde{w}^T Y - \epsilon \zeta $ , then
\[
 \begin{array}{ll}
 \mbox{minimize}_{w,\zeta}   & {\mathbb E}|\tilde{w}^T Y - \epsilon \zeta| \\
\mbox{subject to}  
& v^T w =1,\\
& \zeta^2 \leq 1
\end{array}
\tag{$P^c_{1,\epsilon}$}
  \]
If we set $\epsilon = 0,$ then we recover 
the original convex problems for noiseless setting
\[
 \begin{array}{ll}
 \mbox{minimize}_{w}   & {\mathbb E}|\tilde{w}^T Y|\\
\mbox{subject to}  
& v^T w =1,
\end{array}
\tag{$P_1$}
  \]
\subsection{Stability for adversarial perturbation of sparse signal }
\paragraph{Mathematical formulation: sparse signal plus adversarial perturbation}

\begin{itemize}
\item
Let $(X_t)_{t \in {\mathbb Z}}$ be IID sampled from  $pN(0,1) + (1-p) \delta_0$, write $X_t = I_t \cdot G_t$ where $I_t$ is IID sampled from Bernoulli, $G_t$ is IID sampled from standard Gaussian.
 \item
The filter $a\in l_1({\mathbb Z}), (\mathcal{F} a)(\omega) \neq 0, \forall \omega\in T.$
\item
$Z$ is a unit vector in $2$ sphere. 
\item
\[
\begin{array}{ll}
Y&= a* (X+\eta Z ) 
\end{array}
\]
\end{itemize}
Let $\tilde{\psi} := w*a$,   $\phi: = \psi - e_0$, then
\[
 \begin{array}{ll}
{\mathbb E}|(w*Y)_0|
&= {\mathbb E}|\psi^T (X+\eta Z )| \\
\end{array}
  \]

  
  
  
 \paragraph{Study the noisy problem}
Under the data generative model, 
\[
\begin{array}{ll}
Y&= a* (X+\eta Z ) 
\end{array}
\]
    the constrained problem becomes
  \[
 \begin{array}{ll}
 \mbox{minimize}_{\psi,\zeta}   & {\mathbb E}|\psi^T (X+\eta Z ) - \epsilon \zeta| \\
\mbox{subject to}  
&u^T \psi =1,\\
& \zeta^2 \leq 1
\end{array}
\tag{$P^c_{1,\epsilon}$}
  \]
  
\paragraph{Worst case objective over $Z$}  
For the constrained problem, we optimize over the worst case objective for all $Z$ that $\|Z\|_2\leq 1$,
    \[
 \begin{array}{ll}
 \mbox{minimize}_{\psi,\zeta} \quad \sup_{Z: \|Z\|_2\leq 1}  &  {\mathbb E}|\psi^T X+\eta  \psi^T Z  - \epsilon \zeta| \\
\mbox{subject to}  
&u^T \psi =1,\\
& \zeta^2 \leq 1
\end{array}
  \]
Which is equivalent to 
  \[
 \begin{array}{ll}
 \mbox{minimize}_{\psi} \quad \inf_{\zeta: \zeta^2 \leq 1} \quad \sup_{Z: \|Z\|_2\leq 1}  &  {\mathbb E}|\psi^T X+\eta  \psi^T Z  - \epsilon \zeta| \\
\mbox{subject to}  
&u^T \psi =1
\end{array}
  \]
In the following we study the min-max objective
  \[
 \begin{array}{ll}
L_{\eta, \epsilon}(\psi) : = \inf_{\zeta: \zeta^2 \leq 1} \quad \sup_{Z: \|Z\|_2\leq 1}  &  {\mathbb E}|\psi^T X+\eta  \psi^T Z  - \epsilon \zeta| 
\end{array}
  \]
  
\paragraph{Robust optimization}
Let the deterministic residual function of $\psi, Z, \zeta$ be
\[
\mu_{\eta, \epsilon}(\psi, Z, \zeta): = \eta  \psi^T Z  - \epsilon \zeta
\]
We reparametrize $\epsilon$ as by defining 
\[
\tau :=  \frac{\epsilon}{\eta}
\]
then 
\[
\mu_{\eta, \epsilon} = \eta ( \psi^T Z  - \tau \zeta)
\]






\begin{lemma}
For any given sign $s =\pm 1$,
\[
\sup_{Z: \|Z\|_p\leq 1}(s \psi^T Z) = \|\psi\|_q,
\]
specifically, $Z$ is along the direction of $s\psi$ so that $s\psi^T Z = \|\psi\|_2$.

\end{lemma}  


\begin{lemma}[Hölder's inequality]
For any given sign $s =\pm 1$, for $p\geq 1, \frac{1}{p}+\frac{1}{q} =1,$
\[
\sup_{Z: \|Z\|_p\leq 1}(s \psi^T Z) = \|\psi\|_q,
\]
specifically, when $|Z|^p$ is along the direction of $s|\psi|^q$ so that $s\psi^T Z = \|\psi\|_q$.

\end{lemma}  


\begin{lemma}
\label{min-maxError}
The min-max quantity  
\[
 \begin{array}{ll}
\inf_{\zeta: \zeta^2 \leq 1} \sup_{Z: \|Z\|_2\leq 1} (\eta  \psi^T Z  - \epsilon \zeta)^2 = (\eta \|\psi\|_2  - \epsilon )_+^2 = \eta^2 ( \|\psi\|_2  - \tau )_+^2
\end{array}
\]
where $(\cdot)_+: = \max{(0, \cdot)}$ is the ReLU function.
\end{lemma}
\begin{proof}
\[
 \begin{array}{ll}
\inf_{\zeta: \zeta^2 \leq 1} \sup_{Z: \|Z\|_2\leq 1} \mu_{\eta, \epsilon}^2 
&=\inf_{\zeta: \zeta^2 \leq 1}\sup_{Z: \|Z\|_2\leq 1} (\eta  \psi^T Z  - \epsilon \zeta)^2\\
&=\inf_{\zeta: \zeta^2 \leq 1}\sup_{Z: \|Z\|_2\leq 1} [(\eta  \psi^T Z)^2+  (\epsilon \zeta)^2 - 2(\eta  \psi^T Z)(\epsilon \zeta)]\\
&=\inf_{\zeta: \zeta^2 \leq 1}\{   \epsilon^2( \zeta)^2+ \eta^2 \sup_{Z: \|Z\|_2\leq 1} [ (\psi^T Z)^2] + 2\eta \epsilon |\zeta|   \sup_{Z: \|Z\|_2\leq 1}[ (-{\rm sign}(\zeta) \psi^T Z)]\}\\
&=\inf_{\zeta: \zeta^2 \leq 1}\{   \epsilon^2( \zeta)^2+ \eta^2 \|\psi\|_2^2 + 2\eta \epsilon |\zeta|   \|\psi\|_2\}\\
&=\inf_{\zeta: \zeta^2 \leq 1}\{  (\eta \|\psi\|_2  - \epsilon |\zeta| )^2\}\\
&= (\eta \|\psi\|_2  - \epsilon )_+^2
\end{array}
\]
The fourth equation is because for any given sign $s =\pm 1$,
\[
\sup_{Z: \|Z\|_2\leq 1}(s \psi^T Z) = \|\psi\|_2.
\]
\[
\sup_{Z: \|Z\|_2\leq 1}( \psi^T Z)^2 = \|\psi\|_2^2.
\]



\end{proof}

We have the following result, generalizing $\|Z\|_2\leq 1$ to $\|Z\|_p\leq 1, \quad, p\geq 1$, and then to $Z\in K$ for a closed compact convex set $K\in {\mathbb R}^N$.



\begin{lemma}
\label{min-maxErrorL_p}
The min-max quantity 
for $p\geq 1, \frac{1}{p}+\frac{1}{q} =1,$
for a one-dimensional convex even function $F$ that is monotonically non-decreasing for positive coordinates,
\[
 \begin{array}{ll}
\inf_{\zeta: \zeta^2 \leq 1} \sup_{Z: \|Z\|_p\leq 1} F( \eta (\psi^T Z  - \tau \zeta)) = 
F(\eta( \|\psi\|_q  - \tau )_+)
\end{array}
\]
where $(\cdot)_+: = \max{(0, \cdot)}$ is the ReLU function.
\end{lemma}

\begin{coro}
 \[
 \begin{array}{ll}
 \inf_{\zeta: \zeta^2 \leq 1} \quad \sup_{Z: \|Z\|_2\leq 1}  &  {\mathbb E}|\psi^T X+\eta (\psi^T Z  - \tau \zeta)| ={\mathbb E}|\psi^T X+\eta( \|\psi\|_2  - \tau )_+|
\end{array}
  \]
  in general, for $p\geq 1, \frac{1}{p}+\frac{1}{q} =1,$
   \[
 \begin{array}{ll}
 \inf_{\zeta: \zeta^2 \leq 1} \quad \sup_{Z: \|Z\|_p\leq 1}  &  {\mathbb E}|\psi^T X+\eta (\psi^T Z  - \tau \zeta)| ={\mathbb E}|\psi^T X+\eta( \|\psi\|_q  - \tau )_+|
\end{array}
  \]
\end{coro}


\subsection{Folded Gaussian}
\paragraph{Folded Gaussian}
For Bernoulli Gaussian $X$ with support $I$, if we condition on $I$ fixed, then for any constant $\mu$, using the rotation invariance of Gaussian, $\psi^T X +\mu$ is  Gaussian with mean $\mu$ and standard deviation $\|\psi_I\|_2$, therefore,
\[
|\psi^T X +\mu |
\]
following the folded Gaussian distribution.
\footnote{see: \url{ https://en.wikipedia.org/wiki/Folded_normal_distribution}.}

From general theory of folded Gaussian $ |\mu+ \sigma G |, \quad  G \sim \mathcal{N}(0, 1).$
Then its mean is
\begin{equation}
\label{FoldedGaussianMean}
E_G |\mu+ \sigma G| =  \sqrt{\frac{2}{\pi}} \sigma \exp\{-\mu^2/(2\sigma^2)\} + \mu \left(1 - 2\Phi\left(\frac{-\mu}{\sigma}\right) \right)
\end{equation}
where $\Phi$  is the normal cumulative distribution function:
\[
\Phi(x) = \frac 1 {\sqrt{2\pi}} \int_{-\infty}^x e^{-t^2/2} \, dt.
\]
Its variance is
\begin{equation}
\label{FoldedGaussianVar}
{\rm Var}_G |\mu+ \sigma G| = E_G( |\mu+ \sigma G|-E_G |\mu+ \sigma G|)^2 =\sigma^2+ \mu^2 -(E_G |\mu+ \sigma G|)^2
\end{equation}  
The variance will be useful for concentration with finite sample.  

\paragraph{Asymptotic expansion for mean of folded Gaussian}
Notice that if $\sigma =0$, then
\[
E_G |\mu+ 0\cdot G| =   |\mu|.
\]
If $\mu =0$, then we go back to mean of half-normal,
\[
E_G |\sigma G| =   \sqrt{\frac{2}{\pi}} \sigma
\]
Let 
\[
\gamma:= \frac{\mu}{\sigma}
\] 

\begin{lemma}
\label{ExpansionMeanFoldedGaussian}
We define the ratio of folded gaussian mean as 
\[
\mathcal{R}^f(\gamma) := \frac{E_G |\mu+ \sigma G|}{E_G |\sigma G|} = \frac{E_G |\mu+ \sigma G|}{\sqrt{\frac{2}{\pi}}\sigma} = \sqrt{\frac{\pi}{2}} E_G |\gamma+  G|
\]
We have three different expressions (asymptotic expansion around  $\gamma$) for $\mathcal{R}^f(\gamma)$:
\begin{eqnarray}
\label{FoldedGaussianMean_ratio}
\mathcal{R}^f(\gamma)
&=&  \exp\{-\gamma^2/2\} + \sqrt{\frac{\pi}{2}}\gamma \left(1 - 2\Phi\left(-\gamma\right) \right)\\
&=&  \exp\{-\gamma^2/2\}\{1 + \frac{1}{2}\gamma \left[ \gamma + \frac{ \gamma^3}{3} + \frac{ \gamma^5}{3\cdot 5} + \cdots + \frac{ \gamma^{2n+1}}{(2n+1)!!} + \cdots\right]\}\\
&=& 1 + \frac{3}{2}\gamma^2 + \frac{1}{24} \gamma^4   - \frac{ \gamma^6}{120}  + O(\gamma^8) 
\end{eqnarray}
 \end{lemma}
 
 \begin{lemma}
 \label{ExpansionMeanFoldedGaussian_LB}
when $\gamma<2,$ 
\[
\mathcal{R}^f(\gamma) \geq 1 + \frac{3}{2}\gamma^2
\]
hence
\[
 \begin{array}{ll}
E_G |\mu+ \sigma G|
&\geq \sqrt{\frac{2}{\pi}}\left[\sigma + \frac{3}{2}\mu^2/\sigma \right]
\end{array}
\]

\end{lemma}
\begin{proof}[Proof of lemma~\ref{ExpansionMeanFoldedGaussian}]
Plug in the formula for mean of folded Gaussian \ref{FoldedGaussianMean}, 
we have
\[
E_G |\mu+ \sigma G| = \sqrt{\frac{2}{\pi}} \sigma \exp\{-\gamma^2/2\} + \mu \left(1 - 2\Phi\left(-\gamma\right) \right)
\]
Divide it by 
\[
E_G |\sigma G| =   \sqrt{\frac{2}{\pi}} \sigma
\]
we get the first equality.

Since the CDF of the standard normal distribution can be expanded by integration by parts into a series:
\[
\Phi( \gamma)=\frac{1}{2} + \frac{1}{\sqrt{2\pi}}\cdot e^{- \gamma^2/2} \left[ \gamma + \frac{ \gamma^3}{3} + \frac{ \gamma^5}{3\cdot 5} + \cdots + \frac{ \gamma^{2n+1}}{(2n+1)!!} + \cdots\right]
\]
where $!!$ denotes the double factorial. 

\[
1 - 2\Phi(-\gamma)  = \frac{1}{\sqrt{2\pi}}\cdot e^{- \gamma^2/2}\left[ \gamma + \frac{ \gamma^3}{3} + \frac{ \gamma^5}{3\cdot 5} + \cdots + \frac{ \gamma^{2n+1}}{(2n+1)!!} + \cdots\right]
\]
This gives the second inequality.

Additionally,
 \[
 \begin{array}{ll}
  e^{- \gamma^2/2} &=\sum_{n=0}^{\infty} \frac{ \gamma^{2n}}{(-2)^n (n)!}\\
  &=  1- \frac{ \gamma^2}{2} + \sum_{n=2}^{\infty} \frac{ \gamma^{2n}}{(-2)^n (n)!}
\end{array}
  \] 
then
 \[
 \begin{array}{ll}
& \exp\{-\gamma^2/2\}\{1 + \frac{1}{2}\gamma \left[ \gamma + \frac{ \gamma^3}{3} + \frac{ \gamma^5}{3\cdot 5} + \cdots + \frac{ \gamma^{2n+1}}{(2n+1)!!} + \cdots\right]\}\\
=& (\sum_{n=0}^{\infty} \frac{ \gamma^{2n}}{(-2)^n (n)!})\{1 + \frac{1}{2}\gamma \left[ \sum_{n=0}^{\infty} \frac{ \gamma^{2n+1}}{(2n+1)!!} \right]\}\\

=& 1 + \frac{3}{2}\gamma^2 + \frac{1}{24} \gamma^4   - \frac{ \gamma^6}{120}  + O(\gamma^8) 
\end{array}
  \] 
\end{proof}


\begin{proof}[Proof of lemma~\ref{ExpansionMeanFoldedGaussian_LB}]
Let $C_\Phi( \gamma^2)$ be a function of $ \gamma$, then it is a composition function with the inner funciton being $ \gamma^2,$ defined as follows: 
\[
 \begin{array}{ll}
C_\Phi( \gamma^2) 
&:= (2\Phi( \gamma)-1)/\left(\frac{ \gamma}{\sqrt{2\pi}}\right) \\
&= e^{- \gamma^2/2} \left[1 + \frac{ \gamma^2}{3} + \frac{ \gamma^4}{3\cdot 5} + \cdots + \frac{ \gamma^{2n}}{(2n+1)!!} + \cdots\right]\\
&= [\sum_{n=0}^{\infty} \frac{ \gamma^{2n}}{(2n+1)!!} ]/[\sum_{n=0}^{\infty} \frac{ \gamma^{2n}}{2^n (n)!}] \\
&= [\sum_{n=0}^{\infty} \frac{ \gamma^{2n}}{(2n+1)!!} ]/[\sum_{n=0}^{\infty} \frac{ \gamma^{2n}}{(2n)!!}] \\
&\leq 1
\end{array}
\]
We remark further that
\[
 \begin{array}{ll}
C_\Phi( \gamma^2) 
&= [\sum_{n=0}^{\infty} \frac{ \gamma^{2n}}{(2n+1)!!} ]/[\sum_{n=0}^{\infty} \frac{ \gamma^{2n}}{(2n)!!}] \\
&= [1 + \frac{ \gamma^2}{3}+ +  \frac{ \gamma^4}{15}+ O( \gamma^6) ]/[1 + \frac{ \gamma^2}{2}+ +  \frac{ \gamma^4}{8}+O( \gamma^6)] \\
&= 1 -\frac{ \gamma^2}{6}+  \frac{ \gamma^4}{40}+ O( \gamma^6)
\end{array}
\]
Then
 \[
 \begin{array}{ll}
\mu \left(1 - 2\Phi\left(-\gamma\right) \right)
&= \frac{1}{\sqrt{2\pi}}\cdot \mu^2 \frac{1}{\sigma} C_\Phi(\gamma^2)\\
&= \frac{1}{\sqrt{2\pi}}\cdot \mu^2 \frac{1}{\sigma} (1-\gamma^2/6 +O(\gamma^4))\\
\end{array}
\] 
Plug in all the fact together
\[
 \begin{array}{ll}
E_G |\mu+ \sigma G|
&=   \left[ \sqrt{\frac{2}{\pi}} \sigma \exp\{-\gamma^2/2\} + \frac{1}{\sqrt{2\pi}}\cdot \mu^2 \frac{1}{\sigma} C_\Phi(\gamma^2)\right]\\
&=   \left[ \sqrt{\frac{2}{\pi}} \sigma \{ 1- \frac{\gamma^2}{2} + \sum_{n=2}^{\infty} \frac{\gamma^{2n}}{(-2)^n (n)!}\} + \frac{1}{\sqrt{2\pi}}\cdot \mu^2 \frac{1}{\sigma} C_\Phi(\gamma^2)\right]\\
&= \sqrt{\frac{2}{\pi}}   \left[ \sigma 
\{ 1 + (1+\frac{1}{2}\cdot C_\Phi(\gamma^2))\gamma^2 +   \sum_{n=2}^{\infty} \frac{\gamma^{2n}}{(-2)^n (n)!}\} \}\right]\\
\end{array}
\] 
Define the residual of this expansion as
\[
 \begin{array}{ll}
\mbox{Res}(\gamma^2) 
&:= \frac{ (2\Phi( \gamma)-1)/\left(\frac{ \gamma}{\sqrt{2\pi}}\right)-1}{2}\cdot \gamma^2 +  (\exp\{-\gamma^2/2\} - 1 + \gamma^2/2)
\\
&= \frac{ C_\Phi(\gamma^2)-1}{2}\cdot \gamma^2 +  (\exp\{-\gamma^2/2\} - 1 + \gamma^2/2) \\
&=    \frac{ C_\Phi(\gamma^2)-1}{2}\cdot \gamma^2  +   \{ \sum_{n=2}^{\infty} \frac{\gamma^{2n}}{(-2)^n (n)!}\} \} \\
&=    \frac{-\gamma^4 }{12}  +    \frac{ \gamma^6}{80} +\frac{\gamma^{4}}{(-2)^2 (2)!} +\frac{\gamma^{6}}{(-2)^3 (3)!}+
O(\gamma^8)\\
&=    \frac{\gamma^4 }{24}  - \frac{ \gamma^6}{120}  + O(\gamma^8)
\end{array}
\] 
We know that if $\gamma^2<1,$ 
\[
 \begin{array}{ll}
\mbox{Res}(\gamma^2) 
&=    \frac{\gamma^4 }{24}  - \frac{ \gamma^6}{120}  + O(\gamma^8)\\
&\geq 0
\end{array}
\] 
(We comment that we could push the range for $\mbox{Res}(\gamma^2) 
\geq 0$ further to at least $\gamma^2<5$ if needed.)
Then we give an lower bound:
\[
 \begin{array}{ll}
E_G |\mu+ \sigma G| 
&= \sqrt{\frac{2}{\pi}} \sigma\left[ 1 + \frac{3}{2}\gamma^2+   \mbox{Res}\right]\\
&\geq \sqrt{\frac{2}{\pi}}\left[  \sigma + \frac{3}{2}\mu^2  \sigma^{-1}\right]
\end{array}
\] 


\end{proof}






\subsection{Other setting on $Z$}
\paragraph{ Adversarial outer noise} 
If our generative model is
\[
\begin{array}{ll}
Y&= a* X + \eta_2 Z_2\\
&= a* (X +\eta_2 a^{-1}* Z_2)
\end{array}
\] 
And we consider the worst case objective over $\|Z_2\|_p\leq 1, \quad p\geq 1.$

Let $C_a^{-1}$ be the linear operator represents $a^{-1}*$, then following previous calculation,
\begin{coro}
For $p\geq 1, \frac{1}{p}+\frac{1}{q} =1,$
   \[
 \begin{array}{ll}
 \inf_{\zeta: \zeta^2 \leq 1} \quad \sup_{Z: \|Z_2\|_p \leq 1}  &  {\mathbb E}|\psi^T X+\eta_2  (\psi^T C_a^{-1} Z  - \tau \zeta)| ={\mathbb E}|\psi^T X+\eta( \|C_{a^{-1}}^T \psi\|_q  - \tau )_+|
\end{array}
  \]
\end{coro}

Let $C_{a^{-1}}^T \psi =\bar{a}^{-1}*\psi$, where $\tilde{a}^{-1}$ is defined by flipping the coordinate of $a^{-1}$ (reflecting the coordinates around zero by map $i$ to $-i$ for $i\in {\mathbb Z}$.)
 
 We can replace the definition of $\gamma$ and $\mu_0$  by:
 \[
 \gamma^{a}_{\eta, \epsilon}(\psi, I): = \inf_{\zeta: \zeta^2 \leq 1} \sup_{Z: \|Z\|_2\leq 1} \eta_2  (\psi^T C_a^{-1} Z  - \tau \zeta)/\|\psi_I\|_2 = \eta \frac{ (\|\bar{a}^{-1}*\psi\|_q  - \tau)_+}{\|\psi_I\|_2}
\]
\[
\mu_0^{a} := \eta( \|\tilde{a}^{-1}\|_q-\tau)_+
\]
We can get the same result as theorem~\ref{Thm:StabilityMSE}.

  

  

\subsection{ Gaussian additional noise}
\paragraph{ Gaussian additional observation inner and outer noise} 
If $G_1$ and $G_2$ are two independent standard normal $N(0,I_N)$. Our generative model is
\[
\begin{array}{ll}
Y&= a* (X+  \sigma_1 G_1) +  \sigma_2 G_2\\
&= a* (X+\sigma_1 G_1+\sigma_2 a^{-1}* G_2)
\end{array}
\] 
let $C_{a^{-1}}$ be the linear operator represents $a^{-1}*$, now using the property for sum of independent Gaussian, then $C_{a^{-1}} = C_a^{-1}$ and 
\[
\begin{array}{ll}
Z: = \sigma_1 G_1+\sigma_2 a^{-1}* G_2
\sim N(0, \sigma_1^2 I_N + \sigma_2^2 C_{a^{-1}} C_{a^{-1}}^T)
\end{array}
\] 
then
\[
\begin{array}{ll}
\psi^T Z
\sim N(0, \sigma_1^2\|\psi\|_2^2 + \sigma_2^2\|C_{a^{-1}}^T \psi\|_2^2 )
\end{array}
\] 
 Let $G$ be another standard normal $N(0,I_N)$, for any fixed vector $\mu \in {\mathbb R}^N, $
 \[
 \begin{array}{ll}
  {\mathbb E}_{G_1} {\mathbb E}_{G_2} {\mathbb E}_X|\psi^T X+\sigma_1 \psi^T G_1+\sigma_2 \psi^T (a^{-1}* G_2)  | = 
   {\mathbb E}_I{\mathbb E}_{G_3}  | \sqrt{ \|\psi_I\|_2^2 + \sigma_1^2\|\psi\|_2^2 + \sigma_2^2\|C_{a^{-1}}^T \psi\|_2^2}G_3  |
\end{array}
  \]
In this case, it seems the previous constrained problem no longer can help us find exact solution $\psi^\star =e_0.$ 



We first study $\sigma_2 = 0$ case.
\[
{\mathbb E}_I{\mathbb E}_{G_3}  | \sqrt{ \|\psi_I\|_2^2 + \sigma_1^2\|\psi\|_2^2}G_3 | = 
\sqrt{\frac{2}{\pi}} \cdot {\mathbb E}_I(\sqrt{ \|\psi_I\|_2^2 + \sigma_1^2\|\psi\|_2^2}) =  \sqrt{\frac{2}{\pi}} \cdot \|\psi\|_2 {\mathbb E}_I(\sqrt{ \frac{\|\psi_I\|_2^2}{\|\psi\|_2^2} + \sigma_1^2})
\]
Define worst case objective
\[
F_{\sigma}(\psi):= \sup_{\sigma_1 \leq \sigma} {\mathbb E}_I(\sqrt{ \|\psi_I\|_2^2 + \sigma_1^2\|\psi\|_2^2}) = {\mathbb E}_I(\sqrt{ \|\psi_I\|_2^2 + \sigma^2\|\psi\|_2^2})
\]
The second equality comes from monotonicity.

Again, we assume the minimizer is $\psi^\star $. Since $e_0$ is  minimizer of $L_{0}$ and $\psi^\star $ is  minimizer of $L_{\sigma}$, we have the chain of inequality:
    \[
 \begin{array}{ll}
F_{\sigma}(e_0)
 \geq F_{\sigma}(\psi^\star ) 
 \geq F_{0}(\psi^\star )
 \geq F_{0}(e_0) 
\end{array}
  \] 
Then
    \[
 \begin{array}{ll}
F_{\eta}(e_0)/F_{0}(e_0)
 \geq (F_{\eta}(\psi^\star )/F_{0}(\psi^\star ))(F_{0}(\psi^\star )/F_{0}(e_0) )
\end{array}
  \] 

   \[
 \begin{array}{ll}
 F_{\eta}(\psi)/F_{0}(\psi) = {\mathbb E}_I(\sqrt{ \frac{\|\psi_I\|_2^2}{\|\psi\|_2^2} + \sigma^2})/{\mathbb E}_I(\frac{\|\psi_I\|_2}{\|\psi\|_2} )
\end{array}
  \] 
  
     \[
 \begin{array}{ll}
 F_{\eta}(e_0)/F_{0}(e_0) = {\mathbb E}_I(\sqrt{ \frac{\|(e_0)_I\|_2^2}{\|e_0\|_2^2} + \sigma^2})/{\mathbb E}_I(\frac{\|(e_0)_I\|_2}{\|e_0\|_2} )= (p\sqrt{ 1 + \sigma^2}+ (1-p) \sqrt{ \sigma^2})/p
\end{array}
  \] 
  
    \[
 \begin{array}{ll}
 F_{0}(\psi)/F_{0}(e_0) 
 &= {\mathbb E}_I\|\psi_I\|_2/{\mathbb E}_I\|(e_0)_I\|_2 \\
 &= 1+({\mathbb E}_I\|\psi_I\|_2-{\mathbb E}_I\|(e_0)_I\|_2)/{\mathbb E}_I\|(e_0)_I\|_2 \\
 &= 1+ (B(p,N)/{\mathbb E}_I\|(e_0)_I\|_2) \|\psi -e_0\|_2\\
 &= 1+ (B(p,N)/p) \|\phi\|_2
\end{array}
  \] 

   \[
 \begin{array}{ll}
1+ (B(p,N)/p) \|\phi\|_2 &\leq (F_{\eta}(e_0)/F_{0}(e_0))/ (F_{\eta}(\psi)/F_{0}(\psi)) \\
&\leq \{(p\sqrt{ 1 + \sigma^2}+ (1-p) \sqrt{ \sigma^2})/p\}/\{{\mathbb E}_I(\sqrt{ \frac{\|\psi_I\|_2^2}{\|\psi\|_2^2} + \sigma^2})/{\mathbb E}_I(\frac{\|\psi_I\|_2}{\|\psi\|_2} )\}
\end{array}
  \] 
  First, a loose bound on $\|\phi^\star\|_2$:
  \[
 \begin{array}{ll}
1+ (B(p,N)/p) \|\phi^\star\|_2 
&\leq \{(p\sqrt{ 1 + \sigma^2}+ (1-p) \sqrt{ \sigma^2})/p\}
\end{array}
  \] 
We found that
 \[
 \begin{array}{ll}
 \|\phi^\star\|_2 
&\leq (\{(p\sqrt{ 1 + \sigma^2}+ (1-p) \sqrt{ \sigma^2})/p\}-1)/(B(p,N)/p) \\
&\leq ((p\sqrt{ 1 + \sigma^2}+ (1-p) \sqrt{ \sigma^2})-p)/B(p,N)
\end{array}
  \] 

In general, when $\sigma_1, \sigma_2$ both non-zero, using the same argument,
\[
 \begin{array}{ll}
 \|\phi^\star\|_2 
&\leq \frac{(p[\sqrt{ 1 + \sigma_1^2+ \sigma_2^2\|\tilde a^{-1}\|_2^2}-1]+ (1-p) \sqrt{ \sigma_1^2+ \sigma_2^2\|\tilde a^{-1}\|_2^2})}{B(p,N)}
\end{array}
  \] 

\subsection{Inner and outer Gaussian and adversarial noise}
\paragraph{Inner and outer Gaussian and adversarial noise together}
When 
If $G_1$ and $G_2$ are two independent standard normal $N(0,I_N)$. 
$Z_1$ and $Z_2$ are inner and outer adversarial noise with $p_1$ and $p_2$ norm bounded by $1$.

Our generative model is
\[
\begin{array}{ll}
Y&= a* (X+ \eta_1 Z_1+  \sigma_1 G_1) + \eta_2 Z_2+ \sigma_2 G_2\\
&= a* (X +\eta_1 Z_1 +\eta_2 a^{-1}* Z_2+  \sigma_1 G_1+\sigma_2 a^{-1}* G_2)
\end{array}
\]
We define our robust objective as   
\[
 \begin{array}{ll}
& F_{\eta_1,\eta_2, \sigma_1,\sigma_2}(\psi)\\
:=& 
\inf_{\zeta^2\leq 1}\sup_{Z_1, Z_2: \|Z_1\|_{p_1} \leq 1; \|Z_2\|_{p_2} \leq 1}
{\mathbb E}_{G_1} {\mathbb E}_{G_2} {\mathbb E}_X|\psi^T X+\sigma_1 \psi^T G_1+\sigma_2 \psi^T (a^{-1}* G_2) + \eta_1 Z_1 +\eta_2 a^{-1}* Z_2-\epsilon \zeta |
\end{array}
  \]  
\begin{lemma}
\label{lemma:Obj_Folded_FullNoise}
Let 
 \[
 \begin{array}{ll}
 \mu(\psi) &=(\eta_1\|\psi\|_{q_1}+ \eta_2\|C_{a^{-1}}^T \psi\|_{q_2}-\epsilon)_+\\
 \sigma_n^2(\psi) &= \sqrt{ \sigma_1^2\|\psi\|_2^2 + \sigma_2^2\|C_{a^{-1}}^T \psi\|_2^2}
 \end{array}
  \]
then the objective is a mean of a folded Gaussian random variable:
\[
 \begin{array}{ll}
 F_{\eta_1,\eta_2, \sigma_1,\sigma_2}(\psi)
&= {\mathbb E}_I{\mathbb E}_{G}  | \sqrt{\|\psi_I\|_2^2 + \sigma_n^2(\psi)} G  + \mu(\psi) |\\
&= \sqrt{\frac{2}{\pi}} {\mathbb E}_I[ \sqrt{\|\psi_I\|_2^2 + \sigma_n^2(\psi)} 
\mathcal{R}^f(\frac{\mu(\psi)}{\sqrt{\|\psi_I\|_2^2 + \sigma_n^2(\psi)}})]
\end{array}
  \] 
  where $G$ is scalar standard Gaussian.
\end{lemma}
From a simple inequality:
   \[
 \begin{array}{ll}
 F_{0,0,0,0}(\psi^\star ) &\leq F_{\eta_1,\eta_2, \sigma_1,\sigma_2}(e_0)
\end{array}
  \]  
we have  
\begin{theorem}
\label{thm:Bound_FullNoise}
\[
 \begin{array}{ll}
 B(p,N) \|\psi^\star -e_0\|_2\leq (1-p) \sigma_n(e_0)^2 \mathcal{R}^f(\frac{\mu(e_0)}{\sigma_n(e_0)})+  
 p\sqrt{1+ \sigma^2_n(e_0)} \mathcal{R}^f(\frac{\mu(e_0)}{\sqrt{1+ \sigma^2_n(e_0)}}) - p
\end{array}
  \]  
where
\[
 \begin{array}{ll}
 \mu(e_0) &= (\eta_1 + \eta_2 \| a^{-1} \|_{q_2}-\epsilon)_+\\
 \sigma_n(e_0) &=\sqrt{ \sigma_1^2 + \sigma_2^2\|a^{-1}\|_2^2} 
 \end{array}
  \]
\end{theorem}  

    
\begin{proof}[Proof of lemma~\ref{lemma:Obj_Folded_FullNoise}]
 First, for any fixed vector $\mu$,
\[
 \begin{array}{ll}
  {\mathbb E}_{G_1} {\mathbb E}_{G_2} {\mathbb E}_X|\psi^T X+\sigma_1 \psi^T G_1+\sigma_2 \psi^T (a^{-1}* G_2) + \mu | = 
   {\mathbb E}_I{\mathbb E}_{G_3}  | \sqrt{ \|\psi_I\|_2^2 + \sigma_1^2\|\psi\|_2^2 + \sigma_2^2\|C_{a^{-1}}^T \psi\|_2^2}G_3  + \mu|
\end{array}
  \]
  Let 
 \[
 \begin{array}{ll}
 \mu(\psi) &= (\eta_1\|\psi\|_{q_1}+ \eta_2\|C_{a^{-1}}^T \psi\|_{q_2}-\epsilon)_+\\
 \sigma_n^2(\psi) &= \sqrt{ \sigma_1^2\|\psi\|_2^2 + \sigma_2^2\|C_{a^{-1}}^T \psi\|_2^2}\\
  \sigma_s^2(\psi_I) &=  \|\psi_I\|_2^2 \\
 \sigma^2 (\psi, \psi_I) &= \sigma_s^2(\psi_I) + \sigma_n^2(\psi) 
 \end{array}
  \]
 Using lemma~\ref{min-maxErrorL_p}, we have
\[
 \begin{array}{ll}
F_{\eta_1,\eta_2, \sigma_1,\sigma_2}(\psi)
&={\mathbb E}_I{\mathbb E}_{G}  | \sigma (\psi, \psi_I) G  + \mu(\psi) | \\
&={\mathbb E}_I{\mathbb E}_{G}  | \sqrt{\sigma_s^2(\psi_I) + \sigma_n^2(\psi)} G  + \mu(\psi) | \\
\end{array}
  \]    

\end{proof}

\begin{proof}[Proof of theorem~\ref{thm:Bound_FullNoise}]  
Additionally, for $\tilde a^{-1} $, define its norm as
\[
M_q: = \|\tilde a^{-1} \|_{q} = \| a^{-1} \|_{q}
\]

 When $\psi = e_0$, then
 \[
 \begin{array}{ll}
 \mu(e_0) &= \eta_1 + \eta_2 M_{q_2}\\
  \sigma_n(e_0) &=\sqrt{ \sigma_1^2 + \sigma_2^2 M_{2}^2} \\
 \sigma_s((e_0)_I) &= I_0 
 \end{array}
  \]

  
We have 
\[
 \begin{array}{ll}
 F_{\eta_1,\eta_2, \sigma_1,\sigma_2}(e_0)
 &= 
  {\mathbb E}_{G} {\mathbb E}_{I_0} |  \sigma(e_0, (e_0)_{I_0})  G  +  \mu(e_0)| \\
  &= 
 (1-p){\mathbb E}_{G} |  \sigma_n(e_0) G  +  \mu(e_0)|+ p{\mathbb E}_{G} | \sqrt{1+ \sigma^2_n(e_0)}  G  +  \mu(e_0)|\\
   &=
 \sqrt{\frac{2}{\pi}}[ (1-p) \sigma_n(e_0)^2 \mathcal{R}^f(\frac{\mu(e_0)}{\sigma_n(e_0)})+  
 p\sqrt{1+ \sigma^2_n(e_0)} \mathcal{R}^f(\frac{\mu(e_0)}{\sqrt{1+ \sigma^2_n(e_0)}}) ]
\end{array}
  \]
  Using the chain of inequality:
    \[
 \begin{array}{ll}
F_{\eta_1,\eta_2, \sigma_1,\sigma_2}((e_0)
 \geq F_{\eta_1,\eta_2, \sigma_1,\sigma_2}((\psi^\star ) 
 \geq F_{0,0,0,0}(\psi^\star )
 \geq F_{0,0,0,0}(e_0) 
\end{array}
  \] 
and the fact that
  \[
 \begin{array}{ll}
 F_{0,0,0,0}(\psi^\star )
 &= {\mathbb E}_I\|\psi_I\|_2 \\
 &= {\mathbb E}_I\|(e_0)_I\|_2 + ({\mathbb E}_I\|\psi_I\|_2-{\mathbb E}_I\|(e_0)_I\|_2) \\
 &= \sqrt{\frac{2}{\pi}}[p+ B(p,N) \|\psi -e_0\|_2]
\end{array}
  \]   
  From
   \[
 \begin{array}{ll}
 F_{0,0,0,0}(\psi^\star ) &\leq F_{\eta_1,\eta_2, \sigma_1,\sigma_2}(e_0)
\end{array}
  \]   
we have
  \[
 \begin{array}{ll}
p+ B(p,N) \|\psi^\star -e_0\|_2\leq (1-p) \sigma_n(e_0)^2 \mathcal{R}^f(\frac{\mu(e_0)}{\sigma_n(e_0)})+  
 p\sqrt{1+ \sigma^2_n(e_0)} \mathcal{R}^f(\frac{\mu(e_0)}{\sqrt{1+ \sigma^2_n(e_0)}}) 
\end{array}
  \]   

\end{proof}  
 
  
  
  
  
  
  
  
  
  
  
  
  
  





  
  
  

  
  
  
  
  
  
  
  
  
  
   
   





 
 
 
  
  
  









    
    
    
  
    
   
   
 
 
 


 
 

























\section{Robustness for noisy observation}
\subsection{Inner and outer Gaussian and adversarial noise}
\paragraph{Mathematical formulation: sparse signal plus inner and outer adversarial perturbation and Gaussian noise}
\begin{itemize}
\item
Let $(X_t)_{t \in {\mathbb Z}}$ be IID sampled from  $pN(0,1) + (1-p) \delta_0$, write $X_t = I_t \cdot G_t$ where $I_t$ is IID sampled from Bernoulli, $G_t$ is IID sampled from standard Gaussian.
 \item
The filter $a\in l_1({\mathbb Z}), (\mathcal{F} a)(\omega) \neq 0, \forall \omega\in T.$
\item
Let $G_1$ and $G_2$ are two independent standard normal $N(0,I_N)$. 
\item
Let $\zeta_1$ and $\zeta_2$ are inner and outer adversarial noise with $p_1$ and $p_2$ norm bounded by $1$, where for $p_1, p_2\geq 1, $ $q_1$ and $q_2$ is defined by $\frac{1}{p_i}+\frac{1}{q_i} =1.$
\item
Our data generative model is
\[
\begin{array}{ll}
Y&= a* (X+ \eta_1 \zeta_1+  \sigma_1 G_1) + \eta_2 \zeta_2+ \sigma_2 G_2\\
&= a* (X +\eta_1 \zeta_1 +\eta_2 a^{-1}* \zeta_2+  \sigma_1 G_1+\sigma_2 a^{-1}* G_2)
\end{array}
\]
\item
Let the total perturbation be
\[
\begin{array}{ll}
Z= \eta_1 \zeta_1 +\eta_2 a^{-1}* \zeta_2+  \sigma_1 G_1+\sigma_2 a^{-1}* G_2
\end{array}
\]
We can write data generative model  simply as
\[
\begin{array}{ll}
Y&= a* (X+ Z ) 
\end{array}
\]
\end{itemize}
Let $\tilde{\psi} := w*a$,   $\phi: = \psi - e_0$, then
\[
 \begin{array}{ll}
(w*Y)_0
&= \psi^T (X+  Z ) 
\end{array}
  \]

 \paragraph{Perturbation analysis of the robust convex problems}
The original convex problems is
\[
 \begin{array}{ll}
 \mbox{minimize}_{w}   & {\mathbb E}|\tilde{w}^T Y|\\
\mbox{subject to}  
& v^T w =1,
\end{array}
\tag{$P_1$}
  \]  
Our objective is
\[
|\psi^T (X+  Z )|  = |\psi^T X+\psi^T[\sigma_1  G_1+\sigma_2 (a^{-1}* G_2) + \eta_1 \zeta_1 +\eta_2 a^{-1}* \zeta_2] |
\]
\paragraph{Worst case robust objective over adversarial perturbation}  
For the constrained problem, we first take expectation over Gaussian noise and over the Bernoulli Gaussian signal, then we optimize over the worst case expected objective for all adversarial perturbation, we define our robust objective as   
\[
 \begin{array}{ll}
 F_{\eta_1,\eta_2, \sigma_1,\sigma_2}(\psi)
:=
\sup_{\zeta_1, \zeta_2: \|\zeta_1\|_{p_1} \leq 1; \|\zeta_2\|_{p_2} \leq 1}
{\mathbb E}_{G_1} {\mathbb E}_{G_2} {\mathbb E}_X|\psi^T (X+  Z )|
\end{array}
  \]  
then our robust constrained problem becomes
  \[
 \begin{array}{ll}
 \mbox{minimize}_{\psi} \quad &F_{\eta_1,\eta_2, \sigma_1,\sigma_2}(\psi) \\
\mbox{subject to}  
&u^T \psi =1
\end{array}
  \]
Assume that the minimizer of the robust problem is $\psi^\star$, then we have the chain of inequality:
    \[
 \begin{array}{ll}
F_{\eta_1,\eta_2, \sigma_1,\sigma_2}(e_0)
 \geq F_{\eta_1,\eta_2, \sigma_1,\sigma_2}(\psi^\star ) 
 \geq F_{0,0,0,0}(\psi^\star )
 \geq F_{0,0,0,0}(e_0) =p
\end{array}
  \] 

  
In the following,
let 
\[
D_R( \psi) :=  F_{\eta_1,\eta_2, \sigma_1,\sigma_2}(\psi)-F_{0,0,0,0}(\psi).
\]
We use this fact and the bi-lipschitz property of noiseless objective around $e_0$ to bound the $\ell_2$ norm deviation $\|\psi^\star-e_0\|_2.$

Let $\phi^\star := \psi^\star-e_0$,
we have
   \[
 \begin{array}{ll}
D_R( e_0) &= F_{\eta_1,\eta_2, \sigma_1,\sigma_2}(e_0)-F_{0,0,0,0}(e_0)\\
 &\geq F_{0,0,0,0}(\psi^\star )-F_{0,0,0,0}(e_0) \\
 &\geq B(p,\phi^\star) \|\psi^\star-e_0\|_2\\
 &\geq p (
(1-p)- \tan(\theta) )\cos{\theta} \|\phi^\star\|_2
\end{array}
  \] 

This bound is not tight but it is tight in order and it is easy to compute since we only need to compute  $F_{\eta_1,\eta_2, \sigma_1,\sigma_2}(e_0)$, the robust objective at $e_0$.
 
\subsection{Folded Gaussian}
\paragraph{Folded Gaussian}
For Bernoulli Gaussian $X$ with support $I$, if we condition on $I$ fixed, then  using the rotation invariance of Gaussian, $\psi^T X$ is  Gaussian with mean $0$ and standard deviation $\|\psi_I\|_2$.

From general theory of folded Gaussian,
\footnote{see: \url{ https://en.wikipedia.org/wiki/Folded_normal_distribution}.}
 $ |\mu+ \sigma G |, \quad  G \sim N(0, 1).$
Then its mean is
\begin{equation}
\label{FoldedGaussianMean}
E_G |\mu+ \sigma G| =  \sqrt{\frac{2}{\pi}} \sigma \exp\{-\mu^2/(2\sigma^2)\} + \mu \left(1 - 2\Phi\left(\frac{-\mu}{\sigma}\right) \right)
\end{equation}
where $\Phi$  is the normal cumulative distribution function:
\[
\Phi(x) = \frac 1 {\sqrt{2\pi}} \int_{-\infty}^x e^{-t^2/2} \, dt.
\]
Its variance is
\begin{equation}
\label{FoldedGaussianVar}
{\rm Var}_G |\mu+ \sigma G| = E_G( |\mu+ \sigma G|-E_G |\mu+ \sigma G|)^2 =\sigma^2+ \mu^2 -(E_G |\mu+ \sigma G|)^2
\end{equation}  
The variance will be useful for concentration with finite sample.  

\paragraph{Asymptotic expansion for mean of folded Gaussian}
Notice that if $\sigma =0$, then
\[
E_G |\mu+ 0\cdot G| =   |\mu|.
\]
If $\mu =0$, then we go back to mean of half-normal,
\[
E_G |\sigma G| =   \sqrt{\frac{2}{\pi}} \sigma
\]
Let 
\[
\gamma:= \frac{\mu}{\sigma}
\] 

\begin{lemma}
\label{ExpansionMeanFoldedGaussian}
We define the ratio of folded gaussian mean as 
\[
\mathcal{R}(\gamma) := \frac{E_G |\mu+ \sigma G|}{E_G |\sigma G|} = \frac{E_G |\mu+ \sigma G|}{\sqrt{\frac{2}{\pi}}\sigma} = \sqrt{\frac{\pi}{2}} E_G |\gamma+  G| = \exp\{-\gamma^2/2\} + \sqrt{\frac{\pi}{2}}\gamma \left(1 - 2\Phi\left(-\gamma\right) \right)
\]
We have three different expressions (asymptotic expansion around  $\gamma$) for $\mathcal{R}(\gamma)$:
\begin{eqnarray}
\label{FoldedGaussianMean_ratio}
\mathcal{R}(\gamma)
&=&  \exp\{-\gamma^2/2\} + \sqrt{\frac{\pi}{2}}\gamma \left(1 - 2\Phi\left(-\gamma\right) \right)\\
&=&  \exp\{-\gamma^2/2\}\{1 + \frac{1}{2}\gamma \left[ \gamma + \frac{ \gamma^3}{3} + \frac{ \gamma^5}{3\cdot 5} + \cdots + \frac{ \gamma^{2n+1}}{(2n+1)!!} + \cdots\right]\}\\
&=& 1 + \frac{3}{2}\gamma^2 + \frac{1}{24} \gamma^4   - \frac{ \gamma^6}{120}  + O(\gamma^8) 
\end{eqnarray}
 \end{lemma}
 
  \begin{lemma}
 \label{ExpansionMeanFoldedGaussian_Monot}
$\mathcal{R}(\gamma)$ is an even function that is monotonically non-decreasing for $\gamma \geq 0$.
\end{lemma}
 
 \begin{lemma}
 \label{ExpansionMeanFoldedGaussian_LB}
when $\gamma<2,$ 
\[
\mathcal{R}(\gamma) \geq 1 + \frac{3}{2}\gamma^2
\]
\end{lemma}


\paragraph{Robust optimization}
\begin{lemma}
From Hölder's inequality, for any vector $a \in {\mathbb R}^N$, for $p\geq 1, \frac{1}{p}+\frac{1}{q} =1,$
\[
\sup_{Z: \|Z\|_p\leq 1}(a^T Z) = \|a\|_q,
\]
specifically, when $|Z|^p$ is along the direction of $\mbox{sign}(a)|a|^q$ so that $a^T Z = \|a\|_q$.

\end{lemma}  



\paragraph{Bound on the noisy solution using robust objective}  
\begin{lemma}
\label{lemma:Obj_Folded_FullNoise}
Let 
 \[
 \begin{array}{ll}
 \mu(\psi) &=\eta_1\|\psi\|_{q_1}+ \eta_2\|C_{a^{-1}}^T \psi\|_{q_2}\\
 \sigma_n^2(\psi) &=  \sigma_1^2\|\psi\|_2^2 + \sigma_2^2\|C_{a^{-1}}^T \psi\|_2^2
 \end{array}
  \]
then the objective is a mean of a folded Gaussian random variable:
\[
 \begin{array}{ll}
 F_{\eta_1,\eta_2, \sigma_1,\sigma_2}(\psi)
&= \sup_{\zeta_1, \zeta_2: \|\zeta_1\|_{p_1} \leq 1; \|\zeta_2\|_{p_2} \leq 1}{\mathbb E}_I{\mathbb E}_{G}  | \sqrt{\|\psi_I\|_2^2 + \sigma_n^2(\psi)} G  + \eta_1 \zeta_1 +\eta_2 a^{-1}* \zeta_2 |\\
&= {\mathbb E}_I{\mathbb E}_{G}  | \sqrt{\|\psi_I\|_2^2 + \sigma_n^2(\psi)} G  + \sup_{\zeta_1, \zeta_2: \|\zeta_1\|_{p_1} \leq 1; \|\zeta_2\|_{p_2} \leq 1}[\eta_1 \zeta_1 +\eta_2 a^{-1}* \zeta_2] |\\
&= {\mathbb E}_I{\mathbb E}_{G}  | \sqrt{\|\psi_I\|_2^2 + \sigma_n^2(\psi)} G  + \mu(\psi) |\\
&= \sqrt{\frac{2}{\pi}} {\mathbb E}_I[ \sqrt{\|\psi_I\|_2^2 + \sigma_n^2(\psi)} 
\mathcal{R}(\frac{\mu(\psi)}{\sqrt{\|\psi_I\|_2^2 + \sigma_n^2(\psi)}})]
\end{array}
  \] 
  where $G$ is scalar standard Gaussian.
 
 The second inequality comes from the fact that $\mathcal{R}(\gamma)$ is an even function that is monotonically non-decreasing for $\gamma \geq 0$.
  
\end{lemma}
From a simple inequality:
   \[
 \begin{array}{ll}
 F_{0,0,0,0}(\psi^\star ) &\leq F_{\eta_1,\eta_2, \sigma_1,\sigma_2}(e_0)
\end{array}
  \]  
we have  
\begin{theorem}
\label{thm:Bound_FullNoise}
For $p<p^\star_1 = 1-\tan(\theta),$
 \[
 \begin{array}{ll}
 \|\phi^\star\|_2\leq 
  \frac{(1-p) \sigma_n(e_0)^2 \mathcal{R}(\frac{\mu(e_0)}{\sigma_n(e_0)})+  
 p\sqrt{1+ \sigma^2_n(e_0)} \mathcal{R}(\frac{\mu(e_0)}{\sqrt{1+ \sigma^2_n(e_0)}}) - p}{p (
(1-p)- \tan(\theta) )\cos{\theta} }
\end{array}
  \]  

 
\end{theorem}  
\begin{proof}
 Since
 \[
 \begin{array}{ll}
  B(p,\phi^\star) \|\phi^\star\|_2\leq 
 D_R(e_0) = F_{\eta_1,\eta_2, \sigma_1,\sigma_2}(e_0)-F_{0,0,0,0}(e_0) 
\end{array}
  \] 
  and
   \[
 \begin{array}{ll}
 p (
(1-p)- \tan(\theta) )\cos{\theta} \|\phi^\star\|_2\leq  B(p,\phi^\star) \|\phi^\star\|_2 
\end{array}
  \]  
  where
\[
 \begin{array}{ll}
 D_R(e_0) = (1-p) \sigma_n(e_0)^2 \mathcal{R}(\frac{\mu(e_0)}{\sigma_n(e_0)})+  
 p\sqrt{1+ \sigma^2_n(e_0)} \mathcal{R}(\frac{\mu(e_0)}{\sqrt{1+ \sigma^2_n(e_0)}}) - p
\end{array}
  \]  
where
\[
 \begin{array}{ll}
 \mu(e_0) &= (\eta_1 + \eta_2 \| a^{-1} \|_{q_2}-\epsilon)_+\\
 \sigma_n(e_0) &=\sqrt{ \sigma_1^2 + \sigma_2^2\|a^{-1}\|_2^2} 
 \end{array}
  \]
\end{proof}
\begin{coro}
When $\sigma_1 = \sigma_2 = 0,$
 \[
 \begin{array}{ll}
\|\psi^\star -e_0\|_2\leq 
 \frac{p\mathcal{R}(\frac{\mu(e_0)}{\sqrt{1+ \sigma^2_n(e_0)}}) - p}{p (
(1-p)- \tan(\theta) )\cos{\theta} }
\end{array}
  \]  

\end{coro}

\begin{coro}
When $\eta_1 = \eta_2 = 0,$
 \[
 \begin{array}{ll}
 \|\psi^\star -e_0\|_2\leq 
 \frac{ (1-p) \sigma_n(e_0)^2 +  
 p\sqrt{1+ \sigma^2_n(e_0)}  - p}{p (
(1-p)- \tan(\theta) )\cos{\theta} }
\end{array}
  \]  

\end{coro}








\subsection{All the proofs}    
\begin{proof}[Proof of lemma~\ref{lemma:Obj_Folded_FullNoise}]
 First, for any fixed vector $\mu$,
\[
 \begin{array}{ll}
  {\mathbb E}_{G_1} {\mathbb E}_{G_2} {\mathbb E}_X|\psi^T X+\sigma_1 \psi^T G_1+\sigma_2 \psi^T (a^{-1}* G_2) + \mu | = 
   {\mathbb E}_I{\mathbb E}_{G_3}  | \sqrt{ \|\psi_I\|_2^2 + \sigma_1^2\|\psi\|_2^2 + \sigma_2^2\|C_{a^{-1}}^T \psi\|_2^2}G_3  + \mu|
\end{array}
  \]
  Let 
 \[
 \begin{array}{ll}
 \mu(\psi) &= (\eta_1\|\psi\|_{q_1}+ \eta_2\|C_{a^{-1}}^T \psi\|_{q_2}-\epsilon)_+\\
 \sigma_n^2(\psi) &= \sqrt{ \sigma_1^2\|\psi\|_2^2 + \sigma_2^2\|C_{a^{-1}}^T \psi\|_2^2}\\
  \sigma_s^2(\psi_I) &=  \|\psi_I\|_2^2 \\
 \sigma^2 (\psi, \psi_I) &= \sigma_s^2(\psi_I) + \sigma_n^2(\psi) 
 \end{array}
  \]
 Using lemma~\ref{min-maxErrorL_p}, we have
\[
 \begin{array}{ll}
F_{\eta_1,\eta_2, \sigma_1,\sigma_2}(\psi)
&={\mathbb E}_I{\mathbb E}_{G}  | \sigma (\psi, \psi_I) G  + \mu(\psi) | \\
&={\mathbb E}_I{\mathbb E}_{G}  | \sqrt{\sigma_s^2(\psi_I) + \sigma_n^2(\psi)} G  + \mu(\psi) | \\
\end{array}
  \]    

\end{proof}

\begin{proof}[Proof of theorem~\ref{thm:Bound_FullNoise}]  
Additionally, for $\tilde a^{-1} $, define its norm as
\[
M_q: = \|\tilde a^{-1} \|_{q} = \| a^{-1} \|_{q}
\]

 When $\psi = e_0$, then
 \[
 \begin{array}{ll}
 \mu(e_0) &= \eta_1 + \eta_2 M_{q_2}\\
  \sigma_n(e_0) &=\sqrt{ \sigma_1^2 + \sigma_2^2 M_{2}^2} \\
 \sigma_s((e_0)_I) &= I_0 
 \end{array}
  \]

  
We have 
\[
 \begin{array}{ll}
 F_{\eta_1,\eta_2, \sigma_1,\sigma_2}(e_0)
 &= 
  {\mathbb E}_{G} {\mathbb E}_{I_0} |  \sigma(e_0, (e_0)_{I_0})  G  +  \mu(e_0)| \\
  &= 
 (1-p){\mathbb E}_{G} |  \sigma_n(e_0) G  +  \mu(e_0)|+ p{\mathbb E}_{G} | \sqrt{1+ \sigma^2_n(e_0)}  G  +  \mu(e_0)|\\
   &=
 \sqrt{\frac{2}{\pi}}[ (1-p) \sigma_n(e_0)^2 \mathcal{R}(\frac{\mu(e_0)}{\sigma_n(e_0)})+  
 p\sqrt{1+ \sigma^2_n(e_0)} \mathcal{R}(\frac{\mu(e_0)}{\sqrt{1+ \sigma^2_n(e_0)}}) ]
\end{array}
  \]
  Using the chain of inequality:
    \[
 \begin{array}{ll}
F_{\eta_1,\eta_2, \sigma_1,\sigma_2}((e_0)
 \geq F_{\eta_1,\eta_2, \sigma_1,\sigma_2}((\psi^\star ) 
 \geq F_{0,0,0,0}(\psi^\star )
 \geq F_{0,0,0,0}(e_0) 
\end{array}
  \] 
and the fact that
  \[
 \begin{array}{ll}
 F_{0,0,0,0}(\psi^\star )
 &= {\mathbb E}_I\|\psi_I\|_2 \\
 &= {\mathbb E}_I\|(e_0)_I\|_2 + ({\mathbb E}_I\|\psi_I\|_2-{\mathbb E}_I\|(e_0)_I\|_2) \\
 &= \sqrt{\frac{2}{\pi}}[p+ B(p,N) \|\psi -e_0\|_2]
\end{array}
  \]   
  From
   \[
 \begin{array}{ll}
 F_{0,0,0,0}(\psi^\star ) &\leq F_{\eta_1,\eta_2, \sigma_1,\sigma_2}(e_0)
\end{array}
  \]   
we have
  \[
 \begin{array}{ll}
p+ B(p,N) \|\psi^\star -e_0\|_2\leq (1-p) \sigma_n(e_0)^2 \mathcal{R}(\frac{\mu(e_0)}{\sigma_n(e_0)})+  
 p\sqrt{1+ \sigma^2_n(e_0)} \mathcal{R}(\frac{\mu(e_0)}{\sqrt{1+ \sigma^2_n(e_0)}}) 
\end{array}
  \]   

\end{proof}  







\begin{proof}[Proof of lemma~\ref{ExpansionMeanFoldedGaussian}]
Plug in the formula for mean of folded Gaussian \ref{FoldedGaussianMean}, 
we have
\[
E_G |\mu+ \sigma G| = \sqrt{\frac{2}{\pi}} \sigma \exp\{-\gamma^2/2\} + \mu \left(1 - 2\Phi\left(-\gamma\right) \right)
\]
Divide it by 
\[
E_G |\sigma G| =   \sqrt{\frac{2}{\pi}} \sigma
\]
we get the first equality.

Since the CDF of the standard normal distribution can be expanded by integration by parts into a series:
\[
\Phi( \gamma)=\frac{1}{2} + \frac{1}{\sqrt{2\pi}}\cdot e^{- \gamma^2/2} \left[ \gamma + \frac{ \gamma^3}{3} + \frac{ \gamma^5}{3\cdot 5} + \cdots + \frac{ \gamma^{2n+1}}{(2n+1)!!} + \cdots\right]
\]
where $!!$ denotes the double factorial. 

\[
1 - 2\Phi(-\gamma)  = \frac{1}{\sqrt{2\pi}}\cdot e^{- \gamma^2/2}\left[ \gamma + \frac{ \gamma^3}{3} + \frac{ \gamma^5}{3\cdot 5} + \cdots + \frac{ \gamma^{2n+1}}{(2n+1)!!} + \cdots\right]
\]
This gives the second inequality.

Additionally,
 \[
 \begin{array}{ll}
  e^{- \gamma^2/2} &=\sum_{n=0}^{\infty} \frac{ \gamma^{2n}}{(-2)^n (n)!}\\
  &=  1- \frac{ \gamma^2}{2} + \sum_{n=2}^{\infty} \frac{ \gamma^{2n}}{(-2)^n (n)!}
\end{array}
  \] 
then
 \[
 \begin{array}{ll}
& \exp\{-\gamma^2/2\}\{1 + \frac{1}{2}\gamma \left[ \gamma + \frac{ \gamma^3}{3} + \frac{ \gamma^5}{3\cdot 5} + \cdots + \frac{ \gamma^{2n+1}}{(2n+1)!!} + \cdots\right]\}\\
=& (\sum_{n=0}^{\infty} \frac{ \gamma^{2n}}{(-2)^n (n)!})\{1 + \frac{1}{2}\gamma \left[ \sum_{n=0}^{\infty} \frac{ \gamma^{2n+1}}{(2n+1)!!} \right]\}\\

=& 1 + \frac{3}{2}\gamma^2 + \frac{1}{24} \gamma^4   - \frac{ \gamma^6}{120}  + O(\gamma^8) 
\end{array}
  \] 
\end{proof}


\begin{proof}[Proof of lemma~\ref{ExpansionMeanFoldedGaussian_LB}]
Let $C_\Phi( \gamma^2)$ be a function of $ \gamma$, then it is a composition function with the inner funciton being $ \gamma^2,$ defined as follows: 
\[
 \begin{array}{ll}
C_\Phi( \gamma^2) 
&:= (2\Phi( \gamma)-1)/\left(\frac{ \gamma}{\sqrt{2\pi}}\right) \\
&= e^{- \gamma^2/2} \left[1 + \frac{ \gamma^2}{3} + \frac{ \gamma^4}{3\cdot 5} + \cdots + \frac{ \gamma^{2n}}{(2n+1)!!} + \cdots\right]\\
&= [\sum_{n=0}^{\infty} \frac{ \gamma^{2n}}{(2n+1)!!} ]/[\sum_{n=0}^{\infty} \frac{ \gamma^{2n}}{2^n (n)!}] \\
&= [\sum_{n=0}^{\infty} \frac{ \gamma^{2n}}{(2n+1)!!} ]/[\sum_{n=0}^{\infty} \frac{ \gamma^{2n}}{(2n)!!}] \\
&\leq 1
\end{array}
\]
We remark further that
\[
 \begin{array}{ll}
C_\Phi( \gamma^2) 
&= [\sum_{n=0}^{\infty} \frac{ \gamma^{2n}}{(2n+1)!!} ]/[\sum_{n=0}^{\infty} \frac{ \gamma^{2n}}{(2n)!!}] \\
&= [1 + \frac{ \gamma^2}{3}+ +  \frac{ \gamma^4}{15}+ O( \gamma^6) ]/[1 + \frac{ \gamma^2}{2}+ +  \frac{ \gamma^4}{8}+O( \gamma^6)] \\
&= 1 -\frac{ \gamma^2}{6}+  \frac{ \gamma^4}{40}+ O( \gamma^6)
\end{array}
\]
Then
 \[
 \begin{array}{ll}
\mu \left(1 - 2\Phi\left(-\gamma\right) \right)
&= \frac{1}{\sqrt{2\pi}}\cdot \mu^2 \frac{1}{\sigma} C_\Phi(\gamma^2)\\
&= \frac{1}{\sqrt{2\pi}}\cdot \mu^2 \frac{1}{\sigma} (1-\gamma^2/6 +O(\gamma^4))\\
\end{array}
\] 
Plug in all the fact together
\[
 \begin{array}{ll}
E_G |\mu+ \sigma G|
&=   \left[ \sqrt{\frac{2}{\pi}} \sigma \exp\{-\gamma^2/2\} + \frac{1}{\sqrt{2\pi}}\cdot \mu^2 \frac{1}{\sigma} C_\Phi(\gamma^2)\right]\\
&=   \left[ \sqrt{\frac{2}{\pi}} \sigma \{ 1- \frac{\gamma^2}{2} + \sum_{n=2}^{\infty} \frac{\gamma^{2n}}{(-2)^n (n)!}\} + \frac{1}{\sqrt{2\pi}}\cdot \mu^2 \frac{1}{\sigma} C_\Phi(\gamma^2)\right]\\
&= \sqrt{\frac{2}{\pi}}   \left[ \sigma 
\{ 1 + (1+\frac{1}{2}\cdot C_\Phi(\gamma^2))\gamma^2 +   \sum_{n=2}^{\infty} \frac{\gamma^{2n}}{(-2)^n (n)!}\} \}\right]\\
\end{array}
\] 
Define the residual of this expansion as
\[
 \begin{array}{ll}
\mbox{Res}(\gamma^2) 
&:= \frac{ (2\Phi( \gamma)-1)/\left(\frac{ \gamma}{\sqrt{2\pi}}\right)-1}{2}\cdot \gamma^2 +  (\exp\{-\gamma^2/2\} - 1 + \gamma^2/2)
\\
&= \frac{ C_\Phi(\gamma^2)-1}{2}\cdot \gamma^2 +  (\exp\{-\gamma^2/2\} - 1 + \gamma^2/2) \\
&=    \frac{ C_\Phi(\gamma^2)-1}{2}\cdot \gamma^2  +   \{ \sum_{n=2}^{\infty} \frac{\gamma^{2n}}{(-2)^n (n)!}\} \} \\
&=    \frac{-\gamma^4 }{12}  +    \frac{ \gamma^6}{80} +\frac{\gamma^{4}}{(-2)^2 (2)!} +\frac{\gamma^{6}}{(-2)^3 (3)!}+
O(\gamma^8)\\
&=    \frac{\gamma^4 }{24}  - \frac{ \gamma^6}{120}  + O(\gamma^8)
\end{array}
\] 
We know that if $\gamma^2<1,$ 
\[
 \begin{array}{ll}
\mbox{Res}(\gamma^2) 
&=    \frac{\gamma^4 }{24}  - \frac{ \gamma^6}{120}  + O(\gamma^8)\\
&\geq 0
\end{array}
\] 
(We comment that we could push the range for $\mbox{Res}(\gamma^2) 
\geq 0$ further to at least $\gamma^2<5$ if needed.)
Then we give an lower bound:
\[
 \begin{array}{ll}
E_G |\mu+ \sigma G| 
&= \sqrt{\frac{2}{\pi}} \sigma\left[ 1 + \frac{3}{2}\gamma^2+   \mbox{Res}\right]\\
&\geq \sqrt{\frac{2}{\pi}}\left[  \sigma + \frac{3}{2}\mu^2  \sigma^{-1}\right]
\end{array}
\] 


\end{proof}




\section{Stability for noisy observation}
\subsection{Inner and outer Gaussian and adversarial noise}
\paragraph{Mathematical formulation: sparse signal plus inner and outer adversarial perturbation and Gaussian noise}
\begin{itemize}
\item
Let $(X_t)_{t \in {\mathbb Z}}$ be IID sampled from  $pN(0,1) + (1-p) \delta_0$, write $X_t = I_t \cdot G_t$ where $I_t$ is IID sampled from Bernoulli, $G_t$ is IID sampled from standard Gaussian.
 \item
The filter $a\in l_1({\mathbb Z}), (\mathcal{F} a)(\omega) \neq 0, \forall \omega\in T.$
\item
Let $G_1$ and $G_2$ are two independent standard normal $N(0,I_N)$. 
\item
Let $Z_1$ and $Z_2$ are inner and outer adversarial noise with $p_1$ and $p_2$ norm bounded by $1$, where for $p_1, p_2\geq 1, $ $q_1$ and $q_2$ is defined by $\frac{1}{p_i}+\frac{1}{q_i} =1.$
\item
Our data generative model is
\[
\begin{array}{ll}
Y&= a* (X+ \eta_1 Z_1+  \sigma_1 G_1) + \eta_2 Z_2+ \sigma_2 G_2\\
&= a* (X +\eta_1 Z_1 +\eta_2 a^{-1}* Z_2+  \sigma_1 G_1+\sigma_2 a^{-1}* G_2)
\end{array}
\]
\item
Let the total perturbation be
\[
\begin{array}{ll}
Z= \eta_1 Z_1 +\eta_2 a^{-1}* Z_2+  \sigma_1 G_1+\sigma_2 a^{-1}* G_2
\end{array}
\]
We can write data generative model  simply as
\[
\begin{array}{ll}
Y&= a* (X+ Z ) 
\end{array}
\]
\end{itemize}
Let $\tilde{\psi} := w*a$,   $\phi: = \psi - e_0$, then
\[
 \begin{array}{ll}
(w*Y)_0
&= \psi^T (X+  Z ) 
\end{array}
  \]

 \paragraph{Perturbation analysis of the robust convex problems}
The original convex problems is
\[
 \begin{array}{ll}
 \mbox{minimize}_{w}   & {\mathbb E}|\tilde{w}^T Y|\\
\mbox{subject to}  
& v^T w =1,
\end{array}
\tag{$P_1$}
  \]  
Our objective is
\[
|\psi^T (X+  Z )-\epsilon \zeta|  = |\psi^T X+\psi^T[\sigma_1  G_1+\sigma_2 (a^{-1}* G_2) + \eta_1 Z_1 +\eta_2 a^{-1}* Z_2]-\epsilon \zeta |
\]
\paragraph{Worst case robust objective over adversarial perturbation}  
For the constrained problem, we first take expectation over Gaussian noise and over the Bernoulli Gaussian signal, then we optimize over the worst case expected objective for all adversarial perturbation, we define our robust objective as   
\[
 \begin{array}{ll}
 F_{\eta_1,\eta_2, \sigma_1,\sigma_2}(\psi)
:=
\inf_{\zeta^2\leq 1}\sup_{Z_1, Z_2: \|Z_1\|_{p_1} \leq 1; \|Z_2\|_{p_2} \leq 1}
{\mathbb E}_{G_1} {\mathbb E}_{G_2} {\mathbb E}_X|\psi^T (X+  Z )-\epsilon \zeta|
\end{array}
  \]  
then our robust constrained problem becomes
  \[
 \begin{array}{ll}
 \mbox{minimize}_{\psi} \quad &F_{\eta_1,\eta_2, \sigma_1,\sigma_2}(\psi) \\
\mbox{subject to}  
&u^T \psi =1
\end{array}
  \]
Assume that the minimizer of the robust problem is $\psi^\star$, then we have the chain of inequality:
    \[
 \begin{array}{ll}
F_{\eta_1,\eta_2, \sigma_1,\sigma_2}(e_0)
 \geq F_{\eta_1,\eta_2, \sigma_1,\sigma_2}(\psi^\star ) 
 \geq F_{0,0,0,0}(\psi^\star )
 \geq F_{0,0,0,0}(e_0) =p
\end{array}
  \] 
In the following, we use this fact and the bi-lipschitz property of noiseless objective around $e_0$ to bound the $\ell_2$ norm deviation $\|\psi^\star-e_0\|_2.$
   \[
 \begin{array}{ll}
F_{\eta_1,\eta_2, \sigma_1,\sigma_2}(e_0)-F_{0,0,0,0}(e_0)
 &\geq F_{0,0,0,0}(\psi^\star )-F_{0,0,0,0}(e_0) \\
 &\geq B(p,N) \|\psi^\star-e_0\|_2
\end{array}
  \] 

This bound is not tight but it is tight in order and it is easy to compute since we only need to compute  $F_{\eta_1,\eta_2, \sigma_1,\sigma_2}(e_0)$, the robust objective at $e_0$.
 
\subsection{Folded Gaussian}
\paragraph{Folded Gaussian}
For Bernoulli Gaussian $X$ with support $I$, if we condition on $I$ fixed, then  using the rotation invariance of Gaussian, $\psi^T X$ is  Gaussian with mean $0$ and standard deviation $\|\psi_I\|_2$.

From general theory of folded Gaussian,
\footnote{see: \url{ https://en.wikipedia.org/wiki/Folded_normal_distribution}.}
 $ |\mu+ \sigma G |, \quad  G \sim \mathcal{N}(0, 1).$
Then its mean is
\begin{equation}
\label{FoldedGaussianMean}
E_G |\mu+ \sigma G| =  \sqrt{\frac{2}{\pi}} \sigma \exp\{-\mu^2/(2\sigma^2)\} + \mu \left(1 - 2\Phi\left(\frac{-\mu}{\sigma}\right) \right)
\end{equation}
where $\Phi$  is the normal cumulative distribution function:
\[
\Phi(x) = \frac 1 {\sqrt{2\pi}} \int_{-\infty}^x e^{-t^2/2} \, dt.
\]
Its variance is
\begin{equation}
\label{FoldedGaussianVar}
{\rm Var}_G |\mu+ \sigma G| = E_G( |\mu+ \sigma G|-E_G |\mu+ \sigma G|)^2 =\sigma^2+ \mu^2 -(E_G |\mu+ \sigma G|)^2
\end{equation}  
The variance will be useful for concentration with finite sample.  

\paragraph{Asymptotic expansion for mean of folded Gaussian}
Notice that if $\sigma =0$, then
\[
E_G |\mu+ 0\cdot G| =   |\mu|.
\]
If $\mu =0$, then we go back to mean of half-normal,
\[
E_G |\sigma G| =   \sqrt{\frac{2}{\pi}} \sigma
\]
Let 
\[
\gamma:= \frac{\mu}{\sigma}
\] 

\begin{lemma}
\label{ExpansionMeanFoldedGaussian}
We define the ratio of folded gaussian mean as 
\[
\mathcal{R}^f(\gamma) := \frac{E_G |\mu+ \sigma G|}{E_G |\sigma G|} = \frac{E_G |\mu+ \sigma G|}{\sqrt{\frac{2}{\pi}}\sigma} = \sqrt{\frac{\pi}{2}} E_G |\gamma+  G|
\]
We have three different expressions (asymptotic expansion around  $\gamma$) for $\mathcal{R}^f(\gamma)$:
\begin{eqnarray}
\label{FoldedGaussianMean_ratio}
\mathcal{R}^f(\gamma)
&=&  \exp\{-\gamma^2/2\} + \sqrt{\frac{\pi}{2}}\gamma \left(1 - 2\Phi\left(-\gamma\right) \right)\\
&=&  \exp\{-\gamma^2/2\}\{1 + \frac{1}{2}\gamma \left[ \gamma + \frac{ \gamma^3}{3} + \frac{ \gamma^5}{3\cdot 5} + \cdots + \frac{ \gamma^{2n+1}}{(2n+1)!!} + \cdots\right]\}\\
&=& 1 + \frac{3}{2}\gamma^2 + \frac{1}{24} \gamma^4   - \frac{ \gamma^6}{120}  + O(\gamma^8) 
\end{eqnarray}
 \end{lemma}
 
 \begin{lemma}
 \label{ExpansionMeanFoldedGaussian_LB}
when $\gamma<2,$ 
\[
\mathcal{R}^f(\gamma) \geq 1 + \frac{3}{2}\gamma^2
\]
\end{lemma}


\paragraph{Robust optimization}
We want to find the min-max value 
\[
\inf_{\zeta^2\leq 1}\sup_{Z_1, Z_2: \|Z_1\|_{p_1} \leq 1; \|Z_2\|_{p_2} \leq 1} H( \psi^T (\eta_1 Z_1 +\eta_2 a^{-1}* Z_2)  - \epsilon \zeta)
\]
for a function $H$,
\[
H(t) := {\mathbb E}_{G_1} {\mathbb E}_{G_2} {\mathbb E}_X|t+ \psi^T (X+  \sigma_1  G_1+\sigma_2 (a^{-1}* G_2) )|
\]
\begin{lemma}
From Hölder's inequality, for any vector $a \in {\mathbb R}^N$, for $p\geq 1, \frac{1}{p}+\frac{1}{q} =1,$
\[
\sup_{Z: \|Z\|_p\leq 1}(a^T Z) = \|a\|_q,
\]
specifically, when $|Z|^p$ is along the direction of $\mbox{sign}(a)|a|^q$ so that $a^T Z = \|a\|_q$.

\end{lemma}  


\begin{lemma}
\label{min-maxError}
For $A>0$, 
\[
 \begin{array}{ll}
\inf_{\zeta: \zeta^2 \leq 1} (A - \epsilon \zeta) = (A - \epsilon )_+
\end{array}
\]
where $(\cdot)_+: = \max{(0, \cdot)}$ is the ReLU function.
\end{lemma}
\begin{lemma}
\label{min-maxErrorL_p}
Let
 \[
 \begin{array}{ll}
 \mu(\psi) &=(\eta_1\|\psi\|_{q_1}+ \eta_2\|C_{a^{-1}}^T \psi\|_{q_2}-\epsilon)_+
 \end{array}
  \]
  where $(\cdot)_+: = \max{(0, \cdot)}$ is the ReLU function,
  then
\[
 \begin{array}{ll}
\inf_{\zeta^2\leq 1}\sup_{Z_1, Z_2: \|Z_1\|_{p_1} \leq 1; \|Z_2\|_{p_2} \leq 1} H( \psi^T (\eta_1 Z_1 +\eta_2 a^{-1}* Z_2)  - \epsilon \zeta) = 
H( \mu(\psi))
\end{array}
\]

\end{lemma}
\paragraph{Bound on the noisy solution using robust objective}  
\begin{lemma}
\label{lemma:Obj_Folded_FullNoise}
Let 
 \[
 \begin{array}{ll}
 \mu(\psi) &=(\eta_1\|\psi\|_{q_1}+ \eta_2\|C_{a^{-1}}^T \psi\|_{q_2}-\epsilon)_+\\
 \sigma_n^2(\psi) &= \sqrt{ \sigma_1^2\|\psi\|_2^2 + \sigma_2^2\|C_{a^{-1}}^T \psi\|_2^2}
 \end{array}
  \]
then the objective is a mean of a folded Gaussian random variable:
\[
 \begin{array}{ll}
 F_{\eta_1,\eta_2, \sigma_1,\sigma_2}(\psi)
&= {\mathbb E}_I{\mathbb E}_{G}  | \sqrt{\|\psi_I\|_2^2 + \sigma_n^2(\psi)} G  + \mu(\psi) |\\
&= \sqrt{\frac{2}{\pi}} {\mathbb E}_I[ \sqrt{\|\psi_I\|_2^2 + \sigma_n^2(\psi)} 
\mathcal{R}^f(\frac{\mu(\psi)}{\sqrt{\|\psi_I\|_2^2 + \sigma_n^2(\psi)}})]
\end{array}
  \] 
  where $G$ is scalar standard Gaussian.
\end{lemma}
From a simple inequality:
   \[
 \begin{array}{ll}
 F_{0,0,0,0}(\psi^\star ) &\leq F_{\eta_1,\eta_2, \sigma_1,\sigma_2}(e_0)
\end{array}
  \]  
we have  
\begin{theorem}
\label{thm:Bound_FullNoise}
\[
 \begin{array}{ll}
 B(p,N) \|\psi^\star -e_0\|_2\leq (1-p) \sigma_n(e_0)^2 \mathcal{R}^f(\frac{\mu(e_0)}{\sigma_n(e_0)})+  
 p\sqrt{1+ \sigma^2_n(e_0)} \mathcal{R}^f(\frac{\mu(e_0)}{\sqrt{1+ \sigma^2_n(e_0)}}) - p
\end{array}
  \]  
where
\[
 \begin{array}{ll}
 \mu(e_0) &= (\eta_1 + \eta_2 \| a^{-1} \|_{q_2}-\epsilon)_+\\
 \sigma_n(e_0) &=\sqrt{ \sigma_1^2 + \sigma_2^2\|a^{-1}\|_2^2} 
 \end{array}
  \]
\end{theorem}  

\subsection{Proofs}    
\begin{proof}[Proof of lemma~\ref{lemma:Obj_Folded_FullNoise}]
 First, for any fixed vector $\mu$,
\[
 \begin{array}{ll}
  {\mathbb E}_{G_1} {\mathbb E}_{G_2} {\mathbb E}_X|\psi^T X+\sigma_1 \psi^T G_1+\sigma_2 \psi^T (a^{-1}* G_2) + \mu | = 
   {\mathbb E}_I{\mathbb E}_{G_3}  | \sqrt{ \|\psi_I\|_2^2 + \sigma_1^2\|\psi\|_2^2 + \sigma_2^2\|C_{a^{-1}}^T \psi\|_2^2}G_3  + \mu|
\end{array}
  \]
  Let 
 \[
 \begin{array}{ll}
 \mu(\psi) &= (\eta_1\|\psi\|_{q_1}+ \eta_2\|C_{a^{-1}}^T \psi\|_{q_2}-\epsilon)_+\\
 \sigma_n^2(\psi) &= \sqrt{ \sigma_1^2\|\psi\|_2^2 + \sigma_2^2\|C_{a^{-1}}^T \psi\|_2^2}\\
  \sigma_s^2(\psi_I) &=  \|\psi_I\|_2^2 \\
 \sigma^2 (\psi, \psi_I) &= \sigma_s^2(\psi_I) + \sigma_n^2(\psi) 
 \end{array}
  \]
 Using lemma~\ref{min-maxErrorL_p}, we have
\[
 \begin{array}{ll}
F_{\eta_1,\eta_2, \sigma_1,\sigma_2}(\psi)
&={\mathbb E}_I{\mathbb E}_{G}  | \sigma (\psi, \psi_I) G  + \mu(\psi) | \\
&={\mathbb E}_I{\mathbb E}_{G}  | \sqrt{\sigma_s^2(\psi_I) + \sigma_n^2(\psi)} G  + \mu(\psi) | \\
\end{array}
  \]    

\end{proof}

\begin{proof}[Proof of theorem~\ref{thm:Bound_FullNoise}]  
Additionally, for $\tilde a^{-1} $, define its norm as
\[
M_q: = \|\tilde a^{-1} \|_{q} = \| a^{-1} \|_{q}
\]

 When $\psi = e_0$, then
 \[
 \begin{array}{ll}
 \mu(e_0) &= \eta_1 + \eta_2 M_{q_2}\\
  \sigma_n(e_0) &=\sqrt{ \sigma_1^2 + \sigma_2^2 M_{2}^2} \\
 \sigma_s((e_0)_I) &= I_0 
 \end{array}
  \]

  
We have 
\[
 \begin{array}{ll}
 F_{\eta_1,\eta_2, \sigma_1,\sigma_2}(e_0)
 &= 
  {\mathbb E}_{G} {\mathbb E}_{I_0} |  \sigma(e_0, (e_0)_{I_0})  G  +  \mu(e_0)| \\
  &= 
 (1-p){\mathbb E}_{G} |  \sigma_n(e_0) G  +  \mu(e_0)|+ p{\mathbb E}_{G} | \sqrt{1+ \sigma^2_n(e_0)}  G  +  \mu(e_0)|\\
   &=
 \sqrt{\frac{2}{\pi}}[ (1-p) \sigma_n(e_0)^2 \mathcal{R}^f(\frac{\mu(e_0)}{\sigma_n(e_0)})+  
 p\sqrt{1+ \sigma^2_n(e_0)} \mathcal{R}^f(\frac{\mu(e_0)}{\sqrt{1+ \sigma^2_n(e_0)}}) ]
\end{array}
  \]
  Using the chain of inequality:
    \[
 \begin{array}{ll}
F_{\eta_1,\eta_2, \sigma_1,\sigma_2}((e_0)
 \geq F_{\eta_1,\eta_2, \sigma_1,\sigma_2}((\psi^\star ) 
 \geq F_{0,0,0,0}(\psi^\star )
 \geq F_{0,0,0,0}(e_0) 
\end{array}
  \] 
and the fact that
  \[
 \begin{array}{ll}
 F_{0,0,0,0}(\psi^\star )
 &= {\mathbb E}_I\|\psi_I\|_2 \\
 &= {\mathbb E}_I\|(e_0)_I\|_2 + ({\mathbb E}_I\|\psi_I\|_2-{\mathbb E}_I\|(e_0)_I\|_2) \\
 &= \sqrt{\frac{2}{\pi}}[p+ B(p,N) \|\psi -e_0\|_2]
\end{array}
  \]   
  From
   \[
 \begin{array}{ll}
 F_{0,0,0,0}(\psi^\star ) &\leq F_{\eta_1,\eta_2, \sigma_1,\sigma_2}(e_0)
\end{array}
  \]   
we have
  \[
 \begin{array}{ll}
p+ B(p,N) \|\psi^\star -e_0\|_2\leq (1-p) \sigma_n(e_0)^2 \mathcal{R}^f(\frac{\mu(e_0)}{\sigma_n(e_0)})+  
 p\sqrt{1+ \sigma^2_n(e_0)} \mathcal{R}^f(\frac{\mu(e_0)}{\sqrt{1+ \sigma^2_n(e_0)}}) 
\end{array}
  \]   

\end{proof}  







\begin{proof}[Proof of lemma~\ref{ExpansionMeanFoldedGaussian}]
Plug in the formula for mean of folded Gaussian \ref{FoldedGaussianMean}, 
we have
\[
E_G |\mu+ \sigma G| = \sqrt{\frac{2}{\pi}} \sigma \exp\{-\gamma^2/2\} + \mu \left(1 - 2\Phi\left(-\gamma\right) \right)
\]
Divide it by 
\[
E_G |\sigma G| =   \sqrt{\frac{2}{\pi}} \sigma
\]
we get the first equality.

Since the CDF of the standard normal distribution can be expanded by integration by parts into a series:
\[
\Phi( \gamma)=\frac{1}{2} + \frac{1}{\sqrt{2\pi}}\cdot e^{- \gamma^2/2} \left[ \gamma + \frac{ \gamma^3}{3} + \frac{ \gamma^5}{3\cdot 5} + \cdots + \frac{ \gamma^{2n+1}}{(2n+1)!!} + \cdots\right]
\]
where $!!$ denotes the double factorial. 

\[
1 - 2\Phi(-\gamma)  = \frac{1}{\sqrt{2\pi}}\cdot e^{- \gamma^2/2}\left[ \gamma + \frac{ \gamma^3}{3} + \frac{ \gamma^5}{3\cdot 5} + \cdots + \frac{ \gamma^{2n+1}}{(2n+1)!!} + \cdots\right]
\]
This gives the second inequality.

Additionally,
 \[
 \begin{array}{ll}
  e^{- \gamma^2/2} &=\sum_{n=0}^{\infty} \frac{ \gamma^{2n}}{(-2)^n (n)!}\\
  &=  1- \frac{ \gamma^2}{2} + \sum_{n=2}^{\infty} \frac{ \gamma^{2n}}{(-2)^n (n)!}
\end{array}
  \] 
then
 \[
 \begin{array}{ll}
& \exp\{-\gamma^2/2\}\{1 + \frac{1}{2}\gamma \left[ \gamma + \frac{ \gamma^3}{3} + \frac{ \gamma^5}{3\cdot 5} + \cdots + \frac{ \gamma^{2n+1}}{(2n+1)!!} + \cdots\right]\}\\
=& (\sum_{n=0}^{\infty} \frac{ \gamma^{2n}}{(-2)^n (n)!})\{1 + \frac{1}{2}\gamma \left[ \sum_{n=0}^{\infty} \frac{ \gamma^{2n+1}}{(2n+1)!!} \right]\}\\

=& 1 + \frac{3}{2}\gamma^2 + \frac{1}{24} \gamma^4   - \frac{ \gamma^6}{120}  + O(\gamma^8) 
\end{array}
  \] 
\end{proof}


\begin{proof}[Proof of lemma~\ref{ExpansionMeanFoldedGaussian_LB}]
Let $C_\Phi( \gamma^2)$ be a function of $ \gamma$, then it is a composition function with the inner funciton being $ \gamma^2,$ defined as follows: 
\[
 \begin{array}{ll}
C_\Phi( \gamma^2) 
&:= (2\Phi( \gamma)-1)/\left(\frac{ \gamma}{\sqrt{2\pi}}\right) \\
&= e^{- \gamma^2/2} \left[1 + \frac{ \gamma^2}{3} + \frac{ \gamma^4}{3\cdot 5} + \cdots + \frac{ \gamma^{2n}}{(2n+1)!!} + \cdots\right]\\
&= [\sum_{n=0}^{\infty} \frac{ \gamma^{2n}}{(2n+1)!!} ]/[\sum_{n=0}^{\infty} \frac{ \gamma^{2n}}{2^n (n)!}] \\
&= [\sum_{n=0}^{\infty} \frac{ \gamma^{2n}}{(2n+1)!!} ]/[\sum_{n=0}^{\infty} \frac{ \gamma^{2n}}{(2n)!!}] \\
&\leq 1
\end{array}
\]
We remark further that
\[
 \begin{array}{ll}
C_\Phi( \gamma^2) 
&= [\sum_{n=0}^{\infty} \frac{ \gamma^{2n}}{(2n+1)!!} ]/[\sum_{n=0}^{\infty} \frac{ \gamma^{2n}}{(2n)!!}] \\
&= [1 + \frac{ \gamma^2}{3}+ +  \frac{ \gamma^4}{15}+ O( \gamma^6) ]/[1 + \frac{ \gamma^2}{2}+ +  \frac{ \gamma^4}{8}+O( \gamma^6)] \\
&= 1 -\frac{ \gamma^2}{6}+  \frac{ \gamma^4}{40}+ O( \gamma^6)
\end{array}
\]
Then
 \[
 \begin{array}{ll}
\mu \left(1 - 2\Phi\left(-\gamma\right) \right)
&= \frac{1}{\sqrt{2\pi}}\cdot \mu^2 \frac{1}{\sigma} C_\Phi(\gamma^2)\\
&= \frac{1}{\sqrt{2\pi}}\cdot \mu^2 \frac{1}{\sigma} (1-\gamma^2/6 +O(\gamma^4))\\
\end{array}
\] 
Plug in all the fact together
\[
 \begin{array}{ll}
E_G |\mu+ \sigma G|
&=   \left[ \sqrt{\frac{2}{\pi}} \sigma \exp\{-\gamma^2/2\} + \frac{1}{\sqrt{2\pi}}\cdot \mu^2 \frac{1}{\sigma} C_\Phi(\gamma^2)\right]\\
&=   \left[ \sqrt{\frac{2}{\pi}} \sigma \{ 1- \frac{\gamma^2}{2} + \sum_{n=2}^{\infty} \frac{\gamma^{2n}}{(-2)^n (n)!}\} + \frac{1}{\sqrt{2\pi}}\cdot \mu^2 \frac{1}{\sigma} C_\Phi(\gamma^2)\right]\\
&= \sqrt{\frac{2}{\pi}}   \left[ \sigma 
\{ 1 + (1+\frac{1}{2}\cdot C_\Phi(\gamma^2))\gamma^2 +   \sum_{n=2}^{\infty} \frac{\gamma^{2n}}{(-2)^n (n)!}\} \}\right]\\
\end{array}
\] 
Define the residual of this expansion as
\[
 \begin{array}{ll}
\mbox{Res}(\gamma^2) 
&:= \frac{ (2\Phi( \gamma)-1)/\left(\frac{ \gamma}{\sqrt{2\pi}}\right)-1}{2}\cdot \gamma^2 +  (\exp\{-\gamma^2/2\} - 1 + \gamma^2/2)
\\
&= \frac{ C_\Phi(\gamma^2)-1}{2}\cdot \gamma^2 +  (\exp\{-\gamma^2/2\} - 1 + \gamma^2/2) \\
&=    \frac{ C_\Phi(\gamma^2)-1}{2}\cdot \gamma^2  +   \{ \sum_{n=2}^{\infty} \frac{\gamma^{2n}}{(-2)^n (n)!}\} \} \\
&=    \frac{-\gamma^4 }{12}  +    \frac{ \gamma^6}{80} +\frac{\gamma^{4}}{(-2)^2 (2)!} +\frac{\gamma^{6}}{(-2)^3 (3)!}+
O(\gamma^8)\\
&=    \frac{\gamma^4 }{24}  - \frac{ \gamma^6}{120}  + O(\gamma^8)
\end{array}
\] 
We know that if $\gamma^2<1,$ 
\[
 \begin{array}{ll}
\mbox{Res}(\gamma^2) 
&=    \frac{\gamma^4 }{24}  - \frac{ \gamma^6}{120}  + O(\gamma^8)\\
&\geq 0
\end{array}
\] 
(We comment that we could push the range for $\mbox{Res}(\gamma^2) 
\geq 0$ further to at least $\gamma^2<5$ if needed.)
Then we give an lower bound:
\[
 \begin{array}{ll}
E_G |\mu+ \sigma G| 
&= \sqrt{\frac{2}{\pi}} \sigma\left[ 1 + \frac{3}{2}\gamma^2+   \mbox{Res}\right]\\
&\geq \sqrt{\frac{2}{\pi}}\left[  \sigma + \frac{3}{2}\mu^2  \sigma^{-1}\right]
\end{array}
\] 


\end{proof}



\subsection{Constrained Convex algorithms }
 \paragraph{Constrained noisy convex problems}
Then we define the constrained convex program that allows noise,
\[
 \begin{array}{ll}
 \mbox{minimize}_{w,z}   & |\alpha| \\
\mbox{subject to}  
& v^T w =1,\\
& {\mathbb E}(\alpha -\tilde{w}^T Y)^2 \leq \epsilon^2
\end{array}
\tag{$P^c_{1,\epsilon}$}
\label{eq:ConstrainProb}
  \]
  If we write $\alpha =\tilde{w}^T Y - \epsilon \zeta $ , then
\[
 \begin{array}{ll}
 \mbox{minimize}_{w,\zeta}   & {\mathbb E}|\tilde{w}^T Y - \epsilon \zeta| \\
\mbox{subject to}  
& v^T w =1,\\
& \zeta^2 \leq 1
\end{array}
\tag{$P^c_{1,\epsilon}$}
  \]
If we set $\epsilon = 0,$ then we recover 
the original convex problems for noiseless setting
\[
 \begin{array}{ll}
 \mbox{minimize}_{w}   & {\mathbb E}|\tilde{w}^T Y|\\
\mbox{subject to}  
& v^T w =1,
\end{array}
\tag{$P_1$}
  \]
\section{Numerical Experiments}








Now we look at numerical experiments of $2-$D image blind deconvolution for different choices of $X^\star$ and $a^{-1,\star}$.

\clearpage
\begin{figure} 
\centering
\includegraphics[width=.2\textwidth]{./Fig_2D_Exp/Sparse2/Y.png}
\includegraphics[width=.2\textwidth]{./Fig_2D_Exp/Sparse2/X_cvx.png}
\includegraphics[width=.2\textwidth]{./Fig_2D_Exp/Sparse2/X_gt.png}
\includegraphics[width=.2\textwidth]{./Fig_2D_Exp/Sparse2/f_cvx.png}
\caption{
Sparse point: $Y$, solved $X$, true $X$, solved $f$. 
}
\label{fig-2D-Sparse}
\end{figure}
\begin{itemize}
\item $X$: $2-$D Bernoulli Gaussian IID of size $80 \times 80$.
\item $f^\star$: $1-$D filter $(1, -s, 0)$, $s=0.9$ centered and rotated by $45$ degree.
\item $Y =a^\star*X$, where the filter inverse $a^\star = f^{\star,-1} $ is defined by discrete Fourier transform on $80 \times 80$ grid.  
\item
The plots in order, first row:
$Y = a^\star * X^\star$, $X^{\mbox{cvx}}$,  $X^\star$, $f^{\mbox{cvx}}= f^\star$.  
\end{itemize} 
\clearpage



\begin{figure} 
\centering
\includegraphics[width=.2\textwidth]{./Fig_2D_Exp/Sparse3/Y.png}
\includegraphics[width=.2\textwidth]{./Fig_2D_Exp/Sparse3/X_cvx.png}
\includegraphics[width=.2\textwidth]{./Fig_2D_Exp/Sparse3/X_gt.png}
\includegraphics[width=.2\textwidth]{./Fig_2D_Exp/Sparse3/f_cvx.png}
\centering
\includegraphics[width=.2\textwidth]{./Fig_2D_Exp/Sparse3/Y_init_prev.png}
\includegraphics[width=.2\textwidth]{./Fig_2D_Exp/Sparse3/a_init.png}
\includegraphics[width=.2\textwidth]{./Fig_2D_Exp/Sparse3/a_gt.png}

\centering
\includegraphics[width=.2\textwidth]{./Fig_2D_Exp/Sparse3/DL_prev/X_dl.png}
\includegraphics[width=.2\textwidth]{./Fig_2D_Exp/Sparse3/DL_prev/a_dl.png}
\includegraphics[width=.2\textwidth]{./Fig_2D_Exp/Sparse3/DL/X_dl.png}
\includegraphics[width=.2\textwidth]{./Fig_2D_Exp/Sparse3/DL/a_dl.png}
\caption{Sphere: $Y$, solved $X$, true $X$, solved $f$;
previous $X$, previous $a$ as initialization, $a$ truth;
$X$ solved by non-convex alternating formulation, $a$ solved by non-convex alternating  formulation.
}
\label{fig-2D-Sphere-BD}
\end{figure}



\begin{itemize}
\item $X$: $2-$D Bernoulli Gaussian IID of size $60 \times 60$.
\item $f^\star$: $1-$D filter $(1, -s, 0)$ with $s=0.99$ centered and rotated by $45$ degree.
\item $Y =a^{\star}*X$, where the filter inverse is defined by discrete Fourier transform on $60 \times 60$ grid.
\item
$a^{\mbox{init}}$ = $f^{\mbox{init},-1}$, where $f^{\mbox{init}}$ is $1-$D filter $(1, -s_{\mbox{init}}, 0)$ with $s_{\mbox{init}}=0.8$ centered and rotated by $45$ degree.
\item $X^{\mbox{init}} = a^{\mbox{init}} * X$ is the initial guess $a^{\mbox{init}}$ convolve with $X^\star$.
\item $a^{\mbox{ncvx}}$ is the forward solved by alternating non-convex algorithm.
\item $X^{\mbox{ncvx}}$ is the initial guess of $X$ solved by alternating non-convex algorithm.
\item
The plots in order, first row:
$Y = a^\star * X^\star$, $X^{\mbox{cvx}}$,  $X^\star$, $f^{\mbox{cvx}}= f^\star$.  
\item
second row:
$X^{\mbox{init}}$, $a^{\mbox{init}}$, $a^{\star}$;
\item
third row:
$X^{\mbox{nc, init}}$, $a^{\mbox{nc,init}}$, $X^{\mbox{nc}}$, $a^{\mbox{nc}}$;
\end{itemize} 

\clearpage

\begin{figure} 
\centering
\includegraphics[width=.2\textwidth]{./Fig_2D_Exp/New_Lis2/Y.png}
\includegraphics[width=.2\textwidth]{./Fig_2D_Exp/New_Lis2/X_gt.png}
\includegraphics[width=.2\textwidth]{./Fig_2D_Exp/New_Lis2/f_gt.png}
\includegraphics[width=.2\textwidth]{./Fig_2D_Exp/New_Lis2/a_gt.png}
\centering
\includegraphics[width=.2\textwidth]{./Fig_2D_Exp/New_Lis2/X_cvx.png}
\includegraphics[width=.2\textwidth]{./Fig_2D_Exp/New_Lis2/f_cvx.png}
\centering
\includegraphics[width=.2\textwidth]{./Fig_2D_Exp/New_Lis2/X_dl.png}
\includegraphics[width=.2\textwidth]{./Fig_2D_Exp/New_Lis2/a_dl.png}
\caption{Sphere: $Y$, solved $X$, true $X$, solved $f$;
previous $X$, previous $a$ as initialization, $a$ truth;
$X$ solved by non-convex alternating formulation, $a$ solved by non-convex alternating  formulation
}
\label{fig-2D-Lis-new2}
\end{figure}

\begin{itemize}
\item $X$: $2-$D centered shaped $8$ with frame size $15$ on size $60 \times 60$ grid.
\item $f^\star$: convolution of two $1-$D filters: 
$(1, -s_1, 0)$ with $s_1=0.9995$ centered and rotated by $45$ degree and 
$(1, -s_2, 0)$ with $s_2=0.9995$ centered and rotated by $0$ degree.
\item $Y =a^{\star}*X$, where the filter inverse is defined by discrete Fourier transform on $60 \times 60$ grid.
\item $a^{\mbox{nc}}$ is the forward solved by alternating non-convex algorithm.
\item $X^{\mbox{nc}}$ is the initial guess of $X$ solved by alternating non-convex algorithm.
\item
The plots in order, first row:
$Y = a^\star * X^\star$,   $X^\star$, $ f^\star$, $a^\star$.  
\item
second row:
$X^{\mbox{cvx}}$, $f^{\mbox{cvx}}$ ,$X^{\mbox{nc}}$, $a^{\mbox{nc}}$;

\end{itemize} 
\clearpage


\begin{figure} 
\centering
\includegraphics[width=.2\textwidth]{./Fig_2D_Exp/New_Lis1/Y.png}
\includegraphics[width=.2\textwidth]{./Fig_2D_Exp/New_Lis1/X_cvx.png}
\includegraphics[width=.2\textwidth]{./Fig_2D_Exp/New_Lis1/f_cvx.png}
\centering
\includegraphics[width=.2\textwidth]{./Fig_2D_Exp/New_Lis1/a_cvx.png}
\includegraphics[width=.2\textwidth]{./Fig_2D_Exp/New_Lis1/X_dl.png}
\includegraphics[width=.2\textwidth]{./Fig_2D_Exp/New_Lis1/a_dl.png}
\caption{ $Y$, solved $X$, true $X$, solved $f$;
previous $X$, previous $a$ as initialization, $a$ truth;
$X$ solved by non-convex alternating formulation, $a$ solved by non-convex alternating  formulation.
}
\label{fig-2D-Lis-new1}
\end{figure}


\begin{itemize}
\item $X$: $2-$D centered shaped $8$ on size $60 \times 60$ grid.
\item $f^\star$: convolution of two $1-$D centered filters:
$(1, -s_1, 0)$, $s_1 = 0.999$, rotated by $45$ degree, .
$(1, -s_12, 0)$, $s_2 = 0.9$, rotated by $0$ degree.
\item $Y =f^{\star,-1}*X$, where the filter inverse is defined by discrete Fourier transform on $60 \times 60$ grid.
\item $a^{\mbox{nc}}$ is the forward solved by alternating non-convex algorithm.
\item $X^{\mbox{nc}}$ is the initial guess of $X$ solved by alternating non-convex algorithm.
\item
The plots in order, first row:
$Y = a^\star * X^\star$,   $X^\star$, $ f^\star$, $a^\star$.  
\item
second row:
$X^{\mbox{cvx}}$, $f^{\mbox{cvx}}$ ,$X^{\mbox{nc}}$, $a^{\mbox{nc}}$;

\end{itemize} 
\clearpage

\begin{figure} 
\centering
\includegraphics[width=.2\textwidth]{./Fig_2D_Exp/Sphere_withINIT/Y.png}
\includegraphics[width=.2\textwidth]{./Fig_2D_Exp/Sphere_withINIT/X_cvx.png}
\includegraphics[width=.2\textwidth]{./Fig_2D_Exp/Sphere_withINIT/X_gt.png}
\includegraphics[width=.2\textwidth]{./Fig_2D_Exp/Sphere_withINIT/f_cvx.png}
\centering
\includegraphics[width=.2\textwidth]{./Fig_2D_Exp/Sphere_withINIT/X_init_prev.png}
\includegraphics[width=.2\textwidth]{./Fig_2D_Exp/Sphere_withINIT/a_init.png}
\includegraphics[width=.2\textwidth]{./Fig_2D_Exp/Sphere_withINIT/a_gt.png}

\centering
\includegraphics[width=.2\textwidth]{./Fig_2D_Exp/Sphere_withINIT/DL_len2/X_dl.png}
\includegraphics[width=.2\textwidth]{./Fig_2D_Exp/Sphere_withINIT/DL_len2/a_dl.png}
\caption{
Sphere: $Y$, solved $X$, true $X$, solved $f$;
previous $X$, previous $a$ as initialization, $a$ truth;
$X$ solved by non-convex alternating formulation, $a$ solved by non-convex alternating  formulation.
}
\label{fig-2D-Sphere-BD2}
\end{figure}


\begin{itemize}
\item $X$: $2-$D circle centered with radius $10$, on size $60 \times 60$ grid.
\item $f^\star$: $1-$D filter $(1, -s, 0)$ centered and rotated by $45$ degree.
\item $Y =f^{\star,-1}*X$, where the filter inverse is defined by discrete Fourier transform on $60 \times 60$ grid.
\item $a^{\mbox{ncvx}}$ is the forward solved by alternating non-convex algorithm.
\item $X^{\mbox{ncvx}}$ is the initial guess of $X$ solved by alternating non-convex algorithm.
\item
The plots in order, first row:
$Y = a^\star * X^\star$, $X^{\mbox{cvx}}$,  $X^\star$, $f^{\mbox{cvx}}= f^\star$.  
\item
second row:
$X^{\mbox{init}}$, $a^{\mbox{init}}$, $a^{\star}$;
\item
third row:
$X^{\mbox{nc}}$, $a^{\mbox{nc}}$, $f^{\mbox{nc}}$;
\end{itemize} 


\clearpage












\begin{figure} 
\centering
\includegraphics[width=.2\textwidth]{./Fig_2D_Exp/Square_tv/Y.png}
\includegraphics[width=.2\textwidth]{./Fig_2D_Exp/Square_tv/X_cvx.png}
\includegraphics[width=.2\textwidth]{./Fig_2D_Exp/Square_tv/X_gt.png}
\includegraphics[width=.2\textwidth]{./Fig_2D_Exp/Square_tv/f_cvx.png}
\caption{Failure mode-square, TV norm: $Y$, solved $X$, true $X$, solved $f$,  }
\label{fig-2D-Square}
\end{figure}


\begin{itemize}
\item $X$: $2-$D centered shaped square with frame size $40But $ on size $60 \times 60$ grid.
\item $f^\star$: convolution of two $1-$D filters: 
$(1, -s_1, 0)$ with $s_1=0.9995$ centered and rotated by $45$ degree and 
$(1, -s_2, 0)$ with $s_2=0.9995$ centered and rotated by $0$ degree.
\item $Y =a^{\star}*X$, where the filter inverse is defined by discrete Fourier transform on $60 \times 60$ grid.
\item $a^{\mbox{nc}}$ is the forward solved by alternating non-convex algorithm.
\item $X^{\mbox{nc}}$ is the initial guess of $X$ solved by alternating non-convex algorithm.
\item
The plots in order, first row:
$Y = a^\star * X^\star$,   $X^\star$, $ f^\star$, $a^\star$.  
\item
second row:
$X^{\mbox{cvx}}$, $f^{\mbox{cvx}}$ ,$X^{\mbox{nc}}$, $a^{\mbox{nc}}$;

\end{itemize} 
\clearpage

\begin{figure} 
\centering
\includegraphics[width=.2\textwidth]{./Fig_2D_Exp/Diamond_tv/Y.png}
\includegraphics[width=.2\textwidth]{./Fig_2D_Exp/Diamond_tv/X_cvx.png}
\includegraphics[width=.2\textwidth]{./Fig_2D_Exp/Diamond_tv/X_gt.png}
\includegraphics[width=.2\textwidth]{./Fig_2D_Exp/Diamond_tv/f_cvx.png}
\caption{Failure mode-diamond, TV norm: $Y$, solved $X$, true $X$, solved $f$,  }
\label{fig-2D-diamond}
\end{figure}

\clearpage







\begin{figure} 
\centering
\includegraphics[width=.2\textwidth]{./Fig_2D_Exp/Camera_img/Y.png}
\includegraphics[width=.2\textwidth]{./Fig_2D_Exp/Camera_img/X_gt.png}
\includegraphics[width=.2\textwidth]{./Fig_2D_Exp/Camera_img/X_TV.png}
\includegraphics[width=.2\textwidth]{./Fig_2D_Exp/Camera_img/X_TV_ma.png}
\caption{`Cameraman', using TV norm and weighted Haar L1 norm ($5$ levels with weights $2^j, j=0, \ldots 4$: $Y$, true $X$, TV norm solved $X$,Weighted Haar L1 norm solved $X$, true $f$, TV norm solved $f$,Weighted Haar L1 norm solved $f$ ,  }
\label{fig-2D-Camera}
\end{figure}
\begin{itemize}
\item $X$: camera man figure, on size $128 \times 128$ grid.
\item $f^\star$: $1-$D filter $(1, -s, 0)$ centered and rotated by $45$ degree.
\item $Y =f^{\star,-1}*X$, where the filter inverse is defined by discrete Fourier transform on $60 \times 60$ grid.
\item
$X^{\mbox{TV}}$ is the solution of convex problem with TV norm objective.
\item
$X^{\mbox{Haar}}$ is the solution of convex problem use weighted Haar L1 norm with $5$ levels as objective, where the weighted Haar L1 norm  weights  level  $j$ coefficient by with weights $2^j, j=0, \ldots 4$.
\item
The plots in order, first row:
$Y = a^\star * X^\star$, $X^\star$,$X^{\mbox{TV}}$,  $X^{\mbox{Haar}}$.  
\end{itemize} 
\clearpage














\subsection{Population Phase transition theorem}
Qingyun: Old proof without subgradient language:

\paragraph{Population phase transition theorem}
\begin{theorem}
Under the previous assumption,
$
         p^\star   =\frac{\sin{\theta}}{\sin{\theta}+ \cos{\theta}} .
$
 The solution of  $P_{1,v}$ is $w^\star$, then 
 $w^\star$ is $a^{-1}$ up to shift and scaling provided $p<p^\star$,
and  $w^\star$ is not $a^{-1}$ up to shift and scaling provided $p> p^\star$.
  \end{theorem}
The population phase transition diagram is in figure~\ref{fig-PT}.

\begin{figure} 
\centering
\includegraphics[width=.7\textwidth]{./figures_BD/Phase_T_sine.png}
\caption{The population phase transition diagram, $x-$axis is $\sin{\theta}$, $y-$axis is the sparsity level $p$ . Red part is failure, Blue part is success.}
\label{fig-PT}
\end{figure}






\paragraph{Proof of the population phase transition theorem}
\begin{itemize}
 \item
  Let $\phi  := \psi -e_0 = \tilde w*\tilde a -e_0,$  then $|[w*Y]_0| = |[(e_0 +\tilde \phi)*X]_0| =  |[(e_0 + \phi)^T X| $, we define
  \[
  L(\phi) = |[(e_0 + \phi)^T X|.
  \] 
  WLOG, we can assume the ground truth $a^{-1}$ is scaled so that $v^T \tilde{a}^{-1} =1$

  We want to find a threshold $p^\star$ so that for all $p<p^\star$, the directional derivative  $ {\mathbb E} \nabla_{\phi}{L}|_{\phi = 0}>0$ for all directions $\phi$ that satisfy $u^T \phi = 0$.

 We define the finite difference
  \[
  D = L(\phi)-L(0) =  |[(e_0 + \phi)^T X| -|X_0| 
  \]
  Let $\epsilon : = \|\phi\|_2 $ and taking limit of $D$ as $\epsilon  \rightarrow 0,$ we have
    \begin{equation}
  \begin{array}{ll}
   {\mathbb E} D &=  p{\mathbb E}[D\mid X_0 \neq 0 ] + 
   (1-p){\mathbb E}[D \mid X_0 = 0 ] \\
   &=  p{\mathbb E}[\mbox{sign}(X_0) ( \phi_0 X_0 + \sum_{ j\neq 0}\phi_{j} X_{j} )\mid X_0 \neq 0 ] + (1-p){\mathbb E}\mid \sum_{ j\neq 0}\phi_{j} X_{j}\mid   \\
   &= p \phi_0 {\mathbb E} [|X_0|\mid X_0 \neq 0 ] + (1-p){\mathbb E}\mid \sum_{ j\neq 0}\phi_{j} X_{j}\mid  
   \end{array}
 \end{equation}
The last equality is because 
 $
 {\mathbb E}[\mbox{sign}(X_0) (\sum_{ j\neq 0}\phi_{j} X_{j} )\mid X_0 \neq 0 ] = 0 
$
 as $X_j$ are sampled independently. Notice that for Bernoulli Gaussian $X$,
$
{\mathbb E} [|X_0|\mid X_0 \neq 0 ]=\sqrt{\frac{2}{\pi}}.
$
 
 Then  $ {\mathbb E} \nabla_{\phi}{L}|_{\phi = 0} = {\mathbb E} \lim_{\epsilon \rightarrow 0}\frac{D}{\epsilon}>0$ 
 in case either $\phi_{0} > 0$, or in case $\phi_0 < 0$ and
\[
          \frac{p}{1-p} < \frac{ {\mathbb E}\mid \sum_{ j\neq 0}\phi_{j} X_{j}\mid  }{|\phi_0| 
          {\mathbb E} [|X_0|\mid X_0 \neq 0 ]} = \frac{\|\phi_{(0)}\|_2}{| \phi_{0}|} \frac{ {\mathbb E} | \phi_{(0)}^T X_{(0)} |}{\|\phi_{(0)}\|_2 \sqrt{\frac{2}{\pi}}}  .
\]
Here  $X_{(0)} = (X_i)_{i\neq 0}$
\item (Geometric lower bound:)
Using the fact that $u^T \phi = 0$. We define $v = \cos{(\theta)}\frac{\tilde{a}}{\|\tilde{a}\|_2} + \sin{(\theta)} (\frac{\tilde{a}}{\|\tilde{a}\|_2} )^{\bot}$, then $u =\tilde{a}^{-1}*v = \tilde{a}^{-1}*(\cos{(\theta)}\frac{\tilde{a}}{\|\tilde{a}\|_2} + \sin{(\theta)} (\frac{\tilde{a}}{\|\tilde{a}\|_2})^{\bot}) =\cos{(\theta)} \frac{e_0}{\|\tilde{a}\|_2}+\sin{(\theta)} \tilde{a}^{-1}*(\frac{\tilde{a}}{\|\tilde{a}\|_2} )^{\bot} $, 
where $\tilde{a}^{-1}*(\frac{\tilde{a}}{\|\tilde{a}\|_2} )^{\bot}$ is perpendicular to $e_0$, 
therefore, 
$0= u^T \phi = \frac{1}{\|\tilde{a}\|_2}[\cos{(\theta)} \phi_0+\sin{(\theta)} \phi_{(0)}]$, therefore,  
$\frac{\|\phi_{(0)}\|_2}{| \phi_{0}|} = \frac{\cos(\theta)}{\sin(\theta)} =\cot(\theta).$
\[
 \frac{\|\phi_{(0)}\|_2}{| \phi_{0}|}\geq  \cot(\theta)
 \]

 
\item
We showed that for IID Bernoulli Gaussian $X$,
\[
\frac{ {\mathbb E} | \phi_{(0)}^T X_{(0)} |}{\sqrt{\frac{2}{\pi}} \|\phi_{(0)}\|_2} = \frac{ {\mathbb E}_{I_{(0)}}\| \phi_{I_{(0)}}\|_2}{ \|\phi_{(0)}\|_2} \in [p,1)
\]
Using semicontinuous extension, we can choose sequence of $\phi$ that is dense on doubly infinite sequence space $V_{-\infty, \infty}$, then we know that $\frac{ {\mathbb E}_{I_{(0)}}\| \phi_{I_{(0)}}\|_2}{ \|\phi_{(0)}\|_2}$ should be replaced by its semicontinuous extension on $V_{-\infty, \infty}$, which is $1$.

The phase transition $p^\star$ is the smallest $p$ so that 
\[
          \frac{p}{1-p} \leq  \cot(\theta) \frac{ {\mathbb E}_{I_{(0)}}\| \phi_{I_{(0)}}\|_2}{ \|\phi_{(0)}\|_2} \leq  \cot(\theta).
\]
Therefore,  generically, $ \frac{p^\star}{1-p^\star}  = \cot(\theta) $.
\EIT
\section{Projection pursuit for planted sparsity}



 
 
 
 
 









 


\section{Phase transition for sub-Gaussian}
From the Khinthine’s inequality for sub-Gaussian and $p = 1$, we have 
\[
E{|h_{(0)}^T X_{(0)}| }\geq  3^{-3/2}C^{-3} K^{-3}  \|h_{(0)}\|_2,
\]
for Bernoulli-SubGaussian, we have,
\[
E{|h_{(0)}^T X_{(0)}| }\geq  (1-(1-p)^M)^{1/2} 3^{-3/2}C^{-3} K^{-3}  \|h_{(0)}\|_2.
\]
In general, we define
\[
\gamma(F,p) = E{|h_{(0)}^T X_{(0)}| }/\|h_{(0)}\|_2 E |X_0|,
\]
as the ratio between the linear combination $E{|h_{(0)}^T X_{(0)}| }$ for $\|h_{(0)}\|_2=1$ and $ E |X_0|$.

Then we need
\[
          \frac{1-p}{p} < \frac{ \gamma(F,p)  \|h_{(0)}\|_2}{| h_{0}|}
\]
for all $h$ such that $   u^T h = 0$ .

Also, using the fact that
 
\[
         \frac{  \|h_{(0)}\|_2} {| h_{0}|} \geq \cot(\theta).
\]
we have 
\[
          \frac{1-p}{p} <\gamma(F,p) \cot(\theta).
\]

\section{Noisy  signal}
 \subsection{Folder Gaussian}
 For a Gaussian RV $Z \sim N(\mu, \sigma)$, then for RV $|Z|$,  
 \[
 \mu_{|Z|} = E|Z|=\sqrt{\frac{2}{\pi}}\sigma e^{-\frac{\mu^2}{2\sigma^2}}+\mu\left[1-2\Phi\left(-\frac{\mu}{\sigma}\right) \right] 
 \]
  \[
 \sigma_{|Z|}^2 = \mu^2 + \sigma^2 - \mu_{|Z|}^2.,
 \]
 
 \subsection{Noisy perturbation analysis}
Consider a noisy perturbation $\bar X_i  =  X_i + \sigma  E_i $, where $E_i$ is standard Gaussian $N(0,1)$, $ X_i = B_i Z_i$ are iid product of Bernolli RV with parameter $p$ and subGaussian unit variance RV with c.d.f $F$. 

Then we let $b^{\sigma}$ be the solution of the optimization problem 
\[
E D_\sigma = E_X E_E( |(1+ {\varepsilon} h_0)(X_0+\sigma  E_0) +  {\varepsilon} h_{(0)}^T (X_{(0)}+\sigma  E_{(0)})| -|X_0+\sigma  E_0|)
\]
for sufficiently small ${\varepsilon}$, and fixed $\sigma<1$, we consider the asymptotic analysis when ${\varepsilon}  << 1 $， $ \sigma<< 1 $. We know $\sigma ((1+ {\varepsilon} h_0)(  E_0)+{\varepsilon} h_{(0)}^T  E_{(0)}))$ is a centered Gaussian with standard deviation 
\[\sigma\sqrt{(1+ {\varepsilon} h_0)^2+({\varepsilon} h_{(0)})^2}\approx \sigma(1+ {\varepsilon} h_0)
\]
therefore,
\begin{eqnarray*}
 E D_\sigma &\approx&   E|(1+ {\varepsilon} h_0)X_0 +  {\varepsilon} h_{(0)}^T X_{(0)}+\sigma (1+ {\varepsilon} h_0) E_0| -E|X_0+\sigma  E_0| \\
 &\approx& 
  p \cdot(|1+{\varepsilon} \cdot  h_{0}|-1) \cdot E |Z_0+\sigma  E_0| + (1-p) \cdot ( E{|{\varepsilon}  \cdot h_{(0)}^T X_{(0)}+\sigma  E_0| -E|\sigma  E_0|} ),
\end{eqnarray*}
 where $p = P(X_0\neq 0)$.
 
 First we derive lower bounds,
 \[
 E |Z_0+\sigma  E_0|
 \geq E |Z_0| - E|\sigma  E_0| =   E |Z_0| - \sqrt{\frac{2}{\pi}}\sigma,
\]
 \begin{eqnarray*}
 E{|{\varepsilon}  \cdot h_{(0)}^T X_{(0)}+\sigma  E_0| -E|\sigma  E_0|}&\geq& E{|{\varepsilon}  \cdot h_{(0)}^T X_{(0)} | -2 E|\sigma  E_0|} \\
 &=&  E|{\varepsilon}  \cdot h_{(0)}^T X_{(0)} | -2 \sqrt{\frac{2}{\pi}}\sigma
\end{eqnarray*}
Then we have 
\begin{eqnarray*}
 E D_\sigma 
\geq  E D_0 - (p+2(1-p)) \cdot \sqrt{\frac{2}{\pi}}\sigma.
\end{eqnarray*}

Therefore, in the existence of noise, we don't have exact recovery result, but we would have a lower bound. 
 
 Second, we use the mean of folded Gaussian to compute the condition expectation.
 
 Here, for a fixed $\mu$, we rescale the variance $\sigma' = \sigma/{\varepsilon}$. Under the assumption ${\varepsilon}/\sigma<< 1 $, using asymptotic expansion, we can compute the condition expectation 
 \begin{eqnarray*}
   E({|\mu+\sigma'  E_0| -|\sigma'  E_0|}\mid \mu)
 &=& \sqrt{\frac{2}{\pi}}\sigma' e^{-\frac{\mu^2}{2\sigma'^2}}+\mu\left[1-2\Phi\left(-\frac{\mu}{\sigma'}\right) \right] - \sqrt{\frac{2}{\pi}}\sigma'
 \\
 &\approx& -\sqrt{\frac{2}{\pi}}\sigma' {\frac{\mu^2}{2\sigma'^2}}+\mu\sqrt{\frac{2}{\pi}}(-\frac{\mu}{\sigma'}  )
 \\
  &=&- \sqrt{\frac{2}{\pi}}  {\frac{3\mu^2}{2\sigma'}}  
\end{eqnarray*}
The first equality is the formula for the mean of folded Gaussian.
The second equality uses the Taylor expansion of Gaussian c.d.f
\[
\Phi (z)  = \frac{1}{2}+  \frac{1}{\sqrt{\pi}} \left(z-\frac{z^3}{3}+\frac{z^5}{10}-\frac{z^7}{42}+\frac{z^9}{216}-\cdots\right)
\]

 Therefore, for $Z_0$ standard Gaussian,
  \begin{eqnarray*}
 E_X E_{E_0}(|Z_0+\sigma  E_0|) 
  &=&  \sqrt{\frac{2}{\pi}}   \sqrt{1+\sigma^2}  
\end{eqnarray*}
 \begin{eqnarray*}
 E_X E_{E_0}({|{\varepsilon}  \cdot h_{(0)}^T X_{(0)} +\sigma  E_0| -|\sigma  E_0|}) 
  &=&-{\varepsilon} \cdot 3\sqrt{\frac{2}{\pi}}  {\frac{E_X( h_{(0)}^T X_{(0)})^2 }{2\sigma'}}  
   \\
  &=&-{\varepsilon} \cdot 3\sqrt{\frac{2}{\pi}}  {\frac{\|h_{(0)}\|^2 }{2\sigma'}}  
\end{eqnarray*}

Then 
\begin{eqnarray*}
 E D_\sigma 
 &\approx& 
 {\varepsilon} \left[ p \cdot h_{0} \cdot \sqrt{\frac{2}{\pi}}   \sqrt{1+\sigma^2}    - (1-p) \cdot 3\sqrt{\frac{2}{\pi}}  {\frac{\|h_{(0)}\|^2 }{2(\sigma/{\varepsilon})}} \right] .
\end{eqnarray*}



 
\section{Population loss and phase transition}
We first study population loss of the sparse deconvolution problem
 \begin{eqnarray*}
    (\mbox{Population sparse deconvolution}) \qquad  \min_{w\in {\mathbb R}^k} &    
    E\| w * Y\|_1\\
    \mbox{ subject to }&  v^T w = 1,\\
    &  w\in {\cal M}.
\end{eqnarray*}
\subsection{Population loss of projection pursuit}
Let's first consider the population objective $E|w^T \Phi X|$.

Consider a vector  $w \in {\cal M}$ which is a small perturbation of $w^\star  \in {\cal M}$, where
$w^T \Phi =e_0$. Let $w = w^\star + {\varepsilon} \delta$ where $\delta \in T_{w^\star} {\cal M}$, the tangent space at $w^\star$. And since $\delta \in T_{w^\star} {\cal M}_1$ is a subspace, we assume the constraint is given by $M \delta = 0$.
Let $h = \Phi^T \delta$, then $   u^T \delta = 0$, where $ v = \Phi u$. We have
\[
    b^T Y = X_0 + {\varepsilon} \delta^T \Phi X=X_0 + {\varepsilon} h^T  X = (1+ {\varepsilon} h_0)X_0 +  + {\varepsilon} h_{(0)}^T X_{(0)} .
\]
 where $h_{(0)}$, $X_{(0)}$ are $h$, $X$ without $0$-th coordinate.
 \subsection{Population phase transition}
 We want to study $E D =  E|(1+ {\varepsilon} h_0)X_0 +  + {\varepsilon} h_{(0)}^T X_{(0)}| -E|X_0|$.
 
 Then for sufficiently small ${\varepsilon}$
\begin{eqnarray*}
 E|(1+ {\varepsilon} h_0)X_0 +  {\varepsilon} h_{(0)}^T X_{(0)}| -E|X_0| &\approx& 
 (|1+{\varepsilon} \cdot  h_{0}|-1) \cdot E |X_0| \cdot (1-\eta)+ {\varepsilon} \cdot  E{|h_{(0)}^T X_{(0)}| } \cdot \eta,
\end{eqnarray*}
 where $p = P(X_0=0)$.
 
 The perturbation increases the objective value 
just in case either $h_{0} > 0$, or in case $h_{0} < 0$ and
\[
          \frac{\eta}{1-\eta} > \frac{| h_{0}| E |X_0|}{ E | h_{(0)}^T X_{(0)} |. }
\]

Now we consider the optimization problem
\begin{eqnarray*}
    (Q_1) \qquad  \min_h&   E | h_{(0)}^T X_{(0)} |\\
    \mbox{ subject to }&  h_{0} = -1, \\
    & h\in \Phi^T T_\beta {\cal M}_1.
\end{eqnarray*}


We know $h = \Phi^T \delta$, then 
\begin{eqnarray*}
    (Q_1) \qquad  \min_\delta &E | \delta^T \Phi_{(0)} X_{(0)} | \\
    \mbox{ subject to }  &\Phi_0^T \delta = -1, \\
     & \delta \in T_\beta {\cal M}_1.
\end{eqnarray*}



Following the quadratic problem reduction, we can related $Q_1$ to the deterministic quadratic problem.

After the study of the general population model, we remark that for convolutional model, $A$ ad $B$ are both generated by the circular or Topelitz matrices from a single vector.





\subsection{Regularized convex problems}
\paragraph{Regularized convex problems}
For any $\epsilon>0$, we can find the Lagrangian form  of the constrained problem to define the regularized convex program, where we can find a corresponding $\lambda = \Lambda(\epsilon)$ so that $\frac{1}{2\lambda}$ is the dual variable.
\[
 \begin{array}{ll}
 \mbox{minimize}_{w,\zeta}   & |\tilde{w}^T Y -\zeta |+ \frac{1}{2\lambda}((\zeta)^2-\epsilon^2) \\
\mbox{subject to}  
& v^T w =1,
\end{array}
\tag{$P_{1,\lambda}$}
  \]
The convention to use for optimization is to ignorne the constant $-\epsilon^2$, and write the problem as
\[
 \begin{array}{ll}
 \mbox{minimize}_{w,\alpha}   & [|\alpha|+ \frac{1}{2\lambda}(\alpha -\tilde{w}^T Y)^2] \\
\mbox{subject to}  
& v^T w =1,
\end{array}
\tag{$P_{1,\lambda}$}
  \]
  Define soft-threshold operator as 
  \[
  S_{\mu}(x): = {\rm sign} (x) (|x| -\mu)_+
  \]
  and define its complement
   \[
  \bar{S}_{\mu}(x): = x-S_{\mu}(x) =  {\rm sign} (x)(x- (|x| -\mu)_+) =  {\rm sign} (x)\max(|x| ,\mu).
  \]
 \begin{lemma}
 If we partially minimize over $\alpha,$ the minimizer is $\alpha = S_{\lambda}(\tilde{w}^T Y)$, 
    \[
 \begin{array}{ll}
\mbox{inf}_{\alpha}   & [|\alpha|+ \frac{1}{2\lambda}(\alpha -\tilde{w}^T Y)^2]   = [(|\tilde{w}^T Y|-\lambda)_+ + \frac{1}{2\lambda}(\max(|\tilde{w}^T Y| ,\lambda) )^2] 

\end{array}
  \]
  \end{lemma} 
  
  
 
\begin{proof}
Due to the proximal operator of $\ell_1$ norm,  the minimizer is $\alpha = S_{\lambda}(\tilde{w}^T Y)$, 
The objective can be further simplified to
  \[
 \begin{array}{ll}
&{\mathbb E}[|\alpha|+ \frac{1}{2\lambda}(\alpha -\tilde{w}^T Y)^2]\\
=&{\mathbb E}[|S_{\lambda}(\tilde{w}^T Y)|+ \frac{1}{2\lambda}(\bar{S}_{\lambda}(\tilde{w}^T Y) )^2]\\
=& [(|\tilde{w}^T Y|-\lambda)_+ + \frac{1}{2\lambda}(\max(|\tilde{w}^T Y| ,\lambda) )^2] 
\end{array}
  \]
 \end{proof} 

Using the data generative model and do change of variable, 
the partially minimized regularized problem is
 \[
 \begin{array}{ll}
 \mbox{minimize}_{\psi}   & {\mathbb E}[|(|\psi^T (X+\eta Z )|-\lambda)_+|+ \frac{1}{2\lambda}(\max(|\psi^T (X+\eta Z )| ,\lambda) )^2] \\
\mbox{subject to}  
& u^T \psi =1,
\end{array}
\tag{$P_{1,\lambda}$}
  \]

\section{Related Work}
Qingyun: Will turn these slides into proper writting latter, just trying to collect all the related reference for now.
\paragraph{Prior work: analysis approach, non-Gaussianity}
  \begin{itemize}
  \item
  Key idea $1$: Find inverse filter $a^{-1}$ so that
$ a^{-1}*y = x $.
\item Key idea $2$: 
If $x$ is non-Gaussian, then $a*x$ is closer to Gaussian. 
\item Find $w\neq 0$  
\[
 \begin{array}{ll}
 \underset{w}{\mbox{maximize}}   & J(w*y)
\end{array}
  \]
\item Non-Gaussianity: $J$ is negative entropy function or normalized $4$-th order cumulant $J(z) = \frac{\|z\|_4^4}{\|z\|_2^4}$ \cite{wiggins1978minimum, donoho1981minimum}
  \item Sparsity: 
  $J(z) = - \| z\|_1/\|z\|_2$,  \cite{gray1979variable,donoho1981minimum,kijko1994seismological}.
  \item varimax norm: $\max_i \frac{|z_i|}{\|z\|_2}$ \cite{cabrelli1985minimum}
\end{itemize}

\paragraph{Prior work: blind equalization for digital signal}
  \begin{itemize}
\item
Digital signal: PAM signaling $\pm 1$ binary signal or QAM signaling with $|x|=1, x\in {\mathbb C}$ \cite{kennedy1992blind,ding2000fast}. 
\item More general,  $x$ are IID and bounded by $M = \mbox{max}(x_i)$ \cite{vembu1994convex,li2001blind}.
\item 
  Convex Cost Functions in Blind Equalization \cite{vembu1994convex}:
  find $w$ so that $w*a = e_0 = (\ldots,0,1,0, \ldots)$ as minimizer of the ideal convex problem (
 Not computable without knowing $a$!)
  \[
 \begin{array}{ll}
 \underset{w}{\mbox{minimize}}   & \| w*a\|_1 
 \\
\mbox{subject to} 
& (w*a)_0 = 1.
\end{array}
  \]  
  
    \item
  When $x$ are IID and bounded by $M = \mbox{max}(x_i)$, using duality, use surrogate objective for $\| w*a\|_1$ as
   \[
 \begin{array}{ll}
 \underset{w}{\mbox{minimize}}   &  \frac{1}{M} E \|w * y\|_\infty 
 \\
\mbox{subject to} 
& \|w*y\|_2 = 1.
\end{array}
  \]
   In \cite{vembu1994convex}, they use projected SGD on $E \|w * y\|_8 $ to approximate $E \|w * y\|_\infty$ .
  
  \item Later, use linear programming to solve $\infty$ norm problem directly \cite{ding2000fast}: 
    \[
 \begin{array}{ll}
 \underset{w}{\mbox{minimize}}   &  E \|w * y\|_\infty
 \\
\mbox{subject to} 
& w_0 = 1.
\end{array}
  \]
\end{itemize}

\paragraph{Prior work: synthesis approach, sparsity}
\begin{itemize}
\item
  Convolutional dictionary learning formulation \cite{bristow2013fast, heide2015fast}, etc.
  \item
  Synthesis blind deconvolution with $2$ norm ball constraint  \cite{zhang2017global,zhang2018structured, kuo2019geometry}, etc.
  \item
  Short $a$ and sparse $x$: 
  \[
 \begin{array}{ll}
 \mbox{minimize}   & \|y-a*x\|_2^2+\lambda \|x\|_1 \\
\mbox{subject to}  & \|a\|_2 = 1.
\end{array}
  \]  
\end{itemize}

\paragraph{Prior work: synthesis approach, subspace assumption}
\[
 \begin{array}{ll}
 y &= a^0 * x^0\\
 \hat y  &= \hat a^0 \circ \hat x^0
\end{array}
  \]
   \begin{itemize}
  \item  $\hat a^0= D_a c_a$.
  \item $\hat x^0 = D_X c_x$.
  \item $D_X$ is often a complex Gaussian matrix.  
\end{itemize}
No translation symmetry in this assumption!


Lifting:
 \[
 \begin{array}{ll}
 \hat y &= (D_a c_a) \circ (D_X c_x)\\
 &= \mathcal{D}(c_a c_x^T)
\end{array}
  \]
 $\mathcal{D}$ is a linear operator defined as 
 \[
 \begin{array}{ll}
\mathcal{D}(c_a c_x^T)_i = \mbox{trace}((D_{a,i}D_{X,i}^T)(c_a c_x^T))
\end{array}
  \]
  Rank one matrix $Z=c_a c_x^T$.
   \[
      \hat y =  \mathcal{D}(Z)
  \]
  \begin{itemize}  
  \item Convex optimization  \cite{ahmed2014blind},\cite{ling2015self}, \cite{chi2016guaranteed},\cite{romberg2013multichannel},\cite{lee2017blind} etc.
  \item Non-convex optimization  \cite{li2016rapid}.
\end{itemize}

\paragraph{Projection pursuit  with linear constraint}
We point out that the population problem formulation for sparse blind deconvolution is also the population problem formulation for sparse projection pursuit, which might also be called convex sparse ICA, or convex sparse dictionary learning.
\begin{equation}
 \begin{array}{ll}
  \underset{\psi \in V}{ \mbox{minimize} }  &  {\mathbb E} |\psi^T X|\\
 \mbox{subject to}  
&  u^T \psi =1.
\end{array}
\tag{$P_{1,v}$}
\end{equation}


In sparse projection pursuit with linear constraint, we observe a matrix $Y_M\in {\mathbb R}^{n\times M }$, and we know that there exists a ground-truth filter $w^0$, such that $w^{0,T} Y_M = X_M$, $X_M$ satisfy Bernoulli$(p)$-Gaussian model.  

We consider the "projection pursuit" problem.
 \begin{eqnarray*}
    (\mbox{Projection pursuit}) \qquad  \min&    \frac{1}{M} \| \tilde{w}^T Y_M\|_1\\
    \mbox{ subject to }&  v^T w = 1.
\end{eqnarray*}
Here $w$ is in a $k$ dimensional subspace of the double infinite sequence. 


\paragraph{Projection pursuit with sphere constraint}
We comment that the results we derive in the following sections are also applicable to the projection pursuit (orthogonal dictionary learning) problem \cite{bai2018subgradient, sun2016complete, sun2015complete}.
 \begin{eqnarray*}
    (\mbox{Projection pursuit with sphere constraint}) \qquad  \min&    \frac{1}{M} \| \tilde{w}^T Y_M\|_1\\
    \mbox{ subject to }&   \frac{1}{M} \| \tilde{w}^T Y_M\|_2 = 1.
\end{eqnarray*}
or
 \begin{eqnarray*}
    (\mbox{Projection pursuit with sphere constraint}) \qquad  \min&    \frac{1}{M} \| \tilde{w}^T Y_M\|_1\\
    \mbox{ subject to }&   \frac{1}{M} \| \tilde{w}^T Y_M\|_2 \geq 1.
\end{eqnarray*}
\begin{lemma}
The population problem is:
\[
 \begin{array}{ll}
 \mbox{minimize}_{\psi} \quad & {\mathbb E} |\psi^T X|  \\
\mbox{subject to}  
&{\mathbb E} |\psi^T X|^2 \geq {\mathbb E} (X_0)^2
\end{array}
  \]
is equivalent to the following convex-concave problem
\[
 \begin{array}{ll}
 \mbox{minimize}_{\psi} \quad &\sqrt{\frac{2}{\pi}}\left[{\mathbb E}_I \|\psi_I\|_2 \right] \\
\mbox{subject to}  
&\|\psi\|_2^2 \geq 1
\end{array}
  \]
We claim that  it has the same minimizer as the \textbf{ projection pursuit (orthogonal dictionary learning) problem with sphere constraint}.
  \[
 \begin{array}{ll}
 \mbox{minimize}_{\psi} \quad & {\mathbb E}_I \|\psi_I\|_2  \\
\mbox{subject to}  
&\|\psi\|_2 = 1
\end{array}
  \]
  which is also equivalent to
   \[
 \begin{array}{ll}
 \mbox{minimize}_{\psi\neq 0} \quad & {\mathbb E}_I \|\psi_I\|_2/\|\psi\|_2 
\end{array}
  \]
  And  also equivalent to
    \[
 \begin{array}{ll}
 \mbox{minimize}_{\psi\neq 0} \quad & {\mathbb E} |\psi^T X|/\sqrt{{\mathbb E} |\psi^T X|^2} 
\end{array}
  \]
\end{lemma}


Inspired by this formulation, we propose a non-convex (convex objective concave constraint) problem for blind deconvultion:
\begin{eqnarray*}
    (\mbox{Projection pursuit with sphere constraint}) \qquad  \min&    \frac{1}{M} \| w*Y_M\|_1\\
    \mbox{ subject to }&   \frac{1}{M} \| w*Y_M\|_2 \geq 1.
\end{eqnarray*}

Another choice is to use infinity norm ball constraint, which allows us to translate it to a convex problem.
\begin{eqnarray*}
    (\mbox{Projection pursuit with sphere constraint}) \qquad  \min&    \frac{1}{M} \| w*Y_M\|_1\\
    \mbox{ subject to }&   \frac{1}{M} \| w*Y_M\|_\infty \geq 1.
\end{eqnarray*}


\paragraph{Infinity norm ball constraint}


Before stating the main theorem and its proof, we make some comments about the formulation.

If we study the population level, 
\begin{equation}
 \begin{array}{ll}
  \underset{w\in V}{ \mbox{minimize} }  &    {\mathbb E} |\tilde{w}^T Y|\\
    \mbox{ subject to }
    & \|w\|_{\infty}= 1.
\end{array}
\tag{$P_{\infty}$ }
\end{equation}
then we only need solve one convex problems
\begin{equation}
 \begin{array}{ll}
  \underset{w\in V}{ \mbox{minimize} }  &  {\mathbb E} |\tilde{w}^T Y|\\
 \mbox{subject to}  
&  w_0 =1.
\end{array}
\tag{$P_{e_0}$ }
\end{equation}
Since the entry with largest magnitude will automatically be shifted to coordinate $0$ due to shift invariance.


If we want to solve the following non-convex problem with infinity norm ball constraint for finite sample,
\begin{equation}
 \begin{array}{ll}
  \underset{w\in V}{ \mbox{minimize} }  &    \frac{1}{2T+1}\|w*Y\|_{1,[-T,T]}\\
    \mbox{ subject to }
    & \|w\|_{\infty}= 1.
\end{array}
\tag{$P_{\infty}^T$}
\end{equation}
We can solve the following convex problems, and select the one with minimal objective value. This is equivalent to solve the problem with fixed infinity norm. 
\begin{equation}
 \begin{array}{ll}
  \underset{w\in V}{ \mbox{minimize} }  &  \frac{1}{2T+1}\|w*Y\|_{1,[-T,T]}\\
 \mbox{subject to}  
&  w_j =1.
\end{array}
\tag{$P_{e_j}^T$ }
\end{equation}

\paragraph{Dirac re-parametrization and connection to residual CNN}
When $v = e_0$, we are imposing a constraint that the largest entry of $w$ is $w_0 =1$.  
We can re-parametrize our filter, as $w = \delta_0 + f$, $f_0 = 0$. Then our problem became 
 \begin{eqnarray*}
    \min_{f, f_0 = 0} &    
    \frac{1}{2T+1}\|(e_0 +f)*Y\|_{1,[-T,T]}.
\end{eqnarray*}
Here $f = (f_L, 0, f_R)$, $f_R$ and $f_L$ are in $k$ dimensional subspace of the double infinite sequence with positive and negative support.

Allowing Noise:
\[
 \begin{array}{ll}
 \mbox{minimize}   & \|x\|_ 1+\lambda/2\|Y+f*Y-x\|_2^2\\
\mbox{subject to}  
& f_0 =0,
\end{array}
  \]
  with variables $f\in {\mathbb R}^k$, $x\in {\mathbb R}^n$. 
  
 \texbf{General multi-channel approximate version}
\[
 \begin{array}{ll}
 \mbox{minimize}   & \sum_i(\|x^i\|_ 1+\lambda^i/2\|Y+f^i*Y-x^i\|_2^2)\\
\mbox{subject to}  
& f^i_0 =0,
\end{array}
  \]
  with variables $f^i\in {\mathbb R}^k$, $x^i\in {\mathbb R}^n$. 
  
\paragraph{Link to CNN}
Partial minimization:(similar to one-layer Residual CNN)
\[
 \begin{array}{ll}
 x^i = S_{1/\lambda^i}(Y+f^i*Y)
\end{array}
  \]
  with $S_{b}(z) :=\mbox{sign}(z)(|z|-b)_+ $. 


\texbf{Method $1$: plug-in}
\[
 \begin{array}{ll}
 \mbox{minimize}   & \sum_i(\|S_{1/\lambda^i}(Y+f^i*Y)\|_ 1+\\
 &\lambda^i/2\|Y+f^i*Y-S_{1/\lambda^i}(Y+f^i*Y)\|_2^2)\\
\mbox{subject to}  
& f^i_0 =0,
\end{array}
  \]
  with variables $f^i\in {\mathbb R}^k$, $x^i\in {\mathbb R}^n$.   
  

\section{ Techinical Overview and Proof Sketch for Phase Transition Theorems }

	
















\subsection{Main Result $1$: Population (Large-$N$) Phase Transition }


\paragraph{Sketch of proof ideas for Theorem~\ref{thm:Population phase transition}}

Here we highlight some of the key ideas in the proof:
\begin{itemize}
\item
\textbf{Change of variable.}
Rewrite the population version of our convex sparse blind deconvolution problem, with the population objective 
${\mathbb E}\frac{1}{N} \| {\bf w}*Y\|_{\ell_1({\cal T})} ={\mathbb E}|({\bf w}*Y)_0|={\mathbb E}|({\bf w}*{\bf a}*X)_0|={\mathbb E}_X|\langle X, ({\bf w}*{\bf a})^{\dagger} \rangle|$ due to  the ergodic property of stationary process and shift invariance, and $(\widetilde{{\bf a}}*{\bf w})_0=((\widetilde{{\bf a}}*{\bf a}^{-1})*({\bf a}*{\bf w}))_0  = \langle (\widetilde{{\bf a}}*{\bf a}^{-1})^{\dagger}, {\bf w}*{\bf a}\rangle$, the convex problem becomes 
\[
 \begin{array}{ll}
 \underset{{\bf w}}{\mbox{minimize}}   & {\mathbb E}_X|\langle X ,  ({\bf a}*{\bf w})^{\dagger} \rangle|\\
\mbox{subject to}  
& \langle \widetilde{{\bf a}}*{\bf a}^{-1} , ({\bf a}*{\bf w})^{\dagger}\rangle =1,
\end{array}
 \]
 \newcommand{{\bf v}}{{\bf v}}
Let ${\bf v}$ denote  the time reversed version of ${\bf a}*{\bf w}$: ${\bf v} := ({\bf a}*{\bf w})^{\dagger}$, and
  let $\widetilde{{\bf e}}:= \widetilde{{\bf a}}*{\bf a}^{-1}$, then by previous assumptions, 
  $\widetilde{{\bf e}}(0)=1$, $\widetilde{{\bf e}}' = \widetilde{{\bf e}}-{\bf e}_0$.
  
 Now we arrive at a simple and fundamental population convex problem: 
 \[
 \begin{array}{ll}
 \underset{{\bf v}}{\mbox{minimize}}   &{\mathbb E}_X|\langle X ,  {\bf v}  \rangle|\\
\mbox{subject to}  
& \langle \widetilde{{\bf e}} ,  {\bf v}  \rangle  =1.
\end{array}
  \]
  \item
\textbf{Expectation using Gaussian.  }
Since $X$ follows Bernoulli-Gaussian IID probability model $X_t = I_t G_t$, we nest the expectation over $I_t$ outside the expectation over Gaussian $G_t$, for which we use $E |N(0,1)| = \sqrt{\frac{2}{\pi}}$:
\[
{\mathbb E}_X|\langle X ,  {\bf v}  \rangle|
= {\mathbb E}_I {\mathbb E}_G {|\sum_{t\in {\mathbb Z}}  I_t G_t {\bf v}(t)| }
= \sqrt{\frac{2}{\pi}} \cdot{\mathbb E}_I \|{\bf v}\cdot I\|_2
\]
\item
\textbf{KKT condition for ${\bf e}_0$.}
Let ${\bf v}^\star$ denote the solution of the optimization problem:
\[
 \begin{array}{ll}
 \underset{\psi}{\mbox{minimize}}   & \sqrt{\frac{2}{\pi}} \cdot{\mathbb E}_I \|{\bf v}\cdot I\|_2\\
\mbox{subject to}  
& \langle   \widetilde{{\bf e}}, {\bf v}\rangle =1
\end{array}
\tag{$Q_1(\widetilde{{\bf e}})$ }
  \]
  
To prove that ${\bf v}^\star={\bf e}_0$, i.e. ${\bf e}_0$ solves ($Q_1(\widetilde{{\bf e}})$), we calculate the directional finite difference at ${\bf e}_0$. Then ${\bf e}_0$ solves this convex problem if the directional finite difference at ${\bf v} = {\bf e}_0$ is non-negative at every direction $\beta$ on unit sphere where $\widetilde{{\bf e}}^T \beta =0$:
	 \[
	 {\mathbb E}\|({\bf e}_0+\beta)\cdot I\|_2 - {\mathbb E}\|({\bf e}_0)\cdot I\|_2 \geq 0.
	 \]


\item
\textbf{Conditional expectation at one sparse element $X_0$.} 
We decompose the objective into a sum of terms, 
conditioning on whether $I_0 = 1_{\{X_0\neq 0\}}$ is zero or not:
	 \[
	 \begin{array}{ll}
	    {\mathbb E}\|({\bf e}_0+\beta)\cdot I\|_2 - {\mathbb E}\|({\bf e}(0))\cdot I\|_2
	    &= p (1+\beta_0)  + (1-p) \nabla_{\beta}{{\mathbb E}_I[\|({\bf e}_0+\beta)'\|_{\ell_2(I-\{0\})} \mid I_0 =0]} -p\\
	    &= p \beta_0  + (1-p) {\mathbb E}_{I'}[\|\beta'\|_{\ell_2(I')} ].
	 \end{array}
	\]
This will be non-negative in case either $\beta(0) > 0$, or else $\beta(0) < 0$ but
	\[
	\frac{p}{1-p} \leq  \frac{{\mathbb E}_{I'}\|\beta'\|_{\ell_2(I')}}{ |\beta_0|}
	\]
	for all $\beta $ that satisfy $ \widetilde{{\bf e}}^T \beta =0$.
\item
\textbf{Reduction to $val(Q_1 (\widetilde{{\bf e}}')).$}	
	We normalize the direction sequence $\beta$ so that $\beta(0)= -1$; 
	using $\tilde{e}(0)=1$, we obtain a lowerbound:
\[
 \inf_{\beta(0)=-1, \langle  \widetilde{{\bf e}}, \beta\rangle  =0} {\mathbb E}_{I'}\|\beta'\|_{\ell_2(I')}
 =\inf_{ \beta(0)=-1, \beta(0) \tilde{e}(0)-\langle  \widetilde{{\bf e}}', \beta'\rangle  =0} {\mathbb E}_{I'}\|\beta'\|_{\ell_2(I')}
= \inf_{\langle   \widetilde{{\bf e}}', \beta'\rangle  =1} {\mathbb E}_{I'} \|\beta'\|_{\ell_2(I')} = val(Q_1 (\widetilde{{\bf e}}'))
\]	
Here $Q_1(\widetilde{{\bf e}}')$ is the optimization problem:
\[
 \begin{array}{ll}
 \underset{\beta \in l_1({\mathbb Z})}{\mbox{minimize}}   &{\mathbb E}_{I'} \|\beta'\|_{\ell_2(I')}\\
\mbox{subject to}  
& \langle  \widetilde{{\bf e}}', \beta'\rangle  =1
\end{array}
  \]

\item 
\textbf{The explicit phase transition condition with upper and lower bound.}
We have shown the existence of  $p^\star$ so that for all $p<p^\star$, the KKT condition is satisfied. And we have represented $p^\star$ as the optimal value of a derived optimization problem $val(Q_1 (\widetilde{{\bf e}}'))$. The following lemma finds simple upper and lower bounds for $p^\star=val(Q_1 (\widetilde{{\bf e}}'))$. 
	\begin{lemma}[Explicit phase transition condition with upper and lower bound]
	\label{lemma:Condition phase transition}
The threshold $p^\star$ determined by
 \[
\frac{p}{1-p}= val(Q_1(\widetilde{{\bf e}}')).
\]
obeys an upper bound and lower
\[
 \frac{p}{\|\widetilde{{\bf e}}'\|_\infty}  \geq val(Q_1(\widetilde{{\bf e}}')) \geq  {\mathbb E}_I\|\widetilde{{\bf e}}'\cdot I\|_{2}^{-1}
\]
where $\|\widetilde{{\bf e}}'\|_\infty  = \frac{|\widetilde{{\bf e}}|_{(2)}}{|\widetilde{{\bf e}}|_{(1)}}  $.
	Additionally, the upper bound 
is sharp if and only if
	\[
	\frac{p}{1-p} \leq val(Q_1(\frac{\widetilde{{\bf e}}'}{\|\widetilde{{\bf e}}'\|_\infty}))=val(Q_1(\widetilde{{\bf e}}''))/\|\widetilde{{\bf e}}'\|_\infty
	\]
	therefore, the upper bound 
holds with equality 
	\[
	p^\star= 1- \frac{|\widetilde{{\bf e}}|_{(2)}}{|\widetilde{{\bf e}}|_{(1)}}
	\]
if 
		\[
	p\leq 1- \frac{|\widetilde{{\bf e}}|_{(3)}}{|\widetilde{{\bf e}}|_{(2)}}
	\]
	Therefore, if 
		\[
	\frac{|\widetilde{{\bf e}}|_{(3)}}{|\widetilde{{\bf e}}|_{(2)}} \leq \frac{|\widetilde{{\bf e}}|_{(2)}}{|\widetilde{{\bf e}}|_{(1)}}
	\]
	then
	\[
	p^\star= 1- \frac{|\widetilde{{\bf e}}|_{(2)}}{|\widetilde{{\bf e}}|_{(1)}}.
	\]
	
	\end{lemma}

The above narrative gives a sketch of our result and its proof. The upper bound $ 1- \frac{|\widetilde{{\bf e}}|_{(2)}}{|\widetilde{{\bf e}}|_{(1)}}$ of $p^\star$ generalized the previous special case of exponential decay filter in theorem~\ref{thm: single root PT} with $p^\star = 1-|s|$.
\item
\begin{lemma}[Geometric lower bound $val(Q_1(\widetilde{{\bf e}}'))$ for the phase transition condition]
\label{lemma: geo lower}
    \[
val(Q_1(\widetilde{{\bf e}}'))
\geq p \cot\angle(\widetilde{{\bf e}}, {\bf e}_0)
\]
where
\[
\cot\angle(\widetilde{{\bf e}}, {\bf e}_0) = \frac{1}{\|\widetilde{{\bf e}}'\|_2} 
\]
\end{lemma}
\item
Using the technical background provided in theorem~\ref{thm:BGPropLemmaNorm}, we can further provide a tighter upper bound to compute phase transition $p^\star$. 


\subsection{Technical Tools:  Landscape of  Expected Homogeneous Function over Bernoulli Support on Sphere}

Let 
\[
V_k({\bf u}) := \frac{{\mathbb E}_J\|{\bf u}\cdot J\|_2^k}{\|{\bf u}\|^k_{2}} 
\]
where ${\bf u}\in {\mathbb R}^N$, $J$ is a Bernoulli sequence indexed from $1$ to $N$.


\paragraph{Expectation $V_1({\bf u}):= {\mathbb E}_J \|{\bf u}\cdot J\|_2^{1}$ on sphere.}





\begin{theorem} \label{thm:BGPropLemmaNorm}
Let ${\bf u} \in {\mathbb R}^N$, then
\[
p \leq V_{1}({\bf u}) \leq V_{1}(\frac{\sum_{j\in [N]}\pm {\bf e}_j}{\sqrt{N}} ) \leq \sqrt{p} .
\]
the lower bound is approached by on-sparse vectors ${\bf u} \in \{ \pm {\bf e}_i, i\in [N]\}$, and the upper bound is approached by 
${\bf u}  \in \{  \frac{1}{\sqrt{N}}\sum_{j\in [N]}\pm {\bf e}_j\}$.

Furthermore, all the stationary points of $V_{1}({\bf u})$ are $\{\frac{\sum_{i\in J} \pm {\bf e}_i}{\sqrt{N_J}}\}$ for different support $J\subset {\cal T}$, where $\{ \pm {\bf e}_i, i\in [N]\}$ are the global minimizers, and $\{  \frac{1}{\sqrt{N}}\sum_{j\in [N]}\pm {\bf e}_j\}$ are the global maximizers. And for $J$ with $1<N_J<N$, $\{\frac{\sum_{i\in J} \pm {\bf e}_i}{\sqrt{N_J}}\}$ are saddle points with value
\[
V_1(\frac{\sum_{i\in J} \pm {\bf e}_i}{\sqrt{N_J}}) = {\mathbb E}_I \sqrt{\frac{\sum_{i\in J} 1_{I_i}}{N_J}} = \sum_{j=0}^{N_J} (1-p)^{N_J-j} p^{j} {N_J \choose j}\sqrt{\frac{j}{N_J}}.
\]
\end{theorem}
To support geometric intuition, Figure~\ref{fig-landscape-V1}
visualizes $V_1$ on the $2-$dimensional sphere.

\begin{figure} 
\includegraphics[width=.7\textwidth]{./figures_BD/Ju_landscape.png}
\caption{The value of $V_1$ on a two-dimensional sphere in ${\mathbb R}^3$, normalized by the affine transform to send the value in $[0,1].$  
}
\label{fig-landscape-V1}
\end{figure}


\paragraph{Expectation $V_{2k}({\bf u}):= {\mathbb E}_J \|{\bf u}\cdot J\|_2^{2k}$ on sphere.}
\begin{theorem} \label{thm:BGPropLemmaSub_k}
For $k\geq 2$, let ${\bf u}$ denote a vector in ${\mathbb R}^N$, then
\[
p^k \leq V_{2k}(\frac{\sum_{j\in [N]}\pm {\bf e}_j}{\sqrt{N}} ) \leq V_{2k}({\bf u}) \leq p.
\]
the upper bound is approached by one-sparse vectors ${\bf u} \in \{ \pm {\bf e}_i, i\in [N]\}$, and the lower bound is approached by 
${\bf u}  \in \{  \frac{1}{\sqrt{N}}\sum_{j\in [N]}\pm {\bf e}_j\}$.

Furthermore, all the stationary points of $V_{2k}(\psi)$ are $\{\frac{\sum_{i\in J} \pm {\bf e}_i}{\sqrt{N_J}}\}$ for different support $J\subset {\cal T}$, where $\{ \pm {\bf e}_i, i\in [N]\}$ are the set of global maximizers, and $\{  \frac{1}{\sqrt{N}}\sum_{j\in [N]}\pm {\bf e}_j\}$ are the set of all the global minimizers. And for $J$ with $1<N_J<N$, $\{\frac{\sum_{i\in J} \pm {\bf e}_i}{\sqrt{N_J}}\}$ are saddle points with value
\[
V_{2k}(\frac{\sum_{i\in J} \pm {\bf e}_i}{\sqrt{N_J}}) = {\mathbb E}_J (\frac{\sum_{i\in J} 1_{I_i}}{N_J})^k = \sum_{j=0}^{N_J} (1-p)^{N_J-j} p^{j} {N_J \choose j}(\frac{j}{N_J})^k.
\]
\end{theorem}

\paragraph{Expectation $V_{-1}({\bf u}):= {\mathbb E}_J \|{\bf u}\cdot J\|_2^{-1}$ on sphere.}
\begin{theorem}
\label{thm:TaylorSqrtInvExpect}
\begin{equation} 
  V_{-1}({\bf u}) 
  \geq p^{-1/2} .
\end{equation}
\end{theorem}


\subsection{Exact representation of $val(Q_1)$}
\paragraph{Exact representation from support  analysis} 
Let $|\widetilde{{\bf e}}|$ be the entry-wise absolute value of $\widetilde{{\bf e}}$. We can rank the entries of $|\widetilde{{\bf e}}|$ to be $|\widetilde{{\bf e}}|_{(1)}, |\widetilde{{\bf e}}|_{(2)}, |\widetilde{{\bf e}}|_{(3)},\ldots$, then the entries of $\widetilde{{\bf e}}'$ will be ranked as $|\widetilde{{\bf e}}|_{(2)}, |\widetilde{{\bf e}}|_{(3)},\ldots$. By scaling we have $|\widetilde{{\bf e}}|_{(1)}=1$. We define $\widetilde{{\bf e}}_{S_m}$ as the vector that only keep the entries $ \widetilde{{\bf e}}_{(2)}, \widetilde{{\bf e}}_{(3)},\ldots, \widetilde{{\bf e}}_{(m+1)}$ of $\widetilde{{\bf e}}$, and send the rest of entries to zero. 
\begin{lemma}[Exact representation of $val(Q_1(\widetilde{{\bf e}}'))$ via support decomposition and $V_1$ function]
\label{lemma: tigher upper}
     \[
val(Q_1(\widetilde{{\bf e}}'))= \inf_{ m\in \{1,2,\ldots, n_e-1\}}val(Q_1^{S_m})
\]

\end{lemma}

\end{itemize}

\textbf{Proof: }
Assume that $\widetilde{{\bf e}}$ has $n_e$ non-zero entry on support $S_{\widetilde{{\bf e}}}$, we can rank the absolute value of entries of $\widetilde{{\bf e}}$ to be $|\widetilde{{\bf e}}|_{(1)}, |\widetilde{{\bf e}}|_{(2)}, |\widetilde{{\bf e}}|_{(3)},\ldots,|\widetilde{{\bf e}}|_{(n_e)}$, then the entries of $\widetilde{{\bf e}}'$ will be ranked as $|\widetilde{{\bf e}}|_{(2)}, |\widetilde{{\bf e}}|_{(3)},\ldots,|\widetilde{{\bf e}}|_{(n_e)}$. 

We know the optimal solution of $(\beta')^\star$ of $\inf_{\langle   \widetilde{{\bf e}}', \beta'\rangle  =1} {\mathbb E}_{I'} \|\beta'\|_{\ell_2(I')}$ must have support $S_{\beta^\star}$ that satisfy $S_{\star}\subset S_{\widetilde{{\bf e}}'}$. From the symmetry of objective, we know if $(\beta')^\star$ is $m$ sparse, then $m\leq n_e-1$ and, its support must be on the top $m$ entries $|\widetilde{{\bf e}}|_{(2)}, |\widetilde{{\bf e}}|_{(3)},\ldots,|\widetilde{{\bf e}}|_{(m+1)}$, we call this support $S_m$, and we know the corresponding entries of $(\beta')^\star$ would have the same sign as entries of $\widetilde{{\bf e}}$. 

We can define the $m$ sparse optimization problem for a random support function on the subset of $S_m$: $J_m \subset S_m$. Let $\beta$ be supported on $S_m$ and each entry positive, then
\[
val(Q_1^{S_m}) :=\inf_{ \beta: \sum_{j=1}^{m}  |\widetilde{{\bf e}}|_{(j+1)} \beta_j  =1, \beta_j> 0} {\mathbb E}_{J_m} \|\beta \cdot J_m\|_{2} 
\]
The linear equality constraint comes from
\[
\langle  |\widetilde{{\bf e}}'_{S_m}|, \beta'\rangle =\sum_{j=1}^{m}  |\widetilde{{\bf e}}|_{(j+1)} \beta_j   =1
\]
Then by breaking the case to find minimum over at most $n_e -1$ non-zero entries  to  the minimum of the cases to find minimum over exactly $m$ non-zero entries, and also use sign symmetry, we have 
\[
val(Q_1(\widetilde{{\bf e}}'))= \inf_{ m\in \{1,2,\ldots, n_e-1\}}val(Q_1^{S_m})
\]



\paragraph{Tighter decomposition on $val(Q_1)$}
Now we can prove a more refine upper bound:
\begin{lemma}[Tighter upper bound on $val(Q_1)$]
    \[
val(Q_1(\widetilde{{\bf e}}'))= \inf_{ m\in \{1,2,\ldots, n_e-1\}}val(Q_1^{S_m})
= \inf_{ m\in \{1,2,\ldots, n_e-1\}} [V_1(\widetilde{{\bf e}}'_{S_m}) \cot\angle(\widetilde{{\bf e}}_{S_m}, {\bf e}_0)]
\]
where $V_1$ function takes value in $[p, \sqrt{p})$.

More specifically, let the unit vector along the direction of $|\widetilde{{\bf e}}'_{S_m}|$ be 
\[
u^m = |\widetilde{{\bf e}}'_{S_m}|/\|\widetilde{{\bf e}}'_{S_m} \|_2
\]
\begin{eqnarray*}
val(Q_1^{S_m}) 
&=& \cot\angle(\widetilde{{\bf e}}_{S_m}, {\bf e}_0)  V_1(|\widetilde{{\bf e}}'_{S_m}|) C(u^m)
\end{eqnarray*}
where 
\[
C(u^m):=\frac{1}{V_1(u^m)}\inf_{ \beta^m:  \beta^m_j> 0} \frac{V_1(\beta^m)}{\cos\angle(\beta^m, u^m)}  = \frac{1}{E_{J_m}\|u^m\cdot J_m\|_2}\inf_{ \beta^m:  \beta^m_j> 0} \frac{E_{J_m}\|\beta^m\cdot J_m\|_2}{\langle  \beta^m, u^m\rangle }
\]
And we can prove that
\[
C(u^m)= 1
\]
\end{lemma}



\begin{proof}[Proof of lemma~\ref{lemma: tigher upper}]
From 
\[
val(Q_1^{S_m}) :=\inf_{ \beta: \sum_{j=1}^{m}  |\widetilde{{\bf e}}|_{(j+1)} \beta_j  =1, \beta_j> 0} {\mathbb E}_{J_m} \|\beta\cdot J_m\|_{2} 
\]

For a fixed $m$, after re-ranking the entries by absolute value, let $S_m$ be the support so that only the top $m$ entries $|\widetilde{{\bf e}}|_{(2)}, |\widetilde{{\bf e}}|_{(3)},\ldots,|\widetilde{{\bf e}}|_{(m+1)}$ are non-zero,  let 
\[
|\widetilde{{\bf e}}'_{S_m}| = (0, |\widetilde{{\bf e}}|_{(2)}, |\widetilde{{\bf e}}|_{(3)},\ldots,|\widetilde{{\bf e}}|_{(m+1)}, 0, \ldots, 0)
\]
\[
\|\widetilde{{\bf e}}'_{S_m}\|_2^2 = |\widetilde{{\bf e}}|^2_{(2)}+ |\widetilde{{\bf e}}|^2_{(3)}+\ldots+|\widetilde{{\bf e}}|^2_{(m+1)} 
\]
let the unit vector along the direction of $|\widetilde{{\bf e}}'_{S_m}|$ be 
\[
u^m = |\widetilde{{\bf e}}'_{S_m}|/\|\widetilde{{\bf e}}'_{S_m} \|_2
\]
so that $u^m_j>0$, $j \in S_m$.
then 
\[
val(Q_1^{S_m}) := \frac{1}{\|\widetilde{{\bf e}}'_{S_m}\|_2}\inf_{ \beta:  \beta_j> 0} \{ {\mathbb E}_{J_m} \| (\beta/\|\beta \|_{2}) \cdot J_m\|_{2} \cdot \frac{1}{\sum_{j=1}^{m}  u^m_j \frac{\beta_j }{\|\beta \|_{2}}} \}
\]
From previous definition, since $\beta$ is supported on $S_m$, we denote it as $\beta^m$, then
$$V_1(\beta^m)= {\mathbb E}_{J_m} \| (\beta/\|\beta \|_{2}) \cdot J_m\|_{2}, $$
$$V_1(\widetilde{{\bf e}}'_{S_m}) = V_1(|\widetilde{{\bf e}}'_{S_m}|) = V_1(u^m).$$


Geometrically,
\[
\cos\angle(\beta^m, u^m) = \sum_{j=1}^{m}  u^m_j \frac{\beta_j }{\|\beta \|_{2}}
\]
\[
\cot\angle(\widetilde{{\bf e}}_{S_m}, {\bf e}_0) =  \frac{1}{\|\widetilde{{\bf e}}'_{S_m}\|_2}
\]
\[
val(Q_1^{S_m}) =  \cot\angle(\widetilde{{\bf e}}_{S_m}, {\bf e}_0) \inf_{ \beta:  \beta_j> 0} \{V_1(\beta^m) \frac{1}{\cos\angle(\beta^m, u^m)} \}
\]



Since $\beta^m = u^m$ is a feasible point of the  linear equality constraint 
\[
\langle  |\widetilde{{\bf e}}'_{S_m}|, \beta'\rangle =\sum_{j=1}^{m}  |\widetilde{{\bf e}}|_{(j+1)} \beta_j   =1,
\]
we have an upper bound 
\[
val(Q_1^{S_m}) \leq \cot\angle(\widetilde{{\bf e}}_{S_m}, {\bf e}_0) V_1(\widetilde{{\bf e}}'_{S_m})
\]
On the other hand,
\begin{eqnarray*}
val(Q_1^{S_m}) 
&=& \cot\angle(\widetilde{{\bf e}}_{S_m}, {\bf e}_0) \inf_{ \beta:  \beta_j> 0} V_1(\beta^m) \cdot \frac{1}{\cos\angle(\beta^m, u^m)}
\\
&=& \cot\angle(\widetilde{{\bf e}}_{S_m}, {\bf e}_0)  V_1(|\widetilde{{\bf e}}'_{S_m}|) \inf_{ \beta:  \beta_j> 0} \frac{V_1(\beta^m)}{V_1(u^m)} \cdot \frac{1}{\cos\angle(\beta^m, u^m)}
\end{eqnarray*}
The lower bound is given by finding the lower bound of
\[
C(u^m):=\frac{1}{V_1(u^m)}\inf_{ \beta^m:  \beta^m_j> 0} \frac{V_1(\beta^m)}{\cos\angle(\beta^m, u^m)}  = \frac{1}{E_{J_m}\|u^m\cdot J_m\|_2}\inf_{ \beta^m:  \beta^m_j> 0} \frac{E_{J_m}\|\beta^m\cdot J_m\|_2}{\langle  \beta^m, u^m\rangle }
\]
First, we know upper bound $C(u^m)\leq 1$ by plug in $\beta^m = u^m$ as feasible point. 

This value $C(u^m)$ can be restricted to the $m$-dimensional subspace supported on $S_m$, therefore its value is independent of ambient space dimension, only dependent on the non-zero entries of $u^m$, without loss of generosity we only need to consider the case when $u^m$ is a $m$-dimensional dense vector lie on the first  quadrant (all positive non-zero entries) of the unit sphere.

If $u^m$ approaches one-sparse point by taking the rest of coordinate close to zero, then $\beta^m$ should equal $u^m$.

On the other extreme, if $u^m$ is has $m$ equal entries $\frac{1}{\sqrt{m}}$, via numerical simulation we could check that \[
C(u^m)=1
\]
We conjecture that $C(u^m)=1$.We only need to prove
$C(u^m)\geq 1$.
To prove the lower bound, we only need to show that, on positive quadrant of $m$-dimensional unit sphere, the function 
$$R_{u^m}(\beta^m)=\frac{1}{E_{J_m}\|u^m\cdot J_m\|_2}\frac{E_{J_m}\|\beta^m\cdot J_m\|_2}{\langle  \beta^m, u^m\rangle }\geq 1 $$
and takes minimum value $1$
  at $\beta^m=u^m$.




In general, by symmetry, expect our guess $\beta^m = u^m$, for any $u^m$ on  we could infer that the $2$ other types of points to check on positive quadrant of unit sphere are $ \beta^m = (0, \sqrt{1-\epsilon^2}, \sqrt{\epsilon},0\ldots,  0)$ and $\beta^m = (\frac{1}{\sqrt{m}}, \ldots, \frac{1}{\sqrt{m}}).$ 
 
First , for $\beta^m = (\frac{1}{\sqrt{m}}, \ldots, \frac{1}{\sqrt{m}})$, then $\langle  \beta^m, u^m\rangle=  \frac{1}{\sqrt{m}} 1^T u^m \leq 1$ by Cauchy inequality,  we have
\[
R_{u^m}(\beta^m) = \frac{1}{E_{J_m}\|u^m\cdot J_m\|_2} \frac{E_{J_m}\|\beta^m\cdot J_m\|_2}{\langle  \beta^m, u^m\rangle} \geq \frac{E_{J_m}\|(\frac{1}{\sqrt{m}}, \ldots, \frac{1}{\sqrt{m}})\cdot J_m\|_2}{E_{J_m}\|u^m\cdot J_m\|_2}\geq 1
\]
The  last  inequality comes from the saddle point property of $V_1$ at point $(\frac{1}{\sqrt{m}}, \ldots, \frac{1}{\sqrt{m}})$.
 since $V_1((\frac{1}{\sqrt{m}}, \ldots, \frac{1}{\sqrt{m}})) = \sum_{j=0}^{m} (1-p)^{m-j} p^{j} {m \choose j}\sqrt{\frac{j}{m}} = \sqrt{p} - \sum_{j=m+1}^{\infty} (1-p)^{m-j} p^{j} {m \choose j}\sqrt{\frac{j}{m}}  \in [p, \sqrt{p}).$

Second, for $ \beta^m = (0, \sqrt{1-\epsilon^2}, \sqrt{\epsilon},0\ldots,  0) $, if $\beta^m$ approaches one-sparse point by taking the rest of coordinate close to zero, by continuity of function $R_{u^m}(\beta^m)$, we can consider the limit point on the boundary $ \beta^m = (0, 1, 0,0\ldots,  0) $, then
\[
R_{u^m}(\beta^m) = \frac{1}{E_{J_m}\|u^m\cdot J_m\|_2} \frac{E_{J_m}\|\beta^m\cdot J_m\|_2}{\langle  \beta^m, u^m\rangle} \geq \frac{p \|u^m\|^2_2}{E_{J_m}\|u^m\cdot J_m\|_2 \|u^m\|_\infty}
\]
 

We can first compute gradient of $R_{u^m}(\beta^m)$, show that its directional gradient are all non-negative at $u^m$. 







\end{proof}
		
		
	
	












\subsection{Technical Tool: Relation between Finite Difference of Objective and Euclidean Distance }
 \paragraph{Bi-Lipschitzness of finite difference of objective}
As an important proof tool, we study functional $B$ that allows us to bound the Euclidean distance $d_2(\psi, {\bf e}_0) := \| \psi - {\bf e}_0\|_2$ by the objective difference ${\mathbb E}_I  \| \psi\cdot I \|_2- {\mathbb E}_I  \| ({\bf e}_0)\cdot I \|_2$  : 
\[
B({\bf e}_0,\phi) := \frac{{\mathbb E}_I  \| ({\bf e}_0+\phi)\cdot I \|_2 - {\mathbb E}_I  \| ({\bf e}_0)\cdot I \|_2}{\|\phi\|_2} 
\]
When we rescale $\phi$ to $\beta:= \frac{\phi}{\|\phi \|_2}$, we have
\[
B({\bf e}_0,t \beta) = \frac{{\mathbb E}_I  \| ({\bf e}_0+t \beta)\cdot I \|_2 - {\mathbb E}_I  \| ({\bf e}_0)\cdot I \|_2}{t} 
\]
From the definition of directional derivative,
\[
\nabla_{\phi}{{\mathbb E}_I  \| ({\bf e}_0+\phi)\cdot I \|_2}\mid_{\phi=0}=\lim_{t\rightarrow 0^+}\frac{{\mathbb E}_I  \| ({\bf e}_0+t \beta)\cdot I \|_2 - {\mathbb E}_I  \| ({\bf e}_0)\cdot I \|_2}{t}=\lim_{t\rightarrow 0^+}B({\bf e}_0,t \beta).
\]

We have upper and lower bound on their difference.
This upper and lower bound allows us to connect objective ${\mathbb E}_I  \| \psi\cdot I \|_2$ and the $2-$norm of $ \psi - {\bf e}_0$:
\begin{theorem}[Bi-Lipschitzness of finite difference of objective near ${\bf e}_0$ for linear constraint]
\label{thm:bound_diff}
We have upper and lower bound of $B({\bf e}_0, \phi)$ :
\begin{eqnarray*}
0\leq B({\bf e}_0,t \beta) -\lim_{t\rightarrow 0^+}B({\bf e}_0,t \beta) 
\leq 
  \frac{t}{2}p\Big( 
  \beta_0^2 
 +
 p  (1-\beta_0^2) \Big) \leq \frac{pt}{2}.
\end{eqnarray*}
This leads to
\[
0 \leq B({\bf e}_0, \phi) -	 \nabla_{\phi}{{\mathbb E}_I  \| ({\bf e}_0+\phi)\cdot I \|_2}\mid_{\phi=0}\leq \frac{p}{2}\|\phi \|_2.
\]
Therefore, when $p<p^\star$, $\nabla_{\phi}{{\mathbb E}_I  \| ({\bf e}_0+\phi)\cdot I \|_2}\geq \epsilon(p,p^\star)>0$, 
we have $B({\bf e}_0, \phi) \geq \epsilon(p,p^\star)>0$, which allows us to bound difference of objective by Euclidean distance .
Reversely, 
$1/B({\bf e}_0, \phi) \leq  1/\epsilon(p,p^\star)$, which allows us to bound Euclidean distance by difference of objective.

\end{theorem}


\section{Supplementary: Main Result $2$ Proof: Guarantee for Finite Observation Window, Finite-Length Inverse }
\subsection{Phase Transition in  Finite Observation Window, Finite-Length Inverse Setting}
\paragraph{Finite sample setting}
\begin{itemize}
\item
Let $(X_t)_{t \in {\mathbb Z}}$ be IID sampled from  $pN(0,1) + (1-p) \delta_0$.
 \item
The filter $a\in l_1({\mathbb Z}), (\mathcal{F} a)(\omega) \neq 0, \forall \omega\in T.$ Additionally, in finite sample setting, we assume $a^{-1}$ is zero outside of a centered window of radius $k$.
\item
Let $Y= a* X$ be a linear process, we are given a series of observations $(Y_t)_{t\in [-(T+k),T+k]}$ from a centered window of radius $T+k$.
\item
 We denote $X^{(i)}$ as the sequence being flipped and shifted for $i$ step from $X$ so that $X^{(i)}_k = X_{-k +i}$. Let $J_N$ be the support of $X$.
 We define the length $N=2T+1$ window as ${\cal T} := [-T, T].$
\end{itemize}
Now we want to find an inverse filter 
$w = (w_{-k}, \ldots, w_0, \ldots, w_k) \in V$ such that the convolution $w*Y$ is sparse,
\begin{equation}
 \begin{array}{ll}
 \underset{w\in V}{  \mbox{minimize} }   & \frac{1}{N}\|w*Y\|_{1,{\cal T}}\\
 \mbox{subject to}  
& (\widetilde{a}* w)_0 =1.
\end{array}
\tag{${BD}_{\widetilde{a},N,k}$}
\end{equation}
\paragraph{Finite sample directional derivative}
Now to study the finite sample convex problem, we need to calculate the directional derivative at $ \psi = e_0$.
Then
\begin{theorem}
\label{thm:fin_deriv}
For any $X$ sequence with support  $J_N$,
\begin{eqnarray*}
& & \lim_{t\rightarrow 0^+}\frac{1}{t}[(\hat{{\mathbb E}}  \| (e_0+t\beta)*X \|_1 - \hat{{\mathbb E}}  \| e_0*X \|_1)
\\
  &=& 
 \langle  \beta, {\frac{1}{N}\sum_{|i|\in {\cal T}} \nabla_{\psi}L_i}|_{e_0} \rangle \\
 &=& 
 \frac{1}{N}\sum_{|i|\in {\cal T}}  \mbox{sign}( X_{-i})X^{(i),T}\beta
 \\
  &=& 
 \frac{1}{N} \left(\|X\|_{1,J_N} \beta_0  +
 \|X*\beta_{(0)}\|_{1,{\cal T}-J_N}+ 
\sum_{i\in J_N} (1_{X_{-i}> 0}-1_{X_{-i}< 0})  \cdot X^{(i),T}\beta_{(0)} \right).
 \end{eqnarray*}
 
\end{theorem}










\subsection{Concentration of Objective}
 \paragraph{Concentration of $\|\psi*X\|_{1}$ }
 \begin{lemma}
 \label{lem:mu_min}
 Let 
 \[
\mu_{\mbox{min}}:=\frac{1}{\sqrt{2}\pi}\lambda_{m}(a)\sqrt{\frac{p}{k}} \|w\|_{1},
\]
then it is the lower bound of ${\mathbb E} \frac{1}{N}\|w*a*X\|_{1,{\cal T}}$:
\[
{\mathbb E} \frac{1}{N}\|w*a*X\|_{1,{\cal T}}  \geq \mu_{\mbox{min}}.
\]
 \end{lemma}

Now we define the intersection of $1-$norm ball of the $k-$dimensional subspace $V_k$ as $B_1(V_k)$, then we consider $w\in B_1(V_k).$ 
 
\begin{lemma}
\label{lem:tail_W}
Hence we consider $\psi = a*w \in a*B_1(V_k),$ and we define
\[
W:= \sup_{\psi \in a*B_1(V_k)}
|\frac{1}{N}\sum_{j\in [-T,T] } (|(\psi*X)_{j}| - {\mathbb E} |(\psi*X)_{j}|) |.
\]

For all $q\geq \max(2, \log(k)),$ there exists constant $C$,
\[
{\mathbb E} W^q\leq 2^{2q}(\frac{\lambda_{M}(a)}{N})^q   k C^q(\sqrt{Npq}+q)^q,
\]
so
\begin{eqnarray*}
 \|W\|_q \leq \frac{4eC}{N} \lambda_{M}(a)  
 (\sqrt{Np(q+\log(k))}+q +\log(k)),
\end{eqnarray*}
and
\begin{eqnarray*}
P( W > \frac{4eC\lambda_{M}(a) }{N}  
 (\sqrt{Np\log(\frac{k}{\delta})}+\log(\frac{k}{\delta})))\leq \delta.
\end{eqnarray*}
\end{lemma}

\begin{theorem}
\label{thm:concen_W}
Let the solution of finite sample convex optimization be $w^\star$, 
for small $\epsilon>0$ and concentration level $\delta>0$, 
there exists universal constant $C$, if there are 
\[
N \geq k\log(\frac{k}{\delta}) ( \frac{C\kappa_a}{\epsilon}  )^2,
\]
and  $ \frac{1}{N}\leq p$, we have:
\begin{eqnarray*}
P( W > \epsilon \mu_{\mbox{min}})\leq \delta.
\end{eqnarray*}

  \end{theorem}
 
\subsection{Concentration for Directional Derivatives} 
\paragraph{Uniform bound of directional derivatives}
We  define the uniform bound of directional derivatives:
\[
W_D  =\sup_{\|\beta\|_2=1, u^T \beta =0, \beta +e_0 \in a*B_1(V_k)}|\langle  \beta, {\frac{1}{N}\sum_{|i|\in {\cal T}} \nabla_{\phi}L_i}|_{\phi = 0} \rangle  -   \langle  \beta, {\mathbb E} \nabla_{\phi}L|_{\phi = 0} \rangle  |.
\]
Notice that
\begin{eqnarray*}
& &\langle  \beta, {\frac{1}{N}\sum_{|i|\in {\cal T}} \nabla_{\phi}L_i}|_{\phi = 0} \rangle  -   \langle  \beta, {\mathbb E} \nabla_{\phi}L|_{\phi = 0} \rangle  
\\
&=& 
\left[\frac{1}{N} \left(\|X\|_{1,J_N} \beta_0  +
 \|X*\beta_{(0)}\|_{1,{\cal T}-J_N}+ 
\sum_{i\in J_N} (1_{X_{-i}> 0}-1_{X_{-i}< 0})  \cdot X^{(i),T}\beta_{(0)}\right)\right]-
\\
& &
\left[\sqrt{\frac{2}{\pi}}(p \cdot \beta_0 + (1-p) \cdot {\mathbb E}_{I}[ \|\beta_{I}\|_2 | 0 \not \in I ])\right].
\end{eqnarray*}

Combining the following three uniform bounds on
\[
|(\frac{1}{N} \|X\|_{1,J_N}-\sqrt{\frac{2}{\pi}}p)|,
\]
\[
|\frac{1}{N}\|X*\beta_{(0)}\|_{1,{\cal T}-J_N}-\sqrt{\frac{2}{\pi}}(1-p) \cdot {\mathbb E}_{I}[ \|\beta_{I}\|_2 | 0 \not \in I ]|,
\]
\[
|\frac{1}{N}\sum_{i\in J_N} (1_{X_{-i}> 0}-1_{X_{-i}< 0})  \cdot X^{(i),T}\beta_{(0)}|,
\]
with high probability, we have  
\[
W_D  < \epsilon \mu_{\mbox{min}}.
\]

We define $\delta_p(N,\epsilon)$ the one side finite $N$ band such that for all $p<p^\star -\delta_p(N,\epsilon)$, 
\[
\langle  \beta, {\mathbb E} \nabla_{\phi}L|_{\phi = 0} \rangle >\epsilon \mu_{\mbox{min}}.
\]
for any direction $\beta \in V,$ the directional derivative at $\phi = 0$
\[
\langle  \beta, {\frac{1}{N}\sum_{|i|\in {\cal T}} \nabla_{\phi}L_i}|_{\phi = 0} \rangle  >0,
\]
then using the convexity argument, $a*w^\star-e_0=0$. 

\paragraph{Finite sample guarantee}
From previous concentration of finite sample directional derivative, we have the following theorem.
	\begin{theorem}
	\label{thm:fin_guarantee_supp}
		When $a^{-1}$ is length $k$, for small constant $\epsilon>0, \delta>0$, when the number of observation $N$ satisfies
		\[
N \geq  k\log(\frac{k}{\delta}) ( \frac{C\kappa_a}{\epsilon}  )^2,
\]
then 
\[
W_D  < \epsilon \mu_{\mbox{min}}.
\]
For $\frac{1}{N}\leq p \leq p^\star - \delta_p(N,\epsilon)$,
\[
\langle  \beta, {\mathbb E} \nabla_{\phi}L|_{\phi = 0} \rangle >\epsilon \mu_{\mbox{min}}.
\]
therefore, 	with probability $1-\delta$, for any direction $\beta \in V,$ the directional derivative at $\phi = 0$
\[
\langle  \beta, {\frac{1}{N}\sum_{|i|\in {\cal T}} \nabla_{\phi}L_i}|_{\phi = 0} \rangle  >0,
\]

		the solution of finite sample convex optimization $w^\star$ is $a^{-1}$ up to scaling and shift 
		with probability $1-\delta$.
	\end{theorem}

\subsection{Proof of Main Result $2$}	
	\begin{proof}[Proof of theorem~\ref{thm:fin_deriv}]
 \begin{eqnarray*}
& & \lim_{t\rightarrow 0^+}\frac{1}{t}[(\hat{{\mathbb E}}  \| (e_0+t\beta)*X \|_1 - \hat{{\mathbb E}}  \| e_0*X \|_1)
\\
  &=& 
 \langle  \beta, {\frac{1}{N}\sum_{|i|\in {\cal T}} \nabla_{\psi}L_i}|_{e_0} \rangle \\
 &=& 
 \frac{1}{N}\sum_{|i|\in {\cal T}}  \mbox{sign}( X_0^{(i)})X^{(i),T}\beta
 \\
 &=& 
 \frac{1}{N}\sum_{|i|\in {\cal T}}  \mbox{sign}( X_{-i})X^{(i),T}\beta
 \\
 &=& 
\frac{1}{N}\sum_{|i|\in {\cal T}} 1_{X_{-i}> 0}  \cdot X^{(i),T}\beta  +
\frac{1}{N}\sum_{|i|\in {\cal T}} 1_{X_{-i}< 0}  \cdot (-X^{(i),T}\beta)  +
\frac{1}{N}\sum_{|i|\in {\cal T}} 1_{X_{-i}= 0} \cdot |X^{(i),T}\beta|
 \\
 &=& 
\frac{1}{N}\sum_{|i|\in {\cal T}} 1_{X_{-i}> 0}  \cdot X^{(i),T}\beta  +
\frac{1}{N}\sum_{|i|\in {\cal T}} 1_{X_{-i}< 0}  \cdot (-X^{(i),T}\beta)  +
\frac{1}{N}\sum_{|i|\in {\cal T}}  |X^{(i),T}\beta_{(0)}|
 \\
 &=& 
\frac{1}{N}\sum_{|i|\in {\cal T}} 1_{X_{-i}> 0}  \cdot X_{-i}\beta_0  +
\frac{1}{N}\sum_{|i|\in {\cal T}} 1_{X_{-i}< 0}  \cdot (-X_{-i}\beta_0)  + 
\frac{1}{N}  \|X*\beta_{(0)}\|_{1,{\cal T}-J_N}+ 
\\
& &
\frac{1}{N}\sum_{|i|\in {\cal T}} 1_{X_{-i}> 0}  \cdot X^{(i),T}\beta_{(0)}  +
\frac{1}{N}\sum_{|i|\in {\cal T}} 1_{X_{-i}< 0}  \cdot (-X^{(i),T}\beta_{(0)})  
 \\
  &=& 
\frac{1}{N}\sum_{|i|\in {\cal T}} 1_{X_{-i}\neq 0}  \cdot |X_{-i}|\beta_0  +
\frac{1}{N}  \|X*\beta_{(0)}\|_{1,{\cal T}-J_N}+ 
\frac{1}{N}\sum_{i\in J_N} (1_{X_{-i}> 0}-1_{X_{-i}< 0})  \cdot X^{(i),T}\beta_{(0)}   
 \\
  &=& 
 \frac{1}{N} \left(\|X\|_{1,J_N} \beta_0  +
 \|X*\beta_{(0)}\|_{1,{\cal T}-J_N}+ 
\sum_{i\in J_N} (1_{X_{-i}> 0}-1_{X_{-i}< 0})  \cdot X^{(i),T}\beta_{(0)} \right).
 \end{eqnarray*}
\end{proof}
	
	
\begin{proof}[Proof of lemma~\ref{lem:mu_min}]
Without loss of generality, we rescale $w$ so that $\|w\|_1 =1$, now
\begin{eqnarray*}
{\mathbb E}_X Z 
&=& \frac{1}{N}\sum_{j\in [-T,T] }{\mathbb E}_X |\sum_{i=1}^k w_i (a*X)_{j-i}|\\
&\geq& \inf_{\|w\|_{1} = 1}\frac{1}{N}\sum_{j\in [-T,T] }{\mathbb E}_X |\sum_{i=1}^k w_i (a*X)_{j-i}|.
\end{eqnarray*}
By symmetry, we know the minimizer is $w = (\frac{1}{k}, \ldots,\frac{1}{k}),$
then we use the symmetrization trick to insert a sequence of $k$ IID Rademacher ($\pm 1$)  random variables 
$\varepsilon_1, \ldots, \varepsilon_k$ in the second inequality, and then apply Khintchine inequality as the third inequality,
\begin{eqnarray*}
{\mathbb E}_X Z 
&\geq&  \frac{1}{k} \frac{1}{N}\sum_{j\in [-T,T] }{\mathbb E}_X |\sum_{i=1}^k (a*X)_{j-i}|\\
&\geq& \frac{1}{2} \frac{1}{k} \frac{1}{N}\sum_{j\in [-T,T] }{\mathbb E}_X |\sum_{i=1}^k \varepsilon_i (a*X)_{j-i}|\\
&\geq&  \frac{1}{2} \frac{1}{k} \frac{1}{N}\sum_{j\in [-T,T] }\frac{1}{\sqrt{2}}{\mathbb E}_X \sqrt{ \sum_{i=1}^k  (a*X)^2_{j-i}}\\
&\geq&  \frac{1}{2} \frac{1}{k} \frac{1}{N}\sum_{j\in [-T,T] }\lambda_{m}(a)\frac{1}{\sqrt{2}}{\mathbb E}_X \sqrt{ \sum_{i=1}^k  X^2_{j-i}}\\
&\geq& \frac{1}{2}  \frac{1}{k} \frac{1}{\sqrt{2}}\frac{1}{N}\sum_{j\in [-T,T] }\lambda_{m}(a)\frac{2}{\pi}\sqrt{pk}\\
&=&  \frac{1}{\sqrt{2}\pi}\lambda_{m}(a)\sqrt{\frac{p}{k}}.\\
\end{eqnarray*}
\end{proof}
 
\begin{proof}[Proof of Lemma~\ref{lem:tail_W}]
Let $\varepsilon_1, \ldots, \varepsilon_N$ be a sequence of IID Rademacher ($\pm 1$)  random variables independent of $X$.  By the symmetrization inequality, see e.g. Lemma~$6.3$ of book \cite{ledoux2013probability}, we have the first inequality. Then since the function $t \rightarrow |t| $ is a contraction, an application of Talagrand’s contraction principle (see Lemma~$8$ of \cite{adamczak2016note}) with $F (x) = |x|^q$ 
conditionally on $X$ gives the second inequality.
\begin{eqnarray*}
{\mathbb E} W^q 
&\leq& 2^q {\mathbb E} \sup_{\psi \in a*B_1(V_k)} 
|\frac{1}{N}\sum_{j\in [-T,T] } \varepsilon_j |(\psi*X)_{j}| |^q\\
&\leq&
2^{2q}{\mathbb E} \sup_{\psi \in a*B_1(V_k)} |\frac{1}{N}\sum_{j\in [-T,T] } \varepsilon_j (\psi*X)_{j} |^q\\
&=&
2^{2q}\frac{1}{N^q}{\mathbb E} \sup_{\psi \in a*B_1(V_k)} |\sum_{i\in [-T,T] }\psi_{i}  \sum_{j\in [-T,T] } \varepsilon_j X_{j-i} |^q\\
&=&
2^{2q}\frac{1}{N^q}{\mathbb E} \sup_{w \in B_1(V_k)} |\sum_{i\in [-T,T] }(w*a)_{i}  \sum_{j\in [-T,T] } \varepsilon_j X_{j-i} |^q\\
&\leq&
2^{2q}\frac{1}{N^q}{\mathbb E} \max_{s\in [k]}|\sum_{i\in [-T,T] }a_{i-s}( \sum_{j\in [-T,T] } \varepsilon_j X_{j-i}) |^q\\
&\leq&
2^{2q}(\frac{\lambda_{M}(a)}{N})^q   \sum_{s\in [k] }{\mathbb E}| \sum_{j\in [-T,T] } \varepsilon_j X_{j-s} |^q.
\end{eqnarray*}
Now by the moment version of Bernstein’s inequality (see Lemma~$7$, equation $(18)$ of~\cite{adamczak2016note}), we know that there exists a universal constant $C$,
\begin{eqnarray*}
{\mathbb E}| \sum_{j\in [-T,T] } \varepsilon_j X_{j-i} |^q \leq C^q(\sqrt{Npq}+q)^q.
\end{eqnarray*}
Therefore, 
\begin{eqnarray*}
{\mathbb E} W^q &\leq& 2^{2q}(\frac{\lambda_{M}(a)}{N})^q    
\sum_{s\in [k] }{\mathbb E}| \sum_{j\in [-T,T] } \varepsilon_j X_{j-s} |^q \\
&\leq& 2^{2q}(\frac{\lambda_{M}(a)}{N})^q   k C^q(\sqrt{Npq}+q)^q.
\end{eqnarray*}
Then
\begin{eqnarray*}
 \|W\|_q \leq \frac{4eC}{N} \lambda_{M}(a)  
 (\sqrt{Np(q+\log(k))}+q +\log(k)).
\end{eqnarray*}
Therefore, we could get the tail bound using the Chebyshev inequality for the moments,
\begin{eqnarray*}
P( W > \frac{4eC\lambda_{M}(a) }{N}  
 (\sqrt{Np(q+\log(k))}+q +\log(k)))\leq e^{-q}.
\end{eqnarray*}
We set $q = \log(\frac{1}{\delta})$, obtaining an upper bound for $W$ which is satisfied with probability at least $1-\delta$:
\begin{eqnarray*}
P( W > \frac{4eC\lambda_{M}(a) }{N}  
 (\sqrt{Np\log(\frac{k}{\delta})}+\log(\frac{k}{\delta})))\leq \delta.
\end{eqnarray*}
\end{proof}	
	
	
	
\begin{proof}[Proof of Theorem~\ref{thm:concen_W}]
From previous lemma, we know that with the conditions, 
\begin{eqnarray*}
P( W > \epsilon \mu_{\mbox{min}})\leq \delta.
\end{eqnarray*}
Therefore, with probability at least $1-\delta$,
\begin{eqnarray*}
W = \sup_{\psi \in a*B_1(V_k)}
|\frac{1}{N}\sum_{j\in [-T,T] } (|(\psi*X)_{j}| - {\mathbb E} |(\psi*X)_{j}|) | \leq \epsilon \mu_{\mbox{min}}.
\end{eqnarray*}
When this is true, for all $\psi \in a*B_1(V_k)$, we have a uniform bound:
\[
 {\mathbb E} |(\psi*X)_{j}|-\epsilon\mu_{\mbox{min}} \leq \frac{1}{N}\sum_{j\in [-T,T] } (|(\psi*X)_{j}| \leq {\mathbb E} |(\psi*X)_{j}|+\epsilon\mu_{\mbox{min}}.
\]
\end{proof}
	
	
	
	
\begin{proof}[Proof of Theorem~\ref{thm:fin_guarantee}]
Notice that
\begin{eqnarray*}
& &\langle  \beta, {\frac{1}{N}\sum_{|i|\in {\cal T}} \nabla_{\phi}L_i}|_{\phi = 0} \rangle  -   \langle  \beta, {\mathbb E} \nabla_{\phi}L|_{\phi = 0} \rangle  
\\
&=& 
\left[\frac{1}{N} \left(\|X\|_{1,J_N} \beta_0  +
 \|X*\beta_{(0)}\|_{1,{\cal T}-J_N}+ 
\sum_{i\in J_N} (1_{X_{-i}> 0}-1_{X_{-i}< 0})  \cdot X^{(i),T}\beta_{(0)}\right)\right]-
\\
& &
\left[\sqrt{\frac{2}{\pi}}(p \cdot \beta_0 + (1-p) \cdot {\mathbb E}_{I}[ \|\beta_{I}\|_2 | 0 \not \in I ])\right].
\end{eqnarray*}
From the previous uniform bound, with high probability, we have the following three uniform bounds on
\[
|(\frac{1}{N} \|X\|_{1,J_N}-\sqrt{\frac{2}{\pi}}p)|,
\]
\[
|\frac{1}{N}\|X*\beta_{(0)}\|_{1,{\cal T}-J_N}-\sqrt{\frac{2}{\pi}}(1-p) \cdot {\mathbb E}_{I}[ \|\beta_{I}\|_2 | 0 \not \in I ]|.
\]
The uniform bound on 
\[
|\frac{1}{N}\sum_{i\in J_N} (1_{X_{-i}> 0}-1_{X_{-i}< 0})  \cdot X^{(i),T}\beta_{(0)}|
\]
comes from symmetric distribution assumption.

Combining all three uniform bounds, with high probability, we have:
\[
W_D  < \epsilon \mu_{\mbox{min}}.
\]

We define $\delta_p(N,\epsilon)$ the one-side finite $N$ band such that for all $p<p^\star -\delta_p(N,\epsilon)$, 
\[
\langle  \beta, {\mathbb E} \nabla_{\phi}L|_{\phi = 0} \rangle >\epsilon \mu_{\mbox{min}}.
\]
for any direction $\beta \in V,$ the directional derivative at $\phi = 0$
\[
\langle  \beta, {\frac{1}{N}\sum_{|i|\in {\cal T}} \nabla_{\phi}L_i}|_{\phi = 0} \rangle  >0,
\]
then using the convexity argument, $a*w^\star-e_0=0$. 

	
\end{proof}

	
	
	
	
	
	
	
	





\section{Supplementary: Main Result 3 Proof: Stability Guarantee with Finite Length Approximation to Infinite Length Inverse  }
\subsubsection{Finite Length Approximation to Infinite Length Inverse Filter}
Now we consider a setting where there is a kernel whose corresponding inverse kernel has infinite support, and we give a finite-length approximation. 

Within the space $V_{-\infty,\infty}$ of bilaterally infinite real-valued sequences $(h(i))_{i \in {\mathbb Z}}$, consider the affine subspace $V_{N_{-}, N_{+}} = \{ (...,0,0,h_{-N_{-}},\ldots, h_{-1}, 1, h_1, h_2, \ldots, h_{N_{+}},0,0,...)\}$, an $N_{-}+N_{+}$-dimensional subspace of bilateral sequences with support at most $N_{-}+N_{+}+1$.
The coordinate that is fixed to one is 
located at index $i=0$.
Each coordinate is zero outside of a window
of size $N_{-}$ on the left of zero 
and  size $N_{+}$ on the right of zero.
The special sequence $e_0 = (\dots,0,0,1,0,0,\dots)$,
vanishing everywhere except the origin,
belongs to $V_{-\infty,\infty}$ and to every
$V_{N_-,N_+}$.

Let $a \in V_{N_-,N_+}$.
Then $a =  (...,0,0,a_{-N_{-}},\ldots, a_{-1}, 1, a_1, a_2, \ldots, a_{N_{+}},0,0,...)$.
We also write $a = e_0 + a_{(0)}$,
where $a_{(0)} \equiv (1-e_0) \cdot a$ 
denotes the `part of $a$ supported away from location
$i=0$'. We also write $a = e_0 + a_L + a_R$,
where $a_L = (a_L(i))_{i \in {\mathbb Z}} 
= (a(i) 1_{\{i < 0\}})_{i \in {\mathbb Z}}$ 
denotes the `part of $a$ supported to
the left of $i=0$' and 
where $a_R = (a_R(i))_{i \in {\mathbb Z}} 
= (a(i) 1_{\{i > 0\}})_{i \in {\mathbb Z}}$
denotes the `part of $a$
supported to the right of $i=0$'.
Finally, we say that $a \in V_{N_{-}, N_{+}}$ 
is a length $ L= N_{-}+N_{+}+1$ filter.

\textbf{First example}: let's look at a simple example of infinite length inverse filter approximation.
Let $s\in (-1,1)$. For $a=( 0, 1, -s)$, then $a^{-1} = (\ldots, 0, 1, s, s^2, s^3, \ldots)$ is an infinite length inverse filter. 

If we choose a length $r$ approximation $w^{r}= (\ldots, 0, 1, s, s^2, s^3, \ldots, s^{r-1})$ then
\[
\|a*w^{r}-e_0\|_2^2 = |s|^{2r}.
\]


Now we consider general finite-length forward filter with infinite-length inverse filter.

\subsection{Finite Length Approximation based on Z Transform}
We construct the finite-length approximation filter explicitly by truncation of Z-transform.

Let the Z-transform of $a$ be 
\[
 \begin{array}{ll}
A(z) =  \sum_{i=-s}^t a_{i} z^{-i}.
\end{array}
  \]
Then Z-transform of the inverse kernel $a^{-1}$ is $1/A(z)$.

\begin{theorem}
Assuming we have a finite length forward filter $a$ with its Z-transform having roots inside the unit circle, namely $s_{k}:= e^{-\rho_k + i \varphi_k }$ with $|s_k|<1$ and $\rho_k>0$
for $k \in \{-N_{-}, \ldots -1, 1, \ldots N_{+}\}$. Let ${\cal I} = \{-N_{-},\ldots, -1, 1, \ldots, N_{+} \}$ as the set of all the possible indicies.
\[
 \begin{array}{ll}
A(z) &=  \sum_{i=-N_{-}}^{N_{+}} a_{i} z^{-i}
\\
&= c_0\prod_{j=1}^{N_{-}} (1- s_{-j} z) \prod_{i=1}^{N_{+}} (1- s_{i} z^{-1}),
\end{array}
  \]
Where $c_0$ is a constant to make sure that the coefficient $a_0=1$.

Then for a vector index $r=(r_{-N_-}, \ldots, r_{-1}, r_{1},\ldots, r_{N_+} ) $, we could construct an approximate inverse filter $w^r$ with Z-transform
\[
 \begin{array}{ll}
W(z) 
&=  \frac{1}{c_0}\prod_{j=1}^{N_{-}} (\sum_{\ell_j=0}^{r_{-j}-1} s^{\ell_j}_{-j} z^{-\ell_j} ) \prod_{i=1}^{N_{+}} (\sum_{\ell_i=0}^{r_{-j}-1} s^{\ell_i}_{i} z^{\ell_i} )\\
&=  \frac{1}{c_0}\prod_{j=1}^{N_{-}} (1- (s_{-j} z)^{r_{-j}})(1- s_{-j} z)^{-1} \prod_{i=1}^{N_{+}}(1- (s_{i}  z^{-1})^{r_i}) (1- s_{i} z^{-1})^{-1}.
\end{array}
\]
Let $\phi^r = w^r*a-e_0$, then
\[
 \begin{array}{ll}
\|\phi^r \|_2^2 = \|a*w^r-e_0\|_2^2
&= \sum_{n=1}^{|{\cal I}|}\sum_{ k_1, \ldots, k_n  \in {\cal I} } 
\exp \Big(- 2 \big(r_{k_1} \rho_{k_1}+ \ldots+  r_{k_n} \rho_{k_n} \big) \Big).
\end{array}
  \]
When $\min_j r_j \rightarrow \infty$, it converges to zero at an exponential rate. The convergence rate is determined by the slowest decaying exponential term as a function of $\min_i (r_i \rho_i)$.
\[
 \begin{array}{ll}
\|\phi^r \|_2
&= O(
\exp \Big(-  \min_i (r_i \rho_i) \Big) ), \quad \min_i r_i \rightarrow \infty.
\end{array}
  \]
 \end{theorem}
\subsection{Stability Theorem}
 \begin{theorem} 
 Let $a\in V_{N_-, N_+} $ be a forward filter with all the roots of Z-transform strictly in the unit circle. Let $w^\star \in V_{(r-1)N_-, (r-1)N_+} $ be the solution of the convex optimization problem. Let $w^r$ be the constructed filter in previous theorem with a uniform vector index $(r, \ldots, r, r,\ldots, r)$.
  Let $\phi^\star = w^\star*a-e_0$, 
 then the solution satisfy
  \[
 \begin{array}{ll}
  B(e_0,\phi^\star) \|\phi^\star \|_2 \leq B(e_0,\phi^r) \|\phi^r \|_2.
\end{array}
  \]
  Additionally, as $p<p^\star$, $B(e_0,\phi^\star)$ and $B(e_0,\phi^r)$ are both upper and lower bounded. 
  Therefore, using the previous asymptotic exponential convergence bound on $\|\phi^r \|_2$ as 
  $r \rightarrow \infty$, it converges to zero at an exponential rate
  \[
 \begin{array}{ll}
   \|\phi^\star \|_2 \leq \frac{B(e_0,\phi^r)}{B(e_0,\phi^\star)} \|\phi^r \|_2
   \leq
   O(
\exp \Big(-  r \min_i (\rho_i) \Big) ), \quad r \rightarrow \infty.
\end{array}
  \]
 \end{theorem} 
 

\subsection{Proof of Stability Theorem}	

\begin{proof}[Proof of Theorem~\ref{thm:RootNorm}]
Now let $\psi^r$ have Z-transform $\Psi(z)$, $\phi = a*w-e_0$ with Z-transform $\Phi(z)$,
\[
 \begin{array}{ll}
\Psi(z): = A(z)W(z) 
&=   \prod_{j=1}^{N_{-}} (1- (s_{-j} z)^{r_{-j}}) \prod_{i=1}^{N_{+}}(1- (s_{i} z^{-1})^{r_i}).
\end{array}
\]
Therefore,
\[
 \begin{array}{ll}
\Phi(z): = A(z)W(z) -1 
&=  \prod_{j=1}^{N_{-}} (1- (s_{-j} z)^{r_{-j}}) \prod_{i=1}^{N_{+}}(1- (s_{i} z^{-1})^{r_i}) -1.
\end{array}
  \]
Let ${\cal I} = \{-N_{-},\ldots, -1, 1, \ldots, N_{+} \}$ as the set of all the possible indicies.
And use polar representation of complex roots:
for any $k \in {\cal I} = \{-N_{-},\ldots, -1, 1, \ldots, N_{+} \}$
\[
s_{k} := |s_{k}| e^{i \varphi_k } = e^{-\rho_k + i \varphi_k },
\]
where 
\[
 \begin{array}{ll}
\varphi_k := \operatorname{Im}(\log( s_{k} )) = -i \log(\frac{s_{k}}{|s_{k}|}),
\end{array}
\]
and since $|s_{k}|<1$ we have 
\[
\rho_k = -\operatorname{Re}(\log( s_{k} ))>0.
\]

Now we consider all $z = e^{2 \pi i t}$ on the unit circle for $t\in [\frac{-1}{2},\frac{1}{2}]$,
then for any $k \in {\cal I}$
\[
 \begin{array}{ll}
- (s_{k} z^{-\mbox{sign}(k)})^{r_{k}} 
&= - |s_{k}|^{r_{k}} \exp \big(i [\varphi_k-  2 \pi t \mbox{sign}(k)]\cdot r_{k} \big)\\
&=   \exp \Big( -r_{k} \rho_k+ i \cdot \big( r_{k} (\varphi_k- 2 \pi t \mbox{sign}(k) ) + \pi  \big) \Big).
\end{array}
  \]
 Now we simplify the notation by defining 
 \[
 \upsilon_k(t) :=r_{k} (\varphi_k- 2 \pi t \mbox{sign}(k) ) + \pi,
 \]
then
 \[
 \begin{array}{ll}
\Phi(z)
&=  \prod_{j=1}^{N_{-}} (1- (s_{-j} z)^{r_{-j}}) \prod_{i=1}^{N_{+}}(1- (s_{i} z^{-1})^{r_{i}}) -1\\
&=  \prod_{k \in {\cal I}} \Big(1- |s_{k}|^{r_{k}} \exp \big(i [\varphi_k-  2 \pi t \mbox{sign}(k)]\cdot r_{k} \big)\Big)  -1\\
&=  \prod_{k \in {\cal I}} \bigg( 1+ \exp \Big( -r_{k} \rho_k+ i \cdot \big( r_{k} (\varphi_k- 2 \pi t \mbox{sign}(k) ) + \pi  \big) \Big) \bigg)     -1\\
&=  \prod_{k \in {\cal I}} \bigg( 1+ \exp \Big( -r_{k} \rho_k+ i \cdot \upsilon_k(t) \Big) \bigg) -1.
\end{array}
  \]
  
 For any $z = \exp{(2 \pi i t )}$ on the unit circle,
 \[
 \begin{array}{ll}
\Phi\big(\exp{(2 \pi i t )}\big)
&=  \prod_{k \in {\cal I}} \bigg( 1+ \exp \Big(- r_{k} \rho_k+ i \cdot \upsilon_k(t) \Big) \bigg)     -1\\
&= \sum_{n=1}^{|{\cal I}|}\sum_{ k_1, \ldots, k_n  \in {\cal I} } 
\exp \Big( -\big(r_{k_1} \rho_{k_1}+ \ldots+  r_{k_n} \rho_{k_n} \big)
+ i \cdot \big( \upsilon_{k_1}(t) + \ldots+  \upsilon_{k_n}(t)  \big) \Big).
\end{array}
  \]
 
Additionally, for any $t \in [\frac{-1}{2},\frac{1}{2}]$,
\[
 \begin{array}{ll}
 |\Phi\big(\exp{(2 \pi i t )}\big)|^2  
= 
\sum_{n=1}^{|{\cal I}|}\sum_{ k_1, \ldots, k_n  \in {\cal I} } 
\exp \Big( -2 \big(r_{k_1} \rho_{k_1}+ \ldots+  r_{k_n} \rho_{k_n} \big) \Big)
\end{array}
  \]
is independent of $t$, since the oscillation integral over the imaginary part is 
  \[
 \begin{array}{ll}
\int_{\frac{-1}{2}}^\frac{1}{2}
\exp  \bigg( i \cdot \Big(  \big(\upsilon_{k_1}(t) - \upsilon_{k_1}(t) \big) + \ldots+  \big(\upsilon_{k_n}(t) -\upsilon_{k_n}(-t)\big)  \Big) \bigg)  dt
= 1.
\end{array}
  \]
Therefore, using Fourier isometry,
 \[
 \begin{array}{ll}
 \|\phi\|_2^2 &= \int_{\frac{-1}{2}}^\frac{1}{2} |\Phi\big(\exp{(2 \pi i t )}\big)|^2 dt \\
&= \sum_{n=1}^{|{\cal I}|}\sum_{ k_1, \ldots, k_n  \in {\cal I} } 
\exp \Big( -2 \big(r_{k_1} \rho_{k_1}+ \ldots+  r_{k_n} \rho_{k_n} \big) \Big).
\end{array}
  \]
When $\min_j r_j \rightarrow \infty$, it converges to zero at an exponential rate. The convergence rate is determined by the slowest decay exponential term as a function of $\min_i (r_i \rho_i)$.
\[
 \begin{array}{ll}
\|\phi^r \|_2
&= O(
\exp \Big(-  \min_i (r_i \rho_i) \Big) ), \quad \min_i r_i \rightarrow \infty.
\end{array}
  \]
  
\end{proof}


\begin{proof}[Proof of Theorem~\ref{thm:bd_stab}]
Following the previous phase transition analysis, 
let the directional finite difference of the objective at $e_0$ be $D(w) = |(w*Y)_0| - |X_0|$, $I$ be the support of $\tilde{X}$, let $\phi = w*a-e_0$,  then we have 
\[
 \begin{array}{ll}
{\mathbb E} D(w) &= \sqrt{\frac{2}{\pi}}{\mathbb E}_I [\|(w*a)_I\|_2 - \|(e_0)_I\|_2]\\
&= \sqrt{\frac{2}{\pi}}{\mathbb E}_I [\|(e_0+\phi)_I\|_2 - \|(e_0)_I\|_2]\\
&= \sqrt{\frac{2}{\pi}}\|{\phi}\|_2 B(p, \phi) . \\
\end{array}
  \]
Now using the optimality of $w^\star \in V_{(r-1)N_-, (r-1)N_+} $, for all $w^r in V_{(r-1)N_-, (r-1)N_+}$,
we have
 \[
 \begin{array}{ll}
{\mathbb E}_I [\|(w^\star*a)_I\|_2 \leq {\mathbb E}_I [\|(w^r*a)_I\|_2,
\end{array}
  \]
  therefore,
  \[
 \begin{array}{ll}
B(p, \phi^\star) \|{\phi^\star}\|_2 \leq B(p, \phi)\|{\phi}\|_2.
\end{array}
  \]
Due to the bi-Lipschitz property of ${\mathbb E}_I \|(e_0+\phi)_I\|_2$ around $e_0$, we know when $p<p^\star$, $B(p, \phi)$ is positive and bounded by a constant.

Therefore, as $p<p^\star$, $B(e_0,\phi^\star)$ and $B(e_0,\phi^r)$ are both upper and lower bounded. 
 \[
 \begin{array}{ll}
   \|\phi^\star \|_2 \leq \frac{B(e_0,\phi^r)}{B(e_0,\phi^\star)} \|\phi^r \|_2.
\end{array}
  \]
 Using the previous theorem, 
 \[
 \begin{array}{ll}
   \|\phi^\star \|_2 \leq \frac{B(e_0,\phi^r)}{B(e_0,\phi^\star)} \sum_{n=1}^{|{\cal I}|}\sum_{ k_1, \ldots, k_n  \in {\cal I} } 
\exp \Big( -2 r\big(\rho_{k_1}+ \ldots+ \rho_{k_n} \big) \Big),
\end{array}
  \]
 using asymptotic exponential convergence bound on $\|\phi^r \|_2$ as 
  $r \rightarrow \infty$, it converges to zero at an exponential rate
  \[
 \begin{array}{ll}
   \|\phi^\star \|_2 
   \leq
   O(
\exp \Big(-  r \min_i (\rho_i) \Big) ), \quad r \rightarrow \infty.
\end{array}
  \]
\end{proof}

\section{Supplementary: Main Result $4$ Proof: Robustness Guarantee against Stochastic and Adversarial  Noises }
\subsection{Robustness Theorem against Stochastic Noise: Moving Average Gaussian Noise }
Consider a convoluted Gaussian noise with an average standard deviation $\sigma$ and a forward moving average filter $b$ with unit norm $\|b\|_2=1$, then
 \[
\begin{array}{ll}
Y&= a* (X+ \sigma b*G).
\end{array}
\]


This model would include the case of IID mixture of sparse Gaussian and small Gaussian $X_t \sim p N(0,1) +(1-p) N(0,\sigma_1)$ with $\sigma = \sigma_1$ and $b=e_0 $.
It also includes the Gaussian observation noise model
\[
\begin{array}{ll}
Y&= a* X+  \sigma_2 G_2 = a* (X+  \sigma_2 a^{-1}*G_2),
\end{array}
\]
with $b = a^{-1}/\|a^{-1}\|_2$, and $\sigma = \sigma_2 \|a^{-1}\|_2$.

 \begin{theorem}
 Let $\psi^\star$ be the solution of the convex optimization problem in Eq.(4.1) for the moving average random noisy model
  \[
\begin{array}{ll}
Y&= a* (X+ \sigma b*G),
\end{array}
\]
 then
\[
B(e_0, \psi^\star-e_0) \|\psi^\star -e_0\|_2 = {\mathbb E}_I  \| (\psi^\star)_I \|_2 -{\mathbb E}_I  \| (e_0)_I \|_2 \leq  (1-p) \sigma +
 p(\sqrt{1+ \sigma^2 }-1).
\]
When $\sigma\leq 1,$
\[
(1-p) \sigma +
 p(\sqrt{1+ \sigma^2 }-1)  \leq \sigma,
\]
therefore, when $p<p^\star$, $\sigma\leq 1,$ there exists a constant $C$,
\[
 \|\psi^\star -e_0\|_2 \leq C \sigma.
\]
\end{theorem}  

\subsection{Robustness Theorem against Adversarial Noise }
In the adversarial noise setting, we observe
\[
	\begin{array}{ll}
	Y&= a* (X+  \zeta) 
	\end{array}
	\]
where $\zeta$ is a sequence chosen by an adversary under constraint:	
\[
	\| \zeta\|_\infty \leq \eta.
	\]
 \begin{theorem}
	Under adversarial noise, 
Let $w^\star$ be the solution of the population convex optimization
	\[
 \begin{array}{ll}
 \underset{w }{\mbox{minimize}}   &{\mathbb E} \frac{1}{N}\|w*Y\|_{\ell_1({\cal T}_N)}\\
\mbox{subject to}  
& (\alpha*w)_0 =1.
\end{array}
  \]
Define $\psi^\star = a*w^\star$; then $\psi^\star$ satisfies the following bound:
 \[
 \begin{array}{ll}
B(e_0, \psi^\star-e_0) \|\psi^\star -e_0\|_2 = {\mathbb E}_I  \| (\psi^\star)_I \|_2 -{\mathbb E}_I  \| (e_0)_I \|_2 \leq 
 p\sqrt{\frac{2}{\pi}}(\mathcal{R}(\eta) - 1)+(1-p)\eta.
\end{array}
  \]  
  Here $\mathcal{R}(\eta)$
is the folded Gaussian mean, for standard Gaussian $G$:
\[
\mathcal{R}(\eta) :=  \sqrt{\frac{\pi}{2}} E_G |\eta+  G| = \exp\{-\eta^2/2\} + \sqrt{\frac{\pi}{2}}\eta \left(1 - 2\Phi\left(-\eta\right) \right).
\]  
$\mathcal{R}(\eta)-1$ is an even function that is monotonically non-decreasing for $\eta \geq 0$ with quadratic upper and lower bound: there exists constants $C_1\leq C_2$,
  \[
C_1 \eta^2 \leq \mathcal{R}(\eta)-1 \leq C_2 \eta^2, \forall \eta.
\]
  
Therefore, when $p<p^\star$, $B(e_0, \psi^\star-e_0)$ is a bounded positive constant, there exists a constant $C$, so that
 \[
 \|\psi^\star -e_0\|_2  \leq C  \eta, \forall \eta > 0.
 \]
\end{theorem}  






\subsection{Proof of Robustness Theorem against Stochastic Noise }
\begin{proof}[Proof of Theorem~\ref{thm:rand_exact_rob}]
Consider the noisy model
 \[
\begin{array}{ll}
Y&= a* (X+  \sigma b*G).
\end{array}
\]
 First, 
\[
 \begin{array}{ll}
 F_{\sigma}(\psi)= {\mathbb E}_{G} {\mathbb E}_X|\psi^T X+\sigma \psi^T (b* G) | = 
   {\mathbb E}_{G} {\mathbb E}_I | \sqrt{ \|\psi_I\|_2^2  + \sigma^2\|C_{b}^T \psi\|_2^2}G |,
\end{array}
  \]
where $C_{b}$ is the Topelitz matrix with the first column being $b$.
  
Let $\psi^\star $ be the optimization solution, when $p<p^\star$,  we have  a chain of inequality:
    \[
 \begin{array}{ll}
F_{\sigma}(e_0)
 \geq F_{\sigma}(\psi^\star ) 
 \geq F_{0}(\psi^\star )
 \geq F_{0}(e_0).
\end{array}
  \]
  We have 
\[
 \begin{array}{ll}
 F_{\sigma}(e_0)
 &= 
{\mathbb E}_{G} {\mathbb E}_I | \sqrt{ \|(e_0)_I\|_2^2  + \sigma^2\|b\|_2^2}G | \\
&= 
{\mathbb E}_{G} {\mathbb E}_I | \sqrt{ \|(e_0)_I\|_2^2  + \sigma^2}G | \\
  &=
 \sqrt{\frac{2}{\pi}}[ 
 (1-p) \sigma +
 p(\sqrt{1+ \sigma^2 }) ].
\end{array}
  \]
  
Therefore, when $p<p^\star$
   \[
 \begin{array}{ll}
F_{\sigma}(e_0)-F_{0}(e_0) 
 \geq F_{0}(\psi^\star )-F_{0}(e_0) \geq 0.
\end{array}
  \]
  \[
B(e_0, \psi^\star-e_0) \|\psi^\star -e_0\|_2 = {\mathbb E}_I  \| (\psi^\star)_I \|_2 -{\mathbb E}_I  \| (e_0)_I \|_2 \leq  (1-p) \sigma +
 p(\sqrt{1+ \sigma^2 }-1). 
\]
When $\sigma\leq 1,$
\[
(1-p) \sigma +
 p(\sqrt{1+ \sigma^2 }-1)  \leq \sigma,
\]
therefore, when $p<p^\star$, $\sigma\leq 1,$ there exists a constant $C$,
\[
 \|\psi^\star -e_0\|_2 \leq C \sigma.
\]
\end{proof}



\subsection{Proof of Robustness Theorem against  Adversarial Noise }
Our data generative model is
\[
	\begin{array}{ll}
	Y&= a* (X+  \zeta),
	\end{array}
	\]
where 	
\[
	\| \zeta\|_\infty \leq \eta.
	\]


\begin{proof}[Proof of Theorem~\ref{thm:adv_exact_rob}]
The population convex optimization is
	\[
 \begin{array}{ll}
 \underset{w, z }{\mbox{minimize}}   & {\mathbb E} \frac{1}{N}\|w*Y\|_{\ell_1({\cal T}_N)}\\
\mbox{subject to}  
& (\alpha*w)_0 =1.
\end{array}
  \]
 By change of variable $\tilde{\psi} = w*a$, it could be reduced to a simpler problem 
  \[
 \begin{array}{ll}
 \mbox{minimize}_{w,s}   & {\mathbb E}|\psi^T (X+  \zeta)| \\
\mbox{subject to}  
& u^T \psi =1.
\end{array}
  \]
 Let $\psi^\star $ be the optimization solution of the worst-case objective  over all possible $\zeta$, defined as $F_{\eta}(\psi)$:
 \[
 \begin{array}{ll}
 F_{\eta}(\psi)
 &= \sup_{\zeta: \| \zeta\|_\infty \leq \eta } {\mathbb E}|\psi^T (X+ \zeta)| \\
 &= \sup_{\zeta: \| \zeta\|_\infty \leq \eta } {\mathbb E}_I  | \|\psi_I\|_2 G  + \psi^T \zeta  |,
\end{array}
  \] 
   when $p<p^\star$,  we have a chain of inequality:
    \[
 \begin{array}{ll}
F_{\eta}(e_0)
 \geq F_{\eta}(\psi^\star ) 
 \geq F_{0}(\psi^\star )
 \geq F_{0}(e_0).
\end{array}
  \]
   Therefore, when $p<p^\star$
   \[
 \begin{array}{ll}
F_{\eta}(e_0)-F_{0}(e_0) 
\geq F(\psi(\zeta),\zeta)-F_{0}(e_0)
 \geq F_{0}(\psi^\star )-F_{0}(e_0) \geq 0.
\end{array}
  \]
  Therefore,
 \[
 \begin{array}{ll}
B(e_0, \psi^\star-e_0) \|\psi^\star -e_0\|_2 = {\mathbb E}_I  \| (\psi^\star)_I \|_2 -{\mathbb E}_I  \| (e_0)_I \|_2 .
\end{array}
  \]   
  
From the folded Gaussian mean formula, let $G$ be scalar standard Gaussian, we have 
\[
 \begin{array}{ll}
 F_{\eta}(e_0)
  &= \sup_{\zeta: \| \zeta\|_\infty \leq \eta } {\mathbb E}_I  {\mathbb E}_G| \|(e_0)_I\|_2 G  +  \zeta_0  |\\
&= 
\sup_{\zeta: \| \zeta\|_\infty \leq \eta} [p{\mathbb E}_G | G  +  \zeta_0  |+ (1-p)|\zeta_0|]\\
  &=
\sup_{\zeta: \| \zeta\|_\infty \leq \eta}[  p \sqrt{\frac{2}{\pi}} \mathcal{R}(\zeta_0) + (1-p)|\zeta_0|]
\\
 &=
p \sqrt{\frac{2}{\pi}} \mathcal{R}(\eta) + (1-p)\eta.
\end{array}
  \]
 The last inequality comes from the fact that $\mathcal{R}(\zeta_0)$ is an even function that is monotonically non-decreasing for $\zeta_0 \geq 0$. 
 We will prove this conclusion below:
 \[
 \begin{array}{ll}
B(e_0, \psi^\star-e_0) \|\psi^\star -e_0\|_2 = {\mathbb E}_I  \| (\psi^\star)_I \|_2 -{\mathbb E}_I  \| (e_0)_I \|_2 \leq 
 (1-p)\eta + p \sqrt{\frac{2}{\pi}} (\mathcal{R}(\eta) - 1).
\end{array}
  \] 



\end{proof}


\paragraph{Tool: folded Gaussian mean formula}
From general theory of folded Gaussian,
 $ |\mu+ \sigma G |, \quad  G \sim N(0, 1).$
Then its mean is
\begin{equation}
\label{FoldedGaussianMean}
E_G |\mu+ \sigma G| =  \sqrt{\frac{2}{\pi}} \sigma \exp\{-\mu^2/(2\sigma^2)\} + \mu \left(1 - 2\Phi\left(\frac{-\mu}{\sigma}\right) \right),
\end{equation}
where $\Phi$  is the normal cumulative distribution function:
\[
\Phi(x) = \frac 1 {\sqrt{2\pi}} \int_{-\infty}^x e^{-t^2/2} \, dt.
\]
As a related remark, its variance is
\begin{equation}
\label{FoldedGaussianVar}
{\rm Var}_G |\mu+ \sigma G| = E_G( |\mu+ \sigma G|-E_G |\mu+ \sigma G|)^2 =\sigma^2+ \mu^2 -(E_G |\mu+ \sigma G|)^2.
\end{equation}  

 \begin{lemma}
\label{lem:FoldedGaussian}
We define the ratio of folded Gaussian mean as 
\[
\mathcal{R}(\gamma) := \frac{E_G |\mu+ \sigma G|}{E_G |\sigma G|} = \frac{E_G |\mu+ \sigma G|}{\sqrt{\frac{2}{\pi}}\sigma} = \sqrt{\frac{\pi}{2}} E_G |\gamma+  G| = \exp\{-\gamma^2/2\} + \sqrt{\frac{\pi}{2}}\gamma \left(1 - 2\Phi\left(-\gamma\right) \right).
\]

$\mathcal{R}(\gamma)$ is an even function that is monotonically non-decreasing for $\gamma \geq 0$.

We have three different expressions (asymptotic expansion around  $\gamma$) for $\mathcal{R}(\gamma)$:
\begin{eqnarray}
\label{FoldedGaussianMean_ratio}
\mathcal{R}(\gamma)
&=&  \exp\{-\gamma^2/2\} + \sqrt{\frac{\pi}{2}}\gamma \left(1 - 2\Phi\left(-\gamma\right) \right)\\
&=&  \exp\{-\gamma^2/2\}\{1 + \frac{1}{2}\gamma \left[ \gamma + \frac{ \gamma^3}{3} + \frac{ \gamma^5}{3\cdot 5} + \cdots + \frac{ \gamma^{2n+1}}{(2n+1)!!} + \cdots\right]\}\\
&=& 1 + \frac{3}{2}\gamma^2 + \frac{1}{24} \gamma^4   - \frac{ \gamma^6}{120}  + O(\gamma^8).
\end{eqnarray}
When $\gamma<1,$ 
\[
 (\frac{3}{2} + \frac{1}{24}) \gamma^2 \geq  \mathcal{R}(\gamma) -1\geq  \frac{3}{2}\gamma^2.
\]
When $\gamma<M$ for $M>1$, this lemma can be generalized:
there exists constants $C'\leq C$,
\[
C' \gamma^2 \leq \mathcal{R}(\gamma)-1 \leq C \gamma^2.
\]
 \end{lemma}
 
 \subsection{Technical Tool: Folded Gaussian Mean Formula }
\paragraph{Proof for folded Gaussian mean}
\begin{proof}[Proof of lemma~\ref{lem:FoldedGaussian}]
Plug in the formula for mean of folded Gaussian \ref{FoldedGaussianMean}, 
we have
\[
E_G |\mu+ \sigma G| = \sqrt{\frac{2}{\pi}} \sigma \exp\{-\gamma^2/2\} + \mu \left(1 - 2\Phi\left(-\gamma\right) \right).
\]
Divide it by 
\[
E_G |\sigma G| =   \sqrt{\frac{2}{\pi}} \sigma,
\]
we get the first equality.

Since the CDF of the standard normal distribution can be expanded by integration by parts into a series:
\[
\Phi( \gamma)=\frac{1}{2} + \frac{1}{\sqrt{2\pi}}\cdot e^{- \gamma^2/2} \left[ \gamma + \frac{ \gamma^3}{3} + \frac{ \gamma^5}{3\cdot 5} + \cdots + \frac{ \gamma^{2n+1}}{(2n+1)!!} + \cdots\right],
\]
where $!!$ denotes the double factorial. 

\[
1 - 2\Phi(-\gamma)  = \frac{1}{\sqrt{2\pi}}\cdot e^{- \gamma^2/2}\left[ \gamma + \frac{ \gamma^3}{3} + \frac{ \gamma^5}{3\cdot 5} + \cdots + \frac{ \gamma^{2n+1}}{(2n+1)!!} + \cdots\right].
\]
This gives the second inequality.

Additionally,
 \[
 \begin{array}{ll}
  e^{- \gamma^2/2} &=\sum_{n=0}^{\infty} \frac{ \gamma^{2n}}{(-2)^n (n)!}\\
  &=  1- \frac{ \gamma^2}{2} + \sum_{n=2}^{\infty} \frac{ \gamma^{2n}}{(-2)^n (n)!},
\end{array}
  \] 
then
 \[
 \begin{array}{ll}
& \exp\{-\gamma^2/2\}\{1 + \frac{1}{2}\gamma \left[ \gamma + \frac{ \gamma^3}{3} + \frac{ \gamma^5}{3\cdot 5} + \cdots + \frac{ \gamma^{2n+1}}{(2n+1)!!} + \cdots\right]\}\\
=& (\sum_{n=0}^{\infty} \frac{ \gamma^{2n}}{(-2)^n (n)!})\{1 + \frac{1}{2}\gamma \left[ \sum_{n=0}^{\infty} \frac{ \gamma^{2n+1}}{(2n+1)!!} \right]\}\\

=& 1 + \frac{3}{2}\gamma^2 + \frac{1}{24} \gamma^4   - \frac{ \gamma^6}{120}  + O(\gamma^8). 
\end{array}
  \] 
  
Let $C_\Phi( \gamma^2)$ be a function of $ \gamma$, then it is a composition with the inner function being $ \gamma^2,$ defined as follows: 
\[
 \begin{array}{ll}
C_\Phi( \gamma^2) 
&:= (2\Phi( \gamma)-1)/\left(\frac{ \gamma}{\sqrt{2\pi}}\right) \\
&= e^{- \gamma^2/2} \left[1 + \frac{ \gamma^2}{3} + \frac{ \gamma^4}{3\cdot 5} + \cdots + \frac{ \gamma^{2n}}{(2n+1)!!} + \cdots\right]\\
&= [\sum_{n=0}^{\infty} \frac{ \gamma^{2n}}{(2n+1)!!} ]/[\sum_{n=0}^{\infty} \frac{ \gamma^{2n}}{2^n (n)!}] \\
&= [\sum_{n=0}^{\infty} \frac{ \gamma^{2n}}{(2n+1)!!} ]/[\sum_{n=0}^{\infty} \frac{ \gamma^{2n}}{(2n)!!}] \\
&\leq 1.
\end{array}
\]
We remark further that
\[
 \begin{array}{ll}
C_\Phi( \gamma^2) 
&= [\sum_{n=0}^{\infty} \frac{ \gamma^{2n}}{(2n+1)!!} ]/[\sum_{n=0}^{\infty} \frac{ \gamma^{2n}}{(2n)!!}] \\
&= [1 + \frac{ \gamma^2}{3}+ +  \frac{ \gamma^4}{15}+ O( \gamma^6) ]/[1 + \frac{ \gamma^2}{2}+ +  \frac{ \gamma^4}{8}+O( \gamma^6)] \\
&= 1 -\frac{ \gamma^2}{6}+  \frac{ \gamma^4}{40}+ O( \gamma^6).
\end{array}
\]
Then
 \[
 \begin{array}{ll}
\mu \left(1 - 2\Phi\left(-\gamma\right) \right)
&= \frac{1}{\sqrt{2\pi}}\cdot \mu^2 \frac{1}{\sigma} C_\Phi(\gamma^2)\\
&= \frac{1}{\sqrt{2\pi}}\cdot \mu^2 \frac{1}{\sigma} (1-\gamma^2/6 +O(\gamma^4)).\\
\end{array}
\] 
Combining the above:
\[
 \begin{array}{ll}
E_G |\mu+ \sigma G|
&=   \left[ \sqrt{\frac{2}{\pi}} \sigma \exp\{-\gamma^2/2\} + \frac{1}{\sqrt{2\pi}}\cdot \mu^2 \frac{1}{\sigma} C_\Phi(\gamma^2)\right]\\
&=   \left[ \sqrt{\frac{2}{\pi}} \sigma \{ 1- \frac{\gamma^2}{2} + \sum_{n=2}^{\infty} \frac{\gamma^{2n}}{(-2)^n (n)!}\} + \frac{1}{\sqrt{2\pi}}\cdot \mu^2 \frac{1}{\sigma} C_\Phi(\gamma^2)\right]\\
&= \sqrt{\frac{2}{\pi}}   \left[ \sigma 
\{ 1 + (1+\frac{1}{2}\cdot C_\Phi(\gamma^2))\gamma^2 +   \sum_{n=2}^{\infty} \frac{\gamma^{2n}}{(-2)^n (n)!}\} \}\right].\\
\end{array}
\] 
Define the residual of this expansion as
\[
 \begin{array}{ll}
\mbox{Res}(\gamma^2) 
&:= \frac{ (2\Phi( \gamma)-1)/\left(\frac{ \gamma}{\sqrt{2\pi}}\right)-1}{2}\cdot \gamma^2 +  (\exp\{-\gamma^2/2\} - 1 + \gamma^2/2)
\\
&= \frac{ C_\Phi(\gamma^2)-1}{2}\cdot \gamma^2 +  (\exp\{-\gamma^2/2\} - 1 + \gamma^2/2) \\
&=    \frac{ C_\Phi(\gamma^2)-1}{2}\cdot \gamma^2  +   \{ \sum_{n=2}^{\infty} \frac{\gamma^{2n}}{(-2)^n (n)!}\} \} \\
&=    \frac{-\gamma^4 }{12}  +    \frac{ \gamma^6}{80} +\frac{\gamma^{4}}{(-2)^2 (2)!} +\frac{\gamma^{6}}{(-2)^3 (3)!}+
O(\gamma^8)\\
&=    \frac{\gamma^4 }{24}  - \frac{ \gamma^6}{120}  + O(\gamma^8).
\end{array}
\] 
We know that if $\gamma^2<1,$ 
\[
 \begin{array}{ll}
\mbox{Res}(\gamma^2) 
&=    \frac{\gamma^4 }{24}  - \frac{ \gamma^6}{120}  + O(\gamma^8)\\
&\geq 0.
\end{array}
\] 

When $\gamma<1,$ 
\[
 (\frac{3}{2} + \frac{1}{24}) \gamma^2 \geq  \mathcal{R}(\gamma) -1\geq  \frac{3}{2}\gamma^2.
\]
When $\gamma<M$ for $M>1$, this lemma can be generalized:
there exist constants $C'\leq C$,
\[
C' \gamma^2 \leq \mathcal{R}(\gamma)-1 \leq C \gamma^2.
\]
\end{proof}



  
  
  
 





\subsection{Examples}
In the following, we study the simple case when there is only one root.
Let $s\in (-1,1)$. For $a=(\ldots, 0, 1, -s, 0, \ldots)$, $a^{-1} = (\ldots, 0, 1, s, s^2, s^3, \ldots)$, we could construct 
\[
w^{r,1}:=  (\ldots, 0, 1, s, s^2, s^3, \ldots, s^r)
\]

For $\gamma\in (0,1)$,
\[
\bar a^{\gamma} :=e_0-\gamma a= (\ldots, 0, 1-\gamma,  \gamma s, 0, \ldots)
\]
\[
 \begin{array}{ll}
w^{r,\gamma} 
&= a^{-1} - a^{-1}*_{r+1} (\bar a^{\gamma})
\end{array}
\]
\[
\begin{array}{ll}
a*(a^{-1}-w^{r,\gamma}) 
&= *_{r+1} (\bar a^{\gamma})\\
&= *_{r+1} (\ldots, 0, 1-\gamma,  \gamma s, 0, \ldots)
\end{array}
\]

\[
\mathcal{F}[(\ldots, 0, 1-\gamma, \gamma s, 0, \ldots)](t)  = (1-\gamma)+   \gamma s  \cdot e^{2\pi i  t}, \quad  t\in T = [\frac{-1}{2}, \frac{1}{2}];
\]

\[
|\mathcal{F}[(\ldots, 0, 1-\gamma, \gamma s, 0, \ldots)] |^2(t) = (1-\gamma)^2+ (\gamma s )^2+ 2 {(1-\gamma)} {\gamma}s \cos{(2\pi  t)} , \quad t\in T = [\frac{-1}{2}, \frac{1}{2}];
\]

\[
 \begin{array}{ll}
\|*_{r+1} (\bar a^{\gamma})\|_2^2 
&= \| (\hat{\bar a}^{\gamma})^{r+1}\|_2^2\\
&=  \int_{\frac{-1}{2}}^\frac{1}{2}  |(1-\gamma)+   \gamma s  \cdot e^{2\pi i  t}|^{2r+2} dt\\
&= \int_{\frac{-1}{2}}^\frac{1}{2}  [(1-\gamma)^2+ (\gamma s )^2+ 2 {(1-\gamma)} {\gamma}s \cos{(2\pi  t)} ]^{r+1} dt
\end{array} \]
Notice the following lemma, 
\begin{lemma}
For odd number, $\int_{\frac{-1}{2}}^\frac{1}{2}  \cos^{2k-1}{(2\pi  t)}  dt = 0$ for any $k\in {\mathbb Z}_+$; for even number, 
\[
\int_{\frac{-1}{2}}^\frac{1}{2}  \cos^{2k}{(2\pi  t)}  dt =  {2k\choose k} 2^{-2k},
\]
\end{lemma}
\begin{proof}
\[
 \begin{array}{ll}
2^n\cos^n (2\pi  t)=(e^{i \cdot 2\pi  t}+e^{-i \cdot 2\pi  t})^n=\sum_{k=0}^n {n\choose k} (e^{i\cdot 2\pi  t})^k\ (e^{-i\cdot 2\pi  t})^{n-k}=\sum_{k=0}^n {n\choose k} e^{(2k-n)i\cdot 2\pi  t},
\end{array} \]

\[
 \begin{array}{ll}
\int_{\frac{-1}{2}}^\frac{1}{2} e^{(2k-n)i\cdot 2\pi  t} dt= \delta_{2k-n}
\end{array} \]
\[
 \begin{array}{ll}
\int_{\frac{-1}{2}}^\frac{1}{2} 2^n\cos^n (2\pi  t) dt =\sum_{k=0}^n {n\choose k} \delta_{2k-n},
\end{array} \]
therefore, for odd number $n=2k-1$, $\int_{\frac{-1}{2}}^\frac{1}{2}  \cos^{2k-1}{(2\pi  t)}  dt = 0$ for any $k\in {\mathbb Z}_+$; for even number $n=2k$, 
\[
\int_{\frac{-1}{2}}^\frac{1}{2} 2^{2k} \cos^{2k}{(2\pi  t)}  dt =  {2k\choose k}.
\]
\end{proof}

\begin{lemma}
Now
$R_1 :=  (1-\gamma)^2+ (\gamma s )^2,$
$R_2 := (1-\gamma) \gamma s,$
\[
 \begin{array}{ll}
 \| (\hat{\bar a}^{\gamma})^{r+1}\|_2^2
=\sum_{k=0}^{\frac{r+1}{2}} [(1-\gamma)^2+ (\gamma s )^2]^{r+1-2k} [ (1-\gamma) \gamma s]^{2k}{2k\choose k}{r+1\choose 2k} 
\end{array} \]
\end{lemma}
\begin{proof}
Now define
$R_1 :=  (1-\gamma)^2+ (\gamma s )^2,$
$R_2 := {(1-\gamma)} {\gamma}s,$
\[
 \begin{array}{ll}
 \| (\hat{\bar a}^{\gamma})^{r+1}\|_2^2
&= \int_{\frac{-1}{2}}^\frac{1}{2}  [(1-\gamma)^2+ (\gamma s )^2+ 2 {(1-\gamma)} {\gamma}s \cos{(2\pi  t)} ]^{r+1} dt\\
&= \int_{\frac{-1}{2}}^\frac{1}{2}  
[R_1+ R_2 \cdot 2 \cos{(2\pi  t)} ]^{r+1} dt\\
&=\sum_{k=0}^{\frac{r+1}{2}} R_1^{r+1-2k} R_2^{2k}{2k\choose k}{r+1\choose 2k} 
\end{array} \]
\end{proof}

Use the Laplace's method theorem \ref{Laplace},
We could instead directly optimize  $\gamma$ over the $\ell_\infty$ norm of $|\hat{\bar a}^{\gamma}|^2$,
\begin{lemma}
\[
 \||\hat{\bar a}^{\gamma}|^2\|_\infty = \max_{ t\in [\frac{-1}{2}, \frac{1}{2}]}\{ (1-\gamma)^2 + (\gamma s)^2+ 2 (1-\gamma) \gamma s \cos{(2\pi  t)} \}  = (1-\gamma)^2 + (\gamma s)^2+ 2 |(1-\gamma) \gamma s| = (|1-\gamma|+|\gamma s|)^2 
\]
then

 \[
  \begin{array}{ll}
\gamma_\infty &= \arg\min_{\gamma>0}\||\hat{\bar a}^{\gamma}|^2\|_\infty \\
&= \arg\min_{\gamma>0}(|1-\gamma|+|\gamma s|)^2 \\
&= 1
 \end{array} \]
\[
\min_{\gamma\in (0,1)}\||\hat{\bar a}^{\gamma}|^2\|_\infty
= (|1-\gamma_\infty|+|\gamma_\infty s|)^2 
= |s|^2
\]
\end{lemma}
And
 \[
 \begin{array}{ll}
\|*_{r+1} (\bar a^{\gamma_\infty  })\|_2^2 
&=  s^{2r+2} 
\end{array} \]

If we instead choose $\gamma_2$ as $ \cos^2{\theta} = \frac{1}{1+s^2}$ to minimize the $\ell_2$ norm instead, then
\[
\|\hat{\bar a}^{\gamma_2}\|_2^2  = \min_{\gamma\in (0,1)}\|\hat{\bar a}^{\gamma}\|_2^2 
=  (1-\gamma_2)^2 + (\gamma_2 s)^2 
= \frac{s^2}{1+s^2}
\]
And in this case,
 \[
  \begin{array}{ll}
\||\hat{\bar a}^{\gamma_2}|^2\|_\infty
&= (|1- \frac{1}{1+s^2}|+| \frac{1}{1+s^2} s|)^2 \\
&=  s^2 (\frac{1+|s|}{1+s^2})^2 \in (s^2, 1)
 \end{array} \]
 
 Therefore, 
 \[
  \begin{array}{ll}
\||\hat{\bar a}^{\gamma_2}|^2\|_\infty \in (s^2, 1)
 \end{array} \]
\section{Supplementary: Background for Convex Blind Deconvolution Problem}
\subsection{Technical Background: Wiener's Lemma and Inverse Filter}
\paragraph{Fourier transform and inverse filter}
The discrete-time Fourier transform $\mathcal{F}$ is defined by $\mathcal{F} {\bf a} $ where
\[
 (\mathcal{F} {\bf a})(\omega) : = \sum_{n=-\infty}^\infty a_n\, e^{2\pi i n \omega}, \quad  \omega\in T = [-\frac{1}{2}, \frac{1}{2}];
\]
the inverse Fourier transform $\mathcal{F}^{-1}$ is defined by
\[
(\mathcal{F}^{-1} f)(n) =  \int_{-\frac{1}{2}}^\frac{1}{2} f(\omega) e^{-2\pi i n \omega}  d\omega, \quad n\in {\mathbb Z}.
\]
Now, our condition on the filter  ${\bf a}$ is:
\[a\in l_1({\mathbb Z}) ,  
(\mathcal{F} {\bf a})(\omega) \neq 0, \forall \omega\in T.
\]
In the following theorem, we show that this condition would provide the existence of an inverse filter in  $l_1({\mathbb Z})$.

First, we present the standard Wiener's lemma.
\begin{lemma}{\textbf{Wiener's lemma on periodic functions:}}
Assume that a function $f$ on unit circle has an absolutely converging Fourier series, and $f(t)\neq 0$ for all $t \in T$, then $1/f$ also has an absolutely convergent Fourier series.
\end{lemma}
Then we can see clearly that the Fourier series version of the previous lemma would guarantee the existence of an inverse filter in  $l_1({\mathbb Z})$.
\begin{lemma}{\textbf{Wiener's lemma on $l_1({\mathbb Z})$ sequences :}}
If  ${\bf a}\in l_1({\mathbb Z}) ,  
(\mathcal{F} {\bf a})(\omega) \neq 0, \forall \omega\in T,$ we could define the inverse filter of  ${\bf a}$ as
${\bf a}^{-1} := \mathcal{F}^{-1}(\frac{1}{\mathcal{F} {\bf a}})$
so that  ${\bf a}^{-1}*a = {\bf e}_0.$ Here $e_0$ is the sequence with $1$ at $0$ coordinate and $0$ elsewhere.
From Wiener's lemma on $\mathcal{F} {\bf a}$,    ${\bf a}^{-1} \in l_1({\mathbb Z})$ .
\end{lemma}

\subsection{Change of Variable and Reduction to Projection Pursuit}
Rewrite the population version of our convex sparse blind deconvolution problem, with the population objective 
${\mathbb E}\frac{1}{N} \| {\bf w}*Y\|_{\ell_1({\cal T})} ={\mathbb E}|({\bf w}*Y)_0|={\mathbb E}|({\bf w}*{\bf a}*X)_0|={\mathbb E}_X|\langle X, ({\bf w}*{\bf a})^{\dagger} \rangle|$ due to  the ergodic property of stationary process and shift invariance, and $(\widetilde{{\bf a}}*{\bf w})_0=((\widetilde{{\bf a}}*{\bf a}^{-1})*({\bf a}*{\bf w}))_0  = \langle (\widetilde{{\bf a}}*{\bf a}^{-1})^{\dagger}, {\bf w}*{\bf a}\rangle$, the convex problem becomes 
\[
 \begin{array}{ll}
 \underset{{\bf w}}{\mbox{minimize}}   & {\mathbb E}_X|\langle X ,  ({\bf a}*{\bf w})^{\dagger} \rangle|\\
\mbox{subject to}  
& \langle \widetilde{{\bf a}}*{\bf a}^{-1} , ({\bf a}*{\bf w})^{\dagger}\rangle =1,
\end{array}
 \]
Let $\psi$ denote  the time reversed version of ${\bf a}*{\bf w}$: $\psi := ({\bf a}*{\bf w})^{\dagger}$, and
  let $\widetilde{{\bf e}}:= \widetilde{{\bf a}}*{\bf a}^{-1}$, then by previous assumptions, 
  $\widetilde{{\bf e}}_0=1$, $\widetilde{{\bf e}}' = \widetilde{{\bf e}}-{\bf e}_0$.
  
 Now we arrive at a simple and fundamental population convex problem: 
 \[
 \begin{array}{ll}
 \underset{\psi}{\mbox{minimize}}   &{\mathbb E}_X|\langle X ,  \psi  \rangle|\\
\mbox{subject to}  
& \langle \widetilde{{\bf e}} ,  \psi  \rangle  =1.
\end{array}
  \]
  \textbf{Expectation using Gaussian.  }
Since $X$ follows Bernoulli-Gaussian IID probability model $X_t = I_t G_t$, we nest the expectation over $I_t$ outside the expectation over Gaussian $G_t$, for which we use $E |N(0,1)| = \sqrt{\frac{2}{\pi}}$:
\[
{\mathbb E}_X|\langle X ,  \psi  \rangle|
= {\mathbb E}_I {\mathbb E}_G {|\sum_{t\in {\mathbb Z}}  I_t G_t \psi(t)| }
= \sqrt{\frac{2}{\pi}} \cdot{\mathbb E}_I \|\psi\cdot I\|_2
\]

\subsection{Technical background: Directional Derivative and Projected Subgradient }
\paragraph{Exact calculation of subgradient for phase transition }
In this section, using sub-gradient and directional derivative, we compute the KKT condition of our problem rigorously.
\begin{lemma}
Let $J \subset [n]$ be the support of $X$, let $\psi_J : = \psi\cdot 1_J$ denote the elementwise product of vector $\psi$ with the indicator vector of subset $J$.  Let $B_J$ denote the central section of the euclidean  ball $B({\mathbb R}^n)$,
where the slice is produced the linear space $\mbox{span} \{e_i : i\in J \}$.
Alternatively, we may write
$B_J: = \{ v_J: \| v_J\|_2 \leq 1\}$.  The set-valued subgradient
operator applied to $\|\psi_J\|$
evaluates as follows:
 \[
 \begin{array}{ll}
\partial_{\psi}    \| \psi_J \|
&=  N(\psi_J).
\end{array}
  \]
Here $N$ maps ${\mathbb R}^n$ into  subsets of ${\mathbb R}^n$,
and is given by:
\[
N(\psi_J):= 
\begin{cases}
    \psi_J/\|\psi_J\|_2, &  \psi_J\neq 0 \\
    B_J, &  \psi_J = 0
  \end{cases}.
\]  
\end{lemma}

\begin{definition}
Consider a probability space containing just the possible  
outcomes  $J \subset [n]$.
Let $S_J$, $J \subset [n]$,  denote a closed compact subset of ${\mathbb R}^n$.
Let $I$ be a random subset of
$[n]$ drawn at random from this probability space with probability $\pi_J = P\{ I = J \}$
of elementary event $J$.
Consider the set-valued random variable $S \equiv S_J$. We define its expectation as
\[
 {{\mathbb E}} S  := \sum_J \pi_J \cdot S_J.
\]
On the right side, we mean the closure of the 
compact set produced by all sums of the form
\[
 \sum_J \pi_J \cdot s_J
\]
where each $s_J \in S_J$.
\end{definition}

\begin{lemma}
For any $\psi \in {\mathbb R}^n$, and $X = (X_i)_{i=1}^n$ with $X_i \sim_{iid} BG(p,0,1)$.
Let now $I \subset [n]$ be the random support of $X$.
Let $\partial_\psi $ denote the set-valued subgradient
operator.
  \[
 \begin{array}{ll}
\partial_\psi [E{|\sum_{i} \psi_i X_i| }  ]
&= \sqrt{\frac{2}{\pi}} \cdot {{\mathbb E}}_I [N(\psi_I)].
\end{array}
  \]
\end{lemma}

\paragraph{Subgradient and directional derivative at $e_0$}
Now we focus on $e_0$. Note that for any subset $J \subset [n]$, $({\bf e}_0)_J$ is either the zero
vector or else $e_0$. Hence $N(({\bf e}_0)_J)$
is either $B_{J}$ or $\{{\bf e}_0\}$. What drives
this dichotomy is whether $0 \in J$ or not.

\begin{lemma}  
\[
 \begin{array}{ll}
{\mathbb E}_I [N(({\bf e}_0)_I) ]
&= p {\bf e}_0 + \sum_{J,0\not\in J } \pi_{J} B_{J} \\
& = p {\bf e}_0 + (1-p) {\cal B}_0,
\end{array}
  \]
where
\[
 \begin{array}{ll}
{\cal B}_0 = {\mathbb E}_{I}[ B_{I} | 0 \not \in I ] = 
(1-p)^{-1} \cdot \sum_{J,0\not\in J } {\pi_{J}} B_{J}.
\end{array}
\]
 
\end{lemma}


\begin{proof}
\begin{eqnarray*}
{\mathbb E}_I [N(({\bf e}_0)_I) ] &=& \sum_{J } \pi_{J} B_{J} \\
&=&  \sum_{J, 0 \in J}   \pi_{J} {\bf e}_0 +  \sum_{J,0\not\in J }  \pi_{J} B_{J}\\
&=& p \cdot {\bf e}_0 + (1-p) \cdot {{\cal B}}_0 .
\end{eqnarray*}
 \end{proof}
To compute with ${\mathbb E}_I [N(({\bf e}_0)_I) ]$,
we need:
\begin{lemma} \label{eq:ComputeExpect}
For each fixed $\beta \in {\mathbb R}^n$:
\[
  \sup_{b \in {\cal B}_0} \langle \beta, b \rangle
   = {\mathbb E}_{I}[ \|\beta_I\|_2 | 0 \not \in I  ].
  \]
\end{lemma}

\begin{proof}
Note that in the definition of the set 
\[
{\cal B}_0 = {\mathbb E}_{I}[ B_{I} | 0 \not \in I ],
\]
 each term $B_J$ obeys the bound
$\sup_{g_J \in B_J} \|g_J\|_2 = 1 $.
Now, given the fixed vector $\beta \in {\mathbb R}^n$, define:
\begin{eqnarray*}
    b^* &=& {\mathbb E}_{I}[ \frac{\beta_I}{\| \beta_I \|_2} | 0 \not \in I  ]\\
     &=&  \sum_{0 \not \in J} \pi_J \frac{\beta_J }{\|\beta_J\|_2}.
 \end{eqnarray*}
 Since each term in this sum has Euclidean norm at most $1$, $b^* \in {{\cal B}}_0$.
Now
\begin{eqnarray*}
\langle  \beta, b^* \rangle &=&
\langle \beta , \sum_{0 \not \in J} \pi_J \frac{\beta_J }{\|\beta_J\|_2} \rangle \\
&=& \sum_{0 \not \in J} \pi_J \langle \beta , \frac{\beta_J }{\|\beta_J\|_2} \rangle \\
&=& \sum_{0 \not \in J} \pi_J \|\beta_J\|_2 .
 \end{eqnarray*}
On the other hand, for any fixed 
vector $g = \sum \pi_J g_J $,
with each $g_J \in B_J$,
we have
\begin{eqnarray*}
\langle  \beta, g \rangle &=&  \sum_{0 \not \in J} \pi_J \langle \beta , g_J  \rangle \\
&\leq& \sum_{0 \not \in J} \pi_J  \|\beta_J \|_2 \|g_J\|_2 \\
& \leq & \sum_{0 \not \in J} \pi_J  \|\beta_J \|_2 
= \langle  \beta, b^* \rangle.
 \end{eqnarray*}
\end{proof}

Lemma (\ref{eq:ComputeExpect}) 
can be viewed as a special instance of
Theorem \textbf{A.15} from \cite{bai2018subgradient}:
\begin{lemma}[Interchangeability of set expectation and support function] 
Suppose a random compact set $S \subset {\mathbb R}^n$ is integrably bounded and the underlying probability space is non-atomic, then ${\mathbb E}[S]$ is a convex set and for any fixed vector $\beta \in {\mathbb R}^n,$
 \begin{equation} \label{eq:SetExpectEquality}
 \sup_{g\in {\mathbb E} S} \langle\beta, g\rangle =  \sup_{g\in  S} {\mathbb E}\langle\beta, g\rangle .
 \end{equation}
\end{lemma}

Define $\beta_{(0)} = \beta \cdot 1_{\{0\}^c}$ as the part of $\beta$ supported away from $0$.







\section{ Main Result $1$ and Its Proof: Phase Transition}
\subsection{KKT Condition for Exact Recovery}
\paragraph{KKT condition for ${\bf e}_0$ to be optimal solution.}
Given the tool defined above, we can calculate KKT rigorously. We first state the overview.

Let $\psi^\star$ be the solution of the optimization problem:
\[
 \begin{array}{ll}
 \underset{\psi}{\mbox{minimize}}   & {\mathbb E}_I \|\psi\cdot I\|_2\\
\mbox{subject to}  
& \langle   \widetilde{{\bf e}}, \psi\rangle =1
\end{array}
\tag{$Q_1(\widetilde{{\bf e}})$ }
  \]
  
We claim that to prove that ${\bf e}_0$ solves ($Q_1(\widetilde{{\bf e}})$), we calculate the directional finite difference at ${\bf e}_0$. Then ${\bf e}_0$ solves this convex problem if the directional finite difference at $\psi = {\bf e}_0$ is non-negative at every direction $\beta$ on unit sphere where $\widetilde{{\bf e}}^T \beta =0$:
	 \[
	 {\mathbb E}\|({\bf e}_0+\beta)\cdot I\|_2 - {\mathbb E}\|({\bf e}_0)\cdot I\|_2 \geq 0.
	 \]
We decompose the objective conditioning on whether $I_0 = 1_{\{X_0\neq 0\}}$ is zero or not:
	 \[
	 \begin{array}{ll}
	    {\mathbb E}\|({\bf e}_0+\beta)\cdot I\|_2 - {\mathbb E}\|({\bf e}_0)\cdot I\|_2
	    &= p (1+\beta_0)  + (1-p) \nabla_{\beta}{{\mathbb E}_I[\|({\bf e}_0+\beta)'\|_{\ell_2(I-\{0\})} \mid I_0 =0]} -p\\
	    &= p \beta_0  + (1-p) {\mathbb E}_{I'}[\|\beta'\|_{\ell_2(I')} ].
	 \end{array}
	\]
This will be non-negative in case either $\beta_{0} > 0$, or else $\beta_0 < 0$ but
	\[
	\frac{p}{1-p} \leq  \frac{{\mathbb E}_{I'}\|\beta'\|_{\ell_2(I')}}{ |\beta_0|}
	\]
	for all $\beta $ that satisfy $ \widetilde{{\bf e}}^T \beta =0$.
	
	In the following, we rigorously prove the last two claims this KKT condition using calculating directional derivative.
	\paragraph{KKT condition in the form of directional derivative and projected subgradient}
\begin{lemma}[Equivalent forms of KKT condition]
For the following optimization problem 
\[
 \begin{array}{ll}
 \underset{\psi}{\mbox{minimize}}   & {\mathbb E}_I \|\psi\cdot I\|_2\\
\mbox{subject to}  
& \langle   \widetilde{{\bf e}}, \psi\rangle =1
\end{array}
\tag{$Q_1(\widetilde{{\bf e}})$ }
  \]
Let $P_{\widetilde{{\bf e}}}^{\perp}$ be the projection onto the hyperplane as the orthogonal complement of  $\widetilde{{\bf e}}$. The following are equivalent forms of KKT condition for ${\bf e}_0$ to be the optimal solution:   
  \begin{itemize}
  \item
  The directional derivative at $ {\bf e}_0$ along every direction $\beta$ on unit sphere where $\widetilde{{\bf e}}^T \beta =0$ is non-negative:
	 \[
	\lim_{t\rightarrow 0^+} \frac{1}{t}[{\mathbb E}\|({\bf e}_0+t\beta)\cdot I\|_2 - {\mathbb E}\|({\bf e}_0)\cdot I\|_2] \geq 0.
	 \]
\item
  \[
 \begin{array}{ll}
0\in P_{\widetilde{{\bf e}}}^{\perp} {\mathbb E}_I [N(({\bf e}_0)_I) ].
\end{array}
  \]
  \item
Equivalently, there exists a subgradient $g\in  P_{\widetilde{{\bf e}}}^{\perp} {\mathbb E}_I [N(({\bf e}_0)_I) ]$ such that for all $\beta$ satisfying  $\widetilde{{\bf e}}^T \beta = 0$,
\[
\beta^T g \geq 0.
\]
\item
Also equivalently, for all $\beta$,
\[
 \begin{array}{ll}
\sup_{g\in  [N(({\bf e}_0)_I)]}{\mathbb E}_I[ \langle   P_{\widetilde{{\bf e}}}^{\perp}  \beta, g\rangle] \quad \geq 0.
\end{array}
  \]
  \end{itemize}
\end{lemma}

\paragraph{KKT condition in directional derivative}
The following upper bound $ 1- \frac{|\widetilde{{\bf e}}|_{(2)}}{|\widetilde{{\bf e}}|_{(1)}}$ of $p^\star$ generalized the previous special case of exponential decay filter in theorem~\ref{thm: single root PT} with $p^\star = 1-|s|$:
\begin{lemma}
 The directional derivative at $e_0$ along $\beta$ 
 evaluates to the following:
 \begin{eqnarray*}
 \lim_{t\rightarrow 0^+}\frac{1}{t}({\mathbb E}_I [\|({\bf e}_0+t\beta)_I\|_2 - \|({\bf e}_0)_I\|_2]) 
 &=&  p \cdot \langle\beta, {\bf e}_0\rangle + (1-p) \cdot {\mathbb E}_{I}[ \|\beta_{I}\|_2 | 0 \not \in I ]
 \end{eqnarray*}
\end{lemma}
 \begin{proof}


Applying Lemma \ref{eq:ComputeExpect}, we proceed as follows:
\[
 \begin{array}{ll}
\lim_{t\rightarrow 0^+}\frac{1}{t}({\mathbb E}_I [\|({\bf e}_0+t\beta)_I\|_2 - \|({\bf e}_0)_I\|_2])
&= \sup_{g\in {\mathbb E}_I [N(({\bf e}_0)_I)]} \langle\beta, g\rangle\\
&= p \cdot \langle\beta, {\bf e}_0\rangle + (1-p) \cdot  \sup_{g\in {\cal B}_0 } \langle\beta, g\rangle\\

&= p \cdot \langle\beta, {\bf e}_0\rangle + (1-p) \cdot {\mathbb E}_{I}[ \|\beta_{I}\|_2 | 0 \not \in I ] 
\end{array}
  \]
  \end{proof}
  
  
\subsection{Formula for Phase Transition Parameter}  
\paragraph{Reduction to $val(Q_1 (\widetilde{{\bf e}}')).$}	
	We normalize the direction sequence $\beta$ so that $\beta_0= -1$; 
	using $\tilde{e}(0)=1$, we obtain a lower bound:
\[
 \inf_{\beta_0=-1, \langle  \widetilde{{\bf e}}, \beta\rangle  =0} {\mathbb E}_{I'}\|\beta'\|_{\ell_2(I')}
 =\inf_{ \beta_0=-1, \beta_0 \tilde{e}_0-\langle  \widetilde{{\bf e}}', \beta'\rangle  =0} {\mathbb E}_{I'}\|\beta'\|_{\ell_2(I')}
= \inf_{\langle   \widetilde{{\bf e}}', \beta'\rangle  =1} {\mathbb E}_{I'} \|\beta'\|_{\ell_2(I')} = val(Q_1 (\widetilde{{\bf e}}'))
\]	
Here $Q_1(\widetilde{{\bf e}}')$ is the optimization problem:
\[
 \begin{array}{ll}
 \underset{\beta \in l_1({\mathbb Z})}{\mbox{minimize}}   &{\mathbb E}_{I'} \|\beta'\|_{\ell_2(I')}\\
\mbox{subject to}  
& \langle  \widetilde{{\bf e}}', \beta'\rangle  =1
\end{array}
  \]
  
  Now we have rigorously proved that there exists a threshold $p^\star > 0$, so that for all 
\begin{itemize}
\item  ${\bf w}^\star$ is ${\bf a}^{-1}$ up to time shift and rescaling provided $p<p^\star$; and
\item   ${\bf w}^\star$ is not ${\bf a}^{-1}$ up to time shift and rescaling, provided $p> p^\star$.
\end{itemize}
The threshold $p^\star$ obeys
 \[
\frac{p}{1-p}= val(Q_1(\widetilde{{\bf e}}')).
\] 
  
  \section{Supplementary: Tight Upper and Lower Bound of Phase Transition Parameter}
  We have shown the existence of  $p^\star$ so that for all $p<p^\star$, the KKT condition is satisfied.   The threshold $p^\star$ determined by
 \[
\frac{p}{1-p}= val(Q_1(\widetilde{{\bf e}}')).
\]
We have represented $p^\star$ as the optimal value of a derived optimization problem $val(Q_1 (\widetilde{{\bf e}}'))$. From now on, we find upper and lower bounds of it.
\subsection{Upper and Lower Bound from Optimization Point of View}
	\begin{lemma}[Explicit phase transition condition with upper bound]
	\label{lemma:Condition phase transition upper}
$val(Q_1(\widetilde{{\bf e}}'))$
obeys an upper bound and lower
\[
 \frac{p}{\|\widetilde{{\bf e}}'\|_\infty}  \geq val(Q_1(\widetilde{{\bf e}}'))  
\]
where $\|\widetilde{{\bf e}}'\|_\infty  = \frac{|\widetilde{{\bf e}}|_{(2)}}{|\widetilde{{\bf e}}|_{(1)}}  $.
	Additionally, the upper bound 
is sharp if and only if
	\[
	\frac{p}{1-p} \leq val(Q_1(\frac{\widetilde{{\bf e}}'}{\|\widetilde{{\bf e}}'\|_\infty}))=val(Q_1(\widetilde{{\bf e}}''))/\|\widetilde{{\bf e}}'\|_\infty
	\]
	therefore, the upper bound 
holds with equality 
	\[
	p^\star= 1- \frac{|\widetilde{{\bf e}}|_{(2)}}{|\widetilde{{\bf e}}|_{(1)}}
	\]
if 
		\[
	p\leq 1- \frac{|\widetilde{{\bf e}}|_{(3)}}{|\widetilde{{\bf e}}|_{(2)}}
	\]
	Therefore, if 
		\[
	\frac{|\widetilde{{\bf e}}|_{(3)}}{|\widetilde{{\bf e}}|_{(2)}} \leq \frac{|\widetilde{{\bf e}}|_{(2)}}{|\widetilde{{\bf e}}|_{(1)}}
	\]
	then
	\[
	p^\star= 1- \frac{|\widetilde{{\bf e}}|_{(2)}}{|\widetilde{{\bf e}}|_{(1)}}.
	\]
	
	\end{lemma}

	\begin{lemma}[Explicit phase transition condition with lower bound]
\label{lemma:Condition phase transition lower}
\[
 val(Q_1(\widetilde{{\bf e}}')) \geq  {\mathbb E}_I\frac{1}{\|\widetilde{{\bf e}}\cdot I\|_{2} }\geq p^{-1/2} \|\widetilde{{\bf e}}'\|_{2}^{-1}=p^{-1/2} \cot{\angle( \widetilde{{\bf e}}, {\bf e}_0)}
\]
	\end{lemma}


  	\begin{proof}[Proof of Lemma~\ref{lemma:Condition phase transition upper}]
	
 Here $Q_1(\widetilde{{\bf e}}')$ is the optimization problem:
\[
 \begin{array}{ll}
 \underset{\beta \in l_1({\mathbb Z})}{\mbox{minimize}}   &{\mathbb E}_{I'} \|\beta'_{I'}\|_{2}\\
\mbox{subject to}  
& \langle  \widetilde{{\bf e}}', \beta'\rangle  =1
\end{array}
  \]
  
The upper bound is achieved at $\beta = e_{i_m}/|\widetilde{{\bf e}}'_{i_m}| =e_{i_m}/\|\widetilde{{\bf e}}'\|_\infty $, where $i_m = \arg\max_i |\widetilde{{\bf e}}'|$.

The upper bound is tight (takes equality) if 
the projection pursuit problem $Q_1(\widetilde{{\bf e}}')$  lead to one-sparse solution
$\beta = e_{i_m}/|\widetilde{{\bf e}}'_{i_m}| =e_{i_m}/\|\widetilde{{\bf e}}'\|_\infty $.
Using the previous condition, it require
\[
	val(Q_1(\frac{\widetilde{{\bf e}}''}{\|\widetilde{{\bf e}}'\|_\infty}))\geq \frac{p}{1-p}
	\]
	therefore, the upper bound 
holds with equality 
	\[
	p^\star= 1- \frac{|\widetilde{{\bf e}}|_{(2)}}{|\widetilde{{\bf e}}|_{(1)}}
	\]
if 
		\[
	p\leq val(Q_1(\frac{\widetilde{{\bf e}}''}{\|\widetilde{{\bf e}}'\|_\infty}))
	\]
	To simplify with upper bound on $val(Q_1(\frac{\widetilde{{\bf e}}''}{\|\widetilde{{\bf e}}'\|_\infty}))$, if 
		\[
	\frac{|\widetilde{{\bf e}}|_{(3)}}{|\widetilde{{\bf e}}|_{(2)}} \leq \frac{|\widetilde{{\bf e}}|_{(2)}}{|\widetilde{{\bf e}}|_{(1)}}
	\]
	then
	\[
	p^\star= 1- \frac{|\widetilde{{\bf e}}|_{(2)}}{|\widetilde{{\bf e}}|_{(1)}}.
	\]
		\end{proof}



	
		\begin{proof}[Proof of Lemma~\ref{lemma:Condition phase transition lower}]
In general, we define $Q_2(\widetilde{{\bf e}}_J)$ for a fixed subset $J$:
\[
 \begin{array}{ll}
 \underset{\beta}{\mbox{minimize}}   &\|\beta\|_{2}\\
\mbox{subject to}  
&  \langle  \widetilde{{\bf e}}_{J}, \beta\rangle  =1
\end{array}
\tag{$Q_2(\widetilde{{\bf e}}_{J})$ }
  \] 	
then $val(Q_2(\widetilde{{\bf e}}_{J})) = \|\widetilde{{\bf e}}_J\|_2^{-1}$, where the optimal is achieved when 
  \[
  \beta_J = \widetilde{{\bf e}}_J/\|\widetilde{{\bf e}}_J\|_2^2 
  \]
then $val(Q_1 )$ has an lower bound:
\[
  val(Q_1(\widetilde{{\bf e}}') ) \geq {\mathbb E}_I val(Q_2(\widetilde{{\bf e}}'_I)) ={\mathbb E}_I\|\widetilde{{\bf e}}_I\|_{2}^{-1} \geq p^{-1/2} \|\widetilde{{\bf e}}'\|_{2}^{-1} 
  \]
  The last inequality is based on $V_{-1}\geq p^{-1/2}$ from theorem~\ref{thm:TaylorSqrtInvExpect}.
	\end{proof}
  \subsection{Upper and Lower Bound from Geometric Point of View}
  \paragraph{Geometric bound}
Let $\theta = \angle({\widetilde{{\bf e}}, {\bf e}_0})$, for $\beta$ need to satisfy a constraint  $\widetilde{{\bf e}}^T \beta =0$, we get
\[
 \begin{array}{ll}
\lim_{t\rightarrow 0^+}\frac{1}{t}({\mathbb E}_I [\|({\bf e}_0+t\beta)_I\|_2 - \|({\bf e}_0)_I\|_2])
&= [p \cos{\angle(\beta, {\bf e}_0)} + (1-p)  V_1(\beta_{(0)}) \sin{\angle(\beta, {\bf e}_0)}] \|\beta\|_2.
\end{array}
  \]
Here $\angle(\beta, {\bf e}_0)\in [0, \pi], \sin{\angle(\beta, {\bf e}_0)}\in [0,1]$. 

\begin{figure}[!tb]
\centering
\includegraphics[width=.55\textwidth]{./figures_BD/Angle_Diagram_label.png}
\caption{Demonstration of the relation between
  $\angle( \beta, -{\bf e}_0)$ and ${\angle(\widetilde{{\bf e}}, {\bf e}_0)}$. In the figure, $u=\widetilde{{\bf e}}/\| \widetilde{{\bf e}}\|_2$.}
\label{fig-Angle}
\end{figure}
  
\begin{lemma}
  \label{Angle_Inequality}
   \[
  \angle(  \beta, - {\bf e}_0) \geq \angle(P_{\widetilde{{\bf e}}}^\perp (-{\bf e}_0), -{\bf e}_0)= \frac{\pi}{2} - \theta
  \]
  
    \[
  \tan(\angle(  \beta, - {\bf e}_0)) \geq \tan\angle(P_{\widetilde{{\bf e}}}^\perp (-{\bf e}_0), -{\bf e}_0) =\cot{\theta}
  \]
\end{lemma}
  
\begin{proof}
   
We know that geometrically, using the property that projection of $e_0$ on the hyperplane with normal vector  $\widetilde{{\bf e}}$ has the smallest angle among all the $\beta$ in that hyperplane, we have that if
   \[
  \angle(  \beta, - {\bf e}_0) \geq \angle(P_{\widetilde{{\bf e}}}^\perp (-{\bf e}_0), -{\bf e}_0) = \frac{\pi}{2} - \angle(\widetilde{{\bf e}}, {\bf e}_0) = \frac{\pi}{2} - \theta,
  \]
then
  \[
  \tan\angle(  \beta, - {\bf e}_0) \geq \tan\angle(P_{\widetilde{{\bf e}}}^\perp (-{\bf e}_0), -{\bf e}_0)= \cot{\theta}.
  \]
 
\end{proof}
  
  


 
  
  
 
  
\begin{theorem}
 \label{thm:PT_lower}
The threshold $p^\star$ satisfies 
		\[ 
 p \cot\angle({\widetilde{{\bf e}}, {\bf e}_0})   \leq val(Q_1(\widetilde{{\bf e}}')) \leq	  \cot\angle({\widetilde{{\bf e}}, {\bf e}_0})  V_1(\widetilde{{\bf e}}').
		\]
\end{theorem}
 

\begin{proof}
  First, we prove the lower bound. Let $H_u=\{\beta : \quad u^T \beta =0 \}$,
    \begin{eqnarray*}
      val(Q_1(\widetilde{{\bf e}}')) &=&  \inf_{\beta\in H_u} [V_1(\beta_{(0)}) \tan\angle(  \beta, - {\bf e}_0)]\\
      &\geq& \inf_{\beta\in H_u} V_1(\beta_{(0)}) \inf_{\beta\in H_u}\tan(\angle(\beta, -{\bf e}_0))\\
      &=& p \tan{(\angle(P_{\widetilde{{\bf e}}}^\perp (-{\bf e}_0), -{\bf e}_0))}\\
       &=& p \cot\angle({\widetilde{{\bf e}}, {\bf e}_0}) 
  \end{eqnarray*}
 
Moreover, the lower bound is achieved when $\widetilde{{\bf e}}'$ is one-sparse.  
 
For the upper bound, we plug in $\beta = P_{\widetilde{{\bf e}}}^\perp (-{\bf e}_0)$, then
	\[ 
	\inf_{\| \beta\|_2=1, u^T \beta =0} 
	\tan(\angle( \beta, -{\bf e}_0))V_1(\beta_{(0)}) \leq
	\tan(\angle( P_{\widetilde{{\bf e}}}^\perp (-{\bf e}_0), -{\bf e}_0))
	V_1(P_{\widetilde{{\bf e}}}^\perp (-{\bf e}_0)).
	\]
It is worth commenting that since $V_1(P_{\widetilde{{\bf e}}}^\perp (-{\bf e}_0))\leq \sqrt{p}$, and $\tan(\angle( P_{\widetilde{{\bf e}}}^\perp (-{\bf e}_0), -{\bf e}_0))=\cot(\angle(\widetilde{{\bf e}}, {\bf e}_0))=\cot\theta$, we have
	\[ 
	 \tan(\angle( P_{\widetilde{{\bf e}}}^\perp (-{\bf e}_0), -{\bf e}_0))
 V_1(P_{\widetilde{{\bf e}}}^\perp (-{\bf e}_0)) 
	\leq \cot(\angle(\widetilde{{\bf e}}, {\bf e}_0)) \sqrt{p}.
	\]
 
  \end{proof}
 \subsection{Tighter Upper and Lower Bound from Refined Analysis} 
 \paragraph{Optimality by support}
Assume that $\widetilde{{\bf e}}$ has $n_e$ non-zero entry on support $S_{\widetilde{{\bf e}}}$, we can rank the absolute value of entries of $\widetilde{{\bf e}}$ to be $|\widetilde{{\bf e}}|_{(1)}, |\widetilde{{\bf e}}|_{(2)}, |\widetilde{{\bf e}}|_{(3)},\ldots,|\widetilde{{\bf e}}|_{(n_e)}$, then the entries of $\widetilde{{\bf e}}'$ will be ranked as $|\widetilde{{\bf e}}|_{(2)}, |\widetilde{{\bf e}}|_{(3)},\ldots,|\widetilde{{\bf e}}|_{(n_e)}$. 

We know the optimal solution of $(\beta')^\star$ of $\inf_{\langle   \widetilde{{\bf e}}', \beta'\rangle  =1} {\mathbb E}_{I'} \|\beta'\|_{\ell_2(I')}$ must have support $S_{\beta^\star}$ that satisfy $S_{\star}\subset S_{\widetilde{{\bf e}}'}$. From the symmetry of objective, we know if $(\beta')^\star$ is $m$ sparse, then $m\leq n_e-1$ and, its support must be on the top $m$ entries $|\widetilde{{\bf e}}|_{(2)}, |\widetilde{{\bf e}}|_{(3)},\ldots,|\widetilde{{\bf e}}|_{(m+1)}$, we call this support $S_m$, and we know the corresponding entries of $(\beta')^\star$ would have the same sign as entries of $\widetilde{{\bf e}}$. 

We can define the $m$ sparse optimization problem for a random support function on the subset of $S_m$: $J_m \subset S_m$. Let $\beta$ be supported on $S_m$ and each entry non-negative, then
\[
val(Q_1^{S_m}) :=\inf_{ \beta: \sum_{j=1}^{m}  |\widetilde{{\bf e}}|_{(j+1)} \beta_j  =1, \beta_j> 0} {\mathbb E}_{J_m} \|\beta \cdot J_m\|_{2} 
\]
Let $z_j:= \beta_j |\widetilde{{\bf e}}|_{(j+1)}$, from the symmetry of objective and the order on $|\widetilde{{\bf e}}|_{(j+1)}$, we know the solution must satisfy $0<z_m \leq z_{m-1} \leq \ldots \leq z_1$. Then we can recast the optimization problem as 
\[
val(Q_1^{S_m}) := 
\inf_{ z: \sum_{j=1}^{m}  z_j=1, 0<z_m \leq z_{m-1} \leq \ldots \leq z_1 } {\mathbb E}_{B} \sqrt{\sum_{j}B_j \frac{ 1}{|\widetilde{{\bf e}}|^2_{(j+1)}} z_j^2 } 
\]
As a special case, when $m=1$, $val(Q_1^{S_1}) = \frac{p}{|\widetilde{{\bf e}}|_{(2)}}$ as discussed above.

Then
\[
val(Q_1(\widetilde{{\bf e}}'))= \inf_{ m\in \{1,2,\ldots, n_e-1\}}val(Q_1^{S_m})
\]
\paragraph{Tighter upper bound on $val(Q_1)$}
Now we can prove a more refine upper bound:
\begin{lemma}[Tighter upper bound on $val(Q_1)$]
    \[
val(Q_1(\widetilde{{\bf e}}'))= \inf_{ m\in \{1,2,\ldots, n_e-1\}}val(Q_1^{S_m})
\leq \inf \{ \frac{p}{|\widetilde{{\bf e}}|_{(2)}}, \frac{V_1(\widetilde{{\bf e}}'_{S_2})}{\|\widetilde{{\bf e}}'_{S_2}\|_2}, \frac{V_1(\widetilde{{\bf e}}'_{S_3})}{\|\widetilde{{\bf e}}'_{S_3}\|_2}, \ldots, \frac{V_1(\widetilde{{\bf e}}')}{\|\widetilde{{\bf e}}'\|_2}   \}
\]
where
\[
\cot\angle(\widetilde{{\bf e}}_{S_m}, {\bf e}_0) =  \frac{1}{\|\widetilde{{\bf e}}'_{S_m}\|_2}.
\]
\end{lemma}

\begin{proof}[Proof of lemma~\ref{lemma: tigher upper}]
From 
\[
val(Q_1^{S_m}) :=\inf_{ \beta: \sum_{j=1}^{m}  |\widetilde{{\bf e}}|_{(j+1)} \beta_j  =1, \beta_j> 0} {\mathbb E}_{J_m} \|\beta\cdot J_m\|_{2} 
\]
we explore geometric point of view to find bounds.

For a fixed $m$, after re-ranking the entries by absolute value, let $S_m$ be the support so that only the top $m$ entries $|\widetilde{{\bf e}}|_{(2)}, |\widetilde{{\bf e}}|_{(3)},\ldots,|\widetilde{{\bf e}}|_{(m+1)}$ are non-zero,  let 
\[
|\widetilde{{\bf e}}'|_{S_m} = (0, |\widetilde{{\bf e}}|_{(2)}, |\widetilde{{\bf e}}|_{(3)},\ldots,|\widetilde{{\bf e}}|_{(m+1)}, 0, \ldots, 0)
\]
\[
\|\widetilde{{\bf e}}'_{S_m}\|_2^2 = |\widetilde{{\bf e}}|^2_{(2)}+ |\widetilde{{\bf e}}|^2_{(3)}+\ldots+|\widetilde{{\bf e}}|^2_{(m+1)} 
\]
let the unit vector along the direction of $|\widetilde{{\bf e}}'|_{S_m}$ be 
\[
u^m = |\widetilde{{\bf e}}'|_{S_m}/\|\widetilde{{\bf e}}'_{S_m} \|_2
\]
then 
\[
val(Q_1^{S_m}) := \frac{1}{\|\widetilde{{\bf e}}'_{S_m}\|_2}\inf_{ \beta:  \beta_j> 0}  {\mathbb E}_{J_m} \| (\beta/\|\beta \|_{2}) \cdot J_m\|_{2} \cdot \frac{1}{\sum_{j=1}^{m}  u^m_j \frac{\beta_j }{\|\beta \|_{2}}}
\]
From previous definition, since $\beta$ is supported on $S_m$, we denote it as $\beta^m$, then
$V_1(\beta^m)= {\mathbb E}_{J_m} \| (\beta/\|\beta \|_{2}) \cdot J_m\|_{2} $,
$V_1(\widetilde{{\bf e}}'_{S_m}) = V_1(|\widetilde{{\bf e}}'|_{S_m}) = V_1(u^m)$.


Geometrically,
\[
\cos\angle(\beta^m, u^m) = \sum_{j=1}^{m}  u^m_j \frac{\beta_j }{\|\beta \|_{2}}
\]
\[
\cot\angle(\widetilde{{\bf e}}_{S_m}, {\bf e}_0) =  \frac{1}{\|\widetilde{{\bf e}}'_{S_m}\|_2}
\]
Since $\beta^m = u^m$ is a feasible point of the constraint, we have an upper bound 
\[
val(Q_1^{S_m}) \leq \frac{V_1(\widetilde{{\bf e}}'_{S_m})}{\|\widetilde{{\bf e}}'_{S_m}\|_2} 
= \cot\angle(\widetilde{{\bf e}}_{S_m}, {\bf e}_0) V_1(\widetilde{{\bf e}}'_{S_m})
\]
\begin{eqnarray*}
val(Q_1^{S_m}) 
&=& \cot\angle(\widetilde{{\bf e}}_{S_m}, {\bf e}_0) \inf_{ \beta:  \beta_j> 0} V_1(\beta^m) \cdot \frac{1}{\cos\angle(\beta^m, u^m)}
\\
&=& \cot\angle(\widetilde{{\bf e}}_{S_m}, {\bf e}_0)  V_1(|\widetilde{{\bf e}}'|_{S_m}) \inf_{ \beta:  \beta_j> 0} \frac{V_1(\beta^m)}{V_1(u^m)} \cdot \frac{1}{\cos\angle(\beta^m, u^m)}
\end{eqnarray*}
The lower bound is given by finding the lower bound of
\[
C(u^m):=\inf_{ \beta:  \beta_j> 0} \frac{V_1(\beta^m)}{V_1(u^m)} \cdot \frac{1}{\cos\angle(\beta^m, u^m)}
\]
We know upper bound $C(u^m)\leq 1$, and lower bound $C(u^m) \geq \frac{1}{p^{1/2}}$ based on the fact that $V_1 \in [p, \sqrt{p}]$.


\end{proof}








  
  

 
 

  
  
  

 



  
  
  
 
 


  
  
  



 
  






































\section{Technical Tool: Tight Bound for Finite Difference of Objective}
We study the upper and lower bound of: 
\[
B({\bf e}_0, \phi) := \frac{{\mathbb E}_I  \| ({\bf e}_0+\phi)_I \|_2 - {\mathbb E}_I  \| ({\bf e}_0)_I \|_2}{\|\phi\|_2} 
\]

This upper and lower bound allows us to connect objective ${\mathbb E}_I  \| \psi_I \|_2$ and the $2-$norm of $ \psi - {\bf e}_0$:
\[
{\mathbb E}_I  \| \psi_I \|_2 - {\mathbb E}_I  \| ({\bf e}_0)_I \|_2 = {\mathbb E}_I  \| \psi_I \|_2 - p=B({\bf e}_0, \psi - {\bf e}_0) \| \psi - {\bf e}_0\|_2
\]














 
  


\paragraph{Bi-Lipschitzness of finite difference of objective near $e_0$ for linear constraint}
We normalize the problem by defining $t:=\|\phi\|_2, \beta:=\frac{\phi}{\|\phi\|_2}$.
After normalization, and taking into account the linear constraint  $\widetilde{{\bf e}}^T \phi = 0$, we will study, for $\beta\in \mathcal{B}_t :=\{\|\beta\|_2=1, u^T \beta = 0,  \|e_0+t\beta\|_\infty\leq 1 \}$ for finite $t$, the upper and lower bound of 
\[
\frac{{\mathbb E}_I  \| ({\bf e}_0+t\beta)_I \|_2 - {\mathbb E}_I  \| ({\bf e}_0)_I \|_2}{t}.
\]

First, if $t\rightarrow 0^+$, then we get the directional derivative along direction of $\beta$. 
\begin{eqnarray*}
\lim_{t\rightarrow 0^+}\frac{{\mathbb E}_I  \| ({\bf e}_0+t \beta)_I \|_2 - {\mathbb E}_I  \| ({\bf e}_0)_I \|_2}{t}.
\end{eqnarray*}

Due to convexity of the function $\beta \rightarrow {\mathbb E}_I  \| ({\bf e}_0+t \beta)_I \|_2$, we have the finite difference lower bounded by directional derivative:
\begin{eqnarray*}
\frac{{\mathbb E}_I  \| ({\bf e}_0+t \beta)_I \|_2 - {\mathbb E}_I  \| ({\bf e}_0)_I \|_2}{t} 
\geq 
 \lim_{t\rightarrow 0^+}\frac{{\mathbb E}_I  \| ({\bf e}_0+t \beta)_I \|_2 - {\mathbb E}_I  \| ({\bf e}_0)_I \|_2}{t} .
\end{eqnarray*}





\begin{theorem}[Bi-Lipschitzness of finite difference of objective near $e_0$ for linear constraint]
We have upper and lower bound
\begin{eqnarray*}
0 \leq \frac{{\mathbb E}_I  \| ({\bf e}_0+t \beta)_I \|_2 - {\mathbb E}_I  \| ({\bf e}_0)_I \|_2}{t} -\lim_{t\rightarrow 0^+}\frac{{\mathbb E}_I  \| ({\bf e}_0+t \beta)_I \|_2 - {\mathbb E}_I  \| ({\bf e}_0)_I \|_2}{t}
\leq 
  \frac{t}{2}p\Big( 
   \beta_0^2 
 +
 p  (1-\beta_0^2) \Big) \leq \frac{pt}{2}.
\end{eqnarray*}
This leads to
\[
0\leq B({\bf e}_0, \phi) -	 \nabla_{\phi}{{\mathbb E}_I  \| ({\bf e}_0+\phi)_I \|_2}\mid_{\phi=0}\leq \frac{p}{2}\|\phi \|_2.
\]
\end{theorem}


\begin{proof}
As mentioned before, the lower bound is derived from convexity.

In the proof, for finite $t$, we calculate the finite difference condition on $I_0$:
\begin{eqnarray*}
\frac{{\mathbb E}_I  \| ({\bf e}_0+t \beta)_I \|_2 - {\mathbb E}_I  \| ({\bf e}_0)_I \|_2}{t}
&=&
(1-p)  {\mathbb E}_{I}[ \|\beta_{I}\|_2 | 0 \not \in I ] + p\big(\frac{{\mathbb E}_I [ \| {\bf e}_0+t\beta_I \|_2 -   1 | 0 \in I ]}{t} \big).
\end{eqnarray*}
And
\begin{eqnarray*}
\lim_{t\rightarrow 0^+}\frac{{\mathbb E}_I  \| ({\bf e}_0+t \beta)_I \|_2 - {\mathbb E}_I  \| ({\bf e}_0)_I \|_2}{t}
&=&
(1-p)  {\mathbb E}_{I}[ \|\beta_{I}\|_2 | 0 \not \in I ] + p\beta_0.
\end{eqnarray*}
In the following, we prove an upper bound 
\begin{eqnarray*}
 \frac{{\mathbb E}_I [ \| {\bf e}_0+t\beta_I \|_2 -   1 | 0 \in I ]}{t}- \beta_0
\leq   \frac{t}{2} p \| \beta_{{(0)}} \|_2^2  + \frac{t}{2} \beta_0^2.
\end{eqnarray*}

For finite $t$,
\begin{eqnarray*}
\frac{{\mathbb E}_I [ \| {\bf e}_0+t\beta_I \|_2 -   1 | 0 \in I ]}{t}
&=&
{\mathbb E}_{I}[ \left( (\frac{1}{t}+ \beta_0)^2+\| \beta_{I_{(0)}} \|_2^2 \right)^{1/2} -\frac{1}{t}| 0 \in I ]\\
&=& \frac{1}{t}{\mathbb E}_{I}[ \left( 1+ 2t\beta_0 + t^2 \beta_0^2+t^2\| \beta_{I_{(0)}} \|_2^2 \right)^{1/2} -1| 0 \in I ].
\end{eqnarray*}



Additionally, using concavity of the function $\sqrt{1+x}$, we have
\begin{eqnarray*}
 && \frac{1}{t}{\mathbb E}_{I}[ \left( 1+ 2t\beta_0 + t^2 \beta_0^2+t^2\| \beta_{I_{(0)}} \|_2^2 \right)^{1/2} -1| 0 \in I ]\\
&\leq&  \frac{1}{t}[ \left( 1+ 2t\beta_0 + t^2 \beta_0^2+t^2{\mathbb E}_{I}\| \beta_{I_{(0)}} \|_2^2| 0 \in I ] \right)^{1/2} -1]\\
&=&  \frac{1}{t}[ \left( 1+ 2t\beta_0 + t^2 \beta_0^2+t^2 p \| \beta_{{(0)}} \|_2^2 \right)^{1/2} -1]\\
&=&  \frac{1}{t}[ \left( 1+ 2t\beta_0 + t^2 \beta_0^2+t^2 p (1-\beta_0^2) \right)^{1/2} -1].
\end{eqnarray*}
Now, apply the inequality: 
for any $x\geq -1$:
\[
\sqrt{1+x} -1 \leq \frac{x}{2},
\]
we have
\begin{eqnarray*}
 \frac{1}{t}[ \left( 1+ 2t\beta_0 + t^2 \beta_0^2+t^2 p \| \beta_{I_{(0)}} \|_2^2 \right)^{1/2} -1]
&\leq&  \beta_0 + \frac{t}{2} (p \| \beta_{{(0)}} \|_2^2  +  \beta_0^2).
\end{eqnarray*}
Therefore, 
\begin{eqnarray*}
\frac{{\mathbb E}_I  \| ({\bf e}_0+t \beta)_I \|_2 - {\mathbb E}_I  \| ({\bf e}_0)_I \|_2}{t} -\lim_{t\rightarrow 0^+}\frac{{\mathbb E}_I  \| ({\bf e}_0+t \beta)_I \|_2 - {\mathbb E}_I  \| ({\bf e}_0)_I \|_2}{t}
\leq 
  \frac{t}{2}p\Big( 
   \beta_0^2 
 +
 p  (1-\beta_0^2) \Big) \leq \frac{pt}{2}.
\end{eqnarray*}
The last inequality is due to $\beta_0^2 \leq 1$.

\end{proof}












































































 

 

\section{Appendix: Phase Transition Condition for Other Probabilistic Model of Signal}

\subsection{ Phase Transition Condition of Projection Pursuit and Convex Blind Deconvolution for General Probabilistic Model}
Now we state the general phase transition with light assumption. Here we don't need to even assume that $X$ are independent, we just need to know the existence of one sparse element $X_0$.

\begin{theorem}[Projection pursuit to find a sparse element]
\label{thm:sparse_element}
Given a sequence $\{X_t\}_{t\in {\mathbb Z}}$ where one element $X_0$ is a sparse element with $p G + (1-p) \delta_0$ for a symmetric distribution $G$.
Let $\psi^\star$ be the solution of  $P_{u}$:
\[
 \begin{array}{ll}
 \underset{\psi}{\mbox{minimize}}   & {\mathbb E}|\psi^T X|\\
\mbox{subject to}  
& u^T \psi =1
\end{array}
\tag{$P_{u}$ }
  \]
then there exists a threshold
$
p^\star
$,
$\psi^\star = e_0$ provided $p<p^\star$,
and  $\psi^\star \neq e_0$ provided $p> p^\star$.

Here $p^\star$ is determined by $G$, the distribution of $\{X_t, t\neq 0\}$ and the direction of $u$:
  \[
  \frac{p^\star}{1-p^\star} =  \inf_{\| \beta\|_2=1, u^T \beta =0} \frac{{\mathbb E}[|X_{(0)}^T \beta_{(0)}| \mid X_0 =0]}{{\mathbb E}[|G|]\|\beta_{(0)}\|_2} \tan(\angle( \beta, -e_0))
  \]
 where $\beta_{(0)}$ is $\beta$ with the $0-$th entry deleted.
   

  \end{theorem}

    
\begin{figure}[!tb]
\centering
\includegraphics[width=.8\textwidth]{./figures_BD/Phase_T_sine_p.png}
\caption{The population phase transition diagram of $ \frac{p^\star}{1-p^\star}  = \cot(\angle(u, e_0)) $ for the special case when $X_t$ is independent, $X_0$ is sampled from Bernoulli Gaussian $p N(0,1) +(1-p) \delta_0,$ and the rest $\{X_t, t\neq 0\}$ are sampled from Gaussian $N(0,1)$. The red region is failure, and the blue region is success. }
\label{fig-PT-project}
\end{figure}


\begin{proof}
The phase transition condition proof is based on  calculating the directional derivative.

First, we want to find a threshold $p^\star$ such that for all $p<p^\star$, the directional derivative at $\psi = e_0$ is non-negative along all direction $\beta$ on unit sphere such that $u^T \beta =0$:
	 \[
	 \nabla_{\beta}{{\mathbb E}|(e_0+\beta)^T X|}\geq 0
	 \]
Second, we calculate $\nabla_{\beta}{{\mathbb E}|(e_0+\beta)^T X|} $ based on whether $X_0$ is zero or not:
	 \[
	 \begin{array}{ll}
	     \nabla_{\beta}{{\mathbb E}|(e_0+\beta)^T X|} = p \beta_0 {\mathbb E}[|X_{0}| \mid X_0 \neq 0]  + (1-p) {\mathbb E}[|X_{(0)}^T \beta_{(0)}| \mid X_0 =0]
	 \end{array}
	\]
It is non-negative in case either $\beta_{0} > 0$, or in case $\beta_0 < 0$ and
	\[
	\frac{p}{1-p} < \frac{{\mathbb E}[|X_{(0)}^T \beta_{(0)}| \mid X_0 =0]}{{\mathbb E}[|G|]||\beta_0|}
	\]
	for all $\beta \in \{\| \beta\|_2=1, u^T \beta =0\}$.
$p^\star$ is the least upper bound of all $p$ satisfying this inequality. 
Therefore,  
	\[
	\frac{p^\star}{1-p^\star} =  \inf_{\| \beta\|_2=1, u^T \beta =0} \frac{{\mathbb E}[|X_{(0)}^T \beta_{(0)}| \mid X_0 =0]}{{\mathbb E}[|G|]|\beta_0|} =  \inf_{\| \beta\|_2=1, u^T \beta =0} \frac{{\mathbb E}[|X_{(0)}^T \beta_{(0)}| \mid X_0 =0]}{{\mathbb E}[|G|]\|\beta_{(0)}\|_2} \tan(\angle( \beta, -e_0))
	\]
\end{proof}


\subsection{Searching for Single Sparse Entry in Gaussian Signal}
\paragraph{Finding one sparse element: Bernoulli Gaussian $X_0$ and Gaussian $\{X_t, t\neq 0\}$}

Now we study the implication of the general phase transition problem in a simplest projection pursuit case, where $X_t$ are independent, $X_0$ is sampled from Bernoulli Gaussian $p N(0,1) +(1-p) \delta_0,$ and the rest $\{X_t, t\neq 0\}$ are sampled from Gaussian $N(0,1)$.

\begin{theorem}
Let $X_t$ be independent, $X_0$ is sampled from Bernoulli Gaussian $p N(0,1) +(1-p) \delta_0,$ the rest $\{X_t, t\neq 0\}$ are sampled from Gaussian $N(0,1)$.
Let $\psi^\star$ be the solution of  $P_{u}$:
\[
 \begin{array}{ll}
 \underset{\psi}{\mbox{minimize}}   & {\mathbb E}|\psi^T X|\\
\mbox{subject to}  
& u^T \psi =1
\end{array}
\tag{$P_{u}$ }
  \]
then there exists a threshold
$
p^\star
$,
$\psi^\star = e_0$ provided $p<p^\star$,
and  $\psi^\star \neq e_0$ provided $p> p^\star$.

Here $p^\star$ is determined by the direction of $u$:
  \[
  \frac{p^\star}{1-p^\star} = \inf_{\| \beta\|_2=1, u^T \beta =0}\tan(\angle( \beta, -e_0)) = \cot(\angle(u, e_0))
  \]
  \end{theorem}
\begin{proof}
First, when the rest of $X_t, t\neq 0$ is independent of $X_0$, and IID sampled from $ N(0,1),$
	\[
 \frac{{\mathbb E}[|X_{(0)}^T \beta_{(0)}| \mid X_0 =0]}{\|\beta_{(0)}\|_2} = 
\sqrt{\frac{2}{\pi}}
\]
Since  
\[
	 {\mathbb E}[|G|] = \sqrt{\frac{2}{\pi}},
	 \]
we have a simple phase transition condition that is completely geometric.
	 \[
  \frac{p^\star}{1-p^\star} = \inf_{\| \beta\|_2=1, u^T \beta =0}\tan(\angle( \beta, -e_0))
  \]
 
Second, we know that there is a geometric structure:
\[
\inf_{\| \beta\|_2=1, u^T \beta =0} \frac{\|\beta_{(0)}\|_2}{|\beta_0|}
= \inf_{\| \beta\|_2=1, u^T \beta =0}\tan(\angle( \beta, -e_0)) = \cot(\angle(u, e_0)),
\]
as indicated by figure~\ref{fig-Angle}. 
When $\beta = P_u^\perp (-e_0)$,
	\[
	\begin{array}{ll}
	\inf_{\| \beta\|_2=1, u^T \beta =0} 
		\tan(\angle( \beta, -e_0)) 
		&=
		\tan(\angle( P_u^\perp (-e_0), -e_0)) \\
		&= \cot(\angle(u, e_0)).
	\end{array}
	\]
	
Therefore, $ \frac{p^\star}{1-p^\star}  = \cot(\angle(u, e_0)) $,
	

\end{proof}





\section{Old concentration results}

 
 

  
 
  
  
  



 





  









 





  
  



 






Now we need to derive a concentration inequality on $D$. 
We focus on deriving a concentration inequality on $\|h*x\|_1$, and the concentration inequality 
on  $\|(h*x)_S\|_1$ can be derived similarly.

We remark that $ \|h*x\|_1 =\sum_{j=1}^n |h^{(j),T} x|= \sum_{j=1}^n |\langle C(a^{(j)}) v, x\rangle|$, 
if they are independent random variables, we can bound the deviation of $ \|h*x\|_1$ from its mean
using tail bounds via Bernstein’s inequality, by using the previous two bound on
variance and absolute value. 
This is the strategy to derive concentration inequality in dictionary learning\cite{luh2016dictionary,blasiok2016improved}.
This is also the strategy to derive concentration inequality for multiple observation blind deconvolution setting, since there are independence across different observations.

However, here due to the nature of the convolution, we cannot directly apply Bernstein’s inequality. There are additional dependencies that cause technical difficulties. 

Therefore, we use the dual representation of the $1$-norm and use Talagrand-Bernstein’s inequality on supreme of random function to derive concentration.
We put this part of the proof in the following lemma, since they are technical and they could be of independent interest for other problems with dependency structure. 




From the lemma, if we let $Z = \|h*x\|_1$, then we have 
 \[
  \begin{array}{ll}
& {\mathbb P} \left[  |\|h*x\|_1- {\mathbb E} \|h*x\|_1 | > t   \right] \\
&= {\mathbb P} \left[  |Z- {\mathbb E} Z | > t   \right] \\
&\leq   \exp\left(  \frac{-t^2/2   }{\sigma^2+K(2{\mathbb E} Z+t/3)}\right)
\end{array}
  \]  
where $\sigma^2\leq  {\mathbb E} Z^2$, $K\leq \|Z\|_\infty$.

From the previous calculation, we recall that we have a lower bound on expectation 
$ {\mathbb E} \|h*x\|_1$,
\[
{\mathbb E} \|h*x\|_1\geq    \sqrt{\frac{p}{2}}  n \| h\|_2 
\] 
\[
{\mathbb E} Z = {\mathbb E} \|h*x\|_1 \geq  \sqrt{\frac{p}{2}}  n \| h\|_2
\]
  









Now we calculate $\sigma^2, K$:
 \[
  \begin{array}{ll}
\sigma^2 &={\mathbb E} \langle w, C(h)x \rangle^2,\\
&={\mathbb E} (C(h)^* w)^T xx^* C(h)^* w  \\
&\leq  p \|C(h)^* w\|_2^2\\
&=p \langle w, C(h)C(h)^* w \rangle 
\end{array}
  \]  
Let $C(h)=UD(\hat{h})U^*$ be the eigen-decomposition, where $U$ is the unitary Fourier matrix, $D(\hat{h})$ is the diagonal matrix with $\hat{h}$ as its diagonal entries. Then using $h=a*v$, $\hat{h}=\hat{a}\hat{v}$, we have
      \[
  \begin{array}{ll}
(C(h)C(h)*) 
&=   UD(\hat{h})U^* U D(\hat{h})^* U^* \\
&=  UD(|\hat{h}|^2) U^* \\
&=  UD(|\hat{a}\hat{v}|^2) U^*. \\
\end{array}
  \] 
Now we have
 \[
  \begin{array}{ll}
\sigma^2 &\leq 
 p \langle w, C(h)C(h)^* w \rangle \\
&= p  \langle w,  UD(|\hat{a}\hat{v}|^2) U^* w \rangle \\
&= p  \langle U^* w,  D(|\hat{a}\hat{v}|^2) U^* w \rangle 
\end{array}
  \] 
Let $\tilde{w} = \langle U^* w $, using the fact that $\|w\|_{\infty}\leq 1$, we have  $\|\tilde{w}\|_{\infty}\leq 1$, therefore,
 \[
  \begin{array}{ll}
\sigma^2 &\leq  p  \langle \tilde{w},  D(|\hat{a}\hat{v}|^2) \tilde{w} \rangle \\
&=  p  \sum_{i=1}^n \tilde{w}_i^2 |\hat{a}_i\hat{v}_i|^2)  \\
&\leq  p  \sum_{i=1}^n  |\hat{a}_i\hat{v}_i|^2 \\
&\leq  p  \max_{i=1}^n |\hat{a}_i|^2\sum_{i=1}^n  |\hat{v}_i|^2 \\
&=  p  \max_{i=1}^n |\hat{a}_i|^2 \|v\|_2^2 \\
&=  p  \max_{i=1}^n |\hat{a}_i|^2 
\end{array}
  \] 
  
  Now we derive the absolute value bound. First, since $x_i$ are iid sampled from the product of Bernoulli random variable and standard Gaussian, the supreme of the absolute value $\tau = \max_i |x_i|$ is upper bounded by the  supreme of standard Gaussian, which is less than $  2\log^{1/2} n$ with probability $1-\exp(-Cn^2)$. In the following, we only consider this case and assume $\tau \leq 2\log^{1/2} n$ and add the tail probability into the tail probability in the concentration inequality.
  
 Using other side of Khintchine inequality, we have
  \[
 \begin{array}{ll}
&  |(h*x)_i|\\
 =&  |\sum_{j=1}^n h_{i-j} \Omega_{j} |G_{j}|\mbox{sign}(G_{j}) |\\
\leq&  
  \sqrt{\sum_{j=1}^{n} (h_{i-j} \Omega_{j}|G_{j}|)^2}\\
 \leq&  \tau  \sum_{i=1}^n \sqrt{\sum_{j=1}^{n}  h_{i-j}^2}\\
   =&   \tau  \| h\|_2 
\end{array}
  \]  
Therefore, we can take $K=  \tau  \| h\|_2  .$

 












We point out that the eigenvalues of $C(a)$ is the Fourier transform of $\hat a$ of $a$, which is the 
the entry-wise inverse $1/\hat w^{0}$. And the singular values of $C(a)$ are the absolute values $|\hat a|$. 
Therefore, using the fact that $\|v\|_2=1$,
\[
 \begin{array}{ll}
 \min_i |\hat a_i| \leq \|h\|_2 =  \|C(a)v\|_2 \leq \max_i |\hat a_i| 
\end{array}
  \]
  
Then the condition number of $C(a)$ can be represented as $\delta_a =   \max_i |\hat a_i|/\min_i |\hat a_i|$.



Overall, for $Z = \|h*x\|_1$, we have ${\mathbb E} Z \geq \sqrt{\frac{p}{2}}  n \| h\|_2$, we choose $t = \sqrt{\frac{p}{2}}  n \| h\|_2$. Since 
 \[
  \begin{array}{ll}
 {\mathbb P} \left[  |Z- {\mathbb E} Z | > t   \right] 
\leq   \exp\left(  \frac{-t^2/2   }{\sigma^2+K(2{\mathbb E} Z+t/3)}\right),
\end{array}
  \]  
we calculate
\[
  \begin{array}{ll}
 &  \frac{-t^2/2   }{\sigma^2+K(2{\mathbb E} Z+t/3)}\\
=&  -\frac{1   }{2(\sigma^2/t^2+K(2{\mathbb E} Z+t/3)/t^2)}\\
\leq &  -\frac{1   }{2(\sigma^2/t^2+K(2+1/3)/t)}\\
 \end{array}
  \]  
Now, we calculate both terms on the denominator,
  \[
  \begin{array}{ll}
   \sigma^2/t^2 &\leq (p  \max_{i=1}^n |\hat{a}_i|^2)/( \sqrt{\frac{p}{2}}  n \| h\|_2)^2 \\
  &\leq 2\max_{i=1}^n |\hat{a}_i|^2)/(\min_{i=1}^n |\hat{a}_i|^2 n^2 \\
   &\leq 2\delta_a^2/ n^2 ,
 \end{array}
  \]  
   \[
  \begin{array}{ll}
  (2+1/3)K/t 
 &\leq  \frac{7}{3}\tau  \| h\|_2 /\sqrt{\frac{p}{2}}  n \| h\|_2\\
  &\leq \frac{7\sqrt{2}}{3}\tau  /\sqrt{p}  n \\
   &\leq \frac{7}{2}\tau  /\sqrt{p}  n 
 \end{array}
  \]  
  
 As we discussed before, since $\tau$ is the maximum of IID Gaussian random variable, 
 $\tau \leq C_1 \log^{1/2}(n)$ with probability $1 - \exp(-C_2 n^2)$ for $C_1$ large enough.
 
And from the assumption we have $ \delta_a \leq C_0 n^{1/2}/ \log^{1/4}(n)$.
  
Therefore, 
\[
  \begin{array}{ll}
& {\mathbb P} \left[  |\|h*x\|_1- {\mathbb E} \|h*x\|_1 | > t   \right] \\
&\leq   \exp\left( \frac{-1   }{ 4\delta_a^2/ n^2  + 7\tau  /\sqrt{p}  n  } \right)\\
&\leq   \exp\left( \frac{-1   }{ 4\delta_a^2/ n^2  + 7 C_1 \log^{1/2}(n)  /\sqrt{p}  n  } -C_2 n^2\right)\\
&\leq   \exp\left( \frac{-1   }{ 4\delta_a^2/ n^2  + 7 C_1 \log^{1/2}(n)  /\sqrt{p}  n  } -C_2 n^2\right)\\
&\leq   \exp\left( \frac{-n   }{  \log^{1/2}(n) (4C_0+  C_3 \sqrt{p})} -C_2 n^2 \right)\\
&\leq   \exp\left( \frac{- C n   }{  (p\log(n))^{1/2}}  \right)
 \end{array}
  \]  
  
  

  
  
  
  
  
  
  
  

\begin{proof}
In the lemma below, we use a covering argument to give a union bound for $v\in B_k=\{v
\in{\mathbb R}^k\mid \|v\|_2=1\}$. 
We want to cover $B_k$ with $\epsilon$ balls. From classical covering argument,
we know that we need at most $|N| = (3/\epsilon)^k$ $\epsilon$ balls to cover $B_k$, and we use symmetrization trick to find the proper $\epsilon$.

Let the deviation be
\[
\tilde{\mathcal{X}}_v =   \sum_{i=1}^n \left( | \langle C(a^{(i)}) v, x\rangle| - {\mathbb E} | \langle C(a^{(i)}) v, x\rangle|\right).
\]  
We will choose $1/\epsilon$ to be the Lipschitz constant of  $\tilde{\mathcal{X}}_v$ as a function of $v$. Let's compute the Lipschitz constant.

 
 

We introduce additional independent random sign $s_j$ , which are IID sampled from Rademacher variable by defining 
\[
\mathcal{X}_v =   \sum_{i=1}^n s_j \left( |\langle C(a^{(i)}) v, x\rangle| \right),
\]  
 then by lemma $24$ of \cite{blasiok2016improved}, for every integer $p$ we have 
 \[
 \| \sup_{v\in B_k} |\tilde{\mathcal{X}}_v | \|_p \lesssim \| \sup_{v\in B_k}  |\mathcal{X}_v| \|_p
 \]
 This is called symmetrization trick. Using this trick, we only need to analyze the tail behavior of the random variable $\mathcal{X}_v$. To obtain tail bounds for maximum  of  $\sup_{v\in B_k}  |\mathcal{X}_v|$, we need to bound tails of the random variables $\mathcal{X}_v-\mathcal{X}_{v'}$ for $v, v'\in B_k$.
 
Since  
\[
\mathcal{X}_v-\mathcal{X}_{v'} =  
\sum_{i=1}^n s_i \left( | \langle C(a^{(i)}) v, x\rangle| -   | \langle C(a^{(i)}) v', x\rangle|\right).
\]

Then use the fact that 
\[
| \langle C(a^{(i)}) v, x\rangle| -   | \langle C(a^{(i)}) v', x\rangle| \leq   \left( | \langle C(a^{(i)}) (v-v'), x\rangle|  \right),
\]
and  both $s_i | \langle C(a^{(i)}) v, x\rangle| -   | \langle C(a^{(i)}) v', x\rangle| $ and  
$s_i \left( | \langle C(a^{(i)}) (v-v'), x\rangle|  \right)$ are symmetric, if we have a tail bound for 
$\sum_{i=1}^n s_i \left( | \langle C(a^{(i)}) (v-v'), x\rangle|  \right)$ , 
we also have tail bounds on $\mathcal{X}_v-\mathcal{X}_{v'}$.

From these calculation, we saw that the Lipschitz constant of  $\tilde{\mathcal{X}}_v$ is bounded by 
 \[
  \begin{array}{ll}
\sqrt{\sum_{j=1}^n (\sum_{i=1}^n s_i |\sum_{t=1}^n a^{i-j+t} x_t|)^2}
&\leq   \sqrt{n (\|a\|_1 \|x\|_1)^2}\\
&\leq  \tau \sqrt{n}np \|a\|_1
\end{array}
  \]  
  
Therefore, we can choose $1/\epsilon = \tau \sqrt{n}np \|a\|_1$, then
at most $|N| = (3\tau \sqrt{n}np \|a\|_1)^k$ balls can cover $B_k$.
We have a concentration inequality in each ball,
We define the tail event 
\[
\mathcal{E}_v = \{|\tilde{\mathcal{X}}_v | > t\}
\]
we can take a union bound,
 \[
  \begin{array}{ll}
 {\mathbb P} \left(\cup_{v \in B_k} \mathcal{E}^c_v \right) &\leq |N| {\mathbb P} \left[  |\|h*x\|_1- {\mathbb E} \|h*x\|_1 | > t   \right] \\
&\leq   (3\tau \sqrt{n}np \|a\|_1)^k {\mathbb P} \left[  |\|h*x\|_1- {\mathbb E} \|h*x\|_1 | > t   \right] 
\end{array}
  \] 
  
Since  $n>  k \log(k/p)$, for large enough constant $C$, we showed in the lemma that there exist a positive constant $C_0$, 
 \[
  \begin{array}{ll}
 {\mathbb P} \left(\cup_{v \in B_k} \mathcal{E}^c_v \right) &\leq  \exp\left(  k\log(3\tau \sqrt{n}np \|a\|_1) - \frac{ C_6 n   }{  (p\log(n))^{1/2}}\right) \\
&\leq    \exp\left( k(\log(3\tau)+\log(\|a\|_1)+\log(p)+3/2 \log(n)) -\frac{ C n   }{  (p\log(n))^{1/2}}\right) 
\end{array}
  \] 
  

Then bound on $\sup_{v \in B_k} |\mathcal{X}_v|$ would imply the same bound on $\sup_{v \in B_k} |\tilde{\mathcal{X}}_v|$.
And with this bound on $\sup_{v \in B_k} |\tilde{\mathcal{X}}_v|$, we proved that 
for all $v\neq 0, v \in B_k$, $D>0$ with probability $1-\delta$, where
\[
\delta \leq  \exp\left(  -C_0 \delta_a^{-2} k\log(kp) \right)
\]
Therefore, when $n>C_1k^2\log(kp)$, 
$
\delta \leq   \exp\left(  -C_0 \delta_a^{-2} k\log(kp) \right),
$
then $\delta$ converges to zero as $k$ goes to infinity.
\end{proof}





\begin{lemma}
Let $Z = \|h*x\|_1$ where $h \in \mathcal{H}$, then 
 \[
  \begin{array}{ll}
& {\mathbb P} \left[  Z - {\mathbb E} Z  > t   \right] \\
&\leq   \exp\left(  \frac{-t^2/2   }{\sigma^2+K(2{\mathbb E} Z+t/3)}\right)
\end{array}
  \]  
where $\sigma^2\leq  {\mathbb E} Z^2$, $K\leq \|Z\|_\infty$.
\end{lemma}
\begin{proof}
Using duality of $1$-norm, we know that
\[
\|h*x\|_1 = \sup_{\|w\|_\infty\leq 1}\langle w, C(h)x \rangle
\]
 From the theorem $8.42$ of \cite{foucart2013mathematical}, we have the following inequality:
 for the supreme of functions $\langle w, C(h)x \rangle$, since ${\mathbb E} \langle w, C(h)x \rangle=0$,
 $ \langle w, C(h)x \rangle\leq K$, 
 \[
 \sigma^2 ={\mathbb E} \langle w, C(h)x \rangle^2,
 \]
 we have
 \[
  \begin{array}{ll}
 {\mathbb P} \left[  Z - {\mathbb E} Z  > t   \right] \leq   \exp\left(  \frac{-t^2/2   }{\sigma^2+K(2{\mathbb E} Z+t/3)}\right)
\end{array}
  \]  
\end{proof}
We also remark that the essential concentration inequality that we are trying to derive is the following:
let's consider the case when $x$ is IID standard Gaussian, we define $z = h*x = C(h)x$,
then $z$ is a non-isotropic Gaussian vector \[\mathcal{N}(0,\Sigma(h)),\] where 
\[  \Sigma(h)_{ij} = \langle h^{(i)},  h^{(j)}\rangle\]
And we are trying to derive a concentration inequality for $\|z\|_1$.






\begin{lemma}
For all $v\in B_k$, the probability 
\[
  \begin{array}{ll}
& {\mathbb P} \left[  |\|v*a*x\|_1- {\mathbb E} \|v*a*x\|_1 | > t, v \in B_k  \right]   \\ 
\leq&  (3\tau \sqrt{n}np \|a\|_1)^k {\mathbb P} \left[  |\|h*x\|_1- {\mathbb E} \|h*x\|_1 | > t   \right] 
\end{array}
  \] 
\end{lemma}
\begin{proof}
Let the deviation be
\[
\tilde{\mathcal{X}}_v =   \sum_{i=1}^n \left( | \langle C(a^{(i)}) v, x\rangle| - {\mathbb E} | \langle C(a^{(i)}) v, x\rangle|\right).
\]  
We will choose $1/\epsilon$ to be the Lipschitz constant of  $\tilde{\mathcal{X}}_v$ as a function of $v$. Let's compute the Lipschitz constant.

 
 

We introduce additional independent random sign $s_j$ , which are IID sampled from Rademacher variable by defining 
\[
\mathcal{X}_v =   \sum_{i=1}^n s_j \left( |\langle C(a^{(i)}) v, x\rangle| \right),
\]  
 then by lemma $24$ of \cite{blasiok2016improved}, for every integer $p$ we have 
 \[
 \| \sup_{v\in B_k} |\tilde{\mathcal{X}}_v | \|_p \lesssim \| \sup_{v\in B_k}  |\mathcal{X}_v| \|_p
 \]
 This is called symmetrization trick. Using this trick, we only need to analyze the tail behavior of the random variable $\mathcal{X}_v$. To obtain tail bounds for maximum  of  $\sup_{v\in B_k}  |\mathcal{X}_v|$, we need to bound tails of the random variables $\mathcal{X}_v-\mathcal{X}_{v'}$ for $v, v'\in B_k$.
 
Since  
\[
\mathcal{X}_v-\mathcal{X}_{v'} =  
\sum_{i=1}^n s_i \left( | \langle C(a^{(i)}) v, x\rangle| -   | \langle C(a^{(i)}) v', x\rangle|\right).
\]

Then use the fact that 
\[
| \langle C(a^{(i)}) v, x\rangle| -   | \langle C(a^{(i)}) v', x\rangle| \leq   \left( | \langle C(a^{(i)}) (v-v'), x\rangle|  \right),
\]
and  both $s_i | \langle C(a^{(i)}) v, x\rangle| -   | \langle C(a^{(i)}) v', x\rangle| $ and  
$s_i \left( | \langle C(a^{(i)}) (v-v'), x\rangle|  \right)$ are symmetric, if we have a tail bound for 
$\sum_{i=1}^n s_i \left( | \langle C(a^{(i)}) (v-v'), x\rangle|  \right)$ , 
we also have tail bounds on $\mathcal{X}_v-\mathcal{X}_{v'}$.

From these calculation, we saw that the Lipschitz constant of  $\tilde{\mathcal{X}}_v$ is bounded by 
 \[
  \begin{array}{ll}
\sqrt{\sum_{j=1}^n (\sum_{i=1}^n s_i |\sum_{t=1}^n a^{i-j+t} x_t|)^2}
&\leq   \sqrt{n (\|a\|_1 \|x\|_1)^2}\\
&\leq  \tau \sqrt{n}np \|a\|_1
\end{array}
  \]  
  
Therefore, we can choose $1/\epsilon = \tau \sqrt{n}np \|a\|_1$, then
at most $|N| = (3\tau \sqrt{n}np \|a\|_1)^k$ balls can cover $B_k$.
We have a concentration inequality in each ball,
We define the tail event 
\[
\mathcal{E}_v = \{|\tilde{\mathcal{X}}_v | > t\}
\]
we can take a union bound,
 \[
  \begin{array}{ll}
 {\mathbb P} \left(\cup_{v \in B_k} \mathcal{E}^c_v \right) &\leq |N| {\mathbb P} \left[  |\|h*x\|_1- {\mathbb E} \|h*x\|_1 | > t   \right] \\
&\leq   (3\tau \sqrt{n}np \|a\|_1)^k {\mathbb P} \left[  |\|h*x\|_1- {\mathbb E} \|h*x\|_1 | > t   \right] 
\end{array}
  \] 
\end{proof}

\section{Finite sample analysis}
\paragraph{Finite sample setting}
\begin{itemize}
\item
Let $(X_t)_{t \in {\mathbb Z}}$ be IID sampled from  $pN(0,1) + (1-p) \delta_0$.
 \item
The filter $a\in l_1({\mathbb Z}), (\mathcal{F} a)(\omega) \neq 0, \forall \omega\in T.$ Additionally, in finite sample setting, we assume $a^{-1}$ is zero outside of a centered window of radius $2k+1$, namely, the non-zero part can be written as $(\beta_{-k}, \ldots, \beta_0, \ldots, \beta_k)\in V$. 

We fix the shift and scaling effect by assuming $v^T a^{-1} = 1$ for a generic $v$ that has one single largest entry. 
\item
Let $Y= a* X$ be a linear process, we are given a series of observation $(Y_t)_{t\in [-(T+k),T+k]}$ from a centered window of radius $T+k$.
\item
 We denote $X^{(i)}$ as the sequence being flipped and shifted for $i$ step from $X$ s that $X^{(i)}_k = X_{-k +i}$. Let $J_N$ be the support of $X$.
 We define the length $N=2T+1$ window as ${\cal T} := [-T, T].$
\end{itemize}
Now we want to find an inverse filter 
$w = (w_{-k}, \ldots, w_0, \ldots, w_k) \in V$ such that the convolution $w*Y$ is sparse,
\begin{equation}
 \begin{array}{ll}
 \underset{w\in V}{  \mbox{minimize} }   & \frac{1}{N}\|w*Y\|_{1,{\cal T}}\\
 \mbox{subject to}  
& v^T w =1.
\end{array}
\tag{$P_{v}^T$}
\end{equation}




First, after change of variable,
\[
 \frac{1}{N}\|w*Y\|_{1,{\cal T}} = \frac{1}{N}\|\psi*X\|_{1,{\cal T}}
\]
Then the empirical average can be represented as sum of shifted and flipped inner products.
\[
\frac{1}{N}\|\psi*X\|_{1,{\cal T}}  = \frac{1}{N}\sum_{|i|\in {\cal T}} |[\psi*X]_i| = \frac{1}{N}\sum_{|i|\in {\cal T}}|\psi^T X^{(i)}|
\]
We define 
\[
p_N = \frac{1}{N}\sum_{i\in {\cal T}} I_i
\]
The population average is being calculated,  as
\[
{\mathbb E}  \|\psi*X \|_1:= {\mathbb E}|C_i(\psi)^T X| = {\mathbb E}|\tilde{\psi}^T X|
\]
We know the mean of $p_N$ is $p$, the variance of $p_N$ as
\[
{\mathbb E}[p_N-p]^2=\frac{1}{N}p(1-p)
\]






 
 
 
 \paragraph{Finite sample directional derivative}
 \begin{eqnarray*}
& & \lim_{t\rightarrow 0^+}\frac{1}{t}[(\hat{{\mathbb E}}  \| (\psi+t\beta)*X \|_1 - \hat{{\mathbb E}}  \| \psi*X \|_1)
\\
  &=& 
 \langle  \beta, {\frac{1}{N}\sum_{|i|\in {\cal T}} \nabla_{\psi}L_i}|_{\psi} \rangle \\
 &=& 
 \frac{1}{N}\sum_{|i|\in {\cal T}}  \mbox{sign}(\psi^T X^{(i)})X^{(i),T}\beta
 \end{eqnarray*}
Now to study the finite sample convex problem, we need to calculate the directional derivative at $ \psi = e_0$.
Then
\begin{theorem}
For any $X$ sequence with support  $J_N$,
\begin{eqnarray*}
& & \lim_{t\rightarrow 0^+}\frac{1}{t}[(\hat{{\mathbb E}}  \| (e_0+t\beta)*X \|_1 - \hat{{\mathbb E}}  \| e_0*X \|_1)
\\
  &=& 
 \langle  \beta, {\frac{1}{N}\sum_{|i|\in {\cal T}} \nabla_{\psi}L_i}|_{e_0} \rangle \\
 &=& 
 \frac{1}{N}\sum_{|i|\in {\cal T}}  \mbox{sign}( X_{-i})X^{(i),T}\beta
 \\
  &=& 
 \frac{1}{N} \left(\|X\|_{1,J_N} \beta_0  +
 \|X*\beta_{(0)}\|_{1,{\cal T}-J_N}+ 
\sum_{i\in J_N} (1_{X_{-i}> 0}-1_{X_{-i}< 0})  \cdot X^{(i),T}\beta_{(0)} \right) 
 \end{eqnarray*}
 
\end{theorem}
\begin{proof}
 \begin{eqnarray*}
& & \lim_{t\rightarrow 0^+}\frac{1}{t}[(\hat{{\mathbb E}}  \| (e_0+t\beta)*X \|_1 - \hat{{\mathbb E}}  \| e_0*X \|_1)
\\
  &=& 
 \langle  \beta, {\frac{1}{N}\sum_{|i|\in {\cal T}} \nabla_{\psi}L_i}|_{e_0} \rangle \\
 &=& 
 \frac{1}{N}\sum_{|i|\in {\cal T}}  \mbox{sign}( X_0^{(i)})X^{(i),T}\beta
 \\
 &=& 
 \frac{1}{N}\sum_{|i|\in {\cal T}}  \mbox{sign}( X_{-i})X^{(i),T}\beta
 \\
 &=& 
\frac{1}{N}\sum_{|i|\in {\cal T}} 1_{X_{-i}> 0}  \cdot X^{(i),T}\beta  +
\frac{1}{N}\sum_{|i|\in {\cal T}} 1_{X_{-i}< 0}  \cdot (-X^{(i),T}\beta)  +
\frac{1}{N}\sum_{|i|\in {\cal T}} 1_{X_{-i}= 0} \cdot |X^{(i),T}\beta|
 \\
 &=& 
\frac{1}{N}\sum_{|i|\in {\cal T}} 1_{X_{-i}> 0}  \cdot X^{(i),T}\beta  +
\frac{1}{N}\sum_{|i|\in {\cal T}} 1_{X_{-i}< 0}  \cdot (-X^{(i),T}\beta)  +
\frac{1}{N}\sum_{|i|\in {\cal T}}  |X^{(i),T}\beta_{(0)}|
 \\
 &=& 
\frac{1}{N}\sum_{|i|\in {\cal T}} 1_{X_{-i}> 0}  \cdot X_{-i}\beta_0  +
\frac{1}{N}\sum_{|i|\in {\cal T}} 1_{X_{-i}< 0}  \cdot (-X_{-i}\beta_0)  + 
\frac{1}{N}  \|X*\beta_{(0)}\|_{1,{\cal T}-J_N}+ 
\\
& &
\frac{1}{N}\sum_{|i|\in {\cal T}} 1_{X_{-i}> 0}  \cdot X^{(i),T}\beta_{(0)}  +
\frac{1}{N}\sum_{|i|\in {\cal T}} 1_{X_{-i}< 0}  \cdot (-X^{(i),T}\beta_{(0)})  
 \\
  &=& 
\frac{1}{N}\sum_{|i|\in {\cal T}} 1_{X_{-i}\neq 0}  \cdot |X_{-i}|\beta_0  +
\frac{1}{N}  \|X*\beta_{(0)}\|_{1,{\cal T}-J_N}+ 
\frac{1}{N}\sum_{i\in J_N} (1_{X_{-i}> 0}-1_{X_{-i}< 0})  \cdot X^{(i),T}\beta_{(0)}   
 \\
  &=& 
 \frac{1}{N} \left(\|X\|_{1,J_N} \beta_0  +
 \|X*\beta_{(0)}\|_{1,{\cal T}-J_N}+ 
\sum_{i\in J_N} (1_{X_{-i}> 0}-1_{X_{-i}< 0})  \cdot X^{(i),T}\beta_{(0)} \right) 
 \end{eqnarray*}
\end{proof}
\begin{coro}
For Gaussian IID sequence $X$, 
 the random variable $ \frac{\|X\|_{1,J_N}}{N} = p_N   ( \frac{\|X\|_{1,J_N}}{|J_N|})$ has mean $p\sqrt{\frac{2}{\pi}}$. 
 And the random expression
\begin{eqnarray*}
& &  \frac{1}{N} \left(\|X\|_{1,J_N} \beta_0  +
 \|X*\beta_{(0)}\|_{1,{\cal T}-J_N} \right) 
\\
  &=& 
p_N   ( \frac{\|X\|_{1,J_N}}{|J_N|}) \beta_0  +
(1-p_N ) \frac{1}{|{\cal T}-J_N|}  \|X*\beta_{(0)}\|_{1,{\cal T}-J_N}
 \end{eqnarray*}
 has mean $\sqrt{\frac{2}{\pi}}(p \cdot \beta_0 + (1-p) \cdot {\mathbb E}_{I}[ \|\beta_{I}\|_2 | 0 \not \in I ])$.
\end{coro}
\begin{proof}
 \begin{eqnarray*}
& & 
\frac{1}{N} \left(\|X\|_{1,J_N} \beta_0  +
 \|X*\beta_{(0)}\|_{1,{\cal T}-J_N} \right) 
 \\
  &=& 
p_N ( \frac{\|X\|_{1,J_N}}{|J_N|})\beta_0  +
(1-p_N ) \frac{1}{|{\cal T}-J_N|}  \|X*\beta_{(0)}\|_{1,{\cal T}-J_N}
 \end{eqnarray*}
where $p_N = \frac{|J_N|}{N} = \frac{1}{N}\sum_{|i|\in {\cal T}}  1_{X_{-i}\neq 0}$ has mean $p$ and variance $\frac{1}{N} p(1-p)$.
And notice that for the Gaussian IID case 
\[
{\mathbb E} [ p_N\frac{\|X\|_{1,J_N}}{|J_N|}] = {\mathbb E}[p_N|X_{0}|\mid X_{0}\neq 0]  = p\sqrt{\frac{2}{\pi}},
\]
\[
{\mathbb E} [(1-p_N ) \frac{1}{|{\cal T}-J_N|}  \|X*\beta_{(0)}\|_{1,{\cal T}-J_N} ] = \sqrt{\frac{2}{\pi}}{\mathbb E}_{I}[ \|\beta_{I}\|_2 | 0 \not \in I ]).
\]
\end{proof}

\begin{coro}
 For symmetric IID sequence $X$, 
  \[
  \frac{1}{N}\sum_{i\in J_N} (1_{X_{-i}> 0}-1_{X_{-i}< 0})  \cdot X^{(i),T}\beta_{(0)}  = \frac{1}{N}\sum_{i\in J_N} (1_{X_{-i}> 0}-1_{X_{-i}< 0})  \cdot (\sum_{j\in J_N-\{i\} } X_j \beta_{i-j} ) 
  \]
  where 
  $\sum_{j\in J_N-\{i\} } X_j \beta_{i-j} $ is independent of $X_i$, therefore independent of $(1_{X_{-i}> 0}-1_{X_{-i}< 0}) $.
  Specifically, it has mean $0$
   \[
 {\mathbb E} \frac{1}{N}\sum_{i\in J_N} (1_{X_{-i}> 0}-1_{X_{-i}< 0})  \cdot X^{(i),T}\beta_{(0)}  = 0
  \]
  for $q\leq 2$,
    \[
  \frac{1}{N}{\mathbb E}|\sum_{i\in J_N} (1_{X_{-i}> 0}-1_{X_{-i}< 0})  \cdot X^{(i),T}\beta_{(0)} |^q \leq 
  \frac{1}{N} {\mathbb E}[]|\sum_{i\in J_N}|1_{X_{-i}> 0}-1_{X_{-i}< 0}||^q\cdot  \max_j |X^{(j),T}\beta_{(0)} |^q]
  \]
  
\end{coro}



 \paragraph{Concerntration of $\|\psi*X\|_{1}$ }
The next step is to provide an uniform bound of the finite sample derivation of $\|X\|_{1,J_N}$ and 
$\|X*\beta_{(0)}\|_{1,{\cal T}-J_N}$.

In general, for any random subset $I$ of ${\cal T}$, we study the uniform concentration of 
\[
\|\psi*X\|_{1,I} 
\]
for $\psi = a*w \in a*B_1(V_k),$ by providing moments and tail bound on
\[
U_I:= \sup_{\psi \in a*B_1(V_k)}
| \|\psi*X\|_{1,I}    - {\mathbb E}  \|\psi*X\|_{1,I}  |
\]

For clarity of the argument, we only state the concentration for $\|\psi*X\|_{1,{\cal T}}.$ Also the result about ${\cal T}$ would easy carry to a random subset $I$.

Let the minimal and maximal eigenvalue of the (convolutional) linear operator $a*$ be
\[
\lambda_{m}(a) := \inf_{y: \|y\|_2\leq 1} \frac{\|a*y\|_2}{\|y\|_2}
\]
\[
\lambda_{M}(a) := \sup_{y: \|y\|_2\leq 1} \frac{\|a*y\|_2}{\|y\|_2}
\]
\begin{lemma}
Using circluar embedding of Topelitz matrix, 
\[
\lambda_{M}(a) \leq \max| \hat{a}|
\]
\end{lemma}

Following the idea of lemma~$4$ of the paper of Adamczak on sparse dictionary learning\cite{adamczak2016note} and lemma~$16$ of the paper \cite{spielman2012exact}, we derive a similar bound for convolutional case.
\begin{lemma}
In the following, we define 
\[
Z :=  \frac{1}{N}\|w*a*X\|_{1,{\cal T}}
\]
We find the lower bound of ${\mathbb E} Z$ as a function of the $1-$norm  of $w$:
\[
{\mathbb E} Z  \geq \frac{1}{\sqrt{2}\pi}\lambda_{m}(a)\sqrt{\frac{p}{k}} \|w\|_{1} 
\]
For convenience, we define
\[
\mu_{\mbox{min}}:=\frac{1}{\sqrt{2}\pi}\lambda_{m}(a)\sqrt{\frac{p}{k}} \|w\|_{1}
\]

\end{lemma}
\begin{proof}
WOLG, we rescale $w$ so that $\|w\|_1 =1$, now
\begin{eqnarray*}
{\mathbb E}_X Z 
&=& \frac{1}{N}\sum_{j\in [-T,T] }{\mathbb E}_X |\sum_{i=1}^k w_i (a*X)_{j-i}|\\
&\geq& \inf_{\|w\|_{1} = 1}\frac{1}{N}\sum_{j\in [-T,T] }{\mathbb E}_X |\sum_{i=1}^k w_i (a*X)_{j-i}|
\end{eqnarray*}
By symmetry, we know the minimizer is $w = (\frac{1}{k}, \ldots,\frac{1}{k}),$
then we use the symmetrization trick to insert a sequence of $k$ IID Rademacher ($\pm 1$)  random variables 
$\varepsilon_1, \ldots, \varepsilon_k$ in the second inequality, and then apply Khintchine inequality as the third inequality,
\begin{eqnarray*}
{\mathbb E}_X Z 
&\geq&  \frac{1}{k} \frac{1}{N}\sum_{j\in [-T,T] }{\mathbb E}_X |\sum_{i=1}^k (a*X)_{j-i}|\\
&\geq& \frac{1}{2} \frac{1}{k} \frac{1}{N}\sum_{j\in [-T,T] }{\mathbb E}_X |\sum_{i=1}^k \varepsilon_i (a*X)_{j-i}|\\
&\geq&  \frac{1}{2} \frac{1}{k} \frac{1}{N}\sum_{j\in [-T,T] }\frac{1}{\sqrt{2}}{\mathbb E}_X \sqrt{ \sum_{i=1}^k  (a*X)^2_{j-i}}\\
&\geq&  \frac{1}{2} \frac{1}{k} \frac{1}{N}\sum_{j\in [-T,T] }\lambda_{m}(a)\frac{1}{\sqrt{2}}{\mathbb E}_X \sqrt{ \sum_{i=1}^k  X^2_{j-i}}\\
&\geq& \frac{1}{2}  \frac{1}{k} \frac{1}{\sqrt{2}}\frac{1}{N}\sum_{j\in [-T,T] }\lambda_{m}(a)\frac{2}{\pi}\sqrt{pk}\\
&=&  \frac{1}{\sqrt{2}\pi}\lambda_{m}(a)\sqrt{\frac{p}{k}}\\
\end{eqnarray*}
\end{proof}


Now we define the intersection of $1-$norm ball of the $k-$dimensional subspace $V_k$ as $B_1(V_k)$, then we consider $w\in B_1(V_k).$ 
\begin{lemma}
Hence we consider $\psi = a*w \in a*B_1(V_k),$ and we define
\[
W:= \sup_{\psi \in a*B_1(V_k)}
|\frac{1}{N}\sum_{j\in [-T,T] } (|(\psi*X)_{j}| - {\mathbb E} |(\psi*X)_{j}|) |
\]

For all $q\geq \max(2, \log(k)),$ there exists constant $C$,
\[
{\mathbb E} W^q\leq 2^{2q}(\frac{\lambda_{M}(a)}{N})^q   k C^q(\sqrt{Npq}+q)^q
\]
so
\begin{eqnarray*}
 \|W\|_q \leq \frac{4eC}{N} \lambda_{M}(a)  
 (\sqrt{Np(q+\log(k))}+q +\log(k))
\end{eqnarray*}
and
\begin{eqnarray*}
P( W > \frac{4eC\lambda_{M}(a) }{N}  
 (\sqrt{Np\log(\frac{k}{\delta})}+\log(\frac{k}{\delta})))\leq \delta
\end{eqnarray*}
\end{lemma}

\begin{proof}
Let $\varepsilon_1, \ldots, \varepsilon_N$ be a sequence of IID Rademacher ($\pm 1$)  random variables independent of $X$,  By the symmetrization inequality, see e.g. lemma~$6.3$ of book \cite{ledoux2013probability}, we have the first inequality. Then since the function $t \rightarrow |t| $ is a contraction, an application of Talagrand’s contraction principle (see lemma~$8$ of \cite{adamczak2016note}) with $F (x) = |x|^q$ 
conditionally on $X$ gives the second inequality.
\begin{eqnarray*}
{\mathbb E} W^q 
&\leq& 2^q {\mathbb E} \sup_{\psi \in a*B_1(V_k)} 
|\frac{1}{N}\sum_{j\in [-T,T] } \varepsilon_j |(\psi*X)_{j}| |^q\\
&\leq&
2^{2q}{\mathbb E} \sup_{\psi \in a*B_1(V_k)} |\frac{1}{N}\sum_{j\in [-T,T] } \varepsilon_j (\psi*X)_{j} |^q\\
&=&
2^{2q}\frac{1}{N^q}{\mathbb E} \sup_{\psi \in a*B_1(V_k)} |\sum_{i\in [-T,T] }\psi_{i}  \sum_{j\in [-T,T] } \varepsilon_j X_{j-i} |^q\\
&=&
2^{2q}\frac{1}{N^q}{\mathbb E} \sup_{w \in B_1(V_k)} |\sum_{i\in [-T,T] }(w*a)_{i}  \sum_{j\in [-T,T] } \varepsilon_j X_{j-i} |^q\\
&\leq&
2^{2q}\frac{1}{N^q}{\mathbb E} \max_{s\in [k]}|\sum_{i\in [-T,T] }a_{i-s}( \sum_{j\in [-T,T] } \varepsilon_j X_{j-i}) |^q\\
&\leq&
2^{2q}(\frac{\lambda_{M}(a)}{N})^q   \sum_{s\in [k] }{\mathbb E}| \sum_{j\in [-T,T] } \varepsilon_j X_{j-s} |^q
\end{eqnarray*}
Now by the moment version of Bernstein’s inequality (see lemma~$7$, equation $(18)$ of\cite{adamczak2016note}), we know that there exists a universal constant $C$,
\begin{eqnarray*}
{\mathbb E}| \sum_{j\in [-T,T] } \varepsilon_j X_{j-i} |^q \leq C^q(\sqrt{Npq}+q)^q
\end{eqnarray*}
Therefore, 
\begin{eqnarray*}
{\mathbb E} W^q &\leq& 2^{2q}(\frac{\lambda_{M}(a)}{N})^q    
\sum_{s\in [k] }{\mathbb E}| \sum_{j\in [-T,T] } \varepsilon_j X_{j-s} |^q \\
&\leq& 2^{2q}(\frac{\lambda_{M}(a)}{N})^q   k C^q(\sqrt{Npq}+q)^q
\end{eqnarray*}
Then
\begin{eqnarray*}
 \|W\|_q \leq \frac{4eC}{N} \lambda_{M}(a)  
 (\sqrt{Np(q+\log(k))}+q +\log(k))
\end{eqnarray*}
therefore, we could get the tail bound using the Chebyshev inequality for the moments,
\begin{eqnarray*}
P( W > \frac{4eC\lambda_{M}(a) }{N}  
 (\sqrt{Np(q+\log(k))}+q +\log(k)))\leq e^{-q}
\end{eqnarray*}
We set $q = \log(\frac{1}{\delta})$, then we will get an upper bound for $W$ which is satisfied with probability at least $1-\delta$:
\begin{eqnarray*}
P( W > \frac{4eC\lambda_{M}(a) }{N}  
 (\sqrt{Np\log(\frac{k}{\delta})}+\log(\frac{k}{\delta})))\leq \delta
\end{eqnarray*}
\end{proof}

\begin{lemma}
Let the condition number of the linear operator $a*$ be
\[
\kappa_a := \frac{\lambda_M(a)}{\lambda_m(a)}.
\]
If
\[
N\geq k\log(\frac{k}{\delta}) (2^7 \pi e^2 \frac{C^2}{\epsilon^2} \kappa_a^2 )
\]
and  $p\geq \frac{1}{N}$, 
then
\begin{eqnarray*}
P( W > \epsilon \mu_{\mbox{min}})\leq \delta
\end{eqnarray*}
\end{lemma}
\begin{proof}
If
\[
N\geq k\log(\frac{k}{\delta}) (2^7 \pi e^2 \frac{C^2}{\epsilon^2} \kappa_a^2 )
\]
then 
\[
\frac{4eC\lambda_{M}(a) }{N}  
 (\sqrt{Np\log(\frac{k}{\delta})} \leq \frac{1}{2}\epsilon (\frac{1}{\sqrt{2}\pi}\lambda_{m}(a)\sqrt{\frac{p}{k}}) \leq \frac{1}{2}\epsilon \mu_{\mbox{min}}
\]
additionally, if $p\geq \frac{1}{N}$, 
then a weaker inequality is also true:
\[
\frac{4eC\lambda_{M}(a) }{N}  
 \log(\frac{k}{\delta}) \leq \frac{1}{2}\epsilon (\frac{1}{\sqrt{2}\pi}\lambda_{m}(a)\sqrt{\frac{p}{k}}) \leq \frac{1}{2}\epsilon \mu_{\mbox{min}}
\]
therefore, combining the two parts and the previous inequality,
\begin{eqnarray*}
P( W > \frac{4eC\lambda_{M}(a) }{N}  
 (\sqrt{Np\log(\frac{k}{\delta})}+\log(\frac{k}{\delta})))\leq \delta
\end{eqnarray*}
we have
\begin{eqnarray*}
P( W > \epsilon \mu_{\mbox{min}})\leq \delta
\end{eqnarray*}
\end{proof}


\begin{theorem}
Let the solution of finite sample convex optimization be $w^\star$, 
for small $\epsilon>0$ and concentration level $\delta>0$, 
there exists universal constant $C$, if there are 
\[
N \geq k\log(\frac{k}{\delta}) ( \frac{C\kappa_a}{\epsilon}  )^2
\]
and  $ \frac{1}{N}\leq p< p^\star(\epsilon)$, we have 
\begin{eqnarray*}
P( W > \epsilon \mu_{\mbox{min}})\leq \delta.
\end{eqnarray*}

  \end{theorem}
\begin{proof}
From previous lemma, we know that with the conditions, 
\begin{eqnarray*}
P( W > \epsilon \mu_{\mbox{min}})\leq \delta.
\end{eqnarray*}
Therefore, with probability at least $1-\delta$,
\begin{eqnarray*}
W = \sup_{\psi \in a*B_1(V_k)}
|\frac{1}{N}\sum_{j\in [-T,T] } (|(\psi*X)_{j}| - {\mathbb E} |(\psi*X)_{j}|) | \leq \epsilon \mu_{\mbox{min}}
\end{eqnarray*}
When this is true, for all $\psi \in a*B_1(V_k)$, we have a uniform bound 
\[
 {\mathbb E} |(\psi*X)_{j}|-\epsilon\mu_{\mbox{min}} \leq \frac{1}{N}\sum_{j\in [-T,T] } (|(\psi*X)_{j}| \leq {\mathbb E} |(\psi*X)_{j}|+\epsilon\mu_{\mbox{min}}
\]
\end{proof}


\begin{coro}
For $\psi = a*w \in a*B_1(V_k),$ 
\[
U_I:= \sup_{\psi \in a*B_1(V_k)}
| \|\psi*X\|_{1,I}    - {\mathbb E}  \|\psi*X\|_{1,I}  |
\]
For all $q\geq \max(2, \log(k)),$ there exists constant $C$,
\[
{\mathbb E} U_I^q\leq 2^{2q}(\lambda_{M}(a))^q   k C^q(\sqrt{Npq}+q)^q
\]
so
\begin{eqnarray*}
 \|U_I\|_q \leq 4eC \lambda_{M}(a)  
 (\sqrt{Np(q+\log(k))}+q +\log(k))
\end{eqnarray*}
and
\begin{eqnarray*}
P( U_I > 4eC\lambda_{M}(a)  
 (\sqrt{Np\log(\frac{k}{\delta})}+\log(\frac{k}{\delta})))\leq \delta
\end{eqnarray*}
\end{coro}

\paragraph{Uniform bound of directional derivative}
We  define the uniform bound of directional derivative
\[
W_D  =\sup_{\|\beta\|_2=1, u^T \beta =0, \beta +e_0 \in a*B_1(V_k)}|\langle  \beta, {\frac{1}{N}\sum_{|i|\in {\cal T}} \nabla_{\phi}L_i}|_{\phi = 0} \rangle  -   \langle  \beta, {\mathbb E} \nabla_{\phi}L|_{\phi = 0} \rangle  | 
\]
Notice that
\begin{eqnarray*}
& &\langle  \beta, {\frac{1}{N}\sum_{|i|\in {\cal T}} \nabla_{\phi}L_i}|_{\phi = 0} \rangle  -   \langle  \beta, {\mathbb E} \nabla_{\phi}L|_{\phi = 0} \rangle  
\\
&=& 
\left[\frac{1}{N} \left(\|X\|_{1,J_N} \beta_0  +
 \|X*\beta_{(0)}\|_{1,{\cal T}-J_N}+ 
\sum_{i\in J_N} (1_{X_{-i}> 0}-1_{X_{-i}< 0})  \cdot X^{(i),T}\beta_{(0)}\right)\right]-
\\
& &
\left[\sqrt{\frac{2}{\pi}}(p \cdot \beta_0 + (1-p) \cdot {\mathbb E}_{I}[ \|\beta_{I}\|_2 | 0 \not \in I ])\right]
\end{eqnarray*}
From previous uniform bound, with high probability, we have the following three uniform bounds on
\[
|\frac{1}{N}\sum_{i\in J_N} (1_{X_{-i}> 0}-1_{X_{-i}< 0})  \cdot X^{(i),T}\beta_{(0)}|
\]
\[
|(\frac{1}{N} \|X\|_{1,J_N}-\sqrt{\frac{2}{\pi}}p)|
\]
\[
|\frac{1}{N}\|X*\beta_{(0)}\|_{1,{\cal T}-J_N}-(1-p) \cdot {\mathbb E}_{I}[ \|\beta_{I}\|_2 | 0 \not \in I ]|
\]
If $p<p^\star(u, \epsilon)$, we know from population result that
\[
\langle  \beta, {\mathbb E} \nabla_{\phi}L|_{\phi = 0} \rangle >\epsilon.
\]
If with high probability,  
\[
W_D  \leq \epsilon/2
\]
then using the convexity argument, 
for any direction $\beta \in V,$ the directional derivative at $\phi = 0$
\[
\langle  \beta, {\frac{1}{N}\sum_{|i|\in {\cal T}} \nabla_{\phi}L_i}|_{\phi = 0} \rangle  >0,
\]
then $a*w^\star-e_0=0$. 





















 








                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           


 
 

 


  







\subsection{$E{| \psi^T X| }$ for IID Bernoulli Gaussian $X$ }
we study in detail for the quantity  
\[
E{| \psi^T X| } = E{|\sum_{i} \psi_i X_i| }.
\]
We know when $X_i$ are unit variance,
\[
   E (\sum_i \psi_i X_i)^2 =  \sum_i   \psi_i^2 E(X_i)^2 = \|a\|_2^2.
\]
The ratio indicate the sparsity level of random variable $Z = \sum_i \psi_i X_i$
\[
\frac{ E{|\sum_{i} \psi_i X_i| }}{\sqrt{E (\sum_i \psi_i X_i)^2}} = \frac{ E{|\sum_{i} \psi_i X_i| }}{\|a\|_2} 
\]
Now, we consider some settings where exact calculation of $E{|\sum_{i} \psi_i X_i| }$ are possible.

\paragraph{Exact calculation for Bernoulli Gaussian}
Let $X_i = B_i G_i$ be IID Bernoulli Gaussian, where $B_i$ is sampled from Bernoulli with parameter $p$, $G_i$ is sampled from $N(0,1)$. Let $I$ be the support where $B_i =1$.
\[
E_X{|\sum_{i} \psi_i X_i| }  
= E_B E_G {|\sum_{i} \psi_i B_i G_i| }
= E_I E_G {|\sum_{i\in I} \psi_i G_i| }
\]
Leveraging the invariance property for linear transform of Gaussian, we get
\[
E_G {|\sum_{i\in I} \psi_i G_i| } = 
\sqrt{\frac{2}{\pi}} \cdot\|\psi_I\|_2.
\]
then
\[
E_X{|\sum_{i} \psi_i X_i| }  
= \sqrt{\frac{2}{\pi}} \cdot E_I \|\psi_I\|_2
= \sqrt{\frac{2}{\pi}} \cdot (1-(1-p)^{\|a\|_0})^{1/2}   \cdot  \|a\|_2.
\]
Here the last equality use the lemma below, and we define that $\|a\|_0= \#\{ t : \psi_t \neq 0\}$.
\begin{lemma} (Expectation of $2-$norm over Bernoulli support)
\label{lem:BLemma}
\begin{equation}
E_I \|\psi_I\|_2 = (1-(1-p)^{\|a\|_0})^{1/2}   \cdot  \|a\|_2
\end{equation}
\end{lemma}
\begin{proof}[Proof of Lemma \ref{lem:BLemma}]
We have the power series representation $\sqrt{x} = \sum_k c_k x^k$ valid on $(1/A,A)$
for large $A \gg 1$. 
Let $N = |{\cal T}|$. Fix $A > 1/{\varepsilon}^2$; then every sum $ \sum_{t \in I} \psi_t^2 \in (1/A,A)$;
using this:
\begin{eqnarray*}
 \sum_{I \subset {\cal T}} \pi(I)   ( \sum_{t \in I} \psi_t^2)^{1/2}  &=&  
 \sum_{I \subset {\cal T}} \pi(I)   \sum_k  c_k \cdot ( \sum_{t \in I} \psi_t^2  )^k \\
 &=& \sum_{I \subset {\cal T}} \pi(I)   \sum_k  c_k \sum_{t_1,\dots, t_k \in I} \prod_{i=1}^k \psi_{t_i}^2  \\
 &=&   \sum_k  c_k \sum_{I \subset {\cal T}} \pi(I)   \sum_{t_1,\dots, t_k \in I} \prod_{i=1}^k \psi_{t_i}^2  \\
 \end{eqnarray*}
 Hence from $ \pi(I) = (p)^{|I|} (1-p)^{|{\cal T} \Delta I|}$
 \begin{eqnarray*}
  \sum_{I \subset {\cal T}} \pi(I)   ( \sum_{t \in I} \psi_t^2)^{1/2}  
 &=&  \sum_k c_k   \sum_{n=1}^{N}  \sum_{|I| = n} (p)^n (1-p)^{N-n}  \cdot  \sum_{t_1,\dots, t_k \in I} \prod_{i=1}^k \psi_{t_i}^2  \\
&=&  \sum_k c_k   \sum_{n=1}^{N}  (p)^n (1-p)^{N-n}  \cdot \sum_{|I| = n} \sum_{t_1,\dots, t_k \in I} \prod_{i=1}^k \psi_{t_i}^2 
 \end{eqnarray*}
 Since the inner two sums  cover $ {N \choose n} n^k$ terms in a uniform way,
 we can write
 \begin{eqnarray*}
  \sum_{I \subset {\cal T}} \pi(I)   ( \sum_{t \in I} \psi_t^2)^{1/2}  
&=&  \sum_k c_k   \sum_{n=1}^{N}  (p)^n (1-p)^{N-n}  \cdot \sum_{|I| = n} \sum_{t_1,\dots, t_k \in I} \prod_{i=1}^k \psi_{t_i}^2  \\
 &=&  \sum_k c_k   \sum_{n=1}^{N}  (p)^n (1-p)^{N-n}  \cdot {N \choose n}  \cdot Ave_{|I| = n}  \cdot {n^k} \cdot  Ave_{t_1,\dots, t_k \in I} \; \prod_{i=1}^k \psi_{t_i}^2  \\
&=&  \sum_k c_k   \sum_{n=1}^{N}  (p)^n (1-p)^{N-n}  \cdot {N \choose n}  \cdot {n^k} \cdot Ave_{t_1,\dots, t_k \in {\cal T}} \; \prod_{i=1}^k \psi_{t_i}^2  \\
 &=&  \sum_k c_k   \cdot \left( \sum_{n=1}^{N}  (p)^n (1-p)^{N-n}  \cdot {N \choose n} \right)  \cdot ( \sum_{t \in {\cal T}} \psi_t^2  )^k \\
 \end{eqnarray*}
Now use the identity $1 = \sum_{n=0}^{N}  (p)^n (1-p)^{N-n}  \cdot {N \choose n}$  for the complete binomial sum, 
hence the incomplete sum $\sum_{n=1}^{N}  (p)^n (1-p)^{N-n}  \cdot {N \choose n}= 1-(1-p)^N$. We get
 \begin{eqnarray*}
  \sum_{I \subset {\cal T}} \pi(I)   ( \sum_{t \in I} \psi_t^2)^{1/2}  
 &=&  \sum_k c_k   \cdot \left( 1-(1-p)^N \right)  \cdot ( \sum_{t \in {\cal T}} \psi_t^2  )^k \\
 &=& \left( 1-(1-p)^N \right) \cdot \sum_k c_k  \left ( \sum_{t \in {\cal T}} \psi_t^2  \right )^k \\
 &=&  \left( 1-(1-p)^N \right) \cdot  \sqrt{\sum_{t \in {\cal T}} \psi_t^2 } .
\end{eqnarray*}
\end{proof}
\section{Technical background}









  
 
  
  
  
  
  
  
  
  
  
  
  
  
  
\section{A common population convex problem: projection pursuit and its phase transition}
In this section, we will show that after change of variables, the population version of blind deconvolution is the same as the population version of sparse projection pursuit, which is a convex formulation used for dictionary learning or sparse blind source separation. 

Therefore, the phase transition phenomenon presented above is a fundamental phase transition that works for a simple but fundamental population convex problem.


\paragraph{Change of variable for  convex sparse blind deconvolution}
We could rewrite the population version of our convex sparse blind deconvolution problem, with the population objective 
${\mathbb E}\frac{1}{N} \| w*Y\|_{\ell_1({\cal T})} ={\mathbb E}|(w*Y)_0|={\mathbb E}|(w*a*X)_0|$ due to  the ergodic property of stationary process and shift invariance. Then the convex problem becomes 
\[
 \begin{array}{ll}
 \underset{\psi}{\mbox{minimize}}   & {\mathbb E}|(w*a*X)_0|\\
\mbox{subject to}  
& ((\alpha*a^{-1})*(a*w))_0 =1,
\end{array}
 \]
Let $\psi$ be the flipped version of $a*w$, such that $\tilde{\psi} := a*w$, then 
 $(w*Y)_0 = (w*a*X)_0 = (\tilde{\psi}*X)_0 = \psi^T X.$  
  Let $u:= \alpha*a^{-1}$. 
 Now we arrive at a simple and fundamental  population convex problem: 
 \[
 \begin{array}{ll}
 \underset{\psi}{\mbox{minimize}}   & {\mathbb E}|\psi^T X|\\
\mbox{subject to}  
& u^T \psi =1.
\end{array}
\tag{$P_{u}$ }
  \]
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
In the following, we show that this population convex problem is also the population convex problem of sparse projection pursuit after change of variables.

\paragraph{Sparse projection pursuit and dictionary learning}
Sparse projection pursuit is the problem of finding  one sparse element $d^T Y$ each time from a high dimensional data matrix $Y\in {\mathbb R}^{n\times M}$. Here $Y= D X$ is a matrix product of dictionary $D\in {\mathbb R}^{n\times d}$ and sparse coefficient $X\in {\mathbb R}^{d\times M}$. 


The sparse projection pursuit formulation is to solve the following linear-constrained convex optimization problem for some vector $v$:
\[
 \begin{array}{ll}
 \underset{q }{\mbox{minimize}}   & \frac{1}{M}\|q^T Y\|_{1}\\
\mbox{subject to}  & v^T q =1.
\end{array}
  \]
 
  
The population version is 
\[
 \begin{array}{ll}
 \underset{d }{\mbox{minimize}}   & {\mathbb E}|q^T D X|\\
\mbox{subject to}  & v^T q =1.
\end{array}
  \]
We could also do a change of variable by assuming $\psi:= D^T q$, $u:=D^T q$, then the population sparse projection pursuit also becomes
   \[
 \begin{array}{ll}
 \underset{\psi}{\mbox{minimize}}   & {\mathbb E}|\psi^T X|\\
\mbox{subject to}  & u^T \psi =1.
\end{array}
\tag{$P_{u}$ }
  \]
Without loss of generality, we could assume $\|u\|_2=1$ by rescaling.
  
  
  
  
  
  








\paragraph{General phase transition for projection pursuit}
Now we state the general phase transition with light assumption. Here we don't need to even assume that $X$ are independent, we just need to know the existence of one sparse element $X_0$.

\begin{theorem}[Projection pursuit to find a sparse element]
Given a sequence $\{X_t\}_{t\in {\mathbb Z}}$ where one element $X_0$ is a sparse element with $p G + (1-p) \delta_0$ for a symmetric distribution $G$.
Let $\psi^\star$ be the solution of  $P_{u}$:
\[
 \begin{array}{ll}
 \underset{\psi}{\mbox{minimize}}   & {\mathbb E}|\psi^T X|\\
\mbox{subject to}  
& u^T \psi =1
\end{array}
\tag{$P_{u}$ }
  \]
then there exists a threshold
$
p^\star
$,
$\psi^\star = e_0$ provided $p<p^\star$,
and  $\psi^\star \neq e_0$ provided $p> p^\star$.

Here $p^\star$ is determined by $G$, the distribution of $\{X_t, t\neq 0\}$ and the direction of $u$:
  \[
  \frac{p^\star}{1-p^\star} =  \inf_{\| \beta\|_2=1, u^T \beta =0} \frac{{\mathbb E}[|X_{(0)}^T \beta_{(0)}| \mid X_0 =0]}{{\mathbb E}[|G|]\|\beta_{(0)}\|_2} \tan(\angle( \beta, -e_0))
  \]
 where $\beta_{(0)}$ is $\beta$ with the $0-$th entry deleted.
   

  \end{theorem}

\begin{proof}
The phase transition condition proof is based on  calculating the directional derivative.

First, we want to find a threshold $p^\star$ such that for all $p<p^\star$, the directional derivative at $\psi = e_0$ is non-negative along all direction $\beta$ on unit sphere such that $u^T \beta =0$:
	 \[
	 \nabla_{\beta}{{\mathbb E}|(e_0+\beta)^T X|}\geq 0
	 \]
Second, we calculate $\nabla_{\beta}{{\mathbb E}|(e_0+\beta)^T X|} $ based on whether $X_0$ is zero or not:
	 \[
	 \begin{array}{ll}
	     \nabla_{\beta}{{\mathbb E}|(e_0+\beta)^T X|} = p \beta_0 {\mathbb E}[|X_{0}| \mid X_0 \neq 0]  + (1-p) {\mathbb E}[|X_{(0)}^T \beta_{(0)}| \mid X_0 =0]
	 \end{array}
	\]
It is non-negative in case either $\beta_{0} > 0$, or in case $\beta_0 < 0$ and
	\[
	\frac{p}{1-p} < \frac{{\mathbb E}[|X_{(0)}^T \beta_{(0)}| \mid X_0 =0]}{{\mathbb E}[|G|]||\beta_0|}
	\]
	for all $\beta \in \{\| \beta\|_2=1, u^T \beta =0\}$.
$p^\star$ is the least upper bound of all $p$ satisfying this inequality. 
Therefore,  
	\[
	\frac{p^\star}{1-p^\star} =  \inf_{\| \beta\|_2=1, u^T \beta =0} \frac{{\mathbb E}[|X_{(0)}^T \beta_{(0)}| \mid X_0 =0]}{{\mathbb E}[|G|]|\beta_0|} =  \inf_{\| \beta\|_2=1, u^T \beta =0} \frac{{\mathbb E}[|X_{(0)}^T \beta_{(0)}| \mid X_0 =0]}{{\mathbb E}[|G|]\|\beta_{(0)}\|_2} \tan(\angle( \beta, -e_0))
	\]
\end{proof}



\paragraph{Special case: geometric phase transition condition for sparse Bernoulli Gaussian $X_0$ and Gaussian $\{X_t, t\neq 0\}$}

Now we study the implication of the general phase transition problem in a simplest projection pursuit case, where $X_t$ are independent, $X_0$ is sampled from Bernoulli Gaussian $p N(0,1) +(1-p) \delta_0,$ and the rest $\{X_t, t\neq 0\}$ are sampled from Gaussian $N(0,1)$.

\begin{theorem}
Let $X_t$ be independent, $X_0$ is sampled from Bernoulli Gaussian $p N(0,1) +(1-p) \delta_0,$ the rest $\{X_t, t\neq 0\}$ are sampled from Gaussian $N(0,1)$.
Let $\psi^\star$ be the solution of  $P_{u}$:
\[
 \begin{array}{ll}
 \underset{\psi}{\mbox{minimize}}   & {\mathbb E}|\psi^T X|\\
\mbox{subject to}  
& u^T \psi =1
\end{array}
\tag{$P_{u}$ }
  \]
then there exists a threshold
$
p^\star
$,
$\psi^\star = e_0$ provided $p<p^\star$,
and  $\psi^\star \neq e_0$ provided $p> p^\star$.

Here $p^\star$ is determined by the direction of $u$:
  \[
  \frac{p^\star}{1-p^\star} = \inf_{\| \beta\|_2=1, u^T \beta =0}\tan(\angle( \beta, -e_0)) = \cot(\angle(u, e_0))
  \]
  \end{theorem}
\begin{proof}
First, when the rest of $X_t, t\neq 0$ is independent of $X_0$, and IID sampled from $ N(0,1),$
	\[
 \frac{{\mathbb E}[|X_{(0)}^T \beta_{(0)}| \mid X_0 =0]}{\|\beta_{(0)}\|_2} = 
\sqrt{\frac{2}{\pi}}
\]
Since  
\[
	 {\mathbb E}[|G|] = \sqrt{\frac{2}{\pi}},
	 \]
we have a simple phase transition condition that is completely geometric.
	 \[
  \frac{p^\star}{1-p^\star} = \inf_{\| \beta\|_2=1, u^T \beta =0}\tan(\angle( \beta, -e_0))
  \]
 
Second, we know that there is a geometric structure:
\[
\inf_{\| \beta\|_2=1, u^T \beta =0} \frac{\|\beta_{(0)}\|_2}{|\beta_0|}
= \inf_{\| \beta\|_2=1, u^T \beta =0}\tan(\angle( \beta, -e_0)) = \cot(\angle(u, e_0)),
\]
as indicated by figure~\ref{fig-Angle}. 
When $\beta = P_u^\perp (-e_0)$,
	\[
	\begin{array}{ll}
	\inf_{\| \beta\|_2=1, u^T \beta =0} 
		\tan(\angle( \beta, -e_0)) 
		&=
		\tan(\angle( P_u^\perp (-e_0), -e_0)) \\
		&= \cot(\angle(u, e_0)).
	\end{array}
	\]
	
Therefore, $ \frac{p^\star}{1-p^\star}  = \cot(\angle(u, e_0)) $,
	

\end{proof}
    
\begin{figure} 
\centering
\includegraphics[width=.8\textwidth]{./figures_BD/Phase_T_sine_p.png}
\caption{The population phase transition diagram of $ \frac{p^\star}{1-p^\star}  = \cot(\angle(u, e_0)) $ for the special case when $X_t$ is independent, $X_0$ is sampled from Bernoulli Gaussian $p N(0,1) +(1-p) \delta_0,$ and the rest $\{X_t, t\neq 0\}$ are sampled from Gaussian $N(0,1)$. The red region is failure, and the blue region is success. }
\label{fig-PT-project}
\end{figure}



\section{Stability for approximation of infinite length inverse filter}

 
 
 

\paragraph{Reduction}
 \begin{lemma} 
 Let $w^\star \in V_{rs, rt} $ be the solution of the convex optimization problem
 \[
 \begin{array}{ll}
  \underset{w}{ \mbox{minimize} }  &  {\mathbb E}|(w*Y)_0|\\
 \mbox{subject to}  
&  w_0 =1, \quad w \in V_{rs, rt}.
\end{array}
  \]
 Then the previous convex optimization problem can be reduced to 
  \[
 \begin{array}{ll}
  \underset{w}{ \mbox{minimize} }  & B(p, \phi) \|\phi \|_2\\
 \mbox{subject to}  
&  \phi = a*w -e_0, \quad w_0 =1, \quad w \in V_{rs, rt}.
\end{array}
  \]
  where
  \[
B(p, \phi) := \frac{{\mathbb E}_I  \| (e_0+\phi)_I \|_2 - {\mathbb E}_I  \| (e_0)_I \|_2}{\|\phi\|_2} 
\]
 \end{lemma} 
  
  
\begin{proof}
Following the previous phase transition analysis, 
let the directional finite difference of the objective at $e_0$ be $D(w) = |(w*Y)_0| - |X_0|$, $I$ be the support of $\tilde{X}$, let $\phi = w*a-e_0$,  then we have 
\[
 \begin{array}{ll}
{\mathbb E} D(w) &= \sqrt{\frac{2}{\pi}}{\mathbb E}_I [\|(w*a)_I\|_2 - \|(e_0)_I\|_2]\\
&= \sqrt{\frac{2}{\pi}}{\mathbb E}_I [\|(e_0+\phi)_I\|_2 - \|(e_0)_I\|_2]\\
&= \sqrt{\frac{2}{\pi}}\|{\phi}\|_2 B(p, \phi)  \\
\end{array}
  \]
The last inequality comes from the bi-Lipschitz property of ${\mathbb E}_I \|(e_0+\phi)_I\|_2$ around $e_0$. 








Now using the optimality of $w^\star$ in $V_{rs, rt}$, for all $w in V_{s,t}, w\neq w^\star$,
we have
 \[
 \begin{array}{ll}
{\mathbb E} D(w^\star) \leq {\mathbb E} D(w)
\end{array}
  \]
We could directly reduce the problem to 
  \[
 \begin{array}{ll}
B(p, \phi^\star) \|{\phi^\star}\|_2 \leq B(p, \phi)\|{\phi}\|_2 
\end{array}
  \]
\end{proof}   
 


\begin{coro}
If there exists a $\phi$ with $\|{\phi}\|_2<1$ such that \[
 \begin{array}{ll}
B(p, \phi^\star) \|{\phi^\star}\|_2 \leq B(p, \phi)\|{\phi}\|_2
\end{array}
  \]
 for  $\epsilon> 0$ when $p\leq p^\star(\epsilon)$, then 
  \[
  B(p, \phi) \geq \epsilon.
  \]
  for all $\phi$, 
  then
    \[
 \begin{array}{ll}
\|{\phi^\star}\|_2 \leq \frac{(1-p)\sqrt{p}  + 
p\Big(1  +
  \frac{\|{\phi}\|_2}{2} 
 +
 p\frac{\|{\phi}\|_2}{2}   \Big)}{\epsilon}\|{\phi}\|_2
\end{array}
  \]
  is also bounded.
\end{coro}
\begin{proof}
From
 \[
 \begin{array}{ll}
B(p, \phi^\star) \|{\phi^\star}\|_2 \leq B(p, \phi)\|{\phi}\|_2
\end{array}
  \]
  We have
  \[
 \begin{array}{ll}
\|{\phi^\star}\|_2 
&\leq \frac{B(p, \phi)}{B(p, \phi^\star)}
\|{\phi}\|_2 \\
&\leq \frac{(1-p)\Big( V_1(\phi_{{(0)}}) \sqrt{1-\phi_0^2}\Big) + 
p\Big(\phi_0  +
  \frac{\|{\phi}\|_2}{2} \phi_0^2 
 +
 p\frac{\|{\phi}\|_2}{2}  (1-\phi_0^2) \Big)}{(1-p)\Big( V_1(\phi^\star_{{(0)}}) \sqrt{1-(\phi^\star_0)^2}\Big) + 
p\Big(\frac{1}{\|{\phi^\star}\|_2} (\left| 1 + \|{\phi^\star}\|_2 \phi^\star_0 \right|-1 \Big)}\|{\phi}\|_2 \\
&\leq \frac{(1-p)\sqrt{p}  + 
p\Big(1  +
  \frac{\|{\phi}\|_2}{2} 
 +
 p\frac{\|{\phi}\|_2}{2}   \Big)}{\epsilon}\|{\phi}\|_2
\end{array}
  \]
  \end{proof}
  In the following we constructed a filter to give an upper bound on $\|{\phi}\|_2$. 


\paragraph{Setting for finite length approximation}
Now we consider a setting where there is a kernel whose corresponding inverse kernel has infinite support, and we give a finite-length approximation. 

Within the space $V_{-\infty,\infty}$ of bilaterally infinite real-valued sequences $(h(i))_{i \in {\mathbb Z}}$, consider the affine subspace $V_{N_{-}, N_{+}} = \{ (...,0,0,h_{-N_{-}},\ldots, h_{-1}, 1, h_1, h_2, \ldots, h_{N_{+}},0,0,...)\}$, an $N_{-}+N_{+}$-dimensional subspace of bilateral sequences with support at most $N_{-}+N_{+}+1$.
The coordinate that is fixed to one is 
located at index $i=0$.
Each coordinate is zero outside of a window
of size $N_{-}$ on the left of zero 
and  size $N_{+}$ on the right of zero.
The special sequence $e_0 = (\dots,0,0,1,0,0,\dots)$,
vanishing everywhere except the origin,
belongs to $V_{-\infty,\infty}$ and to every
$V_{N_-,N_+}$.

Let $a \in V_{N_-,N_+}$.
Then $a =  (...,0,0,a_{-N_{-}},\ldots, a_{-1}, 1, a_1, a_2, \ldots, a_{N_{+}},0,0,...)$.
We also write $a = e_0 + a_{(0)}$,
where $a_{(0)} \equiv (1-e_0) \cdot a$ 
denotes the `part of $a$ supported away from location
$i=0$'. We also write $a = e_0 + a_L + a_R$,
where $a_L = (a_L(i))_{i \in {\mathbb Z}} 
= (a(i) 1_{\{i < 0\}})_{i \in {\mathbb Z}}$ 
denotes the `part of $a$ supported to
the left of $i=0$' and 
where $a_R = (a_R(i))_{i \in {\mathbb Z}} 
= (a(i) 1_{\{i > 0\}})_{i \in {\mathbb Z}}$
denotes the `part of $a$
supported to the right of $i=0$'.
Finally, we say that $a \in V_{N_{-}, N_{+}}$ 
is a length $ L= N_{-}+N_{+}+1$ filter.

An interesting, if not implementable, objective
function for the blind deconvolution problem
is to minimize
${\mathbb E} [|(w*a*X)_0| - |X_0|] $,
particularly when combined with constraints 
somehow ensuring ${{\mathbb E}} |(w*a*X)_0|^2 \geq {{\mathbb E}}|X_0|^2$.
Defining $\phi \equiv w*a-e_0$, 
\[
 \begin{array}{lll}
{\mathbb E} [|(w*a*X)_0| - |X_0|] &=&{\mathbb E} [|((e_0 +\phi)*X)_0| - |X_0|].
\end{array}
  \]
  Below we establish the relation
  \[
   \begin{array}{lll}
{\mathbb E} [|((e_0 +\phi)*X)_0| - |X_0|] 
&\approx&  C \|{\phi}\|_2 ; \\
\end{array}
\]
This suggests that we should  seek to
minimize ${\mathbb E} |(w*Y)_0| $,
while constraining $w$ in some way that
ensures ${{\mathbb E}} |(w*a*X)_0|^2 \geq {{\mathbb E}}|X_0|^2$.
This  the deterministic objective as 
$$
d_a(w, a^{-1}) :=\|{\phi}\|_2 = \|a*w -e_0 \|_2 =  \|a*(w -a^{-1}) \|_2.
$$

Therefore, we want to construct a finite length approximation of $a^{-1}$ to bound $d_a(w^\star, a^{-1}) = \|a*w -e_0 \|_2$ for the optimal point $w^\star$.








We construct the finite length approximation filter explicitly by truncation of Z-transform.

Let the Z-transform of $a$ be 
\[
 \begin{array}{ll}
A(z) =  \sum_{i=-s}^t a_{i} z^{-i},
\end{array}
  \]
Then Z-transform of the inverse kernel $a^{-1}$ is $1/A(z)$.






 
\subsection{ Stability for factorized filter}
\subsection{Examples}
In the following, we study the simple case when there is only one root.
Let $s\in (-1,1)$. For $a=(\ldots, 0, 1, -s, 0, \ldots)$, $a^{-1} = (\ldots, 0, 1, s, s^2, s^3, \ldots)$, we could construct 
\[
w^{r,1}:=  (\ldots, 0, 1, s, s^2, s^3, \ldots, s^r)
\]

If we choose this approximation $w^{r,1},$ then
\[
\|a*w^{r,1}-e_0\|_2^2 = \|*_{r+1} (\bar a^{\gamma})\|_2^2 = s^{2(r+1)} 
\]

In the following, we show that this is indeed the best 

For $\gamma\in (0,1]$,
\[
\bar a^{\gamma} :=e_0-\gamma a= (\ldots, 0, 1-\gamma,  \gamma s, 0, \ldots)
\]
\[
 \begin{array}{ll}
w^{r,\gamma} 
&= a^{-1} - a^{-1}*_{r+1} (\bar a^{\gamma})
\end{array}
\]
\[
\begin{array}{ll}
a*(a^{-1}-w^{r,\gamma}) 
&= *_{r+1} (\bar a^{\gamma})\\
&= *_{r+1} (\ldots, 0, 1-\gamma,  \gamma s, 0, \ldots)
\end{array}
\]

\[
\mathcal{F}[(\ldots, 0, 1-\gamma, \gamma s, 0, \ldots)](t)  = (1-\gamma)+   \gamma s  \cdot e^{2\pi i  t}, \quad  t\in T = [\frac{-1}{2}, \frac{1}{2}];
\]

\[
|\mathcal{F}[(\ldots, 0, 1-\gamma, \gamma s, 0, \ldots)] |^2(t) = (1-\gamma)^2+ (\gamma s )^2+ 2 {(1-\gamma)} {\gamma}s \cos{(2\pi  t)} , \quad t\in T = [\frac{-1}{2}, \frac{1}{2}];
\]

\[
 \begin{array}{ll}
\|*_{r+1} (\bar a^{\gamma})\|_2^2 
&= \| (\hat{\bar a}^{\gamma})^{r+1}\|_2^2\\
&=  \int_{\frac{-1}{2}}^\frac{1}{2}  |(1-\gamma)+   \gamma s  \cdot e^{2\pi i  t}|^{2r+2} dt\\
&= \int_{\frac{-1}{2}}^\frac{1}{2}  [(1-\gamma)^2+ (\gamma s )^2+ 2 {(1-\gamma)} {\gamma}s \cos{(2\pi  t)} ]^{r+1} dt
\end{array} \]
Notice the following lemma, 
\begin{lemma}
For odd number, $\int_{\frac{-1}{2}}^\frac{1}{2}  \cos^{2k-1}{(2\pi  t)}  dt = 0$ for any $k\in {\mathbb Z}_+$; for even number, 
\[
\int_{\frac{-1}{2}}^\frac{1}{2}  \cos^{2k}{(2\pi  t)}  dt =  {2k\choose k} 2^{-2k},
\]
\end{lemma}
\begin{proof}
\[
 \begin{array}{ll}
2^n\cos^n (2\pi  t)=(e^{i \cdot 2\pi  t}+e^{-i \cdot 2\pi  t})^n=\sum_{k=0}^n {n\choose k} (e^{i\cdot 2\pi  t})^k\ (e^{-i\cdot 2\pi  t})^{n-k}=\sum_{k=0}^n {n\choose k} e^{(2k-n)i\cdot 2\pi  t},
\end{array} \]

\[
 \begin{array}{ll}
\int_{\frac{-1}{2}}^\frac{1}{2} e^{(2k-n)i\cdot 2\pi  t} dt= \delta_{2k-n}
\end{array} \]
\[
 \begin{array}{ll}
\int_{\frac{-1}{2}}^\frac{1}{2} 2^n\cos^n (2\pi  t) dt =\sum_{k=0}^n {n\choose k} \delta_{2k-n},
\end{array} \]
therefore, for odd number $n=2k-1$, $\int_{\frac{-1}{2}}^\frac{1}{2}  \cos^{2k-1}{(2\pi  t)}  dt = 0$ for any $k\in {\mathbb Z}_+$; for even number $n=2k$, 
\[
\int_{\frac{-1}{2}}^\frac{1}{2} 2^{2k} \cos^{2k}{(2\pi  t)}  dt =  {2k\choose k}.
\]
\end{proof}

\begin{lemma}
Now
$R_1 :=  (1-\gamma)^2+ (\gamma s )^2,$
$R_2 := (1-\gamma) \gamma s,$
\[
 \begin{array}{ll}
 \| (\hat{\bar a}^{\gamma})^{r+1}\|_2^2
=\sum_{k=0}^{\frac{r+1}{2}} [(1-\gamma)^2+ (\gamma s )^2]^{r+1-2k} [ (1-\gamma) \gamma s]^{2k}{2k\choose k}{r+1\choose 2k} 
\end{array} \]
\end{lemma}
\begin{proof}
Now define
$R_1 :=  (1-\gamma)^2+ (\gamma s )^2,$
$R_2 := {(1-\gamma)} {\gamma}s,$
\[
 \begin{array}{ll}
 \| (\hat{\bar a}^{\gamma})^{r+1}\|_2^2
&= \int_{\frac{-1}{2}}^\frac{1}{2}  [(1-\gamma)^2+ (\gamma s )^2+ 2 {(1-\gamma)} {\gamma}s \cos{(2\pi  t)} ]^{r+1} dt\\
&= \int_{\frac{-1}{2}}^\frac{1}{2}  
[R_1+ R_2 \cdot 2 \cos{(2\pi  t)} ]^{r+1} dt\\
&=\sum_{k=0}^{\frac{r+1}{2}} R_1^{r+1-2k} R_2^{2k}{2k\choose k}{r+1\choose 2k} 
\end{array} \]
\end{proof}

Use the Laplace's method theorem 
We could instead directly optimize  $\gamma$ over the $\ell_\infty$ norm of $|\hat{\bar a}^{\gamma}|^2$,
\begin{lemma}
\[
 \||\hat{\bar a}^{\gamma}|^2\|_\infty = \max_{ t\in [\frac{-1}{2}, \frac{1}{2}]}\{ (1-\gamma)^2 + (\gamma s)^2+ 2 (1-\gamma) \gamma s \cos{(2\pi  t)} \}  = (1-\gamma)^2 + (\gamma s)^2+ 2 |(1-\gamma) \gamma s| = (|1-\gamma|+|\gamma s|)^2 
\]
then

 \[
  \begin{array}{ll}
\gamma_\infty &= \arg\min_{\gamma>0}\||\hat{\bar a}^{\gamma}|^2\|_\infty \\
&= \arg\min_{\gamma>0}(|1-\gamma|+|\gamma s|)^2 \\
&= 1
 \end{array} \]
\[
\min_{\gamma\in (0,1)}\||\hat{\bar a}^{\gamma}|^2\|_\infty
= (|1-\gamma_\infty|+|\gamma_\infty s|)^2 
= |s|^2
\]
\end{lemma}
And
 \[
 \begin{array}{ll}
\|*_{r+1} (\bar a^{\gamma_\infty  })\|_2^2 
&=  s^{2r+2} 
\end{array} \]

 


\paragraph{Finite length approximation constructing using Z-transform truncation on factorization form }
Let the Z-transform of $a$ be 
\[
 \begin{array}{ll}
A(z) =  \sum_{i=-N_{-}}^{N_{+}} a_{i} z^{-i},
\end{array}
  \]
We can factorize the Z-transform as product of factors with roots $s_{-j}$ for $j = 1, \ldots N_{-}$ and  $i = 1, \ldots N_{+}.$
\[
 \begin{array}{ll}
A(z) =  c_0\prod_{j=1}^{N_{-}} (1- s_{-j} z) \prod_{i=1}^{N_{+}} (1- s_{i} z^{-1}),
\end{array}
  \]
  Where $c_0$ is a constant to make sure that the coefficient $a_0=1$.
  
 We assume that $|s_k|<1,$ for any $k = -N_{-}, \ldots , N_+.$  
  
Then Z-transform of the inverse kernel $a^{-1}$ is $1/A(z)$.
\[
 \begin{array}{ll}
1/A(z) &=  \frac{1}{c_0}\prod_{j=1}^{N_{-}} (1- s_{-j} z)^{-1} \prod_{i=1}^{N_{+}} (1- s_{i} z^{-1})^{-1}\\
&=  \frac{1}{c_0}\prod_{j=1}^{N_{-}} (\sum_{\ell_j=0}^{\infty} s^{\ell_j}_{-j} z^{-\ell_j} ) \prod_{i=1}^{N_{+}} (\sum_{\ell_i=0}^{\infty} s^{\ell_i}_{i} z^{\ell_i} )
\end{array}
  \]

From previous single root study, we get a hint to study a single truncation of length $ 1 + \sum_{j=1}^{N_{-}} r_{-j}  +  \sum_{j=1}^{N_{+}}  r_i$,  we construct
\[
 \begin{array}{ll}
W(z) 
&=  \frac{1}{c_0}\prod_{j=1}^{N_{-}} (\sum_{\ell_j=0}^{r_{-j}} s^{\ell_j}_{-j} z^{-\ell_j} ) \prod_{i=1}^{N_{+}} (\sum_{\ell_i=0}^{r_{-j}} s^{\ell_i}_{i} z^{\ell_i} )\\
&=  \frac{1}{c_0}\prod_{j=1}^{N_{-}} (1- (s_{-j} z)^{r_{-j}+1})(1- s_{-j} z)^{-1} \prod_{i=1}^{N_{+}}(1- (s_{i}  z^{-1})^{r_i+1}) (1- s_{i} z^{-1})^{-1}
\end{array}
  \]
Let the corresponding filter being $w$.
  
  
Now let $\psi =  a*w$ with Z-transform $\Psi(z)$, $\phi = a*w-e_0$ with Z-transform $\Phi(z)$,we have
\[
 \begin{array}{ll}
\Psi(z): = A(z)W(z) 
&=   \prod_{j=1}^{N_{-}} (1- (s_{-j} z)^{r_{-j}+1}) \prod_{i=1}^{N_{+}}(1- (s_{i} z^{-1})^{r_i+1}) 
\end{array}
  \]
  therefore,
\[
 \begin{array}{ll}
\Phi(z): = A(z)W(z) -1 
&=  \prod_{j=1}^{N_{-}} (1- (s_{-j} z)^{r_{-j}+1}) \prod_{i=1}^{N_{+}}(1- (s_{i} z^{-1})^{r_i+1}) -1
\end{array}
  \]
\paragraph{Upper Bound on  $\|\phi\|_2^2$ for constructed finite length approximation}
In the following,  we want to give upper bound for
  \[
\|a*w -e_0\|_2^2 =\|\phi\|_2^2 
 \]  
  
Redefine the index $\tau_{-j} := r_{-j}+1; \tau_{i} = r_i+1.$
Let ${\cal I} = \{-N_{-},\ldots, -1, 1, \ldots, N_{+} \}$ as the set of all the possible index.
And we define polar representation of complex roots:
for any $k \in {\cal I} = \{-N_{-},\ldots, -1, 1, \ldots, N_{+} \}$
\[
s_{k} := |s_{k}| e^{i \varphi_k } = e^{\rho_k + i \varphi_k }
\]
where 
\[
 \begin{array}{ll}
\varphi_k := \operatorname{Im}(\log( s_{k} )) = -i \log(\frac{s_{k}}{|s_{k}|}) 
\end{array}
\]
and since $|s_{k}|<1$ we have 
\[
\rho_k = -\operatorname{Re}(\log( s_{k} ))>0
\]

Now we consider all $z = e^{2 \pi i t}$ on the unit circle for $t\in [\frac{-1}{2},\frac{1}{2}]$,
then for any $k \in {\cal I}$
\[
 \begin{array}{ll}
- (s_{k} z^{-\mbox{sign}(k)})^{\tau_{k}} 
&= - |s_{k}|^{\tau_{k}} \exp \big(i [\varphi_k-  2 \pi t \mbox{sign}(k)]\cdot \tau_{k} \big)\\
&=   \exp \Big( \tau_{k} \rho_k+ i \cdot \big( \tau_{k} (\varphi_k- 2 \pi t \mbox{sign}(k) ) + \pi  \big) \Big) 
\end{array}
  \]
 Now we simplify the notation by defining 
 \[
 \upsilon_k(t) :=\tau_{k} (\varphi_k- 2 \pi t \mbox{sign}(k) ) + \pi
 \]
then
 \[
 \begin{array}{ll}
\Phi(z)
&=  \prod_{j=1}^{N_{-}} (1- (s_{-j} z)^{\tau_{-j}}) \prod_{i=1}^{N_{+}}(1- (s_{i} z^{-1})^{\tau_{i}}) -1\\
&=  \prod_{k \in {\cal I}} \Big(1- |s_{k}|^{\tau_{k}} \exp \big(i [\varphi_k-  2 \pi t \mbox{sign}(k)]\cdot \tau_{k} \big)\Big)  -1\\
&=  \prod_{k \in {\cal I}} \bigg( 1+ \exp \Big( \tau_{k} \rho_k+ i \cdot \big( \tau_{k} (\varphi_k- 2 \pi t \mbox{sign}(k) ) + \pi  \big) \Big) \bigg)     -1\\
&=  \prod_{k \in {\cal I}} \bigg( 1+ \exp \Big( \tau_{k} \rho_k+ i \cdot \upsilon_k(t) \Big) \bigg)     -1
\end{array}
  \]
  
\begin{lemma}
 For any $z = \exp{(2 \pi i t )}$ on the unit circle,
 \[
 \begin{array}{ll}
\Phi\big(\exp{(2 \pi i t )}\big)
&=  \prod_{k \in {\cal I}} \bigg( 1+ \exp \Big( \tau_{k} \rho_k+ i \cdot \upsilon_k(t) \Big) \bigg)     -1\\
&= \sum_{n=1}^{|{\cal I}|}\sum_{ k_1, \ldots, k_n  \in {\cal I} } 
\exp \Big( \big(\tau_{k_1} \rho_{k_1}+ \ldots+  \tau_{k_n} \rho_{k_n} \big)
+ i \cdot \big( \upsilon_{k_1}(t) + \ldots+  \upsilon_{k_n}(t)  \big) \Big)  
\end{array}
  \]
 \end{lemma}
 
 \begin{lemma}
Additionally, for any $t \in [\frac{-1}{2},\frac{1}{2}]$,
\[
 \begin{array}{ll}
 |\Phi\big(\exp{(2 \pi i t )}\big)|^2  
= 
\sum_{n=1}^{|{\cal I}|}\sum_{ k_1, \ldots, k_n  \in {\cal I} } 
\exp \Big( 2 \big(\tau_{k_1} \rho_{k_1}+ \ldots+  \tau_{k_n} \rho_{k_n} \big) \Big)   
\end{array}
  \]
is independent of $t$, since the oscillation integral over the imaginary part is 
  \[
 \begin{array}{ll}
\int_{\frac{-1}{2}}^\frac{1}{2}
\exp  \bigg( i \cdot \Big(  \big(\upsilon_{k_1}(t) - \upsilon_{k_1}(t) \big) + \ldots+  \big(\upsilon_{k_n}(t) -\upsilon_{k_n}(-t)\big)  \Big) \bigg)  dt
= 1
\end{array}
  \]
Therefore, using Fourier isometry,
 \[
 \begin{array}{ll}
 \|\phi\|_2^2 &= \int_{\frac{-1}{2}}^\frac{1}{2} |\Phi\big(\exp{(2 \pi i t )}\big)|^2 dt \\
&= \sum_{n=1}^{|{\cal I}|}\sum_{ k_1, \ldots, k_n  \in {\cal I} } 
\exp \Big( 2 \big(\tau_{k_1} \rho_{k_1}+ \ldots+  \tau_{k_n} \rho_{k_n} \big) \Big)   
\end{array}
  \]
  
 \end{lemma}

\begin{theorem}
\label{thm: Root_2_norm}
Since $|s_{k}|<1$, i.e. $\rho_{k}<0$ for all $k\in {\cal I}$.
 \[
 \begin{array}{ll}
 \|\phi\|_2^2 
&= \sum_{n=1}^{|{\cal I}|}\sum_{ k_1, \ldots, k_n  \in {\cal I} } 
\exp \Big( 2 \big(\tau_{k_1} \rho_{k_1}+ \ldots+  \tau_{k_n} \rho_{k_n} \big) \Big) 
\end{array}
  \]
  Let $\gamma^m := \min_{i}\big(\tau_{i} |\rho_{i}| \big) =  \min\{\tau_{-N_-}|\rho_{-N_-}|, \ldots, \tau_{-1}|\rho_{-1}|, \tau_{1}|\rho_{1}|, \ldots, \tau_{N_+}|\rho_{N_+}|\}.$
When $\gamma^m \rightarrow \infty$, we have a finite sum of exponential rate  converging terms: the convergence rate is determined by the exponential term as a function of $\gamma^m$.
   \[
 \begin{array}{ll}
 \|\phi\|_2^2 
=    O(\exp \Big( -2 \min_{i}\big(\tau_{i} |\rho_{i}| \big) \Big))
= O(\exp \Big( -2 \gamma^m  \big) \Big))
\rightarrow 0
\end{array}
  \]

 \end{theorem}

 






