\section{Introduction}
\label{sec:introduction}
Despite the remarkable performance \citep{Krizhevsky2012} of Deep Neural
Networks (NNs), they are found to be rather fragile and easily fooled by
adversarial examples \citep{Szegedy2013}.  More intriguingly, these adversarial
examples are generated by adding imperceptible noise to normal examples, and
thus are indistinguishable for humans.  NNs that are more robust to adversarial
examples tend to have lower standard accuracy \citep{Su}, i.e., the accuracy
measured on natural examples.  The trade-off between robustness and accuracy
has been empirically observed in many works
\citep{DBLP:journals/ml/FawziFF18,Kurakin2017,Madry2017,Turner2018}, and has been theoretically analyzed under the context of simple models,
e.g., linear models \cite{Turner2018}, quadratic models
\cite{DBLP:journals/ml/FawziFF18}, but it is not clear whether the analysis
generalizes to NNs. For example, \citet{Turner2018} show that for linear
models, if examples are close to decision boundaries, robustness provably
conflicts with accuracy, though the proof seems unlikely to generalize to NNs.
Arguably, the most widely used remedy is developed to require NNs to be {\it
$\epsilon$-adversarially robust} (AR), e.g., via Adversarial Training
\citep{Madry2017}, Lipschitz-Margin Training \citep{Tsuzuku}; that is, they
require the model not to change predicted labels when any given input examples
are perturbed within a certain range. In practice, such AR methods are found to
lead to worse performance measured in standard classification accuracy.
Alternatives to build AR into NNs are also being developed. For instance,
\citet{Zhang2019} show that a gap exists between surrogate risk gap and 0-1
risk gap if many examples are close to decision boundaries, and better
robustness can be achieved by pushing examples away from decision boundaries.
But pushing examples away again degrades NN performance in their
experiments. But they are yet to be widely adopted by the community.


We investigate how adversarial robustness built into NNs by the arguably most
established method, i.e., Adversarial Training \citep{Madry2017}, influences
the behaviors of NNs to make them more robust but have lower performance
through the lens of regularization. In an earlier time \citep{Szegedy2013},
adversarial training has been suggested as a form of regularization: it
augments the training of NNs with adversarial examples, and thus might improve
the generalization of the end models.  Note that such a {\it hard requirement}
that the adversarial examples need to be classified correctly is different from
the methods that increase adversarial robustness by adding a soft penalty term to
the risk function employed by \citet{Lyu2016} and \citet{Miyato2018a}, or a
penalty term through curvature reduction \cite{Moosavi-Dezfooli2019}, or local
linearization \cite{Qin2019} (more discussion in
\cref{sec:furth-relat-works}). In these works, regularization is explicitly
enforced by a penalty term, while in adversarial training, it is not clear that
how training with augmented adversarial examples regularizes NNs. For example,
if adversarial training does work as a regularizer, how does a possible
improvement in generalization by using more data end up degrading performance?
Even such a basic problem does not have a clear answer.  To understand the
regularization effects of AR on NNs, we go beyond simple linear or quadratic
models and undertake a comprehensive generalization analysis of AR by
establishing a rigorous generalization bound on NNs, and carrying out a series
of empirical studies theoretically guided by the bound.


Technically, improved generalization implies the reduction in gap between
training errors and test errors.  Regularization achieves the gap reduction by
reducing the size of the hypothesis space, which reduces the variance, but
meanwhile increases the bias of prediction made — a constant classifier can
have zero generalization errors, but also have low test performance.  Thus,
when a hypothesis space is improperly reduced, another possible outcome is
biased poorly performing models with reduced generalization gaps.

\begin{figure*}[t]
    \centering
    \begin{subfigure}[b]{0.41\textwidth}
      \includegraphics[width=\columnwidth]{./Fig/Singular_value_var_on_CIFAR10.png}
      \caption{STD of Singular Values}
      \label{fig:singular_value_CIFAR10_pre}
    \end{subfigure}
    \begin{subfigure}[b]{0.41\textwidth}
      \includegraphics[width=\columnwidth]{./Fig/Probability_Distribution_of_Classification_Margin_on_CIFAR10_Test_Set_vanillaP_test.png}
      \caption{Margin Distribution}
      \label{fig:cifar10_maring_test_pre}
    \end{subfigure}
    \begin{subfigure}[b]{0.16\textwidth}
      \includegraphics[width=\columnwidth]{./Fig/Acc.png}
     
      \caption{Accuracy}
      \label{fig:accuracy_pre}
    \end{subfigure}
    \caption[Adversarial Robustness Regularization Effect Demo]{
      Experiment results on ResNet56 \citep{He2016} trained on the CIFAR10 dataset.
      For the details of the experiments, refer to \cref{sec:exper-valid}.
      {\bf (a)} The standard deviation of singular values of each layer of NNs with adversarial robustness (AR) strength 4, 16 (AR strength 8 is dropped for clarity of the plot). 
      To emphasize, the $x$-axis is the layer index --- overall 56 layers are involved. 
      {\bf (b)} The probability distribution of margins of NNs with AR strength 4, 8, 16. {\bf (c)} The standard and adversarial accuracy of NNs with AR 4, 8, 16.
    }
    \label{fig:diffidence}
\end{figure*}

{\bf Key results.} 
Through a series of theoretically motivated experiments, we find that AR is achieved by regularizing/biasing NNs towards less confident solutions by making the changes in the feature space of most layers (which are induced by changes in the instance space) smoother uniformly in all directions; so to a certain extent, it prevents sudden change in prediction w.r.t. perturbations. 
However, the end result of such smoothing concentrates examples around decision boundaries and leads to worse standard performance. 
We elaborate the above statement in details shortly in
\cref{sec:advers-robusnt-leads}.

{\bf Implications.} We conjecture that the improper reduction comes from the
indistinguishability of the change induced in the intermediate layers of NNs by
adversarial noise and that by inter-class difference.  To guarantee AR, NNs are
asked to smoothe out difference uniformly in all directions in a high
dimensional space, and thus are biased towards less confident solutions that make
similar/concentrated predictions.  We leave the investigation of the conjecture
as future works.

\subsection{AR leads to less confident NNs with more indecisive misclassifications}
\label{sec:advers-robusnt-leads}

This section elaborates the {\it key results} we briefly present previously.

{\it AR reduces the perturbations in the activation/outputs --- the perturbations
that are induced by perturbations in the inputs fed into the layer --- of most layers.}
Through a series of theoretically motivated experiments, the results prompt us
to look at the singular value distributions of the weight matrix of each layer
of the NNs. Shown in \cref{fig:singular_value_CIFAR10_pre}, we find that
overall the standard deviation (STD) of singular values associated with a layer of the NN trained with lower AR strength 4 is larger than that of the NN with higher AR strength 16
\footnote{The AR strength is characterized by the maximally allowed $l_{\infty}$ norm of adversarial examples that are used to train the NNs --- we use adversarial training \citep{Madry2017} to build adversarial robustness into NNs. 
Details can be found in \cref{sec:techn-build-advers}} 
--- the green dots are mostly below the red dots.
Note that given a matrix $\bv{W}$ and an example $\bv{x}$, singular values of $\bv{W}$ determine how the norm $||\bv{Wx}||$ is changed comparing with $||\bv{x}||$. 
More specifically, let $\sigma_{\min}, \sigma_{\max}$ be the minimal and maximal singular values. 
If $\bv{x}$ is not in the null space of $\bv{W}$, then we have $||\bv{Wx}|| \in [\sigma_{\min}||\bv{x}||, \sigma_{\max}||\bv{x}||]$, where $||\cdot||$ denotes $2$-norm. This applies to norm $||\delta\bv{x} ||$ of a perturbation as well; that is, given possible changes $\delta{\bv{x}}$ of $\bv{x}$ of the same norm $||\delta\bv{x}|| = c$, where $c$ is a constant, the variance of $\sigma(\bv{W})$ roughly determines the variance of $||\bv{W}\delta\bv{x}||$, where $\sigma(\bv{W})$ denotes all singular values $\{\sigma_i\}$ of $\bv{W}$. 
In more details, note that by SVD decomposition, $\bv{W}\delta\bv{x} = \sum_{i}\sigma_{i}\bv{u}_i\bv{v}^T_i\delta\bv{x}$, thus $\sigma_i$ determines how the component $\bv{v}^{T}_i\delta\bv{x}$ in the direction of $\bv{v}_i$ is amplified. 
To see an example, suppose that $\sigma_{\min} = \sigma_{\max} = \sigma_0$, then the variance of $\sigma(\bv{W})$ is zero, and $||\bv{W}\delta\bv{x}|| = \sigma_0||\delta\bv{x}||$. 
In this case, the variance of $||\bv{W}\delta\bv{x}||$ (given an ensemble of perturbations $\delta\bv{x}$ of the same norm $c$) is zero as well. 
The conclusion holds as well for $\text{ReLU}(\bv{W}\delta{x})$, where $\bv{W}$ here
is a weight matrix of a layer of a NN, and $\text{ReLU}$ denotes Rectifier
Linear Unit activation function (proved by applying Cauchy interlacing law by
row deletion \citep{Meek} in \cref{lm:1}).  Consequently, by reducing the
variance of singular values of weight matrix of a layer of the NN, AR reduces
the variance of the norms of layer activations, or informally, perturbations in the
activations, induced by input perturbations.

{\it The perturbation reduction in activations concentrates examples, and it
empirically concentrates them around decision boundaries; that is, predictions
are less confident.}
The reduced variance implies that the outputs of each layer of the NN are more concentrated, but it does not tell where they are concentrated. 
Note that in the previous paragraph, the variance relationship discussed between $||\bv{W}\delta\bv{x}||$ and $||\delta\bv{x}||$ equally applies to $||\bv{W}\bv{x}||$ and $||\bv{x}||$, where $\bv{x}$ is an actual example instead of perturbations. 
Thus, to find out the concentration of perturbations, we can look at the
concentration of samples. 
Technically, we look at {\it margins} of examples. In a multi-class setting, suppose a NN computes a score function $f : \bb{R}^{d} \rightarrow \bb{R}^L$, where $L$ is the number of classes; a way to convert this to a classifier is to select the output coordinate with the largest magnitude, meaning $x \mapsto \argmax_{i}f_{i}(x)$. 
The {\it confidence} of such a classifier could be quantified by margins. It measures the gap between the output for the correct label and other labels, meaning $f_y(x) - \max_{i\not=y} f_i(x)$. 
Margin piece-wise linearly depends on the scores, thus the variance of margins is also in a piece-wise linear relationship with the variance of the scores, which are computed linearly from the activation of a NN layer. 
Thus, the consequence of concentration of activation discussed in the previous paragraph can be observed in the distribution of margins. 
More details of the connection between singular values and margins are discussed in \cref{sec:advers-robustn-makes}, after we present \cref{lm:1}.  
A zero margin implies that a classifier has equal propensity to classify an example to two classes, and the example is on the decision boundary.  
We plot the margin distribution of the test set of CIFAR10 in \cref{fig:cifar10_maring_test_pre}, and find that margins are increasingly concentrated around zero --- that is, the decision boundaries --- as AR strength grows.

{\it The sample concentration around decision boundaries smoothes sudden
changes induced perturbations, but also increases indecisive misclassifications.} 
The concentration of test set margins implies that the induced change in margins by the perturbation in the instance space is reduced by AR.
Given two examples $\bv{x}, \bv{x}'$ from the test set, $\delta\bv{x} = \bv{x} - \bv{x}'$ can be taken as a significant perturbation that changes the example $\bv{x}$ to $\bv{x}'$. 
The concentration of overall margins implies the change induced by $\delta\bv{x}$ is smaller statistically in NNs with higher AR strength. 
Thus, for an adversarial perturbation applied on $\bv{x}$, statistically the change of margins is smaller as well --- experimentally it is reflected in the increased adversarial robustness of the network, as shown in the increasing curve in \cref{fig:accuracy_pre}. 
That is, the sudden changes of margins originally induced by adversarial perturbations are {\it smoothed} (to change slowly). 
However, the {\it cost} of such smoothness is lower confidence in prediction, and more test examples are slightly/indecisively moved to the wrong sides of the decision boundaries --- incurring lower accuracy, as shown in the decreasing curve in \cref{fig:accuracy_pre}.
    
Lastly, we note that experiments in this section are used to illustrate our main arguments in this section.
Further consistent quality results are reported in \cref{sec:exper-valid} by
conducting experiments on CIFAR10/100 and Tiny-ImageNet with networks of varied
capacity. And more corroborative experiment results are presented in the
appendices, and outlined in \cref{sec:outl-contr}.

\subsection{Outline and contributions}
\label{sec:outl-contr}

This work carries out generalization analysis on NNs with AR.  
The quantities in the previous section are identified by the generalization errors (GE) upper bound we establish at \cref{thm:main}, which characterizes the regularization of AR on NNs. 
The key result is obtained at the {\it end} of a series of analysis, thus we present the outline of the analysis here.

{\bf Outline.} 
After presenting some preliminaries in \cref{sec:preliminaries}, we proceed to
analyze the regularization of AR on NNs, and establish a GE upper bound in
\cref{sec:gener-bound-guar}.  The bound prompts us to look at the GE gaps in
experiments.  In \cref{sec:regul-effects-nns}, we find that for NNs trained
with higher AR strength, the surrogate risk gaps (GE gaps) decrease for a range
of datasets, i.e., CIFAR10/100 and Tiny-ImageNet.  It implies AR effectively
regularizes NNs.  We then study the finer behavior change of NNs that might
lead to such a gap reduction.  Again, we follow the guidance of
\cref{thm:main}.  We look at the margins in \cref{sec:cms-that-concentrate},
then at the singular value distribution in \cref{sec:advers-robustn-makes}, and
discover the main results described in \cref{sec:advers-robusnt-leads}.  More
corroborative experiments are present in \cref{sec:further-evidence} and
\cref{sec:furth-empir-study} to show that such phenomenon exists in a broad
range of NNs with varied capacity and adversarial training techniques.  More
complementary results are present in \cref{sec:discr-trend-surr} to explain
some seemingly abnormal observations, and in \cref{sec:furth-evid-smooth} to
quantitatively demonstrate the smoothing effects of AR discussed in
\cref{sec:advers-robusnt-leads}.  Related works are present in
\cref{sec:related-works}.


{\bf Contributions}. 
Overall, the core contribution in this work is to show that adversarial robustness (AR) regularizes NNs in a way that hurts its capacity to learn to perform in test. 
More specifically:
\begin{itemize}
\item We establish a generalization error (GE) bound that characterizes the
regularization of AR on NNs. 
The bound connects \emph{margin} with adversarial robustness radius $\epsilon$ via \emph{singular values of weight matrices} of NNs, thus suggesting the two quantities that guide us to investigate the regularization effects of AR empirically.
\item Our empirical analysis tells that AR \emph{effectively} regularizes NNs to reduce the GE gaps. To understand how reduced GE gaps turns out to degrade test performance, we study \emph{variance of singular values} of layer-wise weight matrices of NNs and \emph{distributions of margins} of samples, when different strength of AR are applied on NNs.
\item  The study shows that AR is achieved by regularizing/biasing NNs
towards less confident solutions by making the changes in the feature space of
most layers (which are induced by changes in the instance space) smoother uniformly in all directions; so to a certain extent, it prevents sudden change in
prediction w.r.t. perturbations. 
However, the end result of such smoothing concentrates samples around decision boundaries and leads to worse standard performance.
\end{itemize}

\section{Preliminaries}
\label{sec:preliminaries}

Assume an instance space $\mathcal{Z} = \mathcal{X} \times \mathcal{Y}$, where $\mathcal{X}$ is the space of input data, and $\mathcal{Y}$ is the label space.
$Z := (X, Y)$ are the random variables with an unknown distribution $\mu$, from which we draw samples. 
We use $S_m = \{ z_i = (\bvec{x}_i, y_i)\}_{i=1}^{m}$ to denote the training set of size $m$ whose examples are drawn independently and identically distributed (i.i.d.) by sampling $Z$. 
Given a loss function $l$, the goal of learning is to identify a function $T: \mathcal{X} \mapsto \mathcal{Y}$ in a hypothesis space (a class $\mathcal{T}$ of functions) that minimizes the expected risk
\[
  R(l \circ T) = \mathbb{E}_{Z \sim \mu}\left[l\left(T(X), Y\right)\right] ,
\]
Since $\mu$ is unknown, the observable quantity serving as the proxy to the expected risk $R$ is the empirical risk
\begin{displaymath}
  R_m(l \circ T) = \frac{1}{m}\sum\limits_{i=1}^{m}l\left(T(\bvec{x}_i), y_i\right).
\end{displaymath}
Our goal is to study the discrepancy between $R$ and $R_m$, which is termed as {\it generalization error} --- it is also sometimes termed as generalization gap in the literature
\begin{equation}
  \label{eq:ge}
  \text{GE}(l \circ T) = |R(l \circ T) - R_m(l \circ T)|.
\end{equation}

A NN is a map that takes an input $x$ from the space
$\mathcal{X}$, and builds its output by recursively applying a linear map
$W_{i}$ followed by a pointwise non-linearity $g$:
\[
  x_i = g (\bv{W}_{i} \bv{x}_{i-1}) ,
\]
where $i$ indexes the times of recursion, which is denoted as a layer in the
community, $i = 1, \ldots, L$, $x_0 = x$, and $g$ denotes the activation function.
which is restricted to Rectifier Linear Unit (ReLU) \citep{Glorot2011} or
max pooling operator \citep{Becigneul2017} in this paper. To compactly summarize the operation of $T$, we denote
\begin{equation}
  \label{eq:nn}
  Tx = g(\bv{W}_Lg(\bv{W}_{L-1}\ldots g(\bv{W}_1\bv{x}))) .
\end{equation}


\begin{definition}[Covering number]
    Given a metric space $(\mathcal{S}, \rho)$, and a subset $\tilde{\mathcal{S}} \subset \mathcal{S}$, we say that a subset $\hat{\mathcal{S}}$ of $\tilde{\mathcal{S}}$ is a $\epsilon$-cover of $\tilde{\mathcal{S}}$, if $\forall \tilde{s} \in \tilde{\mathcal{S}}$, $\exists \hat{s} \in \hat{\mathcal{S}}$ such that $\rho(\tilde{s}, \hat{s}) \leq \epsilon$. The $\epsilon$-covering number of $\tilde{\mathcal{S}}$ is
    \begin{displaymath}
        \mathcal{N}_{\epsilon}(\tilde{\mathcal{S}}, \rho) = \min\{ |\hat{\mathcal{S}}|:
        \hat{\mathcal{S}} \text{ is an } \epsilon\text{-covering of } \tilde{\mathcal{S}} \} .
    \end{displaymath}
\end{definition}

Various notions of adversarial robustness have been studied in existing works
\citep{Madry2017,Turner2018,Zhang2019}. 
They are conceptually similar; in this work, we formalize its definition to make clear the object for study.

\begin{definition}[$(\rho, \epsilon)$-adversarial robustness]
    \label{def:ar}
    Given a multi-class classifier $f: \ca{X} \rightarrow \bb{R}^{L}$, and a metric $\rho$ on $\ca{X}$, where $L$ is the number of classes, $f$ is said to be adversarially robust w.r.t. adversarial perturbation of strength $\epsilon$, if there exists an $\epsilon > 0$ such that $\forall z = (\bv{x}, y) \in \ca{Z}$ and $ \delta\bv{x} \in \{\rho(\delta \bv{x}) \leq \epsilon \}$, we have
    \begin{displaymath}
        f_{\hat{y}}(\bv{x} + \delta\bv{x}) - f_{i}(\bv{x} + \delta\bv{x}) \geq 0,
    \end{displaymath}
    where $\hat{y} = \argmax_{j}f_j(\bv{x})$ and $i \not= \hat{y} \in \ca{Y}$.
    $\epsilon$ is called \textit{\textbf{adversarial robustness radius}}.
    When the metric used is clear, we also refer $(\rho, \epsilon)$-adversarial robustness as $\epsilon$-adversarial robustness.
\end{definition}

Note that the definition is an {\it example-wise} one; that is, it requires each example to have a guarding area, in which all examples are of the same class. Also note that the robustness is w.r.t. the predicted class, since ground-truth label is unknown for a $\bv{x}$ in test.

We characterize the GE with ramp risk, which is a typical risk to undertake theoretical analysis \citep{Bartlett2017,Role2019}.

\begin{definition}[Margin Operator]
    \label{def:margin}
    A margin operator $\ca{M}: \bb{R}^{L} \times \{1, \ldots, L\} \rightarrow
    \bb{R}$ is defined as
    \begin{displaymath}
        \ca{M}(\bv{s}, y) := s_{y} - \max_{i\not= y}s_{i}
    \end{displaymath}
    \end{definition}
    \begin{definition}[Ramp Loss]
    \label{def:ramp_loss}
    The ramp loss $l_{\gamma}: \bb{R} \rightarrow \bb{R}^{+}$ is defined as
    \begin{displaymath}
        l_{\gamma} (r) :=
        \begin{cases}
        0            & r < -\gamma        \\
        1 + r/\gamma & r \in [-\gamma, 0] \\
        1            & r > 0
        \end{cases}
    \end{displaymath}
\end{definition}
\begin{definition}[Ramp Risk]
    \label{def:ramp_risk}
    Given a classifier $f$, ramp risk is the risk defined as
    \begin{displaymath}
        R_{\gamma}(f) := \bb{E}(l_{\gamma}(- \ca{M}(f(X), Y))),
    \end{displaymath}
    where $X, Y$ are random variables in the instance space $\ca{Z}$ previously.
\end{definition}

We will use a different notion of margin in \cref{thm:main}, and formalize its
definition as follows. We reserve the unqualified word ``margin'' specifically
for the margin discussed previously --- the output of margin operator for
classification. We call this margin to-be-introduced {\it instance-space margin
(IM)}.

\begin{definition}[Smallest Instance-space Margin]
    \label{def:im}
    Given an element $z = (\bv{x},
    y) \in \ca{Z}$, let $v(\bv{x})$ be the distance from $\bv{x}$ to its closest
    point on the decision boundary, i.e., the {\it instance-space
      margin} (IM) of example $\bv{x}$.
Given a covering set $\hat{S}$ of $\mathcal{Z}$, let
\begin{equation}
    \label{eq:smallest_margin}
  v_{\min} = \min_{\bv{x} \in \{\bv{x} \in \mathcal{X} |  \exists
    \bv{x}' \in \hat{S}_m, ||\bv{x} - \bv{x}'||_2 \leq \epsilon\}} v(\bv{x}),
\end{equation}
where $\hat{S}_m := \{\bv{x}' \in \hat{S} | \exists \bv{x}_i \in S_m, ||\bv{x}_i - \bv{x}'||_2 \leq \epsilon \}$.
    $v_{\min}$ is the {\it smallest} instance-space margin of elements in the
    covering balls that {\it contain training examples}.
\end{definition}




\section{Theoretical instruments for empirical studies on AR}
\label{sec:gener-bound-guar}

In this section, we rigorously establish the bound mentioned in the introduction. 
We study the map $T$ defined in \cref{sec:preliminaries} as a NN (though technically, $T$ now is a map from $\ca{X}$ to $\bb{R}^{L}$, instead of to $\ca{Y}$, such an abuse of notation should be clear in the context). 
To begin with, we introduce an assumption, before we state the generalization error bound guaranteed by adversarial robustness.

\begin{assumption}[Monotony]
    \label{asp:monotony}
    Given a point $\bv{x} \in \ca{X}$, let $\bv{x}'$ be the point on the decision boundary of a NN $T$ that is closest to $\bv{x}$. 
    Then, for all $\bv{x}''$ on the line segment $\bv{x} + t(\bv{x'} - \bv{x}), t \in [0, 1]$, the margin $\ca{M}(T\bv{x}'', y)$ decreases monotonously.
\end{assumption}

The assumption is a regularity condition on the classifier that rules out undesired oscillation between $\bv{x}$ and $\bv{x}'$. 
To see how, notice that the margin defined in \cref{def:margin} reflects how confident the decision is made. 
Since $\bv{x}'$ is on the decision boundary, it means the classifier is unsure how it should be classified. 
Thus, when the difference $\bv{x}' - \bv{x}$ is gradually added to $\bv{x}$, ideally we want the confidence that we have on classifying $\bv{x}$ to decrease in a consistent way to reflect the uncertainty.

\begin{figure*}[t]
    \centering
    \begin{subfigure}[b]{0.48\textwidth}
      \def\columnwidth{\columnwidth}
      \input{ar_demo.pdf_tex}
      \caption{}
      \label{fig:demo}
    \end{subfigure}
    \begin{subfigure}[b]{0.44\textwidth}
      \def\columnwidth{\columnwidth}
      \input{piecewise_linear_demo.pdf_tex}
      \caption{}
      \label{fig:piecewise}
    \end{subfigure}
    \caption[
        Adversarial Robustness Regularization Effect Demo]{{\bf (a)} Illustration of the regularization effect of adversarial robustness. 
        If a NN $T$ is $\epsilon$-adversarially robust, for a given example $\bv{x}$ (drawn as filled squares or circles) and points $\bv{x}'$ in the yellow ball $\{\bv{x}' \,|\, \rho(\bv{x}, \bv{x}') \leq \epsilon\}$ around $\bv{x}$, the predicted labels of $\bv{x}, \bv{x}'$ should be the same, and the loss variation is potentially bigger as $\bv{x}'$ moves from the center to the edge, as shown as intenser yellow color at the edge of a ball. 
        Collectively, the adversarial robustness of each example requires an {\it instance-space margin} (IM) to exist for the decision boundary, shown as the shaded cyan margin. 
        As normally known, margin is related to generalization ability that shrinks the hypothesis space. 
        In this case, the IM required by adversarial robustness would weed out hypotheses that do not have an adequate IM, such as the red dashed line shown in the illustration. 
        {\bf (b)} Illustration of \cref{lm:1}. 
        Given a NN with ReLU activation function, the feature map $\bv{I}_l$ at layer $l$ is divided into regions where $\bv{I}_l(\bv{x})$ is piecewise linear w.r.t. $\bv{x}$. 
        The induced linear map $\bv{W}^q_1$ is given by $\text{diag}(\tau_1(q))\bv{W}_1$, where $\text{diag}(\tau_l(q))$ is a diagonal matrix whose diagonal entries are given by a vector $\tau_1(q)$ that has 0-1 values. 
        For example, in region $p$, $\bv{I}_1 = \bv{W}^{p}_1\bv{x}$ and distance between instances $\bv{x}$ are vertical elongated, while in region $q$, $\bv{I}_1 = \bv{W}^{q}_1\bv{x}$ and distance are horizontally elongated. 
        Thus given $\bv{x}, \bv{x}'$, the difference $\left|\left|  \bv{I}_{l}(\bv{x}) - \bv{I}_{l}(\bv{x}') \right|\right|$ between $\bv{I}_l(\bv{x})$ and $\bv{I}_l(\bv{x}')$ is the length of the transformed line segment $\bv{x} - \bv{x}'$ drawn, of which each segment is linearly transformed in a different way.
    }
\end{figure*}


\begin{theorem}
    \label{thm:main}
    Let $T$ denote a NN with ReLU and MaxPooling nonlinear activation functions (the definition is put at \cref{eq:nn} for readers' convenience), $l_{\gamma}$ the ramp loss defined at \cref{def:ramp_loss}, and $\ca{Z}$ the instance space assumed in \cref{sec:gener-bound-guar}. 
    Assume that $\ca{Z}$ is a $k$-dimensional regular manifold that accepts an $\epsilon$-covering with covering number $(\frac{C_{\ca{X}}}{\epsilon})^{k}$, and assumption \cref{asp:monotony} holds. 
    If $T$ is $\epsilon_0$-adversarially robust (defined at \cref{def:ar}), $\epsilon \leq \epsilon_0$, and denote $v_{\min}$ the smallest IM margin in the covering balls that contain training examples (defined at \cref{def:im}), $\sigma_{\min}^{i}$ the smallest singular values of weight matrices $\bv{W}_{i}, i=1,\ldots,L-1$ of a NN, $\{\bv{w}_{i}\}_{i=1,\ldots, |\ca{Y}|}$ the set of vectors made up with $i$th rows of $\bv{W}_{L}$ (the last layer's weight matrix), then given an i.i.d. training sample $S_m = \{ z_i = (\bvec{x}_i, y_i) \}_{i=1}^{m}$ drawn from $\ca{Z}$, its generalization error $\text{GE}(l \circ T)$ (defined at \cref{eq:ge}) satisfies that, for any $\eta > 0$, with probability at least $1 - \eta$
    \begin{align}
        \text{GE}(l_{\gamma} \circ T) \leq&  \max\{0, 1 - \frac{u_{\min}}{\gamma}\}\nonumber\\
                        &+
                        \sqrt{\frac{2\log(2)C_{\mathcal{X}}^{k}}{\epsilon^{k}m}
        + \frac{2 \log(1/\eta)}{m}}\label{eq:bound_1}
    \end{align}
    where
    \begin{equation}
        u_{\min} = \min_{y, \hat{y} \in \ca{Y}, y\not=\hat{y}}||\bv{w}_{y} - \bv{w}_{\hat{y}}||_2\prod_{i=1}^{L-1}\sigma_{\min}^{i}v_{\min}\label{eq:bound_2}
    \end{equation}
    is a lower bound of margins of examples in covering balls that
    contain training samples.
\end{theorem}
  
The proof of \cref{thm:main} is in \cref{sec:proof-}. {\it The bound identifies quantities that would be studied experimentally in \cref{sec:exper-valid} to understand the regularization of AR on NNs.}
The first term in \cref{eq:bound_1} in \cref{thm:main} suggests that quantities related to the lower bound of {\it margin} $u_{\min}$ might be useful to study how $\epsilon$-adversarial robustness ($\epsilon$-AR) regularizes NNs.
However, $\epsilon$-AR is guaranteed in the instance space that determines the smallest {\it instance-space margin} $v_{\min}$. 
To relate GE bound with $\epsilon$-AR, we characterize in \cref{eq:bound_2} the relationship between margin with IM, via smallest {\it singular values of NNs' weight matrices}, suggesting that quantities related to singular values of NNs' weight matrices might be useful to study how AR regularizes NNs as well.  
An illustration on how AR could influence generalization of NNs through IM is also given in \cref{fig:demo}.
The rightmost term in \cref{eq:bound_1} is a standard term in robust framework \citep{Xu2012a} in learning theory, and is not very relevant to the discussion.
{\it The remaining of this paper are empirical studies that are based on the quantities}, e.g., margin distributions and singular values of NNs' weight matrices, that are related to the identified quantities, i.e., $u_{\min}, \sigma_{\min}^{i}$. 
These studies aim to illuminate with empirical evidence on the phenomena that AR regularizes NNs, reduces GE gaps, but degrades test performance.
\footnote{ 
    Note that in the previous paragraph, though we identifies quantities $u_{\min}$ and $\sigma_{\min}^{i}$ related to the upper bound of GE, the quantities we actually would study empirically are {\it margin distribution} and all {\it singular values} that characterize the GE of all samples, not just the extreme case (upper bound). 
    The analytic characterization of the GE of all samples is not possible since we do not have enough information (we do not know the true distribution of samples). 
    That's why to arrive at close-form analytic characterization of GE, we resort to the extreme non-asymptotic large-sample behaviors. 
    {\it The analytic form is a neat way to present how relevant quantities influence GE.} 
    In the rest of the paper, we would carry on empirical study on the distributions of margins and singular values to investigate AR's influence on GE of all samples.
}

\begin{figure*}[t]
    \centering
    \begin{subfigure}[b]{0.49\textwidth}
      \includegraphics[width=\linewidth]{./Fig/loss_gap.png}
      \caption{}
      \label{fig:loss_gap_full}
    \end{subfigure}
    \begin{subfigure}[b]{0.49\textwidth}
      \includegraphics[width=\linewidth]{./Fig/ErrorPlot/3_Loss_Error_Different_Dataset.png}
      \caption{}
      \label{fig:performance}
    \end{subfigure}
    \caption{
        Experiment results on CIFAR10/100, and Tiny-ImageNet.  The unit of
x-axis is the adversarial robustness (AR) strength of NNs, c.f. the beginning
of \cref{sec:exper-valid}.  {\bf (a)} Plots of loss gap (and error rate gap)
between training and test datasets v.s. AR strength. {\bf (b)} Plots of losses
(and error rates) on training and test datasets v.s. AR strength.
        }
    \label{fig:datasets}
\end{figure*}

Before turning into empirical study, we further present a lemma to illustrate the relation characterized in \cref{eq:bound_2} without the need to jump into proof of \cref{thm:main}. 
It would motivate our experiments later in \cref{sec:advers-robustn-makes}. 
We state the following lemma that relates distances between elements in the instance space with those in the feature space of any intermediate network layers.
\begin{lemma}
\label{lm:1}
    Given two instances $\bv{x}, \bv{x}' \in \ca{X}$, let $\bv{I}_{l}(\bv{x})$ be the activation $g(\bv{W}_lg(\bv{W}_{l-1}\ldots g(\bv{W}_1\bv{x})))$ at layer $l$ of $\bv{x}$, then there exist $n \in \bb{N}$ sets of matrices $\{\bv{W}_i^{q_{j}}\}_{i=1\ldots l}, j=1\ldots n$, that each of the matrix $\bv{W}_{i}^{q_j}$ is obtained by setting some rows of $\bv{W}_{i}$ to zero, and $\{q_j\}_{j=1\ldots n}$ are
    arbitrary distinctive symbols indexed by $j$ that index $\bv{W}^{q_j}_i$,  such that
    \begin{displaymath}
        \left|\left|  \bv{I}_{l}(\bv{x}) - \bv{I}_{l}(\bv{x}') \right|\right| =
        \sum_{j=1}^{n}\int_{s_j}^{e_j}\left|\left|\prod_{i=1}^{l}\bv{W}_{i}^{q_j}dt(\bv{x} - \bv{x}')\right|\right|
    \end{displaymath}
    where $s_{1}=0, s_{j+1}=e_{j}, e_{n}=1, s_j,e_j \in [0, 1]$ --- each $[s_j, e_j]$ is a segment in the line segment parameterized by $t$ that connects $\bv{x}$ and $\bv{x}'$.
\end{lemma}
Its proof is in \cref{sec:proof-}, and an illustration is given in \cref{fig:piecewise}. 
Essentially, it states that difference in the feature space of a NN, induced by the difference between elements in the instance space, is a summation of the norms of the linear transformation ($\prod_{i=1}^{l}\bv{W}^{q_j}_i$) applied on segments of the line segment that connects $\bv{x}, \bv{x}'$ in the instance space. 
Since $\bv{W}_{i}^{q_j}$ is obtained by setting rows of $\bv{W}_i$ to zero, the singular values of these induced matrices are intimately related to weight matrices $\bv{W}_i$ of NN by Cauchy interlacing law by row deletion \citep{Meek}. 
Since the margin of an example $\bv{x}$ is a linear transform of the difference between $I_{L-1}(\bv{x})$ and the $I_{L-1}(\bv{x}')$ of an element $\bv{x}'$ on the decision boundary, singular values of $\{W_{i}\}_{i=1\ldots L-1}$ determine the amplification/shrinkage of the IM $\bv{x} - \bv{x}'$.

\section{Empirical studies on regularization of adversarial robustness}
\label{sec:exper-valid}

In this section, guided by \cref{thm:main}, we undertake empirical studies to explore AR's regularization effects on NNs. We first investigate the behaviors of off-the-shelf architectures of fixed capacity on various datasets in \cref{sec:regul-effects-nns} and \cref{sec:refin-anals-regul}. 
More corroborative controlled studies that explore the regularization effects of AR on NNs with varied capacity are present in \cref{sec:regul-effects-nns-1}.

\subsection{Adversarial robustness effectively regularizes NNs on various datasets}
\label{sec:regul-effects-nns}


This section aims to explore whether AR can effectively reduce generalization
errors --- more specifically, the surrogate risk gaps. 
We use adversarial training \citep{Madry2017} to build adversarial robustness into NNs.  
The AR strength is characterized by the maximally allowed $l_{\infty}$ norm of adversarial examples that are used to train the NNs. 
Details on the technique to build adversarial robustness into NNs is given in \cref{sec:techn-build-advers}.

Our experiments are conducted on CIFAR10, CIFAR100, and Tiny-ImageNet \citep{tiny_imagenet} that represent learning tasks of increased difficulties.
We use ResNet-56 and ResNet-110 \citep{He2016} for CIFAR10/100, and Wide ResNet (WRN-50-2-bottleneck) \citep{Zagoruyko2016} for Tiny-ImageNet \citep{tiny_imagenet}. 
These networks are trained with increasing AR strength. 
Results are plotted in \cref{fig:datasets}.



\paragraph{Regularization of AR on NNs.}
We observe in \cref{fig:loss_gap_full} (shown as blue lines marked by circles) that GE gaps (the gaps between training and test losses) decrease as strength of AR increase; we also observe in \cref{fig:loss_gap_full} that training losses increase as AR strength increase; these results (and more results in subsequent \cref{fig:control}) imply that AR does regularize training of NNs by reducing their capacities to fit training samples. 
Interestingly, in the CIFAR10/100 results in \cref{fig:performance}, the test losses show a decreasing trend even when test error rates increase. 
It suggests that the network actually performs better measured in test loss as contrast to the performance measured in test error rates. 
This phenomenon results from that less confident wrong predictions are made by NNs thanks to adversarial training, which will be explained in details in \cref{sec:refin-anals-regul}, when we carry on finer analysis. 
We note that on Tiny-ImageNet, the test loss does not decrease as those on CIFAR10/100. 
It is likely because the task is considerably harder, and regularization hurts NNs even measured in test loss.

\paragraph{Trade-off between regularization of AR and test error rates.}
The error rate curves in \cref{fig:performance} also tell that the end result of AR regularization leads to biased-performing NNs that achieve degraded test performance. 
These results are consistent across datasets and networks.

\paragraph{Seemingly abnormal phenomenon.}
An seemingly abnormal phenomenon in CIFAR10 observed in \cref{fig:loss_gap_full} is that the error rate gap actually increases. 
It results from the same underlying behaviors of NNs, which we would introduce in \cref{sec:refin-anals-regul}, and an overfitting phenomenon that AR cannot control. Since it would be a digress to explain, it is put in \cref{sec:discr-trend-surr}.

We finally note that the adversarial robustness training reproduced is relevant, of which the defense effect is comparable with existing works.
One may refer to \cref{fig:defense} in \cref{sec:exp_1} for the details. 
We can see from it that similar adversarial robustness to \citet{Madry2017} and \citet{Li2018a} is achieved for CIFAR10/100, Tiny-ImageNet in the NNs we reproduce.


\subsection{Refined analysis through margins and singular values}
\label{sec:refin-anals-regul}

The experiments in the previous sections confirm that AR reduces GE, but decreases accuracy. 
We study the underlying behaviors of NNs to analyze what have led to it here. 
More specifically, we show that adversarial training implements $\epsilon$-adversarial robustness by making NNs biased towards less confident solutions; that is, the key finding we present in \cref{sec:advers-robusnt-leads} that explains both the prevented sudden change in prediction w.r.t. sample perturbation (i.e., the achieved AR), and the reduced test accuracy.


\subsubsection{Margins that concentrate more around zero lead to reduced GE gap}
\label{sec:cms-that-concentrate}

To study how GE gaps are reduced, \cref{thm:main} suggests we first look at the
margins of examples --- a lower bound of margins is $u_{\min}$ in
\cref{eq:bound_2}.  The analysis on margins has been a widely used tool in
learning theory \citep{Bartlett2017}. It reflects the confidence that a
classifier has on an example, which after being transformed by a loss function,
is the surrogate loss.  Thus, the loss difference between examples are
intuitively reflected in the difference in confidence characterized by margins.
To study how AR influences generalization of NNs, distributions of samples
which are obtained by training ResNet-56 on CIFAR10 and CIFAR100 with increased
AR strength (the same setting as for \cref{fig:datasets}).  Applying the same
network of ResNet-56 respectively on \emph{CIFAR-10 and CIFAR-100 of different
learning difficulties} creates learning settings of \emph{larger- and
smaller-capacity} NNs.

\begin{figure}[h]
    \centering
    \begin{subfigure}[b]{0.235\textwidth}
      \includegraphics[width=\columnwidth]{./Fig/Probability_Distribution_of_Classification_Margin_on_CIFAR10_Test_Set_vanillaP_test.png}
      \caption{CIFAR10 Test}
      \label{fig:cifar10_margin_test}
    \end{subfigure}
    \begin{subfigure}[b]{0.235\textwidth}
      \includegraphics[width=\columnwidth]{./Fig/Probability_Distribution_of_Classification_Margin_on_CIFAR100_Test_Set_vanillaP_test.png}
      \caption{CIFAR100 Test}
      \label{fig:cifar100_margin_test}
    \end{subfigure}
    \\
    \begin{subfigure}[b]{0.235\textwidth}
      \includegraphics[width=\columnwidth]{./Fig/Probability_Distribution_of_Classification_Margin_on_CIFAR10_Training_Set_vanillaP_trn.png}
      \caption{CIFAR10 Training}
      \label{fig:cifar10_margin_train}
    \end{subfigure}
    \begin{subfigure}[b]{0.235\textwidth}
      \includegraphics[width=\columnwidth]{./Fig/Probability_Distribution_of_Classification_Margin_on_CIFAR100_Training_Set_without_SP_vanillaP_trn.png}
      \caption{CIFAR100 Training}
      \label{fig:cifar100_margin_train}
    \end{subfigure}
    \caption{Margin distributions of NNs with AR strength 4, 8, 16 on Training and
      Test sets of CIFAR10/100.}
      \label{fig:margin_distribution}
\end{figure}

\paragraph{Concentration and reduced accuracy.}
In \cref{fig:margin_distribution}, we can see that in both CIFAR10/100, the
distributions of margins become more concentrated around zero as AR grows. The
concentration moves the mode of margin distribution towards zero and more
examples slightly across the decision boundaries, where the margins are zero,
which explains the reduced accuracy \footnote{We remark a possibly confusing
  phenomenon here about the margin. The bound \cref{eq:bound_1} might give the
  impression that a smaller margin might lead to a larger generalization error,
  while the empirical study instead shows that the NNs with a smaller margin have
  a smaller generalization error. The hypothesized confusion is a
  misunderstanding of the generalization bound analysis.
The upper bound is a worst case analysis of GE. However, in practice, the interesting object is the average gap between
the training losses and the test losses, i.e., the GE. Unfortunately, the
average gap cannot be analyzed analytically (cf. footnote 2). Thus, we, and also the statistical
learning community, resort to worst case analysis to find an upper bound on GE
to identify quantities that might influence GE. In this case, the phenomenon suggests that the bound might be loose, though this is a
problem that plagues the statistical learning community \cite{Nagarajan2019}. But our focus
in this work is not to derive tight bounds, or reach definite conclusions from
bounds alone, but to guide experiments with the bound.}.

\paragraph{Concentration and reduced loss/GE gap.}
The concentration has different consequences on training and test
losses. Before describing the consequences, to directly relate the concentration
to loss gap, we further introduce {\it estimated probabilities} of
examples. This is because though we use ramp loss in theoretical analysis, in
the experiments, we explore the behaviors of more practically used {\it cross
entropy loss}. The loss maps one-to-one to estimated probability, but not to
margin, though they both serve as a measure of confidence.  Suppose
$\bv{p}(\bv{x})$ is the output of the {\it softmax} function of dimension $L$
($L$ is the number of target classes), and $y$ is the target label. The
estimated probability of $\bv{x}$ would be the $y$-th dimension of
$(\bv{p}(\bv{x}))$, i.e., $(\bv{p}(\bv{x}))_y$.
{\bf On the training sets}, since the NNs are optimized to perform well on the
sets, only a tiny fraction of them are classified wrongly. 
To concentrate the margin distribution more around zero, is to make almost all
of predictions that are correct less confident. 
Thus, a higher expected training loss ensues.
{\bf On the test sets}, the estimated probabilities of the target class
concentrate more around middle values, resulting from lower confidence/margins in
predictions made by NNs, as shown in \cref{fig:prob_histogram} (but the
majority of values are still at the ends).  Note that wrong predictions away
from decision boundaries (with large negative margins) map to
large loss values in the surrogate loss function. Thus, though NNs with larger
AR strength have lower accuracy, they give more predictions whose estimated
probabilities are at the middle (compared with NNs with smaller AR
strength). These predictions, even if relatively more of them are wrong, maps
to smaller loss values, as shown in \cref{fig:loss_histogram}, where we plot
the histogram of loss values of test samples. In the end, it results in
expected test losses that are lower, or increase in a lower rate than the
training losses on CIFAR10/100, Tiny-ImageNet, as shown in
\cref{fig:performance}. {\bf The reduced GE gap} results from
the increased training losses, and decreased or less increased test losses.


\subsubsection{AR makes NNs smoothe predictions w.r.t. input
  perturabtions in all directions}
\label{sec:advers-robustn-makes}

\begin{figure}[h]
  \centering
  \begin{subfigure}[b]{0.235\textwidth}
    \includegraphics[width=\columnwidth]{./Fig/Histogram_of_ProbabilitySNinf_vanillaP_test.png}
    \caption{Prob. Histogram}
    \label{fig:prob_histogram}
  \end{subfigure}
  \begin{subfigure}[b]{0.235\textwidth}
    \includegraphics[width=\columnwidth]{./Fig/Histogram_of_LossSNinf_vanillaP_test.png}
    \caption{Loss Histogram}
    \label{fig:loss_histogram}
  \end{subfigure}
  \\
  \begin{subfigure}[b]{0.235\textwidth}
    \includegraphics[width=\columnwidth]{./Fig/Singular_value_var_on_CIFAR10.png}
    \caption{CIFAR10}
    \label{fig:singular_value_CIFAR10}
  \end{subfigure}
  \begin{subfigure}[b]{0.235\textwidth}
    \includegraphics[width=\columnwidth]{./Fig/Singular_value_var_on_CIFAR100.png}
    \caption{CIFAR100}
    \label{fig:singular_value_CIFAR100}
  \end{subfigure}
  \caption{{
    \bf (a)(b)} are histograms of estimated probabilities and losses of the test set sample of NNs trained with AR strength 4, 8, 16. 
    We plot a subplot of a narrower range inside the plot of the full range to better show the histograms of examples that are around the middle values induced by AR. 
    {\bf (c)(d)} are standard deviations of singular values of weight matrices of NNs at each layer trained on CIFAR10/100 with AR strength 4, 16. The AR strength 8 is dropped for clarity.}
    \label{fig:misc}
\end{figure}

The observation in \cref{sec:cms-that-concentrate} shows that AR make NNs just {\it less confident} by reducing the variance of predictions made and concentrate margins more around zero. 
In this section, we study the {\it underlying factors} of AR that make NNs become less confident.

To begin with, we show that the singular values of the weight matrix of each layer determine the perturbation in margins of samples induced by perturbations in the instance space.  
Such a connection between singular values and the perturbation of outputs of a single layer, i.e., $\text{ReLU}(\bv{W}\delta\bv{x})$, has been discussed in \cref{sec:advers-robusnt-leads}.
In the following, with \cref{lm:1}, we describe how the relatively more complex connection between margins and singular values of each weight matrix of layers of NNs holds. 
Observe that margins are obtained by applying a piece-wise linear mapping (c.f. the margin operator in \cref{def:margin}) to the activation of the last layer of a NN. 
It implies the perturbations in activation of the last layer induce changes in margins in a piece-wise linear way. 
Meanwhile, the perturbation in the activation of the last layer (induced by perturbation in the instance space) is determined by the weight matrix's singular values of each layer of NNs. 
More specifically, this is explained as follows. 
\Cref{lm:1} shows that the perturbation $\delta\bv{I}$ induced by $\delta\bv{x}$, is given by $\sum_{j=1}^{n}\int_{s_j}^{e_j}\left|\left|\prod_{i=1}^{l}\bv{W}_{i}^{q_j}\delta\bv{x}dt\right|\right|$.
Note that for each $i$, $\bv{W}^{q_i}_{i}$ is a matrix. By Cauchy interlacing law by row deletion \citep{Meek}, the singular values of $\bv{W}_{i}$, the weight matrix of layer $i$, determine the singular values of $\bv{W}^{q_j}_i$. 
Thus, suppose $l=1$, we have the change (measured in norm) induced by perturbation as $\sum_{j=1}^{n}\int_{s_j}^{e_j}\left|\left|\bv{W}_{1}^{q_j}\delta\bv{x}dt\right|\right|$. 
The singular values of $\bv{W}_{1}$ would determine the variance (of norms) of
activation perturbations induced by perturbations $\delta\bv{x}$, similarly as explained in
\cref{sec:advers-robusnt-leads} except that the norm perturbation now is obtained by
a summation of $n$ terms $\left|\left|\bv{W}_{1}^{q_j}\delta\bv{x}dt\right|\right|$
(each of which is the exact form discussed in \cref{sec:advers-robusnt-leads})
weighted by $1/(e_j - s_j)$.  Similarly, for the case where $l=2\ldots L-1$, the
singular values of $\bv{W}_{l}$ determine the variance of perturbations in the
output of layer $l$ that induced by the perturbations in the output of the
previous layer (the input to layer $l$), i.e., layer $l-1$.  Consequently, we
choose to study these singular values.

We show the standard deviation of singular values of each layer of ResNet56 trained on CIFAR10/100 earlier in \cref{fig:singular_value_CIFAR10} \cref{fig:singular_value_CIFAR100}.  
Overall, the standard deviation of singular values associated with a layer of the NN trained with AR strength 4 is mostly larger than that of the NN with AR strength 16.  
The STD reduction in CIFAR100 is relatively smaller than CIFAR10, since as observed in \cref{fig:cifar100_margin_test}, the AR induced concentration effect of margin distributions is also relatively less obvious than that in \cref{fig:cifar10_margin_test}. 
More quantitative analysis is given in \cref{sec:quant-analys-vari}.
This leads us to our {\it key results} described in \cref{sec:advers-robusnt-leads}.




\section*{Acknowledgements}

This work is supported in part by National Natural Science Foundation of China
(Grant No. 61771201), Program for Guangdong Introducing Innovative and
Enterpreneurial Teams (Grant No. 2017ZT07X183), Guangdong R\&D Key Project of
China (Grant No. 2019B010155001) and Guangzhou Key Laboratory of Body Data
Science (Grant No. 201605030011).



