\section{Introduction}
\label{sec: intro}

The state of the art in supervised deep learning has dramatically improved over the past decade \cite{lecun2015deep}. Deep learning techniques have led to significant advances in applications such as: face recognition \cite{mehdipour2016comprehensive}; object detection \cite{zhao2019object}; and natural language processing \cite{young2018recent}. Despite these successes, deep learning techniques are not resilient to evasion attacks on test inputs and poisoning attacks on training data \cite{goodfellow2014explaining,szegedy2013intriguing,GDG17}. The adversarial vulnerability of deep neural networks (DNN) have restricted their application, motivating researchers to develop effective defense methods. The focus of this paper is to develop a novel deep defense framework inspired by the mammalian immune system.

Current defense methods can be broadly divided into three classes: (1) Adversarial example detection \cite{metzen2017detecting,feinman2017detecting,grosse2017statistical}; (2) Robust training \cite{madry17,zhang2019theoretically,cohen2019certified}; and (3) Deep classifiers with natural robustness \cite{papernot2018deep,samangouei2018defense}. The first class of methods defends a DNN using simple outlier 
detection models for detecting adversarial examples. However, it has been shown that adversarial detection methods are not perfect and can be easily defeated \cite{carlini2017adversarial}. Robust training aims to harden the model to deactivate attacks such as evasion attacks. Known robust training methods are often tailored to a certain level of attack strength in the context of $\ell_p$-perturbation. Moreover, the trade-off between accuracy and robustness presents challenges \cite{zhang2019theoretically}. Recently alternative defense strategies have been proposed that implement different structures that are naturally resilient to evasion attacks on DNNs. Despite these advances, current methods have difficulty providing an acceptable level of robustness to novel attacks \cite{athalye2018obfuscated}.



To design an effective defense, it is natural to consider a learning strategy that emulates mechanisms of the naturally robust biological immune system. In this paper, we propose a new framework, Robust Adversarial Immune-inspired Learning System (RAILS), that can effectively defend deep learning architectures against aggressive attacks.
Built upon a class-wise k-Nearest Neighbor (kNN) structure, given a test sample RAILS finds an initial small population of proximal samples, balanced across different classes, with uniform label distribution. RAILS then promotes the specificity of the label distribution towards the ground truth label through an evolutionary optimization.
RAILS can efficiently correct adversarial label flipping by balancing label diversity against specificity.
While RAILS can be applied to defending against many different types of attacks, in this paper we restrict attention to evasion attacks on the input. Figure~\ref{fig: overview_result} (a) shows that RAILS outperforms existing methods on various types of evasion attacks.



\begin{figure}[h]
  \centering
 
  	\begin{minipage}[t]{0.5\linewidth}
		\centering
		\includegraphics[trim=190 0 200 0,clip,width=0.95\linewidth]{radar}
				
					{\scriptsize \center (a) }
	\end{minipage}%
	\begin{minipage}[t]{0.5\linewidth}
		\centering
		\includegraphics[trim=0 0 0 0,clip,width=0.95\linewidth]{RAILS_Arch}
				
					{\scriptsize \center (b) }
	\end{minipage}
 
  \caption{\small{\textbf{RAILS launches the best defense against different types of attacks (left panel) using the adversarial deep learning architecture shown on right panel.} (a) Radar plot showing that RAILS has higher robust accuracy than the  adversarially trained CNN \cite{madry17} and Deep k-Nearest Neighbor (DkNN) \cite{papernot2018deep} in defending against four types of attacks: PGD \cite{madry17}, Fast Gradient Sign Method (FGSM) \cite{goodfellow2014explaining}, Square Attack \cite{andriushchenko2020square}, and a (customized) kNN-attack.  (b) For each test input, a special type of data - plasma data - is generated by evolutionary optimization, that contributes to predicting the class of the input (test) sample. Another type of data - memory data - is generated and stored to help defend against similar attacks in the future. Plasma data and memory data are analogous to plasma B cells and memory B cells in the immune system.}}
  \label{fig: overview_result}
 
\end{figure}
  

\paragraph{Contributions.}

Compared to existing defense methods, we make the following contributions:

$\bullet$ RAILS achieves better adversarial robustness by assigning a uniform label distribution to each input and evolving it to a distribution that is concentrated about the input's true label class. (see Section~\ref{sec: RAILS} and Table~\ref{tab2: acc_overall})

$\bullet$ RAILS incorporates a life-long robustifying process by adding synthetic ``virtual data'' to the training data. (see Section~\ref{sec: RAILS} and Table~\ref{tab4: acc_cnn_hard})


$\bullet$ RAILS evolves the distribution via mutation and cross-over mechanisms and is not restricted to $\ell_p$ or any other specific type of attack. (see Section~\ref{sec: alg_detail} and Figure~\ref{fig: overview_result})


$\bullet$ We demonstrate that RAILS improves robustness of existing methods for different types of attacks (Figure~\ref{fig: overview_result}). Specifically, RAILS improves robustness against PGD attack by $\geq 5.62\%/12.5\%/10.32\%$ for the  MNIST, SVHN, and CIFAR-10 datasets (Table~\ref{tab2: acc_overall}). Furthermore, we show that RAILS life-long learning process provides a $2.3\%$ robustness improvement
with only $5\%$ augmentation of the training data  (Table~\ref{tab4: acc_cnn_hard}).

$\bullet$ RAILS is the first adversarial defense framework to be based on the biology of the adaptive immune system. In particular: (a) RAILS computationally emulates the principal mechanisms of the immune response (Figure~\ref{fig: bio_comp}); and (b) our computational and biological experiments demonstrate the fidelity of the emulation -  the learning patterns of RAILS and the immune system are closely aligned (Figure~\ref{fig: curves_bio_rails}).



\paragraph{Related work.}
After it was established that DNNs were vulnerable to evasion attacks \cite{szegedy2013intriguing}, different types of defense mechanisms have been proposed in recent years. One intuitive idea is to eliminate the adversarial examples through outlier detection, including training an additional discrimination sub-network \cite{metzen2017detecting,grosse2017statistical} and using kernel density estimation \cite{feinman2017detecting}. The above approaches rely on the fundamental assumption that the distributions of benign and adversarial examples are distinct, an assumption that has been challenged in \cite{carlini2017adversarial}.

In addition to adversarial attack detection, other methods have been proposed that focus on robustifying the deep architecture during the learning phase \cite{madry17,cohen2019certified,zhang2019theoretically}.
Though these defenses are effective against adversarial examples with a certain level of $\ell_p$ attack strength,
they have limited defense power on stronger attacks, and there is often a sacrifice in overall classification accuracy. In contrast, RAILS is developed to defend against diverse powerful attacks with less sacrifice in accuracy,
and can improve any model's robustness, including robust models.

Another approach is to leverage different structures or classifiers on robustifying deep classifiers \cite{papernot2018deep,samangouei2018defense}. An example is the deep k-Nearest Neighbor (DkNN) classifier \cite{papernot2018deep} that robustifies against instance perturbations 
by applying kNN's to features extracted from each layer. However, a single kNN classifier applied on the whole dataset is easily to be fooled by strong attacks, as illustrated in Figure~\ref{fig: vis_compare_rails}.
On the other side, RAILS incorporates a \textit{diversity} to \textit{specificity} defense mechanism which can provide additional robustness to existing DNNs.

Another line of research relevant to ours is adversarial transfer learning \cite{liu2019transferable,Shafahi2020Adversarially}, which aims to maintain robustness when there is covariate shift from training data to test data. We remark that covariate shift is naturally handled by RAILS as it emulates the immune system which adapts to novel mutated attacks.



\begin{figure}[h]
\centerline{\includegraphics[width=.85\textwidth]{vis_layer_rails}}
\caption{\small{\textbf{Representation examples showing RAILS correcting (improving) the wrong (unconfident) predictions by CNN and kNN.} kNN predict all three examples with $\le 100\%$ confidence of the ground truth class, and CNN gets wrong predictions for images 2 and 4. RAILS provides correct predictions with high confidence scores.}}
\label{fig: vis_compare_rails}
\end{figure}



\paragraph{Notation and preliminaries.}
Given a mapping $f: \mathbb{R}^{d} \rightarrow \mathbb{R}^{d^{\prime}}$ and $\mathbf x_1, \mathbf x_2 \in \mathbb{R}^{d}$, we first define the affinity score $A(f;\mathbf x_1, \mathbf x_2)$ between $\mathbf x_1$ and $\mathbf x_2$. This affinity score can be defined in many ways, e.g, cosine similarity, inner product, or inverse $\ell_p$ distance, but here we use  $A(f;\mathbf x_1, \mathbf x_2) = -\|f(\mathbf x_1) - f(\mathbf x_2)\|_2$,  the negative Euclidean distance. In the context of DNN, $f$ denotes the feature mapping from input to feature representation, and $A$ measures the similarity between two inputs. Higher affinity score indicates higher similarity.


\section{RAILS: overview}\label{sec: RAILS}

In this section, we give an overview of RAILS, and provide a comparison to the natural immune system.




The architecture of RAILS is illustrated in the right panel of Figure~\ref{fig: overview_result}. For each selected hidden layer $l$, RAILS builds class-wise kNN architectures on training samples $\mathcal D_{tr}$. Then for each test input $\mathbf x$, a population of candidates is selected and goes through an evolutionary optimization process to obtain the optimal solution. In RAILS, two types of data are obtained after the evolutionary optimization: `plasma data' for optimal predictions of the present inputs, and `memory data' for the defense against future attacks. These two types of data correspond to plasma B cells and memory B cells in the biological system, and play important roles in \textit{static learning} and \textit{adaptive learning}, respectively.


\paragraph{Defense with static learning.}
 
Adversarial perturbations can severely affect deep classifiers, forcing the predictions to be dominated by adversarial classes rather than the ground truth. For example, a single kNN classifier is vulnerable to adversarial inputs, as shown in Figure~\ref{fig: vis_compare_rails}. The purpose of static learning is to address this issue, i.e., maintaining or increasing the prediction probability of the ground truth $p_l^{y_{\text{true}}(\mathbf x)}(\mathbf x)$ when the input $\mathbf x$ is manipulated by an adversary. The key components include \textbf{(i)} a label initialization via class-wise kNN search on $\mathcal D_{tr}$ that guarantees labels across different classes are uniformly distributed for each input; and \textbf{(ii)} an evolutionary data-label optimization that promotes label distribution specificity towards the input's true label class. Our hypothesis is that the covariate shift of the adversarial examples from the distribution of the ground truth class is small in the input space, and therefore new examples inherited from parents of ground truth class $y_{\text{true}}$ have higher chance of reaching high-affinity. The evolutionary optimization thus promotes the label \textit{specificity} towards the ground truth. The solution denotes the data-label pairs of examples with high-affinity to the input, which we call plasma data. After the process, a majority vote of plasma data is used to make the class prediction. We refer readers to Section~\ref{sec: alg_detail} for more implementation details and Section~\ref{sec: perf_layer} for visualization. In short, static learning defenses seek to correct the predictions of current adversarial inputs and do not plan ahead for future attacks.



\paragraph{Defense with adaptive learning.}
Different from static learning, RAILS adaptive learning tries to use information from past attacks to harden the classifier to defend against future attacks. Hardening is done by leveraging another set of data - memory data generated during evolutionary optimization. Unlike plasma data, memory data is selected from examples with moderate-affinity to the input, which can rapidly adapt to new variants of the adversarial examples. This selection logic is based on maximizing coverage over future attack-space rather than optimizing for the current attack. Adaptive learning is a life-long learning process and can use hindsight to greatly enhance resilience of  $p_l^{y_{\text{true}}(\mathbf x)}(\mathbf x)$ to attacks. This paper will focus on static learning and single-stage adaptive learning that implements a single cycle of classifier hardening.


\begin{figure}[h]
\centerline{\includegraphics[width=.95\textwidth]{bio_comp_v5}}
\caption{\small{\textbf{Simplified immune system (left) and RAILS computational workflow (right):} Both systems are composed of a four-step process, which includes initial detection (sensing), recruiting candidates for diversity (flocking), enlarging population size and promoting specificity (affinity maturation) \cite{de2015dynamics}, and obtaining the final solution (consensus).}}
\label{fig: bio_comp}
\end{figure}



\paragraph{A biological perspective.} RAILS is inspired by and closely associated with the biological immune system. The architecture of the adaptive immune system ensures a robust response to foreign antigens, splitting the work between active sensing and competitive growth to produce an effective antibody. Figure~\ref{fig: bio_comp} displays a comparison between the immune system workflow and the RAILS workflow. Both systems are composed of a four-step process. For example, RAILS emulates flocking from the immune system by initializing a population of candidates that provide \textit{diversity}, and emulates affinity maturation via an evolutionary optimization process to promote \textit{specificity}. Similar to the functions of plasma B cells and memory B cells generated in the immune system, RAILS generates plasma data for predictions of the present inputs (immune system defends antigens though generating plasma B cells) and generates memory data for the defense against future attacks (immune system continuously increases its degree of robustness through generating memory B cells). We refer readers to Section~\ref{sec: app1} in the Appendix for a table of correspondences between RAILS operands and mechanisms in the immune system. In addition, the learning patterns of RAILS and the immune system are closely aligned, as shown in Figure~\ref{fig: curves_bio_rails}.











\section{Details of RAILS workflow}\label{sec: alg_detail}


Algorithm~\ref{RAILS} shows the four-step workflow of RAILS.
We explain each step below.

\paragraph{Sensing.} This step performs an initial discrimination between adversarial and benign inputs to prevent the RAILS computation from becoming overwhelmed by false positives, i.e., only implementing the main steps of RAILS on suspicious inputs. While there are many outlier detection procedures that could be used for this step  \cite{feinman2017detecting,xu2017feature}, we can exploit the fact that the DNN and kNN applied on hidden layers will tend to make similar class predictions for benign inputs. Thus we propose using an cross-entropy measure to generate an {\it adversarial threat score} for each input $\mathbf x$. In the main text results, we skip the sensing stage since the major benefit from sensing is providing an initial detection. We refer readers to Section~\ref{sec: app5} in the Appendix for more details.





\paragraph{Flocking.}
The initial population from each class needs to be selected with a certain degree of affinity measured using the hidden representations in order to satisfy our hypothesis, as illustrated in Figure~\ref{fig: curves_bio_rails}. By constructing class-wise k-Nearest Neighbor (kNN) architecture, we find the kNN that have the highest initial affinity score to the input data from each class and each selected layer. Mathematically, we select
\begin{align}\label{eq: knn}
    \begin{array}{ll}
\mathcal{N}_l^c = \{(\hat{\mathbf x}, y_c)|R_c(\hat{\mathbf x}) \le K, (\hat{\mathbf x}, y_c) \in \mathcal D_c\}~~ \text{Given}\\ A(f_l; \mathbf x_i^c, \mathbf x) \le A(f_l; \mathbf x_j^c, \mathbf x) \Leftarrow R_c(i) > R_c(j), \forall c \in [C], l \in \mathcal{L},  \forall i, j \in [n_c],
    \end{array}
\end{align}
where $x$ is the input. $\mathcal{L}$ is the set of the selected layers. $\mathcal D_c$ is the training data from class $c$ and the size $|\mathcal D_c|=n_c$. $R_c: [n_c] \rightarrow [n_c]$ is a ranking function that sorts the indices based on the affinity score. In the adaptive learning context, if the memory database has been previously populated, flocking will select the nearest neighbors using both the training data and the memory data. The immune system leverages flocking step to find initial B cells and form temporary structures for affinity maturation \cite{de2015dynamics}. Note that in RAILS, the kNN sets $\mathcal{N}_l^c $ are \textit{constructed independently} for each class, thereby ensuring that every class is \textit{fairly represented} in the initial population.


\paragraph{Affinity maturation (evolutionary optimization).} As flocking brings diversity to the label distribution of the initial population, the affinity maturation step, in contrast, promotes specificity towards the ground truth class. Here we use evolutionary optimization to generate new examples (offspring) from the existing examples (parents) in the population. The evolution happens within each class \textit{independently}, and new generated examples from different classes are not affected by one another before the consensus stage. The first generation parents in each class are the $K$ nearest neighbors found by \eqref{eq: knn} in the flocking step, where $K$ is the number of nearest neighbors. Given a total population size $TC$, the $0$-th generation is obtained by copying each nearest neighbor $T/K$ times with random mutations. Given the population of class $c$ in the $(g-1)$-st generation $\mathcal{P}_c^{(g-1)}=[\mathbf x_c(1), \mathbf x_c(2), \cdots, \mathbf x_c(T)] \in \mathbb{R}^{d \times T}$, the candidates for the $g$-th generation are selected as
\begin{align}\label{eq: random}
    \begin{array}{ll}
\mathcal{\hat{P}}_c^{(g-1)} = \mathcal{P}_c^{(g-1)} Z_c^{(g-1)},
    \end{array}
\end{align}
where $Z_c^{(g-1)} \in \mathbb{R}^{T \times T}$ is a binary selection matrix whose columns are independent and identically distributed draws from  Mult$(1, \mathbf P_c)$, the multinomial distribution with probability vector $\mathbf P_c\in [0,1]^T$. The process can also be viewed as creating new nodes from existing nodes in a Preferential Attachment (PA) evolutionary graph \cite{barabasi1999emergence}, where the details can be viewed in Section~\ref{sec: app4} in the Appendix. RAILS generates new examples through the operations \textit{selection}, \textit{mutation}, and \textit{cross-over}, which will be discussed in more detail later. After new examples are generated, RAILS calculates each example’s affinity relative to the input. The new examples are associated with labels that are inherited from their parents, which always come from the same class.
According to our hypothesis in Section \ref{sec: RAILS}, examples inherited from parents of the ground truth class $y_{\text{true}}$ have a higher chance of reaching high-affinity, and thereby the population members with high-affinity are concentrating about the input's true class.


\paragraph{Consensus.} Consensus is responsible for the final selection and predictions. In this step, RAILS selects generated examples with high-affinity scores to be plasma data, and examples with moderate-affinity scores are saved as memory data. The selection is based on a ranking function. 
\begin{align}\label{eq: aff_mat}
    \begin{array}{ll}
S_{\text{opt}} = \{(\Tilde{\mathbf x}, \Tilde{y})|R_g(\Tilde{\mathbf x}) \le \gamma|\mathcal{P}^{(G)}|, (\Tilde{\mathbf x}, \Tilde{y}) \in \mathcal{P}^{(G)}\},
    \end{array}
\end{align}
where $R_g: [|\mathcal{P}^{(G)}|] \rightarrow [|\mathcal{P}^{(G)}|]$ is the same ranking function as $R_c$ except that the domain is the set of cardinality of the final population $\mathcal{P}^{(G)}$. $\gamma$ is a proportionality parameter and is selected as $0.05$ and $0.25$ for plasma data and memory data, respectively. Note that the memory data can be selected in each generation.
For simplicity, we select memory data only in last generation. Memory data will be saved in the secondary database and used for model hardening.

Given that all examples in the population are associated with a label inherited from their parents, RAILS uses majority voting of the plasma data for prediction of the class label of $\mathbf x$.



\begin{algorithm}[h]
\caption{Robust Adversarial Immune-inspired Learning System (RAILS)}
\label{RAILS}
\begin{algorithmic}[1]
\REQUIRE Test data point $\mathbf x$; Training dataset $\mathcal D_{tr} = \{\mathcal D_{1}, \mathcal D_{2},\cdots,\mathcal D_{C}\}$; Number of Classes $C$; Model $M$ with feature mapping $f_l(\cdot)$, $l \in \mathcal{L}$; Affinity function $A$.\\
\noindent{\textbf{First Step: Sensing}}
\STATE{Check the threat score given by an outlier detection strategy to detect the threat of $x$.}\\
\noindent{\textbf{Second Step: Flocking}}
\FOR{$c = 1, 2, \dots, C$}
\STATE{In each layer $l \in \mathcal{L}$, find the k-nearest neighbors $\mathcal{N}_l^c$ of $\mathbf x$ in $\mathcal D_{c}$ by ranking the affinity score.}
\ENDFOR\\
\noindent{\textbf{Third Step: Affinity Maturation}}

\STATE{\textbf{For} each layer $l \in \mathcal{L}$, \textbf{do}}
\STATE{Generate $\mathcal{P}_c^{(0)}$ through mutating each of $\mathbf x^{\prime} \in \mathcal{N}_l^c$ $T/K$ times, $\forall c \in [C]$.}
\FOR{$g = 1, 2, \dots, G$ }
\FOR{$t = 1, 2, \dots, T$}
\STATE{Select data-label pairs $(\mathbf x_c, y_c)$, $(\mathbf x_c^{\prime}, y_c)$ from $\mathcal{P}_c^{(g-1)}$ based on $\mathbf P_c^{(g-1)}$.}
\STATE{$\mathbf x_{\text{os}} = Mutation \big (Crossover(\mathbf x_c,\mathbf x_{c}^{\prime})\big)$; $\mathcal{P}_c^{(g)} \longleftarrow (\mathbf x_{\text{os}}, y_c)$.}
\ENDFOR
\ENDFOR
\STATE{Calculate the affinity score $A(f_{l}; \mathcal{P}^{(G)}, \mathbf x), \forall c \in [C]$ given $\mathcal{P}^{(G)}=\mathcal{P}_1^{(G)}\bigcup \cdots \bigcup \mathcal{P}_C^{(G)}$.}
\STATE{\textbf{end For}}\\
\noindent{\textbf{Fourth Step: Consensus}}
\STATE{Select the top $5\%$ as plasma data $S_{p}^l$ and the top $25\%$ as memory data $S_{m}^l$ based on the affinity scores, $\forall l \in \mathcal{L}$; Obtain the prediction $y$ of $\mathbf x$ using the majority vote of the plasma data.}
\STATE {\bfseries Output:} $y$, the memory data $S_{m} = \{S_{m}^1, S_{m}^2,\cdots,S_{m}^{|\mathcal{L}|}\}$
\end{algorithmic}
\end{algorithm}

\paragraph{Computational cost.} 
The computation cost of RAILS mainly comes from the flocking and affinity maturation stage. kNN structure construction in flocking is a fixed setup cost that can be handled off-line with fast approximate kNN approximation  \cite{bewley2013advantages,rajaraman2011mining}. The additional computational cost in affinity maturation can be \textit{addressed from three perspectives}. First, the evolutionary optimization can be replaced by mean field approximation. Second, leveraging parallel computing can accelerate the process since each sample is generated and utilized separately. Third, the sensing step can prevent the RAILS from being overwhelmed by false positives, therefore reducing computational burden. More discussion can be viewed in Section~\ref{sec: app4} in the Appendix.

\paragraph{Operations in the evolutionary optimization.}
Three operations support the creation of new examples: selection, cross-over, and mutation.
The \textit{selection} operation is shown in \eqref{eq: random}. We compute the selection probability for each candidate through a softmax function.
\begin{align}\label{eq: prob}
    \begin{array}{ll}
\mathbf P(\mathbf x_i) &= Softmax(A(f_{l}; \mathbf x_i, \mathbf x)/\tau) = \frac{\exp{(A(f_{l}; \mathbf x_i, \mathbf x)/\tau)}}{\sum_{\mathbf x_j \in S}\exp{(A(f_{l}; \mathbf x_j, \mathbf x)/\tau)}},
    \end{array}
\end{align}
where $S$ is the set of data points and $x_i \in S$. $\tau>0$ is the sampling temperature that controls sharpness of the softmax operation. Given the selection probability $\mathbf P$, defined on the current generation in \eqref{eq: prob},  the candidate set $\{(\mathbf x_i,y_i)\}_{i=1}^T$ for the next generation is randomly drawn (with replacement).

The \textit{cross-over} operator combines
two parents $\mathbf x_{c}$ and $\mathbf x_{c}^{\prime}$ from the same class, and generates new offspring by randomly selecting each of its elements (pixels) from the corresponding element of either parent. Mathematically,
\begin{align}\label{eq: crossover}
    \begin{array}{ll}
\mathbf x_{\text{os}}^{\prime} = Crossover(\mathbf x_{c}, \mathbf x_{c}^{\prime}) = \left \{ \begin{array}{rcl}
\mathbf x_{c}^{(i)} & \mbox{with prob}~ \frac{A(f_{l}; \mathbf x_{c}, \mathbf x)}{A(f_{l}; \mathbf x_{c}, \mathbf x)+A(f_{l}; \mathbf x_{c}^{\prime}, \mathbf x)}, \\ \mathbf x_{c}^{\prime{(i)}} & \text{with prob}~ \frac{A(f_{l}; \mathbf x_{c}^{\prime}, \mathbf x)}{A(f_{l}; \mathbf x_{c}, \mathbf x)+A(f_{l}; \mathbf x_{c}^{\prime}, \mathbf x)}
\end{array} \right. \forall i \in [d],
    \end{array}
\end{align}
where $i$ represents the $i$-th entry of the example and $d$ is the dimension of the example. The \textit{mutation} operation randomly and independently  mutates an offspring with probability $\rho$, adding uniformly distributed noise in the range $[-\delta_{\text{max}},-\delta_{\text{min}}] \cup [\delta_{\text{min}},\delta_{\text{max}}]$. The resulting perturbation vector is subsequently clipped to satisfy the domain constraint that examples lie in $[0,1]^d$.
\begin{align}\label{eq: mutation}
    \begin{array}{ll}
 \mathbf x_{\text{os}} = Mutation(\mathbf x_{\text{os}}^{\prime}) = \text{Clip}_{[0,1]}\big(\mathbf x_{\text{os}}^{\prime} + \boldsymbol{1}_{[Bernoulli(\rho)]}  \mathbf u([-\delta_{\text{max}},-\delta_{\text{min}}] \cup [\delta_{\text{min}},\delta_{\text{max}}])\big),
    \end{array}
\end{align}
where $\boldsymbol{1}_{[Bernoulli(\rho)]}$ takes value $1$ with probability $\rho$ and value $0$ with probability $1-\rho$. $\mathbf u([-\delta_{\text{max}},-\delta_{\text{min}}] \cup [\delta_{\text{min}},\delta_{\text{max}}])$ is the vector in $\mathbb R^d$ having i.i.d. entries drawn from the punctured uniform distribution $\mathcal{U}([-\delta_{\text{max}},-\delta_{\text{min}}] \cup [\delta_{\text{min}},\delta_{\text{max}}])$. $\text{Clip}_{[0,1]}(\mathbf x)$ is equivalent to $\max(0, \min(\mathbf x, 1))$.








\section{Experimental results}\label{sec: experiments}
We conduct experiments in the context of image classification. We compare RAILS with standard Convolutional Neural Network Classification (CNN) and Deep k-Nearest Neighbors Classification (DkNN) \cite{papernot2018deep} on the MNIST \cite{lecun1998gradient}, SVHN \cite{netzer2011reading}, and CIFAR-10 \cite{Krizhevsky2009learning}. We test our framework using a four-convolutional-layer neural network for MNIST, and VGG16 \cite{SZ2014} for SVHN and CIFAR-10. We refer readers to Section~\ref{sec: app2} in the Appendix for more details of datasets, models, and parameter selection. In addition to the benign test examples, we also generate the same amount of adversarial examples using a $20$($10$)-step PGD attack \cite{madry17} for MNIST, SVHN and CIFAR-10. The attack strength is $\epsilon = 40/60$ for MNIST, and $\epsilon = 8$ for SVHN and CIFAR-10 by default. The performance will be measured by standard accuracy (SA) evaluated using benign (unperturbed) test examples and robust accuracy (RA) evaluated using the adversarial (perturbed) test examples.

\subsection{Performance in single layers}\label{sec: perf_layer}

\begin{wrapfigure}{r}{70mm}
\vspace*{-7mm}
  \centering
  \begin{adjustbox}{max width=0.48\textwidth }
  \begin{tabular}{@{\hskip 0.00in}c  @{\hskip 0.00in}c @{\hskip 0.02in} @{\hskip 0.02in} c @{\hskip 0.02in} @{\hskip 0.02in}c }
 \begin{tabular}{@{}c@{}}  
\vspace*{0.005in}\\
\rotatebox{90}{\parbox{17em}{\centering \Large \textbf{Adv examples ($\epsilon=60$)}}}
 \\
\end{tabular} 
&
\begin{tabular}{@{\hskip 0.02in}c@{\hskip 0.02in}}
     \begin{tabular}{@{\hskip 0.00in}c@{\hskip 0.00in}}
     \parbox{10em}{\centering \Large Conv1}  
    \end{tabular} 
    \\
 \parbox[c]{20em}{\includegraphics[width=20em]{conv1eps60new}} 
\end{tabular}
&
 \begin{tabular}{@{\hskip 0.02in}c@{\hskip 0.02in}}
      \begin{tabular}{@{\hskip 0.00in}c@{\hskip 0.00in}}
     \parbox{10em}{\centering \Large Conv2}
    \end{tabular} 
    \\
 \parbox[c]{20em}{\includegraphics[width=20em]{conv2eps60new}} 
\end{tabular}

\end{tabular}
  \end{adjustbox}
    \caption{\small{\textbf{RAILS has fewer incorrect predictions for those data that DkNN gets wrong.} Confusion Matrices of adversarial examples classification in Conv1 and Conv2 (RAILS vs. DkNN).}}
  \label{fig: conf_mat}
 \vspace*{-5mm}
\end{wrapfigure}
\paragraph{RAILS outperforms DkNN on single layer.}
We first test RAILS in a single layer of the CNN model and compare the obtained accuracy with the results from the DkNN. Table~\ref{tab1: acc_layer} shows the comparisons in the input layer, the first convolutional layer (Conv1), and the second convolutional layer (Conv2) on MNIST. One can see that for both standard accuracy and robust accuracy, RAILS performs better than the DkNN in the hidden layers and achieve better results in the input layer. The input layer results indicate that RAILS can also outperform supervised learning methods like kNN. The confusion matrices in Figure~\ref{fig: conf_mat} show that RAILS has fewer incorrect predictions for those data that DkNN gets wrong. Each value in Figure~\ref{fig: conf_mat} represents the percentage of intersections of RAILS (correct or wrong) and DkNN (correct or wrong).


\begin{table}[h]
\vspace*{-0.01in}
\begin{center}
\caption{\small{
\textbf{RAILS outperforms DkNN on single layers.} SA/RA performance of RAILS versus DkNN in single layer (MNIST).}}
\label{tab1: acc_layer}
\resizebox{0.8\textwidth}{!}{
\begin{tabular}{l||c|c|c|c|c|c}
\hline
\hline
& \multicolumn{2}{c|}{SA}  & \multicolumn{2}{c|}{RA ($\epsilon=40$)} & \multicolumn{2}{c}{RA ($\epsilon=60$)} \\ [0.5ex] 
\hline
& \bf{RAILS} & DkNN & \bf{RAILS} & DkNN & \bf{RAILS} & DkNN \\
\hline 
Input &  \bf{97.53\%} & 96.88\% & \bf{93.78\%} & 91.81\% & \bf{88.83\%} & 85.54\% \\
\hline
 Conv1 &  \bf{97.77\%} & 97.4\% & \bf{92.56\%} & 90.84\% & \bf{84.18\%} & 81.01\% \\
\hline 
Conv2 &  \bf{97.78\%} & 97.42\% & \bf{89.29\%} & 88.26\% & \bf{73.42\%} & 69.18\% \\

\hline
\hline
\end{tabular}}
\end{center}
\vspace*{-2mm}
\end{table}










\paragraph{Ablation study.}
Using the CIFAR-10 dataset and the third convolutional layer of a VGG16 model, we study how each parameter in RAILS affects the performance. We list our major conclusions here: \textbf{(i)} increasing the number of nearest neighbors in a certain range helps to increase performance ; \textbf{(ii)} ; higher mutation probability leads to an increase in robust accuracy \textbf{(iii)} ; the magnitude of mutation is sensitive to the input data, but may be optimized to achieve high robust accuracy. We refer readers to Section~\ref{sec: app6} in the Appendix for more details. We also show that when we turn off the affinity maturation stage, the robust accuracy drops from $59.2\%$ to $55.65\%$ (on $2000$ test examples), indicating that affinity maturation step is critical.




\begin{figure}[h]
  \centering
  \begin{adjustbox}{max width=0.9\textwidth }
    \begin{tabular}{@{\hskip 0.00in}c  @{\hskip 0.02in} c @{\hskip 0.02in} @{\hskip 0.02in} c }

 \begin{tabular}{@{}c@{}}  
\rotatebox{90}{\parbox{10em}{\centering \scriptsize\textbf{\begin{tabular}[c]{@{}c@{}}  Example 1 \\ RAILS: bird \\ DkNN: cat  \end{tabular}}}}
 \\

\rotatebox{90}{\parbox{6.5em}{\raggedleft \scriptsize \textbf{\begin{tabular}[c]{@{}c@{}} Example 2 \\ RAILS: bird \\ DkNN: horse \end{tabular}}}}
\\
\end{tabular} 
&
 \begin{tabular}{@{\hskip 0.02in}c@{\hskip 0.02in}c@{\hskip 0.02in}}
 \begin{tabular}{@{\hskip 0.02in}c@{\hskip 0.02in}c@{\hskip 0.02in}c@{\hskip 0.02in}c@{\hskip 0.02in}}
\begin{tabular}{@{\hskip 0.02in}c@{\hskip 0.02in}}
\\
 \parbox[c]{10em}{              }{\includegraphics[width=6em]{ex0_layer}}
 \\
  \parbox[c]{10em}{              }{\includegraphics[width=6em]{ex3_layer}} 
\end{tabular}

 \begin{tabular}{@{\hskip 0.02in}c@{\hskip 0.02in}}
\\
\parbox[c]{10em}{\includegraphics[trim=5 0 0 0,clip,width=9.5em]{pop0}}
 \\
  \parbox[c]{10em}{\includegraphics[trim=5 0 0 0,clip,width=9.5em]{pop3}} 


\end{tabular}


 \begin{tabular}{@{\hskip 0.02in}c@{\hskip 0.02in}}
\\
 \parbox[c]{10em}{\includegraphics[trim=5 0 0 0,clip,width=9.5em]{fit0}}
  \\
 \parbox[c]{10em}{\includegraphics[trim=5 0 0 0,clip,width=9.5em]{fit3}}


\end{tabular}





\end{tabular}
\end{tabular} 

\vspace*{-0.08in}
\end{tabular}
  \end{adjustbox}
    \caption{\small{Proportion and affinity of the population from the ground truth class of input with respect to the generation number (RAILS on CIFAR-10). We plot all curves by selecting data points with affinity in the top $5\%$ of all classes' data points in each generation. (1) Second column: Data from the true class occupies the majority of the population when the generation number increases (2) Third column: Affinity maturation over multiple generations produces increased affinity (after a temporary decrease in the searching phase) within the true class.}}
  \label{fig: curves_running}
\end{figure}




\subsection{RAILS learning process}\label{sec: vis}
Flocking provides a balanced initial population while affinity maturation within RAILS creates new examples in each generation. To better understand the capability of RAILS, we can visualize the changes of some key indices during runtime.


\begin{wrapfigure}{r}{75mm}
\vspace*{-7mm}
\centerline{\includegraphics[width=.46\textwidth]{bio_rails_curve}}
\caption{\small{\textbf{RAILS emulates the learning patterns of the adaptive immune response.}
Correspondence between learning curves of the natural immune system \textit{in vitro} experiment (left) and the RAILS computational experiment (right). Adversarial input in RAILS corresponds to antigen in the immune system. When the initial candidates are selected based on the input (the green lines), RAILS and the immune system can both jump out of local optimal and find the correct solution. When the candidates are selected based on a different input (the red dashed lines), both responses cannot converge.}}
\label{fig: curves_bio_rails}
\end{wrapfigure}

\paragraph{Visualization of RAILS evolutionary process.}
Picking the top $5\%$ data points with the highest affinity in each generation, Figure~\ref{fig: curves_running} shows how the population proportion and (exponentiated) affinity score of the examples (whose labels are the same as the inputs' ground truth labels) change when the generation number increases. We show two examples from CIFAR-10 here. DkNN makes wrong predictions in both examples. The second column depicts the proportion of the true class in the selected population of each generation. Data from the true class occupies the majority of the population when the generation number increases, which indicates that RAILS can obtain a correct prediction and a high confidence score simultaneously. Meanwhile, affinity maturation over multiple generations yields increasing affinity within the true class, as shown in the third column.




\paragraph{RAILS versus the immune system.}

To demonstrate that the proposed RAILS computational system captures important properties of the immune system, we compare the learning curves of the two systems in Figure~\ref{fig: curves_bio_rails}. In RAILS, we treat each test input (potentially the adversarial example) as an antigen. Figure~\ref{fig: curves_bio_rails} shows that both the immune system and RAILS have similar learning patterns. One can also see that both the immune system and RAILS can escape from a local optimal under strong attacks. The difference between the green and red curves is that the initial population for the red curve is found based on another test input (antigen), which has less correlation to the present input (antigen). The non-convergence of the red curve indicates that the initial population should be selected close to the input, and the flocking with kNN search plays the role. We refer readers to Section~\ref{sec: app3} in the Appendix for more details.









\subsection{Overall performances in different scenarios}

\paragraph{Defend against PGD attack on different datasets.} We compare RAILS with CNN and DkNN in terms of standard accuracy (SA) and robust accuracy (RA).
The results are shown in Table~\ref{tab2: acc_overall}. On MNIST with $\epsilon=60$, one can see that RAILS delivers a $5.62\%$ improvement in RA over DkNN without appreciable loss of SA. On CIFAR-10 (SVHN), RAILS leads to $10.32\%$ ($12.5\%$) and $19.4\%$ ($46\%$) robust accuracy improvements compared to DkNN and CNN, respectively. We refer readers to Section~\ref{sec: app6} in the Appendix for more results under different strengths and types of attacks. Note that there is no competitive relationship between RAILS and robust training since RAILS is a general method that can improve any models' robustness, even a robust trained model.


\begin{table}[h]
\begin{center}
\caption{\small{\textbf{RAILS achieves higher robust accuracy (RA) at small cost of standard accuracy (SA)} for all three datasets (MNIST, SVHN and CIFAR-10) as compared to CNN and DkNN.
}}
\label{tab2: acc_overall}
\resizebox{0.85\textwidth}{!}{
\begin{tabular}{l|c|c|c|c|c|c}
\hline
\hline
& \multicolumn{2}{c|}{MNIST ($\epsilon=60$)}  & \multicolumn{2}{c|}{SVHN ($\epsilon=8$)} & \multicolumn{2}{c}{CIFAR-10 ($\epsilon=8$)} \\ [0.5ex] 
\hline
& SA & RA & SA & RA & SA & RA \\
\hline 
\bf{RAILS (ours)} & 97.95\% & \bf{76.67\%} & 90.62\% & \bf{48.26\%} & 82\% & \bf{52.01\%} \\

CNN & \bf{99.16\%} & 1.01\% & \bf{94.55\%} & 1.66\% & \bf{87.26\%} & 32.57\% \\

DkNN & 97.99\% & 71.05\% & 93.18\% & 35.7\% & 86.63\% & 41.69\%   \\

\hline
\hline
\end{tabular}}
\end{center}
\vspace*{-4mm}
\end{table}

\paragraph{Diversity versus specificity.}
DkNN finds a single group of kNN that achieves label distribution specificity at the beginning. Compared to DkNN, RAILS assigns a uniform label distribution to each input and achieves label specificity through the evolutionary optimization process. Results in Tabel~\ref{tab2: acc_overall} show that the diversity to specificity in RAILS provides additional robustness with a little sacrifice on benign accuracy compared to achieving specificity at the beginning, illustrating a tradeoff between robustness (diversity) and accuracy (specificity).

\paragraph{Defend against various attacks.}
RAILS defends various attacks can be viewed in Figure~\ref{fig: overview_result}. Fast Gradient Sign Method (FGSM) is a fast alternative version of PGD. Square Attack (SA) \cite{andriushchenko2020square} is a type of black-box attack. We also apply a (customized) kNN-attack that is directly applied on the flocking step. On CIFAR-10, RAILS improves the robust accuracy of CNN (DkNN) on PGD/FGSM/SA/kNN Attack by $19.43\%/11.18\%/11.5\%/11.81\%$ ($10.31\%/6.24\%/3.2\%/7.7\%$). More experimental results can be found in Section~\ref{sec: app6} in the Appendix.

\begin{wraptable}{r}{57mm}
\vspace*{-8mm}
\begin{center}
\caption{\small{\textbf{When implemented with memory data and single-stage adaptive learning (SSAL), RAILS hardens DkNN against future attacks.} 
}}
\label{tab4: acc_cnn_hard}
\vspace{.05in}
\resizebox{0.4\textwidth}{!}{
\begin{tabular}{l||c|c}
\hline
\hline 
 & SA & RA ($\epsilon=60$) \\
\hline
DkNN & 98.5\%  & 68.3\%  \\
\hline
\bf{DkNN-SSAL} & 98.5\%  & \bf{70.6\%}  \\

\hline
\hline
\end{tabular}}
\end{center}
\vspace*{-5mm}
\end{wraptable}
\paragraph{Single-Stage Adaptive Learning.}

The previous experiments demonstrate that static learning is effective in predicting present adversarial inputs. Here we show that using single-stage adaptive learning (SSAL), i.e., one-time hardening in the test phase, can improve the robustness of depth classifier. We generate $3000$ memory data from a group of test data in MNIST by feeding them into the RAILS framework. We then test whether the memory data can help DkNN defend the future evasion attack. The memory data works together with the original training data. We then randomly select another group of test data and generate $1000$ adversarial examples for evaluation. Table~\ref{tab4: acc_cnn_hard} shows that the SSAL improves RA of DkNN by $2.3\%$ with no SA loss using $3000$ memory data ($5\%$ of training data).





\section{Conclusion}
Inspired by the immune system, we proposed a new defense framework for deep learning models. The proposed Robust Adversarial Immune-inspired Learning System (RAILS) has a one-to-one mapping to a simplified architecture immune system and its learning behavior aligns with {\em in vitro} biological experiments. RAILS incorporates static learning and adaptive learning, contributing to a robustification of predictions and dynamic model hardening, respectively. The experimental results demonstrate the effectiveness of RAILS. We believe this work is fundamental and delivers valuable principles for designing robust deep models. In future work, we will dig deeper into the mechanisms of the immune system's adaptive learning (life-long learning) and covariate shift adjustment, which will be  consolidated into our computational framework. 










