
\section{Conclusions}
\label{sec:conclutions}
In this work, we presented a detection method for adversarial examples in learning-enabled CPS. 
The method is based on inductive conformal prediction and uses a VAE-based regression model to predict the target variable and compute the nonconformity of new inputs relative to the training set. 
The evaluation is based on an AEBS implemented in an open source simulator for self-driving cars. FGSM is used to generate adversarial examples. The results demonstrate that the method can efficiently detect adversarial examples with a short detection delay.  
Future work includes detection of physically realizable attacks, \added{comparing this new approach with other adversarial detection methods to study into the benifit of taking the output into consideration}, and also
investigating the generation of adversarial examples that are not detectable by the approach.
 
\section{Evaluation}
\label{sec:evaluation}
We evaluate the adversarial detection method using an advanced emergency braking system (AEBS) implemented using CARLA\cite{Dosovitskiy17}. The experiments reported use CARLA 0.9.5 on a 16-core i7 desktop with $\SI{32}{\giga\byte}$ RAM and a single RTX 2080 GPU with $\SI{8}{\giga\byte}$ video memory.

\subsection{Experimental Setup}
A typical architecture (Fig.~\ref{fig:aebs_architecture}) and its desirable behavior (Fig.~\ref{fig:aebs_senario}) of the AEBS are presented in~\cite{CaiICCPS20}. 
The objective of the AEBS is to detect an obstacle and apply appropriate brake force in order to avoid a potential collision. The initial velocity of the host vehicle is $v_0$ and the initial distance between the host car and the obstacle is $d_0$. A perception LEC receives images captured by an on-board camera, detects the nearest front obstacle on the road, and estimates the distance. The estimated distance together with the velocity of the host vehicle are fed to a reinforcement learning controller whose objective is to generate the braking force for safely stopping the vehicle between $L_{\min}$ and $L_{\max}$. 

\begin{figure}[!ht]
	\centering
	\input{./figures/AEBS_architecture}
	\caption{Advanced emergency braking system architecture~\cite{CaiICCPS20}.}
	\label{fig:aebs_architecture}
\end{figure}

\begin{figure}[!ht]
	\centering
	\input{./figures/scenario}
	\caption{Illustration of advanced emergency braking system~\cite{CaiICCPS20}.}
	\label{fig:aebs_senario}
\end{figure}

In order to simulate realistic scenarios, the initial velocity $v_0$ is uniformly sampled between $\SI[per-mode=symbol]{90}{\kilo\meter\per\hour}$ and $\SI[per-mode=symbol]{100}{\kilo\meter\per\hour}$, and the initial distance $d_0$ is based on the camera range in CARLA and is approximately $\SI{100}{\meter}$. 
The reinforcement learning controller is trained using the DDPG algorithm~\cite{DBLP:journals/corr/LillicrapHPHETS15} with $1000$ episodes and reward function which aims to stop the vehicle between $L_{\min}=\SI{1}{\meter}$ and $L_{\max}=\SI{3}{\meter}$.  It should be noted is that the sampling period used in the simulation is $\Delta t = \SI{1/20}{\second}$. The reinforcement learning controller is used only for simulation of the closed loop system and does not affect the proposed approach.
Details about the design of the reinforcement learning controller can be found in~\cite{DBLP:journals/tecs/TranCLMJK19}.


\subsection{LEC Training}
The data set for the perception and detector training consists of $8160$ images obtained by varying the initial distance $d_0$, initial velocity $v_0$ and precipitation. As described in \ref{sec:method}, the VAE and the regression network are jointly trained using the VAE-based regression model implemented as a convolutional neural network (CNN).

In the VAE-based regression model, the input firstly passes through four convolutional layers of $32/64/128/256 \times (5\times 5)$ filters with ELU activations and $2\times2$ max-pooling, and one fully connected layer of 1568 units with ELU activation. Then, the extracted features are fed into a fully connected layer with $1024$ units. The regressor shares the convolutional layers and also has two fully connected layers with $256/1$ units. 
After the regressor, two fully connected layers of $256/1024$ are used to yield distance-specific latent representations. The decoder has symmetric deconvolutional layers.  

A simple two-phase learning schedule is employed with initial searching learning rate $\eta=10^{-4}$ for $250$ epochs, and subsequently fine-tuning $\eta=10^{-5}$ for $100$ epochs. After the training, the mean absolute error of the perception LEC for training and testing are $\SI{0.32}{\meter}$ and $\SI{0.40}{\meter}$ respectively. In addition, multi-dimensional scaling (MDS)~\cite{doi:10.1111/j.1745-3984.2003.tb01108.x} is implemented to seek a low-dimensional representation of the latent space of the VAE. Fig.~\ref{fig:2dlatent} shows the 2D representations of the latent space.
From the plots, see that the dimension related to the distance is disentangled from the latent space. 

A closed-loop simulation run is shown in Fig.~\ref{fig:inDistribution}. Initially, the distance between the host and the lead car is $\SI{98.03}{\meter}$, and the velocity of the host car is $\SI[per-mode=symbol]{96.95}{\kilo\meter\per\hour}$ ($=\SI[per-mode=symbol]{26.93}{\meter\per\second}$). 
After $140$ steps or $\SI{7.0}{\second}$, the host vehicle stops at 
$\SI{1.83}{\meter}$ from the lead car. 

\begin{figure}[!ht]
	\centering

	\input{./figures/2dlatent_attack.tex}
	\caption{2D latent representations estimated by VAE-based regression model.}
	\label{fig:2dlatent}
\end{figure} 

\subsection{Adversarial Examples}
The FGSM is used to generate adversarial examples for the perception LEC. 
For a given input image $x_t$, the neural network output is the estimated distance $c_t = d_t = f(x_t)$. 
The objective is to modify $x_t$ by a small step $\epsilon=0.02$ in the direction that minimizes the loss to $\tilde{x}_t$ such that the predicted distance is close to the target value $y^{\text{target}}$. 
In our experiment, we set  $y^{\text{target}}=\SI{100}{\meter}$. 
The perception module predicts a large distance even when the car is very close to a stopped lead car. 
A comparison of the original image and the adversarial image is shown in Fig.~\ref{fig:PhysicalAttack_a} and ~\ref{fig:PhysicalAttack_c}. 
\begin{figure}[!ht]
	\centering
	\subfloat[Original image\label{fig:PhysicalAttack_a}]{
		\includegraphics[width=0.4\linewidth]{./figures/original}}
	\hfil
	\subfloat[Reconstructed original image\label{fig:PhysicalAttack_b}]{
		\includegraphics[width=0.4\linewidth]{./figures/original_reconstructed}}
	\hfil
	\subfloat[Attacked image\label{fig:PhysicalAttack_c}]{
	\includegraphics[width=0.4\linewidth]{./figures/attacked}}
	\hfil
	\subfloat[Reconstructed attacked image\label{fig:PhysicalAttack_d}]{
	\includegraphics[width=0.4\linewidth]{./figures/attacked_reconstructed}}
	\caption{Comparison of original image, image with attack and their reconstructed images.}
	\label{fig:PhysicalAttack}
\end{figure}

\subsection{Simulation Results}
In order to analyze the performance of the detection method, we generate multiple simulation episodes which contain normal and adversarial examples. 
Adversarial examples are inserted in the simulation in a time step uniformly sampled from $\{20, 21, \ldots, 60 \}$.

To illustrate the approach, we compare two episodes and plot the ground truth and predicted distance to the lead stopped car, the velocity of the host car, the $p$-values and the $S$-value of the CUSUM detector (computed using the logarithm of $M_t$). We use $N=10$ for the number of the examples generated by the VAE and $\delta=12, \tau=80$ for the parameters of the CUSUM detector. Since $M_t$ becomes very large, we choose to use $\log M_t$ to show the results.

The normal case is shown in Fig~\ref{fig:inDistribution}. The $p$-values are randomly distributed between $0$ and $1$, and the martingale is small indicating the inputs are in distribution. 
An episode with adversarial examples is shown in Fig~\ref{fig:AttackPerception}.  The adversarial example starts at $\SI{1.20}{\second}$ trying to cause the regression network to predict a larger distance than the actual one. The error starts to increase and reaches almost $\SI{20}{\meter}$. The controller is misled by the perception LEC and  does  not  stop  the  car  in time which  collides  with  the  lead  car (velocity is greater than $0$ when ground truth distance comes to $0$). The $p$-values become smaller and the detector indicates the test inputs are not from the same distribution as the training data set.  The delay for detection is smaller than $10$ frames or $\SI{0.5}{\second}$ (the sampling rate is $\SI{20}{\hertz}$).

\begin{figure}[!ht]
	\centering
	\input{./figures/inDistribution.tex}
	\caption{Episode with normal input examples.}
	\label{fig:inDistribution}
\end{figure} 

\begin{figure}[!ht]
	\centering
	\input{./figures/attackPerception.tex}
	\caption{Episode with adversarial examples.}
	\label{fig:AttackPerception}
\end{figure} 

We plot the 2D representations of the latent space generated by VAE using one episode of adversarial examples in Fig.~\ref{fig:2dlatent}. Since the latent space is conditioned by the regression prediction, the latent representations of the adversarial examples lie on regions of the space that correspond to different predictions of the regression network. While reconstructing the input for an adversarial example, the VAE generates the new examples from such a region. For example, an image reflecting a small distance is mapped to the region indicating a large distance for an adversarial example. 
Thus, the reconstructed image is sampled from the wrong region and reflects a large distance, and therefore, the reconstruction error or nonconformity score is large. We compare the original image, adversarial example, and their corresponding reconstructed images in Fig.~\ref{fig:PhysicalAttack}.


We also evaluate the approach for $100$ normal episodes and $100$ adversarial episodes by considering different values of $N$.
We run each episode and if an alarm is raised, we stop the simulation, and we check if the alarm is false. We compute the detection delay as the number of frames from the adversarial example corresponding to the alarm raised. 
Table~\ref{tab:vae-based AEBS} shows the false alarms and average delay for different $N$, $\delta$ and $\tau$.  Since the $p$-values for the adversarial examples are almost $0$, the number of the false alarms is very small and the detection delay is smaller than $10$ frames or $\SI{0.5}{\second}$.

\begin{table}[!ht]
	\centering
	\caption{VAE-based detection.}
	\label{tab:vae-based AEBS}
	\begin{tabular}{c c c c}
		\hline
		\multicolumn{1}{m{0.18\columnwidth}}{\centering Parameters $(N, \delta, \tau)$} 
		& \multicolumn{1}{m{0.20\columnwidth}}{\centering False positive}
		& \multicolumn{1}{m{0.20\columnwidth}}{\centering False negative}    
		& \multicolumn{1}{m{0.21\columnwidth}}{\centering Average delay (frames)}  \\
		
		\hline
		$5, 6, 6$ & $0/100$ & $0/100$ & $1.0$\\
		\hline
		$5, 7, 23$ & $0/100$ & $0/100$ & $4.0$\\
		
		\hline
		$10, 10, 62$ & $0/100$ & $0/100$ & $4.0$\\
		\hline
		$10, 12, 80$ & $0/100$ & $0/100$ & $6.0$\\
		
		\hline
		$20, 18, 120$ & $0/100$ & $0/100$ & $3.0$\\
		\hline
		$20, 20, 280$ & $0/100$ & $0/100$ & $9.0$\\
		\hline
	\end{tabular}
\end{table}

\subsection{Computational Efficiency}
The VAE-based regression model can predict the target variable and compute the nonconformity score in real-time without storing training data.
Table~\ref{tab:real-time} reports the minimum $(\min)$, first quartile $(Q_1)$, second quartile or median $(Q_2)$, third quartile $(Q_3)$, and maximum $(\max)$ of (1) the execution times of the LECs in AEBS and (2) the execution times of the VAE-based detectors for different values of $N$.

Since the VAE-based detection uses $N$ examples in each time step, the execution time is proportional to the number of examples generated $N$. 
The execution times are much smaller than the sampling time ($\SI{50}{\milli\second}$ in AEBS, and thus, the methods can be used for real-time out-of-distribution detection.

\begin{table}[!ht]
	\centering
	\caption{Execution times.}
	\label{tab:real-time}
	\begin{tabular}{c c c c c c c}
		\hline
		\multicolumn{1}{m{0.05\columnwidth}}{} 
		& \multicolumn{1}{m{0.05\columnwidth}}{\centering $N$} 
		& \multicolumn{1}{m{0.08\columnwidth}}{\centering $\min$ (ms)}
		& \multicolumn{1}{m{0.08\columnwidth}}{\centering $Q_1$ (ms)}
		& \multicolumn{1}{m{0.08\columnwidth}}{\centering $Q_2$ (ms)}
		& \multicolumn{1}{m{0.08\columnwidth}}{\centering $Q_3$ (ms)}
		& \multicolumn{1}{m{0.08\columnwidth}}{\centering $\max$ (ms)} \\
		\hline
		AEBS &N/A & 3.49 & 3.86 & 3.93 & 3.98 & 4.22 \\
		\hline
		\multirow{3}{*}{VAE} &5 & 18.90 & 18.96 & 18.99 & 19.02 & 19.08 \\
		& 10 & 37.89 & 37.96 & 37.99 & 38.04 & 38.10 \\
		& 20 & 76.32 & 76.47 & 76.52 & 76.71 & 77.32 \\
		\hline
	\end{tabular}
\end{table}










\section{Introduction}
\label{sec:intro}
The use of learning-enabled components (LECs) such as deep neural networks (DNNs) is on the rise in many classes of cyber-physical systems (CPS).
Semi-autonomous and autonomous vehicles, in particular, are applications where LECs play a significant role.
Although DNNs can increase the level of the autonomy by handling the uncertainty and variability of the environment, multiple studies have shown that DNNs are not robust. For example, they are vulnerable to examples with small specially human-crafted perturbations in the input that cause false predictions~\cite{7780651, goodfellow2014explaining}.

LECs rely on design-time data-driven training to learn how to operate in unstructured and dynamic environments. 
However, LECs must be deployed and operate in a real system and the input data may be different than the data used for training and testing. Adversarial examples, in particular, can cause wrong predictions and impact system safety~\cite{sitawarin2018darts}. Adversarial example detection for CPS must have a low false alarm rate while being computational efficient for real-time monitoring.

\added{
Recently, there have been many efforts to defend against adversarial examples especially in the context of classification.
Autoencoder based methods, such as~\cite{DBLP:conf/ccs/MengC17, an2015variational},  utilize the reconstruction error or the reconstruction accuracy to detect adversarial examples. The autoencoders learn and encode the properties of the training data with latent representations. If a test example is from the same distribution as the training set, we expect a small reconstruction error.
The work in~\cite{pmlr-v80-ruff18a} presents an approach which aims to map the in-distribution input data into a hypersphere of minimum volume characterized by center $c$ and radius $R$. The adversarial examples are expected to be mapped to points out of the hypersphere.
Typically, existing methods do not take into consideration the dynamical behavior. 
}

In our previous work~\cite{CaiICCPS20}, we developed an approach which leverages inductive conformal prediction~\cite{balasubramanian2014conformal,volkhonskiy2017inductive} and inductive anomaly detection~\cite{laxhammar2015inductive} for out-of-distribution detection in learning-enabled CPS. The detection algorithm is based on a variational autoencoder (VAE) which is designed independently of the perception LEC, and the approach considers only the input data. Taking into account the predicted output can be beneficial for detection of adversarial examples especially since they are crafted so that a small perturbation of the inputs causes a large change in the output.
Instead of training the regression and VAE model used for detection independently, this paper uses a VAE-based regression model which integrates the regression model into the VAE and takes into account the predicted output for adversarial detection. 


The main contribution of the paper is an approach for detection of adversarial examples for regression LECs in CPS based on inductive conformal anomaly detection. In order to take into consideration the output of the regression LEC, the proposed approach employs a VAE-based regression model which is learned jointly combining the VAE and regressor by conditioning the latent representation of the VAE on the target variable of the regressor. The inductive conformal anomaly detection is based on the VAE-based regression model and can be performed efficiently for high-dimensional inputs. 

Another contribution is the empirical evaluation of the approach using an advanced emergency braking system (AEBS) implemented in CARLA~\cite{Dosovitskiy17}, an open source simulator for self-driving cars.
The AEBS uses a perception LEC to detect the nearest front obstacle and estimate the distance from the host vehicle based on the images captured by an onboard camera. 
The distance estimated by the LEC is used by a reinforcement learning controller together with the vehicle’s velocity to compute the desired braking in order to safely stop the vehicle. 
The fast gradient sign method  (FGSM)~\cite{goodfellow2014explaining} is used to generate adversarial examples for the perception LEC which cause large error in the distance estimation and result to a collision. It should be noted that although such attacks may be not physically realizable, they provide a framework to analyze the robustness of the LEC. The evaluation results show that the approach can detect such adversarial examples effectively with a short detection delay.

The rest of the paper is organized as follows. Section~\ref{sec:problem} introduces system model and formulates the problem. Section~\ref{sec:method} describes the VAE-based regression model and the detection algorithm based in inductive conformal anomaly detection. Section~\ref{sec:evaluation} presents the evaluation results using the AEBS. Section~\ref{sec:conclutions} concludes the paper.

\section{Variational Autoencoder for Regression and Adversarial Examples Detection}
\label{sec:method}
In this section, we propose a detection method based on inductive conformal prediction~\cite{laxhammar2015inductive} for a Variational Autoencoder (VAE) based regression model~\cite{zhao2019variational} that extends the work in~\cite{CaiICCPS20}.
The inductive conformal prediction is  used to generate a nonconformity score of the perception input based on the reconstruction error of the VAE  comparing to the reconstruction error of the training examples to measure the trustworthy of the input.
In the following, we introduce the detection system in details.

\subsection{Variational Autoencoder for Regression}
A VAE is a generative model whose encodings' distribution is regularised during the training in order to ensure that its latent space has good properties allowing the generation of new data\cite{DBLP:journals/corr/KingmaW13}. 
The objective is to model the relationship between the observation $x$ and the low-dimensional latent variable $z$. The architecture presented in~\cite{zhao2019variational} integrates a regression model into the generative model 
to disentangle the regression target variable from the latent space. The architecture of the VAE-based regression model is shown in Fig.~\ref{fig:model}. 

\begin{figure}[!ht]
	\centering
	\input{./figures/model.tex}
	\caption{VAE-based regression model\cite{zhao2019variational}.}
	\label{fig:model}
\end{figure}  

Similar to a traditional VAE, the encoder attempts to represent the input on the latent space using a Gaussian distribution $p(z)$. Instead of using a single Gaussian prior, the VAE-based regression model conditions the prior on the regression target variable $c$ such that the new prior $p(z|c)$, called \textit{latent generator}, can be used to sample a specific latent representation for a given target variable $c$. 
On the other branch of the architecture, a regression network $q(c|x)$ is used to infer the target variable and its uncertainty.
The total loss for the VAE based regression is defined as 
\begin{equation*} \label{eq:vaeLoss}
\begin{split}
\mathcal{L}(\theta, \phi_c, \phi_z; x) = &- D_{\text{KL}}(q_{\phi_c}(c|x) || p(c))\\
&+   \mathbb{E}_{z\sim q_{\phi_z}(z|x)}[\log p_\theta(x|z)]\\
&- \mathbb{E}_{c\sim q_{\phi_c}(c|x)}[ D_{\text{KL}}(q_{\phi_z}(z|x)||p(z|c))],
\end{split}
\end{equation*}
where $\theta$, $\phi_{z}$ and $\phi_{c}$ are the neural network parameters. The first term regularizes the prediction of $c$ with a ground-truth prior $p(c)$. The second term is the model fit trying to reconstruct the input from the latent representations. The third term is the KL divergence between the approximate posterior and the regression-specific prior $p(z|c)$. 
Both the VAE and the regression model can be trained as a single network and the target variable is disentangled from the latent space.

\subsection{Adversarial Examples for Regression Model}
A common approach to generate adversarial examples is the fast gradient sign method (FGSM)~\cite{goodfellow2014explaining}, which computes an adversarial input by adding a small perturbation making the prediction move towards the reversed direction of the gradient descent.
Usually, FGSM is used for classification problems. In this work, we apply this method to generate adversarial examples for regression neural network that can cause large output error.


For a given input $x$ and a target output $y^{\text{target}}$, the neural network output is the regression estimation $c = f(x)$. The objective is to change $x$ to $\tilde{x}$ such that the error between $y^{\text{target}}$ and $f(\tilde{x})$ is minimized, which can be formulated by  $J(\tilde{x}, y^{\text{target}}) = |f(\tilde{x}) - y^{\text{target}} |^2$.


\subsection{Adversarial Detection}
In this subsection, we introduce inductive conformal prediction (ICP)\cite{balasubramanian2014conformal} method for anomaly detection. First, we discuss a general framework for conformal prediction. Next, we show how we define nonconformity score based on VAE's output and use it for anomaly detection.
\subsubsection{Conformal prediction}
\label{sec:cp}
The core idea of the method is the \textit{nonconformity measure} defined by a function $A$ that assigns a value as \textit{nonconformity score} indicating how different a test example is from the training data set. A large nonconformity score corresponds to a strange example with respect to the training data set. The original training set $\{ (x_1,y_1), \ldots, (x_l, y_l)\}$ is split into a proper training set $\{ (x_1,y_1), \ldots, (x_m, y_m)\}$ and a calibration set $\{ (x_{m+1},y_{m+1}), \ldots, (x_l, y_l)\}$. For each calibration example, the nonconformity scores $\alpha_{m+1}, \ldots, \alpha_{l}$ are precomputed relative to the proper training set. 
For a test example $(x^\prime_{k}, y^\prime_{k})$, the nonconformity score $\alpha^\prime_{k}$ is computed using the same way relative to the proper training set and the $p$-value is defined as the fraction of calibration examples that have nonconformity scores greater than or equal to the $\alpha^\prime_{k}$
\begin{equation}
p_k = \frac{|\{i=m+1, \ldots, l\} \, | \, \alpha_i \geq \alpha^\prime_k|}{l-m}.
\label{eq:p-value}
\end{equation}
If the $p$-value is smaller than a predefined threshold $\epsilon \in (0,1)$, the test example can be classified as a conformal anomaly. Using a single $p$-value for detecting the anomaly will make the detector oversensitive.
The robustness of the detector can be considerably improved if multiple $p$-values are used.
In \cite{vovk2005algorithmic}, it  is shown that if the test examples are independent and identically distributed (IID) , the $p$-values are independent and uniformly distributed in $[0,1]$.
Further, a martingale test is used to test the hypothesis that the  $p$-values are independent and uniformly distributed,  and the martingale value is used as an indicator for the unusual test example~\cite{vovk2005algorithmic}. 
in~\cite{Fedorova:2012:PMT:3042573.3042693}, the \textit{simple mixture martingale} is used which is defined as 
\begin{equation}
M = \int_{0}^{1}\prod_{i=1}^{N}\epsilon p_i^{\epsilon-1}d\epsilon.
\label{eq:martingale}
\end{equation}
$ M $ will grow only if there are many small $p$-values in the sequence. More details about the martingales can be found in~\cite{Fedorova:2012:PMT:3042573.3042693}. 

Given the VAE model, we can rely on it  to generate multiple examples sampling from the learned latent space probability distribution. The samples which are similar to the input are IID and can be used to generate $p$-values that are independent and uniformly distributed in $[0,1]$. Then, the martingale can be used to test this hypothesis and detect if the test example comes from the training data set distribution. 
Next, we explain how we compute the nonconformity score.


\subsubsection{Nonconformity measure and adversarial detection algorithm}
In work~\cite{CaiICCPS20}, the reconstruction error (squared error between the input $x$ and the generated output $\hat{x}$) of VAE model is used as the nonconformity measure
\begin{equation*}
\alpha = A_{\text{VAE}}(x, \hat{x}) = ||x-\hat{x}||^2.
\end{equation*}
In this paper, we use the same nonconformity measure based on the output of the VAE-based regression model which integrates the regression model and the VAE (Fig.~\ref{fig:model}). 

During the offline phase, 
the training set $\{ (x_1, y_1),  \ldots, (x_l, y_l)\}$ is split into a proper training set $\{ (x_1, y_1),  \ldots, (x_m, y_m)\}$ and a calibration set $\{ (x_{m+1}, y_{m+1}),  \ldots, (x_l, y_l)\}$. 
The VAE-based regression model is trained by utilizing the proper training data set.
For each example in the calibration data set, the nonconformity score is precomputed using the 
nonconformity measure $A_{\text{VAE}}$.
 
At runtime, a sequence of test inputs $(x_1^\prime, \ldots, x_t^\prime, \ldots)$ is processed one-by-one. 
Consider the input $x^\prime_t$. The method samples from the posterior distribution of the latent space to generate $N$ new examples $\hat{x}_{t,1}^\prime, \ldots, \hat{x}_{t,N}^\prime$, and
the nonconformity score $\alpha^\prime_{t,k}$ and the corresponding $p$-value for each generated example $\hat{x}_{t,k}^\prime$ are computed using $A_{\text{VAE}}$ and Eq.~(\ref{eq:p-value}).  
Since the generated examples are IID, the $p$-values $(p_{t,1}, \ldots, p_{t,N})$ are independent and uniformly distributed in $[0,1]$. Therefore, the martingale in Eq.~(\ref{eq:martingale}) can be used for detection.
The martingale denoted by $M_t$ has a large value if there are many small $p$-values in the sequence $(p_{t,1}, \ldots, p_{t,N})$ which indicates an out-of-distribution input. 

In order to robustly detect when the martingale becomes consistently large, \cite{CaiICCPS20} uses a stateful CUSUM (Cumulative Sum) detector to generate alarms for out-of-distribution inputs by keeping track of the historical information of the martingale values. 
The detector is defined as $S_1=0$ and $S_{t} = \max(0, S_{t-1} + M_{t-1} - \delta)$,  
where $\delta$ prevents $S_t$ from increasing consistently when the inputs are 
in the same distribution as the training data. An alarm is raised whenever $S_t$ is 
greater than a threshold $S_t > \tau $ which can be optimized using empirical
data~\cite{basseville1993detection}. Typically, after an alarm the test is 
reset with $ S_{t+1} = 0$.
Algorithm~\ref{alg:vae} describes the steps of the method. 

\begin{algorithm}[!ht]
	\caption{Adversarial examples detection using the VAE-based regression model}
	\label{alg:vae}
	\begin{algorithmic}[1]
		\renewcommand{\algorithmicrequire}{\textbf{Input:}}
		\renewcommand{\algorithmicensure}{\textbf{Output:}}
		\REQUIRE Input training set $\{ (x_1, y_1),  \ldots, (x_l, y_l)\}$, test examples $(x^\prime_1, \ldots, x^\prime_t, \ldots)$, number of calibration examples $l-m$, 
		number of examples to be sampled $N$, 
		stateful detector threshold $\tau$ and parameter $\delta$
		\ENSURE  Output boolean variable $Anom_{t}$, regression result $c_t$
		
		\renewcommand{\algorithmicrequire}{\textbf{Offline:}}
		\renewcommand{\algorithmicensure}{\textbf{Online:}}
		
		\REQUIRE 
		\STATE Split the training set $\{ (x_1, y_1),  \ldots, (x_l, y_l)\}$ into the proper training set $\{ (x_1, y_1),  \ldots, (x_m, y_m)\}$ and calibration set $\{ (x_{m+1}, y_{m+1}),  \ldots, (x_l, y_l)\}$
		\STATE Train a VAE-based regression model $f(\cdot)$ using the proper training set and get the corresponding nonconformity measure $A_{\text{VAE}}(\cdot)$
		\FOR {$j = m+1$ to $l$}
		\STATE Sample $\hat{x}_j$ using the trained VAE
		\STATE $ \alpha_j = A_{\text{VAE}}(x_j, \hat{x}_j) $
		\ENDFOR
		
		\ENSURE
		\FOR {$t = 1, 2, \ldots$}
		\FOR {$k = 1$ to $N$}
		\STATE Sample $ \hat{x}^\prime_{t,k} $ using the trained VAE
		\STATE $\alpha^\prime_{t,k} = A_{\text{VAE}}(x^\prime_t,\hat{x}^\prime_{t,k})$
		\STATE $p_{t,k} = \frac{|\{i=m+1,\ldots,l\} \,|\, \alpha_i \geq \alpha^\prime_{t,k}|}{l - m}$ 
		\ENDFOR
		\STATE $M_t =  \int_0^1 \prod_{k=1}^N \epsilon p_{t,k}^{\epsilon - 1} d\epsilon$
		\IF {$t=1$} 
		\STATE $S_t = 0$
		\ELSE
		\STATE $S_t = \max(0, S_{t-1} + M_{t-1} -\delta)$
		\ENDIF
		\STATE $Anom_{t} \leftarrow S_t > \tau $
		\STATE $c_t = f(x^\prime_t)$ 
		\ENDFOR
	\end{algorithmic} 
\end{algorithm}
\section{Variational Autoencoder for Regression and Adversarial Examples Detection}
\label{sec:method}
The detection algorithm proposed in this paper extends the work in~\cite{CaiICCPS20} by using a VAE-based regression model~\cite{zhao2019variational}.
The method is based on an LEC architecture which integrates the regression model into the VAE and uses inductive conformal anomaly detection~\cite{laxhammar2015inductive}.  

\subsection{Variational Autoencoder for Regression}
A Variational Autoencoder (VAE) is a generative model whose encodings distribution is regularised during the training in order to ensure that its latent space has good properties allowing the generation of new data~\cite{DBLP:journals/corr/KingmaW13}. 
The objective is to model the relationship between the observation $x$ and the low-dimensional latent variable $z$. The architecture presented in~\cite{zhao2019variational} integrates a regression model into the generative model 
to disentangle the regression target variable from the latent space. The architecture of the VAE-based regression model is shown in Fig.~\ref{fig:model}. 

\begin{figure}[!ht]
	\centering
	\input{./figures/model.tex}
	\caption{VAE-based regression model\cite{zhao2019variational}.}
	\label{fig:model}
\end{figure}  

Similar to a traditional VAE, the encoder attempts to represent the input on the latent space using a Gaussian distribution $p(z)$. Instead of using a single Gaussian prior, the VAE-based regression model conditions the prior on the regression target variable $c$ such that the new prior $p(z|c)$, called \textit{latent generator}, can be used to sample a specific latent representation for a given target variable $c$. 
On the other branch of the architecture, a regression network $q(c|x)$ is used to inference the target variable and its uncertainty.
The loss function is defined as 
\begin{equation*} \label{eq:vaeLoss}
\begin{split}
\mathcal{L}(\theta, \phi_c, \phi_z; x) = &- D_{\text{KL}}(q_{\phi_c}(c|x) || p(c))\\
&+   \mathbb{E}_{z\sim q_{\phi_z}(z|x)}[\log p_\theta(x|z)]\\
&- \mathbb{E}_{c\sim q_{\phi_c}(c|x)}[ D_{\text{KL}}(q_{\phi_z}(z|x)||p(z|c))],
\end{split}
\end{equation*}
where $\theta$, $\phi_{z}$ and $\phi_{c}$ are the neural network parameters. The first term regularizes the prediction of $c$ with a ground-truth prior $p(c)$. The second term is the model fit trying to reconstruct the input from the latent representations. The third term is the KL divergence between the approximate posterior and the regression-specific prior $p(z|c)$. 
Both the VAE and the regression model can be jointly trained as a single network and the target variable is disentangled from the latent space.

\subsection{Adversarial Examples for Regression Model}
A common approach to generate adversarial examples is the fast gradient sign method (FGSM)~\cite{goodfellow2014explaining}, which computes an adversarial input by adding a small perturbation in the direction of the gradient. 
Instead of using this method to make the target network to misclassify the input, in our work, this FGSM is utilized to cause a large error in the output of the regression network. 

For a given input $x$ and a target output $y^{\text{target}}$, the neural network output is the regression estimation $c = f(x)$. The objective is to change $x$ to $\tilde{x}$ such that the error between $y^{\text{target}}$ and $f(\tilde{x})$ is minimized. This can be achieved by considering the cost function $J(\tilde{x}, y^{\text{target}}) = |f(\tilde{x}) - y^{\text{target}} |^2$ for generating the adversarial examples. 


\subsection{Adversarial Detection}
\subsubsection{Conformal prediction}
\label{sec:cp}
Our method is based on inductive conformal prediction (ICP)~\cite{balasubramanian2014conformal}. The core idea of the method is to compute a \textit{nonconformity measure} defined by a function $A$ that assigns a value as \textit{nonconformity score} indicating how different a test example from the training data set. A large nonconformity score corresponds to a strange example with respect to the training data set. The original training set $\{ (x_1,y_1), \ldots, (x_l, y_l)\}$ is split into a proper training set $\{ (x_1,y_1), \ldots, (x_m, y_m)\}$ and a calibration set $\{ (x_{m+1},y_{m+1}), \ldots, (x_l, y_l)\}$. For each calibration example, the nonconformity scores $\alpha_{m+1}, \ldots, \alpha_{l}$ are precomputed relative to the proper training set. 
For a test example $(x^\prime_{k}, y^\prime_{k})$, the nonconformity score $\alpha^\prime_{k}$ is computed using the same way relative to the proper training set and the $p$-value is defined as the fraction of calibration examples that have nonconformity scores greater than or equal to the $\alpha^\prime_{k}$
\begin{equation}
p_k = \frac{|\{i=m+1, \ldots, l\} \, | \, \alpha_i \geq \alpha^\prime_k|}{l-m}.
\label{eq:p-value}
\end{equation}
If the $p$-value is smaller than a predefined threshold $\epsilon \in (0,1)$, the test example can be classified as a conformal anomaly. Using a single $p$-value for detecting the anomaly will make the detector oversensitive.
The robustness of the detector can be considerably improved if multiple $p$-values are used.
In \cite{vovk2005algorithmic}, it  is shown that if the test examples are independent and identically distributed (IID), the $p$-values are independent and uniformly distributed in $[0,1]$.
Further, a martingale test is used to test the hypothesis that the  $p$-values are independent and uniformly distributed,  and the martingale value is used as an indicator for the unusual test example~\cite{vovk2005algorithmic}. 
In~\cite{Fedorova:2012:PMT:3042573.3042693}, the \textit{simple mixture martingale} is used which is defined as 
\begin{equation}
M = \int_{0}^{1}\prod_{i=1}^{N}\epsilon p_i^{\epsilon-1}d\epsilon.
\label{eq:martingale}
\end{equation}
$ M $ will grow only if there are many small $p$-values in the sequence. More details about the martingales can be found in~\cite{Fedorova:2012:PMT:3042573.3042693}. 

The main idea in our approach is to use the generative VAE model to generate multiple examples sampling from the learned latent space probability distribution. The samples which are similar to the input are IID and can be used to generate $p$-values that are independent and uniformly distributed in $[0,1]$. Then, the martingale can be used to test this hypothesis and detect if the test example comes from the training data set distribution. 


\subsubsection{Nonconformity measure and adversarial detection algorithm}
In~\cite{CaiICCPS20}, the reconstruction error (squared error between the input $x$ and the generated output $\hat{x}$) of the VAE model is used as the nonconformity measure
\begin{equation*}
\alpha = A_{\text{VAE}}(x, \hat{x}) = ||x-\hat{x}||^2.
\end{equation*}
In this paper, we use the same nonconformity measure based on the output of the VAE-based regression model which integrates the regression model and the VAE (Fig.~\ref{fig:model}). 

During the offline phase, 
the training set $\{ (x_1, y_1),  \ldots, (x_l, y_l)\}$ is split into a proper training set $\{ (x_1, y_1),  \ldots, (x_m, y_m)\}$ and a calibration set $\{ (x_{m+1}, y_{m+1}),  \ldots, (x_l, y_l)\}$. 
The VAE-based regression model is trained by utilizing the proper training data set.
For each example in the calibration data set, the nonconformity score is precomputed using the 
nonconformity measure $A_{\text{VAE}}$.
 
At runtime, a sequence of test inputs $(x_1^\prime, \ldots, x_t^\prime, \ldots)$ is processed one-by-one. 
Consider the input $x^\prime_t$. The method samples from the posterior distribution of the latent space to generate $N$ new examples $\hat{x}_{t,1}^\prime, \ldots, \hat{x}_{t,N}^\prime$, and
the nonconformity score $\alpha^\prime_{t,k}$ and the corresponding $p$-value for each generated example $\hat{x}_{t,k}^\prime$ are computed using $A_{\text{VAE}}$ and Eq.~(\ref{eq:p-value}).  
Since the generated examples are IID, the $p$-values $(p_{t,1}, \ldots, p_{t,N})$ are independent and uniformly distributed in $[0,1]$. Therefore, the martingale in Eq.~(\ref{eq:martingale}) can be used for detection.
The martingale denoted by $M_t$ has a large value if there are many small $p$-values in the sequence $(p_{t,1}, \ldots, p_{t,N})$ which indicates an out-of-distribution input. 

In order to robustly detect when the martingale becomes consistently large, \cite{CaiICCPS20} uses a stateful CUSUM detector to generate alarms for out-of-distribution inputs by keeping track of the historical information of the martingale values. 
The detector is defined as $S_1=0$ and $S_{t} = \max(0, S_{t-1} + M_{t-1} - \delta)$,  
where $\delta$ prevents $S_t$ from increasing consistently when the inputs are 
in the same distribution as the training data. An alarm is raised whenever $S_t$ is 
greater than a threshold $S_t > \tau $ which can be optimized using empirical
data~\cite{basseville1993detection}. Typically, after an alarm the test is 
reset with $ S_{t+1} = 0$.
Algorithm~\ref{alg:vae} describes the steps of the method. 

\begin{algorithm}[!ht]
	\caption{Adversarial examples detection using the VAE-based regression model}
	\label{alg:vae}
	\begin{algorithmic}[1]
		\renewcommand{\algorithmicrequire}{\textbf{Input:}}
		\renewcommand{\algorithmicensure}{\textbf{Output:}}
		\REQUIRE Input training set $\{ (x_1, y_1),  \ldots, (x_l, y_l)\}$, test examples $(x^\prime_1, \ldots, x^\prime_t, \ldots)$, number of calibration examples $l-m$, 
		number of examples to be sampled $N$, 
		stateful detector threshold $\tau$ and parameter $\delta$
		\ENSURE  Output boolean variable $Anom_{t}$, regression result $c_t$
		
		\renewcommand{\algorithmicrequire}{\textbf{Offline:}}
		\renewcommand{\algorithmicensure}{\textbf{Online:}}
		
		\REQUIRE 
		\STATE Split the training set $\{ (x_1, y_1),  \ldots, (x_l, y_l)\}$ into the proper training set $\{ (x_1, y_1),  \ldots, (x_m, y_m)\}$ and calibration set $\{ (x_{m+1}, y_{m+1}),  \ldots, (x_l, y_l)\}$
		\STATE Train a VAE-based regression model $f(\cdot)$ using the proper training set and get the corresponding nonconformity measure $A_{\text{VAE}}(\cdot)$
		\FOR {$j = m+1$ to $l$}
		\STATE Sample $\hat{x}_j$ using the trained VAE
		\STATE $ \alpha_j = A_{\text{VAE}}(x_j, \hat{x}_j) $
		\ENDFOR
		
		\ENSURE
		\FOR {$t = 1, 2, \ldots$}
		\FOR {$k = 1$ to $N$}
		\STATE Sample $ \hat{x}^\prime_{t,k} $ using the trained VAE
		\STATE $\alpha^\prime_{t,k} = A_{\text{VAE}}(x^\prime_t,\hat{x}^\prime_{t,k})$
		\STATE $p_{t,k} = \frac{|\{i=m+1,\ldots,l\} \,|\, \alpha_i \geq \alpha^\prime_{t,k}|}{l - m}$ 
		\ENDFOR
		\STATE $M_t =  \int_0^1 \prod_{k=1}^N \epsilon p_{t,k}^{\epsilon - 1} d\epsilon$
		\IF {$t=1$} 
		\STATE $S_t = 0$
		\ELSE
		\STATE $S_t = \max(0, S_{t-1} + M_{t-1} -\delta)$
		\ENDIF
		\STATE $Anom_{t} \leftarrow S_t > \tau $
		\STATE $c_t = f(x^\prime_t)$ 
		\ENDFOR
	\end{algorithmic} 
\end{algorithm}

\section{System Model and Problem Formulation}
\label{sec:problem}

A simplified CPS architecture is shown in Fig.~\ref{fig:cps_arch}. The perception component observes and interprets the environment and feeds the information into the controller which, possibly using additional sensors (feedback from the plant), applies an action to the physical plant in order to achieve some task. In responding to the action, the state of the physical plant changes and the environment is observed and interpreted again to continue the system operation. LECs are extensively used for perception and control tasks in CPS in order to increase the level of autonomy.

\begin{figure}[!ht]
	\centering
	\input{./figures/cps}
	\caption{Simplified CPS control architecture~\cite{CaiICCPS20}.}
	\label{fig:cps_arch}
\end{figure}

Perception and control may be implemented using LECs (e.g., neural networks). An LEC can be designed using learning techniques such as supervised, unsupervised, and reinforcement learning. 
It is assumed that the LECs are successfully trained and evaluation of training and testing errors is satisfactory.  
However, adversarial examples with small input perturbations may cause the LECs to generate outputs with large errors and impact the safety of the system. 

The problem considered in this paper is detecting efficiently adversarial examples.
During the system operation, the inputs arrive one by one and an adversarial example can
cause large prediction error. 
The objective of the detection is to efficiently detect if the LEC input is adversarial. It should be noted that although such attacks may be not physically realizable, they provide a framework to analyze the robustness of the LEC.

