\section{Introduction}\label{sec:introduction}

Deep neural networks~(DNN) have been successfully applied to a wide variety of machine learning tasks.
For instance, trained neural networks can reach human-level accuracy in image classification~\citep{russakovsky2015imagenet}.
However, \citet{szegedy2014intriguing} showed that such classifiers can be fooled by adding an imperceptible perturbation to the input image.
Since then, there has been \textcolor{edit}{extensive} research in this area known as \textit{adversarial machine learning}, trying to design more powerful attacks and devising more robust neural networks.
Today, this area encompasses a broader type of data than images, with video~\citep{jiang2019black}, graphs~\citep{zugner2018adversarial}, text~\citep{liang2018deep}, and other types of data classifiers being attacked.

In this regard, the design of stronger adversarial attacks plays a crucial role in understanding the nature of possible real-world threats.
The ultimate goal of such studies is to help neural networks \textcolor{edit}{become more robust} against such adversaries.
This line of research is extremely important as even the slightest flaw in some real-world applications of DNNs such as self-driving cars can \textcolor{edit}{have} severe, irreparable consequences~\citep{eykholt2018robust}.

In general, adversarial attack approaches \textcolor{edit}{can be} classified into two broad categories: white-box and black-box.
In \textit{white-box} adversarial attacks, the assumption is that the threat model has full access to the target DNN.
This way, adversaries can leverage their knowledge about the target model to generate adversarial examples (for instance, by taking the gradient of the neural network).
In contrast, \textit{black-box} attacks assume that they do not know the internal structure of the target model a priori.
Instead, they can only \textit{query} the model about some inputs, and work with the labels or confidence levels associated with them~\citep{yuan2019adversarial}.
Thus, black-box attacks seem to be making more realistic assumptions. 
In the beginning, black-box attacks were mostly thought of as the transferability of white-box adversarial examples to unseen models~\citep{papernot2016transfer}.
Recently, however, there has been more research to attack black-box models directly.

In this paper, we introduce AdvFlow: a black-box adversarial attack that makes use of pre-trained normalizing flows to generate adversarial examples.
In particular, we utilize flow-based methods pre-trained on clean data to model the probability distribution of possible adversarial examples around a given image.
Then, by exploiting the notion of \textit{search gradients} from \textit{natural evolution strategies~(NES)}~\citep{wierstra2008nes, wierstra2014natural}, we solve the black-box optimization problem associated with adversarial example generation to adjust this distribution.
At the end of this process, we wind up having a data distribution whose realizations are likely to be adversarial.
Since this density is constructed on the top of the original data distribution estimated by normalizing flows, we see that the generated perturbations take \textcolor{edit}{on} the structure of data rather than an additive noise (see Figure~\ref{fig:celeba_example}).
This property impedes distinguishing AdvFlow examples from clean data for adversarial example detectors, as they often assume that the adversaries come from a different distribution \textcolor{edit}{than the} clean data.
Moreover, we prove a lemma to conclude that adversarial perturbations generated by the proposed approach can be approximated by a normal distribution with dependent components.
We then put our model under test and show its effectiveness in generating adversarial examples with 1) less detectability, 2) higher success rate, 3) lower number of queries, and 4) higher rate of transferability on defended models compared to the similar method of $\mathcal{N}$\textsc{Attack}~\citep{li2019nattack}.

In summary, we make the following contributions:
\begin{itemize}
    \item We introduce AdvFlow, a black-box adversarial attack that leverages the power of normalizing flows in modeling data distributions. To the best of our knowledge, this is the first work that explores the use of flow-based models in the design of adversarial attacks.
	\item We prove a lemma about the adversarial perturbations generated by AdvFlow. As a result of this lemma, we deduce that AdvFlows can generate perturbations with dependent elements, while this is not the case for $\mathcal{N}$\textsc{Attack}~\citep{li2019nattack}.
	\item We show the power of the proposed approach in generating adversarial examples that have a similar distribution to the data. As a result, our method is able to mislead adversarial example detectors for they often assume adversaries come from a different distribution than the clean data. We then see the performance of the proposed approach in attacking some of the most recent adversarial training defense techniques. 
\end{itemize}

\begin{figure}[tb!]
	\centering
	\begin{subfigure}{.16\textwidth}
		\centering
		\includegraphics[width=1.\textwidth]{CelebA_AdvFlow_Pert.png}
		\caption*{(a)}
	\end{subfigure}
	\hspace*{-0.46em}
	\begin{subfigure}{.16\textwidth}
		\centering
		\includegraphics[width=1.\textwidth]{CelebA_AdvFlow_Sample.png}
		\caption*{(b)}
	\end{subfigure}
	\hspace*{-0.46em}
	\begin{subfigure}{.16\textwidth}
		\centering
		\includegraphics[width=1.\textwidth]{CelebA_Clean.png}
		\caption*{(c)}
	\end{subfigure}
	\hspace*{-0.46em}
	\begin{subfigure}{.16\textwidth}
		\centering
		\includegraphics[width=1.\textwidth]{CelebA_NATTACK_Sample.png}
		\caption*{(d)}
	\end{subfigure}
	\hspace*{-0.46em}
	\begin{subfigure}{.16\textwidth}
		\centering
		\includegraphics[width=1.\textwidth]{CelebA_NATTACK_Pert.png}
		\caption*{(e)}
	\end{subfigure}
	\caption{Adversarial perturbations generated by AdvFlow take the structure of the original image into account, resulting in less detectable adversaries compared to $\mathcal{N}$\textsc{Attack}~\citep{li2019nattack} (see Section~\ref{sec:sec:detectability}).
	The classifier is a VGG19~\citep{simonyan2015vgg} trained to detect smiles in CelebA~\citep{liu2015deep} faces.
    (a) AdvFlow magnified difference (b) AdvFlow adversarial example (c) clean image (d) $\mathcal{N}$\textsc{Attack} adversarial example (e) $\mathcal{N}$\textsc{Attack} magnified difference.}
	\label{fig:celeba_example}
\end{figure}

\section{Related Work}\label{sec:related_works}

In this section, we review some of the most \textcolor{edit}{closely} related work to our proposed approach.
For a complete review of (black-box) adversarial attacks, we refer the interested reader to~\citep{yuan2019adversarial, bhambri2019study}.

\paragraph{Black-box Adversarial Attacks.}
In one of the earliest black-box approaches, \citet{chen2017zoo} used the idea of Zeroth Order Optimization and came up with a method called \textit{ZOO}.
In particular, ZOO uses the target neural network queries to build up a zero-order gradient estimator.
Then, it utilizes the estimated gradient to minimize a \textcolor{edit}{Carlini and Wagner} (C\&W) loss~\citep{carlini2017towards} and find an adversarial image.
Later and inspired by~\citep{wierstra2014natural, salimans2017evolution}, \citet{ilyas2018black} tried to estimate the DNN gradient using a normally distributed search density.
In particular, they estimate the gradient of the classifier ${\mathcal{C}(\mathbf{x})}$ with
\begin{equation}\label{eq:QL}\nonumber
\nabla_{\mathbf{x}}\mathcal{C}(\mathbf{x})\approx\mathbb{E}_{\mathcal{N}(\mathbf{z}|\mathbf{x}, \sigma^2 I)}\left[\mathcal{C}(\mathbf{z}) \nabla_{\mathbf{x}}\log\big(\mathcal{N}(\mathbf{z}|\mathbf{x}, \sigma^2 I)\big)\right],
\end{equation}
which only requires querying the black-box model ${\mathcal{C}(\mathbf{x})}$.
Having the DNN gradient estimate, \citet{ilyas2018black} then take a \textit{projected gradient descent (PGD)} step to minimize their objective for generating an adversarial example.
This idea is further developed in the construction of \textit{$\mathcal{N}$\textsc{Attack}}~\citep{li2019nattack}.
Specifically, instead of trying to minimize the adversarial example generation objective directly, they aim to fit a distribution around the clean data so that its realizations are likely to be adversarial (\textcolor{edit}{see} Section~\ref{sec:sec:NES_NATTACK} for more details).
In another piece of work, \citet{ilyas2019prior} observe that the gradients used in adversarial example generation by PGD exhibit a high correlation both in time and across data.
Thus, the number of queries to attack a black-box model can be reduced if one incorporates this prior knowledge about the gradients.
To this end, \citet{ilyas2019prior} uses a bandit-optimization technique to integrate these priors into their attack, resulting in a method called \textit{Bandits \& Priors}.
Finally, \textit{Simple Black-box Attack}~(SimBA)~\citep{guo2019simba} is a straightforward, intuitive approach to construct black-box adversarial examples.
It is first argued that for any particular direction $\mathbf{q}$ and step size~$\epsilon>0$, either $\mathbf{x} - \epsilon \mathbf{q}$ or $\mathbf{x} + \epsilon \mathbf{q}$ is going to decrease the probability of detecting the correct class label of the input image $\mathbf{x}$.
Thus, we are likely to find an adversary by iteratively taking such steps.
The vectors $\mathbf{q}$ are selected from a set of orthonormal candidate vectors $Q$.
\citet{guo2019simba} use Discrete Cosine Transform~(DCT) to construct such a set, exploiting the observation that ``random noise in low-frequency space is more likely to be adversarial''~\citep{guo2019lowfreq}. 


\paragraph{Adversarial Attacks using Generative Models.}

There has been some prior work that utilizes the power of generative models (mostly generative adversarial networks~(GAN)) to model adversarial perturbations and attack DNNs~\citep{baluja2018learning, xiao2018generating, wang2019adirect, huang2019black}.
The target of these models is mainly white-box attacks.
They require training of their parameters to produce adversarial perturbations using a cost function that involves taking the gradient of a target network.
To adapt themselves to black-box settings, they try to replace this target network with either a distilled version of it~\citep{xiao2018generating}, or a substitute source model~\cite{huang2019black}.
However, as we will see in Section~\ref{sec:proposed_method}, the flow-based part of our model is only pre-trained on some clean training data using the maximum likelihood objective of Eq.~\eqref{eq:ML}.
Thus, AdvFlow can be adapted to any target classifier of the same dataset, without the need to train it again.
Moreover, while prior work is mainly concerned with generating the adversarial perturbations (for example~\citep{tu2019autozoom}), here we use the normalizing flows output as the adversarial example directly.
In this sense, our work is more similar to~\cite{song2018constructing} that generates unrestricted adversarial examples using GANs in a white-box setting, and falls under functional adversarial attacks~\citep{laidlaw2019functional}.
However, besides being black-box, in AdvFlow we restrict the output to be in the vicinity of the original image.

\section{Proposed Method}\label{sec:proposed_method}

In this section, we propose our attack method.
First, we define the problem of black-box adversarial attacks formally.
\textcolor{edit}{Next, we go over normalizing flows and see how we can train a flow-based model.}
Then, we review the idea of Natural Evolution Strategies~(NES)~\citep{wierstra2008nes, wierstra2014natural} and $\mathcal{N}$\textsc{Attack}~\citep{li2019nattack}.
Afterward, we show how normalizing flows can be mixed with NES in the context of black-box adversarial attacks, resulting in a method we call AdvFlow.
Finally, we prove a lemma about the nature of the perturbations generated by the proposed approach and show that $\mathcal{N}$\textsc{Attack} cannot produce the adversarial perturbations generated by AdvFlow.
Our results in Section \ref{sec:experimental_results} support this lemma.
There, we see that AdvFlow can generate adversarial examples that are less detectable than the ones generated by $\mathcal{N}$\textsc{Attack}~\citep{li2019nattack} due to its perturbation structure.

\subsection{Problem Statement}\label{sec:sec:problem_statement}

Let ${\mathcal{C}(\cdot): \mathcal{X}^{d} \rightarrow \mathcal{P}^{k}}$ denote a DNN classifier.
Assume that the classifier \textcolor{edit}{takes} a ${d}$-dimensional \textcolor{edit}{input} ${\mathbf{x} \in \mathcal{X}^{d}}$, and outputs a vector ${\mathbf{p} \in \mathcal{P}^{k}}$.
Each element of the vector ${\mathbf{p}}$ indicates the probability of the input belonging to one of the ${k}$ classes that the classifier is trying to distinguish.
Furthermore, \textcolor{edit}{let} ${y}$ denote the correct class label of the data.
In other words, if the ${y}$-th element of the classifier output ${\mathbf{p}}$ is larger than the rest, then the input has been correctly classified.
Finally, let the well-known Carlini and Wagner (C\&W) loss~\citep{carlini2017towards} be defined as\footnote{
	Note that although we are defining our objective function $\mathcal{L}(\mathbf{x}')$ for un-targeted adversarial attacks, it can be easily modified to targeted attacks. To this end, it suffices to replace ${\max_{c \neq y} \log \mathcal{C}(\mathbf{x}')_{c}}$ in Eq. \eqref{eq:CW_loss} with ${\log \mathcal{C}(\mathbf{x}')_{t}}$, where ${t}$ shows the target class output. In this paper, we only consider un-targeted attacks.}
\begin{equation}\label{eq:CW_loss}
	\mathcal{L}(\mathbf{x}')=\max\big(0, \log \mathcal{C}(\mathbf{x}')_{y} - \max_{c \neq y} \log \mathcal{C}(\mathbf{x}')_{c}\big),	
\end{equation}  
where ${\mathcal{C}(\mathbf{x}')_{y}}$ indicates the ${y}$-th element of the classifier output.
In \textcolor{edit}{the} C\&W objective, we always have ${\mathcal{L}(\mathbf{x}') \geq 0}$.
The minimum occurs when ${\mathcal{C}(\mathbf{x}')_{y} \leq \max_{c \neq y} \mathcal{C}(\mathbf{x}')_{c}}$, which is an indication that our classifier has been fooled.
Thus, finding an adversarial example for the input data ${\mathbf{x}}$ can be written as~\citep{li2019nattack}:
\begin{equation}\label{eq:adv_example}
{\mathbf{x}}_{adv}=\argmin_{\mathbf{x}' \in \mathcal{S}(\mathbf{x})} \mathcal{L}(\mathbf{x}').
\end{equation}
Here, ${\mathcal{S}(\mathbf{x})}$ denotes a set that contains similar data to ${\mathbf{x}}$ in an appropriate manner.
For example, it is common to define
\begin{equation}\label{eq:set}
	\mathcal{S}(\mathbf{x}) = \left\{\mathbf{x}' \in \mathcal{X}^{d}~\big\rvert~\norm{\mathbf{x}' - \mathbf{x}}_{p}\leq \epsilon_{\max} \right\}
\end{equation}
for image data.
In this paper, we define ${\mathcal{S}(\mathbf{x})}$ as in Eq.~\eqref{eq:set} since we deal with the application of our attack on images.

\subsection{\textcolor{edit}{Flow-based Modeling}}\label{sec:background}


\paragraph{Normalizing Flows.}\label{sec:sec:normalizing_flows}

Normalizing flows~(NF)~\citep{tabak2013family, dinh2015nice, rezende2015variational} are a family of generative models that aim at modeling the probability distribution of a given dataset.
To this end, they make use of the well-known \textit{change of variables} formula.
In particular, let ${\mathbf{Z} \in \mathbb{R}^{d}}$ denote a random vector with a straightforward, known distribution such as uniform or standard normal.
The \textit{change of variables} formula states that if we apply an invertible and differentiable transformation ${\mathbf{f}\left(\cdot\right): \mathbb{R}^{d} \rightarrow \mathbb{R}^{d}}$ on ${\mathbf{Z}}$ to obtain a new random vector ${\mathbf{X} \in \mathbb{R}^{d}}$, the relationship between their corresponding distributions can be written as:
\begin{equation}\label{eq:change_of_variable}
p(\mathbf{x}) = p(\mathbf{z})\left|\mathrm{det}\Big(\dfrac{\partial \mathbf{f}}{\partial \mathbf{z}}\Big)\right|^{-1}.
\end{equation} 
Here, ${p(\mathbf{x})}$ and ${p(\mathbf{z})}$ denote the probability distributions of ${\mathbf{X}}$ and ${\mathbf{Z}}$, respectively.
Moreover, the multiplicative term on the right-hand side is called the absolute value of the Jacobian determinant.
This term accounts for the changes in the volume of ${\mathbf{Z}}$ due to applying the transformation ${\mathbf{f}(\cdot)}$.
Flow-based methods model the transformation ${\mathbf{f}(\cdot)}$ using stacked layers of invertible neural networks~(INN).
They then apply this transformation on a \textit{base random vector} ${\mathbf{Z}}$ to model the data density.
In this paper, we assume that the base random vector has a standard normal distribution.

\paragraph{Maximum Likelihood Estimation.}

To fit the parameters of INNs to the i.i.d.~data observations ${\mathbf{x}_1,\mathbf{x}_2, \dots, \mathbf{x}_n}$, NFs use the following maximum likelihood objective~\citep{rezende2015variational}:
\begin{equation}\label{eq:ML}
{\boldsymbol{\theta}}^{*}=\argmax_{\boldsymbol{\theta}} \dfrac{1}{n} \sum_{i=1}^{n} \log p_{\boldsymbol{\theta}}\left(\mathbf{x}_i\right).
\end{equation}
Here, ${\boldsymbol{\theta}}$ denotes the parameter set of the model and $p_{\boldsymbol{\theta}}$ is the density defined in Eq.~\eqref{eq:change_of_variable}. 
Note that INNs should be modeled such that they allow for efficient computation of their Jacobian determinant.
Otherwise, this issue can impose a severe hindrance in the application of NFs to high-dimensional data given the cubic complexity of determinant computation.
For a more detailed review of normalizing flows, we refer the interested reader to~\citep{papamakarios2019normalizing, kobyzev2019nfs} and the references within.

\paragraph{Training the Flow-based Models.}

We assume that we have access to some training data \textcolor{edit}{of the same domain} to pre-train our flow-based model.
However, at the time of adversarial example generation, we use unseen test data.
Note that while in our experiments we use the same training data as the classifier itself, we are not obliged to do so.
We observed that our results remain almost the same even if we separate the flow-based model training data from what the classifier is trained on.
We argue that using the same training data is valid since our flow-based models do not extract discriminative features, and they are only trained on clean data.
This is in contrast to other generative approaches used for adversarial example generation~\citep{baluja2018learning, xiao2018generating, wang2019adirect, huang2019black}.
\textcolor{edit2}{In Appendix~\ref{sec:ap:training_data}, we empirically show that not only is this statement accurate, but we can get almost the same performance by using similar datasets to the true one.}

\subsection{Natural Evolution Strategies and ${\mathcal{N}}$\textsc{Attack}}\label{sec:sec:NES_NATTACK}

\paragraph{Natural Evolution Strategies~(NES).}

Our goal is to solve the optimization problem of Eq.~\eqref{eq:adv_example} in a black-box setting, meaning that we only have access to the inputs and outputs of the classifier ${\mathcal{C}(\cdot)}$.
Natural Evolution Strategies~(NES) use the idea of \textit{search gradients} to optimize Eq.~\eqref{eq:adv_example}~\citep{wierstra2008nes, wierstra2014natural}.
To this end, a so-called \textit{search distribution} is first defined, and then the expected value of the original objective is optimized under this distribution.

In particular, let ${p(\mathbf{x'}|\boldsymbol{\psi})}$ denote the search distribution with parameters ${\boldsymbol{\psi}}$.
Then, in NES we aim to minimize
\begin{equation}\label{eq:nes}
	J(\boldsymbol{\psi})=\mathbb{E}_{p(\mathbf{x'}|\boldsymbol{\psi})}\left[\mathcal{L}(\mathbf{x}')\right]
\end{equation}  
over ${\boldsymbol{\psi}}$ as a surrogate for ${\mathcal{L}(\mathbf{x}')}$~\citep{wierstra2014natural}.
To minimize Eq.~\eqref{eq:nes} using gradient descent, one needs to compute the Jacobian of ${J(\boldsymbol{\psi})}$ with respect to ${\boldsymbol{\psi}}$.
To this end, NES makes use of the ``log-likelihood trick''~\citep{wierstra2014natural} (see Appendix~\ref{sec:ap:ll_trick})
\begin{align}\label{eq:jac_J}
	\nabla_{\boldsymbol{\psi}}J(\boldsymbol{\psi}) = \mathbb{E}_{p(\mathbf{x'}|\boldsymbol{\psi})}\left[\mathcal{L}(\mathbf{x}') \nabla_{\boldsymbol{\psi}}\log\big(p(\mathbf{x'}|\boldsymbol{\psi})\big)\right].
\end{align}
Finally, the parameters of the model are updated using a gradient descent step with learning rate ${\alpha}$:\footnote{Note that this update procedure is not what NES precisely stands for.
	It is rather a \textit{canonical gradient search algorithm} as is called by~\citet{wierstra2014natural}, which only makes use of a \textit{vanilla} gradient~\citep{wierstra2008nes} for evolution strategies.
	In fact, the \textit{natural} term in \textit{natural evolution strategies} represents an update of the form ${\boldsymbol{\psi} \leftarrow \boldsymbol{\psi} - \alpha \tilde{\nabla}_{\boldsymbol{\psi}}J(\boldsymbol{\psi})}$, where  ${\tilde{\nabla}_{\boldsymbol{\psi}}J(\boldsymbol{\psi}) = \mathbf{F}^{-1}\nabla_{\boldsymbol{\psi}}J(\boldsymbol{\psi})}$ is called the \textit{natural gradient}.
	Here, the matrix ${\mathbf{F}}$ is the \textit{Fischer information matrix} of the search distribution ${p(\mathbf{x'}|\boldsymbol{\psi})}$.
	However, since NES in the adversarial learning literature~\citep{ilyas2018black, ilyas2019prior, li2019nattack} points to Eq.~\eqref{eq:nes_update}, we use the same convention here.} 
\begin{equation}\label{eq:nes_update}
	\boldsymbol{\psi} \leftarrow \boldsymbol{\psi} - \alpha \nabla_{\boldsymbol{\psi}}J(\boldsymbol{\psi}).
\end{equation}

\paragraph{${\mathcal{N}}$\textsc{Attack}.}

To find an adversarial example for an input ${\mathbf{x}}$, $\mathcal{N}$\textsc{Attack}~\citep{li2019nattack} tries to find a distribution ${p(\mathbf{x'}|\boldsymbol{\psi})}$ over the set of legitimate adversaries ${\mathcal{S}(\mathbf{x})}$ in Eq.~\eqref{eq:set}.
Therefore, it models $\mathbf{x}'$ as
\begin{equation}\label{eq:NATTACK}
	\mathbf{x}'=\mathrm{proj}_{\mathcal{S}}\big(\tfrac{1}{2}(\tanh(\mathbf{z}) + 1)\big),
\end{equation}
where $\mathbf{z} \sim \mathcal{N}(\mathbf{z}|\boldsymbol{\mu}, \sigma^2 I)$ is an isometric normal distribution with mean~$\boldsymbol{\mu}$ and standard deviation~$\sigma$.
Moreover, $\mathrm{proj}_{\mathcal{S}}(\cdot)$ projects its input back into the set of legitimate adversaries ${\mathcal{S}(\mathbf{x})}$.
\citet{li2019nattack} define their model parameters as ${\boldsymbol{\psi}=\{\boldsymbol{\mu}, \sigma\}}$. 
Then, they find ${\sigma}$ using grid-search and ${\boldsymbol{\mu}}$ by the update rule of Eq.~\eqref{eq:nes_update} exploiting NES.

\subsection{\textcolor{edit}{Our Approach:} AdvFlow}\label{sec:sec:AdvFlow}

Recently, there has been some effort to detect adversarial examples from clean data.
The primary assumption of these methods is often that the adversaries come from a different distribution than the data itself; for instance, see \citep{ma2018characterizing, lee2018simple, zisselman2020resflow}.
Thus, to come up with more powerful adversarial attacks, it seems reasonable to construct adversaries that have a similar distribution to the clean data.
To this end, we propose \textit{AdvFlow}: a black-box adversarial attack that seeks to build inconspicuous adversaries by leveraging the power of normalizing flows~(NF) in exact likelihood modeling of the data~\citep{dinh2016density}.

Let ${\mathbf{f}(\cdot)}$ denote a pre-trained, invertible and differentiable NF model on the clean training data.
To reach our goal of decreasing the attack's detectability, we propose using this pre-trained, fixed NF transformation to model the adversaries.
In an analogy with Eq.~\eqref{eq:NATTACK}, we assume that our adversarial example comes from a distribution that is modeled by
\begin{equation}\label{eq:advflow_dist}
	\mathbf{x}'=\mathrm{proj}_{\mathcal{S}}\big(\mathbf{f}(\mathbf{z})\big),\qquad\mathbf{z}\sim\mathcal{N}(\mathbf{z}|\boldsymbol{\mu}, \sigma^2 I)
\end{equation}
where $\mathrm{proj}_{\mathcal{S}}(\cdot)$ is a projection rule that keeps the generated examples in the set of legitimate adversaries ${\mathcal{S}(\mathbf{x})}$.
By the change of variables formula from Eq.~\eqref{eq:change_of_variable}, we know that ${\mathbf{f}(\mathbf{z})}$ in Eq.~\eqref{eq:advflow_dist} is distributed similar to the clean data distribution.
The only difference is that the base density is transformed by an affine mapping, i.e., from $\mathcal{N}(\mathbf{z}|\mathbf{0}, I)$ to $\mathcal{N}(\mathbf{z}|\boldsymbol{\mu}, \sigma^2 I)$.
This small adjustment can result in an overall distribution for which the generated samples are likely to be adversarial.

Putting the rule of the \textit{lazy statistician}~\citep{wasserman2013all} together with our attack definition in Eq.~\eqref{eq:advflow_dist}, we can write down the objective function of Eq.~\eqref{eq:nes} as
\begin{equation}\label{eq:advflow_cost}
	J(\boldsymbol{\mu}, \sigma)=\mathbb{E}_{p(\mathbf{x'}|\boldsymbol{\mu}, \sigma)}\left[\mathcal{L}(\mathbf{x}')\right]
					    =\mathbb{E}_{\mathcal{N}(\mathbf{z}|\boldsymbol{\mu}, \sigma^2 I)}\left[\mathcal{L}\bigg(\mathrm{proj}_{\mathcal{S}}\big(\mathbf{f}(\mathbf{z})\big)\bigg)\right].
\end{equation}
As in $\mathcal{N}$\textsc{Attack}~\citep{li2019nattack}, we will consider ${\sigma}$ to be a hyperparameter.\footnote{Indeed, we can also optimize $\sigma$ alongside ${\boldsymbol{\mu}}$ to enhance the attack strength. However, since $\mathcal{N}$\textsc{Attack}~\citep{li2019nattack} only optimizes ${\boldsymbol{\mu}}$, we also stick with the same setting.}
Thus, we are only required to minimize ${J(\boldsymbol{\mu}, \sigma)}$ with respect to ${\boldsymbol{\mu}}$.
Using the ``log-likelihood trick'' of Eq.~\eqref{eq:jac_J}, we can derive the Jacobian of ${J(\boldsymbol{\mu}, \sigma)}$ as
\begin{equation}\label{eq:jac_advflow}
	\nabla_{\boldsymbol{\mu}}J(\boldsymbol{\mu}, \sigma) = \mathbb{E}_{\mathcal{N}(\mathbf{z}|\boldsymbol{\mu}, \sigma^2 I)}\left[\mathcal{L}\bigg(\mathrm{proj}_{\mathcal{S}}\big(\mathbf{f}(\mathbf{z})\big)\bigg)	\nabla_{\boldsymbol{\mu}}\log \mathcal{N}(\mathbf{z}|\boldsymbol{\mu}, \sigma^2 I)\right].
\end{equation}
This expectation can then be estimated by sampling from a distribution ${\mathcal{N}(\mathbf{z}|\boldsymbol{\mu}, \sigma^2 I)}$ and forming their sample average.
Next, we update the parameter ${\boldsymbol{\mu}}$ by performing a gradient descent step
\begin{equation}\label{eq:advflow_update}
\boldsymbol{\mu} \leftarrow \boldsymbol{\mu} - \alpha \nabla_{\boldsymbol{\mu}}J(\boldsymbol{\mu}, \sigma).
\end{equation}
In the end, we generate our adversarial example by sampling from Eq.~\eqref{eq:advflow_dist}.

\paragraph{Practical Considerations.}

To help our model to start its search from an appropriate point, we first transform the clean data to its latent space representation.
Then, we aim to find a small additive latent space perturbation in the form of a normal distribution.
Moreover, as suggested in~\citep{li2019nattack}, instead of working with ${\mathcal{L}\big(\mathrm{proj}_{\mathcal{S}}\big(\mathbf{f}(\mathbf{z})\big)\big)}$ directly, we normalize them so that they have zero mean and unit variance to help AdvFlows convergence faster.
Finally, among different flow-based models, it is preferable to choose those that have a straightforward inverse, such as~\citep{dinh2016density, kingma2018glow, durkan2019neural, dolatabadi2020lrs}.
This way, we can efficiently go back and forth between the original data and their base distribution representation. 
Algorithm~\ref{alg:advflow} in Appendix~\ref{sec:ap:advflow} summarizes our black-box attack method.
\textcolor{edit2}{Other variations of AdvFlow can also be found in Appendix~\ref{sec:ap:algs}.
These variations include our solution to high-resolution images and investigation of un-trained AdvFlow.}


\subsubsection{AdvFlow Interpretation}

We can interpret AdvFlow from two different perspectives.

First, there exists a probabilistic view: we use the flow-based model transformation of the original data, and then try to adjust it using an affine transformation on its base distribution.
The amount of this change is determined by our urge to minimize the C\&W cost of Eq.~\eqref{eq:CW_loss} such that we get the minimum value on average.
Thus, if it is successful, we will end up having a distribution whose samples are likely to be adversarial.
Meanwhile, since this distribution is initialized with that of clean data, it resembles the clean data density closely.

Second, we can think of AdvFlow as a search over the latent space of flow-based models.
We map the clean image to the latent space and then try to search in the vicinity of that point to find an adversarial example.
This search is ruled by the objective of Eq.~\eqref{eq:advflow_cost}.
Since our approach exploits a fully invertible, pre-trained flow-based model, we would expect to get an adversarial example that resembles the original image in the structure and look less noisy.
This adjustment gives our model the flexibility to produce perturbations that take the structure of clean data into account (see Figure~\ref{fig:celeba_example}).

\subsubsection{Uniqueness of AdvFlow Perturbations}\label{sec:sec:advflow_lemma}

In this section, we present a lemma about the nature of perturbations generated by AdvFlow and $\mathcal{N}$\textsc{Attack}~\citep{li2019nattack}.
As a direct result of this lemma, we can easily deduce that the adversaries generated by AdvFlow can be approximated by a normal distribution whose components are dependent.
However, this is not the case for $\mathcal{N}$\textsc{Attack} as they always have independent elements.
In this sense, we can then rigorously conclude that the AdvFlow perturbations are unique and cannot be generated by $\mathcal{N}$\textsc{Attack}.
Thus, we cannot expect $\mathcal{N}$\textsc{Attack} to be able to generate perturbations that look like the original data. 
This result can also be generalized to many other attack methods as they often use an additive, independent perturbation. Proofs can be found in Appendix~\ref{sec:ap:proofs}.

\begin{lemma}\label{lemma}
	Let ${\mathbf{f}(\mathbf{x})}$ be an invertible, differentiable function.
	For a small perturbation ${\boldsymbol{\delta}_z}$ we have
	\begin{equation}\label{eq:lemma}\nonumber
		\boldsymbol{\delta} = \mathbf{f}\big(\mathbf{f}^{-1}(\mathbf{x}) + \boldsymbol{\delta}_z\big) - \mathbf{x}\approx\big(\nabla\mathbf{f}^{-1}(\mathbf{x})\big)^{-1}\boldsymbol{\delta}_z.
	\end{equation}
\end{lemma}

\begin{corollary}\label{corollary}
	The adversarial perturbations generated by AdvFlow have dependent components.
	In contrast, ${\mathcal{N}}$\textsc{Attack} perturbation components are independent.
\end{corollary}
 
\section{Experimental Results}\label{sec:experimental_results}

In this section, we present our experimental results.
First, we see how the adversarial examples generated by the proposed model can successfully mislead adversarial example detectors.
Then, we show the attack success rate and the number of queries required to attack both vanilla and defended models.
Finally, we examine the transferability of the generated attacks between defended classifiers.
To see the details of the experiments, please refer to Appendix~\ref{sec:ap:imp_det}.
Also, more simulation results can be found in Appendices~\ref{sec:ap:ext_sim_res} and~\ref{sec:ap:algs}.

For each dataset, we pre-train a flow-based model and fix it across the experiments.
To this end, we use a modified version of Real~NVP~\citep{dinh2016density} as introduced in~\citep{ardizzone2019guided}, the details of which can be found in Appendix~\ref{sec:ap:nf_arch}.
Once trained, we then try to attack target classifiers in a black-box setting using AdvFlow~(Algorithm~\ref{alg:advflow}).

\subsection{Detectability}\label{sec:sec:detectability}

One approach to defend pre-trained classifiers is to employ adversarial example detectors.
This way, a detector is trained and put on top of the classifier.
Before feeding inputs to the un-defended classifier, every input has to be checked whether it is adversarial or not.
One common assumption among such detectors is that the adversaries come from a different distribution than the clean data~\citep{ma2018characterizing, lee2018simple}.
Thus, the performance of these detectors seems to be a suitable measure to quantify the success of our model in generating adversarial examples that have the same distribution as the original data.
To this end, we choose LID~\citep{ma2018characterizing}, Mahalanobis~\citep{lee2018simple}, and Res-Flow~\cite{zisselman2020resflow} adversarial attack detectors to assess the performance of the proposed approach. We compare our results with $\mathcal{N}$\textsc{Attack}~\citep{li2019nattack} \textit{that also approaches the black-box adversarial attack from a distributional perspective} for a fair comparison.
As an ablation study, we also consider the un-trained version of AdvFlows where the weights of the NF models are set randomly.
This way, we can observe the effect of the clean data distribution in misleading adversarial example detectors more precisely. 
We first generate a set of adversarial examples alongside some noisy ones using the test set.
Then, we use $10\%$ of the adversarial, noisy, and clean image data to train adversarial attack detectors.
Details of our experiments in this section can be found in Appendix~\ref{sec:ap:mah_det}.

We report the area under the receiver operating characteristic curve~(AUROC) and the detection accuracy for each case in Table~\ref{tab:Detectability}.
As seen, in almost all the cases the selected adversarial detectors struggle to detect the attacks generated by AdvFlow in contrast to $\mathcal{N}$\textsc{Attack}.
These results support our statement earlier about the distribution of the attacks being more similar to that of data, hence the failure of adversarial example detectors.
Also, we see that pre-training the AdvFlow using clean data is crucial in fooling adversarial example detectors.   

Finally, Figure~\ref{fig:latent_separation} shows the relative change in the base distribution of the flow-based model for adversarial examples of Table~\ref{tab:Detectability}.
Interestingly, we see that AdvFlow adversaries are distinctively closer to the clean data compared to $\mathcal{N}$\textsc{Attack}~\cite{li2019nattack}.
These results highlight the need to reconsider the underlying assumption that adversaries come from a different distribution than the clean data.
Also, it can motivate training classifiers that learn data distributions, as our results reveal this is not currently the case.
\begin{table*}[tb!]
	\caption{Area under the receiver operating characteristic curve~(AUROC) and accuracy of detecting adversarial examples generated by $\mathcal{N}$\textsc{Attack}~\citep{li2019nattack} and AdvFlow (un. for un-trained and tr. for pre-trained NF) using LID~\citep{ma2018characterizing}, Mahalanobis~\citep{lee2018simple}, and Res-Flow~\cite{zisselman2020resflow} adversarial attack detectors.
	In each case, the classifier has a ResNet-34~\cite{he2016deep} architecture.}
	\label{tab:Detectability}
	
	\begin{center}
		\begin{small}
			\resizebox{\columnwidth}{!}{
				\begin{tabular}{cccccccc}
					\toprule
					\parbox[t]{2mm}{\multirow{2}{*}{\rotatebox[origin=c]{90}{Data}}}
					&Metric                                      & \multicolumn{3}{c}{AUROC(\%) \textuparrow}   & \multicolumn{3}{c}{Detection Acc.(\%) \textuparrow}\\%   & AUPR in               & AUPR out\\
					\cmidrule(lr){3-5}\cmidrule(lr){6-8}
					&Method                                      & $\mathcal{N}$\textsc{Attack}    & AdvFlow (un.)  & AdvFlow (tr.)   & $\mathcal{N}$\textsc{Attack}    & AdvFlow (un.)  & AdvFlow (tr.)\\
					\midrule
					\parbox[t]{2mm}{\multirow{3}{*}{\rotatebox[origin=c]{90}{\scriptsize{CIFAR-10}}}}
					&LID~\citep{ma2018characterizing}            & $78.69$ & $84.39$ & $\mathbf{57.59}$    & $72.12$ & $77.11$ & $\mathbf{55.74}$\\%     & $86.25/90.31/\mathbf{72.53}$      & $64.99/73.34/\mathbf{38.61}$\\
					&Mahalanobis~\citep{lee2018simple}           & $97.95$ & $99.50$ & $\mathbf{66.85}$    & $95.59$ & $97.46$ & $\mathbf{62.21}$\\%     & $98.34/99.70/\mathbf{77.14}$      & $97.39/99.18/\mathbf{54.00}$\\
					&Res-Flow~\citep{zisselman2020resflow}       & $97.90$ & $99.40$ & $\mathbf{67.03}$    & $94.55$ & $97.21$ & $\mathbf{62.60}$\\%     & $98.35/99.63/\mathbf{77.38}$      & $97.30/99.03/\mathbf{54.12}$\\
					\midrule
					\parbox[t]{2mm}{\multirow{3}{*}{\rotatebox[origin=c]{90}{\footnotesize{SVHN}}}}
					&LID~\citep{ma2018characterizing}            & $\mathbf{57.70}$  & $58.92$ &$61.11$    & $\mathbf{55.60}$  & $56.43$ & $58.21$\\%     & $\mathbf{72.69}/73.60/74.51$      & $\mathbf{39.08}/40.15/42.77$\\
					&Mahalanobis~\citep{lee2018simple}           & $73.17$ & $74.67$ & $\mathbf{64.72}$    & $68.20$ & $69.46$ & $\mathbf{60.88}$\\%     & $86.09/87.13/\mathbf{77.51}$      & $52.33/53.92/\mathbf{45.30}$\\
					&Res-Flow~\citep{zisselman2020resflow}       & $69.70$ & $74.86$ & $\mathbf{64.68}$    & $64.53$ & $68.41$ & $\mathbf{61.13}$\\%     & $80.93/84.81/\mathbf{77.96}$      & $52.01/57.80/\mathbf{44.53}$\\
					\bottomrule
				\end{tabular}}
		\end{small}
	\end{center}
	
\end{table*}

\begin{figure}[tb!]
	\centering
	\begin{subfigure}{.44\textwidth}
		\centering
		\includegraphics[width=1.0\textwidth]{latent_dist_CIFAR10.png}
	\end{subfigure}\hspace*{0.2em}
	\begin{subfigure}{.44\textwidth}
		\centering
		\includegraphics[width=1.0\textwidth]{latent_dist_SVHN.png}
	\end{subfigure}
	\caption{Relative change in the base distribution of the flow-based model for adversarial examples generated by AdvFlow and $\mathcal{N}$\textsc{Attack} for CIFAR-10~\cite{krizhevsky2009learning} (left) and SVHN~\cite{netzer2011reading} (right) classifiers of Table~\ref{tab:Detectability}.}
	\label{fig:latent_separation}
\end{figure}

\subsection{Success Rate and Number of Queries}\label{sec:sec:succ_query}

Next, we investigate the performance of the proposed model in attacking vanilla and defended image classifiers.
It was shown previously that $\mathcal{N}$\textsc{Attack} struggles to break into adversarially trained models more than any other defense~\citep{li2019nattack}.
Thus, we select some of the most recent defense techniques that are built upon adversarial training~\citep{madry2018towards}.
This selection also helps us in quantifying the attack transferability in our next experiment.
Therefore, we select Free~\citep{shafahi2019free} and Fast~\citep{wong2020fast} adversarial training, alongside adversarial training with auxiliary rotations~\cite{hendrycks2019using} as the defense mechanisms that our classifiers employ.
Note that these models are the most recent defenses built upon adversarial training.
For a brief explanation of each one of these methods, please refer to Appendix~\ref{sec:ap:defense}.
We then train target classifiers on CIFAR-10~\citep{krizhevsky2009learning} and SVHN~\citep{netzer2011reading} datasets.
The architecture that we use here is the well-known Wide-ResNet-32~\cite{zagoruyko2016wresnet} with width $10$.
We then try to attack these classifiers by generating adversarial examples on the test set.
We compare our proposed model with bandits with time and data-dependent priors~\citep{ilyas2019prior}, $\mathcal{N}$\textsc{Attack}~\citep{li2019nattack}, and SimBA~\cite{guo2019simba}. \footnote{Note that SimBA~\citep{guo2019simba} is originally designed for efficient $\ell_2$ attacks, and it may not use the entire $10,000$ query quota for small images. Anyways, we included SimBA~\citep{guo2019simba} in the paper as per one of the reviewers' suggestions.}
To simulate a realistic environment, we set the maximum number of queries to $10,000$.
Moreover, for $\mathcal{N}$\textsc{Attack} and AdvFlow we use a population size of $20$.
More details on the defense methods as well as attack hyperparameters can be found in Appendices~\ref{sec:ap:defense} and~\ref{sec:ap:attack_hyper}.

Tables~\ref{tab:success} and~\ref{tab:query}  show the success rate as well as the average and median number of queries required to successfully attack a vanilla/defended classifier.
Also, Figure~\ref{fig:ap_succ_query} in Appendix~\ref{sec:ap:ext_sim_res} shows the attack success rate for AdvFlow and $\mathcal{N}$\textsc{Attack}~\citep{li2019nattack} versus the maximum number of queries for defended models.
As can be seen, AdvFlows can improve upon the performance of $\mathcal{N}$\textsc{Attack}~\citep{li2019nattack} in all of the defended models in terms of the number of queries and attack success rate.
Also, it should be noted that although our performance on vanilla classifiers is worse than $\mathcal{N}$\textsc{Attack}~\citep{li2019nattack}, we are still generating adversaries that are not easily detectable by adversarial example detectors and come from a similar distribution to the clean data.

\begin{table*}[tb!]
	\caption{Attack success rate of black-box adversarial attacks on CIFAR-10~\citep{krizhevsky2009learning} and SVHN~\citep{netzer2011reading} Wide-ResNet-32~\cite{zagoruyko2016wresnet} classifiers.
	All attacks are with respect to $\ell_{\infty}$ norm with $\epsilon_{\max}=8/255$.}
	\label{tab:success}
	\begin{center}
		\begin{small}
				\begin{tabular}{ccccccc}
					\toprule
					\parbox[t]{2mm}{\multirow{2}{*}{\rotatebox[origin=c]{90}{Data}}} &&& \multicolumn{4}{c}{Success Rate(\%) \textuparrow}\\
					\cmidrule(lr){4-7}
					&Defense                                & Acc.(\%)                & Bandits\citep{ilyas2019prior}  & $\mathcal{N}$\textsc{Attack}~\citep{li2019nattack} & SimBA~\citep{guo2019simba} & AdvFlow \\
					\midrule
					\parbox[t]{2mm}{\multirow{4}{*}{\rotatebox[origin=c]{90}{CIFAR-10}}}
					&Vanilla~\citep{zagoruyko2016wresnet}   & $91.77$                 & $98.81$	                       & $\mathbf{100}$                                     & $99.99$                    & $99.42$\\
					&FreeAdv~\citep{shafahi2019free}        & $81.29$                 & $37.12$	                       & $38.97$	                                        & $35.52$	                 & $\mathbf{41.21}$\\
					&FastAdv~\citep{wong2020fast}           & $86.33$                 & $36.60$                        & $36.90$                                           	& $35.07$	                 & $\mathbf{40.22}$\\
					&RotNetAdv~\citep{hendrycks2019using}   & $86.58$                 & $37.73$	                       & $38.04$	                                        & $35.63$	                 & $\mathbf{40.67}$\\
					\midrule
					\parbox[t]{2mm}{\multirow{4}{*}{\rotatebox[origin=c]{90}{SVHN}}}
					&Vanilla~\citep{zagoruyko2016wresnet}   & $96.45$                 & $87.84$	                       & $\mathbf{98.76}$	                                & $97.26$                    & $90.31$\\
					&FreeAdv~\citep{shafahi2019free}        & $86.47$                 & $49.64$	                       & $50.28$	                                        & $46.28$	                 & $\mathbf{50.76}$\\
					&FastAdv~\citep{wong2020fast}           & $93.90$                 & $40.43$	                       & $35.42$	                                        & $36.19$	                 & $\mathbf{41.49}$\\
					&RotNetAdv~\citep{hendrycks2019using}   & $90.33$                 & $43.47$	                       & $41.49$                                            & $39.01$	                 & $\mathbf{44.22}$\\
					\bottomrule
				\end{tabular}
		\end{small}
	\end{center}
\end{table*}

\begin{table*}[tb!]
	\caption{Average (median) of the number of queries needed to generate an adversarial example for CIFAR-10~\citep{krizhevsky2009learning} and SVHN~\citep{netzer2011reading} Wide-ResNet-32~\cite{zagoruyko2016wresnet} classifiers of Table~\ref{tab:success}.
		For a fair comparison, we first find the samples where all the attack methods are successful, and then compute the average (median) of queries for these samples. 
		Note that for $\mathcal{N}$\textsc{Attack} and AdvFlow we check whether we arrived at an adversarial point every $200$ queries, and hence, the medians are a multiples of $200$.}
	\label{tab:query}
	\begin{center}
		\begin{small}
			\begin{tabular}{cccccc}
				\toprule
				\parbox[t]{2mm}{\multirow{2}{*}{\rotatebox[origin=c]{90}{Data}}} && \multicolumn{4}{c}{Query Average (Median) on Mutually Successful Attacks \textdownarrow}\\
				\cmidrule(lr){3-6}
				&Defense                                & Bandits\citep{ilyas2019prior}    & $\mathcal{N}$\textsc{Attack}~\citep{li2019nattack} & SimBA~\citep{guo2019simba}        & AdvFlow \\
				\midrule
				\parbox[t]{2mm}{\multirow{4}{*}{\rotatebox[origin=c]{90}{CIFAR-10}}}
				&Vanilla~\citep{zagoruyko2016wresnet}   & $552.69~(182)$	               & $\mathbf{237.58}~(200)$	                        & $237.70~(\mathbf{126})$           & $949.31~(400)$\\
				&FreeAdv~\citep{shafahi2019free}        & $1062.7~(354)$	               & $874.91~(400)$                                     & $463.09~(244)$	                & $\mathbf{421.63}~(\mathbf{200})$\\
				&FastAdv~\citep{wong2020fast}           & $1065.92~(358)$                  & $973.05~(400)$                                     & $\mathbf{428.81}~(234)$           & $436.8~(\mathbf{200})$\\
				&RotNetAdv~\citep{hendrycks2019using}   & $1085.43~(408)$                  & $941.67~(400)$                                     & $471.99~(259)$                    & $\mathbf{424.95}~(\mathbf{200})$\\
				\midrule
				\parbox[t]{2mm}{\multirow{4}{*}{\rotatebox[origin=c]{90}{SVHN}}}
				&Vanilla~\citep{zagoruyko2016wresnet}   & $1750.65~(1128)$                 & $408.75~(200)$                                   	& $\mathbf{202.07}~(\mathbf{107})$  & $1572.24~(600)$\\
				&FreeAdv~\citep{shafahi2019free}        & $819.98~(250)$                   & $903.12~(400)$                                     & $\mathbf{365.42}~(216)$           & $692.73~(\mathbf{200})$\\
				&FastAdv~\citep{wong2020fast}           & $755.23~(284)$                   & $1243.38~(600)$	                                & $\mathbf{307.73}~(216)$           & $526.37~(\mathbf{200})$\\
				&RotNetAdv~\citep{hendrycks2019using}   & $663.07~(202)$                   & $756.48~(400)$	                                    & $\mathbf{319.93}~(\mathbf{186})$  & $480.02~(200)$\\
				\bottomrule
			\end{tabular}
		\end{small}
	\end{center}
\end{table*}

\subsection{Transferability}\label{sec:sec:transferability}

Finally, we examine the transferability of the generated attacks for each of the classifiers in Table~\ref{tab:success}.
In other words, we generate attacks using a substitute classifier, and then try to attack another target model.
The results of this experiment are shown in Figure~\ref{fig:ap_confusion_trans} of Appendix~\ref{sec:ap:ext_sim_res}.
As seen, the generated attacks by AdvFlow transfer to other defended models easier than the vanilla one.
This observation precisely matches our intuition about the mechanics of AdvFlow.
More specifically, we know that in AdvFlow the model is learning a distribution that is more expressive than the one used by $\mathcal{N}$\textsc{Attack}.
Also, we have seen in Section~\ref{sec:sec:advflow_lemma} that the perturbations generated by AdvFlow have dependent elements in contrast to $\mathcal{N}$\textsc{Attack}.
As a result, AdvFlow learns to attack classifiers using higher-level features (Figure~\ref{fig:celeba_example}).  
Thus, since vanilla classifiers use different features for classification than the defended ones, AdvFlows are less transferable from defended models to vanilla ones.
In contrast, the expressiveness of AdvFlows enables the attacks to be transferred more successfully between adversarially trained classifiers, and from vanilla to defended ones.
  
\section{Conclusion and Future Directions}\label{sec:conclusion}

In this paper, we introduced AdvFlow: a novel adversarial attack model that utilizes the capacity of normalizing flows in representing data distributions.
We saw that the adversarial perturbations generated by the proposed approach can be approximated using normal distributions with dependent components.
In this sense, $\mathcal{N}$\textsc{Attack}~\citep{li2019nattack} cannot generate such adversaries.
As a result, AdvFlows are less conspicuous to adversarial example detectors in contrast to their $\mathcal{N}$\textsc{Attack}~\citep{li2019nattack} counterpart.
This success is due to AdvFlow being pre-trained on the data distribution, resulting in adversaries that look like the clean data.
We also saw the capability of the proposed method in improving the performance of bandits~\cite{ilyas2019prior}, $\mathcal{N}$\textsc{Attack}~\citep{li2019nattack}, and SimBA~\cite{guo2019simba} on adversarially trained classifiers.
This improvement is in terms of both attack success rate and the number of queries.

Flow-based modeling is an active area of research.
There are numerous extensions to the current work that can be investigated upon successful expansion of normalizing flow models in their range and power.
For example, while $\mathcal{N}$\textsc{Attack}~\citep{li2019nattack} and other similar approaches~\citep{ilyas2018black, ilyas2019prior} are specifically designed for use on image data, the current work can potentially be expanded to entail other forms of data such as graphs~\citep{liu2019gnf, shi2020graphaf}.
Also, since normalizing flows can effectively model probability distributions, finding the distribution of well-known perturbations may lead to increasing classifier robustness against adversarial examples.
We hope that this work can provide a stepping stone to exploiting such powerful models for adversarial machine learning.

\clearpage  
\section*{Broader Impact}
In this paper, we introduce a novel adversarial attack algorithm called AdvFlow.
It uses pre-trained normalizing flows to generate adversarial examples.
This study is crucial as it indicates the vulnerability of deep neural network~(DNN) classifiers to adversarial attacks.

More precisely, our study reveals that the common assumption made by adversarial example detectors (such as the Mahalanobis detector~\citep{lee2018simple}) that the adversaries come from a different distribution than the data may not be an accurate one.
In particular, we show that we can generate adversaries that come from a close distribution to the data, yet they intend to mislead the classifier decision.
Thus, we emphasize that adversarial example detectors need to adjust their assumption about the distribution of adversaries before being deployed in real-world situations.

Furthermore, since our adversarial examples are closely related to the data distribution, our method shows that DNN classifiers are not learning to classify the data based on their underlying distribution.
Otherwise, they would have resisted the attacks generated by AdvFlow.
Thus, it can bring the attention of the machine learning community to training their DNN classifiers in a distributional sense.

All in all, we pinpoint a failure of DNN classifiers to the rest of the community so that they can become familiar with the limitations of the status-quo.
This study, and similar ones, could raise awareness among researchers about the real-world pitfalls of DNN classifiers, with the aim of consolidating them against such threats in the future.


\begin{ack}
We would like to thank the reviewers for their valuable feedback on our work, helping us to improve the final manuscript.
We also would like to thank the authors and maintainers of PyTorch~\citep{paszke2017automatic}, NumPy~\cite{harris2020array}, and Matplotlib~\cite{hunter2007matplotlib}.

This research was undertaken using the LIEF HPC-GPGPU Facility hosted at the University of Melbourne. This Facility was established with the assistance of LIEF Grant LE170100200.
\end{ack}

\bibliographystyle{apalike}
