\section{Introduction}
\label{Intro}

Spiking neural networks~(SNNs) are based on the spiking neural behavior in biological neurons~\cite{hodgkin1952quantitative,izhikevich2003simple}. Each neuron in SNNs elicits a spike when its accumulated membrane potential exceeds a threshold, otherwise, it would stay inactive in the current time step. Compared with ANNs, the activation values in SNNs are binarized (i.e., neuromorphic computing~\cite{roy2019nature}), thus resulting in an advantage of energy efficiency for SNNs. Existing works reveal that on specialized hardware, SNNs can save energy by orders of magnitude compared with ANNs ~\cite{roy2019scaling,DENG2020294}. Another vital attribute of SNN is its ability to make inferences in a spatial-temporal paradigm. Specifically, the forwarding pass in SNN is repeated for $T$ steps to get the final result, where the final result is the expectation of the ultimate layer's output across $T$ steps. This allows the flexibility of adjusting $T$ to balance between the latency and accuracy of SNNs for different application scenarios.  

\begin{table}[t]
\footnotesize
\centering
\begin{adjustbox}{max width=0.48\textwidth}
\begin{tabular}{lccc}
\toprule
\textbf{Features} & \textbf{Training} & \textbf{Conversion} & \textbf{Calibration} \\
\midrule
Accuracy($T\le100$) & High &  Low  & High \\
Scalability & Tiny & Large & Large \\
Training Speed & Slow & Fast & Fast \\
\# Required Data & Full-set & 1000 & 1000 \\
Inference Speed & Fast & Slow & Fast \\
\bottomrule
\end{tabular}
\end{adjustbox}
\caption{Features comparison between SNN direct training, ANN-SNN conversion and our SNN calibration.} \label{tab_compare}
\end{table}

Conventionally, there are two distinct routes to obtain a functional SNN: (1) training SNN from scratch \cite{shrestha2018slayer,kheradpisheh2018stdp}, and (2) converting a pretrained ANN to SNN \cite{cao2015spiking,Diehl2015Fast}. For training from scratch, it is hard to adopt gradient-based optimization methods because of the non-differentiability of the binary activation function in SNNs \cite{neftci2019surrogate}. 
Although several approaches like surrogate gradients~\cite{wu2018spatio,shrestha2018slayer} and synaptic plasticity~\cite{kheradpisheh2018stdp} are proposed to mitigate this problem, training SNN from scratch still lacks the scalability to obtain an effective SNN on the ImageNet dataset. Another notable problem is the tremendous resources required to complete the training process. The binary acceleration cannot be employed in GPU training since no CUDA instructions support this kind of computation. As a result, training an SNN may require $T\times$ more time than ANN training. 

Besides directly training SNNs from scratch, another family of approaches is converting a pretrained ANN into SNN ~\cite{diehl2016conversion,rueckauer2017conversion,Sengupta2018Going}. 
The conversion process demands less computation and memory than training from scratch. Although some progress in SNN conversion is made, such as threshold balancing~\cite{Diehl2015Fast,Sengupta2018Going}, weight normalization~\cite{rueckauer2017conversion}, and soft-reset mechanism~\cite{rueckauer2017conversion, han2020rmp}, all of them fail to convert ANN with BN layers in low latency time steps ($\le 256$), which may significantly increase the latency especially for resource-limited devices. We think the simple \textit{copy-paste} of parameters without any dedicated calibration on SNN will inevitably result in activation mismatch. 


In this work, we aim to obtain an SNN in extremely low latency (less than 256 time steps) and in extremely low cost. We choose to utilize a pre-trained ANN and convert it to SNN. Unlike previous conversion work which simply \textbf{transplants the weights parameters} to the SNN, in this work we show that the \textbf{activation transplating} is much more important. In order to accomplish this, we propose SNN calibration, a new technology family by calibrating the parameters in SNN to match the activation after conversion, and thus significantly narrow the gap in activation distribution between the source ANN and calibrated SNN. We summarize the comparison between our calibration method and the existing conversion \& training methods in \autoref{tab_compare}. 
The novel contributions of the paper are threefold:

\begin{itemize}[leftmargin=*]
    \item We formulate the conversion equation and divide the conventional conversion error into flooring error and clipping error. And then we analyze the error propagation through layers. 
    \item We propose layer-wise calibration algorithm to adjust the network parameters including weights, bias, and initial potential to diminish the conversion error. To accommodate different user requirements, we provide Light Pipeline and Advanced Pipeline to balance accuracy and practical utility.
    \item We verify our algorithms on large-scale datasets like ImageNet~\cite{deng2009imagenet}. In addition to ResNets and VGG networks in previous work, we test a lightweight model MobileNet~\cite{Howard2017MobileNetsEC} and a large model RegNetX-4GF~\cite{radosavovic2020regnet} (79.4\% top-1 accuracy) for the first time in {ANN-to-SNN} conversion. Our method can increase up to 69\% accuracy in Spiking MobileNet conversion with 256 time steps. 
\end{itemize}


\section{Related Work}

For training-based SNN, there are several supervised learning algorithms divided into (1) synaptic plasticity and (2) surrogate gradient. Synaptic plasticity methods are based on time-sensitivity and update the connection weight via the two neurons' firing time interval \cite{kheradpisheh2018stdp,iyer2020classifying,li2019research}. They are more suitable for the neuromorphic image \cite{amir2017low} or rate coding from static images. On the other hand, surrogate gradient (spiking-based backpropagation) methods use a soft relaxed function to replace the hard step function and train SNN like RNN \cite{wu2018spatio,shrestha2018slayer}. They suffer from the computationally expensive and slow during the training process on complex network architecture\cite{rathi2019enabling}.

Unlike training from scratch, ANN-to-SNN conversion methods, such as data-based normalization~\cite{Diehl2015Fast,rueckauer2016theory} or threshold balancing~\cite{Diehl2015Fast,diehl2016conversion}, adapt to more complex situations~\cite{tavanaei2019deep}.
The major bottleneck of these methods is how to balance accuracy and inference latency as they require more than 2k time steps to get accurate results. 
Recently, many methods have been proposed to reduce the conversion loss and simulation length. The soft-reset also called the reset-by-subtraction mechanism, is the most common technique to address the potential reset's information loss \cite{rueckauer2016theory,han2020deep}. Our IF neuron model also adopts this strategy. \citet{rueckauer2017conversion} suggest using percentile threshold, which avoids picking the outlier in the activation distribution.
Spike-Norm~\cite{Sengupta2018Going} tests architectures like VGG-16 and ResNet-20. In this work, we further extend the source architecture to MobileNet and RegNet.
RMP~\cite{han2020rmp} and TSC~\cite{han2020deep} achieves near-to-origin accuracy by adjusting the threshold according to the input and output spike frequency.
\citet{deng2021optimal} decompose the conversion loss into each layer and reduce it via shifting bias. Low latency converted SNN is an on-going research challenge since it still requires a considerable amount of simulation length. At the same time, most SNN conversion work does not address the BN layers in low latency settings.


\section{Preliminaries}
\label{sec_pre}
\textbf{Neuron Model for ANN.} 
Considering the $\ell$-th fully-connected layer or convolutional layer in the ANNs, its forwarding process can be formulated as,
\begin{equation}
    \mathbf{x}^{(\ell+1)} = h(\mathbf{z}^{(\ell)})=h(\mathbf{W}^{(\ell)}\mathbf{x}^{(\ell)}),  1\le\ell\le n, 
    \label{eq_ann}
\end{equation}
where $\mathbf{x}^{(\ell)}$, $\mathbf{W}^{(\ell)}$ denote the input activation and weight parameters in that layer respectively, and $h(\cdot)$ is the ReLU activation function. One can optionally train a bias parameter $\mathbf{b}^{(\ell)}$ and add it to pre-activation.

\textbf{Neuron Model for SNN.} 
Here we use the Integrate-and-Fire (IF) neuron model \cite{liu2001spike,barbi2003stochastic}. In specific, suppose at time step $t$ the spiking neurons in layer $\ell$ receive its binary input $\mathbf{s}^{(\ell)}(t)\in \{0, V_{th}^{(\ell-1)}\}$, the neuron will update its temporary membrane potential by,
\begin{equation}
    \mathbf{v}^{(\ell)}_{temp}(t+1) = \mathbf{v}^{(\ell)}(t) + \mathbf{W}^{(\ell)}\mathbf{s}^{(\ell)}(t),
    \label{eq_integrate}
\end{equation}
where $\mathbf{v}^{(\ell)}(t)$ denotes the membrane potential at time step $t$, and $\mathbf{v}^{(\ell)}_{temp}(t+1)$ denotes the intermediate variable that would be used to determine the update from $\mathbf{v}^{(\ell)}(t)$ to $\mathbf{v}^{(\ell)}(t+1)$.
If this temporary potential exceeds a pre-defined threshold $V_{th}^{(\ell)}$, it would produce a spike output $\mathbf{s}^{(\ell+1)}(t)$ with the value of $V_{th}^{(\ell)}$. Otherwise, it would release no spikes, i.e. $\mathbf{s}^{(\ell+1)}(t) = 0$. The membrane potential at the next time step $t+1$ would then be updated by \textit{soft-reset} mechanism, also known as \textit{reset-by-subtraction}. Mathematically, we describe the updating rule as 
\begin{equation}
    \mathbf{v}^{(\ell)}(t+1) = \mathbf{v}^{(\ell)}_{temp}(t+1) - \mathbf{s}^{(\ell+1)}(t), \label{eq_snn_neuron}
\end{equation}
\begin{equation}
    \mathbf{s}^{(\ell+1)}(t) = \begin{cases}
     V_{th}^{(\ell)} & \text{if }\mathbf{v}^{(\ell)}_{temp}(t+1) \ge V_{th}^{(\ell)} \\
    0 & \text{otherwise } 
    \end{cases}.
    \label{eq_if}
\end{equation}
Note that $V_{th}^{\ell}$ can be distinct in each layer. Thus, we cannot represent the spike in the whole network with binary signals. 
This problem can be avoided by utilizing a weight normalization technique to convert the $\{0, V_{th}^{(\ell-1)}\}$ spike to $\{0,1\}$ spike in every layers, given by:
\begin{equation}
    \mathbf{W}^{(\ell)} \leftarrow \frac{V_{th}^{(\ell-1)}}{V_{th}^{(\ell)}}, \ \ \ V_{th}^{(\ell)} \leftarrow 1. 
\end{equation} 
Recursively applying the above euqalization, we can use 0,1 spike to represent the intermediate activation for each layer. For the rest of the paper, we shall continue using the notation of $\{0, V_{th}^{(\ell-1)}\}$ spike for simplicity.

As for the input to the first layer and the output of the last layer, we do not employ any spiking mechanism. We use the first layer to direct encode the static image to temporal dynamic spikes, this can prevent the undesired information loss of the Poission encoding. For the last layer output, we only integrate the pre-synaptic input and does not firing any spikes. This is because the output can be either positive or negative, yet \autoref{eq_if} can only convert the ReLU activation. 

\textbf{Converting ANN to SNN }
Compared with ANN, SNN employs binary activation (i.e. spikes) at each layer. To compensate the loss in representation capacity, researchers introduce the time dimension to SNN by repeating the forwarding pass $T$ times to get final results. Ideally, the converted SNN is expected to have approximately the same input-output function mapping as the original ANN, i.e.,
\begin{equation}
    \mathbf{x}^{(\ell)} \approx \bar{\mathbf{s}}^{(\ell)}= \frac{1}{T}\sum_{t=0}^T\mathbf{s}^{(\ell)}(t).
    \label{approx_output}
\end{equation}
In practice, the above approximation only holds when $T$ grows to 1k or even higher. However, high $T$ would lead to large inference latency thus damage SNN's practical utility.




\begin{figure}[t]
    \centering
    \includegraphics[width=0.9\linewidth]{figs/floor_clip.pdf}
    \caption{The conversion error between the ReLU activation used in ANN and the output spike in SNN ($V_{th}=10, T=5$) contains flooring error and clipping error.}
    \label{fig_clip_floor}
   \end{figure}

\section{Methodology}
\label{sec_method}
\subsection{Dividing the Conversion Loss}
We first use the derivation in \citet{deng2021optimal} to deduce the relationship between $\bar{\mathbf{s}}^{(\ell)}$ and $\bar{\mathbf{s}}^{(\ell+1)}$.
Suppose the initial membrane potential $\mathbf{v}^{(\ell)}(0) = \mathbf{0}$. Substitute \autoref{eq_integrate} into \autoref{eq_snn_neuron} and sum over $T$, then we get
\begin{equation}
   \mathbf{v}^{(\ell)}(T) = \mathbf{W}^{(\ell)}\left(\sum_{t=0}^{T}\mathbf{s}^{(\ell)}(t) \right) - \sum_{t=0}^{T}\mathbf{s}^{(\ell+1)}(t).
   \label{eq_T_total}
\end{equation}
Since at each time step, the output can be either $0$ or $V_{th}^{(\ell)}$, the accumulated output $\sum_{t=0}^{T}\mathbf{s}^{(\ell+1)}(t)$ can be written to $mV_{th}^{(\ell)}$ where $m \in \{0, 1, \dots, T\}$ denotes the total number of spikes. Note that we assume the terminal membrane potential $\mathbf{v}^{(\ell)}(T)$ lies within the range $[0, V_{th}^{(\ell)})$.
Therefore, according to \autoref{eq_T_total}, we have
\begin{equation}
    T\mathbf{W}^{(\ell)} \bar{\mathbf{s}}^{(\ell)} - V_{th}^{(\ell)} < mV_{th}^{(\ell)} \le T\mathbf{W}^{(\ell)}\bar{\mathbf{s}}^{(\ell)}.
\end{equation}
where $\bar{\mathbf{s}}^{(\ell)}$ is defined in \autoref{approx_output}. Then, we can use floor operation and clip operation to determine the $m$:
\begin{equation}
    m = \mathrm{clip}\left(\left\lfloor\frac{T}{V_{th}^{(\ell)}}\mathbf{W}^{(\ell)}\bar{\mathbf{s}}^{(\ell)}\right\rfloor, 0, T\right).
\end{equation}
Here the clip function sets the upper bound $T$ and lower bound $0$. Floor function $\lfloor x \rfloor$ returns the greatest integer that less than or equal to $x$. Given this formula, we can calculate the expected output spike:
\begin{align}
   \bar{\mathbf{s}}^{(\ell+1)} & = \mathrm{clipfloor}(\mathbf{W}^{(\ell)}\bar{\mathbf{s}}^{(\ell)}, T, V_{th}^{(\ell)}) \nonumber\\ 
   & = \frac{V_{th}^{(\ell)}}{T}\mathrm{clip}\left(\left\lfloor\frac{T}{V_{th}^{(\ell)}}\mathbf{W}^{(\ell)}\bar{\mathbf{s}}^{(\ell)}\right\rfloor, 0, T\right)
   \label{eq_clip_floor}
\end{align}
According to \autoref{eq_clip_floor}, the conversion loss (difference between $\mathbf{x}^{(\ell+1)}$ and $\bar{\mathbf{s}}^{(\ell+1)}$) comes from two aspects, namely the \emph{flooring error} and the \emph{clipping error}. 

In \autoref{fig_clip_floor}, we further indicate that $V_{th}^{(\ell)}$ is crucial for conversion loss because it affects both the flooring and the clipping errors. Increasing $V_{th}^{(\ell)}$ leads to lower clipping error but higher flooring error.
Previous work \cite{Diehl2015Fast,diehl2016conversion} sets $V_{th}^{(\ell)}$ to the maximum pre-activations across samples in ANN to eliminate the clipping error. However, the maximum pre-activations are usually outliers. Given this insight, the outliers may tremendously increase the flooring error. As a result, they have to use a very large $T$ (for example, 2000) to decrease the flooring error. 

\begin{figure}[t]
 \centering
 \includegraphics[width=0.9\linewidth]{figs/mse.pdf}
 \caption{Comparison of the threshold determined by three different approaches. Our MMSE threshold will vary with time steps.}
 \label{fig_mse}
\end{figure}

\subsection{Adaptive Threshold by MMSE}
In an effort to better balance flooring error and the clipping error, we use Minimization of Mean Squared Error (MMSE) to obtain the threshold $V_{th}^{(\ell)}$ under different simulation length $T$. 
Here we adopt the similar layer-wise optimization problem in~\cite{deng2021optimal}, which is formulated by
\begin{equation}
    \min_{V_{th}} \left(\mathrm{clipfloor}(\mathbf{x}^{(\ell+1)}, T, V_{th}^{(\ell)}) - \mathrm{ReLU}({\mathbf{x}}^{(\ell+1)})\right)^2
\end{equation}
Note that the above problem is not guaranteed to be convex, and there is no closed-form solution to this minimization. We hereby sample several batches of training images and use grid search to determine the final result of $V_{th}^{(\ell)}$. Specifically, we linearly sample $N$ grids between $[0, \max({\mathbf{x}}^{(\ell+1)})]$, and find the grid that has lowest MSE. We set $N=100$ and find this option is precise enough to obtain a good solution.
\autoref{fig_mse} shows the dynamics of our proposed method. It is worthwhile to note that $V_{th}^{(\ell)}$ does not monotonically increase along with $T$, because the flooring error may be decreased by slightly increment the threshold. We can further apply MMSE threshold channel-wisely to further decrease the MSE error, as did in~\cite{kim2019spiking}.

\subsection{Layer-wise Calibration}
\label{sec_calibration}
Besides adaptive threshold, we further reduce the conversion error by calibrating the parameters of SNN.
We first analyze how conversion errors accumulate through layers, 
and then present a set of layer-wise algorithms to calibrate different types of SNN parameters, including bias, weights and initial membrane potential. 

As aforementioned, the output layer of our SNN only accumulate the pre-synaptic inputs through time, that is to say, the final output of SNN is arithmetic sum of output from each time step $\sum^T_{t=1}\mathbf{W}^{(n)}\mathbf{s}^{(n)}(t)$.
Note that this modification doesn't introduce additional overhead because network output won't participate in further operation.
With this modified last layer, our object becomes to minimize the difference between the input to the last layer, i.e. $\bar{\mathbf{s}}^{(n)}$ and $\mathbf{x}^{(n)}$.

\begin{lemma}
\label{lemma1}
Denote the Frobenius norm as $||\cdot||$, the conversion error in the last layer in given by
\begin{equation}
    ||\mathbf{e}^{(n)}||=||\mathbf{x}^{(n)} - \bar{\mathbf{s}}^{(n)}|| \le ||\sum_{\ell=1}^{n} Err^{(\ell)} \prod_{k=\ell}^{n}\mathbf{W}^{(k)}||,
    \label{eq_final_err}
\end{equation}
where $Err^{(\ell)}=\mathrm{clipfloor}(\bar{\mathbf{s}}^{(\ell)})-\mathrm{ReLU}(\bar{\mathbf{s}}^{(\ell)})$.
\end{lemma}

We provide the detailed derivation of above lemma in \aref{sec_append_A}. The above lemma indicates that the conversion errors in former layers have a cumulative effect on the subsequent layers. 
In addition, the conversion error in the ultimate layer is upper bounded by weighted linear combination of layer-wise error. Based on this observation, we develop a set of greedy layer-wise calibration algorithm to correct the conversion error in each layer progressively.

We introduce Light Pipeline and Advanced Pipeline, which can be chose by users according to their memory and computation budgets for layer-wise calibration in practice.
The light pipeline achieves fast calibration with less memory and computation consumption by only adjusting bias in SNNs. With a little effort, the light pipeline can outperform state-of-the-art methods by a large margin. 
We also propose an Advanced Pipeline that achieves best results by calibrating the weights as well as the initial membrane potential in a fine-grained way. 



\textbf{Light Pipeline }
Light Pipeline only contains Bias Calibration (BC).
In order to calibrate the bias parameters, we first define a reduced mean function:
\begin{equation}
    \mu_c(\mathbf{x})= \frac{1}{wh}\sum_{i=1}^{w}\sum_{j=1}^{h} \mathbf{x}_{c,i,j}
\end{equation}
where $w, h$ are the width and height of the feature-map, and $\mu_c(\mathbf{x})$ computes the spatial mean of the feature-map in each channel $c$.
We notice that the spatial mean of conversion error $\mathbf{e}^{(\ell)} = \mathbf{x}^{(\ell)} - \bar{\mathbf{s}}^{(\ell)}$ can be written by
\begin{equation}
   \mu_c(\mathbf{x}^{(\ell)}) = \mu_c(\bar{\mathbf{s}}^{(\ell)}) + \mu_c(\mathbf{e}^{(\ell)}). \label{eq_bc}
\end{equation}
To ensure the mean output of SNN is equal to the mean output of ANN, we can add the expected conversion error into the bias term as $\mathbf{b}^{(\ell)}_c\leftarrow\mathbf{b}^{(\ell)}_c+\mu_c(\mathbf{e}^{(\ell+1)})$. In practice, we only sample one batch training images and compute the reduced mean to calibrate the bias. 

\textbf{Advanced Pipeline }
Calibrating the bias only corrects partial error. We need a more fine-grained calibration method. To this end, we propose advanced pipeline which consists of Potential Calibration (PC) and Weights Calibration (WC).

Now consider a non-zero initial membrane potential $\mathbf{v}^{(\ell)}(0)$, we can rewrite \autoref{eq_clip_floor} to 
\begin{equation}
      \tilde{\bar{\mathbf{s}}}^{(\ell+1)} = \frac{V_{th}^{(\ell)}}{T}\mathrm{clip}\left(\left\lfloor\frac{T}{V_{th}^{(\ell)}}\mathbf{W}^{(\ell)}\bar{\mathbf{s}}^{(\ell)}+\frac{\mathbf{v}^{(\ell)}(0)}{V_{th}^{(\ell)}}\right\rfloor, 0, T\right),
\end{equation}
where $\tilde{\bar{\mathbf{s}}}^{(\ell+1)}$ is the calibrated expected output with non-zero initialization of membrane potential.
To obtain a fast calibration for the initial membrane potential, here we make an approximation that:
\begin{align}
    \tilde{\bar{\mathbf{s}}}^{(\ell+1)} & \approx \frac{V_{th}^{(\ell)}}{T}\mathrm{clip}\left(\left\lfloor\frac{T}{V_{th}^{(\ell)}}\mathbf{W}^{(\ell)}\bar{\mathbf{s}}^{(\ell)}\right\rfloor, 0, T\right)+\frac{\mathbf{v}^{(\ell)}(0)}{T} \nonumber \\
    & = \bar{\mathbf{s}}^{(\ell+1)} + \mathbf{v}^{(\ell)}(0)/T
\end{align} 
Similar to BC, $\mathbf{v}^{(\ell)}(0)/T$ can correct the output distribution of SNN. We can directly set $\mathbf{v}^{(\ell)}(0)$ to $T\times \mathbf{e}^{(\ell+1)}$ to calibrate the initial potential. 
Note that Potential Calibration does not need to compute spatial mean.

In advanced pipeline, we also introduce Weights Calibration to correct the conversion error in each layer. In WC, the formulation is given by:
\begin{align}
   \min_{\mathbf{W}^{(\ell)}}  ||\mathbf{e}^{(\ell+1)}||^2 \label{eq_wc}.
\end{align}
Here we optimize the whole weights tensor in SNN layer-by-layer and can reduce the conversion error even further. For practical implementation, we will first store input samples in ANN $\mathbf{x}^{(\ell)}$ and input spikes samples in SNN of each time step $\mathbf{s}^{(\ell)}(t)$ and then compute the expected spike input by $\bar{\mathbf{s}}^{(\ell)}=\sum_{t=1}^T\mathbf{s}^{(\ell)}(t)$. To further compute the gradient of $\mathrm{clipfloor}$ function, we apply the StraightThrough Estimator~\cite{bengio2013ste} of the floor operation, i.e. 
\begin{equation}
    \frac{\partial \lfloor x \rfloor}{\partial x} = 1.
\end{equation} 
To this end, we can use regular training methods like stochastic gradient descent for calibrating the weights.
When conducting calibration for weights, the optimization process is very efficient compared to other direct training methods. This is because we first store the expected input from previous layers, and we do not have to perform $T$ times convolution like direct training methods. The major bottleneck of WC is storing the input of SNN. For example, if we set $T=1024$, then we will do 1024 times forwarding pass for one batch and accumulate them to get the final expected results. 


\begin{algorithm}[t]
\caption{Overall Algorithms}
\label{alg:1}
\begin{algorithmic}
    \INPUT Pretrained ANN; simulation length $T$
    \STATE Fold BN Layers into Conv Layers (cf. \autoref{eq_bn_fold})
    \STATE Replace AvgPooling Layers to depthwise Conv Layers
    \FOR{all $i=1,2,\dots, N$-th layers in the ANN}
        \STATE Collect input data $\mathbf{x}^{(i)}$, output data $\mathbf{x}^{(i+1)}$ in one batch
        \STATE Get MMSE threshold $V_{th}^{(i)}$ using grid search
        \STATE Get SNN output $\bar{\mathbf{s}}^{(i+1)}$
        \STATE Compute Error term $\mathbf{e}^{(i+1)}=\mathbf{x}^{(i+1)}-\bar{\mathbf{s}}^{(i+1)}$
        \IF{\textit{Light Pipeline}}
            \STATE Calibrate bias term $\mathbf{b}^{(i)}\leftarrow\mathbf{b}^{(i)}+\mu(\mathbf{e}^{(i+1)})$
        \ELSE
            \STATE Calibrate Potential $\mathbf{v}^{(i)}(0) \leftarrow T\times\mathbf{e}^{(i+1)}$ 
            \STATE Optimize weights to minimize $||\mathbf{e}^{(\ell+1)}||^2$  via stochastic gradient descent
        \ENDIF
    \ENDFOR
\OUTPUT Converted SNN model
\end{algorithmic}
\end{algorithm}


\subsection{Average Pooling Layers}

Most SNN works do not include the Max Pooling layers since finding the maximum activation neuron ahead of time is impractical, i.e. we cannot determine the maximum neuron $\bar{\mathbf{s}}$ when we only observe $\mathbf{s}(1)$. 
Therefore, they use Average Pooling Layers to downsample the feature-maps and do not convert them in SNN. However, we argue that Average Pooling Layers will produce non-binary information. For example, a $2\times2$ kernel AvgPool layer can output 4 possible values $[0, 0.25, 0.5, 1]$ with spike inputs. 
To make SNN run on corresponding hardware, we convert the AvgPool layer by treating the AvgPool layer as a convolutional layer with specific values. So we can convert the AvgPool layer just like other convolutional layers. More details are included in Appendix.

\subsection{Converting BN layers}
There is no corresponding module in SNN for Batch Normalization (BN) layers. \citet{rueckauer2017conversion} propose to absorb the BN parameters to the weight and bias, which can be represented by:
\begin{equation}
    \mathbf{W}\leftarrow \mathbf{W}\frac{\gamma}{\sigma}, \ \ \ \mathbf{b}\leftarrow \beta + (\mathbf{b}-\mu)\frac{\gamma}{\sigma}, \label{eq_bn_fold}
\end{equation}
where $\mu, \sigma$ are the running mean and standard deviation, and $\gamma, \beta$ are the transformation parameters in the BN layer. 

\section{Experiments}

To demonstrate the effectiveness and the efficiency of the proposed algorithm, we conduct experiments on CIFAR~\cite{cifardataset} and ImageNet~\cite{deng2009imagenet} datasets with extremely low simulation length (say $T\le 256$). In \autoref{sec_ab_study}, we study the impact of the approximations and design choices made in~\autoref{sec_method}. 
In \autoref{sec_sota_comp}, we compare our methods to other methods. 

\subsection{Implementation Details}
For all ANN with BN layers, we fold the BN layer before conversion. We do not convert input images to binary spikes because generating binary spikes requires time and degrades the accuracy. We also do not convert network output to spikes as explained in~\autoref{sec_calibration}.
To correct the bias and membrane potential, we sample one batch of unlabeled data (128 training images). To estimate the MMSE threshold and calibrate weights, we use 1024 training images. 
In our experiments, we apply the bias shift as described in \citet{deng2021optimal}. 
We use Stochastic Gradient Descent with 0.9 momentum to optimize weights in WC, followed by a cosine learning rate decay~\cite{loshchilov2016sgdr}. The learning rate for WC is set to $10^{-5}$, and no L2 regularization is imposed. We optimize the weights in each layer with 5000 iterations. We will analyze the time and space complexity of our algorithm in the next section. Note that the training details of ANN are included in the Appendix.


\subsection{Ablation Study}
In this section, we verify the design choices of our proposed adaptive threshold and layer-wise calibration. In all ablation experiments, we test \textit{VGG-16 and ResNet-20}~\cite{Sengupta2018Going,han2020rmp} \textit{on CIFAR100}.
We also conduct variance studies by running 5 times with different random seeds and report the mean and standard deviation of the (top1) accuracy on the validation set

\label{sec_ab_study}





\begin{table}[t]
\footnotesize
    \centering
    \begin{adjustbox}{max width=0.48\textwidth}
    \begin{tabular}{lcccc}
    \toprule
    \multirow{2}{6em}{\textbf{Method}} & \multicolumn{2}{c}{\textbf{VGG-16 (77.89)}} & \multicolumn{2}{c}{\textbf{ResNet-20 (77.16)}} \\
    \cmidrule(l{2pt}r{2pt}){2-3}\cmidrule(l{2pt}r{2pt}){4-5}
    & T=16 & T=32 & T=16 & T=32 \\
    \midrule
    Maximum Act & 2.38\mypm{0.17} & 5.01\mypm{1.23}& 25.67\mypm{4.67}& 51.27\mypm{3.82}\\
    Percentile 99.9\% & 3.73\mypm{0.38} & 42.11\mypm{0.37} & 55.00\mypm{0.47} & 71.94\mypm{0.19} \\
    MMSE (Ours) & \textbf{19.42\mypm{0.81}} & 43.53\mypm{0.72} & 58.38\mypm{0.62} & 72.13\mypm{0.18} \\
    MMSE* (Ours) & 17.50\mypm{2.46} & \textbf{47.40\mypm{2.71}} & \textbf{63.55\mypm{0.60}} & \textbf{73.57\mypm{0.06}} \\
    \bottomrule
    \end{tabular}
    \end{adjustbox}
    \caption{(Top-1) Accuracy comparison on different threshold determination methods for ANN with BN layers. * denotes channel-wise threshold.} \label{tab_mmse_thresh}
\end{table}

\begin{table}[t]
\footnotesize
    \centering
    \begin{adjustbox}{max width=0.48\textwidth}
    \begin{tabular}{lcccc}
    \toprule
    \multirow{2}{6em}{\textbf{Method}} & \multicolumn{2}{c}{\textbf{VGG-16 (77.89)}} & \multicolumn{2}{c}{\textbf{ResNet-20 (77.16)}} \\
    \cmidrule(l{2pt}r{2pt}){2-3}\cmidrule(l{2pt}r{2pt}){4-5}
    & T=16 & T=32 & T=16 & T=32 \\
    \midrule
    MaxAct + BC& 24.61\mypm{2.19} & 32.60\mypm{2.61} & 55.17\mypm{5.06} & 68.78\mypm{2.48} \\
    Percentile + BC & 40.56\mypm{1.29} & 66.87\mypm{0.78} & 69.60\mypm{0.14}& 75.26\mypm{0.20} \\
    MMSE + BC & 44.95\mypm{0.52} & 67.61\mypm{0.72} & 70.78\mypm{0.15} & 75.53\mypm{0.15}\\
    MMSE* + BC & \textbf{52.96\mypm{1.91}} & \textbf{69.19\mypm{0.75}} & \textbf{72.33\mypm{0.13}} & \textbf{75.94\mypm{0.23}} \\
    \bottomrule
    \end{tabular}
    \end{adjustbox}
    \caption{Light pipeline combines BC with different thresholds for ANN with BN layers and consistently improves accuracy. * denotes channel-wise threshold.} \label{tab_bias_corr}
\end{table}


\textbf{Effect of MMSE Threshold in Conversion}
We study the effect of choosing different threshold $V_{th}$. In \autoref{tab_mmse_thresh}, we show that maximum activation has the lowest effect because of the under-fire problem in the initial stage. Our MMSE threshold achieves better results than maximum activation~\cite{Diehl2015Fast} and percentile~\cite{rueckauer2017conversion} threshold when $T =16$. As an example, our method is 15.7\% higher in accuracy than percentile when converting VGG-16. In ResNet-20, better threshold can significantly improve the accuracy of conversion. 
Finally, we apply the channel-wise MMSE threshold and further boost the accuracy from 43.5\% to 47.4\% in VGG-16 and from 72.1\% to 73.5\% in ResNet-20.



\textbf{Light Pipeline: Combining Bias Calibration}
Next, we verify the effect of the proposed Bias Calibration by applying it to different threshold methods. Results are summarized in \autoref{tab_bias_corr}, where we can find BC can \textit{consistently improve the accuracy of converted SNN by simply tuning the bias parameters.} For example, BC can boost 22\% accuracy in VGG-16 using percentile threshold when $T=16$. On our MMSE threshold, the Bias Calibration can increase up to 35\% accuracy. We should emphasize that BC is cheap and only requires tiny memory space to store the bias term for different $T$. Therefore our light pipeline is flexible to make trade-off between accuracy and latency. 


\newcommand{\cmark}{\color{ForestGreen}\ding{51}}%
\newcommand{\xmark}{\color{Red}\ding{55}}%

\begin{table*}[h]
    \footnotesize
        \centering
        \begin{adjustbox}{max width=\textwidth}
        \begin{tabular}{lcccccccc}
        \toprule
        \textbf{Method} & \textbf{Use BN} & \textbf{Convert AP} & \textbf{ANN Acc.} & \textbf{$T=32$} & \textbf{$T=64$} & \textbf{$T=128$} &\textbf{ $T=256$} & \textbf{ $T\ge 2048$} \\
        \midrule
        \multicolumn{9}{c}{\textbf{ResNet-34~\cite{He2016Deep}    ImageNet}} \\
        \midrule
        Spike-Norm~\cite{Sengupta2018Going} & \xmark & \xmark & 70.69 & - &- &- &- & 65.47 \\ 
        Hybrid Train~\cite{rathi2019enabling} & \xmark & \xmark & 70.20 & - &- &- & 61.48 & 65.10 \\
        RMP~\cite{han2020rmp} & \xmark & \xmark & 70.64 & - & - & - & 55.65 & 69.89 \\
        TSC~\cite{han2020deep} & \xmark & \xmark & 70.64 & - & - & - & 55.65 & 69.93 \\
        Opt.~\cite{deng2021optimal}* & \xmark & \cmark & 70.95 & 33.01 & 59.52 & 67.54 & 70.06 & 70.98 \\
        \textbf{Ours (Light Pipeline)} & \xmark & \cmark & 70.95 & \textbf{62.34} & \textbf{68.38} & \textbf{70.15} & \textbf{70.75} & 70.97 \\
        Opt.~\cite{deng2021optimal}* & \cmark & \cmark & 75.66 & 0.09 & 0.12 & 3.19 & 47.11 & 75.08 \\
        \textbf{Ours (Light Pipeline)} & \cmark & \cmark & 75.66 & \textbf{50.21} & \textbf{63.66} & \textbf{68.89} & \textbf{72.12} &  75.44 \\
        \textbf{Ours (Advanced Pipeline)} & \cmark & \cmark & 75.66 & \textbf{64.54} & \textbf{71.12} & \textbf{73.45} & \textbf{74.61} & 75.45 \\
        \midrule
        \multicolumn{9}{c}{\textbf{VGG-16~\cite{simonyan2014very} ImageNet}} \\
        \midrule
        Spike-Norm~\cite{Sengupta2018Going} & \xmark & \xmark & 70.52 & - &- &- &- & 69.96 \\ 
        Hybrid Train~\cite{rathi2019enabling} & \xmark & \xmark & 69.35 & - &- &- & 62.73 & 65.19 \\
        RMP~\cite{han2020rmp} & \xmark & \xmark & 73.49 & - & - & - & 48.32 & 73.09 \\
        TSC~\cite{han2020deep} & \xmark & \xmark & 73.49 & - & - & - & 69.71 & 73.46 \\
        Opt.~\cite{deng2021optimal}* & \xmark & \cmark & 72.40 & 54.92 & 66.51 & 69.94 & 71.35 & 72.09 \\
        \textbf{Ours (Light Pipeline)} & \xmark & \cmark & 72.40 & \textbf{69.30} & \textbf{71.12} & \textbf{71.85} & \textbf{72.20} & 72.29 \\
        Opt.~\cite{deng2021optimal}* & \cmark & \cmark & 75.36 & 0.114 & 0.118 & 0.122 & 1.81 & 73.88 \\
        \textbf{Ours (Light Pipeline)} & \cmark & \cmark & 75.36 & \textbf{24.88} & \textbf{56.77} & \textbf{70.49} & \textbf{73.66} & 75.15 \\
        \textbf{Ours (Advanced Pipeline)} & \cmark & \cmark & 75.36 & \textbf{63.64} & \textbf{70.69}  & \textbf{73.32}  & \textbf{74.23} & 75.32 \\
        \midrule
        \multicolumn{9}{c}{\textbf{MobileNet~\cite{Howard2017MobileNetsEC} ImageNet}} \\ 
        \midrule
        Opt.~\cite{deng2021optimal}* & \cmark & \cmark & 73.40 & 0.110 & 0.104  & 0.100 & 0.964 & 68.21 \\
        \textbf{Ours (Light Pipeline)} & \cmark & \cmark & 73.40 & 0.254 & \textbf{12.62} & \textbf{53.91}  & \textbf{65.86} & \textbf{72.19}\\
        \textbf{Ours (Advanced Pipeline)} & \cmark & \cmark & 73.40 & \textbf{37.43} & \textbf{56.26} & \textbf{65.40} & \textbf{69.02} & \textbf{72.38} \\ 
         \midrule
        \multicolumn{9}{c}{\textbf{RegNetX-4GF~\cite{radosavovic2020regnet} ImageNet}} \\ 
        \midrule
        Opt.~\cite{deng2021optimal}* & \cmark & \cmark & \textbf{80.02} & 0.218 & 3.542 & 48.60 & 71.22 & 78.33 \\
        \textbf{Ours (Light Pipeline)} & \cmark & \cmark & \textbf{80.02} & \textbf{35.63} & \textbf{65.28} & \textbf{74.37}  & \textbf{77.33} & \textbf{79.15}\\
        \textbf{Ours (Advanced Pipeline)} & \cmark & \cmark & \textbf{80.02} & \textbf{55.70} & \textbf{70.96} & \textbf{75.78} & \textbf{77.50} & \textbf{79.21} \\
        \bottomrule
        \end{tabular}
        \end{adjustbox}
        \caption{Comparison of our algorithm with other existing SNN conversion works. \textit{Use BN} means use BN layers to optimize ANN, \textit{Convert AP} means use depthwise convolutional layers to replace Average Pooling layers. * denotes self-implementation results.} \label{tab_sota_compare}
\end{table*}



\begin{table}[t]
\footnotesize
    \centering
    \begin{adjustbox}{max width=0.48\textwidth}
    \begin{tabular}{lcccc}
    \toprule
    \multirow{2}{6em}{\textbf{Method}} & \multicolumn{2}{c}{\textbf{VGG-16 (77.89)}} & \multicolumn{2}{c}{\textbf{ResNet-20 (77.16)}} \\
    \cmidrule(l{2pt}r{2pt}){2-3}\cmidrule(l{2pt}r{2pt}){4-5}
    & T=16 & T=32 & T=16 & T=32 \\
    \midrule
    MMSE + PC & 59.52\mypm{1.06} & 70.62\mypm{0.47} & 73.37\mypm{0.28} & 76.21\mypm{0.06}\\
    MMSE* + PC & 65.29\mypm{0.86} & 73.53\mypm{0.27} & \textbf{74.22\mypm{0.25}} & \textbf{76.68\mypm{0.12}} \\
    \midrule
    MMSE + PC + WC & 65.02\mypm{0.33}& 73.51\mypm{0.23} & 73.36\mypm{0.28} & 76.32 \mypm{0.13}\\
    MMSE* + PC + WC & \textbf{67.14\mypm{0.88}} & \textbf{74.52\mypm{0.36}} & 74.02\mypm{0.20} & 76.52\mypm{0.16}\\
    \bottomrule
    \end{tabular}
    \end{adjustbox}
    \caption{Advanced pipeline use Potential and Weights Calibration to optimize SNN. * denotes channel-wise threshold.} \label{tab_heavy_pipe}
\end{table}
\textbf{Advanced Pipeline: Potential and Weights Calibration }
Our advanced pipeline contains Potential and Weights Calibration that will alter the ANN's parameters to adapt better in spiking configuration. We validate the effect of them on our MMSE threshold mode in \autoref{tab_heavy_pipe}. We can find that Potential Calibration can substantially improve the accuracy of SNN. As an example, the MMSE + BC on VGG-16 only has 44.95\% accuracy. However, with PC, we can uplift the accuracy to 59.52\%. We also find WC is slightly more stable since the variance of the results is lower. 


\subsection{Comparison to Previous Work}
\label{sec_sota_comp}


In this section, we compare our proposed algorithm with other existing work. We first test ImageNet models\footnote{We include the comparison on CIFAR dataset in \aref{sec_cifar_results}. }. Here we choose the widely adopted ResNet-34~\cite{He2016Deep} and VGG-16~\cite{simonyan2014very} in the existing literature. Note that we test ANNs both with and without BN layers. We additionally verify our algorithm on \textbf{MobileNet}~\cite{Howard2017MobileNetsEC}. To our best knowledge, this is the first work that studies Spiking MobileNet conversion.

Results can be found in \autoref{tab_sota_compare}. For both ResNet-34 and VGG-16 without BN, our light pipeline is within 1\% accuracy loss when $T=128$. On models with BN layers, our method can substantially improve the conversion loss. In particular, the light pipeline can improve 50.1\%, and the advanced pipeline can improve 64.4\% accuracy in ResNet-34 with BN Conversion when $T=32$. Baseline methods still produce a large accuracy gap even on VGG-16 without BN layers and AvgPool Conversion. While our light pipeline reaches 73.66 (less than 2\% accuracy drop) when $T=256$. 
The superiority of our algorithm is also reflected in Spiking MobileNet conversion, where the baseline method \cite{deng2021optimal} crashed when $T\le 256$. To reach acceptable accuracy of Spiking MobileNet, the baseline method has to increase $T$ to 2048. However, our advanced pipeline can achieve higher accuracy while reducing 8$\times$ simulation length (69.02 when $T=256$). Finally, we test our algorithm on a large ANN, RegNetX-4GF~\cite{radosavovic2020regnet} which achieves 79.4\% top-1 accuracy. Our light pipeline reaches 73\% accuracy when $T=128$ and our advanced pipeline reaches 75.8\% accuracy when $T=128$.

\subsection{Complexity Study}
\textbf{Time Complexity }
During run-time, our converted SNN will not produce additional inference time. However, converting SNN using light or advanced pipeline may require time and computing resources. The time needed for each calibration is described in the table below. All experiments were tested on a single NVIDIA GTX 1080TI with 5 runs. In \autoref{tab_convert_time}, we can see that the Bias Calibration and Potential Calibration only takes limited time on CIFAR100 and ImageNet. Using Weights Calibration is much expensive than the other two methods. For example, calibrate a MobileNet on ImageNet may take 30 minutes using the advanced pipeline. We should emphasize that our advanced pipeline is still much cheaper than Hybrid Train~\cite{rathi2019enabling}, which requires 20 epochs of end-to-end training (hundreds of GPU hours). 


\textbf{Space Complexity }
In this section, we report the memory requirements for each calibration algorithm. Since our method will calibrate a new set of parameters for different $T$, therefore it is necessary to study the model size if we want to deploy SNNs under different $T$. Specifically, calibrating the bias of ResNet-34 on ImageNet only requires 0.3653MB memory. However, calibrating the weights and potential requires 83.25MB and 18.76MB, respectively. 
Thus, our proposed light pipeline is both computational and memory cheap and is optimal for flexible SNN conversion. In contrast, the advanced pipeline (PC and WC) requires much more memory space. One may optionally only apply PC to lower down the memory footprint of ResNet-34. Interestingly, some tiny structures like MobileNet share less weights memory (12.21MB) but higher activation memory (19.81MB).

\begin{table}[t]
    \footnotesize
        \centering
        \begin{adjustbox}{max width=0.48\textwidth}
        \begin{tabular}{lccc}
        \toprule
        \textbf{Model} & \textbf{BC}  & \textbf{PC} & \textbf{WC} \\
        \midrule
        VGG-16 & 0.098$\pm$0.003 & 0.106$\pm$0.017 & 4.70$\pm$0.037 \\
        MobileNet & 2.29$\pm$0.005 & 2.19$\pm$0.01 & 27.6$\pm$1.62 \\ 
        \bottomrule
        \end{tabular}
        \end{adjustbox}
        \caption{Conversion time (minutes) for different calibration algorithm. We set $T=64$ and test VGG-16 on CIFAR100 and MobileNet on ImageNet.} \label{tab_convert_time}
\end{table}

\begin{table}[t]
    \footnotesize
        \centering
        \begin{adjustbox}{max width=0.48\textwidth}
        \begin{tabular}{lrrrr}
        \toprule
        \textbf{\#Samples} & 32  & 64 & 128 & 256 \\
        \midrule
        VGG-16 & 64.50$\pm$1.19 & 65.12$\pm$1.02 & 66.04$\pm$0.72 & 66.20$\pm$0.58\\
        ResNet-20 & 75.35$\pm$0.16 & 75.41$\pm$0.18 & 75.43$\pm$0.09 & 75.41$\pm$0.08\\ 
        \bottomrule
        \end{tabular}
        \end{adjustbox}
        \caption{Comparison of the accuracy using different number of data samples for bias correction.} \label{tab_num_sample}
\end{table}


\textbf{Data Sample Complexity}
We study the robustness of the our algorithm by increasing the size of calibration dataset. Here we test Bias Calibration in~\autoref{tab_num_sample} on ResNet-20 and VGG-16 ($T=32$). By increasing the number of samples for calibration, the accuracy will also increase. However, we can see that in ResNet-20 the effect of samples is trivial. While in VGG-16, increasing the number of samples from 32 to 256 can increase 1.7\% mean accuracy. We also find that more samples lead to a stable calibration result. Therefore, we recommend using at least 128 images for calibration. In our experiments, the same trend is also observed in other calibration algorithms.


\subsection{Efficiency and Sparsity}
\begin{figure}[t]
    \centering
    \includegraphics[width=0.45\textwidth]{figs/fire_rate.pdf}
    \caption{Firing rate visualization of VGG-16.}
    \label{fig_fire_rate}
\end{figure}

In this section we visualize the sparsity of our calibrated SNN. We choose the spike VGG-16 on the ImageNet dataset, with $T=64$. We only leverage light pipeline (bias calibration) and record the mean firing ratio across the whole validation dataset, which also corresponds to sparsity of the activation. The firing ratio is demonstrated in \autoref{fig_fire_rate}, where we can find the maximum firing ratio is under 0.08, and the minimum firing ratio can be 0.025. To quantitaively compute the energy saving, we use the energy-estimation euqation in~\citet{rathi2020diet}. For addition, we measure it by 0.9$J$ per operation; for multiplication, we measure it by 4.6$J$ per operation. On the event-driven neuromorphic hardware, a non-firing neuron will not cost any energy. Based on this rule, our calibrated spiking VGG-16 only costs 69.36\% energy of ANN's consumption. 


\section*{Conclusion}
In this work, we analyze the composition of conversion error and its cumulative effect. To reduce the gap between ANN activation and SNN activation, we propose adaptive threshold to determine the threshold in different time steps. We also introduce the layer-wise calibration, which significantly improves the performance of SNN compared with other \emph{simple-copy} methods. Layer-wise calibration is easy to use and only requires a few training images. 
Our method establishes new state-of-the-art performance for SNN conversion. It can successfully convert challenging architectures like MobileNet and RegNetX-4GF with a low latency (less than 256 time steps) for the first time. Even when converting the ANN with Batch Normalization layers, our method can preserve high classification accuracy. 

\section*{Acknowledgement}
This work is supported by NSFC 61876032. We greatly thank annoymous reviewers for their kind suggestions to this work.

