\section{Introduction}
Recently, deep learning models have evolved to deliver acceptable performance in many areas such as computer vision, natural language processing, and speech recognition. However, the number of parameters and computational complexity of most deep learning applications have increased significantly. This ever-increasing computational complexity makes the deployment of deep neural networks harder on edge devices. Furthermore, the energy required to train a deep learning model increases proportionately with computational complexity. Thus, the training cost of these large and complex models also increases significantly on the high-end cloud servers. Although, quantization techniques are commonly used to accelerate \textit{inference} of deep learning models, here we target a fully integer training pipeline \emph{(i.e. forward propagation, back-propagation and SGD)}. Unlike recent proposals in quantized back-propagation,  we directly change the number representation of floating-point values. Furthermore, we show empirically and theoretically that our proposed integer training methodology is effective in training deep learning models with \textit{integer-only arithmetic}.

Despite the fact that accelerated training using integer back-propagation seems intriguing, there are some challenges associated with integer gradient computation. For example, (i) gradients must be scaled correctly in order to be adapted to the limited dynamic range of the integer number (e.g. \texttt{int8}: $[-127,126]$). (ii) The numerical error of the gradient must be \textit{\textbf{unbiased}} in order to preserve the convergence trajectory of the training algorithm. A small numerical error can be accumulated through the course of training and change the convergence behaviour. (iii) The integer training must be \textit{distribution independent}, i.e. the training method should not depend on the distribution of the gradients, weights, or training data.

We propose a novel integer training method designed to address aforementioned challenges (i), (ii) and (iii) simultaneously. Having a linear dynamic fixed-point mapping coupled with a non-linear inverse mapping is the key to full integer training.

To this end, we make the following contributions:
\begin{itemize}
    \item A hardware-friendly integer training method is proposed based on extracting the maximum floating-point exponent of the tensors as the \textit{scale}. We propose linear fixed-point mapping of tensors, while the corresponding inverse mapping for floating-point is non-linear. Our proposed method addresses all the previously mentioned challenges: (i) it computes the scales dynamically, and moreover  the scale does not need to be adjusted in the course of training; (ii)  it provides an unbiased estimation of the gradients and consequently, its convergence trajectory closely follows the floating-point version; (iii) our proposed representation mapping does not depend on the distribution of the training data, weights, or gradients.  
    
    \item We study the optimality gap of our proposed integer training algorithm and show it is analogous to its floating-point counterpart in the course of training (\textbf{Theorem 1}). Our analysis of stochastic gradient descent with our proposed fixed-point gradients shows the original floating-point optimality gap is only shifted by a negligible amount (\textbf{Remark 3}). 
   
    \item Our proposed method effectively performs all operations required in modern neural networks using \textit{integer-only arithmetic}. For instance, the computation of linear layer, convolutional layer, batch-norm and layer-norm, residual connections and also stochastic gradient descent (including gradients, momentum, weight decay, and weight update) are all performed in integer arithmetic. To the best of our knowledge, this is the first time that \textit{back-propagation} of a batch-norm, and  the computation of stochastic gradient descent (SGD) is performed in integer arithmetic with negligible loss in the accuracy for large datasets such as ImageNet. 
\end{itemize}
The rest of this paper is structured as follows. Section \ref{sec:related-works} reviews some previous works in the field of quantized back-propagation and quantifies the similarities and differences with our \textit{representation mapping} method. Section \ref{sec:methodology} discusses our integer training methodology in detail. Theoretical aspects of our integer training method on SGD are studied in Section \ref{sec:sgd-analysis}. Experimental results supporting our integer training methodology and convergence theory are presented in Section \ref{sec:results}.


\section{Related works}\label{sec:related-works}

\citet{banner2018scalable},  proposed a bifurcated back-propagation method where the gradient of weights are in floating-point format and input gradients are in 8-bit fixed-point format. Moreover, they proposed a range-based batch-norm which is more tolerant to quantization noise. The main difference of our proposed algorithm with  \citet{banner2018scalable} is that our integer training method does not demand bifurcation. In other words, our gradients with respect to weights and inputs are represented and computed using \texttt{int8} format. Furthermore, in our method, the back-propagation of batch-norm, as well as stochastic gradient descent (SGD) algorithm,  are also represented and computed using integer values. 
\citet{zhang2020fixed} have proposed a layer-wise ``precision-adaptive'' quantization method that does not affect the distribution of the data. In their method, the quantization error is measured and the quantization scale is adjusted over the course of training accordingly. Similarly, in \citet{zhao2021distribution} a distribution adaptive quantization method is proposed that takes into account the distribution of gradients in the channel dimension. In addition, they introduced a gradient clipping strategy in order to normalize the magnitude of the gradients in the back-propagation. The scale of the quantization is adapted to the distribution of the gradients and adjusted iteratively over the course of training.
Unlike the distribution adaptive methods such as \citep{zhang2020fixed, zhao2021distribution}, our proposed integer training method does not depend on the distribution of training data, weights, and gradients. Thus, there is no need to adjust the scale iteratively in the course of training.
In \citet{zhu2020towards}, a ``direction sensitive gradient clipping method'' is proposed to avoid inappropriate updates in the back-propagation. Moreover, they proposed a learning rate scaling that tackles the problem of unbiased gradient error accumulation. In a similar work, \citet{sakr2018per} proposed a precision assignment methodology to improve the convergence of the quantized back-propagation. This precision assignment is based on some criterion on internal accumulator noise, quantization noise of backward and forward propagation, and gradient clipping.
Unlike \citep{zhu2020towards,sakr2018per}, our proposed method enjoys having the advantage of \textit{unbiased gradients}, hence the convergence trajectory closely follows the floating-point counterpart. Thus, our method does not need any specific gradient correction, gradient clipping strategy, and learning rate corrections as opposed to \citep{zhu2020towards, sakr2018per}. 
\citet{jin2022f8net} proposed a unified fixed-point and parameterized clipping activation to achieve high accuracy. Furthermore, they proposed a method that directly trains the fractional length (i.e. scale) of the fixed-point quantization. Additionally, they use a double forward batch-norm fusion to determine the scaling factor of the batch-norm layer in the forward propagation, while the back-propagation of batch-norm remains floating-point.
In contrast, our proposed integer training method is capable of implementing integer variant of the batch-norm layer in forward propagation without double fusion as opposed to \citep{jin2022f8net}. We also perform batch-norm's back-propagation in integer arithmetic which was ignored in the previous works. Note that back-propagation of batch-norm and layer-norm is sensitive to quantization, as such, naive quantization leads to training divergence.
Additionally, we perform stochastic gradient descent (SGD) computation including weight update, weight decay, and momentum in integer arithmetic with no significant loss of accuracy. 
In our proposed method, we used dynamic fixed-point number format \citep{courbariaux2014training} as the primary number format for integer training. This specific fixed-point number format (also known as block floating-point format) has been used previously to quantize \textbf{\textit{inference}} of deep learning models in \citet{courbariaux2014training,drumond2018training}, and \citet{darvish2020pushing}.


\section{Methodology}\label{sec:methodology}
Common quantization methods, which use division and clipping techniques, are inefficient for back-propagation. Therefore, we decided to go one step deeper and change the number format directly. Here, we propose a hardware-friendly {\textit{number representation mapping}} using the dynamic fixed-point number format where the scale is defined {\textit{per tensor}}. In this approach, each tensor can be represented by its \texttt{int8} version while it is multiplied by a shared scale, as opposed to other methods that allow multiple shared scales for different partitions of the tensor (see \citet[Figure 4]{darvish2020pushing}). One shared scale per tensor makes the computations easier in the computing hardware (e.g. CPU or GPU). However, the training algorithm diverges if the representation mapping is not executed properly. To tackle this problem, we suggest two subtle changes; (i) we propose to perform fixed-point mapping of a tensors in a \textbf{\textit{linear}} fashion while the inverse mapping is \textbf{\textit{non-linear}} as explained in Sections \ref{sec:lin-quant} and \ref{sec:nonlin-dequant}. This is the key to success of our algorithm since a linear fixed-point mapping allows monotonic conversion of floating-point format to fixed-point, while a non-linear inverse mapping allows preserving information.
(ii) We suggest to use stochastic rounding in the back-propagation in a way that preserves the expected value of the tensors as well as their vital statistics, such as the mean. Stochastic rounding in conjunction with representation mapping is crucial to keep the integer training loss trajectory close to its floating-point counterpart.


\begin{figure}[!t]
    \centering
        \includegraphics[width=0.55\textwidth]{figs/quant2.pdf}~~~
        \includegraphics[width=0.45\textwidth]{figs/dequant2.pdf}\\
        \small{(a)}\hspace{8.5cm}\small{(b)}\\
    
    \caption{ (a) Linear fixed-point mapping, (b) Non-linear inverse mapping. 
    }
    \label{fig:quant}
\end{figure}


\subsection{Linear fixed-point mapping}\label{sec:lin-quant}

Our proposed integer training method is based on manipulating the floating-point number format. This is a very simple and effective way of converting floating-point values to fixed-point values as opposed to other commonly used methods that involve division operation. Our method essentially uses \textit{shift} and \textit{round} operations to convert floating-point to fixed-point format, see Figure \ref{fig:quant}(a). 

In this method, a tensor comprising $n$ floating-point numbers $(f_1, f_2, ... , f_n)$ is converted to a fixed-point tensor. First, as shown in Figure \ref{fig:quant}(a), sign, exponent, and mantissa of each floating-point number are extracted using the \textit{unpack to integer} function. For instance, $f_1$ is unpacked to $s_1,e_1,m_1$ where $(s,e,m)$ is used to denote (\textit{sign}, \textit{exponent}, \textit{mantissa}). 
We simply find the \textit{maximum} element of $e_1,e_2,… e_n$ i.e. $e_{\mathrm{max}}=\max(e_1,e_2,… e_n)$ to extract the shared scale of the tensor. Subsequently, the original mantissas are shifted to the right according to the difference of their exponent and $e_\mathrm{max}$, to get the individual mantissas.
For instance, $m_1$ is shifted to right by $e_\mathrm{max}-e_1$ which is denoted by $m_1 >> (e_\mathrm{max}-e_1)$. By shifting mantissas to right, we intentionally push the small values to the \textit{sub-normal} region, where the most significant bit of mantissa is not 1 in binary format i.e. $(1)_2$, see \citet{zuras2008ieee}. Pushing the mantissas to the sub-normal region is performed to align their exponents with $e_\mathrm{max}$ and this unifies the scale for fixed-point values in a tensor. In the next step, to create \texttt{int8} mantissas, the 24-bit single floating-point mantissas are further rounded to 7-bit mantissas to construct signed \texttt{int8} values i.e 7-bit unsigned integer and 1 sign bit.



\subsection{Non-linear inverse mapping}\label{sec:nonlin-dequant}
Our proposed fixed-point mapping method is performed by pushing mantissa values to the sub-normal region and rounding them stochastically. Hence,  the inverse mapping must be performed using a floating-point alignment module that normalizes the mantissas. A \textit{normalized} floating-point mantissa is required to start with the most significant bit $(1)_2$.  For instance, an alignment module converts $2^{127} \times (0.0101)_2 $ to $2^{127-2} \times (1.0100)_2 $.
The alignment module is a very well-known logic circuit that is available in the commodity hardware using Leading Zero Anticipator (LZA) block \citep{schmookler2001lza}.
Coupling a linear fixed-point mapping with a non-linear inverse mapping in this way keeps the number format close to floating-point. Moreover, combining them with stochastic rounding, results in having unbiased fixed-point variant that forces the trajectory of the integer training closely follow the floating-point counterpart.

Figure \ref{fig:quant}(b) shows the inverse mapping unit. This unit receives a tensor of mantissas and a single value of shared exponent computed by an integer layer. Then a tensor of exponents is reconstructed by repeating the value of the shared exponent. The size of this tensor is equal to the mantissa tensor’s size. Moreover, the mantissa tensor is rounded and its sign is extracted. Note that the rounding is used here because the output of the previous layer might have some excessive mantissa bits; this normally happens in matrix multiplication and convolution operations. Then the sign, exponent, and mantissa tensors are sent to an alignment unit that shifts the mantissa and adjust the exponent in order to normalize the floating-point number format \citep{zuras2008ieee}. Finally, the result is packed and carried out to the next layer.

\subsection{Integer layer computations}
When the fixed-point mapping of floating-point values is completed, the fixed-point values are used to perform the \textit{integer-only} layer computations. For instance, let us consider a linear layer which consists of a General Matrix Multiplication (GEMM) operation, but, the idea can be generalized to other types of layers.
Figure \ref{fig:mixed} demonstrates the layer-wise integer computations of a linear layer where the shared exponents and integer mantissas are treated separately. As shown in the Figure \ref{fig:mixed}, in order to perform an integer-only GEMM operation, we need to multiply scales $(2^{e_\mathrm{max1}} \times 2^{e_\mathrm{max2}} )$. This multiplication performs an integer addition operation on the exponents $(e_\mathrm{max1}+e_\mathrm{max2})$. Furthermore, the integer mantissas are sent to an integer GEMM module to compute the output mantissa tensor.


\begin{figure}[!t]
    \centering
        \includegraphics[width=0.8\textwidth]{figs/int-computation2.pdf}~~
    \caption{ A fully integer linear layer.
    }
    \label{fig:mixed}
\end{figure}


\subsection{ Understanding the representation mapping}
Here we provide an intuition of how our proposed representation mapping approach works. Let us denote ${A}$ as a tensor and ${\hat{A}}$ as its fixed-point version in the way that is introduced earlier. In addition, random variables $A_i$ and $\hat{A}_i$ are $i^\text{th}$ element of those tensors. Also note that $A_i$ and $\hat{A}_i$ are different {\textit{representations}} of the same real number, but one is in floating-point format and the other is in dynamic fixed-point format. Thus, one can relate $A_i$ and $\hat{A}_i$ with a random error term $\delta_i$ as  $\hat A_i = {A}_i + \delta_i$ . Since in our training method we used stochastic rounding \citep{connolly2021stochastic}, $\mathbb{E}{\{\hat{A}_i\}} = A_i$, or equivalently $\mathbb{E}{\{\delta_i\}} = 0$ (see  Appendix \ref{appendix:stochastic-rounding}). In other words, the fixed-point value is on the average equal to the floating-point value. Note that here we consider single precision floating-point values as a surrogate of real values.


\textbf{Linear and convolutional layers:} For these two types of layers, both forward and backward propagation computations are based on inner products. Thus, it is easy to see that our proposed fixed-point inner product $\hat{C}_{ij}=  \sum_{k} \hat{A}_{ik}\hat{B}_{kj}$ is an unbiased estimator of the floating-point inner product
\begin{equation}
    \mathbb{E} \{\hat{C}_{ij}\}= \mathbb{E} \left\{\sum_k \hat{A}_{ik}\hat{B}_{kj} \right\} = \mathbb{E} \left\{\sum_k ({A}_{ik} + \delta^A_{ik}) ({B}_{kj} + \delta^B_{kj}) \right\} = \sum_k {A}_{ik}{B}_{kj} = {C}_{ij} .\
\end{equation}

\textbf{Residual connections:} A residual connection involves element-wise addition of two tensors. Each fixed-point element $\hat{C}_{ij}=  \hat{A}_{ij} + \hat{B}_{ij}$ is an unbiased estimator of its floating-point version
\begin{equation}
    \mathbb{E} \{\hat{C}_{ij}\}= \mathbb{E} \{ \hat{A}_{ij} + \hat{B}_{ij}\} = \mathbb{E} \{ ({A}_{ij} + \delta^A_{ij})+({B}_{ij} + \delta^B_{ij})\} =  {A}_{ij}+{B}_{ij} = {C}_{ij} .\
\end{equation}

\textbf{Batch-norm:} A batch-norm layer is defined as 
\begin{equation}
    \hat{\omega}\frac{{\hat A} - \hat\mu }{\sqrt{\hat\sigma^2 + \epsilon}} + \hat{\beta}\text{.}\
    \label{eq:batchnorm}
\end{equation}
For this type of layer, it is important to compute signal statistics such as mean $\hat\mu$ and variance $\hat\sigma^2$ correctly in integer arithmetic. Thus, the fixed-point mean $\hat{\mu}$ is also an unbiased estimator of the true mean $\mu$
\begin{equation}
    \mathbb{E} \left\{\hat{\mu} \right\}= \mathbb{E} \left\{ \frac{\sum_{i=1}^N \hat{A}_i}{N} \right\} = \frac{1}{N} \mathbb{E} \left\{ \sum_{i=1}^N ({A}_i+\delta_i) \right\}=\mu.\
\end{equation}
Then the following derivation holds for fixed-point variance $\hat{\sigma}^2$
\begin{equation}
\begin{gathered}
    \mathbb{E} \{{{\hat\sigma}^2}\}= \mathbb{E} \left\{ \frac{\sum_{i=1}^N (\hat{A}_i - \hat \mu)^2}{N} \right\} = 
   
   
    \frac{1}{N} \mathbb{E} \left\{ \sum_{i=1}^N [({A}_i - \mu)^2\ + \delta_i^2] \right\} =  \sigma^2 + \sigma_\delta^2 ,\
\end{gathered}
\end{equation}
where $\sigma^2$ is the true floating-point variance of the batch and $\sigma_\delta^2$ is the variance of the noise introduced by the linear mapping to fixed-point. Note that the error variance $\sigma_\delta^2$ is rather small and can be integrated to $\epsilon$ in the denominator of equation \eqref{eq:batchnorm}.


\section{Theoretical analysis of SGD using the proposed method} \label{sec:sgd-analysis}
The generic equation of weight update in the $k^\text{th}$ iteration of SGD is
\begin{equation}
    {w}_{k+1}= {w}_{k} + \alpha_k g(w_k,\xi_k),\
    \label{eq:sgd}
\end{equation}
where $g(w_k,\xi_k)$ is the estimated gradients of random samples of the batch generated by the seed $\xi_k$, and $\alpha_k$ is the learning rate. We make the following common assumption in the sequel.

\textbf{Assumption 1 (Lipschitz-continuity).} The loss function $\mathcal{L}(w)$ is continuously differentiable and its gradients $\nabla \mathcal{L}(w)$ satisfies the following inequality where $L > 0$ is the Lipchitz constant
\begin{equation}
    \mathcal{L}(w) \leqslant \mathcal{L}(\bar w) + \nabla \mathcal{L}(\bar w)^\top(w-\bar w) + \frac{1}{2}L|| w- \bar w||^2_2; ~~~ ~~~   \forall ~ w,\bar w \in \mathbb{R}^d.\
\end{equation}
\textbf{Assumption 2.} (i) $\mathcal{L}(w_k)$ is bounded. (ii) Estimated gradients $ g(w_k,\xi_k)$ is an unbiased estimator of the true gradients of the loss function $$\nabla \mathcal{L}( w_k)^\top \mathbb{E}_{\xi_k}\{ g(w_k,\xi_k) \} = ||\nabla \mathcal{L}(w_k)||^2_2 = ||\mathbb{E}_{\xi_k}\{ g(w_k,\xi_k)  \}||^2_2,$$ and (iii,a) there exist scalars $M \geqslant 0$ and $M_V \geqslant 0$ such that for all iterations of SGD $\mathbb{V}_{\xi_k} \{g(w_k,\xi_k)\} \leqslant  M + M_V || \nabla \mathcal{L}( w_k)||^2_2 $ .

Note that here we define $$\mathbb{V}_{\xi_k} \{g(w_k,\xi_k)\} := \mathbb{E}_{\xi_k} \{||g(w_k,\xi_k)||^2_2\} - ||\mathbb{E}_{\xi_k} \{g(w_k,\xi_k)\} ||^2_2.$$

Also from \textit{Assumption 2. (ii) and (iii,a)}, the second moment bound can be derived
\begin{equation}
    \mathbb{E}_{\xi_k} \{||g(w_k,\xi_k)||^2_2\} \leqslant  M + M_G || \nabla \mathcal{L}( w_k)||^2_2 \; ~~~~~ \text{with}~~ M_G := 1+M_V.
\end{equation}

\textbf{Effect of gradient variance on convergence:}
The quality of the estimated gradients $g(w_k,\xi_k)$ directly affects the convergence of the SGD. The effect of first and second moments of gradient are already studied on {\emph{ real numbers }} in the literature.


\textbf{Lemma 2.} Suppose \textit{Assumption 2} is true, then we have
\begin{equation}
    \mathbb{E}_{\xi_k}\{\mathcal{L}(w_{k+1})\} - \mathcal{L}( w_k)\leqslant   -(1-\frac{1}{2}\alpha_kLM_G)\alpha_k ||\nabla \mathcal{L}( w_k)||^2_2  + \frac{1}{2} \alpha^2_k LM. \\
\label{eq:sgd-moments}
\end{equation}
\textit{\textbf{Proof:}} See \citet[Lemma 4.4]{bottou2018optimization}.

Inequality \eqref{eq:sgd-moments} shows the effect of gradient variance bounds, $M$ and $M_G$, on each iterate of SGD, and shows the greater the variance, the more deterioration in the quality of SGD steps. 
The first term, $-(1-\frac{1}{2}\alpha_kLM_G)\alpha_k ||\nabla \mathcal{L}( w_k)||^2_2$ contributes to the decrease of the loss function while the second term, $\frac{1}{2} \alpha^2_k LM$, prevents it. Upon choosing the correct gradient estimates, the right hand side of inequality \eqref{eq:sgd-moments} is bounded by a deterministic quantity, and asymptotically ensures sufficient descent of the loss $\mathcal{L}(w)$. Note that the expectation $\mathbb{E}_{\xi_k}$ is taken over random samples with seed $\xi_k$.

\subsection{Fixed-point gradient noise}
When performing representation mapping in the back-propagation, the quality of the gradients deteriorate. Thus, there is a need to consider the effect of fixed-point mapping variance. Then, \textit{Assumption 2 (iii,a)} should be modified accordingly. 

\textbf{Remark 1.} Note that \textit{Assumption 2 (i), (ii)} still hold after fixed-point mapping because of stochastic rounding, i.e.  the fixed-point gradient remains an unbiased estimator of gradient (refer to Appendix \ref{appendix:stochastic-rounding}).

\textbf{Assumption 2 (iii,b).} When the gradients are in fixed-point format i.e. $\hat{g}(w_k,\xi_k)$, there exist scalars $M \geqslant 0$ , $M_V \geqslant 0$, $M^q \geqslant 0$ and  $M^q_V \geqslant 0$ such that for all iterations of SGD $$\mathbb{V}_{\xi_k} \{\hat{g}(w_k,\xi_k)\} \leqslant  M + M^q + (M_V+ M^q_V )|| \nabla \mathcal{L}( w_k)||^2_2 .$$ 

If $\Tilde M := M + M^q$ and $\Tilde M_V := M_V+ M^q_V $ , then \textit{Assumption 2 (iii,b)} takes the exact form of \textit{Assumption 2 (iii,a)} i.e. $\mathbb{V}_{\xi_k} \{g(w_k,\xi_k)\} \leqslant  \Tilde M + \Tilde M_V || \nabla \mathcal{L}( w_k)||^2_2 $. However, here we separated $M^q$ and $M_V^q$ to emphasize the effect of fixed-point mapping on the true gradients.

\textbf{Remark 2.} If \textit{Assumption 2 (iii,b)} holds true, inequality \eqref{eq:sgd-moments} can be transformed to its {fixed-point} version
\begin{equation}
\begin{gathered}
    \mathbb{E}_{\xi_k}\{\mathcal{L}(w_{k+1})\} - \mathcal{L}( w_k)\leqslant   -(1-\frac{1}{2}\alpha_kL(M_G+M^q_G))\alpha_k ||\nabla \mathcal{L}(w_k)||^2_2  + \frac{1}{2} \alpha^2_k L(M+M^q) \\
    \text{with}~~ M^q_G := 1+M^q_V.
\end{gathered}
\label{eq:sgd-moments-q}
\end{equation}
Inequality \eqref{eq:sgd-moments-q} shows the effect of \textit{\textbf{added}} representation mapping variance with bounds, $M^q$ and $M^q_G$, on each iterate of SGD. This observation shows that fixed-point mapping degrades the convergence of SGD unless its variance bounds are relatively small, or controlled by the learning rate.  
As an example, refer to Appendix \ref{appendix:lin-var-q} for analytical derivations of $M^q$ and $M^q_G$ for the back-propagation of a linear layer involving a fixed-point inner product.

\subsection{Strongly convex and locally convex loss} \label{sec:theorem}
\textbf{Assumption 3 (Strong convexity).} The loss function $\mathcal{L}(w)$ is differentiable and strongly convex.  We recall that strong convexity for differentiable functions is equivalent to the following inequality with constant $c > 0$
\begin{equation}
    \mathcal{L}(w) \geqslant \mathcal{L}(\bar w) + \nabla \mathcal{L}(\bar w)^\top(w-\bar w) + \frac{1}{2}c|| w- \bar w||^2_2; ~~~ ~~~   \forall ~ w,\bar w \in \mathbb{R}^d.\
    \label{eq:convex-1}
\end{equation}
A strongly convex function has a unique minimum point at $w_*$ with the loss value $\mathcal{L}_* = \mathcal{L}(w_*)$.

\textbf{Theorem 1.} Suppose \textit{Assumptions 1, 2(i), 2(ii), 2 (iii,b), 3} are all true, then a SGD method running with fixed-point gradients i.e. $\hat g(w_k,\xi_k)$ and a fixed learning rate $0<\bar\alpha \leqslant \frac{1}{L(M+M^q)}$ satisfies the following bound for its optimality gap with the minimum loss $\mathcal{L}_*$ at the $k^\text{th}$ iteration
\begin{align}
    \mathbb{E}\{\mathcal{L}(w_{k}) - \mathcal{L_*} \}
    \leqslant& \frac{\bar\alpha L(M+M^q)}{2c}  + (1-\bar \alpha c)^{(k-1)} \left(\mathcal{L}(w_1)-\mathcal{L}_*-\frac{\bar \alpha L(M+M^q)}{2c} \right) \nonumber\\
    \xrightarrow{k\xrightarrow{}\infty} &\frac{\bar\alpha L(M+M^q)}{2c}.\
\label{eq:sgd-convex}
\end{align}
\textit{\textbf{Proof.}} See Appendix \ref{appendix:proof-sgd-q}.

\textbf{Remark 3. }Note that when ${k\xrightarrow{}\infty}$, $\frac{\bar\alpha L(M)}{2c}$ is the original optimality gap  (i.e. $\mathbb{E}\{\mathcal{L}(w_{k}) - \mathcal{L_*} \}$) with \textit{real} gradients, see \citet[Theorem 4.6]{bottou2018optimization}, and then, the optimality gap is also increased by $\frac{\bar\alpha L(M^q)}{2c}$ due to fixed-point representation. Here we argue that by keeping the variance bound $M^q$ relatively small, we can theoretically achieve the original performance. On the other hand, optimality gap is related to $\bar\alpha$, which means smaller learning rates leads to smaller optimality gap.

\textbf{Remark 4 (Local convexity). }Strongly convex loss is not a realistic assumption in large deep learning models. However, local convexity is a more realistic assumption i.e. $\mathcal{L}$ is convex around the minimum point ${w}_*$. Hence, inequality \eqref{eq:sgd-convex} still holds around the minimum point ${w}_*$ of a locally convex loss.
We essentially train our ImageNet ResNet18 classification in the same way: first we train the network with a fixed learning rate until reaching a certain optimality gap; then the learning rate is reduced to a smaller fixed value in order to shrink the optimality gap.

\begin{figure}[ht]
    \centering
        \includegraphics[width=0.30\textwidth]{figs/3dsurface_fp.pdf}~~
        \includegraphics[width=0.30\textwidth]{figs/3dsurface_dfx.pdf}~~
        \includegraphics[width=0.30\textwidth]{figs/loss.pdf}\\
        \small{(a)}\hspace{4cm}\small{(b)} \hspace{4cm}\small{(c)}\\

    \caption{ (a) Floating-point loss landscape, (b) Fixed-point \texttt{int8} loss landscape c) Training loss trajectory comparison. 
    \textbf{Integer training setup:} \texttt{int8} linear layer, \texttt{int8} convolutional layer and \texttt{int8} batch-norm layer.
    }
    \label{fig:loss}
   
\end{figure}

\textbf{Empirical evidence:} Figure \ref{fig:loss}(a) demonstrates the locally convex loss landscape of ResNet18 training on CIFAR10 dataset using floating-point computation. Figure \ref{fig:loss}(b) shows the same loss landscape in the fixed-point \texttt{int8} format. We perturbed the weights around the ${w_*}$ in $x$ and $y$  axes using Gaussian noise and evaluated the loss $\mathcal{L}$ in the $z$ axis to acquire Figures \ref{fig:loss}(a) and \ref{fig:loss}(b). Comparing these two figures pronounces our assumption of local convexity in both integer and floating-point tests.  Figure \ref{fig:loss}(c) shows a comparison of the loss trajectory for floating-point and integer training. The fixed-point gradients are unbiased estimators of the true gradients, so the trajectory of the integer training closely follows the trajectory of its floating-point counterpart.

\textbf{Remark 5.}
We implemented an integer weight update, hence, the computations of equation \eqref{eq:sgd} is also performed in integer arithmetic. It is shown in Appendix \ref{appendix:wu} that integer weight update with stochastic rounding is an unbiased estimator of the true weight update.




\section{Experimental results} \label{sec:results}

\textbf{Image classification:} 
Table \ref{tab:classification} reports the experimental results of our proposed integer training method on the conventional vision classification models. In this set of models, we used \texttt{int8} linear layer, \texttt{int8} convolutional layer, \texttt{int8} batch-norm layer, and \texttt{int16} SGD to form a \textit{fully integer training} pipeline. Note that all forward and back-propagation computations of layers are performed in \textit{integer} format. Moreover, we have chosen smaller vision models such as ResNet18 and MobileNetV2 knowing that smaller models are harder to train. Our experimental results show that our proposed integer training method is as good as floating-point with negligible loss of accuracy  ($\leqslant 0.5\%$).

\textbf{Remark 6.} We compare our integer training results with official Pytorch floating-point training. For instance, in conventional vision models  (Table \ref{tab:classification}), we compare the results directly with the accuracy of \textit{torchvision} models that are reported on the official Pytorch website \footnote{Refer to Pytorch Models: \url{https://pytorch.org/vision/stable/models.html}}. We further emphasize that there is no change in hyper-parameters of the training. The full set of hyper-parameters of our experiments are reported in Appendix \ref{appendix:hyper}.  We did not compare our results with quantized back-propagation literature such as \citep{banner2018scalable, zhang2020fixed,zhao2021distribution,zhu2020towards} for two reasons: (i) they often chose to use \textit{mixed-precision} design where some parts of the model such as batch-norm and SGD are in floating-point (ii) they have changed hyper-parameters or used gradient clipping which is not required for our proposed integer training method.


\textbf{Vision transformer:}
We validated the applicability of our proposed integer training on the original vision transformer model, notably ViT-B-16-224 \citep{dosovitskiy2020image}. We took the floating point checkpoint pretrained on ImageNet21K\footnote{ Refer to: \url{https://huggingface.co/google/vit-base-patch16-224}}. We used Huggingface \citep{wolf2019huggingface} to  fine-tune the model on CIFAR10. In this experiment, we used \texttt{int8} linear layer, \texttt{int8} matrix multiplication, \texttt{int8} convolutional layer, and \texttt{int8} layer-norm for our integer training pipeline. The result is reported in Table \ref{tab:classification} demonstrating negligible loss of accuracy ($\leqslant 0.5\%$). 

\textbf{Semantic segmentation:}
To validate our proposed integer training on semantic segmentation task, we used DeepLabV1/V2\footnote{ Refer to : \url{https://github.com/kazuto1011/deeplab-pytorch}} with ResNet-101 as the backbone. We trained the models on PASCAL VOC-2012 \citep{voc} and MS COCO 10K \citep{coco}. For the PASCAL dataset, as suggested by \citet{chen2017deeplab}, the models were initialized with MS COCO checkpoint and data augmentation was used during training. Likewise, for COCO dataset we used PASCAL checkpoint for initialization. The batch-norm layers are frozen in our experiments as suggested in \citep{chen2017deeplab}. We used linear and convolutional layers in \texttt{int8} format. Table \ref{tab:segmentation} shows that the mean intersect of union (mIOU) of our method closely matches the floating-point baseline.

\textbf{Object detection:}
We used Faster R-CNN\citep{fasterrcnn} with ResNet50 as its backbone and Single Shot Detector (SSD)\citep{ssd} which relies on VGG-16 for its backbone. In these experiments, we used \texttt{int8} linear and convolutional layers, while batch-norm layers are frozen. We integrated our integer training method with MMDetection\citep{mmdetection} toolbox and performed our experiments on MS COCO 10K\citep{coco}, PASCAL VOC-2007\citep{voc07}, VOC-2012\citep{voc}, and Cityscapes\citep{cityscapes} datasets. As shown in Table \ref{tab:detection}, the mean average precision (mAP) of our proposed integer training method is very close to floating-point baseline.

\begin{table}[!t]

  \caption{Classification}
  \label{tab:classification}
  \centering
  \begin{tabular}{llll}
    \toprule
    &&\multicolumn{2}{c}{Accuracy}                 \\
    \cmidrule(r){3-4} 
    Model     & Dataset     & \makecell{ \small{\texttt{int8}} \\ \small{\texttt{(top1 - top5)}}} & \makecell{\small{\texttt{Pytorch baseline float}}\\ \small{\texttt{(top1 - top5)}}} \\
    \midrule
    \multicolumn{2}{c}{Conventional Vision Models}\\
    \cmidrule(r){1-2}
    \textbf{ResNet18} & \small{CIFAR10}  &  \makecell{94.84 - \small{~N/A~}}   & \makecell{95.34 - \small{~N/A~}} \\
    \textbf{ResNet18 }    & \small{ImageNet}       & \makecell{69.25 - 88.79} & \makecell{69.75 - 89.07}\\
    \textbf{MobileNetV2}     & \small{ImageNet}       &  \makecell{72.80 - 90.83}  & \makecell{71.87 - 90.28}\\
    \midrule
    \multicolumn{2}{c}{Vision Transformer}\\
    \cmidrule(r){1-2}
    \textbf{ViT-B}~\small{(fine-tuning)}     & \small{CIFAR10 }&   \makecell{98.8 - \small{N/A} }   &  \makecell{99.1 - \small{N/A}}\\
    \bottomrule
  \end{tabular}
    
\end{table}
\begin{table}[!t]
\begin{minipage}{.49\linewidth}
  \caption{Semantic Segmentation}
  \label{tab:segmentation}
  \centering
  \scalebox{0.9}{
  \begin{tabular}{lcccc}
        \toprule
        &&\multicolumn{2}{c}{mIOU}                   \\
        \cmidrule(r){3-4}
        Method & Dataset & \texttt{int8} & \makecell{\texttt{baseline}\\ \texttt{float} } \\
        \toprule
        \textbf{DeepLab-V1} & \small{VOC} &  74.73 & 75.0 \\
        \textbf{DeepLab-V1} & \small{COCO} &  34.8 & 34.7 \\
        \textbf{DeepLab-V2} & \small{VOC} & 77.65 & 77.71 \\
        \textbf{DeepLab-V2} & \small{COCO} &  35.90 & 36.0\\
     
    \bottomrule
 \end{tabular}
 }
 \end{minipage}
 \begin{minipage}{.45\linewidth}
  \caption{Object Detection}
  \label{tab:detection}
  \centering
  \scalebox{0.9}{
  \begin{tabular}{lcccc}
        \toprule
        &&\multicolumn{2}{c}{mAP}                   \\
        \cmidrule(r){3-4}
        Method & Dataset & \texttt{int8} & \makecell{\texttt{baseline}\\ \texttt{float}}  \\
        \toprule
        \textbf{Faster R-CNN} & \small{COCO} & 37.4  & 37.8 \\
        \textbf{Faster R-CNN} & \small{VOC07+12} & 80.14 & 80.31 \\
        \textbf{Faster R-CNN} & \small{Cityscapes} & 39.50 & 40.00 \\
        \textbf{SSD} & \small{COCO} & 42.5 & 43.6 \\

    
    \bottomrule
 \end{tabular}
 }
 \end{minipage}
 \vspace{-5mm}
\end{table}


\section{Conclusion}
We proposed a hardware-friendly integer training method based on coupling a linear fixed-point mapping with a non-linear inverse mapping. This method performs the representation mapping directly on the floating-point number format. Furthermore, our method uses stochastic rounding and produces unbiased estimators of gradients in the back-propagation. Thus, benefiting from unbiased gradients and effective representation mapping, the training loss trajectory closely follows its floating-point version. Moreover, there is no need to tune hyper-parameters or perform gradient clipping to correct the inappropriate gradient and weight updates. Using our proposed technique, we designed the integer version of the most vital components of deep learning such as linear layer, convolutional layer, batch-norm, layer-norm, and stochastic gradient descent (SGD). Our experimental results show the effectiveness of the proposed method, where the accuracy loss is negligible in a wide variety of the training tasks. Furthermore, in our classification tests all training components are in integer arithmetic. We theoretically studied the effect of our proposed method on SGD, and demonstrated why the optimality gap of convergence is shifted by a negligible amount.





\begin{ack}
The authors would like to thank Richard Wu and Vanessa Courville for their constructive comments. The authors would like to thank Sara Elkerdawy for initiating the object detection and semantic segmentation experiments during her internship at Huawei.
\end{ack}


\medskip
\small{
\bibliographystyle{unsrtnat}
