\section{Introduction


	Convolutional neural networks (CNNs) have become the leading content of various computer vision tasks, making marvelous progress in the fields of image classification \cite{He2016, Szegedy2015}, object detection \cite{Ren2015, Girshick2014}, image segmentation \cite{Chen2018, Long2015}, object tracking \cite{Bewley2016, Wojke2017}, 3D reconstruction \cite{tran2017}. Supported by a massive amount of data and excellent GPU computing capability, more and more heavy and complex deep models are proposed, such as VGGNet \cite{Simonyan2015}, ResNet \cite{He2016}, GoogLeNet \cite{Szegedy2015} and DenseNet \cite{Huang2017}.

	However, with the improvement of large model performance, there are huge parameter scale, terrible computing overhead and memory requirements. This becomes the challenge for edge and mobile embedded devices. To be specific, when deploying high-precision model on edge devices, there are three main problems remained to be tackled: 

\begin{itemize}

\item A satisfactory CNN model often has millions of weight parameters, which will consume disk storage and then be loaded into run-time memory during inference.

\item The feature maps cache data generated in the running process may even occupy more memory than the original model parameters.

\item Convolution operation is a kind of intensive computing operation. For example, in VGG-16, one time calculation computing operations required for one convolution layer can reach $1.85 \times 10^9$ times on a image with resolution $224 \times 224$. Convolution operation is the main computing burden.

\end{itemize}

	How to compress and accelerate the neural model has been a high profile in recent years. At present, the typical works include network quantization \cite{Guo2017, Hu2018}, low-rank approximation \cite{Masana2017}, weight sharing \cite{Wang2017, Chen2016, Son2018}, weight pruning \cite{Lee2019, Yang2019}. However, these methods need specific hardware or software library to run, and some of them can not solve the problems described above comprehensively. There is another lightweight technique defined as filter pruning \cite{Li2017, Lin2020, He2017, He2018, Lin2019, Zhao2019}, also known as coarse-grained pruning. By comparison, filter pruning is not restricted to special hardware or software, and suitable as well as universal for CNNs in various types of tasks. 

\begin{figure*}[t]
\begin{center}
   \includegraphics[width=0.7\linewidth]{graph//x.pdf}
\end{center}
   \caption{The left of the image is the output feature maps of the first convolution layer in the unpruned VGG-16 model, and it can be found that there are many similar feature maps. The right of the image is the pruning result of QSFM . Deleting similar information is like obtaining maximal linearly independent system in a two-dimensional matrix.}
\label{fig:long}
\label{fig:onecol}
\label{fig:1}
\end{figure*}

	Filter pruning has been shown to be significant in network slimming \cite{Frankle2019, Malach2020}. On filter pruning, recent works have made great progress. \cite{He2018, Lin2019, Lin2020a} put forward specific optimization objectives and constraints, and utilize heuristic optimization algorithm to train CNNs jointly. These will bring a new set of hyper parameter problems. That means, aiming to deploy strategies, extra tricks to further adjust hyper parameters of heuristic algorithm is needed, which is not flexible for practical application. Moreover, jointed trainning frequently combined with sparsity reguarization penalty, which will destroys the structure of the model itself. In \cite{Huang2018, He2017, Zhao2019, Liu2017, Yu2018, Lin2020, Li2017}, filters are pruned by concrete regulations, which are explicit in generating filter importance. These methods needs sort the importance from high to low, and then delete the low importance filters. It doesn’t take much manual adjustment involved, which means they are robust in varied situations. However, \cite{Yu2018, He2017} originated in experience or intuition, lacking of basic theoretical guidance, though they do success in compressing model. Meanwhile, other rule-based criterion provides sufficient theoretical support to find out this redundant information, but still remain defects. \cite{Li2017} applied the L1 norm to filter as the Evaluation criteria of importance. However, all layers use the same trim scale, which brings suboptimization.  	      \cite{Lin2020} propose a method named Hrank to prune filters with low-rank feature maps. Similar high-rank feature maps may also contain redundant information, but they have not been removed.

	Our work belongs to filter pruning. This idea is inspired by similarity between pictures.The high level of similarity indicates that the feature information contained in the two images is basically the same. The convolutional neural layer produces abundant multi-dimensional tensors, which can be regarded as pictures. In other words, We can apply the idea of similarity to feature maps. Our method first need to generate the feature maps for each convolution layer from training dataset and quantify the similarity between these feature maps. Then we prune convolution filters, layer by layer, based on the corresponding channel feature maps which have the most highest level in similarity. After the filter pruning, the performance of the slimmed model will be promoted through fine-tuning, without obvious loss of accuracy. The similarity of feature maps has been explained and applied in GhostNet \cite{Han2020}, which further proves the feasibility of our method. The linchpin of our method is to opt quantization function, which will be amply discussed in  Sec.\pageref{section3}. It should be emphasized that we have not implement sparsity reguarization penalty in model training. 

	Our main contributions are as follows:

\begin{itemize}

\item Inspired by maximal linearly independent system in the two-dimensional matrices, we propose a method to find out the redundant information in the three-dimensional tensors by quantifying the similarity of any two feature maps, namely QSFM.  

\item QSFM is applicable to all kinds of convolution, including common convolution and depthwise separable convolution. We also use different similarity quantization functions to further explore QSFM.  

\item We do various experiments on CIFAR-10 and CIFAR-100, using VGGNet, MobileNet and ResNet. which shows superior performance of QSFM over existing prior filter pruning methods.

\end{itemize}

\section{Related Work}


	\textbf{Network Quantization}. \cite{Guo2017} proposed the network sketching method, which uses the convolution with binary weight sharing: for convolution calculation with the same input, the result of the previous convolution is retained, and the same part of the convolution filters directly multiplex the result. \cite{Hu2018} projected data into Hamming space by hashing, and transformed the problem of learning binary parameters into a hashing problem under inner product similarity. Different from the traditional 1-Valued or weighted mean, \cite{Zhu2017} proposed trained ternal quantification (TTQ), which uses two trainable full precision scaling coefficients to quantify the weight to $ \{-w_n,0,w_p \}$, and asymmetric weights make the network more flexible. Unlike our approach, these methods aim to quantify the weight of filters into discrete values, while our approach is to find out a quantization function as specific description for similarity between feature maps. These methods obtain a very high level of compression in saving storage space and significantly accelerate inference, but it is extreme, and will bring a considerable loss of accuracy, damaging the model performance. Additionally, the value of the possible weight network needs to be further explored. 

	\textbf{Weight pruning}. Weight pruning can remove any redundant parameter or redundant connection of the expected proportion of the network without restriction, but it will bring the problem of irregular network structure, making it difficult to effectively accelerate after pruning. Recently, \cite{Lee2019} proposed SNIP method that the importance of the connection was determined by sampling the training set several times in the initialization phase of the model, and the pruning template was generated in the meantime. After training, there is no need to iteratively prune and fine-tune by the alternating cycle process. \cite{Yang2019} used weighted sparse projection and input masking to provide quantifiable energy consumption, taking energy consumption budget as the optimization constraint of network training, and the sparse network can be obtained by utilizing dynamic pruning method which can recover the important connections removed by mistake. Due to requirement of special sparse matrix operation library and hardware, it is also not convenient and universal to use weight pruning to implement acceleration for model.

	\textbf{Filter pruing}. Also named as coarse-grained pruning, it considers filters as the minimum prunning units, which can make the network "narrow" , and can directly achieve effective acceleration on existing software or hardware. Compared with human-crafted pruning, \cite{He2018} made the model compression completely automatic and perform better. It utilized DDPG (Deep Deterministic Policy Gradient) as the controller to generate the specific compression ratio in continuous space. Different from the previous hard pruning and label dependent pruning methods, \cite{Lin2019} proposed a label free Generative Adversarial Learning (GAL) method, which uses sparse soft mask pruning network to scale the output of specific structure to zero. It learns pruning networks with sparse soft masks in an end-to-end manner. Due to heuristic optimization algorithm and jointed training, \cite{He2018, Lin2019} will bring a new set of hyper parameter problems. \cite{He2017} propose an iterative two-step algorithm to effectively prune each layer, by a LASSO regression based channel selection and least square reconstruction. \cite{Zhao2019} proposed a variational Bayesian scheme for pruning convolutional neural networks in channel level, introducing a stochastic variational inference to estimate the distribution of channel saliency induced by a sparse prior. Though successful in compressing model, \cite{He2017, Zhao2019} originate in experience or intuition, lacking of basic theoretical guidance. \cite{Li2017} calculated the L1 norm of the filter, cutting out the feature map corresponding to the smaller L1 norm, and retrain after pruning. However, all layers use the same trim scale, which brings suboptimization. \cite{Lin2020} proposed an effective and efficient filter pruning approach that explores the High Rank (HRank) of the feature map in each layer. The principle behind HRank is that low-rank feature maps contain less information, and thus pruned results can be easily reproduced. Nevertheless, similar high-rank feature maps may also contain redundant information, but they have not been removed.

	Our method also belongs to filter pruning. This pruning category can get rid of the limitation of special software library or hardware, achieving the promotion of storage usage, memory occupation and computing speed at the same time. And it retains the model structure with different pruning rate. It can also be combined with other typical categorise mentioned above to further achieve higher compression. 
\section{Our Approach} \label{section3}

\subsection{Find the Redundant Information}

	In mathematics, for a two-dimensional matrix (that is, a set of vectors), all of its vectors can be represented as linear combinations of vectors in its maximal linearly independent system and its rank is the number of vectors contained in the maximal linearly independent system. Assume $A = [\alpha_1,\alpha_2,..., \alpha_n] \in R^{m \times n}$, $\alpha_i \in R^m$, $i = 1, 2, ..., n$. if the maximal linearly independent system of $A$ is $A_{Basic} \in R^{m \times s}$, $s \leq n$, we know that any vector in $A$ can be generated by $A_{Basic}$. To some extent, the maximal linearly independent system of two dimensional vectors represents its most essential information.

	We extend this concept of maximal linearly independent system to three dimensional tensors. Considering that three dimensional tensors are more complex and the condition of linear dependence is not easy to be satisfied in three dimensions, we replace the concept of linear dependence of multiple vectors with the similarity of two tensors. Assume the tensor $B = [\beta_1,\beta_2,...,\beta_k] \in R^{m \times n \times k}$, $\beta_i \in R^{m \times n}$, $i = 1, 2, ..., k$. If $\beta_i$ is similar to $\beta_j$, then $B$ has redundant information. How to define the similarity between $\beta_i$ and $\beta_j$ becomes the key of the problem. Obviously, we can define $\beta_i$ to be similar to $\beta_j$ when $\beta_i = k \cdot \beta_j$ ($q$ is a constant) just like two-dimensional matrix. But in practice, such a definition is too strict, and it is often difficult to find $\beta_i$ and $\beta_j$ that meet such conditions. 

	We define the quantization function $S(\beta_i, \beta_j)$, and the larger the value of $S(\beta_i, \beta_j)$, the more similar $\beta_i$ and $\beta_j$ are. In practice, we use Structural Similarity (SSIM) \cite{Wang2004} and Peak Signal to Noise Ratio (PSNR) , equivalent to Euclidean distance) as quantization functions $S(\beta_i, \beta_j)$. Our task is to find out the most essential information in the three-dimensional tensor through the similarity function, similar to the Maximal Linearly Independent System in the two-dimensional matrix.

\subsection{QSFM}


\subsubsection{Notations}

	Assume an image dataset $Train$ has $M$ images, denoted as $Train = \{Image_1, Image_2, ..., Image_k\} \in R^{M \times N_0 \times X_0 \times Y_0}$, where $N_0$ represents the channel number, $X_0$, $Y_0$ are the height and width. The pre-trained model before pruning is $Model_0$, the Model obtained after the $i$th pruning operation is $Model_i$ the convolution layers of the model are $L_1, L_2, ..., L_n$, $L_i$ contains $N_i$ filters, and the set constituted by $N_i$ filters is $W_i = \{Filter_{(i, 1)}, Filter_{(i, 2)}, ..., Filter_{(i, N_i)}\} \in R^{N_i \times N_{i-1} \times K_i \times K_i}$, where $K_i$ represents the height and width of the convolution filter. Assume the model input is only one image, then the input of $L_i$ is $I_i \in R^{N_{i-1} \times X_{i-1} \times Y_{i-1}}$ and the output of $L_i$ is $O_i \in R^{N_i \times X_i \times Y_i}$. For the $j$th $Filter_{(i, j)} \in R^{N_{i-1} \times K_i \times K_i}$ of layer $L_i$, its output is $Feature\_map_{(i,j)} \in R^{X_i \times Y_i}$, and these $N_i$ Feature maps constitute the set $O_i = \{Feature\_map_{(i,1)}, Feature\_map_{(i,2)}, ...$, $Feature\_map_{(i,N_i)}\} \in R^{N_i \times X_i \times Y_i}$.  For the layer $L_i$, we assume $Filters$ which should be pruned constitute the set $Delete^i$. After identifying the redundant $Feature\_maps$, you can prune them according to their corresponding $Filters$, as shown in Fig.~\ref{fig:1}.

\subsubsection{Quantify similarity}

	The sequence of pruning is from $L_1$ to $L_n$, and it is assumed that the convolution layer of $L_i$ is being pruned.

	First, assume the number of filters to be pruned in layer $L_i$ is $N_{i2}$ according to the compression rate, that is, there are $N_{i2}$ elements in set $Delete^i$.

	The Model to be pruned is $Model_{i-1}$, when the $Image_k$ is input into the $Model_{i-1}$, the output of $L_i$ is $O_i^k = \{Feature\_map_{(i,1)}^k, Feature\_map_{(i,2)}^k, ...$, $Feature\_map_{(i,N_i)}^k\} $. Calculate the quantified similarity function $S(Feature\_map_{(i,m)}^k, Feature\_map_{(i,n)}^k)$ where $Feature\_map_{(i,m)}^k$ and $Feature\_map_{(i,m)}^k$ ($m\neq n$) are any two elements in the set $O_i^k$.  Hereinafter, referred to as $S_{m, n}^{i, k}$, function $S$ takes Structural Similarity (SSIM) or Peak Signal to Noise Ratio (PSNR). 

	Structural Similarity and Peak Signal to Noise Ratio are usually used to measure the image quality after compression. As far as we know, this is the first time that these methods are used to measure the similarity between two feature maps in CNNs.




\begin{small}
\begin{equation}
\begin{aligned}
\begin{array}{l}
S_{m,n}^{i,k}(SSIM)\\
\\ \vspace{1ex}
{\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt}  \vspace{1ex}= SSIM(Feature\_map_{(i,m)}^k,Feature\_map_{(i,n)}^k)\\
{\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt}  = \dfrac{{\displaystyle (2{\mu _m}{\mu _n} + k_1^2{D^2})(2{\sigma _{mn}} + k_2^2{D^2})}}{{\displaystyle (\mu _m^2 + \mu _n^2 + k_1^2{D^2})(\sigma _m^2 + \sigma _n^2 + k_2^2{D^2})}}.
\end{array}
\end{aligned}
\end{equation}
\end{small}



	Here $\mu_m$ and $\mu_n$ are the average of all pixels in $Feature\_map_{(i,m)}^k$ and $Feature\_map_{(i,n)}^k$. $\sigma _m^2$ and $\sigma _n^2$ are the variance of all pixels in $Feature\_map_{(i,m)}^k$ and $Feature\_map_{(i,n)}^k$. $\sigma _{mn}$ is the covariance of $Feature\_map_{(i,m)}^k$ and $Feature\_map_{(i,n)}^k$. Usually $k_1$ is 0.01 and $k_2$ is 0.03. $D$ is determined by the following formula:

\begin{small}
\begin{equation}
\begin{aligned}
D = \max (O_i^k) - \min (O_i^k).
\end{aligned}
\end{equation}
\end{small}

	Here $\max (O_i^k)$ and $\min (O_i^k)$ are the values of the largest and smallest pixels in $(O_i^k)$.

	Since the quantization of PSNR is theoretically equivalent to the Euclidean distance, and we only need to sort the similarity according to the size of the similarity, needless to know the specific value of PSNR. Therefore, the following content does not distinguish the PSNR from the Euclidean distance.

\begin{small}
\begin{equation}
\begin{aligned}
\begin{array}{l}
S_{m,n}^{i,k}(PSNR)\\
\\ \vspace{1ex}
{\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt}  \vspace{1ex}= EuclideanDistance(FM_{(i,m)}^k,FM_{(i,n)}^k)\\
{\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt}  = \sum\limits_{x = 1}^{{X_i}} {\sum\limits_{y = 1}^{{Y_i}} {{{(FM_{(i,m)}^k[x,y] - FM_{(i,n)}^k[x,y])}^2}} }.
\end{array}
\end{aligned}
\end{equation}
\end{small}



	$FM_{(i,m)}^k$ and $FM_{(i,n)}^k$ denote $Feature\_map_{(i,m)}^k$ and $Feature\_map_{(i,n)}^k$. It is noted that $S_{m, n}^{i, k}$ is only the result generated by a single Image ($Image_k$). For the whole data set Train, it contains $M$ images in total. We calculate the statistical average results on $M$ images, and define: 

\begin{small}
\begin{equation}
\begin{aligned}
\begin{array}{l}
S_{m,n}^i = \dfrac{{\displaystyle \sum\limits_{k = 1}^M {S_{m,n}^{i,k}} }}{\displaystyle M},\\ \\ \vspace{1ex}{S^i} = \{ S_{m,n}^i\} ,(m,n = 1,2,3,...,{N_i}).
\end{array}
\end{aligned}
\end{equation}
\end{small}

	The larger $S_{m,n}^i$ is, the more similar is $Feature\_map_{(i,m)}^k$ to $Feature\_map_{(i,n)}^k$. After finding out the feature map groups with high similarity, we need to stipulate an auxiliary condition to determine which of the two feature maps to delete. We can randomly delete a feature map in each similar group, or we can use the L1 norm of the feature map to delete those with a smaller norm, and this part of the experiment is provided in the supplementary material. We use the rank of the two-dimensional matrix as auxiliary condition to assist the quantified similarity function finding the $Delete^i$. The use of this auxiliary condition is inspired by HRank \cite{Lin2020}, but it does not mean that QSFM is similar to that of HRank. In fact, as mentioned above, HRank does not take into account the existence of redundant information (i.e. similarity) in two high-rank feature maps. Meanwhile, our experiment will prove that QSFM is superior to HRank. Calculate the rank of each $Feature\_map_{(i,m)}^k$, then  the statistical average of the Rank of the $Feature\_map_{(i,m)}^k$ in $O_i$ is:

\begin{small}
\begin{equation}
\begin{aligned}
\begin{array}{l}
Rank_m^i = \dfrac{{\displaystyle\sum\limits_{k = 1}^M {Rank_m^{i,k}} }}{\displaystyle M},\\
\\ \vspace{1ex} Ran{k^i} = \{ Rank_m^i\} ,(m = 1,2,3,...,{N_i})
\end{array}
\end{aligned}
\end{equation}
\end{small}

\subsubsection{Filter Pruning}
	
\begin{figure}[t]
\begin{center}
   \includegraphics[width=0.8\linewidth]{graph//Y.pdf}
\end{center}
   \caption{Whatever the convolution process (common convolution or depthwise separable convolution) is, QSFM can prune them to obtain maximal linearly independent system in three-dimensional tensors.}
\label{fig:long}
\label{fig:onecol}
\label{fig:2}
\end{figure}

	Arrange the set $S^i$ in order of highest to lowest, and then match the set, $Rank^i$ to determine the Filter to be pruned. Set the conditions as follows:

	Condition 1: $Filter_{(i,m)}$ and $Filter_{(i,n)}$ are not members of $Delete^i$;
	
	Condition 2: $S_{m,n}^i$ is maximum under the premise that $m$ and $n$ meet condition 1;
	
	Condition 3: $Rank_m^i \textgreater Rank_n^i$;

	Condition 4: $Rank_m^i \leq Rank_n^i$;

	If conditions 1, 2 and 3 are met simultaneously, then $Filter_{(i,n)}$ will be put into $Delete^i$; if conditions 1, 2 and 4 are met simultaneously, then $Filter_{(i,m)}$ will will be put into $Delete^i$. And the above operation will continue until the number of filters in the set $Delete^i$ is $N_{i2}$.

	The model should be fine-tuned after pruning every convolution layer, that is:

\begin{small}
\begin{equation}
\begin{aligned}
Model_i = Fine\_Tuned(Mode{l_{i - 1}} - Delet{e^i})
\end{aligned}
\end{equation}
\end{small}

	Then it enters the pruning of the next convolutional layer until all the convolutional layers are pruned, and the compression of the neural network is completed.


\subsubsection{Special Convolution}

\begin{figure}[t]
\begin{center}
   \includegraphics[width=0.9\linewidth]{graph//B.pdf}
\end{center}
   \caption{The pruning strategy of ResNet-56.}
\label{fig:long}
\label{fig:onecol}
\label{fig:3}
\end{figure}


\begin{figure}[t]
\begin{center}
   \includegraphics[width=0.9\linewidth]{graph//C.pdf}
\end{center}
   \caption{The pruning strategy of MobileNetV2.}
\label{fig:long}
\label{fig:onecol}
\label{fig:4}
\end{figure}

	Our method can also prune special convolution layers, such as depthwise separable convolution. As shown in Fig.~\ref{fig:2}, no matter what kind of convolution layer output, there will be similar feature maps. We can also pruning the network by using the QSFM.

\section{Experiments}

	In our experiments, we use Euclidean distance (PSNR) and Structural Similarity (SSIM) as the function of quantifying the similarity for QSFM, which are denoted as QSFM-PSNR and QSFM-SSIM respectively. We first demonstrated the excellent results obtained by QSFM pruning without any fine-tuning operations in ResNet-56, which is the best as far as we know. We also show the pruning results of QSFM for VGG-16, ResNet-56 and MobileNetV2 \cite{Sandler2018} on the CIFAR-10 \cite{CIFAR10} with fine-tuning operation. Furthermore, we show our results of QSFM for ResNet-56 on CIFAR-100 \cite{CIFAR10}. The original precisions of our model are 93.39\% (VGG-16,CIFAR-10), 93.21\% (ResNet-56, CIFAR-10), 92.54\%( MobileNetV2, CIFAR-10) and 70.62\%( ResNet-56, CIFAR-100).


\subsection{Experimental Settings}

\subsubsection{Configurations}

\begin{figure}[t]
\begin{center}
   \includegraphics[width=1.0\linewidth]{graph//D.pdf}
\end{center}
   \caption{The histogram coordinates represent the proportion of ‘unimportant filters’(filters whose all parameter values are close to $10^{-32}$) in each layer of the three models. The curve represents the change of the accuracy of the model after each pruning, and it can be found that the accuracy of the model has not changed from beginning to end.}
\label{fig:long}
\label{fig:onecol}
\label{fig:5}
\end{figure}

	All the experiments are conducted within Tensorflow(1.14.0) and Keras(2.2.5).

	The convolution operation is usually followed by batch normalization layer and activation layer, in our experiment, we regard these three layers as a block and conduct QSFM pruning on its final output 3D Tensor(feature maps).

	For every residual block in ResNet-56, we only prune the first convolutional layers, which is simple and keeping the output dimension of residual block unchanged, as shown in Fig.~\ref{fig:3}. For every  bottleneck in MobileNetV2, we prune depthwise separable convolution layers, as shown in Fig.~\ref{fig:4}. 

	We pruned the neural network in the order from the first layer to the last. After each pruning, we carried out 30 epochs of fine-tuning operation. After the whole pruning operation, we calculate the FLOPs and Parameters of the pruned network and compare them to existing prior methods \cite{Lin2020, He2018, He2017, Lin2019, Zhao2019}.


\begin{figure}[t]
\begin{center}
   \includegraphics[width=1.0\linewidth]{graph//E.pdf}
\end{center}
   \caption{The histogram shows the FLOPs of the model after each pruning. The curve shows the change of model accuracy after each pruning under four different methods.}
\label{fig:long}
\label{fig:onecol}
\label{fig:6}
\end{figure}

\subsubsection{Compression rate Settings} \label{sec:4.1.2}

\begin{itemize}

\item \textbf{VGG-16:} For the fine-tuning part of VGG-16, to show the universality of our method, we set the compression ratio to 0.5 for all 13 layers.  

\item \textbf{Resnet-56:} In ResNet-56 on CIFAR-10, we found quite a few convolution layers have many filters whose all parameter values close to $10^{-32}$. We counted the proportion of filters whose all parameter values are close to  $10^{-32}$ in the 27 layers of every residual block and pruned these layers. When we pruned these filters, the accuracy of the pruned model was roughly the same as that of the original model (without fine-tuning). In order to prove the generality of the results, we trained several models, all of which had similar results, as shown in Fig.~\ref{fig:5}. \\ This shows that there is a lot of redundancy in the model of ResNet-56 on CIFAR-10, so we think that it may lead to the selection of filters that should be pruned are all of magnitude $10^{-32}$ if the compression ratio of each convolutional layer is too small, which cannot reflect the advantages and disadvantages of the method. It's like a group of dogs(other filters) were mixed in with some cats(filters whose all parameter values close to $10^{-32}$), and the goal is to find a suitable way to pick out the good dogs(compress model). Obviously, the cats need to be removed first, which doesn't show whether the method is good or bad.\\ To sum up, we can only judge the performance of the method if the compression ratio is high enough (larger than the proportion of filters whose all parameter values close to $10^{-32}$). If the compression ratio is too low, then any method will always get a good result because the filters they delete are distinctly useless (just like remove the cat from dogs and can not tell anything). \\ We use this 'high compression ratio' principle to compare the pruning performance of various methods (the compression ratio of each layer of ResNet-56 on CIFAR-10 will not be lower than 0.5), and we also provide a low compression ratio without any fine-tuning operation pruning version of ResNet-56 to demonstrate that our method can also perform well at low compression rates.

\item \textbf{MobileNetV2:} For the fine-tuning part of  MobileNetV2, we set the compression ratio to 0.3 for all 17 depthwise separable convolution layers. 

\end{itemize}


\begin{table}
\scriptsize
\caption{Pruning results of VGG-16 on CIFAR-10(with Fine-tuning). The data of other algorithms are selected from the corresponding papers. 'FLOPs' and 'Parameters' denote the remaining percentage of FLOP and parameter quantities of the compressed models and the lower the better. 'Top-1' represents the Top-1 accuracy. 'Top-5' represents the Top-5 accuracy. 'PR' represents the pruning rate and the higher the better. The Accuracy is represented in the format of pre-trained→pruned (Decreased accuracy). We use boldface to denote our method. The other tables and figures follows the same convention.}
\begin{center}
\addvbuffer[0pt -12pt]{
\setlength{\tabcolsep}{2mm}{
\begin{tabular}{c|c|c|c}
\toprule
Method & Top-1(\%) & FLOPs(PR) &  Parameters(PR) \\
\hline\midrule
\textbf{QSFM-SSIM} & 93.39→91.60(-1.79) & 79.0M(74.8\%) & 3.68M(75.0\%) \\
Hrank \cite{Lin2020} & 93.96→91.23(-2.73) & 73.67M(76.5\%) & 1.75M (88.1\%) \\
\textbf{QSFM-PSNR} & 93.39→90.86(-2.53)  & 79.0M(74.8\%) & 3.68M(75.0\%)\\
\bottomrule
\end{tabular}}}
\end{center}
\label{table:1}
\end{table}


\subsection{Results and Analysis on CIFAR-10}

	We pruned some mainstream models on CIFAR-10, including VGG-16, ResNet-56 and MobileNetV2.

\subsubsection{VGG-16 without Fine-tuning}	
	
	For VGG-16 without Fine-tuning, we only pruned the third and fourth blocks(the $5^{th}$ to $10^{th}$ convolution layers), and the compression ratio is [0.6, 0.4, 0.3, 0.3, 0.3, 0.3]. At this compression ratio, we prune the same network step by step, layer by layer with different methods, including QSFM-PSNR, QSFM-SSIM, Hrank, Random(delete filters randomly). It should be emphasized that we did not make any fine-tuning in this part, and the experimental results are shown in the Fig.~\ref{fig:6}. For fair comparison, the feature map of each method is obtained by averaging the same 5000 images which are randomly generated from CIFAR-10.

Our method is consistently better than HRank method and random method in pruning different convolutional layers. In the experiment, we found that sometimes the accuracy of our method after pruning is even higher than that before pruning(In steps $3^{th}$ and $4^{th}$ of QSFM-SSIM). When the compression ratio is large, HRank needs to delete the high-rank feature map, but it cannot effectively distinguish which are redundant in the high-rank feature maps.

\subsubsection{VGG-16 with Fine-tuning}

\begin{table}
\scriptsize
\caption{Pruning results of ResNet-56 on CIFAR-10(without Fine-tuning)}
\begin{center}
\addvbuffer[0pt -12pt]{
\setlength{\tabcolsep}{1mm}{
\begin{tabular}{c|c|c|c}
\toprule
Method & Top-1(\%) & FLOPs(PR) &  Parameters(PR) \\
\hline\midrule
\textbf{QSFM-SSIM} & 93.21→92.67(-0.54) & 64.92M(48.7\%) & 0.36M(57.9\%) \\
AMC \cite{He2018} & 92.80→90.10(-2.70) & 63.28M(50.0\%) & – \\
He \etal \cite{He2017} & 92.80→90.80(-2.00)  & 62.52M(50.6\%) & –\\
GAL-0.6 \cite{Lin2019} & 93.26→92.98(-0.28)  & 78.30M(37.6\%) & 0.75M(11.80\%)\\
GAL-0.8 \cite{Lin2019} & 93.26→90.36(-2.90)  & 49.99M(60.2\%) & 0.29M(65.90\%)\\
Zhao \etal \cite{Zhao2019} & 93.04→92.26(-0.78)  & 100.86M(20.3\%) & 0.68M(20.49\%)\\
\bottomrule
\end{tabular}}}
\end{center}
\label{table:2}
\end{table}

	We further apply our method to prune the VGG-16 model with Fine-tuning, all 13 convolution layers have a compression ratio of 0.5. The results are displayed in Tab.~\ref{table:1}. Compared with HRank, QSFM-SSIM and QSFM-PSNR nearly compress same FLOPs while maintaining a better accuracy drop (-1.79\% vs. -2.73\% , -2.53\% vs. -2.73\%).

\subsubsection{ResNet-56 without Fine-tuning}

\begin{figure}[t]
\begin{center}
   \includegraphics[width=1.0\linewidth]{graph//H.pdf}
\end{center}
   \caption{The histogram shows the FLOPs of the ResNet-56 model after each pruning. The curve shows the change of model accuracy after each pruning under four different methods.}
\label{fig:long}
\label{fig:onecol}
\label{fig:7}
\end{figure}

	In this part, we first prove that our QSFM method can obtain excellent results at low compression ratio according to the results obtained in Sec.~\ref{sec:4.1.2}. However, as mentioned above, only at high compression rate can the advantages and disadvantages of various methods be fully reflected. Therefore, we compare the performance of different pruning methods at high compression ratio.


\begin{table}
\scriptsize
\caption{Pruning results of ResNet-56 on CIFAR-10(with Fine-tuning).}
\begin{center}
\addvbuffer[0pt -12pt]{
\setlength{\tabcolsep}{1mm}{
\begin{tabular}{c|c|c|c}
\toprule
Method & Top-1(\%) & FLOPs(PR) &  Parameters(PR) \\
\hline\midrule
GAL-0.8 \cite{Lin2019} & 93.26→91.58(-1.68) &50.37M(60.2\%) & 0.29M(65.9\%)\\
\textbf{QSFM-SSIM-1} & 93.21→91.92(-1.29)	& 53.15M(58.0\%) & 0.26M(69.1\%)\\
GAL-0.6 \cite{Lin2019} & 93.26→93.38(0.12) & 78.30M(37.6\%) & 0.75M(11.8\%)\\
AMC \cite{He2018} & 92.80→91.90(-0.9) & 63.28M(50.0\%) & –\\
\textbf{QSFM-PSNR} & 93.21→91.98(-1.23) & 53.15M(58.0\%) & 0.26M(69.1\%)\\
Hrank \cite{Lin2020} & 93.26→90.72(-2.54) & 32.77M(74.1\%) & 0.27M(68.1\%)\\
\textbf{QSFM-SSIM-2} & 93.21→91.88(-1.33) & 50.62M(60.0\%) & 0.25M(71.3\%)\\
\bottomrule
\end{tabular}}}
\end{center}
\label{table:3}
\end{table}


	\textbf{Low compression ratio}. In order to prove the effectiveness of our method under low pruning ratio, we prune ResNet-56 without any fine-tuning operations according to the proportion which is slightly higher than that of the unimportant filters(filters whose all parameter values close to $10^{-32}$), and finally got good results as shown in Tab.~\ref{table:2}. Compared with AMC and He te al., QSFM-PSNR nearly compresses same FLOPs while maintaining a less accuracy drop (-0.54\% vs. -2.7\% and -0.54\% vs. -2.0\%). Compared with GAL-0.6, though QSFM-PSNR has a little higher accuracy drop(-0.54\% vs. -0.28\%), but it gains a larger FLOPs and parameters reduction(64.92M vs. 78.30M and 0.36M vs. 0.75M). Compared with GAL-0.8, QSFM-PSNR gains a less accuracy drop(-0.54\% vs. -2.9\%) while has a little shortage at compression. Compared with Zhao et al., QSFM-PSNR has a less accuracy drop(-0.54\% vs. -0.78\%), larger FLOPs reduction(64.92M vs. 100.86M) and larger parameters reduction(64.92M vs. 78.30M and 0.36M vs. 0.68M).

	\textbf{High Compression Ratio}. For high compression ratio, we only pruned the first 9 blocks, and the compression ratio is [0.5, 0.5, 0.5, 0.5, 0.75, 0.75, 0.75, 0.75, 0.75]. We used the whole training set (50000 images) to calculate the similarity between feature maps. The results are shown in Fig.~\ref{fig:7}. Compared with HRank and Random, QSFM-PSNR provides better maintenance of precision for whole 9 steps, which demonstrates the progressiveness of applying the quantified similarity of feature maps as an importance selection filters. But for QSFM-SSIM, it only performed better in steps $1^{th}$ to $3^{th}$ , which indicates that there is still room for improvement in the selected function to measure similarity. Both QSFM-PSNR and QSFM-SSIM are far better than Random, which proves the correctness of the method.
	
\subsubsection{ResNet-56 with Fine-tuning}

	We further apply our method to prune the ResNet-56 model with Fine-tuning. The results are displayed in Tab.~\ref{table:3}. QSFM-PSNR again demonstrates its ability to obtain a high accuracy of 91.98\%, with 69.1\% parameters reduction and 58.0\% FLOPs reduction. This is significantly better than GAL-0.8. Compared with AMC, which obtains 91.90\% of top-1 accuracy and 50\% FLOPs reduction, QSFM-PSNR gets a larger FLOPs reduction(53.15M vs. 63.28M). Compared with Hrank, QSFM-SSIM-2 gains a less top-1 accuracy drop and parameters reduction( -1.33\% vs. -2.54\% and 0.25M vs. 0.27M).

\subsubsection{MobileNetV2	with Fine-tuning}

\begin{table}
\scriptsize
\caption{Pruning results of MobileNetV2 on CIFAR-10(with Fine-tuning)}
\begin{center}
\addvbuffer[0pt -12pt]{
\setlength{\tabcolsep}{2mm}{
\begin{tabular}{c|c|c|c}
\toprule
Method & Top-1(\%) & FLOPs(PR) &  Parameters(PR) \\
\hline\midrule
MobileNetV2 & 92.54\% & 78.38M(0.0\%) & 2.20M(0.0\%)\\
\textbf{QSFM-PSNR} & 92.06\% & 57.27M(26.93\%) & 1.67M(24.09\%)\\
\bottomrule
\end{tabular}}}
\end{center}
\label{table:4}
\end{table}


	For MobileNetV2, we use QSFM-PSNR to prune the whole 17 depthwise separable convolution layers on CIFAR-10. The compression ratio of each layer is 0.3. The results of pruning experiment are shown in Tab.~\ref{table:4}. After pruning, the accuracy of the model only dropped by 0.48\% (92.54\% $\rightarrow$ 92.06\%), with 26.93\% FLOPs and 24.09\% parameters reduction. The results also show that our proposed method can also achieve good results for depthwise separable convolution layers.

\subsection{Results and Analysis on CIFAR-100}


\begin{table}

\scriptsize
\caption{Pruning results of ResNet-56 on CIFAR-100(with Fine-tuning)}
\begin{center}
\addvbuffer[0pt -12pt]{
\setlength{\tabcolsep}{1mm}{
\begin{tabular}{c|c|c|c|c}
\toprule
Method & Top-1(\%) & Top-5(\%) & FLOPs(PR) &  Parameters(PR) \\
\hline\midrule
ResNet-56 & 70.62\% & 92.00\% & 126.55M(0.0\%) & 0.86M(0.0\%)\\
\textbf{QSFM-PSNR} & 68.33\% & 91.08\% & 58.38M(53.87\%) & 0.42M(51.16\%)\\
\bottomrule
\end{tabular}}}
\end{center}
\label{table:5}
\end{table}

	We use QSFM-PSNR to prune ResNet-56 on CIFAR-100. We only prune the first convolutional layers for every residual block in ResNet-56,and the final results are shown in Tab.~\ref{table:5}. The Top-1 accuracy dropped by 2.29\% (70.62\%$\rightarrow$68.33\%) and the Top-5 accuracy dropped by 0.92\% (92.00\%$\rightarrow$91.08\%), with 53.87\% FLOPs reduction and 51.16\% parameters reduction.

\section{Conclusion}

	In this paper, we propose a novel theory to find the redundant information in the three-dimensional tensors, named QSFM. To be specific, QSFM quantifies the similarity between the feature maps of tensors by using the predefined function, and delete feature maps with high similarity to construct maximal independent system in three-dimensional tensors. We apply QSFM to filter pruning for compressing CNNs, and achieve good results. Experiments show that QSFM can compress CNNs such as VGGNet, ResNet and MobileNet significantly with negligible accuracy drop, which is superior to prior methods. We will continue to explore more quantization similarity functions and even some machine learning methods to quantify the similarity of feature maps, improving the performance of QSFM.



{\small
\bibliographystyle{ieee_fullname}

\section{Introduction


	Convolutional neural networks (CNNs) have become the leading content of various computer vision tasks, making marvelous progress in the fields of image classification \cite{He2016, Szegedy2015}, object detection \cite{Ren2015, Girshick2014}, image segmentation \cite{Chen2018, Long2015}, object tracking \cite{Bewley2016, Wojke2017}, 3D reconstruction \cite{tran2017}. Supported by a massive amount of data and excellent GPU computing capability, more and more heavy and complex deep models are proposed, such as VGGNet \cite{Simonyan2015}, ResNet \cite{He2016}, GoogLeNet \cite{Szegedy2015} and DenseNet \cite{Huang2017}.

	However, with the improvement of large model performance, there are huge parameter scale, terrible computing overhead and memory requirements. This becomes the challenge for edge and mobile embedded devices. To be specific, when deploying high-precision model on edge devices, there are three main problems remained to be tackled: 

\begin{itemize}

\item A satisfactory CNN model often has millions of weight parameters, which will consume disk storage and then be loaded into run-time memory during inference.

\item The feature maps cache data generated in the running process may even occupy more memory than the original model parameters.

\item Convolution operation is a kind of intensive computing operation. For example, in VGG-16, one time calculation computing operations required for one convolution layer can reach $1.85 \times 10^9$ times on a image with resolution $224 \times 224$. Convolution operation is the main computing burden.

\end{itemize}

	How to compress and accelerate the neural model has been a high profile in recent years. At present, the typical works include network quantization \cite{Guo2017, Hu2018}, low-rank approximation \cite{Masana2017}, weight sharing \cite{Wang2017, Chen2016, Son2018}, weight pruning \cite{Lee2019, Yang2019}. However, these methods need specific hardware or software library to run, and some of them can not solve the problems described above comprehensively. There is another lightweight technique defined as filter pruning \cite{Li2017, Lin2020, He2017, He2018, Lin2019, Zhao2019}, also known as coarse-grained pruning. By comparison, filter pruning is not restricted to special hardware or software, and suitable as well as universal for CNNs in various types of tasks. 

\begin{figure*}[t]
\begin{center}
   \includegraphics[width=0.7\linewidth]{graph//x.pdf}
\end{center}
   \caption{The left of the image is the output feature maps of the first convolution layer in the unpruned VGG-16 model, and it can be found that there are many similar feature maps. The right of the image is the pruning result of QSFM . Deleting similar information is like obtaining maximal linearly independent system in a two-dimensional matrix.}
\label{fig:long}
\label{fig:onecol}
\label{fig:1}
\end{figure*}

	Filter pruning has been shown to be significant in network slimming \cite{Frankle2019, Malach2020}. On filter pruning, recent works have made great progress. \cite{He2018, Lin2019, Lin2020a} put forward specific optimization objectives and constraints, and utilize heuristic optimization algorithm to train CNNs jointly. These will bring a new set of hyper parameter problems. That means, aiming to deploy strategies, extra tricks to further adjust hyper parameters of heuristic algorithm is needed, which is not flexible for practical application. Moreover, jointed trainning frequently combined with sparsity reguarization penalty, which will destroys the structure of the model itself. In \cite{Huang2018, He2017, Zhao2019, Liu2017, Yu2018, Lin2020, Li2017}, filters are pruned by concrete regulations, which are explicit in generating filter importance. These methods needs sort the importance from high to low, and then delete the low importance filters. It doesn’t take much manual adjustment involved, which means they are robust in varied situations. However, \cite{Yu2018, He2017} originated in experience or intuition, lacking of basic theoretical guidance, though they do success in compressing model. Meanwhile, other rule-based criterion provides sufficient theoretical support to find out this redundant information, but still remain defects. \cite{Li2017} applied the L1 norm to filter as the Evaluation criteria of importance. However, all layers use the same trim scale, which brings suboptimization.  	      \cite{Lin2020} propose a method named Hrank to prune filters with low-rank feature maps. Similar high-rank feature maps may also contain redundant information, but they have not been removed.

	Our work belongs to filter pruning. This idea is inspired by similarity between pictures.The high level of similarity indicates that the feature information contained in the two images is basically the same. The convolutional neural layer produces abundant multi-dimensional tensors, which can be regarded as pictures. In other words, We can apply the idea of similarity to feature maps. Our method first need to generate the feature maps for each convolution layer from training dataset and quantify the similarity between these feature maps. Then we prune convolution filters, layer by layer, based on the corresponding channel feature maps which have the most highest level in similarity. After the filter pruning, the performance of the slimmed model will be promoted through fine-tuning, without obvious loss of accuracy. The similarity of feature maps has been explained and applied in GhostNet \cite{Han2020}, which further proves the feasibility of our method. The linchpin of our method is to opt quantization function, which will be amply discussed in  Sec.\pageref{section3}. It should be emphasized that we have not implement sparsity reguarization penalty in model training. 

	Our main contributions are as follows:

\begin{itemize}

\item Inspired by maximal linearly independent system in the two-dimensional matrices, we propose a method to find out the redundant information in the three-dimensional tensors by quantifying the similarity of any two feature maps, namely QSFM.  

\item QSFM is applicable to all kinds of convolution, including common convolution and depthwise separable convolution. We also use different similarity quantization functions to further explore QSFM.  

\item We do various experiments on CIFAR-10 and CIFAR-100, using VGGNet, MobileNet and ResNet. which shows superior performance of QSFM over existing prior filter pruning methods.

\end{itemize}

\section{Related Work}


	\textbf{Network Quantization}. \cite{Guo2017} proposed the network sketching method, which uses the convolution with binary weight sharing: for convolution calculation with the same input, the result of the previous convolution is retained, and the same part of the convolution filters directly multiplex the result. \cite{Hu2018} projected data into Hamming space by hashing, and transformed the problem of learning binary parameters into a hashing problem under inner product similarity. Different from the traditional 1-Valued or weighted mean, \cite{Zhu2017} proposed trained ternal quantification (TTQ), which uses two trainable full precision scaling coefficients to quantify the weight to $ \{-w_n,0,w_p \}$, and asymmetric weights make the network more flexible. Unlike our approach, these methods aim to quantify the weight of filters into discrete values, while our approach is to find out a quantization function as specific description for similarity between feature maps. These methods obtain a very high level of compression in saving storage space and significantly accelerate inference, but it is extreme, and will bring a considerable loss of accuracy, damaging the model performance. Additionally, the value of the possible weight network needs to be further explored. 

	\textbf{Weight pruning}. Weight pruning can remove any redundant parameter or redundant connection of the expected proportion of the network without restriction, but it will bring the problem of irregular network structure, making it difficult to effectively accelerate after pruning. Recently, \cite{Lee2019} proposed SNIP method that the importance of the connection was determined by sampling the training set several times in the initialization phase of the model, and the pruning template was generated in the meantime. After training, there is no need to iteratively prune and fine-tune by the alternating cycle process. \cite{Yang2019} used weighted sparse projection and input masking to provide quantifiable energy consumption, taking energy consumption budget as the optimization constraint of network training, and the sparse network can be obtained by utilizing dynamic pruning method which can recover the important connections removed by mistake. Due to requirement of special sparse matrix operation library and hardware, it is also not convenient and universal to use weight pruning to implement acceleration for model.

	\textbf{Filter pruing}. Also named as coarse-grained pruning, it considers filters as the minimum prunning units, which can make the network "narrow" , and can directly achieve effective acceleration on existing software or hardware. Compared with human-crafted pruning, \cite{He2018} made the model compression completely automatic and perform better. It utilized DDPG (Deep Deterministic Policy Gradient) as the controller to generate the specific compression ratio in continuous space. Different from the previous hard pruning and label dependent pruning methods, \cite{Lin2019} proposed a label free Generative Adversarial Learning (GAL) method, which uses sparse soft mask pruning network to scale the output of specific structure to zero. It learns pruning networks with sparse soft masks in an end-to-end manner. Due to heuristic optimization algorithm and jointed training, \cite{He2018, Lin2019} will bring a new set of hyper parameter problems. \cite{He2017} propose an iterative two-step algorithm to effectively prune each layer, by a LASSO regression based channel selection and least square reconstruction. \cite{Zhao2019} proposed a variational Bayesian scheme for pruning convolutional neural networks in channel level, introducing a stochastic variational inference to estimate the distribution of channel saliency induced by a sparse prior. Though successful in compressing model, \cite{He2017, Zhao2019} originate in experience or intuition, lacking of basic theoretical guidance. \cite{Li2017} calculated the L1 norm of the filter, cutting out the feature map corresponding to the smaller L1 norm, and retrain after pruning. However, all layers use the same trim scale, which brings suboptimization. \cite{Lin2020} proposed an effective and efficient filter pruning approach that explores the High Rank (HRank) of the feature map in each layer. The principle behind HRank is that low-rank feature maps contain less information, and thus pruned results can be easily reproduced. Nevertheless, similar high-rank feature maps may also contain redundant information, but they have not been removed.

	Our method also belongs to filter pruning. This pruning category can get rid of the limitation of special software library or hardware, achieving the promotion of storage usage, memory occupation and computing speed at the same time. And it retains the model structure with different pruning rate. It can also be combined with other typical categorise mentioned above to further achieve higher compression. 
\section{Our Approach} \label{section3}

\subsection{Find the Redundant Information}

	In mathematics, for a two-dimensional matrix (that is, a set of vectors), all of its vectors can be represented as linear combinations of vectors in its maximal linearly independent system and its rank is the number of vectors contained in the maximal linearly independent system. Assume $A = [\alpha_1,\alpha_2,..., \alpha_n] \in R^{m \times n}$, $\alpha_i \in R^m$, $i = 1, 2, ..., n$. if the maximal linearly independent system of $A$ is $A_{Basic} \in R^{m \times s}$, $s \leq n$, we know that any vector in $A$ can be generated by $A_{Basic}$. To some extent, the maximal linearly independent system of two dimensional vectors represents its most essential information.

	We extend this concept of maximal linearly independent system to three dimensional tensors. Considering that three dimensional tensors are more complex and the condition of linear dependence is not easy to be satisfied in three dimensions, we replace the concept of linear dependence of multiple vectors with the similarity of two tensors. Assume the tensor $B = [\beta_1,\beta_2,...,\beta_k] \in R^{m \times n \times k}$, $\beta_i \in R^{m \times n}$, $i = 1, 2, ..., k$. If $\beta_i$ is similar to $\beta_j$, then $B$ has redundant information. How to define the similarity between $\beta_i$ and $\beta_j$ becomes the key of the problem. Obviously, we can define $\beta_i$ to be similar to $\beta_j$ when $\beta_i = k \cdot \beta_j$ ($q$ is a constant) just like two-dimensional matrix. But in practice, such a definition is too strict, and it is often difficult to find $\beta_i$ and $\beta_j$ that meet such conditions. 

	We define the quantization function $S(\beta_i, \beta_j)$, and the larger the value of $S(\beta_i, \beta_j)$, the more similar $\beta_i$ and $\beta_j$ are. In practice, we use Structural Similarity (SSIM) \cite{Wang2004} and Peak Signal to Noise Ratio (PSNR) , equivalent to Euclidean distance) as quantization functions $S(\beta_i, \beta_j)$. Our task is to find out the most essential information in the three-dimensional tensor through the similarity function, similar to the Maximal Linearly Independent System in the two-dimensional matrix.

\subsection{QSFM}


\subsubsection{Notations}

	Assume an image dataset $Train$ has $M$ images, denoted as $Train = \{Image_1, Image_2, ..., Image_k\} \in R^{M \times N_0 \times X_0 \times Y_0}$, where $N_0$ represents the channel number, $X_0$, $Y_0$ are the height and width. The pre-trained model before pruning is $Model_0$, the Model obtained after the $i$th pruning operation is $Model_i$ the convolution layers of the model are $L_1, L_2, ..., L_n$, $L_i$ contains $N_i$ filters, and the set constituted by $N_i$ filters is $W_i = \{Filter_{(i, 1)}, Filter_{(i, 2)}, ..., Filter_{(i, N_i)}\} \in R^{N_i \times N_{i-1} \times K_i \times K_i}$, where $K_i$ represents the height and width of the convolution filter. Assume the model input is only one image, then the input of $L_i$ is $I_i \in R^{N_{i-1} \times X_{i-1} \times Y_{i-1}}$ and the output of $L_i$ is $O_i \in R^{N_i \times X_i \times Y_i}$. For the $j$th $Filter_{(i, j)} \in R^{N_{i-1} \times K_i \times K_i}$ of layer $L_i$, its output is $Feature\_map_{(i,j)} \in R^{X_i \times Y_i}$, and these $N_i$ Feature maps constitute the set $O_i = \{Feature\_map_{(i,1)}, Feature\_map_{(i,2)}, ...$, $Feature\_map_{(i,N_i)}\} \in R^{N_i \times X_i \times Y_i}$.  For the layer $L_i$, we assume $Filters$ which should be pruned constitute the set $Delete^i$. After identifying the redundant $Feature\_maps$, you can prune them according to their corresponding $Filters$, as shown in Fig.~\ref{fig:1}.

\subsubsection{Quantify similarity}

	The sequence of pruning is from $L_1$ to $L_n$, and it is assumed that the convolution layer of $L_i$ is being pruned.

	First, assume the number of filters to be pruned in layer $L_i$ is $N_{i2}$ according to the compression rate, that is, there are $N_{i2}$ elements in set $Delete^i$.

	The Model to be pruned is $Model_{i-1}$, when the $Image_k$ is input into the $Model_{i-1}$, the output of $L_i$ is $O_i^k = \{Feature\_map_{(i,1)}^k, Feature\_map_{(i,2)}^k, ...$, $Feature\_map_{(i,N_i)}^k\} $. Calculate the quantified similarity function $S(Feature\_map_{(i,m)}^k, Feature\_map_{(i,n)}^k)$ where $Feature\_map_{(i,m)}^k$ and $Feature\_map_{(i,m)}^k$ ($m\neq n$) are any two elements in the set $O_i^k$.  Hereinafter, referred to as $S_{m, n}^{i, k}$, function $S$ takes Structural Similarity (SSIM) or Peak Signal to Noise Ratio (PSNR). 

	Structural Similarity and Peak Signal to Noise Ratio are usually used to measure the image quality after compression. As far as we know, this is the first time that these methods are used to measure the similarity between two feature maps in CNNs.




\begin{small}
\begin{equation}
\begin{aligned}
\begin{array}{l}
S_{m,n}^{i,k}(SSIM)\\
\\ \vspace{1ex}
{\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt}  \vspace{1ex}= SSIM(Feature\_map_{(i,m)}^k,Feature\_map_{(i,n)}^k)\\
{\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt}  = \dfrac{{\displaystyle (2{\mu _m}{\mu _n} + k_1^2{D^2})(2{\sigma _{mn}} + k_2^2{D^2})}}{{\displaystyle (\mu _m^2 + \mu _n^2 + k_1^2{D^2})(\sigma _m^2 + \sigma _n^2 + k_2^2{D^2})}}.
\end{array}
\end{aligned}
\end{equation}
\end{small}



	Here $\mu_m$ and $\mu_n$ are the average of all pixels in $Feature\_map_{(i,m)}^k$ and $Feature\_map_{(i,n)}^k$. $\sigma _m^2$ and $\sigma _n^2$ are the variance of all pixels in $Feature\_map_{(i,m)}^k$ and $Feature\_map_{(i,n)}^k$. $\sigma _{mn}$ is the covariance of $Feature\_map_{(i,m)}^k$ and $Feature\_map_{(i,n)}^k$. Usually $k_1$ is 0.01 and $k_2$ is 0.03. $D$ is determined by the following formula:

\begin{small}
\begin{equation}
\begin{aligned}
D = \max (O_i^k) - \min (O_i^k).
\end{aligned}
\end{equation}
\end{small}

	Here $\max (O_i^k)$ and $\min (O_i^k)$ are the values of the largest and smallest pixels in $(O_i^k)$.

	Since the quantization of PSNR is theoretically equivalent to the Euclidean distance, and we only need to sort the similarity according to the size of the similarity, needless to know the specific value of PSNR. Therefore, the following content does not distinguish the PSNR from the Euclidean distance.

\begin{small}
\begin{equation}
\begin{aligned}
\begin{array}{l}
S_{m,n}^{i,k}(PSNR)\\
\\ \vspace{1ex}
{\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt}  \vspace{1ex}= EuclideanDistance(FM_{(i,m)}^k,FM_{(i,n)}^k)\\
{\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt}  = \sum\limits_{x = 1}^{{X_i}} {\sum\limits_{y = 1}^{{Y_i}} {{{(FM_{(i,m)}^k[x,y] - FM_{(i,n)}^k[x,y])}^2}} }.
\end{array}
\end{aligned}
\end{equation}
\end{small}



	$FM_{(i,m)}^k$ and $FM_{(i,n)}^k$ denote $Feature\_map_{(i,m)}^k$ and $Feature\_map_{(i,n)}^k$. It is noted that $S_{m, n}^{i, k}$ is only the result generated by a single Image ($Image_k$). For the whole data set Train, it contains $M$ images in total. We calculate the statistical average results on $M$ images, and define: 

\begin{small}
\begin{equation}
\begin{aligned}
\begin{array}{l}
S_{m,n}^i = \dfrac{{\displaystyle \sum\limits_{k = 1}^M {S_{m,n}^{i,k}} }}{\displaystyle M},\\ \\ \vspace{1ex}{S^i} = \{ S_{m,n}^i\} ,(m,n = 1,2,3,...,{N_i}).
\end{array}
\end{aligned}
\end{equation}
\end{small}

	The larger $S_{m,n}^i$ is, the more similar is $Feature\_map_{(i,m)}^k$ to $Feature\_map_{(i,n)}^k$. After finding out the feature map groups with high similarity, we need to stipulate an auxiliary condition to determine which of the two feature maps to delete. We can randomly delete a feature map in each similar group, or we can use the L1 norm of the feature map to delete those with a smaller norm, and this part of the experiment is provided in the supplementary material. We use the rank of the two-dimensional matrix as auxiliary condition to assist the quantified similarity function finding the $Delete^i$. The use of this auxiliary condition is inspired by HRank \cite{Lin2020}, but it does not mean that QSFM is similar to that of HRank. In fact, as mentioned above, HRank does not take into account the existence of redundant information (i.e. similarity) in two high-rank feature maps. Meanwhile, our experiment will prove that QSFM is superior to HRank. Calculate the rank of each $Feature\_map_{(i,m)}^k$, then  the statistical average of the Rank of the $Feature\_map_{(i,m)}^k$ in $O_i$ is:

\begin{small}
\begin{equation}
\begin{aligned}
\begin{array}{l}
Rank_m^i = \dfrac{{\displaystyle\sum\limits_{k = 1}^M {Rank_m^{i,k}} }}{\displaystyle M},\\
\\ \vspace{1ex} Ran{k^i} = \{ Rank_m^i\} ,(m = 1,2,3,...,{N_i})
\end{array}
\end{aligned}
\end{equation}
\end{small}

\subsubsection{Filter Pruning}
	
\begin{figure}[t]
\begin{center}
   \includegraphics[width=0.8\linewidth]{graph//Y.pdf}
\end{center}
   \caption{Whatever the convolution process (common convolution or depthwise separable convolution) is, QSFM can prune them to obtain maximal linearly independent system in three-dimensional tensors.}
\label{fig:long}
\label{fig:onecol}
\label{fig:2}
\end{figure}

	Arrange the set $S^i$ in order of highest to lowest, and then match the set, $Rank^i$ to determine the Filter to be pruned. Set the conditions as follows:

	Condition 1: $Filter_{(i,m)}$ and $Filter_{(i,n)}$ are not members of $Delete^i$;
	
	Condition 2: $S_{m,n}^i$ is maximum under the premise that $m$ and $n$ meet condition 1;
	
	Condition 3: $Rank_m^i \textgreater Rank_n^i$;

	Condition 4: $Rank_m^i \leq Rank_n^i$;

	If conditions 1, 2 and 3 are met simultaneously, then $Filter_{(i,n)}$ will be put into $Delete^i$; if conditions 1, 2 and 4 are met simultaneously, then $Filter_{(i,m)}$ will will be put into $Delete^i$. And the above operation will continue until the number of filters in the set $Delete^i$ is $N_{i2}$.

	The model should be fine-tuned after pruning every convolution layer, that is:

\begin{small}
\begin{equation}
\begin{aligned}
Model_i = Fine\_Tuned(Mode{l_{i - 1}} - Delet{e^i})
\end{aligned}
\end{equation}
\end{small}

	Then it enters the pruning of the next convolutional layer until all the convolutional layers are pruned, and the compression of the neural network is completed.


\subsubsection{Special Convolution}

\begin{figure}[t]
\begin{center}
   \includegraphics[width=0.9\linewidth]{graph//B.pdf}
\end{center}
   \caption{The pruning strategy of ResNet-56.}
\label{fig:long}
\label{fig:onecol}
\label{fig:3}
\end{figure}


\begin{figure}[t]
\begin{center}
   \includegraphics[width=0.9\linewidth]{graph//C.pdf}
\end{center}
   \caption{The pruning strategy of MobileNetV2.}
\label{fig:long}
\label{fig:onecol}
\label{fig:4}
\end{figure}

	Our method can also prune special convolution layers, such as depthwise separable convolution. As shown in Fig.~\ref{fig:2}, no matter what kind of convolution layer output, there will be similar feature maps. We can also pruning the network by using the QSFM.

\section{Experiments}

	In our experiments, we use Euclidean distance (PSNR) and Structural Similarity (SSIM) as the function of quantifying the similarity for QSFM, which are denoted as QSFM-PSNR and QSFM-SSIM respectively. We first demonstrated the excellent results obtained by QSFM pruning without any fine-tuning operations in ResNet-56, which is the best as far as we know. We also show the pruning results of QSFM for VGG-16, ResNet-56 and MobileNetV2 \cite{Sandler2018} on the CIFAR-10 \cite{CIFAR10} with fine-tuning operation. Furthermore, we show our results of QSFM for ResNet-56 on CIFAR-100 \cite{CIFAR10}. The original precisions of our model are 93.39\% (VGG-16,CIFAR-10), 93.21\% (ResNet-56, CIFAR-10), 92.54\%( MobileNetV2, CIFAR-10) and 70.62\%( ResNet-56, CIFAR-100).


\subsection{Experimental Settings}

\subsubsection{Configurations}

\begin{figure}[t]
\begin{center}
   \includegraphics[width=1.0\linewidth]{graph//D.pdf}
\end{center}
   \caption{The histogram coordinates represent the proportion of ‘unimportant filters’(filters whose all parameter values are close to $10^{-32}$) in each layer of the three models. The curve represents the change of the accuracy of the model after each pruning, and it can be found that the accuracy of the model has not changed from beginning to end.}
\label{fig:long}
\label{fig:onecol}
\label{fig:5}
\end{figure}

	All the experiments are conducted within Tensorflow(1.14.0) and Keras(2.2.5).

	The convolution operation is usually followed by batch normalization layer and activation layer, in our experiment, we regard these three layers as a block and conduct QSFM pruning on its final output 3D Tensor(feature maps).

	For every residual block in ResNet-56, we only prune the first convolutional layers, which is simple and keeping the output dimension of residual block unchanged, as shown in Fig.~\ref{fig:3}. For every  bottleneck in MobileNetV2, we prune depthwise separable convolution layers, as shown in Fig.~\ref{fig:4}. 

	We pruned the neural network in the order from the first layer to the last. After each pruning, we carried out 30 epochs of fine-tuning operation. After the whole pruning operation, we calculate the FLOPs and Parameters of the pruned network and compare them to existing prior methods \cite{Lin2020, He2018, He2017, Lin2019, Zhao2019}.


\begin{figure}[t]
\begin{center}
   \includegraphics[width=1.0\linewidth]{graph//E.pdf}
\end{center}
   \caption{The histogram shows the FLOPs of the model after each pruning. The curve shows the change of model accuracy after each pruning under four different methods.}
\label{fig:long}
\label{fig:onecol}
\label{fig:6}
\end{figure}

\subsubsection{Compression rate Settings} \label{sec:4.1.2}

\begin{itemize}

\item \textbf{VGG-16:} For the fine-tuning part of VGG-16, to show the universality of our method, we set the compression ratio to 0.5 for all 13 layers.  

\item \textbf{Resnet-56:} In ResNet-56 on CIFAR-10, we found quite a few convolution layers have many filters whose all parameter values close to $10^{-32}$. We counted the proportion of filters whose all parameter values are close to  $10^{-32}$ in the 27 layers of every residual block and pruned these layers. When we pruned these filters, the accuracy of the pruned model was roughly the same as that of the original model (without fine-tuning). In order to prove the generality of the results, we trained several models, all of which had similar results, as shown in Fig.~\ref{fig:5}. \\ This shows that there is a lot of redundancy in the model of ResNet-56 on CIFAR-10, so we think that it may lead to the selection of filters that should be pruned are all of magnitude $10^{-32}$ if the compression ratio of each convolutional layer is too small, which cannot reflect the advantages and disadvantages of the method. It's like a group of dogs(other filters) were mixed in with some cats(filters whose all parameter values close to $10^{-32}$), and the goal is to find a suitable way to pick out the good dogs(compress model). Obviously, the cats need to be removed first, which doesn't show whether the method is good or bad.\\ To sum up, we can only judge the performance of the method if the compression ratio is high enough (larger than the proportion of filters whose all parameter values close to $10^{-32}$). If the compression ratio is too low, then any method will always get a good result because the filters they delete are distinctly useless (just like remove the cat from dogs and can not tell anything). \\ We use this 'high compression ratio' principle to compare the pruning performance of various methods (the compression ratio of each layer of ResNet-56 on CIFAR-10 will not be lower than 0.5), and we also provide a low compression ratio without any fine-tuning operation pruning version of ResNet-56 to demonstrate that our method can also perform well at low compression rates.

\item \textbf{MobileNetV2:} For the fine-tuning part of  MobileNetV2, we set the compression ratio to 0.3 for all 17 depthwise separable convolution layers. 

\end{itemize}


\begin{table}
\scriptsize
\caption{Pruning results of VGG-16 on CIFAR-10(with Fine-tuning). The data of other algorithms are selected from the corresponding papers. 'FLOPs' and 'Parameters' denote the remaining percentage of FLOP and parameter quantities of the compressed models and the lower the better. 'Top-1' represents the Top-1 accuracy. 'Top-5' represents the Top-5 accuracy. 'PR' represents the pruning rate and the higher the better. The Accuracy is represented in the format of pre-trained→pruned (Decreased accuracy). We use boldface to denote our method. The other tables and figures follows the same convention.}
\begin{center}
\addvbuffer[0pt -12pt]{
\setlength{\tabcolsep}{2mm}{
\begin{tabular}{c|c|c|c}
\toprule
Method & Top-1(\%) & FLOPs(PR) &  Parameters(PR) \\
\hline\midrule
\textbf{QSFM-SSIM} & 93.39→91.60(-1.79) & 79.0M(74.8\%) & 3.68M(75.0\%) \\
Hrank \cite{Lin2020} & 93.96→91.23(-2.73) & 73.67M(76.5\%) & 1.75M (88.1\%) \\
\textbf{QSFM-PSNR} & 93.39→90.86(-2.53)  & 79.0M(74.8\%) & 3.68M(75.0\%)\\
\bottomrule
\end{tabular}}}
\end{center}
\label{table:1}
\end{table}


\subsection{Results and Analysis on CIFAR-10}

	We pruned some mainstream models on CIFAR-10, including VGG-16, ResNet-56 and MobileNetV2.

\subsubsection{VGG-16 without Fine-tuning}	
	
	For VGG-16 without Fine-tuning, we only pruned the third and fourth blocks(the $5^{th}$ to $10^{th}$ convolution layers), and the compression ratio is [0.6, 0.4, 0.3, 0.3, 0.3, 0.3]. At this compression ratio, we prune the same network step by step, layer by layer with different methods, including QSFM-PSNR, QSFM-SSIM, Hrank, Random(delete filters randomly). It should be emphasized that we did not make any fine-tuning in this part, and the experimental results are shown in the Fig.~\ref{fig:6}. For fair comparison, the feature map of each method is obtained by averaging the same 5000 images which are randomly generated from CIFAR-10.

Our method is consistently better than HRank method and random method in pruning different convolutional layers. In the experiment, we found that sometimes the accuracy of our method after pruning is even higher than that before pruning(In steps $3^{th}$ and $4^{th}$ of QSFM-SSIM). When the compression ratio is large, HRank needs to delete the high-rank feature map, but it cannot effectively distinguish which are redundant in the high-rank feature maps.

\subsubsection{VGG-16 with Fine-tuning}

\begin{table}
\scriptsize
\caption{Pruning results of ResNet-56 on CIFAR-10(without Fine-tuning)}
\begin{center}
\addvbuffer[0pt -12pt]{
\setlength{\tabcolsep}{1mm}{
\begin{tabular}{c|c|c|c}
\toprule
Method & Top-1(\%) & FLOPs(PR) &  Parameters(PR) \\
\hline\midrule
\textbf{QSFM-SSIM} & 93.21→92.67(-0.54) & 64.92M(48.7\%) & 0.36M(57.9\%) \\
AMC \cite{He2018} & 92.80→90.10(-2.70) & 63.28M(50.0\%) & – \\
He \etal \cite{He2017} & 92.80→90.80(-2.00)  & 62.52M(50.6\%) & –\\
GAL-0.6 \cite{Lin2019} & 93.26→92.98(-0.28)  & 78.30M(37.6\%) & 0.75M(11.80\%)\\
GAL-0.8 \cite{Lin2019} & 93.26→90.36(-2.90)  & 49.99M(60.2\%) & 0.29M(65.90\%)\\
Zhao \etal \cite{Zhao2019} & 93.04→92.26(-0.78)  & 100.86M(20.3\%) & 0.68M(20.49\%)\\
\bottomrule
\end{tabular}}}
\end{center}
\label{table:2}
\end{table}

	We further apply our method to prune the VGG-16 model with Fine-tuning, all 13 convolution layers have a compression ratio of 0.5. The results are displayed in Tab.~\ref{table:1}. Compared with HRank, QSFM-SSIM and QSFM-PSNR nearly compress same FLOPs while maintaining a better accuracy drop (-1.79\% vs. -2.73\% , -2.53\% vs. -2.73\%).

\subsubsection{ResNet-56 without Fine-tuning}

\begin{figure}[t]
\begin{center}
   \includegraphics[width=1.0\linewidth]{graph//H.pdf}
\end{center}
   \caption{The histogram shows the FLOPs of the ResNet-56 model after each pruning. The curve shows the change of model accuracy after each pruning under four different methods.}
\label{fig:long}
\label{fig:onecol}
\label{fig:7}
\end{figure}

	In this part, we first prove that our QSFM method can obtain excellent results at low compression ratio according to the results obtained in Sec.~\ref{sec:4.1.2}. However, as mentioned above, only at high compression rate can the advantages and disadvantages of various methods be fully reflected. Therefore, we compare the performance of different pruning methods at high compression ratio.


\begin{table}
\scriptsize
\caption{Pruning results of ResNet-56 on CIFAR-10(with Fine-tuning).}
\begin{center}
\addvbuffer[0pt -12pt]{
\setlength{\tabcolsep}{1mm}{
\begin{tabular}{c|c|c|c}
\toprule
Method & Top-1(\%) & FLOPs(PR) &  Parameters(PR) \\
\hline\midrule
GAL-0.8 \cite{Lin2019} & 93.26→91.58(-1.68) &50.37M(60.2\%) & 0.29M(65.9\%)\\
\textbf{QSFM-SSIM-1} & 93.21→91.92(-1.29)	& 53.15M(58.0\%) & 0.26M(69.1\%)\\
GAL-0.6 \cite{Lin2019} & 93.26→93.38(0.12) & 78.30M(37.6\%) & 0.75M(11.8\%)\\
AMC \cite{He2018} & 92.80→91.90(-0.9) & 63.28M(50.0\%) & –\\
\textbf{QSFM-PSNR} & 93.21→91.98(-1.23) & 53.15M(58.0\%) & 0.26M(69.1\%)\\
Hrank \cite{Lin2020} & 93.26→90.72(-2.54) & 32.77M(74.1\%) & 0.27M(68.1\%)\\
\textbf{QSFM-SSIM-2} & 93.21→91.88(-1.33) & 50.62M(60.0\%) & 0.25M(71.3\%)\\
\bottomrule
\end{tabular}}}
\end{center}
\label{table:3}
\end{table}


	\textbf{Low compression ratio}. In order to prove the effectiveness of our method under low pruning ratio, we prune ResNet-56 without any fine-tuning operations according to the proportion which is slightly higher than that of the unimportant filters(filters whose all parameter values close to $10^{-32}$), and finally got good results as shown in Tab.~\ref{table:2}. Compared with AMC and He te al., QSFM-PSNR nearly compresses same FLOPs while maintaining a less accuracy drop (-0.54\% vs. -2.7\% and -0.54\% vs. -2.0\%). Compared with GAL-0.6, though QSFM-PSNR has a little higher accuracy drop(-0.54\% vs. -0.28\%), but it gains a larger FLOPs and parameters reduction(64.92M vs. 78.30M and 0.36M vs. 0.75M). Compared with GAL-0.8, QSFM-PSNR gains a less accuracy drop(-0.54\% vs. -2.9\%) while has a little shortage at compression. Compared with Zhao et al., QSFM-PSNR has a less accuracy drop(-0.54\% vs. -0.78\%), larger FLOPs reduction(64.92M vs. 100.86M) and larger parameters reduction(64.92M vs. 78.30M and 0.36M vs. 0.68M).

	\textbf{High Compression Ratio}. For high compression ratio, we only pruned the first 9 blocks, and the compression ratio is [0.5, 0.5, 0.5, 0.5, 0.75, 0.75, 0.75, 0.75, 0.75]. We used the whole training set (50000 images) to calculate the similarity between feature maps. The results are shown in Fig.~\ref{fig:7}. Compared with HRank and Random, QSFM-PSNR provides better maintenance of precision for whole 9 steps, which demonstrates the progressiveness of applying the quantified similarity of feature maps as an importance selection filters. But for QSFM-SSIM, it only performed better in steps $1^{th}$ to $3^{th}$ , which indicates that there is still room for improvement in the selected function to measure similarity. Both QSFM-PSNR and QSFM-SSIM are far better than Random, which proves the correctness of the method.
	
\subsubsection{ResNet-56 with Fine-tuning}

	We further apply our method to prune the ResNet-56 model with Fine-tuning. The results are displayed in Tab.~\ref{table:3}. QSFM-PSNR again demonstrates its ability to obtain a high accuracy of 91.98\%, with 69.1\% parameters reduction and 58.0\% FLOPs reduction. This is significantly better than GAL-0.8. Compared with AMC, which obtains 91.90\% of top-1 accuracy and 50\% FLOPs reduction, QSFM-PSNR gets a larger FLOPs reduction(53.15M vs. 63.28M). Compared with Hrank, QSFM-SSIM-2 gains a less top-1 accuracy drop and parameters reduction( -1.33\% vs. -2.54\% and 0.25M vs. 0.27M).

\subsubsection{MobileNetV2	with Fine-tuning}

\begin{table}
\scriptsize
\caption{Pruning results of MobileNetV2 on CIFAR-10(with Fine-tuning)}
\begin{center}
\addvbuffer[0pt -12pt]{
\setlength{\tabcolsep}{2mm}{
\begin{tabular}{c|c|c|c}
\toprule
Method & Top-1(\%) & FLOPs(PR) &  Parameters(PR) \\
\hline\midrule
MobileNetV2 & 92.54\% & 78.38M(0.0\%) & 2.20M(0.0\%)\\
\textbf{QSFM-PSNR} & 92.06\% & 57.27M(26.93\%) & 1.67M(24.09\%)\\
\bottomrule
\end{tabular}}}
\end{center}
\label{table:4}
\end{table}


	For MobileNetV2, we use QSFM-PSNR to prune the whole 17 depthwise separable convolution layers on CIFAR-10. The compression ratio of each layer is 0.3. The results of pruning experiment are shown in Tab.~\ref{table:4}. After pruning, the accuracy of the model only dropped by 0.48\% (92.54\% $\rightarrow$ 92.06\%), with 26.93\% FLOPs and 24.09\% parameters reduction. The results also show that our proposed method can also achieve good results for depthwise separable convolution layers.

\subsection{Results and Analysis on CIFAR-100}


\begin{table}

\scriptsize
\caption{Pruning results of ResNet-56 on CIFAR-100(with Fine-tuning)}
\begin{center}
\addvbuffer[0pt -12pt]{
\setlength{\tabcolsep}{1mm}{
\begin{tabular}{c|c|c|c|c}
\toprule
Method & Top-1(\%) & Top-5(\%) & FLOPs(PR) &  Parameters(PR) \\
\hline\midrule
ResNet-56 & 70.62\% & 92.00\% & 126.55M(0.0\%) & 0.86M(0.0\%)\\
\textbf{QSFM-PSNR} & 68.33\% & 91.08\% & 58.38M(53.87\%) & 0.42M(51.16\%)\\
\bottomrule
\end{tabular}}}
\end{center}
\label{table:5}
\end{table}

	We use QSFM-PSNR to prune ResNet-56 on CIFAR-100. We only prune the first convolutional layers for every residual block in ResNet-56,and the final results are shown in Tab.~\ref{table:5}. The Top-1 accuracy dropped by 2.29\% (70.62\%$\rightarrow$68.33\%) and the Top-5 accuracy dropped by 0.92\% (92.00\%$\rightarrow$91.08\%), with 53.87\% FLOPs reduction and 51.16\% parameters reduction.

\section{Conclusion}

	In this paper, we propose a novel theory to find the redundant information in the three-dimensional tensors, named QSFM. To be specific, QSFM quantifies the similarity between the feature maps of tensors by using the predefined function, and delete feature maps with high similarity to construct maximal independent system in three-dimensional tensors. We apply QSFM to filter pruning for compressing CNNs, and achieve good results. Experiments show that QSFM can compress CNNs such as VGGNet, ResNet and MobileNet significantly with negligible accuracy drop, which is superior to prior methods. We will continue to explore more quantization similarity functions and even some machine learning methods to quantify the similarity of feature maps, improving the performance of QSFM.



{\small
\bibliographystyle{ieee_fullname}
