\subsection{Biologically Inspired Connectivity}
Lateral connections between neurons in visual cortex exhibit a Mexican-hat profile \citep{muller2005attentional, kang2003mexican}. For this reason, we initialize lateral connections using a damped \textit{Ricker wavelet} $\omega$ with its peak at the modulated channel, given by

\begin{equation}
\label{eq:ricker}
    \omega(x) = \frac{2\delta}{\sqrt{3\sigma}\pi^{\frac{1}{4}}} \left( 1 - \frac{x^2}{\sigma^2} \right) e^{-x^2 / {2\sigma^2}}.
\end{equation}

\begin{figure}[t!]
\begin{center}
\centerline{\includegraphics[width=\columnwidth]{figures/wavelet.pdf}}
\caption{The Ricker wavelet formalizing the Mexican hat profile at different widths $\sigma$ (for its full function, see Equation \ref{eq:ricker}). As its peak is centered on the neuron at focus, this neuron's activity is facilitated by close neighbours but inhibited by remote cells.}
\label{fig:wavelet}
\vskip -0.2in
\end{center}
\end{figure}

The wavelet is parameterized by its width $\sigma$ and a damping factor $0 < \delta < 1$, which we introduce to avoid numerical instability by down scaling the wavelet. Figure \ref{fig:wavelet} shows examples for different values of $\sigma$. To construct the connectivity profile, the wavelet is discretized and centered around the modulated neuron. Note that we do not form a connection of a neuron to itself and therefore establish a weight of $0$ at the profile's center.

Following this connectivity profile, every channel is excited by close neighbors, but inhibited by filters at further distance. We hypothesize this to amplify regions in the filter sequence where multiple similar filters respond to the same input while increasing certainty in cases where the channels carry multiple hypotheses. A key requirement for this to work is a semantic filter ordering emerging during training. Note that the wavelet is approximately a Difference of Gaussians (DoG) and can hence also be regarded as a feature enhancement filter, which aligns with our hypothesis. 

\subsection{Temporal Dynamics}
We conceptualize the temporal evolution of the net input to a (fully connected) layer in an ANN as a dynamical system \citep[for a more detailed discussion of this refer to][]{yegnanarayana2009artificial}. In the case of constant input, such as image classification, and no recurrent connectivity, the temporal evolution of a layer's net input $\left(\mathbf{z}^l \right)$ is given by 
\begin{equation}
    \frac{d\mathbf{z}^l}{dt} = -\mathbf{z}^l + \mathbf{Wx}(t) + \mathbf{b},
\end{equation}
where $\mathbf{z}^l$ is a passive decay term, $\mathbf{x}(t)$ an external input modulated by the weight matrix $\mathbf{W}$ and $\mathbf{b}$ is a bias term. Finding the equilibrium of this system of independent differential equations (i.e. finding the converged values of $\mathbf{z}^l$) leads to the standard equation of the net input to a fully connected layer:
\begin{equation}
    \mathbf{z}^l = \mathbf{Wx} + \mathbf{b}.
\end{equation}

Introducing lateral connectivity renders the net input of units in a layer mutually dependent such that we obtain a coupled linear differential equation:
\begin{equation}
    \label{eq:diff-eq-lateral}
    \frac{d\mathbf{z}^l}{dt} = -\mathbf{z}^l + \mathbf{Uz}^l + \mathbf{Wx} + \mathbf{b},
\end{equation}
where $\mathbf{U}$ is the recurrent connectivity matrix. For the dynamics of such a system to be stable, its Jacobian ($ \mathbf{J} = \mathbf{U} - \mathbf{I}$, where $\mathbf{I}$ is the identity matrix) must have eigenvalues with a strictly negative real part; that is, the network feedback $\mathbf{U}$ must be sufficiently weak compared to local leakage (this is ensured by the $\delta$ parameter of the wavelet function).

Solving this system of coupled differential equations for the converged values of $\mathbf{z}^l$ one obtains
\begin{equation}
    \label{eq:diff-solved}
    \mathbf{z}^l = -\left(\mathbf{U} - \mathbf{I} \right)^{-1} \left( \mathbf{Wx} + \mathbf{b} \right).
\end{equation}
Simplifying this equation by introducing new terms $\mathbf{Q}=-\left(\mathbf{U} - \mathbf{I} \right)^{-1}$ and $\hat\mathbf{z}^l = \mathbf{Wx} + \mathbf{b}$, it is easy to see that the converged net input to a layer in the presence of linear feedback is equivalent to introducing an additional, linear, feed forward step:

\begin{equation}
    \label{eq:clc}
    \begin{array}{ll}
         \hat\mathbf{z}^l &= \mathbf{Wx} + \mathbf{b} \\
         \mathbf{z}^l &= \mathbf{Q}\hat\mathbf{z}^l
    \end{array}.
\end{equation}

\subsection{Algebraic Implementation}
\label{sec:lc-by-topelitz}
Since we realize lateral connectivity by introducing interactions between output channels of a convolutional layer, the output $\mathbf{Z}^l$ of the feed forward layer is a third-order tensor. In Equation \ref{eq:diff-eq-lateral}, the recurrent component $\mathbf{U}\mathbf{z}^l$ needs to be reformalized to apply to $\mathbf{Z}^l$. That is, a mathematical formulation of applying the connectivity profile to every neuron along the channel domain is required. Since lateral connectivity boils down to a moving weighted average it can in principle be realized by convolution. Specifically, since the connectivity profile ignores the spatial position of a neuron, the problem is a one-dimensional convolution of every depth-column with the wavelet. We implement this convolution in the form of a matrix multiplication with the Toeplitz matrix of the wavelet after unfolding the third-order input tensor. The resultant matrix product can then be folded back into the original input shape. An advantage of using a Toeplitz matrix is the ease of inverting the convolution, as required in Equation \ref{eq:diff-solved} to determine the equilibrium point. Alternatively one may conduct a deconvolution in the frequency domain by use of a fast Fourier transform (FFT). However, our benchmarking results demonstrate that this is substantially slower: a full forward and backward pass with Toeplitz-based SemLC is on average $4.55$ times faster than an FFT implementation. Generally, the added time cost of the former is approximately the same as adding a convolutional layer.

Note that using a Toeplitz matrix naturally renders the convolution circular, which aligns well with the organisation of cortical orientation columns. Laterally connecting kernels in a circle can also be motivated by the intuition that edge-detectors (as commonly emerge in the first layer of a CNN) when sorted by their orientation will form a ring because the last element of the sequence will again be similar to the first. However, choosing circular convolution over zero-padding is solely a conceptual decision, as we observed no significant difference between their influence on performance.

During training, the connectivity profile may either be fixed to its wavelet initialization or be allowed to adapt. Hereafter, we call the latter approach adaptive SemLC, or SemLC-A. Please note that an adaptive profile adds additional network parameters, while fixing it adds only (two) hyperparameters. Thirdly, we consider a variant in which the wavelet's parameters $\sigma$ and $\delta$ can be optimized (SemLC-P). Consequently, the optimization controls at which ranges and to which extent inhibition and excitation occur, but is constrained to the characteristic shape of the wavelet. This grants some adaptability while only adding two network parameters.

For the purpose of integrating lateral connectivity into a convolutional neural network, consider a classic pipeline of a convolutional block (convolution, ReLU, pooling). Semantic lateral connectivity will be added directly after the convolution because i) applying SemLC after pooling might result in relevant information being lost before the lateral connections can factor it in and ii) SemLC requires the dynamical system to be linear in order to analytically obtain the equilibrium.

\subsection{Proof of Concept}
\label{sec:strategy-comparison}

\begin{table*}[t!]
\caption{Comparison of SemLC variants to the baseline networks and Local Response Normalization (LRN) on CIFAR-10, with the classical split between train and test set. Confidence intervals are given at $\alpha = 0.05$.}
\label{tab:strategies}
\vskip 0.15in
\begin{center}
\begin{small}
\begin{sc}
\begin{tabular}{lrrr}
\toprule
Strategy            & AlexNet                           & Simple        & Shallow \\
\midrule
Baseline                & $84.91 \pm .07$           & $84.86 \pm .06$ & $79.52 \pm 0.17$ \\
LRN                 & $85.33 \pm .05$               & $85.38 \pm .06$ & $80.06 \pm 0.15$ \\
SemLC-G             & $85.21 \pm .05$               & $85.09 \pm .05$ & $79.36 \pm 0.23$ \\
\midrule
SemLC               & $85.52 \pm .06$             & $85.46 \pm .05$ & $81.00 \pm 0.11$ \\
SemLC-A             & $\mathbf{85.61} \pm .06$    & $\mathbf{85.65} \pm .06$ & $80.81 \pm 0.10$ \\
SemLC-P             & $85.48 \pm .06$             & $85.54 \pm .06$ & $\mathbf{81.66} \pm 0.19$ \\
\bottomrule
\end{tabular}
\end{sc}
\end{small}
\end{center}
\vskip -0.2in
\end{table*}

Our main set of experiments studies the effect of SemLC on three small-scale CNNs. We assess the different strategies in terms of the performance improvement their integration offers over the bare baseline networks on the classic image classification benchmark CIFAR-10 \footnote{\url{https://www.cs.toronto.edu/~kriz/cifar.html}} \citep{krizhevsky2009cifar}. The first network is a variant of AlexNet \citep{krizhevsky2012alexnet} built for the CIFAR-10 dataset. It comprises two convolutional and two locally connected layers, followed by a single, fully-connected softmax layer. We replace the locally connected layers with convolutional layers since the latter require substantially fewer parameters and add batch normalization \citep{ioffe2015batch} after the first two layers. The original AlexNet for CIFAR-10 is reported to achieve an error rate of $11\%$ on CIFAR-10, our reduced adaption achieves $15.09\%$. The second network, called Simple, is a variant of AlexNet with added dropout \citep{srivastava2014dropout} after the second and fourth convolutional layers and three fully connected layers in the classifier. The third architecture, Shallow, consists of only two convolutional blocks and a two-layer fully-connected classifier. By experimenting on three architectures we can test whether benefits from SemLC are exclusive to a specific network design.

With AlexNet, \citet{krizhevsky2012alexnet} also introduce local response normalization (LRN). This moving average normalizes filter responses to the same receptive field along the channel domain. As this resembles lateral connectivity but involves no connectivity profile we exclude it from the AlexNet baseline and add another baseline where the mechanism is included as a competitor to our mechanism. Additionally, we test SemLC with a Gaussian (SemLC-G) instead of a Mexican hat connectivity profile to control for the effect of the wavelet's shape.   

\paragraph{Hyperparameter Optimization} We determine a good (initial) width $\sigma$ and damping factor $\delta$ by randomly selecting $30$ unique random samples per strategy from the search space spanned by the Cartesian product of $\sigma \in \{3, 4, ..., 10\}$ and $\delta \in \{\frac{1}{10} + \frac{i}{70} | i \in \{1, 2, ..., 7\}\}$. For each random sample, the corresponding strategy is trained three times over $40$ epochs. A strategy's final set of hyperparameters is selected by averaging the best validation performances of the three independent runs. All experiments where conducted on the AlexNet adaption and then generalized to the other networks (which have the same number of filters in their first convolutional layer). We consistently (for all strategies) find $\sigma = 3$ and $\delta = 0.2$ to be the best configuration. 

\paragraph{Methodology} In all experiments, Adam \citep{kingma2015adam} serves as an optimizer with a learning rate threshold multiplied by $10$ once after $100$ and once after $150$ epochs. The training data is augmented by random cropping and horizontal flipping. Each strategy is evaluated by running $90$ independent training repetitions of $180$ epochs. From the training set we randomly sample $10\%$ of the data as a validation set. The final accuracy score of each model is the mean of the repetitions' best model accuracies on the test set. The best model is chosen based on validation accuracy. Confidence intervals are given based on the t-distribution with a $0.95$ percentile.

\subsubsection{Results}

Table \ref{tab:strategies} summarizes the results of the baseline, local response normalization (LRN), SemLC with a Gaussian profile (SemLC-G) and all three Mexican hat based SemLC strategies (SemLC, SemLC-A and SemLC-P) in the networks AlexNet, Simple and Shallow. LRN significantly improves performance of all networks \footnote{In the Shallow network, both the baseline and LRN experiments yielded outliers that substantially lower mean accuracy to $77.99 \pm 2.14$ and $77.72 \pm 2.65$ respectively. We removed them in the Table \ref{tab:strategies}'s data.}. At smaller margins, SemLC-G also significantly improves AlexNet and Simple, but has no significant effect on Shallow. Strikingly, all three Mexican hat based SemLC strategies consistently outperform the baseline models and both LRN and SemLC-G in all three networks. As part of AlexNet and Simple, the adaptive profile performs best, but both SemLC and SemLC-P are close runner-ups. Within the Shallow network, SemLC-P performs better than both alternative SemLC strategies.

\subsection{SemLC in Capsule Networks}
\begin{table}[ht!]
\caption{Added after the first (and sole) convolutional layer, SemLC is compared to LRN and the baseline in the standard CapsNet architecture on MNIST and CIFAR-10. Confidence intervals are given at $\alpha = 0.05$.}
\label{tab:capsnet}
\vskip 0.15in
\begin{center}
\begin{small}
\begin{sc}
\begin{tabular}{lrrr}
\toprule
Network             & MNIST                             & CIFAR-10         \\
\midrule
CapsNet             & $99.56 \pm .03$                   & $81.89 \pm .13$ \\
CapsNet + LRN       & $99.53 \pm .03$                   & $81.07 \pm .24$ \\
CapsNet + SemLC-G   & $99.54 \pm .02$                   & $80.57 \pm .25$   \\
\midrule
CapsNet + SemLC     & $99.56 \pm .02$                   & $82.14 \pm .22$ \\
CapsNet + SemLC-A   & $\textbf{99.58} \pm .02$                  & $81.40 \pm .25$ \\
CapsNet + SemLC-P   & $99.57 \pm .02$          & $\mathbf{82.18} \pm .19$ \\
\bottomrule
\end{tabular}
\end{sc}
\end{small}
\end{center}
\vskip -0.2in
\end{table}

To probe the performance of SemLC in a drastically different but nonetheless shallow architecture, we also incorporate it into the Capsule Networks (CapsNet) introduced by \citet{sabour2017capsules}. Just like \citet{sabour2017capsules}, we apply CapsNet to MNIST \footnote{\url{http://yann.lecun.com/exdb/mnist/}} \citep{lecun1998mnist} but also investigate performance on CIFAR-10 \citep{krizhevsky2009cifar}. CapsNet is a particular interesting case study for SemLC, because it consists of only a single convolutional layer, and as such relies heavily on the quality of its transformation. We optimize the hyperparameter $\sigma$ separately, probing $\sigma \in \{1, 3, 6, 12, 24, 32\}$ and again find $\sigma = 3$ to be the best wavelet width.

\subsubsection{Results}
Table \ref{tab:capsnet} summarizes this experiment. On MNIST, a baseline accuracy of $99.56\%$ already substantially limits possible margins of improvement. Neither LRN nor SemLC-G further improve this result and in fact both decrease accuracy by $0.03$ and $0.02$ percent points respectively. SemLC constitutes no change in accuracy, and SemLC-P and SemLC-A only numerically improve results. Applied to CIFAR-10, the situation is quite different. Whereas LRN and SemLC-G again significantly decrease CapsNet's performance, all other SemLC strategies improve the network by a small but significant margin. 

\subsection{Emerging Filter Organization}
\label{sec:order-experiments}

\begin{table}[t!]
\caption{Mean inter-filter mean squared deviation in the set of all filter pairs and the set of only adjacent filter pairs of different strategies and baselines. The percent decrease in MSD between the two sets indicates the order imposed by a strategy.}
\label{tab:mse}
\vskip 0.15in
\begin{center}
\begin{small}
\begin{sc}
\begin{tabular}{lrrr}
\toprule
Strategy             & All & Adjacent & \% Less Chaos\\
\midrule
None     & 0.231 & 0.230 &  0.4  \\
LRN    & 0.346 & 0.346  & 0.3  \\
SemLC-G & \textbf{0.413} & 0.264 & \textbf{36.1} \\
\midrule
SemLC     & 0.397 & 0.261  & 34.3  \\
SemLC-A   & 0.373 & 0.264  & 29.2  \\
SemLC-P & 0.397 & 0.256 & 35.5  \\
\bottomrule
\end{tabular}
\end{sc}
\end{small}
\end{center}
\vskip -0.1in
\end{table}

The LRN and SemLC networks trained in the previously described experiments all render filters dependent on their neighbors by introducing local connectivity. Naturally, such dependence should evoke an ordering in the filter sequence if the function of these lateral connections is not agnostic to the semantic relationship between the filter and its neighborhood. To study the emergence of such order, we quantify filter similarity by the mean squared deviation \footnote{Mean squared deviation refers to the same metric as mean squared error. We choose the different terminology to distinguish between the notions of an errors and differences.} (MSD) between their weights. Based on this metric, we first propose a numerical method of measuring induced order and then confirm the results in a qualitative analysis of the filter sequences. 

\subsubsection{Numerical Analysis}
\label{sec:numerical-analysis}

To detect order in a filter sequence we first measure the MSD of all $\frac{f(f-1)}{2}$ unique filter pairs in a sequence and then only the $f$ pairs of adjacent filters. Since our padding strategy supposes a circular ordering we also compare the last to the first filter. The proportional decrease from the mean all-pair MSD to the mean adjacent-pair MSD is then an indicator of order. Table \ref{tab:mse} reports these numbers. As expected, the base model exhibits no relevant difference between all-pair MSD and adjacent-pair MSD, reflecting a random ordering of the filter sequence. In contrast, all SemLC strategies show a clear MSD reduction, ranging from $29.2\%$ to $35.5\%$, where the parametric setting achieves the greatest reduction, but standard SemLC comes in as a close second at $34.3\%$. Notably, LRN induces no order despite locally connecting filters. SemLC-G on the other hand produces even higher order, with an MSD decrease of $36.1\%$. Observe, that additionally to the induced order, all SemLC variants and LRN constitute a higher overall inter-filter MSD. This hints towards an increased distinctiveness between the patterns they detect. 

\subsubsection{(Semi-)Qualitative Analysis}
\label{sec:qualitative-analysis}

\begin{figure}[t!]
\begin{center}
\centerline{\includegraphics[width=\columnwidth]{figures/ordering-simple-cifar10.pdf}}
\caption{The original filter order compared to the order suggested by the 2-opt algorithm \citep{Croes58two-opt} with a threshold of $0.003$. The top two plots indicate no structure, but all filter sequences that emerged under some form of SemLC clearly exhibit subsequences of optimal order.}
\label{fig:ordering}
\end{center}
\vskip -0.2in
\end{figure}

\begin{figure*}[t!]
\begin{center}
\centerline{\includegraphics[width=\linewidth]{figures/alexnet-semlc-tuning-curve.pdf}}
\caption{Average correlation of a filter to its neighbors in the full sequence, measured by cosine similarity. Each vector of cosine similarities (one per filter) is then shifted such that the filter based upon which all correlations where measured is always centered (placed at $0$). The filter sequence is that of an AlexNet with added SemLC trained on CIFAR-10.}
\label{fig:correlation}
\end{center}
\vskip -0.2in
\end{figure*}

To visualize the order in a filter sequence, we view the ordering of filters as a \textit{Traveling Salesman Problem} (TSP) where every filter is a vertex and edges are weighted by the inter-filter MSD. An optimal ordering is approximated by minimizing the total weight of a circular tour visiting all vertices. To this end, the 2-opt algorithm \cite{Croes58two-opt} can be applied to the network's order. We plot the result against the original ordering in Figure \ref{fig:ordering} by comparing indices. To better understand the meaning of such a plot consider two extremes: If the original ordering is random, the 2-opt algorithm will entirely restructure it and the plot will display randomly scattered points. If the original ordering is perfect, the two-opt algorithm will immediately stop and every filter will maintain its position in the sequence. The resultant plot is hence a diagonal line. All three SemLC variants impose some structure, but particularly those constrained to the wavelet exhibit well sorted subsequences. This aligns well with our findings from the numerical analysis.

The same conclusion can be drawn from Figure \ref{fig:correlation}. Here, we calculate the correlation of a filter to its neighbors based on their cosine similarity and average the results over all filters after centering the filter at focus. In an unordered sequence of filters, one expects a single peak in the center (since every filter maximally correlates to itself) and small correlation everywhere else (since randomly placed positive and negative correlation will level out through averaging). In Figure \ref{fig:correlation}, such a plot is created for an AlexNet with SemLC trained on CIFAR-10. It can be observed that filters have high positive correlation with their immediate two neighbors in both directions. This peak in positive correlation is flanked by a negative peak over about five filters on both sides, reflecting well the shape of the Mexican hat connectivity. The network is encouraged to place similar filters in the close neighborhood in which they mutually facilitate each others activity. Filters that correlate most negatively should instead be placed in the inhibitory zones of the connectivity profile. In consequence, their deactivation further facilitates the activation of the center neuron through the deactivation or even inversion of inhibitory connections. Filters around the range of the wavelet have low correlation, but group in shallow hills originating from similar filters co-occurring and thus conforming in their correlation with the center filter. 



\section{Introduction}
\subfile{Sections/Introduction}

\section{Related Work}
\subfile{Sections/Related}

\section{Modeling Lateral Connectivity in CNNs}
\subfile{Sections/Approach}

\section{Experiments \& Results}
\subfile{Sections/Experiments}

\section{Connectivity Profile Effects}
\label{sec:profile-impact}
\subfile{Sections/MexicanHatEffect}

\section{Discussion \& Conclusions}
\label{sec:discussion}
\subfile{Sections/Discussion}

\bibliographystyle{unsrtnat}
