\section{Introduction}
Deep Neural Networks (DNNs) achieve state-of-the-art results on various tasks such as speech recognition, object detection or machine translation (\cite{DeepLearning}). Usually, DNNs are trained on a large amount of training data in order to generalize to similar, but unseen test data. However, gathering labeled data, which is needed for supervised learning methods, is both labor-intensive and time-consuming. Thus, it is desirable to increase the data-efficiency, i.e. to ensure good performance even when training data is limited. 

Furthermore, DNNs are afflicted by some key disadvantages when being applied in fields with strict safety requirements such as autonomous driving. In these fields, DNNs need to be robust and explainable in order to enable assessing their behavior in safety-critical cases. This contradicts the current approach of threatening DNNs as a black box and training them in an end-to-end fashion.

Finally, the solution space of all parameters within deep neural networks is high-dimensional which impedes finding the optimal solution to the learning problem. Thus, it makes sense to restrict the solution space using problem-dependent, reasonable constraints.

Consequently, current research tries to combine expert knowledge - e.g. the knowledge utilized to design classical pattern recognition systems - with the architectures and optimization methods of DNNs. This is often called a \textbf{hybrid approach} aiming to combine the best of both worlds: state-of-the-art results by data-driven optimization while using as few labeled training samples as needed. Additionally, an explainable, robust system behavior is desired.

Expert knowledge can be expressed in many different ways. A simple example is knowledge about the expected size of different objects in images for object detection. A more sophisticated one is the exact definition of a filter sequence used to detect those objects. However, this kind of prior knowledge is hard to formalize which makes it difficult to include it into a neural network in a principled manner. Another possibility to incorporate knowledge is to model a prior distribution over the output of a neural network in order to predict a posterior distribution with a Bayesian neural network.  

For many tasks, certain transformations of the input which affect the desired output in a predictable way or do not affect it at all can be determined. Both, knowledge about physical transformations, e.g. when a camera is moved to a novel viewpoint (\cite{CoorsNoVA}), or knowledge about certain transformations frequently occurring in the input data, e.g. rotated and translated patterns, can be leveraged.
The latter is referred to as \textbf{geometrical prior knowledge}. For example, a system for image classification should classify a dog correctly independent of its orientation. Therefore, it needs to learn a representation that does not change when the input is rotated, i.e. it should be \textbf{invariant} with respect to rotations. A common approach to increase the robustness of a neural network to such transformations is to randomly transform the input during the training phase, which is called \textbf{data augmentation}. While this approach is straightforward to implement, the DNN only approximates the desired properties and fails to provide mathematical guarantees. 

A renowned example of incorporating geometrical prior knowledge to DNNs in a mathematically guaranteed way are convolutional neural networks (CNNs) which share a convolutional kernel among their input space. This procedure is called \textit{translational weight tying} and allows to reduce the parameter consumption of DNNs while also facilitating the DNN to recognize patterns independent of their location in the input. Therefore, CNNs are \textbf{equivariant} to translations. The success of CNNs in computer vision tasks confirms that utilizing geometrical prior knowledge is an important \textit{inductive bias} for DNNs.

In this work, we investigate the integration of geometric prior knowledge to neural networks such that their representations are mathematically guaranteed to be in- or equivariant. First, we review the mathematical concepts underlying in- and equivariant representations. We then provide an overview of several different approaches which allow to enforce those properties
by summarizing current work investigating those fields. Afterwards, we give an overview over common datasets and benchmarks which are used to compare the different presented algorithms. Subsequently, we investigate how those methods can be applied to DNNs used for autonomous driving problems. Finally, we summarize our review paper and give a short outlook on open challenges and future work. 

\textbf{Remark:} In this paper, we focus mainly on work related to perception for autonomous driving, i.e. computer vision and processing 3D sensor data. We mention some work incorporating geometrical prior knowledge to other domains, but do not claim any completeness in those fields. We do not propose any new results but hope to give a broad overview over geometrical prior knowledge for neural networks. Thereby, we hope to ease both the entry into this interesting field of research for novel researchers and the comparison of different approaches for experienced researchers. 

\section{Preliminaries}
In this section, we briefly introduce some core concepts needed to understand the work that incorporates geometrical prior knowledge to DNNs. Mainly, we discuss the mathematical concept of groups, group representation theory and steerable filters.
\subsection{Group Theory}
Groups are a mathematical abstraction used to describe invertible geometrical transformations. A group $G$ consists of a set of elements and a group operation, which combines two elements of the group to form a third. It fulfills the axioms of closure, associativity, identity and invertibility.

Group theory is the basis to mathematically describe geometrical symmetries. Therefore, it is used to define in- and equivariance with respect to transformations $g \in G$. A function $f$ is equivariant with respect to a transformation group $G$, if there exists an explicit relationship between transformations $\mathcal{T}_g^{\mathcal{X}}$ of the function's input and the corresponding transformation $\mathcal{T}_g^{\mathcal{Y}}$ of its output:
\begin{equation}
f(\mathcal{T}_g^{\mathcal{X}}[x]) = \mathcal{T}_g^{\mathcal{Y}}[f(x)] \; \forall \; x \in X \text{.}
\end{equation}

It is important to note, that $\mathcal{T}_g^{\mathcal{X}}$ and $\mathcal{T}_g^{\mathcal{Y}}$ do not need to be the same kind of transformation. For example, rotating the input might induce a shift in the output space. Nevertheless, the notation is usually simplified to
\begin{equation}
f(gx) = g^\prime f(x) \; \forall \; x \in X \text{.}
\end{equation}

Invariance is a special case of equivariance, where the transformation in output space is the identity, i.e. the output is unaffected by transformations $g \in G$:
\begin{equation}
f(gx) = f(x) \quad \; \forall \; x \in X \text{.}
\end{equation}

As mentioned before, a common example of an equivariant function $f$ are convolutional neural networks, which introduce translation equivariance.

Depending on the properties of the task to solve, equivariance is more suitable than invariance because it preserves information about the symmetry group. For example, a 3D object detector needs to be equivariant to rotations since one of the tasks is to estimate the detected objects' rotation angles. Invariance would destroy this information in the feature space, which is e.g. beneficial for the classification of those objects.

\subsection{Group Representation Theory}
The mathematical field of Group Representation Theory investigates, how a group $G$ acts linearly on a $n$-dimensional complex vector space $V$. This is an interesting theoretical field for DNNs because feature spaces can be modeled as vector spaces while symmetric input transformations can be modeled as groups. Hence, Group Representation Theory helps to understand how feature spaces of CNNs change under transformations of the input.

For matrix Lie groups, a finite-dimensional complex representation $\Pi$ of $G$ is a group homomorphism $\Pi : G \to \text{GL}(V)$, that maps the group to the general linear group $\text{GL}(n,\mathbb{C})$, i.e. to the group of invertible $n\times n$ matrices representing the linear action of the group on the vector space (\cite{HallGroupRepresentation}).

We define three common group representation types which are used throughout this survey. The \textbf{trivial} representation maps all group elements to the identity matrix, i.e. the vector space $V$ is left invariant under group transformations. 

The \textbf{regular} representation is determined by the action of $G$ on itself. In this case, $V$ is a $|G|$-dimensional vector space which is permuted under group actions $g \in G$.

A representation is called \textbf{irreducible} (\textbf{irrep}), if its only invariant subspaces are the trivial subspaces $W=V$ and $W=\{0\}$. A subspace $W$ of $V$ is called invariant, if $\Pi(g)w \in W$ for all $w \in W$ and all $g \in G$. Importantly, \textit{Maschke's theorem} states that every representation of a finite group consist of a \textit{direct sum} of irreducible representations. Consequently, finding all irreps of a specific group is an interesting aspect of group representation theory.

For a more detailed discussion of Group Representation Theory which forms the backbone of equivariant neural networks, we refer the interested reader to the recent paper by \cite{EstevesSurvey} or to the text book by \cite{HallGroupRepresentation}.

\subsection{Steerable Filters}\label{sec:SteerableFilters}
Steerable filters, proposed by \cite{Adelson}, are filters whose arbitrary rotated version $f^\theta$ can be synthesized using a linear combination of a finite number of basis filters $g_i$ . In the two-dimensional case, rotation-steerable filters are defined as:
\begin{equation}
f^\theta(r,\phi) = \sum_{i=1}^{M}k_i(\theta)g_i(r,\phi)\text{,} 
\end{equation}
where $\theta$ is the rotation angle and $k_i(\theta)$ are called interpolation functions. Steerable filters are defined in the polar space $r=\sqrt{x^2+y^2}$, $\phi = \arctan(|\frac{y}{x}|)$. 

Steerable filters provide the advantage that arbitrary rotated versions of them can be calculated analytically without suffering from sampling effects. This is important for computer vision tasks, where multiple rotated versions of a filter are applied or learned frequently. Additionally, steerable filters can also be expanded towards other transformations in order to facilitate the mentioned advantages for arbitrary transformations.

\section{Geometrical Prior Knowledge for Deep Neural Networks}
For many machine learning tasks, experts can identify transformations, which should affect the output of the algorithm either in a predictable way or not at all. In literature, this kind of expert information is called \textbf{geometrical prior knowledge}.
 
In general, incorporating geometrical prior knowledge to neural networks restricts the solution space of the neural network's learning algorithm. If the prior knowledge is carefully chosen, i.e. it is relevant for the task the network is trying to solve, this helps to raise the data-efficiency of the learning process. Since the available amount of training data is finite - especially for small datasets - incorporating geometrical prior knowledge leads to an improved overall performance of DNNs.

A variety of current research investigates how to incorporate geometrical prior knowledge by enforcing in- or equivariance properties to the neural network. A well-known example are \textbf{Convolutional Neural Networks} (CNNs) which share a convolutional kernel among all positions of the input image or feature map (\cite{Fukushima, LeCun}). This process, called \textit{translational weight tying} makes their response equivariant with respect to translations. Translation equivariance is one of the key properties responsible for the success of CNNs in various practical applications. However, standard convolutions are not equivariant to other transformations such as rotations. Thus, changing or generalizing the convolution to enable equivariance to arbitrary transformations occurs as a promising approach to increase the data-efficiency of CNNs.

We separate the presented ideas into approaches using \textbf{special filter types} to enforce in- or equivariance properties, approaches relying on the \textbf{group convolution} which is equivariant to a group by design and \textbf{learned feature transformations} which try to learn invariant features from data without explicitly enforcing it. Additionally, we explore \textbf{capsules}, \textbf{invariant integration} and other approaches, which do not fit one of the aforementioned categories, but nevertheless provide methods to incorporate geometrical prior knowledge.

\subsection{Restricted Filters}

One possibility to enforce in- or equivariance properties to functions is using restricted or fixed filters to process the input signal. In this section, we introduce methods based on the \textbf{scattering transformation} as well as methods based on circular or spherical \textbf{harmonic filters}. 

\subsubsection{Scattering Neural Networks}
\begin{table*}[t]
	\centering
	\begin{tabular}{|c|c|c|c|}
		\hline
		Reference & Invariance group & Classifier & Learnable \\ \hline
	
		\cite{Scattering} & $\mathbb{Z}^2$ & SVM & \ding{55} \\
	
		\cite{GenericScattering} & $\mathbb{Z}^2$ & Linear SVM &  \ding{55} \\
	
		\cite{DeepHybridNetworks, OyallonCompressingScattering, Scattering2} & $\mathbb{Z}^2$ & CNN & \ding{55} \\
	
		\cite{Cotter15, Cotter2} & $\mathbb{Z}^2$ & CNN & \checkmark\\
	
		\cite{RotoTranslationScattering} & $\mathbf{SE}(2)$ &  Gaussian SVM & \ding{55} \\
	
		\cite{Sifre} & $\mathbf{SE}(2) \rtimes S$ & PCA classifier  & \ding{55} \\ 
		\hline
	\end{tabular}
	\caption{Overview of Scattering CNNs classified by the invariance group $G$, the classifier used on top of the scattering coefficients and if the scattering coefficients are learnable.}
	\label{tab:Scatter}
\end{table*}

\cite{Scattering} propose to use a hand-crafted transformation called scattering transformation that enforces invariance to the representations of a DNN. It consists of a convolution of the input signal $x$ with a family of wavelets $\psi$ followed by a modulus non-linearity and an averaging filter $\phi$. The first-order scattering coefficients $S^{1}(x)$ at rotation and scale $\lambda_1 = \{\theta_1, j_1\}$ are defined as:
\begin{equation}
S^{1}(x) = |x\star\psi_{\lambda_1}|\star\phi_J \text{,}
\end{equation}
where the family of wavelets is computed by rotating and dilating a complex mother-wavelet $\psi$ using $L$ rotations and $J$ scales.
Furthermore, it is possible to cascade multiple scattering transforms to obtain coefficients of order $m\geq0$: 
\begin{equation}
S^{m}x = | \hdots || x \star \psi_{\lambda_1}| \star \psi_{\lambda_2}| \hdots  \star \psi_{\lambda_m}| \star\phi_{J} \text{.}
\end{equation}

\cite{Scattering} use complex Morlet wavelets to obtain the scattering coefficients and process the features with a Support Vector Machine (SVM) for classification. While the calculated scattering coefficients are invariant to translations and stable to small deformations, they are not invariant to other transformation groups such as rotations.

\cite{GenericScattering} compare a two-layer scattering network for image classification to a DNN of the same depth and conclude that the scattering transformation closely resembles the filters that are learned within the early layers of a CNN during training.

\cite{GroupInvScattering} enhances the scattering operator to arbitrary transformations which can be represented as compact Lie groups $G$. He proves that the group scattering transformation is invariant under the action of $G$. Moreover, he proposes to apply this to the group of rotations and translations in 2D, which is conducted by \cite{RotoTranslationScattering}. 

\cite{Sifre} further advance this approach by cascading multiple scattering transformations with global space-scale averaging and invariant linear projectors to obtain a representation that is invariant to translation, rotation, scaling and small deformations.  

Finally, \cite{DeepHybridNetworks,Scattering2} propose to use a hybrid approach: The lower layers of a CNN are replaced by fixed scattering transforms while the upper layers are learned. Thus, the lower layers provide translation-invariant features without the need to learn the filters while the higher layers learn more abstract features. They show that the proposed method enables a performance boost when training on a limited subset of the entire training data. This demonstrates that the scattering transform does not loose any discriminative power while boosting the performance in the limited sample domain by incorporating geometrical prior knowledge.

The scattering transformation can also be used to compress the input images processed by a CNN to reduce the spatial resolution. While this saves parameters and training time, it still captures most of the signal information needed for classification permitting state-of-the-art classification results (\cite{OyallonCompressingScattering}).

\cite{Cotter} visualize the patterns that caused a large activation in each channel of a hybrid scattering CNN in order to further compare the filters used by scattering networks to those learned by conventional CNNs. Therefore, they use a method called DeScatterNet which sequentially inverts the operations of the scattering layers. They find that the CNN filters are more general, i.e. they are able to detect more complex shapes. To resolve this discrepancy, they propose to replace the averaging filters of the original scattering transformation by a more general filter which also has a spatial domain. They show that this procedure allows the scattering network to detect more complex shapes like corners, crosses and curves. 

Additionally, \cite{Cotter2} propose a learnable form of the scattering transformation called locally invariant convolutional layer obtained by multiplying the output of the scattering transformation with learnable weights. They prove that the layer is differentiable with respect to the learnable weights and its input which makes it suitable to use for end-to-end training via the backpropagation algorithm. They apply their novel layer at different depths of a CNN and conclude that it works best, when it is used in an early layer but not within the first layer of the CNN. 

To summarize, the scattering transformation allows to incorporate symmetry properties to neural networks while guaranteeing stability to small deformations, which is a desirable property for many learning tasks. While early proposals use the transformation in combination with classifiers such as the SVM, novel approaches try to include it into DNN architectures. One approach processes the signal with the scattering transformation and uses the calculated scattering coefficients as the input to a DNN, which can be interpreted as introducing hand-crafted filters to lower CNN layers. Another approach is to introduce learnable parameters to the scattering layer itself while ensuring that it is differentiable with respect to its input, making it suitable for end-to-end learning.

\subsubsection{Harmonic Filters}\label{sec:RestrictedFilters}
\cite{HarmonicNetworks} train a CNN where the filters are restricted to circular harmonics so that the convolution's output is equivariant with respect to continuous rotations. In particular, the filters are defined in the polar domain as
\begin{equation}
\mathbf{W}_m(r,\phi,R,\beta) = R(r)e^{i(m\phi+\beta)} \text{,}
\end{equation}
where $r$ and $\phi$ are the radius and angle of the polar representation, $m$ is the rotation-order while $R$ and $\beta$ are the learnable parameters of that layer, namely the radial profile and the phase offset. The filters are defined in the complex domain which automatically induces a complex output of the convolution operation that can easily be calculated using four standard convolutions. While the output's modulus is invariant to rotations of the input, the phase stores the equivariant information about its orientation. 

\cite{TensorFieldNetworks} use a mix of continuous convolutions on point clouds and filters restricted to spherical harmonics to learn representations that are locally equivariant to 3D rotations, translations and point permutations. Point-wise convolutions are used to process vector fields defined on points. The convolution is calculated using the equivariant tensor product. It can be calculated using the Clebsch-Gordan coefficients and the irreducible representations of $SO(3)$.

\subsection{Group Convolution}
\textit{Group equivariant convolutional neural networks} (G-CNNs), first proposed by \cite{GroupEquivariantCNNs}, enforce equivariance to symmetric transformations by replacing the standard convolution with a generalized form, called group convolution. In this section, we first introduce the group convolution by deriving it from the standard convolution. Afterwards, we introduce papers concerning the theory of group equivariant neural networks. Finally, we show how the group convolutional framework is applied to specific transformation groups.

The discrete standard convolution acting on regular input data in two-dimensions is defined as\footnote{To be precise, the formula describes the discrete standard correlation. In neural network literature however, it is constantly referred to as convolution since its slightly easier implementation and the only difference being a flipped kernel. We will follow this nomenclature and subsequently call the presented operations convolution, although they technically are correlations.}:
\begin{equation}
(f\star k)(u) = \sum_{v\in\mathbb{Z}^2} f(v)k(v-u) \text{,} 
\label{eq:Conv}
\end{equation} 
where the convolution kernel $k$ with spatial support $v$ is shifted (translated) over the domain $\mathbb{Z}^2$ of the input function $f$. Usually, the kernel's spatial support is significantly smaller than that of the input domain which results in two advantages: First, the number of parameters which need to be learned during training is reduced. Additionally, the same convolution kernel is applied at each position of the function's domain, which is called spatial parameter sharing and induces equivariance to translations $L_t$:
\begin{align}
\begin{split}
[[L_t f]\star k](u) &= \sum_{v}f(v-t)k(v-u) \\
&= \sum_{v}f(v)k(v-(u-t)) \\
&= [[L_t[f\star k]](u)  \text{.}
\end{split}
\label{eq:Proof_Equivariance}
\end{align}

\textbf{Group convolutions} are the generalization of the standard convolution for an arbitrary transformation group G. The continuous group convolution is defined as:
\begin{equation}
\label{eq:GConvCont}
(f \star_G k)(u) = \int_{G}f(g)k(u^{-1}g)\; d\mu(g)\text{,}
\end{equation}
where $\mu$ is the Haar measure.
The shift, i.e. the action of the translation group, in Formula \ref{eq:Conv} is replaced with the action $g$ of a transformation group $G$. The discrete version is defined as:
\begin{equation}
(f\star_G k)(u) = \sum_{g\in G}f(g)k(u^{-1}g) \text{.} \label{eq:GConv}
\end{equation}
It is easy to see, that the group convolution acts on the domain of the group $G$. Consequently, its input and filters are defined on $G$ as well. 
Again, we can prove that the group convolution defined for the group $G$ is equivariant with respect to transformations $L_h$ of that group:
\begin{align}
\begin{split}
[[L_h f]\star_G k](u) &= \sum_{g\in G}f(h^{-1}g)k(u^{-1}g) \\
&= \sum_{g\in G}f(g)k((h^{-1}u)^{-1}g) \\
&= [[L_h[f\star_G k]](u)  \text{.}
\end{split}
\end{align}

Since the input data is usually not defined on the group manifold (i.e. images lie on a regular grid), a \textbf{lifting layer} needs to be defined, which lifts the input, e.g. from $\mathbb{Z}^2$ in images, to the group domain:
\begin{equation}
(f\star_G k)(u) = \sum_{v \in\mathbb{Z}^2} f(v)k(u^{-1}v) \text{,} \label{eq:GConvLifting}
\end{equation}
where $(f\star_G g) \in G$ and $k$ is a learnable filter defined in the input domain $\mathbb{Z}^2$. 

\subsubsection{Theoretical Aspects of Enforcing Equivariance}
In general, CNNs do not only consist of multiple linear filter operations, but also use non-linearities. It can be shown, that if each single operation enforces or preserves equivariance, the whole neural network is equivariant. \cite{GroupEquivariantCNNs} prove that point-wise non-linearities and pooling operations acting on a region which is a subgroup $H\subset G$ preserves equivariance of the network. Hence, it is sufficient to prove equivariance for the convolutional layers used within a G-CNN, if no other non-linearities are used. 

While \citet{GroupEquivariantCNNs} primarily propose to use G-CNNs focused on an application, i.e. for rotation equivariant classification (see Section \ref{sec:GCNNApps}), several works investigate general theoretical frameworks that enable the classification of existing G-CNNs as well as the expansion towards more complex groups or other input domains, e.g. graphs (\cite{SteerableCNNs, Kondor, GeneralEquivariantCNNs, EstevesSurvey}). 

\citet{Kondor} provide a mathematical prove that a DNN is equivariant to the action of a compact group $G$ of its input if and only if each layer implements the generalized group convolution defined in Equation \ref{eq:GConv}. For their proof, they use concepts from group representation theory. Additionally, it is proven that those convolutions can be defined over quotient spaces of the transformation group $G$ because the network activations are usually defined on homogeneous spaces associated to the group. Finally, they emphasize the benefits of calculating convolutions using irreducible Fourier space representations which lead to sparse matrix multiplications when defined on the appropriate quotient spaces. 

\cite{GeneralEquivariantCNNs} expand Kondor and Trivedi's theory by defining convolutional feature spaces as fields defined over a homogeneous \textit{base space}. This enables a systematic classification of various equivariant convolutional layers based on their symmetry group, the base space the features are attached to and the field type, which can be regular, trivial or irreducible. The field type is closely related to group representation theory and describes, how the values change under transformations of the symmetry groups. Trivial representations are invariant, i.e. the individual values do not change under transformations, the values of regular fields are shifted along the group dimension and irreducible fields change according to the irreducible representation of the transformation group. 
Finally, \cite{GeneralEquivariantCNNs} use the Mackey theory to prove that a convolution with equivariant kernels is not only a necessary and sufficient condition for equivariance, but that this is also the most general class of equivariant linear maps. Finally, they propose a general framework on how to parameterize the learnable equivariant convolutional kernels within DNNs.
 
\cite{EstevesSurvey} gives a broad overview over the mathematical concepts underlying group equivariant CNNs, which are: group representation theory, integration and harmonic analysis on non-Euclidean spaces, and differential geometry. Esteves shows applications of his theory for the group $\mathbf{SO}(3)$ and elucidates that spherical harmonics are directly related to the irreducible representations which arise when calculating the convolution using the Fourier transformation. Finally, based on results from group representation theory, both Kondor and Trivedi's and Cohen et al.'s definition of group equivariant CNNs are discussed.

As mentioned in section \ref{sec:SteerableFilters}, \textbf{steerable filters} can be calculated in arbitrary rotations using a linear combination of basis filters. Similar to the convolution, this principle can be expanded to arbitrary transformations. Consequently, steerable filters provide an attractive alternative to avoid interpolation needed for the computation of arbitrarily transformed filters in equivariant CNNs. 

\cite{SteerableCNNs} propose a general framework where the features of an equivariant CNN are a steerable composition of feature types which encode certain transformation symmetries. This implies that the learned representations transform in a predictable linear manner under transformations of the input. The choice of elementary feature types imposes constraints on the network weights and architecture which reduces the number of learnable parameters. Cohen and Welling use concepts from group representation theory to describe the mappings learned by an equivariant CNN. Feature maps are modeled as fibers at each position of the base space (see Figure \ref{fig:fibers}). An equivariant map processing those fibers is called \textit{intertwiner} and can be expressed as a direct sum of \textit{irreducible representations}. Finally, they elaborate that the steerable CNN framework can be expanded to other forms of homogeneous spaces, e.g. graphs. 

\begin{figure}
	\centering
	\includegraphics[width=0.8\linewidth]{img/fibers.png}
	\caption{A typical CNN feature space can be divided into feature maps (left) or fibers at each position (right) \cite{SteerableCNNs}.}
	\label{fig:fibers}
\end{figure}

\subsubsection{Applications}\label{sec:GCNNApps}
In this section, we present applications of group convolutional neural networks arranged by their equivariance group.  Additionally, we categorize the approaches following the procedure proposed by \cite{GeneralEquivariantCNNs} (see Table \ref{tab:Equi}).

\textbf{2D rotations and translations:} First, we present approaches equivariant to 2D translations and rotations. \cite{GroupEquivariantCNNs} apply the group convolution to the finite wallpaper groups $p4$ and $p4m$ consisting of 2D translations, $90^\circ$ rotations and flips. They show that by transforming the learned filters with all possible rotations and flips and stacking them along the \textit{group dimension}, they can apply the standard convolution to obtain the desired equivariance. An exemplary feature space is depicted in Figure \ref{fig:p4conv}.

\begin{figure}
	\centering
	\includegraphics[width=0.7\linewidth]{img/p4_conv}
	\caption{A feature space obtained by applying a $p4$ group convolution and its rotation \cite{GroupEquivariantCNNs}.}
	\label{fig:p4conv}
\end{figure}

\cite{Bekkers} propose to expand the work by Cohen and Welling to the discrete group $\mathbf{SE}(2, N) \leqslant \mathbf{SE}(2)$, which includes $N$ two-dimensional rotations and all possible translations by applying bilinear interpolation to the points not lying on the regular grid during filter rotation. In their experiments, the best results are achieved using $N=8$ or $N=16$ rotation angles.

Rotation-equivariant Vector Field Networks (\cite{MarcosRot}) reduce the parameter and memory consumption of rotation-equivariant CNNs by only storing the maximum activation and the orientation that caused it at each location. This can be seen as a vector field where the activation is responsible for the vector's length while the orientation is obtained from the corresponding rotation of the filter. The authors conclude that only storing dominant responses and orientations seems sufficient to solve in- and equivariant tasks.

Harmonic networks (\cite{HarmonicNetworks}), mentioned in section \ref{sec:RestrictedFilters}, can also be considered from a group convolutional viewpoint since the circular harmonics are the irreducible representations of the two-dimensional rotation group.

While \cite{SteerableCNNs} implement the steerable CNN framework for the group $p4m$ to obtain steerable features, a variety of other approaches rely on using steerable filters for the convolutional kernels.

\cite{SFCNN} restrict the filters to a linear combination of steerable filters, learn the linear coefficients and apply the $\mathbf{SE}(2,N)$ group convolution. Using steerable filters enables an efficient calculation at arbitrary rotations without suffering from sampling artifacts while the group convolution ensures the desired equivariance properties. Additionally, a generalized weight initialization scheme is proposed that further improves the performance of the layer.

\cite{E2STCNNs} generalize steerable CNNs to the Euclidean group $\mathbf{E}(2)$, which includes rotations, translations and flips in the image plane. The equivariant convolution kernels of $\mathbf{E}(2)$-CNNs are constrained through the symmetries they are designed to express. Those constraints can be formulated using the \textit{irreducible representations} of the symmetry groups. Hence, one can calculate a basis of equivariant kernels by solving the irrep constraints, reverting the change of basis and taking the union over the different blocks. This results in a general formulation of the kernel constraint using a Fourier basis which leads to harmonic basis elements. Additionally, the proposed framework allows for weaker group restrictions, i.e. to subgroups, for higher layers of G-CNNs which might not benefit from a full symmetry.

In contrary to the steerable filter approach, \cite{Diaconu1} propose to replace the interpolation needed to obtain filters at arbitrary rotations by learning a filter basis and all its rotated versions through rotation-invariant coefficients. Therefore, they propose the \textbf{unitary group convolution}:
\begin{equation}
[f\star_G k](g) = \sum_{x\in X} f(x)L_g[k](x) \text{,}
\end{equation}
which is only equivariant, if the inner product is unitary, i.e. it is constant if both $f$ and $\psi$ are transformed by the same transformation group element $g$. The rotated filters are then learned using a novel equivariance loss. The unitary group convolution shows improved robustness to rotated activations and guarantees a better representation stability than the group convolution. 

\cite{Diaconu2} investigate combining the roto-translation group convolution with a self-attention mechanism to obtain equivariant data-dependent filters. By combining two contemporary fields of research, the proposed networks achieve improved results compared to each method on their own while reducing the number of model parameters - allowing to use data-dependent filters and incorporating geometrical prior knowledge at once.

Inspired by the human visual system which is not fully rotation equivariant, \cite{RomeroCoAttentive} propose to use an attention mechanism to learn co-occurring transformations within the dataset. Thereby, the network can exploit information about co-occurring transformations without disrupting its global equivariance. For example, a network for face detection should exploit the relative orientation between the eyes and the nose. The authors show that this concept improves discrete rotation equivariant CNNs in both full and partial rotation equivariant tasks.



\begin{table*}[t]
	\centering
	\begin{tabular}{|c|c|c|c|}
		\hline
		Reference & Equivariance group & Base space & Vector field type \\ \hline
	
		\cite{Fukushima, LeCun} & $\mathbb{Z}^2$ & $\mathbb{Z}^2$ & regular \\
	
		\cite{GroupEquivariantCNNs} & $p4, p4m$ & $\mathbb{Z}^2$ &  regular \\
	
		\cite{SteerableCNNs} & $p4, p4m$ & $\mathbb{Z}^2$ &  irrep \& regular \\
	
		\cite{Diaconu2} & $p4$ & $\mathbb{Z}^2$ & regular \\
	
		\cite{RomeroCoAttentive} & $p4$ & $\mathbb{Z}^2$ & regular \\
	
		\cite{Bekkers} & $\mathbf{SE}(2, N)$ &  $\mathbb{Z}^2$ & regular \\
	
		\cite{SFCNN} & $\mathbf{SE}(2, N)$ &  $\mathbb{R}^2$ & regular \\
	
		\cite{Diaconu1} & $\mathbf{SE}(2, N)$ & $\mathbb{R}^2$ & regular \\	 
	
		\cite{MarcosRot} & $\mathbf{SE}(2, N)$ &  $\mathbb{R}^2$ & irrep \& regular \\
	
		\cite{HarmonicNetworks} & $\mathbf{SE}(2)$ & $\mathbb{R}^2$ & irrep \\
	
		\cite{E2STCNNs} & $\mathbf{E}(2)$ & $\mathbb{R}^2$ & any \\
	
		\cite{SphericalCNNs} & $\mathbf{SE}(3) $ & $S^2$ & regular \\
	
		\cite{Esteves, SphericalHourglass, EquivariantMultiView} & $\mathbf{SE}(3) $ & $S^2$ & trivial \\
	
		\cite{Perraudin} & $\mathbf{SE}(3) $ & $S^2$ & trivial \\
	
		\cite{DefferrardDeepSphere} & $\mathbf{SE}(3) $ & $S^2$ & trivial \\
	
		\cite{Jiang} & $\mathbf{SE}(3) $ & $S^2$ & irrep \\
	
		\cite{ClebschGordanNets} & $\mathbf{SE}(3)$ & $S^2$ & irrep \\  
	
		\cite{Winkels} & $D_4$, $D_{4h}$, $O$, $O_h$ & $\mathbb{Z}^3$ & regular \\
	
		\cite{CubeNet} & $S_4$, $T_{4}$, $V$ & $\mathbb{Z}^3$ & regular \\
	
		\cite{TensorFieldNetworks} & $\mathbf{SE}(3) $ & $\mathbb{R}^3$ & irrep \& regular \\
	
		\cite{3DSTCNNs} & $\mathbf{SE}(3)$ & $\mathbb{R}^3$ & irrep \\ 
	
		\cite{XuScale} & $\mathbb{R}^2 \rtimes S$ & $\mathbb{R}^2 $ & regular \\
	
		\cite{LocalScaleInvariance} & $\mathbb{R}^2 \rtimes S$ & $\mathbb{R}^2 $ & trivial \\
	
		\cite{MarcosScale} & $\mathbb{R}^2 \rtimes S$ & $\mathbb{R}^2 $ & irrep \& regular \\ 
	
		\cite{Worrall19} & $\mathbb{R}^2 \rtimes S$ & $\mathbb{R}^2 $ & regular \\
	
		\cite{ZhuScale} & $\mathbb{R}^2 \rtimes S$ & $\mathbb{R}^2 $ & regular \\
	
		\cite{GhoshScale} & $\mathbb{R}^2 \rtimes S$ & $\mathbb{R}^2 $ & regular \& trivial \\
	
		\cite{Sosnovik} & $\mathbb{R}^2 \rtimes S$ & $\mathbb{R}^2 $ & regular \\
	
		\cite{IssakkimuthuS19} & $\mathbf{SE}(2) \rtimes S$ & $\mathbb{R}^2 $ & irrep \\
	
		\cite{GIFT} & $\mathbf{SE}(2) \rtimes S$ & $\mathbb{R}^2$ & trivial\\
	
		\cite{BekkersLie} & $\mathbb{R}^2 \rtimes H$ & $\mathbb{R}^2$ & regular \\
		\hline
	\end{tabular}
	\caption{Overview of in- and equivariant CNNs classified by the equivariance group $G$, the base space $G/H$ and the field type $\rho$. Based on \cite{GeneralEquivariantCNNs}.}
	\label{tab:Equi}
\end{table*}


\textbf{3D rotations and translations:}
A variety of approaches expand equivariance towards 3D signals and groups. Since finite groups grow exponentially with increasing dimension, sophisticated analytical solutions for equivariance are even more crucial than in the 2D case. 

First, we present work which achieves equivariance to 3D rotations and translations for inputs defined on the sphere $S^2$, e.g. spherical images or global climate models.
 
\cite{SphericalCNNs} propose a spherical correlation incorporating the aforementioned equivariance properties. Therefore, they utilize the group Fourier transformation that allows to efficiently compute the convolution in the Fourier domain using a multiplication. \cite{Esteves} independently suggest to calculate the spherical convolution in the Fourier domain but use trivial instead of regular group representations. Both apply the spherical convolution for 3D shape recognition. 

\cite{SphericalHourglass} use the spherical convolutional layer to enforce invariance to the camera-pose in a DNN for semantic segmentation of spherical images. \cite{EquivariantMultiView} create a group representation from multiple views of a single object and process it with spherical CNNs in order to encode shape information equivariant to the icosahedral group. Additionally, they use the log-polar transformation which allows to use less viewpoints by exploiting the equivariance of related viewpoints since in-plane rotations act as translations in the log-polar domain.

\cite{Perraudin} approximate a spherical convolution by treating the discrete sphere as a graph and applying a graph convolution. Their approach, called DeepSphere, achieves equivariance to 3D rotations by restricting the learned filters to be radial and is applied to cosmology images. While this approach fails to achieve exact equivariance, it allows to significantly reduce the computational complexity of spherical convolutions. In a subsequent paper, \cite{DefferrardDeepSphere} propose minor improvements of DeepSphere and show that the number of connected neighbors on the spherical graph can be used to trade-off between equivariance guarantees and computational complexity.

Comparably, \cite{Jiang} process the sphere as a unstructured grid (mesh) and apply a mesh convolution. Rotation-equivariance is guaranteed by using a linear combination of parameterized differential operators. While the weights of the linear combination are learnable, the differential operators are efficiently estimated using one-ring neighbors. The operation itself is applicable to arbitrary unstructured grids and shows promising results for tasks in the spherical domain while being parameter-efficient.

\cite{ClebschGordanNets} further generalize spherical $\mathbf{SO}(3)$-equivariant CNNs by proposing to use the Clebsch-Gordan transformation as a general purpose nonlinearity for rotation-equivariant spherical CNNs. The Clebsch-Gordan transformation, which decomposes a tensor into a product of its irreducible representations, was first used in neural networks for rotation-symmetric physical systems by \cite{Kondor1}. Since the transformation is calculated directly in the Fourier space, it avoids the frequent calculation of forward and backward Fourier transformations after each layer. Moreover, \cite{ClebschGordanNets} generalize the Fourier transformation to compact continuous groups.

Multiple proposals achieve equivariance to 3D rotations and translations for three-dimensional inputs. \cite{Winkels} transfer the group convolution of \cite{GroupEquivariantCNNs} to 3D by transforming the filters using the finite 3D groups $D_4, D_{4h}, O$ and $O_h$ which describe the symmetries of cuboids and cubes, respectively. \cite{CubeNet} independently propose the same procedure using the cube group $S_4$ containing 24 right-angle rotations, the tetrahedral group $T_4$ including the 12 even rotations and Klein's four-group $V$, which is commutative and the smallest non-cyclic group.

\cite{3DSTCNNs} expand the theory of steerable equivariant CNNs to 3D. They propose to parameterize 3D rotation steerable convolutional kernels as linear combinations of spherical harmonics with learnable weights. Additionally, they show that only the angular part of the spherical harmonics is restricted by the equivariance constraints, while the radial part can be chosen arbitrarily.

\textbf{Scale}: Another interesting transformation that naturally occurs in many Computer Vision tasks is scale, which for example results from variable recording distances. Consequently, investigating how to expand group-equivariant CNNs to scaling transformations seems beneficial. Compared to rotations, this is more challenging because the scale group is not cyclic and unbounded. Additionally, scaling is often modeled as a semi-group due to the information loss when down-scaling an image which makes the group action non-invertible. In mathematical terms, scale transformations are often called dilation.

\cite{XuScale} use a fairly similar approach to G-CNNs to obtain scale invariance in a convolutional neural network which they call multi-column CNN. The core idea is to share scaled versions of the same convolutional filter among different columns, each resulting in a maximum response at a different scale of the same pattern. The scaled versions of the filters are calculated using bilinear interpolation for upscaling while the minimum $L_2$-norm filter is used for downscaling. The columns, which can be seen as the group transformation channel in the G-CNN framework, process the input independently, resulting in a column-flip when a pattern is scaled. Finally, the column activations are concatenated and used for classification.

\begin{figure}
	\centering
	\includegraphics[width=0.85\linewidth]{img/ScaleInvariantConv}
	\caption{A scale-invariant convolution layer proposed by \cite{LocalScaleInvariance}.}
	\label{fig:ScaleConv}
\end{figure}

Conversely, \cite{LocalScaleInvariance} process multiple scaled versions of the same input obtained using bilinear interpolation at each convolutional layer and keep the maximum response at each location by applying the max pooling operation over all scales (see Figure \ref{fig:ScaleConv}). Thereby, each layer guarantees local scale-invariance. While this approach leads to promising results, it involves two transformations of the input per convolution layer. \cite{MarcosScale} expand this approach by additionally storing the information about which scale caused the maximum response at each location. Thereby, they disentangle the scale and the magnitude of the response, which is stored as a vector field. Higher convolutional layers process both information at once using a vector field convolution. 

\cite{GhoshScale} adapt the local scale-invariant convolutions of Kanazawa et al. by transforming scale-steerable filters instead of the signal itself. The filters are composed of linear combinations of log-radial harmonics, an adapted version of the circular harmonics. By using the steerable filter approach, interpolation artifacts are avoided which leads to an improved scale robustness and performance. Additionally, the second scale operation of Kanazawa's approach is avoided because the filters are transformed instead of the signals.

\cite{Worrall19} propose to formally enhance the group convolution such that it can be applied to transformations modeled as a semigroup:
\begin{equation}
	[k \star_S f](s) = \sum_{x\in X} k(x)L_s[f](x) \text{.}
\end{equation}
In comparison to the group convolution (Equation \ref{eq:GConv}), the signal is transformed instead of the filter, which is similar to the approach used by \cite{LocalScaleInvariance}. Again, a transformation acting on the input induces a shift in the semi-group convolution's output. By defining the input images as scale-spaces, i.e. the input and multiple blurred versions of it, and modeling dilations as a discrete semigroup, a scale equivariant NN can be defined. \cite{Worrall19} restrict their approach to integer scaling to avoid interpolation when creating the scale space, which on the downside leads to unwanted boundary effects. 

\cite{ZhuScale} propose a scale-equivariant CNN using joint convolutions across the space and scaling group. In order to reduce the model complexity, they decompose the convolutional filters under two pre-fixed separable Fourier-Bessel bases with trainable expansion coefficients. At the same time, they truncate the filters to low frequency components which leads to an improved deformation robustness and a reduced parameter consumption.

\cite{Sosnovik} use steerable filters and scale-equivariant group convolutions in combination with a discrete number of scales. This approach allows to efficiently calculate filters for arbitrary real-valued scaling factors. Two-dimensional Hermite polynomials with a Gaussian envelope are used as the steerable filter basis. As usual for steerable filter CNNs, the filter basis is pre-computed while the weights of the linear combinations are learned using the backpropagation algorithm. While the network could learn from inter-scale interactions in theory, the experiments show that the DNNs perform better using limited inter-scale interactions.

\cite{IssakkimuthuS19} propose to learn rotation-equivariant filter bases using an autoencoder with $\mathbf{SO}(2)$-equivariant mapping and tensor nonlinearities. Additionally, scale-coupled bases are used in combination with the tensor nonlinearity to obtain scale-robust filter bases. Finally, they show that the learned bases closely resemble their Fourier counterparts and allow for good performance in classification tasks.

\cite{GIFT} use a two-step approach to obtain pixel-wise visual descriptors invariant to scale and rotations, which can be used to find correspondence points in a set of images. First, they process transformed versions of the input image using a standard CNN. Second, they use the group convolution for rotations and scales as well as bilinear group pooling to obtain invariant features, which they call GIFT descriptors.

\textbf{Arbitrary Groups}: 
\cite{Gauge} extend group convolutions to local gauge transformations, which enables equivariant CNNs on manifolds instead of on homogeneous base spaces. The convolutions only depend on the intrinsic geometry of the manifold. Compared to previous work, equivariance is guaranteed with respect to local transformations instead of global ones. This framework is applied to signals on a icosahedron which is an approximation of the sphere.

\cite{Finzi2020} propose a convolutional layer equivariant to transformations which can be expressed as a Lie group based on a surjective exponential map. The novel layer can be applied to arbitrary continuous data, including regular grids (images) and point clouds. Equivariance to a new group can be achieved by only implementing the group exponential and logarithm maps and using the general layer framework. Hence, the proposed layer is flexible and can be used for a variety of different problem settings.

Comparably, \cite{BekkersLie} introduces a convolutional layer equivariant to groups that are a semidirect product of the translation group and arbitrary Lie groups. Therefore, the convolutional kernels are parameterized using B-spline basis functions defined on the Lie algebra of the corresponding Lie group. By altering the respective type of the B-spline basis functions localized, dilated and deformable group convolutions can be implemented. 

In summary, group convolutions have been applied to a variety of transformations in both two and three dimensions. While most applications thus far are limited to simple transformations on homogeneous base spaces, e.g. scale and rotations, current research investigates how to generalize the group convolution to more complex groups and input domains. 

\subsection{Learned Feature Transformations}
\begin{figure}[t]
	\centering
	\begin{subfigure}[b]{\linewidth}
		\centering
		\includegraphics[width=0.8\linewidth]{img/SpatialTransformer}
		\caption{Spatial Transfomer Network (\cite{STN})}
		\label{fig:STN}
	\end{subfigure}
	\begin{subfigure}[b]{\linewidth}
		\centering
		\includegraphics[width=0.85\linewidth]{img/EquivariantTransformer}
		\caption{Equivariant Transformer Network (\cite{ETN})}
		\label{fig:ETN}
	\end{subfigure}
	\caption{Architecture of spatial and equivariant transformer networks \cite{ETN}.}
\end{figure}
Another possibility to obtain a transformation invariant feature space is to estimate the transformations acting on the input in order to subsequently re-transform it to its \textit{canonical} form. Moreover, this procedure can be applied in the feature space of DNNs - not only on the input. Hence, we refer to this approach as \textbf{learned feature transformations}.

Spatial Transformer Networks (STNs, \cite{STN}) implicitly learn invariance to affine transformations - including translation, rotation and scale - at once without additional supervision. A three-step approach is used to estimate and perform transformations of the input or feature spaces of CNNs to obtain invariant representations: A localization network estimates the transformation parameters of an affine transformation, which is used to calculate a parameterized sampling grid. A differentiable image sampling, e.g. bilinear interpolation, is then used to reverse transform the input (see Figure \ref{fig:STN}). This facilitates the usage of the backpropagation algorithm to train both the convolution operator as well as the localization network at once.  

\cite{PolarTN} enhance STNs by using a polar coordinate transformation to learn features for object classification that are invariant to translations and equivariant to rotations and scale. A CNN predicts the object center which is used as the origin for the log-polar transformation of the input. Effectively, the transformed representation is invariant with respect to the object location. Additionally, rotations and scales in the regular image domain appear as shifts in polar space because they are the \textit{canonical coordinates} with respect to those transformations. Consequently, a standard CNN can be used processing the log-polar representation for a rotation- and scale-equivariant classification. Since the log-polar transformation is differentiable, the whole architecture including the polar origin predictor can be optimized end-to-end.

\cite{ETN} further generalize this method towards arbitrary continuous transformation groups and call their method Equivariant Transformer Networks (ETNs). Similar to the procedure in STNs, a separate network is trained to estimate, how the input is transformed. The estimated parameters are used to re-transform the input back to its \textit{canonical} form. However, in contrast to STNs, the input is transformed to the canonical coordinate system corresponding to the desired symmetry transformation. Thereby, the inverse transformation parameters are estimated in an equivariant manner using standard convolutions. In comparison to Polar Transformer Networks, ETNs only transform their input to canonical coordinates to estimate the transformation parameters. The classification itself is performed in the regular Euclidean domain, see Figure \ref{fig:ETN}.




While learned feature transformations facilitate learning invariant representations, they do not incorporate geometrical prior knowledge in a guaranteed way. In contrast, invariance is learned in a task-specific manner without any additional supervision. However, \cite{ETN} and \cite{PolarTN} show how to combine equivariant convolutions and learned feature transformations.

\subsection{Capsules}
\begin{figure*}[t]
	\centering
	\includegraphics[width=0.75\linewidth]{img/Capsule}
	\caption{Example of a capsule network architecture \cite{Capsules}.}
	\label{fig:Capsule}
\end{figure*}

\cite{HintonTransformingAE} introduce the notion of a group of neurons called \textbf{capsules} that perform internal calculations to output a small vector of highly informative outputs. This vector consists of the probability that the visual entity the capsule specializes on is present and its instantiation parameters, e.g. its pose, lightning conditions or deformation. While the former part should be invariant to changes of the instantiation parameters, the latter part should be equivariant. The proposed capsules are trained on pairs of transformed images with known transformation matrix using a transforming autoencoder.

Capsules can be arranged in a \textit{capsule network} similar to other DNNs consisting of multiple layers with multiple nodes (see Figure \ref{fig:Capsule}). In capsule networks, a \textit{routing algorithm} is used to determine, which lower-layer capsules send information to their higher-layer counterparts.

\cite{DynamicRoutingCapsules} propose a variation of capsules by encoding the instantiation parameters using the orientation of the capsule's output vector, while its length determines the probability that the entity exists in the input. This enables to use a dynamic routing approach between lower- and higher-level capsules. It is based on calculating a scalar product between their respective vectors to determine part-whole relationships. Convolutional layers are used within the capsules to benefit from weight sharing. The approach enables capsules to learn the pose information inherently through the routing mechanism. 

\cite{Capsules} introduce a novel type of capsules consisting of a logistic unit indicating the presence of an entity and a pose matrix storing the entity's pose information. The logistic unit allows to optimize an objective function specialized on detecting entities while the pose matrix allows to calculate simplified transformation matrices between capsules. Moreover, a new routing algorithm based on Expectation Maximization (EM) is proposed such that active capsules receive a cluster of similar pose votes from lower-level capsules. 

\cite{LenssenGroupCapsules} combine capsules with group convolutions to guarantee transformation-invariant output activations and equivariant pose vectors. Therefore, the capsules utilize group convolutions for their calculations, store the pose information as group elements and adapt the dynamic routing algorithm to guarantee equivariance under specified conditions. The group equivariant capsules are applied to SO(2) using spline-based convolutional kernels to avoid interpolation. While the proposed capsules now incorporate guaranteed equivariance properties, restricting the pose vector to group representations prevents the capsule from extracting arbitrary pose information, e.g. lighting.

\cite{VenkDeepEquivariantCapsules} propose space-of-variation capsules in order to improve the scalability of capsule networks and guaranteeing equivariance properties at the same time. Instead of learning pair-wise relationships between capsules, each capsule learns to encode the manifold of legal pose-variations, called space-of-variations, through a neural network that uses group convolutions to benefit from increased parameter sharing. Additionally, a provably equivariant routing procedure guarantees that learned part-whole relationships are preserved under transformations. In comparison to \cite{LenssenGroupCapsules}, the pose information is not embedded as group elements. While this reduces the transformation efficiency of the representation, compositionalities involving non-geometrical properties can be learned more efficiently. 

A broad field of research applies capsules to different problem settings or adapts their technicalities, e.g. the routing mechanism. However, listing these proposals is beyond the scope of this survey which focuses on
methods to obtain in- or equivariant representations.





\subsection{Invariant Integration}

Invariant Integration is an approach to construct a complete feature space with respect to a transformation first proposed for pattern recognition tasks by \cite{SM_Existence}. A complete feature space maps all equivalent patterns according to a transformation group $G$ from the signal space $S$ to the same point in the feature space $F$, whereas all non-equivalent patterns are mapped to distinct points. \cite{SM_Existence} mathematically defines conditions for the existence of complete feature spaces in pattern recognition tasks. For finite groups, a complete feature space can be computed by calculating all possible monomials
\begin{equation}
f(x) = x_0^{b_0}x_1^{b_1} \hdots x_{n-1}^{b_n-1} \quad \text{with} \quad b_1 + b_2 + \hdots + b_{n-1} \leq |G|
\end{equation}
with the monomial exponents $b_i$ and calculating the group average, which is defined as
\begin{equation}\label{eq:groupAverage}
A[f](x) := \int_{G} f(gx)dg \text{.}
\end{equation}

It is closely related to the group convolution (cf. Eq. \ref{eq:GConvCont}) and can be interpreted as a special case using a uniform kernel $k$ and the monomial output $f(x)$.

For finite groups, equation \ref{eq:groupAverage} is reduced to a sum over all group elements. The group average using all possible monomials is a valid basis representing the set of invariants. However, it is just an upper bound and computationally inefficient, especially for larger groups. Thus, it is important to reduce the number of monomials by carefully selecting them to improve the separability of the invariant feature space. The separability can also be improved by constructing weak $G$-commutative maps acting on them (\cite{SM_Existence}). 

\cite{SM_Algos} expands this theory to non-compact groups and continuous signals by introducing the notion that combining invariance to compact subgroups induces invariance to more general groups. For example, features invariant to the general linear group $GL(n,\mathbb{C})$ can be built by a quotient of homogeneous features invariant to the special unitary group $SU(n, \mathbb{C})$. Continuous signals within a Hilbert space are decomposed using a basis and an appropriate inner product to construct stable subsets. Subsequently, invariants to these subsets can be determined.

\cite{SM_Gray} applies the derived methods to 2D transformations and rotations on gray-scale images. The calculated invariant features are used to classify objects in an image. Twelve different monomials are processed via the group average to obtain invariance to global rotations and translations. In practice however, there might not only exist global transformations but also multiple local ones. In this case, the group-averaged monomials are not fully invariant, but vary only slowly. This also holds for small overlaps and articulated objects. Multiple objects within the same image are additive, i.e. their invariant feature values add, as long as they do not overlap. While the obtained features are invariant, they are not optimal for the specific application. Additional desirable properties, e.g. separability or robustness to distortions, need to be included by designing additional maps before applying invariant integration.

\cite{Condurache} combine multiple approaches in order to achieve invariance to multiple transformations. First, they compute Fourier descriptors of humans' contours to obtain features invariant to color changes, starting point, rotation and translation. Moreover, they use invariant integration with monomials in Fourier space which guarantees invariance to rotations in Fourier space. This corresponds to invariance with respect to anthropomorphic changes, i.e. different size and build among humans, in signal space. In total, this approach can be modeled as using a chain of invariant transformations to achieve an induced invariance in the input domain. Eventually, a Support Vector Machine (SVM) processes the invariant features for human event detection in video scenes.

\cite{Rath} propose to use invariant integration in combination with equivariant group convolutions to compute a representation invariant to rotations and translations for image classification. The proposed Invariant Integration Layer's exponents $b_i$ can be optimized using the backpropagation algorithm which makes it a drop-in replacement for pooling layers that are usually utilized for the transition between the equivariant and invariant feature space in equivariant neural networks. Thereby, the data-efficiency of those networks is improved. 

In summary, Invariant Integration is a method to construct a complete feature space with respect to a transformation. Comparably to the scattering transformation, most proposals use it to construct a feature space for classifiers such as a SVM, while novel approaches try to combine it with deep learning approaches in order to increase their data-efficiency.


\subsection{Other Methods}
\begin{figure}
	\centering
	\includegraphics[width=\linewidth]{img/TIPooling}
	\caption{Network architecture using TI Pooling \cite{Laptev}.}
	\label{fig:TIPooling}
\end{figure}
In this section, we introduce approaches which do not fit into one of the previous subcategories.

\cite{GensDeepSymmetry} suggest to form feature maps over arbitrary symmetry groups. A kernel-based interpolation scheme is used to find all transformed points in the lower layer contributing to a point of the symmetry feature map. The same kernel is used at every point of the symmetry feature map which leads to weight-tying among transformations. To reduce the computational complexity especially for higher-order transformations, a sub-sampling of the $N$ most important points in a neighborhood, determined by Gauss-Newton optimization, is performed.

TI-Pooling (\cite{Laptev}) is a method to compute transformation-invariant features. The input of a CNN is transformed with all transformation elements the network should be invariant to, one forward-pass per transformation element is computed using the same weights and max-pooling is applied among all responses, as shown in Figure \ref{fig:TIPooling}. While this procedure is a straight-forward way to obtain transformation-invariance, it is computationally expensive since - due to its \textit{brute-force} nature - $|G|$ forward-passes need to be computed per input. This is especially problematic for bigger transformation groups.

\cite{HenriquesWarpedConv} propose to warp the input of a CNN using the exponential map of arbitrary transformation groups such that transformations of the input result in translations of the warped image map. Hence, a standard convolution can be applied in the warped space to achieve equivariance - decreasing computational requirements compared to group convolutions. This approach is applied to three transformations: scale and aspect ratio, scale and rotation and perspective 3D rotations. However, in their implementation, the arbitrary groups need to be Abelian and only have two parameters in order to avoid more complex computations. 

\cite{RegularizedInvariance} model the worst-case transformations a DNN needs to be robust to as spatial adversarial attacks. This facilitates the use of a regularization term in combination with data augmentation to encourage learning constant values for all transformed versions. Consequently, invariance to transformation sets is guaranteed. This is a weaker requirement than invariance to transformation groups that allows to impose the desired properties on subsets not forming a group, e.g. rotations in $[-90^\circ, 90^\circ]$. In contrary to usual adversarial defense regularization, the proposed method does not include a trade-off between accuracy and adversarial robustness, but is able to improve the overall performance of the DNN.

\cite{Feige} uses a Variational Autoencoder (VAE) to learn an invariant class representation and equivariant information about the transformation needed to obtain the specific sample from the canonical group sample. During reconstruction, the decoder only gets the invariant latent variable from samples of the same class, but not of the sample itself. Contrarily, the equivariant information is sample-specific and is thus used to adapt the reconstructed class sample to the specific instance using a smooth transformation. Hence, the equivariant representation is able to encode arbitrary smooth transformations. The learned latent representation of the VAE can be used for other tasks, e.g. classification.


\section{Datasets and Benchmarks}
\begin{figure}
	\centering
	\begin{subfigure}{0.15\linewidth}
		\includegraphics[width=\linewidth]{img/mnist_rot/0}
	\end{subfigure}
	\begin{subfigure}{0.15\linewidth}
		\includegraphics[width=\linewidth]{img/mnist_rot/1}
	\end{subfigure}
	\begin{subfigure}{0.15\linewidth}
		\includegraphics[width=\linewidth]{img/mnist_rot/2}
	\end{subfigure}
	\begin{subfigure}{0.15\linewidth}
		\includegraphics[width=\linewidth]{img/mnist_rot/3}
	\end{subfigure}

	\vspace{2pt}
	\begin{subfigure}{0.15\linewidth}
	\includegraphics[width=\linewidth]{img/mnist_scale/0}
	\end{subfigure}
	\begin{subfigure}{0.15\linewidth}
		\includegraphics[width=\linewidth]{img/mnist_scale/1}
	\end{subfigure}
	\begin{subfigure}{0.15\linewidth}
		\includegraphics[width=\linewidth]{img/mnist_scale/2}
	\end{subfigure}
	\begin{subfigure}{0.15\linewidth}
		\includegraphics[width=\linewidth]{img/mnist_scale/3}
	\end{subfigure}

	\vspace{2pt}
	\begin{subfigure}{0.15\linewidth}
		\includegraphics[width=\linewidth]{img/affnist/0}
	\end{subfigure}
	\begin{subfigure}{0.15\linewidth}
		\includegraphics[width=\linewidth]{img/affnist/1}
	\end{subfigure}
	\begin{subfigure}{0.15\linewidth}
		\includegraphics[width=\linewidth]{img/affnist/2}
	\end{subfigure}
	\begin{subfigure}{0.15\linewidth}
		\includegraphics[width=\linewidth]{img/affnist/3}
	\end{subfigure}
	\caption{Examples from rotated MNIST (top), MNIST-Scale (middle) and affNIST (bottom).}
	\label{fig:MNIST}
\end{figure}
\begin{table*}[t]
	\centering
	\begin{tabular}{|l|c|}
		\hline
		Method & Test Error [$\%$] \\ 
		\hline
		Conventional CNN & 5.03 \\
		\cite{Scattering} & 4.4 \\
		\cite{LenssenGroupCapsules} & 2.6\\
		\cite{PolarTN} & 1.83 \\
		\cite{HarmonicNetworks} & 1.69 \\
	
		\cite{Laptev} & 1.2 \\
		\cite{MarcosRot} & 1.09 \\
		\cite{Rath} & 0.72 \\
		\cite{SFCNN} & 0.71\\
		\cite{E2STCNNs} & 0.68 \\
		\hline
	\end{tabular}
	\caption{Mean test error $[\%]$ on the rotated MNIST dataset.}
	\label{tab:RotMNIST}
\end{table*}
\begin{table*}[t]
	\centering
	\begin{tabular}{|l|c|}
		\hline
		Method & Test Error [$\%$] \\ 
		\hline
		Conventional CNN & 3.48 \\
		\cite{LocalScaleInvariance} & 3.13 \\
		\cite{MarcosScale} & 2.44 \\
		\cite{STN} & 1.97 \\
		\cite{GhoshScale} & 1.91 \\
		\cite{Sosnovik} & 1.68 \\
		\hline
	\end{tabular}
	\caption{Mean test error $[\%]$ on the MNIST-scale dataset.}
	\label{tab:ScaledMNIST}
\end{table*}\begin{table*}[t]
\centering
\begin{tabular}{|l|c|}
	\hline
	Method & Test Error [$\%$] \\ 
	\hline
	Conventional CNN (\cite{DynamicRoutingCapsules}) & 34.0 \\
	\cite{DynamicRoutingCapsules} & 21.0 \\
	Conventional CNN (\cite{ETN}) & 12.7 \\
	\cite{LenssenGroupCapsules} & 10.9 \\
	\cite{Capsules} & 6.9 \\
	\cite{VenkDeepEquivariantCapsules} & 2.99 \\
	\cite{ETN} & 1.7 \\
	\hline
\end{tabular}
\caption{Test error $[\%]$ on the affNIST dataset.}
\label{tab:AffNist}
\end{table*}
Most of the algorithms presented in this paper evaluate their performance on artificially transformed versions of MNIST (\cite{MNIST}), a well-known dataset for hand-written digit recognition, to prove the incorporated in- or equivariance properties. We summarize the reported performances of different in- or equivariant algorithms on transformed MNIST datasets in this section to allow an easy comparison.

The rotated MNIST dataset (\cite{RotMNIST}) contains digits that are randomly rotated by angles $\phi \in (0, 360)^\circ$. It consists of a training set with $10,000$, a validation set with $2,000$ and a test set with $50,000$ samples. Examples of the dataset are shown in Figure \ref{fig:MNIST} (top).
Due to the relatively small amount of training samples, the variability of all possible rotations is not captured in the training dataset. Hence, DNNs incorporating in- or equivariance to rotations achieve comparably better results. The reported results of various proposals are shown in Table \ref{tab:RotMNIST}.

The MNIST-Scale dataset (\cite{MNIST-Scale}) consists of digits which are randomly scaled by uniformly sampled factors $s \in (0.3, 1)$. The size of the training, validation and test set is the same as for rotated MNIST. Examples are depicted in Figure \ref{fig:MNIST} (middle) and the reported results are shown in Table \ref{tab:ScaledMNIST}. 

Finally, the affNIST dataset (\cite{affNIST}) is used to test the robustness of algorithms, mainly capsule networks, to unseen affine transformations. Therefore, the networks are trained on randomly translated digits from the MNIST dataset and tested on digits which are randomly transformed with small affine transformations. Examples are shown in Figure \ref{fig:MNIST} (bottom) while the reported results are listed in Table \ref{tab:AffNist}.

Additionally, most authors also apply their approach to a real world dataset where the incorporated prior knowledge often allows to outperform conventional algorithms. We refrain from listing the latter since their uniqueness and variety prevents us from comparing multiple algorithms. 


\section{Towards Geometrical Prior Knowledge for Autonomous Driving}
Systems for automated driving need to solve a variety of problems that could benefit from deep learning approaches. This includes processing and fusing signals from different sensors, perceiving the environment, i.e. reliably detecting and locating other objects in a driving scene, understanding and assessing the environment, localizing the ego vehicle and finally, planning and executing driving maneuvers. Especially for the perception part, deep learning approaches have been established to provide state-of-the-art results on several benchmarks (e.g. \cite{KITTI3D, nuScenes}). Moreover, DNNs are used within cutting edge prototype vehicles already driving on real streets autonomously. 
We refer the reader to \cite{SurveyAutonomous} for a concise overview over datasets, methods and challenges concerning deep learning and autonomous driving.

Nevertheless, there remain some open problems which need to be solved before DNNs can safely be used within self-driving cars. Foremost, collecting and labelling the data needed for supervised learning is both time-consuming and cost-expensive. Particularly, labelling three-dimensional objects in driving scenes is a non-trivial task. Furthermore, DNNs are hard to evaluate due to their black box behavior which makes it hard to predict outcomes given data-points the network has not been trained on. This is delicate since self-driving cars must operate safely even in unexpected circumstances, e.g. a car behaving in a way that was not included in the training data.

We argue that incorporating geometrical prior knowledge to deep learning systems for autonomous driving helps to mitigate both depicted problems. On the one hand, the data-efficiency of DNNs is increased which means that less data needs to be collected, stored and labelled to achieve the same performance level. On the other hand, in- or equivariance guarantees aid the interpretability of DNNs. 

\textbf{Equivariant 3D Object Detection}:
As an example, we show how 3D object detection can benefit from in- or equivariance to various transformations. In this section, we list several transformation groups that affect the output of an 3D object detector in a predictable way. 
An ideal 3D object detector should be stable to shape deformation and varying object size (i.e. scale) and equivariant to translation and rotation of the object to be detected since the pose information is needed to predict the exact location. Moreover, neural nets processing point clouds must be invariant to permutations and varying point cloud densities. Contrarily, color changes in images, including variable illumination, should be considered. Ideally, the detection should also be equivariant to more complicated transformations, e.g. (partial) occlusions. 

Object detection can be divided in several sub-tasks: Recognition, object type classification and estimation of the relative position, orientation and size. While recognition and classification build invariants, it is important to preserve information about position, orientation and size to be able to estimate them. Therefore, equivariant outputs are more appropriate in this case. 


\section{Conclusion}
In this section, we first summarize the various methods presented and their different advantages and disadvantages. Afterwards, we draw a conclusion and give a short outlook on possible future work. 

\textbf{Data augmentation} is easy to implement and highly adaptable towards a variety of transformations. Additionally, it can easily be used in combination with other approaches. On the downside, it does not provide any mathematical guarantees, augmenting the data with all possible transformations is computationally inefficient, the solution space is not restricted effectively and the equivariance is only learned for the network as a whole, not at each layer.

\textbf{Restricted filters} provide a provable way to obtain in- or equivariant representations in neural networks and reduce the space of learnable parameters. However, the restrictions might be too strict and prevent the network from learning informative representations, especially when the filters are fixed rather than learnable. 

Enforcing equivariance properties using \textbf{group convolutions} provides a mathematically guaranteed way to incorporate geometrical prior knowledge to DNNs and leads to state-of-the-art results in a variety of tasks benefiting from in- or equivariance properties. Up to now, group convolutions are restricted to fairly simple and often finite transformation groups and suffer from a computational overhead. Moreover, important symmetries occurring in tasks are not expressible using the concept of groups, e.g. viewpoint changes in images. Incorporating in- or equivariance to multiple transformation groups at once is also an area which needs further investigation.

\textbf{Learned Feature Transformations} provide a way to learn invariance from data rather than defining and incorporating it manually. Feature transformation layers are easy to include to existing neural network architectures. The transformations are learned from data and not restricted to groups. Nonetheless, learned feature transformations usually lack mathematical guarantees.

\textbf{Capsules} disentangle learned object representations into invariant object information and equivariant pose parameters. This includes sophisticated transformations such as viewpoint-changes or lighting which can not be modeled as groups. However, capsules do not guarantee the desired properties. Adaptations leveraging group symmetry exist but either suffer from a reduced transformation efficiency or restrictions regarding the equivariant pose information. Capsule networks are non-straightforward to train because they need a carefully-designed routing algorithm to assign activations to other layers as well as for backpropagation. 

\textbf{Invariant Integration} is a method to guarantee invariance while increasing the separability of distinct patterns. However, it suffers from computational complexity, is restricted to group transformations, the monomials are hard to choose in a principled manner and disposes the information about the symmetry group. Consequently, it is best used in combination with equivariant G-CNNs to improve the separability of learned invariant representations while exploiting the properties of equivariant learnable filters.

In this survey, we showed how utilizing prior knowledge can enhance state-of-the-art deep learning systems and used 3D object detection as an example. Foremost, defining problem-specific symmetry groups and introducing in- or equivariance to them can greatly improve the performance of neural networks, especially when the amount of training data is scarce. This has been proven in a broad variety of interesting applications from medical imaging to 3D shape recognition. Moreover, this approach improves the interpretability of neural network layers which is important for validation.

We expect future work to investigate further generalizations to more complex input domains and symmetry groups as well as to more complicated, non-invertible transformations, e.g. occlusions. Moreover, DNNs incorporating multiple symmetries at once could be of interest. 

{\small
