\section{Introduction}

\IEEEPARstart{V}{isual} inspection using imaging technologies is an essential part of
material quality control.
For example, solar modules are subject to regular inspection by \acl{el}
imaging~\cite{jahn2018review} for material defect detection.
\changed{To inspect every single cell of a solar module, the entire module needs to be located in the \ac{el} image first, to allow for a subsequent segmentation into single cells~(see~\cref{fig:motivation})\chmark{ch001}}. In addition, removal of perspective and lens distortion simplifies further
analysis~\cite{deitsch2018segmentation}.
Modern CMOS sensors allow \ac{el} imaging at high frame rates, which is
beneficial for drone-based in-field inspection.
However, due to practical requirements on the signal-to-noise ratio and the
sensor integration time, their spatial resolution for \ac{el} imaging is limited,
challenging the detection of small material defects.  
In this respect, the use of \ac{sr} techniques for software-based
resolution enhancement is promising.
  
\input{8eec2f53d49941d7aed78e331312b4ec.tex}

\input{b0779e5953294e699eb62d5078e79ab2.tex}

In this work, we propose to capture a sequence of \ac{lr} images using fast CMOS sensors and to subsequently apply \ac{mfsr} in order cell images at higher resolution. Prior to the reconstruction of a \ac{hr} image, the sequence of \ac{lr} images needs to be registered. Traditionally, one of the frames is used as a reference frame that all others are registered to. However, this approach does not compensate for perspective or lens distortion. We propose to integrate the rectification and undistortion steps into the motion estimation procedure of a \ac{mfsr} approach by performing registration w.\,r.\,t\onedot a virtual reference of the module.

The contributions of our work are as follows:
\begin{enumerate}
    \item We fuse rectification and undistortion with \ac{sr} which leads to a uniform spacing of pixels in the module plane, regardless of perspective or lens distortion.
    \item We conduct an extensive user study showing that \ac{mfsr} simplifies manual inspection of solar cells by human experts and that our proposed registration outperforms the state of the art.
    \item We apply automated crack detection and segmentation on the resulting images and show that \ac{mfsr} enhances automated inspection of solar cells.
\end{enumerate}

\section{Related Work}
\label{sec:RelatedWork}

Visual inspection of solar cells is an active research area. In previous studies, it has been shown that, for example, micro cracks result in power loss of the module, especially after simulated aging~\cite{kontges2010quantifying}. Hence, reliable detection of these and other defect types is of particular interest. Only recently, \ac{dl}-based methods are applied to automatically detect defects on solar cells using \ac{el} imaging~\cite{deitsch2019automatic}. Many of these works focus on the detection~\cite{akram2019cnn} and segmentation~\cite{rahman2020defects} of cracks. However, a reliable detection of small cracks is challenging and requires a sufficient image resolution. For this reason, we focus on the automated detection and segmentation of cracks and show that super-resolution is beneficial for this task.

~\\\noindent
\textbf{Observation model for super-resolution.}
Given a sequence of \ac{lr} images captured from a scene of interest, \ac{sr} can be modelled by the observation model:  
\begin{equation}\label{eq:observation-model}
	\bm{g}_i = \bm{W}_i \bm{f} + \bm{n}_i\quad \text{for}\quad 1 \leq i \leq N\,,
\end{equation}
where $\bm{f}$ denotes the desired \ac{hr} image, $\bm{g}_i$ is the $i$-th frame of the \ac{lr} sequence, and $\bm{n}_i$ is additive noise. The system matrix $\bm{W}_i$ describes the imaging process including motion between the \ac{hr} and \ac{lr} domain, the \acl{psf} of the optical system, and sampling on the sensor array. \ac{sr} algorithms aim at inverting this model w.\,r.\,t\onedot $\bm{f}$.

~\\\noindent
\textbf{Single-frame super-resolution.}
\changed{Many mainstream methods use single images ($N = 1$) to compute a super-resolved image~\chmark{ch002}}. Here, especially deep neural networks~\cite{dong2014learning, Kim2016, Lim2017,zhang2020deep} and generative adversarial learning~\cite{wang2018esrgan,rakotonirina2020esrganplus} advanced the state of the art in \ac{sfsr}. They greatly benefit from large natural image databases, enabling supervised learning of mappings from \ac{lr} to the \ac{hr} domain. However, without fine-tuning to the target application, their performance can drastically drop. This is particularly severe in imaging applications such as ours, where comprehensive training datasets are still unavailable and image properties like noise or blur models can considerably deviate from those of natural images~\cite{kohler2017benchmarking}. \changed{To overcome this issue, zero-short \ac{sfsr} methods have been developed~\cite{shocher2018zero,soh2020meta}. These methods exploit recurring information within a single image to train a convolutional neural network at test time. Only recently, it has been shown that combining classification and \ac{sfsr} into a single network in terms of multi-task learning helps to improve the performance of \ac{sfsr} and classification at the same time~\cite{schirrmacher2020sr}.} Still, existing \ac{sfsr} methods mostly target to satisfy quality perceptions of humans and might even hallucinate \ac{hr} additional details. This is caused by the fact that they only use the information available in a single image and incorporate prior knowledge to reconstruct \ac{hr} the image. In contrast, visual inspection requires authentic data for quantitative and objective analyses, like solar cell defect detection. This prohibits the simple use of existing pre-trained models.

~\\\noindent
\textbf{Image-based multi-frame super-resolution.}
\ac{mfsr} aims at the inversion of \cref{eq:observation-model} using multiple frames ($N > 1$) assuming sub-pixel inter-frame motion, to fuse them into an \ac{hr} image. This can be done by non-uniform interpolation~\cite{alam2000infrared,batz2016multi}, iterative reconstruction to invert \cref{eq:observation-model}~\changed{\cite{elad1997restoration, Liu2014, kohler2016robust,liu2018robust,honda2019multi,borsoi2020robust}}, or deep learning~\changed{\cite{kappeler2016video,liao2015video,wang2019edvr,isobe2020video,li2020mucan,xu2020deformable,pan2020deep}} to directly predict the \ac{hr} image.

During the imaging process, crucial details are lost in \ac{lr} frames, which is mainly caused by the sampling process on the sensor. In principle, \ac{mfsr} is able to reconstruct crucial \ac{hr} details on solar cells from aliasing artifacts in the \ac{lr} images~\cite{kohler2018phd}.
However, such approaches are entirely image-based.
Specifically, inter-frame motion is directly obtained on the image plane by
classical image-to-image registration or learned motion estimation modules. 
We show that the robustness of these image-based methods for \acl{el} imaging of
solar cells is strongly limited by the accuracy of motion estimation. 
In contrast, we propose to incorporate 3D geometry into the underlying model to
increase robustness of \ac{mfsr}.  

~\\\noindent
\textbf{3D geometry-based multi-frame super-resolution.}
In a closely related work to ours, Park and Lee~\cite{park2017joint} have
coupled the estimation of the 3D camera pose, depth, and \ac{hr} data from
\ac{lr} frames via joint optimization based on a projective model. 
Similar to this approach, we consider 3D camera motion in a static scene. 
Additionally, we exploit the fact that imaged solar cells are planar and
distortion-free objects in space. 
Therefore, a purely projective model is not sufficient and we propose to include a
pinhole camera model along with radial distortion, such that the overall model
complies to the imaging process. 
This allows a joint rectification and \ac{sr} in \acl{el} imaging of solar cells.
We also show how to seamlessly integrate our general-purpose model in
regularized reconstruction based \ac{mfsr}.

\section{Methodology}\label{sec:method}

An overview of our method is shown in~\cref{fig:overview}. It consists of two steps: First, \ac{lr} images are
registered~(\cref{subsec:registration}). As a result, we know the motion $m_i$ from the reference to every low resolution frame $\bm{g}_i$. As opposed to previous works on \ac{sr}, this reference is not an arbitrary low resolution frame. We perform registration w.\,r.\,t\onedot to an image of the module that is free of lens and perspective distortion and find that this increases registration stability. Hence, $m_i$ describes the deformation of $\bm{f}$ by perspective projection and lens distortion (see~\cref{fig:motionmod}). To simplify notation, we define $m$ such that it includes the downsampling as well. Hence, $m$ is not a pure motion model. In a second step, this virtual module image is reconstructed from the \ac{lr} images using the estimated motion~(\cref{subsec:reconstruction}).

\subsection{Registration}\label{subsec:registration}

\input{f3d56bb2ff5f4ba8aef7754428f88d8f.tex}

\input{93abd320714c4d04aaba5839e3572acc.tex}

In the general case, $m$ may be any motion and downsampling model that relates two corresponding points $\lc_{i,k}$ and $\bm{y}_k$ in either view. Hence, it holds that
\begin{equation}\label{eq:motionmod}
\lc_{i,k} = m_i(\bm{y}_k)\,,
\end{equation}
where $\lc_{i,k}$ is the \ac{lr} correspondence to $\bm{y}_k$ and $k$ denotes the \ac{hr} pixel index.We assume that the virtual reference of solar modules lies in the $x$-$y$-plane of the reference coordinate system, which we refer to as the module coordinate system. A 2D\xspace point in the module coordinate system in homogeneous coordinates $\tilde{\mc} = \left[\tilde{y}_1,\,\tilde{y}_2,\,s\right]$ with scaling factor $s$ and its image $\lch_i = \left[\tilde{u}_i,\,\tilde{v}_i,\,s_i\right]$ in multiple views $1 \leq i \leq N$ are related by the non-linear model~\cite{hartley2003multiple,fryer1986lens}:
\begin{equation}\label{eq:pinhole-homo-dist}
    m_i \coloneqq \lch_i \simeq \bm{K} d(\bm{H}_i\tilde{\mc},\, \bm{p})\,.
\end{equation}
Here, $\bm{H} \in \mathbb{R}^{3\times3}$ encodes the camera pose, $\bm{K}$ are the intrinsic camera parameters, and $d(\cdot,\, \bm{p})$ is a distortion model with parameters $\bm{p}$. We assume that $\bm{H}$ changes between frames, whereas $\bm{K}$ and $\bm{p}$ are constant. For simplicity, we resort to radial distortion with a single coefficient. We further assume that the center of distortion is located at the origin. 
Let $\tilde{\mc}' = \bm{H}\tilde{\mc} = \left[\tilde{y}_1',\,\tilde{y}_2',\,s'\right]$, $y_1' = \frac{\tilde{y}_1'}{s'}$ and $y_2' = \frac{\tilde{y}_2'}{s'}$. Then, $d$ follows as
\begin{equation}\label{eq:distortion}
    d(\tilde{\mc}',\,\bm{p}) = \left(
        \frac{r_d}{r_u}y_1',\,
        \frac{r_d}{r_u}y_2'
    \right)^\top,
\end{equation}
where $r_u$ is the distance to the center of distortion, $\kappa$ is the distortion coefficient and $r_d$ is given by~\cite{brown1971close}
\begin{equation}\label{eq:brown}
    r_d = r_u + \kappa r_u^3\,.
\end{equation}

For registration, we use the approach by \etal{Hoffmann} to find corresponding points between \ac{lr} frames~\cite{hoffmann2019fast} and Zhang's method to initially estimate the intrinsic and extrinsic camera parameters~\cite{zhang2000flexible}. Then, registered images are obtained by warping each \ac{lr} frame into the common module coordinate system. To achieve sub-pixel  registrations in the presence of noisy point correspondences, we refine the initial parameters by non-linear least-squares minimization:
\begin{equation}
    \hat{\bm{H}}_i = \argmin_{\bm{H}_i} \lVert \bm{g}_i - \tau(\bm{f},m_i) \rVert_2\,.
\end{equation}
Here $\tau(\bm{f},m_i)$ denotes inverse warping of $\bm{f}$ according to the motion model $m_i$. To improve stability, we perform registration in a multi-scale scheme~\cite{modersitzki2004numerical}.

\subsection{Joint super-resolution and rectification}\label{subsec:reconstruction}

By the observation model in \cref{eq:observation-model}, the \ac{hr} image is transformed to obtain a \ac{lr} view and we can relate any point in the module coordinate frame $\bm{y}_k$ with its image $\lc_{i,k}$ in the $i$-th \ac{lr} frame by~\cref{eq:motionmod,eq:pinhole-homo-dist} (cf\onedot~\cref{fig:motionmod}). However, for smooth results, it is preferable to implement~\cref{eq:observation-model} by inverse warping. Therefore, we need to relate $\lc_{i,l}$ with its image in $\bm{y}$, where $l$ is the \ac{lr} pixel index. Hence, the inverse of $m$ is required. Using the motion model in \cref{eq:pinhole-homo-dist}, we need to invert $\bm{K}$, $\bm{H}_i$ and $\bm{p}$. Since $\bm{K}$ and $\bm{H}_i$ are square matrices and have full rank, they are invertible right away. According to~\cref{eq:brown}, distorted and undistorted radii are related by a polynomial of degree 3, which has an analytic inverse by the formula of Cardano. The solution depends on the discriminant $D$. For $D > 0$, the undistorted radius $r_u$ is given by
\begin{gather*}
    r_u = \frac{1}{6\kappa} \left( \sqrt[3]{-4q + 4\sqrt{q^2 + 4p^3}} + \sqrt[3]{-4q - 4\sqrt{q^2 + 4p^3}} \right)\,,
\end{gather*}
for $D < 0$ it is
\begin{gather*}
    r_u = 6\kappa \sqrt{-p} \cos\left( \frac{1}{3}\arccos\left( \frac{-q}{2\sqrt{-p^3}} \right) + \frac{4\pi}{3} \right)
\end{gather*}
and for $D=0$ it holds that $r_u = r_d$. Note that we substituted $p = 3\kappa$ and $q = -27 \kappa^2 r_d$ and that the discriminant is $D = q^2 + 4p^3$. A detailed derivation of this result is given in~\cref{sec:inverse-distortion}.

Inter-frame motion can be expressed in various forms and we use motion vector fields as a general model. The motion vector for the $l$'th pixel of the $i$'th \ac{lr} frame is computed as $\bm{v}_{i,l} = m^{-1}(\lc_{i,l})-\lc_{i,l}$. Using the derived motion fields, a known camera \acl{psf}, and the desired \ac{sr} magnification factor, we parameterize the system matrices $\bm{W}_i$, $i = 1, \ldots, N$ of the observation model in \cref{eq:observation-model} similar to~\cite{kohler2016robust}. Then, $\bm{W}_i$ relates the \ac{hr}, undistorded and rectified image $\bm{f}$ of the virtual module to the \ac{lr} frame $\bm{g}_i$. Given $N$ frames, we aim at reconstructing $\bm{f}$ as \acl{map} estimate: 
\begin{equation}
    \hat{\hrimg} = \argmin_{\bm{f}} \sum_{i = 1}^N|| \bm{g}_i - \bm{W}_i \bm{f} ||_2^2 + \lambda R(\bm{f})\quad,
\end{equation}
where $R(\bm{f})$ denotes a regularization term with regularization weight
$\lambda \geq 0$ to induce prior knowledge on the desired solution. 
In this paper, we use the weighted bilateral total
variation~\cite{kohler2016robust} for sparse and detail-preserving
regularization. 

\section{Experiments and Results}\label{sec:experiments}

This section reports experimental results on five solar module datasets. An example image for every sequence in shown in~\cref{fig:dataset-examples}. The first sequence (\textsc{D1}\xspace) exhibits strong noise artifacts while the second sequence (\textsc{D2}\xspace) is characterized by heavy lens distortion and streak artifacts. The third sequence (\textsc{D3}\xspace) shows substantial motion blur. The last two sequences are not corrupted by any particular kind of artifacts. The sequences have been recorded with four different CMOS cameras at a spatial resolution of \SI{640 x 512}{px}. Here, \textsc{D1}\xspace, \textsc{D2}\xspace and \textsc{D3}\xspace use different cameras, while \textsc{D4}\xspace and \textsc{D5}\xspace share the same camera.

Furthermore, we have \ac{hr} images available for the solar module \textsc{D2-5}\xspace, which will be used as reference for the crack detection in~\cref{subsec:segmentation}. For \textsc{D1}\xspace, there is no such data available, since these images have been captured by a drone on-site.

\subsection{Reference methods and parametrization}\label{subsec:reference-methods}

\input{1a44595e35f64d6fa800fe79d9addb68.tex}

We evaluated multiple \ac{sfsr} and \ac{mfsr} methods. For \ac{sfsr}, we use
bicubic upsampling as a baseline and also compare against
\mbox{\textsc{SRCNN}}\xspace~\cite{dong2014learning} and the recent \mbox{\textsc{ESRGAN}}\xspace~\cite{wang2018esrgan}. For
\ac{mfsr}, we exploit $N = 20$ \ac{lr} frames and compare our proposed approach
against EDVR\xspace~\cite{wang2019edvr}. Due to the lack of comprehensive solar cell datasets to train \acl{dl} methods for our application, we use the pretrained models provided by the authors for \mbox{\textsc{SRCNN}}\xspace, \mbox{\textsc{ESRGAN}}\xspace, and EDVR\xspace. \changed{In contrast, our approach is completely unsupervised and does not require any training\chmark{ch012}.}

\input{553c41c5f40f4a00b0d4ef69c1d47380.tex}

\input{c8ffb5ec6d9147fab18f96b8c1b3c1bf.tex}

\input{3ef49756be2b48959e8b3e0eb358f7ab.tex}

A qualitative comparison of the results is given in
\cref{fig:resultimgs}. The \ac{dl}-based \ac{sfsr} methods (for example \mbox{\textsc{ESRGAN}}\xspace) tend to only sharpen the result and may generate high frequency artifacts as well. Overall, they fail to reconstruct additional details. In contrast, \ac{dl}-based \ac{mfsr} (EDVR\xspace) only performs well on \textsc{D3}\xspace and \textsc{D4}\xspace. Further investigation shows that EDVR\xspace breaks down completely with stronger noise or artifacts, which is in agreement with the finding of another study in a different context~\cite{schirrmacher2019adaptive}. \changed{A detailed comparison of EDVR\xspace and our method is given in~\cref{fig:resultimgs-detail}\chmark{ch008}.} In contrast, the proposed method (\mbox{\textsc{Ours-P}}\xspace) performs well on most datasets, giving smooth results and showing many details without enhancing noise or artifacts. On~\textsc{D1}\xspace, \mbox{\textsc{Ours-P}}\xspace does not perform well. We found that this is caused by strong motion blur in the \ac{lr} images that lead to a larger error in the initial parameter estimation using~\cite{hoffmann2019fast} and~\cite{zhang2000flexible}. In addition, shift-variant motion blur is currently not covered by the reconstruction approach.

\changed{\subsection{Ablation study}\label{subsec:ablation-study}

\chmark{ch010}For an ablation study concerning the impact
of motion estimation and rectification, we replace our registration scheme
by standard in-plane homography estimation between planar solar cells using the image alignment toolbox~\cite{iat2013}. Then, we determine motion vectors for regularized reconstructions from the homographies. In the following, we denote this method as \mbox{\textsc{Ours-H}}\xspace. The impact of our proposed motion estimation can be qualitatively seen from~\cref{fig:resultimgs}. It turns out that the homography-based approach (\mbox{\textsc{Ours-H}}\xspace) gives a relatively blurry result, compared to \mbox{\textsc{Ours-P}}\xspace. This is mostly caused by unstable registration.}


\subsection{User study}\label{subsec:user-study}

\input{4c97cc2de1934089bdb167c2a7ff94f7.tex}

As opposed to many traditional \ac{sr} benchmarks, where \ac{lr} images are obtained by downsampling, we are working with low resolution cameras, such that there are no exact ground truth \ac{hr} images available. On the upside, this corresponds to a realistic application case. On the downside, we cannot report traditional metrics like PSNR or SSIM that require a pixel-perfect \ac{hr} reference image. To this end, we conduct a user study. Here, we assess, if \ac{sr} is suitable for manual inspection of defects by human experts. For the user study, we include all reference methods and our registration in combination with the regularized reconstruction. Further, we include $20$ randomly selected cells from every sequence and compare all $6$ methods using $3\times$ as well as $4\times$ magnification.

In summary, the user study comprises 3000 pairwise comparisons. It was performed by seven experts in \acl{el} imaging. They were presented a pair through a web interface and were asked to decide, which of the samples is better suited for detecting defects. \changed{During the study, the arrangement of methods (left or right) and the order of samples was randomized\chmark{ch004}.} Similar to the benchmark in~\cite{kohler2017benchmarking}, we use Bradley-Terry (BT) scores~\cite{david1963method} derived from pair-wise comparisons to compute a ranking of the methods. Using the scores $p_k$ and $p_l$, the probability that method $k$ is preferred over method $l$ is computed as $P(k>l) = \exp(p_k)/(\exp(p_k) + \exp(p_l))$.

The result is presented in~\cref{fig:bt-scores-all}. On average, the \ac{dl} approaches do not perform well. In contrast, \mbox{\textsc{Ours-H}}\xspace shows moderate performance, but is limited by instable registration. Our proposed method clearly outperforms the other methods. Using the BT scores, we find that \mbox{\textsc{Ours-P}}\xspace is preferred over \mbox{\textsc{Ours-H}}\xspace with a probability of $P=0.57$ ($3\times$) or $P=0.61$ ($4\times$). Regarding the magnification factor, it turns out that the difference in perceived performance is increased for $4\times$ magnification. This is explained by the negative impact of instable registration for the \mbox{\textsc{Ours-H}}\xspace approach, which is amplified with the increased magnification (see~\cref{fig:compare-magn}).

\changed{Finally, we evaluate the inter-observer variance to determine the amount of agreement between users. We compute the ranking of methods per user and Kendall's coefficient of concordance $W \in [-1, 1]$ to obtain the inter-observer variance of method rankings, where $W=1$ denotes total agreement between users, $W=0$ denotes no agreement and $W=-1$ states that user rankings are the exact inverse of each other. We find that the value of $W$ depends on the magnification. For $3\times$ magnification, the coefficient computes as $W_{3\times} = 0.33$, whereas for $4\times$ magnification the agreement is stronger and amounts to $W_{4\times}=0.44$. The same effect has been reported by \etal{K\"ohler} in a similar study~\cite{kohler2017benchmarking}. We conclude that, with higher magnification factors, different methods are easier to distinguish for human observers, which leads to a higher overall consensus\chmark{ch005}.}

\subsection{Automated crack segmentation and detection}\label{subsec:segmentation}

\input{162bb1c35c534aadaf428ec8c9f4f373.tex}

We also investigate if the proposed method improves automated inspection. For this analysis, we include all cell images from \textsc{D2-5}\xspace, since \ac{hr} measurements are available for this data. The results are summarized by~\cref{tab:results} and \cref{fig:resultimgs-seg}.

For crack segmentation, we apply the weakly supervised approach proposed by \etal{Mayr}~\cite{mayr2019weakly} to the results with $3\times$ and $4\times$ magnification as well as to the \ac{hr} images. We use the segmentation result obtained on the \ac{hr} images as ground truth. Obviously, this automatically generated ground truth is not perfect. However, the bias is constant for every method. Hence, the relative performance of the methods remains unaffected.

\etal{Mayr} use a fully convolutional network trained for classification of cracks. The segmentation is then derived from the class activation maps. As a result, the resolution of segmentation maps is reduced by a factor of $16$, compared to the original image. This is beneficial to our task, since a registration of \ac{hr} images that are used as pseudo ground-truth with the super-resolved cell images is challenging and the registration mismatch is hidden by the subsampling.

We report the pixel-wise metrics with respect to the automated segmentation obtained from the \ac{hr} images. It turns out that EDVR\xspace results in a recall that is comparable to our method. However, it generates many false positives, as indicated by the low precision and average precision (AP) in~\cref{tab:results}. On the other hand, \mbox{\textsc{Ours-H}}\xspace results in a high precision but a small recall. Overall, our method performs $\sim2\times$ better than EDVR\xspace and $\sim3\times$ better than \mbox{\textsc{Bicubic}}\xspace regarding the average precision and $F_1$ score.

\input{8be8742d198f44e0a8158c7a2d708dd9.tex}

In addition to the pixel-wise metrics, we compare the detection of crack instances too. For many applications, it is more interesting to know the number of cracks, rather than having a perfect segmentation. Furthermore, a segmentation that is only one pixel off results in a significant drop in the quantitative segmentation performance, since cracks are usually thin.

We perform connected component analysis on the coarse segmentation masks, assuming that a single connected component corresponds to a crack instance. Then, we compute the fraction of crack instances for the \ac{hr} segmentation masks, where at least one pixel of the corresponding area on the \ac{sr} image is classified as a crack. This is the recall of the crack instance detection. Inversely, the fraction of detected crack instances on the \ac{sr} image that overlaps with a crack instance detected on the \ac{hr} image by at least a single pixel, is the precision of crack instance detection. We find that our method outperforms the other methods with respect to recall and $F_1$ score. \mbox{\textsc{Ours-H}}\xspace performs a little better regarding the precision, but fails in the recall of crack instances. Furthermore, EDVR\xspace has a recall that is close to ours, but has a much lower precision. Regarding the $F_1$ score, which combines recall and precision into a single score, our method performs $\sim35\%$ better than the best reference method.

\changed{Finally, we analyze the correlation between the results obtained from the user study to the quantitative results. Spearman's rank correlation coefficient $\rho \in [-1, 1]$ is widely used to quantify the amount of correlation between rankings of ordinal variables. As opposed to, for example, Pearson's correlation, it is not limited to linear associations. The results are summarized by~\cref{tab:results}. We find that, apart from the precision score, the association between the metrics and the user study rankings is weaker for the higher magnification. This complies to a previous study, that reported a weaker correlation for reference metrics at higher magnifications, as opposed to reference-less metrics~\cite{kohler2017benchmarking}. Overall, the experiment reveals a fair correlation of the user study results to the quantitative results for $3\times$ magnification, while they do not correlate well for $4\times$ magnification. This shows that, for higher magnifications, qualitative analysis of the results cannot substitute the qualitative analysis and vice versa\chmark{ch006}.}

\changed{\subsection{Computational performance}\label{subsec:time}

\chmark{ch009}From an application perspective, not only quality of results, but also runtime matters. To this end, we conduct an evaluation of the computational performance. The experiments have been conducted using a workstation equipped with an Intel i7-7820X CPU, \SI{64}{\giga\byte} or RAM and a Nvidia Titan Xp GPU using a magnification of $4\times$ with an initial resolution of $512^2$~pixels. Note that the results were averaged over \textsc{D1-5}\xspace. In~\cref{fig:computation-time}, we summarize the computational performance and compare it to the quantitative segmentation metrics. In summary, we find that, in our experiments, \ac{sfsr} methods are much faster than \ac{mfsr} and might be suitable for online image processing. However, we have shown that they are not well suited for the task at hand, since \ac{hr} details are not authentic. In contrast, \ac{mfsr} methods require about \SIrange{30}{50}{\second} of processing time, with the proposed method being faster than EDVR\xspace. As a result, \ac{mfsr} needs to run offline. This is not prohibitive for many applications: For example, given a scenario, where data is collected on-site and later analyzed to identify and possibly replace faulty modules, this is an acceptable amount of computation time.}

\section{Conclusion}\label{sec:conclusion}


This paper presents a new method to increase the spatial image resolution of
solar modules.
It is based on parametric registration and can be combined with various
reconstruction approaches.
As opposed to non-parametric registration, such as optical flow, this ensures
that the registration result is plausible given the imaging process.
In contrast to parametric registration based on plain homography estimation, it accounts for lens distortion, too.

We present a user study showing that our approach outperforms state of the art single-frame and \acl{mfsr} methods. Furthermore, we combine the proposed method with automatic crack detection and segmentation and show that~\ac{sr} is not only beneficial for manual inspection by human experts, but also for automated inspection. In particular, we show that the proposed method is better suited for subsequent automated inspection than state of the art methods. \changed{Furthermore, we investigate the methods in terms of computational performance and find that the proposed method is computationally fast, compared to other state of the art \ac{mfsr} methods.}

In future works, we aim to integrate the proposed motion model with \ac{sr} to form a single optimization problem, as shown for the purely projective case in~\cite{park2017joint}. This would also allow for joint estimation of motion. Furthermore, an interesting opportunity is to combine the conventional and \ac{dl}-based methods in terms of known operator learning~\cite{maier2019learning} to join the robustness of conventional methods with the power of learning individual steps of the algorithm from the data\chmark{ch007}.

\vspace{.2cm}
\noindent\textbf{Acknowledgements}
\noindent We gratefully acknowledge funding of the Federal Ministry for Economic Affairs and Energy (BMWi: Grant No. 0324286, iPV4.0). In addition, we would like to thank the user study participants: J.~Hepp, M.~Mayr, A.~Vetter, J.~Denz, T.~Winkler and C.~Buerhop. Further, we would like to thank Ircam GmbH and Greateyes GmbH for providing the cameras and images and Rauschert Heinersdorf-Pressig GmbH for access to their photovoltaic plants.

{\small
\bibliographystyle{IEEEbib}
