

\subsection{Network Architectures}
LASSR is inherited from our previous GAN-based network \citep{cap2019super}. 
It is composed of two CNN models: the generator $G$, which generates SR images, and the discriminator $D$, which distinguishes SR images from HR images. 
The networks are trained together to solve an adversarial min-max problem. 

\subsubsection{The Generator}
Our scheme uses the architecture of a generator $G$, in the same way as in ESRGAN \citep{wang2018esrgan}. 
$G$ is composed of 23 residual-in-residual dense (RRDB) blocks, resulting in a total of 115 convolutional layers. 
Our network $G$ up-scales 4x from the input LR image. 
Fig. \ref{fig:fig_2} illustrates the architecture of the generator $G$ used in our experiments. 
The reader is referred to the ESRGAN paper for more technical details. 

\subsubsection{The Discriminator}
Our discriminator $D$ is designed in the same way as in our previous model \citep{cap2019super}. 
It is deeper than the discriminator used in ESRGAN, and has a larger input size of 192x192. 
The architecture of our discriminator $D$ is illustrated in Fig. \ref{fig:fig_3}. 
We define a convolution block as a block of two convolutional (Conv) layers. 
Following a Conv, we use either a leaky rectified linear function (LReLU) \citep{maas2013rectifier} or a combination of LReLU and batch normalization (BN) \citep{ioffe2015batch}. 
We use LReLU with $\alpha=0.2$ as the activation function for all layers except for the last. The reader is referred to our previous paper \citep{cap2019super} for more details. 

\subsection{Loss Functions of LASSR}
The objective functions in LASSR are extended from ESRGAN. 
To train our generator $G$, we minimize the loss function $\mathcal{L}_\mathrm{G}$ as follows:
\begin{equation}
\mathcal{L}_\mathrm{G}=\lambda \mathcal{L}_\mathrm{G}^\mathrm{adv}+\mathcal{L}_\mathrm{percep}+\eta{\left|I_\mathrm{HR}-I_\mathrm{SR}\right|}_1+\beta \mathcal{L}_\mathrm{ARM},
\end{equation}
where $\mathcal{L}_\mathrm{G}^\mathrm{adv}$, $\mathcal{L}_\mathrm{percep}$, and ${\left|I_\mathrm{HR}-I_\mathrm{SR}\right|}_1$ appear in the original loss function in ESRGAN. 
Here, $\mathcal{L}_\mathrm{G}^\mathrm{adv}$  is the adversarial loss for the generator $G$, and $\mathcal{L}_\mathrm{percep}$ is the perceptual loss \citep{johnson2016perceptual}, which minimizes the similarity between the HR and SR images in the feature space of the VGG19 model \citep{Simonyan15} pre-trained with the ImageNet dataset \citep{deng2009imagenet}. 
$I_\mathrm{HR}$ and $I_\mathrm{SR}$ are the HR and SR images, respectively. 
$\mathcal{L}_\mathrm{ARM}$ is our proposed novel loss term for calibrating the artifact effects, and is formed based on our proposed ARM (described in detail in the next section). 
$\lambda$, $\eta$, and $\beta$ are coefficients used to balance the different loss terms. 

To train our discriminator $D$, we use the same adversarial loss $\mathcal{L}_\mathrm{D}$ as in ESRGAN: 
\begin{equation}
\mathcal{L}_\mathrm{D}=-\mathbb{E}_{I_\mathrm{HR}}[\mathrm{log}(D(I_\mathrm{HR},I_\mathrm{SR}))]-\\
\mathbb{E}_{I_\mathrm{SR}}[\mathrm{log}(1-D(I_\mathrm{SR},I_\mathrm{HR}))].
\end{equation}
Finally, the $G$ and $D$ networks are trained together to solve an adversarial min-max problem \citep{goodfellow2014generative}. 
For more details of the loss functions, the reader is referred to the ESRGAN literature \citep{wang2018esrgan}. 

\subsubsection{The Artifact Removal Module}
In this work, we propose a novel ARM that detects and suppress artifacts on-the-fly during the training of our SR model. 
The ARM acts like a dynamic training strategy, and provides guidance allowing LASSR to suppress the occurrence of artifacts. 
The key idea of the ARM is to detect artifacts and then to minimize the differences between the areas of these artifacts and the corresponding areas of the ground-truth HR images. 
In most cases, artifacts appear in the form of similar specks or rubber stamps, as shown in Fig. \ref{fig:fig_1}. 
We refer to these artifact regions as blobs, and apply the difference of Gaussians (DoG) to detect them. 
DoG is an algorithm that finds scale-space maxima by subtracting different blurred versions of an original image. 
These blurred images are obtained by convolving the original images with Gaussian kernels with differing standard deviations. 
After obtaining the scale-space maxima, the areas of blobs are defined by the local maxima points and their corresponding Gaussian kernels. 

Fig. \ref{fig:fig_4} illustrates the steps in the process of detecting artifact areas (blobs) in SR images. 
Given a SR image and a ground-truth HR image, we first subtract two images to obtain the subtracted version. 
DoG is then applied to detect the locations of artifacts (blobs). 
Based on our preliminary experiments, artifact areas can be detected effectively using only two Gaussian kernels with corresponding values of $\sigma_1=0.078 \times N$ and $\sigma_2=0.104 \times N$, where $N \times N$ is the size of the cropped image used for training (i.e. $192 \times 192$ in our study). 

Once the artifacts have been detected, we add a new loss term $\mathcal{L}_\mathrm{ARM}$ in Eq. (1) to LASSR to allow the network to suppress the artifact. $\mathcal{L}_\mathrm{ARM}$ is defined as:
\begin{equation}
\mathcal{L}_\mathrm{ARM}=\sum_{b_i\in\mathfrak B}b_i,
\end{equation}
where $b_i$ is the sum of the pixel values of a blob detected in the subtracted image, and $\mathfrak B=\left\{b_1,b_2,...,b_{\left|\mathfrak B\right|}\right\}$ is the set of detected blobs. 
\input{tables/table_I}
\subsection{Datasets}
In these experiments, we used two datasets, Dataset-A and Dataset-B. 
Dataset-A was used to train and evaluate both LASSR and the ESRGAN model as a comparison. 
Dataset-B was used to train and evaluate the classifiers, in order to assess the effectiveness of the disease diagnostic ability of SR. 
These datasets are independent of each other. 

\subsubsection{Dataset-A for SR Models}
In this experiment, we used a cucumber dataset previously reported in the literature \citep{hiroki2018diagnosis,cap2019super} as Dataset-A. 
This is a multiple infection dataset with 25 classes containing a total of 48,311 cucumber leaf images, of which 38,821 show single infections, 1,814 show multiple infections, and 7,676 contain healthy leaves. 
Each image has a size of $316\times 316$ pixels. 
We divided this dataset into separate training and testing datasets. 
The training set contained 36,233 images (roughly 75\% of the dataset, referred to here as Dataset-$\mathrm{A}_\mathrm{Train}$), while the testing set contained 12,078 images (roughly 25\% of the dataset, referred to as Dataset-$\mathrm{A}_\mathrm{Test}$).

\subsubsection{Dataset-B for Disease Classifier}
Dataset-B was another cucumber leaf dataset collected from multiple locations in Japan, taken during the period 2015–2019. 
Table \ref{tab:table-I} summarizes the statistics for this dataset. 
It contained four classes of disease (\emph{Cucurbit chlorotic yellows virus} (CCYV), \emph{Melon yellow spot virus} (MYSV), \emph{Brown spot}, \emph{Downy mildew}) and healthy. 
Each image in this dataset had a size of $512 \times 512$ pixels. 
We divided this dataset into two sets, Dataset-$\mathrm{B}_\mathrm{Train/Val}$ and Dataset-$\mathrm{B}_\mathrm{Test}$. 
Images in Dataset-$\mathrm{B}_\mathrm{Test}$ were taken at different times and in different locations from those in Dataset-$\mathrm{B}_\mathrm{Train/Val}$ in order to avoid the problem of latent similarities among datasets, as mentioned earlier. 
Note that the appearances of the images in these sets varied widely, due to the differences in the circumstances (i.e. photographic conditions and background) in which they were taken.

\subsection{Experiment I: LASSR Improving Image Quality}
We first evaluated the improvement in image quality by LASSR and compared it to existing ESRGAN methods. 
We used FID scores \citep{heusel2017gans} as our evaluation criteria, in the same way as in other SR methods.  
This is because other standard quantitative measures such as PSNR and SSIM have been reported as being unable to capture and accurately assess the perceptual image quality which is highly correlated with human perception \citep{toderici2017full,ledig2017photo,zhang2018unreasonable}. 
We trained our LASSR and ESRGAN using Dataset-$\mathrm{A}_\mathrm{Train}$. 
During training, HR images were obtained by random cropping from training images with the size of $192 \times 192$. 
LR images were then created by $1/4 \times$ down-sampling from HR images, using bicubic interpolation. 
Both LR and HR images were augmented with random horizontal flips and random 90 degrees rotations on-the-fly. 
It should be noted that these are the same conditions that were used in the original training of ESRGAN. 

The ESRGAN model was trained using the loss functions from the original paper, while our LASSR was trained using the loss functions in Eqs. (1) and (2), with $\lambda = \beta = 5 \times 10^{-3}$ and $\eta = 10^{-2}$. We set the mini-batch size to 32 images and used the Adam optimizer \citep{kingma2015adam} for both $G$ and $D$ models. The training process was completed after 400 epochs. More details are given in the Results section.
\begin{figure}[!t]
\centering
\includegraphics[width=0.99\linewidth]{figures/Fig6.pdf}
\caption{
    Line profiles for the SR and the original HR images. We can see similar profiles of LASSR and HR, while that of ESRGAN shows significant dissimilarities due to the presence of artifacts.
}
\label{fig:fig_6}
\end{figure}
\subsection{Experiment II: LASSR Improving Diagnostic Performance on an Unseen Dataset}
In this experiment, we trained similar plant disease classifiers using LR, HR and SR images using both LASSR and ESRGAN for comparison. 
Specifically, we trained the following five classiﬁers based on the pre-trained ResNet50 \citep{he2016deep} model: 
\renewcommand{\theenumi}{\arabic{enumi}}
\begin{enumerate}
    \item ResNet\_LR: The classifier was trained on a 1/4 x down-sampled Dataset-$\mathrm{B}_\mathrm{Train}$ (i.e. input size $128 \times 128$) using bicubic interpolation.
    \item ResNet\_HR: The classifier was trained with Dataset-$\mathrm{B}_\mathrm{Train}$ (i.e. input size $512 \times 512$).
    \item ResNet\_Bicubic: As in (1), except with training images 4x SRed (i.e. $512 \times 512$) using bi-cubic interpolation.
    \item ResNet\_ESRGAN: As in (1), except with training images 4x SRed (i.e. $512 \times 512$) using ESRGAN.
    \item ResNet\_LASSR [proposed model]: As in (1), except with training images 4x SRed (i.e. $512 \times 512$) using LASSR.
\end{enumerate}
All five ResNet50-based classifiers were fine-tuned at all layers with the Adam optimizer \citep{kingma2015adam}. 
To handle the class imbalance in Dataset-$\mathrm{B}_\mathrm{Train}$, we applied the softmax class-balanced loss \citep{cui2019class} with $\beta=0.9999$ to all classifiers. 
During the training process, we applied random horizontal and vertical flips to each image. 
Training was complete after 20 epochs. 
\subsection{LASSR: Improving Image Quality}
Figs. \ref{fig:fig_5} and \ref{fig:fig_6} show the visual comparison and line profiles of the generated images and the original HR image.
Table \ref{tab:table-II} shows the FID scores of the images generated using the bicubic, ESRGAN and LASSR techniques versus the original HR images. 
\input{tables/table_II}
In a similar way as for Dataset-$\mathrm{A}_\mathrm{Test}$, we calculated the FID score for the entire Dataset-B, since it is exclusive from Dataset-$\mathrm{A}_\mathrm{Train}$. Note that the test images were $1/4 \times$ down-sampled before being fed to the SR models. 

Our LASSR scheme successfully suppressed the artifacts, and generated images that were more natural than the ESRGAN method. 
We also observed that for Dataset-$\mathrm{A}_\mathrm{Test}$, ESRGAN produced over 1,300 artifact images (11.39\% of the dataset), while our LASSR created only 177 cases (1.47\%). 
On Dataset-B, which was exclusive of Dataset-A, ESRGAN still produced 380 cases of artifacts (0.66\%) while our LASSR generated zero artifact (artifact-free) images. 
LASSR also achieved a better (lower) FID score than ESRGAN on both datasets. 

\subsection{LASSR: Improving Diagnostic Performance on Unseen Dataset}
Table \ref{tab:table-III} presents a comparison of the diagnostic performance of disease classiﬁers in Experiment II. 
We can see that there is a large gap between the micro-average accuracy of the Dataset-$\mathrm{B}_\mathrm{Val}$ and Dataset-$\mathrm{B}_\mathrm{Test}$, since the two sets are inherently different in nature due to differences in the photographed locations of the images and other factors as mentioned earlier. 
However, our LASSR model significantly helped boosting the performance on unseen dataset from the baseline model, achieving a competitive result. 
\subsection{Improvement in Image Quality}
From the results of the Experiment I, thanks to the introduction of the novel ARM, our LASSR effectively addresses the problem of artifacts and produces more natural SR images compared to the results from ESRGAN (see Figs. \ref{fig:fig_1}, \ref{fig:fig_5} and \ref{fig:fig_6}), and achieves better FID scores (Table \ref{tab:table-II}). 
Although some artifact cases remain in the images generated by LASSR on Dataset-$\mathrm{A}_\mathrm{Test}$, it should be noted that our LASSR produced much weaker artifacts than ESRGAN (see Fig. \ref{fig:fig_5}, last row). 
Moreover, LASSR generated artifact-free SR images on Dataset-B.

The incidence of ESRGAN artifacts was 11.4\% for Dataset-$\mathrm{A}_\mathrm{Test}$, which consisted of images taken in the same field as the training data. 
The rate was far lower for Dataset-B at 0.66\%, on the completely unknown dataset. 
In addition, the artifacts generated by ESRGAN were almost all of the same size (where each side of the boundary box was about 1/10th the size of the image) and had circle-like shapes, as if the image had been stamped with a rubber stamp. 
This may be due to the particular combination of the input image and kernels in a relatively forward convolutional layer corresponding to a specific receptive field size. 
However, we have not been able to identify the reason for this. 
Although our ARM effectively suppressed and reduced these artifact effects, it seems likely that further investigation of the behavior of the convolutional layers could be a way to be toward the artifact-free SR, and we aim to address this in the near future.
\input{tables/table_III}
\subsection{Improvement in Disease Diagnosis Performance}
In Experiment II, although the diagnostic accuracy of all of the classifiers on the two subsets Dataset-$\mathrm{B}_\mathrm{Train/Val}$ was similar and very high, the performance was significantly reduced compared to Dataset-$\mathrm{B}_\mathrm{Test}$. 
As discussed above, this is due to the problem of latent similarities between data \citep{quan2018,suwa2019comparable,saikawa2019aop}. 
As frequently reported in the literature, the performance was higher for data from the same imaging environment (Dataset-$\mathrm{B}_\mathrm{Train/Val}$) due to overfitting, but lower for data from a different environment (Dataset-$\mathrm{B}_\mathrm{Test}$). 
This reduction in performance is generally known as a covariate shift. 

The ResNet\_LR and ResNet\_Bicubic models showed poor performance on Dataset-$\mathrm{B}_\mathrm{Test}$ (with micro-average accuracies of only around 54\% and 61\%, respectively, as shown in Table \ref{tab:table-III}.a). 
This was because the size of the LR input images was not sufficient for the diagnosis of plant diseases. 
Our LASSR scheme successfully reconstructed the information from LR images and helped the ResNet\_LASSR model to achieve high diagnostic performance, significantly outperforming the model trained on LR images (ResNet\_LR) by nearly 22\% in terms of micro-average accuracy (Table \ref{tab:table-III}.a). 
Moreover, ResNet\_LASSR achieved performance that was 2.2\% and 2.1\% better than ResNet\_ESRGAN in terms of micro- and macro-average accuracy, respectively (Table \ref{tab:table-III}). 

For the healthy plants, ResNet\_LASSR (86.6\%) was less numerically accurate than ResNet\_ESRGAN (90.2\%), unlike in the cases of disease. 
In this case, ResNet\_LR also shows a high degree of accuracy in diagnosis (85.5\%), and the difference from ResNet\_HR (89.1\%) is small, with ResNet\_LASSR falling in between. 
This may be due to the fact that healthy cases have essentially no signs of disease that can be detected, and there is therefore little need for HR information. 
Although we cannot determine a direct reason for the inferiority of ResNet\_LASSR compared to ResNet\_ESRGAN, we believe that the diagnostic ability of ResNet\_LASSR for healthy plants is sufficiently effective from a practical standpoint. 

It should also be noted that in terms of CCYV diagnosis, there was a reversal in the performance of ResNet\_LR and ResNet\_Bicubic. 
It appears that neither model could detect the essential features of this disease, although the latter had a wastefully larger degree of freedom, which reduced its generalizability. 

Except for the healthy images, ResNet\_LASSR performed significantly better than ResNet\_ESRGAN in all cases, achieving the closest result to the model using the original HR images. 
This reinforces our argument for the benefits of using the SR method. Since high-quality training data are not always available in practice, SR techniques are confirmed to be effective in generating reliable training resources and improving the robustness of diagnosis systems. 
Note that we also trained another classifier with HR images of size $224\times 224$ (as commonly used in many disease classifiers) and recorded a diagnostic of around 72\% in terms of micro-accuracy. This implies that the quality of the training image is important. 

Although LASSR achieved promising results, we note that the selection of hyper-parameters for the ARM is still done manually, as it depends on the input size. 
We believe that the development of an ARM that dynamically adapts the training set will further improve the effectiveness of LASSR. 

\section{Introduction}
    \input{01_introduction}

\section{Proposed Method - LASSR}
    \input{02_method}

\section{Experiments}
    \input{03_experiment}

\section{Results}
    \input{04_results}

\section{Discussion}
    \input{05_discussion}

\section{Conclusion}
    \input{06_conclusion}

\section*{Acknowledgment}
This research was conducted as part of a research project on future agricultural production and was supported by the Ministry of Agriculture, Forestry and Fisheries of Japan.

\nocite{*}
\footnotesize{
\bibliographystyle{IEEEtran}



