\section{Introduction}

High-level computer vision tasks, including object detection and segmentation, are highly dependent on the quality of captured images. However, outdoor vision systems such as security monitoring and vision-based self-driving cars or driving assistance systems are easily influenced by severe weather or other environments. As shown in Fig.~\ref{fig:1}(a), raindrops adhered to camera lens or windshield can highly degrade the image or obstruct the visibility.

\begin{figure}
	\centering
	\includegraphics[width=0.9\linewidth]{figures/fig1_0306}
	\caption{Examples of adherent raindrops and/or mist. (a) Raindrops adhered to a camera lens (Image source: The Nature Conservancy). (b) Inhomogeneous mist adhered to a windshield. (c) Mist and raindrops adhered to a camera lens}
	\label{fig:1}
\end{figure}

Adherent raindrops (or waterdrops) exist widely in different scenes. Without enough shelter, rain streaks will result in raindrops naturally. In special environments like fishing-boats, splashing water can also form waterdrops. Raindrops can have various visual effects because of various drop sizes, different distances to the camera, and complex refraction, which makes it extremely difficult to model the raindrops manually.

Besides raindrops, another severe and commonly-observed interference, adherent mist, is rarely mentioned in computer vision. As shown in Fig.~\ref{fig:1}(b), mist adhered to the windshield can also severely hamper the vision, which is vitally dangerous for driving.

Adherent mist is often caused by the temperature difference of air between two sides of the glass window. Unlike the haze in the air, mist adhered to glass windows is usually inhomogeneous (e.g., Fig.~\ref{fig:1}(b)).

Interestingly, adherent mists can sometimes develop along with the raindrops, especially for driving, where there is a temperature difference between the warm steam inside and the cooled glass by the raindrops outside. Therefore, it is necessary to handle both adherent mist and raindrops together. Fig.~\ref{fig:1}(c) shows the overlapping of mist and raindrops adhered to a camera lens.

By modifying the formulas in \cite{chen2019gated,li2019single,qian2018attentive,ren2018gated}, the degraded image $I^{cap}$ captured by the camera can be modeled as a combination of raindrops $R$, the scattering of adherent mist $A$, and the clean background $B$ as follows:
\begin{align}
I^{cap} = ((1-M)\otimes B + R)\odot t + A \odot (1-t)
\end{align}
where M denotes to a binary mask showing the existence of raindrops, $\odot$ means element-wise multiplication, and t is the transmission map indicating how much information has passed through the adherent mist. $I^{cap}$, $M$, $B$, $R$, $t$, and $A$ are location-related maps (i.e., matrices) instead of constants.

In this study, we aim to remove the adherent mist and raindrops from a single impaired image to obtain a clean image. The proposed approach, based on convolutional neural networks (CNNs), can automatically visually remove raindrops and mist of varying degrees. It can benefit high-level vision algorithms and improve their robustness in severe weather.

As both raindrops and mist might block some objects partially or even entirely, it is challenging to restore a purely clean image. To better solve the problem, we apply smoothed dilated convolution \cite{wang2018smoothed} into our model to obtain a relatively large receptive field instead of general pooling operations that might lose spatial information. Dual attention modules consisting of channel attention and spatial attention are adopted for information selection and efficient learning. Perceptual loss is also applied in order to generate intuitively realistic images and avoid over-smoothing details \cite{johnson2016perceptual,ledig2017photo}.

Besides, to strengthen the spatial attention and handle the inhomogeneity, the classification activation map (CAM) \cite{zhou2016learning} for each captured image is then calculated based on a newly trained classification network and acts as the spatial attention input concatenated with the original RGB image. The classification network is designed to accurately judge if the collected image is clean, with raindrops only, or with both raindrops and adherent mist. The proposed CAM attention can effectively locate important regions of the input image and improve the restoration performance, without hand-crafted priors.

Thus, our contributions are four-fold as follows:
\begin{itemize}
	\item We address the visual interference problem caused by adherent mist and raindrops, which is frequent and essential but less studied. Without appropriate solutions, this problem possibly limits extensive outdoor camera-based systems.
	\item A CNN-based method is proposed to jointly remove adherent mist and raindrops. The proposed CNN is designed with smoothed dilated convolution, dual attention module, and perceptual loss to achieve state-of-the-art performance. The peak signal-to-noise ratio (PSNR) and structural similarity index (SSIM) \cite{wang2004image} values of the test set is improved from 17.9213 and 0.6283 to 21.8073 and 0.7549, compared with the degraded input.
	\item We introduce CAM attention as an efficient spatial attention input. Except for accessible classification labels, it does not require standard assumptions to find out worthy-attention pixel locations. This technique can be easily implemented in other low-level image generation problems.
	\item The improvements of high-level algorithms using enhanced images are demonstrated, showing the importance of both the problem and the proposed method. We also provide a new public dataset, which contains image pairs of ground truth (clean) images and degraded images (with raindrops and adherent mist). Further studies could use our dataset to develop new algorithms.
\end{itemize}


\section{Related Work}

Most visibility enhancement algorithms related to rain weather focused on rain streaks \cite{fu2017removing,li2019single,li2017single,wang2019erl,zhang2018density}. However, the shapes and physical effects of raindrops exceedingly differ from those of rain streaks. Thus, these streak removal algorithms can not be directly applied to raindrop removal. 

A few studies about raindrop detection have been proposed for years. Kurihata et al. \cite{kurihata2005rainy} utilize principal component analysis to learn the characteristics of raindrops, while Ito et al. \cite{ito2015adherent} employ the maximally stable extremal regions to detect raindrop candidates. However, these methods do not concentrate on raindrop removal. You et al. \cite{you2015adherent} pay attention to both raindrop detection and removal in videos. They notice the temporal change of intensity of raindrop pixels is different from that of non-raindrop pixels, which is not applicable to the single image case. Eigen et al. \cite{eigen2013restoring} may be first to address the single image raindrop removal problem. They use a straightforward idea which trains a shallow CNN with pairs of raindrop images and clean images. Unfortunately, the limited capacity of the CNN model restricts its performance, especially for large and dense raindrops.

Qian et al.'s work (DeRaindrop) in 2018 \cite{qian2018attentive} might be the first practical solution for the raindrop removal task. Specifically, they build a generative adversarial network (GAN) \cite{goodfellow2014generative} to generate raindrop-free images. Besides the GAN, DeRaindrop also includes a recurrent network to gradually locate raindrops’ location in the input image, working as the spatial attention for the GAN. The performance of DeRaindrop is related to this raindrop mask, whose ground truth is calculated by the difference between the degraded image and the clean image with a hand-crafted thresholding 30. However, it is difficult to apply this manual threshold to images with adherent mist, which contain complex variations of pixel values. Quan et al. \cite{quan2019deep} propose shape-driven attention for raindrop regions, which is based on priors about the shape of raindrops.

Few works primarily focus on the adherent mist. Some similar studies mainly concentrate on atmospheric haze (or fog) \cite{cai2016dehazenet,chen2019gated,fattal2008single,he2010single,ren2018gated}. On the one hand, they both have a scattering effect, which results in “white” color and degrades the original information. But compared to atmospheric haze, adherent mist is more likely to be inhomogeneous as it is related to the specific design of the equipment, such as which direction the warm steam comes from and how the window looks (e.g., mist adhered to the windshield near the driver’s seat in Fig.~\ref{fig:1}(b)).

Single image dehazing is initially explored using some priors-based methods. Fattal et al. \cite{fattal2008single} work on estimating the albedo of the scene for single image dehazing. He et al. \cite{he2010single} propose a dark channel prior which talks about the local minimum of the dark channel varies between haze and haze-free images.
 
Then some learning-based methods have been proposed for dehazing using a single image. Cai et al. \cite{cai2016dehazenet} introduce an end-to-end CNN to estimate the transmissions from images with haze. Ren et al. \cite{ren2018gated} propose multi-scale gated fusion network to improve the performance. And the smoothed dilation technique is adopted by Chen et al. \cite{chen2019gated} to remove gridding artifacts.

However, in our work, the superposition of adherent mist and raindrops results in complex local features. The influence of raindrops limits general dehazing algorithms using transmission estimation. It is also difficult to manually input an attention map similar to that in \cite{qian2018attentive} or \cite{quan2019deep}. Therefore, the proposed network is designed to efficiently utilize features and restore images for the practical scene.


\section{Methods}
\subsection{Network Architecture}

The proposed network is shown in Fig.~\ref{fig:2}. The degraded image concatenated with its CAM attention map is used as the input for a generative network to output the clean image. The CAM attention is generated through a newly trained classification network. Since the classification network can accurately distinguish the type of the degraded image, it can also inform us which regions contribute more to the classification result. The generative network applies the architecture of an autoencoder with some shortcut connections. The input is initially convolved by two convolutional layers with dual attention modules. After a convolutional layer with stride 2, the spatial size of the features is decreased. Smoothed dilated convolution modules, as well as dual attention modules, are successively applied to handle those features further. Shortcut connections are inspired by the residual learning concept \cite{he2016deep}. Finally, the feature maps are converted back to the original size by a transpose convolutional layer and tuned by two convolutional layers to generate the output. Instance normalization \cite{ulyanov2016instance} is also applied in the generative network. 

\begin{figure}[t]
	\centering
	\includegraphics[width=1.0\linewidth]{figures/fig2_0306_2}
	\caption{The architecture of the proposed network. The degraded input image is firstly used to generate a CAM attention map with a classification network. Then the input image concatenated with the CAM attention map is handled by the generative network to obtain the clean output. The details of the smoothed dilated convolution module and the dual attention module are elaborated in the following figure}
	\label{fig:2}
\end{figure}

\subsubsection{CAM Attention}
It might be helpful to handle the inhomogeneous features \cite{qian2018attentive} as shown in Fig.~\ref{fig:1}, with additional spatial attention maps instead of pure CNNs. Although related works (i.e., \cite{qian2018attentive,quan2019deep}) have generated such attention maps, they use a manually set threshold or shape prior, which might be less feasible to do so for complex images with adherent mist and raindrops. Instead, we utilize CAM to adaptively generate the attention maps.

CAM is proposed by Zhou et al. \cite{zhou2016learning}, which contributes to the visual interpretation of a CNN ``black box''. For a classification network, CAM can show the importance of various spatial regions in the image to the prediction result. This mechanism is achieved by weighting different channels of the final convolutional layer's features (denoting as $f_k(x,y)$ for a spatial position $(x,y)$ of channel $k$) using the weights of the dense layer (denoting as $\omega_k$) with respect to the prediction result $t$, which can be described as \cite{zhou2016learning}:
\begin{align}
M_t(x,y) = \sum_{k}\omega_{k}^{t}f_k(x,y)
\end{align}
where $M_t$ denotes to the CAM result. The features are resized to the shape of the input before multiplication, and the activation map is generated by adding values from all channels for each spatial position and normalization.

The architecture of our simple classification network is briefly shown in Fig.~\ref{fig:2}, which is similar to the structure in \cite{he2020restoration}, with 7 convolutional layers and 1 dense layer. A global maximum pooling layer is omitted in Fig.~\ref{fig:2}, which is used to flatten 2D features to 1D feature vector before the dense layer. A softmax activation function is applied to obtain the one-hot result. 

Specifically, our classification task aims to distinguish images in three situations: clean, with raindrops (and without mist), and with both adherent mist and raindrops. Once the class of the image is accurately identified, CAM can hopefully explain which regions in the input image contribute more to the prediction. Four examples of CAM attention are shown in Fig.~\ref{fig:3}.

\begin{figure}
	\centering
	\includegraphics[width=1.0\linewidth]{figures/fig3_0306}
	\caption{Examples of CAM attention. The first row shows the original RGB images, and the corresponding CAM is shown below. (a--b) Images from \cite{qian2018attentive} degraded by raindrop only. (c--d) with raindrops only. (c--d) Images collected in our dataset with both adherent mist and raindrops. All CAM results share the same color bar on the right}
	\label{fig:3}
\end{figure}

For images with raindrops only (Fig.~\ref{fig:3}(a--b)), CAMs show high activation for raindrop regions. And for images with both adherent mist and raindrops (Fig.~\ref{fig:3}(c--d)), except for some highlighted raindrops, we can visualize some highlighted edge textures in CAMs (e.g., the tall building on the left of Fig.~\ref{fig:3}(c), and the grids on the wall in Fig.~\ref{fig:3}(d)). The attention results are reasonable as the CNN might classify the image based on the existence of raindrops and the edge variation behind the mist.

Raindrops regions and edge textures (i.e., high-frequency details) are actually the parts that we mostly focus on when observing the restoration results, which means regions that are important to the classification task are probably significant for the restoration task too. Therefore, we concatenate the original RGB image with its CAM and feed them to the generative network for image restoration.

\begin{figure}
	\centering
	\includegraphics[width=0.9\linewidth]{figures/fig4_0305}
	\caption{The architecture of the smoothed dilated convolution module and the dual attention module. (a) The smoothed dilated convolution module.  The same module adopts the same dilation rate. The adding operation indicates the element-wise sum. (b) The dual attention module. The two dense layers in the channel attention part share the same weights for output from both GAP and GMP in each iteration. The multiplication operation in channel attention weights different channels, and the multiplication operation in spatial attention processes various spatial positions}
	\label{fig:4}
\end{figure}

\subsubsection{Smoothed Dilated Convolution Module}
Dilated convolution is one of the useful ways to obtain a relatively large receptive field without decreasing the feature size like pooling. The convolution kernel covers small regions larger than the kernel size with some skipped pixels. However, dilated convolution might lead to gridding artifacts as described in \cite{chen2019gated}. To handle this disadvantage, \cite{wang2018smoothed} propose smoothed dilated convolution by adding a general convolution layer before the dilated convolution operation. As a result, the result of the dilated convolution is not only relied on spaced pixels but also some continuous information. 


The smoothed dilated convolution module applied in our method is shown in Fig.~\ref{fig:4}(a). A standard convolution layer is added before each dilated convolution layer. When the dilation rate of the dilated convolution kernel is $d$, the kernel size of the corresponding standard convolution kernel should be $2d-1$ in each dimension. In this way, the information from the skipped feature pixels in the input feature map could also be used in the dilated convolution to avoid gridding artifacts. Specifically, there are 6 smoothed dilated convolution modules in our network as indicated in Fig.~\ref{fig:2}. The first 3 modules apply the dilation rate of 2, and the last 3 modules adopt the dilation rate of 4. 

\subsubsection{Dual Attention Module}
Considering the inhomogeneous distribution of mist and raindrops, another spatial attention mechanism with a self-attention concept besides the CAM attention is also adopted for feature selection. Besides, as introduced in \cite{hu2018squeeze}, channel attention can also be useful to improve information extraction. These two attention parts form the dual attention module. And 12 modules are inserted in the generative network as indicated in Fig.~\ref{fig:2}.

As shown in Fig.~\ref{fig:4}(b), we build the dual attention module based on \cite{woo2018cbam}. Firstly, feature maps are processed by global average pooling (GAP) and global maximum pooling (GMP), individually, and become 1D vectors with the length of the number of input channels. Then, the two vectors are fed into two shared dense layers individually before being added together. The shared dense layer means the same weights are used to the two inputs from GAP and GMP. The multiplication operation in the channel attention gives different channels in the feature tensor different weights according to the vector.

The spatial attention is followed by the channel attention. Maximum pooling along the channel axis (MPC) indicates that the maximum value at each spatial position is obtained by comparing pixels from all channels at the same position. Average pooling along the channel axis (APC) is a similar operation for average values. The multiplication operation weights different spatial positions in the input tensor according to the 1-channel attention map.

\subsection{Loss Function}
As indicated in \cite{johnson2016perceptual,ledig2017photo}, although pixel-wise mean squared error (MSE) might bring improvements on metrics like peak signal-to-noise (PSNR) and structural similarity index (SSIM) \cite{wang2004image}, the generated image is likely to be overly smoothed. Instead, a perceptual loss term can be applied to calculate the error from high-level views, which might be helpful in generating subjectively realistic images.

We calculate the perceptual loss based on the MSE of the output features from the 7th convolution layer of VGG19 \cite{ledig2017photo,simonyan2014very}. The VGG19 model is pretrained on the ImageNet dataset \cite{deng2009imagenet}, which might give a relatively high-level description of an image. Based on the restored image $I^{out}$ and the corresponding ground truth $I^{gt}$, the two output feature maps of the 7th convolutional layer are $f(I^{out})$ and $f(I^{gt})$, respectively. Then the perceptual loss is:
\begin{align}
Loss_{per} = \frac{1}{n_x \times n_y \times n_c} \sum_{x=1}^{n_x} \sum_{y=1}^{n_y} \sum_{c=1}^{n_c}(f(I^{gt})_{x,y,c} - f(I^{out})_{x,y,c})^2
\end{align}
where $x$, $y$, $c$ denote the dimensions of the feature map.

Finally, the perceptual loss is added together with the general MSE loss as:
\begin{align}
Loss_{total} = \lambda_1Loss_{mse} + \lambda_2Loss_{per}
\end{align}
where $\lambda_1$, $\lambda_2$ are the weights of items and $Loss_{mse}$ is calculated by directly comparing $I^{out}$ and $I^{gt}$. $\lambda_1$, $\lambda_2$ was set as 1.0 and 0.05, respectively.

\subsection{Dataset Preparation}
To prepare the image pairs consisting of clean and degraded images, we take photos following strategies similar to those in \cite{qian2018attentive}. Specifically, we utilize two same glass panels to simulate different cases. Firstly, we fixed the camera using a tripod. Secondly, one of the panels was randomly sprinkled with waterdrops on one side, and randomly sprayed with mist by a cosmetic sprayer on the other side. Then, we placed the blurred panel in front of the camera and took a photo as the degraded image. At last, the corresponding ground truth (i.e., clean) image was taken with the other clean glass panel.

The distributions of waterdrops and mist are random to keep good generalization. And the distance between the camera and the glass panel is also a random value between 0.5cm--4.0cm, thus the shape of waterdrops can have different appearances in the image due to the influence of camera focal length. The integrated camera of Redmi K20 Pro is used for taking pictures.

Finally, all the collected images were resized with bicubic interpolation to the shape of $640\times480$ to form the dataset. There are 572 image pairs for training, 73 image pairs for validation, and 72 image pairs for test. All the following evaluation results are based on images in the testing set.


\section{Experiments}
\subsection{Experiment Setup}

Both the classification network and the generative network are implemented on Keras framework with Tensorflow backend. Both models are trained using an Nvidia Titan RTX GPU with Adam optimizer. The learning rate of 1e-4 and the batch size of 8 were utilized for the classification network, while for the generative network, the learning rate is set as 1e-3, and the batch size is 4. During training, data augmentation operations, including flipping, rotation (180 degrees), brightness and contrast fluctuation, and gamma tuning, were applied.

In addition, only the collected dataset described in Sec. 3.3 is used to train the generative network. However, the image pairs of raindrops dataset from \cite{qian2018attentive} are mixed with our dataset to train the classification network for raindrops classification. To avoiding data leaking, the testing images of our captured data that will be used to evaluate the performance of the image restoration are not used for training the classification network.

PSNR and SSIM are adopted for evaluating our model's performance. Their values are calculated based on the luminance channel, so the RGB images are converted into YCrCb mode before evaluation.


\subsection{Classification Results}
The classification model was evaluated with 210 test images (66 from the dataset of \cite{qian2018attentive}, and 144 from our dataset). The classification network achieves the accuracy and f1-score of both about 96\%. The relatively high classification performance might contribute to generate relatively reliable CAM attention results.


\subsection{Restoration Results}
To our knowledge, there are no other comparable studies that work on the exact same problem (i.e., jointly removal of adherent mist and raindrops). Thus, we compare the proposed method with the following three strategies:
\begin{itemize}
	\item[(1)] Our method is compared with two two-step combinations. A two-step combination method consists of a pre-trained general dehazing approach (that originally focuses on atmospheric haze) and the pre-trained raindrop removal model DeRaindrop (D) by \cite{qian2018attentive} to restore the image. In other words, the degraded image is firstly processed by a dehazing model (i.e., GCANet \cite{chen2019gated} (G), or FFANet \cite{qin2019ffa} (F)) and then by D. The two combinations are named as ``G+D'' and ``F+D'', individually.
	\item[(2)] We also tried to apply D before G (or F) for comparison. These two inverse combinates are denoted as ``D+G'' and ``D+F'', respectively.
	\item[(3)] Although originally designed for other tasks, D, G, and F are advanced generative network that could be applied to our dataset. Therefore, we re-trained these three networks using our dataset. The three re-trained methods are denoted as ``re-D'', ``re-G'', and ``re-F'', respectively.
\end{itemize}
The quantitative results of the three strategies above are listed in Table~\ref{table:1}. DI stands for ``Degraded Images''.
\setlength{\tabcolsep}{2pt}
\begin{table}
	\begin{center}
		\caption{The quantitative results of different methods}
		\label{table:1}
		\begin{tabular}{p{25pt}p{30pt}p{30pt}p{30pt}p{30pt}p{30pt}p{30pt}p{30pt}p{30pt}p{30pt}}
			\hline\noalign{\smallskip}
			   & DI & G+D & F+D & D+G & D+F & re-D & re-G & re-F & Ours\\
			\noalign{\smallskip}
			\hline
			\noalign{\smallskip}
			PSNR & 17.9213 & 17.4249 & 19.4054 & 18.1926 & 19.3557 & 19.3927 & 19.4788 & 20.3022 & \it{21.8073}\\
			SSIM & 0.6283 & 0.6601 & 0.6825 & 0.6758 & 0.6831 & 0.6982 & 0.6844 & 0.7197 & \it{0.7549}\\
			\hline
		\end{tabular}
	\end{center}
\end{table}


The results of Table~\ref{table:1} show that, firstly, it is important to handle adherent mist and raindrops together. The co-existence of mist and raindrops might lead to horrible results that even worse than the degraded images (e.g., the PSNR result of G+D in Table~\ref{table:1}). Secondly, the combination of F and D outperforms the combination of G and D with both processing orders. Thirdly, the three re-trained models generally show the effectiveness of re-training, especially for G and F. But our method always outperforms other methods.

In practical scenes, the combination of adherent mist and raindrops might severely impair the image, which might block objects from the subjective view. Examples in Fig.~\ref{fig:5} and Fig.~\ref{fig:6} exhibit the awful shooting environment and the restoration results from several methods above. Due to the space limit, we select D+G and F+D methods (Fig.~\ref{fig:5}), and re-F and re-D methods (Fig.~\ref{fig:6}) as representatives for visual comparison. The intuitive restoration examples of all the above methods are provided in the supplementary material.

\begin{figure}[t]
	\centering
	\includegraphics[width=1.0\linewidth]{figures/fig5_0305}
	\caption{Examples of results from D+G, F+D, and our method. Each row corresponds to the same sample. The values below images (except ground truth images) indicate the corresponding values of PSNR / SSIM}
	\label{fig:5}
\end{figure}
\begin{figure}[t]
	\centering
	\includegraphics[width=1.0\linewidth]{figures/fig6_0305}
	\caption{Examples of results from re-F, re-D, and our method. Each row corresponds to the same sample. The values below images (except ground truth images) indicate the corresponding values of PSNR / SSIM. The black bars in three of the sub-images of the second row are used to hide the location information}
	\label{fig:6}
\end{figure}

From Fig.~\ref{fig:5} and Fig.~\ref{fig:6}, we can find that some objects are not distinguishable in the degraded image. For example, in the first row of Fig.~\ref{fig:5}, it is hard to count how many cars in the degraded image, and in the second row of Fig.~\ref{fig:6}, the bicycles in front of the car almost disappear in the degraded image, which is vital for self-driving applications. If without the correct judgment, the safety of driving algorithms is to be doubted. This problem can be effectively handled by our method, as shown in the fifth column in Fig.~\ref{fig:5} and Fig.~\ref{fig:6}. In addition, the proposed method can adaptively handle degradations with different extents. For both light degradation (e.g., the last example in Fig.~\ref{fig:6}) and severe degradation (e.g., the second example in Fig.~\ref{fig:5}), our method can always keep the reasonable brightness and fine textures.

According to the attached PSNR / SSIM values (Fig.~\ref{fig:5} and Fig.~\ref{fig:6}), which are used to measure the similarity between the given image and the ground truth, it is proved that the proposed method can obtain relatively cleaner images and fewer artifacts than the other methods. The two combination methods (D+G and F+D in Fig.~\ref{fig:5}) sometimes result in serious luminance deviations. And the two re-trained methods (re-F and re-D in Fig.~\ref{fig:6}) show more artifacts than the proposed method.


\subsection{Ablation Study}
To verify the effectiveness of various components of the proposed method, we investigate the ablation study as follows. Besides the proposed architecture, we trained a model without CAM attention (Non-CAM), a model without channel attention (Non-CA), a model without spatial attention (Non-SA), and a similar model that replaces the smoothed dilated convolution by standard dilated convolution (Non-SD). Other designs of each model remain the same.

\setlength{\tabcolsep}{7pt}
\begin{table}
	\begin{center}
		\caption{The ablation study results from different network designs}
		\label{table:2}
		\begin{tabular}{cccccc}
			\hline\noalign{\smallskip}
			& Non-CAM & Non-CA & Non-SA & Non-SD & Ours\\
			\noalign{\smallskip}
			\hline
			\noalign{\smallskip}
			PSNR & 21.2289 & 21.4883 & 21.4294 & 21.2753 & \it{21.8073}\\
			SSIM & 0.7494 & 0.7546 & 0.7514 & 0.7442 & \it{0.7549}\\
			\hline
		\end{tabular}
	\end{center}
\end{table}

The statistical results of these four models using our testing set are shown in Table~\ref{table:2}. The performance decreases if any one of the above components is removed, especially for PSNR metrics. For example, without the CAM attention mechanism, the PSNR decrease by about 0.6dB. It is demonstrated that all those techniques effectively contribute to our model's performance.



\subsection{Applications}
Besides the quantitative and qualitative evaluation in Sec. 4.2, the importance of the proposed problem and the effectiveness of our method are also validated in practical applications, such as for instance segmentation. We apply a cutting-edge segmentation algorithm, Cascade R-CNN method \cite{cai2019cascade} from Detectron2 (https://github.com/facebookresearch/detectron2) model zoo to segment the images including the ground truth images, degraded images, and restored images as shown in Fig.~\ref{fig:7}. This advanced algorithm above also detects 2D objects with bounding boxes. For a fair and representative comparison, methods with relatively high-performance such as F+D and re-F are applied in this subsection. Segmentation results based on images restored by other methods are provided in the supplementary material.


\begin{figure}[t]
	\centering
	\includegraphics[width=1.0\linewidth]{figures/fig7_0305}
	\caption{Instance segmentation results using images from different sources: (From left to right) ground truth, degraded, F+D, re-F, and our method. Each row corresponds to the same sample. Small objects are best viewed by zoom-in}
	\label{fig:7}
\end{figure}

\begin{figure}[t]
	\centering
	\includegraphics[width=1.0\linewidth]{figures/fig8_0903}
	\caption{Panoptic segmentation results using images from various sources: (From left to right) ground truth, degraded, F+D, re-F, and our method. For each scene, the segmentation results are attached below their corresponding images. The white bars in the second sample are added to hide the location information. Small objects are best viewed by zoom-in}
	\label{fig:8}
\end{figure}

As shown in the first row of Fig.~\ref{fig:7}, there is a car ``missed'' in the degraded image, the image from F+D method, and the image from re-F method. And in the second row, the segmentation algorithm can not find the bicycles in the degraded image, and F+D recovered image. Thus, it is proved that adherent mist degraded images not only impede our subjective views, but also impair the performance of computer algorithms. For the third sample, the segmentation algorithm mistakenly regards the building as a train in the degraded image, and F+D recovered image. The error possibly leads to severe consequences in practical applications.




In addition, panoptic segmentation is a newly proposed segmentation task \cite{kirillov2019panoptic} which tries to label all the pixels in the image. The results of panoptic segmentation based on various images are shown in Fig.~\ref{fig:8}. We applied R101-FPN method in Detectron2 (https://github.com/facebookresearch/detectron2) model zoo as the panoptic segmentation algorithm.



Panoptic segmentation might be more natural to show the influence of adherent mist and raindrops, as it demands all the spatial pixels of the image. In the first sample (1st and 2nd rows in Fig.~\ref{fig:8}), the segmentation algorithm can not clearly find the border between the road and the grass, as well as the border between the tree and the sky in the degraded image. In the second sample (3rd and 4th rows in Fig.~\ref{fig:8}), segmentation results of the middle three images are horrible, especially for the degraded image. Because of the impact of adherent mist and raindrops, the trees, the building, and some bicycles can not be identified by the panoptic segmentation algorithm. The segmentation result with the output from our method shows relatively distinguishable contours for the bicycles, the pavement, the trees, and the building.


\section{Conclusion}
In this work, we propose an image degradation problem caused by the co-existence of mist and raindrops adhered to a glass window or camera lens. This phenomenon shows different image characteristics compared to atmospheric haze or rain streaks. The problem is widely observed in reality but lacks investigation. Instead of using hand-crafted priors to generate spatial attention maps to handle this problem, we introduce CAM attention to generate attention maps without priors, and utilize dual attention modules as well as smoothed dilated convolution modules in the restoration CNN to generate relatively clean images. Both quantitative evaluation and intuitive examples show quality improvement with the proposed method. The restoration with our method can also benefit high-level tasks such as instance segmentation and panoptic segmentation for generating reliable results, which demonstrates a vital significance in vision applications.


\bibliographystyle{splncs04}
