\section{Introduction}
Images captured under complicated rain scenarios inevitably suffer from the noticeable degradation of visual quality. The degradation causes detrimental impacts on many computer vision tasks such as image segmentation \cite{liu2020deep}, object detection \cite{wojna2019devil} and video surveillance \cite{sultani2018real}. Thus, it is indispensable to develop effective algorithms to recover high-quality rain-free images, which is referred to as \textit{image deraining}.

Image deraining is an ill-posed problem. 
To make it well-posed, traditional methods usually exploit various image priors, such as sparse coding \cite{luo2015removing}, Gaussian mixture model \cite{li2016rain} and low-rank representation \cite{chang2017transformed}. However, these hand-crafted priors possess a limited representation ability,  leading to poor deraining results under complicated and varied rainy scenes.
By learning from massive synthetic clean/rainy image pairs, the performance of deep learning based techniques is substantially improved \cite{yang2017deep,fu2021rain}. 
However, these supervised methods usually achieve sub-optimal performance on real-world rainy images, due to the fact that i) there exists the domain gap between synthetic and real-world rainy images, and ii) collection of real-world and large-scale clean/rainy image pairs is usually not possible. 

To alleviate the aforementioned problems, semi-supervised deraining techniques, as promising solutions, 
leverage paired synthetic data for good initialization and unpaired real-world data for generalization \cite{wei2019semi,ye2021closing,yasarla2020syn2real}. Unfortunately, their performance still depends on the domain gap between the synthetic and real-world rainy images. Once the gap is much larger, the deraining results would be most unsatisfactory. 
Further, CycleGAN-based unsupervised methods are proposed for real-world image deraining, avoiding paired training data \cite{wei2021semi,han2020decomposed,jin2019unsupervised,zhu2019singe,chen2022unpaired}. But these methods are known as being difficult to train, due to their complex adversarial objectives. Thus, they easily suffer from a series of problems of premature convergence, model collapse, and robust overfitting, leading to image degradation including the loss of image details, remnant rain, halo artifacts, and/or color distortion. 


Recently, various diffusion models \cite{sohl2015deep,ho2020denoising,song2019generative} have gained wide interest in computer vision. Compared to GANs, diffusion models provide a stable training process and are more powerful for modeling image pixel distribution.
However, 
standard diffusion models usually focus on synthetic degradation, such as image inpainting, image colorization, and super-resolution, in which it is easy to simulate large-scale paired synthetic data rather than real-world ones. 
No work to-date exploits diffusion models for real-world image deraining.

In this paper, we propose a novel unsupervised learning paradigm based on denoising diffusion probabilistic models (DDPM), called RainDiffusion, to tackle the unfavorable prevailing problem of real-world image deraining. 
To enable unsupervised training for unpaired real-world data, RainDiffusion includes two main interactive branches:
Non-diffusive Translation Branch (NTB) and Diffusive Translation Branch (DTB).
To bypass the difficulty in unpaired training of standard diffusion models, NTB, only used in the model training phase, makes the best use of cycle-consistent architecture to generate initial clean/rainy image pairs for conditional diffusion model training. 
Given initial image pairs, DTB designs two conditional diffusion modules for high-quality real-world image deraining and rain generation. 
Instead of popular unsupervised adversarial networks, RainDiffusion exploits the powerful generative ability of DDPMs to mitigate this limitation of unstable training and significantly improve the quality of final derained images (see Fig. \ref{ab_reals}). 
Experiments on several well-known benchmark datasets show that RainDiffusion outperforms existing un/semi-supervised methods and achieves comparable performance against fully-supervised ones. 
The main contributions are as follows:
\begin{itemize}
\item \textit{First Diffusion-based Model for Real-world Image Deraining.} We propose a novel unsupervised image deraining method based on denoising diffusion probabilistic models, called RainDiffusion. RainDiffusion is a non-adversarial training paradigm, serving as a new standard bar for real-world image deraining.
 

\item \textit{Stable Non-GAN-based Unsupervised Training for Unpaired Real-world Data.} 
RainDiffusion consists of two main interactive branches: Non-diffusive Translation Branch (NTB) and Diffusive Translation Branch (DTB). Through these two branches, RainDiffusion can remove the dependency on paired training data, and avoid training unstable adversarial objectives.

\item \textit{State-of-the-art Image Deraining Performance.} Extensive experiments show the superiority of RainDiffusion in real-world image deraining. Moreover, our approach performs favorably against existing semi-supervised, unsupervised, and even supervised methods on synthetic datasets.
\end{itemize}

\begin{figure*}[!t] \centering
	\includegraphics[width=1\linewidth]{pipline2.pdf}
	\caption{The pipeline of RainDiffusion. RainDiffusion consists of two main interactive branches: Non-diffusive Translation Branch (NTB) and Diffusive Translation Branch (DTB).
	In detail, NTB, only used in the model training phase, fully exploits cycle-consistent architecture to generate initial clean/rainy image pairs. Note that $G_{\phi}^{A}$ and $G_{\phi}^{B}$ are employed to produce the rainy and rain-free images, respectively. $x$, $y$, $x'$, $y'$, $x''$ and $y''$ refer to the 
	given rain-free image, given rainy image, generated rainy image, generated rain-free image, reconstructed rain-free image, and reconstructed rainy image.
	Given initial image pairs, DTB leverages two conditional diffusion modules for translation between clean and rainy images. Especially, the deraining diffusion module learns a conditional reverse process $p_{\theta}^{A}(x_{0:T}|\Tilde{x})$ without modifying the diffusion process $q(x_{1:T}|x_{0})$ for $x$. The diffusion module for rain generation is similar to it, except it works oppositely.
    }
	\label{fig:network framework}
\end{figure*}
\section{Related Work}
\textbf{Single Image Deraining.} Early single image deraining methods employ several hand-crafted priors to restore the rainy images. For example, in \cite{luo2015removing,kang2011automatic}, it is assumed that rain streaks are sparse and of high frequency in the image, and thus the problem is translated into progressive image decomposition via dictionary learning. Li et al.~\cite{li2016rain} adopt the Gaussian Mixture Model (GMM) to accommodate multiple orientations and scales of rain streaks. In \cite{chen2013generalized,chang2017transformed}, the low-rank representations are incorporated into deraining. In \cite{gu2017joint} and \cite{wang2017hierarchical}, guided filter and analysis sparse representation are integrated into sparse coding for finer layer decomposition.

Recently, deep learning-based methods have been substantiated to be effective in image deraining \cite{fu2017clearing}. Fu et al.~\cite{fu2017clearing} first introduce an end-to-end residual CNN to simplify the learning processing. Network modules, such as dense block \cite{li2018non,wang2019erl}, recursive block \cite{fan2018residual,li2018recurrent} and dilated convolution \cite{deng2020detail}, and structures, such as RNN \cite{li2018recurrent,ren2019progressive}, GAN \cite{zhang2019image,chen2022unpaired} and multi-stream networks \cite{yang2017deep,zhang2018density}, are validated to be effective in image deraining.
Although we have witnessed promising deraining results on synthetic datasets, these learning-based efforts trained on synthetic images generalize poorly to real-world images, typically because of the obvious domain gap.

To bridge the domain gap between synthetic and real-world rainy images, Some semi-supervised frameworks have been proposed \cite{wei2019semi,ye2021closing,yasarla2020syn2real}. For example, Wei et al.~\cite{wei2019semi} first propose a semi-supervised rain removal framework where they adopt a likelihood term imposed on Gaussian Mixture Model (GMM) and minimize the Kullback-Leibler (KL) divergence between synthetic and real rain. Yasarla et al. \cite{yasarla2020syn2real} further propose a Gaussian Process-based semi-supervised learning framework to obtain better generalization performance. However, these supervised/semi-supervised methods still require paired data. Such paired data is very difficult or even impossible to collect under real-world rainy scenes. 
Motivated by a popular image-to-image translation architecture, i.e., CycleGAN \cite{zhu2017unpaired}, recent works \cite{zhu2019singe,jin2019unsupervised,wei2021deraincyclegan,chen2022unpaired} attempt to exploit the improved CycleGAN architecture and constrained transfer learning to jointly the
rainy and rain-free image domains. However, the aforementioned methods all rely on their complex adversarial objective to develop their algorithms, which leads to the difficulty of stable training. Thus, they are also not capable of obtaining promising real-world image deraining results.

\textbf{Denoising Diffusion Probabilistic Models.} Recently, denoising diffusion probabilistic models (DDPM) \cite{sohl2015deep,ho2020denoising,song2019generative} have exhibited their powerful ability in many vision tasks such as image super-resolution \cite{saharia2022image}, text-to-image generation \cite{gu2022vector}, and image segmentation \cite{amit2021segdiff}. Different from other GANs, diffusion models have a stable training process and provide more diversity because they are likelihood-based.
However, regular diffusion models still require paired data for training, and are so far not considered for real-world image deraining. This motivates us to propose the first diffusion-based model for real-world image deraining.

\section{Proposed Method}
Fig. \ref{fig:network framework} presents the overall architecture of RainDiffusion. To the best of our knowledge, the proposed RainDiffusion is the first diffusion-based model for real-world image deraining. To enable unsupervised training for unpaired real-world data, it consists of two main interactive branches: Non-diffusive Translation Branch (NTB) and Diffusive Translation Branch (DTB).
Especially, NTB is used to guide the generation of initial clean/rainy image pairs. 
Then, DTB inputs image pairs generated from NTB into two conditional diffusion modules for translation between clean and rainy images. 
In the following, we first briefly introduce the denoising diffusion probabilistic models. After that, we elaborate on our NTB and DTB. 



\subsection{Denoising Diffusion Probabilistic Models}
Denoising diffusion probabilistic models (DDPM) slowly corrupt the training data with Gaussian noise and learn to reverse the corruption as a generative model \cite{sohl2015deep,ho2020denoising,song2019generative}. In the forward process, Gaussian noise is added sequentially onto an input image $x_{0} \sim q(x_{0})$ over $T$ time steps according to the following Markovian process:
\begin{equation}
q(x_{t}|x_{t-1})= \mathcal{N}(x_{t};\sqrt{1-\beta_{t}}x_{t-1},\beta_{t}I), \forall t \in \{1,...,T\}
\end{equation}
\begin{equation}
q(x_{1:T}|x_{0})= \prod_{t=1}^{T} q(x_{t}|x_{t-1})
\end{equation}
where $\beta_{t}$ is noise variance, $\mathcal{N}(.)$ is a Gaussian distribution, $I$ is an identity covariance matrix with the same dimensions as the input image $x_{0}$. An important property of this recursive formulation is that it also allows the direct sampling of $x_{t}$, when $t$ is drawn from a uniform distribution,
i.e., $\forall t \sim \mathcal{U}(\{1,...,T\})$:
\begin{equation}
\begin{aligned}
q(x_{t}|x_{0})&= \mathcal{N}(x_{t};\sqrt{\overline{\alpha}_{t}}x_{0},(1-\overline{\alpha}_{t})I) \Longrightarrow \\
x_{t} &= \sqrt{\overline{\alpha}_{t}}x_{0}+\epsilon_{t} \sqrt{1-\overline{\alpha}_{t}}
\end{aligned}
\end{equation}
where $\epsilon_{t} \sim \mathcal{N}(0,I)$, $\alpha_{t}=1-\beta_{t}$ and $\overline{\alpha}_{t}=\prod_{i=1}^{t}\alpha_{i}$. If we have the original image $x_{0}$ and fix a variance schedule $\beta_{t}$, we can sample any noisy
version $x_{t}$ via a single step. Similarly, reverse diffusion also adopts a Markov chain from $x_{T}$ onto $x_{0}$, albeit each step aims to gradually denoise the samples. Under small $\beta_{t}$ and large $T$, the reverse transition probability between $x_{t}$ and
$x_{t-1}$ can be approximated as a Gaussian distribution:
\begin{equation}
\label{reverse}
p_{\theta}(x_{t-1}|x_{t})= \mathcal{N}(x_{t-1};\mu_{\theta}(x_{t},t),\sigma_{\theta}(x_{t},t))
\end{equation}
\begin{equation}
\label{reverse_0:T}
p_{\theta}(x_{0:T})= p(x_{T}) \prod_{t=1}^{T} p_{\theta}(x_{t-1}|x_{t})
\end{equation}
where the reverse process is parameterized by a neural network that estimates $\mu_{\theta}(x_{t},t)$ and $\sigma_{\theta}(x_{t},t)$. Especially, $\beta_{t}$ can be learned jointly with the model or just kept constant to ensure that $x_{t}$ approximately follows a standard normal distribution. 

The model then can be trained by optimizing a variational bound on negative data log likelihood $\mathbb{E}_{q(x_{0})}[-\log p_{\theta}(x_{0})] \leq L_{\theta}$, which can be  performed by:
\begin{equation}
\begin{aligned}
L_{\theta}=\mathbb{E}_{q}[\begin{matrix}\underbrace{D_{KL}(q(x_{T}|x_{0})||p(x_{T}))}\\ L_{T}\end{matrix}\, \begin{matrix}\underbrace{-\log p_{\theta}(x_{0}|x_{1})}\\ L_{0}\end{matrix}\\
+\begin{matrix}\sum_{t>1}\underbrace{ D_{KL}(q(x_{t-1}|x_{t}),x_{0})||p_{\theta}(x_{t-1}|x_{t}))}\\ L_{t-1}\end{matrix}]
\end{aligned}
\end{equation}
where $D_{KL}$ denotes the Kullback-Leibler divergence between two probability distributions. A common parametrization focuses on $\mu_{\theta}(x_{t},t)$ while ignoring $\sigma_{\theta}(x_{t},t)$ \cite{ho2020denoising}, as follows:
\begin{equation}
\label{epsilon}
\mu_{\theta}(x_{t},t)= \frac{1}{\sqrt{\alpha_{t}}}(x_{t}-\frac{\beta_{t}}{\sqrt{1-\overline{\alpha}_{t}}}\epsilon_{\theta}(x_{t},t))
\end{equation}
In this case, the network can be used to estimate the added noise $\epsilon_{t}$ by minimizing the following loss:
\begin{equation}
\label{noise_loss}
L_{err}= \mathbb{E}_{x_{0},t,\epsilon_{t}\sim \mathcal{N}(0,I)}[||\epsilon_{t}- \epsilon_{\theta}(\sqrt{\overline{\alpha}_{t}}x_{0}+\epsilon_{t} \sqrt{1-\overline{\alpha}_{t}},t)||^{2}]
\end{equation}
During inference, reverse diffusion steps are performed starting from a random sample $x_{T} \sim \mathcal{N}(0,I)$. For each step $t \in \{T,...,1\}$, $\mu$ is derived using Eq. \ref{epsilon} based on the estimated $\epsilon_{\theta}$, and $x_{t-1}$ is sampled based on Eq. \ref{reverse}, which can be expressed in closed form:
\begin{equation}
\label{noise_loss2}
x_{t-1}=\frac{1}{\sqrt{\alpha_{t}}}(x_{t}-\frac{\beta_{t}}{\sqrt{1-\overline{\alpha}_{t}}}\epsilon_{\theta}(x_{t},t))+\sigma_{t}z
\end{equation}
where $z \sim \mathcal{N}(0,I)$), which resembles one step of sampling via Langevin dynamics \cite{welling2011bayesian}.

To enable high-quality image deraining, we should learn a conditional reverse process $p_{\theta}(x_{0:T}|\Tilde{x})$ without modifying the diffusion process $q(x_{1:T}|x_{0})$ for $x$, where $x_{0}$ and $\Tilde{x}$ represent a clean image and a rainy image. During the training phase, we sample $(x_{0},\Tilde{x}) \sim q(x_{0},\Tilde{x})$ from a paired data distribution and learn a conditional diffusion model. We input $\Tilde{x}$ to the reverse process as follows:
\begin{equation}
\label{reverse_rain}
p_{\theta}(x_{0:T}|\Tilde{x})= p(x_{T}) \prod_{t=1}^{T} p_{\theta}(x_{t-1}|x_{t},\Tilde{x})
\end{equation}
The noise estimator network in Eq. \ref{noise_loss}, \ref{noise_loss2} is also replaced by $\epsilon_{\theta}(x_{t},\Tilde{x},t)$.

\subsection{Non-diffusive Translation Branch (NTB)}
Although conditional diffusion models have shown state-of-the-art image-conditional data synthesis and restoration capabilities, the implementation of the diffusion models in Eq. \ref{reverse_rain} still requires paired clean/rainy images. We will tackle this problem by designing a new Non-diffusive Translation Branch (NTB) without requiring adversarial training. As shown in Fig. \ref{fig:network framework}, NTB introduces cycle-consistent architecture to generate initial clean/rainy image pairs for conditional diffusion model training. Since the training rainy image $y$ and clean image $x$ are unpaired, NTB employs two non-diffusive generators with parameters $\phi^{A,B}$ to obtain initial translation estimates:
\begin{equation}\centering
\begin{split}
x' = G_{\phi}^{A}(x)\qquad x'' = G_{\phi}^{B}(x')\\
y' = G_{\phi}^{B}(y)\qquad y'' = G_{\phi}^{A}(y')
\end{split}
\end{equation}
where $x$, $y$, $x'$, $y'$, $x''$ and $y''$ refer to the given rain-free image, given rainy image, generated rainy image, generated rain-free image, reconstructed rain-free image, and reconstructed rainy image. Notably, we adopt U-Net \cite{ronneberger2015u} as our non-diffusive generators. Then the cycle-consistency loss can be used to limit the space of generated samples and retain the contents of generated images:
\begin{equation}\label{eq:cyc}
\begin{aligned}
L_{cyc} = E_{x\sim P_{data(x)}}[\left \| x''-x \right \|_{1}]+E_{y\sim P_{data(y)}}[\left \| y''-y \right \|_{1}]
\end{aligned}
\end{equation}


\textbf{The difference between NTB and other circulatory structures.} In unsupervised learning, the circulatory structures with cycle-consistency loss are commonly used for model training \cite{chen2022unpaired,wei2021deraincyclegan,zhu2017unpaired}. The differences between NTB and others lie in the following two aspects: 1) NTB does not require any discriminators for adversarial training. This means that our NTB provides a more stable training process. 2) The generators of NTB are designed to generate paired clean/rainy images for diffusion model training. Thus, both non-diffusive generators $G_{\phi}^{A}$ and $G_{\phi}^{B}$ are only used in the training phase and do not involve the testing phase.

\begin{figure}[!t]
	\centering
	\includegraphics[width=1.0\linewidth]{derain_raing.pdf}
    \caption{Training RainDiffusion on unpaired data. From (a) to (d) are the input real-world rainy images, real-world clean images, generated derained images and generated rainy images, respectively.}
	\label{ab_real1}
\end{figure}

\begin{algorithm}[!t]
    \caption{RainDiffusion training}
    \label{alg:training}
    \begin{algorithmic}[1]
    \REQUIRE 
    Unpaired clean image $x$ and rainy image $y$
       
        \REPEAT
        \STATE Randomly sample a binary patch mask $P_{i}$\\
        \STATE $x_{i}= Crop(P_{i}\circ x), y_{i}= Crop(P_{i}\circ y)$
        \STATE $x'_{i} = Crop(P_{i}\circ G_{\phi}^{A}(x)), x''_{i} = Crop(P_{i}\circ G_{\phi}^{B}(x'))$\\
        \STATE $y'_{i} = Crop(P_{i}\circ G_{\phi}^{B}(y)), y''_{i} = Crop(P_{i}\circ G_{\phi}^{A}(y'))$\\
        \STATE $t^{A}, t^{B} \sim$ Uniform $(\{ 1,...,T \})$\\
        \STATE $\epsilon^{A},\epsilon^{B} \sim \mathcal{N}(0,I)$\\
        \STATE Take gradient descent step on\\
        \STATE $\nabla_{\theta,\phi}[||\epsilon^{A}- \epsilon_{\theta}^{A}(\sqrt{\hat{\alpha}_{t^{A}}}x_{i}+\epsilon_{t^{A}} \sqrt{1-{\hat{\alpha}_{t^{A}}}},x'_{i},t^{A})||^{2}$\\
        \STATE $+||\epsilon^{B}- \epsilon_{\theta}^{B}(\sqrt{\hat{\alpha}_{t^{B}}}x'_{i}+\epsilon_{t^{B}} \sqrt{1-{\hat{\alpha}_{t^{B}}}},x''_{i},t^{B})||^{2}$\\
        \STATE $+||\epsilon^{B}- \epsilon_{\theta}^{B}(\sqrt{\hat{\alpha}_{t^{B}}}y_{i}+\epsilon_{t^{B}} \sqrt{1-{\hat{\alpha}_{t^{B}}}},y'_{i},t^{B})||^{2}$\\
        \STATE $+||\epsilon^{A}- \epsilon_{\theta}^{A}(\sqrt{\hat{\alpha}_{t^{A}}}y'_{i}+\epsilon_{t^{A}} \sqrt{1-{\hat{\alpha}_{t^{A}}}},y''_{i},t^{A})||^{2}$\\
        \STATE $+\lambda_{cyc}L_{cyc}]$
        \UNTIL{converged}\\
    \STATE \textbf{return} $\theta^{A},\theta^{B}$(optional)
    \end{algorithmic}
\end{algorithm}
\subsection{Diffusive Translation Branch (DTB)}
Following the advantage of denoising diffusion probabilistic models in the area of generative modeling, DTB also designs two conditional diffusion modules, to mitigate the limitation of unstable training and improve the generalization ability for real-world image deraining and rain generation (as shown in Fig. \ref{ab_real1}). 

As shown in Fig. \ref{fig:network framework}, the deraining diffusion module learns a conditional reverse process $p_{\theta}^{A}(x_{0:T}|\Tilde{x})$ without modifying the diffusion process $q(x_{1:T}|x_{0})$ for $x$. 
Since real-world rainy benchmarks usually consist of images with various sizes, we adopt the path-based DDPM \cite{ozdenizci2022restoring} as our backbone for size-agnostic image deraining. Notably, we also try other conditional diffusion models as our backbone, elaborated in Sec. \ref{Ablation_Study}. 
Especially, we rewrite the conditional reverse process $p_{\theta}^{A}(x_{0:T}|\Tilde{x})$ as follows:
\begin{equation}
\label{crop_rain}
p_{\theta}(x_{0:T}^{i}|\Tilde{x}^{i})= p(x_{T}^{i}) \prod_{t=1}^{T} p_{\theta}(x_{t-1}^{i}|x_{t}^{i},|\Tilde{x}^{i})
\end{equation}
where $x_{0}^{i}= Crop(P_{i}\circ x_{0})$ and $\Tilde{x}^{i}= Crop(P_{i}\circ \Tilde{x})$ denote $p \times p$ patches from a training set image pair $(x_{0},\Tilde{x})$, and the $Crop(.)$ operation extracts the patch from the location indicated by $P_{i}$. During the training phase, we randomly sample (with uniform probability) the $p \times p$ patch location for $P_{i}$ within the complete range of image dimensions. Note that we use $p = 128$ pixel for $P_{i}$. The pipeline of the diffusion module for rain generation is similar to it, except it works oppositely. The training phase of our RainDiffusion is outlined in Algorithm \ref{alg:training}.

\begin{algorithm}[!t]
    \caption{RainDiffusion testing (image deraining)}
    \label{alg:test}
    \begin{algorithmic}[1]
    \REQUIRE 
    Rainy image $\Tilde{x}$, conditional diffusion model $\epsilon_{\theta}^{A}(x_{t},\Tilde{x},t)$, number of implicit sampling steps $S$, dictionary of $D$ overlapping patch locations.
    \STATE $x_{t} \sim \mathcal{N}(0,I)$
    \FOR{each $i= S,...,1$}
        \STATE $t = (i-1)\cdot T/S+1$
        \STATE $t_{next} = (i-2)\cdot T/S+1$ \textbf{if} $i>1$ \textbf{else} 0
        \STATE $\widehat{\Omega}_{t} = 0, M=0$
        \FOR{each $d= 1,...,D$}
        \STATE $x_{t}^{d} = Crop(P_{d}\circ x_{t}), \Tilde{x}^{d}= Crop(P_{d}\circ \Tilde{x})$
        \STATE $\widehat{\Omega}_{t}=\widehat{\Omega}_{t}+P_{d}\cdot \epsilon_{\theta}(x_{t}^{d},\Tilde{x}^{d},t)$ 
        \STATE $M=M+P_{d}$
        \ENDFOR
        \STATE $\widehat{\Omega}_{t} = \widehat{\Omega}_{t}\oslash M$
        \STATE $x_{t} \Longleftarrow \sqrt{\overline{\alpha}_{t_{next}}}(\frac{x_{t}-\sqrt{1-\overline{\alpha}_{t}}\cdot\widehat{\Omega}_{t}}{\sqrt{\overline{\alpha}_{t}}})+\sqrt{1-\overline{\alpha}_{t_{next}}}\cdot \widehat{\Omega}_{t}$
    \ENDFOR
    \STATE\textbf{return} $x_{t}$
    \end{algorithmic}
\end{algorithm}
\begin{figure}[!t]
	\centering
	\includegraphics[width=1.0\linewidth]{testing.pdf}
    \caption{Illustration of testing phase of RainDiffusion detailed in Algorithm \ref{alg:test}. }
	\label{fig_test}
\end{figure}

\begin{table*}[!t]\footnotesize
\centering
\footnotesize
\caption{Comparisons on metrics (PSNR/SSIM) with the different types of deraining methods. \textcolor{red}{Red} and \textcolor{blue}{blue} colors are used to indicate the $1^{st}$ and $2^{nd}$ ranks, respectively. }
\label{synthetic_result}
\begin{tabular}{c|c|c|c|c|c|c|c}
\toprule
\multicolumn{2}{c|}{Datasets}                      & \multicolumn{2}{c|}{Rain200L} & \multicolumn{2}{c|}{Rain800} & \multicolumn{2}{c}{SPA-Data} \\
\midrule
\multicolumn{2}{c|}{Metrics}                       & PSNR          & SSIM         & PSNR         & SSIM         & PSNR          & SSIM         \\\midrule
\multirow{2}{*}{Prior-based}     & DSC \cite{luo2015removing}& 27.34         & 0.849        & 18.56        & 0.600        & 34.95         & 0.942        \\
                                 &GMM \cite{li2016rain}& 29.05         & 0.872        & 20.46        & 0.730        & 34.30         & 0.943        \\\midrule
\multirow{5}{*}{Supervised}      &DDN \cite{detail_layer}& 32.38         & 0.926        & 21.16        & 0.732        & 34.70         & 0.934        \\
                                 &DID-MDN \cite{zhang2018density}& \textcolor{blue}{35.40}         & 0.961        & 21.89        & 0.795        & 34.68         & 0.930        \\
                                 &SPA-Net \cite{wang2019spatial}& 31.59         & 0.919        & 24.37        & \textcolor{blue}{0.861}        & 35.13         & 0.944        \\
                                 &MSPFN \cite{jiang2020multi}& 32.40         & 0.933        & 25.52        & 0.830        & 35.21         & \textcolor{red}{0.951}        \\
                                 &AirNet \cite{li2022all}& 34.90 & \textcolor{blue}{0.969}         & 23.77& 0.833 & 33.45 & 0.924        \\
                                 \midrule
\multirow{3}{*}{Semi-supervised} &SIRR \cite{wei2019semi}& 34.47         & \textcolor{blue}{0.969 }       & 24.36        & 0.859        & 34.85         & 0.936        \\
                                 &Syn2Real \cite{yasarla2020syn2real}& 34.39         & 0.965        & 23.74        & 0.799        & 33.66         & 0.938        \\
                                 &JRGR \cite{ye2021closing}& 26.54         & 0.872        & 22.16        & 0.822        & 31.12         & 0.902        \\\midrule
\multirow{6}{*}{Unsupervised}    &CycleGAN \cite{zhu2017unpaired}& 24.61         & 0.834        & 23.95        & 0.819        & 22.40         & 0.860        \\
                                 &RR-GAN \cite{zhu2019singe}& /             & /            & 23.51        & 0.757        & /             & /            \\
                                 &DerainCycleGAN \cite{wei2021deraincyclegan}& 31.49         & 0.936        & 24.32        & 0.842        & 34.12         & 0.950        \\
                                 &DCD-GAN \cite{chen2022unpaired}& /             & /            & \textcolor{blue}{25.61}        & 0.813        & \textcolor{blue}{35.30}         & 0.943        \\
                                 &NLCL \cite{ye2022unsupervised}             &17.06             &0.653  &15.91            & 0.540             & 33.82         & 0.946        \\
                                 & \textbf{Ours}  & \textcolor{red}{37.02}              &\textcolor{red}{0.979}             &  \textcolor{red}{26.24}            &   \textcolor{red}{0.862}            & \textcolor{red}{36.96}              &\textcolor{blue}{0.948} \\
                                 \bottomrule
\end{tabular}
\end{table*}
The testing phase of RainDiffusion is illustrated in Fig. \ref{fig_test} and outlined in Algorithm \ref{alg:test}. We observe that a large $T$ inevitably leads to costly sampling, e.g., when $T = 1000$. To address this problem, we use an implicit sampling strategy \cite{song2020denoising} to accelerate our sampling process (lines 3-4 in Alg. \ref{alg:test}). Implicit sampling using a noise estimator network can be performed by
\begin{equation}
\label{implicit}
\begin{split}
x_{t-1}=&\sqrt{\overline{\alpha}_{t-1}}(\frac{x_{t}-\sqrt{1-\overline{\alpha}_{t}}\cdot \epsilon_{\theta}(x_{t},\Tilde{x},t)}{\sqrt{\overline{\alpha}_{t}}})\\
&+\sqrt{1-\overline{\alpha}_{t-1}}\cdot \epsilon_{\theta}(x_{t},\Tilde{x},t)
\end{split}
\end{equation}
during accelerated sampling we only needs a subsequence $\tau_{1},...,\tau_{S}$ of the complete ${1,...,T}$ timestep indices, which can be performed by
\begin{equation}
\label{implicit2}
\begin{split}
\tau_{i}=(i-1) \cdot T / S+1
\end{split}
\end{equation}
Given any denoising time step $t$, we first extract all overlapping $p \times p$ patches from a grid-like arranged parsing scheme to decompose the image $\Tilde{x}$ of arbitrary size, where $D$ represents the total number of extracted patches, defining a dictionary of overlapping patch locations (line 7 in Alg. \ref{alg:test}). 
Then, we use $\epsilon_{\theta}(x_{t}^{d},\Tilde{x}^{d},t)$ to estimate the additive noise for all overlapping patch locations $d \in {1,...,D}$ and accumulate these overlapping noise estimates at their respective patch locations in a matrix $\widehat{\Omega}_{t}$ of same size as the whole image (line 8 in Alg. \ref{alg:test}).
Finally, we normalize $\widehat{\Omega}_{t}$ based on the number of received estimates for each pixel (line 11 in Alg. \ref{alg:test}), and perform an implicit sampling update using the smoothed whole-image noise estimate $\widehat{\Omega}_{t}$ (line 12 in Alg. \ref{alg:test}).
























\section{Experiment}
We implement RainDiffusion using Pytorch 1.6 on a system with 11th Gen Intel(R) Core(TM) i7-11700F CPU and Nvidia GeForce RTX 3090 GPU. For optimizing RainDiffusion, we use the Adam optimizer with a min-batch size of 4 to train the paradigm, where the momentum parameters $ \beta_{1}$ and $\beta_{2}$ take the values of 0.5 and 0.999, respectively. Besides, the initial learning rate for both generators and discriminators is set to 0.00002. In the experiment, we set $\lambda_{cyc}$ to $1.0$. For training, a 128 $\times$128 image patch is randomly cropped from the original image (or its horizontal flipped version). During inference, only the noise estimator network responsible for the desired image deraining was used.
\subsection{Evaluation Settings and Datasets}
We qualitatively and quantitatively compare the performance of RainDiffusion with different types of deraining approaches, including supervised, semi-supervised and unsupervised methods. Supervised methods include DSC \cite{luo2015removing}, GMM \cite{li2016rain}, DDN \cite{detail_layer}, DID-MDN \cite{zhang2018density}, SPA-Net \cite{wang2019spatial}, MSPFN \cite{jiang2020multi} and AirNet \cite{li2022all}, while semi-supervised methods include SIRR \cite{wei2019semi}, Syn2Real \cite{yasarla2020syn2real}, and JRGR \cite{ye2021closing}. For unsupervised methods, CycleGAN \cite{zhu2017unpaired}, RR-GAN \cite{zhu2019singe}, DerainCycleGAN \cite{wei2021deraincyclegan}, DCD-GAN \cite{chen2022unpaired} and NLCL \cite{ye2022unsupervised} are adopted. As the code of the
methods \cite{zhu2019singe,wei2021deraincyclegan,chen2022unpaired} is not available, we refer to some comparison results presented in their paper. For other methods, we retrain them with the codes provided by authors.

As RainDiffusion is trained in an unsupervised learning manner, real-world clean and rainy images are utilized for training our model. For evaluation, both synthetic and real datasets are employed. The synthetic datasets are: 1) Rain200L \cite{yang2017deep} containing 1800 synthetic images for training and 200 images for testing; 2) Rain800 \cite{zhang2019image} containing 700 synthetic images for training and 192 images (100 synthetic images and 92 real-world images) for testing. The real datasets are: 1) SIRR-Data \cite{wei2019semi} containing 147 images without ground truths; 2) Google search with ``real-world rainy images" containing 300 images without ground truths; 3) SPA-Data \cite{wang2019spatial} containing 29500 high-quality rainy/clean image pairs (28500 images for training and 1000 images for testing). Finally, we adopt the average Peak Signal to Noise Ratio (PSNR) and Structural Similarity Index (SSIM) metrics for quantitative evaluation of the deraining images.

\begin{figure*}[!t]
	\centering
	\includegraphics[width=1\linewidth]{real_result.pdf}
    \caption{Comparison of deraining performance on real-world rainy images. It is clearly shown that most deraining methods fail to remove rain streaks in heavy rain. Comparatively, our method is more successful to remove rain accumulation and obtain the cleanest result with very clear details.}
	\label{morecompare}
\end{figure*}
\subsection{Comparison with State-of-the-arts}
\textbf{Comparison on Synthetic Datasets.} 
Table \ref{synthetic_result} reports the quantitative results of 15 baseline models on synthetic datasets including Rain200L and Rain800. We have the following observations: 1) compared with two common types of unsupervised methods, i.e, CycleGAN-based \cite{zhu2017unpaired,zhu2019singe,wei2021deraincyclegan} and constractive learning \cite{ye2022unsupervised}, our method obtains higher values of PSNR and SSIM, which verifies the robustness of the proposed RainDiffusion. 2) There is an obvious gap between semi-supervised and supervised methods, even worse than unsupervised ones. 3) Our method can achieve competitive results to several existing supervised ones. More qualitative results on synthetic datasets are provided in the supplementary material.

\textbf{Comparison on Real-world Datasets.} We further compare our method with 15 baseline models on the SPA-Data dataset, which contains real-world rainy images with corresponding ground truth. As shown in Table~\ref{synthetic_result}, our method performs better than all semi-supervised and unsupervised methods and shows competitive advantages to the fully-supervised methods. Further, Fig. \ref{morecompare} illustrates the deraining performance of compared approaches on some real-world rainy images without ground truth, which are more complex and challenging. RainDiffusion successfully handles various rain streaks with different patterns, and achieves better visual results than all compared methods. Especially, the derained results of the unsupervised methods will cause color and structure distortion, while our method can better preserve the color and structure of the image. More qualitative results on real-world rainy images are provided in the supplementary material.

\begin{table}[!t]
\centering
\footnotesize
\caption{Ablation study for different loss functions. $L_{err}(x,x')$ indicates the noise estimator network in Eq. \ref{noise_loss} is replaced by $\epsilon_{\theta}(x_{t},x',t)$ (line 8 in Alg. \ref{alg:training}).}
\label{loss_ab}
\setlength{\tabcolsep}{1.5mm}{
\begin{tabular}{cccc}
\toprule
Model       & w/o $L_{err}(x,x')$& w/o $L_{err}(x',x'')$& w/o $L_{cyc}$\\
PSNR / SSIM &24.80/0.817     &25.36/0.842     &20.07/0.708      \\\midrule
Model       & w/o $L_{err}(y,y')$& w/o $L_{err}(y',y'')$& Ours \\
PSNR / SSIM &25.09/0.824     &25.78/0.851     & \textbf{26.24/0.872}   \\
\bottomrule
\end{tabular}
}
\end{table}

\begin{figure}[!t] \centering
	\includegraphics[width=0.8\linewidth]{loss_cyc.pdf}
	\caption{Effectiveness of the balanced weight $\lambda_{cyc}$.}
	\label{fig:weight_cyc}
\end{figure}

\begin{table}[!t]
\centering
\footnotesize
\caption{Quantitative comparisons with different diffusion-based models on the Rain800 dataset.}
\label{ab_ddpm}
\begin{tabular}{cccc}
\toprule
Method       & SR3&DDPM & WeatherDiffusion \\
PSNR / SSIM &26.12/0.866     &26.01/0.863     &\textbf{26.24/0.872}      \\
\bottomrule
\end{tabular}
\end{table}
\subsection{Ablation Study}
\label{Ablation_Study}

\textbf{The Effect of Loss Function.} We evaluate the effectiveness of our hybrid loss function on the Rain800 dataset. Especially, we remove one component to each configuration at one time. For fair comparision, the same training settings are kept for all models testing. As shown in Table \ref{loss_ab}, the full structure of RainDiffusion achieves the highest performance in terms of PSNR and SSIM, which indicates that all the components of RainDiffusion are beneficial for effective deraining. Moreover, we also evaluate the effectiveness of the balanced weight $\lambda_{cyc}$. As shown in Fig. \ref{fig:weight_cyc}, the balanced weight $\lambda_{cyc}$ is chosen experimentally, which optimizes our network effectively.

\textbf{The Effect of Different Diffusion Models.} 
We also the effects of different diffusion models used in DTB, including improved DDPMs (WeatherDiffusion \cite{ozdenizci2022restoring} and SR3 \cite{saharia2022image}), and the standard DDPM \cite{ho2020denoising}. The above DDPMs are conditional diffusion models, which require paired data for training. As shown in Table \ref{ab_ddpm}, benefit from RainDiffusion, they both can achieve satisfactory performance on unsupervised image deraining.









\section{Conclusion}
In this paper, we do not plan to bridge the gap between the synthetic/real rainy images and depend on any adversarial objectives. We propose a new unsupervised learning paradigm based on denoising diffusion probabilistic models, dubbed RainDiffusion, to tackle the unfavorable prevailing problem of real-world image deraining. RainDiffusion consists of two main interactive branches: Non-diffusive Translation Branch (NTB) and Diffusive Translation Branch (DTB). Especially, NTB fully exploits cycle-consistent architecture to generate initial clean/rainy image pairs for conditional diffusion model training. Given initial image pairs, DTB leverages two conditional diffusion modules to progressively refine the desired output. Experiments on both synthetic datasets and real-world rainy images demonstrate that RainDiffusion performs favorably against current unsupervised and semi-supervised methods, and also possesses competitive advantages to the fully-supervised ones. 



{\small
\bibliographystyle{ieee_fullname}
