


\subsection{Global-Local Stepwise Generative Framework}
As shown in Figure~\ref{fig1:frame}, the three local pathways of our \emph{GLSGN} operate on three different sizes of images with decreasing resolutions and perform image restoration in a stepwise manner. The first pathway operates on each of sixteen local patches partitioned equally from the full-resolution image ($4K$) and the decoded feature maps are fed into the second local pathways, which performs restoration on patches from scaled-down images with a quarter of original resolution ($2K$). The third and the last pathway takes the input image with $1/16$ of initial size as input and combines the decoded features from the second pathway to perform image restoration. In this way, these three local pathways restore the background successively and achieve progressively cleaner background image. Meanwhile, the global pathway performs image restoration on the whole input image but with only $1/16$ of initial resolution to obtain a rough restoration result efficiently to provide cues in a global view for the three local pathways.

Formally, the $i$-th local pathway $\mathcal{S}_i$ of our \emph{GLSGN} performs image restoration by:
\begin{equation}
    \hat{\mathbf{B}}_i = \mathcal{S}_i(\mathbf{I}_i,\textbf{F}_{i-1},\textbf{F}_{\text{g}}),
\label{eqn:content_consis}
\end{equation}
where $\hat{\mathbf{B}}_i$ is the restored background image from the input patches $\mathbf{I}_i$, and $\mathbf{F}_{i-1}$ and $\mathbf{F}_{\text{g}}$ are the decoded features from the previous pathway $\mathcal{S}_{i-1}$ and the global pathway $\mathcal{S}_{g}$ respectively ($\mathcal{S}_{1}$ have no previous pathway and only receives feedback from $\mathcal{S}_{g}$). 
All four pathways share the same structure of encoders but with independent parameters. Note that the 
input patches for different pathways have different sizes of receptive fields corresponding to the initial intact image: the patches from higher resolution images correspond to smaller receptive field in the initial image.

\begin{comment}
As illustrated in Figure~\ref{fig2:frame}, main framework of our \emph{GLSGN} employs a multi-stage learning strategy(three patch-based stages and one global view stage) to collaboratively separate noise constituents from the background image.
Since the resolution of input image much increases, the diversity of noise becomes larger, and current one-stage methods cannot sufficiently learn noise distribution due to the limited receptive field. Thus, our \emph{GLSGN} mainly leverages two methods to enlarge the receptive field, the multi-stage learning strategy and the perceptual consistency attention.

\noindent\textbf{Multi-stage Learning Strategy.}
In the proposed \emph{GLSGN}, the number of cropped patches are decreasing from patch-based stage-1 to stage-3.
Unlike previous multi-stage frameworks~\cite{zhang2019deep,suin2020spatially,zamir2021multi} for image restoration, we employ an additional global stage to supplement global semantic information for patch-based stages. Combined with features from global stage and last local stage, our \emph{GLSGN} is able to obtain more multi-scale noise perception:
\begin{equation}
    \hat{\textbf{B}}_i = \mathcal{S}_i(\textbf{I}_i,\textbf{F}_{i-1},\textbf{F}_{\text{global}} )
\label{eqn:content_consis}
\end{equation}
where $\hat{\textbf{B}}_i$ is the restored image in the $i$-th patch-based stage $\mathcal{S}_i$, and \textbf{I} is the input image.
As can be seen in Figure~\ref{fig1:pipline}, even if the image scale is reducing in patch-based stages, the receptive field is increasing. 
For instance, the restored image in the first stage has the same resolution as the input image, and it is cropped into 16 patches during restoration. In the second stage, we reduce the scale of original image $\mathbf{I}$ into a half, and crop it into 8 patches.
Thus, the first stage owns higher fidelity of texture details, yet the last stage is able to more accurately model the noise patterns due to larger receptive fields.
Such strategy not merely enables our framework to learn richer contextual information, but also can model more complex noise patterns in the ultra high-resolution image. 
\end{comment}

\subsection{Global-Local Consistency between Pathways for Collaborative Image Restoration}
Since the input patches of different pathways have different resolutions and different sizes of receptive fields, the crux of our \emph{GLSGN} lies in the global-local consistency between four pathways to perform  image restoration collaboratively. Our \emph{GLSGN} is designed to ensure the global-local consistency in four aspects: 1) restored low-level content, 2) perceptual attention, 3) restoring intensity and 4) high-level semantics.

\begin{figure}[!t]
    \centering
   
   
    \includegraphics[width=\linewidth]{fig3.pdf
   
    \vspace{-8pt}
    \caption{(a) The structure of perceptual attention consistency (PAC) mechanism. (b) The structure of spatial attention.
    }
    \label{fig2:PAC}
   
\end{figure}
\smallskip\noindent\textbf{Consistency of restored low-level content by iterative decoding.}
To guarantee the consistency of low-level content restored by different pathways, we feed the decoded features of the previous pathway and the global pathway into the current pathway by concatenating them with the corresponding encoded layer of current pathway, as indicated in Equation~\ref{eqn:content_consis} and Figure~\ref{fig1:frame}. As a result, the decoding process of current pathway is performed referring to the global cues from the global pathways and fine-grained cues from the previous pathways. 


\smallskip\noindent\textbf{Perceptual Attention Consistency.}
Intuitively, the regions with heavy noise in an input image tend to obtain more attention than the regions with light noise during background restoration. The noise patterns in different scales of a same original image typically remain the same, thus four pathways in our \emph{GLSGN} should share the consistent perceptual attention during background restoration. To this end, we propose the Perceptual Attention Consistency (\emph{PAC}) mechanism as shown in Figure~\ref{fig2:PAC}(a). Specifically, for each pathway we employ a spatial attention module~\cite{feng2021deep} shown in Figure~\ref{fig2:PAC}(b) to calculate an attention map from the encoded features, which indicates the spatial perceptual attention for the input image. Then we normalize the perceptual attention of current pathway by fusing the attention maps from the previous pathway and the global pathway:
\begin{equation}
    \mathbf{A}_i = \frac{f^i_\text{att}(\mathbf{E}_i)+\sigma_1\cdot f^{i-1}_\text{att}(\mathbf{E}_{i-1})+\sigma_2\cdot f^g_\text{att}(\mathbf{F}_g)}{1+\sigma_1+\sigma_2},
   
\end{equation}
where $\mathbf{E}_i$ denotes the encoded features in the $i$-th pathway. Here we use the decoded features of global pathway $\mathbf{F}_g$ to provide a more accurate estimation for attention distribution. $f^i_\text{att}$ denotes the spatial attention module while $\sigma_1$ and $\sigma_2\in$[0, 1] are the hyper-parameters to balance between terms. $\mathbf{A}_i$ is the normalized attention map for $i$-th pathway, which is further used to re-weight the feature maps $\mathbf{E}_i$ by element-wise multiplication $\odot$:
\begin{equation}
    \mathbf{E}_i = \mathbf{A}_i \odot \mathbf{E}_i.
   
\end{equation}

\begin{comment}
Another way of increasing receptive fields in our \emph{GLSGN} is that we develop a perceptual attention consistent (PAC) module which combines information from different stages and ensures consistency of noise modeling between various receptive fields among stages, leading to cleaner restoration results with exquisite details.
Specifically, stages have different perceptual focuses due to processing images of various resolutions.
For example, the first stage pays more attention to the texture information, yet the last stage focuses more on the semantic structure in the image.
Thus, to normalize the perceptual focuses of various stages and estimate the distribution of noise patterns, we propose the perceptual consistency attention (PCA).
In each stage, we obtain its perceptual focus $\text{A}_i$ by computing attention maps of spatial information in the feature space.
Then, we normalize the perceptual focus of current stage by cooperating with the perceptual attention in the global stage and previous patch-based stage:
\begin{equation}
    \textbf{A}_i = \frac{f^1_{att}(\mathbf{F}_i)+\sigma_1\cdot f^2_{att}(\mathbf{F}_{i-1})+\sigma_2\cdot f^3_{att}(\mathbf{F}_g)}{1+\sigma_1+\sigma_2}
\end{equation}
where $\mathbf{F}_i$ denotes the features in the $i$-th stage, $f_{att}$ is the spatial attention mechanism~\cite{feng2021deep}, $\sigma_1$ and $\sigma_2\in$ [0, 1] are the hyper-parameters to balance between terms, and $\mathbf{A}_i$ is the regularized perceptual focus in the $i$-th stage.
\end{comment}

\smallskip\noindent\textbf{Consistency of restoring intensity by inter-patch normalization.}
\begin{figure}[!t]
    \centering
   
    \begin{minipage}[b]{0.95\linewidth}
    \includegraphics[width=\linewidth]{fig4.pdf}\vspace{4pt}
    \end{minipage}
    \vspace{-10pt}
    \caption{An example of reflection removal for our proposed patch normalization(PN). (a) The visualization of equation~\ref{equ1:PN}. (b) The restored result without patch normalization. (c) The restored result by patch normalization.
    }
    \label{fig4:PN}
   
\end{figure}
Each local pathway operates on the equally partitioned patches individually during both encoding and decoding. To keep the consistency of restoring intensity between different patches by a same pathway, 
we introduce the patch normalization (PN) that leverages statistics of adjacent patches to regularize the restoration of local patch, which is integrated into each residual block of decoder for each local pathway. Concretely, we first measure the restoring intensity for a patch by calculating the average variation ratios of all pixel values. Then for a patch to be regularized, we calculate its restoring intensity and the average restoring intensity of four adjacent patches $x_r$, and compute the ratio as the regularizing factor. Note that all calculations are based per channel, thus the regularizing factor 
$\mathcal{A}\in\mathbb{R}^{C}$ for patch $x_g$ is calculated as:
\begin{equation}
    \label{equ1:PN}
    \mathcal{A}_{\hat{x}_g}=\frac{\sum\limits^n_{r=1}\sum\limits^{W}_{i=1}\sum\limits^{H}_{j=1}\left|\frac{\hat{x}_{r, \langle i,j \rangle}}{x_{r, \langle i,j \rangle}}\right|}{n\cdot\sum\limits^{W}_{i=1}\sum\limits^{H}_{j=1}\left|\frac{\hat{x}_{g, \langle i,j \rangle}}{x_{g, \langle i,j \rangle}}\right|},
   
\end{equation}
where ${x}_{g, \langle i,j \rangle},\hat{x}_{g, \langle i,j \rangle}\in\mathbb{R}^{C\times H\times W}$ are the input and output channel-wise pixel of patch-wise feature maps at position $\langle i,j \rangle$ in the same residual block, and $n$ is the number of adjacent patches. In our implementation, we select four adjacent patches around patch $x_g$ as shown in Figure~\ref{fig4:PN}(a) and thus $n$ is 4.
Then the patch
\begin{equation}
    \label{equ2:PN}
    \hat{x}'_g=\mathcal{A}_{\hat{x}_g}\cdot\hat{x}_g+\mathcal{B}(\hat{x}_g),
   
\end{equation}
where $\mathcal{B}$ learns a bias by additional convolution layer. 
Figure~\ref{fig4:PN}(b) and Figure~\ref{fig4:PN}(c) further illustrate the effect of applying our proposed PN module, and it's easy to observe that our PN modules enable our framework to obtain more consistent background image between different patches.

\begin{figure}[!t]
    \centering
   
    \begin{minipage}[b]{1\linewidth}
    \includegraphics[width=\linewidth]{fig5.pdf}
    \end{minipage}
   
    \caption{Laplacian Pyramid for reconstructing full resolution ultra high-resolution background image. The green arrow denotes the mask updating steps.
    }
    \label{fig4:laplacian}
   
\end{figure}
\smallskip\noindent\textbf{Semantic consistency using Laplacian Pyramid for background synthesis.}
Four pathways generate four restored background images with different resolutions. It is crucial to ensure the high-level semantic consistency of these four background images for synthesizing a high-resolution background image from these four background images.
Inspired by Laplacian Pyramid (LP)~\cite{liang2021high,burt1987laplacian} for synthesizing high resolution images, we establish the Laplacian Pyramid based on four restored background images to reconstruct the final high-resolution background image.

\begin{figure*}[!t]
    \centering
    \begin{minipage}[b]{1.0\linewidth}
    \includegraphics[width=\linewidth]{fig6.pdf}
    \end{minipage}
    \caption{The 4K resolution samples from our UHR4K dataset. Top: images with reflection or rain streak. Bottom: the corresponding groundtruth background images.}
    \label{fig5:dataset}
\end{figure*}
Laplacian pyramid (LP)~\cite{burt1987laplacian}  has a widespread application in image processing, especially image blending~\cite{burt1987laplacian} and super-resolution~\cite{lai2017deep}. The crux of LP is to linearly decouple an image into high-frequency and low-frequency components with different resolutions, from which the original image can be reconstructed invertibly and accurately. Specifically, let $d(\cdot)$ and $u(\cdot)$ be a downsampling operation and an upsampling operator, respectively.
Given an arbitrary image $I_0 \in \mathbb{R}^{H \times W} $, it is first downsampled to obtain a low-frequency prediction $I_1 \in \mathbb{R}^{\frac{H}{2} \times \frac{W}{2}}$, as $I_1=d(I_0)$. To guarantee invertible reconstruction of the original image, the LP records the high-frequency residual $h_0$, as $h_0 = I_0-u(I_1)$. To further reduce image resolution, the LP iteratively performs above operations, obtaining a sequence of low- and high-frequency components. 
Reconstruction from the LP sequence is conducted using the backward recurrence: $I_k = u(I_{k+1}) + h_k$, where $k$ is the number of levels in the pyramid.


Following the above steps, we establish the LP based on four restored background images to reconstruct the final high-resolution background image, as illustrated in Figure~\ref{fig4:laplacian}. We first extract the high-frequency components [$h_1$, $h_2$, $h_3$] from the outputs of the two local pathways $\mathcal{S}_1$, $\mathcal{S}_2$ and the global pathway $\mathcal{S}_g$ for constructing the Laplacian Pyramid. Since the local pathway $\mathcal{S}_3$ performs restoration last in all four pathways, we use its restored background image as the low-frequency component $l$. By learning a mask $\mathbf{M}$ for removing the noise constituents of the high-frequency components, our Laplacian Pyramid can obtain noise-free high-frequency components:
\begin{equation}
    h'_i = h_i \odot \mathbf{M}_i,
   
\end{equation}
where $h_i$ is the high-frequency components in the $i$-th pathway.
Finally, according to the reversibility of Laplacian Pyramid, we can synthesize the final restored image $\hat{\mathbf{B}}_{\text{out}}$ by combining the high-frequency components [$h'_1, h'_2, h'_3$] and low-frequency component $l$:
\begin{equation}
    \hat{\mathbf{B}}_{\text{out}} = h'_1 + \text{Up}(h'_2+\text{Up}(h'_3+l)),
\end{equation}
where $\text{Up}$ denotes bi-linear interpolation upsampling.
\subsection{Jointly Supervised Parameter Learning}
We optimize the parameters of our \emph{GLSGN} in an end-to-end manner.

\smallskip\noindent\textbf{Multi-stage Pixel Reconstruction Loss.} We employ the $L_1$ loss to learn the pixel level reconstruction of each pathway:
\begin{small}
\begin{equation}
    \mathcal{L}_{\text{pixel}}=\alpha_1\sum_{i=1}^{N}{\mathcal{L}_1\left(\mathbf{B},{\hat{\mathbf{B}}}_i\right)} + \beta_1\mathcal{L}_1\left(\mathbf{B},{\hat{\mathbf{B}}}_{\text{out}}\right),
   
\end{equation}
\end{small}\noindent 
where ${\hat{\mathbf{B}}}_i$ denotes the restored image in the $i$-th pathway, and ${\hat{\mathbf{B}}}_{\text{out}}$ is the restored image of Laplacian Pyramid. Besides, we obtain groundtruth $\mathbf{B}$ of various scales by interpolation. Empirically, we set $\alpha_1\!=\!0.1$ and $\beta_1=0.5$.

\smallskip\noindent\textbf{Multi-stage Perceptual Loss.} To learn the consistency of semantic information, we perform supervision of the perceptual loss~\cite{johnson2016perceptual} in each pathway:
\begin{small}
\begin{equation}
    \mathcal{L}_\text{perc}=\alpha_2\sum_{i=1}^{N}{\mathcal{L}_{\text{VGG}}\left(\mathbf{B},{\hat{\mathbf{B}}}_i\right)} + \beta_2\mathcal{L}_{\text{VGG}}\left(\mathbf{B},{\hat{\mathbf{B}}}_{\text{out}}\right),
   
\end{equation}
\end{small} 

\noindent where $\mathcal{L}_{\text{VGG}}$ denotes perceptual distance between two images measured by a pre-trained VGG-19~\cite{simonyan2014very}. Empirically, we set $\alpha_2\!=\!0.1$ and $\beta_2\!=\!0.5$. 

\smallskip\noindent \textbf{Conditional Adversarial Loss}, which encourages the restored image $\hat{\mathbf{B}}_{\text{out}}$ to be as realistic as the ground-truth background image $\mathbf{B}$. We employ the spectral normalization~\cite{miyato2018spectral} in our discriminator to stabilize the adversarial learning process:
\begin{equation}
    \mathcal{L}_{\text{adv}}  = -\mathbb{E}_{\text{B}\backsim \mathbb{P}_\text{GLSGN}}[D^{sn}(G(\mathbf{I}))], 
   
\end{equation}
where $D^{sn}$ is the discriminator with the spectral normalization module after each convolution layer.
In sum, the total loss function is defined as follows:
\begin{equation}
   
    \mathcal{L} = \lambda_1\cdot\mathcal{L}_{\text{pixel}}+\lambda_2\cdot\mathcal{L}_{\text{perc}}+\lambda_3\cdot\mathcal{L}_{\text{adv}}.
   
\end{equation}
We set the hyper-parameters $\lambda_1=1$ and $\lambda_2=\lambda_3=0.01$ to balance between different loss functions.





\subsection{Image Reflection Removal}
\noindent\textbf{UHR4K-Real.}
We collect real-world 4K images using two cameras, Nikon D300 and Huawei mobile phone.
We follow two steps to collect paired background and degraded images:
1) The reflection-contaminated image is first captured through the glass; 2) The groundtruth background image is captured by removing the glass.
Altogether 336 image pairs are captured and we divide them into 316 image pairs as training set and 20 image pairs as test set.
To ensure that the model learns the reflection patterns rather than the image content, the real images from diverse scenes are collected. We also consider various reflection conditions while photographing real-world images including 1) circumstance: indoor and outdoor; 2) illumination: lamplight and sunlight; 3) glass thickness: 3mm and 9mm; 4) camera exposure value: 8.0-16.0; 5) camera view: front view and oblique view; 6) camera aperture: f/4.0 and f/16.0.

Due to the refraction effect of the glass, the inevitable spatial shifts still exist between the blended image taken behind the glass and the background image without the glass. To align the images as closely as possible, like Wan \emph{et al}.~\cite{wan2017benchmarking}, we first extract the ORB feature points~\cite{rublee2011orb} from the two images. Then the RANSAC algorithm~\cite{fischler1981random} is employed to calculate the homographic transformation matrix. Finally, the blended image is aligned to the background image by using the estimated transformation.
\begin{figure*}[!t]
    \centering
    \begin{minipage}[b]{1\linewidth}
    \includegraphics[width=\linewidth]{ablation_img.pdf}\vspace{4pt}
    \end{minipage}
   
    \caption{Reflection removal results by five variants of \emph{GLSGN} on a test image from Real20~\cite{zhang2018single} dataset.}
    \label{fig7:ablation}
   
\end{figure*}
\input{table5}

\noindent\textbf{UHR4K-Syn.}
To further expand the reflection scenes, we extract 2,167 frames from numerous 4K videos as background, covering a wide variety of scenes.
Following the previous approach~\cite{fan2017generic}, we synthesize the reflection image by applying the Gaussian kernel to blur the images of DIV2K dataset~\cite{agustsson2017ntire}. 
Finally, the reflection image and background image are combined to synthesize the degraded image according to the physical principle~\cite{fan2017generic} of reflection formation.
Consequently, we obtain 2,167 image pairs, and divide 2,117 image pairs as training set and 50 image pairs as test set.


\begin{table}[!t]
\centering
\caption{Comparison with related real-world datasets for image reflection removal. The statistics are based on the image resolution, image amount, and scenes.}
\renewcommand\arraystretch{1.1}
\resizebox{0.875\linewidth}{!}{
\begin{tabular}{l|cccc}
\toprule
Dataset&Resolution & Amount & Scenes &  \\
\midrule
Solid~\cite{wan2017benchmarking}&540$\times$400&200&20 \\
Postcard~\cite{wan2017benchmarking}&540$\times$400&200&5  \\
Wild~\cite{wan2017benchmarking}&540$\times$400&55&50 \\
Nature~\cite{li2020single}&600$\times$400&220&67 \\
Real20~\cite{zhang2018single}&$\approx$1106$\times$902&110&83 \\
UHR4K-Real&\textbf{3840$\times$2160}&\textbf{336}&\textbf{116} \\
\bottomrule
\end{tabular}}
\label{tab:eff}
\end{table}

\subsection{Image Rain Streak Removal}
\noindent\textbf{UHR4K-Rain.} Due to the difficulty of capturing real-world image pairs for image rain streak removal, we simulate the rain conditions by employing  the photoshop tutorial~\cite{RN16} to synthesize rain streaks. 
First, we randomly generate noise points with different densities. Then, we simulate rain streak of different sizes, and directions by stretching and rotating the direction of the noise with the Gaussian blur kernel. Finally, we superimpose the generated rain streaks mask onto the background image for the synthetic rain image. 
To ensure diversity, the generated rain streaks images have different orientations, densities, sizes and contrasts. Similar to the synthetic reflection dataset, we synthesize 2,167 rainy images using the background images we collected. Among them, 2,117 image pairs are used for training and 50 are used for testing.


\subsection{Comparison with Existing Datasets}
Compared to existing datasets~\cite{wan2017benchmarking, zhang2018single, li2020single, yang2017deep, zhang2018density} for image reflection removal and rain streak removal, our UHR4K enjoys the following advantages:
1) Much higher resolution (3840$\times$2160) of images, which challenges most frameworks for image restoration; 2) containing both synthetic and real-world images, which enables more comprehensive evaluation; 3) various degradations considered to make the dataset more realistic.

There is no real-world dataset and only one synthetic dataset~\cite{zheng2021ultra} for 4K image restoration to date, thus we construct UHR4K by collecting both synthetic and real-world images.
Compared with popular real-world datasets for reflection removal in Table~\ref{tab:eff}, our dataset contains more real-world images covering more scenes.
We refer to Real20~\cite{li2020single}
and Nature~\cite{zhang2018single} to set the split ratio of training and test set and amount of test images.
Our dataset has no overlapped scenes between training and test set while the other two datasets do.
We randomly select 20 images as held-out validation set to guide the model training.
\subsection{The UHD4K Dataset for image restoration}





\subsection{Implementation Details}
We implement our \emph{GLSGN} in distribution mode with 4 RTX 3090 GPUs under Pytorch, and employ the Adam~\cite{kingma2014adam} to optimize the process of gradient descent with batch-size 4. The learning rate is 2$\times10^{-4}$ during 500 training epochs. In our experiments, we randomly crop training images into 960$\times$540, and augment training by random flipping and rotation. 
In addition, we retrain all methods according to their released official codes and papers for fair comparison on ultra high-resolution images.




\begin{figure}[!t]
    \begin{center}
    \centering
    \begin{minipage}[b]{0.9\linewidth}
    \includegraphics[width=\linewidth]{runtimes.pdf}
    \end{minipage}
    \caption{Runtime vs Performance.}
    \label{fig8:runtime}
    \end{center}
\end{figure}
\input{table1}
\subsection{Ablation Study}
\noindent\textbf{Ablation experiments on reflection removal.} To investigate functional technique components in our \emph{GLSGN}, we conduct ablation study on the reflection removal task,
and perform experiments on five variants of our \emph{GLSGN}:
1) \textbf{Global-Only}, which only employs the global pathway; 
2) \textbf{Global-Local}, which collaboratively employs both the global pathway and local pathways; 
3) \textbf{\emph{GLSGN}-PN} that brings in the patch-normalization; 
4) \textbf{\emph{GLSGN}-PAC} that leverages the proposed perceptual attention consistency; 
5) \textbf{\emph{GLSGN}(complete)} further employs the Laplacian Pyramid for image reconstruction.

Table~\ref{tab1:ablation} presents the experimental results of ablation study. The comparison between \textbf{Global-Only} and \textbf{Global-Local} demonstrates the necessity of global-local collaboration. The increasingly better quantitative performance of five variants shows the effectiveness of proposed technical components. Furthermore, we visualize the restored results of various variants in Figure~\ref{fig7:ablation} on a randomly selected testing image, which also illustrates the superiority of each component in our \emph{GLSGN} for thoroughly removing noise patterns.

\begin{figure}[!t]
    \centering
   
    \begin{minipage}[b]{1\linewidth}
    \includegraphics[width=\linewidth]{fig7.pdf
    \end{minipage}
   
    \caption{Visualization of rain streak perception in the second local pathway on a randomly selected 4K test image for rain streak removal. A$_{S\!P}$ is the attention maps of plain spatial attention, and A$_{P\!A\!C}$ is the attention maps of PAC mechanism.
    }
    \label{fig7:PCA}
    \vspace{-4pt}
\end{figure}
\begin{figure}[!t]
    \centering
    \begin{minipage}[b]{0.925\linewidth}
    \includegraphics[width=\linewidth]{Double.pdf
    \end{minipage}
   
    \caption{Experimental results of different numbers of local pathways in \emph{GLSGN} on 4KID~\cite{zheng2021ultra} dataset. The local pathway number N=3 yields the best performance in terms of PSNR and SSIM. 
    }
    \label{fig8:stage}
   
\end{figure}

\noindent\textbf{Effect of the perceptual attention consistency}. Figure~\ref{fig7:PCA} visualizes the attention maps of our perceptual attention consistency and plain spatial attention~\cite{feng2021deep}. Our cross-pathway perceptual attention consistency obviously enables the framework to model rain streaks more completely and synthesize higher quality results than plain spatial attention. 


\begin{table}[!t]
\centering
\caption{Runtime and memory usage on different devices.}
\renewcommand\arraystretch{1.1}
\resizebox{1\linewidth}{!}{
\Huge
\begin{tabular}{l|ccccc|cc}
\toprule
Method &\multicolumn{5}{c|}{GLSGN (ours)}&MPRNet~\cite{zamir2021multi}&DMGN~\cite{feng2021deep} \\

\cmidrule(lr){0-1}
\cmidrule(lr){2-6}
\cmidrule(lr){7-8}

Device &RTX3090&RTX Titan& RTX 2080 &Titan xp &CPU &\multicolumn{2}{c}{RTX3090} \\

\midrule
Runtime(s)&0.082 &0.113&0.142&0.155&3.980 &0.549&0.767 \\
Memory(GB) &\multicolumn{5}{c|}{8.626}&17.72&21.60 \\
\bottomrule
\end{tabular}}
\label{tab:devices}
\end{table}

\noindent\textbf{Investigation on the model efficiency.}
To investigate whether the performance superiority of our \emph{GLSGN} over competing methods is benefited from the advantages of model design or model complexity, we compare the model efficiency of \emph{GLSGN} with six state-of-the-art methods~\cite{wen2019single, wei2019single, li2020single, dong2020multi, feng2021deep, zheng2021ultra} for image restoration in Table~\ref{tab:efficiency}. 
Besides, as shown in Figure~\ref{fig8:runtime}, we also perform the comparison between \emph{GLSGN} and other methods on 4KID~\cite{zheng2021ultra} dataset in terms of model performance and runtime on the dehazing task. 
It demonstrates our \emph{GLSGN} not only enjoys less TFLOPS, trainable parameters and runtime, but also achieves the best performance than most competing methods.
To further investigate the efficiency of our proposed \emph{GLSGN} on a 4K test image, we list the runtime and memory usage on various computational devices in Table~\ref{tab:devices}. The results demonstrate the high efficiency of our proposed \emph{GLSGN}.


\noindent\textbf{Investigation on the number of local pathways.} To explore how many pathways are optimal to predict the clean background image, we train the model with different numbers of local pathways on ultra high-resolution image dehazing.
The experimental results are illustrated in Figure~\ref{fig8:stage}, which shows deploying 3 local pathways achieves the best performance.

\begin{figure*}[!t]
    \centering
    \begin{minipage}[b]{1\linewidth}
    \includegraphics[width=\linewidth]{dereflection_syn.pdf}
    \end{minipage}
    \vspace{-16pt}
    \caption{Visualization of restored images for reflection removal on UHR4K-Syn dataset.
    }
   
    \label{fig7:syn}
   
\end{figure*}
\begin{figure*}[!t]
    \centering
    \begin{minipage}[b]{1\linewidth}
    \includegraphics[width=\linewidth]{dereflection_real.pdf}
    \end{minipage}
    \vspace{-16pt}
    \caption{
    Visualization of restored images for reflection removal on UHR4K-Real dataset
    }
    \label{fig7:real}
   
\end{figure*}

\begin{table}[!t]
    \centering
    \caption{Analysis of SSIM loss and high-frequency scales.}
    \renewcommand\arraystretch{1.1}
    \resizebox{1\linewidth}{!}{
        \large
        \begin{tabular}{l|c|cccc}
    \toprule
    \multicolumn{1}{l|}{Dataset} &Metric& GLSGN$_{\text{SSIM}}$ &GLSGN$_{h_0}$&GLSGN$_{h_0+h_1}$ & Ours \\
    \midrule
    \multirow{2}{*}{UHR4K-Real}&PSNR&23.54&23.58&23.86&\textbf{24.35}\\
    ~&SSIM&0.832&0.831&0.836&\textbf{0.841}\\
    \bottomrule
    \end{tabular}%
    }
    \label{tab1:ssim}
\end{table}
\noindent\textbf{Investigation on Laplacian pyramid (LP).}
we conduct reflection removal experiments on UHR4K-Real dataset to investigate 1) the difference between LP and SSIM loss, 2) the effect of different scale of high-frequency information in LP, which is shown in Table~\ref{tab1:ssim}. The results show that LP performs better than SSIM loss distinctly and it is important to incorporate different scales of high-frequency information.
\begin{table}[!t]
    \centering
    \caption{Human evaluation on the reflection removal results. 50 human subjects are asked to perform comparison between our model and other three methods on the restored background images of 50 randomly selected test samples. Our model obtains $75.2\%$ votes among $50\times 50 = 2500$ comparisons and wins on 44 samples.}
    \renewcommand\arraystretch{1.1}
   
    \resizebox{0.8\linewidth}{!}{
    \begin{tabular}{l|cc}
   
    \toprule
    Model  &Share of the vote &Winning samples \\
    \midrule
ERRNet~\cite{wei2019single} & 1.8$\%$  &0 \\
IBCLN~\cite{li2020single}  &10.6$\%$   &2 \\
DMGN~\cite{feng2021deep}   & 12.4$\%$    &4 \\
Ours &75.2$\%$ &44 \\
\bottomrule
    \end{tabular}
    }
    \label{tab5_1:human_evalution}
\end{table}

\noindent\textbf{User study.} This section further presents the results of user study because PSNR and SSIM have their bias for quality evaluation of restored background images. We perform human evaluation
to compare our model with other top-three most powerful methods for reflection removal including ERRNet~\cite{wei2019single}, IBCLN~\cite{li2020single} and DMGN~\cite{feng2021deep}. We randomly select 50 test samples and present the generated background images by our model and other three methods to 50 human subjects for manual comparison of restoring quality. As shown in Table~\ref{tab5_1:human_evalution}, among total 50×50 = 2500 comparisons, our model wins on 75.2$\%$ of 2,500 votes which is much more than other methods. Besides, we aggregate evaluation results of all subjects for each sample, and our model wins on 44 test samples and fails on only 6 test samples.
\begin{figure*}[htbp]
    \centering
    \begin{minipage}[b]{1\linewidth}
    \includegraphics[width=\linewidth]{derain_sample.pdf}\vspace{4pt}
    \end{minipage}
   
    \caption{Visualization of restored images for rain streak removal on UHR4K-Rain dataset.
    }
    \label{fig8:derain}
   
\end{figure*}

\input{table2}

\subsection{Comparison with State-of-the-art Methods}
\subsubsection{\textbf{Experiments on image reflection removal.}}
We conduct experiments to compare our \emph{GLSGN} with nine state-of-the-art methods~\cite{zou2020deep,feng2021deep,zamir2021multi,wen2019single,zheng2021ultra,wei2019single,li2020single,feng2021contrastive,Zamir2021Restormer} for image reflection removal on five datasets including our proposed ultra high-resolution datasets as UHR4K-Real and UHR4K-Syn, and multiple public benchmark datasets with regular resolution as Real20~\cite{zhang2018single}, Nature~\cite{li2020single}, and $\text{SIR}^2$~\cite{wan2017benchmarking}. Table~\ref{tab3:reflection} lists the quantitative results of various methods for image reflection removal on five datasets in terms of PSNR and SSIM. Our \emph{GLSGN} achieves significant improvement on ultra high-resolution datasets compared to other competing methods. Meanwhile, on several public benchmark datasets, our method also obtains comparable results against state-of-the-art methods. It is reasonable that stronger capability of noise modeling drives our model to enjoy higher performance and thus \emph{GLSGN} is able to restore higher-quality image from the degraded image. The large gap between Zheng \emph{et al.}~\cite{zheng2021ultra} and our \emph{GLSGN} demonstrates that our model has significant superiority for restoring ultra high-resolution images over the method specifically designed for high-resolution image restoration, even on challenging reflection removal task.

To gain more insights into the model performance of various methods, we randomly select two examples from the synthetic and real-world test sets and visualize their restored images in Figure~\ref{fig7:syn} and Figure~\ref{fig7:real} respectively. Our \emph{GLSGN} is able to remove most of reflection in the input image and synthesize cleaner background images. 

\begin{figure*}[htbp]
    \centering
    \begin{minipage}[b]{1\linewidth}
    \includegraphics[width=\linewidth]{dehaze_sample.pdf}\vspace{4pt}
    \end{minipage}
   
    \caption{Visualization of restored images for image dehazing on dehazing dataset.
    }
    \label{fig7:dehaze}
   
\end{figure*}
\input{table3}

\subsubsection{\textbf{Experiments on image rain streak removal.}}
We conduct experiments to compare our \emph{GLSGN} with nine state-of-the-art methods~\cite{li2018recurrent,ren2019progressive,chen2021robust,wang2021rain,zou2020deep,feng2021deep,zamir2021multi,zheng2021ultra,Zamir2021Restormer} for image rain streak removal on our collected UHR4K-Rain dataset and two public benchmark datasets with regular resolution: Rain100H~\cite{yang2017deep} and Rain100L~\cite{yang2017deep} datasets.
Table~\ref{tab4:derain} presents the quantitative results of various methods for rain streak removal, and our \emph{GLSGN} consistently achieves the best performance on UHR4K-Rain and Rain100L datasets over nine methods. In particular, on UHR4K-Rain dataset, our method achieves significant improvement over other methods, indicating the strength of our model in processing ultra high-resolution images. On Rain100H dataset, our method obtains 
similar performance compared to the state-of-the-art methods for image rain streak removal. The experimental results indicate the superiority of our \emph{GLSGN} on both regular resolution and high-resolution images.


To have a qualitative comparison, we visualize the restored images by different methods for image rain streak removal on two randomly selected examples from the test set in Figure~\ref{fig8:derain}. In particular, our \emph{GLSGN} performs distinctly better than other competing methods for rain streak removal.
Benefiting from effective collaboration between various pathways, our \emph{GLSGN} can more thoroughly remove rain streaks in ultra high-resolution images.


\subsubsection{\textbf{Experiments on image dehazing.}}
We conduct ultra high-resolution image dehazing experiments on existing datasets OHaze~\cite{ancuti2018haze} and 4KID~\cite{zheng2021ultra} to compare our proposed \emph{GLSGN} with nine current state-of-the-art methods~\cite{he2010single,qin2020ffa,dong2020multi,zou2020deep,feng2021deep,wu2021contrastive,zamir2021multi,zheng2021ultra,Zamir2021Restormer}.
Table~\ref{tab:dehaze} presents the quantitative results of different methods for image dehazing in terms of PSNR and SSIM, and our \emph{GLSGN} surpasses all other state-of-the-art models by a large margin on both OHaze~\cite{ancuti2018haze} and 4KID~\cite{zheng2021ultra} datasets.
\input{table4}

We visualize the restored details of the background image by different methods on two randomly selected test images in Figure~\ref{fig7:dehaze}.
It manifests that haze patterns are removed more thoroughly by our \emph{GLSGN} than other competing methods. Especially, compared to
Zheng \emph{et al}.~\cite{zheng2021ultra} which is a method specifically designed for ultra high-resolution image dehazing, 
our \emph{GLSGN} is able to synthesize higher quality background image due to the consideration of collaborative consistency among pathways.



\section{Introduction}
\label{sec:intro}
\input{1.introduction}

\section{Related Work}
\label{sec:relatedwork}
\input{2.relatedwork}

\section{Global-Local Stepwise Generative Network}
\label{sec:method}
\input{3.method}

\section{UHR4K Dataset}
\label{sec:data}
\input{4.0.dataset}

\section{Experiments}
\label{sec:exp}
\input{4.experiments}

\section{Conclusion}
\label{sec:con}
\input{5.conclusion}


\bibliographystyle{IEEEtran}

\section{Introduction}
\IEEEPARstart{T}{his} file is intended to serve as a ``sample article file''
for IEEE journal papers produced under \LaTeX\ using
IEEEtran.cls version 1.8b and later. The most common elements are covered in the simplified and updated instructions in ``New\_IEEEtran\_how-to.pdf''. For less common elements you can refer back to the original ``IEEEtran\_HOWTO.pdf''. It is assumed that the reader has a basic working knowledge of \LaTeX. Those who are new to \LaTeX \ are encouraged to read Tobias Oetiker's ``The Not So Short Introduction to \LaTeX ,'' available at: \url{http://tug.ctan.org/info/lshort/english/lshort.pdf} which provides an overview of working with \LaTeX.

\section{The Design, Intent, and \\ Limitations of the Templates}
The templates are intended to {\bf{approximate the final look and page length of the articles/papers}}. {\bf{They are NOT intended to be the final produced work that is displayed in print or on IEEEXplore\textsuperscript{\textregistered}}}. They will help to give the authors an approximation of the number of pages that will be in the final version. The structure of the \LaTeX\ files, as designed, enable easy conversion to XML for the composition systems used by the IEEE. The XML files are used to produce the final print/IEEEXplore pdf and then converted to HTML for IEEEXplore.

\section{Where to Get \LaTeX \ Help --- User Groups}
The following online groups are helpful to beginning and experienced \LaTeX\ users. A search through their archives can provide many answers to common questions.
\begin{list}{}{}
\item{\url{http://www.latex-community.org/}} 
\item{\url{https://tex.stackexchange.com/} }
\end{list}

\section{Other Resources}
See \cite{ref1,ref2,ref3,ref4,ref5} for resources on formatting math into text and additional help in working with \LaTeX .

\section{Text}
For some of the remainer of this sample we will use dummy text to fill out paragraphs rather than use live text that may violate a copyright.

Itam, que ipiti sum dem velit la sum et dionet quatibus apitet voloritet audam, qui aliciant voloreicid quaspe volorem ut maximusandit faccum conemporerum aut ellatur, nobis arcimus.
Fugit odi ut pliquia incitium latum que cusapere perit molupta eaquaeria quod ut optatem poreiur? Quiaerr ovitior suntiant litio bearciur?

Onseque sequaes rectur autate minullore nusae nestiberum, sum voluptatio. Et ratem sequiam quaspername nos rem repudandae volum consequis nos eium aut as molupta tectum ulparumquam ut maximillesti consequas quas inctia cum volectinusa porrum unt eius cusaest exeritatur? Nias es enist fugit pa vollum reium essusam nist et pa aceaqui quo elibusdandis deligendus que nullaci lloreri bla que sa coreriam explacc atiumquos simolorpore, non prehendunt lam que occum\cite{ref6} si aut aut maximus eliaeruntia dia sequiamenime natem sendae ipidemp orehend uciisi omnienetus most verum, ommolendi omnimus, est, veni aut ipsa volendelist mo conserum volores estisciis recessi nveles ut poressitatur sitiis ex endi diti volum dolupta aut aut odi as eatquo cullabo remquis toreptum et des accus dolende pores sequas dolores tinust quas expel moditae ne sum quiatis nis endipie nihilis etum fugiae audi dia quiasit quibus.
\IEEEpubidadjcol
Ibus el et quatemo luptatque doluptaest et pe volent rem ipidusa eribus utem venimolorae dera qui acea quam etur aceruptat.
Gias anis doluptaspic tem et aliquis alique inctiuntiur?

Sedigent, si aligend elibuscid ut et ium volo tem eictore pellore ritatus ut ut ullatus in con con pere nos ab ium di tem aliqui od magnit repta volectur suntio. Nam isquiante doluptis essit, ut eos suntionsecto debitiur sum ea ipitiis adipit, oditiore, a dolorerempos aut harum ius, atquat.

Rum rem ditinti sciendunti volupiciendi sequiae nonsect oreniatur, volores sition ressimil inus solut ea volum harumqui to see\eqref{deqn_ex1a} mint aut quat eos explis ad quodi debis deliqui aspel earcius.

\begin{equation}
\label{deqn_ex1a}
x = \sum_{i=0}^{n} 2{i} Q.
\end{equation}

Alis nime volorempera perferi sitio denim repudae pre ducilit atatet volecte ssimillorae dolore, ut pel ipsa nonsequiam in re nus maiost et que dolor sunt eturita tibusanis eatent a aut et dio blaudit reptibu scipitem liquia consequodi od unto ipsae. Et enitia vel et experferum quiat harum sa net faccae dolut voloria nem. Bus ut labo. Ita eum repraer rovitia samendit aut et volupta tecupti busant omni quiae porro que nossimodic temquis anto blacita conse nis am, que ereperum eumquam quaescil imenisci quae magnimos recus ilibeaque cum etum iliate prae parumquatemo blaceaquiam quundia dit apienditem rerit re eici quaes eos sinvers pelecabo. Namendignis as exerupit aut magnim ium illabor roratecte plic tem res apiscipsam et vernat untur a deliquaest que non cus eat ea dolupiducim fugiam volum hil ius dolo eaquis sitis aut landesto quo corerest et auditaquas ditae voloribus, qui optaspis exero cusa am, ut plibus.


\section{Some Common Elements}
\subsection{Sections and Subsections}
Enumeration of section headings is desirable, but not required. When numbered, please be consistent throughout the article, that is, all headings and all levels of section headings in the article should be enumerated. Primary headings are designated with Roman numerals, secondary with capital letters, tertiary with Arabic numbers; and quaternary with lowercase letters. Reference and Acknowledgment headings are unlike all other section headings in text. They are never enumerated. They are simply primary headings without labels, regardless of whether the other headings in the article are enumerated. 

\subsection{Citations to the Bibliography}
The coding for the citations is made with the \LaTeX\ $\backslash${\tt{cite}} command. 
This will display as: see \cite{ref1}.

For multiple citations code as follows: {\tt{$\backslash$cite\{ref1,ref2,ref3\}}}
 which will produce \cite{ref1,ref2,ref3}. For reference ranges that are not consecutive code as {\tt{$\backslash$cite\{ref1,ref2,ref3,ref9\}}} which will produce  \cite{ref1,ref2,ref3,ref9}

\subsection{Lists}
In this section, we will consider three types of lists: simple unnumbered, numbered, and bulleted. There have been many options added to IEEEtran to enhance the creation of lists. If your lists are more complex than those shown below, please refer to the original ``IEEEtran\_HOWTO.pdf'' for additional options.\\

\subsubsection*{\bf A plain  unnumbered list}
\begin{list}{}{}
\item{bare\_jrnl.tex}
\item{bare\_conf.tex}
\item{bare\_jrnl\_compsoc.tex}
\item{bare\_conf\_compsoc.tex}
\item{bare\_jrnl\_comsoc.tex}
\end{list}

\subsubsection*{\bf A simple numbered list}
\begin{enumerate}
\item{bare\_jrnl.tex}
\item{bare\_conf.tex}
\item{bare\_jrnl\_compsoc.tex}
\item{bare\_conf\_compsoc.tex}
\item{bare\_jrnl\_comsoc.tex}
\end{enumerate}

\subsubsection*{\bf A simple bulleted list}
\begin{itemize}
\item{bare\_jrnl.tex}
\item{bare\_conf.tex}
\item{bare\_jrnl\_compsoc.tex}
\item{bare\_conf\_compsoc.tex}
\item{bare\_jrnl\_comsoc.tex}
\end{itemize}





\subsection{Figures}
Fig. 1 is an example of a floating figure using the graphicx package.
 Note that $\backslash${\tt{label}} must occur AFTER (or within) $\backslash${\tt{caption}}.
 For figures, $\backslash${\tt{caption}} should occur after the $\backslash${\tt{includegraphics}}.

\begin{figure}[!t]
\centering
\includegraphics[width=2.5in]{fig1}
\caption{Simulation results for the network.}
\label{fig_1}
\end{figure}

Fig. 2(a) and 2(b) is an example of a double column floating figure using two subfigures.
 (The subfig.sty package must be loaded for this to work.)
 The subfigure $\backslash${\tt{label}} commands are set within each subfloat command,
 and the $\backslash${\tt{label}} for the overall figure must come after $\backslash${\tt{caption}}.
 $\backslash${\tt{hfil}} is used as a separator to get equal spacing.
 The combined width of all the parts of the figure should do not exceed the text width or a line break will occur.
\begin{figure*}[!t]
\centering
\subfloat[]{\includegraphics[width=2.5in]{fig1}%
\label{fig_first_case}}
\hfil
\subfloat[]{\includegraphics[width=2.5in]{fig1}%
\label{fig_second_case}}
\caption{Dae. Ad quatur autat ut porepel itemoles dolor autem fuga. Bus quia con nessunti as remo di quatus non perum que nimus. (a) Case I. (b) Case II.}
\label{fig_sim}
\end{figure*}

Note that often IEEE papers with multi-part figures do not place the labels within the image itself (using the optional argument to $\backslash${\tt{subfloat}}[]), but instead will
 reference/describe all of them (a), (b), etc., within the main caption.
 Be aware that for subfig.sty to generate the (a), (b), etc., subfigure
 labels, the optional argument to $\backslash${\tt{subfloat}} must be present. If a
 subcaption is not desired, leave its contents blank,
 e.g.,$\backslash${\tt{subfloat}}[].


 

\section{Tables}
Note that, for IEEE-style tables, the
 $\backslash${\tt{caption}} command should come BEFORE the table. Table captions use title case. Articles (a, an, the), coordinating conjunctions (and, but, for, or, nor), and most short prepositions are lowercase unless they are the first or last word. Table text will default to $\backslash${\tt{footnotesize}} as
 the IEEE normally uses this smaller font for tables.
 The $\backslash${\tt{label}} must come after $\backslash${\tt{caption}} as always.
 
\begin{table}[!t]
\caption{An Example of a Table\label{tab:table1}}
\centering
\begin{tabular}{|c||c|}
\hline
One & Two\\
\hline
Three & Four\\
\hline
\end{tabular}
\end{table}

\section{Algorithms}
Algorithms should be numbered and include a short title. They are set off from the text with rules above and below the title and after the last line.

\begin{algorithm}[H]
\caption{Weighted Tanimoto ELM.}\label{alg:alg1}
\begin{algorithmic}
\STATE 
\STATE {\textsc{TRAIN}}$(\mathbf{X} \mathbf{T})$
\STATE \hspace{0.5cm}$ \textbf{select randomly } W \subset \mathbf{X}  $
\STATE \hspace{0.5cm}$ N_\mathbf{t} \gets | \{ i : \mathbf{t}_i = \mathbf{t} \} | $ \textbf{ for } $ \mathbf{t}= -1,+1 $
\STATE \hspace{0.5cm}$ B_i \gets \sqrt{ \textsc{max}(N_{-1},N_{+1}) / N_{\mathbf{t}_i} } $ \textbf{ for } $ i = 1,...,N $
\STATE \hspace{0.5cm}$ \hat{\mathbf{H}} \gets  B \cdot (\mathbf{X}^T\textbf{W})/( \mathbb{1}\mathbf{X} + \mathbb{1}\textbf{W} - \mathbf{X}^T\textbf{W} ) $
\STATE \hspace{0.5cm}$ \beta \gets \left ( I/C + \hat{\mathbf{H}}^T\hat{\mathbf{H}} \right )^{-1}(\hat{\mathbf{H}}^T B\cdot \mathbf{T})  $
\STATE \hspace{0.5cm}\textbf{return}  $\textbf{W},  \beta $
\STATE 
\STATE {\textsc{PREDICT}}$(\mathbf{X} )$
\STATE \hspace{0.5cm}$ \mathbf{H} \gets  (\mathbf{X}^T\textbf{W} )/( \mathbb{1}\mathbf{X}  + \mathbb{1}\textbf{W}- \mathbf{X}^T\textbf{W}  ) $
\STATE \hspace{0.5cm}\textbf{return}  $\textsc{sign}( \mathbf{H} \beta )$
\end{algorithmic}
\label{alg1}
\end{algorithm}

Que sunt eum lam eos si dic to estist, culluptium quid qui nestrum nobis reiumquiatur minimus minctem. Ro moluptat fuga. Itatquiam ut laborpo rersped exceres vollandi repudaerem. Ulparci sunt, qui doluptaquis sumquia ndestiu sapient iorepella sunti veribus. Ro moluptat fuga. Itatquiam ut laborpo rersped exceres vollandi repudaerem. 
\section{Mathematical Typography \\ and Why It Matters}

Typographical conventions for mathematical formulas have been developed to {\bf provide uniformity and clarity of presentation across mathematical texts}. This enables the readers of those texts to both understand the author's ideas and to grasp new concepts quickly. While software such as \LaTeX \ and MathType\textsuperscript{\textregistered} can produce aesthetically pleasing math when used properly, it is also very easy to misuse the software, potentially resulting in incorrect math display.

IEEE aims to provide authors with the proper guidance on mathematical typesetting style and assist them in writing the best possible article. As such, IEEE has assembled a set of examples of good and bad mathematical typesetting \cite{ref1,ref2,ref3,ref4,ref5}. 

Further examples can be found at \url{http://journals.ieeeauthorcenter.ieee.org/wp-content/uploads/sites/7/IEEE-Math-Typesetting-Guide-for-LaTeX-Users.pdf}

\subsection{Display Equations}
The simple display equation example shown below uses the ``equation'' environment. To number the equations, use the $\backslash${\tt{label}} macro to create an identifier for the equation. LaTeX will automatically number the equation for you.
\begin{equation}
\label{deqn_ex1}
x = \sum_{i=0}^{n} 2{i} Q.
\end{equation}

\noindent is coded as follows:
\begin{verbatim}
\begin{equation}
\label{deqn_ex1}
x = \sum_{i=0}^{n} 2{i} Q.
\end{equation}
\end{verbatim}

To reference this equation in the text use the $\backslash${\tt{ref}} macro. 
Please see (\ref{deqn_ex1})\\
\noindent is coded as follows:
\begin{verbatim}
Please see (\ref{deqn_ex1})\end{verbatim}

\subsection{Equation Numbering}
{\bf{Consecutive Numbering:}} Equations within an article are numbered consecutively from the beginning of the
article to the end, i.e., (1), (2), (3), (4), (5), etc. Do not use roman numerals or section numbers for equation numbering.

\noindent {\bf{Appendix Equations:}} The continuation of consecutively numbered equations is best in the Appendix, but numbering
 as (A1), (A2), etc., is permissible.\\

\noindent {\bf{Hyphens and Periods}}: Hyphens and periods should not be used in equation numbers, i.e., use (1a) rather than
(1-a) and (2a) rather than (2.a) for subequations. This should be consistent throughout the article.

\subsection{Multi-Line Equations and Alignment}
Here we show several examples of multi-line equations and proper alignments.

\noindent {\bf{A single equation that must break over multiple lines due to length with no specific alignment.}}
\begin{multline}
\text{The first line of this example}\\
\text{The second line of this example}\\
\text{The third line of this example}
\end{multline}

\noindent is coded as:
\begin{verbatim}
\begin{multline}
\text{The first line of this example}\\
\text{The second line of this example}\\
\text{The third line of this example}
\end{multline}
\end{verbatim}

\noindent {\bf{A single equation with multiple lines aligned at the = signs}}
\begin{align}
a &= c+d \\
b &= e+f
\end{align}
\noindent is coded as:
\begin{verbatim}
\begin{align}
a &= c+d \\
b &= e+f
\end{align}
\end{verbatim}

The {\tt{align}} environment can align on multiple  points as shown in the following example:
\begin{align}
x &= y & X & =Y & a &=bc\\
x' &= y' & X' &=Y' &a' &=bz
\end{align}
\noindent is coded as:
\begin{verbatim}
\begin{align}
x &= y & X & =Y & a &=bc\\
x' &= y' & X' &=Y' &a' &=bz
\end{align}
\end{verbatim}





\subsection{Subnumbering}
The amsmath package provides a {\tt{subequations}} environment to facilitate subnumbering. An example:

\begin{subequations}\label{eq:2}
\begin{align}
f&=g \label{eq:2A}\\
f' &=g' \label{eq:2B}\\
\mathcal{L}f &= \mathcal{L}g \label{eq:2c}
\end{align}
\end{subequations}

\noindent is coded as:
\begin{verbatim}
\begin{subequations}\label{eq:2}
\begin{align}
f&=g \label{eq:2A}\\
f' &=g' \label{eq:2B}\\
\mathcal{L}f &= \mathcal{L}g \label{eq:2c}
\end{align}
\end{subequations}

\end{verbatim}

\subsection{Matrices}
There are several useful matrix environments that can save you some keystrokes. See the example coding below and the output.

\noindent {\bf{A simple matrix:}}
\begin{equation}
\begin{matrix}  0 &  1 \\ 
1 &  0 \end{matrix}
\end{equation}
is coded as:
\begin{verbatim}
\begin{equation}
\begin{matrix}  0 &  1 \\ 
1 &  0 \end{matrix}
\end{equation}
\end{verbatim}

\noindent {\bf{A matrix with parenthesis}}
\begin{equation}
\begin{pmatrix} 0 & -i \\
 i &  0 \end{pmatrix}
\end{equation}
is coded as:
\begin{verbatim}
\begin{equation}
\begin{pmatrix} 0 & -i \\
 i &  0 \end{pmatrix}
\end{equation}
\end{verbatim}

\noindent {\bf{A matrix with square brackets}}
\begin{equation}
\begin{bmatrix} 0 & -1 \\ 
1 &  0 \end{bmatrix}
\end{equation}
is coded as:
\begin{verbatim}
\begin{equation}
\begin{bmatrix} 0 & -1 \\ 
1 &  0 \end{bmatrix}
\end{equation}
\end{verbatim}

\noindent {\bf{A matrix with curly braces}}
\begin{equation}
\begin{Bmatrix} 1 &  0 \\ 
0 & -1 \end{Bmatrix}
\end{equation}
is coded as:
\begin{verbatim}
\begin{equation}
\begin{Bmatrix} 1 &  0 \\ 
0 & -1 \end{Bmatrix}
\end{equation}\end{verbatim}

\noindent {\bf{A matrix with single verticals}}
\begin{equation}
\begin{vmatrix} a &  b \\ 
c &  d \end{vmatrix}
\end{equation}
is coded as:
\begin{verbatim}
\begin{equation}
\begin{vmatrix} a &  b \\ 
c &  d \end{vmatrix}
\end{equation}\end{verbatim}

\noindent {\bf{A matrix with double verticals}}
\begin{equation}
\begin{Vmatrix} i &  0 \\ 
0 & -i \end{Vmatrix}
\end{equation}
is coded as:
\begin{verbatim}
\begin{equation}
\begin{Vmatrix} i &  0 \\ 
0 & -i \end{Vmatrix}
\end{equation}\end{verbatim}

\subsection{Arrays}
The {\tt{array}} environment allows you some options for matrix-like equations. You will have to manually key the fences, but there are other options for alignment of the columns and for setting horizontal and vertical rules. The argument to {\tt{array}} controls alignment and placement of vertical rules.

A simple array
\begin{equation}
\left(
\begin{array}{cccc}
a+b+c & uv & x-y & 27\\
a+b & u+v & z & 134
\end{array}\right)
\end{equation}
is coded as:
\begin{verbatim}
\begin{equation}
\left(
\begin{array}{cccc}
a+b+c & uv & x-y & 27\\
a+b & u+v & z & 134
\end{array} \right)
\end{equation}
\end{verbatim}

A slight variation on this to better align the numbers in the last column
\begin{equation}
\left(
\begin{array}{cccr}
a+b+c & uv & x-y & 27\\
a+b & u+v & z & 134
\end{array}\right)
\end{equation}
is coded as:
\begin{verbatim}
\begin{equation}
\left(
\begin{array}{cccr}
a+b+c & uv & x-y & 27\\
a+b & u+v & z & 134
\end{array} \right)
\end{equation}
\end{verbatim}

An array with vertical and horizontal rules
\begin{equation}
\left( \begin{array}{c|c|c|r}
a+b+c & uv & x-y & 27\\ \hline
a+b & u+v & z & 134
\end{array}\right)
\end{equation}
is coded as:
\begin{verbatim}
\begin{equation}
\left(
\begin{array}{c|c|c|r}
a+b+c & uv & x-y & 27\\
a+b & u+v & z & 134
\end{array} \right)
\end{equation}
\end{verbatim}
Note the argument now has the pipe "$\vert$" included to indicate the placement of the vertical rules.


\subsection{Cases Structures}
Many times cases can be miscoded using the wrong environment, i.e., {\tt{array}}. Using the {\tt{cases}} environment will save keystrokes (from not having to type the $\backslash${\tt{left}}$\backslash${\tt{lbrace}}) and automatically provide the correct column alignment.
\begin{equation*}
{z_m(t)} = \begin{cases}
1,&{\text{if}}\ {\beta }_m(t) \\ 
{0,}&{\text{otherwise.}} 
\end{cases}
\end{equation*}
\noindent is coded as follows:
\begin{verbatim}
\begin{equation*}
{z_m(t)} = 
\begin{cases}
1,&{\text{if}}\ {\beta }_m(t),\\ 
{0,}&{\text{otherwise.}} 
\end{cases}
\end{equation*}
\end{verbatim}
\noindent Note that the ``\&'' is used to mark the tabular alignment. This is important to get  proper column alignment. Do not use $\backslash${\tt{quad}} or other fixed spaces to try and align the columns. Also, note the use of the $\backslash${\tt{text}} macro for text elements such as ``if'' and ``otherwise.''

\subsection{Function Formatting in Equations}
Often, there is an easy way to properly format most common functions. Use of the $\backslash$ in front of the function name will in most cases, provide the correct formatting. When this does not work, the following example provides a solution using the $\backslash${\tt{text}} macro:

\begin{equation*} 
  d_{R}^{KM} = \underset {d_{l}^{KM}} {\text{arg min}} \{ d_{1}^{KM},\ldots,d_{6}^{KM}\}.
\end{equation*}

\noindent is coded as follows:
\begin{verbatim}
\begin{equation*} 
 d_{R}^{KM} = \underset {d_{l}^{KM}} 
 {\text{arg min}} \{ d_{1}^{KM},
 \ldots,d_{6}^{KM}\}.
\end{equation*}
\end{verbatim}

\subsection{ Text Acronyms Inside Equations}
This example shows where the acronym ``MSE" is coded using $\backslash${\tt{text\{\}}} to match how it appears in the text.

\begin{equation*}
 \text{MSE} = \frac {1}{n}\sum _{i=1}^{n}(Y_{i} - \hat {Y_{i}})^{2}
\end{equation*}

\begin{verbatim}
\begin{equation*}
 \text{MSE} = \frac {1}{n}\sum _{i=1}^{n}
(Y_{i} - \hat {Y_{i}})^{2}
\end{equation*}
\end{verbatim}

\section{Conclusion}
The conclusion goes here.


\section*{Acknowledgments}
This should be a simple paragraph before the References to thank those individuals and institutions who have supported your work on this article.



{




