\section{Introduction}
\label{sec.introduction}
\IEEEPARstart{P}{ositron} emission tomography (PET) is an imaging modality widely used in oncology, neurology, and cardiology studies  \cite{sweet1951uses, cherry2018total, fletcher2008recommendations, beyer2000combined, di2007clinical, el2011improvement}.  It can observe molecular-level activities {\it{in vivo}} through the injection of specifically designed radioactive tracers.  Due to various physical degradation factors and limited counts received,  the signal-to-noise ratio (SNR) of PET is inferior to other imaging modalities, which comprises its clinical values in diagnosis, prognosis, staging and treatment monitoring.  Further enhancing PET image quality is essential to improve its diagnosis and treatment-monitoring accuracy.  

For the past decades, various post-processing methods have been investigated to further improve PET image quality. Gaussian filtering is widely used in clinical scanners to reduce the noise but it can also smooth out image structures. Non-local mean  \cite{dutta2013non}, wavelet \cite{boussion2009incorporation}, highly constrained back-projection processing \cite{christian2010dynamic,bevington20214}, and guided filtering \cite{yan2015mri} have been proposed to further preserve image details based on pre-defined filters or similarity calculation.  Recently,  the convolutional neural network (CNN)-based approaches have become a dominant trend in various PET research topics such as image denoising \cite{xiang2017deep,gong2018pet,kaplan2019full,cui2019pet,chen2019ultra,hashimoto2019dynamic,da2020micro,schramm2021approximating,mehranian2022image,daveau2022deep} and image reconstruction \cite{kim2018penalized,gong2018iterative,gong2018petre,haggstrom2019deeppet,lim2020improved,mehranian2020model},  due to their better performance than other state-of-the-art methods.  The CNN-based Unet architecture \cite{ronneberger2015u} is the most widely used network architecture currently.  Performance of CNN for PET image denoising can be further improved by different loss-function designs,  e.g.,  a perceptual loss \cite{gong2018pet,ouyang2019ultra,hu2020dpir} formed by a pre-trained network or a learned discriminative loss from the additional discriminator network \cite{goodfellow2014generative,ouyang2019ultra,lei2019whole,zhou2020supervised,song2020pet,hu2020dpir,sanaat2021deep,xue2022cross}.

One pitfall of CNN is that it specifically focuses on local spatial information and thus has a limited receptive field.  The multi-head self-attention (MSA)-based structure,  i.e., transformer,  has the ability to capture long-range information \cite{vaswani2017attention, dosovitskiy2020image, han2022survey}.  Based on the aggregation of tokens, global MSAs were first developed for natural language processing \cite{vaswani2017attention}.  Vision Transformer (ViT) was then proposed where global MSAs were successfully applied to sequences of spatial image patches for image classification  \cite{dosovitskiy2020image}.  Due to global MSA's quadratically growing computational complexity along the spatial dimension,  Swin Transformer \cite{liu2021swin} was proposed to efficiently calculate local MSAs using shifted windows that offered linear computational complexity.  Swin Transformer calculated local MSAs on multiple stages and achieved better classification performance than global MSAs. However, local MSAs of Swin Transformer is still limited regarding the receptive field of MSA. Restormer \cite{zamir2021restormer} was recently proposed to efficiently compute global MSAs along the channel dimension with linear computational complexity for image restoration tasks.  Regarding medical imaging, various transformer-based architectures were investigated for medical image segmentation \cite{chen2021transunet,  xie2021cotr, wang2021transbts, zhang2021transfuse, hatamizadeh2022unetr, chen2021transmorph}.  For PET image denoising, transformer-based architectures have not yet been fully investigated.

\begin{figure}[!t]
	\centerline{\includegraphics[width=\columnwidth]{figure_proposed_3.eps}}
	\caption{A brief overview of Spach Transformer for PET image denoising.}
	\label{fig.proposed}
\end{figure}
\begin{figure*}[!t]
	\centerline{\includegraphics[width=17cm]{figure_proposed_2.eps}}
	\caption{Diagram of the proposed Spach Transformer for PET image denoising.  Details of the channel-wise transformer block and the spatial-wise transformer block are given in Fig.~\ref{fig.sw} and Fig.~\ref{fig.cw}, respectively.}
	\label{fig.proposed2}
\end{figure*}

In this work, we proposed a {\it{spa}}tial and {\it{ch}}annel-wise encoder-decoder {\it{transformer}},  denoted as {\it{Spach Transformer}}, that can leverage spatial and channel information from oncological PET images.  The proposed Spach Transformer includes spatial-wise and channel-wise transformer blocks,  which can efficiently perform image denoising while also preserving image details, e.g., small lesion uptakes.  The spatial-wise block is based on local attention and the channel-wise block is based on global attention.  As the channel-wise transformer block may ignore the spatial information, a gated-conv feed-forward network (GCFN) block was further designed to incorporate spatial attributes between layers in the channel-wise transformer block.  A brief overview of the proposed Spach Transformer is presented in Fig. \ref{fig.proposed}.  It encoded the input and built two deep latent features in spatial and channel directions.  The decoder further took the two latent features concatenated as input to generate the final denoised PET image based on skip connections. To evaluate the effectiveness of the proposed Spach Transformer for PET image denoising,  clinical $^{18}$F-FDG,  $^{18}$F-ACBC,  $^{18}$F-DCFPyL, and $^{68}$Ga-DOTATATE datasets were utilized to quantitatively compare the performance of different CNN and transformer-based methods. 

The main contributions of this work include : (1) proposing a novel spatial and channel-wise encoder-decoder transformer integrating local and global attentions for PET image denoising and tumor-uptake preserving; (2)  designing a GCFN block to incorporate spatial attributes in the channel-wise transformer block to boost the performance; (3) evaluations using clinical datasets based on different PET tracers. This paper is organized as follows.  Section 2 presents the related works and the proposed Spach Transformer in detail.  Experiments, results and discussions are shown in Sections 3, 4 and 5, respectively.  Finally,  conclusions are drawn in Section 5.


\section{Method} \label{sec.proposed}
Fig. \ref{fig.proposed2} shows the detailed diagram of the proposed Spach Transformer.  The encoder part of Spach Transformer consists of {\color{black}two levels of spatial-wise transformer blocks and four levels of channel-wise transformer blocks.}  The decoder part of Spach Transformer includes {\color{black}four levels of channel-wise transformer blocks}.  Below we will describe major elements of the Spach Transformer (Sec.~\ref{sec.proposed}) as well as details of the spatial-wise transformer block (Sec.~\ref{sec.spatial}) and the channel-wise transformer block. (Sec.~\ref{sec.channel}).  

\subsection{Spach Transformer: Spatial and Channel-wise Transformer} \label{sec.proposed}
Given a low-dose PET image ${\bf X} \in \mathbb{R}^{H\times W\times D\times 1}$, Spach Transformer first obtained low-level feature embeddings ${\bf F}_0 \in \mathbb{R}^{H\times W\times D\times C}$ based on a $3\times 3\times 3$ convolution, where $H$, $W$, and $D$ indicate the voxel dimension and $C$ is the number of channels.  The four layers of the channel-wise encoder transformer blocks further processed the embedding  ${\bf F}_0$ to extract deep latent features  ${\bf F}_{l_c}$. 

Addtionally,  following \cite{dosovitskiy2020image,liu2021swin},  a patch partitioning module produced non-overlapping patches $\mathbf{x}_{i}\in \mathbb{R}^{P^3}$ from the low-dose PET image ${\bf X}$, where $P$ is the patch size and was set to 4 in our implementation.  A linear embedding layer was added to project these non-overlapping voxels to an arbitrary dimension as follows:
\begin{equation} 
	\mathbf{z}_{0} =[\mathbf{x}_{0}\mathbf{E} ;\mathbf{x}_{1}\mathbf{E} ;\dotsc ;\mathbf{x}_{N_p}\mathbf{E}] \in \mathbb{R}^{N_{P} \times d},
\end{equation}
where $\mathbf{E} \in \mathbb{R}^{P^3 \times d}$ is a projection matrix, and 
$\displaystyle N_{P} =\tfrac{H}{P} \times \tfrac{W}{P} \times \tfrac{D}{P}$ is the number of patches. 
The spatial-wise encoder transformer blocks and the patch-merging layer used the tokens $\mathbf{z}_{0}$ to extract deep latent features  ${\bf F}_{l_s}$.  Role of the patch-merging layer is to reduce the spatial size by half along each dimension.  

The encoder part described above gradually reduced the spatial size while expanding the channel capacity. The final latent features were generated by concatenating the latent features from the two transformer paths as  $ \mathbf{F}_{l} = \left[\mathbf{F}_{l_{c}} ,\mathbf{F}_{l_{s}}\right]$.  The final latent features $\mathbf{F}_{l}$ were further supplied to the decoder path to generate the high-quality PET image $\hat{\mathbf{X}}$.   Following \cite{zamir2021restormer},  a $1 \times 1 \times 1$ convolution and a concatenation operation was utilized to reduce the channel size by half at each decoder layer, followed by a channel-wise transformer block to recover the spatial information. The deep features  $\mathbf{F}_{d}$ were further refined to obtain robust features $\mathbf{F}_{r}$ at the PET image resolution scale.  Finally,  a $3 \times 3 \times 3$ convolution layer was applied on $\mathbf{F}_{r}$ to produce the residual image $\mathbf{R}$,  and the final denoised PET image was generated as $\hat{\mathbf{X}} = \mathbf{R} + \mathbf{X}$.
 
\subsection{Spatial-wise Transformer Block} \label{sec.spatial}
Swin Transformer \cite{liu2021swin} proposed to use window-based and shift window-based MSA modules to replace the standard MSA modules.  Fig. \ref{fig.sw_block} shows the diagram of the transformer block,  which is based on the following operations:
\begin{equation}\label{eq.swin}
	\begin{aligned}
		\hat{\mathbf{z}}_{l} 	&=MSA_W\left(LN(\mathbf{z}_{l-1})\right) +\mathbf{z}_{l-1}, 		\\
		\mathbf{z}_{l} 			&=MLP\left(LN(\hat{\mathbf{z}}_{l})\right) +\hat{\mathbf{z}}_{l}, \\
		\hat{\mathbf{z}}_{l+1} 	&=MSA_{SW}\left(LN(\mathbf{z}_{l})\right) +\mathbf{z}_{l}, 			\\
		\mathbf{z}_{l+1} 		&=MLP\left(LN(\hat{\mathbf{z}}_{l+1})\right) +\hat{\mathbf{z}}_{l+1},
	\end{aligned}
\end{equation}
where $MSA_W$ and $MSA_{SW}$ are regular window-based MSA (W-MSA) and shifted window-based MSA (SW-MSA),  respectively,  $MLP$ is the multi-layer perceptron (MLP) module with the Gaussian error linear unit (GELU) \cite{hendrycks2016gaussian} as the activation layer,  and $LN$ is the layer normalization (LN) module \cite{ba2016layer} which was inserted before the W-MAS, SW-MSA and MLP modules.  Residual connection was used after each module.  Fig. ~\ref{fig.sw_sa} shows the window-based self-attention diagram.  The self-attention was calculated as  \cite{liu2021swin} :
\begin{equation}
	\text{Attention}(\mathbf{Q}_s ,\mathbf{K}_s ,\mathbf{V}_s) =\text{Softmax}\left(\mathbf{Q}_s \mathbf{K}_s^{T} /\sqrt{d} +\mathbf{B}\right)\mathbf{V}_s,
\end{equation}
where $\mathbf{B}\in \mathbb{R}^{M^3 \times M^3}$ is the relative position of tokens in each window, $d$ is the query/key dimension. and $M^3$ is the number of patches in the 3D image window. The query $\mathbf{Q}_s$, key $\mathbf{K}_s$,  and value $\mathbf{V}_s$ matrices were computed as:
\begin{equation}
	\mathbf{Q}_s =W_{l}^{Q}\mathbf{Y} ,\mathbf{K}_s =W_{l}^{K} \mathbf{Y} ,\mathbf{V}_s =W_{l}^{V}  \mathbf{Y},
\end{equation}
where $W_{l}^Q\in \mathbb{R}^{M^3 \times M^3}$, $W_{l}^K\in \mathbb{R}^{M^3 \times M^3}$ , and $W_{l}^V\in \mathbb{R}^{M^3 \times M^3}$ are the linear projection matrices,   and $\mathbf{Y} \in \mathbb{R}^{M^3 \times d}$ is the input after the LN layer.  
\subsection{Channel-wise Transformer Block}\label{sec.channel}
The Restormer work \cite{zamir2021restormer} designed the multi-Dconv head transposed attention (MDTA) modules to replace the standard MSA modules,  which computed self-attention along the channel dimension instead of the spatial dimension.  This can allow a global attention calculation with linearly computational complexity.  In Restormer,  each transformer block included a MDTA and a gated-Dconv feed-forward network (GDFN) module.  Though Restormer showed competing performance in natural image enhancement,  based on our experiments,  we found that this configuration has the pitfall of losing local spatial information.  Apart from transferring local spatial information in parallel using the spatial-wise transformer blocks as described in Sec.~\ref{sec.spatial},  in the proposed channel-wise transformer block (see Fig. \ref{fig.cw_block}), we further designed a gated-conv feed-forward network (GCFN) to better capture local spatial information and thus improve the lesion-uptake recovery.  Below we will introduce the details of MDTA, GDFN and GCFN. 
\subsubsection{MDTA}
Fig. \ref{fig.cw_mdta} shows the diagram of the MDTA module.  Given an input feature map ${\bf F} \in \mathbb{R}^{\hat{H} \times \hat{W} \times \hat{D} \times \hat{C}}$, a layer normalization module was first applied to obtain $\mathbf{Y} \in \mathbb{R}^{\hat{H} \times \hat{W} \times \hat{D} \times \hat{C}}$. The query $\mathbf{Q}_c  \in \mathbb{R}^{\hat{H} \times \hat{W} \times \hat{D} \times \hat{C}}$,  key $\mathbf{K}_c \in \mathbb{R}^{\hat{H} \times \hat{W} \times \hat{D} \times \hat{C}}$,  and value $\mathbf{V}_c \in \mathbb{R}^{\hat{H} \times \hat{W} \times \hat{D} \times \hat{C}}$ matrices were then obtained after a $1 \times 1 \times 1$ pixel-wise convolution operation (encoding channel-wise context) and a $3 \times 3 \times 3$ channel-wise convolution operation (aggregating pixel-wise cross-channel context).  By reshaping operations, $\hat{\mathbf{Q}}_c \in \mathbb{R}^{\hat{H}  \hat{W} \hat{D} \times \hat{C}}$, 
$\hat{\mathbf{K}}_c \in \mathbb{R}^{\hat{C} \times \hat{H}  \hat{W} \hat{D}}$, and 
$\hat{\mathbf{V}}_c \in \mathbb{R}^{\hat{H}  \hat{W} \hat{D} \times \hat{C}}$  were  obtained from  $\mathbf{Q}_c$, $\mathbf{K}_c$, and $\mathbf{V}_c$, respectively, and were used to generate a transposed-attention map $\mathbf{A} \in \mathbb{R}^{\hat{C} \times \hat{C}}$ instead of the massive spatial attention map of size $  \hat{H}  \hat{W} \hat{D} \times  \hat{H}  \hat{W} \hat{D}$ \cite{vaswani2017attention, dosovitskiy2020image}. The output feature of the MDTA module $\hat{\mathbf{F}}$ can be expressed as
\begin{equation}
	\begin{aligned}
	& \hat{\mathbf{F}} =W_{p}\text{Attention}(\hat{\mathbf{Q}}_{c} ,\hat{\mathbf{K}}_{c} ,\hat{\mathbf{V}}_{c})+\mathbf{F},\\
	& \text{Attention}(\hat{\mathbf{Q}}_{c} ,\hat{\mathbf{K}}_{c} ,\hat{\mathbf{V}}_{c}) =\hat{\mathbf{V}}_{c} \cdot \text{Softmax}({\hat{\mathbf{Q}}_{c} \cdot }\hat{\mathbf{K}}_{c} / \alpha ),
	\end{aligned}
\end{equation}
where $\alpha$ is a learnable scaling parameter to control the magnitude of the dot product output ${\hat{\mathbf{Q}}_{c} \cdot }\hat{\mathbf{K}}_{c}$,  and $W_{p}$ is a linear projection matrix.  Note that the output of the self-attention operation was reshaped to have the same size as the input ${\bf F}$.

\begin{figure}[htp]
	\centering
	\begin{subfigure}[b]{\columnwidth}
		\centering
		\includegraphics[width=\columnwidth]{figure_transformer_blocks_1.eps}
		\caption{The spatial-wise transformer block.}
		\label{fig.sw_block}
	\end{subfigure}
	\hfill
	\begin{subfigure}[b]{\columnwidth}
		\centering
		\includegraphics[width=\columnwidth]{figure_transformer_blocks_2.eps}
		\caption{The window-based multi-head self-attention.}
		\label{fig.sw_sa}
	\end{subfigure}
	\caption{Diagrams of (a) the spatial-wise transformer block and (b) the window-based attention calculation. }
	\label{fig.sw}
\end{figure}
\begin{figure}[t]
	\centering
	\begin{subfigure}[b]{\columnwidth}
		\centerline{\includegraphics[width=5.5cm]{figure_transformer_blocks_3.eps}}
		\caption{The proposed channel-wise transformer block.}
		\label{fig.cw_block}
	\end{subfigure}
	\hfill
	\begin{subfigure}[b]{\columnwidth}
		\centerline{\includegraphics[width=\columnwidth]{figure_transformer_blocks_4.eps}}
		\caption{The multi-dconv head transposed attention (MDTA).}
		\label{fig.cw_mdta}
	\end{subfigure}
	\hfill
	\begin{subfigure}[b]{\columnwidth}
		\centerline{\includegraphics[width=7cm]{figure_transformer_blocks_5.eps}}
		\caption{The gated-dconv feed-forward network (GDFN).}
		\label{fig.cw_gdfn}
	\end{subfigure}
	\hfill
	\begin{subfigure}[b]{\columnwidth}
		\centerline{\includegraphics[width=7cm]{figure_transformer_blocks_6.eps}}
		\caption{The gated-conv feed-forward network (GCFN).}
		\label{fig.cw_gcfn}
	\end{subfigure}
	\caption{Diagrams of (a) the proposed channel-wise transformer block,  (b) the MDTA module, (c) the GDFN module and (d) the GCFN module.}
	\label{fig.cw}
\end{figure}
\begin{table}[t] \centering
	\caption{According to channel number change $C$, an ablation study of different model parameters of Spach Transformer (GCFN).}
	\label{tbl.param_abl}
	\begin{tabular}{|@{\hskip3pt}c@{\hskip3pt}|c@{\hskip3pt}|@{\hskip3pt}r@{\hskip3pt}|cc|cc|}
		\hline
		{\rule{0pt}{3ex}}\multirow{2}{*}{}           & \multirow{2}{*}{C} & \multirow{2}{*}{\begin{tabular}[c]{@{}c@{}}Model\\ Parameters\end{tabular}} & \multicolumn{2}{c|}{PSNR}             & \multicolumn{2}{c|}{SSIM}            \\ \cline{4-7} 
		{\rule{0pt}{3ex}}&                    &                                                                             & \multicolumn{1}{c|}{Avg}     & Std    & \multicolumn{1}{c|}{Avg}    & Std    \\ \hline
		{\rule{0pt}{3ex}}\multirow{5}{*}{\begin{tabular}[c]{@{}c@{}}Spach \\ Transformer\end{tabular}} & 6                  & 7,034,951                                                                   & \multicolumn{1}{c|}{54.9753} & 3.6359 & \multicolumn{1}{c|}{0.9367} & 0.0147 \\ \cline{2-7} 
		{\rule{0pt}{3ex}}& 12                 & 19,218,323                                                                  & \multicolumn{1}{c|}{55.0840} & 3.6092 & \multicolumn{1}{c|}{\textbf{0.9403}} & \textbf{0.0128} \\ \cline{2-7} 
		{\rule{0pt}{3ex}}& 24                 & 59,350,043                                                                  & \multicolumn{1}{c|}{55.0911} & 3.6103 & \multicolumn{1}{c|}{0.9358} & 0.0147 \\ \cline{2-7} 
		{\rule{0pt}{3ex}}& 36                 & 120,501,731                                                                 & \multicolumn{1}{c|}{55.1003} & 3.6193 & \multicolumn{1}{c|}{0.9379} & 0.0137 \\ \cline{2-7} 
		{\rule{0pt}{3ex}}& 48                 & 202,673,387                                                                 & \multicolumn{1}{c|}{\textbf{55.1443}} & \textbf{3.6036} & \multicolumn{1}{c|}{0.9368} & 0.0138 \\ \hline
	\end{tabular}
\end{table}
\begin{figure*}[t]
	\centering
	\begin{subfigure}[b]{\textwidth}
		\centering
		\includegraphics[width=0.9\textwidth]{figure_ablation_1.eps}
		\caption{The predicted images and the distance images of the whole body.}
		\label{fig.abl1_1}
	\end{subfigure}
	\hfill
	\begin{subfigure}[b]{\textwidth}
		\centering
		\includegraphics[width=0.9\textwidth]{figure_ablation_2.eps}
		\caption{The enlarged images where the blue rectangles are located in (a).}
		\label{fig.abl1_2}
	\end{subfigure}
	\caption{Coronal and zoomed-in views of the denoised PET images of Spach Transformer according to different numbers of channels $C$ based on an  $^{18}$F-DCFPyL dataset}
	\label{fig.abl1}
\end{figure*}


\begin{figure*}[t]
	\centering
	\begin{subfigure}[b]{\textwidth}
		\centering
		\includegraphics[width=0.9\textwidth]{figure_experiments_1.eps}
		\caption{The predicted images and the distance images of the whole body.}
		\label{fig.exp1_1}
	\end{subfigure}
	\hfill
	\begin{subfigure}[b]{\textwidth}
		\centering
		\includegraphics[width=0.9\textwidth]{figure_experiments_2.eps}
		\caption{The enlarged images where the blue rectangles are located in (a).}
		\label{fig.exp1_2}
	\end{subfigure}
	\caption{Coronal and zoomed-in views of the denoised PET results of different methods based on an  $^{18}$F-DCFPyL dataset}
	\label{fig.exp1}
\end{figure*}

\begin{figure*}[!h]
	\centering
	\begin{subfigure}[b]{\textwidth}
		\centering
		\includegraphics[width=0.9\textwidth]{figure_experiments_3.eps}
		\caption{The predicted images and the distance images of the whole body.}
		\label{fig.exp2_1}
	\end{subfigure}
	\hfill
	\begin{subfigure}[b]{\textwidth}
		\centering
		\includegraphics[width=0.9\textwidth]{figure_experiments_4.eps}
		\caption{The enlarged images where the blue rectangles are located in (a).}
		\label{fig.exp2_2}
	\end{subfigure}
	\caption{Coronal and zoomed-in views of the denoised PET results of the state-of-the-art methods based on an  $^{18}$F-FDG dataset}
	\label{fig.exp2}
\end{figure*}
\begin{figure*}[t]
	\centering
	\begin{subfigure}[b]{\textwidth}
		\centering
		\includegraphics[width=0.9\textwidth]{figure_experiments_5.eps}
		\caption{The predicted images and the distance images of the whole body.}
		\label{fig.exp3_1}
	\end{subfigure}
	\hfill
	\begin{subfigure}[b]{\textwidth}
		\centering
		\includegraphics[width=0.9\textwidth]{figure_experiments_6.eps}
		\caption{The enlarged images where the blue rectangles are located in (a).}
		\label{fig.exp3_2}
	\end{subfigure}
	\caption{Coronal and zoomed-in views of the denoised PET results of different methods based on an  $^{18}$F-ACBC dataset}
	\label{fig.exp3}
\end{figure*}

\begin{figure*}[!h]
	\centering
	\begin{subfigure}[b]{\textwidth}
		\centering
		\includegraphics[width=0.9\textwidth]{figure_experiments_7.eps}
		\caption{The predicted images and the distance images of the whole body.}
		\label{fig.exp4_1}
	\end{subfigure}
	\hfill
	\begin{subfigure}[b]{\textwidth}
		\centering
		\includegraphics[width=0.9\textwidth]{figure_experiments_8.eps}
		\caption{The enlarged images where the blue rectangles are located in (a).}
		\label{fig.exp4_2}
	\end{subfigure}
	\caption{Coronal and zoomed-in views of the denoised PET results of different methods based on a $^{68}$Ga DOTATATE dataset}
	\label{fig.exp4}
\end{figure*}



\subsubsection{GDFN}
{\color{black}A gating mechanism \cite{dauphin2017language} can help selectively pass useful information to the next layer in the network architecture.
Due to this, the gating network could suppress less informative features.
In \cite{zamir2021restormer}, the gating mechanism was utilized in the GDFN module  (see Fig. \ref{fig.cw_gdfn}), which was calculated as}
\begin{equation}
	\begin{aligned}
	& \hat{\mathbf{F}} =W_{p}^{0}\text{Gating}(\mathbf{F}) +\mathbf{F}, \\
	& \text{Gating}(\mathbf{F}) =\phi \left( W_{c}^{1} W_{p}^{1}{LN}(\mathbf{F})\right) \odot W_{c}^{2} W_{p}^{2}{LN}(\mathbf{F}).
	\end{aligned}
\end{equation}
Here ${\mathbf{F}}$ and $\hat{\mathbf{F}} $ are the input and output features, $\odot$ is an element-wise multiplication,  $\phi$ is the GELU non-linearity activation layer,  and $LN$ indicates the LN layer. $W_p^0$, $W_p^1$, and $W_p^2$ indicate the linear projection matrices.  $W_{c}^{1}$ and $W_{c}^{1}$ represent  $3 \times 3 \times 3$ channel-wise convolutions.
 
\subsubsection{GCFN}
Since the above MDTA and GDFN modules entirely focus on utilizing channel information for image denoising, 
it may lose spatial information.  To preserve spatial attributes between layers,  a gated-conv feed-forward network (GCFN) module (see Fig. \ref{fig.cw_gcfn}) was added to the channel-wise transformer block.  The only difference between GCFN and GDFN are that the $3 \times 3 \times 3$ channel-wise convolutions used in GDFN ($W_{c}^{1}$ and $W_{c}^{1}$) were replaced by $3 \times 3 \times 3$ spatial-wise convolutions ($W_{s}^{1}$ and $W_{s}^{1}$).  We hypothesized that the spatial-wise convolutions could better utilize spatial information and thus further help recovery lesion uptakes.







\section{Experiments}
\subsection{Datasets}

To evaluate performance of the proposed Spach Transformer,  clinical whole-body $^{18}$F-FDG,  $^{18}$F-ACBC,  $^{18}$F-DCFPyL,  and $^{68}$Ga-DOTATATE datasets acquired from the GE DMI PET/CT scanner were utilized.  These tracers have unique distributions in the organs and are currently the most widely used tracers in oncology PET studies.  For each dataset,  1/4 of the events were extracted from the listmode data to generate the 1/4 low-dose data,  which was reconstructed and supplied as the network input.  The corresponding normal-dose image was adopted as the training label.  Both the low-dose and normal-dose PET images were reconstructed based on the ordered subset expectation maximization (OSEM) algorithm (3 iterations and 17 subsets) with point spread function (PSF) and time of flight (TOF) modeling.  For the training set, 30  $^{18}$F-FDG and 30  $^{18}$F-ACBC datasets were included.  For the validation set,  4 $^{18}$F-FDG datasets were used.  For the testing set, 10  $^{18}$F-FDG, 10  $^{18}$F-ACBC,  10 $^{18}$F-DCFPyL,  and 18 $^{68}$Ga-DOTATATE datasets were employed.  Note that $^{18}$F-DCFPyL and  $^{68}$Ga-DOTATATE datasets were not included in the training set,  but in the testing set.  This is to evaluate the robustness of the deep learning structures to datasets of new tracers unseen during the training phase. The whole-body PET images had a dimension of $256 \times 256 \times 345$ and a voxel size of $2.73 \times 2.73 \times 2.8$ mm$^3$.  During network training, the input image was divided into $96 \times 96 \times 96$ cubes due to the GPU memory limit.



\subsection{Implementation and quantification}
The Unet \cite{ronneberger2015u},  Swin Transformer-based structure \cite{chen2021transmorph}, and Restormer \cite{zamir2021restormer} were adopted as the reference methods.  All the architectures utilized the Charbonnier loss \cite{barron2019general} as the training loss function.  For Spach Transformer,  the channel size of the transformer blocks can influence the network parameters a lot.  We have investigated the performance of the Spach Transformer with different channel sizes.  During network training, the AdamW optimizer was used and 300 epochs were run with the initial learning rate of 1e-5 gradually reduced to 1e-8 using cosine annealing \cite{loshchilov2016sgdr}.  The proposed network was implemented on the Nvidia Tesla V100 GPU.

As for quantification,  the peak signal-to-noise ratio (PSNR) and structural similarity index (SSIM) were utilized to quantify the global performance of different methods.  For oncological PET images,  preserving tumor uptake is essential for disease diagnosis and monitoring.  To evaluate the performance of different methods regarding tumor-uptake recovery,  the contrast-to-noise (CNR) was calculated at the tumor region as
\begin{equation}
\text{CNR} = (V_{\text{tumor}} - V_{\text{ref}})/{\sigma}_{\text{ref}}.
\end{equation}
Here $V_{\text{tumor}}$ is the mean value of the tumor region of interest (ROI). $V_{\text{ref}}$ and $\sigma_{\text{ref}}$ are the reference ROIs' mean value and standard deviation, respectively. The reference ROIs were selected from uniform regions inside the liver region.


\section{Results}\label{sec.rst}
\subsection{Effect of model parameters}\label{subsec.abl}
{\color{black}We want to first investigate the performance of Spach Transformer with different network parameters. The channel number used in the transformer architecture has a big impact on the model size and was varied to investigate the network performance with different network parameters.} Table~\ref{tbl.param_abl} shows the averaged PSNR and SSIM values for different model parameters of the Spach Transformer based on varying the channel size $C$ ($C=\{6, 12, 24, 36, 48\}$).  It can be observed that the best PSNR was achieved when $C=48$, while the best SSIM was obtained when $C=12$.  Fig. \ref{fig.abl1} shows the denoised PET images from one $^{18}$F-DCFPyL  dataset with different channel numbers.  To evaluate how close the predicted images are to the ground truth (GT), the images of $L2$-norm distance  between the predicted and GT images were also generated.  It can be observed that when $C=6$,  the denoised image is blurred and the uptake in the zoomed-in tumor region cannot be fully recovered.  When the channel number $C$ was increased to 12,  the network can better recover the tumor uptake and also has a better denoising performance.  After further increasing the network parameter,  performance of the network on tumor-uptake recovery increased a little and became stable.  Based on the PSNR and SSIM results as well as the image appearance,  the channel size $C$ was set to 12 in all our studies. The comparison methods were all set to have a similar number of network parameters as the Spach Transformer with $C=12$.









\begin{table*}[htp] \centering
	\caption{Quantitative comparison of PSNR with the state-of-the-art methods.}
	\label{tbl.psnr_sota}
	\begin{tabular}{|c|cc|cc|cc|cc|cc|}
		\hline
		{\rule{0pt}{3ex}}\multirow{3}{*}{Tracer}           & \multicolumn{2}{c|}{Unet}             & \multicolumn{2}{c|}{Swin Transformer} & \multicolumn{2}{c|}{Restormer}        & \multicolumn{2}{c|}{\begin{tabular}[c]{@{}c@{}}{\rule{0pt}{3ex}}Spach Transformer\\ w/o GCFN\end{tabular}} & \multicolumn{2}{c|}{Spach Transformer}                          \\ \cline{2-11} 
		{\rule{0pt}{3ex}}& \multicolumn{1}{c|}{Mean}    & Std    & \multicolumn{1}{c|}{Mean}    & Std    & \multicolumn{1}{c|}{Mean}    & Std    & \multicolumn{1}{c|}{Mean}                          & Std                          & \multicolumn{1}{c|}{Mean}             & Std             \\ \hline
		{\rule{0pt}{3ex}}$^{18}$F-DCFPyL  & \multicolumn{1}{c|}{53.9726} & 5.7870 & \multicolumn{1}{c|}{53.5862} & 5.7802 & \multicolumn{1}{c|}{53.8871} & 5.7728 & \multicolumn{1}{c|}{53.9818}                       & 5.7754                       & \multicolumn{1}{c|}{\textbf{54.0927}} & \textbf{5.7879} \\ \hline
		{\rule{0pt}{3ex}}$^{18}$F-FDG     & \multicolumn{1}{c|}{52.6028} & 3.1194 & \multicolumn{1}{c|}{52.1079} & 3.2607 & \multicolumn{1}{c|}{52.5813} & 3.0841 & \multicolumn{1}{c|}{52.6135}                       & 3.0513                       & \multicolumn{1}{c|}{\textbf{52.8678}} & \textbf{2.9585} \\ \hline
		{\rule{0pt}{3ex}}$^{18}$F-ACBC    & \multicolumn{1}{c|}{58.3996} & 7.0715 & \multicolumn{1}{c|}{57.6852} & 7.2052 & \multicolumn{1}{c|}{58.4845} & 7.0278 & \multicolumn{1}{c|}{58.6009}                       & 7.0026                       & \multicolumn{1}{c|}{\textbf{58.7500}} & \textbf{6.9883} \\ \hline
		{\rule{0pt}{3ex}}$^{68}$Ga DOTATATE   & \multicolumn{1}{c|}{54.0784} & 3.7722 & \multicolumn{1}{c|}{53.3199} & 3.8628 & \multicolumn{1}{c|}{54.4369} & 3.7322 & \multicolumn{1}{c|}{54.4289}                       & 3.7438                       & \multicolumn{1}{c|}{\textbf{54.8294}} & \textbf{3.6679} \\ \hline
		{\rule{0pt}{3ex}}Avg             & \multicolumn{1}{c|}{54.6492} & 5.3646 & \multicolumn{1}{c|}{54.0323} & 5.4194 & \multicolumn{1}{c|}{54.7790} & 5.3398 & \multicolumn{1}{c|}{54.8267}                       & 5.3439                       & \multicolumn{1}{c|}{\textbf{55.0840}} & \textbf{5.2985} \\ \hline \hline
		{\rule{0pt}{3ex}}Model Parameters & \multicolumn{2}{c|}{21,781,520}       & \multicolumn{2}{c|}{20,631,864}       & \multicolumn{2}{c|}{21,156,805}       & \multicolumn{2}{c|}{21,022,921}                                                   & \multicolumn{2}{c|}{19,218,323}                         \\ \hline
	\end{tabular}
\end{table*}
\begin{table*}[htp] \centering
	\caption{The SSIM values of different methods on  $^{18}$F-DCFPyL,  $^{18}$F-FDG,  $^{18}$F-ACBC and $^{68}$Ga DOTATATE datasets.}
	\label{tbl.ssim_sota}
	\begin{tabular}{|c|cc|cc|cc|cc|cc|}
		\hline
		{\rule{0pt}{3ex}}\multirow{4}{*}{Tracer} & \multicolumn{2}{c|}{Unet}            & \multicolumn{2}{c|}{Swin Transformer} & \multicolumn{2}{c|}{Restormer}       & \multicolumn{2}{c|}{\begin{tabular}[c]{@{}c@{}}{\rule{0pt}{3ex}}Spach Transformer\\ w/o GCFN\end{tabular}} & \multicolumn{2}{c|}{Spach Transformer}                         \\ \cline{2-11} 
		{\rule{0pt}{3ex}}& \multicolumn{1}{c|}{Mean}   & Std    & \multicolumn{1}{c|}{Mean}    & Std    & \multicolumn{1}{c|}{Mean}   & Std    & \multicolumn{1}{c|}{Mean}                          & Std                          & \multicolumn{1}{c|}{Mean}            & Std             \\ \hline
		{\rule{0pt}{3ex}}$^{18}$F-DCFPyL         & \multicolumn{1}{c|}{0.9234} & 0.0076 & \multicolumn{1}{c|}{0.9125}  & 0.0080 & \multicolumn{1}{c|}{0.9370} & 0.0078 & \multicolumn{1}{c|}{0.9369}                        & 0.0079                       & \multicolumn{1}{c|}{\textbf{0.9439}} & \textbf{0.0069} \\ \hline
		{\rule{0pt}{3ex}}$^{18}$F-FDG            & \multicolumn{1}{c|}{0.9364} & 0.0063 & \multicolumn{1}{c|}{0.9282}  & 0.0079 & \multicolumn{1}{c|}{0.9472} & 0.0075 & \multicolumn{1}{c|}{0.9477}                        & 0.0076                       & \multicolumn{1}{c|}{\textbf{0.9539}} & \textbf{0.0074} \\ \hline
		{\rule{0pt}{3ex}}$^{18}$F-ACBC           & \multicolumn{1}{c|}{0.8894} & 0.0059 & \multicolumn{1}{c|}{0.8759}  & 0.0052 & \multicolumn{1}{c|}{0.8955} & 0.0061 & \multicolumn{1}{c|}{0.8975}                        & 0.0059                       & \multicolumn{1}{c|}{\textbf{0.9138}} & \textbf{0.0048} \\ \hline
		{\rule{0pt}{3ex}}$^{68}$Ga-DOTATATE          & \multicolumn{1}{c|}{0.9121} & 0.0168 & \multicolumn{1}{c|}{0.9166}  & 0.0122 & \multicolumn{1}{c|}{0.9432} & 0.0119 & \multicolumn{1}{c|}{0.9417}                        & 0.0129                       & \multicolumn{1}{c|}{\textbf{0.9458}} & \textbf{0.0127} \\ \hline
		{\rule{0pt}{3ex}}Avg                    & \multicolumn{1}{c|}{0.9148} & 0.0196 & \multicolumn{1}{c|}{0.9097}  & 0.0204 & \multicolumn{1}{c|}{0.9328} & 0.0215 & \multicolumn{1}{c|}{0.9327}                        & 0.0208                       & \multicolumn{1}{c|}{\textbf{0.9404}} & \textbf{0.0169} \\ \hline
	\end{tabular}
	
	
\end{table*}
\begin{figure}[t!]
	\centering   \hspace{-0.2cm}
	\begin{subfigure}[h]{0.5\columnwidth}
		\centering
		\includegraphics[width=4.5cm]{figure_CNR_1.eps} 
		\caption{$^{18}$F-DCFPyL} 
		\label{fig.CNR1} 
	\end{subfigure}  \hspace{-0.2cm}
	\begin{subfigure}[h]{0.5\columnwidth} 
		\centering
		\includegraphics[width=4.5cm]{figure_CNR_2.eps}
		\caption{$^{18}$F-FDG}
		\label{fig.CNR2}
	\end{subfigure} 
	\hfill
	\hspace{-2.5cm}
	\begin{subfigure}[b]{0.5\columnwidth}  
		\centering
		\hspace{-0.6cm}\includegraphics[width=4.5cm]{figure_CNR_3.eps} \hspace{-0.6cm}
		\caption{$^{18}$F-ACBC}
		\label{fig.CNR3}
	\end{subfigure} \hspace{-0.3cm}
	\begin{subfigure}[b]{0.5\columnwidth}
		\centering
		\includegraphics[width=4.5cm]{figure_CNR_4.eps}
		\caption{$^{68}$Ga DOTATATE}
		\label{fig.CNR4}
	\end{subfigure} 
	\caption{The CNR values of different methods based on (a) $^{18}$F-DCFPyL, (b) $^{18}$F-FDG, (c) $^{18}$F-ACBC and (d) $^{68}$Ga DOTATATE datasets.}
	\label{fig.CNR}
\end{figure}


\subsection{Comparisons with reference methods}
Fig. \ref{fig.exp1}, \ref{fig.exp2}, \ref{fig.exp3}, and \ref{fig.exp4} present the coronal view of the results generated by the Spach Transformer and the reference methods based on one $^{18}$F-DCFPyL,  one $^{18}$F-FDG,  one $^{18}$F-ACBC, and  one $^{68}$Ga DOTATATE dataset. 
From Fig. \ref{fig.exp1_1}, \ref{fig.exp2_1}, \ref{fig.exp3_1}, and \ref{fig.exp4_1},  it can be observed that Restormer has better denoising performance than that of the Unet and Swin Transformer.   It can also be observed that the Unet and Swin Transformer can achieve relatively higher uptake values than that of the Restormer in the tumor regions.  This might be because the Unet and Swin Transformer focus more on spatial information, whereas the Restormer is highly dependent on the channel information.  As the Spach Transformer can utilize both spatial and channel information together, it can better preserve the tumor details while also achieving competitive denoising performance than other reference methods.


Table \ref{tbl.psnr_sota} and \ref{tbl.ssim_sota} show that the Spach Transformer achieved average PSNR of 55.0840 and aveerage SSIM of 0.9403, better than the Unet, Swin Transformer, Restormer, and Spach Transformer without GCFN. Compared to Restormer,  Spach Transformer achieved a PSNR gain of 0.305 and an SSIM gain of 0.008 because of the extra spatial information utilized.  Compared to Swin Transformer,  Spach Transformer achieved a PSNR gain of 1.052 and an SSIM gain of 0.031 because of the global channel information leveraged. 

To examine the local performance of different methods at the tumor regions,  we have further quantified the tumor CNR for datasets with confirmed tumors.  Fig. ~\ref{fig.CNR} shows the CNR results of all the methods for datasets with different tracers.  As observed from Fig. \ref{fig.exp1}, \ref{fig.exp2}, \ref{fig.exp3}, and \ref{fig.exp4},  the Restormer has worse performance regarding tumor-uptake preserving, which is confirmed by the CNR plot in Fig. \ref{fig.CNR}.  The Spach Transformer obtained better CNR results in the tumor regions than other reference methods because of the combination of spatial and channel information. 

\section{Discussion}
For PET image denoising,  one challenge is to reduce the noise while also preserving the tumor uptake.  The latest deep learning architectures (e.g., the Swin Transformer and the Restormer) have focused on utilizing spatial or channel information in an efficient way to improve the specific image-processing task.  In this work, we proposed a novel spatial and channel-wise transformer architecture, Spach Transformer, aiming to suppress the noise and at the same time preserving the image details.  Oncology PET datasets from different tracers were utilized to evaluate the performance of different network structures.

From the evaluations, we observed that the spatial information-oriented network architectures (e.g., the Unet and Swin Transformer) can highlight the tumor region well but the results are still noisy.  In contrast, the channel information-oriented network architecture (e.g., the Restormer) can achieve better denoising performance but failed to preserve the tumor uptakes, which can be observed through the CNR plot and the zoomed-in tumor regions of the coronal views.  The SSIM, PSNR and CNR quantification show that the proposed Spach Transformer can achieve the best performance,  demonstrating the benefits of combining both spatial and channel information.

Another thing to note is that all the network structures were trained using $^{18}$F-FDG and $^{18}$F-ACBC datasets,  while the testing datasets include two additional tracers, i.e.,  $^{18}$F-DCFPyL and  $^{68}$Ga DOTATATE, which were unseen during the training phase.  $^{18}$F-DCFPyL is a newly FDA approved prostate-specific membrane antigen (PSMA) tracer for prostate cancer diagnosis,  which does not have many cases available when we began this work. $^{18}$Ga-DOTATATE is for neuroendocrine tumor diagnosis and has worse image quality as compared to the $^{18}$F-based tracer.  Evaluations on datasets from these two tracers can help evaluate the robustness of the network to unseen data distributions and explore the possibility of deep learning-based cross-tracer applications.  Results show that all the networks have good performance on those unseen data distributions and the proposed Spach Transformer has the best performance.  Apart from cross-tracer testing,  further evaluating the performance of the networks on datasets from different scanners is one of our future tasks.  

\section{Conclusion}
In this work, we proposed a spatial and channel-wise transformer that utilized the spatial and channel information for PET image denoising. The proposed Spach Transformer achieved better performances in terms of CNR, PSNR, and SSIM on $^{18}$F-FDG,  $^{18}$F-ACBC,  $^{18}$F-DCFPyL, and $^{68}$Ga-DOTATATE datasets compared to other reference methods. Our future work will focus on further quantitative evaluations based more clinical datasets.
\balance
