
\subsection{Simulated perceptual noise analysis}
\label{sec:analysis}
With 6 target images as the references, as shown in study program file (Ref~\cite{userStudyCode}), we compute the loss of mean-squared error (IMMSE) for image-wise noise level, together with perception-based image quality evaluator (PIQE) \cite{venkatanath2015blind} and HDR-VDP \cite{mantiuk05PredVisDiff} for perceptual quality. 
For a uniform \emph{lower-means-better} error metric across all conditions, we inverted the HDR-VDP metric scores, whose otherwise original value is positively correlated to the perceptual image quality. For a fair comparison among different image conditions, we normalized the error with the BASELINE condition (the dashed horizontal line). 

The results are visualized in \Cref{fig:evaluation:metric}. Among all metrics, both OURS and PSF-ONLY conditions demonstrated significantly lower error than BASELINE: both are $95\%+$ lower than BASELINE/FOVEATION-ONLY with IMMSE. The trend is consistent with perceptual metrics: OURS showed $22.9\%$/$65.4\%$ lower PIQE/HDR-VDP error. The FOVEATION-ONLY condition, however, did not show a significant difference from BASELINE. Meanwhile, OURS and PSF-ONLY are noticeably similar with all three metrics (max $8.8\%$ difference). 
We note that quantitative metrics show slightly higher error for OURS compared to the PSF-only condition as the metrics are evaluated over all image pixels evenly. However, we argue that standard image metrics are not the best way of evaluating our technique as they do not fully capture the human visual system properties in the peripheral vision.
Evaluating foveal and peripheral regions separately is challenging due to continuous visual acuity fall-off. Precisely evaluating the perceived quality and noise level with a novel vision-aware metric is an exciting future work while orthogonal to the scope of the current work. 
An alternative and a more direct evaluation method are perceptual experiments. Therefore, we conducted perceptual evaluation which is more important for perceived display image quality.

\subsection{User Study}
\label{sec:study}

To study the perceptual image quality of the end-user, we performed a subjective evaluation with all conditions.
We used remote experimental studies following the safety protocols due to the restrictions imposed by COVID-19. The programmed user studies were sent to the users in a zip package and written instructions were provided to the users for setting up the experiment. One of the authors also monitored the remote experiment via video conference.
The code used for our user evaluations is attached as the Supplementary material with this manuscript.

\paragraph{Task and Stimuli}
Our perceptual study program required the users to input the size and resolution of their display monitor, and the program automatically computed the eye-to-display viewing distance to be maintained, as well as scaled the stimuli images in order to maintain a fixed field of view for all the users. Maintaining the appropriate viewing distance, the users kept their head fixed and one of their eyes covered. The stimuli images were placed at the center of the display. A green cross was shown throughout to help with gaze fixation. For each trial, a reference image was shown followed by pairs of stimuli (temporal interleaving).
Specifically, we used $6$ ($1080\times1080$) different target images and their corresponding reconstructed images from the four test conditions: OURS, FOVEATION-ONLY, BASELINE, and PSF-ONLY. 
The stimuli were shown as images on screen while the subjects were instructed to remain seated and gaze at the image center during the whole study. 

The task was a two-alternative-choice (2AFC). 
Before the experiment of each target image, the subjects were shown ground truth original images for 5 seconds and were asked to examine it as the quality reference. A pair of images were then presented for 1 second each as generated by any two of the four test conditions described. The order of the images presented was randomized. 
The 2AFC task for each trial was to choose the image with less perceived noise compared to the reference target which was then followed by the next trial.
The subjects used the keyboard to record their responses.
Each experiment consisted of $36$ trials, $6$ per target image. Twelve users (mean age $24.1$, $3$ females) participated in the study. All had a normal or corrected-to-normal vision. None of the subjects were aware of the number of conditions, our technique, and the goal of the study. The experiment was conducted with pre-computed test images distributed to users to adhere to the ongoing COVID-19 restrictions.

\paragraph{Results}
\Cref{fig:evaluation:study} shows the study results. Comparing with all other conditions, the majority of users chose OURS as the one with less noise. By observing OURS vs PSF-ONLY between \Cref{sec:analysis} and this study, we discovered the significant difference between objective metrics and higher-order end-user perception. That is, the foveation effect in OURS introduces strong perceptual quality gain despite the subtle numerical differences. The result demonstrates the perceptual benefits of both foveation (by comparing with PSF-ONLY) and the introduction of PSF (by comparing with FOVEATION-ONLY) in the optimization.

\begin{figure}[bht]
    \centering
    \subfloat[objective metrics]{\includegraphics[width=0.96\linewidth]{Figures/evaluation_new.jpg}\label{fig:evaluation:metric}}
    
    \subfloat[user study]{\includegraphics[width=0.85\linewidth]{Figures/study_new.jpg}\label{fig:evaluation:study}}
    \caption{Evaluation results of our method. (a) shows the objective errors described in  \protect\Cref{sec:analysis}. Lower error values mean better quality. (b) plots the users' 2AFC choice percentage as being ``less noisy'' from the study in \protect\Cref{sec:study}.}
    \label{fig:evaluation}
\end{figure}


\paragraph{Ensuring Gaze Fixation without Eye Tracker}
In principle, the best evaluation of our method requires an eye tracker to account for dynamic gaze changes that occur naturally~\cite{lu2020improved}. However, we argue that even without it, our evaluation is valid. This is because the stimuli images have noise in the periphery, and any deviation of the participants’ gaze from the assumed fovea location (green cross) would indeed be disadvantageous for our technique. In practice, such a situation would lead to lower metric scores of our method due to visible noise in regions which were assumed to lie in the periphery. The higher subjective scores for our method, as described below, only confirm that the users maintained gaze fixation and the experiments were conducted appropriately.


\begin{figure*}[ht!]
\centering
    \includegraphics[width=0.98\linewidth]{Figures/gauss_opt_retina_recon.jpg}
    \caption{Holograms optimized using existing methods~\cite{chakravarthula2019wirtinger,peng2020neural} do not consider the speckle noise caused by retinal PSF and hence the perceived images are noisy. However, optimizing for individual PSFs which might differ from person-to-person and even from left to right eye is challenging. Here we validate that optimizing hologram phase with a Gaussian PSF consistently suppresses retinal speckle noise and is a reasonable generalization.}
    \label{fig:gauss_opt_retina_recon}
\end{figure*}


\begin{table}[htbp]
	\centering 
	\caption{Quantitative metrics computed on a custom dataset with randomly picked images. It can be observed that optimizing for a Gaussian PSF but reconstructing with a real retinal PSF produces results on par with optimizing directly for the real retinal PSF.} 
	\label{table:psnr_metrics} 
	\begin{tabular}{c|c|c}  
		& & \\[-6pt]  
		 & PSNR & SSIM \\ 
		\hline
		& & \\[-6pt]  
		Baseline & 22.28 & 0.71 \\ 
		\hline
		& & \\[-6pt]  
		Direct retinal optimization & 35.55 & 0.95 \\
		\hline
		& & \\[-6pt]  
		Gaussian PSF optimization & 31.6 & 0.91 \\
		\hline
	\end{tabular}
\end{table}




\subsection{Holographic Displays}
Holographic displays are gaining increasing interest as the ultimate display technology, promising compact 3D displays with per-pixel focus control and aberration correction, which are essential features especially for future immersive near-eye displays. Most holographic display designs rely on a single phase-only spatial light modulator (SLM)~\cite{chen2015improved}, although configurations using two phase SLMs~\cite{levin2016passive,chakravarthula2019wirtinger,choi2021optimizing}, or a combination of both amplitude and phase SLMs~\cite{shi2017near} are also explored. While typically holographic displays have been large and bulky benchtop setups, recent works have shown path to miniaturization of such displays via holographic optics HOEs~\cite{maimone2017holographic, li2016holographic, jang2019holographic} or waveguides~\cite{yeom20153d}. Two critical limitations of existing holographic near-eye displays are the tiny eyebox and poor image quality. Several recent works have achieved an increased eyebox size by using eyetracking~\cite{jang2017retinal, jang2019holographic,xia2020towards}. While the display design plays a major role in the final image quality, existing designs only differ in the specific implementation details. In contrast, the algorithms used for generating the holograms, which restrict the achievable image quality, remain the same or similar across the displays. However, recent optimization and deep learning approaches have shown significant algorithmic improvements in image quality~\cite{chakravarthula2019wirtinger,chakravarthula2020learned,Shi2021:Nature,peng2020neural}.

\begin{figure*}[thb]
\centering
    \includegraphics[width=0.8\linewidth]{Figures/concept.jpg}
    \caption{Speckle noise due to PSF on the image plane: [Top] Simulated holographic reconstructions as perceived by a human observer when there is no overlap in the image plane (complex amplitude) pixels. [Bottom] Simulated holographic reconstructions as perceived by a human observer when there is an overlap in the image plane (complex amplitude) pixels. Overlapping (complex amplitude) image pixels with random phase result in interference and causes speckle noise. [Inset] Visualization of Airy disk for a circular aperture.}
    \label{fig:speckle}
\end{figure*}

\subsection{Limitations in Holographic Display Setups}

In this section, we briefly review holographic display configurations and the most important limitations that restrict image quality in holographic displays. Note that owing to these several factors deteriorating image quality in holographic displays, we are unable to separate the effects of speckle noise due to retinal PSF on a real holographic display. Therefore, our user evaluations are conducted on an emulated holographic display. 

\paragraph{SLM technology}
Phase-only SLMs are often preferred for computer generated holography due to their higher diffraction efficiency. Achieving accurate, complex modulation of both phase and amplitude with a single SLM device still remains an open problem~\cite{reichelt2012full}. However, these phase-only SLMs require a trillion sub-wavelength sized pixels to display holograms comparable to conventional displays. Unfortunately, existing SLMs only have resolutions ranging up to $3840 \times 2160$ (4K UHD) with pixel pitches limited to approximately $\unit[4]{\mu m}$ and fill factors less than $95\%$. While decreasing pixel pitches result in increasing maximum diffraction angle and thus field of view, it also causes a significant drop in the image quality. The space for electronics between the active pixel areas further leads to zero-order undiffracted light which often causes severe artifacts in the holographic images.

\paragraph{Phase wrapping and quantization errors}
Most existing SLMs can only present quantized phase modulation with a limited bit-depth. The CGH phase patterns are typically wrapped to lie within $[0, 2\pi]$, which are further quantized to the bit-depth of the SLM. The phase errors caused due to quantization and wrapping operations significantly deteriorate the quality of holographic images~\cite{dallas1972phase}. This quantized digital modulation of the SLM pixels correspond to a phase modulation of the incident light via a calibrated lookup table (LUT), which determines the phase modulation accuracy and diffraction efficiency of the SLM. Any errors in the LUTs cause non-linear phase modulation which further degrade the display quality.

\paragraph{Coherent Laser Speckle}
Illumination sources such as single-mode lasers have a large coherence length and produce coherent speckle noise. Many works aim at reducing speckle by using rotating diffusers~\cite{bianco2016quasi}, modulating or quickly repositioning the laser beam~\cite{kang2008effective} or superposition of multiple reconstructions~\cite{golan2009speckle}. While such coherence speckle noise can also be mitigated by using partially-coherent light sources, they result in blur or loss of depth perception~\cite{dainty1977statistics}. For a comprehensive discussion on many techniques to reduce speckle noise in CGH, we refer the reader to Bianco et al.~\cite{bianco2018strategies}. However, note that effective holographic noise suppression still remains an open problem.

\subsection{Iterative Algorithms for CGH} 
Holography for displays rely on diffraction and interference of light for generating imagery. Using a phase-only SLMs requires computing phase-only holograms that are capable of projecting imagery that can closely mimic the target image. The phase retrieval problem is generally non-convex and ill-posed. Early methods to phase retrieval include error reduction using iterative optimization~\cite{lesem1969kinoform,gerchberg1972practical} and hybrid input-output (HIO) methods~\cite{fienup1982phase, bauschke2003hybrid}. First-order non-linear optimization~\cite{gonsalves1976phase,lane1991phase,fienup1993phase}, alternative direction methods for phase retrieval~\cite{wen2012alternating,marchesini2016alternating}, non-convex optimization~\cite{zhang20173d}, and methods overcoming the non-convex nature of the phase retrieval problem by lifting; i.e., relaxation, to a semidefinite~\cite{candes2013phaselift} or linear program~\cite{goldstein2018phasemax,bahmaniPhaseMax} are also explored. Recent optimization methods for holographic phase retrieval utilize Wirtinger gradients and stochastic gradient descent methods to compute high-quality holograms with flexible loss functions~\cite{chakravarthula2019wirtinger, chakravarthula2020computing,peng2020neural}. Deep learning based methods are also gaining interest among researchers for their ability to predict high-quality phase-only holograms~\cite{peng2020neural, chakravarthula2020computing,Shi2021:Nature,eybposh2020deepcgh}. We refer the reader to Barbastathis et. al~\shortcite{barbastathis2019use} for an overview of learned phase retrieval methods. 

All of these methods have in common that they assume a perfect image formation model and ignore deviations from the perfect forward model in the human eye. The proposed method addresses this very important but under-investigated aspect of holographic displays.

\section{Related Work}
\input{Sections/Relatedwork}

\section{Computer Generated Holography}
\input{Sections/Preliminary}

\section{Retinal Sampling Model and the Point Spread Function of the Human Eye}

\input{Sections/PSFmodel}

\section{Method}
\input{Sections/Method}


\section{Evaluation}
\input{Sections/Evaluation}

\section{Discussion and Conclusion}
\input{Sections/Conclusion}


\acknowledgments{
This work is supported by the European Research Council (ERC) under the European Union’s Horizon 2020 research and innovation program (grant agreement N$^{\circ}$ 804226 – PERDY), NSF grants 1840131 and 1405847, and a generous gift from Intel.}

\bibliographystyle{abbrv}

