# Motion-enhanced Holography

Zhenxing Dong1 1Department of Electronic Engineering, Shanghai Jiao Tong
University, Shanghai 200240, China
2State Key Lab of Advanced Optical Communication Systems and Networks,
Shanghai Jiao Tong University, Shanghai 200240, China
*Corresponding author<EMAIL_ADDRESS>
Yuye Ling1* 1Department of Electronic Engineering, Shanghai Jiao Tong
University, Shanghai 200240, China
2State Key Lab of Advanced Optical Communication Systems and Networks,
Shanghai Jiao Tong University, Shanghai 200240, China
*Corresponding author<EMAIL_ADDRESS>
Yan Li1 1Department of Electronic Engineering, Shanghai Jiao Tong University,
Shanghai 200240, China
2State Key Lab of Advanced Optical Communication Systems and Networks,
Shanghai Jiao Tong University, Shanghai 200240, China
*Corresponding author<EMAIL_ADDRESS>
Yikai Su2 1Department of Electronic Engineering, Shanghai Jiao Tong
University, Shanghai 200240, China
2State Key Lab of Advanced Optical Communication Systems and Networks,
Shanghai Jiao Tong University, Shanghai 200240, China
*Corresponding author<EMAIL_ADDRESS>

Holographic displays, which enable pixel-level depth control and aberration
correction, are considered the key technology for the next-generation virtual
reality (VR) and augmented reality (AR) applications. However, traditional
holographic systems suffer from limited spatial bandwidth product (SBP), which
makes them impossible to reproduce realistic 3D displays. Time-multiplexed
holography creates different speckle patterns over time and then averages them
to achieve a speckle-free 3D display. However, this approach requires spatial
light modulators (SLMs) with ultra-fast refresh rates, and current algorithms
cannot update holograms at such speeds. To overcome the aforementioned
challenge, we proposed a novel architecture, motion-enhanced holography, that
achieves realistic 3D holographic displays without artifacts by continuously
shifting a special hologram. We introduced an iterative algorithm to
synthesize motion-enhanced holograms and demonstrated that our method achieved
a 10 dB improvement in the peak signal-to-noise ratio (PSNR) of 3D focal
stacks in numerical simulations compared to traditional holographic systems.
Furthermore, we validated this idea in optical experiments utilizing a high-
speed and high-precision programmable three-axis displacement stage to display
full-color and high-quality 3D focal stacks. Our code and data are available
at https://github.com/Zhenxing-Dong/Motion-enhanced-Holography.

##  1 Introduction

Holographic displays are considered the ideal candidate for the next-
generation virtual reality (VR) and augmented reality (AR) displays [1],
thanks to their powerful ability to provide pixel-level depth control,
aberration correction, and compact form factor. Computer-generated holography
(CGH) is a digital simulation method to synthesize holographic patterns. Then
these holograms are loaded onto spatial light modulators (SLMs) to dynamically
reproduce the wavefronts of three-dimensional (3D) virtual objects through
laser illumination.

Recently, state-of-the-art (SOTA) CGH methods [2, 3, 4, 5] have delivered
high-fidelity displayed images with less artifacts. Peng et al. [2] introduced
a novel camera-in-the-loop (CITL) optimization strategy to improve display
quality of optical experiments, and developed a HoloNet architecture to
synthesize real-time 2D holographic images. Shi et al. [3] demonstrated a
deep-learning-based CGH pipeline capable of synthesizing a photorealistic
colour 3D hologram from a single RGB-depth image in real time. Choi et al. [4]
proposed a new differentiable camera-calibrated 3D model to achieve high-
quality 3D multi-plane holography. Shi et al. [5] further presented a layered
depth image (LDI) as an efficient 3D scene representation to solve the
occlusion problem and utilized supervised and unsupervised learning to
generate high-quality phase-only holograms.

However, these outstanding algorithms specifically pursue the exceptional
quality of in-focus objects, while neglecting constraints on the defocus
objects, leading to unnatural ringing artifacts. To address this issue, Yang
et al. [6] and Kavaklı et al . [7] both attempted to optimize the multi-plane
natural blur target to alleviate the unnatural defocus problem. Unfortunately,
supervising with a focal stack over-constrains the system, because natural
blur targets are generated from incoherent light, while holographic
reconstruction is based on coherent light principles. This approach would
exceed the degrees of freedom of current holographic systems. Therefore, a
fundamental question in attaining realistic 3D holographic displays is how to
improve the spatial bandwidth product (SBP) of traditional holographic
systems.

To overcome this challenge, Choi et al. [8] proposed time-multiplexed
holography, which utilizes a high-speed refresh rate micro-electromechanical
system (MEMS) to display multiple frames with unique speckle patterns in rapid
succession. This allows the visual averaging of these frames by the human eye
to achieve true 3D displays without artifacts. However, current algorithms are
unable to support real-time content updates at such a high refresh rate.
Moreover, it is impractical to transmit multiple sub-holograms in real time
over current wireless communication networks in practical VR and AR
applications. Recently, researchers have embarked on exploring hardware-
assisted holography, seeking to elevate the SBP of naïve holographic systems
through innovative hardwares. Kuo et al. [9] introduced multisource
holography, which incorporates an array of sources and two SLMs to achieve
high-quality focal stacks with natural defocus cues. However, this
architecture requires 16 light sources and 2 SLMs, making it challenging to
develop a compact form factor. Nam et al. [10] considered polarization as a
new degree of freedom by integrating metasurface into a holographic display.
However, it is complex to fabricate a precise metasurface, and the content-
dependent metasurface is difficult to apply in practical displays due to the
ever-changing real-world scenes.

Here, we proposed a novel holographic framework, motion-enhanced holography,
that presents fully life-like 3D scenes with realistic defocus blur. Unlike
time-multiplexed holography, our approach only relies on a single hologram
frame that is moved along a predefined path, synthesizing specific motion
speckle patterns. These patterns are then averaged to achieve realistic 3D
holographic displays. Moreover, compared to other hardware-assisted holography
that demands complicated hardware, our method only requires a programmable
displacement stage to move the SLM in optical experiments to reproduce true 3D
display fields. To do this, we further introduced an iterative algorithm to
optimize motion-enhanced holograms and demonstrated that the proposed method
achieved a 10 dB improvement in the PSNR of focal stacks in numerical
simulations compared to traditional holographic systems. Furthermore, we
developed a corresponding full-color holographic display system and provided a
proof-of-concept by validating the effectiveness of our method in optical
experiments.

##  2 Framework Overview

In this section, we briefly review traditional holography before introducing a
novel motion-enhanced holography.

###  2.1 Traditional Holography

Traditional holographic displays employ a coherent collimated laser beam to
illuminate a phase-only SLM, generating an optical field $u_{\mathrm{source}}$
in the SLM plane. The optical field is then propagated over a specific
distance $z$ to reconstruct 2D or 3D images. It is classical to utilize the
angular spectrum method (ASM) [11] as the ideal light propagation to simulate
complex waves from the SLM plane to the target plane. The expression is as
follows:

$\begin{gathered}u_{\mathrm{target}}(x,y)=\iint\mathcal{F}\left(a(x,y,\lambda)e^{i\phi(x,y,\lambda)}u_{\mathrm{source}}(x,y)\right)\\\
\cdot\mathcal{H}_{c}\left(f_{x},f_{y},\lambda,z\right)e^{i2\pi\left(f_{x}x+f_{y}y\right)}df_{x}df_{y},\\\
\mathcal{H}_{c}\left(f_{x},f_{y},\lambda,z\right)=\begin{cases}e^{i\frac{2\pi}{\lambda}\sqrt{1-\left(\lambda
f_{x}\right)^{2}-\left(\lambda f_{y}\right)^{2}}z},&\text{ if
}\sqrt{f_{x}^{2}+f_{y}^{2}}<\frac{1}{\lambda},\\\ 0&\text{ otherwise
}\end{cases}\end{gathered}$ (1)

where $u_{\mathrm{target}}$ is an optical field in the target plane, $\phi$ is
a phase-only hologram in the SLM, and $a$ is a constant value of 1 in the
ideal case. $\lambda$ is the wavelength, $f_{x}$, $f_{y}$ are spatial
frequencies, $z$ is the propagation distance between SLM and display plane,
$\mathcal{H}\\{\cdot\\}$ is ASM transfer function and $\mathcal{F}\\{\cdot\\}$
denotes the Fourier transform. The reconstruction images
$|u_{\mathrm{target}}(x,y)|$ can be obtained by calculating the absolute value
of $u_{\mathrm{target}}(x,y)$.

CGH adopts the ASM model to address the inverse problem of optimizing $\phi$
for a 2D image or 3D focal stack intensity $GT_{\mathrm{target}}^{j}$ located
at the set of distances $z^{j}$, $j=1,2\ldots J$. The objective is to solve
the following optimization problem:

$\underset{\phi}{\operatorname{minimize}}\mathcal{L}\left(\left|u_{\mathrm{target}}(x,y,z^{j})\right|,\sqrt{GT_{\text{target
}}^{j}}\right).$ (2)

.

To achieve natural 3D display, the GT focal stack based on the RGB image
$I(x,y)$ is synthesized from spatially incoherent wave propagation to reflect
the focus cues in nature. The intensity of incoherent propagation [12] is
given by the equation:

$GT_{\mathrm{target}}^{j}=\iint\mathcal{F}\left(I(x,y)\right)\mathcal{H}_{inc}\left(f_{x},f_{y},\lambda,z^{j}\right)e^{i2\pi\left(f_{x}x+f_{y}y\right)z^{j}}df_{x}df_{y},$
(3)

where $\mathcal{H}_{inc}$ is is the incoherent free-space propagation
operator. Furthermore, based on the relationship between incoherent and
coherent transfer functions, $\mathcal{H}_{inc}$ can be described as:

$\mathcal{H}_{inc}=\mathcal{H}_{c}\left(f_{x},f_{y},\lambda,z^{j}\right)\otimes\mathcal{H}_{c}\left(f_{x},f_{y},\lambda,z^{j}\right),$
(4)

Here $\otimes$ denotes auto-correlation. Therefore, if a single SLM pattern is
to directly control the 3D volume with natural focus cues in traditional
holographic displays, it would surpass the degrees of freedom of the SLM. This
is because Equation 2 would have many more target observations than unknown
phase values.

###  2.2 Motion-enhanced Holography

Figure 1: Motion-enhanced Holography.

To overcome this challenge, we proposed a novel motion-enhanced holography to
increase the SBP of holographic systems. In contrast to time-multiplexed
holography, we attempt to create a unique speckle pattern in space by shifting
the holograms. By superimposing the reconstructed images from each shifted
hologram, this speckle pattern is effectively averaged out, achieving the
high-fidelity 3D scenes with realistic defocus effects. The detailed framework
to synthesize motion-enhanced holograms is illustrated in Figure 1. For an SLM
phase $\phi$, we initially designed a shifted path to generate a sequence of
translated sub-holograms. This process can be represented as follows:

$\left\\{\phi_{\mathrm{s}1},\phi_{\mathrm{s}2},\ldots,\phi_{\mathrm{sK}}\right\\}={Shiftedpath}_{k}^{d,p}\\{\phi\\},$
(5)

where $K$ represents the number of movements, $d$ denotes the direction of
each movement and $p$ is the pixel value per move.

Then, each translated sub-hologram is propagated in free space to reconstruct
a 3D scene with specific speckle pattern. Lastly, the final reconstruction
result is obtained by superimposing and averaging the individual
reconstructions. We can express it as follows:

$\left|u_{\text{motion
}}^{j}\right|=\frac{1}{K}\sum_{k}\left|f_{\mathrm{ASM}}\left(\phi_{\mathrm{sk}},z^{j}\right)\right|.$
(6)

Therefore, our goal is to solve the following optimization problem:

$\underset{\phi}{\operatorname{minimize}}\mathcal{L}\left(\left|u_{\mathrm{motion}}(x,y,z^{j})\right|,\sqrt{GT_{\text{target
}}^{j}}\right).$ (7)

.

We consider integrating these parameters into the overall framework for joint
optimization as an interesting exploration. However, for the convenience of
highlighting the advantages of our framework, our shifted path is pre-
designed, specifying the number of movements, pixel values, and directions for
each movement in this work. These details will be elaborated in next section.

For specific loss functions to optimize our framework, we believe that the in-
focus planes require more attention in terms of accuracy compared to the
defocus planes:

$L=L_{2}\left(\left|u_{\text{motion }}^{j}\right|,\sqrt{GT_{\text{target
}}^{j}}\right)+w*L_{2}\left(\mathrm{M}^{j}*\left|u_{\text{motion
}}^{j}\right|,\mathrm{M}^{j}*\sqrt{GT_{\text{target }}^{j}}\right),$ (8)

where $L_{2}$ is the mean square error (MSE). The first item involves
evaluating the overall reconstructed images and target focal stack, while the
second item focuses only on assessing the in-focus region of the
reconstruction images and target focal stack. $w$ represents the weight
assigned to the second item and $\mathrm{M}^{j}$ is a binary mask where the
value is 1 in the in-focus region and 0 in other areas.

##  3 Results

###  3.1 Implementation Details

#### Details of optimizing motion-enhanced holograms.

To render target focal stacks with natural focus cues, we leverage DeepFocus
[13], an end-to-end convolutional neural network (CNN) designed to efficiently
synthesize defocus blur with accommodation-supporting. We rendered 7 target
planes at 0.1, 0.4, 0.83, 1.28, 1.74, 2.22, and 2.73 D and measured these
distances to correspond to -12, -8, -4, 0, 4, 8, and 12 mm away from the SLM
physically based on our optical setup. For the light sources, the wavelengths
of 633nm, 520nm, and 450nm correspond to red, green, and blue laser diodes,
respectively. The resolution of the hologram is 1920$\times$1024 with a 6.4
$\mu$m pixel pitch, while the resolution of the target image of interest is
1600$\times$880\. We move the hologram one pixel at a time, with a total of
nine movements of the hologram. All simulations were optimized and tested in
PyTorch using an NVIDIA GeForce RTX 3090 GPU Card.

Figure 2: Holographic display setup.

#### Details of the experimental setup.

The optical setup is shown in Figure 2, and it employs a HOLOEYE LETO-3 phase-
only liquid crystal on silicon that has a resolution of 1920 $\times$ 1080
pixels and a pixel pitch of 6.4 $\mu$m. This SLM has a refresh rate of 60 Hz
(monochrome) and an 8-bit depth. The laser used is a FISBA RGBeam single-mode
fiber-coupled module, which consists of three laser diodes optically aligned
at wavelengths of 633 nm, 520 nm, and 450 nm. The collimating lens (CL) is an
achromatic doublet with a focal length of 100 mm. The beam expander (BE) is
used to ensure that the laser incident on the SLM is as uniform as possible.
The neutral density (ND) filter is used to control the intensity of the laser,
while polarization filters are used to match the polarization direction of the
SLM. All images are captured with a FLIR BFS-U3-31S4M-C USB 3.1 vision sensor
with a resolution of 2048 $\times$ 1536 pixels and a pixel pitch of 3.45
$\mu$m. We provide a 4F system, where the first achromatic lens has a focal
length of 100mm, and the second achromatic lens has a focal length of 50mm,
ensuring that the camera can capture the complete reconstructed image. An iris
is positioned at the Fourier plane to block excessive light diffracted by the
grating structure and higher-order diffractions. We use a 3-axis nanomax
flexure stage with bundled controller to move the SLM.

###  3.2 Numerical Simulation

We qualitatively and quantitatively evaluated motion-enhanced holography in
simulation. Figure 3 compares the depth-of-field images reconstructed by the
conventional holography, time-multiplexed holography, proposed method, and the
ground truth. Due to the limited degrees of freedom provided by a single SLM,
traditional methods often exhibit significant speckle noise and chromatic
aberration (see Figure 3, Column 2) artifacts. The proposed method
meaningfully eliminates speckle artifacts and accurately reproduces the in-
focus region with the naturally defocused planes. Furthermore, we demonstrate
that the proposed method is visually and numerically comparable to the four-
frame time-multiplexed holography and significantly outperforms the
conventional method (10 dB improvement in terms of PSNR). This simulation
shows that motion-enhanced holography can create natural defocus cues with low
speckle, no ringing artifacts, and no chromatic aberration.

Figure 3: Numerical Simulation.

###  3.3 Optical Experiments

To further demonstrate the effectiveness of the proposed method, we conducted
the optical experiment in the real world. The experimental reconstructions are
shown in Figure 4. As expected from our simulations, the conventional method
yields a 3D scene with speckles and low contrast, and certain regions are
unable to be accurately reproduced. The observations clearly indicate that the
hologram generated by our method achieves higher-contrast and higher quality
compared to the conventional method. During hardware capture, we encountered
challenges such as ringing artifacts at the edges and color non-uniformity.
These obstacles impeded the reconstruction process from attaining outcomes
comparable to the simulated results. It is crucial to acknowledge that these
challenges arise from the imperfect nature of the hardware prototype and are
independent of the framework. To overcome these constraints, our intention is
to investigate and apply CITL optimization strategy. Through the utilization
of these strategy, our objective is to improve the display’s quality and
alleviate the hardware-related issues that have been observed.

Figure 4: Optical experiment.

##  4 Ablation Study

In this section, we conducted ablation experiments to observe the influence of
the number of movements on the final reconstruction results. The relevant
quantitative and qualitative analyses are presented in Figure 5. It is evident
that as the number of movements increases, the reconstructed images show
continuous improvement both visually and in terms of objective metrics. An
interesting phenomenon observed is that the improvement tends to be higher
when the number of movements is odd compared to even. Particularly, the image
quality of the reconstruction with 6 movements may be lower than that of the
reconstruction with 5 movements. We hypothesize that employing a symmetric
movement path during the scanning process would be advantageous for the
reconstruction of 3D scenes.

Figure 5: Ablation study: shifted number.

##  5 Conclusion

In summary, we proposed a novel architecture, motion-enhanced holography, that
achieves a realistic 3D holographic display without speckle by continuously
translating the same hologram. Compared to other complex hardware-assisted
holography, our method stands out as it only requires a single light source, a
single SLM, and a readily achievable three-axis displacement stage. We
conducted extensive numerical simulations and optical experiments to
demonstrate that our method enables the realization of holographic 3D displays
with realistic defocus effects. We believe that motion-enhanced holography
provides a promising solution and approach to address long-standing issues in
holographic displays, such as view-dependent effects and limited eye box size.

Limitations and Future Work. In this work, we employ a discrete movement
approach rather than continuous movement, which may introduce new issues when
compared to practical dynamic holographic displays, such as motion blur.
However, we hypothesize that the blurring caused by the continuous movement is
unlikely to have a detrimental impact on the final results when the SLM is
moved at high speeds. In the future, we will further explore this potential
issue.

In the current framework, the number of movements, their directions, and the
distances of movement are all pre-designed manually. We believe that
incorporating the use of reinforcement learning to optimize these hyper-
parameters within our framework would further enhance its potential. We employ
an iterative method to optimize the motion-enhanced hologram, and optimizing a
1080p hologram takes approximately 8 minutes with 2000 iterations. In the
future, we intend to leverage neural networks to expedite the entire inference
process, enabling real-time hologram generation.

## References

  * [1] C. Chang, K. Bang, G. Wetzstein, B. Lee, and L. Gao, “Toward the next-generation vr/ar optics: a review of holographic near-eye displays from a human-centric perspective,” _Optica_ , vol. 7, no. 11, pp. 1563–1578, Nov 2020.
  * [2] Y. Peng, S. Choi, N. Padmanaban, and G. Wetzstein, “Neural holography with camera-in-the-loop training,” _ACM Trans. Graph._ , vol. 39, no. 6, nov 2020.
  * [3] L. Shi, B. Li, C. Kim, P. Kellnhofer, and W. Matusik, “Towards real-time photorealistic 3d holography with deep neural networks,” _Nature_ , vol. 591, no. 7849, pp. 234–239, 2021.
  * [4] S. Choi, M. Gopakumar, Y. Peng, J. Kim, and G. Wetzstein, “Neural 3d holography: Learning accurate wave propagation models for 3d holographic virtual and augmented reality displays,” _ACM Trans. Graph._ , vol. 40, no. 6, dec 2021.
  * [5] L. Shi, B. Li, and W. Matusik, “End-to-end learning of 3d phase-only holograms for holographic display,” _Light: Science & Applications_, vol. 11, no. 1, pp. 1–18, 2022.
  * [6] D. Yang, W. Seo, H. Yu, S. I. Kim, B. Shin, C.-K. Lee, S. Moon, J. An, J.-Y. Hong, G. Sung _et al._ , “Diffraction-engineered holography: Beyond the depth representation limit of holographic displays,” _Nature Communications_ , vol. 13, no. 1, p. 6012, 2022.
  * [7] K. Kavaklı, Y. Itoh, H. Urey, and K. Akşit, “Realistic defocus blur for multiplane computer-generated holography,” in _2023 IEEE Conference Virtual Reality and 3D User Interfaces (VR)_. IEEE, 2023, pp. 418–426.
  * [8] S. Choi, M. Gopakumar, Y. Peng, J. Kim, M. O’Toole, and G. Wetzstein, “Time-multiplexed neural holography: A flexible framework for holographic near-eye displays with fast heavily-quantized spatial light modulators,” in _ACM SIGGRAPH 2022 Conference Proceedings_ , ser. SIGGRAPH ’22. New York, NY, USA: Association for Computing Machinery, 2022.
  * [9] G. Kuo, F. Schiffers, D. Lanman, O. Cossairt, and N. Matsuda, “Multisource holography,” _ACM Trans. Graph._ , vol. 42, no. 6, dec 2023.
  * [10] S.-W. Nam, Y. Kim, D. Kim, and Y. Jeong, “Depolarized holography with polarization-multiplexing metasurface,” _ACM Trans. Graph._ , vol. 42, no. 6, dec 2023.
  * [11] J. W. Goodman, _Introduction to Fourier optics_. Roberts and Company publishers, 2005.
  * [12] B. Lee, D. Kim, S. Lee, C. Chen, and B. Lee, “High-contrast, speckle-free, true 3d holography via binary cgh optimization,” _Scientific reports_ , vol. 12, no. 1, p. 2811, 2022.
  * [13] L. Xiao, A. Kaplanyan, A. Fix, M. Chapman, and D. Lanman, “Deepfocus: Learned image synthesis for computational display,” in _ACM SIGGRAPH 2018 Talks_ , ser. SIGGRAPH ’18. New York, NY, USA: Association for Computing Machinery, 2018.