Large Étendue 3D Holographic Display with Content-adpative Dynamic Fourier Modulation
Brian Chao Manu Gopakumar Suyeon Choi Jonghyun Kim Liang Shi Gordon Wetzstein





0000-0002-4581-6850

  Stanford University
  Stanford
  CA
  94305
  USA
brianchc@stanford.edu


0000-0001-9017-4968

  Stanford University
  Stanford
  CA
  94305
  USA
manugopa@stanford.edu


0000-0001-9030-0960

  Stanford University
  Stanford
  CA
  94305
  USA
suyeon@stanford.edu


0000-0002-1197-368X

  NVIDIA
  Santa Clara
  CA
  95051
  USA
jonghyunk@nvidia.com



0000-0002-4442-4679

  Massachusetts Institute of Technology
  Cambridge
  MA
  02139
  USA
liangs@mit.edu


0000-0002-9243-6885

  Stanford University
  Stanford
  CA
  94305
  USA
gordon.wetzstein@stanford.edu

§ ABSTRACT 
Emerging holographic display technology offers unique capabilities for next-generation virtual reality systems. Current holographic near-eye displays, however, only support a small étendue, which results in a direct tradeoff between achievable field of view and eyebox size. Étendue expansion has recently been explored, but existing approaches are either fundamentally limited in the image quality that can be achieved or they require extremely high-speed spatial light modulators. We describe a new étendue expansion approach that combines multiple coherent sources with content-adaptive amplitude modulation of the hologram spectrum in the Fourier plane. To generate time-multiplexed phase and amplitude patterns for our spatial light modulators, we devise a pupil-aware gradient-descent-based computer-generated holography algorithm that is supervised by a large-baseline target light field. Compared with relevant baseline approaches, ours demonstrates significant improvements in image quality and étendue in simulation and with an experimental holographic display prototype.









<ccs2012>
   <concept>
       <concept_id>10010583.10010786</concept_id>
       <concept_desc>Hardware Emerging technologies</concept_desc>
       <concept_significance>500</concept_significance>
       </concept>
   <concept>
       <concept_id>10010147.10010371.10010387</concept_id>
       <concept_desc>Computing methodologies Graphics systems and interfaces</concept_desc>
       <concept_significance>500</concept_significance>
       </concept>
 </ccs2012>


[500]Hardware Emerging technologies
[500]Computing methodologies Graphics systems and interfaces









  
    < g r a p h i c s >

   
Conventional holographic displays use a single laser source that provides a limited étendue, here visualized by a recorded spectrum that only covers a small area of the Fourier plane (b). Single-source holograms therefore only support a limited eyebox size, which means that an image can be observed only when the user's pupil (b, red) is well aligned with the eyebox (c). The image quickly degrades and fades into black as the pupil (b, blue) shifts even a small amount (d). Using multi-source illumination (a), our holographic display creates a significantly expanded coverage of addressable spatial frequencies (e) which, combined with our content-adaptive Fourier modulation strategy, achieves a large étendue with better image quality across an expanded eyebox (f,g). 

	
  
 

Large Étendue 3D Holographic Display with Content-adaptive Dynamic Fourier Modulation
    Gordon Wetzstein
    Received: 20 June 2024 / Accepted: 05 July 2024
=====================================================================================





§ INTRODUCTION



Holographic near-eye displays offer unique benefits to virtual and augmented reality (VR/AR) applications. For example, holographic displays can present perceptually realistic 3D images with natural parallax to the user in lightweight device form factors <cit.>. Yet, the étendue of holographic displays is fundamentally limited by the pixel count of the underlying spatial light modulators (SLMs), preventing current holographic near-eye displays from achieving a large field of view and eyebox simultaneously. This limitation is a fundamental barrier towards making this a practical display technology. 


Increasing the pixel count of an SLM seems like the natural solution. However, developing large-area phase-only SLMs with pixel pitches matching the small feature sizes (i.e., tens of nanometers) of analog holographic films <cit.> is simply not feasible with today's hardware solutions. To overcome this problem, étendue expansion techniques have been described in the literature, including those based on static, high-resolution masks <cit.>, pupil replication <cit.>, steered or multi-source illumination <cit.>, and making use of higher-diffraction orders and pupil optimization <cit.>. However, each of these approaches has its limitations, as mask-based systems do not have sufficient degrees of freedom to achieve a high image quality, pupil replication approaches cannot create natural 3D effects and parallax over the eyebox, and steered sources are hindered by the requirement for high-speed SLMs as well as high diffraction orders (HDOs) that fundamentally limit the image quality. As a result, none of these solutions is able to achieve high-quality 3D holography with a large étendue.


Our work is motivated by the hypothesis that a holographic display requires sufficient degrees of freedom to achieve a large field of view and eyebox simultaneously. In the absence of an extremely high-resolution SLM, this is only achievable using steered or multi-source illumination. We thus build on the latter approach but address its major shortcomings, HDOs and symmetric illumination copies, by introducing a dynamic, programmable amplitude modulation mechanism in the Fourier plane, after the SLM. This unique optical setup allows us to extend steered / multi-source configurations such that they modulate the frequency spectrum of the display image in a content-adaptive manner. For this purpose, we leverage a stochastic optimization approach that factors a target light field into a set of time-multiplexed phase SLM and corresponding Fourier amplitude masks that are displayed in rapid succession while being integrated by the user's eye.


Using the proposed system, we demonstrate improved 3D image quality over a large étendue, surpassing the performances of existing approaches in both simulation and experiment. Specifically, our contributions include 


    
  * A novel optical holographic display configuration that combines a time-multiplexed phase SLM near the image plane and a dynamic amplitude SLM that controls the frequency spectrum.
    
  * A computer-generated holography framework that uses stochastic optimization to factor a target light field into a set of phase–amplitude image pairs.
    
  * Demonstration of improved 3D image quality among high-étendue holographic displays.


Our method should be clearly distinguished from Multisource Holography, a system recently proposed by <cit.> for speckle reduction that also leverages a multi-source laser array. In Multisource Holography, the multi-source laser and two phase-only SLMs placed in close proximity are used to remove speckles, but the system étendue remains limited since the spacing between each source is relatively small. In our system, we place the laser sources much farther apart to create a high-étendue backlight for the phase-only SLM to greatly increase the eyebox size and place an amplitude display at the Fourier plane.



§ RELATED WORK





  
Holographic Near-eye Displays. Holographic displays are a promising technology for virtual and augmented reality applications due to their unique capability to display true 3D content and significant progress has been made recently <cit.>. In particular, the advancement in computer graphics, machine learning, and computing infrastructures have enabled real-time hologram rendering based on neural networks <cit.>, significantly improved image quality with end-to-end optimization <cit.>, higher light efficiency and brightness with simultaneous control of multiple wavelengths and energy-efficiency loss function <cit.>, and thin form factors in eyeglasses-like design <cit.>. Despite offering these unique capabilities, current holographic displays fail to provide a comfortable immersive experience as they cannot simultaneously provide a wide field of view (FoV) and a sufficiently large eyebox (i.e., the region in which a user's eye perceives the displayed content). 

In a given display system, the product of the FoV and the eyebox is a constant, referred to as the étendue. For a holographic display, the étendue is directly proportional to the number of pixels in the SLM. A 1080p SLM, for instance, can either support a wide field of view (e.g., 80 degrees) with an eyebox smaller than 1 mm or vice versa. 
However, increasing the SLM resolution to the point where both large field of view and eyebox can be achieved simultaneously faces significant challenges in manufacturing, cost, and addressing speed, accuracy, and bandwidth. Instead, efforts have been made to increase the étendue of holographic displays without increasing SLM resolution.
The approaches fall under two categories: (i) increasing étendue after the SLM and (ii) increasing étendue before the SLM.





  
Post-SLM Étendue Expansion
The most representative method in this category is mask-based étendue expansion <cit.>, where a static mask at a resolution higher than the SLM is placed after the SLM to increase the diffraction angles of the SLM-modulated wavefront, thus increasing the étendue. However, such systems suffer from difficulties in alignment and from reduced image quality and low contrast because their effective degrees of freedom <cit.> are insufficient to synthesize a high-quality large-étendue wavefront. Pupil replication <cit.> is another popular approach. It is implemented either by putting a pupil-replicating waveguide after the SLM to replicate pupil locations at its out-coupler or using a pupil-replicating holographic optical element (HOE) as the eyepiece, effectively expanding the eyebox of the system. However, pupil-replicating displays cannot display 3D content or natural parallax across the expanded eyebox since the content within the eyebox are merely copies of the same wavefront. Higher-diffraction orders combined with pupil optimization can also be leveraged to slightly expand the eyebox in the single-source case <cit.>. Finally, a regular eyepiece can be replaced by a lens array to partition an unexpanded eyebox into an array of smaller chunks that cover an expanded area <cit.>. However, this comes at an explicit cost of image quality and brightness nonuniformity, especially when observed with a small pupil.



  
Pre-SLM Étendue Expansion
Methods in this category modify the laser illumination to expand étendue either through a multi-source configuration or beam-steering. <cit.> used a micro-electromechanical-system (MEMS) mirror to temporally change the laser illumination and steer the resulting pupils over a larger eye box. <cit.> implemented the same principle by arranging individual laser diodes into a 2D array and sequentially turning each one on to create temporal directional illumination. <cit.>  implemented per-pixel beam steering of the phase SLM by using transmissive LCD panels and polarization gratings and demonstrated that the étendue expansion amount scales exponentially with the number of LCD layers.
To permanently expand the étendue, <cit.> activated all illumination sources simultaneously. They introduced a random mask at the Fourier plane to break the correlation among copies in the spectrum formed by directional illuminations. This effectively eliminates duplicate images within the expanded eyebox. However, they did not demonstrate view-dependent effects across the expanded eyebox and the random mask is not content adaptive, resulting in reduced 3D realism and low image quality. Instead of using multiple laser diodes that are incoherent with each other, <cit.> implemented a mutually coherent multi-laser source using a lens array. The mutually coherent sources can interfere constructively and destructively with each other, granting the hologram optimization process more degrees of freedom. However, their system requires eye tracking and a new hologram needs to be optimized for each dynamic pupil location, making the system challenging for real-time applications. 



  
Fourier Modulation. 
Holographic display systems often require Fourier plane filtering to remove HDOs created by the pixelated structure of the phase SLM <cit.>. However, it is not straightforward to apply this to a multi-source or beam steering setting since the directional illuminations create shifted copies of the wavefront from normally incident illumination and the associated HDOs in the Fourier domain. When using beam-steering, the filter position needs to be dynamically adjusted to block the HDOs of the shifted wavefront. To achieve this, <cit.> placed a programmable polarization shutter at the Fourier plane and synchronized the laser sources to filter out the HDOs. However, when using multi-source illumination for étendue expansion, HDOs associated with one illumination intermingle with the shifted wavefront of another illumination at the Fourier domain, making it impossible to separate. It is therefore crucial to model HDOs to precisely characterize how they contribute to different angular views. Explicit modeling of HDOs has been demonstrated for 2D images <cit.> and 3D focal stacks <cit.>, but not for 4D light fields under multi-source illumination.



Inspired by <cit.>, we employ a multisource laser illumination and additionally place a dynamic amplitude SLM at the Fourier plane to enable content adaptive modulation and time multiplexing. We jointly optimize the patterns for the phase and amplitude SLMs to reproduce a 4D light field rendered over an expanded eyebox. We also explicitly model the HDOs and demonstrate notable improvement in image quality and contrast. Collectively, this new hardware and software co-design enables a dynamic view-dependent holographic display with a large eyebox.







§ METHOD
























In this section, we first review the conventional single-source holographic image formation model before introducing the multi-source image formation model of our system.





 §.§ Single-Source Holographic Image Formation Model
 

 
For on-axis Fresnel holography, a collimated beam from a laser source illuminates an SLM with a normally incident, coherent field . The SLM imparts a spatially varying phase delay ϕ to the field which propagates a distance z along the optical axis. The wavefront at this plane can be mathematically described using the angular spectrum method (ASM) <cit.> as a function of the phase pattern and distance from the SLM:


    f (ϕ, z ) = {{ e^iϕ(x, y )(x, y ) }·(, ; z ) }
       (, ; z ) = 
        e^i2π/λ z √(1- ( λ) ^2- ( λ) ^2)   if √(^2+^2) < 1/λ.
    
        0    otherwise.


Here, λ is the wavelength of light, x, y are the spatial coordinates on the SLM, , are the frequency coordinates, 
and  is the transfer function of the ASM. The operator f models free-space propagation between the parallel SLM and target planes separated by a distance z. For notational convenience, we omit the dependence
of the fields on x and y. The intensity generated by a holographic display at a distance z in front of the SLM is therefore | f (ϕ, z ) |^2. If a high-speed SLM is available, a time-multiplexed variant of the image formation is ∑_t=1^T | f (ϕ^(t), z ) |^2  / T, where T phase SLM patterns ϕ^(t), t = 1, …, T are rapidly displayed in sequence, and the resulting intensities are averaged by the users' eye <cit.>.




 §.§ Multi-Source Holographic Image Formation Model with Fourier Modulation

To extend the single-source image formation model to our system, we modify the formulation to incorporate off-axis collimated illumination traveling in direction
𝐤 = (k_x, k_y, k_z) and a programmable amplitude mask  at the Fourier plane of the holographic display system. This results in the model


    f^(j)(ϕ, , z ) = 
        {^(j)(, ; ϕ) ·(, ; z ) ·(, ) }
       ^(j)(,  ; ϕ) = { e^iϕ(x, y )^(j)(x, y ) e^i  𝐤^(j)·𝐱}


where j is the index of the source, 

and ^(j)(x, y ) is the complex-valued field modeling any deviations in amplitude and phase of source j=1, …, J from a perfect plane wave e^i  𝐤^(j)·𝐱, 𝐱 = (x,y,z). Moreover, ^(j)(x, y ) can optionally also include per-source, time-dependent modulation, such as switching individual lasers on and off. In our setup, we do not consider this case and assume that all sources are turned on at all times. Please refer to the supplemental material for more discussions about the generalized configuration with amplitude-controllable laser sources ^(j)(x, y ).





 §.§ Stochastic Optimization of Light Field Holograms


To reconstruct a light field, we use gradient descent to optimize a set of time-multiplexed phase patterns ϕ^(t) and corresponding Fourier masks ^(t) by minimizing the following objective:

    {ϕ^(t), ^(t)}minimize‖ s √(1/T∑_t=1^T∑_j=1^J |H2LF( f^(j)(ϕ^(t), ^(t), z) ))|^2 - lf_target‖

where s is a scale factor, lf_target is the amplitude of the target light field, and H2LF is a hologram-to-light field transformation, such as the Short-Time Fourier Transform (STFT) <cit.>. 

The memory consumption of the above optimization problem is huge due to time multiplexing, multiple sources, and the explicit modeling of HDOs. Therefore, it is impractical to realize H2LF using the STFT since it reconstructs a whole light field and the memory consumption would explode for a dense lf_target. To solve this problem, we devise a stochastic version of Eq. <ref> that allows us to optimize a single light-field view rather than a full light field in each iteration of the optimization routine. 

For this purpose, we randomly chose a view p of the target light lf^(p)_target in each iteration and run a gradient descent step of Eq. <ref>. A binary pupil mask ^(p) in the Fourier plane in the hologram-to-light field transform is applied to reconstruct one specific view as 


    H2LF^(p)( f^(j)(ϕ^(t), ^(t), z) ) = f^(j)(ϕ^(t), ^(t)·^(p), z), 
       ^(p)(, ) = 
            1,    if  ( - p)^2 + ( - p)^2 ≤ r_p^2, 
    
            0,    otherwise


where ^(p) is a binary pupil mask in the Fourier plane, r_p is the radius of the pupil and p, p are the spatial coordinates of the center of the pupil. This procedure is similar to the pupil-supervision techniques described in <cit.>. Please refer to the supplemental material for more details on our stochastic light field optimization procedure.




 §.§ Implementation Details


Since we are using a highly-quantized 4-bit phase SLM, the quantization of pixel values needs to be taken into consideration. Such quantization constraints can be enforced using techniques described in prior work <cit.>. Higher diffraction orders (HDOs) are modeled using the wave propagation model described in <cit.>. We use PyTorch to implement all our algorithms and run optimization.

In all our experiments, the radius r_p of the pupils are set to be 2 mm, resulting in a 4 mm diameter pupil. 81 pupils are equally spaced in the Fourier plane (eyebox plane), where each pupil corresponds to a single view in a 9 × 9 light field. The illumination directions of the multisource laser are set such that they match the diffraction angle of the ± 1^st higher diffraction orders of the blue wavelength. This allows the blue spectrum copies to be perfectly tiled in the Fourier plane, while removing the gaps between the red and green wavelength spectrum copies. Please see the supplemental materials for detailed discussion on the choosing the appropriate illumination angles.





§ ANALYSIS








 §.§ Optical System Analysis

The étendue G of a display is defined as the product of the display area with the solid angle of emitted light:

    G = 4Asin^2θ,

where A is the display area and 2θ is the solid angle of the emission cone of each display pixel. Étendue is conserved through reflections, refractions, and free space propagation in an optical system. When illuminated with a normal incidence light of wavelength λ, the diffraction angle θ_SLM of an SLM with pixel pitch p can be expressed as θ_SLM = ±sin^-1λ/2p.
   
For an SLM of physical size L_x × L_y, its étendue G_SLM can be expressed as:

    G_SLM = 4L_x L_ysin^2θ_SLM = λ^2 N_x N_y

where N_x × N_y is the pixel resolution of the SLM. This means that the étendue of a holographic display is directly proportional to the number of pixels of the SLM.





In a Fresnel holography display system, the 1D field-of-view (FoV) and eyebox size w can be expressed as follows:

    FoV = 2tan^-1(L/2 ) = 2tan^-1(Np/2 ),     w = λ/p,

where L, N are the size of the SLM and the number of SLM pixels in the x or y axis, respectively, and  is the eyepiece focal length. Under paraxial assumptions (θ≈sinθ≈tanθ), we see that the product of the 2D FoV and eyebox of a holographic display system is exactly the étendue of the system:

    FoV_x ·FoV_y · w^2 = 
    
        2tan^-1(N_xp/2 ) · 2tan^-1(N_yp/2 ) ·^2λ^2/p^2≈λ^2 N_x N_y = G_SLM

This implies that there is an inherent tradeoff between the FoV and the eyebox of a holographic display.

When the SLM is illuminated with a grid of α×α off-axis, directional illuminations, the system eyebox is expanded due to shifted copies of the original spectrum. Specifically, if the directional illumination is selected such that the illumination direction matches the higher-order diffraction angles, the system 1D eyebox is exactly expanded by α while the FoV remains the same, resulting in an expanded 1D eyebox size of w = α fλ/p. Therefore, the 2D étendue of the system is expanded by a factor of α^2. 

We show how the FoV and eyebox size relates to the required number of sources in Fig. <ref>. We assume an SLM pixel pitch of 10.8  μm and resolution of 1000 × 1000 and laser wavelength of 632.8 nm. The FoV and eyebox size move along each white line in opposite directions as we vary the eyepiece focal length  while the system étendue remains fixed. As we increase the number of sources, the étendue of the system also increases, as the white lines move further towards to upper-right of the plot.



 §.§ Baseline Configurations


We next discuss a number of holographic display system configurations that serve as baselines to our proposed design shown in Fig. <ref>. Illustrations of these baselines are shown in Fig. <ref>.



  
I. Single Source with Fourier Filter. The conventional holographic display setup with a single laser source and a Fourier filter to block HDOs, including <cit.>. Such systems suffer from small étendue and non-uniform brightness across the eyebox.



  
II. Single Source with Phase Mask. A high-res­o­lu­tion phase mask is placed in front of the SLM to increase the diffraction angle of the SLM and therefore increase the étendue of the system. The phase masks can be random <cit.> or optimized <cit.>. These approaches have been shown to expand the étendue at the cost of decreased image quality and contrast.




  
III. Multiple Sources. Multiple mutually incoherent lasers illuminate the SLM from different angles simultaneously. Due to the absence of a Fourier filter, the frequency spectrum contains multiple shifted, potentially overlapping copies of the same hologram. These constraints limit this system's capability to perfectly reconstruct a light field.





  
IV. Multiple Sources with Fixed Random Fourier Mask Multiple lasers illuminate the SLM from different angles simultaneously while a fixed random mask is placed at the Fourier plane to break the correlation between the image copies, as demonstrated by Jo et al. jo2022binary. Time multiplexing and content-adaptive filtering are not feasible since the random masks are custom-printed and fixed.






  
V. Multiple Sources with Dynamic Fourier Filter (ours). Multiple lasers illuminate the SLM from different angles simultaneously while an amplitude SLM is placed at the Fourier plane. The amplitude SLM can be dynamically refreshed and is synchronized with the phase SLM, allowing for time-multiplexed and content-adaptive Fourier modulation. We additionally compare with a generalized configuration V^* where the amplitude of the laser sources are controllable rather than fixed. More discussions on this generalized configuration can be found in the supplement.



  
VI. Steered Illumination. Multiple individually controllable sources or a single, swept source illuminate the SLM from different angles without Fourier filtering, including <cit.>. Reduced image quality due to HDOs remains an issue due to the lack of filtering. Furthermore, such methods only apply to very high-speed SLMs, because each source is sequentially turned on or steered in sequence.



  
VII. Steered Illumination with Shifting Fourier Filter. Multiple individually controllable sources or a single, swept source illuminate the SLM from different angles while a synchronizable, dynamic filter is placed at the Fourier plane to filter out the HDOs. One example is the steered illumination system described in <cit.>. This method is still sequential in nature, as each source is turned on one at a time, and requires a very high-speed SLM.










 §.§ Assessment



Table <ref> and Figure <ref> shows the light field reconstruction performance of different baseline configurations in simulation. For configurations with multiple sources (III, IV, V, V*, VI, and VII), we consider a 3×3 grid of sources. We simulate a 800 × 1280 phase SLM for single-SLM configurations (I, II, III, VI, VII) and an additional 20 × 20 Fourier display for configurations IV, V, and V^*. We run our optimization algorithm on all configurations to reconstruct a 9 × 9 light field. Additionally, we optimize the single source configuration to reconstruct a smaller, 3 × 3 light field in Table <ref>. Time multiplexing is not used for configurations I, II, III, and IV. Please refer to the supplemental material for additional discussions on the optimization parameters, degrees of freedom of the system, and ablation studies on the resolution of the Fourier display.

The naive single-source configuration (I) is able to reconstruct a small 3 × 3 light field, but fails to reconstruct a larger-baseline 9 × 9 light field and cannot support uniform brightness across the expanded eyebox (Fig. <ref>, b–d). Mask-based étendue expansion techniques (II) reconstruct low-contrast and speckly images. By using multiple sources (III), the eyebox is expanded but the light field reconstruction quality is poor due to copies created by multiple sources and HDOs. Introducing a fixed random mask at the Fourier plane (IV) improves image quality, although the improvement is limited due to the lack of time multiplexing and content-adaptive Fourier mask optimization. Steered illumination options (VI, VII) achieve decent image quality, however both configurations reconstruct speckly images due to HDOs and can only be implemented using high-speed SLMs due to the large number of required time-multiplexed frames.

Our methods (V, V^*) achieves the best image reconstruction quality when using one frame and is better than the steered illumination baseline (VII) while using fewer frames (6 vs. 9). This is achieved through time multiplexing and our novel content-adaptive Fourier modulation optimization framework. Our method successfully removes the copies created by multiple sources and HDOs, reconstructing clean and speckless light field views. More importantly, a minimal increase in degrees of freedom in the Fourier plane (a low-resolution 20 × 20 Fourier display) is sufficient to achieve good image quality, and we perform extensive experiments to validate this claim in the supplemental material. Finally, although our generalized configuration V^* with amplitude-controllable sources achieves the best quantitative image quality, the improvement is marginal (<0.5 dB in terms of PSNR) and suffers from a much higher system complexity. Hence, we opted for configuration V for our hardware implementation.









§ EXPERIMENTAL RESULTS






  
Hardware Implementation We implement the proposed 3D holographic display design and evaluate our algorithms on the system. The hardware setup and the optical path are shown in Fig. <ref>. We implement our multi-source laser by cascading multiple 1:4 fiber splitters (Thorlabs TWQ560HA) and arranging 9 customized fiber tip outputs into a 3×3 array, which is then held together using a custom-printed 3D mount. The spacing between each source is 8.17 mm and a 200 mm lens is used to collimate the multi-source laser. Each collimated source field is, therefore, incident on the phase SLM with a 2.34^∘ incident angle. We use a TI DLP6750Q1EVM phase SLM for phase modulation and a 1080p SiliconMicroDisplay liquid crystal on silicon (LCoS) display for Fourier amplitude modulation. A 75 mm Fourier transform lens is used to image the spectrum of the phase-modulated wavefront onto the amplitude SLM. Our final design has a diagonal field-of-view (FoV) of 7.78 degrees and an eyebox size of 8.53 mm × 8.53 mm. A FLIR Grasshopper 2.3 MP color USB3 vision sensor paired with a Canon EF 50mm f/1.4 USM camera lens is used to capture all experimental results. Please refer to the supplemental material for additional details on the degrees of freedom of our system and the relevant optimization parameters.






  
Experimental Capture Details 

To capture light field views, we place pupil masks at the Fourier plane to mimic the movement of the user's eyes, which is a technique used in prior works <cit.>. We implement this with a Thorlabs SM1D12 adjustable iris on a translation stage at the Fourier plane. To capture focal stacks, we center the pupil at the Fourier plane and adjust the camera focus to capture images at different depths. 




  
Assessment 
Experimentally captured results are shown in Figs. <ref>, <ref>, Table <ref>, and in the supplemental material. The PSNR and SSIM values are averaged across all captured light field views. We observe the same trends as predicted by our simulations both quantitatively and qualitatively: the single-source configuration only supports a limited eyebox and suffers from severe brightness falloff at peripheral viewpoints; 3D multi-source holography without a Fourier filter cannot achieve a high image quality due to the copies created the multiple sources; a static random mask placed in the Fourier plane only provides limited degrees of freedom and suffers from low contrast; our approach without time multiplexing (i.e., 1 frame) improves the quality over the random mask as it optimizes the amplitude mask pattern in a content-adaptive manner; our method with 6-frame time multiplexing achieves the highest image quality with the largest amount of empirically observed parallax.





§ DISCUSSION


In summary, we present a novel hardware system for étendue expansion and an algorithmic framework for 4D light-field-supervised computer-generated holography. The hardware system includes a multi-source laser array to create a large-étendue coherent backlight for the phase SLM and an amplitude SLM for dynamic Fourier-amplitude modulation. The algorithmic framework includes the joint optimization of time-multiplexed amplitude SLM and phase SLM patterns and a memory-efficient, stochastic light field supervision procedure to create 4D light field holograms. We compare our method with a number of étendue expansion baselines and verify in simulation and experimentally that our system achieves the highest-quality light field reconstruction results for large étendue settings. 



  
Limitations and Future Work 
We demonstrate our results on a benchtop display setup but futher efforts are required to miniaturize this system. Currently, our multisource laser array is implemented using bulky fiber splitters and could be miniaturized using nanophotonic phased arrays <cit.>. Folding the propagation distance of holograms using optical waveguides could further remove the need of beam splitters and subsequently shrink the form factor, as demonstrated in <cit.>. We illustrate potential compact designs in the supplemental material. The frame rate of our system is limited by our amplitude display (240 Hz native frame rate). This translates to a ∼13.33 Hz frame rate when operating in color-sequential mode with 6-frames time-multiplexing. The frame rate can be improved by using more advanced LCoS displays with frame rate > 720 Hz <cit.>. Real-time synthesis of light field holograms are necessary for practical holographic displays, but is not currently not supported by our system. Extending recent neural network-based hologram synthesis methods <cit.> to work for 4D light field holograms would be an interesting future direction. Finally, we did not attempt to calibrate a neural network–parameterized wave propagation model of our prototype display system, which has been demonstrated to significantly improve experimentally captured holographic image quality for other types of optical configurations <cit.>.



  
Conclusion The novel hardware design and algorithmic framework presented in this work improves the étendue of holographic displays and allows for light field holograms synthesis with improved image quality. These help make holographic displays a more practical technology for augmented and virtual reality applications. 


We thank Grace Kuo for helpful advice regarding the implementation of the multisource laser setup. Brian Chao is supported by the Stanford Graduate Fellowship and the NSF GRFP. Manu Gopakumar is supported by the Stanford Gradudate Fellowship. Suyeon Choi is supported by the Meta Research PhD Fellowship.








ACM-Reference-Format


