1. Bibliographic Information

1.1. Title

Wide-FOV 3D Pancake VR Enabled by a Light Field Display Engine

1.2. Authors

Qimeng Wang, Yifan Ding, Mingjing Wang, Yaya Huang, Bo-Ru Yang, and Zong Qin All authors are affiliated with the School of Electronics and Information Technology, Sun Yat-Sen University, Guangzhou, China. Zong Qin is the corresponding author. Their research background appears to be in display technologies, particularly related to VR, light field displays, and micro-LCDs.

1.3. Journal/Conference

The paper does not explicitly state the journal or conference where it was published. However, the accompanying metadata (Presentation type: Oral preferred, Presenter: Student, Primary Topic: AR/VR/MR (AVR), Secondary Topic: Display Systems (DSY)) suggests it was presented at a conference related to display technologies or augmented/virtual reality. Given the nature of the research, it is likely a prestigious conference in optics, photonics, or display technology (e.g., SID Display Week, OSA conferences, SPIE Photonics West).

1.4. Publication Year

The publication year is not explicitly stated in the provided text. However, a reference [10] from 2024 is cited, and the paper itself includes "SID Symp. Dig. Tech. 35(1), 1271-1274 (2024)", indicating a likely publication year of 2024 or late 2023.

1.5. Abstract

This paper introduces a novel approach for a true-3D Virtual Reality (VR) headset that combines Pancake optics with a light field display (LFD) engine. The LFD engine generates intermediate images that incorporate computational focus cues, providing depth information. To achieve high resolution, the system utilizes a field-sequential-color (FSC) micro-LCD. A key innovation addresses the common issue of field-of-view (FOV) reduction in LFDs, often caused by optical aberrations, by employing a telecentric optical path. The research experimentally validates the system's ability to produce clear 3D images with a wide FOV of 68.6 degrees.

1.6. Original Source Link

/files/papers/6937898ba1be66f6e380326b/paper.pdf The publication status is likely officially published, as it provides detailed methodology and experimental results typical of a peer-reviewed paper.

2. Executive Summary

2.1. Background & Motivation

The core problem the paper aims to solve is the vergence-accommodation conflict (VAC) in current Virtual Reality (VR) displays. Most contemporary VR headsets, especially those utilizing Pancake optics, offer a compact and lightweight design with a large field of view (FOV) but typically present images at a fixed virtual image distance. This fixed distance causes a mismatch between the vergence (angle of the eyes when focusing on an object) and accommodation (the eye's adjustment of focus), leading to eye strain, fatigue, and an unnatural viewing experience, thus not supporting true-3D display.

Previous attempts to address VAC in Pancake VR, such as mechanically moving lenses or inserting varifocal elements (e.g., LC lenses), are limited. Mechanical solutions are complex and slow, while varifocal elements can only adjust diopter (focus distance) but cannot render true-3D scenes with multiple focal planes simultaneously. Other VAC-free technologies, like Maxwellian view displays and holographic displays, also have limitations. Maxwellian view restricts the eyebox (the region where the viewer's eye can be placed to see the full image), and holographic displays, while offering true-3D, often require coherent light sources, leading to bulky systems, though recent advancements are making them more compact.

Light Field Displays (LFDs) are promising for VAC-free viewing due to their ability to encode computational focus cues. However, directly integrating an LFD as a near-eye display faces significant challenges: a sharp drop in visual resolution because microlens arrays (MLAs) magnify pixels, and a severely limited FOV due to aberrations (optical distortions) from the MLA.

This paper's innovative idea is to combine the advantages of both LFD and Pancake optics. It proposes using an LFD engine to generate intermediate images with computational focus cues (thereby providing true-3D capabilities) and then relaying these images through a Pancake module (for compactness, lightweight design, and a large FOV). The paper specifically addresses the FOV limitation of LFDs by integrating them into the telecentric path of Pancake optics.

2.2. Main Contributions / Findings

The primary contributions and key findings of this paper are:

1. True-3D Pancake VR System: The paper successfully proposes and demonstrates a true-3D Pancake VR headset by integrating a light field display (LFD) engine with a Pancake module. This system effectively overcomes the vergence-accommodation conflict (VAC) by providing computational focus cues, allowing for multiple virtual image depths in a single scene.
2. High-Resolution LFD Engine: It incorporates a field-sequential-color (FSC) micro-LCD with a 2.3Kx2.3K resolution. This design removes the color filter array, achieving a tripled resolution and significantly improving optical efficiency, which is crucial for the inherently resolution-sacrificing nature of LFDs and the low efficiency of Pancake optics.
3. Expanded Field of View (FOV): The paper effectively addresses the aberration-induced FOV reduction typically seen in LFDs by utilizing the object-space telecentric path inherent in Pancake optics. This ensures that chief rays (central rays from an object point) pass through the microlens array (MLA) nearly paraxially (close to the optical axis), thereby minimizing aberrations and enabling a wider FOV.
4. Image Quality Matching Strategy: A sophisticated strategy is developed to match the image quality variations between the LFD engine and the Pancake module across different depth planes. This involves intentionally placing the LFD engine's Central Depth Plane (CDP) at a Pancake object plane that might not be its absolute best, but provides a balanced overall image quality for the entire depth range.
5. Experimental Verification: A prototype was built and experimentally verified. It successfully demonstrated clear 3D images with computationally adjustable virtual image distances, showcasing true-3D capability. The measured FOV was 68.6 degrees, significantly larger than what a standalone LFD engine could achieve and comparable to commercial Pancake VR systems. The system introduced an acceptable additional optical track of 2.1 cm.

3. Prerequisite Knowledge & Related Work

3.1. Foundational Concepts

To understand this paper, a foundational grasp of several optical and display technologies is essential:

• Pancake Optics: This is a compact optical design commonly used in modern VR headsets. It uses a folded optical path, typically involving a polarizing beam splitter (or half mirror), quarter-wave plate (QWP), and reflective polarizers, to achieve a large FOV in a thin form factor. Light from a microdisplay is circularly polarized by a QWP, reflected multiple times within the lens module (between the half mirror and reflective polarizer), and finally exits into the eye. This folding reduces the physical length of the optical path, making the headset more compact.

  As shown in Figure 1 (image 1.jpg in VLM), the Pancake module works by having light from the display pass through a quarter-wave plate (QWP), which converts its linear polarization into circular polarization. This circularly polarized light then enters the front lens, which contains a half mirror or a polarizing beam splitter. The light is reflected multiple times within the cavity between the lens and the reflective polarizer before exiting the module and reaching the user's eye. This folded path allows for a shorter physical distance while maintaining a longer effective focal length, leading to a compact form factor.

  该图像是一个示意图，展示了宽视场3D Pancake VR系统的光学结构。通过QWP、半透镜和反射偏振器，显示器生成三维图像，并引导光线进入观察者的眼睛。

  Figure 1. Working principle of the Pancake.

• Light Field Display (LFD): An LFD aims to reproduce the light field (the distribution of light rays in space) of a scene, providing true-3D perception without the need for special glasses. It typically uses a microdisplay (like an LCD) in conjunction with a microlens array (MLA). The microdisplay shows an elemental image array (EIA), where each elemental image is viewed through a corresponding microlens. By encoding different perspectives into these elemental images, the LFD can generate computational focus cues and parallax (the apparent shift of an object's position due to a change in viewing angle), allowing the viewer's eyes to naturally focus at different depths and observe true-3D scenes.

  Figure 2 (image 2.jpg in VLM) illustrates this principle. An elemental image array (EIA) is displayed on a microdisplay panel. A microlens array is placed in front of this display. Each microlens projects its corresponding elemental image into space. The combined effect of these projected images, each showing a slightly different perspective, reconstructs a 3D image (represented by the red apple in the figure). The voxel represents a volumetric pixel in the reconstructed 3D space.

  该图像是一个示意图，展示了通过显示面板、透镜阵列和体素重建三维图像的过程。图中涉及的元素包括元素图像阵列、显示面板和重构的三维图像，一个红色苹果的形象通过光线的分布实现了三维效果。

  Figure 2. Working principle of the light field display.

• Vergence-Accommodation Conflict (VAC): This is a fundamental problem in conventional stereoscopic 3D displays, including most VR headsets. Vergence is the inward or outward rotation of the eyes to fixate on an object at a certain distance. Accommodation is the eye's ability to change its focal length (by adjusting the shape of the lens) to bring an object at a specific distance into sharp focus on the retina. In stereoscopic 3D, separate images are presented to each eye to create the illusion of depth (vergence cues). However, the virtual image is typically fixed at a constant distance (e.g., 2 meters), so the eye's accommodation remains fixed at that distance, regardless of where the vergence cues suggest an object is. This conflict between vergence and accommodation leads to visual fatigue, discomfort, and limits the realism of the 3D experience. VAC-free displays, like LFDs, resolve this by providing natural focus cues, allowing the eye to accommodate naturally to different depths.

• Field-Sequential-Color (FSC) Micro-LCD: This is a type of liquid crystal display (LCD) that achieves full color without a traditional color filter array (CFA). Instead, it rapidly displays successive monochrome images in red, green, and blue (RGB) light. The human eye's visual persistence (the ability of the retina to retain an image for a short period after its removal) blends these rapidly changing color fields into a single, full-color image. By removing the CFA, each pixel can display any color at full resolution, effectively tripling the perceived spatial resolution compared to a display with RGB subpixels. It also significantly increases optical efficiency because color filters absorb a considerable amount of light.

• Microlens Array (MLA): An MLA is a sheet containing a regular grid of very small lenses (microlenses). In LFDs, it is placed in front of a microdisplay. Each microlens acts as a small projector, displaying a portion of the overall light field. The design of the MLA (e.g., lens pitch, focal length, shape) is critical for the performance of the LFD in terms of resolution, depth range, and viewing angle.

• Telecentric Path (Object-Space Telecentric): An object-space telecentric optical system is one where the chief rays (the central rays from object points) are parallel to the optical axis in the object space. This is achieved by placing the aperture stop (the component that limits the diameter of the light bundle, often the eye pupil in near-eye displays) at the image-space focal point of the lens system. The primary benefit in display systems is that it makes the magnification constant regardless of the object's distance from the lens, and, crucially for LFDs, it ensures that light rays enter the MLA at near-perpendicular angles, even for off-axis (large FOV) points. This minimizes aberrations that typically arise from highly oblique (angled) rays passing through lenses.

• Modulation Transfer Function (MTF): MTF is a measure of an optical system's ability to transfer contrast from the object to the image at different spatial frequencies. In simpler terms, it quantifies how well an optical system can reproduce fine details. A higher MTF value at a given spatial frequency indicates better image quality (sharper details). MTF is often plotted as a curve, showing how contrast decreases with increasing spatial frequency. A system with good MTF maintains high contrast even for very fine patterns.

• Elemental Image Array (EIA): In an LFD, the EIA is the pattern displayed on the microdisplay. It consists of many small elemental images, each corresponding to a microlens in the MLA. Each elemental image is a miniature, slightly different perspective of the 3D scene. When viewed through the MLA, these elemental images combine to reconstruct the light field and the 3D scene.

• Reconstructed Depth Plane (RDP): In an LFD, the RDP refers to the specific depth plane in the 3D space where the image is reconstructed in sharp focus. By computationally manipulating the EIA, the LFD can reconstruct images at various RDPs, providing depth cues and allowing natural accommodation.

• Central Depth Plane (CDP): The CDP is a specific RDP in an LFD. It is the depth plane where the LFD engine inherently achieves its highest resolution and best image quality, typically corresponding to the native image plane of the microlens array. As the RDP moves further away from the CDP (either closer or farther), the image quality (resolution) of the LFD tends to decrease due to optical defocus and magnification effects.

3.2. Previous Works

The paper discusses several existing technologies and their limitations, forming the backdrop for its proposed solution:

• Fixed Virtual Image Distance Pancake VR: Most current Pancake VR headsets (e.g., as mentioned in [1], [2]) provide compact form factors and large FOVs. However, they typically offer only a fixed virtual image distance, which inevitably leads to the VAC. This is the primary problem the paper aims to solve.
• Mechanical Movement/Varifocal Elements in Pancake VR: One approach to achieve depth variability in Pancake VR is mechanically moving the lenses. However, this is complex, slow, and cannot support multiple focal planes simultaneously. Another approach involves inserting varifocal elements like LC lenses [2], which can adjust the diopter (focus distance) but still cannot present true-3D scenes with multiple objects at different depths simultaneously, as they only shift the single focal plane.
• Maxwellian View Display: These displays [3] project images directly onto the retina, ensuring always-in-focus retinal images regardless of accommodation, thereby resolving VAC. However, they are highly dependent on a fixed pupil position, leading to a very restricted eyebox (the area where the eye can see the full image), which is impractical for comfortable VR use.
• Holographic Display: Holographic displays [4] are theoretically ideal for true-3D as they record and reproduce complete wavefronts, including phase information, offering full depth cues. However, they typically require coherent light sources (like lasers), which historically have made the optical systems bulky and challenging to integrate into compact near-eye displays. Recent advancements, such as integrating AI-driven digital holography with metasurface waveguides [4], are making holographic AR glasses more compact, but a VAC-free VR solution with more affordable sources is still needed.
• Direct Near-Eye LFDs: Light Field Displays (LFDs) [5] offer computational focus cues and feasible hardware. However, directly using a microdisplay with an MLA as a near-eye display suffers from two major drawbacks:
  • Resolution Drop: The MLA significantly magnifies pixels [6], leading to a sharp drop in visual resolution.
  • FOV Limitation: The FOV is severely limited by aberrations (optical distortions) induced by oblique rays passing through the MLA, especially since MLAs usually have spherical profiles [7], [12].
• Freeform Prism-based LFD AR: Hua et al. [8] proposed a FOV-expanded near-eye LFD combined with a freeform prism and a tunable lens. While addressing FOV, freeform prism-based VR architectures tend to be bulkier than the compact Pancake solutions prevalent today, making them less suitable for lightweight VR.

3.3. Technological Evolution

The field of VR displays has evolved significantly from simple stereoscopic displays to more sophisticated VAC-free solutions. Early VR focused on achieving a wide FOV and basic 3D perception through stereoscopy. The challenge of VAC soon became apparent, leading to research into solutions like varifocal displays, Maxwellian displays, and holographic displays.

Pancake optics emerged as a key technology to achieve compact and lightweight headsets with large FOVs, addressing ergonomic concerns. However, Pancake optics traditionally retained the VAC issue. Simultaneously, Light Field Displays (LFDs) developed as a promising approach for true-3D by synthesizing light fields and providing natural depth cues, but struggled with resolution and FOV when implemented directly as near-eye displays.

This paper represents a crucial step in this evolution by attempting to merge the best aspects of Pancake optics (compactness, wide FOV) with LFDs (true-3D, VAC-free). It positions itself as a solution that builds upon the compactness of Pancake designs while introducing the sophisticated depth cues of LFDs, overcoming the inherent limitations of each technology when used in isolation. The integration of FSC micro-LCDs further pushes the boundaries of resolution and efficiency in such combined systems.

3.4. Differentiation Analysis

Compared to the main methods in related work, this paper's approach offers several core differences and innovations:

• Unique Combination: Unlike previous Pancake VR systems that only offered fixed virtual distances or limited varifocal capabilities, this paper integrates a full LFD engine. This is the first reported Pancake VR system that leverages an LFD engine to generate intermediate images with computational focus cues for true-3D.
• VAC-Free with Pancake Compactness: It is one of the few solutions that aims to deliver a VAC-free true-3D experience while retaining the highly desired compact and lightweight form factor of Pancake optics. This differentiates it from bulkier freeform prism-based LFDs [8] or holographic displays that may compromise on compactness.
• FOV Enhancement for LFDs: The paper directly tackles the critical FOV limitation of LFDs by using the telecentric optical path of the Pancake module. Instead of complex freeform optics [8] or multiple MLA dithering, it leverages an existing advantageous feature of Pancake optics to ensure near-paraxial rays through the MLA, significantly expanding the useful FOV.
• High-Resolution and Efficient Display Source: The adoption of a Field-Sequential-Color (FSC) micro-LCD provides a native resolution triple that of traditional RGB subpixel displays, without the need for complex mechanical dithering [9]. This, combined with the higher optical efficiency due to the absence of color filters, is a significant improvement for LFDs where resolution and light throughput are critical.
• Optimized System Integration: The paper goes beyond simply combining components by proposing a detailed image quality matching strategy between the Pancake and LFD engine. This systematic approach to balancing image quality across multiple depth planes accounts for the individual optical characteristics of each module, leading to a more robust and optimized system performance.

4. Methodology

4.1. Principles

The core idea of this method is to overcome the limitations of both Pancake optics (fixed focus, VAC) and Light Field Displays (LFDs) (limited FOV, resolution drop) by integrating them synergistically. The LFD engine is designed to be the true-3D display component, generating intermediate images that inherently carry computational focus cues and parallax information. These intermediate images effectively create multiple, adjustable virtual image depths. The Pancake module then acts as a relay optic, taking these intermediate images and presenting them to the user with its characteristic compactness and wide Field of View (FOV).

A key principle in this integration is to leverage the object-space telecentric path of the Pancake module. By placing the LFD engine in this telecentric path, the rays passing through the LFD's microlens array (MLA) become nearly paraxial (parallel to the optical axis), even for off-axis (large FOV) views. This minimizes the aberrations that typically limit the FOV of standalone LFDs. Furthermore, a high-resolution Field-Sequential-Color (FSC) micro-LCD is used within the LFD engine to mitigate the inherent resolution sacrifice of LFDs. Finally, a careful matching strategy is employed to balance the optical performance (specifically, Modulation Transfer Function (MTF)) of both the Pancake and LFD engine across the range of reconstructed depth planes (RDPs).

Figure 3 (image 3.jpg in VLM) visually represents this principle. The LFD engine (on the left) generates an intermediate image which includes depth cues. This image is then fed into the Pancake module (on the right), which relays it to the eye. The LFD engine comprises a microdisplay and a microlens array (MLA). The Pancake module uses a polarizing beam splitter and quarter-wave plate elements to fold the optical path and present the image to the eye. The dashed lines show the light path, indicating how the intermediate image from the LFD is processed by the Pancake to create the final true-3D virtual image for the observer.

该图像是一个示意图，展示了光场3D引擎与煎饼模块的光学结构。图中描绘了中间图像的生成路径，强调了光场显示引擎如何通过一个光学系统产生清晰的3D图像。红色方块表示光的传播路径和关键部件。

Figure 3. Proposed VAC-free Pancake using an LFD engine.

4.2. Core Methodology In-depth (Layer by Layer)

The methodology involves three main aspects: microdisplay selection for resolution, optical design for FOV expansion, and system-level matching for balanced image quality.

4.2.1. Microdisplay Panel

The resolution of Light Field Displays (LFDs) is inherently limited because the pixels on the microdisplay must encode both spatial (positional) and angular (directional) information. This means that a single pixel on the display contributes to a specific ray, rather than just a point in space.

To overcome this inherent resolution sacrifice, the authors adopt a 2.1-inch field-sequential-color (FSC) micro-LCD.

• High Resolution: This specific FSC micro-LCD boasts a 2.3K-by-2.3K resolution. By comparison, traditional LCDs use subpixels (e.g., separate red, green, and blue subpixels for each perceived pixel) which effectively reduces the addressable resolution.
• Color Filter Removal: In FSC LCDs, the color filter array (CFA) is removed. Instead of having dedicated RGB subpixels with color filters that absorb a significant portion of light, the display rapidly cycles through full-screen red, green, and blue illumination, synchronized with the display content. The visual persistence of the human eye then fuses these rapidly displayed monochromatic subframes into a full-color image.
• Resolution Tripling: The removal of subpixels means that each physical pixel on the LCD can be used for any color, effectively tripling the perceived spatial resolution compared to an RGB subpixel display of the same physical pixel count.
• Optical Efficiency: Eliminating the color filter array also means a significantly increased optical efficiency because color filters typically block about two-thirds of the light. This enhanced efficiency is particularly beneficial for Pancake optics, which are known for their relatively low light throughput due to multiple reflections and polarizing elements.
• Color Breakup Mitigation: The authors mention their previous work [11] on significantly suppressing the color breakup issue (a common artifact in FSC displays where rapid eye movements can separate the color components) using deep learning.

4.2.2. Expanded FOV

A significant challenge for LFDs used as near-eye displays is the limited Field of View (FOV). Directly placing a microdisplay with a microlens array (MLA) close to the eye results in severe aberrations for large fields (i.e., when looking at peripheral parts of the image). This is because MLAs typically have spherical profiles, and oblique beams (light rays entering at steep angles) passing through them experience strong distortions, quickly degrading image quality.

Figure 4 (image 6.jpg in VLM) illustrates this problem:

• Figure 4(a) shows a simulation model of a directly near-eye LFD.

• Figure 4(b) demonstrates how visual resolution decreased with field (angle) due to aberration. The PPD (Pixels Per Degree), a measure of visual resolution, drops sharply as the field angle increases.

• Figure 4(c) displays PSFs (Point Spread Functions) at different fields. The PSF describes how a point of light is rendered by the optical system; a larger, more spread-out PSF indicates worse image quality. It shows that no image can be formed when the FOV exceeds 10 degrees (unilateral) due to severe degradation.

  该图像是图表，展示了宽视场3D Pancake VR中的光场显示引擎的分辨率随视场角变化的趋势。在图(b)中，分辨率（PPD）与视场（度）之间的关系以曲线形式呈现，图(c)则显示了不同视场角下的光斑分布情况。

  Figure 4. (a) Simulation model of a directly near-eye LFD; (b) visual resolution decreased with field to demonstrate the FOV limited by aberration; (c) PSFs of different fields.

To address this aberration-induced FOV limitation, the paper leverages the object-space telecentric optical path commonly found in modern Pancake optics.

• Object-Space Telecentric Path: In a telecentric system, the chief rays (rays passing through the center of the aperture stop) are parallel to the optical axis in either the object space or image space. For an object-space telecentric path, this means that chief rays from all points on the object plane (in this case, the LFD's microlens array) enter the subsequent optical system (the eye) parallel to the optical axis. This condition is achieved by positioning the aperture stop (which is the eye pupil in near-eye displays) at the image-space focal point of the lens module.

• Benefit for LFD: By placing the LFD engine within this telecentric path of the Pancake module, all lenslets (microlenses) within the MLA work with near-paraxial rays. This means that even light from the edges of the microdisplay (corresponding to large Field of View angles) passes through the MLA at relatively shallow angles, significantly reducing aberrations that would otherwise occur with oblique rays. This strategy is crucial for ensuring low aberrations and maintaining image quality across a large FOV.

  Figure 5 (image 7.jpg in VLM) illustrates the object-space telecentric path of the Pancake and its benefit:

• Figure 5(a) shows a typical Pancake model in Zemax simulation. The telecentric path is achieved by placing the aperture stop (representing the eye pupil) at the image-space focal point of the Pancake lens module.

• Figure 5(b) visually represents how the telecentric path ensures that chief rays (dashed lines) from the LFD engine enter the Pancake module parallel to the optical axis. This, in turn, ensures that these rays pass through the MLA at near-perpendicular angles, preventing severe aberrations and expanding the usable FOV.

  该图像是一个示意图，展示了通过FSC-LCD生成中间图像的过程，其中标示了光路从微型液晶显示器(Micro-LCD)经过MLA至瞳孔的路径，说明了Pancake VR系统的结构与工作原理。

  Figure 5. The object-space telecentric path of Pancake and its benefit in suppressing the aberrations induced oblique rays through MLA in the LFD engine.

4.2.3. Matching between Pancake and the LFD Engine

A Pancake module is typically optimized for a specific virtual image distance. When the LFD engine varies its virtual image distance (by adjusting the position of its intermediate image), residual aberration can occur in the Pancake module. To address this, the authors analyze and match the image quality variations of both components.

• Pancake MTF Variation: The Modulation Transfer Function (MTF) of the Pancake system varies with the virtual image distance. This is simulated using Zemax, considering the conic profile of the Pancake's optical surfaces.

  • Figure 6(a) (image 4.jpg in VLM) shows that the MTF of the Pancake changes non-negligibly across different virtual image distances.
  • Due to the difficulty in accurately modeling commercial Pancake optics, MTFs are acquired by placing a microdisplay at various positions relative to the Pancake's native object plane. MTFs are calculated from a knife edge scan. The blue solid line in Figure 6(b) shows the MTF for object planes to the left of the native object plane. The blue dashed line predicts the MTF for object planes to the right, where the microdisplay cannot physically be submerged into the Pancake module.

• LFD Engine MTF Variation: The reconstructed depth plane (RDP) provided by the LFD engine achieves its highest resolution at the MLA's native image plane, which is defined as the central depth plane (CDP). As the RDP moves away from the CDP (either closer or farther), the image quality of the LFD decreases due to defocus and changes in transverse magnification, which influences the effective voxel size on the RDP.

  The LFD-determined MTF is given by Equation (1): $ \begin{array}{rl} & {\mathrm{MTF} = \left{\hat{\mathbf{P}} (\mathbf{s},\mathbf{t})\otimes \hat{\mathbf{P}} (\mathbf{s},\mathbf{t})\right} \cdot \mathrm{sinC}\left(\frac{\mathbf{g}}{\mathrm{P}}\right),}\ & {\mathrm{where}\hat{\mathbf{P}} (\mathbf{s},\mathbf{t}) = \mathbf{P}(\mathbf{s},\mathbf{t})\mathrm{exp}\left[\mathrm{i}\mathbf{k}\cdot \left(\frac{\mathbf{G}}{\mathrm{CDP}} -\frac{\mathbf{1}}{\mathrm{I_{RDP}}}\right)\frac{\mathbf{s}^{2} + \mathbf{t}^{2}}{2}\right]} \end{array} \quad (1) $ Where:

• $\mathrm{MTF}$ is the Modulation Transfer Function.

• $\hat{\mathbf{P}} (\mathbf{s},\mathbf{t})$ represents the complex pupil function at pupil coordinates $(\mathbf{s},\mathbf{t})$. It contains information about the optical system's phase and amplitude response.

• $\otimes$ denotes the convolution operator. The term $\{\hat{\mathbf{P}} (\mathbf{s},\mathbf{t})\otimes \hat{\mathbf{P}} (\mathbf{s},\mathbf{t})\}$ effectively represents the Autocorrelation of the pupil function, which is proportional to the Optical Transfer Function (OTF). The MTF is the magnitude of the OTF.

• $\mathrm{sinC}\left(\frac{\mathbf{g}}{\mathrm{P}}\right)$ is the sinc function, which accounts for the discrete nature of pixels and the sampling effect. This term models the MTF degradation due to the pixel pitch and the voxel size.

  • $\mathrm{sinC}(x) = \frac{\mathrm{sin}(\pi x)}{\pi x}$.
  • $\mathbf{g}$ is related to the voxel size or the sampling grid on the Reconstructed Depth Plane (RDP).
  • $\mathrm{P}$ is the pixel pitch of the MLA or the effective pixel pitch on the display.

• $\mathbf{P}(\mathbf{s},\mathbf{t})$ is the pupil function itself, representing the transmission characteristics of the MLA at pupil coordinates $(\mathbf{s},\mathbf{t})$.

• $\mathrm{exp}\left[\mathrm{i}\mathbf{k}\cdot \left(\frac{\mathbf{G}}{\mathrm{CDP}} -\frac{\mathbf{1}}{\mathrm{I_{RDP}}}\right)\frac{\mathbf{s}^{2} + \mathbf{t}^{2}}{2}\right]$ is a phase term that accounts for defocus when the Reconstructed Depth Plane (RDP) is not at the Central Depth Plane (CDP).

  • $\mathrm{i}$ is the imaginary unit ($\sqrt{-1}$).

  • $\mathbf{k}$ is the wave vector (related to the wavelength of light).

  • $\mathbf{G}$ is a constant related to the MLA properties (e.g., focal length).

  • $\mathrm{CDP}$ is the distance to the Central Depth Plane.

  • $\mathrm{I_{RDP}}$ is the distance to the Reconstructed Depth Plane.

  • $\mathbf{s}^{2} + \mathbf{t}^{2}$ represents the squared radial distance in the pupil plane. This quadratic term models the parabolic wavefront curvature associated with defocus.

    The red line in Figure 6(b) (image 4.jpg in VLM) shows the LFD-determined MTF as predicted by this equation, indicating how image quality drops as the RDP moves away from the CDP.

• Compromised Configuration: Since both the LFD and Pancake have varying image quality across different depth planes, a compromised configuration is adopted. This means the LFD engine's CDP is intentionally aligned with a Pancake object plane that might not represent the Pancake's absolute best MTF but offers a better overall balance across the entire range of virtual image distances that the system needs to produce. This ensures that no single depth plane has exceptionally high quality while others are unacceptably poor, leading to more consistent true-3D viewing.

  该图像是插图，包含两部分内容。部分 (a) 展示了不同距离（0.1m, 0.5m, 1m, 2m）下的调制传递函数（MTF）随空间频率变化的曲线图。部分 (b) 左侧是一个示意图，描述了光场显示器的光学路径，右侧是不同距离下（以 CDP 为基准）的分辨率变化曲线（包括模拟和实验数据）。底部为10个不同图像的排列，可能展示实验结果或图像清晰度的比较。

  Figure 6. (a) MTF varying with the virtual image distance of the Pancake. (b) Image quality matching between the Pancake and the LFD engine.

4.2.4. Image Rendering for the LFD Engine

The depth of the Reconstructed Depth Plane (RDP) in the LFD engine is controlled by how the Elemental Image Array (EIA) is rendered.

• Viewpoint-based Projection: A standard rendering method is viewpoint-based projection. In this approach, each lenslet (microlens) in the MLA is conceptually treated as a virtual camera. The 3D target scene is then rendered from the perspective of each of these virtual cameras, generating the corresponding elemental images.
• Ray Manipulation: When this EIA is displayed on the microdisplay, the MLA optically manipulates the directions of the light rays originating from these elemental images. This manipulation causes the rays to converge or diverge in such a way that they inversely project the elemental images onto the desired specific depth plane (the RDP), creating the illusion of a true-3D object at that depth.
• Accelerated Rendering: The authors also reference their previous work [13] on an accelerated rendering method that reduces computational complexity, suggesting that real-time rendering of these EIAs for dynamic 3D scenes is a key consideration.

5. Experimental Setup

5.1. Datasets

The paper does not use a traditional dataset for training or evaluation in the machine learning sense. Instead, for experimental verification, a specific sample scene was generated and displayed. This scene contained two distinct objects located at two different virtual depths. This setup was chosen to demonstrate the true-3D capability of the system, specifically its ability to render multiple focal planes simultaneously and allow for selective focusing.

• Sample Data Illustration: Figure 7(b) (image 5.jpg in VLM) shows the Elemental Image Array (EIA) generated for this sample scene. This EIA contains the encoded parallax information for two objects at different depths. One object appears to be a red cat, and the other is a blue cat. The differences in the elemental images across the array encode the depth information for these two objects.

  该图像是图示，展示了宽视场3D Pancake VR 技术的原理和应用示例。图 (c) 和 (d) 展示了不同焦点下的图像效果，分别为清晰和模糊的卡通猫咪图像，表明 FOV 为 68.6 度，$FOV: 68.6^ ext{°}$。

  Figure 7. (a) Experimental setup; (b) EIA of the sample scene; (c) and (d) reconstructed images on two depth planes and the measured FOV.

5.2. Evaluation Metrics

The primary evaluation metrics used in this paper are:

1. Field of View (FOV):

   • Conceptual Definition: Field of View refers to the extent of the observable world that is seen at any given moment. In VR displays, it's the angular size of the displayed virtual world visible to the user. A larger FOV contributes to a more immersive experience.
   • Mathematical Formula: FOV is typically measured in degrees. For a given display and optical system, it's related to the display size and the effective focal length of the optics. For a simple optical system, the FOV can be approximated by: $ \mathrm{FOV} = 2 \cdot \mathrm{atan}\left(\frac{H}{2 \cdot f}\right) $
   • Symbol Explanation:
     • $H$: The horizontal dimension of the display (or the image projected to the eye).
     • $f$: The effective focal length of the optical system.
     • $\mathrm{atan}$: The arctangent function.
   • Measurement in Paper: In this paper, the FOV was measured experimentally using a smartphone camera. By capturing images through the Pancake module and using the camera's known specifications (focal length) and the size of the picture on the image sensor, the angle subtended by the displayed image could be calculated.

2. Image Clarity/Sharpness (Qualitative & Quantitative via MTF):

   • Conceptual Definition: This refers to how well fine details and contrast are preserved in the displayed image. For true-3D displays, it also implies the ability to render objects at different depths with appropriate focus. Modulation Transfer Function (MTF) is a quantitative measure of this.
   • Mathematical Formula (MTF): While the paper provides a formula for LFD-determined MTF (Equation 1), a general formula for MTF (which is the magnitude of the Optical Transfer Function, OTF) is given by: $ \mathrm{MTF}(f_x, f_y) = |\mathrm{OTF}(f_x, f_y)| $ Where the OTF is the Fourier Transform of the Point Spread Function (PSF): $ \mathrm{OTF}(f_x, f_y) = \mathcal{F}{\mathrm{PSF}(x, y)} $
   • Symbol Explanation:
     • $\mathrm{MTF}(f_x, f_y)$: The Modulation Transfer Function at spatial frequencies $f_x$ and $f_y$.
     • $\mathrm{OTF}(f_x, f_y)$: The Optical Transfer Function at spatial frequencies $f_x$ and $f_y$.
     • $\mathrm{PSF}(x, y)$: The Point Spread Function, which describes the optical system's response to a point source of light in the spatial domain (x, y).
     • $\mathcal{F}\{\dots\}$: The Fourier Transform operator.
     • $|\dots|$: The magnitude operator.
   • Measurement in Paper: The paper qualitatively assesses image clarity by capturing photographs focused at different depth planes (Figure 7c and 7d). Quantitatively, MTF was simulated using Zemax for the Pancake module and calculated using Equation (1) for the LFD engine. The Pancake's MTF was also acquired experimentally from a knife edge measurement.

5.3. Baselines

The paper implicitly compares its proposed system against several existing approaches by highlighting their limitations in the introduction, rather than conducting direct comparative experiments with full baseline systems. These implicit baselines include:

• Conventional Pancake VR: This serves as a baseline for compactness and FOV, but is limited by VAC and fixed virtual image distance. The paper's system aims to maintain the advantages of Pancake while adding true-3D capability.

• Direct Near-Eye LFDs: These are baselined for their VAC-free capability but are limited by severe FOV reduction due to MLA aberrations and resolution issues. The paper explicitly states that its 68.6-degree FOV is "significantly larger than the LFD engine used alone."

• Pancake with Mechanical Varifocal Elements: These are mentioned as complex, slow, and unable to support multiple focal planes simultaneously, which the proposed LFD-Pancake system addresses.

• Maxwellian View Displays: Acknowledged for their VAC-free nature but limited by a restricted eyebox, which the LFD-Pancake system avoids.

• Holographic Displays: Praised for true-3D but often criticized for bulkiness and requiring coherent sources, which the proposed system aims to avoid while still offering true-3D.

• Freeform Prism-based LFD AR: Mentioned as a FOV-expanded LFD but noted for inducing "bulkier volume than today's Pancake solution," contrasting with the proposed compact design.

  The experimental results demonstrate that the proposed system achieves a wide FOV similar to Pancake optics while also providing true-3D and computational focus cues, thereby overcoming the key limitations of the aforementioned baseline approaches.

6. Results & Analysis

6.1. Core Results Analysis

The authors built a prototype to experimentally verify their Wide-FOV 3D Pancake VR system.

• Prototype Components:

  • FSC Micro-LCD: A 1500-ppi (pixels per inch) Field-Sequential-Color (FSC) micro-LCD based on a mini-LED backlight. This choice underpins the high resolution and optical efficiency discussed in the methodology.
  • Microlens Array (MLA): An MLA with a 1-mm lens pitch.
  • Pancake Module: A commercial Pancake module.

• Experimental Setup:

  • Figure 7(a) (image 5.jpg in VLM) displays the experimental setup.
  • A critical design parameter was the placement of the Pancake module's designed object plane at 6 mm from the LFD's Central Depth Plane (CDP). This specific distance was chosen to achieve optimal image quality, based on the image quality matching analysis presented in Section 2.3.
  • The LFD engine introduced an additional optical track (physical length) of 2.1 cm. The authors consider this an acceptable trade-off for the added true-3D functionality in near-eye displays.

• EIA and Depth Planes:

  • Figure 7(b) (image 5.jpg in VLM) shows the Elemental Image Array (EIA) for a sample scene containing two objects (a red cat and a blue cat) at different virtual depths.
  • The LFD engine reconstructs these objects at two distinct intermediate image planes:
    • The first plane is 9.7 mm from the MLA, corresponding to the CDP of the LFD. This object is intended to be in the foreground.
    • The second plane is positioned 16 mm from the MLA. This object is intended for the background.
  • The intermediate RDP for the background object, though slightly out-of-focus from the LFD's CDP, was intentionally placed on a Pancake object plane identified to have better MTF during the image quality matching process, showcasing the deliberate compromise for balanced overall performance.

• True-3D Verification (Computational Focus Cues):

  • A smartphone camera (with a focal length of 5.5 mm) was used to capture virtual images through the Pancake module.
  • Figure 7(c) demonstrates the system's ability to focus on the foreground object (the red cat, reconstructed at the CDP). The camera is focused on this object, showing sharp details, while the background object (the blue cat) appears blurred with visible subviews (artifacts of LFD reconstruction when out of focus).
  • Figure 7(d) shows the camera refocused on the background object (the blue cat, reconstructed at 16 mm from the MLA). This object now appears sharper, while the foreground object (red cat) becomes blurred.
  • This experiment clearly verifies computationally adjustable virtual image distances, demonstrating the true-3D feature of the system, where the user's eye (or the camera in this case) can naturally accommodate to different virtual depths within the scene.

• Field of View (FOV) Measurement:

  • Using the camera's specifications and the captured picture size on its image sensor, the FOV was measured to be 68.6 degrees.

  • This FOV is close to the original Pancake module's inherent FOV, confirming that the integration of the LFD engine did not significantly compromise the wide viewing angle provided by the Pancake optics.

  • Crucially, this 68.6-degree FOV is significantly larger than what a direct near-eye LFD engine used alone could achieve (as shown in Figure 4, where LFDs alone were limited to about 10 degrees unilateral FOV before severe degradation), validating the effectiveness of the telecentric path strategy.

    The results strongly validate the effectiveness of the proposed method in achieving a true-3D Pancake VR headset with a wide FOV and computational focus cues, overcoming the limitations of previous approaches.

6.2. Data Presentation (Tables)

The paper does not contain any data presented in tabular format within the results section. All quantitative and qualitative results are discussed in the text and supported by figures.

6.3. Ablation Studies / Parameter Analysis

The paper does not explicitly present separate ablation studies in the results section, where specific components of the proposed system are removed or altered to quantify their individual contribution. However, the Methodology section (specifically 2.3 Matching between Pancake and the LFD engine) implicitly performs a parameter analysis and design optimization that serves a similar purpose:

• Pancake MTF vs. Virtual Image Distance (Figure 6a): This analysis explores how the Pancake module's image quality (represented by MTF) changes as the virtual image distance varies. This is a crucial parameter analysis for understanding the Pancake's performance characteristics and limitations when used with a dynamic LFD engine.

• LFD MTF vs. Reconstructed Depth Plane (Figure 6b) & Equation (1): The analysis of how the LFD engine's MTF degrades as the Reconstructed Depth Plane (RDP) moves away from the Central Depth Plane (CDP) is a form of parameter analysis for the LFD component. Equation (1) models this relationship.

• Image Quality Matching (Figure 6b): The decision to utilize a "compromised configuration" where the LFD's CDP intentionally uses a relatively worse object plane of the Pancake is a direct result of this parameter analysis. It demonstrates how the authors optimized the system by balancing the performance curves of both components to achieve acceptable image quality across the entire depth range, rather than optimizing for a single, perfect depth. This design choice is a direct consequence of understanding how key parameters (image depth) affect the individual components and the overall system.

  These analyses are foundational to the system's design and demonstrate that components' performance and interactions are well understood and accounted for, even if not presented as a formal ablation study.

7. Conclusion & Reflections

7.1. Conclusion Summary

This paper successfully presents a novel true-3D Pancake VR headset by ingeniously combining a light field display (LFD) engine with a Pancake module. The system addresses the critical vergence-accommodation conflict (VAC) by enabling computational focus cues and variable virtual image distances. Key to its high performance is the use of a field-sequential-color (FSC) micro-LCD, which ensures high resolution and improved optical efficiency by removing color filters. The persistent problem of aberration-induced FOV reduction in LFDs is effectively mitigated by strategically integrating the LFD engine into the object-space telecentric path of the Pancake optics. Furthermore, the authors implemented a careful image quality matching strategy to achieve balanced image clarity across multiple depth planes. A prototype experimentally validated the system, demonstrating sharp images at two distinct depth planes and achieving a wide FOV of 68.6 degrees, which significantly surpasses standalone LFDs. The integration did result in an additional optical track length of 2.1 cm, considered an acceptable trade-off.

7.2. Limitations & Future Work

The authors explicitly mention one limitation:

• Increased Optical Track: The integration of the LFD engine introduced an additional optical track length of 2.1 cm. While deemed "acceptable" by the authors, this still represents an increase in the physical size of the optical module, which is typically a critical parameter for compact VR headsets. Further miniaturization efforts could be a potential future research direction.

  The paper does not explicitly outline future work. However, based on the discussion, potential future research directions could include:

• Further Miniaturization: Reducing the additional optical track length (2.1 cm) while maintaining or improving performance would be a valuable area of research.

• Dynamic Range of Focus Cues: While two depth planes were demonstrated, exploring the practical limits and quality of a broader range of focus cues for more complex 3D scenes could be a next step.

• Rendering Optimization: The paper references prior work on accelerated rendering [13]. Further advancements in real-time, high-fidelity EIA rendering for complex and dynamic light fields would be crucial for a practical VR experience.

• Human Factors Evaluation: Conducting comprehensive user studies to evaluate visual comfort, presence, and long-term effects of VAC-free Pancake VR would be important for commercialization.

• Manufacturing and Cost: Optimizing the design for mass manufacturability and reducing production costs of the specialized FSC micro-LCDs and precisely aligned MLA/Pancake modules.

• Color Breakup Mitigation in FSC Displays: While the authors mention previous work on mitigating color breakup using deep learning [11], continuous improvement in this area remains important for FSC displays to ensure a flawless visual experience.

7.3. Personal Insights & Critique

This paper presents a highly innovative and practical approach to addressing the vergence-accommodation conflict (VAC) in VR, a long-standing challenge. The core strength lies in the intelligent combination of two powerful optical technologies: the true-3D capability of light field displays (LFDs) and the compactness and wide FOV of Pancake optics. This synergistic approach not only leverages the strengths but also mitigates the weaknesses of each technology when used in isolation (e.g., LFD's limited FOV is overcome by Pancake's telecentric path).

The use of a Field-Sequential-Color (FSC) micro-LCD is a smart choice for enhancing resolution and optical efficiency, directly tackling the inherent pixel-sharing issue of LFDs. The detailed analysis of Modulation Transfer Function (MTF) variation for both components and the subsequent image quality matching strategy highlight a rigorous engineering approach to system design, ensuring balanced performance rather than a compromise in one area for the sake of another.

A potential area for critique or further investigation could be the practical implementation of the "compromised configuration" for image quality. While theoretically sound, the perceptual impact of intentionally "worsening" the CDP image quality for the sake of overall balance might need further subjective evaluation. Also, the 2.1 cm increase in optical track, while deemed acceptable, is still a design trade-off that will be scrutinized in the context of increasingly smaller and lighter VR headsets.

From a broader perspective, this work demonstrates how combining mature and emerging optical technologies in novel ways can lead to significant breakthroughs in near-eye display performance. The methods and conclusions are highly relevant to the entire $AR/VR/MR$ industry and could inspire similar hybrid optical designs that tackle specific limitations of current display technologies. The concept of using one optical system to enhance the fundamental weaknesses of another, particularly for FOV and depth cues, is broadly transferable. This paper provides a clear roadmap for developing VAC-free VR headsets that are both immersive and comfortable for prolonged use, moving closer to truly natural visual experiences in virtual environments.