Ray Traced Synthesis & Microlens Arrays

For high-fidelity holographic displays, we use Vulkan Ray Tracing (VRT) to simulate the exact physical optics of a Microlens Array (MLA).

Ecosystem Perspective

This chapter falls under the category: Beyond the OpenXR Standard.

Standard OpenXR runtimes are built around traditional projection models. You use Vulkan’s Ray Tracing pipelines to simulate complex virtual microlens optics, allowing for vergence-accommodation solutions and sparse lightfield traversal that exceed the capabilities of the standard runtime compositor.

Physical Optics: The Microlens Array (MLA)

To understand ray-traced synthesis, you must understand how lightfield cameras work. They use a Microlens Array—a grid of thousands of tiny lenses placed in front of a sensor. In our ray tracing pipeline, we reverse this process:

Primary Ray: We trace a ray from the virtual eye to the screen.
Microlens Intersection: We simulate the ray passing through a virtual microlens.
Refraction: We calculate how the lens bends the ray based on focal length and curvature.
LightField Lookup: The refracted ray now points to a specific 4D plenoptic coordinate.

The runtime provides the focal length, pitch, and curvature parameters of the physical MLA via vendor extensions, and may even provide a pre-built acceleration structure for the lens geometry.

Implementing the Virtual Lens in Slang

Using the RaytracingAccelerationStructure in Vulkan, we can simulate complex lenses that a raster shader cannot. We use designated initializers to configure the ray generation pass:

// Configuring the ray tracing pipeline using designated initializers
vk::RayTracingPipelineCreateInfoKHR pipelineInfo{
    .pNext = &dynamicRenderingInfo, // Link to our dynamic rendering state
    .stageCount = static_cast<uint32_t>(stages.size()),
    .pStages = stages.data(),
    .groupCount = static_cast<uint32_t>(groups.size()),
    .pGroups = groups.data(),
    .maxPipelineRayRecursionDepth = 1,
    .layout = *pipelineLayout
};

In our Slang RayGen shader, we perform the physical optics simulation:

// RayGen shader in Slang for plenoptic synthesis
[shader("raygeneration")]
void rayGenMain() {
    uint2 pixelID = DispatchRaysIndex().xy;
    float2 uv = float2(pixelID) / float2(DispatchRaysDimensions().xy);

    // 1. Trace a ray through the virtual microlens optics
    RayDesc ray = generateRayThroughLens(uv);

    // 2. Intersect the ray with our 4D LightField "box"
    float4 color = intersectLightField(ray, lightFieldData);

    // 3. Write the final result to the OpenXR swapchain image
    outputImage[pixelID] = color;
}

Advanced: Holographic Persistence and MLA Optimization

Vulkan allows you to push synthesis beyond standard view-dependent models:

Temporal Holographic Persistence: Using Vulkan’s Ray Tracing Motion Blur extensions, you can simulate the persistence of a holographic display by tracing rays across the predicted display window. This ensures moving objects remain sharp and stable.
Subgroup MLA Intersections: You can use Vulkan’s Subgroup Operations to implement highly efficient ray-circle or ray-hexagon intersection logic, sharing the computational cost of the lens simulation across multiple threads.

Why Ray Trace? Physical Fidelity

Ray tracing allows us to simulate Vergence-Accommodation Conflict (VAC) solutions. By adjusting the virtual focal length of our microlens array in real-time, we can change the "Plane of Focus" of the holographic image. This allows the user’s eyes to naturally focus on near or far virtual objects, eliminating eye strain.

Furthermore, ray tracing handles Non-Uniform LightFields (like octrees) much more efficiently than nested loops in a fragment shader.

For more information on ray tracing and holographic displays, consult the official Vulkan Specification on Ray Tracing, the Vulkan Guide, and our main tutorial series.

Previous | Next