GPU Resource Sharing

Consider a typical frame in a game or graphics application running at 60 FPS. You have 16.67 milliseconds to render the frame and run any ML inference. Let’s trace through what happens without GPU resource sharing:

Step 1: Render to texture (2ms) - Vulkan renders your scene to a texture. This is fast—the GPU is doing what it does best.

Step 2: Read texture to CPU (3-5ms) - You issue a vkCmdCopyImageToBuffer command, insert a pipeline barrier, and wait for the transfer to complete. For a 1920×1080 RGBA texture, that’s 8.3MB of data. Even with PCIe 3.0’s theoretical 16 GB/s bandwidth, real-world transfers are slower due to overhead, synchronization, and other bus traffic.

Step 3: ML inference on CPU or separate GPU context (5-10ms) - Your ML library processes the data. If it’s using the GPU, it has to upload the data again—another transfer.

Step 4: Read results back (1-2ms) - If the ML library produced GPU results, you need to read them back to CPU, then potentially upload to Vulkan again.

Total: 11-19ms. You’ve already blown your 16.67ms frame budget, and we haven’t even accounted for the actual rendering work or other game logic. The result? Dropped frames, stuttering, poor user experience.

With GPU resource sharing, the same workflow looks like this:

Step 1: Render to shared texture (2ms) - Vulkan renders to a texture that’s allocated with external memory support.

Step 2: Synchronize (<0.1ms) - Signal a semaphore or fence to indicate rendering is complete.

Step 3: ML inference on shared memory (5-10ms) - The ML library reads directly from the shared GPU memory. No transfer needed.

Step 4: Synchronize (<0.1ms) - Wait for inference to complete before using results.

Total: 7-12ms. You’ve eliminated 4-7ms of transfer overhead—enough to make the difference between hitting your frame budget and missing it. For real-time applications, this is the difference between feasible and infeasible.

GPU resource sharing also reduces memory usage. Without sharing, you need:

That’s at least 25MB for a single frame’s worth of data. With sharing, you need only the Vulkan texture memory—the ML library uses the same memory. For applications processing multiple frames in flight or working with higher resolutions, the memory savings multiply.

Several extensions work together to enable external memory:

VK_KHR_external_memory_capabilities - Queries what types of external memory the device supports. You use this during device selection to ensure your target platform supports the sharing mechanisms you need.

VK_KHR_external_memory - Core extension that defines the structures and concepts for external memory. This is the base that other extensions build on.

VK_KHR_external_memory_fd (Linux) - Enables sharing via POSIX file descriptors. This is the standard mechanism on Linux and Android.

VK_KHR_external_memory_win32 (Windows) - Enables sharing via Windows NT handles. This is the standard mechanism on Windows.

VK_EXT_external_memory_dma_buf (Linux) - Enables sharing via DMA-BUF, a Linux kernel mechanism for sharing buffers between devices. Particularly useful for embedded systems and when working with hardware video decoders.

VK_KHR_external_semaphore and VK_KHR_external_fence - Enable sharing synchronization primitives, which we’ll discuss later.

Before attempting to use external memory, you need to verify that your device supports it. This happens during device selection:

This query tells you whether the device supports exporting memory in the format and usage you need. Different devices have different capabilities—some might support external memory for images but not buffers, or support certain formats but not others.

Once you’ve confirmed support, you allocate memory with external memory support enabled. The key is adding VkExportMemoryAllocateInfo to the pNext chain when allocating memory:

The critical addition is VkExternalMemoryImageCreateInfo in the image creation and VkExportMemoryAllocateInfo in the memory allocation. These tell Vulkan that this memory needs to be shareable with external APIs.

After allocating exportable memory, you export a handle that other APIs can import:

The handle (NT handle on Windows, file descriptor on Linux) is an opaque reference to the GPU memory. You pass this handle to your ML library, which imports it into its own memory management system.

On Linux, the primary mechanism for sharing GPU memory is POSIX file descriptors. Vulkan exports memory as a file descriptor, which other APIs can import.

POSIX File Descriptors - The standard mechanism. File descriptors are integers that reference kernel objects. When you export Vulkan memory as a file descriptor, you’re getting a reference to the underlying kernel memory object. This works with OpenCL and most other GPU APIs on Linux.

DMA-BUF - A Linux kernel framework specifically designed for sharing buffers between devices. DMA-BUF is particularly important for embedded systems and when working with hardware video encoders/decoders. Vulkan supports DMA-BUF through VK_EXT_external_memory_dma_buf.

The key consideration on Linux is file descriptor ownership. File descriptors are reference-counted by the kernel—the memory isn’t freed until all file descriptors referencing it are closed. When you export a file descriptor from Vulkan, you’re responsible for closing it when you’re done (or passing that responsibility to the importing API).

On Windows, the mechanism is NT handles (also called Win32 handles). These are similar in concept to file descriptors but with Windows-specific semantics.

Opaque Handles - The most common type. These are handles that don’t expose the underlying memory directly—they’re references managed by the graphics driver. Use VK_EXTERNAL_MEMORY_HANDLE_TYPE_OPAQUE_WIN32_BIT_KHR.

Named Handles - Handles that have a name, allowing them to be shared across processes by name. Useful for inter-process communication but less common for ML inference scenarios.

Windows handles have different lifetime semantics than Linux file descriptors. When you export a handle, you can specify whether it should be duplicated (creating a new reference) or transferred (moving ownership). For most ML inference scenarios, you want duplication—both Vulkan and the ML library maintain references, and the memory is freed when both release their references.

On Apple platforms, the mechanism is IOSurface, a macOS/iOS framework for sharing surfaces between processes and APIs.

IOSurface is specifically designed for image data and provides efficient sharing between Metal (Apple’s graphics API), Core Image, Core Video, and other frameworks. While Vulkan on macOS/iOS (via MoltenVK) can work with IOSurface, the integration is more complex because you’re bridging between Vulkan and Metal under the hood.

For ML inference on Apple platforms, the typical approach is to use Core ML, which integrates naturally with IOSurface. If you’re using ONNX Runtime with Core ML execution provider, the runtime handles the IOSurface integration for you.

Android uses AHardwareBuffer, a framework for sharing buffers between different Android subsystems. AHardwareBuffer is built on top of Linux’s DMA-BUF but provides a higher-level, Android-specific API.

Vulkan on Android supports importing and exporting AHardwareBuffer through VK_ANDROID_external_memory_android_hardware_buffer. This is the standard mechanism for sharing GPU memory with Android’s ML frameworks (TensorFlow Lite with GPU delegate, NNAPI, etc.).

The key advantage of AHardwareBuffer is that it’s the native Android mechanism—all Android GPU APIs understand it, and the Android framework handles the platform-specific details.

ONNX Runtime’s OpenCL execution provider can import external memory using OpenCL’s external memory extensions:

Once you have the OpenCL buffer, you can create an ONNX Runtime tensor that wraps it:

The key insight: ONNX Runtime doesn’t copy the data. It runs inference directly on the shared GPU memory.

TensorFlow Lite’s GPU delegate can work with external memory on Android using AHardwareBuffer:

On desktop platforms, TensorFlow Lite’s GPU delegate integration is more limited. You might need to use OpenCL interop or fall back to CPU-GPU transfers.

DirectML, being a DirectX 12 API, integrates naturally with Vulkan on Windows through shared NT handles:

DirectML’s integration is particularly clean on Windows because both Vulkan and DirectML are built on top of the same underlying Windows graphics infrastructure.

GPUs are asynchronous. When you submit a command buffer to Vulkan, it returns immediately—the GPU executes the commands later. Similarly, when you launch an OpenCL kernel or DirectML operation, it returns immediately. Without synchronization, you have race conditions:

Synchronization primitives solve these problems by providing explicit control over execution order and memory visibility.

Vulkan provides two main synchronization primitives:

Semaphores - GPU-GPU synchronization. Semaphores coordinate work between queues. One queue signals a semaphore when work completes, another queue waits on that semaphore before starting work.

Fences - GPU-CPU synchronization. Fences allow the CPU to wait for GPU work to complete. You signal a fence from the GPU, then wait on it from the CPU.

For external synchronization, you need external semaphores and external fences—versions that can be shared with other APIs.

External semaphores work similarly to external memory:

You then import this semaphore into your ML library (e.g., OpenCL):

Now you can coordinate between Vulkan and OpenCL:

This creates a dependency chain: Vulkan renders → signals semaphore → OpenCL waits → OpenCL runs inference → signals semaphore → Vulkan waits → Vulkan uses results. All GPU-side, no CPU involvement, maximum performance.

Vulkan 1.2 introduced timeline semaphores, which are even more powerful. Unlike binary semaphores (signaled or unsignaled), timeline semaphores have a 64-bit counter value. You can wait for specific values and signal specific values, enabling more complex synchronization patterns.

Timeline semaphores are particularly useful for pipelining—running multiple frames in flight with ML inference on each frame, where you need to track which frame each operation corresponds to.

Vulkan and your ML library might have different expectations about memory layout. Vulkan images use implementation-defined tiling for optimal performance, while ML libraries often expect linear layouts (row-major, tightly packed).

For sharing to work, you typically need to use VK_IMAGE_TILING_LINEAR, which forces a linear layout that ML libraries can understand. The tradeoff is performance—linear tiling is slower for rendering than optimal tiling.

Alternatively, you can use optimal tiling for rendering and insert a layout transition + copy to a linear staging image before ML inference. This adds overhead but might be faster overall if the rendering performance gain outweighs the copy cost.

Vulkan and ML libraries might use different pixel formats. Vulkan commonly uses VK_FORMAT_B8G8R8A8_UNORM (BGRA), while ML models often expect RGB or planar YUV formats. You need to handle these conversions, either in a compute shader before inference or in preprocessing code.

Not all drivers support all external memory features. Some older drivers might not support external semaphores, or might support them only for certain memory types. Always check capabilities at runtime and have fallback paths.

Mobile platforms, in particular, have more limitations. Android’s AHardwareBuffer support varies by device and Android version. iOS’s IOSurface integration through MoltenVK has quirks.

When GPU resource sharing goes wrong, it’s often hard to debug. Symptoms include:

Use validation layers aggressively. Enable Vulkan validation layers, OpenCL’s built-in error checking, and any debugging tools your ML library provides. Insert explicit synchronization (fences, clFinish) during debugging to isolate timing issues.