VK_EXT_graphics_pipeline_library

For VK_KHR_ray_tracing_pipeline, pipelines contain a significant number of shaders - making monolithic compilation very slow. VK_KHR_pipeline_library allowed applications to create partial pipelines (pipeline libraries) containing only a subset of the final shaders. These pipeline libraries can be linked together to form a final executable. Ray pipelines were relatively straightforward as only shaders are linked, and there’s no “state” for ray shaders beyond the shader groups.

Graphics pipelines by comparison contain a lot of static state that needs to be separated carefully, retaining any “interface” information. However, this extension reuses the same underlying mechanism.

The following feature is exposed by this extension:

graphicsPipelineLibrary is the core feature enabling this functionality.

The following properties are exposed by this extension:

graphicsPipelineLibraryFastLinking indicates whether the cost of linking pipelines without VK_PIPELINE_CREATE_LINK_TIME_OPTIMIZATION_BIT_EXT is comparable to recording a command in a command buffer, such that applications can link pipelines on demand while recording commands. If this property is not supported, linking should still be cheaper than a full pipeline compilation.

If graphicsPipelineLibraryIndependentInterpolationDecoration is not supported, applications must provide matching interpolation decorations in both the last geometry stage and the fragment stage; if it is supported, any geometry stage decorations are ignored.

Four sets of state that have been identified as often recombined by applications are:

The intent is to allow each of those to be independently compiled as far as possible, along with relevant pieces of state that may need to match for the final linked pipeline.

Pipeline libraries are created for the parts specified, and any parameters required to create a library with those parts must be provided.

For all pipeline libraries VkPipelineCache, basePipelineHandle, basePipelineIndex, VkPipelineCreationFeedbackCreateInfo, and VkPipelineCompilerControlCreateInfoAMD parameters are independently consumed and do not need to match between libraries or for any final pipeline. VkPipelineCreateFlags are also independent, though VK_PIPELINE_CREATE_LIBRARY_BIT_KHR is required for all pipeline libraries. Only dynamic states that affect state consumed by a library are used, other dynamic states are ignored and play no part in linked pipelines. Where multiple pipeline libraries are built with the same required piece of state, those states must match exactly when linked together.

The subset of VkGraphicsPipelineCreateInfo used to compile each kind of pipeline library is listed in the following sections, along with any pitfalls, quirks, or interactions that need calling out. Any state not explicitly listed for a particular library part will be ignored when compiling that part.

To allow descriptor sets to be independently specified for each of the two shader library types, a new pipeline layout create flag is added:

When specified, fragment and pre-rasterization shader pipeline libraries only need to specify the descriptor sets used by that library. Descriptor set layouts unused by a library may be set to VK_NULL_HANDLE.

Linking is performed by including the existing VkPipelineLibraryCreateInfoKHR structure in the pNext chain of VkGraphicsPipelineCreateInfo.

Libraries can be linked into other libraries recursively while there are still state blobs that can be linked together. E.g an application could create a library for the vertex input interface and pre-rasterization shaders separately, then link them into a new library.

A newly created graphics pipeline consists of the parts defined by linked libraries, plus those defined by VkGraphicsPipelineLibraryCreateInfoEXT. Parts specified in the pipeline must not overlap those defined by libraries, and similarly multiple libraries must not provide the same parts. Any state required by multiple parts must match.

Graphics pipelines that contain a full set of libraries are executable, may not be used for further linking, and must not have the VK_PIPELINE_CREATE_LIBRARY_BIT_KHR set. Graphics pipelines that contain only a subset of stages are not executable, may be used for further linking, and must have VK_PIPELINE_CREATE_LIBRARY_BIT_KHR set.

If rasterizerDiscardEnable is enabled, the complete set of parts does not include fragment shader or fragment output interface libraries.

Two additional bits control how linking is performed:

VK_PIPELINE_CREATE_LINK_TIME_OPTIMIZATION_BIT_EXT allows applications to specify that linking should perform an optimization pass; when this bit is specified, additional optimizations will be performed at link time, and the resulting pipeline should perform equivalently to a pipeline created monolithically.

To perform link time optimizations, VK_PIPELINE_CREATE_RETAIN_LINK_TIME_OPTIMIZATION_INFO_BIT_EXT must be specified on all pipeline libraries that are being linked together. Implementations should retain any additional information needed to perform optimizations at the final link step when this bit is present.

If the application created the final linked pipeline with pipeline layouts including the VK_PIPELINE_LAYOUT_CREATE_INDEPENDENT_SETS_BIT_EXT flag, the final linked pipeline layout is the union of the layouts provided for shader stages. However, in the specific case that a final link is being performed between stages and VK_PIPELINE_CREATE_LINK_TIME_OPTIMIZATION_BIT_EXT is specified, the application can override the pipeline layout with one that is compatible with that union but does not have the VK_PIPELINE_LAYOUT_CREATE_INDEPENDENT_SETS_BIT_EXT flag set, allowing a more optimal pipeline layout to be used when generating the final pipeline.

To make single-shader compilation consistent, shader modules will be deprecated by allowing VkShaderModuleCreateInfo to be chained to VkPipelineShaderStageCreateInfo, and allowing the VkShaderModule to be VK_NULL_HANDLE in this case. Applications can continue to use shader modules as they are not being removed; but it’s strongly recommended to not use them. The primary reason for this would be to allow bypassing what is in many cases a useless copy, along with potential wasted storage if they are retained. There have been previous efforts to allow shader modules to be precompiled in some way, but this functionality is now being made available in a more reliable and portably agreed way, negating the need to focus efforts in this area moving forward.

Initial compilation can now be organized into separate chunks, allowing consistent earlier compilation for applications that have this information available separately, and potentially allows more multithreading opportunities for applications that do not.

Below is an example of the information needed to compile a vertex shader:

Linking is relatively straightforward - pipeline libraries in, executable pipeline out, with the option of optimizing the pipeline or not.

While splitting the geometry stages may be possible, it’s a significant amount of additional work for many vendors, the advantage for most developers is unclear, and it would be difficult to make some of the guarantees in this extension.

When a novel shader/stage combination is seen that requires compilation, it should be compiled into a separate pipeline library as early as possible; this should be possible alongside usual material/object loading (e.g. texture/mesh streaming). If an application has its own material cache, the library should be cached there. Applications should still use pipeline caches to amortize compilation across similar stage blobs but should avoid mixing different stage types in the same VkPipelineCache, to avoid unnecessary lookup overhead.

Basic linking should then be done as early as the application is able. Applications should ideally store/cache this pipeline with relevant objects. Using a VkPipelineCache for this suboptimal pipeline is recommended; implementations where this would provide no benefit should ignore the cache lookup request for fast linking.

Once a basic link is done, the application should schedule a task for a separate thread to create an optimized pipeline. This should use pipeline caches in the same manner as existing monolithic compilation, sharing this cache with fast-linked pipelines. Implementations should prefer returning optimized pipelines from these caches. Applications should switch to the optimized pipeline as soon as they are available.

Work to change the way pipelines are cached is ongoing; to avoid scope creep the minimum set of features required to ensure things worked were added. A future extension may change how a lot of this works, so it was undesirable to design something that would be thrown away later.

A fast-linked pipeline may have a significant device performance penalty compared to the final pipeline on some implementations. Some vendors may have a negligible performance penalty; others will have performance penalties differing based on which shader stages are compiled together. A rough estimate given by vendors is that it could be as bad as a 50% penalty in the general case, with outliers performing even worse.

As general advice, applications should be aiming to keep the amount of work in each frame performed by unoptimized pipelines relatively low (<10%); profiling may be necessary to identify problematic areas. Developers are strongly encouraged create optimized pipelines as soon as they are able to replace the linked pipeline. Relying completely on fast linked pipelines could result in unacceptable performance degradation on some implementations.

No. This extension doesn’t change how specialization constants work – they work as they do for existing pipelines. If they’re provided, implementations are free to specialize the pipeline or not, and cache pipelines that are specialized, unspecialized, or both. Specialization constants must be provided alongside the shader stages using them and cannot be provided at link time. This may be something we want to address in a future extension.

Some implementations require the interpolation decorations in the last geometry shader stage if pipeline libraries are used, and this is advertised by the graphicsPipelineLibraryIndependentInterpolationDecoration property. It is expected that these implementations are serving markets where OpenGL ES is dominant, where this requirement was never dropped for separate shader objects, unlike OpenGL.

Yes. This simplifies compilation, avoids an unnecessary copy, and brings developers and implementations onto the same page.

Yes, as developers may want to adjust the way they manage pipelines. If linking is more or less free, the expectation is that applications may link pipelines on demand when recording draw calls. If linking is going to take more time, they may try to more aggressively pre-cache pipelines. This has been added as the graphicsPipelineLibraryFastLinking property.

Implementation and developer guidance is that if this feature bit is advertised, applications should be able to link on demand, so the cost of linking should be comparable to recording commands in a command buffer.

The depth format affects how depth bias is applied, but these are currently provided in separate parts of the pipeline. Nobody has claimed this to be a problem.

Caches containing pipeline libraries will necessarily increase the total memory consumption of compiled pipelines, as applications will generally try to keep these available while pipeline could be streamed in/out. Implementations may be able to use data in the library caches for the final pipelines in some circumstances, which could help mitigate it - but this is not guaranteed and will vary by vendor.

Fast-linked pipelines should not contribute to the total memory consumption if applications destroy the fast-linked pipeline once an optimized version exists.

Improvements to pipeline caches allowing selective eviction of individual caches could help with memory management here, but as this intersects with other known pipeline cache problems, this should be dealt with in a separate extension.

Yes, but it will not necessarily be subject to the same quality guarantees. Ray tracing pipeline libraries were not designed with this directly in mind, so while implementations should make use of these bits as best they can, it is not possible to make the same quality guarantees as for graphics pipelines.

Any future extensions using the pipeline library interface should be aware of these interactions and try to follow the intent of these bits as much as possible.

Yes.

Some vendors will make use of this information if it is available and would rather not see it move - however notably all of this state can be made dynamic. Applications wanting to avoid setting this state with the fragment shader library should use this dynamic state.

The fragment shader library only needs this information when sample shading is enabled.

Moving it to the output interface removes the need to create multiple fragment shader libraries for different MSAA rates, which some applications do as a part of dynamic performance tuning. This only works when used in conjunction with dynamic rendering; when render pass objects are used, the sample rate will effectively be sourced from any subpass attachments due to validation constraints. This could be made to work with subpasses with no attachments, but the additional complexity of adding that path had no clear benefit, so is disallowed.

No, as the complexity of handling this does not clearly translate to significant application wins.

Not unconditionally - if an implementation does not do anything with that flag then yes, but there is a functional difference then it cannot. I.e. the same as other pipeline state.