Image Operations

Images are addressed by texel coordinates. There are three texel coordinate systems:

SPIR-V OpImageFetch, OpImageSparseFetch, OpImageRead, OpImageSparseRead, opImageBlockMatchSADQCOM, opImageBlockMatchSSDQCOM, opImageBlockMatchWindowSADQCOM, opImageBlockMatchWindowSSDQCOM, and OpImageWrite instructions use integer texel coordinates.

Other image instructions can use either normalized or unnormalized texel coordinates (selected by the unnormalizedCoordinates state of the sampler used in the instruction), but there are limitations on what operations, image state, and sampler state is supported. Normalized coordinates are logically converted to unnormalized as part of image operations, and certain steps are only performed on normalized coordinates. The array layer coordinate is always treated as unnormalized even when other coordinates are normalized.

Normalized texel coordinates are referred to as (s,t,r,q,a), with the coordinates having the following meanings:

The coordinates are extracted from the SPIR-V operand based on the dimensionality of the image variable and type of instruction. For Proj instructions, the components are in order (s, [t,] [r,] q), with t and r being conditionally present based on the Dim of the image. For non-Proj instructions, the coordinates are (s [,t] [,r] [,a]), with t and r being conditionally present based on the Dim of the image and a being conditionally present based on the Arrayed property of the image. Projective image instructions are not supported on Arrayed images.

Unnormalized texel coordinates are referred to as (u,v,w,a), with the coordinates having the following meanings:

Only the u and v coordinates are directly extracted from the SPIR-V operand, because only 1D and 2D (non-Arrayed) dimensionalities support unnormalized coordinates. The components are in order (u [,v]), with v being conditionally present when the dimensionality is 2D. When normalized coordinates are converted to unnormalized coordinates, all four coordinates are used.

Integer texel coordinates are referred to as (i,j,k,l,n), with the coordinates having the following meanings:

They are extracted from the SPIR-V operand in order (i [,j] [,k] [,l] [,n]), with j and k conditionally present based on the Dim of the image, and l conditionally present based on the Arrayed property of the image. n is conditionally present and is taken from the Sample image operand.

If an accessed image was created from a view using VkImageViewSlicedCreateInfoEXT and accessed through a VK_DESCRIPTOR_TYPE_STORAGE_IMAGE descriptor, then the value of k is incremented by VkImageViewSlicedCreateInfoEXT::sliceOffset, giving k ← sliceOffset + k. The image’s accessible range in the third dimension is k < sliceOffset + sliceCount. If VkImageViewSlicedCreateInfoEXT::sliceCount is VK_REMAINING_3D_SLICES_EXT, the range is inherited from the image’s depth extent as specified by Image Mip Level Sizing.

For all coordinate types, unused coordinates are assigned a value of zero.

The Texel Coordinate Systems - For the example shown of an 8×4 texel two dimensional image.

The Texel Coordinate Systems - For the example shown of an 8×4 texel two dimensional image.

For corner-sampled images, the texel samples are located at the grid intersections instead of the texel centers.

An RGB color (red, green, blue) is transformed to a shared exponent color (red_shared, green_shared, blue_shared, exp_shared) as follows:

First, the components (red, green, blue) are clamped to (red_clamped, green_clamped, blue_clamped) as:

where:

The largest clamped component, max_clamped is determined:

A preliminary shared exponent exp' is computed:

The shared exponent exp_shared is computed:

Finally, three integer values in the range 0 to 2^N are computed:

A shared exponent color (red_shared, green_shared, blue_shared, exp_shared) is transformed to an RGB color (red, green, blue) as follows:

where:

Texel input validation operations inspect instruction/image/sampler state or coordinates, and in certain circumstances cause the texel value to be replaced or become undefined. There are a series of validations that the texel undergoes.

Texels undergo a format conversion from the VkFormat of the image view to a vector of either floating-point or signed or unsigned integer components, with the number of components based on the number of components present in the format.

Each component is converted based on its type and size (as defined in the Format Definition section for each VkFormat), using the appropriate equations in 16-Bit Floating-Point Numbers, Unsigned 11-Bit Floating-Point Numbers, Unsigned 10-Bit Floating-Point Numbers, Fixed-Point Data Conversion, and Shared Exponent to RGB. Signed integer components smaller than 32 bits are sign-extended.

If the image view format is sRGB, the color components are first converted as if they are UNORM, and then sRGB to linear conversion is applied to the R, G, and B components as described in the “sRGB EOTF” section of the Khronos Data Format Specification. The A component, if present, is unchanged.

If VkSamplerYcbcrConversionYcbcrDegammaCreateInfoQCOM::enableYDegamma is equal to VK_TRUE, then sRGB to linear conversion is applied to the G component as described in the “sRGB EOTF” section of the Khronos Data Format Specification. If VkSamplerYcbcrConversionYcbcrDegammaCreateInfoQCOM::enableCbCrDegamma is equal to VK_TRUE, then sRGB to linear conversion is applied to the R and B components as described in the “sRGB EOTF” section of the Khronos Data Format Specification. The A component, if present, is unchanged.

If the image view format is block-compressed, then the texel value is first decoded, then converted based on the type and number of components defined by the compressed format.

A texel is replaced if it is one (and only one) of:

Border texels are replaced with a value based on the image format and the borderColor of the sampler. The border color is:

The custom border color (U) may be rounded by implementations prior to texel replacement, but the error introduced by such a rounding must not exceed one ULP of the image’s format.

This is substituted for the texel value by replacing the number of components in the image format

† S = B_g may be substituted as the replacement method by the implementation when VkSamplerCreateInfo::borderColor is VK_BORDER_COLOR_INT_CUSTOM_EXT and VkSamplerCustomBorderColorCreateInfoEXT::format is VK_FORMAT_UNDEFINED. Implementations should use S = B_r as the replacement method.

The value returned by a read of an invalid texel is undefined, unless that read operation is from a buffer resource and the robustBufferAccess feature is enabled. In that case, an invalid texel is replaced as described by the robustBufferAccess feature. If the access is to an image resource and the x, y, z, or layer coordinate validation fails and the robustImageAccess feature is enabled, then zero must be returned for the R, G, and B components, if present. Either zero or one must be returned for the A component, if present. If If the robustImageAccess2 feature is enabled, zero values must be returned. If only the sample index was invalid, the values returned are undefined.

Additionally, if the robustImageAccess feature is enabled, but the robustImageAccess2 feature is not, any invalid texels may be expanded to four components prior to texel replacement. This means that components not present in the image format may be replaced with 0 or may undergo conversion to RGBA as normal.

Loads from a null descriptor return a four component color value of all zeros. However, for storage images and storage texel buffers using an explicit SPIR-V Image Format, loads from a null descriptor may return an alpha value of 1 (float or integer, depending on format) if the format does not include alpha.

If the VkPhysicalDeviceSparseProperties::residencyNonResidentStrict property is VK_TRUE, a sparse unbound texel is replaced with 0 or 0.0 values for integer and floating-point components of the image format, respectively.

If residencyNonResidentStrict is VK_FALSE, the value of the sparse unbound texel is undefined.

If the image view has a depth/stencil format, the depth component is selected by the aspectMask, and the operation is an OpImage*Dref* instruction, a depth comparison is performed. The result is 1.0 if the comparison evaluates to true, and 0.0 otherwise. This value replaces the depth component D.

The compare operation is selected by the VkCompareOp value set by VkSamplerCreateInfo::compareOp. The reference value from the SPIR-V operand D_ref and the texel depth value D_tex are used as the reference and test values, respectively, in that operation.

If the image being sampled has an unsigned normalized fixed-point format, then D_ref is clamped to [0,1] before the compare operation.

The texel is expanded from one, two, or three components to four components based on the image base color:

where one = 1.0f for floating-point formats and depth aspects, and one = 1 for integer formats and stencil aspects.

All texel input instructions apply a swizzle based on:

The swizzle can rearrange the components of the texel, or substitute zero or one for any components. It is defined as follows for each color component:

where:

If the border color is one of the VK_BORDER_COLOR_*_OPAQUE_BLACK enums and the VkComponentSwizzle is not the identity swizzle for all components, the value of the texel after swizzle is undefined.

If the image view has a depth/stencil format and the VkComponentSwizzle is VK_COMPONENT_SWIZZLE_ONE, and VkPhysicalDeviceMaintenance5Properties::depthStencilSwizzleOneSupport is not VK_TRUE, the value of the texel after swizzle is undefined.

OpImageSparse* instructions return a structure which includes a residency code indicating whether any texels accessed by the instruction are sparse unbound texels. This code can be interpreted by the OpImageSparseTexelsResident instruction which converts the residency code to a boolean value.

In some color models, the color representation is defined in terms of monochromatic light intensity (often called “luma”) and color differences relative to this intensity, often called “chroma”. It is common for color models other than RGB to represent the chroma components at lower spatial resolution than the luma component. This approach is used to take advantage of the eye’s lower spatial sensitivity to color compared with its sensitivity to brightness. Less commonly, the same approach is used with additive color, since the green component dominates the eye’s sensitivity to light intensity and the spatial sensitivity to color introduced by red and blue is lower.

Lower-resolution components are “downsampled” by resizing them to a lower spatial resolution than the component representing luminance. This process is also commonly known as “chroma subsampling”. There is one luminance sample in each texture texel, but each chrominance sample may be shared among several texels in one or both texture dimensions.

The process of reconstructing a full color value for texture access involves accessing both chroma and luma values at the same location. To generate the color accurately, the values of the lower-resolution components at the location of the luma samples are reconstructed from the lower-resolution sample locations, an operation known here as “chroma reconstruction” irrespective of the actual color model.

The location of the chroma samples relative to the luma coordinates is determined by the xChromaOffset and yChromaOffset members of the VkSamplerYcbcrConversionCreateInfo structure used to create the sampler Y′C_BC_R conversion.

The following diagrams show the relationship between unnormalized (u,v) coordinates and (i,j) integer texel positions in the luma component (shown in black, with circles showing integer sample positions) and the texel coordinates of reduced-resolution chroma components, shown as crosses in red.

Reconstruction is implemented in one of two ways:

If the format of the image that is to be sampled sets VK_FORMAT_FEATURE_SAMPLED_IMAGE_YCBCR_CONVERSION_CHROMA_RECONSTRUCTION_EXPLICIT_BIT, or the VkSamplerYcbcrConversionCreateInfo’s forceExplicitReconstruction is VK_TRUE, reconstruction is performed as an explicit step independent of filtering, described in the Explicit Reconstruction section.

If the format of the image that is to be sampled does not set VK_FORMAT_FEATURE_SAMPLED_IMAGE_YCBCR_CONVERSION_CHROMA_RECONSTRUCTION_EXPLICIT_BIT and if the VkSamplerYcbcrConversionCreateInfo’s forceExplicitReconstruction is VK_FALSE, reconstruction is performed as an implicit part of filtering prior to color model conversion, with no separate post-conversion texel filtering step, as described in the Implicit Reconstruction section.

Sampler Y′C_BC_R conversion performs the following operations, which an implementation may combine into a single mathematical operation:

Texel output validation operations inspect instruction/image state or coordinates, and in certain circumstances cause the write to have no effect. There are a series of validations that the texel undergoes.

The integer texel coordinates are validated according to the same rules as for texel input coordinate validation.

If the texel fails integer texel coordinate validation, then the write has no effect.

If the texel attempts to write to an unbound region of a sparse image, the texel is a sparse unbound texel. In such a case, if the VkPhysicalDeviceSparseProperties::residencyNonResidentStrict property is VK_TRUE, the sparse unbound texel write has no effect. If residencyNonResidentStrict is VK_FALSE, the write may have a side effect that becomes visible to other accesses to unbound texels in any resource, but will not be visible to any device memory allocated by the application.

If the image format is sRGB, a linear to sRGB conversion is applied to the R, G, and B components as described in the “sRGB EOTF” section of the Khronos Data Format Specification. The A component, if present, is unchanged.

Texels then undergo a format conversion from the floating-point, signed, or unsigned integer type of the texel data to the VkFormat of the image view. If the number of components in the texel data is larger than the number of components in the format, additional components are discarded.

Each component is converted based on its type and size (as defined in the Format Definition section for each VkFormat). Floating-point outputs are converted as described in Floating-Point Format Conversions and Fixed-Point Data Conversion. Integer outputs are converted such that their value is preserved. The converted value of any integer that cannot be represented in the target format is undefined.

If the VkImageView format has an X component in its format description, undefined values are written to those bits.

If the underlying VkImage format has an X component in its format description, undefined values are also written to those bits, even if result format conversion produces a valid value for those bits because the VkImageView format is different.

For Proj image operations, the normalized texel coordinates (s,t,r,q,a) and (if present) the D_ref coordinate are transformed as follows:

Derivatives are used for LOD selection. These derivatives are either implicit (in an ImplicitLod image instruction in a mesh, task, compute, or fragment shader) or explicit (provided explicitly by shader to the image instruction in any shader).

For implicit derivatives image instructions, the derivatives of texel coordinates are calculated in the same manner as derivative operations. That is:

Partial derivatives not defined above for certain image dimensionalities are set to zero.

For explicit LOD image instructions, if the optional SPIR-V operand Grad is provided, then the operand values are used for the derivatives. The number of components present in each derivative for a given image dimensionality matches the number of partial derivatives computed above.

If the optional SPIR-V operand Lod is provided, then derivatives are set to zero, the cube map derivative transformation is skipped, and the scale factor operation is skipped. Instead, the floating-point scalar coordinate is directly assigned to λ_base as described in LOD Operation.

If the image or sampler object used by an implicit derivative image instruction is not uniform across the quad and quadDivergentImplicitLod is not supported, then the derivative and LOD values are undefined. Implicit derivatives are well-defined when the image and sampler and control flow are uniform across the quad, even if they diverge between different quads.

If quadDivergentImplicitLod is supported, then derivatives and implicit LOD values are well-defined even if the image or sampler object are not uniform within a quad. The derivatives are computed as specified above, and the implicit LOD calculation proceeds for each shader invocation using its respective image and sampler object.

For cube map image instructions, the (s,t,r) coordinates are treated as a direction vector (r_x,r_y,r_z). The direction vector is used to select a cube map face. The direction vector is transformed to a per-face texel coordinate system (s_face,t_face), The direction vector is also used to transform the derivatives to per-face derivatives.

The direction vector selects one of the cube map’s faces based on the largest magnitude coordinate direction (the major axis direction). Since two or more coordinates can have identical magnitude, the implementation must have rules to disambiguate this situation.

The rules should have as the first rule that r_z wins over r_y and r_x, and the second rule that r_y wins over r_x. An implementation may choose other rules, but the rules must be deterministic and depend only on (r_x,r_y,r_z).

The layer number (corresponding to a cube map face), the coordinate selections for s_c, t_c, r_c, and the selection of derivatives, are determined by the major axis direction as specified in the following two tables.

The partial derivatives of the Cube Map Coordinate Transformations can be computed as:

The other derivatives are simplified similarly, resulting in

LOD selection can be either explicit (provided explicitly by the image instruction) or implicit (determined from a scale factor calculated from the derivatives). The LOD must be computed with mipmapPrecisionBits of accuracy.