Optical Flow

Optical Flow Queues

VK_NV_optical_flow adds an optical flow queue type bit VK_QUEUE_OPTICAL_FLOW_BIT_NV to VkQueueFlagBits. Optical flow operations are supported by queues with an advertised queue capability of VK_QUEUE_OPTICAL_FLOW_BIT_NV. As in the case of other queue types, an application must use vkGetPhysicalDeviceQueueFamilyProperties to query whether the physical device has support for the Optical Flow Queue. When the implementation reports the VK_QUEUE_OPTICAL_FLOW_BIT_NV bit for a queue family, it advertises general support for Vulkan queue operations described in Devices and Queues.

Optical Flow Image Formats

To enumerate the supported image formats for a specific optical flow usage, call:

// Provided by VK_NV_optical_flow
VkResult vkGetPhysicalDeviceOpticalFlowImageFormatsNV(
    VkPhysicalDevice                            physicalDevice,
    const VkOpticalFlowImageFormatInfoNV*       pOpticalFlowImageFormatInfo,
    uint32_t*                                   pFormatCount,
    VkOpticalFlowImageFormatPropertiesNV*       pImageFormatProperties);

physicalDevice is the physical device being queried.
pOpticalFlowImageFormatInfo is a pointer to a VkOpticalFlowImageFormatInfoNV structure specifying the optical flow usage for which information is returned.
pFormatCount is a pointer to an integer related to the number of optical flow properties available or queried, as described below.
pImageFormatProperties is a pointer to an array of VkOpticalFlowImageFormatPropertiesNV structures in which supported formats and image parameters are returned.

If pImageFormatProperties is NULL, then the number of optical flow properties supported for the given physicalDevice is returned in pFormatCount. Otherwise, pFormatCount must point to a variable set by the application to the number of elements in the pImageFormatProperties array, and on return the variable is overwritten with the number of values actually written to pImageFormatProperties. If the value of pFormatCount is less than the number of optical flow properties supported, at most pFormatCount values will be written to pImageFormatProperties, and VK_INCOMPLETE will be returned instead of VK_SUCCESS, to indicate that not all the available values were returned.

Before creating an image to be used as an optical flow frame, obtain the supported image creation parameters by querying with vkGetPhysicalDeviceFormatProperties2 and vkGetPhysicalDeviceImageFormatProperties2 using one of the reported formats and adding VkOpticalFlowImageFormatInfoNV to the pNext chain of VkPhysicalDeviceImageFormatInfo2.

When querying the parameters with vkGetPhysicalDeviceImageFormatProperties2 for images used for optical flow operations, the VkOpticalFlowImageFormatInfoNV::usage field must contain one or more of the bits defined in VkOpticalFlowUsageFlagBitsNV.

Valid Usage (Implicit)

VUID-vkGetPhysicalDeviceOpticalFlowImageFormatsNV-physicalDevice-parameter
physicalDevice must be a valid VkPhysicalDevice handle
VUID-vkGetPhysicalDeviceOpticalFlowImageFormatsNV-pOpticalFlowImageFormatInfo-parameter
pOpticalFlowImageFormatInfo must be a valid pointer to a valid VkOpticalFlowImageFormatInfoNV structure
VUID-vkGetPhysicalDeviceOpticalFlowImageFormatsNV-pFormatCount-parameter
pFormatCount must be a valid pointer to a uint32_t value
VUID-vkGetPhysicalDeviceOpticalFlowImageFormatsNV-pImageFormatProperties-parameter
If the value referenced by pFormatCount is not 0, and pImageFormatProperties is not NULL, pImageFormatProperties must be a valid pointer to an array of pFormatCount VkOpticalFlowImageFormatPropertiesNV structures

Return Codes

Success

VK_SUCCESS
VK_INCOMPLETE

Failure

VK_ERROR_EXTENSION_NOT_PRESENT
VK_ERROR_INITIALIZATION_FAILED
VK_ERROR_FORMAT_NOT_SUPPORTED

VK_FORMAT_B8G8R8A8_UNORM, VK_FORMAT_R8_UNORM and VK_FORMAT_G8_B8R8_2PLANE_420_UNORM are initially supported for images with optical usage VK_OPTICAL_FLOW_USAGE_INPUT_BIT_NV.

VK_FORMAT_R16G16_SFIXED5_NV is initially supported for images with optical flow usage VK_OPTICAL_FLOW_USAGE_OUTPUT_BIT_NV, VK_OPTICAL_FLOW_USAGE_HINT_BIT_NV and VK_OPTICAL_FLOW_USAGE_GLOBAL_FLOW_BIT_NV.

VK_FORMAT_R8_UINT and VK_FORMAT_R32_UINT are initially supported for images with optical flow usage VK_OPTICAL_FLOW_USAGE_COST_BIT_NV. It is recommended to use VK_FORMAT_R8_UINT because of the lower bandwidth.

The VkOpticalFlowImageFormatInfoNV structure is defined as:

// Provided by VK_NV_optical_flow
typedef struct VkOpticalFlowImageFormatInfoNV {
    VkStructureType              sType;
    const void*                  pNext;
    VkOpticalFlowUsageFlagsNV    usage;
} VkOpticalFlowImageFormatInfoNV;

sType is a VkStructureType value identifying this structure.
pNext is NULL or a pointer to a structure extending this structure.
usage is a bitmask of VkOpticalFlowUsageFlagBitsNV describing the intended optical flow usage of the image.

Valid Usage (Implicit)

VUID-VkOpticalFlowImageFormatInfoNV-sType-sType
sType must be VK_STRUCTURE_TYPE_OPTICAL_FLOW_IMAGE_FORMAT_INFO_NV
VUID-VkOpticalFlowImageFormatInfoNV-usage-parameter
usage must be a valid combination of VkOpticalFlowUsageFlagBitsNV values
VUID-VkOpticalFlowImageFormatInfoNV-usage-requiredbitmask
usage must not be 0

Bits which can be set in VkOpticalFlowImageFormatInfoNV::usage, controlling optical flow usage, are:

// Provided by VK_NV_optical_flow
typedef enum VkOpticalFlowUsageFlagBitsNV {
    VK_OPTICAL_FLOW_USAGE_UNKNOWN_NV = 0,
    VK_OPTICAL_FLOW_USAGE_INPUT_BIT_NV = 0x00000001,
    VK_OPTICAL_FLOW_USAGE_OUTPUT_BIT_NV = 0x00000002,
    VK_OPTICAL_FLOW_USAGE_HINT_BIT_NV = 0x00000004,
    VK_OPTICAL_FLOW_USAGE_COST_BIT_NV = 0x00000008,
    VK_OPTICAL_FLOW_USAGE_GLOBAL_FLOW_BIT_NV = 0x00000010,
} VkOpticalFlowUsageFlagBitsNV;

VK_OPTICAL_FLOW_USAGE_INPUT_BIT_NV specifies that the image can be used as input or reference frame for an optical flow operation.
VK_OPTICAL_FLOW_USAGE_OUTPUT_BIT_NV specifies that the image can be used as output flow vector map for an optical flow operation.
VK_OPTICAL_FLOW_USAGE_HINT_BIT_NV specifies that the image can be used as hint flow vector map for an optical flow operation.
VK_OPTICAL_FLOW_USAGE_COST_BIT_NV specifies that the image can be used as output cost map for an optical flow operation.
VK_OPTICAL_FLOW_USAGE_GLOBAL_FLOW_BIT_NV specifies that the image can be used as global flow vector for an optical flow operation.

// Provided by VK_NV_optical_flow
typedef VkFlags   VkOpticalFlowUsageFlagsNV;

VkOpticalFlowUsageFlagsNV is a bitmask type for setting a mask of zero or more VkOpticalFlowUsageFlagBitsNV.

The VkOpticalFlowImageFormatPropertiesNV structure is defined as:

// Provided by VK_NV_optical_flow
typedef struct VkOpticalFlowImageFormatPropertiesNV {
    VkStructureType    sType;
    const void*        pNext;
    VkFormat           format;
} VkOpticalFlowImageFormatPropertiesNV;

sType is a VkStructureType value identifying this structure.
pNext is NULL or a pointer to a structure extending this structure.
format is a VkFormat that specifies the format that can be used with the specified optical flow image usages.

Valid Usage (Implicit)

VUID-VkOpticalFlowImageFormatPropertiesNV-sType-sType
sType must be VK_STRUCTURE_TYPE_OPTICAL_FLOW_IMAGE_FORMAT_PROPERTIES_NV
VUID-VkOpticalFlowImageFormatPropertiesNV-pNext-pNext
pNext must be NULL

Optical Flow Session

Optical Flow Session Object

Optical flow session objects are abstracted and represented by VkOpticalFlowSessionNV handles:

// Provided by VK_NV_optical_flow
VK_DEFINE_NON_DISPATCHABLE_HANDLE(VkOpticalFlowSessionNV)

Creating an Optical Flow Session

To create an optical flow session object, call:

// Provided by VK_NV_optical_flow
VkResult vkCreateOpticalFlowSessionNV(
    VkDevice                                    device,
    const VkOpticalFlowSessionCreateInfoNV*     pCreateInfo,
    const VkAllocationCallbacks*                pAllocator,
    VkOpticalFlowSessionNV*                     pSession);

device is the logical device that creates the optical flow session object.
pCreateInfo is a pointer to a VkOpticalFlowSessionCreateInfoNV structure containing parameters specifying the creation of the optical flow session.
pAllocator controls host memory allocation as described in the Memory Allocation chapter.
pSession is a pointer to a VkOpticalFlowSessionNV handle specifying the optical flow session object which will be created by this function when it returns VK_SUCCESS

Valid Usage (Implicit)

VUID-vkCreateOpticalFlowSessionNV-device-parameter
device must be a valid VkDevice handle
VUID-vkCreateOpticalFlowSessionNV-pCreateInfo-parameter
pCreateInfo must be a valid pointer to a valid VkOpticalFlowSessionCreateInfoNV structure
VUID-vkCreateOpticalFlowSessionNV-pAllocator-parameter
If pAllocator is not NULL, pAllocator must be a valid pointer to a valid VkAllocationCallbacks structure
VUID-vkCreateOpticalFlowSessionNV-pSession-parameter
pSession must be a valid pointer to a VkOpticalFlowSessionNV handle

Return Codes

Success

VK_SUCCESS

Failure

VK_ERROR_OUT_OF_HOST_MEMORY
VK_ERROR_INITIALIZATION_FAILED

The VkOpticalFlowSessionCreateInfoNV structure is defined as:

// Provided by VK_NV_optical_flow
typedef struct VkOpticalFlowSessionCreateInfoNV {
    VkStructureType                      sType;
    void*                                pNext;
    uint32_t                             width;
    uint32_t                             height;
    VkFormat                             imageFormat;
    VkFormat                             flowVectorFormat;
    VkFormat                             costFormat;
    VkOpticalFlowGridSizeFlagsNV         outputGridSize;
    VkOpticalFlowGridSizeFlagsNV         hintGridSize;
    VkOpticalFlowPerformanceLevelNV      performanceLevel;
    VkOpticalFlowSessionCreateFlagsNV    flags;
} VkOpticalFlowSessionCreateInfoNV;

sType is a VkStructureType value identifying this structure.
pNext is NULL or a pointer to a structure extending this structure.
width is the width in pixels of the input or reference frame to be bound to this optical flow session.
height is the height in pixels of the input or reference frame to be bound to this optical flow session.
imageFormat is the VkFormat of the input and reference frame to be bound to this optical flow session.
flowVectorFormat is the VkFormat of the flow vector maps (output or hint) to be bound to this optical flow session.
costFormat is the VkFormat of the cost maps to be bound to this optical flow session.
outputGridSize is exactly one bit of VkOpticalFlowGridSizeFlagsNV specifying the grid size of the output flow and cost maps to be bound to this optical flow session. The size of the output flow and cost maps is determined by VkOpticalFlowSessionCreateInfoNV::width and VkOpticalFlowSessionCreateInfoNV::height divided by VkOpticalFlowSessionCreateInfoNV::outputGridSize.
hintGridSize is one exactly bit of VkOpticalFlowGridSizeFlagsNV specifying the grid size of the hint flow vector maps to be bound to this optical flow session. The size of the hint maps is determined by VkOpticalFlowSessionCreateInfoNV::width and VkOpticalFlowSessionCreateInfoNV::height divided by VkOpticalFlowSessionCreateInfoNV::hintGridSize.
performanceLevel is the VkOpticalFlowPerformanceLevelNV used for this optical flow session.
flags are the VkOpticalFlowSessionCreateFlagsNV used for this optical flow session.

Valid Usage

VUID-VkOpticalFlowSessionCreateInfoNV-width-07581
width must be greater than or equal to VkPhysicalDeviceOpticalFlowPropertiesNV::minWidth and less than or equal to VkPhysicalDeviceOpticalFlowPropertiesNV::maxWidth
VUID-VkOpticalFlowSessionCreateInfoNV-height-07582
height must be greater than or equal to VkPhysicalDeviceOpticalFlowPropertiesNV::minHeight and less than or equal to VkPhysicalDeviceOpticalFlowPropertiesNV::maxHeight
VUID-VkOpticalFlowSessionCreateInfoNV-imageFormat-07583
imageFormat must be one of the formats returned by vkGetPhysicalDeviceOpticalFlowImageFormatsNV for VK_OPTICAL_FLOW_USAGE_INPUT_BIT_NV
VUID-VkOpticalFlowSessionCreateInfoNV-flowVectorFormat-07584
flowVectorFormat must be one of the formats returned by vkGetPhysicalDeviceOpticalFlowImageFormatsNV for VK_OPTICAL_FLOW_USAGE_OUTPUT_BIT_NV
VUID-VkOpticalFlowSessionCreateInfoNV-costFormat-07585
costFormat must be one of the formats returned by vkGetPhysicalDeviceOpticalFlowImageFormatsNV for VK_OPTICAL_FLOW_USAGE_COST_BIT_NV if VK_OPTICAL_FLOW_SESSION_CREATE_ENABLE_COST_BIT_NV is set in flags
VUID-VkOpticalFlowSessionCreateInfoNV-outputGridSize-07586
outputGridSize must be exactly one of the bits reported in VkPhysicalDeviceOpticalFlowPropertiesNV::supportedOutputGridSizes
VUID-VkOpticalFlowSessionCreateInfoNV-hintGridSize-07587
hintGridSize must be exactly one of the bits reported in VkPhysicalDeviceOpticalFlowPropertiesNV::supportedHintGridSizes if VK_OPTICAL_FLOW_SESSION_CREATE_ENABLE_HINT_BIT_NV is set in flags
VUID-VkOpticalFlowSessionCreateInfoNV-flags-07588
VK_OPTICAL_FLOW_SESSION_CREATE_ENABLE_HINT_BIT_NV must not be set in flags if VkPhysicalDeviceOpticalFlowPropertiesNV::hintSupported is VK_FALSE
VUID-VkOpticalFlowSessionCreateInfoNV-flags-07589
VK_OPTICAL_FLOW_SESSION_CREATE_ENABLE_COST_BIT_NV must not be set in flags if VkPhysicalDeviceOpticalFlowPropertiesNV::costSupported is VK_FALSE
VUID-VkOpticalFlowSessionCreateInfoNV-flags-07590
VK_OPTICAL_FLOW_SESSION_CREATE_ENABLE_GLOBAL_FLOW_BIT_NV must not be set in flags if VkPhysicalDeviceOpticalFlowPropertiesNV::globalFlowSupported is VK_FALSE
VUID-VkOpticalFlowSessionCreateInfoNV-flags-07591
VK_OPTICAL_FLOW_SESSION_CREATE_ALLOW_REGIONS_BIT_NV must not be set in flags if VkPhysicalDeviceOpticalFlowPropertiesNV::maxNumRegionsOfInterest is 0
VUID-VkOpticalFlowSessionCreateInfoNV-flags-07592
VK_OPTICAL_FLOW_SESSION_CREATE_BOTH_DIRECTIONS_BIT_NV must not be set in flags if VkPhysicalDeviceOpticalFlowPropertiesNV::bidirectionalFlowSupported is VK_FALSE

Valid Usage (Implicit)

VUID-VkOpticalFlowSessionCreateInfoNV-sType-sType
sType must be VK_STRUCTURE_TYPE_OPTICAL_FLOW_SESSION_CREATE_INFO_NV
VUID-VkOpticalFlowSessionCreateInfoNV-pNext-pNext
pNext must be NULL or a pointer to a valid instance of VkOpticalFlowSessionCreatePrivateDataInfoNV
VUID-VkOpticalFlowSessionCreateInfoNV-sType-unique
The sType value of each struct in the pNext chain must be unique
VUID-VkOpticalFlowSessionCreateInfoNV-imageFormat-parameter
imageFormat must be a valid VkFormat value
VUID-VkOpticalFlowSessionCreateInfoNV-flowVectorFormat-parameter
flowVectorFormat must be a valid VkFormat value
VUID-VkOpticalFlowSessionCreateInfoNV-costFormat-parameter
If costFormat is not 0, costFormat must be a valid VkFormat value
VUID-VkOpticalFlowSessionCreateInfoNV-outputGridSize-parameter
outputGridSize must be a valid combination of VkOpticalFlowGridSizeFlagBitsNV values
VUID-VkOpticalFlowSessionCreateInfoNV-outputGridSize-requiredbitmask
outputGridSize must not be 0
VUID-VkOpticalFlowSessionCreateInfoNV-hintGridSize-parameter
hintGridSize must be a valid combination of VkOpticalFlowGridSizeFlagBitsNV values
VUID-VkOpticalFlowSessionCreateInfoNV-performanceLevel-parameter
If performanceLevel is not 0, performanceLevel must be a valid VkOpticalFlowPerformanceLevelNV value
VUID-VkOpticalFlowSessionCreateInfoNV-flags-parameter
flags must be a valid combination of VkOpticalFlowSessionCreateFlagBitsNV values

The VkOpticalFlowSessionCreatePrivateDataInfoNV structure is for NV internal use only and is defined as:

// Provided by VK_NV_optical_flow
typedef struct VkOpticalFlowSessionCreatePrivateDataInfoNV {
    VkStructureType    sType;
    void*              pNext;
    uint32_t           id;
    uint32_t           size;
    const void*        pPrivateData;
} VkOpticalFlowSessionCreatePrivateDataInfoNV;

sType is a VkStructureType value identifying this structure.
pNext is NULL or a pointer to a structure extending this structure.
id is an identifier for data which is passed at a memory location specified in VkOpticalFlowSessionCreatePrivateDataInfoNV::pPrivateData.
size is the size of data in bytes which is passed at a memory location specified in VkOpticalFlowSessionCreatePrivateDataInfoNV::pPrivateData.
pPrivateData is a pointer to NV internal data.

Valid Usage (Implicit)

VUID-VkOpticalFlowSessionCreatePrivateDataInfoNV-sType-sType
sType must be VK_STRUCTURE_TYPE_OPTICAL_FLOW_SESSION_CREATE_PRIVATE_DATA_INFO_NV
VUID-VkOpticalFlowSessionCreatePrivateDataInfoNV-pPrivateData-parameter
pPrivateData must be a pointer value

Optical flow vectors are generated block-wise, one vector for each block of NxN pixels (referred to as grid).

Bits which can be set in VkOpticalFlowSessionCreateInfoNV::outputGridSize and VkOpticalFlowSessionCreateInfoNV::hintGridSize, or which are returned in VkPhysicalDeviceOpticalFlowPropertiesNV::supportedOutputGridSizes and VkPhysicalDeviceOpticalFlowPropertiesNV::supportedHintGridSizes controlling optical flow grid sizes, are:

// Provided by VK_NV_optical_flow
typedef enum VkOpticalFlowGridSizeFlagBitsNV {
    VK_OPTICAL_FLOW_GRID_SIZE_UNKNOWN_NV = 0,
    VK_OPTICAL_FLOW_GRID_SIZE_1X1_BIT_NV = 0x00000001,
    VK_OPTICAL_FLOW_GRID_SIZE_2X2_BIT_NV = 0x00000002,
    VK_OPTICAL_FLOW_GRID_SIZE_4X4_BIT_NV = 0x00000004,
    VK_OPTICAL_FLOW_GRID_SIZE_8X8_BIT_NV = 0x00000008,
} VkOpticalFlowGridSizeFlagBitsNV;

VK_OPTICAL_FLOW_GRID_SIZE_1X1_BIT_NV specifies that grid is 1x1 pixel.
VK_OPTICAL_FLOW_GRID_SIZE_2X2_BIT_NV specifies that grid is 2x2 pixel.
VK_OPTICAL_FLOW_GRID_SIZE_4X4_BIT_NV specifies that grid is 4x4 pixel.
VK_OPTICAL_FLOW_GRID_SIZE_8X8_BIT_NV specifies that grid is 8x8 pixel.

// Provided by VK_NV_optical_flow
typedef VkFlags   VkOpticalFlowGridSizeFlagsNV;

VkOpticalFlowGridSizeFlagsNV is a bitmask type for setting a mask of zero or more VkOpticalFlowGridSizeFlagBitsNV.

Optical flow exposes performance levels which the application can choose based on the desired performance and quality requirement.

The optical flow performance level types are defined with the following:

// Provided by VK_NV_optical_flow
typedef enum VkOpticalFlowPerformanceLevelNV {
    VK_OPTICAL_FLOW_PERFORMANCE_LEVEL_UNKNOWN_NV = 0,
    VK_OPTICAL_FLOW_PERFORMANCE_LEVEL_SLOW_NV = 1,
    VK_OPTICAL_FLOW_PERFORMANCE_LEVEL_MEDIUM_NV = 2,
    VK_OPTICAL_FLOW_PERFORMANCE_LEVEL_FAST_NV = 3,
} VkOpticalFlowPerformanceLevelNV;

VK_OPTICAL_FLOW_PERFORMANCE_LEVEL_SLOW_NV is a level with slower performance but higher quality.
VK_OPTICAL_FLOW_PERFORMANCE_LEVEL_MEDIUM_NV is a level with medium performance and medium quality.
VK_OPTICAL_FLOW_PERFORMANCE_LEVEL_FAST_NV is a preset with higher performance but lower quality.

Bits which can be set in VkOpticalFlowSessionCreateInfoNV::flags, controlling optical flow session operations, are:

// Provided by VK_NV_optical_flow
typedef enum VkOpticalFlowSessionCreateFlagBitsNV {
    VK_OPTICAL_FLOW_SESSION_CREATE_ENABLE_HINT_BIT_NV = 0x00000001,
    VK_OPTICAL_FLOW_SESSION_CREATE_ENABLE_COST_BIT_NV = 0x00000002,
    VK_OPTICAL_FLOW_SESSION_CREATE_ENABLE_GLOBAL_FLOW_BIT_NV = 0x00000004,
    VK_OPTICAL_FLOW_SESSION_CREATE_ALLOW_REGIONS_BIT_NV = 0x00000008,
    VK_OPTICAL_FLOW_SESSION_CREATE_BOTH_DIRECTIONS_BIT_NV = 0x00000010,
} VkOpticalFlowSessionCreateFlagBitsNV;

VK_OPTICAL_FLOW_SESSION_CREATE_ENABLE_HINT_BIT_NV specifies that a VkImageView with external flow vectors will be used as hints in performing the motion search and must be bound to VK_OPTICAL_FLOW_SESSION_BINDING_POINT_HINT_NV.
VK_OPTICAL_FLOW_SESSION_CREATE_ENABLE_COST_BIT_NV specifies that the cost for the forward flow is generated in a VkImageView which must be bound to VK_OPTICAL_FLOW_SESSION_BINDING_POINT_COST_NV. Additionally, if VK_OPTICAL_FLOW_SESSION_CREATE_BOTH_DIRECTIONS_BIT_NV is also set, the cost for backward flow is generated in a VkImageView which must be bound to VK_OPTICAL_FLOW_SESSION_BINDING_POINT_BACKWARD_COST_NV. The cost is the confidence level of the flow vector for each grid in the frame. The Cost implies how (in)accurate the flow vector is. Higher cost value implies the flow vector to be less accurate and vice-versa.
VK_OPTICAL_FLOW_SESSION_CREATE_ENABLE_GLOBAL_FLOW_BIT_NV specifies that a global flow vector is estimated from forward flow in a single pixel VkImageView which must be bound to VK_OPTICAL_FLOW_SESSION_BINDING_POINT_GLOBAL_FLOW_NV.
VK_OPTICAL_FLOW_SESSION_CREATE_ALLOW_REGIONS_BIT_NV specifies that regions of interest can be specified in VkOpticalFlowExecuteInfoNV.
VK_OPTICAL_FLOW_SESSION_CREATE_BOTH_DIRECTIONS_BIT_NV specifies that backward flow is generated in addition to forward flow which is always generated.

VkOpticalFlowSessionCreateFlagsNV is a bitmask type for setting a mask of zero or more VkOpticalFlowSessionCreateFlagBitsNV.

// Provided by VK_NV_optical_flow
typedef VkFlags   VkOpticalFlowSessionCreateFlagsNV;

Destroying an Optical Flow Session

To destroy an optical flow session object, call:

// Provided by VK_NV_optical_flow
void vkDestroyOpticalFlowSessionNV(
    VkDevice                                    device,
    VkOpticalFlowSessionNV                      session,
    const VkAllocationCallbacks*                pAllocator);

device is the device that was used for the creation of the optical flow session.
session is the optical flow session to be destroyed.
pAllocator controls host memory allocation as described in the Memory Allocation chapter.

Valid Usage (Implicit)

VUID-vkDestroyOpticalFlowSessionNV-device-parameter
device must be a valid VkDevice handle
VUID-vkDestroyOpticalFlowSessionNV-session-parameter
session must be a valid VkOpticalFlowSessionNV handle
VUID-vkDestroyOpticalFlowSessionNV-pAllocator-parameter
If pAllocator is not NULL, pAllocator must be a valid pointer to a valid VkAllocationCallbacks structure
VUID-vkDestroyOpticalFlowSessionNV-session-parent
session must have been created, allocated, or retrieved from device

Binding Vulkan Image Views to an Optical Flow Session

To bind a vulkan image to an optical flow session object, call:

// Provided by VK_NV_optical_flow
VkResult vkBindOpticalFlowSessionImageNV(
    VkDevice                                    device,
    VkOpticalFlowSessionNV                      session,
    VkOpticalFlowSessionBindingPointNV          bindingPoint,
    VkImageView                                 view,
    VkImageLayout                               layout);

device is the device which owns the optical flow session object session.
session is the optical flow session object to which the image view is to be bound.
bindingPoint specifies the binding point VkOpticalFlowSessionBindingPointNV to which the image view is bound.
view is a VkImageView to be bound.
layout must specify the layout that the image subresources accessible from view will be in at the time the optical flow vectors are calculated with vkCmdOpticalFlowExecuteNV on a VkDevice.

Valid Usage (Implicit)

VUID-vkBindOpticalFlowSessionImageNV-device-parameter
device must be a valid VkDevice handle
VUID-vkBindOpticalFlowSessionImageNV-session-parameter
session must be a valid VkOpticalFlowSessionNV handle
VUID-vkBindOpticalFlowSessionImageNV-bindingPoint-parameter
bindingPoint must be a valid VkOpticalFlowSessionBindingPointNV value
VUID-vkBindOpticalFlowSessionImageNV-view-parameter
If view is not VK_NULL_HANDLE, view must be a valid VkImageView handle
VUID-vkBindOpticalFlowSessionImageNV-layout-parameter
layout must be a valid VkImageLayout value
VUID-vkBindOpticalFlowSessionImageNV-session-parent
session must have been created, allocated, or retrieved from device
VUID-vkBindOpticalFlowSessionImageNV-view-parent
If view is a valid handle, it must have been created, allocated, or retrieved from device

Return Codes

Success

VK_SUCCESS

Failure

VK_ERROR_OUT_OF_HOST_MEMORY
VK_ERROR_INITIALIZATION_FAILED

The optical flow session binding points are defined with the following:

// Provided by VK_NV_optical_flow
typedef enum VkOpticalFlowSessionBindingPointNV {
    VK_OPTICAL_FLOW_SESSION_BINDING_POINT_UNKNOWN_NV = 0,
    VK_OPTICAL_FLOW_SESSION_BINDING_POINT_INPUT_NV = 1,
    VK_OPTICAL_FLOW_SESSION_BINDING_POINT_REFERENCE_NV = 2,
    VK_OPTICAL_FLOW_SESSION_BINDING_POINT_HINT_NV = 3,
    VK_OPTICAL_FLOW_SESSION_BINDING_POINT_FLOW_VECTOR_NV = 4,
    VK_OPTICAL_FLOW_SESSION_BINDING_POINT_BACKWARD_FLOW_VECTOR_NV = 5,
    VK_OPTICAL_FLOW_SESSION_BINDING_POINT_COST_NV = 6,
    VK_OPTICAL_FLOW_SESSION_BINDING_POINT_BACKWARD_COST_NV = 7,
    VK_OPTICAL_FLOW_SESSION_BINDING_POINT_GLOBAL_FLOW_NV = 8,
} VkOpticalFlowSessionBindingPointNV;

VK_OPTICAL_FLOW_SESSION_BINDING_POINT_INPUT_NV specifies the binding point for the input frame.
VK_OPTICAL_FLOW_SESSION_BINDING_POINT_REFERENCE_NV specifies the binding point for the input reference frame.
VK_OPTICAL_FLOW_SESSION_BINDING_POINT_HINT_NV specifies the binding point for the optional external hint flow vectors.
VK_OPTICAL_FLOW_SESSION_BINDING_POINT_FLOW_VECTOR_NV specifies the binding point for output flow vectors of default forward flow calculation.
VK_OPTICAL_FLOW_SESSION_BINDING_POINT_BACKWARD_FLOW_VECTOR_NV specifies the binding point for the optional output flow vector map of optional backward flow calculation.
VK_OPTICAL_FLOW_SESSION_BINDING_POINT_COST_NV specifies the binding point for the optional output cost map of default forward flow calculation.
VK_OPTICAL_FLOW_SESSION_BINDING_POINT_BACKWARD_COST_NV specifies the binding point for the optional output cost map of optional backward flow calculation.
VK_OPTICAL_FLOW_SESSION_BINDING_POINT_GLOBAL_FLOW_NV specifies the binding point for the optional global flow value of default forward flow calculation.

Optical Flow Execution

Default direction of flow estimation is forward which calculates the optical flow from input frame to reference frame. Optionally backward flow estimation can be additionally calculated. An output flow vector (Vx, Vy) means that current pixel (x, y) of input frame can be found at location (x+Vx, y+Vy) in reference frame. A backward flow vector (Vx, Vy) means that current pixel (x, y) of reference frame can be found at location (x+Vx, y+Vy) in input frame.

To calculate optical flow vectors from two input frames, call:

// Provided by VK_NV_optical_flow
void vkCmdOpticalFlowExecuteNV(
    VkCommandBuffer                             commandBuffer,
    VkOpticalFlowSessionNV                      session,
    const VkOpticalFlowExecuteInfoNV*           pExecuteInfo);

commandBuffer is the command buffer into which the command will be recorded.
session is the optical flow session object on which this command is operating.
pExecuteInfo Info is a pointer to a VkOpticalFlowExecuteInfoNV.

Valid Usage (Implicit)

VUID-vkCmdOpticalFlowExecuteNV-commandBuffer-parameter
commandBuffer must be a valid VkCommandBuffer handle
VUID-vkCmdOpticalFlowExecuteNV-session-parameter
session must be a valid VkOpticalFlowSessionNV handle
VUID-vkCmdOpticalFlowExecuteNV-pExecuteInfo-parameter
pExecuteInfo must be a valid pointer to a valid VkOpticalFlowExecuteInfoNV structure
VUID-vkCmdOpticalFlowExecuteNV-commandBuffer-recording
commandBuffer must be in the recording state
VUID-vkCmdOpticalFlowExecuteNV-commandBuffer-cmdpool
The VkCommandPool that commandBuffer was allocated from must support optical flow operations
VUID-vkCmdOpticalFlowExecuteNV-renderpass
This command must only be called outside of a render pass instance
VUID-vkCmdOpticalFlowExecuteNV-videocoding
This command must only be called outside of a video coding scope
VUID-vkCmdOpticalFlowExecuteNV-commonparent
Both of commandBuffer, and session must have been created, allocated, or retrieved from the same VkDevice

Host Synchronization

Host access to the VkCommandPool that commandBuffer was allocated from must be externally synchronized

Command Properties

Command Buffer Levels	Render Pass Scope	Video Coding Scope	Supported Queue Types	Command Type
Primary Secondary	Outside	Outside	Opticalflow	Action

Command Buffer Levels

Render Pass Scope

Video Coding Scope

Supported Queue Types

Command Type

Primary
Secondary

Outside

Opticalflow

Action

The VkOpticalFlowExecuteInfoNV structure is defined as:

// Provided by VK_NV_optical_flow
typedef struct VkOpticalFlowExecuteInfoNV {
    VkStructureType                sType;
    void*                          pNext;
    VkOpticalFlowExecuteFlagsNV    flags;
    uint32_t                       regionCount;
    const VkRect2D*                pRegions;
} VkOpticalFlowExecuteInfoNV;

sType is a VkStructureType value identifying this structure.
pNext is NULL or a pointer to a structure extending this structure.
flags are the VkOpticalFlowExecuteFlagsNV used for this command.
regionCount is the number of regions of interest specified in pRegions.
pRegions is a pointer to regionCount VkRect2D regions of interest.

Valid Usage

VUID-VkOpticalFlowExecuteInfoNV-regionCount-07593
regionCount must be 0 if VK_OPTICAL_FLOW_SESSION_CREATE_ALLOW_REGIONS_BIT_NV was not set for VkOpticalFlowSessionNV on which this command is operating

Valid Usage (Implicit)

VUID-VkOpticalFlowExecuteInfoNV-sType-sType
sType must be VK_STRUCTURE_TYPE_OPTICAL_FLOW_EXECUTE_INFO_NV
VUID-VkOpticalFlowExecuteInfoNV-pNext-pNext
pNext must be NULL
VUID-VkOpticalFlowExecuteInfoNV-flags-parameter
flags must be a valid combination of VkOpticalFlowExecuteFlagBitsNV values
VUID-VkOpticalFlowExecuteInfoNV-pRegions-parameter
If regionCount is not 0, pRegions must be a valid pointer to an array of regionCount VkRect2D structures

Bits which can be set in VkOpticalFlowExecuteInfoNV::flags, controlling optical flow execution, are:

// Provided by VK_NV_optical_flow
typedef enum VkOpticalFlowExecuteFlagBitsNV {
    VK_OPTICAL_FLOW_EXECUTE_DISABLE_TEMPORAL_HINTS_BIT_NV = 0x00000001,
} VkOpticalFlowExecuteFlagBitsNV;

VK_OPTICAL_FLOW_EXECUTE_DISABLE_TEMPORAL_HINTS_BIT_NV specifies that temporal hints from previously generated flow vectors are not used. If temporal hints are enabled, optical flow vectors from previous vkCmdOpticalFlowExecuteNV call are automatically used as hints for the current vkCmdOpticalFlowExecuteNV call, to take advantage of temporal correlation in a video sequence. Temporal hints should be disabled if there is a-priori knowledge of no temporal correlation (e.g. a scene change, independent successive frame pairs).

VkOpticalFlowExecuteFlagsNV is a bitmask type for setting a mask of zero or more VkOpticalFlowExecuteFlagBitsNV.

// Provided by VK_NV_optical_flow
typedef VkFlags   VkOpticalFlowExecuteFlagsNV;