[SPEC] External GPU memory interop (OpenGL, Vulkan, DirectX)

[kekekeks]

To avoid jittering and rendering artifacts, any user-rendered GPU images need to be properly synchronized. By properly synchronized we mean:

• user rendering commands are already completed when Avalonia tries to consume the rendered contents
• rendered contents are synchronized with the rest of the changes to the visual and composition trees, so they should be applied with the rest of the composition batch
• rendering should be synchronized with monitor refresh rate

We are adding CompositionDrawingSurface and CompositionSurfaceVisual APIs (CompositionBrush and CompositionSpriteVisual would come later).

User code could use ElementComposition.SetElementChildVisual to display the CompositionSurfaceVisual on top of their control.

public class CompositionSurface : CompositionObject
{
}

public class CompositionSurfaceVisual : CompositionVisual
{
    public CompositionSurface { get; set; }
}


public class Compositor
{
...
    public CompositionDrawingSurface CreateDrawingSurface();
    public CompositionSurfaceVisual CreateSurfaceVisual();
...
}

CompositionDrawingSurface retains its own copy of the image. To update said copy one needs to provide a GPU-backed image with some means to wait for rendering to be completed:

public class CompositionDrawingSurface : CompositionSurface
{
    /// <summary>
    /// Updates the surface contents using an imported memory image using a keyed mutex as the means of synchronization
    /// </summary>
    /// <param name="image">GPU image with new surface contents</param>
    /// <param name="acquireIndex">The mutex key to wait for before accessing the image</param>
    /// <param name="releaseIndex">The mutex key to release for after accessing the image </param>
    /// <returns>A task that completes when update operation is completed and user code is free to destroy or dispose the image</returns>
    public Task UpdateWithKeyedMutex(ICompositionImportedGpuImage image, uint acquireIndex, uint releaseIndex);

    /// <summary>
    /// Updates the surface contents using an imported memory image using a semaphore pair as the means of synchronization
    /// </summary>
    /// <param name="image">GPU image with new surface contents</param>
    /// <param name="waitForSemaphore">The semaphore to wait for before accessing the image</param>
    /// <param name="signalSemaphore">The semaphore to signal after accessing the image</param>
    /// <returns>A task that completes when update operation is completed and user code is free to destroy or dispose the image</returns>
    public Task UpdateWithSemaphores(ICompositionImportedGpuImage image,
        ICompositionImportedGpuSemaphore waitForSemaphore,
        ICompositionImportedGpuSemaphore signalSemaphore);

    /// <summary>
    /// Updates the surface contents using an unspecified automatic means of synchronization
    /// provided by the underlying platform
    /// </summary>
    /// <param name="image">GPU image with new surface contents</param>
    /// <returns>A task that completes when update operation is completed and user code is free to destroy or dispose the image</returns>
    public Task UpdateWithAutomaticSync(ICompositionImportedGpuImage image);
}

What is ICompositionImportedGpuImage? It's an object that represents an externally allocated GPU memory, that's been imported to use with the compositor and the current GPU context (it will become invalid once that context is lost).

public class Compositor
{
...
    public ValueTask<ICompositionGpuInterop?> TryGetGpuInterop();
...
}


public interface ICompositionGpuInterop
{
    /// <summary>
    /// Returns the list of image handle types supported by the current GPU backend, see <see cref="KnownPlatformGraphicsExternalImageHandleTypes"/>
    /// </summary>
    IReadOnlyList<string> SupportedImageHandleTypes { get; }
    
    /// <summary>
    /// Returns the list of semaphore types supported by the current GPU backend, see <see cref="KnownPlatformGraphicsExternalSemaphoreTypes"/>
    /// </summary>
    IReadOnlyList<string> SupportedSemaphoreTypes { get; }

    /// <summary>
    /// Returns the supported ways to synchronize access to the imported GPU image
    /// </summary>
    /// <returns></returns>
    CompositionGpuImportedImageSynchronizationCapabilities GetSynchronizationCapabilities(string imageHandleType);
    
    /// <summary>
    /// Asynchronously imports a texture. The returned object is immediately usable.
    /// </summary>
    ICompositionImportedGpuImage ImportImage(IPlatformHandle handle,
        PlatformGraphicsExternalMemoryProperties properties);

    /// <summary>
    /// Asynchronously imports a texture. The returned object is immediately usable.
    /// </summary>
    /// <param name="image">An image that belongs to the same GPU context or the same GPU context sharing group as one used by compositor</param>
    ICompositionImportedGpuImage ImportImage(ICompositionImportableSharedGpuContextImage image);

    /// <summary>
    /// Asynchronously imports a semaphore object. The returned object is immediately usable.
    /// </summary>
    ICompositionImportedGpuSemaphore ImportSemaphore(IPlatformHandle handle);
    
    /// <summary>
    /// Asynchronously imports a semaphore object. The returned object is immediately usable.
    /// </summary>
    /// <param name="image">A semaphore that belongs to the same GPU context or the same GPU context sharing group as one used by compositor</param>
    ICompositionImportedGpuImage ImportSemaphore(ICompositionImportableSharedGpuContextSemaphore image);
    
    /// <summary>
    /// Indicates if the device context this instance is associated with is no longer available
    /// </summary>
    public bool IsLost { get; }
    
}

[Flags]
public enum CompositionGpuImportedImageSynchronizationCapabilities
{
    /// <summary>
    /// Pre-render and after-render semaphores must be provided alongside with the image
    /// </summary>
    Semaphores = 1,
    /// <summary>
    /// Image must be created with D3D11_RESOURCE_MISC_SHARED_KEYEDMUTEX or in other compatible way
    /// </summary>
    KeyedMutex = 2,
    /// <summary>
    /// Synchronization and ordering is somehow handled by the underlying platform
    /// </summary>
    Automatic = 4
}

/// <summary>
/// An imported GPU object that's usable by composition APIs 
/// </summary>
public interface ICompositionGpuImportedObject : IDisposable
{
    /// <summary>
    /// Tracks the import status of the object. Once the task is completed,
    /// the user code is allowed to free the resource owner in case when a non-owning
    /// sharing handle was used
    /// </summary>
    Task ImportCompeted { get; }
    /// <summary>
    /// Indicates if the device context this instance is associated with is no longer available
    /// </summary>
    bool IsLost { get; }
}

/// <summary>
/// An imported GPU image object that's usable by composition APIs 
/// </summary>
[NotClientImplementable]
public interface ICompositionImportedGpuImage : ICompositionGpuImportedObject
{

}

/// <summary>
/// An imported GPU semaphore object that's usable by composition APIs 
/// </summary>
[NotClientImplementable]
public interface ICompositionImportedGpuSemaphore : ICompositionGpuImportedObject
{

}

/// <summary>
/// An GPU object descriptor obtained from a context from the same share group as one used by the compositor
/// </summary>
[NotClientImplementable]
public interface ICompositionImportableSharedGpuContextObject : IDisposable
{
}

/// <summary>
/// An GPU image descriptor obtained from a context from the same share group as one used by the compositor
/// </summary>
[NotClientImplementable]
public interface ICompositionImportableSharedGpuContextImage : IDisposable
{
}

/// <summary>
/// An GPU semaphore descriptor obtained from a context from the same share group as one used by the compositor
/// </summary>
[NotClientImplementable]
public interface ICompositionImportableSharedGpuContextSemaphore : IDisposable
{
}

public struct PlatformGraphicsExternalMemoryProperties
{
    public int Width { get; set; }
    public int Height { get; set; }
    public PlatformGraphicsExternalMemoryFormat Format { get; set; }
}

public enum PlatformGraphicsExternalMemoryFormat
{
    R8G8B8A8UNorm,
    B8G8R8A8UNorm
}

/// <summary>
/// Describes various GPU memory handle types that are currently supported by Avalonia graphics backends
/// </summary>
public static class KnownPlatformGraphicsExternalImageHandleTypes
{
    /// <summary>
    /// An DXGI global shared handle returned by IDXGIResource::GetSharedHandle D3D11_RESOURCE_MISC_SHARED or D3D11_RESOURCE_MISC_SHARED_KEYEDMUTEX flag.
    /// The handle does not own the reference to the underlying video memory, so the provider should make sure that the resource is valid until
    /// the handle has been successfully imported
    /// </summary>
    public const string D3D11TextureGlobalSharedHandle = nameof(D3D11TextureGlobalSharedHandle);
    /// <summary>
    /// A DXGI NT handle returned by IDXGIResource1::CreateSharedHandle for a texture created with D3D11_RESOURCE_MISC_SHARED_NTHANDLE or flag
    /// </summary>
    public const string D3D11TextureNtHandle = nameof(D3D11TextureNtHandle);
    /// <summary>
    /// A POSIX file descriptor that's exported by Vulkan using VK_EXTERNAL_MEMORY_HANDLE_TYPE_OPAQUE_FD_BIT or in a compatible way
    /// </summary>
    public const string VulkanOpaquePosixFileDescriptor = nameof(VulkanOpaquePosixFileDescriptor);
}

/// <summary>
/// Describes various GPU semaphore handle types that are currently supported by Avalonia graphics backends
/// </summary>
public static class KnownPlatformGraphicsExternalSemaphoreTypes
{
    /// <summary>
    /// A POSIX file descriptor that's been exported by Vulkan using VK_EXTERNAL_SEMAPHORE_HANDLE_TYPE_OPAQUE_FD_BIT or in a compatible way
    /// </summary>
    public const string VulkanOpaquePosixFileDescriptor = nameof(VulkanOpaquePosixFileDescriptor);
    
    /// <summary>
    /// A NT handle that's been exported by Vulkan using VK_EXTERNAL_SEMAPHORE_HANDLE_TYPE_OPAQUE_WIN32_BIT or in a compatible way
    /// </summary>
    public const string VulkanOpaqueNtHandle = nameof(VulkanOpaqueNtHandle);
    
    // A global shared handle that's been exported by Vulkan using VK_EXTERNAL_SEMAPHORE_HANDLE_TYPE_OPAQUE_WIN32_KMT_BIT or in a compatible way
    public const string VulkanOpaqueKmtHandle = nameof(VulkanOpaqueKmtHandle);
    
    /// A DXGI NT handle returned by ID3D12Device::CreateSharedHandle or ID3D11Fence::CreateSharedHandle
    public const string Direct3D12FenceNtHandle = nameof(Direct3D12FenceNtHandle);
}

ICompositionImportableSharedGpuContextObject and friends would be obtained from IGlContext and are required to support platforms where we provide OpenGL rendering support via OpenGL context sharing and for those who wish to reuse the same VkDevice for both Avalonia and user code.

OpenGlControlBase, VulkanControlBase and swapchains are outside of the scope of this spec. We are getting complaints about our built-in base controls being too inflexible for various applications, so the goal of this spec is to define a flexible API that can be used to implement both our built-in simple controls an complex user scenarios.

For vsync-synchronized rendering, we'll just add RequestAnimationFrame API that would allow one to do rendering just before the composition batch is sent to the render thread:

public class Compositor
{
...
    public void RequestAnimationFrame(Action<object> callback, object state);
...
}

[abenedik]

It is great that there will be a generic texture interop mechanism in Avalonia.

I think that I understand the specification and the steps that user would need to take to render a shared texture.
The specification is well written and allows a very generic approach to shared texture - Avalonia that runs on OpenGL or Vulkan can use shared textures from OpenGL, DirectX and Vulkan.

From the user's point of view, it would be good if ICompositionGpuInterop would also have the LUID and UUID of the device that Avalonia uses (those two properties may be nullable, but it would be good if Avalonia would set at least one). This way the correct device could be chosen for the external renderer. It is easier to work with LUID because it is ulong (UUID is a byte array), but I think that OpenGL supports only UUID. For example UUID on OpenGL can be retrieved by:

var uuidBytes = stackalloc byte[16];
_glExt.GetUnsignedByte(GlExtrasInterface.GL_DEVICE_UUID_EXT, index: 0, uuidBytes); // calling "glGetUnsignedBytei_vEXT"

On Vulkan the following can be used:

// We use GetPhysicalDeviceProperties2 to also get LUID
var physicalDeviceIDProperties = new PhysicalDeviceIDProperties()
{
    SType = StructureType.PhysicalDeviceIdProperties
};

var physicalDeviceProperties2 = new PhysicalDeviceProperties2()
{
    SType = StructureType.PhysicalDeviceProperties2,
    PNext = &physicalDeviceIDProperties
};

VulkanInstance.Vk.GetPhysicalDeviceProperties2(PhysicalDevice, &physicalDeviceProperties2);

if (physicalDeviceIDProperties.DeviceLUIDValid)
{
    // Convert from fixed 8 byte array to ulong:
    _deviceLUID = *((ulong*)&physicalDeviceIDProperties.DeviceLUID[0]);
}
else
{
    _deviceLUID = 0;
}

// Copy UUID
_deviceUUID = new byte[Vk.UuidSize];
for (int i = 0; i < Vk.UuidSize; i++)
    _deviceUUID[i] = physicalDeviceIDProperties.DeviceUUID[i];

Note that calling GetPhysicalDeviceProperties2 required Vulkan API 1.1 or in the case of API 1.0 the VK_KHR_get_physical_device_properties2 instance extension needs to be enabled (the following is the case on Rasberry Pi 4).

Some other minor spelling / conceptual issues or ideas:

ICompositionGpuInterop.SupportedImageHandleTypes should probably return a list of KnownPlatformGraphicsExternalImageHandleTypes and not a list of strings.

ICompositionGpuInterop.SupportedSemaphoreTypes should probably return a list of KnownPlatformGraphicsExternalSemaphoreTypes and not a list of strings.

ICompositionGpuInterop.GetSynchronizationCapabilities should probably take a KnownPlatformGraphicsExternalImageHandleTypes as a parameter and not a string.

ICompositionGpuInterop.ImportImage method should probably also take a KnownPlatformGraphicsExternalImageHandleTypes as a parameter.

ICompositionGpuInterop.ImportSemaphore method should probably also take a KnownPlatformGraphicsExternalSemaphoreTypes as a parameter.

KnownPlatformGraphicsExternalImageHandleTypes.VulkanOpaquePosixFileDescriptor should probably be named only OpaquePosixFileDescriptor because this handle type is not Vulkan specific (it existed way before Vulkan). It can also be used to share OpenGL images. See: https://en.wikipedia.org/wiki/File_descriptor and https://registry.khronos.org/OpenGL/extensions/EXT/EXT_external_objects_fd.txt

The same applies for KnownPlatformGraphicsExternalSemaphoreTypes.VulkanOpaquePosixFileDescriptor

Similarly the KnownPlatformGraphicsExternalSemaphoreTypes.VulkanOpaqueNtHandle and VulkanOpaqueKmtHandle are not Vulkan specific but Win32 specific. Before Vulkan existed, they can be used to share a DirectX 9 or DirectX 11 textures. So the prefix should be Win32 and not Vulkan.

ICompositionGpuImportedObject.ImportCompeted => ImportCompleted

I am looking forward to a working demo of that spec.

[kekekeks]

and not a list of strings.

We are using strings to describe IPlatformHandle types in other parts of the framework. That way APIs can be extended from 3rd-party backends if needed.

because this handle type is not Vulkan specific

According to the Vulkan spec: it "specifies a POSIX file descriptor handle that has only limited valid usage outside of Vulkan and other compatible APIs". This fd can only be exported from Vulkan, it can't be created by OpenGL. A universal API-agnostic (and usable by multiple physical devices) handle would be the DMABUF one. We'll add support for those extra handle types at some point too.

Before Vulkan existed, they can be used to share a DirectX 9 or DirectX 11 textures

DirectX-compatible DXGI share handles are represented by VK_EXTERNAL_MEMORY_HANDLE_TYPE_D3D11_TEXTURE_BIT and VK_EXTERNAL_MEMORY_HANDLE_TYPE_D3D11_TEXTURE_KMT_BIT respectively.

Vulkan has its own VK_EXTERNAL_MEMORY_HANDLE_TYPE_OPAQUE_WIN32_BIT and VK_EXTERNAL_MEMORY_HANDLE_TYPE_OPAQUE_WIN32_KMT_BIT handle types that are not compatible with DirectX, but can be consumed from OpenGL via external objects extension just like on Linux.

[abenedik]

Similarly the KnownPlatformGraphicsExternalSemaphoreTypes.VulkanOpaqueNtHandle and VulkanOpaqueKmtHandle are not Vulkan specific but Win32 specific.
Uh yes, semaphores are of course Vulkan specific. DirectX 11 synchronization is done by using a keyed mutex.

[kekekeks]

DirectX 11 supports semaphores via ID3D11Device5::CreateFence / ID3D11Device5::OpenSharedFence. In vulkan those are represented by VK_EXTERNAL_SEMAPHORE_HANDLE_TYPE_D3D12_FENCE_BIT.

VK_EXTERNAL_SEMAPHORE_HANDLE_TYPE_OPAQUE_WIN32_BIT and VK_EXTERNAL_SEMAPHORE_HANDLE_TYPE_OPAQUE_WIN32_KMT_BIT are Vulkan/OpenGL specific

[Sergio0694]

This looks very promising 😄

Just trying to understand this better - a while back I've prototyped implementing a custom D3D12 swap chain in Avalonia (see here), which I did by using a platform native control and then just plugged a DXGI swap chain from the given HWND (full source code in this branch). I'm trying to understand what the benefit of this would be vs. using that approach, as it seems this would also involve some additional steps to plug everything together. Is that because this would allow the swap chain to better interoperate with other visual tree elements compared to using a NativeControlHost-derived control? 🤔

Really cool stuff though! Also love how some of the API names are (roughly) the same as on UWP ahah

[Sergio0694]

Nit: I would strongly recommend adding the "Async" suffix to UpdateWithKeyedMutex and UpdateWithSemaphores 🙂

[maxkatz6]

@Sergio0694 problem with NativeControlHost-derived control is level of integration into the app, yes. As on most platforms it's basically another window glued in the middle of parent window. But it's still probably the easiest way to integrate native controls into the Avalonia app.
I also can see benefits of renderer-interop API for the application like video players, game engine editors, or maybe even graphics editors where developers might need even lower access than Skia can provide.

[NielsAtlant3D]

I can't get this to run on Mac (M2 MacBook Pro) at all - It complains that the backend does not support image sharing. Is this a known issue? Can it be fixed? Is there another (platform independent) way to access the GPU within an Avalonia application?

[kekekeks]

On macOS we currently only support interop via shared OpenGL context.
Use compositor.TryGetRenderInterfaceFeature(typeof(IOpenGlTextureSharingRenderInterfaceContextFeature)) for shared context creation and relevant shared texture import/update methods from ICompositionGpuInterop

IOSurface/CVPixelBuffer are planned but no ETA.

[NielsAtlant3D]

Thanks, I took a quick look at it, but I think I need a deeper understanding of how the compositing works in Avalonia before I can make any use of that information :)

[martinrhan]

I am working with this and I encountered problem that I cannot solve. can you please check my question #17689