RFC: Introduce ExternalTexture

[i509VCB]

Related to the discussion around a texture memory import api, it may be desirable to import memory from external sources, such as video.

The current state of externally created textures in wgpu restricts the types of textures that can be imported. In particular, textures right now must meet the following requirements:

1. Be RGB or some permutation of this
2. Must be a format known by wgpu. The graphics apis and platforms that wgpu using may support more formats that are common in for example video.
3. Not multi-planar. Multi-planar formats are common in video, such as decoding from a video codec or the memory objects from something like vaapi.

The solution to this would be to introduce an ExternalTexture type which represents an opaque, read-only, 2d, implementation dependent view over some texture memory that can be copied and or sampled from.

This type would be analogous to the existing GPUExternalTexture from webgpu. In order to keep the maintenance burden down, creation of ExternalTextures on native from platform memory handles is considered to be out of scope of wgpu's public api (except for an external_texture_from_hal function). Hal implementations inside wgpu-hal will be responsible for creating any ExternalTexture instances and should expose mechanisms to allow this. The web backend will provide a function to create an ExternalTexture from an HTMLVideoElement.

Along with the new type, ExternalTexture like TextureView would become a valid resource to bind in a bind group.

Naga would need to grow infrastructure to support the texture_external sampler.

Implementation

One thing that would differ between webgpu and the native backends would be how external textures are managed. On webgpu, external textures are only valid for the lifetime of the current frame. Essentially you need to reimport the texture every time you render. On native, the external texture is valid for the lifetime of the instance of the external texture. One solution to keep uniform behavior would be to have wgpu on web reimport the GPUExternalTexture every time it is used. This would likely involve owning a copy of a HTMLVideoElement object used the import the texture.

For native, any format which Vulkan supports with our current infrastructure should work. YUV and multi-planar formats may require internal support for VK_KHR_ycbcr_sampler_conversion. On Android, any AHardwareBufferFormat should work with Vulkan, as the extensions which allow importing an AHB in Vulkan allow using formats not known by Vulkan, but are still valid AHB formats.

EGL I am not so familiar with, but there could be some integration with the TEXTURE_EXTERNAL texture type.

I am not familiar with how DX12 and Metal would work here. I would like some advice on how DX12 and Metal should be integrated into this.

[pdiaz]

I'm looking forward to this feature!