Video camera input support

[stephanemagnenat]

Many applications, notably video conferencing, augmented reality, and video/image capture require access to the device's camera, which are numerous and high-resolution in modern mobile devices.

I originally filled an issue discussing this question in wgpu-rs. Since then, there were progress in WebGPU with support for things such as importExternalTexture.

The goal of this issue is to discuss and keep track of the support for video camera input in wgpu. This would certainly require quite some system-specific glue code, and maybe it belongs to a sister library, but given the effort required and the user benefit of a close integration with wgpu I am under the impression that it makes sense to talk about it here.

Here are the references currently found so far:

• Vulkan on Android: There is an extension, VK_ANDROID_external_memory_android_hardware_buffer allowing Vulkan to access Android's native hardware buffer. There is some code samples here.
• On Mac/iOS, Metal seems to have tight integration with AVFoundation, there is already a tutorial on how to do the link.
• On Windows, it is probably possible to use DirectShow to capture to a DirectX texture, probably through an Enhanced Video Renderer, although it is surprisingly difficult to find recent samples. No idea if there is a way to do camera-to-texture directly with Vulkan on Windows.
• Non-Android Linux, I did not find anything, I'm afraid one has to get the V4L2 stream in software and then send it to the texture.

It is probably interesting to see how the browsers do it, Chromium has a design document.

[valpackett]

Non-Android Linux, I did not find anything, I'm afraid one has to get the V4L2 stream in software and then send it to the texture

v4l2 can export dmabufs actually so you want VK_EXT_external_memory_dma_buf. (Which there's plenty of other reasons to support…)

related: #965