Implement Camera module for webcam access on all platforms (currently only for iOS and macOS)

[Ansraer]

Godot version:
Any version

OS/device including version:
Anything other than ios/osx

Issue description:
Godot's Camera Server and module only have support for ios and OSX. This means that trying to interact with a webcam on other platforms always fails.

There have been efforts to add support for other platforms, but none of these attempts have ever been successful.
The most promising one (#44498) attempts to add support for Windows using Media Foundation but is currently only compatible with msvc.

Steps to reproduce:
Try to use the camera server on any non apple platform

Minimal reproduction project:
https://github.com/BastiaanOlij/godot3_test_projects/tree/master/TestCameraServer

UPDATE:
There are a couple of different open PRs.
Windows: #49763 (3.x only, but should be easily portable to master)
Linux: #47967 and #53666

[Calinou]

Related to #13139 (another feature that needs to be implemented on non-Apple platforms).

[BastiaanOlij]

Note that it does also work with ARCore though that work never ended up getting merged and is now moving to a plugin.
There were attempts at implementing native Android support, can't remember what that got stuck on.

There have also been a few projects attempting to implement webcam support on Linux but getting stuck on less then stellar support for various cameras

The problem is that some of the more complete cross platform camera libraries are all behemoths unsuitable to be included in Godot (like OpenCV)

[Ansraer]

The problem is that some of the more complete cross platform camera libraries are all behemoths unsuitable to be included in Godot (like OpenCV)

Its even worse than than that. While there are smaller libraries all they did was to write a wrapper around ffmpeg or opencv and hide that fact in a footnote of the readme.
In addition, both ffmpeg and opencv use various deprecated native libraries (e.g. directshow) and, to add insult to injury, their buildsystem is a horrible outdated mess that is NOT cross platform.
After spending a few days trying to write a PR to add more platform support I came to the conclusion that the best solution will be to manually add support for each platform using native apis.
At the point of writing that would be v4l2 for linux (which appears to be actually useable!), media foundation for windows (this one is more problematic. MS uses various Macros and shortcuts that cause most compilers to dislike compiling it. I managed to get it working by manually redefining some of the macros and ignoring certain warnings, but it wasnt pretty.) and ndk for android (also looks useable).

[Schmetzler]

What is with the HTML5 support. How would one go about that? There seems to be a pretty easy way to use webcams with JavaScript... But I don't have an idea how to use this with godot?

[Calinou]

What is with the HTML5 support. How would one go about that? There seems to be a pretty easy way to use webcams with JavaScript... But I don't have an idea how to use this with godot?

Like with many advanced HTML5 features, enabling webcam access requires permission from the user. Exposing webcam support to Godot would also require handling the permission request, converting the data so it can be read from Godot, etc. It's a lot of work.

[benjarmstrong]

I have a feature complete CameraServer implementation for Windows over at #49763 which partially fixes this issue.

[tavurth]

I was hoping to implement a cross-platform QR code scanner using ZXing.net, you can find my progress here:

https://github.com/tavurth/godot-qr-scanner

However without Android CameraServer support this seems to be blocked.

Are there currently any known workarounds to get images from the Camera on Android?

[ModProg]

Maybe it would be good practice to document these limitations in the docs?
I started implementing following: https://docs.godotengine.org/de/stable/classes/class_cameraserver.html
But then only after not getting any streams searched for the cause, finding this issue.

I'd suggest either linking this issue there or providing a table like in the MDN docs with platform compatibility.

[Calinou]

@ModProg This is already documented in the class reference in master:

godot/doc/classes/CameraServer.xml

Line 9 in deefce7

[b]Note:[/b] This class is currently only implemented on macOS and iOS. On other platforms, no [CameraFeed]s will be available.

However, this wasn't backported to 3.x yet, so I did it in #56489.

[apouche]

Do you guys know if anyone is working on the Android implementation?

[FlooferLand]

Bump?
I've noticed a few users having projects that depend on this feature being implemented for Android, aswell as being one of those users myself.

There is already an Android ARCore plugin for Godot under MIT, and it seems to be grabbing camera data so perhaps it could be of help?

[Calinou]

Nobody is currently working on CameraServer for Android, so it'll take a while for this to be implemented.

[Lamelynx]

I'm working on two android plugin that may help someone.
The first request a image through a activity/intent link .
The second is a extremly basic "inten" wrapper that can request a QR scan. link to example

[MichaelMacha]

In the interim, I don't suppose it might be possible to implement a GDExtension library that pulls the camera from FFMPEG? FFMPEG is flawed, sure, but it runs on everything and is reasonably lightweight, and a dynamic link to it might help some of us out until something better is implemented internally.

The documentation isn't great, but it should be straightforward. Has anyone else started on something like that?

EDIT: Shoot, I'm sorry, I'm thinking of the ImageMagick documentation, not FFMPEG. I'll have to look at the FFMPEG documentation again.

[ludolpif]

Hi, I'm newcomer on Godot, I hope making a basic video compositing software from it (then use OBS for the live broadcasting part). CameraServer is a key feature needed for this idea.

The best digest is the update at top of this issue, considering the addition about 3.X->master for Win) :
Windows: #49763 (3.x only, someone ported to master : RedMser@70921fe)
Linux: #47967 and #53666

I think we can say that OBS has just working things for huge user base on this topic, without dependencies on big libs or specific compilers it think, nor many commits on the 4 past years to maintain it. For win, it's dshow, for lin it's v4l2 on their side.

May I help somehow with already existing PR that are around there ?

[Roemer]

Any progress here? I would love to create a scanner to scan an image and put the scanned object into a virtual world in Godot.

[brunodeangelis]

I was thinking of making a game with a similar concept to Before Your Eyes, where I'd use the webcam to recognise certain facial gestures and affect gameplay. It would be awesome if Godot could add support for something like that :)

[bezark]

I'd also love to see this implemented. I don't know much about working with external libraries but @MichaelMacha 's idea seems smart. I wonder if any of the Open frameworks libraries could be helpful?

https://github.com/openframeworks/openFrameworks/tree/master/libs%2FopenFrameworks%2Fvideo

Does anyone know if this is a viable route?

[Calinou]

Regarding external libraries, we have an existing commit in a branch for Windows and a PR for Linux already. These need to be rebased against the latest master branch though. We should avoid linking against large external libraries that are difficult to build from source, as Godot aims to remain quick and easy to build from source.

[MichaelMacha]

@Calinou I am in full agreement about not making Godot depend on something like FFMPEG, for the record. This is a proposal that would fully depend on an external library, FFMPEG, installed in and of itself along side the game—and given what the rest of my life has been doing to my programming schedule, probably just if it is necessary. Much of what I love so much about Godot is its extreme portability.

I'm also of the feeling that Godot's progress is zipping along right now, and if that carries over to webcam feeds across operating systems, binding to FFMPEG may not be as pressing.

One side thought I had, in the way of binding to FFMPEG, would be to create a video texture that allows for an FFMPEG script to alter it. I know that that's probably an advanced use case, but it's a great one.

[longPtrCall]

I don't understand why we have this issue opened for more than a year. Every system has its camera driver and we have to learn how to use them.

Using OpenCV sounds like a horrible idea, but we can refer to their sources to learn how they did a cross-platform solution (which drivers they've used). I know about deprecated APIs, but for a while we need to implement it somehow.

[Calinou]

I don't understand why we have this issue opened for more than a year. Every system has its camera driver and we have to learn how to use them.

We have multiple open PRs but they need to be rebased and tested before they can be merged. (Also, 4.2 is in feature freeze, so any new features will target 4.3 at the earliest.)

[Account-Accelerator]

Happy New Year to the Godot Community and Developers! I'm reaching out with keen interest about the progress on implementing webcam support in Godot. Is there any update or development in this area? As someone eager to integrate webcam functionalities into my projects, I'm looking for any available information, documentation, or discussions on this topic. If there are ways I can assist or contribute to this feature, I would be more than willing to help. Thanks for all your hard work, and I'm excited to see what we can achieve together in the new year.

Best regards, Nicholas

[Schmetzler]

Happy New Year to the Godot Community and Developers! I'm reaching out with keen interest about the progress on implementing webcam support in Godot. Is there any update or development in this area? As someone eager to integrate webcam functionalities into my projects, I'm looking for any available information, documentation, or discussions on this topic. If there are ways I can assist or contribute to this feature, I would be more than willing to help. Thanks for all your hard work, and I'm excited to see what we can achieve together in the new year.

Best regards, Nicholas

As a matter of fact it seems like there exist PRs for the major platforms (besides Android)... it wouldn't be ideal but it should be possible to mash them together and build the engine with em. (well it also depends on the Godot version you are going to use).

Maybe it can also be implemented as GDExtension (by using for example OpenCV)

[Exw27]

Any progress for webcam access?

[MichaelMacha]

It's good to hear that there are so many active PRs across platforms, and I can understand why this has taken so long as I come from a multimedia background and know how crazy-complex, and divergent between hardware providers, video stream formatting can be.

That said, once we have all of these PRs merged, should we expect a consistent high-level interface? It would still be progress, but accessing a webcam differently on Windows, Macintosh, Linux, and Android could be a bit of a rigamarole. It would be much better if we had a single high-level texture or resource which pulled from a camera based on its current platform.

[Calinou]

Any progress for webcam access?

The Linux camera PR is good to go on my end, but it needs someone to take a final look and merge it. The Windows branch didn't work on my end though, so it's not in a mergeable state (no PR exists for this rebased branch yet).

That said, once we have all of these PRs merged, should we expect a consistent high-level interface?

It's possible to provide a consistent API across platforms, but some of the PRs like #53666 have already implemented their own API on top that PRs for other platforms haven't implemented yet.

[achatham]

Has anyone tried getting CameraServer to work on the web platform?

I have it working via javascript but it requires GPU->CPU->GPU image copies, so it's pretty unusable. In a straight javascript implementation (without godot), you'd use gl.texImage2D and pass in the <video> DOM element, and then all the data would stay on the GPU (the same approach would implement VideoStreamPlayer).

I attempted to do this myself in a module/GDExtension but am not sure it's possible without poking holes through lower-level parts of the engine. I'm hoping someone can confirm that before I waste more time.