Linux texture support

[huanghongxun]

Add support for external textures to the Linux platform shell.

Related issues

Fixes flutter/flutter#64188.

Design

For API, I introduce an abstract class FlTexture for abstration of different OpenGL textures, allowing any other textures to be supported in the future.
Subclass FlPixelBufferTexture support loading textures from given pixel buffers with RGBA format. For other formats, or textures not from pixel buffers, developer may implement their own texture class.

Pre-launch Checklist

• I read the Contributor Guide and followed the process outlined there for submitting PRs.
• I read the Tree Hygiene wiki page, which explains my responsibilities.
• I read and followed the Flutter Style Guide and the C++, Objective-C, Java style guides.
• I listed at least one issue that this PR fixes in the description above.
• I added new tests to check the change I am making or feature I am adding, or Hixie said the PR is test-exempt. See testing the engine for instructions on
  writing and running engine tests.
• I updated/added relevant documentation (doc comments with ///).
• I signed the CLA.
• All existing and new tests are passing.
• The reviewer has submitted any presubmit flakes in this PR using the engine presubmit flakes form before re-triggering the failure.

If you need help, consider asking for advice on the #hackers-new channel on Discord.

[stuartmorgan]

@cbracken See #20714 and #19405 for design discussion context.

[anirudhb]

Has anything changed from the original PR?

[huanghongxun]

Has anything changed from the original PR?

In one word, FlExternalTextureGL has been replaced with an abstract class FlTexture for developer customization and future support, as well as an official implementation FlPixelBufferTexture.

Also code is ported to be compatible with master branch. Comments are refined.

[dnsoumik]

This is great hope this time it will get merged 👍 I Appreciate all the hard work everyone is doing :)

[escamoteur]

@huanghongxun Amazing to see progress here. I currently work on https://github.com/FlutterGL/flutter_web_gl
And am waiting for Linux support. Could you tell me how your Testure support works? I use OpenGL ES from the Angle framework internally if there would be a way to avaoid an additional copying of the image data this would be amazing

[huanghongxun]

@escamoteur
FlTexture::populate_texture returns a texture id. So subclassing FlTexture and rewriting populate_texture to return your texture id should prevent from copying data. Please be careful of OpenGL context. You may find a way to share Flutters' context.

[escamoteur]

But that's inside the Flutter Engine

[huanghongxun]

@escamoteur egl provides eglGetCurrentContext which can be used for creating shared context.

GdkWindow* window = gtk_widget_get_parent_window(#FlView)
GdkGLContext* context = gdk_window_create_gl_context(window, error);

above code should create a context shared with the one Flutter is using.

[escamoteur]

@huanghongxun lets have a more detailed chat when this is merged and on master :-)

[RossComputerGuy]

Will this allow transparent windows (or at least a transparent FlView)?

[huanghongxun]

Will this allow transparent windows (or at least a transparent FlView)?

Already supported, but not in this PR. The only thing to do is to set app paintable for FlView.

[TimWhiting]

Hooray! Thanks @huanghongxun for the patience and dedication and thanks for the reviews and merge flutter team!

[HedgeHao]

Is there any tutorial about how to use this texture on Linux? Does that mean I can write a native OpenGL function render the image then display in flutter texture widget?

[huanghongxun]

Is there any tutorial about how to use this texture on Linux? Does that mean I can write a native OpenGL function render the image then display in flutter texture widget?

You can render your texture in Flutter's OpenGL context with hardware acceleration.
We have detailed implementation guide in comments. Please read them.
Also, an example: https://github.com/lucaszanella/external_texture_flutter_linux/blob/master/external_texture/linux/external_texture_plugin.cc#L40

[lattice0]

@HedgeHao the repo linked by @huanghongxun is mine. It uses a version that is a little older than the one merged, but it should amost work. If you get it working, I'd appreciate a PR, as I have no time right now

[HedgeHao]

I follow this repo and try to use PixelBufferTexture on my own but texture widget cannot render the data in buffer correctly.

Here is my version.

else if (strcmp(method, "registerTexture") == 0)
  {
    self->myTextxure = MY_TEXTURE(g_object_new(my_texture_get_type(), nullptr));
    FL_PIXEL_BUFFER_TEXTURE_GET_CLASS(self->myTextxure)->copy_pixels = my_texture_copy_pixels;
    fl_texture_registrar_register_texture(self->texture_registrar, FL_TEXTURE(self->myTextxure));
    MY_TEXTURE_GET_CLASS(self->myTextxure)->texture_id = reinterpret_cast<int64_t>(FL_TEXTURE(self->myTextxure));
    fl_texture_registrar_mark_texture_frame_available(self->texture_registrar, FL_TEXTURE(self->myTextxure));
    response = FL_METHOD_RESPONSE(fl_method_success_response_new(fl_value_new_int(MY_TEXTURE_GET_CLASS(self->myTextxure)->texture_id)));
  }
  else if (strcmp(method, "nativeClick") == 0)
  {
    int len = 300 * 300 * 4;
    uint8_t buffer[len];
    for (int i = 0; i < len; i++)
    {
      buffer[i] = 127;
    }
    MY_TEXTURE_GET_CLASS(self->myTextxure)->buffer = buffer;
    fl_texture_registrar_mark_texture_frame_available(self->texture_registrar, FL_TEXTURE(self->myTextxure));
    response = FL_METHOD_RESPONSE(fl_method_success_response_new(nullptr));
  }

I except after registerTexture, each time when I call nativeClick some data will set to the buffer and texture widget in Flutter will refresh. Here's a screenshot. I set all the bytes in buffer to constant 127. There's some color dots at the bottom of the widget and it did change a little bit when I click every time. But it's obviously wrong since it should all be 127.

Can someone please help? Thanks a lot!

@alexmercerind dart_vlc is such a great work! Wonder if you could help me out with this.

[alexmercerind]

@HedgeHao I'm happy dart_vlc could serve as an example, only difference is that I used G_DECLARE_DERIVABLE_TYPE instead of G_DECLARE_FINAL_TYPE (as in the tests of this PR) since I wanted to store some additional data like dimensions & id (and wanted to access it within frame callbacks aswell) etc.

I can't help out much, however I think that the buffer (whose pointer you are assigning) gets deallocated as soon as the flutter_hello_plugin_handle_method_call call ends. Maybe store it somewhere else or pass the ownership (std::move on std::unique_ptr? I don't know much about GObject style but standard C++).

[lattice0]

I'm willing to try the merged code, but I couldn't compile my code in the dev channel. Anyone knows when this will land on any of the channels?

[HedgeHao]

@LucasZanella I open a PR. It should work on the latest flutter version on master channel

Then I have a further question. Instead of create a pixel buffer directly. I try to use openGL to draw a triangle and get the pixel buffer data then create a texture with that buffer. I thought I can have a triangle render in flutter texture widget but it doesn't work. The code is here. The code work great with GLUT window but not in Flutter. Any ideas?

[lattice0]

Thanks @HedgeHao, PR merged.

[lattice0]

Anyone knows if direct GPU texture is supported yet? Or does it still copy from GPU to CPU and to GPU back again?

[charafau]

@lattice0 someone please correct me if I'm wrong but I think it still copies from GPU to CPU and to GPU. Engine doesn't support that yet but there is an issue for direct GPU texture support

[alexmercerind]

Anyone knows if direct GPU texture is supported yet? Or does it still copy from GPU to CPU and to GPU back again?

I kindly request the answer to the same question (same for macOS).

Hardware accelerated rendering is a big concern. Pixel buffers are just not good.

If the GL texture in this PR also copies the frame to an intermediate CPU RAM buffer internally, what's the point???

The decodeImageFromPixels from dart:ui already supports rendering such pixel buffers without any platform specific care. Besides a performant zero-copy open-gl (for Linux) & METAL (for macOS) texture API, there's no point. It's a must.

Playing a 1080p video through pixel buffers will cramp the CPU, redundantly drain the battery.
4K videos won't even play due to buffer copy latency. All this while knowing that a better solution exists.

Please consider. Thankyou!

[knopp]

I haven't tested but the linux texture takes GL Texture name and target. Why would you assume it's copied to CPU and then GPU?

On macOS the texture is backed by IOSurface, which is a GPU buffer.