Status of DriftFX integration into LWJGL3

[mipastgt]

I raised this issue already on the LWJGL3 project but was redirected to report it here, so that's what I do.

LWJGL/lwjgl3#522

The main questions are:

1. Is there any intention to make this production-ready?
2. Is there any intention to do that for Java 11+ too?

I'd love to see that working but it must be stable to be usable.

[tomsontom]

@mipastgt I can answer the Java11+ Question.

Yes there is and we have started working on it already - #18 - The Java-Bits are already in place (we produce an MR-Jar supporting Java8 and Java-11+, we won't support 9 & 10) and there are NO changes to the c/c++ story - but we need to port that to Github-Actions as travis is just to unreliable lately.

[QuattroDome]

Hi everbody

What is the status of the LWJGL3 intergation and Java11?

I am very interested to get a pure Java application with JavaFX + DriftFX + LWJGL3 running on Java 11.

If it is already possible, am I happy to contribute a sample application to show case the LWJGL3 intergation.

[Spasi]

The LWJGL bindings will be updated when Java 11 support in DriftFX is complete (i.e. when #18 is closed).

[QuattroDome]

@Spasi #18 seems to be already merged. Or is somethink still missing?

[QuattroDome]

A native OpenGL node is what JavaFX has long been missing. I appreciate your work very much.

[tomsontom]

BTW we are actively working on a new version 2.x who will have an improved api and hopefully fix problems we currently face eg with backpressure

[Spasi]

Could we have a progress update on the new version?

[tomsontom]

moving the code from an internal dev branches to the public repo is in progress - we expect that the code is available at the end of the month.

[QuattroDome]

@tomsontom any news? I am happy to support you.

[ReikaKalseki]

For those of us not in "high priority" projects where perfect stability and/or performance are of utmost concern, is DriftFX in a usable state for LWJGL 3.2, Java 8, and JavaFX to match? If so, is there a guide beyond this thing from the main page on how to build it into a JavaFX application, or examples showing the same?

[tomsontom]

Hi - the last few months we worked hard to get all this stuff into place and we are happy to report success and one can now use the stock lwjgl release together with the lastest nightly builds - https://tomsondev.bestsolution.at/2020/08/28/news-from-dirftfx-approaching-v1-0/ - feel free to give it a spin and report your findings

[ReikaKalseki]

I spy an example. I am going to give this a try over the next few days. :)

[ReikaKalseki]

It mostly works! It renders, including legacy OGL calls. I took the OBJ demo and ripped out all its render code and supplied my own simple test, and sure enough we have a result:

		GL11.glLineWidth(4);
		GL11.glBegin(GL11.GL_LINE_LOOP);
		GL11.glColor4f(1, 0, 0, 1);
		GL11.glVertex2d(0, 0.5);
		GL11.glColor4f(0, 1, 0, 1);
		GL11.glVertex2d(-0.5, -0.5);
		GL11.glColor4f(0, 0, 1, 1);
		GL11.glVertex2d(0.5, -0.5);
		GL11.glEnd();

The only thing is that every frame or so it puts [J] [DEBUG] *downloaded 1 => <some hex address> in the console, which quickly grows problematic.

I am going to try integrating it into my actual project now and see how that goes.

[tomsontom]

Looks like we missed a debug statement somewhere - @redrezo ?

[redrezo]

Happy to See people using drift ;-)
I think that output comes from the Main memory Mode, IOSurface and NVDXInterop should be quiet, ive added an issue for it #27

[ReikaKalseki]

IOSurface is not available on my system - the transfer type dropdown said something to that effect (as did that big debug console dump of the system parameters near the start). NVDX is, though I did not notice a change in console behavior (I also am a bit leery of using it given the apparently known issue of crashing the main driver, which will take down my entire PC).

[redrezo]

IOSurface is the MacOS mechanism of sharing textures across processes. NVDX is the bridge between d3d and opengl - in windows javafx renders with d3d while and drift allows you to get your opengl texture via nvdx in there.
MainMemory mode transfers the texture to main memory and uploads it to the graphic card again.
I never experienced a complete crash of the system while using nvdx. only the program itself crashed on some platforms.

The mode is specified when creating the swap chain - if you copied my sample you should be able to switch it on the fly (i think..) maybe you need to stop, switch and start again

[ReikaKalseki]

So I have managed to integrate DriftFX into my actual project, and it renders mostly correctly (and yes, NVDX is seemingly working fine). I will have an image coming when it is more...displayable.

However, two things do come up.

The first is that the size of the renderer (ie the returned Vec2i in the getSize()) always seems to come out to 0x0 unless the surface is contained in a BorderPane via that pane's setCenter method; anything else (eg simply adding it as a child of a TitledPane) creates a size-zero node and thus breaks the framebuffer setup. This might be a misunderstanding or lack of knowledge on my part regarding JavaFX; I am not that experienced with it, and generally use SceneBuilder pretty much exclusively to set up my GUIs.

The second - and more serious - issue is that now that DFX is creating the GL context, I lose all "setup control" over it. In particular, as far as I can tell, I no longer can choose things like the multisampling level; in a pure GLFW window, that is accomplished by GLFW.glfwWindowHint(GLFW.GLFW_SAMPLES, 4), and from my research that pattern of "set multisampling on context creation" seems to be universal no matter the form the GL context actually takes. Once again, I might simply have a hole in my knowledge here, but for this at least others seem to be having the same family of issue, for example this from a few days ago: https://www.eclipse.org/forums/index.php?t=msg&th=1105005/

[redrezo]

Drift reports its current nodes size (with regard to dpi scaling and your coutom user scale on the node). In javafx the nodes are sized by their parents. I dont think that there are Layouts where the surface will be always 0/0 but if there are this would be an issue on the drift side
Also dont forget that a node is 0/0 until it is attached to a scene and a Layout pass has happened

Drift does not create an OpenGL context for you. However it does use your current context and creates the 'to be rendered to' texture for you
Im Not Sure If multisampling depends on how the texture is created, but this texture is just there for you to transport the Pixel data. So you can setup your rendering Pipeline how you want and all you have to do is blit your final frame to the drift texture.

[ReikaKalseki]

Drift does not create an OpenGL context for you
Is that not what the call to createSharedCompatContext does? Moreover, I do not explicitly create a context; hell, the only way I know how to create contexts in OGL 3+ is with GLFW, which is as stated prior not being used here.

[redrezo]

This is an internal API call i used for the samples.
You should use your own API to create a context
I want to replace it with lwjgl-glfw calls

[ReikaKalseki]

OK, that was not at all clear. I would comment that bit to indicate that such a means of creating the context are not part of the required setup to make DriftFX work, because to someone who has no idea how any of it works, the tendency is to just take a "don't touch anything you do not know is not important" approach.

EDIT:
I want to replace it with lwjgl-glfw calls
Wait, creating a GLFW window inside DFX will work and still pass on the rendered output to the JavaFX GUI? That...surprises me.

[redrezo]

Yeah the New drift Version removed the context coupling the shared contexts were used to allow sharing texture in Linux, but that only worked because in linux the gl restrictions are not that hard
You should be able to render to a glfw Window and pass the final Image at the same time to drift
*i hate the autocorrect on my Phone ;-)

[ReikaKalseki]

I just tested and it indeed works in that my rendered content does appear. An empty GLFW window appears and then closes again (I suspect glfwCreateWindow shows it and glfwHideWindow closes it) alongside the "real" window, but that is not a big deal.

However, the multisampling remains nonfunctional; that implies it probably is in the render-to-texture system in the framebuffer. Clearly GLFW does something on its own end that I need to replicate.

[redrezo]

What is multisampling about? As i said you could just Transport your final Image over to drift with a simple blit Operation..

[ReikaKalseki]

Like anti-aliasing. Without it, all my rendered geometry is "pixel exact" and GL_POINTS are square. With the AA on, I get smooth edges and circular dots instead.

[redrezo]

Yeah but that render result could be put in a texture, which you blit over to drift

[ReikaKalseki]

It is put into a texture, as far as I can tell; that is what the framebuffer it renders to is. Or do you mean something else?

In case you cannot tell, OGL is largely black magic to me, and DFX even more so.

[redrezo]

What i imagine is:
You render into a differrent fbo which produce your perfekt result - you could Look at it with your glfw window
You then blit the result via another fbo to drift
And IT should have all the fancy stuff like multisampling included

[ReikaKalseki]

Ah I see. I can try that.

[ReikaKalseki]

It took a while to figure out how to make this work, but I have done it. So I needed to actually render to the MSAA buffer first, then blit that to another custom buffer (this one standard) and then rather than blit that to the DFX one (which fails as the DFX one is coded as DEPTH_32F while 'normal' buffers get standard data types), just render the standard buffer as a quad onto the DFX buffer. And the result works great:

With this, I can confirm that DFX appears fully functional, if somewhat nonintuitive and rather hard to debug with.

[tomsontom]

It's cool that you got it working! We are happy to take feedback on what you think we have to improve to make it easier to adopt for you and hopefully for the community at large!

[ReikaKalseki]

I am sure some of my headaches were the result of my fairly limited experience with the "mechanical" side of OGL (I have a lot of experience with using OGL in general, but only in FFP and "drawing" contexts, plus a little shader code), but the big thing I can think of that would help with both adoption and use is more detailed documentation.

For example, as you saw earlier, I had no idea how to actually stick DFX into a JavaFX program, and even once I had a sample, I could not effectively discern what was a necessary step to make it work and what was just part of the example's implementation.

Similarly, without said example I would have never guessed as to the actual nature of the framebuffer required in terms of its encoding (ie GL11.glTexImage2D(GL11.GL_TEXTURE_2D, 0, GL32.GL_DEPTH_COMPONENT32F, width, height, 0, GL11.GL_DEPTH_COMPONENT, GL11.GL_FLOAT, (ByteBuffer)null);). Even the people I know who do have more experience with this sort of thing seem to find that an odd requirement.

Finally, I had a bunch of issues along the way that made absolutely no sense, but which might be predictable "gotcha" moments.

For example, I had an issue where the texture ID (usually 1) from GLRenderer.getGLTextureId(target); represented what GL seemed to think was an unallocated texture (using it threw errors, and checking residency returned false). That seemed to be caused by the timing of the GLFW context creation, but that was far from intuitive and even farther from being pleasant to debug. This seems believably an artifact of how DFX works, but without knowing that in advance it is something of a trap one can fall into.

I also had issues where there seemed to be some kind of behavior dependency on the actual structure of the code, for example where unless the DFX integration code was called from inside my main render loop (and NOT in a function called by said loop), the JFX window would never call its init(Stage primary) and thus never actually initialize the DriftFXSurface (which in turn meant that the render loop never even tried to call said code). That might have been something inside eclipse or the JVM - editing and hotswapping the class, or the presence of breakpoints or console spam would make everything load properly - but the fact it where and whether DFX code was being called was the root trigger seems to make that less likely. Again, for all I know this is a fairly predictable situation with DFX, but without knowing how it (or JavaFX, for that matter) work, I have no real way to know. "A literal heisenbug" indeed.

[redrezo]

First of all: Thanks for using drift and for the feedback!

You are right, we indeed need more detailed documentation ;-)
Here are some comments trying to provide some drift insight

The depth component texture has actually nothing to do with drift. Drift provides just a texture which you may use in a framebuffer to draw to. And the sample creates a depth texture and adds it to its framebuffer as GL_DEPTH_ATTACHMENT to enable culling the backface of the model. The drift texture itself is attached as GL_COLOR_ATTACHMENT0.

The texture you get from drift is created with glTexImage2D(GL_TEXTURE_2D, 0, GL_RGBA, size.x, size.y, 0, GL_BGRA, GL_UNSIGNED_BYTE, 0) in MainMemory Mode, with dxDevice.CreateTexture(size.x, size.y, 0, Win32.D3DUSAGE_DYNAMIC, Win32.D3DFMT_A8R8G8B8, Win32.D3DPOOL_DEFAULT) in NVDXInterop and with CGLTexImageIOSurface2D(cglContext, GL_TEXTURE_RECTANGLE, GL_RGBA, size.x, size.y, GL_BGRA, GL_UNSIGNED_INT_8_8_8_8_REV, ioSurface, 0) in IOSurface mode. The thing is that in NVDX and IOSurface the OpenGL texture is actually backed by a Direct3D texture and an IOSurface.
(you can see it in the different org.eclipse.fx.drift.internal.backend.Image#allocate() implementations)

As for your timing issues there may be a lot of reasons why this could happen. Drift itself needs to hook the QuantumRenderer (the javafx renderer) and your renderer. We hook the QuantumRenderer by implementing the render function within our NGNode. If your drift surface is not visible this render method will be never invoked which could cause issues. Also the reported size in the renderer will always be 0/0 until your surface is added to the scene and layouted.
And on your renderer side drift needs a current valid opengl context to create and dispose the swapchain (to dispose the swapchain it also needs the quantum renderer running, because it has to disconnect from the textures on some implementations). Acquire will give you a texture from said swapchain, and if this texture is somehow invalid this already happend on swapchain creation.

Also if your javafx window does not get initialized it seems to me that you somehow blocked the javafx main thread.

[ReikaKalseki]

I have for the most part completed my project; is there somewhere you would like me to post a link to it, either as a "how to use" example, for your own interest, or if you have some sort of "projects using us" page (like some other libraries do)?

[tomsontom]

Well probably you want to provide a PR against the README.md, adding an section "Who uses DriftFX" pointing to a blog post, github repo, ...

[Aloento]

Applications should not extend the Node class directly.
LWJGL/lwjgl3#617 (comment)

[Displee]

Is there an example somewhere with anti-aliasing?

[tomsontom]

Applications should not extend the Node class directly.
LWJGL/lwjgl3#617 (comment)

On what version of OpenJFX is that? DriftFX has to use internal API hence you need to open/export those internal packages using command-line swiches.

[Displee]

DriftFX now also used here: https://github.com/markaren/three.kt/blob/master/examples/src/main/kotlin/info/laht/threekt/examples/javafx/HelloDriftFX.kt with LWJGL3.

Would be nice if some one could add anti-aliasing :P

[ReikaKalseki]

@Displee

Is there an example somewhere with anti-aliasing?

See my "Game Calendar" project:

https://github.com/ReikaKalseki/GameCalendar/blob/master/src/Reika/GameCalendar/Rendering/RenderLoop.java
https://github.com/ReikaKalseki/GameCalendar/blob/master/src/Reika/GameCalendar/Rendering/Framebuffer.java

[Spasi]

Hey all,

I've finally been able to migrate the LWJGL bindings for DriftFX to the current version. For those that are not aware, there has been a major refactoring to DriftFX and I'm happy to report that based on my testing:

• Great performance, didn't notice any bugs or weird debug messages.
• Most of the implementation has been moved to Java code, LWJGL-style. That's the way to do it!
• DriftFX handles loading of its shared library in a sane manner and avoids classloader issues, similar to LWJGL.
• With published Maven artifacts, it is now easy to add to Java projects.

Given the above, DriftFX gains almost nothing from being bundled within LWJGL and the DriftFX bindings will be removed. Vanilla LWJGL + the official DriftFX release work great together and there is no need for tighter integration. The only remaining difficulty will be OpenGL context creation, but LWJGL already provides all the necessary (platform-specific) APIs and users that need more control (compared to org.eclipse.fx.drift.internal.GL.createContext/createSharedCompatContext) will be able to implement it.

I think this issue may be closed now. :)

[chengenzhao]

@AlmasB

Could it be possible for #FXGL to provide a component to support LWJGL integration?

Cheers

[tomsontom]

@Spasi thanks for the feedback