OpenXR: Add documentation page about the new composition layers

[BastiaanOlij]

This new help page documents the new composition layer support @dsnopek added to Godot.

I've detailed a simple example project for this. I'll see if I can upload that somewhere so we can reference the source.

[mhilbrunner]

Thank you! Only reviewed the first half so far.

[devloglogan]

A few more typos

[m4gr3d]

I'd add MR in addition to AR in that sentence to avoid giving the impression we're only referring to AR glasses:

This is a feature that is specifically targeted at AR and MR use cases...

[BastiaanOlij]

As mentioned during the XR meeting, should have written this down before :) In my humble opinion I want to avoid the MR terminology here. AR and VR reference two distinct approaches, VR referencing fully immersive virtual worlds, while AR references any approach where the real world is augmented with virtual content.

Just for completeness sake for this discussion, XR is a grouping term that was adopted by Khronos and as such by a part of the industry (including by Godot).

MR is a confusing term that means something different depending on who you talk to.

• Originally it referred to mixing real world capture with virtual world capture to create a composite such as solutions as MixCast, Liv and Oculus own Mixed Reality iPad app implement.
• Then Microsoft adopted MR to mean Mixed Reality as the name for their WMR platform, in this case defining it as a platform supported both AR (through Hololens) and VR (through other WMR devices, none of which do passthrough I believe). I also guess they hoped people would see it as an abbreviation of Microsoft Reality, but that's pure speculation on my part :)
• Then in most recent years MR meaning Mixed Reality has become an alternative to XR as an encompassing term for people who don't like the XR moniker.
• And finally, Meta seems to have rebranded it recently to specifically mean AR through the means of passthrough though I am not sure if that has created more clarification or more confusion in the market space. All though Metas marketing here IMHO is inconsistent as its also often referring to the above meaning as a catch all for AR and VR.

So MR in its meaning in the context of this document as I understand it, e.g. AR through passthrough, is a meaning that is fairly contested.

As Godot is a platform agnostic development tool and we try not to care how AR is achieved but want our applications to be portable between various AR devices, it is my personal opinion that AR in this context is sufficient capturing all use cases of this feature, while MR is confusing.

There are features that are only possible on devices that use a camera feed to achieve AR and where passthrough should be named specifically, but this is no such feature as it is as applicable on a passthrough device such as the Quest, as it is on an optical AR device such as the Magic Leap 2.

However, I stress that this is my personal standpoint on the matter, so this is absolutely up for debate.

[m4gr3d]

It may be worth giving a use-case to illustrate how this is being used in actual project (e.g: adding a virtual window to the environment with passthrough in the 'background'.

[devloglogan]

I said I was on board with this use case last night at the XR meeting but I need to run that back: hole punched composition layers aren't the correct method for a "window into a virtual world". Since a subviewport can't be rendered in stereo to make it look actually 3D. If you wanted to achieve that effect you'd be better off having the "virtual world" be the main scene, and then have everything that isn't the window be covered up by (hole punched) geometry passthrough.

The main point of hole punching in composition layers is to just let things be displayed in front of the composition layer, like hand models, etc. Adding that use case would make sense.

[BastiaanOlij]

@devloglogan actually for those type of use cases you're better off rendering geometry directly with the shadow to opacity option and skip composition layers all together.

Hole punching in relation to composition layers is purely meant to allow the composition layer to be rendered behind the main content so that you can have virtual foreground objects occlusion part of the content of the composition layer.

Right now composition layers are always visible on top even if they clip through geometry.

[devloglogan]

@BastiaanOlij I totally agree, and was attempting to say as much in my previous comment. :) Would not use composition layers when going for that use case.

[BastiaanOlij]

Ah we're on the same page then :)

[BastiaanOlij]

One more bit of feedback from the XR meeting was the suggestion to add some image captures to the documentation.

Especially seeing we're naming this by the technical solution, not by what you can achieve with this solution, a picture tells a thousand words to let people know what this functionality is used for.

[devloglogan]

Perhaps an image like this would suffice to illustrate hole punching?

[BastiaanOlij]

Owh thats pretty clear!

Interesting btw that this shows its also viable for VR, so possibly I should rewrite that part of the text to not point out AR at all.

[dsnopek]

@devloglogan That screenshot looks great!

@BastiaanOlij:

Interesting btw that this shows its also viable for VR, so possibly I should rewrite that part of the text to not point out AR at all.

+1 to this!

[m4gr3d]

Owh thats pretty clear!

Interesting btw that this shows its also viable for VR, so possibly I should rewrite that part of the text to not point out AR at all.

@BastiaanOlij +1! A VR specific example would be to use a quad composition layer for text display to improve the text rendering quality.

hole punched composition layers aren't the correct method for a "window into a virtual world".

@devloglogan During the meeting, I was suggesting the opposite, showing a window into the real world, which should be doable using the OpenXRFbPassthroughGeometry node.

If you wanted to achieve that effect you'd be better off having the "virtual world" be the main scene, and then have everything that isn't the window be covered up by (hole punched) geometry passthrough.

This would make for an interesting sample, especially if the geometry is destructable so the user can 'break out' from the real world into the virtual world (e.g: similar to what First Encounters does).

[BastiaanOlij]

@m4gr3d asked this on rocketchat as well, but I don't see whats special about FB passthrough geometry, we can already do this with the shadow_to_opacity shader and it will work cross platform on any AR capable device?
I'm probably missing something?

[dsnopek]

This is looking really great! I have just a couple minor notes.

Also, I wonder if it's worth pointing out that an equirect layer can also be used for rendering a sky box? While Godot can of course render the sky, using a composition layer has the same advantages with regard to clarity and performance that using a composition layer for 2D UI has.

This is mentioned specifically in the OpenXR spec:

[dsnopek]

Is it worth specifically saying something about "virtual hands appearing behind the composition layer"? I feel like that's the first thing developers are going to notice about composition layers in VR.

[BastiaanOlij]

Well, we do have the screenshot logan provided, it kind of shows that use case.

[m4gr3d]

This is dependent upon the sort order, so it may be worth clarifying that.

[BastiaanOlij]

It does say that later on:

You also need to set ``Sort Order`` to a negative value,
the XR compositor will now draw the viewport first, and then overlay our rendering result.

[m4gr3d]

Minor comments, but looks good overall!

[skyace65]

Only one minor phrasing issue, but after that is fixed this should be good to merge.

[skyace65]

Thanks!