Switch normal map convention from OpenGL-style (Y-) to DirectX-style (Y+)

[Calinou]

Describe the project you are working on

The Godot editor 🙂

Describe the problem or limitation you are having in your project

Godot expects OpenGL-style normal maps, which generally does not line up with the normal maps people have at hand (either exported from third-party software, or downloaded from various websites and asset stores).

While I've done extensive work to document this everywhere relevant, DirectX-style normal maps are still more prevalent overall, especially on websites that distribute PBR materials like ambientCG.

Most popular 3D DCCs and game engines have settled on DirectX-style normal maps, making Godot an exception with its OpenGL-style normal maps.
There's a comment in the decal shader hinting this as well: https://github.com/godotengine/godot/blob/7bc52e56c093342255c389fcc652eda2ced40b39/servers/rendering/renderer_rd/shaders/scene_forward_clustered.glsl#L818

There is now an import option to invert the Y axis of a normal map. However, having this happen out of the box would make setting up scenes faster, while also preventing most cases where people don't notice that their normal maps are the wrong way around on one axis. If this proposal is accepted and you need to use OpenGL-style normal maps, you can still enable the aforementioned import option.

This is a compatibility-breaking change, so we need to take a decision before Godot 4.0 is released.

Note: Despite the name, DirectX-style normal maps can be used in OpenGL and Vulkan just fine. "DirectX" just refers to where the convention originally comes from.

Describe the feature / enhancement and how it helps to overcome the problem or limitation

Make Godot expect DirectX-style normal maps (Y+) instead of OpenGL-style normal maps (Y-).

Describe how your proposal will work, with code, pseudo-code, mock-ups, and/or diagrams

Tweak the renderer's shaders to expect DirectX orientation in normal maps, in both 2D and 3D.

If this enhancement will not be used often, can it be worked around with a few lines of script?

No, as this is about changing a default decision in the renderer.

Is there a reason why this should be core and not an add-on in the asset library?

See above.

[zinnschlag]

Maybe the default could be made a project-wide and/or editor option (still with DirectX being the default setting for this option)? Different users will have different requirements. Someone who pipelines his assets thought Blender (using OpenGL normals, normal baking and so on) would probably prefer the OpenGL option (I know I would).

[YuriSizov]

I use Blender's MatCap to render normals for an image, and I have to invert it to use properly with Godot. So it seems to also follow DirectX?

[zinnschlag]

https://www.blenderkit.com/docs/tutorials/normal-maps/

[YuriSizov]

Oh well, I guess the MatCap doesn't produce normals that are usable by Blender then.

[Calinou]

Maybe the default could be made a project-wide and/or editor option (still with DirectX being the default setting for this option)? Different users will have different requirements. Someone who pipelines his assets thought Blender (using OpenGL normals, normal baking and so on) would probably prefer the OpenGL option (I know I would).

I would prefer using the same orientation everywhere, so that you can port assets easily across several Godot projects without them breaking in subtle ways. Not making the behavior dependent on a project setting also makes troubleshooting easier.

I think Blender also uses DirectX normal maps anyway, or that's how I remember it working at least back when I did normal map baking.

[soundsbeard]

DirectX-style normal maps are still more prevalent overall

i strongly disagree with this statement. i set up levels for different projects using all available assets, including cc-0/cc-by props from blendswap, sketchfab and numerous websites with free models and textures. and i would say that its 50/50 division of usage normal map types.

for some people its hard to detect the actual type of a normal map just looking at a 2d image. and sadly, most of 3d artists do not burden themselves to name their file names correctly, like car_normal_dx or thingy_normal_ogl.
and i doubt, that

having this happen out of the box prevent most cases where people don't notice that their normal maps are the wrong way around

so the most difficult thing here is to detect the correct map type.
i want to point out, that unity also uses ogl maps by default.
https://docs.unity3d.com/Manual/StandardShaderMaterialParameterNormalMap.html
(they do not state it directly, but all the examples shown are clearly ogl maps)

and while you putting the question about compatibility-breaking change, i want to say that unity also has an option of generating normal map straight out of height map
https://docs.unity3d.com/uploads/Main/BumpMapImportInspectorGreyscale.png
just like substance painter converts all the bumps including heights and normals into any desired export map.

so, as a 3d artist, i would like to see this option instead of constant checking the correct map type and flipping damned green channel over and over again.
please consider discussing this compatibility-breaking change.

[Calinou]

i want to point out, that unity also uses ogl maps by default.
docs.unity3d.com/Manual/StandardShaderMaterialParameterNormalMap.html
(they do not state it directly, but all the examples shown are clearly ogl maps)

Taking this example from the page you linked:

I'm pretty sure this is a DirectX normal map, since the green channel is highlighting the upper parts of the bumps, not the lower parts. To the human eyes, what's pointing upwards looks brighter.

and while you putting the question about compatibility-breaking change, i want to say that unity also has an option of generating normal map straight out of height map

Godot provides this via the Image.bumpmap_to_normalmap() method, but it leaves to be desired for general-purpose use and is not exposed to the editor. It's mainly intended for procedural generation purposes.

[soundsbeard]

Godot provides this via the Image.bumpmap_to_normalmap() method, but it leaves to be desired for general-purpose use and is not exposed to the editor. It's mainly intended for procedural generation purposes.

thanks, good to know, but how a level designer can possibly know that if its hidden in the editor?

[Calinou]

thanks, good to know, but how a level designer can possibly know that if its hidden in the editor?

It could be exposed as an import option. If it provides good quality, it can be a good tradeoff for reducing project file size on the VCS (which is important when using Git). However, there are currently several problems with doing this:

• The resulting quality is lower than a manually generated normal map, often in a noticeable way. This requires Make Image.bumpmap_to_normalmap() more general and adjustable #659 to be resolved first. High quality mode would likely always be enabled when using it as an import option.
• In master, the importer doesn't know anymore whether a texture is repeating or not. For proper normal map generation, you need to know whether the texture is repeating. This would require adding another import option to specify it, which further adds complexity. This would likely default to "repeating", since that's how most textures are used in 3D.
• Godot expects a 1:1 mapping between source files and the imported resources. Therefore, if you want to use the height map both as a normal map and as a height map (for parallax mapping), you have to include two copies of the image, which unnecessarily increases the project's file size.
  • There would still be a file size advantage when duplicating the image (since a 8-bit luminance image is 3 times smaller than a 8 bpc RGB image on disk), but not as much as in an "ideal" situation without any file duplication.

[soundsbeard]

To the human eyes

you can detect the correct normal map type pretty easily.

1. you can visually separate bright blue from bright red with slash in ogl (like separation for folder paths on linux).
   and with backslash for dx map (like separation for folder paths on windows) https://i.imgur.com/0D4yrPR.png

2. Roy G. Biv - a mnemonic for rainbow colors. opengl has these colors counter-clockwise no matter if its a bump or a hole.
   while dx has it clockwise. https://i.imgur.com/qnSMPVD.png

so you can follow this logic on any of provided unity maps, you can come to the conclusion, that its an ogl map type.

[Calinou]

After inspecting this normalmap from ambientCG (https://ambientcg.com/view?id=Bricks066), it appears you're correct. However, my point still stands: ambientCG distributes DirectX-style normal maps, and I want them to look correct out of the box 🙂
I edited my first post accordingly.

Detecting whether a normal map is correctly oriented in a programmatic way is an interesting problem, but it sounds pretty complex, especially if you want it to be reasonably fast.

[soundsbeard]

i also prepared everything in blender, so you can just hit bake to see the final result.
https://drive.google.com/file/d/1yrA-EL9PSM0fPVMRpks773cuMDocLMPW/view?usp=sharing

you will see this: https://i.imgur.com/pvBAo92.png
you can also set blender to flip y while baking to see how dx map will look like

[soundsbeard]

also there is a mistake in thread name. ogl is y+, while dx is y-
i also think that foss engine should follow foss standards by default, while even proprietary untiy uses ogl maps.

[Zireael07]

i also think that foss engine should follow foss standards by default,

The thing is, there are no standards, it's a coin flip whether the asset you have has one or the other convention.

[soundsbeard]

Most popular 3D DCCs and game engines have settled on DirectX-style normal maps

while saying this you also give a link, where ogl (y+) has 5 points, and dx has 4.

@Zireael07 most props (~60%) i download from sketchfab have ogl maps. this coin flips to ogl more frequently, at least for me.

[Zireael07]

Yeah, so far most normalmaps I've used (sketchfab, blendswap, cc0textures, some Open Game Art stuff, and the like), have had normalmaps that could be used out of the box.

Still, because there are outliers that are the other way around, it would be good maybe not to switch to the other convention as @Calinou suggests, but to have a dropdown somewhere clearly visible on import where you can choose which convention this particular asset uses

[Calinou]

but to have a dropdown somewhere clearly visible on import where you can choose which convention this particular asset uses

This already exists: godotengine/godot#39202

It will be available in 3.4beta1 too.

[Calinou]

Closing in favor of godotengine/godot-docs#4192, as the current orientation seems to work well enough as long as there's an easy way to invert the normal map direction (Normal Map Invert Y in the Import dock).