Integrate AMD FSR 2 in the Vulkan renderer (currently, FSR 1.0 is supported)

[ArseniyMirniy]

Bugsquad edit: Related to #2809, which is about the FSR 1.0 implementation (which is now in master).

Describe the project you are working on

This feature will help almost any project to improve FPS on low-end systems.

Describe the problem or limitation you are having in your project

There is never too many FPS.

Describe the feature / enhancement and how it helps to overcome the problem or limitation

AMD showcased their new FSR 2.0 that has vector data and previous frames data included into the algorithm, so the quality is way better, than the original FSR 1.x branch. It allows to properly work with thin lines, like ropes or netting fences.

Describe how your proposal will work, with code, pseudo-code, mock-ups, and/or diagrams

AMD Described it better: https://gpuopen.com/fidelityfx-superresolution-2/

[All images ZIP – 155MB]

Scene 1 – Native
Scene 1 – Quality
Scene 1 – Balanced
Scene 1 – Performance
Scene 2 – Native
Scene 2 – Quality
Scene 2 – Balanced
Scene 2 – Performance

If this enhancement will not be used often, can it be worked around with a few lines of script?

Nope, not even close. Unlike FSR 1.0 this feature must be enabled on rendering pipeline, so Vector Data and Previous Frames will be available for the algorithm.

Is there a reason why this should be core and not an add-on in the asset library?

Unlike FSR 1.0 this feature must be enabled on rendering pipeline, so Vector Data and Previous Frames will be available for the algorithm. Only with this data small and thin details can be saved on the output.

[Calinou]

Related to #3401, which is a prerequisite for this proposal to be implemented (as motion vectors are required).

Some notes:

• FSR 2.0 does not have a Vulkan/GLSL version yet – it's currently limited to Direct3D 12/HLSL.
• FSR 2.0 isn't open source yet. Given how fast FSR 1.0 was open sourced after being unveiled, I expect FSR 2.0 to be open sourced in about 3 months from now (June-July 2022).

This feature will help almost any project to improve FPS on low-end systems.

Remember that this is a temporal upscaling algorithm, and these aren't cheap to run on very low-end GPUs (like integrated graphics). FSR 2.0 should work out nicely on mid-range dedicated graphics, but old/low-end dedicated GPUs might struggle with it too (except if targeting 30 FPS).

For mobile, FSR Lite (which is a derivative of FSR 1.0) will likely remain the best approach for the foreseeable future.

[ArseniyMirniy]

Remember that this is a temporal upscaling algorithm, and these aren't cheap to run on very low-end GPUs (like integrated graphics). FSR 2.0 should work out nicely on mid-range dedicated graphics, but old/low-end dedicated GPUs might struggle with it too (except if targeting 30 FPS).

For mobile, #3786 (which is a derivative of FSR 1.0) will likely remain the best approach for the foreseeable future.

For my own projects Mid-range devices are probably even more important area, and also, if internal solution with TAA is possible outside of FSR, it's fine to have it. If FSR 2.0 will have Vulkan version, it's also fine to see, which one is better. Totally up to you, guys.

[yhyu13]

@Calinou

FSR 2.0 does not have a Vulkan/GLSL version yet – it's currently limited to Direct3D 12/HLSL.

Do you mean the shader code or does FSR 2.0 relies on DX specific APIs?

[Calinou]

Do you mean the shader code or does FSR 2.0 relies on DX specific APIs?

It's a mixture of both right now. I'm fairly sure Vulkan equivalents do exist for anything FSR 2.0 does, but AMD still needs to do the porting work 🙂

[Calinou]

FSR 2.0 is out in Deathloop (but its code hasn't landed on GPUOpen yet). Here's a comparison against native rendering and DLSS: https://www.techpowerup.com/review/amd-fidelity-fx-fsr-20/

FSR 2.0 seems to compare favorably to DLSS, and it can also be paired with dynamic resolution scaling.

For reference, TAA support in Godot (which is a prerequisite for FSR 2.0 due to the motion vectors) is still being developed as we speak.

[Zireael07]

motion vectors

Does that mean we're finally getting built-in motion blur in Godot?

[ArseniyMirniy]

Does that mean we're finally getting built-in motion blur in Godot?

Yes, it would be great to have the ability to build a proper motion blur (currently, you can make camera-based one)

[ArseniyMirniy]

In addition, it seems, that FSR 2.0 is not just good, but "Better than native" even at 4K resolution, it nearly eliminates common AA artifacts, like shimmery lines:

https://www.youtube.com/watch?v=s25cnyTMHHM

[Calinou]

motion vectors

Does that mean we're finally getting built-in motion blur in Godot?

Motion blur is not planned for 4.0, but contributions on this subject are welcome. Since motion vectors will be exposed to custom shaders, it should also be possible to write a shader that makes use of it to create better motion blur than the current motion blur shaders out there.

[Butzwack]

https://github.com/GPUOpen-Effects/FidelityFX-FSR2
The source code for FSR 2.0 is now available.

[ArseniyMirniy]

https://github.com/GPUOpen-Effects/FidelityFX-FSR2 The source code for FSR 2.0 is now available.

The recent Alpha 10 also features TAA, which means, they implemented vector data and are able to implement FSR 2.0 (or the equivalent).

[Anutrix]

With Beta(feature freeze) being so close, just wanted to know if this will make it to 4.0.

It took less than 90 mins for Spartan Engine on live stream(https://www.youtube.com/watch?v=QhyMal6RY7M). And we know that every engine is different, we think it's possible for interested Godot Devs to make it before Beta.

[Calinou]

With Beta(feature freeze) being so close, just wanted to know if this will make it to 4.0.

Probably not for 4.0, maybe for 4.1. Spartan removed its internal TAA implementation in favor of FSR 2.0¹ (see godotengine/godot#62865), but the goal is to finalize the existing TAA implementation in Godot (especially motion vector generation). Right now, TAA doesn't even work properly if you have skeletal-animated meshes or moving UVs, and this needs to be fixed first. Jittering various screen-space effects over time to improve their quality is also important, and it can be done with traditional TAA too.

Spartan also uses HLSL instead of GLSL, which probably makes the FSR 2.0 integration a bit easier. (FSR 2.0 does have a GLSL mode, but I assume it has received less testing as most AAA developers use HLSL in their engines.)

Lastly, remember that since FSR 2.0 is significantly more demanding than FSR 1.0, you'll have to lower your input resolution slightly to match performance with FSR 1.0. This means that if the camera moves very fast, FSR 2.0 won't look any better than FSR 1.0 as it won't have the necessary time to converge frames.

Footnotes

1. It's debated whether removing TAA in favor of FSR 2.0 is currently a good idea, as FSR 2.0 is currently not designed to provide TAA at native resolution. It can be done, but it'll have a much larger performance impact compared to traditional TAA (albeit with higher quality/less blurriness, I presume). ↩

[Anutrix]

Thx for the info.

It's debated whether removing TAA in favor of FSR 2.0 is currently a good idea, as FSR 2.0 is currently not designed to provide TAA at native resolution. It can be done, but it'll have a much larger performance impact compared to traditional TAA (albeit with higher quality/less blurriness, I presume). ↩

I agree that it shouldn't replace TAA. And if it's possible to keep it neat and maintainable, keeping both TAA and FSR 2.0 as options would nice. Good to know the status.
Will Wait For it in Godot 4.1, if not 4.0.

[ghost]

There is FSR 2.2 already, very thicc

[TheJackiMonster]

I would like to note here that the first step integrate AMD FSR 2.x into Godot it to migrate the existing FSR 1.0 implementation to ffx-fsr2-api [...]

I would recommend to offer both (FSR 1 and FSR 2) since they might have different advantages depending on the game. For example FSR 2 can not handle animated textures without manually hacking in proper motion vectors for those. FSR 1 on the other hand does not have issues with those.

So since we can't estimate what kind of games people create and which features they use, it should be best for developers to decide which option they include for upscaling.

[duplexsystem]

duplexsystem

Feel free to look into it! I am not certain that the FSR2 spatial upscaler is a drop-in solution for FSR 1 as I understand they are processed in different stages of the pipeline. But it is definitely worth checking out. It would be nice to reduce our external dependencies .

Are you comfortable taking a stab at implementing FSR 2.x at the same time?

Yeah I think there are some nuances with tone-mapping and such but considering it's already a part of FSR 2's api it's probably the right path forward to use it. And I can take a stab at FSR 2 (spatial) but no promises :P.

[duplexsystem]

I would like to note here that the first step integrate AMD FSR 2.x into Godot it to migrate the existing FSR 1.0 implementation to ffx-fsr2-api [...]

I would recommend to offer both (FSR 1 and FSR 2) since they might have different advantages depending on the game. For example FSR 2 can not handle animated textures without manually hacking in proper motion vectors for those. FSR 1 on the other hand does not have issues with those.

So since we can't estimate what kind of games people create and which features they use, it should be best for developers to decide which option they include for upscaling.

Sorry I don't thank you understood what I said. FSR 2 the API/Gtihub repo has a spatial upscaler (named fsr1) which should take the place of the spatial upscaler from the FSR 1 API/Github repo as it's currently unmaintained. I was saying I think the first step to adding FSR 2 the temporal upscaler (named fsr2) into Godot would be to replace the spatial upscaler from the FSR 1 API/Github repo with the one from the FSR 2 the API/Gtihub repo. This has nothing to do with getting rid of a spatial upscaler :)

[Butzwack]

AMD just released their FidelityFX SDK.
Announcement: https://gpuopen.com/learn/fidelityfx-sdk-available-now/
Github repo: https://github.com/GPUOpen-LibrariesAndSDKs/FidelityFX-SDK

I don't know much easier this would make the FSR2 implementation, but it includes Variable Rate Shading #3859, Constrast-adaptive Sharpening #1519 and Single Pass Downsampler #3179, effectively killing multiple birds with one stone.
Ray tracing isn't a priority for Godot right now, but the SDK also provides a denoiser and hybrid implementations of reflections and shadows with a reasonable performance cost, which could be useful a few minor versions down the line.

[Calinou]

but it includes Variable Rate Shading #3859

Note that VRS is already implemented in the engine since Godot 4.0. What we're missing is a built-in way to automatically generate a VRS density map based on motion vectors, previous frame brightness, etc. This would allow for improving performance without decreasing quality too much, compared to using a VRS texture that reduces the sharing rate on all pixels that aren't in the center of the screen.

AMD's VRS implementation includes such a way to generate this density texture, which we could extract.

[TheJackiMonster]

AMD just released their FidelityFX SDK. Announcement: https://gpuopen.com/learn/fidelityfx-sdk-available-now/ Github repo: https://github.com/GPUOpen-LibrariesAndSDKs/FidelityFX-SDK

I don't know much easier this would make the FSR2 implementation, but it includes Variable Rate Shading #3859, Constrast-adaptive Sharpening #1519 and Single Pass Downsampler #3179, effectively killing multiple birds with one stone. Ray tracing isn't a priority for Godot right now, but the SDK also provides a denoiser and hybrid implementations of reflections and shadows with a reasonable performance cost, which could be useful a few minor versions down the line.

The most issues will be related to AMD only releasing the SDK for Windows at the moment using Windows specific binaries. It's pretty similar to many of the FidelityFX projects using HLSL shaders which do not work properly with glslangValidator yet. For their ray tracing tools you need the denoiser for example which relies on HLSL shaders and those seem to require a different shader compiler than glslangValidator (most likely DXC which has some issues on Linux from my experience).

Two things I definitely like about that release:

• They finally published the source code used for their custom shader-compiler front-end (which I needed to reverse engineer for FSR2 to have something similar on Linux without wine). ^^'
• Given that they bundle FidelityFX projects now, they will likely all have Vulkan support in the future (The newest projects already supported Vulkan and DirectX but it was never fully clear they would implement Vulkan support in every case).

Edit: For anyone interested in helping to bring the FidelityFX Denoiser support to Linux, here are some issues in glslang which would help compiling the required shaders to SPIRV:

• Constant unsigned integer expressions in HLSL unsupported KhronosGroup/glslang#3089
• Unsupported types in HLSL: float16_t2 and float16_t3 KhronosGroup/glslang#3093

[DarioSamo]

I've been prototyping (at least on Windows so far) what exactly an integration would require and ran into some key decisions that might be necessary.

I've posted all my findings in a document for anyone who wishes to take a look and provide some feedback.

The one key decision as explained in the document would be to decide to choose either integration route: use the existing (and more convenient) APIs that take native resource handles or spend the time and effort for an implementation that uses the FSR interface and integrates into Godot's RenderingDevice (which is up for debate as it'll possibly get refactored in the near future).

There's some other key problems that have been identified due to lack of motion vector information which might need to be split off into its own issue reports and PR as they'd fix problems in the existing TAA implementation as well.

[Calinou]

The one key decision as explained in the document would be to decide to choose either integration route: use the existing (and more convenient) APIs that take native resource handles or spend the time and effort for an implementation that uses the FSR interface and integrates into Godot's RenderingDevice (which is up for debate as it'll possibly get refactored in the near future).

I'm split about which approach is best for Godot here.

Solution A would allow shipping FSR2 as a DLL on Windows, but it can't be used on macOS at all (and not on Linux either to my knowledge, or not with dynamic libraries at least). However, it also has a significant upside: shipping FSR2 as a DLL allows players to update the library themselves without requiring developer intervention. This can be used to improve the appearance of upscaling post-launch if a game is no longer being supported by its developer, and is a common tweak performed with DLSS. With FSR2, it's possible to link statically or dynamically. It seems AAA games roughly have a 50/50 split in terms of whether they go with static or dynamic linking for FSR2. DLSS doesn't provide a choice – it's always dynamically linked.
Another upside of this approach is that we could use the D3D12 version of FSR2 directly when using the D3D12 rendering backend in Godot.

Solution B would better fit Godot's single-binary ethos, which aims to avoid requiring any auxiliary files to work (for non-Mono builds). That said, when using the D3D12 backend, it would be much more involved to use the D3D12 version of FSR2 directly (without going through shader conversion, which may have a performance impact).

The rest of the points outlined in the document make a lot of sense, though I wonder if the sky background could be written to the reactive mask to avoid ghosting artifacts in motion. I also wonder if the shimmering in Bistro is due to vertex jittering being too strong at a distance (I remember @clayjohn spending some time tweaking TAA jitter). See godotengine/godot#61319 for context.

[DarioSamo]

Solution A would allow shipping FSR2 as a DLL on Windows, but it can't be used on macOS at all (and not on Linux either to my knowledge, or not with dynamic libraries at least). However, it also has a significant upside: shipping FSR2 as a DLL allows players to update the library themselves without requiring developer intervention. This can be used to improve the appearance of upscaling post-launch if a game is no longer being supported by its developer, and is a common tweak performed with DLSS. With FSR2, it's possible to link statically or dynamically. It seems AAA games roughly have a 50/50 split in terms of whether they go with static or dynamic linking for FSR2. DLSS doesn't provide a choice – it's always dynamically linked. Another upside of this approach is that we could use the D3D12 version of FSR2 directly when using the D3D12 rendering backend in Godot.

I should note that Solution A is actually using static linking even on my prototype. Using the DLL was phased out by AMD a while ago with 2.2 and static linking is the preferred way of shipping FSR2. In fact, integrating Solution A would require the work of porting over the build system from the repository and compiling the shaders, so I'm fairly sure it'd still work in the supported platforms with enough effort.

I also wonder if the shimmering in Bistro is due to vertex jittering being too strong at a distance (I remember @clayjohn spending some time tweaking TAA jitter). See godotengine/godot#61319 for context.

I believe the shimmering in Bistro is possibly related to the jittering suffering from deviation at long distances with very thin objects, but I'm unaware how other engines fix this issue in raster (as my background for using this comes from RT rendering and it's not so visible there). It'd be interesting to get some input from people who've experienced this issue if possible.

[DarioSamo]

After some discussion with @clayjohn we determined Solution B seems like the most reasonable long term approach, so I'll likely dive into prototyping that later and see if there's any blockers with the current RD.

[DarioSamo]

Happy to report my prototype with Solution B is currently working in this branch. Feel free to give it a go if you're curious, but ghosting issues are expected until the rest of the problems identified in the document are solved.

https://github.com/DarioSamo/godot/tree/fsr2-rd

Most of the issues pointed out in the document still remain to be solved, although some of them like motion vectors for animated meshes are now part of the upstream as of this month.

There's a PR which might be of interest to use as the basis for refactoring this implementation into a rendering effect instead of adding the option as it is right now.

godotengine/godot#80925

[romulasry]

fsr 3, some may guess in early september (6th?)https://www.pcworld.com/article/2041810/fsr-3-amds-open-dlss-3-rival-starts-supercharging-games-in-september.html. See:

[ArseniyMirniy]

fsr 3, some may guess in early september (6th?)

Open Source version will be available much later

In addition, AMD is aiming to make FSR3 automatically compatible with DX11 and DX12 games without integration (probably in worse form with more issues). But Vulkan isn't mentioned in that statement.

[TheJackiMonster]

@Rinocrosser They stated though it will be part of their FidelityFX SDK which aims to support DX12 and Vulkan. So I assume we will see Vulkan support when it's released open-source on Github or at least after some delay.

[mrjustaguy]

That doesn't mean that AMD's Driver solution that he's talking about will be able to work on Vulkan.

[mrjustaguy]

FSR 2 PR is out as a draft - godotengine/godot#81197

[ArseniyMirniy]

FSR 2 PR is out as a draft - godotengine/godot#81197

Thanks a lot! Any known issues?

[mrjustaguy]

Likely, but I'm unaware of any right now, but I'll get to testing it as soon as the artifacts are done

[TheJackiMonster]

That doesn't mean that AMD's Driver solution that he's talking about will be able to work on Vulkan.

As far as I understand it AMD's driver solution only applies to Windows since it will be available as toggle via their Radeon driver software. So I'm not sure whether it makes sense to consider supporting that over an FSR3 implementation inside the engine or game anyway? I think they pitched it as optional feature for older games which do not see any integration ever.

[DarioSamo]

There should be no concern over APIs right now if the implementation on the PR is followed. It's implemented on top of Godot's own abstraction which should run in all backends supported by Forward+ at the moment (which is just Vulkan but DX12 backend is in another PR).