Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

GDExtension: double-precision builds differ in method hashes and lack compatibility #86346

Closed
Bromeon opened this issue Dec 19, 2023 · 5 comments · Fixed by #88188
Closed

GDExtension: double-precision builds differ in method hashes and lack compatibility #86346

Bromeon opened this issue Dec 19, 2023 · 5 comments · Fixed by #88188
Labels
bug confirmed topic:gdextension
Milestone
4.3

Comments

@Bromeon
Copy link
Contributor

Bromeon commented Dec 19, 2023 •
edited
Loading

Tested versions

4.3.dev.custom_build.3ce73e5d4 (built from 3ce73e5)

System information

Windows 10

Issue description

I built a recent master version with both precision=single and precision=double options in scons.

Now I would like to export the extension_api.json. For the single version, it works fine, but for the double version I'm met with dozens of warnings about hash incompatibilities, like this:

WARNING: Compatibility hash 1057544502 mapped to non-existent hash 3656773645. Please update gdextension_compat_hashes.cpp.
     at: GDExtensionCompatHashes::get_legacy_hashes (core\extension\gdextension_compat_hashes.cpp:68)
WARNING: Compatibility hash 2055804584 mapped to non-existent hash 1980270704. Please update gdextension_compat_hashes.cpp.
     at: GDExtensionCompatHashes::get_legacy_hashes (core\extension\gdextension_compat_hashes.cpp:68)
WARNING: Compatibility hash 2055804584 mapped to non-existent hash 1980270704. Please update gdextension_compat_hashes.cpp.
Full output 
λ godot.windows.editor.dev.double.x86_64.console.exe --dump-extension-api --headless
Dumping Extension API
Godot Engine v4.3.dev.custom_build.3ce73e5d4 - https://godotengine.org

WARNING: Compatibility hash 1057544502 mapped to non-existent hash 3656773645. Please update gdextension_compat_hashes.cpp.
     at: GDExtensionCompatHashes::get_legacy_hashes (core\extension\gdextension_compat_hashes.cpp:68)
WARNING: Compatibility hash 2055804584 mapped to non-existent hash 1980270704. Please update gdextension_compat_hashes.cpp.
     at: GDExtensionCompatHashes::get_legacy_hashes (core\extension\gdextension_compat_hashes.cpp:68)
WARNING: Compatibility hash 2055804584 mapped to non-existent hash 1980270704. Please update gdextension_compat_hashes.cpp.
     at: GDExtensionCompatHashes::get_legacy_hashes (core\extension\gdextension_compat_hashes.cpp:68)
WARNING: Compatibility hash 1634855856 mapped to non-existent hash 153818295. Please update gdextension_compat_hashes.cpp.
     at: GDExtensionCompatHashes::get_legacy_hashes (core\extension\gdextension_compat_hashes.cpp:68)
WARNING: Compatibility hash 3283884939 mapped to non-existent hash 288975085. Please update gdextension_compat_hashes.cpp.
     at: GDExtensionCompatHashes::get_legacy_hashes (core\extension\gdextension_compat_hashes.cpp:68)
WARNING: Compatibility hash 2437345566 mapped to non-existent hash 4175465202. Please update gdextension_compat_hashes.cpp.
     at: GDExtensionCompatHashes::get_legacy_hashes (core\extension\gdextension_compat_hashes.cpp:68)
WARNING: Compatibility hash 3836314258 mapped to non-existent hash 2931053748. Please update gdextension_compat_hashes.cpp.
     at: GDExtensionCompatHashes::get_legacy_hashes (core\extension\gdextension_compat_hashes.cpp:68)
WARNING: Compatibility hash 860410478 mapped to non-existent hash 3042891259. Please update gdextension_compat_hashes.cpp.
     at: GDExtensionCompatHashes::get_legacy_hashes (core\extension\gdextension_compat_hashes.cpp:68)
WARNING: Compatibility hash 1358907026 mapped to non-existent hash 1816663697. Please update gdextension_compat_hashes.cpp.
     at: GDExtensionCompatHashes::get_legacy_hashes (core\extension\gdextension_compat_hashes.cpp:68)
WARNING: Compatibility hash 1868867708 mapped to non-existent hash 1579955111. Please update gdextension_compat_hashes.cpp.
     at: GDExtensionCompatHashes::get_legacy_hashes (core\extension\gdextension_compat_hashes.cpp:68)
WARNING: Compatibility hash 1222433716 mapped to non-existent hash 3344325384. Please update gdextension_compat_hashes.cpp.
     at: GDExtensionCompatHashes::get_legacy_hashes (core\extension\gdextension_compat_hashes.cpp:68)
WARNING: Compatibility hash 1149405976 mapped to non-existent hash 1851767612. Please update gdextension_compat_hashes.cpp.
     at: GDExtensionCompatHashes::get_legacy_hashes (core\extension\gdextension_compat_hashes.cpp:68)
WARNING: Compatibility hash 3489634142 mapped to non-existent hash 703945977. Please update gdextension_compat_hashes.cpp.
     at: GDExtensionCompatHashes::get_legacy_hashes (core\extension\gdextension_compat_hashes.cpp:68)
WARNING: Compatibility hash 2747409789 mapped to non-existent hash 1282766827. Please update gdextension_compat_hashes.cpp.
     at: GDExtensionCompatHashes::get_legacy_hashes (core\extension\gdextension_compat_hashes.cpp:68)
WARNING: Compatibility hash 3123400617 mapped to non-existent hash 2882425029. Please update gdextension_compat_hashes.cpp.
     at: GDExtensionCompatHashes::get_legacy_hashes (core\extension\gdextension_compat_hashes.cpp:68)
WARNING: Compatibility hash 4067663783 mapped to non-existent hash 2086826090. Please update gdextension_compat_hashes.cpp.
     at: GDExtensionCompatHashes::get_legacy_hashes (core\extension\gdextension_compat_hashes.cpp:68)
WARNING: Compatibility hash 1002852006 mapped to non-existent hash 2754756483. Please update gdextension_compat_hashes.cpp.
     at: GDExtensionCompatHashes::get_legacy_hashes (core\extension\gdextension_compat_hashes.cpp:68)
WARNING: Compatibility hash 496058220 mapped to non-existent hash 4288681949. Please update gdextension_compat_hashes.cpp.
     at: GDExtensionCompatHashes::get_legacy_hashes (core\extension\gdextension_compat_hashes.cpp:68)
WARNING: Compatibility hash 496058220 mapped to non-existent hash 4288681949. Please update gdextension_compat_hashes.cpp.
     at: GDExtensionCompatHashes::get_legacy_hashes (core\extension\gdextension_compat_hashes.cpp:68)
WARNING: Compatibility hash 496058220 mapped to non-existent hash 4288681949. Please update gdextension_compat_hashes.cpp.
     at: GDExtensionCompatHashes::get_legacy_hashes (core\extension\gdextension_compat_hashes.cpp:68)
WARNING: Compatibility hash 1002852006 mapped to non-existent hash 2754756483. Please update gdextension_compat_hashes.cpp.
     at: GDExtensionCompatHashes::get_legacy_hashes (core\extension\gdextension_compat_hashes.cpp:68)
WARNING: Compatibility hash 1002852006 mapped to non-existent hash 2754756483. Please update gdextension_compat_hashes.cpp.
     at: GDExtensionCompatHashes::get_legacy_hashes (core\extension\gdextension_compat_hashes.cpp:68)
WARNING: Compatibility hash 1002852006 mapped to non-existent hash 2754756483. Please update gdextension_compat_hashes.cpp.
     at: GDExtensionCompatHashes::get_legacy_hashes (core\extension\gdextension_compat_hashes.cpp:68)
WARNING: Compatibility hash 754862190 mapped to non-existent hash 339056240. Please update gdextension_compat_hashes.cpp.
     at: GDExtensionCompatHashes::get_legacy_hashes (core\extension\gdextension_compat_hashes.cpp:68)
WARNING: Compatibility hash 754862190 mapped to non-existent hash 339056240. Please update gdextension_compat_hashes.cpp.
     at: GDExtensionCompatHashes::get_legacy_hashes (core\extension\gdextension_compat_hashes.cpp:68)
WARNING: Compatibility hash 34330743 mapped to non-existent hash 205485391. Please update gdextension_compat_hashes.cpp.
     at: GDExtensionCompatHashes::get_legacy_hashes (core\extension\gdextension_compat_hashes.cpp:68)
WARNING: Compatibility hash 34330743 mapped to non-existent hash 205485391. Please update gdextension_compat_hashes.cpp.
     at: GDExtensionCompatHashes::get_legacy_hashes (core\extension\gdextension_compat_hashes.cpp:68)
WARNING: Compatibility hash 34330743 mapped to non-existent hash 205485391. Please update gdextension_compat_hashes.cpp.
     at: GDExtensionCompatHashes::get_legacy_hashes (core\extension\gdextension_compat_hashes.cpp:68)
WARNING: Compatibility hash 4040559639 mapped to non-existent hash 3711419014. Please update gdextension_compat_hashes.cpp.
     at: GDExtensionCompatHashes::get_legacy_hashes (core\extension\gdextension_compat_hashes.cpp:68)
WARNING: Compatibility hash 4040559639 mapped to non-existent hash 3711419014. Please update gdextension_compat_hashes.cpp.
     at: GDExtensionCompatHashes::get_legacy_hashes (core\extension\gdextension_compat_hashes.cpp:68)
WARNING: Compatibility hash 110375048 mapped to non-existent hash 390416203. Please update gdextension_compat_hashes.cpp.
     at: GDExtensionCompatHashes::get_legacy_hashes (core\extension\gdextension_compat_hashes.cpp:68)
WARNING: Compatibility hash 110375048 mapped to non-existent hash 390416203. Please update gdextension_compat_hashes.cpp.
     at: GDExtensionCompatHashes::get_legacy_hashes (core\extension\gdextension_compat_hashes.cpp:68)
WARNING: Compatibility hash 110375048 mapped to non-existent hash 390416203. Please update gdextension_compat_hashes.cpp.
     at: GDExtensionCompatHashes::get_legacy_hashes (core\extension\gdextension_compat_hashes.cpp:68)
WARNING: Compatibility hash 4252992020 mapped to non-existent hash 2468082605. Please update gdextension_compat_hashes.cpp.
     at: GDExtensionCompatHashes::get_legacy_hashes (core\extension\gdextension_compat_hashes.cpp:68)
WARNING: Compatibility hash 832527510 mapped to non-existent hash 2406300660. Please update gdextension_compat_hashes.cpp.
     at: GDExtensionCompatHashes::get_legacy_hashes (core\extension\gdextension_compat_hashes.cpp:68)
WARNING: Compatibility hash 338791288 mapped to non-existent hash 244650101. Please update gdextension_compat_hashes.cpp.
     at: GDExtensionCompatHashes::get_legacy_hashes (core\extension\gdextension_compat_hashes.cpp:68)
WARNING: Compatibility hash 1278520651 mapped to non-existent hash 1062245816. Please update gdextension_compat_hashes.cpp.
     at: GDExtensionCompatHashes::get_legacy_hashes (core\extension\gdextension_compat_hashes.cpp:68)
WARNING: Compatibility hash 2180266943 mapped to non-existent hash 1333997032. Please update gdextension_compat_hashes.cpp.
     at: GDExtensionCompatHashes::get_legacy_hashes (core\extension\gdextension_compat_hashes.cpp:68)
WARNING: Compatibility hash 3548053052 mapped to non-existent hash 316450961. Please update gdextension_compat_hashes.cpp.
     at: GDExtensionCompatHashes::get_legacy_hashes (core\extension\gdextension_compat_hashes.cpp:68)
WARNING: Compatibility hash 3346058748 mapped to non-existent hash 3580801207. Please update gdextension_compat_hashes.cpp.
     at: GDExtensionCompatHashes::get_legacy_hashes (core\extension\gdextension_compat_hashes.cpp:68)
WARNING: Compatibility hash 311501835 mapped to non-existent hash 4061635501. Please update gdextension_compat_hashes.cpp.
     at: GDExtensionCompatHashes::get_legacy_hashes (core\extension\gdextension_compat_hashes.cpp:68)
WARNING: Compatibility hash 496058220 mapped to non-existent hash 4288681949. Please update gdextension_compat_hashes.cpp.
     at: GDExtensionCompatHashes::get_legacy_hashes (core\extension\gdextension_compat_hashes.cpp:68)
WARNING: Compatibility hash 496058220 mapped to non-existent hash 4288681949. Please update gdextension_compat_hashes.cpp.
     at: GDExtensionCompatHashes::get_legacy_hashes (core\extension\gdextension_compat_hashes.cpp:68)
WARNING: Compatibility hash 496058220 mapped to non-existent hash 4288681949. Please update gdextension_compat_hashes.cpp.
     at: GDExtensionCompatHashes::get_legacy_hashes (core\extension\gdextension_compat_hashes.cpp:68)
WARNING: Compatibility hash 1002852006 mapped to non-existent hash 2754756483. Please update gdextension_compat_hashes.cpp.
     at: GDExtensionCompatHashes::get_legacy_hashes (core\extension\gdextension_compat_hashes.cpp:68)
WARNING: Compatibility hash 1002852006 mapped to non-existent hash 2754756483. Please update gdextension_compat_hashes.cpp.
     at: GDExtensionCompatHashes::get_legacy_hashes (core\extension\gdextension_compat_hashes.cpp:68)
WARNING: Compatibility hash 1002852006 mapped to non-existent hash 2754756483. Please update gdextension_compat_hashes.cpp.
     at: GDExtensionCompatHashes::get_legacy_hashes (core\extension\gdextension_compat_hashes.cpp:68)
WARNING: Compatibility hash 2613124475 mapped to non-existent hash 2498990330. Please update gdextension_compat_hashes.cpp.
     at: GDExtensionCompatHashes::get_legacy_hashes (core\extension\gdextension_compat_hashes.cpp:68)

Furthermore, the two generated extension_api.json files are different.
Primarily, method parameters differ:

  1. Parameters have "meta": "double" instead of "meta": "float"
  2. hash_compatibility is absent
  3. hash is sometimes different
  4. very rarely, PackedFloat64Array is used instead of PackedFloat32Array

Which of the above are intended, and which are not?

  • If the signatures are supposed to be different, why do we have the builtin_class_sizes JSON object? That one lists all possible configurations, not just the current one -- so why would other places be specific to the current build?
  • This makes it harder to build a GDExtension for double precision, without first recompiling the engine. Typically people have the engine around, but theoretically one can just program against the API and run it elsewhere (different machine, deployment server, CI, ...).
  • Could we use the real type (real_t in C++) to abstract some of those differences?

I'll attach the two files, as well as the diff from single to double.
This was referenced Dec 20, 2023
Merged
Merged
@nickpolet
Copy link

I get similar issues to you with precision=double builds. This results in an error when trying to call a specific function (PhysicsServer3D.body_add_shape in my case).

image

I also get similar output when running --dump-extension-api on the built binary. Single precision doesn't have any warnings, but double has a lot.

@nickpolet nickpolet mentioned this issue Jan 9, 2024
Closed
@dsnopek
Copy link
Contributor

dsnopek commented Jan 10, 2024

Some amount of differences between double and single precision are expected, but based on this error message:

WARNING: Compatibility hash 1057544502 mapped to non-existent hash 3656773645. Please update gdextension_compat_hashes.cpp.
     at: GDExtensionCompatHashes::get_legacy_hashes (core\extension\gdextension_compat_hashes.cpp:68)

I think what you might be hitting is a bug in gdextension_compat_hashes.cpp, because we didn't account for those differences when building the table of hashes in that file.

I suspect we'll need to figure out which of those need to change for double precision builds, and then add some #ifdefs to switch to them.

@YuriSizov
Copy link
Contributor

Furthermore, the two generated extension_api.json files are different.

Note that the official CI is also affected since we generate the dump with the double-precision build, coincidentally. Warnings are non-blocking, but if files are different then we might have issues with the test godot-cpp build we run afterwards. Should we use a non-DP build for the API dump to make sure we don't miss any issues?

This also applies to 4.2, not just 4.3, btw.

@dsnopek
Copy link
Contributor

dsnopek commented Jan 26, 2024

Discussed at the GDExtension meeting, and we agreed that it's OK that the hashes are different between single and double precision builds. This may even be useful, since the functions aren't binary compatible, so you'd find out early, rather than calling the function and getting strange behavior.

(However, there still is a bug in gdextension_compat_hashes.cpp that needs to be resolved)

@dsnopek dsnopek mentioned this issue Feb 11, 2024
Merged
@dsnopek
Copy link
Contributor

dsnopek commented Feb 11, 2024

Sorry it took me so long to get to this one! It's just... really, really, really tedious to do :-)

I just posted PR #88188 to fix the incorrect hashes in gdextension_compat_hashes.cpp for double precision builds

@akien-mga akien-mga added this to the 4.3 milestone Feb 12, 2024
@dsnopek dsnopek mentioned this issue Feb 24, 2024
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug confirmed topic:gdextension
Projects
None yet
Milestone
4.3
Development

Successfully merging a pull request may close this issue.

Fix gdextension_compat_hashes.cpp for double precision builds dsnopek/godot
6 participants
@Calinou @dsnopek @Bromeon @akien-mga @nickpolet @YuriSizov