Embind problems with val::as<T>() in wasm32 with MAXIMUM_MEMORY=4GB

[pavshen]

There is a problem with raw pointers previously returned from embind-wrapped C++ functions by val(ptr) and structs embind-registered as value_object used later as in-parameters of type val of embind-wrapped C++ functions. Their conversion from val to native types by val::as<T>() inside the C++ function produces garbage in case of memory locations above 2GB (in raw pointer when its $$.ptr is negative; as for value_object, I don't know when). On the other hand, when they are used in functions with parameters of their native types without the usage of val they are marshaled OK. The same is with JS string literals passed as val and converted with val::as<std::string>().

(I need to pass them as val in order to support variable-type and nullable parameters).

I thought #19755 would fix this, but the problem still exists.

Version of emscripten/emsdk:

emcc (Emscripten gcc/clang-like replacement + linker emulating GNU ld) 3.1.44 (bec42dac7873903d09d713963e34020c22a8bd2d)
clang version 17.0.0 (https://github.com/llvm/llvm-project a8cbd27d1f238e104a5d5ca345d93bc1f4d4ab1f)
Target: wasm32-unknown-emscripten
Thread model: posix

[sbc100]

I have fix pending in 5fbde92

[pavshen]

I merged 5fbde92 into my compiled version of tag 3.1.44 (I also replaced several additional relevant files that were changed after 3.1.44), It worked below 2GB, but above 2GB all three mentioned cases didn't work. I saw that $$.ptr's were positive for raw pointers above 2GB, but as<T>() still produced garbage.

[pavshen]

I have fix pending in 5fbde92

@sbc100, any ideas why the fix doesn't work (see above)?

[sbc100]

I am working on a bunch of >4gb changes right now. I imagine once they all land this issue should be resolved.

[pavshen]

Do those changes include fixes in WebGL? #18744 fixes WebGL1 problems beyond 2GB, WebGL2 still crashes in many functions.

[sbc100]

The embind fixes have now landed and all the tests pass. If you example still fails perhaps you could come up with a small example of the failure code and we can turn that into a new test and fix it?

[pavshen]

@sbc100, I can try to test it tomorrow, but I saw #20077 and #20079. Should I either test without them or try to take them as well or wait until they have been merged?

Are you planning to fix WebGL, too? #18744 is old (for 3.1.38) and it fixes only WebGL1 problems beyond 2GB, WebGL2 still crashes in many functions.

[sbc100]

Yes, I would wait until both #20077 and #20079.

And yes, the plan to get everything working with wasm64 >4gb and with wasm32 >2gb. We are making good progress towards that.

[pavshen]

Let me know when it is a good time to test embind and/or WebGL with wasm32 >2GB and I will update you about the results.

I'll be very glad to test wasm64 >4GB when it is ready. But, TypedArrays with long indices are not supported yet by browsers, are they?

[sbc100]

For large typed arrays we now have a polyfil that allows us to get stuff running: #19737

[pavshen]

My example still fails with 3.1.47. I will check which cases of the above-mentioned fail and will try to prepare small examples.

[pavshen]

I checked again my example with 3.1.57 and 3.1.60. Now it works! I think the issue can be closed.