wasm32-unknown-unknown "C" ABI doesn't use proper ABI adjustments

[RalfJung]

rustc has a PassMode enum to declare how arguments should be passed between functions. PassMode::Direct is used for the simple case where we can just generate LLVM IR with an argument type that matches the usual LLVM type for this Rust type. It should only be used for types with Scalar or Vector ABI; for larger types, Rust will generate LLVM struct/union types and those

• are sometimes adjusted to obtain better code generation
• make it very hard to reason about our effective ABI, since it depends on how LLVM lowers those complex struct types into an effective ABI. In particular, this causes a huge risk of violating our ABI compatibility guarantees, e.g. for repr(transparent) types.

Unfortunately the compiler never had an assertion ensuring that PassMode::Direct is only used with Scalar/Vector types, and so one target accidentally broke this rule: wasm32-unknown-unknown. Later all wasm targets got an extern "wasm" ABI that has the same issue. We should find some way to fix this target, as the current situation makes life harder for anyone wanting to tweak our Rust-to-LLVM-type-mapping, and for alternative backends that have to match our effective ABI.

The issue is that last time @bjorn3 tried to fix this, it lead to #81386 and had to be reverted. It will be basically impossible to do this change without breaking the ABI at least for some cases. I'm not sure how much ABI breakage we can get away with. This might have to be blocked on rust-lang/compiler-team#672 which should help to implement the current effective ABI at least for types that are ABI-stable (i.e., repr(C) and repr(transparent)).

Cc #122532

[LegNeato]

FWIW, rust-gpu had to use the new make_direct_deprecated(): https://github.com/EmbarkStudios/rust-gpu/pull/1109/files#diff-9ff3a35aa9e19911b0f93cebdc627768b3ca62ee19cb2c233c787606df7a4426R67. I don't know enough about either project to know if it is the same issue as this or if it should be fixed on the rust-gpu side, but I wanted to add a link here for posterity.

[RalfJung]

Hm yeah that's not great, though it might be related to #117271 given both are about GPUs.

[eddyb]

Hm yeah that's not great, though it might be related to #117271 given both are about GPUs.

I'm reviewing the change @LegNeato mentioned and ended up here semi-randomly, so I might as well answer this:

Rust-GPU only started using PassMode::Direct:

• mostly to avoid PassMode::Cast, as it's transmute-like
  _{(as it gets into "antiquated GPU typed memory lacking unions" territory, or at least ptr<->int transmutes - and even with those being theoretically solvable, in the short term it helps to not even have the problem)}
• skipping PassMode::Indirect as well avoids having to eliminate memory accesses (i.e. via equivalents of LLVM mem2reg+SROA)
  _{(not strictly necessary: calls or even just moving values around already likely introduces this problem intra-function, and any calls with problematic indirection would get inlined anyway)}

So it probably makes most sense to turn PassMode::Cast into PassMode::Indirect instead, anyway (which AFAICT is equivalent to using ArgAbi::new and just never doing any adjustment logic beyond "ignore ZSTs").

And while writing this comment I went ahead and tried it and the only issues stem from lacking extra logic to handle the indirect case in a few places _{^{(mainly the format_args! "decompiler")}}, so Rust-GPU wouldn't get impacted by make_direct_deprecated being removed, beyond a bit of busywork.
_{(which I'm only punting on in the very short term because of similar-yet-different changes needed to update to newer nightlies etc., but after that I'll be happy to get rid of another hack from Rust-GPU)}