llvm17/rust: random variations in binaries #72206
Comments
asl
added
incomplete
|
Have you reported this issue to Rust? Are you sure it's an LLVM problem and not Rust frontend? |
|
Not yet. It was not clear to me how to distinguish LLVM/rust problems. And most past rust problems turned out to be in libLLVM. e.g. rust-lang/rust#57041 |
|
btw: re-building llvm 17.0.4 itself produces similar diffs in random appendages in one file: |
|
Also I've been unable to locate the source for this |
|
Looks like it's https://github.com/ms-jpq/sad, looking at the opensuse package. |
|
Thanks! The nondeterminisim there seems to come from its build script, which executes I would not be surprised if all the above crates are causing you trouble because of implementation decisions in their source, not issues with the toolchain. I will not be debugging any more here. |
|
I filed rust-lang/rust#128675 yesterday and the rust people say it might be the build tools' fault. Which it apparently was for sad + tiny, but that was not obvious from the diff that had llvm random/hash IDs all over the place. That still leaves the part about llvm's own |
|
As documented with the description of symbol-mangling, this is done by LLVM during LTO: https://github.com/rust-lang/rust/blob/b586701f78a6d5c7f618b76e7ae3cace9a6fbf37/src/doc/rustc/src/symbol-mangling/v0.md?plain=1#L996-L1013 What LTO settings are you using, and what linker? |
|
https://github.com/bmwiedemann/openSUSE/blob/master/packages/l/llvm18/llvm18.spec#L1091 says we use build log shows the line that creates libomp as |
|
that's not LLVM 17? can you verify that the libomp.so output is non-reproducible with the same build command passed each time? |
|
I always compare with the same build tools. The libomp issue just affects several llvm versions in the same way. Here is the log from llvm17: |
|
I see. Does altering the number of ThinLTO jobs or not using ThinLTO change this? |
|
...hm. Apparently the build winds up passing -flto=thin twice? and both |
If there is fault, it lies with the authors of those codebases. Not the build tools. I debugged both of those examples very quickly by diffing the disassembly with |
|
I think, building without lto gives a clearer view on the diff, that makes debugging the actual issue much easier. e.g. I was looking at https://rb.zq1.de/compare.factory-20240731/diffs/sad-compare.out that had 25 unrelated diffs from lto and rust's codegen-units=16 in the disassembly. |
|
Without LTO, libomp.so becomes reproducible. |
|
There seems to be a |
|
This is the only hint I can find: https://github.com/llvm/llvm-project/blob/main/llvm/test/ThinLTO/X86/promote-local-name.ll#L14 |
|
It seems the names added by ThinLTO are based on a hash of the module's IR, which suggests to me that it should be more deterministic than it actually seems to be? It seems like it would be best to refocus your efforts on that. |
While working on reproducible builds for openSUSE, I found that
various rust packages produced different binaries in every build,
even when trying to run builds as similar as possible.
versions:
affected packages:
Diffs (from
filterdiff strings {1,2}/usr/bin/sad) look like this:so it seems there is some random number appended and that influences ordering.
The text was updated successfully, but these errors were encountered: