Bit-for-bit deterministic / reproducible builds #34902

infinity0 · 2016-07-18T16:40:21Z

It would be good if rustc could generate bit-for-bit reproducible results, even in the presence of minor system environment differences. Currently we have quite a large diff: e.g. see txt diff for 1.9.0 or perhaps txt diff for 1.10.0 a few days after I'm posting this. (You might want to "save link as" instead of displaying it directly in the browser.)

Much of the diff output is due to build-id differences, which can be ignored since they are caused by other deeper issues and will go away once these deeper issues are fixed. One example of a deeper issue is this:

│   │   │   ├── ./usr/lib/i386-linux-gnu/librustc_data_structures-e8edd0fd.so
│   │   │   │   ├── readelf --wide --file-header {}
│   │   │   │   │ @@ -6,15 +6,15 @@
│   │   │   │   │    OS/ABI:                            UNIX - System V
│   │   │   │   │    ABI Version:                       0
│   │   │   │   │    Type:                              DYN (Shared object file)
│   │   │   │   │    Machine:                           Intel 80386
│   │   │   │   │    Version:                           0x1
│   │   │   │   │    Entry point address:               0x4620
│   │   │   │   │    Start of program headers:          52 (bytes into file)
│   │   │   │   │ -  Start of section headers:          338784 (bytes into file)
│   │   │   │   │ +  Start of section headers:          338780 (bytes into file)
│   │   │   │   │    Flags:                             0x0
│   │   │   │   │    Size of this header:               52 (bytes)
│   │   │   │   │    Size of program headers:           32 (bytes)
│   │   │   │   │    Number of program headers:         8
│   │   │   │   │    Size of section headers:           40 (bytes)
│   │   │   │   │    Number of section headers:         30
│   │   │   │   │    Section header string table index: 29
│   │   │   │   ├── readelf --wide --program-header {}

Here are the system variations that might be causing these issues. I myself am not that familiar with ELF, but perhaps someone else here would know why the section header is 4 bytes later in the first vs second builds.

The text was updated successfully, but these errors were encountered:

steveklabnik · 2016-07-18T16:49:23Z

In general, being reproducible is something we're interested in; we try to tackle it bug by bug.

eefriedman · 2016-07-18T18:02:18Z

This probably has some obvious cause:

│   │   │   │ -<h4 id='method.unzip' class='method'><code>fn <a href='../../core/iter/trait.Iterator.html#method.unzip' class='fnname'>unzip</a>&lt;A, B, FromA, FromB&gt;(self) -&gt; (FromA, FromB) <span class='where'>where FromB: <a class='trait' href='../../core/default/trait.Default.html' title='core::default::Default'>Default</a> + <a class='trait' href='../../core/iter/trait.Extend.html' title='core::iter::Extend'>Extend</a>&lt;B&gt;, FromA: <a class='trait' href='../../core/default/trait.Default.html' title='core::default::Default'>Default</a> + <a class='trait' href='../../core/iter/trait.Extend.html' title='core::iter::Extend'>Extend</a>&lt;A&gt;, Self: <a class='trait' href='../../core/iter/trait.Iterator.html' title='core::iter::Iterator'>Iterator</a>&lt;Item=(A, B)&gt;</span></code></h4>
│   │   │   │ +<h4 id='method.unzip' class='method'><code>fn <a href='../../core/iter/trait.Iterator.html#method.unzip' class='fnname'>unzip</a>&lt;A, B, FromA, FromB&gt;(self) -&gt; (FromA, FromB) <span class='where'>where FromB: <a class='trait' href='../../core/default/trait.Default.html' title='core::default::Default'>Default</a> + <a class='trait' href='../../core/iter/trait.Extend.html' title='core::iter::Extend'>Extend</a>&lt;B&gt;, Self: <a class='trait' href='../../core/iter/trait.Iterator.html' title='core::iter::Iterator'>Iterator</a>&lt;Item=(A, B)&gt;, FromA: <a class='trait' href='../../core/default/trait.Default.html' title='core::default::Default'>Default</a> + <a class='trait' href='../../core/iter/trait.Extend.html' title='core::iter::Extend'>Extend</a>&lt;A&gt;</span></code></h4>

If someone wants to look at the code generation differences, it's probably best to start with libcore. The reproducible-builds.org diff isn't that useful for that because it doesn't recognize .rlib files as archives.

infinity0 · 2016-07-18T19:12:11Z

Thanks for the heads up, I just added AR support in diffoscope and hopefully the diff output will see that in the next few weeks, when the website updates.

infinity0 · 2016-08-16T10:37:39Z

Here's a diff of rust.metadata.bin from libcore.rlib, from 1.10.0. (2.0 MB, ~1700 lines, might make your browser slow.) Mostly ordering differences. @eddyb has suggested HashMap ordering as a culprit, and that would be replaced with FnvHashMap instead.

eddyb · 2016-08-16T14:08:23Z

If it's a HashMap, I don't know which one. cc @rust-lang/compiler

infinity0 · 2016-08-20T19:58:37Z

I tried replacing HashMap with FnvHashMap in src/librustc_metadata/loader.rs but it didn't seem to help. Here is the Makefile i'm using to automate my rebuilds:

TRIPLET = x86_64-unknown-linux-gnu

all: libcore.diff

libcore.diff: rust.metadata.bin.1 rust.metadata.bin.2
    diffoscope --html-dir "$@" rust.metadata.bin.1 rust.metadata.bin.2 \
        --max-diff-block-lines 5000 \
        --max-diff-input-lines 10000000 \
        --max-report-size 204800000; true # diffoscope returns 1 for "there were diffs"

rust.metadata.bin.1 rust.metadata.bin.2: Makefile
    rm -f $(TRIPLET)/stage2/lib/rustlib/$(TRIPLET)/lib/libcore-*.rlib
    rm -f $(TRIPLET)/stage2/lib/rustlib/$(TRIPLET)/lib/stamp.core
    $(MAKE) $(TRIPLET)/stage2/lib/rustlib/$(TRIPLET)/lib/stamp.core
    ar x $(TRIPLET)/stage2/lib/rustlib/$(TRIPLET)/lib/libcore-*.rlib rust.metadata.bin
    mv rust.metadata.bin "$@"

.PHONY: all rust.metadata.bin.1 rust.metadata.bin.2

perhaps i'm Doing It Wrong.

infinity0 · 2016-08-20T20:02:11Z

As background, HashMap in rust is non-deterministic to protect against certain types of DoS attack. You can switch it to the deterministic FnvHashMap if you're sure your code will always be called in a safe manner. This ought to be true for rustc itself. (I notice some online "try-it-yourself" rust web services let me run "ls /" and other shell commands, so I could also exploit this HashMap ordering issue, but I also assume that they're clever enough to set a ulimit and/or containerise the thing.)

jonas-schievink · 2016-08-22T18:11:21Z

I wrote a small script to diff metadata (I couldn't really get the makefile to work).

It looks like even more is changing between two compiles using the current master (or nightly): The crate hash and/or disambiguator, which are stored right after the target triple. @infinity0 can you confirm?

EDIT: That might get fixed by #35854

jonas-schievink · 2016-08-22T20:13:48Z

Okay, apparently replacing Hash{Map,Set} in resolve with FnvHash{Map,Set} got rid of the first (smaller) blocks of the diff. The big block at the bottom is still there. Will continue to randomly sed the compiler and report back ;)

jonas-schievink · 2016-08-22T21:23:04Z

The remaining issue is that (some?) predicates are encoded in non-deterministic order. Maybe we need to do this for all bounds? Not sure what would be the correct way to do that (or why these are nondeterministic in the first place).

michaelwoerister · 2016-08-22T21:26:21Z

Just for reference, there's this: #34805

jonas-schievink · 2016-08-22T21:32:04Z

@michaelwoerister Thanks! Too tired to think about it currently. Will have a look tomorrow :)

sanxiyn · 2016-08-23T11:27:43Z

#24473 is rustdoc subset of this.

Use `FnvHashMap` in more places * A step towards rust-lang#34902 (see my comments there) * More stable error messages in some places related to crate loading * Possible slight performance improvements since all `HashMap`s replaced had small keys where `FnvHashMap` should be faster (although I didn't measure) (likewise for `HashSet` -> `FnvHashSet`)

Steps towards reproducible builds cc #34902 Running `make dist` twice will result in a rustc tarball where only `librustc_back.so`, `librustc_llvm.so` and `librustc_trans.so` differ. Building `libstd` and `libcore` twice with the same compiler and flags produces identical artifacts. The third commit should close #24473

nikomatsakis · 2016-09-02T08:56:22Z

Potentially relevant, from IRC:

<eddyb> nmatsakis, infinity0: fwiw, I just found another hash table. not random, but may cause issues around incremental recompilation. see rustc_metadata::encoder::encode_reachable (the reachable NodeSet)

though I think that it's not a big issue unless the hashtable is truly random.

eddyb · 2016-09-02T08:57:45Z

Fix suggestion is changing that to be a BTreeSet<DefIndex>. Other things could also use btrees.

infinity0 · 2016-12-10T10:00:47Z

In case this helps to motivate anyone, we got a successful reproduction on i386 on Debian testing!

https://tests.reproducible-builds.org/debian/rb-pkg/testing/i386/rustc.html

This may or may not be "an accident", we'll have to see what future tests show. Also on this arch/platform we are fixing the build path (just to see how it does). For other arch/platforms we vary the build path, and haven't seen rustc reproduce there yet.

michaelwoerister · 2016-12-12T16:55:02Z

At least when building with debuginfo, the build path will show up in the resulting binaries as the DW_AT_comp_dir attribute of the DW_TAG_compile_unit.

codyps · 2016-12-12T17:42:13Z

@michaelwoerister

In gcc (and clang), the -fdebug-prefix-map argument allows one to avoid this. Is a similar option for rustc available?

michaelwoerister · 2016-12-12T18:08:32Z

@jmesmon No, we don't have an option like that. If you open an issue, I'd be happy to discuss it with the @rust-lang/tools team.

tarcieri · 2019-12-06T23:44:23Z

@jgalenson awesome! I'm working on a project for BFT consensus-driven proofs of reproducible builds, and that'd make an amazing test case: https://github.com/iqlusioninc/synchronicity

jgalenson · 2019-12-13T18:35:56Z

I was able to get reproducible builds of rustc with debuginfo by upgrading to a newer version of the compiler_builtins crate that contains rust-lang/compiler-builtins@ca423fe (so at least 0.1.20 should work, although I used 0.1.22). With that, I no longer needed my last remaining patch.

I also had to use a newer C++ compiler that supports the -ffile-prefix-map argument. gcc-8 does, but unfortunately using it did not work (it seems to embed source paths into files generated from assembly). However, since https://reviews.llvm.org/D49466 was merged, Clang supports that option now, so using a random nightly build of LLVM from a few days ago worked.

That's a convoluted setup, but @infinity0, can you reproduce it? Your diff seemed to contain some other differences mine did not, so some things might remain.

@tarcieri that seems pretty cool. Have you gotten it working yet?

infinity0 · 2019-12-29T22:08:50Z

@jgalenson I'm already including that compiler-builtins -ffile-prefix-map patch in Debian's rustc and it doesn't work, though we're using GCC not LLVM. What exactly is this issue with "it seems to embed source paths into files generated from assembly" you are talking about? I don't remember seeing that last time I was working on GCC reproducibility a few years ago.

tests.r-b.org results for rustc 1.39 are out the results are pretty similar to mine above - unreproducible including build paths relating to the syn crate.

infinity0 · 2019-12-29T22:10:12Z

I should mention that the differences include more that just C++ compiler output, which is what you seem to be suggesting is the last remaining thing, from your previous comment. My own testing and tests.r-b.org's testing indicates otherwise. At least the syn crate is embedding build paths.

jgalenson · 2019-12-30T18:40:01Z

I investigated the gcc bug I mentioned. If I take a trivial .c file and compile it with -g and -ffile-prefix-map, I don't see source paths in the output. But if I use the same command line for a trivial assembly file, I do see source paths in the output. Those go away if I change -ffile-prefix-map to -fdebug-prefix-map. I see this with gcc 8.3 but not with recent LLVM builds. Can you reproduce that?

This comes up because compiler-builtins includes a few .S files. This miscompilation then causes a number of other crates to be non-reproducible, including rustc_{driver,llvm} and cargo. Those all go away one I use Clang instead of gcc. I don't, however, see a difference in syn.

The tests.r-b.org link you posted no longer works for me because it seems to fail to build. But if you can send me the config.toml you're using, I'll see if I can reproduce it myself. In addition, you can try using my config and adding in debuginfo-level{,-std,-tools} = 2 and using a recent Clang.

comex · 2020-01-22T01:14:56Z

Filed https://gcc.gnu.org/bugzilla/show_bug.cgi?id=93371 for the GCC bug. As a short-term workaround, compiler-builtins could pass both -ffile-prefix-map and -fdebug-prefix-map...

infinity0 · 2020-08-05T21:06:39Z

I didn't get around to looking into the above issues in more detail, but I just noticed that tests.r-b.org for rustc has been showing "reproducible" for 1.44.1 including under varying build paths (unstable, experimental)!

1.45.0 is in-progress and we should have results in just a few more days.

sanxiyn · 2020-08-06T04:00:31Z

We should use individual issues tagged A-reproducibility going forward. Closing.

infinity0 · 2020-08-07T21:09:59Z

OK, well results for 1.45.0 say "unreproducible" again, and the auto-generated diff did not complete after 2 hours. Shall I open an issue for CI integration tests on this? If I understand correctly there are concerns about it taking up too much resources, or something.

sanxiyn · 2020-08-10T05:29:30Z

Yes, I think we should open a new issue for reproducible build CI. I also think it would be impractical for a while due to computational demand, but then I am not a member of infra team.

steveklabnik added the A-compiler label Jul 18, 2016

infinity0 mentioned this issue Aug 20, 2016

perf.rust-lang.org test for syntex-syntax messed up #35855

Closed

jonas-schievink mentioned this issue Aug 22, 2016

Use FnvHashMap in more places #35909

Closed

jonas-schievink mentioned this issue Aug 25, 2016

Steps towards reproducible builds #35984

Merged

codyps mentioned this issue Dec 12, 2016

Allow remapping source path prefixes in debug output #38322

Closed

jgalenson mentioned this issue Jan 28, 2020

Occasional non-reproducibility when building librls and libcargo #68608

Closed

jgalenson mentioned this issue Feb 21, 2020

Occasional non-reproducibility when building rustc #69352

Open

gendx mentioned this issue Mar 25, 2020

[Tracking] Reproducible builds tock/tock#1666

Open

12 tasks

gendx mentioned this issue Apr 20, 2020

Non-reproducible builds depending on the compiling OS #71361

Closed

frol mentioned this issue May 14, 2020

A docker image for building contracts near/near-sdk-rs#151

Closed

jgalenson mentioned this issue Jul 13, 2020

Non-reproducible builds rust-lang/rust-analyzer#5351

Open

sanxiyn closed this as completed Aug 6, 2020

infinity0 mentioned this issue Aug 10, 2020

CI for deterministic / reproducible builds #75362

Open

segiddins mentioned this issue Aug 25, 2020

Add swift.use_tmpdir_for_module_cache feature bazelbuild/rules_swift#456

Closed

zone117x mentioned this issue Feb 23, 2021

Create Signed Releases stacks-network/stacks-core#2435

Closed

maurolacy mentioned this issue Apr 30, 2021

Make a deterministic build system CosmWasm/wasmvm#136

Open

the8472 mentioned this issue Sep 16, 2021

Compiling large projects with over-heating CPUs causes weird crashes #88902

Closed

jedisct1 mentioned this issue Jul 12, 2022

Add Cargo.lock to repository (remove from gitignore) DNSCrypt/encrypted-dns-server#90

Closed

jonathanpdunne mentioned this issue Aug 30, 2022

Provide build instructions to verify bytecode polkadot-js/wasm#423

Closed

HCastano mentioned this issue Sep 7, 2022

Add information for reproducible builds to metadata use-ink/cargo-contract#525

Closed

5 tasks

flaub mentioned this issue Jan 25, 2023

Make ELF binaries build deterministically risc0/risc0#116

Closed

raboof mentioned this issue May 4, 2023

rustc: not reproducible NixOS/nixpkgs#229858

Closed

taiki-e mentioned this issue Jan 22, 2024

Trojan:Win32/Sabsik.TE.A!ml in target x86_64-pc-windows-msvc taiki-e/upload-rust-binary-action#63

Closed

jieyouxu mentioned this issue Aug 14, 2024

Tracking Issue for Reproducible Build bugs and challenges #129080

Open

81 tasks

grenade mentioned this issue Nov 5, 2024

provide a mechanism for public verification of release binaries QuilibriumNetwork/ceremonyclient#330

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Bit-for-bit deterministic / reproducible builds #34902

Bit-for-bit deterministic / reproducible builds #34902

infinity0 commented Jul 18, 2016

steveklabnik commented Jul 18, 2016

eefriedman commented Jul 18, 2016 •

edited

Loading

infinity0 commented Jul 18, 2016

infinity0 commented Aug 16, 2016

eddyb commented Aug 16, 2016

infinity0 commented Aug 20, 2016

infinity0 commented Aug 20, 2016

jonas-schievink commented Aug 22, 2016 •

edited

Loading

jonas-schievink commented Aug 22, 2016

jonas-schievink commented Aug 22, 2016

michaelwoerister commented Aug 22, 2016

jonas-schievink commented Aug 22, 2016

sanxiyn commented Aug 23, 2016

nikomatsakis commented Sep 2, 2016 •

edited by eddyb

Loading

eddyb commented Sep 2, 2016

infinity0 commented Dec 10, 2016

michaelwoerister commented Dec 12, 2016

codyps commented Dec 12, 2016

michaelwoerister commented Dec 12, 2016

tarcieri commented Dec 6, 2019

jgalenson commented Dec 13, 2019

infinity0 commented Dec 29, 2019 •

edited

Loading

infinity0 commented Dec 29, 2019

jgalenson commented Dec 30, 2019

comex commented Jan 22, 2020

infinity0 commented Aug 5, 2020

sanxiyn commented Aug 6, 2020

infinity0 commented Aug 7, 2020

sanxiyn commented Aug 10, 2020

Bit-for-bit deterministic / reproducible builds #34902

Bit-for-bit deterministic / reproducible builds #34902

Comments

infinity0 commented Jul 18, 2016

steveklabnik commented Jul 18, 2016

eefriedman commented Jul 18, 2016 • edited Loading

infinity0 commented Jul 18, 2016

infinity0 commented Aug 16, 2016

eddyb commented Aug 16, 2016

infinity0 commented Aug 20, 2016

infinity0 commented Aug 20, 2016

jonas-schievink commented Aug 22, 2016 • edited Loading

jonas-schievink commented Aug 22, 2016

jonas-schievink commented Aug 22, 2016

michaelwoerister commented Aug 22, 2016

jonas-schievink commented Aug 22, 2016

sanxiyn commented Aug 23, 2016

nikomatsakis commented Sep 2, 2016 • edited by eddyb Loading

eddyb commented Sep 2, 2016

infinity0 commented Dec 10, 2016

michaelwoerister commented Dec 12, 2016

codyps commented Dec 12, 2016

michaelwoerister commented Dec 12, 2016

tarcieri commented Dec 6, 2019

jgalenson commented Dec 13, 2019

infinity0 commented Dec 29, 2019 • edited Loading

infinity0 commented Dec 29, 2019

jgalenson commented Dec 30, 2019

comex commented Jan 22, 2020

infinity0 commented Aug 5, 2020

sanxiyn commented Aug 6, 2020

infinity0 commented Aug 7, 2020

sanxiyn commented Aug 10, 2020

eefriedman commented Jul 18, 2016 •

edited

Loading

jonas-schievink commented Aug 22, 2016 •

edited

Loading

nikomatsakis commented Sep 2, 2016 •

edited by eddyb

Loading

infinity0 commented Dec 29, 2019 •

edited

Loading