Spurious rebuilds due to filesystem rounding file modification timestamps

[kornelski]

Problem

The issue #2874 was not completely fixed by #2880. I'm still seeing the exact same problem in latest Cargo:

Dirty libc v0.2.142: the dependency build_script_build was rebuilt (1682215868.647155753s, 647155753ns after last build at 1682215868.000000000s)
Dirty openssl-sys v0.9.86: the dependency build_script_main was rebuilt (1682214777.891768399s, 891768399ns after last build at 1682214777.000000000s)

Steps

I'm using Docker and mounting target/ dir as a Docker volume (in order to cache it). I think the docker volume rounds the timestamps to 1-second precision. However, this doesn't always cause spurious rebuilds, only maybe 25% of the time, and it's always build_script_build or build_script_main, not other files.

Possible Solution(s)

I assume Cargo wouldn't want to introduce imprecision in comparison, such as always comparing the integer number of seconds or allowing 1-second difference between timestamps. Therefore, I propose detecting this problem explicitly: there's only one in a billion chance that the nanosecond part will be exactly 0. So if either of the timestamps being compared has exactly 0 nanosecond part, compare only the integer number of seconds. Otherwise compare in full precision. I can make a PR with that.

Notes

No response

Version

cargo 1.71.0-nightly (d0a4cbcee 2023-04-16)
release: 1.71.0-nightly
commit-hash: d0a4cbcee614fdb7ba66e860e603a00a644d71f8
commit-date: 2023-04-16
host: aarch64-apple-darwin
libgit2: 1.6.3 (sys:0.17.0 vendored)
libcurl: 7.88.1 (sys:0.4.61+curl-8.0.1 system ssl:(SecureTransport) LibreSSL/3.3.6)
ssl: OpenSSL 1.1.1t  7 Feb 2023
os: Mac OS 13.4.0 [64-bit]

[weihanglo]

The proposed solution sounds hacky initially but actually makes some sense.

The precision of mtime timestamp varies in filesystem implementations I guess? I think we need research about what the precision is of each major file system handling timestamp. I know macbook once use 1-second accuracy HFS+ but not APFS is good at nanosecond level. I also wonder if there is any other factor affecting the round-up, perhaps different libc implementations? What if people in a less-accurate filesystem with powerful computing resources, leading to a lot of rebuild skipped.

[Emilgardis]

I'm also seeing this in docker for windows using wsl2

This has actually been something users using cross-rs (which uses a container engine) have dealt with and haven't located the exact reason until now

    Dirty serde_json v1.0.108: the dependency build_script_build was rebuilt (1701461274.529880500s, 529880500ns after last build at 1701461274.000000000s)
Compiling serde_json v1.0.108
    Dirty anyhow v1.0.75: the dependency build_script_build was rebuilt (1701461276.248774900s, 248774900ns after last build at 1701461276.000000000s)

I'm curious if the project would be interested in making the suggested workaround, or if maybe we could enable a way to tell cargo to be less exact in mtime checks (like a cargo flag, available through config)

[kornelski]

@weihanglo I'm unable to perform the research of filesystems and libc implementations that you suggested.

I've been using a fork of Cargo with this change, and it worked fine for me.

If someone else wants to develop a less hacky fix or investigate all platforms, please do.

[weihanglo]

@kornelski, could you share the patch so we can weigh on how we can integrate that, or provide a config for low precision filesystem as a escape hatch. Using fingerprint instead of mtime is also under the consideration #6529. Since we have experimental SQLite integration, #6529 might be more possible to move forward.

[yuyang-ok]

I think I have this issue too.
Sometimes cargo tries to rebuild from libc .

[weihanglo]

Sometimes cargo tries to rebuild from libc .

Hi, @yuyang-ok. could you share what filesystem/OS/environment you are on? That would help evaluate possible solutions.

[bobrik]

I'm seeing this as well with z3:

       Dirty z3-sys v0.8.1: the file `target/release/build/z3-sys-0afb7a39553d3e75/out/param_kind.rs` has changed (1703474932.081130554s, 81130554ns after last build at 1703474932.000000000s)

I'm building in a Linux VM running on macOS host. The filesystem with the project is a virtfs passed to QEMU as:

-virtfs local,path=/Users/ivan/projects,security_model=none,mount_tag=projects

Inside of the VM:

ivan@vm:~$ mount | grep projects
projects on /mnt/data/homes/ivan/projects type 9p (rw,relatime,dirsync,access=client,trans=virtio,_netdev)

I'm not sure where the rounded timestamp comes from. Both VM's fs and remote fs produce ns precision timestamps:

$ stat target/release/build/z3-sys-0afb7a39553d3e75/out/param_kind.rs ~/.rustup
  File: target/release/build/z3-sys-0afb7a39553d3e75/out/param_kind.rs
  Size: 295       	Blocks: 8          IO Block: 131072 regular file
Device: 0,42	Inode: 213528511   Links: 1
Access: (0644/-rw-r--r--)  Uid: (  501/    ivan)   Gid: (   20/ dialout)
Access: 2023-12-25 03:28:52.416478161 +0000
Modify: 2023-12-25 03:28:52.081130554 +0000
Change: 2023-12-25 03:28:52.081130554 +0000
 Birth: -
  File: /home/ivan/.rustup
  Size: 94        	Blocks: 0          IO Block: 4096   directory
Device: 259,1	Inode: 103875877   Links: 6
Access: (0755/drwxr-xr-x)  Uid: (  501/    ivan)   Gid: (   20/ dialout)
Access: 2023-12-25 03:38:09.415500979 +0000
Modify: 2023-07-23 20:06:26.016396583 +0000
Change: 2023-07-23 20:06:26.016396583 +0000
 Birth: 2023-07-23 20:06:12.887530972 +0000

Running cargo build on the host only builds once.

Running it in a VM almost always produces a rebuild, but not 100% of the time, as it converges eventually:

ivan@vm:~/projects/advent-of-code-2023/problem-42$ rm -rf target

ivan@vm:~/projects/advent-of-code-2023/problem-42$ cargo build --release
   Compiling glob v0.3.1
   Compiling proc-macro2 v1.0.71
   Compiling unicode-ident v1.0.12
   Compiling libc v0.2.151
   Compiling cfg-if v1.0.0
   Compiling memchr v2.6.4
   Compiling regex-syntax v0.8.2
   Compiling minimal-lexical v0.2.1
   Compiling libloading v0.7.4
   Compiling nom v7.1.3
   Compiling bindgen v0.66.1
   Compiling clang-sys v1.6.1
   Compiling lazy_static v1.4.0
   Compiling rustc-hash v1.1.0
   Compiling lazycell v1.3.0
   Compiling bitflags v2.4.1
   Compiling shlex v1.2.0
   Compiling peeking_take_while v0.1.2
   Compiling regex-automata v0.4.3
   Compiling radium v0.7.0
   Compiling quote v1.0.33
   Compiling cexpr v0.6.0
   Compiling syn v2.0.42
   Compiling regex v1.10.2
   Compiling tap v1.0.1
   Compiling wyz v0.5.1
   Compiling log v0.4.20
   Compiling funty v2.0.0
   Compiling fnv v1.0.7
   Compiling bitvec v1.0.1
   Compiling z3-sys v0.8.1
   Compiling z3 v0.12.1
   Compiling problem-42 v0.1.0 (/mnt/data/homes/ivan/projects/advent-of-code-2023/problem-42)
    Finished release [optimized + debuginfo] target(s) in 35.51s

ivan@vm:~/projects/advent-of-code-2023/problem-42$ cargo build --release
   Compiling proc-macro2 v1.0.71
   Compiling clang-sys v1.6.1
   Compiling bindgen v0.66.1
   Compiling quote v1.0.33
   Compiling syn v2.0.42
   Compiling z3-sys v0.8.1
   Compiling z3 v0.12.1
   Compiling problem-42 v0.1.0 (/mnt/data/homes/ivan/projects/advent-of-code-2023/problem-42)
    Finished release [optimized + debuginfo] target(s) in 24.45s

ivan@vm:~/projects/advent-of-code-2023/problem-42$ cargo build --release
   Compiling clang-sys v1.6.1
   Compiling bindgen v0.66.1
   Compiling z3-sys v0.8.1
   Compiling z3 v0.12.1
   Compiling problem-42 v0.1.0 (/mnt/data/homes/ivan/projects/advent-of-code-2023/problem-42)
    Finished release [optimized + debuginfo] target(s) in 21.92s

ivan@vm:~/projects/advent-of-code-2023/problem-42$ cargo build --release
   Compiling z3-sys v0.8.1
   Compiling z3 v0.12.1
   Compiling problem-42 v0.1.0 (/mnt/data/homes/ivan/projects/advent-of-code-2023/problem-42)
    Finished release [optimized + debuginfo] target(s) in 9.06s

ivan@vm:~/projects/advent-of-code-2023/problem-42$ cargo build --release
   Compiling z3-sys v0.8.1
   Compiling z3 v0.12.1
   Compiling problem-42 v0.1.0 (/mnt/data/homes/ivan/projects/advent-of-code-2023/problem-42)
    Finished release [optimized + debuginfo] target(s) in 6.17s

ivan@vm:~/projects/advent-of-code-2023/problem-42$ cargo build --release
    Finished release [optimized + debuginfo] target(s) in 0.56s

ivan@vm:~/projects/advent-of-code-2023/problem-42$ cargo build --release
    Finished release [optimized + debuginfo] target(s) in 0.71s

[gilescope]

Pragmatically this would be the best thing to do for the environment. I hate to think how much CO2 has been produced due to this 'feature' over the years of our inaction.