Maturin develop always rebuilds every dependency

[velvia]

Please provide the following information:

• Your python version (python -V): 3.7
• Your pip version (pip -V): 19.2.3
  • If this isn't the latest pip version, have you checked using the latest pip version?
• The version of the bindings you're using, if any (e.g. pyo3, rust-cpython or cffi):
• Does cargo build work? yes
• If on windows, have you checked that you aren't accidentally using unix path (those with the forward slash /)?

Please list the exact steps required to reproduce your error with all command output and if possible with a repository:

My crate has 280 dependencies. When I run Maturin develop it always rebuilds every single one, even when I don't change the dependencies.... it's almost like it is running clean every time. Is there a way to not rebuild all the deps?

thanks

[konstin]

Could you check the output of maturin develop --cargo-extra-args="-v"? In general, the decision whether to rebuild or to use the cached build belongs to cargo, but it could be that maturin influences the environment variables so that cargo rebuilds. This means it is sometimes unavoidable that a maturin develop rebuilds most of the crates, but a second consecutive maturin develop shouldn't.

[velvia]

@konstin I don't see anything unusual in the output:

🔗 Found pyo3 bindings
🐍 Found CPython 3.7m at python
   Compiling libc v0.2.86
   Compiling proc-macro2 v1.0.24
   Compiling unicode-xid v0.2.1
   Compiling syn v1.0.60
   Compiling cfg-if v1.0.0
   Compiling autocfg v1.0.1
   Compiling serde_derive v1.0.123
   Compiling serde v1.0.123
   Compiling log v0.4.14
   Compiling memchr v2.3.4
   Compiling proc-macro-hack v0.5.19
   Compiling pin-project-lite v0.2.4
     Running `rustc --crate-name build_script_build /Users/evan/.cargo/registry/src/github.com-1ecc6299db9ec823/libc-0.2.86/build.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type bin --emit=dep-info,link -C embed-bitcode=no -C debuginfo=2 --cfg 'feature="align"' --cfg 'feature="default"' --cfg 'feature="std"' -C metadata=363869d85faea57d -C extra-filename=-363869d85faea57d --out-dir /Users/evan/src/ul/ul/libs/multiparquet/target/debug/build/libc-363869d85faea57d -L dependency=/Users/evan/src/ul/ul/libs/multiparquet/target/debug/deps --cap-lints allow`
     Running `rustc --crate-name build_script_build --edition=2018 /Users/evan/.cargo/registry/src/github.com-1ecc6299db9ec823/syn-1.0.60/build.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type bin --emit=dep-info,link -C embed-bitcode=no -C debuginfo=2 --cfg 'feature="clone-impls"' --cfg 'feature="default"' --cfg 'feature="derive"' --cfg 'feature="extra-traits"' --cfg 'feature="full"' --cfg 'feature="parsing"' --cfg 'feature="printing"' --cfg 'feature="proc-macro"' --cfg 'feature="quote"' --cfg 'feature="visit"' --cfg 'feature="visit-mut"' -C metadata=c848fa94ad01805d -C extra-filename=-c848fa94ad01805d --out-dir /Users/evan/src/ul/ul/libs/multiparquet/target/debug/build/syn-c848fa94ad01805d -L dependency=/Users/evan/src/ul/ul/libs/multiparquet/target/debug/deps --cap-lints allow`
     Running `rustc --crate-name build_script_build --edition=2018 /Users/evan/.cargo/registry/src/github.com-1ecc6299db9ec823/proc-macro2-1.0.24/build.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type bin --emit=dep-info,link -C embed-bitcode=no -C debuginfo=2 --cfg 'feature="default"' --cfg 'feature="proc-macro"' -C metadata=0601c921986e3de8 -C extra-filename=-0601c921986e3de8 --out-dir /Users/evan/src/ul/ul/libs/multiparquet/target/debug/build/proc-macro2-0601c921986e3de8 -L dependency=/Users/evan/src/ul/ul/libs/multiparquet/target/debug/deps --cap-lints allow`
     Running `rustc --crate-name build_script_build /Users/evan/.cargo/registry/src/github.com-1ecc6299db9ec823/serde_derive-1.0.123/build.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type bin --emit=dep-info,link -C embed-bitcode=no -C debuginfo=2 --cfg 'feature="default"' -C metadata=83408131f60a7502 -C extra-filename=-83408131f60a7502 --out-dir /Users/evan/src/ul/ul/libs/multiparquet/target/debug/build/serde_derive-83408131f60a7502 -L dependency=/Users/evan/src/ul/ul/libs/multiparquet/target/debug/deps --cap-lints allow`
     Running `rustc --crate-name build_script_build /Users/evan/.cargo/registry/src/github.com-1ecc6299db9ec823/serde-1.0.123/build.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type bin --emit=dep-info,link -C embed-bitcode=no -C debuginfo=2 --cfg 'feature="default"' --cfg 'feature="derive"' --cfg 'feature="rc"' --cfg 'feature="serde_derive"' --cfg 'feature="std"' -C metadata=ff9b00ffec515a48 -C extra-filename=-ff9b00ffec515a48 --out-dir /Users/evan/src/ul/ul/libs/multiparquet/target/debug/build/serde-ff9b00ffec515a48 -L dependency=/Users/evan/src/ul/ul/libs/multiparquet/target/debug/deps --cap-lints allow`
     Running `rustc --crate-name build_script_build /Users/evan/.cargo/registry/src/github.com-1ecc6299db9ec823/log-0.4.14/build.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type bin --emit=dep-info,link -C embed-bitcode=no -C debuginfo=2 -C metadata=21152c524a92b587 -C extra-filename=-21152c524a92b587 --out-dir /Users/evan/src/ul/ul/libs/multiparquet/target/debug/build/log-21152c524a92b587 -L dependency=/Users/evan/src/ul/ul/libs/multiparquet/target/debug/deps --cap-lints allow`
     Running `rustc --crate-name build_script_build /Users/evan/.cargo/registry/src/github.com-1ecc6299db9ec823/memchr-2.3.4/build.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type bin --emit=dep-info,link -C embed-bitcode=no -C debuginfo=2 --cfg 'feature="default"' --cfg 'feature="std"' --cfg 'feature="use_std"' -C metadata=6bd68f452d94d1bf -C extra-filename=-6bd68f452d94d1bf --out-dir /Users/evan/src/ul/ul/libs/multiparquet/target/debug/build/memchr-6bd68f452d94d1bf -L dependency=/Users/evan/src/ul/ul/libs/multiparquet/target/debug/deps --cap-lints allow`
     Running `rustc --crate-name build_script_build --edition=2018 /Users/evan/.cargo/registry/src/github.com-1ecc6299db9ec823/proc-macro-hack-0.5.19/build.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type bin --emit=dep-info,link -C embed-bitcode=no -C debuginfo=2 -C metadata=0439868cee90cad0 -C extra-filename=-0439868cee90cad0 --out-dir /Users/evan/src/ul/ul/libs/multiparquet/target/debug/build/proc-macro-hack-0439868cee90cad0 -L dependency=/Users/evan/src/ul/ul/libs/multiparquet/target/debug/deps --cap-lints allow`
     Running `rustc --crate-name autocfg /Users/evan/.cargo/registry/src/github.com-1ecc6299db9ec823/autocfg-1.0.1/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type lib --emit=dep-info,metadata,link -C embed-bitcode=no -C debuginfo=2 -C metadata=fa83e66836aadcae -C extra-filename=-fa83e66836aadcae --out-dir /Users/evan/src/ul/ul/libs/multiparquet/target/debug/deps -L dependency=/Users/evan/src/ul/ul/libs/multiparquet/target/debug/deps --cap-lints allow`
     Running `rustc --crate-name cfg_if --edition=2018 /Users/evan/.cargo/registry/src/github.com-1ecc6299db9ec823/cfg-if-1.0.0/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C embed-bitcode=no -C debuginfo=2 -C metadata=7313dec648596e9f -C extra-filename=-7313dec648596e9f --out-dir /Users/evan/src/ul/ul/libs/multiparquet/target/debug/deps -L dependency=/Users/evan/src/ul/ul/libs/multiparquet/target/debug/deps --cap-lints allow`
     Running `rustc --crate-name unicode_xid /Users/evan/.cargo/registry/src/github.com-1ecc6299db9ec823/unicode-xid-0.2.1/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C embed-bitcode=no -C debuginfo=2 --cfg 'feature="default"' -C metadata=c67a224201df1e1d -C extra-filename=-c67a224201df1e1d --out-dir /Users/evan/src/ul/ul/libs/multiparquet/target/debug/deps -L dependency=/Users/evan/src/ul/ul/libs/multiparquet/target/debug/deps --cap-lints allow`
     Running `rustc --crate-name pin_project_lite --edition=2018 /Users/evan/.cargo/registry/src/github.com-1ecc6299db9ec823/pin-project-lite-0.2.4/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C embed-bitcode=no -C debuginfo=2 -C metadata=c23b92df33239e01 -C extra-filename=-c23b92df33239e01 --out-dir /Users/evan/src/ul/ul/libs/multiparquet/target/debug/deps -L dependency=/Users/evan/src/ul/ul/libs/multiparquet/target/debug/deps --cap-lints allow`
   Compiling version_check v0.9.2

I wonder if it has to do with that I put the recommended OSX fixes in .cargo/config.toml:

[target.x86_64-apple-darwin]
rustflags = [
  "-C", "link-arg=-undefined",
  "-C", "link-arg=dynamic_lookup",
]

[velvia]

The last few times I have not been able to reproduce this, and it seems Maturin develop is now only building changed crates, so I'm closing.

[kaimast]

I see this issue on my setup.

I also noticed that maturin seems to ignore the flags .cargo/config.toml as velvia suggested. In particular I am setting rustc to use llvm linker.

[target.'cfg(target_os = "linux")']
rustflags = [
  "-C", "linker=clang",
  "-Z", "macro-backtrace",
  "-C", "link-arg=-Wl,-E",
]

However, even without these flags I see a rebuild of the python bindings whenever I have compiled some other crate in the same workspace...

[davidhewitt]

This might be relevant: PyO3/pyo3#1708

[konstin]

So am I reading this correctly that this is currently waiting on rust-lang/rustup#2849 (comment)?

[kaimast]

This might be relevant: PyO3/pyo3#1708

I'm not sure.

I used to build my python bindings without maturin and never encountered this issue from what I remembered.

[konstin]

The problem (and also cargo rebuilding way to much in other circumstances) always confuses me, because maturin is just passing some arguments like you normally and also i don't understand how to get cargo to tell me why it's recompiling -vvv didn't help

[kaimast]

The only "fix" that works for me is to set CARGO_TARGET_DIR to something other than target before invoking maturin.

[nipunn1313]

I was able to repro this and narrow it down
I believe it to be taiki-e/cargo-config2#17 - where an interaction between ~/.cargo/config.toml and $myrepo/.cargo/config.toml - is merging in different order between cargo and maturin. I suspect this is why I'm seeing recompiles.

Ofc recompiles are a complex issue that appears as a symptom that can have many causes, so your mileage may vary.

[kaimast]

I now only see pyo3 being rebuild. But it happens consistently whenever I run maturin develop after cargo clippy.