Add dep-info generation #3557

raphlinus · 2017-01-18T00:24:10Z

Work in progress: add a --dep-info flag to cargo build (and also
rustc) that outputs dependency information in a form compatible with
make and ninja, to a specified file. This will help in integrating
into other build systems.

rust-highfive · 2017-01-18T00:24:22Z

Thanks for the pull request, and welcome! The Rust team is excited to review your changes, and you should hear from @brson (or someone else) soon.

If any changes to this PR are deemed necessary, please add them as extra commits. This ensures that the reviewer can see what has changed since they last reviewed the code. Due to the way GitHub handles out-of-date commits, this should also make it reasonably obvious what issues have or haven't been addressed. Large or tricky changes may require several passes of review and changes.

Please see the contribution instructions for more information.

raphlinus · 2017-01-18T00:27:26Z

This is very rough, but I wanted to checkpoint my work in progress. Comments are welcome (though I'm out most of the rest of this week). See https://internals.rust-lang.org/t/proposal-make-cargo-output-dep-info/4603/ for background discussion. I also want to create a tracking issue, but haven't done that yet.

One limitation is that it doesn't follow path dependencies for libraries. I agree this would be a good thing.

Another discussion question: most paths will be absolute, but it might be more desirable to make them relative to some base directory. The code has a mechanism for this, but I haven't plumbed it through from the command line, out of concern for proliferating options.

Other issues are identified by TODO.

alexcrichton

I'm curious, what do you think about avoiding an explicit argument for this and doing it unconditionally? We could always emit foo.d files next to all targets perhaps?

alexcrichton · 2017-01-18T23:21:25Z

src/cargo/ops/cargo_output_depinfo.rs

+    } else {
+        &profiles.dev
+    };
+    let mut context = Context::new(&ws, &resolve, &packages, &config, build_config, &profiles)?;


Perhaps this could be moved into cargo_rustc to avoid creating multiple contexts? It's not expensive or anything but it tends to be subtle enough to introduce lots of bugs, so deduplicating the logic would be best.

Yes, probably moving into cargo_rustc would be better. By way of context, this started out as a plugin using cargo as a lib, so it makes sense to change things to be more integrated.

alexcrichton · 2017-01-18T23:22:10Z

src/cargo/ops/cargo_output_depinfo.rs

+                    profile: &profile,
+                    kind: Kind::Target,
+                };
+                let filename = ["dep-", kind, "-", &context.file_stem(&unit)].concat();


Here I think we may want to tweak how fingerprints are managed slightly. Could we maintain a map of fingerprints for units (e.g. active paths) on the Context and then we scrape that list afterwards? That way we don't have to re-parse anything or re-guess filenames and such.

Yes, agreed.

raphlinus · 2017-01-23T16:45:55Z

Yes, I'm totally fine with always writing the depfile. It occurs to me it could potentially be useful for speeding up a null build, assuming we get it precise enough.

alexcrichton · 2017-01-23T16:57:54Z

Ok! In that case we can probably drop the command line argument for now and start out with just target/debug/foo.d and such perhaps?

raphlinus · 2017-01-25T01:38:27Z

Ok, I uploaded a new version, for discussion. I believe the functionality of this is about what we want (though happy to discuss), but no doubt the implementation can be cleaned up, as it still shows signs of being an external tool scraping cargo's internals.

Happily, this version does traverse path dependencies. Build script inputs are still missing, but I think I understand better how to add them.

Also, I see the merge problem; TargetKind::Example has been split. I'll rebase.

Make cargo output a ".d" file containing dependency info (in a format that make and ninja can consume) for each artifact it produces. This will help in integrating into other build systems.

raphlinus · 2017-01-25T01:58:48Z

Rebased. I ended up doing push -f, sorry about that (I'm used to very different git workflows).

alexcrichton · 2017-01-25T17:23:55Z

src/cargo/ops/cargo_rustc/output_depinfo.rs

+    let mut outfile = File::create(output_path)?;
+    write!(outfile, "{}:", target_fn)?;
+    for dep in &deps {
+        write!(outfile, " {}", render_filename(dep, basedir)?)?;


I think here we'll want to escape spaces in paths, right? (that's done elsewhere I believe)

Oh right we parse escapes, we don't personally escape, nvmd

Yeah, I do need to re-escape spaces. Thanks for the heads-up!

alexcrichton · 2017-01-25T17:24:29Z

src/cargo/ops/cargo_rustc/output_depinfo.rs

+pub fn output_depinfo(context: &mut Context, unit: &Unit) -> CargoResult<()> {
+    let mut deps = HashSet::new();
+    add_deps_for_unit(&mut deps, context, unit)?;
+    for dep_unit in &context.dep_targets(unit)? {


I think this'll want to be recursive to pick up path dependencies of path dependencies

alexcrichton · 2017-01-25T17:25:02Z

src/cargo/ops/cargo_rustc/output_depinfo.rs

+    }
+}
+
+// TODO: probably better to use Context::target_filenames for this


Yeah I think we'll just want to emit one .d file with the same contents for each filename returned from target_filenames

alexcrichton · 2017-01-25T17:25:38Z

src/cargo/ops/cargo_rustc/output_depinfo.rs

+    relpath.to_str().ok_or(internal("path not utf-8")).map(ToOwned::to_owned)
+}
+
+fn read_dep_file<P: AsRef<Path>>(path: P) -> CargoResult<DepFile> {


Could this implementation be shared with the fingerprint module? I think the fingerprint module could store information in a side table in the context perhaps to avoid rereading the filesystem here.

I shared the implementation, but have not yet implemented a change to stash the info so we can avoid rereading the file system. I don't think that's hard, but for reference here are some measurements from a null build of cargo: a null build takes about 215ms. The time between reading the input fingerprints and writing the output .d is about 400us, repeated twice. Also, for perspective, this only reads the deps for path dependencies. So I think the impact on performance is pretty minimal. I'll happily stash if you wanna chase that down, though.

(However, in looking at the strace, I see I'm losing several ms by not buffering the file writing - fixing that now!)

Use Context::target_filenames rather than trying to redo that. Traverse path dependencies recursively, so we get transitive deps. Use existing fingerprint parsing.

Use BufWriter when generating .d files with dep-info, to avoid excessive numbers of write syscalls.

Also pick up the rerun-if-changed dependencies from build scripts.

raphlinus · 2017-01-26T21:48:24Z

Latest version has a bit of performance tuning, and also picks up dependencies generated by build scripts. My personal feeling is that this is ready now, but I'd very much welcome any feedback.

alexcrichton · 2017-01-26T22:21:19Z

src/cargo/ops/cargo_rustc/output_depinfo.rs

+    if visited.contains(unit) {
+        return Ok(());
+    }
+    visited.insert(unit.clone());


You can combine these together as:

if !visited.insert(unit.clone()) { return Ok(()) }

I believe clone is just copying something like 4 pointers, so it shouldn't cost anything.

alexcrichton · 2017-01-26T22:22:15Z

src/cargo/ops/cargo_rustc/output_depinfo.rs

+
+    // Add dependencies from rustc dep-info output (stored in fingerprint directory)
+    let dep_info_loc = fingerprint::dep_info_loc(context, unit);
+    if let Some(paths) = fingerprint::parse_dep_info(&dep_info_loc)? {


I think here we'd want to return an error in the None case, right? If we fail to read dep info we expect to exists, that seems fatal for this operation at least.

So if I error on None here, the following tests fail:

cargo_platform_specific_dependency freshness_ignores_excluded

I haven't dug into exactly why; might be worth investigating.

Odd! I'd be fine ignoring the error case here, but I think that may want to translate to not writing out a dependency file at all or perhaps a warning, it seems semi-serious to not list deps by accident.

alexcrichton · 2017-01-26T22:23:44Z

src/cargo/ops/cargo_rustc/output_depinfo.rs

+    let basedir = None; // TODO
+    for (_filename, link_dst, _linkable) in context.target_filenames(unit)? {
+        if let Some(link_dst) = link_dst {
+            let output_path = link_dst.with_extension("d");


I think we may want to unconditionally append .d here to handle case like libfoo.so.d, right?

I had that before, but I changed it to follow https://github.com/rust-lang/cargo/blob/master/src/cargo/ops/cargo_rustc/mod.rs#L281 which does it this way. I'm ok either way but feel it should be consistent.

Eh that sounds reasonable to me! If the compiler's doing it we may as well mirror that.

alexcrichton · 2017-01-26T22:26:12Z

Looks great! I think that rereading from the filesystem is very well factored here, so let's keep as-is.

Could you also be sure to add a test to make sure these files are generated? Other than that I'm all for this!

raphlinus · 2017-01-27T16:23:08Z

Ok, here's where I am regarding the strict checks that dependencies are present. I tracked down the failure for cargo_platform_specific_dependency. First, the unit that has run_custom_build set doesn't have an entry in the .fingerprint dir; it represents running the script rather than building it. I think skipping the parsing of the fingerprint makes sense for run_custom_build or doc in the profile.

That didn't quite fix things. The unit that runs the build script depends on another unit to build it. However, that wasn't getting picked up, as Context::dep_run_custom_build has a short-circuit to skip the dependency if it's already been satisfied. I got much better results when I added a full_deps flag to dep_targets that forces into the traversal.

I'm still getting some test failures when I have strict checking, apparently in the interaction between build script and features. What I'd like to do is clean up my current patch, turn off the strict checking, and file an issue regarding the failures when strict checking is on (and that the feature should be considered experimental until then). I think it's going to take some fairly deep digging to get all of them, and I believe it now works well enough for all our needs for a while.

Sound good?

alexcrichton · 2017-01-27T17:30:33Z

Thanks for tracking that down! I think though that instead of adding full_deps as a flag we'll want to just precompute the list before compilation starts. I think a flag like full_deps isn't quite right because it'd take into account dependencies that overrides cause Cargo to not lock at. In that sense precomputing the graph should have perfect knowledge of what's already available and what isn't, so we can just ferry that over to the dep info generation.

I'm ok not having strict checking as well, yeah, but what do you think is the best way to represent that in the dep-info file? If we missed a dependency b/c we couldn't find the files associated with it, should we just leave out those files? Delete the dep-info file?

raphlinus · 2017-01-27T18:39:12Z

I agree grabbing the graph is a good idea, but it doesn't seem simple to me. We'd need to grab the graph before running any of the compilation steps, then gather up the depfiles after. Where should the graph be stored? What type would it have?

I'm considering deferring the build script stuff for now, if it's going to be really complicated. In the targets we're building now, we're not using build scripts in the primary targets or their path deps. But I'm definitely willing to put in some more work if there's a way to get it right that's not too hard.

Deleting the dep-info file seems most correct; in my quick test with ninja, it causes the build to always be re-run. I'll do that next.

Tests for existence for dep-info output in simple compilation cases. Deletes the dep-info file if it fails (for example, if it can't find one of the dep-info inputs).

raphlinus · 2017-01-27T19:45:29Z

I believe the latest version is in a mergeable state. It doesn't try to do anything fancy in cases where it can't find the dep-info output (like the build scripts), just deletes the .d file. This works for our cases and shouldn't cause incorrect behavior. If we merge this, I'll definitely document the further work in an issue.

alexcrichton · 2017-01-27T21:37:15Z

@bors: r+

Looks great to me! I'll try to poke around the failure cases as well soon.

bors · 2017-01-27T21:37:16Z

📌 Commit 5cb6995 has been approved by alexcrichton

bors · 2017-01-27T22:55:09Z

⌛ Testing commit 5cb6995 with merge 609371f...

Add dep-info generation Work in progress: add a --dep-info flag to cargo build (and also rustc) that outputs dependency information in a form compatible with make and ninja, to a specified file. This will help in integrating into other build systems.

bors · 2017-01-28T00:34:56Z

☀️ Test successful - status-appveyor, status-travis
Approved by: alexcrichton
Pushing 609371f to master...

Version 1.16.0 (2017-03-16) =========================== Language -------- * Lifetimes in statics and consts default to `'static`. [RFC 1623] * [The compiler's `dead_code` lint now accounts for type aliases][38051]. * [Uninhabitable enums (those without any variants) no longer permit wildcard match patterns][38069] * [Clean up semantics of `self` in an import list][38313] * [`Self` may appear in `impl` headers][38920] * [`Self` may appear in struct expressions][39282] Compiler -------- * [`rustc` now supports `--emit=metadata`, which causes rustc to emit a `.rmeta` file containing only crate metadata][38571]. This can be used by tools like the Rust Language Service to perform metadata-only builds. * [Levenshtein based typo suggestions now work in most places, while previously they worked only for fields and sometimes for local variables][38927]. Together with the overhaul of "no resolution"/"unexpected resolution" errors (#[38154]) they result in large and systematic improvement in resolution diagnostics. * [Fix `transmute::<T, U>` where `T` requires a bigger alignment than `U`][38670] * [rustc: use -Xlinker when specifying an rpath with ',' in it][38798] * [`rustc` no longer attempts to provide "consider using an explicit lifetime" suggestions][37057]. They were inaccurate. Stabilized APIs --------------- * [`VecDeque::truncate`] * [`VecDeque::resize`] * [`String::insert_str`] * [`Duration::checked_add`] * [`Duration::checked_sub`] * [`Duration::checked_div`] * [`Duration::checked_mul`] * [`str::replacen`] * [`str::repeat`] * [`SocketAddr::is_ipv4`] * [`SocketAddr::is_ipv6`] * [`IpAddr::is_ipv4`] * [`IpAddr::is_ipv6`] * [`Vec::dedup_by`] * [`Vec::dedup_by_key`] * [`Result::unwrap_or_default`] * [`<*const T>::wrapping_offset`] * [`<*mut T>::wrapping_offset`] * `CommandExt::creation_flags` * [`File::set_permissions`] * [`String::split_off`] Libraries --------- * [`[T]::binary_search` and `[T]::binary_search_by_key` now take their argument by `Borrow` parameter][37761] * [All public types in std implement `Debug`][38006] * [`IpAddr` implements `From<Ipv4Addr>` and `From<Ipv6Addr>`][38327] * [`Ipv6Addr` implements `From<[u16; 8]>`][38131] * [Ctrl-Z returns from `Stdin.read()` when reading from the console on Windows][38274] * [std: Fix partial writes in `LineWriter`][38062] * [std: Clamp max read/write sizes on Unix][38062] * [Use more specific panic message for `&str` slicing errors][38066] * [`TcpListener::set_only_v6` is deprecated][38304]. This functionality cannot be achieved in std currently. * [`writeln!`, like `println!`, now accepts a form with no string or formatting arguments, to just print a newline][38469] * [Implement `iter::Sum` and `iter::Product` for `Result`][38580] * [Reduce the size of static data in `std_unicode::tables`][38781] * [`char::EscapeDebug`, `EscapeDefault`, `EscapeUnicode`, `CaseMappingIter`, `ToLowercase`, `ToUppercase`, implement `Display`][38909] * [`Duration` implements `Sum`][38712] * [`String` implements `ToSocketAddrs`][39048] Cargo ----- * [The `cargo check` command does a type check of a project without building it][cargo/3296] * [crates.io will display CI badges from Travis and AppVeyor, if specified in Cargo.toml][cargo/3546] * [crates.io will display categories listed in Cargo.toml][cargo/3301] * [Compilation profiles accept integer values for `debug`, in addition to `true` and `false`. These are passed to `rustc` as the value to `-C debuginfo`][cargo/3534] * [Implement `cargo --version --verbose`][cargo/3604] * [All builds now output 'dep-info' build dependencies compatible with make and ninja][cargo/3557] * [Build all workspace members with `build --all`][cargo/3511] * [Document all workspace members with `doc --all`][cargo/3515] * [Path deps outside workspace are not members][cargo/3443] Misc ---- * [`rustdoc` has a `--sysroot` argument that, like `rustc`, specifies the path to the Rust implementation][38589] * [The `armv7-linux-androideabi` target no longer enables NEON extensions, per Google's ABI guide][38413] * [The stock standard library can be compiled for Redox OS][38401] * [Rust has initial SPARC support][38726]. Tier 3. No builds available. * [Rust has experimental support for Nvidia PTX][38559]. Tier 3. No builds available. * [Fix backtraces on i686-pc-windows-gnu by disabling FPO][39379] Compatibility Notes ------------------- * [Uninhabitable enums (those without any variants) no longer permit wildcard match patterns][38069] * In this release, references to uninhabited types can not be pattern-matched. This was accidentally allowed in 1.15. * [The compiler's `dead_code` lint now accounts for type aliases][38051]. * [Ctrl-Z returns from `Stdin.read()` when reading from the console on Windows][38274] * [Clean up semantics of `self` in an import list][38313] [37057]: rust-lang/rust#37057 [37761]: rust-lang/rust#37761 [38006]: rust-lang/rust#38006 [38051]: rust-lang/rust#38051 [38062]: rust-lang/rust#38062 [38062]: rust-lang/rust#38622 [38066]: rust-lang/rust#38066 [38069]: rust-lang/rust#38069 [38131]: rust-lang/rust#38131 [38154]: rust-lang/rust#38154 [38274]: rust-lang/rust#38274 [38304]: rust-lang/rust#38304 [38313]: rust-lang/rust#38313 [38314]: rust-lang/rust#38314 [38327]: rust-lang/rust#38327 [38401]: rust-lang/rust#38401 [38413]: rust-lang/rust#38413 [38469]: rust-lang/rust#38469 [38559]: rust-lang/rust#38559 [38571]: rust-lang/rust#38571 [38580]: rust-lang/rust#38580 [38589]: rust-lang/rust#38589 [38670]: rust-lang/rust#38670 [38712]: rust-lang/rust#38712 [38726]: rust-lang/rust#38726 [38781]: rust-lang/rust#38781 [38798]: rust-lang/rust#38798 [38909]: rust-lang/rust#38909 [38920]: rust-lang/rust#38920 [38927]: rust-lang/rust#38927 [39048]: rust-lang/rust#39048 [39282]: rust-lang/rust#39282 [39379]: rust-lang/rust#39379 [`<*const T>::wrapping_offset`]: https://doc.rust-lang.org/std/primitive.pointer.html#method.wrapping_offset [`<*mut T>::wrapping_offset`]: https://doc.rust-lang.org/std/primitive.pointer.html#method.wrapping_offset [`Duration::checked_add`]: https://doc.rust-lang.org/std/time/struct.Duration.html#method.checked_add [`Duration::checked_div`]: https://doc.rust-lang.org/std/time/struct.Duration.html#method.checked_div [`Duration::checked_mul`]: https://doc.rust-lang.org/std/time/struct.Duration.html#method.checked_mul [`Duration::checked_sub`]: https://doc.rust-lang.org/std/time/struct.Duration.html#method.checked_sub [`File::set_permissions`]: https://doc.rust-lang.org/std/fs/struct.File.html#method.set_permissions [`IpAddr::is_ipv4`]: https://doc.rust-lang.org/std/net/enum.IpAddr.html#method.is_ipv4 [`IpAddr::is_ipv6`]: https://doc.rust-lang.org/std/net/enum.IpAddr.html#method.is_ipv6 [`Result::unwrap_or_default`]: https://doc.rust-lang.org/std/result/enum.Result.html#method.unwrap_or_default [`SocketAddr::is_ipv4`]: https://doc.rust-lang.org/std/net/enum.SocketAddr.html#method.is_ipv4 [`SocketAddr::is_ipv6`]: https://doc.rust-lang.org/std/net/enum.SocketAddr.html#method.is_ipv6 [`String::insert_str`]: https://doc.rust-lang.org/std/string/struct.String.html#method.insert_str [`String::split_off`]: https://doc.rust-lang.org/std/string/struct.String.html#method.split_off [`Vec::dedup_by_key`]: https://doc.rust-lang.org/std/vec/struct.Vec.html#method.dedup_by_key [`Vec::dedup_by`]: https://doc.rust-lang.org/std/vec/struct.Vec.html#method.dedup_by [`VecDeque::resize`]: https://doc.rust-lang.org/std/collections/vec_deque/struct.VecDeque.html#method.resize [`VecDeque::truncate`]: https://doc.rust-lang.org/std/collections/vec_deque/struct.VecDeque.html#method.truncate [`str::repeat`]: https://doc.rust-lang.org/std/primitive.str.html#method.repeat [`str::replacen`]: https://doc.rust-lang.org/std/primitive.str.html#method.replacen [cargo/3296]: rust-lang/cargo#3296 [cargo/3301]: rust-lang/cargo#3301 [cargo/3443]: rust-lang/cargo#3443 [cargo/3511]: rust-lang/cargo#3511 [cargo/3515]: rust-lang/cargo#3515 [cargo/3534]: rust-lang/cargo#3534 [cargo/3546]: rust-lang/cargo#3546 [cargo/3557]: rust-lang/cargo#3557 [cargo/3604]: rust-lang/cargo#3604 [RFC 1623]: https://github.com/rust-lang/rfcs/blob/master/text/1623-static.md

rust-highfive assigned brson Jan 18, 2017

alexcrichton reviewed Jan 18, 2017

View reviewed changes

raphlinus added 2 commits January 24, 2017 17:51

Add dep-info generation

2b2b55f

Make cargo output a ".d" file containing dependency info (in a format that make and ninja can consume) for each artifact it produces. This will help in integrating into other build systems.

Track TargetKind::Example split

1ed7f69

raphlinus force-pushed the master branch from 6e9124e to 1ed7f69 Compare January 25, 2017 01:56

alexcrichton reviewed Jan 25, 2017

View reviewed changes

raphlinus added 3 commits January 26, 2017 12:47

More improvements

f9fdc76

Use Context::target_filenames rather than trying to redo that. Traverse path dependencies recursively, so we get transitive deps. Use existing fingerprint parsing.

Buffer file writing for dep-info

abf8e35

Use BufWriter when generating .d files with dep-info, to avoid excessive numbers of write syscalls.

Add build script rerun-if-changed dependencies

d79249b

Also pick up the rerun-if-changed dependencies from build scripts.

alexcrichton reviewed Jan 26, 2017

View reviewed changes

Add simple tests, cleanup

5cb6995

Tests for existence for dep-info output in simple compilation cases. Deletes the dep-info file if it fails (for example, if it can't find one of the dep-info inputs).

bors merged commit 5cb6995 into rust-lang:master Jan 28, 2017

matklad mentioned this pull request Sep 4, 2017

eRFC: Cargo build system integration rust-lang/rfcs#2136

Merged

matklad mentioned this pull request Dec 11, 2017

Avoid rebuilding a project when cwd changes #4788

Merged

rtsuk mentioned this pull request Mar 10, 2018

Have --dep-info generate gn-compatible dependency files #5162

Closed

ehuss added this to the 1.16.0 milestone Feb 6, 2022

Add dep-info generation #3557

Add dep-info generation #3557

Conversation

raphlinus commented Jan 18, 2017

rust-highfive commented Jan 18, 2017

raphlinus commented Jan 18, 2017

alexcrichton left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

raphlinus commented Jan 23, 2017

alexcrichton commented Jan 23, 2017

raphlinus commented Jan 25, 2017

raphlinus commented Jan 25, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

raphlinus commented Jan 26, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

alexcrichton commented Jan 26, 2017

raphlinus commented Jan 27, 2017

alexcrichton commented Jan 27, 2017

raphlinus commented Jan 27, 2017

raphlinus commented Jan 27, 2017

alexcrichton commented Jan 27, 2017

bors commented Jan 27, 2017

bors commented Jan 27, 2017

bors commented Jan 28, 2017