Tracking Issue for trim-paths RFC 3127 #111540

ehuss · 2023-05-13T17:34:50Z

This is a tracking issue for the RFC 3127 (rust-lang/rfcs#3127).

This enhancement adds the --remap-path-scope command-line flag to control the scoping of how paths get remapped in the resulting binary.

Issues: F-trim-paths Feature: trim-paths
Documentation (rustc): https://doc.rust-lang.org/nightly/unstable-book/compiler-flags/remap-path-scope.html
Documentation (cargo): https://doc.rust-lang.org/nightly/cargo/reference/unstable.html#profile-trim-paths-option
Cargo tracking issue: rust-lang/cargo#12137

About tracking issues

Tracking issues are used to record the overall progress of implementation.
They are also used as hubs connecting to other relevant issues, e.g., bugs or open design questions.
A tracking issue is however not meant for large scale discussion, questions, or bug reports about a feature.
Instead, open a dedicated issue for the specific matter and add the relevant feature gate label.

Steps

Implement the RFC (Implement rustc part of RFC 3127 trim-paths #115214)
Adjust documentation (see instructions on rustc-dev-guide)
Stabilization PR (see instructions on rustc-dev-guide)

Unresolved Questions

Should we use a slightly more complex remapping rule, like distinguishing packages from registry, git and path, as proposed in Enable --remap-path-prefix for absolute paths by default #40552?
What concerns are there regarding the use of relative paths causing issues with backtraces and debuggers?
Should we treat the current package the same as other packages?
Will these cover all potentially embedded paths? Have we missed anything?
What exactly is the intended use case for the "diagnostics" scope, and when would someone want to turn it on? Is that something someone would really want to enable in cargo? See --remap-path-prefix no longer works for compiler messages #87745.
Should rustdoc support remapping (like for diagnostics)? Does remapping matter for doctests?
- doctest has supported remapping: Simplify trim-paths feature by merging all debuginfo options together #122450
- rustdoc diagnostics hasn't: rustdoc: use remap-path-prefix #69264 (comment)

Implementation history

The text was updated successfully, but these errors were encountered:

…r-errors Implement rustc part of RFC 3127 trim-paths This PR implements (or at least tries to) [RFC 3127 trim-paths](rust-lang#111540), the rustc part. That is `-Zremap-path-scope` with all of it's components/scopes. `@rustbot` label: +F-trim-paths

Implement rustc part of RFC 3127 trim-paths This PR implements (or at least tries to) [RFC 3127 trim-paths](rust-lang/rust#111540), the rustc part. That is `-Zremap-path-scope` with all of it's components/scopes. `@rustbot` label: +F-trim-paths

weihanglo · 2023-11-05T03:01:03Z

I was trying to remove OSO symbols on macOS. Did a research and summarized in this issue: #116948 (comment). Now, I can see that there is no way for trim-paths to really trim all paths unless rustc does some dirty trick for older macOS platforms. Would it be a blocker if we want to stabilize this feature, say, in 2024 edition?

cc @Urgau you might be interested.

michaelwoerister · 2023-12-12T13:23:39Z

What concerns are there regarding the use of relative paths causing issues with backtraces and debuggers?

I talked about this with @danielframpton the other day and he noticed that the current [package name]-[version] rule for remapping might make things unnecessarily complicated for debuggers. For making the debugging experience easy to set up, it would be beneficial to require as little configuration as possible. The RFC does not explicitly make file system paths (as used by Cargo) match with how things are renamed, so in the general case the debugger needs a separate source file search path for each crate involved.

However, if we prescribed that each crate from was remapped to the relative path Cargo uses inside its $CARGO_HOME/registry/src and $CARGO_HOME/git/checkouts directories, then we have a fixed number of source directories a debugger needs to know about. The renaming would then be [registry]/[package name]-[version] for registry crates and [package name]-[sha] for git dependencies.

The current [package name]-[version] rule is already pretty close, but that seems to be by accident. It would be good to make this something that can be relied on.

This might also help with backtraces?

I don't know if there is something similarly useful we can do for path dependencies.

Should we treat the current package the same as other packages?

This also came up. Unconditionally stripping the path to the current package might be too inflexible in some cases. E.g. when building a staticlib, one might want to make the paths coming from the current package identifiable in some way. If they all start with src/.. then they might clash with the paths of the codebase that uses the staticlib as a dependency.

weihanglo · 2023-12-13T15:17:00Z

However, if we prescribed that each crate from was remapped to the relative path Cargo uses inside its $CARGO_HOME/registry/src and $CARGO_HOME/git/checkouts directories, then we have a fixed number of source directories a debugger needs to know about. The renaming would then be [registry]/[package name]-[version] for registry crates and [package name]-[sha] for git dependencies.

This plan sounds really tenable to me! Will do a change on cargo side for this, and see how people think about it.

For path dependencies, maybe we could have a common prefix for them, like cargo_deps@local/<pkg>-<version>. Might not be useful as others dep kinds, but at least people can have a single location for path dependencies?

then they might clash with the paths of the codebase that uses the staticlib as a dependency.

Sorry I don't fully understand. Could you give a more concrete example of this?

Implement RFC 3127 sysroot path handling changes Fix rust-lang#105907 Fix rust-lang#85463 Implement parts of rust-lang#111540 Right now, backtraces into sysroot always shows /rustc/$hash in diagnostics, e.g. ``` thread 'main' panicked at 'hello world', map-panic.rs:2:50 stack backtrace: 0: std::panicking::begin_panic at /rustc/a55dd71d5fb0ec5a6a3a9e8c27b2127ba491ce52/library/std/src/panicking.rs:616:12 1: map_panic::main::{{closure}} at ./map-panic.rs:2:50 2: core::option::Option<T>::map at /rustc/a55dd71d5fb0ec5a6a3a9e8c27b2127ba491ce52/library/core/src/option.rs:929:29 3: map_panic::main at ./map-panic.rs:2:30 4: core::ops::function::FnOnce::call_once at /rustc/a55dd71d5fb0ec5a6a3a9e8c27b2127ba491ce52/library/core/src/ops/function.rs:248:5 note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace. ``` [RFC 3127 said](https://rust-lang.github.io/rfcs/3127-trim-paths.html#changing-handling-of-sysroot-path-in-rustc) > We want to change this behaviour such that, when rust-src source files can be discovered, the virtual path is discarded and therefore the local path will be embedded, unless there is a --remap-path-prefix that causes this local path to be remapped in the usual way. This PR implements this behaviour. When `rust-src` is present at compile time, rustc replaces /rustc/$hash with a real path into local rust-src with best effort. To sanitise this, users must explicitly supply `--remap-path-prefix=<path to rust-src>=foo`.

michaelwoerister · 2023-12-14T09:03:13Z

For path dependencies, maybe we could have a common prefix for them, like cargo_deps@local/-. Might not be useful as others dep kinds, but at least people can have a single location for path dependencies?

Maybe, yes. It's definitely worth thinking about this some more (and collecting ideas from other people).

Sorry I don't fully understand. Could you give a more concrete example of this?

As I understand it, the RFC states that paths from the workspace being compiled are trimmed by simply making them relative to the workspace root, that is, all file paths from workspace crates would look like src/foo/bar.rs or some_crate_in_the_workspace/src/foo/bar.rs. Those paths don't contain anything that would allow disambiguating them from other paths that are purely relative. One example where this is sub-optimal is when one has multiple Rust projects, each of which gets compiled into a staticlib that is then consumed within another, non-Rust project. Each of these Rust projects would have debuginfo file paths that just start with src/ and the paths for some of the files might clash , e.g. src/lib.rs. The debugger would have no way to know which src/lib.rs file to pick, even if both are in its source search path.

Therefore it would be good, if there was some way to avoid this problem. Allowing to prepend a prefix to the relative paths might be an option.

weihanglo · 2023-12-14T22:31:50Z

@michaelwoerister. Thank you for your feedback! I just opened a new issue rust-lang/cargo#13171 for further discussion about remap rules in Cargo.

michaelwoerister · 2024-02-09T08:58:27Z

For cargo install I wouldn't want trim paths. And ditto for precompiled opensource binaries as created using eg cargo-dist and downloaded using cargo-binstall.

To clarify: trim-paths will still give file paths of the form some_crate-1.0.1/src/lib.rs, so the information in a backtrace is still rather useful. If debuginfo is enabled (which currently is not the default for release builds), then the debugger should still be able to find the symbols, it just won't be able to find the source files currently.

EDIT: @bjorn3, would that be sufficient for your use case?

…=wesleywiser link.exe: Don't embed full path to PDB file in binary. This PR makes `rustc` unconditionally pass `/PDBALTPATH:%_PDB%` to MSVC-style linkers, causing the linker to only embed the filename of the PDB in the binary instead of the full path. This will help implement the [trim-paths RFC](rust-lang#111540) for `*-msvc` targets. Passing `/PDBALTPATH:%_PDB%` to the linker is already done by many projects that need reproducible builds and [debugger's should still be able to find the PDB](https://learn.microsoft.com/cpp/build/reference/pdbpath) if it is in the same directory as the binary. r? `@ghost` Fixes rust-lang#87825

michaelwoerister · 2024-03-13T10:05:14Z

Here is a table that shows the various combinations of the trim-paths and split-debuginfo Cargo options and (to the right side of the ⇒) what rustc options these map to and what the consequences of this are.

trim-paths	split-debuginfo	⇒	--remap-path-scope	Binary	Separate Debuginfo	Surprising?	Useful?	Remarks
none	off	⇒		unsanitzed			Yes
macro	off	⇒	macro	semi-sanitized			?
object	off	⇒	macro + split-debuginfo + split-debuginfo-path + unsplit-debuginfo	sanitized		Kind of	?	Must sanitize debuginfo too, otherwise "object" won't be sanitized
all	off	⇒	macro + split-debuginfo + split-debuginfo-path + unsplit-debuginfo	sanitized			Yes

trim-paths	split-debuginfo	⇒	--remap-path-scope	Binary	Separate Debuginfo	Surprising?	Useful?	Remarks
none	packed	⇒		unsanitzed	unsanitzed		Yes
macro	packed	⇒	macro	semi-sanitized	unsanitzed		?
object	packed	⇒	macro + unsplit-debuginfo + split-debuginfo-path	sanitized	unsanitzed	Yes	?	Debugger won't find source files
all	packed	⇒	all	sanitized	sanitized		Yes (?)	Debugger won't find source files

trim-paths	split-debuginfo	⇒	--remap-path-scope	Binary	Separate Debuginfo	Surprising?	Useful?	Remarks
none	unpacked	⇒		unsanitzed	unsanitzed		Yes	Fast builds with debuginfo
macro	unpacked	⇒	macro	semi-sanitized	unsanitzed		?
object	unpacked	⇒	macro + unsplit-debuginfo + split-debuginfo-path	sanitized	unsanitzed	Yes	?	Debugger won't find source files AND dwos
all	unpacked	⇒	all	sanitized	sanitized		?	Debugger won't find source files AND dwos

trim-paths	split-debuginfo	⇒	--remap-path-scope	Binary	Separate Debuginfo	Surprising?	Useful?	Remarks
none	post-link	⇒		unsanitzed	unsanitzed		Kind of	Not much use in splitting
macro	post-link	⇒	macro	sanitized	unsanitzed	A little	Kind of
object	post-link	⇒	macro + unsplit-debuginfo + split-debuginfo-path + split-debuginfo	sanitized	unsanitzed		Kind of	Might be useful when only distributing binary?
all	post-link	⇒	all	sanitized	sanitized		Yes

Observations:

Some of the combinations don't have a real use case
Some of the combinations are surprising. They don't do what one this they would do. E.g. trim-paths=object, split-debuginfo=packed leads to semi-broken debuginfo because line-tables, contained in the binary, have to be sanitized. But it the combination sounds like "don't touch debuginfo".
Only two cases make a distinction between the split-debuginfo and unsplit-debuginfo remapping scopes -- and they fall into the surprising and useless category.
No case makes a distinction between unsplit-debuginfo and split-debuginfo-path

My conclusion:

We should only have a single debuginfo scope and not further subdivide that into split-debuginfo, unsplit-debuginfo, and split-debuginfo-path. The only cases this makes a difference for are trim-paths=object, split-debuginfo=packed and trim-paths=object, split-debuginfo=unpacked but these cases have no real use case to begin with.

…=wesleywiser link.exe: Don't embed full path to PDB file in binary. This PR makes `rustc` unconditionally pass `/PDBALTPATH:%_PDB%` to MSVC-style linkers, causing the linker to only embed the filename of the PDB in the binary instead of the full path. This will help implement the [trim-paths RFC](rust-lang#111540) for `*-msvc` targets. Passing `/PDBALTPATH:%_PDB%` to the linker is already done by many projects that need reproducible builds and [debugger's should still be able to find the PDB](https://learn.microsoft.com/cpp/build/reference/pdbpath) if it is in the same directory as the binary. r? `@ghost` Fixes rust-lang#87825

link.exe: Don't embed full path to PDB file in binary. This PR makes `rustc` unconditionally pass `/PDBALTPATH:%_PDB%` to MSVC-style linkers, causing the linker to only embed the filename of the PDB in the binary instead of the full path. This will help implement the [trim-paths RFC](rust-lang/rust#111540) for `*-msvc` targets. Passing `/PDBALTPATH:%_PDB%` to the linker is already done by many projects that need reproducible builds and [debugger's should still be able to find the PDB](https://learn.microsoft.com/cpp/build/reference/pdbpath) if it is in the same directory as the binary. r? `@ghost` Fixes rust-lang/rust#87825

Zalathar · 2024-03-18T13:16:00Z

Via #122450, it’s come to my attention that this will affect coverage instrumentation, because the coverage mappings embedded in the binary usually contain at least one absolute path, and any adjustment to those paths is constrained by the capabilities of the llvm-cov tool.

michaelwoerister · 2024-03-18T13:23:55Z

For reference, Clang allows to separately control path remapping for code coverage: https://clang.llvm.org/docs/ClangCommandLineReference.html#cmdoption-clang-fcoverage-prefix-map

michaelwoerister · 2024-03-22T10:03:33Z

FYI, @Urgau's PR #122450 will merge all debuginfo scopes into a single one, since splitting them did not actually solve the problem it was intended to solve (see #111540 (comment)).

I'll wait for a few days before approving the PR, as to give everyone here a chance to speak up in case they disagree with the change.

…=michaelwoerister Simplify trim-paths feature by merging all debuginfo options together This PR simplifies the trim-paths feature by merging all debuginfo options together, as described in rust-lang#111540 (comment). And also do some correctness fixes found during the review. cc `@weihanglo` r? `@michaelwoerister`

…oerister Simplify trim-paths feature by merging all debuginfo options together This PR simplifies the trim-paths feature by merging all debuginfo options together, as described in rust-lang/rust#111540 (comment). And also do some correctness fixes found during the review. cc `@weihanglo` r? `@michaelwoerister`

redactedontop · 2024-07-21T21:20:38Z

...updates?

liciana24 · 2024-07-29T13:16:21Z

any movement here? still waiting on this and hoping scope didnt change from stripping all in release mode BY DEFAULT.

weihanglo · 2024-07-29T18:31:35Z

Not entirely sure on rustc side of this, but I feel like it is waiting for Cargo to report back the integration status. See also: rust-lang/cargo#12137 (comment)

estebank · 2024-08-09T18:16:04Z

Random question: are these flags meant to affect the output of diagnostics as well? One could argue that they should.

Urgau · 2024-08-09T18:56:50Z

Yes, that's the diagnostics scope, see the --remap-path-scope flag for more details.

ehuss mentioned this issue May 13, 2023

Tracking Issue for trim-paths RFC 3127 rust-lang/cargo#12137

Open

6 tasks

ehuss added the B-RFC-approved Blocker: Approved by a merged RFC but not yet implemented. label May 13, 2023

ehuss mentioned this issue May 13, 2023

New rustc and Cargo options to allow path sanitisation by default rust-lang/rfcs#3127

Merged

jyn514 added the A-reproducibility Area: Reproducible / Deterministic builds label Jun 16, 2023

jyn514 mentioned this issue Jun 16, 2023

Enable --remap-path-prefix for absolute paths by default #40552

Closed

Urgau mentioned this issue Aug 25, 2023

Implement rustc part of RFC 3127 trim-paths #115214

Merged

pietroalbini mentioned this issue Sep 18, 2023

Remap Cargo dependencies to /rust/deps #115872

Merged

epage mentioned this issue Oct 19, 2023

Reproducible builds: Automatically remap $CARGO_HOME and $PWD rust-lang/cargo#5505

Closed

weihanglo mentioned this issue Oct 24, 2023

feat: implement RFC 3127 -Ztrim-paths rust-lang/cargo#12625

Merged

8 tasks

Nemo157 mentioned this issue Nov 12, 2023

Using -Ztrim-paths with -Zbuild-std gives bad version numbers for standard library paths #117839

Closed

cbeuw mentioned this issue Nov 21, 2023

Implement RFC 3127 sysroot path handling changes #118149

Closed

graydon mentioned this issue Nov 30, 2023

build paths are embedded in wasms stellar/rs-soroban-env#1268

Open

weihanglo mentioned this issue Dec 14, 2023

trim-paths: Remap rules for different dependency kinds rust-lang/cargo#13171

Open

michaelwoerister mentioned this issue Feb 19, 2024

link.exe: Don't embed full path to PDB file in binary. #121297

Merged

Urgau mentioned this issue Mar 1, 2024

Removing absolute path in proc-macro #121842

Closed

Urgau mentioned this issue Mar 13, 2024

Simplify trim-paths feature by merging all debuginfo options together #122450

Merged

michaelwoerister mentioned this issue Apr 3, 2024

fix(trim-paths): trim SO and DW_AT_comp_dir symbols for root DI node #118518

Closed

jin-eld mentioned this issue Jul 19, 2024

QA warning on recipes that inherit cargo_bin rust-embedded/meta-rust-bin#183

Closed

weihanglo mentioned this issue Aug 1, 2024

rustdoc: use remap-path-prefix #69264

Closed

jieyouxu mentioned this issue Aug 14, 2024

Tracking Issue for Reproducible Build bugs and challenges #129080

Open

81 tasks

This was referenced Aug 21, 2024

Separate --remap-path-prefix for diagnostics and debug info #73740

Closed

Add flag to modify the working directory assumed by the diagnostics #88754

Closed

jieyouxu added A-debuginfo Area: Debugging information in compiled programs (DWARF, PDB, etc.) A-backtrace Area: backtraces labels Aug 21, 2024

Urgau mentioned this issue Aug 29, 2024

Implement RFC3137 trim-paths sysroot changes - take 2 #129687

Open

shr-project mentioned this issue Aug 29, 2024

uutils-coreutils: disable buildpaths error openembedded/meta-openembedded#865

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Tracking Issue for trim-paths RFC 3127 #111540

Tracking Issue for trim-paths RFC 3127 #111540

ehuss commented May 13, 2023 •

edited by Urgau

Loading

weihanglo commented Nov 5, 2023

michaelwoerister commented Dec 12, 2023

weihanglo commented Dec 13, 2023

michaelwoerister commented Dec 14, 2023

weihanglo commented Dec 14, 2023

michaelwoerister commented Feb 9, 2024 •

edited

Loading

michaelwoerister commented Mar 13, 2024

Zalathar commented Mar 18, 2024

michaelwoerister commented Mar 18, 2024

michaelwoerister commented Mar 22, 2024

redactedontop commented Jul 21, 2024

liciana24 commented Jul 29, 2024

weihanglo commented Jul 29, 2024

estebank commented Aug 9, 2024

Urgau commented Aug 9, 2024

Tracking Issue for trim-paths RFC 3127 #111540

Tracking Issue for trim-paths RFC 3127 #111540

Comments

ehuss commented May 13, 2023 • edited by Urgau Loading

About tracking issues

Steps

Unresolved Questions

Implementation history

weihanglo commented Nov 5, 2023

michaelwoerister commented Dec 12, 2023

weihanglo commented Dec 13, 2023

michaelwoerister commented Dec 14, 2023

weihanglo commented Dec 14, 2023

michaelwoerister commented Feb 9, 2024 • edited Loading

michaelwoerister commented Mar 13, 2024

Zalathar commented Mar 18, 2024

michaelwoerister commented Mar 18, 2024

michaelwoerister commented Mar 22, 2024

redactedontop commented Jul 21, 2024

liciana24 commented Jul 29, 2024

weihanglo commented Jul 29, 2024

estebank commented Aug 9, 2024

Urgau commented Aug 9, 2024

ehuss commented May 13, 2023 •

edited by Urgau

Loading

michaelwoerister commented Feb 9, 2024 •

edited

Loading