unify bootstrap and shim binaries #95164

fee1-dead · 2022-03-21T06:37:45Z

they are now dispatched based on the binary name it is called with.
target/debug/rust{c, doc} are now hard linked to bootstrap.

This fixes #95141, cc @jyn514.

rust-highfive · 2022-03-21T06:37:48Z

r? @Mark-Simulacrum

(rust-highfive has picked a reviewer for you, use r? to override)

jyn514

This is awesome ❤️

src/bootstrap/bin/main.rs

src/bootstrap/Cargo.toml

src/bootstrap/bin/main.rs

bjorn3 · 2022-03-23T10:26:27Z

src/bootstrap/lib.rs

+        if bootstrap_out.join("bootstrap").exists() && !bootstrap_out.join("rustc").exists() {
+            let bin = bootstrap_out.join("bootstrap");
+            fs::hard_link(&bin, bootstrap_out.join("rustc"))
+                .expect("failed to link rustc to bootstrap shim");


Maybe use Build::copy instead? It hardlinks and falls back to copying if it fails. Some systems may not support hardlinks.

Build::copy defeats the point, it means the new files won't be updated on changes.

I don't know a good solution here :/ maybe we could do test somewhere of whether the fs supports file links and copy unconditionally if so?

it means the new files won't be updated on changes

Not sure what you meant. The current implementation is as follows:

/// Copies a file from `src` to `dst` pub fn copy(&self, src: &Path, dst: &Path) { if self.config.dry_run { return; } self.verbose_than(1, &format!("Copy {:?} to {:?}", src, dst)); if src == dst { return; } let _ = fs::remove_file(&dst); let metadata = t!(src.symlink_metadata()); if metadata.file_type().is_symlink() { let link = t!(fs::read_link(src)); t!(symlink_file(link, dst)); } else if let Ok(()) = fs::hard_link(src, dst) { // Attempt to "easy copy" by creating a hard link // (symlinks don't work on windows), but if that fails // just fall back to a slow `copy` operation. } else { if let Err(e) = fs::copy(src, dst) { panic!("failed to copy `{}` to `{}`: {}", src.display(), dst.display(), e) } t!(fs::set_permissions(dst, metadata.permissions())); let atime = FileTime::from_last_access_time(&metadata); let mtime = FileTime::from_last_modification_time(&metadata); t!(filetime::set_file_times(dst, atime, mtime)); } }

Which is akin to an unconditional hard_link.

No, because if you hit the fs::copy path, then the next time bootstrap is executed "rustc".exists() will return true and it will never recopy the binary. Which defeats the whole point of this change.

That is true, I will just remove the check for rustc.exists() because it was triggering a filesystem error because the dest file already existed.

Ah, I suppose that works too. The hard link should be very cheap compared to everything else bootstrap does 👍

Mark-Simulacrum

Can we name the "primary" (i.e., the one produced by rustc) binary something less difficult to refer to? Maybe rustbuild-binary-dispatch-shim?

Please also squash the commits as you make changes, no need for "review changes" commits.

src/bootstrap/bin/main.rs

src/bootstrap/bin/rustc.rs

src/bootstrap/lib.rs

bors · 2022-04-10T14:48:55Z

☔ The latest upstream changes (presumably #95253) made this pull request unmergeable. Please resolve the merge conflicts.

they are now dispatched based on the binary name it is called with. `target/debug/rust{c, doc}` are now hard linked to `bootstrap`.

fee1-dead · 2022-05-10T23:25:08Z

Umm, I took a look and it looks like it already tries to remove the destination before copying: https://cs.github.com/rust-lang/rust/blob/fee75fbe11b1fad5d93c723234178b2a329a3c03/src/bootstrap/lib.rs#L1418

bjorn3 · 2022-05-11T09:36:46Z

Windows doesn't allow deleting files that are currently open by default. I believe FILE_DISPOSITION_POSIX_SEMANTICS allows deleting open files though.

fee1-dead · 2022-05-11T13:09:09Z

So I tried to use posix_delete on windows with the latest commit. Should we see whether that will work?

Mark-Simulacrum · 2022-05-11T13:22:08Z

I'm not sure that function is publicly exposed, but we can try. @bors r+

bors · 2022-05-11T13:22:09Z

📌 Commit 64a8af9 has been approved by Mark-Simulacrum

bors · 2022-05-11T18:28:37Z

⌛ Testing commit 64a8af9 with merge c0220834cf72ccea19c0454ad82db04e18ea67a7...

bors · 2022-05-11T18:34:09Z

💔 Test failed - checks-actions

rust-log-analyzer · 2022-05-11T18:34:35Z

The job x86_64-msvc-tools failed! Check out the build log: (web) (plain)

Click to see the possible cause of the failure (guessed by this bot)

   Compiling xz2 v0.1.6
error[E0603]: module `sys` is private
    --> src\bootstrap\lib.rs:1430:36
     |
1430 |             if let Ok(file) = std::sys::fs::File::open(dst, &std::sys::fs::OpenOptions::new()) {
     |                                    ^^^ private module
note: the module `sys` is defined here

error[E0603]: module `sys` is private
    --> src\bootstrap\lib.rs:1430:67
    --> src\bootstrap\lib.rs:1430:67
     |
1430 |             if let Ok(file) = std::sys::fs::File::open(dst, &std::sys::fs::OpenOptions::new()) {
     |                                                                   ^^^ private module
note: the module `sys` is defined here


error[E0624]: associated function `posix_delete` is private
    --> src\bootstrap\lib.rs:1431:30
     |
1431 |                 let _ = file.posix_delete();

Some errors have detailed explanations: E0603, E0624.
For more information about an error, try `rustc --explain E0603`.
error: could not compile `bootstrap` due to 3 previous errors
---
   Compiling bootstrap v0.0.0 (D:\a\rust\rust\src\bootstrap)
error[E0603]: module `sys` is private
    --> src\bootstrap\lib.rs:1430:36
     |
1430 |             if let Ok(file) = std::sys::fs::File::open(dst, &std::sys::fs::OpenOptions::new()) {
     |                                    ^^^ private module
note: the module `sys` is defined here

error[E0603]: module `sys` is private
    --> src\bootstrap\lib.rs:1430:67
    --> src\bootstrap\lib.rs:1430:67
     |
1430 |             if let Ok(file) = std::sys::fs::File::open(dst, &std::sys::fs::OpenOptions::new()) {
     |                                                                   ^^^ private module
note: the module `sys` is defined here


error[E0624]: associated function `posix_delete` is private
    --> src\bootstrap\lib.rs:1431:30
     |
1431 |                 let _ = file.posix_delete();

Some errors have detailed explanations: E0603, E0624.
For more information about an error, try `rustc --explain E0603`.
error: could not compile `bootstrap` due to 3 previous errors
---
   Compiling bootstrap v0.0.0 (D:\a\rust\rust\src\bootstrap)
error[E0603]: module `sys` is private
    --> src\bootstrap\lib.rs:1430:36
     |
1430 |             if let Ok(file) = std::sys::fs::File::open(dst, &std::sys::fs::OpenOptions::new()) {
     |                                    ^^^ private module
note: the module `sys` is defined here

error[E0603]: module `sys` is private
    --> src\bootstrap\lib.rs:1430:67
    --> src\bootstrap\lib.rs:1430:67
     |
1430 |             if let Ok(file) = std::sys::fs::File::open(dst, &std::sys::fs::OpenOptions::new()) {
     |                                                                   ^^^ private module
note: the module `sys` is defined here


error[E0624]: associated function `posix_delete` is private
    --> src\bootstrap\lib.rs:1431:30
     |
1431 |                 let _ = file.posix_delete();

Some errors have detailed explanations: E0603, E0624.
For more information about an error, try `rustc --explain E0603`.
error: could not compile `bootstrap` due to 3 previous errors
---
   Compiling bootstrap v0.0.0 (D:\a\rust\rust\src\bootstrap)
error[E0603]: module `sys` is private
    --> src\bootstrap\lib.rs:1430:36
     |
1430 |             if let Ok(file) = std::sys::fs::File::open(dst, &std::sys::fs::OpenOptions::new()) {
     |                                    ^^^ private module
note: the module `sys` is defined here

error[E0603]: module `sys` is private
    --> src\bootstrap\lib.rs:1430:67
    --> src\bootstrap\lib.rs:1430:67
     |
1430 |             if let Ok(file) = std::sys::fs::File::open(dst, &std::sys::fs::OpenOptions::new()) {
     |                                                                   ^^^ private module
note: the module `sys` is defined here


error[E0624]: associated function `posix_delete` is private
    --> src\bootstrap\lib.rs:1431:30
     |
1431 |                 let _ = file.posix_delete();

Some errors have detailed explanations: E0603, E0624.
For more information about an error, try `rustc --explain E0603`.
error: could not compile `bootstrap` due to 3 previous errors
---
   Compiling bootstrap v0.0.0 (D:\a\rust\rust\src\bootstrap)
error[E0603]: module `sys` is private
    --> src\bootstrap\lib.rs:1430:36
     |
1430 |             if let Ok(file) = std::sys::fs::File::open(dst, &std::sys::fs::OpenOptions::new()) {
     |                                    ^^^ private module
note: the module `sys` is defined here

error[E0603]: module `sys` is private
    --> src\bootstrap\lib.rs:1430:67
    --> src\bootstrap\lib.rs:1430:67
     |
1430 |             if let Ok(file) = std::sys::fs::File::open(dst, &std::sys::fs::OpenOptions::new()) {
     |                                                                   ^^^ private module
note: the module `sys` is defined here


error[E0624]: associated function `posix_delete` is private
    --> src\bootstrap\lib.rs:1431:30
     |
1431 |                 let _ = file.posix_delete();

Some errors have detailed explanations: E0603, E0624.
For more information about an error, try `rustc --explain E0603`.
error: could not compile `bootstrap` due to 3 previous errors

bors · 2022-05-14T08:14:44Z

⌛ Testing commit 64a8af9 with merge 83698375db8b68e4b56f5a650443f96accc44471...

bors · 2022-05-14T08:20:06Z

💔 Test failed - checks-actions

rust-log-analyzer · 2022-05-14T08:23:44Z

The job i686-msvc-1 failed! Check out the build log: (web) (plain)

Click to see the possible cause of the failure (guessed by this bot)

   Compiling xz2 v0.1.6
error[E0603]: module `sys` is private
    --> src\bootstrap\lib.rs:1430:36
     |
1430 |             if let Ok(file) = std::sys::fs::File::open(dst, &std::sys::fs::OpenOptions::new()) {
     |                                    ^^^ private module
note: the module `sys` is defined here

error[E0603]: module `sys` is private
    --> src\bootstrap\lib.rs:1430:67
    --> src\bootstrap\lib.rs:1430:67
     |
1430 |             if let Ok(file) = std::sys::fs::File::open(dst, &std::sys::fs::OpenOptions::new()) {
     |                                                                   ^^^ private module
note: the module `sys` is defined here


error[E0624]: associated function `posix_delete` is private
    --> src\bootstrap\lib.rs:1431:30
     |
1431 |                 let _ = file.posix_delete();

Some errors have detailed explanations: E0603, E0624.
For more information about an error, try `rustc --explain E0603`.
error: could not compile `bootstrap` due to 3 previous errors
---
   Compiling bootstrap v0.0.0 (D:\a\rust\rust\src\bootstrap)
error[E0603]: module `sys` is private
    --> src\bootstrap\lib.rs:1430:36
     |
1430 |             if let Ok(file) = std::sys::fs::File::open(dst, &std::sys::fs::OpenOptions::new()) {
     |                                    ^^^ private module
note: the module `sys` is defined here

error[E0603]: module `sys` is private
    --> src\bootstrap\lib.rs:1430:67
    --> src\bootstrap\lib.rs:1430:67
     |
1430 |             if let Ok(file) = std::sys::fs::File::open(dst, &std::sys::fs::OpenOptions::new()) {
     |                                                                   ^^^ private module
note: the module `sys` is defined here


error[E0624]: associated function `posix_delete` is private
    --> src\bootstrap\lib.rs:1431:30
     |
1431 |                 let _ = file.posix_delete();

Some errors have detailed explanations: E0603, E0624.
For more information about an error, try `rustc --explain E0603`.
error: could not compile `bootstrap` due to 3 previous errors
---
   Compiling bootstrap v0.0.0 (D:\a\rust\rust\src\bootstrap)
error[E0603]: module `sys` is private
    --> src\bootstrap\lib.rs:1430:36
     |
1430 |             if let Ok(file) = std::sys::fs::File::open(dst, &std::sys::fs::OpenOptions::new()) {
     |                                    ^^^ private module
note: the module `sys` is defined here

error[E0603]: module `sys` is private
    --> src\bootstrap\lib.rs:1430:67
    --> src\bootstrap\lib.rs:1430:67
     |
1430 |             if let Ok(file) = std::sys::fs::File::open(dst, &std::sys::fs::OpenOptions::new()) {
     |                                                                   ^^^ private module
note: the module `sys` is defined here


error[E0624]: associated function `posix_delete` is private
    --> src\bootstrap\lib.rs:1431:30
     |
1431 |                 let _ = file.posix_delete();

Some errors have detailed explanations: E0603, E0624.
For more information about an error, try `rustc --explain E0603`.
error: could not compile `bootstrap` due to 3 previous errors
---
   Compiling bootstrap v0.0.0 (D:\a\rust\rust\src\bootstrap)
error[E0603]: module `sys` is private
    --> src\bootstrap\lib.rs:1430:36
     |
1430 |             if let Ok(file) = std::sys::fs::File::open(dst, &std::sys::fs::OpenOptions::new()) {
     |                                    ^^^ private module
note: the module `sys` is defined here

error[E0603]: module `sys` is private
    --> src\bootstrap\lib.rs:1430:67
    --> src\bootstrap\lib.rs:1430:67
     |
1430 |             if let Ok(file) = std::sys::fs::File::open(dst, &std::sys::fs::OpenOptions::new()) {
     |                                                                   ^^^ private module
note: the module `sys` is defined here


error[E0624]: associated function `posix_delete` is private
    --> src\bootstrap\lib.rs:1431:30
     |
1431 |                 let _ = file.posix_delete();

Some errors have detailed explanations: E0603, E0624.
For more information about an error, try `rustc --explain E0603`.
error: could not compile `bootstrap` due to 3 previous errors
---
   Compiling bootstrap v0.0.0 (D:\a\rust\rust\src\bootstrap)
error[E0603]: module `sys` is private
    --> src\bootstrap\lib.rs:1430:36
     |
1430 |             if let Ok(file) = std::sys::fs::File::open(dst, &std::sys::fs::OpenOptions::new()) {
     |                                    ^^^ private module
note: the module `sys` is defined here

error[E0603]: module `sys` is private
    --> src\bootstrap\lib.rs:1430:67
    --> src\bootstrap\lib.rs:1430:67
     |
1430 |             if let Ok(file) = std::sys::fs::File::open(dst, &std::sys::fs::OpenOptions::new()) {
     |                                                                   ^^^ private module
note: the module `sys` is defined here


error[E0624]: associated function `posix_delete` is private
    --> src\bootstrap\lib.rs:1431:30
     |
1431 |                 let _ = file.posix_delete();

Some errors have detailed explanations: E0603, E0624.
For more information about an error, try `rustc --explain E0603`.
error: could not compile `bootstrap` due to 3 previous errors

jyn514 · 2022-05-25T23:25:37Z

This should no longer be necessary with the new plan, since no one will run cargo run locally (they'll either download it from CI or build all binaries from source through x.py): #95141 (comment)

Sorry to spend so much of your time on this.

fee1-dead · 2022-05-26T00:28:35Z

I did take a look at rustup, and it looks like they retry the deletion with timeout instead of invoking the functions once. Maybe that will resolve the issue (perhaps due to concurrency?) But if this is not needed anymore I'm happy to leave it.

rust-highfive assigned Mark-Simulacrum Mar 21, 2022

rust-highfive added the S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. label Mar 21, 2022

This comment has been minimized.

Sign in to view

fee1-dead force-pushed the unify-shim-bins branch from a4ecadb to eeb4fe9 Compare March 21, 2022 06:50

This comment has been minimized.

Sign in to view

fee1-dead force-pushed the unify-shim-bins branch from eeb4fe9 to 2fdf200 Compare March 21, 2022 07:14

This comment has been minimized.

Sign in to view

jyn514 reviewed Mar 21, 2022

View reviewed changes

src/bootstrap/bin/main.rs Outdated Show resolved Hide resolved

src/bootstrap/Cargo.toml Outdated Show resolved Hide resolved

jyn514 mentioned this pull request Mar 21, 2022

Tracking Issue for removing python as the main bootstrap entry point #94829

Open

15 tasks

bjorn3 reviewed Mar 23, 2022

View reviewed changes

src/bootstrap/bin/main.rs Outdated Show resolved Hide resolved

bjorn3 reviewed Mar 23, 2022

View reviewed changes

src/bootstrap/bin/main.rs Outdated Show resolved Hide resolved

bjorn3 reviewed Mar 23, 2022

View reviewed changes

fee1-dead force-pushed the unify-shim-bins branch from c94b2b3 to 52a1dc0 Compare March 23, 2022 10:44

jyn514 approved these changes Mar 23, 2022

View reviewed changes

Mark-Simulacrum reviewed Apr 6, 2022

View reviewed changes

src/bootstrap/bin/main.rs Outdated Show resolved Hide resolved

src/bootstrap/bin/main.rs Outdated Show resolved Hide resolved

src/bootstrap/bin/rustc.rs Show resolved Hide resolved

src/bootstrap/lib.rs Outdated Show resolved Hide resolved

src/bootstrap/lib.rs Outdated Show resolved Hide resolved

Mark-Simulacrum added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Apr 6, 2022

fee1-dead force-pushed the unify-shim-bins branch 2 times, most recently from a6aa04c to 578826a Compare April 7, 2022 04:56

This comment has been minimized.

Sign in to view

fee1-dead force-pushed the unify-shim-bins branch from 578826a to 57c250f Compare April 7, 2022 05:01

This comment has been minimized.

Sign in to view

fee1-dead force-pushed the unify-shim-bins branch from 57c250f to 812d6d9 Compare April 7, 2022 05:10

This comment has been minimized.

Sign in to view

fee1-dead force-pushed the unify-shim-bins branch from 812d6d9 to cc29d18 Compare April 7, 2022 07:06

This comment has been minimized.

Sign in to view

fee1-dead force-pushed the unify-shim-bins branch from cc29d18 to cdfa1a8 Compare April 7, 2022 08:20

fee1-dead force-pushed the unify-shim-bins branch from cdfa1a8 to 59d85f4 Compare April 12, 2022 09:25

Mark-Simulacrum added the S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. label Apr 30, 2022

unify bootstrap and shim binaries

b616d7b

they are now dispatched based on the binary name it is called with. `target/debug/rust{c, doc}` are now hard linked to `bootstrap`.

fee1-dead force-pushed the unify-shim-bins branch from b86bec7 to b616d7b Compare May 10, 2022 23:25

fee1-dead added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. and removed S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. labels May 10, 2022

Use posix_delete on windows

64a8af9

bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels May 11, 2022

bors added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. and removed S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. labels May 11, 2022

Mark-Simulacrum added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels May 14, 2022

apiraino added the T-infra Relevant to the infrastructure team, which will review and decide on the PR/issue. label May 23, 2022

fee1-dead closed this May 26, 2022

fee1-dead deleted the unify-shim-bins branch March 25, 2023 09:02

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

unify bootstrap and shim binaries #95164

unify bootstrap and shim binaries #95164

fee1-dead commented Mar 21, 2022

rust-highfive commented Mar 21, 2022

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

jyn514 left a comment

bjorn3 Mar 23, 2022

jyn514 Mar 23, 2022

fee1-dead Mar 23, 2022 •

edited

Loading

jyn514 Mar 23, 2022

fee1-dead Mar 23, 2022 •

edited

Loading

jyn514 Mar 23, 2022

Mark-Simulacrum left a comment

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

bors commented Apr 10, 2022

fee1-dead commented May 10, 2022

bjorn3 commented May 11, 2022

fee1-dead commented May 11, 2022

Mark-Simulacrum commented May 11, 2022

bors commented May 11, 2022

bors commented May 11, 2022

bors commented May 11, 2022

rust-log-analyzer commented May 11, 2022

bors commented May 14, 2022

bors commented May 14, 2022

rust-log-analyzer commented May 14, 2022

jyn514 commented May 25, 2022

fee1-dead commented May 26, 2022

unify bootstrap and shim binaries #95164

unify bootstrap and shim binaries #95164

Conversation

fee1-dead commented Mar 21, 2022

rust-highfive commented Mar 21, 2022

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

jyn514 left a comment

Choose a reason for hiding this comment

bjorn3 Mar 23, 2022

Choose a reason for hiding this comment

jyn514 Mar 23, 2022

Choose a reason for hiding this comment

fee1-dead Mar 23, 2022 • edited Loading

Choose a reason for hiding this comment

jyn514 Mar 23, 2022

Choose a reason for hiding this comment

fee1-dead Mar 23, 2022 • edited Loading

Choose a reason for hiding this comment

jyn514 Mar 23, 2022

Choose a reason for hiding this comment

Mark-Simulacrum left a comment

Choose a reason for hiding this comment

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

bors commented Apr 10, 2022

fee1-dead commented May 10, 2022

bjorn3 commented May 11, 2022

fee1-dead commented May 11, 2022

Mark-Simulacrum commented May 11, 2022

bors commented May 11, 2022

bors commented May 11, 2022

bors commented May 11, 2022

rust-log-analyzer commented May 11, 2022

bors commented May 14, 2022

bors commented May 14, 2022

rust-log-analyzer commented May 14, 2022

jyn514 commented May 25, 2022

fee1-dead commented May 26, 2022

fee1-dead Mar 23, 2022 •

edited

Loading

fee1-dead Mar 23, 2022 •

edited

Loading