Make miri a subtree instead of a submodule #102028

oli-obk · 2022-09-19T19:09:59Z

rustup

When cargo-miri is executed as a cargo test runner or rustdoc runtool, external tools expect what they launch as the runner/runtool to be the process actually running the test. But in the implementation, we launch the Miri interpreter as a subprocess using std::process::Command. This tends to confuse other tools (like nextest) and users (like the author). What we really want is to call POSIX exec so that the cargo-miri process becomes the interpreter. So this implements just that; we call execve via a cfg(unix) extension trait. Windows has no such mechanism, but it also doesn't have POSIX signals, which is the primary tripping hazard this change fixes.

Use real exec on cfg(unix) targets Closes rust-lang/miri#2421 The standard library has a platform extension trait that lets us get the behavior we want on cfg(unix), so why not use it? I tried this out and it produces the correct behavior in concert with nextest.

Use cargo_metadata in cargo-miri Closes rust-lang#2393 Added `cargo_metadata` to `cargo-miri` and changed metadata from manual parsing to `cargo_metadata` invocations. Thus, removed local `Metadata` struct too. Happy to fix if anything isn't right :)

Fix typo in eval.rs I just found some typos while reading the code

rustup Should work again, now that rust-lang#99984 landed.

avoid double-space in test logging

Add shim for realpath on unix Salvaged from rust-lang/miri#2294 by `@LegNeato`

Co-authored-by: Ralf Jung <post@ralfj.de>

Add `mkstemp` shim for unix

also greatly extend the 'who calls who' comment

split cargo-miri into multiple files also greatly extend the 'who calls who' comment

…ture Some crates are using nightly and failing when mapping these errors, for example <https://miri.saethlin.dev/?crate=remove_dir_all&version=0.7.0>: ``` error: unsupported operation: io error NotADirectory cannot be translated into a raw os error --> /root/.rustup/toolchains/miri/lib/rustlib/src/rust/library/std/src/sys/unix/fs.rs:1203:19 ```

Add additional raw error mappings for the nightly `io_error_more` feature Some crates are using nightly and failing when mapping these errors, for example <https://miri.saethlin.dev/?crate=remove_dir_all&version=0.7.0>: ``` error: unsupported operation: io error NotADirectory cannot be translated into a raw os error --> /root/.rustup/toolchains/miri/lib/rustlib/src/rust/library/std/src/sys/unix/fs.rs:1203:19 ```

avoid strerror_r failure on unknown errnum This is an informative function anyway, so as fallback just return a string with the raw errnum. Avoids panics / interpreter aborts in std on unknown errnum in from_raw_os_error.

fix an ICE in nanosleep()

and a bit of cleanup

add support for env::home_dir and a bit of cleanup

RalfJung · 2022-09-21T17:34:31Z

uh...?

 Documenting miri v0.1.0 (/checkout/src/tools/miri)
error[E0275]: overflow evaluating the requirement `rustc_ast::token::TokenKind: std::marker::Send`
[...]
error: could not document `miri`

oli-obk · 2022-09-21T17:46:44Z

Lol have we ever tried documenting miri?

saethlin · 2022-09-21T17:51:17Z

Yes, when Miri builds its docs have been available on https://doc.rust-lang.org/nightly-rustc

Which is a relatively recent thing, I'm told. The docs did build a week or so ago.

RalfJung · 2022-09-21T17:54:04Z

Yeah the docs currently work (well they did before the most recent toolstate breakage).

We can revert #98764 for now if it helps.

jyn514 · 2022-09-21T18:01:54Z

I expect raising the type length limit like it suggests would work fine too.

RalfJung · 2022-09-22T07:16:38Z

(that's a very old log, the bot just a long time before posting it)

RalfJung · 2022-09-22T07:17:52Z

I expect raising the type length limit like it suggests would work fine too.

But why would this start being a problem now?

@oli-obk should we do a regular submodule update first, both to unblock the distributed Miri and to make sure that this PR is a no-op for the Miri folder (so all build failures must definitely be due to the subtree switch)?

oli-obk · 2022-09-22T07:19:50Z

I just disabled the (newly enabled in this PR) check that docs actually build and we can fix them later

@bors r+

bors · 2022-09-22T07:19:52Z

📌 Commit 2ce88a5 has been approved by oli-obk

It is now in the queue for this repository.

oli-obk · 2022-09-22T07:21:21Z

But yes, if this one doesn't go through, let's do a submodule bump, I have enough experience manually "rebasing" (recreating) this PR now for that not to be annoying

bors · 2022-09-22T09:45:11Z

⌛ Testing commit 2ce88a5 with merge c10f7d7...

bors · 2022-09-22T12:27:45Z

☀️ Test successful - checks-actions
Approved by: oli-obk
Pushing c10f7d7 to master...

rust-timer · 2022-09-22T14:55:20Z

Finished benchmarking commit (c10f7d7): comparison URL.

Overall result: no relevant changes - no action needed

@rustbot label: -perf-regression

Instruction count

This benchmark run did not return any relevant results for this metric.

Max RSS (memory usage)

Results

This is a less reliable metric that may be of interest but was not used to determine the overall result at the top of this comment.

	mean¹	range	count²
Regressions ❌ (primary)	0.6%	[0.6%, 0.6%]	1
Regressions ❌ (secondary)	3.7%	[2.6%, 4.8%]	2
Improvements ✅ (primary)	-	-	0
Improvements ✅ (secondary)	-4.5%	[-6.5%, -2.5%]	2
All ❌✅ (primary)	0.6%	[0.6%, 0.6%]	1

Cycles

Results

This is a less reliable metric that may be of interest but was not used to determine the overall result at the top of this comment.

	mean¹	range	count²
Regressions ❌ (primary)	-	-	0
Regressions ❌ (secondary)	-	-	0
Improvements ✅ (primary)	-2.4%	[-2.7%, -2.2%]	2
Improvements ✅ (secondary)	-7.2%	[-7.2%, -7.2%]	1
All ❌✅ (primary)	-2.4%	[-2.7%, -2.2%]	2

the arithmetic mean of the percent change ↩ ↩²
number of relevant changes ↩ ↩²

jyn514 · 2022-09-22T17:28:19Z

@RalfJung @oli-obk would one of you mind making a follow-up PR to address the comments I had earlier?

oli-obk · 2022-09-22T17:58:41Z

Thanks for the reminder, should get to it tomorrow

RalfJung · 2022-10-14T06:34:22Z

src/ci/docker/host-x86_64/x86_64-gnu-tools/checktools.sh

@@ -23,3 +22,4 @@ cat /tmp/toolstate/toolstates.json
 python3 "$X_PY" test --stage 2 check-tools
 python3 "$X_PY" test --stage 2 src/tools/clippy
 python3 "$X_PY" test --stage 2 src/tools/rustfmt
+python3 "$X_PY" test --stage 2 src/tools/miri


Is it possible that this only gave us back test coverage on Linux, but not on Windows? That's a problem...

bors and others added 30 commits July 27, 2022 00:39

Auto merge of rust-lang#2447 - RalfJung:rustup, r=RalfJung

530abac

rustup

Use cargo_metadata in cargo-miri

38e7bcf

Fix formatting

01a6109

Fix typo in eval.rs

66f6fa6

Auto merge of rust-lang#2452 - Hiroki6:eval-rs-comments-typo, r=RalfJung

8aab6bb

Fix typo in eval.rs I just found some typos while reading the code

rustup

f6badf0

clippy

9ffea91

Auto merge of rust-lang#2453 - RalfJung:rustup, r=RalfJung

7865255

rustup Should work again, now that rust-lang#99984 landed.

Add shim for realpath on unix

b43bede

output realpath as a path, and remove a bogus test

8356f4c

avoid double-space in test logging

87b9075

Auto merge of rust-lang#2458 - RalfJung:less-space, r=RalfJung

25df001

avoid double-space in test logging

Auto merge of rust-lang#2457 - RalfJung:realpath, r=RalfJung

9f4612a

Add shim for realpath on unix Salvaged from rust-lang/miri#2294 by `@LegNeato`

Add mkstemp shim for unix

b29e7b8

Co-authored-by: Ralf Jung <post@ralfj.de>

Auto merge of rust-lang#2346 - LegNeato:mkstemp, r=RalfJung

c24c638

Add `mkstemp` shim for unix

split cargo-miri into multiple files

29e9a8a

also greatly extend the 'who calls who' comment

Auto merge of rust-lang#2459 - RalfJung:cargo-miri-split, r=RalfJung

730a799

split cargo-miri into multiple files also greatly extend the 'who calls who' comment

make errno table syntactically more similar to rustc library code

e1e1f42

avoid strerror_r failure on unknown errnum

6d14a5e

Auto merge of rust-lang#2465 - RalfJung:strerror_r, r=RalfJung

74c5f1b

avoid strerror_r failure on unknown errnum This is an informative function anyway, so as fallback just return a string with the raw errnum. Avoids panics / interpreter aborts in std on unknown errnum in from_raw_os_error.

fix an ICE in nanosleep()

d7875ea

Auto merge of rust-lang#2466 - RalfJung:longsleep, r=RalfJung

4208764

fix an ICE in nanosleep()

add support for env::home_dir

d61e55a

and a bit of cleanup

really, clippy?!?

889c073

Auto merge of rust-lang#2467 - RalfJung:home, r=RalfJung

d6b750e

add support for env::home_dir and a bit of cleanup

bors added the S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. label Sep 21, 2022

This comment was marked as resolved.

Sign in to view

Allow miri docs to keep failing for now

2ce88a5

bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Sep 22, 2022

RalfJung mentioned this pull request Sep 22, 2022

remove Miri rust-lang-nursery/rust-toolstate#26

Merged

bors added the merged-by-bors This PR was explicitly merged by bors. label Sep 22, 2022

bors merged commit c10f7d7 into rust-lang:master Sep 22, 2022

rustbot added this to the 1.66.0 milestone Sep 22, 2022

bors mentioned this pull request Sep 22, 2022

Add tools to sysroot after build. #102010

Closed

3 tasks

oli-obk deleted the miri_subtree branch September 22, 2022 12:40

This comment has been minimized.

Sign in to view

RalfJung reviewed Oct 14, 2022

View reviewed changes

luqmana mentioned this pull request Oct 20, 2022

miri is no longer a submodule but a subtree. rust-lang/rustc-dev-guide#1488

Merged

dilyanpalauzov mentioned this pull request Aug 16, 2024

Include miri in config.example.toml[build] # tools= #129171

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Make miri a subtree instead of a submodule #102028

Make miri a subtree instead of a submodule #102028

oli-obk commented Sep 19, 2022 •

edited by RalfJung

Loading

RalfJung commented Sep 21, 2022

oli-obk commented Sep 21, 2022

saethlin commented Sep 21, 2022 •

edited

Loading

RalfJung commented Sep 21, 2022

jyn514 commented Sep 21, 2022

This comment was marked as resolved.

RalfJung commented Sep 22, 2022

RalfJung commented Sep 22, 2022

oli-obk commented Sep 22, 2022

bors commented Sep 22, 2022

oli-obk commented Sep 22, 2022

bors commented Sep 22, 2022

bors commented Sep 22, 2022

rust-timer commented Sep 22, 2022

jyn514 commented Sep 22, 2022

oli-obk commented Sep 22, 2022

This comment has been minimized.

RalfJung Oct 14, 2022

Make miri a subtree instead of a submodule #102028

Make miri a subtree instead of a submodule #102028

Conversation

oli-obk commented Sep 19, 2022 • edited by RalfJung Loading

RalfJung commented Sep 21, 2022

oli-obk commented Sep 21, 2022

saethlin commented Sep 21, 2022 • edited Loading

RalfJung commented Sep 21, 2022

jyn514 commented Sep 21, 2022

This comment was marked as resolved.

RalfJung commented Sep 22, 2022

RalfJung commented Sep 22, 2022

oli-obk commented Sep 22, 2022

bors commented Sep 22, 2022

oli-obk commented Sep 22, 2022

bors commented Sep 22, 2022

bors commented Sep 22, 2022

rust-timer commented Sep 22, 2022

Overall result: no relevant changes - no action needed

Instruction count

Max RSS (memory usage)

Cycles

Footnotes

jyn514 commented Sep 22, 2022

oli-obk commented Sep 22, 2022

This comment has been minimized.

RalfJung Oct 14, 2022

Choose a reason for hiding this comment

oli-obk commented Sep 19, 2022 •

edited by RalfJung

Loading

saethlin commented Sep 21, 2022 •

edited

Loading