Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Support testing of cg_clif in rust's CI #1357

Merged
merged 20 commits into from
Jun 15, 2023
Merged

Support testing of cg_clif in rust's CI #1357

merged 20 commits into from
Jun 15, 2023

Conversation

bjorn3
Copy link
Member

@bjorn3 bjorn3 commented Feb 13, 2023

This will need a change on the rust side too to run ./y.rs prepare --download-dir some/dir (if not building from a source tarball), vendor all deps as necessary and run ./y.rs test --out-dir some/other/dir --download-dir some/dir.

r? @jyn514

Part of #1290

@bjorn3
Copy link
Member Author

bjorn3 commented Feb 13, 2023

@hexagonal-sun I think this will help with building on NixOS too. Setting CARGO, RUSTC and RUSTDOC when building should allow building and running without rustup.

jyn514
jyn514 approved these changes Feb 14, 2023
View reviewed changes
Copy link
Member

@jyn514 jyn514 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks! This broadly looks fine to me - I suspect it might need a couple more changes when you write the test code in bootstrap, but I don't have any concerns :)

@bjorn3
Copy link
Member Author

bjorn3 commented Feb 14, 2023

Yeah, I needed a lot more changes. I got a branch over at https://github.com/bjorn3/rust/tree/test_cg_clif_in_ci. I'm going to call it a day for now though. I will sync the changes to this PR tomorrow and after this PR is merged I will first sync all changes to the rust repo before opening a PR with the actual CI integration.

@bjorn3 bjorn3 force-pushed the build_system_rework6 branch 3 times, most recently from ab3e907 to 0ecaca7 Compare February 18, 2023 18:45
@bjorn3 bjorn3 mentioned this pull request Feb 19, 2023
Merged
@jyn514
Copy link
Member

jyn514 commented Feb 19, 2023

@bjorn3 I've only read the commit descriptions but this looks broadly fine to me still :) let me know if you want a full review.

@bjorn3
Copy link
Member Author

bjorn3 commented Feb 19, 2023

There are a couple of things left to implement for me. After that I would appreciate a full review. I will let you know when it is ready.

@bjorn3 bjorn3 force-pushed the build_system_rework6 branch 3 times, most recently from 490dcbf to f20bffc Compare February 26, 2023 13:49
@bjorn3 bjorn3 mentioned this pull request Mar 21, 2023
Closed
@bjorn3 bjorn3 force-pushed the build_system_rework6 branch from f20bffc to caec60b Compare March 21, 2023 15:50
@bjorn3
Copy link
Member Author

bjorn3 commented Apr 13, 2023

Landed part of the changes directly on master: dfaeab8...4ece6d0

@bjorn3 bjorn3 force-pushed the build_system_rework6 branch from caec60b to d262780 Compare April 13, 2023 13:45
@bjorn3 bjorn3 mentioned this pull request May 9, 2023
Closed
@bjorn3 bjorn3 force-pushed the build_system_rework6 branch 2 times, most recently from 8acf0c2 to c252660 Compare May 14, 2023 12:58
@bjorn3
Copy link
Member Author

bjorn3 commented May 14, 2023 •
edited
Loading

Split out https://github.com/bjorn3/rustc_codegen_cranelift/pull/1373 and landed 8acf0c2 and 316d04b directly on master.

@bjorn3 bjorn3 force-pushed the build_system_rework6 branch from c252660 to eceb0df Compare May 14, 2023 13:00
@bjorn3 bjorn3 force-pushed the build_system_rework6 branch from eceb0df to 08b1ec4 Compare May 22, 2023 17:14
@bjorn3
Copy link
Member Author

bjorn3 commented May 22, 2023

Split out https://github.com/bjorn3/rustc_codegen_cranelift/pull/1374

@bjorn3 bjorn3 force-pushed the build_system_rework6 branch 4 times, most recently from e3fe724 to 9002f8f Compare May 24, 2023 17:35
@bjorn3
Copy link
Member Author

bjorn3 commented May 24, 2023

Landed 5b3bc29 and 134dc33 directly on master.

@bjorn3 bjorn3 force-pushed the build_system_rework6 branch 2 times, most recently from 9ca3e12 to 0f4c8dc Compare May 29, 2023 12:30
@bjorn3 bjorn3 force-pushed the build_system_rework6 branch from 17e8c04 to 4088952 Compare June 5, 2023 18:47
@bjorn3
Copy link
Member Author

bjorn3 commented Jun 5, 2023

https://github.com/bjorn3/rustc_codegen_cranelift/pull/1378 made it possible to test entirely with an LLVM compiled standard library. This avoids the need to vendor the standard library dependencies.

@bjorn3
Copy link
Member Author

bjorn3 commented Jun 5, 2023

@jyn514 This PR is ready for review. The rust side (rust-lang/rust@master...bjorn3:rust:test_cg_clif_in_ci#diff-8479eab02701e686aedb15b567dc8fc31220c6e4efb9565ccc9d662b7fee2214) is still missing vendoring, but that shouldn't be too hard to wire up now. The download dir created by ./y.sh prepare now includes Cargo.lock files to allow vendoring and I'm entirely avoiding building the standard library now, so there is no need to find a way to handle vendoring for the standard library anymore.

@bjorn3 bjorn3 force-pushed the build_system_rework6 branch 2 times, most recently from 49be594 to b4e4611 Compare June 7, 2023 15:58
bjorn3 added 19 commits June 13, 2023 16:39
@bjorn3
Add --download-dir option to specify download dir separate from --out…
ce3f300 
…-dir
@bjorn3
Allow building the build system using cargo
d0b8896 
Rust's build system only handles cargo, not rustc.
@bjorn3
Put patched sources in build/ instead of download/
0e41399
@bjorn3
Reuse existing download in y.sh prepare if fresh
75327f8
@bjorn3
Don't patch in place in apply_patches
2c38eff 
This will make it easier to skip patching if unnecessary in the future
@bjorn3
Lazily patch coretests
fc23a8a
@bjorn3
Rename a couple of build dirs for consistency
b9129c0
@bjorn3
Enable compiler-builtins no-asm feature using --features flag
54eaa53
@bjorn3
Rework standard library building
3baee66
@bjorn3
Lazily patch all test projects
67f9fe6
@bjorn3
Write stdlib workspace Cargo.toml directly in prepare.rs
d3da972
@bjorn3
Ensure everything has a lockfile
8ad9e9f
@bjorn3
Move some profile settings out of the stdlib cargo workspace patch
eb3e8bb
@bjorn3
Disable all incremental compilation for CARGO_BUILD_INCREMENTAL=false
6b9af8c
@bjorn3
Only copy library dir for stdlib
d0ea8bb 
When building as part of rust, the sysroot source dir is symlinked to
the main source dir, which contains the build dir to which we are likely
copying.
@bjorn3
Lazily patch the standard library
bcac222
@bjorn3
Copy Cargo.lock over in ./y.sh prepare
8c1c84d 
This makes it easier for ./x.py to vendor all dependencies
@bjorn3
Allow skipping tests from the commandline
a691b14
@bjorn3
Remove outdated fixme
485d7e1
@bjorn3 bjorn3 force-pushed the build_system_rework6 branch from b4e4611 to 485d7e1 Compare June 13, 2023 16:40
@jyn514
Copy link
Member

jyn514 commented Jun 15, 2023

i'm not sure when i'll get a chance to look at this, sorry. you could ask someone on bootstrap to review or just open a PR directly to rust-lang/rust.

@bjorn3
Copy link
Member Author

bjorn3 commented Jun 15, 2023

I'm going to skip implementing vendoring support on the rust side for now and instead skip the tests that need external crates. I intend to implement it later, but for now even running part of the tests is better than nothing.

@jyn514 no worries. I intend to merge this PR and put up a PR for rust-lang/rust today. Most of the complexity of this PR is vendoring support which isn't relevant for now given the above.

@bjorn3 bjorn3 merged commit f2895f3 into master Jun 15, 2023
@bjorn3 bjorn3 deleted the build_system_rework6 branch June 15, 2023 17:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@jyn514 jyn514 jyn514 approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@bjorn3 @jyn514