Compile rustc+LLVM to wasm #6
Conversation
|
Thanks a lot for this PR! I'm currently at a conference, so I probably won't get to reviewing for a couple of days. If I don't review it within a week or so feel free to ping me. |
There was a problem hiding this comment.
Sorry for the late reply. I've rebased the wasm changes and created the new compile_rustc_for_wasm16 branch for this. Would you mind rebasing and changing the target of this PR to that branch? You can use the same edit button that allows changing the PR title and then clicking on the button next to "oligamiq wants to merge 86 commits into". Also squashing your commits into either a single commit or a logical division of commits would be appreciated.
walker.py
Outdated
There was a problem hiding this comment.
It searches for symbols to identify .c files and other files that use symbols that cannot be used in WASM. It was useful for debugging, but can be eliminated.
| .define("CMAKE_CXX_COMPILER_TARGET", &wasi_target_llvm) | ||
| .define("CMAKE_C_FLAGS", format!("{wasi_sysroot} {wasi_cflags_llvm}")) | ||
| .define("CMAKE_CXX_FLAGS", format!("{wasi_sysroot} {wasi_cflags_llvm}")) | ||
| .define("CMAKE_EXE_LINKER_FLAGS", wasi_ldflags_llvm) |
There was a problem hiding this comment.
Isn't there a cmake toolchain file in the wasi-sdk that sets all these flags as necessary?
wrapper_linker_clang.sh
Outdated
| shift 2 | ||
| continue | ||
| fi | ||
| if [[ $1 == "-Wl,--max-memory=1073741824" ]]; then |
There was a problem hiding this comment.
Should this just remove the -Wl, prefix? I was genuinely running out of memory without this argument. Wasm shared memories need to declare a maximum size and lld gives a very small size by default.
oligamiq
force-pushed
the
compile_rustc_for_wasm15_dev
branch
4 times, most recently
from
73bc239 to
881d67f
Compare
oligamiq
changed the base branch from
compile_rustc_for_wasm15
to
compile_rustc_for_wasm16
|
Since I was assuming version 15, there was a conflict. |
|
You can drop the first three commits that are currently in this PR when rebasing on top of compile_rustc_for_wasm16. |
|
I've shut down the work server, so I'll do it tomorrow. I'll give it a try. |
oligamiq
force-pushed
the
compile_rustc_for_wasm15_dev
branch
from
881d67f to
3fe3834
Compare
|
Looks like the first commit is still there. If you want I could do the rebase for you and if you enable the github option for this have me push it directly to your PR branch. |
|
What is the 'first commit'? |
|
I'm referring to 295aca5. This commit originates from the compile_rustc_for_wasm15 branch of mine despite this PR now targetting the compile_rustc_for_wasm16 branch. |
|
If we lose it, won't it cause the build to possibly fail or fail to compile on the cranelift backend? |
|
Sorry. It makes me crazy because it already exists. I'll turn it off. Add Message |
oligamiq
force-pushed
the
compile_rustc_for_wasm15_dev
branch
2 times, most recently
from
d67fd0c to
1e6418a
Compare
|
I believe the work is complete. |
compiler/rustc_target/src/spec/targets/wasm32_wasip1_threads.rs
Outdated
Show resolved
Hide resolved
|
It seems like it still isn't rebased on top of the compile_rustc_for_wasm16 branch. I've got it rebased locally and am currently building it to try it out. If you enable the option "Allow edits and access to secrets by maintainers" as explained by https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/allowing-changes-to-a-pull-request-branch-created-from-a-fork I can push my rebased version to this PR. |
|
ok, understand. You want to create a new branch from wasm16 and cherry-pick it. I've given you the permissions for now, so please take care of it this time. I thought github would do it automatically... |
bjorn3
force-pushed
the
compile_rustc_for_wasm15_dev
branch
from
578bd85 to
d77d78b
Compare
|
Pushed
I think github preserves your choice, so if you ever disabled it, it will be kept disabled for all future PR's by default. |
|
No, I thought I could just apply the changes to the commit in a nice way regardless of the top of branch. |
|
There were some conflicts between the top of the branch and your commits, so it wouldn't have been possible to merge until those conflicts were resolved. I resolved them for you while rebasing. |
|
I'm getting the following locally when building LLVM: Edit: Messed up the llvm-project submodule while I did the rebase. |
|
I believe there was a .define("HAVE_DLOPEN", ""); followed by a .define to turn it OFF. I'll go look for it. I have restored it, though, I understand, as I also had trouble with submodule changes. |
|
about [Fix linking stdc++ on wasi] config.llvm.toml: src/bootstrap/src/core/build_steps/compile.rs: |
|
In this case the |
bjorn3
force-pushed
the
compile_rustc_for_wasm15_dev
branch
from
3ab3533 to
ed82c66
Compare
bjorn3
force-pushed
the
compile_rustc_for_wasm15_dev
branch
from
ed82c66 to
ee85060
Compare
This is really slow when producing a 156MB rustc.wasm output.
|
I finally managed to get it to work locally. It does deadlock most of the time when running in wasmtime however unless I use |
|
really? It should have worked without any issues, but I will also test it with the copied version to ensure it works. Add: |
|
Updating Wasmtime seems to have fixed the deadlocks. |
|
oh... |
|
Checked that this PR doesn't break compiling cg_clif for single threaded wasm32-wasip1. I've spent enough time on this for today. I'm going to take another look at the changes tomorrow to see if anything is unnecessary or can be improved. After that I will merge this PR. Thanks again for working on this! |
|
This should be updated to use the https://github.com/YoWASP/llvm-project/releases/tag/rustc-2024-09-24-do-not-remove tag for the llvm-project submodule to ensure that it remains possible to build in the future. |
|
I think LLVM_ENABLE_THREADS=OFF was broken |
|
Indeed. That commit was just a couple of changes that are necessary anyway to compile for wasm32-wasip1 once LLVM_ENABLE_THREADS=OFF gets fixed. |
|
Also, RustCompiler have code to check if LLVM has multithreaded. |
…raheemdev
Optimize `Box::default` and `Arc::default` to construct more types in place
Both the `Arc` and `Box` `Default` impls currently call `T::default()` before allocating, and then moving the resulting `T` into the allocation.
Most `Default` impls are trivial, which should in theory allow
LLVM to construct `T: Default` directly in the `Box` allocation when calling
`<Box<T>>::default()`.
However, the allocation may fail, which necessitates calling `T`'s destructor if it has one.
If the destructor is non-trivial, then LLVM has a hard time proving that it's
sound to elide, which makes it construct `T` on the stack first, and then copy it into the allocation.
Change both of these impls to allocate first, and then call `T::default` into the uninitialized allocation, so that LLVM doesn't have to prove that it's sound to elide the destructor/initial stack copy.
For example, given the following Rust code:
```rust
#[derive(Default, Clone)]
struct Foo {
x: Vec<u8>,
z: String,
y: Vec<u8>,
}
#[no_mangle]
pub fn src() -> Box<Foo> {
Box::default()
}
```
<details open>
<summary>Before this PR:</summary>
```llvm
`@__rust_no_alloc_shim_is_unstable` = external global i8
; drop_in_place() generated in case the allocation fails
; core::ptr::drop_in_place<playground::Foo>
; Function Attrs: nounwind nonlazybind uwtable
define internal fastcc void `@"_ZN4core3ptr36drop_in_place$LT$playground..Foo$GT$17hff376aece491233bE"(ptr` noalias nocapture noundef readonly align 8 dereferenceable(72) %_1) unnamed_addr #0 personality ptr `@rust_eh_personality` {
start:
%_1.val = load i64, ptr %_1, align 8
%0 = icmp eq i64 %_1.val, 0
br i1 %0, label %bb6, label %"_ZN63_$LT$alloc..alloc..Global$u20$as$u20$core..alloc..Allocator$GT$10deallocate17heaa87468709346b1E.exit.i.i.i4.i"
"_ZN63_$LT$alloc..alloc..Global$u20$as$u20$core..alloc..Allocator$GT$10deallocate17heaa87468709346b1E.exit.i.i.i4.i": ; preds = %start
%1 = getelementptr inbounds i8, ptr %_1, i64 8
%_1.val6 = load ptr, ptr %1, align 8, !nonnull !3, !noundef !3
tail call void `@__rust_dealloc(ptr` noundef nonnull %_1.val6, i64 noundef %_1.val, i64 noundef 1) #8
br label %bb6
bb6: ; preds = %"_ZN63_$LT$alloc..alloc..Global$u20$as$u20$core..alloc..Allocator$GT$10deallocate17heaa87468709346b1E.exit.i.i.i4.i", %start
%2 = getelementptr inbounds i8, ptr %_1, i64 24
%.val9 = load i64, ptr %2, align 8
%3 = icmp eq i64 %.val9, 0
br i1 %3, label %bb5, label %"_ZN63_$LT$alloc..alloc..Global$u20$as$u20$core..alloc..Allocator$GT$10deallocate17heaa87468709346b1E.exit.i.i.i4.i.i11"
"_ZN63_$LT$alloc..alloc..Global$u20$as$u20$core..alloc..Allocator$GT$10deallocate17heaa87468709346b1E.exit.i.i.i4.i.i11": ; preds = %bb6
%4 = getelementptr inbounds i8, ptr %_1, i64 32
%.val10 = load ptr, ptr %4, align 8, !nonnull !3, !noundef !3
tail call void `@__rust_dealloc(ptr` noundef nonnull %.val10, i64 noundef %.val9, i64 noundef 1) #8
br label %bb5
bb5: ; preds = %"_ZN63_$LT$alloc..alloc..Global$u20$as$u20$core..alloc..Allocator$GT$10deallocate17heaa87468709346b1E.exit.i.i.i4.i.i11", %bb6
%5 = getelementptr inbounds i8, ptr %_1, i64 48
%.val4 = load i64, ptr %5, align 8
%6 = icmp eq i64 %.val4, 0
br i1 %6, label %"_ZN4core3ptr46drop_in_place$LT$alloc..vec..Vec$LT$u8$GT$$GT$17hb5ca95423e113cf7E.exit16", label %"_ZN63_$LT$alloc..alloc..Global$u20$as$u20$core..alloc..Allocator$GT$10deallocate17heaa87468709346b1E.exit.i.i.i4.i15"
"_ZN63_$LT$alloc..alloc..Global$u20$as$u20$core..alloc..Allocator$GT$10deallocate17heaa87468709346b1E.exit.i.i.i4.i15": ; preds = %bb5
%7 = getelementptr inbounds i8, ptr %_1, i64 56
%.val5 = load ptr, ptr %7, align 8, !nonnull !3, !noundef !3
tail call void `@__rust_dealloc(ptr` noundef nonnull %.val5, i64 noundef %.val4, i64 noundef 1) #8
br label %"_ZN4core3ptr46drop_in_place$LT$alloc..vec..Vec$LT$u8$GT$$GT$17hb5ca95423e113cf7E.exit16"
"_ZN4core3ptr46drop_in_place$LT$alloc..vec..Vec$LT$u8$GT$$GT$17hb5ca95423e113cf7E.exit16": ; preds = %bb5, %"_ZN63_$LT$alloc..alloc..Global$u20$as$u20$core..alloc..Allocator$GT$10deallocate17heaa87468709346b1E.exit.i.i.i4.i15"
ret void
}
; Function Attrs: nonlazybind uwtable
define noalias noundef nonnull align 8 ptr `@src()` unnamed_addr #1 personality ptr `@rust_eh_personality` {
start:
; alloca to place `Foo` in.
%_1 = alloca [72 x i8], align 8
call void `@llvm.lifetime.start.p0(i64` 72, ptr nonnull %_1)
store i64 0, ptr %_1, align 8
%_2.sroa.4.0._1.sroa_idx = getelementptr inbounds i8, ptr %_1, i64 8
store ptr inttoptr (i64 1 to ptr), ptr %_2.sroa.4.0._1.sroa_idx, align 8
%_2.sroa.5.0._1.sroa_idx = getelementptr inbounds i8, ptr %_1, i64 16
%_3.sroa.4.0..sroa_idx = getelementptr inbounds i8, ptr %_1, i64 32
call void `@llvm.memset.p0.i64(ptr` noundef nonnull align 8 dereferenceable(16) %_2.sroa.5.0._1.sroa_idx, i8 0, i64 16, i1 false)
store ptr inttoptr (i64 1 to ptr), ptr %_3.sroa.4.0..sroa_idx, align 8
%_3.sroa.5.0..sroa_idx = getelementptr inbounds i8, ptr %_1, i64 40
%_4.sroa.4.0..sroa_idx = getelementptr inbounds i8, ptr %_1, i64 56
call void `@llvm.memset.p0.i64(ptr` noundef nonnull align 8 dereferenceable(16) %_3.sroa.5.0..sroa_idx, i8 0, i64 16, i1 false)
store ptr inttoptr (i64 1 to ptr), ptr %_4.sroa.4.0..sroa_idx, align 8
%_4.sroa.5.0..sroa_idx = getelementptr inbounds i8, ptr %_1, i64 64
store i64 0, ptr %_4.sroa.5.0..sroa_idx, align 8
%0 = load volatile i8, ptr `@__rust_no_alloc_shim_is_unstable,` align 1, !noalias !4
%_0.i.i.i = tail call noalias noundef align 8 dereferenceable_or_null(72) ptr `@__rust_alloc(i64` noundef 72, i64 noundef 8) #8, !noalias !4
%1 = icmp eq ptr %_0.i.i.i, null
br i1 %1, label %bb2.i, label %"_ZN5alloc5boxed12Box$LT$T$GT$3new17h0864de14f863a27aE.exit"
bb2.i: ; preds = %start
; invoke alloc::alloc::handle_alloc_error
invoke void `@_ZN5alloc5alloc18handle_alloc_error17h98142d0d8d74161bE(i64` noundef 8, i64 noundef 72) rust-lang#9
to label %.noexc unwind label %cleanup.i
.noexc: ; preds = %bb2.i
unreachable
cleanup.i: ; preds = %bb2.i
%2 = landingpad { ptr, i32 }
cleanup
; call core::ptr::drop_in_place<playground::Foo>
call fastcc void `@"_ZN4core3ptr36drop_in_place$LT$playground..Foo$GT$17hff376aece491233bE"(ptr` noalias noundef nonnull align 8 dereferenceable(72) %_1) rust-lang#10
resume { ptr, i32 } %2
"_ZN5alloc5boxed12Box$LT$T$GT$3new17h0864de14f863a27aE.exit": ; preds = %start
; Copy from stack to heap if allocation is successful
call void `@llvm.memcpy.p0.p0.i64(ptr` noundef nonnull align 8 dereferenceable(72) %_0.i.i.i, ptr noundef nonnull align 8 dereferenceable(72) %_1, i64 72, i1 false)
call void `@llvm.lifetime.end.p0(i64` 72, ptr nonnull %_1)
ret ptr %_0.i.i.i
}
```
</details>
<details>
<summary>After this PR</summary>
```llvm
; Notice how there's no `drop_in_place()` generated as well
define noalias noundef nonnull align 8 ptr `@src()` unnamed_addr #0 personality ptr `@rust_eh_personality` {
start:
; no stack allocation
%0 = load volatile i8, ptr `@__rust_no_alloc_shim_is_unstable,` align 1
%_0.i.i.i.i.i = tail call noalias noundef align 8 dereferenceable_or_null(72) ptr `@__rust_alloc(i64` noundef 72, i64 noundef 8) #5
%1 = icmp eq ptr %_0.i.i.i.i.i, null
br i1 %1, label %bb3.i, label %"_ZN5alloc5boxed16Box$LT$T$C$A$GT$13new_uninit_in17h80d6355ef4b73ea3E.exit"
bb3.i: ; preds = %start
; call alloc::alloc::handle_alloc_error
tail call void `@_ZN5alloc5alloc18handle_alloc_error17h98142d0d8d74161bE(i64` noundef 8, i64 noundef 72) #6
unreachable
"_ZN5alloc5boxed16Box$LT$T$C$A$GT$13new_uninit_in17h80d6355ef4b73ea3E.exit": ; preds = %start
; construct `Foo` directly into the allocation if successful
store i64 0, ptr %_0.i.i.i.i.i, align 8
%_8.sroa.4.0._1.sroa_idx = getelementptr inbounds i8, ptr %_0.i.i.i.i.i, i64 8
store ptr inttoptr (i64 1 to ptr), ptr %_8.sroa.4.0._1.sroa_idx, align 8
%_8.sroa.5.0._1.sroa_idx = getelementptr inbounds i8, ptr %_0.i.i.i.i.i, i64 16
%_8.sroa.7.0._1.sroa_idx = getelementptr inbounds i8, ptr %_0.i.i.i.i.i, i64 32
tail call void `@llvm.memset.p0.i64(ptr` noundef nonnull align 8 dereferenceable(16) %_8.sroa.5.0._1.sroa_idx, i8 0, i64 16, i1 false)
store ptr inttoptr (i64 1 to ptr), ptr %_8.sroa.7.0._1.sroa_idx, align 8
%_8.sroa.8.0._1.sroa_idx = getelementptr inbounds i8, ptr %_0.i.i.i.i.i, i64 40
%_8.sroa.10.0._1.sroa_idx = getelementptr inbounds i8, ptr %_0.i.i.i.i.i, i64 56
tail call void `@llvm.memset.p0.i64(ptr` noundef nonnull align 8 dereferenceable(16) %_8.sroa.8.0._1.sroa_idx, i8 0, i64 16, i1 false)
store ptr inttoptr (i64 1 to ptr), ptr %_8.sroa.10.0._1.sroa_idx, align 8
%_8.sroa.11.0._1.sroa_idx = getelementptr inbounds i8, ptr %_0.i.i.i.i.i, i64 64
store i64 0, ptr %_8.sroa.11.0._1.sroa_idx, align 8
ret ptr %_0.i.i.i.i.i
}
```
</details>
LLVM backend is now available.
It works by changing config.llvm.toml to config.toml.
This allows rustc to compile against wasm32-wasip1-threads.
https://github.com/YoWASP/clang was used as a reference.
Since wasm-ld can also be used on wasi by that project, All that remains is for browser_wasi_shim to support threading, and it should be possible to compile on the browser.
cargo is going to create a project that wraps this project like the project above, which together with rustc will generate one wasm as a driver like YoWASP/clang project.
Problem is that llvm is not patched by rust and the version of llvm is higher than rust. compiler/rustc_target/src/target_features.rs has been changed because of that.
Also, the cc library has been raised to a version that supports building to wasi.
In addition, I upgraded the wasi-sdk version.
wasm and sysroot are available at https://github.com/oligamiq/rust_wasm/
so please use them if you want to try them.
Please forgive the messy commit.