implement thread-spawn #81

oligamiq · 2024-09-20T05:33:16Z

I have implemented a multi-thread accessible file system, which was a prerequisite for wasi-threads.

I haven’t created any tests as I’m not quite sure how to do that.

Features:
Usable across multiple workers
Create a WASIFarm and obtain a WASIFarmRef using .get_ref().
WASIFarmRef can be sent via postMessage.
Be cautious, as classes will also be sent as objects via postMessage.
Multiple WASIFarm can be used to create WASIFarmAnimal.
For Stdio, the WASIRef at the end takes precedence.
Files passed to WASIRef will be treated as if they exist.
WASIFarmAnimal can be accessed using .wasiImport.
Each WASIFarmAnimal can only have one .wasiImport.
When creating a WASIFarmAnimal, you can pass override_fd_maps to change the files accessible by each WASIFarmRef. This allows passing file descriptors to child processes.
For usage examples, please refer to the wasi_threads example.
I ported the example rustc and it works fine!
Issues:
Uses SharedArrayBuffer
Since the communication portion is separated, by understanding the polyfill found at https://github.com/WebReflection/sabayon, it should be possible to implement this in a non-SharedArrayBuffer environment.

Commands used across multiple files cannot yet be used across multiple Farms
I will address this eventually.

File pointers are not process-dependent
This could be achieved by using pread to implement read, but I’m not sure if it's correct to make file pointers process-dependent in the first place.

Parts that haven’t been tested may be unreliable
If 100% is the amount of code I can manage, then this project has reached 120%, so I may not be able to manage everything.

It should be correct, but atomics.waitAsync gives tsc error.

Motivation is running low

oligamiq · 2024-09-27T11:35:49Z

rustc on LLVM confirmed to work properly

oligamiq · 2024-09-27T11:39:20Z

I try running wasm-ld using
https://github.com/YoWASP/clang

bjorn3 · 2024-09-27T11:50:59Z

bjorn3/rust#8 links lld directly into the rustc.wasm binary such that it doesn't need to run an external linker executable.

oligamiq · 2024-09-27T11:52:30Z

Have you reached the level of availability yet?

bjorn3 · 2024-09-27T11:56:24Z

What do you mean?

oligamiq · 2024-09-27T11:58:00Z

Does it work?

bjorn3 · 2024-09-27T12:03:04Z

It works in wasmtime at least:

$ echo 'fn main() { println!("Hello World!"); }' | wasmtime run -Sthreads=y -Spreview2=n --dir tmp::/ --dir dist --env RUST_MIN_STACK=16777216 dist/bin/rustc.wasm - --sysroot dist --target wasm32-wasip1
Linking using LC_ALL="C" VSLANG="1033" "rust-lld" "-flavor" "wasm" [...]
$ wasmtime run tmp/rust_out.wasm
Hello World!

oligamiq · 2024-09-27T12:06:31Z

ok, I'll try it after it's compiled on my server

Change to use the first stdio fd_write can be async enabled move but wasm throw error: memory access out of bounds

oligamiq · 2024-09-27T15:47:41Z

Report on the result of using wasm-ld
Clang exits without any output. It might be caused by #80.
When passing a blob pattern to wasm-ld, it directly passes it to path_open, which causes an issue (should this be addressed?).
When passing proper arguments, it fails midway with a "move but wasm throw error: memory access out of bounds" error.

I should test wasm-ld with wasmtime, but first, I plan to try bjorn3/rust#8.

bjorn3 · 2024-09-27T15:51:29Z

When passing a blob pattern to wasm-ld, it directly passes it to path_open, which causes an issue (should this be addressed?).

On Unix glob patterns are normally handled by the shell, not by the program itself.

oligamiq · 2024-09-27T17:45:04Z

Using bjorn3/rust#8, I successfully compiled and executed it directly on the web!

example: wasi_multi_threads_rustc

bjorn3 · 2024-09-27T18:33:34Z

Great work! Just as a heads up: It will almost certainly take me at least a week before I have time to review this.

oligamiq · 2024-09-27T18:46:10Z

I've written quite a lot of comments, but since there are many files and the structure is quite complex, feel free to ask if you have any concerns!

oligamiq · 2024-09-27T18:46:21Z

I don't understand poll_oneoff at all, but since I'm implementing a file system that can be accessed by multiple workers, it's possible that I can implement poll_oneoff as well.

bjorn3 · 2024-09-27T18:51:20Z

I don't understand poll_oneoff at all

It's similar to epoll on Linux, kqueue on BSD. Basically it allows you to ask a bunch of fd's at once if they have any data ready to ready from or can be written too and I believe also allows setting a timeout. This is what async rust internally uses on wasi. It is mostly useful for streams of data like network sockets or reading from stdin.

bjorn3

This is a lot more code than I expected. I've left a couple of small comments, but I'm no way near done reviewing this. It will likely take at least a couple of weeks before I've fully reviewed this. Would it be possible to have the thread support in a separate package? If so if it works locally for me, I think it could be merged mostly as is with a disclaimer that it is less production ready than the rest of browser_wasi_shim.

src/wasi.ts

src/fs_mem.ts

src/wasi_farm/shared_array_buffer/worker_background/worker_blob.ts

test/run-testsuite.sh

.vscode/settings.json

.gitmodules

oligamiq · 2024-10-04T10:33:28Z

We are dependent on classes such as Fd, File, and Directory. Since I have only modified src/wasi.ts and have isolated everything in the src/wasi_farm directory, I believe it is quite feasible to create a separate npm package that has @bjorn3/browser_wasi_shim as a dependency.

oligamiq · 2024-10-06T09:09:56Z

Regarding the wasm-ification of cargo:
For the time being, I’ve managed to run cargo run, so I’ll pause the work for now. I’ve disabled all git-related functionalities, so I can’t use any libraries. Due to the heavy use of threads, the process has become extremely slow. Therefore, I plan to wait for the implementation of thread pools, fd_filestat_set_times, and further maturity of gitoxide, as well as WASI Preview 2 reaching Tier 2.

I won’t be submitting a pull request because the current state of cargo is too problematic. However, in the process, I did discover a bug related to this pull request. Additionally, I implemented a special function to run wasm within virtual files, and I’d like to add that as well. Would that be acceptable?

bjorn3 · 2024-10-06T09:30:47Z

Additionally, I implemented a special function to run wasm within virtual files, and I’d like to add that as well. Would that be acceptable?

If you add it outside of the wasi_snapshot_preview1 module import, it should be possible for the users of browser_wasi_shim to add it without direct browser_wasi_shim support, right? I'm fine with adding it to one of the examples, but I'm hesitant to add extensions to wasi to the core of browser_wasi_shim.

oligamiq · 2024-10-06T09:50:11Z

Indeed. I'll submit the bug fix only.

oligamiq · 2024-10-06T10:49:05Z

This is a lot more code than I expected. I've left a couple of small comments, but I'm no way near done reviewing this. It will likely take at least a couple of weeks before I've fully reviewed this. Would it be possible to have the thread support in a separate package? If so if it works locally for me, I think it could be merged mostly as is with a disclaimer that it is less production ready than the rest of browser_wasi_shim.

What shall we do?

…itten fd_renumber. add .vscode on .gitignore. rm unused change. Pass fd_map as is to child threads. Change .gitmodules to tabs. reset run-testsuite.sh Minor modifications to the code. patch about null equal undefined on parse JSON.

bjorn3 · 2024-10-06T19:14:24Z

What shall we do?

Please split the thread support into a new package that depends on the main browser_wasi_shim package.

oligamiq · 2024-10-06T19:30:05Z

Would you prefer to go with a monorepo, or should I set up a separate repository?

bjorn3 · 2024-10-06T20:58:32Z

Having it in this repo is fine.

oligamiq · 2024-10-06T23:18:47Z

Ok, I try monorepo.
However, it will take time because I don't know the publishing system of this repository.

oligamiq · 2024-10-06T23:20:22Z

By the way, do you have a good package name?

bjorn3 · 2024-10-07T16:27:28Z

Maybe browser_wasi_shim-threads?

oligamiq · 2024-10-08T18:06:49Z

I splited, so check please.

bjorn3

Couple minor changes and I will merge it.

.gitmodules

threads/README.md

bjorn3 · 2024-11-09T16:46:23Z

threads/README.md

@@ -0,0 +1,21 @@
+# A pure javascript shim for WASI Preview 1 threads
+
+This project is implement threads on browser_wasi_shim


Suggested change

This project is implement threads on browser_wasi_shim

This project implement threads on browser_wasi_shim

nit

bjorn3 · 2024-11-10T10:58:46Z

(did you forget to push?)

Co-authored-by: bjorn3 <17426603+bjorn3@users.noreply.github.com>

oligamiq · 2024-11-10T11:02:18Z

Oh, I see. I couldn’t pull because I resolved it without committing. My apologies and thank you so much!

bjorn3 · 2024-11-10T11:11:14Z

Thanks a lot for working on this!

oligamiq · 2024-11-10T11:15:09Z

Oh, my apologies, I just finished making the corrections, so it was a bit too early to merge.

bjorn3 · 2024-11-10T11:24:27Z

My bad. I fixed the typo you hadn't resolved yet when I merged this PR myself, but for any other changes please open another PR.

oligamiq · 2024-11-10T11:25:43Z

ok.

oligamiq force-pushed the main branch 2 times, most recently from 80f8d74 to 9959142 Compare September 23, 2024 18:31

implement usable across multiple workers WASI

63d184e

oligamiq force-pushed the main branch from 9959142 to 2d924a6 Compare September 23, 2024 18:35

feature wasi thread-spawn

b389c72

oligamiq force-pushed the main branch from 2d924a6 to b389c72 Compare September 27, 2024 11:22

oligamiq changed the title ~~Usable across multiple workers WASI~~ threads-spwan Sep 27, 2024

oligamiq changed the title ~~threads-spwan~~ implement thread-spawn Sep 27, 2024

Using LLVM's wasm-ld

2a0189f

Change to use the first stdio fd_write can be async enabled move but wasm throw error: memory access out of bounds

compile and run on web only!!

c198838

bjorn3 reviewed Oct 4, 2024

View reviewed changes

oligamiq force-pushed the main branch from bf16572 to 6e6347e Compare October 6, 2024 10:48

oligamiq force-pushed the main branch from 6e6347e to 08ffd65 Compare October 6, 2024 10:52

oligamiq force-pushed the main branch 2 times, most recently from 52cf4ec to 51822df Compare October 8, 2024 17:50

split threads feature

3dc520f

oligamiq force-pushed the main branch from 51822df to 3dc520f Compare October 8, 2024 18:03

patch about build on threads feature

5da03cc

bjorn3 approved these changes Nov 9, 2024

View reviewed changes

Update threads/README.md

622e4c0

Co-authored-by: bjorn3 <17426603+bjorn3@users.noreply.github.com>

bjorn3 merged commit 3f705d1 into bjorn3:main Nov 10, 2024
1 check passed

		@@ -0,0 +1,21 @@
		# A pure javascript shim for WASI Preview 1 threads

		This project is implement threads on browser_wasi_shim

	This project is implement threads on browser_wasi_shim
	This project implement threads on browser_wasi_shim

implement thread-spawn #81

implement thread-spawn #81

Conversation

oligamiq commented Sep 20, 2024

oligamiq commented Sep 27, 2024 • edited Loading

oligamiq commented Sep 27, 2024

bjorn3 commented Sep 27, 2024

oligamiq commented Sep 27, 2024

bjorn3 commented Sep 27, 2024

oligamiq commented Sep 27, 2024

bjorn3 commented Sep 27, 2024

oligamiq commented Sep 27, 2024

oligamiq commented Sep 27, 2024

bjorn3 commented Sep 27, 2024

oligamiq commented Sep 27, 2024 • edited Loading

bjorn3 commented Sep 27, 2024

oligamiq commented Sep 27, 2024

oligamiq commented Sep 27, 2024

bjorn3 commented Sep 27, 2024 • edited Loading

bjorn3 left a comment

Choose a reason for hiding this comment

oligamiq commented Oct 4, 2024

oligamiq commented Oct 6, 2024 • edited Loading

bjorn3 commented Oct 6, 2024

oligamiq commented Oct 6, 2024

oligamiq commented Oct 6, 2024

bjorn3 commented Oct 6, 2024

oligamiq commented Oct 6, 2024

bjorn3 commented Oct 6, 2024

oligamiq commented Oct 6, 2024

oligamiq commented Oct 6, 2024

bjorn3 commented Oct 7, 2024

oligamiq commented Oct 8, 2024

bjorn3 left a comment

Choose a reason for hiding this comment

bjorn3 Nov 9, 2024

Choose a reason for hiding this comment

bjorn3 commented Nov 10, 2024

oligamiq commented Nov 10, 2024

bjorn3 commented Nov 10, 2024

oligamiq commented Nov 10, 2024

bjorn3 commented Nov 10, 2024

oligamiq commented Nov 10, 2024

oligamiq commented Sep 27, 2024 •

edited

Loading

oligamiq commented Sep 27, 2024 •

edited

Loading

bjorn3 commented Sep 27, 2024 •

edited

Loading

oligamiq commented Oct 6, 2024 •

edited

Loading