wasm: support new WASI interface

[johanbrandhorst]

Mozilla, Fastly and others have started work on a new standard interface for WASM to interact with system calls, they're calling it WASI, the WebAssembly System Interface.

https://hacks.mozilla.org/2019/03/standardizing-wasi-a-webassembly-system-interface/

I think we should consider supporting this in the Go WASM architecture, possibly under a new OS (GOOS=WASI).

This issue should track the development of such an effort.

Mozilla have invited interested parties to join the #wasi channel on the Mozilla IRC server (https://wiki.mozilla.org/IRC).

[johanbrandhorst]

@neelance @agnivade

[neelance]

I'm already looking into this. Yes, this is likely to become the GOOS/GOARCH wasi/wasm. My current long term plan is to also make js/wasm partially use WASI as soon as a feature can be expressed with WASI without losing any functionality.

[johanbrandhorst]

https://github.com/CraneStation/wasmtime/blob/master/docs/WASI-api.md this looks like a good list to reference.

[neelance]

My WIP for WASI can be found here: https://github.com/neelance/go/tree/wasi

There is no GOOS=wasi yet, use GOARCH=wasm + GOOS=js to target WASI. The fork is thus not compatible with wasm_exec.

[ghost]

This is a very important thing.
Wasi can be a new containerisation approach but running on mobiles, desktops, IOT and servers. It solves is able to solve a fundamental problem of trusting code from someone else with the host deciding what file / network IO the third party code has access to.
Sort of like how android will prompt you when an app wants to access your contacts or file system.

Wasmer git repo has it working for golang, python using a rust wasm / wasi engine currently btw. It's fast too.
https://github.com/wasmerio/go-ext-wasm

It seems that there is not IDL / RPC standard yet for WASI though. I looked but could not find. If anyone knows would be great.

[tiborvass]

@gedw99 you should probably ask on the WASI repo instead. Not sure if WebAssembly/WASI#49 is what you're looking for. Either way WASI is still in active development so it is normal that certain things are lacking.

[philippgille]

It's great to see there's some interest here! How is the progress? I see a recent commit from Dec 26, 2019 and one before that on Mar 29, 2019 in the fork of @neelance.

I recently stumbled on https://webassembly.sh, which drops you into a shell where you can execute WebAssembly binaries that were packaged with the WebAssembly Package Manager (wapm, by the Wasmer guys) and target WASI: https://wapm.io/interface/wasi

There are small command line programs like cowsay and lolcat, but some people seem to try to experiment with bigger things as well (TiDB, but didn't try it out).

So naturally, I'd like to experiment with this new tech and create a WASI-targeting WASM file with Go, upload it to wapm and try it out on webassembly.sh :)

[neelance]

@philippgille I've updated my fork and just successfully ran a Go program on webassembly.sh. Try hello-go.

[superloach]

@neelance can we get a status update on WASI support? What's still missing? Do you need help with anything?

Very excited to get this working - combining the portability of WASI with the strength of Go will open up lots of opportunities. 😃

[3goats]

Any update on this?

[mvdan]

A reminder that this is open source and there are no ETAs, especially when it comes to external contributions such as Richard's. He hasn't posted any update in this thread since March, so that's the latest update. If you want to be notified of updates, subscribe to the thread. Aside from that, the best way to speed up progress is to check out his work, figure out how it works, and try to improve it.

[inliquid]

Open source doesn't mean there is no ETA.

[ianlancetaylor]

@inliquid That is true but I hope that the meaning is reasonably clear. Please be charitable in your understanding of what other people write. Thanks.

[neelance]

See #38248 for what I was working on. No ETA though. Depends on how much time I get to spend on contributing to the Go project.

[inliquid]

That is true but I hope that the meaning is reasonably clear. Please be charitable in your understanding of what other people write. Thanks.

Meaning is clear and it's not correct. At least in a form when someone replies to very reasonable question "this is open source". Open source doesn't mean work is not paid or not compensated in other ways. In most cases it's just one of the business models. So no one should feel like asking for ETA or just a progress is something wrong.

[icholy]

@inliquid the ETA is whenever @neelance feels like it.

[neelance]

Open source doesn't mean work is not paid or not compensated in other ways.

@inliquid Just to be clear: I am not getting compensated for my work on WebAssembly/WASI in any way. I am not even using it at my day job. I just work on it because I believe it should exist.

@icholy The phrasing of your comment is ambiguous, at least to me.

[torch2424]

Hello! 👋

So I noticed that in Go 1.16, Go for OpenBSD is using libc syscalls.

I was curious after #38248 is merged, if WASI libc could be used as an easy(-er) way to bring Go to WASI?

If not, what's missing from wasi-libc that could make this happen? 🤔 😄

[Hywan]

wasmer-go is embedding Wasmer inside Go (already mentionned in this thread). It supports WASI. When using TinyGo to produce Wasm modules, there is no problem. When using Go, you need an adapter to support the Go JS ABI. So far, there are 2 libraries to support it, https://github.com/mattn/gowasmer/ and https://github.com/go-wasm-adapter/go-wasm/, both are assuming the runtime is wasmer-go.

It would be easier if Go could add support for WASI. We have many users reporting they try to run Go inside Go via WebAssembly, or even running Go in other environments (like Python, PHP, C, Rust and so on) via WebAssembly. Obviously we may be able to implement the Go JS ABI for all those hosts, but (i) there is no garantee that the JS ABI is stable and won't break, (ii) it's time consuming, (iii) it's a JS ABI, so it won't fit in all the other hosts, some stuff are deeply tied to the Web.

Thank you for your efforts :-).

[dosgo]

Some TinyGo standard libraries do not support it, so there is still no way to compile golang to WASI at present.

[dankegel]

More of go's standard libraries compile and pass tests with each release of tinygo.
If you need wasi, do try tinygo now and then to see how close it is to meeting your needs.

[backkem]

I wonder if we should avoid focusing on specific interface implementations in the stdlib and instead offer an easy way to support any interface instead. My reasoning:

1. The WASI spec will likely keep evolving, forming a moving target. De-coupling its evolution from the mainline release schedule may be advisable.
2. It gives other runtime providers (embedded, serverless, smart contracts, neural net and cryptography accelerators, plug-in systems, ...) an idiomatic way to support their custom system interfaces. This seems like an integral part of the WebAssembly ecosystem moving forward.
3. By building tooling around the witx interface definition format, the guest interface, including the WASI interface, won't have to be hand-crafted. It can be generated, E.g.: witx-codegen. For reference: an example of the WASI interface in witx.

[superloach]

Perhaps a golang.org/x/wasi package would work for now, until the WASI spec is stable? I feel like relying on outside tooling like witx doesn't fit with Go's goals here.

…

On Thu, 20 Jan 2022, 08:56 Michiel De Backker, ***@***.***> wrote: I wonder if we should avoid focusing on specific interface implementations in the stdlib and instead offer an easy way to support any interface instead. My reasoning: 1. The WASI spec will likely keep evolving <https://github.com/WebAssembly/WASI/blob/main/docs/Proposals.md>, forming a moving target. De-coupling its evolution from the mainline release schedule may be advisable. 2. It gives other runtime providers (embedded, serverless, smart contracts, neural net and cryptography accelerators, plug-in systems, ...) an idiomatic way to support their custom system interfaces. This seems like an integral part of the WebAssembly ecosystem moving forward. 3. By building tooling around the witx <https://github.com/WebAssembly/WASI/blob/main/docs/witx.md> interface definition format, the guest interface, including the WASI interface, won't have to be hand-crafted. It can be generated, E.g.: witx-codegen <https://github.com/jedisct1/witx-codegen>. For reference: an example of the WASI interface in witx <https://github.com/WebAssembly/WASI/blob/main/phases/snapshot/witx/wasi_snapshot_preview1.witx> . — Reply to this email directly, view it on GitHub <#31105 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AHIZZ6BCAYRXC36UFMAMEOTUXAICHANCNFSM4HB6TJAQ> . You are receiving this because you commented.Message ID: ***@***.***>

[backkem]

I tried to make the argument in the go:wasmimport directive ticket. I'm not sure if there would be any other blockers to working on a golang.org/x/wasi package.

I still believe the investment in witx-based code generation would not be wasted. We can write our own generator in Go if we don't want to rely on another community project. It will support the communities that target different host interfaces and it may support future work on things like module linking.

[inkeliz]

I think, instead of supporting WASI, could Golang suport GOOS=none GOARCH=wasm or GOOS=freestanding GOARCH=wasm?

I think that would be the first step to support GOOS=WASI. The difference between GOOS=none and GOOS=js is that would no long require the js glue, which is used on the runtime too (far I remember). That will make possible to implement WASI on top of that, even by third-party packages at first.

However, I think #42372 should be solved first, because exporting function is almost mandatory for WASM.

[johanbrandhorst]

To subscribers wanting an update on the progress of this, the current blocker is the implementation of this (accepted) proposal: #38248. It's a pretty big change so it will need some coordination from the compiler team to get it over the line, I think.

[inkeliz]

Honestly, I don't think #38248 is the current blocker. Instead, #42372 is the real blocker, or both of them.

Currently it's already possible to call imported functions: using CallImport magic assembly code, you can see that here for instance (https://github.com/gioui/gio-x/blob/main/explorer/explorer_js.s, https://github.com/gioui/gio-x/blob/main/explorer/explorer_js.js).

Of course, the "ABI" is odd, and will not match perfectly with the WASM exported functions. So, yes, that should be also change.

However, and for me most important: currently, it's not possible to export one function. So you can't have func Foo(i int) and the wasm-host (such as Wasmer) call Foo. The export feature is mandatory for any WASI and currently that is impossible. There's no equivalent of CallExport or even //go:export. TinyGo have such features (and also supports WASI).

[gcsfred2]

What about upper-case versus lower-case names of functions? Could that native mechanism be used to map to exported and non-exported functions?

…

On Thu, Nov 10, 2022 at 1:40 PM inkeliz ***@***.***> wrote: Honestly, I don't think #38248 <#38248> is the current blocker. Instead, #42372 <#42372> is the real blocker, or both of them. Currently it's already possible to call imported functions: using CallImport magic assembly code, you can see that here for instance ( https://github.com/gioui/gio-x/blob/main/explorer/explorer_js.s, https://github.com/gioui/gio-x/blob/main/explorer/explorer_js.js). Of course, the "ABI" *is odd*, and will not match perfectly with the WASM exported functions. So, yes, that should be also change. However, and for me most important: currently, it's not possible to export one function. So you can't have func Foo(i int) and the wasm-host (such as Wasmer) call Foo. The export feature is mandatory for any WASI and currently that is impossible. There's no equivalent of CallExport or even //go:export. *TinyGo have such features (and also supports WASI).* — Reply to this email directly, view it on GitHub <#31105 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AJMERJMP2PVTARFMT5NZVC3WHU6QPANCNFSM4HB6TJAQ> . You are receiving this because you commented.Message ID: ***@***.***>

[KiYugadgeter]

Now is time to send feedback If you hopes WASI support on Go!!
https://go.dev/blog/survey2023-q1

[johanbrandhorst]

Closing this in favor of #58141, where we will track the effort to implement the accepted proposal 🎉.

[hurae]

For who concerns and clicked into this, golang is supporting wasi preview 1 in version 1.21.

refer to this: https://tip.golang.org/doc/go1.21