Add BuildTargets.md #378
Add BuildTargets.md #378
Conversation
|
Would love to see this added and I am happy to test this. Let me know how/if I can support. |
lukewagner
force-pushed
the
add-build-targets
branch
from
3ae88c0 to
dca900a
Compare
|
@elewis787 Great to hear! Probably the next thing we need for this PR to merge is a concrete implementation just to validate what's here makes sense. In particular, one great proof-of-concept would be extending the wit-component crate to accept any core module targeting |
|
@lukewagner sounds great. A very basic example showing wasip1 and wasip2 and using WIT to generate bindings can be found here https://github.com/elewis787/wasip1-wasip2. The goal is to show that Tinygo ( dev branch supporting wasip1 and wasip2 ) can be used to target a component or a module. Once these are compiled, this example shows that the wasmtime-go bindings ( currently wasip1 support only ) can be used for core modules, and wasmtime-cli ( or rust wasmtime, or jco ) can be used for components. Let me know if this is on the right track for what you would like to see. It may be a slightly different use case. This focuses more on proving that a wit component ( with limitations ) can be a core module per wasip1. Regardless, I will start digging into the wit-component crate. |
|
I am looking into this as well. Does this mean there would be build target support for generating modules (wasip1 only) versus generating components (wasip2) from compilers? I have been testing out WebAssembly modules with JCO and through their But is there a plan to make core modules without WASI P2 imports a build target for things like this? |
|
From a technical perspective almost all components today start as single core modules. This is the case for nearly all compiles for TinyGo, Rust, and C (e.g. LLVM-based toolchains). In all of these cases LLVM emits a single core wasm module which is then fed into
is "yes that's sort of already supported today". I say "sort of" in that the way this typically works is that the original binary is using a mixture of The goal of this proposal is to formalize the core-module-to-component step of tooling. It makes the core module artifact (which today becomes a component) a first-class artifact. It additionally defines what it means to import/export various names from a core module and this proposal means that most core modules emitted by compilers today are not valid to become a component because they import `wasi_snapshot_preview1. This means that one big chunk in implementing this proposal is going to be updating language standard libraries to exclusively use WASIp2 APIs instead of a mixture of WASIp1 and WASIp2. This would likely start with wasi-libc and start bubbling up from there for example. |
|
Thanks @alexcrichton! This is helpful and clear. as a naive example, does this demonstrate what you are discussing?
If so, is a byproduct of this a core module that is produced but using wasip2 imports before componentization? This means that in the example I provided the obvious difference is that the module produced does not use wasip2 syntax. Are there any plans for taking a component to a module to be backward compatible with wasip1? I know wasip1 does not include some of the wasip2 system interfaces and many have changed but I have been able to side step some of the tooling support by implementing what is needed by the core module. This is still useful. Thanks for walking through regardless! |
|
Indeed yeah the differences will come down to imports. The first one you listed is WASIp1, and the second one is WASIp2-as-defined-by- (import "cm32p2|ns:wasi:filesystem/types@0.2" "[method]descriptor.write" (func (param ...)))Notably the prefix
Yes, and this document is describing the exact name of component imports/exports to be "componentized", optionally, later on. This document is a formalization of the names used today which is intended to become standard (as opposed to whatever was implemented to begin with)
Not currently, but that's also sort of out of the scope of this PR |
|
@alexcrichton, thanks for explaining this. @jeff1010322 and I are both in the jco exploration phase. As mentioned, we are playing around with the transpile flags to produce core modules. As a next step, do you feel it is helpful to look into the generation of the core module and take a stab at outlining the behavior listed here? As far as backward compatibility, do you know of a place to discuss/contribute to this? After using WIT and a few various tools, it seems like there may be some options/benefits in the bindings generated. Candidly, I am still getting up to speed with WASIp2 and may be incorrect. |
|
Oh no worries! Always great to have more folks help and happy to help out where I can! I think the first step is what you're already doing here which is to make sure you feel comfortable with the current process of how a component is created. The high-level overview of that is that source code is written with the help of The next step is going to be working backwards in this pipeline from the end back to the front. For example if you were to change the core wasm module today nothing would be able to turn it into a component. In that sense you'll want to do as @lukewagner suggested which is to start with Once that's all implemented the next step is to probably work on From there the next hard part will be to flesh out language standard libraries, but that's probably best tackled once we're closer to that. |
|
Great overview. Thanks again! I'll start looking into this. I am still getting up to date on the various tools but I believe I understand what's required now. |
This commit implements recognition of the `_initialize` function from WASIp1 in the componentization process of `wasm-tools component new`. This additionally corresponds to the same function in the proposed [BuildTargets.md](WebAssembly/component-model#378). This is implemented by having a small core wasm module which is just an import and a `start` section get instantiated at the end of a component to run `_initialize` before all other exports.
This commit implements recognition of the `_initialize` function from WASIp1 in the componentization process of `wasm-tools component new`. This additionally corresponds to the same function in the proposed [BuildTargets.md](WebAssembly/component-model#378). This is implemented by having a small core wasm module which is just an import and a `start` section get instantiated at the end of a component to run `_initialize` before all other exports.
lukewagner
force-pushed
the
main
branch
3 times, most recently
from
824fdc5 to
74bd278
Compare
This commit decouples the string encodings listed for imports/exports from their core wasm names to instead being registered with WIT-level constructs instead. Previously the parsing phase of a module would register a string encoding for core wasm import/export names but this subverted the logic of validation where detection of how exactly an import lines up with WIT-level items is determined. The goal of this commit is to decouple this relation. Worlds are encoding into custom sections with a known string encoding for all imports/exports of that world. This can possibly differ for different parts of an application to theoretically enable one interface to be imported with UTF-8 and another with UTF-16. This means that encodings are tracked per-import/export rather than per-world. Previously this process would assume that there is a single name for an import's/export's encoding but with new detection and names coming down the line this is no longer going to be the case. For example with the new names in WebAssembly/component-model#378 there are new names to be supported meaning that there's not one single name to register encodings with. To help bridge this gap the abstraction here is changed to where metadata for a module records string encodings on a WIT level, for example per WIT import/export, instead of per core wasm import/export. Then during encoding of a component the WIT level constructs are matched up instead of the core names to determine the string encoding in the lift/lower operation. The end goal is that the connection between core wasm names and WIT names continues to be decoupled where validation is the only location concerned about this.
This commit adds support for WebAssembly/component-model#378 to `wit-component`. Notably a new set of alternative names are registered and recognized during the module-to-component translation process. Support for the previous set of names are all preserved and will continue to be supported for some time. The new names are, for now, recognized in parallel to the old names. This involved some refactoring to the validation part of `wit-component` and further encapsulation of various names to one small location instead of a shared location for everywhere else to use as well.
|
I have an implementation of this at bytecodealliance/wasm-tools#1828 which I believe captures everything here. |
|
Nice! Let me know once things seem far enough along that you feel confident that what's in this PR can match reality. |
…odel (#1828) * Use `ExportMap` for naming component exports Use the map's metadata to determine what the core wasm name is for each export instead of recalculating it in the encoder which would duplicate work done in validation. * Decouple import/export encodings from core names This commit decouples the string encodings listed for imports/exports from their core wasm names to instead being registered with WIT-level constructs instead. Previously the parsing phase of a module would register a string encoding for core wasm import/export names but this subverted the logic of validation where detection of how exactly an import lines up with WIT-level items is determined. The goal of this commit is to decouple this relation. Worlds are encoding into custom sections with a known string encoding for all imports/exports of that world. This can possibly differ for different parts of an application to theoretically enable one interface to be imported with UTF-8 and another with UTF-16. This means that encodings are tracked per-import/export rather than per-world. Previously this process would assume that there is a single name for an import's/export's encoding but with new detection and names coming down the line this is no longer going to be the case. For example with the new names in WebAssembly/component-model#378 there are new names to be supported meaning that there's not one single name to register encodings with. To help bridge this gap the abstraction here is changed to where metadata for a module records string encodings on a WIT level, for example per WIT import/export, instead of per core wasm import/export. Then during encoding of a component the WIT level constructs are matched up instead of the core names to determine the string encoding in the lift/lower operation. The end goal is that the connection between core wasm names and WIT names continues to be decoupled where validation is the only location concerned about this. * Remove core wasm name guess in adapter GC This commit removes the need for the GC pass on the adapter module to guess what core wasm export names are needed for WIT. Previously it was assumed that certain exports would have exact core wasm names but that's going to change soon so this refactoring is empowering these future changes. The GC pass for adapters is restructured to run validation over the non-GC'd adapter first. This validation pass will identify WIT export functions and such and then this information is used to determine the set of live exports. These live exports are then used to perform a GC pass, and then afterwards the validation pass is run a second time to recalculate information with possibly-removed imports. * Support the new name mangling scheme for components This commit adds support for WebAssembly/component-model#378 to `wit-component`. Notably a new set of alternative names are registered and recognized during the module-to-component translation process. Support for the previous set of names are all preserved and will continue to be supported for some time. The new names are, for now, recognized in parallel to the old names. This involved some refactoring to the validation part of `wit-component` and further encapsulation of various names to one small location instead of a shared location for everywhere else to use as well. * Update `embed --dummy` with new ABI names This commit updates the `wasm-tools component embed` subcommand, specifically the `--dummy` flag. This flag now uses the new "standard32" names for the core module that is generated. Additionally a new `--dummy-names $FOO` option has been added to enable generating the old names as well as the new names. Utilities have also been added to `Resolve` for bindings generators to avoid hardcoding ABI names and instead use the add categories of imports/exports to name items. * Add a flag to require the new mangling scheme This commit adds a new `--reject-legacy-names` flag to the `wasm-tools component new` subcommand which can be used to disable support for the legacy naming scheme. This is intended to help with testing out the new naming scheme for tools and to help evaluate in the future if it's theoretically possible to remove support for the old naming scheme. * Fix tests * Update some test expectations
This PR adds BuildTargets.md to define this new concept of "build targets" as presented in both CG-06 and WASI-06-12, which itself was a revision of the earlier "
wasit2" idea that came up in WASI/#595.Currently, BuildTargets.md only defines one build target,
wasm32, but with the intention of adding more in the future (e.g.,wasm64andwasmgc). See BuildTargets.md in this PR for more details.The goal, mentioned at the end of BuildTargets.md, is that
wasi-sdkand various other currently-component-producing toolchains would be able to emit core modules that matchedwasm32as defined here (with "componentification" being merely a final linker option of whether to runwasm-ldorwasm-component-ld). I plan to leave this open until we can get some implementation feedback to test this out.