JSCT Initial Migration #1
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This commit is the start of a refactoring which will move from each code
generator having one monolithic test into many separate sub-tests. This
should enable smaller and more focused tests which benefits bringing new
bindings generators online (can get some tests working before others)
and also allows writing tests for some generators but not others.
The way that this is currently architected is that there's a new
`tests/runtime/*` directory in which a test-per-directory lives. For
each tests `tests/runtime/foo` there will be a few conventions:
* `tests/runtime/foo/imports.witx` - these are what hosts must
implement and what the wasm can import from the host. This file is
required to exist.
* `tests/runtime/foo/exports.witx` - this is what wasms must implement
and what hosts can call. This file must exist.
* `tests/runtime/foo/host.*` - these are separate host implementations
to execute the wasm code. The extension of the file indicates which
generator it's used for (e.g. right now `host.{ts,rs,py}` are
supported). If `host.py` doesn't exist for the test `foo` then testing
the Python code generator will skip the test `foo`. Otherwise it's
expected to work. At least one `host.*` file is required to work, but
there's no need to have a file-per-host.
* `tests/runtime/foo/wasm.*` - these are separate guest compiled-to-wasm
implementations. Currently only `wasm.{c,rs}` are supported, but the
intent is to support `wasm.js` for #71 as well. Not all languages need
to have an implementation here either, they'll be skipped if omitted.
The general goal is that adding a test for some witx functionality
should be "drop a few files in `tests/runtime/your_new_test` and you're
good to go. Over time more generators can add more tests if it becomes
necessary, and generator-specific tests can also be added to this
directory such as the one test exercising JS instantiation added here
too.
Overall I think this turned out quite well. The biggest downside is that
Rust-compiled-to-wasm needs to list explicitly each test that's being
compiled to wasm, which is a bit of a bummer.
Starts the extraction process from the monolithic tests that already exist into smaller tests.
There was an accidentaly `.to_snake_case()` when it should have been using the raw name from the witx. Closes #76
This commit is the initial steps towards supporting and implementing `async` function as-specified in `*.witx` itself. The goal of this is to implement the callback-based ABI of async functions and streams as presented in WASI subgroup meetings. This commit itself only implements async functions for the compiled-to-Rust guest and a JS host. The next PR is to implement async functions for a Wasmtime host, followed by an implementation of streams afterwards. Included here is: * Updates to the `*.witx` syntax and parsing. * Assertions in code generators that don't support async that async isn't used. * An implementation of the callback ABI for `async` functions for compiled-to-wasm Rust modules. This includes a simple "executor" to track the state of async imports/exports. Everything is only exposed with `async` functions in Rust, no callbacks are part of the interface of the generated bindings. * An implementation of the callback ABI for JS hosts. This is exposed with `async` functions and only uses callbacks under-the-hood. This change had some significant ramifications on the generated bindings for JS hosts. Namely both imports and exports needed to access the same state of callbacks so they are no longer generated separately in isolation but rather together in the same manner as the spidermonkey bindings. I expect that most generators will migrate to this pattern in the future. Work was done internally to deduplicate intrinsics across bindings modules, namely the promise-related intrinsics. The testing so far isn't super strenuous, but a major thing I made sure to implement is that when an asynchronous wasm function is called and later some asynchronous import throws an exception that should carry the exception to the original wasm export's promise in JS. This is intended to map to basically what `async` and `await` feel like in JS normally. Otherwise though there's codegen tests for flavorful types being used in parameters/results for async functions (and this test is ignored in most generators at the moment) and runtime tests for some concurrency.
* feat: initial work to make nullable fields use option `?` * fix: unwrap nullable type and update test to use `?` * fix: missing semicolon * fix: refactor to return option when unwrapping nullables * feat: refactor nullable methods to iface struct * fix: fmt forgot to run it in root * chore: refactor nullable methods back to js impl
I today started looking at updating this project to the new draft of the canonical ABI specified at WebAssembly/component-model#23 and while there's quite a few changes that need to happen the first thing that struck me was that maintaining all of the support for old-style witx files is pretty onerous. Especially the ABI-calculating code has lots of special cases for various witx constructs and the Preview1 ABI, and I wasn't really sure how to update this in many situations. Overall the original purpose of the witx support was to prove out that it's possible to support both the old witx abi and the new canonical ABI in the same generator. The canonical ABI has changed significantly in the meantime however and this doesn't necessarily make sense to do any more. I think it would be best now to reevaluate at the point when WASI is ready to switch to the component model what to do with the old witx support. I no longer think that "build it in here" is the obvious option. As this diff shows there's quite a bit of weight to carry the old witx abis as this commit clocks in at nearly 7000 lines removed. The specifics being dropped here are: * Parsing support for `*.witx` * Support for `Pointer` and `ConstPointer` * Support for `WitxInstruction` * Support for push/pull buffers The push/pull buffer feature was never actually fully implemented, even for Rust hosts they only kind-of worked. Most other generators never even implemented support for them. Additionally support for other `*.witx` constructs was somewhat spotty at best with very few tests. My hope is that there are no existing users of this support. If there are then I think it's best to re-evaluate how best to solve the scenario on-hand.
* Rename f32/f64 to float32/float64 This commit renames the previously-known `f32` and `f64` interface types to `float32` and `float64` to match the upstream specification. * Fix a test
* Add support for params passed indirectly This commit adds support for the `MAX_FLAT_PARAMS` variable in the current canonical ABI draft. Argument lists which exceed a static number of parameters are now always passed indirectly through memory. In export this is dynamically allocated/freed and in imports this is statically passed. * Review comments
* Split out the `Option` and `Expected` types from `Variant`
This commit is like prior PRs to split out specializations of types into
their own AST type to avoid conflicting with the main type (in this case
`variant`). I originally thought these two types would be relatively
simple but this is probably one of the more complicated transitions, as
evidenced by the lines changed here. The main churn was that variants
already have a significant amount of code to support them and this is in
some places "duplicating" code for option/expected and in other cases
splitting what was already an if/else.
Overall I think that the generated code gets a little better since it's
clear when something is and `option` vs `expected` now rather than
trying to have everything shoehorned into one. Notably the C code
generator now generates descriptive fields like `bool is_some` or `bool
is_err` instead of a bland `uint8_t tag` with some comments about how to
use it.
* Remove `Variant::as_{option,expected}`
... as these are separate variants now.
* Review comments
* Split out a `Union` type from `Variant` This commit splits out the union specialization from `variant`, finalizing the transition to the new AST of types available in the current component model draft. This is mostly like the prior splittings-out where there's some duplication but it's relatively localized and hopefully not too hard to follow. * Fill out TODO
* Update enum representation * Generate variant docs * Fix demo
* wit-bindgen-rust: fix lowering and lifting of zero-length lists. This commit fixes the Rust bindings generated for lifting and lowering of lists to account for lists of zero-length. It is undefined behavior to call `std::alloc::alloc` with a zero-sized layout. Additionally, as the behavior of the `canonical_abi_realloc` implemented in `wit-bindgen` is to return a non-null dummy pointer for a zero-sized alloc, the bindings were calling `std::alloc::dealloc` with a pointer that was not allocated with `std::alloc::alloc` (also undefined behavior). To fix this, the bindings now account for zero-length lists and skip allocation and deallocation where appropriate. * Update `list` runtime tests to pass empty lists and strings.
* initial work * Fix formatting * Fix references to Rust library for guest code Improve documentation Other minor fixes * Fix wastime -> host_wasmtime_rust references Fixed feature flags * Fix demo deploy update readme improve help text * Update readme Co-authored-by: Kyle Brown <kbrown@singlestore.com> Co-authored-by: Kyle Brown <kyleb@liquidrocketry.com>
* Update JS integer lowering Updates JS integer lowering to match the component model explainer (assuming that WebAssembly/component-model#52 is merged). The main differences are that out-of-range numbers now wrap instead of throwing, and that non-integers are now rounded down instead of throwing.
* Remove `async func` support. The `async` keyword has been removed from the [async proposal], in favor of using plain functions that with `future` or `stream` types. See also the [component-model PR to remove the syntax]. This removes the (now) old `async func` suppport from wit-bindgen, to make it easier to develop the current Canonical ABI proposal. [async proposal]: https://docs.google.com/presentation/d/1MNVOZ8hdofO3tI0szg_i-Yoy0N2QPU2C--LzVuoGSlE/edit#slide=id.g1270ef7d5b6_0_111 [component-model PR to remove the syntax]: WebAssembly/component-model#98 * Remove support for the `async` checkbox in wit-bindgen-demo.
* Update flag representation * Use `flags.repr().count()` instead of manually matching * Remove flag validation intrinsics * Validate that flags don't have extraneous bits set when lifting * grammar nit Co-authored-by: Alex Crichton <alex@alexcrichton.com>
* Rename `canonical_abi_realloc` to `cabi_realloc`. This follows the current Canonical ABI. This doesn't rename `canonical_abi_free`, as that's expected to be removed when post-return functions are implemented. * Rename the "expected" type to "result". This follows the current Canonical ABI. * Implement function and value type name mangling. This implements the name-mangling scheme in the current canonical ABI, with the modification proposed in WebAssembly/component-model#104, though that can be easily removed if the proposal is declined. * Use name mangling in the bindings generators. * Use the export base name rather than the mangled name for python identifiers.
Signed-off-by: GitHub <noreply@github.com> Signed-off-by: GitHub <noreply@github.com>
* WIP: Update for component model multireturn and removal of unit This PR updates wit-bindgen for the latest changes to the component model: - the return of multireturn - removal of the `unit` type - a new syntax for optional types in results, streams, futures and variants all of which go hand in hand. This also pulls in the latest versions of wasm-encoder, wasmprinter, and wasmparser crates to get their updates for these component model changes. * Get all wit-bindgen tests working * Implement tests for multi-return Exercise this throughout the runtime tests and additionally add a few variants in codegen tests to ensure that this is exercised. * Update markdown generator Co-authored-by: George Kulakowski <gkulakowski@fastly.com>
This adds a few new pseudo-instructions plus a new method on `Interface`
to generate a `post-return` function. For now these are simply named
`{name}_post_return` and the integration point there will probably
change as the component model shapes up.
The integration here is intended to still be relatively primitive in
that the actual component model integration will likely look different
in the future. Given how `wit-bindgen` works today, though, this should
at least get things part of the way there.
* remove Canonical ABI name mangling this reverts a big chunk of the functionality introduced in bytecodealliance/wit-bindgen#309 * multi-return.wit: namespace these export functions the tests already export functions named a,b,c in strings.wit. We can't define more functions with that same name. * remove mangled names from tests/runtime/invalid/wasm.rs
Try to reduce the number of places versions are mentioned and encourage shared dependencies between crates.
…409) * gen-host-js: Error wrapping and unwrapping for singular results When an exported component function has a singular return of type result, treat this as a throwable in the JS enviroment, by throwing the error payload. Similarly, when an imported function has a singular return of type result, add a catch handler and wrap the function in a result object when handling. Further optimization work should be done to avoid unnecessary wrapping in future. * better payload check, return fixup * fixup demo * improved error payload extraction * fixup tests, intrinsic dependence * remove unnecessary tostring * pr feedback * remove unnecessary include * simpler throws check * review feedback * explicit result flattening * throws signature * fixup throws signature
* Add empty `imports.wit` for `exports_only` test Otherwise the build process thinks this is always missing so tests are always slowly rebuilt. * Fix compiling the `runtime` test for JS in isolation This requires the `clap` feature to be enabled now which works with `cargo test --workspace` but isn't working with individual crate testing. * Don't test wasm size in exports_only test Different versions of `wasi-libc` will produce differently sized binaries, so only test the generated JS which this repo controls.
…ining (#425) In addition, --instantiation now supports a third instantiateCore optional argument to replace the --load=custom wrapping use case
* Update all codegen tests to world syntax This enables simplifying codegen tests as well by having each test exercise imports/exports instead of having all code generators have one case for imports and one for exports. * Update codegen tests for the C guest generator * Update teavm codegen tests for worlds * Remove the `--name` argument from the CLI No longer needed as the name is inferred from the `*.wit` world. * Get all js host generator tests working Update all `runtime/*` tests with new naming conventions, world files, etc. Minor updates were made to names as I ended up using different conventions for the `*.wit` world files than were previously exercised. This also fixes a few minor issues with the output for the JS generator in the default ESM mode. * Get all host-wasmtime tests working with worlds Various small updates here and there to namings and such. * Get all Python tests working with worlds Fix support for exported instances by using correct import paths for various types/intrinsics/etc. Additionally update the codegen tests to specify a mypy cache dir to avoid racing between tests. * Update demo build for worlds Also fix an issue with the Rust host generator where it used an interface's name instead of the name of the import for import module names. * Update demo with world files
* Bump version of dependency crates This commit updates this repository to published versions of the `wasm-tools` family of crates. Additionally Wasmtime is updated with the same upstream versions of the `wasm-tools` crates. Put together this updates everything to the most recent version of the component model with optional URLs encoded and an updated version of Wasmtime. Part of this transition is the removal of `wit_component::ComponentInterfaces` which is plumbed through as part of this commit. * Update to wasmtime main * Fix demo build * Update wasm-tools binary * Fix cli build * Install wasm-tools from crates.io, not git
guybedford
added a commit
that referenced
this pull request
Nov 28, 2023
* feat: add initial impl for node sockets * chore: update sockets tests * feat: add TcpSocketImpl * chore: temp disable eslint in lib/sockets/** * chore: refactor instanceNetwork + dropNetwork * chore: tcp socket impl wip * chore: createTcpSocket/startBind/finishBind * fix: dupe-class-members * chore: add throws entry to jsdoc * chore: improve startBind() logic * chore: improve startConnect() + test * chore: improve tcp-socket impl * refactor: start-* finish-* methods to match specs * chore: use assertion statement in tcp-socket-impl * refactor: tcp-socket-impl * chore: use err.code to catch sockets errors * chore: migrate tcp-socket-impl to use tcp_wrap * chore: mock listen and connect calls * fix: bind connectReq.oncomplete to class method * chore: reorder test suites * chore: add e2e test for tcp-socket-impl * chore: hook up internal events * chore: update impl to match latest wit specs * fix: support bind6 and connect6 * chore: move assert to own file * chore: use assert * chore: use ipv6 in tests * chore: add streams to tcp-socket-impl (wip) * fix: add error code -49 address-not-bindable * feat: add udp-socket-impl method signatures * fix: handle error -99 * chore: document udp errors * chore: wip * feat(udp): wip impl * chore: clean imports * chore: use new folder structure * chore(udp): use Symbols for local state * chore: sync impl with lastest specs * chore: remove useless logs * chore(udp): fix according to conformance tests * fix(tcp): removed deprecated methods * chore: try to reuse an existing udp socket (wip) * fix: make conformance preview2_udp_states pass * feat: add ip-name-lookup (wip) * chore: socket resolve addresses (#1) * chore: add missing import from node:dns/promises * chore: resolve ipv6 :: and ::1 * fix(udp): make preview2_udp_sockopts apss * chore: delete unused code * chore: add comments * chore: more conformance tests passing! * chore: use enums for socket conn state * chore: make preview2_tcp_states pass * fix: make preview2_tcp_connect pass (wip) * fix: refactor isUnicastIpAddress * fix: make preview2_tcp_bind pass * fix: make preview2_tcp_connect pass * chore: refactor code * chore: preview2_tcp_sample_application wip * fix: preview2_tcp_bind and preview2_udp_bind * chore: stashing wip fixes * fix: improve tests and add more comments for hop limits * chore: rename errorState property * fix: make unit tests pass * chore: fix linting --------- Co-authored-by: Guy Bedford <guybedford@gmail.com>
2 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This performs the initial migration of
gen-host-jsfrom wit-bindgen into this repo, along with the initial associated JS component tools CLI functions including component optimization and asm.js transpilation. The commit history from gen-host-js is preserved.This includes the full usage:
The main library is built as two single-file optimized JS builds -
dist/cli.mjsanddist/api.mjsfor CLI and API usage respectively.Remaining work to land this draft PR:
Once this lands, the remaining gen-host-js issues can be ported to this repo.