Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Proposal phase process requirements for test suite #1430

Closed
ngzhian opened this issue Mar 17, 2022 · 0 comments · Fixed by #1431
Closed

Proposal phase process requirements for test suite #1430

ngzhian opened this issue Mar 17, 2022 · 0 comments · Fixed by #1431

Comments

@ngzhian
Copy link
Member

ngzhian commented Mar 17, 2022

In phase 2, a test suite is added.
The entry requirement for phase 3 is:

Test suite has been updated to cover the feature in its forked repo.

But it doesn't need to run against the reference interpreter.

The test suite should run against some implementation, though it need not be the reference interpreter.

It is actually quite difficult to have tests but not have reference interpreter implementation because:

So in the current state, to be able to generate JS test files, you will need to implement it in the reference interpreter first.

I suggest that we make a tiny tweak to https://github.com/WebAssembly/spec/blob/main/test/core/run.py#L68 to take a flag to just generate js, and skip running the tests (and round tripping).
ngzhian added a commit to ngzhian/spec that referenced this issue Mar 18, 2022
@ngzhian
Add --generate-js-only flag to test runner
eeecec1 
This will return early right after generating JS from the wast test
files. It will not attempt to run the tests, or do the round trip
conversion from wasm <-> wast.

This is convenient for proposals to add tests without having to
update the reference interpreter with implementation, and generate those
tests to JS to run in other Wasm engines.

Fixes WebAssembly#1430.
@ngzhian ngzhian mentioned this issue Mar 18, 2022
Merged
Ms2ger pushed a commit that referenced this issue Mar 21, 2022
@ngzhian @Ms2ger
Add --generate-js-only flag to test runner
daee490 
This will return early right after generating JS from the wast test
files. It will not attempt to run the tests, or do the round trip
conversion from wasm <-> wast.

This is convenient for proposals to add tests without having to
update the reference interpreter with implementation, and generate those
tests to JS to run in other Wasm engines.

Fixes #1430.
dhil added a commit to effect-handlers/wasm-spec that referenced this issue Mar 17, 2023
@dhil @rossberg
@hanshoglund @ngzhian @sbc100 @tlively @ggreif @codefromthecrypt @titzer @keithw @gahaas @wooster0 @tpmccallum @ia0 @vapourismo @JamesMenetrey @ivan-pan @ictrobot @aathan @fifofefe @mnordine @CosineP @ariez-xyz @surma @takikawa @ianh @tomstuart @Jamesernator @whirlicote @Ms2ger @Adzz @osa1 @eutro @michaelficarra @yamt @candymate @bongjunj @ShinWonho @suhdonghwi @jimblandy @aheejin @littledan
Merge pull request #25 from dhil/wasmfx-funcref-merge
9f861ea 
* [spec] Add reference types to overview (WebAssembly#1394)

* [interpreter] Remove use of physical equality on characters (WebAssembly#1396)

* Merge SIMD proposal (WebAssembly#1391)

SIMD is [phase 5](WebAssembly/simd#507), merge all the changes back into main spec.

* Remove merge conflict marker

* [spec] Handle v128 in validation algorithm (WebAssembly#1399)

* [spec] Fix instruction table (WebAssembly#1402)

* Add tests for functions without end marker. NFC (WebAssembly#1405)

Inspired by this downstream test in wabt:
WebAssembly/wabt#1775

Fixes: WebAssembly#1404

* Describe correct tail call behavior across modules

Whether tail calls across module boundaries would guarantee tail call behavior was previously an open question, but @thibaudmichaud confirmed that they would guarantee tail call behavior in V8 in WebAssembly/tail-call#15 (comment).

* [interpreter] Fix a typo in README (WebAssembly#1406)

* Add a link to the proposals repo (WebAssembly#1409)

Fixes WebAssembly#1407.

* [spec] Add note regarding parameter names (WebAssembly#1412)

* Comments WIP

* Merge upstream (WebAssembly#55)

* Typo

* [spec] Clarifying note on text format (WebAssembly#1420)

Signed-off-by: Adrian Cole <adrian@tetrate.io>
Co-authored-by: Andreas Rossberg <rossberg@mpi-sws.org>

* Eps

* Eps

* [test] Fix section size in binary test (WebAssembly#1424)

* Update document README to install six

* Disallow type recursion (WebAssembly#56)

* Fix import order

* Add --generate-js-only flag to test runner

This will return early right after generating JS from the wast test
files. It will not attempt to run the tests, or do the round trip
conversion from wasm <-> wast.

This is convenient for proposals to add tests without having to
update the reference interpreter with implementation, and generate those
tests to JS to run in other Wasm engines.

Fixes WebAssembly#1430.

* Remove use of let from func.bind test

* Add call3

* [spec] Fix missing mention of vectype (WebAssembly#1436)

Fixed WebAssembly#1435.

* [spec] Fix single-table limitation in module instantiation (WebAssembly#1434)

* [spec] Fix missing immediate on table.set (WebAssembly#1441)

* [docs] Update syntax in examples (WebAssembly#1442)

* Clarification in proposals README

* [interpreter] Tweak start section AST to match spec

* [spec] Bump release to 2 (WebAssembly#1443)

At yesterday's WG meeting, we decided to make a new release, now switching to the Evergreen model. For administrative and technical reasons having to do with W3C procedure, we decided to bump the release number to 2.

From now on, the standard will iterate at version 2 from the W3C's official perspective. We use minor release numbers internally to distinguish different iterations.

(@ericprud, I hope I understood correctly that the Bikeshed "level" also needed to be bumped to 2.)

* Remove test cases with let

* Sync wpt test (WebAssembly#1449)

* [spec] Fix typo (WebAssembly#1448)

* [proposals] Add missing start to example (WebAssembly#1454)

* [spec] "version 2.0" -> "release 2.0" (WebAssembly#1452)

* [spec] Fix typo (WebAssembly#1458)

* [test] Add assert_trap for unreached valid case (WebAssembly#1460)

* [interpreter] Name the type Utf8.unicode

* [spec] Fix binary format of data/elem tags to allow LEB (WebAssembly#1461)

* [spec] Fix typos in numeric operations (WebAssembly#1467)

* [spec] Fix syntax error in element segments validation rule (WebAssembly#1465)

* [spec] Fix typo in global instance syntax (WebAssembly#1466)

* [spec] Fix typos in module instantiation (WebAssembly#1468)

* [interpreter] Turn into a Dune package (WebAssembly#1459)

* [spec] Fix typos in instruction validation rules (WebAssembly#1462)

* [bib] Update latex .bib file for webassembly 2.0 (WebAssembly#1463)

* [spec] Add missing default for vector types (WebAssembly#1464)

* [spec] Fix typos in binary and text formats (WebAssembly#1469)

* [spec] Fix various typos (WebAssembly#1470)

* TypeError for Global constructor with v128

At the moment the spec requires a `LinkError` to be thrown when the `WebAssembly.Global` constructor is called for type `v128`. This was introduced in WebAssembly/simd#360, but according to the PR description, actually a `TypeError` should be thrown. The PR refers to https://github.com/WebAssembly/simd/blob/master/proposals/simd/SIMD.md#javascript-api-and-simd-values, and there a `TypeError` is required.

* [spec] Fix LEB opcodes in instruction index (WebAssembly#1475)

* [spec] Fix v128.loadX_splat in instruction index (WebAssembly#1477)

* [interpreter] Dune test suite (WebAssembly#1478)

* [interpreter] Fix warning flags for OCaml 4.13 (WebAssembly#1481)

* [interpreter] Simplify lexer and avoid table overflow on some architectures (WebAssembly#1482)

* [spec] Editorial nit (WebAssembly#1484)

* [interpreter] Produce error messages in encoder (WebAssembly#1488)

* [spec] Add missing close paren on table abbreviation (WebAssembly#1486)

Also remove an unnecessary space in the previous table abbreviation.

* [spec] Remove outdated note (WebAssembly#1491)

* Eps

* [interpreter] Factor data and element segments into abstract types (WebAssembly#1492)

* [spec] Update note on module initialization trapping (WebAssembly#1493)

* Fix type equality

* Fix typo

* [spec] Add note about control stack invariant to algorithm (WebAssembly#1498)

* [spec] Tweak tokenisation for text format (WebAssembly#1499)

* [test] Use still-illegal opcode (func-refs) (WebAssembly#1501)

* Fix minor typos and consistency issues in the validation algorithm. (WebAssembly#61)

* Add definition of defaultable types (WebAssembly#62)

No rules for locals yet, since those are still being discussed.

* Remove func.bind (WebAssembly#64)

* Implement 1a (WebAssembly#63)

* Subtyping on vector types & heap bottom check (WebAssembly#66)

* [interpreter] Bring AST closer to spec

* [spec] Fix typo (WebAssembly#1508)

* WIP

* Remove polymorphic variants

* Minor syntactic consistency fix (WebAssembly#68)

Change pseudo equality operator from `==` to `=`.

* Bump version

* [spec] Fix table.copy validation typo (WebAssembly#1511)

* More fixes

* Fix Latex

* Adjust intro

* Spec local initialization (WebAssembly#67)

* Add table initialiser (WebAssembly#65)

* [spec] Remove outdated note (WebAssembly#1491)

* [interpreter] Factor data and element segments into abstract types (WebAssembly#1492)

* [spec] Update note on module initialization trapping (WebAssembly#1493)

* [spec] Add note about control stack invariant to algorithm (WebAssembly#1498)

* [spec] Tweak tokenisation for text format (WebAssembly#1499)

* [test] Use still-illegal opcode (func-refs) (WebAssembly#1501)

* [spec] Fix typo (WebAssembly#1508)

* [spec] Fix table.copy validation typo (WebAssembly#1511)

* Merge fallout

* Latex fixes

* [spec] Minor copy edit (WebAssembly#1512)

* Spec changelog

* [spec] Trivial editorial fix

* Update embedding

* Oops

* Argh

* Rename Sem to Dyn

* Readd match.mli

* [interpreter] Build wast.js with Js_of_ocaml (WebAssembly#1507)

* [interpreter] Add flag for controlling call budget

* Spec zero byte

* Fix table/elem expansion (WebAssembly#71)

* Fix merge artefact

* Restrict init from stack-polymorphism (WebAssembly#75)

* [spec] Simplify exec rule for if (WebAssembly#1517)

* [spec] Formatting tweak (WebAssembly#1519)

* [spec] Fix typing rule in appendix (WebAssembly#1516)

* [spec] Fix “invertible” typo (WebAssembly#1520)

* [spec] Correct use of opdtype and stacktype (WebAssembly#1524)

* [spec] Add note to instruction index (WebAssembly#1528)

* Add type annotation to call_ref (WebAssembly#76)

* [spec] Tweak wording to avoid first person

* Eps

* Eps2

* Eps3

* Remove unneeded assumption type

* [spec/test] Fix scoping of non-imported globals (WebAssembly#1525)

* Fix test

* A couple of tests

* Performance improvement

* Typo

* Another typo

* [spec] Fix language config

* Fix null subtyping being wrong way around (WebAssembly#79)

* [spec] Fix naming typo (WebAssembly#1532)

* Defunctorise types again

* [spec] Add citation for WasmCert (WebAssembly#1533)

* [test] Fix async_index.js

* [test] Enable the i64 tests in imports.wast.

Fixes WebAssembly#1514.

* Minor tweak

* [js-api][web-api] Editorial: Fix some minor issues.

Fixes WebAssembly#1064.

* Update README.md (WebAssembly#1540)

Improve wording.

* [spec] Fix typo in element execution (WebAssembly#1544)

* [spec] Remove obsolete note (WebAssembly#1545)

* cccccc[klghketetivvtnnhvntikigrnueuhdkkukljgjuest/meta/generate_*.js: sync upstream JS with tests (WebAssembly#1546)

* [spec] Editorial tweak

* [test] test segment/table mismatch and externref segment (WebAssembly#1547)

* [interpreter] Remove duplicate token declarations (WebAssembly#1548)

* Update Soundness appendix (WebAssembly#72)

* [spec] Formatting eps

* Remove oboslete note in README (WebAssembly#82)

* Add `print_i64` to generated spec tests

WebAssembly@82a613d added `print_i64` to the standalone test files, but not to the ones generated by the spec interpreter.

* [test] Tweak binary-leb128 and simd_lane (WebAssembly#1555)

* [spec] Allow explicit keyword definitions (WebAssembly#1553)

Rather than describing keyword tokens as always being defined implicitly by terminal symbols in syntactic productions, describe them as being defined implicitly or explicitly. This accounts for the explicit definitions of `offset` and `align` phrases, which are lexically keywords, later in the chapter.

Fixes WebAssembly#1552.

* [js-api] editorial: adjust link for v128 type

* Factor local init tests to local_init.wast; add more (WebAssembly#84)

* Update JS API for no-frills

* [spec] Add missing case for declarative elem segments

Fixes WebAssembly#1562.

* [spec] Hotfix last accidental commit

* [spec] Fix hyperref (WebAssembly#1563)

* [spec] Bump sphinx version to fix Python problem

* [spec] Fix minor errors and inconsistencies (WebAssembly#1564)

* Spacing

* Fix a couple more superfluous brackets

* [spec] Eps

* [interpreter] Refactor parser to handle select & call_indirect correctly (WebAssembly#1567)

* [spec] Remove dead piece of grammar

* [test] elem.wast: force to use exprs in a element (WebAssembly#1561)

* Fix typos in SIMD exec/instructions

* Update interpreter README (WebAssembly#1571)

It previously stated that the formal spec did not exist, but the spec has existed for years now.

* [spec] Remove an obsolete exec step (WebAssembly#1580)

* [test] Optional tableidx for table.{get,set,size,grow,fill} (WebAssembly#1582)

* [spec] Fix abstract grammar for const immediate (WebAssembly#1577)

* [spec] Fix context composition in text format (WebAssembly#1578)

* [spec] Fix label shadowing (WebAssembly#1579)

* Try bumping OCaml

* Try bumping checkout

* Adjust for multi-return

* Tweak reduction rules

* Spec return_call_ref

* Fix

* Text format

* [spec] Fix typos in instruction index (WebAssembly#1584)

* [spec] Fix typo (WebAssembly#1587)

* [spec] Remove inconsistent newline (WebAssembly#1589)

* [interpreter] Remove legacy bigarray linking (WebAssembly#1593)

* [spec] Show scrolls for overflow math blocks (WebAssembly#1594)

* [interpreter] Run JS tests via node.js (WebAssembly#1595)

* [spec] Remove stray `x` indices (WebAssembly#1598)

* [spec] Style tweak for cross-refs

* [spec] Style eps (WebAssembly#1601)

* Separate subsumption from instr sequencing

* State principal types

* Add statements about glbs, lubs, and disjoint hierarchies

* Add missing bot

* [spec] Clarify that atoms can be symbolic (WebAssembly#1602)

* [test] Import v128 global (WebAssembly#1597)

* Update Overview.md

* [js-api] Expose everywhere

* [js-api] Try to clarify NaN/infinity handling. (WebAssembly#1535)

* [web-api] Correct MIME type check. (WebAssembly#1537)

Fixes WebAssembly#1138.

* [ci] Pin nodejs version to avoid fetching failures (WebAssembly#1603)

The issues appears to be related to actions/runner-images#7002.

Co-authored-by: Ms2ger <Ms2ger@igalia.com>

* [spec] Add missing value to table.grow reduction rule (WebAssembly#1607)

* [test] Move SIMD linking test to simd dir (WebAssembly#1610)

* Editorial: Clarify the name of the instantiate algorithm.

* Add notes to discourage using synchronous APIs.

* [jsapi] Normative: Always queue a task during asynchronous instantiation

JSC will have to do asynchronous compilation work during some instantiations.
To be consistent, this PR always queues a task to complete instantiation,
except through the synchronous Instance(module) API, to ensure consistency
across platforms.

This patch also cleans up the specification in various surrounding ways:
- Include notes about APIs whose use is discouraged/may be limited

Closes WebAssembly#741
See also webpack/webpack#6433

* [test] Exception -> Tag in wasm-module-builder.js

The section name has changed to the tag section a few years ago. This
adds the corresponding changes added in
WebAssembly/exception-handling#252 and
WebAssembly/exception-handling#256.

* [spec] Fix reduction rule for label (WebAssembly#1612)

Fix WebAssembly#1605.

* [spec] Clarifying note about canonical NaNs (WebAssembly#1614)

* [spec] Tweak crossref

* [test] Fix invalid section ID tests (WebAssembly#1615)

* [tests] Disable node run for now

* [spec] Don't check in generated index, to avoid spurious merge conflicts

* [spec] Rename script

* [ci] deactivate node run for now

* Fix uses of \to; compositionality

* Fix typo in text expansion

* Follow-up fix

* Fix compilation errors after merge.

This commit fixes the errors introduced by the merge of
function-references/main into this tree.

---------

Signed-off-by: Adrian Cole <adrian@tetrate.io>
Co-authored-by: Andreas Rossberg <rossberg@dfinity.org>
Co-authored-by: Hans Höglund <hanshoglund@users.noreply.github.com>
Co-authored-by: Ng Zhi An <zhin@chromium.org>
Co-authored-by: Ng Zhi An <zhin@google.com>
Co-authored-by: Sam Clegg <sbc@chromium.org>
Co-authored-by: Thomas Lively <7121787+tlively@users.noreply.github.com>
Co-authored-by: Gabor Greif <ggreif@gmail.com>
Co-authored-by: Andreas Rossberg <rossberg@mpi-sws.org>
Co-authored-by: Crypt Keeper <64215+codefromthecrypt@users.noreply.github.com>
Co-authored-by: Andreas Rossberg <rossberg@chromium.org>
Co-authored-by: Ng Zhi An <ngzhian@gmail.com>
Co-authored-by: Ben L. Titzer <ben.titzer@gmail.com>
Co-authored-by: Keith Winstein <keithw@cs.stanford.edu>
Co-authored-by: gahaas <ahaas@google.com>
Co-authored-by: r00ster <r00ster91@protonmail.com>
Co-authored-by: Timothy McCallum <tpmccallum@users.noreply.github.com>
Co-authored-by: Julien Cretin <github@ia0.eu>
Co-authored-by: Julien Cretin <cretin@google.com>
Co-authored-by: Ole Krüger <ole@vprsm.de>
Co-authored-by: Jämes Ménétrey <james@menetrey.me>
Co-authored-by: Ivan Panchenko <39594356+ivan-pan@users.noreply.github.com>
Co-authored-by: Ethan Jones <ictrobot@outlook.com>
Co-authored-by: Ole Krüger <ole.kruger@trili.tech>
Co-authored-by: aathan <aathan_github@memeplex.com>
Co-authored-by: Alberto Fiori <9143617+fifofefe@users.noreply.github.com>
Co-authored-by: mnordine <marknordine@gmail.com>
Co-authored-by: cosine <CosineP@users.noreply.github.com>
Co-authored-by: ariez-xyz <41232910+ariez-xyz@users.noreply.github.com>
Co-authored-by: Surma <surma@surma.dev>
Co-authored-by: Asumu Takikawa <asumu@igalia.com>
Co-authored-by: Ian Henderson <ian@ianhenderson.org>
Co-authored-by: Tom Stuart <hi@tomstu.art>
Co-authored-by: James Browning <thejamesernator@gmail.com>
Co-authored-by: whirlicote <78504913+whirlicote@users.noreply.github.com>
Co-authored-by: Ms2ger <Ms2ger@igalia.com>
Co-authored-by: Adam Lancaster <Adzz@users.noreply.github.com>
Co-authored-by: Ömer Sinan Ağacan <omeragacan@gmail.com>
Co-authored-by: B Szilvasy <beatrice.szilvasy@gmail.com>
Co-authored-by: Thomas Lively <tlively123@gmail.com>
Co-authored-by: Michael Ficarra <github@michael.ficarra.me>
Co-authored-by: YAMAMOTO Takashi <yamamoto@midokura.com>
Co-authored-by: Thomas Lively <tlively@google.com>
Co-authored-by: candymate <31069474+candymate@users.noreply.github.com>
Co-authored-by: Bongjun Jang <bongjun.jang@kaist.ac.kr>
Co-authored-by: ShinWonho <50018375+ShinWonho@users.noreply.github.com>
Co-authored-by: 서동휘 <hwidongsuh@gmail.com>
Co-authored-by: Jim Blandy <jimb@red-bean.com>
Co-authored-by: Heejin Ahn <aheejin@gmail.com>
Co-authored-by: Daniel Ehrenberg <littledan@chromium.org>
backes pushed a commit to backes/spec that referenced this issue Jul 12, 2023
@ngzhian @rossberg
Add --generate-js-only flag to test runner
714441c 
This will return early right after generating JS from the wast test
files. It will not attempt to run the tests, or do the round trip
conversion from wasm <-> wast.

This is convenient for proposals to add tests without having to
update the reference interpreter with implementation, and generate those
tests to JS to run in other Wasm engines.

Fixes WebAssembly#1430.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Add --generate-js-only flag to test runner ngzhian/spec
1 participant
@ngzhian