[Merged by Bors] - Add ES5 and ES6 Conformance calculation to boa_tester #2690

ZackMitkin · 2023-03-18T17:26:35Z

This is calculated based the tests es5id or es6id.

Judging by this it's probably not the best method of doing it, but the alternative would require a lot of rework on the boa_tester so that it can pull an older version of the test262 spec which has it's own problems since there's not really an "ES5" only version.

I would think that if we're 100% passing on es5id's then it's safe to assume boa supports es5 (assuming test262's es5id is covering all the es5 cases)

This Pull Request fixes/closes #2629.

It changes the following:

Store spec_version in TestResult based on the tests es[6/5]id
Count all es5, es6 and their test outcome during TestSuite::run()
Print the conformance.

I'm serializing the spec_version outcomes so that it can be displayed in test262 github page. I'd like to work on that too if possible. Let me know if there's anything more I should cover in this PR, I'll gladly do it :)

codecov · 2023-03-18T17:53:30Z

Codecov Report

Merging #2690 (e6e2c8d) into main (7e6d3c9) will decrease coverage by 0.06%.
The diff coverage is 0.00%.

@@            Coverage Diff             @@
##             main    #2690      +/-   ##
==========================================
- Coverage   49.36%   49.30%   -0.06%     
==========================================
  Files         397      396       -1     
  Lines       39568    39776     +208     
==========================================
+ Hits        19533    19612      +79     
- Misses      20035    20164     +129

Impacted Files	Coverage Δ
boa_tester/src/exec/mod.rs	`0.00% <0.00%> (ø)`
boa_tester/src/main.rs	`0.00% <0.00%> (ø)`
boa_tester/src/results.rs	`0.00% <0.00%> (ø)`

... and 28 files with indirect coverage changes

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

Razican

It seems that the Test262 comparator fails because the current "main" branch doesn't have the new fields in JSON. I guess they should be optional fields.

ZackMitkin · 2023-03-18T17:56:36Z

It seems that the Test262 comparator fails because the current "main" branch doesn't have the new fields in JSON. I guess they should be optional fields.

Yep, just noticed too, I've just added #[serde(default)] to the TestResult for es5 and es6

jedel1043

Thanks for the contribution! I just have a small suggestion for development ease :)

boa_tester/src/main.rs

This is calculated based the tests es5id or es6id

boa_tester/src/main.rs

boa_tester/src/exec/mod.rs

ZackMitkin · 2023-03-20T10:57:54Z

I've separated the SuiteResults stats by es-version and all stats. This will break the current CI but I can create a PR to back-fill the previous runs and fix the test262 github page sometime this week

boa_tester/src/results.rs

boa_tester/src/main.rs

jedel1043 · 2023-03-20T19:31:40Z

Opened #2706 to solve the CI issue. We can merge that PR after we agree on the new data model here and this PR after that.

boa_tester/src/main.rs

boa_tester/src/exec/mod.rs

this allows us to organize total, passed, ignored, and panic stats by es version

jedel1043

Thanks for the contribution! Looks good to me

HalidOdat

Looks good to me! :)

jedel1043 · 2023-03-20T20:13:49Z

Ok, gonna merge #2706 and then run CI on this.

ZackMitkin · 2023-03-20T20:27:39Z

Ok, gonna merge #2706 and then run CI on this.

Wouldn't it be better to merge your PR into this branch to check CI? Not sure why run it on main...

jedel1043 · 2023-03-20T20:32:24Z

Wouldn't it be better to merge your PR into this branch to check CI? Not sure why run it on main...

Because #2706 doesn't target main, but gh-pages, so we cannot combine both PRs into one.

ZackMitkin · 2023-03-20T20:33:33Z

Copying the ../gh-pages/test262/refs/heads/main/results.json file to ../results/test262/pull/results.json in order to add the results
Writing the results to ../results/test262/pull...
Error: 
   0: could not write the results to the output JSON file
   1: missing field `a5` at line 1 column 136

No idea what's happening here... I have seen it before though during testing. It worked fine if I wrote to a clean or non-existing dir, but if I tried to overwrite an older schema it'd fail... didn't think serde cared about the overwritten schema...

jedel1043 · 2023-03-20T20:38:19Z

Yeah, serde is very strict with serializations. Our best option here would be to derive Default for Statistics and add a serde(default) to the new a5 and a6 fields.

boa_tester/src/main.rs

jedel1043 · 2023-03-20T22:21:45Z

All good now!

jasonwilliams

LGTM
Great contribution!

HalidOdat

Looks good to me! :)

jedel1043 · 2023-03-20T22:30:53Z

bors r+

This is calculated based the tests `es5id` or `es6id`. Judging by [this](tc39/test262#1557) it's probably not the best method of doing it, but the alternative would require a lot of rework on the boa_tester so that it can pull an older version of the test262 spec which has it's own problems since there's not really an "ES5" only version. I would think that if we're 100% passing on es5id's then it's safe to assume boa supports es5 (assuming test262's es5id is covering all the es5 cases) This Pull Request fixes/closes #2629. It changes the following: - Store `spec_version` in TestResult based on the tests `es[6/5]id` - Count all es5, es6 and their test outcome during `TestSuite::run()` - Print the conformance. I'm serializing the `spec_version` outcomes so that it can be displayed in test262 github page. I'd like to work on that too if possible. Let me know if there's anything more I should cover in this PR, I'll gladly do it :) Co-authored-by: jedel1043 <jedel0124@gmail.com>

bors · 2023-03-20T22:42:51Z

Pull request successfully merged into main.

Build succeeded:

ZackMitkin force-pushed the main branch from 3bde68d to 189183e Compare March 18, 2023 17:54

Razican reviewed Mar 18, 2023

View reviewed changes

jedel1043 requested changes Mar 18, 2023

View reviewed changes

boa_tester/src/main.rs Outdated Show resolved Hide resolved

ZackMitkin force-pushed the main branch from 189183e to 28c71de Compare March 19, 2023 12:03

ZackMitkin requested a review from jedel1043 March 19, 2023 12:07

ZackMitkin force-pushed the main branch 3 times, most recently from 05d0659 to cacdddb Compare March 19, 2023 13:07

Add ES5 and ES6 Conformance calculation to boa_tester

6f43cb7

This is calculated based the tests es5id or es6id

ZackMitkin force-pushed the main branch from cacdddb to 6f43cb7 Compare March 19, 2023 13:34

jedel1043 requested changes Mar 19, 2023

View reviewed changes

boa_tester/src/main.rs Outdated Show resolved Hide resolved

boa_tester/src/main.rs Outdated Show resolved Hide resolved

boa_tester/src/main.rs Outdated Show resolved Hide resolved

boa_tester/src/exec/mod.rs Outdated Show resolved Hide resolved

ZackMitkin requested a review from jedel1043 March 20, 2023 10:58

ZackMitkin force-pushed the main branch from 757752c to 077e136 Compare March 20, 2023 16:32

ZackMitkin requested review from Razican and jedel1043 and removed request for jedel1043 and Razican March 20, 2023 16:33

jedel1043 reviewed Mar 20, 2023

View reviewed changes

boa_tester/src/results.rs Outdated Show resolved Hide resolved

jedel1043 reviewed Mar 20, 2023

View reviewed changes

boa_tester/src/main.rs Outdated Show resolved Hide resolved

jedel1043 mentioned this pull request Mar 20, 2023

Migrate data to new structure #2706

Merged

ZackMitkin force-pushed the main branch from 077e136 to eb82ace Compare March 20, 2023 19:41

ZackMitkin requested a review from jedel1043 March 20, 2023 19:52

jedel1043 reviewed Mar 20, 2023

View reviewed changes

boa_tester/src/main.rs Outdated Show resolved Hide resolved

Razican reviewed Mar 20, 2023

View reviewed changes

boa_tester/src/main.rs Outdated Show resolved Hide resolved

boa_tester/src/exec/mod.rs Outdated Show resolved Hide resolved

Seperate boa_tester SuiteResults stats by es version.

6f0ac5c

this allows us to organize total, passed, ignored, and panic stats by es version

ZackMitkin force-pushed the main branch from eb82ace to 6f0ac5c Compare March 20, 2023 19:59

ZackMitkin requested a review from jedel1043 March 20, 2023 20:03

jedel1043 approved these changes Mar 20, 2023

View reviewed changes

jedel1043 added enhancement New feature or request test Issues and PRs related to the tests. labels Mar 20, 2023

jedel1043 added this to the v0.17.0 milestone Mar 20, 2023

HalidOdat approved these changes Mar 20, 2023

View reviewed changes

ZackMitkin requested review from jedel1043 and removed request for Razican March 20, 2023 20:43

jedel1043 reviewed Mar 20, 2023

View reviewed changes

boa_tester/src/main.rs Outdated Show resolved Hide resolved

add serde default to SuiteResult and ReducedResultInfo Statistics fields

fd3146b

ZackMitkin force-pushed the main branch from 7b68d04 to fd3146b Compare March 20, 2023 20:46

ZackMitkin requested a review from HalidOdat March 20, 2023 20:52

jedel1043 added 3 commits March 20, 2023 15:05

Add serde(rename) attribute

6aa4e3f

Add serde(rename) attribute pt. 2

d2021ab

Derive Default for SpecVersion

e6e2c8d

jedel1043 requested a review from Razican March 20, 2023 22:21

jasonwilliams approved these changes Mar 20, 2023

View reviewed changes

HalidOdat approved these changes Mar 20, 2023

View reviewed changes

bors bot changed the title ~~Add ES5 and ES6 Conformance calculation to boa_tester~~ [Merged by Bors] - Add ES5 and ES6 Conformance calculation to boa_tester Mar 20, 2023

bors bot closed this Mar 20, 2023

jedel1043 mentioned this pull request Mar 20, 2023

Display ES5 conformance information in our webpage #2710

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Merged by Bors] - Add ES5 and ES6 Conformance calculation to boa_tester #2690

[Merged by Bors] - Add ES5 and ES6 Conformance calculation to boa_tester #2690

ZackMitkin commented Mar 18, 2023 •

edited

Loading

codecov bot commented Mar 18, 2023 •

edited

Loading

Razican left a comment

ZackMitkin commented Mar 18, 2023

jedel1043 left a comment

ZackMitkin commented Mar 20, 2023

jedel1043 commented Mar 20, 2023

jedel1043 left a comment

HalidOdat left a comment

jedel1043 commented Mar 20, 2023

ZackMitkin commented Mar 20, 2023

jedel1043 commented Mar 20, 2023 •

edited

Loading

ZackMitkin commented Mar 20, 2023

jedel1043 commented Mar 20, 2023 •

edited

Loading

jedel1043 commented Mar 20, 2023

jasonwilliams left a comment

HalidOdat left a comment

jedel1043 commented Mar 20, 2023

bors bot commented Mar 20, 2023

[Merged by Bors] - Add ES5 and ES6 Conformance calculation to boa_tester #2690

[Merged by Bors] - Add ES5 and ES6 Conformance calculation to boa_tester #2690

Conversation

ZackMitkin commented Mar 18, 2023 • edited Loading

codecov bot commented Mar 18, 2023 • edited Loading

Codecov Report

Razican left a comment

Choose a reason for hiding this comment

ZackMitkin commented Mar 18, 2023

jedel1043 left a comment

Choose a reason for hiding this comment

ZackMitkin commented Mar 20, 2023

jedel1043 commented Mar 20, 2023

jedel1043 left a comment

Choose a reason for hiding this comment

HalidOdat left a comment

Choose a reason for hiding this comment

jedel1043 commented Mar 20, 2023

ZackMitkin commented Mar 20, 2023

jedel1043 commented Mar 20, 2023 • edited Loading

ZackMitkin commented Mar 20, 2023

jedel1043 commented Mar 20, 2023 • edited Loading

jedel1043 commented Mar 20, 2023

jasonwilliams left a comment

Choose a reason for hiding this comment

HalidOdat left a comment

Choose a reason for hiding this comment

jedel1043 commented Mar 20, 2023

bors bot commented Mar 20, 2023

ZackMitkin commented Mar 18, 2023 •

edited

Loading

codecov bot commented Mar 18, 2023 •

edited

Loading

jedel1043 commented Mar 20, 2023 •

edited

Loading

jedel1043 commented Mar 20, 2023 •

edited

Loading