Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Merged by Bors] - Migrate tests to new test API #2619

Closed
wants to merge 6 commits into from

Conversation

jedel1043
Copy link
Member

This PR migrates our entire test suite to our new testing API.

It changes the following:

  • Migrates tests to new test API.
  • Cleans up the API to be a bit more descriptive and maintainable.
  • Prettifies our test failure display to show the failed scripts.
  • Splits our massive tests.rs file into smaller sub-suites.

Example output of a failing test:

image

@jedel1043 jedel1043 added test Issues and PRs related to the tests. Internal Category for changelog labels Feb 27, 2023
@jedel1043 jedel1043 added this to the v0.17.0 milestone Feb 27, 2023
@github-actions
Copy link

github-actions bot commented Feb 27, 2023

Test262 conformance changes

Test result main count PR count difference
Total 94,181 94,181 0
Passed 71,950 71,950 0
Ignored 17,218 17,218 0
Failed 5,013 5,013 0
Panics 12 12 0
Conformance 76.40% 76.40% 0.00%

@codecov
Copy link

codecov bot commented Feb 27, 2023

Codecov Report

Merging #2619 (e909f81) into main (423c16a) will increase coverage by 0.17%.
The diff coverage is 75.94%.

@@            Coverage Diff             @@
##             main    #2619      +/-   ##
==========================================
+ Coverage   49.39%   49.56%   +0.17%     
==========================================
  Files         387      388       +1     
  Lines       39278    39275       -3     
==========================================
+ Hits        19400    19468      +68     
+ Misses      19878    19807      -71     
Impacted Files Coverage Δ
boa_engine/src/builtins/weak/weak_ref.rs 71.79% <ø> (ø)
boa_engine/src/context/mod.rs 44.13% <ø> (ø)
boa_engine/src/value/serde_json.rs 74.19% <ø> (ø)
boa_parser/src/lexer/error.rs 58.33% <50.00%> (ø)
boa_engine/src/error.rs 33.33% <66.66%> (-3.71%) ⬇️
boa_engine/src/lib.rs 78.20% <76.38%> (-21.80%) ⬇️
boa_engine/src/builtins/bigint/mod.rs 85.18% <100.00%> (ø)
boa_parser/src/error.rs 53.03% <100.00%> (ø)
boa_engine/src/value/display.rs 66.41% <0.00%> (-20.15%) ⬇️
boa_parser/src/parser/cursor/buffered_lexer/mod.rs 87.01% <0.00%> (-7.80%) ⬇️
... and 41 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.


let main_timer = Profiler::global().start_event("Main", "Main");
/// Runs `source`, panickinf if the execution throws.
pub(crate) fn run(source: impl Into<Cow<'static, str>>) -> Self {
Copy link
Member

@jasonwilliams jasonwilliams Feb 28, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What's the reason for using Cow here instead of just the static string? Are these test strings ever being mutated?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There are some tests that programmatically generate the tested scripts:

run_test(cases.into_iter().map(|(case, msg)| {
TestAction::assert_native_error(format!("'use strict'; {case}"), ErrorKind::Syntax, msg)
}));

The other option was to take a String instead of a &'static str, but that would make almost all tests slower, so I opted for taking a Cow instead.

typeof a;
"#;
assert_eq!(&exec(typeof_object), "\"string\"");
run_test([TestAction::assert_eq(
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Most of these tests are only passing 1 item in the array, do we want an API which can handle just a single test?
The name run_test doens't sound right when it's designed to take in an array of tests. But I understand this is just bikeshedding.

Copy link
Member Author

@jedel1043 jedel1043 Feb 28, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

run_test_actions maybe? Or could be even run_actions, I think.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think that name would be describing better what is happening yeah. (Even though it is longer)

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Most of these tests are only passing 1 item in the array, do we want an API which can handle just a single test?

Thought about it, but I didn't think it would be worth increasing the complexity of the API just to save two ([ and ]) keystrokes...

... but now that I think about it, we could implement IntoIterator for TestAction, which doesn't increase our API complexity, though it makes run_test_actions(TestAction::exec(..)) look a bit weird.

@jasonwilliams
Copy link
Member

Looks good so far, but improvement to what we had before!

@jedel1043 jedel1043 force-pushed the test-api-migration branch from 6c0482e to e909f81 Compare March 1, 2023 03:27
Copy link
Member

@HalidOdat HalidOdat left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The tests look much cleaner now! Great work @jedel1043

@jedel1043 jedel1043 requested a review from jasonwilliams March 3, 2023 05:13
Copy link
Member

@raskad raskad left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice improvement!

@jedel1043
Copy link
Member Author

Gonna merge this since I've already resolved Jason's comments, and It'll be better to merge this ASAP to avoid merge conflicts.

@jedel1043
Copy link
Member Author

bors r+

bors bot pushed a commit that referenced this pull request Mar 5, 2023
This PR migrates our entire test suite to our new testing API.

It changes the following:

- Migrates tests to new test API.
- Cleans up the API to be a bit more descriptive and maintainable.
- Prettifies our test failure display to show the failed scripts.
- Splits our massive `tests.rs` file into smaller sub-suites.

Example output of a failing test:

![image](https://user-images.githubusercontent.com/38230983/221502567-9e219371-b4ab-49d0-b42b-94a9b1a9c002.png)
@bors
Copy link

bors bot commented Mar 5, 2023

Pull request successfully merged into main.

Build succeeded:

@bors bors bot changed the title Migrate tests to new test API [Merged by Bors] - Migrate tests to new test API Mar 5, 2023
@bors bors bot closed this Mar 5, 2023
@bors bors bot deleted the test-api-migration branch March 5, 2023 20:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Internal Category for changelog test Issues and PRs related to the tests.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants