Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

test: improve ssz tests consistency #6776

Merged
merged 2 commits into from
May 16, 2024
Merged

test: improve ssz tests consistency #6776

merged 2 commits into from
May 16, 2024

Conversation

jeluard
Copy link
Contributor

@jeluard jeluard commented May 14, 2024

Motivation

Make sure all tests always run

Description

Do not throw an error when an SSZ type is missing, but rather fail a test.
This let subsequent generated tests run not matter what, ensuring the tests count is consistent.

@jeluard jeluard requested a review from a team as a code owner May 14, 2024 13:51
nflaig
nflaig reviewed May 14, 2024
View reviewed changes
packages/beacon-node/test/spec/presets/ssz_static.test.ts
@@ -56,9 +56,9 @@ const sszStatic =
(ssz.bellatrix as Types)[typeName] ||
(ssz.altair as Types)[typeName] ||
(ssz.phase0 as Types)[typeName];
if (!sszType) {
throw Error(`No type for ${typeName}`);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

doesn't this just result in strange errors downstream if we don't throw here and exit the test run?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Makes sense. I updated so that when sszType is null, associated tests are not run at all; also improved the error message to be more meaningfull.
Compared to the original Error based approach, a missing SSZ type will not prevent further tests to be run (the Error would short-circuit the logic)

nflaig
nflaig reviewed May 15, 2024
View reviewed changes
Copy link
Member

@nflaig nflaig left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Still not sure this improves the situation, could you provide the output of running the ssz_static tests with a missing ssz type before vs. after your changes

@jeluard
Copy link
Contributor Author

jeluard commented May 15, 2024

The main thing that it improves, besides better error message giving some context, is that further tests are not short-circuited.
Currently all tests related to types defined after the missing one will not be generated/Executed as the Error short-circuit the whole dynamic test generation.
With this change, a single test fail related to the missing types, and all tests are run for further types (if they are existing).

@nflaig
Copy link
Member

nflaig commented May 15, 2024

besides better error message giving some context, is that further tests are not short-circuited

I understand that but can you provide a before vs. after example

wemeetagain
wemeetagain approved these changes May 16, 2024
View reviewed changes
Copy link
Member

@wemeetagain wemeetagain left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Makes sense to me.
The test count is still variable, but at least one missing ssz type won't cause an unknown number of unrelated ssz type tests from being suppressed.

@wemeetagain wemeetagain merged commit 68ded3d into ChainSafe:electra-fork May 16, 2024
12 of 17 checks passed
This was referenced May 16, 2024
Merged
Merged
g11tech pushed a commit that referenced this pull request May 24, 2024
@jeluard @g11tech
test: improve ssz tests consistency (#6776)
ed0dbe9 
* test: improve ssz tests consistency

* chore: address comments
g11tech pushed a commit to g11tech/lodestar that referenced this pull request Jun 19, 2024
@jeluard @g11tech
test: improve ssz tests consistency (ChainSafe#6776)
fdefed8 
* test: improve ssz tests consistency

* chore: address comments
g11tech pushed a commit that referenced this pull request Jun 25, 2024
@jeluard @g11tech
test: improve ssz tests consistency (#6776)
c20e7ae 
* test: improve ssz tests consistency

* chore: address comments
g11tech pushed a commit that referenced this pull request Jun 25, 2024
@jeluard @g11tech
test: improve ssz tests consistency (#6776)
c015abc 
* test: improve ssz tests consistency

* chore: address comments
g11tech pushed a commit that referenced this pull request Jul 1, 2024
@jeluard @g11tech
test: improve ssz tests consistency (#6776)
5147fb7 
* test: improve ssz tests consistency

* chore: address comments
g11tech pushed a commit that referenced this pull request Jul 30, 2024
@jeluard @g11tech
test: improve ssz tests consistency (#6776)
6241272 
* test: improve ssz tests consistency

* chore: address comments
g11tech pushed a commit that referenced this pull request Jul 31, 2024
@jeluard @g11tech
test: improve ssz tests consistency (#6776)
d64d5d1 
* test: improve ssz tests consistency

* chore: address comments
g11tech pushed a commit that referenced this pull request Aug 2, 2024
@jeluard @g11tech
test: improve ssz tests consistency (#6776)
31e98d2 
* test: improve ssz tests consistency

* chore: address comments
g11tech pushed a commit that referenced this pull request Aug 9, 2024
@jeluard @g11tech
test: improve ssz tests consistency (#6776)
aad56b3 
* test: improve ssz tests consistency

* chore: address comments
g11tech pushed a commit that referenced this pull request Aug 9, 2024
@jeluard @g11tech
test: improve ssz tests consistency (#6776)
1f97284 
* test: improve ssz tests consistency

* chore: address comments
g11tech pushed a commit that referenced this pull request Aug 23, 2024
@jeluard @g11tech
test: improve ssz tests consistency (#6776)
a6ba9e8 
* test: improve ssz tests consistency

* chore: address comments
g11tech pushed a commit that referenced this pull request Aug 27, 2024
@jeluard @g11tech
test: improve ssz tests consistency (#6776)
7b76bfc 
* test: improve ssz tests consistency

* chore: address comments
philknows pushed a commit that referenced this pull request Sep 3, 2024
@jeluard @philknows
test: improve ssz tests consistency (#6776)
f83a8a1 
* test: improve ssz tests consistency

* chore: address comments
@wemeetagain
Copy link
Member

🎉 This PR is included in v1.22.0 🎉

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@nflaig nflaig nflaig left review comments

@wemeetagain wemeetagain wemeetagain approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@jeluard @nflaig @wemeetagain