Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fixed flakes and added parallelization to ChainComponents tests #15537

Merged
merged 2 commits into from
Dec 23, 2024
Merged

Fixed flakes and added parallelization to ChainComponents tests #15537

merged 2 commits into from
Dec 23, 2024

Conversation

silaslenihan
Copy link
Contributor

@silaslenihan silaslenihan commented Dec 5, 2024 •
edited
Loading

This PR changes the ChainComponents EVM testing interface to support parallel tests, enabling isolated test state and removing potential race conditions.

Local verification to confirm the tests are flake free
Total runs: 500
Failures: 0
Pass rate: 100.00%
Average test time: 3m11.98371491s
Total time: 26h39m51.85745504s

Requires

Supports

@silaslenihan silaslenihan force-pushed the BCFR-1083-Fix-flakes-and-optimize-CC-tests branch 2 times, most recently from dff9e0e to 0ba8baa Compare December 6, 2024 00:02
@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Dec 6, 2024 •
edited
Loading

AER Report: CI Core

aer_workflow , commit , Detect Changes , Clean Go Tidy & Generate , Scheduled Run Frequency , Flakeguard Root Project / Get Tests To Run , GolangCI Lint (core/scripts) , Core Tests (go_core_tests) , GolangCI Lint (.) , test-scripts , Core Tests (go_core_tests_integration) , GolangCI Lint (integration-tests) , Core Tests (go_core_ccip_deployment_tests) , GolangCI Lint (deployment) , Core Tests (go_core_fuzz) , Core Tests (go_core_race_tests) , Flakeguard Deployment Project / Get Tests To Run , Flakeguard Root Project / Run Tests , Flakeguard Root Project / Report , Flakeguard Deployment Project / Run Tests , Flakeguard Deployment Project / Report , lint , SonarQube Scan , Flakey Test Detection

1. File not found error during SonarScanner execution:[SonarQube Scan]

Source of Error: 
SonarQube Scan	2024-12-23T18:55:09.0484804Z ERROR: Error during SonarScanner execution
SonarQube Scan	2024-12-23T18:55:09.0485994Z java.io.UncheckedIOException: java.nio.file.NoSuchFileException: /github/workspace/.git/.probe-cacec0ca-c1d9-4af7-bdd9-970c8f20ea9c
SonarQube Scan	2024-12-23T18:55:09.0559776Z 	at java.base/java.nio.file.FileTreeIterator.fetchNextIfNeeded(FileTreeIterator.java:83)
SonarQube Scan	2024-12-23T18:55:09.0560444Z 	... 45 more

Why: The error occurred because the SonarScanner attempted to access a file (/github/workspace/.git/.probe-cacec0ca-c1d9-4af7-bdd9-970c8f20ea9c) that does not exist in the specified directory. This could be due to a misconfiguration or a missing file in the repository.

Suggested fix: Ensure that the .git directory and all necessary files are correctly checked out in the workspace. Verify the repository configuration and the presence of the required files before running the SonarScanner.

AER Report: Operator UI CI ran successfully ✅

aer_workflow , commit

@silaslenihan silaslenihan force-pushed the BCFR-1083-Fix-flakes-and-optimize-CC-tests branch 3 times, most recently from b4c5545 to f12df7a Compare December 11, 2024 17:55
@silaslenihan silaslenihan force-pushed the BCFR-1083-Fix-flakes-and-optimize-CC-tests branch 4 times, most recently from 536d752 to 146dd9f Compare December 17, 2024 14:54
@github-actions GitHub Actions
Copy link
Contributor

Flakeguard Summary

Ran new or updated tests between develop and 146dd9f (BCFR-1083-Fix-flakes-and-optimize-CC-tests).

View Flaky Detector Details | Compare Changes

Found Flaky Tests ❌

Name Pass Ratio Panicked? Timed Out? Race? Runs Successes Failures Skips Package Package Panicked? Avg Duration Code Owners
TestRunChainComponentsMultipleTimes 0.00% true true false 3 0 3 0 github.com/smartcontractkit/chainlink/v2/core/services/relay/evm true 0s Unknown

Artifacts

For detailed logs of the failed tests, please refer to the artifact failed-test-results-with-logs.json.

@silaslenihan silaslenihan mentioned this pull request Dec 17, 2024
Merged
@silaslenihan silaslenihan force-pushed the BCFR-1083-Fix-flakes-and-optimize-CC-tests branch 8 times, most recently from 96ccc8a to 41b47e6 Compare December 18, 2024 19:07
@silaslenihan silaslenihan mentioned this pull request Dec 18, 2024
Merged
@silaslenihan silaslenihan force-pushed the BCFR-1083-Fix-flakes-and-optimize-CC-tests branch 6 times, most recently from ba0f7a2 to a1f48db Compare December 18, 2024 21:28
@silaslenihan silaslenihan marked this pull request as ready for review December 18, 2024 21:31
@silaslenihan silaslenihan requested review from a team as code owners December 18, 2024 21:31
ilija42
ilija42 reviewed Dec 23, 2024
View reviewed changes
core/services/relay/evm/chain_components_test.go
t.Run("RunChainComponentsEvmTests", func(t *testing.T) {
t.Parallel()
it := &EVMChainComponentsInterfaceTester[*testing.T]{Helper: helper, DeployLock: &deployLock}
// These tests are broken in develop as well, so disable them for now
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

CC @poopoothegorilla Can you create an epic for general Chain Components work that is not urgent, but has to be done? Maybe just have 1 for now and then have them per chain

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Created this epic https://smartcontract-it.atlassian.net/browse/NONEVM-1084 used the word "Hardening" which some other projects use for this type of work

ilija42
ilija42 previously approved these changes Dec 23, 2024
View reviewed changes
Copy link
Contributor

@ilija42 ilija42 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good job!

@silaslenihan silaslenihan force-pushed the BCFR-1083-Fix-flakes-and-optimize-CC-tests branch 3 times, most recently from 6450e32 to ae33309 Compare December 23, 2024 15:18
ilija42
ilija42 previously approved these changes Dec 23, 2024
View reviewed changes
Farber98
Farber98 previously approved these changes Dec 23, 2024
View reviewed changes
Copy link
Contributor

@Farber98 Farber98 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

great job with this! 💪

@silaslenihan silaslenihan dismissed stale reviews from Farber98 and ilija42 via 30edf9d December 23, 2024 15:33
@silaslenihan silaslenihan force-pushed the BCFR-1083-Fix-flakes-and-optimize-CC-tests branch from ae33309 to 30edf9d Compare December 23, 2024 15:33
@cl-sonarqube-production
Copy link

Quality Gate passed Quality Gate passed

Issues
1 New issue
1 Fixed issue
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
7.8% Duplication on New Code

See analysis details on SonarQube

Farber98
Farber98 previously approved these changes Dec 23, 2024
View reviewed changes
ilija42
ilija42 previously approved these changes Dec 23, 2024
View reviewed changes
@ilija42 ilija42 added this pull request to the merge queue Dec 23, 2024
@ilija42 ilija42 removed this pull request from the merge queue due to a manual request Dec 23, 2024
@silaslenihan silaslenihan dismissed stale reviews from ilija42 and Farber98 via 7d11d5a December 23, 2024 18:46
@silaslenihan silaslenihan force-pushed the BCFR-1083-Fix-flakes-and-optimize-CC-tests branch from 30edf9d to 7d11d5a Compare December 23, 2024 18:46
@ilija42 ilija42 added this pull request to the merge queue Dec 23, 2024
Merged via the queue into develop with commit c1e92af Dec 23, 2024
172 of 173 checks passed
@ilija42 ilija42 deleted the BCFR-1083-Fix-flakes-and-optimize-CC-tests branch December 23, 2024 20:38
@github-actions github-actions bot mentioned this pull request Dec 23, 2024
Draft
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@poopoothegorilla poopoothegorilla poopoothegorilla left review comments

@ilija42 ilija42 ilija42 approved these changes

@jmank88 jmank88 jmank88 approved these changes

@Farber98 Farber98 Awaiting requested review from Farber98

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@silaslenihan @jmank88 @poopoothegorilla @Farber98 @ilija42