Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix test-runner race conditions #269

Merged
merged 4 commits into from
Jan 25, 2023
Merged

Fix test-runner race conditions #269

merged 4 commits into from
Jan 25, 2023

Conversation

adzialocha
Copy link
Member

@adzialocha adzialocha commented Jan 24, 2023

Some of our tests use the underlying static schema provider which is a static mutable data store (due to a workaround for async-graphql). This mutable store is accessible across all test runner threads in parallel mode as they all run in the same process. To prevent overwriting data across threads (race condition) we have to run these test in serial.

Read more: https://users.rust-lang.org/t/static-mutables-in-tests/49321

Background

This PR is blantantly stolen from #266 which is the in-depth research of @sandreae to find out where the race-conditions came from.

The following changes have been made in comparison to PR #266:

  1. Added comments around the places where we add [#serial]. It's fairly special-case and I can see how that same problem will come up in the future when we (and others) add new tests and forget to make them execute in serial. These comments will not necessarily prevent that but it's better than nothing maybe.
  2. Removed all code which was somewhat optional around solving these kinds of race conditions. State messaging channel for services to report on async work for better tests #266 adds a feature to reliably inform about if an async task (like building a new schema) was finished. There is more race conditions to tackle, but they are more unlikely to happen (for now we can naively bump the waiting times).

I'm wondering if there's another way to tackle the issue in the future. We could have a mocked static schema store only for test environments which puts the data behind a randomized key which is set once per thread. Then all parallelly-running tests would have the schema store behind the same memory address, but access different areas inside of it.

Closes: #247

📋 Checklist

  • Add tests that cover your changes
  • Add this PR to the Unreleased section in CHANGELOG.md
  • Link this PR to any issues it closes
  • New files contain a SPDX license header

@codecov
Copy link

codecov bot commented Jan 24, 2023

Codecov Report

Base: 93.95% // Head: 93.97% // Increases project coverage by +0.02% 🎉

Coverage data is based on head (6565fea) compared to base (35e1d3e).
Patch coverage: 100.00% of modified lines in pull request are covered.

Additional details and impacted files
@@            Coverage Diff             @@
##             main     #269      +/-   ##
==========================================
+ Coverage   93.95%   93.97%   +0.02%     
==========================================
  Files          59       59              
  Lines        4928     4950      +22     
==========================================
+ Hits         4630     4652      +22     
  Misses        298      298              
Impacted Files Coverage Δ
aquadoggo/src/db/stores/document.rs 92.06% <ø> (ø)
aquadoggo/src/graphql/client/dynamic_query.rs 91.01% <100.00%> (+0.14%) ⬆️
...quadoggo/src/graphql/client/dynamic_types/tests.rs 100.00% <100.00%> (ø)
aquadoggo/src/graphql/client/mutation.rs 100.00% <100.00%> (ø)
aquadoggo/src/graphql/client/static_query.rs 100.00% <100.00%> (ø)
aquadoggo/src/graphql/client/tests.rs 100.00% <100.00%> (ø)
aquadoggo/src/graphql/schema.rs 92.85% <100.00%> (+0.10%) ⬆️
aquadoggo/src/tests.rs 100.00% <100.00%> (ø)

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report at Codecov.
📢 Do you have feedback about the report comment? Let us know in this issue.

@adzialocha adzialocha closed this Jan 24, 2023
@adzialocha adzialocha deleted the fix-test-race-conditions branch January 24, 2023 10:20
@adzialocha adzialocha restored the fix-test-race-conditions branch January 24, 2023 10:36
@adzialocha adzialocha reopened this Jan 24, 2023
@adzialocha adzialocha marked this pull request as ready for review January 25, 2023 10:50
Copy link
Member

@sandreae sandreae left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me!

You mentioned still seeing one test fail sometimes here but I couldn't reproduce this myself.

@sandreae sandreae merged commit 4776b89 into main Jan 25, 2023
@adzialocha
Copy link
Member Author

Looks good to me!

You mentioned still seeing one test fail sometimes here but I couldn't reproduce this myself.

Thanks! I'll make an issue about it and then we keep looking out for it!

@adzialocha adzialocha deleted the fix-test-race-conditions branch January 25, 2023 11:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Race conditions with failing tests
2 participants