CI tests appear to run indefinitely when parallelized, but only on Github, not locally #25
Labels
Milestone
Comments
abelbraaksma
added a commit
that referenced
this issue
Oct 14, 2022
abelbraaksma
added a commit
that referenced
this issue
Oct 14, 2022
Revert "Troubleshooting CI, remove OfXXX, CE, Other and ToXXX" This reverts commit 6856b9e. Turn TaskSeq.empty into a unit-function Troubleshooting CI, remove Item, Pick, CE Revert "Troubleshooting CI, remove Item, Pick, CE" This reverts commit bc6c260. Several attempts at troubleshooting CI issue, see log details and #25 Temporarily killing most tests, to troubleshoot hanging CI Revert "Temporarily killing most tests, to troubleshoot hanging CI" This reverts commit b84fff1. Troubleshooting CI, adding back several new tests Temporarily killing most tests, to troubleshoot hanging CI Revert "Temporarily killing most tests, to troubleshoot hanging CI" This reverts commit b84fff1. Troubleshooting CI, adding back several new tests Revert "Troubleshooting CI, adding back several new tests" This reverts commit 5ecf179. Troubleshooting CI, re-enabling only some tests, including new perf tests Revert "Troubleshooting CI, re-enabling only some tests, including new perf tests" This reverts commit 602cb3a. Troubleshooting CI, removing only the new perf tests Revert "Troubleshooting CI, removing only the new perf tests" This reverts commit d16f739. Troubleshooting CI, up until Iter is included Revert "Troubleshooting CI, up until Iter is included" This reverts commit 7f63395. Troubleshooting CI, up until Iter is removed (excluded) This reverts commit 7f63395. Revert "Troubleshooting CI, up until Iter is removed (excluded)" This reverts commit 32d463b. Troubleshooting CI, tests for Last, Map and Pick excluded Revert "Troubleshooting CI, tests for Last, Map and Pick excluded" This reverts commit 9132fee. Troubleshooting CI: remove all tests but Choose tests Troubleshooting CI, add hang timeout test Revert "Troubleshooting CI: remove all tests but Choose tests" This reverts commit 93cfda5. Troubleshooting CI, removing deliberately hanging test, leave blame-detection in test.yaml Troubleshooting hanging CI: order test cases (currently only within a class, not sure how to change it) Troubleshooting CI, re-enable parallelization of tests and clean code
abelbraaksma
added a commit
that referenced
this issue
Oct 14, 2022
abelbraaksma
added a commit
that referenced
this issue
Oct 14, 2022
Revert "Troubleshooting CI, remove OfXXX, CE, Other and ToXXX" This reverts commit 6856b9e. Turn TaskSeq.empty into a unit-function Troubleshooting CI, remove Item, Pick, CE Revert "Troubleshooting CI, remove Item, Pick, CE" This reverts commit bc6c260. Several attempts at troubleshooting CI issue, see log details and #25 Temporarily killing most tests, to troubleshoot hanging CI Revert "Temporarily killing most tests, to troubleshoot hanging CI" This reverts commit b84fff1. Troubleshooting CI, adding back several new tests Temporarily killing most tests, to troubleshoot hanging CI Revert "Temporarily killing most tests, to troubleshoot hanging CI" This reverts commit b84fff1. Troubleshooting CI, adding back several new tests Revert "Troubleshooting CI, adding back several new tests" This reverts commit 5ecf179. Troubleshooting CI, re-enabling only some tests, including new perf tests Revert "Troubleshooting CI, re-enabling only some tests, including new perf tests" This reverts commit 602cb3a. Troubleshooting CI, removing only the new perf tests Revert "Troubleshooting CI, removing only the new perf tests" This reverts commit d16f739. Troubleshooting CI, up until Iter is included Revert "Troubleshooting CI, up until Iter is included" This reverts commit 7f63395. Troubleshooting CI, up until Iter is removed (excluded) This reverts commit 7f63395. Revert "Troubleshooting CI, up until Iter is removed (excluded)" This reverts commit 32d463b. Troubleshooting CI, tests for Last, Map and Pick excluded Revert "Troubleshooting CI, tests for Last, Map and Pick excluded" This reverts commit 9132fee. Troubleshooting CI: remove all tests but Choose tests Troubleshooting CI, add hang timeout test Revert "Troubleshooting CI: remove all tests but Choose tests" This reverts commit 93cfda5. Troubleshooting CI, removing deliberately hanging test, leave blame-detection in test.yaml Troubleshooting hanging CI: order test cases (currently only within a class, not sure how to change it) Troubleshooting CI, re-enable parallelization of tests and clean code
abelbraaksma
added a commit
that referenced
this issue
Oct 14, 2022
|
This issue was not caused by us, but by an XUnit bug, see: xunit/xunit#2587. CI will run tests sequentially for now, and maybe I should move over to a framework that supports this kind of tests (more) properly. |
abelbraaksma
added
bug
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Attempt at fixing and more analysis: #27.
While working on PR #23, it turned out that the test runner never finished, leading to empty logs (while running or after canceling). The bad behavior of the Github reporting is explained here: actions/runner#1326 (and it was closed with no fix).
The cause appears to be a (potential) race condition. It is yet unclear whether that is caused by the resumable code or something else. The temporary solution in #23 is to disable parallel runs in CI and to improve reporting.
Unfortunately, xUnit reporting is quite useless for cases like this. It only reports when a test has finished, not when it started. While
dotnet runhas a--blame-hang-timeout, which at least solves the issue with GH action reporting, as at the very least, this results in a failed test run.Opening this issue to investigate further, so that we can close #23 meanwhile, as the implementations there have nothing to do with this behavior (removing enough tests removes the issue, but it doesn't matter which ones you remove, ultimately, the race condition comes back).
Example of a run that went on for over 5 hours (!): https://github.com/abelbraaksma/TaskSeq/actions/runs/3250819154/jobs/5334999458 (note: after retention period is over, this log will be gone).
The text was updated successfully, but these errors were encountered: