Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

actions: Use pytest-rerunfailures for pytest-xdist worker crash #1273

Merged
merged 1 commit into from
Feb 14, 2024

Conversation

zmedico
Copy link
Member

@zmedico zmedico commented Feb 14, 2024

Since pytest-xdist workers crash intermittently for the multiprocessing spawn start method, use pytest-rerunfailures to detect and handle this case. Only use pytest-rerunfailures for the spawn start-method since that is the only case where we've observed intermittent pytest-xdist worker crashes, and use --only-rerun 'worker .* crashed while running' to ensure that rerun only triggers for worker crashes.

Bug: https://bugs.gentoo.org/924416

@thesamesam
Copy link
Member

thesamesam commented Feb 14, 2024

bleh. I'm not hard-against it, just worried we're papering over the problem with this. But I suppose it's better than flaky CI which may-or-may not fail for unrelated changes...

I haven't looked at what the actual crash is - do you have any ideas for it yet? (Are we to blame?)

@zmedico
Copy link
Member Author

zmedico commented Feb 14, 2024

bleh. I'm not hard-against it, just worried we're papering over the problem with this. But I suppose it's better than flaky CI which may-or-may not fail for unrelated changes...

It's a pytest-xdist problem though, since it doesn't trigger without it, and pytest-rerunfailures is designed to handle it since pytest-dev/pytest-rerunfailures#158.

I haven't looked at what the actual crash is - do you have any ideas for it yet? (Are we to blame?)

I haven't been able to trigger it locally, so that makes it extra mysterious. Since using a private multiprocessing context for the spawn start method didn't help in #1270, I think we can safely say it's not our fault. It might be pytest-dev/execnet#96 since that's an "intermittent" issue.

The --only-rerun 'worker .* crashed while running' bit makes the workaround not so bad, since rerun is only triggered when absolutely necessary.

@zmedico zmedico force-pushed the bug_924416_pytest-rerunfailures branch from 4cab84d to 5c66967 Compare February 14, 2024 18:46
@zmedico zmedico marked this pull request as ready for review February 14, 2024 18:46
@zmedico zmedico requested a review from thesamesam February 14, 2024 18:47
@thesamesam
Copy link
Member

AH, thank you, sorry, I thought it was an interaction between one of our setup fixtures and multiprocessing, just exposed by pytest-xdist.

Since pytest-xdist workers crash intermittently for the multiprocessing
spawn start method, use pytest-rerunfailures to detect and handle this
case. Only use pytest-rerunfailures for the spawn start-method since
that is the only case where we've observed intermittent pytest-xdist
worker crashes, and use --only-rerun 'worker .* crashed while running'
to ensure that rerun only triggers for worker crashes.

Bug: https://bugs.gentoo.org/924416
See: pytest-dev/execnet#96
Signed-off-by: Zac Medico <zmedico@gentoo.org>
@zmedico zmedico force-pushed the bug_924416_pytest-rerunfailures branch from 5c66967 to ffcb1eb Compare February 14, 2024 19:08
@gentoo-bot gentoo-bot merged commit ffcb1eb into gentoo:master Feb 14, 2024
12 checks passed
@zmedico zmedico deleted the bug_924416_pytest-rerunfailures branch February 14, 2024 19:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants