Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix flaky sigstore tests on Windows #297

Merged
merged 1 commit into from
Aug 20, 2024

Conversation

mihaimaruseac
Copy link
Collaborator

@mihaimaruseac mihaimaruseac commented Aug 20, 2024

Summary

It seems that I failed to properly mock Sigstore in testing and that was causing some flakiness on Windows. This should be fixed now.

I'm running each CI 5 times before sending this for review to make sure.

Release Note

NONE

Documentation

NONE

Signed-off-by: Mihai Maruseac <mihaimaruseac@google.com>
@mihaimaruseac mihaimaruseac changed the title WIP Fix flaky sigstore tests on Windows Aug 20, 2024
@mihaimaruseac
Copy link
Collaborator Author

Five attempts of CI all passed, so I think this is good to go

@mihaimaruseac mihaimaruseac marked this pull request as ready for review August 20, 2024 21:16
@mihaimaruseac mihaimaruseac requested review from a team as code owners August 20, 2024 21:16
@mihaimaruseac mihaimaruseac added this to the V1 release milestone Aug 20, 2024
@spencerschrock
Copy link
Contributor

Should we look into running unit tests without networking? I took a look at what sigstore-python does, and played around with the following (I was unable to get it to work with hatch test):

unshare --map-root-user --net pytest -v .

before:

=============================================================================== short test summary info ===============================================================================
FAILED tests/signing/sigstore_test.py::TestSigstoreSigning::test_sign_verify_artifacts - sigstore.errors.TUFError: Failed to refresh TUF metadata
FAILED tests/signing/sigstore_test.py::TestSigstoreSigning::test_sign_verify_dsse_single_digest - sigstore.errors.TUFError: Failed to refresh TUF metadata
FAILED tests/signing/sigstore_test.py::TestSigstoreSigning::test_sign_verify_dsse_digest_of_digests - sigstore.errors.TUFError: Failed to refresh TUF metadata
FAILED tests/signing/sigstore_test.py::TestSigstoreSigning::test_sign_verify_dsse_digests - sigstore.errors.TUFError: Failed to refresh TUF metadata
FAILED tests/signing/sigstore_test.py::TestSigstoreSigning::test_sign_verify_dsse_digest_of_digests_sharded - sigstore.errors.TUFError: Failed to refresh TUF metadata
FAILED tests/signing/sigstore_test.py::TestSigstoreSigning::test_sign_verify_dsse_digests_sharded - sigstore.errors.TUFError: Failed to refresh TUF metadata
FAILED tests/signing/sigstore_test.py::TestSigstoreSigning::test_sign_verify_mocked_prod - sigstore.errors.TUFError: Failed to refresh TUF metadata
FAILED tests/signing/sigstore_test.py::TestSigstoreSigning::test_sign_verify_mocked_prod_oidc - sigstore.errors.TUFError: Failed to refresh TUF metadata
FAILED tests/signing/sigstore_test.py::TestSigstoreSigning::test_sign_verify_mocked_ambient - sigstore.errors.TUFError: Failed to refresh TUF metadata
FAILED tests/signing/sigstore_test.py::TestSigstoreSigning::test_sign_digest_as_bytes - sigstore.errors.TUFError: Failed to refresh TUF metadata
FAILED tests/signing/sigstore_test.py::TestSigstoreSigning::test_sign_bytes_as_digest - sigstore.errors.TUFError: Failed to refresh TUF metadata
FAILED tests/signing/sigstore_test.py::TestSigstoreSigning::test_verify_not_into_json_payload - sigstore.errors.TUFError: Failed to refresh TUF metadata
FAILED tests/signing/sigstore_test.py::TestSigstoreSigning::test_verify_not_intoto_statement - sigstore.errors.TUFError: Failed to refresh TUF metadata
FAILED tests/signing/sigstore_test.py::TestSigstoreSigning::test_verify_intoto_predicate_not_matched - sigstore.errors.TUFError: Failed to refresh TUF metadata
FAILED tests/signing/sigstore_test.py::TestSigstoreSigning::test_verify_intoto_single_digest_more_than_one_digests - sigstore.errors.TUFError: Failed to refresh TUF metadata
FAILED tests/signing/sigstore_test.py::TestSigstoreSigning::test_verify_intoto_digest_of_digests_more_than_one_digests - sigstore.errors.TUFError: Failed to refresh TUF metadata
FAILED tests/signing/sigstore_test.py::TestSigstoreSigning::test_verify_intoto_digest_of_digests_invalid_root_digest - sigstore.errors.TUFError: Failed to refresh TUF metadata
FAILED tests/signing/sigstore_test.py::TestSigstoreSigning::test_verify_intoto_digest_of_shard_digests_more_than_one_digests - sigstore.errors.TUFError: Failed to refresh TUF metadata
FAILED tests/signing/sigstore_test.py::TestSigstoreSigning::test_verify_intoto_digest_of_shard_digests_invalid_root_digest - sigstore.errors.TUFError: Failed to refresh TUF metadata
=========================================================================== 19 failed, 235 passed in 6.56s ============================================================================

Testing on this PR everything passes :)

Copy link
Contributor

@spencerschrock spencerschrock left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not 100% familiar with unittest.mock but the CI seems stable and my unshare experiment looked good 👍

@mihaimaruseac
Copy link
Collaborator Author

Running with networking disabled was a good way to test that we completely mocked out Sigstore, this is a good suggestion -- thank you. Might be too late for this now, but we can try if we detect other flakyness / need to mock other services?

@mihaimaruseac mihaimaruseac merged commit 5a88752 into sigstore:main Aug 20, 2024
17 checks passed
@mihaimaruseac mihaimaruseac deleted the fix-mocking-on-windows branch August 20, 2024 21:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants