Investigate test flake in securedrop-admin integration tests

[redshiftzero]

It looks like there's a test flake in the integration tests: see this build

We should resolve quickly as test flakes can be very annoying

[kushaldas]

I could not reproduce this :(

[msheiny]

yeah i havent caught it yet locally either :|

[redshiftzero]

Thanks for checking y'all - removing P-high label - if it doesn't happen again in the next week or so we can close and chalk it up to temporary network issues (a reasonable explanation in that case imho).

[redshiftzero]

Ah, here we go, it happened again today: https://circleci.com/gh/freedomofpress/securedrop/15182

These looks to be keyserver issues. Note that this might be turning up an intermittent issue in securedrop-admin update, not necessarily an issue in the integration test, so that should be investigated

[redshiftzero]

Once more: https://circleci.com/gh/freedomofpress/securedrop/15266

(and I put the P-high label back on as flaky tests in CI is a recipe for much sadness)

[conorsch]

Caught another one: https://circleci.com/gh/freedomofpress/securedrop/15947

[conorsch]

Still getting these: https://circleci.com/gh/freedomofpress/securedrop/24343

[eloquence]

We've committed do a time-boxed (2 hour) investigation during the 4/3-4/17 sprint, hopefully getting closer towards resolution.

[conorsch]

Spent some time tinkering with the tests in question. The problematic tests are:

tests/test_integration.py::TestGitOperations::test_update
tests/test_integration.py::TestGitOperations::test_update_fails_when_no_signature_present
tests/test_integration.py::TestGitOperations::test_update_with_duplicate_branch_and_tag

Briefly experimented with adding the @flaky decorator on the class, similar to #4004, but that was insufficient to resolve the problems. The errors reported after patching with the decorator included:

================================= 60 passed in 29.59 seconds ====
  py2 runtests: commands[1] | /home/user/securedrop/admin/.tox/py2/bin/coverage combine --append
  No data to combine
  ERROR: InvocationError: '/home/user/securedrop/admin/.tox/py2/bin/coverage combine --append'
  _____________________________________ summary ________________________________________
    pylint: commands succeeded
      flake8: commands succeeded
      ERROR:   py2: commands failed

It appears that the coverage combine action is failing, due to an interaction with the test reruns triggered by flaky. Cursory investigation shows we may be able to leverage the .coveragerc file to accommodate, but haven't taken a stab at it yet.