Flaky bot being timed out before it can report issue

[anguillanneuf]

python-pubsublite samples were quietly failing from 2/23 to 3/9, but Flakybot didn't report any failures because the tests timed out (>180 minutes) before Flakybot ran.

TL;DR: The PR that broke these samples came from python-pubsub on 2/22. It caused a method that cancels a streaming pull future from a Pub/Sub Lite subscribe call to hang.

Is there something that Flakybot could do to catch such failures in the future?

[tbpg]

Thanks for the report. I've been thinking about this, and I'm not sure of a good solution.

One idea is to set a bash TRAP so that flakybot always runs? From the log, I'm not sure it would have run, though:

ERROR: Aborting VM command due to timeout of 10800 seconds

We can use flakybot to say "the build failed" even when it doesn't give a specific test case that failed. But, in this case, I don't think that would help. We know what case failed, the VM just gets shut down.

[JustinBeckwith]

Hmmm. Brainstorming time - I wonder if we could retroactively look at CheckSuiteRuns on a cron for tests that run on the default branch? Yeah we don't have access to run code if the kokoro VM loses it's gourd, but GitHub would still see a failed status check, right? I think there's some nuance to how the GitHub API works with Kokoro registered jobs vs GitHub Action jobs, so some research would be needed.

[chingor13]

Note: nightly jobs are not guaranteed to create status checks on the commits they run on unless we start a policy that the should.

[tmatsuo]

Here is my 2 cents.

The nightly build can fail even before flakybot can do anything useful. We should have a way of checking nightly build status without flakybot. Closing, but feel free to reopen with a rationale.