Add jenkins:report task for test failures #22682
Conversation
joshdover
force-pushed
the
flaky-reporting
branch
from
cf35518 to
6af1aea
Compare
💚 Build Succeeded |
| owner: GITHUB_OWNER, | ||
| repo: GITHUB_REPO, | ||
| number: existingIssue.number, | ||
| state: 'open', // Reopen issue if it was closed. |
There was a problem hiding this comment.
I am leaning towards creating a new issue here. The reason for the failure could be different than previous failures. Thoughts?
There was a problem hiding this comment.
It could be, but I think it's also more likely that we think we fixed a test and closed the issue just for it to fail a few days later (maybe it only fails 1% of the time). In that case, it seems useful to know that it failed recently and to be able to quickly see what we tried to fix it.
I can't think of a great reason that the historical context for a given test failure would be bad to surface up.
There was a problem hiding this comment.
Ok - we can stick with this and always change later. It's possible we could also create an arbitrary timeout. So after an issue was closed for X amount of time a new issue is created.
💔 Build Failed |
|
retest |
💔 Build Failed |
|
retest |
💚 Build Succeeded |
💚 Build Succeeded |
|
|
|
@tylersmalley Should I backport this to 6.x and 5.x? |
* Add jenkins:report task * PR comments
Fixes #21965
Adds a new
jenkins:reportgrunt task that will pick up any test failures found in JUnit XML files and creates / updates issues in Github with theflaky-failing-testlabel. This will only be ran on tracked branches, I have an upcoming infra change for adding this to the jenkins jobs for those branches.Here's an example of what an automated issue will look like: joshdover#23