You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Had a discussion with @nreese about #42989 and how currently the Percy check is done at the Github level, which is working, however the Jenkins CI job sends an email about the test status, which does not look at the Percy results. So you can get a Github failure check on the Percy tests, but an email that is passing (green heart) which can be confusing. Also according to Nathan, the email is the primary tool used to merge their PR and it might be missed seeing the red 'X' on the PR itself, so we would need to add a validation so it's part of the email notifications too.
I personally don't see the fact that Percy validation status is separate from the Jenkins test status as something that we should fix. Just like the CLA check, up-to-date check, or label checks, the status of the Percy validation is not managed or tracked by Jenkins.
When a user intentionally makes visual changes to Kibana the only way to update the snapshots is to run CI, visit the Percy UI, and then approve the changes. If we included the percy status in the Jenkins status then users would also have to re-run Jenkins in order to have a complete CI run without any visual changes, which would be a bad user experience.
Github already solves this for us by merging the different statuses from the Github checks into a single status for the whole commit/PR, I don't think we need to invent anything new here.
This is mainly for developers who rely on email notifications on whether their PR passed or failed, it would be nice to add some indication to those emails if there are visual regressions, as right now they would have to check it in Github. If only visual regression tests failed, then they don't need to rerun, they just need to approve the changes in the Percy dashboard and the Github check will turn green for the PR.
Had a discussion with @nreese about #42989 and how currently the Percy check is done at the Github level, which is working, however the Jenkins CI job sends an email about the test status, which does not look at the Percy results. So you can get a Github failure check on the Percy tests, but an email that is passing (green heart) which can be confusing. Also according to Nathan, the email is the primary tool used to merge their PR and it might be missed seeing the red 'X' on the PR itself, so we would need to add a validation so it's part of the email notifications too.
cc: @spalger
Part of #33817
The text was updated successfully, but these errors were encountered: