Slack Receives Incorrect or Duplicate Alerts from Jenkins

[7hunderbird]

User Story or Problem Statement

Today we've noticed that there are distracting and in-accurate alerts coming into the #cms-notifications slack channel.

1. Alerts for jobs that don't exist.
2. Duplicate alerts.

Alerts for jobs that don't exist

When this issue emerged, the most recent build in the cms-db-sanitize job was number 103121.

As you can see in the figure above, the errors showed a URL to a build job number 103187. Each of these job numbers that were about 60 builds ahead led to a 404 page not found because the builds had not run yet.

When we looked at the live status of this job, it was succeeding and so the main issue here is that the noise and distraction that this causes. 😢

Duplicate alerts

At first the notification that the prod deploy was about to go out was duplicated.

Then, when the prod deploy had determined that there was no new version to ship out, it ALSO duplicated the "ABORT" which is why I created this ticket.

Not only did we get duplicate errors from the prod deploy "ABORT" notification, but the first one also had the problem of linking to the wrong job that was somehow in the future.

Description or Additional Context

Steps for Implementation

Acceptance Criteria

• There are no more duplicate alerts coming into #cms-notifications slack channel
• There are no alerts for non-existent jobs

[7hunderbird]

Searching in slack there was a time period around April 17th and 18th where there were repeated "cms/cms-db-sanitize failed" errors.

[7hunderbird]

With the other jenkins server turned off the duplicate alerts from Jenkins are gone and this is resolved.