Notify before silence ends

[marcusboden]

Enhancement Proposal

When a silence expires, we will get the alert again. That makes sense and should be this way.
However, oftentimes the reason for the silence has still not been fixed. The alert itself does not contain any information about the expired silence, sending us down the rabbit hole of investigating this alert again.

I would like a notification before the silence expires so that the person responding to alerts will know what to expect and can extend the silence on time.

amtool has some features for that and I'm thinking about building that with a cron job running amtool silence query --within 1h and using the output to create a manual alert with amtool. But it would be nice for this to be integrated into cos.

[marcusboden]

Sorry, wrong repo. Meant to open it here: canonical/alertmanager-k8s-operator#268