Review notification engine

[dnsmichi]

Will be updated, @lippserd

Engine

• Reminder notifications should use a "scheduler" similar to checks, instead of the current timer [dev.icinga.com #12584] Add notification scheduler (similar to check scheduler) #4594
• Avoid code duplication from filter checks:
  • event triggered from process check result
  • reminder notifications
• Keep track of filtered notifications and schedule them at a later point: Add support to track filtered/unfiltered notifications and do not re-notify if filtered states don't change #4503
  • Start notifications after a downtime ends Notification behaviour after Downtime ends #5919
• Evaluate a better design for "one time notifications", instead of interval=0 Inconsistent behavior between notifications outside timeperiods #7099
• Notification results Add notification result store/sync #6722

Cluster

• Delayed notifications (times.begin/end) in HA enabled zones times. begin/end ignored in HA cluster setup #4995

Others

• Problem IDs for notifications Runtime Macros "last_problem_start_time" - alternative to "problem_id" #5158

[Crunsher]

Also relevant: #7086
Being able to reschedule Notifcations to after the Startup is done will fix this.

[dgoetz]

Is #6787 the same as #5919 or should it be added to the list?

[dnsmichi]

Technically the same issue/solution required.

[dnsmichi]

Assigning to 2.11 as ongoing tasks. Cannot guarantee that all issues will be solved with that release.

[Crunsher]

Current state and TODOS:

• Using a scheduler for problem notifications is great.
• Unit tests for a component are hard

More on the issue of HA notifications:
When losing a node in HA, it takes a while for the remaining Node to update paused on the Notifications. Paused notifications are not sent because we assume someone else will take care of it. That way it's quite easy to lose Notifications. For problem notifications that is not too bad as you'll just miss one renotification and receive the next in the worst case.
For all other (Recovery, DowntimeStart... etc) this is a known Problem.
The only 100% way I can come up with to guarantee each and every notification is sent is for the notification components to start communicating.

[dnsmichi]

Assigned the selected issues to 2.11, the rest is up for 2.12.

[widhalmt]

ref/NC/627311

[widhalmt]

Could you add #6167 to list of issues to tackle during a rework? We have a setup where an object gets not-Ok outside of notification periods and no notification is sent when the notification period is reached. This should not be the case because problems could get away unnoticed.

[dnsmichi]

This issue is on hold for now, and needs a reorganisation with also planning the steps and time again. The previous attempt in solving this was not satisfying.