Icinga not sending notifications from primary master in HA setup

[Sec42]

Describe the bug

I am having an issue in icinga where the primary master is sometimes not sending notifications despite writing to IDO that it did.

When it works, the log looks like this:

Apr 30 15:14:15 munnvmonpmac11 icinga2: Checking for configured notifications for object 'i-03578159cbb444b9a-node-mi-mgmt-icinga!ti-ftp-feed-1E_w'
Apr 30 15:14:15 munnvmonpmac11 icinga2: Completed sending 'Problem' notification 'i-03578159cbb444b9a-node-mi-mgmt-icinga!ti-ftp-feed-1E_w!topx' for checkable 'i-03578159cb
Apr 30 15:14:15 munnvmonpmac11 icinga2: Sending 'Problem' notification 'i-03578159cbb444b9a-node-mi-mgmt-icinga!ti-ftp-feed-1E_w!topx' for user 'topx'
Apr 30 15:14:15 munnvmonpmac12 icinga2: Checking for configured notifications for object 'i-03578159cbb444b9a-node-mi-mgmt-icinga!ti-ftp-feed-1E_w'

And the IDO has a record for it:

Apr 30 15:14:15 | 440292 | i-03578159cbb444b9a-node-mi-mgmt-icinga | ti-ftp-feed-1E_w | ERR: 1E_w: 0 detecte | ERR: 1E_w: | Amadeus To |   1

But one other time when the notification was not sent, the log looks like this:

Jun 10 00:04:13 munnvmonpmac12 icinga2: Checking for configured notifications for object 'i-03578159cbb444b9a-node-mi-mgmt-icinga!ti-ftp-feed-1E_w'

note that only the secondary master (munnvmonpmac12) wrote anything into the log.

The IDO record is present, though:

Jun 10 00:04:13 | 447222 | i-03578159cbb444b9a-node-mi-mgmt-icinga | ti-ftp-feed-1E_w | ERR: 1E_w: 0 detecte | ERR: 1E_w: | Amadeus To |   1

The server was not being reloaded at that time.

Your Environment

Include as many relevant details about the environment you experienced the problem in

• Version used (icinga2 --version):
  icinga2 - The Icinga 2 network monitoring daemon (version: r2.10.5-1)

Additional context

Above you can see that in the working case both servers logged a
"Checking for configured notifications" message, while in the broken case only one of them logged that message, I wrote a quick script to search the logs for these instances.

For the current logfile, I get the following numbers:
4248 cases where both servers logged this message
59 cases where only the secondary logged this message

I am not sure what causes this, but it seems like the icinga primary master is loosing about 1% of notification events.

I'm currently trying to catch one of these cases with the debuglog enabled, but due to it being very verbose/large, I have not yet succeeded.

If you have any suggestions on how to proceed/debug, please let me know.

[dnsmichi]

At the time when those missing notifications happen, is there a reload going on which allows the secondary note to take over and also write to the IDO database? I.e. are you able to correlate the missing timestamps to those on the secondary master's log at that time?

If there's a reload going on, I could think of #7086 which has been fixed for 2.11. The RC packages are available for testing, maybe you can give them a whirl.

[Al2Klimov]

IMAO the lack of external feedback for a long time indicates that that feedback will never happen. Therefore closing this one.

Feel free to re-open if the problem persists with the latest Icinga 2 version as long as you provide the desired information.