Add notification result store/sync

[dnsmichi]

Story

Every now and then notifications do not work, and the last resort is the executed command script. Instead of greps inside the icinga2.log, it would be nice to have the results available.

Having an HA enabled zone with two masters also renders this kind of complicated, as notifications run balanced similar to checkables. Some commands are logged on masterA, others on masterB. Synchronising the notification result between them allows us to keep track of it independently.

This also comes to mind with having one backend writer active, while the secondary is paused. Both need to synchronise the notification result event, and then one of them writes that to the backend (IDO won't receive that, just IcingaDB).

Solution

Instead of only logging the executed notification command line (notice level if ok, warning if failed), we want to have the command line available in memory for better analysis and debugging.

Furthermore, we'll need the command line via event available for Icinga DB and store the historical relevant parts next to the notified user, etc.

[dnsmichi]

@Crunsher This might be relevant for IcingaDB.

[dnsmichi]

I'll merge this now, and let everyone test the snapshot packages then.