times. begin/end ignored in HA cluster setup #4995
Comments
|
As discussed offline, more details such as the debug log entries are required. The secondary notification which triggers a reminder notification would obviously mean that the initial notification was skipped somehow. |
dnsmichi
added
needs feedback
|
Any updates? |
|
as I enabled the debug mode and reduced the notification interval to 1m (it was standing on default, not 60s) the notifications worked fine. So I can not reproduce the problem anymore. So I think I will close the Issue, but I need to do some testing to be sure. |
|
Please check that this affects a checkable object which is checked on the left side, but the notification object is triggered and executed on the right side (feedback from @lippserd). |
|
I tested like this: I have one host object for both masters (.10/.20). Set Icinga2a to down in Icingaweb2(node icinga2b). And in the debug logs I get: debug logs: I've started ~15:23 with the notifications. Setup: #---zones.conf (both nodes)---# #---global-templates/notifications.conf---# |
|
Please use code blocks for better formatting (3 backticks each). Can you please summarize what your tests now mean for the reported problem? |
|
The Problem is that I don't get any notification for the down host. Icinga2 is waiting for the 2 minutes as it states in the logs "Not sending reminder notifications for notification object 'icinga2a-host!mail-icingaadmin': before specified begin time (2 minutes)". But after that message I don't get any notification or log message regarding that host. Without the time.begin rows the notifications are working fine. I've got some comments regarding this Issue, stating that the notifications work again if you change the notification name, but that's not the case, at lest for version 2.6.1. I've tested that. I can reproduce the Issue, so if you need more info or a test with a modified setup let me know. |
|
I know now why I've got no notifications. I wasn't testing correctly. The host recovered before the 2 minute limit. I set the check_interval to 3 minutes and it's working now. I have tested it with disabling the notification module on one node and it's working, too. The other node does the sending. So I'm in the same situation as before, I'm not able to reproduce the problem. I will close this Issue. |
|
Hi, I must reopen this Issue. I've tested with two new fresh VMs and I'm able to reproduce the problem now. How to reproduce: I've tested with different combinations (with times.begin, without default assign rule and so on). There is a scenario where the custom notifications over Icingaweb2 are not send but logged in the logs. As I rename the notification object the notifications are working again. You can see the irregularities if you compare the mail log with the debug log(attached). notifications.conf(last state) zones.conf |
|
Comparing to the original issue description, this one seems a little bit different. It would be nice if you could extract your observation from the logs and post it here too in the future. box17 Sending custom notifications as a test case is invalid here, as they are forced and ignore the type and state filters (and so are times and other filters).
Then the notification object is renamed and therefore Icinga 2 restarted in There's also a mismatch between box17 and box18 here, as the logs for box18 introduce a new notification object called 'master_box18!mail-icingaadmin-02`. Timestamps do not match box17, so it is unclear what exactly happened here in your test setup. TL;DR - please explain in detail step by step
Furthermore query the REST API endpoint /v1/objects/notifications for 'master_box18!mail-icingaadmin-01' on both nodes and extract the object attributes. "paused" highlights which box feels responsible for triggering notifications. And please avoid renaming objects to provide a clear strategy on how to reliably reproduce the issue. |
|
Hi, ok, that explains why I couldn't send custom notifications. I have tested again and I get no notifications. Infos: extracted entries from box17 debug.log (no send entries): /etc/icinga2/zones.d/global-templates/notifications.conf /etc/icinga2/zones.d/master/hosts.conf box18 is flagged true: box17: Regards |
|
Boils down, that both instances already exchanged the details about sending a problem notification to the "icingaadmin" user, somehow at least. box17 box18
box17 feels responsible for sending notifications, So, that above and below is what I initially expected in your issue report. There is no further attempt in sending a notification. Which would indicate that the first attempt already set I'd try again with more sane default filters, like box18 does nothing in this regard. Remember the timestamp for Log entries for that specific notification are not available, so it is hard to guess. One thing which seems odd is the Please re-check again and do the following:
|
|
before notification: after process check results to down(before recovery): after Recovery notification 12:51:06 on Regardless I haven't got any notifications, except the recovery notification on box17. Regards |
dnsmichi
removed
the
needs feedback
and others
Hi,
in HA Setup, the times {begin & end) in a notification object are ignored(I get the notification as soon as its changing to hard state). See logs. And I should get the notifications in a 60s interval(service), but I just get it one time.
Config Master:
notifications.conf:
Icinga2 Logs:
/var/mail/icinga:
Regards
The text was updated successfully, but these errors were encountered: