Adding Prometheus alert manager exporter #437
Conversation
Codecov Report
@@ Coverage Diff @@
## master #437 +/- ##
==========================================
+ Coverage 67.80% 67.91% +0.10%
==========================================
Files 103 104 +1
Lines 7148 7178 +30
Branches 795 798 +3
==========================================
+ Hits 4847 4875 +28
- Misses 2064 2066 +2
Partials 237 237
|
| host = alert_cfg.alert_manager_host | ||
| port = alert_cfg.alert_manager_port | ||
| for alert in alerts: | ||
| print(alert) |
There was a problem hiding this comment.
remove this print. Or Is someone listening on stdout?
delfin/api/v1/alerts.py
Outdated
| @@ -67,6 +69,11 @@ def show(self, req, id): | |||
| for alert in alert_list: | |||
| alert_util.fill_storage_attributes(alert, storage) | |||
|
|
|||
| try: | |||
| self.alert_exporter.dispatch(ctx, alert_list) | |||
There was a problem hiding this comment.
Any reason for adding alert exporter dispatch in wsgi server? Can we handle it in task manager server?
There was a problem hiding this comment.
Removed this. there sync API in alert to handle alert pull task from task manager. GET Alert will respond directly to client through REST.
| @@ -280,7 +280,56 @@ def clear_alert(self, context, alert): | |||
| pass | |||
|
|
|||
| def list_alerts(self, context, query_para=None): | |||
| pass | |||
| alert_list = [{ | |||
There was a problem hiding this comment.
Can we make some fields random ?
There was a problem hiding this comment.
Done, alert id and occur_time is random
| ] | ||
|
|
||
| CONF.register_opts(alert_mngr_opts, "PROMETHEUS_ALERT_MANAGER_EXPORTER") | ||
| alert_cfg = CONF.PROMETHEUS_ALERT_MANAGER_EXPORTER |
There was a problem hiding this comment.
Any reason to make alert_cfg global ?
There was a problem hiding this comment.
Values are read once and never changed after that.. No other specific reason, all conf read in delfin is global to file scope AFIK.
| dict["annotations"]["summary"] = alert.get("description") | ||
|
|
||
| self.alerts.append(dict) | ||
| requests.post('http://' + host + ":" + port + '/api/v1/alerts', |
There was a problem hiding this comment.
Can we check post response status and report?
|
LGTM |
Get K8s Cluster IP before deploying Multi-Cloud Pods.
What this PR does / why we need it:
New alert exporter to prometheus alert manager
recieved alert in delfin ( through trap and sync ) will be pushed to alert manager through this exporter.
Which issue this PR fixes (optional, in
fixes #<issue number>(, fixes #<issue_number>, ...)format, will close that issue when PR gets merged): fixes #NASpecial notes for your reviewer:
This is rwork of PR #429
Release note:
Test Report:
Sync alerts for storage <storage_id>
Check task process logs
Check alert manager for alerts
