How to Alertmanager running shell-script

[ihtiking]

Server CentOS7 and intalled alertmanager in OpenShift 3.11.

When some alert is triggered, i want run shell-script.

Now, yml-file in Config Map alertmanager contains block receivers:

• name: alert1
  webhook_configs:
  • url: http://......

and he work fine.

In end point i want run command "curl -X POST "https://some-site/?var=text1""

Please, help.

Linux 3.10.0-862.14.4.el7.x86_64 x86_64

alertmanager, version 0.15.1 (branch: HEAD, revision: 8397de1)
build user: root@prometheus-alertmanager-binary-3-build
build date: 20180720-09:32:24
go version: go1.10.3

prometheus, version 2.3.2 (branch: HEAD, revision: 71af5e29e815795e9dd14742ee7725682fa14b7b)
build user: root@prometheus-binary-6-build
build date: 20180720-09:19:12
go version: go1.10.3

[simonpasquier]

You should have a look at https://github.com/imgix/prometheus-am-executor or https://github.com/adnanh/webhook

[ihtiking]

Big thanks. I'm gotting this - https://github.com/adnanh/webhook. This is solution IMHO very simple.

My steps:

1. Create hooks.json:
   [
   {
   "id": "runscript",
   "execute-command": "/usr/bin/sendmessage.sh",
   "command-working-directory": "/usr/bin"
   }
   ]

2. Create shell-script sendmessage.sh:
   #!/bin/bash
   curl -I -X POST "https://<your_site_which_you_want_execute>"

3. Create Dockerfile:
   #FROM almir/webhook - this is string i used when create compiled file "webhook" in first build docker.
   FROM centos:7
   COPY webhook /usr/bin/
   COPY hooks.json /var/webhook/
   COPY sendmessage.sh /usr/bin/
   RUN chmod +x /usr/bin/sendmessage.sh
   CMD ["/usr/bin/webhook", "-verbose", "-hooks=/var/webhook/hooks.json", "-hotreload"]

4. Create some folder and copy here webhook-binary file, hooks.json and shell-script sendmessage.sh.

5. Next, run command:
   docker build -t webhook-image . --tag=webhook-image && docker run --network="host" -d -p 9000:9000 --name=webhook webhook-image && docker start webhook

6. Next in browser open url http://<your_ip_addr>:9000/hooks/runscript
   When open your site on port 9000, webhook is executed shell-script.

It's very great!!!

[simonpasquier]

I'm closing the issue then.

[mchugh19]

The proposed webhook service seems like a bit more work and additional points of complexity and failure than having alert manager support script running. Can this issue be reopened as a feature request?

[simonpasquier]

@mchugh19 unfortunately no:

1. Alertmanager's scope isn't about remediation but rather notifying 3rd party systems.
2. We don't accept new integrations currently and the webhook integration is the recommended approach.

[alwaysastudent]

It would be great if alert manager can support running shell scripts.

[sukhinin]

Running external scripts is the standard and flexible way to integrate with third-party systems. Definitely would like to see it supported.

[dgl]

https://github.com/adnanh/webhook is a generic webhook receiver written in Go, you can easily use this with alertmanager's webhooks in order to run shell commands.

[sukhinin]

@dgl I'm aware of it. That's one more service we have to run, monitor and support. It just makes system more complex and adds yet another moving part.

[MalloZup]

in case it help anyone, I do have written a blogpost for helping people using this pattern:

https://mallozup.github.io/posts/self-healing-systems-with-prometheus/

[zffocussss]

https://github.com/adnanh/webhook is a generic webhook receiver written in Go, you can easily use this with alertmanager's webhooks in order to run shell commands.

it is a very useful and common way to perform many kinds of tasks, like scripting,http and tcp request.

[JohnVillalovos]

@mchugh19 unfortunately no:

1. Alertmanager's scope isn't about remediation but rather notifying 3rd party systems.

2. We don't accept new integrations currently and the webhook integration is the recommended approach.

@simonpasquier

Did this policy change? I see that Telegram support was added in 2022 in #2827

Would that mean a PR for shell-script support would be considered?

Seems like if shell-script support was added then it would be very easy for people to add support for notifying 3rd party systems without needing changes to alertmanager. For example Telegram support may have been able to have been done using a shell-script.

[venerari]

How do we put priority container on adnanh webhook beside auto replication and priority network if it run as container? So the email will for sure run first and it should be on another different kubernetes/docker cluster right?

[ankur07garg]

having issues is sending payload from alertmanager.yml to Google space/google chat via webhook. getting 400, it is not letting use template or payload in webhook for sending message to google space. Please assist or some example config would be helpful