Skip to content

Notify_splunk

Jump to bottom
Chris Caron edited this page May 16, 2024 · 1 revision

Splunk/VictorOps On-Call Notifications

Account Setup

  1. Create an account with Splunk On-Call (previously VictorOps). Then set up your REST endoint
    splunk-01
  2. Access your API Key from this link.
    splunk-02
    It will look something like this: 
    https://alert.victorops.com/integrations/generic/20131114/alert/1234abcd-c11c-1ad1-a1a1-12345678abcd/$routing_key
                                                                ^                                  ^ ^          ^
                                                                |------------ apikey --------------| |          |
                                                                                                     | routing  |
                                                                                                    /    key     \
                                                                                                   | placeholder |
                                                                                                   |-------------|
  3. Finally you will need to define a routing_key which can be done from Settings -> Route Keys
    splunk-03
  4. The entity_id is used to ensure your message can be triggered and acknowledged. It's effectively a key. If you don't provide one then Apprise will generate one for you (the same one every time).

Syntax

Valid syntax is as follows:

  • splunk://{routing_key}@{apikey}
  • splunk://{routing_key}@{apikey}/{entity_id}
  • victorops://{routing_key}@{apikey}
  • victorops://{routing_key}@{apikey}/{entity_id}
  • https://alert.victorops.com/integrations/generic/20131114/ alert/{apikey}/{routing_key}
  • https://alert.victorops.com/integrations/generic/20131114/ alert/{apikey}/{routing_key}/{entity_id}

Parameter Breakdown

Variable Required Description
apikey Yes The REST API key associated with your Splunk account
routing_key Yes One of the routing_key values you associated within your Splunk account
entity_id No A key you wish to generate your trigger from. Keys allow you to alert, ackowledge and/or resolve the same notification later on.
action No The action you wish to perform with your Splunk/VictorOps Apprise notification. the following options are available to you:
⚪ map: Use Apprise (or custom) action mappings based on the Notification Type. Hence a warning from Apprise triggers a WARNING on Splunk, while a failure triggers a CRITICAL Splunk message (triggering an incident). Finally a success triggers a RECOVERY Spunk message (clearing an incident). map is the default action if nothing is specified.
⚪ warning: Reguardless of the Apprise notification, ALWAYS trigger a Splunk WARNING message.
⚪ critical: Reguardless of the Apprise notification, ALWAYS trigger a Splunk CRITICAL message.
⚪ acknowledgement: Reguardless of the Apprise notification, ALWAYS trigger a Splunk ACKNOWLEDGEMENT message.
⚪ info: Reguardless of the Apprise notification, ALWAYS trigger a Splunk INFO message.
⚪ recovery: Reguardless of the Apprise notification, ALWAYS trigger a Splunk RECOVERY message.

Custom Splunk/On-Call Event Mapping

You can have Apprise take a unique Splunk/On-Call action depending on the notification that is triggered if you use the (default) action of map with this integration.

First consider that Splunk supports the following settings:

  1. CRITICAL: Triggers an incident
  2. WARNING: May trigger an incident, depending on your settings
  3. ACKNOWLEDGEMENT: Acks an incident
  4. INFO: Creates a timeline event but does not trigger an incident
  5. RECOVERY: Resolves an incident

By default if the action is set to map, then Apprise maps itself to the following:

  • Apprise info 👉 Splunk INFO
  • Apprise warning 👉 Splunk WARNING
  • Apprise failure 👉 Splunk CRITICAL
  • Apprise success 👉 Splunk RECOVERY

If you wish to map these differently, you simply need to use the : (colon) when over-riding an apprise variable. Hence, if you wanted to map the (Apprise) info to (Splunk) ACKNOWLEDGEMENT instead, your URL would have ?:info=acknowledgement. You can also short-form it like ?i=a if you wanted to as well (same effect).

You can add as many re-mappings as you want. Just be certain to add a colon (:) infront of the Apprise notificaiton type first.

Testing

Send a Spunk On-Call alert to fail our database service:

# Assuming we want to trigger a Splunk CRITICAL message (we send a Apprise Failure)
# Assuming our {apikey} is 134b8gh0-eba0-4fa9-ab9c-257ced0e8221
# Assuming our {route_key} is database
apprise -vv -t "Test Message Title" -b "Test Message Body" -n failure \
   splunk://database@134b8gh0-eba0-4fa9-ab9c-257ced0e8221

We can recover from the failure above by just doing the following:

# Assuming we want to trigger a Splunk ACKNOWLEDGEMENT message (we send a Apprise Success)
# Assuming our {apikey} is 134b8gh0-eba0-4fa9-ab9c-257ced0e8221
# Assuming our {route_key} is database
apprise -vv -t "Test Message Title" -b "Test Message Body" -n success \
   splunk://database@134b8gh0-eba0-4fa9-ab9c-257ced0e8221

Send a Spunk message while re-mapping our keys around:

# Assuming we want the (Apprise) `info` to to trigger a Splunk RECOVERY
# Assuming we want the (Apprise) `warning` to always trigger a Splunk CRITICAL
# Assuming our {apikey} is 134b8gh0-eba0-4fa9-ab9c-257ced0e8221
# Assuming our {route_key} is database
# In this example we'll send a warning message (which will be a CRITICAL)
apprise -vv -t "Test Message Title" -b "Test Message Body" -n warning \
   splunk://database@134b8gh0-eba0-4fa9-ab9c-257ced0e8221?:info=rec&:warn=crit

Reguardless of what message type we sent, we always set it as RECOVERY:

# Assuming we always trigger a recovery
# Assuming our {apikey} is 134b8gh0-eba0-4fa9-ab9c-257ced0e8221
# Assuming our {route_key} is database
# In this example we'll send a failure message (which will be a RECOVERY due to our settings)
apprise -vv -t "Test Message Title" -b "Test Message Body" -n failure  \
   splunk://database@134b8gh0-eba0-4fa9-ab9c-257ced0e8221?:action=recovery

Quick Links:

Clone this wiki locally