Expose SLM Metrics

[Evesy]

The exporter currently exposes metrics directly around snapshots but it would be good to also expose metrics from SLM [1] around scheduled snapshots.

When using SLM snapshots a scheduled snapshot can fail without the underlying Elastic snapshots having any failure, e.g. with ignore_unavailable set to false if I purposefully turn a index red you can get a snapshot to fail like the below:

{
  "type": "snapshot_exception",
  "reason": "[gcs:kibana-2022.03.24-rmug_mrwtauw_hac6mjchg/uw4vqewLQVSTUDMWUxZutA] Indices don't have primary shards [.kibana_7.15.1_001, .kibana-event-log-7.15.1-000002, .kibana_task_manager_pre7.4.0_001, .kibana_task_manager_7.15.1_001]",
  "stack_trace": "SnapshotException[[gcs:kibana-2022.03.24-rmug_mrwtauw_hac6mjchg/uw4vqewLQVSTUDMWUxZutA] Indices don't have primary shards [.kibana_7.15.1_001, .kibana-event-log-7.15.1-000002, .kibana_task_manager_pre7.4.0_001, .kibana_task_manager_7.15.1_001]]\n\tat org.elasticsearch.snapshots.SnapshotsService$2.execute(SnapshotsService.java:574)\n\tat org.elasticsearch.repositories.blobstore.BlobStoreRepository$1.execute(BlobStoreRepository.java:487)\n\tat org.elasticsearch.cluster.ClusterStateUpdateTask.execute(ClusterStateUpdateTask.java:51)\n\tat org.elasticsearch.cluster.service.MasterService.executeTasks(MasterService.java:836)\n\tat org.elasticsearch.cluster.service.MasterService.calculateTaskOutputs(MasterService.java:403)\n\tat org.elasticsearch.cluster.service.MasterService.runTasks(MasterService.java:243)\n\tat org.elasticsearch.cluster.service.MasterService.access$100(MasterService.java:63)\n\tat org.elasticsearch.cluster.service.MasterService$Batcher.run(MasterService.java:170)\n\tat org.elasticsearch.cluster.service.TaskBatcher.runIfNotProcessed(TaskBatcher.java:146)\n\tat org.elasticsearch.cluster.service.TaskBatcher$BatchedTask.run(TaskBatcher.java:202)\n\tat org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:718)\n\tat org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:262)\n\tat org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:225)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)\n\tat java.base/java.lang.Thread.run(Thread.java:829)\n"
}

This is not represented in the existing snapshot metrics such as elasticsearch_snapshot_stats_snapshot_number_of_failures, presumably as the actual snapshot is not attempted, however it is present in the SLM stats [2] and can be tied back to a specific policy:

{
    "retention_runs": 1,
    "retention_failed": 0,
    "retention_timed_out": 0,
    "retention_deletion_time": "0s",
    "retention_deletion_time_millis": 0,
    "total_snapshots_taken": 17,
    "total_snapshots_failed": 2,
    "total_snapshots_deleted": 0,
    "total_snapshot_deletion_failures": 0,
    "policy_stats": [
        {
            "policy": "kibana",
            "snapshots_taken": 17,
            "snapshots_failed": 2,
            "snapshots_deleted": 0,
            "snapshot_deletion_failures": 0
        }
    ]
}

[1] https://www.elastic.co/guide/en/elasticsearch/reference/current/snapshots-take-snapshot.html#automate-snapshots-slm
[2] https://www.elastic.co/guide/en/elasticsearch/reference/current/slm-api-get-stats.html