Alerting framework: Ability to do multiple queries and arithmetics between Metric threshold's metric values

[georgivalentinov]

Describe the feature:

Metric threshold alerts count on each metric having the right high-level granularity in order to alert on something meaningful, e.g. container_cpu_limit_pct (for getting container cpu utilization percent of limit). Unfortunately a lot of metrics out there are more of a lower-level ones and need a bit of arithmetics to get to something useful.
Currently we can do basic aggregations (min, max, sum, cardinality, percentile, ...) on metric values but nothing more complex.

They say a picture is worth a thousand words, so here's one simplified example (it's how Datadog does this):

A set of lower-level metrics, with which one is still able to get the same high-level "percent utilization of cpu limit" number by doing multiple queries and some arithmetic on them.

Describe a specific use case for the feature:

As hinted above - most metrics are more of a lower-level ones, which severely limits what the current Metric threshold rule can be used for.

Here's another example use case:

{
    "prometheus": {
        "labels": {
            "instance": "monitoring-kube-state-metrics:8080",
            "job": "prometheus",
            "namespace": "<my_namespace>",
            "horizontalpodautoscaler": "<my_hpa_name>"
        },
        "metrics": {
            "kube_horizontalpodautoscaler_spec_max_replicas": 5,
            "kube_horizontalpodautoscaler_status_current_replicas": 1,
            "kube_horizontalpodautoscaler_metadata_generation": 0,
            "kube_horizontalpodautoscaler_status_desired_replicas": 1,
            "kube_horizontalpodautoscaler_spec_min_replicas": 1
        }
    }
}

This is a snippet from a document with metrics collected from a Kubernetes horizontal pod autoscaler with the prometheus/collector Metricbeat's metricset.
With the current capabilities one can't compute a ratio and thus percentage of currently running replicas compared to the maximum possible (kube_horizontalpodautoscaler_status_current_replicas / kube_horizontalpodautoscaler_spec_max_replicas * 100).

A third example:

We have the following AWS ALB metrics at hand: HTTPCode_ELB_5XX_Count, HTTPCode_Target_5XX_Count & RequestCount. First two represent http 5xx errors for an application, where one is being reported by an AWS target group and the other by an AWS LB. The third one represents total http request processed by the same AWS LB.
Frequently one wants to sum the first two and then compare that number to the total requests in order to get percentage 5xx errors of all traffic - so it's a similar formula as the first example: (a + b) / c * 100. Which value can then be compared to the warning and/or alert % thresholds.

Sorry if all this has already been requested, but I couldn't find such an issue back.

[elasticmachine]

Pinging @elastic/response-ops (Team:ResponseOps)

[elasticmachine]

Pinging @elastic/actionable-observability (Team: Actionable Observability)

[maryam-saeidi]

cc @vinaychandrasekhar