toolkit: Reconciliation status as Prometheus metrics

[stefanprodan]

The toolkit controllers are recording the reconciliation status in a condition named Ready:

status:
  conditions:
  - type: Ready
    lastTransitionTime: "<last status change timestamp>"
    message: "<human readable description including errors if any>"
    reason: "<Progressing|Suspended|DependencyNotReady|ReconciliationSucceeded|etc>"
    status: "<Unknown|True|False>"

Metric spec

We could export the ready condition as a Prometheus gauge:

gitops_toolkit_ready_condition{kind, name, namespace, reason} [-2,-1,0,1]

Gauge values:

• status.Unknown = 0
• status.True = 1
• status.False = -1
• finalizer = -2

Alert manager example:

groups:
- name: GitOpsToolkit
  rules:
  - alert: ReconciliationFailure
    expr: gitops_toolkit_ready_condition == -1
    for: 10m
    labels:
      severity: page
    annotations:
      summary: '{{ $labels.kind }} {{ $labels.namespace }}/{{ $labels.name }} reconciliation has been failing with {{ $labels.reason }} for more than ten minutes.'

Implementation spec

Create a metrics library in fluxcd/pkg that:

• contains the Ready condition struct
• registers the ready condition as a controller-runtime metric
• exposes a function to record the status changes

Monitoring stack

The ready condition metrics would allow us to build Grafana dashboards for monitoring the reconciliation process and drilldown to individual objects. The dashboards could be used to diagnose specific issues if we plot the reconciliations failures grouped by kind/namespace/name/reason.

[seaneagan]

What about just exposing all conditions by including condition (or maybe type) as one of the labels (see #122)? Also are we sure we want reason as a label since there should only be one condition of a given type at a time, not multiple with different reasons, and thus only one gauge should exist? There is a way to delete old metrics, so perhaps that could be used to make sure only one gauge exists at a time for each resource.

gitops_toolkit_condition_info{kind, name, namespace, condition} [-2,-1,0,1]

[stefanprodan]

Good point about reason... I don't think we should be deleting metrics as that requires access to Prometheus API.

Any idea how we can include the reason so people can see in a dashboard why something fails? Or maybe we should not expose object level metrics, the more I think I about it, feels like Prometheus is not intended for this.

[squaremo]

This is a misuse of gauges, which are to record a quantity. The things you'd expect to do with a gauge are:

• aggregate it over several metrics (permutations of labels)
• chart its value over time;

Neither of these yields anything useful when the gauge value represents a discrete state. (OK, you could make a chart of the state transitions of a single object, provided you could select a single object with labels. But there are better ways to get that information, like events).

A lesser misuse might be to give an aggregate count of the number of items in each state, which (since this can fall as well as rise) would need to be a gauge.

gitops_toolkit_ready_state{kind, name, namespace, status} = 0 | 1

This gives a sensible value when aggregated (and charted in aggregate), and can be used for alerts. However, you'd need to set the gauge for all values of status each time, which tells me this is not a great use of gauges either.

[stefanprodan]

Here is how kube-state-metrics exposes deployments conditions as gauges kubernetes/kube-state-metrics#889

[squaremo]

OK, so that more or less corresponds to my suggestion, but with individual conditions rather than the ready status. I'm still a little taken aback that people are happy with all these time series {object x condition x value}, but assuming they don't care, that might be the way to do it.

[stefanprodan]

The custom metrics implementation is tracked here #329