Alloy Test and Investigation for Metrics

[Rotfuks]

Motivation

We want to unify all of our agents to use the new opentelemetry agent from grafanalabs: alloy. For this we need to first test out if alloy can deliver exactly the same capability as the prometheus/grafana agents when collecting metrics. 

Todo

• Deploy Alloy for one test installation (like gazelle)
  • Create a feature flag with which we can select which agents are active - for this we still need to deploy both
• Check if mimir still gets the same metrics as we expect
• Compare the resource consumption of alloy to the previous agents - is it less, is it more, is it the same?

Outcome

• We gained experience with alloy and are confident enough to roll it out as agent for collecting metrics everywhere

[TheoBrigitte]

I managed to use alloy to send metrics to Mimir.

Settings kept:

• affinity for karpenter and alloy itself
• external labels
• service and pod monitor selectors
• remote write
• scrape interval
• priority class

Settings abandoned:

• retention: there's no explicit retention setting in Alloy, the retention logic is dictated by the WAL truncation logic, see https://grafana.com/docs/alloy/latest/reference/components/prometheus/prometheus.remote_write/#data-retention
• additional team label: currently no way to add our team label, we can probably push this upstream though
• keep dropped target: we currently have this value set to 0 (unlimited), meaning we keep in memory every dropped target, I am unsure why we have this set (more context)

Decision made:

• deployment mode: statefulset, this is the recommended deployment mode to run Alloy as metrics ingester, it allows to keep WAL on disk, and its "easy" to scale with the clustering mode. see https://grafana.com/docs/alloy/latest/set-up/deploy/
• clustering: enable clustering in order to allow for multiple Alloy pods to share the load, see https://grafana.com/docs/alloy/latest/reference/components/prometheus/prometheus.operator.servicemonitors/#clustering-block
• hpa: TODO enable horizontal pod autoscaling

Here is the value file I used to deploy Alloy as metrics ingester values.yaml.gz

helm install alloy giantswarm-test/alloy --version 0.3.1-c58378e71cbb5e9da677957500cf43b951d870a1 --values values.yaml

The amount of metrics sent to Mimir stays the same as with Prometheus agent

• left side is Prometheus Agent sending metrics
• middle just me playing with Alloy; note the spike is when I ran 2 Alloy replicas without clustering which resulted in nearly x2 req./s
• right side is Alloy sending metrics

[TheoBrigitte]

Here are the results of running Prometheus agent and Alloy as metrics agent. All test have been ran on the same installation (golem), each test was run for 1h.

I used 4 different test cases:

• 1 Prometheus agent replica/shard
• 1 Alloy replica
• 2 Prometheus agent replicas/shards
• 2 Alloy replicas

Agents

agent	replicas	CPU	Memory
Alloy	1	< 0.1	< 3GiB
Prometheus agent	1	> 0.1	> 4Gib
Alloy	2	< 0.05	< 3GiB
Prometheus agent	2	< 0.1	< 3GiB

Mimir

The amount of metrics, network and resources load on Mimir stayed approximately the same across all tests. Some Mimir ingester restarted and had some impact on the values shown in the graphs here but values are mostly within the same range.

Summary

Those tests showed that Alloy tends to consume about the same amount of resources or less than Prometheus agent and Mimir load stayed the same across all tests.

Here are the results as Grafana dashboard screenshots prometheus-agent_vs_alloy.tar.gz

[TheoBrigitte]

Most of the work and testing was done in giantswarm/observability-operator#66

I decided to go with our current custom autoscaling solution as it would otherwise differ to much from what we have currently and its also more complex to find a fit for every different installations size.

• alloy-app v0.4.0 was release with support for secret values
• observability-bundle v1.6.0 was release with the new alloy-metrics app as v0.4.0
• observability-operator v0.4.0 was release with support for Alloy as monitoring agent

Deployment to an installation is currently blocked as this feature is only supported on CAPI installations and we need a new release to get the new observability-bundle out.

[TheoBrigitte]

v29.1.0 is on its way, once this is release to a CAPA installation we can proceed with our live testing of Alloy as monitoring agent. We would then only need to toggle the monitoring agent flag for the observability-operator (example: https://github.com/giantswarm/giantswarm-configs/pull/135/files).

[QuentinBisson]

As an FYI, the release was merged :)

[TheoBrigitte]

Now we need to have it deployed to MCs https://github.com/giantswarm/giantswarm-management-clusters/pull/749

[QuentinBisson]

We can try it on a WC right?

[QuentinBisson]

Oh wait no we cannot because of this https://github.com/giantswarm/observability-operator/blob/09ddfe046e6a81cc6b874ac537941be9a495bc18/internal/controller/cluster_monitoring_controller.go#L181

Maybe the ervices should be created for each reconciliation then so the agent is always injected? Or passed as a function parameter

[QuentinBisson]

Yes we can test it out on the gazelle/cicddev cluster as it is running 29.1.0 :)

[TheoBrigitte]

There were actually few issue preventing this to be rolled out

• incorrect catalog name in observability-bundle for alloyMetrics
• invalid Alloy configuration due to missing comma in external labels map
• broken release pipeline in observability-operator

Those are all fixed now, but we now need to wait for an upgrade of observability-bundle to v1.6.2, most likely in capa v30.0.0 > giantswarm/releases#1357 (review)

[TheoBrigitte]

This is running on golem now and would be available from:

• capa v29.2.0 CAPA: Release v29.2.0. releases#1419
• capz v29.1.0 CAPZ: Release v29.1.0. releases#1389

Reminder: make sure we make an announcement to customers before releasing alloy-metrics.

[QuentinBisson]

@TheoBrigitte as this is an investigation story and not the rollout, should this be put in tracking or closed?

[Rotfuks]

Done on our side for now.