Migrate to Alloy

[Rotfuks]

Motivation

OpenTelemetry is the future of monitoring :)
At least GrafanaLabs sees it this way, that's why they introduced Grafana Alloy, which is an Agent ingesting opentelemetry data - from metrics over logs to tracing, all managed in one agent. This will replace the Prometheus and Grafana Agents in the near future and we'll just have one agent running, that can handle all, which will be a nice improvement resource usage wise.

Investigation

Preview Give feedback

1. Replace Grafana-Agent Rules with Alloy #3521
   team/atlas +1
   
2. Alloy Test and Investigation for Logs #3524
   blocked team/atlas +2
   
3. Alloy Test and Investigation for Metrics #3522
   blocked team/atlas +2
   
4. Alloy monitoring #3590
   team/atlas +1
   
5. Alloy testing #3613
   blocked team/atlas +2
   

IF we've decided to go for it:

Rollout

Preview Give feedback

1. Use Alloy to collect logs on all installations #3525
   team/atlas +1
   
2. Rollout Alloy for metrics to all Installations #3523
   team/atlas +1
   
3. Check the resource usage of Alloy #3724
   team/atlas +1
   
4. Collect Kubernetes Events with Alloy #3750
   team/atlas +1
   

Outcome

• We have replaced all agents for collecting metrics and logs with Alloy running on all installations.