Set prometheus in agent mode and upgrade kube-prometheus-stack to 30.0.0

[NissesSenap]

Upgrade kube-prometheus-stack to 30.0.0, this to get the latest prometheus-operator version so we can use
prometheus in agent mode.
Also run prometheus in agent mode, this to lower resource usage and a smarter way of handling WAL.

[NissesSenap]

When upgrading the helm chart I got the following error:

│ Error: cannot patch "prometheus-kube-state-metrics" with kind Deployment: Deployment.apps "prometheus-kube-state-metrics" is invalid: spec.selector: Invalid value: v1.LabelSelector{MatchLabels:map[string]string{"app.kubernetes.io/instance":"prometheus", "app.kubernetes.io/name":"kube-state-metrics"}, MatchExpressions:[]v1.LabelSelectorRequirement(nil)}: field is immutable
│ 
│   with module.aks1_core.module.prometheus["prometheus"].helm_release.prometheus,
│   on .terraform/modules/aks1_core/modules/kubernetes/prometheus/main.tf line 38, in resource "helm_release" "prometheus":
│   38: resource "helm_release" "prometheus" {

I solved it by deleting the deployment:

k delete deployment prometheus-kube-state-metric -n prometheus

Just as it says in the error:

# OLD
  selector:
    matchLabels:
      app.kubernetes.io/name: kube-state-metrics

# new 
  selector:
    matchLabels:
      app.kubernetes.io/instance: prometheus
      app.kubernetes.io/name: kube-state-metrics

[NissesSenap]

It's possible to run in agent mode but it's currently not a first class citizen: prometheus-operator/prometheus-operator#3989

But I will try to apply the workaround described here: https://github.com/prometheus-operator/kube-prometheus/blob/main/docs/customizations/prometheus-agent.md

[NissesSenap]

As a part of upgrading to this version you will need to delete the old prometheus PVC.
Ether delete the prometheus xks
kubectl delete prometheus xks -n prometheus

or you can also set the replicas to 0
kubectl edit prometheus xks -n prometheus

Then delete the PVC

kubectl delete pvc prometheus-xks-db-prometheus-xks-0 prometheus-xks-db-prometheus-xks-1 -n prometheus

[phillebaba]

LGTM! Great Work 🌹