[grafana] Grafana probes customizable only on path & port

[cstanislawski]

All the following is applicable to both readiness & liveness probes, but I'll showcase the issue I'm experiencing using only the readinessProbe.

Helm version:

% helm version  
version.BuildInfo{Version:"v3.15.3", GitCommit:"3bb50bbbdd9c946ba9989fbe4fb4104766302a64", GitTreeState:"clean", GoVersion:"go1.22.5"}

Grafana chart version: 8.4.1 - this is applicable to older versions as well.

Issue:
I'm unable to customize the grafana probes outside of modifying the path & port - while modifying the default values to:

readinessProbe:
  exec:
    command:
    - /bin/sh
    - -c
    - exec curl -f http://localhost:3000/api/health

I got the following readinessProbe as a result:

          readinessProbe:
            exec:
              command:
              - /bin/sh
              - -c
              - exec curl -f http://localhost:3000/api/health
            httpGet:
              path: /api/health
              port: 3000

i.e. the default value of Grafana's readinessProbe, httpGet, will get populated into the output manifest, unless I very explicitly set the httpGet in values to null, so:

readinessProbe:
  exec:
    command:
    - /bin/sh
    - -c
    - exec curl -f http://localhost:3000/api/health
  httpGet: null

while this workaround could work, it's not possible for me as I'm using the Grafana chart embedded into another chart, kube-prometheus-stack, and setting httpGet: null does not get populated into the Grafana-embedded values.
While this arguably could be considered an issue with Helm / kube-prometheus-stack, the Grafana chart is the only chart I've had issues with.

Weirdly enough, this is not an issue while testing with the local chart checked out on tag grafana-8.4.1 (same results with main):
customized values:

readinessProbe:
  exec:
    command:
    - /bin/sh
    - -c
    - exec curl -f http://localhost:3000/api/health

running commands:

cd charts/grafana
helm template --name-template grafana . -f values.yaml  > manifests-local.yaml

result in:

          readinessProbe:
            exec:
              command:
              - /bin/sh
              - -c
              - exec curl -f http://localhost:3000/api/health

To make this even weirder, the sidecar containers are not affected by this difference between local vs published chart - the results are the same in both scenarios.

---

While this is not a common use case I need to customize the readinessProbe to configure Grafana correctly with:

  readinessProbe:
    exec:
      command:
        - /bin/sh
        - -c
        - |-
          curl -X GET http://localhost:3000/api/health
          curl -X POST http://localhost:3000/api/v1/ngalert/admin_config --user "$GF_SECURITY_ADMIN_USER:$GF_SECURITY_ADMIN_PASSWORD" -H 'Content-Type: application/json' -H 'Accept: application/json, text/plain, */*' --data '{"alertmanagersChoice":"external"}'

to configure the Alerting > Admin settings to use the external alertmanagers in the kube-prometheus-stack, which is caused by the absence of provisioning for ngalert - related to:
grafana/grafana#64081
grafana/grafana#80349

I do not need a workaround as I've already created one - a workaround for a workaround.

I would just love to see what's the cause of this, as I cannot identify the issue myself.