[kube-prometheus-stack] Use namespaceOverrides when set

[sathieu]

What this PR does / why we need it:

Uses grafana.namespaceOverride, kube-state-metrics.namespaceOverride and prometheus-node-exporter.namespaceOverride when set, for created resource.

(includes #1342).

Checklist

• DCO signed
• Chart Version bumped
• Title of the PR starts with chart name (e.g. [prometheus-couchdb-exporter])

[vsliouniaev]

Sorry, noticed another potential issue with namespace selectors in serivcemonitors:

https://github.com/prometheus-community/helm-charts/pull/1343/files#diff-ef9695f03f05e41615990a96c6a839fd0858697075287a8be909cc1acc18509aR16-R20

Could you try testing this end-to-end with all possible namespace overrides set to different values and see if everything remains fully operational?

[sathieu]

@vsliouniaev Actually, I have a problem when doing Helm template (from ArgoCD):

`helm template . [...]` failed exit status 1: Error: template: prometheus-stack/charts/kube-prometheus-stack/templates/_helpers.tpl:103:16: executing "grafana.namespace" at <.Values.grafana.namespaceOverride>: nil pointer evaluating interface {}.namespaceOverride Use --debug flag to render out invalid YAML

The CI has the same message.

Also, your link is not pointing to a specific file. Could you update it?

[sathieu]

@vsliouniaev I found the cause: from the doc:

Recall how the labels on templates are globally shared. Thus, the labels chart can be included from any other chart.

... and defining a template twice leads to mixed scope.

To fix this, I removed the templates from kube-prometheus-stack, so that subchart's templates are used.

[sathieu]

In can confirm that it works:

$ helm template  --set grafana.namespaceOverride=grafana --set kube-state-metrics.namespaceOverride=kube-state-metrics --set prometheus-node-exporter.namespaceOverride=prometheus-node-exporter .  | grep -A 5 'kind: ServiceMonitor'
kind: ServiceMonitor
metadata:
  name: RELEASE-NAME-kube-promethe-alertmanager
  namespace: default
  labels:
    app: kube-prometheus-stack-alertmanager
--
kind: ServiceMonitor
metadata:
  name: RELEASE-NAME-kube-promethe-coredns
  namespace: default
  labels:
    app: kube-prometheus-stack-coredns
--
kind: ServiceMonitor
metadata:
  name: RELEASE-NAME-kube-promethe-apiserver
  namespace: default
  labels:
    app: kube-prometheus-stack-apiserver
--
kind: ServiceMonitor
metadata:
  name: RELEASE-NAME-kube-promethe-kube-controller-manager
  namespace: default
  labels:
    app: kube-prometheus-stack-kube-controller-manager
--
kind: ServiceMonitor
metadata:
  name: RELEASE-NAME-kube-promethe-kube-etcd
  namespace: default
  labels:
    app: kube-prometheus-stack-kube-etcd
--
kind: ServiceMonitor
metadata:
  name: RELEASE-NAME-kube-promethe-kube-proxy
  namespace: default
  labels:
    app: kube-prometheus-stack-kube-proxy
--
kind: ServiceMonitor
metadata:
  name: RELEASE-NAME-kube-promethe-kube-scheduler
  namespace: default
  labels:
    app: kube-prometheus-stack-kube-scheduler
--
kind: ServiceMonitor
metadata:
  name: RELEASE-NAME-kube-promethe-kube-state-metrics
  namespace: kube-state-metrics
  labels:
    app: kube-prometheus-stack-kube-state-metrics
--
kind: ServiceMonitor
metadata:
  name: RELEASE-NAME-kube-promethe-kubelet
  namespace: default
  labels:
    app: kube-prometheus-stack-kubelet
--
kind: ServiceMonitor
metadata:
  name: RELEASE-NAME-kube-promethe-node-exporter
  namespace: prometheus-node-exporter
  labels:
    app: kube-prometheus-stack-node-exporter
--
kind: ServiceMonitor
metadata:
  name: RELEASE-NAME-kube-promethe-grafana
  namespace: grafana
  labels:
    app: kube-prometheus-stack-grafana
--
kind: ServiceMonitor
metadata:
  name: RELEASE-NAME-kube-promethe-operator
  namespace: default
  labels:
    app: kube-prometheus-stack-operator
--
kind: ServiceMonitor
metadata:
  name: RELEASE-NAME-kube-promethe-prometheus
  namespace: default
  labels:
    app: kube-prometheus-stack-prometheus

[vsliouniaev]

There are namespace selectors in a few servicemonitors that I think will cause issues: https://github.com/prometheus-community/helm-charts/blob/main/charts/kube-prometheus-stack/templates/exporters/kube-state-metrics/serviceMonitor.yaml#L50-L54

[sathieu]

thanks @vsliouniaev I've removed kubeStateMetrics.serviceMonitor.namespaceOverride and bumped major for this.

I've also fixed the prometheus Datasource.

Currently testing ...

[sathieu]

At 4f92516, with grafana.namespaceOverride, kube-state-metrics.namespaceOverride and prometheus-node-exporter.namespaceOverride, I have tested:

• ✔️ Prometheus Datasource is set-up and working
• ✔️ Dashboards are created
• ✔️ All tested dashboards have filled graphs, including
  • ✔️ Node Exporter / Nodes (prometheus-node-exporter.namespaceOverride)
• However, I have the following dashboards empty:
  • 🔴 Alertmanager / Overview (serviceMonitor is unhealthy server returned HTTP status 404 Not Found, maybe because of istio)
  • Kubernetes / Proxy (serviceMonitor is unhealthy server returned HTTP status 503 Service Unavailable) fixed (this was a Kubespray config)
  • 🟠 etcd (expected, my etcd install is not using containers)

[sathieu]

@vsliouniaev Please review again.

NB: I've rebased to resolve a merge conflict.

[vsliouniaev]

From the code, I can see there is an inconsistency in the namespaces in charts/kube-prometheus-stack/templates/exporters/node-exporter/servicemonitor.yaml.

[sathieu]

@vsliouniaev Thanks. Fixed. back to you 🏓 !

[vsliouniaev]

This generally looks like it's working. Not sure what the use-case is for such an added complication to the chart.

[sathieu]

@vsliouniaev I think it's a good idea to separate the components in different namespaces. Namespaces are security boundaries in k8s, and usage of network policies or roles is easier. Monitoring is also easier.

Also, I'm not the only one using namespace overrides (See older bugs #984, #820), this PR fixes all remaining problems about nsOverrides.

Note: I've resolved conflict and force-pushed.

[sathieu]

Thanks @vsliouniaev !

[james-callahan]

This breaks namespace independence.
namespaceSelector shouldn't be set unless it's pointing at a different namespace

Created #1382