keda-operator-metrics-apiserver crashed due to lack of configmap related rbac

[fivesheep]

A clear and concise description of what the bug is.

Expected Behavior

keda-operator-metrics-apiserver shall be able to start properly

Actual Behavior

keda-operator-metrics-apiserver crashed with the following error:

I1030 02:42:41.365718       1 main.go:101] keda_metrics_adapter "msg"="KEDA Version: 2.4.0"
I1030 02:42:41.365765       1 main.go:102] keda_metrics_adapter "msg"="KEDA Commit: "
I1030 02:42:41.365770       1 main.go:103] keda_metrics_adapter "msg"="Go Version: go1.15.13"
I1030 02:42:41.365776       1 main.go:104] keda_metrics_adapter "msg"="Go OS/Arch: linux/amd64"
I1030 02:42:43.516909       1 request.go:655] Throttling request took 1.048327779s, request: GET:https://172.20.0.1:443/apis/node.k8s.io/v1?timeout=32s
I1030 02:42:44.868483       1 provider.go:46] keda_metrics_adapter/provider "msg"="starting"
I1030 02:42:44.868508       1 main.go:166] keda_metrics_adapter "msg"="starting adapter..."
2021-10-30 02:42:44.868582 I | Starting metrics server at :9022
I1030 02:42:45.310328       1 serving.go:325] Generated self-signed cert (apiserver.local.config/certificates/apiserver.crt, apiserver.local.config/certificates/apiserver.key)
W1030 02:42:45.676594       1 requestheader_controller.go:193] Unable to get configmap/extension-apiserver-authentication in kube-system.  Usually fixed by 'kubectl create rolebinding -n kube-system ROLEBINDING_NAME --role=extension-apiserver-authentication-reader --serviceaccount=YOUR_NS:YOUR_SA'
E1030 02:42:45.676642       1 main.go:121] keda_metrics_adapter "msg"="unable to run external metrics adapter" "error"="unable to load configmap based request-header-client-ca-file: configmaps \"extension-apiserver-authentication\" is forbidden: User \"system:serviceaccount:kube-system:keda-operator-sa\" cannot get resource \"configmaps\" in API group \"\" in the namespace \"kube-system\""

Steps to Reproduce the Problem

helm apply with -n kube-system (or whatever)

monitor the keda-operator-metrics-apiserver pod

Specifications

• KEDA Version: 2.4.0
• Platform & Version: EKS 1.21
• Kubernetes Version: Kube 1.21
• Scaler(s): Please elaborate

[fivesheep]

In my use case, the service account was created outside, and the service account name was set via the value file. in the service account file, it uses {{ .Values.serviceAccount.name }} for service account

{{- if .Values.serviceAccount.create -}}
apiVersion: v1
kind: ServiceAccount
metadata:
  labels:
    app.kubernetes.io/name: {{ .Values.serviceAccount.name }}
    app.kubernetes.io/version: {{ .Chart.AppVersion }}
    app.kubernetes.io/part-of: {{ .Values.operator.name }}
    app.kubernetes.io/managed-by: {{ .Release.Service }}
    app.kubernetes.io/instance: {{ .Release.Name }}
  {{- if .Values.serviceAccount.annotations }}
  annotations:
  {{- toYaml .Values.serviceAccount.annotations | nindent 6}}
  {{- end }}
  name: {{ .Values.serviceAccount.name }}
  namespace: {{ .Release.Namespace }}
{{- end -}}

and they were used by the deployments

templates/01-serviceaccount.yaml
6:    app.kubernetes.io/name: {{ .Values.serviceAccount.name }}
15:  name: {{ .Values.serviceAccount.name }}

templates/12-keda-deployment.yaml
43:      serviceAccountName: {{ .Values.serviceAccount.name }}

templates/22-metrics-deployment.yaml
45:      serviceAccountName: {{ .Values.serviceAccount.name }}

however, within the rolebinding files, the service account was the {{ .Values.operator.name }} which is inconsistent when passing a serviceAccount name from outside

apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  labels:
    app.kubernetes.io/name: {{ .Values.operator.name }}-auth-reader
    app.kubernetes.io/version: {{ .Chart.AppVersion }}
    app.kubernetes.io/part-of: {{ .Values.operator.name }}
    app.kubernetes.io/managed-by: {{ .Release.Service }}
    app.kubernetes.io/instance: {{ .Release.Name}}
  name: {{ .Values.operator.name }}-auth-reader
  namespace: kube-system
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: extension-apiserver-authentication-reader
subjects:
- kind: ServiceAccount
  name: {{ .Values.operator.name }}
  namespace: {{ .Release.Namespace }}
---

[joebowbeer]

Should be closed? Was fixed by #190 ?

[facorazza]

I'm still getting this using the Helm chart 2.14.2 with no custom values on k3s in a dedicated ns

[marianhromiak]

Same problem here. Error message from log:

"error"="unable to load configmap based request-header-client-ca-file: configmaps \"extension-apiserver-authentication\" is forbidden: User \"system:serviceaccount:keda-system:keda-metrics-server\" cannot get resource \"configmaps\" in API group \"\" in the namespace \"kube-system\"" "logger"="keda_metrics_adapter"

[yegorka6]

Same issue when upgrading from 2.13.2 to 2.15.1 (no custom values):
unable to run external metrics adapter" "error"="unable to load configmap based request-header-client-ca-file: configmaps "extension-apiserver-authentication" is forbidden: User "system:serviceaccount:devops:keda-metrics-server" cannot get resource "configmaps" in API group "" in the namespace "kube-system"" "logger"="keda_metrics_adapter

Any update on this?

[rupertgti]

I have the issue with chart in 2.15.2 with no custom values (I have the problem from 2.13.2 and cannot upgrade it)

[msvticket]

I also had this problem when upgrading to 2.15.2 from 2.13.0. In my case my CD system deploys a chart in specified namespace. But in the chart the rolebinding keda-operator-auth-reader is specified to go to kube-system. Since the CD system overrides that this problem occurs.

[rupertgti]

thank you @msvticket , its true. I use helmfile to apply the helm chart and it override the namespace for all elements. I need to search some workaround about this