[cluster-autoscaler] RBAC issue when running in namespaced mode

[rbjorklin]

Which component are you using?: cluster-autoscaler

What version of the component are you using?: 1.31.0

Component version: registry.k8s.io/autoscaling/cluster-autoscaler:v1.31.0

What k8s version are you using (kubectl version)?:

Output

❯ kubectl version
Client Version: v1.29.3
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.30.0

What environment is this in?: Hetzner with cluster-api-provider

What did you expect to happen?:

I expected autoscaling to work when setting clusterScoped: false as suggested in the values.yaml file.

What happened instead?:

The cluster-autoscaler throws errors failing to read resources at the cluster scope.

Log messages, click to expand.

E0912 00:32:11.707800       1 reflector.go:158] "Unhandled Error" err="pkg/mod/k8s.io/client-go@v0.31.0/tools/cache/reflector.go:243: Failed to watch cluster.x-k8s.io/v1beta1, Resource=machinedeployments: failed to list cluster.x-k8s.io/v1beta1, Resource=machinedeployments: machinedepl
oyments.cluster.x-k8s.io is forbidden: User \"system:serviceaccount:management:cluster-autoscaler-clusterapi-cluster-autoscaler\" cannot list resource \"machinedeployments\" in API group \"cluster.x-k8s.io\" at the cluster scope" logger="UnhandledError"
I0912 00:32:27.780015       1 reflector.go:341] Listing and watching cluster.x-k8s.io/v1beta1, Resource=machinepools from pkg/mod/k8s.io/client-go@v0.31.0/tools/cache/reflector.go:243
W0912 00:32:27.781182       1 reflector.go:561] pkg/mod/k8s.io/client-go@v0.31.0/tools/cache/reflector.go:243: failed to list cluster.x-k8s.io/v1beta1, Resource=machinepools: machinepools.cluster.x-k8s.io is forbidden: User "system:serviceaccount:management:cluster-autoscaler-clusterap
i-cluster-autoscaler" cannot list resource "machinepools" in API group "cluster.x-k8s.io" at the cluster scope
E0912 00:32:27.781213       1 reflector.go:158] "Unhandled Error" err="pkg/mod/k8s.io/client-go@v0.31.0/tools/cache/reflector.go:243: Failed to watch cluster.x-k8s.io/v1beta1, Resource=machinepools: failed to list cluster.x-k8s.io/v1beta1, Resource=machinepools: machinepools.cluster.x-
k8s.io is forbidden: User \"system:serviceaccount:management:cluster-autoscaler-clusterapi-cluster-autoscaler\" cannot list resource \"machinepools\" in API group \"cluster.x-k8s.io\" at the cluster scope" logger="UnhandledError"
I0912 00:32:33.296921       1 reflector.go:341] Listing and watching cluster.x-k8s.io/v1beta1, Resource=machinesets from pkg/mod/k8s.io/client-go@v0.31.0/tools/cache/reflector.go:243
W0912 00:32:33.297764       1 reflector.go:561] pkg/mod/k8s.io/client-go@v0.31.0/tools/cache/reflector.go:243: failed to list cluster.x-k8s.io/v1beta1, Resource=machinesets: machinesets.cluster.x-k8s.io is forbidden: User "system:serviceaccount:management:cluster-autoscaler-clusterapi-
cluster-autoscaler" cannot list resource "machinesets" in API group "cluster.x-k8s.io" at the cluster scope
E0912 00:32:33.297799       1 reflector.go:158] "Unhandled Error" err="pkg/mod/k8s.io/client-go@v0.31.0/tools/cache/reflector.go:243: Failed to watch cluster.x-k8s.io/v1beta1, Resource=machinesets: failed to list cluster.x-k8s.io/v1beta1, Resource=machinesets: machinesets.cluster.x-k8s
.io is forbidden: User \"system:serviceaccount:management:cluster-autoscaler-clusterapi-cluster-autoscaler\" cannot list resource \"machinesets\" in API group \"cluster.x-k8s.io\" at the cluster scope" logger="UnhandledError"
I0912 00:32:40.377287       1 reflector.go:341] Listing and watching cluster.x-k8s.io/v1beta1, Resource=machines from pkg/mod/k8s.io/client-go@v0.31.0/tools/cache/reflector.go:243
W0912 00:32:40.378531       1 reflector.go:561] pkg/mod/k8s.io/client-go@v0.31.0/tools/cache/reflector.go:243: failed to list cluster.x-k8s.io/v1beta1, Resource=machines: machines.cluster.x-k8s.io is forbidden: User "system:serviceaccount:management:cluster-autoscaler-clusterapi-cluste
r-autoscaler" cannot list resource "machines" in API group "cluster.x-k8s.io" at the cluster scope
E0912 00:32:40.378558       1 reflector.go:158] "Unhandled Error" err="pkg/mod/k8s.io/client-go@v0.31.0/tools/cache/reflector.go:243: Failed to watch cluster.x-k8s.io/v1beta1, Resource=machines: failed to list cluster.x-k8s.io/v1beta1, Resource=machines: machines.cluster.x-k8s.io is fo
rbidden: User \"system:serviceaccount:management:cluster-autoscaler-clusterapi-cluster-autoscaler\" cannot list resource \"machines\" in API group \"cluster.x-k8s.io\" at the cluster scope" logger="UnhandledError"

How to reproduce it (as minimally and precisely as possible):

My values.yaml looks like this:

cloudProvider: clusterapi

autoDiscovery:
  labels:
    - cluster-autoscaler: enabled

clusterAPIMode: "kubeconfig-incluster"

clusterAPIWorkloadKubeconfigPath: /etc/kubernetes/management-kubeconfig/value

clusterAPIKubeconfigSecret: "management-kubeconfig"

rbac:
  clusterScoped: false

Anything else we need to know?:

This comment makes me think this has worked at some point in time.

To be clear the autoscaling works as intended when setting clusterScoped: true.

[adrianmoisey]

/area cluster-autoscaler