forbidden: User "system:Reconciler error serviceaccount:karpenter:karpenter cannot patch resource "nodes/status" in API group "\" at the cluster scope

[andrewhibbert]

Version

Karpenter Version: v0.20.1

Kubernetes Version: v1.21.0

Expected Behavior

No errors patching node status

Actual Behavior

karpenter-697465c6f4-5gxsh controller 2023-01-04T18:39:53.936Z ERROR controller Reconciler error {"commit": "06cb81f-dirty", "controller": "node", "controllerGroup": "", "controllerKind": "Node", "Node": {"name":"ip-10-138-110-47.eu-west-1.compute.internal"}, "namespace": "", "name": "ip-10-138-110-47.eu-west-1.compute.internal", "reconcileID": "cd065963-bd9e-4dcb-b19b-1966aca98e0e", "error": "nodes \"ip-10-138-110-47.eu-west-1.compute.internal\" is forbidden: User \"system:serviceaccount:karpenter:karpenter\" cannot patch resource \"nodes/status\" in API group \"\" at the cluster scope"}

As in #3085

Steps to Reproduce the Problem

Unsure seems sporadic

Resource Specs and Logs

karpenter-697465c6f4-5gxsh controller 2023-01-04T18:39:53.936Z ERROR controller Reconciler error {"commit": "06cb81f-dirty", "controller": "node", "controllerGroup": "", "controllerKind": "Node", "Node": {"name":"ip-10-138-110-47.eu-west-1.compute.internal"}, "namespace": "", "name": "ip-10-138-110-47.eu-west-1.compute.internal", "reconcileID": "cd065963-bd9e-4dcb-b19b-1966aca98e0e", "error": "nodes \"ip-10-138-110-47.eu-west-1.compute.internal\" is forbidden: User \"system:serviceaccount:karpenter:karpenter\" cannot patch resource \"nodes/status\" in API group \"\" at the cluster scope"}

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

[andrewhibbert]

From what I've seen this is for spot instances

[ellistarn]

Odd to me that no one else is seeing this. We don't patch node status as far as I'm aware. Are you on EKS or self managed? Anything weird about your setup?

[andrewhibbert]

Happened once in about an hour, I probably only noticed as I've just upgraded and checking the logs. I'll check tomorrow to see if it has happened some more. EKS, I think it is fairly standard

[sergibarroso-datarobot]

I'm having the same issue with spot instances.

I've consolidation and spot interruption-handling enabled.

[joaosilva15]

It also happened to us. Changed the cluster role to allow nodes/status instead of only nodes makes the error disappear but I also find it strange that no one else had the same issue. We observed this issue on bottlerocket nodes not running in spot instances.

[andrewhibbert]

Happened once in about an hour, I probably only noticed as I've just upgraded and checking the logs. I'll check tomorrow to see if it has happened some more. EKS, I think it is fairly standard

Just to circle back on this, I am still seeing these messages in the logs and it does seem to be spot instances