UnPause does not happen immediately

[alt-dima]

Report

We have about 500 ScaledObjects.
Some with CPU, some with RabbitMQ and some with Kafka scalers (only one scaler per ScaledObject)

Pausing of ScaledObjects happens immediately after autoscaling.keda.sh/paused-replicas annotation added to ScaledObject:

1. In the keda-operator immediately appear Reconcile message
2. In the keda-operator immediately appear HPA updated message
3. HPA actually paused

But unPausing does not happen same way.
For ScaledObject with CPU scaler it may happen in couple of minutes
For ScaledObject with Kafka/RabbitMQ scaler it may never happen or happen with a big delay (tens of minutes)

Expected Behavior

1. Reconcile for unpausing object should happen immediately right after autoscaling.keda.sh/paused-replicas annotation was removed from ScaledObject.
2. HPA should be "unpaused" (scaling restored)

Actual Behavior

1. autoscaling.keda.sh/paused-replicas annotation removed from ScaledObject
2. Reconciling ScaledObject not happen
3. HPA remains "paused" (with annotation autoscaling.keda.sh/paused-replicas and min=max=paused-replicas)

Steps to Reproduce the Problem

1. Create a ScaledObject with Kafka scaler
2. Pause it kubectl annotate scaledObject test-workload --overwrite autoscaling.keda.sh/paused-replicas=2
3. Check HPA for test-workload = it was paused (annotated and min=max=2)
4. UnPause it kubectl annotate scaledObject test-workload --overwrite autoscaling.keda.sh/paused-replicas-
5. Check HPA for test-workload = it still paused (annotated and min=max=2)
6. Check ScaledObject for test-workload = annotation removed!

Logs from KEDA operator

example

KEDA Version

2.10.1

Kubernetes Version

1.24

Platform

Amazon Web Services

Scaler Details

Kafka, RabbitMQ, CPU

Anything else?

Also interesting, why Reconcile for CPU-based ScaledObjects happens (based on the keda-operator logs) very often?
And for Kafka/RabbitMQ Reconcile happens very rarery.

Maybe actual problem in this behavior? Keda does not do immediately UnPause, but periodically scans all the ScaledObjects and "refresh/realign" HPA?

[alt-dima]

[alt-dima]

I tried to restart keda-operator and in 1-2 minutes after restart it updated/unpaused the HPA.

So it looks like the are 2 problems

1. some loop over all scaledObjects (to refresh all of them?) does not reach some workloads (seems only Kafka and RabbitMQ are affected, because for CPU-based I do see Reconcile messages very often ).
2. missing the logic to execute Reconcile after annotation removed from ScaledObject

[zroubalik]

There's new implementation of pausing ScaledObject coming in 2.11: #4550

this should resolve this issue

[zroubalik]

there was a typo in the orignal post, missing the 0 - #4550

[alt-dima]

Updated to 2.11.
But it remains the same.
Paused (to 0) => hpa removed almost immediately.
Un-Paused (removed annotation) => after 10 minutes it is still paused (hpa still not created and amount of pods = 0)

[zroubalik]

@alt-dima you are hitting a new regression, that has been introduced in this version. We will release 2.11.1 very soon, which fixes this. #4734

[tomkerkhove]

This will be shipped as part of #4743.

Closing this issue since it was fixed already

[alt-dima]

For history.
With version 2.11.1 it works fine! UnPause happens immediately!
Thank you.