-
Notifications
You must be signed in to change notification settings - Fork 1.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
The paused-replicas annotation not fully respected #4253
Comments
add pull request #4254 thanks |
I understand the rationale behind your ask. Currently we just pause scaling but scalers are connected, also HPA is still in place. We would have to do a bigger change. In the controller in the reconcile loop stop the scaling logic, so we are not connecting the to scalers. keda/controllers/keda/scaledobject_controller.go Lines 442 to 476 in 30b2f8e
And also delete the generated HPA, so it is not querying KEDA Metric Server for metrics
@aharonh are you willing to give it a try? |
Signed-off-by: aharonh <aharon.haravon@gmail.com>
Hi @zroubalik ,
I just re-did the changes with the gpg signature that was missing. Yep, I tested it, looked like it works as expected:
Can you think of any further tests that would be useful to perform? Thanks, |
I see though what you are saying... I guess the reason it works in our case is that we are in addition setting the deployment replicas to 0 when scaling down the namespace to 0. I guess that may be the reason that this fix helps us resolve the issue at least in our case. |
Do you agree to merge this fix and we can open another issue I can separately work on that will fix the issue in wider sense as you suggested? |
Asking just to make sure that our production will not have to run with patched version for longer time ;) |
Signed-off-by: aharonh <aharon.haravon@gmail.com>
@aharonh yeah I see. I would actually prefer if we go straight with the approach I suggested, it is more clear solution. We are going to do a new release in ~2 weeks, so there's still some time. Just let me know if you are interested in contribute that, if not we can try to ask someone else :) |
ok, @zroubalik I will try to have a look into it, thanks! |
This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 7 days if no further activity occurs. Thank you for your contributions. |
still intend to have a look into it, not immediately though so if anyone else has the time and want's to handle this, you are welcome. |
Hello @JorTurFer and @aharonh. |
That would be wonderful, thank you @tobotg! |
Signed-off-by: Tobo Atchou <tobo.atchou@gmail.com>
@tomkerkhove @zroubalik |
…dacore#4253 Signed-off-by: Tobo Atchou <tobo.atchou@gmail.com>
Signed-off-by: Tobo Atchou <tobo.atchou@gmail.com>
Signed-off-by: Tobo Atchou <tobo.atchou@gmail.com>
Signed-off-by: Tobo Atchou <tobo.atchou@gmail.com>
Signed-off-by: Tobo Atchou <tobo.atchou@gmail.com>
Signed-off-by: Tobo Atchou <tobo.atchou@gmail.com>
Signed-off-by: Tobo Atchou <tobo.atchou@gmail.com>
Signed-off-by: Tobo Atchou <tobo.atchou@gmail.com>
…or msg and new condition msg Signed-off-by: Tobo Atchou <tobo.atchou@gmail.com>
…sed condition Signed-off-by: Tobo Atchou <tobo.atchou@gmail.com>
… get hpa name and paused condition Signed-off-by: Tobo Atchou <tobo.atchou@gmail.com>
…ifest for paused condition Signed-off-by: Tobo Atchou <tobo.atchou@gmail.com>
Signed-off-by: Tobo Atchou <tobo.atchou@gmail.com>
…or msg and new condition msg Signed-off-by: Tobo Atchou <tobo.atchou@gmail.com>
…sed condition Signed-off-by: Tobo Atchou <tobo.atchou@gmail.com>
… get hpa name and paused condition Signed-off-by: Tobo Atchou <tobo.atchou@gmail.com>
…ifest for paused condition Signed-off-by: Tobo Atchou <tobo.atchou@gmail.com>
…rease test wait timeout Signed-off-by: Tobo Atchou <tobo.atchou@gmail.com>
Signed-off-by: Tobo Atchou <tobo.atchou@gmail.com>
…or msg and new condition msg Signed-off-by: Tobo Atchou <tobo.atchou@gmail.com>
…sed condition Signed-off-by: Tobo Atchou <tobo.atchou@gmail.com>
… get hpa name and paused condition Signed-off-by: Tobo Atchou <tobo.atchou@gmail.com>
…ifest for paused condition Signed-off-by: Tobo Atchou <tobo.atchou@gmail.com>
…rease test wait timeout Signed-off-by: Tobo Atchou <tobo.atchou@gmail.com>
…ed up test with slim container image and wait for monitored deployment Signed-off-by: Tobo Atchou <tobo.atchou@gmail.com>
Signed-off-by: Tobo Atchou <tobo.atchou@gmail.com>
…or msg and new condition msg Signed-off-by: Tobo Atchou <tobo.atchou@gmail.com>
…sed condition Signed-off-by: Tobo Atchou <tobo.atchou@gmail.com>
… get hpa name and paused condition Signed-off-by: Tobo Atchou <tobo.atchou@gmail.com>
…ifest for paused condition Signed-off-by: Tobo Atchou <tobo.atchou@gmail.com>
…rease test wait timeout Signed-off-by: Tobo Atchou <tobo.atchou@gmail.com>
…ed up test with slim container image and wait for monitored deployment Signed-off-by: Tobo Atchou <tobo.atchou@gmail.com>
Signed-off-by: Tobo Atchou <tobo.atchou@gmail.com>
…acore#4550) Signed-off-by: Tobo Atchou <tobo.atchou@gmail.com>
Report
Althogh the scaledobject annotation autoscaling.keda.sh/paused-replicas does pause the autoscaling as expected, keda will still log error messages originating from triggers for such paused scaledobjects. This is a problem if you are managing multiple environments that are being fully scaled down (to 0), including the data sources for the scaled object triggers - in our case mysql database is also scaled down so the database connection failure is all the time logged.
Expected Behavior
No error messages will be created whatsoever for scaledobjects that are paused.
Actual Behavior
many errors are logged in loop - see logs from keda operator
Steps to Reproduce the Problem
kubectl annotate scaledobject session-scaler autoscaling.keda.sh/paused-replicas="0"
k logs -l app=keda-operator -f
Logs from KEDA operator
KEDA Version
2.8.1
Kubernetes Version
1.25
Platform
Amazon Web Services
Scaler Details
mysql
Anything else?
willing to help fix this
The text was updated successfully, but these errors were encountered: