Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

The paused-replicas annotation not fully respected #4253

Closed
aharonh opened this issue Feb 20, 2023 · 13 comments · Fixed by #4550
Closed

The paused-replicas annotation not fully respected #4253

aharonh opened this issue Feb 20, 2023 · 13 comments · Fixed by #4550
Assignees
Labels
bug Something isn't working

Comments

@aharonh
Copy link

aharonh commented Feb 20, 2023

Report

Althogh the scaledobject annotation autoscaling.keda.sh/paused-replicas does pause the autoscaling as expected, keda will still log error messages originating from triggers for such paused scaledobjects. This is a problem if you are managing multiple environments that are being fully scaled down (to 0), including the data sources for the scaled object triggers - in our case mysql database is also scaled down so the database connection failure is all the time logged.

Expected Behavior

No error messages will be created whatsoever for scaledobjects that are paused.

Actual Behavior

many errors are logged in loop - see logs from keda operator

Steps to Reproduce the Problem

  1. to pause, type
kubectl annotate scaledobject session-scaler autoscaling.keda.sh/paused-replicas="0"
  1. to checkout the operator logs:

k logs -l app=keda-operator -f

Logs from KEDA operator

2023-02-20T14:36:52Z	ERROR	mysql_scaler	Found error when pinging database: dial tcp 100.64.186.148:3306: connect: connection refused	{"type": "ScaledObject", "namespace": "invu-sre", "name": "keda-session-scaler", "error": "dial tcp 100.64.186.148:3306: connect: connection refused"}
github.com/kedacore/keda/v2/pkg/scalers.NewMySQLScaler
	/workspace/pkg/scalers/mysql_scaler.go:53
github.com/kedacore/keda/v2/pkg/scaling.buildScaler
	/workspace/pkg/scaling/scale_handler.go:431
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).buildScalers.func1
	/workspace/pkg/scaling/scale_handler.go:325
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).buildScalers
	/workspace/pkg/scaling/scale_handler.go:328
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).GetScalersCache
	/workspace/pkg/scaling/scale_handler.go:194
github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).getScaledObjectMetricSpecs
	/workspace/controllers/keda/hpa.go:202
github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).newHPAForScaledObject
	/workspace/controllers/keda/hpa.go:74
github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).updateHPAIfNeeded
	/workspace/controllers/keda/hpa.go:152
github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).ensureHPAForScaledObjectExists
	/workspace/controllers/keda/scaledobject_controller.go:410
github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).reconcileScaledObject
	/workspace/controllers/keda/scaledobject_controller.go:241
github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).Reconcile
	/workspace/controllers/keda/scaledobject_controller.go:192
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/internal/controller/controller.go:121
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/internal/controller/controller.go:320
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/internal/controller/controller.go:273
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/internal/controller/controller.go:234
2023-02-20T14:36:52Z	ERROR	scalehandler	error resolving auth params	{"scalerIndex": 0, "object": {"apiVersion": "keda.sh/v1alpha1", "kind": "ScaledObject", "namespace": "invu-sre", "name": "keda-session-scaler"}, "error": "error establishing MySQL connection: dial tcp 100.64.186.148:3306: connect: connection refused"}
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).GetScalersCache
	/workspace/pkg/scaling/scale_handler.go:194
github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).getScaledObjectMetricSpecs
	/workspace/controllers/keda/hpa.go:202
github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).newHPAForScaledObject
	/workspace/controllers/keda/hpa.go:74
github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).updateHPAIfNeeded
	/workspace/controllers/keda/hpa.go:152
github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).ensureHPAForScaledObjectExists
	/workspace/controllers/keda/scaledobject_controller.go:410
github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).reconcileScaledObject
	/workspace/controllers/keda/scaledobject_controller.go:241
github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).Reconcile
	/workspace/controllers/keda/scaledobject_controller.go:192
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/internal/controller/controller.go:121
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/internal/controller/controller.go:320
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/internal/controller/controller.go:273
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/internal/controller/controller.go:234
2023-02-20T14:36:52Z	ERROR	Error getting scalers	{"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "scaledObject": {"name":"keda-session-scaler","namespace":"invu-sre"}, "namespace": "invu-sre", "name": "keda-session-scaler", "reconcileID": "d6eaaaf7-5633-4e16-9093-7108dbfb736e", "error": "error establishing MySQL connection: dial tcp 100.64.186.148:3306: connect: connection refused"}
github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).newHPAForScaledObject
	/workspace/controllers/keda/hpa.go:74
github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).updateHPAIfNeeded
	/workspace/controllers/keda/hpa.go:152
github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).ensureHPAForScaledObjectExists
	/workspace/controllers/keda/scaledobject_controller.go:410
github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).reconcileScaledObject
	/workspace/controllers/keda/scaledobject_controller.go:241
github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).Reconcile
	/workspace/controllers/keda/scaledobject_controller.go:192
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/internal/controller/controller.go:121
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/internal/controller/controller.go:320
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/internal/controller/controller.go:273
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/internal/controller/controller.go:234
2023-02-20T14:36:52Z	ERROR	Failed to create new HPA resource	{"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "scaledObject": {"name":"keda-session-scaler","namespace":"invu-sre"}, "namespace": "invu-sre", "name": "keda-session-scaler", "reconcileID": "d6eaaaf7-5633-4e16-9093-7108dbfb736e", "HPA.Namespace": "invu-sre", "HPA.Name": "keda-hpa-keda-session-scaler", "error": "error establishing MySQL connection: dial tcp 100.64.186.148:3306: connect: connection refused"}
github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).ensureHPAForScaledObjectExists
	/workspace/controllers/keda/scaledobject_controller.go:410
github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).reconcileScaledObject
	/workspace/controllers/keda/scaledobject_controller.go:241
github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).Reconcile
	/workspace/controllers/keda/scaledobject_controller.go:192
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/internal/controller/controller.go:121
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/internal/controller/controller.go:320
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/internal/controller/controller.go:273
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/internal/controller/controller.go:234
2023-02-20T14:36:52Z	ERROR	Failed to check HPA for possible update	{"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "scaledObject": {"name":"keda-session-scaler","namespace":"invu-sre"}, "namespace": "invu-sre", "name": "keda-session-scaler", "reconcileID": "d6eaaaf7-5633-4e16-9093-7108dbfb736e", "error": "error establishing MySQL connection: dial tcp 100.64.186.148:3306: connect: connection refused"}
github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).reconcileScaledObject
	/workspace/controllers/keda/scaledobject_controller.go:241
github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).Reconcile
	/workspace/controllers/keda/scaledobject_controller.go:192
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/internal/controller/controller.go:121
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/internal/controller/controller.go:320
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/internal/controller/controller.go:273
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/internal/controller/controller.go:234
2023-02-20T14:36:52Z	ERROR	Failed to ensure HPA is correctly created for ScaledObject	{"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "scaledObject": {"name":"keda-session-scaler","namespace":"invu-sre"}, "namespace": "invu-sre", "name": "keda-session-scaler", "reconcileID": "d6eaaaf7-5633-4e16-9093-7108dbfb736e", "error": "error establishing MySQL connection: dial tcp 100.64.186.148:3306: connect: connection refused"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/internal/controller/controller.go:121
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/internal/controller/controller.go:320
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/internal/controller/controller.go:273
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/internal/controller/controller.go:234
2023-02-20T14:36:52Z	DEBUG	events	Warning	{"object": {"kind":"ScaledObject","namespace":"invu-sre","name":"keda-session-scaler","uid":"495af290-9075-4185-81bd-8f2dcebaa40b","apiVersion":"keda.sh/v1alpha1","resourceVersion":"258835449"}, "reason": "KEDAScalerFailed", "message": "error establishing MySQL connection: dial tcp 100.64.186.148:3306: connect: connection refused"}
2023-02-20T14:36:52Z	DEBUG	events	Warning	{"object": {"kind":"ScaledObject","namespace":"invu-sre","name":"keda-session-scaler","uid":"495af290-9075-4185-81bd-8f2dcebaa40b","apiVersion":"keda.sh/v1alpha1","resourceVersion":"258835449"}, "reason": "ScaledObjectCheckFailed", "message": "Failed to ensure HPA is correctly created for ScaledObject"}
2023-02-20T14:36:52Z	ERROR	Reconciler error	{"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "scaledObject": {"name":"keda-session-scaler","namespace":"invu-sre"}, "namespace": "invu-sre", "name": "keda-session-scaler", "reconcileID": "d6eaaaf7-5633-4e16-9093-7108dbfb736e", "error": "error establishing MySQL connection: dial tcp 100.64.186.148:3306: connect: connection refused"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/internal/controller/controller.go:273
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/internal/controller/controller.go:234
2023-02-20T14:36:55Z	INFO	Reconciling ScaledObject	{"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "scaledObject": {"name":"keda-session-scaler","namespace":"invu-sre"}, "namespace": "invu-sre", "name": "keda-session-scaler", "reconcileID": "a09319ae-39da-4f3b-bf57-b1ddf85f18c3"}
2023-02-20T14:36:55Z	DEBUG	Parsed Group, Version, Kind, Resource	{"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "scaledObject": {"name":"keda-session-scaler","namespace":"invu-sre"}, "namespace": "invu-sre", "name": "keda-session-scaler", "reconcileID": "a09319ae-39da-4f3b-bf57-b1ddf85f18c3", "GVK": "apps/v1.Deployment", "Resource": "deployments"}
2023-02-20T14:36:55Z	DEBUG	scalehandler	cannot resolve env to a value. fieldRef and resourceFieldRef env are skipped	{"type": "ScaledObject", "namespace": "invu-sre", "name": "keda-session-scaler", "env-var-name": "INSTANCE"}
2023-02-20T14:36:55Z	DEBUG	scalehandler	cannot resolve env to a value. fieldRef and resourceFieldRef env are skipped	{"type": "ScaledObject", "namespace": "invu-sre", "name": "keda-session-scaler", "env-var-name": "MY_POD_NAMESPACE"}
2023-02-20T14:36:55Z	ERROR	mysql_scaler	Found error when pinging database: dial tcp 100.64.186.148:3306: connect: connection refused	{"type": "ScaledObject", "namespace": "invu-sre", "name": "keda-session-scaler", "error": "dial tcp 100.64.186.148:3306: connect: connection refused"}
github.com/kedacore/keda/v2/pkg/scalers.NewMySQLScaler
	/workspace/pkg/scalers/mysql_scaler.go:53
github.com/kedacore/keda/v2/pkg/scaling.buildScaler
	/workspace/pkg/scaling/scale_handler.go:431
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).buildScalers.func1
	/workspace/pkg/scaling/scale_handler.go:325
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).buildScalers
	/workspace/pkg/scaling/scale_handler.go:328
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).GetScalersCache
	/workspace/pkg/scaling/scale_handler.go:194
github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).getScaledObjectMetricSpecs
	/workspace/controllers/keda/hpa.go:202
github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).newHPAForScaledObject
	/workspace/controllers/keda/hpa.go:74
github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).updateHPAIfNeeded
	/workspace/controllers/keda/hpa.go:152
github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).ensureHPAForScaledObjectExists
	/workspace/controllers/keda/scaledobject_controller.go:410
github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).reconcileScaledObject
	/workspace/controllers/keda/scaledobject_controller.go:241
github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).Reconcile
	/workspace/controllers/keda/scaledobject_controller.go:192
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/internal/controller/controller.go:121
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/internal/controller/controller.go:320
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/internal/controller/controller.go:273
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/internal/controller/controller.go:234
2023-02-20T14:36:55Z	ERROR	scalehandler	error resolving auth params	{"scalerIndex": 0, "object": {"apiVersion": "keda.sh/v1alpha1", "kind": "ScaledObject", "namespace": "invu-sre", "name": "keda-session-scaler"}, "error": "error establishing MySQL connection: dial tcp 100.64.186.148:3306: connect: connection refused"}
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).GetScalersCache
	/workspace/pkg/scaling/scale_handler.go:194
github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).getScaledObjectMetricSpecs
	/workspace/controllers/keda/hpa.go:202
github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).newHPAForScaledObject
	/workspace/controllers/keda/hpa.go:74
github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).updateHPAIfNeeded
	/workspace/controllers/keda/hpa.go:152
github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).ensureHPAForScaledObjectExists
	/workspace/controllers/keda/scaledobject_controller.go:410
github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).reconcileScaledObject
	/workspace/controllers/keda/scaledobject_controller.go:241
github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).Reconcile
	/workspace/controllers/keda/scaledobject_controller.go:192
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/internal/controller/controller.go:121
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/internal/controller/controller.go:320
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/internal/controller/controller.go:273
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/internal/controller/controller.go:234
2023-02-20T14:36:55Z	ERROR	Error getting scalers	{"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "scaledObject": {"name":"keda-session-scaler","namespace":"invu-sre"}, "namespace": "invu-sre", "name": "keda-session-scaler", "reconcileID": "a09319ae-39da-4f3b-bf57-b1ddf85f18c3", "error": "error establishing MySQL connection: dial tcp 100.64.186.148:3306: connect: connection refused"}
github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).newHPAForScaledObject
	/workspace/controllers/keda/hpa.go:74
github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).updateHPAIfNeeded
	/workspace/controllers/keda/hpa.go:152
github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).ensureHPAForScaledObjectExists
	/workspace/controllers/keda/scaledobject_controller.go:410
github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).reconcileScaledObject
	/workspace/controllers/keda/scaledobject_controller.go:241
github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).Reconcile
	/workspace/controllers/keda/scaledobject_controller.go:192
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/internal/controller/controller.go:121
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/internal/controller/controller.go:320
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/internal/controller/controller.go:273
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/internal/controller/controller.go:234
2023-02-20T14:36:55Z	ERROR	Failed to create new HPA resource	{"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "scaledObject": {"name":"keda-session-scaler","namespace":"invu-sre"}, "namespace": "invu-sre", "name": "keda-session-scaler", "reconcileID": "a09319ae-39da-4f3b-bf57-b1ddf85f18c3", "HPA.Namespace": "invu-sre", "HPA.Name": "keda-hpa-keda-session-scaler", "error": "error establishing MySQL connection: dial tcp 100.64.186.148:3306: connect: connection refused"}
github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).ensureHPAForScaledObjectExists
	/workspace/controllers/keda/scaledobject_controller.go:410
github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).reconcileScaledObject
	/workspace/controllers/keda/scaledobject_controller.go:241
github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).Reconcile
	/workspace/controllers/keda/scaledobject_controller.go:192
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/internal/controller/controller.go:121
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/internal/controller/controller.go:320
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/internal/controller/controller.go:273
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/internal/controller/controller.go:234
2023-02-20T14:36:55Z	ERROR	Failed to check HPA for possible update	{"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "scaledObject": {"name":"keda-session-scaler","namespace":"invu-sre"}, "namespace": "invu-sre", "name": "keda-session-scaler", "reconcileID": "a09319ae-39da-4f3b-bf57-b1ddf85f18c3", "error": "error establishing MySQL connection: dial tcp 100.64.186.148:3306: connect: connection refused"}
github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).reconcileScaledObject
	/workspace/controllers/keda/scaledobject_controller.go:241
github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).Reconcile
	/workspace/controllers/keda/scaledobject_controller.go:192
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/internal/controller/controller.go:121
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/internal/controller/controller.go:320
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/internal/controller/controller.go:273
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/internal/controller/controller.go:234
2023-02-20T14:36:55Z	ERROR	Failed to ensure HPA is correctly created for ScaledObject	{"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "scaledObject": {"name":"keda-session-scaler","namespace":"invu-sre"}, "namespace": "invu-sre", "name": "keda-session-scaler", "reconcileID": "a09319ae-39da-4f3b-bf57-b1ddf85f18c3", "error": "error establishing MySQL connection: dial tcp 100.64.186.148:3306: connect: connection refused"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/internal/controller/controller.go:121
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/internal/controller/controller.go:320
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/internal/controller/controller.go:273
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/internal/controller/controller.go:234
2023-02-20T14:36:55Z	DEBUG	events	Warning	{"object": {"kind":"ScaledObject","namespace":"invu-sre","name":"keda-session-scaler","uid":"495af290-9075-4185-81bd-8f2dcebaa40b","apiVersion":"keda.sh/v1alpha1","resourceVersion":"258835449"}, "reason": "KEDAScalerFailed", "message": "error establishing MySQL connection: dial tcp 100.64.186.148:3306: connect: connection refused"}
2023-02-20T14:36:55Z	DEBUG	events	Warning	{"object": {"kind":"ScaledObject","namespace":"invu-sre","name":"keda-session-scaler","uid":"495af290-9075-4185-81bd-8f2dcebaa40b","apiVersion":"keda.sh/v1alpha1","resourceVersion":"258835449"}, "reason": "ScaledObjectCheckFailed", "message": "Failed to ensure HPA is correctly created for ScaledObject"}
2023-02-20T14:36:55Z	ERROR	Reconciler error	{"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "scaledObject": {"name":"keda-session-scaler","namespace":"invu-sre"}, "namespace": "invu-sre", "name": "keda-session-scaler", "reconcileID": "a09319ae-39da-4f3b-bf57-b1ddf85f18c3", "error": "error establishing MySQL connection: dial tcp 100.64.186.148:3306: connect: connection refused"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/internal/controller/controller.go:273
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/internal/controller/controller.go:234

KEDA Version

2.8.1

Kubernetes Version

1.25

Platform

Amazon Web Services

Scaler Details

mysql

Anything else?

willing to help fix this

@aharonh aharonh added the bug Something isn't working label Feb 20, 2023
aharonh added a commit to aharonh/keda that referenced this issue Feb 20, 2023
@aharonh
Copy link
Author

aharonh commented Feb 20, 2023

add pull request #4254
please review and merge

thanks

@zroubalik
Copy link
Member

zroubalik commented Feb 21, 2023

I understand the rationale behind your ask. Currently we just pause scaling but scalers are connected, also HPA is still in place.

We would have to do a bigger change. In the controller in the reconcile loop stop the scaling logic, so we are not connecting the to scalers.

// requestScaleLoop tries to start ScaleLoop handler for the respective ScaledObject
func (r *ScaledObjectReconciler) requestScaleLoop(ctx context.Context, logger logr.Logger, scaledObject *kedav1alpha1.ScaledObject) error {
logger.V(1).Info("Notify scaleHandler of an update in scaledObject")
key, err := cache.MetaNamespaceKeyFunc(scaledObject)
if err != nil {
logger.Error(err, "Error getting key for scaledObject")
return err
}
if err = r.ScaleHandler.HandleScalableObject(ctx, scaledObject); err != nil {
return err
}
// store ScaledObject's current Generation
r.scaledObjectsGenerations.Store(key, scaledObject.Generation)
return nil
}
// stopScaleLoop stops ScaleLoop handler for the respective ScaledObject
func (r *ScaledObjectReconciler) stopScaleLoop(ctx context.Context, logger logr.Logger, scaledObject *kedav1alpha1.ScaledObject) error {
key, err := cache.MetaNamespaceKeyFunc(scaledObject)
if err != nil {
logger.Error(err, "Error getting key for scaledObject")
return err
}
if err := r.ScaleHandler.DeleteScalableObject(ctx, scaledObject); err != nil {
return err
}
// delete ScaledObject's current Generation
r.scaledObjectsGenerations.Delete(key)
return nil
}

And also delete the generated HPA, so it is not querying KEDA Metric Server for metrics

func (r *ScaledObjectReconciler) ensureHPAForScaledObjectExists(ctx context.Context, logger logr.Logger, scaledObject *kedav1alpha1.ScaledObject, gvkr *kedav1alpha1.GroupVersionKindResource) (bool, error) {

@aharonh are you willing to give it a try?

aharonh added a commit to aharonh/keda that referenced this issue Feb 21, 2023
Signed-off-by: aharonh <aharon.haravon@gmail.com>
@aharonh
Copy link
Author

aharonh commented Feb 21, 2023

Hi @zroubalik ,

Hi, did you actually test this change? I don't think it will help with your issue, because majority of the scaling logic is being driven by scaler handler: https://github.com/kedacore/keda/blob/main/pkg/scaling/scale_handler.go

I think that we should have a different approach.

I just re-did the changes with the gpg signature that was missing. Yep, I tested it, looked like it works as expected:

  • There were no errors when the ScaledObject was paused
  • When unpaused, it worked as expected

Can you think of any further tests that would be useful to perform?

Thanks,
Aharon

@aharonh
Copy link
Author

aharonh commented Feb 21, 2023

I see though what you are saying... I guess the reason it works in our case is that we are in addition setting the deployment replicas to 0 when scaling down the namespace to 0. I guess that may be the reason that this fix helps us resolve the issue at least in our case.

@aharonh
Copy link
Author

aharonh commented Feb 21, 2023

Do you agree to merge this fix and we can open another issue I can separately work on that will fix the issue in wider sense as you suggested?

@aharonh
Copy link
Author

aharonh commented Feb 21, 2023

Asking just to make sure that our production will not have to run with patched version for longer time ;)

aharonh added a commit to aharonh/keda that referenced this issue Feb 21, 2023
Signed-off-by: aharonh <aharon.haravon@gmail.com>
@zroubalik
Copy link
Member

zroubalik commented Feb 21, 2023

@aharonh yeah I see. I would actually prefer if we go straight with the approach I suggested, it is more clear solution. We are going to do a new release in ~2 weeks, so there's still some time. Just let me know if you are interested in contribute that, if not we can try to ask someone else :)

@aharonh
Copy link
Author

aharonh commented Feb 21, 2023

ok, @zroubalik I will try to have a look into it, thanks!

@stale
Copy link

stale bot commented Apr 22, 2023

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 7 days if no further activity occurs. Thank you for your contributions.

@stale stale bot added the stale All issues that are marked as stale due to inactivity label Apr 22, 2023
@aharonh
Copy link
Author

aharonh commented Apr 23, 2023

still intend to have a look into it, not immediately though so if anyone else has the time and want's to handle this, you are welcome.

@stale stale bot removed the stale All issues that are marked as stale due to inactivity label Apr 23, 2023
@tobotg
Copy link
Contributor

tobotg commented Apr 23, 2023

Hello @JorTurFer and @aharonh.
I'll be happy to look into it and contribute if you're ok. Please let me know.
Thanks,

@tomkerkhove
Copy link
Member

That would be wonderful, thank you @tobotg!

tobotg added a commit to tobotg/keda that referenced this issue Apr 24, 2023
Signed-off-by: Tobo Atchou <tobo.atchou@gmail.com>
@tobotg
Copy link
Contributor

tobotg commented Apr 24, 2023

@tomkerkhove @zroubalik
I tried to follow the indications @zroubalik gave above and filed a PR. I'm happy to rework it as review and feedback come.
Thanks

tobotg added a commit to tobotg/keda that referenced this issue Apr 24, 2023
tobotg added a commit to tobotg/keda that referenced this issue May 2, 2023
Signed-off-by: Tobo Atchou <tobo.atchou@gmail.com>
tobotg added a commit to tobotg/keda that referenced this issue May 6, 2023
Signed-off-by: Tobo Atchou <tobo.atchou@gmail.com>
tobotg added a commit to tobotg/keda that referenced this issue May 17, 2023
Signed-off-by: Tobo Atchou <tobo.atchou@gmail.com>
tobotg added a commit to tobotg/keda that referenced this issue May 18, 2023
Signed-off-by: Tobo Atchou <tobo.atchou@gmail.com>
tobotg added a commit to tobotg/keda that referenced this issue May 18, 2023
Signed-off-by: Tobo Atchou <tobo.atchou@gmail.com>
tobotg added a commit to tobotg/keda that referenced this issue May 18, 2023
Signed-off-by: Tobo Atchou <tobo.atchou@gmail.com>
tobotg added a commit to tobotg/keda that referenced this issue May 31, 2023
Signed-off-by: Tobo Atchou <tobo.atchou@gmail.com>
tobotg added a commit to tobotg/keda that referenced this issue May 31, 2023
…or msg and new condition msg

Signed-off-by: Tobo Atchou <tobo.atchou@gmail.com>
tobotg added a commit to tobotg/keda that referenced this issue May 31, 2023
…sed condition

Signed-off-by: Tobo Atchou <tobo.atchou@gmail.com>
tobotg added a commit to tobotg/keda that referenced this issue May 31, 2023
… get hpa name and paused condition

Signed-off-by: Tobo Atchou <tobo.atchou@gmail.com>
tobotg added a commit to tobotg/keda that referenced this issue May 31, 2023
…ifest for paused condition

Signed-off-by: Tobo Atchou <tobo.atchou@gmail.com>
tobotg added a commit to tobotg/keda that referenced this issue Jun 4, 2023
Signed-off-by: Tobo Atchou <tobo.atchou@gmail.com>
tobotg added a commit to tobotg/keda that referenced this issue Jun 4, 2023
…or msg and new condition msg

Signed-off-by: Tobo Atchou <tobo.atchou@gmail.com>
tobotg added a commit to tobotg/keda that referenced this issue Jun 4, 2023
…sed condition

Signed-off-by: Tobo Atchou <tobo.atchou@gmail.com>
tobotg added a commit to tobotg/keda that referenced this issue Jun 4, 2023
… get hpa name and paused condition

Signed-off-by: Tobo Atchou <tobo.atchou@gmail.com>
tobotg added a commit to tobotg/keda that referenced this issue Jun 4, 2023
…ifest for paused condition

Signed-off-by: Tobo Atchou <tobo.atchou@gmail.com>
tobotg added a commit to tobotg/keda that referenced this issue Jun 4, 2023
…rease test wait timeout

Signed-off-by: Tobo Atchou <tobo.atchou@gmail.com>
tobotg added a commit to tobotg/keda that referenced this issue Jun 8, 2023
Signed-off-by: Tobo Atchou <tobo.atchou@gmail.com>
tobotg added a commit to tobotg/keda that referenced this issue Jun 8, 2023
…or msg and new condition msg

Signed-off-by: Tobo Atchou <tobo.atchou@gmail.com>
tobotg added a commit to tobotg/keda that referenced this issue Jun 8, 2023
…sed condition

Signed-off-by: Tobo Atchou <tobo.atchou@gmail.com>
tobotg added a commit to tobotg/keda that referenced this issue Jun 8, 2023
… get hpa name and paused condition

Signed-off-by: Tobo Atchou <tobo.atchou@gmail.com>
tobotg added a commit to tobotg/keda that referenced this issue Jun 8, 2023
…ifest for paused condition

Signed-off-by: Tobo Atchou <tobo.atchou@gmail.com>
tobotg added a commit to tobotg/keda that referenced this issue Jun 8, 2023
…rease test wait timeout

Signed-off-by: Tobo Atchou <tobo.atchou@gmail.com>
tobotg added a commit to tobotg/keda that referenced this issue Jun 8, 2023
…ed up test with slim container image and wait for monitored deployment

Signed-off-by: Tobo Atchou <tobo.atchou@gmail.com>
tobotg added a commit to tobotg/keda that referenced this issue Jun 14, 2023
Signed-off-by: Tobo Atchou <tobo.atchou@gmail.com>
tobotg added a commit to tobotg/keda that referenced this issue Jun 14, 2023
…or msg and new condition msg

Signed-off-by: Tobo Atchou <tobo.atchou@gmail.com>
tobotg added a commit to tobotg/keda that referenced this issue Jun 14, 2023
…sed condition

Signed-off-by: Tobo Atchou <tobo.atchou@gmail.com>
tobotg added a commit to tobotg/keda that referenced this issue Jun 14, 2023
… get hpa name and paused condition

Signed-off-by: Tobo Atchou <tobo.atchou@gmail.com>
tobotg added a commit to tobotg/keda that referenced this issue Jun 14, 2023
…ifest for paused condition

Signed-off-by: Tobo Atchou <tobo.atchou@gmail.com>
tobotg added a commit to tobotg/keda that referenced this issue Jun 14, 2023
…rease test wait timeout

Signed-off-by: Tobo Atchou <tobo.atchou@gmail.com>
tobotg added a commit to tobotg/keda that referenced this issue Jun 14, 2023
…ed up test with slim container image and wait for monitored deployment

Signed-off-by: Tobo Atchou <tobo.atchou@gmail.com>
tobotg added a commit to tobotg/keda that referenced this issue Jun 14, 2023
Signed-off-by: Tobo Atchou <tobo.atchou@gmail.com>
zroubalik pushed a commit that referenced this issue Jun 18, 2023
Signed-off-by: Tobo Atchou <tobo.atchou@gmail.com>
SpiritZhou pushed a commit to SpiritZhou/keda that referenced this issue Jun 30, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
Archived in project
4 participants