Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[BUG] kubeblocks CrashLoopBackOff Observed a panic in reconciler cluster scaleOut ops #8651

Closed
JashBook opened this issue Dec 16, 2024 · 1 comment · Fixed by #8659
Closed

[BUG] kubeblocks CrashLoopBackOff Observed a panic in reconciler cluster scaleOut ops #8651

JashBook opened this issue Dec 16, 2024 · 1 comment · Fixed by #8659
Assignees
@kubeJocker
Labels
kind/bug Something isn't working
Milestone
Release 0.9.2

Comments

@JashBook
Copy link
Collaborator

Describe the bug

kbcli version
Kubernetes: v1.30.6
KubeBlocks: 0.9.3-beta.12
kbcli: 0.9.2

To Reproduce
Steps to reproduce the behavior:

  1. Go to '...'
  2. Click on '....'
  3. Scroll down to '....'
  4. See error
kubectl get pod  -n kb-iccwnqzrvb                                                
NAME                                              READY   STATUS             RESTARTS      AGE
kubeblocks-66ccd795cf-2qrsc                       0/1     CrashLoopBackOff   8 (26s ago)   41m
kubeblocks-dataprotection-76d5978d8c-pjqx7        1/1     Running            0             41m

logs kubeblocks pod

kubectl logs -n kb-iccwnqzrvb  kubeblocks-66ccd795cf-2qrsc

2024-12-16T02:41:03.046Z	ERROR	failed to validate config template!	{"controller": "componentdefinition", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "ComponentDefinition", "ComponentDefinition": {"name":"loki-read"}, "namespace": "", "name": "loki-read", "reconcileID": "ce3ec727-174f-43e9-aecd-95ad14c8c820", "component": {"name":"loki-read"}, "templateName": "loki-config", "configMapName": "loki", "error": "configmaps \"loki\" not found"}
github.com/apecloud/kubeblocks/controllers/apps/configuration.validateConfigTemplate
	/src/controllers/apps/configuration/config_util.go:375
github.com/apecloud/kubeblocks/controllers/apps/configuration.checkConfigTemplate.func1
	/src/controllers/apps/configuration/config_util.go:332
github.com/apecloud/kubeblocks/controllers/apps/configuration.handleConfigTemplate
	/src/controllers/apps/configuration/config_util.go:277
github.com/apecloud/kubeblocks/controllers/apps/configuration.checkConfigTemplate
	/src/controllers/apps/configuration/config_util.go:334
github.com/apecloud/kubeblocks/controllers/apps/configuration.ReconcileConfigSpecsForReferencedCR[...]
	/src/controllers/apps/configuration/config_util.go:106
github.com/apecloud/kubeblocks/controllers/apps.(*ComponentDefinitionReconciler).validateConfigs
	/src/controllers/apps/componentdefinition_controller.go:340
github.com/apecloud/kubeblocks/controllers/apps.(*ComponentDefinitionReconciler).validate
	/src/controllers/apps/componentdefinition_controller.go:201
github.com/apecloud/kubeblocks/controllers/apps.(*ComponentDefinitionReconciler).reconcile
	/src/controllers/apps/componentdefinition_controller.go:107
github.com/apecloud/kubeblocks/controllers/apps.(*ComponentDefinitionReconciler).Reconcile
	/src/controllers/apps/componentdefinition_controller.go:85
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:119
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:316
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:266
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:227
2024-12-16T02:41:03.067Z	INFO	failed to check config template: configmaps "loki" not found	{"controller": "componentdefinition", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "ComponentDefinition", "ComponentDefinition": {"name":"loki-read"}, "namespace": "", "name": "loki-read", "reconcileID": "ce3ec727-174f-43e9-aecd-95ad14c8c820", "component": {"name":"loki-read"}}
2024-12-16T02:41:03.067Z	ERROR	Reconciler error	{"controller": "componentdefinition", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "ComponentDefinition", "ComponentDefinition": {"name":"loki-read"}, "namespace": "", "name": "loki-read", "reconcileID": "ce3ec727-174f-43e9-aecd-95ad14c8c820", "error": "failed to check config template: configmaps \"loki\" not found"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:329
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:266
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:227
2024-12-16T02:41:05.358Z	INFO	reconcile	{"controller": "opsrequest", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "OpsRequest", "OpsRequest": {"name":"camellia-qszmws-restart-w579d","namespace":"ns-doqwtruiby"}, "namespace": "ns-doqwtruiby", "name": "camellia-qszmws-restart-w579d", "reconcileID": "3565f58e-2e56-4ff9-8bbc-792004692b73", "opsRequest": {"name":"camellia-qszmws-restart-w579d","namespace":"ns-doqwtruiby"}, "opsRequest": {"name":"camellia-qszmws-restart-w579d","namespace":"ns-doqwtruiby"}}
2024-12-16T02:41:05.379Z	INFO	reconcile	{"controller": "opsrequest", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "OpsRequest", "OpsRequest": {"name":"camellia-qszmws-restart-w579d","namespace":"ns-doqwtruiby"}, "namespace": "ns-doqwtruiby", "name": "camellia-qszmws-restart-w579d", "reconcileID": "28bc11b2-b6b2-4082-814e-09b944f51348", "opsRequest": {"name":"camellia-qszmws-restart-w579d","namespace":"ns-doqwtruiby"}, "opsRequest": {"name":"camellia-qszmws-restart-w579d","namespace":"ns-doqwtruiby"}}
2024-12-16T02:41:05.525Z	INFO	reconcile	{"controller": "opsrequest", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "OpsRequest", "OpsRequest": {"name":"camellia-qszmws-restart-w579d","namespace":"ns-doqwtruiby"}, "namespace": "ns-doqwtruiby", "name": "camellia-qszmws-restart-w579d", "reconcileID": "016a9e1c-b17b-45d1-a796-2d80efc3247b", "opsRequest": {"name":"camellia-qszmws-restart-w579d","namespace":"ns-doqwtruiby"}, "opsRequest": {"name":"camellia-qszmws-restart-w579d","namespace":"ns-doqwtruiby"}}
2024-12-16T02:41:05.548Z	INFO	reconcile	{"controller": "opsrequest", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "OpsRequest", "OpsRequest": {"name":"camellia-qszmws-restart-w579d","namespace":"ns-doqwtruiby"}, "namespace": "ns-doqwtruiby", "name": "camellia-qszmws-restart-w579d", "reconcileID": "ee4ec143-c36e-4cfa-a385-293a1ab92da7", "opsRequest": {"name":"camellia-qszmws-restart-w579d","namespace":"ns-doqwtruiby"}, "opsRequest": {"name":"camellia-qszmws-restart-w579d","namespace":"ns-doqwtruiby"}}
2024-12-16T02:41:05.714Z	INFO	reconcile	{"controller": "opsrequest", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "OpsRequest", "OpsRequest": {"name":"strsent-tvqwed-restart-nfrdd","namespace":"ns-zvwzoswjnh"}, "namespace": "ns-zvwzoswjnh", "name": "strsent-tvqwed-restart-nfrdd", "reconcileID": "a174b54c-218d-4d68-a78f-925a5fcca1d5", "opsRequest": {"name":"strsent-tvqwed-restart-nfrdd","namespace":"ns-zvwzoswjnh"}, "opsRequest": {"name":"strsent-tvqwed-restart-nfrdd","namespace":"ns-zvwzoswjnh"}}
2024-12-16T02:41:05.738Z	INFO	reconcile	{"controller": "opsrequest", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "OpsRequest", "OpsRequest": {"name":"strsent-tvqwed-restart-nfrdd","namespace":"ns-zvwzoswjnh"}, "namespace": "ns-zvwzoswjnh", "name": "strsent-tvqwed-restart-nfrdd", "reconcileID": "cc4c35f1-7746-4d50-8a00-858c2d045b7d", "opsRequest": {"name":"strsent-tvqwed-restart-nfrdd","namespace":"ns-zvwzoswjnh"}, "opsRequest": {"name":"strsent-tvqwed-restart-nfrdd","namespace":"ns-zvwzoswjnh"}}
2024-12-16T02:41:05.891Z	INFO	status conditions, creating: false, available: true, its running: true, has failure: false, updating: false, config synced: true	{"controller": "component", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Component", "Component": {"name":"strsent-tvqwed-be","namespace":"ns-zvwzoswjnh"}, "namespace": "ns-zvwzoswjnh", "name": "strsent-tvqwed-be", "reconcileID": "0e8c482a-7822-448e-9ceb-70c14d8e5349", "component": {"name":"strsent-tvqwed-be","namespace":"ns-zvwzoswjnh"}}
2024-12-16T02:41:05.915Z	INFO	reconcile	{"controller": "opsrequest", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "OpsRequest", "OpsRequest": {"name":"strsent-tvqwed-restart-nfrdd","namespace":"ns-zvwzoswjnh"}, "namespace": "ns-zvwzoswjnh", "name": "strsent-tvqwed-restart-nfrdd", "reconcileID": "853b765b-cc43-4294-bce0-ff522273f793", "opsRequest": {"name":"strsent-tvqwed-restart-nfrdd","namespace":"ns-zvwzoswjnh"}, "opsRequest": {"name":"strsent-tvqwed-restart-nfrdd","namespace":"ns-zvwzoswjnh"}}
2024-12-16T02:41:05.918Z	INFO	cluster component phase transition: Updating -> Running (component is Running)	{"controller": "cluster", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Cluster", "Cluster": {"name":"strsent-tvqwed","namespace":"ns-zvwzoswjnh"}, "namespace": "ns-zvwzoswjnh", "name": "strsent-tvqwed", "reconcileID": "801ac33c-b699-4a19-8cef-3d0d19c88a66", "cluster": {"name":"strsent-tvqwed","namespace":"ns-zvwzoswjnh"}}
2024-12-16T02:41:05.949Z	INFO	reconcile	{"controller": "opsrequest", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "OpsRequest", "OpsRequest": {"name":"strsent-tvqwed-restart-nfrdd","namespace":"ns-zvwzoswjnh"}, "namespace": "ns-zvwzoswjnh", "name": "strsent-tvqwed-restart-nfrdd", "reconcileID": "947ab51b-3669-460e-a5d2-8671ed93fa87", "opsRequest": {"name":"strsent-tvqwed-restart-nfrdd","namespace":"ns-zvwzoswjnh"}, "opsRequest": {"name":"strsent-tvqwed-restart-nfrdd","namespace":"ns-zvwzoswjnh"}}
2024-12-16T02:41:06.010Z	INFO	reconcile	{"controller": "opsrequest", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "OpsRequest", "OpsRequest": {"name":"strsent-tvqwed-restart-nfrdd","namespace":"ns-zvwzoswjnh"}, "namespace": "ns-zvwzoswjnh", "name": "strsent-tvqwed-restart-nfrdd", "reconcileID": "97d0ca7e-2525-4aeb-8756-9c90beb1bcfc", "opsRequest": {"name":"strsent-tvqwed-restart-nfrdd","namespace":"ns-zvwzoswjnh"}, "opsRequest": {"name":"strsent-tvqwed-restart-nfrdd","namespace":"ns-zvwzoswjnh"}}
2024-12-16T02:41:06.083Z	INFO	status conditions, creating: false, available: true, its running: true, has failure: false, updating: false, config synced: true	{"controller": "component", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Component", "Component": {"name":"strsent-tvqwed-be","namespace":"ns-zvwzoswjnh"}, "namespace": "ns-zvwzoswjnh", "name": "strsent-tvqwed-be", "reconcileID": "8cb01d43-efd0-49d9-8194-76edba3928d2", "component": {"name":"strsent-tvqwed-be","namespace":"ns-zvwzoswjnh"}}
2024-12-16T02:41:06.192Z	INFO	reconcile	{"controller": "opsrequest", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "OpsRequest", "OpsRequest": {"name":"camellia-qszmws-volumeexpansion-x2s6s","namespace":"ns-doqwtruiby"}, "namespace": "ns-doqwtruiby", "name": "camellia-qszmws-volumeexpansion-x2s6s", "reconcileID": "c8205871-93f0-4257-942e-228abb1f69bc", "opsRequest": {"name":"camellia-qszmws-volumeexpansion-x2s6s","namespace":"ns-doqwtruiby"}, "opsRequest": {"name":"camellia-qszmws-volumeexpansion-x2s6s","namespace":"ns-doqwtruiby"}}
2024-12-16T02:41:06.227Z	INFO	reconcile	{"controller": "opsrequest", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "OpsRequest", "OpsRequest": {"name":"camellia-qszmws-volumeexpansion-x2s6s","namespace":"ns-doqwtruiby"}, "namespace": "ns-doqwtruiby", "name": "camellia-qszmws-volumeexpansion-x2s6s", "reconcileID": "5a3840ff-b046-436b-8d85-aa2116c32047", "opsRequest": {"name":"camellia-qszmws-volumeexpansion-x2s6s","namespace":"ns-doqwtruiby"}, "opsRequest": {"name":"camellia-qszmws-volumeexpansion-x2s6s","namespace":"ns-doqwtruiby"}}
2024-12-16T02:41:06.243Z	INFO	reconcile	{"controller": "opsrequest", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "OpsRequest", "OpsRequest": {"name":"camellia-qszmws-volumeexpansion-x2s6s","namespace":"ns-doqwtruiby"}, "namespace": "ns-doqwtruiby", "name": "camellia-qszmws-volumeexpansion-x2s6s", "reconcileID": "ceb5afc4-8011-462f-a1ae-376432418ae6", "opsRequest": {"name":"camellia-qszmws-volumeexpansion-x2s6s","namespace":"ns-doqwtruiby"}, "opsRequest": {"name":"camellia-qszmws-volumeexpansion-x2s6s","namespace":"ns-doqwtruiby"}}
2024-12-16T02:41:06.277Z	INFO	reconcile	{"controller": "opsrequest", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "OpsRequest", "OpsRequest": {"name":"camellia-qszmws-volumeexpansion-x2s6s","namespace":"ns-doqwtruiby"}, "namespace": "ns-doqwtruiby", "name": "camellia-qszmws-volumeexpansion-x2s6s", "reconcileID": "d409a0b4-01db-42c0-bde6-7e02b4095368", "opsRequest": {"name":"camellia-qszmws-volumeexpansion-x2s6s","namespace":"ns-doqwtruiby"}, "opsRequest": {"name":"camellia-qszmws-volumeexpansion-x2s6s","namespace":"ns-doqwtruiby"}}
2024-12-16T02:41:06.312Z	INFO	update cluster status after applying resources, generation: 6	{"controller": "cluster", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Cluster", "Cluster": {"name":"camellia-qszmws","namespace":"ns-doqwtruiby"}, "namespace": "ns-doqwtruiby", "name": "camellia-qszmws", "reconcileID": "32ad1297-9829-4549-8784-a8d7fffc814c", "cluster": {"name":"camellia-qszmws","namespace":"ns-doqwtruiby"}}
2024-12-16T02:41:06.331Z	INFO	reconcile	{"controller": "opsrequest", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "OpsRequest", "OpsRequest": {"name":"camellia-qszmws-volumeexpansion-x2s6s","namespace":"ns-doqwtruiby"}, "namespace": "ns-doqwtruiby", "name": "camellia-qszmws-volumeexpansion-x2s6s", "reconcileID": "88400017-837f-4704-ba58-4b77cf3a0b55", "opsRequest": {"name":"camellia-qszmws-volumeexpansion-x2s6s","namespace":"ns-doqwtruiby"}, "opsRequest": {"name":"camellia-qszmws-volumeexpansion-x2s6s","namespace":"ns-doqwtruiby"}}
2024-12-16T02:41:06.355Z	INFO	reconcile	{"controller": "opsrequest", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "OpsRequest", "OpsRequest": {"name":"camellia-qszmws-volumeexpansion-x2s6s","namespace":"ns-doqwtruiby"}, "namespace": "ns-doqwtruiby", "name": "camellia-qszmws-volumeexpansion-x2s6s", "reconcileID": "db34385d-709a-4c31-87f7-5d063ae22902", "opsRequest": {"name":"camellia-qszmws-volumeexpansion-x2s6s","namespace":"ns-doqwtruiby"}, "opsRequest": {"name":"camellia-qszmws-volumeexpansion-x2s6s","namespace":"ns-doqwtruiby"}}
2024-12-16T02:41:06.468Z	INFO	update status after applying new spec, generation: 4	{"controller": "component", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Component", "Component": {"name":"camellia-qszmws-proxy","namespace":"ns-doqwtruiby"}, "namespace": "ns-doqwtruiby", "name": "camellia-qszmws-proxy", "reconcileID": "10cd9406-5e9a-4922-b7e5-591b3ebbab17", "component": {"name":"camellia-qszmws-proxy","namespace":"ns-doqwtruiby"}}
2024-12-16T02:41:06.496Z	INFO	reconcile	{"controller": "opsrequest", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "OpsRequest", "OpsRequest": {"name":"camellia-qszmws-volumeexpansion-x2s6s","namespace":"ns-doqwtruiby"}, "namespace": "ns-doqwtruiby", "name": "camellia-qszmws-volumeexpansion-x2s6s", "reconcileID": "677c5415-188f-4e93-b63d-04f734f11984", "opsRequest": {"name":"camellia-qszmws-volumeexpansion-x2s6s","namespace":"ns-doqwtruiby"}, "opsRequest": {"name":"camellia-qszmws-volumeexpansion-x2s6s","namespace":"ns-doqwtruiby"}}
2024-12-16T02:41:06.504Z	INFO	reconcile	{"controller": "opsrequest", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "OpsRequest", "OpsRequest": {"name":"camellia-qszmws-volumeexpansion-x2s6s","namespace":"ns-doqwtruiby"}, "namespace": "ns-doqwtruiby", "name": "camellia-qszmws-volumeexpansion-x2s6s", "reconcileID": "057e9557-7ed5-40bf-a725-025e0872d205", "opsRequest": {"name":"camellia-qszmws-volumeexpansion-x2s6s","namespace":"ns-doqwtruiby"}, "opsRequest": {"name":"camellia-qszmws-volumeexpansion-x2s6s","namespace":"ns-doqwtruiby"}}
2024-12-16T02:41:06.539Z	INFO	reconcile	{"controller": "opsrequest", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "OpsRequest", "OpsRequest": {"name":"camellia-qszmws-volumeexpansion-x2s6s","namespace":"ns-doqwtruiby"}, "namespace": "ns-doqwtruiby", "name": "camellia-qszmws-volumeexpansion-x2s6s", "reconcileID": "ccb6c4ef-21e7-453e-8b95-60726e7c70d7", "opsRequest": {"name":"camellia-qszmws-volumeexpansion-x2s6s","namespace":"ns-doqwtruiby"}, "opsRequest": {"name":"camellia-qszmws-volumeexpansion-x2s6s","namespace":"ns-doqwtruiby"}}
2024-12-16T02:41:06.707Z	INFO	reconcile	{"controller": "opsrequest", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "OpsRequest", "OpsRequest": {"name":"camellia-qszmws-volumeexpansion-x2s6s","namespace":"ns-doqwtruiby"}, "namespace": "ns-doqwtruiby", "name": "camellia-qszmws-volumeexpansion-x2s6s", "reconcileID": "e9e1bf25-5c16-4e54-ad66-8d6b3e6ac28d", "opsRequest": {"name":"camellia-qszmws-volumeexpansion-x2s6s","namespace":"ns-doqwtruiby"}, "opsRequest": {"name":"camellia-qszmws-volumeexpansion-x2s6s","namespace":"ns-doqwtruiby"}}
2024-12-16T02:41:06.770Z	INFO	status conditions, creating: false, available: true, its running: true, has failure: false, updating: true, config synced: true	{"controller": "component", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Component", "Component": {"name":"camellia-qszmws-proxy","namespace":"ns-doqwtruiby"}, "namespace": "ns-doqwtruiby", "name": "camellia-qszmws-proxy", "reconcileID": "2615d6ad-ddb8-46b9-82bd-de37a618a1fa", "component": {"name":"camellia-qszmws-proxy","namespace":"ns-doqwtruiby"}}
2024-12-16T02:41:06.914Z	INFO	reconcile	{"controller": "opsrequest", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "OpsRequest", "OpsRequest": {"name":"strsent-tvqwed-restart-nfrdd","namespace":"ns-zvwzoswjnh"}, "namespace": "ns-zvwzoswjnh", "name": "strsent-tvqwed-restart-nfrdd", "reconcileID": "80473fda-6b80-4f8d-b04e-37b316b7ec59", "opsRequest": {"name":"strsent-tvqwed-restart-nfrdd","namespace":"ns-zvwzoswjnh"}, "opsRequest": {"name":"strsent-tvqwed-restart-nfrdd","namespace":"ns-zvwzoswjnh"}}
2024-12-16T02:41:06.914Z	INFO	reconcile	{"controller": "opsrequest", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "OpsRequest", "OpsRequest": {"name":"camellia-qszmws-volumeexpansion-x2s6s","namespace":"ns-doqwtruiby"}, "namespace": "ns-doqwtruiby", "name": "camellia-qszmws-volumeexpansion-x2s6s", "reconcileID": "93e35a93-3703-4a54-8cc4-100440eaa47d", "opsRequest": {"name":"camellia-qszmws-volumeexpansion-x2s6s","namespace":"ns-doqwtruiby"}, "opsRequest": {"name":"camellia-qszmws-volumeexpansion-x2s6s","namespace":"ns-doqwtruiby"}}
2024-12-16T02:41:07.060Z	INFO	cluster component phase transition: Running -> Updating (component is Updating)	{"controller": "cluster", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Cluster", "Cluster": {"name":"camellia-qszmws","namespace":"ns-doqwtruiby"}, "namespace": "ns-doqwtruiby", "name": "camellia-qszmws", "reconcileID": "32f4145a-f178-47d2-9a4e-37ee3de47d3d", "cluster": {"name":"camellia-qszmws","namespace":"ns-doqwtruiby"}}
2024-12-16T02:41:07.180Z	INFO	reconcile	{"controller": "opsrequest", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "OpsRequest", "OpsRequest": {"name":"strsent-tvqwed-restart-nfrdd","namespace":"ns-zvwzoswjnh"}, "namespace": "ns-zvwzoswjnh", "name": "strsent-tvqwed-restart-nfrdd", "reconcileID": "7802c9b4-b115-4431-b02f-3f880c37a94f", "opsRequest": {"name":"strsent-tvqwed-restart-nfrdd","namespace":"ns-zvwzoswjnh"}, "opsRequest": {"name":"strsent-tvqwed-restart-nfrdd","namespace":"ns-zvwzoswjnh"}}
2024-12-16T02:41:07.195Z	INFO	status conditions, creating: false, available: true, its running: true, has failure: false, updating: true, config synced: true	{"controller": "component", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Component", "Component": {"name":"camellia-qszmws-proxy","namespace":"ns-doqwtruiby"}, "namespace": "ns-doqwtruiby", "name": "camellia-qszmws-proxy", "reconcileID": "07a1f593-e94e-4132-b119-ca5cc9c949b6", "component": {"name":"camellia-qszmws-proxy","namespace":"ns-doqwtruiby"}}
2024-12-16T02:41:07.210Z	INFO	reconcile	{"controller": "opsrequest", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "OpsRequest", "OpsRequest": {"name":"camellia-qszmws-volumeexpansion-x2s6s","namespace":"ns-doqwtruiby"}, "namespace": "ns-doqwtruiby", "name": "camellia-qszmws-volumeexpansion-x2s6s", "reconcileID": "8ea571dd-55c4-49d0-98c1-118b1c507d79", "opsRequest": {"name":"camellia-qszmws-volumeexpansion-x2s6s","namespace":"ns-doqwtruiby"}, "opsRequest": {"name":"camellia-qszmws-volumeexpansion-x2s6s","namespace":"ns-doqwtruiby"}}
2024-12-16T02:41:07.459Z	INFO	reconcile	{"controller": "opsrequest", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "OpsRequest", "OpsRequest": {"name":"strsent-tvqwed-restart-nfrdd","namespace":"ns-zvwzoswjnh"}, "namespace": "ns-zvwzoswjnh", "name": "strsent-tvqwed-restart-nfrdd", "reconcileID": "23b5f4db-61d7-45c3-a8ba-357924c11eb5", "opsRequest": {"name":"strsent-tvqwed-restart-nfrdd","namespace":"ns-zvwzoswjnh"}, "opsRequest": {"name":"strsent-tvqwed-restart-nfrdd","namespace":"ns-zvwzoswjnh"}}
2024-12-16T02:41:07.464Z	INFO	reconcile	{"controller": "opsrequest", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "OpsRequest", "OpsRequest": {"name":"camellia-qszmws-volumeexpansion-x2s6s","namespace":"ns-doqwtruiby"}, "namespace": "ns-doqwtruiby", "name": "camellia-qszmws-volumeexpansion-x2s6s", "reconcileID": "9acaf61e-adee-442d-b375-f96cba251de6", "opsRequest": {"name":"camellia-qszmws-volumeexpansion-x2s6s","namespace":"ns-doqwtruiby"}, "opsRequest": {"name":"camellia-qszmws-volumeexpansion-x2s6s","namespace":"ns-doqwtruiby"}}
2024-12-16T02:41:07.707Z	INFO	reconcile	{"controller": "opsrequest", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "OpsRequest", "OpsRequest": {"name":"strsent-tvqwed-restart-nfrdd","namespace":"ns-zvwzoswjnh"}, "namespace": "ns-zvwzoswjnh", "name": "strsent-tvqwed-restart-nfrdd", "reconcileID": "5946a109-86ad-49f4-aaac-6411ac4262db", "opsRequest": {"name":"strsent-tvqwed-restart-nfrdd","namespace":"ns-zvwzoswjnh"}, "opsRequest": {"name":"strsent-tvqwed-restart-nfrdd","namespace":"ns-zvwzoswjnh"}}
2024-12-16T02:41:12.199Z	INFO	reconcile	{"controller": "opsrequest", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "OpsRequest", "OpsRequest": {"name":"orch-twuybv-horizontalscaling-8vdzj","namespace":"ns-rrzicitpck"}, "namespace": "ns-rrzicitpck", "name": "orch-twuybv-horizontalscaling-8vdzj", "reconcileID": "a27d3b08-651a-43f8-a496-51d3b04583d7", "opsRequest": {"name":"orch-twuybv-horizontalscaling-8vdzj","namespace":"ns-rrzicitpck"}, "opsRequest": {"name":"orch-twuybv-horizontalscaling-8vdzj","namespace":"ns-rrzicitpck"}}
2024-12-16T02:41:12.199Z	INFO	reconcile	{"controller": "opsrequest", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "OpsRequest", "OpsRequest": {"name":"orch-twuybv-restart-b7vm8","namespace":"ns-rrzicitpck"}, "namespace": "ns-rrzicitpck", "name": "orch-twuybv-restart-b7vm8", "reconcileID": "22840f55-c3fe-4f89-bb6f-7b8715f90c35", "opsRequest": {"name":"orch-twuybv-restart-b7vm8","namespace":"ns-rrzicitpck"}, "opsRequest": {"name":"orch-twuybv-restart-b7vm8","namespace":"ns-rrzicitpck"}}
2024-12-16T02:41:12.237Z	INFO	reconcile	{"controller": "opsrequest", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "OpsRequest", "OpsRequest": {"name":"orch-twuybv-restart-b7vm8","namespace":"ns-rrzicitpck"}, "namespace": "ns-rrzicitpck", "name": "orch-twuybv-restart-b7vm8", "reconcileID": "f340c5ce-71af-4947-b983-ba4b8195682f", "opsRequest": {"name":"orch-twuybv-restart-b7vm8","namespace":"ns-rrzicitpck"}, "opsRequest": {"name":"orch-twuybv-restart-b7vm8","namespace":"ns-rrzicitpck"}}
2024-12-16T02:41:12.259Z	INFO	reconcile	{"controller": "opsrequest", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "OpsRequest", "OpsRequest": {"name":"orch-twuybv-horizontalscaling-8vdzj","namespace":"ns-rrzicitpck"}, "namespace": "ns-rrzicitpck", "name": "orch-twuybv-horizontalscaling-8vdzj", "reconcileID": "994e222e-c91d-4b0c-a33b-ca90a514ef29", "opsRequest": {"name":"orch-twuybv-horizontalscaling-8vdzj","namespace":"ns-rrzicitpck"}, "opsRequest": {"name":"orch-twuybv-horizontalscaling-8vdzj","namespace":"ns-rrzicitpck"}}
2024-12-16T02:41:12.269Z	INFO	reconcile	{"controller": "opsrequest", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "OpsRequest", "OpsRequest": {"name":"orch-twuybv-horizontalscaling-8vdzj","namespace":"ns-rrzicitpck"}, "namespace": "ns-rrzicitpck", "name": "orch-twuybv-horizontalscaling-8vdzj", "reconcileID": "91e331fc-ed7c-4121-ad65-9aa1ce8f07c5", "opsRequest": {"name":"orch-twuybv-horizontalscaling-8vdzj","namespace":"ns-rrzicitpck"}, "opsRequest": {"name":"orch-twuybv-horizontalscaling-8vdzj","namespace":"ns-rrzicitpck"}}
2024-12-16T02:41:12.328Z	INFO	status conditions, creating: false, available: false, its running: false, has failure: false, updating: false, config synced: true	{"controller": "component", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Component", "Component": {"name":"orch-twuybv-orchestrator","namespace":"ns-rrzicitpck"}, "namespace": "ns-rrzicitpck", "name": "orch-twuybv-orchestrator", "reconcileID": "1e1b36ea-3b0c-48da-8fe8-ad0983c0250d", "component": {"name":"orch-twuybv-orchestrator","namespace":"ns-rrzicitpck"}}
2024-12-16T02:41:12.512Z	INFO	status conditions, creating: false, available: false, its running: false, has failure: false, updating: false, config synced: true	{"controller": "component", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Component", "Component": {"name":"orch-twuybv-orchestrator","namespace":"ns-rrzicitpck"}, "namespace": "ns-rrzicitpck", "name": "orch-twuybv-orchestrator", "reconcileID": "40bee6dd-d3dc-422c-83e8-7fce7c02dcec", "component": {"name":"orch-twuybv-orchestrator","namespace":"ns-rrzicitpck"}}
2024-12-16T02:41:12.647Z	INFO	status conditions, creating: false, available: true, its running: false, has failure: true, updating: true, config synced: true	{"controller": "component", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Component", "Component": {"name":"apepg-dhzoqi-postgresql","namespace":"ns-smhezznipw"}, "namespace": "ns-smhezznipw", "name": "apepg-dhzoqi-postgresql", "reconcileID": "08385fd5-2550-4dba-932c-f5082a1af6a3", "component": {"name":"apepg-dhzoqi-postgresql","namespace":"ns-smhezznipw"}}
2024-12-16T02:41:12.853Z	INFO	status conditions, creating: false, available: true, its running: false, has failure: true, updating: true, config synced: true	{"controller": "component", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Component", "Component": {"name":"apepg-dhzoqi-postgresql","namespace":"ns-smhezznipw"}, "namespace": "ns-smhezznipw", "name": "apepg-dhzoqi-postgresql", "reconcileID": "06ad6496-0262-4df0-b310-011cf7c948f1", "component": {"name":"apepg-dhzoqi-postgresql","namespace":"ns-smhezznipw"}}
2024-12-16T02:41:13.469Z	INFO	status conditions, creating: false, available: true, its running: false, has failure: true, updating: true, config synced: true	{"controller": "component", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Component", "Component": {"name":"apepg-dhzoqi-postgresql","namespace":"ns-smhezznipw"}, "namespace": "ns-smhezznipw", "name": "apepg-dhzoqi-postgresql", "reconcileID": "a4a78849-f03d-4393-beaa-d16254de8c69", "component": {"name":"apepg-dhzoqi-postgresql","namespace":"ns-smhezznipw"}}
2024-12-16T02:41:13.752Z	INFO	status conditions, creating: false, available: true, its running: true, has failure: false, updating: true, config synced: true	{"controller": "component", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Component", "Component": {"name":"apepg-dhzoqi-postgresql","namespace":"ns-smhezznipw"}, "namespace": "ns-smhezznipw", "name": "apepg-dhzoqi-postgresql", "reconcileID": "cb453bc8-f395-49c5-a557-6ebdbae5cf47", "component": {"name":"apepg-dhzoqi-postgresql","namespace":"ns-smhezznipw"}}
2024-12-16T02:41:13.784Z	INFO	cluster component phase transition: Abnormal -> Updating (component is Updating)	{"controller": "cluster", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Cluster", "Cluster": {"name":"apepg-dhzoqi","namespace":"ns-smhezznipw"}, "namespace": "ns-smhezznipw", "name": "apepg-dhzoqi", "reconcileID": "2ddfff84-905e-44ba-b55b-0c691d1024be", "cluster": {"name":"apepg-dhzoqi","namespace":"ns-smhezznipw"}}
2024-12-16T02:41:13.926Z	INFO	status conditions, creating: false, available: true, its running: true, has failure: false, updating: true, config synced: true	{"controller": "component", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Component", "Component": {"name":"apepg-dhzoqi-postgresql","namespace":"ns-smhezznipw"}, "namespace": "ns-smhezznipw", "name": "apepg-dhzoqi-postgresql", "reconcileID": "dea41ee7-2f3f-4a7e-b842-69c43b581873", "component": {"name":"apepg-dhzoqi-postgresql","namespace":"ns-smhezznipw"}}
2024-12-16T02:41:14.980Z	INFO	reconcile	{"controller": "opsrequest", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "OpsRequest", "OpsRequest": {"name":"strsent-tvqwed-horizontalscaling-xzjh5","namespace":"ns-zvwzoswjnh"}, "namespace": "ns-zvwzoswjnh", "name": "strsent-tvqwed-horizontalscaling-xzjh5", "reconcileID": "ddf922d0-51fa-4752-bb4b-3fcebc140226", "opsRequest": {"name":"strsent-tvqwed-horizontalscaling-xzjh5","namespace":"ns-zvwzoswjnh"}, "opsRequest": {"name":"strsent-tvqwed-horizontalscaling-xzjh5","namespace":"ns-zvwzoswjnh"}}
2024-12-16T02:41:15.007Z	INFO	reconcile	{"controller": "opsrequest", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "OpsRequest", "OpsRequest": {"name":"strsent-tvqwed-horizontalscaling-xzjh5","namespace":"ns-zvwzoswjnh"}, "namespace": "ns-zvwzoswjnh", "name": "strsent-tvqwed-horizontalscaling-xzjh5", "reconcileID": "e7d0d961-703d-47c8-8377-0d391565deb7", "opsRequest": {"name":"strsent-tvqwed-horizontalscaling-xzjh5","namespace":"ns-zvwzoswjnh"}, "opsRequest": {"name":"strsent-tvqwed-horizontalscaling-xzjh5","namespace":"ns-zvwzoswjnh"}}
2024-12-16T02:41:15.022Z	INFO	reconcile	{"controller": "opsrequest", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "OpsRequest", "OpsRequest": {"name":"strsent-tvqwed-horizontalscaling-xzjh5","namespace":"ns-zvwzoswjnh"}, "namespace": "ns-zvwzoswjnh", "name": "strsent-tvqwed-horizontalscaling-xzjh5", "reconcileID": "5703efe9-450a-4a49-83ee-bdbd9bad24c0", "opsRequest": {"name":"strsent-tvqwed-horizontalscaling-xzjh5","namespace":"ns-zvwzoswjnh"}, "opsRequest": {"name":"strsent-tvqwed-horizontalscaling-xzjh5","namespace":"ns-zvwzoswjnh"}}
2024-12-16T02:41:15.065Z	INFO	reconcile	{"controller": "opsrequest", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "OpsRequest", "OpsRequest": {"name":"strsent-tvqwed-horizontalscaling-xzjh5","namespace":"ns-zvwzoswjnh"}, "namespace": "ns-zvwzoswjnh", "name": "strsent-tvqwed-horizontalscaling-xzjh5", "reconcileID": "75d37cc2-f7c0-4ecb-b851-547af2464f1a", "opsRequest": {"name":"strsent-tvqwed-horizontalscaling-xzjh5","namespace":"ns-zvwzoswjnh"}, "opsRequest": {"name":"strsent-tvqwed-horizontalscaling-xzjh5","namespace":"ns-zvwzoswjnh"}}
2024-12-16T02:41:15.099Z	INFO	reconcile	{"controller": "opsrequest", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "OpsRequest", "OpsRequest": {"name":"strsent-tvqwed-horizontalscaling-xzjh5","namespace":"ns-zvwzoswjnh"}, "namespace": "ns-zvwzoswjnh", "name": "strsent-tvqwed-horizontalscaling-xzjh5", "reconcileID": "3b0a283b-1f12-4253-8e5c-091e0996664d", "opsRequest": {"name":"strsent-tvqwed-horizontalscaling-xzjh5","namespace":"ns-zvwzoswjnh"}, "opsRequest": {"name":"strsent-tvqwed-horizontalscaling-xzjh5","namespace":"ns-zvwzoswjnh"}}
2024-12-16T02:41:15.137Z	INFO	update cluster status after applying resources, generation: 6	{"controller": "cluster", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Cluster", "Cluster": {"name":"strsent-tvqwed","namespace":"ns-zvwzoswjnh"}, "namespace": "ns-zvwzoswjnh", "name": "strsent-tvqwed", "reconcileID": "71a3c68d-0b2b-40ce-bcde-1ff5b663cda4", "cluster": {"name":"strsent-tvqwed","namespace":"ns-zvwzoswjnh"}}
2024-12-16T02:41:15.145Z	INFO	reconcile	{"controller": "opsrequest", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "OpsRequest", "OpsRequest": {"name":"strsent-tvqwed-horizontalscaling-xzjh5","namespace":"ns-zvwzoswjnh"}, "namespace": "ns-zvwzoswjnh", "name": "strsent-tvqwed-horizontalscaling-xzjh5", "reconcileID": "85bdfdff-dc5d-4634-8df8-20e9056adc9b", "opsRequest": {"name":"strsent-tvqwed-horizontalscaling-xzjh5","namespace":"ns-zvwzoswjnh"}, "opsRequest": {"name":"strsent-tvqwed-horizontalscaling-xzjh5","namespace":"ns-zvwzoswjnh"}}
2024-12-16T02:41:15.167Z	INFO	reconcile	{"controller": "opsrequest", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "OpsRequest", "OpsRequest": {"name":"strsent-tvqwed-horizontalscaling-xzjh5","namespace":"ns-zvwzoswjnh"}, "namespace": "ns-zvwzoswjnh", "name": "strsent-tvqwed-horizontalscaling-xzjh5", "reconcileID": "abdcb03f-231c-4843-a2cf-710f8820ad0a", "opsRequest": {"name":"strsent-tvqwed-horizontalscaling-xzjh5","namespace":"ns-zvwzoswjnh"}, "opsRequest": {"name":"strsent-tvqwed-horizontalscaling-xzjh5","namespace":"ns-zvwzoswjnh"}}
2024-12-16T02:41:15.176Z	INFO	reconcile	{"controller": "opsrequest", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "OpsRequest", "OpsRequest": {"name":"strsent-tvqwed-horizontalscaling-xzjh5","namespace":"ns-zvwzoswjnh"}, "namespace": "ns-zvwzoswjnh", "name": "strsent-tvqwed-horizontalscaling-xzjh5", "reconcileID": "47fc51a7-3d43-4cbf-b4bc-b8a1c61ed708", "opsRequest": {"name":"strsent-tvqwed-horizontalscaling-xzjh5","namespace":"ns-zvwzoswjnh"}, "opsRequest": {"name":"strsent-tvqwed-horizontalscaling-xzjh5","namespace":"ns-zvwzoswjnh"}}
2024-12-16T02:41:15.189Z	INFO	reconcile	{"controller": "opsrequest", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "OpsRequest", "OpsRequest": {"name":"strsent-tvqwed-horizontalscaling-xzjh5","namespace":"ns-zvwzoswjnh"}, "namespace": "ns-zvwzoswjnh", "name": "strsent-tvqwed-horizontalscaling-xzjh5", "reconcileID": "797e7c8c-0b19-429f-acd8-9ecc47db4a3a", "opsRequest": {"name":"strsent-tvqwed-horizontalscaling-xzjh5","namespace":"ns-zvwzoswjnh"}, "opsRequest": {"name":"strsent-tvqwed-horizontalscaling-xzjh5","namespace":"ns-zvwzoswjnh"}}
2024-12-16T02:41:15.323Z	INFO	Observed a panic in reconciler: runtime error: invalid memory address or nil pointer dereference	{"controller": "component", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Component", "Component": {"name":"strsent-tvqwed-be","namespace":"ns-zvwzoswjnh"}, "namespace": "ns-zvwzoswjnh", "name": "strsent-tvqwed-be", "reconcileID": "57caf399-04c7-46c1-b93a-7a619fb90f86"}
panic: runtime error: invalid memory address or nil pointer dereference [recovered]
	panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x20 pc=0x2515305]

goroutine 917 [running]:
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile.func1()
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:116 +0x1e5
panic({0x288b3c0?, 0x4b8db50?})
	/usr/local/go/src/runtime/panic.go:914 +0x21f
github.com/apecloud/kubeblocks/controllers/apps.(*componentWorkloadOps).annotateInstanceSetForMemberJoin(0x2ced05b?)
	/src/controllers/apps/transformer_component_workload.go:581 +0x25
github.com/apecloud/kubeblocks/controllers/apps.(*componentWorkloadOps).scaleOut(0xc0076bc2a0, 0xc00917e600)
	/src/controllers/apps/transformer_component_workload.go:532 +0xf2
github.com/apecloud/kubeblocks/controllers/apps.(*componentWorkloadOps).horizontalScale(0xc0076bc2a0)
	/src/controllers/apps/transformer_component_workload.go:437 +0x1b2
github.com/apecloud/kubeblocks/controllers/apps.(*componentWorkloadTransformer).handleWorkloadUpdate(0x0?, {{0x3347828, 0xc00ac37590}, {{{0x0, 0x0}, {0x0, 0x0}}}, {{0x334d928, 0xc00ac372f0}, 0x0}, ...}, ...)
	/src/controllers/apps/transformer_component_workload.go:206 +0x92
github.com/apecloud/kubeblocks/controllers/apps.(*componentWorkloadTransformer).handleUpdate(0xc00acde030, {{0x3347828, 0xc00ac37590}, {{{0x0, 0x0}, {0x0, 0x0}}}, {{0x334d928, 0xc00ac372f0}, 0x0}, ...}, ...)
	/src/controllers/apps/transformer_component_workload.go:177 +0xac
github.com/apecloud/kubeblocks/controllers/apps.(*componentWorkloadTransformer).Transform(0xc00acde020?, {0x3347470?, 0xc00a399ef0}, 0xc00acde040?)
	/src/controllers/apps/transformer_component_workload.go:122 +0x2e9
github.com/apecloud/kubeblocks/pkg/controller/graph.TransformerChain.ApplyTo({0xc008335340?, 0x16, 0x4c131e0?}, {0x3347470, 0xc00a399ef0}, 0x417852?)
	/src/pkg/controller/graph/transformer.go:57 +0x9b
github.com/apecloud/kubeblocks/controllers/apps.(*componentPlanBuilder).Build(0xc006e32370)
	/src/controllers/apps/component_plan_builder.go:117 +0xb6
github.com/apecloud/kubeblocks/controllers/apps.(*ComponentReconciler).Reconcile(0xc000bf0b70, {0x3347828, 0xc00ac37290}, {{{0xc008dd61e0, 0xd}, {0xc002cdbec0, 0x11}}})
	/src/controllers/apps/component_controller.go:188 +0xa0e
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile(0x334d928?, {0x3347828?, 0xc00ac37290?}, {{{0xc008dd61e0?, 0xb?}, {0xc002cdbec0?, 0x0?}}})
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:119 +0xb7
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler(0xc000c9b4a0, {0x3347860, 0xc000cb89b0}, {0x29ca3a0?, 0xc0082bfe60?})
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:316 +0x3cc
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem(0xc000c9b4a0, {0x3347860, 0xc000cb89b0})
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:266 +0x1af
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2()
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:227 +0x79
created by sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2 in goroutine 101
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:223 +0x565

Expected behavior
A clear and concise description of what you expected to happen.

Screenshots
If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information):

  • OS: [e.g. iOS]
  • Browser [e.g. chrome, safari]
  • Version [e.g. 22]

Additional context
Add any other context about the problem here.
@JashBook JashBook added the kind/bug Something isn't working label Dec 16, 2024
@JashBook JashBook changed the title [BUG] kubeblocks CrashLoopBackOff [BUG] kubeblocks CrashLoopBackOff Observed a panic in reconciler starrocks ent be ops Dec 16, 2024
@JashBook JashBook changed the title [BUG] kubeblocks CrashLoopBackOff Observed a panic in reconciler starrocks ent be ops [BUG] kubeblocks CrashLoopBackOff Observed a panic in reconciler starrocks ent be scaleOut ops Dec 16, 2024
@JashBook
Copy link
Collaborator Author

kubeblocks CrashLoopBackOff Observed a panic in reconciler tidb scaleOut ops

kubectl get pod -n kb-afmdycuamd   kubeblocks-6d7dc688c7-6k7t2 
NAME                          READY   STATUS             RESTARTS      AGE
kubeblocks-6d7dc688c7-6k7t2   0/1     CrashLoopBackOff   1 (13s ago)   118s

logs kubeblocks pod

kubectl logs -n kb-afmdycuamd   kubeblocks-6d7dc688c7-6k7t2  --previous

2024-12-16T02:47:27.192Z	ERROR	failed to get config template cm object!	{"controller": "componentdefinition", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "ComponentDefinition", "ComponentDefinition": {"name":"loki-read"}, "namespace": "", "name": "loki-read", "reconcileID": "ba0739ed-67a2-41f5-b089-49e627d146f3", "component": {"name":"loki-read"}, "configMapName": "loki", "error": "configmaps \"loki\" not found"}
github.com/apecloud/kubeblocks/controllers/apps/configuration.getConfigMapByTemplateName
	/src/controllers/apps/configuration/config_util.go:79
github.com/apecloud/kubeblocks/controllers/apps/configuration.validateConfigTemplate.func1
	/src/controllers/apps/configuration/config_util.go:349
github.com/apecloud/kubeblocks/controllers/apps/configuration.validateConfigTemplate
	/src/controllers/apps/configuration/config_util.go:373
github.com/apecloud/kubeblocks/controllers/apps/configuration.checkConfigTemplate.func1
	/src/controllers/apps/configuration/config_util.go:332
github.com/apecloud/kubeblocks/controllers/apps/configuration.handleConfigTemplate
	/src/controllers/apps/configuration/config_util.go:277
github.com/apecloud/kubeblocks/controllers/apps/configuration.checkConfigTemplate
	/src/controllers/apps/configuration/config_util.go:334
github.com/apecloud/kubeblocks/controllers/apps/configuration.ReconcileConfigSpecsForReferencedCR[...]
	/src/controllers/apps/configuration/config_util.go:106
github.com/apecloud/kubeblocks/controllers/apps.(*ComponentDefinitionReconciler).validateConfigs
	/src/controllers/apps/componentdefinition_controller.go:340
github.com/apecloud/kubeblocks/controllers/apps.(*ComponentDefinitionReconciler).validate
	/src/controllers/apps/componentdefinition_controller.go:201
github.com/apecloud/kubeblocks/controllers/apps.(*ComponentDefinitionReconciler).reconcile
	/src/controllers/apps/componentdefinition_controller.go:107
github.com/apecloud/kubeblocks/controllers/apps.(*ComponentDefinitionReconciler).Reconcile
	/src/controllers/apps/componentdefinition_controller.go:85
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:119
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:316
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:266
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:227
2024-12-16T02:47:27.192Z	ERROR	failed to get config template cm object!	{"controller": "componentdefinition", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "ComponentDefinition", "ComponentDefinition": {"name":"loki-read"}, "namespace": "", "name": "loki-read", "reconcileID": "ba0739ed-67a2-41f5-b089-49e627d146f3", "component": {"name":"loki-read"}, "templateName": "loki-config", "configMapName": "loki", "error": "configmaps \"loki\" not found"}
github.com/apecloud/kubeblocks/controllers/apps/configuration.validateConfigTemplate.func1
	/src/controllers/apps/configuration/config_util.go:350
github.com/apecloud/kubeblocks/controllers/apps/configuration.validateConfigTemplate
	/src/controllers/apps/configuration/config_util.go:373
github.com/apecloud/kubeblocks/controllers/apps/configuration.checkConfigTemplate.func1
	/src/controllers/apps/configuration/config_util.go:332
github.com/apecloud/kubeblocks/controllers/apps/configuration.handleConfigTemplate
	/src/controllers/apps/configuration/config_util.go:277
github.com/apecloud/kubeblocks/controllers/apps/configuration.checkConfigTemplate
	/src/controllers/apps/configuration/config_util.go:334
github.com/apecloud/kubeblocks/controllers/apps/configuration.ReconcileConfigSpecsForReferencedCR[...]
	/src/controllers/apps/configuration/config_util.go:106
github.com/apecloud/kubeblocks/controllers/apps.(*ComponentDefinitionReconciler).validateConfigs
	/src/controllers/apps/componentdefinition_controller.go:340
github.com/apecloud/kubeblocks/controllers/apps.(*ComponentDefinitionReconciler).validate
	/src/controllers/apps/componentdefinition_controller.go:201
github.com/apecloud/kubeblocks/controllers/apps.(*ComponentDefinitionReconciler).reconcile
	/src/controllers/apps/componentdefinition_controller.go:107
github.com/apecloud/kubeblocks/controllers/apps.(*ComponentDefinitionReconciler).Reconcile
	/src/controllers/apps/componentdefinition_controller.go:85
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:119
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:316
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:266
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:227
2024-12-16T02:47:27.192Z	ERROR	failed to validate config template!	{"controller": "componentdefinition", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "ComponentDefinition", "ComponentDefinition": {"name":"loki-read"}, "namespace": "", "name": "loki-read", "reconcileID": "ba0739ed-67a2-41f5-b089-49e627d146f3", "component": {"name":"loki-read"}, "templateName": "loki-config", "configMapName": "loki", "error": "configmaps \"loki\" not found"}
github.com/apecloud/kubeblocks/controllers/apps/configuration.validateConfigTemplate
	/src/controllers/apps/configuration/config_util.go:375
github.com/apecloud/kubeblocks/controllers/apps/configuration.checkConfigTemplate.func1
	/src/controllers/apps/configuration/config_util.go:332
github.com/apecloud/kubeblocks/controllers/apps/configuration.handleConfigTemplate
	/src/controllers/apps/configuration/config_util.go:277
github.com/apecloud/kubeblocks/controllers/apps/configuration.checkConfigTemplate
	/src/controllers/apps/configuration/config_util.go:334
github.com/apecloud/kubeblocks/controllers/apps/configuration.ReconcileConfigSpecsForReferencedCR[...]
	/src/controllers/apps/configuration/config_util.go:106
github.com/apecloud/kubeblocks/controllers/apps.(*ComponentDefinitionReconciler).validateConfigs
	/src/controllers/apps/componentdefinition_controller.go:340
github.com/apecloud/kubeblocks/controllers/apps.(*ComponentDefinitionReconciler).validate
	/src/controllers/apps/componentdefinition_controller.go:201
github.com/apecloud/kubeblocks/controllers/apps.(*ComponentDefinitionReconciler).reconcile
	/src/controllers/apps/componentdefinition_controller.go:107
github.com/apecloud/kubeblocks/controllers/apps.(*ComponentDefinitionReconciler).Reconcile
	/src/controllers/apps/componentdefinition_controller.go:85
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:119
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:316
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:266
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:227
2024-12-16T02:47:27.208Z	INFO	failed to check config template: configmaps "loki" not found	{"controller": "componentdefinition", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "ComponentDefinition", "ComponentDefinition": {"name":"loki-read"}, "namespace": "", "name": "loki-read", "reconcileID": "ba0739ed-67a2-41f5-b089-49e627d146f3", "component": {"name":"loki-read"}}
2024-12-16T02:47:27.208Z	ERROR	Reconciler error	{"controller": "componentdefinition", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "ComponentDefinition", "ComponentDefinition": {"name":"loki-read"}, "namespace": "", "name": "loki-read", "reconcileID": "ba0739ed-67a2-41f5-b089-49e627d146f3", "error": "failed to check config template: configmaps \"loki\" not found"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:329
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:266
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:227
2024-12-16T02:47:27.256Z	INFO	reconcile	{"controller": "opsrequest", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "OpsRequest", "OpsRequest": {"name":"solr-ozvcqt-verticalscaling-4kpjx","namespace":"ns-fcrehgwceq"}, "namespace": "ns-fcrehgwceq", "name": "solr-ozvcqt-verticalscaling-4kpjx", "reconcileID": "032a9292-380a-437a-a633-d876bf3e5e9e", "opsRequest": {"name":"solr-ozvcqt-verticalscaling-4kpjx","namespace":"ns-fcrehgwceq"}, "opsRequest": {"name":"solr-ozvcqt-verticalscaling-4kpjx","namespace":"ns-fcrehgwceq"}}
2024-12-16T02:47:27.473Z	INFO	Observed a panic in reconciler: runtime error: invalid memory address or nil pointer dereference	{"controller": "component", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Component", "Component": {"name":"tidb-qhtkrl-tidb","namespace":"ns-ctwdmtmstr"}, "namespace": "ns-ctwdmtmstr", "name": "tidb-qhtkrl-tidb", "reconcileID": "ba31c42d-084a-42e6-ab83-dad03239b882"}
panic: runtime error: invalid memory address or nil pointer dereference [recovered]
	panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x20 pc=0x2515305]

goroutine 904 [running]:
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile.func1()
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:116 +0x1e5
panic({0x288b3c0?, 0x4b8db50?})
	/usr/local/go/src/runtime/panic.go:914 +0x21f
github.com/apecloud/kubeblocks/controllers/apps.(*componentWorkloadOps).annotateInstanceSetForMemberJoin(0x2ced05b?)
	/src/controllers/apps/transformer_component_workload.go:581 +0x25
github.com/apecloud/kubeblocks/controllers/apps.(*componentWorkloadOps).scaleOut(0xc004f04000, 0xc0010e9800)
	/src/controllers/apps/transformer_component_workload.go:532 +0xf2
github.com/apecloud/kubeblocks/controllers/apps.(*componentWorkloadOps).horizontalScale(0xc004f04000)
	/src/controllers/apps/transformer_component_workload.go:437 +0x1b2
github.com/apecloud/kubeblocks/controllers/apps.(*componentWorkloadTransformer).handleWorkloadUpdate(0x0?, {{0x3347828, 0xc004f585a0}, {{{0x0, 0x0}, {0x0, 0x0}}}, {{0x334d928, 0xc004f58330}, 0x0}, ...}, ...)
	/src/controllers/apps/transformer_component_workload.go:206 +0x92
github.com/apecloud/kubeblocks/controllers/apps.(*componentWorkloadTransformer).handleUpdate(0xc004f01bf0, {{0x3347828, 0xc004f585a0}, {{{0x0, 0x0}, {0x0, 0x0}}}, {{0x334d928, 0xc004f58330}, 0x0}, ...}, ...)
	/src/controllers/apps/transformer_component_workload.go:177 +0xac
github.com/apecloud/kubeblocks/controllers/apps.(*componentWorkloadTransformer).Transform(0xc004f01be0?, {0x3347470?, 0xc001ab9e60}, 0xc004f01c00?)
	/src/controllers/apps/transformer_component_workload.go:122 +0x2e9
github.com/apecloud/kubeblocks/pkg/controller/graph.TransformerChain.ApplyTo({0xc004e5de40?, 0x16, 0x4?}, {0x3347470, 0xc001ab9e60}, 0x417852?)
	/src/pkg/controller/graph/transformer.go:57 +0x9b
github.com/apecloud/kubeblocks/controllers/apps.(*componentPlanBuilder).Build(0xc004f54190)
	/src/controllers/apps/component_plan_builder.go:117 +0xb6
github.com/apecloud/kubeblocks/controllers/apps.(*ComponentReconciler).Reconcile(0xc000c90360, {0x3347828, 0xc004f582d0}, {{{0xc0005e16b0, 0xd}, {0xc000f9daf0, 0x10}}})
	/src/controllers/apps/component_controller.go:188 +0xa0e
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile(0x334d928?, {0x3347828?, 0xc004f582d0?}, {{{0xc0005e16b0?, 0xb?}, {0xc000f9daf0?, 0x0?}}})
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:119 +0xb7
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler(0xc000aaea00, {0x3347860, 0xc000a7fae0}, {0x29ca3a0?, 0xc001079300?})
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:316 +0x3cc
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem(0xc000aaea00, {0x3347860, 0xc000a7fae0})
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:266 +0x1af
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2()
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:227 +0x79
created by sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2 in goroutine 100
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:223 +0x565

@JashBook JashBook changed the title [BUG] kubeblocks CrashLoopBackOff Observed a panic in reconciler starrocks ent be scaleOut ops [BUG] kubeblocks CrashLoopBackOff Observed a panic in reconciler cluster scaleOut ops Dec 16, 2024
@JashBook JashBook added this to the Release 0.9.2 milestone Dec 16, 2024
@kubeJocker kubeJocker mentioned this issue Dec 16, 2024
Merged
@kubeJocker kubeJocker linked a pull request Dec 16, 2024 that will close this issue
fix: nilptr for memberjoin #8659
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@kubeJocker kubeJocker

Labels
kind/bug Something isn't working
Projects
None yet
Milestone
Release 0.9.2
Development

Successfully merging a pull request may close this issue.

fix: nilptr for memberjoin apecloud/kubeblocks
4 participants
@iziang @kubeJocker @JashBook @shanshanying