Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG]: CSM Operator Crashing #1603

Open
anurajpd opened this issue Nov 22, 2024 · 5 comments
Open

[BUG]: CSM Operator Crashing #1603

anurajpd opened this issue Nov 22, 2024 · 5 comments
Assignees
Labels
area/csm-operator type/bug Something isn't working. This is the default label associated with a bug issue.
Milestone

Comments

@anurajpd
Copy link

anurajpd commented Nov 22, 2024

Bug Description

CSM Operator Crashing

Logs

manifest file used to create csm

apiVersion: storage.dell.com/v1
kind: ContainerStorageModule
metadata:
name: powerflex
namespace: powerflex
spec:
driver:
csiDriverType: "powerflex"
configVersion: v2.12.0
replicas: 2
sideCars:
- name: provisioner
args: ["--volume-name-prefix=ocp03"]
initContainers:
- name: sdc
image: dellemc/sdc:4.5.2.1
node:
envs:
- name: X_CSI_RENAME_SDC_ENABLED
value: "true"
- name: X_CSI_RENAME_SDC_PREFIX
value: "ocp03"


[anuraj@adm01 pf06]$ oc apply -f csm-powerflex.yaml
containerstoragemodule.storage.dell.com/powerflex created
[anuraj@adm01 pf06]$ oc get csm
\NAME CREATIONTIME CSIDRIVERTYPE CONFIGVERSION STATE
powerflex 3s powerflex v2.12.0
[anuraj@adm01 pf06]$ oc get csm
NAME CREATIONTIME CSIDRIVERTYPE CONFIGVERSION STATE
powerflex 5s powerflex v2.12.0
[anuraj@adm01 pf06]$ oc get csm
NAME CREATIONTIME CSIDRIVERTYPE CONFIGVERSION STATE
powerflex 7s powerflex v2.12.0
[anuraj@adm01 pf06]$ oc get csm
NAME CREATIONTIME CSIDRIVERTYPE CONFIGVERSION STATE
powerflex 8s powerflex v2.12.0
[anuraj@adm01 pf06]$ oc get pod
No resources found in powerflex namespace.
[anuraj@adm01 pf06]$ oc get pod -n dell-csm-operator
NAME READY STATUS RESTARTS AGE
dell-csm-operator-controller-manager-7c5b4c7f84-mxhbp 1/1 Running 2 (18s ago) 15h
[anuraj@adm01 pf06]$ oc get pod -n dell-csm-operator -w
NAME READY STATUS RESTARTS AGE
dell-csm-operator-controller-manager-7c5b4c7f84-mxhbp 0/1 Error 2 (25s ago) 15h
dell-csm-operator-controller-manager-7c5b4c7f84-mxhbp 0/1 CrashLoopBackOff 2 (11s ago) 15h
dell-csm-operator-controller-manager-7c5b4c7f84-mxhbp 0/1 Running 3 (12s ago) 15h
dell-csm-operator-controller-manager-7c5b4c7f84-mxhbp 1/1 Running 3 (21s ago) 15h
dell-csm-operator-controller-manager-7c5b4c7f84-mxhbp 0/1 Error 3 (28s ago) 15h
dell-csm-operator-controller-manager-7c5b4c7f84-mxhbp 0/1 CrashLoopBackOff 3 (3s ago) 15h
dell-csm-operator-controller-manager-7c5b4c7f84-mxhbp 0/1 Running 4 (31s ago) 15h
dell-csm-operator-controller-manager-7c5b4c7f84-mxhbp 1/1 Running 4 (43s ago) 15h


2024-11-22T10:42:02Z INFO Starting workers {"controller": "containerstoragemodule", "controllerGroup": "storage.dell.com", "contr
ollerKind": "ContainerStorageModule", "worker count": 1}
2024-11-22T10:42:02.547Z INFO controllers/csm_controller.go:251 ################Starting Reconcile############## {"Trac
eId": "powerflex-1"}
2024-11-22T10:42:02.547Z INFO controllers/csm_controller.go:254 reconcile for {"TraceId": "powerflex-1", "Namespace": "power
flex", "Name": "powerflex", "Attempt": 1}
2024-11-22T10:42:02Z INFO Observed a panic in reconciler: runtime error: index out of range [0] with length 0 {"controller": "contai
nerstoragemodule", "controllerGroup": "storage.dell.com", "controllerKind": "ContainerStorageModule", "ContainerStorageModule": {"name":"power
flex","namespace":"powerflex"}, "namespace": "powerflex", "name": "powerflex", "reconcileID": "bfaf7401-5490-4dc5-ad30-4fb09a108679"}
panic: runtime error: index out of range [0] with length 0 [recovered]
panic: runtime error: index out of range [0] with length 0

goroutine 203 [running]:
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile.func1()
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.5/pkg/internal/controller/controller.go:111 +0x1da
panic({0x1c0f480?, 0xc00197c360?})
/usr/local/go/src/runtime/panic.go:785 +0x132
github.com/dell/csm-operator/pkg/drivers.PrecheckPowerFlex({0x200e838, 0xc001494360}, 0xc0014b2008, {0x1, {{0x0, 0x0}, {{0xc00005e450, 0x2f},
{0xc0002c0840, 0x32}, ...}}, ...}, ...)
/workspace/pkg/drivers/powerflex.go:100 +0xea6
github.com/dell/csm-operator/controllers.(*ContainerStorageModuleReconciler).PreChecks(0xc0001787e0, {0x200e838, 0xc001494360}, 0xc0014b2008,
{0x1, {{0x0, 0x0}, {{0xc00005e450, 0x2f}, {0xc0002c0840, ...}, ...}}, ...})
/workspace/controllers/csm_controller.go:1325 +0x3ae
github.com/dell/csm-operator/controllers.(*ContainerStorageModuleReconciler).Reconcile(0xc0001787e0, {0x0?, 0x0?}, {{{0xc00105dd70?, 0x5?}, {0
xc00105dd20?, 0xc001445d50?}}})
/workspace/controllers/csm_controller.go:282 +0x4b8
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile(0x2012b68?, {0x200e838?, 0xc0014942d0?}, {{{0xc00105dd70?, 0xb?
}, {0xc00105dd20?, 0x0?}}})
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.5/pkg/internal/controller/controller.go:114 +0xa5
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler(0xc0002360b0, {0x200e870, 0xc00045d9f0}, {0x1b62320, 0xc
00149e000})
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.5/pkg/internal/controller/controller.go:311 +0x39c
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem(0xc0002360b0, {0x200e870, 0xc00045d9f0})
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.5/pkg/internal/controller/controller.go:261 +0x1a8
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2()
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.5/pkg/internal/controller/controller.go:222 +0x73
created by sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2 in goroutine 195
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.5/pkg/internal/controller/controller.go:218 +0x46c

Screenshots

No response

Additional Environment Information

No response

Steps to Reproduce

deploy csm

Expected Behavior

sucessfull creation of csm

CSM Driver(s)

csm 1.12
powerflex

Installation Type

CSM Operator

Container Storage Modules Enabled

No response

Container Orchestrator

OpenShift 4.17

Operating System

CoreOS

@anurajpd anurajpd added needs-triage Issue requires triage. type/bug Something isn't working. This is the default label associated with a bug issue. labels Nov 22, 2024
@csmbot
Copy link
Collaborator

csmbot commented Nov 22, 2024

@anurajpd: Thank you for submitting this issue!

The issue is currently awaiting triage. Please make sure you have given us as much context as possible.

If the maintainers determine this is a relevant issue, they will remove the needs-triage label and respond appropriately.


We want your feedback! If you have any questions or suggestions regarding our contributing process/workflow, please reach out to us at container.storage.modules@dell.com.

@shanmydell shanmydell added this to the v1.13.0 milestone Nov 22, 2024
@shanmydell
Copy link
Collaborator

/sync

@gallacher gallacher added area/csm-operator and removed needs-triage Issue requires triage. labels Nov 22, 2024
@csmbot
Copy link
Collaborator

csmbot commented Nov 22, 2024

link: 30620

@shefali-malhotra
Copy link
Collaborator

@anurajpd Please confirm if you provided MDM values in the Powerflex CR under initContainers section.

If you are adding initContainers section in the CR, please provided MDM values as below:

initContainers:
  - image: docker.io/dellemc/sdc:4.5.2.1
    imagePullPolicy: IfNotPresent
    name: sdc
    envs:
      - name: MDM
        value: "10.xx.xx.xx,10.xx.xx.xx"  # provide MDM value

If you don't want to provide MDM details, please skip the initContainers section so that it picks from the default configuration.

@anurajpd
Copy link
Author

The MDM value is already provided in the secret, so don't want to provide the same information at multiple places but need to change the sdc init container image repository, so just override that only.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/csm-operator type/bug Something isn't working. This is the default label associated with a bug issue.
Projects
None yet
Development

No branches or pull requests

6 participants