Incomplete CRDs in latest published helm repository?

[zumbuschms]

Describe your question
As we have encountered issues with deletion of PVCs during uninstallation of Redis replication charts using 0.15.1 of the operator, I was searching the current documentation and found the keepAfterDelete option. As that feature was added in 0.16.0, I decided to update the operator to the latest version and downloaded the archive from the helm repo, as our deployment environment doesn't have Internet access (helm pull ot-helm/redis-operator). Checking the downloaded archive, I was not able to find any definitions for keepAfterDelete in the CRDs and when I compared the content of the 0.16.1 download with the 0.15.1 version we had used since September 2023, all files were identical.

Is the new feature not yet available in the published helm charts, although the info announces version 0.17.0 of the operator?

Output of helm show chart ot-helm/redis-operator

apiVersion: v2
appVersion: 0.17.0
dependencies:
- alias: certmanager
  condition: certmanager.enabled
  name: cert-manager
  repository: https://charts.jetstack.io
  version: v1.12.4
description: Provides easy redis setup definitions for Kubernetes services, and deployment.
home: https://github.com/OT-CONTAINER-KIT/redis-operator
icon: https://github.com/OT-CONTAINER-KIT/redis-operator/raw/master/static/redis-operator-logo.svg
keywords:
- operator
- redis
- opstree
- kubernetes
- openshift
maintainers:
- name: iamabhishek-dubey
- name: sandy724
- name: shubham-cmyk
name: redis-operator
sources:
- https://github.com/OT-CONTAINER-KIT/redis-operator
version: 0.16.1

What version of redis-operator are you using?
chart version: 0.16.1
app version: 0.17.0

[drivebyer]

@zumbuschms , Please try https://github.com/OT-CONTAINER-KIT/helm-charts/releases/tag/redis-operator-0.16.4

[zumbuschms]

The keepAfterDelete option doesn't work with that release either, but this time the operator recognizes the parameter but the operator pod crashes trying to deploy the redis-replication instance. I've downloaded redis-operator-0.16.4, redis-replication-0.16.0 and redis-sentinl-0.16.0 from the helm-charts releases, extracted each into a separate folder to have it easier to search the contents of the helmcharts and installed the operator from that folder:

helm upgrade redis-operator .\redis-operator-0.16.4 --install

Next I installed the redis-replication chart setting the keepAfterDelete parameter and a custom storage class with retaining reclaim policy:

helm upgrade redist-replication .\redis-replication-0.16.0 --install --set storageSpec.keepAfterDelete=true --set storageSpec.volumeClaimTemplate.spec.storageClassName=azure-disk-retain

The Redis replication pods are never created and the Redis operator pod is in a restart cycle. This is the outout of kubectl logs for that pod:

{"level":"info","ts":"2024-06-24T10:41:05Z","logger":"setup","msg":"starting manager"}
{"level":"info","ts":"2024-06-24T10:41:05Z","logger":"controller-runtime.metrics","msg":"Starting metrics server"}
{"level":"info","ts":"2024-06-24T10:41:05Z","msg":"starting server","kind":"health probe","addr":"[::]:8081"}
{"level":"info","ts":"2024-06-24T10:41:05Z","logger":"controller-runtime.metrics","msg":"Serving metrics server","bindAddress":":8080","secure":false}
I0624 10:41:05.933505 1 leaderelection.go:250] attempting to acquire leader lease default/6cab913b.redis.opstreelabs.in...
I0624 10:41:21.343740 1 leaderelection.go:260] successfully acquired lease default/6cab913b.redis.opstreelabs.in
{"level":"info","ts":"2024-06-24T10:41:21Z","msg":"Starting EventSource","controller":"redis","controllerGroup":"redis.redis.opstreelabs.in","controllerKind":"Redis","source":"kind source: *v1beta2.Redis"}
{"level":"info","ts":"2024-06-24T10:41:21Z","msg":"Starting Controller","controller":"redis","controllerGroup":"redis.redis.opstreelabs.in","controllerKind":"Redis"}
{"level":"info","ts":"2024-06-24T10:41:21Z","msg":"Starting EventSource","controller":"redisreplication","controllerGroup":"redis.redis.opstreelabs.in","controllerKind":"RedisReplication","source":"kind source: *v1beta2.RedisReplication"}
{"level":"info","ts":"2024-06-24T10:41:21Z","msg":"Starting Controller","controller":"redisreplication","controllerGroup":"redis.redis.opstreelabs.in","controllerKind":"RedisReplication"}
{"level":"info","ts":"2024-06-24T10:41:21Z","msg":"Starting EventSource","controller":"redissentinel","controllerGroup":"redis.redis.opstreelabs.in","controllerKind":"RedisSentinel","source":"kind source: *v1beta2.RedisSentinel"}
{"level":"info","ts":"2024-06-24T10:41:21Z","msg":"Starting EventSource","controller":"redissentinel","controllerGroup":"redis.redis.opstreelabs.in","controllerKind":"RedisSentinel","source":"kind source: *v1beta2.RedisReplication"}
{"level":"info","ts":"2024-06-24T10:41:21Z","msg":"Starting Controller","controller":"redissentinel","controllerGroup":"redis.redis.opstreelabs.in","controllerKind":"RedisSentinel"}
{"level":"info","ts":"2024-06-24T10:41:21Z","msg":"Starting EventSource","controller":"rediscluster","controllerGroup":"redis.redis.opstreelabs.in","controllerKind":"RedisCluster","source":"kind source: *v1beta2.RedisCluster"}
{"level":"info","ts":"2024-06-24T10:41:21Z","msg":"Starting Controller","controller":"rediscluster","controllerGroup":"redis.redis.opstreelabs.in","controllerKind":"RedisCluster"}
{"level":"info","ts":"2024-06-24T10:41:21Z","msg":"Starting workers","controller":"redis","controllerGroup":"redis.redis.opstreelabs.in","controllerKind":"Redis","worker count":1}
{"level":"info","ts":"2024-06-24T10:41:21Z","msg":"Starting workers","controller":"rediscluster","controllerGroup":"redis.redis.opstreelabs.in","controllerKind":"RedisCluster","worker count":1}
{"level":"info","ts":"2024-06-24T10:41:21Z","msg":"Starting workers","controller":"redisreplication","controllerGroup":"redis.redis.opstreelabs.in","controllerKind":"RedisReplication","worker count":1}
{"level":"info","ts":"2024-06-24T10:41:21Z","msg":"Starting workers","controller":"redissentinel","controllerGroup":"redis.redis.opstreelabs.in","controllerKind":"RedisSentinel","worker count":1}
{"level":"info","ts":"2024-06-24T10:41:21Z","logger":"controllers.RedisReplication","msg":"Reconciling opstree redis replication controller","Request.Namespace":"default","Request.Name":"redis-replication"}
{"level":"info","ts":"2024-06-24T10:41:21Z","msg":"Observed a panic in reconciler: runtime error: invalid memory address or nil pointer dereference","controller":"redisreplication","controllerGroup":"redis.redis.opstreelabs.in","controllerKind":"RedisReplication","RedisReplication":{"name":"redis-replication","namespace":"default"},"namespace":"default","name":"redis-replication","reconcileID":"73a2452f-4377-45b5-b98c-87826b55d01c"}
panic: runtime error: invalid memory address or nil pointer dereference [recovered]
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x17a70e8]

goroutine 128 [running]:
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile.func1()
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:116 +0x1e5
panic({0x19cf900?, 0x2cc0ca0?})
/usr/local/go/src/runtime/panic.go:914 +0x21f
github.com/OT-CONTAINER-KIT/redis-operator/k8sutils.getProbeInfo(0x0, 0x0?, 0x0, 0x0)
/workspace/k8sutils/statefulset.go:617 +0x3e8
github.com/OT-CONTAINER-KIT/redis-operator/k8sutils.generateContainerDef({_, }, {{0xc00005ae20, 0x1d}, {0xc0005c6d50, 0xc}, 0x0, 0x0, {0xc0000579b0, 0x26}, ...}, ...)
/workspace/k8sutils/statefulset.go:369 +0x159
github.com/OT-CONTAINER-KIT/redis-operator/k8sutils.generateStatefulSetsDef({{0xc000058e10, 0x11}, {0x0, 0x0}, {0xc0005c6d36, 0x7}, {0x0, 0x0}, {0x0, 0x0}, ...}, ...)
/workspace/k8sutils/statefulset.go:234 +0x467
github.com/OT-CONTAINER-KIT/redis-operator/k8sutils.CreateOrUpdateStateFul({, }, {{, _}, }, {, _}, {{0xc000058e10, 0x11}, {0x0, ...}, ...}, ...)
/workspace/k8sutils/statefulset.go:100 +0x1a5
github.com/OT-CONTAINER-KIT/redis-operator/k8sutils.CreateReplicationRedis(0xc0006f5680, {0x1f12bd0, 0xc0000ab1e0})
/workspace/k8sutils/redis-replication.go:64 +0x873
github.com/OT-CONTAINER-KIT/redis-operator/controllers.(*RedisReplicationReconciler).Reconcile(0xc0000b6fc0, {0x1ef8ed0, 0xc0006f9350}, {{{0xc0005c6d36?, 0x5?}, {0xc000058e10?, 0xc0001c7d08?}}})
/workspace/controllers/redisreplication_controller.go:60 +0x36c
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile(0x1efc1d0?, {0x1ef8ed0?, 0xc0006f9350?}, {{{0xc0005c6d36?, 0xb?}, {0xc000058e10?, 0x0?}}})
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:119 +0xb7
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler(0xc000019400, {0x1ef8f08, 0xc0003b9180}, {0x1a86860?, 0xc0001118e0?})
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:316 +0x3cc
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem(0xc000019400, {0x1ef8f08, 0xc0003b9180})
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:266 +0x1af
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2()
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:227 +0x79
created by sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2 in goroutine 82
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:223 +0x565

I found a Redis replication deployment in the Git repo for the operator that seems to be used for testing (/tests/e2e-chainsaw/v1beta2/keep-pvc/redis-replication/replication.yaml) and added the custom storage class and installed it using kubectl:

---
apiVersion: redis.redis.opstreelabs.in/v1beta2
kind: RedisReplication
metadata:
  name: redis-replication
spec:
  clusterSize: 3
  podSecurityContext:
    runAsUser: 1000
    fsGroup: 1000
  kubernetesConfig:
    image: quay.io/opstree/redis:latest
    imagePullPolicy: Always
    resources:
      requests:
        cpu: 101m
        memory: 128Mi
      limits:
        cpu: 101m
        memory: 128Mi
  storage:
    keepAfterDelete: true
    volumeClaimTemplate:
      spec:
        accessModes: [ReadWriteOnce]
        resources:
          requests:
            storage: 1Gi

kubectl apply -f .\redis-replication.yml

But the operator keeps crashing and the log is almost exactly the same:

{"level":"info","ts":"2024-06-24T10:41:05Z","logger":"setup","msg":"starting manager"}
{"level":"info","ts":"2024-06-24T10:41:05Z","logger":"controller-runtime.metrics","msg":"Starting metrics server"}
{"level":"info","ts":"2024-06-24T10:41:05Z","msg":"starting server","kind":"health probe","addr":"[::]:8081"}
{"level":"info","ts":"2024-06-24T10:41:05Z","logger":"controller-runtime.metrics","msg":"Serving metrics server","bindAddress":":8080","secure":false}
I0624 10:41:05.933505 1 leaderelection.go:250] attempting to acquire leader lease default/6cab913b.redis.opstreelabs.in...
I0624 10:41:21.343740 1 leaderelection.go:260] successfully acquired lease default/6cab913b.redis.opstreelabs.in
{"level":"info","ts":"2024-06-24T10:41:21Z","msg":"Starting EventSource","controller":"redis","controllerGroup":"redis.redis.opstreelabs.in","controllerKind":"Redis","source":"kind source: *v1beta2.Redis"}
{"level":"info","ts":"2024-06-24T10:41:21Z","msg":"Starting Controller","controller":"redis","controllerGroup":"redis.redis.opstreelabs.in","controllerKind":"Redis"}
{"level":"info","ts":"2024-06-24T10:41:21Z","msg":"Starting EventSource","controller":"redisreplication","controllerGroup":"redis.redis.opstreelabs.in","controllerKind":"RedisReplication","source":"kind source: *v1beta2.RedisReplication"}
{"level":"info","ts":"2024-06-24T10:41:21Z","msg":"Starting Controller","controller":"redisreplication","controllerGroup":"redis.redis.opstreelabs.in","controllerKind":"RedisReplication"}
{"level":"info","ts":"2024-06-24T10:41:21Z","msg":"Starting EventSource","controller":"redissentinel","controllerGroup":"redis.redis.opstreelabs.in","controllerKind":"RedisSentinel","source":"kind source: *v1beta2.RedisSentinel"}
{"level":"info","ts":"2024-06-24T10:41:21Z","msg":"Starting EventSource","controller":"redissentinel","controllerGroup":"redis.redis.opstreelabs.in","controllerKind":"RedisSentinel","source":"kind source: *v1beta2.RedisReplication"}
{"level":"info","ts":"2024-06-24T10:41:21Z","msg":"Starting Controller","controller":"redissentinel","controllerGroup":"redis.redis.opstreelabs.in","controllerKind":"RedisSentinel"}
{"level":"info","ts":"2024-06-24T10:41:21Z","msg":"Starting EventSource","controller":"rediscluster","controllerGroup":"redis.redis.opstreelabs.in","controllerKind":"RedisCluster","source":"kind source: *v1beta2.RedisCluster"}
{"level":"info","ts":"2024-06-24T10:41:21Z","msg":"Starting Controller","controller":"rediscluster","controllerGroup":"redis.redis.opstreelabs.in","controllerKind":"RedisCluster"}
{"level":"info","ts":"2024-06-24T10:41:21Z","msg":"Starting workers","controller":"redis","controllerGroup":"redis.redis.opstreelabs.in","controllerKind":"Redis","worker count":1}
{"level":"info","ts":"2024-06-24T10:41:21Z","msg":"Starting workers","controller":"rediscluster","controllerGroup":"redis.redis.opstreelabs.in","controllerKind":"RedisCluster","worker count":1}
{"level":"info","ts":"2024-06-24T10:41:21Z","msg":"Starting workers","controller":"redisreplication","controllerGroup":"redis.redis.opstreelabs.in","controllerKind":"RedisReplication","worker count":1}
{"level":"info","ts":"2024-06-24T10:41:21Z","msg":"Starting workers","controller":"redissentinel","controllerGroup":"redis.redis.opstreelabs.in","controllerKind":"RedisSentinel","worker count":1}
{"level":"info","ts":"2024-06-24T10:41:21Z","logger":"controllers.RedisReplication","msg":"Reconciling opstree redis replication controller","Request.Namespace":"default","Request.Name":"redis-replication"}
{"level":"info","ts":"2024-06-24T10:41:21Z","msg":"Observed a panic in reconciler: runtime error: invalid memory address or nil pointer dereference","controller":"redisreplication","controllerGroup":"redis.redis.opstreelabs.in","controllerKind":"RedisReplication","RedisReplication":{"name":"redis-replication","namespace":"default"},"namespace":"default","name":"redis-replication","reconcileID":"73a2452f-4377-45b5-b98c-87826b55d01c"}
panic: runtime error: invalid memory address or nil pointer dereference [recovered]
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x17a70e8]

goroutine 128 [running]:
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile.func1()
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:116 +0x1e5
panic({0x19cf900?, 0x2cc0ca0?})
/usr/local/go/src/runtime/panic.go:914 +0x21f
github.com/OT-CONTAINER-KIT/redis-operator/k8sutils.getProbeInfo(0x0, 0x0?, 0x0, 0x0)
/workspace/k8sutils/statefulset.go:617 +0x3e8
github.com/OT-CONTAINER-KIT/redis-operator/k8sutils.generateContainerDef({_, }, {{0xc00005ae20, 0x1d}, {0xc0005c6d50, 0xc}, 0x0, 0x0, {0xc0000579b0, 0x26}, ...}, ...)
/workspace/k8sutils/statefulset.go:369 +0x159
github.com/OT-CONTAINER-KIT/redis-operator/k8sutils.generateStatefulSetsDef({{0xc000058e10, 0x11}, {0x0, 0x0}, {0xc0005c6d36, 0x7}, {0x0, 0x0}, {0x0, 0x0}, ...}, ...)
/workspace/k8sutils/statefulset.go:234 +0x467
github.com/OT-CONTAINER-KIT/redis-operator/k8sutils.CreateOrUpdateStateFul({, }, {{, _}, }, {, _}, {{0xc000058e10, 0x11}, {0x0, ...}, ...}, ...)
/workspace/k8sutils/statefulset.go:100 +0x1a5
github.com/OT-CONTAINER-KIT/redis-operator/k8sutils.CreateReplicationRedis(0xc0006f5680, {0x1f12bd0, 0xc0000ab1e0})
/workspace/k8sutils/redis-replication.go:64 +0x873
github.com/OT-CONTAINER-KIT/redis-operator/controllers.(*RedisReplicationReconciler).Reconcile(0xc0000b6fc0, {0x1ef8ed0, 0xc0006f9350}, {{{0xc0005c6d36?, 0x5?}, {0xc000058e10?, 0xc0001c7d08?}}})
/workspace/controllers/redisreplication_controller.go:60 +0x36c
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile(0x1efc1d0?, {0x1ef8ed0?, 0xc0006f9350?}, {{{0xc0005c6d36?, 0xb?}, {0xc000058e10?, 0x0?}}})
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:119 +0xb7
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler(0xc000019400, {0x1ef8f08, 0xc0003b9180}, {0x1a86860?, 0xc0001118e0?})
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:316 +0x3cc
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem(0xc000019400, {0x1ef8f08, 0xc0003b9180})
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:266 +0x1af
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2()
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:227 +0x79
created by sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2 in goroutine 82
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:223 +0x565

Did I miss something or is this a new issue with the operator?

[drivebyer]

upgrade to 0.18.0 plz @zumbuschms

[zumbuschms]

I've started to test the latest releases and I already identified a few issues in the redis-sentinel and redis-replication charts. As soon as I have figured them completely out, I'll file the issue and try to provide a PR to fix them. The issues are around broken templates for sentinel and replication where the templates no longer match the CRDs.