Upgrading operator version cannot update sts

[broken-ufa]

I'm trying to upgrade operator from 0.9 to newer versions and operator cannot upgrade statefulsets because of this commit ab7bd65

Upgrading spec.selector.matchLabels is forbidden.

What version of redis operator are you using?
0.9

kubectl -n common logs redis-cluster-operator-d7d9b5c7b-gpkzv
{"level":"info","ts":1673947667.5821903,"logger":"controllers.RedisCluster","msg":"Reconciling opstree redis Cluster controller","Request.Namespace":"common","Request.Name":"redis-cluster"}
{"level":"info","ts":1673947667.5850625,"logger":"controller_redis","msg":"Redis statefulset get action was successful","Request.StatefulSet.Namespace":"common","Request.StatefulSet.Name":"redis-cluster-leader"}
{"level":"info","ts":1673947667.5869122,"logger":"controller_redis","msg":"Changes in statefulset Detected, Updating...","Request.StatefulSet.Namespace":"common","Request.StatefulSet.Name":"redis-cluster-leader","patch":"{\"metadata\":{\"annotations\":{\"meta.helm.sh/release-name\":\"local-localenv\",\"meta.helm.sh/release-namespace\":\"common\",\"project.werf.io/env\":\"local\",\"project.werf.io/name\":\"localenv\",\"prometheus.io/port\":null,\"prometheus.io/scrape\":null,\"redis.opstreelabs.instance\":\"redis-cluster\",\"werf.io/version\":\"v1.2.196\"},\"labels\":{\"app\":\"redis-cluster\",\"app.kubernetes.io/managed-by\":\"Helm\"}},\"spec\":{\"selector\":{\"matchLabels\":{\"app\":\"redis-cluster\",\"app.kubernetes.io/managed-by\":\"Helm\"}},\"serviceName\":\"redis-cluster-leader-headless\",\"template\":{\"metadata\":{\"annotations\":{\"meta.helm.sh/release-name\":\"local-localenv\",\"meta.helm.sh/release-namespace\":\"common\",\"project.werf.io/env\":\"local\",\"project.werf.io/name\":\"localenv\",\"redis.opstreelabs.in\":\"true\",\"redis.opstreelabs.instance\":\"redis-cluster\",\"werf.io/version\":\"v1.2.196\"},\"labels\":{\"app\":\"redis-cluster\",\"app.kubernetes.io/managed-by\":\"Helm\"}},\"spec\":{\"$setElementOrder/containers\":[{\"name\":\"redis-cluster-leader\"}],\"containers\":[{\"$setElementOrder/env\":[{\"name\":\"REDIS_ADDR\"},{\"name\":\"REDIS_ADDR\"},{\"name\":\"SERVER_MODE\"},{\"name\":\"SETUP_MODE\"}],\"$setElementOrder/volumeMounts\":[{\"mountPath\":\"/etc/redis/external.conf.d\"}],\"env\":[{\"$patch\":\"delete\",\"name\":\"PERSISTENCE_ENABLED\"}],\"livenessProbe\":{\"failureThreshold\":3,\"initialDelaySeconds\":1,\"periodSeconds\":10,\"timeoutSeconds\":1},\"name\":\"redis-cluster-leader\",\"readinessProbe\":{\"failureThreshold\":3,\"initialDelaySeconds\":1,\"periodSeconds\":10,\"timeoutSeconds\":1},\"volumeMounts\":[{\"$patch\":\"delete\",\"mountPath\":\"/data\"}]}]}},\"volumeClaimTemplates\":null}}"}
{"level":"error","ts":1673947667.5906074,"logger":"controller_redis","msg":"Redis stateful update failed","Request.StatefulSet.Namespace":"common","Request.StatefulSet.Name":"redis-cluster-leader","error":"StatefulSet.apps \"redis-cluster-leader\" is invalid: spec: Forbidden: updates to statefulset spec for fields other than 'replicas', 'template', 'updateStrategy', 'persistentVolumeClaimRetentionPolicy' and 'minReadySeconds' are forbidden","stacktrace":"redis-operator/k8sutils.patchStatefulSet\n\t/workspace/k8sutils/statefulset.go:179\nredis-operator/k8sutils.CreateOrUpdateStateFul\n\t/workspace/k8sutils/statefulset.go:78\nredis-operator/k8sutils.RedisClusterSTS.CreateRedisClusterSetup\n\t/workspace/k8sutils/redis-cluster.go:158\nredis-operator/k8sutils.CreateRedisLeader\n\t/workspace/k8sutils/redis-cluster.go:114\nredis-operator/controllers.(*RedisClusterReconciler).Reconcile\n\t/workspace/controllers/rediscluster_controller.go:73\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:114\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:311\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:227"}
{"level":"error","ts":1673947667.5906496,"logger":"controller_redis","msg":"Cannot create statefulset for Redis","Request.StatefulSet.Namespace":"common","Request.StatefulSet.Name":"redis-cluster-leader","Setup.Type":"leader","error":"StatefulSet.apps \"redis-cluster-leader\" is invalid: spec: Forbidden: updates to statefulset spec for fields other than 'replicas', 'template', 'updateStrategy', 'persistentVolumeClaimRetentionPolicy' and 'minReadySeconds' are forbidden","stacktrace":"redis-operator/k8sutils.CreateRedisLeader\n\t/workspace/k8sutils/redis-cluster.go:114\nredis-operator/controllers.(*RedisClusterReconciler).Reconcile\n\t/workspace/controllers/rediscluster_controller.go:73\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:114\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:311\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:227"}
{"level":"error","ts":1673947667.590679,"logger":"controller.rediscluster","msg":"Reconciler error","reconciler group":"redis.redis.opstreelabs.in","reconciler kind":"RedisCluster","name":"redis-cluster","namespace":"common","error":"StatefulSet.apps \"redis-cluster-leader\" is invalid: spec: Forbidden: updates to statefulset spec for fields other than 'replicas', 'template', 'updateStrategy', 'persistentVolumeClaimRetentionPolicy' and 'minReadySeconds' are forbidden","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:227"}
{"level":"info","ts":1673947677.8313842,"logger":"controllers.RedisCluster","msg":"Reconciling opstree redis Cluster controller","Request.Namespace":"common","Request.Name":"redis-cluster"}
{"level":"info","ts":1673947677.836951,"logger":"controller_redis","msg":"Redis statefulset get action was successful","Request.StatefulSet.Namespace":"common","Request.StatefulSet.Name":"redis-cluster-leader"}
{"level":"info","ts":1673947677.8410318,"logger":"controller_redis","msg":"Changes in statefulset Detected, Updating...","Request.StatefulSet.Namespace":"common","Request.StatefulSet.Name":"redis-cluster-leader","patch":"{\"metadata\":{\"annotations\":{\"meta.helm.sh/release-name\":\"local-localenv\",\"meta.helm.sh/release-namespace\":\"common\",\"project.werf.io/env\":\"local\",\"project.werf.io/name\":\"localenv\",\"prometheus.io/port\":null,\"prometheus.io/scrape\":null,\"redis.opstreelabs.instance\":\"redis-cluster\",\"werf.io/version\":\"v1.2.196\"},\"labels\":{\"app\":\"redis-cluster\",\"app.kubernetes.io/managed-by\":\"Helm\"}},\"spec\":{\"selector\":{\"matchLabels\":{\"app\":\"redis-cluster\",\"app.kubernetes.io/managed-by\":\"Helm\"}},\"serviceName\":\"redis-cluster-leader-headless\",\"template\":{\"metadata\":{\"annotations\":{\"meta.helm.sh/release-name\":\"local-localenv\",\"meta.helm.sh/release-namespace\":\"common\",\"project.werf.io/env\":\"local\",\"project.werf.io/name\":\"localenv\",\"redis.opstreelabs.in\":\"true\",\"redis.opstreelabs.instance\":\"redis-cluster\",\"werf.io/version\":\"v1.2.196\"},\"labels\":{\"app\":\"redis-cluster\",\"app.kubernetes.io/managed-by\":\"Helm\"}},\"spec\":{\"$setElementOrder/containers\":[{\"name\":\"redis-cluster-leader\"}],\"containers\":[{\"$setElementOrder/env\":[{\"name\":\"REDIS_ADDR\"},{\"name\":\"REDIS_ADDR\"},{\"name\":\"SERVER_MODE\"},{\"name\":\"SETUP_MODE\"}],\"$setElementOrder/volumeMounts\":[{\"mountPath\":\"/etc/redis/external.conf.d\"}],\"env\":[{\"$patch\":\"delete\",\"name\":\"PERSISTENCE_ENABLED\"}],\"livenessProbe\":{\"failureThreshold\":3,\"initialDelaySeconds\":1,\"periodSeconds\":10,\"timeoutSeconds\":1},\"name\":\"redis-cluster-leader\",\"readinessProbe\":{\"failureThreshold\":3,\"initialDelaySeconds\":1,\"periodSeconds\":10,\"timeoutSeconds\":1},\"volumeMounts\":[{\"$patch\":\"delete\",\"mountPath\":\"/data\"}]}]}},\"volumeClaimTemplates\":null}}"}
{"level":"error","ts":1673947677.8497245,"logger":"controller_redis","msg":"Redis stateful update failed","Request.StatefulSet.Namespace":"common","Request.StatefulSet.Name":"redis-cluster-leader","error":"StatefulSet.apps \"redis-cluster-leader\" is invalid: spec: Forbidden: updates to statefulset spec for fields other than 'replicas', 'template', 'updateStrategy', 'persistentVolumeClaimRetentionPolicy' and 'minReadySeconds' are forbidden","stacktrace":"redis-operator/k8sutils.patchStatefulSet\n\t/workspace/k8sutils/statefulset.go:179\nredis-operator/k8sutils.CreateOrUpdateStateFul\n\t/workspace/k8sutils/statefulset.go:78\nredis-operator/k8sutils.RedisClusterSTS.CreateRedisClusterSetup\n\t/workspace/k8sutils/redis-cluster.go:158\nredis-operator/k8sutils.CreateRedisLeader\n\t/workspace/k8sutils/redis-cluster.go:114\nredis-operator/controllers.(*RedisClusterReconciler).Reconcile\n\t/workspace/controllers/rediscluster_controller.go:73\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:114\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:311\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:227"}
{"level":"error","ts":1673947677.8498228,"logger":"controller_redis","msg":"Cannot create statefulset for Redis","Request.StatefulSet.Namespace":"common","Request.StatefulSet.Name":"redis-cluster-leader","Setup.Type":"leader","error":"StatefulSet.apps \"redis-cluster-leader\" is invalid: spec: Forbidden: updates to statefulset spec for fields other than 'replicas', 'template', 'updateStrategy', 'persistentVolumeClaimRetentionPolicy' and 'minReadySeconds' are forbidden","stacktrace":"redis-operator/k8sutils.CreateRedisLeader\n\t/workspace/k8sutils/redis-cluster.go:114\nredis-operator/controllers.(*RedisClusterReconciler).Reconcile\n\t/workspace/controllers/rediscluster_controller.go:73\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:114\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:311\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:227"}

redis-operator version:

Does this issue reproduce with the latest release?
yes

What operating system and processor architecture are you using (kubectl version)?
Ubuntu 22.04, kubectl v1.25.4

kubectl version Output

broken@g3:~$ kubectl version
WARNING: This version information is deprecated and will be replaced with the output from kubectl version --short.  Use --output=yaml|json to get the full version.
Client Version: version.Info{Major:"1", Minor:"25", GitVersion:"v1.25.4", GitCommit:"872a965c6c6526caa949f0c6ac028ef7aff3fb78", GitTreeState:"clean", BuildDate:"2022-11-09T13:36:36Z", GoVersion:"go1.19.3", Compiler:"gc", Platform:"linux/amd64"}
Kustomize Version: v4.5.7
Server Version: version.Info{Major:"1", Minor:"25", GitVersion:"v1.25.4-rc4+k3s1", GitCommit:"0dc63334c0db3e7b99244427615e091909fc486e", GitTreeState:"clean", BuildDate:"2022-11-15T23:58:33Z", GoVersion:"go1.19.3", Compiler:"gc", Platform:"linux/amd64"}

What did you do?

I'm trying to upgrade operator version from 0.9 to 0.13 with already existed rediclusters in k8s-cluster
If I remove existed sts - it recreates with passed thru labels

There is diff between removed sts and just autocreated one:

apiVersion: apps/v1                                             apiVersion: apps/v1
kind: StatefulSet                                               kind: StatefulSet
metadata:                                                       metadata:
  annotations:                                                    annotations:
    banzaicloud.com/last-applied: UEsDBBQACAAIAAAAAAAAAAAAAAA |     banzaicloud.com/last-applied: UEsDBBQACAAIAAAAAAAAAAAAAAA
    prometheus.io/port: "9121"                                |     meta.helm.sh/release-name: local-localenv
    prometheus.io/scrape: "true"                              |     meta.helm.sh/release-namespace: common
                                                              >     project.werf.io/env: local
                                                              >     project.werf.io/name: localenv
    redis.opstreelabs.in: "true"                                    redis.opstreelabs.in: "true"
  creationTimestamp: "2023-01-16T15:06:56Z"                   |     redis.opstreelabs.instance: redis-cluster
  generation: 3                                               |     werf.io/version: v1.2.195
                                                              >   creationTimestamp: "2023-01-16T15:17:08Z"
                                                              >   generation: 1
  labels:                                                         labels:
    app: redis-cluster-leader                                 |     app: redis-cluster
                                                              >     app.kubernetes.io/managed-by: Helm
    redis_setup_type: cluster                                       redis_setup_type: cluster
    role: leader                                                    role: leader
  name: redis-cluster-leader                                      name: redis-cluster-leader
  namespace: common                                               namespace: common
  ownerReferences:                                                ownerReferences:
  - apiVersion: redis.redis.opstreelabs.in/v1beta1                - apiVersion: redis.redis.opstreelabs.in/v1beta1
    controller: true                                                controller: true
    kind: RedisCluster                                              kind: RedisCluster
    name: redis-cluster                                             name: redis-cluster
    uid: 6e015c57-f728-48ab-99eb-fad0149fc9aa                       uid: 6e015c57-f728-48ab-99eb-fad0149fc9aa
  resourceVersion: "44284"                                    |   resourceVersion: "44859"
  uid: be5d9b93-6814-4aa3-86d6-b48a3f736496                   |   uid: 1a145502-4e69-48a8-8e2c-431023eebb71
spec:                                                           spec:
  podManagementPolicy: OrderedReady                               podManagementPolicy: OrderedReady
  replicas: 3                                                     replicas: 3
  revisionHistoryLimit: 10                                        revisionHistoryLimit: 10
  selector:                                                       selector:
    matchLabels:                                                    matchLabels:
      app: redis-cluster-leader                               |       app: redis-cluster
                                                              >       app.kubernetes.io/managed-by: Helm
      redis_setup_type: cluster                                       redis_setup_type: cluster
      role: leader                                                    role: leader
  serviceName: redis-cluster-leader                               serviceName: redis-cluster-leader
  template:                                                       template:
    metadata:                                                       metadata:
                                                              >       annotations:
                                                              >         meta.helm.sh/release-name: local-localenv
                                                              >         meta.helm.sh/release-namespace: common
                                                              >         project.werf.io/env: local
                                                              >         project.werf.io/name: localenv
                                                              >         redis.opstreelabs.in: "true"
                                                              >         redis.opstreelabs.instance: redis-cluster
                                                              >         werf.io/version: v1.2.195
      creationTimestamp: null                                         creationTimestamp: null
      labels:                                                         labels:
        app: redis-cluster-leader                             |         app: redis-cluster
                                                              >         app.kubernetes.io/managed-by: Helm
        redis_setup_type: cluster                                       redis_setup_type: cluster
        role: leader                                                    role: leader

What did you expect to see?

Upgraging operator version without any issues

What did you see instead?

Errors in logs and impossibility of reconciling existed redisclusters

---

Can you make flag in values to turn this behaviour off?

[iamabhishek-dubey]

Fixed in #411