mongodb-operator cannot remove tolerations from statefulset/pods as specified in CR

[tylergu]

What did you do to encounter the bug?
We first deploy the MongoDB cluster with a toleration sepecified.
Then we tried to remove the toleration from the statefulSet/pods, by deleting the toleration in spec.statefulSet.template.spec.tolerations.
Then we realized that the operator is unable to remove the toleration.

Steps to reproduce the behavior:

1. Deploy the MongoDB cluster with spec:

apiVersion: mongodbcommunity.mongodb.com/v1
kind: MongoDBCommunity
metadata:
  namespace: mongodb
  name: test-cluster
spec:
  automationConfig:
      processes:
      - disabled: false
        name: test-cluster-1
  members: 3
  type: ReplicaSet
  version: "4.4.0"
  security:
    authentication:
      modes: ["SCRAM"]
  users:
    - name: my-user
      db: admin
      passwordSecretRef: # a reference to the secret that will be used to generate the user's password
        name: my-user-password
      roles:
        - name: clusterAdmin
          db: admin
        - name: userAdminAnyDatabase
          db: admin
      scramCredentialsSecretName: my-scram
  statefulSet:
    spec:
      template:
        spec:
          containers:
          - name: mongod
            resources:
              limits:
                cpu: '1'
                memory: 1000M
              requests:
                cpu: '1'
                memory: 1000M
          - name: mongodb-agent
            resources:
              limits:
                cpu: '1'
                memory: 1000M
              requests:
                cpu: '1'
                memory: 1000M
          tolerations:
          - effect: NoSchedule
            key: test-key
            operator: Equal
            value: test-value

2. Remove the toleration from the spec.statefulSet.template.spec.tolerations:

apiVersion: mongodbcommunity.mongodb.com/v1
kind: MongoDBCommunity
metadata:
  namespace: mongodb
  name: test-cluster
spec:
  automationConfig:
      processes:
      - disabled: false
        name: test-cluster-1
  members: 3
  type: ReplicaSet
  version: "4.4.0"
  security:
    authentication:
      modes: ["SCRAM"]
  users:
    - name: my-user
      db: admin
      passwordSecretRef: # a reference to the secret that will be used to generate the user's password
        name: my-user-password
      roles:
        - name: clusterAdmin
          db: admin
        - name: userAdminAnyDatabase
          db: admin
      scramCredentialsSecretName: my-scram
  statefulSet:
    spec:
      template:
        spec:
          containers:
          - name: mongod
            resources:
              limits:
                cpu: '1'
                memory: 1000M
              requests:
                cpu: '1'
                memory: 1000M
          - name: mongodb-agent
            resources:
              limits:
                cpu: '1'
                memory: 1000M
              requests:
                cpu: '1'
                memory: 1000M

What did you expect?
The operator should be able to remove the toleration from the statefulset as specified in the CR.

What happened instead?
The toleration is left on the statefulset and cannot be removed

Operator Information

• Operator Version - 0.7.4
• MongoDB Image used - 4.4.0

Kubernetes Cluster Information

kubectl version --short --output=yaml

$ kubectl version --short --output=yaml
clientVersion:
  buildDate: "2022-05-24T12:26:19Z"
  compiler: gc
  gitCommit: 3ddd0f45aa91e2f30c70734b175631bec5b5825a
  gitTreeState: clean
  gitVersion: v1.24.1
  goVersion: go1.18.2
  major: "1"
  minor: "24"
  platform: linux/amd64
kustomizeVersion: v4.5.4
serverVersion:
  buildDate: "2021-05-21T23:01:33Z"
  compiler: gc
  gitCommit: 5e58841cce77d4bc13713ad2b91fa0d961e69192
  gitTreeState: clean
  gitVersion: v1.21.1
  goVersion: go1.16.4
  major: "1"
  minor: "21"
  platform: linux/amd64

[tylergu]

mongodb/mongodb-kubernetes-operator#1251

[Spedoske]

Root cause: improper implementation for updating tolerations.
https://github.com/mongodb/mongodb-kubernetes-operator/blob/master/pkg/util/merge/merge.go#L506