Miscellaneous questions for Rabbitmq Operator

[xixiangzouyibian]

Hi, I using rabbitmq operator to deploy rabbitmq cluster for some time. Here post my questions which I can't find the documentation to support.

• Can I not deploy the rabbitmq messging topology operator from rabbitmq cluster operator helm charts ?
  • We are using istio to handle the same things (TLS, Routing, Permissions...), so messaging topology operator is not necessary to us.
• If I failed to upgrade rabbitmq cluster by operator, which means the ReconcileSuccess shows false, how can I rollback my upgrade and make ReconcileSuccess back to true ?
  • It doesn't seem to work if I rollback the changes in definition.yaml and apply it again.
    • Like: if I modify the storage size, the operator will report: Failed to scale PVCs: shrinking persistent volumes is not supported, then rollback the storage size to previous value, helm upgrade to apply it but the ReconcileSuccess still shows false
  • If last upgrade failed, the next upgrade (not the same change compared with last upgrde) will be blocked either ?

[xixiangzouyibian]

I'm using helm to deploy, but helm history only keeps lastest10 revisions, the correct revision has been drowned out, not sure if helm rollback can work for the case of "RconcileSuccess is false"

[ChunyiLyu]

Hi @xixiangzouyibian, if you would like install only the cluster operator, installing via the cluster operator manifest instead of the helm chart is a good option. You can simply run kubectl apply -f https://github.com/rabbitmq/cluster-operator/releases/latest/download/cluster-operator.yml (or point to any release version).

For the reconcile success status condition, if the failure is caused by a configuration (for instance in the example you gave, you were trying to down size the persistent volume size which is a forbidden operation) you could update the manifest to the previous successful version, and reconcile should succeed and the status condition will go back to true.

If an update failed, further updates are not blocked.

[xixiangzouyibian]

Hi @ChunyiLyu , Thanks for you response.
For some normalization requirments, we must use helm to deploy. So hope Rabbitmq cluster operator helm chats support toggle to disable Rabbitmq messaging topology operator deployment in the furture release.

As you mentioned:

you could update the manifest to the previous successful version

Do you mean I need to edit manifest back and apply it again ? Does "helm rollback" work for this case ?

If an update failed, further updates are not blocked

As my observed, when I failed to update storage size without handling, then rabbitmq cluster will not do rolling upgrade if I modify "spec.rabbitmq.additionalConfig", which really confused me.

[ChunyiLyu]

So hope Rabbitmq cluster operator helm chats support toggle to disable Rabbitmq messaging topology operator deployment in the furture release.
This sounds like a perfectly reasonable feature request. Would you mind creating an issue in https://github.com/bitnami/charts/pulls? The chart is under a different repo and creating a request there should get the right people's attention.

Do you mean I need to edit manifest back and apply it again ? Does "helm rollback" work for this case ?
I was referring to kubectl apply .... The rabbit team does development using kubectl, so I'm not familiar with using helm for Cluster Operator. Looking at https://helm.sh/docs/helm/helm_rollback/, if you updated the configuration using helm in the first place, then helm rollback should work.

As my observed, when I failed to update storage size without handling, then rabbitmq cluster will not do rolling upgrade if I modify "spec.rabbitmq.additionalConfig", which really confused me.

This could be caused by using helm, because this is not the behavior using the operator. If you change the persistent volume size, then proceed to update spec.rabbitmq.additionalConfig and change the the persistent volume size back to the original value in the same kubectl apply, it will succeed. The Cluster Operator, like many other native k8s controllers, tries to reconcile to the current configuration. Previous failed configuration shouldn't matter in this case. I suggest creating an issue in the helm repo if you keep seeing this problem.

[xixiangzouyibian]

Sure, will raise an issue about adding toggle to rabbitmq messaging topology operator.

For the last comment, let me describe my operation in a little more detail to make no difference in our understanding:

• Change pv size, helm upgrade (reconcile as false, not trigger rolling upgrade)
• Change "spec.rabbitmq.additionalConfig", helm upgrade (not working, not trigger rolling upgrade)
• Change pv size to original value, helm upgrade (reconcile as false, not trigger rolling upgrade)
• Change "spec.rabbitmq.additionalConfig" to original value (everything is rollback now), helm upgrade (reconcile as false, not trigger rolling upgrade)

If what I said is in line with what your understand, which has nothing to do with the cluster operator, I think I should ask for helm team indeed.

[xixiangzouyibian]

From my last try, even if I executed helm upgrade from original helm chart package, the events still said "shrinking persistent volumes is not supported". It seems that previous upgrade failures will be followed by subsequent upgrades and cause "ReconcileSuccess" always false.

[ChunyiLyu]

I think this is more to do with helm upgrade.

I tried similar steps with kubectl apply and cannot reproduce the issue. I did

1. create a RMQ with manifest

apiVersion: rabbitmq.com/v1beta1
kind: RabbitmqCluster
metadata:
  name: sample
  namespace: rabbitmq-system
spec:
  persistence:
    storage: 10Gi
  replicas: 1
  service:
    type: LoadBalancer

2. scale down PVC to 9Gi and get an error

  Warning  FailedReconcilePersistence  5s (x11 over 10s)       rabbitmqcluster-controller  Failed to scale PVCs: shrinking persistent volumes is not supported

3. scale back to 10Gi and add a RMQ config

apiVersion: rabbitmq.com/v1beta1
kind: RabbitmqCluster
metadata:
  name: sample
  namespace: rabbitmq-system
spec:
  persistence:
    storage: 10Gi
  replicas: 1
  service:
    type: LoadBalancer
  rabbitmq:
    additionalConfig: |
      log.console.level = debug

Status condition is updated and no new FailedReconcile events (the old failure still shows but from the time details you can tell it's a past event rather than new one)

    Last Transition Time:  2023-07-03T13:40:14Z
    Message:               Finish reconciling
    Reason:                Success
    Status:                True
    Type:                  ReconcileSuccess
...
Events:
  Type     Reason                      Age                     From                        Message
  ----     ------                      ----                    ----                        -------
  Normal   SuccessfulUpdate            7m34s (x50 over 2d13h)  rabbitmqcluster-controller  updated resource sample-server of Type *v1.StatefulSet
  Warning  FailedReconcilePersistence  4m58s (x11 over 5m3s)   rabbitmqcluster-controller  Failed to scale PVCs: shrinking persistent volumes is not supported
  Normal   SuccessfulUpdate            2m28s (x75 over 2d13h)  rabbitmqcluster-controller  updated resource sample-nodes of Type *v1.Service

I suggest look into behaviors of helm upgrade or use kubectl to manage your RMQ instances.

[xixiangzouyibian]

Thanks @ChunyiLyu for great support, will try with kubectl

[xixiangzouyibian]

@ChunyiLyu I tried with kubectl but unlucky for the attempt:

• Rolling upgrade was still not triggered although I changed rabbitmq version and resource limits, the rabbitmq pods age was old, and rabbitmqCluster showed "reconcileSuccess" was false.
• It seems the rabbitmqCluster status is always stuck in "observedGeneration: 5" and ignore any following upgrades, that is the changes applied in manifest, but not sync up changes to rabbitmq statefulset.
• I tried to "kubectl rollout deploy rabbitmq-operators", but the failure record is still kept.

1. kubectl get rmq rabbitmq -o yaml > rabbitmq_upgrade.yaml

apiVersion: rabbitmq.com/v1beta1
kind: RabbitmqCluster
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"rabbitmq.com/v1beta1","kind":"RabbitmqCluster","metadata":{"annotations":{},"labels":{"app.kubernetes.io/name":"rabbitmq"},"name":"rabbitmq","namespace":"xxx"},"spec":{"affinity":{"podAntiAffinity":{"requiredDuringSchedulingIgnoredDuringExecution":[{"labelSelector":{"matchLabels":{"app.kubernetes.io/name":"rabbitmq"}},"topologyKey":"kubernetes.io/hostname"}]}},"image":"dockerrepo:30500/bitnami/rabbitmq:3.11.17-debian-11-r3","persistence":{"storage":"20Gi","storageClassName":"local-path"},"rabbitmq":{"additionalConfig":"cluster_formation.peer_discovery_backend = rabbit_peer_discovery_k8s\ncluster_formation.k8s.host = kubernetes.default.svc.cluster.local\ncluster_formation.k8s.address_type = hostname\nvm_memory_high_watermark_paging_ratio = 0.85\ncluster_formation.node_cleanup.interval = 10\ncluster_partition_handling = autoheal\nqueue_leader_locator = balanced\nqueue_master_locator = min-masters\nloopback_users.guest = false\ndefault_user = guest\ndefault_pass = guest","additionalPlugins":["rabbitmq_management","rabbitmq_peer_discovery_k8s"]},"replicas":3,"resources":{"limits":{"cpu":2,"memory":"4Gi"},"requests":{"cpu":2,"memory":"4Gi"}}}}
    meta.helm.sh/release-name: xxx
    meta.helm.sh/release-namespace: xxx
  creationTimestamp: "2023-06-30T04:59:16Z"
  finalizers:
  - deletion.finalizers.rabbitmqclusters.rabbitmq.com
  generation: 12
  labels:
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: rabbitmq
    helm.toolkit.fluxcd.io/name: xxx
    helm.toolkit.fluxcd.io/namespace: xxx
  name: rabbitmq
  namespace: xxx
  resourceVersion: "75147544"
  uid: c446ec79-b7d0-4337-8249-a3378be085d9
spec:
  affinity:
    podAntiAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
      - labelSelector:
          matchLabels:
            app.kubernetes.io/name: rabbitmq
        topologyKey: kubernetes.io/hostname
  delayStartSeconds: 30
  image: dockerrepo:30500/bitnami/rabbitmq:3.11.17-debian-11-r3
  override: {}
  persistence:
    storage: 20Gi
    storageClassName: local-path
  rabbitmq:
    additionalConfig: |-
      cluster_formation.peer_discovery_backend = rabbit_peer_discovery_k8s
      cluster_formation.k8s.host = kubernetes.default.svc.cluster.local
      cluster_formation.k8s.address_type = hostname
      vm_memory_high_watermark_paging_ratio = 0.85
      cluster_formation.node_cleanup.interval = 10
      cluster_partition_handling = autoheal
      queue_master_locator = min-masters
      loopback_users.guest = false
      default_user = guest
      default_pass = guest
    additionalPlugins:
    - rabbitmq_management
    - rabbitmq_peer_discovery_k8s
  replicas: 3
  resources:
    limits:
      cpu: "1"
      memory: 1Gi
    requests:
      cpu: "2"
      memory: 4Gi
  secretBackend:
    externalSecret: {}
  service:
    type: ClusterIP
  terminationGracePeriodSeconds: 604800
  tls: {}
status:
  binding:
    name: rabbitmq-default-user
  conditions:
  - lastTransitionTime: "2023-06-30T10:47:53Z"
    reason: AllPodsAreReady
    status: "True"
    type: AllReplicasReady
  - lastTransitionTime: "2023-06-30T05:00:50Z"
    reason: AtLeastOneEndpointAvailable
    status: "True"
    type: ClusterAvailable
  - lastTransitionTime: "2023-07-04T07:54:35Z"
    reason: NoWarnings
    status: "True"
    type: NoWarnings
  - lastTransitionTime: "2023-06-30T10:45:41Z"
    message: shrinking persistent volumes is not supported
    reason: FailedReconcilePVC
    status: "False"
    type: ReconcileSuccess
  defaultUser:
    secretReference:
      keys:
        password: password
        username: username
      name: rabbitmq-default-user
      namespace: xxx
    serviceReference:
      name: rabbitmq
      namespace: xxx
  observedGeneration: 5

2. kubectl apply -f rabbitmq_upgrade.yaml
   rabbitmqcluster.rabbitmq.com/rabbitmq configured
3. kubectl get rmq

NAME       ALLREPLICASREADY   RECONCILESUCCESS   AGE
rabbitmq   True               False              4d3h

4. kubectl get pods | grep rabbitmq ("2/2" because of istio container injection)

rabbitmq-operators-5b8c446968-klq7h                               2/2     Running     1 (18m ago)    18m
rabbitmq-operators-messaging-topology-operator-7bbdf598b7-9wlhc   1/1     Running     1 (9h ago)     4d3h
rabbitmq-server-0                                                 2/2     Running     0              3d22h
rabbitmq-server-1                                                 2/2     Running     0              3d22h
rabbitmq-server-2                                                 2/2     Running     0              3d22h

[xixiangzouyibian]

List all of verions I'm using:

    <rabbitmq.operator.chart.version>3.4.2</rabbitmq.operator.chart.version>
    <rabbitmq.operator.image.version>2.3.0-scratch-r0</rabbitmq.operator.image.version>
    <rabbitmq.msgTopology.operator.image.version>1.11.0-scratch-r0</rabbitmq.msgTopology.operator.image.version>
    <rabbitmq.image.version>3.11.18-debian-11-r0</rabbitmq.image.version>

[ChunyiLyu]

@xixiangzouyibian Checking the operator logs would be useful. If you see that the observed generation is not changed, that generally means that something went wrong with the Operator trying to apply the change.

If the Operator had issues with applying your updates, restart the Operator deployment via kubectl rollout deploy rabbitmq-operators won't help. Please refer to the Operator logs and see if there are any errors.

[xixiangzouyibian]

@ChunyiLyu ok, Thanks again. I have re-installed rabbitmq cluster operator and cluster.
Maybe we can add or enhance webhook to verify the modification if valid to avoid a lot of troubles instead of operator fails in applying changes then blocking subsequent operations.

By the way, the request of adding toggle to rabbitmq cluster operator helm install has been accepted by bitnami:
bitnami/charts#17498

[ChunyiLyu]

Yes we have plans adding validation for some configurations in the future using: https://kubernetes.io/docs/tasks/extend-kubernetes/custom-resources/custom-resource-definitions/#validation-rules. That said, not all configuration related errors can be caught before applying, debugging with Operator logs and status condition will still be needed.

[xixiangzouyibian]

That's great