Operator crushes when replicasCount is set in multiple fields

[xinze-zheng]

Operator version: 0.22.2

The description for spec.configuration.clusters.layout.shards.replicasCount depicts it will override the top-level replicas configuration in spec.configuration.clusters.layout.replicasCount. However, when the spec.configuration.clusters.layout.shards.replicasCount is set to be greater than spec.configuration.clusters.layout.replicasCount, the operator crushed.

CR that crushes the operator:

kind: ClickHouseInstallation
metadata:
  name: test-cluster
  namespace: acto-clickhouse
spec:
  configuration:
    clusters:
    - layout:
        replicasCount: 1
        shards:
          - name: shard1
            replicasCount: 2
          - name: shard2
            replicasCount: 2

      name: replicated
    zookeeper:
      nodes:
      - host: zookeeper.zoo3.ns
  defaults:
    templates:
      dataVolumeClaimTemplate: default
      podTemplate: clickhouse:19.6
  templates:
    podTemplates:
    - name: clickhouse:19.6
      spec:
        containers:
        - image: clickhouse/clickhouse-server:22.3
          name: clickhouse-pod
          volumeMounts:
            - name: default2
              mountPath: /var/lib/clickhouse
    volumeClaimTemplates:
    - name: default
      spec:
        accessModes:
        - ReadWriteOnce
        resources:
          requests:
            storage: 500Mi
    - name: default2
      spec:
        accessModes:
        - ReadWriteOnce
        resources:
          requests:
            storage: 200Mi

[Slach]

could you share crash logs from clickhouse-operator container?

[Slach]

@xinze-zheng any news from your side about logs from clickhouse-operator container? We will much appreciate for stack trace with crash.

[xinze-zheng]

Hey, sorry for getting back late. I've pasted the error log below with an attached full log.
The process of reproducing is first apply operator and then apply the CR described above.
Thanks!
operator_log.txt

E0131 16:26:44.645708       1 runtime.go:78] Observed a panic: runtime.boundsError{x:1, y:1, signed:true, code:0x0} (runtime error: index out of range [1] with length 1)
goroutine 334 [running]:
k8s.io/apimachinery/pkg/util/runtime.logPanic({0x182dea0?, 0xc000c948d0})
	/clickhouse-operator/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:74 +0x99
k8s.io/apimachinery/pkg/util/runtime.HandleCrash({0x0, 0x0, 0x0?})
	/clickhouse-operator/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:48 +0x75
panic({0x182dea0, 0xc000c948d0})
	/usr/local/go/src/runtime/panic.go:884 +0x212
github.com/altinity/clickhouse-operator/pkg/apis/clickhouse.altinity.com/v1.(*HostsField).GetOrCreate(...)
	/clickhouse-operator/pkg/apis/clickhouse.altinity.com/v1/type_hosts_field.go:51
github.com/altinity/clickhouse-operator/pkg/apis/clickhouse.altinity.com/v1.(*Cluster).GetOrCreateHost(...)
	/clickhouse-operator/pkg/apis/clickhouse.altinity.com/v1/type_cluster.go:174
github.com/altinity/clickhouse-operator/pkg/model.(*Normalizer).normalizeShardHosts(...)
	/clickhouse-operator/pkg/model/normalizer.go:1632
github.com/altinity/clickhouse-operator/pkg/model.(*Normalizer).normalizeShard(0x450152?, 0xc0002a2300, 0xc000016280, 0x0)
	/clickhouse-operator/pkg/model/normalizer.go:1535 +0x426
github.com/altinity/clickhouse-operator/pkg/model.(*Normalizer).normalizeCluster.func1(0x450152?, 0x8?)
	/clickhouse-operator/pkg/model/normalizer.go:1356 +0x27
github.com/altinity/clickhouse-operator/pkg/apis/clickhouse.altinity.com/v1.(*Cluster).WalkShards(0xc000016280, 0xc001009e50)
	/clickhouse-operator/pkg/apis/clickhouse.altinity.com/v1/type_cluster.go:231 +0xf4
github.com/altinity/clickhouse-operator/pkg/model.(*Normalizer).normalizeCluster(0xc000b16d20, 0xc000b16d20?)
	/clickhouse-operator/pkg/model/normalizer.go:1355 +0x479
github.com/altinity/clickhouse-operator/pkg/model.(*Normalizer).normalizeClusters(0xc000b16d20, {0xc0009a4040?, 0x20?, 0x1783460?})
	/clickhouse-operator/pkg/model/normalizer.go:860 +0xe6
github.com/altinity/clickhouse-operator/pkg/model.(*Normalizer).normalizeConfiguration(0x0?, 0x0?)
	/clickhouse-operator/pkg/model/normalizer.go:522 +0x1d8
github.com/altinity/clickhouse-operator/pkg/model.(*Normalizer).normalize(0xc000b16d20)
	/clickhouse-operator/pkg/model/normalizer.go:245 +0x35c
github.com/altinity/clickhouse-operator/pkg/model.(*Normalizer).CreateTemplatedCHI(0xc000b16d20, 0xc000372fc0, 0xc000d99170)
	/clickhouse-operator/pkg/model/normalizer.go:127 +0x2ff
github.com/altinity/clickhouse-operator/pkg/controller/chi.(*worker).normalize(0xc0001e07e0, 0x0?)
	/clickhouse-operator/pkg/controller/chi/worker.go:323 +0x6e
github.com/altinity/clickhouse-operator/pkg/controller/chi.(*worker).reconcileCHI(0xc0001e07e0, {0x1ba4530, 0xc0001c1540}, 0x0, 0xc000372fc0)
	/clickhouse-operator/pkg/controller/chi/worker-chi-reconciler.go:73 +0x1209
github.com/altinity/clickhouse-operator/pkg/controller/chi.(*worker).updateCHI(0xc0001e07e0, {0x1ba4530, 0xc0001c1540}, 0x0, 0xc000372fc0)
	/clickhouse-operator/pkg/controller/chi/worker.go:462 +0xf7e
github.com/altinity/clickhouse-operator/pkg/controller/chi.(*worker).processReconcileCHI(0x100000003?, {0x1ba4530?, 0xc0001c1540?}, 0x0?)
	/clickhouse-operator/pkg/controller/chi/worker.go:199 +0x4d
github.com/altinity/clickhouse-operator/pkg/controller/chi.(*worker).processItem(0xc0001e07e0, {0x1ba4530, 0xc0001c1540}, {0x1704920?, 0xc0007677a0?})
	/clickhouse-operator/pkg/controller/chi/worker.go:301 +0x45d
github.com/altinity/clickhouse-operator/pkg/controller/chi.(*worker).run(0xc0001e07e0)
	/clickhouse-operator/pkg/controller/chi/worker.go:180 +0x414
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x0?)
	/clickhouse-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:155 +0x3e
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0x0?, {0x1b8c460, 0xc00077e150}, 0x1, 0xc0001200c0)
	/clickhouse-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:156 +0xb6
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc0008dc7b0?, 0x3b9aca00, 0x0, 0x0?, 0x0?)
	/clickhouse-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133 +0x89
k8s.io/apimachinery/pkg/util/wait.Until(0x89c1ea?, 0x0?, 0x0?)
	/clickhouse-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:90 +0x25
created by github.com/altinity/clickhouse-operator/pkg/controller/chi.(*Controller).Run
	/clickhouse-operator/pkg/controller/chi/controller.go:509 +0x79d
panic: runtime error: index out of range [1] with length 1 [recovered]
	panic: runtime error: index out of range [1] with length 1

[sunsingerus]

Please check latest 0.23.3 build.
Fixed.

[Slach]

@sunsingerus is it fixed in 0.23.3?

[alex-zaitsev]

Fixed in https://github.com/Altinity/clickhouse-operator/releases/tag/release-0.23.3