[Bug]: Deploy obcluster failed with latest version

[silentacorn]

Describe the bug

Try deploy a simple oceanbase cluster by ob-operator but always failed as follow error logs while pods shows running already:

{"level":"INFO","ts":"2024-08-01T19:41:02+08:00","msg":"Create observers","controller":"obzone","controllerGroup":"oceanbase.oceanbase.com","controllerKind":"OBZone","OBZone":{"name":"obcluster-1-zone1","namespace":"oceanbase"},"namespace":"oceanbase","name":"obcluster-1-zone1","reconcileID":"fe450abf-de5d-440b-b5ed-e7b6fb85d396"}
{"level":"INFO","ts":"2024-08-01T19:41:02+08:00","msg":"Create observer","controller":"obzone","controllerGroup":"oceanbase.oceanbase.com","controllerKind":"OBZone","OBZone":{"name":"obcluster-1-zone1","namespace":"oceanbase"},"namespace":"oceanbase","name":"obcluster-1-zone1","reconcileID":"fe450abf-de5d-440b-b5ed-e7b6fb85d396","server":"obcluster-1-zone1-272987e9035e"}
{"level":"INFO","ts":"2024-08-01T19:41:02+08:00","msg":"Newly created server, init status","controller":"observer","controllerGroup":"oceanbase.oceanbase.com","controllerKind":"OBServer","OBServer":{"name":"obcluster-1-zone1-272987e9035e","namespace":"oceanbase"},"namespace":"oceanbase","name":"obcluster-1-zone1-272987e9035e","reconcileID":"cab46ab5-f960-4474-be95-2a0dadbdf53a"}
{"level":"INFO","ts":"2024-08-01T19:41:02+08:00","msg":"Create observers","controller":"obzone","controllerGroup":"oceanbase.oceanbase.com","controllerKind":"OBZone","OBZone":{"name":"obcluster-1-zone2","namespace":"oceanbase"},"namespace":"oceanbase","name":"obcluster-1-zone2","reconcileID":"17e81701-0ccd-43ba-b3b2-bf99b6ed0623"}
{"level":"INFO","ts":"2024-08-01T19:41:02+08:00","msg":"Create observer","controller":"obzone","controllerGroup":"oceanbase.oceanbase.com","controllerKind":"OBZone","OBZone":{"name":"obcluster-1-zone2","namespace":"oceanbase"},"namespace":"oceanbase","name":"obcluster-1-zone2","reconcileID":"17e81701-0ccd-43ba-b3b2-bf99b6ed0623","server":"obcluster-1-zone2-c557ffed90ab"}
{"level":"INFO","ts":"2024-08-01T19:41:02+08:00","msg":"Newly created server, init status","controller":"observer","controllerGroup":"oceanbase.oceanbase.com","controllerKind":"OBServer","OBServer":{"name":"obcluster-1-zone2-c557ffed90ab","namespace":"oceanbase"},"namespace":"oceanbase","name":"obcluster-1-zone2-c557ffed90ab","reconcileID":"ec6f9e99-3765-4ac8-897e-f0a00a0a6a9f"}
{"level":"INFO","ts":"2024-08-01T19:41:02+08:00","msg":"Create observers","controller":"obzone","controllerGroup":"oceanbase.oceanbase.com","controllerKind":"OBZone","OBZone":{"name":"obcluster-1-zone3","namespace":"oceanbase"},"namespace":"oceanbase","name":"obcluster-1-zone3","reconcileID":"09711cad-b115-48a5-b497-846cb93840aa"}
{"level":"INFO","ts":"2024-08-01T19:41:02+08:00","msg":"Create observer","controller":"obzone","controllerGroup":"oceanbase.oceanbase.com","controllerKind":"OBZone","OBZone":{"name":"obcluster-1-zone3","namespace":"oceanbase"},"namespace":"oceanbase","name":"obcluster-1-zone3","reconcileID":"09711cad-b115-48a5-b497-846cb93840aa","server":"obcluster-1-zone3-b92a1921f4dd"}
{"level":"INFO","ts":"2024-08-01T19:41:02+08:00","msg":"Newly created server, init status","controller":"observer","controllerGroup":"oceanbase.oceanbase.com","controllerKind":"OBServer","OBServer":{"name":"obcluster-1-zone3-b92a1921f4dd","namespace":"oceanbase"},"namespace":"oceanbase","name":"obcluster-1-zone3-b92a1921f4dd","reconcileID":"e628e425-3d9d-4afd-a21f-83ca582afe83"}
{"level":"INFO","ts":"2024-08-01T19:41:03+08:00","msg":"Create observer when create obcluster","controller":"observer","controllerGroup":"oceanbase.oceanbase.com","controllerKind":"OBServer","OBServer":{"name":"obcluster-1-zone1-272987e9035e","namespace":"oceanbase"},"namespace":"oceanbase","name":"obcluster-1-zone1-272987e9035e","reconcileID":"0eac2fb7-ec9f-4400-a6c3-3433eec52d7d"}
{"level":"INFO","ts":"2024-08-01T19:41:03+08:00","msg":"Create observer when create obcluster","controller":"observer","controllerGroup":"oceanbase.oceanbase.com","controllerKind":"OBServer","OBServer":{"name":"obcluster-1-zone2-c557ffed90ab","namespace":"oceanbase"},"namespace":"oceanbase","name":"obcluster-1-zone2-c557ffed90ab","reconcileID":"a3366ec0-475f-422f-b976-60f4e6d67eb9"}
{"level":"INFO","ts":"2024-08-01T19:41:03+08:00","msg":"Create observer when create obcluster","controller":"observer","controllerGroup":"oceanbase.oceanbase.com","controllerKind":"OBServer","OBServer":{"name":"obcluster-1-zone3-b92a1921f4dd","namespace":"oceanbase"},"namespace":"oceanbase","name":"obcluster-1-zone3-b92a1921f4dd","reconcileID":"34dbd673-0d8c-4272-9a51-77c6e1930ae3"}
{"level":"INFO","ts":"2024-08-01T19:41:10+08:00","msg":"static ip not supported, set empty annotation","controller":"observer","controllerGroup":"oceanbase.oceanbase.com","controllerKind":"OBServer","OBServer":{"name":"obcluster-1-zone1-272987e9035e","namespace":"oceanbase"},"namespace":"oceanbase","name":"obcluster-1-zone1-272987e9035e","reconcileID":"095a8d75-4893-493d-ada8-f6d30f5976b4"}
{"level":"INFO","ts":"2024-08-01T19:41:10+08:00","msg":"static ip not supported, set empty annotation","controller":"observer","controllerGroup":"oceanbase.oceanbase.com","controllerKind":"OBServer","OBServer":{"name":"obcluster-1-zone2-c557ffed90ab","namespace":"oceanbase"},"namespace":"oceanbase","name":"obcluster-1-zone2-c557ffed90ab","reconcileID":"03fccac2-3b74-457b-b2ca-46109752f957"}
{"level":"INFO","ts":"2024-08-01T19:41:10+08:00","msg":"static ip not supported, set empty annotation","controller":"observer","controllerGroup":"oceanbase.oceanbase.com","controllerKind":"OBServer","OBServer":{"name":"obcluster-1-zone3-b92a1921f4dd","namespace":"oceanbase"},"namespace":"oceanbase","name":"obcluster-1-zone3-b92a1921f4dd","reconcileID":"ad92b607-f1ed-412e-9e69-dda2adf0e4c0"}
{"level":"INFO","ts":"2024-08-01T19:41:23+08:00","msg":"Pod is ready","controller":"observer","controllerGroup":"oceanbase.oceanbase.com","controllerKind":"OBServer","OBServer":{"name":"obcluster-1-zone1-272987e9035e","namespace":"oceanbase"},"namespace":"oceanbase","name":"obcluster-1-zone1-272987e9035e","reconcileID":"0033dca3-0681-44ac-b720-9a3b0bcc2843"}
{"level":"INFO","ts":"2024-08-01T19:41:25+08:00","msg":"Pod is ready","controller":"observer","controllerGroup":"oceanbase.oceanbase.com","controllerKind":"OBServer","OBServer":{"name":"obcluster-1-zone2-c557ffed90ab","namespace":"oceanbase"},"namespace":"oceanbase","name":"obcluster-1-zone2-c557ffed90ab","reconcileID":"9dd82dbe-0c90-4a08-ba8d-808ea5bb7133"}
{"level":"INFO","ts":"2024-08-01T19:41:33+08:00","msg":"Pod is ready","controller":"observer","controllerGroup":"oceanbase.oceanbase.com","controllerKind":"OBServer","OBServer":{"name":"obcluster-1-zone3-b92a1921f4dd","namespace":"oceanbase"},"namespace":"oceanbase","name":"obcluster-1-zone3-b92a1921f4dd","reconcileID":"c5b30b83-a796-4e88-9ed9-6ede5e9e1252"}
{"level":"ERROR","ts":"2024-08-01T19:42:12+08:00","msg":"get oceanbase operation manager failed","controller":"obcluster","controllerGroup":"oceanbase.oceanbase.com","controllerKind":"OBCluster","OBCluster":{"name":"obcluster","namespace":"oceanbase"},"namespace":"oceanbase","name":"obcluster","reconcileID":"44f8cc7e-6701-4fa7-8f20-788b32676d0f","error":"Can not get oceanbase operation manager of obcluster obcluster after checked all server","errorVerbose":"Can not get oceanbase operation manager of obcluster obcluster after checked all server\ngh.neting.cc/oceanbase/ob-operator/internal/resource/utils.getSysClient\n\t/workspace/internal/resource/utils/util.go:138\ngh.neting.cc/oceanbase/ob-operator/internal/resource/utils.GetSysOperationClient\n\t/workspace/internal/resource/utils/util.go:53\ngh.neting.cc/oceanbase/ob-operator/internal/resource/obcluster.(*OBClusterManager).getOceanbaseOperationManager\n\t/workspace/internal/resource/obcluster/obcluster_task.go:266\ngh.neting.cc/oceanbase/ob-operator/internal/resource/obcluster.(*OBClusterManager).Bootstrap\n\t/workspace/internal/resource/obcluster/obcluster_task.go:281\ngh.neting.cc/oceanbase/ob-operator/pkg/task.runTask\n\t/workspace/pkg/task/task_manager.go:63\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1598","stacktrace":"github.com/oceanbase/ob-operator/internal/resource/obcluster.(*OBClusterManager).Bootstrap\n\t/workspace/internal/resource/obcluster/obcluster_task.go:291\ngh.neting.cc/oceanbase/ob-operator/pkg/task.runTask\n\t/workspace/pkg/task/task_manager.go:63"}

The obcluster.yaml file:
apiVersion: oceanbase.oceanbase.com/v1alpha1 kind: OBCluster metadata: name: obcluster namespace: oceanbase spec: clusterName: obcluster clusterId: 1 serviceAccount: "default" userSecrets: root: root-password proxyro: proxyro-password monitor: monitor-password operator: operator-password topology: - zone: zone1 replica: 1 nodeSelector: zone: hp1 - zone: zone2 replica: 1 nodeSelector: zone: kos - zone: zone3 replica: 1 nodeSelector: zone: uos observer: image: oceanbase/oceanbase-cloud-native:4.2.1.6-106000012024042515 resource: cpu: 4 memory: 16Gi storage: dataStorage: storageClass: rook-ceph-block size: 1000Gi redoLogStorage: storageClass: rook-ceph-block size: 500Gi logStorage: storageClass: rook-ceph-block size: 200Gi monitor: image: oceanbase/obagent:4.2.1-100000092023101717 resource: cpu: 1 memory: 1Gi

Environment

ob-operator:2.2.2
observer:
image: oceanbase/oceanbase-cloud-native:4.2.1.6-106000012024042515
monitor:
image: oceanbase/obagent:4.2.1-100000092023101717
k8s: v1.28.2

Fast reproduce steps

1. deploy ob-operator 2.2.2 with helm in oceanbase-system namespace
2. create security
3. deploy oceanbase by apply -f obcluster.yaml in oceanbase namespace

Expected behavior

No response

Actual behavior

kubectl -n oceanbase get obcluster
NAME STATUS AGE
obcluster failed 14m

kubectl -n oceanbase describe obcluster obcluster
...
Events:
Type Reason Age From Message

---

Normal 13m obcluster-controller newly created cluster, init status
Normal 13m obcluster-controller Create obzone obcluster-1-zone1 successfully
Normal 13m obcluster-controller Create obzone obcluster-1-zone2 successfully
Normal 13m obcluster-controller Create obzone obcluster-1-zone3 successfully
Warning Task failed 12m obcluster-controller get oceanbase operation manager: Can not get oceanbase operation manager of obcluster obcluster after checked all server

Additional context

No response

[chris-sun-star]

Pod turned to running status only indicates that the observer process is up and listens the desired part, but there's still extra steps to successfully bootstrap an OceanBase cluster.
The events show that ob-operator failed to obtain a connection after tried all the observers, and the time was about 1 minute after obzone was created, maybe observer is not ready to handle connection cause it still has to do some initializing procedure after the process is started.
You may try to connect to the observer using mysql client using root user and without password, see if the observer is connectable, and get the first log file of observer, the file may be observer.log.20.... We've encountered some similar issues due to slow IO, you may also check the disk iops as well.