start.sh script timing out event tho broker seems to be running

[karlem]

BUG REPORT

Expected behavior

Was expecting the start.sh script run smoothly and start container, but it's being timeout all the time even tho broker says that is up and running.

Actual behavior

Process is being killed due timeout error in start.sh script.

Test code

Simple K8S statefulSet

{
  "spec": {
    "updateStrategy": { "type": "RollingUpdate" },
    "template": {
      "spec": {
        "serviceAccountName": "emqtt-serviceaccount",
        "containers": [
          {
            "readinessProbe": {
              "tcpSocket": { "port": 1883 },
              "periodSeconds": 10,
              "initialDelaySeconds": 5
            },
            "ports": [
              { "containerPort": 1883 },
              { "containerPort": 8883 },
              { "containerPort": 8080 },
              { "containerPort": 18083 },
              { "containerPort": 4369 },
              { "containerPort": 4370 },
              { "containerPort": 6369 },
              { "containerPort": 6370 },
              { "containerPort": 6371 },
              { "containerPort": 6372 },
              { "containerPort": 6373 },
              { "containerPort": 6374 },
              { "containerPort": 6375 },
              { "containerPort": 6376 },
              { "containerPort": 6377 },
              { "containerPort": 6378 }
            ],
            "name": "emqtt",
            "image":
              "emqx/emqx:v3.0.0",
            "env": [
              { "value": "k8s", "name": "EMQX_CLUSTER__DISCOVERY" },
              { "value": "emqtt", "name": "EMQX_NAME" },
              {
                "value": "https://kubernetes.default:443",
                "name": "EMQX_CLUSTER__K8S__APISERVER"
              },
              {
                "value": "analytics-chatkit",
                "name": "EMQX_CLUSTER__K8S__NAMESPACE"
              },
              { "value": "emqtt", "name": "EMQX_CLUSTER__K8S__SERVICE_NAME" },
              { "value": "ip", "name": "EMQX_CLUSTER__K8S__ADDRESS_TYPE" },
              { "value": "emqtt", "name": "EMQX_CLUSTER__K8S__APP_NAME" },
              {
                "value": "d0c1680d3abb74a14d7feafb012e08df",
                "name": "EMQX_NODE__COOKIE"
              },
              { "value": "emqtt-cluster", "name": "EMQX_CLUSTER__NAME" },
              { "value": "on", "name": "EMQX_CLUSTER__AUTOHEAL" },
              { "value": "5m", "name": "EMQX_CLUSTER__AUTOCLEAN" },
              {
                "value": "emqx_recon,emqx_retainer,emqx_dashboard",
                "name": "EMQX_LOADED_PLUGINS"
              },
              { "value": "0.0.0.0:8080", "name": "EMQX_LISTENER__API__MGMT" },
              { "value": "200KB", "name": "EMQX_MQTT__MAX_PACKET_SIZE" },
              { "value": "console", "name": "EMQX_LOG__CONSOLE" },
              { "value": "info", "name": "EMQX_LOG__CONSOLE__LEVEL" },
              {
                "value": "1d",
                "name": "EMQX_ZONE__EXTERNAL__SESSION_EXPIRY_INTERVAL"
              }
            ]
          }
        ]
      },
      "metadata": {
        "name": "emqtt",
        "labels": { "app": "emqtt" }
      }
    },
    "serviceName": "emqtt",
    "selector": { "matchLabels": { "app": "emqtt" } },
    "replicas": 2
  },
  "metadata": { "namespace": "abcd", "name": "emqtt" },
  "kind": "StatefulSet",
  "apiVersion": "apps/v1"
}

EMQ version

v3.0.0

Docker version

18.6.1

Startup log

cluster.k8s.address_type=ip
cluster.k8s.address_type=ip
node.max_ports=1048576
cluster.name=emqtt-cluster
cluster.name=emqtt-cluster
listener.tcp.external.acceptors=64
dashboard.default_user.password=verysecret
mqtt.max_packet_size=200KB
zone.external.session_expiry_interval=1d
listener.ssl.external.acceptors=32
node.process_limit=2097152
node.max_ets_tables=2097152
cluster.k8s.service_name=emqtt
cluster.k8s.service_name=emqtt
cluster.discovery=k8s
cluster.discovery=k8s
listener.ws.external.acceptors=16
node.cookie=d0c1680d3abb74a14d7feafb012e08df
node.name=emqtt@10.2.19.54
cluster.k8s.app_name=emqtt
cluster.k8s.app_name=emqtt
cluster.k8s.apiserver=https://kubernetes.default:443
cluster.k8s.apiserver=https://kubernetes.default:443
cluster.k8s.namespace=analytics-chatkit
cluster.k8s.namespace=analytics-chatkit
cluster.autoclean=5m
cluster.autoclean=5m
cluster.autoheal=on
cluster.autoheal=on
EMQX_LOADED_PLUGINS=emqx_recon,emqx_retainer,emqx_dashboard
emqx 3.0 is started successfully!

===== LOGGING STARTED Wed Feb 27 15:05:34 UTC 2019

Exec: /opt/emqx/erts-10.2.1/bin/erlexec -boot /opt/emqx/releases/3.0/emqx -mode embedded -boot_var ERTS_LIB_DIR /opt/emqx/erts-10.2.1/../lib -mnesia dir "/opt/emqx/data/mnesia/emqtt@10.2.19.54" -config /opt/emqx/data/configs/app.2019.02.27.15.05.35.config -args_file /opt/emqx/data/configs/vm.2019.02.27.15.05.35.args -vm_args /opt/emqx/data/configs/vm.2019.02.27.15.05.35.args -- console
Root: /opt/emqx
/opt/emqx
Starting emqx on node emqtt@10.2.19.54
Start http:dashboard listener on 18083 successfully.
Start mqtt:tcp listener on 127.0.0.1:11883 successfully.
Start mqtt:tcp listener on 0.0.0.0:1883 successfully.
Start mqtt:ws listener on 0.0.0.0:8083 successfully.
Start mqtt:ssl listener on 0.0.0.0:8883 successfully.
Start mqtt:wss listener on 0.0.0.0:8084 successfully.
EMQ X Broker v3.0.0 is running now!
Eshell V10.2.1 (abort with ^G)
(emqtt@10.2.19.54)1> ['2019-02-27T15:05:40Z']:waiting emqx
['2019-02-27T15:05:42Z']:waiting emqx
['2019-02-27T15:05:44Z']:waiting emqx
['2019-02-27T15:05:45Z']:waiting emqx
['2019-02-27T15:05:47Z']:waiting emqx
['2019-02-27T15:05:49Z']:waiting emqx
['2019-02-27T15:05:49Z']:timeout error

System

Kubernetes

[karlem]

@zhanghongtong Hello mate.

Any idea what it could be please?

Thank you

[Rory-Z]

Sorry for replying so late, you can try to deploy emqx cluster with helm, you can use our chart