Device MQTT does not start, is unstable and restarts multiple times

[M0hanrajp]

[ Issue ]

Device-mqtt does not start, always restarts multiple times, not able to use the service.

[ Description ]

From the provided docker-compose.yml, only run docker-compose up -d & multiple failures are observed which are captured in the logs attached.

Few of the error messages included below:

app=device-mqtt source=validation.go:59 msg="An error occurred while performing a 'Subscribe' operation: connection lost before Subscribe completed"
app=device-mqtt source=bootstrap.go:146 msg="Un-Registering service from the Registry"
app=device-mqtt source=config.go:669 msg="Watching for 'MQTTBrokerInfo/Writable' configuration changes has stopped"
app=device-mqtt source=config.go:790 msg="Watching for 'Writable' configuration changes has stopped"
app=device-mqtt source=httpserver.go:146 msg="Web server shut down"
app=device-mqtt source=driver.go:223 msg="driver is stopping, disconnect the MQTT conn"
app=device-mqtt source=bootstrap.go:27 msg="Device Service bootstrapping failed

But sometimes there was segmentation violation:

edgex-device-mqtt  | level=INFO ts=2024-09-03T06:05:49.196442146Z app=device-mqtt source=driver.go:223 msg="driver is stopping, disconnect the MQTT conn"
edgex-device-mqtt  | panic: runtime error: invalid memory address or nil pointer dereference
edgex-device-mqtt  | [signal SIGSEGV: segmentation violation code=0x1 addr=0x30 pc=0x7b94f0]

EdgeX community please help check why there are multiple errors for device mqtt during starting of the service, service is not usable.

Thanks.

device-mqtt-errors-logs.txt

docker-compose.txt

[cloudxxx8]

could you try to check the config from Consul is expected, under edgex/v3/device-mqtt/MQTTBrokerInfo/Host?
it looks like your device-mqtt can't connect to the broker properly.

[M0hanrajp]

@cloudxxx8, Thanks for your response.

Sorry, I did not understand check the config from Consul is expected, under edgex/v3/device-mqtt/MQTTBrokerInfo/Host?

do you mean the following ?

curl http://localhost:59982/api/v3/config | json_pp

{
   "apiVersion" : "v3",
   "config" : {
      "Clients" : {
         "core-metadata" : {
            "Host" : "edgex-core-metadata",
            "Port" : 59881,
            "Protocol" : "http",
            "UseMessageBus" : false
         }
      },
      "CustomConfiguration" : {
         "MQTTBrokerInfo" : {
            "AuthMode" : "none",
            "ClientId" : "device-mqtt",
            "ConnEstablishingRetry" : 10,
            "ConnRetryWaitTime" : 5,
            "CredentialsName" : "credentials",
            "CredentialsRetryTime" : 120,
            "CredentialsRetryWait" : 1,
            "Host" : "edgex-mqtt-broker",
            "IncomingTopic" : "incoming/data/#",
            "KeepAlive" : 3600,
            "Port" : 1883,
            "Qos" : 0,
            "ResponseTopic" : "command/response/#",
            "Schema" : "tcp",
            "Writable" : {
               "ResponseFetchInterval" : 500
            }
         }
      },
      "Device" : {
         "AsyncBufferSize" : 16,
         "DataTransform" : true,
         "DevicesDir" : "./res/devices",
         "Discovery" : {
            "Enabled" : false,
            "Interval" : "30s"
         },
         "EnableAsyncReadings" : true,
         "Labels" : [
            "[]"
         ],
         "MaxCmdOps" : 128,
         "MaxCmdValueLen" : 256,
         "ProfilesDir" : "./res/profiles",
         "ProvisionWatchersDir" : ""
      },
      "Driver" : null,
      "MaxEventSize" : 0,
      "MessageBus" : {
         "AuthMode" : "none",
         "BaseTopicPrefix" : "edgex",
         "Disabled" : false,
         "Host" : "edgex-mqtt-broker",
         "Optional" : {
            "AutoProvision" : "true",
            "AutoReconnect" : "true",
            "ClientId" : "device-mqtt",
            "ConnectTimeout" : "5",
            "DefaultPubRetryAttempts" : "2",
            "Deliver" : "new",
            "Durable" : "",
            "Format" : "nats",
            "KeepAlive" : "10",
            "Qos" : "0",
            "QueueGroup" : "",
            "Retained" : "false",
            "RetryOnFailedConnect" : "true",
            "SkipCertVerify" : "false",
            "Subject" : "edgex/#"
         },
         "Port" : 1883,
         "Protocol" : "tcp",
         "SecretName" : "redisdb",
         "Type" : "mqtt"
      },
      "Registry" : {
         "Host" : "edgex-core-consul",
         "Port" : 8500,
         "Type" : "consul"
      },
      "Service" : {
         "CORSConfiguration" : {
            "CORSAllowCredentials" : false,
            "CORSAllowedHeaders" : "Authorization, Accept, Accept-Language, Content-Language, Content-Type, X-Correlation-ID",
            "CORSAllowedMethods" : "GET, POST, PUT, PATCH, DELETE",
            "CORSAllowedOrigin" : "https://localhost",
            "CORSExposeHeaders" : "Cache-Control, Content-Language, Content-Length, Content-Type, Expires, Last-Modified, Pragma, X-Correlation-ID",
            "CORSMaxAge" : 3600,
            "EnableCORS" : false
         },
         "EnableNameFieldEscape" : false,
         "HealthCheckInterval" : "10s",
         "Host" : "edgex-device-mqtt",
         "MaxRequestSize" : 0,
         "MaxResultCount" : 1024,
         "Port" : 59982,
         "RequestTimeout" : "5s",
         "ServerBindAddr" : "",
         "StartupMsg" : "device mqtt started"
      },
      "Writable" : {
         "InsecureSecrets" : {
            "DB" : {
               "SecretData" : {
                  "password" : "",
                  "username" : ""
               },
               "SecretName" : "redisdb"
            },
            "MQTT" : {
               "SecretData" : {
                  "password" : "",
                  "username" : ""
               },
               "SecretName" : "credentials"
            }
         },
         "LogLevel" : "INFO",
         "Reading" : {
            "ReadingUnits" : true
         },
         "Telemetry" : {
            "Interval" : "30s",
            "Metrics" : {
               "EventsSent" : false,
               "LastConnected" : false,
               "ReadingsSent" : false,
               "SecurityConsulTokenDuration" : false,
               "SecurityConsulTokensRequested" : false,
               "SecurityGetSecretDuration" : false,
               "SecurityRuntimeSecretTokenDuration" : false,
               "SecuritySecretsRequested" : false,
               "SecuritySecretsStored" : false
            },
            "Tags" : null
         }
      }
   },
   "serviceName" : "device-mqtt"
}

or did you mean to check from consul the following ?

Please let me know if anything else is neeeded, I'm just using default generated configuration from compose builder i.e. make gen no-secty ds-mqtt mqtt-bus mqtt-verbose

Please note that this issue occurs regularly and only some times the device-mqtt starts properly.

[M0hanrajp]

@cloudxxx8, Please have a look at fail & pass logs for device-mqtt service.

I observed that it takes almost around 10 mins for the service to be up and running and till that it keeps failing.

device-mqtt-errors-logs.txt

device-mqtt-startup-fail.txt

Thanks,

[M0hanrajp]

Please note that when the service runs, the device under the service also encountered multiple errors for that which I have raised discussion #308

[M0hanrajp]

Hello All, please note that #308 is resolved.

[M0hanrajp]

Regarding this issue: Device MQTT does not start, is unstable and restarts multiple times

I think the due to the following reasons device-mqtt keeps failing:

• From the docker compose file in comment 1, the below is device-mqtt configuration:

    environment:
      EDGEX_STARTUP_DURATION: "120"
      EDGEX_SECURITY_SECRET_STORE: "false"
      MQTTBROKERINFO_HOST: edgex-mqtt-broker
      MQTTBROKERINFO_PORT: 1883

• Note in the provided configuration file ( Comment-1), all services are configured with MQTT as the message bus.
• Device-mqtt MQTTBROKERINFO_HOST is set as edgex-mqtt-broker. So when docker compose up -d is done all services are communicating with the single broker.
• It is the reason why device-mqtt keeps failing multiple times during service startup.

The same issue is not observed if redis is used as the message bus for all services, if MQTTBROKERINFO_HOST is provided with external broker url then device-mqtt startup is normal.

@cloudxxx8,

If we generate a compose file make gen no-secty ds-mqtt mqtt-bus then the issue is reproduced.
Can we optimize this part ?

Thanks

[cloudxxx8]

Yes, you have to change MQTTBROKERINFO_HOST , but that's different.
Why do you need MQTT Device Service to subscribe the internal message bus?
The MQTT Device data should be sent into the internal message bus.

[cloudxxx8]

Unless it's a simple test with simulator like we wrote in the document.
The internal message bus can be Redis pub/sub. They should be saperated.

[M0hanrajp]

Yes, you have to change MQTTBROKERINFO_HOST , but that's different. Why do you need MQTT Device Service to subscribe the internal message bus? The MQTT Device data should be sent into the internal message bus.

Actually I do not want device-mqtt service to use internal message bus but if you generate a docker compose file using make gen no-secty ds-mqtt mqtt-bus in compose builder then by default MQTTBROKERINFO_HOST in device-mqtt will be set to edgex-mqtt-broker& all services internally will be using edgex-mqtt-broker as the message bus.

There is no error reported by makefile during this case.

[cloudxxx8]

the Makefile use the internal mqtt broker as the default broker for testing purpose, and users need to replace it with their real broker address.

[M0hanrajp]

the Makefile use the internal mqtt broker as the default broker for testing purpose, and users need to replace it with their real broker address.

Leaving a note here, if anyone encounters the issue, please replace the broker address.
Thanks