KafkaJS is not re-auhtenticating to all brokers

[flucivja]

Describe the bug
When I run kafkajs with multiple brokers I see in logs that it authenticated to all of them successfuly

[KafkaJS] [SASLOAuthBearerAuthenticator][DEBUG] - Authenticate with SASL OAUTHBEARER kafka-1.flucivja.local:9094
[KafkaJS] [SASLOAuthBearerAuthenticator][DEBUG] - SASL OAUTHBEARER authentication successful kafka-1.flucivja.local:9094

[KafkaJS] [SASLOAuthBearerAuthenticator][DEBUG] - Authenticate with SASL OAUTHBEARER kafka-2.flucivja.local:9094
[KafkaJS] [SASLOAuthBearerAuthenticator][DEBUG] - SASL OAUTHBEARER authentication successful kafka-2.flucivja.local:9094

then I see it succesfully receving topics and when reauthenticationThreshold is hit then re-authentication happes but only to first broker

[KafkaJS] [SASLOAuthBearerAuthenticator][DEBUG] - Authenticate with SASL OAUTHBEARER kafka-1.flucivja.local:9094
[KafkaJS] [SASLOAuthBearerAuthenticator][DEBUG] - SASL OAUTHBEARER authentication successful kafka-1.flucivja.local:9094

then it is working for a while and suddenly I got this error where it is clearly visible that connection was closed by second broker because client didn't re-authenticate to it.

[KafkaJS] [Connection][DEBUG] - Kafka server has closed connection kafka-2.flucivja.local:9094
[KafkaJS] [Connection][DEBUG] - disconnecting...
[KafkaJS] [Connection][DEBUG] - disconnected
[KafkaJS] [Runner][DEBUG] - Error while fetching data, trying again...
[KafkaJS] [ConsumerGroup][DEBUG] - Fetching from 1 partitions for 1 out of 1 topics
 [KafkaJS] [Connection][DEBUG] - Request Fetch(key: 1, version: 11)
 [KafkaJS] [Connection][DEBUG] - Response Fetch(key: 1, version: 11)
 [KafkaJS] [Runner][DEBUG] - Error while fetching data, trying again...
... repeated several times
[KafkaJS] [Consumer][ERROR] - Crash: KafkaJSNumberOfRetriesExceeded: Not connected
[KafkaJS] [Runner][DEBUG] - stop consumer group
[KafkaJS] [Consumer][INFO] - Stopped
[KafkaJS] [Consumer][DEBUG] - consumer has stopped, disconnecting
[KafkaJS] [Connection][DEBUG] - disconnecting...
[KafkaJS] [Connection][DEBUG] - disconnected
[KafkaJS] [Consumer][ERROR] - Restarting the consumer
...then consumer is restarted and same happens every 5 minutes

To Reproduce
You need to run KafkaJS with multiple brokers with SASL OAUTHBEARER auth and server brokers must be configured that client needs to reauthenticate every e.g. 5 minutes.

Expected behavior
KafkaJS re-authenticates to all connected brokers

Environment:

• Mac OS and also Linux
• KafkaJS version 1.15.0
• NodeJS version 14

[flucivja]

I found possible root cause:

• it connects to consumer group leader and coordinator
• it does re-authentication to group leader but not to coordinator
• it runs heartbeat requests only to coordinator
• when session expires, coordinator disconnects consumer and re-connects again

Possible fix: re-authenticate before heartbeat when needed. I created PR #1241 with this fix which runs ok and without any errors within my environment.

[giulliano-bueno]

Great contribution, @flucivja. Thanks.