BrokerNotAvailableError: Broker not available (sendRequest -> ensureBrokerReady)

[vgnanasekaran]

A few Producer.send call fails due to BrokerNotAvailableError: Broker not available (sendRequest -> ensureBrokerReady). The reason for it is, just before the send call, the socket connection (idle) was closed . Please refer to the logs below. At 1:39:42 the broker was connected, idle was false and ready was true. At 1:48:02 (~9 minutes later) socket was closed. At 1:48:03, 1 second after socket was closed, a producer send call was invoked. As the connection to the broker was false, the call resulted in BrokerNotAvailableError: Broker not available (sendRequest -> ensureBrokerReady) failure. Also at the same time 1:48:03 createBroker established a new connection and the subsequent producer send calls go through successfully.

Logs

Fri, 17 Nov 2023 01:39:42 GMT kafka-node:KafkaClient 1.3.6.8:9092|[BrokerWrapper 1.3.6.8:9092 (connected: true) (ready: true) (idle: false) (needAuthentication: false) (authenticated: false)]
 
Fri, 17 Nov 2023 01:48:02 GMT kafka-node:KafkaClient kafka-node-client socket closed 1.3.6.8:9092 (hadError: false) 

Fri, 17 Nov 2023 01:48:03 GMT kafka-node:KafkaClient compressing messages if needed
Fri, 17 Nov 2023 01:48:03 GMT kafka-node:KafkaClient brokerForLeader: object.keys: 1.3.6.4:9092,1.3.6.8:9092,1.3.6.9:9092
Fri, 17 Nov 2023 01:48:03 GMT kafka-node:KafkaClient 1.3.6.4:9092|[BrokerWrapper 1.3.6.4:9092 (connected: true) (ready: true) (idle: false) (needAuthentication: false) (authenticated: false)]
Fri, 17 Nov 2023 01:48:03 GMT kafka-node:KafkaClient 1.3.6.8:9092|[BrokerWrapper 1.3.6.8:9092 (connected: false) (ready: true) (idle: true) (needAuthentication: false) (authenticated: false)]
Fri, 17 Nov 2023 01:48:03 GMT kafka-node:KafkaClient 1.3.6.9:9092|[BrokerWrapper 1.3.6.9:9092 (connected: true) (ready: true) (idle: true) (needAuthentication: false) (authenticated: false)]
Fri, 17 Nov 2023 01:48:03 GMT kafka-node:KafkaClient brokerForLeader: leader:  2
Fri, 17 Nov 2023 01:48:03 GMT kafka-node:KafkaClient ensureBrokerReady: broker after brokerForLeader call:  [BrokerWrapper 1.3.6.8:9092 (connected: false) (ready: true) (idle: true) (needAuthentication: false) (authenticated: false)]
Fri, 17 Nov 2023 01:48:03 GMT kafka-node:KafkaClient kafka-node-client refreshBrokerMetadata()
Fri, 17 Nov 2023 01:48:03 GMT kafka-node:KafkaClient found 2 connected broker(s)
^[[31merror^[[39m: [/client/change]. kafka producer err : BrokerNotAvailableError: Broker not available (sendRequest -> ensureBrokerReady)
Fri, 17 Nov 2023 01:48:03 GMT kafka-node:KafkaClient kafka-node-client updated internal metadata
Fri, 17 Nov 2023 01:48:03 GMT kafka-node:KafkaClient kafka-node-client reconnecting to 1.3.6.8:9092
Fri, 17 Nov 2023 01:48:03 GMT kafka-node:KafkaClient kafka-node-client createBroker 1.3.6.8:9092
Fri, 17 Nov 2023 01:48:03 GMT kafka-node:KafkaClient kafka-node-client sending versions request to 1.3.6.8:9092

Environment

• Node version: v11.10.1
• Kafka-node version: 5.0.0
• Kafka version: kafka_2.12-2.1.1

For specific cases also provide

• Number of Brokers: 3
• Number partitions for topic:10

Checking to see if the below mentioned changes can be made to handle this failure scenario.

In socket.on('close', function (hadError) event handler as a first step remove the broker from brokers list.

If that is done then in brokerForLeader function the below mentioned code will take care of creating new instance

return (
brokers[addr] ||
this.setupBroker(broker.host, broker.port, longpolling, brokers, err => {
if (err) {
this.emit('error', err);
}
})
);

Please share your thoughts about the fix or any other better approach to handle this failure.

[vgnanasekaran]

Xiaoxin Lu @hyperlink can you please check and share your thoughts.