BB-624 Retry connection to kafka if backbeat processes started before

[BourgoisMickael]

• Suppress AWS SDK warning in stderr
• Improve container image size from 2.34GB to 1.55GB (layer chown + dockerignore)

REPOSITORY                   TAG                         IMAGE ID       SIZE         BLOB SIZE
scality/backbeat             7.70.15.rc1-nodesvc-base    ea2e98003485    1.6 GiB      491.7 MiB
scality/backbeat             9.4.0.0                     cf4ecf197272    2.3 GiB      1.8 GiB

• Backport / cherrypick some fixes from zenko to crash exit after a timeout on connection error with kafka client (for CRR, Lifecycle, Bucket notification). Even for bucket notification destination.
• Prevent ProbeServer crash on startup by fixing metric handling when the consumer is not set up and delaying it after the queue components are started.

New behavior: components will try to connect for 60s instead of 30s and crash exit and restart if kafka client can't connect with error like:

{"name":"Backbeat:QueueProcessor:task","time":1730775546353,"method":"QueueProcessor::task","error":{"origin":"local","message":"broker transport failure","code":-195,"errno":-195,"stack":"Error: Local: Broker transport failure\n    at Function.createLibrdkafkaError [as create] (/home/scality/backbeat/node_modules/node-rdkafka/lib/error.js:454:10)\n    at /home/scality/backbeat/node_modules/node-rdkafka/lib/client.js:350:28"},"level":"error","message":"error during queue processor initialization","hostname":"MDM-RING-46789-store-1","pid":594}

{"name":"Backbeat:QueueProcessor:task","time":1730776197182,"error":{"origin":"local","message":"timed out","code":-185,"errno":-185,"stack":"Error: Local: Timed out\n    at Function.createLibrdkafkaError [as create] (/home/scality/backbeat/node_modules/node-rdkafka/lib/error.js:454:10)\n    at /home/scality/backbeat/node_modules/node-rdkafka/lib/client.js:350:28"},"method":"MetricsProducer::setupProducer","level":"error","message":"error starting metrics producer for queue processor","hostname":"MDM-RING-46789-store-2","pid":146}

[bert-e]

Hello bourgoismickael,

My role is to assist you with the merge of this
pull request. Please type @bert-e help to get information
on this process, or consult the user documentation.

Available options

name	description	privileged	authored
/after_pull_request	Wait for the given pull request id to be merged before continuing with the current one.
/bypass_author_approval	Bypass the pull request author's approval	⭐
/bypass_build_status	Bypass the build and test status	⭐
/bypass_commit_size	Bypass the check on the size of the changeset TBA	⭐
/bypass_incompatible_branch	Bypass the check on the source branch prefix	⭐
/bypass_jira_check	Bypass the Jira issue check	⭐
/bypass_peer_approval	Bypass the pull request peers' approval	⭐
/bypass_leader_approval	Bypass the pull request leaders' approval	⭐
/approve	Instruct Bert-E that the author has approved the pull request.		✍️
/create_pull_requests	Allow the creation of integration pull requests.
/create_integration_branches	Allow the creation of integration branches.
/no_octopus	Prevent Wall-E from doing any octopus merge and use multiple consecutive merge instead
/unanimity	Change review acceptance criteria from one reviewer at least to all reviewers
/wait	Instruct Bert-E not to run until further notice.

Available commands

name	description	privileged
/help	Print Bert-E's manual in the pull request.
/status	Print Bert-E's current status in the pull request TBA
/clear	Remove all comments from Bert-E from the history TBA
/retry	Re-start a fresh build TBA
/build	Re-start a fresh build TBA
/force_reset	Delete integration branches & pull requests, and restart merge process from the beginning.
/reset	Try to remove integration branches unless there are commits on them which do not appear on the source branch.

Status report is not available.

[nicolas2bert]

Suggestions:

• add tests for "Prevent ProbeServer crash on startup by delaying it after the queue components are started".
• add tests for Consumer service crashes in the event of a consumer error?

[bert-e]

Waiting for approval

The following approvals are needed before I can proceed with the merge:

• the author

• 2 peers

The following options are set: create_pull_requests, create_integration_branches

[BourgoisMickael]

Suggestions:

• add tests for "Prevent ProbeServer crash on startup by delaying it after the queue components are started".
• add tests for Consumer service crashes in the event of a consumer error?

@nicolas2bert
I added a simple test for the crash on probing, but for testing the whole task file that stops on error it's not easily testable, I'll try to use vm module but it might take some time.

[BourgoisMickael]

About this it looks like on zenko it could crash with a "callback already called" if destination Kafka is not up on startup

[bert-e]

Waiting for approval

The following approvals are needed before I can proceed with the merge:

• the author

• 2 peers

The following options are set: create_pull_requests

[BourgoisMickael]

Here is an Integration e2e run: https://github.com/scality/Integration/actions/runs/11699734386/job/32582300296

It should not trigger any exit implemented in this PR as all kafka dependencies are already UP in those tests

[BourgoisMickael]

/create_integration_branches

[BourgoisMickael]

/approve

[bert-e]

I have successfully merged the changeset of this pull request
into targetted development branches:

• ✔️ development/7.10

• ✔️ development/7.70

• ✔️ development/8.5

• ✔️ development/8.6

• ✔️ development/8.7

The following branches have NOT changed:

• development/7.4

Please check the status of the associated issue BB-624.

Goodbye bourgoismickael.

The following options are set: approve, create_integration_branches