Error: Cannot set headers after they are sent to the client

[nflaig]

Describe the bug

When shutting down the beacon node it can happen that there are uncaught exceptions trying to set headers on a response which is already sent / aborted /closed and no longer writable.

[node-1-cl-lodestar] [8460]: Eph 6/0 2.220[node-1-cl-lodestar] �[31merror�[39m: uncaughtException: Cannot set headers after they are sent to the client
Error: Cannot set headers after they are sent to the client
    at new NodeError (node:internal/errors:405:5)
    at ServerResponse.setHeader (node:_http_outgoing:648:11)
    at ServerResponse.writeHead (node:_http_server:381:21)
    at onSendEnd (/home/runner/actions-runner/_work/lodestar/lodestar/node_modules/fastify/lib/reply.js:597:7)
    at onSendHook (/home/runner/actions-runner/_work/lodestar/lodestar/node_modules/fastify/lib/reply.js:530:5)
    at fallbackErrorHandler (/home/runner/actions-runner/_work/lodestar/lodestar/node_modules/fastify/lib/error-handler.js:127:3)
    at handleError (/home/runner/actions-runner/_work/lodestar/lodestar/node_modules/fastify/lib/error-handler.js:61:5)
    at onErrorHook (/home/runner/actions-runner/_work/lodestar/lodestar/node_modules/fastify/lib/reply.js:743:5)
    at Reply.send (/home/runner/actions-runner/_work/lodestar/lodestar/node_modules/fastify/lib/reply.js:133:5)
    at defaultErrorHandler (/home/runner/actions-runner/_work/lodestar/lodestar/node_modules/fastify/lib/error-handler.js:92:9) Cannot set headers after they are sent to the client
Error: Cannot set headers after they are sent to the client

See node-1-cl-lodestar.log from this sim tests run. This error seems to happen consistently in sim tests.

There was a previous issue #5706 and an attempt to fix it #5722 but the error still happens.

Expected behavior

No uncaught exceptions when shutting down the beacon node.

Steps to reproduce

The issue happens consistently in sim tests.

Additional context

This error can also happen while node is running and client aborts

• see Error using Lodestar as a failover #5323

Operating system

Linux

Lodestar version or commit hash

unstable (6a5225f)