Beacon API events SSE stream unexpected disconnections

[Savid]

Description

When connecting to the /eth/v1/events event stream, clients will get unexpected disconnections from the lighthouse beacon node. Especially on the attestation topic for mainnet while subscribing all subnets.

Version

Rust - 1.69.0 (84c898d65 2023-04-16)
unstable - commit 7456e1e8faac7a581705f8e71b0dc4f09a36ee5c
stable - commit 693886b94176faa4cb450f024696cb69cda2fe58

Present Behaviour

To replicate the current behaviour easily connect via curl;

curl "http://127.0.0.1:5052/eth/v1/events?topics=attestation" -H "accept: text/event-stream"

Eventually you'll get disconnected with the libcurl error;

curl: (18) transfer closed with outstanding read data remaining

If you want to speed this up run the beacon node with --subscribe-all-subnets --import-all-attestations flags on mainnet to have more attestation events published. With a "fast" machine you will get disconnected multiple times a slot.

Expected Behaviour

Should not disconnect unexpectedly.

[michaelsproul]

I think this could be due to the channel filling up, and the stream being terminated by our error handling here:

lighthouse/beacon_node/http_api/src/lib.rs

Lines 3820 to 3822 in dfcb336

	Err(e) => Err(warp_utils::reject::server_sent_event_error(
	format!("{:?}", e),
	)),

The broadcast receiver will return "lag errors" as described here when the channel fills up:

If a value is sent when the channel is at capacity, the oldest value currently held by the channel is released. This frees up space for the new value. Any receiver that has not yet seen the released value will return RecvError::Lagged the next time recv is called.

I think we could map these RecvError::Lagged errors into no-op events, maybe a comment that says ":dropped some events".

We could also set some more generous defaults for the channel capacity, which is currently hardcoded at 16 here:

lighthouse/beacon_node/beacon_chain/src/events.rs

Line 7 in dfcb336

const DEFAULT_CHANNEL_CAPACITY: usize = 16;

Different capacities for different events would probably make sense, and a way to adjust them via the CLI (maybe an integer multiplier that applies to all channel capacities).

[michaelsproul]

Implementation of this fix should happen on top of #4462

[michaelsproul]

WIP PR for initial testing here: #4500

I haven't been able to get it to produce the :error - dropped N messages comment yet, but will run it for a few hours and see what happens

[Savid]

awesome, thanks. Just built an image. Will roll it out to one of our mainnet sentries and let you know how it looks