Fix chain_subscribeAllHeads not recreating the channel if it dies

[tomaka]

Subscribing to chain_subscribeAllHeads opens a channel of notifications with the runtime service.
If this channel gets too full (i.e. more than 32 notifications in queue), the channel is force-closed.

This typically happens if the finalized block jumps more than 32 blocks at once, or if there are 32 new blocks in a short amount of time, because in that situation we will report one notification per finalized block or per new block.

In theory, the finalized block isn't supposed to jump more than 32 blocks at once and we're not supposed to get 32 new blocks at once. But this tends to happen for example if we lose Internet connectivity for a while then gain it back.

Unfortunately, when the channel is closed, chain_subscribeAllHeads will not reopen a new one, and no new block will ever be reported to the user. The consequence, in practice, is that PolkadotJS will look frozen.
This is what this PR fixes. We now properly reopen a channel. Some blocks might be missed (as this is the entire reason why the channel needs to be closed), but because nothing in the API of chain_subscribeAllHeads exists to handle these kind of situations, this is the best that we can do.

The diff looks big, but what I did is basically move let new_blocks = within the asynchronous task.

[mergify]

Automatically approving tomaka's pull requests. This auto-approval will be removed once more maintainers are active.

[tomaka]

I've also bumped the buffer size from 32 to 64, because ideally we would like to avoid the situation altogether.

[github-actions]

twiggy diff report

Difference in .wasm size before and after this pull request.

---

 Delta Bytes │ Item
─────────────┼─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
       -1998 ┊ <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll::h1d664ca45fed2e25
       +1607 ┊ <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll::h82c907fbbd6e392c
       +1339 ┊ <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll::hd454f9fc6be6e0c5
       -1061 ┊ <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll::h603bbb19fdd11805
        -403 ┊ core::ptr::drop_in_place<core::future::from_generator::GenFuture<smoldot_light_base::json_rpc_service::state_chain::<impl smoldot_light_base::json_rpc_service::Background<smoldot_light_wasm::platform::Platform>>::chain_subscribe_all_heads::{{closure}}>>::h9a47f4047fc2c1a3
        +321 ┊ core::ptr::drop_in_place<core::future::from_generator::GenFuture<smoldot_light_base::json_rpc_service::state_chain::<impl smoldot_light_base::json_rpc_service::Background<smoldot_light_wasm::platform::Platform>>::chain_subscribe_all_heads::{{closure}}::{{closure}}>>::hbdc49d75f6bfdd13
        +242 ┊ core::ptr::drop_in_place<core::future::from_generator::GenFuture<smoldot_light_base::json_rpc_service::state_chain::<impl smoldot_light_base::json_rpc_service::Background<smoldot_light_wasm::platform::Platform>>::chain_subscribe_all_heads::{{closure}}>>::h3f4b9713273867f4
        -154 ┊ core::ptr::drop_in_place<core::future::from_generator::GenFuture<smoldot_light_base::json_rpc_service::state_chain::<impl smoldot_light_base::json_rpc_service::Background<smoldot_light_wasm::platform::Platform>>::chain_subscribe_all_heads::{{closure}}::{{closure}}>>::h2bf110727804ecc9
        +143 ┊ <futures_util::future::future::map::Map<Fut,F> as core::future::future::Future>::poll::hb8f876e9a7120fdb
        -140 ┊ <futures_util::future::future::map::Map<Fut,F> as core::future::future::Future>::poll::h2a82e0e55d61810f
         -85 ┊ futures_util::abortable::Abortable<T>::try_poll::h737f91432fd2581a
         +79 ┊ futures_util::abortable::Abortable<T>::try_poll::h547b2d61b014ec53
         +46 ┊ futures_util::future::future::map::_::<impl futures_util::future::future::map::Map<Fut,F>>::project_replace::h9ed8d64109ee506a
         -46 ┊ futures_util::future::future::map::_::<impl futures_util::future::future::map::Map<Fut,F>>::project_replace::hb1c4da07d5f54b03
         +26 ┊ core::ptr::drop_in_place<futures_util::future::future::Map<futures_util::abortable::Abortable<core::future::from_generator::GenFuture<smoldot_light_base::json_rpc_service::state_chain::<impl smoldot_light_base::json_rpc_service::Background<smoldot_light_wasm::platform::Platform>>::chain_subscribe_all_heads::{{closure}}::{{closure}}>>,smoldot_light_base::json_rpc_service::state_chain::<impl smoldot_light_base::json_rpc_service::Background<smoldot_light_wasm::platform::Platform>>::chain_subscribe_all_heads::{{closure}}::{{closure}}>>::h42075b22d4611af2
         -26 ┊ core::ptr::drop_in_place<futures_util::future::future::Map<futures_util::abortable::Abortable<core::future::from_generator::GenFuture<smoldot_light_base::json_rpc_service::state_chain::<impl smoldot_light_base::json_rpc_service::Background<smoldot_light_wasm::platform::Platform>>::chain_subscribe_all_heads::{{closure}}::{{closure}}>>,smoldot_light_base::json_rpc_service::state_chain::<impl smoldot_light_base::json_rpc_service::Background<smoldot_light_wasm::platform::Platform>>::chain_subscribe_all_heads::{{closure}}::{{closure}}>>::h5ff55e06bb8ef3a3
         -24 ┊ core::ptr::drop_in_place<futures_util::abortable::Abortable<core::future::from_generator::GenFuture<smoldot_light_base::json_rpc_service::state_chain::<impl smoldot_light_base::json_rpc_service::Background<smoldot_light_wasm::platform::Platform>>::chain_subscribe_all_heads::{{closure}}::{{closure}}>>>::h6ce5f372fb016a0b
         +24 ┊ core::ptr::drop_in_place<futures_util::abortable::Abortable<core::future::from_generator::GenFuture<smoldot_light_base::json_rpc_service::state_chain::<impl smoldot_light_base::json_rpc_service::Background<smoldot_light_wasm::platform::Platform>>::chain_subscribe_all_heads::{{closure}}::{{closure}}>>>::h77896ffc2dbf781c
         -16 ┊ data[0]
         -14 ┊ <futures_util::future::future::Map<Fut,F> as core::future::future::Future>::poll::h2decd8f178b7302a
         +14 ┊ ... and 1 more.
        -126 ┊ Σ [21 Total Rows]