web3.js: after fixing issue #25133 app still can freeze

[awojciak]

Problem

Copied from here: solana-labs/solana-web3.js#1177 because I didn't get any response.
App still freezes and there is a lot of these errors:

Code line indicated in error:

For me it seems like there is still some recursion retrying way too many times. Here you can check preview: invariant-labs/webapp#312 - reproduction steps are the same as earlier (Open dev tools console in browser, set network in app to devnet and for example try to swap two tokens (use faucet to get our mock tokens) - then app will freeze and console will be full of these errors).

[steveluscher]

You set up a ton of accountSubscribe subscriptions, with calls like these:

https://github.com/invariant-labs/webapp/blob/a7f9b51eb103e5e46df50ade3dabfc19fdc30007/src/containers/EventsHandlers/market.ts#L173-L185

I counted 271 subscriptions on startup of the app. Interestingly, the devnet RPC drops the connections after it's responded to ~240–250 of them. Sounds like a magic number to me. I bet the RPC drops the connection as soon as it receives 250 subscriptions.

1. The effect of the RPC dropping the socket so often is trashing your app on devnet. You can reconnect and resubscribe for, for instance, signatureSubscribe, but while you were disconnected you will almost have certainly missed the signature confirmation. web3.js: signature confirmation is only as reliable as the websocket is reliable #25436
2. There's probably a race condition either in our code or inside https://github.com/elpheria/rpc-websockets that's causing the connectedness of the socket to fall out of sync with reality. I myself, using your repro here, observed rpc-websockets trying to call send() on this.socket which at the moment was undefined!

[steveluscher]

If I comment out most of the subscriptions that set up hundreds of accountSubscribe subscriptions, I seem to be able to swap happily without the system blowing up.

This isn't to say I don't have work to do to make the socket connection more resilient! It means that I think both of us have work. You, to figure out another way to get the data you need without having to make hundreds of subscriptions, and me to look for bugs in the socket tracker.

[steveluscher]

@brianlong, does this ~250 number sound plausible? Do you know of any code on api.devnet that would kick a subscriber out with a 1013 Please Try Again websocket error if they try to make 250 subscriptions?

[awojciak]

If I comment out most of the subscriptions that set up hundreds of accountSubscribe subscriptions, I seem to be able to swap happily without the system blowing up.

This isn't to say I don't have work to do to make the socket connection more resilient! It means that I think both of us have work. You, to figure out another way to get the data you need without having to make hundreds of subscriptions, and me to look for bugs in the socket tracker.

Ok, thanks for the response, I'm gonna look for better solution. And good luck with looking for bugs!

[steveluscher]

More confirmation that you're getting rate limited with respect to the number of subscriptions/connection you open: https://discord.com/channels/428295358100013066/560496939779620864/978331039581745193

[steveluscher]

@awojciak, are you still having this problem in production? It looks like https://invariant.app is making far fewer accountSubscribe subscriptions these days. Was that your mitigation?

[awojciak]

@steveluscher I've forgot about that issue, hadn't stumbled upon it recently