Beacon chain lags behind frequently when running the latest tree-states branch

[jimmygchen]

Description

After merging in latest unstable to tree-states in PR #4514 , the BN running tree-states version seems to lag behind quite frequently for some reasons.

[jimmygchen]

looks like gossip_block processing is possibly broken, each processing is taking up to 10s. I think this might be causing the queues to fill up (and hence chain lagging behind) and the high memory usage.

[jimmygchen]

I'm not able to reproduce this when running locally, I've tried the following 3 versions

• 8423e9f: Latest tree-states branch
• 5d2063d: One commit prior to the unstable merge
• 6954de6: Alpha release version v4.2.990-exp

I haven't seen any significant difference in terms of performance and lagging behind, but maybe i didn't keep them up long enough (I ran each version for about 15 mins)

I'll try doing a fresh deployment to the same node to see if the problem persists.

[jimmygchen]

Might be worth trying this version as well to narrow down the possible changes that broke it:
#4458

[jimmygchen]

Our tree states node seems to be back to normal now, 8 hours after rolling back to the alpha release version 🤔

The slowness seems to be correlated to the frequency of State diff applied log occurrences, which wasn't showing up at all until I upgraded to the latest version, and now stopped after I rolled back to the alpha release version.

[jimmygchen]

Re-deployed latest tree-state branch again and looks fine - I think there's still something fishy with the new version though, because the old version seems to be able to recover from the slowness, but the performance with new version just keep degrading over time.

Will leave it for a bit and keep monitoring. (Will try to reproduce with the attestation rewards API call later if it all looks good)

[jimmygchen]

The node running the latest tree-states branch version degraded over time :/ I think we need to take a deeper look.
Latest alpha release version is fine and not affected.

[michaelsproul]

I guess the state diffs being applied are for historic states. It's possible this is a manifestation of the HTTP death spiral issue in combination with some slight slowdown in the new version of tree-states vs the previous alpha. I think the fact we recently pointed the checkpointz server at this node might be part of the reason we've seen a change.

We could try bumping up the hdiff buffer cache size to see if that helps take some of the time off the calls. Grepping the logs for "diff" should show long each diff is taking to be applied and whether the buffer cache is being hit. We probably also want to tweak the cache algorithm so that it prefers to keep "deeper" (more general) diffs, rather than ones for e.g. single epochs.

It will also be interesting to test the HTTP API fix once that's applied in unstable.

[michaelsproul]

Something else it could be is the database migration frequency. The alpha has it set to 4, but the latest version has it set to 1. This was an attempt to reduce cache misses (by making it more likely the new finalized state is in the cache) but it may have backfired

[jimmygchen]

Thanks, would be interesting to try changing the database migration frequency and see how it peforms. It's worth noting that the state-transition test benchmark (block processing times) hasn't changed much since the last time we looked at it together, so I'm inclined to agree that it could be related to the HTTP death spiral issue.

[michaelsproul]

I think I found part of the problem in #4573. I'll deploy that now and see if it shows much improvement. I haven't seen the crazy 20GB+ memory usage running the latest tree-states yet, but it does look a lot bumpier than v4.2.990-exp.

[jimmygchen]

This has been fixed in #4576, closing. 🎉