The number of accounts/ files in TdS and SLP snapshots is too high

[mvines]

Snapshots are taking 1000 slots or worse to unpack and process on TdS/SLP, due to the large number of accounts/ files in them. At this time, TdS has over 4500 files in accounts/, SLP has almost 3000.

This slows down the ability of validators to quickly boot and restart.

[ryoqun]

This is partly affected by this #8148, which causes some unneeded storage entries not to be deleted forever.

[mvines]

This is starting to be a real problem. I'm getting snapshots on TdS and SLP that are from more than 10 minutes ago, which causes new validators to catch up very slowly

[ryoqun]

Status update: I've created a fix locally which reduces the current TdS snapshot into 1/5. PRs are coming shortly after.

[mvines]

oops, this is not fixed!

[ryoqun]

Finally, my almost 1-month worth labor is paying off:

TdS:

SLP:

[mvines]

@ryoqun - I think we can call this issue fixed now right?

[ryoqun]

@mvines yes from operational standpoint!

There is still minor security issue, which I noticed and could be mitigated by more aggressive cleaning implementation. But there are bunch of similar low-priority security issues when deserializing a snapshot, which is tracked by #7167. So, closing this now.