Reclaims more old accounts in clean

[brooksprumo]

Problem

In clean_accounts(), if there are old storages and old accounts beyond what is expected, we do not always reclaim them. This is because when scanning the candidates during clean, if the account's latest slot in the slot list is not in the uncleaned_roots list, we don't collect reclaims.

This can happen if there are candidates from old/ancient slots, where their newest entry is in a slot that is not being visited by clean. We would find multiple entries for this candidate in the index, yet determine not to collect reclaims.

Summary of Changes

Collect reclaims on candidates that have slot lists greater than one entry, and if they are all cleanable (i.e. rooted).

[jeffwashington]

this looks correct. This seems like we can explain now why the brute force method wasn't working, too. And, since we were handling duplicates on the index correctly when we were starting with the in-mem index only. This seems correct.

[HaoranYi]

Nice find.
looks correct to me.

[brooksprumo]

I spun up a mnb node with the disk index disabled to see how it performed. Here's some results:

clean always has a lot of work to do on the first iteration. And we see it removing the old slots:

And since we're cleaning up reclaims, shrink can do a better job now too, purging lots of zero lamport accounts:

[HaoranYi]

lgtm

[HaoranYi]

If max_clean_root_inclusive is none, we need to follow the old rule, i.e. slots must be in self.uncleaned_roots...

[HaoranYi]

Recently, we have seen a few clean related issues with different CLI configuration for validators.

I am trying to think carefully about different ways that clean might be broken when the validator is running with different configurations, i.e. w/o disk index; w/o snapshot generation ...

[jeffwashington]

seems like we could get rid of this metric code and stat. or at least make it a fn to simplify the code in this more critical code path to make it easier to understand and verify correct. THis can be done later.

[brooksprumo]

Yep, I agree. I'll save it for later to keep this PR small, to make it simpler to backport.

[mergify]

Backports to the stable branch are to be avoided unless absolutely necessary for fixing bugs, security issues, and perf regressions. Changes intended for backport should be structured such that a minimum effective diff can be committed separately from any refactoring, plumbing, cleanup, etc that are not strictly necessary to achieve the goal. Any of the latter should go only into master and ride the normal stabilization schedule.

[mergify]

Backports to the beta branch are to be avoided unless absolutely necessary for fixing bugs, security issues, and perf regressions. Changes intended for backport should be structured such that a minimum effective diff can be committed separately from any refactoring, plumbing, cleanup, etc that are not strictly necessary to achieve the goal. Any of the latter should go only into master and ride the normal stabilization schedule. Exceptions include CI/metrics changes, CLI improvements and documentation updates on a case by case basis.