revive sub-du in utils/frame

[kianenigma]

EDIT: There's already a PR open to build upon, with some comments.

I am suggesting that we bring sub-du, originally implemented with rust and using an older version of metadata here:

https://github.com/paritytech/substrate-debug-kit/blob/master/sub-du/src/main.rs

back to substrate. This can be a useful tool to report the state size of any substrate based chain.

Notes:

1. There is an argument that this could easily be implemented in TS as well, which is correct, but it is also not too much of a back bend to do it in rust and it is a good application to use jsonrpsee client.
2. There is argument to not bring this to substrate, but I disagree with this. Managing dependencies is a PITA and we already have a mono-repo, with multiple aux tools like chain-spec-builder and subkey here so that we prevent breaking changes.

cc @ggwpez I was told you would also be interested in this, perhaps for try-runtime diagnosis.

Example output from 2020:

60  M ├─┬ polkadot
55  M │ │─┬ Staking
28  M │ │ │─ ErasStakers => Map(29,915,344 bytes, 17691 keys)
20  M │ │ │─ ErasStakersClipped => Map(21,571,602 bytes, 17691 keys)
3   M │ │ │─ Ledger => Map(3,830,167 bytes, 10881 keys)
2   M │ │ │─ Nominators => Map(2,235,784 bytes, 8476 keys)
622 K │ │ │─ ErasRewardPoints => Map(637,242 bytes, 85 keys)
340 K │ │ │─ Bonded => Map(348,192 bytes, 10881 keys)
65  K │ │ │─ ErasValidatorPrefs => Map(67,533 bytes, 17691 keys)
17  K │ │ │─ Payee => Map(18,177 bytes, 10881 keys)
2   K │ │ │─ SpanSlash => Map(2,688 bytes, 84 keys)
2   K │ │ │─ Validators => Map(2,173 bytes, 580 keys)
1   K │ │ │─ SlashingSpans => Map(1,795 bytes, 103 keys)
1   K │ │ │─ ErasTotalStake => Map(1,360 bytes, 85 keys)
1   K │ │ │─ ErasValidatorReward => Map(1,344 bytes, 84 keys)
340 B │ │ │─ ErasStartSessionIndex => Map(340 bytes, 85 keys)
233 B │ │ │─ BondedEras => Value(233 bytes)
16  B │ │ │─ CanceledSlashPayout => Value(16 bytes)
13  B │ │ │─ ActiveEra => Value(13 bytes)
4   B │ │ │─ EarliestUnappliedSlash => Value(4 bytes)
4   B │ │ │─ SlashRewardFraction => Value(4 bytes)
4   B │ │ │─ CurrentEra => Value(4 bytes)
4   B │ │ │─ MinimumValidatorCount => Value(4 bytes)
4   B │ │ │─ ValidatorCount => Value(4 bytes)
4   B │ │ │─ HistoryDepth => Value(4 bytes)
1   B │ │ │─ StorageVersion => Value(1 bytes)
1   B │ │ │─ IsCurrentSessionFinal => Value(1 bytes)
1   B │ │ │─ EraElectionStatus => Value(1 bytes)
1   B │ │ │─ Invulnerables => Value(1 bytes)
0   B │ │ │─ QueuedScore => Value(0 bytes)
0   B │ │ │─ QueuedElected => Value(0 bytes)
0   B │ │ │─ SnapshotNominators => Value(0 bytes)
0   B │ │ │─ SnapshotValidators => Value(0 bytes)
0   B │ │ │─ NominatorSlashInEra => Map(0 bytes, 0 keys)
0   B │ │ │─ ValidatorSlashInEra => Map(0 bytes, 0 keys)
0   B │ │ │─ UnappliedSlashes => Map(0 bytes, 0 keys)
0   B │ │ │─ ForceEra => Value(0 bytes)
3   M │ │─┬ System
3   M │ │ │─ Account => Map(3,580,632 bytes, 49731 keys)
75  K │ │ │─ BlockHash => Map(76,832 bytes, 2401 keys)
19  B │ │ │─ Events => Value(19 bytes)
16  B │ │ │─ BlockWeight => Value(16 bytes)
10  B │ │ │─ LastRuntimeUpgrade => Value(10 bytes)
4   B │ │ │─ EventCount => Value(4 bytes)
1   B │ │ │─ UpgradedToU32RefCount => Value(1 bytes)
0   B │ │ │─ ExecutionPhase => Value(0 bytes)
0   B │ │ │─ EventTopics => Map(0 bytes, 0 keys)
0   B │ │ │─ Digest => Value(0 bytes)
0   B │ │ │─ ExtrinsicsRoot => Value(0 bytes)
0   B │ │ │─ ParentHash => Value(0 bytes)
0   B │ │ │─ Number => Value(0 bytes)
0   B │ │ │─ ExtrinsicData => Map(0 bytes, 0 keys)
0   B │ │ │─ AllExtrinsicsLen => Value(0 bytes)
0   B │ │ │─ ExtrinsicCount => Value(0 bytes)
324 K │ │─┬ Balances
324 K │ │ │─ Locks => Map(332,483 bytes, 11358 keys)
16  B │ │ │─ TotalIssuance => Value(16 bytes)
1   B │ │ │─ StorageVersion => Value(1 bytes)
0   B │ │ │─ Account => Map(0 bytes, 0 keys)
285 K │ │─┬ Session
116 K │ │ │─ KeyOwner => Map(119,520 bytes, 3735 keys)
116 K │ │ │─ NextKeys => Map(119,200 bytes, 745 keys)
45  K │ │ │─ QueuedKeys => Value(46,082 bytes)
7   K │ │ │─ Validators => Value(7,682 bytes)
4   B │ │ │─ CurrentIndex => Value(4 bytes)
1   B │ │ │─ QueuedChanged => Value(1 bytes)
0   B │ │ │─ DisabledValidators => Value(0 bytes)
203 K │ │─┬ Indices
203 K │ │ │─ Accounts => Map(208,348 bytes, 4252 keys)
68  K │ │─┬ Offences
57  K │ │ │─ Reports => Map(59,237 bytes, 161 keys)
5   K │ │ │─ ReportsByKindIndex => Map(5,839 bytes, 2 keys)
5   K │ │ │─ ConcurrentReportsIndex => Map(5,223 bytes, 71 keys)
1   B │ │ │─ DeferredOffences => Value(1 bytes)
54  K │ │─┬ Democracy
49  K │ │ │─ VotingOf => Map(51,172 bytes, 707 keys)
1   K │ │ │─ Preimages => Map(1,827 bytes, 17 keys)
1   K │ │ │─ DepositOf => Map(1,608 bytes, 8 keys)
545 B │ │ │─ PublicProps => Value(545 bytes)
156 B │ │ │─ ReferendumInfoOf => Map(156 bytes, 12 keys)
4   B │ │ │─ LowestUnbaked => Value(4 bytes)
4   B │ │ │─ ReferendumCount => Value(4 bytes)
4   B │ │ │─ PublicPropCount => Value(4 bytes)
1   B │ │ │─ StorageVersion => Value(1 bytes)
1   B │ │ │─ Cancellations => Map(1 bytes, 1 keys)
0   B │ │ │─ Blacklist => Map(0 bytes, 0 keys)
0   B │ │ │─ NextExternal => Value(0 bytes)
0   B │ │ │─ LastTabledWasExternal => Value(0 bytes)
0   B │ │ │─ Locks => Map(0 bytes, 0 keys)
47  K │ │─┬ Vesting
47  K │ │ │─ Vesting => Map(48,492 bytes, 1347 keys)
45  K │ │─┬ Identity
22  K │ │ │─ IdentityOf => Map(23,482 bytes, 258 keys)
11  K │ │ │─ SuperOf => Map(11,967 bytes, 315 keys)
10  K │ │ │─ SubsOf => Map(10,963 bytes, 50 keys)
115 B │ │ │─ Registrars => Value(115 bytes)
29  K │ │─┬ ElectionsPhragmen
27  K │ │ │─ Voting => Map(28,145 bytes, 193 keys)
961 B │ │ │─ RunnersUp => Value(961 bytes)
625 B │ │ │─ Members => Value(625 bytes)
33  B │ │ │─ Candidates => Value(33 bytes)
4   B │ │ │─ ElectionRounds => Value(4 bytes)
22  K │ │─┬ Babe
12  K │ │ │─ UnderConstruction => Map(12,901 bytes, 3 keys)
9   K │ │ │─ Authorities => Value(9,602 bytes)
32  B │ │ │─ NextRandomness => Value(32 bytes)
32  B │ │ │─ Randomness => Value(32 bytes)
8   B │ │ │─ CurrentSlot => Value(8 bytes)
8   B │ │ │─ GenesisSlot => Value(8 bytes)
8   B │ │ │─ EpochIndex => Value(8 bytes)
4   B │ │ │─ SegmentIndex => Value(4 bytes)
0   B │ │ │─ Lateness => Value(0 bytes)
0   B │ │ │─ AuthorVrfRandomness => Value(0 bytes)
0   B │ │ │─ Initialized => Value(0 bytes)
0   B │ │ │─ NextEpochConfig => Value(0 bytes)
20  K │ │─┬ Claims
11  K │ │ │─ Claims => Map(12,128 bytes, 758 keys)
5   K │ │ │─ Preclaims => Map(5,660 bytes, 283 keys)
1   K │ │ │─ Vesting => Map(1,980 bytes, 55 keys)
780 B │ │ │─ Signing => Map(780 bytes, 780 keys)
16  B │ │ │─ Total => Value(16 bytes)
16  K │ │─┬ Proxy
16  K │ │ │─ Proxies => Map(16,584 bytes, 273 keys)
0   B │ │ │─ Announcements => Map(0 bytes, 0 keys)
8   K │ │─┬ ImOnline
7   K │ │ │─ Keys => Value(7,682 bytes)
956 B │ │ │─ AuthoredBlocks => Map(956 bytes, 239 keys)
198 B │ │ │─ ReceivedHeartbeats => Map(198 bytes, 2 keys)
4   B │ │ │─ HeartbeatAfter => Value(4 bytes)
8   K │ │─┬ Treasury
6   K │ │ │─ Tips => Map(6,269 bytes, 19 keys)
1   K │ │ │─ Reasons => Map(1,985 bytes, 19 keys)
288 B │ │ │─ Proposals => Map(288 bytes, 3 keys)
9   B │ │ │─ Approvals => Value(9 bytes)
4   B │ │ │─ ProposalCount => Value(4 bytes)
1   B │ │ │─ BountyApprovals => Value(1 bytes)
0   B │ │ │─ BountyDescriptions => Map(0 bytes, 0 keys)
0   B │ │ │─ Bounties => Map(0 bytes, 0 keys)
0   B │ │ │─ BountyCount => Value(0 bytes)
2   K │ │─┬ Multisig
2   K │ │ │─ Multisigs => Map(3,043 bytes, 27 keys)
0   B │ │ │─ Calls => Map(0 bytes, 0 keys)
2   K │ │─┬ RandomnessCollectiveFlip
2   K │ │ │─ RandomMaterial => Value(2,594 bytes)
999 B │ │─┬ Council
444 B │ │ │─ Voting => Map(444 bytes, 2 keys)
417 B │ │ │─ Members => Value(417 bytes)
65  B │ │ │─ Proposals => Value(65 bytes)
37  B │ │ │─ ProposalOf => Map(37 bytes, 2 keys)
32  B │ │ │─ Prime => Value(32 bytes)
4   B │ │ │─ ProposalCount => Value(4 bytes)
996 B │ │─┬ Grandpa
988 B │ │ │─ SetIdSession => Map(988 bytes, 247 keys)
8   B │ │ │─ CurrentSetId => Value(8 bytes)
0   B │ │ │─ Stalled => Value(0 bytes)
0   B │ │ │─ NextForced => Value(0 bytes)
0   B │ │ │─ PendingChange => Value(0 bytes)
0   B │ │ │─ State => Value(0 bytes)
102 B │ │─┬ TechnicalCommittee
97  B │ │ │─ Members => Value(97 bytes)
4   B │ │ │─ ProposalCount => Value(4 bytes)
1   B │ │ │─ Proposals => Value(1 bytes)
0   B │ │ │─ Prime => Value(0 bytes)
0   B │ │ │─ Voting => Map(0 bytes, 0 keys)
0   B │ │ │─ ProposalOf => Map(0 bytes, 0 keys)
21  B │ │─┬ Scheduler
20  B │ │ │─ Agenda => Map(20 bytes, 2 keys)
1   B │ │ │─ StorageVersion => Value(1 bytes)
0   B │ │ │─ Lookup => Map(0 bytes, 0 keys)
17  B │ │─┬ TransactionPayment
16  B │ │ │─ NextFeeMultiplier => Value(16 bytes)
1   B │ │ │─ StorageVersion => Value(1 bytes)
8   B │ │─┬ Timestamp
8   B │ │ │─ Now => Value(8 bytes)
0   B │ │ │─ DidUpdate => Value(0 bytes)
1   B │ │─┬ TechnicalMembership
1   B │ │ │─ Members => Value(1 bytes)
0   B │ │ │─ Prime => Value(0 bytes)
1   B │ │─┬ Authorship
1   B │ │ │─ Uncles => Value(1 bytes)
0   B │ │ │─ DidSetUncles => Value(0 bytes)
0   B │ │ │─ Author => Value(0 bytes)

• A feature that we might want to add here is that I am not sure if we have RPC calls to get the size of child-trees. All in all, child trees were not used in production back then and are not considered.

[kianenigma]

@harrysolovay has build this with CAPI: https://github.com/paritytech/capi/blob/main/examples/size_tree.ts

[kianenigma]

Please see: #4341 (review)

[bkchr]

@kianenigma sounds like we should close this issue and create a new one for the subxt explorer?

I think also @josepot at some point added/wanted to add something similar to papi?

[ggwpez]

I wrote a prototype to gauge the throughput needed for the Asset-Hub migration. Not sure if it is worth integrating into the SDK though, it could just exist as script somewhere.

It consumes try-runtime snapshots to be re-usable with CI caches. Output for Rococo AH:

[kianenigma]

Noice 🚀 If it works on the basis of try-runtime, I would say it could come and live here in the sdk, but if not, it can also somewhere in your gh.

Would be good if you share the link here + result of running it on DOT and KSM relay chains.

[ggwpez]

Code is here now, did not check in detail yet: https://github.com/ggwpez/pdu