[Storehouse] Implement Register store

[zhangchiqing]

This PR implements the RegisterStore.

Closes: #4659

[codecov-commenter]

Codecov Report

All modified and coverable lines are covered by tests ✅

Comparison is base (fa477d9) 56.23% compared to head (f7d8ede) 61.15%.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #4940      +/-   ##
==========================================
+ Coverage   56.23%   61.15%   +4.92%     
==========================================
  Files         969       20     -949     
  Lines       90431     1555   -88876     
==========================================
- Hits        50854      951   -49903     
+ Misses      35789      551   -35238     
+ Partials     3788       53    -3735     

Flag	Coverage Δ
unittests	61.15% <ø> (+4.92%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[sideninja]

can you add a comment explaining what is the difference between height and pruned height

[sideninja]

@zhangchiqing is this PR the same as the one I already reviewed?

[janezpodhostnik]

Suggested change

	// finalizationProcessingLoop notify the register store when a block is finalized
	// and handle the error if any
	// finalizationProcessingLoop notifies the register store when a block is finalized.

[janezpodhostnik]

could we join the GetUpdatedRegisters an Prune calls into one call.

something like PopUpdatedRegisters

[zhangchiqing]

I can't prune before successfully storing it in diskstore, otherwise if there is a query getting register at the pruned height, it will fail, because the in-mem store has pruned its value, and disk store has not stored it yet.

I also can't store into disk store first, because I need to retrieve the updated registers from memStore first.

[sideninja]

would this always be one bigger than pruned height? how could it be bigger than that?

[zhangchiqing]

It could be a few blocks bigger, not just one. For instance:

X is 3 at height 10,
X is 6 at height 15.
block 12 is finalized, now (X: 3, height: 10) is stored in OnDiskRegisterStore, and (X: 6, height: 15) is stored in InMemoryRegisterStore.
Then GetRegister(14, X) will first query memStore, which returns prunedError with prunedHeight as 12. So the height 14 is bigger than prunedHeight 12 by 2 (bigger than one), now we need to query diskStore with (X, 12), which returns 3. Meaning X at height 14 is the same as X at height 12, because X wasn't updated from height 11 to 14.

[sideninja]

will it be ok if the OnBlockFinalized is not run since it's being used by another goroutine?

[zhangchiqing]

It's ok, because:

1. another goroutine will recursively call OnBlockFinalized for the next block, until there is no more finalized blocks.
2. it's also ok to miss a call to onBlockFinalized, because the correct register value can still be returned regardless it's stored in in-memory store or on-disk store. We are also resistant from crash-failure, because during startup recovery, we will re-execute all blocks since on-disk store's latest executed height.

[sideninja]

is it ok to prune with every block finalized, would it be a valuable improvement if we only prune once we don't have more finalized blocks? not sure how expensive is to prune just an idea.

[zhangchiqing]

prune should be fast and cheap, since it's just acquire a lock and updating some index in memory.

I think the performance is quite similar comparing the following two approach:

1. pruning once to prune 3 finalized blocks
2. pruning 3 times each time to prune 1 finalized block