refactor(pkg/trie/triedb): implement generics, refactor nibbles, revise iterator #4221

timwu20 · 2024-10-03T19:39:53Z

Changes

Introduces and integrates a new hash.Hash and hash.Hasher interface constraints into triedb package.
- This is a local interface that is similar to runtime.Hash and runtime.Hasher already on development.
- Updates all the tests to use hash.H256 for the hash and runtime.BlakeTwo256 as the hasher.
Refactors how we handle nibbles in triedb. Introduces NibbleSlice, Nibbles, and LeftNibbles. Each of these introduced types has their own distinct public interface and are used in different components within triedb.
- Nibbles is analogous to trie_db::nibble::NibbleSlice. Primarily used to avoid costly slice manipulation and unnecessary copying of underlying bytes. Able to represent different nibble starting points, but not mutable to be able to reconstruct full keys.
- NibbleSlice is analogous to trie_db::nibble::NibbleVec. Mutable in the sense you can push and pop nibbles as well as truncate. Used to reconstruct full keys.
- LeftNibbles is analogous to trie_db::nibble::LeftNibbleSlice. Used in proof generation and verification. Since NibbleSlice and NIbbles are right aligned it makes it more expensive to truncate. LeftNibbles left aligns the nibbles which facilitates faster truncation.
Revises the iterator to mirror TrieDBRawIterator which is imported by the state machine primitives in substrate. This is embedded in other types to be able to iterate and return expected types. The rawItem introduced that is returned by rawIterator.nextRawItem includes the partial key as NibbleSlice, hash of the node, and an in-memory type that fulfills the codec.EncodedNode interface (Branch or Leaf).

Notes

Removes caching and introduces a temporary Cache interface to avoid too many code changes to how we construct TrieDBs and the options we provide it.
Future PR will introduce the new caching interface as required from future client db work. The implementation of the cache is actually in substrate and will not be included as part triedb package.
Backwards compatibility is now broken given we have revised how we store the nodes in underlying k/v store. We are now inline with the rust trie impl on how we store keys for nodes.
- We were using an "expanded" view of the nibbles as part of the storage keys for the nodes. An expanded view is essentially where each nibble occupies an entire byte. So keys were 2x longer than we actually needed.
- We also now prefix hash values based on child to avoid any key collisions. It was highly unlikely we would retrieve the wrong value, but I think it's a prudent decision to prefix it in our impl.

Tests

go test -tags integration github.com/ChainSafe/gossamer

Issues

contributes to #4157

…, remove caching temporarily

pkg/trie/triedb/child_tries.go

pkg/trie/triedb/codec/decode.go

dimartiro

A few comments but overall ok!
Thanks for spending time working on this changes Tim! I tried to keep the TrieDB as simple as possible but it seems that it wasn't enough 💔

dimartiro · 2024-10-04T17:22:37Z

pkg/trie/triedb/codec/node.go

@@ -23,11 +24,11 @@ type (
 	// InlineNode contains bytes of the encoded node data
 	InlineNode []byte
 	// HashedNode contains a hash used to lookup in db for encoded node data
-	HashedNode common.Hash
+	HashedNode[H any] struct{ Hash H }


Why not?

Suggested change

HashedNode[H any] struct{ Hash H }

HashedNode[H hash.Hash] struct{ Hash H }

I lean towards trying to be succinct and generic as possible per type. I think it's understood that hash.Hash is used exclusively in TrieDB, but it doesn't necessarily mean that we need it to be for HashedNode. If you feel strongly the other way I'm open to changing it.

mmm since we are always storing hashes in HashedNode I would say that it will be right to use the hash.Hash instead of any to prevent issues and also being consistent with HashedValue[H hash.Hash]. What do you think? do you see any issue?

dimartiro · 2024-10-04T17:27:29Z

pkg/trie/triedb/lookup.go

 }

-func NewTrieLookup(db db.DBGetter, hash common.Hash, cache cache.TrieCache, recorder *Recorder) TrieLookup {
-	return TrieLookup{
+func NewTrieLookup[H hash.Hash, Hasher hash.Hasher[H], QueryItem any](


Where are you using the QueryItem? 🤔

Whoops! 😅 This is part of the caching changes coming next. I'll remove on this branch.

pkg/trie/triedb/lookup.go

dimartiro · 2024-10-04T17:38:35Z

pkg/trie/triedb/node.go

@@ -143,32 +150,38 @@ func inMemoryFetchedValue(value nodeValue, prefix []byte, db db.DBGetter) ([]byt
 	}
 }

+type NodeTypes[H hash.Hash] interface {


Are we using this?

No not yet. I think it's good to semantically describe what the Node types are.

dimartiro · 2024-10-04T17:39:22Z

pkg/trie/triedb/proof/generate.go

@@ -18,20 +19,22 @@ type nodeHandle interface {
 }

 type (
-	nodeHandleHash   common.Hash
+	nodeHandleHash[H any] struct {


Why not?

Suggested change

nodeHandleHash[H any] struct {

nodeHandleHash[H hash.Hash] struct {

#4221 (comment)

dimartiro · 2024-10-04T17:44:15Z

pkg/trie/triedb/node_storage.go

@@ -63,34 +62,34 @@ type (
 	NewStoredNode struct {
 		node Node
 	}
-	CachedStoredNode struct {
+	CachedStoredNode[H any] struct {


Why not hash.Hash?

#4221 (comment)

dimartiro · 2024-10-04T17:44:32Z

pkg/trie/triedb/node_storage.go

 	return n.node
 }

 // nodeStorage is a struct that contains all the temporal nodes that are stored
 // in the trieDB before being written to the backed db
-type nodeStorage struct {
+type nodeStorage[H any] struct {


Suggested change

type nodeStorage[H any] struct {

type nodeStorage[H hash.Hash] struct {

#4221 (comment)

timwu20 · 2024-10-04T17:57:42Z

A few comments but overall ok! Thanks for spending time working on this changes Tim! I tried to keep the TrieDB as simple as possible but it seems that it wasn't enough 💔

You did all the heavy lifting @dimartiro! The fact that I have something to integrate with and contribute to is a testament to your hard work. 💪

timwu20 changed the title ~~refactor(pkg/trie/triedb) implement generics, refactor nibbles, revise iterator~~ refactor(pkg/trie/triedb): implement generics, refactor nibbles, revise iterator Oct 3, 2024

timwu20 force-pushed the tim/refactor-triedb branch 14 times, most recently from 6759481 to cc55742 Compare October 4, 2024 02:32

timwu20 changed the base branch from development to tim/update-golangci-lint October 4, 2024 02:32

timwu20 force-pushed the tim/refactor-triedb branch 2 times, most recently from 1f89c69 to 895baf8 Compare October 4, 2024 02:56

timwu20 added 2 commits October 3, 2024 23:19

triedb changes, implement generics, refactor nibbles, revise iterator…

664b8ce

…, remove caching temporarily

remove unnecessary code

1a55c3e

timwu20 force-pushed the tim/refactor-triedb branch from 895baf8 to 1a55c3e Compare October 4, 2024 03:20

timwu20 marked this pull request as ready for review October 4, 2024 15:19

timwu20 requested review from dimartiro, EclesioMeloJunior, jimjbrettj and P1sar as code owners October 4, 2024 15:19

dimartiro reviewed Oct 4, 2024

View reviewed changes

pkg/trie/triedb/child_tries.go Outdated Show resolved Hide resolved

dimartiro reviewed Oct 4, 2024

View reviewed changes

pkg/trie/triedb/codec/decode.go Show resolved Hide resolved

timwu20 force-pushed the tim/refactor-triedb branch from 3ec8390 to 665a370 Compare October 4, 2024 17:34

cr feedback, code cleanup

9c82571

timwu20 force-pushed the tim/refactor-triedb branch from 665a370 to 9c82571 Compare October 4, 2024 17:42

dimartiro approved these changes Oct 4, 2024

View reviewed changes

cr feedback, remove QueryItem

94949b5

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

refactor(pkg/trie/triedb): implement generics, refactor nibbles, revise iterator #4221

refactor(pkg/trie/triedb): implement generics, refactor nibbles, revise iterator #4221

timwu20 commented Oct 3, 2024 •

edited

Loading

dimartiro left a comment

dimartiro Oct 4, 2024

timwu20 Oct 4, 2024

dimartiro Oct 4, 2024

dimartiro Oct 4, 2024

timwu20 Oct 4, 2024

dimartiro Oct 4, 2024

timwu20 Oct 4, 2024

dimartiro Oct 4, 2024

timwu20 Oct 4, 2024

dimartiro Oct 4, 2024

timwu20 Oct 4, 2024

dimartiro Oct 4, 2024

timwu20 Oct 4, 2024

timwu20 commented Oct 4, 2024

	HashedNode[H any] struct{ Hash H }
	HashedNode[H hash.Hash] struct{ Hash H }

	nodeHandleHash[H any] struct {
	nodeHandleHash[H hash.Hash] struct {

	type nodeStorage[H any] struct {
	type nodeStorage[H hash.Hash] struct {

refactor(pkg/trie/triedb): implement generics, refactor nibbles, revise iterator #4221

Are you sure you want to change the base?

refactor(pkg/trie/triedb): implement generics, refactor nibbles, revise iterator #4221

Conversation

timwu20 commented Oct 3, 2024 • edited Loading

Changes

Notes

Tests

Issues

dimartiro left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

timwu20 commented Oct 4, 2024

timwu20 commented Oct 3, 2024 •

edited

Loading