trie: Derivesha with stacktrie

[gballet]

This has the updated version of stacktrie, with Commit.

[rjl493456442]

The commit seems useless. All nodes will be committed directly during the insertion.
I guess "if the db passed is not nil, then commit is enabled" is OKish?

[gballet]

yeah that's what I'm shooting for. My understanding of the role of StackTrie is to be "the type of trie that you use when you can insert all keys sequentially and want to calculate the hash". I don't foresee a use case where it could be used for some kind of caching. So presumably, if the data in the trie is to be persisted eventually, it might as well be persisted as it is being inserted.

[rjl493456442]

StackTrie requires all entries are inserted in a sort way. But if we use it for deriving hash, the insertion is not guaranteed.

[holiman]

Benchmark results? 👀 ?

[MariusVanDerWijden]

I think there is something fishy with the shortcut in line 269 as it does not write to the DB.
I created the following test:

func TestCommit(t *testing.T) {
	db := memorydb.New()
	st := NewStackTrie(db)

	kvs := []struct {
		K string
		V string
	}{
		{K: "405787fa12a823e0f2b7631cc41b3ba8828b3321ca811111fa75cd3aa3bb5ace", V: "1111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111"},
	}

	for _, kv := range kvs {
		st.TryUpdate(common.FromHex(kv.K), common.FromHex(kv.V))
	}
	//st.Hash()
	newDB := memorydb.New()

	st.Commit(newDB)
	if db.Len() != newDB.Len() {
		t.Fatalf("Did not insert correctly %v, %v", db.Len(), newDB.Len())
	}

}

If I let st.Hash() commented out the result is Did not insert correctly 0, 1 if I delete the comment it results in Did not insert correctly 1, 0 So it depends on st.Hash() into which db we insert.

[holiman]

Added another commit, making it use the existing hasherpool.

[user@work trie]$ benchstat before.txt after.txt 
name                       old time/op    new time/op    delta
DeriveSha200/std_trie-6       579µs ±11%     580µs ±11%     ~     (p=1.000 n=5+5)
DeriveSha200/stack_trie-6     708µs ±18%     624µs ± 5%  -11.90%  (p=0.016 n=5+4)

name                       old alloc/op   new alloc/op   delta
DeriveSha200/std_trie-6       274kB ± 0%     274kB ± 0%     ~     (p=0.175 n=5+5)
DeriveSha200/stack_trie-6     397kB ± 0%     198kB ± 0%  -50.22%  (p=0.008 n=5+5)

name                       old allocs/op  new allocs/op  delta
DeriveSha200/std_trie-6       2.86k ± 0%     2.86k ± 0%     ~     (all equal)
DeriveSha200/stack_trie-6     3.16k ± 0%     2.52k ± 0%  -20.40%  (p=0.008 n=5+5)

[holiman]

After this last commit, It's still a bit slower than std_trie, but uses less allocs

[holiman]

On a larger list, with 2K items, it's faster aswell:

BenchmarkDeriveSha2000
BenchmarkDeriveSha2000/std_trie
BenchmarkDeriveSha2000/std_trie-6         	     141	  10510602 ns/op	 4117591 B/op	   33935 allocs/op
BenchmarkDeriveSha2000/stack_trie
BenchmarkDeriveSha2000/stack_trie-6       	     166	   6670803 ns/op	 1958921 B/op	   25090 allocs/op

[holiman]

Pushed another fix, now stacktrie beats regular trie on 200 item lists:

name                       old time/op    new time/op    delta
DeriveSha200/std_trie-6       585µs ± 8%     577µs ± 2%     ~     (p=0.690 n=5+5)
DeriveSha200/stack_trie-6     804µs ±69%     541µs ± 1%  -32.62%  (p=0.016 n=5+4)

name                       old alloc/op   new alloc/op   delta
DeriveSha200/std_trie-6       274kB ± 0%     274kB ± 0%     ~     (p=0.175 n=5+5)
DeriveSha200/stack_trie-6     198kB ± 0%     116kB ± 0%  -41.27%  (p=0.008 n=5+5)

name                       old allocs/op  new allocs/op  delta
DeriveSha200/std_trie-6       2.86k ± 0%     2.86k ± 0%     ~     (all equal)
DeriveSha200/stack_trie-6     2.52k ± 0%     1.87k ± 0%  -25.63%  (p=0.008 n=5+5)

[holiman]

total effect of the last 4 commits:

name                       old time/op    new time/op    delta
DeriveSha200/std_trie-6       579µs ±11%     578µs ±17%     ~     (p=0.841 n=5+5)
DeriveSha200/stack_trie-6     708µs ±18%     532µs ± 2%  -24.88%  (p=0.016 n=5+4)

name                       old alloc/op   new alloc/op   delta
DeriveSha200/std_trie-6       274kB ± 0%     274kB ± 0%     ~     (p=0.587 n=5+5)
DeriveSha200/stack_trie-6     397kB ± 0%     116kB ± 0%  -70.76%  (p=0.008 n=5+5)

name                       old allocs/op  new allocs/op  delta
DeriveSha200/std_trie-6       2.86k ± 0%     2.86k ± 0%     ~     (all equal)
DeriveSha200/stack_trie-6     3.16k ± 0%     1.87k ± 0%  -40.80%  (p=0.008 n=5+5)

[holiman]

I think there is something fishy with the shortcut in line 269 as it does not write to the DB.

Yes, it's a bit of an oddity. So the Hash does write to database, if non-nil. And the Commit just sets the db to non-nil and calls Hash. Maybe it would be neater to change the signature of hash into hash(db), and have Hash() utilize hash(nil), and have Commit(db) use hash(db) under the hood?

[holiman]

So the Hash does write to database, if non-nil. And the Commit just sets the db to non-nil and calls Hash. Maybe it would be neater to change the signature of hash into hash(db), and have Hash() utilize hash(nil), and have Commit(db) use hash(db) under the hood?

It's actually even more complicated than that. Even Update (via insert) may write to the database. So the Commit is just a way to "write leftovers to the db", but in fact db operations happen all the time. Which is totally understandable, since the idea iwth the stacktrie is that we shouldn't build the full trie in memory, but actually get rid of things too.

And that throws the model of having a totally separate 'Commit`-stage out the window, basically. I'm not sure what the proper thing to do is.

[gballet]

I think there is something fishy with the shortcut in line 269 as it does not write to the DB.
I created the following test:

func TestCommit(t *testing.T) {
	db := memorydb.New()
	st := NewStackTrie(db)

	kvs := []struct {
		K string
		V string
	}{
		{K: "405787fa12a823e0f2b7631cc41b3ba8828b3321ca811111fa75cd3aa3bb5ace", V: "1111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111"},
	}

	for _, kv := range kvs {
		st.TryUpdate(common.FromHex(kv.K), common.FromHex(kv.V))
	}
	//st.Hash()
	newDB := memorydb.New()

	st.Commit(newDB)
	if db.Len() != newDB.Len() {
		t.Fatalf("Did not insert correctly %v, %v", db.Len(), newDB.Len())
	}

}

If I let st.Hash() commented out the result is Did not insert correctly 0, 1 if I delete the comment it results in Did not insert correctly 1, 0 So it depends on st.Hash() into which db we insert.

You're missing the point here: the role of StackTrie is to save memory by hashing/committing nodes as soon as it is known they will no longer be updated. Even if Hash and Commit were updated, most of the nodes after you call Hash would already be hashed, and the subsequent call to Commit would miss most nodes.

[gballet]

It's actually even more complicated than that. Even Update (via insert) may write to the database. So the Commit is just a way to "write leftovers to the db", but in fact db operations happen all the time. Which is totally understandable, since the idea iwth the stacktrie is that we shouldn't build the full trie in memory, but actually get rid of things too.

Exactly, you have to commit the nodes as soon as you know they can be hashed, so that you don't run out of memory. We could implement some sort of caching and batch them. I don't think that it is possible to totally separate hashing from committing, though, precisely because memory has to be freed.

[holiman]

This could be rewritten as

		var nodes [17]node

		for i, child := range st.children {
			if child != nil {
				childhash := child.hash()
				if len(childhash) < 32 {
					nodes[i] = rawNode(childhash)
				}else{
					nodes[i]= hashNode(childhash)
				}
			} else {
				nodes[i] = nilValueNode
			}
		}
		nodes[16] = nilValueNode
...
		if err := rlp.Encode(&h.tmp, nodes); err != nil {
			panic(err)
		}

As it is now, if you look at fullNode.EncodeRLP, it basically does this whole dance again, iterating over the children to produce a []node replacing nil will nilValueNodes. So that can be skipped.

Doesn't impact the benchmark a whole lot, but might do a bit of a difference on a larger trie?

[holiman]

... so essentially

	case branchNode:
		var nodes [17]node
		for i, child := range st.children {
			if child == nil{
				nodes[i] = nilValueNode
				continue
			}
			if childhash := child.hash(); len(childhash) < 32 {
				nodes[i] = rawNode(childhash)
			}else{
				nodes[i]= hashNode(childhash)
			}
		}
		nodes[16] = nilValueNode
		h = newHasher(false)
		defer returnHasherToPool(h)
		h.tmp.Reset()
		if err := rlp.Encode(&h.tmp, nodes); err != nil {
			panic(err)
		}

[gballet]

rebased to see if this could fix the linter issue, it works fine on my local machine

[holiman]

CPU profile, running the stacktrie benchmark (commenting out the standard trie test)

And mem profile:

[holiman]

Before merging this, I think it would be prudent to iterate through the full ancients of mainnet, and do the DeriveSha for all receipt- and transaction-lists, to verify that it still works.
As an alternative, I guess a fast-sync would work just as well.

[gballet]

Ok, running a mainnet fastsync 🤞

[holiman]

LGTM (but I wrote parts of it, so my thumb-up doesn't count)

[holiman]

Final numbers:

name                 old time/op    new time/op    delta
DeriveSha200/trie-6     625µs ±20%     515µs ±17%  -17.65%  (p=0.001 n=10+9)

name                 old alloc/op   new alloc/op   delta
DeriveSha200/trie-6     275kB ± 0%      55kB ± 0%  -80.03%  (p=0.000 n=10+10)

name                 old allocs/op  new allocs/op  delta
DeriveSha200/trie-6     2.87k ± 0%     1.27k ± 0%  -55.73%  (p=0.000 n=10+10)

[holiman]

With larger numbers, 2000 keys-values:

BenchmarkDeriveSha2000/std_trie-6                    100          10962903 ns/op         4119401 B/op      33950 allocs/op
BenchmarkDeriveSha2000/std_trie-6                    150           8169317 ns/op         4118595 B/op      33947 allocs/op
BenchmarkDeriveSha2000/std_trie-6                    156           7748590 ns/op         4118925 B/op      33948 allocs/op
BenchmarkDeriveSha2000/stack_trie-6                  234           4876869 ns/op          536370 B/op      12566 allocs/op
BenchmarkDeriveSha2000/stack_trie-6                  248           4913092 ns/op          536256 B/op      12565 allocs/op
BenchmarkDeriveSha2000/stack_trie-6                  249           5034490 ns/op          536270 B/op      12565 allocs/op

[holiman]

With 20K:

BenchmarkDeriveSha20000/std_trie-6                    12          89740864 ns/op        47107630 B/op     348241 allocs/op
BenchmarkDeriveSha20000/std_trie-6                    13          88671409 ns/op        47107734 B/op     348244 allocs/op
BenchmarkDeriveSha20000/stack_trie-6                  24          52058080 ns/op         5336531 B/op     125446 allocs/op
BenchmarkDeriveSha20000/stack_trie-6                  24          52906663 ns/op         5336178 B/op     125444 allocs/op

[holiman]

I experimented with ininite_trie:

package main

import (
	"fmt"
	"math/big"
	"time"

	"github.com/ethereum/go-ethereum/common/math"
	"github.com/ethereum/go-ethereum/trie"
)

func main() {
	fmt.Println("Let's find out the trie hash of a fully saturated hexary trie!\n")
	var (
		tr             = trie.NewStackTrie(nil)
		b              = big.NewInt(1)
		big1           = big.NewInt(1)
		end, _         = new(big.Int).SetString("ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff", 16)
		lastLog, start = time.Now(), time.Now()
	)
	for b.Cmp(end) < 0 {
		key := math.PaddedBigBytes(b, 32)
		tr.Update(key, b.Bytes())
		b.Add(b, big1)
		if time.Since(lastLog) > 10*time.Second {
			lastLog = time.Now()
			elapsed := time.Since(start)
			progress := big.NewInt(0).Mul(b, big.NewInt(100*100))
			progress.Div(progress, end)
			percent := float64(progress.Uint64()) / 100
			rate := big.NewInt(0).Div(b, big.NewInt(int64(elapsed/time.Second)))
			fmt.Printf("Nearly there! %.02f %% done. %d additions in %v. Rate: %v inserts/s\n",
				percent,
				b,
				elapsed, rate)
		}
	}
}

output:

Let's find out the trie hash of a fully saturated hexary trie!

Nearly there! 0.00 % done. 6229553 additions in 10.000065072s. Rate: 622955 inserts/s
Nearly there! 0.00 % done. 12256962 additions in 20.000141602s. Rate: 612848 inserts/s
Nearly there! 0.00 % done. 18237832 additions in 30.000143089s. Rate: 607927 inserts/s
Nearly there! 0.00 % done. 24298523 additions in 40.000384147s. Rate: 607463 inserts/s
Nearly there! 0.00 % done. 30163618 additions in 50.000383426s. Rate: 603272 inserts/s
Nearly there! 0.00 % done. 36033613 additions in 1m0.00038604s. Rate: 600560 inserts/s
Nearly there! 0.00 % done. 41822215 additions in 1m10.000387331s. Rate: 597460 inserts/s
Nearly there! 0.00 % done. 47460650 additions in 1m20.000389449s. Rate: 593258 inserts/s
Nearly there! 0.00 % done. 53231214 additions in 1m30.000390272s. Rate: 591457 inserts/s
Nearly there! 0.00 % done. 58747349 additions in 1m40.000456935s. Rate: 587473 inserts/s
Nearly there! 0.00 % done. 64471917 additions in 1m50.000458735s. Rate: 586108 inserts/s
Nearly there! 0.00 % done. 70125990 additions in 2m0.000460157s. Rate: 584383 inserts/s
Nearly there! 0.00 % done. 75797575 additions in 2m10.000463053s. Rate: 583058 inserts/s
Nearly there! 0.00 % done. 81589186 additions in 2m20.000464204s. Rate: 582779 inserts/s
Nearly there! 0.00 % done. 87353647 additions in 2m30.000466191s. Rate: 582357 inserts/s
Nearly there! 0.00 % done. 92978947 additions in 2m40.000467703s. Rate: 581118 inserts/s
Nearly there! 0.00 % done. 98872961 additions in 2m50.000469521s. Rate: 581605 inserts/s
Nearly there! 0.00 % done. 104711106 additions in 3m0.000471009s. Rate: 581728 inserts/s
Nearly there! 0.00 % done. 110690132 additions in 3m10.000472722s. Rate: 582579 inserts/s
Nearly there! 0.00 % done. 116521819 additions in 3m20.000474889s. Rate: 582609 inserts/s
Nearly there! 0.00 % done. 122380953 additions in 3m30.000476468s. Rate: 582766 inserts/s

[gballet]

one infinity later... ♾️

[holiman]

I'll merge this in the next couple of days or so, right now investigating some statesync issues