cmd, core, eth, light, trie: add trie read caching layer

[karalabe]

This is an alternative approach to #17873.

In short, this PR introduces a read cache into the trie.Database (previously it was a write cache/gc only). The reason behind this is twofold:

• Storage tries are currently not cached. If subsequent blocks access the same read-only data, they keep hammering the disk to retrieve it. It's also a griefing vector since we can create transactions that try to max out storage trie reads (not dangerous at current gas levels, but sure annoying).
• Executing pending transactions load parts of the account and storage tries into memory during their execution, but annoyingly dump all that information when a full block arrives. The next full block will however most probably share a lot of disk accesses with our pending state, so might as well reuse anything that we've already loaded.

My original attempt in #17873 cached entire storage tries, and introduced a fancy way to maximize useful data across transactions and blocks. Unfortunately, we have no means to measure the memory usage of that approach, neither easily control it. Alas, although it's cleaner and faster, it was not provably immune to DoS attacks.

This current approach in this PR adds the caching layer in between the database and our internal trie (same way we did for writes/pruning). This is suboptimal because we can only cache rlp encoded nodes, requiring reparsing them every time on access. On the flipside though, it avoid the disk reads the same way my old PR did, but also guarantees memory caps. As a bonus point, the data structure used avoids GC overhead on the millions of read-cached trie nodes, so it my actually be more performant with large enough caches.

---

Benchmark results (purple = PR, blue = master):

Disk writes grow at the same rate as with master. The absolute value here is not relevant because I restarted the code on top of an existing chain, causing quite some blocks to be reprocessed, but different number on master and pr. The charts are after the system synced. Read wise we can see this pr saves about 75% of disk reads compared to master on mainnet (running with --cache=2048).

Perhaps a more interesting metric is the propagated block processing time. This PR manages to cut new block times down by about 30%. You can see that the PR doesn't help old blocks that much during sync, since there's no preloading (pending transactions are the preloaders), however after sync is done, new import times are much lower than master ones. Our metrics library uses 1024 samples (blocks) for averaging, hence the couple hours it takes for the speedup to be fully visible on the charts.

---

Full sync benchmarks

i3.2xlarge: 8vcpu, 61 GiB mem, 1.9 TB NVMe SSD

--syncmode==full --cache=2048

build	time	disk reads	disk writes
master	6d 2h 30m	33.28TB	23.27TB
cacher	5d 4h 40m	30.72TB	21.74TB
	-15.27%	-7.69%	-6.57%

[holiman]

It looks good to me, fwiw, but should be checked by @fjl aswell

[holiman]

Comparing another branch with this pr on TestDifficulty/difficultyRopsten.json/DifficultyTest1023 (adding an artificial fail)

        --- FAIL: TestDifficulty/difficultyRopsten.json/DifficultyTest1023 (0.00s)
        	difficulty_test.go:85: parent[time 15025775352 diff 5197261454665822249 unclehash:1dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347] child[time 15025775372 number 4300000] diff 5194723729346161203 != expected 5194723729346161203



        --- FAIL: TestDifficulty/difficultyRopsten.json/DifficultyTest1023 (0.00s)
        	difficulty_test.go:85: parent[time 9323269331 diff 7371607815720194386 unclehash:1dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347] child[time 9323269351 number 4300000] diff 7368008397841424760 != expected 7368008397841426808

both parent time and child time seems off

EDIT: I used the wrong output , fixed now

[holiman]

Do you get the same errors locally?

[holiman]

Test

[user@work go-ethereum]$ cat ./tests/testdata/BasicTests/difficultyRopsten.json  | grep DifficultyTest1023 -A8
 "DifficultyTest1023" : {
		"parentTimestamp" : "0x037f9b22f8",
		"parentDifficulty" : "0x482061c1ba2fb029",
		"parentUncles" : "0x1dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347",
		"currentTimestamp" : "0x037f9b230c",
		"currentBlockNumber" : "0x419ce0",
		"currentDifficulty" : "0x48175db581f86a33"
	},

0x037f9b22f8 == 15025775352

[karalabe]

Yeah, so I accidentally pushed some old version of the consensus test repo into this PR. Fixed it now.

[karalabe]

@holiman I've rebased on master, renamed the nodes field to dirties to make it clearer what it is, and also added 4 metrics to be able to measure the hits/misses, data stores/loads from the read cache. These will be particularly handy when comparing a simple node vs. a light server to see how much strain is put on the caches.

[karalabe]

@fjl I've reworked the API so the 0 is not mandatory, rather there's a second set of methods WithCache. PTAL

[fjl]

Changes look much cleaner now. Let's investigate if this change also allows disabling the generation-based caching.

[karalabe]

Sure, but lets please do that in a followup PR. Removing the cache generations is a non negligible amount of change, after which at least a full sync is needed to validate it, which is +1 week. If this change is ok and works well, I don't think we should postpone it just to add more to it.

[crackcomm]

@karalabe I would consider it unsafe in some circumstances but it's just being paranoid, is'nt it?

this is why I submitted #18118