feat: add tree snapshots

[alexghr]

This PR adds two different algorithms to create snapshots of merkle trees.

Notation

N - number of non-zero nodes
H - height of tree
M - number of snapshots

Incremental Full snapshots

This algorithm stores the trees in a database instance in the same way they would be stored using pointers in memory. Each node has two key-value entries in the database: one for its left child and one for its right child.

Taking a snapshot means just walking the tree (BF order), storing any new nodes. If we hit a node that already exists in the database, that means the subtree rooted at that node has not changed and so we can skip checking it.

Building a sibling path is just a traversal of the tree from historic root to historic leaf.

Pros:

1. its generic enough that it works with any type of merkle tree (one caveat: we'll need an extension to store historic leaf values for indexed-trees)
2. it shares structure with other versions of the tree: if some subtree hasn't changed between two snapshots then it will be reused for both snapshots (would work well for append-only trees)
3. getting a historic sibling path is extremely cheap only requiring O(H) database reads

Cons:

1. it takes up a space to store every snapshot. Worst case scenario it would take up an extra O(N * M) space (ie. the tree changes entirely after every snapshot). For append-only trees it would be more space-efficient (e.g. once we start filling the right subtree, the left subtree will be shared by every future snapshot).

More details in this comment #3207 (comment).

Append-only snapshots

For append-only trees we can have an infinite number of snapshots with just O(N + M) extra space (N - number of nodes in the tree, M - number of snapshots) at the cost of sibling paths possibly requiring O(H) hashes.

This way of storing snapshots only works for AppendOnlyTree and exploits two properties of this type of tree:

1. once a leaf is set, it can never be changed (i.e. the tree is append-only). This property can be generalised to: once a subtree is completely filled then none of its nodes can ever change.
2. there are no gaps in the leaves: at any moment in time T we can say that the first K leaves of the tree are filled and the last 2^H-K leaves are zeroes.

The algorithm stores a copy of the tree in the database:

• the last snapshot of that node (ie v1, v2, etc)
• the node value at that snapshot

Taking the snapshot is also a BF traversal of the tree, comparing the current value of nodes with its previously stored value. If the value is different we update both entries. If the values are the same then the node hasn't changed and by the first property, none of the nodes in its subtree have changed so it returns early. For each snapshot we also store how many leaves have been filled (e.g. at v1 we had 10 leaves, at v2 33 leaves, etc)

Building a sibling path is more involved and could potentially require O(H) hashes:

1. walk the historic tree from leaf to root
2. check the sibling's last snapshot

• if it's before the target snapshot version then we can safely use the node (e.g. node A was last changed at time T3 and we're building a proof for the tree at time T10, then by property 1 we can conclude that neither node A nor the subtree rooted at A will ever change at any moment in time Tn where n > 3)
• if the node has changed then we have to rebuild its historic value at the snapshot we're interested in
  • to do this we check how "wide" the subtree rooted at that node is (e.g. what's the first leaf index of the subtree) and if it intersects with the filled leaf set at that snapshot. If it doesn't intersect at all (e.g. my subtree's first leaf is 11 but only 5 leaves were filled at the time) then we can safely conclude that the whole subtree has some hash of zeros
  • if it does intersect go down one level and level apply step 2 again.
  • the invariant is: we will either reach a leaf and that leaf was changed in the version we're interested in, or we reach a node that was changed before the version we're interested in (and we return early) or we reach a node that was historically a hash of zero

Two (big) SVGs showing the sibling path algorithm step-by-step

Average case

Drawing of sibling path algorithm in the worst case

Pros:

• low space requirements only needing O(N+M) extra space.

Cons:

• generating a sibling path involves mixed workload: partly reading from the database and partly hashing. Worst case scenario O(H) db reads + O(H) hashing
• doesn't work for IndexedTrees because even though it only supports appending new leaves, internally it updates its leaf values.

Fix #3207

[alexghr]

Maybe I should rename "incremental" to "full snapshots" to differentiate the two implementations better?

[AztecBot]

Benchmark results

Metrics with a significant change:

• node_database_size_in_bytes (10): 5,015,749 (+38%)
• note_history_trial_decrypting_time_in_ms (10): 349 (+52%)

Detailed results

All benchmarks are run on txs on the Benchmarking contract on the repository. Each tx consists of a batch call to create_note and increment_balance, which guarantees that each tx has a private call, a nested private call, a public call, and a nested public call, as well as an emitted private note, an unencrypted log, and public storage read and write.

This benchmark source data is available in JSON format on S3 here.

Values are compared against data from master at commit a138860b and shown if the difference exceeds 1%.

L2 block published to L1

Each column represents the number of txs on an L2 block published to L1.

Metric	8 txs	32 txs	128 txs
l1_rollup_calldata_size_in_bytes	45,444	179,588	716,132
l1_rollup_calldata_gas	222,972	868,172	3,447,800
l1_rollup_execution_gas	842,059	3,595,280	22,203,169
l2_block_processing_time_in_ms	2,255 (+9%)	8,440 (+4%)	35,278 (+6%)
note_successful_decrypting_time_in_ms	317	931 (+1%)	3,327 (-3%)
note_trial_decrypting_time_in_ms	25.5 (-8%)	57.9 (+16%)	199
l2_block_building_time_in_ms	20,627 (+1%)	80,990	324,176
l2_block_rollup_simulation_time_in_ms	17,063 (+1%)	67,035	268,176
l2_block_public_tx_process_time_in_ms	3,514	13,806	55,444

L2 chain processing

Each column represents the number of blocks on the L2 chain where each block has 16 txs.

Metric	5 blocks	10 blocks
node_history_sync_time_in_ms	24,731 (+7%)	47,284 (+5%)
note_history_successful_decrypting_time_in_ms	2,206 (+2%)	4,278 (+2%)
note_history_trial_decrypting_time_in_ms	164 (+3%)	⚠️ 349 (+52%)
node_database_size_in_bytes	2,799,587 (-15%)	⚠️ 5,015,749 (+38%)
pxe_database_size_in_bytes	29,748	59,307

Circuits stats

Stats on running time and I/O sizes collected for every circuit run across all benchmarks.

Circuit	circuit_simulation_time_in_ms	circuit_input_size_in_bytes	circuit_output_size_in_bytes
private-kernel-init	199	43,077	20,441
private-kernel-ordering	115	25,833	9,689
base-rollup	2,957	664,108	873
root-rollup	84.6 (-3%)	4,072	1,097
private-kernel-inner	259	64,484	20,441
public-kernel-private-input	171	25,203	20,441
public-kernel-non-first-iteration	168	25,245	20,441
merge-rollup	10.8 (-2%)	2,592	873

Miscellaneous

Transaction sizes based on how many contracts are deployed in the tx.

Metric	0 deployed contracts	1 deployed contracts
tx_size_in_bytes	10,323	25,938

[LHerskind]

Good job. Think this makes a lot of sense, mostly got some nits really.

The main thing we might need to consider is that require all the snapshots here to be taking throughout the process of building.

Something that could be done to reduce space requirements would be to only store snapshots for every 10'th block, and then apply a few blocks if needing values from in between these.

[LHerskind]

Where is M used?

It is not fully clear to me that there are no influence of M, I would say you are storing at the least "something" for every snapshot because you need the meta etc.

[alexghr]

Ah yes, you're right, I totally forgot. So the space requirements would be O(N + M). O(N) to store a copy of the tree and O(M) to store for each snapshot up to which leaf index it's written to.

[LHerskind]

Would rename the var to convey the "count" as well. Use leafCountAtVersion as later as well.

[LHerskind]

What is the BF in this case, breadth first?

[alexghr]

Yes

[LHerskind]

I don't think this one is actually previous, consider changing to just nodeValueKey.

[LHerskind]

I think the naming version is a bit weird for the snapshots. As I read it, you are using it as a snapshot id, so I would rather use that to not confuse version as the variant of snapshot (full/append-only).
Also since we are expected to be using it for block numbers, might be useful to think about that.

[LHerskind]

Would be neat with a one line comment for each of these keys for what is to be stored at them.

[LHerskind]

What is the _: boolean in here? What is its purpose?

[alexghr]

It comes from the SiblingPathSource interface and it's whether to include uncommitted data in the path. Oh I could even remove the parameter and TS should still be happy 🤔

[LHerskind]

Not clear to me if we want it to throw or just to return the one it already made?

[alexghr]

Oh good shout!

[LHerskind]

The values you are matching are both defined before you modify anything so it don't make a lot of difference. Seems like the historicPaths should only be computed after the second snapshot to show it. Otherwise the test is the same as returns the same path if tree has not advanced.

I think this test can be removed as it is covered be the other ones.

[alexghr]

Fair, I think I wanted to prove to myself that snapshotting again does not modify any prior sibling paths I had generated. But now that I say it out loud the test indeed doesn't make sense 😅

[LHerskind]

As before I'm not a fan of the version naming.

[PhilWindle]

This is great. So with this we have a space efficient way of generating snapshots for append only trees. Sparse trees probably more space than we would ideally like but we may be getting rid of the only sparse tree in favour of an indexed tree.

The indexed tree is still to be modified. Hopefully we can find a way to snapshot it that is more efficient than the sparse tree.

[alexghr]

Something that could be done to reduce space requirements would be to only store snapshots for every 10'th block, and then apply a few blocks if needing values from in between these.

Yeah, that's a good idea Lasse. Controlling the granularity of snapshots would be a good thing to add. Do you think that would something we need right away or would we be able to work on it at a later date?

The indexed tree is still to be modified. Hopefully we can find a way to snapshot it that is more efficient than the sparse tree.

Just pushed a 3rd "full snapshot" implementation for indexed trees that saves the leaf values as well

[alexghr]

Heya folks, I've updated WorldState to snapshot the trees after every block. The world state now exposes a getSnapshot(block: number): MerkleTreeOperations API to look at the state as it was at the given time. LMK what you think