feat: refactor StandardIndexedTree for abstract leaves and preimages and optimized it

[sirasistant]

This PR prepares the standard indexed tree to be used for the public data tree. Changes:

• Make StandardIndexedTree abstract in leaves in preparation to accomodate other trees than the nullifier tree
• Removed the leaves in memory in the StandardIndexedTree and switch to db queries for low leaf finding (O(1))
• FindLeafIndex is now O(1) too

Checklist:

Remove the checklist to signal you've completed it. Enable auto-merge if the PR is ready to merge.

• If the pull request requires a cryptography review (e.g. cryptographic algorithm implementations) I have added the 'crypto' tag.
• I have reviewed my diff in github, line by line and removed unexpected formatting changes, testing logs, or commented-out code.
• Every change is related to the PR description.
• I have linked this pull request to relevant issues (if any exist).

[AztecBot]

Benchmark results

Metrics with a significant change:

• node_database_size_in_bytes (5): 3,628,812 (+30%)
• node_database_size_in_bytes (10): 3,914,928 (-22%)
• note_history_trial_decrypting_time_in_ms (10): 367 (+59%)
• note_trial_decrypting_time_in_ms (8): 17.1 (-36%)
• note_trial_decrypting_time_in_ms (32): 39.8 (-36%)

Detailed results

All benchmarks are run on txs on the Benchmarking contract on the repository. Each tx consists of a batch call to create_note and increment_balance, which guarantees that each tx has a private call, a nested private call, a public call, and a nested public call, as well as an emitted private note, an unencrypted log, and public storage read and write.

This benchmark source data is available in JSON format on S3 here.

Values are compared against data from master at commit 9f682cb8 and shown if the difference exceeds 1%.

L2 block published to L1

Each column represents the number of txs on an L2 block published to L1.

Metric	8 txs	32 txs	128 txs
l1_rollup_calldata_size_in_bytes	45,444	179,588	716,132
l1_rollup_calldata_gas	222,972	867,920	3,449,540
l1_rollup_execution_gas	842,059	3,595,028	22,204,909
l2_block_processing_time_in_ms	2,207	8,455 (-2%)	34,946 (-1%)
note_successful_decrypting_time_in_ms	309 (-2%)	913 (-2%)	3,311 (-5%)
note_trial_decrypting_time_in_ms	⚠️ 17.1 (-36%)	⚠️ 39.8 (-36%)	200 (-1%)
l2_block_building_time_in_ms	20,146 (-1%)	80,155 (-2%)	320,349 (-2%)
l2_block_rollup_simulation_time_in_ms	16,624 (-1%)	66,297 (-2%)	264,734 (-2%)
l2_block_public_tx_process_time_in_ms	3,490	13,788 (-1%)	55,373 (-1%)

L2 chain processing

Each column represents the number of blocks on the L2 chain where each block has 16 txs.

Metric	5 blocks	10 blocks
node_history_sync_time_in_ms	24,614 (-2%)	48,042 (-1%)
note_history_successful_decrypting_time_in_ms	2,231 (-2%)	4,355 (-1%)
note_history_trial_decrypting_time_in_ms	165 (-1%)	⚠️ 367 (+59%)
node_database_size_in_bytes	⚠️ 3,628,812 (+30%)	⚠️ 3,914,928 (-22%)
pxe_database_size_in_bytes	29,748	59,307

Circuits stats

Stats on running time and I/O sizes collected for every circuit run across all benchmarks.

Circuit	circuit_simulation_time_in_ms	circuit_input_size_in_bytes	circuit_output_size_in_bytes
private-kernel-init	199	43,109	20,441
private-kernel-ordering	115	25,833	9,689
base-rollup	2,956	667,692 (+1%)	873
root-rollup	84.5 (-8%)	4,072	881
private-kernel-inner	260	64,516	20,441
public-kernel-private-input	172	25,203	20,441
public-kernel-non-first-iteration	168 (-1%)	25,245	20,441
merge-rollup	10.8 (-2%)	2,592	873

Miscellaneous

Transaction sizes based on how many contracts are deployed in the tx.

Metric	0 deployed contracts	1 deployed contracts
tx_size_in_bytes	10,323	25,938

[LeilaWang]

Can we call it treeBuilder so that we don't have to change its name when exporting?

[LeilaWang]

This is very cute 😂

[LeilaWang]

Should be index !== undefined?

[LeilaWang]

Suggested change

	const cachedLowLeafPreimage =
	cachedLowLeafIndex !== undefined ? this.getCachedPreimage(cachedLowLeafIndex) : undefined;
	if (cachedLowLeafIndex && (!lowLeafIndex || cachedLowLeafPreimage!.getKey() > lowLeafPreimage!.getKey())) {
	if (cachedLowLeafIndex !== undefined) {
	const cachedLowLeafPreimage = this.getCachedPreimage(cachedLowLeafIndex)!;
	if (!lowLeafPreimage || cachedLowLeafPreimage.getKey() > lowLeafPreimage.getKey()) {}
	}

[LeilaWang]

Shouldn't treeHeight always be this.depth?

[sirasistant]

Yeah, I included this as a class method but I was facing some generics funkyness when trying to use this.getHeight() as the generic of LowLeafWitnessdata. I'll see if I can get it resolved :D

[LeilaWang]

This won't work for sparse tree though, and sparse tree extends this.

[sirasistant]

Oh, this was previously in the merkle_trees db classes that potentially applied this algorithm for any tree. Let me check...

[LeilaWang]

Can it always use treeBuilder here so we can still pass treeClass to it?

[sirasistant]

So the issue here was:

• newTree and loadTree were called with constructor (typed with new) functions, which all had the same shape before
• now indexed trees have different constructors, because they require the leaf factory and the preimage factory
• Typescript won't let me use the new operator if the function is not typed as a constructor (which makes sense)
• So now newTree and loadTree sometimes need to issue "new" and sometimes not

My solution was ok, let's accept regular functions, and if you are passing a regular constructor just wrap it in a regular function via treeBuilder and you're set. If you are passing a more complex constructor then just wrap it in a closure that provides the factories needed for that specific tree instance.

I tried to type newTree and loadTree with a union type of constructor and regular function, but I didn't find a clean way to narrow the type so it allowed me to do new sometimes and just calling other times