feat: benchmark Protogalaxy rounds

[codygunton]

Splits folding into multiple functions for benchmarking.

Parallelizes the loop in compute_full_honk_evaluations, which results in a 5x speedup in the perturbator round.

Effect on Client IVC

Without parallel construction of perturbator

ivc_bench                                                                                   100%   17MB  58.9MB/s   00:00    
2024-02-21T12:13:53+00:00
Running ./ivc_bench
Run on (16 X 3000 MHz CPU s)
CPU Caches:
  L1 Data 32 KiB (x8)
  L1 Instruction 32 KiB (x8)
  L2 Unified 1024 KiB (x8)
  L3 Unified 36608 KiB (x1)
Load Average: 0.00, 0.24, 0.84
----------------------------------------------------------
Benchmark                Time             CPU   Iterations
----------------------------------------------------------
IvcBench/Full/6      53047 ms        50603 ms            1

With parallel construction of perturbator

ivc_bench                                                                                   100%   17MB  62.3MB/s   00:00    
2024-02-21T12:05:37+00:00
Running ./ivc_bench
Run on (16 X 3000 MHz CPU s)
CPU Caches:
  L1 Data 32 KiB (x8)
  L1 Instruction 32 KiB (x8)
  L2 Unified 1024 KiB (x8)
  L3 Unified 36608 KiB (x1)
Load Average: 0.00, 0.62, 1.25
----------------------------------------------------------
Benchmark                Time             CPU   Iterations
----------------------------------------------------------
IvcBench/Full/6      42092 ms        39692 ms            1

Other benchmarks

Before optimization:

Benchmarking lock created at ~/BENCHMARK_IN_PROGRESS.
protogalaxy_bench                                                             100% 8050KB  37.1MB/s   00:00    
2024-02-09T21:51:45+00:00
Running ./protogalaxy_bench
Run on (16 X 3000 MHz CPU s)
CPU Caches:
  L1 Data 32 KiB (x8)
  L1 Instruction 32 KiB (x8)
  L2 Unified 1024 KiB (x8)
  L3 Unified 36608 KiB (x1)
Load Average: 0.00, 0.00, 0.00
------------------------------------------------------
Benchmark            Time             CPU   Iterations
------------------------------------------------------
fold_one/14        311 ms          302 ms            2
fold_one/15        601 ms          582 ms            1
fold_one/16       1159 ms         1125 ms            1
fold_one/17       2317 ms         2248 ms            1
fold_one/18       4578 ms         4423 ms            1
fold_one/19       8997 ms         8694 ms            1
fold_one/20      17837 ms        17273 ms            1

After optimization (comparison only with UltraHonk) and addition of new benchmarks:

Benchmarking lock created at ~/BENCHMARK_IN_PROGRESS.
protogalaxy_bench                                                             100% 8038KB  36.8MB/s   00:00    
2024-02-09T21:54:16+00:00
Running ./protogalaxy_bench
Run on (16 X 3000 MHz CPU s)
CPU Caches:
  L1 Data 32 KiB (x8)
  L1 Instruction 32 KiB (x8)
  L2 Unified 1024 KiB (x8)
  L3 Unified 36608 KiB (x1)
Load Average: 1.06, 1.10, 0.47
---------------------------------------------------------------------------
Benchmark                                 Time             CPU   Iterations
---------------------------------------------------------------------------
fold_one<UltraComposer>/14              234 ms          223 ms            3
fold_one<UltraComposer>/15              438 ms          418 ms            2
fold_one<UltraComposer>/16              836 ms          800 ms            1
fold_one<UltraComposer>/17             1657 ms         1585 ms            1
fold_one<UltraComposer>/18             3316 ms         3131 ms            1
fold_one<UltraComposer>/19             6471 ms         6131 ms            1
fold_one<UltraComposer>/20            12638 ms        12022 ms            1
fold_one<GoblinUltraComposer>/14        633 ms          553 ms            1
fold_one<GoblinUltraComposer>/15       1175 ms         1069 ms            1
fold_one<GoblinUltraComposer>/16       2271 ms         2092 ms            1
fold_one<GoblinUltraComposer>/17       4794 ms         4218 ms            1
fold_one<GoblinUltraComposer>/18       9317 ms         8415 ms            1
fold_one<GoblinUltraComposer>/19      18220 ms        16743 ms            1
fold_one<GoblinUltraComposer>/20      38419 ms        34026 ms            1

The round benchmarks:

Benchmarking lock created at ~/BENCHMARK_IN_PROGRESS.
protogalaxy_round_bench                                                                                                                                                                                                                                                                                    100% 8049KB  40.3MB/s   00:00    
2024-02-12T21:51:43+00:00
Running ./protogalaxy_round_bench
Run on (16 X 3000 MHz CPU s)
CPU Caches:
  L1 Data 32 KiB (x8)
  L1 Instruction 32 KiB (x8)
  L2 Unified 1024 KiB (x8)
  L3 Unified 36608 KiB (x1)
Load Average: 0.26, 1.03, 0.92
-----------------------------------------------------------------------------------------
Benchmark                                               Time             CPU   Iterations
-----------------------------------------------------------------------------------------
bench_round_goblin_ultra/preparation/14               345 ms          234 ms            3
bench_round_goblin_ultra/preparation/15               575 ms          446 ms            2
bench_round_goblin_ultra/preparation/16              1045 ms          861 ms            1
bench_round_goblin_ultra/preparation/17              2376 ms         1606 ms            1
bench_round_goblin_ultra/preparation/18              4161 ms         3160 ms            1
bench_round_goblin_ultra/preparation/19              7819 ms         6411 ms            1
bench_round_goblin_ultra/preparation/20             18265 ms        12433 ms            1
bench_round_goblin_ultra/perturbator/14              39.2 ms         39.0 ms           18
bench_round_goblin_ultra/perturbator/15              78.8 ms         78.3 ms            9
bench_round_goblin_ultra/perturbator/16               160 ms          159 ms            4
bench_round_goblin_ultra/perturbator/17               325 ms          324 ms            2
bench_round_goblin_ultra/perturbator/18               660 ms          657 ms            1
bench_round_goblin_ultra/perturbator/19              1327 ms         1319 ms            1
bench_round_goblin_ultra/perturbator/20              2680 ms         2659 ms            1
bench_round_goblin_ultra/combiner_quotient/14         189 ms          187 ms            4
bench_round_goblin_ultra/combiner_quotient/15         380 ms          372 ms            2
bench_round_goblin_ultra/combiner_quotient/16         748 ms          742 ms            1
bench_round_goblin_ultra/combiner_quotient/17        1509 ms         1488 ms            1
bench_round_goblin_ultra/combiner_quotient/18        3017 ms         2998 ms            1
bench_round_goblin_ultra/combiner_quotient/19        6024 ms         5978 ms            1
bench_round_goblin_ultra/combiner_quotient/20       12055 ms        11989 ms            1
bench_round_goblin_ultra/accumulator_update/14       89.0 ms         89.0 ms            8
bench_round_goblin_ultra/accumulator_update/15        169 ms          169 ms            4
bench_round_goblin_ultra/accumulator_update/16        327 ms          327 ms            2
bench_round_goblin_ultra/accumulator_update/17        686 ms          686 ms            1
bench_round_goblin_ultra/accumulator_update/18       1371 ms         1371 ms            1
bench_round_goblin_ultra/accumulator_update/19       3102 ms         3102 ms            1
bench_round_goblin_ultra/accumulator_update/20       6552 ms         6552 ms            1

[AztecBot]

Benchmark results

No metrics with a significant change found.

Detailed results

All benchmarks are run on txs on the Benchmarking contract on the repository. Each tx consists of a batch call to create_note and increment_balance, which guarantees that each tx has a private call, a nested private call, a public call, and a nested public call, as well as an emitted private note, an unencrypted log, and public storage read and write.

This benchmark source data is available in JSON format on S3 here.

Values are compared against data from master at commit 382626cd and shown if the difference exceeds 1%.

L2 block published to L1

Each column represents the number of txs on an L2 block published to L1.

Metric	8 txs	32 txs	128 txs
l1_rollup_calldata_size_in_bytes	45,956	181,604	724,196
l1_rollup_calldata_gas	227,168	888,800	3,536,048
l1_rollup_execution_gas	320,359	1,007,248	3,756,426
l2_block_processing_time_in_ms	1,155 (-4%)	4,392 (-2%)	17,375 (-2%)
note_successful_decrypting_time_in_ms	321	992 (+1%)	3,662
note_trial_decrypting_time_in_ms	31.1 (-4%)	125 (+4%)	146 (-15%)
l2_block_building_time_in_ms	18,211	72,075	289,497
l2_block_rollup_simulation_time_in_ms	13,505	53,422	215,185
l2_block_public_tx_process_time_in_ms	4,670 (-1%)	18,576 (-1%)	74,049 (-1%)

L2 chain processing

Each column represents the number of blocks on the L2 chain where each block has 16 txs.

Metric	5 blocks	10 blocks
node_history_sync_time_in_ms	14,925 (-2%)	28,768 (-6%)
note_history_successful_decrypting_time_in_ms	2,313 (-7%)	4,539 (-8%)
note_history_trial_decrypting_time_in_ms	79.5 (-20%)	200 (-12%)
node_database_size_in_bytes	17,264,720	33,243,216
pxe_database_size_in_bytes	29,923	59,478

Circuits stats

Stats on running time and I/O sizes collected for every circuit run across all benchmarks.

Circuit	circuit_simulation_time_in_ms	circuit_input_size_in_bytes	circuit_output_size_in_bytes
private-kernel-init	327	44,528	25,313
private-kernel-ordering	196	43,633	16,193
base-rollup	1,457	129,774	933
root-rollup	83.1 (-2%)	4,192	729
private-kernel-inner	436	70,819	25,313
public-kernel-private-input	239 (-1%)	33,663	26,945
public-kernel-non-first-iteration	238	33,705	26,945
merge-rollup	7.86 (-1%)	2,712	933

Tree insertion stats

The duration to insert a fixed batch of leaves into each tree type.

Metric	1 leaves	2 leaves	8 leaves	16 leaves	32 leaves	128 leaves	64 leaves	512 leaves	1024 leaves	2048 leaves	8192 leaves
batch_insert_into_append_only_tree_16_depth_ms	10.2 (-1%)	10.8 (-11%)	12.5 (-2%)	16.7 (-3%)	23.1	64.1	N/A	N/A	N/A	N/A	N/A
batch_insert_into_append_only_tree_16_depth_hash_count	16.9	17.5	23.0	31.6	47.0	143	N/A	N/A	N/A	N/A	N/A
batch_insert_into_append_only_tree_16_depth_hash_ms	0.594 (-1%)	0.601 (-7%)	0.533 (-2%)	0.517 (-3%)	0.482	0.442	N/A	N/A	N/A	N/A	N/A
batch_insert_into_append_only_tree_32_depth_ms	N/A	N/A	N/A	N/A	N/A	75.0	47.5	238 (-1%)	459 (-2%)	906 (+1%)	3,570 (+1%)
batch_insert_into_append_only_tree_32_depth_hash_count	N/A	N/A	N/A	N/A	N/A	159	96.0	543	1,055	2,079	8,223
batch_insert_into_append_only_tree_32_depth_hash_ms	N/A	N/A	N/A	N/A	N/A	0.463	0.487	0.435	0.431 (-2%)	0.430 (+1%)	0.429 (+1%)
batch_insert_into_indexed_tree_20_depth_ms	N/A	N/A	N/A	N/A	N/A	104 (-1%)	56.8	353	686 (-2%)	1,384 (+1%)	5,458
batch_insert_into_indexed_tree_20_depth_hash_count	N/A	N/A	N/A	N/A	N/A	197	104	691	1,363	2,707	10,771
batch_insert_into_indexed_tree_20_depth_hash_ms	N/A	N/A	N/A	N/A	N/A	0.494 (-1%)	0.496	0.477	0.472 (-1%)	0.479 (+1%)	0.474
batch_insert_into_indexed_tree_40_depth_ms	N/A	N/A	N/A	56.5 (-1%)	N/A	N/A	N/A	N/A	N/A	N/A	N/A
batch_insert_into_indexed_tree_40_depth_hash_count	N/A	N/A	N/A	94.1	N/A	N/A	N/A	N/A	N/A	N/A	N/A
batch_insert_into_indexed_tree_40_depth_hash_ms	N/A	N/A	N/A	0.577	N/A	N/A	N/A	N/A	N/A	N/A	N/A

Miscellaneous

Transaction sizes based on how many contracts are deployed in the tx.

Metric	0 deployed contracts	1 deployed contracts
tx_size_in_bytes	16,859	39,794 (+12%)

Transaction processing duration by data writes.

Metric	0 new commitments	1 new commitments
tx_pxe_processing_time_ms	553 (-1%)	1,447

Metric	0 public data writes	1 public data writes
tx_sequencer_processing_time_ms	0.519 (-5%)	574

[ludamad]

Any reason these are lambdas instead of normal funcs?

[lucasxia01]

it’s an intermediate step that Cody took changing between one big function and many split up functions

[ludamad]

As mentioned this is never actually ran in parallel

[ledwards2225]

LGTM

[codygunton]

Sanity check:
For size 14, the round times sum to $345 + 39.2 + 189 + 89.0 = 662.2$, and the standalone bench is at $633$ ($5 \%$ more).
For size 20, the round times sum to $18265 + 2680 +12055 + 6552 = 39552$, and the standalone bench is at $38419$ ($3\%$ less).