feat: Relations vs widgets benchmarking

[codygunton]

Add code to measure Plonk widget execution vs Honk relation execution. Outputs below.

Summary of results

Here are values of widget time/relation time:

Arithmetic  0.8342541436464088
GenPermSort 1.0814663951120163
Elliptic    0.8155940594059405
Auxiliary   0.7053435114503817
Plookup     0.7008310249307479
Permutation 1.192233009708738

Benchmark outputs

% ./bin/relations_bench                                                                                                      17s ~/barretenberg-cpp/build cg/relation-check-bench + mainframe
2024-01-10T19:27:00+00:00
Running ./bin/relations_bench
Run on (128 X 2649.99 MHz CPU s)
CPU Caches:
  L1 Data 32 KiB (x64)
  L1 Instruction 32 KiB (x64)
  L2 Unified 512 KiB (x64)
  L3 Unified 32768 KiB (x8)
Load Average: 12.24, 23.55, 50.13
--------------------------------------------------------------------------------------------------------------------------------------------
Benchmark                                                                                                  Time             CPU   Iterations
--------------------------------------------------------------------------------------------------------------------------------------------
execute_relation<honk::flavor::Ultra, UltraArithmeticRelation<Fr>>                                       543 ns          543 ns      1289230
execute_relation<honk::flavor::Ultra, GenPermSortRelation<Fr>>                                           491 ns          491 ns      1423778
execute_relation<honk::flavor::Ultra, EllipticRelation<Fr>>                                              808 ns          808 ns       866448
execute_relation<honk::flavor::Ultra, AuxiliaryRelation<Fr>>                                            1965 ns         1965 ns       357614
execute_relation<honk::flavor::Ultra, LookupRelation<Fr>>                                                722 ns          722 ns       969890
execute_relation<honk::flavor::Ultra, UltraPermutationRelation<Fr>>                                      515 ns          515 ns      1358778

% taskset -c 0 ./bin/widget_bench                                                                                        1m 57s ~/barretenberg-cpp/build cg/relation-check-bench + mainframe
2024-01-10T19:15:11+00:00
Running ./bin/widget_bench
Run on (128 X 2649.99 MHz CPU s)
CPU Caches:
  L1 Data 32 KiB (x64)
  L1 Instruction 32 KiB (x64)
  L2 Unified 512 KiB (x64)
  L3 Unified 32768 KiB (x8)
Load Average: 25.77, 74.54, 79.34
----------------------------------------------------------------------------------------------------------------------------------------
Benchmark                                                                                              Time             CPU   Iterations
----------------------------------------------------------------------------------------------------------------------------------------
quotient_contribution<ProverPlookupArithmeticWidget<ultra_settings>>/iterations:1/manual_time        453 ns       186172 ns            1
quotient_contribution<ProverGenPermSortWidget<ultra_settings>>/iterations:1/manual_time              531 ns       165501 ns            1
quotient_contribution<ProverEllipticWidget<ultra_settings>>/iterations:1/manual_time                 659 ns       174561 ns            1
quotient_contribution<ProverPlookupAuxiliaryWidget<ultra_settings>>/iterations:1/manual_time        1386 ns       206071 ns            1
quotient_contribution<ProverPlookupWidget<4>>/iterations:1/manual_time                               506 ns       188280 ns            1
quotient_contribution<ProverPermutationWidget<4, true>>/iterations:1/manual_time                     614 ns       167110 ns            1

[ludamad]

What's the rationale with making them all iterations 1? just that it takes long? (the way I did rounds was pretty inefficient but simpler with how googlebench is setup)

[codygunton]

I think I got impatient while working on it, will revert.

[AztecBot]

Benchmark results

No metrics with a significant change found.

Detailed results

All benchmarks are run on txs on the Benchmarking contract on the repository. Each tx consists of a batch call to create_note and increment_balance, which guarantees that each tx has a private call, a nested private call, a public call, and a nested public call, as well as an emitted private note, an unencrypted log, and public storage read and write.

This benchmark source data is available in JSON format on S3 here.

Values are compared against data from master at commit fbaa7264 and shown if the difference exceeds 1%.

L2 block published to L1

Each column represents the number of txs on an L2 block published to L1.

Metric	8 txs	32 txs	128 txs
l1_rollup_calldata_size_in_bytes	45,444	179,588	716,132
l1_rollup_calldata_gas	221,004	866,456	3,448,028
l1_rollup_execution_gas	934,148	3,968,930	24,791,564
l2_block_processing_time_in_ms	1,337	5,157 (+1%)	21,119
note_successful_decrypting_time_in_ms	327	1,016 (+1%)	3,717
note_trial_decrypting_time_in_ms	38.5 (+4%)	38.4 (-2%)	136 (-27%)
l2_block_building_time_in_ms	14,998 (+1%)	59,231	237,253
l2_block_rollup_simulation_time_in_ms	10,517	42,007 (+1%)	168,645
l2_block_public_tx_process_time_in_ms	4,398 (+2%)	17,146	68,345

L2 chain processing

Each column represents the number of blocks on the L2 chain where each block has 16 txs.

Metric	5 blocks	10 blocks
node_history_sync_time_in_ms	15,061 (-2%)	29,134 (-1%)
note_history_successful_decrypting_time_in_ms	2,582 (+6%)	4,624 (-3%)
note_history_trial_decrypting_time_in_ms	78.0 (-10%)	156 (-15%)
node_database_size_in_bytes	3,619,830	3,719,966
pxe_database_size_in_bytes	29,923	59,478

Circuits stats

Stats on running time and I/O sizes collected for every circuit run across all benchmarks.

Circuit	circuit_simulation_time_in_ms	circuit_input_size_in_bytes	circuit_output_size_in_bytes
private-kernel-init	211 (-1%)	44,709	25,561
private-kernel-ordering	138 (-1%)	30,953	14,809
base-rollup	1,064 (-1%)	128,338	881
root-rollup	80.7 (-1%)	4,088	709
private-kernel-inner	288 (-1%)	71,236	25,561
public-kernel-private-input	203	31,891	25,561
public-kernel-non-first-iteration	201 (-1%)	31,933	25,561
merge-rollup	8.42	2,608	881

Tree insertion stats

The duration to insert a fixed batch of leaves into each tree type.

Metric	1 leaves	2 leaves	8 leaves	16 leaves	32 leaves	64 leaves	128 leaves	512 leaves	1024 leaves	2048 leaves	8192 leaves
batch_insert_into_append_only_tree_ms	11.1 (-1%)	88.2 (+2%)	13.6 (-12%)	18.1 (+1%)	23.8	49.6 (-1%)	88.1 (-5%)	244 (+1%)	468 (-2%)	916 (+1%)	3,563
batch_insert_into_indexed_tree_ms	N/A	N/A	N/A	63.9 (-2%)	N/A	60.0	120 (+8%)	362	729 (-2%)	1,415 (+1%)	5,568 (-2%)

Miscellaneous

Transaction sizes based on how many contracts are deployed in the tx.

Metric	0 deployed contracts	1 deployed contracts
tx_size_in_bytes	15,487	31,907

Transaction processing duration by data writes.

Metric	0 new commitments	1 new commitments
tx_pxe_processing_time_ms	345	965 (-1%)

Metric	1 public data writes
tx_sequencer_processing_time_ms	532

[ledwards2225]

Basically looks good but I'm confused about the input of 80 to the arith circuit construction function. Once that's cleared up all else looks good.

[ledwards2225]

Pretty confused about what's going on here. Why is this 80? This is creating a circuit with 2^80 gates, no? The input was intended to be log_2 number of gates and as far as I can tell that's still how it works. Is this not being used as I think?

[codygunton]

Lol it's overflowing: $2^{80} = 2^{16} \mod 2^{32}$. I printed passes in the utility function body to double check and indeed it is equal to $2^{16-2}-4$. I'll just set 80 to 16.

[ledwards2225]

This isn't new to this PR but the (hidden) comment below:
"// NOTE: this simply calls the following 3 functions it does NOT try to replicate ProverPlookupAuxiliaryWidget
// logic exactly"
is now outdated

[codygunton]

Removed

[ledwards2225]

LGTM