refactor: towards plain struct flavor entities

[ludamad]

Step towards removing the array backing in the flavor entity classes. This systematically removes all usage of operator[] so that the array backing can be deleted in a followup. This is achieved by having a pointer_view method that is defined in const and non-const forms (cleanest way I could find was through a simple macro) and returns an array of each pointer. Pointers were used over references to not have to deal with references decaying into values.

Happy to take feedback on the approach, it isn't cleaning up much just yet, but will make way for a followup PR that removes the backing arrays.

Also:

• Adds a zip_view class that aims to get us using zip_view before C++2023 fully lands. I yoinked some open source and slimmed it down a bit for what we're likely to use.

[ledwards2225]

Everything here seems fine to me but it's a bit hard for me to evaluate without having a full picture of how the final structure will look. Normally I'm all for the incremental approach but do you feel confident that you've thought through the simplifications that this will allow in the flavor classes? I'm a bit worried that if this ultimately relies on different collections of polys being adjacent in the struct (e.g. all "precomputed" or "to-be-shifted" are grouped together and accessible via a contiguous range) then this will lead to difficulties and possibly be too restrictive. Of course maybe we could explicitly specify subsets but then I'm not sure how much we've gained. Perhaps I just don't have enough insight into the plan - just want to make sure the path forward is clear enough to get in this intermediate work. Happy that you're on the case!

[ludamad]

Summarizing slack discussion: Luke and I got to the understanding that this won't be that much of an overhaul, it will largely keep stuff as-is, this is the most dramatic of the rewriting right here, the followup would just be using this to remove the array backing so these structs don't have self-references to their own data (and thus won't be footguns that can't be copy-constructed)

[AztecBot]

Benchmark results

No metrics with a significant change found.

Detailed results

All benchmarks are run on txs on the Benchmarking contract on the repository. Each tx consists of a batch call to create_note and increment_balance, which guarantees that each tx has a private call, a nested private call, a public call, and a nested public call, as well as an emitted private note, an unencrypted log, and public storage read and write.

This benchmark source data is available in JSON format on S3 here.

Values are compared against data from master at commit ec2b805e and shown if the difference exceeds 1%.

L2 block published to L1

Each column represents the number of txs on an L2 block published to L1.

Metric	8 txs	32 txs	128 txs
l1_rollup_calldata_size_in_bytes	45,444	179,588	716,132
l1_rollup_calldata_gas	222,912	868,268	3,449,648
l1_rollup_execution_gas	841,999	3,595,376	22,205,017
l2_block_processing_time_in_ms	2,023 (+1%)	7,576 (-1%)	29,890 (-1%)
note_successful_decrypting_time_in_ms	292 (-1%)	870 (-1%)	3,181 (-2%)
note_trial_decrypting_time_in_ms	93.7 (-4%)	78.2 (-11%)	134
l2_block_building_time_in_ms	13,374 (+1%)	52,695	210,635
l2_block_rollup_simulation_time_in_ms	12,021 (+1%)	47,399	189,520
l2_block_public_tx_process_time_in_ms	1,312 (-1%)	5,166 (-1%)	20,605

L2 chain processing

Each column represents the number of blocks on the L2 chain where each block has 16 txs.

Metric	5 blocks	10 blocks
node_history_sync_time_in_ms	21,802	42,320
note_history_successful_decrypting_time_in_ms	2,021	3,948 (+1%)
note_history_trial_decrypting_time_in_ms	121	143 (+1%)
node_database_size_in_bytes	1,628,701	1,096,881
pxe_database_size_in_bytes	29,748	59,307

Circuits stats

Stats on running time and I/O sizes collected for every circuit run across all benchmarks.

Circuit	circuit_simulation_time_in_ms	circuit_input_size_in_bytes	circuit_output_size_in_bytes
private-kernel-init	762	61,697	18,905
private-kernel-ordering	125	24,297	8,153
base-rollup	1,766	656,311	814
root-rollup	169	4,072	1,097
private-kernel-inner	786	81,568	18,905
public-kernel-private-input	41.1	41,519	18,841
public-kernel-non-first-iteration	25.8	41,497	18,841
merge-rollup	15.1	2,592	873

Miscellaneous

Transaction sizes based on how many contracts are deployed in the tx.

Metric	0 deployed contracts	1 deployed contracts
tx_size_in_bytes	8,787	27,547

[ludamad]

This was not building in its own folder, nor was it compatible with the gdb on mainframe

[ledwards2225]

All LGTM

[ledwards2225]

Can this not be zipped?