[Tracking Issue] Benchmarks #1663
Labels
A-benchmarking
Area: relates to benchmarking
Comments
Merged
|
Tracking here the status for each metric:
|
spalladino
added a commit
that referenced
this issue
Oct 3, 2023
Initial version of benchmarks for end-to-end tests. - Adds support for structured logging. All log methods now accept a string as message (requiring the caller to use `util.format` if needed) and an optional object. The object is passed directly to each entry when logging to JSONL. - Exports stats associated to publishing a rollup to L1 as first use case. Stats are collected using viem and exported with the structured logging format. Adds a first `bench-` e2e test that runs multiple rollup sizes to collect stats. - Collects benchmark data as JSONL from all e2e tests and uploads it to a new S3 bucket `aztec-ci-artifacts` (private for now) using the name of the job and the commit identifier. Benchmark data is any entry that has an `eventName` associated. - Once all e2e tests are finished, aggregates benchmark data from e2e tests named `bench*`, filtering by the metrics requested in #1663, and reuploading it to S3 and adding it to the PR as a comment. The JSONL files extracted from each test look like: ```jsonl {"blockNumber":1,"calldataGas":104732,"calldataSize":22340,"encryptedLogCount":2,"encryptedLogSize":516,"eventName":"rollup-published-to-l1","gasPrice":"1500000008","gasUsed":"422583","level":"info","message":"Published L2 block to L1 rollup contract","namespace":"aztec:sequencer:publisher","timestamp":"2023-09-29T20:14:30.534Z","transactionHash":"0x47724f679d813ec6bdb87002d8ed994abd8f11382eadc81bb41b3de0d7087db7","txCount":4,"unencryptedLogCount":0,"unencryptedLogSize":28} {"blockNumber":2,"calldataGas":93852,"calldataSize":21828,"encryptedLogCount":0,"encryptedLogSize":24,"eventName":"rollup-published-to-l1","gasPrice":"1500000008","gasUsed":"370654","level":"info","message":"Published L2 block to L1 rollup contract","namespace":"aztec:sequencer:publisher","timestamp":"2023-09-29T20:14:35.246Z","transactionHash":"0x1f9ad018f0a7a8f4414a7f789db7baa9b885e524c10a43e2e00d864b4f700721","txCount":4,"unencryptedLogCount":0,"unencryptedLogSize":24} {"blockNumber":3,"calldataGas":94016,"calldataSize":21860,"encryptedLogCount":0,"encryptedLogSize":24,"eventName":"rollup-published-to-l1","gasPrice":"1500000008","gasUsed":"373541","level":"info","message":"Published L2 block to L1 rollup contract","namespace":"aztec:sequencer:publisher","timestamp":"2023-09-29T20:14:41.191Z","transactionHash":"0xf919ea0083420d3bf45263c28797b024f6447ba046ea3d2fba7c116abbac54a7","txCount":4,"unencryptedLogCount":0,"unencryptedLogSize":28} {"blockNumber":4,"calldataGas":93680,"calldataSize":21860,"encryptedLogCount":0,"encryptedLogSize":24,"eventName":"rollup-published-to-l1","gasPrice":"1500000008","gasUsed":"373205","level":"info","message":"Published L2 block to L1 rollup contract","namespace":"aztec:sequencer:publisher","timestamp":"2023-09-29T20:14:46.089Z","transactionHash":"0x34f386ead9e72cbef2ed8b79cc1cffc4540ec9317811d7381d0311e980c90493","txCount":4,"unencryptedLogCount":0,"unencryptedLogSize":28} {"blockNumber":5,"calldataGas":186888,"calldataSize":43044,"encryptedLogCount":0,"encryptedLogSize":68,"eventName":"rollup-published-to-l1","gasPrice":"1500000008","gasUsed":"730533","level":"info","message":"Published L2 block to L1 rollup contract","namespace":"aztec:sequencer:publisher","timestamp":"2023-09-29T20:14:59.600Z","transactionHash":"0xa4ebaeb35dee76bdf8ff3313ede44cf2c8553108d724285eac8c7f1ac47adfdb","txCount":8,"unencryptedLogCount":0,"unencryptedLogSize":100} {"blockNumber":1,"calldataGas":104720,"calldataSize":22340,"encryptedLogCount":2,"encryptedLogSize":516,"eventName":"rollup-published-to-l1","gasPrice":"1500000008","gasUsed":"422571","level":"info","message":"Published L2 block to L1 rollup contract","namespace":"aztec:sequencer:publisher","timestamp":"2023-09-29T20:15:08.321Z","transactionHash":"0x5fd0ea8e8f0bddcd62d178751154f98d10ef234f69ed44f265628eac2b6e3110","txCount":4,"unencryptedLogCount":0,"unencryptedLogSize":28} {"blockNumber":2,"calldataGas":93900,"calldataSize":21828,"encryptedLogCount":0,"encryptedLogSize":24,"eventName":"rollup-published-to-l1","gasPrice":"1500000008","gasUsed":"370702","level":"info","message":"Published L2 block to L1 rollup contract","namespace":"aztec:sequencer:publisher","timestamp":"2023-09-29T20:15:12.976Z","transactionHash":"0x4c30b46ae0ba3f90622843fe363a70300ae8c71e25ad3487e6d43650326d41b1","txCount":4,"unencryptedLogCount":0,"unencryptedLogSize":24} {"blockNumber":3,"calldataGas":94028,"calldataSize":21860,"encryptedLogCount":0,"encryptedLogSize":24,"eventName":"rollup-published-to-l1","gasPrice":"1500000008","gasUsed":"373553","level":"info","message":"Published L2 block to L1 rollup contract","namespace":"aztec:sequencer:publisher","timestamp":"2023-09-29T20:15:16.832Z","transactionHash":"0x7641b6ad965ab2bacc4d6c860de2eb8a08b0e8fd4e01e99f10c5541c20c996a5","txCount":4,"unencryptedLogCount":0,"unencryptedLogSize":28} {"blockNumber":4,"calldataGas":93692,"calldataSize":21860,"encryptedLogCount":0,"encryptedLogSize":24,"eventName":"rollup-published-to-l1","gasPrice":"1500000008","gasUsed":"373217","level":"info","message":"Published L2 block to L1 rollup contract","namespace":"aztec:sequencer:publisher","timestamp":"2023-09-29T20:15:21.969Z","transactionHash":"0x7c13aa9ebd8ec280f99b4dbcc2505cfca08034ec8d22fcf850ffb17fd5493228","txCount":4,"unencryptedLogCount":0,"unencryptedLogSize":28} {"blockNumber":5,"calldataGas":365728,"calldataSize":85348,"encryptedLogCount":0,"encryptedLogSize":132,"eventName":"rollup-published-to-l1","gasPrice":"1500000008","gasUsed":"1444415","level":"info","message":"Published L2 block to L1 rollup contract","namespace":"aztec:sequencer:publisher","timestamp":"2023-09-29T20:15:45.668Z","transactionHash":"0x0ec877bc7a6c30a7879fffe5c9ffd2d72df28f5e46971519def7385790bd1deb","txCount":16,"unencryptedLogCount":0,"unencryptedLogSize":196} ``` Sample S3 structure after running this PR: ``` aztec-ci-artifacts ├── benchmarks-v1 [public] │ └── pulls │ └── 2605.json └── logs-v1 [private] └── pulls └── 2605 ├── archiver.jsonl ├── benchmarks_publish_rollup.jsonl ├── e2e_2_pxes.jsonl ├── e2e_account_contracts.jsonl ├── e2e_card_game.jsonl ├── e2e_cross_chain_messaging.jsonl ├── e2e_deploy_contract.jsonl ├── e2e_escrow_contract.jsonl ├── e2e_multiple_accounts_1_enc_key.jsonl ├── e2e_nested_contract.jsonl ├── e2e_non_contract_account.jsonl ├── e2e_ordering.jsonl ├── e2e_p2p_network.jsonl ├── e2e_pending_commitments_contract.jsonl ├── e2e_private_token_contract.jsonl ├── e2e_public_cross_chain_messaging.jsonl ├── e2e_public_token_contract.jsonl ├── e2e_public_to_private_messaging.jsonl ├── guides_writing_an_account_contract.jsonl ├── multi-transfer.jsonl ├── private.jsonl ├── uniswap_trade_on_l1_from_l2.jsonl └── unknown.jsonl ``` And the aggregated benchmark file has one entry for each metric, and then one entry for each of the `ROLLUP_SIZES` configured: ```json { "l1_rollup_calldata_gas": { "8": 186888, "32": 723612 }, "l1_rollup_calldata_size_in_bytes": { "8": 43044, "32": 169956 }, "l1_rollup_execution_gas": { "8": 730533, "32": 3092494 }, "timestamp": "2023-10-02T17:44:19.237Z" } ``` Note that the `AZTEC_GITHUB_TOKEN` in CCI at the moment doesn't have rights for commenting. I'll review this with Adam once he's back.
Maddiaa0
pushed a commit
that referenced
this issue
Oct 6, 2023
Initial version of benchmarks for end-to-end tests. - Adds support for structured logging. All log methods now accept a string as message (requiring the caller to use `util.format` if needed) and an optional object. The object is passed directly to each entry when logging to JSONL. - Exports stats associated to publishing a rollup to L1 as first use case. Stats are collected using viem and exported with the structured logging format. Adds a first `bench-` e2e test that runs multiple rollup sizes to collect stats. - Collects benchmark data as JSONL from all e2e tests and uploads it to a new S3 bucket `aztec-ci-artifacts` (private for now) using the name of the job and the commit identifier. Benchmark data is any entry that has an `eventName` associated. - Once all e2e tests are finished, aggregates benchmark data from e2e tests named `bench*`, filtering by the metrics requested in #1663, and reuploading it to S3 and adding it to the PR as a comment. The JSONL files extracted from each test look like: ```jsonl {"blockNumber":1,"calldataGas":104732,"calldataSize":22340,"encryptedLogCount":2,"encryptedLogSize":516,"eventName":"rollup-published-to-l1","gasPrice":"1500000008","gasUsed":"422583","level":"info","message":"Published L2 block to L1 rollup contract","namespace":"aztec:sequencer:publisher","timestamp":"2023-09-29T20:14:30.534Z","transactionHash":"0x47724f679d813ec6bdb87002d8ed994abd8f11382eadc81bb41b3de0d7087db7","txCount":4,"unencryptedLogCount":0,"unencryptedLogSize":28} {"blockNumber":2,"calldataGas":93852,"calldataSize":21828,"encryptedLogCount":0,"encryptedLogSize":24,"eventName":"rollup-published-to-l1","gasPrice":"1500000008","gasUsed":"370654","level":"info","message":"Published L2 block to L1 rollup contract","namespace":"aztec:sequencer:publisher","timestamp":"2023-09-29T20:14:35.246Z","transactionHash":"0x1f9ad018f0a7a8f4414a7f789db7baa9b885e524c10a43e2e00d864b4f700721","txCount":4,"unencryptedLogCount":0,"unencryptedLogSize":24} {"blockNumber":3,"calldataGas":94016,"calldataSize":21860,"encryptedLogCount":0,"encryptedLogSize":24,"eventName":"rollup-published-to-l1","gasPrice":"1500000008","gasUsed":"373541","level":"info","message":"Published L2 block to L1 rollup contract","namespace":"aztec:sequencer:publisher","timestamp":"2023-09-29T20:14:41.191Z","transactionHash":"0xf919ea0083420d3bf45263c28797b024f6447ba046ea3d2fba7c116abbac54a7","txCount":4,"unencryptedLogCount":0,"unencryptedLogSize":28} {"blockNumber":4,"calldataGas":93680,"calldataSize":21860,"encryptedLogCount":0,"encryptedLogSize":24,"eventName":"rollup-published-to-l1","gasPrice":"1500000008","gasUsed":"373205","level":"info","message":"Published L2 block to L1 rollup contract","namespace":"aztec:sequencer:publisher","timestamp":"2023-09-29T20:14:46.089Z","transactionHash":"0x34f386ead9e72cbef2ed8b79cc1cffc4540ec9317811d7381d0311e980c90493","txCount":4,"unencryptedLogCount":0,"unencryptedLogSize":28} {"blockNumber":5,"calldataGas":186888,"calldataSize":43044,"encryptedLogCount":0,"encryptedLogSize":68,"eventName":"rollup-published-to-l1","gasPrice":"1500000008","gasUsed":"730533","level":"info","message":"Published L2 block to L1 rollup contract","namespace":"aztec:sequencer:publisher","timestamp":"2023-09-29T20:14:59.600Z","transactionHash":"0xa4ebaeb35dee76bdf8ff3313ede44cf2c8553108d724285eac8c7f1ac47adfdb","txCount":8,"unencryptedLogCount":0,"unencryptedLogSize":100} {"blockNumber":1,"calldataGas":104720,"calldataSize":22340,"encryptedLogCount":2,"encryptedLogSize":516,"eventName":"rollup-published-to-l1","gasPrice":"1500000008","gasUsed":"422571","level":"info","message":"Published L2 block to L1 rollup contract","namespace":"aztec:sequencer:publisher","timestamp":"2023-09-29T20:15:08.321Z","transactionHash":"0x5fd0ea8e8f0bddcd62d178751154f98d10ef234f69ed44f265628eac2b6e3110","txCount":4,"unencryptedLogCount":0,"unencryptedLogSize":28} {"blockNumber":2,"calldataGas":93900,"calldataSize":21828,"encryptedLogCount":0,"encryptedLogSize":24,"eventName":"rollup-published-to-l1","gasPrice":"1500000008","gasUsed":"370702","level":"info","message":"Published L2 block to L1 rollup contract","namespace":"aztec:sequencer:publisher","timestamp":"2023-09-29T20:15:12.976Z","transactionHash":"0x4c30b46ae0ba3f90622843fe363a70300ae8c71e25ad3487e6d43650326d41b1","txCount":4,"unencryptedLogCount":0,"unencryptedLogSize":24} {"blockNumber":3,"calldataGas":94028,"calldataSize":21860,"encryptedLogCount":0,"encryptedLogSize":24,"eventName":"rollup-published-to-l1","gasPrice":"1500000008","gasUsed":"373553","level":"info","message":"Published L2 block to L1 rollup contract","namespace":"aztec:sequencer:publisher","timestamp":"2023-09-29T20:15:16.832Z","transactionHash":"0x7641b6ad965ab2bacc4d6c860de2eb8a08b0e8fd4e01e99f10c5541c20c996a5","txCount":4,"unencryptedLogCount":0,"unencryptedLogSize":28} {"blockNumber":4,"calldataGas":93692,"calldataSize":21860,"encryptedLogCount":0,"encryptedLogSize":24,"eventName":"rollup-published-to-l1","gasPrice":"1500000008","gasUsed":"373217","level":"info","message":"Published L2 block to L1 rollup contract","namespace":"aztec:sequencer:publisher","timestamp":"2023-09-29T20:15:21.969Z","transactionHash":"0x7c13aa9ebd8ec280f99b4dbcc2505cfca08034ec8d22fcf850ffb17fd5493228","txCount":4,"unencryptedLogCount":0,"unencryptedLogSize":28} {"blockNumber":5,"calldataGas":365728,"calldataSize":85348,"encryptedLogCount":0,"encryptedLogSize":132,"eventName":"rollup-published-to-l1","gasPrice":"1500000008","gasUsed":"1444415","level":"info","message":"Published L2 block to L1 rollup contract","namespace":"aztec:sequencer:publisher","timestamp":"2023-09-29T20:15:45.668Z","transactionHash":"0x0ec877bc7a6c30a7879fffe5c9ffd2d72df28f5e46971519def7385790bd1deb","txCount":16,"unencryptedLogCount":0,"unencryptedLogSize":196} ``` Sample S3 structure after running this PR: ``` aztec-ci-artifacts ├── benchmarks-v1 [public] │ └── pulls │ └── 2605.json └── logs-v1 [private] └── pulls └── 2605 ├── archiver.jsonl ├── benchmarks_publish_rollup.jsonl ├── e2e_2_pxes.jsonl ├── e2e_account_contracts.jsonl ├── e2e_card_game.jsonl ├── e2e_cross_chain_messaging.jsonl ├── e2e_deploy_contract.jsonl ├── e2e_escrow_contract.jsonl ├── e2e_multiple_accounts_1_enc_key.jsonl ├── e2e_nested_contract.jsonl ├── e2e_non_contract_account.jsonl ├── e2e_ordering.jsonl ├── e2e_p2p_network.jsonl ├── e2e_pending_commitments_contract.jsonl ├── e2e_private_token_contract.jsonl ├── e2e_public_cross_chain_messaging.jsonl ├── e2e_public_token_contract.jsonl ├── e2e_public_to_private_messaging.jsonl ├── guides_writing_an_account_contract.jsonl ├── multi-transfer.jsonl ├── private.jsonl ├── uniswap_trade_on_l1_from_l2.jsonl └── unknown.jsonl ``` And the aggregated benchmark file has one entry for each metric, and then one entry for each of the `ROLLUP_SIZES` configured: ```json { "l1_rollup_calldata_gas": { "8": 186888, "32": 723612 }, "l1_rollup_calldata_size_in_bytes": { "8": 43044, "32": 169956 }, "l1_rollup_execution_gas": { "8": 730533, "32": 3092494 }, "timestamp": "2023-10-02T17:44:19.237Z" } ``` Note that the `AZTEC_GITHUB_TOKEN` in CCI at the moment doesn't have rights for commenting. I'll review this with Adam once he's back.
|
@iAmMichaelConnor do you think we can flag this issue as Completed? |
|
Yep, I do :) Thanks for the robust way you've implemented this! |
codygunton
pushed a commit
that referenced
this issue
Jan 23, 2024
* Copied index.html to output (#1661) * Generate htnl file (#1662) * Deployment fix (#1663) * Deployment fixes (#1665) * Deployment fix attempt (#1667) * Updated falafel and kebab terraform with new project directory structure (#1670) * Reduce the amount of logging in kebab (#1668) * Reduce the amount of logging in kebab * Logging improvement * Bug fix * Reverted erroneous change * Pw/increase faucet drip (#1669) * Increased the faucet drip size * Force contract redeployment * Deploy terraform from correct directory (#1671) * JB/Website grants updates (#1666) * Add new grant * Improve research images * Mini typo * JB/Website build manifest (#1672) * Add website to build manifest * JB/Brand iteration (#1673) * Small build change * JB/Fix website deployment (#1676) * Fix deployment * JB/Fix website deploy (#1677) * Fix deploy Co-authored-by: Jonathan Bursztyn <jobur93@gmail.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
We should collect some benchmarks about the current protocol.
We should generate this table programmatically. The crypto team have done this nicely; take inspiration from there.
Please add to the table directly, if you can think of more :)
Combine with the cryptography metrics being pulled-together by Cody.
* "Core protocol" = any circuit which is not a Noir smart contract circuit.
Tasks
The text was updated successfully, but these errors were encountered: