feat: Goblin proof construction #3332

codygunton · 2023-11-16T23:21:08Z

The last PR that splits out and completes work from #3022

We add composer, prover and verifier classes for the Goblin Translator VM. We also further flesh out the full Goblin composer tests. Modulo some missing challenge generation, we have our first Goblin proofs (i.e., something worth measuring)!

Spawned issues
AztecProtocol/barretenberg#783
AztecProtocol/barretenberg#784
AztecProtocol/barretenberg#785
AztecProtocol/barretenberg#786
AztecProtocol/barretenberg#787

Checklist:

Remove the checklist to signal you've completed it. Enable auto-merge if the PR is ready to merge.

If the pull request requires a cryptography review (e.g. cryptographic algorithm implementations) I have added the 'crypto' tag.
I have reviewed my diff in github, line by line and removed unexpected formatting changes, testing logs, or commented-out code.
Every change is related to the PR description.
I have linked this pull request to relevant issues (if any exist).

AztecBot · 2023-11-16T23:59:10Z

Benchmark results

Metrics with a significant change:

note_trial_decrypting_time_in_ms (8): 100 (+433%)
note_trial_decrypting_time_in_ms (32): 88.8 (+555%)

Detailed results

All benchmarks are run on txs on the Benchmarking contract on the repository. Each tx consists of a batch call to create_note and increment_balance, which guarantees that each tx has a private call, a nested private call, a public call, and a nested public call, as well as an emitted private note, an unencrypted log, and public storage read and write.

This benchmark source data is available in JSON format on S3 here.

Values are compared against data from master at commit 5cfbb68b and shown if the difference exceeds 1%.

L2 block published to L1

Each column represents the number of txs on an L2 block published to L1.

Metric	8 txs	32 txs	128 txs
l1_rollup_calldata_size_in_bytes	45,444	179,588	716,132
l1_rollup_calldata_gas	222,900	868,160	3,449,516
l1_rollup_execution_gas	841,987	3,595,268	22,204,885
l2_block_processing_time_in_ms	2,005 (-1%)	7,710 (-1%)	29,908 (-1%)
note_successful_decrypting_time_in_ms	297 (-6%)	878 (-3%)	3,243 (-1%)
note_trial_decrypting_time_in_ms	⚠️ 100 (+433%)	⚠️ 88.8 (+555%)	142 (+1%)
l2_block_building_time_in_ms	20,747 (-2%)	81,839 (-2%)	326,577 (-1%)
l2_block_rollup_simulation_time_in_ms	12,088 (-2%)	47,435 (-2%)	188,619 (-1%)
l2_block_public_tx_process_time_in_ms	8,617 (-2%)	34,263 (-2%)	137,442 (-1%)

L2 chain processing

Each column represents the number of blocks on the L2 chain where each block has 16 txs.

Metric	5 blocks	10 blocks
node_history_sync_time_in_ms	22,044 (-1%)	42,811 (-1%)
note_history_successful_decrypting_time_in_ms	2,038 (-4%)	4,042 (-2%)
note_history_trial_decrypting_time_in_ms	125 (-2%)	158 (+1%)
node_database_size_in_bytes	1,629,370	1,099,814
pxe_database_size_in_bytes	29,748	59,307

Circuits stats

Stats on running time and I/O sizes collected for every circuit run across all benchmarks.

Circuit	circuit_simulation_time_in_ms	circuit_input_size_in_bytes	circuit_output_size_in_bytes
private-kernel-init	774	61,697	18,905
private-kernel-ordering	128	24,297	8,153
base-rollup	1,753 (-1%)	656,428	873
root-rollup	169 (-1%)	4,072	1,097
private-kernel-inner	799	81,568	18,905
public-kernel-private-input	574	41,519	18,905
public-kernel-non-first-iteration	411	41,561	18,905
merge-rollup	15.4 (-1%)	2,592	873

Miscellaneous

Transaction sizes based on how many contracts are deployed in the tx.

Metric	0 deployed contracts	1 deployed contracts
tx_size_in_bytes	8,787	27,547

This reverts commit 835e77e.

codygunton · 2023-11-21T18:43:47Z

barretenberg/cpp/src/barretenberg/goblin/full_goblin_composer.test.cpp

-    }
-}
-
+// TODO(https://github.com/AztecProtocol/barretenberg/issues/787) Expand these tests.


Left a note to add more tests once we have a properly encapsulated Goblin. Will make it easier to write the tests, and this test wasn't doing anything Goblin-specific.

codygunton · 2023-11-21T18:44:26Z

barretenberg/cpp/src/barretenberg/proof_system/circuit_builder/eccvm/eccvm_circuit_builder.hpp


-    ECCVMCircuitBuilder(std::vector<VMOperation> vm_operations)
-        : vm_operations(vm_operations){};
+    ECCVMCircuitBuilder(std::shared_ptr<ECCOpQueue>& op_queue)


Without this, the shared pointer initializes to null.

How did it work before? Was this API actually never used?

A pointer wasn't passed, there was just a vector of operations, but now we need to pass in this op_queue to share data between the ECCVM and the Translator.

codygunton · 2023-11-21T19:18:31Z

barretenberg/cpp/src/barretenberg/translator_vm/goblin_translator_verifier.hpp

+    BF evaluation_input_x = 0;
+    BF batching_challenge_v = 0;
+    std::shared_ptr<VerificationKey> key;
+    std::map<std::string, Commitment> commitments;


Idk why we're using maps here, but I see they're also in use in the other flavors. Will leave an issue

Rumata888 · 2023-11-22T17:09:04Z

barretenberg/cpp/src/barretenberg/eccvm/eccvm_verifier.cpp

@@ -270,7 +270,7 @@ template <typename Flavor> bool ECCVMVerifier_<Flavor>::verify_proof(const plonk
        univariate_opening_verified = PCS::verify(pcs_verification_key, batched_univariate_claim, transcript);
    }

-    return multivariate_opening_verified && univariate_opening_verified;
+    return sumcheck_verified.value() && multivariate_opening_verified && univariate_opening_verified;


Haven't we already verified the sumcheck by this time?

Yeah but it felt safer to add this check (why not? also what if somehow the option doesn't have a value and the early return is missed? not sure what happens then).

Rumata888 · 2023-11-22T17:12:30Z

barretenberg/cpp/src/barretenberg/goblin/full_goblin_composer.test.cpp


-namespace {
-auto& engine = numeric::random::get_debug_engine();


Do we not use debug engine for reproducible randomness anymore?

I think randomness by default in tests is better. More chances to discover weird edge cases! You said you agree.

Rumata888 · 2023-11-22T17:15:40Z

barretenberg/cpp/src/barretenberg/goblin/translation_evaluations.hpp

+#include "barretenberg/ecc/curves/bn254/fq.hpp"
+
+namespace barretenberg {
+struct TranslationEvaluations {


Maybe this should be in the EccOpQueue? It seems natural to have it there, since it has no goblin translator specific members.

But then OpQueue is not a good name...

Rumata888 · 2023-11-22T17:36:44Z

barretenberg/cpp/src/barretenberg/translator_vm/goblin_translator_prover.cpp

+}
+
+/**
+ * @brief Compute permutation and lookup grand product polynomials and commitments


No lookup grand product

Rumata888 · 2023-11-22T17:41:12Z

barretenberg/cpp/src/barretenberg/translator_vm/goblin_translator_composer.cpp

+ * @tparam Flavor provides the circuit constructor type and the number of wires.
+ * @param circuit_builder
+ * @param dyadic_circuit_size Power of 2 circuit size
+ * @todo TODO(https://github.com/AztecProtocol/barretenberg/issues/783) Optimize memory operatations.


operatations

🤖 I have created a release *beep* *boop* --- <details><summary>aztec-packages: 0.16.0</summary> ## [0.16.0](aztec-packages-v0.15.1...aztec-packages-v0.16.0) (2023-11-27) ### ⚠ BREAKING CHANGES * deprecate circuits/cpp ([#3421](#3421)) * call stack validation optimisation. ([#3387](#3387)) ### Features * Base rollup in noir ([#3257](#3257)) ([4a1e9c3](4a1e9c3)) * Call stack validation optimisation. ([#3387](#3387)) ([d06d5db](d06d5db)) * Goblin proof construction ([#3332](#3332)) ([6a7ebb6](6a7ebb6)) * More logs relevant for debugging failures of 2 pixies test ([#3370](#3370)) ([683a0f3](683a0f3)) * Noir subrepo. ([#3369](#3369)) ([d94d88b](d94d88b)) * Noir_wasm compilation of noir programs ([#3272](#3272)) ([f9981d5](f9981d5)) * Rollback public state changes on failure ([#3393](#3393)) ([0e276fb](0e276fb)) ### Bug Fixes * **docs:** Doc explaining noir debug_log ([#3322](#3322)) ([eed023d](eed023d)) * Naming inconsistency in private kernel ([#3384](#3384)) ([4743486](4743486)) * Race condition in `PXE.getTxReceipt(...)` ([#3411](#3411)) ([9557a66](9557a66)) ### Miscellaneous * Deprecate circuits/cpp ([#3421](#3421)) ([4973cfb](4973cfb)) * Deterministically deduplicate `cached_partial_non_native_field_multiplication` across wasm32 and native compilations ([#3425](#3425)) ([5524933](5524933)) * **docs:** Common patterns and anti patterns in aztec.nr ([#3413](#3413)) ([65bd855](65bd855)) * Fix and reenable e2e quick start ([#3403](#3403)) ([112740e](112740e)), closes [#3356](#3356) * Fix intermittent failures for block-building e2e test ([#3404](#3404)) ([e76e2d4](e76e2d4)), closes [#3358](#3358) * Formatted `noir-contracts` and `aztec-nr` ([a73c4aa](a73c4aa)) * Initial clone of noir to subrepo ([#3409](#3409)) ([8f1cb83](8f1cb83)) * **noir-contracts:** Remove redundant return value of 1 ([#3415](#3415)) ([2001d47](2001d47)), closes [#2615](#2615) * Plumbs noir subrepo into yarn-project. ([#3420](#3420)) ([63173c4](63173c4)) * Remove pxe / node /p2p-bootstrap docker images ([#3396](#3396)) ([c236143](c236143)) * Skip artifacts for prettier ([#3399](#3399)) ([98d9e04](98d9e04)) * Update path to acir artifacts ([#3426](#3426)) ([f56f88d](f56f88d)) </details> <details><summary>barretenberg.js: 0.16.0</summary> ## [0.16.0](barretenberg.js-v0.15.1...barretenberg.js-v0.16.0) (2023-11-27) ### Miscellaneous * Plumbs noir subrepo into yarn-project. ([#3420](#3420)) ([63173c4](63173c4)) </details> <details><summary>barretenberg: 0.16.0</summary> ## [0.16.0](barretenberg-v0.15.1...barretenberg-v0.16.0) (2023-11-27) ### Features * Goblin proof construction ([#3332](#3332)) ([6a7ebb6](6a7ebb6)) * Noir subrepo. ([#3369](#3369)) ([d94d88b](d94d88b)) ### Miscellaneous * Deterministically deduplicate `cached_partial_non_native_field_multiplication` across wasm32 and native compilations ([#3425](#3425)) ([5524933](5524933)) * Plumbs noir subrepo into yarn-project. ([#3420](#3420)) ([63173c4](63173c4)) * Update path to acir artifacts ([#3426](#3426)) ([f56f88d](f56f88d)) </details> --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please).

🤖 I have created a release *beep* *boop* --- <details><summary>aztec-packages: 0.16.0</summary> ## [0.16.0](AztecProtocol/aztec-packages@aztec-packages-v0.15.1...aztec-packages-v0.16.0) (2023-11-27) ### ⚠ BREAKING CHANGES * deprecate circuits/cpp ([#3421](AztecProtocol/aztec-packages#3421)) * call stack validation optimisation. ([#3387](AztecProtocol/aztec-packages#3387)) ### Features * Base rollup in noir ([#3257](AztecProtocol/aztec-packages#3257)) ([4a1e9c3](AztecProtocol/aztec-packages@4a1e9c3)) * Call stack validation optimisation. ([#3387](AztecProtocol/aztec-packages#3387)) ([d06d5db](AztecProtocol/aztec-packages@d06d5db)) * Goblin proof construction ([#3332](AztecProtocol/aztec-packages#3332)) ([6a7ebb6](AztecProtocol/aztec-packages@6a7ebb6)) * More logs relevant for debugging failures of 2 pixies test ([#3370](AztecProtocol/aztec-packages#3370)) ([683a0f3](AztecProtocol/aztec-packages@683a0f3)) * Noir subrepo. ([#3369](AztecProtocol/aztec-packages#3369)) ([d94d88b](AztecProtocol/aztec-packages@d94d88b)) * Noir_wasm compilation of noir programs ([#3272](AztecProtocol/aztec-packages#3272)) ([f9981d5](AztecProtocol/aztec-packages@f9981d5)) * Rollback public state changes on failure ([#3393](AztecProtocol/aztec-packages#3393)) ([0e276fb](AztecProtocol/aztec-packages@0e276fb)) ### Bug Fixes * **docs:** Doc explaining noir debug_log ([#3322](AztecProtocol/aztec-packages#3322)) ([eed023d](AztecProtocol/aztec-packages@eed023d)) * Naming inconsistency in private kernel ([#3384](AztecProtocol/aztec-packages#3384)) ([4743486](AztecProtocol/aztec-packages@4743486)) * Race condition in `PXE.getTxReceipt(...)` ([#3411](AztecProtocol/aztec-packages#3411)) ([9557a66](AztecProtocol/aztec-packages@9557a66)) ### Miscellaneous * Deprecate circuits/cpp ([#3421](AztecProtocol/aztec-packages#3421)) ([4973cfb](AztecProtocol/aztec-packages@4973cfb)) * Deterministically deduplicate `cached_partial_non_native_field_multiplication` across wasm32 and native compilations ([#3425](AztecProtocol/aztec-packages#3425)) ([5524933](AztecProtocol/aztec-packages@5524933)) * **docs:** Common patterns and anti patterns in aztec.nr ([#3413](AztecProtocol/aztec-packages#3413)) ([65bd855](AztecProtocol/aztec-packages@65bd855)) * Fix and reenable e2e quick start ([#3403](AztecProtocol/aztec-packages#3403)) ([112740e](AztecProtocol/aztec-packages@112740e)), closes [#3356](AztecProtocol/aztec-packages#3356) * Fix intermittent failures for block-building e2e test ([#3404](AztecProtocol/aztec-packages#3404)) ([e76e2d4](AztecProtocol/aztec-packages@e76e2d4)), closes [#3358](AztecProtocol/aztec-packages#3358) * Formatted `noir-contracts` and `aztec-nr` ([a73c4aa](AztecProtocol/aztec-packages@a73c4aa)) * Initial clone of noir to subrepo ([#3409](AztecProtocol/aztec-packages#3409)) ([8f1cb83](AztecProtocol/aztec-packages@8f1cb83)) * **noir-contracts:** Remove redundant return value of 1 ([#3415](AztecProtocol/aztec-packages#3415)) ([2001d47](AztecProtocol/aztec-packages@2001d47)), closes [#2615](AztecProtocol/aztec-packages#2615) * Plumbs noir subrepo into yarn-project. ([#3420](AztecProtocol/aztec-packages#3420)) ([63173c4](AztecProtocol/aztec-packages@63173c4)) * Remove pxe / node /p2p-bootstrap docker images ([#3396](AztecProtocol/aztec-packages#3396)) ([c236143](AztecProtocol/aztec-packages@c236143)) * Skip artifacts for prettier ([#3399](AztecProtocol/aztec-packages#3399)) ([98d9e04](AztecProtocol/aztec-packages@98d9e04)) * Update path to acir artifacts ([#3426](AztecProtocol/aztec-packages#3426)) ([f56f88d](AztecProtocol/aztec-packages@f56f88d)) </details> <details><summary>barretenberg.js: 0.16.0</summary> ## [0.16.0](AztecProtocol/aztec-packages@barretenberg.js-v0.15.1...barretenberg.js-v0.16.0) (2023-11-27) ### Miscellaneous * Plumbs noir subrepo into yarn-project. ([#3420](AztecProtocol/aztec-packages#3420)) ([63173c4](AztecProtocol/aztec-packages@63173c4)) </details> <details><summary>barretenberg: 0.16.0</summary> ## [0.16.0](AztecProtocol/aztec-packages@barretenberg-v0.15.1...barretenberg-v0.16.0) (2023-11-27) ### Features * Goblin proof construction ([#3332](AztecProtocol/aztec-packages#3332)) ([6a7ebb6](AztecProtocol/aztec-packages@6a7ebb6)) * Noir subrepo. ([#3369](AztecProtocol/aztec-packages#3369)) ([d94d88b](AztecProtocol/aztec-packages@d94d88b)) ### Miscellaneous * Deterministically deduplicate `cached_partial_non_native_field_multiplication` across wasm32 and native compilations ([#3425](AztecProtocol/aztec-packages#3425)) ([5524933](AztecProtocol/aztec-packages@5524933)) * Plumbs noir subrepo into yarn-project. ([#3420](AztecProtocol/aztec-packages#3420)) ([63173c4](AztecProtocol/aztec-packages@63173c4)) * Update path to acir artifacts ([#3426](AztecProtocol/aztec-packages#3426)) ([f56f88d](AztecProtocol/aztec-packages@f56f88d)) </details> --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please).

codygunton self-assigned this Nov 16, 2023

codygunton changed the base branch from master to lde/eccvm_univariate_openings November 17, 2023 17:59

Base automatically changed from lde/eccvm_univariate_openings to master November 17, 2023 19:24

codygunton assigned Rumata888 Nov 17, 2023

codygunton force-pushed the cg/full-goblin branch from 07aa6de to 2dff430 Compare November 17, 2023 21:44

codygunton changed the base branch from master to cg-lde/zm_updates November 17, 2023 21:45

Base automatically changed from cg-lde/zm_updates to master November 17, 2023 21:46

codygunton added 3 commits November 17, 2023 21:54

Squash

2fc62de

Everything builds

34c8dd1

Fix test

6669d2b

codygunton force-pushed the cg/full-goblin branch from a0dc285 to 6669d2b Compare November 17, 2023 22:34

codygunton added 18 commits November 17, 2023 23:46

Re-link univariate evaluations

c8c3794

Rename struct

0c0ea53

Revert shplonk

7c3d80f

Cheat... ~> TranslationEvaluations

9dc88fb

Clean up

1490b7d

Scrap test for now

d9ebe64

cleanup

fe01fd3

Untemplate composer

ee565cf

Untemplate prover and verifier

981b9f4

Reintroduce warnings

b534033

Some composer cleanup

c55c858

Remove one unneeded layer of copying.

835e77e

Use lambda

03fc165

1/3 reduction in compilation time.

98272a1

Revert "Remove one unneeded layer of copying."

87a4438

This reverts commit 835e77e.

temp add -ftime-trace

6e2edd3

Make bb-tests run translator_vm_tests

32439fb

Leave note on reverted block

2e5b735

codygunton added 6 commits November 20, 2023 20:44

Leave notes

64184aa

Remove compilation profiling

66f979b

some cleanup

8f6b899

Cleanup

2f80643

Initialize op_queue

0d11fae

Delete notes

183fb56

codygunton commented Nov 21, 2023

View reviewed changes

codygunton requested a review from Rumata888 November 21, 2023 19:19

codygunton added 2 commits November 21, 2023 14:20

Merge branch 'master' into cg/full-goblin

97373ff

Update sumcheck after rebase.

4e9002f

codygunton mentioned this pull request Nov 21, 2023

feat: Goblin Translator (final part) #3022

Closed

4 tasks

codygunton marked this pull request as ready for review November 21, 2023 20:56

Rumata888 reviewed Nov 22, 2023

View reviewed changes

Rumata888 approved these changes Nov 22, 2023

View reviewed changes

codygunton added 4 commits November 22, 2023 18:33

Fix typos

058430e

Reintroduce engine

075b0e4

Merge remote-tracking branch 'origin/master' into cg/full-goblin

9c81e24

Merge branch 'master' into cg/full-goblin

678a8b8

codygunton enabled auto-merge (squash) November 22, 2023 19:32

codygunton disabled auto-merge November 22, 2023 19:32

codygunton enabled auto-merge (squash) November 22, 2023 19:32

codygunton merged commit 6a7ebb6 into master Nov 22, 2023
76 checks passed

codygunton deleted the cg/full-goblin branch November 22, 2023 20:04

AztecBot mentioned this pull request Nov 22, 2023

chore(master): Release 0.16.0 #3385

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: Goblin proof construction #3332

feat: Goblin proof construction #3332

codygunton commented Nov 16, 2023 •

edited

Loading

AztecBot commented Nov 16, 2023 •

edited

Loading

L2 block published to L1

L2 chain processing

Circuits stats

Miscellaneous

codygunton Nov 21, 2023

codygunton Nov 21, 2023

Rumata888 Nov 22, 2023

codygunton Nov 22, 2023

codygunton Nov 21, 2023

Rumata888 Nov 22, 2023

codygunton Nov 22, 2023

Rumata888 Nov 22, 2023

codygunton Nov 22, 2023 •

edited

Loading

Rumata888 Nov 22, 2023

codygunton Nov 22, 2023

Rumata888 Nov 22, 2023

Rumata888 Nov 22, 2023


		namespace {
		auto& engine = numeric::random::get_debug_engine();

feat: Goblin proof construction #3332

feat: Goblin proof construction #3332

Conversation

codygunton commented Nov 16, 2023 • edited Loading

Checklist:

AztecBot commented Nov 16, 2023 • edited Loading

Benchmark results

L2 block published to L1

L2 chain processing

Circuits stats

Miscellaneous

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codygunton Nov 22, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codygunton commented Nov 16, 2023 •

edited

Loading

AztecBot commented Nov 16, 2023 •

edited

Loading

codygunton Nov 22, 2023 •

edited

Loading