Whisk (SSLE) with Curdleproofs - rebased (#3342)

* Introduce consensus code for Whisk

* polish, simplify, clean up (~100 fewer lines)

@asn-d6: As discussed, I fixed a few bugs along the way but likely also introduced some bugs :)

* minor cleanups and fixes

* simplify is_k_commitment_unique

* Update beacon-chain.md

* Update beacon-chain.md

* Initialize `k` in `get_validator_from_deposit()`

* minor cleanups

* Update beacon-chain.md

* Create beacon-chain.md

This PR changes the Whisk tracker format to be of the form `(r * pubkey, r * BLS_GT_GENERATOR)` instead of `(r * k * BLS_G1_GENERATOR, r * BLS_G1_GENERATOR)`. This allows for non-interactive tracker registrations from validator pubkeys, removing ~50 lines the code. It also significantly reduces the amount of state overhead. This PR also removes permutation commitments, though those can be easily readded if deemed necessary.

* A couple of fixes to the no-registration simplification

@asn-d6: Readded a consistency check for `IsValidWhiskOpeningProof` (involving `pubkey` instead of `k_commitment`).

* remove unused helpers

* use Mary's suggested tracker

* Update beacon-chain.md

* Revert G_t element optimization

This needs its own ethresearch post, and some additional analysis to see if we can do the shuffle ZKP in the allowed
timeframe.

This reverts commit 8517aca.

* Implement new shuffling strategy

Ditch the Feistel logic and instead have each shuffler pick the row they shuffle using their RANDAO reveal.

* Curdleproofs edits

* working whisk eth2spec

* working whisk dummy test

* add more boilerplate set up code

* rebase constants

* Implement even newer and simplified shuffling strategy

This commit further simplifies 0faef30 by removing the entire squareshuffle.

The latest version of https://eprint.iacr.org/2022/560 proposes that each shuffler picks random indices from the entire
candidate set instead of organizing validators into a square.

* Move to _features

* remove dummy test

* Run doctoc

* Change Whisk's previous fork to Capella instead of Bellatrix. Make linter happier.

* Fix lint

* Fix pylint

* Fix mypy issues

* Clean-up get_beacon_proposer_index

* Fix doc headers

* Fix capella link

* Update apply_deposit

* Rename process_shuffled_trackers

---------

Co-authored-by: George Kadianakis <desnacked@riseup.net>
Co-authored-by: Justin <drakefjustin@gmail.com>
Co-authored-by: Nalin Bhardwaj <nalinbhardwaj@nibnalin.me>
Co-authored-by: Hsiao-Wei Wang <hsiaowei.eth@gmail.com>

.gitignore

@@ -21,6 +21,7 @@ tests/core/pyspec/eth2spec/bellatrix/
 tests/core/pyspec/eth2spec/capella/
 tests/core/pyspec/eth2spec/deneb/
 tests/core/pyspec/eth2spec/eip6110/
+tests/core/pyspec/eth2spec/whisk/
 
 # coverage reports
 .htmlcov

Makefile

@@ -34,7 +34,7 @@ MARKDOWN_FILES = $(wildcard $(SPEC_DIR)/*/*.md) \
                  $(wildcard $(SPEC_DIR)/_features/*/*/*.md) \
                  $(wildcard $(SSZ_DIR)/*.md)
 
-ALL_EXECUTABLE_SPECS = phase0 altair bellatrix capella deneb eip6110
+ALL_EXECUTABLE_SPECS = phase0 altair bellatrix capella deneb eip6110 whisk
 # The parameters for commands. Use `foreach` to avoid listing specs again.
 COVERAGE_SCOPE := $(foreach S,$(ALL_EXECUTABLE_SPECS), --cov=eth2spec.$S.$(TEST_PRESET_TYPE))
 PYLINT_SCOPE := $(foreach S,$(ALL_EXECUTABLE_SPECS), ./eth2spec/$S)

setup.py

@@ -48,6 +48,7 @@ def installPackage(package: str):
 CAPELLA = 'capella'
 DENEB = 'deneb'
 EIP6110 = 'eip6110'
+WHISK = 'whisk'
 
 
 # The helper functions that are used when defining constants
@@ -733,9 +734,31 @@ def imports(cls, preset_name: str):
 '''
 
 
+#
+# WhiskSpecBuilder
+#
+class WhiskSpecBuilder(CapellaSpecBuilder):
+    fork: str = WHISK
+
+    @classmethod
+    def imports(cls, preset_name: str):
+        return super().imports(preset_name) + f'''
+from eth2spec.capella import {preset_name} as capella
+'''
+
+    @classmethod
+    def hardcoded_custom_type_dep_constants(cls, spec_object) -> str:
+        # Necessary for custom types `WhiskShuffleProof` and `WhiskTrackerProof`
+        constants = {
+            'WHISK_MAX_SHUFFLE_PROOF_SIZE': spec_object.constant_vars['WHISK_MAX_SHUFFLE_PROOF_SIZE'].value,
+            'WHISK_MAX_OPENING_PROOF_SIZE': spec_object.constant_vars['WHISK_MAX_OPENING_PROOF_SIZE'].value,
+        }
+        return {**super().hardcoded_custom_type_dep_constants(spec_object), **constants}
+
+
 spec_builders = {
     builder.fork: builder
-    for builder in (Phase0SpecBuilder, AltairSpecBuilder, BellatrixSpecBuilder, CapellaSpecBuilder, DenebSpecBuilder, EIP6110SpecBuilder)
+    for builder in (Phase0SpecBuilder, AltairSpecBuilder, BellatrixSpecBuilder, CapellaSpecBuilder, DenebSpecBuilder, EIP6110SpecBuilder, WhiskSpecBuilder)
 }
 
 
@@ -1045,15 +1068,15 @@ def finalize_options(self):
         if len(self.md_doc_paths) == 0:
             print("no paths were specified, using default markdown file paths for pyspec"
                   " build (spec fork: %s)" % self.spec_fork)
-            if self.spec_fork in (PHASE0, ALTAIR, BELLATRIX, CAPELLA, DENEB, EIP6110):
+            if self.spec_fork in (PHASE0, ALTAIR, BELLATRIX, CAPELLA, DENEB, EIP6110, WHISK):
                 self.md_doc_paths = """
                     specs/phase0/beacon-chain.md
                     specs/phase0/fork-choice.md
                     specs/phase0/validator.md
                     specs/phase0/weak-subjectivity.md
                     specs/phase0/p2p-interface.md
                 """
-            if self.spec_fork in (ALTAIR, BELLATRIX, CAPELLA, DENEB, EIP6110):
+            if self.spec_fork in (ALTAIR, BELLATRIX, CAPELLA, DENEB, EIP6110, WHISK):
                 self.md_doc_paths += """
                     specs/altair/light-client/full-node.md
                     specs/altair/light-client/light-client.md
@@ -1065,7 +1088,7 @@ def finalize_options(self):
                     specs/altair/validator.md
                     specs/altair/p2p-interface.md
                 """
-            if self.spec_fork in (BELLATRIX, CAPELLA, DENEB, EIP6110):
+            if self.spec_fork in (BELLATRIX, CAPELLA, DENEB, EIP6110, WHISK):
                 self.md_doc_paths += """
                     specs/bellatrix/beacon-chain.md
                     specs/bellatrix/fork.md
@@ -1074,7 +1097,7 @@ def finalize_options(self):
                     specs/bellatrix/p2p-interface.md
                     sync/optimistic.md
                 """
-            if self.spec_fork in (CAPELLA, DENEB, EIP6110):
+            if self.spec_fork in (CAPELLA, DENEB, EIP6110, WHISK):
                 self.md_doc_paths += """
                     specs/capella/light-client/fork.md
                     specs/capella/light-client/full-node.md
@@ -1104,6 +1127,11 @@ def finalize_options(self):
                     specs/_features/eip6110/beacon-chain.md
                     specs/_features/eip6110/fork.md
                 """
+            if self.spec_fork == WHISK:
+                self.md_doc_paths += """
+                    specs/_features/whisk/beacon-chain.md
+                    specs/_features/whisk/fork.md
+                """
             if len(self.md_doc_paths) == 0:
                 raise Exception('no markdown files specified, and spec fork "%s" is unknown', self.spec_fork)
 
@@ -1259,5 +1287,6 @@ def run(self):
         "lru-dict==1.1.8",
         MARKO_VERSION,
         "py_arkworks_bls12381==0.3.4",
+        "curdleproofs @ git+https://github.com/nalinbhardwaj/curdleproofs.pie@master#egg=curdleproofs&subdirectory=curdleproofs",
     ]
 )