EIP-4844 progress

[karalabe]

This is a tracking issue for EIP-4844. Some of the issues might get broken down into very low level piecemeal ones that might help review and/or work on a different larger ones. Ideally we'll end up with a batch of PRs people can take a look at to wrap their head around 4844 and it's progress / coding history.

• Integrate an SSZ library / generator for the blob transactions
  • Work is ongoing via https://github.com/OffchainLabs/methodical-ssz
• Integrate a KZG library for blob verifications
  • Pull in the C and Go dependencies for different systems, wrap together
    crypto/kzg4844: pull in the C and Go libs for KZG cryptography #27155
  • Benchmark batched vs concurrent verifications and extend the API if needed
  • Pull in the final trusted setup parameters
• Define the blob transaction
  • Define the new transaction type and integrate the interfaces and signers
    core/types, params: add blob transaction type, RLP encoded for now #27049
  • Convert the blob transaction encodings from RLP to SSZ
• Update the header with the 4844 fields and implement validations
  • Add excessDataGas field to header, disable before Cancun
    consensus, core/typer: add 4844 excessDataGas to header, tie it to Cancun #27046
  • Implement excessDataGas to blob fee conversions
    consensus/misc, params: add EIP-4844 blobfee conversions #27041
  • Implement excessDataGas validations and calculations across blocks
    consensus, core, eth/downloader: add 4844 excessDataGas validations #27340
    consensus, core, eth/downloader, params: 4844 chain validation #27382
  • Validate blob hashes in bundled transactions (block processing and snap sync)
    core, eth/downloader, params: validate blob tx bodies #27392
    core, eth/downloader: validate blobtx.To at serialization time #27393
• Update the receipts with the blob gas usage and pricing fields
  core/types: add data gas fields in Receipt #27743
• Implement BLOBHASH opcode and point evaluation precompile
  core: 4844 opcode and precompile #27356
• Implement blob transaction pool
  • Abstract out transaction validation to support reuse across multiple tx pools
    core/txpool: make transaction validation reusable across packages (pools) #27429
  • Make the old transaction pool a subpool / sibling of the blobpool
    all: move main transaction pool into a subpool #27463
  • Define the integration points between the old txpool and the new blob pool
    core/txpool: make transaction validation reusable across packages (pools) #27429
    all: move main transaction pool into a subpool #27463
  • Implement the standalone blob transaction pool
    core/txpool/blobpool: 4844 blob transaction pool #26940
• Implement blob transaction propagation
  • Update the eth protocol to support announce-only for blob txs
    eth: implement eth/68 to support only announcing certain tx types #25980
  • Implement a blob transaction fetcher to throttle the data rate of blob propagations
• Integrate the blob transactions with the miner
• Implement the engine API extensions for the blob transactions
  beacon/engine, eth/catalyst: EIP-4844 updates for the engine API #27736
• Add support for delaying the Cancun fork
  cmd, eth, les: replace Shanghai override flag with Cancun #27171

Note, an initial proof of concept was prepared by Coinbase and Optimism at #26283.

[jtraglia]

For KZG implementations, there are two options for Go clients:

• Go-KZG-4844
  • Pure Go (with gnark-crypto)
  • Supports parallelization
  • Go 1.18 required
  • CGO not required
• C-KZG-4844 (Go bindings)
  • Core implementation in C (with blst)
  • All single-threaded
  • Go 1.19 required
  • CGO required

Right now, our recommendation is for Go clients to use Go-KZG-4844. While C-KZG-4844 is expected to be slightly faster when run on single-threaded systems, Go-KZG-4844 also provides parallelized functions that can be faster (though less efficient) on systems with multiple threads. Most clients will use C-KZG-4844 bindings, but that's because there isn't a great alternative for them (right now). From a diversity perspective, using a different implementation would be beneficial. Both implementations will be audited and have been differentially fuzzed against each other.

Also, some information about the audit. There will be a one month audit (starting on April 17th 2023) for both of these implementations (and their bindings) by a reputable security firm. For Go-KZG-4844, code from gnark-crypto that it needs but was not audited has been copied into its project so it gets audited.

Benchmarks for Go-KZG-4844

With parallelization enabled on a system with 20 cores:

Benchmark/BlobToKZGCommitment-20         	     307	   3901151 ns/op
Benchmark/ComputeKZGProof-20             	     254	   4697532 ns/op
Benchmark/ComputeBlobKZGProof-20         	     246	   4861995 ns/op
Benchmark/VerifyKZGProof-20              	     858	   1390373 ns/op
Benchmark/VerifyBlobKZGProof-20          	     585	   2032785 ns/op
Benchmark/VerifyBlobKZGProofBatch(count=1)-20         	     585	   2035749 ns/op
Benchmark/VerifyBlobKZGProofBatch(count=2)-20         	     370	   3227315 ns/op
Benchmark/VerifyBlobKZGProofBatch(count=4)-20         	     243	   4920444 ns/op
Benchmark/VerifyBlobKZGProofBatch(count=8)-20         	     145	   8207486 ns/op
Benchmark/VerifyBlobKZGProofBatch(count=16)-20        	      82	  14729597 ns/op
Benchmark/VerifyBlobKZGProofBatch(count=32)-20        	      43	  27678134 ns/op
Benchmark/VerifyBlobKZGProofBatch(count=64)-20        	      21	  53038206 ns/op
Benchmark/VerifyBlobKZGProofBatchPar(count=1)-20      	     583	   2039326 ns/op
Benchmark/VerifyBlobKZGProofBatchPar(count=2)-20      	     538	   2224141 ns/op
Benchmark/VerifyBlobKZGProofBatchPar(count=4)-20      	     501	   2383239 ns/op
Benchmark/VerifyBlobKZGProofBatchPar(count=8)-20      	     451	   2656815 ns/op
Benchmark/VerifyBlobKZGProofBatchPar(count=16)-20     	     366	   3253719 ns/op
Benchmark/VerifyBlobKZGProofBatchPar(count=32)-20     	     187	   6241685 ns/op
Benchmark/VerifyBlobKZGProofBatchPar(count=64)-20     	     100	  11882722 ns/op

For a better comparision of efficiency, with GOMAXPROCS=1:

Benchmark/BlobToKZGCommitment         	      26	  43713460 ns/op
Benchmark/ComputeKZGProof             	      25	  45990877 ns/op
Benchmark/ComputeBlobKZGProof         	      25	  46085868 ns/op
Benchmark/VerifyKZGProof              	     859	   1390975 ns/op
Benchmark/VerifyBlobKZGProof          	     585	   2035813 ns/op
Benchmark/VerifyBlobKZGProofBatch(count=1)         	     588	   2035962 ns/op
Benchmark/VerifyBlobKZGProofBatch(count=2)         	     355	   3381252 ns/op
Benchmark/VerifyBlobKZGProofBatch(count=4)         	     225	   5297926 ns/op
Benchmark/VerifyBlobKZGProofBatch(count=8)         	     135	   8825641 ns/op
Benchmark/VerifyBlobKZGProofBatch(count=16)        	      74	  15600914 ns/op
Benchmark/VerifyBlobKZGProofBatch(count=32)        	      39	  29332908 ns/op
Benchmark/VerifyBlobKZGProofBatch(count=64)        	      20	  55860546 ns/op
Benchmark/VerifyBlobKZGProofBatchPar(count=1)      	     584	   2047389 ns/op
Benchmark/VerifyBlobKZGProofBatchPar(count=2)      	     286	   4179565 ns/op
Benchmark/VerifyBlobKZGProofBatchPar(count=4)      	     141	   8363224 ns/op
Benchmark/VerifyBlobKZGProofBatchPar(count=8)      	      72	  16752780 ns/op
Benchmark/VerifyBlobKZGProofBatchPar(count=16)     	      34	  33594562 ns/op
Benchmark/VerifyBlobKZGProofBatchPar(count=32)     	      16	  67546948 ns/op
Benchmark/VerifyBlobKZGProofBatchPar(count=64)     	       8	 134386234 ns/op

Benchmarks for C-KZG-4844 (Go bindings)

Benchmark/BlobToKZGCommitment-20         	      28	  40685543 ns/op
Benchmark/ComputeKZGProof-20             	      27	  42668193 ns/op
Benchmark/ComputeBlobKZGProof-20         	      27	  42853123 ns/op
Benchmark/VerifyKZGProof-20              	    1162	   1020716 ns/op
Benchmark/VerifyBlobKZGProof-20          	     742	   1609853 ns/op
Benchmark/VerifyBlobKZGProofBatch(count=1)-20         	     751	   1596980 ns/op
Benchmark/VerifyBlobKZGProofBatch(count=2)-20         	     452	   2648943 ns/op
Benchmark/VerifyBlobKZGProofBatch(count=4)-20         	     260	   4590053 ns/op
Benchmark/VerifyBlobKZGProofBatch(count=8)-20         	     141	   8483093 ns/op
Benchmark/VerifyBlobKZGProofBatch(count=16)-20        	      73	  16198360 ns/op
Benchmark/VerifyBlobKZGProofBatch(count=32)-20        	      37	  31688225 ns/op
Benchmark/VerifyBlobKZGProofBatch(count=64)-20        	      18	  62790266 ns/op

[sauliusgrigaitis]

There is a third option - rust-kzg. It exposes the same C API as c-kzg-4844 so you can use the same Go bindings that are provided with c-kzg-4844. See here for instructions.

[karalabe]

@jtraglia The trusted setup file seems to be slightly different for the two libs: The Go version requires some Lagrange form of G1 points whereas the C does not. Any particular reason for that / not being able to calculate it internally?

[sauliusgrigaitis]

Conversion is expensive. But I'm in favor that all the libs should use exactly the same trusted setup file.

[karalabe]

The trusted setup file in the Go lib has 4096 G2 points but the C lib rejects that because it wants 65 G2 points. Which is correct?

[sauliusgrigaitis]

65 G2 points is more than enough.

[jtraglia]

Hey @karalabe, good question. For both presets, there should only be 65 G2 points. I suspect Go-KZG-4844 has 4096 because that's how the trusted setup was defined initially; about 4 months ago, this PR changed the count from 4096 to 65. Also, you should use the trusted setups (mainnet and minimal) from the consensus-specs.

[lightclient]

4844 shipped in March 🎊 https://ethereum.org/en/roadmap/dencun/