Calldata compression

[boyuan-chen]

Calldata compression discussion #180 - closed #180

The calldata is compressed in the batch submitter. The core-utils handles the decompression. The data transport layer uses functions from core-utils and stores the decoded payloads in the database and shares with l2geth.

Unit tests are added and one new integration test is added. The new integration test uses the same method in data transport layer to decompress the calldata and search the transactions in sequencer and verifier nodes. This test helps us know that the verifier and sequencer have the same blocks and sync up correctly.

[codecov-commenter]

Codecov Report

Merging #184 (8d0f587) into develop (b47739e) will increase coverage by 0.05%.
The diff coverage is 48.00%.

@@             Coverage Diff             @@
##           develop     #184      +/-   ##
===========================================
+ Coverage    47.14%   47.20%   +0.05%     
===========================================
  Files           99       99              
  Lines         3875     3866       -9     
  Branches       761      762       +1     
===========================================
- Hits          1827     1825       -2     
+ Misses        2048     2041       -7     

Flag	Coverage Δ
boba-contracts	20.39% <ø> (ø)
contracts	77.71% <ø> (ø)
core-utils	86.77% <100.00%> (ø)
data-transport-layer	31.17% <13.33%> (+0.31%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
.../l1-ingestion/handlers/sequencer-batch-appended.ts	12.96% <13.33%> (-0.72%)	⬇️
packages/core-utils/src/optimism/batch-encoding.ts	89.30% <100.00%> (ø)

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update b47739e...8d0f587. Read the comment docs.

[InoMurko]

Awesome!

[InoMurko]

am I wrong at assuming that batch type legacy is applicable only if we want to keep the ability to NOT compress batches?

afaik - the moment we deploy - all new batches are compressed, the only reason why we would have "legacy" batches is for the verification to be able to read the uncompressed batches - like (/core-utils/src/optimism/batch-encoding.ts for DTL). correct?

[boyuan-chen]

Yes

[InoMurko]

I don't think this is needed anymore, right?
If all batches after deploy are compressed batches - brotli - why have different types of batches.
Or am I missing something in the implementation?

[boyuan-chen]

It's used for the Reader function. The Reader function is only used for the unit tests. We have to test the case of uncompressed data to the original data.

[InoMurko]

We have to test the case of uncompressed data to the original data.

in batch submitter? why?

[boyuan-chen]

This is the test data.

    {
      "name": "complex",
      "hex_encoding": "0102030405060708000004000102030405060708090a0b0c0d0e0f000102030405060708090a0b0c0d0e0f000102030405060708090a0b0c0d0e0f000102030405060708090a0b0c0d0e0f00000ac980808080808080808000000ac980808080808080808000000ac980808080808080808000000ac9808080808080808080",
      "should_start_at_element": 4328719365,
      "total_elements_to_append": 395016,
      "contexts": [
        {
          "num_sequenced_txs": 258,
          "num_subsequent_queue_txs": 197637,
          "timestamp": 25887770890,
          "block_number": 47446822415
        },
        {
          "num_sequenced_txs": 258,
          "num_subsequent_queue_txs": 197637,
          "timestamp": 25887770890,
          "block_number": 47446822415
        },
        {
          "num_sequenced_txs": 258,
          "num_subsequent_queue_txs": 197637,
          "timestamp": 25887770890,
          "block_number": 47446822415
        },
        {
          "num_sequenced_txs": 258,
          "num_subsequent_queue_txs": 197637,
          "timestamp": 25887770890,
          "block_number": 47446822415
        }
      ],
      "txs": [
        "c9808080808080808080",
        "c9808080808080808080",
        "c9808080808080808080",
        "c9808080808080808080"
      ],
      "error": false
    },

The actual tx data is decoded (Reader) from hex_encoding and contexts. The batch submitter compresses data using the Write() function and decompresses data using Reader() function. We need to compare the actual tx data with the decompressed data from Reader() function. It's very hard to compare the tx data without the BatchTypeLegacy type in the Reader function that actually decodes the payload.