feat: rewritten gossip sync to be async from block processing #711

pompon0 · 2023-12-19T13:27:13Z

Simplified configs to not require genesis block.
Removed consensus fields from jsonrpc sync api, to prevent having gaps in miniblock quorum certificates.
Moved consensus fields to a separate table to make first/last entry lookup fast and reliable.
Migrated consensus integration to new era-consensus API: Simplified block storage era-consensus#53
Deduplicated code between EN consensus integration and main node consensus integration
Implemented support for cloning a db in tests

…fig-work

popzxc

Looks good to me, but I feel that my domain knowledge isn't sufficient to grok changes fully. I'd highly recommend getting a 2nd review from someone with enough domain knowledge.

core/lib/dal/src/connection/mod.rs

core/lib/dal/src/consensus_dal.rs

core/lib/dal/src/models/proto/mod.proto

RomanBrodetski

Looks good! @slowli were you also planning to review this?

core/lib/dal/src/sync_dal.rs

core/lib/dal/migrations/20240103123456_move_consensus_fields_to_new_table.up.sql

core/lib/dal/src/consensus_dal.rs

core/lib/zksync_core/src/consensus/storage.rs

core/lib/zksync_core/src/consensus/mod.rs

core/lib/dal/src/models/proto/mod.proto

…fig-work

core/lib/dal/src/sync_dal.rs

core/lib/dal/src/models/storage_sync.rs

core/lib/dal/src/models/proto/mod.proto

core/lib/zksync_core/src/consensus/storage/testonly.rs

core/lib/zksync_core/src/consensus/testonly.rs

core/lib/zksync_core/src/consensus/storage.rs

core/lib/zksync_core/src/consensus/storage/mod.rs

core/lib/zksync_core/src/consensus/mod.rs

core/lib/zksync_core/src/consensus/storage/mod.rs

slowli

As discussed privately, there's a couple of outstanding questions (e.g., whether it make sense to have Payload defined / validated in the DAL crate), but they can be revisited later.

Also, an operational nit: I think this PR merits a "feat" prefix; it brings quite a few improvements, and "refactor" PRs don't show up in the changelog.

🤖 I have created a release *beep* *boop* --- ## [19.2.0](core-v19.1.1...core-v19.2.0) (2024-01-17) ### Features * adds `zk linkcheck` to zk tool and updates zk env for `zk linkcheck` ci usage ([#868](#868)) ([d64f584](d64f584)) * **contract-verifier:** Support zkVM solc contract verification ([#854](#854)) ([1ed5a95](1ed5a95)) * **en:** Make batch status updater work with pruned data ([#863](#863)) ([3a07890](3a07890)) * rewritten gossip sync to be async from block processing ([#711](#711)) ([3af4644](3af4644)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please).

🤖 I have created a release *beep* *boop* --- [19.2.0](core-v19.1.1...core-v19.2.0) (2024-01-17) * adds `zk linkcheck` to zk tool and updates zk env for `zk linkcheck` ci usage ([#868](#868)) ([d64f584](d64f584)) * **contract-verifier:** Support zkVM solc contract verification ([#854](#854)) ([1ed5a95](1ed5a95)) * **en:** Make batch status updater work with pruned data ([#863](#863)) ([3a07890](3a07890)) * rewritten gossip sync to be async from block processing ([#711](#711)) ([3af4644](3af4644)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please).

pompon0 added 7 commits December 13, 2023 17:16

strongly typed config

bcc0c8d

WIP making it compile

26e43f1

compiles with todos

6f9539d

wip shifting payload to dal

cb26d29

snapshot: before FinalBlock -> CommitQC

5ee4456

simplified stuff

b896301

snapshot

dc103b0

pompon0 changed the title ~~Gprusak consensus config work~~ [WIP] rewriting gossip sync to be async from block processing Dec 19, 2023

pompon0 added 11 commits December 20, 2023 13:36

last_in_batch in payload

0ea4776

deduplicated gossip storage

033d690

test db copying

19850df

cloning db works

3039ef1

tests implemented, but fail

93b7fa0

fast compilation

31666b8

consensus tests and documentation

2c3b870

Merge remote-tracking branch 'origin/main' into gprusak-consensus-con…

432ada4

…fig-work

merged

261ad59

zk fmt

926cd26

zk fmt, reverted en configuration

cc83a75

pompon0 changed the title ~~[WIP] rewriting gossip sync to be async from block processing~~ [refactor] rewritten gossip sync to be async from block processing Jan 3, 2024

pompon0 changed the title ~~[refactor] rewritten gossip sync to be async from block processing~~ refactor: rewritten gossip sync to be async from block processing Jan 3, 2024

updated deps

27ed6b8

pompon0 requested a review from slowli January 3, 2024 12:10

pompon0 marked this pull request as ready for review January 3, 2024 12:10

pompon0 requested a review from popzxc January 3, 2024 12:10

pompon0 added 2 commits January 3, 2024 13:23

spellcheck

16e73f9

removed unused deps

827e2d1

popzxc previously approved these changes Jan 3, 2024

View reviewed changes

core/lib/dal/src/connection/mod.rs Outdated Show resolved Hide resolved

core/lib/dal/src/consensus_dal.rs Outdated Show resolved Hide resolved

core/lib/dal/src/models/proto/mod.proto Outdated Show resolved Hide resolved

pompon0 requested a review from brunoffranca January 3, 2024 13:18

applied comments

4a9d4f3

RomanBrodetski previously approved these changes Jan 3, 2024

View reviewed changes

core/lib/dal/src/sync_dal.rs Show resolved Hide resolved

slowli reviewed Jan 5, 2024

View reviewed changes

added test of fetcher backfilling certs

ef6f949

pompon0 dismissed RomanBrodetski’s stale review via ef6f949 January 8, 2024 14:03

pompon0 requested a review from slowli January 8, 2024 14:03

pompon0 added 3 commits January 8, 2024 15:21

removed the dal testonly method

7e70bc0

bumped era-consensus

706559a

Merge remote-tracking branch 'origin/main' into gprusak-consensus-con…

24ea1a6

…fig-work

slowli reviewed Jan 10, 2024

View reviewed changes

pompon0 added 3 commits January 10, 2024 13:44

missing changes

5f12db2

sqlx queries

3a6d733

applied comments

7ec7094

pompon0 requested review from slowli and popzxc January 10, 2024 14:06

pompon0 added 2 commits January 10, 2024 16:17

lint

dfa7497

Merge branch 'main' into gprusak-consensus-config-work

622d26d

slowli approved these changes Jan 12, 2024

View reviewed changes

pompon0 changed the title ~~refactor: rewritten gossip sync to be async from block processing~~ feat: rewritten gossip sync to be async from block processing Jan 13, 2024

Merge branch 'main' into gprusak-consensus-config-work

556e429

pompon0 enabled auto-merge January 13, 2024 09:55

pompon0 added this pull request to the merge queue Jan 13, 2024

github-merge-queue bot removed this pull request from the merge queue due to failed status checks Jan 13, 2024

pompon0 added this pull request to the merge queue Jan 13, 2024

Merged via the queue into main with commit 3af4644 Jan 13, 2024
27 checks passed

pompon0 deleted the gprusak-consensus-config-work branch January 13, 2024 12:07

zksync-era-bot mentioned this pull request Jan 13, 2024

chore(main): release core 19.2.0 #867

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: rewritten gossip sync to be async from block processing #711

feat: rewritten gossip sync to be async from block processing #711

pompon0 commented Dec 19, 2023 •

edited

Loading

popzxc left a comment

RomanBrodetski left a comment

slowli left a comment •

edited

Loading

feat: rewritten gossip sync to be async from block processing #711

feat: rewritten gossip sync to be async from block processing #711

Conversation

pompon0 commented Dec 19, 2023 • edited Loading

popzxc left a comment

Choose a reason for hiding this comment

RomanBrodetski left a comment

Choose a reason for hiding this comment

slowli left a comment • edited Loading

Choose a reason for hiding this comment

pompon0 commented Dec 19, 2023 •

edited

Loading

slowli left a comment •

edited

Loading