Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat(vm-runner): Implement batch data prefetching #2724

Merged
merged 20 commits into from
Aug 30, 2024
Merged

feat(vm-runner): Implement batch data prefetching #2724

merged 20 commits into from
Aug 30, 2024

Conversation

slowli
Copy link
Contributor

@slowli slowli commented Aug 23, 2024 •
edited
Loading

What ❔

  • Implements prefetching of storage slots / bytecodes accessed by a VM in a batch. Enables it for the VM playground. Optionally shadows prefetched snapshot storage.
  • Makes RocksDB cache optional for VM playground.

Why ❔

  • Prefetching will allow to load storage slots / bytecodes for a batch in O(1) DB queries, which is very efficient for local debugging etc. It may be on par or faster than using RocksDB cache. (There's a caveat: prefetching doesn't work w/o protective reads.)
  • Disabling RocksDB cache is useful for local testing, since the cache won't catch up during a single batch run anyway.

Checklist

  • PR title corresponds to the body of PR (we generate changelog entries from PRs).
  • Tests for the changes have been added / updated.
  • Documentation comments have been added / updated.
  • Code has been formatted via zk fmt and zk lint.

@slowli slowli marked this pull request as ready for review August 27, 2024 13:13
@slowli slowli mentioned this pull request Aug 27, 2024
Merged
4 tasks
itegulov
itegulov previously approved these changes Aug 29, 2024
View reviewed changes
Copy link
Contributor

@itegulov itegulov left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good overall, just a couple of nits/suggestions

core/lib/state/src/shadow_storage.rs Outdated Show resolved Hide resolved
core/lib/state/src/shadow_storage.rs Outdated Show resolved Hide resolved
core/lib/state/src/storage_factory.rs Show resolved Hide resolved
@github-actions GitHub Actions
Copy link
Contributor

Detected VM performance changes

Benchmark name change in estimated runtime change in number of opcodes executed
call_far -2.2% +0 (NaN%)

Changes in number of opcodes executed indicate that the gas price of the benchmark has changed, which causes it run out of gas at a different time. Or that it is behaving completely differently.

perekopskiy
perekopskiy approved these changes Aug 29, 2024
View reviewed changes
Copy link
Contributor

@perekopskiy perekopskiy left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice!

@slowli slowli added this pull request to the merge queue Aug 30, 2024
@github-merge-queue github-merge-queue bot removed this pull request from the merge queue due to failed status checks Aug 30, 2024
@slowli slowli added this pull request to the merge queue Aug 30, 2024
github-merge-queue bot pushed a commit that referenced this pull request Aug 30, 2024
@slowli
feat(vm-runner): Implement batch data prefetching (#2724)
6ab7c62 
## What ❔

- Implements prefetching of storage slots / bytecodes accessed by a VM
in a batch. Enables it for the VM playground. Optionally shadows
prefetched snapshot storage.
- Makes RocksDB cache optional for VM playground.

## Why ❔

- Prefetching will allow to load storage slots / bytecodes for a batch
in O(1) DB queries, which is very efficient for local debugging etc. It
may be on par or faster than using RocksDB cache. (There's a caveat:
prefetching doesn't work w/o protective reads.)
- Disabling RocksDB cache is useful for local testing, since the cache
won't catch up during a single batch run anyway.

## Checklist

- [x] PR title corresponds to the body of PR (we generate changelog
entries from PRs).
- [x] Tests for the changes have been added / updated.
- [x] Documentation comments have been added / updated.
- [x] Code has been formatted via `zk fmt` and `zk lint`.
@github-merge-queue github-merge-queue bot removed this pull request from the merge queue due to failed status checks Aug 30, 2024
@slowli slowli added this pull request to the merge queue Aug 30, 2024
Merged via the queue into main with commit d01840d Aug 30, 2024
48 checks passed
@slowli slowli deleted the aov-pla-1022-implement-batch-data-prefetching-in-vm-runner branch August 30, 2024 10:55
@zksync-era-bot zksync-era-bot mentioned this pull request Aug 30, 2024
Merged
slowli added a commit that referenced this pull request Sep 3, 2024
@slowli
Revert "feat(vm-runner): Implement batch data prefetching (#2724)"
878c995 
This reverts commit d01840d.
slowli added a commit that referenced this pull request Sep 3, 2024
@slowli
Revert "feat(vm-runner): Implement batch data prefetching (#2724)"
221b5b9 
This reverts commit d01840d.
slowli added a commit that referenced this pull request Sep 3, 2024
@slowli
Revert "feat(vm-runner): Implement batch data prefetching (#2724)"
b7c969e 
This reverts commit d01840d.
github-merge-queue bot pushed a commit that referenced this pull request Sep 5, 2024
@zksync-era-bot
chore(main): release core 24.24.0 (#2773)
fcffb06 
🤖 I have created a release *beep* *boop*
---


##
[24.24.0](core-v24.23.0...core-v24.24.0)
(2024-09-05)


### Features

* conditional cbt l1 updates
([#2748](#2748))
([6d18061](6d18061))
* **eth-watch:** do not query events from earliest block
([#2810](#2810))
([1da3f7e](1da3f7e))
* **genesis:** Validate genesis config against L1
([#2786](#2786))
([b2dd9a5](b2dd9a5))
* Integrate tracers and implement circuits tracer in vm2
([#2653](#2653))
([87b02e3](87b02e3))
* Move prover data to
/home/popzxc/workspace/current/zksync-era/prover/data
([#2778](#2778))
([62e4d46](62e4d46))
* Remove prover db from house keeper
([#2795](#2795))
([85b7346](85b7346))
* **vm-runner:** Implement batch data prefetching
([#2724](#2724))
([d01840d](d01840d))
* **vm:** Extract batch executor to separate crate
([#2702](#2702))
([b82dfa4](b82dfa4))
* **vm:** Simplify VM interface
([#2760](#2760))
([c3bde47](c3bde47))
* **zk_toolbox:** add multi-chain CI integration test
([#2594](#2594))
([05c940e](05c940e))


### Bug Fixes

* **config:** Do not panic for observability config
([#2639](#2639))
([1e768d4](1e768d4))
* **core:** Batched event processing support for Reth
([#2623](#2623))
([958dfdc](958dfdc))
* return correct witness inputs
([#2770](#2770))
([2516e2e](2516e2e))
* **tee-prover:** increase retries to reduce spurious alerts
([#2776](#2776))
([4fdc806](4fdc806))
* **tee-prover:** mitigate panic on redeployments
([#2764](#2764))
([178b386](178b386))
* **tee:** lowercase enum TEE types
([#2798](#2798))
([0f2f9bd](0f2f9bd))
* **vm-runner:** Fix statement timeouts in VM playground
([#2772](#2772))
([d3cd553](d3cd553))


### Performance Improvements

* **vm:** Fix VM performance regression on CI loadtest
([#2782](#2782))
([bc0d7d5](bc0d7d5))

---
This PR was generated with [Release
Please](https://github.com/googleapis/release-please). See
[documentation](https://github.com/googleapis/release-please#release-please).

---------

Co-authored-by: zksync-era-bot <zksync-era-bot@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@perekopskiy perekopskiy perekopskiy approved these changes

@itegulov itegulov itegulov left review comments

@RomanBrodetski RomanBrodetski Awaiting requested review from RomanBrodetski

@popzxc popzxc Awaiting requested review from popzxc

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@slowli @itegulov @perekopskiy