Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Flat storage upgrade on nightly #8501

Closed
exalate-issue-sync bot opened this issue Feb 3, 2023 · 3 comments
Closed

Flat storage upgrade on nightly #8501

exalate-issue-sync bot opened this issue Feb 3, 2023 · 3 comments

Comments

@exalate-issue-sync
Copy link

exalate-issue-sync bot commented Feb 3, 2023 •
edited by Longarithm
Loading

Remove feature protocol_feature_flat_state and partially enable FlatStorage for master. In this stage, by default, the binary will maintain the db columns needed for FlatStorage, but storage reads won't need FlatStorage at all. This should have minimal impact in node performance since it only requires writes to flat storage columns, DB migration will be done in this stage, either offline or while the node is running, depending on how long the migration takes.

Then release flat storage in a protocol upgrade, together with the fee adjustment.
@Longarithm Longarithm mentioned this issue Feb 3, 2023
Closed
26 tasks
@Longarithm
Copy link
Member

We've started discussing part of rollout - possible change of RAM requirements - starting from https://near.zulipchat.com/#narrow/stream/345766-pagoda.2Fstorage.2Fflat-storage/topic/limiting.20RAM.20usage/near/338942828.

Currently we need 2 GB of RAM more, but it would lead to increasing minimal HW requirements from 16 to 18 GB, and it would decrease profitability of small validators, because it would move them to next tier for most cloud providers. We will investigate how to proceed here.

Another blocker here is partially #8577 - for smooth release we want to get rid of excessive feature flags.

@Longarithm Longarithm mentioned this issue Mar 9, 2023
Merged
near-bulldozer bot pushed a commit that referenced this issue Mar 20, 2023
@Longarithm
feat: flat storage rollout on nightly (#8697)
bc25a2d 
Here we switch from assertions of flat storage vs. trie reads to reads from flat storage only. Part of #8501.

Also moving `create_flat_storage_manager` to `FlatStorageManager::test` - I need to reuse it in Runtime tests.

## Testing

* `test_storage_read_write_costs_runtime` - runtime-specific test to check costs for subsequent storage read and write. It emphasizes the difference in costs before and after flat storage. Note that without flat storage we still can call contracts, but results, especially costs, will likely diverge from results on chain.
* `test_flat_storage_upgrade` - integration test for protocol upgrade correctness check
* ~https://nayduck.near.org/#/run/2901~ https://nayduck.near.org/#/run/2907
@Longarithm
Copy link
Member

We agreed on reducing RAM usage for shard 3 to 1 GB, which allows us to rollout flat storage without extra announcements.
We also agreed on costs, we just need to wait until compute costs implementation is fully merged.
#8761 should close the issue.

@Longarithm Longarithm changed the title Flat storage rollout Flat storage upgrade on nightly Mar 28, 2023
@Longarithm
Copy link
Member

We didn't achieve full flat storage rollout as planned, because we weren't confident enough in the code and also we need to wait for compute costs release.
Still the flat storage feature is ready, we just need to make sure it goes to mainnet. I will create separate issue for that.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@Longarithm