Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[ProjectTracking]: Stateless validation StatelessNet #20

Open
42 of 64 tasks
Tracked by #72
walnut-the-cat opened this issue Jan 10, 2024 · 8 comments
Open
42 of 64 tasks
Tracked by #72

[ProjectTracking]: Stateless validation StatelessNet #20

walnut-the-cat opened this issue Jan 10, 2024 · 8 comments
Assignees
@wacban @walnut-the-cat
Labels
P0 project-tracking T-core T-node

Comments

@walnut-the-cat
Copy link

walnut-the-cat commented Jan 10, 2024 •
edited
Loading

Goals

  • Demonstrate stateless validation to public so they have better understanding on what we are working on
  • Leverage community power to identify critical issues and stress test the network with stateless validation
  • Figure out detailed MainNet launch requirement and plan based on findings

Links to external documentations and discussions

  • StatelessNet planning doc: LINK
  • NEP: LINK

Estimated effort

  • StatelessNet launch preparation: 2-3 weeks
  • StatelessNet duration: 8+ weeks
  • StatelessNet data analysis/post-mortem: 2 weeks
  • [Out of scope] Implementation of the follow up items identified during StatelessNet

Timeline

Start StatelessNet stage 1: Late January/Early February

We aim to open StatelessNet to the public on January 29th so community members can test features and generate traffic.

Requirements

  • 3 weeks before
  • 2 weeks before
    • Stateless validation MVP should go through basic tests (unit tests, integration tests, load tests).
  • 1 week before
    • Stateless validation MVP must be merged into master and be ready for distribution.
    • StatelessNet network should be set up along with necessary monitoring tools.

EthDenver: Feb 29th - March 3rd, 2024

Bowen Wang plans to share findings from Stateless validation StatelessNet during EthDenver.

Requirements

  • 1 week before
    • Data from StatelessNet should be gathered and analyzed in presentable format.
  • 2 weeks before
    • At least 1-2 rounds of performance optimization and stability enhancement should be in place based on incoming data.

Start StatelessNet Stage 2: Early March (at least 4 weeks after stage 1)

Expand scenario coverage of StatelessNet to start testing corner cases.

Mid-point check-in: March 15th, 2024

Continue collecting data and make improvements.

Complete StatelessNet: March 31st, 2024 (or TBD)

  • Wrap up the StatelessNet test and share the final report.
  • Need to have enough confidence on resilience, scalability, reliability of the network before completing StatelessNet and the end date may move around.

Assumptions

  • NF will support public communication amplification
  • Project is approved by NF with enough budgets
  • At least 20 validators will be willing to participate to test stateless validation before MainNet launch
  • Public interest will be high enough to run meaningful reward program

Pre-requisites

  • MVP is complete and StatelessNet binary is ready by end of January
  • StatelessNet set up and configuration are ready by end of January

Out of scope

  • TestNet/MainNet launch
  • Implementation of items identified during StatelessNet
  • StatelessNet generalization for future use

TODO

Before StatelessNet launch - Core

  • Complete MVP
  • E2E integration test
  • Internal sanity test

Before StatelessNet launch - Node

Before StatelessNet launch - Operational

  • Finalize
    • Public communication message
    • Reward program
    • High level timeline
  • Get budget alignment with NEAR Foundation
  • Create
    • An official support group on Telegram (Olga) #32
      • Potential name:
        • Stake Wars IV general support
        • Stake Wars IV validator support
      • Should promote channel in the following telegram channels:
        • NEAR Dev
        • NEAR Protocol community
        • NEAR Tools Community
        • Protocol validator channel (Do not know the exact name...)
    • Create Zulip Channel
    • GitHub repository for reward program
      • Note: a single Github repo maybe used for both program description as well as reward program
      • Potential name: near/stakewars_iv (similar to https://github.com/near/stakewars-iii)
      • Necessary contents
        • Program overview
        • Reward program overview
        • Issue template / label for submission
        • Reference links
  • Get X account access (or figure out a way to make a post on NEAR X account)
    • account to use: @NearStakeWars
  • Locate proper communication channel on Near.org
  • Partner communication
    • Aurora
    • SWEAT
    • Kaiching
  • Document to write
    • Reward program repo README
    • Stateless validation overview
    • NEP draft
  • Communicate with validators on StatelessNet

Before StatelessNet Stage 2

TBD

Operational

  • Google form for validator application
    • https://forms.gle/Nmdq6YAgTKD4pMqGA
    • What are the information we should gather?
      • MainNet account name
      • Node resource set up that will be used for validator node
      • Region
      • agreement to enable telemetry/profiling/debug page
      • agreement for automatic binary update?

Techincal

  • Stress test / load test
  • Performance test: Due. Feb 21st
    • We would like to have some performance data on what stateless validation and In-memory trie can enable. For instance, performance improvement data for fungible token transfer per sec per shard. We acknowledge that the number may not be accurate until all optimization/stabilization/bug-fixes are in, but Bowen can use such data to give high level overview during EthDenver.

After StatelessNet completion

TBD
@Longarithm Longarithm mentioned this issue Jan 29, 2024
Merged
github-merge-queue bot pushed a commit to near/nearcore that referenced this issue Jan 29, 2024
@Longarithm
feat: statelessnet protocol (#10518)
abb2b28 
Set special protocol feature and version to launch
near/near-one-project-tracking#20 against.

---------

Co-authored-by: Longarithm <the.aleksandr.logunov@gmail.com>
@staffik staffik mentioned this issue Feb 1, 2024
Closed
7 tasks
@Longarithm
Copy link
Member

Longarithm commented Feb 7, 2024 •
edited by walnut-the-cat
Loading

Feb 8th Core team update

@Longarithm Longarithm mentioned this issue Feb 9, 2024
Merged
github-merge-queue bot pushed a commit to near/nearcore that referenced this issue Feb 10, 2024
@Longarithm
test: statelessnet ci (#10589)
4756b78 
Introduce automatic CI checks for latest statelessnet binary used for
near/near-one-project-tracking#20. This PR
adopts new check automatically.
The check is not required, in order not to block main development flow.
Wanted to make this trigger only on demand, but there is no simple way
to do it.

---------

Co-authored-by: Longarithm <the.aleksandr.logunov@gmail.com>
@walnut-the-cat
Copy link
Author

walnut-the-cat commented Feb 26, 2024 •
edited by pugachAG
Loading

Feb 26th update

  • Majority of issues discovered at the beginning of stake wars program have been resolved now
  • Orphan state pool implementation is now merged. The team will re-enable mainnet traffic mirroring soon
  • [WIP] Testloop refactoring is in progress; the work is necessary to enable in-memory trie with proper testing
  • Proper genesis handling & test fixes
  • [WIP] Performance refactoring iteration based on Bowen's latest feedback. Plan to share the result by Feb 27th.
  • The team started reviewing incoming reward program proposals to prepare for start of the program with Stage 2
  • Community validator onboarding in progress
  • Other MainNet release related topics will now be tracked in the tracking issue 46

@bowenwang1996
Copy link

[WIP] Testloop refactoring is in progress; the work is necessary to enable in-memory trie with proper testing

How much time is this going to take?

Proper genesis handling & test fixes

What does this mean?

@walnut-the-cat
Copy link
Author

Regarding testloop, it's being discussed in this Zulip thread. I expect it to be mostly done this week, but am double checking with Robin.

@walnut-the-cat
Copy link
Author

Proper genesis handling & test fixes

This is part of @pugachAG 's Better Engineering effort to make code base more manageable and easy to develop. So far, these two PRs are related:

It will be followed by removal of mock epoch manager

@wacban
Copy link

wacban commented Mar 11, 2024 •
edited by walnut-the-cat
Loading

March 11th Update

  • discovered and debugging a number of issues
    • stack overflow - almost fixed
    • one shard stalling under mainnet load - under investigation
    • crash in memtrie - under investigation (cc. @staffik)
    • empty blocks and missing chunks - looking for an owner
  • the next milestone, in-memory-trie and state sync integration - implementation is in progressder

@wacban
Copy link

wacban commented Apr 5, 2024 •
edited
Loading

April 5th Update

We deployed a number of new features to statelessnet but unfortunately the chain suffered tremendously due to bugs. We decided to shut down the chain as maintaining it on unstable code was consuming too much developers' time. We are going to continue development and testing on dedicated test networks. We may resume statelessnet once we've gained more confidence in it.

  • finished features
    • single shard tracking using state sync
    • memtrie loading and unloading during state sync
    • soft state witness size limit
  • fixed bugs
    • missing trie value issue
    • missing storage recording issue
    • (statelessnet only) reward calculation panic due to arithmetic overflows
    • (statelessnet only) maintenance scripts accidentaly unstaking all tokens
  • misc
    • Attempted mirroring mainnet traffic but failed due to networking constraints. Search for a solution is in progress.

@walnut-the-cat
Copy link
Author

Moving the task to 'Ready to be prioritized' from 'WIP' as we are pausing StatelessNet

@walnut-the-cat walnut-the-cat mentioned this issue Jun 25, 2024
Open
28 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@wacban wacban

@walnut-the-cat walnut-the-cat

Labels
P0 project-tracking T-core T-node
Projects
Near One project tracking
Status: Done
Milestone
No milestone
Development

No branches or pull requests
4 participants
@wacban @Longarithm @bowenwang1996 @walnut-the-cat