Spec compliance testing

[JorikSchellekens]

To ensure that Starknet is ready for a major update, we should have a simple set of tooling which can test compliance of the Starknet Stack with the specification. This set of tools should have an easy CLI wrapper which pulls in all the latest implementations of the tested software and run all of the tests against these implementations. It should be easy to configure new spec compliant implementations. There should also be an informative readiness status on a Starknet Status page. The testing framework should be similar to Ethereum's HIVE, making no assumptions about the underlying implementations (except perhaps in virtual machines, where it should only make assumptions about the format of the resulting state diffs and traces).

A modification to the Starknet specifications SHOULD be merged with a modification of the relevant testing framework, effectively making an ecosystem-wide test-driven development process. I use SHOULD in the IETF sense here because that may not always be feasible.

The following seem like important areas which require testing:

• RPC compliance testing
  • Automated property tester
  • Handwritten tests to test the set and structure of endpoints
  • DSL to test discrete scenarios, similar to https://github.com/keep-starknet-strange/gomu-gomu-no-gatling but without emphasis on performance
• RPC load testing and benchmarks
  • https://github.com/keep-starknet-strange/gomu-gomu-no-gatling
  • Random data - I'm told this is not so useful from the Nethermind Ethereum team
  • Accelerated tx replay - effectively run a shadow fork at Nx speed.
• VM benchmarks
  • Standard programs
• VM correctness tests
  • Unit: 1 instruction tests
  • Unit: 1 builtin tests
  • Unit: Hint tests
  • Sample: Handwritten programs
  • End to end: https://github.com/NethermindEth/cairo-property-tester
• Sierra correctness tests
  • As above but requires a formalisation of Sierra (being worked on by StarkWare, Nethermind can help)
• Sierra - Cairo VM differential testing
  • Run a large collection of programs and compare their state updates
  • Can be auto-generated programs as in https://github.com/NethermindEth/cairo-property-tester
• VM differential testing
  • As above, but between Cairo VMs and compare their traces
  • This is already the defacto testing done by the LambdaClass VM
• Networking tests
  • p2p syncing tests
    • Scenario testing
    • Random network configuration testing
    • Nethermind has already started with Kurtosis
• Sync benchmarks

I believe that all of these tests should reside in a repo in the Starknet-io. I want to open a discussion about what is missing from this list and how to collaborate on this effectively across all of the teams.

Open questions:

• Should we test the wallets according to https://community.starknet.io/t/snip-standardizing-starknet-wallet-connections-with-write-api-and-json-rpc-integration/98049
• Can/should we test SDKs?
• What structure should be used to represent each of these tests?

[Mirko-von-Leipzig]

RPC compliance testing

• Handwritten tests to test the set and structure of endpoints

This can be somewhat automated by extracting the information from the specification. We've done this recently in pathfinder for method availability. A json-rpc compliant server should return MethodNotFound error if the endpoint does not contain a method which allows for a very simple test.

Generating valid input is much more complicated to test that is much more complicated. And probably only worth the effort for ossified parts of the specification for now.

[kuanglixu]

我爱你

[w1986]

good

[antonini13]

yew

[mb-billions]

Awesome idea graet

[Armankhan566]

Why I am not eligible for starknet airdrop

0x0326e95A1E7Be0011313C320cF88A57F77fDDE5C330443b35D048E03f9eE794D

This is my address 600 dollar involved in this address please eligible me starknet

[Valrob28]

Have you commits anything there ? or at least do u have 0,005 eth in your STARK wallet ? how maybe ur stack is locked into some LP. Arrrh hope this comments could help ! at least we should be eligible for the second round if they too or conversation into the snapchoot .