Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

MantaPay stress-test benchmark #928

Merged
merged 61 commits into from
Dec 25, 2022
Merged

Conversation

ghzlatarev
Copy link
Contributor

@ghzlatarev ghzlatarev commented Dec 19, 2022

Description

closes: #861

  • The idea of the stress test is to send lots of transactions on top of a large ledger, and making sure they succeed in a certain amount of time.
  • Specifically the amount of transactions is 700 in the form to-private -> to-private -> to-private -> private-transfer -> to-private -> to-private -> to-public.
  • The size of the ledger is ~100k transactions of the same form, which populate all the storage items of the pallet.
  • The asserts are that transactions per second from start to last finalized transaction is more than 0.5. Also that all expected success events are recorded on the ledger.
  • The test runs on the reference hardware that we also use for benchmarking weights

Before we can approve this PR for merge, please make sure that all the following items have been checked off:

  • Linked to Github issue with discussion and accepted design OR have an explanation in the PR that describes this work.
  • Added one label out of the L- group to this PR
  • Added one or more labels from the A- and C- groups to this PR
  • Explicitly labelled A-calamari, A-dolphin and/or A-manta if your changes are meant for/impact either of these (CI depends on it)
  • This PR is targeted against the current Milestone ( otherwise discuss if it can be added in time)
  • Re-reviewed Files changed in the Github PR explorer.

Situational Notes:

  • If adding functionality, write unit tests!
  • If importing a new pallet, choose a proper module index for it, and allow it in BaseFilter. Ensure every extrinsic works from front-end. If there's corresponding tool, ensure both work for each other.
  • If needed, update our Javascript/Typescript APIs. These APIs are officially used by exchanges or community developers.
  • If modifying existing runtime storage items, make sure to implement storage migrations for the runtime and test them with try-runtime. This includes migrations inherited from upstream changes, and you can search the diffs for modifications of #[pallet::storage] items to check for any.
  • If runtime changes, need to update the version numbers properly:
    • authoring_version: The version of the authorship interface. An authoring node will not attempt to author blocks unless this is equal to its native runtime.
    • spec_version: The version of the runtime specification. A full node will not attempt to use its native runtime in substitute for the on-chain Wasm runtime unless all of spec_name, spec_version, and authoring_version are the same between Wasm and native.
    • impl_version: The version of the implementation of the specification. Nodes are free to ignore this; it serves only as an indication that the code is different; as long as the other two versions are the same then while the actual code may be different, it is nonetheless required to do the same thing. Non-consensus-breaking optimizations are about the only changes that could be made which would result in only the impl_version changing.
    • transaction_version: The version of the extrinsics interface. This number must be updated in the following circumstances: extrinsic parameters (number, order, or types) have been changed; extrinsics or pallets have been removed; or the pallet order in the construct_runtime! macro or extrinsic order in a pallet has been changed. You can run the metadata_diff.yml workflow for help. If this number is updated, then the spec_version must also be updated
  • Verify benchmarks & weights have been updated for any modified runtime logics

Signed-off-by: Georgi Zlatarev <georgi.zlatarev@manta.network>
Signed-off-by: Georgi Zlatarev <georgi.zlatarev@manta.network>
@ghzlatarev ghzlatarev added A-calamari Area: Issues and PRs related to the Calamari Runtime A-dolphin Area: Issues and PRs related to the Dolphin Runtime labels Dec 19, 2022
Signed-off-by: Georgi Zlatarev <georgi.zlatarev@manta.network>
Signed-off-by: Georgi Zlatarev <georgi.zlatarev@manta.network>
Signed-off-by: Georgi Zlatarev <georgi.zlatarev@manta.network>
Signed-off-by: Georgi Zlatarev <georgi.zlatarev@manta.network>
Signed-off-by: Georgi Zlatarev <georgi.zlatarev@manta.network>
Signed-off-by: Georgi Zlatarev <georgi.zlatarev@manta.network>
Signed-off-by: Georgi Zlatarev <georgi.zlatarev@manta.network>
Signed-off-by: Georgi Zlatarev <georgi.zlatarev@manta.network>
Signed-off-by: Georgi Zlatarev <georgi.zlatarev@manta.network>
Signed-off-by: Georgi Zlatarev <georgi.zlatarev@manta.network>
Signed-off-by: Georgi Zlatarev <georgi.zlatarev@manta.network>
Signed-off-by: Georgi Zlatarev <georgi.zlatarev@manta.network>
Signed-off-by: Georgi Zlatarev <georgi.zlatarev@manta.network>
Signed-off-by: Georgi Zlatarev <georgi.zlatarev@manta.network>
Signed-off-by: Georgi Zlatarev <georgi.zlatarev@manta.network>
Signed-off-by: Georgi Zlatarev <georgi.zlatarev@manta.network>
Signed-off-by: Georgi Zlatarev <georgi.zlatarev@manta.network>
Signed-off-by: Georgi Zlatarev <georgi.zlatarev@manta.network>
Signed-off-by: Georgi Zlatarev <georgi.zlatarev@manta.network>
Signed-off-by: Georgi Zlatarev <georgi.zlatarev@manta.network>
Signed-off-by: Georgi Zlatarev <georgi.zlatarev@manta.network>
Signed-off-by: Georgi Zlatarev <georgi.zlatarev@manta.network>
Signed-off-by: Georgi Zlatarev <georgi.zlatarev@manta.network>
Signed-off-by: Georgi Zlatarev <georgi.zlatarev@manta.network>
Signed-off-by: Georgi Zlatarev <georgi.zlatarev@manta.network>
@ghzlatarev ghzlatarev marked this pull request as ready for review December 23, 2022 09:56
Signed-off-by: Georgi Zlatarev <georgi.zlatarev@manta.network>
Signed-off-by: Georgi Zlatarev <georgi.zlatarev@manta.network>
Signed-off-by: Georgi Zlatarev <georgi.zlatarev@manta.network>
Copy link
Contributor

@ferrell-code ferrell-code left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some nits

tests/stress_benchmark_test.ts Outdated Show resolved Hide resolved
tests/stress_benchmark_test.ts Outdated Show resolved Hide resolved
tests/append_storage.js Show resolved Hide resolved
Signed-off-by: Georgi Zlatarev <georgi.zlatarev@manta.network>
ferrell-code
ferrell-code previously approved these changes Dec 23, 2022
.gitattributes Outdated Show resolved Hide resolved
Copy link
Contributor

@Garandor Garandor left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

main comment on avoiding git lfs, looks good otherwise

Adam Reif and others added 2 commits December 23, 2022 15:39
Signed-off-by: Georgi Zlatarev <georgi.zlatarev@manta.network>
ghzlatarev and others added 4 commits December 24, 2022 15:09
Signed-off-by: Georgi Zlatarev <georgi.zlatarev@manta.network>
Signed-off-by: Georgi Zlatarev <georgi.zlatarev@manta.network>
Signed-off-by: Georgi Zlatarev <georgi.zlatarev@manta.network>
@ghzlatarev ghzlatarev merged commit 4bda399 into manta Dec 25, 2022
@ghzlatarev ghzlatarev deleted the ghzlatarev/stress-benchmark-test branch December 25, 2022 21:15
ferrell-code pushed a commit that referenced this pull request Dec 27, 2022
* Initial commit for stress test

Signed-off-by: Georgi Zlatarev <georgi.zlatarev@manta.network>

* Remove other workflows for testing. Also trigger on this branch

Signed-off-by: Georgi Zlatarev <georgi.zlatarev@manta.network>

* Update

Signed-off-by: Georgi Zlatarev <georgi.zlatarev@manta.network>

* Update

Signed-off-by: Georgi Zlatarev <georgi.zlatarev@manta.network>

* Update

Signed-off-by: Georgi Zlatarev <georgi.zlatarev@manta.network>

* Add LFS storage.json

Signed-off-by: Georgi Zlatarev <georgi.zlatarev@manta.network>

* Check

Signed-off-by: Georgi Zlatarev <georgi.zlatarev@manta.network>

* Trigger

Signed-off-by: Georgi Zlatarev <georgi.zlatarev@manta.network>

* Use append_storage.js

Signed-off-by: Georgi Zlatarev <georgi.zlatarev@manta.network>

* Hard code prefixes

Signed-off-by: Georgi Zlatarev <georgi.zlatarev@manta.network>

* Hard code prefixes

Signed-off-by: Georgi Zlatarev <georgi.zlatarev@manta.network>

* Bump sleep in dolphin test

Signed-off-by: Georgi Zlatarev <georgi.zlatarev@manta.network>

* add .gitattributes

Signed-off-by: Georgi Zlatarev <georgi.zlatarev@manta.network>

* Add lfs:true to ci

Signed-off-by: Georgi Zlatarev <georgi.zlatarev@manta.network>

* Install git-lfs in github action

Signed-off-by: Georgi Zlatarev <georgi.zlatarev@manta.network>

* Use sudo for apt-get

Signed-off-by: Georgi Zlatarev <georgi.zlatarev@manta.network>

* Check

Signed-off-by: Georgi Zlatarev <georgi.zlatarev@manta.network>

* Check

Signed-off-by: Georgi Zlatarev <georgi.zlatarev@manta.network>

* Check

Signed-off-by: Georgi Zlatarev <georgi.zlatarev@manta.network>

* Check

Signed-off-by: Georgi Zlatarev <georgi.zlatarev@manta.network>

* Check

Signed-off-by: Georgi Zlatarev <georgi.zlatarev@manta.network>

* Check

Signed-off-by: Georgi Zlatarev <georgi.zlatarev@manta.network>

* Check

Signed-off-by: Georgi Zlatarev <georgi.zlatarev@manta.network>

* Check

Signed-off-by: Georgi Zlatarev <georgi.zlatarev@manta.network>

* Check

Signed-off-by: Georgi Zlatarev <georgi.zlatarev@manta.network>

* Check

Signed-off-by: Georgi Zlatarev <georgi.zlatarev@manta.network>

* Check

Signed-off-by: Georgi Zlatarev <georgi.zlatarev@manta.network>

* Check

Signed-off-by: Georgi Zlatarev <georgi.zlatarev@manta.network>

* Check

Signed-off-by: Georgi Zlatarev <georgi.zlatarev@manta.network>

* Use reference hardware

Signed-off-by: Georgi Zlatarev <georgi.zlatarev@manta.network>

* More sleep before start of test

Signed-off-by: Georgi Zlatarev <georgi.zlatarev@manta.network>

* Fix

Signed-off-by: Georgi Zlatarev <georgi.zlatarev@manta.network>

* Clean up

Signed-off-by: Georgi Zlatarev <georgi.zlatarev@manta.network>

* Clean up

Signed-off-by: Georgi Zlatarev <georgi.zlatarev@manta.network>

* Run stress test last

Signed-off-by: Georgi Zlatarev <georgi.zlatarev@manta.network>

* Run stress test separately

Signed-off-by: Georgi Zlatarev <georgi.zlatarev@manta.network>

* Append storage for second run

Signed-off-by: Georgi Zlatarev <georgi.zlatarev@manta.network>

* In new workflow

Signed-off-by: Georgi Zlatarev <georgi.zlatarev@manta.network>

* In new workflow

Signed-off-by: Georgi Zlatarev <georgi.zlatarev@manta.network>

* Same workflow

Signed-off-by: Georgi Zlatarev <georgi.zlatarev@manta.network>

* Correct TransferPost sizes

Signed-off-by: Georgi Zlatarev <georgi.zlatarev@manta.network>

* Bump expected_tps to 2

Signed-off-by: Georgi Zlatarev <georgi.zlatarev@manta.network>

* Remove redundant workflow

Signed-off-by: Georgi Zlatarev <georgi.zlatarev@manta.network>

* Update asserts and format the ts file

Signed-off-by: Georgi Zlatarev <georgi.zlatarev@manta.network>

* Redce sleep between iterations

Signed-off-by: Georgi Zlatarev <georgi.zlatarev@manta.network>

* Update

Signed-off-by: Georgi Zlatarev <georgi.zlatarev@manta.network>

* Incrase delay again

Signed-off-by: Georgi Zlatarev <georgi.zlatarev@manta.network>

* Update gitattributes

Signed-off-by: Georgi Zlatarev <georgi.zlatarev@manta.network>

* Fix path to precomputed coins

Signed-off-by: Georgi Zlatarev <georgi.zlatarev@manta.network>

* CHange expected tps

Signed-off-by: Georgi Zlatarev <georgi.zlatarev@manta.network>

* Add commetn and upload alice artifact

Signed-off-by: Georgi Zlatarev <georgi.zlatarev@manta.network>

* Bump CI job timeout to 180

Signed-off-by: Georgi Zlatarev <georgi.zlatarev@manta.network>

* Fix dolphin

Signed-off-by: Georgi Zlatarev <georgi.zlatarev@manta.network>

* Add comments and consts

Signed-off-by: Georgi Zlatarev <georgi.zlatarev@manta.network>

* Get rid of erroneous echos

Signed-off-by: Georgi Zlatarev <georgi.zlatarev@manta.network>

* Get rid of unsub

Signed-off-by: Georgi Zlatarev <georgi.zlatarev@manta.network>

* Simplify

Signed-off-by: Georgi Zlatarev <georgi.zlatarev@manta.network>

* Use s3 instead of git lfs:

Signed-off-by: Georgi Zlatarev <georgi.zlatarev@manta.network>

Signed-off-by: Georgi Zlatarev <georgi.zlatarev@manta.network>
Co-authored-by: Adam Reif <garandor@manta.network>
Signed-off-by: Charles Ferrell <charlie@manta.network>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
A-calamari Area: Issues and PRs related to the Calamari Runtime A-dolphin Area: Issues and PRs related to the Dolphin Runtime C-enhancement Category: An issue proposing an enhancement or a PR with one L-added Log: Issues and PRs related to addition
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Stress test for MantaPay pallet
3 participants