-
Notifications
You must be signed in to change notification settings - Fork 2
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
continuous testing - bot like traffic to stress test reliability and availability #59
Comments
Follow up look into:
|
@InoMurko @achiurizo
is it ok? |
Before you do 2 and 3, we need to look at what |
Versioning the tests repo or adding feature tags to tests |
can you elaborate on that? 🤔 |
|
regarding
|
|
Ayrat works on /v2/block.get and adds tests into cabbage branch ayrat/v2_block_get If you merge your PR into childchain2, how does Arhur in his PR reference your newly added tests when it gets into childchain2 master? So... what I'm trying to avoid is having to maintain As long as every stakeholder (@achiurizo @T-Dnzt ) are aware of the associated cost and risk. |
maintain a syncing your |
I think for the current interim, this will probably be the case until we decommission
It really probably has to come to some form of branch management like this. Ignoring the API example above, contract changes alone(like the v2 contracts being worked on) will need some tracking branch on the other repositories to test the changes(so that would be now I'll also mention that we are also hiring for a QA role 🔜 . Hopefully this |
@ayrat555 could we get a review of the current |
We have a watcher_info tests in perf: https://github.com/omgnetwork/elixir-omg/blob/master/priv/perf/apps/load_test/test/load_tests/runner/watcher_info_test.exs The main logic is in this file: https://github.com/omgnetwork/elixir-omg/blob/master/priv/perf/apps/load_test/lib/scenario/account_transactions.ex This one simulates a user using watcher info APIs to create transaction and send transaction (and randomly hit other watcher info APIs). The script was tested to be able to run for a day long time ago (note, at the time hakney retry was turned on). Might be a good one to use it for simulate production load testing (plus this is more similar to how our user would hit our service). |
omg-js integration tests (org-mode format)** addTokenTest *** add token that the chain had *** add token that the chain did not have ** amountTypesTest *** validates types that approveToken acceps ** challengeExitTest *** challenge a dishonest exit ** challengeInFlightExitInputSpentTest *** challenge an in-flight exit as non canonical and challenge an invalid input piggyback ** challengeInFlightExitOutputSpentTest *** challenge output from a canonical invalid output piggyback ** createSubmitTypedTransactionTest *** create and submit a single currency transaction using submitTyped ** createTransactionTest *** create and submit a single currency transaction *** create a single currency transaction *** create and submit a mixed currency transaction *** create an incomplete transaction *** split a utxo into 4 ** deleteNonPiggybackedInFlightExitTest *** delete an ife if not piggybacked after the first phase and return ife bond ** depositTest *** deposit ETH to the Plasma contract *** deposit ERC20 tokens to the Plasma contract ** exitWithoutWatcherTest *** creates required exit data ** getExitQueueTest *** processes exits and returnds empty queue ** inFlightExitChallengeResponseTest *** invalid IFE challenge response ** inFlightExitChallengeTest *** challenge an in-flight exit by providing a competitor *** challenge an in-flight exit as non canonical ** inFlightExitTest *** exit a ChildChain transaction *** exit a ChildChain transaction that is not included ** mergeUtxoTest *** merge utxos to a single utxo ** metadataTest *** checks if childchain adds different types of metadata ** simpleStartStandardExitTest *** start standard exit for a ChildChain transaction ** standardExitTest *** exit a deposit *** exit a ChildChain transaction *** exit ERC20 token ** transferTest *** transfer ETH and ERC20 tokens on the childchain *** transfer funds on the childchain with eth currency *** transfer ERC20 tokens on the childchain *** transfer ETH on the childchain *** send a transaction with 4 inputs and 4 outputs
Do we really need to delete anything from omg-js integration tests? I think the main purpose of them is to test the correctness of omg-js, not elixir-omg.
I think it's possible to use
So my suggestion is to create separate cases which will cover points in the issue |
No, I don't think we need to delete anything here - apart from testing the omg-js code itself, they also provide good example code for JS developers.
There's already deposit and transact code in there, and PR omgnetwork/elixir-omg#1732 (review needed 😄 ) introduces a test for starting standard exits. Exit processing, IFEs, challenges, etc still need to be added.
Yeah, the perf tests are just that - to measure perf/load/stress. It's assumed that correctness is already tested elsewhere. One thing I find cumbersome about Chaperon tests is how they're configured - if you want to change e.g. how many iterations, you have to edit the config file, recompile and run. It would be great to be able to pass params on the command line. There are lots of configuration params though, so ideally we could set defaults in a file and override them on the command line. I don't know whats the best way of doing that in Elixir though... |
So if we agree that omg-js and perf/ are two separate projects, not supposed to cover the same thing... I propose we do the following: Lets make perf usable all around by:
Of course, the question is, why would we built a frontend if we have omg-js - I think we will see that the amount of pressure elixir node can create (in terms of IO and concurrency) would far exceed any other tooling. As this is considered developer tooling and testing, we will specify our requirements on the fly. |
can you elaborate a bit more on why the frontend is needed? |
To elaborate more on what’s needed for release pipeline, I think it would either being run directly on the release pipeline’s agent to respond back test succeed or not or perf need to call a web hook back the release pipeline to continue or abort the automated promotion |
In terms of usability, APIs are great for machine 2 machine collaboration. APIs are less suitable for observability and introspection where humans are needed. And as we turn these tests into stateful services that might run tests for more then 24hours this kind of tooling will become useful. Imagine Datadog being just APIs... |
I think I kind of get but still not following well. I get UI is better for human, curious what do you expect it to show/use from UI. From description comment above it seems to provide ability to set config via a form UI? For other part, if we need any inspection of the perf test running, should we just get ability of observation on datadog. Or do you want to build fancy UI showing how loads are hitting 😱😱 |
I was playing with LiveView over the weekend. It seems it's pretty neat because you can write interactive web pages almost without writing any js code. But I have the same question as @boolafish. what kind of real-time information will be shown there (errors, rps, current test, progress, etc) ? |
Will provide details by EOD! |
Because these tests are going to be statefull, we would display:
|
I think this is great. Thanks for looking into this! I approve. 👍 |
This is just a suggestion, while the form and UI does sound nice to work with, Datadog also provides synethtic testing, this allows us to create custom API/browser tests within Datadog for a service. A couple of nice things;
|
what can we do now as a proof of concept:
|
FYI, share some knowledges:
|
@boolafish I have a question the third point. Do you suggest adding datadog metrics to the current tests? will it be useful? |
ah...was inspired by this: https://k6.io/blog/datadog-alerts-fail-load-test But something not going so far as 👆above would be like getting API latency metric (eg P90/P99) and assert on that. That is sth probably not that great if we use metric from I guess above might be some basic metrics we would can assert against (latency, TPS, error rate) for performance tests. Then we can have automation tests saying we can run TPS 100 for 30 minutes with P99 latency to not exceed Xms and error rate < Y%. Currently.....lol it really depends on the release manager to decide what to see after running the load test. (we run load test on sandbox when there is release. But there is no good interpretation atm) For sth more fancy like memory leak, CPU rate we might apply the idea from the article. |
Link to some pipeline requirement and random thought: #141 (comment) |
An update on the release tooling. We have decide to go forward with spinnaker which have out of box support to trigger/monitor kubernate job. As a result, the way to hook perf and release pipeline would be trigger perf with k job. |
Done part of performance and spinnaker. thanks @ayrat555 @boolafish |
We would like to stress test our services (childchain + vault + postgres specifically) and how they work under continuous trafic.
The testing kit should do the following:
Backoffice testing kit will do accounting like operations:
Update. Task list based on #59 (comment) and #59 (comment) :
Create a module in the perf project that will create deposits and test balances on the child chain and the root chain. TPS, addresses, period, geth node url have to be configurable
Create a simple UI which will show test parameters (status, configuration, balances etc). Also, UI can start and configure tests.
Integrate datadog to collect stats (https://app.datadoghq.com/dashboard/pz5-694-h7e/app-metrics-vm-stats?from_ts=1598185098067&live=true&to_ts=1600777098067&tpl_var_dashboard_app_env=development) and metrics
Prepare docker container/release so the app can be deployed
Create API so circle ci builds can trigger the perf app
The text was updated successfully, but these errors were encountered: