interpreting results of performance tests

[ayrat555]

related issues/prs:

#59
#141
#145
omgnetwork/elixir-omg#1745

During code review discussions, the question about the interpretation of performance tests raised several times.

Currently, by default successful outcome is defined by error_rate mean = 0. Optionally you can check error_rate percentile (10%-90%) = 0.

[boolafish]

I think for several tests it will need:

1. server side API latency (each API should have one metric). P9x version instead of max.
2. client side error rate
3. server side error rate (optional?)

Personally I think this is test specific. So it might be better to allow each test to inject each logic that it want to check and developers can add new checks for each test. (eg. the API for deposit test would be different from a watcher one)

[boolafish]

https://omgnetworkhq.slack.com/archives/CMX790Q5V/p1603252908142800?thread_ts=1603249243.137000&channel=CMX790Q5V&message_ts=1603252908.142800

Add feedback from infra team

[ayrat555]

Ok. I'll try adding metrics like:
https://app.datadoghq.com/dashboard/rvw-u8w-hh3/traefik-ingress-metrics?from_ts=1603462350880&live=true&to_ts=1603465950880

https://www.weave.works/blog/the-red-method-key-metrics-for-microservices-architecture/

[InoMurko]

I think this is all done