Testing Improvement Plan

[Stebalien]

• CLI/API
  • Coverage
    • Merge Implementation status dashboard #6075 (maybe with a few tweaks) and then fill out the flags that aren't covered.
  • Language Interop
    • Fix the current interop tests (IPNS Pubsub tests failing on master for linux interop#71).
    • Run sharness tests gainst both go-ipfs and js-ipfs (where possible).
  • Backwards Compatibility
    • Run sharness tests with previous go-ipfs binary as a client. Need to mark the required client version on sharness tests.
    • Interop tests: These may be sufficient.
• Performance
  • Benchmarks in CI.
    • End-to-end bitswap (ci: benchmark #6446)
    • Garbage Collection
    • File Add
    • File Cat/Get
    • Directory Listing
    • Pin ls/add/etc.
    • Dial Latency (testing the dialer, transports, etc.).
    • Stream throughput (using ipfs p2p).
  • Performance history (Testing Improvement Plan #6483 (comment))
• Reliability/Stability
  • Nightly Mirror Gateway: All requests to a specific gateway get mirrored to this gateway with nginx.
    • Track and compare errors/latencies versus the gateway serving the response.
      • Auto-file Issues.
    • Auto deploy a nightly build.
    • Auto-file issues on crash.
    • Automatically pull goroutine dumps, cpu profiles, and memory profiles.
    • Automatically analyze with stackparse and auto-file issues when a specific goroutine count increases by 100x (usually indicates an issues). Note: this'll have a bunch of false positives but it's a good signal.
    • Same with the heap: File an issue if the "top memory user in a steady state" changes.
    • Same with the CPU profile.
  • Nightly Bootstrapper
    • Auto deploy a nightly build.
    • Auto-file issues on crash.
    • Automatically pull goroutine dumps, cpu profiles, and memory profiles.
    • Automatically analyze with stackparse and auto-file issues when a specific goroutine count increases by 100x (usually indicates an issues). Note: this'll have a bunch of false positives but it's a good signal.
    • Same with the heap: File an issue if the "top memory user in a steady state" changes.
    • Same with the CPU profile.
  • Network Simulation: tests run on a simulated network so we can predict how are changes will affect the network. We should be able to run each test with X% of the network running the current release and the old release.
    • DHT Tests - test node run as a client and server.
      • FindPeer
      • Provide/FindProviders
      • IPNS
    • End-to-end Bitswap (including finding providers, multiple providers, etc.).
  • Network Canary - Continuous background tests against the main network.
    • DHT Tests - test node run as a client and server.
      • FindPeer
      • Provide/FindProviders
      • IPNS
    • End-to-end Bitswap (including finding providers, multiple providers, etc.).
    • Gateway Test - Add a file to the test node and fetch it on the ipfs.io gateway (measuring latency).
  • Release Candidates
    • Opt-out telemetry is built into RC binaries for collection of network operation and performance (required to be able to confirm any hypothesises formed, from testlab tests, in the production network)

In both network tests, we should track and compare metrics on:

• Number of dials.
• Total network bandwidth.
• Latencies/times for each step.
• CPU Usage
• Memory Usage
• Goroutine Counts.

---

The Plan: https://docs.google.com/spreadsheets/d/1xyqyGUF-oe3x9ln88YonVeOMWWdknik74lVgL_3dBY8

[Stebalien]

cc @ipfs/wg-infrastructure

[lanzafame]

As part of the end-to-end benchmarking, can we look at some tooling around the storage of the benchmarks, over just comparing on a PR by PR basis, i.e. https://github.com/golang/perf or https://github.com/influxdata/grade

[Stebalien]

@lanzafame I've added a item to the todo list.

[lanzafame]

@Stebalien I have updated the issue with the point about RC telemetry.

[yiannisbot]

On the issue of Network Simulation for libp2p TestLab, I've come across this: https://chepeftw.github.io/NS3DockerEmulator/ - which is a docker emulator that can then be imported in NS-3 (https://nsnam.org). The production IPFS/libp2p/DHT code is packaged in a docker container, which represents a node in the network. Through NS-3 we then do the network setup and run the simulation (it's more emulation than simulation TBH). This can produce realistic results using production code. It might not be useful for all purposes, but surely it can be for many cases.

[jbenet]

Links

• Leveling up testing infrastructure notes#202
• Real world IPFS scenarios we want to test  notes#211
• Document a few use cases for submitting tests to the Orchestrator ipfs-inactive/pm-test-lab#10 (comment)

[Jorropo]

I don't think Dial Latency (testing the dialer, transports, etc.). is an IPFS part, I think this test is libp2p releated because its impossible to fix any problem about that in IPFS without changing the libp2p codebase, also that test and his result impact all libp2p users, such as filecoin, IPFS, ethereum 2.0...

[Stebalien]

@Jorropo these are tests needed by go-ipfs, not necessarily tests of go-ipfs. That is, we want to test these libp2p features before we cut a release.