forked from libp2p/test-plans
-
Notifications
You must be signed in to change notification settings - Fork 0
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Organization: add DESIGN.md and ROADMAP.md (libp2p#44)
Co-authored-by: Steve Loeppky <stvn@loeppky.com> Co-authored-by: Prithvi Shahi <shahi.prithvi@gmail.com>
- Loading branch information
Showing
3 changed files
with
347 additions
and
0 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,158 @@ | ||
| # libp2p testing story | ||
|
|
||
| ``` | ||
| Date: 2022-10-18 | ||
| Status: In Progress | ||
| ``` | ||
|
|
||
| --- | ||
|
|
||
| ## Overview | ||
|
|
||
| This document describes our process for testing interoperability & backward compatibility across libp2p implementations. | ||
|
|
||
| **Why:** | ||
|
|
||
| - Interoperability is a shared concern. | ||
| - There is no single blessed libp2p reference implementation that we use for conformance testing. | ||
| - No single maintainer (go|rust|js-libp2p or IPDX) will succeed without everyone's involvement. | ||
| - We want to share a Testing Story with the world that shows we care about quality & interop. | ||
| - We want to encourage other implementations to join the testing party. | ||
|
|
||
| **Historical Context:** | ||
|
|
||
| - We completed a “PING” interop test with Testground. It is running in the go-libp2p and rust-libp2p CI pipeline. | ||
| - It means we “proved” that we can write and run interop tests between versions AND implementations. | ||
|
|
||
| # Libp2p Testing Matrix | ||
|
|
||
| *What do we want to test next?* | ||
|
|
||
| | | go-libp2p | rust-libp2p | js-libp2p (node) | js-libp2p (browser) | jvm-libp2p | nim-libp2p | | ||
| | --- | --- | --- | --- | --- | --- | --- | | ||
| | Simple PING [#35][issue-35] | ✅ | ✅ | 🍎 | 🔥 | | | | ||
| | Circuit Relay | | | | | | | | ||
| | WebTransport Transport | 🔥 | | 🔥 (depends on https://github.com/libp2p/js-libp2p-webtransport/issues/1) | 🔥 (depends on https://github.com/libp2p/js-libp2p-webtransport/issues/1) | | | | ||
| | WebRTC Transport | 🔥 (depends on working implementation) | 🔥 (depends on working implementation) | 🔥 (depends on working implementation) | 🔥 (depends on working implementation) | | | | ||
| | NAT Traversal | | | | | | | | ||
| | Hole Punching (STUN) | | | | | | | | ||
| | Identify Protocol | | | | | | | | ||
| | AutoNAT | | | | | | | | ||
| | DHT | | | | | | | | ||
| | QUIC | | | | | | | | ||
| | Benchmarking? | | | | | | | | ||
|
|
||
| **Dependencies** | ||
|
|
||
| - Anything `js-libp2p` related requires the `ping` test to start | ||
| - Benchmarking must relate to [Remote Runners][remote-runners] | ||
| - https://github.com/testground/testground/pull/1425 | ||
| - https://github.com/testground/testground/issues/1392 | ||
|
|
||
| **Questions** | ||
|
|
||
| - When do we revisit this table to discuss priorities and add new tests? | ||
|
|
||
| **Legend** | ||
|
|
||
| - ✅ Done | ||
| - 🚚 In Progress | ||
| - 🔥 Highest Priority | ||
| - 🍎 Low-hanging fruit | ||
| - 🧊 Lowest priority | ||
|
|
||
| # How does libp2p test interoperability? | ||
|
|
||
| --- | ||
|
|
||
| --- | ||
|
|
||
|
|
||
| ## Background | ||
| The approach outlined below is pretty much what happen with the go|rust-libp2p ping tests in 2022Q3. | ||
|
|
||
| libp2p implementations aren't forced to adopt this approach, but it is the approach that has been taken by some of the longer-lived implementations (go, JS, and rust). | ||
|
|
||
| I (@laurent) haven’t had time to look at [libp2p/interop](https://github.com/libp2p/interop/actions/runs/3021456724) yet. Some information may be missing. | ||
|
|
||
| ## 202210 Proposal | ||
| <aside> | ||
| 1️⃣ Before working on a new feature, the libp2p maintainers come together and agree on a description of the new test plan.* | ||
|
|
||
| </aside> | ||
|
|
||
| **Example:** | ||
|
|
||
| - [IPFS Test Story in libp2p/interop](https://github.com/libp2p/interop/blob/master/pdd/PDD-THE-IPFS-BUNDLE.md) | ||
|
|
||
| **Question:** | ||
|
|
||
| - What should be the format for this description? | ||
| - Can we live with a rough “here is a general idea of what the test should do”, and let the first implementor figure out the details? | ||
| - Do we need to make these decisions now? (09-09-2022) | ||
|
|
||
| <aside> | ||
| 2️⃣ *The maintainers agree on which implementation will provide the reference test implementation (go, rust, js, or other). This implementation is written for Testground and merged in the `libp2p/test-plan` repository.* | ||
|
|
||
| </aside> | ||
|
|
||
| **Example:** | ||
|
|
||
| - https://github.com/libp2p/test-plans/pull/9 “add an instructional libp2p ping test plan” | ||
|
|
||
| **Why:** | ||
|
|
||
| - During implementation, some decisions might be taken on how coordination works, details of the tests, etc. It will be easier to clear the path from one implementation. | ||
|
|
||
| <aside> | ||
| 3️⃣ Once this implementation is merged, the reference implementation enables the test in their CI. It will be a “simple” test that runs the current branch against the last N implementations. | ||
|
|
||
| </aside> | ||
|
|
||
| **Example:** | ||
|
|
||
| - https://github.com/libp2p/go-libp2p/pull/1625 “ci: run testground:ping plan on pull requests” in go-libp2p | ||
|
|
||
| <aside> | ||
| 4️⃣ Other implementation will provide their version of the test. And enable a similar test in CI | ||
|
|
||
| </aside> | ||
|
|
||
| **Example:** | ||
|
|
||
| - https://github.com/libp2p/test-plans/pull/26 “ping/rust: introduce rust cross-version test” | ||
| - https://github.com/libp2p/rust-libp2p/pull/2835 “.github: introduce interop tests” in rust-libp2p | ||
|
|
||
| <aside> | ||
| 5️⃣ Once multiple implementations have been provided and are running the test in CI, each project will add a “big” test workflow in their Release Process. | ||
| This “big test” runs the test between every known implementation & version. | ||
| It might be enabled in a nightly job too. | ||
|
|
||
| </aside> | ||
|
|
||
| **Example:** | ||
|
|
||
| - TODO: add the `full` interop test to `go-libp2p` + update their release documentation. | ||
|
|
||
| ## Open Questions | ||
|
|
||
| - When do we revisit this scenario to improve and gather feedback? | ||
| - How do we evaluate progress & success? | ||
| - When we’re able to use these tests for benchmarking probably. | ||
| - What’s the plan for the day when everything starts to break? | ||
| - What’s the plan for the time when we start to crumble under test complexity? | ||
| - Maintenance | ||
| - Tests will need updates on new releases, etc. | ||
| - What are the dependencies between tests? | ||
| - ex: Does it make sense to test HOLE PUNCHING if you don’t test AUTONAT first? | ||
|
|
||
| ## Refs | ||
|
|
||
| - [https://docs.libp2p.io/concepts/protocols/](https://docs.libp2p.io/concepts/protocols/) | ||
| - libp2p interop in [Interop Repository](https://github.com/libp2p/interop) | ||
| - [libp2p interop issue](https://github.com/libp2p/interop/issues/70) | ||
| - [libp2p/interop test plans](https://github.com/libp2p/interop/blob/master/pdd/PDD-THE-IPFS-BUNDLE.md) | ||
|
|
||
|
|
||
| [issue-35]: https://github.com/libp2p/test-plans/issues/35 | ||
| [remote-runners]: https://pl-strflt.notion.site/Remote-Runners-c4ad4886c4294fb6a6f8afd9c0c5b73c |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,182 @@ | ||
| # test-plans roadmap Q4’22/Q1’23 <!-- omit in toc --> | ||
|
|
||
| ``` | ||
| Date: 2022-10-20 | ||
| Status: Accepted | ||
| Notes: Internal test-plans stakeholders have aligned on this roadmap. Please add any feedback or questions in: | ||
| https://github.com/libp2p/test-plans/issues/58 | ||
| ``` | ||
|
|
||
| ## Table of Contents <!-- omit in toc --> | ||
|
|
||
| - [About the Roadmap](#about-the-roadmap) | ||
| - [Vision](#vision) | ||
| - [Sections](#sections) | ||
| - [🛣️ Milestones](#️-milestones) | ||
| - [2022](#2022) | ||
| - [Early Q4 (October)](#early-q4-october) | ||
| - [Mid Q4 (November)](#mid-q4-november) | ||
| - [End of Q4 (December)](#end-of-q4-december) | ||
| - [2023](#2023) | ||
| - [Early Q1 (January)](#early-q1-january) | ||
| - [End of Q1 (March)](#end-of-q1-march) | ||
| - [Up Next](#up-next) | ||
| - [📖 Appendix](#-appendix) | ||
| - [A. Multi-dimensional Testing/Interop Visibility](#a-multi-dimensional-testinginterop-visibility) | ||
| - [1. User configured interop tests & dashboard](#1-user-configured-interop-tests--dashboard) | ||
| - [2. Interop test plans for all existing/developing libp2p transports](#2-interop-test-plans-for-all-existingdeveloping-libp2p-transports) | ||
| - [3. Canonical interop tests & dashboard](#3-canonical-interop-tests--dashboard) | ||
| - [B. Hardening test infrastructure](#b-hardening-test-infrastructure) | ||
| - [1. Track test suite stability](#1-track-test-suite-stability) | ||
| - [2. Design process for adding new tests](#2-design-process-for-adding-new-tests) | ||
| - [3. Be the home for all interop tests](#3-be-the-home-for-all-interop-tests) | ||
| - [C. Future-proof Benchmarking](#c-future-proof-benchmarking) | ||
| - [1. Benchmarking using nix-builders](#1-benchmarking-using-nix-builders) | ||
| - [2. Benchmarking using remote runners](#2-benchmarking-using-remote-runners) | ||
| - [D. Expansive protocol test coverage](#d-expansive-protocol-test-coverage) | ||
| - [1. DHT server mode scale test](#1-dht-server-mode-scale-test) | ||
| - [2. AutoNat](#2-autonat) | ||
| - [3. Hole Punching](#3-hole-punching) | ||
| - [4. AutoRelay](#4-autorelay) | ||
| - [5. Custom topologies](#5-custom-topologies) | ||
| - [6. MTU Fixes](#6-mtu-fixes) | ||
|
|
||
| ## About the Roadmap | ||
|
|
||
| ### Vision | ||
| We, the maintainers, are committed to upholding libp2p's shared core tenets and ensuring libp2p implementations are: [**Secure, Stable, Specified, and Performant.**](https://github.com/libp2p/specs/blob/master/ROADMAP.md#core-tenets) | ||
|
|
||
| This roadmap is complementary to those of [go-libp2p](https://github.com/libp2p/go-libp2p/blob/master/ROADMAP.md), [rust-libp2p](https://github.com/libp2p/rust-libp2p/blob/master/ROADMAP.md), and [js-libp2p](https://github.com/libp2p/js-libp2p/blob/master/ROADMAP.md). | ||
|
|
||
| It aims to encompass the **stability** and **performance** tenets of the libp2p team. | ||
| Projects outlined here are shared priorities of the different implementations. | ||
|
|
||
| ### Sections | ||
| This document consists of two sections: [Milestones](#️-milestones) and the [Appendix](#-appendix) | ||
|
|
||
| [Milestones](#️-milestones) is our best educated guess (not a hard commitment) around when we plan to ship the key features. | ||
| Where possible projects are broken down into discrete sub-projects e.g. project "A" may contain two sub-projects: A.1 and A.2 | ||
|
|
||
| A project is signified as "complete" once all of it's sub-projects are shipped. | ||
|
|
||
| The [Appendix](#-appendix) section describes a project's high-level motivation, goals, and lists sub-projects. | ||
|
|
||
| Each Appendix header is linked to a GitHub Epic. Latest information on progress can be found in the Epics and child issues. | ||
|
|
||
| ## 🛣️ Milestones | ||
|
|
||
| ### 2022 | ||
|
|
||
| #### Early Q4 (October) | ||
| - [A.1 User Configured Interop Tests & Dashboard](#1-user-configured-interop-tests--dashboard) | ||
|
|
||
| #### Mid Q4 (November) | ||
| - [A.2 Interop tests for all existing/developing libp2p transports](#2-interop-test-plans-for-all-existingdeveloping-libp2p-transports) | ||
| - [C.1 Benchmarking using nix-builders](#1-benchmarking-using-nix-builders) | ||
|
|
||
| #### End of Q4 (December) | ||
| - [A.3 Canonical Interop Tests & Dashboard](#3-canonical-interop-tests--dashboard) | ||
|
|
||
| ### 2023 | ||
|
|
||
| #### Early Q1 (January) | ||
|
|
||
| - [D.1 DHT Server Mode Scale Test](#1-dht-server-mode-scale-test) | ||
|
|
||
| #### End of Q1 (March) | ||
| - [C.2 Benchmarking using remote runners](#2-benchmarking-using-remote-runners) | ||
|
|
||
| ### Up Next | ||
|
|
||
| ## 📖 Appendix | ||
|
|
||
| **Projects are listed in descending priority.** | ||
|
|
||
| ### [A. Multi-dimensional Testing/Interop Visibility](https://github.com/libp2p/test-plans/issues/53) | ||
| **Why:** libp2p supports a variety of transport protocols, muxers, & security protocols across implementations in different languages. Until we actually test them together, we can't guarantee implementation interoperability. We need to ensure and demonstrate that: interoperable features work with each other as expected and we don't introduce degradations that break interoperability in new releases. | ||
|
|
||
| **Goal:** libp2p implementers run tests across permutations of libp2p implementations, versions, and supported transports, muxers, and security protocols. Implementers and users can reference a single website with a dashboard to view the Pass/Fail/Implemented/Not Implemented results of multi-dimensional tests. | ||
|
|
||
| This effort depends on [Testground Milestone 1](https://github.com/testground/testground/blob/master/ROADMAP.md#1-bootstrap-libp2ps-interoperability-testing-story) | ||
|
|
||
| **How:** | ||
| #### [1. User configured interop tests & dashboard](https://github.com/libp2p/test-plans/issues/55) | ||
| Enable test-plans maintainers to define a configuration (of libp2p impls, versions, transports, expected RTT), run Testground tests per configuration, and retrieve resulting data in a standard format. | ||
| The test case results can be formatted as a "dashboard" (simple Markdown table of Pass/Fail results.) | ||
|
|
||
| #### [2. Interop test plans for all existing/developing libp2p transports](https://github.com/libp2p/test-plans/issues/61) | ||
| Using tooling from A.1, all features of go-libp2p, rust-libp2p, and js-libp2p that should be interoperable are tested (i.e. transports (TCP, QUIC, WebRTC, WebTransport), multiplexers (mplex, yamux), secure channels (TLS, Noise), etc.) across versions. | ||
|
|
||
| Features currently in development across implementations (like WebRTC in go-libp2p and rust-libp2p, or QUIC & TLS in rust-libp2p) are not merged without at least manually running these test suites. | ||
|
|
||
| Test suites are run in `libp2p/test-plans` CI and before releasing a version of go-libp2p, rust-libp2p, and js-libp2p (GitHub workflow added so that these suites run against the `master` branch on a nightly basis (updating the status check.)) | ||
|
|
||
| **Note:** | ||
| - Dependency on [C.1](#1-benchmarking-using-nix-builders) to run node.js-libp2p in Testground. | ||
| - Dependency on [testground/Investigate browser test support](https://github.com/testground/testground/issues/1386) to run interoperability test for js-libp2p WebRTC against Go and Rust. | ||
|
|
||
| #### [3. Canonical interop tests & dashboard](https://github.com/libp2p/test-plans/issues/62) | ||
| A comprehensive and canonical dashboard is generated and hosted in a publicly discoverable site that displays latest test suite results (Pass/Fail/Implemented/Not Implemented/Unimplementable) from a nightly CI run. | ||
| All permutations of libp2p implementations, versions, and supported transports, muxers, & security protocols should be visible. | ||
|
|
||
| An enhancement of A.1 to make it easier for users and implementers to see the full state of libp2p interoperability. | ||
|
|
||
| ### B. Hardening test infrastructure | ||
|
|
||
| #### 1. Track test suite stability | ||
| <!-- TODO: Assign a quarter --> | ||
| <!-- TODO: Create issue --> | ||
| `libp2p/test-plans` maintainers have a straightforward way to track the test suite stability and performance. | ||
| - We can track the status of every test combination stability from the interop project itself | ||
| - We can easily measure the consequence (improvements) of a pull request to the libp2p/interop repository | ||
| - We are alerted when an interop test starts failing on one of our client repositories, and we can dispatch the alert to repo maintainers. | ||
|
|
||
| #### 2. Design process for adding new tests | ||
| <!-- TODO: Assign a quarter --> | ||
| <!-- TODO: Create issue --> | ||
| We have an explicit, working, Design Process for adding new tests | ||
| - The design is documented in `./DESIGN.md`. | ||
| - The design is followed by the team when we add new features. | ||
| - There is a clear path when it comes to testing new features. This might mean testing multiple `masters` against each other. | ||
|
|
||
| #### 3. Be the home for all interop tests | ||
| <!-- TODO: Assign a quarter --> | ||
| <!-- TODO: Create issue --> | ||
| We have ported the tests from `libp2p/interop` | ||
| - This repository implement tests `connect`, `dht`, `pubsub` ([ref](https://github.com/libp2p/interop/blob/ce0aa3749c9c53cf5ad53009b273847b94106d40/src/index.ts#L32-L35)) | ||
| - At of writing (2022-09-27), it is disabled in `go-libp2p` ([ref](https://github.com/libp2p/go-libp2p/actions/workflows/interop.yml)), and it is used in `js-libp2p` ([ref](https://github.com/libp2p/js-libp2p/actions/runs/3111413168/jobs/5050929689)). | ||
|
|
||
|
|
||
| ### [C. Future-proof Benchmarking](https://github.com/libp2p/test-plans/issues/63) | ||
| **Why**: For libp2p to be competitive, it needs to delivers comparable performance to widely used protocols on the internet, namely HTTP/2 and HTTP/3. | ||
|
|
||
| **Goal**: We have a test suite that runs libp2p transfers between nodes located at different locations all over the world, proving that libp2p is able to achieve performance on par with HTTP. The test suite is run on a continuous basis and results are published to a public performance dashboard. | ||
|
|
||
| #### [1. Benchmarking using nix-builders](https://github.com/testground/testground/pull/1425) | ||
| - [Benchmark go-libp2p, rust-libp2p, and go-libp2p](https://github.com/libp2p/test-plans/issues/27) | ||
| - [Specifically add js-libp2p-transfer-performance as a test-plan and CI job to benchmark transfer times across releases](https://github.com/libp2p/test-plans/issues/65) to catch issues like [#1342](https://github.com/libp2p/js-libp2p/issues/1342) | ||
| - (Dependency: remote machines need Nix installed) | ||
| #### [2. Benchmarking using remote runners](https://github.com/testground/testground/issues/1392) | ||
| Benchmarking using first class support for remote runners (using `remote:exec`) in Testground | ||
|
|
||
| ### [D. Expansive protocol test coverage](https://github.com/libp2p/test-plans/issues/64) | ||
| **Why:** Having interoperability tests with lots of transports, encryption mechanisms, and stream muxers is great. However, we need to stay backwards-compatible with legacy libp2p releases, with other libp2p implementations, and less advanced libp2p stacks. | ||
|
|
||
| **Goal:** Expand beyond unit tests and have expansive test-plan coverage that covers all protocols. | ||
|
|
||
| This effort depends on [Testground Milestone 6](https://github.com/testground/testground/blob/master/ROADMAP.md#6-support-libp2ps-interoperability-testing-story-and-probelabs-work-as-a-way-to-drive-critical-testground-improvements) | ||
|
|
||
| <!-- TODO: List all major protocol test backlog items here. | ||
| Decide as a team which ones to prioritize and then assign to quarters.--> | ||
| #### 1. DHT server mode scale test | ||
| Test js-libp2p DHT Server Mode at scale (testbed of at least >20 nodes; ideally 100/1000+) in Testground | ||
| Depends on [C.1](#1-benchmarking-using-nix-builders) | ||
| Relates to [Testground Milestone 4 (for large scale tests.)](https://github.com/testground/testground/blob/master/ROADMAP.md#4-provide-a-testground-as-a-service-cluster-used-by-libp2p--ipfs-teams) | ||
| #### 2. AutoNat | ||
| Depends on [testground/NAT and/or firewall support](https://github.com/testground/testground/issues/1299) | ||
| #### [3. Hole Punching](https://github.com/libp2p/test-plans/issues/21) | ||
| Depends on [testground/NAT and/or firewall support](https://github.com/testground/testground/issues/1299) | ||
| #### 4. AutoRelay | ||
| #### 5. Custom topologies | ||
| #### 6. MTU Fixes | ||
| Depends on [testground/Network Simulation Fixes](https://github.com/testground/testground/issues/1492) |