From ce8674936cdc31f601dc06d23595e79f5d356087 Mon Sep 17 00:00:00 2001 From: Ed Page Date: Wed, 7 Jun 2023 13:22:53 -0500 Subject: [PATCH 1/5] feat(testing): Initial draft --- text/0000-t-test.md | 73 +++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 73 insertions(+) create mode 100644 text/0000-t-test.md diff --git a/text/0000-t-test.md b/text/0000-t-test.md new file mode 100644 index 00000000000..723ccc7b65a --- /dev/null +++ b/text/0000-t-test.md @@ -0,0 +1,73 @@ +- Feature Name: n/a +- Start Date: (2023-06-07) +- RFC PR: [rust-lang/rfcs#0000](https://github.com/rust-lang/rfcs/pull/0000) +- Rust Issue: n/a + +# Summary +[summary]: #summary + +Create a team focused on the testing in the development workflow under primarily under [t-devtools](https://www.rust-lang.org/governance/teams/dev-tools) but also working closely with [t-libs](https://www.rust-lang.org/governance/teams/library) when it comes to the API and implementation of the built-in test harness. + +# Motivation +[motivation]: #motivation + +In [rust 1.70.0](https://blog.rust-lang.org/2023/06/01/Rust-1.70.0.html), a bug was fixed so unstable test features, like `cargo test -- --format json`, require using a nightly toolchain, like other unstable features. The problem is IDEs and CIs have been relying on this behavior to get programmatic test data and little progress has been made in the last 5 years ([#49359](https://github.com/rust-lang/rust/issues/49359)). + +Before this, there was a growing interest in improving testing generally, like [better CI integration](https://internals.rust-lang.org/t/pre-rfc-implementing-test-binary-list-format-json-for-use-by-ide-test-explorers-runners/18308). + +One challenge with improving the situation is that a lot of concerns cross multiple teams, in particular cargo (for `cargo test`), libs (for libtest). This makes it more difficult to establish a vision and coordinate efforts to achieve that vision. + +There is also precedence for this today with t-devtools having more specific sub-teams like rustdoc and rustfmt. + +## Mission and responsibilities +[mission]: #mission + +This team would primarily be focused on iterating on the test writing experience, `cargo test`, and enabling integration with external tools like CI or IDEs. + +Examples issues to resolve: +- [Stabilize support for programmatic (json) test output](https://github.com/rust-lang/rust/issues/49359) +- What parts of [cargo nextest](https://nexte.st/) can we stabilize? +- With the proliferation of test frameworks (e.g. [rstest], [trybuild], [trycmd], [cargo_test_support], [criterion]), are there underlying needs we can resolve? + +## Relationships to other teams + +T-devtools: This will be the parent team. + +**T-cargo**: This is a sibling team that T-testing will need to work with similarly to T-rustfmt, T-clippy, etc. + +**T-rustdoc**: This is a sibling team that T-testing will likely coordinate with if any changes are need to how we do doctesting + +**T-libs**: This will effectively be another parent team as they are ultimately responsible for libtest. + +## Processes + +For decisions on vision and direction, T-testing will use a standard FCP process. T-testing will be subject to [T-cargo's processes](https://doc.crates.io/contrib/team.html#decision-process) when dealing with `cargo test` and T-libs's processes for libtest. For any newly developed crates and libraries, we will follow [T-cargo's processes](https://doc.crates.io/contrib/team.html#decision-process). + +## Membership + +Team members are expected to shepherd testing discussions and vote on FCPs and is independent of regular code contributions though contributing can help build up the relevant experience and/or demonstrate the qualities for team membership. + +Qualifications that will be taken into account for membership include: + +- Does the person have the judgement for when deciding when a larger consensus is needed? +- Does the person understand the constraints of backwards compatibility within `cargo test` and libtest and exercise caution when extending the compatibly constraints? + +Someone can become a member of the Cargo Team by requesting a review or being nominated by one of the existing members. They can be added by unanimous consent of the team. The team lead or another member of the team will also confirm with the moderation team that there are no concerns involving the proposed team member. + +Leads are responsible for scheduling and facilitating team meetings and will be selected from the team members by consensus. + +The initial members of the testing team shall be: +- Lead: Caleb Cartwright (@calebcartwright) +- Ed Page (@epage) +- Weihang Lo (@weihanglo) +- Scott Schafer (@Muscraft) + +## Drawbacks and Alternatives + +- The ownership lines are becoming more muddled with the two teams dealing with `cargo test` and two teams dealing with libtest. + +[rstest]: https://crates.io/crates/rstest +[trybuild]: https://crates.io/crates/trybuild +[trycmd]: https://crates.io/crates/trycmd +[cargo_test_support]: https://doc.rust-lang.org/nightly/nightly-rustc/cargo_test_support/ +[criterion]: https://crates.io/crates/criterion From 06e84401d00f646ee041c4835374fc3121f3aa6d Mon Sep 17 00:00:00 2001 From: Caleb Cartwright Date: Sun, 25 Jun 2023 14:35:27 -0500 Subject: [PATCH 2/5] docs: elaborate on a few points --- text/0000-t-test.md | 65 ++++++++++++++++++++++++++++++++++----------- 1 file changed, 50 insertions(+), 15 deletions(-) diff --git a/text/0000-t-test.md b/text/0000-t-test.md index 723ccc7b65a..8d9db4f220c 100644 --- a/text/0000-t-test.md +++ b/text/0000-t-test.md @@ -1,43 +1,67 @@ - Feature Name: n/a -- Start Date: (2023-06-07) +- Start Date: (2023-06-27) - RFC PR: [rust-lang/rfcs#0000](https://github.com/rust-lang/rfcs/pull/0000) - Rust Issue: n/a # Summary [summary]: #summary -Create a team focused on the testing in the development workflow under primarily under [t-devtools](https://www.rust-lang.org/governance/teams/dev-tools) but also working closely with [t-libs](https://www.rust-lang.org/governance/teams/library) when it comes to the API and implementation of the built-in test harness. +Create a new subteam focused on testing in the development workflow and to be responsible for +ensuring Rust has a high quality automated testing experience that includes the capabilities +developers expect when working with a modern programming language. # Motivation [motivation]: #motivation -In [rust 1.70.0](https://blog.rust-lang.org/2023/06/01/Rust-1.70.0.html), a bug was fixed so unstable test features, like `cargo test -- --format json`, require using a nightly toolchain, like other unstable features. The problem is IDEs and CIs have been relying on this behavior to get programmatic test data and little progress has been made in the last 5 years ([#49359](https://github.com/rust-lang/rust/issues/49359)). +Currently, the overall automated testing experience spans multiple components owned by different +teams across the Rust Project (e.g. T-cargo with `cargo test` being the primary +touch point for most users, T-libs for libtest, T-rustdoc for doctests, etc.). +This makes it more difficult to establish a vision and coordinate efforts to achieve that vision. +However, there isn't any single team focused on the testing picture holistically. +Simultaneously, there are a number of well known needs and feature requests in the space that do not have an explicit +owner driving the efforts to completion. -Before this, there was a growing interest in improving testing generally, like [better CI integration](https://internals.rust-lang.org/t/pre-rfc-implementing-test-binary-list-format-json-for-use-by-ide-test-explorers-runners/18308). +For example, there's been some long standing requests to have additional test output +formats, such as JSON ([#49359]) and JUnit, available on Stable Rust. +While some of these are available as unstable features in Nightly Rust, +in [Rust 1.70.0](https://blog.rust-lang.org/2023/06/01/Rust-1.70.0.html), a bug was fixed +so unstable test features, like `cargo test -- --format json`, require using a nightly +toolchain, like other unstable features. This caused breakage in certain editor/IDE and CI +related tooling as they had been relying on the prior behavior to get test data +programmatically, and little progress has been made in the last 5 years ([#49359]). -One challenge with improving the situation is that a lot of concerns cross multiple teams, in particular cargo (for `cargo test`), libs (for libtest). This makes it more difficult to establish a vision and coordinate efforts to achieve that vision. +Furthermore, there's been a growing interest in improving testing generally, like [better CI integration][ci] as well as requests for things like better support for custom test harnesses and frameworks (.e.g [#2318]). -There is also precedence for this today with t-devtools having more specific sub-teams like rustdoc and rustfmt. +The new Testing team is intended to establish an overarching vision and provide focused attention on these areas. + +[#49359]: https://github.com/rust-lang/rust/issues/49359 +[#50297]: https://github.com/rust-lang/rust/issues/50297 +[#2318]: https://github.com/rust-lang/rfcs/pull/2318 +[ci]: (https://internals.rust-lang.org/t/pre-rfc-implementing-test-binary-list-format-json-for-use-by-ide-test-explorers-runners/18308) ## Mission and responsibilities [mission]: #mission -This team would primarily be focused on iterating on the test writing experience, `cargo test`, and enabling integration with external tools like CI or IDEs. +This team would be primarily focused on iterating on the test writing and reporting experience, `cargo test`, and enabling integration points and features for external tools like CI or IDEs. -Examples issues to resolve: +Examples of issues to resolve: - [Stabilize support for programmatic (json) test output](https://github.com/rust-lang/rust/issues/49359) - What parts of [cargo nextest](https://nexte.st/) can we stabilize? - With the proliferation of test frameworks (e.g. [rstest], [trybuild], [trycmd], [cargo_test_support], [criterion]), are there underlying needs we can resolve? ## Relationships to other teams -T-devtools: This will be the parent team. +With the aforementioned breadth across the Project, the Testing team will need to have collaborative relationships with many other teams, and is conceptually a subteam of both the Libs and Dev Tools teams. + +The rust-lang/team repo does not currently support representing dual-parent subteams, so for now the Test team will be primarily under the Dev Tools team + +T-devtools: This will be the primary top level parent team. **T-cargo**: This is a sibling team that T-testing will need to work with similarly to T-rustfmt, T-clippy, etc. **T-rustdoc**: This is a sibling team that T-testing will likely coordinate with if any changes are need to how we do doctesting -**T-libs**: This will effectively be another parent team as they are ultimately responsible for libtest. +**T-libs**: This will be a second/secondary top level parent team as they are ultimately responsible for libtest. ## Processes @@ -45,16 +69,16 @@ For decisions on vision and direction, T-testing will use a standard FCP process ## Membership -Team members are expected to shepherd testing discussions and vote on FCPs and is independent of regular code contributions though contributing can help build up the relevant experience and/or demonstrate the qualities for team membership. +Team members are expected to shepherd testing discussions and vote on FCPs. Team membership is independent of regular code contributions though contributing can help build up the relevant experience and/or demonstrate the qualities for team membership. Qualifications that will be taken into account for membership include: - Does the person have the judgement for when deciding when a larger consensus is needed? - Does the person understand the constraints of backwards compatibility within `cargo test` and libtest and exercise caution when extending the compatibly constraints? -Someone can become a member of the Cargo Team by requesting a review or being nominated by one of the existing members. They can be added by unanimous consent of the team. The team lead or another member of the team will also confirm with the moderation team that there are no concerns involving the proposed team member. +Someone can become a member of the Testing Team by requesting a review or being nominated by one of the existing members. They can be added by unanimous consent of the team. The team lead or another member of the team will also confirm with the moderation team that there are no concerns involving the proposed team member. -Leads are responsible for scheduling and facilitating team meetings and will be selected from the team members by consensus. +Team Leads are responsible for scheduling and facilitating team meetings and will be selected from the team members by consensus. The initial members of the testing team shall be: - Lead: Caleb Cartwright (@calebcartwright) @@ -62,10 +86,21 @@ The initial members of the testing team shall be: - Weihang Lo (@weihanglo) - Scott Schafer (@Muscraft) -## Drawbacks and Alternatives +## Drawbacks + +The proposed Testing team bears some similarity to other Rust teams (e.g. Types team) in the sense +that it would complicate and muddle the ownership of specific problems. +For example, there would be two teams dealing with `cargo test` and two dealing with libtest. + +## Rationale and alternatives + +- This could be a working group instead of a team. However, we believe the [reasoning articulated in the Types team RFC][team-not-wg] is applicable here as well. There is a need for focused effort on driving work to completion along with associated maintenance work; not a shorter-lived initiative to create recommendations. +- The Testing team could be a dual-parent subteam, but with the primary team under the Libs team. However, we believe Dev Tools is the better primary parent given the purview of the Testing team would extend well beyond libtest +- The Testing team could be a single-parent subteam. We think there's too much overlap with too many teams across multiple top level teams to be a single-parent subteam. +- We could do nothing and not form a new subteam nor a new working group. This would perpetuate the status quo and would most likely result in continued stagnation/lack of progress on the aforementioned focus areas. -- The ownership lines are becoming more muddled with the two teams dealing with `cargo test` and two teams dealing with libtest. +[team-not-wg]: https://rust-lang.github.io/rfcs/3254-types-team.html#why-a-team-and-not-a-working-group-what-is-the-difference-between-those-anyway [rstest]: https://crates.io/crates/rstest [trybuild]: https://crates.io/crates/trybuild [trycmd]: https://crates.io/crates/trycmd From f327e00a350885978c6a75cb331e1a3d3c43a4c5 Mon Sep 17 00:00:00 2001 From: Caleb Cartwright Date: Mon, 3 Jul 2023 13:28:00 -0500 Subject: [PATCH 3/5] rewording and case corrections --- text/0000-t-test.md | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/text/0000-t-test.md b/text/0000-t-test.md index 8d9db4f220c..56c0e9b0a03 100644 --- a/text/0000-t-test.md +++ b/text/0000-t-test.md @@ -42,7 +42,7 @@ The new Testing team is intended to establish an overarching vision and provide ## Mission and responsibilities [mission]: #mission -This team would be primarily focused on iterating on the test writing and reporting experience, `cargo test`, and enabling integration points and features for external tools like CI or IDEs. +This team would be primarily focused on iterating on the test writing and analysis experience, `cargo test`, and enabling integration points and features for external tools like CI or IDEs. Examples of issues to resolve: - [Stabilize support for programmatic (json) test output](https://github.com/rust-lang/rust/issues/49359) @@ -53,7 +53,7 @@ Examples of issues to resolve: With the aforementioned breadth across the Project, the Testing team will need to have collaborative relationships with many other teams, and is conceptually a subteam of both the Libs and Dev Tools teams. -The rust-lang/team repo does not currently support representing dual-parent subteams, so for now the Test team will be primarily under the Dev Tools team +The rust-lang/team repo does not currently support representing dual-parent subteams, so for now the Testing team will be primarily under the Dev Tools team T-devtools: This will be the primary top level parent team. @@ -61,6 +61,8 @@ T-devtools: This will be the primary top level parent team. **T-rustdoc**: This is a sibling team that T-testing will likely coordinate with if any changes are need to how we do doctesting +**T-IDEs and Editors**: This is a sibling team that T-testing will likely coordinate with to understand the needs of IDEs/editors related to incorporating test related capabilities + **T-libs**: This will be a second/secondary top level parent team as they are ultimately responsible for libtest. ## Processes @@ -80,7 +82,7 @@ Someone can become a member of the Testing Team by requesting a review or being Team Leads are responsible for scheduling and facilitating team meetings and will be selected from the team members by consensus. -The initial members of the testing team shall be: +The initial members of the Testing team shall be: - Lead: Caleb Cartwright (@calebcartwright) - Ed Page (@epage) - Weihang Lo (@weihanglo) From 43ad2a298dce57e165a3fcb21de3ba57c09a3b0e Mon Sep 17 00:00:00 2001 From: Ed Page Date: Mon, 3 Jul 2023 13:40:15 -0500 Subject: [PATCH 4/5] fix: Update RFC number --- text/{0000-t-test.md => 3455-t-test.md} | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) rename text/{0000-t-test.md => 3455-t-test.md} (98%) diff --git a/text/0000-t-test.md b/text/3455-t-test.md similarity index 98% rename from text/0000-t-test.md rename to text/3455-t-test.md index 56c0e9b0a03..58c0a43f5fa 100644 --- a/text/0000-t-test.md +++ b/text/3455-t-test.md @@ -1,6 +1,6 @@ - Feature Name: n/a - Start Date: (2023-06-27) -- RFC PR: [rust-lang/rfcs#0000](https://github.com/rust-lang/rfcs/pull/0000) +- RFC PR: [rust-lang/rfcs#3455](https://github.com/rust-lang/rfcs/pull/3455) - Rust Issue: n/a # Summary From aee38982603b4c4b481a8bc5efc12cde0b79d2c6 Mon Sep 17 00:00:00 2001 From: Ed Page Date: Tue, 18 Jul 2023 13:48:34 -0500 Subject: [PATCH 5/5] fix(testing): Add thom to the initial team --- text/3455-t-test.md | 1 + 1 file changed, 1 insertion(+) diff --git a/text/3455-t-test.md b/text/3455-t-test.md index 58c0a43f5fa..39d6bf3a5a6 100644 --- a/text/3455-t-test.md +++ b/text/3455-t-test.md @@ -87,6 +87,7 @@ The initial members of the Testing team shall be: - Ed Page (@epage) - Weihang Lo (@weihanglo) - Scott Schafer (@Muscraft) +- Thom Chiovoloni (@thomcc) ## Drawbacks