2024 CNTI Improvements and potential major refactors

[taylor]

There is a need for a reassessment of the test suite goals and overall improvements for usability and maintainability has been building within the Test Suite community (eg. end users, contributors, upstream projects).

This need has come as a result of direct explicit and indirect analysis of usage and engagement within the community such as people going through the certification process which uses the test suite, companies utilizing the test suite as part of their product life cycle (eg. in development + testing CI/CD pipelines), upstream projects attempting to contribute to tests, and explicitly by community members wanting to help with usability and maintenance improvements.

For example TietoEvry has proposed improvements and major updates for the Test Suite. The original presentation given to the CNTI Test Catalog community (on March 12th, 2024) can be viewed at https://docs.google.com/presentation/d/1nuobTVOK1N1o7XvOJCLfMFJhTzgZJBG-/edit

• Summary of TietoEvry's motivations for refactor proposal:
  • End-user and maintainer usability updates
  • Decrease needed work on non-core assets
    • Allow LFN-CNTI community focus on core assets
  • Improve test case traceability
  • Improvements to framework and non-test features.
    • For example: Test logging, Test progress reporting, Spec test result reporting
• A document for discussing the improvements and related items is here https://docs.google.com/document/d/1OCN40Oap4-acZ70nxwaFghZQMAHY6hpQNcd17JomvVA/edit#heading=h.4zj5gq5h9l3z

[taylor]

As indicated on the test catalog call today there have been restrictions and motivations in the past that may no longer be relevant.

There have also been some motivations and areas that were ambiguous and "externally applied to the project" which slowed down the progress of the test suite.

If we can motivations, use cases, and goals which have more clarity as well as remove unnecessary limits and ambiguity then the progress, long term viability, and chances for success will improve

Next steps:

• Documenting and discussing current and future motivation and related goals for the test suite. Eg. high-level goals increase contributors and maintainers for the test suite. Increase the number of end-users of certification (which implies unspecified goals for the test suite to support the certification).
• List use cases for the test suite - (brainstorming ideas). Prioritize the use cases (with community input)