Trilinos Testing Policy

Test writing is an essential software development activity. Without adequate, independent and easily executed tests, software viability is at risk of breaking due changes in upstream dependencies, build environments, internal refactorings and enhancements. Under-tested software is unsustainable.

Trilinos Testing Policy:

1. All functionality will be covered by testing: Numerous approaches are possible. Chapter 22 of Steve McConnell's book Code Complete is an excellent resource. Structured Basis Testing (having as many tests as there are logic paths through your code) is a good starting point. Formal unit testing is even better.
2. Untested functionality is itself a bug: If any user or developer identifies an untested code segment in a Production Growth package, there is a right and obligation to report it.
3. Contributed code must be covered: While we appreciate non-developer contributions, they are subject to the same code coverage requirements. In particular, Trilinos developers may refuse to accept contributions that do not have tests.

Notes:

1. Policy Scope: This policy formally applies to any Production Growth package development. It is encouraged for packages in other phases (Research Stable or Production Maintenance).
2. Exceptions: Yes, there will be exceptional circumstances where you need to commit some functionality without a corresponding test, but it should be the exception and addressed as soon as possible. If you must commit code without testing coverage, file an issue. Issues with management, user or client support for taking time to write tests should be forward to the Trilinos project leader.
3. Test writing is part of your day job: Time spent writing tests is a core job responsibility, justifiable as a normal part of your work day.
4. Design for testability: Without careful design, state variations can be so complicated that it becomes impossible to confidently test all, or even the most important software capabilities.
5. The src-test diagnostic: A quick check for testing compliance: While not sufficient, one simple diagnostic for testing compliance is checking whether or not a commit log message lists changes to both src and test directories.

Questions?

Contact Mike Heroux maherou@sandia.gov

Helpful Links:

Code Complete, 2nd Edition, by Steve McConnell