Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

change "regression tests" in documentation to "consistency tests" #434

Closed
edwardhartnett opened this issue Apr 6, 2021 · 1 comment
Closed
Assignees
Labels
documentation Improvements or additions to documentation

Comments

@edwardhartnett
Copy link
Collaborator

We are using a very different meaning for the term "regression testing" than is usually meant.

Regression testing is when a programmer runs the unit test suite (or a subset) after every small change in code. This is usually done on the programmers computer (i.e. with "make test"), but can be accomplished with GitHub continuous integration, if the programmer commits changes frequently, and pays attention to the results of the CI run on each commit.

The name comes from the agile term "regression", which is when a programmer working on the code accidentally breaks code that was already working. It is intended to be done frequently, because the cost of fixing a bug goes up dramatically over time.

For more on regression testing: https://en.wikipedia.org/wiki/Regression_testing

What we are doing is not regression testing for two reasons:
1 - we are not testing any results for correctness, but only that they have not changed.
2 - the tests are not being run by the programmer as the work is done, but instead done at the end, when work is submitted as a PR. Finding bugs months after they were introduced is not what anyone else means by "regression testing."

Let us re-term this "consistency checking." All we are checking is that we did not change any results. We are not checking that the results are actually correct.

When we have unit tests, and the programmers are using them properly, then we will have regression testing. Let's not claim to have something that we don't have.

@edwardhartnett
Copy link
Collaborator Author

I believe this has been changed everywhere it needs to change. Moving forward, please stop calling these "regression" tests, because that is claiming something we don't yet have - comprehensive unit testing.

One day, when we have comprehensive unit testing, we will have regression testing.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
documentation Improvements or additions to documentation
Projects
None yet
Development

No branches or pull requests

1 participant