-
Notifications
You must be signed in to change notification settings - Fork 35
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add answer testing framework #220
Conversation
Typo in your test filename |
Comment from @mabruzzo during triage: How and can the tests be run in both single and double precision? Similar to how we test the compilation in both modes. |
I've got a solution for this that works with setting environment variables. I will make sure to include this in the documentation. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Sorry for taking so long to review this. This looks great to me!
Two very minor points:
- I did have a question about a website-documentation change that you made.
- I also wanted to ask what you thought about enabling the answer-tests on CI for single precision. In the past, I've anecdotally found that the single precision tests catch a fair amount of mistakes (since I primarily develop the code in double precision mode). It's not strictly necessary to do this right now since we don't have many pytest-tests. But, it might be easier to address this now (so that mistakes in newly implemented answer tests are caught more quickly). What does Enzo do?
@mabruzzo I'm happy to run single precision answer tests on circleci. I wasn't sure we wanted to use resources for both and chose arbitrarily. |
This adds a framework for performing answer testing. The
EnzoETest
class is a base class for creating answer tests. Test files are run in a temporary directory which is then removed when the tests are done. TheTestHLLCCloud
class gives an example of using this with a simple test problem.I have added documentation for running and creating new answer tests. This required some slight reorganization and rewording of the existing testing documentation. I've removed instructions from this comment so reviewers can try the instructions in the documentation. Go here to see a rendered version of the documentation. In particular, the new documentation can be found here.
This is now working within circleci with a single test. @mabruzzo, I used one of your vlct simulations to provide a template. Please, feel free to alter that test however you see fit. I implemented something very simple.