Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add answer testing framework #220

Merged
merged 21 commits into from
Jun 30, 2022
Merged

Add answer testing framework #220

merged 21 commits into from
Jun 30, 2022

Conversation

brittonsmith
Copy link
Contributor

@brittonsmith brittonsmith commented Jun 14, 2022

This adds a framework for performing answer testing. The EnzoETest class is a base class for creating answer tests. Test files are run in a temporary directory which is then removed when the tests are done. The TestHLLCCloud class gives an example of using this with a simple test problem.

I have added documentation for running and creating new answer tests. This required some slight reorganization and rewording of the existing testing documentation. I've removed instructions from this comment so reviewers can try the instructions in the documentation. Go here to see a rendered version of the documentation. In particular, the new documentation can be found here.

This is now working within circleci with a single test. @mabruzzo, I used one of your vlct simulations to provide a template. Please, feel free to alter that test however you see fit. I implemented something very simple.

@jwise77
Copy link
Contributor

jwise77 commented Jun 15, 2022

Typo in your test filename test_vclt.py ➡️ test_vlct.py. But looks good so far!

@jwise77
Copy link
Contributor

jwise77 commented Jun 15, 2022

Comment from @mabruzzo during triage: How and can the tests be run in both single and double precision? Similar to how we test the compilation in both modes.

@jwise77 jwise77 added this to the v1.0 milestone Jun 15, 2022
@brittonsmith
Copy link
Contributor Author

Comment from @mabruzzo during triage: How and can the tests be run in both single and double precision? Similar to how we test the compilation in both modes.

I've got a solution for this that works with setting environment variables. I will make sure to include this in the documentation.

@brittonsmith brittonsmith changed the title [work in progress] Add answer testing framework Add answer testing framework Jun 22, 2022
@mabruzzo mabruzzo self-requested a review June 22, 2022 13:18
Copy link
Contributor

@mabruzzo mabruzzo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry for taking so long to review this. This looks great to me!

Two very minor points:

  • I did have a question about a website-documentation change that you made.
  • I also wanted to ask what you thought about enabling the answer-tests on CI for single precision. In the past, I've anecdotally found that the single precision tests catch a fair amount of mistakes (since I primarily develop the code in double precision mode). It's not strictly necessary to do this right now since we don't have many pytest-tests. But, it might be easier to address this now (so that mistakes in newly implemented answer tests are caught more quickly). What does Enzo do?

doc/source/conf.py Outdated Show resolved Hide resolved
@brittonsmith
Copy link
Contributor Author

@mabruzzo I'm happy to run single precision answer tests on circleci. I wasn't sure we wanted to use resources for both and chose arbitrarily.

@fearmayo fearmayo merged commit 6168a61 into enzo-project:main Jun 30, 2022
@brittonsmith brittonsmith deleted the atest branch June 30, 2022 11:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants