WIP: Rename regression_test.py to use pytest

[rjleveque]

Also updated regression data for 2 examples with minor changes in results.

See also

• WIP: Rename regression_test.py to use pytest geoclaw#562
• WIP: Using pytest instead of nose requires renaming to regression_test.py classic#90

[rjleveque]

Two tests are failing, tests/acoustics_2d_radial and tests/acoustics_2d_adjoint. The former ran fine on my laptop and the latter did not, but I updated the regression_data in commit e1ff225, and now it runs fine on my laptop.

@ketch or @mandli, could you try running pytest with this branch and see if how they behave for you?

[mandli]

I have the following with the full pytest output attached below.

• acoustics_2d_radial

E           AssertionError:
E           Not equal to tolerance rtol=1e-14, atol=1e-08
E
E           Mismatched elements: 18 / 44 (40.9%)
E           Max absolute difference: 0.0021096
E           Max relative difference: 0.02010603

• acoustics_2d_adjoint

E           AssertionError:
E           Not equal to tolerance rtol=1e-14, atol=1e-08
E
E           Mismatched elements: 3 / 51 (5.88%)
E           Max absolute difference: 0.00021523
E           Max relative difference: 0.00756248
E           Gauge Match Failed for gauge = 1
E             failures in fields: 0

pytest_output.txt

[mandli]

I noticed that some of the tests were behaving oddly and made some adjustments to the adjoint tests. Unfortunately, while I think it fixed one problem, it caused another for the 1D version to surface.

[rjleveque]

I tried running it on a linux machine and I got the same results as @mandli, and digging into the tests/acoustics_2d_radial results, the level 3 grids it makes are slightly different larger and hence the results start to diverge near the patch boundary. I thought it might be because I was using the ifort compiler, but redoing it with gfortran on linux didn't change the results (still different than with gfortran on my MacBook). In both tests I used a fresh clone of the repos plus this PR, so I think all the code should have been identical.

Maybe there's a cell with a value that's within rounding error of the flagging tolerance and if it gets flagged another row of cells is required in the patch.

But strangely it's not just in one patch but in all 5 of the level 3 grids in the figure below. In one computation they all start at y = -0.26 and in the other at y = -0.24.

@mjberger is this something worth pursuing, or should we just change the tolerance a bit for example and hope it goes away?

[mjberger]

it's hugely annoying, sometimes impossible, to track down this kind of thing. How soon does it diverge into the calculation? Is it the same for enough time to be a meaningful test if you shorten the run time? — Marsha

…

On Jun 19, 2024, at 11:08 PM, Randall J. LeVeque ***@***.***> wrote: I tried running it on a linux machine and I got the same results as @mandli <https://github.com/mandli>, and digging into the tests/acoustics_2d_radial results, the level 3 grids it makes are slightly different larger and hence the results start to diverge near the patch boundary. I thought it might be because I was using the ifort compiler, but redoing it with gfortran on linux didn't change the results (still different than with gfortran on my MacBook). In both tests I used a fresh clone of the repos plus this PR, so I think all the code should have been identical. Maybe there's a cell with a value that's within rounding error of the flagging tolerance and if it gets flagged another row of cells is required in the patch. But strangely it's not just in one patch but in all 5 of the level 3 grids in the figure below. In one computation they all start at y = -0.26 and in the other at y = -0.24. @mjberger <https://github.com/mjberger> is this something worth pursuing, or should we just change the tolerance a bit for example and hope it goes away? frame1.png (view on web) <https://github.com/clawpack/amrclaw/assets/720122/e2d0541e-0e44-4893-a103-60245a6ba26c> — Reply to this email directly, view it on GitHub <#279 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAGUGC6VM4X7YHUJ6RJMGLDZIJBSJAVCNFSM6AAAAABJEYUUPSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCNZZG4ZDMMBUGU>. You are receiving this because you were mentioned.

[rjleveque]

It's a small unit test problem and looking at fort.amr after setting verbosity and verbosity_regrid to 3, I see that the number of cells flagged already differs at time 0, on both levels 2 and 3. So it seems like it must be something dumb I'm missing...