Re-organisation/tidying of functional tests submodule

[MichaelClerx]

Hi all,
This is a proposal for a slight re-organisation of the functional tests.

Main changes:

1. All functional tests have names that match the method they test
2. Problems are all imported into functionaltests, so you can do e.g. import pints.functionaltests.ft, ft.TestMethodOn...
3. Functional tests have lost their prefix, so you can write e.g. ft.dream_mcmc.two_dim_gaussian()
4. Added docstrings everywhere
5. Added a check to ensure the default warm-up isn't longer than the number of iterations
6. Tests are explicitly listed for each method (instead of using a name-matching system).

Test discovery example:

import pints
import pints.functionaltests as ft

print('All tests')
for x in ft.tests():
    print(x)

print()
print('All tests for population MCMC')
for x in ft.tests(pints.PopulationMCMC):
    print(x)

[MichaelClerx]

Hi functional testers @ben18785 @fcooper8472 @chonlei @DavAug @rcw5890 and anyone I'm forgetting,

I've done a proposal to make the stuff we've been adding here a bit more streamlined and usable. Changes are listed above.

Due to a naming change, the diff shows up as old fine gone, new file added, sorry about that.

Please let me know what you think!

[MichaelClerx]

I think this is still all in line with @fcooper8472 's original design: pints-team/change-point-testing#38

Although I think the updated plan was to have functional tests live somewhere outside PINTS.

One solution would be to have:

• pints
  • code, docs, unit tests, examples, etc
• pints method-merge-tests
  • the stuff that's currently under pints.functionaltests
  • the notebooks currently under https://github.com/pints-team/method-merge-tests
• pints functional testing
  • imports pints and pints method-merge-tests
  • runs the tests that are currently under pints.functionaltests, stores the results, etc.

That fits with the idea that you make the pre-merge test first, then use that to create the functional test, and then re-use it if the functional test ever fails.

Not sure how the method-merge-tests get access to unmerged new methods though (that's a problem with the current set-up too)?

[fcooper8472]

If there was an updated plan to have the functional testing methods themselves outside PINTS then I don't think I was involved in that conversation. The existing structure was the result of the most recent decision that I'm aware of.

The changes in this PR look fine to me, although I haven't looked at any of the changes in individual files because, as you say, the diffs are all obscured due to renaming.

[MichaelClerx]

Thanks!

If there was an updated plan to have the functional testing methods themselves outside PINTS then I don't think I was involved in that conversation. The existing structure was the result of the most recent decision that I'm aware of.

I just meant your updated ideas here: pints-team/change-point-testing#38 (comment)

Although reading that more carefully I'm not sure what you mean there 😅

I'm happy to keep the functional tests in PINTS!

[fcooper8472]

I think the plan was to have the methods themselves in PINTS, but the script that handles the automated running of those methods and the storing/processing of the data (the "functional testing" infrastructure) would live in another repo.

[MichaelClerx]

Gotcha!

If we're doing that, I wouldn't mind having the method merge tests in the pints repo too, if (big if!) we're OK with them not being tested and using dependencies not required by pints.

[MichaelClerx]

@ben18785 can you have a quick look at this one? Hoping to merge this soon so that it doesn't keep going out of date

[ben18785]

Thanks @MichaelClerx -- I've been through and checked things (mostly that the number of iterations, warmup and chains match) and looks good. Given that these things are part of PINTS, how should we handle the Jupyter notebooks in the method-merge-tests repo? Do they need a particular PINTS commit associated with them somewhere?

[MichaelClerx]

Thanks @MichaelClerx -- I've been through and checked things (mostly that the number of iterations, warmup and chains match) and looks good. Given that these things are part of PINTS, how should we handle the Jupyter notebooks in the method-merge-tests repo? Do they need a particular PINTS commit associated with them somewhere?

Thanks @ben18785 !

I think we can just update those whenever we need to run them? There shouldn't be many (further) updates to this code anyway, I think

[MichaelClerx]

I'll merge this in tomorrow and update the method merge tests