Software priorities for paper

[MichaelClerx]

Hi all!

Just talking to @ben18785 about the things we really need to do for the paper. Thought it'd be good to make an overview ticket.

MCMC Methods

• Metropolis Add standard Metropolis algorithm #253
• Adaptive covariance adaptive covariance MCMC functionality #4
• Diff ev. Add differential evolution MCMC #149 Differential evolution example has undocumented stuff at the end #189
• Emcee hammer Add emcee hammer MCMC sampler #148 Add emcee hammer test #184 Update emcee MC Hammer to ask-and-tell interface? #225 Recreate emcee hammer in Pints (without wrapper) #281 (ticket) i281 Recreate emcee hammer #319 (PR) Update mcmc ask/tell to allow None to be returned #320 (interface) (@MichaelClerx @ben18785 ) --> Almost there
• Population Add population MCMC #186
• HMC Add Hamiltonian MC (HMC) to Pints #104 (@ben18785 @MichaelClerx ) --> Nearly there
• NUTS (No from Ben & Sanmitra)
• Dream Get DREAM mcmc into ask-and-tell interface #219 (@MichaelClerx)
• MALA Implement MALA #294 --> Can be adapted from HMC quite easily (we think)

Nested methods

• Nested sampling Add nested sampling to Pints #107 (@MichaelClerx @ben18785 ) --> Needs ask-tell and notebooks
• Ellipsoid nested sampling Add nested sampling to Pints #107 (@MichaelClerx @ben18785 ) --> Needs ask-tell and notebook fix
• Multinest Recreate Multinest in Pints #282 (@ben18785 ) --> Have study, need to test if it's ok and work into sampler class

Particle-based methods

• SMC Add sequential MC (SMC) #120 (@ben18785 ) --> Needs tests, docs, etc.

Likelihood-free methods

• ABC --> Leaving this for paper 3, Would be a good student project too

Toy problems #119

• At least one toy problem with sensitivities add sensitivities for fitzhugh nagumo model #301
• Logistic Add example page for logistic toy model #125
• HH IK
• HH AP Add HH AP model to toy models #126 AP model Add Beeler Reuter AP model to toy models #395 (@chonlei)
• Fitzhugh-Nagumo Add Fitzhugh-Nagumo toy model #239 Add Fitzhugh-Nagumo MultiSeriesProblem example #275
• Lotke-Volterra Add lotka-volterra toy model #327
• Multi-output Goodwin oscillator Goodwin oscillator toy model #268
• Michaelis-menten Add a Michaelis-Menten toy problem #122 (Schnell?) Add HES1 Michaelis-Menten model #286
• Banana Add Banana problem #123
• Rosenbrock
• Repressilator Add Repressilator toy model #227
• SIR epidemiology model Add SIR toy model #366

Features

• Allow non-rectangular boundaries for optimisers Replace Boundaries object by Boundaries interface and RectangularBoundaries object #382 (@MichaelClerx )
• Allow weighted error measures Add weighting output option in SumOfSquaresError #397 (@chonlei )

Functional tests

• Tests for Optimisers Add functional tests for optimisers #241 (@ben18785 @MichaelClerx @martinjrobins )
• Tests for MCMC implementations Come up with a good way to validate inference methods #181 (@ben18785 @MichaelClerx @martinjrobins @sanmitraghosh )
• Infrastructure for functional tests (@martinjrobins @MichaelClerx )
• Add (slow) notebook testing to functional tests
• Tests for samplers (adapt MCMC tests)

Unit tests

• Full tests for diagnostics Add tests for diagnostics methods #173 (@ben18785 )
• Tests for nested sampling Add nested sampling to Pints #107 (@MichaelClerx )
• Full tests for pints.io i43 sensitivities #347
• Full tests (no output inspection) for plot.py Get pints.plot test coverage to 100% #373
• Full tests for pints._optimisers Get pints.optimisers test coverage to 100% #372
• Full tests for pints._mcmc Get coverage of pints._mcmc to 100% #374
• All tests should pass on windows Make all tests pass on windows #406 (@MichaelClerx )

Documentation

• Notebooks are so slow they keep failing the daily test

Infra

• Allow None returned from MCMC tell, for emcee hammer Update mcmc ask/tell to allow None to be returned #320 (@MichaelClerx ) --> Almost there
• Ask/tell where possible for parallelisation, assuming expensive evaluation step Change Optimiser/MCMC objects to work with 'ask-and-tell' interface #132
• Work out interface to get 1st order sensitivities into Pints Work out interface to get 1st order sensitivities into Pints #43 #298 added sensitivities for logistic model #299 Add Gaussian known noise log likelihood with first order sensitivities #343 Add LogLikelhoodWithSensitivities interface (and LogPdfWithSensitivities) #344
• Make Pints pip installable (once everything is done)

Comparison

• Matrix for optimisers, matrix for inference Comparison matrix of models versus optimisers versus inference methods #229 (@martinjrobins) --> Up and running
• Bayesian optimisation of hyper parameters (i.e. method settings)(setting hyper parameters of samplers #391 #391 add method to MCMCSampling to access underlying samplers #392 interface for setting hyper parameters of optimisers and samplers #393)

[MichaelClerx]

Questions:

• Should we mention the hierarchical MCMC examples? Rewrite hierarchical gibbs sampling code to use new framework #38
• Should we talk about software design, or leave that for a separate paper? (@sanmitraghosh thinks we should write a separate paper about it)

[martinjrobins]

Should we talk about software design, or leave that for a separate paper?

I'm guessing we'll need a separate paper for this, to concentrate on it fully, although we should describe the ask-and-tell interface in broad terms and point to a hierarchical MCMC example (which I still need to do) as an example of its utility.

[MichaelClerx]

Should we add MALA to the MCMC list for the paper? @sanmitraghosh @ben18785 ?

[sanmitraghosh]

Yes MALA, needs L'(p) and smMALA, needs L''(p)

[sanmitraghosh]

@MichaelClerx @martinjrobins @ben18785 I just want to touch up on this point that we should test all single chain MCMC methods with the FN model and particle ones with GW model. I think this is extremely important at this points or else we might run into the scenario that we have many algorithms (and associated infrastructures) that are not suited for complex ODE models. There is no point in collecting algorithms that can solve the logistic problem.

I think any algorithm that has not been published (and hopefully cited) with a working non-linear ODE example in the originating paper should be subjected to the aforementioned models FN/GW (performance on the logistic model is absolutely irrelevant for our chosen endeavour) before even implementing within PINTS. This will save us a lot of time and hassle.

[MichaelClerx]

Closing this issue now, as it has been replaced by a GitHub Project!