Provide interface to the inference package EP-BOLFI

[YannickNoelStephanKuhn]

Feature description

Makes EP-BOLFI available, an optimizer developed specifically for difficulties that are common with battery parameterization, such as, but not limited to:

• noisy data,
• incomplete models,
• interdependent model parameters, and
• ill-posed parameterizations.

Motivation

While EP-BOLFI, as in EP plus BOLFI, is expected to be superseded by its implementation within SOBER, its own interface here will serve as the basis of benchmarking the two against each other. Additionally, there are a lot of "self-driving" features within EP-BOLFI, the library, that make week-long runs on HPC resources viable. Finally, it might be the case that EP-BOLFI, the library, works better with limited resources, e.g., your decade-old work laptop.

Possible implementation

I would extend BaseOptimiser in a minimal fashion, rather than the way it is done for AdamWImpl. While EP-BOLFI internally has a lot of similar structure to what is shown in AdamWImpl, it would require a complete refactor of EP-BOLFI to expose it in this manner.
I will also have to extend OptimisationResult to a BayesOptimisationResult to catch parameter correlations and general uncertainty information. I would give it its own file, but I'm open to other suggestions.
Next, I need to figure out how to deal with the features in EP as a cost function and the Likelihoods involved as per pybop.costs. I'm unsure whether this is necessary; if so, this requires a minor refactor of EP-BOLFI.
Optionally, I would look into the way probabilistic Parameters are handled right now, checking if they can handle inherent parameter uncertainties that EP-BOLFI was designed to handle. These are not parameters to fit, but parameters that would be randomly generated in each model evaluation during optimization.
Finally, there may need to be some additional plotting routines for the BayesOptimisationResult.

Additional context

I've been asked to provide an interface to EP-BOLFI in the past, and I'm happy to finally have the resources to oblige.
This will also help me with addressing the SOBER and BASQ interfaces mentioned in #228 afterwards, as I can practice a Bayesian Optimizer interface with something I'm more intimately familiar with.

[BradyPlanden]

Hi Yannick,

Thanks for opening this issue. This sounds like a reasonable path forward, feel free to ping me as you go through the process of integrating the cost/likelihood and parameters. If you can structure this functionality within the experimental directory (see #481) for the initial landing, we can move it outside once it's stabilised. Cheers!

[YannickNoelStephanKuhn]

Hi Brady,
will do, thanks for the offer. For the time being, I will just document my progress here.
As a first step, I drafted how PyBOP might handle multivariate distributions; the multivariate priors are drafted in YannickNoelStephanKuhn@15e75fc and the multivariate parameters are drafted in YannickNoelStephanKuhn@6ef018a.
While not strictly necessary, it enables a technique common in Bayesian Optimisation: re-using a Posterior as a Prior for further refinement.

[YannickNoelStephanKuhn]

Drafted a minimal EP-BOLFI interface in YannickNoelStephanKuhn@20dd5a6.
It translates PyBOP input -> EP-BOLFI routine -> PyBOP output, but does not integrate the sampling in any way with PyBOP. I will also have to check if it works out to keep parameter transformations on the PyBOP side, or if I have to move them over to EP-BOLFI.
Posteriors are tacked onto a BayesianOptimisationResult, handled via the BasePrior object type.
The Likelihood is available via the bolfi_posterior attribute; I could wrap that within a BaseMetaLikelihood, but it is a result, not an input like the other PyBOP Likelihoods.