Long term idea - use tools like SOBER to do model selection in PyBOP

[davidhowey]

Feature description

@ma921 has developed some great ideas e.g. as described in this paper and more recently SOBER for high speed Bayesian quadrature that we can use for model selection. It would be great, as a long term aim, to get this into PyBOP.

Motivation

No response

Possible implementation

No response

Additional context

No response

[BradyPlanden]

I have been thinking about this for a while now. I think there are a few technical decisions we need to sort out to make this work, here's a first list:

• How does SOBER want to be distributed? There is an open PR to release it on PyPI: configured setup files to work with PyPI ma921/SOBER#5, if so, is the goal to maintain and update the package?
• Depending on the above, we can import SOBER as a dependency and integrate it into the PyBOP codebase, this might be a bit clumsy as the API's will be a bit different. Alternatively, we can incorporate SOBER into our codebase and then modify/maintain it as needed. In this case it would essentially be like forking the repo, but into PyBOP. I think this is probably the better way forward as, as we can have a better integration of SOBER.
• A similar discussion is also needed for BASQ. From the looks of it, SOBER does the same thing with BASQ as I described above, so perhaps this is the expected method of use.

@ma921, I'd be interested to hear your thoughts on this. There's a follow-up discussion about how to handle SOBER's dependencies, and perhaps some of them wouldn't be needed after ingestion into PyBOP.

[ma921]

Thank @davidhowey and @BradyPlanden!
I have no preference—the PyPI matter is not as I had planned.
As you mentioned, I think it's a great idea to integrate the SOBER codebase into one of the optimizers.
Since BASQ is an inner loop function of SOBER, including SOBER means BASQ is automatically incorporated.

Several potential issues to consider:

• Numpy vs. PyTorch: SOBER relies on PyTorch, GPyTorch, and BoTorch. These dependencies are essential. The simplest method to integrate SOBER is to have an interface that accepts numpy arrays but internally converts them into torch tensors. This approach is inefficient but common (for example, in BoTorch). I prefer using PyTorch since it supports GPU acceleration and automatic differentiation.
• Interface: SOBER is based on Bayesian principles, requiring a prior distribution over all parameters. Fortunately, the pybop/parameters/priors.py file is quite similar to the SOBER framework, so adding additional functions there should be compatible with SOBER as well.

How should we proceed to the next steps?

[BradyPlanden]

Excellent, thanks for the response @ma921. In which case, it seems ingesting SOBER (and by extension BASQ) into the PyBOP codebase makes the most sense. We haven't finalised the bayes architecture yet, so this would be a good time to align requirements. I'll take a further look at SOBER's API and let's have a chat early next week if that works.

In terms of dependancies, I think this is fine. We will probably want an optional installation and import like we do with plotly. Cross-posting to #229, as pytorch is a dependancy requirement there as well.

I'll create a branch for this issue, and if you want to start integrating that would be great. Opening a WIP PR would be a great way for us to review and collaborate as this progresses. For reference, here are the contributing guidelines . I'm happy to support development of this, so give me a shout if you want a hand!

[ma921]

Excellent @BradyPlanden! I'm proceeding with merging my codebases into branch 228 and will inform you through a WIP PR once completed.
After the deployment of my SOBER code base, it will establish a foundation that enables compatibility with other Bayesian methodologies, including MCMCs and Bayesian optimization. We might encounter minor differences, such as setting up priors via different libraries like torch.distributions, Pyro, or PyMC3. However, these issues primarily pertain to interfacing and should be manageable.
Stay tuned for updates!

[YannickNoelStephanKuhn]

Hello everyone,
I am commenting on this stale issue, since I experimented on how to use SOBER for battery-related optimization tasks. It's at a point now where I am confident that I figured out the intricacies of SOBER (and BASQ) and wrote some code that makes SOBER more accessible for non-ML people.
I can not guarantee yet in which state that code will release; I just wanted to let you know that
a) someone has figured out a way of using SOBER within PyBOP, and
b) that someone will make it available soonish.

[YannickNoelStephanKuhn]

We figured out how we would like to commit our code: I would make a fork of the PyBOP repository, where we figure out if what I wrote aligns with the general design principle behind PyBOP.
Is that okay with the main developers?

[BradyPlanden]

Hi @YannickNoelStephanKuhn, thanks for posting an update on this issue. Yes, forking and opening a PR for integration would be the recommended way to go. For reference, here is our Contributing Guide which should get you up to speed on running tests, pre-commit, etc. Looking forward to your updates!