PiBO #813

hvarfner · 2022-01-17T17:40:49Z

PiBO - BO with user Priors over the Optimum

New attempt on PiBO - here are the changes from last time:

New Added unit tests for PriorAcquisitionFunction (and required supporting classes)

Tests for the _compute_prior for different shapes
Tests for standard computation for both EI (does not require rescaling) and TS (does require rescaling)
Tests for discretizing the prior (relevant for random forests

Added unit test for LocalAndSortedPriorRandomSearch

Since this one is pretty much tested (it just appends two different random searches to one another), there is really only one thing to test to me, but I may be wrong

Removed the need to sample the prior to get the co-domain of the prior

Moved to closed-form in ConfigSpace

Changed input kwargs in facade

Now called user_prior_kwargs (never got acquisition_function_kwargs to work, since the underlying acquisition function gets arguments it does not expect, like decay_beta)

Confirmed issue combining inactive with Normal / Beta hyperparameters

Due to what I believe is some old TPE support, Normal parameters are not normalized to 0-1 range (presumably since they can be used without upper and lower bounds
Inactive parameters are flagged with a -1 value (right?) for Uniform parameters, which does not mesh with the possibly unbounded nature of Normal parameters
As such, one could force Normal parameters to be bounded, normalize, and allow for PiBO with inactive parameters, but there may be some other issue regaring this that I fail to see. Simply put, I don't know if it's OK to forcibly bound and normalize Normal parameters in ConfigSpace.

Added beauitified example usage

May not be the most intuitive, but at least it showcases how to use it

And of course, right branch this time

Distributions for GPs. Added: - Method get_configspace in base_epm (needed in PriorAcquisitionFunction) - Allow for NormalInteger- and NormalFloatHyperparameter in util_funcs check - Added support for enabling user priors in optimization in facade - Added PriorAcquisitionFunction class in acquisition.py

hyperparameters

incorporated it into Local Search with priors

still haven't discretized the prior, though

with random forests and continous parameters

models

prior_optimization_kwargs

verbose

hyperparameters

example and removed finished TODOs

PriorAcquisitionFunction

in ConfigSpace (Hyperparameter.get_max_density())

search space based on the prior (not only the uniform one).

retrieved form the uniform configspace

benjamc · 2022-03-24T08:49:40Z

We need to wait for the new ConfigSpace release and then see if the tests are running through. After that, we can merge. 🌞

eddiebergman · 2022-03-24T09:39:52Z

I'll release a new ConfigSpace later today :)

hvarfner · 2022-03-24T09:40:33Z

Great, thanks!

eddiebergman · 2022-03-24T11:44:17Z

You can check progress here, I bumped the version notes and now the wheels are building for Windows/Linux/Mac ... Once those are done I can publish a release :)

eddiebergman · 2022-03-24T11:46:20Z

One more point: Considering it's a fairly major change to ConfigSpace, I bumped from 0.4.21 -> 0.5.0. This means the requirements.txt will need to be updated. I did this change as it will also effect many other libraries that rely on ConfigSpace, it should hopefully prevent anyone being safe and relying on the 0.4.x from experiencing sudden issues.

eddiebergman · 2022-03-24T15:33:21Z

@hvarfner, @benjamc It's live, https://pypi.org/project/ConfigSpace/#files :)

Just going to reiterate the requirements file needs to be changed here.

eddiebergman · 2022-03-24T15:44:07Z

Sorry more notes, the dependency cycle continues. Autosklearn can't use new ConfigSpace until there's a new SMAC release. It's not urgent but just that I will be nudging this along while possible.

hvarfner · 2022-04-03T20:00:29Z

Hi,

If I'm not mistaking, we waited for configspace to be in place before, right? Is there more that needs to be done? If so, please let me know what I can do!

eddiebergman · 2022-04-04T11:43:30Z

Hi @hvarfner,

As previously mentioned, the requirements file needs to be updated so that the right version of ConfigSpace gets pulled in. If you look at the test errors, you can see that ConfigSpace has no variable BetaFloatHyperparameter and other such errors. Hopefully there'll be no bugs to fix with that but we'll have to see :)

Best,
Eddie

renesass · 2022-04-04T11:47:30Z

We have another PR going on right now, which updates the files based on updated tools (black, flake8, ...). We would love to have PiBO in this "cleaned" state, although it takes a bit more time. I can't promise it but most likely we can merge it Thursday.

Sorry for the late responses - a lot going on recently.

mlindauer · 2022-04-04T11:52:24Z

Hi Rene and Eddie,
ICLR is in a few weeks and of course, we want to advertise piBO as part of SMAC (and Hypermapper). So please make sure that this is done before the ICLR starts -- earlier would be even better because people have watched Carl's yt video already.

renesass · 2022-04-07T12:25:16Z

smac/facade/smac_ac_facade.py

+                model_instance, RFRImputator
+            )
+            acquisition_function_instance = PriorAcquisitionFunction(
+                acquisition_function=acquisition_function_instance, **user_prior_kwargs, **acq_def_kwargs


Both decay_beta and discretize are not used. Should they be part of "user_prior_kwargs"?

After this issue is fixed, we can merge it. :)

Great to hear! I'll check it right away.

Addressed the issue now, thanks for spotting.

…o_dev

- PiBO: Augment the acquisition function by multiplying by a pdf given by the user. The prior then decays over time, allowing for the optimization to carry on as per default. - The `RunHistory` can now act as a `Mapping` in that you can use the usual methods you can use on dicts, i.e. `len(rh)`, `rh.items()`, `rh[key]`. Previously this was usually done by accessing `rh.data` which is still possible. - Updated the signature of the `ROAR` facade to match with it's parent class `SMAC4AC`. Anyone relying on the output directory without specifying an explicit `run_id` to a `ROAR` facade should now expect to see the output directory at `run_0` instead of `run_1`. See #827. - Updated and integrated flake8, mypy, black, and isort. - SMAC uses `automl_sphinx_theme` now and therefore the API is displayed nicer. Co-authored-by: Carl Hvarfner <58733990+hvarfner@users.noreply.github.com> Co-authored-by: Difan Deng <deng@dengster.tnt.uni-hannover.de> Co-authored-by: Carolin Benjamins <benjamins@tnt.uni-hannover.de> Co-authored-by: Tim Ruhkopf <timruhkopf@gmail.com> Co-authored-by: eddiebergman <eddiebergmanhs@gmail.com>

hvarfner added 30 commits December 8, 2021 16:53

Changed one line where the wrong PDF was evaluated

f2829d6

Added support for new hyperparametersAdded support for new

22a27c1

hyperparameters

Added Local Search which makes use of both uniform and prior samples

7b97916

Changed facade to accept uniform configspace when using priors, and

520a97d

incorporated it into Local Search with priors

Addded support for Beta and Normal hyperparameters in random forest -

d7e30c9

still haven't discretized the prior, though

Commited all stuff regarding pibo so far

fd996ec

Implemented working acquisition fucntion discretization when working

09fe6da

with random forests and continous parameters

Ensured support for Normal and Beta parameters in the random forest

770af74

models

Forgot to remove some debugging code

f628169

Small edit - mistakenly had the wrong attribute

7e7eaea

Added support for LCB and TS as well

76c6e44

Removed debugging functikonality

4fa6f1b

Removed test example for priors

e74a935

Changed the prior parameters from acquisition function kwargs to

a35d6fe

prior_optimization_kwargs

Renamed prior_optimization_kwargs to user_prior_kwargs to be a bit less

bf468b8

verbose

Renamed optimize_with_priors to user_priors to be consistent

f42d516

Added an example of how to use PiBO to optimize with priors on the

e7af657

hyperparameters

Some small changes - removed unnecessary line of code, fixed bug in

4d21f0c

example and removed finished TODOs

Added skeleton for testing acquisition function

dd57f17

Added a bunch of compute_prior unit tests and testing infrastructure for

a387930

PriorAcquisitionFunction

Added all the test compute-funcitonality for PriorAcquisitionFunction

f4428ad

Added test for TS to cover the rescaling possibility

b057b38

Finished with unittests for PriorAcquisitionFunction

2b98dc4

Removed get_pdf_bounds - no longer needed since it is now solved solely

3aa2fc9

in ConfigSpace (Hyperparameter.get_max_density())

Slight cleanup in example

f4c5945

Added missing documentation to the new local search

95e677a

Added (the one) test for the new local search that also samples from the

3cf1eb3

search space based on the prior (not only the uniform one).

Changed prior-based local search so that the neighbor generation is

76ee4a8

retrieved form the uniform configspace

Changed local search for prior-based local search back

99acba0

Merge branch 'development' into pibo_dev

d820f7a

eddiebergman mentioned this pull request Mar 24, 2022

Bump ConfigSpace to 0.5 version automl/auto-sklearn#1428

Closed

renesass added 4 commits April 7, 2022 13:33

Merged update_tools

72fbc2e

Updated configspace version

69d4e77

Fixed pre-commit bugs

ae3f685

Import fixes

8379dc3

renesass suggested changes Apr 7, 2022

View reviewed changes

renesass and others added 5 commits April 7, 2022 14:33

Fixes some codefactor issues

1221a15

Updated changelog

cd57f18

Fixed initiation of PriorAcquisitionFunction

c574c14

Merge branch 'pibo_dev' of https://github.com/hvarfner/SMAC3 into pib…

e9554ab

…o_dev

Fixed more pre-commit bugs

732eb06

renesass changed the title ~~Pibo dev~~ PiBO Apr 7, 2022

Fixed isort

306149d

renesass merged commit e0e892f into automl:development Apr 7, 2022

renesass mentioned this pull request Apr 7, 2022

Version bump 1.3 #830

Merged

github-actions bot pushed a commit that referenced this pull request Apr 19, 2022

René Sass: Version 1.3 (PiBO #813, Tools #824)

82ceb7f

eddiebergman mentioned this pull request Nov 16, 2023

fix pyrfr int cast seed #1083

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

PiBO #813

PiBO #813

hvarfner commented Jan 17, 2022

benjamc commented Mar 24, 2022

eddiebergman commented Mar 24, 2022

hvarfner commented Mar 24, 2022

eddiebergman commented Mar 24, 2022

eddiebergman commented Mar 24, 2022

eddiebergman commented Mar 24, 2022 •

edited

Loading

eddiebergman commented Mar 24, 2022

hvarfner commented Apr 3, 2022

eddiebergman commented Apr 4, 2022

renesass commented Apr 4, 2022

mlindauer commented Apr 4, 2022

renesass Apr 7, 2022

renesass Apr 7, 2022

hvarfner Apr 7, 2022

hvarfner Apr 7, 2022

PiBO #813

PiBO #813

Conversation

hvarfner commented Jan 17, 2022

PiBO - BO with user Priors over the Optimum

New attempt on PiBO - here are the changes from last time:

New Added unit tests for PriorAcquisitionFunction (and required supporting classes)

Added unit test for LocalAndSortedPriorRandomSearch

Removed the need to sample the prior to get the co-domain of the prior

Changed input kwargs in facade

Confirmed issue combining inactive with Normal / Beta hyperparameters

Added beauitified example usage

And of course, right branch this time

benjamc commented Mar 24, 2022

eddiebergman commented Mar 24, 2022

hvarfner commented Mar 24, 2022

eddiebergman commented Mar 24, 2022

eddiebergman commented Mar 24, 2022

eddiebergman commented Mar 24, 2022 • edited Loading

eddiebergman commented Mar 24, 2022

hvarfner commented Apr 3, 2022

eddiebergman commented Apr 4, 2022

renesass commented Apr 4, 2022

mlindauer commented Apr 4, 2022

renesass Apr 7, 2022

Choose a reason for hiding this comment

renesass Apr 7, 2022

Choose a reason for hiding this comment

hvarfner Apr 7, 2022

Choose a reason for hiding this comment

hvarfner Apr 7, 2022

Choose a reason for hiding this comment

eddiebergman commented Mar 24, 2022 •

edited

Loading