Alternative termination features other than max iteration?

[akmandor]

Are there other features that we can use to terminate the optimization other than setting max iteration number (set by 'runcount-limit' in Scenario)?

For example, termination when the difference between expected (estimate) and given value of the function is less than some threshold for some number of iterations.

[renesass]

That would be indeed beneficial. I will integrate this feature in our upcoming Version 2.0.
I'm thinking of the following:

• Scenario gets a new argument called termination_cost_threshold
• SMAC checks every iteration if the incumbent passed the threshold and gracefully terminates

Does this work for you?

[renesass]

I just implemented it:
c4823e0

[eddiebergman]

I'll also add that you can pass custom callbacks to SMAC with IncorporateRunResultsCallback. I'm pretty sure there's no example, the issue to create one #823 was closed a bit early during a stale-bot sweep.

You can use this test as a reference, I also implemented a really dumb version early stopping you can use if you need, with an example in this PR in auto-sklearn but you would have to wrap it in IncorporateRunResultsCallback yourself.

[eddiebergman]

From @renesass in the other issue, you can use what I suggested for now but this will change significantly in the next release by the looks of things.

Check this out:
https://github.com/automl/SMAC3/blob/development-2.0/smac/callback.py

Integrated the callbacks here:
https://github.com/automl/SMAC3/blob/development-2.0/smac/smbo.py

Docs is also (nearly) ready:
https://automl.github.io/SMAC3/development-2.0/advanced_usage/callbacks.html

[akmandor]

• First of all, thank you so much @renesass for providing the solution in such a short time!

• I checked out to the development-2.0 branch to test the functionality. However I noticed that the structure of the v2 is considerably different from the current main. I could not run the given Minimal Example.

• I also tried to run the same example in development-2.0 branch but it seems like it has not updated yet.

• I also tried synthetic function example in development-2.0` branch but it also gave me importing errors as such:

Traceback (most recent call last):
  File "smac_example2.py", line 15, in <module>
    from smac import BlackBoxFacade, Scenario
  File "/home/nesetunver.akmandor/.local/lib/python3.8/site-packages/smac/__init__.py", line 26, in <module>
    from smac.facade import BlackBoxFacade, HyperparameterFacade, MultiFidelityFacade, AlgorithmConfigurationFacade
  File "/home/nesetunver.akmandor/.local/lib/python3.8/site-packages/smac/facade/__init__.py", line 1, in <module>
    from smac.facade.facade import Facade
  File "/home/nesetunver.akmandor/.local/lib/python3.8/site-packages/smac/facade/facade.py", line 11, in <module>
    from smac.acquisition.functions.abstract_acquisition_function import AbstractAcquisitionFunction
  File "/home/nesetunver.akmandor/.local/lib/python3.8/site-packages/smac/acquisition/__init__.py", line 1, in <module>
    from smac.acquisition.abstract_acqusition_optimizer import AbstractAcquisitionOptimizer
  File "/home/nesetunver.akmandor/.local/lib/python3.8/site-packages/smac/acquisition/abstract_acqusition_optimizer.py", line 15, in <module>
    from smac.utils.stats import Stats
  File "/home/nesetunver.akmandor/.local/lib/python3.8/site-packages/smac/utils/stats.py", line 20, in <module>
    from smac.scenario import Scenario
ImportError: cannot import name 'Scenario' from 'smac.scenario' (/home/nesetunver.akmandor/.local/lib/python3.8/site-packages/smac/scenario/__init__.py)

• Could provide me a simple example (like the Minimal Example) that I can run quickly to test the new functionality?

[renesass]

The documentation still has to be updated drastically. But you could use following examples already:

https://github.com/automl/SMAC3/blob/development-2.0/examples/mlp_multi_fidelity.py
https://github.com/automl/SMAC3/blob/development-2.0/examples/sgd_instances.py

or also the other examples but you have to set deterministic in the scenario to false (a bug i just discovered...).
Please make sure to install SMAC v2.0 again (I recommend using another environment and python 3.10) as we use ConfigSpace 0.6 and Pynisher 1.0, both of which are not released yet.

pip install -e .[dev]