Default parameter value is ignored for initial design

[jendrikseipp]

Steps/Code to Reproduce

import logging

import numpy as np

from ConfigSpace.hyperparameters import UniformIntegerHyperparameter

from smac.configspace import ConfigurationSpace
from smac.scenario.scenario import Scenario
from smac.facade.smac_hpo_facade import SMAC4HPO


def evaluate_cfg(cfg):
    print(cfg)
    return None

logging.basicConfig(level=logging.INFO)

cs = ConfigurationSpace()

cs.add_hyperparameters([
    UniformIntegerHyperparameter("a", lower=0, upper=600, default_value=1),
])

scenario = Scenario({
    "run_obj": "quality",
    # max. number of function evaluations
    "runcount-limit": "inf",
    "cs": cs,
    "deterministic": "true",
    # memory limit for target algorithm (3.5 GB)
    "memory-limit": 3584,
})

default_cfg = cs.get_default_configuration()
print("Default config:", default_cfg)

print("Optimizing...")
smac = SMAC4HPO(
    scenario=scenario, rng=np.random.RandomState(42), tae_runner=evaluate_cfg)
incumbent = smac.optimize()
inc_value = evaluate_cfg(incumbent)
print("Optimized value: %.2f" % (inc_value))

Expected Results

The default value 1 is used for the initial design.

Actual Results

INFO:smac.utils.io.cmd_reader.CMDReader:Output to smac3-output_2019-09-13_08:45:09_553534
Default config: Configuration:
  a, Value: 1

Optimizing...
INFO:smac.facade.smac_hpo_facade.SMAC4HPO:Optimizing a deterministic scenario for quality without a tuner timeout - will make SMAC deterministic and only evaluate one configuration per iteration!
INFO:smac.initial_design.sobol_design.SobolDesign:Running initial design for 10 configurations
INFO:smac.facade.smac_hpo_facade.SMAC4HPO:<class 'smac.facade.smac_hpo_facade.SMAC4HPO'>
Configuration:
  a, Value: 300

For some reason the default value 1 is ignored and instead (lower+upper)/2 is used as the default value.

Versions

0.11.0

[mlindauer]

Dear Jendrik,

Thank you for reporting the issue.
I can confirm that the default configuration is ignored if the SMAC4AC facade is not used or if the initial design is not explicitly set to the default initial design (in case of the SMAC4HPO or SMAC4BO facade).

I have to discuss this with the other developers, because it will change the results on some standard benchmarks quite a bit; but my intuition is that we want to change that in the next release.

Best,
Marius

[jendrikseipp]

Thanks for the info!

[stheid]

Is there any workaround? my searchspace is very large. i would like to give smac at least a little hint by starting close to a usefull spot.

[mfeurer]

Not yet, but a pull request would be very welcome (it should add this as an option, though).

[stheid]

Workaround A

Modified sobol_design.py by prepending the default configuration

SMAC3/smac/initial_design/default_configuration_design.py

Lines 27 to 29 in 2b27e0d

	config = self.scenario.cs.get_default_configuration()
	config.origin = 'Default'
	return [config]

Workaround B

replace 'SobolDesign' with 'None'

SMAC3/smac/facade/smac_hpo_facade.py

Line 39 in 2b27e0d

kwargs['initial_design'] = kwargs.get('initial_design', SobolDesign)

Cause

I tracked down the bug. however, i am not able to predict the intended behavior.

Both Bayesian optimizers select Sobol as initial design:

SMAC3/smac/facade/smac_hpo_facade.py

Line 39 in 2b27e0d

kwargs['initial_design'] = kwargs.get('initial_design', SobolDesign)

SMAC3/smac/facade/smac_bo_facade.py

Line 56 in 2b27e0d

kwargs['initial_design'] = kwargs.get('initial_design', SobolDesign)

Therefore the initial_design is not None and the value for initial_incumbant in scenario is ignored

SMAC3/smac/facade/smac_ac_facade.py

Lines 427 to 430 in 2b27e0d

	elif initial_design is None:
	if scenario.initial_incumbent == "DEFAULT":
	init_design_def_kwargs['max_config_fracs'] = 0.0
	initial_design = DefaultConfiguration(**init_design_def_kwargs)

additionally Sobol design

SMAC3/smac/initial_design/sobol_design.py

Lines 46 to 50 in 2b27e0d

	sobol = sobol_seq.i4_sobol_generate(len(params) - constants, self.init_budget)
	return self._transform_continuous_designs(design=sobol,
	origin='Sobol',
	cs=cs)

does not use the default configuration.

I hope that helps others in tackle the issue.

[stheid]

The documentation is not very verbose on what Sobol even is.

[jendrikseipp]

I used the following workaround:

from smac.facade.smac_hpo_facade import SMAC4HPO
from smac.initial_design.default_configuration_design import DefaultConfiguration
smac = SMAC4HPO(
    initial_design=DefaultConfiguration,
   ...
)

[stheid]

Has basically the same effect as Workaround B, but yours is obvoiusly better.

[stheid]

@mfeurer is it recommended to disable sobol like #533 (comment)

[mlindauer]

It depends on what you want to achieve.

The HPO-facade uses a sobol sequence but in turn, it also uses less interleaved random configurations. Thus, if you disable the sobol sequence (workaround b) and you still use the HPO facade, SMAC will be more greedy compared to its default settings.

Workaround A is most likely what you want in most cases.

[stheid]

Thanks for the information.

workaround A leads SMAC to prematurely stop after it finished its n+1 initial configurations (while n is the number SOBOL calcualtes from the time budget)

so for 100 runs, it will stop after 25+1=26

[mfeurer]

@gamer01 could you please open a new issue on SMAC stopping prematurely with code to reproduce this?

[stheid]

It only happens for my modified version. Therefore i would not deem it a bug in smac. (The Initializer seems to assume it has 25 elements, as it has 26 now, it stops)

[stheid]

solved by skiping one sobol element

MySobol.py.txt

from MySobol import SobolDesign
smac = SMAC4HPO(scenario=scenario, tae_runner=tae, initial_design=SobolDesign)

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[timruhkopf]

Good things may take a while, but with our latest release (2.0), this becomes an easy-to-fix issue:

simply add to the <facade>.get_initial_design call (e.g. in this SVM example) the default config as an additional_configs like this:

classifier = SVM()

[....]

 initial_design = HyperparameterOptimizationFacade.get_initial_design(
        scenario,
        n_configs=5,
        additional_configs=[  # to get the default configuration in the initial design
            classifier.configspace.get_default_configuration()
        ]
    )

This will result in SMAC executing the n_configs from the initial design and, subsequently, the default config you specified.

Notice, that it is still not default behaviour to execute the configspace default (if available)

[timruhkopf]

At long last, this is now an optional feature; see PR #995.

Setting Scenario([...], use_default_config=True) will make use of the default config. but it is not the default for legacy reasons, particularly because of the benchmarks, which would be solved by the default config as per Marius' comment above:

Dear Jendrik,

Thank you for reporting the issue. I can confirm that the default configuration is ignored if the SMAC4AC facade is not used or if the initial design is not explicitly set to the default initial design (in case of the SMAC4HPO or SMAC4BO facade).

I have to discuss this with the other developers, because it will change the results on some standard benchmarks quite a bit; but my intuition is that we want to change that in the next release.

Best, Marius