HPOBench

HPOBench is a library for hyperparameter optimization and black-box optimization benchmark with a focus on reproducibility.

Note: HPOBench is under active construction. Stay tuned for more benchmarks. Information on how to contribute a new benchmark will follow shortly.

Note: If you are looking for a different or older version of our benchmarking library, you might be looking for HPOlib1.5

In 4 lines of code

Run a random configuration within a singularity container

from hpobench.container.benchmarks.ml.xgboost_benchmark import XGBoostBenchmark
b = XGBoostBenchmark(task_id=167149, container_source='library://phmueller/automl', rng=1)
config = b.get_configuration_space(seed=1).sample_configuration()
result_dict = b.objective_function(configuration=config, fidelity={"n_estimators": 128, "subsample": 0.5}, rng=1)

All benchmarks can also be queried with fewer or no fidelities:

from hpobench.container.benchmarks.ml.xgboost_benchmark import XGBoostBenchmark
b = XGBoostBenchmark(task_id=167149, container_source='library://phmueller/automl', rng=1)
config = b.get_configuration_space(seed=1).sample_configuration()
result_dict = b.objective_function(configuration=config, fidelity={"n_estimators": 128,}, rng=1)
result_dict = b.objective_function(configuration=config, rng=1)

Containerized benchmarks do not rely on external dependencies and thus do not change. To do so, we rely on Singularity (version 3.5).

Further requirements are: ConfigSpace, scipy and numpy

Note: Each benchmark can also be run locally, but the dependencies must be installed manually and might conflict with other benchmarks. This can be arbitrarily complex and further information can be found in the docstring of the benchmark.

A simple example is the XGBoost benchmark which can be installed with pip install .[xgboost]

from hpobench.benchmarks.ml.xgboost_benchmark import XGBoostBenchmark
b = XGBoostBenchmark(task_id=167149)
config = b.get_configuration_space(seed=1).sample_configuration()
result_dict = b.objective_function(configuration=config, fidelity={"n_estimators": 128, "subsample": 0.5}, rng=1)

Installation

Before we start, we recommend using a virtual environment. To run any benchmark using its singularity container, run the following:

git clone https://github.com/automl/HPOBench.git
cd HPOBench 
pip install .

Note: This does not install singularity (version 3.5). Please follow the steps described here: user-guide.

Available Containerized Benchmarks

Benchmark Name	Container Name	Container Source	Hosted at	Additional Info
XGBoostBenchmark	xgboost_benchmark	library://phmueller/automl/xgboost_benchmark	Sylabs	Works with OpenML task ids
SupportVectorMachine	svm_benchmark	library://phmueller/automl/svm_benchmark	Sylabs	Works with OpenML task ids
BNNOnToyFunction	pybnn	library://phmueller/automl/pybnn	Sylabs
BNNOnBostonHousing	pybnn	library://phmueller/automl/pybnn	Sylabs
BNNOnProteinStructure	pybnn	library://phmueller/automl/pybnn	Sylabs
BNNOnYearPrediction	pybnn	library://phmueller/automl/pybnn	Sylabs
CartpoleFull	cartpole	library://phmueller/automl/cartpole	Sylabs	Not deterministic
CartpoleReduced	cartpole	library://phmueller/automl/cartpole	Sylabs	Not deterministic
Learna	learna_benchmark	library://phmueller/automl/learna_benchmark	Sylabs	Not deterministic
MetaLearna	learna_benchmark	library://phmueller/automl/learna_benchmark	Sylabs	Not deterministic
SliceLocalizationBenchmark	tabular_benchmarks	library://phmueller/automl/tabular_benchmarks	Sylabs	Loading may take several minutes
ProteinStructureBenchmark	tabular_benchmarks	library://phmueller/automl/tabular_benchmarks	Sylabs	Loading may take several minutes
NavalPropulsionBenchmark	tabular_benchmarks	library://phmueller/automl/tabular_benchmarks	Sylabs	Loading may take several minutes
ParkinsonsTelemonitoringBenchmark	tabular_benchmarks	library://phmueller/automl/tabular_benchmarks	Sylabs	Loading may take several minutes
NASCifar10ABenchmark	nasbench_101	library://phmueller/automl/nasbench_101	Sylabs	Loading may take several minutes
NASCifar10BBenchmark	nasbench_101	library://phmueller/automl/nasbench_101	Sylabs	Loading may take several minutes
NASCifar10CBenchmark	nasbench_101	library://phmueller/automl/nasbench_101	Sylabs	Loading may take several minutes
Cifar10NasBench201Benchmark	nasbench_201	library://phmueller/automl/nasbench_201	Sylabs	Loading may take several minutes
Cifar100NasBench201Benchmark	nasbench_201	library://phmueller/automl/nasbench_201	Sylabs	Loading may take several minutes
Cifar10ValidNasBench201Benchmark	nasbench_201	library://phmueller/automl/nasbench_201	Sylabs	Loading may take several minutes
ImageNetNasBench201Benchmark	nasbench_201	library://phmueller/automl/nasbench_201	Sylabs	Loading may take several minutes
NASBench1shot1SearchSpace1Benchmark	nasbench_1shot1	library://phmueller/automl/nasbench_1shot1	Sylabs	Loading may take several minutes
NASBench1shot1SearchSpace2Benchmark	nasbench_1shot1	library://phmueller/automl/nasbench_1shot1	Sylabs	Loading may take several minutes
NASBench1shot1SearchSpace3Benchmark	nasbench_1shot1	library://phmueller/automl/nasbench_1shot1	Sylabs	Loading may take several minutes
ParamNetAdultOnStepsBenchmark	paramnet	library://phmueller/automl/paramnet	Sylabs
ParamNetAdultOnTimeBenchmark	paramnet	library://phmueller/automl/paramnet	Sylabs
ParamNetHiggsOnStepsBenchmark	paramnet	library://phmueller/automl/paramnet	Sylabs
ParamNetHiggsOnTimeBenchmark	paramnet	library://phmueller/automl/paramnet	Sylabs
ParamNetLetterOnStepsBenchmark	paramnet	library://phmueller/automl/paramnet	Sylabs
ParamNetLetterOnTimeBenchmark	paramnet	library://phmueller/automl/paramnet	Sylabs
ParamNetMnistOnStepsBenchmark	paramnet	library://phmueller/automl/paramnet	Sylabs
ParamNetMnistOnTimeBenchmark	paramnet	library://phmueller/automl/paramnet	Sylabs
ParamNetOptdigitsOnStepsBenchmark	paramnet	library://phmueller/automl/paramnet	Sylabs
ParamNetOptdigitsOnTimeBenchmark	paramnet	library://phmueller/automl/paramnet	Sylabs
ParamNetPokerOnStepsBenchmark	paramnet	library://phmueller/automl/paramnet	Sylabs
ParamNetPokerOnTimeBenchmark	paramnet	library://phmueller/automl/paramnet	Sylabs
ParamNetVehicleOnStepsBenchmark	paramnet	library://phmueller/automl/paramnet	Sylabs
ParamNetVehicleOnTimeBenchmark	paramnet	library://phmueller/automl/paramnet	Sylabs

Further Notes

How to build a container locally

With singularity installed run the following to built the xgboost container

cd hpobench/container/recipes/ml
sudo singularity build xgboost_benchmark Singularity.XGBoostBenchmark

You can use this local image with:

from hpobench.container.benchmarks.ml.xgboost_benchmark import XGBoostBenchmark
b = XGBoostBenchmark(task_id=167149, container_name="xgboost_benchmark", 
                     container_source='./') # path to hpobench/container/recipes/ml
config = b.get_configuration_space(seed=1).sample_configuration()
result_dict = b.objective_function(config, fidelity={"n_estimators": 128, "subsample": 0.5})

Remove all caches

HPOBench data

HPOBench stores downloaded containers and datasets at the following locations:

$XDG_CONFIG_HOME # ~/.config/hpobench
$XDG_CACHE_HOME # ~/.cache/hpobench
$XDG_DATA_HOME # ~/.local/share/hpobench

For crashes or when not properly shutting down containers, there might be socket files left under /tmp/.

OpenML data

OpenML data additionally maintains it's own cache which is located at ~/.openml/

Singularity container

Singularity additionally maintains it's own cache which can be removed with singularity cache clean

Troubleshooting

• Singularity throws an 'Invalid Image format' exception Use a singularity version > 3. For users of the Meta-Cluster in Freiburg, you have to set the following path: export PATH=/usr/local/kislurm/singularity-3.5/bin/:$PATH

• A Benchmark fails with SystemError: Could not start a instance of the benchmark. Retried 5 times but the container can be started locally with singularity instance start <pathtocontainer> test See whether in ~/.singularity/instances/sing/$HOSTNAME/*/ there is a file that does not end with '}'. If yes delete this file and retry.

Status

Status for Master Branch:

Status for Development Branch: