-
Notifications
You must be signed in to change notification settings - Fork 26
Newbie Guide
This page provides general guidelines on how to best use HyperMapper on a new application. These guidelines are answers to common questions new users have when adopting HyperMapper. We assume users have already installed HyperMapper in this guide. Contact us if you have doubts that are not covered on this page.
The first step to optimize a function with HyperMapper is to define your search space and your budget. The search space is the input parameters that you wish to optimize with HyperMapper, in this guide, we will use D to refer to the number of input parameters being optimized. In this guide, we will also call a combination of input parameter values a "configuration". The budget is for how long you are willing to run the optimization, usually defined by how many function evaluations you are willing to perform or how many hours you are willing to wait.
For example, you may want to optimize the hyperparameters of a neural network in order to find the hyperparameters that lead to the lowest loss. Let's say you are optimizing the learning rate, batch size, and the number of hidden layers, in this case, you have D = 3 input parameters. Also, let's say the network takes approximately 5 minutes to train and we are willing to wait 5 hours for the optimization, then our budget is 2 hours or approximately 60 function evaluations.
Next, you have to decide on which optimization method to use. HyperMapper supports three optimization methods: Bayesian Optimization, Local Search, and Evolutionary Optimization. You can find more details on each optimization method on this page. The choice of which optimization method to use depends on the number of your budget. We recommend using Bayesian optimization (bayesian_optimization) if your budget is small, e.g. in the order of a few thousands of function evaluations, and either the local search (local_search) or evolutionary optimization (evolutionary_optimization) otherwise. This is defined in the json with the optimization_method field:
"optimization_method" = "bayesian_optimization"
By default, HyperMapper will use Bayesian optimization.
HyperMapper's Bayesian optimization method has a random initialization phase called Design of Experiment (DoE) phase. In this phase, a number of configurations are randomly chosen and evaluated. You can control how many samples are sampled and how this sampling is done in the json file with the design_of_experiment field:
"design_of_experiment": {
"doe_type": "random sampling"
"number_of_samples": 10
}
HyperMapper supports random sampling and standard latin hypercube as DoE types. We recommend using standard latin hypercube if your search space contains more than 10 input parameters and random sampling otherwise. For the number_of_samples, a good rule of thumb is to use D+1 samples if your search space contains less than 5 input parameters and 10D samples otherwise. However, avoid spending more than 25% of your budget in the DoE phase, if 10D samples are more than 25% of your budget, use 25% of your budget or D+1 instead. We discourage using less than D+1 samples for DoE in all cases.
HyperMapper supports Gaussian Processes (gaussian_process) and Random Forests (random_forest) as models for Bayesian Optimization. The model can be selected with the optimization_model field:
"models": {
"model": "gaussian_process"
}
Gaussian Processes are more effective if the search space contains up to 10 input parameters and is continuous, i.e., contains only real parameters, for larger or discrete search spaces, the Random Forests model is recommended.
HyperMapper can interface with other languages using its Client-Server mode. See the Client-Server mode page for details and examples for Python, C++, and Scala.
HyperMapper provides a number of scripts that allow users to visualize the results of optimization. Users can plot regret for mono-objective functions, to see the performance of optimization and also compare the performance for different approaches. For multi-objective functions, users can compute and plot the Pareto fronts, to visualize a trade-off between optimization objectives.
HyperMapper provides a few advanced features that can be used to make the search more efficient.
-
Constrained Search Spaces. HyperMapper can be used to optimize functions where some configurations in the search space are invalid. For instance, you may be interested in optimizing the runtime of a neural network without dropping the accuracy below a threshold, any point that leads to accuracy below the threshold is considered to be invalid (or "infeasible", as we call them). For these search spaces, you can use HyperMapper's feasibility predictor, which leads HyperMapper to prioritize feasible configurations during optimization. See this page, for an example application with feasibility constraints.
-
Prior Injection. HyperMapper allows users to inject their knowledge into the optimization process. This is done via a prior distribution that informs where in the input space the user expects to find "good" function values. The prior helps guide optimization towards "good" regions of the space, speeding up convergence. Even if the prior is wrong, HyperMapper is robust and will still find "good" performing configurations. See this page, for details on prior injection.
We recommend users avoid discrete parameters with hundreds of possible values in their ranges. From our past experience, this type of parameter leads to degraded performance, further, it is often possible to prune the possible values of these parameters to facilitate optimization without losing much in terms of performance. Pruning leads to more thoughtful design of the search space and consequently better optimization. However, pruning is not possible in some applications and then discrete parameters with large ranges are required. In these cases, generating the JSON configuration file manually can become cumbersome, instead, we recommend using a script to automatically generate the JSON file. We provide an example on how to do that here.
HyperMapper writes all execution information to a "hypermapper_logfile.log" file. In this file, you will find a log of the output of HyperMapper as well as additional internal information. If you run into any issues while running HyperMapper, you can check this file for more information. Additionally, you can ask HyperMapper to print the importance of each input parameter for optimization in this file, by using the print_parameter_importance field, see this page for more information. The most common errors new users run into are: