Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Reduce number of initialization evaluations if external data is supplied #137

Merged
merged 10 commits into from
Nov 9, 2023

Conversation

AngelFP
Copy link
Member

@AngelFP AngelFP commented Nov 7, 2023

Currently, it is not possible to skip the initialization step of the Ax generators. This is not convenient, because it means that the first trials suggested by the generator will always be quasi-random, even if the user supplied external data to initialize the model. This PR fixes this issue by reducing the number of Sobol trials if external data is provided by the user.

There were several options to implement this:

  1. Allow the user to set n_init=0: In this case, we would simply not add a Sobol step to the AxClient. However, this is not safe in all cases, because the BO step requires some initial data to be able to start. Also, it you want to resume a previous exploration, you would need to change the input script to set n_init=0 to avoid rerunning the Sobol step.
  2. Do not add a Sobol generation step if external data is given: Currently not possible, because the AxClient is initialized when creating the generator, and this is done before the exploration object exists. Therefore, the generator cannot know in advance whether external data will be loaded. This could of course be reimplemented so that the AxClient is only initialized once the Exploration starts, but this would break a current workflow in which we allow the user to access the AxClient of the generator.
  3. Always have a Sobol step with n_init>0, and only reduce the number of Sobol trials once external data is given: This solves the issue with option 1. If a user resumes an exploration by running the same original script, the optimization will continue from where it had stopped, without generating Sobol trials again. It also maintains compatibility with the workflow mentioned in 2 (the user can get a reference to the AxClient in the input script as soon as the generator is initialized). This approach is also more flexible, as it does not require the user to decide in advance whether external data will be given. This is useful for interactive optimization on a notebook (e.g., with n_init=6, the user can first run 2 Sobol trials, then add an external one, then continue with the remaining 3 Sobol trials). This was the implemented option.

@AngelFP AngelFP added the enhancement New feature or request label Nov 8, 2023
Copy link
Collaborator

@MaxThevenet MaxThevenet left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks great!

@MaxThevenet MaxThevenet merged commit 1e8c782 into main Nov 9, 2023
10 checks passed
@AngelFP AngelFP deleted the feature/adjust_n_init branch November 9, 2023 12:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants