[Feature] Hook to implement early stopping

[thomashlvt]

Short Question Description

I would like to implement a hook for the user to be able to implement his own stopping strategy. Is this interesting for you? How would I go about implementing the hook myself?

Context Information

First off, I really like the project and I'm very much impressed with what you have accomplished. The autoML engine works extremely well for our use cases.

Different datasets require different training lengths. In some cases, I noticed that the autoML engine already finds the most optimal configuration in a matter of seconds, whereas for others, longer training times do benefit the performance. Without knowing the dataset in advance, it is hard to find the optimal training time - we are trying to minimize computation time while keeping the same model performance.

This could be done by providing a hook to the user that is called after every new model is trained. I would then check the new model's performance with the best one so far, and make a heuristic decision to continue training based on this. The time_left_for_this_task would still be the maximum training time - the hook would thus implement some sort of early stopping strategy.

Similar Work

I did not find a similar example/tutorial in the documentation, nor a similar GitHub Issue.

[janhenriklambrechts]

I agree that a time_left_for_this_task-hook would be extremely useful

[eddiebergman]

Hi @thomashlvt,

I think this would be an excellent feature especially if it's completely optional. I will have a look tomorrow and point you to the relevant places to consider.

There are some challenges to this though which essentially means your stopping criteria won't fully reflect the final performance.

• Autosklearn searches for a list of candidate models for the allotted time.
• It then creates an ensemble from the candidates models, once time has elapsed.
• This ensemble is the final product which is used for prediction

The early-stopping criteria would only be able to consider individual model performance and metrics, with no information of how strong an ensemble created at that point would be. To create an ensemble after each model evaluation would be too costly.

If you think this would be enough for your feature idea, here are some extra question so I can give you the information you would need!

• Are you familiar with the internal code of autosklearn?
• Have you contributed to open-source github projects before?

We are working on a contribution guide which should be here soon and should help!

[eddiebergman]

Hi @thomashlvt,

After looking at how one might implement this and talking to some colleagues, it appears it's not as easy as it seems. Autosklearn relies on a library called SMAC for performing the model search. The two ways to currently stop SMAC are:

• When the budget is exhausted
• One of the target runs returned STOP

I agree this use case seems to be something we should support so if you were very keen to get this feature working we would be very appreciative! We'd also be happy to help you work through this. I've written about a possible solution to do so.

Solution

There is a mechanism within autosklearn to pass callbacks to SMAC with the argument get_trials_callback in the __init__() methods of autosklearn. Currently this callback is called in SMAC but does not use any return values from the callback. This prevents you from passing information to SMAC during its running.

Therefore, the solution I see would be to have SMAC look at the return values of the callbacks and use this in some way to either tell SMAC to continue or stop.

[thomashlvt]

Hi @eddiebergman -

I'm happy you're also excited about this feature!

Are you familiar with the internal code of autosklearn?
Our team and I have been using it for the past 2 months, so I'm familiar with the API and docs. I've also read the autosklearn2.0 paper. I have taken a glance at the internal code, but cannot really say I'm entirely familiar, however, I think I'll find my way given some pointers.

Have you contributed to open-source github projects before?
I have not, but I'm very keen to make a first contribution! I do have experience contributing to large (closed) codebases through some internships (on GitLab though).

Solution
Thanks for the pointers!

I can start working on a PR in the SMAC3 repository.

What interface do you suggest?

• Would you simply return a boolean to indicate whether the optimization loop should be interrupted (or vica versa)? For backwards compatibility returning None also should not stop the optimization loop.
• Or would you create some StopSMBOException that the callback can raise to interrupt the loop?

I think interrupting the loop itself can be easily done by setting the self._stop flag.

Thanks!

[thomashlvt]

I have created a PR in SMAC :)

[eddiebergman]

Ello again @thomashlvt, glad to see you could get it done so quickly!

I think the boolean method you implemented should be fine but I am not familiar with the full development of SMAC.
I left my own little review there to spark some involvement from the SMAC side of things but we would have to wait until someone more involved approved the changes.

You should be able to work off that branch quite safely without any API changes to test an early stopping mechanism with AutoSklearn.

Some more pointer about what I think would make an excellent PR!

Implementation

I'm not sure anything would actually be need to put into AutoSklearn as the callbacks are already forwarded. If you do however it would make sense to put it into the AutoML class.

Unit Tests

For creating unit tests, you could put that in test/test_automl.py

• A unit test could be done very similar to how you currently have it done in SMAC.
• A unit test that documents what occurs when the callback returns false after the first call. My assumption would be that type(AutoML.ensemble_) == SingleBest, indicating that only a single model was trained. This assumption may be wrong however a unit test to discover this would be nice so we can document its behaviour. (Single Best)
• A unit test that performs your intended purposes would be nice but it would be hard to verify.
  • Assuming you stop when a model accuracy hits .9, this is not reflective of the actual performance of the ensemble created by autosklearn once model searching has stopped. The performance of the ensemble should stay above .9 based on the greedy ensembling approach we use now but if we change this method in the future, the test would have to be rethought. The only other alternative I can think of is finding the model that triggered the early stopping and then testing that the model has a performance over .9. This is possible but convoluted for two reasons
    • SMAC's id and the id we give to models are not always aligned (we hope to fix this soon)
    • Getting models by an id is not the most straight forward at the moment, you could use AutoML.models_ which is a dict { (seed, model_id, budget) : model }

Documentation

We would need some documentation on this functionality

• In the SklearnEstimator class arguments
• Seeing as this feature would be hidden in the docstring of an argument, I think it makes sense to document this with a section in manual.rst and an Advanced example runnable by sphinx-gallery that showcases your use case.
• You can look at this example if you're unfamiliar with how to write an example that can be run with sphinx-gallery. Note that the example must be prefixed with example_X.py.
  • To build doc and test

  cd doc
  make html  # Will take a while as it runs examples and generates html
  xdg-open ./build/html/index.html  # View in browser and see that it's there

  We are working on a contribution guide which should document this for the future!

Please let me know if there's any more information you need!

As a side note, I think this feature will actually make testing some other components of autosklearn easier, being able to interrupt it as required :)

[thomashlvt]

Hi there! Thanks for the additional pointers, I'm looking forward to trying to make this work :)

I indeed think, given the change in SMAC, that there have to be no additional changes to make this work, besides actually implementing the callback.

Would you suggest to also put this callback into the actual codebase and expose it in the API somehow, or only test it and document it in the examples?

[eddiebergman]

I think for clarity sake, it might make sense to expose a specific parameter, something like early_stopping_callback in our interface. While functionally it's no different from the current parameter to pass callbacks, it's a lot clearer as an entry point to using this functionality. As to where this goes, we will do some rework of the main classes AutoML, AutoMLClassifier, AutoMLRegressor, AutoSklearnEstimator, AutoSklearnClassifier, AutoSklearnRegressor but I would imagine it needs to be present in each of them.

The last decision is then whether it should be an argument to fit or __init__(). I would prefer __init__ so more is known at object construction but it might make more contextual sense to put it in fit. @mfeurer would you have any additional thoughts on this?

[mfeurer]

While functionally it's no different from the current parameter to pass callbacks, it's a lot clearer as an entry point to using this functionality.

Wouldn't it maybe suffice to rename the current argument and improve its documentation?

[eddiebergman]

While functionally it's no different from the current parameter to pass callbacks, it's a lot clearer as an entry point to using this functionality.

Wouldn't it maybe suffice to rename the current argument and improve its documentation?

That's a much better idea! I think having the dedicated doc section in SMAC that we can link to would make it a lot easier. A simple description of the callback here, one or two use cases and then simply refer to SMAC documentation for more details.

[eddiebergman]

Raising this in a new issue so we can clearly state what needs to be done on our end.
Please see #1304