Do not get stuck due to adaptive capping

[filipbartek]

If the incumbent has all the instance-seed pairs covered
and the challenger is ruled out due to adaptive capping,
we cancel all the runs on the current challenger
to proceed to the next challenger.

This is a possibly dirty quick fix.
I am not well acquainted with the code.
I welcome suggestions for improvements of the fix.

Resolves #748

[renesass]

Thanks for your contribution! We'll look into this shortly.

[mlindauer]

Hi,

this looks good and the change makes sense in principle. However, I still don't fully see why the bug happens.
@renesass could you please run the example from @filipbartek 's issue and check that everything is fine now.

@franchuterivera do you have any comment on this? I think this bug was introduced because of the new IntensifierStage, which I'm not super familiar with.

[renesass]

I ran @filipbartek code on both the old and the new code and got the same results. However, we should investigate why this problem happens in the first place. Since more time is required to find the bugs, we can't solve it by the upcoming release.

[mlindauer]

@franchuterivera @mfeurer could you please comment on this?
I don't understand why this change is required if we do a stage change anyway.
After we got not incumbent runs, I would expect that we could consider the same challenger again, since the adaptive capping allows now for more time.

[mlindauer]

@filipbartek please redirect your PR to development. Other than new releases, there is never a PR directed to master.

[dengdifan]

Simply setting self.to_run = [] should suffice,
'self.continue_challenger' will be modified anyway under Intensification._process_racer_results (line 422)

[filipbartek]

Having tested this on an example that I attached to issue #748, I agree that setting self.to_run suffices.
I have simplified the PR accordingly.

[mfeurer]

So if I understand this correctly, the function _process_racer_results does not know that the evaluation of the current challenger should be stopped because adaptive capping mandates that (as that functionality was simply not programmed) and therefore the stage change doesn't happen properly. So what you're doing right now @filipbartek is "faking" the stop mechanism that _process_racer_results knows of. A cleaner solution would then be that we add a new argument to _process_racer_results which has the same effect as clearing self.to_run but is more descriptive. Is that correct @dengdifan?

Why the stage change wasn't sufficient is something I currently don't understand myself either :(

[dengdifan]

adding an argument to _process_racer_results could also be an option.
Alternatively, we could add a docstring saying that "since the challenger fails to outperform the incumbent due to adaptive capping, we discard all the forthcoming runs"

Why the stage change wasn't sufficient is something I currently don't understand myself either :(

Intensifier will skip the 4th line of _process_racer_results if self.to_run is not empty. As self.to_run still contains elements, intensifier will continue evaluating the remaining instance-seed pairs in self.to_run.

[codecov]

Codecov Report

Merging #749 (215e3b6) into development (c4a745e) will decrease coverage by 0.01%.
The diff coverage is 0.00%.

@@               Coverage Diff               @@
##           development     #749      +/-   ##
===============================================
- Coverage        87.02%   87.01%   -0.02%     
===============================================
  Files               68       68              
  Lines             6221     6222       +1     
===============================================
  Hits              5414     5414              
- Misses             807      808       +1     

Impacted Files	Coverage Δ
smac/intensification/intensification.py	95.43% <0.00%> (-0.37%)	⬇️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update c4a745e...215e3b6. Read the comment docs.