Halt ML jobs before node restart

[anyasabo]

Opening an issue for the outstanding TODO to halt ML jobs before we restart a node here. This is described in the official rolling upgrade ES docs:
https://www.elastic.co/guide/en/elasticsearch/reference/current/rolling-upgrades.html

[racevedoo]

Hey @anyasabo, I can try to take on this one. So basically we should call the POST _ml/set_upgrade_mode?enabled=true, and then, at the end, we call POST _ml/set_upgrade_mode?enabled=false.

Did I miss anything?

[anyasabo]

Yep that's my understanding @racevedoo

[sebgl]

This is not on easy one, we may want to discuss it further first. Some thoughts:

• Disabling ML jobs is optional. If not done, jobs will simply be spawned on another node. This might be the behaviour we want in most/some cases? I don't know enough about how ML works to decide what's best.
• We may want the user to configure this (should we stop ML jobs during upgrade? yes/no).
• Stopping the ML jobs is easy since we know when we have a rolling upgrade to perform. Re-enabling it is a lot harder. Basically we don't know when a rolling upgrade is over (was performed before). An approach would be to call POST _ml/set_upgrade_mode?enabled=false at every single reconciliation, which feels wrong. See Don't clear shard allocation excludes at every reconciliation #1522 where we discuss a similar problem.

[pebrc]

Closing as this should be covered by the Node Shutdown API #4597