Remove metadata customs that can break serialization #30945
Conversation
This commit removes ML and persistent task custom metadata from the cluster state API to avoid the possibility of sending such custom metadata to a client that can not understand. This can arise in a rolling upgrade to the default distribution from a prior version that did not have X-Pack.
jasontedor
added
>breaking
blocker
review
discuss
:Distributed Coordination/Task Management
|
Pinging @elastic/ml-core |
|
Pinging @elastic/es-distributed |
| @Override | ||
| public Version getMinimalSupportedVersion() { | ||
| return Version.V_5_4_0; | ||
| return MINIMAL_SUPPORTED_VERSION; |
There was a problem hiding this comment.
are we sure that's OK? this is also used when publishing a cluster state to nodes. What happens when people upgrade from 6.2 + xpack to 6.3 + xpack in a rolling fashion?
There was a problem hiding this comment.
The effect of losing the persistent tasks in a 6.2 -> 6.3 upgrade would be that running ML jobs and datafeeds would be killed and would have to be manually restarted.
It's currently documented best practice to stop ML jobs during upgrades. We are trying to make things more robust so this advice can be removed, but because the advice exists today I don't think it would be a complete disaster if ML jobs got killed during a rolling upgrade to 6.3 if it wasn't followed. (For upgrades beyond 6.3 this wouldn't be acceptable, as ML will be supported in Cloud and they'll be relying on ML jobs staying open during rolling upgrades from 6.3 to higher versions.)
We could also potentially document the es.persistent_tasks.custom_metadata_minimum_version system property and tell people who want to leave ML jobs running during a rolling upgrade to 6.3 that they need to set it.
|
We are going to end up taking so different approaches. |
This commit removes ML and persistent task custom metadata from the cluster state API to avoid the possibility of sending such custom metadata to a client that can not understand. This can arise in a rolling upgrade to the default distribution from a prior version that did not have X-Pack.
Relates #30731, relates #30857