[ML] Prevent job creation/opening if .ml indices cannot be created

[droberts195]

In https://www.elastic.co/guide/en/elasticsearch/reference/current/zip-targz.html#zip-targz-enable-indices we document that auto-creation must be permitted for .ml* indices even if auto-creation of indices is disabled in general. Unfortunately, if a user does not follow the guidance then the resulting failure is very hard to diagnose.

We should explicitly check that auto-creation of .ml* indices is permitted at a few key points where we have the ability to fail early and report a clear error message. If we wait until results and/or state cannot be written then we cannot report the problem in a REST response.

[elasticmachine]

Pinging @elastic/ml-core

[droberts195]

The other option here is that we switch to a strategy of creating our indices without relying on the auto-create mechanism at all. Security does this:

elasticsearch/x-pack/plugin/security/src/main/java/org/elasticsearch/xpack/security/support/SecurityIndexManager.java

Line 260 in 305b584

public void prepareIndexIfNeededThenExecute(final Consumer<Exception> consumer, final Runnable andThen) {

However, there is only ever one active security index, whereas ML has several, and in future we'll want to use rollover. Rollover works best with the standard auto-create mechanism, so that points to us continuing to use auto-create.

[dimitris-athanasiou]

Don't we win this for free once we need to store the job config in an index?

[droberts195]

Don't we win this for free once we need to store the job config in an index?

I don't think it's completely free.

For the support case that caused this issue to be raised some .ml indices had been created. So I imagine they restricted auto-create after having previously run ML. The same thing could happen in the future - a user could create their first job, thus causing the .ml-config index to be created, then restrict auto-creation of indices leading to subsequent problems with ML. So it would be nice to have some other checks that can report the underlying problem with restricted auto-creation.

[davidkyle]

The .ml-anomalies-{shared|specific} index is created explicitly in JobResultsProvider.createJobResultIndex we do this because of the need to update mappings and check the number of fields in the mapping. The other indices - state, notifications, meta - are created automatically on write.

The chain of events that led to the failure is

1. Create job and the results index
2. Run job writing results and model snapshots to the results index
3. Close job causing the model state to be persisted which then fails because .ml-state could not be auto-created
4. Re-opening the job fails because there is a model snapshot doc saying model state exists but that state cannot be found

This is important to consider for the config migration change #32905. If the config cannot be written to .ml-config due to an auto-create restriction the background migration task will fail repeated. The only affect will be the spam the log files and the user won't have any feedback on how to remedy the issue. For this reason the migrator must explicitly create the index prior to migration.

[sophiec20]

closing as out of date