Handling of _all field in indices migrated from 5.x to 6.0

[imotov]

In #22144 we disabled _all field by default, which means that that unless _all was explicitly enabled in a 5.x mapping, it will be disabled in 6.0 after upgrade. In other words, 5.x indices with _all explicitly enabled will continue to work in 6.0 as before, while indices with _all not explicitly enabled will lose _all fields and will start to use our fallback mechanism for all fields queries.

After discussions with @archanid and @dakrone we couldn't come to a clear conclusion on how this situation should be handled. We see at least 2 options here.

1. It can be fixed by updating all existing 5.x mappings to have _all explicitly enabled when the first 6.0 node shows up in the cluster.

2. Alternatively we can just update our documentation that doesn't seem to explicitly say what happens to the existing mappings.

[clintongormley]

This is a bug. 5.x indices should keep working in the same way as before when the cluster is upgraded to 6.x. This means that _all should continue to be enabled by default for indices created in 5.x.

I'd also be OK with upgrading the 5.x mappings to make _all explicitly enabled, rather than introducing version logic into the default.

[rhoboat]

@clintongormley @dakrone Can you help fill out the table? Given the 5.x scenario on the left, what should happen when they upgrade, and should we raise a deprecation issue in the migration tool?

5.x setting	6.0?	Deprecation Issue?
"_all": { "enabled": false }	no setting	no
"_all": { "enabled": true }	"_all": { "enabled": true }	yes
nothing or improperly configured	"_all": { "enabled": true }	yes

[clintongormley]

@archanid i've updated the table

[rhoboat]

thanks @clintongormley Just to clarify, what I meant by "improperly configured", and I should have been more clear, is "_all": { } which is a possible thing. After talking with @imotov and @dakrone this morning we think that case is the same as "nothing". We'll still want to keep the "enabled": true functionality if the index was created in 5.x.

[rhoboat]

🎉