org.elasticsearch.cluster.metadata.MetadataIndexAliasesService#applyAliasActions can become very slow when adding aliases to large data streams

[original-brownbear]

When adding aliases that include filters to data streams we validate the filter for every index in the data stream. This entails instantiating a temporary index service for every index (at least for every index not on the master node) which in turn means parsing each index's mapping and setting up a mapper instance.
This can take many seconds for larger data streams. It seems we could avoid validating the filter for every index here and just validate for every unique mapping only?

relates #89924 and #77466

[elasticsearchmachine]

Pinging @elastic/es-data-management (Team:Data Management)

[DaveCTurner]

I'm guessing that making a whole new Metadata each time round the loop, rebuilding the index abstraction lookup map, is also not very cheap?

[original-brownbear]

I'm guessing that making a whole new Metadata each time round the loop, rebuilding the index abstraction lookup map, is also not very cheap?

You'd think so but parsing mappings for something like Beats mappings is so absurdly expensive that rebuilding a 5k indices Metadata is probably still less expensive than parsing the mappings once. It makes sense if you think about it, parsing a mapping with 4k fields probably means something like 10k map puts etc. :)

[DaveCTurner]

Sure that makes sense for the case we looked at, but the metadata-building cost scales with the total number of indices/aliases in the cluster, not just the target data stream, and in another situation we could have 10x or more total indices. We do see some folk using aliases very heavily.

[original-brownbear]

Right, that makes sense. If you have simple mappings you might bottle neck more on the metadata rebuilding. Though that has been way optimized lately while mapping parsing hasn't and I think rebuilding a 50k indices/DS/aliases metadata was about as expensive as parsing one Beats mapping.