[Lens] Avoid index pattern remapping for each visualization

[flash1293]

In our current setup, the datasource initializes itself when a Lens visualization is loaded on a dashboard. This includes calling loadIndexPatterns, which loads the used index patterns and builds a map for all existing fields. As this is happening for each visualization individually, a large dashboard built with large index patterns can cause performance issues:

• Lots of memory used for storing the same field maps/lists
• Lots of time spent mapping and indexing the field lists

There are two options how we can improve this:

• As we never change the field list, cache the mapped index pattern and reuse it for multiple initializations (downside: how to correctly clear the cache? it seems like the index pattern object is not immutable, so we could run into consistency issues)
• Change Lens to not require any remapping, but simply use the index pattern instance provided by the data plugin cache (downside: this requires a lot of refactoring)

[elasticmachine]

Pinging @elastic/kibana-app (Team:KibanaApp)

[dej611]

I've run some investigation in the topic and found that, maybe, we're already taking advantage of the data plugin cache:

• in the embeddable output the component is calling indexPatternService.get(id) which internally relies on the caching system
• in the indexpattern_datasource the loadIndexPatterns function is relying on the same indexPatternService.get(id) method

After discussing with @wylieconlon , there are still probably some open questions around performance here:

• what happens when loading 10 Lens embeddables in a dashboard with a big indexpattern? Will the cache be useful "in time"?
• even if the cache reduces the network time, we're still making a copy of the indexpattern for each Lens embeddable. Can this be a bottleneck in terms of performance for big indexpatterns?

[flash1293]

Yes, we are hitting the data plugin index pattern cache, this issue is about the re-mapping of index patterns within Lens (

kibana/x-pack/plugins/lens/public/indexpattern_datasource/loader.ts

Line 121 in dcc34e4

const currentIndexPattern: IndexPattern = {

). Sorry if that wasn't clear from the issue.

what happens when loading 10 Lens embeddables in a dashboard with a big indexpattern? Will the cache be useful "in time"?

The data plugin takes care of that, it will only do the work once even if the same index pattern is requested at once multiple times.

[flash1293]

Chatted with @mattkime about this and the data view instances delivered from the service are not immutable and might change field lists in some cases, so we can't use it as a cache key. However I'm not sure whether refactoring as described above would be a good idea as the instances not serializable as well.

Not sure about the right way forward here... @mbondyra what do you think about putting the data view instances directly into the redux store?

[mbondyra]

Data views have some unserialisable data so we'd break one of the redux state rules there, but it's not the first time we'd do it anyway.

I wonder if we could load data views before creating the store and have it as a dependency same way as datasourceMap or visualizationMap. But I guess not, given that the field list can change, eg. while adding runtime fields.

[flash1293]

I wonder if we could load data views before creating the store and have it as a dependency same way as datasourceMap or visualizationMap. But I guess not, given that the field list can change, eg. while adding runtime fields.

It's too expensive to preload all data views - we need an approach to do this on the fly. I like the idea of moving this to the frame level though. Maybe we can introduce a mechanism on the datasource to report a list of data views it requires, and the frame loads them async and then calls the sync functions of the data source, passing in the list of data view instances. I'm not sure whether they need to be in the redux state, we just put them in there so the datasource can use it in all places as the state is already passed into every function.

That would mean there are no data views in the state anymore, neither re-mapped nor unmapped. It's a pretty large refactoring though, but one that would reduce some complexity as we could remove the async initialization method of data sources; we just keep that around for the data views.

[stratoula]

Caching the fieldlist solves this problem! Will be available in 8.13!