Kibana Index Patterns going out-of-sync

[sorenlouv]

Today the APM index pattern in Kibana is not created automatically. The user can create it manually through the Kibana UI, or by running ./apm-server setup.

There are two cases where the APM index pattern is required in the upcoming 6.4 release:

• when importing dashboards/visualization (from Kibana Getting Started Guide)
• when using the query bar (from APM UI)

For the above cases to work correctly we either need to:

• handle the case where it does not exists and provide the user with instruction for creating it manually
• create it automatically

In both cases we also need to handle the case where the apm-server schema changes, which should cause an existing APM index pattern to be deleted and recreated.

Possible solutions

1. Auto-creating from Kibana by inferring the schema
I initially tried to use the saved_objects API in Kibana like:

POST /api/saved_objects/index-pattern/apm-*
{
  attributes: { title: 'apm-*', timeFieldName: '@timestamp' }
}

This will infer the field types from elasticsearch, and create the index pattern. The downside of this is:

• not all field types can be correctly inferred (afaik)
• I didn't find a good way to determine when the APM-server schema had changed, which requires the APM index pattern to be re-created

2. Auto-creating from Kibana using a hard-coded index pattern object
Same as above, but instead of inferring the index pattern fields, a pre-defined index pattern will be created.

This would solve the problem from above where field types can't be inferred since they can now be explicitly defined in the hardcoded object.
This does not solve the problem with an existing APM index pattern coming out-of-sync when apm-server schema changes after an upgrade.

It also adds the extra requirement, that whenever apm-server schema changes, the file in Kibana containing the index pattern object must be updated.

3. Auto-creating from apm-server on startup
On startup apm-server should remove the existing APM index pattern (if it exists) and create a new APM index pattern.
This will ensure that there is always an up-to-date APM index pattern.

Is is possible to create the index pattern by calling elasticsearch directly but it is probably easier to use the saved_objects API in Kibana. Is it a problem that the apm-server has a dependency on Kibana?

Conclusion

We should find a way to create the APM Index Pattern in Kibana, that is either easy to do for the user, or fully automated. The created index pattern needs to stay in sync with the apm-server schema.

Personally I'm most inclined to using the last solution (3), since this seems to address the problems with 1) and 2)

@simitt @roncohen @jalvz @graphaelli

[roncohen]

For (3), a slight adoption of this change would enable us to export the index pattern to a file using APM Server.

[jalvz]

You obviously thought about this more than me, but I do think that a dependency on Kibana is a problem...
For detecting schema changes would it help if we add a numeric schema_version to each document?

[makwarth]

Thanks for the write up @sqren. I agree 3 is the best option. One concern: If a user has old and current APM servers / Beats, and one of the older ones is restarted, it'd end up replacing a current index pattern with an old one?

[graphaelli]

This is possible now (unless I'm missing something) with:

setup.dashboards.enabled=true
setup.dashboards.retry.enabled=true

That does overwrite the existing pattern unconditionally and allows apm-server to continue operating if kibana is unavailable, reducing the kibana dependency. I would argue these should be the default on cloud - would making these the default solve the issue? Is it ok to get the dashboards along with the index pattern? Did I completely miss something?

[sorenlouv]

@graphaelli The user has to run ./apm-server setup manually, right? If they haven't done that the index patterns won't exist, and the UI will not work (correctly).

[sorenlouv]

Oh, are you saying that just adding those to the config does the trick? Then yeah, maybe just having them in there by default would work.

[sorenlouv]

Since we are moving importing of dashboards to Kibana, the option should perhaps be renamed to setup.index_pattern.enabled or similar.

[graphaelli]

Since we are moving importing of dashboards to Kibana

Do we plan to remove dashboard loading from apm-server when kibana is able to import them or will they do the same thing? If they're removed setup.dashboards.only_index=true should also be set.

option should perhaps be renamed to setup.index_pattern.enabled

We can probably do that. The setup.dashboards.* options are inherited from libbeat so it might be tricky but I expect we can figure that out if it's important.

There's still the issue @makwarth brought up (if this defaults to on):

One concern: If a user has old and current APM servers / Beats, and one of the older ones is restarted, it'd end up replacing a current index pattern with an old one?

[sorenlouv]

Do we plan to remove dashboard loading from apm-server when kibana is able to import them

@roncohen mentioned that we should leave it in the apm-server but I'm not sure why that is useful. The dashboard import feature will be in the Getting Started guide (not APM ui) so will be able to both xpack and OSS users.

We can probably do that. The setup.dashboards.* options are inherited from libbeat so it might be tricky but I expect we can figure that out if it's important.

Not important at all. Just thought it would be helpful to separate the notion of creating dashboards vs creating index patterns

There's still the issue @makwarth brought up

True, and @ruflin also brought that up. Not sure how that should be handled tbh.

[makwarth]

Agree, it doesn't make sense to have dashboards in the Server any longer, imo.

[sorenlouv]

@graphaelli Let's move forward with your suggestion #991 (comment). I'll join the apm-server meeting today with @makwarth and we can hash out the details.

[roncohen]

@roncohen mentioned that we should leave it in the apm-server but I'm not sure why that is useful.

I don't think i was talking about the dashboards. What i was maybe saying was that we should think twice about removing index pattern generation from the APM Server and store a hardcoded file in Kibana because as opposed to in Kibana, we have a high level way to define the index pattern in the APM Server (fields.yml).

[makwarth]

It's only been a few days, but wanted to follow up on this as all new 6.4 UI features depend on this.

Regarding this:

One concern: If a user has old and current APM servers / Beats, and one of the older ones is restarted, it'd end up replacing a current index pattern with an old one?

I don't think it's a blocker for 6.4, but would something like this be possible: In the APM Server, check if the Server's version is lower than other Servers ingesting to the ES cluster. If yes, the APM Server should not update the index pattern on start/restart. (I write APM Server, but I assume this functionality is really in libbeat)

[alvarolobato]

We've been talking about this again, we think that creating the indexpatterns from the server seems to be the first solution. We are still deciding on how to handle different versions.
@simitt is going to do a few tests and come back with the results, it should be ready by the begining of the next week.

@sqren is there a way to use more than one index pattern or you can use only one?

[sorenlouv]

@alvarolobato @simitt and I talked about different solutions. We are currently investigating what's possibly works better, and will get back to you when we have something concrete.

[simitt]

@sqren I investigated the version handling idea.
I can confirm your finding that although we provide a version in the Kibana Index Pattern that we send to the Kibana API no version shows up in the indexed document in .kibana index.

Adding the version to the id of the index-pattern seems to be the only way to store multiple apm index patterns atm. Setting following config options in the apm-server.yml leads to auto-import and overwrite the id of the index pattern and dashboards when the APM Server is started:

setup.dashboards.enabled: true
setup.dashboards.index: "apm-7.0.0-alpha1*"
setup.dashboards.retry.enabled: true

Issue: libbeat does not handle using "apm-%{[beat.version]}-*" properly for the setup.dashboards.index, which means we would need to fix this in libbeat or hardcode a version.

Using this approach I tested the behaviour of dashboards and the curated UI:

• Dashboards use only one of the imported Index Pattern, as this gets hardcoded to the dashboards when they are imported. This means the dashboards would always only show the data matching the version of the latest import, which is not necessarily the latest version.
• Curated UI seems to work fine with multiple Index Patterns, starting with apm. I imported data from two different versions, and two versioned Index Pattern, also one where I removed all the fields and all data showed up and seemed to work fine in the curated UI.

If we want to go forward with this idea we would need to

• come up with a solution for the dashboards, e.g. separating the dashboards and the index pattern import (would probably affect other beats).
• fix that libbeat properly resolves the version for the setup.dashboards.index
• do further investigation from the UI side

[sorenlouv]

Curated UI seems to work fine with multiple Index Patterns, starting with apm. I imported data from two different versions, and two versioned Index Pattern, also one where I removed all the fields and all data showed up and seemed to work fine in the curated UI.

fyi: Query bar and ML are the only consumers of index patterns, and they are not in master yet, so might get a false positive result.

[sorenlouv]

To fetch the index patterns, I assume the UI would have to get a list of index patterns, and select the latest one. Is that correct?

[sorenlouv]

Solution A: Overwriting logic is in Kibana
On Friday @simitt and I discussed the different approaches to importing index patterns, and settled on a model where the logic determining whether to overwrite the previous index pattern would reside in Kibana. A requirement for this was to add a beatVersion field to the index pattern object. Unfortunately the .kibana mapping is set to dynamic: strict, so unknown fields will cause an exception to be thrown. Only way around this is to update the mapping.json with the new field, which will only be applied once. It will therefore not be applied for users upgrading from 6.3 to 6.4 and our solution is not viable.

Solution B: No overwriting logic. APM-Server overwrites every time regardless of version
Instead we are going back to the original plan of letting APM-Server create the index pattern on startup, and overwrite the previous index pattern without any version checks. This could cause the index pattern to be overwritten with an older version. If this is changed in 6.5 we shouldn't have too many people affected by this (only people who have changed their 6.3 configs to automatically import index patterns AND are running different versions of APM-Server).

APM UI still needs a strategy when no index pattern has been created. This could happen if the user is running apm-server 6.3 or below.

[makwarth]

Thanks for the update. Few comments / questions:

1/

If this is changed in 6.5

With this we're targeting to remove the index pattern creation from the APM Server in 6.5 and move the functionality to Kibana with the update of mapping.json? If yes, I'll look into the progress on this / create a new issue in the Kibana repo.
(Without knowing the details, this likely can slip into 6.6. Hopefully we won't have many schema changes between 6.5 and 6.6. and therefore a 6.6 delay won't be a blocker.)

2/

Does this mean that Import Dashboards in the Kibana Add data tutorial will be disabled until it can detect an index pattern? That index pattern can come from booting up the APM Server or creating it manually in Management/Index Patterns

3/

Follow up on 2): If I remember correctly, when manually creating the Index Patterns in Kibana, the field types weren't set correctly? Is this a blocker?

4/

We need a state for the Query bar and the ML integration, if there's no index pattern. (cc @formgeist)

[formgeist]

@makwarth Created a design enhancements issue for those tasks elastic/kibana#19995 cc @sqren let's chat if there's any more things needed before release.

[sorenlouv]

With this we're targeting to remove the index pattern creation from the APM Server in 6.5 and move the functionality to Kibana with the update of mapping.json? If yes, I'll look into the progress on this / create a new issue in the Kibana repo.

Yes, I hope we can add the version check in Kibana by 6.5.

Does this mean that Import Dashboards in the Kibana Add data tutorial will be disabled until it can detect an index pattern?

Yes, that is one solution. Another is to fall back to a hardcoded index pattern (which might not match with the APM-Server version). A third is to create the index pattern automatically (with the disadvantages you describe below).

If I remember correctly, when manually creating the Index Patterns in Kibana, the field types weren't set correctly? Is this a blocker?

This is how I understand it, yes. But I have always used auto-created index patterns and haven't had problems, so I wonder how big a deal it is.

[simitt]

After another round of discussions, we are aiming for moving the dashboards and the index patterns to Kibana.
This would mean having the current versions hardcoded in Kibana. We always aim for backwards compatibility in minor versions, therefore running a newer version of APM Server than of Kibana and the other way around should not be an issue, it just means that newly introduced fields or bugfixes might not be usable unless updated. The curated APM UI is already dealing with the same compatibility scenario.

Action items for 6.4

• @sqren looks into how feasible this is for 6.4
• APM Server would mark the apm-server setup command and the usage of the setup.dashboards* in the apm-server.yml file as deprecated.

[makwarth]

Just to clarify: The index pattern and dashboards will be hardcoded in Kibana, as part of the "Add data" instructions, not the APM UI. This means they will also be accessible for non-X-Pack users.
We hope to get this in by 6.4, but it's really up to Nathan who owns the Add data functionality in Kibana. Søren will reach out to Nathan, and we already have an issue about this from before this discussion. Closing this issue for elastic/kibana#17803

[makwarth]

Notes:

• The setup instructions will ideally have two buttons (instead of the -setup cmd)
  • "Import index pattern"
  • "Import index pattern and dashboards"
• The APM UI will not do specific checks for the index pattern. Elements in the APM UI that require and index pattern - like query bar or ML integration - will have a state to inform the user to go import the index pattern. (cc @formgeist)