[filebeat] Upgrade Kibana module to prepare for 8.0 breaking changes

[lukeelmers]

The @elastic/kibana-core team has been working on improvements to our new platform logging service in preparation for removing Kibana's legacy logging system in 8.0.

As a part of these changes, the log formats have been updated to be more in line with the Elasticsearch logs which are based on log4j 2. In particular, our log format for http request/response logs has been updated to include ECS-compliant meta, and we've also adjusted how ops metrics are logged, among other changes.

After looking at the ingest pipeline for the Kibana module, it appears that several of these referenced fields will break once the legacy logger is deprecated.

Wanted to raise this issue now so that we can discuss implications on Beats & answer any questions you might have around these changes. We have not yet removed the legacy logs from Kibana's master branch, but will likely do so as we get closer to the 8.0 alpha.

cc @alexh97 @kobelb

[elasticmachine]

Pinging @elastic/integrations-services (Team:Services)

[ycombinator]

The impact here (from a Beats perspective) is on the kibana Filebeat module. It will need to be enhanced to read the new format of Kibana logs while also being able to read the legacy format (for the period that both formats are supported). Then, at some future point, the module can be changed again to remove support for the legacy format.

@lukeelmers what's the plan for Kibana logs through the deprecation period? Will each log entry contain both sets of fields (legacy + new)? Or are the new logs being written to a separate file along side the legacy ones (if so, how would this work in a containerized environment where logs might be emitted to STDOUT)? Or something else?

[ycombinator]

cc: @sayden

[lukeelmers]

Will each log entry contain both sets of fields (legacy + new)?

@ycombinator @sayden The short answer is that there are duplicate log entries (one in new & one in old format).

what's the plan for Kibana logs through the deprecation period?

The deprecation strategy looks like this:

• We are marking Kibana's legacy logging configuration options as deprecated starting in 7.13
• Legacy config options will continue to work until 8.0, but a deprecation warning will be logged prompting the user to update to the new config
• Once the user switches to use the new config, the new format is included in the same stream alongside the old format, so there are essentially duplicate log entries: one in new format, one in old format (see example snippets below)
• In 8.0, the old format goes away entirely, so there are no longer duplicate entries, and only the new format remains
• Other than one remaining modification we are shipping in 7.13, the new format is done and we don't anticipate any other changes to it.

Example log snippets

7.x through 7.last - using old/deprecated config

json

{"type":"log","@timestamp":"2021-03-10T08:20:38-07:00","tags":["info","savedobjects-service"],"pid":85059,"message":"Waiting until all Elasticsearch nodes are compatible with Kibana before starting saved objects migrations..."}
{"type":"log","@timestamp":"2021-03-10T08:20:39-07:00","tags":["info","savedobjects-service"],"pid":85059,"message":"Starting saved objects migrations"}

stdout

server    log   [08:30:57.067] [info][savedobjects-service] Waiting until all Elasticsearch nodes are compatible with Kibana before starting saved objects migrations...
server    log   [08:30:57.121] [info][savedobjects-service] Starting saved objects migrations

7.x through 7.last - using new config

json

{"@timestamp":"2021-03-10T08:20:38.101-07:00","message":"Waiting until all Elasticsearch nodes are compatible with Kibana before starting saved objects migrations...","log":{"level":"INFO","logger":"savedobjects-service"},"process":{"pid":85059}}
{"type":"log","@timestamp":"2021-03-10T08:20:38-07:00","tags":["info","savedobjects-service"],"pid":85059,"message":"Waiting until all Elasticsearch nodes are compatible with Kibana before starting saved objects migrations..."}
{"@timestamp":"2021-03-10T08:20:39.997-07:00","message":"Starting saved objects migrations","log":{"level":"INFO","logger":"savedobjects-service"},"process":{"pid":85059}}
{"type":"log","@timestamp":"2021-03-10T08:20:39-07:00","tags":["info","savedobjects-service"],"pid":85059,"message":"Starting saved objects migrations"}

stdout

[2021-03-10T08:30:57.067-07:00][INFO ][savedobjects-service] Waiting until all Elasticsearch nodes are compatible with Kibana before starting saved objects migrations...
server    log   [08:30:57.067] [info][savedobjects-service] Waiting until all Elasticsearch nodes are compatible with Kibana before starting saved objects migrations...
[2021-03-10T08:30:57.121-07:00][INFO ][savedobjects-service] Starting saved objects migrations
server    log   [08:30:57.121] [info][savedobjects-service] Starting saved objects migrations

8.0

json

{"@timestamp":"2021-03-10T08:20:38.101-07:00","message":"Waiting until all Elasticsearch nodes are compatible with Kibana before starting saved objects migrations...","log":{"level":"INFO","logger":"savedobjects-service"},"process":{"pid":85059}}
{"@timestamp":"2021-03-10T08:20:39.997-07:00","message":"Starting saved objects migrations","log":{"level":"INFO","logger":"savedobjects-service"},"process":{"pid":85059}}

stdout

[2021-03-10T08:30:57.067-07:00][INFO ][savedobjects-service] Waiting until all Elasticsearch nodes are compatible with Kibana before starting saved objects migrations...
[2021-03-10T08:30:57.121-07:00][INFO ][savedobjects-service] Starting saved objects migrations

It will need to be enhanced to read the new format of Kibana logs while also being able to read the legacy format (for the period that both formats are supported)

Technically Kibana has shipped for the last several minors with the behavior described above (both formats alongside each other), however in practice folks will most likely have not seen any change as we haven't yet communicated to users that the legacy settings are deprecated. That's what will be changing in 7.13 -- users will start to see the warnings and switch over to the new configs, and then start to get the duplicate entries.

As the entries are duplicates, that means users can safely throw away/ignore the new format until 8.0 and not lose any data.

[lukeelmers]

One other thing to add: The new logs will be ECS compliant after we address elastic/kibana#90406, so hopefully that makes things easier 🙂

[ycombinator]

Thanks for all the details, including sample log snippets, @lukeelmers — super helpful!

@sayden let me know if you have any questions about handling any of this from the Filebeat kibana module perspective!

[lukeelmers]

Hey @sayden, just wanted to follow up now that we've closed elastic/kibana#90406. At this point we are considering our new logging system "done". Do you have any idea yet where this might fall on the Beats roadmap?

One thing we are hoping to do soon is lift the requirement that folks log everything to both the legacy and the new system. Currently we force everything to still be logged to the legacy system for backwards compatibility, however we'd like to allow folks to opt-out of the legacy system when they are ready to do so. Ideally we'd do this soon in 7.x to give folks as much time as possible to shift to the new system and ensure a smooth 8.0 upgrade. We decided that we want to wait to lift this restriction until the filebeat module is ready (see discussion on this PR), so let me know if there's any further resources we could provide to ensure you have all the info you need.

We have updated the Kibana docs with details, including a general overview and a guide on migrating to the new system. The planned breaking changes have also been documented in our 8.0 breaking changes docs.

Lastly, I've put together a 30 minute crash course screencast which demonstrates some of the changes, including a look at how the log formats differ. I'd be happy to sync if you want to chat about any of this or have questions -- feel free to ping me. 🙂

[lukeelmers]

Just checking in to see if this was still in the works, as we are planning to remove the legacy logging format from Kibana's master branch in the next month or two.

Don't hesitate to reach out if any questions come up!

[lukeelmers]

@elastic/integrations-services just a heads up that we are planning on merging the breaking changes to Kibana's master branch in the next week or two -- we will follow up soon with an exact date.

Are these changes in progress on your side?

[lukeelmers]

Was just talking to @yspotts on the Cloud team, and learned that these changes to Filebeat will be a blocker to Cloud's ability to update to support the new logging system.

In addition to the near-term goal of supporting 8.0, Cloud will eventually aim to support the new log format in 7.16 (as this will help us greatly for debugging purposes). While this won't necessarily need to coincide with the 7.16 release on Cloud's side -- they can add support after the fact -- they would still require a 7.16-compatible version of Filebeat to be able to handle the new format.

The implication here is that Beats would need to support both formats in 7.16, and then just the new format in 8.0.

[yspotts]

@lukeelmers @sayden

//cc @andrew-moldovan @gmarz @nikulinivan

Talking with @AlexP-Elastic Just discovered another fly in the ointment that we need to be aware of:
Its a bit complicated but we actually have 2 separate filebeats running in the cloud:

• One that is "co-located" with the deployment. This one ships logs via the "log delivery" feature for user consumption. This filebeat is included in the docker image together with the ES/Kibana from the unified build. So for 8.0 deployments, it will use the 8.0 filebeat, and thus we need filebeat support for the new format in 8.0. This filebeat is the responsibility of the Application/Solutions team. So in this regard, we are fine.

• However, there is another filebeat (called the beats-runner) which runs "globally" in the "ECE" infrastructure. This ships logs to the the regional logging cluster (ESS) or the logging and monitoring deployment (ECE). Currently, this is on a very old version of filebeats (see https://github.com/elastic/cloud/issues/79840 for plans to upgrade). This filebeat is owned by the Platform team.
  What this means is that the cloud will need support for new format in filebeat for a version before 8.0 - since we cannot wait until 8.0 is GAed in order to upgrade this "global" filebeat. We will have a chicken egg issue, and in addition, there is no way to test out the logging in advance in QA/Staging which is a requirement on our side.

In order not to step on anyone's toes, I bring it up here, and allow the stack team to work with @elastic/cloud-platform-data-services to co-ordinate strategy on this point.

[yspotts]

@gigerdo FYI

[s-nel]

What this means is that the cloud will need support for new format in filebeat for a version before 8.0 - since we cannot wait until 8.0 is GAed in order to upgrade this "global" filebeat. We will have a chicken egg issue, and in addition, there is no way to test out the logging in advance in QA/Staging which is a requirement on our side.

I don't really understand that, can you elaborate? I believe our stance is that we won't upgrade the L+M (ECE) and RLC (ESS) to 8.0 right away, and we would keep beats-runner on 7.x until we upgrade L+M and RLC.

[yspotts]

@s-nel

Yes, your understanding is correct - I expressed in inelegantly.
That is essentially my point - that since we will keep beats-runner on 7.x for now, we need to backport filebeat support for the new logging output to some version of 7.x.

[s-nel]

Thanks I understand now 🙇

[gmarz]

@yspotts Thanks for pinging us 👍

On the Cloud Platform side (in terms of the beats-runner upgrade) we are OK with the changes proposed in elastic/kibana#50660, as long as support for the legacy format is backported to some 7.x version that we can upgrade to before 8.0. @planadecu Can you coordinate with the beats team on determining what version that will be?

Separately, @lucasmoore we'll need to work out how we plan to roll out the beats-runner upgrade (https://github.com/elastic/cloud/issues/79840) to ESS in order to prepare for this change. I'll send something out to discuss when @przemek-grzedzielski (who has been driving the beats upgrade) returns next week.

[lukeelmers]

as long as support for the legacy format is backported to some 7.x version that we can upgrade to before 8.0. @planadecu Can you coordinate with the beats team on determining what version that will be?

@gmarz Were you meaning to say "as long as support for the new format is backported"? If so, I think that's a question for @sayden to confirm, but we will need to support both as I mentioned above:

The implication here is that Beats would need to support both formats in 7.16, and then just the new format in 8.0.

[planadecu]

Maybe the folks from the beats team can shed some light on the topic @ruflin @ravikesarwani. How feasible is it that Filebeats supports both versions? In which version do you think you could do it?

[sayden]

I managed to spend some time on this today and started to modify the pipeline.

So as far as I can understand, a TLDR is that Filebeat needs to support ASAP Kibana logs for 7.x and 8.x at the same time, right? (all blame on my about the ASAP, don't assume malice).

[yspotts]

@sayden thank you!
I would say ASAP for 8.x. 7.x is important, but some decisions are TBD on 7.x

(And I certainly dont ascribe any malice; we are all working hard on so many things and it seems like there is a never ending list of stuff that is ASAP, no worries at all my friend!)

[lukeelmers]

@sayden Thanks for looking into this! Yep, 8.x is most critical, however I know we have 7.16 FF coming up first, so unless this is something you'd envision going out in a 7.16.x patch release (which would be up to you all), you may want to consider tackling both of them at once. There would still be benefit to users to have support on 7.x, even if we are still working out some of the details on the cloud side.

And of course, Filebeat would only need to support both versions on 7.x. (On master we'd only need to support the new format)

[jlind23]

Backlog grooming: @lukeelmers as 8.0 is already out for a while, shouldn't we close this issue?