Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cherry-pick #9056 to 6.x: Use _doc if ES major version is 7 #9390

Closed
wants to merge 1 commit into from

Conversation

urso
Copy link

@urso urso commented Dec 5, 2018

Cherry-pick of PR #9056 to 6.x branch. Original message:

Update the Elasticsearch output and template generator to set the type
to _doc if Elasticsearch major version is 7.

Update the Elasticsearch output and template generator to set the type
to _doc if Elasticsearch major version is 7.

Use common.Version throughout
- Use common version instead of strings + parsing over and over again.
- Having the parsed version available earlier, we can now configure the
  default document type based on version ranges.
- Use `_doc` if version is >= 7.0.

Introduce unit tests for version aware bulk encoding.

(cherry picked from commit 41f87a4)
@ruflin
Copy link
Member

ruflin commented Dec 6, 2018

The failures are presistent so I wonder how they are related to this change. Does the template need adjustments or the tests at least?

@ruflin
Copy link
Member

ruflin commented Dec 6, 2018

@urso What should be the upgrade behaviour here? Someone using MB 6.6 with ES 6.7 and then ES is upgraded, will still have the old template. As soon as he goes to the next version of MB, the new template will be loaded. Is that ok?

@urso
Copy link
Author

urso commented Dec 6, 2018

It's a system test that fails. Unfortunately all system tests make assumptions about the ES version being used and how template format should look like. This will hunt us a while.

@urso
Copy link
Author

urso commented Dec 6, 2018

What should be the upgrade behaviour here? Someone using MB 6.6 with ES 6.7 and then ES is upgraded, will still have the old template. As soon as he goes to the next version of MB, the new template will be loaded. Is that ok?

Oh good point... this might be a problem. We have to update/overwrite the template when we detect ES is updated. Otherwise ES will reject indexing due to us having two types doc and _doc.

In the future (not yet supported by ES) we will remove the _type from the bulk request. Then we won't have this issue. But I don't think we will have this behavior ready in ES when we have Beats 6.6.

Thinking about it, we should wait with backporting the change until 6.6 has it's own branch AND we do not set _type in bulk requests anymore.

@urso urso closed this Dec 6, 2018
@urso urso deleted the backport_9056_6.x branch February 19, 2019 19:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants