Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Streamline no-op template updates #57662

Closed
DaveCTurner opened this issue Jun 4, 2020 · 6 comments · Fixed by #57851 or #64493
Closed

Streamline no-op template updates #57662

DaveCTurner opened this issue Jun 4, 2020 · 6 comments · Fixed by #57851 or #64493
Assignees
@dakrone
Labels
:Data Management/Indices APIs APIs to create and manage indices and templates >enhancement Team:Data Management Meta label for data/management team

Comments

@DaveCTurner
Copy link
Contributor

Today every put-template request (both composable and legacy) results in a cluster state update, even if the cluster state already contains exactly the template that we're trying to put. There's no need to update the cluster state if the template is already there, so it would be preferable to skip any such requests without needing to publish a new cluster state version.
@DaveCTurner DaveCTurner added >enhancement :Data Management/Indices APIs APIs to create and manage indices and templates labels Jun 4, 2020
@elasticmachine
Copy link
Collaborator

Pinging @elastic/es-core-features (:Core/Features/Indices APIs)

@elasticmachine elasticmachine added the Team:Data Management Meta label for data/management team label Jun 4, 2020
@dakrone
Copy link
Member

dakrone commented Jun 4, 2020

@DaveCTurner this sounds pretty reasonable to me, out of curiosity, was this causing any problems anywhere?

@DaveCTurner
Copy link
Contributor Author

Yes, we've seen a few cases (and other issues linked to this one) where users unwittingly leave template auto-creation on across thousands of Beats, resulting in lot of no-op cluster state updates at HIGH priority preventing the master from doing anything more useful for quite some time. Although the requests do time out after 30s by default this doesn't really help since clients will often retry until successful.

There's changes in progress on the Beats side too to mitigate this kind of problem, but that won't help other clients.

@dakrone dakrone self-assigned this Jun 8, 2020
dakrone added a commit to dakrone/elasticsearch that referenced this issue Jun 8, 2020
@dakrone
Make noop template updates be cluster state noops
69b9a92 
When updating an existing component or composable template that is identical to the existing
template, we now skip doing a cluster state update.

Resolves elastic#57662
@dakrone dakrone mentioned this issue Jun 8, 2020
Merged
dakrone added a commit that referenced this issue Jun 9, 2020
@dakrone
Make noop template updates be cluster state noops (#57851)
08fc2ca 
When updating an existing component or composable template that is identical to the existing
template, we now skip doing a cluster state update.

Resolves #57662
dakrone added a commit to dakrone/elasticsearch that referenced this issue Jun 9, 2020
@dakrone
Make noop template updates be cluster state noops (elastic#57851)
cc21afc 
When updating an existing component or composable template that is identical to the existing
template, we now skip doing a cluster state update.

Resolves elastic#57662
@Leaf-Lin Leaf-Lin mentioned this issue Jul 1, 2020
Closed
@DaveCTurner
Copy link
Contributor Author

@dakrone I recently learned that Beats will continue to default to using legacy templates (elastic/beats#21212) for BWC reasons, but #57851 doesn't address that case, so I think this will continue to affect users for a while yet. Could you do something similar to #57851 for legacy templates?

@DaveCTurner DaveCTurner reopened this Nov 2, 2020
dakrone added a commit to dakrone/elasticsearch that referenced this issue Nov 2, 2020
@dakrone
Make legacy template noop updates not change cluster state
afa77e5 
When a legacy template does not change, there is no need to create and publish a new cluster state,
rather the update should be a noop.

Relates to elastic#57851
Resolves elastic#57662
@dakrone dakrone mentioned this issue Nov 2, 2020
Merged
@dakrone
Copy link
Member

dakrone commented Nov 2, 2020

@DaveCTurner I agree with the concern, I opened #64493 for this for at least 7.11+

dakrone added a commit that referenced this issue Nov 2, 2020
@dakrone
Make legacy template noop updates not change cluster state (#64493)
b323d4c 
When a legacy template does not change, there is no need to create and publish a new cluster state,
rather the update should be a noop.

Relates to #57851
Resolves #57662
dakrone added a commit to dakrone/elasticsearch that referenced this issue Nov 2, 2020
@dakrone
Make legacy template noop updates not change cluster state (elastic#6…
5539260 
…4493)

When a legacy template does not change, there is no need to create and publish a new cluster state,
rather the update should be a noop.

Relates to elastic#57851
Resolves elastic#57662
@Leaf-Lin
Copy link
Contributor

Just want to point out this could happen to APM as well.

https://www.elastic.co/guide/en/apm/server/current/configuration-template.html

setup.template.overwrite defaults to false.

Leaf-Lin added a commit to elastic/apm-server that referenced this issue Nov 27, 2020
@Leaf-Lin
Add warnings to setup.template.overwrite
3b299ee 
Similar to elastic/beats#22357, setup.template.overwrite could potentially overload Elasticsearch with too many update requests. 
On Elasticsearch side, it will be addressed in newer versions by introducing no-op updates which will be available from 7.11+ elastic/elasticsearch#64493

See also elastic/elasticsearch#57662

Thought having some warnings in the APM setting here should prevent users unwittingly leave template auto-creation on across large number of APM.
@Leaf-Lin Leaf-Lin mentioned this issue Nov 27, 2020
Closed
10 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@dakrone dakrone

Labels
:Data Management/Indices APIs APIs to create and manage indices and templates >enhancement Team:Data Management Meta label for data/management team
Projects
None yet
Milestone
No milestone
4 participants
@dakrone @DaveCTurner @elasticmachine @Leaf-Lin