Parse alias filters on the coordinating node #20916
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Today we don't parse alias filters on the coordinating node, we only forward the alias patters to executing node and resolve it late. This has several problems like requests that go through filtered aliases are never cached if they use date math, since the parsing happens very late in the process even without rewriting. It also used to be processed on every shard while we can only do it once per index on the coordinating node. Another nice sideeffect is that we are never prone to clusterstate updates that change an alias, all nodes will execute the exact same alias filter since they are process based on the same cluster state.
s1monw
added
>enhancement
review
:Search/Search
|
@colings86 can you take a look |
colings86
approved these changes
Oct 13, 2016
| // we need to bench here a bit, to see maybe it makes sense to use OrFilter | ||
| BoolQueryBuilder combined = new BoolQueryBuilder(); | ||
| for (String aliasName : aliasNames) { | ||
| AliasMetaData alias = aliases.get(aliasName); |
There was a problem hiding this comment.
Looks like there is an indentation issue here?
|
Looks like org.elasticsearch.search.SearchServiceTests.testSearchWhileIndexDeleted failed in the PR build. Not sure if this failure is related to this change? |
…s rewritten once context is created
|
@colings86 I added additional testing can you take another look? |
colings86
approved these changes
Oct 14, 2016
| final Map<String, AliasFilter> queryBuilderMap = new HashMap<>(); | ||
| for (String index : concreteIndices) { | ||
| clusterState.blocks().indexBlockedRaiseException(ClusterBlockLevel.READ, index); | ||
| AliasFilter queryBuilder = searchService.buildAliasFilter(clusterState, index, request.indices()); |
There was a problem hiding this comment.
nit: should this be called aliasFilter?
| } | ||
|
|
||
| private Map<String, AliasFilter> buildPerIndexAliasFilter(SearchRequest request, ClusterState clusterState, String...concreteIndices) { | ||
| final Map<String, AliasFilter> queryBuilderMap = new HashMap<>(); |
There was a problem hiding this comment.
nit: should this be called aliasFilterMap?
s1monw
added a commit
that referenced
this pull request
Oct 14, 2016
Today we don't parse alias filters on the coordinating node, we only forward the alias patters to executing node and resolve it late. This has several problems like requests that go through filtered aliases are never cached if they use date math, since the parsing happens very late in the process even without rewriting. It also used to be processed on every shard while we can only do it once per index on the coordinating node. Another nice side-effect is that we are never prone to cluster-state updates that change an alias, all nodes will execute the exact same alias filter since they are process based on the same cluster state.
|
@s1monw I think that we should add a bw comp layer on top of this change, which otherwise breaks backwards compatibility on the transport layer. Maybe our bw comp tests don't catch it because we have no REST tests that search, or validate query, or explain against a filtered alias? |
|
@javanna you were wrong, this is fully bw compatible :) |
the bwc layer is inside the new class... it's not very obvious from the high level but when you look closer you see that all kinds of BWC tests are present. |
javanna
added a commit
to javanna/elasticsearch
that referenced
this pull request
Nov 22, 2016
The search shards api returns info about which shards are going to be hit by executing a search with provided parameters: indices, routing, preference. Indices can also be aliases, which can also hold filters. The output includes an array of shards and a summary of all the nodes the shards are allocated on. This commit adds a new indices section to the search shards output that includes one entry per index, where each index can be associated with an optional filter in case the index was hit through a filtered alias. This is relevant since we have moved parsing of alias filters to the coordinating node. Relates to elastic#20916
javanna
added a commit
that referenced
this pull request
Nov 22, 2016
Add indices and filter information to search shards api output The search shards api returns info about which shards are going to be hit by executing a search with provided parameters: indices, routing, preference. Indices can also be aliases, which can also hold filters. The output includes an array of shards and a summary of all the nodes the shards are allocated on. This commit adds a new indices section to the search shards output that includes one entry per index, where each index can be associated with an optional filter in case the index was hit through a filtered alias. This is relevant since we have moved parsing of alias filters to the coordinating node. Relates to #20916
javanna
added a commit
that referenced
this pull request
Nov 22, 2016
Add indices and filter information to search shards api output The search shards api returns info about which shards are going to be hit by executing a search with provided parameters: indices, routing, preference. Indices can also be aliases, which can also hold filters. The output includes an array of shards and a summary of all the nodes the shards are allocated on. This commit adds a new indices section to the search shards output that includes one entry per index, where each index can be associated with an optional filter in case the index was hit through a filtered alias. This is relevant since we have moved parsing of alias filters to the coordinating node. Relates to #20916
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Today we don't parse alias filters on the coordinating node, we only forward
the alias patters to executing node and resolve it late. This has several problems
like requests that go through filtered aliases are never cached if they use date math,
since the parsing happens very late in the process even without rewriting. It also used
to be processed on every shard while we can only do it once per index on the coordinating node.
Another nice side effect is that we are never prone to cluster-state updates that change an alias,
all nodes will execute the exact same alias filter since they are process based on the same
cluster state.