[8.6] [Security Solution][Alerts] improves performance of new terms multi fields (#145167) #145697
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…ields (elastic#145167) ## Summary This PR improves performance of new terms multiple fields implementation released in elastic#143943 In comparison with single value fields it's faster in 2-2.5 times In comparison with array field the win even more significant: 3-4 times ### Technical implementation: Value for runtime field is emitted only if any of new terms fields present in `include` clause of terms aggregation. It's achieved by passing a values map of new terms fields to the runtime script and checking new terms values in it. So, there is a significant performance wins if runtime field is not matched with includes values: - new terms fields are not encoded in base64 within runtime script - runtime script doesn't emit any values - runtime field doesn't have any value to be compared against during aggregation, as its empty - it eliminates possible unyielding execution branches early: if one of items in array of first new terms field is not present in values map, no need to run through the rest of combinations As a result, this implementation overcomes the issue with non exhaustive results due to the large number of emitted fields. ES doesn't allow emitting more than 100 values in the runtime script, so if the number of all combinations in new terms fields is greater than 100, only the first 100 combinations will be used for terms aggregation. With this new implementation only matched fields will be emitting. Even if its number is greater than 100, we will hit circuit breaker in Security Solution: as rule run can't generate more than 100 alerts ### Performance measurements Implementation | Shards | Docs per shard | Simultaneous Rule Executions | Fields cardinality | Rule Execution Time (improved) | Rule Execution Time (old) -- | -- | -- | -- | -- | -- | -- Terms 1 field | 10 | 900,000 | 1 | 100,000 | 7s | Terms 2 fields | 10 | 900,000 | 1 | 100,000 | 17s | 33s Terms 2 fields | 10 | 900,000 | 2 | 100,000 | 19s | Terms 3 fields | 10 | 900,000 | 1 | 100,000 | 18s | 46s Terms 3 fields | 10 | 900,000 | 2 | 100,000 | 20s | | | | | | | Terms 1 field | 20 | 900,000 | 1 | 100,000 | 10.5s | Terms 2 fields | 20 | 900,000 | 1 | 100,000 | 28s | 55s Terms 2 fields | 20 | 900,000 | 2 | 100,000 | 28.5s | 56s Terms 3 fields | 20 | 900,000 | 1 | 100,000 | 30s | 75s Terms 3 fields | 20 | 900,000 | 2 | 100,000 | 31s | 75s | | | | | | Terms 1 field | 10 | 1,800,000 | 1 | 100,000 | 7s | Terms 2 fields | 10 | 1,800,000 | 1 | 100,000 | 24s | 50s Terms 3 fields | 10 | 1,800,000 | 1 | 100,000 | 26s | 68s | | | | | | array of unique values length 10 | | | | | | Terms 1 field | 10 | 900,000 | 1 | 100,000 | 9.5s | Terms 2 fields | 10 | 900,000 | 1 | 100,000 | 75s | 3.5m Terms 3 fields | 10 | 900,000 | 1 | 100,000 | 83s | 6m ### Checklist Delete any items that are not applicable to this PR. - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com> Co-authored-by: Marshall Main <55718608+marshallmain@users.noreply.github.com> (cherry picked from commit b985ec1)
1 task
💚 Build Succeeded
Metrics [docs]Unknown metric groupsESLint disabled in files
ESLint disabled line counts
Total ESLint disabled count
To update your PR or re-run it, just comment with: cc @vitaliidm |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Backport
This will backport the following commits from
mainto8.6:Questions ?
Please refer to the Backport tool documentation