[Security Solution] Support rule type changes in the rule upgrade workflow #161247
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
banderror
added
bug
|
Tested for all rules that changed rule types with
|
|
Pinging @elastic/security-detections-response (Team:Detections and Resp) |
|
Pinging @elastic/security-solution (Team: SecuritySolution) |
💛 Build succeeded, but was flaky
Failed CI StepsMetrics [docs]Unknown metric groupsESLint disabled line counts
Total ESLint disabled count
History
To update your PR or re-run it, just comment with: |
jpdjere
approved these changes
Jul 5, 2023
kibanamachine
pushed a commit
to kibanamachine/kibana
that referenced
this pull request
Jul 5, 2023
…kflow (elastic#161247) **Fixes: elastic#161094 ## Summary - Adds support for rule type changes in the `/internal/detection_engine/prebuilt_rules/upgrade/_review` endpoint. - Previously, if any rule had a different `type` in its `current_version` compared to its `target_version` the request would fail with `500`. - This PR: - updates this behaviour to accept rule type changes - creates a new `calculateAllFieldsDiff` method that is responsible for calculating diffs among all fields of all rule types. Used exclusively when there has been a rule type change between the current version and the target version (which can normally happen through upgrades of the `security_detection_engine` package) OR when the base version has a different type as the current version (which should not happen under normal conditions and user behaviour). - updates the diffable fields types for each specifc rule type (e.g.: `DiffableCustomQueryFields`,`DiffableEqlFields`,`DiffableThreatMatchFields`, etc) , replacing the `data_query` field name for either `eql_query` (for EQL type rules) or `kql_query` (for all others). ## How to test 1. With a clean Kibana state, use the `xpack.securitySolution.prebuiltRulesPackageVersion` config to force Kibana to install a package that contains the rules with their original type: ``` xpack.securitySolution.prebuiltRulesPackageVersion: '8.3.1' ``` 2. Install the four "offending" rules, [listed below.](elastic#161247 (comment)) 3. Remove the config, restart Kibana and navigate to the Rules Page so that the latest package is installed. 4. Navigate to the Rule Updates table. The four installed rules should have updates available. Update them. 5. All the listed rule types should be updated, as well as their corresponding fields. ### For maintainers - [ ] This was checked for breaking API changes and was [labeled appropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process) --------- Co-authored-by: jpdjere <jpdjeredjian@gmail.com> (cherry picked from commit 9e52f70)
💚 All backports created successfully
Note: Successful backport PRs will be merged automatically after passing CI. Questions ?Please refer to the Backport tool documentation |
kibanamachine
added a commit
that referenced
this pull request
Jul 5, 2023
…de workflow (#161247) (#161304) # Backport This will backport the following commits from `main` to `8.9`: - [[Security Solution] Support rule type changes in the rule upgrade workflow (#161247)](#161247) <!--- Backport version: 8.9.7 --> ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sqren/backport) <!--BACKPORT [{"author":{"name":"Georgii Gorbachev","email":"georgii.gorbachev@elastic.co"},"sourceCommit":{"committedDate":"2023-07-05T20:42:21Z","message":"[Security Solution] Support rule type changes in the rule upgrade workflow (#161247)\n\n**Fixes: https://github.com/elastic/kibana/issues/161094**\r\n\r\n## Summary\r\n\r\n- Adds support for rule type changes in the\r\n`/internal/detection_engine/prebuilt_rules/upgrade/_review` endpoint.\r\n- Previously, if any rule had a different `type` in its\r\n`current_version` compared to its `target_version` the request would\r\nfail with `500`.\r\n- This PR:\r\n - updates this behaviour to accept rule type changes\r\n- creates a new `calculateAllFieldsDiff` method that is responsible for\r\ncalculating diffs among all fields of all rule types. Used exclusively\r\nwhen there has been a rule type change between the current version and\r\nthe target version (which can normally happen through upgrades of the\r\n`security_detection_engine` package) OR when the base version has a\r\ndifferent type as the current version (which should not happen under\r\nnormal conditions and user behaviour).\r\n- updates the diffable fields types for each specifc rule type (e.g.:\r\n`DiffableCustomQueryFields`,`DiffableEqlFields`,`DiffableThreatMatchFields`,\r\netc) , replacing the `data_query` field name for either `eql_query` (for\r\nEQL type rules) or `kql_query` (for all others).\r\n\r\n\r\n## How to test\r\n1. With a clean Kibana state, use the\r\n`xpack.securitySolution.prebuiltRulesPackageVersion` config to force\r\nKibana to install a package that contains the rules with their original\r\ntype:\r\n```\r\nxpack.securitySolution.prebuiltRulesPackageVersion: '8.3.1'\r\n```\r\n2. Install the four \"offending\" rules, [listed\r\nbelow.](https://github.com/elastic/kibana/pull/161247#issuecomment-1622132120)\r\n3. Remove the config, restart Kibana and navigate to the Rules Page so\r\nthat the latest package is installed.\r\n4. Navigate to the Rule Updates table. The four installed rules should\r\nhave updates available. Update them.\r\n5. All the listed rule types should be updated, as well as their\r\ncorresponding fields.\r\n\r\n\r\n### For maintainers\r\n\r\n- [ ] This was checked for breaking API changes and was [labeled\r\nappropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)\r\n\r\n---------\r\n\r\nCo-authored-by: jpdjere <jpdjeredjian@gmail.com>","sha":"9e52f7064f4016b9c52606ab865149d6b48f5ae4","branchLabelMapping":{"^v8.10.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["bug","release_note:skip","impact:critical","Team:Detections and Resp","Team: SecuritySolution","Team:Detection Rule Management","Feature:Prebuilt Detection Rules","v8.9.0","v8.10.0"],"number":161247,"url":"https://github.com/elastic/kibana/pull/161247","mergeCommit":{"message":"[Security Solution] Support rule type changes in the rule upgrade workflow (#161247)\n\n**Fixes: https://github.com/elastic/kibana/issues/161094**\r\n\r\n## Summary\r\n\r\n- Adds support for rule type changes in the\r\n`/internal/detection_engine/prebuilt_rules/upgrade/_review` endpoint.\r\n- Previously, if any rule had a different `type` in its\r\n`current_version` compared to its `target_version` the request would\r\nfail with `500`.\r\n- This PR:\r\n - updates this behaviour to accept rule type changes\r\n- creates a new `calculateAllFieldsDiff` method that is responsible for\r\ncalculating diffs among all fields of all rule types. Used exclusively\r\nwhen there has been a rule type change between the current version and\r\nthe target version (which can normally happen through upgrades of the\r\n`security_detection_engine` package) OR when the base version has a\r\ndifferent type as the current version (which should not happen under\r\nnormal conditions and user behaviour).\r\n- updates the diffable fields types for each specifc rule type (e.g.:\r\n`DiffableCustomQueryFields`,`DiffableEqlFields`,`DiffableThreatMatchFields`,\r\netc) , replacing the `data_query` field name for either `eql_query` (for\r\nEQL type rules) or `kql_query` (for all others).\r\n\r\n\r\n## How to test\r\n1. With a clean Kibana state, use the\r\n`xpack.securitySolution.prebuiltRulesPackageVersion` config to force\r\nKibana to install a package that contains the rules with their original\r\ntype:\r\n```\r\nxpack.securitySolution.prebuiltRulesPackageVersion: '8.3.1'\r\n```\r\n2. Install the four \"offending\" rules, [listed\r\nbelow.](https://github.com/elastic/kibana/pull/161247#issuecomment-1622132120)\r\n3. Remove the config, restart Kibana and navigate to the Rules Page so\r\nthat the latest package is installed.\r\n4. Navigate to the Rule Updates table. The four installed rules should\r\nhave updates available. Update them.\r\n5. All the listed rule types should be updated, as well as their\r\ncorresponding fields.\r\n\r\n\r\n### For maintainers\r\n\r\n- [ ] This was checked for breaking API changes and was [labeled\r\nappropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)\r\n\r\n---------\r\n\r\nCo-authored-by: jpdjere <jpdjeredjian@gmail.com>","sha":"9e52f7064f4016b9c52606ab865149d6b48f5ae4"}},"sourceBranch":"main","suggestedTargetBranches":["8.9"],"targetPullRequestStates":[{"branch":"8.9","label":"v8.9.0","labelRegex":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"main","label":"v8.10.0","labelRegex":"^v8.10.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/161247","number":161247,"mergeCommit":{"message":"[Security Solution] Support rule type changes in the rule upgrade workflow (#161247)\n\n**Fixes: https://github.com/elastic/kibana/issues/161094**\r\n\r\n## Summary\r\n\r\n- Adds support for rule type changes in the\r\n`/internal/detection_engine/prebuilt_rules/upgrade/_review` endpoint.\r\n- Previously, if any rule had a different `type` in its\r\n`current_version` compared to its `target_version` the request would\r\nfail with `500`.\r\n- This PR:\r\n - updates this behaviour to accept rule type changes\r\n- creates a new `calculateAllFieldsDiff` method that is responsible for\r\ncalculating diffs among all fields of all rule types. Used exclusively\r\nwhen there has been a rule type change between the current version and\r\nthe target version (which can normally happen through upgrades of the\r\n`security_detection_engine` package) OR when the base version has a\r\ndifferent type as the current version (which should not happen under\r\nnormal conditions and user behaviour).\r\n- updates the diffable fields types for each specifc rule type (e.g.:\r\n`DiffableCustomQueryFields`,`DiffableEqlFields`,`DiffableThreatMatchFields`,\r\netc) , replacing the `data_query` field name for either `eql_query` (for\r\nEQL type rules) or `kql_query` (for all others).\r\n\r\n\r\n## How to test\r\n1. With a clean Kibana state, use the\r\n`xpack.securitySolution.prebuiltRulesPackageVersion` config to force\r\nKibana to install a package that contains the rules with their original\r\ntype:\r\n```\r\nxpack.securitySolution.prebuiltRulesPackageVersion: '8.3.1'\r\n```\r\n2. Install the four \"offending\" rules, [listed\r\nbelow.](https://github.com/elastic/kibana/pull/161247#issuecomment-1622132120)\r\n3. Remove the config, restart Kibana and navigate to the Rules Page so\r\nthat the latest package is installed.\r\n4. Navigate to the Rule Updates table. The four installed rules should\r\nhave updates available. Update them.\r\n5. All the listed rule types should be updated, as well as their\r\ncorresponding fields.\r\n\r\n\r\n### For maintainers\r\n\r\n- [ ] This was checked for breaking API changes and was [labeled\r\nappropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)\r\n\r\n---------\r\n\r\nCo-authored-by: jpdjere <jpdjeredjian@gmail.com>","sha":"9e52f7064f4016b9c52606ab865149d6b48f5ae4"}}]}] BACKPORT--> Co-authored-by: Georgii Gorbachev <georgii.gorbachev@elastic.co>
11 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fixes: #161094
Summary
/internal/detection_engine/prebuilt_rules/upgrade/_reviewendpoint.typein itscurrent_versioncompared to itstarget_versionthe request would fail with500.calculateAllFieldsDiffmethod that is responsible for calculating diffs among all fields of all rule types. Used exclusively when there has been a rule type change between the current version and the target version (which can normally happen through upgrades of thesecurity_detection_enginepackage) OR when the base version has a different type as the current version (which should not happen under normal conditions and user behaviour).DiffableCustomQueryFields,DiffableEqlFields,DiffableThreatMatchFields, etc) , replacing thedata_queryfield name for eithereql_query(for EQL type rules) orkql_query(for all others).How to test
xpack.securitySolution.prebuiltRulesPackageVersionconfig to force Kibana to install a package that contains the rules with their original type:For maintainers