Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Security Solution] Add ES|QL Query editable component #199887

Merged
merged 45 commits into from
Nov 28, 2024

Conversation

maximpn
Copy link
Contributor

@maximpn maximpn commented Nov 12, 2024

Partially addresses: #171520

Summary

This PR adds is built on top of #193828 and #196948 and adds an ES|QL Query editable component for Three Way Diff tab's final edit side of the upgrade prebuilt rule workflow.

Details

This PR extracts ES|QL Query edit component from Define rule form step and makes it reusable. The following changes were made

  • ES|QL validator was refactored and covered by unit tests
  • Query persistence was addressed and covered by tests (previous functionality didn't work out of the box and didn't have tests)

How to test

The simplest way to test is via patching installed prebuilt rules (a.k.a. downgrading a prebuilt rule) via Rule Patch API. Please follow steps below

  • Ensure the prebuiltRulesCustomizationEnabled feature flag is enabled
  • Run Kibana locally
  • Install an ES|QL prebuilt rule, e.g. AWS Bedrock Guardrails Detected Multiple Violations by a Single User Over a Session with rule_id 0cd2f3e6-41da-40e6-b28b-466f688f00a6
  • Patch the installed rule by running a query below
curl -X PATCH --user elastic:changeme  -H 'Content-Type: application/json' -H 'kbn-xsrf: 123' -H "elastic-api-version: 2023-10-31" -d '{"rule_id":"0cd2f3e6-41da-40e6-b28b-466f688f00a6","version":1,"query":"from logs-*","language":"esql"}' http://localhost:5601/kbn/api/detection_engine/rules
  • Open Detection Rules (SIEM) Page -> Rule Updates -> click on AWS Bedrock Guardrails Detected Multiple Violations by a Single User Over a Session rule -> expand EQL Query to see EQL Query -> press Edit button

Screenshots

image image

@maximpn maximpn added release_note:skip Skip the PR/issue when compiling release notes v9.0.0 Team:Detections and Resp Security Detection Response Team Team: SecuritySolution Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc. Team:Detection Rule Management Security Detection Rule Management Team Feature:Prebuilt Detection Rules Security Solution Prebuilt Detection Rules area backport:version Backport to applied version labels v8.17.0 labels Nov 12, 2024
@maximpn maximpn self-assigned this Nov 12, 2024
@elasticmachine
Copy link
Contributor

🤖 Jobs for this PR can be triggered through checkboxes. 🚧

ℹ️ To trigger the CI, please tick the checkbox below 👇

  • Click to trigger kibana-pull-request for this PR!
  • Click to trigger kibana-deploy-project-from-pr for this PR!

@maximpn maximpn force-pushed the add-esql-query-editable-component branch 3 times, most recently from 3c63ea7 to 5a0566c Compare November 14, 2024 12:11
@maximpn maximpn marked this pull request as ready for review November 14, 2024 14:01
@maximpn maximpn requested review from a team as code owners November 14, 2024 14:01
@elasticmachine
Copy link
Contributor

Pinging @elastic/security-detections-response (Team:Detections and Resp)

@maximpn maximpn requested a review from xcrzx November 14, 2024 14:01
@elasticmachine
Copy link
Contributor

Pinging @elastic/security-solution (Team: SecuritySolution)

@elasticmachine
Copy link
Contributor

Pinging @elastic/security-detection-rule-management (Team:Detection Rule Management)

@xcrzx
Copy link
Contributor

xcrzx commented Nov 27, 2024

Probably not related to this PR, but the help text doesn't seem to be correct here:

image

@xcrzx
Copy link
Contributor

xcrzx commented Nov 27, 2024

Kibana crashes when I try to open the upgrade flyout for this specific rule: Attempts to Brute Force a Microsoft 365 User Account

image

@maximpn
Copy link
Contributor Author

maximpn commented Nov 27, 2024

Hi @xcrzx,

are unrelated to this PR

@xcrzx
Copy link
Contributor

xcrzx commented Nov 27, 2024

Do you have a list of issues that will be addressed separately?

@maximpn
Copy link
Contributor Author

maximpn commented Nov 27, 2024

Do you have a list of issues that will be addressed separately?

The ticket #171520 has tasks (under TODO (click to expand). Add help texts is there primarily for popup texts. Added your screenshot to make visually clear.

@xcrzx
Copy link
Contributor

xcrzx commented Nov 27, 2024

Not directly related to this PR, the diff view highlights changes incorrectly:

image

Removed parts (in bold) are in the wrong places

@xcrzx
Copy link
Contributor

xcrzx commented Nov 27, 2024

Auto-merge is also behaving incorrectly for the query fields. Incoming changes are simply ignored, and the current version is proposed as the target.

For context:

  • The target version includes a new line added to the query.
  • The current version has a different line modified.

There’s no actual conflict since different lines were touched, yet the merge does not account for the incoming changes.

image

The correctly merged version would look like this:

image

@xcrzx
Copy link
Contributor

xcrzx commented Nov 27, 2024

Not sure if the editor itself is broken or there's something related to its integration, but when I try to go to a new line (by hitting Enter), random symbols are inserted instead:

Screen.Recording.2024-11-27.at.13.22.58.mov

Copy link
Contributor

@xcrzx xcrzx left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tested the PR locally, found one regression in the ESQL field validation function, but other than that looks okay.

I mentioned several other issues during testing, but they can be addressed separately. I'll create tickets

@maximpn maximpn requested a review from xcrzx November 27, 2024 16:49
@maximpn
Copy link
Contributor Author

maximpn commented Nov 27, 2024

@xcrzx,

Thanks for review. I addressed you last comments. Issue you found in #199887 (comment), #199887 (comment) and #199887 (comment) lay out of scope of this PR. Let's address them separately.

@vitaliidm
Copy link
Contributor

The recent changes in ES|QL query columns fetching, when errors are not handled, pollute dev console. Every input error is displaying there now

Screen.Recording.2024-11-28.at.10.01.54.mov

Copy link
Contributor

@xcrzx xcrzx left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verified the latest changes locally regarding rule validation, all good. Thank you @maximpn 👍

@maximpn
Copy link
Contributor Author

maximpn commented Nov 28, 2024

The recent changes in ES|QL query columns fetching, when errors are not handled, pollute dev console. Every input error is displaying there now

@vitaliidm,

Yes. It's a known drawback. We need to fetch ES|QL columns for the other input via useEsqlQueryColumns hook. Since ES|QL query isn't valid getESQLQueryColumns() throws an error. I'm trying to find a solution to suppress unwanted errors in the console.

@vitaliidm
Copy link
Contributor

@maximpn

The nature of ES|QL API is that it returns error on invalid input from user. It includes syntax error, not fully finished query or missing index/data on server side(query is valid from syntax wise).

That's why we had to catch error from response and process it, depends on our case:

  • suppress when retrieving columns
  • resurface when doing validation

Now, all error go through.
I suggest to revert to previous implementation or have more sophisticated error catch: suppress only 400, throw rest of errors.
But since error anyway is shown in query bar error, I don't think we really need it

@maximpn maximpn force-pushed the add-esql-query-editable-component branch from 08d36c7 to 625ffa9 Compare November 28, 2024 13:01
@maximpn
Copy link
Contributor Author

maximpn commented Nov 28, 2024

@vitaliidm,

I rolled back the last commit.

@maximpn maximpn enabled auto-merge (squash) November 28, 2024 13:02
@maximpn maximpn merged commit e55232f into elastic:main Nov 28, 2024
8 checks passed
@kibanamachine
Copy link
Contributor

Starting backport for target branches: 8.x

https://github.com/elastic/kibana/actions/runs/12071134090

@elasticmachine
Copy link
Contributor

💚 Build Succeeded

Metrics [docs]

Module Count

Fewer modules leads to a faster build time

id before after diff
securitySolution 6264 6274 +10

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id before after diff
securitySolution 13.4MB 13.4MB +2.0KB
Unknown metric groups

References to deprecated APIs

id before after diff
@kbn/securitysolution-utils 1 0 -1
securitySolution 437 435 -2
total -3

History

cc @maximpn

@kibanamachine
Copy link
Contributor

💔 All backports failed

Status Branch Result
8.x Backport failed because of merge conflicts

Manual backport

To create the backport manually run:

node scripts/backport --pr 199887

Questions ?

Please refer to the Backport tool documentation

@maximpn maximpn deleted the add-esql-query-editable-component branch November 28, 2024 14:56
@maximpn
Copy link
Contributor Author

maximpn commented Nov 28, 2024

💚 All backports created successfully

Status Branch Result
8.x

Note: Successful backport PRs will be merged automatically after passing CI.

Questions ?

Please refer to the Backport tool documentation

maximpn added a commit to maximpn/kibana that referenced this pull request Nov 28, 2024
**Partially addresses:** elastic#171520

## Summary

This PR adds is built on top of elastic#193828 and elastic#196948 and adds an ES|QL Query editable component for Three Way Diff tab's final edit side of the upgrade prebuilt rule workflow.

## Details

This PR extracts ES|QL Query edit component from Define rule form step and makes it reusable. The following changes were made

- ES|QL validator was refactored and covered by unit tests
- Query persistence was addressed and covered by tests (previous functionality didn't work out of the box and didn't have tests)

## How to test

The simplest way to test is via patching installed prebuilt rules (a.k.a. downgrading a prebuilt rule) via Rule Patch API. Please follow steps below

- Ensure the `prebuiltRulesCustomizationEnabled` feature flag is enabled
- Run Kibana locally
- Install an ES|QL prebuilt rule, e.g. `AWS Bedrock Guardrails Detected Multiple Violations by a Single User Over a Session` with rule_id `0cd2f3e6-41da-40e6-b28b-466f688f00a6`
- Patch the installed rule by running a query below

```bash
curl -X PATCH --user elastic:changeme  -H 'Content-Type: application/json' -H 'kbn-xsrf: 123' -H "elastic-api-version: 2023-10-31" -d '{"rule_id":"0cd2f3e6-41da-40e6-b28b-466f688f00a6","version":1,"query":"from logs-*","language":"esql"}' http://localhost:5601/kbn/api/detection_engine/rules
```

- Open `Detection Rules (SIEM)` Page -> `Rule Updates` -> click on `AWS Bedrock Guardrails Detected Multiple Violations by a Single User Over a Session` rule -> expand `EQL Query` to see EQL Query -> press `Edit` button

## Screenshots

<img width="2550" alt="image" src="https://github.com/user-attachments/assets/f65d04d5-9fd9-4d3f-8741-eba04a3be8a6">

<img width="2552" alt="image" src="https://github.com/user-attachments/assets/dd0a2613-5262-44b2-bbeb-d0ed34d57d9c">

(cherry picked from commit e55232f)

# Conflicts:
#	x-pack/plugins/security_solution/public/detection_engine/rule_creation/logic/esql_validator.test.ts
#	x-pack/plugins/security_solution/public/detection_engine/rule_creation/logic/esql_validator.ts
#	x-pack/plugins/translations/translations/zh-CN.json
maximpn added a commit that referenced this pull request Nov 29, 2024
#202224)

# Backport

This will backport the following commits from `main` to `8.x`:
- [[Security Solution] Add ES|QL Query editable component
(#199887)](#199887)

<!--- Backport version: 8.9.8 -->

### Questions ?
Please refer to the [Backport tool
documentation](https://github.com/sqren/backport)

<!--BACKPORT [{"author":{"name":"Maxim
Palenov","email":"maxim.palenov@elastic.co"},"sourceCommit":{"committedDate":"2024-11-28T14:48:48Z","message":"[Security
Solution] Add ES|QL Query editable component (#199887)\n\n**Partially
addresses:** https://github.com/elastic/kibana/issues/171520\r\n\r\n##
Summary\r\n\r\nThis PR adds is built on top of
#193828 and
#196948 and adds an ES|QL Query
editable component for Three Way Diff tab's final edit side of the
upgrade prebuilt rule workflow.\r\n\r\n## Details\r\n\r\nThis PR
extracts ES|QL Query edit component from Define rule form step and makes
it reusable. The following changes were made\r\n\r\n- ES|QL validator
was refactored and covered by unit tests\r\n- Query persistence was
addressed and covered by tests (previous functionality didn't work out
of the box and didn't have tests)\r\n\r\n## How to test\r\n\r\nThe
simplest way to test is via patching installed prebuilt rules (a.k.a.
downgrading a prebuilt rule) via Rule Patch API. Please follow steps
below\r\n\r\n- Ensure the `prebuiltRulesCustomizationEnabled` feature
flag is enabled\r\n- Run Kibana locally\r\n- Install an ES|QL prebuilt
rule, e.g. `AWS Bedrock Guardrails Detected Multiple Violations by a
Single User Over a Session` with rule_id
`0cd2f3e6-41da-40e6-b28b-466f688f00a6`\r\n- Patch the installed rule by
running a query below\r\n\r\n```bash\r\ncurl -X PATCH --user
elastic:changeme -H 'Content-Type: application/json' -H 'kbn-xsrf: 123'
-H \"elastic-api-version: 2023-10-31\" -d
'{\"rule_id\":\"0cd2f3e6-41da-40e6-b28b-466f688f00a6\",\"version\":1,\"query\":\"from
logs-*\",\"language\":\"esql\"}'
http://localhost:5601/kbn/api/detection_engine/rules\r\n```\r\n\r\n-
Open `Detection Rules (SIEM)` Page -> `Rule Updates` -> click on `AWS
Bedrock Guardrails Detected Multiple Violations by a Single User Over a
Session` rule -> expand `EQL Query` to see EQL Query -> press `Edit`
button\r\n\r\n## Screenshots\r\n\r\n<img width=\"2550\" alt=\"image\"
src=\"https://github.com/user-attachments/assets/f65d04d5-9fd9-4d3f-8741-eba04a3be8a6\">\r\n\r\n<img
width=\"2552\" alt=\"image\"
src=\"https://github.com/user-attachments/assets/dd0a2613-5262-44b2-bbeb-d0ed34d57d9c\">","sha":"e55232f87732594eb9502993766203a219be44db","branchLabelMapping":{"^v9.0.0$":"main","^v8.18.0$":"8.x","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:skip","v9.0.0","Team:Detections
and Resp","Team: SecuritySolution","Team:Detection Rule
Management","Feature:Prebuilt Detection
Rules","backport:version","v8.18.0"],"number":199887,"url":"https://github.com/elastic/kibana/pull/199887","mergeCommit":{"message":"[Security
Solution] Add ES|QL Query editable component (#199887)\n\n**Partially
addresses:** https://github.com/elastic/kibana/issues/171520\r\n\r\n##
Summary\r\n\r\nThis PR adds is built on top of
#193828 and
#196948 and adds an ES|QL Query
editable component for Three Way Diff tab's final edit side of the
upgrade prebuilt rule workflow.\r\n\r\n## Details\r\n\r\nThis PR
extracts ES|QL Query edit component from Define rule form step and makes
it reusable. The following changes were made\r\n\r\n- ES|QL validator
was refactored and covered by unit tests\r\n- Query persistence was
addressed and covered by tests (previous functionality didn't work out
of the box and didn't have tests)\r\n\r\n## How to test\r\n\r\nThe
simplest way to test is via patching installed prebuilt rules (a.k.a.
downgrading a prebuilt rule) via Rule Patch API. Please follow steps
below\r\n\r\n- Ensure the `prebuiltRulesCustomizationEnabled` feature
flag is enabled\r\n- Run Kibana locally\r\n- Install an ES|QL prebuilt
rule, e.g. `AWS Bedrock Guardrails Detected Multiple Violations by a
Single User Over a Session` with rule_id
`0cd2f3e6-41da-40e6-b28b-466f688f00a6`\r\n- Patch the installed rule by
running a query below\r\n\r\n```bash\r\ncurl -X PATCH --user
elastic:changeme -H 'Content-Type: application/json' -H 'kbn-xsrf: 123'
-H \"elastic-api-version: 2023-10-31\" -d
'{\"rule_id\":\"0cd2f3e6-41da-40e6-b28b-466f688f00a6\",\"version\":1,\"query\":\"from
logs-*\",\"language\":\"esql\"}'
http://localhost:5601/kbn/api/detection_engine/rules\r\n```\r\n\r\n-
Open `Detection Rules (SIEM)` Page -> `Rule Updates` -> click on `AWS
Bedrock Guardrails Detected Multiple Violations by a Single User Over a
Session` rule -> expand `EQL Query` to see EQL Query -> press `Edit`
button\r\n\r\n## Screenshots\r\n\r\n<img width=\"2550\" alt=\"image\"
src=\"https://github.com/user-attachments/assets/f65d04d5-9fd9-4d3f-8741-eba04a3be8a6\">\r\n\r\n<img
width=\"2552\" alt=\"image\"
src=\"https://github.com/user-attachments/assets/dd0a2613-5262-44b2-bbeb-d0ed34d57d9c\">","sha":"e55232f87732594eb9502993766203a219be44db"}},"sourceBranch":"main","suggestedTargetBranches":["8.x"],"targetPullRequestStates":[{"branch":"main","label":"v9.0.0","labelRegex":"^v9.0.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/199887","number":199887,"mergeCommit":{"message":"[Security
Solution] Add ES|QL Query editable component (#199887)\n\n**Partially
addresses:** https://github.com/elastic/kibana/issues/171520\r\n\r\n##
Summary\r\n\r\nThis PR adds is built on top of
#193828 and
#196948 and adds an ES|QL Query
editable component for Three Way Diff tab's final edit side of the
upgrade prebuilt rule workflow.\r\n\r\n## Details\r\n\r\nThis PR
extracts ES|QL Query edit component from Define rule form step and makes
it reusable. The following changes were made\r\n\r\n- ES|QL validator
was refactored and covered by unit tests\r\n- Query persistence was
addressed and covered by tests (previous functionality didn't work out
of the box and didn't have tests)\r\n\r\n## How to test\r\n\r\nThe
simplest way to test is via patching installed prebuilt rules (a.k.a.
downgrading a prebuilt rule) via Rule Patch API. Please follow steps
below\r\n\r\n- Ensure the `prebuiltRulesCustomizationEnabled` feature
flag is enabled\r\n- Run Kibana locally\r\n- Install an ES|QL prebuilt
rule, e.g. `AWS Bedrock Guardrails Detected Multiple Violations by a
Single User Over a Session` with rule_id
`0cd2f3e6-41da-40e6-b28b-466f688f00a6`\r\n- Patch the installed rule by
running a query below\r\n\r\n```bash\r\ncurl -X PATCH --user
elastic:changeme -H 'Content-Type: application/json' -H 'kbn-xsrf: 123'
-H \"elastic-api-version: 2023-10-31\" -d
'{\"rule_id\":\"0cd2f3e6-41da-40e6-b28b-466f688f00a6\",\"version\":1,\"query\":\"from
logs-*\",\"language\":\"esql\"}'
http://localhost:5601/kbn/api/detection_engine/rules\r\n```\r\n\r\n-
Open `Detection Rules (SIEM)` Page -> `Rule Updates` -> click on `AWS
Bedrock Guardrails Detected Multiple Violations by a Single User Over a
Session` rule -> expand `EQL Query` to see EQL Query -> press `Edit`
button\r\n\r\n## Screenshots\r\n\r\n<img width=\"2550\" alt=\"image\"
src=\"https://github.com/user-attachments/assets/f65d04d5-9fd9-4d3f-8741-eba04a3be8a6\">\r\n\r\n<img
width=\"2552\" alt=\"image\"
src=\"https://github.com/user-attachments/assets/dd0a2613-5262-44b2-bbeb-d0ed34d57d9c\">","sha":"e55232f87732594eb9502993766203a219be44db"}},{"branch":"8.x","label":"v8.18.0","labelRegex":"^v8.18.0$","isSourceBranch":false,"state":"NOT_CREATED"}]}]
BACKPORT-->
CAWilson94 pushed a commit to CAWilson94/kibana that referenced this pull request Dec 12, 2024
**Partially addresses:** elastic#171520

## Summary

This PR adds is built on top of elastic#193828 and elastic#196948 and adds an ES|QL Query editable component for Three Way Diff tab's final edit side of the upgrade prebuilt rule workflow.

## Details

This PR extracts ES|QL Query edit component from Define rule form step and makes it reusable. The following changes were made

- ES|QL validator was refactored and covered by unit tests
- Query persistence was addressed and covered by tests (previous functionality didn't work out of the box and didn't have tests)

## How to test

The simplest way to test is via patching installed prebuilt rules (a.k.a. downgrading a prebuilt rule) via Rule Patch API. Please follow steps below

- Ensure the `prebuiltRulesCustomizationEnabled` feature flag is enabled
- Run Kibana locally
- Install an ES|QL prebuilt rule, e.g. `AWS Bedrock Guardrails Detected Multiple Violations by a Single User Over a Session` with rule_id `0cd2f3e6-41da-40e6-b28b-466f688f00a6`
- Patch the installed rule by running a query below

```bash
curl -X PATCH --user elastic:changeme  -H 'Content-Type: application/json' -H 'kbn-xsrf: 123' -H "elastic-api-version: 2023-10-31" -d '{"rule_id":"0cd2f3e6-41da-40e6-b28b-466f688f00a6","version":1,"query":"from logs-*","language":"esql"}' http://localhost:5601/kbn/api/detection_engine/rules
```

- Open `Detection Rules (SIEM)` Page -> `Rule Updates` -> click on `AWS Bedrock Guardrails Detected Multiple Violations by a Single User Over a Session` rule -> expand `EQL Query` to see EQL Query -> press `Edit` button

## Screenshots

<img width="2550" alt="image" src="https://github.com/user-attachments/assets/f65d04d5-9fd9-4d3f-8741-eba04a3be8a6">

<img width="2552" alt="image" src="https://github.com/user-attachments/assets/dd0a2613-5262-44b2-bbeb-d0ed34d57d9c">
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backport:version Backport to applied version labels Feature:Prebuilt Detection Rules Security Solution Prebuilt Detection Rules area release_note:skip Skip the PR/issue when compiling release notes Team:Detection Rule Management Security Detection Rule Management Team Team:Detections and Resp Security Detection Response Team Team: SecuritySolution Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc. v8.18.0 v9.0.0
Projects
None yet
Development

Successfully merging this pull request may close these issues.

7 participants