Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Security Solution] Add EQL query editable component with EQL options fields #199115

Merged
merged 28 commits into from
Nov 22, 2024

Conversation

maximpn
Copy link
Contributor

@maximpn maximpn commented Nov 6, 2024

Partially addresses: #171520

Summary

This PR adds is built on top of #193828 and #196948 and adds an EQL Query editable component with EQL Options fields (event_category_override, timestamp_field and tiebreaker_field) for Three Way Diff tab's final edit side of the upgrade prebuilt rule workflow.

Details

This PR make a set of changes to make existing EQL Query bar component easily reusable and type safe when used in forms. In particular the following was done

  • EQL query bar was wrapped in EqlQueryEdit component with UseField inside. It helps to make it type safe avoiding issues like passing invalid types to EqlQueryBar. UseField types component properties as Record<string, any> so potentially any refactoring can break some functionality. For example code in Timeline passes DataViewSpec where DataViewBase is expected while these two types aren't fully compatible.
  • Validation was added directly to EqlQueryEdit. Passing field configuration to UseField rewrites field configuration defined in from schema. It leads to cases when validation is defined in both form schema and as a field configuration for UseFields. Additionally we can reduce reusing complexity by incapsulating absolutely required validation in EqlQueryEdit component.
  • Empty string tiebreakerField was removed in Timelines. tiebreakerField is part of EQL options used for EQL validation. EQL validation endpoint /internal/search/eql returns an error when an empty string provided for tiebreakerField. This problem didn't surface earlier since It looks like EQL options weren't provided correctly before this PR. Timeline EQL validation requests were sent without EQL options.

How to test

The simplest way to test is via patching installed prebuilt rules via Rule Patch API. Please follow steps below

  • Ensure the prebuiltRulesCustomizationEnabled feature flag is enabled
  • Run Kibana locally
  • Install an EQL prebuilt rule, e.g. Potential Code Execution via Postgresql with rule_id 2a692072-d78d-42f3-a48a-775677d79c4e
  • Patch the installed rule by running a query below
curl -X PATCH --user elastic:changeme  -H 'Content-Type: application/json' -H 'kbn-xsrf: 123' -H "elastic-api-version: 2023-10-31" -d '{"rule_id":"2a692072-d78d-42f3-a48a-775677d79c4e","version":1,"query":"process where process.name == \"cmd.exe\"","language":"eql","event_category_override": "test","timestamp_field": "@timestamp","tiebreaker_field": "tiebreaker"}' http://localhost:5601/kbn/api/detection_engine/rules
  • Open Detection Rules (SIEM) Page -> Rule Updates -> click on Potential Code Execution via Postgresql rule -> expand EQL Query to see EQL Query -> press Edit button

Screenshots

  • EQL Query in Prebuilt Rules Update workflow
image
  • event_category_override + tiebreaker_field + timestamp_field (aka EQL options) in Prebuilt Rules Update workflow
image
  • Examples of invalid EQL
image image

@maximpn maximpn added release_note:skip Skip the PR/issue when compiling release notes v9.0.0 Team:Detections and Resp Security Detection Response Team Team: SecuritySolution Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc. Team:Detection Rule Management Security Detection Rule Management Team Feature:Prebuilt Detection Rules Security Solution Prebuilt Detection Rules area backport:prev-minor Backport to (8.x) the previous minor version (i.e. one version back from main) v8.17.0 labels Nov 6, 2024
@maximpn maximpn self-assigned this Nov 6, 2024
@maximpn maximpn requested a review from nikitaindik November 6, 2024 14:28
@@ -33,7 +33,6 @@ export const timelineDefaults: SubsetTimelineModel &
description: '',
eqlOptions: {
eventCategoryField: 'event.category',
tiebreakerField: '',
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

tiebreakerField as an empty doesn't look correct for the EQL validator. It sends requests to /internal/search/eql to validate the request. It returns an error when a tiebreaker field has an empty string.

Additionally it looks like EQL options weren't provided correctly before this PR. Timeline EQL validation requests were sent without EQL options.

@maximpn maximpn marked this pull request as ready for review November 7, 2024 07:49
@maximpn maximpn requested review from a team as code owners November 7, 2024 07:49
@maximpn maximpn requested a review from nkhristinin November 7, 2024 07:49
@elasticmachine
Copy link
Contributor

Pinging @elastic/security-detections-response (Team:Detections and Resp)

@elasticmachine
Copy link
Contributor

Pinging @elastic/security-solution (Team: SecuritySolution)

@elasticmachine
Copy link
Contributor

Pinging @elastic/security-detection-rule-management (Team:Detection Rule Management)

@banderror banderror added backport:version Backport to applied version labels and removed backport:prev-minor Backport to (8.x) the previous minor version (i.e. one version back from main) labels Nov 7, 2024
@maximpn maximpn force-pushed the add-eql-query-edit-component branch from 2291ad0 to db856ba Compare November 7, 2024 21:15
@yctercero
Copy link
Contributor

Thanks for all the work here! Pulled down and tested in rule creation/edit flow. So far LGTM. Confirmed that logic around disabling suppression for sequence queries remains. cc @dhurley14

QueryLanguageEnum,
} from '../../../../../../../../../common/api/detection_engine';
import { EqlQueryEditAdapter } from './eql_query_edit_adapter';
// import {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Commented out code. Can be removed.

eqlQuery: {
validations: [
// {
// validator: debounceAsync(eqlValidator, 300),
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Commented out code. Can be removed.

);
}

const kqlQuerySchema = {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since there are no validations we can probably remove the schema

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

RuleFieldEditFormWrapper's ruleFieldFormSchema property is required. We still should pass an empty object. So removing it completely it's possible.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Got it. Then let's rename it to eqlQuerySchema.

@nikitaindik
Copy link
Contributor

nikitaindik commented Nov 11, 2024

For some reason getting an EQL validation error in the upgrade flyout, but on the the Rule Editing page.

The error is shown right away, after opening the edit mode. Prevents me from saving.

Scherm­afbeelding 2024-11-11 om 16 41 06
EQL query I used
api where host.os.type == "win" and 
  process.Ext.api.name in ("OpenProcess", "OpenThread") and Target.process.name : "lsass.exe" and 
  not 
  (
    process.executable : (
        "?:\\ProgramData\\GetSupportService*\\Updates\\Update_*.exe",
        "?:\\ProgramData\\Microsoft\\Windows Defender\\Platform\\*\\MsMpEng.exe",
        "?:\\Program Files (x86)\\Asiainfo Security\\OfficeScan Client\\NTRTScan.exe",
        "?:\\Program Files (x86)\\Blackpoint\\SnapAgent\\SnapAgent.exe",
        "?:\\Program Files (x86)\\CheckPoint\\Endpoint Security\\EFR\\EFRService.exe",
        "?:\\Program Files (x86)\\CyberCNSAgent\\osqueryi.exe",
        "?:\\Program Files (x86)\\cisco\\cisco anyconnect secure mobility client\\vpnagent.exe",
        "?:\\Program Files (x86)\\cisco\\cisco anyconnect secure mobility client\\aciseagent.exe",
        "?:\\Program Files (x86)\\cisco\\cisco anyconnect secure mobility client\\vpndownloader.exe",
        "?:\\Program Files (x86)\\eScan\\reload.exe",
        "?:\\Program Files (x86)\\Google\\Update\\GoogleUpdate.exe",
        "?:\\Program Files (x86)\\Kaspersky Lab\\*\\avp.exe",
        "?:\\Program Files (x86)\\microsoft intune management extension\\microsoft.management.services.intunewindowsagent.exe",
        "?:\\Program Files (x86)\\N-able Technologies\\Reactive\\bin\\NableReactiveManagement.exe",
        "?:\\Program Files (x86)\\N-able Technologies\\Windows Agent\\bin\\agent.exe",
        "?:\\Program Files (x86)\\Tanium\\Tanium Client\\TaniumClient.exe",
        "?:\\Program Files (x86)\\Trend Micro\\*\\CCSF\\TmCCSF.exe",
        "?:\\Program Files (x86)\\Trend Micro\\Security Agent\\TMASutility.exe",
        "?:\\Program Files*\\Windows Defender\\MsMpEng.exe",
        "?:\\Program Files\\Bitdefender\\Endpoint Security\\EPSecurityService.exe",
        "?:\\Program Files\\Cisco\\AMP\\*\\sfc.exe",
        "?:\\Program Files\\Common Files\\McAfee\\AVSolution\\mcshield.exe",
        "?:\\Program Files\\EA\\AC\\EAAntiCheat.GameService.exe",
        "?:\\Program Files\\Elastic\\Agent\\data\\elastic-agent-*\\components\\agentbeat.exe",
        "?:\\Program Files\\Elastic\\Agent\\data\\elastic-agent-*\\components\\metricbeat.exe",
        "?:\\Program Files\\Elastic\\Agent\\data\\elastic-agent-*\\components\\osqueryd.exe",
        "?:\\Program Files\\Elastic\\Agent\\data\\elastic-agent-*\\components\\packetbeat.exe",
        "?:\\Program Files\\ESET\\ESET Security\\ekrn.exe",
        "?:\\Program Files\\Fortinet\\FortiClient\\FortiProxy.exe",
        "?:\\Program Files\\Fortinet\\FortiClient\\FortiSSLVPNdaemon.exe",
        "?:\\Program Files\\Goverlan Inc\\GoverlanAgent\\GovAgentx64.exe",
        "?:\\Program Files\\Huntress\\HuntressAgent.exe",
        "?:\\Program Files\\LogicMonitor\\Agent\\bin\\sbshutdown.exe",
        "?:\\Program Files\\Malwarebytes\\Anti-Malware\\MBAMService.exe",
        "?:\\Program Files\\Microsoft Monitoring Agent\\Agent\\Health Service State\\*\\pmfexe.exe", 
        "?:\\Program Files\\Microsoft Security Client\\MsMpEng.exe",
        "?:\\Program Files\\Qualys\\QualysAgent\\QualysAgent.exe",
        "?:\\Program Files\\smart-x\\controlupagent\\version*\\cuagent.exe",
        "?:\\Program Files\\TDAgent\\ossec-agent\\ossec-agent.exe",
        "?:\\Program Files\\Topaz OFD\\Warsaw\\core.exe",
        "?:\\Program Files\\Trend Micro\\Deep Security Agent\\netagent\\tm_netagent.exe",
        "?:\\Program Files\\VMware\\VMware Tools\\vmtoolsd.exe",
        "?:\\Program Files\\Windows Defender Advanced Threat Protection\\MsSense.exe",
        "?:\\Program Files\\Wise\\Wise Memory Optimizer\\WiseMemoryOptimzer.exe",
        "?:\\Windows\\AdminArsenal\\PDQDeployRunner\\*\\exec\\Sysmon64.exe",
        "?:\\Windows\\Sysmon.exe",
        "?:\\Windows\\Sysmon64.exe",
        "?:\\Windows\\System32\\csrss.exe",
        "?:\\Windows\\System32\\MRT.exe",
        "?:\\Windows\\System32\\msiexec.exe",
        "?:\\Windows\\System32\\taskhostw.exe",
        "?:\\Windows\\System32\\RtkAudUService64.exe",
        "?:\\Windows\\System32\\wbem\\WmiPrvSE.exe",
        "?:\\Windows\\SysWOW64\\wbem\\WmiPrvSE.exe",
        "?:\\Windows\\tenable_mw_scan_142a90001fb65e0beb1751cc8c63edd0.exe"
    ) and not ?process.code_signature.trusted == false
  )

@maximpn maximpn force-pushed the add-eql-query-edit-component branch from 326eeb0 to 64b371f Compare November 11, 2024 17:00
@maximpn maximpn force-pushed the add-eql-query-edit-component branch from d11b456 to 45bfabc Compare November 21, 2024 16:57
@maximpn maximpn enabled auto-merge (squash) November 21, 2024 22:09
@elasticmachine
Copy link
Contributor

elasticmachine commented Nov 21, 2024

💔 Build Failed

Failed CI Steps

Test Failures

  • [job] [logs] FTR Configs #69 / discover/group4 data view flyout "after all" hook for "update data view with no time field"
  • [job] [logs] FTR Configs #69 / discover/group4 data view flyout create saved data view
  • [job] [logs] FTR Configs #91 / Rules Management - Prebuilt Rules - Prebuilt Rules Management @ess @serverless @skipInServerlessMKI Perform Prebuilt Rules Upgrades - mode: SPECIFIC_RULES successful updates successfully resolves a non-resolvable conflict by using pick_version:RESOLVED for that field
  • [job] [logs] FTR Configs #106 / Rules Management - Prebuilt Rules - Prebuilt Rules Management @ess @serverless @skipInServerlessMKI Perform Prebuilt Rules Upgrades - mode: SPECIFIC_RULES successful updates successfully resolves a non-resolvable conflict by using pick_version:RESOLVED for that field
  • [job] [logs] FTR Configs #106 / Rules Management - Prebuilt Rules - Prebuilt Rules Management @ess @serverless @skipInServerlessMKI Perform Prebuilt Rules Upgrades - mode: SPECIFIC_RULES successful updates successfully resolves a non-resolvable conflict by using pick_version:RESOLVED for that field
  • [job] [logs] FTR Configs #91 / Rules Management - Prebuilt Rules - Prebuilt Rules Management @ess @serverless @skipInServerlessMKI Perform Prebuilt Rules Upgrades - mode: SPECIFIC_RULES successful updates successfully resolves a non-resolvable conflict by using pick_version:RESOLVED for that field

Metrics [docs]

Module Count

Fewer modules leads to a faster build time

id before after diff
lists 342 343 +1
securitySolution 6218 6222 +4
total +5

Public APIs missing comments

Total count of every public API that lacks a comment. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats comments for more detailed information.

id before after diff
@kbn/securitysolution-utils 54 55 +1

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id before after diff
securitySolution 13.4MB 13.4MB +3.5KB
Unknown metric groups

API count

id before after diff
@kbn/securitysolution-utils 60 63 +3

References to deprecated APIs

id before after diff
securitySolution 438 437 -1

History

cc @maximpn

@maximpn maximpn force-pushed the add-eql-query-edit-component branch from 568ea87 to 9f5d1f8 Compare November 22, 2024 05:38
@maximpn maximpn merged commit c0c803c into elastic:main Nov 22, 2024
48 checks passed
@kibanamachine
Copy link
Contributor

Starting backport for target branches: 8.x

https://github.com/elastic/kibana/actions/runs/11968313271

@kibanamachine
Copy link
Contributor

💔 All backports failed

Status Branch Result
8.x Backport failed because of merge conflicts

Manual backport

To create the backport manually run:

node scripts/backport --pr 199115

Questions ?

Please refer to the Backport tool documentation

@maximpn
Copy link
Contributor Author

maximpn commented Nov 22, 2024

💚 All backports created successfully

Status Branch Result
8.x

Note: Successful backport PRs will be merged automatically after passing CI.

Questions ?

Please refer to the Backport tool documentation

maximpn added a commit to maximpn/kibana that referenced this pull request Nov 22, 2024
… fields (elastic#199115)

**Partially addresses:** elastic#171520

## Summary

This PR adds is built on top of elastic#193828 and elastic#196948 and adds an EQL Query editable component with EQL Options fields (`event_category_override`, `timestamp_field` and `tiebreaker_field`) for Three Way Diff tab's final edit side of the upgrade prebuilt rule workflow.

## Details

This PR make a set of changes to make existing EQL Query bar component easily reusable and type safe when used in forms. In particular the following was done

- EQL query bar was wrapped in `EqlQueryEdit` component with `UseField` inside. It helps to make it type safe avoiding issues like passing invalid types to `EqlQueryBar`. `UseField` types component properties as `Record<string, any>` so potentially any refactoring can break some functionality. For example code in Timeline passes `DataViewSpec` where `DataViewBase` is expected while these two types aren't fully compatible.
- Validation was added directly to `EqlQueryEdit`. Passing field configuration to `UseField` rewrites field configuration defined in from schema. It leads to cases when validation is defined in both form schema and as a field configuration for `UseFields`. Additionally we can reduce reusing complexity by incapsulating absolutely required validation in `EqlQueryEdit` component.
- Empty string `tiebreakerField` was removed in Timelines. `tiebreakerField` is part of EQL options used for EQL validation. EQL validation endpoint `/internal/search/eql` returns an error when an empty string provided for `tiebreakerField`. This problem didn't surface earlier since It looks like EQL options weren't provided correctly before this PR. Timeline EQL validation requests were sent without EQL options.

## How to test

The simplest way to test is via patching installed prebuilt rules via Rule Patch API. Please follow steps below

- Ensure the `prebuiltRulesCustomizationEnabled` feature flag is enabled
- Run Kibana locally
- Install an EQL prebuilt rule, e.g. `Potential Code Execution via Postgresql` with rule_id `2a692072-d78d-42f3-a48a-775677d79c4e`
- Patch the installed rule by running a query below

```bash
curl -X PATCH --user elastic:changeme  -H 'Content-Type: application/json' -H 'kbn-xsrf: 123' -H "elastic-api-version: 2023-10-31" -d '{"rule_id":"2a692072-d78d-42f3-a48a-775677d79c4e","version":1,"query":"process where process.name == \"cmd.exe\"","language":"eql","event_category_override": "test","timestamp_field": "@timestamp","tiebreaker_field": "tiebreaker"}' http://localhost:5601/kbn/api/detection_engine/rules
```

- Open `Detection Rules (SIEM)` Page -> `Rule Updates` -> click on `Potential Code Execution via Postgresql` rule -> expand `EQL Query` to see EQL Query -> press `Edit` button

## Screenshots

- EQL Query in Prebuilt Rules Update workflow
<img width="2560" alt="image" src="https://github.com/user-attachments/assets/59d157b2-6aca-4b21-95d0-f71a2e174df2">

- event_category_override + tiebreaker_field + timestamp_field (aka EQL options) in Prebuilt Rules Update workflow
<img width="2552" alt="image" src="https://github.com/user-attachments/assets/1886d3b4-98f9-40a7-954c-2a6d4b8e925a">

- Examples of invalid EQL
<img width="2560" alt="image" src="https://github.com/user-attachments/assets/d584deca-7903-45c5-9499-718552df441c">

<img width="2548" alt="image" src="https://github.com/user-attachments/assets/b734e22c-ab62-4624-85d0-e4e6dbb9d523">

(cherry picked from commit c0c803c)

# Conflicts:
#	x-pack/plugins/security_solution/public/timelines/components/timeline/query_bar/eql/index.tsx
maximpn added a commit that referenced this pull request Nov 22, 2024
…ptions fields (#199115) (#201314)

# Backport

This will backport the following commits from `main` to `8.x`:
- [[Security Solution] Add EQL query editable component with EQL options
fields (#199115)](#199115)

<!--- Backport version: 8.9.8 -->

### Questions ?
Please refer to the [Backport tool
documentation](https://github.com/sqren/backport)

<!--BACKPORT [{"author":{"name":"Maxim
Palenov","email":"maxim.palenov@elastic.co"},"sourceCommit":{"committedDate":"2024-11-22T07:27:40Z","message":"[Security
Solution] Add EQL query editable component with EQL options fields
(#199115)\n\n**Partially addresses:**
https://github.com/elastic/kibana/issues/171520\r\n\r\n##
Summary\r\n\r\nThis PR adds is built on top of
#193828 and
#196948 and adds an EQL Query
editable component with EQL Options fields (`event_category_override`,
`timestamp_field` and `tiebreaker_field`) for Three Way Diff tab's final
edit side of the upgrade prebuilt rule workflow.\r\n\r\n##
Details\r\n\r\nThis PR make a set of changes to make existing EQL Query
bar component easily reusable and type safe when used in forms. In
particular the following was done\r\n\r\n- EQL query bar was wrapped in
`EqlQueryEdit` component with `UseField` inside. It helps to make it
type safe avoiding issues like passing invalid types to `EqlQueryBar`.
`UseField` types component properties as `Record<string, any>` so
potentially any refactoring can break some functionality. For example
code in Timeline passes `DataViewSpec` where `DataViewBase` is expected
while these two types aren't fully compatible.\r\n- Validation was added
directly to `EqlQueryEdit`. Passing field configuration to `UseField`
rewrites field configuration defined in from schema. It leads to cases
when validation is defined in both form schema and as a field
configuration for `UseFields`. Additionally we can reduce reusing
complexity by incapsulating absolutely required validation in
`EqlQueryEdit` component.\r\n- Empty string `tiebreakerField` was
removed in Timelines. `tiebreakerField` is part of EQL options used for
EQL validation. EQL validation endpoint `/internal/search/eql` returns
an error when an empty string provided for `tiebreakerField`. This
problem didn't surface earlier since It looks like EQL options weren't
provided correctly before this PR. Timeline EQL validation requests were
sent without EQL options.\r\n\r\n## How to test\r\n\r\nThe simplest way
to test is via patching installed prebuilt rules via Rule Patch API.
Please follow steps below\r\n\r\n- Ensure the
`prebuiltRulesCustomizationEnabled` feature flag is enabled\r\n- Run
Kibana locally\r\n- Install an EQL prebuilt rule, e.g. `Potential Code
Execution via Postgresql` with rule_id
`2a692072-d78d-42f3-a48a-775677d79c4e`\r\n- Patch the installed rule by
running a query below\r\n\r\n```bash\r\ncurl -X PATCH --user
elastic:changeme -H 'Content-Type: application/json' -H 'kbn-xsrf: 123'
-H \"elastic-api-version: 2023-10-31\" -d
'{\"rule_id\":\"2a692072-d78d-42f3-a48a-775677d79c4e\",\"version\":1,\"query\":\"process
where process.name ==
\\\"cmd.exe\\\"\",\"language\":\"eql\",\"event_category_override\":
\"test\",\"timestamp_field\": \"@timestamp\",\"tiebreaker_field\":
\"tiebreaker\"}'
http://localhost:5601/kbn/api/detection_engine/rules\r\n```\r\n\r\n-
Open `Detection Rules (SIEM)` Page -> `Rule Updates` -> click on
`Potential Code Execution via Postgresql` rule -> expand `EQL Query` to
see EQL Query -> press `Edit` button\r\n\r\n## Screenshots\r\n\r\n- EQL
Query in Prebuilt Rules Update workflow\r\n<img width=\"2560\"
alt=\"image\"
src=\"https://github.com/user-attachments/assets/59d157b2-6aca-4b21-95d0-f71a2e174df2\">\r\n\r\n-
event_category_override + tiebreaker_field + timestamp_field (aka EQL
options) in Prebuilt Rules Update workflow\r\n<img width=\"2552\"
alt=\"image\"
src=\"https://github.com/user-attachments/assets/1886d3b4-98f9-40a7-954c-2a6d4b8e925a\">\r\n\r\n-
Examples of invalid EQL\r\n<img width=\"2560\" alt=\"image\"
src=\"https://github.com/user-attachments/assets/d584deca-7903-45c5-9499-718552df441c\">\r\n\r\n<img
width=\"2548\" alt=\"image\"
src=\"https://github.com/user-attachments/assets/b734e22c-ab62-4624-85d0-e4e6dbb9d523\">","sha":"c0c803c8830c10f1df1b204a7d7b859f1f584991","branchLabelMapping":{"^v9.0.0$":"main","^v8.18.0$":"8.x","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:skip","v9.0.0","Team:Detections
and Resp","Team: SecuritySolution","Team:Detection Rule
Management","Feature:Prebuilt Detection
Rules","backport:version","v8.18.0"],"number":199115,"url":"https://github.com/elastic/kibana/pull/199115","mergeCommit":{"message":"[Security
Solution] Add EQL query editable component with EQL options fields
(#199115)\n\n**Partially addresses:**
https://github.com/elastic/kibana/issues/171520\r\n\r\n##
Summary\r\n\r\nThis PR adds is built on top of
#193828 and
#196948 and adds an EQL Query
editable component with EQL Options fields (`event_category_override`,
`timestamp_field` and `tiebreaker_field`) for Three Way Diff tab's final
edit side of the upgrade prebuilt rule workflow.\r\n\r\n##
Details\r\n\r\nThis PR make a set of changes to make existing EQL Query
bar component easily reusable and type safe when used in forms. In
particular the following was done\r\n\r\n- EQL query bar was wrapped in
`EqlQueryEdit` component with `UseField` inside. It helps to make it
type safe avoiding issues like passing invalid types to `EqlQueryBar`.
`UseField` types component properties as `Record<string, any>` so
potentially any refactoring can break some functionality. For example
code in Timeline passes `DataViewSpec` where `DataViewBase` is expected
while these two types aren't fully compatible.\r\n- Validation was added
directly to `EqlQueryEdit`. Passing field configuration to `UseField`
rewrites field configuration defined in from schema. It leads to cases
when validation is defined in both form schema and as a field
configuration for `UseFields`. Additionally we can reduce reusing
complexity by incapsulating absolutely required validation in
`EqlQueryEdit` component.\r\n- Empty string `tiebreakerField` was
removed in Timelines. `tiebreakerField` is part of EQL options used for
EQL validation. EQL validation endpoint `/internal/search/eql` returns
an error when an empty string provided for `tiebreakerField`. This
problem didn't surface earlier since It looks like EQL options weren't
provided correctly before this PR. Timeline EQL validation requests were
sent without EQL options.\r\n\r\n## How to test\r\n\r\nThe simplest way
to test is via patching installed prebuilt rules via Rule Patch API.
Please follow steps below\r\n\r\n- Ensure the
`prebuiltRulesCustomizationEnabled` feature flag is enabled\r\n- Run
Kibana locally\r\n- Install an EQL prebuilt rule, e.g. `Potential Code
Execution via Postgresql` with rule_id
`2a692072-d78d-42f3-a48a-775677d79c4e`\r\n- Patch the installed rule by
running a query below\r\n\r\n```bash\r\ncurl -X PATCH --user
elastic:changeme -H 'Content-Type: application/json' -H 'kbn-xsrf: 123'
-H \"elastic-api-version: 2023-10-31\" -d
'{\"rule_id\":\"2a692072-d78d-42f3-a48a-775677d79c4e\",\"version\":1,\"query\":\"process
where process.name ==
\\\"cmd.exe\\\"\",\"language\":\"eql\",\"event_category_override\":
\"test\",\"timestamp_field\": \"@timestamp\",\"tiebreaker_field\":
\"tiebreaker\"}'
http://localhost:5601/kbn/api/detection_engine/rules\r\n```\r\n\r\n-
Open `Detection Rules (SIEM)` Page -> `Rule Updates` -> click on
`Potential Code Execution via Postgresql` rule -> expand `EQL Query` to
see EQL Query -> press `Edit` button\r\n\r\n## Screenshots\r\n\r\n- EQL
Query in Prebuilt Rules Update workflow\r\n<img width=\"2560\"
alt=\"image\"
src=\"https://github.com/user-attachments/assets/59d157b2-6aca-4b21-95d0-f71a2e174df2\">\r\n\r\n-
event_category_override + tiebreaker_field + timestamp_field (aka EQL
options) in Prebuilt Rules Update workflow\r\n<img width=\"2552\"
alt=\"image\"
src=\"https://github.com/user-attachments/assets/1886d3b4-98f9-40a7-954c-2a6d4b8e925a\">\r\n\r\n-
Examples of invalid EQL\r\n<img width=\"2560\" alt=\"image\"
src=\"https://github.com/user-attachments/assets/d584deca-7903-45c5-9499-718552df441c\">\r\n\r\n<img
width=\"2548\" alt=\"image\"
src=\"https://github.com/user-attachments/assets/b734e22c-ab62-4624-85d0-e4e6dbb9d523\">","sha":"c0c803c8830c10f1df1b204a7d7b859f1f584991"}},"sourceBranch":"main","suggestedTargetBranches":["8.x"],"targetPullRequestStates":[{"branch":"main","label":"v9.0.0","labelRegex":"^v9.0.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/199115","number":199115,"mergeCommit":{"message":"[Security
Solution] Add EQL query editable component with EQL options fields
(#199115)\n\n**Partially addresses:**
https://github.com/elastic/kibana/issues/171520\r\n\r\n##
Summary\r\n\r\nThis PR adds is built on top of
#193828 and
#196948 and adds an EQL Query
editable component with EQL Options fields (`event_category_override`,
`timestamp_field` and `tiebreaker_field`) for Three Way Diff tab's final
edit side of the upgrade prebuilt rule workflow.\r\n\r\n##
Details\r\n\r\nThis PR make a set of changes to make existing EQL Query
bar component easily reusable and type safe when used in forms. In
particular the following was done\r\n\r\n- EQL query bar was wrapped in
`EqlQueryEdit` component with `UseField` inside. It helps to make it
type safe avoiding issues like passing invalid types to `EqlQueryBar`.
`UseField` types component properties as `Record<string, any>` so
potentially any refactoring can break some functionality. For example
code in Timeline passes `DataViewSpec` where `DataViewBase` is expected
while these two types aren't fully compatible.\r\n- Validation was added
directly to `EqlQueryEdit`. Passing field configuration to `UseField`
rewrites field configuration defined in from schema. It leads to cases
when validation is defined in both form schema and as a field
configuration for `UseFields`. Additionally we can reduce reusing
complexity by incapsulating absolutely required validation in
`EqlQueryEdit` component.\r\n- Empty string `tiebreakerField` was
removed in Timelines. `tiebreakerField` is part of EQL options used for
EQL validation. EQL validation endpoint `/internal/search/eql` returns
an error when an empty string provided for `tiebreakerField`. This
problem didn't surface earlier since It looks like EQL options weren't
provided correctly before this PR. Timeline EQL validation requests were
sent without EQL options.\r\n\r\n## How to test\r\n\r\nThe simplest way
to test is via patching installed prebuilt rules via Rule Patch API.
Please follow steps below\r\n\r\n- Ensure the
`prebuiltRulesCustomizationEnabled` feature flag is enabled\r\n- Run
Kibana locally\r\n- Install an EQL prebuilt rule, e.g. `Potential Code
Execution via Postgresql` with rule_id
`2a692072-d78d-42f3-a48a-775677d79c4e`\r\n- Patch the installed rule by
running a query below\r\n\r\n```bash\r\ncurl -X PATCH --user
elastic:changeme -H 'Content-Type: application/json' -H 'kbn-xsrf: 123'
-H \"elastic-api-version: 2023-10-31\" -d
'{\"rule_id\":\"2a692072-d78d-42f3-a48a-775677d79c4e\",\"version\":1,\"query\":\"process
where process.name ==
\\\"cmd.exe\\\"\",\"language\":\"eql\",\"event_category_override\":
\"test\",\"timestamp_field\": \"@timestamp\",\"tiebreaker_field\":
\"tiebreaker\"}'
http://localhost:5601/kbn/api/detection_engine/rules\r\n```\r\n\r\n-
Open `Detection Rules (SIEM)` Page -> `Rule Updates` -> click on
`Potential Code Execution via Postgresql` rule -> expand `EQL Query` to
see EQL Query -> press `Edit` button\r\n\r\n## Screenshots\r\n\r\n- EQL
Query in Prebuilt Rules Update workflow\r\n<img width=\"2560\"
alt=\"image\"
src=\"https://github.com/user-attachments/assets/59d157b2-6aca-4b21-95d0-f71a2e174df2\">\r\n\r\n-
event_category_override + tiebreaker_field + timestamp_field (aka EQL
options) in Prebuilt Rules Update workflow\r\n<img width=\"2552\"
alt=\"image\"
src=\"https://github.com/user-attachments/assets/1886d3b4-98f9-40a7-954c-2a6d4b8e925a\">\r\n\r\n-
Examples of invalid EQL\r\n<img width=\"2560\" alt=\"image\"
src=\"https://github.com/user-attachments/assets/d584deca-7903-45c5-9499-718552df441c\">\r\n\r\n<img
width=\"2548\" alt=\"image\"
src=\"https://github.com/user-attachments/assets/b734e22c-ab62-4624-85d0-e4e6dbb9d523\">","sha":"c0c803c8830c10f1df1b204a7d7b859f1f584991"}},{"branch":"8.x","label":"v8.18.0","labelRegex":"^v8.18.0$","isSourceBranch":false,"state":"NOT_CREATED"}]}]
BACKPORT-->
paulinashakirova pushed a commit to paulinashakirova/kibana that referenced this pull request Nov 26, 2024
… fields (elastic#199115)

**Partially addresses:** elastic#171520

## Summary

This PR adds is built on top of elastic#193828 and elastic#196948 and adds an EQL Query editable component with EQL Options fields (`event_category_override`, `timestamp_field` and `tiebreaker_field`) for Three Way Diff tab's final edit side of the upgrade prebuilt rule workflow.

## Details

This PR make a set of changes to make existing EQL Query bar component easily reusable and type safe when used in forms. In particular the following was done

- EQL query bar was wrapped in `EqlQueryEdit` component with `UseField` inside. It helps to make it type safe avoiding issues like passing invalid types to `EqlQueryBar`. `UseField` types component properties as `Record<string, any>` so potentially any refactoring can break some functionality. For example code in Timeline passes `DataViewSpec` where `DataViewBase` is expected while these two types aren't fully compatible.
- Validation was added directly to `EqlQueryEdit`. Passing field configuration to `UseField` rewrites field configuration defined in from schema. It leads to cases when validation is defined in both form schema and as a field configuration for `UseFields`. Additionally we can reduce reusing complexity by incapsulating absolutely required validation in `EqlQueryEdit` component.
- Empty string `tiebreakerField` was removed in Timelines. `tiebreakerField` is part of EQL options used for EQL validation. EQL validation endpoint `/internal/search/eql` returns an error when an empty string provided for `tiebreakerField`. This problem didn't surface earlier since It looks like EQL options weren't provided correctly before this PR. Timeline EQL validation requests were sent without EQL options.

## How to test

The simplest way to test is via patching installed prebuilt rules via Rule Patch API. Please follow steps below

- Ensure the `prebuiltRulesCustomizationEnabled` feature flag is enabled
- Run Kibana locally
- Install an EQL prebuilt rule, e.g. `Potential Code Execution via Postgresql` with rule_id `2a692072-d78d-42f3-a48a-775677d79c4e`
- Patch the installed rule by running a query below

```bash
curl -X PATCH --user elastic:changeme  -H 'Content-Type: application/json' -H 'kbn-xsrf: 123' -H "elastic-api-version: 2023-10-31" -d '{"rule_id":"2a692072-d78d-42f3-a48a-775677d79c4e","version":1,"query":"process where process.name == \"cmd.exe\"","language":"eql","event_category_override": "test","timestamp_field": "@timestamp","tiebreaker_field": "tiebreaker"}' http://localhost:5601/kbn/api/detection_engine/rules
```

- Open `Detection Rules (SIEM)` Page -> `Rule Updates` -> click on `Potential Code Execution via Postgresql` rule -> expand `EQL Query` to see EQL Query -> press `Edit` button

## Screenshots

- EQL Query in Prebuilt Rules Update workflow
<img width="2560" alt="image" src="https://github.com/user-attachments/assets/59d157b2-6aca-4b21-95d0-f71a2e174df2">

- event_category_override + tiebreaker_field + timestamp_field (aka EQL options) in Prebuilt Rules Update workflow
<img width="2552" alt="image" src="https://github.com/user-attachments/assets/1886d3b4-98f9-40a7-954c-2a6d4b8e925a">

- Examples of invalid EQL
<img width="2560" alt="image" src="https://github.com/user-attachments/assets/d584deca-7903-45c5-9499-718552df441c">

<img width="2548" alt="image" src="https://github.com/user-attachments/assets/b734e22c-ab62-4624-85d0-e4e6dbb9d523">
CAWilson94 pushed a commit to CAWilson94/kibana that referenced this pull request Dec 12, 2024
… fields (elastic#199115)

**Partially addresses:** elastic#171520

## Summary

This PR adds is built on top of elastic#193828 and elastic#196948 and adds an EQL Query editable component with EQL Options fields (`event_category_override`, `timestamp_field` and `tiebreaker_field`) for Three Way Diff tab's final edit side of the upgrade prebuilt rule workflow.

## Details

This PR make a set of changes to make existing EQL Query bar component easily reusable and type safe when used in forms. In particular the following was done

- EQL query bar was wrapped in `EqlQueryEdit` component with `UseField` inside. It helps to make it type safe avoiding issues like passing invalid types to `EqlQueryBar`. `UseField` types component properties as `Record<string, any>` so potentially any refactoring can break some functionality. For example code in Timeline passes `DataViewSpec` where `DataViewBase` is expected while these two types aren't fully compatible.
- Validation was added directly to `EqlQueryEdit`. Passing field configuration to `UseField` rewrites field configuration defined in from schema. It leads to cases when validation is defined in both form schema and as a field configuration for `UseFields`. Additionally we can reduce reusing complexity by incapsulating absolutely required validation in `EqlQueryEdit` component.
- Empty string `tiebreakerField` was removed in Timelines. `tiebreakerField` is part of EQL options used for EQL validation. EQL validation endpoint `/internal/search/eql` returns an error when an empty string provided for `tiebreakerField`. This problem didn't surface earlier since It looks like EQL options weren't provided correctly before this PR. Timeline EQL validation requests were sent without EQL options.

## How to test

The simplest way to test is via patching installed prebuilt rules via Rule Patch API. Please follow steps below

- Ensure the `prebuiltRulesCustomizationEnabled` feature flag is enabled
- Run Kibana locally
- Install an EQL prebuilt rule, e.g. `Potential Code Execution via Postgresql` with rule_id `2a692072-d78d-42f3-a48a-775677d79c4e`
- Patch the installed rule by running a query below

```bash
curl -X PATCH --user elastic:changeme  -H 'Content-Type: application/json' -H 'kbn-xsrf: 123' -H "elastic-api-version: 2023-10-31" -d '{"rule_id":"2a692072-d78d-42f3-a48a-775677d79c4e","version":1,"query":"process where process.name == \"cmd.exe\"","language":"eql","event_category_override": "test","timestamp_field": "@timestamp","tiebreaker_field": "tiebreaker"}' http://localhost:5601/kbn/api/detection_engine/rules
```

- Open `Detection Rules (SIEM)` Page -> `Rule Updates` -> click on `Potential Code Execution via Postgresql` rule -> expand `EQL Query` to see EQL Query -> press `Edit` button

## Screenshots

- EQL Query in Prebuilt Rules Update workflow
<img width="2560" alt="image" src="https://github.com/user-attachments/assets/59d157b2-6aca-4b21-95d0-f71a2e174df2">

- event_category_override + tiebreaker_field + timestamp_field (aka EQL options) in Prebuilt Rules Update workflow
<img width="2552" alt="image" src="https://github.com/user-attachments/assets/1886d3b4-98f9-40a7-954c-2a6d4b8e925a">

- Examples of invalid EQL
<img width="2560" alt="image" src="https://github.com/user-attachments/assets/d584deca-7903-45c5-9499-718552df441c">

<img width="2548" alt="image" src="https://github.com/user-attachments/assets/b734e22c-ab62-4624-85d0-e4e6dbb9d523">
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backport:version Backport to applied version labels Feature:Prebuilt Detection Rules Security Solution Prebuilt Detection Rules area release_note:skip Skip the PR/issue when compiling release notes Team:Detection Rule Management Security Detection Rule Management Team Team:Detections and Resp Security Detection Response Team Team: SecuritySolution Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc. v8.18.0 v9.0.0
Projects
None yet
Development

Successfully merging this pull request may close these issues.

9 participants