[Security Solution][Alerts] Alert suppression per rule execution

[marshallmain]

Summary

Addresses #130699

This PR implements alert throttling per rule execution for query and saved query rules. The implementation is very similar in concept to threshold rules. We allow users to pick one or more fields to group source documents by and use a composite aggregation to collect documents bucketed by those fields. We create 1 alert for each bucket based on the first document in the bucket and add metadata to the alert that represents how to retrieve the rest of the documents in the bucket.

The metadata fields are:

• kibana.alert.suppression.terms: Array<{field: string; value: string | number | null}> An array of objects, each object represents one of the terms used to group these alerts
• kibana.alert.suppression.start: Date The timestamp of the first document in the bucket
• kibana.alert.suppression.end: Date The timestamp of the last document in the bucket
• kibana.alert.suppression.docs_count: number The number of suppressed alerts

There is one new rule parameter, currently implemented at the solution level, to enable this feature: alertSuppression.groupBy: string[].

Similar to threshold rules, the throttled query rules keep track of created alerts in the rule state in order to filter out duplicate documents in subsequent rule executions. When a throttled alert is created, we store the bucket information including field names, values, and end date in the rule state. Subsequent rule executions convert this state into a filter that excludes documents that have already been covered by existing alerts. This is necessary because consecutive rule executions will typically query overlapping time ranges.

Licensing

Alert suppression is licensed as a platinum-level feature. However, the rule management APIs do not prevent the suppression configuration fields from being set without a platinum license. Instead, the license is checked at rule execution time: if the license is insufficient, then the suppression configuration is ignored and the rule executes without suppressing alerts. This avoids potential issues with rule management and license downgrades - if a customer has rules enabled with suppression and the license expires/downgrades, they are not prevented from exporting/importing/editing their rules in any way, but the suppression will no longer be applied until the license is reinstated.

We check the license in the UI and display the suppression configuration accordingly. If the license is insufficient, then in rule creation the suppression configuration is disabled. When editing a rule, if the license is insufficient and the rule does not already have suppression configured, then suppression configuration is also disabled. If the license is insufficient but a rule does have suppression configured already, then we allow editing the suppression configuration in the UI.

The rule details page shows an indicator if suppression is configured and the license is insufficient to notify users that the suppression configuration will not be applied during rule execution.

Screenshots

Rule Create/Edit With License

Rule Details With License

Rule Create, or Rule Edit of a rule without existing suppression configuration, Without License

Editing a rule that has existing suppression configuration, but without the correct license, still allows changing the configuration (to allow removing the params)

Rule Details Without License

Alerts table

Known issues

• The layers icon in the rule name for suppressed alerts does not show up in the rule preview table

[dhurley14]

platform changes look good

[vitaliidm]

Great work, @marshallmain. This feature will bring more capabilities to our customers!
Thank you for addressing my comments.

I left a few minor as well, that can be addressed later. The only big question I have it's regarding licensing. What's the plan on it? Will it be added in 8.6 release? As currently it's only handled on UI

[vitaliidm]

can we use value for colour from eui theme?

[marshallmain]

Yeah, this should come from the theme

[vitaliidm]

I can see check for license is only present in UI.
What about having it on API level of Security and Platform? Is there a consideration to have it in 8.6?

[marshallmain]

I switched from validating the params on create/edit of a rule to checking the license at runtime in the executor to avoid edge cases around license expiration/degradation. E.g. if the license expires, rules would be exportable but then would fail to import, patch would allow editing a rule while leaving suppression params untouched but update would not, etc.

The behavior now is that if a license is insufficient and suppression is enabled then rules will continue to work, but the suppression configuration will be ignored. If the platinum license is restored, then suppression will start to work again on those rules automatically. This way we can keep the functionality behind the license appropriately but not cause issues with rule management if the license level does drop.

[vitaliidm]

Thanks for explaining, that makes sense 👍

I switched from validating the params on create/edit of a rule to checking the license at runtime in the executor to avoid edge cases around license expiration/degradation

Btw, can we use there minimumLicenseForSuppression instead of hardcoded platinum value?

[marshallmain]

Oops, yeah that should be minimumLicenseForSuppression as well. I'll add it to the follow up list.

[vitaliidm]

why bucket.count need to be decremented?

Is an idea behind it: if there is only one document, we still generate an alert, but display count as 0, because no other documents exist to be actually suppressed?

[marshallmain]

Yeah the idea is to display here the number of alerts that were not created due to suppression, so we subtract 1 because the created alert is included in bucket.count as well.

[vitaliidm]

I would suggest to test state involved executions through actual rule run, as it involves storing state in task manager and operating with it.
As for preview, it's stored in memory during preview execution, so we don't have full coverage of this functionality

[marshallmain]

Good idea, will add in a follow up

[xcrzx]

Code changes in the rules area LGTM! While testing locally found a couple of minor UI bugs, though.

Values should be comma separated:

Long values overflow outside of the screen:

The total number of alerts displayed on the chart (11) doesn't match the total number of alerts in the table (9). Not sure where that difference comes from. Probably not related to this PR at all.

[marshallmain]

Values should be comma separated:

Good point, updated to handle the array of values correctly (and updated screenshots in the PR description to match).

The total number of alerts displayed on the chart (11) doesn't match the total number of alerts in the table (9). Not sure where that difference comes from.

I've seen this before if one event has multiple event.category values, as it gets counted multiple times in the histogram then -was that the case for your test data?

[kibana-ci]

💚 Build Succeeded

• Buildkite Build
• Commit: 5d88e35
• Cloud Deployment

Metrics [docs]

Module Count

Fewer modules leads to a faster build time

id	before	after	diff
securitySolution	3286	3290	+4

Public APIs missing comments

Total count of every public API that lacks a comment. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats comments for more detailed information.

id	before	after	diff
@kbn/rule-data-utils	83	89	+6

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
observability	487.7KB	488.3KB	+582.0B
securitySolution	9.6MB	9.6MB	+20.1KB
triggersActionsUi	663.6KB	665.6KB	+2.0KB
total			+22.7KB

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id	before	after	diff
apm	29.9KB	30.1KB	+169.0B
cases	126.7KB	126.8KB	+169.0B
infra	85.6KB	85.7KB	+169.0B
synthetics	25.1KB	25.3KB	+173.0B
timelines	138.8KB	139.0KB	+169.0B
total			+849.0B

Unknown metric groups

API count

id	before	after	diff
@kbn/rule-data-utils	86	92	+6

ESLint disabled in files

id	before	after	diff
osquery	1	2	+1

ESLint disabled line counts

id	before	after	diff
enterpriseSearch	19	21	+2
fleet	59	65	+6
osquery	108	113	+5
securitySolution	441	447	+6
total			+19

Total ESLint disabled count

id	before	after	diff
enterpriseSearch	20	22	+2
fleet	67	73	+6
osquery	109	115	+6
securitySolution	518	524	+6
total			+20

History

• 💚 Build #87733 succeeded 135944e
• 💚 Build #87722 succeeded f8b4250
• 💔 Build #87708 failed 3823357
• 💔 Build #86889 failed d757582
• 💔 Build #86733 failed 66fda9d

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

[xcrzx]

I've seen this before if one event has multiple event.category values, as it gets counted multiple times in the histogram then -was that the case for your test data?

Oh, yeah, that was the reason. Then I think it is okay to have one event counted multiple times.

And thanks for fixing the layout issues 👍

[mikecote]

Rule registry changes look good to me at this time! We'll meet later this week with y'all to discuss proposed changes but we won't block this PR from merging.