[Security Solution][Alerts][Discuss] Alert suppression on docs with missing fields

[marshallmain]

With the introduction of alert suppression users can select one or more fields to group source events by, creating a single alert per unique value in the field. However, "no value" is currently (in 8.6 and 8.7) treated as a unique value for this grouping. As a result, all documents that don't populate the suppression field will end up in the same group, which may not be what users expect to happen.

There are (at least) 3 ways we might choose to handle docs that are missing the suppression field:

1. Exclude them from the results entirely. It's possible that when choosing a suppression field, users are not interested in docs that don't populate the field.
2. (current behavior) Include them in the results, but group them all together in one bucket. "No value" is treated as though it's a unique value the same way a host.name of host-1 is a unique value.
3. Fall back to the unsuppressed custom query rule logic for documents that don't populate the suppression field. In this case, we could perhaps first create alerts for each of the unique values in the suppression group, then, if maxSignals was not hit yet, create more alerts (up to maxSignals) that correspond 1-1 with the source docs that don't populate the suppression field.

(3) seems to be the most intuitive default behavior. However, we may also want to provide the ability to configure how these docs that are missing the suppression field are handled.

Questions:
Should we allow this behavior to be configured, perhaps at the rule level?
Since alert suppression is in technical preview, are we free to change the default behavior from option 2 to option 3?

cc @paulewing

[vitaliidm]

Draft version of UI:

Rule create page

Rule Details

cc: @elastic/security-docs , @joepeeples

[vitaliidm]

POC for hiding suppression settings in accordion

Screen.Recording.2023-04-19.at.19.01.33.mov

Few questions:

1. Maybe it would make sense to move it below timeline?
2. Maybe we could leave input outside of the accordion? And put only radio buttons there?
3. How this would affect discoverability of feature?

[vitaliidm]

We agreed to add accordion with Suppression Configuration button title

cc: @paulewing

[ARWNightingale]

@paulewing

Ill put some quick options here
https://www.figma.com/file/9v6cvRqd17Xvsn5o7MHFCl/Alert-suppression-on-docs-with-missing-fields?node-id=501%3A42630&t=qqkoLqrnuRu5yNUR-1

[vitaliidm]

on advanced correlation WG meeting we decided to remove accordion and instead of, we will just add indentation for suppression settings

In future, we might also consider an option to add info icons or links to documentation for each suppress/unsupress mode

[vitaliidm]

@joepeeples , can you please assist with wording for information icons, we want to add for missing fields options?

[joepeeples]

Here's my first stab at the UI text, tooltip, etc. This is a combo of @vitaliidm's draft and a condensed version of the draft docs I'm working on here.

If a suppression field is missing

• Suppress by missing fields
• Do not suppress by missing fields

Help tooltip: Choose how to handle events with missing Suppress alerts by fields. Either group events with missing fields together, or create a separate alert for each event. Learn more

I know the option names are a little repetitive, but I thought it'd help to be specific. "Do not suppress" on its own felt a little broad to me, like the user was turning off suppression overall, but it's really only under specific circumstances.

Also, FYI the Learn more link is following the UX Writing team's guidelines for docs links, so no ending punctuation, etc.

[vitaliidm]

Thanks @joepeeples
I am thinking, maybe we should change Suppress by missing fields to something like Suppress when fields missing
As missing field here refers more to the value of the field we suppress by.
So, if we suppress alerts by agent.name, if agent.name is missing in document, we are not suppressing by missing field value, i.e. null, we just include those documents in suppress buckets.

Although, Suppress when fields missing sounds very repetitive if combined with header If a suppression field is missing

[joepeeples]

@vitaliidm and all: I see what you mean. Maybe something like:

If a suppression field is missing

• Suppress and group alerts with missing fields
• Do not suppress alerts with missing fields

[vitaliidm]

@joepeeples
Maybe something like this?

If a suppression field is missing

• Suppress and group alerts for events with missing fields
• Do not suppress alerts for events with missing fields

Or maybe just

If a suppression field is missing

• Suppress and group alerts
• Do not suppress alerts

[joepeeples]

@joepeeples Maybe something like this?

If a suppression field is missing

• Suppress and group alerts for events with missing fields
• Do not suppress alerts for events with missing fields

@vitaliidm I like this first option a lot. It's a little wordier, but I think in this case that's a good thing, because this explains the behavior more fully and accurately.

[vitaliidm]

UI changes implemented in #155839

[umutsagenet]

Hello Team,

Do you have any specific time when this suppression feature on specific fields for a specific time will be available in Elastic Security? This is really a need to prevent multiple alerts/noise triggering in front of analysts. Splunk has this feature named as throttle for a while and it is very useful. Especially working in an MSSP, we sometimes see multiple logs for the same event in a specific rule query and this throttle feature is needed to prevent alerting for the same username and/or same hostname and/or same target.user.name etc. for an hour or so, therefore alerts are not triggered for a specific time on a specific field. Also there are cases that a rule search may trigger in a client environment for a network misconfiguration after a maintenance or a general update can match with the rule search query which may result in benign activities, however throttle can help in such cases to prevent excessive alerting. I was asking questions about throttle option to Elastic Support and Maggie informed me about this github page. Thank you.

Umut.

[haodketernal]

hmm. Why does it need platinum license ? so basic