[Security Solution][Alerts] adds support for multi fields in new terms rule

[vitaliidm]

Summary

• addresses [Security Solution][Alerts] Allow using multiple fields in New Terms rule #142862
  • allows up to 3 fields in New terms
  • displays new terms fields in alerts details
• For multiple new terms fields(['source.host', 'source.ip']), in terms aggregation uses a runtime field. Which is created by joining values from new terms fields into one single keyword. Field values encoded in base64 and joined with a configured delimiter symbol, which is not part of base64 symbols(a–Z, 0–9, +, /, =) to avoid a situation when delimiter can be part of field value. Include parameter consists of encoded in base64 results from Phase 1.
  For single field, implementation remains the same to avoid performance penalties
• Performance measurements:
  • msearch POC for one field
  • historical POC of multi fields
  • current implementation

UI

Alert details

Before

After

Delete any items that are not applicable to this PR.

• Documentation was added for features that require explanation or tutorials
• Unit or functional tests were updated or added to match the most common scenarios

For maintainers

• This was checked for breaking API changes and was labeled appropriately

[marshallmain]

This is awesome! Just a couple questions about how the runtime field handles edge cases, then I think it's good to go.

[marshallmain]

If we filter out individual null values from a particular bucket, is there any chance of the encoded value from the bucket matching the runtime field-calculated value since the runtime field will still use all of the fields, including the nulls? I wonder if we should exclude the entire bucket if any values are null, or include the nulls and base64 encode them as well.

[vitaliidm]

Looks like, we don't need this check at all.
Our query is following

POST /null-*/_search?allow_no_indices=true&ignore_unavailable=true
{
   "query":{
      "bool":{
         "filter":[
            {
               "bool":{
                  "must":[
                     
                  ],
                  "filter":[
                     {
                        "query_string":{
                           "query":"*"
                        }
                     }
                  ],
                  "should":[
                     
                  ],
                  "must_not":[
                     
                  ]
               }
            },
            {
               "range":{
                  "@timestamp":{
                     "lte":"2022-11-10T14:46:11.680Z",
                     "gte":"2022-11-08T14:45:11.680Z",
                     "format":"strict_date_optional_time"
                  }
               }
            },
            {
               "match_all":{
                  
               }
            }
         ]
      }
   },
   "fields":[
      {
         "field":"*",
         "include_unmapped":true
      },
      {
         "field":"@timestamp",
         "format":"strict_date_optional_time"
      }
   ],
   "aggregations":{
      "new_terms":{
         "composite":{
            "sources":[
               {
                  "host.name":{
                     "terms":{
                        "field":"host.name"
                     }
                  }
               }
            ],
            "size":10000
         }
      }
   },
   "runtime_mappings":{
      
   },
   "sort":[
      {
         "@timestamp":{
            "order":"asc",
            "unmapped_type":"date"
         }
      }
   ],
   "size":0
}

And it doesn't return null values in buckets. Per documentation, we need to introduce a parameter in "missing_bucket": true, on sources terms item.
I think we can dismiss this check entirely then. Downside of this, we would need to cast typing here, as bucket type here is following:

export interface AggregationsCompositeBucketKeys extends AggregationsMultiBucketBase {
    key: Record<string, any>;
}

On the other note: should we treat null value in term as a new one? In current implementation we dismiss nulls. But could it be a valid case: some field has become nullable for the first time in comparison to historical, do we want to generate alert about it?

[marshallmain]

True, the current phase 1 query excludes null values, but I think it's good to be defensive here so the rule is robust in case we do end up getting null values back by accident - either a subtle change in ES, or the query changes in the future. The include param on the terms agg for phase 2 rejects null values, so if a null did slip in with a single field rule it would break the rule.

null as a new value seems like it could be a valid use case, and seems like it would be possible with the new runtime field implementation for multiple fields since the null could be base64 encoded before inclusion in the terms include param. We'd just need to add some kind of special handling for it in the single field case - maybe a single extra query checking for null in that field over the history window when null is detected in the recent window?

I think adding full support for nulls is something we can leave for the future though, don't want to rush it in here. My immediate focus is just ensuring the implementation is robust.

[vitaliidm]

added filtering for buckets with multiple fields and functional tests for it

[marshallmain]

This is awesome! Definitely the most complicated runtime field I've seen so far, but well worth it for this feature IMO.
A couple questions:

1. Does anything break if we select multiple fields that each have many values? Certain fields such as process.args can be very large arrays of values, so even though we limit it to 3 fields, if each one has 100 values we'd be emitting 1,000,000 values here. Adding a hard cap on the number of emitted values could be useful, if ES doesn't have a circuit breaker built in already.
2. Some integration test coverage to execute this runtime field specifically against test data and verify that it creates the expected values, particularly in edge cases like null values and large arrays, would be beneficial.

[vitaliidm]

Does anything break if we select multiple fields that each have many values? Certain fields such as process.args can be very large arrays of values, so even though we limit it to 3 fields, if each one has 100 values we'd be emitting 1,000,000 values here. Adding a hard cap on the number of emitted values could be useful, if ES doesn't have a circuit breaker built in already.

I've tried to perform test on large array: there is a limit on 100 emitted fields

    {
        "shard": 0,
        "index": "array-high-cardinality-data-fake_hosts-2022-11-04",
        "node": "yZmoJFdNRHe4uovryh5mDw",
        "reason": {
          "type": "script_exception",
          "reason": "runtime error",
          "script_stack": [
            "org.elasticsearch.server@8.6.0-SNAPSHOT/org.elasticsearch.script.AbstractFieldScript.checkMaxSize(AbstractFieldScript.java:125)",
            "org.elasticsearch.server@8.6.0-SNAPSHOT/org.elasticsearch.script.StringFieldScript$Emit.emit(StringFieldScript.java:145)",
            """emit(line);
            } else {
              """,
            "     ^---- HERE"
          ],
          "script": " ...",
          "lang": "painless",
          "position": {
            "offset": 140,
            "start": 135,
            "end": 182
          },
          "caused_by": {
            "type": "illegal_argument_exception",
            "reason": "Runtime field [new_terms_values] is emitting [101] values while the maximum number of values allowed is [100]"
          }
        }
      },

Which is, doesn't look to many indeed, but would prevent overuse of resources on ES side

[marshallmain]

Does the exception prevent results from being returned at all for that query, even for other documents that do not have too many values?

[vitaliidm]

@marshallmain
Added following functionality

• ensured that runtime field script does not emit more than 100 fields
• added tests for preview that cover arrays with large number of items

TODO:

• add tests for runtime script

[vitaliidm]

added tests for runtime script bc50e91

[marshallmain]

LGTM, nice work!

[kibana-ci]

💛 Build succeeded, but was flaky

• Buildkite Build
• Commit: cb135ad
• Cloud Deployment

Failed CI Steps

• Security Solution Tests #1

Test Failures

• [job] [logs] Security Solution Tests #1 / Alert Flyout Opens a new timeline investigation with alert ids from the process ancestry

Metrics [docs]

Module Count

Fewer modules leads to a faster build time

id	before	after	diff
securitySolution	3285	3286	+1

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
securitySolution	9.6MB	9.6MB	+1.3KB

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id	before	after	diff
securitySolution	51.0KB	51.0KB	+41.0B

Unknown metric groups

ESLint disabled in files

id	before	after	diff
osquery	1	2	+1

ESLint disabled line counts

id	before	after	diff
enterpriseSearch	19	21	+2
fleet	59	65	+6
osquery	108	113	+5
securitySolution	441	447	+6
total			+19

Total ESLint disabled count

id	before	after	diff
enterpriseSearch	20	22	+2
fleet	67	73	+6
osquery	109	115	+6
securitySolution	518	524	+6
total			+20

History

• 💛 Build #87165 was flaky 7d5dec8
• 💔 Build #87087 failed 64d6075
• 💛 Build #86804 was flaky 858e5c4
• 💚 Build #86749 succeeded b0b7ce2
• 💔 Build #86710 failed e26280c

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @vitaliidm