[Security Solution][Exceptions] Add lowercase normalized fields for case-insensitive matching

[madirey]

This PR adds *.caseless multifields for fields that will be used for case-insensitive exceptions, so that the detection engine can perform case-insensitive comparisons in the same way that the endpoint does (without tokenization).

Related:

• [Security Solution][Exceptions] Add lowercase normalizer for case-insensitivity + deprecate _tags field (new OS field) kibana#77379
• Add normalizer multi-field capability ecs#971

[jonathan-buttner]

Should these changes go in the Ext section under each file since they modify the core ECS fields? Thoughts @marshallmain ?

[jonathan-buttner]

process.parent isn't explicitly defined in the core ECS as of version 1.6. It's now created by duplicating all the process fields and storing them on process.parent. So I think you can remove the parent.* one heres since you already have them defined on process.

[jonathan-buttner]

👍

[madirey]

@jonathan-buttner @marshallmain Let's talk about this today... these are just multi-field additions to existing fields, so I was thinking we could get away with this... if they go into Ext, that makes our job a little harder as we'd need to set them in the pipeline. Not sure what's the best option going forward, but happy to discuss.

Alternatively, we could use this as a stop-gap and push to get these included in ECS?

[jonathan-buttner]

@jonathan-buttner @marshallmain Let's talk about this today... these are just multi-field additions to existing fields, so I was thinking we could get away with this... if they go into Ext, that makes our job a little harder as we'd need to set them in the pipeline. Not sure what's the best option going forward, but happy to discuss.

Alternatively, we could use this as a stop-gap and push to get these included in ECS?

Yeah that makes sense. I'm happy to chat whenever.

[marshallmain]

I think it's simpler to add the extra multi-fields to the core ECS fields rather than having a new custom field. We've had user confusion about the existing multi-fields on core ECS fields (.text) already and having a completely separate field might be more confusing.

The downside is if ECS does add official .caseless fields and they're not exactly the same as our caseless fields then upgrading the package to a new ECS version is potentially a breaking change. Does the ingest manager have any way of dealing with new package versions that have breaking changes compared to the installed version? @jonathan-buttner

[jonathan-buttner]

@madirey since ECS already defines the path.text as type text, if you remove that from here does it still get outputted or does it get removed?

Basically I'm wondering if we can avoid having to define the -name: text fields in these files.

[jonathan-buttner]

Looks like ECS doesn't support merging custom and core multi_fields. I'll open an issue in the ECS repo.

[jonathan-buttner]

I think it's simpler to add the extra multi-fields to the core ECS fields rather than having a new custom field. We've had user confusion about the existing multi-fields on core ECS fields (.text) already and having a completely separate field might be more confusing.

The downside is if ECS does add official .caseless fields and they're not exactly the same as our caseless fields then upgrading the package to a new ECS version is potentially a breaking change. Does the ingest manager have any way of dealing with new package versions that have breaking changes compared to the installed version? @jonathan-buttner

The general guidance from the ingest team is to avoid making breaking changes until a major version change 7.x -> 8.0. I think the way the ingest manager app handles installing a new package is it forces a rollover on the index so the new mapping being applied shouldn't collide with the old index. Is that accurate @ruflin ?

@madirey It'd probably be a good idea to create a ticket in ECS to discuss the .caseless field as well. I think I'm ok with adding them here but if we can get them in ECS core eventually too that'd be great.

[ruflin]

@jonathan-buttner At first, ingest manager does try to update the mapping. This works if there is no conflict. If there is a conflict, it will update the template and trigger a rollover.

[marshallmain]

LGTM