Add common dynamic mappings and properties from ECS automatically #1073

mrodm · 2022-12-15T16:46:35Z

Relates #1018

This PR adds support to include some dynamic templates and properties from ECS that are commonly used and can be a source of conflicts if they don't have any mapping.

These mappings are added if packages enable the flag import_mappings in _dev/build/build.yml. This flag is introduced in elastic/package-spec#455 , and it would be available in a future version 2.3.0 of the spec.

Option 1:
All the dynamic templates names are prefixed with _embedded_ecs, so they can be distinguished easily by Fleet in case it is needed. For instance:

elasticsearch:
  index_template:
    mappings:
      dynamic_templates:
        - _embedded_ecs.ecs_timestamp:
            mapping:
              ignore_malformed: false
              type: date
            path_match: '@timestamp'
        - _embedded_ecs.data_stream_to_constant:
            mapping:
              type: constant_keyword
            path_match: data_stream.*

Option 2 (Discarded for now):
All the mappings added by this flag are added into the _meta field. Thereby, they are marked in case it is needed to know which ones were automatically added. Properties and dynamic templates are added to ecs_properties_added and ecs_dynamic_templates_added keys, respectively.
Example:

    "mappings": {
      "_meta": {
        "ecs_properties_added": [
          "@timestamp"
        ],
        "package": {
          "name": "elastic_package_registry"
        },
        "managed_by": "fleet",
        "managed": true,
        "ecs_dynamic_templates_added": [
          "data_stream_to_constant",
          "resolved_ip_to_ip",
          "forwarded_ip_to_ip",
          "ip_to_ip",
          "port_to_long",
          "thread_id_to_long",
          "status_code_to_long",
          "line_to_long",
          "priority_to_long",
          "code_to_long",
          "code_to_long",
          "bytes_to_long",
          "packets_to_long",
          "public_key_exponent_to_long",
          "severity_to_long",
          "duration_to_long",
          "pid_to_long",
          "uptime_to_long",
          "sequence_to_long",
          "entropy_to_long",
          "size_to_long",
          "entrypoint_to_long",
          "ttl_to_long",
          "major_to_long",
          "minor_to_long",
          "as_number_to_long",
          "pgid_to_long",
          "exit_code_to_long",
          "chi_to_long",
          "args_count_to_long",
          "virtual_address_to_long",
          "io_text_to_wildcard",
          "strings_to_wildcard",
          "path_to_wildcard",
          "message_id_to_wildcard",
          "command_line_to_multifield",
          "error_stack_trace_to_multifield",
          "http_content_to_multifield",
          "url_full_to_multifield",
          "url_original_to_multifield",
          "user_agent_original_to_multifield",
          "error_message_to_match_only",
          "message_match_only_text",
          "agent_name_to_keyword",
          "service_name_to_keyword",
          "sections_name_to_keyword",
          "resource_name_to_keyword",
          "observer_name_to_keyword",
          "question_name_to_keyword",
          "group_name_to_keyword",
          "geo_name_to_keyword",
          "host_name_to_keyword",
          "severity_name_to_keyword",
          "title_to_multifield",
          "executable_to_multifield",
          "file_path_to_multifield",
          "file_target_path_to_multifield",
          "name_to_multifield",
          "full_name_to_multifield",
          "os_full_to_multifield",
          "working_directory_to_multifield",
          "timestamp_to_date",
          "delivery_timestamp_to_date",
          "not_after_to_date",
          "not_before_to_date",
          "accessed_to_date",
          "origination_timestamp_to_date",
          "created_to_date",
          "installed_to_date",
          "creation_date_to_date",
          "ctime_to_date",
          "mtime_to_date",
          "ingested_to_date",
          "start_to_date",
          "end_to_date",
          "score_base_to_float",
          "score_temporal_to_float",
          "score_to_float",
          "score_norm_to_float",
          "usage_to_float",
          "location_to_geo_point",
          "same_as_process_to_boolean",
          "established_to_boolean",
          "resumed_to_boolean",
          "max_bytes_per_process_exceeded_to_boolean",
          "interactive_to_boolean",
          "exists_to_boolean",
          "trusted_to_boolean",
          "valid_to_boolean",
          "go_stripped_to_boolean",
          "coldstart_to_boolean",
          "exports_to_flattened",
          "structured_data_to_flattened",
          "imports_to_flattened",
          "attachments_to_nested",
          "segments_to_nested",
          "elf_sections_to_nested",
          "pe_sections_to_nested",
          "macho_sections_to_nested",
          "trigger_to_nested"
        ]
      },

…t exist

elasticmachine · 2022-12-15T16:51:26Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2022-12-22T19:07:32.984+0000
Duration: 35 min 8 sec

Test stats 🧪

Test	Results
Failed	0
Passed	870
Skipped	0
Total	870

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.

elasticmachine · 2022-12-15T17:22:54Z

🌐 Coverage report

Name	Metrics % (`covered/total`)	Diff
Packages	100.0% (`35/35`)	💚
Files	65.909% (`87/132`)	👎 -1.278
Classes	61.17% (`115/188`)	👎 -0.786
Methods	47.613% (`379/796`)	👎 -0.958
Lines	30.886% (`3449/11167`)	👎 -0.982
Conditionals	100.0% (`0/0`)	💚

jsoriano

Looks good, but added a couple of comments that may help to simplify this feature.

Ideally we will also need to take these mappings into account when validating field definitions on tests. Tests will start failing if developers remove their ECS definitions after enabling this import, although the final template will be correct.
But let's think about this in a followup. Current workaround would be to keep definitions required by tests.

internal/builder/dynamic_mappings.go

jsoriano · 2022-12-20T11:57:11Z

internal/builder/_static/ecs_mappings.json

+      "@timestamp": {
+        "type": "date",
+        "ignore_malformed": false
+      }


Was there a strong reason to keep this field as an static property?

This single field forces us to maintain the special case of having also static properties defined here, requiring for example the additional meta field and so on. Would it be possible to define this as a dynamic template instead? It would be something like this:

"dynamic_templates": [ { "ecs_timestamp": { "path_match": "@timestamp", "mapping": { "type": "date", "ignore_malformed": false }, },

And we could think on this feature as something like adds only dynamic templates. Also in the context of the spec change.

I used the same json that was built here https://gist.github.com/P1llus/e0de7b3a7824a41a29660e253c6cce6b

I think it could be changed to be a dynamic template as you mentioned. @P1llus do you think in any inconvenient if it is changed ?

I would say there is no problem and it would simplify the code to just manage dynamic templates.

For now I've moved that property to a dynamic template as suggested by @jsoriano.

If there is any inconvenient because of that, it could be re-introduced.

Currently, it has been followed the approach to add a prefix (_embedded_ecs.*) to all the names of the dynamic templates in the static file, so they can be identified easily. If properties are needed too, it should be added another way (using _meta field and lists as it was in a previous change?).

internal/builder/dynamic_mappings.go

Review errors raised and their messages

mrodm · 2022-12-20T17:35:18Z

go.mod

@@ -160,3 +160,5 @@ require (
 	sigs.k8s.io/structured-merge-diff/v4 v4.2.3 // indirect
 	sigs.k8s.io/yaml v1.3.0 // indirect
 )
+
+replace github.com/elastic/package-spec/v2 => github.com/mrodm/package-spec/v2 v2.0.0-20221220145202-70d4563fbf33


to be removed before merging

mrodm · 2022-12-20T18:47:12Z

/test

internal/builder/packages.go

internal/builder/dynamic_mappings.go

…ppings

internal/builder/_static/ecs_mappings.json

hop-dev

I added one question but looks good to me, I like the new style of prefixing the name

mrodm · 2022-12-22T16:42:20Z

/test

In kibana versions < 8.6.0 the dot in the names caused that packages were not installed. It was interpreted that the string after the dot was an inner element of the map, causing validation errors.

mrodm · 2022-12-22T19:07:48Z

I've checked building and installing a test package in different versions of Kibana. In kibana 8.6.0-SNASPSHOT there was no issue. But in versions < 8.6.0 , the dot in the dynamic template names is causing this error:

Error installing elastic_package_registry 0.0.7: mapper_parsing_exception: [mapper_parsing_exception] Reason: Failed to parse mapping: Illegal dynamic template parameter: [ecs_timestamp]

I've updated how the name of each dynamic template is generated to replace the dot "." by a dash "-" @jsoriano @hop-dev

Thereby, the dynamic template generated would be like:

elasticsearch:
  index_template:
    mappings:
      dynamic_templates:
        - _embedded_ecs-ecs_timestamp:
            mapping:
              ignore_malformed: false
              type: date
            path_match: '@timestamp'
        - _embedded_ecs-data_stream_to_constant:
            mapping:
              type: constant_keyword
            path_match: data_stream.*
        - _embedded_ecs-resolved_ip_to_ip:
            mapping:
              type: ip
            match: resolved_ip

Tested with the following Kibana versions 7.17.8-SNAPSHOT, 8.0.0-SNAPSHOT, 8.5.0-SNAPSHOT and 8.6.0-SNAPSHOT

Add dynamic mappings and properties - pending to create if they do no…

6e51991

…t exist

Write files in destinationDir

89e943c

mrodm added 6 commits December 15, 2022 18:57

Create nodes if they do not exist

9b7624b

Test new version package-spec

f51aef7

Check wether or not import setting is true in build manifest

c663afe

Added list of mappings added into _meta field

3c78387

Add tests for methods adding ecs mappings

001a312

Add license header

ddea438

mrodm mentioned this pull request Dec 20, 2022

Import hard-coded list of common fields #1025

Closed

mrodm marked this pull request as ready for review December 20, 2022 10:31

mrodm changed the title ~~Add dynamic mappings and properties ECS - WIP~~ Add common dynamic mappings and properties from ECS automatically Dec 20, 2022

mrodm requested a review from a team December 20, 2022 10:31

mrodm self-assigned this Dec 20, 2022

jsoriano requested a review from hop-dev December 20, 2022 11:26

jsoriano reviewed Dec 20, 2022

View reviewed changes

mrodm added 2 commits December 20, 2022 16:35

Update field used to import mappings

e85ada3

Rename dynamic template names to include a common prefix

44763fc

Review errors raised and their messages

mrodm commented Dec 20, 2022

View reviewed changes

mrodm added 2 commits December 20, 2022 18:44

Remove properties

311ee70

Remove the addition of _meta fields

f6fcff5

mrodm requested a review from jsoriano December 20, 2022 18:50

jsoriano approved these changes Dec 21, 2022

View reviewed changes

internal/builder/packages.go Outdated Show resolved Hide resolved

internal/builder/dynamic_mappings.go Outdated Show resolved Hide resolved

mrodm added 2 commits December 21, 2022 09:52

Add debug message before validation

17d6ac3

Update debug messages

31fa9a2

jsoriano approved these changes Dec 21, 2022

View reviewed changes

mrodm added 2 commits December 21, 2022 11:34

Add method to check if mappings should be imported

aaa83e2

Merge remote-tracking branch 'upstream/main' into add_temporal_ecs_ma…

57da695

…ppings

Use semver v3

f21c828

hop-dev reviewed Dec 21, 2022

View reviewed changes

internal/builder/_static/ecs_mappings.json Show resolved Hide resolved

hop-dev approved these changes Dec 21, 2022

View reviewed changes

mrodm requested a review from jsoriano December 21, 2022 12:50

jsoriano approved these changes Dec 21, 2022

View reviewed changes

mrodm mentioned this pull request Dec 22, 2022

Temporarily import hard-coded list of common fields #1018

Closed

3 tasks

Remove package-spec replace from go.mod

a46de7b

Replace dot by a dash in the imported template names

8290c65

In kibana versions < 8.6.0 the dot in the names caused that packages were not installed. It was interpreted that the string after the dot was an inner element of the map, causing validation errors.

mrodm force-pushed the add_temporal_ecs_mappings branch from 633f842 to 8290c65 Compare December 22, 2022 19:07

mrodm mentioned this pull request Jan 10, 2023

Add ECS mappings during test runner executions #1090

Merged

1 task

mrodm merged commit d46148d into elastic:main Jan 13, 2023

mrodm deleted the add_temporal_ecs_mappings branch January 13, 2023 12:34

This was referenced Jan 16, 2023

[GCS] Adding new integration for Custom GCS Input elastic/integrations#4692

Merged

[Azure Blob Storage] Adding integration for Custom Azure Blob Storage Input elastic/integrations#4693

Merged

jsoriano mentioned this pull request Jan 17, 2023

Add build information of the package at build time elastic/package-spec#446

Open

P1llus mentioned this pull request Jan 19, 2023

Dynamic ECS mapping progress elastic/integrations#5055

Closed

muthu-mps mentioned this pull request Jan 31, 2023

[Service Integrations] ECS Version sync with stack version elastic/integrations#4792

Closed

5 tasks

jsoriano mentioned this pull request Apr 17, 2023

[RFC] Stage 0 - TSDB Dimensions elastic/ecs#2172

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add common dynamic mappings and properties from ECS automatically #1073

Add common dynamic mappings and properties from ECS automatically #1073

mrodm commented Dec 15, 2022 •

edited

Loading

elasticmachine commented Dec 15, 2022 •

edited

Loading

Build stats

Test stats 🧪

elasticmachine commented Dec 15, 2022 •

edited

Loading

jsoriano left a comment

jsoriano Dec 20, 2022

mrodm Dec 20, 2022

mrodm Dec 20, 2022

mrodm Dec 20, 2022

mrodm commented Dec 20, 2022

hop-dev left a comment

mrodm commented Dec 22, 2022

mrodm commented Dec 22, 2022

Add common dynamic mappings and properties from ECS automatically #1073

Add common dynamic mappings and properties from ECS automatically #1073

Conversation

mrodm commented Dec 15, 2022 • edited Loading

elasticmachine commented Dec 15, 2022 • edited Loading

💚 Build Succeeded

Build stats

Test stats 🧪

🤖 GitHub comments

elasticmachine commented Dec 15, 2022 • edited Loading

🌐 Coverage report

jsoriano left a comment

Choose a reason for hiding this comment

jsoriano Dec 20, 2022

Choose a reason for hiding this comment

mrodm Dec 20, 2022

Choose a reason for hiding this comment

mrodm Dec 20, 2022

Choose a reason for hiding this comment

mrodm Dec 20, 2022

Choose a reason for hiding this comment

mrodm commented Dec 20, 2022

hop-dev left a comment

Choose a reason for hiding this comment

mrodm commented Dec 22, 2022

mrodm commented Dec 22, 2022

mrodm commented Dec 15, 2022 •

edited

Loading

elasticmachine commented Dec 15, 2022 •

edited

Loading

elasticmachine commented Dec 15, 2022 •

edited

Loading