From fc14145222b5c2fc48fab7dda38f381212797bb4 Mon Sep 17 00:00:00 2001 From: Dougal Seeley Date: Wed, 5 May 2021 12:56:44 +0100 Subject: [PATCH] Upstream ECS merge of v1.9.0 (#4) * bumping version for 1.x release branch (#921) * [1.x] add related.hosts (#913) (#924) * [1.x][DOCS] Fixes SIEM links (#936) * [1.x] Consolidate field-details doc template (#897) (#946) * Add http.[request|response].mime_type (#944) (#949) * [1.x] Cut 1.6 Changelog (#933) (#952) (#953) Co-authored-by: Mathieu Martin * [1.x] Add threat.technique.subtechnique (#951) (#956) Co-authored-by: Ross Wolf <31489089+rw-access@users.noreply.github.com> * [1.x] Nest as for foreign reuse (#960) (#962) * [1.x] Remove `expected_event_types` from protocol (#964) (#965) * [1.x] Expand definitions of source and destination field sets (#967) (#973) * [1.x] Introduce `--strict` flag (#937) (#975) * [1.x] Add example value composite type checking (#966) (#976) * Add example value composite type checking (#966) * generate csv artifact * [1.x] Add event category configuration (#963) (#977) * [1.x] Add normalizer multi-field capability (#971) (#978) Co-authored-by: Eric Beahan Co-authored-by: Madison Caldwell * [1.x] Add mapping network event guidance doc (#969) (#983) * [1.x] Removing unneeded link under `Additional Information` (#984) (#985) * [1.x] Add discrete attribute to field details page headers (#989) (#990) * [1.x] Uniformity across domain name breakdown fields (#981) (#994) Co-authored-by: Mathieu Martin * Add --oss flag to the ECS generator script (#991) (#995) * Add network directions ingress and egress (#945) (#997) * Mention ECS Mapper in the main documentation (#987) (#1000) Co-authored-by: Dan Roscigno * [1.x] Introduce experimental artifacts (#993) (#1001) Co-authored-by: Mathieu Martin * Bump version to 1.8.0-dev in branch 1.x (#1011) * Cut 1.7 changelog (#1010) (#1012) * [1.x] Clarify that file extension should exclude the dot. (#1016) (#1020) * [1.x] Add usage docs section (#988) (#1024) Co-authored-by: Mathieu Martin * [1.x] feat: include alias path when generating template (#877) (#1035) Co-authored-by: Richard Gomez <32133502+rgmz@users.noreply.github.com> * [1.x] Add support for `scaling_factor` in the generator (#1042) (#1055) Co-authored-by: Mathieu Martin * [1.x] Add fallback for constant_keyword (#1046) (#1056) Co-authored-by: Mathieu Martin * [1.x] Add wildcard type support to go code generator (#1050) (#1057) * add wildcard type support * also add version and constant_keyword * changelog * [1.x] New default make task that generates main and experimental artifacts. (#1041) (#1060) Also changing the order of the 'generate' task: it now starts with the new generator, then runs the legacy scripts. * [1.x] Change the index pattern in the sample template. (#1048) (#1068) * [1.x] Prepare link to Logs docs changing with the 7.10 release in "getting-started" (#1073) (#1079) Co-authored-by: EamonnTP * [1.x] Prepare link to Logs docs changing with the 7.10 release in "products-solutions" page (#1074) (#1083) Co-authored-by: EamonnTP * [1.x] Add event.category session. (#1049) (#1093) Co-authored-by: Mathieu Martin * [1.x] Add event.category registry (#1040) (#1094) Co-authored-by: Mathieu Martin * [1.x] Add --ref support for experimental artifacts (#1063) (#1101) Co-authored-by: Mathieu Martin * [1.x] Remove experimental event.original definition (#1053) (#1104) * [1.x] Add missing `process.thread.name` to experimental definitions (#1103) (#1106) * [1.x] Remove index parameter for wildcard fields (#1115) (#1119) * [1.x] Add dns.answer object into experimental schema (#1118) (#1121) * [1.x] Clarify x509 definition guidance for network events with only one cert (#1114) (#1123) * [1.x] Indicate when artifacts include experimental changes (#1117) (#1125) * [1.x] Add os.type field, with list of allowed values (#1111) (#1130) * [1.x] Add support for constant_keyword's 'value' parameter (#1112) (#1132) * [1.x] Beta label support (#1051) (#1133) Co-authored-by: Mathieu Martin * [1.x] Backport #1134 and #1135 (#1136) * Remove temporary ifeval in "getting started" page, add link to Metrics docs (#1134) * Remove temporary ifeval from products page, add link to Metrics (#1135) * Two small documentation backports (#1149) * Remove an incorrect `event.type` from the 'converting' page (#1146) * Mention Logstash support for ECS in the 'products' page (#1147) * [1.x] Reinforce the exclusion of the leading dot from url.extension (#1151) (#1152) * [1.x] Make all fields linkable directly via an HTML ID (#1148) (#1154) * [1.x] Tracing fields should be at the root (#1165) * Add notice to the tracing field set, about not nesting field names. (#1162) * Tracing fields should be at top level in Beats artifact (#1164) * [1.x] Usage of brackets for a URL containing IPv6 address (#1131) (#1168) * [1.x] 6.x index template data type fallback (#1171) (#1172) * [1.x] Apply RFC 0007 stage 3 changes - multi-user (#1066) (#1175) Conflict: deleted file rfcs/text/0007-multiple-users.md as RFCs are not backported to version branches. * [1.x] Handle `error.stack_trace` case for ES 6.x template (#1176) (#1177) * [1.x] Add composable index templates artifacts (#1156) (#1179) * [1.x] Move _meta section back inside mappings, in legacy templates. (#1186) (#1187) Backports the following commits to 1.x: * Move _meta section back inside mappings, in legacy templates. (#1186) This fixes an issue introduced by #1156, discovered in #1180. Composable templates support `_meta` at the template's root, but legacy templates don't. So we're just putting it back inside the mappings for legacy templates. This also fixes missing updates to the component template, after the introduction of wildcard in #1098. * [1.x] Apply the RFC 0005 stage 2 (host metrics) changes in the experimental artifacts (#1159) (#1184) Co-authored-by: Mathieu Martin * [1.x] Stage 3 changes for wildcard RFC 0001 (#1098) (#1183) * [1.x] Conditional handling in es_template.template_settings (#1191) (#1192) * [1.x] Artifacts docs page (#1189) (#1195) * [1.x] Remove beta warning label from categorization fields docs (#1067) (#1196) * [1.x] Correct wording of `event.reference` description (#1181) (#1197) * Bump version to 1.9.0-dev in branch 1.x (#1198) * [1.x] Cut 1.8 FF changelog.next.md #1199 (#1201) * Merge custom and core multi_fields arrays (#982) (#1213) Co-authored-by: Jonathan Buttner <56361221+jonathan-buttner@users.noreply.github.com> * [1.x] Stage 2 changes for RFC 0009 - data_stream fields (#1215) (#1222) * [1.x] add http.request.id (#1208) (#1223) Co-authored-by: Eric Beahan Co-authored-by: Gil Raphaelli * [1.x] add cloud.service.name (#1204) (#1224) * add cloud.platform * expand cloud.platform description * move to cloud.service.name Co-authored-by: Gil Raphaelli * [1.x] Add ssdeep hash (#1169) (#1227) Co-authored-by: Andrew Stucki * [CI] Switch to GitHub actions (#1236) (#1245) Co-authored-by: Eric Beahan Co-authored-by: Andrew Stucki * Revert wildcard adoption back to experimental stage (#1235) (#1243) * Add scaled_float type to go generator (#1250) (#1251) * add scaled_float * changelog * Add categorization fields usage docs (#1242) (#1257) * add time_zone, postal_code, and continent_code (#1229) (#1258) * Specify MAC address format (#456) (#1260) Co-authored-by: Robin Schneider <36660054+ypid-geberit@users.noreply.github.com> * finalize 1.8.0 changelog (#1262) (#1265) * Add additional host fields (#1248) (#1267) Co-authored-by: kaiyan-sheng * Stage 1 changes for RFC 0014 - extend pe fields (#1256) (#1270) * Add 2 fields to code_signature (#1269) (#1272) Co-authored-by: Yamin Tian <56367679+Trinity2019@users.noreply.github.com> * Stage 3 changes for RFC 0007 - remove beta attribute (#1271) (#1273) * Stage 1 experimental changes for RFC 0008 - threat.indicator fields (#1268) (#1274) * Stage 1 changes for RFC 0015 - add elf fieldset (#1261) (#1275) * Cut 1.9 FF CHANGELOG.next.md (#1277) * lock go version in actions (#1283) (#1290) * Bump jinja2 from 2.11.2 to 2.11.3 in /scripts (#1310) (#1320) * Bump jinja2 from 2.11.2 to 2.11.3 in /scripts * Bump pyyaml from 5.3b1 to 5.4 in /scripts (#1318) (#1325) Co-authored-by: Eric Beahan Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Adjust terminology - change whitelist to allowlist (#1315) (#1331) Co-authored-by: Dominic Page <11043991+djptek@users.noreply.github.com> * Remove -dev label from 1.9 version (#1329) * remove -dev label from 1.9 version * generate artifacts * removing rules artifacts * Cut 1.9 changelog (#1328) * move 1.9 changes to changelog * add 1.9 release changes --- .github/workflows/test.yml | 19 + .gitignore | 4 + .travis.yml | 29 - CHANGELOG.md | 234 + CHANGELOG.next.md | 77 - Makefile | 21 +- USAGE.md | 89 +- code/go/ecs/client.go | 15 + code/go/ecs/cloud.go | 6 + code/go/ecs/code_signature.go | 10 + code/go/ecs/destination.go | 24 +- code/go/ecs/event.go | 6 +- code/go/ecs/file.go | 4 +- code/go/ecs/geo.go | 11 + code/go/ecs/hash.go | 9 +- code/go/ecs/host.go | 37 +- code/go/ecs/http.go | 20 + code/go/ecs/network.go | 23 +- code/go/ecs/observer.go | 18 +- code/go/ecs/os.go | 9 + code/go/ecs/related.go | 4 + code/go/ecs/server.go | 15 + code/go/ecs/source.go | 24 +- code/go/ecs/threat.go | 39 +- code/go/ecs/tracing.go | 3 + code/go/ecs/url.go | 19 +- code/go/ecs/version.go | 2 +- code/go/ecs/x509.go | 13 +- docs/additional.asciidoc | 2 + docs/artifacts.asciidoc | 6 + docs/converting.asciidoc | 21 +- docs/field-details.asciidoc | 2712 ++- docs/field-values-usage.asciidoc | 175 + docs/field-values.asciidoc | 82 +- docs/fields.asciidoc | 2 +- docs/index.asciidoc | 4 +- docs/products-solutions.asciidoc | 9 +- docs/usage/README.md | 40 + docs/usage/user.asciidoc | 430 + docs/using-conventions.asciidoc | 2 +- docs/using-getting-started.asciidoc | 5 +- docs/using-mapping-network-events.asciidoc | 267 + docs/using.asciidoc | 6 +- experimental/README.md | 26 + experimental/code/go/ecs/agent.go | 26 + experimental/code/go/ecs/as.go | 26 + experimental/code/go/ecs/client.go | 29 + experimental/code/go/ecs/data_stream.go | 66 + experimental/code/go/ecs/destination.go | 29 + experimental/code/go/ecs/dns.go | 32 + experimental/code/go/ecs/elf.go | 120 + experimental/code/go/ecs/error.go | 29 + experimental/code/go/ecs/file.go | 32 + experimental/code/go/ecs/geo.go | 26 + experimental/code/go/ecs/hash.go | 23 + experimental/code/go/ecs/host.go | 26 + experimental/code/go/ecs/http.go | 32 + experimental/code/go/ecs/log.go | 29 + experimental/code/go/ecs/orchestrator.go | 52 + experimental/code/go/ecs/organization.go | 26 + experimental/code/go/ecs/os.go | 29 + experimental/code/go/ecs/pe.go | 135 + experimental/code/go/ecs/process.go | 41 + experimental/code/go/ecs/registry.go | 32 + experimental/code/go/ecs/server.go | 29 + experimental/code/go/ecs/source.go | 29 + experimental/code/go/ecs/threat.go | 119 + experimental/code/go/ecs/tls.go | 35 + experimental/code/go/ecs/url.go | 38 + experimental/code/go/ecs/user.go | 32 + experimental/code/go/ecs/user_agent.go | 26 + experimental/code/go/ecs/version.go | 23 + experimental/code/go/ecs/x509.go | 29 + experimental/generated/beats/fields.ecs.yml | 9165 +++++++++ experimental/generated/csv/fields.csv | 1089 ++ experimental/generated/ecs/ecs_flat.yml | 13440 +++++++++++++ experimental/generated/ecs/ecs_nested.yml | 15999 ++++++++++++++++ .../generated/elasticsearch/7/template.json | 4926 +++++ .../elasticsearch/component/agent.json | 43 + .../elasticsearch/component/base.json | 26 + .../elasticsearch/component/client.json | 183 + .../elasticsearch/component/cloud.json | 80 + .../elasticsearch/component/container.json | 43 + .../elasticsearch/component/data_stream.json | 25 + .../elasticsearch/component/destination.json | 183 + .../elasticsearch/component/dll.json | 244 + .../elasticsearch/component/dns.json | 89 + .../elasticsearch/component/ecs.json | 20 + .../elasticsearch/component/error.json | 40 + .../elasticsearch/component/event.json | 109 + .../elasticsearch/component/file.json | 545 + .../elasticsearch/component/group.json | 28 + .../elasticsearch/component/host.json | 244 + .../elasticsearch/component/http.json | 88 + .../elasticsearch/component/log.json | 85 + .../elasticsearch/component/network.json | 86 + .../elasticsearch/component/observer.json | 213 + .../elasticsearch/component/organization.json | 29 + .../elasticsearch/component/package.json | 66 + .../elasticsearch/component/process.json | 862 + .../elasticsearch/component/registry.json | 45 + .../elasticsearch/component/related.json | 31 + .../elasticsearch/component/rule.json | 56 + .../elasticsearch/component/server.json | 183 + .../elasticsearch/component/service.json | 48 + .../elasticsearch/component/source.json | 183 + .../elasticsearch/component/threat.json | 572 + .../elasticsearch/component/tls.json | 346 + .../elasticsearch/component/tracing.json | 36 + .../elasticsearch/component/url.json | 78 + .../elasticsearch/component/user.json | 240 + .../elasticsearch/component/user_agent.json | 83 + .../component/vulnerability.json | 79 + .../generated/elasticsearch/template.json | 72 + experimental/schemas/agent.yml | 5 + experimental/schemas/as.yml | 9 + experimental/schemas/client.yml | 7 + experimental/schemas/data_stream.yml | 60 + experimental/schemas/destination.yml | 7 + experimental/schemas/dns.yml | 9 + experimental/schemas/elf.yml | 198 + experimental/schemas/error.yml | 9 + experimental/schemas/file.yml | 13 + experimental/schemas/geo.yml | 9 + experimental/schemas/hash.yml | 5 + experimental/schemas/host.yml | 4 + experimental/schemas/http.yml | 9 + experimental/schemas/log.yml | 7 + experimental/schemas/organization.yml | 5 + experimental/schemas/os.yml | 7 + experimental/schemas/pe.yml | 234 + experimental/schemas/process.yml | 15 + experimental/schemas/registry.yml | 13 + experimental/schemas/server.yml | 7 + experimental/schemas/source.yml | 7 + experimental/schemas/threat.yml | 196 + experimental/schemas/tls.yml | 11 + experimental/schemas/url.yml | 13 + experimental/schemas/user.yml | 9 + experimental/schemas/user_agent.yml | 5 + experimental/schemas/x509.yml | 7 + generated/README.md | 25 +- generated/beats/fields.ecs.yml | 1082 +- generated/csv/fields.csv | 1434 +- generated/ecs/ecs_flat.yml | 1350 +- generated/ecs/ecs_nested.yml | 1539 +- generated/elasticsearch/6/template.json | 432 +- generated/elasticsearch/7/template.json | 432 +- generated/elasticsearch/README.md | 178 +- generated/elasticsearch/component/agent.json | 44 + generated/elasticsearch/component/base.json | 26 + generated/elasticsearch/component/client.json | 190 + generated/elasticsearch/component/cloud.json | 80 + .../elasticsearch/component/container.json | 43 + .../elasticsearch/component/destination.json | 190 + generated/elasticsearch/component/dll.json | 109 + generated/elasticsearch/component/dns.json | 91 + generated/elasticsearch/component/ecs.json | 20 + generated/elasticsearch/component/error.json | 44 + generated/elasticsearch/component/event.json | 109 + generated/elasticsearch/component/file.json | 298 + generated/elasticsearch/component/group.json | 28 + generated/elasticsearch/component/host.json | 251 + generated/elasticsearch/component/http.json | 91 + generated/elasticsearch/component/log.json | 87 + .../elasticsearch/component/network.json | 86 + .../elasticsearch/component/observer.json | 216 + .../elasticsearch/component/organization.json | 30 + .../elasticsearch/component/package.json | 66 + .../elasticsearch/component/process.json | 370 + .../elasticsearch/component/registry.json | 48 + .../elasticsearch/component/related.json | 31 + generated/elasticsearch/component/rule.json | 56 + generated/elasticsearch/component/server.json | 190 + .../elasticsearch/component/service.json | 48 + generated/elasticsearch/component/source.json | 190 + generated/elasticsearch/component/threat.json | 80 + generated/elasticsearch/component/tls.json | 354 + .../elasticsearch/component/tracing.json | 36 + generated/elasticsearch/component/url.json | 83 + generated/elasticsearch/component/user.json | 252 + .../elasticsearch/component/user_agent.json | 86 + .../component/vulnerability.json | 79 + generated/elasticsearch/template.json | 71 + schemas/README.md | 34 +- schemas/client.yml | 22 + schemas/cloud.yml | 12 + schemas/code_signature.yml | 22 + schemas/destination.yml | 30 +- schemas/dns.yml | 4 +- schemas/event.yml | 43 +- schemas/file.yml | 7 +- schemas/geo.yml | 28 + schemas/hash.yml | 11 +- schemas/host.yml | 77 +- schemas/http.yml | 41 + schemas/network.yml | 19 +- schemas/observer.yml | 23 +- schemas/os.yml | 14 + schemas/process.yml | 2 +- schemas/related.yml | 10 + schemas/rule.yml | 2 +- schemas/server.yml | 22 + schemas/source.yml | 30 +- schemas/threat.yml | 89 +- schemas/tls.yml | 6 +- schemas/tracing.yml | 7 +- schemas/url.yml | 26 +- schemas/user.yml | 6 + schemas/x509.yml | 10 +- scripts/cmd/gocodegen/gocodegen.go | 4 +- scripts/generator.py | 24 +- scripts/generators/asciidoc_fields.py | 282 +- scripts/generators/beats.py | 23 +- ...yml => beats_default_fields_allowlist.yml} | 0 scripts/generators/ecs_helpers.py | 32 + scripts/generators/es_template.py | 214 +- scripts/requirements.txt | 4 +- scripts/schema/cleaner.py | 59 +- scripts/schema/finalizer.py | 9 +- scripts/schema/loader.py | 57 +- scripts/schema/oss.py | 31 + scripts/templates/field_details.j2 | 152 + .../field_details/acceptable_value_names.j2 | 8 - .../field_details/field_reuse_section.j2 | 6 - .../templates/field_details/nestings_row.j2 | 7 - .../field_details/nestings_table_header.j2 | 11 - scripts/templates/field_details/row.j2 | 14 - .../templates/field_details/table_header.j2 | 14 - ...eld_values_template.j2 => field_values.j2} | 21 +- .../{fields_template.j2 => fields.j2} | 0 scripts/tests/test_asciidoc_fields.py | 142 + scripts/tests/test_ecs_helpers.py | 8 + scripts/tests/test_es_template.py | 113 + scripts/tests/unit/test_schema_cleaner.py | 96 + scripts/tests/unit/test_schema_finalizer.py | 44 +- scripts/tests/unit/test_schema_loader.py | 127 + scripts/tests/unit/test_schema_oss.py | 44 + version | 2 +- 239 files changed, 67301 insertions(+), 2044 deletions(-) create mode 100644 .github/workflows/test.yml delete mode 100644 .travis.yml create mode 100644 docs/artifacts.asciidoc create mode 100644 docs/field-values-usage.asciidoc create mode 100644 docs/usage/README.md create mode 100644 docs/usage/user.asciidoc create mode 100644 docs/using-mapping-network-events.asciidoc create mode 100644 experimental/README.md create mode 100644 experimental/code/go/ecs/agent.go create mode 100644 experimental/code/go/ecs/as.go create mode 100644 experimental/code/go/ecs/client.go create mode 100644 experimental/code/go/ecs/data_stream.go create mode 100644 experimental/code/go/ecs/destination.go create mode 100644 experimental/code/go/ecs/dns.go create mode 100644 experimental/code/go/ecs/elf.go create mode 100644 experimental/code/go/ecs/error.go create mode 100644 experimental/code/go/ecs/file.go create mode 100644 experimental/code/go/ecs/geo.go create mode 100644 experimental/code/go/ecs/hash.go create mode 100644 experimental/code/go/ecs/host.go create mode 100644 experimental/code/go/ecs/http.go create mode 100644 experimental/code/go/ecs/log.go create mode 100644 experimental/code/go/ecs/orchestrator.go create mode 100644 experimental/code/go/ecs/organization.go create mode 100644 experimental/code/go/ecs/os.go create mode 100644 experimental/code/go/ecs/pe.go create mode 100644 experimental/code/go/ecs/process.go create mode 100644 experimental/code/go/ecs/registry.go create mode 100644 experimental/code/go/ecs/server.go create mode 100644 experimental/code/go/ecs/source.go create mode 100644 experimental/code/go/ecs/threat.go create mode 100644 experimental/code/go/ecs/tls.go create mode 100644 experimental/code/go/ecs/url.go create mode 100644 experimental/code/go/ecs/user.go create mode 100644 experimental/code/go/ecs/user_agent.go create mode 100644 experimental/code/go/ecs/version.go create mode 100644 experimental/code/go/ecs/x509.go create mode 100644 experimental/generated/beats/fields.ecs.yml create mode 100644 experimental/generated/csv/fields.csv create mode 100644 experimental/generated/ecs/ecs_flat.yml create mode 100644 experimental/generated/ecs/ecs_nested.yml create mode 100644 experimental/generated/elasticsearch/7/template.json create mode 100644 experimental/generated/elasticsearch/component/agent.json create mode 100644 experimental/generated/elasticsearch/component/base.json create mode 100644 experimental/generated/elasticsearch/component/client.json create mode 100644 experimental/generated/elasticsearch/component/cloud.json create mode 100644 experimental/generated/elasticsearch/component/container.json create mode 100644 experimental/generated/elasticsearch/component/data_stream.json create mode 100644 experimental/generated/elasticsearch/component/destination.json create mode 100644 experimental/generated/elasticsearch/component/dll.json create mode 100644 experimental/generated/elasticsearch/component/dns.json create mode 100644 experimental/generated/elasticsearch/component/ecs.json create mode 100644 experimental/generated/elasticsearch/component/error.json create mode 100644 experimental/generated/elasticsearch/component/event.json create mode 100644 experimental/generated/elasticsearch/component/file.json create mode 100644 experimental/generated/elasticsearch/component/group.json create mode 100644 experimental/generated/elasticsearch/component/host.json create mode 100644 experimental/generated/elasticsearch/component/http.json create mode 100644 experimental/generated/elasticsearch/component/log.json create mode 100644 experimental/generated/elasticsearch/component/network.json create mode 100644 experimental/generated/elasticsearch/component/observer.json create mode 100644 experimental/generated/elasticsearch/component/organization.json create mode 100644 experimental/generated/elasticsearch/component/package.json create mode 100644 experimental/generated/elasticsearch/component/process.json create mode 100644 experimental/generated/elasticsearch/component/registry.json create mode 100644 experimental/generated/elasticsearch/component/related.json create mode 100644 experimental/generated/elasticsearch/component/rule.json create mode 100644 experimental/generated/elasticsearch/component/server.json create mode 100644 experimental/generated/elasticsearch/component/service.json create mode 100644 experimental/generated/elasticsearch/component/source.json create mode 100644 experimental/generated/elasticsearch/component/threat.json create mode 100644 experimental/generated/elasticsearch/component/tls.json create mode 100644 experimental/generated/elasticsearch/component/tracing.json create mode 100644 experimental/generated/elasticsearch/component/url.json create mode 100644 experimental/generated/elasticsearch/component/user.json create mode 100644 experimental/generated/elasticsearch/component/user_agent.json create mode 100644 experimental/generated/elasticsearch/component/vulnerability.json create mode 100644 experimental/generated/elasticsearch/template.json create mode 100644 experimental/schemas/agent.yml create mode 100644 experimental/schemas/as.yml create mode 100644 experimental/schemas/client.yml create mode 100644 experimental/schemas/data_stream.yml create mode 100644 experimental/schemas/destination.yml create mode 100644 experimental/schemas/dns.yml create mode 100644 experimental/schemas/elf.yml create mode 100644 experimental/schemas/error.yml create mode 100644 experimental/schemas/file.yml create mode 100644 experimental/schemas/geo.yml create mode 100644 experimental/schemas/hash.yml create mode 100644 experimental/schemas/host.yml create mode 100644 experimental/schemas/http.yml create mode 100644 experimental/schemas/log.yml create mode 100644 experimental/schemas/organization.yml create mode 100644 experimental/schemas/os.yml create mode 100644 experimental/schemas/pe.yml create mode 100644 experimental/schemas/process.yml create mode 100644 experimental/schemas/registry.yml create mode 100644 experimental/schemas/server.yml create mode 100644 experimental/schemas/source.yml create mode 100644 experimental/schemas/threat.yml create mode 100644 experimental/schemas/tls.yml create mode 100644 experimental/schemas/url.yml create mode 100644 experimental/schemas/user.yml create mode 100644 experimental/schemas/user_agent.yml create mode 100644 experimental/schemas/x509.yml create mode 100644 generated/elasticsearch/component/agent.json create mode 100644 generated/elasticsearch/component/base.json create mode 100644 generated/elasticsearch/component/client.json create mode 100644 generated/elasticsearch/component/cloud.json create mode 100644 generated/elasticsearch/component/container.json create mode 100644 generated/elasticsearch/component/destination.json create mode 100644 generated/elasticsearch/component/dll.json create mode 100644 generated/elasticsearch/component/dns.json create mode 100644 generated/elasticsearch/component/ecs.json create mode 100644 generated/elasticsearch/component/error.json create mode 100644 generated/elasticsearch/component/event.json create mode 100644 generated/elasticsearch/component/file.json create mode 100644 generated/elasticsearch/component/group.json create mode 100644 generated/elasticsearch/component/host.json create mode 100644 generated/elasticsearch/component/http.json create mode 100644 generated/elasticsearch/component/log.json create mode 100644 generated/elasticsearch/component/network.json create mode 100644 generated/elasticsearch/component/observer.json create mode 100644 generated/elasticsearch/component/organization.json create mode 100644 generated/elasticsearch/component/package.json create mode 100644 generated/elasticsearch/component/process.json create mode 100644 generated/elasticsearch/component/registry.json create mode 100644 generated/elasticsearch/component/related.json create mode 100644 generated/elasticsearch/component/rule.json create mode 100644 generated/elasticsearch/component/server.json create mode 100644 generated/elasticsearch/component/service.json create mode 100644 generated/elasticsearch/component/source.json create mode 100644 generated/elasticsearch/component/threat.json create mode 100644 generated/elasticsearch/component/tls.json create mode 100644 generated/elasticsearch/component/tracing.json create mode 100644 generated/elasticsearch/component/url.json create mode 100644 generated/elasticsearch/component/user.json create mode 100644 generated/elasticsearch/component/user_agent.json create mode 100644 generated/elasticsearch/component/vulnerability.json create mode 100644 generated/elasticsearch/template.json rename scripts/generators/{beats_default_fields_whitelist.yml => beats_default_fields_allowlist.yml} (100%) create mode 100644 scripts/schema/oss.py create mode 100644 scripts/templates/field_details.j2 delete mode 100644 scripts/templates/field_details/acceptable_value_names.j2 delete mode 100644 scripts/templates/field_details/field_reuse_section.j2 delete mode 100644 scripts/templates/field_details/nestings_row.j2 delete mode 100644 scripts/templates/field_details/nestings_table_header.j2 delete mode 100644 scripts/templates/field_details/row.j2 delete mode 100644 scripts/templates/field_details/table_header.j2 rename scripts/templates/{field_values_template.j2 => field_values.j2} (71%) rename scripts/templates/{fields_template.j2 => fields.j2} (100%) create mode 100644 scripts/tests/test_asciidoc_fields.py create mode 100644 scripts/tests/unit/test_schema_oss.py diff --git a/.github/workflows/test.yml b/.github/workflows/test.yml new file mode 100644 index 0000000000..f5aad7d831 --- /dev/null +++ b/.github/workflows/test.yml @@ -0,0 +1,19 @@ +name: Tests + +on: [push, pull_request] + +jobs: + tests: + runs-on: ubuntu-20.04 + name: Unit Tests + steps: + - uses: actions/checkout@v2 + - uses: actions/setup-go@v2 + with: + go-version: '1.15.x' + - uses: actions/setup-python@v2 + with: + python-version: '3.x' + - run: git fetch --prune --unshallow --tags + - run: make setup + - run: make check diff --git a/.gitignore b/.gitignore index 20c4de146e..a3cabe1d6a 100644 --- a/.gitignore +++ b/.gitignore @@ -7,3 +7,7 @@ build .idea *.iml .vscode/* + +# experimental exclusions +experimental/generated/elasticsearch/6 +experimental/generated/docs diff --git a/.travis.yml b/.travis.yml deleted file mode 100644 index a56a30a073..0000000000 --- a/.travis.yml +++ /dev/null @@ -1,29 +0,0 @@ -sudo: false - -language: go - -os: -- linux - -dist: bionic - -go: -- 1.13.x - -install: -- git fetch --tags --all -- make setup - -addons: - apt: - update: true - packages: - - libxml2-utils - - python3-venv - - xsltproc - -jobs: - include: - - stage: check - script: - - make check diff --git a/CHANGELOG.md b/CHANGELOG.md index 5510520c9f..83526636a4 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -3,6 +3,240 @@ # CHANGELOG All notable changes to this project will be documented in this file based on the [Keep a Changelog](http://keepachangelog.com/) Standard. This project adheres to [Semantic Versioning](http://semver.org/). +## [1.9.0](https://github.com/elastic/ecs/compare/v1.8.0...v1.9.0) + +### Schema Changes + +#### Added + +* Added `hash.ssdeep`. #1169 +* Added `cloud.service.name`. #1204 +* Added `http.request.id`. #1208 +* `data_stream.*` fieldset introduced in experimental schema and artifacts. #1215 +* Added `geo.timezone`, `geo.postal_code`, and `geo.continent_code`. #1229 +* Added `beta` host metrics fields. #1248 +* Added `code_signature.team_id`, `code_signature.signing_id`. #1249 +* Extended `pe` fields added to experimental schema. #1256 +* Add `elf` fieldset to experimental schema. #1261 +* Add `threat.indicator` fields to experimental schema. #1268 + +#### Improvements + +* Include formatting guidance and examples for MAC address fields. #456 +* New section in ECS detailing event categorization fields usage. #1242 +* `user.changes.*`, `user.effective.*`, and `user.target.*` field reuses are GA. #1271 + +### Tooling and Artifact Changes + +#### Improvements + +* Update Python dependencies #1310, #1318 +* Adjustments to use terminology that doesn't have negative connotation. #1315 + + +## [1.8.0](https://github.com/elastic/ecs/compare/v1.7.0...v1.8.0) + +### Schema Changes + +#### Bugfixes + +* Clean up `event.reference` description. #1181 +* Go code generator fails if `scaled_float` type is used. #1250 + +#### Added + +* Added `event.category` "registry". #1040 +* Added `event.category` "session". #1049 +* Added usage documentation for `user` fields. #1066 +* Added `user` fields at `user.effective.*`, `user.target.*` and `user.changes.*`. #1066 +* Added `os.type`. #1111 + +#### Improvements + +* Event categorization fields GA. #1067 +* Note `[` and `]` bracket characters may enclose a literal IPv6 address when populating `url.domain`. #1131 +* Reinforce the exclusion of the leading dot from `url.extension`. #1151 + +#### Deprecated + +* Deprecated `host.user.*` fields for removal at the next major. #1066 + +### Tooling and Artifact Changes + +#### Bugfixes + +* `tracing` fields should be at root of Beats `fields.ecs.yml` artifacts. #1164 + +#### Added + +* Added the `path` key when type is `alias`, to support the [alias field type](https://www.elastic.co/guide/en/elasticsearch/reference/current/alias.html). #877 +* Added support for `scaled_float`'s mandatory parameter `scaling_factor`. #1042 +* Added ability for --oss flag to fall back `constant_keyword` to `keyword`. #1046 +* Added support in the generated Go source go for `wildcard`, `version`, and `constant_keyword` data types. #1050 +* Added support for marking fields, field sets, or field reuse as beta in the documentation. #1051 +* Added support for `constant_keyword`'s optional parameter `value`. #1112 +* Added component templates for ECS field sets. #1156, #1186, #1191 +* Added functionality for merging custom and core multi-fields. #982 + +#### Improvements + +* Make all fields linkable directly. #1148 +* Added a notice highlighting that the `tracing` fields are not nested under the + namespace `tracing.` #1162 +* ES 6.x template data types will fallback to supported types. #1171, #1176, #1186 +* Add a documentation page discussing the experimental artifacts. #1189 + +## [1.7.0](https://github.com/elastic/ecs/compare/v1.6.0...v1.7.0) + +### Schema Changes + +#### Bugfixes + +* The `protocol` allowed value under `event.type` should not have the `expected_event_types` defined. #964 +* Clarify the definition of `file.extension` (no dots). #1016 + +#### Added + +* Added Mime Type fields to HTTP request and response. #944 +* Added network directions ingress and egress. #945 +* Added `threat.technique.subtechnique` to capture MITRE ATT&CK® subtechniques. #951 +* Added `configuration` as an allowed `event.category`. #963 +* Added a new directory with experimental artifacts, which includes all changes + from RFCs that have reached stage 2. #993, #1053, #1115, #1117, #1118 + +#### Improvements + +* Expanded field set definitions for `source.*` and `destination.*`. #967 +* Provided better guidance for mapping network events. #969 +* Added the field `.subdomain` under `client`, `destination`, `server`, `source` + and `url`, to match its presence at `dns.question.subdomain`. #981 +* Clarified ambiguity in guidance on how to use x509 fields for connections with + only one certificate. #1114 + +### Tooling and Artifact Changes + +#### Breaking changes + +* Changed the index pattern of the sample Elasticsearch template from `ecs-*` to + `try-ecs-*` to avoid conflicting with Logstash' `ecs-logstash-*`. #1048 + +#### Bugfixes + +* Addressed issue where foreign reuses weren't using the user-supplied `as` value for their destination. #960 +* Experimental artifacts failed to install due to `event.original` index setting. #1053 + +#### Added + +* Introduced `--strict` flag to perform stricter schema validation when running the generator script. #937 +* Added check under `--strict` that ensures composite types in example fields are quoted. #966 +* Added `ignore_above` and `normalizer` support for keyword multi-fields. #971 +* Added ability to supply free-form usage documentation per fieldset. #988 +* Added `--oss` flag for users who want to generate ECS templates for use on OSS clusters. #991 + +#### Improvements + +* Field details Jinja2 template components have been consolidated into one template #897 +* Add `[discrete]` marker before each section header in field details. #989 +* `--ref` now loads `experimental/schemas` based on git ref in addition to `schemas`. #1063 + + +## [1.6.0](https://github.com/elastic/ecs/compare/v1.5.0...v1.6.0) + +### Schema Changes + +#### Bugfixes + +* Field `registry.data.strings` should have been marked as an array field. #790 + +#### Added + +* Added `x509.*` field set. #762 +* Add architecture and imphash for PE field set. #763 +* Added `agent.build.*` for extended agent version information. #764 +* Added `log.file.path` to capture the log file an event came from. #802 +* Added more account and project cloud metadata. #816 +* Added missing field reuse of `pe` at `process.parent.pe` #868 +* Added `span.id` to the tracing fieldset, for additional log correlation #882 +* Added `event.reason` for the reason why an event's outcome or action was taken. #907 +* Added `user.roles` to capture a list of role names that apply to the user. #917 + +#### Improvements + +* Removed misleading pluralization in the description of `user.id`, it should + contain one ID, not many. #801 +* Clarified misleading wording about multiple IPs in src/dst or cli/srv. #804 +* Improved verbiage about the MITRE ATT&CK® framework. #866 +* Removed the default `object_type=keyword` that was being applied to `object` fields. + This attribute is Beats-specific. It's still supported, but needs to be set explicitly + on a case by case basis now. This default being removed affects `dns.answers`, + `log.syslog`, `network.inner`, `observer.egress`, and `observer.ingress`. #871 +* Improved attribute `dashed_name` in `generated/ecs/*.yml` to also + replace `@` with `-`. #871 +* Updated several URLs in the documentation with "example.com" domain. #910 + +#### Deprecated + +* Deprecate guidance to lowercase `http.request.method` #840 + + +### Tooling and Artifact Changes + +#### Breaking changes + +* Removed field definitions at the root of documents for fieldsets that + had `reusable.top_level:false`. This PR affects `ecs_flat.yml`, the csv file + and the sample Elasticsearch templates. #495, #813 +* Removed the `order` attribute from the `ecs_nested.yml` and `ecs_flat.yml` files. #811 +* In `ecs_nested.yml`, the array of strings that used to be in `reusable.expected` + has been replaced by an array of objects with 3 keys: 'as', 'at' and 'full'. #864 +* The subset format now requires `name` and `fields` keys at the top level. #873 + +#### Bugfixes + +* Subsets are created after duplicating reusable fields now so subsets can + be applied to each reused instance independently. #753 +* Quoted the example for `labels` to avoid YAML interpreting it, and having + slightly different results in different situations. #782 +* Fix incorrect listing of where field sets are nested in asciidoc, + when they are nested deep. #784 +* Allow beats output to be generated when using `--include` or `--subset` flags. #814 +* Field parameter `index` is now correctly populated in the Beats field definition file. #824 + +#### Improvements + +* Add support for reusing official fieldsets in custom schemas. #751 +* Add full path names to reused fieldsets in `nestings` array in `ecs_nested.yml`. #803 +* Allow shorthand notation for including all subfields in subsets. #805 +* Add support for Elasticsearch `enabled` field parameter. #824 +* Add `ref` option to generator allowing schemas to be built for a specific ECS version. #851 +* Add `template-settings` and `mapping-settings` options to allow override of defaults in generated ES templates. #856 +* When overriding ECS field sets via the `--include` flag, it's no longer necessary + to duplicate the field set's mandatory attributes. The customizations are merged + before validation. #864 +* Add ability to nest field sets as another name. #864 +* Add ability to nest field sets within themselves (e.g. `process` => `process.parent`). #864 +* New attribute `reused_here` is added in `ecs_nested.yml`. It obsoletes the + previous attribute `nestings`, and is able to fully capture details of other + field sets reused under this one. #864 +* When chained reuses are needed (e.g. `group` => `user`, then `user` => many places), + it's now necessary to force the order with new attribute `reusable.order`. This + attribute is otherwise optional. It's currently only needed for `group`. #864 +* There's a new representation of ECS at `generated/ecs/ecs.yml`, which is a deeply nested + representation of the fields. This file is not in git, as it's only meant for + developers working on the ECS tools. #864 +* Jinja2 templates now define the doc structure for the AsciiDoc generator. #865 +* Intermediate `ecs_flat.yml` and `ecs_nested.yml` files are now generated for each individual subset, + in addition to the intermediate files generated for the combined subset. #873 + +#### Deprecated + +* In `ecs_nested.yml`, we're deprecating the attribute `nestings`. It will be + removed in a future release. The deprecated `nestings` attribute was an array of + flat field names describing where fields are nested within the field set. + This is replaced with the attribute `reused_here`, which is an array of objects. + The new format still lists where the fields are nested via the same flat field name, + but also specifies additional information about each field reuse. #864 + ## [1.5.0](https://github.com/elastic/ecs/compare/v1.4.0...v1.5.0) diff --git a/CHANGELOG.next.md b/CHANGELOG.next.md index ae6e775639..4c99b1e12d 100644 --- a/CHANGELOG.next.md +++ b/CHANGELOG.next.md @@ -10,103 +10,26 @@ Thanks, you're awesome :-) --> ### Schema Changes -* Added `log.file.path` to capture the log file an event came from. #802 - #### Breaking changes #### Bugfixes -* Field `registry.data.strings` should have been marked as an array field. #790 - #### Added -* Add architecture and imphash for PE field set. (#763) -* Added `agent.build.*` for extended agent version information. (#764) -* Added `x509.*` field set. (#762) -* Added more account and project cloud metadata. (#816) -* Added missing field reuse of `pe` at `process.parent.pe` #868 -* Added `span.id` to the tracing fieldset, for additional log correlation (#882) -* Added `event.reason` for the reason why an event's outcome or action was taken. #907 - #### Improvements -* Removed misleading pluralization in the description of `user.id`, it should - contain one ID, not many. #801 -* Clarified misleading wording about multiple IPs in src/dst or cli/srv. #804 -* Improved verbiage about the MITRE ATT&CK® framework. #866 -* Removed the default `object_type=keyword` that was being applied to `object` fields. - This attribute is Beats-specific. It's still supported, but needs to be set explicitly - on a case by case basis now. This default being removed affects `dns.answers`, - `log.syslog`, `network.inner`, `observer.egress`, and `observer.ingress`. #871 -* Improved attribute `dashed_name` in `generated/ecs/*.yml` to also - replace `@` with `-`. #871 -* Updated several URLs in the documentation with "example.com" domain. #910 - -#### Deprecated - -* Deprecate guidance to lowercase `http.request.method` #840 -* In `ecs_nested.yml`, we're deprecating the attribute `nestings`. It will be - removed in a future release. The deprecated `nestings` attribute was an array of - flat field names describing where fields are nested within the field set. - This is replaced with the attribute `reused_here`, which is an array of objects. - The new format still lists where the fields are nested via the same flat field name, - but also specifies additional information about each field reuse. - - ### Tooling and Artifact Changes #### Breaking changes -* Removed field definitions at the root of documents for fieldsets that - had `reusable.top_level:false`. This PR affects `ecs_flat.yml`, the csv file - and the sample Elasticsearch templates. #495, #813 -* Removed the `order` attribute from the `ecs_nested.yml` and `ecs_flat.yml` files. #811 -* In `ecs_nested.yml`, the array of strings that used to be in `reusable.expected` - has been replaced by an array of objects with 3 keys: 'as', 'at' and 'full'. #864 -* The subset format now requires `name` and `fields` keys at the top level. #873 - #### Bugfixes -* Subsets are created after duplicating reusable fields now so subsets can - be applied to each reused instance independently. #753 -* Quoted the example for `labels` to avoid YAML interpreting it, and having - slightly different results in different situations. #782 -* Fix incorrect listing of where field sets are nested in asciidoc, - when they are nested deep. #784 -* Allow beats output to be generated when using `--include` or `--subset` flags. #814 -* Field parameter `index` is now correctly populated in the Beats field definition file. #824 - #### Added #### Improvements -* Add support for reusing official fieldsets in custom schemas. #751 -* Add full path names to reused fieldsets in `nestings` array in `ecs_nested.yml`. #803 -* Allow shorthand notation for including all subfields in subsets. #805 -* Add support for Elasticsearch `enabled` field parameter. #824 -* Add `ref` option to generator allowing schemas to be built for a specific ECS version. #851 -* Add `template-settings` and `mapping-settings` options to allow override of defaults in generated ES templates. #856 -* When overriding ECS field sets via the `--include` flag, it's no longer necessary - to duplicate the field set's mandatory attributes. The customizations are merged - before validation. #864 -* Add ability to nest field sets as another name. #864 -* Add ability to nest field sets within themselves (e.g. `process` => `process.parent`). #864 -* New attribute `reused_here` is added in `ecs_nested.yml`. It obsoletes the - previous attribute `nestings`, and is able to fully capture details of other - field sets reused under this one. #864 -* When chained reuses are needed (e.g. `group` => `user`, then `user` => many places), - it's now necessary to force the order with new attribute `reusable.order`. This - attribute is otherwise optional. It's currently only needed for `group`. #864 -* There's a new representation of ECS at `generated/ecs/ecs.yml`, which is a deeply nested - representation of the fields. This file is not in git, as it's only meant for - developers working on the ECS tools. #864 -* Jinja2 templates now define the doc structure for the AsciiDoc generator. #865 -* Intermediate `ecs_flat.yml` and `ecs_nested.yml` files are now generated for each individual subset, - in addition to the intermediate files generated for the combined subset. #873 - #### Deprecated -