Invalid ECS field usages at root-level

[andrewkroh]

Across packages owned by elastic/security-external-integrations the following fields are being used at the document root, but according to ECS they are only allowed be nested under other ECS namespaces like host or source. These usages need to be changed to align with ECS. And fixing these issues will be required to move to package-spec 3.0.0.

Field Name	Usage count
as.number	2
geo.city_name	6
geo.continent_name	4
interface.id	2
interface.name	3
os.family	3
os.name	7
os.type	2
vlan.id	2
x509.issuer.common_name	2

This was detected by looking at fields.yml mappings only. It's possible that the fields are not actually used in some cases. If I accidentally included a deprecated or rsa2elk package then please ignore that field.

Source Locations

Package	Data Stream	Field	Github Source
1password	item_usages	os.name	View Source
1password	signin_attempts	os.name	View Source
azure_frontdoor	access	geo.city_name	View Source
azure_frontdoor	access	geo.continent_name	View Source
azure_frontdoor	waf	geo.city_name	View Source
azure_frontdoor	waf	geo.continent_name	View Source
bluecoat	director	geo.city_name	View Source
carbonblack_edr	log	os.type	View Source
cisco_aironet	log	interface.id	View Source
cisco_meraki	events	geo.city_name	View Source
cisco_meraki	log	geo.city_name	View Source
cloudflare_logpush	network_session	vlan.id	View Source
crowdstrike	fdr	os.type	View Source
f5	bigipafm	geo.city_name	View Source
f5	bigipapm	geo.city_name	View Source
fireeye	nx	interface.name	View Source
infoblox_nios	log	interface.name	View Source
juniper_junos	log	geo.city_name	View Source
juniper_netscreen	log	geo.city_name	View Source
juniper_srx	log	as.number	View Source
juniper_srx	log	geo.city_name	View Source
juniper_srx	log	geo.continent_name	View Source
juniper_srx	log	interface.id	View Source
juniper_srx	log	interface.name	View Source
juniper_srx	log	os.family	View Source
juniper_srx	log	os.name	View Source
juniper_srx	log	vlan.id	View Source
juniper_srx	log	x509.issuer.common_name	View Source
lyve_cloud	audit	os.name	View Source
netflow	log	as.number	View Source
netflow	log	geo.city_name	View Source
netflow	log	geo.continent_name	View Source
netflow	log	os.family	View Source
netflow	log	os.name	View Source
panw	panos	x509.issuer.common_name	View Source
sentinel_one	activity	os.family	View Source
sentinel_one	alert	os.name	View Source
trend_micro_vision_one	detection	os.name	View Source

(List generated with an agg on top of query @attributes.deprecated:false and @attributes.rsa2elk:false and @owner:elastic/security-external-integrations and @type:field and name:(vlan.id or geo.continent_name or os.type or interface.id or os.name or interface.name or as.number or os.name or os.name or as.number or os.family or os.type or interface.name or x509.issuer.common_name or geo.city_name) to https://github.com/andrewkroh/go-examples/tree/main/fleetpkg-indexer)

[elasticmachine]

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

[kgeller]

Just for my own record keeping sake I am converting the above table into categories: 1) in need of pipeline updates 2) false alarms that only need cleanup to their ecs field declarations 3) omitted

Integrations in need of pipeline updates:

• 1password [ECS][1password] Correcting invalid ECS field usages at root-level #7963
• cisco_aironet [ECS][cisco_aironet] Correcting invalid ECS field usages at root-level #7966
• cloudflare_logpush [ECS][cloudflare_logpush] Correcting invalid ECS field usages at root-level #7967
• crowdstrike [ECS][crowdstrike] Correcting invalid ECS field usages at root-level #7968
• fireeye [ECS][fireeye] Correcting invalid ECS field declarations at root-level #7969
• infoblox_nios [ECS][infoblox_nios] Correcting invalid ECS field usages at root-level #7981
• lyve_cloud [ECS][lyve_cloud] Correcting invalid ECS field usages at root-level #7982
• panw [ECS][panw] Correcting invalid ECS field usages at root-level #7984
• sentinel_one [ECS][sentinel_one] Correcting invalid ECS field usages at root-level #7985
• trend_micro_vision_one [ECS][trend_micro_vision_one] Correcting invalid ECS field usages at root-level #7986

Integrations for ecs field declaration updates:

• (Combined PR [ECS] Cleaning up incorrect invalid ECS field usages at root-level #7965 )
• azure_frontdoor
• carbonblack_edr
• cisco_meraki
• juniper_srx
• netflow

Omitted

• bluecoat - deprecated
• f5 - deprecated
• juniper_junos - deprecated
• juniper_netscreen - deprecated

[kgeller]

All non-deprecated packages noted above have now been updated.

[andrewkroh]

juniper_srx and netflow still have issues.

packages/juniper_srx/data_stream/log/fields/ecs.yml:15:3 as.organization.name
packages/juniper_srx/data_stream/log/fields/ecs.yml:81:3 code_signature.exists
packages/juniper_srx/data_stream/log/fields/ecs.yml:83:3 code_signature.status
packages/juniper_srx/data_stream/log/fields/ecs.yml:85:3 code_signature.subject_name
packages/juniper_srx/data_stream/log/fields/ecs.yml:87:3 code_signature.trusted
packages/juniper_srx/data_stream/log/fields/ecs.yml:89:3 code_signature.valid
packages/juniper_srx/data_stream/log/fields/ecs.yml:367:3 hash.md5
packages/juniper_srx/data_stream/log/fields/ecs.yml:369:3 hash.sha1
packages/juniper_srx/data_stream/log/fields/ecs.yml:371:3 hash.sha256
packages/juniper_srx/data_stream/log/fields/ecs.yml:373:3 hash.sha512
packages/juniper_srx/data_stream/log/fields/ecs.yml:555:3 pe.architecture
packages/juniper_srx/data_stream/log/fields/ecs.yml:557:3 pe.company
packages/juniper_srx/data_stream/log/fields/ecs.yml:559:3 pe.description
packages/juniper_srx/data_stream/log/fields/ecs.yml:561:3 pe.file_version
packages/juniper_srx/data_stream/log/fields/ecs.yml:563:3 pe.imphash
packages/juniper_srx/data_stream/log/fields/ecs.yml:565:3 pe.original_file_name
packages/juniper_srx/data_stream/log/fields/ecs.yml:567:3 pe.product
packages/netflow/data_stream/log/fields/ecs.yml:13:3 as.organization.name
packages/netflow/data_stream/log/fields/ecs.yml:283:3 geo.city_name
packages/netflow/data_stream/log/fields/ecs.yml:285:3 geo.continent_name
packages/netflow/data_stream/log/fields/ecs.yml:287:3 geo.country_iso_code
packages/netflow/data_stream/log/fields/ecs.yml:289:3 geo.country_name
packages/netflow/data_stream/log/fields/ecs.yml:291:3 geo.location
packages/netflow/data_stream/log/fields/ecs.yml:293:3 geo.name
packages/netflow/data_stream/log/fields/ecs.yml:295:3 geo.region_iso_code
packages/netflow/data_stream/log/fields/ecs.yml:297:3 geo.region_name
packages/netflow/data_stream/log/fields/ecs.yml:305:3 hash.md5
packages/netflow/data_stream/log/fields/ecs.yml:307:3 hash.sha1
packages/netflow/data_stream/log/fields/ecs.yml:309:3 hash.sha256
packages/netflow/data_stream/log/fields/ecs.yml:311:3 hash.sha512
packages/netflow/data_stream/log/fields/ecs.yml:467:3 os.family
packages/netflow/data_stream/log/fields/ecs.yml:469:3 os.full
packages/netflow/data_stream/log/fields/ecs.yml:471:3 os.kernel
packages/netflow/data_stream/log/fields/ecs.yml:473:3 os.name
packages/netflow/data_stream/log/fields/ecs.yml:475:3 os.platform
packages/netflow/data_stream/log/fields/ecs.yml:477:3 os.version