Rename source_ecs to source #8983

andrewkroh · 2018-11-07T23:16:04Z

Update Suricata module to use source instead of source_ecs now that the Filebeat source
field has been changed. This updates libbeat/_meta/fields.ecs.yml to include the ECS
source.* fields.

This updates the dashboards and visualizations to use the new source fields. And I also updated
the saved searches to use event.module instead of fileset.module.

Update Suricata module to use `source` instead of `source_ecs` now that the Filebeat source field has been changed. This updates `libbeat/_meta/fields.ecs.yml` to include the ECS `source.*` fields. This updates the dashboards and visualizations to use the new `source` fields. And I also updated the saved searches to use `event.module` instead of `fileset.module`. Remove source.ip, source,port, and source.hostname from auditbeat's auditd module fields. Remove source.ip, source.port, and source.mac from packetbeat's flow fields.

ruflin · 2018-11-08T11:07:16Z

libbeat/_meta/fields.ecs.yml

+
+          Can be one or multiple IPv4 or IPv6 addresses.
+
+      - name: hostname


@andrewkroh @webmat This does not exist anymore in ECS. Do we need it for suricata? Can you two sync up?

The ECS fields I used are stale because I moved this source definition from filebeat to libbeat, rather than getting an entirely new copy from elastic/ecs. We should do a full update in a separate PR (we have to verify that any fields removed from ECS are still accounted for elsewhere in fields.yml files when we do this).

Auditbeat had source.hostname in its fields.yml, so I had to remove it because it was now duplicated. I think Auditbeat uses source.hostname in cases where the SSH daemon reports both the remote client's IP and the reverse DNS name of that IP.

We can proceed with a rename in Auditbeat or a change ECS. Similarly we are using source.geo.* which is not in ECS.

Just to clarify, geo is a reuseable object, and source and destination are both expected places where geo can be nested. The template.json doesn't reflect it yet, but the official spec is the readme :-)

So source.geo.* is perfectly ECS compliant.

Agreed that source.hostname should be replaced by source.domain. Same goes for destination.hostname.

If you want to store an ambiguous value as well, before determining if it's a hostname or an IP, ECS no longer has a field for this (at least for now). You may store it in a custom field, or you may add logic to save to .ip only when it's an IP, or save to .domain when it's a hostname/domain.

Ok, I'll update Auditbeat to send source.domain instead of source.hostname.

I removed the source.hostname and destination.hostname fields. And I added the source.geo.* and destination.geo.*.

I reverted the changes to Auditbeat's mapping w.r.t. source.hostname as I will do that in an independent PR since it's separate change.

ruflin · 2018-11-08T11:08:21Z

x-pack/filebeat/module/suricata/_meta/kibana/6/dashboard/Filebeat-Suricata-Alert-Overview.json

@@ -272,7 +272,7 @@
                  "alias": null,
                  "disabled": false,
                  "index": "filebeat-*",
-                  "key": "fileset.module",
+                  "key": "event.module",


Not sure if we should update the dashboards here as I planned to this in bulk. Like this we have some dashboards converted but not all. But not sure if it will have an affect on the migration.

I needed to update them to test my UI changes, but I could undo these lines. However if the bulk migration just does a find/replace on fileset.module this won't be a problem. I'm in favor of leaving it as is, but I'll follow your direction.

ruflin · 2018-11-08T11:08:54Z

x-pack/filebeat/module/suricata/eve/test/eve-alerts.log-expected.json

-        "source_ecs.ip": "192.168.1.146", 
-        "source_ecs.port": 52340, 
+        "source.ip": "192.168.1.146",
+        "source.port": 52340,
        "suricata.eve.alert.action": "allowed", 
        "suricata.eve.alert.category": "Not Suspicious Traffic", 


@tsg category ;-)

webmat

LGTM overall.

I'd just make sure to get rid of source.hostname and destination.hostname.

I don't have a strong opinion on whether or not to rename fileset.module now vs as part of Nic's PR. I agree that if it's done via a find/replace, it shouldn't matter :-)

webmat · 2018-11-08T20:10:36Z

dev-tools/ecs-migration.yml

+  alias: true
+  copy_to: false
+
+- from: source_ecs.geo.continent_name


Needlessly verbose, IMO. What about the following?

- from: source_ecs.geo to: source.geo

I don't know how ecs-migration.yml will be used so I can't say for sure whether this is needlessly verbose. But if I can speculate, I think the level of detail here is appropriate if this will be used to create aliases because aliases must target concrete fields, not objects like source.geo.

webmat · 2018-11-08T20:17:12Z

libbeat/_meta/fields.ecs.yml

+
+          Can be one or multiple IPv4 or IPv6 addresses.
+
+      - name: hostname


Agreed that source.hostname should be replaced by source.domain. Same goes for destination.hostname.

If you want to store an ambiguous value as well, before determining if it's a hostname or an IP, ECS no longer has a field for this (at least for now). You may store it in a custom field, or you may add logic to save to .ip only when it's an IP, or save to .domain when it's a hostname/domain.

Removes `source.hostname` and `destination.hostname` from fields.ecs.yml. Adds `source.geo` and `destination.geo` to fields.ecs.yml.

webmat

LGTM

webmat · 2018-11-09T15:09:43Z

@andrewkroh Actually there's a mention of source_ecs left in the Alerts-Overview dashboard

* Rename source_ecs to source Update Suricata module to use `source` instead of `source_ecs` now that the Filebeat source field has been changed. This updates `libbeat/_meta/fields.ecs.yml` to include the ECS `source.*` fields. This updates the dashboards and visualizations to use the new `source` fields. And I also updated the saved searches to use `event.module` instead of `fileset.module`. - Remove source.ip, source,port from auditbeat's auditd module fields (source.hostname remains, will remove separately). - Remove source.ip, source.port, and source.mac from packetbeat's flow fields. - Adds `source.geo` and `destination.geo` to fields.ecs.yml.

andrewkroh force-pushed the feature/fb/rename-source_ecs branch from ad51a65 to 93d9d6c Compare November 7, 2018 23:16

andrewkroh added review Filebeat Filebeat labels Nov 7, 2018

andrewkroh force-pushed the feature/fb/rename-source_ecs branch from 93d9d6c to 3eb2749 Compare November 8, 2018 00:38

ruflin reviewed Nov 8, 2018

View reviewed changes

webmat added the ecs label Nov 8, 2018

webmat reviewed Nov 8, 2018

View reviewed changes

Remove source.hostname and destination.hostname

64dbd42

Removes `source.hostname` and `destination.hostname` from fields.ecs.yml. Adds `source.geo` and `destination.geo` to fields.ecs.yml.

webmat mentioned this pull request Nov 9, 2018

Convert Filebeat's traefik.access to ECS. #9005

Merged

19 tasks

webmat approved these changes Nov 9, 2018

View reviewed changes

Update dashboard to remove source_ecs

e3e0b7c

webmat mentioned this pull request Nov 9, 2018

[WIP] Import ECS 1.0.0 Beta 1 field definitions #9014

Closed

20 tasks

andrewkroh merged commit 2b748a3 into elastic:master Nov 9, 2018

ruflin mentioned this pull request Nov 14, 2018

Transition Beats to ECS #8655

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Rename source_ecs to source #8983

Rename source_ecs to source #8983

andrewkroh commented Nov 7, 2018

ruflin Nov 8, 2018

andrewkroh Nov 8, 2018

webmat Nov 8, 2018

webmat Nov 8, 2018

andrewkroh Nov 8, 2018

andrewkroh Nov 8, 2018

ruflin Nov 8, 2018

andrewkroh Nov 8, 2018

ruflin Nov 8, 2018

webmat left a comment

webmat Nov 8, 2018

andrewkroh Nov 8, 2018

webmat Nov 9, 2018

webmat Nov 8, 2018

webmat left a comment

webmat commented Nov 9, 2018 •

edited

Loading


		Can be one or multiple IPv4 or IPv6 addresses.

		- name: hostname

Rename source_ecs to source #8983

Rename source_ecs to source #8983

Conversation

andrewkroh commented Nov 7, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

webmat left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

webmat left a comment

Choose a reason for hiding this comment

webmat commented Nov 9, 2018 • edited Loading

webmat commented Nov 9, 2018 •

edited

Loading