From cc8c923fe099e0700a32289c00e780a8646cead9 Mon Sep 17 00:00:00 2001 From: Tudor Golubenco Date: Thu, 21 Mar 2019 10:59:55 +0100 Subject: [PATCH 1/2] Adding categorization fields for the system/auth module (#11334) * Adding categorization fields for the system/auth module This PR adds the following fields for the SSH login events: * `event.category: authentication` * `event.action: ssh_login` * `event.type` either `authentication_success` or `authentication_failure` The `event.outcome` is currently not quite ECS compliant, but I didn't touch it to avoid a breaking change. The PR doesn't attempt to categorize other logs besides the SSH login attempts, so it's a subset of #9905, but it's what we need for the UI. * Normalized event.outcome and brought back `system.auth.ssh.event`. * changelog (cherry picked from commit a9f567b4a43fe9978a6d1671cb1e10357b0972ea) --- CHANGELOG.next.asciidoc | 11 ++++++++ dev-tools/ecs-migration.yml | 5 ---- filebeat/docs/fields.asciidoc | 5 ++-- filebeat/module/system/auth/_meta/fields.yml | 7 ++--- .../module/system/auth/ingest/pipeline.json | 11 ++++++-- .../system/auth/test/test.log-expected.json | 26 +++++++++++++++---- filebeat/module/system/fields.go | 2 +- 7 files changed, 49 insertions(+), 18 deletions(-) diff --git a/CHANGELOG.next.asciidoc b/CHANGELOG.next.asciidoc index 0fb134bf49f..202878e605d 100644 --- a/CHANGELOG.next.asciidoc +++ b/CHANGELOG.next.asciidoc @@ -77,6 +77,8 @@ https://github.com/elastic/beats/compare/v7.0.0-beta1...master[Check the HEAD di - Fix a bug when converting NetFlow fields to snake_case. {pull}10950[10950] - Add on_failure handler for Zeek ingest pipelines. Fix one field name error for notice and add an additional test case. {issue}11004[11004] {pull}11105[11105] - Fix issue preventing docker container events to be stored if the container has a network interface without ip address. {issue}11225[11225] {pull}11247[11247] +- Add on_failure handler for Zeek ingest pipelines. Fix one field name error for notice and add an additional test + case. {issue}11004[11004] {pull}11105[11105] - Change URLPATH grok pattern to support brackets. {issue}11135[11135] {pull}11252[11252] - Add support for iis log with different address format. {issue}11255[11255] {pull}11256[11256] @@ -120,6 +122,15 @@ https://github.com/elastic/beats/compare/v7.0.0-beta1...master[Check the HEAD di - Add ISO8601 timestamp support in syslog metricset. {issue}8716[8716] {pull}10736[10736] - Add support for loading custom NetFlow and IPFIX field definitions to netflow input. {pull}10945[10945] {pull}11223[11223] +- Add more info to message logged when a duplicated symlink file is found {pull}10845[10845] +- Add option to configure docker input with paths {pull}10687[10687] +- Add Netflow module to enrich flow events with geoip data. {pull}10877[10877] +- Set `event.category: network_traffic` for Suricata. {pull}10882[10882] +- Add configuration knob for auto-discover hints to control whether log harvesting is enabled for the pod/container. {issue}10811[10811] {pull}10911[10911] +- Change Suricata module pipeline to handle `destination.domain` being set if a reverse DNS processor is used. {issue}10510[10510] +- Add the `network.community_id` flow identifier to field to the IPTables, Suricata, and Zeek modules. {pull}11005[11005] +- Add support for loading custom NetFlow and IPFIX field definitions to netflow input. {pull}10945[10945] +- Added categorization fields for SSH login events in the system/auth fileset. {pull}11334[11334] *Heartbeat* diff --git a/dev-tools/ecs-migration.yml b/dev-tools/ecs-migration.yml index f12e7db4f37..2b55db8d50d 100644 --- a/dev-tools/ecs-migration.yml +++ b/dev-tools/ecs-migration.yml @@ -486,11 +486,6 @@ alias: true beat: filebeat -- from: system.auth.ssh.event - to: event.action - alias: true - beat: filebeat - - from: system.auth.program to: process.name alias: true diff --git a/filebeat/docs/fields.asciidoc b/filebeat/docs/fields.asciidoc index a4102407afe..4236704c5ff 100644 --- a/filebeat/docs/fields.asciidoc +++ b/filebeat/docs/fields.asciidoc @@ -13084,9 +13084,10 @@ The client IP from SSH connections that are open and immediately dropped. *`system.auth.ssh.event`*:: + -- -type: alias +example: Accepted + +The SSH event as found in the logs (Accepted, Invalid, Failed, etc.) -alias to: event.action -- diff --git a/filebeat/module/system/auth/_meta/fields.yml b/filebeat/module/system/auth/_meta/fields.yml index ece161df3ab..de859681432 100644 --- a/filebeat/module/system/auth/_meta/fields.yml +++ b/filebeat/module/system/auth/_meta/fields.yml @@ -43,9 +43,10 @@ The client IP from SSH connections that are open and immediately dropped. - name: event - type: alias - path: event.action - migration: true + example: Accepted + description: > + The SSH event as found in the logs (Accepted, Invalid, Failed, etc.) + - name: ip type: alias path: source.ip diff --git a/filebeat/module/system/auth/ingest/pipeline.json b/filebeat/module/system/auth/ingest/pipeline.json index a5496839970..c1fc3db92d5 100644 --- a/filebeat/module/system/auth/ingest/pipeline.json +++ b/filebeat/module/system/auth/ingest/pipeline.json @@ -9,8 +9,8 @@ "GREEDYMULTILINE" : "(.|\n)*" }, "patterns": [ - "%{SYSLOGTIMESTAMP:system.auth.timestamp} %{SYSLOGHOST:host.hostname} %{DATA:process.name}(?:\\[%{POSINT:process.pid:long}\\])?: %{DATA:event.outcome} %{DATA:system.auth.ssh.method} for (invalid user )?%{DATA:user.name} from %{IPORHOST:source.ip} port %{NUMBER:source.port:long} ssh2(: %{GREEDYDATA:system.auth.ssh.signature})?", - "%{SYSLOGTIMESTAMP:system.auth.timestamp} %{SYSLOGHOST:host.hostname} %{DATA:process.name}(?:\\[%{POSINT:process.pid:long}\\])?: %{DATA:event.outcome} user %{DATA:user.name} from %{IPORHOST:source.ip}", + "%{SYSLOGTIMESTAMP:system.auth.timestamp} %{SYSLOGHOST:host.hostname} %{DATA:process.name}(?:\\[%{POSINT:process.pid:long}\\])?: %{DATA:system.auth.ssh.event} %{DATA:system.auth.ssh.method} for (invalid user )?%{DATA:user.name} from %{IPORHOST:source.ip} port %{NUMBER:source.port:long} ssh2(: %{GREEDYDATA:system.auth.ssh.signature})?", + "%{SYSLOGTIMESTAMP:system.auth.timestamp} %{SYSLOGHOST:host.hostname} %{DATA:process.name}(?:\\[%{POSINT:process.pid:long}\\])?: %{DATA:system.auth.ssh.event} user %{DATA:user.name} from %{IPORHOST:source.ip}", "%{SYSLOGTIMESTAMP:system.auth.timestamp} %{SYSLOGHOST:host.hostname} %{DATA:process.name}(?:\\[%{POSINT:process.pid:long}\\])?: Did not receive identification string from %{IPORHOST:system.auth.ssh.dropped_ip}", "%{SYSLOGTIMESTAMP:system.auth.timestamp} %{SYSLOGHOST:host.hostname} %{DATA:process.name}(?:\\[%{POSINT:process.pid:long}\\])?: \\s*%{DATA:user.name} :( %{DATA:system.auth.sudo.error} ;)? TTY=%{DATA:system.auth.sudo.tty} ; PWD=%{DATA:system.auth.sudo.pwd} ; USER=%{DATA:system.auth.sudo.user} ; COMMAND=%{GREEDYDATA:system.auth.sudo.command}", "%{SYSLOGTIMESTAMP:system.auth.timestamp} %{SYSLOGHOST:host.hostname} %{DATA:process.name}(?:\\[%{POSINT:process.pid:long}\\])?: new group: name=%{DATA:group.name}, GID=%{NUMBER:group.id}", @@ -61,6 +61,13 @@ "target_field": "source.geo", "ignore_failure": true } + }, + { + "script": { + "lang": "painless", + "ignore_failure": true, + "source": "if (ctx.system.auth.ssh.event == \"Accepted\") { if (!ctx.containsKey(\"event\")) { ctx.event = [:]; } ctx.event.type = \"authentication_success\"; ctx.event.category = \"authentication\"; ctx.event.action = \"ssh_login\"; ctx.event.outcome = \"success\"; } else if (ctx.system.auth.ssh.event == \"Invalid\" || ctx.system.auth.ssh.event == \"Failed\") { if (!ctx.containsKey(\"event\")) { ctx.event = [:]; } ctx.event.type = \"authentication_failure\"; ctx.event.category = \"authentication\"; ctx.event.action = \"ssh_login\"; ctx.event.outcome = \"failure\"; }" + } } ], "on_failure" : [{ diff --git a/filebeat/module/system/auth/test/test.log-expected.json b/filebeat/module/system/auth/test/test.log-expected.json index c780956e8a5..b33632f39fd 100644 --- a/filebeat/module/system/auth/test/test.log-expected.json +++ b/filebeat/module/system/auth/test/test.log-expected.json @@ -3,7 +3,11 @@ "ecs.version": "1.0.0", "event.dataset": "system.auth", "event.module": "system", - "event.outcome": "Accepted", + "system.auth.ssh.event": "Accepted", + "event.outcome": "success", + "event.category": "authentication", + "event.action": "ssh_login", + "event.type": "authentication_success", "fileset.name": "auth", "host.hostname": "localhost", "input.type": "log", @@ -21,7 +25,11 @@ "ecs.version": "1.0.0", "event.dataset": "system.auth", "event.module": "system", - "event.outcome": "Accepted", + "system.auth.ssh.event": "Accepted", + "event.outcome": "success", + "event.category": "authentication", + "event.action": "ssh_login", + "event.type": "authentication_success", "fileset.name": "auth", "host.hostname": "localhost", "input.type": "log", @@ -38,7 +46,11 @@ "ecs.version": "1.0.0", "event.dataset": "system.auth", "event.module": "system", - "event.outcome": "Invalid", + "system.auth.ssh.event": "Invalid", + "event.outcome": "failure", + "event.category": "authentication", + "event.action": "ssh_login", + "event.type": "authentication_failure", "fileset.name": "auth", "host.hostname": "localhost", "input.type": "log", @@ -53,7 +65,11 @@ "ecs.version": "1.0.0", "event.dataset": "system.auth", "event.module": "system", - "event.outcome": "Failed", + "system.auth.ssh.event": "Failed", + "event.outcome": "failure", + "event.category": "authentication", + "event.action": "ssh_login", + "event.type": "authentication_failure", "fileset.name": "auth", "host.hostname": "slave22", "input.type": "log", @@ -170,4 +186,4 @@ "user.id": "48", "user.name": "apache" } -] \ No newline at end of file +] diff --git a/filebeat/module/system/fields.go b/filebeat/module/system/fields.go index a34fbdc06d7..4cfa76bf8c7 100644 --- a/filebeat/module/system/fields.go +++ b/filebeat/module/system/fields.go @@ -32,5 +32,5 @@ func init() { // AssetSystem returns asset data. // This is the base64 encoded gzipped contents of module/system. func AssetSystem() string { - return "eJzsmM9u4zYQxu9+ioEve1o/gA9FgQWKFug/ILn05GXIkTRYiiOQo3XUpy8oyY5iUw6luO0lOhmWZn4fP3HIET/DN+z2ELogWG8AhMTiHrYP/R/bDYDBoD01Quz28MMGAOA3Nq1FKNhDo3wgV44JwHIJBVkMuw1AqNjLQbMrqNyD+BY3AAWhNWHf5/kMTtU4ocdLugb3UHpum/GfhIJ4/dRngsJzDVJhUkK8psQpVbVSnf9McW+wU/xfybXPfVb29LeKIVHMWUdKy1SPUI1BVN28unsSpiypcHGnUVLt4dOP58hPFw/UVHo1aB/tv8ZWHCT+WkiNYbuZ2Bxs47n0ql5IbTxrDGG3FkpmJfA6ModXYwiqXGptOiqH1wb0C2ExZImdSW4IVRJ7WU0wUwJTx6TiS6tvluFwPVYIDw8/9+WHTkgP9Tek28EX5eAJgR0CF7BtVAhH9mYL7GHbtE+W9DfstrsZVYFKp6T1lyZlCjuHR3hcKrQldAIDOK7Ac2DjuWnQHOjSxZPDiRtZkkYFv/w5rF/ROs3OoY5hAaRSAioKbtCBcgaortGQErTdSdUuPSle1ON3dDIjPDUjT3Oyj9upXsvVI7fq4AU9a9gtbuDWa9wlYvOgDfs1wx2xyeisMnxRUCLPjjxVjHPlOM2p2Qk5dHJILBNvj/FqnCXyLiPnbctfC2yd+O5AgQ+azd0kZmTNF2l5WJLuI+5mtnxRHktid8f3+lbCBS+VpLvnhLuZbrFj951qOUmX7cet4ewN+Y39YmxzQ4OaCtIgPKzPAbRHJWjgqes3ta+R+hU017Vy5nJHu73to/d82bkA4LOqGzv2NfD7H49Arh8b+mtns7a9nnNqsGI2rQIOXw+t4ZN2KBRZvBrCuVGXbh398fEvOFboE0QKgM+oW5nHNseVfZFuvY+bvSGPWth37xCR6DAn78kzX29hWRpF+RJleNHCcKxIV0mB4UiiK3LlnMLx2ZVWjaCTDfCdVC9hpteZ2qJM+uPiX625Ebyy7CpOLIhTbTM2xTgo2Br0/SlAlOLwOHxMzPXQFVq7BmawUK2VIcEN3Aw3uei/3ZTNfRfl9oLt1QdjNjURmccsVzH7+bkEmqyAPsv/UgIn8soaWDk/BtfeMUH+o5c1OWOzXG4uWQvOu76wE0UujH6+OnizPD0C/DjySmI/jrzeeeT1TwAAAP//TwGvPg==" + return "eJzsmM2O2zYQx+9+ioEvaYFED+BD0SJA0AD9AnYvPTlcciQRoTgCOVqv+vQF9bHRypRNad3msjoZkmd+fw05H+IH+IrtAXzrGasdAGs2eID9XXdjvwNQ6KXTNWuyB/hpBwDwO6nGIOTkoBbOa1sMDsBQAbk26LMdgC/J8VGSzXVxAHYN7gByjUb5Q+fnA1hR4YQeLm5rPEDhqKmHOxEF4frUeYLcUQVcYlRCuKbEKVU0XD7fjHEvsGP837Rtnjqv5PQ/IpgEMc86YlqmelhX6FlU9YunozBhtPCzJ7Xg8gDvfn62fDf7Q6ULJ3rtQ/jPsSV5Dr9WUoNZtmCbgq0dFU5UK6m1I4neZ1uhWm0Enlum8Cr0XhRrQxu3SuE1Ht1KWDBZE84o1/syip1nEyykwDRiXNI81BfTsL/uS4S7u1+79EPLWvb517vL4KOw8IBAFoFy2NfC+xM5tQdysK+bB6PlV2z32YIqrwsruHHzICUKezYP8FAqpNFoGXpwqMBLYOWorlEd9TyKY4QjD5IkDQo+/9XXrxA6SdaiDGYeuBQMIgiu0YKwCnRVodKC0bSjqiy+Kb6px0e0fCYAn0RVhybzi5RYM75itTsACA85NUGj7aIbii78MHp/D5/tozBavYdPQptwA1lmP14TvxjyWC6N2eSpcRKziO2l9J0UKHLnAUvGRq2TEvmbggJp8c1j6byU0FOfkixri5aPkUJz/R3P3rNAyhJ8Xg75S4GNZdcetaejJHUziQle00Ua6ovabcRd9JYuymGhyd5wXa85XLGomttbbriL7lZH7LZbLcXpuo7eKEpu6Vfq9TAo+xqlzrUEpr5we5AOBaOCh7Yr3F8C9QtIqiph1bwnXh4c0Dmazz6TXhPGHPjjz/vQIwIF3Xlkk9pOxxlHtOBNCo/990ejaNQOeddqlto6c7uNfn//N5xKdBGi9oBPKBtextanjb1WNs6FTqu0Q8nk2leIiMyok3VyROctLEkjC1cg9wvNBKdSyzIq0J80y1LbYknh8N+NoRpAYxjgUYtOwsK0NA2LUPHPk/805wbwxrQrKVIQp9oWwhTsICej0HXnCEGKxVP/ObI0hZdozBaYwlw0hnsHF3AL3GjRvz6ULX1Zpc6CzdknZzI1YpnGLDYxu/25BhrNgM7Ld0mBkbwxBzbujz5qr9gg/9NiTU7pDBW7OWvFidlHsiy09UM8XxzdGZoeIr4dmkWxb4dmrzw0+zcAAP//vfvB9g==" } From 4431b7b55a1872be89861b40f2388c0ab7216a47 Mon Sep 17 00:00:00 2001 From: Tudor Golubenco Date: Thu, 21 Mar 2019 11:04:57 +0100 Subject: [PATCH 2/2] cleanup changelog --- CHANGELOG.next.asciidoc | 10 ---------- 1 file changed, 10 deletions(-) diff --git a/CHANGELOG.next.asciidoc b/CHANGELOG.next.asciidoc index 202878e605d..5979f3f8572 100644 --- a/CHANGELOG.next.asciidoc +++ b/CHANGELOG.next.asciidoc @@ -77,8 +77,6 @@ https://github.com/elastic/beats/compare/v7.0.0-beta1...master[Check the HEAD di - Fix a bug when converting NetFlow fields to snake_case. {pull}10950[10950] - Add on_failure handler for Zeek ingest pipelines. Fix one field name error for notice and add an additional test case. {issue}11004[11004] {pull}11105[11105] - Fix issue preventing docker container events to be stored if the container has a network interface without ip address. {issue}11225[11225] {pull}11247[11247] -- Add on_failure handler for Zeek ingest pipelines. Fix one field name error for notice and add an additional test - case. {issue}11004[11004] {pull}11105[11105] - Change URLPATH grok pattern to support brackets. {issue}11135[11135] {pull}11252[11252] - Add support for iis log with different address format. {issue}11255[11255] {pull}11256[11256] @@ -122,14 +120,6 @@ https://github.com/elastic/beats/compare/v7.0.0-beta1...master[Check the HEAD di - Add ISO8601 timestamp support in syslog metricset. {issue}8716[8716] {pull}10736[10736] - Add support for loading custom NetFlow and IPFIX field definitions to netflow input. {pull}10945[10945] {pull}11223[11223] -- Add more info to message logged when a duplicated symlink file is found {pull}10845[10845] -- Add option to configure docker input with paths {pull}10687[10687] -- Add Netflow module to enrich flow events with geoip data. {pull}10877[10877] -- Set `event.category: network_traffic` for Suricata. {pull}10882[10882] -- Add configuration knob for auto-discover hints to control whether log harvesting is enabled for the pod/container. {issue}10811[10811] {pull}10911[10911] -- Change Suricata module pipeline to handle `destination.domain` being set if a reverse DNS processor is used. {issue}10510[10510] -- Add the `network.community_id` flow identifier to field to the IPTables, Suricata, and Zeek modules. {pull}11005[11005] -- Add support for loading custom NetFlow and IPFIX field definitions to netflow input. {pull}10945[10945] - Added categorization fields for SSH login events in the system/auth fileset. {pull}11334[11334] *Heartbeat*