Add fileset for parsing linux auditd logs (elastic#3750) (elastic#3923)…

… (elastic#3941) (elastic#3962) The PR adds an auditd module for parsing logs from the auditd daemon. There is a sample dashboard for viewing this data. Features - Parses audit event type, unix epoch time, audit event counter, and the arbitrary key/value pairs that follow. - Applies geoip lookup to the `auditd.log.addr` field which is present for some remote login events. - Decodes hex encoded ascii values that are sometimes used for the `auditd.log.exe` and `auditd.log.cmd` fields. - Remove key/value pairs where the value is `?`. Missing Features - Decoder for `auditd.log.saddr` field present in SOCKADDR audit events. A recipe is described [here](https://unix.stackexchange.com/questions/102926/how-to-interpret-the-saddr-field-of-an-audit-log) and could probably be ported to painless. Sample value: `type=SOCKADDR msg=audit(1481078693.491:873): saddr=01002F7661722F72756E2F6E7363642F736F636B657400008983B330BE7F0000000000000000000070830130BE7F000004000000000000001B00000000000000D128B7ED000000008B8BB330BE7F00003B00000000000000403E4BF7FD7F0000447F0130BE7F000080150230BE7F`
andrewkroh · Apr 10, 2017 · 529f983 · 529f983
1 parent f59240e
commit 529f983
Show file tree

Hide file tree

Showing 33 changed files with 3,390 additions and 17 deletions.
diff --git a/CHANGELOG.asciidoc b/CHANGELOG.asciidoc
@@ -153,6 +153,7 @@ https://github.com/elastic/beats/compare/v5.2.2...v5.3.0[View commits]
 - The `symlinks` and `harverster_limit` settings are now GA, instead of experimental. {pull}3525[3525]
 - close_timeout is also applied when the output is blocking. {pull}3511[3511]
 - Improve handling of different path variants on Windows. {pull}3781[3781]
+- Add auditd module for reading audit logs on Linux. {pull}3750[3750] {pull}3941[3941]
 
 
 *Metricbeat*

diff --git a/filebeat/docs/fields.asciidoc b/filebeat/docs/fields.asciidoc
@@ -13,6 +13,7 @@ This document describes the fields that are exported by Filebeat. They are
 grouped in the following categories:
 
 * <<exported-fields-apache2>>
+* <<exported-fields-auditd>>
 * <<exported-fields-beat>>
 * <<exported-fields-cloud>>
 * <<exported-fields-log>>
@@ -285,6 +286,154 @@ type: keyword
 The module producing the logged message.
 
 
+[[exported-fields-auditd]]
+== Auditd Fields
+
+Module for parsing auditd logs.
+
+
+
+[float]
+== auditd Fields
+
+Fields from the auditd logs.
+
+
+
+[float]
+== log Fields
+
+Fields from the Linux audit log. Not all fields are documented here because they are dynamic and vary by audit event type.
+
+
+
+[float]
+=== auditd.log.record_type
+
+The audit event type.
+
+
+[float]
+=== auditd.log.old_auid
+
+For login events this is the old audit ID used for the user prior to this login.
+
+
+[float]
+=== auditd.log.new_auid
+
+For login events this is the new audit ID. The audit ID can be used to trace future events to the user even if their identity changes (like becoming root).
+
+
+[float]
+=== auditd.log.old_ses
+
+For login events this is the old session ID used for the user prior to this login.
+
+
+[float]
+=== auditd.log.new_ses
+
+For login events this is the new session ID. It can be used to tie a user to future events by session ID.
+
+
+[float]
+=== auditd.log.sequence
+
+type: long
+
+The audit event sequence number.
+
+
+[float]
+=== auditd.log.acct
+
+The user account name associated with the event.
+
+
+[float]
+=== auditd.log.pid
+
+The ID of the process.
+
+
+[float]
+=== auditd.log.ppid
+
+The ID of the process.
+
+
+[float]
+=== auditd.log.items
+
+The number of items in an event.
+
+
+[float]
+=== auditd.log.item
+
+The item field indicates which item out of the total number of items. This number is zero-based; a value of 0 means it is the first item.
+
+
+[float]
+=== auditd.log.a0
+
+The first argument to the system call.
+
+
+[float]
+=== auditd.log.res
+
+The result of the system call (success or failure).
+
+
+[float]
+== geoip Fields
+
+Contains GeoIP information gathered based on the `auditd.log.addr` field. Only present if the GeoIP Elasticsearch plugin is available and used.
+
+
+
+[float]
+=== auditd.log.geoip.continent_name
+
+type: keyword
+
+The name of the continent.
+
+
+[float]
+=== auditd.log.geoip.city_name
+
+type: keyword
+
+The name of the city.
+
+
+[float]
+=== auditd.log.geoip.region_name
+
+type: keyword
+
+The name of the region.
+
+
+[float]
+=== auditd.log.geoip.country_iso_code
+
+type: keyword
+
+Country ISO code.
+
+
+[float]
+=== auditd.log.geoip.location
+
+type: geo_point
+
+The longitude and latitude.
+
+
 [[exported-fields-beat]]
 == Beat Fields
 

diff --git a/filebeat/docs/images/kibana-audit-auditd.png b/filebeat/docs/images/kibana-audit-auditd.png
diff --git a/filebeat/docs/modules/auditd.asciidoc b/filebeat/docs/modules/auditd.asciidoc
@@ -0,0 +1,41 @@
+////
+This file is generated! See scripts/docs_collector.py
+////
+
+[[filebeat-module-auditd]]
+== Auditd module
+
+This module collects and parses logs from the audit daemon (`auditd`).
+
+[float]
+=== Compatibility
+
+This module was tested with logs from `auditd` on OSes like CentOS 6 and
+CentOS 7.
+
+This module is not available for Windows.
+
+[float]
+=== Dashboard
+
+This module comes with a sample dashboard showing an overview of the audit log
+data. You can build more specific dashboards that are tailored to the audit
+rules that you use on your systems.
+
+image::./images/kibana-audit-auditd.png[]
+
+[float]
+=== Syslog fileset settings
+
+[float]
+==== var.paths
+
+An array of paths where to look for the log files. If left empty, Filebeat
+will choose the paths depending on your operating systems.
+
+
+=== Fields
+
+For a description of each field in the metricset, see the
+<<exported-fields-auditd,exported fields>> section.
+
diff --git a/filebeat/docs/modules/system.asciidoc b/filebeat/docs/modules/system.asciidoc
@@ -5,18 +5,21 @@ This file is generated! See scripts/docs_collector.py
 [[filebeat-module-system]]
 == System module
 
-This module collects and parses logs created by system logging server of common Unix/Linux based
-distributions.
+This module collects and parses logs created by system logging server of common
+Unix/Linux based distributions.
 
+[float]
 === Compatibility
 
-This module was tested with logs from OSes like Ubuntu 12.04, Centos 7, macOS Sierra, and others.
+This module was tested with logs from OSes like Ubuntu 12.04, Centos 7, and
+macOS Sierra.
 
 This module is not available for Windows.
 
+[float]
 === Dashboard
 
-This module comes with a sample dashboard.
+This module comes with a sample dashboard showing syslog data.
 
 image::./images/kibana-system.png[]
 

diff --git a/filebeat/docs/modules_list.asciidoc b/filebeat/docs/modules_list.asciidoc
@@ -4,6 +4,7 @@ This file is generated! See scripts/docs_collector.py
 
   * <<filebeat-modules-overview>>
   * <<filebeat-module-apache2>>
+  * <<filebeat-module-auditd>>
   * <<filebeat-module-mysql>>
   * <<filebeat-module-nginx>>
   * <<filebeat-module-system>>
@@ -14,6 +15,7 @@ This file is generated! See scripts/docs_collector.py
 
 include::modules-overview.asciidoc[]
 include::modules/apache2.asciidoc[]
+include::modules/auditd.asciidoc[]
 include::modules/mysql.asciidoc[]
 include::modules/nginx.asciidoc[]
 include::modules/system.asciidoc[]

diff --git a/filebeat/filebeat.full.yml b/filebeat/filebeat.full.yml
@@ -56,6 +56,19 @@ filebeat.modules:
     # can be added under this section.
     #prospector:
 
+#------------------------------- Auditd Module -------------------------------
+#- module: auditd
+  #log:
+    #enabled: true
+
+    # Set custom paths for the log files. If left empty,
+    # Filebeat will choose the paths depending on your OS.
+    #var.paths:
+
+    # Prospector configuration (advanced). Any prospector configuration option
+    # can be added under this section.
+    #prospector:
+
 #-------------------------------- MySQL Module -------------------------------
 #- module: mysql
   # Error logs

diff --git a/filebeat/filebeat.template-es2x.json b/filebeat/filebeat.template-es2x.json
@@ -169,6 +169,104 @@
             }
           }
         },
+        "auditd": {
+          "properties": {
+            "log": {
+              "properties": {
+                "a0": {
+                  "ignore_above": 1024,
+                  "index": "not_analyzed",
+                  "type": "string"
+                },
+                "acct": {
+                  "ignore_above": 1024,
+                  "index": "not_analyzed",
+                  "type": "string"
+                },
+                "geoip": {
+                  "properties": {
+                    "city_name": {
+                      "ignore_above": 1024,
+                      "index": "not_analyzed",
+                      "type": "string"
+                    },
+                    "continent_name": {
+                      "ignore_above": 1024,
+                      "index": "not_analyzed",
+                      "type": "string"
+                    },
+                    "country_iso_code": {
+                      "ignore_above": 1024,
+                      "index": "not_analyzed",
+                      "type": "string"
+                    },
+                    "location": {
+                      "type": "geo_point"
+                    },
+                    "region_name": {
+                      "ignore_above": 1024,
+                      "index": "not_analyzed",
+                      "type": "string"
+                    }
+                  }
+                },
+                "item": {
+                  "ignore_above": 1024,
+                  "index": "not_analyzed",
+                  "type": "string"
+                },
+                "items": {
+                  "ignore_above": 1024,
+                  "index": "not_analyzed",
+                  "type": "string"
+                },
+                "new_auid": {
+                  "ignore_above": 1024,
+                  "index": "not_analyzed",
+                  "type": "string"
+                },
+                "new_ses": {
+                  "ignore_above": 1024,
+                  "index": "not_analyzed",
+                  "type": "string"
+                },
+                "old_auid": {
+                  "ignore_above": 1024,
+                  "index": "not_analyzed",
+                  "type": "string"
+                },
+                "old_ses": {
+                  "ignore_above": 1024,
+                  "index": "not_analyzed",
+                  "type": "string"
+                },
+                "pid": {
+                  "ignore_above": 1024,
+                  "index": "not_analyzed",
+                  "type": "string"
+                },
+                "ppid": {
+                  "ignore_above": 1024,
+                  "index": "not_analyzed",
+                  "type": "string"
+                },
+                "record_type": {
+                  "ignore_above": 1024,
+                  "index": "not_analyzed",
+                  "type": "string"
+                },
+                "res": {
+                  "ignore_above": 1024,
+                  "index": "not_analyzed",
+                  "type": "string"
+                },
+                "sequence": {
+                  "type": "long"
+                }
+              }
+            }
+          }
+        },
         "beat": {
           "properties": {
             "hostname": {