Syslog inputs parses RFC3164 events via TCP or UDP (#6842)

The Syslog inputs will use the UDP and TCP source lib, allowing the same socket behavior and the
same options as the two existing inputs. The parser is a state machine build with ragel[1] and allow
to parse FC3164[2] events with some less than perfect variants, if the received event is a complete
RFC3164 we will extract all of them, for us the minimum valid message MUST have the `date` and the
`message` defined.

Anything else we will log and drop them.

Fields:
* priority
* timestamp
* program
* pid
* message
* facility: extracted from the priority
* severity: extracted from the priority
* severity_label: mapped from the official list.
* facility_label: mapped from the official list[2]

Sample Configuration:

```yaml
  #enabled: false

  #protocol.tcp:
    # The host and port to receive the new event
    #host: "localhost:9000"

    # Character used to split new message
    #line_delimiter: "\n"

    # Maximum size in bytes of the message received over TCP
    #max_message_size: 20MiB

    # The number of seconds of inactivity before a remote connection is closed.
    #timeout: 300s

  #protocol.udp:
    # The host and port to receive the new event
    #host: "localhost:9000"

    # Maximum size of the message received over UDP
    #max_message_size: 10240
```

Limitations:
* Doesn't support multiline events like darwin can do, we need to extract the multiline logic from the log input.
* Only support RFC3164, RFC5424 will require more work on the parser.

close #5862

[1]: http://www.colm.net/open-source/ragel/
[2]: https://tools.ietf.org/html/rfc3164

CHANGELOG.asciidoc

@@ -129,6 +129,7 @@ https://github.com/elastic/beats/compare/v6.0.0-beta2...master[Check the HEAD di
 - Add support of log_format combined to NGINX access logs. {pull}6858[6858]
 - Release config reloading feature as GA.
 - Add support human friendly size for the UDP input. {pull}6886[6886]
+- Add Syslog input to ingest RFC3164 Events via TCP and UDP {pull}6842[6842]
 
 *Heartbeat*
 

filebeat/_meta/common.reference.p2.yml

@@ -241,7 +241,7 @@ filebeat.inputs:
   # Maximum size of the message received over UDP
   #max_message_size: 10KiB
 
-#------------------------------ TCP prospector --------------------------------
+#------------------------------ TCP input --------------------------------
 # Experimental: Config options for the TCP input
 #- type: tcp
   #enabled: false
@@ -258,6 +258,35 @@ filebeat.inputs:
   # The number of seconds of inactivity before a remote connection is closed.
   #timeout: 300s
 
+#------------------------------ Syslog input --------------------------------
+# Experimental: Config options for the Syslog input
+# Accept RFC3164 formatted syslog event via UDP.
+#- type: syslog
+  #enabled: false
+  #protocol.udp:
+    # The host and port to receive the new event
+    #host: "localhost:9000"
+
+    # Maximum size of the message received over UDP
+    #max_message_size: 10KiB
+
+# Accept RFC3164 formatted syslog event via TCP.
+#- type: syslog
+  #enabled: false
+
+  #protocol.tcp:
+    # The host and port to receive the new event
+    #host: "localhost:9000"
+
+    # Character used to split new message
+    #line_delimiter: "\n"
+
+    # Maximum size in bytes of the message received over TCP
+    #max_message_size: 20MiB
+
+    # The number of seconds of inactivity before a remote connection is closed.
+    #timeout: 300s
+
 #========================== Filebeat autodiscover ==============================
 
 # Autodiscover allows you to detect changes in the system and spawn new modules

filebeat/_meta/fields.common.yml

@@ -54,3 +54,45 @@
     - name: fileset.name
       description: >
         The Filebeat fileset that generated this event.
+
+    - name: syslog.facility
+      type: long
+      required: false
+      description: >
+        The facility extracted from the priority.
+
+    - name: syslog.priority
+      type: long
+      required: false
+      description: >
+        The priority of the syslog event.
+
+    - name: syslog.severity_label
+      type: keyword
+      required: false
+      description: >
+        The human readable severity.
+
+    - name: syslog.facility_label
+      type: keyword
+      required: false
+      description: >
+        The human readable facility.
+
+    - name: process.program
+      type: keyword
+      required: false
+      description: >
+        The name of the program.
+
+    - name: process.pid
+      type: long
+      required: false
+      description: >
+        The pid of the process.
+
+    - name: event.severity
+      type: long
+      required: false
+      description: >
+        The severity of the event.

filebeat/docs/fields.asciidoc

@@ -1806,6 +1806,90 @@ The Filebeat module that generated this event.
 The Filebeat fileset that generated this event.
 
 
+--
+
+*`syslog.facility`*::
++
+--
+type: long
+
+required: False
+
+The facility extracted from the priority.
+
+
+--
+
+*`syslog.priority`*::
++
+--
+type: long
+
+required: False
+
+The priority of the syslog event.
+
+
+--
+
+*`syslog.severity_label`*::
++
+--
+type: keyword
+
+required: False
+
+The human readable severity.
+
+
+--
+
+*`syslog.facility_label`*::
++
+--
+type: keyword
+
+required: False
+
+The human readable facility.
+
+
+--
+
+*`process.program`*::
++
+--
+type: keyword
+
+required: False
+
+The name of the program.
+
+
+--
+
+*`process.pid`*::
++
+--
+type: long
+
+required: False
+
+The pid of the process.
+
+
+--
+
+*`event.severity`*::
++
+--
+type: long
+
+required: False
+
+The severity of the event.
+
+
 --
 
 [[exported-fields-logstash]]

filebeat/docs/filebeat-options.asciidoc

@@ -48,6 +48,7 @@ You can configure {beatname_uc} to use the following inputs:
 * <<{beatname_lc}-input-udp>>
 * <<{beatname_lc}-input-docker>>
 * <<{beatname_lc}-input-tcp>>
+* <<{beatname_lc}-input-syslog>>
 
 
 
@@ -62,3 +63,5 @@ include::inputs/input-udp.asciidoc[]
 include::inputs/input-docker.asciidoc[]
 
 include::inputs/input-tcp.asciidoc[]
+
+include::inputs/input-syslog.asciidoc[]

filebeat/docs/inputs/input-common-tcp-options.asciidoc

@@ -0,0 +1,29 @@
+//////////////////////////////////////////////////////////////////////////
+//// This content is shared by Filebeat inputs that use the TCP inputsource
+//// If you add IDs to sections, make sure you use attributes to create
+//// unique IDs for each input that includes this file. Use the format:
+//// [id="{beatname_lc}-input-{type}-option-name"]
+//////////////////////////////////////////////////////////////////////////
+[float]
+[id="{beatname_lc}-input-{type}-tcp-max-message-size"]
+==== `max_message_size`
+
+The maximum size of the message received over TCP. The default is `20MiB`.
+
+[float]
+[id="{beatname_lc}-input-{type}-tcp-host"]
+==== `host`
+
+The host and TCP port to listen on for event streams.
+
+[float]
+[id="{beatname_lc}-input-{type}-tcp-line-delimiter"]
+==== `line_delimiter`
+
+Specify the characters used to split the incoming events. The default is '\n'.
+
+[float]
+[id="{beatname_lc}-input-{type}-tcp-timeout"]
+==== `timeout`
+
+The number of seconds of inactivity before a remote connection is closed. The default is `300s`.

filebeat/docs/inputs/input-common-udp-options.asciidoc

@@ -0,0 +1,17 @@
+//////////////////////////////////////////////////////////////////////////
+//// This content is shared by Filebeat inputs that use the UDP inputsource
+//// If you add IDs to sections, make sure you use attributes to create
+//// unique IDs for each input that includes this file. Use the format:
+//// [id="{beatname_lc}-input-{type}-option-name"]
+//////////////////////////////////////////////////////////////////////////
+[float]
+[id="{beatname_lc}-input-{type}-udp-max-message-size"]
+==== `max_message_size`
+
+The maximum size of the message received over UDP. The default is `10KiB`.
+
+[float]
+[id="{beatname_lc}-input-{type}-udp-host"]
+==== `host`
+
+The host and UDP port to listen on for event streams.

filebeat/docs/inputs/input-syslog.asciidoc

@@ -0,0 +1,47 @@
+:type: syslog
+
+[id="{beatname_lc}-input-{type}"]
+=== Syslog input
+
+++++
+<titleabbrev>Syslog</titleabbrev>
+++++
+
+Use the `syslog` input to read events over TCP or UDP, this input will parse BSD (rfc3164)
+event and some variant.
+
+Example configurations:
+
+["source","yaml",subs="attributes"]
+----
+{beatname_lc}.inputs:
+- type: syslog
+  protocol.udp:
+    host: "localhost:9000"
+----
+
+["source","yaml",subs="attributes"]
+----
+{beatname_lc}.inputs:
+- type: syslog
+  protocol.tcp:
+    host: "localhost:9000"
+----
+
+==== Configuration options
+
+The `syslog` input supports protocol specific configuration options plus the
+<<{beatname_lc}-input-{type}-common-options>> described later.
+
+Protocol `udp`:
+
+include::../inputs/input-common-udp-options.asciidoc[]
+
+Protocol `tcp`:
+
+include::../inputs/input-common-tcp-options.asciidoc[]
+
+[id="{beatname_lc}-input-{type}-common-options"]
+include::../inputs/input-common-options.asciidoc[]
+
+:type!:

filebeat/docs/inputs/input-tcp.asciidoc

@@ -15,7 +15,7 @@ Example configuration:
 ----
 {beatname_lc}.inputs:
 - type: tcp
-  max_message_size: 10240
+  max_message_size: 10MiB
   host: "localhost:9000"
 ----
 
@@ -25,29 +25,7 @@ Example configuration:
 The `tcp` input supports the following configuration options plus the
 <<{beatname_lc}-input-{type}-common-options>> described later.
 
-[float]
-[id="{beatname_lc}-input-{type}-max-message-size"]
-==== `max_message_size`
-
-The maximum size of the message received over TCP. The default is `20MiB`.
-
-[float]
-[id="{beatname_lc}-input-{type}-host"]
-==== `host`
-
-The host and TCP port to listen on for event streams.
-
-[float]
-[id="{beatname_lc}-input-{type}-line-delimiter"]
-==== `line_delimiter`
-
-Specify the characters used to split the incoming events. The default is '\n'.
-
-[float]
-[id="{beatname_lc}-input-{type}-timeout"]
-==== `timeout`
-
-The number of seconds of inactivity before a remote connection is closed. The default is `300s`.
+include::../inputs/input-common-tcp-options.asciidoc[]
 
 [id="{beatname_lc}-input-{type}-common-options"]
 include::../inputs/input-common-options.asciidoc[]

filebeat/docs/inputs/input-udp.asciidoc

@@ -25,17 +25,7 @@ Example configuration:
 The `udp` input supports the following configuration options plus the
 <<{beatname_lc}-input-{type}-common-options>> described later.
 
-[float]
-[id="{beatname_lc}-input-{type}-max-message-size"]
-==== `max_message_size`
-
-The maximum size of the message received over UDP. The default is `10KiB`.
-
-[float]
-[id="{beatname_lc}-input-{type}-host"]
-==== `host`
-
-The host and UDP port to listen on for event streams.
+include::../inputs/input-common-udp-options.asciidoc[]
 
 [id="{beatname_lc}-input-{type}-common-options"]
 include::../inputs/input-common-options.asciidoc[]