Cherry-pick #10006 to 6.x: Populate more ECS fields in the Suricata m…

…odule (#10537)

Cherry-pick of PR #10006 to 6.x branch. Original message: 

A few more ECS fields are populated by the ingest pipeline that enriches Suricata's eve.json events.

Additions:

- http.request.referrer (from suricata.eve.http.http_refer)

- event.action (from suricata.eve.alert.category)
  describes the action that caused the event.
  Examples: "Attempted Denial of Service", "Successful Administrator Privilege Gain"

- event.outcome (from suricata.eve.alert.action)
  Possible values: "allowed", "blocked"

- event.severity (from suricata.eve.alert.severity)
  Possible values: 1, 2 or 3.

- network.transport (from suricata.eve.proto)
  Examples: "tcp", "udp", "ipv6-icmp"

(cherry picked from commit 184149f4a18b4162b0d6c89adba3bb924a2db0b8)

filebeat/_meta/fields.common.yml

@@ -150,11 +150,34 @@
         @timestamp is the processed timestamp from the log line. If both are identical
         only @timestamp should be used.
 
+    - name: event.duration
+      type: long
+      format: duration
+      input_format: nanoseconds
+      description: >
+        Duration of the event in nanoseconds.
+
+        If event.start and event.end are known this value should be the
+        difference between the end and start time.
+
+    - name: event.end
+      type: date
+      description: >
+        event.end contains the date when the event ended or when the activity
+        was last observed.
+
     - name: event.type
       type: keyword
       description: >
         A type given to this kind of event which can be used for grouping.
 
+    - name: event.start
+      level: extended
+      type: date
+      description: >
+        event.start contains the date when the event started or when the
+        activity was first observed.
+
     - name: http.response.status_code
       type: long
       description: >
@@ -166,6 +189,12 @@
       description: >
         Elapsed time between request and response in milli seconds.
 
+    - name: http.response.body.bytes
+      type: long
+      format: bytes
+      description: >
+        Size in bytes of the response body.
+
     - name: http.response.content_length
       type: long
       description: >
@@ -179,6 +208,12 @@
     - name: source_ecs
       type: group
       fields:
+      - name: bytes
+        type: long
+        format: bytes
+        description: >
+          Bytes sent from the source to the destination.
+
       - name: ip
         type: ip
         description: >
@@ -192,6 +227,12 @@
         description: >
           MAC address of the source.
 
+      - name: packets
+        level: core
+        type: long
+        description: >
+          Packets sent from the client to the server.
+
       - name: port
         type: long
         description: >
@@ -235,6 +276,19 @@
     - name: destination
       type: group
       fields:
+        - name: bytes
+          level: core
+          type: long
+          format: bytes
+          description: >
+            Bytes sent from the destination to the source.
+
+        - name: domain
+          level: core
+          type: keyword
+          description: >
+            Destination domain.
+
         - name: ip
           type: ip
           description: >
@@ -248,6 +302,12 @@
           description: >
             MAC address of the destination.
 
+        - name: packets
+          level: core
+          type: long
+          description: >
+            Packets sent from the destination to the source.
+
         - name: port
           type: long
           description: >
@@ -376,13 +436,29 @@
 
       type: group
       fields:
+
+      - name: domain
+        level: extended
+        type: keyword
+        description: >
+          Domain of the request, such as "www.elastic.co".
+
+          In some cases a URL may refer to an IP and/or port directly, without a
+          domain name. In this case, the IP address would go to the `domain` field.
+
       - name: hostname
         type: keyword
         description: >
           Hostname of the request, such as "elastic.co".
           In some cases a URL may refer to an IP and/or port directly, without a
           domain name. In this case, the IP address would go to the `hostname` field.
 
+      - name: path
+        level: extended
+        type: keyword
+        description: >
+          Path of the request, such as "/search".
+
     - name: file
       description: >
         File fields provide details about each file.
@@ -406,6 +482,29 @@
         details about the network activity associated with an event.
       type: group
       fields:
+      - name: bytes
+        level: core
+        type: long
+        format: bytes
+        description: >
+          Total bytes transferred in both directions.
+          If `source.bytes` and `destination.bytes` are known, `network.bytes` is their sum.
+
+      - name: packets
+        level: core
+        type: long
+        description: >
+          Total packets transferred in both directions.
+          If `source.packets` and `destination.packets` are known, `network.packets` is their sum.
+
+      - name: protocol
+        level: core
+        type: keyword
+        description: >
+          L7 Network protocol name. ex. http, lumberjack, transport protocol.
+
+          The field value must be normalized to lowercase for querying. See
+          "Lowercase Capitalization" in the "Implementing ECS" section.
 
       - name: transport
         level: core

filebeat/docs/fields.asciidoc

@@ -3431,6 +3431,29 @@ type: date
 event.created contains the date on which the event was created. In case of log events this is when the log line was read by Filebeat. In comparison @timestamp is the processed timestamp from the log line. If both are identical only @timestamp should be used.
 
 
+--
+
+*`event.duration`*::
++
+--
+type: long
+
+format: duration
+
+Duration of the event in nanoseconds.
+If event.start and event.end are known this value should be the difference between the end and start time.
+
+
+--
+
+*`event.end`*::
++
+--
+type: date
+
+event.end contains the date when the event ended or when the activity was last observed.
+
+
 --
 
 *`event.type`*::
@@ -3441,6 +3464,16 @@ type: keyword
 A type given to this kind of event which can be used for grouping.
 
 
+--
+
+*`event.start`*::
++
+--
+type: date
+
+event.start contains the date when the event started or when the activity was first observed.
+
+
 --
 
 *`http.response.status_code`*::
@@ -3463,6 +3496,18 @@ type: long
 Elapsed time between request and response in milli seconds.
 
 
+--
+
+*`http.response.body.bytes`*::
++
+--
+type: long
+
+format: bytes
+
+Size in bytes of the response body.
+
+
 --
 
 *`http.response.content_length`*::
@@ -3486,6 +3531,18 @@ Request method.
 --
 
 
+*`source_ecs.bytes`*::
++
+--
+type: long
+
+format: bytes
+
+Bytes sent from the source to the destination.
+
+
+--
+
 *`source_ecs.ip`*::
 +
 --
@@ -3505,6 +3562,16 @@ type: keyword
 MAC address of the source.
 
 
+--
+
+*`source_ecs.packets`*::
++
+--
+type: long
+
+Packets sent from the client to the server.
+
+
 --
 
 *`source_ecs.port`*::
@@ -3584,6 +3651,28 @@ Region ISO code.
 --
 
 
+*`destination.bytes`*::
++
+--
+type: long
+
+format: bytes
+
+Bytes sent from the destination to the source.
+
+
+--
+
+*`destination.domain`*::
++
+--
+type: keyword
+
+Destination domain.
+
+
+--
+
 *`destination.ip`*::
 +
 --
@@ -3603,6 +3692,16 @@ type: keyword
 MAC address of the destination.
 
 
+--
+
+*`destination.packets`*::
++
+--
+type: long
+
+Packets sent from the destination to the source.
+
+
 --
 
 *`destination.port`*::
@@ -3817,6 +3916,17 @@ URL fields provide a complete URL, with scheme, host, and path. The URL object c
 
 
 
+*`url.domain`*::
++
+--
+type: keyword
+
+Domain of the request, such as "www.elastic.co".
+In some cases a URL may refer to an IP and/or port directly, without a domain name. In this case, the IP address would go to the `domain` field.
+
+
+--
+
 *`url.hostname`*::
 +
 --
@@ -3825,6 +3935,16 @@ type: keyword
 Hostname of the request, such as "elastic.co". In some cases a URL may refer to an IP and/or port directly, without a domain name. In this case, the IP address would go to the `hostname` field.
 
 
+--
+
+*`url.path`*::
++
+--
+type: keyword
+
+Path of the request, such as "/search".
+
+
 --
 
 [float]
@@ -3859,6 +3979,39 @@ The network is defined as the communication path over which a host or network ev
 
 
 
+*`network.bytes`*::
++
+--
+type: long
+
+format: bytes
+
+Total bytes transferred in both directions. If `source.bytes` and `destination.bytes` are known, `network.bytes` is their sum.
+
+
+--
+
+*`network.packets`*::
++
+--
+type: long
+
+Total packets transferred in both directions. If `source.packets` and `destination.packets` are known, `network.packets` is their sum.
+
+
+--
+
+*`network.protocol`*::
++
+--
+type: keyword
+
+L7 Network protocol name. ex. http, lumberjack, transport protocol.
+The field value must be normalized to lowercase for querying. See "Lowercase Capitalization" in the "Implementing ECS" section.
+
+
+--
+
 *`network.transport`*::
 +
 --