Add Filebeat kubernetes module. {issue}10812[10812] #10912

alakahakai · 2019-02-22T19:28:01Z

Add Filebeat kubernetes module. #10812. Closed old PR #10844.

elasticmachine · 2019-02-22T19:28:53Z

Pinging @elastic/secops

jsoriano

Thanks @alakahakai for this. I think that supporting envoy and coredns is going to be a great addition.

But I think that they should be two different modules, and not linked to kubernetes. Even if both envoy and coredns are services that are usually deployed on kubernetes they can also be run with other orchestrators or even as standalone services.
Filebeat modules should focus on supporting specific services, no matter how they are deployed.

jsoriano · 2019-02-25T17:06:13Z

x-pack/filebeat/filebeat.reference.yml

+  coredns: 
+    enabled: true
+  envoy:
+    enabled: true


I think they should be two modules, one for coredns and another one for envoy.

I like the suggestion. Let me see whether doing these two modules separately will allow them to work in kubernetes deployments. Some additional work might still be needed.

jsoriano · 2019-02-25T17:07:17Z

x-pack/filebeat/module/kubernetes/_meta/fields.yml

+    description: >
+      Fields from coredns logs after normalization
+    fields:
+    - name: duration


Please add descriptions for fields in this file.

Check out how I handled event.duration across very different sources in #10274. I think that's a pattern we can likely reuse broadly. This will reduce compilations a lot across the board, if we can always use the same snippet.

x-pack/filebeat/module/kubernetes/_meta/fields.yml

jsoriano · 2019-02-25T17:10:22Z

x-pack/filebeat/module/kubernetes/_meta/fields.yml

+      type: integer
+
+    - name: response_flags
+      type: keyword


Please use ECS fields for http requests and responses https://github.com/elastic/ecs#-http-fields

I agree with Jaime's comment here and below. I've added a few more as well.

Is the intent of this module to preserve the original information under envoy.*, and only copy out to the ECS fields?

In most places we've simply moved all of the data to the ECS field directly, and not defined the field for the source naming. Of course you still need to define all of the fields that don't actually map to ECS.

jsoriano · 2019-02-25T17:12:05Z

x-pack/filebeat/module/kubernetes/_meta/fields.yml

+    - name: response_flags
+      type: keyword
+
+    - name: bytes_received


Use http.request.bytes for this?

jsoriano · 2019-02-25T17:12:57Z

x-pack/filebeat/module/kubernetes/_meta/fields.yml

+    - name: bytes_sent
+      type: long
+
+    - name: duration


event.duration? (https://github.com/elastic/ecs#-event-fields)

jsoriano · 2019-02-25T17:14:42Z

x-pack/filebeat/module/kubernetes/coredns/config/coredns.yml

+      field: "message"
+      target_prefix: "coredns"
+  - drop_fields: 
+      fields: [time]


It'd be better to place all processing in the ingest pipeline.

Just like for Zeek, I'm carrying the fight over here ;-)

I like the idea of splitting out the logs here. It lets people use Beat processors that are inherently agent-side. There's no PID in this specific log, but one may want to do reverse DNS on the IPs.

I would directly name the fields following ECS here.

"%{timestamp} [%{event.level}] %{client.ip}:%{client.port} ...

And make sure to copy client.* to source.*. We're trying to always have source.* as a baseline in ECS.

jsoriano · 2019-02-25T17:15:43Z

x-pack/filebeat/module/kubernetes/envoy/config/envoy.yml

+      target_prefix: "envoy"
+  - drop_fields: 
+      fields: [time]
+


It'd be better to do all processing in the ingest pipeline.

jsoriano · 2019-02-25T17:19:26Z

x-pack/filebeat/module/kubernetes/k8s-ingest.yaml

+  labels:
+    k8s-app: ingest
+---
+


There are already deploy files in deploy/kubernetes, could this be added there?

jsoriano · 2019-02-25T17:29:48Z

x-pack/filebeat/module/kubernetes/envoy/manifest.yml

+var:
+  - name: paths
+    default:
+      - /var/lib/docker/containers/*/*-json.log


This will take logs from all containers running in the system, we probably want to set here some safe default, or nothing at all so setting something is required.
Take into account that if this module is used with kubernetes or docker autodiscover this path will be easily set to the one of the specific envoy container.

Please see file ingest-k8s.yaml and pull 10911 for addressing this. Note that the file names are using some generated 64-byte id here hence hard to specify without auto-discovery,.

x-pack/filebeat/include/list.go

webmat

Unless specifically desired, I think we should as much as possible use the ECS fields directly, without defining the source fields that are mapped cleanly to ECS. Of course all of the fields that don't map should be defined, as you're doing.

Noted a few other minor things.

webmat · 2019-02-25T19:14:27Z

CHANGELOG.next.asciidoc

@@ -244,6 +244,7 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...master[Check the HEAD d
 - Add ISO8601 timestamp support in syslog metricset. {issue}8716[8716] {pull}10736[10736]
 - Add more info to message logged when a duplicated symlink file is found {pull}10845[10845]
 - Add Netflow module to enrich flow events with geoip data. {pull}10877[10877]
+- Add Filebeat kubernetes module. {issue}10812[10812] 


I would link the Changelog to this PR which will have all of the action, not the original issue.

webmat · 2019-02-25T19:17:56Z

x-pack/filebeat/module/kubernetes/_meta/fields.yml

+    - name: upstream_service_time
+      type: double
+
+    - name: forwarded_for


Nothing for xff in ECS yet

webmat · 2019-02-25T19:18:39Z

x-pack/filebeat/module/kubernetes/_meta/fields.yml

+    - name: request_id
+      type: keyword
+
+    - name: authority


If this is an RFC Authority (name or IP + port), this can go in .address

webmat · 2019-02-25T19:18:55Z

x-pack/filebeat/module/kubernetes/_meta/fields.yml

+    - name: forwarded_for
+      type: keyword
+
+    - name: http_user_agent


ECS user_agent.original

In general, all the related ECS fields are added via the beats or ingest pipeline processors. The original fields are kept for reference. I guess I should remove these original fields instead.

Yes. The team has been going back and forth on this.

For all plaintext logs, the cutover has been pretty straightforward and drastic. We parse right into the ECS fields as much as we can, and add custom fields for everything not covered.

For Suricata, we started with the assumption that some pre-existing content or community experience existed, based on the exact Suricata field names & so on, because of the existence of their EVE JSON log format. I'm not sure if that's the case or not.

But since 7.0, we've started "hollowing out" the suricata.eve.* namespace, and replacing all of the fields that map to ECS with field aliases anyway (examples).

So I think we should do the same here. Especially since this is a brand new module. Let's not create legacy we'll have to clean up in 8.0 ;-)

webmat · 2019-02-25T19:19:17Z

x-pack/filebeat/module/kubernetes/_meta/fields.yml

+    - name: upstream_port
+      type: integer
+
+    - name: message


ECS message

webmat · 2019-02-25T19:27:03Z

x-pack/filebeat/module/kubernetes/coredns/config/coredns.yml

+      field: "message"
+      target_prefix: "coredns"
+  - drop_fields: 
+      fields: [time]


Just like for Zeek, I'm carrying the fight over here ;-)

I like the idea of splitting out the logs here. It lets people use Beat processors that are inherently agent-side. There's no PID in this specific log, but one may want to do reverse DNS on the IPs.

I would directly name the fields following ECS here.

"%{timestamp} [%{event.level}] %{client.ip}:%{client.port} ...

And make sure to copy client.* to source.*. We're trying to always have source.* as a baseline in ECS.

webmat · 2019-02-25T19:31:27Z

x-pack/filebeat/module/kubernetes/_meta/fields.yml

+    description: >
+      Fields from coredns logs after normalization
+    fields:
+    - name: duration


Check out how I handled event.duration across very different sources in #10274. I think that's a pattern we can likely reuse broadly. This will reduce compilations a lot across the board, if we can always use the same snippet.

webmat · 2019-02-25T19:34:39Z

x-pack/filebeat/module/kubernetes/coredns/ingest/pipeline.json

+      "script": {
+        "lang": "painless",
+        "source": "ctx.event.created = ctx['@timestamp']; ctx['@timestamp'] = ctx['coredns']['timestamp']; ctx.coredns.remove('timestamp');",
+        "ignore_failure" : true


You should parse the date in ingest node, to ensure the document content is consistent. Even if the Elasticsearch date datatype is able to parse many things, we want the source document itself to be consistent.

It's a thing ECS will have to make more clear in the future, IMO. (See what happened with the Zeek Unix ts)

webmat · 2019-02-25T19:37:09Z

x-pack/filebeat/module/kubernetes/envoy/ingest/pipeline.json

+    {
+      "set": {
+        "field": "user_agent.original",
+        "value": "{{envoy.http_user_agent}}",


You don't need to copy it around to avoid clobbering anymore. Since 7.0, the UA parser re-outputs the .original.

So you can directly populate user_agent.original and have:

{ "user_agent": { "field": "user_agent.original", "ignore_missing": true } }

and the right thing will happen :-)

webmat · 2019-02-25T19:39:21Z

x-pack/filebeat/module/kubernetes/k8s-ingest.yaml

+        ]
+        env:
+        - name: ELASTICSEARCH_HOST
+          value: 192.168.99.1


Should this IP be here (same for KIBANA_HOST below)? Is this a Kubernetes default for the gateway / your host?

alakahakai · 2019-02-25T20:50:04Z

I will try to break this into two separate modules - one for coredns, the other for envoy; And move them to non x-pack. Should additional things required for these to work in Kubernetes, a separate kubernetes module would be added then.

alakahakai · 2019-02-28T23:17:08Z

Close this PR. I will open new issues and PRs for coredns and envoy.

Add Filebeat kubernetes module. {issue}10812[10812]

d66c43c

alakahakai requested review from a team as code owners February 22, 2019 19:28

alakahakai requested review from andrewkroh and tsg February 22, 2019 19:28

alakahakai added Filebeat Filebeat SecOps labels Feb 22, 2019

alakahakai added review enhancement labels Feb 22, 2019

Ray Qiu added 4 commits February 22, 2019 16:46

Fix conflicts on include/list.go

f109170

Resolve conflicts

382667f

Merge branch 'master' into issue-10812

850d1cc

Resolve conflicts

0e731eb

jsoriano requested changes Feb 25, 2019

View reviewed changes

webmat reviewed Feb 25, 2019

View reviewed changes

x-pack/filebeat/include/list.go Show resolved Hide resolved

webmat suggested changes Feb 25, 2019

View reviewed changes

alakahakai added in progress Pull request is currently in progress. and removed review labels Feb 28, 2019

alakahakai closed this Feb 28, 2019

alakahakai deleted the issue-10812 branch February 28, 2019 23:18

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add Filebeat kubernetes module. {issue}10812[10812] #10912

Add Filebeat kubernetes module. {issue}10812[10812] #10912

alakahakai commented Feb 22, 2019 •

edited by jsoriano

Loading

elasticmachine commented Feb 22, 2019

jsoriano left a comment

jsoriano Feb 25, 2019

alakahakai Feb 25, 2019

jsoriano Feb 25, 2019

webmat Feb 25, 2019

jsoriano Feb 25, 2019

webmat Feb 25, 2019

jsoriano Feb 25, 2019

jsoriano Feb 25, 2019

jsoriano Feb 25, 2019

webmat Feb 25, 2019

jsoriano Feb 25, 2019

jsoriano Feb 25, 2019

jsoriano Feb 25, 2019

alakahakai Feb 25, 2019

webmat left a comment

webmat Feb 25, 2019

webmat Feb 25, 2019

webmat Feb 25, 2019

webmat Feb 25, 2019

alakahakai Feb 25, 2019

webmat Feb 25, 2019

webmat Feb 25, 2019

webmat Feb 25, 2019

webmat Feb 25, 2019

webmat Feb 25, 2019

webmat Feb 25, 2019

webmat Feb 25, 2019

alakahakai commented Feb 25, 2019

alakahakai commented Feb 28, 2019

Add Filebeat kubernetes module. {issue}10812[10812] #10912

Add Filebeat kubernetes module. {issue}10812[10812] #10912

Conversation

alakahakai commented Feb 22, 2019 • edited by jsoriano Loading

elasticmachine commented Feb 22, 2019

jsoriano left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

webmat left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

alakahakai commented Feb 25, 2019

alakahakai commented Feb 28, 2019

alakahakai commented Feb 22, 2019 •

edited by jsoriano

Loading