[Filebeat] Add module for Kibana audit logs #22696
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,8 +1,16 @@ | ||
| - module: kibana | ||
| # Server logs | ||
| log: | ||
| enabled: true | ||
|
|
||
| # Set custom paths for the log files. If left empty, | ||
| # Filebeat will choose the paths depending on your OS. | ||
| #var.paths: | ||
|
|
||
| # Audit logs | ||
| audit: | ||
| enabled: true | ||
|
|
||
| # Set custom paths for the log files. If left empty, | ||
| # Filebeat will choose the paths depending on your OS. | ||
| #var.paths: |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,16 @@ | ||
| type: log | ||
| paths: | ||
| {{ range $i, $path := .paths }} | ||
| - {{$path}} | ||
| {{ end }} | ||
| exclude_files: [".gz$"] | ||
|
|
||
| processors: | ||
| - add_locale: ~ | ||
| - add_fields: | ||
| target: '' | ||
| fields: | ||
| ecs.version: 1.6.0 | ||
| - decode_json_fields: | ||
| fields: [message] | ||
| target: kibana._audit_temp | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,94 @@ | ||
| description: Pipeline for parsing Kibana audit logs in JSON format | ||
| processors: | ||
|
|
||
| - set: | ||
| field: "@timestamp" | ||
| value: "{{kibana._audit_temp.@timestamp}}" | ||
|
|
||
| - set: | ||
| field: message | ||
| value: "{{kibana._audit_temp.message}}" | ||
|
|
||
| - set: | ||
| if: ctx.kibana._audit_temp.event.action != null | ||
| field: event.action | ||
| value: "{{kibana._audit_temp.event.action}}" | ||
| - set: | ||
| if: ctx.kibana._audit_temp.event.category != null | ||
| field: event.category | ||
| value: "{{kibana._audit_temp.event.category}}" | ||
| - set: | ||
| if: ctx.kibana._audit_temp.event.outcome != null | ||
| field: event.outcome | ||
| value: "{{kibana._audit_temp.event.outcome}}" | ||
| - set: | ||
| if: ctx.kibana._audit_temp.event.type != null | ||
| field: event.type | ||
| value: "{{kibana._audit_temp.event.type}}" | ||
|
|
||
| - rename: | ||
| if: ctx.kibana._audit_temp.url != null | ||
| field: kibana._audit_temp.url | ||
| target_field: "url" | ||
|
|
||
| - set: | ||
| if: ctx.url?.query == null | ||
| field: url.original | ||
| value: '{{url.path}}' | ||
| ignore_empty_value: true | ||
| - set: | ||
| if: ctx.url?.path != null && ctx.url?.query != null | ||
| field: url.original | ||
| value: '{{url.path}}?{{url.query}}' | ||
|
|
||
| - rename: | ||
| if: ctx.kibana._audit_temp.http != null | ||
| field: kibana._audit_temp.http | ||
| target_field: http | ||
|
|
||
| - rename: | ||
| if: ctx.kibana._audit_temp.user != null | ||
| field: kibana._audit_temp.user | ||
| target_field: user | ||
|
|
||
| - rename: | ||
| if: ctx.kibana._audit_temp.trace != null | ||
| field: kibana._audit_temp.trace | ||
| target_field: trace | ||
|
|
||
| - rename: | ||
| if: ctx.kibana._audit_temp.process?.pid != null | ||
| target_field: process | ||
| field: kibana._audit_temp.process | ||
|
|
||
| - rename: | ||
| if: ctx.kibana._audit_temp.error != null | ||
| target_field: error | ||
| field: kibana._audit_temp.error | ||
|
|
||
| - rename: | ||
| if: ctx.kibana._audit_temp.kibana.space_id != null | ||
| target_field: kibana.space_id | ||
| field: kibana._audit_temp.kibana.space_id | ||
thomheymann marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
|
||
| - rename: | ||
| if: ctx.kibana._audit_temp.kibana.saved_object != null | ||
| target_field: kibana.saved_object | ||
| field: kibana._audit_temp.kibana.saved_object | ||
|
|
||
| - rename: | ||
| if: ctx.kibana._audit_temp.kibana.add_to_spaces != null | ||
| target_field: kibana.add_to_spaces | ||
| field: kibana._audit_temp.kibana.add_to_spaces | ||
|
|
||
| - rename: | ||
| if: ctx.kibana._audit_temp.kibana.delete_from_spaces != null | ||
| target_field: kibana.delete_from_spaces | ||
| field: kibana._audit_temp.kibana.delete_from_spaces | ||
|
There was a problem hiding this comment. I'm a bit worried about picking fields within There was a problem hiding this comment.
I'm torn on this too. On one hand, it would be great if this module "just worked" whenever a new field was added. On the other hand, unknown fields will be dynamically mapped, which might not be what we want. @P1llus are there recommendations on how filesets should handle dynamic fields? Should we accept them so they are mapped dynamically, or should we attempt to keep the mapping in sync with changes in Kibana going forward? There was a problem hiding this comment. The general way we approach it, is ensuring that all fields coming from filebeat modules would be appended under In terms of dynamic mapping, I would recommend to map as many fields as possible, however if you know that there is certain fields with complex or everchanging field names, you can apply the " Hope that makes sense @legrego @thomheymann There was a problem hiding this comment. I am not a fan of changing the JSON structure from what's logged since it is already in the format that we want and it would be super confusing for users otherwise. I'm starting to think that it then might be better to restrict the ECS schema in Kibana so that only known fields are allowed to be logged which we can map manually in the Filebeat pipeline. Is there a way of defining this pipeline using an npm module in the Kibana repository or can we publish a Go module from within Kibana? I'm a bit worried about these things being dislocated and going out of sync. (Developers adding fields without updating Filebeat pipeline) There was a problem hiding this comment.
++ I think this is the safest approach for now too, until we get to the point where we actually require dynamic fields that can't be known ahead of time.
I wonder if there's something we could do within Kibana to enforce this. If we had the audit logger run alongside the functional test suites, we could inspect it to ensure that it's not capturing anything unexpected. If it wouldn't be so expensive to do, we could do a runtime enforcement, but I don't think that's worth the overhead at this point. There was a problem hiding this comment. I've typed the |
||
|
|
||
thomheymann marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| - remove: | ||
| field: 'kibana._audit_temp' | ||
| on_failure: | ||
| - set: | ||
| field: error.message | ||
| value: '{{ _ingest.on_failure_message }}' | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,21 @@ | ||
| description: Pipeline for parsing Kibana audit logs | ||
| processors: | ||
| - set: | ||
| field: event.ingested | ||
| value: '{{_ingest.timestamp}}' | ||
| - rename: | ||
| field: '@timestamp' | ||
| target_field: event.created | ||
| - pipeline: | ||
| name: '{< IngestPipeline "pipeline-json" >}' | ||
| - set: | ||
legrego marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| field: event.kind | ||
| value: event | ||
| - append: | ||
| field: related.user | ||
| value: "{{user.name}}" | ||
| if: "ctx?.user?.name != null" | ||
| on_failure: | ||
| - set: | ||
| field: error.message | ||
| value: '{{ _ingest.on_failure_message }}' | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,12 @@ | ||
| module_version: 1.0 | ||
|
|
||
| var: | ||
| - name: paths | ||
| default: | ||
| - /var/log/kibana/*_audit.json | ||
|
|
||
| ingest_pipeline: | ||
| - ingest/pipeline.yml | ||
| - ingest/pipeline-json.yml | ||
|
|
||
| input: config/audit.yml |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,7 @@ | ||
| {"@timestamp":"2020-11-20T12:05:14.528-05:00","message":"User is updating config [id=8.0.0]","log":{"level":"INFO","logger":"plugins.security.audit.ecs"},"process":{"pid":86516},"event":{"action":"saved_object_update","category":"database","type":"change","outcome":"unknown"},"kibana":{"space_id":"marketing","saved_object":{"type":"config","id":"8.0.0"}},"user":{"name":"elastic","roles":["superuser"]},"trace":{"id":"e0bd67a1-a1b0-424d-9652-a350f88188eb"}} | ||
| {"@timestamp":"2020-11-20T12:05:14.849-05:00","message":"User is requesting [/foo/s/marketing/api/saved_objects/_find] endpoint","log":{"level":"INFO","logger":"plugins.security.audit.ecs"},"process":{"pid":86516},"event":{"action":"http_request","category":"web","outcome":"unknown"},"http":{"request":{"method":"get"}},"url":{"domain":"0.0.0.0","path":"/foo/s/marketing/api/saved_objects/_find","port":5603,"query":"default_search_operator=AND&has_reference=%5B%5D&page=1&per_page=1000&search_fields=title%5E3&search_fields=description&type=dashboard","scheme":"https:"},"user":{"name":"elastic","roles":["superuser"]},"kibana":{"space_id":"marketing"},"trace":{"id":"ae67a156-3847-4d89-9c97-86769df5bc2e"}} | ||
| {"@timestamp":"2020-11-20T12:05:15.841-05:00","message":"User is requesting [/foo/s/marketing/api/saved_objects/_bulk_get] endpoint","log":{"level":"INFO","logger":"plugins.security.audit.ecs"},"process":{"pid":86516},"event":{"action":"http_request","category":"web","outcome":"unknown"},"http":{"request":{"method":"post"}},"url":{"domain":"0.0.0.0","path":"/foo/s/marketing/api/saved_objects/_bulk_get","port":5603,"scheme":"https:"},"user":{"name":"elastic","roles":["superuser"]},"kibana":{"space_id":"marketing"},"trace":{"id":"cef382d1-7442-4f9a-8bee-0512c2b1da5a"}} | ||
| {"@timestamp":"2020-11-20T12:05:15.853-05:00","message":"User has accessed index-pattern [id=metrics-*]","log":{"level":"INFO","logger":"plugins.security.audit.ecs"},"process":{"pid":86516},"event":{"action":"saved_object_get","category":"database","type":"access","outcome":"success"},"kibana":{"space_id":"marketing","saved_object":{"type":"index-pattern","id":"metrics-*"}},"user":{"name":"elastic","roles":["superuser"]},"trace":{"id":"cef382d1-7442-4f9a-8bee-0512c2b1da5a"}} | ||
| {"@timestamp":"2020-11-20T12:05:24.103-05:00","message":"User is requesting [/foo/s/marketing/api/saved_objects/_find] endpoint","log":{"level":"INFO","logger":"plugins.security.audit.ecs"},"process":{"pid":86516},"event":{"action":"http_request","category":"web","outcome":"unknown"},"http":{"request":{"method":"get"}},"url":{"domain":"0.0.0.0","path":"/foo/s/marketing/api/saved_objects/_find","port":5603,"query":"fields=title&per_page=10&search=%22My%20Dashboard%22&search_fields=title&type=dashboard","scheme":"https:"},"user":{"name":"elastic","roles":["superuser"]},"kibana":{"space_id":"marketing"},"trace":{"id":"b10ee3ab-8102-4122-b4b5-5727e9b3d6a3"}} | ||
| {"@timestamp":"2020-11-20T12:05:24.143-05:00","message":"User is requesting [/foo/s/marketing/api/saved_objects/dashboard] endpoint","log":{"level":"INFO","logger":"plugins.security.audit.ecs"},"process":{"pid":86516},"event":{"action":"http_request","category":"web","outcome":"unknown"},"http":{"request":{"method":"post"}},"url":{"domain":"0.0.0.0","path":"/foo/s/marketing/api/saved_objects/dashboard","port":5603,"query":"overwrite=true","scheme":"https:"},"user":{"name":"elastic","roles":["superuser"]},"kibana":{"space_id":"marketing"},"trace":{"id":"4995c6bd-903c-42c2-af28-5cf17cc1cb6b"}} | ||
| {"@timestamp":"2020-11-20T12:05:24.150-05:00","message":"User is creating dashboard [id=undefined]","log":{"level":"INFO","logger":"plugins.security.audit.ecs"},"process":{"pid":86516},"event":{"action":"saved_object_create","category":"database","type":"creation","outcome":"unknown"},"kibana":{"space_id":"marketing","saved_object":{"type":"dashboard"}},"user":{"name":"elastic","roles":["superuser"]},"trace":{"id":"4995c6bd-903c-42c2-af28-5cf17cc1cb6b"}} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is this something we should log from Kibana instead?
If we want to update to a later version of ECS that should be controlled by Kibana, not the Filebeat module.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Good question; I think this would make sense for Kibana to log instead of Filebeat. I copied this over from the ES config, but it probably doesn't make sense for us to follow suit here.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Cool, I'll create a PR to log that in Kibana: elastic/kibana#85390
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I came across this when testing. It appears that Filebeat is expecting that it'll set an explicit version itself:
beats/filebeat/tests/system/test_modules.py
Line 104 in 4dd8061
I worked around this, but I don't know if that's "ok" or not 😕