Ingest ES structured audit logs #10352

ycombinator · 2019-01-27T14:09:25Z

This is a "forward port" of #8852.

In #8852, we taught Filebeat to ingest either structured or unstructured ES audit logs but the resulting fields conformed to the 6.x mapping structure.

In this PR we also teach Filebeat to ingest either structured or unstructured ES audit logs but the resulting fields conform to the 7.0 (ECS-based) mapping structure.

elasticmachine · 2019-01-27T14:09:26Z

Pinging @elastic/stack-monitoring

ycombinator · 2019-01-27T16:38:10Z

While working on this PR I realized that we don't have sample lines for the structured elasticsearch audit log containing a request body (which is supposed to be parsed into the http.request.body.content field). I'm working with @albertzaharovits to get such a sample and will incorporate it into follow up PRs (for master and 6.x).

ycombinator · 2019-01-27T23:08:20Z

jenkins, test this

ruflin

Can you also update the ecs-migration.yml file?

filebeat/module/elasticsearch/audit/_meta/fields.yml

ycombinator · 2019-01-28T12:39:56Z

Can you also update the ecs-migration.yml file?

Updated now. Do I need to run any make/mage commands as well?

This reverts commit ab7cf63.

ycombinator · 2019-01-28T16:41:10Z

@ruflin I'm not sure what you meant by #10352 (review):

Can you also update the ecs-migration.yml file?

Could you please clarify? Otherwise this PR is ready for review, IMHO.

ycombinator · 2019-01-28T16:42:06Z

jenkins, test this

webmat

Noticed very minor things. Looking pretty good!

webmat · 2019-01-28T21:08:48Z

filebeat/module/elasticsearch/audit/ingest/pipeline-json.json

+        },
+        {
+            "dot_expander": {
+                "field": "origin.address",


I would instead rename this field to source.address (and keep it around).

webmat · 2019-01-28T21:17:42Z

filebeat/module/elasticsearch/audit/ingest/pipeline-json.json

+        {
+            "rename": {
+                "field": "elasticsearch.audit.user.name",
+                "target_field": "user.name"


Can't we dot_expand in place? If the original key is "user.name", I would think that the output to "user": { "name": "..." } doesn't conflict.

It would simplify the code in a few places where you have the same pattern happening.

I'm not sure I follow what you mean by doing "dot_expand in place"? The dot_expander processor right above this one is necessary to go from:

{ "elasticsearch.audit.user.name": "foo" }

to:

{ "elasticsearch.audit.user": { "name": "foo" } }

That then allows us to call the rename processor as we are doing over here.

I haven't used dot_expander yet, so perhaps I'm misunderstanding it. But I was under the impression that the following did the equivalent of the 2 processors above:

{ "dot_expander": { "field": "user.name" } }

And in cases where the output isn't the object equivalent of the dotted notation, you would use path this way, to get the equivalent of the two node.name processors above:

{ "dot_expander": { "field": "node.name", "path": "elasticsearch.node" } }

If that's not the case, you can ignore this ;-)

webmat · 2019-01-28T21:19:09Z

filebeat/module/elasticsearch/audit/ingest/pipeline.json

            }
        },
        {
            "date": {
-                "field": "elasticsearch.audit.timestamp",
+                "field": "elasticsearch.audit.@timestamp",


Nit: prior to grabbing the real timestamp from the log, could you populate event.created with Beat's @timestamp?

This is already being done in the very first processor in this pipeline. It's collapsed in the diff since nothing changed there:

https://github.com/elastic/beats/pull/10352/files#diff-a007340015953dcf31f9d48f74358ac3R4

Missed that 👍

webmat · 2019-01-28T21:28:45Z

Also, the dev-tools/ecs-migration.yml file is where we document the field migrations as you're doing here.

It's not the format as the various fields.yml fields Beats uses to generate templates. Rather it's optimized for us to automate some of this migration (e.g. rename fields in dashboards).

But in the ecs-migration.yml file, you should document all of the fields for which you have migration: true in the fields.yml file. This file can indicate:

whether the field could get a forward-compatible alias in 6.7, provided the mapping is 1:1, with alias6: true
whether the field in 7.x will alias to the old field. Most of the time this is true. Use alias: true.
in case of duration scale differences (ms to ns), you can use scale:
if your field is only present in one beat, you identify it with beat: filebeat, as should be the case here.

ycombinator · 2019-01-29T00:07:33Z

@webmat Thanks for the detailed explanation of how to use the ecs-migration.yml file — really appreciate it written down so I can go back and look at it in the future as well!

I've addressed all your comments in the review. Only one of them resulted in a code change, however. So you might want to look at my replies to your other two comments.

Thanks!

Follow up to #10352 per #10352 (comment): > While working on this PR I realized that we don't have sample lines for the **structured** elasticsearch audit log containing a request body (which is supposed to be parsed into the `http.request.body.content` field). I'm working with `@albertzaharovits` to get such a sample and will incorporate it into follow up PRs (for `master` and `6.x`). Accordingly, this PR adds sample lines to the structured and unstructured log file test fixtures for the `elasticsearch/audit` fileset and teaches the fileset to parse any new fields encountered in these sample lines.

webmat

LGTM

Muy understanding of dot_expander may be flawed, haven't used it before. So feel free to ignore, if what I'm saying in my response below doesn't make sense ;-)

webmat · 2019-01-29T14:28:28Z

filebeat/module/elasticsearch/audit/ingest/pipeline-json.json

+        {
+            "rename": {
+                "field": "elasticsearch.audit.user.name",
+                "target_field": "user.name"


I haven't used dot_expander yet, so perhaps I'm misunderstanding it. But I was under the impression that the following did the equivalent of the 2 processors above:

{ "dot_expander": { "field": "user.name" } }

And in cases where the output isn't the object equivalent of the dotted notation, you would use path this way, to get the equivalent of the two node.name processors above:

{ "dot_expander": { "field": "node.name", "path": "elasticsearch.node" } }

If that's not the case, you can ignore this ;-)

webmat · 2019-01-29T14:29:00Z

filebeat/module/elasticsearch/audit/ingest/pipeline.json

            }
        },
        {
            "date": {
-                "field": "elasticsearch.audit.timestamp",
+                "field": "elasticsearch.audit.@timestamp",


Missed that 👍

Follow up to #10352 per #10352 (comment): > While working on this PR I realized that we don't have sample lines for the **structured** elasticsearch audit log containing a request body (which is supposed to be parsed into the `http.request.body.content` field). I'm working with `@albertzaharovits` to get such a sample and will incorporate it into follow up PRs (for `master` and `6.x`). Accordingly, this PR adds sample lines to the structured and unstructured log file test fixtures for the `elasticsearch/audit` fileset and teaches the fileset to parse any new fields encountered in these sample lines.

This PR teaches the `elasticsearch/slowlog` fileset to ingest structured Elasticsearch search and indexing slow logs. This PR takes the same approach as #10352, in that it creates an entrypoint pipeline, `pipeline.json`, that delegates further processing of a log entry depending on what it sees as the first character of the entry: - If the first character is `{`, it assumes the log line is structured as JSON and delegates further processing to `pipeline-json.json`. - Else, it assumes the log line is plaintext and delegates further processing to `pipeline-plaintext.json`.

ycombinator added in progress Pull request is currently in progress. module Filebeat Filebeat v7.0.0 Feature:Stack Monitoring labels Jan 27, 2019

ycombinator requested review from a team as code owners January 27, 2019 14:09

Ingest ES structured audit logs

e9e01be

ycombinator force-pushed the fb-es-structured-audit-log branch from 2c0ff8b to e9e01be Compare January 27, 2019 15:49

ycombinator added 3 commits January 27, 2019 07:51

Removing unnecessary processor

08cda85

More updates

01eed12

Adding CHANGELOG entries

b19e4ee

ycombinator changed the title ~~[WIP] Ingest ES structured audit logs~~ Ingest ES structured audit logs Jan 27, 2019

ycombinator added review and removed in progress Pull request is currently in progress. labels Jan 27, 2019

ycombinator requested a review from ruflin January 27, 2019 16:34

ycombinator mentioned this pull request Jan 28, 2019

Parse more fields from elasticsearch audit log #10356

Merged

ruflin reviewed Jan 28, 2019

View reviewed changes

filebeat/module/elasticsearch/audit/_meta/fields.yml Show resolved Hide resolved

ycombinator added 2 commits January 28, 2019 04:35

Updating ecs-migration.yml

ab7cf63

Forgot to remove old HTTP request body field

84ae341

ycombinator added 2 commits January 28, 2019 07:44

Revert "Updating ecs-migration.yml"

cd1018b

This reverts commit ab7cf63.

Undoing migration changes to fields.yml

0b11092

webmat reviewed Jan 28, 2019

View reviewed changes

Renaming to and preserving source.address

1692e94

ycombinator mentioned this pull request Jan 29, 2019

Parse more fields from elasticsearch audit log #10385

Merged

webmat approved these changes Jan 29, 2019

View reviewed changes

ycombinator merged commit 5e460ed into elastic:master Jan 29, 2019

ycombinator mentioned this pull request Jan 30, 2019

Ingest structured ES server logs #10428

Merged

This was referenced Jan 31, 2019

Ingest structured ES deprecation logs #10445

Merged

Ingest structured ES slow logs #10447

Merged

ycombinator deleted the fb-es-structured-audit-log branch December 25, 2019 11:15

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Ingest ES structured audit logs #10352

Ingest ES structured audit logs #10352

ycombinator commented Jan 27, 2019 •

edited

Loading

elasticmachine commented Jan 27, 2019

ycombinator commented Jan 27, 2019 •

edited

Loading

ycombinator commented Jan 27, 2019

ruflin left a comment

ycombinator commented Jan 28, 2019

ycombinator commented Jan 28, 2019

ycombinator commented Jan 28, 2019

webmat left a comment

webmat Jan 28, 2019

webmat Jan 28, 2019

ycombinator Jan 29, 2019

webmat Jan 29, 2019

webmat Jan 28, 2019

ycombinator Jan 28, 2019

webmat Jan 29, 2019

webmat commented Jan 28, 2019

ycombinator commented Jan 29, 2019

webmat left a comment

webmat Jan 29, 2019

webmat Jan 29, 2019

Ingest ES structured audit logs #10352

Ingest ES structured audit logs #10352

Conversation

ycombinator commented Jan 27, 2019 • edited Loading

elasticmachine commented Jan 27, 2019

ycombinator commented Jan 27, 2019 • edited Loading

ycombinator commented Jan 27, 2019

ruflin left a comment

Choose a reason for hiding this comment

ycombinator commented Jan 28, 2019

ycombinator commented Jan 28, 2019

ycombinator commented Jan 28, 2019

webmat left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

webmat commented Jan 28, 2019

ycombinator commented Jan 29, 2019

webmat left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ycombinator commented Jan 27, 2019 •

edited

Loading

ycombinator commented Jan 27, 2019 •

edited

Loading