[Filebeat] Iptables / ubiquiti module #10176

adriansr · 2019-01-18T16:41:14Z

This adds a new x-pack module to parse iptables logs from a file. The module understands additional fields added by Ubiquiti firewalls.

Example iptables event:

"iptables": {
      "ether_type": 2048,
      "input_device": "eth0",
      "output_device": "wan1",
      "tcp": {
        "reserved_bits": 0,
        "flags": "ACK",
        "window": 128
      },
      "precedence_bits": 0,
      "length": 123,
      "tos": 0,
      "output_device": "lo",
      "id": 12345,
      "ttl": 255
    },

For Ubiquiti logs, it parses the Ubiquiti tag before iptables logs.

Format: RuleSet-RuleNumber-Action

It's a common pattern to format the RuleSet as "InputZone-OutputZone". Action can be A for allow and D for Deny. RuleNumber can be a number or "default".

For [source-dest-42-D]:

        "iptables.ubiquiti.input_zone": "source",
        "iptables.ubiquiti.output_zone": "dest",
        "iptables.ubiquiti.rule_number": "42",
        "iptables.ubiquiti.rule_set": "source-dest",
        "event.outcome": "deny",

The following ECS fields are populated:

        "destination.ip": "10.4.0.5",
        "destination.mac": "90:10:20:76:8d:20",
        "destination.port": 443,
        "ecs.version": "1.0.0-beta2",
        "event.dataset": "iptables.log",
        "event.module": "iptables",
        "event.outcome": "deny", (only when Ubiquiti extension found)
        "fileset.name": "log",
        "network.transport": "tcp",
        "network.type": "ipv4",
        "source.geo.city_name": "Bellaterra",
        "source.geo.continent_name": "Europe",
        "source.geo.country_iso_code": "ES",
        "source.geo.location.lat": 41.5026,
        "source.geo.location.lon": 2.0875,
        "source.geo.region_iso_code": "ES-B",
        "source.geo.region_name": "Barcelona",
        "source.ip": "158.109.0.1",
        "source.mac": "90:10:65:29:b6:2a",
        "source.port": 38842

Closes #8781

elasticmachine · 2019-01-18T16:41:49Z

Pinging @elastic/secops

adriansr

some questions

x-pack/filebeat/module/iptables/log/ingest/pipeline.json

adriansr · 2019-01-18T16:50:15Z

x-pack/filebeat/module/iptables/log/test/iptables.log-expected.json

+        "iptables.tcp_window": 8192,
+        "iptables.ttl": 115,
+        "log.offset": 0,
+        "message": "Jan  8 03:37:09 example-host kernel: iptables DROP_INPUT: IN=eth0 OUT= MAC=90:10:35:5a:1e:3a:90:10:9e:ec:2c:71:08:00 SRC=203.0.113.36 DST=172.16.54.114 LEN=52 TOS=0x00 PREC=0x00 TTL=115 ID=15743 DF PROTO=TCP SPT=17805 DPT=445 WINDOW=8192 RES=0x00 SYN URGP=0 ",


shall I keep or remove the original message field?

@webmat How is this handled in the other modules?

In some modules where there's an actual user-readable message, I grokked the structured headers and sometimes message content, and left only the user-readable message in message. I think that's helpful for those situations.

These firewall logs are entirely structured content, however. So I think you can remove them for now.

There's two corollaries to this:

We've discussed in SecOps always populating message with an event summary, but I will leave it up to @adriansr on whether to do that or not in this PR. I think it could be tackled later, though.

I saw some activity around a beat-wide option to keep/discard the original messages (in event.original, not message). I think @kvch is the best person to tell you about this. For this reason, I don't think this module should handle preservation of the original explicitly.

I decided on log.original, as the message could've been partially parsed by the syslog input

x-pack/filebeat/module/iptables/_meta/docs.asciidoc

x-pack/filebeat/module/iptables/_meta/kibana/6/dashboard/Filebeat-Suricata-Alert-Overview.json

webmat · 2019-01-18T19:22:48Z

x-pack/filebeat/module/iptables/log/test/iptables.log-expected.json

+        "iptables.tcp_window": 8192,
+        "iptables.ttl": 115,
+        "log.offset": 0,
+        "message": "Jan  8 03:37:09 example-host kernel: iptables DROP_INPUT: IN=eth0 OUT= MAC=90:10:35:5a:1e:3a:90:10:9e:ec:2c:71:08:00 SRC=203.0.113.36 DST=172.16.54.114 LEN=52 TOS=0x00 PREC=0x00 TTL=115 ID=15743 DF PROTO=TCP SPT=17805 DPT=445 WINDOW=8192 RES=0x00 SYN URGP=0 ",


In some modules where there's an actual user-readable message, I grokked the structured headers and sometimes message content, and left only the user-readable message in message. I think that's helpful for those situations.

These firewall logs are entirely structured content, however. So I think you can remove them for now.

There's two corollaries to this:

We've discussed in SecOps always populating message with an event summary, but I will leave it up to @adriansr on whether to do that or not in this PR. I think it could be tackled later, though.

I saw some activity around a beat-wide option to keep/discard the original messages (in event.original, not message). I think @kvch is the best person to tell you about this. For this reason, I don't think this module should handle preservation of the original explicitly.

webmat · 2019-01-18T19:24:24Z

I realized this was a WIP, so I didn't mean to post the comments about Suricata LOL

x-pack/filebeat/module/iptables/log/_meta/fields.yml

jsoriano · 2019-01-22T11:52:23Z

x-pack/filebeat/module/iptables/log/ingest/pipeline.json

+                    "field": "source.port",
+                    "type": "integer",
+                    "ignore_missing": true
+                }


Could you cast directly in the grok patterns with the %{SYNTAX:SEMANTIC:TYPE} syntax instead of using convert?

Thanks, I didn't know about casting support in Grok

andrewkroh

Great job as usual.

Note to other reviewers: I didn't look too closely at the script processors in the pipeline.

filebeat/docs/fields.asciidoc

filebeat/docs/modules/iptables.asciidoc

x-pack/filebeat/module/iptables/fields.go

x-pack/filebeat/module/iptables/_meta/docs.asciidoc

webmat

Did a full read through. Noticed minor details, one question.

Amazing work!

webmat · 2019-01-25T21:42:37Z

x-pack/filebeat/module/iptables/log/ingest/pipeline.json

+                        "IPTABLES_IPV6": "SRC=%{IPV6:source.ip} DST=%{IPV6:destination.ip} LEN=%{UNSIGNED_INT:iptables.length:int} TC=%{UNSIGNED_INT:iptables.tos} HOPLIMIT=%{UNSIGNED_INT:iptables.ttl:int} FLOWLBL=%{UNSIGNED_INT:iptables.flow_label:int} %{IPTABLES_IP_PAYLOAD}",
+                        "IPTABLES": "%{IPTABLES_ETHERNET} (:?%{IPTABLES_IP}|%{IPTABLES_IPV6})",
+                        "UBIQUITI_FIELD": "[^-\\]]*",
+                        "UBIQUITI_RULESET_NAME": "[^\\]]*",


Should UBIQUITI_FIELD and UBIQUITI_RULESET_NAME be 1+ characters, instead of 0+?

Also, my head is spinning from so much grokking 😆

I'd rather make the regex as tolerant as possible. I'm worried for example about a case where the rule set name could be empty (it's user supplied). Is there a scenario where 0+ causes a problem?

No, just wanted to make sure that was desired. If you're worried there may be situations where it's empty, then * is fine :-)

x-pack/filebeat/module/iptables/log/ingest/pipeline.json

webmat · 2019-01-25T21:57:53Z

x-pack/filebeat/module/iptables/log/ingest/pipeline.json

+                            }
+                        ]
+                    },
+                    "source": "for (action in params.mappings) { def src = ctx[action.source.object]; if (src != null) { Map map = action.map; String key = src[action.source.key]; String mapping = map[key]; if (mapping != null) { Map dst = ctx[action.destination.object]; if (dst == null) { dst = new HashMap(); ctx[action.destination.object] = dst;} dst[action.destination.key] = mapping; } } }"


Could you explain what this does, please?

Given a source and destination fields in the form "object.key",
it looks up the value of the source field in the given map, and if it exists, it will set the destination field to the mapped value.

for example, it sets network.type to ipv6 if iptables.ether_type is 86:dd, or sets network.transport to ipv6-icmp if its original value was icmpv6.

Is there a better way to do it?

Ah, I see it now. Thanks for explaining.

Only thing I see with this is that right now you're limited to a depth of 2 fields (e.g. network.transport) for source and destination field names. But I'm not sure it's worth modifying for this detail, as we don't need to support deeper nesting for now.

Check out how much I could pack in a screenful in 10192, though. Have you considered just using the basic processors and formatting them more aggressively? ;-)

I still see a need for some custom code for the mapping of values, so just something to think about. No action needed.

By the way, I've been wanting something similar to this in Ingest Node in general. Since right now is a bit late to make requests for 7.0 (understatement of the day), I'm making a note of your code. We could potentially extract it out to a reusable pipeline, pack a bunch of mappings in a temp variable, and call this shared "mapping pipeline". I've noted down this idea in issue elastic/ecs#181

webmat · 2019-01-28T14:26:53Z

x-pack/filebeat/module/iptables/log/ingest/pipeline.json

@@ -32,7 +32,8 @@
                        "IPTABLES_IP_FRAGFLAG": "((?<= )(CE|DF|MF))*",
                        "IPTABLES_IP_START": "SRC=%{IPV4:source.ip} DST=%{IPV4:destination.ip} LEN=%{UNSIGNED_INT:iptables.length:int} TOS=0x%{BASE16NUM:iptables.tos} PREC=0x%{BASE16NUM:iptables.precedence_bits} TTL=%{UNSIGNED_INT:iptables.ttl:int} ID=%{UNSIGNED_INT:iptables.id:int}(?: %{IPTABLES_IP_FRAGFLAG:iptables.fragment_flags})?(?: FRAG: %{UNSIGNED_INT:iptables.fragment_offset:int})?",
                        "IPTABLES_IP": "%{IPTABLES_IP_START} %{IPTABLES_IP_PAYLOAD}",
-                        "IPTABLES_IPV6": "SRC=%{IPV6:source.ip} DST=%{IPV6:destination.ip} LEN=%{UNSIGNED_INT:iptables.length:int} TC=%{UNSIGNED_INT:iptables.tos} HOPLIMIT=%{UNSIGNED_INT:iptables.ttl:int} FLOWLBL=%{UNSIGNED_INT:iptables.flow_label:int} %{IPTABLES_IP_PAYLOAD}",
+                        "IPTABLES_IPV6_START": "SRC=%{IPV6:source.ip} DST=%{IPV6:destination.ip} LEN=%{UNSIGNED_INT:iptables.length:int} TC=%{UNSIGNED_INT:iptables.tos} HOPLIMIT=%{UNSIGNED_INT:iptables.ttl:int} FLOWLBL=%{UNSIGNED_INT:iptables.flow_label:int}",
+                        "IPTABLES_IPV6": "%{IPTABLES_IPV6_START} %{IPTABLES_IP_PAYLOAD}",


❤️ Makes the similarity (... %{IPTABLES_IP_PAYLOAD}) more obvious :-)

webmat

LGTM

Feel free to keep the mapping code as is. I think part of it could have been done in a more compact way, like in the link I pasted up there.

But on the other hand, we may want to extract this out to a reusable pipeline eventually. So let's keep it in for now, as a test bed for this approach.

This adds a new x-pack module to parse iptables logs from a file. The module understands additional fields added by Ubiquiti firewalls. Example iptables event: ``` "iptables": { "ether_type": 2048, "input_device": "eth0", "output_device": "wan1", "tcp": { "reserved_bits": 0, "flags": "ACK", "window": 128 }, "precedence_bits": 0, "length": 123, "tos": 0, "output_device": "lo", "id": 12345, "ttl": 255 }, ``` For Ubiquiti logs, it parses the Ubiquiti tag before iptables logs. Format: RuleSet-RuleNumber-Action It's a common pattern to format the RuleSet as "InputDevice-OutputDevice". Action can be A for allow and D for Deny. RuleNumber can be a number or "default". For [source-dest-42-D]: ``` "iptables.ubiquiti.input_zone": "source", "iptables.ubiquiti.output_zone": "dest", "iptables.ubiquiti.rule_number": "42", "iptables.ubiquiti.rule_set": "source-dest", "event.outcome": "deny", ``` The following ECS fields are populated: ``` "destination.ip": "10.4.0.5", "destination.mac": "90:10:20:76:8d:20", "destination.port": 443, "ecs.version": "1.0.0-beta2", "event.dataset": "iptables.log", "event.module": "iptables", "event.outcome": "deny", (only when Ubiquiti extension found) "fileset.name": "log", "network.transport": "tcp", "network.type": "ipv4", "source.geo.city_name": "Bellaterra", "source.geo.continent_name": "Europe", "source.geo.country_iso_code": "ES", "source.geo.location.lat": 41.5026, "source.geo.location.lon": 2.0875, "source.geo.region_iso_code": "ES-B", "source.geo.region_name": "Barcelona", "source.ip": "158.109.0.1", "source.mac": "90:10:65:29:b6:2a", "source.port": 38842 ``` Closes elastic#8781 Co-Authored-by: James Spiteri <james.spiteri@elastic.co>

adriansr · 2019-01-29T10:35:24Z

jenkins, test this

This adds a new x-pack module to parse iptables logs from a file. The module understands additional fields added by Ubiquiti firewalls. Example iptables event: ``` "iptables": { "ether_type": 2048, "input_device": "eth0", "output_device": "wan1", "tcp": { "reserved_bits": 0, "flags": "ACK", "window": 128 }, "precedence_bits": 0, "length": 123, "tos": 0, "output_device": "lo", "id": 12345, "ttl": 255 }, ``` For Ubiquiti logs, it parses the Ubiquiti tag before iptables logs. Format: RuleSet-RuleNumber-Action It's a common pattern to format the RuleSet as "InputDevice-OutputDevice". Action can be A for allow and D for Deny. RuleNumber can be a number or "default". For [source-dest-42-D]: ``` "iptables.ubiquiti.input_zone": "source", "iptables.ubiquiti.output_zone": "dest", "iptables.ubiquiti.rule_number": "42", "iptables.ubiquiti.rule_set": "source-dest", "event.outcome": "deny", ``` The following ECS fields are populated: ``` "destination.ip": "10.4.0.5", "destination.mac": "90:10:20:76:8d:20", "destination.port": 443, "ecs.version": "1.0.0-beta2", "event.dataset": "iptables.log", "event.module": "iptables", "event.outcome": "deny", (only when Ubiquiti extension found) "fileset.name": "log", "network.transport": "tcp", "network.type": "ipv4", "source.geo.city_name": "Bellaterra", "source.geo.continent_name": "Europe", "source.geo.country_iso_code": "ES", "source.geo.location.lat": 41.5026, "source.geo.location.lon": 2.0875, "source.geo.region_iso_code": "ES-B", "source.geo.region_name": "Barcelona", "source.ip": "158.109.0.1", "source.mac": "90:10:65:29:b6:2a", "source.port": 38842 ``` Closes elastic#8781 Co-Authored-by: James Spiteri <james.spiteri@elastic.co> (cherry picked from commit d7c14df)

…10462) This adds a new x-pack module to parse iptables logs from a file. The module understands additional fields added by Ubiquiti firewalls. Example iptables event: ``` "iptables": { "ether_type": 2048, "input_device": "eth0", "output_device": "wan1", "tcp": { "reserved_bits": 0, "flags": "ACK", "window": 128 }, "precedence_bits": 0, "length": 123, "tos": 0, "output_device": "lo", "id": 12345, "ttl": 255 }, ``` For Ubiquiti logs, it parses the Ubiquiti tag before iptables logs. Format: RuleSet-RuleNumber-Action It's a common pattern to format the RuleSet as "InputDevice-OutputDevice". Action can be A for allow and D for Deny. RuleNumber can be a number or "default". For [source-dest-42-D]: ``` "iptables.ubiquiti.input_zone": "source", "iptables.ubiquiti.output_zone": "dest", "iptables.ubiquiti.rule_number": "42", "iptables.ubiquiti.rule_set": "source-dest", "event.outcome": "deny", ``` The following ECS fields are populated: ``` "destination.ip": "10.4.0.5", "destination.mac": "90:10:20:76:8d:20", "destination.port": 443, "ecs.version": "1.0.0-beta2", "event.dataset": "iptables.log", "event.module": "iptables", "event.outcome": "deny", (only when Ubiquiti extension found) "fileset.name": "log", "network.transport": "tcp", "network.type": "ipv4", "source.geo.city_name": "Bellaterra", "source.geo.continent_name": "Europe", "source.geo.country_iso_code": "ES", "source.geo.location.lat": 41.5026, "source.geo.location.lon": 2.0875, "source.geo.region_iso_code": "ES-B", "source.geo.region_name": "Barcelona", "source.ip": "158.109.0.1", "source.mac": "90:10:65:29:b6:2a", "source.port": 38842 ``` Closes #8781 Co-Authored-by: James Spiteri <james.spiteri@elastic.co> (cherry picked from commit d7c14df)

adriansr requested a review from a team as a code owner January 18, 2019 16:41

adriansr added in progress Pull request is currently in progress. Filebeat Filebeat SecOps labels Jan 18, 2019

adriansr removed the request for review from a team January 18, 2019 16:44

adriansr force-pushed the feature-iptables-ubiquiti branch from c968354 to 4bec0c3 Compare January 18, 2019 16:48

adriansr commented Jan 18, 2019

View reviewed changes

adriansr force-pushed the feature-iptables-ubiquiti branch from 4bec0c3 to 55dfcb0 Compare January 18, 2019 16:58

webmat reviewed Jan 18, 2019

View reviewed changes

ruflin reviewed Jan 21, 2019

View reviewed changes

x-pack/filebeat/module/iptables/log/_meta/fields.yml Outdated Show resolved Hide resolved

adriansr requested a review from a team as a code owner January 22, 2019 11:51

jsoriano reviewed Jan 22, 2019

View reviewed changes

adriansr changed the title ~~[WIP][Filebeat] Iptables / ubiquiti module~~ [Filebeat] Iptables / ubiquiti module Jan 24, 2019

adriansr added review and removed in progress Pull request is currently in progress. labels Jan 24, 2019

andrewkroh approved these changes Jan 24, 2019

View reviewed changes

adriansr force-pushed the feature-iptables-ubiquiti branch 3 times, most recently from 42ec570 to 026d904 Compare January 25, 2019 16:31

webmat reviewed Jan 26, 2019

View reviewed changes

webmat reviewed Jan 28, 2019

View reviewed changes

webmat approved these changes Jan 28, 2019

View reviewed changes

adriansr force-pushed the feature-iptables-ubiquiti branch from 3033e8e to 938b032 Compare January 28, 2019 21:58

adriansr merged commit d7c14df into elastic:master Jan 29, 2019

adriansr mentioned this pull request Jan 31, 2019

Cherrypick #10176 to 6.x branch: [Filebeat] Iptables / ubiquiti module #10461

Closed

adriansr mentioned this pull request Jan 31, 2019

Cherry-pick of #10176 to 6.x: [Filebeat] Iptables / ubiquiti module #10462

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Filebeat] Iptables / ubiquiti module #10176

[Filebeat] Iptables / ubiquiti module #10176

adriansr commented Jan 18, 2019 •

edited

Loading

elasticmachine commented Jan 18, 2019

adriansr left a comment

adriansr Jan 18, 2019

andrewkroh Jan 18, 2019

webmat Jan 18, 2019

adriansr Jan 24, 2019

webmat Jan 18, 2019

webmat commented Jan 18, 2019

jsoriano Jan 22, 2019

adriansr Jan 22, 2019

andrewkroh left a comment

webmat left a comment

webmat Jan 25, 2019

webmat Jan 25, 2019

adriansr Jan 27, 2019

webmat Jan 28, 2019

webmat Jan 25, 2019

adriansr Jan 27, 2019

webmat Jan 28, 2019

webmat Jan 28, 2019

webmat left a comment

adriansr commented Jan 29, 2019

[Filebeat] Iptables / ubiquiti module #10176

[Filebeat] Iptables / ubiquiti module #10176

Conversation

adriansr commented Jan 18, 2019 • edited Loading

elasticmachine commented Jan 18, 2019

adriansr left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

webmat commented Jan 18, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

andrewkroh left a comment

Choose a reason for hiding this comment

webmat left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

webmat left a comment

Choose a reason for hiding this comment

adriansr commented Jan 29, 2019

adriansr commented Jan 18, 2019 •

edited

Loading