Fix errors in filebeat Zeek dashboard and README files. Update field descriptions. Add notice.log support. #10916

alakahakai · 2019-02-23T04:33:11Z

Update Filebeat Zeek dashboard to address {issue}10915[10915]

elasticmachine · 2019-02-23T17:02:04Z

Pinging @elastic/secops

FrankHassanabad · 2019-02-23T17:08:24Z

Nit: One small thing I noticed when setting up zeek was that in the readme you need

networks.cfg

Rather than network.cfg

For: /etc/bro/network.cfg

So I think it should be: /etc/bro/networks.cfg

In the readme.

alakahakai · 2019-02-23T17:36:16Z

Nit: One small thing I noticed when setting up zeek was that in the readme you need

networks.cfg

Rather than network.cfg

For: /etc/bro/network.cfg

So I think it should be: /etc/bro/networks.cfg

In the readme.

Yes. Sorry for the typo.

x-pack/filebeat/module/zeek/_meta/fields.yml

x-pack/filebeat/module/zeek/notice/config/notice.yml

webmat

Noted a few things that need to change.

Good stuff overall, though. And I love the idea of doing the field renames right in Beats. It's lightweight and allows for the use of Beats processors on the predictable ECS field names, instead of the arbitrary field names of each different sources.

x-pack/filebeat/module/zeek/README.md

x-pack/filebeat/module/zeek/_meta/fields.yml

x-pack/filebeat/module/zeek/_meta/kibana/7/dashboard/Filebeat-Zeek-Overview.json

x-pack/filebeat/module/zeek/connection/manifest.yml

x-pack/filebeat/module/zeek/notice/config/notice.yml

x-pack/filebeat/module/zeek/notice/ingest/pipeline.json

x-pack/filebeat/module/zeek/notice/test/notice-json.log-expected.json

…ocal_resp

webmat

Almost done.

I think we should avoid painless for simple cases such as when copying a value.

Once that's fixed, I'm good with the PR

webmat · 2019-02-26T14:00:24Z

x-pack/filebeat/module/zeek/connection/test/connection-json.log

@@ -1 +1,3 @@
 {"ts":1547188415.857497,"uid":"CAcJw21BbVedgFnYH3","id.orig_h":"192.168.86.167","id.orig_p":38339,"id.resp_h":"192.168.86.1","id.resp_p":53,"proto":"udp","service":"dns","duration":0.076967,"orig_bytes":75,"resp_bytes":178,"conn_state":"SF","local_orig":true,"local_resp":true,"missed_bytes":0,"history":"Dd","orig_pkts":1,"orig_ip_bytes":103,"resp_pkts":1,"resp_ip_bytes":206,"tunnel_parents":[]}
+{"ts":1547188416.857497,"uid":"CAcJw21BbVedgFnYH4","id.orig_h":"192.168.86.167","id.orig_p":38340,"id.resp_h":"8.8.8.8","id.resp_p":53,"proto":"udp","service":"dns","duration":0.076967,"orig_bytes":75,"resp_bytes":178,"conn_state":"SF","local_orig":true,"local_resp":false,"missed_bytes":0,"history":"Dd","orig_pkts":1,"orig_ip_bytes":103,"resp_pkts":1,"resp_ip_bytes":206,"tunnel_parents":[]}
+{"ts":1547188417.857497,"uid":"CAcJw21BbVedgFnYH5","id.orig_h":"4.4.2.2","id.orig_p":383341,"id.resp_h":"8.8.8.8","id.resp_p":53,"proto":"udp","service":"dns","duration":0.076967,"orig_bytes":75,"resp_bytes":178,"conn_state":"SF","local_orig":false,"local_resp":false,"missed_bytes":0,"history":"Dd","orig_pkts":1,"orig_ip_bytes":103,"resp_pkts":1,"resp_ip_bytes":206,"tunnel_parents":[]}


❤️ moar test cases! 😂

webmat · 2019-02-26T14:03:32Z

x-pack/filebeat/module/zeek/notice/ingest/pipeline.json

+    {
+      "script": {
+        "lang": "painless",
+        "source": "ctx.destination.ip = ctx['destination']['address'];",


Sorry for not catching this earlier, but this copy and the following two can be done with set and the same if.

We should avoid using Painless in cases like this, as it adds unnecessary overhead (even if small).

Check out an example in Suricata module

It was using set processors before. I did not see there was an "if" for set, and set processor was creating null value fields. I will fix this.

…oard-fix

webmat

LGTM

webmat

Oops, the test failures appear related.

When Elasticsearch hangs like this, it may mean one of the pipelines or your template is at fault. Ping me if you need help diagnosing this.

alakahakai · 2019-02-27T07:12:20Z

Oh jeez, never noticed either ;-)

This actually was not the problem. There was a typo in suricata pipeline that caused a parse error, so the test was stuck.

webmat

This PR shouldn't change which version of the stack is used in master.

This is my last complaint for real, this time ;-)

testing/environments/snapshot.yml

alakahakai · 2019-02-27T14:56:34Z

I think this needs to be backported to 7.x. Will open a new PR for that.

webmat

LGTM

* Update Zeek dashboard and README.md. Add support for notice.log. Update field descriptions

* Fix errors in filebeat Zeek dashboard and README files. Update field descriptions. Add notice.log support. (#10916)

Ray Qiu added 2 commits February 22, 2019 18:39

Update Zeek dashboard

a03d6db

Update README.md

c356dc2

alakahakai requested a review from a team as a code owner February 23, 2019 04:33

alakahakai requested a review from a team February 23, 2019 04:33

alakahakai added bug review SecOps labels Feb 23, 2019

Ray Qiu added 2 commits February 23, 2019 09:37

Fix a typo in README - network.cfg should be networks.cfg

f5fbd24

Fix a typo in README - network.cfg should be networks.cfg

1fca544

alakahakai changed the title ~~Update Filebeat Zeek dashboard~~ Update Filebeat Zeek dashboard and README files Feb 23, 2019

Add support for notice.log

a3061ac

alakahakai changed the title ~~Update Filebeat Zeek dashboard and README files~~ Update Filebeat Zeek dashboard and README files. Add notice.log support. Feb 23, 2019

alakahakai requested a review from a team as a code owner February 23, 2019 23:09

alakahakai added the Filebeat Filebeat label Feb 23, 2019

jsoriano reviewed Feb 25, 2019

View reviewed changes

x-pack/filebeat/module/zeek/_meta/fields.yml Show resolved Hide resolved

x-pack/filebeat/module/zeek/notice/config/notice.yml Show resolved Hide resolved

webmat suggested changes Feb 25, 2019

View reviewed changes

webmat mentioned this pull request Feb 25, 2019

Add Filebeat kubernetes module. {issue}10812[10812] #10912

Closed

Ray Qiu added 2 commits February 25, 2019 16:01

Update field descriptions, ingest processors, and others

0c89f54

Rename visualizations for Kibana dashboards

3f1be85

alakahakai requested a review from webmat February 26, 2019 00:15

Ray Qiu added 3 commits February 25, 2019 16:28

Add ingest processor code to add tags for connection local_orig and l…

904f1bc

…ocal_resp

Add pull to CHANGELOG.next.asciidoc

43e416b

Add two more test cases for connection logs related to tags

5c6b8fc

alakahakai changed the title ~~Update Filebeat Zeek dashboard and README files. Add notice.log support.~~ Fix errors in filebeat Zeek dashboard and README files. Update field descriptions. Add notice.log support. Feb 26, 2019

webmat suggested changes Feb 26, 2019

View reviewed changes

Ray Qiu added 2 commits February 26, 2019 08:31

Use set processors with if conditions

93a8c61

Merge remote-tracking branch 'upstream/master' into zeek-kibana-dashb…

237bd10

…oard-fix

Merge with upstream/master

95d72cd

webmat approved these changes Feb 26, 2019

View reviewed changes

webmat suggested changes Feb 26, 2019

View reviewed changes

Ray Qiu added 3 commits February 26, 2019 11:47

Fix copy/paste errors

6c2ad76

Fix copy/paste errors

31412e5

Fix a typo in Suricata eve pipeline that caused test to fail

e73fce2

webmat suggested changes Feb 27, 2019

View reviewed changes

testing/environments/snapshot.yml Outdated Show resolved Hide resolved

Leave snapshot at 7.1.0, will have a new PR to update it to 8.0.0

e8cbbfe

webmat approved these changes Feb 27, 2019

View reviewed changes

alakahakai merged commit 9cc9725 into elastic:master Feb 27, 2019

alakahakai deleted the zeek-kibana-dashboard-fix branch February 27, 2019 19:49

alakahakai pushed a commit that referenced this pull request Feb 28, 2019

Backport PR #10916 to 7.0 (#10972)

062e16c

* Update Zeek dashboard and README.md. Add support for notice.log. Update field descriptions

alakahakai pushed a commit that referenced this pull request Feb 28, 2019

Backport PR #10916 to 7.x (#10967)

7865bff

* Fix errors in filebeat Zeek dashboard and README files. Update field descriptions. Add notice.log support. (#10916)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix errors in filebeat Zeek dashboard and README files. Update field descriptions. Add notice.log support. #10916

Fix errors in filebeat Zeek dashboard and README files. Update field descriptions. Add notice.log support. #10916

alakahakai commented Feb 23, 2019

elasticmachine commented Feb 23, 2019

FrankHassanabad commented Feb 23, 2019 •

edited

Loading

alakahakai commented Feb 23, 2019

webmat left a comment

webmat left a comment

webmat Feb 26, 2019

webmat Feb 26, 2019

alakahakai Feb 26, 2019 •

edited

Loading

webmat left a comment

webmat left a comment

alakahakai commented Feb 27, 2019

webmat left a comment

alakahakai commented Feb 27, 2019

webmat left a comment

Fix errors in filebeat Zeek dashboard and README files. Update field descriptions. Add notice.log support. #10916

Fix errors in filebeat Zeek dashboard and README files. Update field descriptions. Add notice.log support. #10916

Conversation

alakahakai commented Feb 23, 2019

elasticmachine commented Feb 23, 2019

FrankHassanabad commented Feb 23, 2019 • edited Loading

alakahakai commented Feb 23, 2019

webmat left a comment

Choose a reason for hiding this comment

webmat left a comment

Choose a reason for hiding this comment

webmat Feb 26, 2019

Choose a reason for hiding this comment

webmat Feb 26, 2019

Choose a reason for hiding this comment

alakahakai Feb 26, 2019 • edited Loading

Choose a reason for hiding this comment

webmat left a comment

Choose a reason for hiding this comment

webmat left a comment

Choose a reason for hiding this comment

alakahakai commented Feb 27, 2019

webmat left a comment

Choose a reason for hiding this comment

alakahakai commented Feb 27, 2019

webmat left a comment

Choose a reason for hiding this comment

FrankHassanabad commented Feb 23, 2019 •

edited

Loading

alakahakai Feb 26, 2019 •

edited

Loading