Add zeek package #245

leehinman · 2020-08-12T22:20:51Z

What does this PR do?

Add package for the filebeat Zeek module. Includes capture_loss,
connection, dce_rpc, dhcp, dnp3, dns, dpd, files, ftp, http, intel
irc, kerberos, modbus, mysql, notice, ntlm, ocsp, pe, radius, rdp
rfb, sip, smb_cmd, smb_files, smb_mapping, smtp, snmp, socks
ssh, ssl, stats, syslog, traceroute, tunnel, weird, x509

Checklist

I have reviewed tips for building integrations and this pull request is aligned with them.
I have verified that all datasets collect metrics or logs.

Related issues

Closes Migrate zeek #225

Screenshots

elasticmachine · 2020-08-12T22:20:53Z

Pinging @elastic/integrations (Team:Integrations)

elasticmachine · 2020-08-12T22:20:53Z

Pinging @elastic/siem (Team:SIEM)

elasticmachine · 2020-08-12T22:24:50Z

💚 Build Succeeded

Expand to view the summary

Build stats

Build Cause: [Pull request #245 updated]
Start Time: 2020-09-18T16:52:46.692+0000
Duration: 4 min 35 sec

andrewstucki · 2020-08-13T23:41:09Z

I'll go through this... slowly... reviews will be in chunks

andrewkroh · 2020-09-14T14:57:20Z

packages/zeek/manifest.yml

+    title: 'Collect Zeek capture_loss, connection, dce_rpc, dhcp, dnp3, dns, dpd, files, ftp, http, intel, irc, kerberos, modbus, mysql, notice, ntlm, ocsp, pe, radius, rdp, rfb, sip, smb_cmd, smb_files, smb_mapping, smtp, snmp, socks, ssh, ssl, stats, syslog, traceroute, tunnel, weird and x509 logs'
+    description: 'Collecting capture_loss, connection, dce_rpc, dhcp, dnp3, dns, dpd, files, ftp, http, intel, irc, kerberos, modbus, mysql, notice, ntlm, ocsp, pe, radius, rdp, rfb, sip, smb_cmd, smb_files, smb_mapping, smtp, snmp, socks, ssh, ssl, stats, syslog, traceroute, tunnel, weird and x509 logs from Zeek instances'
+owner:
+  github: elastic/siem


Suggested change

github: elastic/siem

github: elastic/security-ingest

andrewkroh · 2020-09-14T14:58:56Z

packages/zeek/manifest.yml

+  description: Collect logs from Zeek instances
+  inputs:
+  - type: logfile
+    title: 'Collect Zeek capture_loss, connection, dce_rpc, dhcp, dnp3, dns, dpd, files, ftp, http, intel, irc, kerberos, modbus, mysql, notice, ntlm, ocsp, pe, radius, rdp, rfb, sip, smb_cmd, smb_files, smb_mapping, smtp, snmp, socks, ssh, ssl, stats, syslog, traceroute, tunnel, weird and x509 logs'


This seems like rather long title. Maybe we should drop the names of the individual logs.

andrewkroh · 2020-09-14T19:00:25Z

packages/zeek/kibana/visualization/1df7ea80-370d-11e9-aa6d-ff445a78330c.json

+                "indexRefName": "kibanaSavedObjectMeta.searchSourceJSON.index",
+                "query": {
+                    "language": "kuery",
+                    "query": ""


Should these visualizations have a query to only select Zeek data? If so we can take advantage of the constant_keyword data_stream fields for this.

andrewkroh · 2020-09-14T19:26:30Z

packages/zeek/dataset/dhcp/manifest.yml

+    show_user: true
+    default:
+    - zeek.dhcp
+  - name: community_id


This might be a lot of work, but I'd like to have this config option removed from everywhere. The processor should always be included. I can't think of any reason to make it configurable.

andrewkroh

LGTM 🎉

mtojek

Unfortunately CI reported issues for this PR: https://beats-ci.elastic.co/blue/organizations/jenkins/Beats%2Fintegrations/detail/PR-245/13/pipeline

EDIT:

After a short investigation with @ycombinator , it looks that we have a requirement for dataset names to be at least 3 characters: https://github.com/elastic/package-spec/blob/master/versions/1/dataset/spec.yml#L6 . I'm wondering whether it's a good requirement to prevent a non-descriptive, mysterious names like in this case "pe".

It's not a big deal to change it, but maybe it's better to keep it. WDYT @ruflin @ycombinator ?

leehinman · 2020-09-17T14:30:56Z

Unfortunately CI reported issues for this PR: https://beats-ci.elastic.co/blue/organizations/jenkins/Beats%2Fintegrations/detail/PR-245/13/pipeline

EDIT:

After a short investigation with @ycombinator , it looks that we have a requirement for dataset names to be at least 3 characters: https://github.com/elastic/package-spec/blob/master/versions/1/dataset/spec.yml#L6 . I'm wondering whether it's a good requirement to prevent a non-descriptive, mysterious names like in this case "pe".

It's not a big deal to change it, but maybe it's better to keep it. WDYT @ruflin @ycombinator ?

Just a bit of context. The datasets are named after the ~40 log files that zeek produces. And in this case the log file is called pe.log (abbreviation for Portable Executable).

It might be a little weird for all the others to be direct mappings (mysql.log -> mysql) but this one to be pe.log -> portable_executable. I'd vote for an exception in this case, most of the time a 2 letter name would be mysterious but in this context making it not like all the others could be more mysterious. Luckily this is the only 2 letter log name for zeek. https://docs.zeek.org/en/current/script-reference/log-files.html

ycombinator · 2020-09-17T14:39:17Z

I'm in favor of keeping it as pe (which would mean relaxing the constraint in the spec). pe makes sense in the context of this integration so I would expect it would make sense to developers of this integration as well.

However, if we decide to change it to be more descriptive, then I'd suggest doing that consistently for other datasets of this integration as well, e.g. dnp3, dpd, ocsp, rfb. But then you'd again run into an issue of subjectivity: who decides whether an abbreviation need to expanded to be more descriptive vs. leaving it as-is?

So I think the rule-of-thumb should be that we use dataset names that makes sense in the context of an integration.

mtojek · 2020-09-18T06:48:13Z

Ok, I'm going to open PR against package-spec to adjust the schema.

ruflin · 2020-09-18T07:08:14Z

I'm good with changing the constraint. Keep in mind that in most cases this also effects the index name. For short vs long: We probably should have a recommendation but in the end, it is up to the person writing the package to make the call.

mtojek · 2020-09-18T08:41:14Z

@leehinman I updated the dependency on package-spec on master branch. Please rebase this PR against master and mage clean && mage check.

- capture_loss - connection - dce_rpc - dhcp - dnp3 - dns - dpd - files - ftp - http - intel - irc - kerberos - modbus - mysql - notice - ntlm - ocsp - pe - radius - rdp - rfb - sip - smb_cmd - smb_files - smb_mapping - smtp - snmp - socks - ssh - ssl - stats - syslog - traceroute - tunnel - weird - x509 Closes elastic#225

- quoting of = - indentation in a painless script makes it easier to sync with Filebeat

- removed config option for community_id - changed owner to security-ingest - Shortened Title and reworded description - limit visualizations to zeek data

- kerberos, ssl & x509 ECS 1.6.0 changes - fix community_id processor

- mage check changes

leehinman · 2020-09-18T18:29:41Z

@leehinman I updated the dependency on package-spec on master branch. Please rebase this PR against master and mage clean && mage check.

Done. Thanks for all the help.

* Add zeek package - capture_loss - connection - dce_rpc - dhcp - dnp3 - dns - dpd - files - ftp - http - intel - irc - kerberos - modbus - mysql - notice - ntlm - ocsp - pe - radius - rdp - rfb - sip - smb_cmd - smb_files - smb_mapping - smtp - snmp - socks - ssh - ssl - stats - syslog - traceroute - tunnel - weird - x509 - limit visualizations to zeek data - removed config option for communit_id processor - synced with filebeat zeek module for ECS 1.6.0 changes Closes elastic#225

leehinman added enhancement New feature or request Team:Integrations Label for the Integrations team Team:SIEM (Deprecated) labels Aug 12, 2020

This was referenced Aug 19, 2020

[Filebeat] Sync with zeek package elastic/beats#20696

Merged

Cherry-pick #20696 to 7.x: [Filebeat] Sync with zeek package elastic/beats#20712

Merged

andrewkroh reviewed Sep 14, 2020

View reviewed changes

andrewkroh approved these changes Sep 17, 2020

View reviewed changes

mtojek self-requested a review September 17, 2020 12:14

mtojek suggested changes Sep 17, 2020

View reviewed changes

This was referenced Sep 18, 2020

Support two-character dataset names elastic/package-spec#48

Merged

Update dependency on package-spec elastic/elastic-package#106

Merged

Update dependency on elastic-package #274

Merged

leehinman added 5 commits September 18, 2020 11:40

yaml fixes

899a8b4

- quoting of = - indentation in a painless script makes it easier to sync with Filebeat

Incorporate feedback

5448951

- removed config option for community_id - changed owner to security-ingest - Shortened Title and reworded description - limit visualizations to zeek data

sync with recent filebeat changes

2a0ed63

- kerberos, ssl & x509 ECS 1.6.0 changes - fix community_id processor

Reformat yaml

36c7a2f

- mage check changes

leehinman force-pushed the 225_zeek branch from 8c41e34 to 36c7a2f Compare September 18, 2020 16:52

mtojek self-requested a review September 18, 2020 19:12

mtojek approved these changes Sep 18, 2020

View reviewed changes

leehinman merged commit 86425ca into elastic:master Sep 18, 2020

leehinman deleted the 225_zeek branch December 1, 2020 16:32

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add zeek package #245

Add zeek package #245

leehinman commented Aug 12, 2020

elasticmachine commented Aug 12, 2020

elasticmachine commented Aug 12, 2020

elasticmachine commented Aug 12, 2020 •

edited

Loading

Build stats

andrewstucki commented Aug 13, 2020

andrewkroh Sep 14, 2020

andrewkroh Sep 14, 2020

andrewkroh Sep 14, 2020

andrewkroh Sep 14, 2020

andrewkroh left a comment

mtojek left a comment •

edited

Loading

leehinman commented Sep 17, 2020

ycombinator commented Sep 17, 2020

mtojek commented Sep 18, 2020

ruflin commented Sep 18, 2020

mtojek commented Sep 18, 2020

leehinman commented Sep 18, 2020

Add zeek package #245

Add zeek package #245

Conversation

leehinman commented Aug 12, 2020

What does this PR do?

Checklist

Related issues

Screenshots

elasticmachine commented Aug 12, 2020

elasticmachine commented Aug 12, 2020

elasticmachine commented Aug 12, 2020 • edited Loading

💚 Build Succeeded

Build stats

andrewstucki commented Aug 13, 2020

andrewkroh Sep 14, 2020

Choose a reason for hiding this comment

andrewkroh Sep 14, 2020

Choose a reason for hiding this comment

andrewkroh Sep 14, 2020

Choose a reason for hiding this comment

andrewkroh Sep 14, 2020

Choose a reason for hiding this comment

andrewkroh left a comment

Choose a reason for hiding this comment

mtojek left a comment • edited Loading

Choose a reason for hiding this comment

leehinman commented Sep 17, 2020

ycombinator commented Sep 17, 2020

mtojek commented Sep 18, 2020

ruflin commented Sep 18, 2020

mtojek commented Sep 18, 2020

leehinman commented Sep 18, 2020

elasticmachine commented Aug 12, 2020 •

edited

Loading

mtojek left a comment •

edited

Loading