Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Filebeat] Improve ECS categorization field mappings for netflow module #18108

Merged
merged 2 commits into from
May 4, 2020

Conversation

leehinman
Copy link
Contributor

What does this PR do?

Adds ECS categorization fields to netflow input. Specifically:

  • event.category : make array and add network
  • event.type
  • related.ip

Why is it important?

ECS categorization fields improves usability of the data in the SIEM app and improves cross correlation between data sources.

Checklist

  • My code follows the style guidelines of this project
  • I have commented my code, particularly in hard-to-understand areas
    - [ ] I have made corresponding changes to the documentation
    - [ ] I have made corresponding change to the default configuration files
  • I have added tests that prove my fix is effective or that my feature works
  • I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

How to test this PR locally

cd x-pack/filebeat/input/netflow && go test

Related issues

Closes #16135

@leehinman leehinman added enhancement Filebeat Filebeat needs_backport PR is waiting to be backported to other branches. Team:SIEM ecs labels Apr 29, 2020
@elasticmachine
Copy link
Collaborator

Pinging @elastic/siem (Team:SIEM)

@botelastic botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Apr 29, 2020
- event.category : make array and add network
- event.type
- related.ip

Closes elastic#16135
@leehinman leehinman force-pushed the 16135_netflow_ecs_1.4 branch from c328820 to bda7330 Compare April 29, 2020 19:13
@elasticmachine
Copy link
Collaborator

elasticmachine commented Apr 29, 2020

💔 Build Failed

Pipeline View Test View Changes Artifacts preview stats

Expand to view the summary

Build stats

Test stats 🧪

Test Results
Failed 1
Passed 509
Skipped 7
Total 517

Test errors

Expand to view the tests failures

  • Name: runTest – nose.failure.Failure

    • Status: FAILED
    • Age: 1
    • Duration: 0
    • Error Details: Cannot subscript an existing Union. Use Union[u, t] instead.

Steps errors

Expand to view the steps failures

  • Name: Mage update build test
    • Description: mage update build test

    • Result: FAILURE

    • Duration: 6 min 44 sec<

    • Start Time: 2020-04-29T23:38:22.344+0000

Log output

Expand to view the last 100 lines of log output

[2020-04-29T23:38:22.308Z] + git config --get user.email
[2020-04-29T23:38:22.308Z] + [ -z  ]
[2020-04-29T23:38:22.308Z] + git config user.email beatsmachine@users.noreply.github.com
[2020-04-29T23:38:22.308Z] + git config user.name beatsmachine
[2020-04-29T23:38:22.321Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18108/src/github.com/elastic/beats/x-pack/filebeat
[2020-04-29T23:38:22.633Z] + mage update build test
[2020-04-29T23:39:18.921Z] No fields files for module azureeventhub
[2020-04-29T23:39:18.921Z] No fields files for module cloudfoundry
[2020-04-29T23:39:18.921Z] No fields files for module googlepubsub
[2020-04-29T23:39:18.921Z] No fields files for module httpjson
[2020-04-29T23:39:18.921Z] No fields files for module o365audit
[2020-04-29T23:39:18.921Z] Generated fields.yml for filebeat to /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18108/src/github.com/elastic/beats/x-pack/filebeat/fields.yml
[2020-04-29T23:39:18.921Z] >> Building filebeat.yml for linux/amd64
[2020-04-29T23:39:18.921Z] >> Building filebeat.reference.yml for linux/amd64
[2020-04-29T23:39:18.921Z] >> Building filebeat.docker.yml for linux/amd64
[2020-04-29T23:39:45.552Z] exec: go list -m
[2020-04-29T23:39:45.552Z] >> build: Building filebeat
[2020-04-29T23:40:41.819Z] >> go test: Unit Testing
[2020-04-29T23:42:48.356Z] SUMMARY:
[2020-04-29T23:42:48.356Z]   Fail:     0
[2020-04-29T23:42:48.356Z]   Skip:     7
[2020-04-29T23:42:48.356Z]   Pass:     509
[2020-04-29T23:42:48.356Z]   Packages: 19
[2020-04-29T23:42:48.356Z]   Duration: 1m59.757486356s
[2020-04-29T23:42:48.356Z]   Coverage Report: /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18108/src/github.com/elastic/beats/x-pack/filebeat/build/TEST-go-unit.html
[2020-04-29T23:42:48.356Z]   JUnit Report:    /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18108/src/github.com/elastic/beats/x-pack/filebeat/build/TEST-go-unit.xml
[2020-04-29T23:42:48.356Z]   Output File:     /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18108/src/github.com/elastic/beats/x-pack/filebeat/build/TEST-go-unit.out
[2020-04-29T23:42:48.356Z] >> go test: Unit Test Passed
[2020-04-29T23:43:44.705Z] >> python test: Unit Testing
[2020-04-29T23:44:06.657Z] E
[2020-04-29T23:44:06.657Z] ======================================================================
[2020-04-29T23:44:06.657Z] ERROR: Failure: TypeError (Cannot subscript an existing Union. Use Union[u, t] instead.)
[2020-04-29T23:44:06.657Z] ----------------------------------------------------------------------
[2020-04-29T23:44:06.657Z] Traceback (most recent call last):
[2020-04-29T23:44:06.657Z]   File "/var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18108/python-env/build/ve/linux/lib/python3.5/site-packages/nose/failure.py", line 39, in runTest
[2020-04-29T23:44:06.657Z]     raise self.exc_val.with_traceback(self.tb)
[2020-04-29T23:44:06.657Z]   File "/var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18108/python-env/build/ve/linux/lib/python3.5/site-packages/nose/loader.py", line 418, in loadTestsFromName
[2020-04-29T23:44:06.657Z]     addr.filename, addr.module)
[2020-04-29T23:44:06.657Z]   File "/var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18108/python-env/build/ve/linux/lib/python3.5/site-packages/nose/importer.py", line 47, in importFromPath
[2020-04-29T23:44:06.657Z]     return self.importFromDir(dir_path, fqname)
[2020-04-29T23:44:06.657Z]   File "/var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18108/python-env/build/ve/linux/lib/python3.5/site-packages/nose/importer.py", line 94, in importFromDir
[2020-04-29T23:44:06.657Z]     mod = load_module(part_fqname, fh, filename, desc)
[2020-04-29T23:44:06.657Z]   File "/usr/lib/python3.5/imp.py", line 234, in load_module
[2020-04-29T23:44:06.657Z]     return load_source(name, filename, file)
[2020-04-29T23:44:06.657Z]   File "/usr/lib/python3.5/imp.py", line 172, in load_source
[2020-04-29T23:44:06.657Z]     module = _load(spec)
[2020-04-29T23:44:06.657Z]   File "<frozen importlib._bootstrap>", line 693, in _load
[2020-04-29T23:44:06.657Z]   File "<frozen importlib._bootstrap>", line 673, in _load_unlocked
[2020-04-29T23:44:06.657Z]   File "<frozen importlib._bootstrap_external>", line 665, in exec_module
[2020-04-29T23:44:06.657Z]   File "<frozen importlib._bootstrap>", line 222, in _call_with_frames_removed
[2020-04-29T23:44:06.657Z]   File "/var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18108/src/github.com/elastic/beats/x-pack/filebeat/tests/system/test_xpack_modules.py", line 6, in <module>
[2020-04-29T23:44:06.657Z]     import test_modules
[2020-04-29T23:44:06.657Z]   File "/var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18108/src/github.com/elastic/beats/x-pack/filebeat/tests/system/../../../../filebeat/tests/system/test_modules.py", line 12, in <module>
[2020-04-29T23:44:06.657Z]     from deepdiff import DeepDiff
[2020-04-29T23:44:06.657Z]   File "/var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18108/python-env/build/ve/linux/lib/python3.5/site-packages/deepdiff/__init__.py", line 10, in <module>
[2020-04-29T23:44:06.657Z]     from .diff import DeepDiff
[2020-04-29T23:44:06.657Z]   File "/var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18108/python-env/build/ve/linux/lib/python3.5/site-packages/deepdiff/diff.py", line 17, in <module>
[2020-04-29T23:44:06.657Z]     from ordered_set import OrderedSet
[2020-04-29T23:44:06.657Z]   File "/var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18108/python-env/build/ve/linux/lib/python3.5/site-packages/ordered_set.py", line 51, in <module>
[2020-04-29T23:44:06.657Z]     class OrderedSet(MutableSet[T], Sequence[T]):
[2020-04-29T23:44:06.657Z]   File "/var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18108/python-env/build/ve/linux/lib/python3.5/site-packages/ordered_set.py", line 186, in OrderedSet
[2020-04-29T23:44:06.657Z]     def update(self, sequence: SetLike[T]) -> int:
[2020-04-29T23:44:06.657Z]   File "/usr/lib/python3.5/typing.py", line 546, in __getitem__
[2020-04-29T23:44:06.657Z]     "Cannot subscript an existing Union. Use Union[u, t] instead.")
[2020-04-29T23:44:06.657Z] TypeError: Cannot subscript an existing Union. Use Union[u, t] instead.
[2020-04-29T23:44:06.657Z] 
[2020-04-29T23:44:06.657Z] [error] 100.00% nose.failure.Failure.runTest: 0.0025s
[2020-04-29T23:44:06.657Z] ----------------------------------------------------------------------
[2020-04-29T23:44:06.657Z] Ran 1 test in 0.003s
[2020-04-29T23:44:06.657Z] 
[2020-04-29T23:44:06.657Z] FAILED (errors=1)
[2020-04-29T23:44:06.657Z] >> python test: Unit Testing Complete
[2020-04-29T23:44:06.657Z] Error: running "/var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18108/python-env/build/ve/linux/bin/nosetests --process-timeout=90 --with-timer --with-xunit --xunit-file=build/TEST-python-unit.xml tests/system/test_xpack_modules.py" failed with exit code 1
[2020-04-29T23:44:06.721Z] Recording test results
[2020-04-29T23:44:08.862Z] Archiving artifacts
[2020-04-29T23:44:09.864Z] + curl -sSLo codecov https://codecov.io/bash
[2020-04-29T23:44:10.134Z] + FILE=auditbeat/build/coverage/full.cov
[2020-04-29T23:44:10.134Z] + [ -f auditbeat/build/coverage/full.cov ]
[2020-04-29T23:44:10.134Z] + FILE=filebeat/build/coverage/full.cov
[2020-04-29T23:44:10.134Z] + [ -f filebeat/build/coverage/full.cov ]
[2020-04-29T23:44:10.134Z] + FILE=heartbeat/build/coverage/full.cov
[2020-04-29T23:44:10.134Z] + [ -f heartbeat/build/coverage/full.cov ]
[2020-04-29T23:44:10.134Z] + FILE=libbeat/build/coverage/full.cov
[2020-04-29T23:44:10.134Z] + [ -f libbeat/build/coverage/full.cov ]
[2020-04-29T23:44:10.134Z] + FILE=metricbeat/build/coverage/full.cov
[2020-04-29T23:44:10.134Z] + [ -f metricbeat/build/coverage/full.cov ]
[2020-04-29T23:44:10.134Z] + FILE=packetbeat/build/coverage/full.cov
[2020-04-29T23:44:10.134Z] + [ -f packetbeat/build/coverage/full.cov ]
[2020-04-29T23:44:10.134Z] + FILE=winlogbeat/build/coverage/full.cov
[2020-04-29T23:44:10.134Z] + [ -f winlogbeat/build/coverage/full.cov ]
[2020-04-29T23:44:10.135Z] + FILE=journalbeat/build/coverage/full.cov
[2020-04-29T23:44:10.135Z] + [ -f journalbeat/build/coverage/full.cov ]
[2020-04-29T23:44:11.704Z] Failed in branch Filebeat x-pack
[2020-04-29T23:44:12.420Z] Running on Jenkins in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18108
[2020-04-29T23:44:12.631Z] [INFO] getVaultSecret: Getting secrets
[2020-04-29T23:44:12.692Z] Masking supported pattern matches of $VAULT_ADDR or $VAULT_ROLE_ID or $VAULT_SECRET_ID
[2020-04-29T23:44:13.746Z] + chmod 755 generate-build-data.sh
[2020-04-29T23:44:13.747Z] + ./generate-build-data.sh https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats-beats-mbp/PR-18108/ https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats-beats-mbp/PR-18108/runs/3 FAILURE 2005866
[2020-04-29T23:44:14.297Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats-beats-mbp/PR-18108/runs/3/steps/?limit=10000 -o steps-info.json
[2020-04-29T23:44:14.848Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats-beats-mbp/PR-18108/runs/3/tests/?status=FAILED -o tests-errors.json

Copy link
Member

@andrewkroh andrewkroh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

if len(relatedIP) > 0 {
ecsRelated := common.MapStr{}
ecsRelated["ip"] = relatedIP
event.Fields["related"] = ecsRelated
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
event.Fields["related"] = ecsRelated
event.Fields["related"] = common.MapStr{"ip": relatedIP}

@andresrc andresrc removed the needs_team Indicates that the issue/PR needs a Team:* label label May 2, 2020
@leehinman leehinman merged commit e6d4787 into elastic:master May 4, 2020
@leehinman leehinman deleted the 16135_netflow_ecs_1.4 branch May 4, 2020 13:33
leehinman added a commit to leehinman/beats that referenced this pull request May 4, 2020
…le (elastic#18108)

* Improve ECS categorization field mappings for netflow module

- event.category : make array and add network
- event.type
- related.ip

Closes elastic#16135

(cherry picked from commit e6d4787)
leehinman added a commit that referenced this pull request May 4, 2020
…le (#18108) (#18173)

* Improve ECS categorization field mappings for netflow module

- event.category : make array and add network
- event.type
- related.ip

Closes #16135

(cherry picked from commit e6d4787)
@andrewkroh andrewkroh removed the needs_backport PR is waiting to be backported to other branches. label May 6, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Filebeat][Netflow] Populate new ECS fields for netflow
4 participants