Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[core][tests] csv parser bug and unit test descriptions #54

Merged
merged 2 commits into from
Mar 3, 2017

Conversation

jacknagz
Copy link
Contributor

@jacknagz jacknagz commented Mar 3, 2017

to @airbnb/streamalert-maintainers

size: small

changes

  • Ensure we are calling the hints option with .get(). Not every parser has a hints property, so this would throw a KeyError if a given stream is analyzing both KV and CSV data.
  • Add a TestKVParser unit testing class.
  • Add descriptions to unit test methods. Produces a nicer and readable output when running tests:
$ ./test/scripts/unit_tests.sh
Payload Classify CSV ... ok
Payload Classify Nested CSV ... ok
Payload Classify JSON ... ok
Payload Classify KV ... ok
Payload Classify Nested JSON ... ok
Payload Classify Nested JSON Missing Subkeys ... ok
Payload Classify JSON osquery ... ok
Payload Classify Nested JSON Generic ... ok
Payload Classify Syslog ... ok
Payload Source Mapping 1 ... ok
Payload Source Mapping 2 ... ok
Payload Record Refresh ... ok
Config Validator - Valid Config ... ok
Config Validator - No Parsers ... ok
Config Validator - No Logs ... ok
Parse CloudWatch JSON ... ok
Parse Non-nested JSON objects ... ok
Parse Cloudtrail JSON ... ok
Parse Inspec JSON ... ok
Parse Multi-layered JSON ... ok
Parse KV - 'key:value,key:value' ... ok
Helpers - In Set ... ok
Helpers - Last Hour ... ok
Rule Engine - Alert Format ... ok
Rule Engine - Basic Rule/Matcher ... ok
Rule Engine - CSV Rule ... ok
Rule Engine - KV Rule ... ok
Rule Engine - Req Subkeys ... ok
Rule Engine - Syslog Rule ... ok

Name                           Stmts   Miss  Cover
--------------------------------------------------
stream_alert.py                    1      0   100%
stream_alert/classifier.py       114      4    96%
stream_alert/config.py            42     13    69%
stream_alert/parsers.py          193      4    98%
stream_alert/pre_parsers.py       58     40    31%
stream_alert/rules_engine.py      85      0   100%
--------------------------------------------------
TOTAL                            493     61    88%
----------------------------------------------------------------------
Ran 29 tests in 0.161s

OK

Not every parser requires the 'hints' key, yet the csv parser was calling it
without a .get().  In the case of KV data, this would throw a KeyError.
@jacknagz jacknagz force-pushed the jacknagz-csv-parser-bug-and-unit-test-descriptions branch from f4b28f4 to ebe43e6 Compare March 3, 2017 21:58
@jacknagz jacknagz force-pushed the jacknagz-csv-parser-bug-and-unit-test-descriptions branch from ebe43e6 to 4ba0153 Compare March 3, 2017 21:59
@jacknagz jacknagz merged commit e850298 into master Mar 3, 2017
@jacknagz jacknagz deleted the jacknagz-csv-parser-bug-and-unit-test-descriptions branch March 3, 2017 22:38
@jacknagz jacknagz added the bug label Mar 3, 2017
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant