Refactor StreamPayload, Parsers, Testing, and more #34
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
added 9 commits
February 8, 2017 17:48
* this change reduces complexity in classifier * uses a consistent naming scheme for pre-parser classes/methods * updates unit tests to reflect changes
Previously, all matchers and rules were evaluated in parallel. We can save processing if we only evaluate rules if matchers return True. This change also cleans up the rules_engine.process flow.
Overall, this refactor provides a common interface for all parsers. By namespacing parsers into their own classes, we get the benefit of parser specific helper methods, along with a consistent structure for all parsers. Change summary: * Move all parser logic out of StreamClassifier into a new parsers.py file * Create an Abstract Base Class for Parsers to inherit from * Register new parser classes with a parser decorator * Expose a get_parser method to other modules
Please enter the commit message for your changes. Lines starting
* add error handing for improper JSON files * fix a bug in the return statement spacing where it was skipping checks * add unit test cases for validating configs
* Add functional test fixtures, rules, and conf for users to work off of * Fix a bug in the CLI test to strip all records of newline characters * Abstract the testing commands into shell scripts to be used by devs and Travis CI
* We don't need the Terraform binary to run Lambda commands (it uses the AWS CLI) * Resolves #31
austinbyers
approved these changes
Feb 14, 2017
There was a problem hiding this comment.
LGTM
I still think using JSONSchema or similar would be simpler than using your own JSON schema library, but I haven't looked at it in depth
This was referenced Feb 14, 2017
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
to @airbnb/streamalert-maintainers
size: large
resolves: #32, #31, #30, #27, and #26
Change Summary
This PR refactors and simplifies a lot of core classes in StreamAlert to be more modular and extensible:
configloading.