[Data Normalization] Make logs optional when datatypes is defined

[chunyong-lin]

to @ryandeivert @mime-frame
cc @airbnb/streamalert-maintainers
size: tiny
contributes to: #304

Summary

Remove log_source constrain which means logs keyword is optional when we define a rule if datatypes keyword present. Otherwise logs keyword is required.
Apply Data Normalization feature to a rule, we can define a rule in this format

from helpers.base import fetch_values_by_datatype

@rule(datatypes=['command'])
def alert_suspicious_wget(rec):
    results = fetch_values_by_datatype(rec, 'command')
    for result in results:
        if fnmatch(result, "wget *"):
            return true
    return false

Changes

Testing

• Integration test is passed, it covers three cases
  • logs is present, datatypes is not
  • logs is present, datatypes is
  • logs is not present, datatypes is
• Unit test is passed, and it covers four cases
  • logs is present, datatypes is not - working as expected
  • logs is present, datatypes is - working as expected
  • logs is not present, datatypes is not - log an ERROR
  • logs is not present, datatypes is - working as expected.
• Testing in AWS testing account is passed

[coveralls]

Coverage increased (+0.003%) to 95.298% when pulling 2f16953 on remove_log_source_constrain into d3ac7f3 on master.

[ryandeivert]

Optional: The is not None here is implicit with just if rule.datatypes:

[ryandeivert]

Is this logging statement no longer helpful? I understand why the one above was removed but can you explain why this one was also?

[chunyong-lin]

Since we removed log_sources constrain, one rule will matched to multiple records. Normalized types dictionaries are defined based on each log source though. It is not an error if normalized type foo doesn't defined in Record A, it just doesn't match and return False is enough. Does it make sense to you?

[ghost]

@chunyong-lin - For the testing, please ensure there is a test for all possible permutations and update the PR description:

1. logs is present, datatypes is not
2. logs is present, datatypes is
3. logs is not present, datatypes is not - should log an ERROR
4. logs is not present, datatypes is

[coveralls]

Coverage increased (+0.003%) to 95.298% when pulling c725c7c on remove_log_source_constrain into 81044e1 on master.

[chunyong-lin]

@mime-frame @ryandeivert PTAL, I also updated my test status to PR description. Thanks 😸

[coveralls]

Coverage remained the same at 95.295% when pulling 3dd0e15 on remove_log_source_constrain into 6610ff6 on master.

[coveralls]

Coverage remained the same at 95.295% when pulling 3dd0e15 on remove_log_source_constrain into 6610ff6 on master.

[coveralls]

Coverage decreased (-0.06%) to 95.231% when pulling 201aa1e on remove_log_source_constrain into 6610ff6 on master.

[ghost]

@chunyong-lin - to confirm, this is ready for review?

[chunyong-lin]

@mime-frame Yes, it is ready for review.

[ryandeivert]

LGTM