[lambda][rule] processed logs will now be sent immediately to the alert processor

[ryandeivert]

to @airbnb/streamalert-maintainers
size: medium

changes

• Refactoring the rule processor so alerts are sent to the alert processor (via sns) as they are processed.
  • Prior to these changes, we would first iterator over all logs and apply the rule logic against them, caching any that matched a rule along the way. We would finally send them to the alert processor (one at a time) after all logs had been processed.
  • These change will make this all happen sequentially and any logs that match a rule will get sent as we process (instead of caching and sending at the end).
• Will log a message on every 100 records processed from s3 so we can track progress.
  • Adding some logic that will also try to estimate the number of records in the entire file during processing (we use a generator to avoid reading all the lines multiple times, and getting the exact line count would require reading the entire file twice).
• Other indirect gains:
  • boto3 client creation now happens when the StreamSink class is instantiated and recycled to avoid continuously re-creating it.
  • the StreamSink class does not cache the alerts list and will send on demand
• Removing/updating some old code (ie: things related to staging) that is no longer relevant.
• Updating tests where applicable

[jacknagz]

can we add SNS tests via moto to verify the messages would send?

[jacknagz]