[parsers] add cloudtrail via cloudwatch logs support

[ryandeivert]

to: @austinbyers
cc: @airbnb/streamalert-maintainers
size: small

Background

Logs that are ingested via CloudWatch logs subscription filter have a unique format. The format is described here in step 8 under Example # 1. The message objects within the list of logEvents are stored as embedded json objects. However, currently, there is no easy way to use a json path in combination with loading an embedded json object. This adds support for loading of embedded json objects for json records extracted via a json path.

Changes

• Adding new logs.json config option of embedded_json that is a boolean indicating if the extracted json object is an embedded object (encoded json within json).
• Adding json loading logic for embedded json objects via new method of _extract_records. This new method also handles the json regex extraction & loading.
• Adding CloudTrail log via CloudWatch logs schema support. See here for more info on the contents of the log.

Testing

• Adding unit tests for the extraction of the embedded json object.

[austinbyers]

A really clean solution! 👏

[austinbyers]

Should this be a warning? If I understand correctly, this logic parses embedded json (if specified) for each matched log schema. If that fails, then the config is probably wrong? Or will this happen more frequently?

[ryandeivert]

It's probably safe to have it as warning or error. just updated!

[jacknagz]

👏 @ryandeivert could you also add the new embedded_json config option to our docs so other users have an understanding of how it works?

[ryandeivert]

@jacknagz I will make a todo for it and address soon