[lookup_tables] support for S3 lookup table files

[chunyong-lin]

to: @ryandeivert or @austinbyers
cc: @airbnb/streamalert-maintainers
size: medium
resolves N/A

Background

This PR is to support lookup tables in Rule Processor. The lookup tables are globally shared and can be referenced in a rule.

The lookup tables feature will download json files from S3 buckets and load them into memory. The buckets info can be configured in conf/global.json, for example,

"infrastructure": {
    "lookup_tables": {
      "buckets": {
        "bucket_name": [
          "foo.json",
          "bar.json"
        ]
      },
      "cache_refresh_minutes": 10,
      "enabled": false
    },
...
}

Changes

• Add a new class LookupTables to handle downloading files from S3 buckets and loading them into memory. It also refreshes lookup tables, similar to how to refresh rule table.
• Mock S3 bucket for integration and unit testing.

Testing

• Rule testing

python manage.py lambda test --processor all
...
StreamAlertCLI [INFO]: (66/66) Successful Tests
StreamAlertCLI [INFO]: (36/36) Alert Tests Passed

• Unit testing

./tests/scripts/unit_tests.sh
...
Ran 749 tests in 12.404s

OK

[austinbyers]

❇️

[coveralls]

Coverage remained the same at 97.649% when pulling 7fc9d7d on support_lookup_tables into 5c24f16 on master.