Create re-usable Lambda Terraform module

[austinbyers]

to: @ryandeivert and @jacknagz
cc: @airbnb/streamalert-maintainers
size: medium

Background

The number of different Lambda functions has grown over time and will continue to do so this quarter. To simplify the process of managing Terraform resources related to each Lambda function, this creates a generic re-usable Lambda module. BinaryAlert has a similar module which provided inspiration for this implementation.

Changes

• Creates a new module tf_lambda. The alert processor (and future alert dispatcher) will use this soon and other Lambda functions will be migrated to this over time. The module contains the following components:
  • IAM execution role with basic permissions
  • Lambda function
  • Production alias
  • CloudWatch log group
  • CloudWatch metric alarms related to Lambda
• Fixes IAM permissions for the alert processor when using S3 as an output
• Terraform format fixes newlines in other module files

Providing a re-usable module like this ensures that all Lambda functions have the same base permissions, metrics, and tagging. For example, right now tagging is not enabled for CloudWatch log groups in most (all?) of StreamAlert's Lambda functions.

This PR simply creates the new module, it does not use it any way. The existing infrastructure remains unchanged.

Testing

• Terraform format/validate
• Deploy to a test account. Modified main.tf.json to create a global alert processor like so:

{
  "module": {
    "global_alert_processor": {
      "environment_variables": {
        "TEST": "VALUE"
      },
      "function_name": "global_alert_processor",
      "handler": "stream_alert.alert_processor.main.handler",
      "source": "modules/tf_lambda",
      "source_bucket": "SOURCE_BUCKET",
      "source_object_key": "SOURCE_OBJECT_KEY"
    }
  }
}

• Added "enable_iterator_age_alarm": true - iterator age alarm was created
• Added VPC configuration - function was destroyed and recreated in VPC
• Added "enable_metric_alarms": false - metric alarms were destroyed
• Added "enabled": false - all resources were destroyed

[coveralls]

Coverage remained the same at 95.401% when pulling c44a446 on austin-alert-decluster into 0bb9ae2 on release-2.0.0.

[austinbyers]

Update: Due to hashicorp/terraform-provider-aws#443, this is going to have to be re-worked so that there are 2 different Lambda resources (VPC and non-VPC). It works the first time, and then subsequent deploys with no VPC always fail. 👎

Not ready for review yet - apologies

[austinbyers]

@jacknagz @ryandeivert this is now ready for review. Thanks!

[jacknagz]

One quick comment, and then another general question: How will this module be used with the SA CLI? Are those changes coming after this?

[jacknagz]

Has there been an example where this would be used? I'm trying to contrast this approach with just deleting the module altogether

[austinbyers]

Yes - the threat intel downloader is optional. Once the downloader uses this module (instead of building the Lambda resource directly), it needs a way to indicate whether or not the function should be enabled. This is, for example, how the optional BinaryAlert CarbonBlack downloader works

[jacknagz]

I see, so your vision is to pre-allocate the Terraform code, and then just flip the switch as needed. Generally with the SA CLI, we would just not marshal out the Terraform code to generate the Lambda. Either approach works though.

[austinbyers]

Ah, I see. Yeah, having the feature just adds flexibility - whichever way works best

[austinbyers]

@jacknagz this PR just provides a new module for future use, it does not change the existing infrastructure or workflow in any way

After this merges, we will create a global alert processor, at which point the CLI, config files, and Terraform generation process will need to be changed accordingly.

[jacknagz]

Sounds good, thanks! @austinbyers