Merge pull request #205 from airbnb/jacknaglieri-cloudtrail-patterns

Bug Fixes (Initialization, AWS Perms, Config), CloudWatch Event Patterns, and More

conf/clusters/prod.json

@@ -10,22 +10,22 @@
         "s3_bucket_suffix": "streamalert.results"
       },
       "streams": {
-        "retention": 36,
-        "shards": 5
+        "retention": 24,
+        "shards": 1
       }
     },
     "kinesis_events": {
       "enabled": true
     },
     "stream_alert": {
       "alert_processor": {
-        "current_version": 7,
+        "current_version": "$LATEST",
         "memory": 128,
-        "timeout": 25
+        "timeout": 10
       },
       "rule_processor": {
-        "current_version": 8,
-        "memory": 256,
+        "current_version": "$LATEST",
+        "memory": 128,
         "timeout": 10
       }
     }

docs/source/account.rst

@@ -26,33 +26,61 @@ prefix
 
 Open ``conf/global.json`` and ``conf/lambda.json`` and replace ``PREFIX_GOES_HERE`` with your company or organization name.
 
-Administrator
-~~~~~~~~~~~~~
-
-To successfully deploy StreamAlert, you need to create an administrative user in the AWS account.
-
-Steps:
+user account
+~~~~~~~~~~~~
+
+To deploy StreamAlert, you need to create an AWS user for administration.
+
+First, create the policy to attach to the user:
+
+* Go to: Services => IAM => Policies
+* Click: Create policy
+* Select: Create your Own Policy
+* Name the policy ``streamalert``, and paste the following as the ``Policy Document``:
+* Clock: Create Policy
+
+.. code-block::
+
+  {
+    "Version": "2012-10-17",
+    "Statement": [
+        {
+            "Effect": "Allow",
+            "Action": [
+                "athena:*",
+                "cloudtrail:*",
+                "cloudwatch:*",
+                "ec2:*FlowLogs",
+                "events:*",
+                "firehose:*",
+                "iam:*",
+                "kinesis:*",
+                "kms:*",
+                "lambda:*",
+                "logs:*",
+                "s3:*",
+                "sns:*"
+            ],
+            "Resource": "*"
+        }
+    ]
+  }
+
+Next, create the user:
 
 * Go to: Services => IAM => Users
 * Click: Add user
-* Username: streamalert
+* Username: ``streamalert``
 * Access type: Programmatic access
-* Click: Next
+* Click: ``Next: Permissions``
 * Select: Attach existing policies directly
-* Attach the following policies::
+* Attach the previously created ``streamalert`` policy
+* Click: ``Next: Review``, and then ``Create user``
 
-  * AmazonKinesisFirehoseFullAccess
-  * AmazonKinesisFullAccess
-  * AmazonS3FullAccess
-  * AmazonSNSFullAccess
-  * AWSLambdaFullAccess
-  * CloudWatchFullAccess
-  * CloudWatchLogsFullAccess
-  * IAMFullAccess
-* Click:  Next (Review), and then Create User
-
-Take the Access Key and Secret Key and export them to your environment variables::
+Copy the Access Key ID and Secret Access Key and export them to your environment variables::
 
   $ export AWS_ACCESS_KEY_ID="REPLACE_ME"
   $ export AWS_SECRET_ACCESS_KEY="REPLACE_ME"
   $ export AWS_DEFAULT_REGION="us-east-1"
+
+.. note:: Remember to save your credentials in a safe place!

docs/source/clusters.rst

@@ -79,11 +79,11 @@ An example ``production`` cluster::
 Customizing Clusters
 ~~~~~~~~~~~~~~~~~~~~
 
-Each cluster can be broken up into multiple modules to make up a StreamAlert cluster.
+Each StreamAlert cluster is made up of multiple modules.
 
 Each module corresponds to a Terraform module found in the ``terraform/modules`` directory, and serves a specific purpose in a StreamAlert cluster.
 
-After making modifications to a cluster's file, make sure you apply it with::
+After making modifications to a cluster file, make sure you apply the changes with::
 
   $ python stream_alert_cli.py terraform build
   
@@ -92,12 +92,18 @@ This will regenerate the necessary Terraform files and then apply the changes.
 Module: StreamAlert
 --------------------
 
-See `Lambda Settings <lambda.html>`_ for customization options.
+The main module for StreamAlert.
+
+It creates both AWS Lambda functions, aliases, an SNS topic, IAM permissions, and more.
+
+See `Lambda Settings <lambda.html>`_ for all customization options.
 
 Module: Kinesis
 ---------------
 
-See `Kinesis <kinesis.html>`_ for customization options.
+This module contains configuration for the Kinesis Streams and Kinesis Firehose infrastructure.
+
+See `Kinesis <kinesis.html>`_ for all customization options.
 
 Module: CloudWatch Monitoring
 -----------------------------
@@ -134,19 +140,32 @@ Template::
 Module: CloudTrail
 ------------------
 
-AWS CloudTrail is a service that enables compliance, operational auditing, and risk auditing of your AWS account.
+`AWS CloudTrail <https://aws.amazon.com/cloudtrail/>`_ is a service that enables compliance, operational auditing, and risk auditing of your AWS account.
 
 StreamAlert has native support for enabling and monitoring CloudTrail logs with the ``cloudtrail`` module.
 
 When writing rules for CloudTrail data, use the ``cloudwatch:event`` log source.
 
 By default, all API calls will be logged and accessible from rules.
 
-Template::
+**template**
 
-  "cloudtrail": {
-    "enabled": true
-  }
+.. code-block::
+
+    "cloudtrail": {
+      "enabled": true
+    }
+
+**options**
+
+=============        ========     =======                            ===========
+Key                  Required     Default                            Description
+-------------        ---------    -------                            -----------
+``enabled``          Yes          -                                  To enable/disable the CloudTrail.
+``existing_trail``   No           ``false``                          Set to ``true`` if the account has an existing CloudTrail.  This is to avoid duplication of data collected by multiple CloudTrails.
+``is_global_trail``  No           ``true``                           If the CloudTrail should collect events from any region.
+``event_pattern``    No           ``{"account": ["<accound_id>"]}``  The CloudWatch Events pattern to send to Kinesis.  `More information <http://docs.aws.amazon.com/AmazonCloudWatch/latest/events/EventTypes.html>`_.
+=============        =========    =======                            ===========
 
 Module: Flow Logs
 -----------------
@@ -157,7 +176,9 @@ In the settings below, an arbitrary amount of subnets, vpcs, and enis can be ena
 
 When writing rules for this data, use the ``cloudwatch:flow_logs`` log source.
 
-Template::
+**template**
+
+.. code-block::
 
   "flow_logs": {
     "enabled": true,
@@ -175,3 +196,16 @@ Template::
       "..."
     ]
   }
+
+**options**
+
+=============        ========     =======                              ===========
+Key                  Required     Default                              Description
+-------------        ---------    -------                              -----------
+``enabled``          Yes          -                                    To enable/disable the Flow log creation.
+``log_group_name``   No           prefix_cluster_streamalert_flow_logs The name of the CloudWatch Log group.
+``subnets``          No           None                                 The list of AWS VPC subnet IDs to collect flow logs from.
+``vpcs``             No           None                                 The list of AWS VPC IDs to collect flow logs from.
+``enis``             No           None                                 The list of AWS ENIs to collect flow logs from.
+=============        =========    =======                            ===========
+

stream_alert_cli.py

@@ -178,7 +178,13 @@ def build_parser():
     # add subcommand options for the terraform sub-parser
     tf_parser.add_argument(
         'subcommand',
-        choices=['build', 'destroy', 'init', 'init-backend', 'generate', 'status']
+        choices=['build',
+                 'clean',
+                 'destroy',
+                 'init',
+                 'init-backend',
+                 'generate',
+                 'status']
     )
 
     tf_parser.add_argument(