Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Bug Fixes (Initialization, AWS Perms, Config), CloudWatch Event Patterns, and More #205

Merged
merged 10 commits into from
Jun 27, 2017
Merged

Bug Fixes (Initialization, AWS Perms, Config), CloudWatch Event Patterns, and More #205

Show file tree
Hide file tree
Changes from 8 commits
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
9b521a2
[conf] reset the alert/rule processor versions to LATEST
Jun 27, 2017
2ecf848
[conf] use minimums for cluster lambda/kinesis settings
Jun 27, 2017
7865f2e
[tf] support all cloudtrail tf module options + unit tests
Jun 27, 2017
9cce9d8
[conf] force_destroy enabled for prerequisite buckets
Jun 27, 2017
dc17bbe
[conf] add a terraform clean command
Jun 27, 2017
291f629
[cli] fix bug in runner init targets
Jun 27, 2017
6dadb70
[cli] add defaults and tests to flow_logs module
Jun 27, 2017
7098ac1
[docs] update AWS permissions and cluster file options
Jun 27, 2017
3c74456
[cli] error handling up and down the CLI stack
Jun 27, 2017
61d5a13
[cli] autopep8 on runner/tf generate
Jun 27, 2017
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 6 additions & 6 deletions conf/clusters/prod.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,22 +10,22 @@
"s3_bucket_suffix": "streamalert.results"
},
"streams": {
"retention": 36,
"shards": 5
"retention": 24,
"shards": 1
}
},
"kinesis_events": {
"enabled": true
},
"stream_alert": {
"alert_processor": {
"current_version": 7,
"current_version": "$LATEST",
"memory": 128,
"timeout": 25
"timeout": 10
},
"rule_processor": {
"current_version": 8,
"memory": 256,
"current_version": "$LATEST",
"memory": 128,
"timeout": 10
}
}
Expand Down
68 changes: 48 additions & 20 deletions docs/source/account.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,33 +26,61 @@ prefix

Open ``conf/global.json`` and ``conf/lambda.json`` and replace ``PREFIX_GOES_HERE`` with your company or organization name.

Administrator
~~~~~~~~~~~~~

To successfully deploy StreamAlert, you need to create an administrative user in the AWS account.

Steps:
user account
~~~~~~~~~~~~

To deploy StreamAlert, you need to create an AWS user for administration.

First, create the policy to attach to the user:

* Go to: Services => IAM => Policies
* Click: Create policy
* Select: Create your Own Policy
* Name the policy ``streamalert``, and paste the following as the ``Policy Document``:
* Clock: Create Policy

.. code-block::

{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"athena:*",
"cloudtrail:*",
"cloudwatch:*",
"ec2:*FlowLogs",
"events:*",
"firehose:*",
"iam:*",
"kinesis:*",
"kms:*",
"lambda:*",
"logs:*",
"s3:*",
"sns:*"
],
"Resource": "*"
}
]
}

Next, create the user:

* Go to: Services => IAM => Users
* Click: Add user
* Username: streamalert
* Username: ``streamalert``
* Access type: Programmatic access
* Click: Next
* Click: ``Next: Permissions``
* Select: Attach existing policies directly
* Attach the following policies::
* Attach the previously created ``streamalert`` policy
* Click: ``Next: Review``, and then ``Create user``

* AmazonKinesisFirehoseFullAccess
* AmazonKinesisFullAccess
* AmazonS3FullAccess
* AmazonSNSFullAccess
* AWSLambdaFullAccess
* CloudWatchFullAccess
* CloudWatchLogsFullAccess
* IAMFullAccess
* Click: Next (Review), and then Create User

Take the Access Key and Secret Key and export them to your environment variables::
Copy the Access Key ID and Secret Access Key and export them to your environment variables::

$ export AWS_ACCESS_KEY_ID="REPLACE_ME"
$ export AWS_SECRET_ACCESS_KEY="REPLACE_ME"
$ export AWS_DEFAULT_REGION="us-east-1"

.. note:: Remember to save your credentials in a safe place!
54 changes: 44 additions & 10 deletions docs/source/clusters.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -79,11 +79,11 @@ An example ``production`` cluster::
Customizing Clusters
~~~~~~~~~~~~~~~~~~~~

Each cluster can be broken up into multiple modules to make up a StreamAlert cluster.
Each StreamAlert cluster is made up of multiple modules.

Each module corresponds to a Terraform module found in the ``terraform/modules`` directory, and serves a specific purpose in a StreamAlert cluster.

After making modifications to a cluster's file, make sure you apply it with::
After making modifications to a cluster file, make sure you apply the changes with::

$ python stream_alert_cli.py terraform build

Expand All @@ -92,12 +92,18 @@ This will regenerate the necessary Terraform files and then apply the changes.
Module: StreamAlert
--------------------

See `Lambda Settings <lambda.html>`_ for customization options.
The main module for StreamAlert.

It creates both AWS Lambda functions, aliases, an SNS topic, IAM permissions, and more.

See `Lambda Settings <lambda.html>`_ for all customization options.

Module: Kinesis
---------------

See `Kinesis <kinesis.html>`_ for customization options.
This module contains configuration for the Kinesis Streams and Kinesis Firehose infrastructure.

See `Kinesis <kinesis.html>`_ for all customization options.

Module: CloudWatch Monitoring
-----------------------------
Expand Down Expand Up @@ -134,19 +140,32 @@ Template::
Module: CloudTrail
------------------

AWS CloudTrail is a service that enables compliance, operational auditing, and risk auditing of your AWS account.
`AWS CloudTrail <https://aws.amazon.com/cloudtrail/>`_ is a service that enables compliance, operational auditing, and risk auditing of your AWS account.

StreamAlert has native support for enabling and monitoring CloudTrail logs with the ``cloudtrail`` module.

When writing rules for CloudTrail data, use the ``cloudwatch:event`` log source.

By default, all API calls will be logged and accessible from rules.

Template::
**template**

"cloudtrail": {
"enabled": true
}
.. code-block::

"cloudtrail": {
"enabled": true
}

**options**

============= ======== ======= ===========
Key Required Default Description
------------- --------- ------- -----------
``enabled`` Yes - To enable/disable the CloudTrail.
``existing_trail`` No ``false`` Set to ``true`` if the account has an existing CloudTrail. This is to avoid duplication of data collected by multiple CloudTrails.
``is_global_trail`` No ``true`` If the CloudTrail should collect events from any region.
``event_pattern`` No ``{"account": ["<accound_id>"]}`` The CloudWatch Events pattern to send to Kinesis. `More information <http://docs.aws.amazon.com/AmazonCloudWatch/latest/events/EventTypes.html>`_.
============= ========= ======= ===========

Module: Flow Logs
-----------------
Expand All @@ -157,7 +176,9 @@ In the settings below, an arbitrary amount of subnets, vpcs, and enis can be ena

When writing rules for this data, use the ``cloudwatch:flow_logs`` log source.

Template::
**template**

.. code-block::

"flow_logs": {
"enabled": true,
Expand All @@ -175,3 +196,16 @@ Template::
"..."
]
}

**options**

============= ======== ======= ===========
Key Required Default Description
------------- --------- ------- -----------
``enabled`` Yes - To enable/disable the Flow log creation.
``log_group_name`` No prefix_cluster_streamalert_flow_logs The name of the CloudWatch Log group.
``subnets`` No None The list of AWS VPC subnet IDs to collect flow logs from.
``vpcs`` No None The list of AWS VPC IDs to collect flow logs from.
``enis`` No None The list of AWS ENIs to collect flow logs from.
============= ========= ======= ===========

8 changes: 7 additions & 1 deletion stream_alert_cli.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -178,7 +178,13 @@ def build_parser():
# add subcommand options for the terraform sub-parser
tf_parser.add_argument(
'subcommand',
choices=['build', 'destroy', 'init', 'init-backend', 'generate', 'status']
choices=['build',
'clean',
'destroy',
'init',
'init-backend',
'generate',
'status']
)

tf_parser.add_argument(
Expand Down
44 changes: 27 additions & 17 deletions stream_alert_cli/runner.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -137,10 +137,9 @@ def terraform_handler(options):
LOGGER_CLI.info('Building Initial Infrastructure')
init_targets = [
'aws_s3_bucket.lambda_source',
'aws_s3_bucket.integration_testing',
'aws_s3_bucket.terraform_state',
'aws_s3_bucket.stream_alert_secrets',
'aws_s3_bucket.logging_bucket',
'aws_s3_bucket.stream_alert_secrets',
'aws_s3_bucket.terraform_remote_state',
'aws_kms_key.stream_alert_secrets',
'aws_kms_alias.stream_alert_secrets'
]
Expand All @@ -162,6 +161,9 @@ def terraform_handler(options):
LOGGER_CLI.info('Building Remainder Infrastructure')
tf_runner()

elif options.subcommand == 'clean':
terraform_clean()

elif options.subcommand == 'destroy':
if options.target:
target = options.target
Expand All @@ -181,26 +183,34 @@ def terraform_handler(options):
sys.exit(1)

# Remove old Terraform files
LOGGER_CLI.info('Removing old Terraform files')
cleanup_files = ['{}.tf'.format(cluster) for cluster in CONFIG.clusters()]
cleanup_files.extend([
'main.tf',
'terraform.tfstate',
'terraform.tfstate.backup'
])
for tf_file in cleanup_files:
file_to_remove = 'terraform/{}'.format(tf_file)
if not os.path.isfile(file_to_remove):
continue
os.remove(file_to_remove)
# Finally, delete the Terraform directory
shutil.rmtree('terraform/.terraform/')
terraform_clean()

# get a quick status on our declared infrastructure
elif options.subcommand == 'status':
status()


def terraform_clean():
"""Remove leftover Terraform statefiles and main/cluster files"""
LOGGER_CLI.info('Cleaning Terraform files')

cleanup_files = ['{}.tf'.format(cluster) for cluster in CONFIG.clusters()]
cleanup_files.extend([
'main.tf',
'terraform.tfstate',
'terraform.tfstate.backup'
])
for tf_file in cleanup_files:
file_to_remove = 'terraform/{}'.format(tf_file)
if not os.path.isfile(file_to_remove):
continue
os.remove(file_to_remove)

# Finally, delete the Terraform directory
if os.path.isdir('terraform/.terraform/'):
shutil.rmtree('terraform/.terraform/')


def run_command(args=None, **kwargs):
"""Alias to CLI Helpers.run_command"""
return helpers.run_command(args, **kwargs)
Expand Down
42 changes: 39 additions & 3 deletions stream_alert_cli/terraform_generate.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -53,7 +53,7 @@ def generate_s3_bucket(**kwargs):
'target_bucket': logging_bucket,
'target_prefix': '{}/'.format(bucket_name)
}
force_destroy = kwargs.get('force_destroy', False)
force_destroy = kwargs.get('force_destroy', True)
versioning = kwargs.get('versioning', True)
lifecycle_rule = kwargs.get('lifecycle_rule')

Expand Down Expand Up @@ -352,6 +352,32 @@ def generate_cloudtrail(cluster_name, cluster_dict, config):
"""
modules = config['clusters'][cluster_name]['modules']
cloudtrail_enabled = bool(modules['cloudtrail']['enabled'])
existing_trail_default = False
existing_trail = modules['cloudtrail'].get('existing_trail', existing_trail_default)
is_global_trail_default = True
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe I'm just not seeing it elsewhere, but it looks like existing_trail_default and is_global_trail_default are just booleans and only used to hold default values for the .get() calls. Can we just place the boolean value directly in the get for brevity?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Correct, they're just booleans. I defined them as variables to be really explicit about what the defaults are for these options.

is_global_trail = modules['cloudtrail'].get('is_global_trail', is_global_trail_default)
event_pattern_default = {
'account': [config['global']['account']['aws_account_id']]
}
event_pattern = modules['cloudtrail'].get('event_pattern', event_pattern_default)

# From here:
# http://docs.aws.amazon.com/AmazonCloudWatch/latest/events/CloudWatchEventsandEventPatterns.html
valid_event_pattern_keys = {
'version',
'id',
'detail-type',
'source',
'account',
'time',
'region',
'resources',
'detail'
}
if not set(event_pattern.keys()).issubset(valid_event_pattern_keys):
LOGGER_CLI.error('Invalid CloudWatch Event Pattern!')
sys.exit(1)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should probably start returning a value in places like this (maybe just a simple bool == False, or a more comprehensive solution would be returning a non-zero error number) that we could check up the call stack and exit on error. It would avoid having sys.exit(1) calls scattered about the CLI and could allow for us to start using useful exit codes for certain errors (that are not 1). Not a necessary change now, but something to keep in mind.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed, I'll make some changes


cluster_dict['module']['cloudtrail_{}'.format(cluster_name)] = {
'account_id': config['global']['account']['aws_account_id'],
'cluster': cluster_name,
Expand All @@ -360,7 +386,11 @@ def generate_cloudtrail(cluster_name, cluster_dict, config):
'enable_logging': cloudtrail_enabled,
'source': 'modules/tf_stream_alert_cloudtrail',
's3_logging_bucket': '{}.streamalert.s3-logging'.format(
config['global']['account']['prefix'])}
config['global']['account']['prefix']),
'existing_trail': existing_trail,
'is_global_trail': is_global_trail,
'event_pattern': json.dumps(event_pattern)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why do we want event_pattern to be a json string within a dictionary as opposed to just another python dict? I assume this has something to do with the config stuff below?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

https://www.terraform.io/docs/providers/aws/r/cloudwatch_event_rule.html

}


def generate_flow_logs(cluster_name, cluster_dict, config):
Expand All @@ -373,11 +403,17 @@ def generate_flow_logs(cluster_name, cluster_dict, config):
config [dict]: The loaded config from the 'conf/' directory
"""
modules = config['clusters'][cluster_name]['modules']
flow_log_group_name_default = '{}_{}_streamalert_flow_logs'.format(
config['global']['account']['prefix'],
cluster_name
)
flow_log_group_name = modules['flow_logs'].get('log_group_name', flow_log_group_name_default)

if modules['flow_logs']['enabled']:
cluster_dict['module']['flow_logs_{}'.format(cluster_name)] = {
'source': 'modules/tf_stream_alert_flow_logs',
'destination_stream_arn': '${{module.kinesis_{}.arn}}'.format(cluster_name),
'flow_log_group_name': modules['flow_logs']['log_group_name']}
'flow_log_group_name': flow_log_group_name}
for flow_log_input in ('vpcs', 'subnets', 'enis'):
input_data = modules['flow_logs'].get(flow_log_input)
if input_data:
Expand Down
15 changes: 4 additions & 11 deletions terraform/modules/tf_stream_alert_cloudtrail/main.tf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -79,17 +79,10 @@ data "aws_iam_policy_document" "cloudtrail_bucket" {

// Cloudwatch event to capture Cloudtrail API calls
resource "aws_cloudwatch_event_rule" "all_events" {
name = "${var.prefix}_${var.cluster}_streamalert_all_events"
description = "Capture all CloudWatch events"
role_arn = "${aws_iam_role.streamalert_cloudwatch_role.arn}"

event_pattern = <<PATTERN
{
"detail-type": [
"AWS API Call via CloudTrail"
]
}
PATTERN
name = "${var.prefix}_${var.cluster}_streamalert_all_events"
description = "Capture all CloudWatch events"
role_arn = "${aws_iam_role.streamalert_cloudwatch_role.arn}"
event_pattern = "${var.event_pattern}"
}

// The Kinesis destination for Cloudwatch events
Expand Down
Loading