Skip to content

CloudWatch Events Firehose Method

Jump to bottom
Paul Reeves edited this page Dec 9, 2024 · 5 revisions

CloudWatch Events Firehose Method

CloudWatch Events Firehose Method Overview

Some AWS services generate events that can be retrieved with Amazon CloudWatch Events, sent onto other destinations such as Amazon Data Firehose, then sent onto Splunk from Firehose over the HTTP Event Collector (HEC). In the event that the Firehose cannot reach Splunk, events are sent to an S3 backsplash bucket for later retrieval.

Visual overview:

Mermaid code of visual overview:

graph TB;
	source[Event Source]
	cwEvents[Cloud Watch Events]
	adf[Amazon Data Firehose]
	splunk[Splunk]
	s3Backsplash[(Firehose Backsplash Bucket)]
	source-->cwEvents
	cwEvents-->adf
	adf-->|HEC|splunk
	adf-->|On failure sending to Splunk|s3Backsplash

Splunk Dashboard

There is code for a Splunk Dashboard Studio dashboard located in https://github.com/splunk/splunk-aws-gdi-toolkit/blob/main/CloudWatchEvents-Firehose-Resources/splunkDashboard.json that can be used to check the health of the different events coming in through this method.

Deployment Instructions

Instructions to deploy this ingestion method depend on what type of data (data sourcetype essentially) you are looking to send to Splunk.

GuardDuty Deployment Instructions

These instructions are for configuring Amazon GuardDuty logs to be sent to Splunk. GuardDuty is a service that watches resources in your AWS Account for potentially malicious activity, and generates findings based on that potentially malicious activity. There are a number of data sources GuardDuty uses to identify this activity, and the list is growing as time goes on. GuardDuty is not a replacement for a full SIEM like Splunk Enterprise Security (ES), but instead can feed into ES to help generate notables.

  1. Install the Splunk Add-on for Amazon Web Services (AWS).
    • If you use Splunk Cloud, install the add-on on the ad-hoc search head or ad-hoc search head cluster.
    • If you do not use Splunk Cloud, install this add-on on the HEC endpoint (probably either the indexer(s) or heavy forwarder), your indexer(s), and your search head(s).
  2. Configure any firewall rules in front of Splunk to receive data from Amazon Data Firehose.
    • Reference the AWS documentation for the IP ranges required. Make sure to add the IP ranges from the region you'll be deploying the CloudFormation to.
    • If you use Splunk Cloud, you'll want to add the relevant IP range(s) to the HEC feature in the IP allowlist.
    • If you do not use Splunk Cloud, you'll need to consult with your Splunk Architect, Splunk Admin, and/or network team to determine which firewall rules to change and where.
  3. Create a HEC token with Indexer acknowledgment turned on, in Splunk to ingest the events, with these specific instructions:
    • Make sure to enable indexer acknowledgment.
    • Set the source name override to aws_cloudwatchevents_guardduty.
    • Set the sourcetype to aws:cloudwatch:guardduty.
    • Select the index you want the data to be sent to.
    • AWS Kinesis Data Firehose does check the format of the tokens, so we recommend letting Splunk generate this rather than setting it manually through inputs.conf.
    • If you use Splunk Cloud, follow these instructions
    • If you do not use Splunk Cloud, the HEC token will need to be created on the Splunk instance that will be receiving this data (probably either the indexer(s) or a heavy forwarder). Instructions for this can be found here.
  4. Configure GuardDuty by following the documented instructions from AWS.
  5. Deploy the CloudWatchEvents-Firehose-Resources/sourceToSplunk.yml CloudFormation Template in the administrator account used in step 4. This will create the necessary resources to pick up the GuardDuty events and send them to Splunk. Specifically for GuardDuty events to be sent to Splunk, these parameters need to be changed from the default values:
    • logType: guardduty
    • splunkHECEndpoint: https://{{url}}:{{port}}
      • For Splunk Cloud, this will be https://http-inputs-firehose-{{stackName}}.splunkcloud.com:443, where {{stackName}} is the name of your Splunk Cloud stack.
      • For non-Splunk Cloud deployments, consult with your Splunk Architect or Splunk Admin.
    • splunkHECToken: The value of the HEC token from step 3
    • stage: If this is going into production, set this to something like prod
  6. Verify the data is being ingested. The easiest way to do this is to wait a few minutes, then run a search like index={{splunkIndex}} sourcetype=aws:cloudwatch:guardduty | head 100, where {{splunkIndex}} is the destination index selected in step 3.
  • Deploy sourceToSplunk.yml:
aws cloudformation create-stack --region us-west-2 --stack-name guardDutyFindings-to-splunk --capabilities CAPABILITY_NAMED_IAM --template-body file://sourceToSplunk.yml --parameters ParameterKey=logType,ParameterValue=guardduty ParameterKey=splunkHECEndpoint,ParameterValue=https://http-inputs-firehose-contoso.splunkcloud.com:443 ParameterKey=splunkHECToken,ParameterValue=01234567-89ab-cdef-0123-456789abcdef ParameterKey=stage,ParameterValue=prod ParameterKey=cloudWatchAlertEmail,ParameterValue=jsmith@contoso.com ParameterKey=contact,ParameterValue=jsmith@contoso.com

Security Hub Deployment Instructions

These instructions are for configuring AWS Security Hub logs to be sent to Splunk. Security Hub is a service that aggregates security findings across multiple built-in services to AWS (at the time of writing this includes Amazon GuardDuty, Amazon Inspector, AWS Firewall Manager, IAM Access Analyzer, and Amazon Macie) , and third-party services. Security Hub is not a replacement for a full SIEM like Splunk Enterprise Security (ES), but instead can feed into ES to help generate notables.

  1. Install the Splunk Add-on for Amazon Web Services (AWS).
    • If you use Splunk Cloud, install the add-on on the ad-hoc search head or ad-hoc search head cluster.
    • If you do not use Splunk Cloud, install this add-on on the HEC endpoint (probably either the indexer(s) or heavy forwarder), your indexer(s), and your search head(s).
  2. Configure any firewall rules in front of Splunk to receive data from Amazon Data Firehose.
    • Reference the AWS documentation for the IP ranges required. Make sure to add the IP ranges from the region you'll be deploying the CloudFormation to.
    • If you use Splunk Cloud, you'll want to add the relevant IP range(s) to the HEC feature in the IP allowlist.
    • If you do not use Splunk Cloud, you'll need to consult with your Splunk Architect, Splunk Admin, and/or network team to determine which firewall rules to change and where.
  3. Create a HEC token with Indexer acknowledgment turned on, in Splunk to ingest the events, with these specific instructions:
    • Make sure to enable indexer acknowledgment.
    • Set the source name override to aws_cloudwatchevents_securityhub.
    • Leave the sourcetype to aws:securityhub:finding.
    • Select the index(es) you want the data to be sent to.
    • AWS Kinesis Data Firehose does check the format of the tokens, so we recommend letting Splunk generate this rather than setting it manually through inputs.conf.
    • If you use Splunk Cloud, follow these instructions
    • If you do not use Splunk Cloud, the HEC token will need to be created on the Splunk instance that will be receiving this data (probably either the indexer(s) or a heavy forwarder). Instructions for this can be found here.
  4. Configure Security Hub by following the documented instructions from AWS.
  5. Deploy the CloudWatchEvents-Firehose-Resources/sourceToSplunk.yml CloudFormation Template in the administrator account used in step 4. This will create the necessary resources to pick up the GuardDuty events and send them to Splunk. Specifically for GuardDuty events to be sent to Splunk, these parameters need to be changed from the default values:
    • logType: securityhub
    • splunkHECEndpoint: https://{{url}}:{{port}}
      • For Splunk Cloud, this will be https://http-inputs-firehose-{{stackName}}.splunkcloud.com:443, where {{stackName}} is the name of your Splunk Cloud stack.
      • For non-Splunk Cloud deployments, consult with your Splunk Architect or Splunk Admin.
    • splunkHECToken: The value of the HEC token from step 3
    • stage: If this is going into production, set this to something like prod
  6. Verify the data is being ingested. The easiest way to do this is to wait a few minutes, then run a search like index={{splunkIndex}} sourcetype=aws:securityhub:finding | head 100, where {{splunkIndex}} is the destination index selected in step 3.

Example Security Hub Deployment via AWS CLI:

  • Deploy sourceToSplunk.yml:
aws cloudformation create-stack --region us-west-2 --stack-name securityHubFindings-to-splunk --capabilities CAPABILITY_NAMED_IAM --template-body file://sourceToSplunk.yml --parameters ParameterKey=logType,ParameterValue=securityhub ParameterKey=splunkHECEndpoint,ParameterValue=https://http-inputs-firehose-contoso.splunkcloud.com:443 ParameterKey=splunkHECToken,ParameterValue=01234567-89ab-cdef-0123-456789abcdef ParameterKey=stage,ParameterValue=prod ParameterKey=cloudWatchAlertEmail,ParameterValue=jsmith@contoso.com ParameterKey=contact,ParameterValue=jsmith@contoso.com

Uninstallation Instructions

  1. On each log source (GuardDuty or Security Hub), unconfigure the logging on the AWS side.
  2. Empty the bucket named {{accountId}}-{{region}}-{{logType}}-bucket of all files.
    • The logType is guardduty if you deployed using the GuardDuty instructions.
    • The logType is securityhub if you deployed using the Security Hub instructions.
  3. Delete the CloudFormation stack that deployed the resources in sourceToSplunk.yml.
  4. Delete the HEC token that was deployed to Splunk to ingest the data.
  5. Remove any firewall exceptions that were created for Amazon Data Firehose to send to Splunk over HEC.

FAQ

  • What if I want to send data to Edge Processor? The recommended way to get the data to Edge Processor is to set up the architecture defined in the EP SVA, with Firehose acting as the HEC source. You will need to configure a signed certificate on the load balancer since Amazon Data Firehose requires the destination it's sending to to have a signed certificate and appropriate DNS entry. The DNS Round Robin architecture could also be used.
  • What if I want to send data to Ingest Processor? To get data to Ingest Processor, first send it to the Splunk Cloud environment like normal, then when creating the pipeline to interact with the data specify a partition that will apply the data being sent from the Splunk AWS GDI Toolkit. For more information on how to do this, refer to the Ingest Processor documentation.

Troubleshooting

  • The CloudFormation template fails to deploy when I try to deploy it..
    • Verify that the role you are using to deploy the CloudFormation template has the appropriate permissions to deploy the resources you're trying to deploy.
    • Verify that the parameters are set correctly on the stack.
    • Also, in the CloudFormation console, check the events on the failed stack for hints to where it failed.
    • Also see the official Troubleshooting CloudFormation documentation.
  • Events aren't getting to Splunk. What should I check? In this order, check the following:
    1. That the CloudFormation template deployed without error.
    2. That the parameters (especially splunkHECEndpoint and splunkHECToken) are correct.
    3. That the event source (ex GuardDuty or Security Hub) are generating events and sending to CloudWatch Events.
    4. That there are no errors related to ingestion in Splunk.
    5. That any firewall ports are open from Firehose to Splunk.
  • Not all of the events are getting to Splunk or Amazon Data Firehose is being throttled.
    • Amazon Data Firehose has a number of quotas associated with each Firehose. You can check whether you're being throttled by navigating to the monitoring tab in the AWS console for the Firehose and checking if the "Throttled records (Count)" value is greater than zero. If the Firehose is being throttled, you can use the Kinesis Firehose Service quota increase form to request that quota be increased.
Clone this wiki locally