processing_configuration not available for AWS Splunk Firehose Delivery Stream

[cwiggs]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Description

Should be able to define lambda transformer for Splunk destination the same way a processor can be defined for extended s3 destinations.

New or Affected Resource(s)

• aws_kinesis_firehose_delivery_stream

Potential Terraform Configuration

resource "aws_kinesis_firehose_delivery_stream" "vpc_flow_logs" {
  name = "splunk_ingrestion_stream"
  destination = "splunk"

  s3_configuration {
    role_arn = "firehose-iam-role"
    bucket_arn = "arn:aws:s3:::bucket"
    prefix = "logs/logs"
    buffer_size = 10
    buffer_interval = 400
    compression_format = "GZIP"
  }

  splunk_configuration {
    hec_endpoint = "${var.hec_endpoint}"
    hec_token = "${var.hec_token}"
    hec_acknowledgment_timeout = 600
    hec_endpoint_type = "Event"
    s3_backup_mode = "FailedEventsOnly"
  }

  processing_configuration = [
    {
      enabled = "true"

      processors = [
        {
          type = "Lambda"

          parameters = [
            {
              parameter_name  = "LambdaArn"
              parameter_value = "${aws_lambda_function.transform.arn}:$LATEST"
            },
          ]
        },
      ]
    },
  ]

References

Same issue for elasticsearch: #2729

[bflad]

Hi @cwiggs 👋 Sorry you're running into trouble here.

I am presuming this issue is a continuation of #3944 (comment), on the merged pull request that added the functionality requested here.

In that comment you mention:

I'm using the aws provider version 1.25 but i'm getting this error when I try to use the "processing_configuration" option under "splunk_configuration"

Error: module.kinesis-firehose.aws_kinesis_firehose_delivery_stream.vpc_flow_logs: : invalid or unknown key: processing_configuration

There's two possibilities that we can easily check:

• Double check terraform -v to ensure your AWS provider is at least 1.14.0 as expected
• Ensure that your processing_configuration is nested under splunk_configuration -- the configuration example above has it outside splunk_configuration

We explicitly acceptance test this functionality daily without issue, using this test configuration: https://github.com/terraform-providers/terraform-provider-aws/blob/867b227ca892f5d609cf721651a92e20e439b45c/aws/resource_aws_kinesis_firehose_delivery_stream_test.go#L1751-L1798

Please let us know, thanks.

[cwiggs]

Hello @bflad!

moving the processing_configuration into the splunk_configuration block fixed the issue. I was a bit confused because the docs here don't show "processing_configuration" as an option under "splunk_configuration"

[bflad]

You are correct! Let me submit a fix to the documentation. Here we go: #5102

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thanks!