Unable to create Kinesis Analytics application: InvalidArgumentException

[zentavr]

Terraform Version

Terraform v0.11.11
+ provider.aws v1.54.0

Affected Resource(s)

• aws_kinesis_analytics_application

Terraform Configuration Files

##
##  SDK Health Analytics Application
##
resource "aws_iam_role" "ks_app_sdk_health_analytics" {
  name = "kinesisapp-${var.environment}-sdk-health-analytics-role"
  path = "/service-role/"
  assume_role_policy = "${data.aws_iam_policy_document.ks_app_sdk_health_analytics_sts_policy_doc.json}"
  description = "Regulates the permissions for ${var.environment}-sdk-health-analytics application stream"

  tags = {
    Name = "kinesisapp-${var.environment}-sdk-health-analytics-role"
    Environment = "${var.environment}"
    Origin = "terraform"
  }
}

data "aws_iam_policy_document" "ks_app_sdk_health_analytics_sts_policy_doc" {
  statement {
    effect = "Allow"

    actions = ["sts:AssumeRole"]

    principals {
      identifiers = ["kinesisanalytics.amazonaws.com"]
      type = "Service"
    }
  }
}

resource "aws_iam_policy" "app_sdk_health_analytics_policy" {
  name = "kinesisapp-${var.environment}-sdk-health-analytics-policy"
  path = "/service-role/"
  description = "Manages the Kinesis Data Analytics application permissions for ${var.environment}-sdk-health-analytics"
  policy = "${data.aws_iam_policy_document.ks_app_sdk_health_analytics_permissions_policy_doc.json}"
}

data "aws_iam_policy_document" "ks_app_sdk_health_analytics_permissions_policy_doc" {
  statement {
    sid = "ReadInputKinesis"

    effect = "Allow"

    actions = [
      "kinesis:GetShardIterator",
      "kinesis:GetRecords",
      "kinesis:DescribeStream"
    ]

    resources = [
      "${aws_kinesis_stream.ks_core_api_stream.arn}"
    ]
  }

  statement {
    sid = "UseLambdaFunction"

    effect = "Allow"

    actions = [
      "lambda:InvokeFunction",
      "lambda:GetFunctionConfiguration"
    ]

    resources = [
      "${module.lambda_sdk_health_to_sqs.arn}"
    ]
  }
}

resource "aws_iam_role_policy_attachment" "app_sdk_health_analytics_policy_attach" {
  role = "${aws_iam_role.ks_app_sdk_health_analytics.name}"
  policy_arn = "${aws_iam_policy.app_sdk_health_analytics_policy.arn}"
}

resource "aws_kinesis_analytics_application" "ks_app_sdk_health_analytics" {
  name = "${var.environment}-sdk-health-analytics"

  depends_on = [
    "aws_iam_role_policy_attachment.app_sdk_health_analytics_policy_attach"
  ]

  inputs {
    name_prefix = "SOURCE_SQL_STREAM"

    kinesis_stream {
      resource_arn = "${aws_kinesis_stream.ks_core_api_stream.arn}"
      role_arn = "${aws_iam_role.ks_app_sdk_health_analytics.arn}"
    }

    parallelism {
      count = 1
    }

    schema {
      # 1
      record_columns {
        mapping  = "$.recorded_at"
        name     = "recorded_at"
        sql_type = "TIMESTAMP"
      }

      # 2
      record_columns {
        mapping  = "$.type"
        name     = "type0"
        sql_type = "VARCHAR(16)"
      }

      # 3
      record_columns {
        mapping  = "$.account_id"
        name     = "account_id"
        sql_type = "VARCHAR(64)"
      }

      # 4
      record_columns {
        mapping  = "$.device_id"
        name     = "device_id"
        sql_type = "VARCHAR(64)"
      }

      # 5
      record_columns {
        mapping  = "$.data.value"
        name     = "value0"
        sql_type = "VARCHAR(64)"
      }

      record_encoding = "UTF-8"

      record_format {
        mapping_parameters {
          json {
            record_row_path = "$"
          }
        }
      }
    }
  }

  outputs {
    name = "IN_APP_STREAM"

    schema {
      record_format_type = "JSON"
    }

    lambda {
      resource_arn = "${module.lambda_sdk_health_to_sqs.arn}"
      role_arn = "${aws_iam_role.ks_app_sdk_health_analytics.arn}"
    }
  }
}

Debug Output

* aws_kinesis_analytics_application.ks_app_sdk_health_analytics: 1 error(s) occurred:

* aws_kinesis_analytics_application.ks_app_sdk_health_analytics: Unable to create Kinesis Analytics application: InvalidArgumentException: Given IAM role arn : arn:aws:iam::203867187697:role/service-role/kinesisapp-dev-sdk-health-analytics-role does not provide Invoke permissions on the Lambda resource : arn:aws:lambda:us-west-2:203867187697:function:dev_sdk_health_to_sqs
	status code: 400, request id: 9d45d802-1032-11e9-a999-874cf3c8ef5d

--->

Expected Behavior

Kinesis application should be created, because I specified

  depends_on = [
    "aws_iam_role_policy_attachment.app_sdk_health_analytics_policy_attach"
  ]

...when creating aws_kinesis_analytics_application.ks_app_sdk_health_analytics resource

Actual Behavior

Creation of Kinesis Application gets failed after the first execution. When I apply the same scripts second time - everyting gets created.

Important Factoids

I have my environment in the VPC

[bflad]

Hi @zentavr 👋 Thanks for the detailed report about this and apologies for the unexpected behavior.

When working with IAM resources across other AWS services, eventual consistency within AWS can cause issues immediately after changes in IAM even with proper Terraform resource ordering. Each downstream AWS service handles these types of IAM permissions errors differently so we have to add custom logic for each case, but generally we just need to retry on the error (like the one you provided) for a reasonable period of time (up to 1 or 2 minutes). We currently handle one slightly different error case already in the resource:

https://github.com/terraform-providers/terraform-provider-aws/blob/d8a32cd7caaddb6430c81aef5effc7bdfa3da61a/aws/resource_aws_kinesis_analytics_application.go#L578-L588

I also notice that our acceptance testing does not include an output Lambda function like yours above so hopefully adding a similar test configuration will trigger the error you see so we can ensure its properly fixed and regressions are prevented in the future. I have submitted #7039 to add the testing and retries.

[zentavr]

Hello @bflad ,

Thank you for the reply and all the explanations.
I wonder in which version of the plugin the fix would be public?

We are on the development stage right now and it’s not very critical, but would be a problem a little bit later :)

[bflad]

The fix for this should be merged and will release with version 1.55.0 of the Terraform AWS provider, likely middle of this week.

[bflad]

This has been released in version 1.55.0 of the AWS provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

[zentavr]

@bflad - I will test and let you know.

[zentavr]

@bflad - Thank you, it works! :)

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thanks!