Cannot update lambda EventSourceMapping after creation

[Jimmy89]

What happened?

If I create a EventSourceMapping for a AWS Lambda function and I want to update the resource, I get the error

 aws-native:lambda:EventSourceMapping (event-source):
    error: operation error CloudControl: UpdateResource, https response error StatusCode: 400, RequestID: XXXXXX, api error ValidationException: Model validation failed (#/Topics: expected minimum item count: 1, found: 0
    #/Queues: expected minimum item count: 1, found: 0)

Also after running pulumi refresh I keep getting it.

Expected Behavior

Being able to update an event source map.

Steps to reproduce

Stack code example:

import * as pulumi from "@pulumi/pulumi";
import * as aws from "@pulumi/aws-native";
import * as awsClassic from "@pulumi/aws";

const dynamodbTable = new aws.dynamodb.Table(`dynamodb`, {
    attributeDefinitions: [
        {
            attributeName: "PK",
            attributeType: "S",
        },
        {
            attributeName: "SK",
            attributeType: "S",
        },
    ],
    keySchema: [
        {
            attributeName: "PK",
            keyType: "HASH",
        },
        {
            attributeName: "SK",
            keyType: "RANGE",
        },
    ],
    streamSpecification: {
        streamViewType: "NEW_AND_OLD_IMAGES"
    },
    tableName: "TABLE_NAME",
    billingMode: "PAY_PER_REQUEST",
});

const policyDynamodb = new awsClassic.iam.Policy(`dynamodb-table`, {
    description: "Allow adding and changing dynamodb items",
    policy: awsClassic.iam.getPolicyDocumentOutput({
        statements: [{
            actions: [
                "dynamodb:UpdateItem",
                "dynamodb:GetItem",
                "dynamodb:Query",
                "dynamodb:ConditionCheckItem",
                "dynamodb:BatchGetItem",
                "dynamodb:BatchWriteItem",
                "dynamodb:DeleteItem",
                "dynamodb:PutItem",
            ],
            resources: [dynamodbTable.arn],
            effect: "Allow",
        }],
    }).json,
});

const policyDynamodbStream = new awsClassic.iam.Policy(`dynamodb-table-stream`, {
    description: "Allow reading from the dynamodb table stream",
    policy: awsClassic.iam.getPolicyDocumentOutput({
        statements: [{
            actions: [
                "dynamodb:DescribeStream",
                "dynamodb:GetRecords",
                "dynamodb:GetShardIterator",
                "dynamodb:ListStreams"
            ],
            resources: [dynamodbTable.streamArn],
            effect: "Allow",
        }],
    }).json,
});

const lambdaRole = new awsClassic.iam.Role(`lambda-role`, {
    assumeRolePolicy: awsClassic.iam.getPolicyDocumentOutput({
        statements: [{
            actions: ["sts:AssumeRole"],
            principals: [{
                identifiers: ["lambda.amazonaws.com"],
                type: "Service",
            }],
            effect: "Allow",
        }],
    }).json,
    managedPolicyArns: [
        awsClassic.iam.ManagedPolicy.AWSLambdaBasicExecutionRole,
        awsClassic.iam.ManagedPolicy.AWSXRayDaemonWriteAccess,
        awsClassic.iam.ManagedPolicy.CloudWatchLambdaInsightsExecutionRolePolicy
        policyDynamodb.arn,
        policyDynamodbStream.arn,
    ],
});

const lambda = new awsClassic.lambda.Function('lambda', {
    code: new pulumi.asset.AssetArchive({
        ".": new pulumi.asset.FileArchive("./FOLDER"),
    }),
    runtime: "nodejs18.x",
    handler: "index.handler",
    architectures: ["x86_64"],
    role: lambdaRole.arn,
    timeout: 5,
    memorySize: 8000,
});

// This is the function what it's all about
new aws.lambda.EventSourceMapping("event-source", {
  functionName: lambda.arn,
  startingPosition: "LATEST",
  batchSize: 1,
  enabled: true,
  eventSourceArn: dynamodbTable.streamArn,
  filterCriteria: {
    filters: [
      { pattern: JSON.stringify({
          dynamodb: {
            NewImage: {
              type: { S: ["select"] }
            }
          }
        })
      },
    ]
  }
});

Run the above stack (sorry if I overlooked any missing variable). After being upped remove the filterCriteria key from the EventSourceMapping and up again (not specific for this key, but that was my update).

You should get the error:

 aws-native:lambda:EventSourceMapping (event-source):
    error: operation error CloudControl: UpdateResource, https response error StatusCode: 400, RequestID: XXXXXX, api error ValidationException: Model validation failed (#/Topics: expected minimum item count: 1, found: 0
    #/Queues: expected minimum item count: 1, found: 0)

Output of pulumi about

Dependencies:
NAME                VERSION
@pulumi/aws         5.42.0
@pulumi/aws-native  0.73.0
@pulumi/pulumi      3.78.1

Additional context

I found out that the requested items by AWS are for Kafka and MQ clusters, which is not related to dynamodb streams at all.

Contributing

Vote on this issue by adding a 👍 reaction.
To contribute a fix for this issue, leave a comment (and link to your pull request, if you've opened one already).

[rquitales]

Hello @Jimmy89,

We appreciate you providing the comprehensive details and the complete code, which facilitated quick reproduction of the issue. After attempting to replicate the problem myself, I have indeed encountered the error you described. The error seems to be originating from validation within the underlying AWS CloudControl API. This issue might potentially be a bug on their end.

As you accurately identified, the queue and topics fields are related to Kafka or MQ clusters and should be irrelevant to the changes you're attempting. It appears that there might have been a oversight in their validation check, causing the queue and topics fields to be erroneously validated for non-Kafka or non-MQ streams.

Since this bug seems to affect EventSourceMappings that are being updated in-place, a possible workaround is to always replace the eventsourcemapping if changes are required. Here's an example of how you might implement this workaround:

new aws.lambda.EventSourceMapping("event-source", {
  functionName: lambda.arn,
  startingPosition: "LATEST",
  batchSize: 1,
  enabled: true,
  eventSourceArn: dynamodbTable.streamArn,
  // ... other configuration ...
}, { replaceOnChanges: ["*"] });

Thank you for your contribution and patience in helping us identify this issue. I will discuss this matter with my team to determine the next steps and how to alert the AWS team about this issue in their CloudControl API. Your feedback is invaluable as we work towards a resolution. If you have any further questions or insights, feel free to share.