AssertionError in lambda.EventSourceMapping for version 6.13

[chrispyduck]

What happened?

I'm testing an upgrade of the python pulumi-aws from 5.10 to 6.13 and i'm seeing an AssertionError: Unexpected type. Expected 'list' got '<class 'str'>' that I believe I've narrowed down to uses of aws.lambda_.EventSourceMapping resources (by moving a return statement around to find the resource that causes it to fail), e.g.:

aws.lambda_.EventSourceMapping(
    "queue",
    event_source_arn=queue.arn,
    function_name=lambda_.arn,
    opts=opts,
)

I've verified that queue.arn and lambda_.arn are strings and that the documentation and SDK code both list these parameters as str not list, and this code was working as-is before the upgrade, so I suspect this is a bug.

Example

I'm trying to create a minimal project to reproduce this issue but I haven't been able to, yet. I'll update once I figure it out. For now all I can report is that every occurrence of EventSourceMapping has this problem in my (private) repo.

Output of pulumi about

CLI          
Version      3.94.0
Go Version   go1.21.3
Go Compiler  gc

Plugins
NAME        VERSION
aws         6.13.0
aws-native  0.90.0
datadog     4.23.0
python      unknown

Host     
OS       darwin
Version  14.1.1
Arch     arm64

This project is written in python: executable='./env/bin/python' version='3.9.16'

Backend        
Name           pulumi.com
URL            *removed*
User           *removed*
Organizations  *removed*
Token type     personal

Dependencies:
NAME                     VERSION
blinker                  1.7.0
boltons                  20.2.1
boto3                    1.28.17
datadog-api-client       2.19.0
dnspython                2.4.2
Flask                    2.2.3
hiredis                  2.2.3
inflection               0.5.1
launchdarkly-server-sdk  7.6.1
lxml                     4.9.3
orjson                   3.9.10
pip                      22.3.1
pulumi-aws               6.13.0
pulumi-aws-native        0.90.0
pulumi-datadog           4.23.0
pydantic                 2.5.2
redis                    4.6.0
regret                   0.45.2
retry                    0.9.2
rich                     13.7.0
sentry-sdk               1.38.0
setuptools               69.0.2
tomli                    2.0.1
tomlkit                  0.12.3

Additional context

I was hoping to work around this using aws-native's version of this resource but I ran into this instead: pulumi/pulumi-aws-native#1038. This is preventing me from upgrading

Contributing

Vote on this issue by adding a 👍 reaction.
To contribute a fix for this issue, leave a comment (and link to your pull request, if you've opened one already).

[chrispyduck]

Per @t0yv0 this may be related to #3075

[VenelinMartinov]

Thank you for reporting @chrispyduck, looks like the issue in the native provider has a workaround - pulumi/pulumi-aws-native#1038 (comment) - did you try that? It might be the quickest way to unblocking your use case.

[t0yv0]

Getting another report of this stack trace from a customer. Not yet certain which resource.

[VenelinMartinov]

Got a repro:

import pulumi
import pulumi_aws as aws
import pulumi_archive as archive

assume_role = aws.iam.get_policy_document(
    statements=[
        aws.iam.GetPolicyDocumentStatementArgs(
            effect="Allow",
            principals=[
                aws.iam.GetPolicyDocumentStatementPrincipalArgs(
                    type="Service",
                    identifiers=["lambda.amazonaws.com"],
                )
            ],
            actions=["sts:AssumeRole"],
        )
    ]
)
iam_for_lambda = aws.iam.Role(
    "iamForLambda",
    assume_role_policy=assume_role.json,
    managed_policy_arns=["arn:aws:iam::aws:policy/AmazonSQSFullAccess"],
)
lambda_ = archive.get_file(
    type="zip", source_file="lambda.js", output_path="lambda_function_payload.zip"
)
test_lambda = aws.lambda_.Function(
    "testLambda",
    code=pulumi.FileArchive("lambda_function_payload.zip"),
    role=iam_for_lambda.arn,
    handler="index.test",
    runtime="nodejs18.x",
)

queue = aws.sqs.Queue("queue")


esm = aws.lambda_.EventSourceMapping(
    "example", event_source_arn=queue.arn, function_name=test_lambda.arn
)

steps:

1. pulumi up with pulumi-aws version 5.42.0.
2. pulumi up with pulumi-aws version 6.13.1.
3. Observe failure:

  pulumi:pulumi:Stack (aws_esm_py-dev):
    error: Program failed with an unhandled exception:
    Traceback (most recent call last):
      File "/opt/homebrew/bin/pulumi-language-python-exec", line 197, in <module>
        loop.run_until_complete(coro)
      File "/Users/vvm/.pyenv/versions/3.11.6/lib/python3.11/asyncio/base_events.py", line 653, in run_until_complete
        return future.result()
               ^^^^^^^^^^^^^^^
      File "/Users/vvm/code/programs/aws_esm_py/venv/lib/python3.11/site-packages/pulumi/runtime/stack.py", line 137, in run_in_stack
        await run_pulumi_func(lambda: Stack(func))
      File "/Users/vvm/code/programs/aws_esm_py/venv/lib/python3.11/site-packages/pulumi/runtime/stack.py", line 51, in run_pulumi_func
        await wait_for_rpcs()
      File "/Users/vvm/code/programs/aws_esm_py/venv/lib/python3.11/site-packages/pulumi/runtime/stack.py", line 83, in wait_for_rpcs
        raise exn from cause
      File "/Users/vvm/code/programs/aws_esm_py/venv/lib/python3.11/site-packages/pulumi/runtime/rpc_manager.py", line 71, in rpc_wrapper
        result = await rpc
                 ^^^^^^^^^
      File "/Users/vvm/code/programs/aws_esm_py/venv/lib/python3.11/site-packages/pulumi/runtime/resource.py", line 1029, in do_register
        rpc.resolve_outputs(
      File "/Users/vvm/code/programs/aws_esm_py/venv/lib/python3.11/site-packages/pulumi/runtime/rpc.py", line 1122, in resolve_outputs
        translated_value = translate_output_properties(
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/Users/vvm/code/programs/aws_esm_py/venv/lib/python3.11/site-packages/pulumi/runtime/rpc.py", line 1001, in translate_output_properties
        element_type = _get_list_element_type(typ)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/Users/vvm/code/programs/aws_esm_py/venv/lib/python3.11/site-packages/pulumi/runtime/rpc.py", line 166, in _get_list_element_type
        raise AssertionError(f"Unexpected type. Expected 'list' got '{typ}'")
    AssertionError: Unexpected type. Expected 'list' got '<class 'str'>'

Same code works fine if pulumi up-d with 6.13.1 from the start, so likely some migration thing.

[VenelinMartinov]

Looks like the schema for the queue parameter for EventSourceMapping changed:
In 5.43.0 it was:

"queues": {
                    "type": "array",
                    "items": {
                        "type": "string"
                    },
                    "description": "The name of the Amazon MQ broker destination queue to consume. Only available for MQ sources. A single queue name must be specified.\n"
                },

And now it is:

"queues": {
                    "type": "string",
                    "description": "The name of the Amazon MQ broker destination queue to consume. Only available for MQ sources. The list must contain exactly one queue name.\n"
                },

I guess it was always meant to have only one parameter but the change of the type is a breaking change. Specifically in the repro above, when we call "/pulumirpc.ResourceMonitor/RegisterResource" for EventSourceMapping, we used to pass "queues": [] but now we pass "queues":null. Passing [] triggers the exception.

[t0yv0]

This is very interesting. The new provider looked at queues: [] state and responded with DIFF_NONE.

{
    "method": "/pulumirpc.ResourceProvider/Diff",
    "request": {
        "id": "a36b1ca7-080b-482a-a685-0b087016048d",
        "urn": "urn:pulumi:dev::aws_esm_py::aws:lambda/eventSourceMapping:EventSourceMapping::example",
        "olds": {
            "amazonManagedKafkaEventSourceConfig": null,
            "batchSize": 10,
            "bisectBatchOnFunctionError": false,
            "destinationConfig": null,
            "documentDbEventSourceConfig": null,
            "enabled": true,
            "eventSourceArn": "arn:aws:sqs:us-east-1:616138583583:queue-d91b62c",
            "filterCriteria": null,
            "functionArn": "arn:aws:lambda:us-east-1:616138583583:function:testLambda-597cec3",
            "functionName": "arn:aws:lambda:us-east-1:616138583583:function:testLambda-597cec3",
            "functionResponseTypes": [],
            "id": "a36b1ca7-080b-482a-a685-0b087016048d",
            "lastModified": "2023-12-07T15:11:47Z",
            "lastProcessingResult": "",
            "maximumBatchingWindowInSeconds": 0,
            "maximumRecordAgeInSeconds": 0,
            "maximumRetryAttempts": 0,
            "parallelizationFactor": 0,
            "queues": [],
            "scalingConfig": null,
            "selfManagedEventSource": null,
            "selfManagedKafkaEventSourceConfig": null,
            "sourceAccessConfigurations": [],
            "startingPosition": "",
            "startingPositionTimestamp": "",
            "state": "Enabled",
            "stateTransitionReason": "USER_INITIATED",
            "topics": [],
            "tumblingWindowInSeconds": 0,
            "uuid": "a36b1ca7-080b-482a-a685-0b087016048d"
        },
        "news": {
            "__defaults": [
                "enabled"
            ],
            "enabled": true,
            "eventSourceArn": "arn:aws:sqs:us-east-1:616138583583:queue-d91b62c",
            "functionName": "arn:aws:lambda:us-east-1:616138583583:function:testLambda-597cec3"
        },
        "oldInputs": {
            "__defaults": [
                "enabled"
            ],
            "enabled": true,
            "eventSourceArn": "arn:aws:sqs:us-east-1:616138583583:queue-d91b62c",
            "functionName": "arn:aws:lambda:us-east-1:616138583583:function:testLambda-597cec3"
        }
    },
    "response": {
        "stables": [
            "queues",
            "startingPosition",
            "selfManagedEventSource",
            "amazonManagedKafkaEventSourceConfig",
            "topics",
            "selfManagedKafkaEventSourceConfig",
            "eventSourceArn",
            "startingPositionTimestamp"
        ],
        "changes": "DIFF_NONE",
        "hasDetailedDiff": true
    },
    "metadata": {
        "kind": "resource",
        "mode": "client",
        "name": "aws"
    }
}

After that the engine stopped interacting with the provider and emitted queues: [] from RegisterResource back to the Python language SDK, violating the type contract, and causing the runtime failure in the Python SDK.

[t0yv0]

I'm wondering if the intent of this change pulumi/pulumi-terraform-bridge@f0a3ffc is making it worse here, in a sense that the bridge pretends that []->null translation is done implicitly but does not inform the engine..

[VenelinMartinov]

We are still working on a fix but the issue has a workaround - one can specify a value for queues. For example:

esm = aws.lambda_.EventSourceMapping(
    "example",
    event_source_arn=queue.arn,
    function_name=test_lambda.arn,
    queues="queue",
    opts=pulumi.ResourceOptions(delete_before_replace=True),
)

This will force the resource to update and work around the issue.
Once updated, one can then remove the queues value and run pulumi up again, which should get to the desired end state.
NOTE: This will cause a REPLACEMENT of the resource, which could trigger replacements of any resources which depend on it.
Also note that delete_before_replace is needed here.

[VenelinMartinov]

This was released in v6.13.3