@aws-cdk/integ-tests-alpha: (Integ-test's does not have the proper role permission)

[UlaJiang]

Describe the bug

@aws-cdk/integ-tests-alpha module
https://docs.aws.amazon.com/cdk/api/v2/docs/integ-tests-alpha-readme.html

By default, the AwsApiCall construct will automatically add the correct IAM policies to allow the Lambda function to make the API call. It does this based on the service and api that is provided.

During the deployment, there are several lambda function and execution role are auto-generated by CDK.

Stack ARN in case helps for troubleshooting: arn:aws:cloudformation:us-west-2:673604154507:stack/IntegTestdevDefaultTestDeployAssert31A9C52F/5746bdf0-7d00-11ee-9e23-06a38c3b8f8b

Inside the state machine, we can see the error

AccessDeniedException: User: arn:aws:sts::673604154507:assumed-role/IntegTestdevDefaultTestDe-SingletonFunction76b3e830-5BWQ2YQ5iN2A/IntegTestdevDefaultTestDe-SingletonFunction76b3e83-fLl2b7TfQLmk is not authorized to perform: lambda:InvokeFunction on resource: arn:aws:lambda:us-west-2:673604154507:function:CDKTemplate-Microscope-Scheduler-dev because no identity-based policy allows the lambda:InvokeFunction action

However, checking the auto-generated lambda execution role,one of the role IntegTestdevDefaultTestDe-SingletonFunction76b3e830-QIh9L3ag1iz4 has the permission "lambda:Invoke", but there is no "lambda:InvokeFunction" permission, another role IntegTestdevDefaultTestDe-SingletonFunction1488541a-PkTEiGn4sLHb has both "lambda:Invoke" and "lambda:InvokeFunction" permission automatically generated

Expected Behavior

inside AWS console, the below action is an invalid action, and console will throw validation error, may I know if this is a bug in CDK that auto-generate "lambda:Invoke" in action rather than "lambda:InvokeFunction"?

Current Behavior

We tried to use escape hatch to down to the L2/L1 construct to override the role permission, however, the L1 construct of Role is CfnResource not CfnRole, which is also a bit different from other construct like laws-lambda as I can get CfnRole and can be used by add_property_override
example of aws_lambda construct:

        fn = lambda_.Function(self, "MyLambda",
            code=lambda_.InlineCode("exports.handler = async () => { console.log('hello world'); };"),
            handler="index.handler",
            runtime=lambda_.Runtime.PYTHON_3_7
        )
        
        children = fn.node.find_all()
        for child in children:
            print(child, "->", child.node.id, "->", child.node.path)

printout with CfnRole as the construct

<aws_cdk.aws_lambda.Function object at 0x7f1df63e8d90> -> MyLambda -> cdk-1-xx/MyLambda
<aws_cdk.aws_iam.Role object at 0x7f1de43f80d0> -> ServiceRole -> cdk-1-xx/MyLambda/ServiceRole
<aws_cdk.aws_iam.CfnRole object at 0x7f1de43f8310> -> Resource -> cdk-1-xx/MyLambda/ServiceRole/Resource
<aws_cdk.aws_lambda.CfnFunction object at 0x7f1df62a3410> -> Resource -> cdk-1-xx/MyLambda/Resource

However, for @aws-cdk/integ-tests-alpha module, I can only get CfnResource construct (see attachment) and cannot use add_property_override to change the policy

Reproduction Steps

Question1: may I know how does the code intergated in the backend to choose which IAM Role to make the invokefunction action inside statemachine ? and how can we customised this IAM role please?

Question2: inside AWS console, the below action is an invalid action, and console will throw validation error, may I know if this is a bug in CDK that auto-generate "lambda:Invoke" in action rather than "lambda:InvokeFunction"?

Possible Solution

No response

Additional Information/Context

No response

CDK CLI Version

2.104.0

Framework Version

No response

Node.js Version

n/a

OS

mac

Language

Python

Language Version

No response

Other information

No response

[oqian]

The other workaround that I have tried is to leverage the api_call.provider.add_to_role_policy, hoping to add policies to the singletonFunction so support the invocation of lambda function. After checking the resources created by the stack, it seems that the added policy was not properly added to the singletonFunction's execution role. Rather, it was added to some other singletonFunctions that are not in charge of invoking the function-of-interest. So there might also be a mismatch issue between the provider and the one that is actually triggering the function.

[pahud]

I think at least lambda:InvokeFunction is required and probably lambda:Invoke* as well.

Are you able to provide a minimal sample that we can reproduce in our account?

[github-actions]

This issue has not received a response in a while. If you want to keep this issue open, please leave a comment below and auto-close will be canceled.

[oqian]

Hi, @pahud ,
An example for the IAM action issue that can be reproduced:

lambda_invocation = integ.assertions.invoke_function(    
    function_name=stack_to_be_tested.lambda_function.function_name,
    invocation_type=InvocationType.EVENT,
    payload=json.dumps({
        "days":1
    })
).expect(
    ExpectedResult.object_like(
        {
            "execution_arn": Match.string_like_regexp("arn:aws:states:us-.*")
        }
    )
).wait_for_assertions(
    interval=Duration.seconds(10),
    total_timeout=Duration.minutes(90)
)

Here integ is an aws_cdk.integ_tests_alpha.IntegTest object.

[oqian]

An example that integ_test fails to add the role policy to the assertion provider:
lambda_invocation.provider.add_to_role_policy({ "Effect": "Allow", "Action": ["lambda:*"], "Resource": ["*"] })
The additional user-made IAM policy was not added to the right assertion provider that executes the lambda call/api call in the integ-test.

[oqian]

In the singletonFunction, I kept seeing the following message:

INFO	AccessDeniedException: User: arn:aws:sts::673604154507:assumed-role/IntegTestdevDefaultTestDe-SingletonFunction76b3e830-YcJyEyB1Llkg/IntegTestdevDefaultTestDe-SingletonFunction76b3e83-MePrE2rfZIja is not authorized to perform: lambda:InvokeFunction on resource: arn:aws:lambda:us-west-2:673604154507:function:CDKTemplate-Microscope-Scheduler-test because no identity-based policy allows the lambda:InvokeFunction action

and when I go into the role that is attached to this lambda function, the in-line policy is defined as follows:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": [
                "lambda:Invoke"
            ],
            "Resource": [
                "*"
            ],
            "Effect": "Allow"
        },
        {
            "Action": [
                "states:DescribeExecution"
            ],
            "Resource": [
                "*"
            ],
            "Effect": "Allow"
        }
    ]
}

Apparently, for lambda-related IAM actions, there is no such action as "Invoke", so it has to be a typo. Ref: https://docs.aws.amazon.com/service-authorization/latest/reference/list_awslambda.html

The other question for this inline-policy is that the added role_policy was not reflected, so there are primarily 2 issues here.

Hope that these details will help you reproduce the issue.

[pahud]

Yes I can investigate this with CDK in TypeScript. Details below:

export class DemoStack extends Stack {
  readonly fn: lambda.IFunction;
  constructor(scope: Construct, id: string, props: StackProps) {
    super(scope, id, props);

    this.fn = new lambda.Function(this, 'Fn', {
      code: lambda.Code.fromInline(`def handler(event, context): return {'foo': 'bar'}`),
      handler: 'index.handler',
      runtime: lambda.Runtime.PYTHON_3_11,
      functionName: PhysicalName.GENERATE_IF_NEEDED,
    });
    
  }
}

integ-demo.ts

const app = new cdk.App();
const env = { region: process.env.CDK_DEFAULT_REGION, account: process.env.CDK_DEFAULT_ACCOUNT };

const demoStack = new DemoStack(app, 'demo-stack9', { env });
const integ = new IntegTest(app, 'Integ', {
  testCases: [demoStack],
});
integ.assertions.invokeFunction({
  functionName: demoStack.fn.functionName,
  invocationType: InvocationType.EVENT,
  payload: JSON.stringify({ 'days': 1 }),
})
.expect(ExpectedResult.objectLike({'execution_arn': Match.stringLikeRegexp('arn:aws:states:us-.*')}))
.waitForAssertions({
  interval: cdk.Duration.seconds(30),
  totalTimeout: cdk.Duration.minutes(90),
});

Run yarn integ-runner integ.demo.ts --update-on-failed

And check the state machine:

And the Lambda Log error message:

2023-11-21T16:06:33.356Z	c428e6b6-1a2e-43df-ac5c-69c0f8b6291e	INFO	AccessDeniedException: User: arn:aws:sts::XXXXXXXXXXXX:assumed-role/IntegDefaultTestDeployAss-SingletonFunction76b3e830-f14egzaVWLg4/IntegDefaultTestDeployAss-SingletonFunction76b3e83-K7rP3RHVgNjc is not authorized to perform: lambda:InvokeFunction on resource: arn:aws:lambda:us-east-1:903779448426:function:demo-stack9demostack9fn3ada946fb19bc1d14d04 because no identity-based policy allows the lambda:InvokeFunction action  

And the inline policy of the assumed role

Obviously we need lambda:InvokeFunction rather than lambda:Invoke.

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.

[oqian]

Hi @pahud , I dont think this issue was fully addressed by the most recent PR #28424 . In the latest version 2.118.0, although the wait provider was granted the correct permission, the actual singleton function that is in charge of invoking the function-to-be-tested is still not granted the "Lambda:invokeFunction " action. You can try to reproduce the issue by following the exact same process that is mentioned earlier in this issue report.