Logs: Faulty Resource Policy generated

[ezra-agathos]

What is the problem?

I'm attempting to create a simple stack that houses a task definition for some fargate tasks.
When I add a logging configuration to a container definition inside, a resource policy is added in the generated cf template:

  "Resources": {
    "ecsstackloggroupPolicy0078989C": {
      "Type": "AWS::Logs::ResourcePolicy",
      "Properties": {
        "PolicyDocument": "{\"Statement\":[{\"Action\":[\"logs:CreateLogStream\",\"logs:PutLogEvents\"],\"Effect\":\"Allow\",\"Principal\":
        ...
        ...

Reproduction Steps

from aws_cdk import (
    core as cdk,
    aws_ecs as ecs,
    aws_iam as iam,
    aws_ecr as ecr,
    aws_logs as logs,
)
REGION="us-east-2"
FAMILY="example"
TASK_ARN="arn:aws:iam::<example existing iam role arn>"
EXEC_ARN="arn:aws:iam::<example existing iam role arn>"
ECR_PIPELINE_ARN="arn:aws:ecr:<example existing ECR arn>"
LOG_GROUP_ARN="arn:aws:logs:<example existing log group arn>"
class EcsStack(cdk.Stack):

    def __init__(self, scope: cdk.Construct, construct_id: str, **kwargs) -> None:
        super().__init__(scope, construct_id, **kwargs)

        # reference execution role
        exec_role = iam.Role.from_role_arn(
            self,
            'fargate-exec-role',
            EXEC_ARN,
            mutable=False
        )

        # reference task role
        task_role = iam.Role.from_role_arn(
            self,
            'fargate-task-role',
            TASK_ARN,
            mutable=False
        )

        # reference pipeline ecr image
        repo = ecr.Repository.from_repository_arn(
            self,
            'repo',
            ECR_PIPELINE_ARN
        )

        image = ecs.ContainerImage.from_ecr_repository(
            repo,
            'image-name'
        )
        
        # logging
        log_group = logs.LogGroup.from_log_group_arn(
            self,
            'ecs-stack-log-group',
            log_group_arn=LOG_GROUP_ARN
        )
        log_driver = ecs.LogDriver.aws_logs(
            stream_prefix="test",
            log_group=log_group
        )

        # construct base task definition
        base_def = ecs.TaskDefinition(
            self,
            'base-definition',
            family=FAMILY,
            execution_role=exec_role,
            task_role=task_role,
            compatibility=ecs.Compatibility.FARGATE,
            cpu='512',
            memory_mib='1024'
        )
        base_def.add_container(
            id='pipeline',
            container_name='pipeline',
            image=image,
            cpu=512,
            memory_limit_mib=1024,
            memory_reservation_mib=512,
            essential=True,
            environment=[],
            secrets=[],
            logging=log_driver
        )

What did you expect to happen?

I was attempting to generate a template that behaved similarly to a CF template I've written below.

When deployed, a task definition with a single container definition is created, with references to the pre-existing resources I've specified using ARNs.

{
    "AWSTemplateFormatVersion": "2010-09-09",
    "Resources": {
        "TaskDefinition": {
            "Type": "AWS::ECS::TaskDefinition",
            "Properties": {
                "Family": "test-deploy-definition",
                "ExecutionRoleArn": "...",
                "TaskRoleArn": "...",
                "RequiresCompatibilities": [
                    "FARGATE"
                ],
                "Cpu": "512",
                "Memory": "1024",
                "NetworkMode": "awsvpc",
                "ContainerDefinitions": [
                    {
                        "Name": "...",
                        "Image": "...",
                        "Command": [],
                        "EntryPoint": [],
                        "MountPoints": [],
                        "Cpu": 512,
                        "Memory": 1024,
                        "MemoryReservation": 512,
                        "PortMappings": [],
                        "Essential": true,
                        "Environment": [
                        ],
                        "Secrets": [],
                        "LogConfiguration": {
                            "LogDriver": "awslogs",
                            "Options": {
                              "awslogs-group": "/test/log",
                              "awslogs-region": "us-east-2",
                              "awslogs-stream-prefix": "test"
                            }
                    
                        }
                    }
                ]
            }
        }
    }

What actually happened?

1. The resource policy request generates an error:

11:06:15 AM | CREATE_FAILED        | AWS::Logs::ResourcePolicy | ecsstackloggroupPolicy0078989C
Resource handler returned message: "Invalid request provided: AWS::Logs::ResourcePolicy" (RequestToken: 24b0df2b-73fa-056d-4371-7b19f637cb1c, HandlerErrorCode: InvalidRequest)

        new ResourcePolicy (/private/var/folders/d4/lz99k_413039dlfrmtm3zb9h0000gn/T/jsii-kernel-RXZcI5/node_modules/@aws-cdk/aws-logs/lib/policy.js:17:9)
        \_ Import.addToResourcePolicy (/private/var/folders/d4/lz99k_413039dlfrmtm3zb9h0000gn/T/jsii-kernel-RXZcI5/node_modules/@aws-cdk/aws-logs/lib/log-group.js:125:27)
        \_ Function.addToPrincipalOrResource (/private/var/folders/d4/lz99k_413039dlfrmtm3zb9h0000gn/T/jsii-kernel-RXZcI5/node_modules/@aws-cdk/aws-iam/lib/grant.js:70:49)
        \_ Import.grant (/private/var/folders/d4/lz99k_413039dlfrmtm3zb9h0000gn/T/jsii-kernel-RXZcI5/node_modules/@aws-cdk/aws-logs/lib/log-group.js:97:26)
        \_ Import.grantWrite (/private/var/folders/d4/lz99k_413039dlfrmtm3zb9h0000gn/T/jsii-kernel-RXZcI5/node_modules/@aws-cdk/aws-logs/lib/log-group.js:89:21)
        \_ AwsLogDriver.bind (/private/var/folders/d4/lz99k_413039dlfrmtm3zb9h0000gn/T/jsii-kernel-RXZcI5/node_modules/@aws-cdk/aws-ecs/lib/log-drivers/aws-log-driver.js:48:23)
        \_ new ContainerDefinition (/private/var/folders/d4/lz99k_413039dlfrmtm3zb9h0000gn/T/jsii-kernel-RXZcI5/node_modules/@aws-cdk/aws-ecs/lib/container-definition.js:114:50)
        \_ TaskDefinition.addContainer (/private/var/folders/d4/lz99k_413039dlfrmtm3zb9h0000gn/T/jsii-kernel-RXZcI5/node_modules/@aws-cdk/aws-ecs/lib/base/task-definition.js:265:16)
        \_ /private/var/folders/d4/lz99k_413039dlfrmtm3zb9h0000gn/T/tmpp_n4_w2q/lib/program.js:8248:134
        \_ Kernel._wrapSandboxCode (/private/var/folders/d4/lz99k_413039dlfrmtm3zb9h0000gn/T/tmpp_n4_w2q/lib/program.js:8860:24)
        \_ /private/var/folders/d4/lz99k_413039dlfrmtm3zb9h0000gn/T/tmpp_n4_w2q/lib/program.js:8248:107
        \_ Kernel._ensureSync (/private/var/folders/d4/lz99k_413039dlfrmtm3zb9h0000gn/T/tmpp_n4_w2q/lib/program.js:8841:28)
        \_ Kernel.invoke (/private/var/folders/d4/lz99k_413039dlfrmtm3zb9h0000gn/T/tmpp_n4_w2q/lib/program.js:8248:34)
        \_ KernelHost.processRequest (/private/var/folders/d4/lz99k_413039dlfrmtm3zb9h0000gn/T/tmpp_n4_w2q/lib/program.js:9757:36)
        \_ KernelHost.run (/private/var/folders/d4/lz99k_413039dlfrmtm3zb9h0000gn/T/tmpp_n4_w2q/lib/program.js:9720:22)
        \_ Immediate._onImmediate (/private/var/folders/d4/lz99k_413039dlfrmtm3zb9h0000gn/T/tmpp_n4_w2q/lib/program.js:9721:46)
        \_ processImmediate (internal/timers.js:461:21)


 ❌  EcsStack failed: Error: The stack named EcsStack failed to deploy: UPDATE_ROLLBACK_COMPLETE
...

2. It is unclear if this resource policy was even required, given that the defined task and execution roles have necessary log permissions. The documentation does not explain why this is generated in the first place when a logging configuration is attached to the container definition.

CDK CLI Version

1.132.0 (build 5c75891)

Framework Version

No response

Node.js Version

v12.21.0

OS

MacOS Catalina 10.15.7

Language

Python

Language Version

3.9.6

Other information

Our AWS account is controlled by a information security service provider - I thought that perhaps this was a personal account permissions issue. However, the error is InvalidRequest and not AccessDenied so I'm gut checking here first.

[vramanlal]

I have just ran into this issue. A CDK Stack that created a log group now fails after upgrade to 1.132.0. In my case, I am creating a CodeBuild project in the stack and the IAM role assigned already has the required permissions.

Looks like resource policy support was added and it is adding the policy by default (auto permissions I guess). Maybe an option is required when creating the LogGroup whether resource policy should be created automatically or not.

Reverting back to 1.130.0 works for my existing stack.

[ivstiv]

Same problem here:
Resource handler returned message: "Invalid request provided: AWS::Logs::ResourcePolicy"

Reverting to 1.130.0 fixed it for me as well. The issue seems to persist on 1.133.0, so there is no point in updating.

[bjornhandersson]

Same problem here but for MSK
Using CDK 2.0.0

[comcalvi]

Unable to reproduce on CDK 2.0.0. @vramanlal @ivstiv @bjornhandersson can you share a stack that reproduces this?

[bjornhandersson]

@comcalvi
This stack reproduce the issue. I fixed the issue by removing:

    logGroup.grantWrite(new iam.ArnPrincipal(cluster.ref));

I guess this is no longer needed when the ResourcePolicy is "automatically" created in newer versions of CDK.

Stack:

import { Stack, StackProps } from 'aws-cdk-lib';
import { Vpc } from 'aws-cdk-lib/aws-ec2';
import * as logs from 'aws-cdk-lib/aws-logs';
import * as msk from 'aws-cdk-lib/aws-msk';
import * as iam from 'aws-cdk-lib/aws-iam';
import { Construct } from 'constructs';
// import * as sqs from 'aws-cdk-lib/aws-sqs';

export class LogPolicyReprodStack extends Stack {
  constructor(scope: Construct, id: string, props?: StackProps) {
    super(scope, id, props);

    var vpc = new Vpc(this, 'SomeVpc', {
      maxAzs: 2
    });

    
    const logGroup = new logs.LogGroup(this, 'BrokerLogs', {
      retention: logs.RetentionDays.ONE_MONTH
    });

    // I'm using Cfn resource in this stack since when it was originally created no higher level construct existed. 
    const cluster = new msk.CfnCluster(this, 'SomeCluster', {
      clusterName: 'SomeClusterV1',
      kafkaVersion: '2.6.0',
      numberOfBrokerNodes: 2,
      loggingInfo: {
        brokerLogs: {
          cloudWatchLogs: {
            enabled: true, 
            logGroup: logGroup.logGroupName
          }
        }
      },
      enhancedMonitoring: 'PER_TOPIC_PER_BROKER',
      encryptionInfo: {
        encryptionInTransit: {
          clientBroker: 'TLS_PLAINTEXT',
          inCluster: false
        }
      },
      brokerNodeGroupInfo: {
        clientSubnets: vpc.privateSubnets.map(subnet => subnet.subnetId),
        instanceType: 'kafka.t3.small',
        storageInfo: {
          ebsStorageInfo: {
            volumeSize: 8
          }
        }
      }
    });

    // This is the line that causes the problem.  
    logGroup.grantWrite(new iam.ArnPrincipal(cluster.ref));
  }
}

[comcalvi]

@bjornhandersson thanks for the stack, I can confirm that it reproduces the issue. However the line that you rightfully pointed out causes the problem:

logGroup.grantWrite(new iam.ArnPrincipal(cluster.ref));

is working as intended. The cluster.ref returns a resource ARN, which is not a valid IAM principal (see docs).

This reproduces the error for the wrong reason (the error is desired here). Did this successfully deploy on a version prior to 1.132.0? Can someone provide a stack that successfully deployed on older versions but was failed deployment with that error message when you upgraded to 1.132.0?

[bjornhandersson]

@comcalvi

Thanks for the reply.
I realised that as well but posted since the info might be useful for someone else.
It did deploy successfully on version 1.98.0 but not 1.32.0 or above.
I don't know exactly at which version it stoped working.

[ivstiv]

@comcalvi

I did a few runs with 1.133.0 & 1.136.0 stack dependencies (npm) and could not replicate its previous behaviour from a few weeks ago when I consistently experienced the error and had to roll back the update of the dependencies. I have also updated my aws-cdk npm package since then which might also be related somehow. I guess the issue has been resolved?! Unfortunately (or perhaps fortunately 👍 ) I can't help with any broken examples at this point of time.

[comcalvi]

@bjornhandersson I can confirm that it did successfully deploy on version 1.98.0. However, if the stack is first deployed without

logGroup.grantWrite(new iam.ArnPrincipal(cluster.ref));

and then that line is added and you run cdk diff, then the diff is empty. This means that in version 1.98.0, that line didn't add any CloudFormation to the template. The cluster class does not implement IGrantable, so you can't call grantWrite() on it the way you would a lambda function (see docs). You would need to create an IAM role that the cluster can assume and then call grantWrite() on that role. The iam.ArnPrincipal(cluster.ref) results in an invalid principal, so the deployment fails.

@ivstiv thanks for the reply. Glad to hear that the issue has been resolved.

[vramanlal]

The following code with comments show when it works and when it does not. I appears to be related to Fn.importValue when importing a role via ARN created in another stack. Also if the stack props do not provide the env values, then the Fn.importValue lines work. That is no LogResourcPolicy is output. This code has been tested with CDK 1.1.36.0.

import { Stack, Construct, Fn, RemovalPolicy, StackProps, App } from "@aws-cdk/core";
import { Project, Source, FilterGroup, EventAction, BuildSpec, LinuxBuildImage, ComputeType } from "@aws-cdk/aws-codebuild";
import { Role } from "@aws-cdk/aws-iam";
import { Key } from "@aws-cdk/aws-kms";
import { LogGroup, RetentionDays } from "@aws-cdk/aws-logs";

export class CodeBuildStack extends Stack {
    constructor( scope: Construct, id: string, props: StackProps ) {
        super( scope, id, props );

        const kmsKey = Key.fromKeyArn( this, "KmsKeyCodeBuild", Fn.importValue( "devops-kms-key-codebuild-arn" ) );
        const logsKey = Key.fromKeyArn( this, "KmsKeyLogs", Fn.importValue( "devops-kms-key-logs-arn" ) );

        // const type = "node";
        // const projectType = "develop";

        const codeBuildRole = Role.fromRoleArn( this, "CodeBuildRoleImport",
            "arn:aws:iam::<devops account no>:role/role-name", // works - no logs resource policy output
            // Fn.importValue( "some-exported-name-in-another-stack-with-same-arn-value-as-above" ), // does not work - logs resource policy is output
            {
                mutable: false
            } );

        const webhookFilters = [
            FilterGroup.inEventOf( EventAction.PULL_REQUEST_CREATED, EventAction.PULL_REQUEST_REOPENED, EventAction.PUSH ).andHeadRefIsNot( "^refs/tags/release/.*" ),
        ];

        // Source
        const githubSource = Source.gitHub( {
            owner: "<github-owner>",
            repo: "<github-repo>",
            webhook: true,
            webhookFilters: webhookFilters,
            reportBuildStatus: true,
            cloneDepth: 1
        } );

        const description = "Runs unit tests whenever code is pushed.";
        const logGroup = new LogGroup( this, "CodeBuildLogGroup", {
            logGroupName: "/aws/codebuild/test-log-group",
            retention: RetentionDays.ONE_MONTH,
            removalPolicy: RemovalPolicy.DESTROY,
            encryptionKey: logsKey
        } );

        new Project( this, "CodeBuild", { // nosonar
            projectName: "test-project",
            description: description,
            source: githubSource,
            grantReportGroupPermissions: false,
            role: codeBuildRole,
            buildSpec: BuildSpec.fromSourceFilename( "buildspec.yml" ),
            encryptionKey: kmsKey,
            badge: true,
            logging: {
                cloudWatch: {
                    enabled: true,
                    logGroup: logGroup
                }
            },
            environment: {
                buildImage: LinuxBuildImage.STANDARD_5_0,
                computeType: ComputeType.SMALL,
                privileged: false
            }
        } );
    }
}
const app = new App();
new CodeBuildStack( app, "CodeBuildStack", {
    stackName: "TEST-PROJECT",
    env: {
        account: "<devops account no>",
        region: "<region>"
    }
} );

[Stazz-Sphinx]

Hello there.
Any updates regarding that issue. I have the same problem when upgraded from version 1.123.0 till version 1.139.0. ?

I figure out with that issue just let CDK create the IAM ExecutionRole + IAM policy for ECS service byself.

[FrontierPsychiatrist]

I also have this issue when trying to create a log group and use it for an ECS Task container with version 1.139.0.

[nomadme]

having a same issue on v1.135.0

[comcalvi]

@nomadme @FrontierPsychiatrist @Stazz-Sphinx @vramanlal How is the exported value generated? Is it from a CDK stack, CloudFormation only stack, or a resource created via the console / SDKs?

[ezra-agathos]

update on the original bug: turns out I did a rookie mistake and didn't set the appropriate environment in app.py 🤦
e.g. env = cdk.Environment(account=os.getenv('CDK_DEFAULT_ACCOUNT'), region=os.getenv('CDK_DEFAULT_REGION'))
Since then I've also upgraded past v 2.0.0, and haven't seen the issue since - I am curious about the root issue though

[FrontierPsychiatrist]

@comcalvi I am not sure what you mean by "exported value". I am using a CDK stack (in Typescript if that's relevant). Today I upgraded from 1.141.0 to 2.10.0 and still get the same error when I try to deploy the stack.

9:58:33 AM | CREATE_FAILED        | AWS::Logs::ResourcePolicy | LogGroupPolicyResourcePolicy6FA18555
Resource handler returned message: "Invalid request provided: AWS::Logs::ResourcePolicy" (RequestToken: 469a133f-e000-4d55-e8f0-b33a77787491, HandlerErrorCode: InvalidRequest)

I am creating a log group manually and then attaching it to an ECS task container

const logGroup = new logs.LogGroup(this, 'LogGroup', {
  logGroupName: '/ecs/someName',
  retention: logs.RetentionDays.ONE_MONTH,
  removalPolicy: RemovalPolicy.DESTROY,
})

task.addContainer('Reporter', {
  image: ecs.RepositoryImage.fromEcrRepository(repository, imageTag.valueAsString),
  environment: {
    // ...
  },
  logging: ecs.AwsLogDriver.awsLogs({
    streamPrefix: '',
    logGroup: logGroup
  }),
  secrets: {
    // ...
  }
})

The task is allowed to create log streams via a shared execution role that I load via it's ARN (but with mutable: false).

I initially only had the ECR repository in the stack and successfully deployed it, but now after adding the other resources deployments always fail.

Even with multiple -v I don't get a message what actually is invalid in the request though.

[nomadme]

@comcalvi Yes, from CDK.

[vramanlal]

@comcalvi yes the exported values "devops-kms-key-codebuild-arn" and "devops-kms-key-logs-arn" are in another CDK stack. Interesting note is that if in the StackProps is DO NOT provide the env object (account and region), the stack deploys as expected. However when I provide the env object - then it fails. The exported values created by the other stack as also in the same account and region.

[badaldavda8]

@comcalvi - On digging through Ctrail, this is what I find - "Principal section of policy contains ARN instead of account ID: arn:aws:iam::8428xxx:role/BlueGreenContainerImageSt-EcsBlueGreenRolesecsTask-1ETUWEFPQ9D7L",

[nomadme]

@comcalvi So our deployment is returning with this msg. It seems like the underlying cloudformation template is creating a malformed request.

LogGroup/Policy (LogGroupPolicyC8294C50) Resource handler returned message: "Invalid request provided: AWS::Logs::ResourcePolicy" (RequestToken: 02d7fb75-af32-4f3d-003c-a12d13aa9a30, HandlerErrorCode: InvalidRequest)

Please help.

[nomadme]

@rix0rrr You closed this ticket, but you didn't provide clear direction of how to remediate this error.

WHAT IS THE SOLUTION TO RESOLVE THIS ERROR?

[comcalvi]

The other ticket was closed because it is the same issue as this one. We are working on a fix for this now. There is currently no known workaround to this bug without downgrading to one of the versions mentioned above that had successful deployments.

[nomadme]

Thanks @comcalvi for the status update.

Do you have any ETA when the patch will be released? If this is going to take a while, we can rollback to previous version.

[comcalvi]

likely 1-3 weeks.

[BuruY]

This is a possible workaround for logging issue

const logGroup = new LogGroup(this, `${API_PREFIX}LogGroup`, {
        ... some props
});

const taskDefinition = new FargateTaskDefinition(this, `${API_PREFIX}TaskDefinition`, {
        ... some props
});
       
taskDefinition.addContainer(`${API_PREFIX}Container`, {
        ... some props

        // -- Disabled logging in container, this is where the bug in CDK lives. 
        // logging: LogDrivers.awsLogs({
        //     streamPrefix: API_PREFIX,
        //     logGroup: logGroup
        // })
    }
);

// -- Logging is added with a cfnTaskDefinition override
const cfnTaskDefinition = taskDefinition.node.defaultChild as CfnTaskDefinition;
cfnTaskDefinition.addOverride("Properties.ContainerDefinitions.0.LogConfiguration", {
    "LogDriver": "awslogs",
    "Options": {
        "awslogs-group": (logGroup.node.defaultChild as CfnLogGroup).ref,
        "awslogs-stream-prefix": API_PREFIX,
        "awslogs-region": "*****" // change this to actual region
    }
});

[nomadme]

@BuruY Thanks, using the level 1 class works.

[ghost]

Thanks @BuruY !
This one was a real doozie.
Here's the override part of the code for the Python crowd:

cfn_taskdef: ecs.CfnTaskDefinition = fargate_taskdef.node.default_child
cfn_loggroup: logs.CfnLogGroup = log_group.node.default_child
cfn_taskdef.add_override(
  "Properties.ContainerDefinitions.0.LogConfiguration",
  {
    "LogDriver": "awslogs",
    "Options": {
        "awslogs-group": cfn_loggroup.ref,
        "awslogs-stream-prefix": "your-log-group-prefix",
        "awslogs-region": "us-east-1"
    }
  }
)

[simonscheurer]

Any update on this? The 1 to 3 weeks are over since some time. Still not working on my side. Also on 1.147 the same issue happens.
"Invalid request provided: AWS::Logs::ResourcePolicy" (RequestToken: ce0942e0-30c6-4221-bbac-88a8bbc09e25, HandlerErrorCode: InvalidRequest)

[hao-public]

This is taking too long. Can someone provide a L1 example fix using Python code, please :-)

[ytsipun]

The same issue after updating from CDK v1.123.0 to v2.10.0:

CloudTrail:

Principal section of policy contains ARN instead of account ID

[UPD]
Applied a workaround by removing the AWS::Logs::ResourcePolicy resource:
logGroup.node.tryRemoveChild('Policy')
Logs are still pushed (thanks to the ECS Task Exec role permissions).

[ghost]

FWIW this issue seems to be fixed in the latest Python CDK version 2.17.0

[badaldavda8]

Not fixed in Typescript CDK Version 2.17.0. Still facing the same issue.

[kbakk]

FWIW this issue seems to be fixed in the latest Python CDK version 2.17.0

Not for us. Started seeing this after moving a log group for ECS task to a different construct (new resource ID). Have tried to upgrade from 2.15.0 to 2.17.0, and it didn't fix it.

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.