Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[sqs] Circular dependency between resources when encrypting sqs queue with kms and assigning target to rule #11158

Closed
prabint opened this issue Oct 27, 2020 · 13 comments · Fixed by #14638
Assignees
Labels
@aws-cdk/aws-sqs Related to Amazon Simple Queue Service bug This issue is a bug. effort/small Small work item – less than a day of effort good first issue Related to contributions. See CONTRIBUTING.md p1

Comments

@prabint
Copy link

prabint commented Oct 27, 2020

I am setting up an event bridge rule that will forward events from Macie to sqs. Everything works fine until I add encryptionMasterKey to the queue. I either have to remove the .encryptionMasterKey or remove the .target() to avoid circular dependency error.

Reproduction Steps

val queue = Queue.Builder.create(scope, "MacieReportQueue")
             .queueName("MacieReportQueue")
             // TODO: FIXME: KMS Circular dependency error when combined with target
             //  https://github.com/aws/aws-cdk/issues/3067
             .encryption(QueueEncryption.KMS)
             .encryptionMasterKey(key)
             .build()

     val target = SqsQueue.Builder.create(queue).build()

     val rule = Rule.Builder.create(scope, "MacieToSqsEventBridgeRule")
             .ruleName("MacieToSqsEventBridgeRule")
             .description("Sends messages to SQS if Macie finds sensitive information")
             .targets(listOf(target))
             .eventPattern(EventPattern.builder()
                     .source(listOf("aws.macie"))
                     .build()
             )
             .build()

CF Graph

What did you expect to happen?

No circular dependency error. Queue should be encrypted.

What actually happened?

Circular dependency between resources: [MacieReportQueue7D1A954B, MacieReportQueuePolicy84D89C73, MacieToSqsEventBridgeRule498CF606, MacieReportQueueKeyAliasA0A0C899, MacieReportQueueKeyF608AAD8]

Environment

  • **CLI Version :1.67.0 (build 2b4dd71)
  • **Framework Version:1.67.0
  • Node.js Version: v14.13.1
  • OS : Mac 10.14.6
  • Language (Version): Java 8

Other


This is 🐛 Bug Report

@prabint prabint added bug This issue is a bug. needs-triage This issue or PR still needs to be triaged. labels Oct 27, 2020
@prabint prabint changed the title Circular dependency between resources when using encrypting sqs queue with kms and assigning target to rule Circular dependency between resources when encrypting sqs queue with kms and assigning target to rule Oct 27, 2020
@SomayaB SomayaB changed the title Circular dependency between resources when encrypting sqs queue with kms and assigning target to rule [sqs] Circular dependency between resources when encrypting sqs queue with kms and assigning target to rule Nov 2, 2020
@github-actions github-actions bot added the @aws-cdk/aws-sqs Related to Amazon Simple Queue Service label Nov 2, 2020
@MrArnoldPalmer MrArnoldPalmer added effort/small Small work item – less than a day of effort p2 and removed needs-triage This issue or PR still needs to be triaged. labels Nov 10, 2020
@jamesholmes-linktree
Copy link

jamesholmes-linktree commented Nov 13, 2020

I hit the same issue with an encrypted queue. The issue is this addition to the Key Policy when the target is created:

{
              "Action": [
                "kms:Decrypt",
                "kms:Encrypt",
                "kms:ReEncrypt*",
                "kms:GenerateDataKey*"
              ],
              "Condition": {
                "ArnEquals": {
                  "aws:SourceArn": {
                    "Fn::GetAtt": [
                      "Rule123456",
                      "Arn"
                    ]
                  }
                }
              },
              "Effect": "Allow",
              "Principal": {
                "Service": "events.amazonaws.com"
              },
              "Resource": "*"
            }

The Key now depends on the Rule ARN as a condition; the rule depends on the queue ARN as a target and; the queue depends on the key ARN to encrypt it. There's the circle.

@rix0rrr
Copy link
Contributor

rix0rrr commented Dec 24, 2020

This is pretty much unfixable by design the way Key Policies work in KMS, as long as this is not implemented: aws-cloudformation/cloudformation-coverage-roadmap#322

@rix0rrr
Copy link
Contributor

rix0rrr commented Dec 24, 2020

I would be okay dropping the Condition and just trusting AWS Events wholesale. Looping in @njlynch as the master of KMS.

@rix0rrr rix0rrr added good first issue Related to contributions. See CONTRIBUTING.md p1 and removed p2 labels Dec 24, 2020
@rix0rrr
Copy link
Contributor

rix0rrr commented Dec 24, 2020

Think I disagree with priority as well. This should work and there's no workaround.

@element7771
Copy link

+1

Faced this problem on project. Still present in cdk 1.80.0.
It will be nice, if you are able to fix it in the nearest versions.

Thanks!

@njlynch
Copy link
Contributor

njlynch commented Feb 11, 2021

Missed this mention over the holidays... I agree that removing the condition is likely the simplest approach here.

@sudopla
Copy link

sudopla commented Feb 12, 2021

Running into this same problem. Removing the condition would be great

@Rajesh734
Copy link

I am having the exact same issue with my cdk project. I created a SQS queue with masterkey and created EventBridge Target. By default AWS-CDK adds SQSQueuePolicy which I can't alter and remove the condition defined in the Policy. Hence, can't remove the circular dependency.

How can this be fixed in aws cdk project?

I am using: Nodejs, aws-cdk

@sudopla
Copy link

sudopla commented Feb 16, 2021

I'm using the CFN class to circumvent this problem for now.

const cfnRule = eventRule.node.defaultChild as events.CfnRule
cfnRule.targets = [
    {
        arn: queue.queueArn,
        id: 'EventQueue-Target',
    }
]

You will also need to define the following permissions

// Gives EventBridge rule permission to send message to queue target            
queue.addToResourcePolicy(new iam.PolicyStatement({
    actions: ['sqs:SendMessage'],
    resources: [queue.queueArn],
    principals: [new iam.ServicePrincipal('events.amazonaws.com')],
    conditions: {
        'ArnEquals': {'aws:SourceArn': eventRule.ruleArn}
    }
}))
// Give EventBridge access to KMS key
sqsKmsKey.addToResourcePolicy(new iam.PolicyStatement({
    principals: [new iam.ServicePrincipal('events.amazonaws.com')],
    actions: ['kms:Decrypt', 'kms:GenerateDataKey'],
    resources: ['*']
}))

@Rajesh734
Copy link

I'm using the CFN class to circumvent this problem for now.

const cfnRule = eventRule.node.defaultChild as events.CfnRule
cfnRule.targets = [
    {
        arn: queue.queueArn,
        id: 'EventQueue-Target',
    }
]

You will also need to define the following permissions

// Gives EventBridge rule permission to send message to queue target            
queue.addToResourcePolicy(new iam.PolicyStatement({
    actions: ['sqs:SendMessage'],
    resources: [queue.queueArn],
    principals: [new iam.ServicePrincipal('events.amazonaws.com')],
    conditions: {
        'ArnEquals': {'aws:SourceArn': eventRule.ruleArn}
    }
}))
// Give EventBridge access to KMS key
sqsKmsKey.addToResourcePolicy(new iam.PolicyStatement({
    principals: [new iam.ServicePrincipal('events.amazonaws.com')],
    actions: ['kms:Decrypt', 'kms:GenerateDataKey'],
    resources: ['*']
}))

Didn't think about CFN class. Thank you @sudopla !

@madeline-k madeline-k self-assigned this May 10, 2021
@madeline-k
Copy link
Contributor

Taking a look at this one. I'm planning to remove the condition from being added here: https://github.com/aws/aws-cdk/blob/master/packages/@aws-cdk/aws-events-targets/lib/sqs.ts#L58-L60 in the case where the queue is encrypted.

madeline-k added a commit to madeline-k/aws-cdk that referenced this issue May 11, 2021
…eue encrypted with KMS as a target of an AWS Events Rule

fixes aws#11158
madeline-k added a commit to madeline-k/aws-cdk that referenced this issue May 11, 2021
…eue encrypted with KMS as a target of an AWS Events Rule

fixes aws#11158
@mergify mergify bot closed this as completed in #14638 May 13, 2021
mergify bot pushed a commit that referenced this issue May 13, 2021
…SQS queue (#14638)

fixes #11158


----

*By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*
@github-actions
Copy link

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.

BenChaimberg added a commit that referenced this issue May 18, 2021
commit 35a6202
Author: Ben Chaimberg <chaimber@amazon.com>
Date:   Tue May 18 10:18:39 2021 -0700

    move supported conditions to function-base and minor name changes

commit 7c6c217
Author: Ben Chaimberg <chaimber@amazon.com>
Date:   Mon May 17 20:23:55 2021 -0700

    remove extraneous whitespace

commit 02cf427
Author: Ben Chaimberg <chaimber@amazon.com>
Date:   Mon May 17 17:31:01 2021 -0700

    add conditions snippet to README

commit 8ebf049
Author: Ben Chaimberg <chaimber@amazon.com>
Date:   Mon May 17 17:22:25 2021 -0700

    format README

commit af2be84
Merge: ea53bcc 8856482
Author: Ben Chaimberg <chaimber@amazon.com>
Date:   Mon May 17 17:19:14 2021 -0700

    Merge branch 'master' of github.com:aws/aws-cdk into chaimber/lambda_perm_cond

commit ea53bcc
Merge: 988b66c 1a695e2
Author: Ben Chaimberg <chaimber@amazon.com>
Date:   Mon May 17 17:16:10 2021 -0700

    Merge branch 'chaimber/lambda_perm_cond' of github.com:aws/aws-cdk into chaimber/lambda_perm_cond

commit 988b66c
Author: Ben Chaimberg <chaimber@amazon.com>
Date:   Mon May 17 17:10:48 2021 -0700

    add documentation for Permission to README

commit 8856482
Author: Elad Ben-Israel <benisrae@amazon.com>
Date:   Mon May 17 23:19:30 2021 +0300

    chore: set license of eslint-plugin-cdk (#14720)

    Fixes #14594

    ----

    *By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*

commit b70a5fa
Author: Carter Van Deuren <carterv@users.noreply.github.com>
Date:   Mon May 17 11:29:47 2021 -0700

    docs(kinesis): correct grantRead and grantWrite comments (#14707)

    This commit swaps the comments for `grantRead` and `grantWrite` so that the comments match the permissions being granted.

    No extra verification was done for this change, as it only effects comments.

    ----

    *By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*

commit f65d826
Author: Jared Short <jaredlshort@gmail.com>
Date:   Mon May 17 13:38:57 2021 -0400

    docs(stepfunctions-tasks): fix integration patterns of step-function-task docs (#14722)

    ----

    *By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*

commit 50d486a
Author: Adam Wong <55506708+wong-a@users.noreply.github.com>
Date:   Mon May 17 10:12:49 2021 -0700

    feat(stepfunctions): Add support for ResultSelector (#14648)

    ### Description

    Adds support for `ResultSelector`. [ResultSelector](https://docs.aws.amazon.com/step-functions/latest/dg/input-output-inputpath-params.html#input-output-resultselector) was added to ASL in August 2020 and is currently missing coverage in CDK.

    This change exposes a new `resultSelector` field to Task, Map, and Parallel state props. This is a JSON object that functions similarly to `Parameters` but where `$` refers to the state's raw result instead of the state input. This allows you to reshape the result without using extra Pass states.

    The implementation mimics what exists for Parameters. I'm not convinced we need extra types here.

    #### Example

    ```ts
    new tasks.LambdaInvoke(this, 'Invoke Handler', {
      lambdaFunction: fn,
      resultSelector: {
        lambdaOutput: sfn.JsonPath.stringAt('$.Payload'),
        invokeRequestId: sfn.JsonPath.stringAt('$.SdkResponseMetadata.RequestId'),
        staticValue: 'foo',
      },
    })
    ```

    Which produces the following ASL:

    ```json
    {
      "Type": "Task",
      "Resource": "arn:aws:states:::lambda:invoke",
      "Parameters": {
        "FunctionName": ${functionName},
        "Payload.$": "$"
      },
      "ResultSelector": {
          "lambdaOutput.$": "$.Payload",
          "invokeRequestId.$": "$.SdkResponseMetadata.RequestId",
          "staticValue": "foo",
      },
      "Next": ${nextState}
    }
    ```

    ### Testing
    * Unit tests for Map, Task, and Parallel states to include `resultSelector`
    * Unit test with ResultSelector for `LambdaInvoke` state updated to include `resultSelector`
    * Updated LambdaInvoke integ test to use ResultSelector for one of the states. Executed state machine manually through the AWS console to ensure the example actually works too.

    Closes #9904

    ----

    *By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*

commit 4da78f6
Author: Kyle Roach <kroach.work@gmail.com>
Date:   Mon May 17 12:45:39 2021 -0400

    feat(apigatewayv2): http api - lambda authorizer (#13181)

    Second part of #10534

    Had to make small changes to `authorizerType` that route expects, since Route and Authorizer enums are not the same. See #10534 (comment).

    ----

    *By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*

commit 49d18ab
Author: Austin Yattoni <austinyattoni@gmail.com>
Date:   Mon May 17 11:18:49 2021 -0500

    fix(lambda): unable to access SingletonFunction vpc connections (#14533)

    fixes #6261

    ----

    *By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*

commit 8154e91
Author: Nick Lynch <nlynch@amazon.com>
Date:   Mon May 17 15:29:02 2021 +0100

    chore: skip Go proxy for lambda-go integ tests (#14727)

    Privacy-conscious users and/or organizations may choose to skip sending all
    package requests to the Google proxy (as is default). Some corporate
    environments may block network access to proxy.golang.org altogether.

    ----

    *By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*

commit 3bca822
Author: Nick Lynch <nlynch@amazon.com>
Date:   Mon May 17 11:47:57 2021 +0100

    chore(msk): add ignore-assets pragma to cluster integ test (#14725)

    The MSK module relies on custom resources, which in turn create a Lambda
    function with assets. The way the current (lerna/yarn) build works includes
    the .ts file (as well as the .d.ts and .js) files in the asset bundle. Using the
    new `nozem` build (correctly) only includes the .d.ts and .js files, leading to a
    different asset hash.

    Since we don't care about the actual hash anyway, adding the ignore-assets
    pragma so this test can pass with either build tool.

    ----

    *By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*

commit fb0977a
Merge: 1effc9f df89694
Author: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Date:   Sat May 15 00:59:26 2021 +0000

    chore(merge-back): 1.104.0 (#14708)

    See [CHANGELOG](https://github.com/aws/aws-cdk/blob/merge-back/1.104.0/CHANGELOG.md)

commit df89694
Merge: 44d3383 1effc9f
Author: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Date:   Sat May 15 00:33:23 2021 +0000

    Merge branch 'master' into merge-back/1.104.0

commit 1effc9f
Author: Hsing-Hui Hsu <hsinghui@amazon.com>
Date:   Fri May 14 15:18:14 2021 -0700

    docs(ecs): add contributing guide (#14672)

    ----

    *By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*

commit 44d3383
Merge: bc13a66 aaa0d05
Author: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Date:   Fri May 14 21:59:31 2021 +0000

    chore(release): 1.104.0 (#14706)

    See [CHANGELOG](https://github.com/aws/aws-cdk/blob/bump/1.104.0/CHANGELOG.md)

commit aaa0d05
Author: Neta Nir <neta1nir@gmail.com>
Date:   Fri May 14 14:33:52 2021 -0700

    Update CHANGELOG.md

commit 0328b03
Author: AWS CDK Team <aws-cdk@amazon.com>
Date:   Fri May 14 21:30:07 2021 +0000

    chore(release): 1.104.0

commit 348e11e
Author: Madeline Kusters <80541297+madeline-k@users.noreply.github.com>
Date:   Fri May 14 14:27:19 2021 -0700

    fix(ecs): Classes FargateService and Ec2Service have no defaultChild (#14691)

    * fix(ecs): Class FargateService has no defaultChild

    fixes #14665

    * update unit tests

commit 1a695e2
Merge: 84045fd d82de05
Author: Ben Chaimberg <chaimber@amazon.com>
Date:   Fri May 14 11:45:50 2021 -0700

    Merge branch 'master' into chaimber/lambda_perm_cond

commit d82de05
Author: Bryan Pan <bryanpan342@gmail.com>
Date:   Thu May 13 08:48:46 2021 -0700

    chore(appsync): rds data source service integration with grantDataApi (#14671)

    Utilize the `grantDataApi` from RDS to complete service integration.

    Fixes: #13189

    ----

    *By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*

commit 8296623
Author: Hsing-Hui Hsu <hsinghui@amazon.com>
Date:   Thu May 13 08:22:23 2021 -0700

    test(ecs-patterns): update l3 fargate integ tests (#14668)

    This adds integ tests for NLB fargate services -- previously, there were
    duplicate ALB fargate services being spun up. Also gives integ test
    stacks unique names.

    ----

    *By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*

commit b240f6e
Author: Nick Lynch <nlynch@amazon.com>
Date:   Thu May 13 14:21:05 2021 +0100

    feat(cloudwatch): GraphWidget supports period and statistic (#14679)

    Dashboard metric widgets support overridding/setting both period and stat on the
    widget as a whole. This is often useful in combination with `MathExpression`
    metrics.

    Reference: https://docs.aws.amazon.com/AmazonCloudWatch/latest/APIReference/CloudWatch-Dashboard-Body-Structure.html#CloudWatch-Dashboard-Properties-Metric-Widget-Object

    ----

    *By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*

commit 3063818
Author: Madeline Kusters <80541297+madeline-k@users.noreply.github.com>
Date:   Thu May 13 02:29:45 2021 -0700

    fix(events-targets): circular dependency when adding a KMS-encrypted SQS queue  (#14638)

    fixes #11158

    ----

    *By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*

commit 9d97b7d
Author: Mitchell Valine <valinm@amazon.com>
Date:   Thu May 13 02:01:08 2021 -0700

    chore: init templates use node jest environment (#14632)

    Remove usage of the `jsdom` test environment in init templates to speed
    up unit testing by default.

    Testing: ran cdk init --language=(typescript|javascript) against local build of
    CLI then ran yarn test to verify that the testing config was valid and jest correctly
    used the node environment.

    fix: #14630

    ----

    *By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*

commit 282d242
Author: Adam Ruka <adamruka@amazon.com>
Date:   Thu May 13 00:37:33 2021 -0700

    chore(custom-resources): import the AWSLambda package explicitly (#14643)

    When linking the aws-cdk repository to a CDK app using the `link-all.sh` script,
    if the app uses `ts-node`,
    the Lambda code in the @aws-cdk/custom-resources package gets picked up by the TypeScript compiler.
    That code relied on the `aws-lambda` package being implicitly available,
    but that would cause `ts-node` to fail.
    Add an explicit import of it in the code -
    I checked the only difference in the generated JS code is the sourceMappingUrl,
    so it shouldn't make a difference at runtime,
    but allows `ts-node` to load that file successfully.

    Fixes #11627

    ----

    *By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*

commit 9a4d624
Author: Oliver Bowman <oliverbowman@protonmail.com>
Date:   Thu May 13 07:40:55 2021 +0100

    docs(lambda-nodejs): Example for esbuild missing comma in property (#13520)

    ----
    Example under esbuild appears to be missing comma.

    *By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*
hollanddd pushed a commit to hollanddd/aws-cdk that referenced this issue Aug 26, 2021
…SQS queue (aws#14638)

fixes aws#11158


----

*By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*
@ryparker
Copy link
Contributor

ryparker commented Nov 30, 2021

Ran into this when troubleshooting a similar issue on StackOverflow

A circular dependency error is thrown when using a KMS key for encrypting an S3 bucket and an SQS queue, then setup a notification on the S3 bucket to message the SQS queue.

See repro code

It looks like the root problem is that the following condition is added to the KMS key:

KMS key resource Properties.KeyPolicy.Statement[1]:

{
	"Action": [
		"kms:Decrypt",
	 	"kms:Encrypt",
		 "kms:ReEncrypt*",
		 "kms:GenerateDataKey*"
	 ],
	 "Condition": {
	 	"ArnLike": {
	    	"aws:SourceArn": {
	      		"Fn::GetAtt": [
	        		"ssls3bucketrawkms4B1E1122",
	        		"Arn"
	     		 ]
	   		 }
	 	 }
 	 },
},

The S3 bucket depends on the KMS key for encryption, and the KMS key has a condition that depends on the S3 bucket.

Workaround

I was able to deploy the stack after using escape hatches to delete the condition:

# Delete the circular reference
cfn_kms_key = kms_key.node.default_child
cfn_kms_key.add_property_deletion_override("KeyPolicy.Statement.1")

workaround code available here


Created a new issue for this.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
@aws-cdk/aws-sqs Related to Amazon Simple Queue Service bug This issue is a bug. effort/small Small work item – less than a day of effort good first issue Related to contributions. See CONTRIBUTING.md p1
Projects
None yet
Development

Successfully merging a pull request may close this issue.

10 participants