Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Changing ssm parameter from string -> securestring throws me an cdk deployment error #30129

Closed
maddyexplore opened this issue May 9, 2024 · 8 comments
Assignees
Labels
@aws-cdk/aws-ssm Related to AWS Systems Manager closed-for-staleness This issue was automatically closed because it hadn't received any attention in a while. response-requested Waiting on additional info and feedback. Will move to "closing-soon" in 7 days.

Comments

@maddyexplore
Copy link

Describe the bug

I was trying to secure all strings in ssm parameter group and I found the cdk-bootstrap version there and I changed it into securestring, after that any deployment with cdk fails with error

AutomationStack: SSM parameter /cdk-bootstrap/<blabla>/version not a number

Expected Behavior

it should sense and decrypt with the km kwy

Current Behavior

throws me error

Reproduction Steps

secure all strings in ssm parameter group and I found the cdk-bootstrap version there and I changed it into securestring, after that any deployment with cdk fails

Possible Solution

No response

Additional Information/Context

No response

CDK CLI Version

2.124.0

Framework Version

No response

Node.js Version

v21.6.1

OS

Windows

Language

Python

Language Version

python 3.10

Other information

No response

@maddyexplore maddyexplore added bug This issue is a bug. needs-triage This issue or PR still needs to be triaged. labels May 9, 2024
@github-actions github-actions bot added the @aws-cdk/aws-ssm Related to AWS Systems Manager label May 9, 2024
@ashishdhingra ashishdhingra added investigating This issue is being investigated and/or work is in progress to resolve the issue. and removed needs-triage This issue or PR still needs to be triaged. labels May 9, 2024
@ashishdhingra ashishdhingra self-assigned this May 9, 2024
@ashishdhingra
Copy link
Contributor

ashishdhingra commented May 9, 2024

Bootstrap version SSM parameter is defined as DEFAULT_BOOTSTRAP_STACK_VERSION_SSM_PARAMETER here. During synthesize() operation, CheckBootstrapVersion rule would be added in the generated CloudFormation template along with the CfnParameter named BootstrapVersion as shown in example below:

Parameters:
  BootstrapVersion:
    Type: AWS::SSM::Parameter::Value<String>
    Default: /cdk-bootstrap/hnb659fds/version
    Description: Version of the CDK Bootstrap resources in this environment, automatically retrieved from SSM Parameter Store. [cdk:skip]
Resources:
...
...
Rules:
  CheckBootstrapVersion:
    Assertions:
      - Assert:
          Fn::Not:
            - Fn::Contains:
                - - "1"
                  - "2"
                  - "3"
                  - "4"
                  - "5"
                - Ref: BootstrapVersion
        AssertDescription: CDK bootstrap stack version 6 required. Please run 'cdk bootstrap' with a recent version of the CDK CLI.

As per description Version of the CDK Bootstrap resources in this environment, automatically retrieved from SSM Parameter Store. of BootstrapVersion parameter in the generated CloudFormation template, the value for this parameter is automatically retrieved from SSM Parameter Store. The CDK deployment is essentially a CloudFormation deployment and is handled by the CloudFormation service.

Per CDK Bootstrapping documentation,

  • AWS CDK's built-in stack synthesizers is called DefaultStackSynthesizer (refer Stack synthesizers)
  • During initialization of a stack, it is possible to specify a synthesizer as per below example in the above documentation:
    new MyStack(this, 'MyStack', {
      // stack properties
      synthesizer: new DefaultStackSynthesizer({
        // synthesizer properties
      }),
    });
  • Here it is possible to override certain properties (as mentioned in Customizing synthesis section)
    • The property generateBootstrapVersionRule could be set to false and as per logic here, it should not emit BootstrapVersion CfnParameter and CheckBootstrapVersion CfnRule (these are added here).
      Unsure if this should be used since bootstrap version should always be validated.
    • Per Roles section at The bootstrapping template contract, the DefaultStackSynthesizer requires five IAM roles for five different purposes. For your use case, since the issue happens during deployment process, in your AWS environment, you could add necessary IAM permissions to the cloudFormationExecutionRole, which per above documentation is ARN of the role passed to CloudFormation to execute the deployments. and has format like arn:${AWS::Partition}:iam::${AWS::AccountId}:role/cdk-${Qualifier}-cfn-exec-role-${AWS::AccountId}-${AWS::Region} (here the default value for ${Qualifier} is hnb659fds) deployment role created by CDK bootstrapping process. You could refer IAM permissions for using AWS default keys and customer managed keys for example policy.

I would rather go with 2nd option to add necessary IAM permissions. Hope it unblocks you.

On a side note, is there any specific reason for which you would you like to make the SSM parameter /cdk-bootstrap/<blabla>/version encrypted as it's not sensitive data and you will need to manage a key for that.

Thanks,
Ashish

@ashishdhingra ashishdhingra added response-requested Waiting on additional info and feedback. Will move to "closing-soon" in 7 days. and removed investigating This issue is being investigated and/or work is in progress to resolve the issue. labels May 9, 2024
@maddyexplore
Copy link
Author

maddyexplore commented May 10, 2024

I tried both option but still failed,

❌ Deployment failed: Error: Stack: SSM parameter /cdk-bootstrap/hnb659fds/version not a number:

@github-actions github-actions bot removed the response-requested Waiting on additional info and feedback. Will move to "closing-soon" in 7 days. label May 10, 2024
@ashishdhingra
Copy link
Contributor

I tried both option but still failed,

❌ Deployment failed: Error: Stack: SSM parameter /cdk-bootstrap/hnb659fds/version not a number:

@maddyexplore I would need to investigate the workaround (above and if any) at my end. In the meanwhile, could you please share if there any specific reason for which you would you like to make the SSM parameter /cdk-bootstrap/<qualifier>/version encrypted as it's not sensitive data and you will need to manage a key for that?

Thanks,
Ashish

@ashishdhingra
Copy link
Contributor

Investigation:

  • Recreated parameter /cdk-bootstrap/hnb659fds/version as SecureString with value 20 (previous value for String parameter).
  • Tried to redeploy stack. Got below error:
    ❌  TypescriptStack failed: Error: TypescriptStack: SSM parameter /cdk-bootstrap/hnb659fds/version not a number: AQICAHhR+jDBT8khUi2kIUiYp9ASFV9fu/hR4Kd0IcVviBHv4QH0RI0XMNm+356Jc2K9pcn1AAAAYDBeBgkqhkiG9w0BBwagUTBPAgEAMEoGCSqGSIb3DQEHATAeBglghkgBZQMEAS4wEQQM8DFLR3BB/zQQ9UzjAgEQgB13/bQdv8d3PJdqZ1SQpVe8tBkPBRgS3LWIk/9rfQ==
        at Deployments.validateBootstrapStackVersion (/usr/local/lib/node_modules/aws-cdk/lib/index.js:438:12210)
        at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
        at async Deployments.deployStack (/usr/local/lib/node_modules/aws-cdk/lib/index.js:438:5821)
        at async Object.deployStack2 [as deployStack] (/usr/local/lib/node_modules/aws-cdk/lib/index.js:438:199515)
        at async /usr/local/lib/node_modules/aws-cdk/lib/index.js:438:181237
    
     ❌ Deployment failed: Error: TypescriptStack: SSM parameter /cdk-bootstrap/hnb659fds/version not a number: AQICAHhR+jDBT8khUi2kIUiYp9ASFV9fu/hR4Kd0IcVviBHv4QH0RI0XMNm+356Jc2K9pcn1AAAAYDBeBgkqhkiG9w0BBwagUTBPAgEAMEoGCSqGSIb3DQEHATAeBglghkgBZQMEAS4wEQQM8DFLR3BB/zQQ9UzjAgEQgB13/bQdv8d3PJdqZ1SQpVe8tBkPBRgS3LWIk/9rfQ==
        at Deployments.validateBootstrapStackVersion (/usr/local/lib/node_modules/aws-cdk/lib/index.js:438:12210)
        at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
        at async Deployments.deployStack (/usr/local/lib/node_modules/aws-cdk/lib/index.js:438:5821)
        at async Object.deployStack2 [as deployStack] (/usr/local/lib/node_modules/aws-cdk/lib/index.js:438:199515)
        at async /usr/local/lib/node_modules/aws-cdk/lib/index.js:438:181237
    
    TypescriptStack: SSM parameter /cdk-bootstrap/hnb659fds/version not a number: AQICAHhR+jDBT8khUi2kIUiYp9ASFV9fu/hR4Kd0IcVviBHv4QH0RI0XMNm+356Jc2K9pcn1AAAAYDBeBgkqhkiG9w0BBwagUTBPAgEAMEoGCSqGSIb3DQEHATAeBglghkgBZQMEAS4wEQQM8DFLR3BB/zQQ9UzjAgEQgB13/bQdv8d3PJdqZ1SQpVe8tBkPBRgS3LWIk/9rfQ==
    
  • Modified bin\typescript.ts (project name is typescript):
    const app = new cdk.App();
    new TypescriptStack(app, 'TypescriptStack', { 
      synthesizer: new cdk.DefaultStackSynthesizer({
        generateBootstrapVersionRule: false
      })
    });
    This didn't generate the rule in CloudFormation template, but gave the below error during cdk deploy:
    ✨  Synthesis time: 4.49s
    
    
     ❌ Deployment failed: Error: TypescriptStack: SSM parameter /cdk-bootstrap/hnb659fds/version not a number: AQICAHhR+jDBT8khUi2kIUiYp9ASFV9fu/hR4Kd0IcVviBHv4QH0RI0XMNm+356Jc2K9pcn1AAAAYDBeBgkqhkiG9w0BBwagUTBPAgEAMEoGCSqGSIb3DQEHATAeBglghkgBZQMEAS4wEQQM8DFLR3BB/zQQ9UzjAgEQgB13/bQdv8d3PJdqZ1SQpVe8tBkPBRgS3LWIk/9rfQ==
      at Deployments.validateBootstrapStackVersion (/usr/local/lib/node_modules/aws-cdk/lib/index.js:438:12210)
      at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
      at async Deployments.buildSingleAsset (/usr/local/lib/node_modules/aws-cdk/lib/index.js:438:10975)
      at async Object.buildAsset (/usr/local/lib/node_modules/aws-cdk/lib/index.js:438:197148)
      at async /usr/local/lib/node_modules/aws-cdk/lib/index.js:438:181290
    
    TypescriptStack: SSM parameter /cdk-bootstrap/hnb659fds/version not a number: AQICAHhR+jDBT8khUi2kIUiYp9ASFV9fu/hR4Kd0IcVviBHv4QH0RI0XMNm+356Jc2K9pcn1AAAAYDBeBgkqhkiG9w0BBwagUTBPAgEAMEoGCSqGSIb3DQEHATAeBglghkgBZQMEAS4wEQQM8DFLR3BB/zQQ9UzjAgEQgB13/bQdv8d3PJdqZ1SQpVe8tBkPBRgS3LWIk/9rfQ==
    
    • Reverted the changes made in previous step. Modified IAM role cdk-hnb659fds-deploy-role-<ACCOUNT_ID>-<REGION> to add the following permission:
    {
        "Version": "2012-10-17",
        "Statement": [
            {
                "Sid": "VisualEditor0",
                "Effect": "Allow",
                "Action": [
                    "kms:Decrypt",
                    "kms:Encrypt",
                    "kms:GenerateDataKey"
                ],
                "Resource": "<KMS_KEY_ARN>"
            }
        ]
    }
    Running cdk deploy still gave the above error.

Finding:
Following stack trace, it appears error is thrown at at Deployments.validateBootstrapStackVersion (/usr/local/lib/node_modules/aws-cdk/lib/api/deployments.ts:665:13). Code path is buildSingleAsset() > validateBootstrapStackVersion() > validateVersion() > versionFromSsmParameter().

The method versionFromSsmParameter() uses AWS JS SDK SSM.getParameter() without using the WithDecryption flag/option. Hence, encrypted value for SecureString is retrieved from SSM (in other words, it is not decrypted automatically).

The document Read Systems Manager values at synthesis time mentions that Only plain Systems Manager strings may be retrieved. Secure strings cannot be retrieved. The latest version will always be returned. Specific versions cannot be requested..

Need to discuss with Core CLI team.

@ashishdhingra ashishdhingra added investigating This issue is being investigated and/or work is in progress to resolve the issue. p2 effort/small Small work item – less than a day of effort labels May 10, 2024
@maddyexplore
Copy link
Author

I tried both option but still failed,
❌ Deployment failed: Error: Stack: SSM parameter /cdk-bootstrap/hnb659fds/version not a number:

@maddyexplore I would need to investigate the workaround (above and if any) at my end. In the meanwhile, could you please share if there any specific reason for which you would you like to make the SSM parameter /cdk-bootstrap/<qualifier>/version encrypted as it's not sensitive data and you will need to manage a key for that?

Even though its not a sensitive data, we thought that storing it in a secure way since SSM parameter supports the SecureString

would appreciate the quicker fix for this

@ashishdhingra
Copy link
Contributor

I tried both option but still failed,
❌ Deployment failed: Error: Stack: SSM parameter /cdk-bootstrap/hnb659fds/version not a number:

@maddyexplore I would need to investigate the workaround (above and if any) at my end. In the meanwhile, could you please share if there any specific reason for which you would you like to make the SSM parameter /cdk-bootstrap/<qualifier>/version encrypted as it's not sensitive data and you will need to manage a key for that?

Even though its not a sensitive data, we thought that storing it in a secure way since SSM parameter supports the SecureString

would appreciate the quicker fix for this

@maddyexplore Good morning. Per review from the CDK team, this is not an issue. Resources created by bootstrap template should not be manually altered. Doing so may lead to unexpected failures, as observed in this issue.

Thanks,
Ashish

@ashishdhingra ashishdhingra added response-requested Waiting on additional info and feedback. Will move to "closing-soon" in 7 days. and removed p2 investigating This issue is being investigated and/or work is in progress to resolve the issue. effort/small Small work item – less than a day of effort labels May 13, 2024
@TheRealAmazonKendra TheRealAmazonKendra removed the bug This issue is a bug. label May 13, 2024
Copy link

This issue has not received a response in a while. If you want to keep this issue open, please leave a comment below and auto-close will be canceled.

@github-actions github-actions bot added closing-soon This issue will automatically close in 4 days unless further comments are made. closed-for-staleness This issue was automatically closed because it hadn't received any attention in a while. and removed closing-soon This issue will automatically close in 4 days unless further comments are made. labels May 15, 2024
@aws-cdk-automation
Copy link
Collaborator

Comments on closed issues and PRs are hard for our team to see. If you need help, please open a new issue that references this one.

@aws aws locked as resolved and limited conversation to collaborators Jul 25, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
@aws-cdk/aws-ssm Related to AWS Systems Manager closed-for-staleness This issue was automatically closed because it hadn't received any attention in a while. response-requested Waiting on additional info and feedback. Will move to "closing-soon" in 7 days.
Projects
None yet
Development

No branches or pull requests

4 participants