CDK deploy not detecting credentials when stack contains assets

[chris-leach]

I followed the CDK Python workshop as far as adding a Lambda function (Hello Lambda), and cdk deploy fails with the error message The environment aws://unknown-account/us-east-2 doesn't have the CDK toolkit stack (CDKToolkit) installed. Clearly this is because the stack now includes assets, but the CDKToolkit stack is up:

$ aws cloudformation describe-stacks --stack-name CDKToolkit
{
    "Stacks": [
        {
            "StackId": "arn:aws:cloudformation:eu-west-2:86XXXXXXXXXX:stack/CDKToolkit/73949470-fbcf-11e9-9a4f-06b0998bdefc",
            "StackName": "CDKToolkit",
            "Description": "The CDK Toolkit Stack. It was created by `cdk bootstrap` and manages resources necessary for managing your Cloud Applications with AWS CDK.",
            "CreationTime": "2019-10-31T11:13:22.776Z",
            "LastUpdatedTime": "2019-10-31T11:13:29.212Z",
            "StackStatus": "CREATE_COMPLETE",
            ...
        }
    ]
}

"unknown-account/us-east-2" implies that CDK is not detecting the account and region from ~/.aws, but it deployed successfully on the previous step (without any assets), and the verbose log (see below) seems to indicate that it is detecting the account and region, at least initally.

Reproduction Steps

As above - sample-app with hello-world lambda added.

~/.aws/config:

[default]
region = eu-west-2
ca_bundle = /etc/ssl/certs/ca-certificates.crt

~/.aws/credentials:

[default]
aws_secret_access_key = XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
aws_access_key_id = XXXXXXXXXXXXXXXXXXX

Error Log

$ cdk deploy hello-cdk-1 -v --require-approval never
CDK toolkit version: 1.15.0 (build bdbe3aa)
Command line arguments: { _: [ 'deploy' ],
  v: true,
  verbose: true,
  'require-approval': 'never',
  requireApproval: 'never',
  'ignore-errors': false,
  ignoreErrors: false,
  json: false,
  j: false,
  ec2creds: undefined,
  i: undefined,
  'version-reporting': undefined,
  versionReporting: undefined,
  'path-metadata': true,
  pathMetadata: true,
  'asset-metadata': true,
  assetMetadata: true,
  'role-arn': undefined,
  r: undefined,
  roleArn: undefined,
  staging: true,
  'no-color': false,
  noColor: false,
  'build-exclude': [],
  E: [],
  buildExclude: [],
  ci: false,
  '$0': 'cdk',
  STACKS: [ 'hello-cdk-1' ],
  stacks: [ 'hello-cdk-1' ] }
Determining whether we're on an EC2 instance.
Does not look like EC2 instance.
cdk.json: {
  "app": "python3 app.py"
}
merged settings: { versionReporting: true,
  pathMetadata: true,
  output: 'cdk.out',
  app: 'python3 app.py',
  context: {},
  tags: [],
  assetMetadata: true,
  requireApproval: 'never',
  toolkitBucket: {},
  staging: true }
Setting "CDK_DEFAULT_REGION" environment variable to eu-west-2
Resolving default credentials
Retrieved account ID 86XXXXXXXXXX from disk cache
Setting "CDK_DEFAULT_ACCOUNT" environment variable to 86XXXXXXXXXX
context: { 'aws:cdk:enable-path-metadata': true,
  'aws:cdk:enable-asset-metadata': true }
outdir: cdk.out
env: { CDK_DEFAULT_REGION: 'eu-west-2',
  CDK_DEFAULT_ACCOUNT: '86XXXXXXXXXX',
  CDK_CONTEXT_JSON: '{"aws:cdk:enable-path-metadata":true,"aws:cdk:enable-asset-metadata":true}',
  CDK_OUTDIR: 'cdk.out',
  CDK_CLI_ASM_VERSION: '1.10.0',
  CDK_CLI_VERSION: '1.15.0' }
hello-cdk-1: deploying...
Using default AWS SDK credentials for account 86XXXXXXXXXX
Waiting for stack CDKToolkit to finish creating or updating...
(node:26542) Warning: Ignoring extra certs from `/etc/ssl/certs/ca-certificates.crt`, load failed: error:0B07C065:x509 certificate routines:X509_STORE_add_cert:cert already in hash table

Stack CDKToolkit does not exist
The environment aws://unknown-account/us-east-2 doesn't have the CDK toolkit stack (CDKToolkit) installed. Use cdk bootstrap "aws://unknown-account/us-east-2" to setup your environment for use with the toolkit.

 ❌  hello-cdk-1 failed: Error: This stack uses assets, so the toolkit stack must be deployed to the environment (Run "cdk bootstrap aws://unknown-account/us-east-2")
This stack uses assets, so the toolkit stack must be deployed to the environment (Run "cdk bootstrap aws://unknown-account/us-east-2")
Error: This stack uses assets, so the toolkit stack must be deployed to the environment (Run "cdk bootstrap aws://unknown-account/us-east-2")
    at Object.prepareAssets (/usr/lib/node_modules/aws-cdk/lib/assets.ts:24:11)
    at Object.deployStack (/usr/lib/node_modules/aws-cdk/lib/api/deploy-stack.ts:50:24)
    at CloudFormationDeploymentTarget.deployStack (/usr/lib/node_modules/aws-cdk/lib/api/deployment-target.ts:68:12)
    at <anonymous>
    at process._tickDomainCallback (internal/process/next_tick.js:229:7)

Environment

• CLI Version: aws-cli/1.16.270 Python/3.6.9 Linux/4.4.0-62-generic botocore/1.13.6
• Framework Version: 1.15.0 (build bdbe3aa)
• OS: Ubuntu 16.04.6 LTS
• Language: Python 3.6.9

Other

Possibly related to #2853 - another regression?

---

This is 🐛 Bug Report

[rhboyd]

I know this won't fix your issue, but running cdk bootstrap will properly bootstrap the account used by your default profile. It will then work correctly for asset deployments after that.

[chris-leach]

Feeling extremely silly now, but I figured it out. Earlier in the tutorial, the env is explicitly set (MyStack(app, "hello-cdk-1", env={'region': 'us-east-2'})), so the stack is not in fact region-agnostic as I had assumed. This also explains the unknown account. Deleting the env from the stack definition fixes the issue.

[serg06]

cdk bootstrap kept failing for me with "... already exists". I was able to fix it by specifying a custom qualifier for cdk bootstrap and in my cdk.json.