Make the CLI experience of multi-account CDK apps awesome

[skinny85]

After all of the issues/PRs for cross-account CodePipelines are delivered (#3208, #3323 , #3387, #3388, #3389 ), we will have a great story for making it very easy to work with cross-account CodePipelines.

However, we also need to make it easy to work with cross-account CDK apps from the perspective of authentication from the command line. Most likely, this will involve a deeper integration with AWS profiles defined in the ~/.aws/credentials / ~/.aws/config files. A possible solution would be to allow passing profile when creating a Stack:

new Stack(app, 'Stack1', {
  env: {
     profile: 'account1',
  },
});

The ideal customer experience we want here is to be able to say:

$ cdk deploy '*'

, and all of the different credentials for the different accounts the Stacks belong to will be automatically wired together.

[skinny85]

An interesting point: the CloudFormation cross-account actions defined in #3208 create a new stack that contains the IAM roles needed to deploy into the target account. How do we want to model authorization for that use-case? Maybe passing the profile in Stack's env like above is not a great idea; perhaps we need some sort of configurable account-to-profile mapping instead?

[tader]

(Probably an option to specify the CloudFormation role ARN should also be added.)

[skinny85]

(Probably an option to specify the CloudFormation role ARN should also be added.)

That's already possible (you can pass a role when creating the action).

[tader]

I don’t think it is in the context of this issue, i.e., using profiles for various accounts. Each account would require its own CF role.

…

(Probably an option to specify the CloudFormation role ARN should also be added.) That's already possible (you can pass a role when creating the action).

[skinny85]

I'm pretty sure the credentials that the profiles include refer to an IAM user or role already, so I'm not sure what "specifying a role" would mean in this context.

[tader]

As a developer I could only have the permission to start a CloudFormation deploy. I can tell CloudFormation to assume an other role with elevated permissions. See the `—role-arn` option of the cdk command

…

I'm pretty sure the credentials that the profiles include refer to an IAM user or role already, so I'm not sure what "specifying a role" would mean in this context.

[skinny85]

Ah, got it. Thanks for the explanation.

Yes, that's something that this feature should definitely include.

[slotnick]

It could be as easy as specifying an optional .aws/credentials profile in the Environment, in addition to region and account.

[btotharye]

Is the wildcard option truly working?

cdk deploy * -c stage=dev
No stack found matching 'README.md'. Use "list" to print manifest

Seems to do this no matter what I do I only tried wildcards cause the cdk deploy command mentions it.

[skinny85]

Yes, this is a common gotcha. The "No stack found matching 'README.md'" should be a hint :).

The * is expanded by your shell, by default, to a list of all files in the current directory. To pass the * into the CDK, you need to escape it; so either:

cdk deploy \* -c stage=dev

or

cdk deploy '*' -c stage=dev

[btotharye]

Yes, this is a common gotcha. The "No stack found matching 'README.md'" should be a hint :).

The * is expanded by your shell, by default, to a list of all files in the current directory. To pass the * into the CDK, you need to escape it; so either:

cdk deploy \* -c stage=dev

or

cdk deploy '*' -c stage=dev

ha thanks good to know

[tijoer]

Thanks for this.
Will this also fix the detection for non existing circular references like #3300 ?

[eladb]

I am curious what people think about this idea which came up as part of the CI/Cd design: My current thinking is that when you bootstrap an environment using the cdk you will be able to specify a trusted account that will be able to deploy into this environment and then the cli will just assume a role in the target and deploy into it. No profiles needed...

[tijoer]

There will be two scenarios where CDK is used:

• Developers deploying infrastructure
• Pipelines deploying infrastructure

If we can attach a policy to pipelines that sufficient rights are granted everything would be fine. The same should go for developers. Let me define a role that allows them to execute cloudformation in each account and make the generation of cloudformation possible without cyclic reference checking.

Using profiles would be a pain in the long way, as I already have three different entries in my ~/.aws/credentials file for some accounts right now and the other developers have named their credentials differently, however we try to enforce a common naming scheme right now. I never liked profiles and they feel kinda off, but I can understand why they are needed.

env: {
  account: '123456789',
}

should be better for tooling then

env: {
     profile: 'account1', //might change for each developer
}

[bgdnlp]

I wouldn't assume too much about how the CDK is used, tends to force people into patterns that might not apply to them. Defining a trusted account sounds great as an option and applies to many use cases I can think of, but it may be a pain in more restricted environments, where whoever does the deploying doesn't have enough access to also establish a trust between accounts. I think that the original idea of deeper integration with existing AWS authentication mechanism is right, because it's in line with how other AWS tools work, like aws-cli and the SDKs.

Maybe the CDK needs a local configuration file/option that isn't packaged with the app. One that is specific to whoever is currently using the CDK. Similar to profiles, actually. It could even be stored in the same directory, ~/.aws/cdk. And in that file the user could say "when deploying or synthesizing for this account, use this profile, or these credentials. also, define these account-specific variables, which are like cloudformation parameters".

This is almost possible now. What I currently do is use context variables, from cdk.json and command line. In cdk.json I have a structure like:

{
  "accounts":{
    "account_id":"12345",
    "regions":{
      "region":"eu-west-1",
      "environments":[
        {
          "env":"staging",
          "load_balancer":"arn:aws:etc",
          "subnets":[
            "snet-1234",
            "snet-5432"
          ]
        },
        {
          "env":"production",
          "load_balancer":"arn:aws:etc",
          "subnets":[
            "snet-7890",
            "snet-0987"
          ]
        }
      ]
    }
  }
}

And then in code I create the stack by looping through the above, like:

for environ in environments:
  stack = Stack(app, 'MyStack-' + environ)

I also mix SDK and CDK (because in Python it's easy) and need to know in code which profile is in use, so I run the CDK with:

cdk --profile myprofile --context profile=myprofile destroy ecs-prd

But:

• I can't use multiple profiles in the same command, so I need to run the cdk once for each account. Which I do manually, but it could be scripted.
• cdk.json gets stored in version control. In some cases it's exactly what I want, because I'm working on an app that is specific to this company/team/environment. In other cases I might want to package the app for some team that uses completely different accounts, so I don't want these defined in cdk.json.

Because I mix the CDK and SDK in the same code, I'd like the ability to pass the credentials used by the CDK to the SDK. But that can be worked around.

Because of the second point above, I'm thinking that maybe ~/.aws/cdk should have the same format as cdk.json and if the same variable is defined in both, then ~/aws/cdk should take precedence and overwrite what's in cdk.json. In which case the local file should probably be named ~/.aws/cdk.json.

Again, the option of defining a trusted account and sidestepping all this personal configuration sounds great, as long as it's not the only option.

[davidbarsky]

Commenting to register my support in making multi-account support better. I'm also interested in seeing how this could be used to support cellular architectures, where new accounts can be spun up, on the fly, using a service like AWS Organizations.

I suspect that designing this feature around supporting cellular accounts might have a rising-tide effect for less demanding use-cases.

[gsdwait]

Can the example from the aws-events overview section on Cross-account targets actually be run from the command line using cdk CLI? I'm trying to do something similar to this with cross-account targets and I don't know how to access two different accounts in a single pass. This feature is to make the experience awesome. I'd be OK right now with any experience that works.

https://docs.aws.amazon.com/cdk/api/latest/docs/aws-events-readme.html#cross-account-targets

[saltman424]

I came here with exactly the same question as @gsdwait

Any info?

[faridux]

I also face the challenge, in organisation we centalize the logs,notification and alarms. Setup in one shot the tooling for client and admin account will ease a lot the deployement.
Congrats for the CDK tool by the way, infra as an app!

[hupe1980]

@saltman424 @faridux We solved it with a plugin: https://github.com/hupe1980/cdk-multi-profile-plugin

[pcgeek86]

It would be great if we could specify an IAM Role to assume in the Environment object.

Example

from aws_cdk.core import App, Environment
from awscdktest.awscdktest_stack import AwscdktestStack

env = Environment(region="us-west-2", assume_role='arn:aws:iam:us-west-2:000000000000:rolename')
app = App()
AwscdktestStack(app, "awscdktest", env=env)
app.synth()

[lazharichir]

Can the example from the aws-events overview section on Cross-account targets actually be run from the command line using cdk CLI? I'm trying to do something similar to this with cross-account targets and I don't know how to access two different accounts in a single pass. This feature is to make the experience awesome. I'd be OK right now with any experience that works.

https://docs.aws.amazon.com/cdk/api/latest/docs/aws-events-readme.html#cross-account-targets

The problem here is that you would need to add en EventBusPolicy which would look like that:

new events.CfnEventBusPolicy(this, `myapp-event-bus-policy`, {
	eventBusName: bus.eventBusName,
	action: `events:PutEvents`,
	principal: `*`,
	statementId: `myapp-event-bus-policy-statement`,
	condition: {
		type: `StringEquals`,
		key: `aws:PrincipalOrgID`,
		value: orgPrincipal.organizationId,
	}
})

But this causes an Internal Failure upon deploying via CDK or CloudFormation, as per aws-cloudformation/cloudformation-coverage-roadmap#156.

So kind of stuck for now when it comes to EventBridge's cross-account eventing.

[mrpackethead]

I'm working on mulit-account / multiregion deployments, with a pipeline that deploys the stacks, when i commit to a repository. My build stage does a cdk synth, the interates through the manifest.json, picks up the region/account details, and deploys the templates using a boto3.client. create_stack ( after switching to an 'automation' role ) that exisits in all my accounts..
Its worked well, but i'm faced with a 'new' problem. My deployment process works fine, for creating new stacks, or even editing them.. CF just deals to stuff. However i have got no effective way to remove a stack that already exists. If i remove a stack from my app, that wont' result in it be deleting it.
I have been toying on some different ideas on how to deal with this. I could for example register the exisitance of my stack in a small dynamodb table, when the stack is reployed, i coudl find the differnces and delete it.
I was hoping for something more 'native' than this though.

import json
import boto3
import argparse
parser = argparse.ArgumentParser(description='Get Credentials')
parser.add_argument('--profile', help="profile", default='default')
args = parser.parse_args()
session = boto3.session.Session(profile_name=args.profile)
with open('manifest.json') as manifest_file:
    data = json.load(manifest_file)
for artifact in data['artifacts']:
    if artifact == 'Tree':
        continue
    template = data['artifacts'][artifact]['properties']['templateFile']
    environment = data['artifacts'][artifact]['environment']
    region =  environment.split('/')[3]
    account = environment.split('/')[2]
    print("\n ********* Deploying Templates to accounts ***************")
    print(artifact, region, account)
    with open(template) as cftemp:
        cftemplate = json.load(cftemp)
        sts = session.client('sts')
        assumedrole = 'arn:aws:iam::' + account + ':role/TAR-Automation'           
        sessionname = 'cdkxadeployment'
        
       
    
        try:
            taskcreds = sts.assume_role(RoleArn = assumedrole,RoleSessionName = sessionname )
            tasksession = boto3.session.Session(  aws_access_key_id=taskcreds['Credentials']['AccessKeyId'],
                                                    aws_secret_access_key=taskcreds['Credentials']['SecretAccessKey'],
                                                    aws_session_token=taskcreds['Credentials']['SessionToken'] )
            
            cf = tasksession.client('cloudformation') 
            print(tasksession)
        except Exception as e:
            print('\tThe account:', account, " can not be accessed")
            print(e)
            continue  # this would occur if there is an auth issue / account does not exisit  
        try:
            stack = cf.create_stack(
                StackName = artifact,
                TemplateBody = json.dumps(cftemplate),
                Capabilities = ['CAPABILITY_NAMED_IAM'],
                OnFailure ='DELETE'
            )
            print(stack)
        except Exception as e:
            print(e)`

[brettswift]

Something as simple as introducing an 'app' tag could work but it would need to be validated as unique.

…

On Sat., Mar. 7, 2020, 22:43 Andrew, ***@***.***> wrote: I'm working on mulit-account / multiregion deployments, with a pipeline that deploys the stacks, when i commit to a repository. My build stage does a cdk synth, the interates through the manifest.json, picks up the region/account details, and deploys the templates using a boto3.client. create_stack ( after switching to an 'automation' role ) that exisits in all my accounts.. Its worked well, but i'm faced with a 'new' problem. My deployment process works fine, for creating new stacks, or even editing them.. CF just deals to stuff. However i have got no effective way to remove a stack that already exists. If i remove a stack from my app, that wont' result in it be deleting it. I have been toying on some different ideas on how to deal with this. I could for example register the exisitance of my stack in a small dynamodb table, when the stack is reployed, i coudl find the differnces and delete it. I was hoping for something more 'native' than this though. `import json import boto3 import argparse parser = argparse.ArgumentParser(description='Get Credentials') parser.add_argument('--profile', help="profile", default='default') args = parser.parse_args() session = boto3.session.Session(profile_name=args.profile) with open('manifest.json') as manifest_file: data = json.load(manifest_file) for artifact in data['artifacts']: if artifact == 'Tree': continue template = data['artifacts'][artifact]['properties']['templateFile'] environment = data['artifacts'][artifact]['environment'] region = environment.split('/')[3] account = environment.split('/')[2] print("\n ********* Deploying Templates to accounts ***************") print(artifact, region, account) with open(template) as cftemp: cftemplate = json.load(cftemp) sts = session.client('sts') assumedrole = 'arn:aws:iam::' + account + ':role/TAR-Automation' # this needs to be sessionname = 'cdkxadeployment' try: taskcreds = sts.assume_role(RoleArn = assumedrole,RoleSessionName = sessionname ) tasksession = boto3.session.Session( aws_access_key_id=taskcreds['Credentials']['AccessKeyId'], aws_secret_access_key=taskcreds['Credentials']['SecretAccessKey'], aws_session_token=taskcreds['Credentials']['SessionToken'] ) cf = tasksession.client('cloudformation') print(tasksession) except Exception as e: print('\tThe account:', account, " can not be accessed") print(e) continue # this would occur if there is an auth issue / account does not exisit try: stack = cf.create_stack( StackName = artifact, TemplateBody = json.dumps(cftemplate), Capabilities = ['CAPABILITY_NAMED_IAM'], OnFailure ='DELETE' ) print(stack) except Exception as e: print(e)` — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#3401?email_source=notifications&email_token=AAFQYFOQFGRPQSH66AWCWM3RGMWA5A5CNFSM4IGHXG32YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEOENFUY#issuecomment-596169427>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAFQYFPV3FXCBUMW6PFW2Q3RGMWA5ANCNFSM4IGHXG3Q> .

[0xdevalias]

For my use-case my credentials are for the 'root' Organisation account, and I am wanting to be able to deploy into the 'production' account linked to my organisation.

Since I use aws-vault to manage my creds (which I believe makes them available by ENV var) I don't want to have to provide any profile names, etc.

Ideally some 'automagic' assume role stuff would occur, and then the deploy would succeed as though I had directly used credentials for the production account:

• https://docs.aws.amazon.com/cli/latest/reference/sts/assume-role.html
• https://medium.com/faun/step-by-step-aws-iam-assumerole-with-aws-vault-configuration-9d5986373c33

Based on the blog post above, I can edit my config file as follows, which allows CDK to deploy easily cross-account:

~/.aws/config

[profile redacted]
region=ap-southeast-2

[profile redacted-prod]
source_profile=redacted
role_arn=arn:aws:iam::123456789012:role/OrganizationAccountAccessRole

[polothy]

If your CodeBuild has:

iam.PolicyStatement(
    actions=["sts:AssumeRole"],
    resources=[
        "arn:*:iam::*:role/*-deploy-role-*",
        "arn:*:iam::*:role/*-publishing-role-*",
    ],
)

Would be nice if CDK just assumed the the deploy-role from the other account even though current credentials are for a different account. Otherwise, I think I have to create a role over in the other account just so CDK can then assume the deploy-role.

[tysonclugg]

A more 12-factor style solution would be extending the existing AWS environment processing with a new env_prefix/AWS_ENV_PREFIX setting that allows sourcing different AWS config environment variables for each stack. For example, instead of looking for an AWS_PROFILE environment variable, the variable name is prefixed and we instead source from ${AWS_ENV_PREFIX}_AWS_PROFILE.

The CDK source would then be free of any dev specific AWS_PROFILE settings or similar:

new Stack(app, 'Production', {
  env: {
     env_prefix: 'PROD',
  },
});
new Stack(app, 'Development', {
  env: {
     env_prefix: 'DEV',
  },
});

The dev specific configs would support all existing configs, including Using an IAM role in the AWS CLI:
~/.aws/config

[profile deploy-dev]
role_arn = arn:aws:iam::523456789099:role/developmentadminrole
source_profile = user1

[profile deploy-prod]
role_arn = arn:aws:iam::123456789012:role/productionadminrole
source_profile = user1

Tying all of the above together would be applying config via prefixed AWS environment variables:

$ export PROD_AWS_PROFILE="deploy-prod"
$ export DEV_AWS_PROFILE="deploy-dev"
# now we can deploy using "the ideal customer experience":
$ cdk deploy '*'

Otherwise, simply adding profile/AWS_PROFILE into each stack's environment may be enough and drops a layer of indirection, at the expense of needing specific profiles defined in ~/.aws/config for each developer. This is not ideal when different projects (perhaps from different organisations) have overlapping profile names defined in ~/.aws/config.

[skinny85]

This was solved with new-style synthesis (the default in CDK v2).

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.

[briandefiant]

@skinny85 Is there any way for this to work with AWS CLI credential profiles? I only see the deploy_role_arn option on DefaultStackSynthesizer, but no way to specify a credential profile rather than a role ARN.

(CliCredentialsStackSynthesizer also seemed promising, but it seems to only be able to use the "current" CLI credentials, and not accept a custom credential profile name.)

[skinny85]

Yes. You need to pass the --profile option when invoking the CLI, for example cdk deploy --profile my_profile.

[briandefiant]

Got it, thank you. I'll have to script something to associate stacks with credential profiles. (I was hoping for something that supported cdk deploy --all.)

[skinny85]

You shouldn't have to - if you set things up so that your base account that you're deploying from is trusted by the target accounts (see here for details), you won't need anything beyond the profile for the base account.