feat(app-delivery): continuous delivery for CDK apps

packages/@aws-cdk/app-delivery/README.md

@@ -1,154 +1,132 @@
-## Continuous Integration / Continuous Delivery for CDK Applications
-This library includes a *CodePipeline* composite Action for deploying AWS CDK Applications.
+# App Delivery
 
-This module is part of the [AWS Cloud Development Kit](https://github.com/awslabs/aws-cdk) project.
+> **Experimental**
 
-### Limitations
-The construct library in it's current form has the following limitations:
-1. It can only deploy stacks that are hosted in the same AWS account and region as the *CodePipeline* is.
-2. Stacks that make use of `Asset`s cannot be deployed successfully.
+Continuous delivery for AWS CDK apps.
 
-### Getting Started
-In order to add the `PipelineDeployStackAction` to your *CodePipeline*, you need to have a *CodePipeline* artifact that
-contains the result of invoking `cdk synth -o <dir>` on your *CDK App*. You can for example achieve this using a
-*CodeBuild* project.
+## Overview
 
-The example below defines a *CDK App* that contains 3 stacks:
-* `CodePipelineStack` manages the *CodePipeline* resources, and self-updates before deploying any other stack
-* `ServiceStackA` and `ServiceStackB` are service infrastructure stacks, and need to be deployed in this order
+The app delivery solution for AWS CDK apps is based on the idea of a
+**bootstrap pipeline**. It's an AWS CodePipeline which monitors your source
+control branch for changes, picks them up, builds them and runs `cdk deploy`
+against a set of stacks from your application (by default it will simply deploy
+all stacks).
 
+The bootstrap pipeline may be sufficient for simple applications that do not
+require customization of their deployment process. However, this solution can be
+extended using **deployment pipelines** to allow users to define arbitrary
+CodePipeline models which can deploy complex applications across regions and
+accounts.
+
+## Bootstrap Pipeline
+
+Normally, you will set up a single bootstrap pipeline per CDK app, which is
+bound to the source control repository in which you store your application.
+
+The `cdk-pipeline` program, which is included in this module can be used to create/update
+bootstrap pipelines in your account.
+
+To use it, create a file called `cdk.pipelines.yaml` with a map where the key is
+the name of the bootstrap pipeline and the value is an object with the following options:
+
+* `source`: the GitHub repository to monitor. Must be in the form **http://github.com/ACCOUNT/REPO**.
+* `oauthSecret`: the ARN of an AWS Secrets Manager secret that contains the GitHub OAuth key.
+* `branch` (optional): branch to use (default is `master`)
+* `workdir` (optional): the directory in which to run the build command (defaults to the root of the repository).
+* `stacks` (optional): array of stack names to deploy (defaults to all stacks not marked `autoDeploy: false`).
+* `environment` (optional): the CodeBuild environment to use (defaults to node.js 10.1)
+* `install` (optional): install command (defaults: `npm install`)
+* `build` (optional): build command (defaults: `npm run build && npm test`)
+* `version` (optional): semantic version requirement of the CDK CLI to use for deployment (defaults: `latest`)
+
+Here's an example for the bootstrap pipeline for the [CDK workshop](https://github.com/aws-samples/aws-cdk-intro-workshop):
+
+```yaml
+cdk-workshop:
+  source: https://github.com/aws-samples/aws-cdk-intro-workshop
+  oauthSecret: arn:aws:secretsmanager:us-east-1:111111111111:secret:github-token-aaaaa
+  workdir: code/typescript
 ```
-  ┏━━━━━━━━━━━━━━━━┓  ┏━━━━━━━━━━━━━━━━┓  ┏━━━━━━━━━━━━━━━━━┓  ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
-  ┃     Source     ┃  ┃     Build      ┃  ┃  Self-Update    ┃  ┃             Deploy              ┃
-  ┃                ┃  ┃                ┃  ┃                 ┃  ┃                                 ┃
-  ┃ ┌────────────┐ ┃  ┃ ┌────────────┐ ┃  ┃ ┌─────────────┐ ┃  ┃ ┌─────────────┐ ┌─────────────┐ ┃
-  ┃ │   GitHub   ┣━╋━━╋━▶ CodeBuild  ┣━╋━━╋━▶Deploy Stack ┣━╋━━╋━▶Deploy Stack ┣━▶Deploy Stack │ ┃
-  ┃ │            │ ┃  ┃ │            │ ┃  ┃ │PipelineStack│ ┃  ┃ │ServiceStackA│ │ServiceStackB│ ┃
-  ┃ └────────────┘ ┃  ┃ └────────────┘ ┃  ┃ └─────────────┘ ┃  ┃ └─────────────┘ └─────────────┘ ┃
-  ┗━━━━━━━━━━━━━━━━┛  ┗━━━━━━━━━━━━━━━━┛  ┗━━━━━━━━━━━━━━━━━┛  ┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛
-```
 
-#### `index.ts`
-
-```typescript
-import codebuild = require('@aws-cdk/aws-codebuild');
-import codepipeline = require('@aws-cdk/aws-codepipeline');
-import codepipeline_actions = require('@aws-cdk/aws-codepipeline-actions');
-import cdk = require('@aws-cdk/cdk');
-import cicd = require('@aws-cdk/cicd');
-
-const app = new cdk.App();
-
-// We define a stack that contains the CodePipeline
-const pipelineStack = new cdk.Stack(app, 'PipelineStack');
-const pipeline = new codepipeline.Pipeline(pipelineStack, 'CodePipeline', {
-  // Mutating a CodePipeline can cause the currently propagating state to be
-  // "lost". Ensure we re-run the latest change through the pipeline after it's
-  // been mutated so we're sure the latest state is fully deployed through.
-  restartExecutionOnUpdate: true,
-  /* ... */
-});
-
-// Configure the CodePipeline source - where your CDK App's source code is hosted
-const source = new codepipeline_actions.GitHubSourceAction({
-  actionName: 'GitHub',
-  /* ... */
-});
-pipeline.addStage({
-  name: 'source',
-  actions: [source],
-});
-
-const project = new codebuild.PipelineProject(pipelineStack, 'CodeBuild', {
-  /** 
-  * Choose an environment configuration that meets your use case.
-  * For NodeJS, this might be:
-  *
-  * environment: {
-  *   buildImage: codebuild.LinuxBuildImage.UBUNTU_14_04_NODEJS_10_1_0,
-  * },
-  */
-});
-const buildAction = new codepipeline_actions.CodeBuildBuildAction({
-  actionName: 'CodeBuild',
-  project,
-  inputArtifact: source.outputArtifact,
-});
-pipeline.addStage({
-  name: 'build',
-  actions: [buildAction],
-});
-const synthesizedApp = buildAction.outputArtifact;
-
-// Optionally, self-update the pipeline stack
-const selfUpdateStage = pipeline.addStage({ name: 'SelfUpdate' });
-new cicd.PipelineDeployStackAction(pipelineStack, 'SelfUpdatePipeline', {
-  stage: selfUpdateStage,
-  stack: pipelineStack,
-  inputArtifact: synthesizedApp,
-});
-
-// Now add our service stacks
-const deployStage = pipeline.addStage({ name: 'Deploy' });
-const serviceStackA = new MyServiceStackA(app, 'ServiceStackA', { /* ... */ });
-// Add actions to deploy the stacks in the deploy stage:
-const deployServiceAAction = new cicd.PipelineDeployStackAction(pipelineStack, 'DeployServiceStackA', {
-  stage: deployStage,
-  stack: serviceStackA,
-  inputArtifact: synthesizedApp,
-  // See the note below for details about this option.
-  adminPermissions: false, 
-});
-// Add the necessary permissions for you service deploy action. This role is
-// is passed to CloudFormation and needs the permissions necessary to deploy
-// stack. Alternatively you can enable [Administrator](https://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies_job-functions.html#jf_administrator) permissions above, 
-// users should understand the privileged nature of this role.
-deployServiceAAction.addToRolePolicy(
-  new iam.PolicyStatement()
-    .addAction('service:SomeAction')
-    .addResource(myResource.myResourceArn)
-    // add more Action(s) and/or Resource(s) here, as needed
-);
-
-const serviceStackB = new MyServiceStackB(app, 'ServiceStackB', { /* ... */ });
-new cicd.PipelineDeployStackAction(pipelineStack, 'DeployServiceStackB', {
-  stage: deployStage,
-  stack: serviceStackB,
-  inputArtifact: synthesizedApp,
-  createChangeSetRunOrder: 998,
-  adminPermissions: true, // no need to modify the role with admin
-}); 
+Next, use the `cdk-pipeline` command to create/update this bootsrapping pipeline
+into your account (assumes you have the CDK CLI installed on your system).
+
+```console
+$ npx -p @aws-cdk/app-delivery cdk-pipeline
 ```
 
-#### `buildspec.yml`
-The repository can contain a file at the root level named `buildspec.yml`, or
-you can in-line the buildspec. Note that `buildspec.yaml` is not compatible.
-
-For example, a *TypeScript* or *Javascript* CDK App can add the following `buildspec.yml`
-at the root of the repository:
-
-```yml
-version: 0.2
-phases:
-  install:
-    commands:
-      # Installs the npm dependencies as defined by the `package.json` file
-      # present in the root directory of the package
-      # (`cdk init app --language=typescript` would have created one for you)
-      - npm install
-  build:
-    commands:
-      # Builds the CDK App so it can be synthesized
-      - npm run build
-      # Synthesizes the CDK App and puts the resulting artifacts into `dist`
-      - npm run cdk synth -- -o dist
-artifacts:
-  # The output artifact is all the files in the `dist` directory
-  base-directory: dist
-  files: '**/*'
+This command will deploy a stack called `cdk-pipelines` in your AWS account,
+which will contain all the bootstrap pipelines defines in
+`cdk.pipelines.yaml`.
+
+To add/remove/update pipelines, simply update the .yaml file and re-run
+`cdk-pipelines`.
+
+This pipeline will now monitor the GitHub repository and it will deploy the
+stacks defined in your app to your account.
+
+## Deployment Pipeline
+
+As mentioned above, the bootstrap pipeline is useful for simple applications
+where you basically just want your CDK app to continuously be deployed into your
+AWS account.
+
+For more complex scenarios, such as multi-stack/multi-account/multi-region
+deployments or when you want more control over how your application is deployed,
+the CDK allows you to harness the full power of AWS CodePipeline in order to
+model complex deployment scenarios.
+
+The basic idea of **deployment pipelines** is that they are defined like any
+other stack in your CDK application (and therefore can reason about the
+structure of your application, reference resources and stacks, etc), and are
+also continuously deployed through the bootstrap pipeline.
+
+The CDK is shipped with a class called `DeploymentPipeline` which extends
+the normal `codepipeline.Pipeline` and is automatically wired to the CDK
+application produced from your bootstrap pipeline.
+
+To deploy CDK stacks from your application through a deployment pipeline, you
+can simply add a `DeployStackAction` to your pipeline.
+
+The following is a CDK application that consists of two stacks (`workshop-stack`
+and `random-stack`) which are deployed in parallel by the application pipeline:
+
+```ts
+class MyAppPipeline extends Stack {
+  constructor(scope: Construct, id: string, props: StackProps) {
+    super(scope, id, props);
+
+    new codepipelinePipeline(this, 'Pipeline', {
+      bootstrap: 'cdk-workshop',
+      stages: [
+        {
+          name: 'Deploy',
+          actions: [
+            new CdkDeployAction({ stacks: [ new WorkshopStack(app, 'workshop-stack').name ], admin: true }),
+            new CdkDeployAction({ stack: [ new RandomStack(app, 'random-stack').name ], admin: true })
+          ]
+        }
+      ]
+    });
+  }
+}
+
+const app = new App();
+new MyAppPipeline(app, 'workshop-app-pipeline');
 ```
 
-The `PipelineDeployStackAction` expects it's `inputArtifact` to contain the result of 
-synthesizing a CDK App using the `cdk synth -o <directory>`.
+We would need to modify our `cdk.pipelines.yaml` file to only deploy the
+`workshop-app-pipeline` (because the other two stacks are now deployed by our
+deployment pipeline):
+
+```yaml
+cdk-workshop:
+  source: https://github.com/aws-samples/aws-cdk-intro-workshop
+  oauthSecret: arn:aws:secretsmanager:us-east-1:111111111111:secret:github-token-aaaaa
+  workdir: code/typescript
+  stacks: [ 'workshop-app-pipeline ]
+```
 
+## TODO
 
+- [ ] Should we automatically set `autoDeploy` to false if a stack is associated with a `DeployStackAction`.

packages/@aws-cdk/app-delivery/bin/cdk-pipeline

@@ -0,0 +1,4 @@
+#!/bin/bash
+set -euo pipefail
+scriptdir=$(cd $(dirname $0) && pwd)
+exec cdk -a ${scriptdir}/../bootstrap-app/app.js deploy

packages/@aws-cdk/app-delivery/bootstrap-app/app.ts

@@ -0,0 +1,31 @@
+import cdk = require('@aws-cdk/cdk');
+import fs = require('fs');
+import yaml = require('yaml');
+import { BootstrapPipeline, BootstrapPipelineProps } from './pipeline';
+
+const config = readConfig();
+const app = new cdk.App();
+
+for (const [ id, props ] of Object.entries(config)) {
+  const stack = new cdk.Stack(app, `cdk-bootstrap-${id}`);
+  new BootstrapPipeline(stack, id, props);
+}
+
+interface Config {
+  [name: string]: BootstrapPipelineProps
+}
+
+function readConfig(): Config {
+  const files = [
+    'cdk.pipelines.yaml',
+    'cdk.pipelines.json'
+  ];
+
+  for (const file of files) {
+    if (fs.existsSync(file)) {
+      return yaml.parse((fs.readFileSync(file, 'utf-8')));
+    }
+  }
+
+  throw new Error(`Unable to find pipeline configuration in one of: ${files.join(', ')}`);
+}