Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(app-delivery): continuous delivery for CDK apps #2073

Closed
wants to merge 9 commits into from

Conversation

eladb
Copy link
Contributor

@eladb eladb commented Mar 21, 2019

Rewrite the app delivery library to allow full deployment of CDK apps. A CDK pipeline can be created using a command line tool called cdk-pipeline (which we will eventually integrate into the toolkit). This pipeline is also called "a bootstrapping pipeline". It's behavior is fixed - it builds and deploys a CDK app from a source control repo (currently only GitHub is supported).

In order to customize the deployment behavior, users can configure the bootstrapping pipeline to only deploy a deployment stack instead, which will in turn, deploy the rest of the CDK app. This approach provides full flexibility to CDK users as it offers CI/CD for both the pipeline itself and the rest of the app, with full customizability.

Fixed #308
Fixes #1917
Fixes #1151

Moral support: @NetaNir

Co-authored-by: rix0rrr huijbers@amazon.com
Co-authored-by: RomainMuller rmuller@amazon.com
Co-authored-by: skinny85 adamruka@amazon.com


Pull Request Checklist

  • Testing
    • Unit test added (prefer not to modify an existing test, otherwise, it's probably a breaking change)
    • CLI change?: coordinate update of integration tests with team
    • cdk-init template change?: coordinated update of integration tests with team
  • Docs
    • jsdocs: All public APIs documented
    • README: README and/or documentation topic updated
  • Title and Description
    • Change type: title prefixed with fix, feat will appear in changelog
    • Title: use lower-case and doesn't end with a period
    • Breaking?: last paragraph: "BREAKING CHANGE: <describe what changed + link for details>"
    • Issues: Indicate issues fixed via: "Fixes #xxx" or "Closes #xxx"
  • Sensitive Modules (requires 2 PR approvers)
    • IAM Policy Document (in @aws-cdk/aws-iam)
    • EC2 Security Groups and ACLs (in @aws-cdk/aws-ec2)
    • Grant APIs (only if not based on official documentation with a reference)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license.

@eladb eladb requested a review from a team as a code owner March 21, 2019 14:02
@eladb eladb changed the title wip: bootstapping and application pipelines wip: cdk pipelines Mar 21, 2019
@eladb eladb changed the title wip: cdk pipelines feat: cdk pipelines - continuous delivery for cdk apps Mar 24, 2019
@RomainMuller RomainMuller changed the title feat: cdk pipelines - continuous delivery for cdk apps feat(cdk pipelines): Continuous delivery for CDK apps Mar 25, 2019
@eladb eladb changed the title feat(cdk pipelines): Continuous delivery for CDK apps feat(app-delivery): continuous delivery for CDK apps Mar 25, 2019
@@ -1,151 +1,132 @@
## Continuous Integration / Continuous Delivery for CDK Applications
This library includes a *CodePipeline* composite Action for deploying AWS CDK Applications.
# App Delivery
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Have we made the previous method of building a CDK pipeline impossible now, or is that code still available? We're not really explaining how it works anymore at least.

I'm concerned because existing customers could be depending on this right now, and we'd break them with no real replacement. I would like this library to both export the CDK primitives to build this pipeline, as well as a the helper tool that builds a standard pipeline using those primitives from a config file, and explain both methods in the README.

The refactor is so big I can't really tell whether that is what's going on, but if not that's what I want :).

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is basically a full replacement of the existing module. Start with the README file and judge if it offers the right building blocks. I’ll also see if we can expose more of the building blocks.

I’ll make sure to include a breaking change message in the commit. I don’t believe the current offering was of much value to anyone, especially given it did not support assets.

@@ -0,0 +1,4 @@
#!/bin/bash
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can't really do this because it won't work on Windows.

Has to be a JavaScript file.

(And it's not "just a build script", it's in the hot path of every CDK user)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good point about Windows. I’ll convert to JavaScript.

@eladb eladb mentioned this pull request Mar 26, 2019
@eladb eladb changed the base branch from master to benisrae/secrets-refactor April 2, 2019 16:18
@eladb eladb changed the base branch from benisrae/secrets-refactor to master April 2, 2019 16:18
@eladb
Copy link
Contributor Author

eladb commented Apr 2, 2019

@skinny85, I've extracted the various pipeline actions from the app-delivery module into the codepipeline-actions module (CfnSourceAction, CfnBuildAction and CfnDeployAction).

You will see that I've implemented them as constructs, which made it much easier and less convoluted. I am not super happy that those don't look & feel like the other pipeline actions, so we should re-visit once we refactor the Action API.

Let's align the terminology:

  • "Boot pipeline": the pipeline with the fixed structure that's responsible for building the app and deploying all the stacks not marked with autoDeploy: false.
  • "App pipeline": optionally, the pipeline that apps can define)

The app-delivery module now includes only the boot pipeline (and app) and a command line program that reads a yaml file and creates/updates your boot pipeline.

I am not 100% happy with the API of CdkDeployAction. It currently accepts stacks: string[] which is the low-level API we need for the boot pipeline, but for the app pipelines, I imagine users would want to be able to pass in a Stack object. I wonder if we should create CdkDeployStackAction for that purpose.

Other remaining work:

  • End-to-end test of an app with a "complex" pipeline inside it.
  • Rename the concept of bootstrapping pipeline to something nicer. I was thinking that maybe "boot pipeline" is good (just shorter).
  • Update READMEs in app-delivery and pipeline-actions so they reflect that latest APIs.
  • Integ test for app-delivery which instantiates the bootstrapping pipeline
  • Integ test for pipeline-actions (basically define a "deployment pipeline" for a CDK app with a few stacks). We need to be creative how we can integ test this...
  • Unit tests for all actions and boot pipeline.
  • Go through PR comments (up here) and address them
  • I think we should rename the tool from cdk-pipelines to cdk-boot.
  • Rebase after I merge the secrets change into master (this code currently relies on it).

@eladb
Copy link
Contributor Author

eladb commented Apr 3, 2019

seed

@eladb
Copy link
Contributor Author

eladb commented Apr 3, 2019

🌱

@skinny85 skinny85 force-pushed the benisrae/app-delivery branch from 76be772 to 71625fe Compare April 10, 2019 22:01
@skinny85
Copy link
Contributor

Did the rebase. Let's see the if the build passes.

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
To use it, create a file called `cdk.pipelines.yaml` with a map where the key is
the name of the bootstrap pipeline and the value is an object with the following options:

* `source`: the GitHub repository to monitor. Must be in the form **http://github.com/ACCOUNT/REPO**.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would CodeCommit work here?

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes works for me

@statik
Copy link
Contributor

statik commented Aug 6, 2019

This looks really cool! I'm excited to try out this feature.

@eladb
Copy link
Contributor Author

eladb commented Aug 7, 2019

Superseded by #3437

@mergify
Copy link
Contributor

mergify bot commented Sep 23, 2019

Thanks so much for taking the time to contribute to the AWS CDK ❤️

We will shortly assign someone to review this pull request and help get it
merged. In the meantime, please take a minute to make sure you follow this
checklist
:

  • PR title type(scope): text
    • type: fix, feat, refactor go into CHANGELOG, chore is hidden
    • scope: name of module without aws- or cdk- prefix or postfix (e.g. s3 instead of aws-s3-deployment)
    • text: use all lower-case, do not end with a period, do not include issue refs
  • PR Description
    • Rationale: describe rationale of change and approach taken
    • Issues: indicate issues fixed via: fixes #xxx or closes #xxx
    • Breaking?: last paragraph: BREAKING CHANGE: <describe what changed + link for details>
  • Testing
    • Unit test added. Prefer to add a new test rather than modify existing tests
    • CLI or init templates change? Re-run/add CLI integration tests
  • Documentation
    • README: update module README to describe new features
    • API docs: public APIs must be documented. Copy from official AWS docs when possible
    • Design: for significant features, follow design process

This was referenced Dec 12, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
contribution/core This is a PR that came from AWS.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

CI/CD for CDK apps Correct docs for app-delivery & improve usability
8 participants