pipelines: "No such key" error on Lambda deployment with selfMutation: false

[bmmin]

Describe the bug

When I set selfMutation: false on the pipeline, I cannot deploy Lambdas any more.

My vague understanding is that when generating the templates, the lamnda assets are hashed, and the hash is the s3 key under which they land before being deployed. The problem is, that the asset upload happens in the Assets step, where the hashes are hardcoded in the buildspec of the assets. This means, that if selfMutation is off, the pipeline does not update these codeBuild projects, so the assets are not uploaded, and the lambda deployments fail.

Expected Behavior

I can still use the pipeline to deploy the stacks which were originally configured to be deployable until I dont need a new pipeline shape (e.g. cicd change - any build/deploy step needs more inputs, or a new stack).

Current Behavior

On any code change the lambda deployment fails.

Reproduction Steps

Take any pipeline that deploys a single lambda. Turn off self mutation, and generate the pipeline. Now any lambda source change will result in a failed deployment.

Possible Solution

When using selfMutate, either pass the asset hash to the assets task, or maybe better: just use a single asset step, that uploads all the assets to the correct buckets for example based on a assetsInfo.json file generated during the pipeline synth in the synth step. This would allow you to support the non self mutating use case across additional assets as well.

If none is possible, at least remove the flag, since it just breaks functionality, as such is confusing.

Additional Information/Context

Why turning self mutation off?

We are using cdk-pipelines to deploy our application across multiple regions. Some of these are not ready yet, and as such, they are failing. Since CodePipelines cannot handle such failures, having these regions anywhere but the end of the pipeline would prevent any deployments. Still these regions not only turn the pipeline red all the time, but require a full deployment (across multiple regions, lots of bake time) for every iteration. In order to speed up new region iteration, I want to put them in a separate pipeline. I could have obviously opted for separate self mutating pipelines for every region in build (~6 atm), but it has several downsides:

• manual management of all pipeline stacks
• cannot easily share resources across pipelines, like cache or assets buckets, codeBuild projects, etc. All shared resources now need an additional stack.

Instead I wanted to create in the main pipeline stack a lot of additional pipelines without self mutation. These pipelines could even point to temporary branches, allowing faster iteration on the regions, but once the region is ready, moving the region definition to a different array would in one commit add the new region in the main pipeline, and clean up the additional temporary pipeline. In this use case I dont have the issue of the temporary pipelines not being able to add additional stacks, or different pipeline configuration, since these regions have the same shape as the live regions. As such, iteration is related just to configuretion in code, and retrying when dependencies go live. Updating these in build regions from the main pipeline just simplifies the management. Also the fact that the pipelines are not rerun on mutation is not an issue, since none of these regions are live, and retriggering a run is fast (since they only have 1 wave).

CDK CLI Version

1.167.0

Framework Version

No response

Node.js Version

16

OS

macOS

Language

Typescript

Language Version

TypeScript (4.7.4)

Other information

No response

[peterwoodworth]

Hey @bmmin, thanks for the thorough report here.

We're aware of this functionality, and are tracking the issue here #9080. It's something we plan to fix at some point - I recommend giving the issue a thumbs up and adding a comment in this thread to show support

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.