pipelines: support batch enabled CodeBuild projects in CodePipeline

[HansFalkenberg-Visma]

Describe the feature

CodeBuild projects can run a single BuildSpec in parallel. This greatly reduces the pipeline wall time for large suites of integration tests or other long running parallelizable tasks. It is achieved by enabling batch building for the project.

The L2 construct aws-codepipeline.Pipeline supports this as easy as setting executeBatchBuild on the properties when creating a CodeBuildAction.

The L3 construct pipelines.CodePipeline maps into a L2 Pipeline, so it seems it would be a simple affair to have a new property for its CodeBuildStep that is conveyed to the L2 construct.

Use Case

I want the self-updating feature of the L3 construct, but also want to run tests in parallel without having to rewrite to the L2 construct.

Proposed Solution

Add a new property CodeBuildStepProps.buildType (same name as in the Console), that has enum values Single (default) and Batch.

Or simply CodeBuildStepProps.executeBatchBuild with a boolean value same as CodeBuildAction since the CodeBuildStep is CodeBuild specific, after all. And this avoids confusing it with CodeBuildStepProps.type, which I couldn't find in the Console.

Other Information

No response

Acknowledgements

• I may be able to implement this feature request
• This feature might incur a breaking change

CDK version used

2.132.1 (build 9df7dd3)

Environment details (OS name and version, etc.)

Windows 10 Version 22H2

[HansFalkenberg-Visma]

While trying to enable batch builds, I encountered a number of other requirements and issues:

1. CDK serializes "version": 0.2 in the BuildSpec as "version": "0.2".
   • Regardless of whether it is supplied by me in CodeBuildStepProps.partialBuildSpec or supplied by CDK itself.
   • This was reported in 2022, but ignored because there were other issues in the example provided for that ticket.
     • aws-cdk: Batch build configuration #22087
   • The specification clearly has a number there, not a string.
   • A CodeBuild project without batching is forgiving and allows this breach of schema, but a CodeBuild project with batch enabled fails in DOWNLOAD_SOURCE with cannot unmarshal !!str `0.2` into float64.
   • Batch builds in L2 work because the BuildSpec can be supplied as a file path, but L3 only supports inline build specs. Presumably L2 also doesn't work if you try to supply an inline BuildSpec there, but I haven't tried.
   • Unless this issue is fixed, a somewhat complicated workaround will still be required to make batch builds work in L3.
2. Remembering to call enableBatchBuild on the CodeBuild project.
   • This can only be done after the pipeline has been built, so tightly related code is forced far apart.
   • Wouldn't be necessary if this issue is implemented.
3. Adding permission to the action's role to do codebuild:BatchGetBuildBatches and codebuild:StartBuildBatch.
   • Should be part of the solution for this issue.
4. Adding the BatchEnabled property on the pipeline action.
   • I couldn't figure out how to use escape hatches to do this.
   • Lest you count modifying the CloudFormation template JSON directly as an escape hatch, which is what I ended up doing.
   • Wouldn't be necessary if this issue is implemented.
5. Setting a sensible timeout for the batch build timeout.
   • CodeBuildStepProps.timeout only applies to the individual builds.
   • The default for the batch build part is 12 hours.
     • I couldn't find documentation for this default, but it doesn't come from CDK or CloudFormation.
     • Maybe the API does it, or maybe there is no timeout until the project is viewed in the Console?
     • The Console only allows 8 hours, so you can't edit the project there unless you also change the timeout.
   • The API has no restriction at all, which seems like a suboptimal specification.
   • Maybe the batch timeout could be set to CodeBuildStepProps.timeout while fixing this ticket?
     • Or specifically set to 8 hours since that seems the indented maximum.
     • Although technically a breaking change, it seems very minor and if it's done in L3 it doesn't affect many since I doubt many have gone to my efforts to get L3 batch builds to work.
6. CloudFormation doesn't recognize a change in a pipeline action configuration.
   • So when adding BatchEnabled you have to make another change to the pipeline too, or CloudFormation refuses to deploy it.
   • Somebody should probably make an issue about this, if nobody has already.

The last issue is pretty simple to work around when you encounter it, but could otherwise yield undetectable drift.

The first five can all be worked around using the three functions in the attached file, batch-build-workaround.zip. Batch timeout will be set to 1 hour.

import { App } from 'aws-cdk-lib';
import { CodeBuildStep, CodePipeline, ... } from 'aws-cdk-lib/pipelines';
import {
  doEscapeHatchFixesToEnableBatchBuilds,
  doTemplateManipulationsToEnableBatchBuilds,
  scheduleEnableBatchBuilds,
} from './batch-build-workaround';

const app = new App();

...

const pipeline1Name = 'pipeline-1';
const pipeline2Name = 'pipeline-2';


const pipeline1 = new CodePipeline(..., { ..., pipelineName: pipeline1Name; });

const stepA = new CodeBuildStep(..., { ..., partialBuildSpec: ... });
scheduleEnableBatchBuilds(pipeline1, pipeline1Name, stepA);

const stepB = new CodeBuildStep(..., { ..., partialBuildSpec: ... });
scheduleEnableBatchBuilds(pipeline1, pipeline1Name, stepB);

pipeline1.addWave(..., { post: [stepA, stepB] });
pipeline1.buildPipeline();
doEscapeHatchFixesToEnableBatchBuilds(pipeline1Name);


const pipeline2 = new CodePipeline(..., { ..., pipelineName: pipeline2Name; });

const stepC = new CodeBuildStep(..., { ..., partialBuildSpec: ... });
scheduleEnableBatchBuilds(pipeline2, pipeline2Name, stepC);

pipeline2.addWave(..., { pre: [stepC] });
pipeline2.buildPipeline();
doEscapeHatchFixesToEnableBatchBuilds(pipeline2Name);


doTemplateManipulationsToEnableBatchBuilds(app.synth());

[HansFalkenberg-Visma]

If you came here and also don't know what a valid build specification looks like, here is one example:

new CodeBuildStep(..., {
  ...,
  partialBuildSpec: BuildSpec.fromObject({
    version: 0.2,
    batch: {
      'build-list': [
        { identifier: 'Machine_1' },
        { identifier: 'Machine_2' },
        { identifier: 'Machine_3' },
        { identifier: 'Machine_4' },
      ],
    },
  }),
});

There is much else you could do, so that's a pretty minimalistic example. See Batch build buildspec reference

[tim-finnigan]

Thanks for creating this feature request and highlighting the issues above.