bug: Accounts pipeline often times out

[mbevc1]

Describe the bug
When applying changes very often Accounts pipeline would break with timeouts changing resources.

Failed resources:
AWSAccelerator-AccountsStack-050451375726-us-east-1 | 10:42:41 AM | UPDATE_FAILED        | Custom::CreateServiceLinkedRole | SecurityHubServiceLinkedRole/CreateServiceLinkedRoleResource/Default (SecurityHubServiceLinkedRoleCreateServiceLinkedRoleResource4CC7EFAA) Received response status [FAILED] from custom resource. Message returned: TimeoutError: {"state":"TIMEOUT","reason":"Waiter has timed out"}
    at checkExceptions (/var/runtime/node_modules/@aws-sdk/node_modules/@smithy/util-waiter/dist-cjs/index.js:59:26)
    at waitUntilFunctionActiveV2 (/var/runtime/node_modules/@aws-sdk/client-lambda/dist-cjs/index.js:5826:49)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async defaultInvokeFunction (/var/task/outbound.js:1:1024)
    at async invokeUserFunction (/var/task/framework.js:1:2287)
    at async onEvent (/var/task/framework.js:1:369)
    at async Runtime.handler (/var/task/cfn-response.js:1:1676) (RequestId: ab51f95c-8b66-4b99-9420-f4b775751368)
    new CustomResource (/codebuild/output/src2823/src/s3/00/source/node_modules/aws-cdk-lib/core/lib/custom-resource.js:1:1075)
    \_ new ServiceLinkedRole (/codebuild/output/src2823/src/s3/00/source/packages/@aws-accelerator/constructs/lib/aws-iam/service-linked-role.ts:90:21)
    \_ AccountsStack.createServiceLinkedRole (/codebuild/output/src2823/src/s3/00/source/packages/@aws-accelerator/accelerator/lib/stacks/accelerator-stack.ts:918:31)
    \_ AccountsStack.createSecurityHubServiceLinkedRole (/codebuild/output/src2823/src/s3/00/source/packages/@aws-accelerator/accelerator/lib/stacks/accelerator-stack.ts:563:12)
    \_ new AccountsStack (/codebuild/output/src2823/src/s3/00/source/packages/@aws-accelerator/accelerator/lib/stacks/accounts-stack.ts:67:12)
    \_ createAccountsStack (/codebuild/output/src2823/src/s3/00/source/packages/@aws-accelerator/accelerator/utils/stack-utils.ts:502:27)
    \_ createManagementAccountStacks (/codebuild/output/src2823/src/s3/00/source/packages/@aws-accelerator/accelerator/bin/app.ts:105:22)
    \_ main (/codebuild/output/src2823/src/s3/00/source/packages/@aws-accelerator/accelerator/bin/app.ts:246:5)
    \_ processTicksAndRejections (node:internal/process/task_queues:95:5)
    \_ async /codebuild/output/src2823/src/s3/00/source/packages/@aws-accelerator/accelerator/bin/app.ts:260:5

To Reproduce
Apply changes to the manifest, I've noticed this usually with security related changes, e.g. SH, GD, Macie,...

Expected behavior
A clear and concise description of what you expected to happen.
Successful apply of the pipeline.

Please complete the following information about the solution:

• Version: v1.9.2

To get the version of the solution, you can look at the description of the created AWS CloudFormation stack used to install the LZA (AWSAccelerator-InstallerStack). For example, "(SO0199) Landing Zone Accelerator on AWS. Version 1.5.1.". If the description does not contain the version information, you can look at the Parameters of the stack for the RepositoryBranchName as that should contain the version number.

• Region: eu-west-2
• Was the solution modified from the version published on this repository? No
• If the answer to the previous question was yes, are the changes available on GitHub?
• Have you checked your service quotas for the services this solution uses?
• Were there any errors in the CloudWatch Logs?

Screenshots
If applicable, add screenshots to help explain your problem (please DO NOT include sensitive information).

Additional context
Possibly related to: #237

[mbevc1]

There is also retry we could use as well: https://aws.amazon.com/about-aws/whats-new/2024/10/aws-codebuild-retrying-builds-automatically/

[n3mawashi]

Can confirm that I get his almost ever run. While I'm grateful for the retry action/stage, it usually means having force delete the template every run.