Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Bug]: account-bootstrapping step function failing on CreateOrUpdateBaseStack with "Error": "Runtime.OutOfMemory" #758

Open
1 of 2 tasks
ethanBaird opened this issue Sep 10, 2024 · 0 comments
Labels
bug Something isn't working

Comments

@ethanBaird
Copy link

Is there an existing issue for this?

  • I have searched the existing issues

Describe the bug

Management account codepipeline failing on UploadAndUpdateBaseStacks step during stepfunction adf-account-bootstrapping when creating a new account.

Expected Behavior

  • Account creation workflow to succeed end to end
  • New account to be bootstrapped by during adf-build/main.py OR adf-account-bootstrapping not both?

Current Behavior

In the account creating process, During the CreateOrUpdateBaseStack workflow we are seeing the following error

"error": {
    "Error": "Runtime.OutOfMemory",
    "Cause": "{\"errorType\":\"Runtime.OutOfMemory\",\"errorMessage\":\"RequestId: 74646775-3735-49fd-a93c-0830f3dc7db4 Error: Runtime exited with error: signal: killed\"}"
  }

This is causing the step function to crash and report a failure back to the Codepipeline in the Management account with

2024-09-10 12:33:24,267 | INFO | __main__ | Waiting for 30 seconds for the executions of arn:aws:states:us-east-1:914779079504:stateMachine:adf-account-bootstrapping to finish. | (main.py:426)
2024-09-10 12:33:54,617 | INFO | __main__ | Found 1 state machine execution that is running. | (main.py:452)
2024-09-10 12:33:54,617 | ERROR | __main__ | Account Bootstrapping State Machine encountered a failed, timed out, or aborted execution. Please look into this problem before retrying the bootstrap pipeline. You can navigate to: https://us-east-1.console.aws.amazon.com/states/home?region=us-east-1#/statemachines/view/arn:aws:states:us-east-1:914779079504:stateMachine:adf-account-bootstrapping | (main.py:400)

As such we need to rerun the pipeline.

On second execution this stepfunction does not run, and the bootstrap pipeline succeeds as normal.

I'm honestly not sure what the value of this step function is, considering that the bootstrap process runs during the adf-build/main.py anyway which is being executed by the adf-bootstrap-pipeline-build anyway?

It appears this step function is triggered by the MoveAccount call being made in the adf-account-management step function responsible for creating the account and moving it to the correct OU.

Steps To Reproduce

Create account with ADF 4.0.0

We are adding a new definition to a yml file under adf-accounts/

Possible Solution

Should this step function exist? Wouldn't the account be bootstrapped by the Codebuild which is waiting for this stepfunction to finish anyway?

What value does it add?

Additional Information/Context

No response

ADF Version

4.0.0

Contributing a fix?

  • Yes, I am working on a fix to resolve this issue
@ethanBaird ethanBaird added the bug Something isn't working label Sep 10, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

1 participant