[AWS][GHA] On-demand runners are not cleaning up correctly

[oseasmoran73]

Description

On-demand runners are not cleaning up correctly when retry occurs. If it is successful, then it will clean up that last successful one

Background/context

Repo

If you search in instances up and running in EC2, you will find on-demand runners that have not been spun down

---

Tasks

• If on-demand runners fail, all instances spun up should be properly shut down and terminated

Acceptance Criteria

• Content-build workflow properly cleans up on-demand runners

[oseasmoran73]

Adding slack convo

Would be great if we can get this prioritize in the next sprint or two @laineymajor

[mydesignrocks]

update: all the workflows tied to content-build are migrated over to either self-hosted/asg or ubuntu-latest (github runner) with the exception of content-release. We are working with the respective teams to address the runner definition for the content-release in this sprint. That should resolve the requirement of this ticket.
In the meantime, we will terminate the lingering unused ec2 on-demand runners, manually.

[olivereri]

This can likely be closed to due to work in these issues:

Conent-build no longer uses on-demand runners:
department-of-veterans-affairs/va.gov-cms#11402

126 orphaned On-demand Content-build runners terminated:
department-of-veterans-affairs/va.gov-cms#11405

[oseasmoran73]

@CBonade They should have all been cleaned up with the retries. Massive thanks to @olivereri for handling all those orphans from content-build!

I think the last step would be to sunset archive (preferably delete) that repository in its entirety to resources being orphaned. I will defer that to you as COP lead

Tasks

• Reach out got Github Team to delete or archive repo
• Final sanity check with AWS for orphaned content-build (rule of thumb, and > 30 days)

[CBonade]

https://github.com/department-of-veterans-affairs/github-user-requests/issues/15404 issue request here with GitHub opened to archive this repository.

[CBonade]

This was archived by the GitHub team so ready for closing.