Create Custom Cypress Container Image for Content-Build

[olivereri]

Description

Implement a solution that allows Cypress test containers to execute without requiring root privileges.

Two Content-Build Github Actions (GHA) workflows run Cypress tests using container images. To execute without failure the containers are run with root privileges. When launched with those elevated privileges GHA runner servers file system privileges is changed to root. Other GHA workflows that run on the same runner servers fail due to the previous file system permission changes. To harmonize these workflows create a custom Cypress container image.

Acceptance Criteria

• Custom Cypress container image exists that does not require root permissions to use.
• Verify the new container image runs the continuous integration and accessibility Cypress jobs
• Content-Build workflows that use a Cypress container image for tests don't run as root
• Content-Build workflows runs the continuous integration and accessibility Cypress jobs without failures.

Relations:

department-of-veterans-affairs/va.gov-team#50148

Implementation Details

Proposed solution described here:
https://github.com/cypress-io/cypress-docker-images/tree/master/examples/included-as-non-root-mapped

Proposed tasks:

• Create AWS IAM user (service account) with credentials to provide permissions to image build GHA workflow.
  • Example PR https://github.com/department-of-veterans-affairs/devops/pull/12512
• Create AWS Elastic Container Registry to store custom container images for Content-Build.
  • Example PR https://github.com/department-of-veterans-affairs/devops/pull/12511
• Create GHA workflow in Content-Build repository to build custom Cypress container images.
  • Example workflow https://github.com/department-of-veterans-affairs/vsp-infra-k8s-sorry-cypress/actions/runs/3814358570/workflow
• Create follow-on tickets to modify workflows to use new container images.

Team

Please check the team(s) that will do this work.

• CMS Team
• Public Websites
• Facilities
• User support

[timcosgrove]

@olivereri just a note that when running the Cypress docker container build command, the user/uid that needs to be added to the Docker container is runner and the uid is 1001

[EWashb]

Let's determine success metrics for this effort working. Do we have data on how much time we have lost for this issue? Can we gather the baseline metrics to track success over time?

[olivereri]

Let's determine success metrics for this effort working. Do we have data on how much time we have lost for this issue? Can we gather the baseline metrics to track success over time?

I think from the last failure (mid Jan.) we pretty much lost a sprint to it. Other than that @timcosgrove did a good job discovering the failure rate:

https://dsva.slack.com/archives/CT4GZBM8F/p1676487279441419

Lastly, Github Actions integration with Datadog will definitely help record the metrics we want. We'll unfortunately have to wait for that functionality:

https://dsva.slack.com/archives/C01G6J7UGGH/p1676577201346119

[olivereri]

Ultimately this didn't require creating custom containers to fix the base issue. The initial assumptions weren't entirely wrong. During local testing starting a container with -u 1001:1001 would fail. However, Content-Build repository GHA workflows accept -u 1001:1001 as an option and run the container without failing. This is despite the container not having a user with that UID.

All that was required to remediate this issue was to remove any options to run Cypress Containers that included the root user UID. The other part was to make sure all the ASG GHA runner instances had their permissions fixed in the /home/runner directory.

With the above two (2) items corrected Cypress tests that run in a container on Content-Build repository GHA runners no longer require root and are no longer a problem for other workflows.

[olivereri]

Raw troubleshooting slack thread
https://dsva.slack.com/archives/CT4GZBM8F/p1677089769027479

[olivereri]

The solution that closes this issue didn't require creating a custom container. It was determined that including the option to run as UID 1001 ran successfully without needing to modify the container. The ACs changed but that had no affect on the pointing for this issue.