Provide a Backup image directory to Profs/TAs to use in case of RHODS image directory failure

[joachimweyl]

Motivation

we wish to avoid interruption when the default image directory has issues

Completion Criteria

create a secondary location for images that can be used as failover.

Description

• Find/create a backup location for the image
• update code so that the backup will be used if the primary image is not available
• test failover
• test if there are any issues if the backup is used one day and the default the next

Completion dates

Desired - 2024-04-17
Required - TBD

[larsks]

We sort of have this already: our clusters all run an internal image registry. We can populate this using ImageStream resources and have people pull those instead of the upstream images.

[larsks]

Related documentation: https://docs.openshift.com/container-platform/4.14/openshift_images/image-streams-manage.html

This is probably a better choice than #521 when we know the images in advance.

[IsaiahStapleton]

The class images are already stored internally in the internal image registry (using the image registry operator found in openshift-image-registry namespace). The reason the students were unable to build their containers from the image when registry.redhat.io was down was because the RHOAI/RHODS operator pulls a second container that it used for Oauth authentication (registry.redhat.io/openshift4/ose-oauth-proxy@sha256:4bef31eb993feb6f1096b51b4876c65a6fb1f4401fee97fa4f4542b6b7c9bc46). I have stored that image in the internal image registry and am looking into patching the operator so that it pulls from this internal registry rather than the external one.

[IsaiahStapleton]

After talking with Lars, it seems patching the operator isn't as easy as I thought it would be. We would have to make a fork of the operator, make the changes, and then deploy that. An alternative solution I am looking into is an Admission Controller to intercept the request in order to modify the resource to request the Oauth container from the internal registry rather than the external one. I am also going to follow up with the RHODS team to see if they can include a feature to change the location where this oauth image is being pulled in a new release.

[larsks]

I also posted this in slack, but probably worth posting here as well:

The policy enforcement tool I was thinking about while on the call is the Open Policy Agent Gatekeeper, currently configured through https://github.com/OCP-on-NERC/gatekeeper. Docs are at https://open-policy-agent.github.io/gatekeeper/website/docs/. That may let you do what you want without needing to write a bunch of code. We haven't paid much attention to this since deploying it, so we'll definitely want to verify that it is still operating as expected.

[IsaiahStapleton]

I added the oauth-proxy image to the internal image registry, this image stream is stored within redhat-ods-applications namespace (image-registry.openshift-image-registry.svc:5000/redhat-ods-applications/oauth-proxy). I then used gatekeeper to create a policy for mutating the request for the created pods in rhods-notebooks to pull the oauth-container from the internal registry rather than the external one. The pull request for that is here: OCP-on-NERC/gatekeeper#6

[IsaiahStapleton]

This issue was solved by OCP-on-NERC/nerc-ocp-config#543 and OCP-on-NERC/gatekeeper#6