Outdated images in opendatahub-operator breaks ISVC reconcile

[bartoszmajsak]

Summary

I am encountering errors while running a simple smoke test against KServe. Specifically, I am seeing reconcile errors from the ODH Model Controller and am unable to access the model.

Details

It appears that the manifests for odh-model-controller are referencing an outdated image (v0.12.0-latest) created 2 months ago. I suspect this might be related to the issue (see attached screenshot for reference).

Steps to Reproduce

1. Deploy the latest incubation of odh-operator sources using manifests from here.
2. This deployment uses the image referenced here.
3. Run the smoke test against KServe.
4. Observe the reconcile errors.

Observed Behavior

The following error is logged during the reconcile process:

{"controller": "inferenceservice", "controllerGroup": "serving.kserve.io", "controllerKind": "InferenceService", "InferenceService": {"name":"sklearn-v2-iris","namespace":"kserve-test"}, "namespace": "kserve-test", "name": "sklearn-v2-iris", "reconcileID": "b23200b9-879d-4075-a88c-a7f363e2cd47", "error": "2 errors occurred:\n\t* servicemeshmemberrolls.maistra.io \"default\" is forbidden: User \"system:serviceaccount:opendatahub:odh-model-controller\" cannot update resource \"servicemeshmemberrolls\" in API group \"maistra.io\" in the namespace \"istio-system\"\n\t* failed to load serviceHost from InferenceService status\n\n"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.6/pkg/internal/controller/controller.go:329
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.6/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.6/pkg/internal/controller/controller.go:235

Workaround

Switching to the v0.12.0 image (which is actually newer than v0.12.0-latest as of the time of writing this issue) resolves the reconcile errors, and the SMM is created successfully.

[israel-hdez]

AFAIK, CI was fixed, and the image reference was also fixed.
Closing as done.