"FATA Install failed; trident pod spec reports a different image" in air gap installation

[heikocane]

I'm upgrading trident to version 21.07 with tridentctl on our "air gap" cluster.
The steps are:

./tridentctl install -n trident --image-registry artifactory.company.com/docker-k8s-official-remote/sig-storage --trident-image artifactory.company.com/docker-se-virtual/netapp/trident:21.07.0 --autosupport-image artifactory.company.com/docker-dockerhub-release-remote/netapp/trident-autosupport:21.01.0 --generate-custom-yaml --silence-autosupport

./tridentctl install -n trident --use-custom-yaml

and then the output:

INFO Created Kubernetes clients.                   namespace=default version=v1.20.7
INFO Starting Trident installation.                namespace=trident
INFO Created service account.                      path=/root/trident-installer/setup/trident-serviceaccount.yaml
INFO Created cluster role.                         path=/root/trident-installer/setup/trident-clusterrole.yaml
INFO Created cluster role binding.                 path=/root/trident-installer/setup/trident-clusterrolebinding.yaml
INFO Trident tridentbackendconfigs.trident.netapp.io CRD present.
INFO Trident tridenttransactions.trident.netapp.io CRD present.
INFO Trident tridentsnapshots.trident.netapp.io CRD present.
INFO Trident tridentbackends.trident.netapp.io CRD present.
INFO Installer will create a fresh tridentmirrorrelationships.trident.netapp.io CRD.
INFO Created custom resource definitions tridentmirrorrelationships.trident.netapp.io.  namespace=trident
INFO Installer will create a fresh tridentsnapshotinfos.trident.netapp.io CRD.
INFO Created custom resource definitions tridentsnapshotinfos.trident.netapp.io.  namespace=trident
INFO Trident tridentstorageclasses.trident.netapp.io CRD present.
INFO Trident tridentvolumes.trident.netapp.io CRD present.
INFO Trident tridentnodes.trident.netapp.io CRD present.
INFO Trident tridentversions.trident.netapp.io CRD present.
INFO Created custom resource definitions.
INFO Created Trident pod security policy.          path=/root/trident-installer/setup/trident-podsecuritypolicy.yaml
INFO Added finalizers to custom resource definitions.
WARN Unable to get VolumeSnapshot CRD: customresourcedefinitions.apiextensions.k8s.io "volumesnapshots.snapshot.storage.k8s.io" not found  CRD=volumesnapshots.snapshot.storage.k8s.io
INFO Created Trident service.                      path=/root/trident-installer/setup/trident-service.yaml
INFO Created Trident secret.                       path=/root/trident-installer/setup/trident-service.yaml
INFO Created Trident deployment.                   path=/root/trident-installer/setup/trident-deployment.yaml
INFO Created Trident daemonset.                    path=/root/trident-installer/setup/trident-daemonset.yaml
INFO Waiting for Trident pod to start.
ERRO Trident pod was not running after 180.00 seconds. Pod status is Running. Use 'kubectl describe pod trident-csi-66d448dd89-249rr -n trident' for more information.
FATA Install failed; trident pod spec reports a different image (artifactory.company.com/docker-se-virtual/netapp/trident:21.07.0) than required (netapp/trident:21.07.0).  Resolve the issue; use 'tridentctl uninstall' to clean up; and try again.

All trident pods are running (and working)

 kubectl get pods -n trident
NAME                           READY   STATUS    RESTARTS   AGE
trident-csi-66d448dd89-249rr   6/6     Running   0          14m
trident-csi-6t6l2              2/2     Running   0          14m
trident-csi-7cxvv              2/2     Running   0          14m
trident-csi-8nsfp              2/2     Running   0          14m
trident-csi-9b9gk              2/2     Running   0          14m
trident-csi-mt6r6              2/2     Running   0          14m
trident-csi-v4qgr              2/2     Running   0          14m
trident-csi-vgdxv              2/2     Running   0          14m
trident-csi-x6f4p              2/2     Running   0          14m

and also

tridentctl logs -n trident

don't throw any errors.

Environment
Provide accurate information about the environment to help us reproduce the issue.

• Trident version: 21.07
• Trident installation flags used: tridentctl install -n trident --use-custom-yaml
• Container runtime: ocker version 19.03.15, build 99e3ed8919
• Kubernetes version: v1.20.7
• Kubernetes orchestrator: Kubespray
• OS: Ubuntu 18.04.5
• NetApp backend types: ontap
• Other:

[balaramesh]

@heikocane I observed the same issue. What I found was:

When the install YAMLs are generated with tridentctl (tridentctl install --generate-custom-yaml -n trident), and trident-deployment.yaml is updated to have the trident-main container use a different container image (other than netapp/trident:21.07.0, tridentctl returns this error/warning. This was my log trace:

DEBU[2021-08-04T09:35:46-04:00] Trident pod not yet running, waiting.         increment=2.340488664s status="trident pod spec reports a different image (localhost:5000/netapp/trident:21.07.0) than required (netapp/trident:21.07.0)"
ERRO[2021-08-04T09:38:33-04:00] Trident pod was not running after 180.00 seconds. Pod status is Running. Use 'kubectl describe pod trident-csi-bf755c4d-zdw2g -n trident' for more information.
FATA[2021-08-04T09:38:33-04:00] Install failed; trident pod spec reports a different image (localhost:5000/netapp/trident:21.07.0) than required (netapp/trident:21.07.0).  Resolve the issue; use 'tridentctl uninstall' to clean up; and try again.

As you can see, Trident observes the controller pod to be running, but still thinks there's an error somewhere. I believe you are good to go ahead and use Trident at this stage. This is an error with the tridentctl installer.

[gnarl]

This issue is fixed with commit b60b06e and is available in the Trident 21.07.1 release.