Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

trident-csi controller is CrashLoopBackOff on updating #508

Closed
ysakashita opened this issue Jan 19, 2021 · 3 comments
Closed

trident-csi controller is CrashLoopBackOff on updating #508

ysakashita opened this issue Jan 19, 2021 · 3 comments
Labels
enhancement tracked

Comments

@ysakashita
Copy link

Describe the bug
When trident-operator update version, the trident-csi controller sometimes down(CrashLoopBackOff).
I believe that it is a serious bug because it cannot be automatically recovered.
Especially, this is a big problem for us, as we are managing so many Kubernetes clusters.

$ kubectl get pod 
NAME                                           READY   STATUS              RESTARTS   AGE
trident-csi-55cd7b56fd-jrhzd                   3/6     CrashLoopBackOff    15         3m18s
trident-csi-5ws2k                              0/2     ContainerCreating   0          3m18s
trident-csi-9rlk2                              0/2     ContainerCreating   0          3m18s
trident-csi-l62bj                              0/2     ContainerCreating   0          3m18s
trident-csi-ld4s2                              0/2     ContainerCreating   0          3m18s
trident-csi-tck29                              0/2     ContainerCreating   0          3m18s
trident-operator-867b86848b-bwwtq              1/1     Running             0          3m13s

$ kubectl describe pod trident-csi-55cd7b56fd-jrhzd
...
Events:
  Type     Reason       Age                    From               Message
  ----     ------       ----                   ----               -------
  Normal   Scheduled    4m20s                  default-scheduler  Successfully assigned trident/trident-csi-55cd7b56fd-jrhzd to demo-ysaka-net-worker-default-d81eff7-scxg7
  Normal   Pulled       4m20s                  kubelet            Container image "quay.io/k8scsi/csi-provisioner:v2.0.3" already present on machine
  Normal   Created      4m20s                  kubelet            Created container csi-provisioner
  Normal   Created      4m19s                  kubelet            Created container csi-resizer
  Normal   Created      4m19s                  kubelet            Created container csi-snapshotter
  Normal   Pulled       4m19s                  kubelet            Container image "quay.io/k8scsi/csi-snapshotter:v3.0.1" already present on machine
  Normal   Started      4m19s                  kubelet            Started container csi-resizer
  Normal   Started      4m19s                  kubelet            Started container csi-snapshotter
  Normal   Created      4m19s (x2 over 4m20s)  kubelet            Created container trident-main
  Normal   Started      4m19s                  kubelet            Started container csi-provisioner
  Normal   Pulled       4m19s (x2 over 4m20s)  kubelet            Container image "netapp/trident:20.10.0" already present on machine
  Normal   Created      4m19s                  kubelet            Created container csi-attacher
  Normal   Started      4m19s                  kubelet            Started container csi-attacher
  Normal   Pulled       4m19s                  kubelet            Container image "quay.io/k8scsi/csi-resizer:v1.0.1" already present on machine
  Normal   Pulled       4m19s                  kubelet            Container image "quay.io/k8scsi/csi-attacher:v3.0.1" already present on machine
  Warning  FailedMount  4m19s                  kubelet            MountVolume.SetUp failed for volume "trident-csi-token-57wbw" : secret "trident-csi-token-57wbw" not found

Environment
Provide accurate information about the environment to help us reproduce the issue.

  • Trident version: v20.10.1
  • Trident installation flags used: -n trident
  • Container runtime: Docker 19.3.13
  • Kubernetes version: 1.19.3
  • Kubernetes orchestrator: none
  • Kubernetes enabled feature gates: none
  • OS: Ubuntu 20.04.1
  • NetApp backend types: ONTAP AFF 9.5

To Reproduce

I tried to repeat over 100 times updating trident-operator from 20.10.0 to 20.10.1 using the following script.
Then the trident-csi controller was not restarted.

$ cat update-trident-operator.sh
#!/bin/sh

tag="20.10.0"

while true
do

  if [ $tag = "20.10.0" ]; then
    tag="20.10.1"
  else
    tag="20.10.0"
  fi

  echo "update image to $tag"
  kubectl set image -n trident deploy trident-operator trident-operator=netapp/trident-operator:$tag
  sleep 30
  status=`kubectl get pod -l app=controller.csi.trident.netapp.io -o json |jq -r '.items[0].status.phase'`

  while [ $status != "Running" ]
  do
    status=`kubectl get pod -l app=controller.csi.trident.netapp.io -o json |jq -r '.items[0].status.phase'`
    echo "$status"
    sleep 5
  done

done

$ ./update-trident-operator.sh

Expected behavior
I expect correct running the trident-csi controller after updating trident-operator.

Additional context

We guess that it works like the following steps.

  1. new trident-operator(v20.10.1) is updating
  2. old trident-operator(v20.10.0) return to image old one at same timing step1.
  3. old triednt-operator delete service account
  4. re-create the trident-csi controller after running the new trident-operator.
  5. the trident-csi controller doesn't get the secret of service account.

Additional, we try to change the .spec.strategy.type: Recreate in trident-csi deployment.
As a result, I think the issue doesn’t occur.

logs

  • trident-csi controller's log (trident-main)
time="2021-01-19T00:24:04Z" level=debug msg=Environment PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
time="2021-01-19T00:24:04Z" level=debug msg=Environment HOSTNAME=trident-csi-55cd7b56fd-jrhzd
time="2021-01-19T00:24:04Z" level=debug msg=Environment KUBE_NODE_NAME=demo-ysaka-net-worker-default-d81eff7-scxg7
time="2021-01-19T00:24:04Z" level=debug msg=Environment CSI_ENDPOINT="unix://plugin/csi.sock"
time="2021-01-19T00:24:04Z" level=debug msg=Environment TRIDENT_SERVER="127.0.0.1:8000"
time="2021-01-19T00:24:04Z" level=debug msg=Environment KUBERNETES_SERVICE_HOST=10.27.0.1
time="2021-01-19T00:24:04Z" level=debug msg=Environment TRIDENT_CONTROLLER_PROM_PORT_8001_TCP_PORT=8001
time="2021-01-19T00:24:04Z" level=debug msg=Environment TRIDENT_CONTROLLER_PROM_PORT_8001_TCP_ADDR=10.27.2.207
time="2021-01-19T00:24:04Z" level=debug msg=Environment TRIDENT_CONTROLLER_PROM_PORT_8001_TCP="tcp://10.27.2.207:8001"
time="2021-01-19T00:24:04Z" level=debug msg=Environment TRIDENT_CSI_SERVICE_HOST=10.27.11.80
time="2021-01-19T00:24:04Z" level=debug msg=Environment TRIDENT_CSI_SERVICE_PORT_METRICS=9220
time="2021-01-19T00:24:04Z" level=debug msg=Environment TRIDENT_CSI_PORT="tcp://10.27.11.80:34571"
time="2021-01-19T00:24:04Z" level=debug msg=Environment TRIDENT_CSI_PORT_34571_TCP_ADDR=10.27.11.80
time="2021-01-19T00:24:04Z" level=debug msg=Environment TRIDENT_CSI_SERVICE_PORT_HTTPS=34571
time="2021-01-19T00:24:04Z" level=debug msg=Environment TRIDENT_CONTROLLER_PROM_PORT="tcp://10.27.2.207:8001"
time="2021-01-19T00:24:04Z" level=debug msg=Environment KUBERNETES_PORT_443_TCP_ADDR=10.27.0.1
time="2021-01-19T00:24:04Z" level=debug msg=Environment TRIDENT_CSI_PORT_9220_TCP_ADDR=10.27.11.80
time="2021-01-19T00:24:04Z" level=debug msg=Environment KUBERNETES_PORT="tcp://10.27.0.1:443"
time="2021-01-19T00:24:04Z" level=debug msg=Environment KUBERNETES_PORT_443_TCP="tcp://10.27.0.1:443"
time="2021-01-19T00:24:04Z" level=debug msg=Environment KUBERNETES_PORT_443_TCP_PORT=443
time="2021-01-19T00:24:04Z" level=debug msg=Environment KUBERNETES_SERVICE_PORT=443
time="2021-01-19T00:24:04Z" level=debug msg=Environment KUBERNETES_PORT_443_TCP_PROTO=tcp
time="2021-01-19T00:24:04Z" level=debug msg=Environment TRIDENT_CSI_PORT_34571_TCP="tcp://10.27.11.80:34571"
time="2021-01-19T00:24:04Z" level=debug msg=Environment TRIDENT_CSI_PORT_34571_TCP_PROTO=tcp
time="2021-01-19T00:24:04Z" level=debug msg=Environment KUBERNETES_SERVICE_PORT_HTTPS=443
time="2021-01-19T00:24:04Z" level=debug msg=Environment TRIDENT_CONTROLLER_PROM_SERVICE_PORT_HTTP_METRICS=8001
time="2021-01-19T00:24:04Z" level=debug msg=Environment TRIDENT_CSI_PORT_34571_TCP_PORT=34571
time="2021-01-19T00:24:04Z" level=debug msg=Environment TRIDENT_CSI_PORT_9220_TCP="tcp://10.27.11.80:9220"
time="2021-01-19T00:24:04Z" level=debug msg=Environment TRIDENT_CONTROLLER_PROM_SERVICE_PORT=8001
time="2021-01-19T00:24:04Z" level=debug msg=Environment TRIDENT_CONTROLLER_PROM_PORT_8001_TCP_PROTO=tcp
time="2021-01-19T00:24:04Z" level=debug msg=Environment TRIDENT_CSI_SERVICE_PORT=34571
time="2021-01-19T00:24:04Z" level=debug msg=Environment TRIDENT_CSI_PORT_9220_TCP_PROTO=tcp
time="2021-01-19T00:24:04Z" level=debug msg=Environment TRIDENT_CSI_PORT_9220_TCP_PORT=9220
time="2021-01-19T00:24:04Z" level=debug msg=Environment TRIDENT_CONTROLLER_PROM_SERVICE_HOST=10.27.2.207
time="2021-01-19T00:24:04Z" level=debug msg=Environment SSL_CERT_FILE=/etc/ssl/certs/ca-certificates.crt
time="2021-01-19T00:24:04Z" level=debug msg=Environment PORT=8000
time="2021-01-19T00:24:04Z" level=debug msg=Environment BIN=trident_orchestrator
time="2021-01-19T00:24:04Z" level=debug msg=Environment CLI_BIN=tridentctl
time="2021-01-19T00:24:04Z" level=debug msg=Environment K8S=
time="2021-01-19T00:24:04Z" level=debug msg=Environment TRIDENT_IP=localhost
time="2021-01-19T00:24:04Z" level=debug msg=Environment HOME=/root
time="2021-01-19T00:24:04Z" level=info msg="Running Trident storage orchestrator." binary=/trident_orchestrator build_time="Fri Oct 30 20:52:25 UTC 2020" version=20.10.0
time="2021-01-19T00:24:04Z" level=debug msg=Flag name=address value=127.0.0.1
time="2021-01-19T00:24:04Z" level=debug msg=Flag name=crd_persistence value=true
time="2021-01-19T00:24:04Z" level=debug msg=Flag name=csi_endpoint value="unix://plugin/csi.sock"
time="2021-01-19T00:24:04Z" level=debug msg=Flag name=csi_node_name value=demo-ysaka-net-worker-default-d81eff7-scxg7
time="2021-01-19T00:24:04Z" level=debug msg=Flag name=csi_role value=controller
time="2021-01-19T00:24:04Z" level=debug msg=Flag name=debug value=true
time="2021-01-19T00:24:04Z" level=debug msg=Flag name=https_port value=8443
time="2021-01-19T00:24:04Z" level=debug msg=Flag name=https_rest value=true
time="2021-01-19T00:24:04Z" level=debug msg=Flag name=k8s_pod value=true
time="2021-01-19T00:24:04Z" level=debug msg=Flag name=log_format value=text
time="2021-01-19T00:24:04Z" level=debug msg=Flag name=metrics value=true
time="2021-01-19T00:24:04Z" level=debug msg="Trident is configured with a CRD client."
time="2021-01-19T00:24:04Z" level=debug msg="Creating CRDv1 persistent store client." requestID=a170a554-8827-4e7c-87e4-020e69a516f6 requestSource=Internal
time="2021-01-19T00:24:04Z" level=fatal msg="Unable to create the Kubernetes store client. could not initialize Kubernetes client; couldn't retrieve API server's version: the server has asked for the client to provide credentials"
  • trident-operator's log
time="2021-01-19T00:18:37Z" level=debug msg=Environment PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
time="2021-01-19T00:18:37Z" level=debug msg=Environment HOSTNAME=trident-operator-867b86848b-bwwtq
time="2021-01-19T00:18:37Z" level=debug msg=Environment OPERATOR_NAME=trident-operator
time="2021-01-19T00:18:37Z" level=debug msg=Environment POD_NAME=trident-operator-867b86848b-bwwtq
time="2021-01-19T00:18:37Z" level=debug msg=Environment TRIDENT_CONTROLLER_PROM_PORT_8001_TCP_PROTO=tcp
time="2021-01-19T00:18:37Z" level=debug msg=Environment KUBERNETES_SERVICE_HOST=10.27.0.1
time="2021-01-19T00:18:37Z" level=debug msg=Environment TRIDENT_CSI_PORT="tcp://10.27.11.80:34571"
time="2021-01-19T00:18:37Z" level=debug msg=Environment TRIDENT_CSI_PORT_34571_TCP_PORT=34571
time="2021-01-19T00:18:37Z" level=debug msg=Environment TRIDENT_CONTROLLER_PROM_SERVICE_HOST=10.27.2.207
time="2021-01-19T00:18:37Z" level=debug msg=Environment TRIDENT_CONTROLLER_PROM_PORT_8001_TCP_PORT=8001
time="2021-01-19T00:18:37Z" level=debug msg=Environment TRIDENT_CSI_PORT_34571_TCP="tcp://10.27.11.80:34571"
time="2021-01-19T00:18:37Z" level=debug msg=Environment TRIDENT_CONTROLLER_PROM_SERVICE_PORT=8001
time="2021-01-19T00:18:37Z" level=debug msg=Environment KUBERNETES_PORT="tcp://10.27.0.1:443"
time="2021-01-19T00:18:37Z" level=debug msg=Environment KUBERNETES_PORT_443_TCP_ADDR=10.27.0.1
time="2021-01-19T00:18:37Z" level=debug msg=Environment TRIDENT_CSI_PORT_9220_TCP_ADDR=10.27.11.80
time="2021-01-19T00:18:37Z" level=debug msg=Environment TRIDENT_CONTROLLER_PROM_PORT_8001_TCP="tcp://10.27.2.207:8001"
time="2021-01-19T00:18:37Z" level=debug msg=Environment KUBERNETES_SERVICE_PORT_HTTPS=443
time="2021-01-19T00:18:37Z" level=debug msg=Environment KUBERNETES_PORT_443_TCP="tcp://10.27.0.1:443"
time="2021-01-19T00:18:37Z" level=debug msg=Environment TRIDENT_CSI_PORT_9220_TCP_PROTO=tcp
time="2021-01-19T00:18:37Z" level=debug msg=Environment TRIDENT_CONTROLLER_PROM_SERVICE_PORT_HTTP_METRICS=8001
time="2021-01-19T00:18:37Z" level=debug msg=Environment KUBERNETES_PORT_443_TCP_PROTO=tcp
time="2021-01-19T00:18:37Z" level=debug msg=Environment TRIDENT_CSI_SERVICE_PORT_HTTPS=34571
time="2021-01-19T00:18:37Z" level=debug msg=Environment TRIDENT_CSI_PORT_34571_TCP_PROTO=tcp
time="2021-01-19T00:18:37Z" level=debug msg=Environment KUBERNETES_PORT_443_TCP_PORT=443
time="2021-01-19T00:18:37Z" level=debug msg=Environment TRIDENT_CSI_SERVICE_HOST=10.27.11.80
time="2021-01-19T00:18:37Z" level=debug msg=Environment TRIDENT_CSI_PORT_34571_TCP_ADDR=10.27.11.80
time="2021-01-19T00:18:37Z" level=debug msg=Environment TRIDENT_CONTROLLER_PROM_PORT="tcp://10.27.2.207:8001"
time="2021-01-19T00:18:37Z" level=debug msg=Environment KUBERNETES_SERVICE_PORT=443
time="2021-01-19T00:18:37Z" level=debug msg=Environment TRIDENT_CSI_SERVICE_PORT=34571
time="2021-01-19T00:18:37Z" level=debug msg=Environment TRIDENT_CSI_PORT_9220_TCP="tcp://10.27.11.80:9220"
time="2021-01-19T00:18:37Z" level=debug msg=Environment TRIDENT_CSI_PORT_9220_TCP_PORT=9220
time="2021-01-19T00:18:37Z" level=debug msg=Environment TRIDENT_CONTROLLER_PROM_PORT_8001_TCP_ADDR=10.27.2.207
time="2021-01-19T00:18:37Z" level=debug msg=Environment TRIDENT_CSI_SERVICE_PORT_METRICS=9220
time="2021-01-19T00:18:37Z" level=debug msg=Environment SSL_CERT_FILE=/etc/ssl/certs/ca-certificates.crt
time="2021-01-19T00:18:37Z" level=debug msg=Environment BIN=trident-operator
time="2021-01-19T00:18:37Z" level=debug msg=Environment K8S=
time="2021-01-19T00:18:37Z" level=debug msg=Environment HOME=/root
time="2021-01-19T00:18:37Z" level=info msg="Running Trident operator." binary=/trident-operator build_time="Fri Oct 30 20:52:25 UTC 2020" version=20.10.0
time="2021-01-19T00:18:37Z" level=debug msg=Flag name=debug value=true
time="2021-01-19T00:18:37Z" level=debug msg="Creating in-cluster Kubernetes clients."
time="2021-01-19T00:18:37Z" level=debug msg="Initialized Kubernetes API client." cli=kubectl flavor=k8s namespace=trident timeout=30s version=v1.19.3
time="2021-01-19T00:18:37Z" level=info msg="Created Kubernetes clients." namespace=trident version=v1.19.3
time="2021-01-19T00:18:37Z" level=info msg="Initializing Trident Provisioner controller."
time="2021-01-19T00:18:37Z" level=info msg="Activating Trident Provisioner controller."
time="2021-01-19T00:18:37Z" level=info msg="Starting workers"
time="2021-01-19T00:18:37Z" level=info msg="Started workers"
time="2021-01-19T00:18:37Z" level=info msg="TridentProvisioner CR added." name=trident namespace=trident
time="2021-01-19T00:18:37Z" level=info msg="Deployment changed." name=trident-csi namespace=trident
time="2021-01-19T00:18:37Z" level=info msg="Daemonset changed." name=trident-csi namespace=trident
time="2021-01-19T00:18:37Z" level=info msg="An operator based Trident CSI deployment named 'trident-csi' was found in the namespace 'trident'."
time="2021-01-19T00:18:37Z" level=debug msg="Found atleast one CSI Trident deployment created by the operator in namespace: trident."
time="2021-01-19T00:18:37Z" level=info msg="Reconciler found Trident installation." callingCRName=trident callingCRNamespace=trident callingResourceType=resourceTridentProvisionerCR controllingCRBasedOnStatus=trident controllingCRBasedOnStatusNamespace=trident isCSI=true namespace=trident operatorBasedCSIDeployments="[trident-csi]"
time="2021-01-19T00:18:37Z" level=debug msg="Identifying controlling CRs from the list of CRs found in Trident installation namespace."
time="2021-01-19T00:18:37Z" level=info msg="Found CR that controls current Trident deployment." name=trident namespace=trident
time="2021-01-19T00:18:37Z" level=debug msg="Controlling CR: 'trident', Controlling CR Namespace: 'trident'"
time="2021-01-19T00:18:37Z" level=debug msg="'trident' in namespace 'trident' is a controlling CR based on status."
time="2021-01-19T00:18:37Z" level=debug msg="Initialized Kubernetes API client." cli=kubectl flavor=k8s namespace=trident timeout=30s version=v1.19.3
time="2021-01-19T00:18:37Z" level=debug msg="Initialized installer." namespace=trident
time="2021-01-19T00:18:37Z" level=info msg="A Trident deployment named 'trident-csi' was found by label in CR namespace 'trident'."
W0119 00:18:37.299241       1 warnings.go:67] apiextensions.k8s.io/v1beta1 CustomResourceDefinition is deprecated in v1.16+, unavailable in v1.22+; use apiextensions.k8s.io/v1 CustomResourceDefinition
W0119 00:18:37.310068       1 warnings.go:67] apiextensions.k8s.io/v1beta1 CustomResourceDefinition is deprecated in v1.16+, unavailable in v1.22+; use apiextensions.k8s.io/v1 CustomResourceDefinition
W0119 00:18:37.322860       1 warnings.go:67] apiextensions.k8s.io/v1beta1 CustomResourceDefinition is deprecated in v1.16+, unavailable in v1.22+; use apiextensions.k8s.io/v1 CustomResourceDefinition
W0119 00:18:37.328671       1 warnings.go:67] apiextensions.k8s.io/v1beta1 CustomResourceDefinition is deprecated in v1.16+, unavailable in v1.22+; use apiextensions.k8s.io/v1 CustomResourceDefinition
W0119 00:18:37.335365       1 warnings.go:67] apiextensions.k8s.io/v1beta1 CustomResourceDefinition is deprecated in v1.16+, unavailable in v1.22+; use apiextensions.k8s.io/v1 CustomResourceDefinition
W0119 00:18:37.342158       1 warnings.go:67] apiextensions.k8s.io/v1beta1 CustomResourceDefinition is deprecated in v1.16+, unavailable in v1.22+; use apiextensions.k8s.io/v1 CustomResourceDefinition
time="2021-01-19T00:18:37Z" level=debug msg="Kubernetes version 'v1.19.3' is supported."
time="2021-01-19T00:18:37Z" level=debug msg="Initialized Kubernetes API client." cli=kubectl flavor=k8s namespace=trident timeout=30s version=v1.19.3
time="2021-01-19T00:18:37Z" level=debug msg="Initialized installer." namespace=trident
time="2021-01-19T00:18:37Z" level=debug msg="Using default Trident image 'netapp/trident:20.10.0', version 'v20.10.0'"
time="2021-01-19T00:18:37Z" level=info msg="Starting Trident installation." namespace=trident
time="2021-01-19T00:18:37Z" level=info msg="A Trident Service account named 'trident-csi' was found by label."
time="2021-01-19T00:18:37Z" level=debug msg="Patching Trident Service account." namespace=trident service=trident-csi
time="2021-01-19T00:18:37Z" level=debug msg="Patched Kubernetes Service Account." label="app=controller.csi.trident.netapp.io" namespace=trident serviceAccount=trident-csi
time="2021-01-19T00:18:37Z" level=debug msg="Patched Trident service account."
time="2021-01-19T00:18:37Z" level=info msg="A Trident cluster role named 'trident-csi' was found by label."
time="2021-01-19T00:18:37Z" level=debug msg="Patching Trident Cluster role." clusterRole=trident-csi
time="2021-01-19T00:18:37Z" level=debug msg="Patched Kubernetes cluster role." clusterRole=trident-csi label="app=controller.csi.trident.netapp.io"
time="2021-01-19T00:18:37Z" level=debug msg="Patched Trident Cluster role."
time="2021-01-19T00:18:37Z" level=info msg="A Trident Cluster role binding named 'trident-csi' was found by label."
time="2021-01-19T00:18:37Z" level=debug msg="Patching Trident Cluster role binding." clusterRoleBinding=trident-csi
time="2021-01-19T00:18:37Z" level=debug msg="Patched Kubernetes cluster role binding." clusterRoleBinding=trident-csi label="app=controller.csi.trident.netapp.io"
time="2021-01-19T00:18:37Z" level=debug msg="Patched Trident Cluster role binding."
W0119 00:18:37.442092       1 warnings.go:67] apiextensions.k8s.io/v1beta1 CustomResourceDefinition is deprecated in v1.16+, unavailable in v1.22+; use apiextensions.k8s.io/v1 CustomResourceDefinition
time="2021-01-19T00:18:37Z" level=info msg="Trident tridentversions.trident.netapp.io CRD present."
W0119 00:18:37.445998       1 warnings.go:67] apiextensions.k8s.io/v1beta1 CustomResourceDefinition is deprecated in v1.16+, unavailable in v1.22+; use apiextensions.k8s.io/v1 CustomResourceDefinition
time="2021-01-19T00:18:37Z" level=info msg="Trident tridentbackends.trident.netapp.io CRD present."
W0119 00:18:37.449150       1 warnings.go:67] apiextensions.k8s.io/v1beta1 CustomResourceDefinition is deprecated in v1.16+, unavailable in v1.22+; use apiextensions.k8s.io/v1 CustomResourceDefinition
time="2021-01-19T00:18:37Z" level=info msg="Trident tridentstorageclasses.trident.netapp.io CRD present."
W0119 00:18:37.452915       1 warnings.go:67] apiextensions.k8s.io/v1beta1 CustomResourceDefinition is deprecated in v1.16+, unavailable in v1.22+; use apiextensions.k8s.io/v1 CustomResourceDefinition
time="2021-01-19T00:18:37Z" level=info msg="Trident tridentvolumes.trident.netapp.io CRD present."
W0119 00:18:37.460178       1 warnings.go:67] apiextensions.k8s.io/v1beta1 CustomResourceDefinition is deprecated in v1.16+, unavailable in v1.22+; use apiextensions.k8s.io/v1 CustomResourceDefinition
time="2021-01-19T00:18:37Z" level=info msg="Trident tridentnodes.trident.netapp.io CRD present."
W0119 00:18:37.463461       1 warnings.go:67] apiextensions.k8s.io/v1beta1 CustomResourceDefinition is deprecated in v1.16+, unavailable in v1.22+; use apiextensions.k8s.io/v1 CustomResourceDefinition
time="2021-01-19T00:18:37Z" level=info msg="Trident tridenttransactions.trident.netapp.io CRD present."
W0119 00:18:37.467137       1 warnings.go:67] apiextensions.k8s.io/v1beta1 CustomResourceDefinition is deprecated in v1.16+, unavailable in v1.22+; use apiextensions.k8s.io/v1 CustomResourceDefinition
time="2021-01-19T00:18:37Z" level=info msg="Trident tridentsnapshots.trident.netapp.io CRD present."
time="2021-01-19T00:18:37Z" level=debug msg="Found API resource." group=apiextensions.k8s.io kind=CustomResourceDefinition resource=customresourcedefinitions version=v1beta1
time="2021-01-19T00:18:37Z" level=debug msg="Adding finalizers to CRD object tridentbackends.trident.netapp.io." kind=CustomResourceDefinition name=tridentbackends.trident.netapp.io
W0119 00:18:37.470951       1 warnings.go:67] apiextensions.k8s.io/v1beta1 CustomResourceDefinition is deprecated in v1.16+, unavailable in v1.22+; use apiextensions.k8s.io/v1 CustomResourceDefinition
time="2021-01-19T00:18:37Z" level=debug msg="Trident finalizer already present on Kubernetes CRD object tridentbackends.trident.netapp.io, nothing to do"
time="2021-01-19T00:18:37Z" level=debug msg="Adding finalizers to CRD object tridentnodes.trident.netapp.io." kind=CustomResourceDefinition name=tridentnodes.trident.netapp.io
W0119 00:18:37.474328       1 warnings.go:67] apiextensions.k8s.io/v1beta1 CustomResourceDefinition is deprecated in v1.16+, unavailable in v1.22+; use apiextensions.k8s.io/v1 CustomResourceDefinition
time="2021-01-19T00:18:37Z" level=debug msg="Trident finalizer already present on Kubernetes CRD object tridentnodes.trident.netapp.io, nothing to do"
time="2021-01-19T00:18:37Z" level=debug msg="Adding finalizers to CRD object tridentstorageclasses.trident.netapp.io." kind=CustomResourceDefinition name=tridentstorageclasses.trident.netapp.io
W0119 00:18:37.478205       1 warnings.go:67] apiextensions.k8s.io/v1beta1 CustomResourceDefinition is deprecated in v1.16+, unavailable in v1.22+; use apiextensions.k8s.io/v1 CustomResourceDefinition
time="2021-01-19T00:18:37Z" level=debug msg="Trident finalizer already present on Kubernetes CRD object tridentstorageclasses.trident.netapp.io, nothing to do"
time="2021-01-19T00:18:37Z" level=debug msg="Adding finalizers to CRD object tridenttransactions.trident.netapp.io." kind=CustomResourceDefinition name=tridenttransactions.trident.netapp.io
W0119 00:18:37.482004       1 warnings.go:67] apiextensions.k8s.io/v1beta1 CustomResourceDefinition is deprecated in v1.16+, unavailable in v1.22+; use apiextensions.k8s.io/v1 CustomResourceDefinition
time="2021-01-19T00:18:37Z" level=debug msg="Trident finalizer already present on Kubernetes CRD object tridenttransactions.trident.netapp.io, nothing to do"
time="2021-01-19T00:18:37Z" level=debug msg="Adding finalizers to CRD object tridentversions.trident.netapp.io." kind=CustomResourceDefinition name=tridentversions.trident.netapp.io
W0119 00:18:37.485462       1 warnings.go:67] apiextensions.k8s.io/v1beta1 CustomResourceDefinition is deprecated in v1.16+, unavailable in v1.22+; use apiextensions.k8s.io/v1 CustomResourceDefinition
time="2021-01-19T00:18:37Z" level=debug msg="Trident finalizer already present on Kubernetes CRD object tridentversions.trident.netapp.io, nothing to do"
time="2021-01-19T00:18:37Z" level=debug msg="Adding finalizers to CRD object tridentvolumes.trident.netapp.io." kind=CustomResourceDefinition name=tridentvolumes.trident.netapp.io
W0119 00:18:37.489572       1 warnings.go:67] apiextensions.k8s.io/v1beta1 CustomResourceDefinition is deprecated in v1.16+, unavailable in v1.22+; use apiextensions.k8s.io/v1 CustomResourceDefinition
time="2021-01-19T00:18:37Z" level=debug msg="Trident finalizer already present on Kubernetes CRD object tridentvolumes.trident.netapp.io, nothing to do"
time="2021-01-19T00:18:37Z" level=debug msg="Adding finalizers to CRD object tridentsnapshots.trident.netapp.io." kind=CustomResourceDefinition name=tridentsnapshots.trident.netapp.io
W0119 00:18:37.493496       1 warnings.go:67] apiextensions.k8s.io/v1beta1 CustomResourceDefinition is deprecated in v1.16+, unavailable in v1.22+; use apiextensions.k8s.io/v1 CustomResourceDefinition
time="2021-01-19T00:18:37Z" level=debug msg="Trident finalizer already present on Kubernetes CRD object tridentsnapshots.trident.netapp.io, nothing to do"
time="2021-01-19T00:18:37Z" level=info msg="Added finalizers to custom resource definitions."
time="2021-01-19T00:18:37Z" level=info msg="A Trident Pod security policy named 'tridentpods' was found by label."
time="2021-01-19T00:18:37Z" level=debug msg="Patching Trident Pod security policy." podSecurityPolicy=tridentpods
time="2021-01-19T00:18:37Z" level=debug msg="Patched Kubernetes pod security policy." deployment=tridentpods label="app=controller.csi.trident.netapp.io"
time="2021-01-19T00:18:37Z" level=debug msg="Patched Trident Pod security policy."
W0119 00:18:37.516127       1 warnings.go:67] storage.k8s.io/v1beta1 CSIDriver is deprecated in v1.19+, unavailable in v1.22+; use storage.k8s.io/v1 CSIDriver
time="2021-01-19T00:18:37Z" level=info msg="A Trident CSI driver CR named 'csi.trident.netapp.io' was found by label."
time="2021-01-19T00:18:37Z" level=debug msg="Patching Trident CSI driver CR." CSIDriver=csi.trident.netapp.io
W0119 00:18:37.521346       1 warnings.go:67] storage.k8s.io/v1beta1 CSIDriver is deprecated in v1.19+, unavailable in v1.22+; use storage.k8s.io/v1 CSIDriver
W0119 00:18:37.526952       1 warnings.go:67] storage.k8s.io/v1beta1 CSIDriver is deprecated in v1.19+, unavailable in v1.22+; use storage.k8s.io/v1 CSIDriver
time="2021-01-19T00:18:37Z" level=debug msg="Patched Kubernetes CSI driver." deployment=csi.trident.netapp.io label="app=controller.csi.trident.netapp.io"
time="2021-01-19T00:18:37Z" level=debug msg="Patched Trident CSI driver."
time="2021-01-19T00:18:37Z" level=info msg="A Trident service name trident-csi was found by label in CR namespace 'trident'."
time="2021-01-19T00:18:37Z" level=debug msg="Patching Trident service." namespace=trident service=trident-csi
time="2021-01-19T00:18:37Z" level=debug msg="Patched Kubernetes service." deployment=trident-csi label="app=controller.csi.trident.netapp.io" namespace=trident
time="2021-01-19T00:18:37Z" level=debug msg="Patched Trident Service."
time="2021-01-19T00:18:37Z" level=info msg="A Trident secret named 'trident-csi' was found by label in CR namespace 'trident'."
time="2021-01-19T00:18:37Z" level=info msg="A Trident deployment named 'trident-csi' was found by label in CR namespace 'trident'."
time="2021-01-19T00:18:37Z" level=debug msg="Patching Trident deployment." deployment=trident-csi namespace=trident
time="2021-01-19T00:18:37Z" level=debug msg="Patched Kubernetes deployment." deployment=trident-csi label="app=controller.csi.trident.netapp.io" namespace=trident
time="2021-01-19T00:18:37Z" level=debug msg="Patched Trident deployment."
time="2021-01-19T00:18:37Z" level=info msg="A Trident daemonset named 'trident-csi' was found by label in CR namespace 'trident'."
time="2021-01-19T00:18:37Z" level=debug msg="Patching Trident daemonset." daemontset=trident-csi namespace=trident
time="2021-01-19T00:18:37Z" level=debug msg="Patched Kubernetes DaemonSet." DaemonSet=trident-csi label="app=node.csi.trident.netapp.io" namespace=trident
time="2021-01-19T00:18:37Z" level=debug msg="Patched Trident DaemonSet."
time="2021-01-19T00:18:37Z" level=debug msg="Waiting for 7s after the patch to make sure we get the right trident-pod name"
time="2021-01-19T00:18:44Z" level=info msg="Waiting for Trident pod to start."
time="2021-01-19T00:18:44Z" level=info msg="Trident pod started." namespace=trident pod=trident-csi-55cd7b56fd-jrhzd
time="2021-01-19T00:18:44Z" level=info msg="Waiting for Trident REST interface."
time="2021-01-19T00:18:44Z" level=debug msg="Invoking tunneled command: 'tridentctl -s 127.0.0.1:8000 version -o json'"
time="2021-01-19T00:18:44Z" level=debug msg="REST interface not yet up, waiting." increment=403.222236ms
time="2021-01-19T00:18:45Z" level=debug msg="Invoking tunneled command: 'tridentctl -s 127.0.0.1:8000 version -o json'"
time="2021-01-19T00:18:45Z" level=debug msg="REST interface not yet up, waiting." increment=886.22811ms
time="2021-01-19T00:18:46Z" level=debug msg="Invoking tunneled command: 'tridentctl -s 127.0.0.1:8000 version -o json'"
time="2021-01-19T00:18:46Z" level=debug msg="REST interface not yet up, waiting." increment=1.232920549s
time="2021-01-19T00:18:47Z" level=debug msg="Invoking tunneled command: 'tridentctl -s 127.0.0.1:8000 version -o json'"
time="2021-01-19T00:18:47Z" level=debug msg="REST interface not yet up, waiting." increment=1.520389807s
time="2021-01-19T00:18:48Z" level=debug msg="Invoking tunneled command: 'tridentctl -s 127.0.0.1:8000 version -o json'"
time="2021-01-19T00:18:48Z" level=debug msg="REST interface not yet up, waiting." increment=3.382296942s
time="2021-01-19T00:18:52Z" level=debug msg="Invoking tunneled command: 'tridentctl -s 127.0.0.1:8000 version -o json'"
time="2021-01-19T00:18:52Z" level=debug msg="REST interface not yet up, waiting." increment=3.494898005s
time="2021-01-19T00:18:55Z" level=debug msg="Invoking tunneled command: 'tridentctl -s 127.0.0.1:8000 version -o json'"
time="2021-01-19T00:18:55Z" level=debug msg="REST interface not yet up, waiting." increment=3.962647571s
time="2021-01-19T00:18:59Z" level=debug msg="Invoking tunneled command: 'tridentctl -s 127.0.0.1:8000 version -o json'"
time="2021-01-19T00:18:59Z" level=debug msg="REST interface not yet up, waiting." increment=9.702007862s
time="2021-01-19T00:19:09Z" level=debug msg="Invoking tunneled command: 'tridentctl -s 127.0.0.1:8000 version -o json'"
time="2021-01-19T00:19:09Z" level=error msg="Trident REST interface was not available after 30.00 seconds; err: unable to upgrade connection: container not found (\"trident-main\"); unable to upgrade connection: container not found (\"trident-main\")"
time="2021-01-19T00:19:09Z" level=debug msg="Updating Trident Provisioner CR after failed installation." name=trident namespace=trident
time="2021-01-19T00:19:09Z" level=error msg="error syncing 'trident/trident': reconcile failed; error re-installing Trident 'trident' in namespace 'trident'; err: reconcile failed; unable to upgrade connection: container not found (\"trident-main\"); unable to upgrade connection: container not found (\"trident-main\"); use 'tridentctl logs' to learn more, requeuing"
time="2021-01-19T00:19:09Z" level=info msg=-------------------------------------------------
time="2021-01-19T00:19:09Z" level=info msg=-------------------------------------------------
time="2021-01-19T00:19:09Z" level=error msg="error syncing 'trident/trident': reconcile failed; error re-installing Trident 'trident' in namespace 'trident'; err: reconcile failed; unable to upgrade connection: container not found (\"trident-main\"); unable to upgrade connection: container not found (\"trident-main\"); use 'tridentctl logs' to learn more, requeuing"
time="2021-01-19T00:19:09Z" level=info msg="'TridentProvisioner' CR updated." name=trident namespace=trident
time="2021-01-19T00:19:09Z" level=info msg="An operator based Trident CSI deployment named 'trident-csi' was found in the namespace 'trident'."
time="2021-01-19T00:19:09Z" level=debug msg="Found atleast one CSI Trident deployment created by the operator in namespace: trident."
time="2021-01-19T00:19:09Z" level=info msg="Reconciler found Trident installation." callingCRName=trident-csi callingCRNamespace=trident callingResourceType=resourceDeployment controllingCRBasedOnStatus=trident controllingCRBasedOnStatusNamespace=trident isCSI=true namespace=trident operatorBasedCSIDeployments="[trident-csi]"
time="2021-01-19T00:19:09Z" level=debug msg="Identifying controlling CRs from the list of CRs found in Trident installation namespace."
time="2021-01-19T00:19:09Z" level=info msg="Found CR that controls current Trident deployment." name=trident namespace=trident
time="2021-01-19T00:19:09Z" level=debug msg="Controlling CR: 'trident', Controlling CR Namespace: 'trident'"
time="2021-01-19T00:19:09Z" level=debug msg="'trident' in namespace 'trident' is a controlling CR based on status."
time="2021-01-19T00:19:09Z" level=debug msg="Initialized Kubernetes API client." cli=kubectl flavor=k8s namespace=trident timeout=30s version=v1.19.3
time="2021-01-19T00:19:09Z" level=debug msg="Initialized installer." namespace=trident
time="2021-01-19T00:19:09Z" level=info msg="A Trident deployment named 'trident-csi' was found by label in CR namespace 'trident'."
W0119 00:19:09.665284       1 warnings.go:67] apiextensions.k8s.io/v1beta1 CustomResourceDefinition is deprecated in v1.16+, unavailable in v1.22+; use apiextensions.k8s.io/v1 CustomResourceDefinition
W0119 00:19:09.680193       1 warnings.go:67] apiextensions.k8s.io/v1beta1 CustomResourceDefinition is deprecated in v1.16+, unavailable in v1.22+; use apiextensions.k8s.io/v1 CustomResourceDefinition
W0119 00:19:09.690603       1 warnings.go:67] apiextensions.k8s.io/v1beta1 CustomResourceDefinition is deprecated in v1.16+, unavailable in v1.22+; use apiextensions.k8s.io/v1 CustomResourceDefinition
W0119 00:19:09.695573       1 warnings.go:67] apiextensions.k8s.io/v1beta1 CustomResourceDefinition is deprecated in v1.16+, unavailable in v1.22+; use apiextensions.k8s.io/v1 CustomResourceDefinition
W0119 00:19:09.702654       1 warnings.go:67] apiextensions.k8s.io/v1beta1 CustomResourceDefinition is deprecated in v1.16+, unavailable in v1.22+; use apiextensions.k8s.io/v1 CustomResourceDefinition
W0119 00:19:09.710436       1 warnings.go:67] apiextensions.k8s.io/v1beta1 CustomResourceDefinition is deprecated in v1.16+, unavailable in v1.22+; use apiextensions.k8s.io/v1 CustomResourceDefinition
time="2021-01-19T00:19:09Z" level=debug msg="Kubernetes version 'v1.19.3' is supported."
time="2021-01-19T00:19:09Z" level=debug msg="Initialized Kubernetes API client." cli=kubectl flavor=k8s namespace=trident timeout=30s version=v1.19.3
time="2021-01-19T00:19:09Z" level=debug msg="Initialized installer." namespace=trident
time="2021-01-19T00:19:09Z" level=debug msg="Using default Trident image 'netapp/trident:20.10.0', version 'v20.10.0'"
time="2021-01-19T00:19:09Z" level=info msg="Starting Trident installation." namespace=trident
time="2021-01-19T00:19:09Z" level=info msg="A Trident Service account named 'trident-csi' was found by label."
time="2021-01-19T00:19:09Z" level=debug msg="Patching Trident Service account." namespace=trident service=trident-csi
time="2021-01-19T00:19:09Z" level=debug msg="Patched Kubernetes Service Account." label="app=controller.csi.trident.netapp.io" namespace=trident serviceAccount=trident-csi
time="2021-01-19T00:19:09Z" level=debug msg="Patched Trident service account."
time="2021-01-19T00:19:09Z" level=info msg="A Trident cluster role named 'trident-csi' was found by label."
time="2021-01-19T00:19:09Z" level=debug msg="Patching Trident Cluster role." clusterRole=trident-csi
time="2021-01-19T00:19:09Z" level=debug msg="Patched Kubernetes cluster role." clusterRole=trident-csi label="app=controller.csi.trident.netapp.io"
time="2021-01-19T00:19:09Z" level=debug msg="Patched Trident Cluster role."
time="2021-01-19T00:19:09Z" level=info msg="A Trident Cluster role binding named 'trident-csi' was found by label."
time="2021-01-19T00:19:09Z" level=debug msg="Patching Trident Cluster role binding." clusterRoleBinding=trident-csi
time="2021-01-19T00:19:09Z" level=debug msg="Patched Kubernetes cluster role binding." clusterRoleBinding=trident-csi label="app=controller.csi.trident.netapp.io"
time="2021-01-19T00:19:09Z" level=debug msg="Patched Trident Cluster role binding."
W0119 00:19:09.789268       1 warnings.go:67] apiextensions.k8s.io/v1beta1 CustomResourceDefinition is deprecated in v1.16+, unavailable in v1.22+; use apiextensions.k8s.io/v1 CustomResourceDefinition
time="2021-01-19T00:19:09Z" level=info msg="Trident tridentversions.trident.netapp.io CRD present."
W0119 00:19:09.792314       1 warnings.go:67] apiextensions.k8s.io/v1beta1 CustomResourceDefinition is deprecated in v1.16+, unavailable in v1.22+; use apiextensions.k8s.io/v1 CustomResourceDefinition
time="2021-01-19T00:19:09Z" level=info msg="Trident tridentbackends.trident.netapp.io CRD present."
W0119 00:19:09.795867       1 warnings.go:67] apiextensions.k8s.io/v1beta1 CustomResourceDefinition is deprecated in v1.16+, unavailable in v1.22+; use apiextensions.k8s.io/v1 CustomResourceDefinition
time="2021-01-19T00:19:09Z" level=info msg="Trident tridentstorageclasses.trident.netapp.io CRD present."
W0119 00:19:09.799027       1 warnings.go:67] apiextensions.k8s.io/v1beta1 CustomResourceDefinition is deprecated in v1.16+, unavailable in v1.22+; use apiextensions.k8s.io/v1 CustomResourceDefinition
time="2021-01-19T00:19:09Z" level=info msg="Trident tridentvolumes.trident.netapp.io CRD present."
W0119 00:19:09.802506       1 warnings.go:67] apiextensions.k8s.io/v1beta1 CustomResourceDefinition is deprecated in v1.16+, unavailable in v1.22+; use apiextensions.k8s.io/v1 CustomResourceDefinition
time="2021-01-19T00:19:09Z" level=info msg="Trident tridentnodes.trident.netapp.io CRD present."
W0119 00:19:09.806209       1 warnings.go:67] apiextensions.k8s.io/v1beta1 CustomResourceDefinition is deprecated in v1.16+, unavailable in v1.22+; use apiextensions.k8s.io/v1 CustomResourceDefinition
time="2021-01-19T00:19:09Z" level=info msg="Trident tridenttransactions.trident.netapp.io CRD present."
W0119 00:19:09.810118       1 warnings.go:67] apiextensions.k8s.io/v1beta1 CustomResourceDefinition is deprecated in v1.16+, unavailable in v1.22+; use apiextensions.k8s.io/v1 CustomResourceDefinition
time="2021-01-19T00:19:09Z" level=info msg="Trident tridentsnapshots.trident.netapp.io CRD present."
time="2021-01-19T00:19:09Z" level=debug msg="Found API resource." group=apiextensions.k8s.io kind=CustomResourceDefinition resource=customresourcedefinitions version=v1beta1
time="2021-01-19T00:19:09Z" level=debug msg="Adding finalizers to CRD object tridentbackends.trident.netapp.io." kind=CustomResourceDefinition name=tridentbackends.trident.netapp.io
W0119 00:19:09.814701       1 warnings.go:67] apiextensions.k8s.io/v1beta1 CustomResourceDefinition is deprecated in v1.16+, unavailable in v1.22+; use apiextensions.k8s.io/v1 CustomResourceDefinition
time="2021-01-19T00:19:09Z" level=debug msg="Trident finalizer already present on Kubernetes CRD object tridentbackends.trident.netapp.io, nothing to do"
time="2021-01-19T00:19:09Z" level=debug msg="Adding finalizers to CRD object tridentnodes.trident.netapp.io." kind=CustomResourceDefinition name=tridentnodes.trident.netapp.io
W0119 00:19:09.818893       1 warnings.go:67] apiextensions.k8s.io/v1beta1 CustomResourceDefinition is deprecated in v1.16+, unavailable in v1.22+; use apiextensions.k8s.io/v1 CustomResourceDefinition
time="2021-01-19T00:19:09Z" level=debug msg="Trident finalizer already present on Kubernetes CRD object tridentnodes.trident.netapp.io, nothing to do"
time="2021-01-19T00:19:09Z" level=debug msg="Adding finalizers to CRD object tridentstorageclasses.trident.netapp.io." kind=CustomResourceDefinition name=tridentstorageclasses.trident.netapp.io
W0119 00:19:09.824114       1 warnings.go:67] apiextensions.k8s.io/v1beta1 CustomResourceDefinition is deprecated in v1.16+, unavailable in v1.22+; use apiextensions.k8s.io/v1 CustomResourceDefinition
time="2021-01-19T00:19:09Z" level=debug msg="Trident finalizer already present on Kubernetes CRD object tridentstorageclasses.trident.netapp.io, nothing to do"
time="2021-01-19T00:19:09Z" level=debug msg="Adding finalizers to CRD object tridenttransactions.trident.netapp.io." kind=CustomResourceDefinition name=tridenttransactions.trident.netapp.io
W0119 00:19:09.828492       1 warnings.go:67] apiextensions.k8s.io/v1beta1 CustomResourceDefinition is deprecated in v1.16+, unavailable in v1.22+; use apiextensions.k8s.io/v1 CustomResourceDefinition
time="2021-01-19T00:19:09Z" level=debug msg="Trident finalizer already present on Kubernetes CRD object tridenttransactions.trident.netapp.io, nothing to do"
time="2021-01-19T00:19:09Z" level=debug msg="Adding finalizers to CRD object tridentversions.trident.netapp.io." kind=CustomResourceDefinition name=tridentversions.trident.netapp.io
W0119 00:19:09.832592       1 warnings.go:67] apiextensions.k8s.io/v1beta1 CustomResourceDefinition is deprecated in v1.16+, unavailable in v1.22+; use apiextensions.k8s.io/v1 CustomResourceDefinition
time="2021-01-19T00:19:09Z" level=debug msg="Trident finalizer already present on Kubernetes CRD object tridentversions.trident.netapp.io, nothing to do"
time="2021-01-19T00:19:09Z" level=debug msg="Adding finalizers to CRD object tridentvolumes.trident.netapp.io." kind=CustomResourceDefinition name=tridentvolumes.trident.netapp.io
W0119 00:19:09.836285       1 warnings.go:67] apiextensions.k8s.io/v1beta1 CustomResourceDefinition is deprecated in v1.16+, unavailable in v1.22+; use apiextensions.k8s.io/v1 CustomResourceDefinition
time="2021-01-19T00:19:09Z" level=debug msg="Trident finalizer already present on Kubernetes CRD object tridentvolumes.trident.netapp.io, nothing to do"
time="2021-01-19T00:19:09Z" level=debug msg="Adding finalizers to CRD object tridentsnapshots.trident.netapp.io." kind=CustomResourceDefinition name=tridentsnapshots.trident.netapp.io
W0119 00:19:09.839835       1 warnings.go:67] apiextensions.k8s.io/v1beta1 CustomResourceDefinition is deprecated in v1.16+, unavailable in v1.22+; use apiextensions.k8s.io/v1 CustomResourceDefinition
time="2021-01-19T00:19:09Z" level=debug msg="Trident finalizer already present on Kubernetes CRD object tridentsnapshots.trident.netapp.io, nothing to do"
time="2021-01-19T00:19:09Z" level=info msg="Added finalizers to custom resource definitions."
time="2021-01-19T00:19:09Z" level=info msg="A Trident Pod security policy named 'tridentpods' was found by label."
time="2021-01-19T00:19:09Z" level=debug msg="Patching Trident Pod security policy." podSecurityPolicy=tridentpods
time="2021-01-19T00:19:09Z" level=debug msg="Patched Kubernetes pod security policy." deployment=tridentpods label="app=controller.csi.trident.netapp.io"
time="2021-01-19T00:19:09Z" level=debug msg="Patched Trident Pod security policy."
W0119 00:19:09.859599       1 warnings.go:67] storage.k8s.io/v1beta1 CSIDriver is deprecated in v1.19+, unavailable in v1.22+; use storage.k8s.io/v1 CSIDriver
time="2021-01-19T00:19:09Z" level=info msg="A Trident CSI driver CR named 'csi.trident.netapp.io' was found by label."
time="2021-01-19T00:19:09Z" level=debug msg="Patching Trident CSI driver CR." CSIDriver=csi.trident.netapp.io
W0119 00:19:09.864053       1 warnings.go:67] storage.k8s.io/v1beta1 CSIDriver is deprecated in v1.19+, unavailable in v1.22+; use storage.k8s.io/v1 CSIDriver
W0119 00:19:09.870209       1 warnings.go:67] storage.k8s.io/v1beta1 CSIDriver is deprecated in v1.19+, unavailable in v1.22+; use storage.k8s.io/v1 CSIDriver
time="2021-01-19T00:19:09Z" level=debug msg="Patched Kubernetes CSI driver." deployment=csi.trident.netapp.io label="app=controller.csi.trident.netapp.io"
time="2021-01-19T00:19:09Z" level=debug msg="Patched Trident CSI driver."
time="2021-01-19T00:19:09Z" level=info msg="A Trident service name trident-csi was found by label in CR namespace 'trident'."
time="2021-01-19T00:19:09Z" level=debug msg="Patching Trident service." namespace=trident service=trident-csi
time="2021-01-19T00:19:09Z" level=debug msg="Patched Kubernetes service." deployment=trident-csi label="app=controller.csi.trident.netapp.io" namespace=trident
time="2021-01-19T00:19:09Z" level=debug msg="Patched Trident Service."
time="2021-01-19T00:19:09Z" level=info msg="A Trident secret named 'trident-csi' was found by label in CR namespace 'trident'."
time="2021-01-19T00:19:09Z" level=info msg="A Trident deployment named 'trident-csi' was found by label in CR namespace 'trident'."
time="2021-01-19T00:19:09Z" level=debug msg="Patching Trident deployment." deployment=trident-csi namespace=trident
time="2021-01-19T00:19:09Z" level=debug msg="Patched Kubernetes deployment." deployment=trident-csi label="app=controller.csi.trident.netapp.io" namespace=trident
time="2021-01-19T00:19:09Z" level=debug msg="Patched Trident deployment."
time="2021-01-19T00:19:09Z" level=info msg="A Trident daemonset named 'trident-csi' was found by label in CR namespace 'trident'."
time="2021-01-19T00:19:09Z" level=debug msg="Patching Trident daemonset." daemontset=trident-csi namespace=trident
time="2021-01-19T00:19:09Z" level=debug msg="Patched Kubernetes DaemonSet." DaemonSet=trident-csi label="app=node.csi.trident.netapp.io" namespace=trident
time="2021-01-19T00:19:09Z" level=debug msg="Patched Trident DaemonSet."
time="2021-01-19T00:19:09Z" level=debug msg="Waiting for 7s after the patch to make sure we get the right trident-pod name"
time="2021-01-19T00:19:16Z" level=info msg="Waiting for Trident pod to start."
time="2021-01-19T00:19:16Z" level=info msg="Trident pod started." namespace=trident pod=trident-csi-55cd7b56fd-jrhzd
time="2021-01-19T00:19:16Z" level=info msg="Waiting for Trident REST interface."
time="2021-01-19T00:19:16Z" level=debug msg="Invoking tunneled command: 'tridentctl -s 127.0.0.1:8000 version -o json'"
time="2021-01-19T00:19:17Z" level=debug msg="REST interface not yet up, waiting." increment=338.666141ms
time="2021-01-19T00:19:17Z" level=debug msg="Invoking tunneled command: 'tridentctl -s 127.0.0.1:8000 version -o json'"
time="2021-01-19T00:19:17Z" level=debug msg="REST interface not yet up, waiting." increment=786.476203ms
time="2021-01-19T00:19:18Z" level=debug msg="Invoking tunneled command: 'tridentctl -s 127.0.0.1:8000 version -o json'"
time="2021-01-19T00:19:18Z" level=debug msg="REST interface not yet up, waiting." increment=1.601808555s
time="2021-01-19T00:19:19Z" level=debug msg="Invoking tunneled command: 'tridentctl -s 127.0.0.1:8000 version -o json'"
time="2021-01-19T00:19:19Z" level=debug msg="REST interface not yet up, waiting." increment=2.147952255s
time="2021-01-19T00:19:22Z" level=debug msg="Invoking tunneled command: 'tridentctl -s 127.0.0.1:8000 version -o json'"
time="2021-01-19T00:19:22Z" level=debug msg="REST interface not yet up, waiting." increment=1.628564377s
time="2021-01-19T00:19:23Z" level=debug msg="Invoking tunneled command: 'tridentctl -s 127.0.0.1:8000 version -o json'"
time="2021-01-19T00:19:23Z" level=debug msg="REST interface not yet up, waiting." increment=4.312980659s
time="2021-01-19T00:19:28Z" level=debug msg="Invoking tunneled command: 'tridentctl -s 127.0.0.1:8000 version -o json'"
time="2021-01-19T00:19:28Z" level=debug msg="REST interface not yet up, waiting." increment=6.029302542s
time="2021-01-19T00:19:34Z" level=debug msg="Invoking tunneled command: 'tridentctl -s 127.0.0.1:8000 version -o json'"
time="2021-01-19T00:19:34Z" level=debug msg="REST interface not yet up, waiting." increment=5.053333139s
time="2021-01-19T00:19:39Z" level=debug msg="Invoking tunneled command: 'tridentctl -s 127.0.0.1:8000 version -o json'"
time="2021-01-19T00:19:39Z" level=debug msg="REST interface not yet up, waiting." increment=7.002509428s
time="2021-01-19T00:19:46Z" level=debug msg="Invoking tunneled command: 'tridentctl -s 127.0.0.1:8000 version -o json'"
time="2021-01-19T00:19:46Z" level=error msg="Trident REST interface was not available after 30.00 seconds; err: unable to upgrade connection: container not found (\"trident-main\"); unable to upgrade connection: container not found (\"trident-main\")"
time="2021-01-19T00:19:46Z" level=debug msg="Updating Trident Provisioner CR after failed installation." name=trident namespace=trident
time="2021-01-19T00:19:46Z" level=info msg="New status is same as the old status, no update needed." name=trident namespace=trident
time="2021-01-19T00:19:46Z" level=error msg="error syncing 'trident/trident-csi': reconcile failed; error re-installing Trident 'trident' in namespace 'trident'; err: reconcile failed; unable to upgrade connection: container not found (\"trident-main\"); unable to upgrade connection: container not found (\"trident-main\"); use 'tridentctl logs' to learn more, requeuing"
time="2021-01-19T00:19:46Z" level=info msg=-------------------------------------------------
time="2021-01-19T00:19:46Z" level=info msg=-------------------------------------------------
time="2021-01-19T00:19:46Z" level=error msg="error syncing 'trident/trident-csi': reconcile failed; error re-installing Trident 'trident' in namespace 'trident'; err: reconcile failed; unable to upgrade connection: container not found (\"trident-main\"); unable to upgrade connection: container not found (\"trident-main\"); use 'tridentctl logs' to learn more, requeuing"
time="2021-01-19T00:19:46Z" level=info msg="An operator based Trident CSI deployment named 'trident-csi' was found in the namespace 'trident'."
time="2021-01-19T00:19:46Z" level=debug msg="Found atleast one CSI Trident deployment created by the operator in namespace: trident."
time="2021-01-19T00:19:46Z" level=info msg="Reconciler found Trident installation." callingCRName=trident-csi callingCRNamespace=trident callingResourceType=resourceDaemonset controllingCRBasedOnStatus=trident controllingCRBasedOnStatusNamespace=trident isCSI=true namespace=trident operatorBasedCSIDeployments="[trident-csi]"
time="2021-01-19T00:19:46Z" level=debug msg="Identifying controlling CRs from the list of CRs found in Trident installation namespace."
time="2021-01-19T00:19:46Z" level=info msg="Found CR that controls current Trident deployment." name=trident namespace=trident
time="2021-01-19T00:19:46Z" level=debug msg="Controlling CR: 'trident', Controlling CR Namespace: 'trident'"
time="2021-01-19T00:19:46Z" level=debug msg="'trident' in namespace 'trident' is a controlling CR based on status."
time="2021-01-19T00:19:46Z" level=debug msg="Initialized Kubernetes API client." cli=kubectl flavor=k8s namespace=trident timeout=30s version=v1.19.3
time="2021-01-19T00:19:46Z" level=debug msg="Initialized installer." namespace=trident
time="2021-01-19T00:19:46Z" level=info msg="A Trident deployment named 'trident-csi' was found by label in CR namespace 'trident'."
W0119 00:19:46.361614       1 warnings.go:67] apiextensions.k8s.io/v1beta1 CustomResourceDefinition is deprecated in v1.16+, unavailable in v1.22+; use apiextensions.k8s.io/v1 CustomResourceDefinition
W0119 00:19:46.368116       1 warnings.go:67] apiextensions.k8s.io/v1beta1 CustomResourceDefinition is deprecated in v1.16+, unavailable in v1.22+; use apiextensions.k8s.io/v1 CustomResourceDefinition
W0119 00:19:46.374156       1 warnings.go:67] apiextensions.k8s.io/v1beta1 CustomResourceDefinition is deprecated in v1.16+, unavailable in v1.22+; use apiextensions.k8s.io/v1 CustomResourceDefinition
W0119 00:19:46.379077       1 warnings.go:67] apiextensions.k8s.io/v1beta1 CustomResourceDefinition is deprecated in v1.16+, unavailable in v1.22+; use apiextensions.k8s.io/v1 CustomResourceDefinition
W0119 00:19:46.386792       1 warnings.go:67] apiextensions.k8s.io/v1beta1 CustomResourceDefinition is deprecated in v1.16+, unavailable in v1.22+; use apiextensions.k8s.io/v1 CustomResourceDefinition
W0119 00:19:46.394805       1 warnings.go:67] apiextensions.k8s.io/v1beta1 CustomResourceDefinition is deprecated in v1.16+, unavailable in v1.22+; use apiextensions.k8s.io/v1 CustomResourceDefinition
@ysakashita ysakashita added the bug label Jan 19, 2021
@rohit-arora-dev
Copy link
Contributor

Hello @ysakashita,

It appears that when you are updating Trident Operator, you are updating the existing Trident Operator deployment with the newer Trident Operator image. The preferred workflow to update the Trident Operator (and thus Trident) is to delete the old Trident Operator bundle.yaml once you do that it does not uninstall or affects your Trident installation, however in that period of time Trident installation is not be managed by the Operator. Once the old bundle.yaml is completely gone (including the old Trident Operator deployment) you should install the new bundle.yaml.

@ysakashita
Copy link
Author

@ntap-arorar

I understood bundle.yaml way.
However, I want to enhance the way without bundle.yaml.
Because I believe that the way is not declarative management.
So it loses the advantage of operator.
For example, the trident-operator is not able to manage by GitOps (e.g. ArgoCD).
Ref: #358

Especially in an environment where many Kubernetes are running,
the way will increase the management cost.

It's just ideas for enhancement without uninstall operation(bundle.yaml)

(Idea1): Add .spec.strategy.type: Recreate in trident-operator's deployment.
(Idea2): Select leader trident-operator using leader election of client-go. (see: https://github.com/kubernetes/client-go/blob/master/examples/leader-election/main.go)

@gnarl gnarl added tracked and removed tracked labels Feb 16, 2021
@ysakashita ysakashita mentioned this issue Mar 1, 2021
Closed
@gnarl gnarl added tracked and removed tracked labels Mar 11, 2021
@gnarl
Copy link
Contributor

gnarl commented May 4, 2021

The "Recreate" strategy was added with this commit and is included in Trident 21.04.0.

@gnarl gnarl closed this as completed May 4, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
enhancement tracked
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@gnarl @ysakashita @rohit-arora-dev