FailedAttachVolume - Unable to attach or mount volumes

[smangelkramer]

Describe the bug

Hi all,

just did a fresh installation of Trident in an current Rancher/RKE environment following the official docs of Trident. Everything looks fine, but got an error when trying to create and use my first vol inside a pod/container:

AttachVolume.Attach failed for volume "pvc-c17af9f3-3d2d-45f4-820f-36b91caca697" : CSINode demo-netapp-trident-w2 does not contain driver csi.trident.netapp.io

Environment

• Trident version: 20.04
• Trident installation flags used: no custom YAML
• Container runtime: 19.3.5
• Kubernetes version: 1.17.5
• Kubernetes orchestrator: Rancher 2.4.3
• Kubernetes enabled feature gates:
• OS: RancherOS 1.5.5
• NetApp backend types: ontap-nas (FAS3220 ONTAP 9.1)

To Reproduce

Default installation, create a Pod with PVC.
Volume gets created, but cannot be mounted:

CSINode <NODE> does not contain driver csi.trident.netapp.io

Cant find any issue, docs, ... regarding this.
Some deeper Troubleshooting was not successful.
Perhaps it will help to investigate further...

Expected behavior

Additional context

[clintonk]

Hello, @smangelkramer. What is the path to the kubelet directory on your nodes? Other Rancher users have reported that it is not /var/lib/kubelet, as is typical with most Kubernetes distributions. Please see #314 as well as this issue. Did you specify --kubelet-dir when installing Trident?

[smangelkramer]

Hello @clintonk i edited my RKE Cluster YAML before installing Trident like this:

kubelet:
      extra_binds:
        - "/var/lib/kubelet/plugins_registry:/var/lib/kubelet/plugins_registry"

But absolutely agree, we have an containerized kubelet with RKE/RancherOS.
Will check this asap and will reply. THX!

[smangelkramer]

Just re-deployed the Operator with Kubelet Dir: /opt/rke/var/lib/kubelet :

Name:         trident
Namespace:    trident
Labels:       <none>
Annotations:  <none>
API Version:  trident.netapp.io/v1
Kind:         TridentProvisioner
Metadata:
  Creation Timestamp:  2020-05-28T13:44:41Z
  Generation:          1
  Resource Version:    252275
  Self Link:           /apis/trident.netapp.io/v1/namespaces/trident/tridentprovisioners/trident
  UID:                 dc15c6b9-6b98-414e-ac7b-fdbcd5e633c0
Spec:
  Debug:        true
  Kubelet Dir:  /opt/rke/var/lib/kubelet
Status:
  Message:  Trident installed
  Status:   Installed
  Version:  v20.04
Events:
  Type    Reason      Age   From                        Message
  ----    ------      ----  ----                        -------
  Normal  Installing  31s   trident-operator.netapp.io  Installing Trident
  Normal  Installed   12s   trident-operator.netapp.io  Trident installed

But i ran into the same issue. I will try to deploy it again with tridentctl and customized YAML.

[rohit-arora-dev]

In 20.04.0 Trident Operator, there is a bug because of which Trident Operator does not honor the kubeletDir value set in the TridentProvisioner Spec and instead resorts to the default kubeletDir value (/var/lib/kubelet), the fix has been checked in 09161e3 and will be delivered as part of 20.07.0 Trident Operator release.

The current alternative is to use tridentctl for Trident installation and specify alternate path by using --kubelet-dir flag.