dell · khareRajshree · Aug 22, 2023 · Aug 21, 2023
@@ -41,7 +41,7 @@ The CSI Drivers by Dell implement an interface between [CSI](https://kubernetes-
 | Create VolumeSnapshot    | yes      | yes       | yes       | yes        | yes        |
 | Create Volume from Snapshot | yes   | yes       | yes       | yes        | yes        |
 | Delete Snapshot          | yes      | yes       | yes       | yes        | yes        |
-| [Access Mode](https://kubernetes.io/docs/concepts/storage/persistent-volumes/#access-modes)| **FC/iSCSI:** <br>RWO/<br>RWOP<br> **Raw block:** <br>RWO/<br>RWX/<br>ROX/<br>RWOP | RWO/ROX/RWOP<br><br>RWX (Raw block only) | RWO/ROX/RWOP<br><br>RWX (Raw block & NFS only) | RWO/RWX/ROX/<br>RWOP | RWO/RWOP<br>(FC/iSCSI)<br>RWO/<br>RWX/<br>ROX/<br>RWOP<br>(RawBlock, NFS) |
+| [Access Mode](https://kubernetes.io/docs/concepts/storage/persistent-volumes/#access-modes)| **FC/iSCSI:** <br>RWO/<br>RWOP<br> **Raw block:** <br>RWO/<br>RWX/<br>ROX/<br>RWOP | RWO/ROX/RWOP<br><br>RWX (Raw block & NFS only) | RWO/ROX/RWOP<br><br>RWX (Raw block & NFS only) | RWO/RWX/ROX/<br>RWOP | RWO/RWOP<br>(FC/iSCSI)<br>RWO/<br>RWX/<br>ROX/<br>RWOP<br>(RawBlock, NFS) |
 | CSI Volume Cloning       | yes      | yes       | yes       | yes        | yes        |
 | CSI Raw Block Volume     | yes      | yes       | yes       | no         | yes        |
 | CSI Ephemeral Volume     | no       | yes       | yes       | yes        | yes        |
@@ -66,7 +66,7 @@ The CSI Drivers by Dell implement an interface between [CSI](https://kubernetes-
 | NVMeFC        | N/A              | N/A                | N/A              | N/A              | yes              |
 | NFS           | N/A              | N/A                | yes              | yes              | yes              |
 | Other         | N/A              | ScaleIO protocol   | N/A              | N/A              | N/A              |
-| Supported FS  | ext4 / xfs       | ext4 / xfs         | ext4 / xfs / NFS | NFS       | ext3 / ext4 / xfs / NFS |
+| Supported FS  | ext4 / xfs       | ext4 / xfs / NFS   | ext4 / xfs / NFS | NFS       | ext3 / ext4 / xfs / NFS |
 | Thin / Thick provisioning | Thin  | Thin              | Thin/Thick       | N/A              | Thin             |
 | Platform-specific configurable settings | Service Level selection<br>iSCSI CHAP | - | Host IO Limit<br>Tiering Policy<br>NFS Host IO size<br>Snapshot Retention duration | Access Zone<br>NFS version (3 or 4);Configurable Export IPs | iSCSI CHAP |
 | Auto RDM(vSphere)  | Yes(over FC)  | N/A              | N/A      | N/A              | N/A             |

@@ -761,3 +761,165 @@ The user can set the volume limit for a node by creating a node label `max-vxfle
 The user can also set the volume limit for all the nodes in the cluster by specifying the same to `maxVxflexosVolumesPerNode` attribute in values.yaml file.
 
 >**NOTE:** <br>To reflect the changes after setting the value either via node label or in values.yaml file, user has to bounce the driver controller and node pods using the command `kubectl get pods -n vxflexos --no-headers=true | awk '/vxflexos-/{print $1}'| xargs kubectl delete -n vxflexos pod`. <br><br> If the value is set both by node label and values.yaml file then node label value will get the precedence and user has to remove the node label in order to reflect the values.yaml value. <br><br>The default value of `maxVxflexosVolumesPerNode` is 0. <br><br>If `maxVxflexosVolumesPerNode` is set to zero, then Container Orchestration decides how many volumes of this type can be published by the controller to the node.<br><br>The volume limit specified to `maxVxflexosVolumesPerNode` attribute is applicable to all the nodes in the cluster for which node label `max-vxflexos-volumes-per-node` is not set.
+
+## NFS volume support
+Starting with version 2.8, the CSI driver for PowerFlex will support NFS volumes for PowerFlex storage systems version 4.0 and greater.
+
+CSI driver will support following operations for NFS volumes:
+
+* Creation and deletion of a NFS volume with RWO/RWX/ROX access modes.
+* Support of tree quotas while volume creation.
+* Expand the size of a NFS volume.
+* Creation and deletion of snapshot of a NFS volume while retaining file permissions.
+* Create NFS volume from the snapshot.
+
+To enable the support of NFS volumes operations from CSI driver, there are a few new keys introduced which needs to be set before performing the operations for NFS volumes.
+* `nasName`: defines the NAS server name that should be used for NFS volumes.
+* `nfsAcls`: defines permissions - POSIX mode bits or NFSv4 ACLs, to be set on NFS target mount directory. NFSv4 ACLs are supported for NFSv4 shares on NFSv4 enabled NAS servers only. POSIX ACLs are not supported and only POSIX mode bits are supported for NFSv3 shares.
+* `externalAccess`: allows you to specify additional entries for host to access NFS volumes.
+* `enableQuota`: when enabled will set quota limit for a newly provisioned NFS volume.
+
+> NOTE:
+> * `nasName`
+>   * nasName is a mandatory parameter and has to be provided in secret yaml, else it will be an error state and will be captured in driver logs.
+>   * nasName can be given at storage class level as well.
+>   * If specified in both, secret and storage class, then precedence is given to storage class value.
+>   * If nasName not given in secret, irrespective of it specified in SC, then it's an error state and will be captured in driver logs.
+> * `nfsAcls`
+>   * nfsAcls is an optional parameter, with a default value 0777.
+>   * nfsAcls can be specified in secret or storage class or values.
+>   * If nfsAcls is specified in more than one places, then the order of precedence will be:
+>     1. storage class
+>     2. secret
+>     3. values
+>     4. specified nowhere, then set to default value: 0777
+
+The user has to update the `secret.yaml`, `values.yaml` and `storageclass-nfs.yaml` with the above keys as like below:
+
+[`samples/secret.yaml`](https://github.com/dell/csi-powerflex/blob/main/samples/secret.yaml)
+```yaml
+- username: "admin"
+  password: "Password123"
+  systemID: "2b11bb111111bb1b"
+  endpoint: "https://127.0.0.2"
+  skipCertificateValidation: true
+  isDefault: true
+  mdm: "10.0.0.3,10.0.0.4"
+  nasName: "nas-server"
+  nfsAcls: "0777"
+```
+
+[`samples/storageclass/storageclass-nfs.yaml`](https://github.com/dell/csi-powerflex/blob/main/samples/storageclass/storageclass-nfs.yaml)
+```yaml
+apiVersion: storage.k8s.io/v1
+kind: StorageClass
+metadata:
+  name: vxflexos-nfs
+provisioner: csi-vxflexos.dellemc.com
+reclaimPolicy: Delete
+allowVolumeExpansion: true
+parameters:
+  storagepool: "pool2" # Insert Storage pool
+  systemID: <SYSTEM_ID> # Insert System ID
+  csi.storage.k8s.io/fstype: nfs
+  nasName: "nas-server"
+  allowRoot: "false"
+  nfsAcls: "0777"
+#  path: /csi
+#  softLimit: "80"
+#  gracePeriod: "86400"
+volumeBindingMode: WaitForFirstConsumer
+allowedTopologies:
+- matchLabelExpressions:
+  - key: csi-vxflexos.dellemc.com/<SYSTEM_ID>-nfs # Insert System ID
+    values:
+    - "true"
+```
+
+[`helm/csi-vxflexos/values.yaml`](https://github.com/dell/csi-powerflex/blob/main/helm/csi-vxflexos/values.yaml)
+```yaml
+...
+externalAccess:
+nfsAcls: "0777"
+enableQuota: false
+...
+```
+
+## Usage of Quotas to Limit Storage Consumption for NFS volumes
+Starting with version 2.8, the CSI driver for PowerFlex will support enabling tree quotas for limiting capacity for NFS volumes. To use the quota feature user can specify the boolean value `enableQuota` in values.yaml.
+
+To enable quota for NFS volumes, make the following edits to [values.yaml](https://github.com/dell/csi-powerflex/blob/main/helm/csi-vxflexos/values.yaml) file:
+```yaml
+...
+...
+# enableQuota: a boolean that, when enabled, will set quota limit for a newly provisioned NFS volume.
+# Allowed values:
+#   true: set quota for volume
+#   false: do not set quota for volume
+# Optional: true
+# Default value: none
+enableQuota: true
+...
+...
+```
+
+For example, if the user creates a PVC with 3 Gi of storage and quotas have already been enabled in PowerFlex system for the specified volume.
+
+When `enableQuota` is set to `true`
+
+* The driver sets the hard limit of the PVC to 3Gi.
+* The user adds data of 2Gi to the PVC (by logging into POD). It works as expected.
+* The user tries to add 2Gi more data.
+* Driver doesn't allow the user to enter more data as total data to be added is 4Gi and PVC limit is 3Gi.
+* The user can expand the volume from 3Gi to 6Gi. The driver allows it and sets the hard limit of PVC to 6Gi.
+* User retries adding 2Gi more data (which has been errored out previously).
+* The driver accepts the data.
+
+When `enableQuota` is set to `false`
+
+* Driver doesn't set any hard limit against the PVC created.
+* The user adds 2Gi data to the PVC, which has a limit of 3Gi. It works as expected.
+* The user tries to add 2Gi more data. Now the total size of data is 4Gi.
+* Driver allows the user to enter more data irrespective of the initial PVC size (since no quota is set against this PVC)
+* The user can expand the volume from an initial size of 3Gi to 4Gi or more. The driver allows it.
+
+If enableQuota feature is set, user can also set other tree quota parameters such as soft limit, soft grace period and path using storage class yaml file.
+
+* `path`: relative path to the root of the associated NFS volume.
+* `softLimit`: soft limit set to quota. Specified as a percentage w.r.t. PVC size.
+* `gracePeriod`: grace period of quota, must be mentioned along with softLimit, in seconds. Soft Limit can be exceeded until the grace period.
+
+> NOTE:
+> * `hardLimit` is set to same size as that of PVC size.
+> *  When a volume with quota enabled is expanded then the hardLimit and softLimit are also recalculated by driver w.r.t. to the new PVC size.
+> * `sofLimit` cannot be set to unlimited value (0), otherwise it will become greater than hardLimit (PVC size).
+> * `softLimit` should be lesser than 100%, since hardLimit will be set to 100% (PVC size) internally by the driver.
+
+### Storage Class Example with Quota Limit Parameters:
+[`samples/storageclass/storageclass-nfs.yaml`](https://github.com/dell/csi-powerflex/blob/main/samples/storageclass/storageclass-nfs.yaml)
+
+```yaml
+apiVersion: storage.k8s.io/v1
+kind: StorageClass
+metadata:
+  name: vxflexos-nfs
+provisioner: csi-vxflexos.dellemc.com
+reclaimPolicy: Delete
+allowVolumeExpansion: true
+parameters:
+  storagepool: "pool2" # Insert Storage pool
+  systemID: <SYSTEM_ID> # Insert System ID
+  csi.storage.k8s.io/fstype: nfs
+  nasName: "nas-server"
+  allowRoot: "false"
+  nfsAcls: "0777"
+  path: /csi
+  softLimit: "80"
+  gracePeriod: "86400"
+volumeBindingMode: WaitForFirstConsumer
+allowedTopologies:
+  - matchLabelExpressions:
+    - key: csi-vxflexos.dellemc.com/<SYSTEM_ID>-nfs # Insert System ID
+      values:
+        - "true"
+```
@@ -70,6 +70,12 @@ For detailed PowerFlex installation procedure, see the [Dell PowerFlex Deploymen
     - For Red Hat Enterprise Linux and CentOS, run `rpm -iv ./EMC-ScaleIO-sdc-*.x86_64.rpm`, where * is the SDC name corresponding to the PowerFlex installation version.
 4. To add more MDM_IP for multi-array support, run `/opt/emc/scaleio/sdc/bin/drv_cfg --add_mdm --ip 10.xx.xx.xx.xx,10.xx.xx.xx`
 
+#### Installation Wizard prerequisite, secret update:
+When the driver is installed using values generated by installation wizard, then the user needs to update the secret for driver by patching the MDM keys, as follows:
+
+**Steps**
+* `echo -n '<MDM_IPS>' | base64`
+* `kubectl patch secret vxflexos-config -n vxflexos -p "{\"data\": { \"MDM\": \"<GENERATED_BASE64>\"}}"`
 
 ### (Optional) Volume Snapshot Requirements
   For detailed snapshot setup procedure, [click here.](../../../../snapshots/#optional-volume-snapshot-requirements)
@@ -96,6 +102,8 @@ Note that the namespace can be any user-defined name that follows the convention
     | skipCertificateValidation  | Determines if the driver is going to validate certs while connecting to PowerFlex REST API interface. | true     | true    |
     | isDefault | An array having isDefault=true is for backward compatibility. This parameter should occur once in the list. | false    | false   |
     | mdm       | mdm defines the MDM(s) that SDC should register with on start. This should be a list of MDM IP addresses or hostnames separated by comma. | true     | -       |
+    | nasName       | nasName defines what NAS should be used for NFS volumes. NFS volumes are supported on arrays version >= 4.0. | false     | -       |
+    | nfsAcls       | nfsAcls enables setting permissions on NFS mount directory. This value will be used if a storage class does not have the NFS ACL (nfsAcls) parameter specified. | false     | 0777       |
 
     Example: `samples/secret.yaml`
 
@@ -151,7 +159,7 @@ Use the below command to replace or update the secret:
 
 | Parameter                | Description                                                                                                                                                                                                                                                                                                                                                                                                    | Required | Default |
 | ------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------- | ------- |
-| version | Set to verify the values file version matches driver version and used to pull the image as part of the image name. | Yes | 2.7.0 |
+| version | Set to verify the values file version matches driver version and used to pull the image as part of the image name. | Yes | 2.8.0 |
 | driverRepository | Set to give the repository containing the driver image (used as part of the image name). | Yes | dellemc |
 | powerflexSdc | Set to give the location of the SDC image used if automatic SDC deployment is being utilized. | Yes | dellemc/sdc:3.6.1 |
 | certSecretCount | Represents the number of certificate secrets, which the user is going to create for SSL authentication. | No | 0 |
@@ -164,7 +172,9 @@ Use the below command to replace or update the secret:
 | enablesnapshotcgdelete | A boolean that, when enabled, will delete all snapshots in a consistency group everytime a snap in the group is deleted. | Yes | false |
 | enablelistvolumesnapshot | A boolean that, when enabled, will allow list volume operation to include snapshots (since creating a volume from a snap actually results in a new snap). It is recommend this be false unless instructed otherwise. | Yes | false |
 | allowRWOMultiPodAccess | Setting allowRWOMultiPodAccess to "true" will allow multiple pods on the same node to access the same RWO volume. This behavior conflicts with the CSI specification version 1.3. NodePublishVolume description that requires an error to be returned in this case. However, some other CSI drivers support this behavior and some customers desire this behavior. Customers use this option at their own risk. | Yes | false |
-| maxVxflexosVolumesPerNode | Maximum number of volumes that controller can publish to the node. | No | 0 |
+| nfsAcls | Setting nfsAcls enables setting permissions on NFS mount directory. This value acts as default value for NFS ACL (nfsAcls), if not specified for an array config in secret. NFS volumes are supported on arrays version >= 4.0. Allowed values: for Unix mode: valid octal mode number like: "0777", "777", "0755"; for NFSv4 acls: valid NFSv4 acls, seperated by comma, like: "A::OWNER@:RWX,A::GROUP@:RWX", "A::OWNER@:rxtncy" | No | " " |
+| externalAccess | Setting externalAccess allows to specify additional entries for host to access NFS volumes. Both single IP address and subnet are valid entries. Allowed Values: x.x.x.x/xx or x.x.x.x. | No | " " |
+| enableQuota | A boolean that, when enabled, will set quota limit for a newly provisioned NFS volume. | No | false |
 | **controller**           | This section allows the configuration of controller-specific parameters. To maximize the number of available nodes for controller pods, see this section. For more details on the new controller pod configurations, see the [Features section](../../../features/powerflex#controller-ha) for Powerflex specifics.              | -        | -       |
 | volumeNamePrefix | Set so that volumes created by the driver have a default prefix. If one PowerFlex/VxFlex OS system is servicing several different Kubernetes installations or users, these prefixes help you distinguish them. | Yes | "k8s" |
 | controllerCount | Set to deploy multiple controller instances. If the controller count is greater than the number of available nodes, excess pods remain in a pending state. It should be greater than 0. You can increase the number of available nodes by configuring the "controller" section in your values.yaml. For more details on the new controller pod configurations, see the [Features section](../../../features/powerflex#controller-ha) for Powerflex specifics. | Yes | 2 |