- 1. Overview
- 2. Supported versions
- 3. Driver Installation
- 4. Performing Volume operations
- 5. Collecting Logs
- 6. Troubleshooting CSI driver installation
- 7. Odd Case Environment Variables
- 8. Note on K8S setup through Rancher
- 9. Driver upgrades and downgrades
Datera is a fully disaggregated scale-out storage platform, that runs over multiple standard protocols (iSCSI, Object/S3), combining both heterogeneous compute platform/framework flexibility (HPE, Dell, Fujitsu, Cisco and others) with rapid deployment velocity and access to data from anywhere. Datera® is a software-defined data infrastructure for virtualized environments, databases, cloudstacks, DevOps, microservices and container deployments. It provides operations-free delivery and orchestration of data at scale for any application within a traditional datacenter, private cloud or hybrid cloud setting.
Datera gives Kubernetes (K8s) enterprise customers the peace of mind of a future-proof data services platform that is ready for diverse and demanding workloads -- as K8s continues to dominate the container orchestration arena, it is likely to containerize higher-end workloads, as well.
The Datera CSI Volume Plugin uses Datera storage backend as distributed data storage for containers.
Datera CSI Plugin Version | Supported CSI Versions | Supported Kubernetes Versions |
---|---|---|
v1.0.4 | v1.0 | v1.13.X+ |
v1.0.5 | v1.0 | v1.13.X+ |
v1.0.6 | v1.0 | v1.13.X+ |
v1.0.7 | v1.0 | v1.13.X+ |
v1.0.8 | v1.0 | v1.13.X+ |
v1.0.9 | v1.0 | v1.13.X+ |
v1.0.10 | v1.0 | v1.13.X+ |
v1.0.11 | v1.0 | v1.13.X+ |
v1.0.12 | v1.0 | v1.13.X+ |
Kubernetes Installation/Configuration (Kubernetes v1.13+ required). Note that container-based ISCSID is no longer supported. The Datera implementation runs an iscsi-send inside the driver containers and an iscsi-recv on the kubernetes hosts. The iscsi-recv would further use the iscsid on the kubernetes hosts for performing iSCSI operations.
Ensure iscsid and iscsi-recv are running on the hosts. These MUST be performed before installing the CSI plugin:
First install iscsid on the kubernetes hosts
Ubuntu
$ apt install open-iscsi
Centos
$ yum install iscsi-initiator-utils
Verify iscsid is running:
$ ps -ef | grep iscsid
root 12494 996 0 09:41 pts/2 00:00:00 grep --color=auto iscsid
root 13326 1 0 Dec17 ? 00:00:01 /sbin/iscsid
root 13327 1 0 Dec17 ? 00:00:05 /sbin/iscsid
Clone the datera-csi repository
$ git clone http://github.com/Datera/datera-csi
Run the iscsi-recv service installer
$ ./assets/setup_iscsi.sh
[INFO] Dependency checking
[INFO] Downloading iscsi-recv
[INFO] Verifying checksum
[INFO] Changing file permissions
[INFO] Registering iscsi-recv service
Created symlink from /etc/systemd/system/multi-user.target.wants/iscsi-recv.service to /lib/systemd/system/iscsi-recv.service.
[INFO] Starting iscsi-recv service
[INFO] Verifying service started correctly
root 4879 1 0 19:50 ? 00:00:00 /var/datera/iscsi-recv -addr unix:////var/datera/csi-iscsi.sock
Check that the iscsi-recv service is running
$ systemctl --all | grep iscsi-recv
iscsi-recv.service loaded active running iscsi-recv container to host iscsiadm adapter service
Modify deploy/kubernetes/releases/1.0/csi-datera-v1.0.x.yaml and update the values for the following environment variables in the yaml:
DAT_MGMT
-- The management IP of the Datera systemDAT_USER
-- The username of your Datera accountDAT_PASS
-- The password for your Datera accountDAT_TENANT
-- The tenant to use with your Datera accountDAT_API
-- The API version to use when communicating (should be 2.2, currently the only version the plugin supports)
There are 2 locations for each value within the yaml that should be modified. One is under the StatefulSet and the other is under the DaemonSet. Note that the yaml does not come with a built-in StorageClass. Create one using the following example shown and modifying it depending on deployment needs.
Instead of putting the username/password in the yaml file directly, you can use the kubernetes secrets capabilities.
NOTE: This must be done before installing the CSI driver.
First create the secrets. They're base64 encoded strings. The two required secrets are "username" and "password". Modify and save the below yaml as secrets.yaml.
apiVersion: v1
kind: Secret
metadata:
name: datera-secret
namespace: kube-system
type: Opaque
data:
# base64 encoded username
# generate this via "$ echo -n 'your-username' | base64"
username: YWRtaW4=
# base64 encoded password
# generate this via "$ echo -n 'your-password' | base64"
password: cGFzc3dvcmQ=
Then create the secrets.
$ kubectl create -f secrets.yaml
Now install the CSI driver like above, but using the "secrets" yaml:
$ kubectl create -f csi-datera-secrets-v1.0.10.yaml
The only difference between the "secrets" yaml and the regular yaml is the use of secrets for the "username" and "password" fields.
The driver install manifest file is available under deploy/kubernetes/release/1.0 directory. Pick up the latest version. Run the following command to install Datera CSI driver.
$ kubectl create -f csi-datera-v1.0.x.yaml
For example:
# kubectl create -f csi-datera-secrets-1.0.10.yaml
storageclass.storage.k8s.io/dat-block-storage created
serviceaccount/csi-datera-controller-sa created
clusterrole.rbac.authorization.k8s.io/csi-datera-provisioner-role created
clusterrolebinding.rbac.authorization.k8s.io/csi-datera-provisioner-binding created
clusterrole.rbac.authorization.k8s.io/csi-datera-attacher-role created
clusterrolebinding.rbac.authorization.k8s.io/csi-datera-attacher-binding created
clusterrole.rbac.authorization.k8s.io/csi-datera-snapshotter-role created
clusterrolebinding.rbac.authorization.k8s.io/csi-datera-snapshotter-binding created
statefulset.apps/csi-provisioner created
serviceaccount/csi-datera-node-sa created
clusterrole.rbac.authorization.k8s.io/csi-datera-node-driver-registrar-role created
clusterrolebinding.rbac.authorization.k8s.io/csi-datera-node-driver-registrar-binding created
daemonset.apps/csi-node created
#
You should see one csi-provisioner pod and 'N' csi-node pods running in kube-system namespace, where N = number of kubernetes worker nodes. The csi-provisioner pod can run on any kubernetes node.
# kubectl get pods -n kube-system -o wide | egrep 'NAME|csi-'
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
csi-node-cwpfk 3/3 Running 0 164m 1.1.1.1 node1 <none> <none>
csi-node-lx66c 3/3 Running 0 164m 2.2.2.2 node2 <none> <none>
csi-provisioner-0 6/6 Running 0 163m 1.1.1.1 node1 <none> <none>
[root@tlx51cp tmp]#
Before creating a Volume, a StorageClass needs to be created. This StorageClass acts like a template where you can specify your Volume and QoS parameters. The parameters can be placed within the parameters
section of the StorageClass.
In the following example we configure volumes with a replica of 3 and a QoS of 1000 IOPS max. All parameters must be strings (pure numbers and booleans should be enclosed in quotes). Save the following as 'csi-storageclass.yaml' file.
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
name: dat-block-storage
namespace: kube-system
annotations:
storageclass.kubernetes.io/is-default-class: "true"
provisioner: dsp.csi.daterainc.io
parameters:
replica_count: "3"
total_iops_max: "1000"
$ kubectl create -f csi-storageclass.yaml
Here are a list of supported parameters for the plugin:
Name | Default |
---|---|
replica_count |
3 |
placement_mode |
hybrid (Use this for Datera OS versions < 3.3) |
placement_policy |
default (Use this for Datera OS versions >= 3.3) |
ip_pool |
default |
template |
"" |
round_robin |
false |
read_iops_max |
0 |
write_iops_max |
0 |
total_iops_max |
0 |
read_bandwidth_max |
0 |
write_bandwidth_max |
0 |
total_bandwidth_max |
0 |
iops_per_gb |
0 |
bandwidth_per_gb |
0 |
fs_type |
ext4 (Currently the only supported values are 'ext4' and 'xfs') |
fs_args |
-E lazy_itable_init=0,lazy_journal_init=0,nodiscard -F |
delete_on_unmount |
false |
NOTE:
-
All parameters MUST be strings in the yaml file, otherwise the kubectl parser will fail. If in doubt, enclose each in double quotes ("")
-
The 'placement_mode' will continue to work in Datera OS versions >= 3.3, however the 'placement_policy' takes precedence.
-
StorageClass parameters cannot be patched using "kubectl apply -f <>" command. Any changes needs a delete and re-create of the StorageClass with modified parameters. You can also use "kubectl replace .." which does delete and replace of StorageClass. Only subsequent PVCs/PVs which references this modified StorageClass will see the change. There is no impact to existing PVCs/PVs.
$ kubectl replace -f csi-storageclass.yaml --force
storageclass.storage.k8s.io "csi-storageclass" deleted
storageclass.storage.k8s.io/csi-storageclass replaced
$
A volume on Datera backend is created automatically when a Persistent Volume Claim (PVC) is created. This PVC can be further referenced in a Pod manifest to use the volume.
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: csi-pvc
namespace: default
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 100Gi
storageClassName: dat-block-storage
$ kubectl create -f csi-pvc.yaml
Create and save the following as csi-app.yaml. Note that this Pod claims a volume by specifying the name of a PVC claim "csi-pvc".
kind: Pod
apiVersion: v1
metadata:
name: my-csi-app
spec:
containers:
- name: my-app-image
image: alpine
volumeMounts:
- mountPath: "/data"
name: my-app-volume
command: [ "sleep", "1000000" ]
volumes:
- name: my-app-volume
persistentVolumeClaim:
claimName: csi-pvc
$ kubectl create -f csi-app.yaml
To create a volume snapshot in kubernetes you can use the following VolumeSnapshotClass and VolumeSnapshot as an example. Save the following as csi-snap-class.yaml.
apiVersion: snapshot.storage.k8s.io/v1beta1
kind: VolumeSnapshotClass
metadata:
name: csi-snap-class
driver: dsp.csi.daterainc.io
deletionPolicy: Retain
parameters:
Here are a list of supported snapshot parameters for the plugin: (v1.0.7+)
Name | Default |
---|---|
remote_provider_uuid |
"" |
type |
local options: local, remote, local_and_remote |
Example VolumeSnapshotClass yaml file with parameters (when saving snapshot to a remote provider):
apiVersion: snapshot.storage.k8s.io/v1beta1
kind: VolumeSnapshotClass
metadata:
name: csi-snap-class
driver: dsp.csi.daterainc.io
deletionPolicy: Retain
parameters:
remote_provider: c7f97223-81d9-44fe-ae7b-7c27daf6c288
type: local_and_remote
$ kubectl create -f csi-snap-class.yaml
Create and save the following as csi-snap.yaml.
apiVersion: snapshot.storage.k8s.io/v1beta1
kind: VolumeSnapshot
metadata:
name: csi-pvc-snap
spec:
volumeSnapshotClassName: csi-snap-class
source:
persistentVolumeClaimName: csi-pvc
$ kubectl create -f csi-snap.yaml
We can now view the snapshot using kubectl command.
# kubectl get volumesnapshot
NAME READYTOUSE SOURCEPVC SOURCESNAPSHOTCONTENT RESTORESIZE SNAPSHOTCLASS SNAPSHOTCONTENT CREATIONTIME AGE
csi-pvc-snap true csi-pvc 1Gi csi-snap-class snapcontent-4ea69ee8-444c-4106-b000-0b7c91cb847f 22m 22m
#
# kubectl get volumesnapshotcontents
NAME READYTOUSE RESTORESIZE DELETIONPOLICY DRIVER VOLUMESNAPSHOTCLASS VOLUMESNAPSHOT AGE
snapcontent-4ea69ee8-444c-4106-b000-0b7c91cb847f true 1073741824 Retain dsp.csi.daterainc.io csi-snap-class csi-pvc-snap 22m
#
Now we can use this snapshot to create a new PVC.
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: csi-pvc-restore
namespace: default
spec:
storageClassName: dat-block-storage
dataSource:
name: csi-pvc-snap
kind: VolumeSnapshot
apiGroup: snapshot.storage.k8s.io
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 1Gi
For other examples, such as creating/managing volume snapshots, resizing volumes, adding CHAP support, overriding Datera templates, using PVCs for deployment, etc., please check the 'deploy/examples' folder.
You can collect logs from the entire Datera CSI plugin via the csi_log_collect.sh
script in the datera-csi/assets
folder. Basic log collection is very simple. To collect logs from all the master and worker nodes, run the script with no arguments on the Kubernetes master node. To collect logs from a specific worker node or master node, specify the '-p ' option after the script. See examples below.
$ chmod +x ./assets/csi_log_collect.sh
$ ./assets/csi_log_collect.sh
$ ./assets/csi_log_collect.sh -p csi-provisioner
$ ./assets/csi_log_collect.sh -p csi-node
If driver pods have already crashed or terminated, use the -p or --previous option to get the logs.
$ kubectl --v=8 logs <podname> -n <namespace> --previous
If the CSI driver pod(s) is/are not coming up to a stable Running state, check the following:
(1) Are there any health check failures?
$ kubectl describe pod csi-provisioner-0 -n kube-system
$ kubectl describe pod csi-node-xxxxx -n kube-system
(2) Check whether Kubernetes killed the Pod for lack of resources or excess usage of cpu/mem resources. The kubelet on the kubernetes host node will provide additional insights into failure.
$ kubectl get pod csi-provisioner-0 -n kube-system -o wide
$ kubectl get pod csi-node-xxxxx -n kube-system -o wide
$ ssh user@k8s-hostnode
$ grep <podname> /var/log/containers
$ sudo journalctl -u kubelet -n 100 --no-pager
(3) Is the iscsid deamon running on all the worker nodes?
$ systemctl --all | grep iscsi-recv
(4) Check for kubernetes events.
$ kubectl get events -n kube-system --sort-by='.metadata.creationTimestamp'
$ kubectl get events -n kube-system --sort-by='.lastTimestamp'
Sometimes customer setups require a bit of flexibility. These environment variables allow for tuning the plugin to behave in atypical ways. USE THESE WITH CAUTION.
- DAT_SOCKET -- Socket that driver listens on
- DAT_HEARTBEAT -- Interval to perform Datera heartbeat function
- DAT_TYPE -- Which CSI services to expose on the binary
- DAT_VOL_PER_NODE -- Max volumes per node setting
- DAT_DISABLE_MULTIPATH -- Disable multipath (for use with bonded nics)
- DAT_REPLICA_OVERRIDE -- Override set replica counts to 1 (for single-node systems)
- DAT_METADATA_DEBUG -- Calculates metadata size before sending (for checking 2KB hard-limit)
- DAT_DISABLE_LOGPUSH -- Disables pushing plugin logs to the Datera system
- DAT_LOGPUSH_INTERVAL -- Sets interval between logpushes to the Datera system
- DAT_FORMAT_TIMEOUT -- Sets the timeout duration for volume format calls (default 60 seconds)
In Rancher setup, the kubelet is run inside a container and hence may not have access to the socket /var/datera/csi-iscsi.sock on the host. Run '# nc -U /var/datera/csi-iscsi.sock' from inside the kubelet container and verify whether the socket is listening. If not, a bind mount would be needed as specified here: https://docs.docker.com/storage/bind-mounts/
--mount type=bind,source="/var/datera/csi-iscsi.sock"/target,target=/var/datera/csi-iscsi.sock
Driver upgrades and downgrades can be done by running a 'kubectl delete -f <csi_driver_yaml_used_to_create>' followed by 'kubectl delete -f <csi_driver_yaml_for_new_version>'. For example, a downgrade from v1.0.11 to v1.0.10 can be done as follows:
# kubectl delete -f csi-datera-1.0.11.yaml
# kubectl create -f csi-datera-1.0.10.yaml