sriov-device-plugin cannot be created

[peng-isi]

What issue would you like to bring attention to

I tried to attach a VF of Mellanox IB interface to a pod. However, when I added one item to the ConfigMap.yaml as follows:

        {
        "resourceName": "mlnx_sriov_netdevice",
        "resourcePrefix": "mellanox.com",
        "selectors": {
            "linkTypes": ["infiniband"],
            "vendors": ["15b3"],
            "devices": ["1003"],
            "drivers": ["mlx4_core"],
        }
      }

I tried to run Daemonset by the command "kubectl apply -f deployments/k8s-v1.16/sriovdp-daemonset.yaml", the kube-sriov-device-plugin pod can not be created. What's wrong with the the configMap.yaml? Any help is appreciated!

What is the impact of this issue?

Do you have a proposed response or remediation for the issue?

[martinkennelly]

Hi,
Can you post logs from the daemon set?
Can you post the full configmap?
What version of k8 are you using?

[peng-isi]

Martinkennelly,

Thanks for help.

The full configmap is appended as follows:

apiVersion: v1
kind: ConfigMap
metadata:
name: sriovdp-config
namespace: kube-system
data:
config.json: |
{
"resourceList": [{
"resourceName": "intel_sriov_netdevice",
"selectors": {
"vendors": ["8086"],
"devices": ["154c", "10ed"],
"drivers": ["i40evf", "ixgbevf"]
}
},
{
"resourceName": "intel_sriov_dpdk",
"selectors": {
"vendors": ["8086"],
"devices": ["154c", "10ed"],
"drivers": ["vfio-pci"],
"pfNames": ["enp0s0f0","enp2s2f1"]
}
},
{
"resourceName": "mlnx_sriov_netdevice",
"resourcePrefix": "mellanox.com",
"selectors": {
"linkTypes": ["infiniband"],
"vendors": ["15b3"],
"devices": ["1003"],
"drivers": ["mlx4_core"],
}
}
]
}
~

The log for the daemon set is appended as

Name: kube-sriov-device-plugin-amd64-ph6hl
Namespace: kube-system
Priority: 0
Node: hp3/10.13.206.63
Start Time: Fri, 31 Jul 2020 17:23:33 -0400
Labels: app=sriovdp
controller-revision-hash=67d75dbf9f
name=sriov-device-plugin
pod-template-generation=1
tier=node
Annotations:
Status: Running
IP: 10.13.206.63
IPs:
IP: 10.13.206.63
Controlled By: DaemonSet/kube-sriov-device-plugin-amd64
Containers:
kube-sriovdp:
Container ID: docker://4715c95e5fbf2f98353be8cd9a185cf3741006bcfa7a20a74bde21c6bccef61f
Image: nfvpe/sriov-device-plugin
Image ID: docker-pullable://nfvpe/sriov-device-plugin@sha256:46d9b7002addcf94a6799648a2b7f50a0f6f99d0ce52ce35e2fd6c889c7989eb
Port:
Host Port:
Args:
--log-dir=sriovdp
--log-level=10
State: Waiting
Reason: CrashLoopBackOff
Last State: Terminated
Reason: Completed
Exit Code: 0
Started: Tue, 04 Aug 2020 12:11:30 -0400
Finished: Tue, 04 Aug 2020 12:11:30 -0400
Ready: False
Restart Count: 1069
Environment:
Mounts:
/etc/pcidp from config-volume (rw)
/var/lib/kubelet/ from devicesock (rw)
/var/log from log (rw)
/var/run/secrets/kubernetes.io/serviceaccount from sriov-device-plugin-token-j46d5 (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
devicesock:
Type: HostPath (bare host directory volume)
Path: /var/lib/kubelet/
HostPathType:
log:
Type: HostPath (bare host directory volume)
Path: /var/log
HostPathType:
config-volume:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: sriovdp-config
Optional: false
sriov-device-plugin-token-j46d5:
Type: Secret (a volume populated by a Secret)
SecretName: sriov-device-plugin-token-j46d5
Optional: false
QoS Class: BestEffort
Node-Selectors: beta.kubernetes.io/arch=amd64
Tolerations: node-role.kubernetes.io/master:NoSchedule
node.kubernetes.io/disk-pressure:NoSchedule
node.kubernetes.io/memory-pressure:NoSchedule
node.kubernetes.io/network-unavailable:NoSchedule
node.kubernetes.io/not-ready:NoExecute
node.kubernetes.io/pid-pressure:NoSchedule
node.kubernetes.io/unreachable:NoExecute
node.kubernetes.io/unschedulable:NoSchedule
Events:
Type Reason Age From Message

---

Warning BackOff 61s (x25270 over 3d18h) kubelet, hp3 Back-off restarting failed container

The k8 on master node is v1.18.1 and the one on the slave node (HP6) is v1.18.6.

The lspci on hp6 is

21:00.0 Network controller [0280]: Mellanox Technologies MT27500 Family [ConnectX-3] [15b3:1003]
Subsystem: Hewlett-Packard Company InfiniBand FDR/EN 10/40Gb Dual Port 544QSFP Adapter [103c:18d6]
Physical Slot: 4
Flags: bus master, fast devsel, latency 0, IRQ 28
Memory at fbf00000 (64-bit, non-prefetchable) [size=1M]
Memory at f8000000 (64-bit, prefetchable) [size=32M]
Capabilities: [40] Power Management version 3
Capabilities: [48] Vital Product Data
Capabilities: [9c] MSI-X: Enable+ Count=128 Masked-
Capabilities: [60] Express Endpoint, MSI 00
Capabilities: [100] Alternative Routing-ID Interpretation (ARI)
Capabilities: [148] Device Serial Number 00-02-c9-03-00-f6-ef-f0
Capabilities: [108] Single Root I/O Virtualization (SR-IOV)
Capabilities: [154] Advanced Error Reporting
Capabilities: [18c] #19
Kernel driver in use: mlx4_core
Kernel modules: mlx4_core

21:00.1 Network controller [0280]: Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function] [15b3:1004]
Subsystem: Hewlett-Packard Company MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function] [103c:61b0]
Physical Slot: 4
Flags: bus master, fast devsel, latency 0
[virtual] Memory at d8000000 (64-bit, prefetchable) [size=32M]
Capabilities: [60] Express Endpoint, MSI 00
Capabilities: [9c] MSI-X: Enable+ Count=220 Masked-
Capabilities: [40] Power Management version 0
Kernel driver in use: mlx4_core
Kernel modules: mlx4_core

21:00.2 Network controller [0280]: Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function] [15b3:1004]
Subsystem: Hewlett-Packard Company MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function] [103c:61b0]
Physical Slot: 4
Flags: bus master, fast devsel, latency 0
[virtual] Memory at da000000 (64-bit, prefetchable) [size=32M]
Capabilities: [60] Express Endpoint, MSI 00
Capabilities: [9c] MSI-X: Enable+ Count=220 Masked-
Capabilities: [40] Power Management version 0
Kernel driver in use: mlx4_core
Kernel modules: mlx4_core

21:00.3 Network controller [0280]: Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function] [15b3:1004]
Subsystem: Hewlett-Packard Company MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function] [103c:61b0]
Physical Slot: 4
Flags: bus master, fast devsel, latency 0
[virtual] Memory at dc000000 (64-bit, prefetchable) [size=32M]
Capabilities: [60] Express Endpoint, MSI 00
Capabilities: [9c] MSI-X: Enable+ Count=220 Masked-
Capabilities: [40] Power Management version 0
Kernel driver in use: mlx4_core
Kernel modules: mlx4_core

21:00.4 Network controller [0280]: Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function] [15b3:1004]
Subsystem: Hewlett-Packard Company MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function] [103c:61b0]
Physical Slot: 4
Flags: bus master, fast devsel, latency 0
[virtual] Memory at de000000 (64-bit, prefetchable) [size=32M]
Capabilities: [60] Express Endpoint, MSI 00
Capabilities: [9c] MSI-X: Enable+ Count=220 Masked-
Capabilities: [40] Power Management version 0
Kernel driver in use: mlx4_core
Kernel modules: mlx4_core

[peng-isi]

I finally found out the problem with the configMap: the "kube-sriov-device-plugin" pod can be correctly created when I add " \"IsRdma\": true" to the "mlnx_sriov_networkdevice" resource as follows:

{
"resourceName": "mlnx_sriov_netdevice",
"resourcePrefix": "mellanox.com",
"selectors": {
"linkTypes": ["infiniband"],
"vendors": ["15b3"],
"devices": ["1003"],
"drivers": ["mlx4_core"],
"isRdma": true
}
}

I don't know why, but it allows the device plugin to be created.

[martinkennelly]

Good to hear.
CC @adrianchiris

[adrianchiris]

Hi There,
So, ConnectX3 family devices is not officially supported as mentioned (or rather, lack of mention) in the README

Also i think we would need the device plugin pod logs for more logging info. (kubectl logs <pod-name>)

Also in the devices entry you need to provide the VF's PCI device ID which is 1004 not 1003
1003 is the PCI PF device ID.

[moshe010]

we did enabled it to one customer so it should work, you need to know how to use it with VF selector

[zshi-redhat]

Hi There,
So, ConnectX3 family devices is not officially supported as mentioned (or rather, lack of mention) in the README

Also i think we would need the device plugin pod logs for more logging info. (kubectl logs <pod-name>)

+1, The log of sriov-device-plugin daemon when isRdma is not specified would help identify the issue.
@peng-isi , if it's ok for you to provide the log message ^

[moshe010]

Also if you merged this #226 we can see the issue with isRdma

[peng-isi]

thank @adrianchiris and @zshi-redhat. I finally get it work.

I made two errors in my configMap: 1) use PF ID instead of VF ID; and 2) I didn't delete the comma following last item in IB configuration when I deleted the "isRama" item, but there is no error when I create the map.

Right now, the device plugin daemonset works with or without the "isRdma" item specified.

[peng-isi]

@moshe010 thanks. If possible, could you point me to some examples on VF selector?

[zshi-redhat]

thank @adrianchiris and @zshi-redhat. I finally get it work.

I made two errors in my configMap: 1) use PF ID instead of VF ID; and 2) I didn't delete the comma following last item in IB configuration when I deleted the "isRama" item, but there is no error when I create the map.

The 2) explains why the device plugin pod crashed. it failed to parse the configMap in the wrong format.