Add support for running inside a VM with vfio-noiommu #272

atyronesmith · 2020-10-07T20:03:48Z

This PR is required for openshift/sriov-network-operator#369.

This PR enables the mapping of a no-iommu vfio device into a container.

Associated system configuration enables no-iommu for the vfio-pci driver.

This PR is required to allow Kubernetes to run in a virtual environment that does not provide a virtualized iommu. The initial target environment is OpenStack. OpenStack Nova does not currently provide a virtualized iommu. Support for vfio-noiommu was added to linux a few years ago with the caveats outlined below...

`There is really no way to safely give a user full access to a DMA
capable device without an IOMMU to protect the host system. There is
also no way to provide DMA translation, for use cases such as device
assignment to virtual machines. However, there are still those users
that want userspace drivers even under those conditions. The UIO
driver exists for this use case, but does not provide the degree of
device access and programming that VFIO has. In an effort to avoid
code duplication, this introduces a No-IOMMU mode for VFIO.

This mode requires building VFIO with CONFIG_VFIO_NOIOMMU and enabling
the "enable_unsafe_noiommu_mode" option on the vfio driver. This
should make it very clear that this mode is not safe. Additionally,
CAP_SYS_RAWIO privileges are necessary to work with groups and
containers using this mode. Groups making use of this support are
named /dev/vfio/noiommu-$GROUP and can only make use of the special
VFIO_NOIOMMU_IOMMU for the container. Use of this mode, specifically
binding a device without a native IOMMU group to a VFIO bus driver
will taint the kernel and should therefore not be considered
supported. This patch includes no-iommu support for the vfio-pci bus
driver only.`

In the future, the current QEMU machine type in use for OpenStack is i440fx. The
machine type should be switched to Q35 in the future. The Q35 machine provides
a virtualized iommu implementation and will obsolete this patch.

In summary, the use of noiommu introduces security concerns, but the performance
boost provided by vfio-pci can be justified in many cases.

pkg/utils/utils.go

zshi-redhat · 2020-10-20T01:12:14Z

/lgtm

zshi-redhat · 2020-10-20T01:12:48Z

/cc @ahalim-intel @adrianchiris @martinkennelly @moshe010 PTAL

pkg/resources/vfioResource.go

adrianchiris

Overall code looks OK to me.
Small comment posted.

moshe010

Can update the readme for the VM with vfio-noiommu?
According to your PR description the pod should have SYS_RAWIO in the capabilities or to be privileged maybe you can have examples like we have for dpdk or rdma https://github.com/k8snetworkplumbingwg/sriov-network-device-plugin/tree/master/docs/dpdk.
It is not explicitly stated in the PR, but my understanding is that the use-case is for running dpdk POD in the VM. Am I right? if that the case I think we need to update plugin/tree/master/docs/dpdk/README file with that use-case.

martinkennelly · 2020-10-27T12:54:09Z

Second the three points made by @moshe010 especially emphasizing the addition of documentation. PR looks good. I am currently testing it locally. Thank you for this.

Fixed link path Check for missing name fix name missing fix err name Add dual return Additional Documentation

atyronesmith · 2020-11-05T00:32:18Z

@moshe010 @martinkennelly Docs pushed

docs/dpdk/README-virt.md

martinkennelly · 2020-11-07T16:55:21Z

docs/dpdk/README-virt.md

+
+With `vfio-pci` an application must run privilege Pod with  **IPC_LOCK** and **CAP_SYS_RAWIO** capability.
+
+# Example deployment


Since this feature does not work with SRIOV CNI - can you add in an example CNI to use and state this feature doesn't work with SRIOV CNI. Thank you.

I tried with host-device, but also failed.
I am getting:

failed to find host device: no net directory under pci device 0000:00:06.0

Is there a patch outstanding on a CNI to fix this for vfio-pci devices?

so at this point no CNI is expected to be used if i understood correctly

martinkennelly · 2020-11-09T08:55:34Z

LGTM overall besides nits / CNI Question.

atyronesmith · 2020-11-09T13:51:01Z

@martinkennelly Aplogies, forgot to document the SR-IOV CNI situation. I will include an example that I have been using for testing. Basically, just remove the annotations...

  annotations:
    k8s.v1.cni.cncf.io/networks: sriov-net1

and keep the resources...

    resources:
      requests:
        intel.com/intel_sriov_netdevice: '1'
      limits:
        intel.com/intel_sriov_netdevice: '1'
```
unchanged.

adrianchiris · 2020-11-10T07:41:18Z

docs/dpdk/README-virt.md

+
+# Pod Usage
+
+Unfortunately, the SR-IOV CNI is not compatible with the noiommu feature.  To use the noiommu feature, remove the 


This section kinda hints that SR-IOV CNI should support this, which i am not sure is the case.

suggestion:
just have the pod example, then a note:

Note: When consuming a VFIO device in a virtual environment secondary network is not required as network configuration for the underlying VF should be performed at the hypervisor level.

docs/dpdk/pod_testpmd_virt.yaml

adrianchiris · 2020-11-10T09:10:43Z

README.md

@@ -367,6 +368,45 @@ $ kubectl get node node1 -o json | jq '.status.allocatable'

 ```

+### Virtual environments with no iommu


This seems to me a rather specialized usecase, IMO we move this section under docs/dpdk/README-virt

we can have a general description here and point to docs/dpdk/README-virt
we should also explain that not all selectors are applicable.

e.g

Virtual environments

SR-IOV network device plugin supports running in a virtualized environment, however, not all device selectors are applicable as the VFs are passthrough to the VM without any association to their respective PF, hence any device selector that relies on the association between a VF and its PF will not work.

The following selectors will not work in a virtualized environment: pfNames, rootDevices

Virtual environments with no iommu

SR-IOV network device plugin supports allocating VFIO devices in a virtualized environment without a virtualized iommu
for more information refer to {add link}

P.S
I realize we are missing some documentation on consuming sr-iov resources in virtualized environment in general, however thats not related to this PR.

adrianchiris · 2020-11-11T08:52:00Z

docs/dpdk/pod_testpmd_virt.yaml

+  - name: testpmd
+    image: <DPDK testpmd image>
+    securityContext:
+        privileged: true


so in the example pod should it be privileged ? or will it generally work with CAP_IPC_LOCK and CAP_SYS_RAWIO ?
id prefer to list the minimum capabilities.

adrianchiris · 2020-11-11T08:55:32Z

docs/dpdk/README-virt.md

+
+# Usage
+
+_When consuming a VFIO device in a virtual environment, a secondary network is not required as network configuration for the underlying VF should be performed at the hypervisor level._


nit: formatting her should be with a > to emphasize its a note/commet similarly as done in other part of the docs e.g L40 in this file

When consuming a VFIO device in a virtual environment, a secondary network is not required as network configuration for the underlying VF should be performed at the hypervisor level

zshi-redhat · 2020-11-18T14:56:47Z

@adrianchiris could you take a look at the new doc update?

adrianchiris

LGTM, Thank you !

martinkennelly

LGTM! Thanks!

atyronesmith force-pushed the master branch 3 times, most recently from 11b4cfb to bcd1c5c Compare October 9, 2020 02:43

zshi-redhat reviewed Oct 13, 2020

View reviewed changes

pkg/utils/utils.go Show resolved Hide resolved

pkg/utils/utils.go Show resolved Hide resolved

adrianchiris approved these changes Oct 22, 2020

View reviewed changes

pkg/resources/vfioResource.go Outdated Show resolved Hide resolved

adrianchiris approved these changes Oct 22, 2020

View reviewed changes

atyronesmith force-pushed the master branch from efc882e to 395931e Compare October 23, 2020 13:46

atyronesmith changed the title ~~WIP - Add support for running inside a VM with vfio-noiommu~~ Add support for running inside a VM with vfio-noiommu Oct 23, 2020

moshe010 reviewed Oct 27, 2020

View reviewed changes

Add support for running inside a VM with vfio-noiommu

cba9f30

Fixed link path Check for missing name fix name missing fix err name Add dual return Additional Documentation

atyronesmith force-pushed the master branch from 395931e to cba9f30 Compare November 4, 2020 20:01

zshi-redhat approved these changes Nov 5, 2020

View reviewed changes

martinkennelly reviewed Nov 7, 2020

View reviewed changes

Additional Documentation

b4c23b6

atyronesmith requested a review from moshe010 November 10, 2020 01:45

adrianchiris reviewed Nov 10, 2020

View reviewed changes

docs/dpdk/pod_testpmd_virt.yaml Outdated Show resolved Hide resolved

adrianchiris requested changes Nov 10, 2020

View reviewed changes

Additional example information

7d6b1f2

atyronesmith requested a review from adrianchiris November 11, 2020 02:36

adrianchiris reviewed Nov 11, 2020

View reviewed changes

Additional clarification in documentation and examples

b71551f

atyronesmith requested review from adrianchiris and ahalimx86 November 13, 2020 00:59

adrianchiris approved these changes Nov 18, 2020

View reviewed changes

martinkennelly approved these changes Nov 18, 2020

View reviewed changes

zshi-redhat merged commit 4563418 into k8snetworkplumbingwg:master Nov 18, 2020

adrianchiris mentioned this pull request Nov 24, 2020

sriov network device plugin in a virtualized environment #289

Open

adrianchiris mentioned this pull request Jan 4, 2021

Enable the use of vfio-noiommu when kubernetes is run in a virtual environment. #278

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add support for running inside a VM with vfio-noiommu #272

Add support for running inside a VM with vfio-noiommu #272

atyronesmith commented Oct 7, 2020 •

edited

Loading

zshi-redhat commented Oct 20, 2020

zshi-redhat commented Oct 20, 2020

adrianchiris left a comment

moshe010 left a comment

martinkennelly commented Oct 27, 2020

atyronesmith commented Nov 5, 2020

martinkennelly Nov 7, 2020

martinkennelly Nov 7, 2020

adrianchiris Nov 10, 2020

martinkennelly commented Nov 9, 2020

atyronesmith commented Nov 9, 2020

adrianchiris Nov 10, 2020

adrianchiris Nov 10, 2020

adrianchiris Nov 10, 2020

adrianchiris Nov 11, 2020

adrianchiris Nov 11, 2020

zshi-redhat commented Nov 18, 2020

adrianchiris left a comment

martinkennelly left a comment


		With `vfio-pci` an application must run privilege Pod with IPC_LOCK and CAP_SYS_RAWIO capability.

		# Example deployment


		# Pod Usage

		Unfortunately, the SR-IOV CNI is not compatible with the noiommu feature. To use the noiommu feature, remove the

		@@ -367,6 +368,45 @@ $ kubectl get node node1 -o json \| jq '.status.allocatable'

		```

		### Virtual environments with no iommu


		# Usage

		_When consuming a VFIO device in a virtual environment, a secondary network is not required as network configuration for the underlying VF should be performed at the hypervisor level._

Add support for running inside a VM with vfio-noiommu #272

Add support for running inside a VM with vfio-noiommu #272

Conversation

atyronesmith commented Oct 7, 2020 • edited Loading

zshi-redhat commented Oct 20, 2020

zshi-redhat commented Oct 20, 2020

adrianchiris left a comment

Choose a reason for hiding this comment

moshe010 left a comment

Choose a reason for hiding this comment

martinkennelly commented Oct 27, 2020

atyronesmith commented Nov 5, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

martinkennelly commented Nov 9, 2020

atyronesmith commented Nov 9, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Virtual environments

Virtual environments with no iommu

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

zshi-redhat commented Nov 18, 2020

adrianchiris left a comment

Choose a reason for hiding this comment

martinkennelly left a comment

Choose a reason for hiding this comment

atyronesmith commented Oct 7, 2020 •

edited

Loading