Add support to virtio-vDPA devices #2664

amorenoz · 2021-11-18T11:04:12Z

- What this PR does and why is it needed

This PR works in combination with this PR in the SR-IOV Network Device Plugin to provide support for vDPA devices.

This functionality is based on OvS Hardware Offload and allows ovn-kubernetes to expose an open standard netdev such as a virtio-net to the pod.

Most of the heavy-lifting (detecting vdpa devices, ensuring they are bound the right driver, etc) is done by the SR-IOV Network Device Plugin. From a ovn-kubernetes perspective, it's just a matter of selecting the right netdev to move to the pod's namespace.

- Special notes for reviewers

I would like feedback on a number of topics:

Right now, this PR just detects if the PCI device has a virtio-vdpa device using govdpa and exposes the virtio-net netdev if so. It is my understanding that supported cards, even if they allow the creation of two simultaneous netdevs (the vendor one and the virtio-vdpa device), the vdpa netdev takes precedence if it exists. Therefore, this logic would be sane.

If that is not the case we would need to add support to the Device Information Spec so we can read the information coming from the Device Plugin and unambiguously determine the type of device we're handling

I've placed the code that returns the virtio-net netdev in pkg/util/sriovnet_linux.go. It might not be the perfect place as it seems more of a wrapper around https://github.com/Mellanox/sriovnet. So this is a question for @moshe010 and @adrianchiris: WDYT? Would you accept this logic into https://github.com/Mellanox/sriovnet?

- How to verify it

You need a working OvS Hardware Offload setup with a ConnectX6-DX NIC. Then, follow the instructions that are added in docs/ovs_offload.md as part of this PR to configure virtio-vDPA devices.

- Description for the changelog

Add support for virtio-vdpa devices

amorenoz · 2021-11-18T11:04:27Z

/cc @zshi-redhat

adrianchiris · 2021-11-18T16:21:13Z

ve placed the code that returns the virtio-net netdev in pkg/util/sriovnet_linux.go. It might not be the perfect place as it seems more of a wrapper around https://github.com/Mellanox/sriovnet. So this is a question for @moshe010 and @adrianchiris: WDYT? Would you accept this logic into https://github.com/Mellanox/sriovnet?

i think it makes sense. maybe govdpa should go to npwg.

amorenoz · 2021-11-18T16:30:33Z

ve placed the code that returns the virtio-net netdev in pkg/util/sriovnet_linux.go. It might not be the perfect place as it seems more of a wrapper around https://github.com/Mellanox/sriovnet. So this is a question for @moshe010 and @adrianchiris: WDYT? Would you accept this logic into https://github.com/Mellanox/sriovnet?

i think it makes sense. maybe govdpa should go to npwg.

Agree, wasn't there an initiative to move some common stuff to a library in npwg that could be used by more projects?
I can propose govdpa to npwg in the next meeting

Should sriovnet also go to npwg too?
In any case I'll send a PR to add this part of vdpa code into sriovnet

coveralls · 2021-11-27T17:48:48Z

Coverage decreased (-0.09%) to 53.176% when pulling 4b108c3 on amorenoz:vdpa into 1a2bdaf on ovn-org:master.

amorenoz · 2021-12-14T12:09:58Z

Hi @adrianchiris , as we discussed, having sriovnet depend on govdpa does not make much sense since the goal of vdpa is to (eventually) work with all possible vdpa parents (sriov is just one of them).

I've moved the govdpa-related stuff to an independent vdpa_linux.go wrapper so it can be mocked and added unit tests. PTAL

amorenoz · 2022-01-10T12:27:52Z

Rebased on top of @girishmg 's latest change: 60eeb3a

amorenoz · 2022-04-07T06:57:44Z

@girishmg @dcbw @danwinship any comments on this?

dcbw · 2022-07-12T14:39:59Z

@amorenoz could you rebase the PR? Looks OK to me, just needs the rebase.

lmilleri · 2022-09-20T16:11:02Z

@dcbw I rebased and pushed the PR again, but got an error in the ovn-ci / e2e-dual-conversion. Hope this is an expected failure and not due to my changes?

zshi-redhat · 2022-09-21T08:33:34Z

/retest

ovn-robot · 2022-09-21T08:33:52Z

Oops, something went wrong:

Unable to retry this workflow run because it was created over a month ago

zshi-redhat · 2022-09-21T08:38:14Z

Oops, something went wrong:

Unable to retry this workflow run because it was created over a month ago

Interesting, retest doesn't work for long living PR, @dcbw do you know how to rerun the test w/o creating a new PR (the CI failure doesn't seem to be related to the vdpa change)?

zshi-redhat · 2022-09-21T08:47:55Z

go-controller/go.sum

@@ -497,6 +497,8 @@ github.com/juju/utils v0.0.0-20180808125547-9dfc6dbfb02b/go.mod h1:6/KLg8Wz/y2KV
 github.com/juju/version v0.0.0-20161031051906-1f41e27e54f2/go.mod h1:kE8gK5X0CImdr7qpSKl3xB2PmpySSmfj7zVbkZFs81U=
 github.com/julienschmidt/httprouter v1.2.0/go.mod h1:SYymIcj16QtmaHHD7aYtjjsJG7VTCxuUUipMqKk8s4w=
 github.com/julienschmidt/httprouter v1.3.0/go.mod h1:JR6WtHb+2LUe8TCKY3cZOxFyyO8IZAc4RVcycCCAKdM=
+github.com/k8snetworkplumbingwg/govdpa v0.1.3 h1:FZRhTMB1e3yWwSEy+l4eS73WioyMaL+vmFZ8JNwn+Uk=


@lmilleri I vaguely remembered that you mentioned we need latest changes in govdpa lib, but this is still using 0.1.3 version, is this expected?

trozet · 2022-09-23T19:31:08Z

Oops, something went wrong:
Unable to retry this workflow run because it was created over a month ago
Interesting, retest doesn't work for long living PR, @dcbw do you know how to rerun the test w/o creating a new PR (the CI failure doesn't seem to be related to the vdpa change)?

@zshi-redhat I can restart it, but i think it wont restart now because there are conflicting files. If this PR is still important can you rebase @amorenoz and we will review.

zshi-redhat · 2022-09-26T01:16:38Z

@lmilleri could you please rebase?

lmilleri · 2022-09-26T11:06:39Z

@zshi-redhat there are some changes in the govdpa library under review: k8snetworkplumbingwg/govdpa#15.
I think it would make sense to push the above PR first, create a new version and then rebase this PR afterwards

Add support to sriov-based virtio-vdpa devices. When a sr-iov device has a vdpa device created, and it is bound to virtio_vdpa driver, use the associated virtio-net device instead of the vendor one. In order to determine whether a device is vdpa or not as well as its device path, use external library github.com/k8snetworkplumbingwg/govdpa Signed-off-by: Adrian Moreno <amorenoz@redhat.com>

vdpa is an addition on top of ovs_offload so it makes sense to document it as part of it. Signed-off-by: Adrian Moreno <amorenoz@redhat.com>

lmilleri · 2022-11-04T12:17:09Z

Rebased from master and updated dependency to the newly govdpa v0.1.4 release.
Please review

lmilleri · 2022-11-08T10:02:30Z

Please consider this PR on hold, I found a potential issue in govdpa v0.1.4

Signed-off-by: Leonardo Milleri <lmilleri@redhat.com>

lmilleri · 2022-11-17T17:06:47Z

Fixed integration issue with govdpa v.0.1.4.
Please review.

@zshi-redhat @dcbw

zshi-redhat · 2022-11-30T10:00:52Z

/retest

ovn-robot · 2022-11-30T10:11:50Z

Oops, something went wrong:

Unable to retry this workflow run because it was created over a month ago

lmilleri · 2022-12-02T11:33:59Z

@trozet @dcbw do you think this PR can be merged?
If not, please advise on the way forward.
Apparently "/retest" is not working, any workaround?

amorenoz · 2022-12-13T07:52:05Z

@trozet @zshi-redhat @dcbw Can you please take a look at this PR?

dcbw · 2023-01-04T15:46:30Z

/lgtm

mostly just helper code

lmilleri · 2023-01-09T12:04:33Z

Yes, it is mostly helper code apart from go-controller/pkg/cni/cni.go, in which resides the logic for returning the virtio/vdpa device if vdpa device exists.

lmilleri · 2023-01-11T09:50:52Z

@girishmg can you please review it?

amorenoz force-pushed the vdpa branch 2 times, most recently from e9d23be to 77d4b68 Compare December 14, 2021 12:04

amorenoz changed the title ~~RFC: Add support to virtio-vDPA devices~~ Add support to virtio-vDPA devices Dec 14, 2021

amorenoz force-pushed the vdpa branch from 77d4b68 to 5ff8c51 Compare December 14, 2021 12:09

amorenoz force-pushed the vdpa branch from 5ff8c51 to d531392 Compare December 29, 2021 10:23

lmilleri force-pushed the vdpa branch from d531392 to b57447d Compare September 20, 2022 13:54

zshi-redhat reviewed Sep 21, 2022

View reviewed changes

amorenoz mentioned this pull request Sep 27, 2022

Bump containernetworking #3137

Merged

amorenoz added 2 commits November 2, 2022 17:25

docs: add vdpa section in ovs_offload docs

8897972

vdpa is an addition on top of ovs_offload so it makes sense to document it as part of it. Signed-off-by: Adrian Moreno <amorenoz@redhat.com>

lmilleri force-pushed the vdpa branch from b57447d to ee08251 Compare November 4, 2022 09:54

Updated dependency to govdpa v0.1.4

4b108c3

Signed-off-by: Leonardo Milleri <lmilleri@redhat.com>

lmilleri force-pushed the vdpa branch from ee08251 to 4b108c3 Compare November 17, 2022 13:06

lmilleri mentioned this pull request Nov 22, 2022

Vdpa introduction k8snetworkplumbingwg/sriov-network-operator#377

Merged

trozet approved these changes Jan 13, 2023

View reviewed changes

trozet merged commit 57bcf26 into ovn-org:master Jan 13, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add support to virtio-vDPA devices #2664

Add support to virtio-vDPA devices #2664

amorenoz commented Nov 18, 2021

amorenoz commented Nov 18, 2021

adrianchiris commented Nov 18, 2021

amorenoz commented Nov 18, 2021

coveralls commented Nov 27, 2021 •

edited

Loading

amorenoz commented Dec 14, 2021

amorenoz commented Jan 10, 2022

amorenoz commented Apr 7, 2022

dcbw commented Jul 12, 2022

lmilleri commented Sep 20, 2022

zshi-redhat commented Sep 21, 2022

ovn-robot commented Sep 21, 2022

zshi-redhat commented Sep 21, 2022

zshi-redhat Sep 21, 2022

trozet commented Sep 23, 2022

zshi-redhat commented Sep 26, 2022

lmilleri commented Sep 26, 2022

lmilleri commented Nov 4, 2022

lmilleri commented Nov 8, 2022

lmilleri commented Nov 17, 2022

zshi-redhat commented Nov 30, 2022

ovn-robot commented Nov 30, 2022

lmilleri commented Dec 2, 2022

amorenoz commented Dec 13, 2022

dcbw commented Jan 4, 2023

lmilleri commented Jan 9, 2023

lmilleri commented Jan 11, 2023

Add support to virtio-vDPA devices #2664

Add support to virtio-vDPA devices #2664

Conversation

amorenoz commented Nov 18, 2021

amorenoz commented Nov 18, 2021

adrianchiris commented Nov 18, 2021

amorenoz commented Nov 18, 2021

coveralls commented Nov 27, 2021 • edited Loading

amorenoz commented Dec 14, 2021

amorenoz commented Jan 10, 2022

amorenoz commented Apr 7, 2022

dcbw commented Jul 12, 2022

lmilleri commented Sep 20, 2022

zshi-redhat commented Sep 21, 2022

ovn-robot commented Sep 21, 2022

zshi-redhat commented Sep 21, 2022

zshi-redhat Sep 21, 2022

Choose a reason for hiding this comment

trozet commented Sep 23, 2022

zshi-redhat commented Sep 26, 2022

lmilleri commented Sep 26, 2022

lmilleri commented Nov 4, 2022

lmilleri commented Nov 8, 2022

lmilleri commented Nov 17, 2022

zshi-redhat commented Nov 30, 2022

ovn-robot commented Nov 30, 2022

lmilleri commented Dec 2, 2022

amorenoz commented Dec 13, 2022

dcbw commented Jan 4, 2023

lmilleri commented Jan 9, 2023

lmilleri commented Jan 11, 2023

coveralls commented Nov 27, 2021 •

edited

Loading