Does Antrea work as a secondary Multus CNI?

[xagent003]

This guide: https://antrea.io/docs/main/docs/cookbooks/multus/

Only talks about using Antrea as a primary CNI and macvlan as secondary CNI. But:

1. Can we use Antrea as the CNI for Multus NetworkAttachmentDefinitions? If so:
   a. Does Antrea also need to be primary CNI?
   b. Can we use existing CNI like Calico and use Antrea only for secondary Multus networks?
   c. What features will work and what won't?

[antoninbas]

We don't test for this use case and this is not a use case that we had in mind when building Antrea.

There is no way to make this work without modifying Antrea code. One may be able to get close by using Antrea in noEncap mode and enabling Antrea Flexible IPAM (https://github.com/antrea-io/antrea/blob/main/docs/antrea-ipam.md#antrea-flexible-ipam). @jianjuns what do you think?

IMO, requirements for secondary networks are typically different from requirements for the primary network. For example, there may be higher throughput requirements (e.g., requiring hardware acceleration), but simpler connectivity requirements, and no need for features like NetworkPolicy and Egress.

I think Antrea could provide value as a secondary network CNI provider, by also leveraging OVS for secondary networks. We have been taking steps in that direction (for example, one can use Antrea IPAM for secondary networks, as described in our Multus cookbook), but there is still a long way to go and it's not been a priority so far.

The following items are ranked from most likely to happen in the future to least likely to happen in the future (in my opinion):

1. secondary network support (limited feature set) with Antrea as the primary CNI (with or without Multus?)
2. secondary network support (limited feature set) without Antrea as the primary network
3. secondary network support with the same full feature set as provided by Antrea for primary networks.

It would be good to understand your use case & requirements:

• why are you interested in using Antrea for secondary networks, and why would you want to use a different primary CNI?
• for secondary networks, would you like overlay support, or do you want to rely on the underlay to route Pod traffic?
• besides connectivity (across Pods and potentially to external networks) which features would you care about for secondary networks (security, etc.)?

[github-actions]

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment, or this will be closed in 90 days

[devopstales]

Any update on this? It would be nice to use network separation with multiple virtual networks in kubernetes not just with networ policies/firewall rules.

[antoninbas]

We have some progress for secondary network support in the upcoming Antrea v1.14 with support for #5278. Antrea can now attach a secondary pNIC to a new OVS bridge, dedicated to Pod secondary networks. However, the only use case that is supported is connecting Pods to VLANs in the physical network. There is no support for creating overlays.

And we do not support this scenario: arbitrary primary CNI + Multus + Antrea as secondary CNI. The current support is for Antrea as the only CNI (for primary and secondary networks), with no Multus.

[devopstales]

Thank you @antoninbas for the update. This is a big step forward, but for my use case the missing part is the support for multiple overlay networks. Is there any plan for that? Thank you for the hard work on this.

[antoninbas]

I think we should be able to capitalize on the work we have done to support overlays.
cc @jianjuns

I can bring it up at the next Antrea community meeting

[sspreitzer]

We are looking for this functionality. Multus CNI, Default: OVN, Secondary: Antrea
We would like to support hybrid cloud flexibility by flexible secondary interface CNIs.

[antoninbas]

We discussed this briefly at our community meeting yesterday.

I think this issue really should be split into 2 parts:

1. Support for overlay secondary networks (in addition to VLAN networks) when using Antrea as the only CNI (for both primary and secondary networks), without Multus. This is not too complicated IMO, although we may want to wait for the current SecondaryNetwork feature to be a bit more stable before we add this functionality. We may want to open a separate issue to track this. cc @jianjuns
2. Support for using Antrea to provision secondary networks when using Multus and possibly a different CNI as the "primary". I believe this is what @xagent003 is asking for with this issue. This is not a use case we had in mind for Antrea TBH, and so a lot of changes may be required. We are still waiting for compelling user stories for this: what limitations do other secondary CNIs have? how can Antrea help? which features (networking / security) do users expect for the secondary networks?

[antoninbas]

@sspreitzer

We are looking for this functionality. Multus CNI, Default: OVN, Secondary: Antrea

I assume you mean that you are using Openshift with ovn-kubernetes as the primary / default CNI?

We would like to support hybrid cloud flexibility by flexible secondary interface CNIs.

Are you looking at overlay networks? Any requirement besides basic isolation and Pod-to-Pod connectivity? It would be good to have more information about your use case.

[github-actions]

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment, or this will be closed in 90 days

[sspreitzer]

@antoninbas

We are looking for this functionality. Multus CNI, Default: OVN, Secondary: Antrea

I assume you mean that you are using Openshift with ovn-kubernetes as the primary / default CNI?

Exactly. We are using the OpenShift Container Platform with version 4.14, default CNI with Multus, primary CNI is ovn-kubernetes, secondary aspiring Antrea and/or others.

We would like to support hybrid cloud flexibility by flexible secondary interface CNIs.

Are you looking at overlay networks? Any requirement besides basic isolation and Pod-to-Pod connectivity? It would be good to have more information about your use case.

The use case is to implement a hybrid compute capable OpenShift integration. One cluster should be capable of running on bare metal, vSphere and cloud instances, while being able to take advantage of the vSphere capabilities through the Antrea plugin or similar with other CNIs.

In general imho any CNI solution should not make assumptions on primary or secondary use and offer an agnostic implementation.

[antoninbas]

In general imho any CNI solution should not make assumptions on primary or secondary use and offer an agnostic implementation.

It depends on what you mean exactly by CNI here. With the current K8s networking model, I would say that these 2 use cases are significantly different by nature. K8s only cares about the "primary" network (i.e., it's the only network "defined" by K8s natively). Constructs like Services and NetworkPolicies are specific to that network. There is ongoing discussions to support "multi-network" constructs in K8s, but AFAIK this is all still pretty early-stage.

Now if we are only looking at the CNI interface specifically, yes we could argue that its implementation in Antrea should support both use cases. Currently we do not use CNI ADD / DEL to provision secondary network interfaces in Antrea, instead we use a controller-based approach. The advantage of this approach is that we can support "hot-pluggable" network interfaces, rather than tying secondary network interfaces to the Pod's lifecyle.

I do not see "uniform" primary and secondary networks any time soon, whether at the API-level (that's dependent on changes in K8s IMO) or at the implementation-level (datapath implementation, feature availability). To be honest, I don't even have a clear mental model of what it would look like. Will we be able to tie a Service to a specific network? Should all Antrea features, such as Egress, be available for all networks? Who should take care of cross-network communications?

[sspreitzer]

@antoninbas

Thank you for clarification. Allow me to paraphrase the situation:

• Kubernetes does not provide an interface to handle multiple CNIs
• Multus CNI provides an interface to handle multiple CNIs
• Antrea CNI cannot be used with Multus CNI
  • Antrea CNI by design assumes it is the only CNI in a Kubernetes cluster, similar to Multus CNI, which results in mechanisms that are incompatible with Multus CNI

Causing the following strategic impact:

• Antrea CNI locks Kubernetes clusters into VMware virtualisation
• Antrea CNI cannot be used with Multus CNI

[antoninbas]

@sspreitzer Most of this is inaccurate

Antrea CNI by design assumes it is the only CNI in a Kubernetes cluster, similar to Multus CNI, which results in mechanisms that are incompatible with Multus CNI

No, Antrea just assumes it's the primary network CNI. It can still work with Multus as the meta-CNI as long as Antrea provisions the primary network interfaces ("eth0"): https://github.com/antrea-io/antrea/tree/main/docs/cookbooks/multus. You can use any secondary network CNI alongside it (e.g., macvlan)

Antrea CNI locks Kubernetes clusters into VMware virtualisation

I don't see how it's remotely true. Antrea works on any infrastructure: bare-metal, vSphere, public cloud. I routinely run Antrea on VirtualBox for development.

Antrea CNI cannot be used with Multus CNI

As described above, Antrea can be used with Multus.

It feels like the points I was trying to make were lost. The fact that we are not planning to have support for "standalone Antrea as a secondary network provider" has nothing to do with what infrastructure we support. I still have to see a compelling case for secondary networks provisioned by Antrea with the primary network managed by another CNI. And I'll point out that Antrea can be used as the primary CNI on OpenShift.

[github-actions]

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment, or this will be closed in 90 days