support for noEncap mode

[suwang48404]

Describe what you are trying to solve
Antrea to support other encapsulation mode for inter-worker-node pod traffic.

Describe the solution you have in mind
Antrea to allow users to specify additional encapsulation mode.

Describe how your solution impacts user flows
An user would need to modify antrea-cni.conf in Antrea deployment manifest and/or additional knobs (see later) that updates Antrea-agent.conf current content.

Describe the main design/architecture of your solution
The following knobs shall be added to Antrea-agent.conf.
PodTrafficEncapMode : determines how inter-node pod traffic's encapsulation mode. It has the following options
:Encap -- pod's service (request/reply of services provided by pods) traffic always encapsulated, encapsulation protocol is provided by TunnelType knob. Pod's own request traffic to other destinations are masqueraded. This will be the default mode, and is already supported
:NoEncapMasq-- Pod's traffic is not encapsulated nor masqueraded. Pod's IP is visible to worker node's network always. This mode shall works best in cloud or on-prem where pod's IP are known to worker node's network. This mode may also work on-prem where with out-of-band configure, worker nodes's network is made aware of POD CIDRs
:NoEncap -- Pod's service traffic is not encapsulated , however pod's request traffic are masqueraded. This mode works in on-prem case where all worker nodes are on the same l2 subnet.
:Auto -- In this mode, depends on worker node's subnet, Pod's service traffic is in Encap mode if worker nodes are on the same subnet, and in NoEncap mode if worker nodes are on different subnet. Pod's request traffic are masqueraded.

please find detailed proposal at https://docs.google.com/document/d/1y0P-BZ63-JqALZjPPONXNJdIlnyWJhOGHjd4OLA0hDA/edit#

Alternative solutions that you considered
A list of the alternative solutions that you considered, and why they fell short. You can list the pros and cons of each solution.

Test plan
Describe what kind of tests you plan on adding to exercise your changes.

Additional context
Any other relevant information.

[jianjuns]

Thanks for making the proposal.

I have thought about no-encap mode too, and a little different from what you described I am thinking to use a single config option like networkingMode to indicate one of multiple different modes including noEncap mode. So, we could have:

1. encap - always encap with the specified protocol (VXLAN or Geneve or GRE or STT).
2. ipsecEncap - always encap and enable IPSec ESP.
3. hybrid - when the remote Node is in a different (underlay network) subnet from the source Node, encap the traffic; otherwise just do routing (add a static route of remote Pod subnet -> remote Node IP on the source Node).
4. noEncap - never encap. If the remote Node is the same subnet as the source Node, still add a route (remote Pod subnet -> remote Node IP) on the source Node; if the remote Node is in a different subnet, just follow the routes configured on the Node (here we assume underlay network/router knows about to forward the traffic. Maybe some other code configured underlay router already).
   I assume most likely "noEncap" should be for the case that all Nodes are in the same underlay subnet. But then why we still introduce "noEncap" besides "hybrid" - in "noEncap" mode we will not change the interface MTU, but for "hybrid" as we do not know in advance all Nodes must be in a single subnet or not, we have to decrease interface MTU.

Then let us talk about cloud native networking, another aspect I have different ideas with you. I would add a separate mode for native networking, like:
5. underlayNetwork
The reasons are:

1. we will not add static routes to remote Node but always let underlay network handle cross-Nodes routing and forwarding;
2. for Pod -> non-Pod CIDR traffic we will do SNAT (now with iptables) to Node IP for other modes, but we should not do that for underlayNetwork mode;
3. we will add the Node's physical interface to the uplink of the bridge (and move the IPs to the bridge if there are IPs on the physical interface) - but you could argue even for other modes we might support an option of adding the physical interface to OVS for better performance.
   Last, in some cases the underlay network might expect to tag the traffic in some way (e.g. VLAN tags for NSX-T). We could consider it to be a sub-option of the same mode or another mode.

I would think about IPAM separately. For how to do IPAM for cloud managed K8s, we need to discuss with public cloud teams (they might and might not like to make their CNI to be an IPAM plugin of Antrea).

Let me know what you think. Also like to learn others' opinions: @antoninbas @tnqn @salv-orlando.

[jianjuns]

The major difference between my 1-4, and 5: in 1-4 Pod IPs are in different network from Nodes and the underlay network might not recognize Pod IPs; but in 5 underlay network is aware of Pods IPs and Pods IP and Node IPs could be in the same network (and even subnet).

[jianjuns]

Thought over. We could also choose to remove 4 and document MTU should be explicitly set if all Nodes are in the same subnet. Just that it wont work for MTU auto-discovery that we might support later. But anyway I would prioritize 3 over 4.

[suwang48404]

let us have some discussion tomorrow?
thx, Su

[suwang48404]

@jianjuns incorporated your comments into the updated proposal. Hopefully it covered all our use cases.

[antoninbas]

Adding this tentatively to the v0.3.0 milestone, although we probably need some additional design work / scoping to decide what should go into the release.

[suwang48404]

@antoninbas what is timeline for v0.3.0 release?

[antoninbas]

@susanwu88 Jan 22nd. You can see that by clicking on the milestone name.

[antoninbas]

Moving this out of the 0.3.0 milestone, as discussed at the 01/15/2020 community meeting.

[antoninbas]

Moving this out of the 0.4.0 milestone, as discussed at the 01/29/2020 community meeting.

[suwang48404]

feature implemented in #340