Add documentation for Antrea's OVS pipeline #206

antoninbas · 2019-12-06T22:21:10Z

Some detailed documentation for the OVS pipeline, including a
description of each table. This is directed at developers and people
trying to troubleshoot issues.

It includes a SVG high-level diagram of the pipeline. We use SVG
directly so it renders better on all screens and to avoid having to
check-in a "large" PNG image that may need to be updated often.

More documentation specific to the Network Policy implementation will
follow later.

Fixes #27

antrea-bot · 2019-12-06T22:21:12Z

Thanks for your PR.
Unit tests and code linters are run automatically every time the PR is updated.
E2e tests can only be triggered by a member of the vmware-tanzu organization. Regular contributors to the project should join the org.

The following commands are available:

/test-e2e: to trigger e2e tests. This command can only be run by members of the vmware-tanzu organization
/skip-e2e: to skip e2e tests. This command can only be run by members of the vmware-tanzu organization

jianjuns · 2019-12-07T06:15:36Z

docs/ovs-pipeline.md

+
+All traffic is finally resubmitted to the [DnatTable].
+
+### DnatTable (40)


Probably: "DNATTable"

abhiraut

nice!

abhiraut · 2019-12-10T01:48:01Z

docs/ovs-pipeline.md

+* *table miss-miss flow entry*: a "catch-all" entry in a OpenFlow table, which
+  is used if no other flow is a match. If the table-miss flow entry does not
+  exist, by default packets unmatched by flow entries are dropped (discarded).
+* *conjuctive match fields*: an efficient way in OVS to implement conjunctive


conjuctive -> conjunctive

abhiraut · 2019-12-10T02:08:11Z

docs/ovs-pipeline.md

+```
+
+After this table, ARP traffic is resubmitted to [ARPResponderTable], while IP
+traffic is resubmitted to [ConnectionTrackingTable]. Traffic which does not


ConnectionTrackingTable -> ConntrackTable

tnqn · 2019-12-10T11:28:20Z

docs/ovs-pipeline.md

+  action, then the packets in the flow go through the switch in the same way
+  that they would if OpenFlow was not configured on the switch. Antrea uses this
+  action to process ARP traffic as a regular learning L2 switch would.
+* *table miss-miss flow entry*: a "catch-all" entry in a OpenFlow table, which


Do you mean table-miss flow entry?

tnqn · 2019-12-10T12:12:02Z

docs/ovs-pipeline.md

+This table complements [EgressRuleTable] for Network Policy egress rule
+implementation. In K8s, when a Network Policy is applied to a set of Pods, the
+default behavior for these Pods become "deny" (it becomes an [isolated
+Pod](https://kubernetes.io/docs/concepts/services-networking/network-policies/#isolated-and-non-isolated-pods). This


Suggested change

Pod](https://kubernetes.io/docs/concepts/services-networking/network-policies/#isolated-and-non-isolated-pods). This

Pod](https://kubernetes.io/docs/concepts/services-networking/network-policies/#isolated-and-non-isolated-pods)). This

Otherwise it appears as ... "deny" (it becomes an isolated Pod. This ...

tnqn · 2019-12-10T12:18:44Z

docs/ovs-pipeline.md

+can go through.
+
+The rest of the flows read as follows: if the source IP address is in set
+{10.10.1.2, 10.10.1.3}, and the destination port is in the set {3, 4} (which


maybe "the destination OF port" to avoid confusion.

abhiraut · 2019-12-10T22:35:54Z

docs/ovs-pipeline.md

+
+In the future this table may support an additional mode of operations, in which
+it will implement kube-proxy functionality and take care of performing
+laod-balancing / DNAT on traffic destined to services.


laod -> load

abhiraut · 2019-12-10T22:49:47Z

docs/ovs-pipeline.md

+
+If the `conjunction` action is matched, packets are "allowed" and resubmitted
+directly to [L3ForwardingTable]. Other packets go to [EgressDefaultTable]. If a
+connection is established - as a remainder all connections are committed in


abhiraut · 2019-12-10T22:55:50Z

docs/ovs-pipeline.md

+```
+1. table=60, priority=200,ip,nw_src=10.10.1.2 actions=drop
+2. table=60, priority=200,ip,nw_src=10.10.1.3 actions=drop
+3. table=60, priority=80,ip actions=resubmit(,70)


for the sake of completeness, as you have done throughout this document, also include one line about this resubmit rule..

abhiraut · 2019-12-10T23:36:38Z

docs/ovs-pipeline.md

+can go through.
+
+The rest of the flows read as follows: if the source IP address is in set
+{10.10.1.2, 10.10.1.3}, and the destination port is in the set {3, 4} (which


abhiraut · 2019-12-10T23:36:54Z

docs/ovs-pipeline.md

+### IngressDefaultTable (100)
+
+This table is similar in its purpose to [EgressDefaultTable], and it complements
+[EgressRuleTable] for Network Policy egress rule implementation. In K8s, when a


this should be [IngressRuleTable]

abhiraut · 2019-12-10T23:37:51Z

docs/ovs-pipeline.md

+### IngressDefaultTable (100)
+
+This table is similar in its purpose to [EgressDefaultTable], and it complements
+[EgressRuleTable] for Network Policy egress rule implementation. In K8s, when a


egress rule impl.. -> ingress rule impl..

abhiraut · 2019-12-10T23:38:26Z

docs/ovs-pipeline.md

+[EgressRuleTable] for Network Policy egress rule implementation. In K8s, when a
+Network Policy is applied to a set of Pods, the default behavior for these Pods
+become "deny" (it becomes an [isolated
+Pod](https://kubernetes.io/docs/concepts/services-networking/network-policies/#isolated-and-non-isolated-pods). This


non-isolated-pods). -> non-isolated-pods)).

abhiraut · 2019-12-10T23:39:37Z

docs/ovs-pipeline.md

+Network Policy is applied to a set of Pods, the default behavior for these Pods
+become "deny" (it becomes an [isolated
+Pod](https://kubernetes.io/docs/concepts/services-networking/network-policies/#isolated-and-non-isolated-pods). This
+table is in charge of dropping traffic originating from Pods to which a Network


this should be traffic destined to .. since it is ingress rule

abhiraut · 2019-12-10T23:43:39Z

docs/ovs-pipeline.md

+
+## Tables
+
+![OVS pipeline](/docs/assets/ovs-pipeline.svg)


in the diagram i noticed an arrow from table70 -> table60.. i believe it should be the other way around

abhiraut · 2019-12-11T20:01:00Z

docs/ovs-pipeline.md

@@ -0,0 +1,561 @@
+# Antrea OVS Pipeline
+
+## Terminology


maybe somewhere in this doc add the ovs-ofctl command used to dump the flows?

antoninbas · 2019-12-12T12:38:06Z

docs/ovs-pipeline.md

+This table handles all "tracked" packets (all packets are moved to the tracked
+state by the previous table, [ConntrackTable]). It serves the following
+purposes:
+ * keeps track of connections going through the gateway port; for all packets


I think I got this wrong: this mechanism applies to reverse traffic from a tunnel as well, not just from local backend Pods. @wenyingd could you confirm?

ConntrackStateTable also commits packets from the tunnel port into the ct_zone. It works by flow "table=31, priority=190,ct_state=+new+trk,ip actions=ct(commit,table=40,zone=65520)"

antoninbas · 2019-12-12T12:38:52Z

docs/ovs-pipeline.md

+document](http://docs.openvswitch.org/en/latest/tutorials/ovs-conntrack/) for
+more information on connection tracking in OVS.
+
+### ConntrackStateTable (31)


Needs to be updated because of #213

docs/ovs-pipeline.md

wenyingd · 2019-12-13T11:11:23Z

docs/ovs-pipeline.md

+  action, then the packets in the flow go through the switch in the same way
+  that they would if OpenFlow was not configured on the switch. Antrea uses this
+  action to process ARP traffic as a regular learning L2 switch would.
+* *table miss-miss flow entry*: a "catch-all" entry in a OpenFlow table, which


wenyingd · 2019-12-13T11:12:19Z

docs/ovs-pipeline.md

+  that they would if OpenFlow was not configured on the switch. Antrea uses this
+  action to process ARP traffic as a regular learning L2 switch would.
+* *table miss-miss flow entry*: a "catch-all" entry in a OpenFlow table, which
+  is used if no other flow is a match. If the table-miss flow entry does not


"is used if no other flow is matched"?

wenyingd · 2019-12-13T11:19:00Z

docs/ovs-pipeline.md

+   is "known", i.e. corresponds to an entry in [L2ForwardingCalcTable], which is
+   essentially a "dmac" table.
+ * reg1 (NXM_NX_REG1): it is used to store the egress OF port for the packet and
+ is set by [L2ForwardingCalcTable].


reg1 is also set in L3ForwardingTable for the packet that targets at Pods on peer Node (PR: #209), and in DNATTable for the packet targets at a Service IP(PR: #220)

wenyingd · 2019-12-13T11:37:21Z

docs/ovs-pipeline.md

+2. table=0, priority=200,in_port=tun0 actions=load:0->NXM_NX_REG0[0..15],resubmit(,30)
+3. table=0, priority=190,in_port="coredns5-8ec607" actions=load:0x2->NXM_NX_REG0[0..15],resubmit(,10)
+4. table=0, priority=190,in_port="coredns5-9d9530" actions=load:0x2->NXM_NX_REG0[0..15],resubmit(,10)
+5. table=0, priority=80,ip actions=resubmit(,10)


This flow(no: 5) might be removed after PR: #199 is checked in, and the action of the table-miss should be drop.

wenyingd · 2019-12-13T12:03:37Z

docs/ovs-pipeline.md

+If you dump the flows for this table, you should see something like this:
+```
+1. table=40, priority=200,ip,nw_dst=10.96.0.0/12 actions=output:gw0
+2. table=40, priority=80,ip actions=resubmit(,50)


priority=80 will be replaced by priority=0 in all tables once PR: #195 is merged.

wenyingd · 2019-12-13T12:05:31Z

docs/ovs-pipeline.md

+If the `conjunction` action is matched, packets are "allowed" and resubmitted
+directly to [L3ForwardingTable]. Other packets go to [EgressDefaultTable]. If a
+connection is established - as a remainder all connections are committed in
+[ConntrackStateTable] - its packets go straight to [L3ForwardingTable], with no


Do you mean "Established" connections?

antoninbas · 2019-12-18T03:06:48Z

It seems that all the patches to the OVS pipeline have been merged, so I'll update my PR to reflect the latest changes.

antrea-bot · 2019-12-18T22:07:56Z

Thanks for your PR.
Unit tests and code linters are run automatically every time the PR is updated.
E2e, conformance and network policy tests can only be triggered by a member of the vmware-tanzu organization. Regular contributors to the project should join the org.

The following commands are available:

/test-e2e: to trigger e2e tests.
/skip-e2e: to skip e2e tests.
/test-conformance: to trigger conformance tests.
/skip-conformance: to skip conformance tests.
/test-networkpolicy: to trigger networkpolicy tests.
/skip-networkpolicy: to skip networkpolicy tests.
/test-all: to trigger all tests.
/skip-all: to skip all tests.

These commands can only be run by members of the vmware-tanzu organization.

antoninbas · 2019-12-19T00:40:07Z

I addressed review comments and updated the doc to reflect the latest OVS pipeline. PTAL.

antoninbas · 2020-01-07T21:25:10Z

Any chance we can review / merge this?
Any change to the OVS pipeline recently?

jianjuns · 2020-01-07T21:39:16Z

Any chance we can review / merge this?
Any change to the OVS pipeline recently?

Sorry, I did not review yet. IPSec introduced some changes: the IPSec tunnel itself which we could document separately, and the change to load tunnel ofport in the L3 table and skip L2 and ingress policy tables. Check the commit description: 0d2e4d9

@wenyingd could comment on any other changes.

antoninbas · 2020-01-07T21:50:57Z

@jianjuns For IPSec support, I am also leaning towards a separate document or a future PR. I will update this PR to indicate that some flows are different when IPSec is enabled and to take into account the new table bypass for tunnelled traffic.

jianjuns · 2020-01-07T22:19:47Z

@antoninbas: what I mean is the IPSec PR also changes the L3 flows for the normal tunnels. You might want to include that part into your doc.

antrea-bot · 2020-01-07T22:36:08Z

Thanks for your PR.
Unit tests and code linters are run automatically every time the PR is updated.
E2e, conformance and network policy tests can only be triggered by a member of the vmware-tanzu organization. Regular contributors to the project should join the org.

The following commands are available:

/test-e2e: to trigger e2e tests.
/skip-e2e: to skip e2e tests.
/test-conformance: to trigger conformance tests.
/skip-conformance: to skip conformance tests.
/test-networkpolicy: to trigger networkpolicy tests.
/skip-networkpolicy: to skip networkpolicy tests.
/test-all: to trigger all tests.
/skip-all: to skip all tests.

These commands can only be run by members of the vmware-tanzu organization.

antoninbas · 2020-01-07T22:37:20Z

@jianjuns yes that's what I meant. I have updated the document to account for the changes for normal tunnels.

docs/ovs-pipeline.md

antrea-bot · 2020-01-08T19:40:17Z

Thanks for your PR.
Unit tests and code linters are run automatically every time the PR is updated.
E2e, conformance and network policy tests can only be triggered by a member of the vmware-tanzu organization. Regular contributors to the project should join the org.

The following commands are available:

/test-e2e: to trigger e2e tests.
/skip-e2e: to skip e2e tests.
/test-conformance: to trigger conformance tests.
/skip-conformance: to skip conformance tests.
/test-networkpolicy: to trigger networkpolicy tests.
/skip-networkpolicy: to skip networkpolicy tests.
/test-all: to trigger all tests.
/skip-all: to skip all tests.

These commands can only be run by members of the vmware-tanzu organization.

antoninbas · 2020-01-08T19:45:46Z

Thanks for the review @wenyingd

wenyingd

Overall LGTM, expect for one comment.

wenyingd · 2020-01-09T03:51:34Z

docs/ovs-pipeline.md

+
+As for [EgressRuleTable], flow 1 (highest priority) ensures that for established
+connections - as a remainder all connections are committed in
+[ConntrackStateTable] - packets go straight to [L2ForwardingOutTable], with no


all connections are committed in [ConntrackCommitTable]?

Done. Thanks for catching this.

Some detailed documentation for the OVS pipeline, including a description of each table. This is directed at developers and people trying to troubleshoot issues. It includes a SVG high-level diagram of the pipeline. We use SVG directly so it renders better on all screens and to avoid having to check-in a "large" PNG image that may need to be updated often. More documentation specific to the Network Policy implementation will follow later. Fixes antrea-io#27

wenyingd

LGTM.

antoninbas · 2020-01-10T02:05:32Z

/skip-all

vmwclabot added the cla-not-required label Dec 6, 2019

antoninbas requested review from abhiraut, jianjuns, salv-orlando, tnqn and wenyingd December 6, 2019 22:21

jianjuns reviewed Dec 7, 2019

View reviewed changes

docs/ovs-pipeline.md Outdated

All traffic is finally resubmitted to the [DnatTable].

### DnatTable (40)

Copy link

Contributor

jianjuns Dec 7, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Probably: "DNATTable"

abhiraut reviewed Dec 10, 2019

View reviewed changes

tnqn reviewed Dec 10, 2019

View reviewed changes

antoninbas mentioned this pull request Dec 10, 2019

Rewrite dst MAC with host gateway's MAC #213

Merged

abhiraut reviewed Dec 10, 2019

View reviewed changes

abhiraut reviewed Dec 11, 2019

View reviewed changes

antoninbas commented Dec 12, 2019

View reviewed changes

wenyingd reviewed Dec 13, 2019

View reviewed changes

antoninbas mentioned this pull request Dec 17, 2019

Considering support annotation for open flows #243

Closed

antoninbas force-pushed the add-documentation-for-OVS-pipeline branch from 5a1174b to 3a34ae6 Compare December 18, 2019 22:07

antoninbas force-pushed the add-documentation-for-OVS-pipeline branch from 3a34ae6 to 46db62e Compare December 19, 2019 00:37

antoninbas mentioned this pull request Dec 19, 2019

Document OVS Datapath Flows #27

Closed

antoninbas force-pushed the add-documentation-for-OVS-pipeline branch from 46db62e to bba519a Compare January 7, 2020 22:35

wenyingd reviewed Jan 8, 2020

View reviewed changes

docs/ovs-pipeline.md Outdated Show resolved Hide resolved

docs/ovs-pipeline.md Outdated Show resolved Hide resolved

docs/ovs-pipeline.md Show resolved Hide resolved

docs/ovs-pipeline.md Outdated Show resolved Hide resolved

docs/ovs-pipeline.md Show resolved Hide resolved

antoninbas force-pushed the add-documentation-for-OVS-pipeline branch from bba519a to 1eb0978 Compare January 8, 2020 19:40

wenyingd reviewed Jan 9, 2020

View reviewed changes

antoninbas force-pushed the add-documentation-for-OVS-pipeline branch from 1eb0978 to b0484c4 Compare January 9, 2020 18:33

wenyingd approved these changes Jan 10, 2020

View reviewed changes

antoninbas merged commit bff28db into antrea-io:master Jan 10, 2020

antoninbas deleted the add-documentation-for-OVS-pipeline branch January 10, 2020 02:06

zyiou added a commit to zyiou/antrea that referenced this pull request Jul 2, 2021

remove element map from record (antrea-io#206)

0ca84f4


		All traffic is finally resubmitted to the [DnatTable].

		### DnatTable (40)

	Pod](https://kubernetes.io/docs/concepts/services-networking/network-policies/#isolated-and-non-isolated-pods). This
	Pod](https://kubernetes.io/docs/concepts/services-networking/network-policies/#isolated-and-non-isolated-pods)). This

Add documentation for Antrea's OVS pipeline #206

Add documentation for Antrea's OVS pipeline #206

Conversation

antoninbas commented Dec 6, 2019

antrea-bot commented Dec 6, 2019

Choose a reason for hiding this comment

abhiraut left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

antoninbas commented Dec 18, 2019

antrea-bot commented Dec 18, 2019

antoninbas commented Dec 19, 2019

antoninbas commented Jan 7, 2020

jianjuns commented Jan 7, 2020

antoninbas commented Jan 7, 2020

jianjuns commented Jan 7, 2020

antrea-bot commented Jan 7, 2020

antoninbas commented Jan 7, 2020

antrea-bot commented Jan 8, 2020

antoninbas commented Jan 8, 2020

wenyingd left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

wenyingd left a comment

Choose a reason for hiding this comment

antoninbas commented Jan 10, 2020

wenyingd left a comment •

edited

Loading