Preserve OVS data path flows when Antrea is restarted gracefully #355
Assignees
Labels
area/component/agent
Issues or PRs related to the agent component
area/dependency
Issues or PRs related to dependency changes.
area/ovs/openflow
Issues or PRs related to Open vSwitch Open Flow.
area/test/e2e
Issues or PRs related to Antrea specific end-to-end testing.
kind/design
Categorizes issue or PR as related to design.
priority/important-soon
Must be staffed and worked on either currently, or very soon, ideally in time for the next release.
proposal
A concrete proposal for adding a feature
Milestone
Comments
antoninbas
added a commit
to antoninbas/antrea
that referenced
this issue
Feb 4, 2020
Without this patch, when the antrea-ovs container exits gracefully (e.g. as part of a RollingUpdate for the DaemonSet), flows are deleted, which breaks connectivity (potentially for > 10s). A recent OVS patch changed the default behavior of ovs-vswitchd so that data path flows are no longer deleted on exit. Because this patch is not yet available in any released OVS version (and won't be for > 6 months), we apply it manually when we build our base OVS Docker image. With this support, it seems that the Antrea agent logic which is in charge of replaying the flows after each reconnection from the OpenFlow client to OVS, is no longer needed. While it doesn't hurt to keep it, I cannot think of a scenario where it is useful. I suggest we open a separate issue to keep track of removing that code. Fixes antrea-io#355
antoninbas
added a commit
to antoninbas/antrea
that referenced
this issue
Feb 5, 2020
Without this patch, when the antrea-ovs container exits gracefully (e.g. as part of a RollingUpdate for the DaemonSet), datapath flows are deleted, which breaks existing connections. It takes 10 seconds for exiting connections to start working again. A recent OVS patch changed the default behavior of ovs-vswitchd so that datapath flows are no longer deleted on exit. Because this patch is not yet available in any released OVS version (and won't be for > 6 months), we apply it manually when we build our base OVS Docker image. Fixes antrea-io#355
|
To clarify: the changes in #358 only apply to kernel datapath flows. New connections that don't match any flow in the datapath will not be established until ovs-vswitchd has restarted and Antrea has replayed the flows. If kube-proxy selects an endpoint on a Node that's affected for a ClusterIP, the connection won't be established until then (or until there is a TCP SYN timeout and the app tries opening a new connection). |
antoninbas
added a commit
to antoninbas/antrea
that referenced
this issue
Feb 6, 2020
Without this patch, when the antrea-ovs container exits gracefully (e.g. as part of a RollingUpdate for the DaemonSet), datapath flows are deleted, which breaks existing connections. It takes 10 seconds for exiting connections to start working again. A recent OVS patch changed the default behavior of ovs-vswitchd so that datapath flows are no longer deleted on exit. Because this patch is not yet available in any released OVS version (and won't be for > 6 months), we apply it manually when we build our base OVS Docker image. Fixes antrea-io#355 Disable new e2e test for Kind When using the netdev datapath, packet forwarding is done by the ovs-vswitchd daemon itself, so even existing connections are broken.
antoninbas
added a commit
to antoninbas/antrea
that referenced
this issue
Feb 7, 2020
Without this patch, when the antrea-ovs container exits gracefully (e.g. as part of a RollingUpdate for the DaemonSet), datapath flows are deleted, which breaks existing connections. It takes 10 seconds for exiting connections to start working again. A recent OVS patch changed the default behavior of ovs-vswitchd so that datapath flows are no longer deleted on exit. Because this patch is not yet available in any released OVS version (and won't be for > 6 months), we apply it manually when we build our base OVS Docker image. Fixes antrea-io#355 Disable new e2e test for Kind When using the netdev datapath, packet forwarding is done by the ovs-vswitchd daemon itself, so even existing connections are broken.
McCodeman
added
area/component/agent
antoninbas
added a commit
to antoninbas/antrea
that referenced
this issue
Feb 18, 2020
Without this patch, when the antrea-ovs container exits gracefully (e.g. as part of a RollingUpdate for the DaemonSet), datapath flows are deleted, which breaks existing connections. It takes 10 seconds for exiting connections to start working again. A recent OVS patch changed the default behavior of ovs-vswitchd so that datapath flows are no longer deleted on exit. Because this patch is not yet available in any released OVS version (and won't be for > 6 months), we apply it manually when we build our base OVS Docker image. Fixes antrea-io#355 Disable new e2e test for Kind When using the netdev datapath, packet forwarding is done by the ovs-vswitchd daemon itself, so even existing connections are broken.
antoninbas
added a commit
to antoninbas/antrea
that referenced
this issue
Feb 18, 2020
Without this patch, when the antrea-ovs container exits gracefully (e.g. as part of a RollingUpdate for the DaemonSet), datapath flows are deleted, which breaks existing connections. It takes 10 seconds for exiting connections to start working again. A recent OVS patch changed the default behavior of ovs-vswitchd so that datapath flows are no longer deleted on exit. Because this patch is not yet available in any released OVS version (and won't be for > 6 months), we apply it manually when we build our base OVS Docker image. Fixes antrea-io#355 Disable new e2e test for Kind When using the netdev datapath, packet forwarding is done by the ovs-vswitchd daemon itself, so even existing connections are broken.
antoninbas
added a commit
to antoninbas/antrea
that referenced
this issue
Feb 19, 2020
Without this patch, when the antrea-ovs container exits gracefully (e.g. as part of a RollingUpdate for the DaemonSet), datapath flows are deleted, which breaks existing connections. It takes 10 seconds for exiting connections to start working again. A recent OVS patch changed the default behavior of ovs-vswitchd so that datapath flows are no longer deleted on exit. Because this patch is not yet available in any released OVS version (and won't be for > 6 months), we apply it manually when we build our base OVS Docker image. This is a step towards non-disruptive upgrade. However, there are other factors in play: 1) when the OVS daemons are stopped, the VXLAN device (vxlan_sys_4789) goes away, which impacts connectivity between Nodes, and 2) when ovs-vswitchd restarts, datapath flows are flushed and we have to wait until the Antrea Agent has replayed the flows. * Disable new e2e test for Kind When using the netdev datapath, packet forwarding is done by the ovs-vswitchd daemon itself, so even existing connections are broken. * Use arping instead of ping for test When using ping the test can be flaky: if one of the hosts need to send an ARP request to refresh its ARP entry while the test is ongoing, odds are that there is no cached flow in the datapath. If this happens, the host will mark the ARP entry has "incomplete" and the ping will start failing. Fixes antrea-io#355
antoninbas
added a commit
that referenced
this issue
Feb 20, 2020
Without this patch, when the antrea-ovs container exits gracefully (e.g. as part of a RollingUpdate for the DaemonSet), datapath flows are deleted, which breaks existing connections. It takes 10 seconds for exiting connections to start working again. A recent OVS patch changed the default behavior of ovs-vswitchd so that datapath flows are no longer deleted on exit. Because this patch is not yet available in any released OVS version (and won't be for > 6 months), we apply it manually when we build our base OVS Docker image. This is a step towards non-disruptive upgrade. However, there are other factors in play: 1) when the OVS daemons are stopped, the VXLAN device (vxlan_sys_4789) goes away, which impacts connectivity between Nodes, and 2) when ovs-vswitchd restarts, datapath flows are flushed and we have to wait until the Antrea Agent has replayed the flows. * Disable new e2e test for Kind When using the netdev datapath, packet forwarding is done by the ovs-vswitchd daemon itself, so even existing connections are broken. * Use arping instead of ping for test When using ping the test can be flaky: if one of the hosts need to send an ARP request to refresh its ARP entry while the test is ongoing, odds are that there is no cached flow in the datapath. If this happens, the host will mark the ARP entry has "incomplete" and the ping will start failing. Fixes #355
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Describe what you are trying to solve
At the moment, data path flows are deleted by ovs-vswitchd when the daemon gracefully exits. This means that for planned rolling upgrades of Antrea, we are currently bringing the network down. A simple experiment reveals that it can take more than 10s for the network to come back up:
That time may go up as the number of flows increase, and it may take even longer for network policies to be enforced again.
Describe the solution you have in mind
A patch was recently merged into OVS master to change this behavior: openvswitch/ovs@79eadaf. If we include that patch, we should be able to support hitless upgrades for Antrea.
Unfortunately Antrea currently uses OVS 2.11.0, and the patch is not even included in the latest OVS release (2.13.0). Until we can upgrade to a release > 2.13, we should consider applying the patch manually when we build the
antrea/openvswitchDocker image.Describe the main design/architecture of your solution
Alternative solutions that you considered
Have 2 different DaemonSets, one for antrea-agent and one for antrea-ovs, so that antrea-agent can be upgraded without disrupting the OVS daemons. It seems undesirable to have 2 DaemonSets for Antrea though, and we may also want to upgrade the Docker image for antrea-ovs from time to time (security vulnerabilities).
Test plan
I will add an e2e test which will send UDP packets using iperf between 2 Pods (or send ICMP packets with a small interval?). The test will verify that no packets have been lost after restarting Antrea.
The text was updated successfully, but these errors were encountered: