What should flannel do when it loses a lease?

[rosenhouse]

Hello,

This question cuts across flanneld and the flannel CNI plugin. Apologies we're asking in the wrong place.

We (@cloudfoundry/cf-container-networking) are exploring some of the failure states of flannel and etcd. One simple scenario is when the flannel daemon loses a lease on a subnet. To test this, we delete the key from etcd.

In this case, flannel may acquire a lease on a different subnet. This can cause problems if the flannel CNI plugin has already created a bridge with the old subnet, and had attached containers to that bridge. In this scenario, existing containers with IPs on the old subnet are now disconnected from the rest of the network. And if a new container is started on the host, it is assigned an IP from the new subnet, which is incorrect for the existing bridge.

In this scenario, is there any way to recover?

Should the flannel CNI plugin create a new bridge, with the new IP? Should the existing bridge and connected containers be destroyed?

cc: @rusha19 @jaydunk

[tomdee]

Sorry @rosenhouse I don't have a good answer for this at the moment. I'm going to close this issue and just track making this better as part of #29