dual-stack support

[danwinship]

Tracking for what needs to be done for dual-stack support.

• Configuration (pkg/config/):

  • Allow configuring dual-stack service CIDRs (either via --secondary-service-cidr like upstream kube, or via --service-cidrs) (fix up config for dual-stack, add dual-stack service CIDRs #1189)
  • Validate that cluster and service CIDR combination is valid. (ie, either both same-single-stack or both dual-stack) (fix up config for dual-stack, add dual-stack service CIDRs #1189)
  • Revisit config.IPv6Mode. Should we add a parallel config.IPv4Mode? Or have config.IPMode with IPv4, IPv6, and DualStack options? Note that in many places where we currently check config.IPv6Mode, the more dual-stack friendly option is to just look at the IP address currently being operated on and act based on that. (fix up config for dual-stack, add dual-stack service CIDRs #1189)

• Node startup/configuration (pkg/node/):

  • k8s.ovn.org/l3-gateway-config annotation needs to allow dual-stack ip-address and next-hop (dual-stack support for l3-gateway-config annotation #1239)
  • pkg/node/management-port*.go:createPlatformManagementPort() needs to configure dual-stack IP/default route and service CIDR route (dual stack management port creation #1241)
  • pkg/util/kube.go:GetNodeIP() doesn't actually need to be dual-stack for the one place it's used (default ovn-encap-ip), but it should have a better name in that case (Misc dual-stack bits #1461)

• Node management (mostly pkg/ovn/):

  • Need dual stack SubnetAllocator support for host subnet/join subnet (Dual subnet allocation for dual-stack #1164)
  • Need to update annotations ("k8s.ovn.org/node-subnets", "k8s.ovn.org/node-join-subnets") to be dual-stack (Make host subnet/join subnet annotation dual-stack #1283)
  • pkg/ovn/ovn.go:WatchNodes() needs to handle IPv4 and IPv6 host subnets (Make host subnet/join subnet annotation dual-stack #1283)
  • pkg/ovn/master.go:addNode() needs to allocate and annotate both IPv4 and IPv6 host subnets (Make host subnet/join subnet annotation dual-stack #1283)
  • pkg/ovn/master.go:addNode() / ensureNodeLogicalNetwork() need to configure a single logical switch with information about both IPv4 and IPv6. (Make host subnet/join subnet annotation dual-stack #1283)
  • pkg/ovn/master.go:syncNodeManagementPort() needs to set up management port for both IPv4 and IPv6 (Make host subnet/join subnet annotation dual-stack #1283)
  • pkg/ovn/master.go:syncGatewayLogicalNetwork() (Make host subnet/join subnet annotation dual-stack #1283)
    • needs to create both IPv4 and IPv6 join subnets
    • needs to add both IPv4 and IPv6 static routes in shared gateway mode
  • pkg/util/gateway_init.go:GatewayInit() needs to take, and handle, multiple joinSubnetStr, nicIP, defaultGW, rampoutIPSubnet (dual-stack support for creating gateway in OVN #1256)
  • pkg/util/gateway_cleanup.go:GatewayCleanup() needs to staticRouteCleanup() both IPv4 and IPv6 routes (dual-stack support for creating gateway in OVN #1256)

• Pods

  • Update the pod annotation (pkg/util/util.go, etc) to indicate both IPv4 and IPv6 IPs. (update pod annotation and CNI shim communication for dual-stack #1129)
  • pkg/ovn/pods.go:getPodAddresses() / waitForPodAddresses() / pkg/util/net.go:GetPortAddresses() need to handle multiple IPs (dualstack support for GetPortAddresses #1325)
  • pkg/ovn/pods.go:getRoutesGatewayIP() needs to return multiple IPs and routes. (Redo it to take a util.PodAnnotation object and fill in the gateway/routes fields) (dualstack support for GetPortAddresses #1325)
  • pkg/ovn/pods.go:addLogicalPort()
    • updates for other pods.go and pod annotation changes
    • needs to pass both IPv4 and IPv6 addresses to lsp-set-addresses (dualstack support for GetPortAddresses #1325)
    • needs to pass both IPv4 and IPv6 addresses to lsp-set-port-security (dualstack support for GetPortAddresses #1325)
  • pkg/ovn/port_cache.go: needs to handle multiple pod IPs (ovn: add multiple pod IPs to port cache #1373)

• CNI (pkg/cni/):

  • Update the cniserver-to-cnishim communication to include both IPv4 and IPv6 IPs (update pod annotation and CNI shim communication for dual-stack #1129)
  • Update the cnishim to configure both IPs and return both IPs to kubelet (update pod annotation and CNI shim communication for dual-stack #1129)
  • ConfigureInterface must set both IPs as external_ids. (update pod annotation and CNI shim communication for dual-stack #1129)

• Services. (Note that for services, dual-stack just means allowing both single-stack-IPv4 Services and single-stack-IPv6 Services in the same cluster; no individual Service is itself dual-stack)

  • NodePort handling (eg createGatewaysVIP, handleNodePortLB) needs to create both IPv4 and IPv6 VIPs, and pick the correct one to use for each Service (initial dual-stack loadbalancer support #1234)
  • pkg/node/gateway_localnet.go:localnetAddService() / localnetDeleteService() need to add/delete IPv4 or IPv6 rules depending on the service IP (node: dual-stack gateway setup #1480)
  • pkg/node/gateway_localnet.go:localnetNodePortWatcher() needs to set up both iptables and ip6tables base rules (node: dual-stack gateway setup #1480)
  • pkg/util/gateway_init.go:GetDefaultGatewayRouterIP(), used by ExternalIP handling, is IPv4-only, but it's also totally broken anyway... (ExternalP for ovn-kubernetes  #978)

• NetworkPolicy
  We need separate IPv4 and IPv6 address sets. (The idea about using port groups instead of address sets didn't pan out; inport only indicates the ingress port on the current logical switch, so for inter-node traffic we lose the original source port.)

  • pkg/ovn/namespace.go:addPodToNamespace() / deletePodFromNamespace() / AddNamespace() need to check/add/remove multiple IPs from address sets (Dual stack support for address set and network policies. #1393)
  • pkg/ovn/common.go needs to update naming to allow separate IPv4 and IPv6 sets for each namespace (Dual stack support for address set and network policies. #1393)
  • pkg/ovn/policy_common.go:getL3MatchFromAddressSet() needs to match "IPv4-expression || IPv6-expression" (Dual stack support for address set and network policies. #1393)
  • pkg/ovn/policy_common.go:ipMatch() needs to be based on the IP address in question, not the global mode (Dual stack support for address set and network policies. #1393 / Dual stack support for IP blocks in network policy #1428)
  • pkg/ovn/policy_common.go:addAllowACLFromNode() needs to allow from both IPv4 or IPv6 address (Dual stack support for address set and network policies. #1393)

• Unknowns

  • Any additional fixes necessitated by hybrid overlay extensions (VXLAN tunnels to other things) #889 hybrid overlay
  • Any additional fixes necessitated by ExternalP for ovn-kubernetes  #978 ExternalIP fixes
  • Any additional fixes necessitated by Node: add iptables rules and openflow flows for certain cloud load balancers #1126 cloud loadbalancer iptables rules
  • Any additional fixes to shared gateway mode after fixing shared gateway mode does not support IPv6 / dual-stack #1141
  • Any additional fixes necessitated by PRs filed after this issue

cc @dcbw @russellb

[danwinship]

(I'm currently working on all the annotation-related stuff.)

[danwinship]

doing config stuff now too

[Billy99]

@danwinship Looking into "pkg/ovn/port_cache.go: needs to handle multiple pod IPs" now. (#1270)

[Billy99]

Looking into the remaining "pods" bullet item above.

[gvbalaji]

I am looking into these two items under Services

1. pkg/node/gateway_localnet.go:localnetAddService() / localnetDeleteService() need to add/delete IPv4 or IPv6 rules depending on the service IP
2. pkg/node/gateway_localnet.go:localnetNodePortWatcher() needs to set up both iptables and ip6tables base rules

i will discuss the code changes I have done with Dan Winship on these two items

[gvbalaji]

PR's for these two items under services:

1. pkg/node/gateway_localnet.go:localnetAddService() / localnetDeleteService() need to add/delete IPv4 or IPv6 rules depending on the service IP
   2.pkg/node/gateway_localnet.go:localnetNodePortWatcher() needs to set up both iptables and ip6tables base rules

#1326
#1327

[gvbalaji]

@danwinship I am planning to look at this item under "NetworkPolicy".

pkg/ovn/namespace.go:addPodToNamespace() / deletePodFromNamespace() / AddNamespace() need to check/add/remove multiple IPs from address sets

Please let me know if this is ok Or you want me to look at some other policy item first.

[gvbalaji]

pkg/ovn/namespace.go:addPodToNamespace() / deletePodFromNamespace() / AddNamespace() need to check/add/remove multiple IPs from address sets is under progress
PR link
#1376

I think this also addresses this line item where the structure definition has moved to ovn.go

pkg/ovn/common.go needs to update naming to allow separate IPv4 and IPv6 sets for each namespace

[gvbalaji]

Looking into these items now.
pkg/ovn/policy_common.go:getL3MatchFromAddressSet() needs to match "IPv4-expression || IPv6-expression"
pkg/ovn/policy_common.go:ipMatch() needs to be based on the IP address in question, not the global mode
pkg/ovn/policy_common.go:addAllowACLFromNode() needs to allow from both IPv4 or IPv6 address

PR link
#1428

@danwinship

[gvbalaji]

Additional items I found that needs dual stack support are

1. hybrid overlay
   hybrid-overlay/pkg/controller/master.go : Method handleOverlayPort needs to handle multiple subnets in case of dual stack
   hybrid-overlay/pkg/controller/node.go : Method getNodeSubnetAndIP needs to handle multiple CIDRs in case of dual stack
   pkg/ovn/pods.go : Method addRoutesGatewayIP needs to handle dual stack aspects for hybridOverlayExternalGW.

2. pkg/ovn/gateway_init.go : Method gatewayInit needs to configure multiple IPs for lb_force_snat_ip values

3. pkg/ovn/master.go : Method 'StartClusterMaster' needs to handle multiple subnets for shared gateway mode

cc : @danwinship PTAL

[danwinship]

All of the original work is done now... I'm going to update the ipv6 shared gateway bug, and add a new one for hybrid overlay, and we can just fix other things as we find them