Add WireGuard tunnels for Antrea Multi-cluster cross-cluster traffic

Add a traffic encryption mode for cross-cluster traffic. If WireGuard enabled, corresponding WireGuard configuration will be created on the Gateway Node. And all cross-cluster traffic will go through the WireGuard tunnel to remote Gateway. Signed-off-by: hjiajing <hjiajing@vmware.com>
antrea-io · May 15, 2023 · 11ae823 · 11ae823
1 parent 6c500e9
commit 11ae823
Show file tree

Hide file tree

Showing 40 changed files with 1,058 additions and 133 deletions.
diff --git a/build/charts/antrea/README.md b/build/charts/antrea/README.md
@@ -90,6 +90,8 @@ Kubernetes: `>= 1.16.0-0`
 | multicluster.enablePodToPodConnectivity | bool | `false` | Enable Multi-cluster Pod to Pod connectivity. |
 | multicluster.enableStretchedNetworkPolicy | bool | `false` | Enable Multi-cluster NetworkPolicy. Multi-cluster Gateway must be enabled to enable StretchedNetworkPolicy. |
 | multicluster.namespace | string | `""` | The Namespace where Antrea Multi-cluster Controller is running. The default is antrea-agent's Namespace. |
+| multicluster.trafficEncryptionMode | string | `"none"` | Determines how cross-cluster traffic is encrypted. It has the following options: - none (default):  Cross-cluster traffic will not be encrypted. - wireGuard:       Enable WireGuard for tunnel traffic encryption. |
+| multicluster.wireGuard.port | int | `51821` | WireGuard tunnel port for cross-cluster traffic. |
 | noSNAT | bool | `false` | Whether or not to SNAT (using the Node IP) the egress traffic from a Pod to the external network. |
 | nodeIPAM.clusterCIDRs | list | `[]` | CIDR ranges to use when allocating Pod IP addresses. |
 | nodeIPAM.enable | bool | `false` | Enable Node IPAM in Antrea |
@@ -114,7 +116,7 @@ Kubernetes: `>= 1.16.0-0`
 | tlsCipherSuites | string | `""` | Comma-separated list of cipher suites that will be used by the Antrea APIservers. If empty, the default Go Cipher Suites will be used. See https://golang.org/pkg/crypto/tls/#pkg-constants. |
 | tlsMinVersion | string | `""` | TLS min version from: VersionTLS10, VersionTLS11, VersionTLS12, VersionTLS13. |
 | trafficEncapMode | string | `"encap"` | Determines how traffic is encapsulated. It must be one of "encap", "noEncap", "hybrid", or "networkPolicyOnly". |
-| trafficEncryptionMode | string | `"none"` | Determines how tunnel traffic is encrypted. Currently encryption only works with encap mode.It must be one of "none", "ipsec", "wireGuard". |
+| trafficEncryptionMode | string | `"none"` | Determines how tunnel traffic is encrypted. Currently encryption only works with encap mode. It must be one of "none", "ipsec", "wireGuard". |
 | transportInterface | string | `""` | Name of the interface on Node which is used for tunneling or routing the traffic across Nodes. |
 | transportInterfaceCIDRs | list | `[]` | Network CIDRs of the interface on Node which is used for tunneling or routing the traffic across Nodes. |
 | tunnelCsum | bool | `false` | TunnelCsum determines whether to compute UDP encapsulation header (Geneve or VXLAN) checksums on outgoing packets. For Linux kernel before Mar 2021, UDP checksum must be present to trigger GRO on the receiver for better performance of Geneve and VXLAN tunnels. The issue has been fixed by https://github.com/torvalds/linux/commit/89e5c58fc1e2857ccdaae506fb8bc5fed57ee063, thus computing UDP checksum is no longer necessary. It should only be set to true when you are using an unpatched Linux kernel and observing poor transfer performance. |

diff --git a/build/charts/antrea/conf/antrea-agent.conf b/build/charts/antrea/conf/antrea-agent.conf
@@ -346,6 +346,16 @@ multicluster:
   enableStretchedNetworkPolicy: {{ .enableStretchedNetworkPolicy }}
 # Enable Pod to Pod connectivity.
   enablePodToPodConnectivity: {{ .enablePodToPodConnectivity }}
+# Determines how cross-cluster traffic is encrypted.
+# It has the following options:
+# - none (default):  Cross-cluster traffic will not be encrypted.
+# - wireGuard:       Use WireGuard to encrypt traffic.
+  trafficEncryptionMode: {{ .trafficEncryptionMode | quote }}
+# WireGuard tunnel configuration for cross-cluster traffic.
+# It only works when multicluster.trafficEncryptionMode is wireGuard.
+  wireGuard:
+    # WireGuard tunnel port for cross-cluster traffic.
+    port: {{ .wireGuard.port }}
 {{- end }}
 
 {{- if .Values.featureGates.SecondaryNetwork }}

diff --git a/build/charts/antrea/templates/agent/clusterrole.yaml b/build/charts/antrea/templates/agent/clusterrole.yaml
@@ -209,6 +209,7 @@ rules:
     - get
     - list
     - watch
+    - patch
   - apiGroups:
     - multicluster.crd.antrea.io
     resources:

diff --git a/build/charts/antrea/values.yaml b/build/charts/antrea/values.yaml
@@ -24,7 +24,7 @@ tunnelPort: 0
 # observing poor transfer performance.
 tunnelCsum: false
 # -- Determines how tunnel traffic is encrypted. Currently encryption only works
-# with encap mode.It must be one of "none", "ipsec", "wireGuard".
+# with encap mode. It must be one of "none", "ipsec", "wireGuard".
 trafficEncryptionMode: "none"
 # -- Enable bridging mode of Pod network on Nodes, in which the Node's transport
 # interface is connected to the OVS bridge.
@@ -342,6 +342,15 @@ multicluster:
   enableStretchedNetworkPolicy: false
   # -- Enable Multi-cluster Pod to Pod connectivity.
   enablePodToPodConnectivity: false
+  # -- Determines how cross-cluster traffic is encrypted.
+  # It has the following options:
+  # - none (default):  Cross-cluster traffic will not be encrypted.
+  # - wireGuard:       Enable WireGuard for tunnel traffic encryption.
+  trafficEncryptionMode: "none"
+  # WireGuard tunnel configuration for cross-cluster traffic.
+  wireGuard:
+    # -- WireGuard tunnel port for cross-cluster traffic.
+    port: 51821
 
 testing:
   ## -- enable code coverage measurement (used when testing Antrea only).

diff --git a/build/yamls/antrea-aks.yml b/build/yamls/antrea-aks.yml
@@ -3262,6 +3262,16 @@ data:
       enableStretchedNetworkPolicy: false
     # Enable Pod to Pod connectivity.
       enablePodToPodConnectivity: false
+    # Determines how cross-cluster traffic is encrypted.
+    # It has the following options:
+    # - none (default):  Cross-cluster traffic will not be encrypted.
+    # - wireGuard:       Use WireGuard to encrypt traffic.
+      trafficEncryptionMode: "none"
+    # WireGuard tunnel configuration for cross-cluster traffic.
+    # It only works when multicluster.trafficEncryptionMode is wireGuard.
+      wireGuard:
+        # WireGuard tunnel port for cross-cluster traffic.
+        port: 51821
   antrea-cni.conflist: |
     {
         "cniVersion":"0.3.0",
@@ -3744,6 +3754,7 @@ rules:
     - get
     - list
     - watch
+    - patch
   - apiGroups:
     - multicluster.crd.antrea.io
     resources:
@@ -4323,7 +4334,7 @@ spec:
         kubectl.kubernetes.io/default-container: antrea-agent
         # Automatically restart Pods with a RollingUpdate if the ConfigMap changes
         # See https://helm.sh/docs/howto/charts_tips_and_tricks/#automatically-roll-deployments
-        checksum/config: 3af5beafa4cc20ba7f963ed5409de8af66dbd1e185d98a56601d18edf74faba1
+        checksum/config: 1deeecf01f782d7520200f50723554203d03b1130dc698488aea43ff6588e522
       labels:
         app: antrea
         component: antrea-agent
@@ -4564,7 +4575,7 @@ spec:
       annotations:
         # Automatically restart Pod if the ConfigMap changes
         # See https://helm.sh/docs/howto/charts_tips_and_tricks/#automatically-roll-deployments
-        checksum/config: 3af5beafa4cc20ba7f963ed5409de8af66dbd1e185d98a56601d18edf74faba1
+        checksum/config: 1deeecf01f782d7520200f50723554203d03b1130dc698488aea43ff6588e522
       labels:
         app: antrea
         component: antrea-controller

diff --git a/build/yamls/antrea-eks.yml b/build/yamls/antrea-eks.yml
@@ -3262,6 +3262,16 @@ data:
       enableStretchedNetworkPolicy: false
     # Enable Pod to Pod connectivity.
       enablePodToPodConnectivity: false
+    # Determines how cross-cluster traffic is encrypted.
+    # It has the following options:
+    # - none (default):  Cross-cluster traffic will not be encrypted.
+    # - wireGuard:       Use WireGuard to encrypt traffic.
+      trafficEncryptionMode: "none"
+    # WireGuard tunnel configuration for cross-cluster traffic.
+    # It only works when multicluster.trafficEncryptionMode is wireGuard.
+      wireGuard:
+        # WireGuard tunnel port for cross-cluster traffic.
+        port: 51821
   antrea-cni.conflist: |
     {
         "cniVersion":"0.3.0",
@@ -3744,6 +3754,7 @@ rules:
     - get
     - list
     - watch
+    - patch
   - apiGroups:
     - multicluster.crd.antrea.io
     resources:
@@ -4323,7 +4334,7 @@ spec:
         kubectl.kubernetes.io/default-container: antrea-agent
         # Automatically restart Pods with a RollingUpdate if the ConfigMap changes
         # See https://helm.sh/docs/howto/charts_tips_and_tricks/#automatically-roll-deployments
-        checksum/config: 3af5beafa4cc20ba7f963ed5409de8af66dbd1e185d98a56601d18edf74faba1
+        checksum/config: 1deeecf01f782d7520200f50723554203d03b1130dc698488aea43ff6588e522
       labels:
         app: antrea
         component: antrea-agent
@@ -4565,7 +4576,7 @@ spec:
       annotations:
         # Automatically restart Pod if the ConfigMap changes
         # See https://helm.sh/docs/howto/charts_tips_and_tricks/#automatically-roll-deployments
-        checksum/config: 3af5beafa4cc20ba7f963ed5409de8af66dbd1e185d98a56601d18edf74faba1
+        checksum/config: 1deeecf01f782d7520200f50723554203d03b1130dc698488aea43ff6588e522
       labels:
         app: antrea
         component: antrea-controller

diff --git a/build/yamls/antrea-gke.yml b/build/yamls/antrea-gke.yml
@@ -3262,6 +3262,16 @@ data:
       enableStretchedNetworkPolicy: false
     # Enable Pod to Pod connectivity.
       enablePodToPodConnectivity: false
+    # Determines how cross-cluster traffic is encrypted.
+    # It has the following options:
+    # - none (default):  Cross-cluster traffic will not be encrypted.
+    # - wireGuard:       Use WireGuard to encrypt traffic.
+      trafficEncryptionMode: "none"
+    # WireGuard tunnel configuration for cross-cluster traffic.
+    # It only works when multicluster.trafficEncryptionMode is wireGuard.
+      wireGuard:
+        # WireGuard tunnel port for cross-cluster traffic.
+        port: 51821
   antrea-cni.conflist: |
     {
         "cniVersion":"0.3.0",
@@ -3744,6 +3754,7 @@ rules:
     - get
     - list
     - watch
+    - patch
   - apiGroups:
     - multicluster.crd.antrea.io
     resources:
@@ -4323,7 +4334,7 @@ spec:
         kubectl.kubernetes.io/default-container: antrea-agent
         # Automatically restart Pods with a RollingUpdate if the ConfigMap changes
         # See https://helm.sh/docs/howto/charts_tips_and_tricks/#automatically-roll-deployments
-        checksum/config: 530b8f5633759918bc625c6c3e13b8927c2854687f7a5c5bfd420f1c1e15e3cf
+        checksum/config: 48031b31477519b363d88a8964b59e4f2afc445e00c41578b020eb8cd3d8922c
       labels:
         app: antrea
         component: antrea-agent
@@ -4562,7 +4573,7 @@ spec:
       annotations:
         # Automatically restart Pod if the ConfigMap changes
         # See https://helm.sh/docs/howto/charts_tips_and_tricks/#automatically-roll-deployments
-        checksum/config: 530b8f5633759918bc625c6c3e13b8927c2854687f7a5c5bfd420f1c1e15e3cf
+        checksum/config: 48031b31477519b363d88a8964b59e4f2afc445e00c41578b020eb8cd3d8922c
       labels:
         app: antrea
         component: antrea-controller

diff --git a/build/yamls/antrea-ipsec.yml b/build/yamls/antrea-ipsec.yml
@@ -3275,6 +3275,16 @@ data:
       enableStretchedNetworkPolicy: false
     # Enable Pod to Pod connectivity.
       enablePodToPodConnectivity: false
+    # Determines how cross-cluster traffic is encrypted.
+    # It has the following options:
+    # - none (default):  Cross-cluster traffic will not be encrypted.
+    # - wireGuard:       Use WireGuard to encrypt traffic.
+      trafficEncryptionMode: "none"
+    # WireGuard tunnel configuration for cross-cluster traffic.
+    # It only works when multicluster.trafficEncryptionMode is wireGuard.
+      wireGuard:
+        # WireGuard tunnel port for cross-cluster traffic.
+        port: 51821
   antrea-cni.conflist: |
     {
         "cniVersion":"0.3.0",
@@ -3757,6 +3767,7 @@ rules:
     - get
     - list
     - watch
+    - patch
   - apiGroups:
     - multicluster.crd.antrea.io
     resources:
@@ -4336,7 +4347,7 @@ spec:
         kubectl.kubernetes.io/default-container: antrea-agent
         # Automatically restart Pods with a RollingUpdate if the ConfigMap changes
         # See https://helm.sh/docs/howto/charts_tips_and_tricks/#automatically-roll-deployments
-        checksum/config: 55bb24adab5185aad87b77004a64ea6a5736a85ed5e35d3f2e565d746e26dcf6
+        checksum/config: dae3f1d0ff2e557477f7bbc88eadeb287d2bf4e71a68a7d57cc78fd9c30c8062
         checksum/ipsec-secret: d0eb9c52d0cd4311b6d252a951126bf9bea27ec05590bed8a394f0f792dcb2a4
       labels:
         app: antrea
@@ -4621,7 +4632,7 @@ spec:
       annotations:
         # Automatically restart Pod if the ConfigMap changes
         # See https://helm.sh/docs/howto/charts_tips_and_tricks/#automatically-roll-deployments
-        checksum/config: 55bb24adab5185aad87b77004a64ea6a5736a85ed5e35d3f2e565d746e26dcf6
+        checksum/config: dae3f1d0ff2e557477f7bbc88eadeb287d2bf4e71a68a7d57cc78fd9c30c8062
       labels:
         app: antrea
         component: antrea-controller

diff --git a/build/yamls/antrea.yml b/build/yamls/antrea.yml
@@ -3262,6 +3262,16 @@ data:
       enableStretchedNetworkPolicy: false
     # Enable Pod to Pod connectivity.
       enablePodToPodConnectivity: false
+    # Determines how cross-cluster traffic is encrypted.
+    # It has the following options:
+    # - none (default):  Cross-cluster traffic will not be encrypted.
+    # - wireGuard:       Use WireGuard to encrypt traffic.
+      trafficEncryptionMode: "none"
+    # WireGuard tunnel configuration for cross-cluster traffic.
+    # It only works when multicluster.trafficEncryptionMode is wireGuard.
+      wireGuard:
+        # WireGuard tunnel port for cross-cluster traffic.
+        port: 51821
   antrea-cni.conflist: |
     {
         "cniVersion":"0.3.0",
@@ -3744,6 +3754,7 @@ rules:
     - get
     - list
     - watch
+    - patch
   - apiGroups:
     - multicluster.crd.antrea.io
     resources:
@@ -4323,7 +4334,7 @@ spec:
         kubectl.kubernetes.io/default-container: antrea-agent
         # Automatically restart Pods with a RollingUpdate if the ConfigMap changes
         # See https://helm.sh/docs/howto/charts_tips_and_tricks/#automatically-roll-deployments
-        checksum/config: ad7f3df72a2eadf6a2ab30580d81d6ffc43838b464119d100e0646c348214524
+        checksum/config: 261db0ce9a97fec9ea24c49d637d5f72612024f1ecd158ae0ab0577211a900a6
       labels:
         app: antrea
         component: antrea-agent
@@ -4562,7 +4573,7 @@ spec:
       annotations:
         # Automatically restart Pod if the ConfigMap changes
         # See https://helm.sh/docs/howto/charts_tips_and_tricks/#automatically-roll-deployments
-        checksum/config: ad7f3df72a2eadf6a2ab30580d81d6ffc43838b464119d100e0646c348214524
+        checksum/config: 261db0ce9a97fec9ea24c49d637d5f72612024f1ecd158ae0ab0577211a900a6
       labels:
         app: antrea
         component: antrea-controller

diff --git a/cmd/antrea-agent/agent.go b/cmd/antrea-agent/agent.go
@@ -344,8 +344,9 @@ func run(o *Options) error {
 			ciImportInformer,
 			ofClient,
 			nodeConfig,
-			o.config.Multicluster.EnableStretchedNetworkPolicy,
-			o.config.Multicluster.EnablePodToPodConnectivity,
+			networkConfig,
+			routeClient,
+			o.config.Multicluster,
 		)
 		if networkConfig.TrafficEncapMode != config.TrafficEncapModeEncap {
 			mcPodRouteController = mcroute.NewMCPodRouteController(

diff --git a/cmd/antrea-agent/options.go b/cmd/antrea-agent/options.go
@@ -178,6 +178,9 @@ func (o *Options) setDefaults() {
 	} else {
 		o.setExternalNodeDefaultOptions()
 	}
+	if o.config.Multicluster.EnableGateway {
+		o.setMulticlusterDefaultOptions()
+	}
 }
 
 func (o *Options) validateTLSOptions() error {
@@ -324,6 +327,10 @@ func (o *Options) validateMulticlusterConfig(encapMode config.TrafficEncapModeTy
 	if !o.config.Multicluster.EnableGateway && o.config.Multicluster.EnableStretchedNetworkPolicy {
 		return fmt.Errorf("Multi-cluster Gateway must be enabled to enable StretchedNetworkPolicy")
 	}
+	_, multiclusterEncryptionMode := config.GetTrafficEncryptionModeFromStr(o.config.Multicluster.TrafficEncryptionMode)
+	if multiclusterEncryptionMode == config.TrafficEncryptionModeWireGuard && encryptionMode != config.TrafficEncryptionModeNone {
+		return fmt.Errorf("Antrea Multi-cluster WireGuard does not support in-cluster encryption mode %s", o.config.TrafficEncryptionMode)
+	}
 
 	if encapMode.SupportsEncap() && encryptionMode == config.TrafficEncryptionModeWireGuard {
 		return fmt.Errorf("Multi-cluster Gateway doesn't support in-cluster WireGuard encryption")
@@ -618,3 +625,12 @@ func (o *Options) setExternalNodeDefaultOptions() {
 		o.config.ExternalNode.ExternalNodeNamespace = "default"
 	}
 }
+
+func (o *Options) setMulticlusterDefaultOptions() {
+	_, trafficEncryptionModeType := config.GetTrafficEncryptionModeFromStr(o.config.Multicluster.TrafficEncryptionMode)
+	if trafficEncryptionModeType == config.TrafficEncryptionModeWireGuard {
+		if o.config.Multicluster.WireGuard.Port == 0 {
+			o.config.Multicluster.WireGuard.Port = apis.MulticlusterWireGuardListenPort
+		}
+	}
+}
diff --git a/docs/network-requirements.md b/docs/network-requirements.md
@@ -3,16 +3,17 @@
 Antrea has a few network requirements to get started, ensure that your hosts and
 firewalls allow the necessary traffic based on your configuration.
 
-| Configuration                 | Host(s)             | ports/protocols                            | Other                        |
-|-------------------------------|---------------------|--------------------------------------------|------------------------------|
-| Antrea with VXLAN enabled     | All                 | UDP 4789                                   |                              |
-| Antrea with Geneve enabled    | All                 | UDP 6081                                   |                              |
-| Antrea with STT enabled       | All                 | TCP 7471                                   |                              |
-| Antrea with GRE enabled       | All                 | IP Protocol ID 47                          | No support for IPv6 clusters |
-| Antrea with IPsec ESP enabled | All                 | IP protocol ID 50 and 51, UDP 500 and 4500 |                              |
-| Antrea with WireGuard enabled | All                 | UDP 51820                                  |                              |
-| All                           | kube-apiserver host | TCP 443 or 6443\*                          |                              |
-| All                           | All                 | TCP 10349, 10350, 10351, UDP 10351         |                              |
+| Configuration                                  | Host(s)                    | ports/protocols                            | Other                        |
+|------------------------------------------------|----------------------------|--------------------------------------------|------------------------------|
+| Antrea with VXLAN enabled                      | All                        | UDP 4789                                   |                              |
+| Antrea with Geneve enabled                     | All                        | UDP 6081                                   |                              |
+| Antrea with STT enabled                        | All                        | TCP 7471                                   |                              |
+| Antrea with GRE enabled                        | All                        | IP Protocol ID 47                          | No support for IPv6 clusters |
+| Antrea with IPsec ESP enabled                  | All                        | IP protocol ID 50 and 51, UDP 500 and 4500 |                              |
+| Antrea with WireGuard enabled                  | All                        | UDP 51820                                  |                              |
+| Antrea Multi-cluster with WireGuard encryption | Multi-cluster Gateway Node | UDP 51821                                  |                              |
+| All                                            | kube-apiserver host        | TCP 443 or 6443\*                          |                              |
+| All                                            | All                        | TCP 10349, 10350, 10351, UDP 10351         |                              |
 
 \* _The value passed to kube-apiserver using the --secure-port flag. If you cannot
 locate this, check the targetPort value returned by kubectl get svc kubernetes -o yaml._