-
Notifications
You must be signed in to change notification settings - Fork 550
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
docs: add documentation on using Multus with Talos
Short introduction into running Multus CNI. Signed-off-by: Michael Trip <michael@alcatrash.net> Signed-off-by: Andrey Smirnov <andrey.smirnov@siderolabs.com>
- Loading branch information
Showing
2 changed files
with
286 additions
and
0 deletions.
There are no files selected for viewing
143 changes: 143 additions & 0 deletions
143
website/content/v1.7/kubernetes-guides/network/multus.md
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,143 @@ | ||
| --- | ||
| title: "Multus CNI" | ||
| description: "A brief instruction on howto use Multus on Talos Linux" | ||
| --- | ||
|
|
||
| [Multus CNI](https://github.com/k8snetworkplumbingwg/multus-cni) is a container network interface (CNI) plugin for Kubernetes that enables attaching multiple network interfaces to pods. | ||
| Typically, in Kubernetes each pod only has one network interface (apart from a loopback) -- with Multus you can create a multi-homed pod that has multiple interfaces. | ||
| This is accomplished by Multus acting as a "meta-plugin", a CNI plugin that can call multiple other CNI plugins. | ||
|
|
||
| ## Installation | ||
|
|
||
| Multus can be deployed by simply applying the `thick` `DaemonSet` with `kubectl`. | ||
|
|
||
| ```bash | ||
| kubectl apply -f https://raw.githubusercontent.com/k8snetworkplumbingwg/multus-cni/master/deployments/multus-daemonset-thick.yml | ||
| ``` | ||
|
|
||
| This will create a `DaemonSet` and a CRD: `NetworkAttachmentDefinition`. | ||
| This can be used to specify your network configuration. | ||
|
|
||
| ## Configuration | ||
|
|
||
| ### Patching the `DaemonSet` | ||
|
|
||
| For Multus to properly work with Talos a change need to be made to the `DaemonSet`. | ||
| Instead of of mounting the volume called `host-run-netns` on `/run/netns` it has to be mounted on `/var/run/netns`. | ||
|
|
||
| Edit the `DaemonSet` and change the volume `host-run-netns` from `/run/netns` to `/var/run/netns`. | ||
|
|
||
| ```yaml | ||
| ... | ||
| - name: host-run-netns | ||
| hostPath: | ||
| path: /var/run/netns/ | ||
| ``` | ||
| Failing to do so will leave your cluster crippled. | ||
| Running pods will remain running but new pods and deployments will give you the following error in the events: | ||
| ```text | ||
| Normal Scheduled 3s default-scheduler Successfully assigned virtualmachines/samplepod to virt2 | ||
| Warning FailedCreatePodSandBox 3s kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "3a6a58386dfbf2471a6f86bd41e4e9a32aac54ccccd1943742cb67d1e9c58b5b": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"3a6a58386dfbf2471a6f86bd41e4e9a32aac54ccccd1943742cb67d1e9c58b5b" Netns:"/var/run/netns/cni-1d80f6e3-fdab-4505-eb83-7deb17431293" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=virtualmachines;K8S_POD_NAME=samplepod;K8S_POD_INFRA_CONTAINER_ID=3a6a58386dfbf2471a6f86bd41e4e9a32aac54ccccd1943742cb67d1e9c58b5b;K8S_POD_UID=8304765e-fd7e-4968-9144-c42c53be04f4" Path:"" ERRORED: error configuring pod [virtualmachines/samplepod] networking: [virtualmachines/samplepod/8304765e-fd7e-4968-9144-c42c53be04f4:cbr0]: error adding container to network "cbr0": DelegateAdd: cannot set "" interface name to "eth0": validateIfName: no net namespace /var/run/netns/cni-1d80f6e3-fdab-4505-eb83-7deb17431293 found: failed to Statfs "/var/run/netns/cni-1d80f6e3-fdab-4505-eb83-7deb17431293": no such file or directory | ||
| ': StdinData: {"capabilities":{"portMappings":true},"clusterNetwork":"/host/etc/cni/net.d/10-flannel.conflist","cniVersion":"0.3.1","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","type":"multus-shim"} | ||
| ``` | ||
| ### Creating your `NetworkAttachmentDefinition` | ||
| The `NetworkAttachmentDefinition` configuration is used to define your bridge where your second pod interface needs to be attached to. | ||
| ```yaml | ||
| apiVersion: "k8s.cni.cncf.io/v1" | ||
| kind: NetworkAttachmentDefinition | ||
| metadata: | ||
| name: macvlan-conf | ||
| spec: | ||
| config: '{ | ||
| "cniVersion": "0.3.0", | ||
| "type": "macvlan", | ||
| "master": "eth0", | ||
| "mode": "bridge", | ||
| "ipam": { | ||
| "type": "host-local", | ||
| "subnet": "192.168.1.0/24", | ||
| "rangeStart": "192.168.1.200", | ||
| "rangeEnd": "192.168.1.216", | ||
| "routes": [ | ||
| { "dst": "0.0.0.0/0" } | ||
| ], | ||
| "gateway": "192.168.1.1" | ||
| } | ||
| }' | ||
| ``` | ||
|
|
||
| In this example `macvlan` is used as a bridge type. | ||
| There are 3 types of bridges: `bridge`, `macvlan` and `ipvlan`: | ||
|
|
||
| 1. `bridge` is a way to connect two Ethernet segments together in a protocol-independent way. | ||
| Packets are forwarded based on Ethernet address, rather than IP address (like a router). | ||
| Since forwarding is done at Layer 2, all protocols can go transparently through a bridge. | ||
| In terms of containers or virtual machines, a bridge can also be used to connect the virtual interfaces of each container/VM to the host network, allowing them to communicate. | ||
|
|
||
| 2. `macvlan` is a driver that makes it possible to create virtual network interfaces that appear as distinct physical devices each with unique MAC addresses. | ||
| The underlying interface can route traffic to each of these virtual interfaces separately, as if they were separate physical devices. | ||
| This means that each macvlan interface can have its own IP subnet and routing. | ||
| Macvlan interfaces are ideal for situations where containers or virtual machines require the same network access as the host system. | ||
|
|
||
| 3. `ipvlan` is similar to `macvlan`, with the key difference being that ipvlan shares the parent's MAC address, which requires less configuration from the networking equipment. | ||
| This makes deployments simpler in certain situations where MAC address control or limits are in place. | ||
| It offers two operational modes: L2 mode (the default) where it behaves similarly to a MACVLAN, and L3 mode for routing based traffic isolation (rather than bridged). | ||
|
|
||
| When using the `bridge` interface you must also configure a bridge on your Talos nodes. | ||
| That can be done by updating Talos Linux machine configuration: | ||
|
|
||
| ```yaml | ||
| machine: | ||
| interfaces: | ||
| - interface: br0 | ||
| addresses: | ||
| - 172.16.1.60/24 | ||
| bridge: | ||
| stp: | ||
| enabled: true | ||
| interfaces: | ||
| - eno1 # This must be changed to your matching interface name | ||
| routes: | ||
| - network: 0.0.0.0/0 # The route's network (destination). | ||
| gateway: 172.16.1.254 # The route's gateway (if empty, creates link scope route). | ||
| metric: 1024 # The optional metric for the route. | ||
| ``` | ||
| More information about the configuration of bridges can be found [here](https://github.com/k8snetworkplumbingwg/multus-cni/tree/master/docs) | ||
| ## Attaching the `NetworkAttachmentDefinition` to your `Pod` or `Deployment` | ||
|
|
||
| After the `NetworkAttachmentDefinition` is configured, you can attach that interface to your your `Deployment` or `Pod`. | ||
| In this example we use a pod: | ||
|
|
||
| ```yaml | ||
| apiVersion: v1 | ||
| kind: Pod | ||
| metadata: | ||
| name: samplepod | ||
| annotations: | ||
| k8s.v1.cni.cncf.io/networks: macvlan-conf | ||
| spec: | ||
| containers: | ||
| - name: samplepod | ||
| command: ["/bin/ash", "-c", "trap : TERM INT; sleep infinity & wait"] | ||
| image: alpine | ||
| ``` | ||
|
|
||
| ## Notes on using KubeVirt in combination with Multus | ||
|
|
||
| If you would like to use KubeVirt and expose your virtual machine to the outside world with Multus, make sure to configure a `bridge` instead of `macvlan` or `ipvlan`, because that doesn't work, according to the KubeVirt [Documentation](https://kubevirt.io/user-guide/virtual_machines/interfaces_and_networks/#invalid-cnis-for-secondary-networks). | ||
|
|
||
| > Invalid CNIs for secondary networks | ||
| > The following list of CNIs is known not to work for bridge interfaces - which are most common for secondary interfaces. | ||
| > | ||
| > * macvlan | ||
| > * ipvlan | ||
| > | ||
| > The reason is similar: the bridge interface type moves the pod interface MAC address to the VM, leaving the pod interface with a different address. | ||
| > The aforementioned CNIs require the pod interface to have the original MAC address. |
143 changes: 143 additions & 0 deletions
143
website/content/v1.8/kubernetes-guides/network/multus.md
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,143 @@ | ||
| --- | ||
| title: "Multus CNI" | ||
| description: "A brief instruction on howto use Multus on Talos Linux" | ||
| --- | ||
|
|
||
| [Multus CNI](https://github.com/k8snetworkplumbingwg/multus-cni) is a container network interface (CNI) plugin for Kubernetes that enables attaching multiple network interfaces to pods. | ||
| Typically, in Kubernetes each pod only has one network interface (apart from a loopback) -- with Multus you can create a multi-homed pod that has multiple interfaces. | ||
| This is accomplished by Multus acting as a "meta-plugin", a CNI plugin that can call multiple other CNI plugins. | ||
|
|
||
| ## Installation | ||
|
|
||
| Multus can be deployed by simply applying the `thick` `DaemonSet` with `kubectl`. | ||
|
|
||
| ```bash | ||
| kubectl apply -f https://raw.githubusercontent.com/k8snetworkplumbingwg/multus-cni/master/deployments/multus-daemonset-thick.yml | ||
| ``` | ||
|
|
||
| This will create a `DaemonSet` and a CRD: `NetworkAttachmentDefinition`. | ||
| This can be used to specify your network configuration. | ||
|
|
||
| ## Configuration | ||
|
|
||
| ### Patching the `DaemonSet` | ||
|
|
||
| For Multus to properly work with Talos a change need to be made to the `DaemonSet`. | ||
| Instead of of mounting the volume called `host-run-netns` on `/run/netns` it has to be mounted on `/var/run/netns`. | ||
|
|
||
| Edit the `DaemonSet` and change the volume `host-run-netns` from `/run/netns` to `/var/run/netns`. | ||
|
|
||
| ```yaml | ||
| ... | ||
| - name: host-run-netns | ||
| hostPath: | ||
| path: /var/run/netns/ | ||
| ``` | ||
| Failing to do so will leave your cluster crippled. | ||
| Running pods will remain running but new pods and deployments will give you the following error in the events: | ||
| ```text | ||
| Normal Scheduled 3s default-scheduler Successfully assigned virtualmachines/samplepod to virt2 | ||
| Warning FailedCreatePodSandBox 3s kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "3a6a58386dfbf2471a6f86bd41e4e9a32aac54ccccd1943742cb67d1e9c58b5b": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"3a6a58386dfbf2471a6f86bd41e4e9a32aac54ccccd1943742cb67d1e9c58b5b" Netns:"/var/run/netns/cni-1d80f6e3-fdab-4505-eb83-7deb17431293" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=virtualmachines;K8S_POD_NAME=samplepod;K8S_POD_INFRA_CONTAINER_ID=3a6a58386dfbf2471a6f86bd41e4e9a32aac54ccccd1943742cb67d1e9c58b5b;K8S_POD_UID=8304765e-fd7e-4968-9144-c42c53be04f4" Path:"" ERRORED: error configuring pod [virtualmachines/samplepod] networking: [virtualmachines/samplepod/8304765e-fd7e-4968-9144-c42c53be04f4:cbr0]: error adding container to network "cbr0": DelegateAdd: cannot set "" interface name to "eth0": validateIfName: no net namespace /var/run/netns/cni-1d80f6e3-fdab-4505-eb83-7deb17431293 found: failed to Statfs "/var/run/netns/cni-1d80f6e3-fdab-4505-eb83-7deb17431293": no such file or directory | ||
| ': StdinData: {"capabilities":{"portMappings":true},"clusterNetwork":"/host/etc/cni/net.d/10-flannel.conflist","cniVersion":"0.3.1","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","type":"multus-shim"} | ||
| ``` | ||
| ### Creating your `NetworkAttachmentDefinition` | ||
| The `NetworkAttachmentDefinition` configuration is used to define your bridge where your second pod interface needs to be attached to. | ||
| ```yaml | ||
| apiVersion: "k8s.cni.cncf.io/v1" | ||
| kind: NetworkAttachmentDefinition | ||
| metadata: | ||
| name: macvlan-conf | ||
| spec: | ||
| config: '{ | ||
| "cniVersion": "0.3.0", | ||
| "type": "macvlan", | ||
| "master": "eth0", | ||
| "mode": "bridge", | ||
| "ipam": { | ||
| "type": "host-local", | ||
| "subnet": "192.168.1.0/24", | ||
| "rangeStart": "192.168.1.200", | ||
| "rangeEnd": "192.168.1.216", | ||
| "routes": [ | ||
| { "dst": "0.0.0.0/0" } | ||
| ], | ||
| "gateway": "192.168.1.1" | ||
| } | ||
| }' | ||
| ``` | ||
|
|
||
| In this example `macvlan` is used as a bridge type. | ||
| There are 3 types of bridges: `bridge`, `macvlan` and `ipvlan`: | ||
|
|
||
| 1. `bridge` is a way to connect two Ethernet segments together in a protocol-independent way. | ||
| Packets are forwarded based on Ethernet address, rather than IP address (like a router). | ||
| Since forwarding is done at Layer 2, all protocols can go transparently through a bridge. | ||
| In terms of containers or virtual machines, a bridge can also be used to connect the virtual interfaces of each container/VM to the host network, allowing them to communicate. | ||
|
|
||
| 2. `macvlan` is a driver that makes it possible to create virtual network interfaces that appear as distinct physical devices each with unique MAC addresses. | ||
| The underlying interface can route traffic to each of these virtual interfaces separately, as if they were separate physical devices. | ||
| This means that each macvlan interface can have its own IP subnet and routing. | ||
| Macvlan interfaces are ideal for situations where containers or virtual machines require the same network access as the host system. | ||
|
|
||
| 3. `ipvlan` is similar to `macvlan`, with the key difference being that ipvlan shares the parent's MAC address, which requires less configuration from the networking equipment. | ||
| This makes deployments simpler in certain situations where MAC address control or limits are in place. | ||
| It offers two operational modes: L2 mode (the default) where it behaves similarly to a MACVLAN, and L3 mode for routing based traffic isolation (rather than bridged). | ||
|
|
||
| When using the `bridge` interface you must also configure a bridge on your Talos nodes. | ||
| That can be done by updating Talos Linux machine configuration: | ||
|
|
||
| ```yaml | ||
| machine: | ||
| interfaces: | ||
| - interface: br0 | ||
| addresses: | ||
| - 172.16.1.60/24 | ||
| bridge: | ||
| stp: | ||
| enabled: true | ||
| interfaces: | ||
| - eno1 # This must be changed to your matching interface name | ||
| routes: | ||
| - network: 0.0.0.0/0 # The route's network (destination). | ||
| gateway: 172.16.1.254 # The route's gateway (if empty, creates link scope route). | ||
| metric: 1024 # The optional metric for the route. | ||
| ``` | ||
| More information about the configuration of bridges can be found [here](https://github.com/k8snetworkplumbingwg/multus-cni/tree/master/docs) | ||
| ## Attaching the `NetworkAttachmentDefinition` to your `Pod` or `Deployment` | ||
|
|
||
| After the `NetworkAttachmentDefinition` is configured, you can attach that interface to your your `Deployment` or `Pod`. | ||
| In this example we use a pod: | ||
|
|
||
| ```yaml | ||
| apiVersion: v1 | ||
| kind: Pod | ||
| metadata: | ||
| name: samplepod | ||
| annotations: | ||
| k8s.v1.cni.cncf.io/networks: macvlan-conf | ||
| spec: | ||
| containers: | ||
| - name: samplepod | ||
| command: ["/bin/ash", "-c", "trap : TERM INT; sleep infinity & wait"] | ||
| image: alpine | ||
| ``` | ||
|
|
||
| ## Notes on using KubeVirt in combination with Multus | ||
|
|
||
| If you would like to use KubeVirt and expose your virtual machine to the outside world with Multus, make sure to configure a `bridge` instead of `macvlan` or `ipvlan`, because that doesn't work, according to the KubeVirt [Documentation](https://kubevirt.io/user-guide/virtual_machines/interfaces_and_networks/#invalid-cnis-for-secondary-networks). | ||
|
|
||
| > Invalid CNIs for secondary networks | ||
| > The following list of CNIs is known not to work for bridge interfaces - which are most common for secondary interfaces. | ||
| > | ||
| > * macvlan | ||
| > * ipvlan | ||
| > | ||
| > The reason is similar: the bridge interface type moves the pod interface MAC address to the VM, leaving the pod interface with a different address. | ||
| > The aforementioned CNIs require the pod interface to have the original MAC address. |