[Prpopsal] Support nodepool with host network mode

[Lan-ce-lot]

Summary

We need a new type of node pool that uses host network mode, and components corresponding to non-host networks (such as kube-proxy, flannel, coredns) will not be installed on these nodes. By adding a field node-network-mode=true to describe the host network mode NodePool, modifying the nodepool controller to implement label distribution to the corresponding nodepool nodes, and adding a Pod webhook to modify the Pod network, we can improve the performance and efficiency of edge nodes.

Motivation

In the cloud edge architecture, some edge nodes are only used to install simple applications that do not need to access other Pods through services or DNS. At the same time, these Pods only need to use the host network. Installing kube-proxy/flannel/coredns components on these nodes is a waste of resources because these components are not needed in this scenario. Kube-proxy is responsible for implementing load balancing and traffic forwarding for services, flannel is responsible for implementing network communication between Pods across hosts, and coredns is responsible for implementing domain name resolution between Pods. If Pods on edge nodes only need to use the host network, these components are not necessary. On the contrary, these components will occupy the limited CPU, memory, and network resources of edge nodes, reducing the performance and efficiency of edge nodes. Therefore, they should be avoided or optimized.

It can be seen below that kube-proxy/flannel/coredns components are occupying a significant amount of CPU and memory.

$ k top po -A --sort-by=memory
NAMESPACE      NAME                                       CPU(cores)   MEMORY(bytes)
kube-system    kube-apiserver-master1                     127m         350Mi
kube-system    etcd-master1                               29m          81Mi
kube-system    kube-controller-manager-master1            43m          49Mi
cert-manager   cert-manager-cainjector-74bfccdfdf-4kvf7   3m           30Mi
kube-system    kube-proxy-fttg6                           1m           22Mi
kube-flannel   kube-flannel-ds-cd9f6                      7m           21Mi
cert-manager   cert-manager-b4d6fd99b-bj9jm               2m           17Mi
kube-system    coredns-7d89d9b6b8-vhqj7                   4m           16Mi
kube-system    coredns-7d89d9b6b8-fn9c4                   3m           16Mi
kube-system    kube-scheduler-master1                     6m           16Mi
kube-system    metrics-server-66dd897cc4-zl2sq            8m           14Mi
kube-system    metrics-server-59d8dc4bc-54v59             6m           13Mi
kube-flannel   kube-flannel-ds-cs9dp                      9m           11Mi
cert-manager   cert-manager-webhook-65b766b5f8-z9k6c      5m           11Mi
kube-system    kube-proxy-x2v2r                           3m           10Mi
default        nginx-pod                                  0m           3Mi

Goals

Implement a new type of NodePool that enables Pods on the nodes to use the host network while not using services, CoreDNS, and other components.

Non-Goals/Future Work

Proposal

Definition

• NodePool with host network mode

  A new NodePool type where Pods will only use host networking mode, suitable for lightweight and minimally demanding edge nodes.

Architecture

Here is a NodePool in host network mode:

apiVersion: apps.openyurt.io/v1beta1
kind: NodePool
metadata:
  name: beijing
spec:
  type: Cloud
  networkMode: HostNetwork

The work we need to do is as follows:

• Step1: First, before adding a node to the NodePool, you need to modify the kubelet configuration of the node. This part of the work can be done by yurtadm join.

/var/lib/kubelet/kubeadm-flags.env

—network-plugin=cni

—cni-conf-dir=/etc/cni/net.d

For example:

KUBELET_KUBEADM_ARGS="--network-plugin=cni —cni-conf-dir=/etc/cni/net.d --pod-infra-container-image=registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.5"

Set the network plugin to loopback.

sudo cat <<EOF > /etc/cni/net.d/0-loopback.conf
{
  "cniVersion": "0.3.0",
  "name": "lo",
  "type": "loopback"
}
EOF

Restart kubelet

systemctl daemon-reload & systemctl restart kubelet

• Step2: Design a new field for NodePool called hostNetwork=true/false. True indicates that this is a NodePool of hostNetwork mode type.
• Step3: Modify the NodePool controller and add nodepool.openyurt.io/host-network=true tag to all nodes in the node-network-mode NodePool.
• Step4: Add a webhook to verify if there is a nodepool named xxx for the pods with the nodepool.openyurt.io/desired-nodepool: xxx tag. If it exists, check if the hostNetwork field of the nodepool is true. If it is true, set the pod's hostNetwork to true. Users can add the pod label themselves or use YurtAppSet and YurtAppDaemon to deploy the application.

Step 5: Before installing openyurt, it is required that users add node anti-affinity to components such as kube-proxy(daemonset)/coredns(deployment) of the cluster, and do not schedule nodes without the nodepool.openyurt.io/network-mode=node-network-mode label.

affinity:
  nodeAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:
      nodeSelectorTerms:
      - matchExpressions:
        - key: nodepool.openyurt.io/network-mode
          operator: NotIn
          values:
            - node-network-mode

User Stories

As a user, I hope my edge nodes are as lightweight as possible, consume minimal resources, and have a simple network model.

[rambohe-ch]

@Lan-ce-lot This proposal is followed by pull request #1492 , so i will close this issue.