Create VLAN underlay Network

To deploy a hybridnet cluster with VLAN network, you can install hybridnet with default configuration firstly, then change the default network type to Underlay.

Look at VLAN underlay sample, you will get the basic configuration of creating a VLAN Network. In a word, just create a Network with a VLAN valued spec.mode field (or empty, VLAN is the default mode for an Underlay-type Network):

apiVersion: networking.alibaba.com/v1
kind: Network
metadata:
  name: network1
spec:
  netID: 0                      # Optional.
                                # For Underlay VLAN network, netID refers to VLAN id and can be empty.
                                # If netID is empty, it means the subnets belong to this 
                                # network can have any netID but not empty. 
                                # If netID is not empty, all the subnets belong to this network
                                # can only has an empty or the same netID. 
                                
  type: Underlay                # Required. Underlay or Overlay.
  mode: VLAN                    # Optional. VLAN is the default mode for Underlay network.
  
  nodeSelector:                 # Required only for underlay Network.
    network: "s1"               # Label to select target Nodes, which means every node belongs to 
                                # this network should be patched with this label.

Almost like creating an overlay environment, creating an VLAN underlay container network needs at least one underlay Network object with VLAN mode (followed Subnet is also needed, a Network always needs at least one Subnet to provide address management information).

The biggest difference between creating an Overlay Network and an Underlay Network is that we need to group Nodes in the cluster. The Nodes in the same "Underlay Network" group need to have some common networking properties, e.g., all of them in the same VLAN, all of them belong to the same TOR switch.

The general description for a group of Nodes in the same VLAN Network is: at least one NIC on every node need to be attached to the same VLAN.

Hybridnet use Selector/Label in Kubernetes to differ nodes in different groups. To group Nodes in the cluster, we need to define a label for each group of Nodes and label each group of them with a different label (use kubectl label commands). Then we create a Network object for every Node group with a specific .spec.nodeSelector field.

After a Network is created, the Nodes it select will be updated to its status.nodeList field, as follows.

apiVersion: networking.alibaba.com/v1
kind: Network
...
status:
  ...
  nodeList:                                           # Selected nodes.
  - network-test00002
  - network-test00003
  ...

For VLAN network of hybridnet, hybridnet provides a capability combines the ipvlan and vlan CNI plugins (none of them is actually used in hybridnet). Pod with VLAN mode send packets out though the Nodes' physical NIC directly, with or without a specific VLAN tag, always share the same MAC addresses with Nodes.

Before applying a Subnet object for a VLAN Network, we need to ensure some dependencies in the underlying interconnect fabric. For example, if we want to apply a Subnet like this:

---
apiVersion: networking.alibaba.com/v1
kind: Subnet
metadata:
  name: underlay-subnet1
spec:
  network: underlay-network1
  netID: 0
  range:
    version: "4"
    cidr: "192.168.56.0/24"
    gateway: "192.168.56.1"

Let's assume eth0 is the actual NIC which hybridnet is using. Value of spec.netID determines whether or not hybirdnet will generate a vlan subinterface for eth0.

If spec.netID is 0, hybridnet will use eth0 as the output interface of container network packets, which means the container network packets will not get any VLAN id before getting through eth0. In this case, the gateway, represented by spec.range.gateway field, need to be in the same VLAN as Node, which also means the gateway can respond a ARP request from the Node.

If spec.netID is not 0, let's make it 10, hybridnet will generate a vlan subinterface named eth0.10 as the first-step output interface for container network packets, so the packets will be attached with a VLAN tag of 10 before it's getting through eth0. In this case, we have to make sure the gateway is in the same VLAN as Subnet's spec.netID, which also means the gateway can respond a ARP request with a specified VLAN tag from the Node.

Just like physical machines in a VLAN network, if underlay pods are in the same CIDR, they communicate with each other by L2 direct link-layer packet switching. When it comes to the cross-subnet communication (except the traffic between overlay pods and underlay pods, which we have introduced in Getting Started), it depends on the L3 routing capabilities of the router specified by spec.range.gateway. That also means, if the underlay Pods need to be reachable for Nodes, some extra configuration on physical or virtual router might be required.