SIP Cluster Operator

Overview

The lifecycle of the VM's and their relationship to Cluster will be managed using two operators: vNode-Operator(ViNO) and the Support Infra Provider Operator (SIP) .

Description

The SIP Cluster Operator helps identity appropriate BareMetalHost objects to fulfill a tenant cluster, including initial creation as well as expanding and contracting it over time. It also helps create supporting per-cluster supporting infrastructure such as LoadBalancers, Jump Hosts, and so on as value added cluster services for each cluster.

While ViNO is responsible for setting up VM infrastructure, such as:

• per-node vino pod:
  • libvirt init, e.g.
    • setup vm-infra bridge
    • provisioning tftp/dhcp definition
  • libvirt launch
  • sushi pod
• libvirt domains
• networking
• bmh objects, with labels:
  • location - i.e. rack: 8 and node: rdm8r008c002 - should follow k8s semi-standard
  • vm role - i.e. node-type: worker
  • vm flavor - i.e node-flavor: foobar
  • networks - i.e. networks: [foo, bar] and the details for ViNO can be found here

The Cluster Support Infrastructure Provider, or SIP, is responsible for the lifecycle of:

• identifying the correct BareMetalHost resources to label (or unlabel) based on scheduling constraints.
• extract IP address information from BareMetalHost objects to use in the creation of supporting infrastructure.
• creating support infra for the tenant k8s cluster:
  • load balancers (for tenant k8s api)
  • jump pod to access the cluster and nodes via ssh
  • an OIDC provider for the tenant cluster, i.e. Dex
  • potentially more in the future

SIP Operator High level Algorithm

::::info The expectation is that the operator will only deal with one SIPCluster object at a time -- in other words serially. There will be absolutely no concurrency support. This is critical to avoid race conditions. There is an expectation that all of the operations below are idempotent. ::::

Pseudo Algorithm at a high level after reading the SIPCluster CR:

Gather Phase

Identity BMH VM's

• Gather BMH's that meet the criteria expected for the groups
• Check for existing labeled BMH's
• Complete the expected scheduling contraints :
  • If master
    • collect into list of bmh's to label
  • If worker
    • collect into list of bmh's to label

Extract Info from Identified BMH

• identify and extract the IP address ands other info as needed (***)
  • Use it as part of the service infrastucture configuration
• At this point I have a list of BMH's, and I have the extrapolated data I need for configuring services.

Service Infrastructure Deploy Phase

• Create or Updated the [LB|admin pod] with the appropriate configuration

Label Phase

• Label the collected hosts.
• At this point SIPCluster is done processing a given CR, and can move on the next.

SIPCluster CR will exists within the Control phase for a Tenant cluster.

Development environment

Kind kubernetes cluster

Fastest way to set up a k8s cluster for development env is to use kind to set it up

Install kind on linux (amd64 arch)

# curl -Lo kind https://kind.sigs.k8s.io/dl/v0.9.0/kind-linux-amd64
# sudo install  -m 755 --owner=root --group=root kind /usr/local/bin
# rm kind

More information on how to install kind binary can be found be found here

Create k8s cluster with kind

# make kind-create
# kubectl get nodes

Deploy SIP operator on top of kind cluster

kind-load-image target will build docker image from the current state of your local git repository and upload it to kind cluster to be available for kubelet.

# make kind-load-image
# make deploy

Now you have a working k8s cluster with sip installed on it with your changes to SIP operator

Deliver SIP CRs to kubernetes

Now you are ready to craft and add BaremetalHost CRs into cluster, check samples directory to find BaremetalHost examples there.

Use kubectl apply to deliver SIP CRs and BaremetalHost CRDs to kubernetes cluster

# kustomize build config/samples | kubectl apply -f -

Testing

Need kubebuilder installed to run tests.

Installation of kubebuilder:

# os=$(go env GOOS)
# arch=$(go env GOARCH)

download kubebuilder and extract it to /tmp
# curl -L https://go.kubebuilder.io/dl/2.3.1/${os}/${arch} | tar -xz -C /tmp/

move to a long-term location and put it on your path
(you'll need to set the KUBEBUILDER_ASSETS env var if you put it somewhere else)
# sudo mv /tmp/kubebuilder_2.3.1_${os}_${arch} /usr/local/kubebuilder
# export PATH=$PATH:/usr/local/kubebuilder/bin

Run the tests: Run make test to execute a suite of unit and integration tests against the SIP operator.