v1.6.0: Targets are not adding to the load balancer

[qwinkler]

Probably I misconfigured something, but I can't figure out what exactly.

There are the logs from Cloud Controller Manager:

root@test:~# kubectl logs hcloud-cloud-controller-manager-656bbd88db-n8d7r -n kube-system
Flag --allow-untagged-cloud has been deprecated, This flag is deprecated and will be removed in a future release. A cluster-id will be required on cloud instances.
I0625 08:14:46.827619       1 serving.go:313] Generated self-signed cert in-memory
W0625 08:14:47.474831       1 client_config.go:552] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
I0625 08:14:47.477714       1 controllermanager.go:120] Version: v0.0.0-master+$Format:%h$
Hetzner Cloud k8s cloud controller v1.6.0 started
W0625 08:14:48.027637       1 controllermanager.go:132] detected a cluster without a ClusterID.  A ClusterID will be required in the future.  Please tag your cluster to avoid any future issues
I0625 08:14:48.028791       1 configmap_cafile_content.go:202] Starting client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file
I0625 08:14:48.028815       1 shared_informer.go:223] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file
I0625 08:14:48.028859       1 configmap_cafile_content.go:202] Starting client-ca::kube-system::extension-apiserver-authentication::client-ca-file
I0625 08:14:48.028866       1 shared_informer.go:223] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::client-ca-file
I0625 08:14:48.029437       1 secure_serving.go:178] Serving securely on [::]:10258
I0625 08:14:48.029536       1 tlsconfig.go:240] Starting DynamicServingCertificateController
I0625 08:14:48.030593       1 node_lifecycle_controller.go:78] Sending events to api server
I0625 08:14:48.030656       1 controllermanager.go:247] Started "cloud-node-lifecycle"
I0625 08:14:48.031936       1 controllermanager.go:247] Started "service"
I0625 08:14:48.032093       1 controller.go:208] Starting service controller
I0625 08:14:48.032107       1 shared_informer.go:223] Waiting for caches to sync for service
I0625 08:14:48.125698       1 controllermanager.go:247] Started "route"
I0625 08:14:48.125895       1 route_controller.go:100] Starting route controller
I0625 08:14:48.125949       1 shared_informer.go:223] Waiting for caches to sync for route
I0625 08:14:48.126733       1 node_controller.go:110] Sending events to api server.
I0625 08:14:48.126796       1 controllermanager.go:247] Started "cloud-node"
I0625 08:14:48.129535       1 shared_informer.go:230] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::client-ca-file
I0625 08:14:48.129685       1 shared_informer.go:230] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file
I0625 08:14:48.144444       1 node_controller.go:325] Initializing node test with cloud provider
I0625 08:14:48.226209       1 shared_informer.go:230] Caches are synced for route
I0625 08:14:48.232277       1 shared_informer.go:230] Caches are synced for service
I0625 08:14:48.562019       1 route_controller.go:193] Creating route for node test 10.244.0.0/24 with hint c3a9fdd8-0b4e-4359-a30f-069f57cbd98c, throttled 862ns
I0625 08:14:49.726362       1 route_controller.go:213] Created route for node test 10.244.0.0/24 with hint c3a9fdd8-0b4e-4359-a30f-069f57cbd98c after 1.16434051s
I0625 08:14:49.785607       1 node_controller.go:397] Successfully initialized node test with cloud provider
I0625 08:15:40.063967       1 event.go:278] Event(v1.ObjectReference{Kind:"Service", Namespace:"default", Name:"echoserver", UID:"cec7350c-c004-400a-be2c-f4526e467760", APIVersion:"v1", ResourceVersion:"805", FieldPath:""}): type: 'Normal' reason: 'EnsuringLoadBalancer' Ensuring load balancer
I0625 08:15:40.076748       1 load_balancers.go:81] "ensure Load Balancer" op="hcloud/loadBalancers.EnsureLoadBalancer" service="echoserver" nodes=[]
I0625 08:15:40.077134       1 event.go:278] Event(v1.ObjectReference{Kind:"Service", Namespace:"default", Name:"echoserver", UID:"cec7350c-c004-400a-be2c-f4526e467760", APIVersion:"v1", ResourceVersion:"805", FieldPath:""}): type: 'Warning' reason: 'UnAvailableLoadBalancer' There are no available nodes for LoadBalancer
I0625 08:15:54.987363       1 load_balancer.go:420] "add service" op="hcops/LoadBalancerOps.ReconcileHCLBServices" port=80 loadBalancerID=35028
I0625 08:15:55.824007       1 load_balancers.go:117] "reload HC Load Balancer" op="hcloud/loadBalancers.EnsureLoadBalancer" loadBalancerID=35028
I0625 08:15:55.919779       1 event.go:278] Event(v1.ObjectReference{Kind:"Service", Namespace:"default", Name:"echoserver", UID:"cec7350c-c004-400a-be2c-f4526e467760", APIVersion:"v1", ResourceVersion:"805", FieldPath:""}): type: 'Normal' reason: 'EnsuredLoadBalancer' Ensured load balancer

Steps to reproduce:

# network range: 10.0.0.0/8
# instance specs:
# image: ubuntu-18.04, type: cpx21, server is assigned to the 10.0.0.0/8 network with private IP: 10.0.0.2

curl https://get.docker.com | VERSION=19.03.12 sh
cat > /etc/docker/daemon.json <<EOF
{
  "exec-opts": ["native.cgroupdriver=systemd"],
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "100m"
  },
  "storage-driver": "overlay2"
}
EOF
mkdir -p /etc/systemd/system/docker.service.d
systemctl daemon-reload
systemctl restart docker

cat <<EOF | tee /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
EOF
sysctl --system
apt-get update && apt-get install -y apt-transport-https curl
curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add -
cat <<EOF | tee /etc/apt/sources.list.d/kubernetes.list
deb https://apt.kubernetes.io/ kubernetes-xenial main
EOF
apt-get update
apt-get install -y kubelet=1.16.11-00 kubeadm=1.16.11-00 kubectl=1.16.11-00
mkdir -p /etc/systemd/system/kubelet.service.d/
cat <<EOF | tee /etc/systemd/system/kubelet.service.d/20-hcloud.conf
[Service]
Environment="KUBELET_EXTRA_ARGS=--cloud-provider=external"
EOF
kubeadm init --apiserver-advertise-address=10.0.0.2 --pod-network-cidr=10.244.0.0/16
export KUBECONFIG=/etc/kubernetes/admin.conf
kubectl apply -f https://docs.projectcalico.org/v3.14/manifests/calico.yaml
kubectl taint nodes --all node-role.kubernetes.io/master-
kubectl -n kube-system create secret generic hcloud --from-literal=token=$(cat ~/.htoken) --from-literal=network=my-network
kubectl apply -f https://raw.githubusercontent.com/hetznercloud/hcloud-cloud-controller-manager/master/deploy/v1.6.0-networks.yaml

Verify that cloud controller manager works:

$ kubectl get node -L beta.kubernetes.io/instance-type -L failure-domain.beta.kubernetes.io/region -L failure-domain.beta.kubernetes.io/zone
NAME   STATUS   ROLES    AGE     VERSION    INSTANCE-TYPE   REGION   ZONE
test   Ready    master   2m39s   v1.16.11   cpx21           hel1     hel1-dc2

Create the Service with type LoadBalancer:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: echoserver
spec:
  replicas: 1
  selector:
    matchLabels:
      app: echoserver
  template:
    metadata:
      labels:
        app: echoserver
    spec:
      containers:
      - image: gcr.io/google_containers/echoserver:1.10
        imagePullPolicy: Always
        name: echoserver
        ports:
        - containerPort: 8080
---
apiVersion: v1
kind: Service
metadata:
  name: echoserver
  annotations:
   load-balancer.hetzner.cloud/location: hel1
spec:
  type: LoadBalancer
  ports:
  - port: 80
    targetPort: 8080
    protocol: TCP
  selector:
    app: echoserver

[LKaemmerling]

Masters are not reachable for Load Balancers by design: kubernetes/kubernetes#65618

You could unlabel the node: kubectl label node test node-role.kubernetes.io/master- then it will work.

We found another small bug while reproducing your testcase. You can try:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: echoserver
spec:
  replicas: 1
  selector:
    matchLabels:
      app: echoserver
  template:
    metadata:
      labels:
        app: echoserver
    spec:
      containers:
      - image: gcr.io/google_containers/echoserver:1.10
        imagePullPolicy: Always
        name: echoserver
        ports:
        - containerPort: 8080
---
apiVersion: v1
kind: Service
metadata:
  name: echoserver
  annotations:
   load-balancer.hetzner.cloud/location: hel1
   load-balancer.hetzner.cloud/health-check-interval: 15
spec:
  type: LoadBalancer
  ports:
  - port: 80
    targetPort: 8080
    protocol: TCP
  selector:
    app: echoserver

Until the bugfix release.

[LKaemmerling]

The "bug" i wrote above was fixed with v1.6.1 (which was just released)

[BouweCeunen]

Are there no other implications of removing the label from the master node? This will not cause other things to stop working?

[LKaemmerling]

@BouweCeunen actually this is not a limitation from our side. This is something that comes from the k8s people.

[BouweCeunen]

@BouweCeunen actually this is not a limitation from our side. This is something that comes from the k8s people.

Indeed, just thought that someone would have done it and experienced some side effects. Will take it to the community!

[MatthiasLohr]

Sorry for wakening up this dead cow....

Would it be, for testing issues, acceptable to add an option to hccm which allows to use masters for load balancers if explicitely requested by the one deploying hccm? Removing the master label from the node can have side effects if some applications explicitely require to be scheduled on a master node.

[morsik]

I've just went into this issue, and if anyone still wondering...

In current 1.28 Kubernetes you need to remove node.kubernetes.io/exclude-from-external-load-balancers label if you want your master to be added as target server.

[apricote]

Yes, adding or removing this annotation is the best way to exclude nodes from the load balancer targets.