[solved] Calico doesn't work with minikube

[fasaxc]

@spikecurtis reported that he tried minikube and it didn't seem to work. Felix was faining to program iptables.

Expected Behavior

Should just work.

Current Behavior

iptables failures on the COMMIT line

Possible Solution

Could be a missing kernel module or some incompatibility between kernel and the version of iptables we ship.

Steps to Reproduce (for bugs)

Context

Your Environment

• Calico version
• Orchestrator version (e.g. kubernetes, mesos, rkt):
• Operating System and version:
• Link to your project (optional):

[caseydavenport]

@tmjd has looked into this in the past.

[sbueringer]

Any news on this? We have the same issue and are therefore not able to use Calico on Minikube.

[tmjd]

@sbueringer take a look at #1013 (comment) and the comments below it. A colleague recently used the directions there and was able to run Calico so I believe the directions still work.

[jjacobson93]

It looks like newer minikube versions (v0.27, possibly earlier?) require --extra-config=kubelet.network-plugin.cni now along with --network-plugin=cni (kubernetes/minikube#2828).

[bcreane]

A minikube cluster built with --vmdriver=none flag looks reasonably functional. Further testing required to fully verify.

However any of the other minikube docker drivers are incompatible with Felix. As mentioned in #2016 (comment), the iptables/ipsets that Felix creates on top of a cluster built with one of the kernel docker drivers is substantially incorrect.

In #1013 (comment), @ctaggart mentions that the ability for Calico to function with minikube clusters that use one of the docker drivers (e.g. xhyve-driver) is desirable.

[ctaggart]

@bcreane How are you creating the minikube cluster with --vmdriver=none? I run into problems when I try to create it kubernetes/minikube#2912

[bcreane]

Hi Cameron, I’m back this Monday, will get back to you then. Regards

On Mon, Jun 18, 2018 at 23:30 Cameron Taggart ***@***.***> wrote: @bcreane <https://github.com/bcreane> How are you create the minikube cluster with --vmdriver=none. I run into problems when I try to create it kubernetes/minikube#2912 <kubernetes/minikube#2912> — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1456 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ALg9qr40a4KWw6VC4huq17boOL0lF3APks5t-CnwgaJpZM4Q2WiZ> .

-- <https://tigera.io/> *Brendan* *Creane* Senior Software Engineering Manager Tigera, Inc. brendan@tigera.io | M: +1 510-734-1695 | @bcreane Follow us: Blog <https://blog.tigera.io/> | Twitter <https://twitter.com/tigeraio> | LinkedIn <https://www.linkedin.com/company/tigera/> Secure Application Connectivity for the Cloud Native World <https://tigera.io/>

[bcreane]

Hi @ctaggart - sorry for the long radio silence. I just recreated (and tested with Calico's simple tutorial) the calico+minikube cluster:

# Bring up a minikube cluster. Mine was on an Ubuntu 16.04 VM and using minikube v0.28.2
sudo minikube start --extra-config=kubelet.network-plugin=cni --network-plugin=cni --vm-driver=none

# Install calico-etcd  (likely you could use Kubernetes datastore as well)
kubectl apply -f https://docs.projectcalico.org/v3.2/getting-started/kubernetes/installation/hosted/etcd.yaml

# Create rbac for etcd
kubectl apply -f https://docs.projectcalico.org/v3.2/getting-started/kubernetes/installation/rbac.yaml

# Grab the hosted (etcd) calico manifest
curl https://docs.projectcalico.org/v3.2/getting-started/kubernetes/installation/hosted/calico.yaml -O

# Edit the manifest to make "etcd_endpoints" point to the etcd server you just installed.
sed -i -e "s/10\.96\.232\.137/$(kubectl get service -o json --namespace=kube-system calico-etcd | jq  -r .spec.clusterIP)/" calico.yaml

# Apply your edited calico.yaml
kubectl apply -f calico.yaml

[gflarity]

@bcreane I pasted those commands in verbatum but minikube/calico has issues:

# kubectl get pods --all-namespaces
NAMESPACE     NAME                                       READY     STATUS              RESTARTS   AGE
kube-system   calico-etcd-z7z6k                          1/1       Running             0          2m
kube-system   calico-kube-controllers-5c949c46bf-mc586   0/1       CrashLoopBackOff    3          1m
kube-system   calico-node-fwp7b                          1/2       CrashLoopBackOff    3          1m
kube-system   etcd-minikube                              1/1       Running             0          1m
kube-system   kube-addon-manager-minikube                1/1       Running             6          1m
kube-system   kube-apiserver-minikube                    1/1       Running             0          1m
kube-system   kube-controller-manager-minikube           1/1       Running             0          1m
kube-system   kube-dns-86f4d74b45-8brb5                  0/3       ContainerCreating   0          2m
kube-system   kube-proxy-wkv7b                           1/1       Running             0          2m
kube-system   kube-scheduler-minikube                    1/1       Running             0          1m
kube-system   kubernetes-dashboard-5498ccf677-z55f5      0/1       ContainerCreating   0          2m
kube-system   storage-provisioner                        1/1       Running             0          2m

My host is Redhat EL7...

Linux host 3.10.0-514.10.2.el7.x86_64 #1 SMP Fri Mar 3 00:04:05 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

# minikube version
minikube version: v0.28.2

# kubectl describe pod calico-kube-controllers-5c949c46bf-mc586 --namespace=kube-system
Events:
  Type     Reason                 Age              From               Message
  ----     ------                 ----             ----               -------
  Normal   Scheduled              4m               default-scheduler  Successfully assigned calico-kube-controllers-5c949c46bf-mc586 to minikube
  Normal   SuccessfulMountVolume  4m               kubelet, minikube  MountVolume.SetUp succeeded for volume "etcd-certs"
  Normal   SuccessfulMountVolume  4m               kubelet, minikube  MountVolume.SetUp succeeded for volume "calico-kube-controllers-token-8r9rl"
  Warning  Unhealthy              4m               kubelet, minikube  Readiness probe failed: cannot exec in a stopped state: unknown
  Normal   Pulled                 3m (x4 over 4m)  kubelet, minikube  Container image "quay.io/calico/kube-controllers:v3.2.1" already present on machine
  Normal   Created                3m (x4 over 4m)  kubelet, minikube  Created container
  Normal   Started                3m (x4 over 4m)  kubelet, minikube  Started container
  Warning  Unhealthy              3m (x4 over 4m)  kubelet, minikube  Readiness probe failed: Failed to read status file status.json: open status.json: no such file or directory
  Warning  BackOff                2m (x6 over 4m)  kubelet, minikube  Back-off restarting failed container

# kubectl describe pod calico-node-fwp7b --namespace=kube-system

Events:
  Type     Reason                 Age               From               Message
  ----     ------                 ----              ----               -------
  Normal   Created                6m                kubelet, minikube  Created container
  Normal   SuccessfulMountVolume  6m                kubelet, minikube  MountVolume.SetUp succeeded for volume "cni-bin-dir"
  Normal   SuccessfulMountVolume  6m                kubelet, minikube  MountVolume.SetUp succeeded for volume "lib-modules"
  Normal   SuccessfulMountVolume  6m                kubelet, minikube  MountVolume.SetUp succeeded for volume "var-lib-calico"
  Normal   SuccessfulMountVolume  6m                kubelet, minikube  MountVolume.SetUp succeeded for volume "cni-net-dir"
  Normal   SuccessfulMountVolume  6m                kubelet, minikube  MountVolume.SetUp succeeded for volume "calico-node-token-76s49"
  Normal   SuccessfulMountVolume  6m                kubelet, minikube  MountVolume.SetUp succeeded for volume "etcd-certs"
  Normal   SuccessfulMountVolume  6m                kubelet, minikube  MountVolume.SetUp succeeded for volume "var-run-calico"
  Normal   Pulled                 6m                kubelet, minikube  Container image "quay.io/calico/cni:v3.2.1" already present on machine
  Normal   Started                6m                kubelet, minikube  Started container
  Warning  Unhealthy              6m                kubelet, minikube  Readiness probe errored: rpc error: code = Unknown desc = container not running (99029cb8b4758511e6047eca76183cc54e01270660cfffbb2cfc3192f814d6ad)
  Normal   Created                5m (x3 over 6m)   kubelet, minikube  Created container
  Normal   Started                5m (x3 over 6m)   kubelet, minikube  Started container
  Normal   Pulled                 5m (x3 over 6m)   kubelet, minikube  Container image "quay.io/calico/node:v3.2.1" already present on machine
  Warning  Unhealthy              5m (x3 over 6m)   kubelet, minikube  Readiness probe failed: calico/node is not ready: felix is not ready: Get http://localhost:9099/readiness: dial tcp [::1]:9099: connect: connection refused
  Warning  BackOff                1m (x22 over 6m)  kubelet, minikube  Back-off restarting failed container

# kubectl describe pod kube-dns-86f4d74b45-8brb5 --namespace=kube-system
Events:
  Type     Reason                  Age               From               Message
  ----     ------                  ----              ----               -------
  Normal   Scheduled               8m                default-scheduler  Successfully assigned kube-dns-86f4d74b45-8brb5 to minikube
  Normal   SuccessfulMountVolume   8m                kubelet, minikube  MountVolume.SetUp succeeded for volume "kube-dns-config"
  Normal   SuccessfulMountVolume   8m                kubelet, minikube  MountVolume.SetUp succeeded for volume "kube-dns-token-wdpxb"
  Normal   SandboxChanged          6m (x11 over 8m)  kubelet, minikube  Pod sandbox changed, it will be killed and re-created.
  Warning  FailedCreatePodSandBox  3m (x27 over 8m)  kubelet, minikube  Failed create pod sandbox: rpc error: code = Unknown desc = NetworkPlugin cni failed to set up pod "kube-dns-86f4d74b45-8brb5_kube-system" network: context deadline exceeded

Any ideas?

[bcreane]

@gflarity - anything in the calico/cni and calico/felix logs? Something like:

• kubectl -n kube-system logs calico-node-fwp7b calico-node
• kubectl -n kube-system logs calico-node-fwp7b install-cni

BTW, when you say "verbatum" ... you can't just copy/paste the sed command above - you have to adjust for your calico-etcd IP address.

[gflarity]

@bcreane Thanks, sorry I missed that note :) Things seem better now... Still need to automate this completely though as it's for CI testing...

[bcreane]

@gflarity - no worries - my instructions looked copy/pastable! If you come up with a nice way to automate updating calico.yaml, it'd be nice to see your code on this thread (I'll happily update my comment with your suggestions).

[gflarity]

So it looks like ip in the yaml is accurate for my env... But the following will work if you install the 'jq` tool. There might be a way to do it without jq but this'll do.

# Bring up a minikube cluster. Mine was on an Ubuntu 16.04 VM and using minikube v0.28.2
sudo minikube start --extra-config=kubelet.network-plugin=cni --network-plugin=cni --vm-driver=none

# Install calico-etcd  (likely you could use Kubernetes datastore as well)
kubectl apply -f https://docs.projectcalico.org/v3.2/getting-started/kubernetes/installation/hosted/etcd.yaml

# Create rbac for etcd
kubectl apply -f https://docs.projectcalico.org/v3.2/getting-started/kubernetes/installation/rbac.yaml

# Grab the hosted (etcd) calico manifest
curl https://docs.projectcalico.org/v3.2/getting-started/kubernetes/installation/hosted/calico.yaml -O

# Edit the manifest to make "etcd_endpoints" point to the etcd server you just installed.
# NOTE - this command needs to be adjusted with your etcd endpoint IP address
sed -i -e "s/10\.96\.232\.136/$(kubectl get service -o json --namespace=kube-system calico-etcd | jq  -r .spec.clusterIP)/" calico.yaml

# Apply your edited calico.yaml
kubectl apply -f calico.yaml

[bcreane]

@gflarity - nice use of jq! I'll modify my instructions, thanks.

[gflarity]

FYI I tried the above on OS X with hyperkit and it seems to be working there as well. Yay!

  minikube start --vm-driver=hyperkit  --network-plugin=cni --extra-config=kubelet.network-plugin=cni

[bcreane]

Going to mark this issue "resolved" - both @gflarity and I have created functioning minikube k8s/Calico clusters on a couple of deployments. See #1456 (comment) for instructions.

[gflarity]

👍 Just as an update this has been working successfully in our CI flow for ~14 days now.

[tradel]

FYI @gflarity, you can eliminate jq from the subcommand just by doing:

kubectl get service -o jsonpath='{.spec.clusterIP}' --namespace=kube-system calico-etcd