[solved] Calico doesn't start correctly on minikube

[ceridwen]

Expected Behavior

Installing Calico on minikube should start both containers in the calico-node pod.

Current Behavior

I recently upgraded minikube to 0.21, which means deleting the cluster and reinstalling it. When I tried to reinstall Calico, the calico-node container in the calico node pod never comes up, and networking is broken.

I'm running minikube with minikube start --network-plugin=cni --host-only-cidr=192.168.0.0/16, using virtualbox as the driver.

The logs for calico-node contain entries like:

2017-08-10 00:28:51.825 [PANIC][2269] ipsets.go 283: Failed to update IP sets after mutliple retries. family="inet"
panic: (*logrus.Entry) (0x16df120,0xc4204744b0)
github.com/projectcalico/felix/vendor/github.com/sirupsen/logrus.(*Entry).Panic(0xc42041a870, 0xc4208d5ee0, 0x1, 0x1)

2017-08-10 00:29:15.131 [WARNING][2972] ipsets.go 271: Failed to update IP sets. Marking dataplane for resync. error=exit status 1 family="inet"
2017-08-10 00:29:15.270 [WARNING][2972] ipsets.go 621: Failed to complete ipset restore, IP sets may be out-of-sync. closeErr=<nil> commitErr=<nil> family="inet" flushErr=<nil> input="create cali4tall-hosts hash:ip family inet maxelem 1048576\nadd cali4tall-hosts 10.1.0.1\nswap cali4-all-hosts cali4tall-hosts\ndestroy cali4tall-hosts\ncreate cali4-all-ipam-pools hash:net family inet maxelem 1048576\ncreate cali4tall-ipam-pools hash:net family inet maxelem 1048576\nadd cali4tall-ipam-pools 192.168.0.0/16\nswap cali4-all-ipam-pools cali4tall-ipam-pools\ndestroy cali4tall-ipam-pools\ncreate cali4-masq-ipam-pools hash:net family inet maxelem 1048576\ncreate cali4tmasq-ipam-pools hash:net family inet maxelem 1048576\nadd cali4tmasq-ipam-pools 192.168.0.0/16\nswap cali4-masq-ipam-pools cali4tmasq-ipam-pools\ndestroy cali4tmasq-ipam-pools\nCOMMIT\n" processErr=exit status 1 stderr="ipset v6.32: Error in line 5: Kernel error received: set type not supported\n" stdout="" writeErr=<nil>
2017-08-10 00:29:15.270 [WARNING][2972] ipsets.go 271: Failed to update IP sets. Marking dataplane for resync. error=exit status 1 family="inet"

Steps to Reproduce (for bugs)

1. Install minikube with homebrew (https://github.com/kubernetes/minikube).
2. Install Virtualbox (https://www.virtualbox.org/wiki/Downloads).

3. Start minikube with minikube start --network-plugin=cni --host-only-cidr=192.168.0.0/16
4. Install calico with kubectl apply -f https://docs.projectcalico.org/v2.4/getting-started/kubernetes/installation/hosted/kubernetes-datastore/calico-networking/1.6/calico.yaml

Context

I'm trying to create a working development environment for testing network policy on my local machine so I don't have to depend on a remote cluster for doing development work. I know this is beta software, but it was working so I suspect that a change in Calico or minikube is responsible.

Your Environment

Calico: v2.4
minikube: v0.21.0, Kubernetes version 1.7.0
OS: Mac OS X 10.12.6

[caseydavenport]

@ceridwen thanks for the report - I believe you're encountering this issue: kubernetes/minikube#1779

I think this is essentially an issue with minikube not including the necessary kernel features for Calico to run. However, looking at the upstream issue there appears to be some confusion as to whether or not it should be fixed, so I'll leave this issue open until we can sort it out.

[erasmus74]

What do you mean? What could be the debate? ipset is fairly important for a lot of basic functions and calico is a very important component of many kubernetes applications.

Not trying to sound offensive, just really confused on why that could be.

[tmjd]

I think he was referring to the conversation in this issue kubernetes/minikube#1697
That issue shows that ipset was enabled in the kernel, which from looking at commit history it should be, but the latest release does not seem to have it configured from my testing. Though either way I think the ipset command itself is also missing so I'm working on adding that (and fixing the kernel having it enabled if it is not properly, it is taking a while to build the iso or I'd have that answer already).

[erasmus74]

Thank you so much.

[tmjd]

This was fixed with kubernetes/minikube#1817 and requires using a new minikube.iso. Until the next version of minikube is released (current is 0.21.0), the one built off that PR https://storage.googleapis.com/minikube-builds/1817/minikube.iso can be used when doing minikube start. The option --iso-url https://storage.googleapis.com/minikube-builds/1817/minikube.iso can be used to pass the iso to use.

I'm going to close this issue for now since I believe it is resolved. Please comment or reopen if you do not agree.

[sbueringer]

I verified it today with the new iso, it works.

[javsalgar]

Is this --host-only-cidr=192.168.0.0/16 necessary for making the calico plugin work?

[gtirloni]

minikube start --network-plugin=cni --host-only-cidr "192.168.0.0/16"

E0817 08:12:10.863999   11261 start.go:132] Error starting host: Error creating host: Error executing step: Creating VM.
: Error setting up host only network on machine start: host-only cidr must be specified with a host address, not a network address.

minikube start --network-plugin=cni --host-only-cidr "192.168.0.1/16"

E0817 08:12:45.430440   11649 start.go:132] Error starting host: Error creating host: Error executing step: Creating VM.
: Error setting up host only network on machine start: host-only cidr conflicts with the network address of a host interface.

minikube start --network-plugin=cni (calico datastore)

Calico (datastore) errors
minikube_cni_nohostcidr_calico_logs.txt

minikube start --network-plugin=cni --host-only-cidr "192.168.99.1/24" (existing vboxnet0 address) (calico datastore):

Same calico (datastore) errors as with no --host-only-cidr

minikube start --network-plugin=cni (calico hosted install)

$ kubectl apply -f https://docs.projectcalico.org/v2.4/getting-started/kubernetes/installation/hosted/kubeadm/1.6/calico.yaml
configmap "calico-config" created
daemonset "calico-etcd" created
daemonset "calico-node" created
deployment "calico-policy-controller" created
clusterrolebinding "calico-cni-plugin" created
clusterrole "calico-cni-plugin" created
serviceaccount "calico-cni-plugin" created
clusterrolebinding "calico-policy-controller" created
clusterrole "calico-policy-controller" created
serviceaccount "calico-policy-controller" created
The Service "calico-etcd" is invalid: spec.clusterIP: Invalid value: "10.96.232.136": provided IP is not in the valid range

[sbueringer]

I'm using Calico with (not sure what's exactly necessary):

    --network-plugin=cni \
    --kubernetes-version=v1.6.4 \
    --extra-config=controller-manager.AllocateNodeCIDRs=true \
    --extra-config=controller-manager.ClusterCIDR=192.168.0.0/16 \
    --extra-config=kubelet.PodCIDR=10.180.1.0/24 \
    --iso-url=https://storage.googleapis.com/minikube-builds/1817/minikube.iso "

with https://docs.projectcalico.org/v2.4/getting-started/kubernetes/installation/hosted/kubernetes-datastore/policy-only/1.6/calico.yaml

[gtirloni]

@sbueringer unfortunately that configuration causes localkube to go bananas here (tested 1.6.4 and 1.7.3). I'm using minukube compiled from HEAD, VirtualBox 5.1.26 on Fedora 26.

minikube_sbueringer_v173.txt

minikube logs:

Aug 17 12:10:07 minikube localkube[5062]: Waiting for kubelet to be healthy...
Aug 17 12:10:07 minikube localkube[5062]: I0817 12:10:07.560393    5062 feature_gate.go:144] feature gates: map[]
Aug 17 12:10:07 minikube localkube[5062]: E0817 12:10:07.560921    5062 server.go:318] unable to register configz: register config "componentconfig" twice
Aug 17 12:10:07 minikube localkube[5062]: I0817 12:10:07.572268    5062 client.go:72] Connecting to docker on unix:///var/run/docker.sock
Aug 17 12:10:07 minikube localkube[5062]: I0817 12:10:07.572706    5062 client.go:92] Start docker client with request timeout=2m0s
Aug 17 12:10:07 minikube localkube[5062]: fatal error: unexpected signal during runtime execution
Aug 17 12:10:07 minikube localkube[5062]: [signal SIGSEGV: segmentation violation code=0x1 addr=0xe5 pc=0x7f4f1108e100]
Aug 17 12:10:07 minikube localkube[5062]: runtime stack:
Aug 17 12:10:07 minikube localkube[5062]: runtime.throw(0x4aa306c, 0x2a)
Aug 17 12:10:07 minikube localkube[5062]:         /usr/local/go/src/runtime/panic.go:596 +0x95

$ while [ true ]; do kubectl get nodes; sleep 1; done
The connection to the server 192.168.99.100:8443 was refused - did you specify the right host or port?
The connection to the server 192.168.99.100:8443 was refused - did you specify the right host or port?
The connection to the server 192.168.99.100:8443 was refused - did you specify the right host or port?
The connection to the server 192.168.99.100:8443 was refused - did you specify the right host or port?
The connection to the server 192.168.99.100:8443 was refused - did you specify the right host or port?
NAME       STATUS    AGE       VERSION
minikube   Ready     1m        v1.7.3
NAME       STATUS    AGE       VERSION
minikube   Ready     1m        v1.7.3
NAME       STATUS    AGE       VERSION
minikube   Ready     1m        v1.7.3
The connection to the server 192.168.99.100:8443 was refused - did you specify the right host or port?
The connection to the server 192.168.99.100:8443 was refused - did you specify the right host or port?
The connection to the server 192.168.99.100:8443 was refused - did you specify the right host or port?
The connection to the server 192.168.99.100:8443 was refused - did you specify the right host or port?
The connection to the server 192.168.99.100:8443 was refused - did you specify the right host or port?
The connection to the server 192.168.99.100:8443 was refused - did you specify the right host or port?
The connection to the server 192.168.99.100:8443 was refused - did you specify the right host or port?
NAME       STATUS    AGE       VERSION
minikube   Ready     1m        v1.7.3
NAME       STATUS    AGE       VERSION
minikube   Ready     1m        v1.7.3
NAME       STATUS    AGE       VERSION
minikube   Ready     1m        v1.7.3
The connection to the server 192.168.99.100:8443 was refused - did you specify the right host or port?
The connection to the server 192.168.99.100:8443 was refused - did you specify the right host or port?
The connection to the server 192.168.99.100:8443 was refused - did you specify the right host or port?
The connection to the server 192.168.99.100:8443 was refused - did you specify the right host or port?
The connection to the server 192.168.99.100:8443 was refused - did you specify the right host or port?
The connection to the server 192.168.99.100:8443 was refused - did you specify the right host or port?

[tmjd]

@gtirloni You might need to add a --host-only-cidr 172.17.17.1/24 to the args @sbueringer has suggested. The --host-only-cidr is the address and CIDR that will be used for the vboxnet interface on your host machine that will be connected to the minikube VM (that's the behavior I see).
I'm suggesting you need that because I'm assuming your kubectl get nodes is running on your host and if it is trying to hit the VM at 192.168.99.100 and you have a ClusterCIDR of 192.168.0.0/16, I would expect things to not play nicely.

TBH, I'm not sure that what I've described is your problem though, unless those connection refused messages line up with those localkube/kubelet panics. You don't happen to have multiple VMs running that have that 192.168.99.100 address do you? That kind of intermittent behavior makes me think of issues I've had when I've tried to bring up multiple instances of vagrant machines all trying to use the same IP address.

[gtirloni]

@tmjd specifying --host-only-cidr here means the VM gets a new vboxnet1 interface (instead of using the existing vboxnet0 with 192.168.99.0/24) but the issue is the same as not specifying --host-only-cidr:

$ minikube start --network-plugin=cni --host-only-cidr "172.17.17.1/24"
Starting local Kubernetes v1.7.3 cluster...
Starting VM...
Getting VM IP address...
Moving files into cluster...
Setting up certs...
Starting cluster components...
Connecting to cluster...
Setting up kubeconfig...
Kubectl is now configured to use the cluster.

$ kubectl apply -f https://docs.projectcalico.org/v2.4/getting-started/kubernetes/installation/hosted/kubeadm/1.6/calico.yaml
configmap "calico-config" created
daemonset "calico-etcd" created
daemonset "calico-node" created
deployment "calico-policy-controller" created
clusterrolebinding "calico-cni-plugin" created
clusterrole "calico-cni-plugin" created
serviceaccount "calico-cni-plugin" created
clusterrolebinding "calico-policy-controller" created
clusterrole "calico-policy-controller" created
serviceaccount "calico-policy-controller" created
The Service "calico-etcd" is invalid: spec.clusterIP: Invalid value: "10.96.232.136": provided IP is not in the valid range

I don't have multiple VMs running, only minikube's. I'm not sure this is an issue in minikube or calico.

[tmjd]

@gtirloni Can you try adding --extra-config=controller-manager.ServiceCIDR=10.96.0.0/16? The error you are seeing indicates is that the etcd service address that the manifest specifies does not fall into the service CIDR. Another option would be to download the manifest and update that address to be within the service CIDR, though I'm not sure how to find out the serviceCIDR.

[javsalgar]

I executed it with these options:

minikube start --network-plugin=cni --host-only-cidr 172.17.17.1/24 
--extra-config=controller-manager.ServiceCIDR=10.96.0.0/16 
 --extra-config=controller-manager.AllocateNodeCIDRs=true     
--extra-config=controller-manager.ClusterCIDR=192.168.0.0/16   
--extra-config=controller-manager.ServiceCIDR=10.96.0.0/16   
--extra-config=kubelet.PodCIDR=10.180.1.0/24     
--iso-url=https://storage.googleapis.com/minikube-builds/1817/minikube.iso

And got the same issues shown here: #1013 (comment)

[tmjd]

Could you try just this?

./minikube start --network-plugin=cni --host-only-cidr 172.17.17.1/24 \
    --iso-url=https://storage.googleapis.com/minikube-builds/1817/minikube.iso

You should then be able to deploy calico with
kubectl apply -f https://docs.projectcalico.org/v2.4/getting-started/kubernetes/installation/hosted/kubernetes-datastore/calico-networking/1.6/calico.yaml
This will deploy calico using kubernetes as the datastore.

If you want to use with etcd as the datastore, it is possible to deploy with a few slight modifications to this one https://docs.projectcalico.org/v2.4/getting-started/kubernetes/installation/hosted/kubeadm/1.6/calico.yaml.

• Change the 10.96.232.136 addresses to 10.0.0.136, minikube seems to use a different serviceIP range (I tried changing it but then a couple pods minikube starts, like dns, fail)
• Remove the nodeSelector from the calico-etcd daemonset, this is fine since with minikube there is only one node.

[beeradb]

I was trying to get calico working on minikube today as well, and was having issues and tried out the instructions in this PR. Fair warning, I've never successfully installed calico or used it before, so I'll default to being overly verbose about my experience. Hopefully, this is valuable.

I ran the commands listed in #1013 (comment) and had the following results

• Minikube started as expected
• Calico installed without warnings. The install-cni pod reported "Done configuring CNI" and the calico-node container reports the folowing at the end of startup:

bird: Unable to open configuration file /etc/calico/confd/config/bird.cfg: No such file or directory
2017-08-18 20:16:07.225 [INFO][86] int_dataplane.go 674: Applying dataplane updates
2017-08-18 20:16:07.225 [INFO][86] ipsets.go 224: Asked to resync with the dataplane on next update. family="inet"
2017-08-18 20:16:07.225 [INFO][86] ipsets.go 255: Resyncing ipsets with dataplane. family="inet"
2017-08-18 20:16:07.226 [INFO][86] ipsets.go 297: Finished resync family="inet" numInconsistenciesFound=0 resyncDuration=893.732µs
2017-08-18 20:16:07.226 [INFO][86] int_dataplane.go 689: Finished applying updates to dataplane. msecToApply=1.237469

I think everything was looking good at this point. No big flashing errors, other than the "Unable to open configuration file" in the above output.

I then added a simple nginx-deployment from https://kubernetes.io/docs/concepts/workloads/controllers/deployment/#creating-a-deployment and observed the following results:

• Pods were either reporting 'Error syncing pod" or the more descriptive "Error response from daemon: {"message":"cannot join network of a non running container: 1ffeab283933bb7ef86b086bbd15c648bdf7d7fb88bb1359265e87cd746ae921""

Hopefully, this helps. I'm happy to provide any other follow-up info I can.

[tmjd]

@beeradb Did you do kubectl apply as the command i listed or did you download the 2nd yaml and make the changes I suggested? (I hope you didn't do both, I can imagine that not working.)

[beeradb]

@tmjd I did kubectl apply with the first yaml file listed. No changes applied.

I also tried this both on xhyve and virtualbox with the same results. Not sure if the virtualization method actually matters here, though.

[beeradb]

I believe there may still be some inconsistency in minikube here.

I was able to re-run using the minikube start command listed here: #1013 (comment) and had positive results.

• Minikube started without error
• Calico installed without error
• The nginx deployment pods came up without issue.

However, when removing the --kubernetes-version=v1.6.4 flag (and thus defaulting to v1.7.0) the kube-system namespace ends up being empty for everything but the "kube-addon-manager-minikube" pod.

So, to recap:

• [solved] Calico doesn't start correctly on minikube #1013 (comment) does not work.
• [solved] Calico doesn't start correctly on minikube #1013 (comment) works with v1.6.4
• [solved] Calico doesn't start correctly on minikube #1013 (comment) does not work with v1.7.0. The Minikube API server does not even start correctly.

This was all tested on the following: minikube version: v0.21.0

[tmjd]

@beeradb Thank you for your report on what worked or didn't for you. I've been spending some time trying to understand this and have figured out some options to make Calico work with minikube. There are 2 ways to run Calico policy-only mode and networking (which includes policy) mode. I'm going go through using both here, if you (or anyone) are able to try them out please report back your results.

Note: In both set ups below the pods that are deployed as minikube addons will be brought up with the default networking that minikube sets up. This does not seem to hinder much though because of that it is not possible to create policy that will apply to the addon pods. I attempted to restart the addon pods so they would be networked with the newly configured networking but for some reason after that I could not reach the dns.

Policy-only mode (Kubernetes networking)

In this mode Calico is not setting up the networking but uses the 'host-local' plugin for IPAM so depends on K8s for IPAM. The minikube start command I found to work is:

Minikube v0.21

minikube start --network-plugin=cni --host-only-cidr 172.17.17.1/24 \
    --extra-config=kubelet.ClusterCIDR=192.168.0.0/16 \
    --extra-config=proxy.ClusterCIDR=192.168.0.0/16 \
    --extra-config=controller-manager.ClusterCIDR=192.168.0.0/16 \
    --extra-config=controller-manager.CIDRAllocatorType=RangeAllocator \
    --extra-config=controller-manager.AllocateNodeCIDRs=true \
    --iso-url=https://storage.googleapis.com/minikube-builds/1817/minikube.iso

Minikube v0.22

For minikube v0.22.1 it seems like the options have changed, I found that the following worked with it:
(also notice the --iso-url is not needed with v0.22.1)

minikube start --network-plugin=cni --host-only-cidr 172.17.17.1/24 \
    --extra-config=kubelet.PodCIDR=192.168.0.0/16 \
    --extra-config=proxy.ClusterCIDR=192.168.0.0/16 \
    --extra-config=controller-manager.ClusterCIDR=192.168.0.0/16 \
    --extra-config=controller-manager.CIDRAllocatorType=RangeAllocator \
    --extra-config=controller-manager.AllocateNodeCIDRs=true

I found that without CIDRAllocatorType or ClusterCIDR set on the controller-manager you would not have a working api-server. (@beeradb, I think this is what you saw when you didn't specify kubernetes 1.6.)

After K8s is started with the above then it was possible to install calico with kubectl apply -f https://docs.projectcalico.org/v2.4/getting-started/kubernetes/installation/hosted/kubernetes-datastore/calico-networking/1.6/calico.yaml

Note: I'm not sure why but I had to specify the ClusterCIDR, the default would not work. I also tried specifying the default CIDR to the controller-manager but then I would end up with multiple pods with the same IP addresses.

Calico networking mode

In this mode Calico handles configuring the networking and provides policy.

Minikube v0.21

minikube start --network-plugin=cni --host-only-cidr 172.17.17.1/24 \
    --extra-config=kubelet.ClusterCIDR=192.168.0.0/16 \
    --extra-config=proxy.ClusterCIDR=192.168.0.0/16 \
    --extra-config=controller-manager.ClusterCIDR=192.168.0.0/16 \
    --iso-url=https://storage.googleapis.com/minikube-builds/1817/minikube.iso

Minikube v0.22

minikube start --network-plugin=cni --host-only-cidr 172.17.17.1/24 \
    --extra-config=kubelet.PodCIDR=192.168.0.0/16 \
    --extra-config=proxy.ClusterCIDR=192.168.0.0/16 \
    --extra-config=controller-manager.ClusterCIDR=192.168.0.0/16

After K8s is started with the above then it is possible to install Calico though some slight modifications will need to be made to the manifests. The following commands can be used to pull down a manifest, update it, and then apply the manifest.

curl -O -L https://docs.projectcalico.org/v2.4/getting-started/kubernetes/installation/hosted/kubeadm/1.6/calico.yaml
sed -i -e '/nodeSelector/d' calico.yaml
sed -i -e '/node-role.kubernetes.io\/master: ""/d' calico.yaml
sed -i -e 's/10\.96\.232/10.0.0/' calico.yaml
kubectl apply -f calico.yaml

[beeradb]

Thanks so much for the thorough response. I'll try to carve out some time this afternoon to test this and report back.

[beeradb]

I spent some time trying these out today and both versions worked as expected for me. I did not do a thorough test, but in both cases, I introduced a default-deny policy into the cluster and had networking restricted.

Thanks again for your work on resolving this.

[pksec]

Thanks!!! @tmjd .

I tested your configuration with minikube:v0.23 and it works, thanks a lot. You saved me !

Calico networking mode

Minikube v0.23

minikube start --network-plugin=cni --host-only-cidr 172.17.17.1/24 \
    --extra-config=kubelet.PodCIDR=192.168.0.0/16 \
    --extra-config=proxy.ClusterCIDR=192.168.0.0/16 \
    --extra-config=controller-manager.ClusterCIDR=192.168.0.0/16

I was having issues with kubeadm hosted + Calico install (pls see issue: #1337. ) I think CIDR configuration with kubeadm is the problem there.

[taylorKonigsmark]

@tmjd does this work version 0.24.1?

I'm getting this message when I apply the modified calico.yaml:

The Service "calico-etcd" is invalid: spec.clusterIP: Invalid value: "10.0.0.136": provided IP is not in the valid range. The range of valid IPs is 10.96.0.0/12z

I'm guessing the sed command isn't working for me running on a Mac. what is exactly trying to be changed in the file?

[tmjd]

@taylorKonigsmark I have not tried 0.24.1. It looks like maybe minikube has changed the service CIDR they use. I think you should try again but not do sed -i -e 's/10\.96\.232/10.0.0/' calico.yaml.

What it was attempting to do was put the service IP created for the calico-etcd service into the used service CIDR.

[taylorKonigsmark]

Thank you @tmjd! That helped a lot!

[ctaggart]

Thanks everyone for this thread. I just got Calico v2.4 running on Minikube 2.7 following the above comments.

my environment

I'm using a MacBook Pro with the latest stable Docker for Mac 18.03.1-ci-mac65 installed. It comes with hyperkit v0.20171204-60-g0e5b6b. I installed kubectl with brew install kubectl and it is v1.10.2.

install Minikube

localkube bootstrapper is used instead of the kubeadm default because --extra-config=proxy does not work kubernetes/minikube#2259

minikube start \
  --bootstrapper=localkube \
  --vm-driver=hyperkit \
  --network-plugin=cni \
  --host-only-cidr=172.17.17.1/24 \
  --extra-config=kubelet.network-plugin=cni \
  --extra-config=kubelet.PodCIDR=192.168.0.0/16 \
  --extra-config=proxy.ClusterCIDR=192.168.0.0/16 \
  --extra-config=controller-manager.ClusterCIDR=192.168.0.0/16

install Callico v2.4

This downloads and adjusts the yaml before applying it. It removes the nodeSelector filter. Is it possible to instead make it so that the node is the master node?

curl -O -L https://docs.projectcalico.org/v2.4/getting-started/kubernetes/installation/hosted/kubeadm/1.6/calico.yaml
sed -i -e '/nodeSelector/d' calico.yaml
sed -i -e '/node-role.kubernetes.io\/master: ""/d' calico.yaml
kubectl apply -f calico.yaml

Here is the resulting diff:

Here are all the pods running:

Camerons-MacBook-Pro:calico cameron$ kubectl get pods --all-namespaces
NAMESPACE     NAME                                        READY     STATUS    RESTARTS   AGE
kube-system   calico-etcd-6dkwn                           1/1       Running   0          38m
kube-system   calico-node-xdtjp                           2/2       Running   0          38m
kube-system   calico-policy-controller-6f47884d6f-zkrzv   1/1       Running   0          38m
kube-system   kube-addon-manager-minikube                 1/1       Running   0          51m
kube-system   kube-dns-6dcb57bcc8-bffns                   3/3       Running   0          50m
kube-system   kubernetes-dashboard-5498ccf677-tdmm9       1/1       Running   0          50m
kube-system   storage-provisioner                         1/1       Running   0          50m

I haven't tried with a newer Calico yet, but I'm about to.

reset environment

I'm new to this. Here is what has been working for me to reset my environment.

minikube stop
minikube delete
rm -rf ~/.minikube
sudo rm /var/db/dhcpd_leases

[ctaggart]

This works fine with Calico v3.1 as well.
https://docs.projectcalico.org/v3.1/getting-started/kubernetes/

install Calico v3.1

curl -O -L https://docs.projectcalico.org/v3.1/getting-started/kubernetes/installation/hosted/kubeadm/1.7/calico.yaml
sed -i -e '/nodeSelector/d' calico.yaml
sed -i -e '/node-role.kubernetes.io\/master: ""/d' calico.yaml
kubectl apply -f calico.yaml

Camerons-MacBook-Pro:calico cameron$ kubectl get pods --all-namespaces
NAMESPACE     NAME                                       READY     STATUS    RESTARTS   AGE
kube-system   calico-etcd-krb9d                          1/1       Running   0          18m
kube-system   calico-kube-controllers-685755779f-smwrs   1/1       Running   0          18m
kube-system   calico-node-4cjzb                          2/2       Running   0          18m
kube-system   kube-addon-manager-minikube                1/1       Running   0          22m
kube-system   kube-dns-6dcb57bcc8-pwswp                  3/3       Running   0          22m
kube-system   kubernetes-dashboard-5498ccf677-tvj4t      1/1       Running   0          22m
kube-system   storage-provisioner                        1/1       Running   0          22m

[bcreane]

Quick testing indicates that only --vm-driver=none will work with Calico. I tried out hyperkit and observed that Calico (specifically Felix) was not correctly populating iptables / ipsets rules. I believe this limits Calico+minikube to Linux only.

[ctaggart]

@bcreane thanks for digging in to this. Is there a bug for Felix not correcting populating iptables / ipsets rules when on hyperkit + minikube? It would be great to have that working for dev from a Mac, which is really common.

[bcreane]

@ctaggart - I updated #1456 which is tracking Calico/Felix interoperability with minikube clusters to call out minikube docker driver support.