Replace minikube for e2e tests #2863

aledbf · 2018-07-28T00:40:55Z

What this PR does / why we need it:

Replaces minikube with https://github.com/kubernetes-sigs/kubeadm-dind-cluster to build and run e2e tests.
Also, we update the slow threshold from 5 seconds to 40. This value is more real and can help to detect slow tests.

aledbf · 2018-07-28T00:43:48Z

@antoineco this approach can help us to run the tests in different cloud providers like gce

aledbf · 2018-07-28T04:25:06Z

With this change, we can also skip minikube locally running ./test/e2e/up.sh to start a dind cluster.

codecov-io · 2018-07-28T04:25:44Z

Codecov Report

Merging #2863 into master will increase coverage by 0.12%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master    #2863      +/-   ##
==========================================
+ Coverage   47.71%   47.84%   +0.12%     
==========================================
  Files          75       75              
  Lines        5434     5434              
==========================================
+ Hits         2593     2600       +7     
+ Misses       2510     2504       -6     
+ Partials      331      330       -1

Impacted Files	Coverage Δ
cmd/nginx/main.go	`11.18% <0%> (+4.6%)`	⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 9e2cc65...7032fb7. Read the comment docs.

antoineco · 2018-07-28T05:40:59Z

hack/boilerplate/boilerplate.py

+    "vendor",
+    "test/e2e/framework/framework.go",
+    "test/e2e/generated/bindata.go",
+    "hack/boilerplate/test",


Bindata files are gone.

antoineco · 2018-07-28T05:42:13Z

Makefile

@@ -26,7 +26,9 @@ GOHOSTOS ?= $(shell go env GOHOSTOS)
 # Allow limiting the scope of the e2e tests. By default run everything
 FOCUS ?= .*
 # number of parallel test
-E2E_NODES ?= 3
+E2E_NODES ?= 4


?= is evil, it allows for empty strings. := is safer.

most of the variables could be replaced by env variables, using := avoids that. Also, the scripts we call check for the required variables

:= should handle env vars as well. What version of make do you have installed?

$ make -v
GNU Make 4.1
Built for x86_64-pc-linux-gnu
Copyright (C) 1988-2014 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later http://gnu.org/licenses/gpl.html
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

antoineco · 2018-07-28T06:13:14Z

build/e2e-tests.sh

@@ -34,6 +34,10 @@ if [ -z "${NODE_IP}" ]; then
    echo "NODE_IP must be set"
    exit 1
 fi
+if [ -z "${SLOW_E2E_THRESHOLD}" ]; then


Suggestion:

declare -a mandatory mandatory=( NODE_IP SLOW_E2E_THRESHOLD ... ) for m in ${mandatory[@]}; do if [ -z "${!m}" ]; then echo "$m must be set" declare missing fi done if [ -n $missing ]; then exit 1 fi

Would show all missing variables at once.

ElvinEfendi · 2018-07-28T12:51:08Z

build/go-in-docker.sh

@@ -63,6 +63,7 @@ PWD=${PWD}
 BUSTED_ARGS=${BUSTED_ARGS:-""}
 REPO_INFO=${REPO_INFO:-local}
 NODE_IP=${NODE_IP:-127.0.0.1}
+SLOW_E2E_THRESHOLD=${SLOW_E2E_THRESHOLD:-50}


any reason why is this 50 here but 40 in the Makefile?

ElvinEfendi · 2018-07-28T13:06:04Z

@aledbf could you describe with a few words why is this new tool better than minikube in our case? (I'm not experienced in either of them)

antoineco · 2018-07-28T13:12:59Z

I can comment here: I have been using Docker for Mac for a while so I appreciate having batteries included (local dev script) but swappable (I can run e2e in my own setup easily without seeing failures when minikube is absent).

Also minikube is tough to run in unprivileged environments (most CI systems I use don't allow sudo).

aledbf · 2018-07-28T14:35:57Z

Also, this approach provides a more real cluster, one master, and two workers use just docker and don't require a VM, which means no Virtualbox or QEMU.

aledbf · 2018-07-28T14:36:44Z

why is this new tool better than minikube in our case

Not sure (yet) if it's better or not but for sure it's simpler.

antoineco · 2018-07-28T15:01:54Z

build/build.sh

+  fi
+done
+
+if $missing ; then


Bash doesn't know about booleans, does it? This may always match.

antoineco · 2018-07-28T15:34:35Z

/lgtm

k8s-ci-robot · 2018-07-28T15:34:41Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: aledbf, antoineco

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [aledbf,antoineco]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

k8s-ci-robot added cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. approved Indicates a PR has been approved by an approver from all required OWNERS files. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. labels Jul 28, 2018

aledbf added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jul 28, 2018

aledbf force-pushed the replace-minikube branch 10 times, most recently from 0e3e171 to 80ee721 Compare July 28, 2018 03:49

aledbf force-pushed the replace-minikube branch from 80ee721 to 352a6d9 Compare July 28, 2018 04:26

aledbf assigned ElvinEfendi and antoineco Jul 28, 2018

aledbf removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jul 28, 2018

antoineco reviewed Jul 28, 2018

View reviewed changes

aledbf force-pushed the replace-minikube branch 3 times, most recently from 015bca3 to 1fd6ea0 Compare July 28, 2018 12:47

ElvinEfendi reviewed Jul 28, 2018

View reviewed changes

aledbf force-pushed the replace-minikube branch from 1fd6ea0 to f516dff Compare July 28, 2018 14:32

antoineco reviewed Jul 28, 2018

View reviewed changes

Replace minikube for e2e tests

7032fb7

aledbf force-pushed the replace-minikube branch from f516dff to 7032fb7 Compare July 28, 2018 15:05

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Jul 28, 2018

k8s-ci-robot merged commit 45ba167 into kubernetes:master Jul 28, 2018

aledbf deleted the replace-minikube branch July 28, 2018 15:40

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Replace minikube for e2e tests #2863

Replace minikube for e2e tests #2863

aledbf commented Jul 28, 2018

aledbf commented Jul 28, 2018

aledbf commented Jul 28, 2018

codecov-io commented Jul 28, 2018 •

edited

Loading

antoineco Jul 28, 2018 •

edited

Loading

aledbf Jul 28, 2018

antoineco Jul 28, 2018

aledbf Jul 28, 2018

antoineco Jul 28, 2018

aledbf Jul 28, 2018

antoineco Jul 28, 2018 •

edited

Loading

ElvinEfendi Jul 28, 2018

ElvinEfendi commented Jul 28, 2018

antoineco commented Jul 28, 2018

aledbf commented Jul 28, 2018

aledbf commented Jul 28, 2018

antoineco Jul 28, 2018

aledbf Jul 28, 2018

antoineco commented Jul 28, 2018

k8s-ci-robot commented Jul 28, 2018

Replace minikube for e2e tests #2863

Replace minikube for e2e tests #2863

Conversation

aledbf commented Jul 28, 2018

aledbf commented Jul 28, 2018

aledbf commented Jul 28, 2018

codecov-io commented Jul 28, 2018 • edited Loading

Codecov Report

antoineco Jul 28, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

antoineco Jul 28, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ElvinEfendi commented Jul 28, 2018

antoineco commented Jul 28, 2018

aledbf commented Jul 28, 2018

aledbf commented Jul 28, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

antoineco commented Jul 28, 2018

k8s-ci-robot commented Jul 28, 2018

codecov-io commented Jul 28, 2018 •

edited

Loading

antoineco Jul 28, 2018 •

edited

Loading

antoineco Jul 28, 2018 •

edited

Loading