fix ingress (also for multinode clusters) #13439

prezha · 2022-01-22T23:36:39Z

more specifically, with this pr we should:

align structure and (mostly!) content of deploy/addons/ingress/ingress-deploy.yaml.tmpl with the reference kind and baremetal upstream deployments - this should allow us to more easily keep up with the upstream changes; here there's a slight difference in nginx-ingress-controller cmd params - like leaving out kind's --publish-status-address=localhost to publish node's actual ip instead, having is-default-class set, additional ConfigMaps for tcp/udp-services, gcp-auth secrets handling, etc. but also:
(hopefully still) maintain backwards compatibility with the last 6 minor k8s versions (currently: v1.18+)
include suggestion @alexbaeza made in his pr Fix ingress-controller pod rolling updates #12904 (also in line with the reference deploy.yaml from the first point above)
get us back again on pair with the k8s ingress example using minikube
make sure that ingress is always deployed on the 1st node in multi-node cluster (referenced by minikube ip)
update to the current nginx controller v1.1.1

example (using: https://kind.sigs.k8s.io/docs/user/ingress/)

❯ minikube start --nodes=3
😄 minikube v1.25.1 on Opensuse-Tumbleweed
✨ Automatically selected the docker driver. Other choices: kvm2, virtualbox, ssh
💨 For improved Docker performance, enable the overlay Linux kernel module using 'modprobe overlay'
❗ docker is currently using the btrfs storage driver, consider switching to overlay2 for better performance
👍 Starting control plane node minikube in cluster minikube
🚜 Pulling base image ...
🔥 Creating docker container (CPUs=2, Memory=5333MB) ...
🐳 Preparing Kubernetes v1.23.1 on Docker 20.10.12 ...
▪ kubelet.housekeeping-interval=5m
▪ kubelet.cni-conf-dir=/etc/cni/net.mk
▪ Generating certificates and keys ...
▪ Booting up control plane ...
▪ Configuring RBAC rules ...
🔗 Configuring CNI (Container Networking Interface) ...
🔎 Verifying Kubernetes components...
▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5
🌟 Enabled addons: storage-provisioner, default-storageclass

👍 Starting worker node minikube-m02 in cluster minikube
🚜 Pulling base image ...
🔥 Creating docker container (CPUs=2, Memory=5333MB) ...
🌐 Found network options:
▪ NO_PROXY=192.168.49.2
🐳 Preparing Kubernetes v1.23.1 on Docker 20.10.12 ...
▪ env NO_PROXY=192.168.49.2
🔎 Verifying Kubernetes components...

👍 Starting worker node minikube-m03 in cluster minikube
🚜 Pulling base image ...
🔥 Creating docker container (CPUs=2, Memory=5333MB) ...
🌐 Found network options:
▪ NO_PROXY=192.168.49.2,192.168.49.3
🐳 Preparing Kubernetes v1.23.1 on Docker 20.10.12 ...
▪ env NO_PROXY=192.168.49.2
▪ env NO_PROXY=192.168.49.2,192.168.49.3
🔎 Verifying Kubernetes components...
🏄 Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default

❯ minikube addons enable ingress
▪ Using image k8s.gcr.io/ingress-nginx/controller:v1.1.1
▪ Using image k8s.gcr.io/ingress-nginx/kube-webhook-certgen:v1.1.1
▪ Using image k8s.gcr.io/ingress-nginx/kube-webhook-certgen:v1.1.1
🔎 Verifying ingress addon...
🌟 The 'ingress' addon is enabled

❯ kubectl get pods -n ingress-nginx

NAME                                       READY   STATUS      RESTARTS   AGE
ingress-nginx-admission-create-nxsk2       0/1     Completed   0          104s
ingress-nginx-admission-patch-t2bvc        0/1     Completed   1          104s
ingress-nginx-controller-cc8496874-ct9z9   1/1     Running     0          104s

❯ kubectl apply -f https://kind.sigs.k8s.io/examples/ingress/usage.yaml
pod/foo-app created
service/foo-service created
pod/bar-app created
service/bar-service created
ingress.networking.k8s.io/example-ingress created

❯ kubectl get ingress

NAME              CLASS    HOSTS   ADDRESS   PORTS   AGE
example-ingress   <none>   *                 80      15s

... (wait a bit) ...

❯ kubectl get ingress

NAME              CLASS    HOSTS   ADDRESS        PORTS   AGE
example-ingress   <none>   *       192.168.49.2   80      27s

❯ kubectl get nodes -o wide

NAME           STATUS   ROLES                  AGE   VERSION   INTERNAL-IP    EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION     CONTAINER-RUNTIME
minikube       Ready    control-plane,master   13m   v1.23.1   192.168.49.2   <none>        Ubuntu 20.04.2 LTS   5.16.1-1-default   docker://20.10.12
minikube-m02   Ready    <none>                 13m   v1.23.1   192.168.49.3   <none>        Ubuntu 20.04.2 LTS   5.16.1-1-default   docker://20.10.12
minikube-m03   Ready    <none>                 13m   v1.23.1   192.168.49.4   <none>        Ubuntu 20.04.2 LTS   5.16.1-1-default   docker://20.10.12

❯ kubectl get pods -o wide

NAME      READY   STATUS    RESTARTS   AGE   IP           NODE           NOMINATED NODE   READINESS GATES
bar-app   1/1     Running   0          65s   10.244.2.2   minikube-m03   <none>           <none>
foo-app   1/1     Running   0          65s   10.244.1.2   minikube-m02   <none>           <none>

❯ minikube ip
192.168.49.2

❯ curl `minikube ip`/foo

foo

❯ curl `minikube ip`/bar

bar

for ref - the content of https://kind.sigs.k8s.io/examples/ingress/usage.yaml:

kind: Pod
apiVersion: v1
metadata:
  name: foo-app
  labels:
    app: foo
spec:
  containers:
  - name: foo-app
    image: hashicorp/http-echo:0.2.3
    args:
    - "-text=foo"
---
kind: Service
apiVersion: v1
metadata:
  name: foo-service
spec:
  selector:
    app: foo
  ports:
  # Default port used by the image
  - port: 5678
---
kind: Pod
apiVersion: v1
metadata:
  name: bar-app
  labels:
    app: bar
spec:
  containers:
  - name: bar-app
    image: hashicorp/http-echo:0.2.3
    args:
    - "-text=bar"
---
kind: Service
apiVersion: v1
metadata:
  name: bar-service
spec:
  selector:
    app: bar
  ports:
  # Default port used by the image
  - port: 5678
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: example-ingress
spec:
  rules:
  - http:
      paths:
      - pathType: Prefix
        path: "/foo"
        backend:
          service:
            name: foo-service
            port:
              number: 5678
      - pathType: Prefix
        path: "/bar"
        backend:
          service:
            name: bar-service
            port:
              number: 5678
---

prezha · 2022-01-23T02:06:25Z

/ok-to-test

minikube-pr-bot · 2022-01-23T02:57:19Z

kvm2 driver with docker runtime

+----------------+----------+---------------------+
|    COMMAND     | MINIKUBE | MINIKUBE (PR 13439) |
+----------------+----------+---------------------+
| minikube start | 44.4s    | 44.3s               |
| enable ingress | 39.4s    | 29.7s               |
+----------------+----------+---------------------+

Times for minikube start: 45.5s 44.2s 44.4s 44.2s 43.7s
Times for minikube (PR 13439) start: 44.8s 44.9s 43.6s 43.4s 44.9s

Times for minikube ingress: 85.1s 28.6s 28.6s 25.5s 29.0s
Times for minikube (PR 13439) ingress: 30.1s 29.1s 30.6s 30.0s 28.6s

docker driver with docker runtime

+----------------+----------+---------------------+
|    COMMAND     | MINIKUBE | MINIKUBE (PR 13439) |
+----------------+----------+---------------------+
| minikube start | 26.3s    | 26.4s               |
| enable ingress | 23.0s    | 22.4s               |
+----------------+----------+---------------------+

Times for minikube start: 26.6s 26.6s 26.2s 25.8s 26.2s
Times for minikube (PR 13439) start: 25.6s 25.8s 26.7s 26.8s 27.0s

Times for minikube ingress: 22.9s 23.0s 22.9s 22.4s 23.9s
Times for minikube (PR 13439) ingress: 22.4s 22.9s 22.0s 21.9s 22.9s

docker driver with containerd runtime

+----------------+----------+---------------------+
|    COMMAND     | MINIKUBE | MINIKUBE (PR 13439) |
+----------------+----------+---------------------+
| minikube start | 41.6s    | 41.1s               |
| enable ingress | 29.3s    | 27.4s               |
+----------------+----------+---------------------+

Times for minikube ingress: 23.5s 54.4s 19.9s 20.0s 29.0s
Times for minikube (PR 13439) ingress: 23.4s 23.4s 23.4s 33.5s 33.4s

Times for minikube start: 30.4s 41.7s 45.2s 45.5s 45.5s
Times for minikube (PR 13439) start: 40.8s 41.4s 41.7s 40.7s 41.0s

sharifelgamal

This looks excellent! Thanks for your hard work.

alexbaeza · 2022-02-07T17:04:28Z

Thank you for the mention @prezha Looks really good 👍

k8s-ci-robot · 2022-02-07T17:04:43Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: alexbaeza, prezha, sharifelgamal

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [prezha,sharifelgamal]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

sharifelgamal · 2022-02-07T18:58:24Z

/ok-to-test

minikube-pr-bot · 2022-02-07T19:52:04Z

kvm2 driver with docker runtime

+----------------+----------+---------------------+
|    COMMAND     | MINIKUBE | MINIKUBE (PR 13439) |
+----------------+----------+---------------------+
| minikube start | 51.7s    | 52.5s               |
| enable ingress | 28.2s    | 29.2s               |
+----------------+----------+---------------------+

Times for minikube start: 52.9s 46.8s 53.5s 52.3s 52.9s
Times for minikube (PR 13439) start: 52.0s 52.2s 52.7s 52.9s 52.7s

Times for minikube (PR 13439) ingress: 29.6s 28.6s 29.6s 30.0s 28.5s
Times for minikube ingress: 29.5s 25.5s 29.1s 28.6s 28.1s

docker driver with docker runtime

+----------------+----------+---------------------+
|    COMMAND     | MINIKUBE | MINIKUBE (PR 13439) |
+----------------+----------+---------------------+
| minikube start | 29.6s    | 36.4s               |
| enable ingress | 22.1s    | 22.7s               |
+----------------+----------+---------------------+

Times for minikube start: 42.3s 26.8s 26.5s 26.0s 26.5s
Times for minikube (PR 13439) start: 78.6s 25.9s 25.8s 26.4s 25.4s

Times for minikube ingress: 22.4s 21.9s 21.4s 22.4s 22.4s
Times for minikube (PR 13439) ingress: 22.9s 20.9s 23.9s 22.9s 22.9s

docker driver with containerd runtime

+----------------+----------+---------------------+
|    COMMAND     | MINIKUBE | MINIKUBE (PR 13439) |
+----------------+----------+---------------------+
| minikube start | 43.1s    | 41.7s               |
| enable ingress | 23.0s    | 25.7s               |
+----------------+----------+---------------------+

Times for minikube start: 44.1s 45.7s 42.3s 42.0s 41.4s
Times for minikube (PR 13439) start: 41.5s 41.9s 42.3s 41.2s 41.6s

Times for minikube (PR 13439) ingress: 23.4s 23.9s 23.5s 24.4s 33.4s
Times for minikube ingress: 26.9s 18.9s 22.9s 23.4s 22.9s

minikube-pr-bot · 2022-02-07T21:57:03Z

These are the flake rates of all failed tests.

Environment	Failed Tests	Flake Rate (%)
Docker_Linux	TestStartStop/group/old-k8s-version/serial/DeployApp (gopogh)	0.76 (chart)
Docker_Linux	TestStartStop/group/old-k8s-version/serial/EnableAddonAfterStop (gopogh)	0.76 (chart)
Docker_Linux	TestStartStop/group/old-k8s-version/serial/EnableAddonWhileActive (gopogh)	0.76 (chart)
Docker_Linux	TestStartStop/group/old-k8s-version/serial/FirstStart (gopogh)	0.76 (chart)
Docker_Linux	TestStartStop/group/old-k8s-version/serial/Stop (gopogh)	0.76 (chart)
Docker_Windows	TestPause/serial/VerifyDeletedResources (gopogh)	4.17 (chart)
Docker_Windows	TestDockerFlags (gopogh)	4.26 (chart)
Docker_Windows	TestForceSystemdEnv (gopogh)	6.38 (chart)
Docker_Linux_containerd	TestNetworkPlugins/group/cilium/Start (gopogh)	11.63 (chart)
Docker_Linux	TestStartStop/group/old-k8s-version/serial/Pause (gopogh)	12.21 (chart)
Docker_Windows	TestNetworkPlugins/group/auto/Start (gopogh)	12.77 (chart)
Docker_Linux_containerd	TestAddons/Setup (gopogh)	25.00 (chart)
Docker_Linux_containerd	TestIngressAddonLegacy/serial/ValidateIngressAddonActivation (gopogh)	25.00 (chart)
Docker_Linux_containerd	TestIngressAddonLegacy/StartLegacyK8sCluster (gopogh)	25.00 (chart)
Docker_Linux_containerd	TestJSONOutput/start/Audit (gopogh)	25.00 (chart)
Docker_Linux_containerd	TestJSONOutput/start/Command (gopogh)	25.00 (chart)
Docker_Linux_containerd	TestMultiNode/serial/AddNode (gopogh)	25.00 (chart)
Docker_Linux_containerd	TestMultiNode/serial/DeleteNode (gopogh)	25.00 (chart)
Docker_Linux_containerd	TestMultiNode/serial/FreshStart2Nodes (gopogh)	25.00 (chart)
Docker_Linux_containerd	TestMultiNode/serial/ProfileList (gopogh)	25.00 (chart)
Docker_Linux_containerd	TestMultiNode/serial/RestartKeepsNodes (gopogh)	25.00 (chart)
Docker_Linux_containerd	TestMultiNode/serial/StartAfterStop (gopogh)	25.00 (chart)
Docker_Linux_containerd	TestMultiNode/serial/StopNode (gopogh)	25.00 (chart)
Docker_Linux_containerd	TestPreload (gopogh)	25.00 (chart)
Docker_Linux_containerd	TestMultiNode/serial/RestartMultiNode (gopogh)	25.89 (chart)
Docker_macOS	TestStartStop/group/default-k8s-different-port/serial/SecondStart (gopogh)	26.21 (chart)
Docker_Linux_containerd	TestOffline (gopogh)	28.57 (chart)
Docker_Linux_containerd	TestPause/serial/Start (gopogh)	31.25 (chart)
Docker_Windows	TestNetworkPlugins/group/enable-default-cni/Start (gopogh)	31.91 (chart)
Docker_macOS	TestNetworkPlugins/group/kubenet/DNS (gopogh)	32.08 (chart)
More tests...	Continued...

Too many tests failed - See test logs for more details.

To see the flake rates of all tests by environment, click here.

fix ingress for multinode

422b43e

k8s-ci-robot added cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. labels Jan 22, 2022

k8s-ci-robot requested review from medyagh and prasadkatti January 22, 2022 23:37

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jan 22, 2022

prezha marked this pull request as draft January 22, 2022 23:59

k8s-ci-robot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jan 22, 2022

prezha added 3 commits January 23, 2022 01:00

remove 'publish-status-address=localhost' and fix comment

2be6b99

add 'is-default-class'

9048b0b

add 'is-default-class'

75a63be

k8s-ci-robot added the ok-to-test Indicates a non-member PR verified by an org member that is safe to test. label Jan 23, 2022

prezha changed the title ~~fix ingress for multinode~~ fix ingress (also for multinode clusters) Jan 23, 2022

spowelljr mentioned this pull request Jan 26, 2022

Error enabling ingress addon #13249

Closed

prezha marked this pull request as ready for review January 26, 2022 21:56

k8s-ci-robot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jan 26, 2022

This was referenced Jan 26, 2022

minikube addons enable ingress #13127

Closed

minikube addons enable ingress exits with error #13130

Closed

sharifelgamal approved these changes Jan 27, 2022

View reviewed changes

alexbaeza approved these changes Feb 7, 2022

View reviewed changes

sharifelgamal merged commit 6189742 into kubernetes:master Feb 7, 2022

This was referenced Feb 7, 2022

Fix ingress-controller pod rolling updates #12904

Closed

Cannot start ingress addon: waiting for pod #10544

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix ingress (also for multinode clusters) #13439

fix ingress (also for multinode clusters) #13439

prezha commented Jan 22, 2022 •

edited

Loading

prezha commented Jan 23, 2022

minikube-pr-bot commented Jan 23, 2022

sharifelgamal left a comment

alexbaeza commented Feb 7, 2022

k8s-ci-robot commented Feb 7, 2022

sharifelgamal commented Feb 7, 2022

minikube-pr-bot commented Feb 7, 2022

minikube-pr-bot commented Feb 7, 2022

fix ingress (also for multinode clusters) #13439

fix ingress (also for multinode clusters) #13439

Conversation

prezha commented Jan 22, 2022 • edited Loading

example (using: https://kind.sigs.k8s.io/docs/user/ingress/)

prezha commented Jan 23, 2022

minikube-pr-bot commented Jan 23, 2022

sharifelgamal left a comment

Choose a reason for hiding this comment

alexbaeza commented Feb 7, 2022

k8s-ci-robot commented Feb 7, 2022

sharifelgamal commented Feb 7, 2022

minikube-pr-bot commented Feb 7, 2022

minikube-pr-bot commented Feb 7, 2022

prezha commented Jan 22, 2022 •

edited

Loading