Che is not accessible after being started by chectl on Kubernetes #13840

dmytro-ndp · 2019-07-12T16:34:13Z

Describe the bug

Chectl didn't enable ingress addon on Kubernetes besides there was Enable minikube ingress addon [completed] in output, and it made Che inaccessible by Che Server URL.

Chectl version

https://github.com/che-incubator/chectl/releases/latest/download/chectl-linux

Steps to reproduce

Install Kubernetes using minikube from scratch:

  export MINIKUBE_WANTUPDATENOTIFICATION=false
  export MINIKUBE_WANTREPORTERRORPROMPT=false
  export MINIKUBE_HOME=$HOME
  export CHANGE_MINIKUBE_NONE_USER=true
  export KUBECONFIG=$HOME/.kube/config
  
  mkdir -p $HOME/.kube $HOME/.minikube
  touch $KUBECONFIG
  
  sudo -E  /usr/local/bin/minikube start  \
      --vm-driver=none \
      --cpus 4 \
      --memory 12288 \
      --extra-config=kubelet.cgroup-driver=systemd

Start Che using chectl:

chectl server:start --k8spodreadytimeout=180000 -t=/mnt/hudson_workspace/workspace/CHE-7-Happy-Path-Tests/deploy/ --listr-renderer=verbose

Expected behavior

Che has been installed and is accessible by Che Server URL.

Actual behavior

Che was not accessible by Che Server URL because ingress addon was not actually enabled:

curl -v http://che-che.10.0.148.224.nip.io
* About to connect() to che-che.10.0.148.224.nip.io port 80 (#0)
*   Trying 10.0.148.224...
* Connection refused
* Failed connect to che-che.10.0.148.224.nip.io:80; Connection refused
* Closing connection 0
curl: (7) Failed connect to che-che.10.0.148.224.nip.io:80; Connection refused

There was no ~/.minikube/config/config.json file with "ingress": true parameter

ls -la ~/.minikube/config
total 4
drwxr-xr-x.  2 hudson hudson    6 Jul 12 12:07 .
drwxrwxr-x. 10 hudson hudson 4096 Jul 12 12:07 ..

There was no nginx-ingress-controller running among the pods:

kubectl get pods --all-namespaces
NAMESPACE     NAME                               READY   STATUS    RESTARTS   AGE
che           che-6d6c8b5fb4-rbnr2               1/1     Running   0          3m36s
kube-system   coredns-fb8b8dccf-75bs8            1/1     Running   0          6m23s
kube-system   coredns-fb8b8dccf-nqr5s            1/1     Running   0          6m23s
kube-system   etcd-minikube                      1/1     Running   0          5m9s
kube-system   kube-addon-manager-minikube        1/1     Running   0          6m33s
kube-system   kube-apiserver-minikube            1/1     Running   0          5m21s
kube-system   kube-controller-manager-minikube   1/1     Running   0          5m34s
kube-system   kube-proxy-pfq2l                   1/1     Running   0          6m23s
kube-system   kube-scheduler-minikube            1/1     Running   0          5m29s
kube-system   storage-provisioner                1/1     Running   0          6m22s
kube-system   tiller-deploy-75456b7758-7v9lj     1/1     Running   0          3m54s

Runtime

kubernetes

Client Version: version.Info{Major:"1", Minor:"14", GitVersion:"v1.14.3", GitCommit:"5e53fd6bc17c0dec8434817e69b04a25d8ae0ff0", GitTreeState:"clean", BuildDate:"2019-06-06T01:44:30Z", GoVersion:"go1.12.5", Compiler:"gc", Platform:"linux/amd64"}

minikube v1.1.1, kubelet v1.14.3

Logs

/usr/local/bin/chectl server:start --k8spodreadytimeout=180000 -t=/mnt/hudson_workspace/workspace/CHE-7-Happy-Path-Tests/deploy/ --listr-renderer=verbose
[12:10:00] ✈️  Minikube preflight checklist [started]
[12:10:00] Verify if kubectl is installed [started]
[12:10:00] Verify if kubectl is installed [completed]
[12:10:00] Verify if minikube is installed [started]
[12:10:00] Verify if minikube is installed [completed]
[12:10:00] Verify if minikube is running [started]
[12:10:00] Verify if minikube is running [completed]
[12:10:00] Start minikube [started]
[12:10:00] Start minikube [skipped]
[12:10:00] → Minikube is already running.
[12:10:00] Verify if minikube ingress addon is enabled [started]
[12:10:00] Verify if minikube ingress addon is enabled [completed]
[12:10:00] Enable minikube ingress addon [started]
[12:10:00] Enable minikube ingress addon [completed]
[12:10:00] Retrieving minikube IP and domain for ingress URLs [started]
[12:10:00] Retrieving minikube IP and domain for ingress URLs...10.0.148.224.nip.io. [title changed]
[12:10:00] Retrieving minikube IP and domain for ingress URLs...10.0.148.224.nip.io. [completed]
[12:10:00] ✈️  Minikube preflight checklist [completed]
[12:10:00] 🏃‍  Running Helm to install Che [started]
[12:10:00] Verify if helm is installed [started]
[12:10:00] Verify if helm is installed [completed]
[12:10:00] Create Tiller Role Binding [started]
[12:10:01] Create Tiller Role Binding...done. [title changed]
[12:10:01] Create Tiller Role Binding...done. [completed]
[12:10:01] Create Tiller Service Account [started]
[12:10:01] Create Tiller Service Account...done. [title changed]
[12:10:01] Create Tiller Service Account...done. [completed]
[12:10:01] Create Tiller RBAC [started]
[12:10:01] Create Tiller RBAC [completed]
[12:10:01] Create Tiller Service [started]
[12:10:08] Create Tiller Service...done. [title changed]
[12:10:08] Create Tiller Service...done. [completed]
[12:10:08] Preparing Che Helm Chart [started]
[12:10:08] Preparing Che Helm Chart...done. [title changed]
[12:10:08] Preparing Che Helm Chart...done. [completed]
[12:10:08] Updating Helm Chart dependencies [started]
[12:10:13] Updating Helm Chart dependencies...done. [title changed]
[12:10:13] Updating Helm Chart dependencies...done. [completed]
[12:10:13] Deploying Che Helm Chart [started]
[12:10:13] Deploying Che Helm Chart...done. [title changed]
[12:10:13] Deploying Che Helm Chart...done. [completed]
[12:10:13] 🏃‍  Running Helm to install Che [completed]
[12:10:13] ✅  Post installation checklist [started]
[12:10:13] Che pod bootstrap [started]
[12:10:13] scheduling [started]
[12:10:13] scheduling...done. [title changed]
[12:10:13] scheduling...done. [completed]
[12:10:13] downloading images [started]
[12:10:26] downloading images...done. [title changed]
[12:10:26] downloading images...done. [completed]
[12:10:26] starting [started]
[12:10:48] starting...done. [title changed]
[12:10:48] starting...done. [completed]
[12:10:48] Che pod bootstrap [completed]
[12:10:48] Retrieving Che Server URL [started]
[12:10:48] Retrieving Che Server URL...http://che-che.10.0.148.224.nip.io [title changed]
[12:10:48] Retrieving Che Server URL...http://che-che.10.0.148.224.nip.io [completed]
[12:10:48] Che status check [started]
[12:10:48] Che status check [completed]
[12:10:48] ✅  Post installation checklist [completed]

Full build log: https://codeready-workspaces-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/view/che7-e2e-tests/job/CHE-7-Happy-Path-Tests/338/

Installation method

chectl

Environment

OpenStack on CRW Central CI
- RHEL 7.4

The text was updated successfully, but these errors were encountered:

rhopp · 2019-07-12T18:22:25Z

I've just quickly tried this with minikube with kvm2 driver and "plain" chectl server:start. Ingress addon was enabled without any problems:

Console output:

╭─rhopp@localhost ~
╰─$ minikube start
😄 minikube v1.1.1 on linux (amd64)
🔥 Creating kvm2 VM (CPUs=4, Memory=14336MB, Disk=20000MB) ...
🐳 Configuring environment for Kubernetes v1.14.3 on Docker 18.09.6
🚜 Pulling images ...
🚀 Launching Kubernetes ...
⌛ Verifying: apiserver proxy etcd scheduler controller dns
🏄 Done! kubectl is now configured to use "minikube"

╭─rhopp@localhost ~
╰─$ cat ~/.minikube/config/config.json
{
"cpus": 4,
"dashboard": true,
"memory": 14336,
"vm-driver": "kvm2"
}

╭─rhopp@localhost ~
╰─$ chectl server:start
✔ ✈️ Minikube preflight checklist
✔ Verify if kubectl is installed
✔ Verify if minikube is installed
✔ Verify if minikube is running
↓ Start minikube [skipped]
→ Minikube is already running.
✔ Verify if minikube ingress addon is enabled
✔ Enable minikube ingress addon
✔ Retrieving minikube IP and domain for ingress URLs...192.168.39.241.nip.io.
✔ 🏃 Running Helm to install Che
✔ Verify if helm is installed
✔ Create Tiller Role Binding...done.
✔ Create Tiller Service Account...done.
✔ Create Tiller RBAC
✔ Create Tiller Service...done.
✔ Preparing Che Helm Chart...done.
✔ Updating Helm Chart dependencies...done.
✔ Deploying Che Helm Chart...done.
✔ ✅ Post installation checklist
✔ Che pod bootstrap
✔ scheduling...done.
✔ downloading images...done.
✔ starting...done.
✔ Retrieving Che Server URL...http://che-che.192.168.39.241.nip.io
✔ Che status check
Command server:start has completed successfully.

╭─rhopp@localhost ~
╰─$ cat ~/.minikube/config/config.json
{
"cpus": 4,
"dashboard": true,
"ingress": true,
"memory": 14336,
"vm-driver": "kvm2"
}

╭─rhopp@localhost ~
╰─$ kubectl get pods --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
che che-6d6c8b5fb4-fjjbh 1/1 Running 0 5m16s
kube-system coredns-fb8b8dccf-6qvv5 1/1 Running 0 12m
kube-system coredns-fb8b8dccf-bzm57 1/1 Running 0 12m
kube-system default-http-backend-6864bbb7db-mslv9 1/1 Running 0 5m39s
kube-system etcd-minikube 1/1 Running 0 11m
kube-system kube-addon-manager-minikube 1/1 Running 0 11m
kube-system kube-apiserver-minikube 1/1 Running 0 11m
kube-system kube-controller-manager-minikube 1/1 Running 0 11m
kube-system kube-proxy-gb7zc 1/1 Running 0 12m
kube-system kube-scheduler-minikube 1/1 Running 0 11m
kube-system kubernetes-dashboard-d7c9687c7-zbhfz 1/1 Running 0 12m
kube-system nginx-ingress-controller-586cdc477c-f4rg6 1/1 Running 0 5m38s
kube-system storage-provisioner 1/1 Running 0 12m
kube-system tiller-deploy-64445f4d64-28lcl 1/1 Running 0 5m41s

Have you manage to reproduce somewhere else, than on virtualized openstack machine?
What does command minikube addons enable ingress does on that machine? Does it succeed?

benoitf · 2019-07-12T18:42:50Z

yes on my computer I don't reproduce as well

could you retry by cleaning up $HOME/.minikube folder before starting minikube

dmytro-ndp · 2019-07-12T18:51:34Z

There was no such an error with minikube start command without additional arguments on OpenStack. It might be something specific for --vm-driver=none option of minikube start.

dmytro-ndp · 2019-07-12T18:56:53Z

I would also notice that chectl server:start finished successfully despite ingress addon enabling had failed, and Che wasn't accessible. IMHO, chectl ought to fail in that case.

benoitf · 2019-07-15T15:02:26Z

@dmytro-ndp is it possible to get access to this environment ?

che-bot · 2020-01-15T13:20:02Z

Issues go stale after 180 days of inactivity. lifecycle/stale issues rot after an additional 7 days of inactivity and eventually close.

Mark the issue as fresh with /remove-lifecycle stale in a new comment.

If this issue is safe to close now please do so.

Moderators: Add lifecycle/frozen label to avoid stale mode.

michealharrington · 2020-05-27T19:53:36Z

I'm still having the same issue.

minikube version
minikube version: v1.10.1
commit: 63ab801ac27e5742ae442ce36dff7877dcccb278

chectl server:start --platform minikube --installer=helm
› Current Kubernetes context: 'minikube'
  ✔ Verify Kubernetes API...OK
  ✔ 👀  Looking for an already existing Eclipse Che instance
    ✔ Verify if Eclipse Che is deployed into namespace "che"...it is not
  ✔ ✈️  Minikube preflight checklist
    ✔ Verify if kubectl is installed
    ✔ Verify if minikube is installed
    ✔ Verify if minikube is running
    ↓ Start minikube [skipped]
      → Minikube is already running.
    ✔ Check Kubernetes version: Found v1.18.2.
    ✔ Verify if minikube ingress addon is enabled
    ✔ Enable minikube ingress addon
    ✔ Retrieving minikube IP and domain for ingress URLs...172.17.0.4.nip.io.
    ↓ Check if cluster accessible [skipped]
Eclipse Che logs will be available in '/tmp/chectl-logs/1590608546255'
  ✔ Start following logs
    ↓ Start following Operator logs [skipped]
    ✔ Start following Eclipse Che logs...done
    ✔ Start following Postgres logs...done
    ✔ Start following Keycloak logs...done
    ✔ Start following Plugin registry logs...done
    ✔ Start following Devfile registry logs...done
  ✔ Start following events
    ✔ Start following namespace events...done
 ›   Warning: You can also use features rich 'OLM' installer to deploy Eclipse Che.
  ✔ 🏃‍  Running Helm to install Eclipse Che
    ✔ Verify if helm is installed
    ✔ Check Helm Version: Found v3.2.0+ge11b7ce
    ✔ Create Namespace (che)...done.
    ✔ Check Eclipse Che TLS certificate...going to generate self-signed one
      ✔ Check Cert Manager deployment...not deployed
      ✔ Deploy cert-manager...done
      ✔ Wait for cert-manager...ready
      ✔ Check Cert Manager CA certificate...generating new one
      ✔ Set up Eclipse Che certificates issuer...done
      ✔ Request self-signed certificate...done
      ✔ Wait for self-signed certificate...ready
      ✔ Retrieving Che self-signed CA certificate... is exported to /home/<>/cheCA.crt
    ✔ Check Cluster Role Binding...does not exists.
    ✔ Preparing Eclipse Che Helm Chart...done.
    ✔ Updating Helm Chart dependencies...done.
    ✔ Deploying Eclipse Che Helm Chart...done.
  ✔ ✅  Post installation checklist
    ✔ Devfile registry pod bootstrap
      ✔ scheduling...done.
      ✔ downloading images...done.
      ✔ starting...done.
    ✔ Plugin registry pod bootstrap
      ✔ scheduling...done.
      ✔ downloading images...done.
      ✔ starting...done.
    ✔ Eclipse Che pod bootstrap
      ✔ scheduling...done.
      ✔ downloading images...done.
      ✔ starting...done.
    ✔ Retrieving Eclipse Che server URL... https://che-che.172.17.0.4.nip.io
    ✔ Eclipse Che status check
  ✔ Show important messages
    ✔ ❗[MANUAL ACTION REQUIRED] Please add Che self-signed CA certificate into your browser: /home/<>/cheCA.crt.
   Documentation how to add a CA certificate into a browser: https://www.eclipse.org/che/docs/che-7/installing-che-in-tls-mode-with-self-signed-certificates/#using-che-with-tls_installing-che-in-tls-mode-with-se
lf-signed-certificates
Command server:start has completed successfully.

curl -v https://che-che.172.17.0.4.nip.io
* Could not resolve host: che-che.172.17.0.4.nip.io
* Closing connection 0
curl: (6) Could not resolve host: che-che.172.17.0.4.nip.io

How do I make https://che-che.172.17.0.4.nip.io accessible to the host machine?
Hitting 172.17.0.4 hits an nginx, which means I must be close, but the nip.io doesn't seem to work for me.

I'll gladly give anymore information that is needed.

dmytro-ndp changed the title ~~Che is not accessible after starting by chectl on Kubernetes~~ Che is not accessible after being started by chectl on Kubernetes Jul 12, 2019

dmytro-ndp assigned benoitf Jul 12, 2019

dmytro-ndp added the kind/bug Outline of a bug - must adhere to the bug report template. label Jul 12, 2019

rhopp mentioned this issue Jul 15, 2019

E2E tests suite for Che 7 #12728

Closed

25 tasks

nickboldt added the status/info-needed More information is needed before the issue can move into the “analyzing” state for engineering. label Jul 15, 2019

benoitf removed their assignment Jul 16, 2019

che-bot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Jan 15, 2020

che-bot closed this as completed Jan 22, 2020

michealharrington mentioned this issue May 29, 2020

[helm] Local Che server inaccessible after successful start in minikube 1.10.1 #17046

Closed

22 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Che is not accessible after being started by chectl on Kubernetes #13840

Che is not accessible after being started by chectl on Kubernetes #13840

dmytro-ndp commented Jul 12, 2019 •

edited

Loading

rhopp commented Jul 12, 2019

benoitf commented Jul 12, 2019

dmytro-ndp commented Jul 12, 2019 •

edited

Loading

dmytro-ndp commented Jul 12, 2019

benoitf commented Jul 15, 2019

che-bot commented Jan 15, 2020

michealharrington commented May 27, 2020

Che is not accessible after being started by chectl on Kubernetes #13840

Che is not accessible after being started by chectl on Kubernetes #13840

Comments

dmytro-ndp commented Jul 12, 2019 • edited Loading

Describe the bug

Chectl version

Steps to reproduce

Expected behavior

Actual behavior

Runtime

Logs

Installation method

Environment

rhopp commented Jul 12, 2019

benoitf commented Jul 12, 2019

dmytro-ndp commented Jul 12, 2019 • edited Loading

dmytro-ndp commented Jul 12, 2019

benoitf commented Jul 15, 2019

che-bot commented Jan 15, 2020

michealharrington commented May 27, 2020

dmytro-ndp commented Jul 12, 2019 •

edited

Loading

dmytro-ndp commented Jul 12, 2019 •

edited

Loading