Che can't verify ingress after creating it in k8s (Azure AKS)

[bs-matil]

Description

Hello,
I need help to debug an issue in che. We right now can't create workspaces beacuse the creation of the ingress endpoints always fails. But its unclear why. The attached loggs show the error and the created ingresses. There are no errors on the k8s event log. Any suggestions? To me the created ingress connections look just fine.

The error happens due to the timeout of the future here https://github.com/eclipse/che/blob/07263f1e30089689d71b057f747a44a29283e3c4/infrastructures/kubernetes/src/main/java/org/eclipse/che/workspace/infrastructure/kubernetes/namespace/KubernetesIngresses.java#L109

If I step into here I can evaluate ingress.getStatus().getLoadBalancer().getIngress().isEmpty() and it returns true. So at least something seems to be correct.
https://github.com/eclipse/che/blob/07263f1e30089689d71b057f747a44a29283e3c4/infrastructures/kubernetes/src/main/java/org/eclipse/che/workspace/infrastructure/kubernetes/namespace/KubernetesIngresses.java#L84

If I set a breakpoint here https://github.com/eclipse/che/blob/07263f1e30089689d71b057f747a44a29283e3c4/infrastructures/kubernetes/src/main/java/org/eclipse/che/workspace/infrastructure/kubernetes/namespace/KubernetesIngresses.java#L85

The application never stops. This already confuses me a lot.

Could it be that che does not get the k8s events for some reason? And if yes how can I check this in fabric8.

Anything additional I can look at?

OS and version:
kubernets v 1.10.6
che 6,7.1

Diagnostics:
From che:

2018-08-13 11:54:27,429[io-8080-exec-10]  [INFO ] [o.e.c.a.w.s.WorkspaceManager 459]    - Workspace 'username/wksp-gk9b' with id 'workspaceaqjidosbor1yjmcx' created by user 'username'
2018-08-13 11:54:31,658[nio-8080-exec-1]  [INFO ] [o.e.c.a.w.s.WorkspaceRuntimes 329]   - Starting workspace 'username/wksp-gk9b' with id 'workspaceaqjidosbor1yjmcx' by user 'username'
2018-08-13 11:54:31,919[aceSharedPool-0]  [WARN ] [i.f.k.c.i.VersionUsageUtils 55]      - The client is using resource type 'ingresses' with unstable version 'v1beta1'

Created Ingresses:

apiVersion: v1
items:
- apiVersion: extensions/v1beta1
  kind: Ingress
  metadata:
    annotations:
      kubernetes.io/ingress.class: nginx
      kubernetes.io/tls-acme: "true"
      nginx.ingress.kubernetes.io/proxy-connect-timeout: "3600"
      nginx.ingress.kubernetes.io/proxy-read-timeout: "3600"
      nginx.ingress.kubernetes.io/rewrite-target: /
      nginx.ingress.kubernetes.io/ssl-redirect: "true"
      org.eclipse.che.machine.name: dev-machine
      org.eclipse.che.server.wsagent/http.attributes: '{}'
      org.eclipse.che.server.wsagent/http.path: /api
      org.eclipse.che.server.wsagent/http.port: 4401/tcp
      org.eclipse.che.server.wsagent/http.protocol: https
      org.eclipse.che.server.wsagent/ws.attributes: '{}'
      org.eclipse.che.server.wsagent/ws.path: /wsagent
      org.eclipse.che.server.wsagent/ws.port: 4401/tcp
      org.eclipse.che.server.wsagent/ws.protocol: wss
    creationTimestamp: 2018-08-13T11:54:31Z
    generation: 1
    labels:
      che.original_name: serverhx0eo2y3-dev-machine-server-4401
      che.workspace_id: workspaceaqjidosbor1yjmcx
    name: ingress9aa9v4t8
    namespace: che-workspace
    resourceVersion: "6497308"
    selfLink: /apis/extensions/v1beta1/namespaces/che-workspace/ingresses/ingress9aa9v4t8
    uid: a3ed218e-9eef-11e8-aa28-22c5e7394065
  spec:
    rules:
    - host: example.com
      http:
        paths:
        - backend:
            serviceName: serverhx0eo2y3-dev-machine
            servicePort: server-4401
          path: /serverhx0eo2y3-dev-machine/server-4401
    tls:
    - hosts:
      - example.com
      secretName: che-tls
  status:
    loadBalancer: {}
- apiVersion: extensions/v1beta1
  kind: Ingress
  metadata:
    annotations:
      kubernetes.io/ingress.class: nginx
      kubernetes.io/tls-acme: "true"
      nginx.ingress.kubernetes.io/proxy-connect-timeout: "3600"
      nginx.ingress.kubernetes.io/proxy-read-timeout: "3600"
      nginx.ingress.kubernetes.io/rewrite-target: /
      nginx.ingress.kubernetes.io/ssl-redirect: "true"
      org.eclipse.che.machine.name: dev-machine
      org.eclipse.che.server.exec-agent/http.attributes: '{}'
      org.eclipse.che.server.exec-agent/http.path: /process
      org.eclipse.che.server.exec-agent/http.port: 4412/tcp
      org.eclipse.che.server.exec-agent/http.protocol: https
      org.eclipse.che.server.exec-agent/ws.attributes: '{}'
      org.eclipse.che.server.exec-agent/ws.path: /connect
      org.eclipse.che.server.exec-agent/ws.port: 4412/tcp
      org.eclipse.che.server.exec-agent/ws.protocol: wss
    creationTimestamp: 2018-08-13T11:54:32Z
    generation: 1
    labels:
      che.original_name: serverhx0eo2y3-dev-machine-server-4412
      che.workspace_id: workspaceaqjidosbor1yjmcx
    name: ingressepf5n797
    namespace: che-workspace
    resourceVersion: "6497312"
    selfLink: /apis/extensions/v1beta1/namespaces/che-workspace/ingresses/ingressepf5n797
    uid: a40dc0ce-9eef-11e8-aa28-22c5e7394065
  spec:
    rules:
    - host: example.com
      http:
        paths:
        - backend:
            serviceName: serverhx0eo2y3-dev-machine
            servicePort: server-4412
          path: /serverhx0eo2y3-dev-machine/server-4412
    tls:
    - hosts:
      - example.com
      secretName: che-tls
  status:
    loadBalancer: {}
- apiVersion: extensions/v1beta1
  kind: Ingress
  metadata:
    annotations:
      kubernetes.io/ingress.class: nginx
      kubernetes.io/tls-acme: "true"
      nginx.ingress.kubernetes.io/proxy-connect-timeout: "3600"
      nginx.ingress.kubernetes.io/proxy-read-timeout: "3600"
      nginx.ingress.kubernetes.io/rewrite-target: /
      nginx.ingress.kubernetes.io/ssl-redirect: "true"
      org.eclipse.che.machine.name: dev-machine
      org.eclipse.che.server.tomcat8-debug.attributes: '{}'
      org.eclipse.che.server.tomcat8-debug.port: 8000/tcp
      org.eclipse.che.server.tomcat8-debug.protocol: https
    creationTimestamp: 2018-08-13T11:54:32Z
    generation: 1
    labels:
      che.original_name: serverhx0eo2y3-dev-machine-server-8000
      che.workspace_id: workspaceaqjidosbor1yjmcx
    name: ingressf4al2bpp
    namespace: che-workspace
    resourceVersion: "6497317"
    selfLink: /apis/extensions/v1beta1/namespaces/che-workspace/ingresses/ingressf4al2bpp
    uid: a4328a11-9eef-11e8-aa28-22c5e7394065
  spec:
    rules:
    - host: example.com
      http:
        paths:
        - backend:
            serviceName: serverhx0eo2y3-dev-machine
            servicePort: server-8000
          path: /serverhx0eo2y3-dev-machine/server-8000
    tls:
    - hosts:
      - example.com
      secretName: che-tls
  status:
    loadBalancer: {}
- apiVersion: extensions/v1beta1
  kind: Ingress
  metadata:
    annotations:
      kubernetes.io/ingress.class: nginx
      kubernetes.io/tls-acme: "true"
      nginx.ingress.kubernetes.io/proxy-connect-timeout: "3600"
      nginx.ingress.kubernetes.io/proxy-read-timeout: "3600"
      nginx.ingress.kubernetes.io/rewrite-target: /
      nginx.ingress.kubernetes.io/ssl-redirect: "true"
      org.eclipse.che.machine.name: dev-machine
      org.eclipse.che.server.wsagent-debug.attributes: '{}'
      org.eclipse.che.server.wsagent-debug.port: 4403/tcp
      org.eclipse.che.server.wsagent-debug.protocol: tcp
    creationTimestamp: 2018-08-13T11:54:32Z
    generation: 1
    labels:
      che.original_name: serverhx0eo2y3-dev-machine-server-4403
      che.workspace_id: workspaceaqjidosbor1yjmcx
    name: ingressxhtximbi
    namespace: che-workspace
    resourceVersion: "6497316"
    selfLink: /apis/extensions/v1beta1/namespaces/che-workspace/ingresses/ingressxhtximbi
    uid: a427e8a6-9eef-11e8-aa28-22c5e7394065
  spec:
    rules:
    - host: example.com
      http:
        paths:
        - backend:
            serviceName: serverhx0eo2y3-dev-machine
            servicePort: server-4403
          path: /serverhx0eo2y3-dev-machine/server-4403
    tls:
    - hosts:
      - example.com
      secretName: che-tls
  status:
    loadBalancer: {}
- apiVersion: extensions/v1beta1
  kind: Ingress
  metadata:
    annotations:
      kubernetes.io/ingress.class: nginx
      kubernetes.io/tls-acme: "true"
      nginx.ingress.kubernetes.io/proxy-connect-timeout: "3600"
      nginx.ingress.kubernetes.io/proxy-read-timeout: "3600"
      nginx.ingress.kubernetes.io/rewrite-target: /
      nginx.ingress.kubernetes.io/ssl-redirect: "true"
      org.eclipse.che.machine.name: dev-machine
      org.eclipse.che.server.tomcat8.attributes: '{}'
      org.eclipse.che.server.tomcat8.port: 8080/tcp
      org.eclipse.che.server.tomcat8.protocol: https
    creationTimestamp: 2018-08-13T11:54:32Z
    generation: 1
    labels:
      che.original_name: serverhx0eo2y3-dev-machine-server-8080
      che.workspace_id: workspaceaqjidosbor1yjmcx
    name: ingressy9ok6swa
    namespace: che-workspace
    resourceVersion: "6497310"
    selfLink: /apis/extensions/v1beta1/namespaces/che-workspace/ingresses/ingressy9ok6swa
    uid: a400df48-9eef-11e8-aa28-22c5e7394065
  spec:
    rules:
    - host: example.com
      http:
        paths:
        - backend:
            serviceName: serverhx0eo2y3-dev-machine
            servicePort: server-8080
          path: /serverhx0eo2y3-dev-machine/server-8080
    tls:
    - hosts:
      - example.com
      secretName: che-tls
  status:
    loadBalancer: {}
- apiVersion: extensions/v1beta1
  kind: Ingress
  metadata:
    annotations:
      kubernetes.io/ingress.class: nginx
      kubernetes.io/tls-acme: "true"
      nginx.ingress.kubernetes.io/proxy-connect-timeout: "3600"
      nginx.ingress.kubernetes.io/proxy-read-timeout: "3600"
      nginx.ingress.kubernetes.io/rewrite-target: /
      nginx.ingress.kubernetes.io/ssl-redirect: "true"
      org.eclipse.che.machine.name: dev-machine
      org.eclipse.che.server.terminal.attributes: '{}'
      org.eclipse.che.server.terminal.path: /pty
      org.eclipse.che.server.terminal.port: 4411/tcp
      org.eclipse.che.server.terminal.protocol: wss
    creationTimestamp: 2018-08-13T11:54:32Z
    generation: 1
    labels:
      che.original_name: serverhx0eo2y3-dev-machine-server-4411
      che.workspace_id: workspaceaqjidosbor1yjmcx
    name: ingresszgzyynnz
    namespace: che-workspace
    resourceVersion: "6497309"
    selfLink: /apis/extensions/v1beta1/namespaces/che-workspace/ingresses/ingresszgzyynnz
    uid: a3f7074c-9eef-11e8-aa28-22c5e7394065
  spec:
    rules:
    - host: example.com
      http:
        paths:
        - backend:
            serviceName: serverhx0eo2y3-dev-machine
            servicePort: server-4411
          path: /serverhx0eo2y3-dev-machine/server-4411
    tls:
    - hosts:
      - example.com
      secretName: che-tls
  status:
    loadBalancer: {}
kind: List
metadata:
  resourceVersion: ""
  selfLink: ""

[sleshchenko]

Hi,
The cause of failing I described here #10300 (comment)
I'll create an issue for Che not to wait loadBalancer to be not empty if it is not required.

Just to clarify which server strategy do you use?

[bs-matil]

right now we use single-host

[sleshchenko]

@bsukramlitt I've created an issue for fixing an issue that you faced #10767.

[bs-matil]

how can i fix the issue without this implementation?

[bs-matil]

turns out the fix is as easy as enabling this in the values.yaml of nginx ingress

  ## Allows customization of the external service
  ## the ingress will be bound to via DNS
  publishService:
    enabled: true

afterwards all ingress objects will get the lb ip in the ingress and che continues.

[skabashnyuk]

Closing. Feel free to reopen this issue if you think it's still relevant for you.