CloudShell editor is loaded unstable

[sleshchenko]

Describe the bug

CloudShell editor is loading unstable, like it's loaded 1 from 5-8 times on my installation.
Note that it works just fine on single-user Che

Che version

• latest
• nightly
• other: please specify

Steps to reproduce

1. Create a workspace with CloudShell editor.

Cloud Shell Workspace

apiVersion: 1.0.0
metadata:
  name: cloud-shell
components:
  - alias: cloud-shell
    type: cheEditor
    id: eclipse/cloud-shell/nightly
  - type: dockerimage
    memoryLimit: 256Mi
    alias: dev
    image: 'quay.io/eclipse/che-sidecar-openshift-connector:0.1.2-2601509'
    args: ["tail", "-f", "/dev/null"]
    env:
      - value: '\[\e[34m\]>\[\e[m\]\[\e[33m\]>\[\e[m\]'
        name: PS1

2. Start the workspace.
3. After it's started refresh the page and check if the terminal is successfully initialized.

Actual behavior

Terminal is successfully initialized ~ 1 from 5 tries to refresh a page.

See scheencast:

Expected behavior

Terminal is successfully initialized each time we open it.

Runtime

• kubernetes (include output of kubectl version)
• Openshift (include output of oc version)
• minikube (include output of minikube version and kubectl version)
• minishift (include output of minishift version and oc version)
• docker-desktop + K8S (include output of docker version and kubectl version)
• other: (please specify)

Installation method

• chectl - chectl server:start --platform=minikube --multiuser
• che-operator
• minishift-addon
• I don't know

Environment

• my computer
  • Windows
  • Linux
  • macOS
• Cloud
  • Amazon
  • Azure
  • GCE
  • other (please specify)
• other: please specify

Additional context

Che Machine Exec logs

[GIN-debug] [WARNING] Running in "debug" mode. Switch to "release" mode in production.
 - using env:	export GIN_MODE=release
 - using code:	gin.SetMode(gin.ReleaseMode)
[GIN-debug] GET    /static/*filepath         --> github.com/eclipse/che-machine-exec/vendor/github.com/gin-gonic/gin.(*RouterGroup).createStaticHandler.func1 (3 handlers)
[GIN-debug] HEAD   /static/*filepath         --> github.com/eclipse/che-machine-exec/vendor/github.com/gin-gonic/gin.(*RouterGroup).createStaticHandler.func1 (3 handlers)
[GIN-debug] GET    /                         --> main.main.func1 (3 handlers)
[GIN-debug] GET    /connect                  --> main.main.func2 (3 handlers)
[GIN-debug] GET    /attach/:id               --> main.main.func3 (3 handlers)
⇩ Registered RPCRoutes:
Json-rpc MachineExec Routes:
✓ create
✓ check
✓ resize
[GIN-debug] Listening and serving HTTP on :4444
[GIN] 2020/01/28 - 01:01:32 | 301 |      62.416µs |     172.17.0.16 | GET      /
[GIN-debug] redirecting request 301: /static --> /static/
[GIN] 2020/01/28 - 01:01:32 | 200 |     234.991µs |     172.17.0.16 | GET      /static/
[GIN] 2020/01/28 - 01:01:34 | 301 |       20.15µs |    192.168.99.1 | GET      /?uid=63876
[GIN-debug] redirecting request 301: /static --> /static/
[GIN] 2020/01/28 - 01:01:34 | 200 |     197.971µs |    192.168.99.1 | GET      /static/
[GIN] 2020/01/28 - 01:01:34 | 200 |  104.881597ms |    192.168.99.1 | GET      /static/dist/bundle.js
[GIN] 2020/01/28 - 01:01:35 | 200 |   28.018518ms |    192.168.99.1 | GET      /connect
[GIN] 2020/01/28 - 01:01:49 | 301 |    2.935003ms |    192.168.99.1 | GET      /
[GIN-debug] redirecting request 301: /static --> /static/
[GIN] 2020/01/28 - 01:01:49 | 304 |     118.221µs |    192.168.99.1 | GET      /static/
[GIN] 2020/01/28 - 01:01:49 | 304 |     339.786µs |    192.168.99.1 | GET      /static/dist/bundle.js
[GIN] 2020/01/28 - 01:01:50 | 200 |     422.553µs |    192.168.99.1 | GET      /connect
[GIN] 2020/01/28 - 01:01:51 | 404 |       1.185µs |    192.168.99.1 | GET      /favicon.ico
[GIN] 2020/01/28 - 01:01:54 | 304 |      76.122µs |    192.168.99.1 | GET      /static/
[GIN] 2020/01/28 - 01:01:54 | 400 |      30.496µs |    192.168.99.1 | GET      /connect
Unable to upgrade connection to ws-conn websocket: could not find connection header with token 'upgrade'
failed to read ws-conn message. Cause: websocket: close 1001 
[GIN] 2020/01/28 - 01:01:54 | 400 |      37.723µs |    192.168.99.1 | GET      /attach/2
[GIN] 2020/01/28 - 01:01:54 | 304 |      381.04µs |    192.168.99.1 | GET      /static/dist/bundle.js
[GIN] 2020/01/28 - 01:01:55 | 200 |     254.158µs |    192.168.99.1 | GET      /connect
[GIN] 2020/01/28 - 01:01:57 | 304 |     627.727µs |    192.168.99.1 | GET      /static/
failed to read ws-conn message. Cause: websocket: close 1001 
Unable to upgrade connection to ws-conn websocket: could not find connection header with token 'upgrade'
[GIN] 2020/01/28 - 01:01:57 | 400 |       33.34µs |    192.168.99.1 | GET      /connect
[GIN] 2020/01/28 - 01:01:57 | 400 |     115.209µs |    192.168.99.1 | GET      /attach/3
[GIN] 2020/01/28 - 01:01:57 | 304 |      94.516µs |    192.168.99.1 | GET      /static/dist/bundle.js
[GIN] 2020/01/28 - 01:01:58 | 200 |     362.897µs |    192.168.99.1 | GET      /connect
[GIN] 2020/01/28 - 01:02:04 | 304 |       91.19µs |    192.168.99.1 | GET      /static/
[GIN] 2020/01/28 - 01:02:04 | 400 |      34.688µs |    192.168.99.1 | GET      /connect
[GIN] 2020/01/28 - 01:02:04 | 304 |     878.152µs |    192.168.99.1 | GET      /static/dist/bundle.js
[GIN] 2020/01/28 - 01:02:04 | 200 |     282.781µs |    192.168.99.1 | GET      /connect
[GIN] 2020/01/28 - 01:02:10 | 304 |     224.186µs |    192.168.99.1 | GET      /static/
[GIN] 2020/01/28 - 01:02:10 | 400 |       27.85µs |    192.168.99.1 | GET      /connect
[GIN] 2020/01/28 - 01:02:10 | 304 |     271.296µs |    192.168.99.1 | GET      /static/dist/bundle.js
[GIN] 2020/01/28 - 01:02:10 | 200 |     458.003µs |    192.168.99.1 | GET      /connect
Error occurs on sending ping message to ws-conn. websocket: close sent
Error occurs on sending ping message to ws-conn. websocket: close sent

JTWProxy logs

time="2020-01-28T01:01:24Z" level=info msg="Starting reverse proxy (Listening on ':4400')" 
2020/01/28 01:01:54 [017] WARN: Websocket error: <nil>
2020/01/28 01:01:54 [017] WARN: Websocket error: readfrom tcp 172.17.0.10:4400->172.17.0.4:44682: read tcp 172.17.0.10:60676->10.96.158.190:4444: use of closed network connection
2020/01/28 01:01:54 http: response.WriteHeader on hijacked connection
2020/01/28 01:01:54 http: response.Write on hijacked connection
2020/01/28 01:01:54 [019] WARN: Websocket error: <nil>
2020/01/28 01:01:54 [019] WARN: Websocket error: readfrom tcp 172.17.0.10:4400->172.17.0.4:44980: read tcp 172.17.0.10:60708->10.96.158.190:4444: use of closed network connection
2020/01/28 01:01:54 http: response.WriteHeader on hijacked connection
2020/01/28 01:01:54 http: response.Write on hijacked connection
2020/01/28 01:01:57 [022] WARN: Websocket error: <nil>
2020/01/28 01:01:57 [022] WARN: Websocket error: readfrom tcp 172.17.0.10:4400->172.17.0.4:45028: read tcp 172.17.0.10:60756->10.96.158.190:4444: use of closed network connection
2020/01/28 01:01:57 http: response.WriteHeader on hijacked connection
2020/01/28 01:01:57 http: response.Write on hijacked connection
2020/01/28 01:01:57 [023] WARN: Websocket error: <nil>
2020/01/28 01:01:57 [023] WARN: Websocket error: readfrom tcp 172.17.0.10:4400->172.17.0.4:44940: read tcp 172.17.0.10:60774->10.96.158.190:4444: use of closed network connection
2020/01/28 01:01:57 http: response.WriteHeader on hijacked connection
2020/01/28 01:01:57 http: response.Write on hijacked connection
2020/01/28 01:02:04 [026] WARN: Websocket error: <nil>
2020/01/28 01:02:04 [026] WARN: Websocket error: readfrom tcp 172.17.0.10:4400->172.17.0.4:45088: read tcp 172.17.0.10:60824->10.96.158.190:4444: use of closed network connection
2020/01/28 01:02:04 http: response.WriteHeader on hijacked connection
2020/01/28 01:02:04 http: response.Write on hijacked connection
2020/01/28 01:02:10 [029] WARN: Websocket error: <nil>
2020/01/28 01:02:10 [029] WARN: Websocket error: <nil>
2020/01/28 01:02:10 http: response.WriteHeader on hijacked connection
2020/01/28 01:02:10 http: response.Write on hijacked connection

[sleshchenko]

JWTProxy for some reason doesn't proxy some requests. See the following picture, each connect is browser tab opening, where debug appears - it's where che-machine-exec received connect WebSocket message:

It's needed to investigate JWTProxy + CloudShell frontend possible issues.

[metlos]

Note that now that single user also uses jwtproxy (in a passthrough mode), this issue is present in single user mode as well.

[tolusha]

@sleshchenko
Do you have still problem with that?

[sleshchenko]

@tolusha yeap. CloudShell is still not stable when you run workspace with Che Server. But it's not critical since nobody really uses it at this point. So, there is some issues with integration with JWTProxy.

[che-bot]

Issues go stale after 180 days of inactivity. lifecycle/stale issues rot after an additional 7 days of inactivity and eventually close.

Mark the issue as fresh with /remove-lifecycle stale in a new comment.

If this issue is safe to close now please do so.

Moderators: Add lifecycle/frozen label to avoid stale mode.