Fix image tests: vncserver, websockify, jupyter-remote-desktop-proxy

[consideRatio]

This is work done in parallell to #93, initially thought to be a quick fix to ensure TurboVNC also works. Like @yuvipanda in #93 I ran into issues with different behavior locally and within the GitHub actions environment, but think the biggest difference got resolved by providing a TTY device.

@yuvipanda I propose #93 is completed in a dedicated job inside the test.yaml workflow, side by side to the image-test job finalized in this PR - and then finally in an optional dedicated PR we prune misc bash things I've done in favor of python things you've done.

I consider this to fix #98, which was bugfixed by #99 but not tested by automation to make it work as it is now to some extent at least.

Summary of misc changes

• added test for TurboVNC image in parallel to test for TigerVNC image
• added websockify --help and vncserver --help checks
• added a --fail flag to curl calls (detected by @yuvipanda in Add an integration test for VNC + websockify #93)
• added a test of vncserver startup
• added a test of websockify'ed vncserver startup
• added a test of jupyter-remote-desktop-proxy startup
  • typically fails once, then succeeds - tracked by Initial websocket request can fail #105
• provided a TTY when executing steps - a requirement for vncserver being started
• concluded that there is probably also ipv4 / ipv6 behaviors differing between my local computer and github actions environment, and it could influence testing of an isolated vncserver or websockify'ed vncserver, but not a jupyter server proxying via jupyter-server-proxy.

A blob of old notes, I've worked a lot of hours 😱

Debugging the same issue Yuvi ran into I think:

This works perfectly fine locally, but unfortunately something is causing it to hang when the VNC client is trying to connect to the server on GitHub actions. will have to debug next time i get a chance.

I've debugged this at length with minor progress.

First attempt failure

I've concluded that the initial websocket connection attempt often fails to get sensible responses etc, and I think this relates to jupyter-server-proxy finalizes a websocket handshake even if the handshake to the backend isn't finalized (jupyterhub/jupyter-server-proxy#459). But this shouldn't cause issues for the second attempt that typically works locally.

Future attempt failures

The difference I observe locally and remote, comes down to this, where the green parts represent local and red parts represents failing remote test runs in github actions.

(ignore timestamps, this is a composed set of log lines)

websocat output

 [DEBUG websocat] Done third phase of interpreting options.
 [DEBUG websocat] Done fourth phase of interpreting options.
 [DEBUG websocat] Preparation done. Now actually starting.
 [DEBUG websocat::sessionserve] Serving ThreadedStdio to WsClient("ws://localhost:8888/desktop-websockify/?token=secret") with Options { websocket_text_mode: false, websocket_protocol: None, websocket_reply_protocol: None, udp_oneshot_mode: false, udp_broadcast: false, udp_multicast_loop: false, udp_ttl: None, udp_join_multicast_addr: [], udp_join_multicast_iface_v4: [], udp_join_multicast_iface_v6: [], udp_reuseaddr: false, unidirectional: false, unidirectional_reverse: false, max_messages: None, max_messages_rev: None, exit_on_eof: true, oneshot: false, unlink_unix_socket: false, unix_socket_accept_from_fd: false, exec_args: [], ws_c_uri: "ws://0.0.0.0/", linemode_strip_newlines: false, linemode_strict: false, origin: None, custom_headers: [], custom_reply_headers: [], websocket_version: None, websocket_dont_close: false, websocket_ignore_zeromsg: false, one_message: true, no_auto_linemode: false, buffer_size: 65536, broadcast_queue_len: 16, read_debt_handling: Silent, linemode_zero_terminated: false, restrict_uri: None, serve_static_files: [], exec_set_env: false, no_exit_on_zeromsg: false, reuser_send_zero_msg_on_disconnect: false, process_zero_sighup: false, process_exit_sighup: false, process_exit_on_disconnect: false, socks_destination: None, auto_socks5: None, socks5_bind_script: None, tls_domain: None, tls_insecure: false, headers_to_env: [], max_parallel_conns: None, ws_ping_interval: None, ws_ping_timeout: None, request_uri: None, request_method: None, request_headers: [], autoreconnect_delay_millis: 20, ws_text_prefix: None, ws_binary_prefix: None, ws_binary_base64: false, ws_text_base64: false, close_status_code: None, close_reason: None, asyncstdio: false, foreachmsg_wait_reads: false, announce_listens: false, timestamp_monotonic: false, print_ping_rtts: false, byte_to_exit_on: 28, max_ws_message_length: 209715200, max_ws_frame_length: 104857600, preamble: [], preamble_reverse: [], compress_deflate: false, compress_zlib: false, compress_gzip: false, uncompress_deflate: false, uncompress_zlib: false, uncompress_gzip: false, jsonrpc_omit_jsonrpc: false, inhibit_pongs: None, max_sent_pings: None }
 [INFO  websocat::stdio_threaded_peer] get_stdio_peer (threaded)
 [INFO  websocat::ws_client_peer] get_ws_client_peer
 [INFO  websocat::ws_client_peer] Connected to ws
+[DEBUG websocat::ws_peer] incoming binary
+[DEBUG websocat::readdebt] Fulfilling the debt of 12 bytes
+[DEBUG websocat::my_copy] Once mode requested, so aborting copy
+[DEBUG websocat::my_copy] done
+RFB 003.008
+[INFO  websocat::sessionserve] Reverse finished
+[DEBUG websocat::sessionserve] Reverse shutdown finished
+[INFO  websocat::sessionserve] One of directions finished
+[DEBUG websocat::ws_peer] drop WsWriteWrapper
-[DEBUG websocat::my_copy] zero len
-[DEBUG websocat::my_copy] read_done
-[DEBUG websocat::my_copy] done
-[INFO  websocat::sessionserve] Forward finished
-[DEBUG websocat::sessionserve] Forward shutdown finished
-[DEBUG websocat::ws_peer] drop WsWriteWrapper
-[INFO  websocat::sessionserve] One of directions finished

jupyter_server logs

 [I 2024-03-10 22:04:21.890 ServerApp] Trying to establish websocket connection to ws://localhost:56389/?token=secret
 127.0.0.1: new handler Process
 127.0.0.1 - - [10/Mar/2024 22:04:21] "GET /?token=secret HTTP/1.1" 101 -
 127.0.0.1 - - [10/Mar/2024 22:04:21] 127.0.0.1: Plain non-SSL (ws://) WebSocket connection
 127.0.0.1 - - [10/Mar/2024 22:04:21] 127.0.0.1: Path: '/?token=secret'
 127.0.0.1 - - [10/Mar/2024 22:04:21] connecting to command: '/bin/sh -c cd /home/jovyan && /usr/bin/vncserver -rfbunixpath 
 /tmp/tmpfm_suyv2/vnc-socket -xstartup /opt/install/jupyter_remote_desktop_proxy/share/xstartup -verbose -localhost -fg -geometry 1680x1050 -SecurityTypes None' (port 59467)
 [I 2024-03-10 22:04:21.894 ServerApp] Websocket connection established to ws://localhost:56389/?token=secret
 127.0.0.1 - - [10/Mar/2024 22:04:21] 127.0.0.1:59467: Client closed connection
 127.0.0.1 - - [10/Mar/2024 22:04:21] 127.0.0.1:59467: Closed target

websockify logs

  127.0.0.1: new handler Process
  127.0.0.1 - - [10/Mar/2024 21:15:57] "GET /?token=secret HTTP/1.1" 101 -
  127.0.0.1 - - [10/Mar/2024 21:15:57] 127.0.0.1: Plain non-SSL (ws://) WebSocket connection
  127.0.0.1 - - [10/Mar/2024 21:15:57] 127.0.0.1: Path: '/?token=secret'
  127.0.0.1 - - [10/Mar/2024 21:15:57] connecting to command: '/bin/sh -c cd /home/jovyan && /usr/bin/vncserver -rfbunixpath /tmp/tmpp4fup6x4/vnc-socket -xstartup /opt/install/jupyter_remote_desktop_proxy/share/xstartup -verbose -localhost -fg -geometry 1680x1050 -SecurityTypes None' (port 35719)
  127.0.0.1 - - [10/Mar/2024 21:15:57] 127.0.0.1:35719: Client closed connection
  127.0.0.1 - - [10/Mar/2024 21:15:57] 127.0.0.1:35719: Closed target

vncserver logs

  Sun Mar 10 21:07:15 2024
   Connections: accepted: /tmp/tmpn0g6yfd7/vnc-socket
+  VNCSConnST:  closing /tmp/tmp6wsxj7wa/vnc-socket: Clean disconnection
-  VNCSConnST:  closing /tmp/tmpn0g6yfd7/vnc-socket: read: Connection reset by
-               peer (104)
   EncodeManager: Framebuffer updates: 0
   EncodeManager:   Total: 0 rects, 0 pixels
   EncodeManager:          0 B (1:-nan ratio)
   Connections: closed: /tmp/tmpn0g6yfd7/vnc-socket
   ComparingUpdateTracker: 0 pixels in / 0 pixels out
   ComparingUpdateTracker: (1:-nan ratio)

[yuvipanda]

just casually passing by but i must say i absolutely love the commit messages

[consideRatio]

@yuvipanda I think this may be what you need to get things working locally for you work also in github actions - it made the a difference to me.

I think also there are ipv4 / ipv6 differences, and I now know there were TigerVNC and TurboVNC differences with regards to listening on localhost or not by default, and that websockify will make things listen to localhost anyhow by intercepting the port bind system call or something like that - its messy!

Anyhow, with this, I got it working fully finally.

[consideRatio]

@yuvipanda I'll go for a merge here to trigger a rebuild with jupyterhub 4.1.4 instead of current 4.1.3 i the published images in the main branch, and then see if I can get it to function with the current xsrf complexities when booted with jupyterhub-singleuser - which it doesn't currently with jupyterhub 4.1.3 in the image at least.