quic transport

[totaam]

QUIC could easily replace the old UDP transport.
There are frameworks and APIs that wrap QUIC with easy to use callbacks, like https://github.com/aiortc/aioquic documentation
(see also https://github.com/MagicStack/uvloop)
We would get HTTP/3 and encryption almost for free and a reliable event based framework for managing connection state.

Examples include an http3 server and clients, etc..

The re-send / packet-drop callbacks could easily re-use the techniques that were used for UDP (#639) which was removed in 2022238.

---

Worth noting that aioquic requires Python 3.7 or better, and the OpenSSL development headers so this won't be available on RHEL 8.

[totaam]

aioquic is not available in Fedora and the number of dependencies required to run the http3 server example is a little bit ridiculous: python3-wsproto, python3-httpbin, python3-asgiref, python3-starlette and python3-aiofiles.
Not counting all the dependencies that each one of these packages brings...

[totaam]

It gets worse since we also need to run the GTK main loop until we can drop GTK (#1995).
There are various ways of doing this:

• asyncio-glib
• gbulb

And perhaps soon in pygobject directly: Implement asyncio event loop based on glib

Next, creating asyncio sockets requires using a specific API: asyncio.loop.create_datagram_endpoint / loop.create_connection, which means that socket sharing is going to be more difficult to implement.

This may help in bridging with the rest of the code:

• trio : a friendly Python library for async concurrency and I/O
• greenback : allows you to call back into async code from a syntactically synchronous function

[TijZwa]

I've done some testing with Aioquic. Basically I've just glued asyncio with aioquic to the existing codebase.
The main issue is starting a new main loop inside a thread, inside the existing GTK loop.
Also the list of dependencies is way too big (starlette and what not).

The source of aioquic seems pretty neat. I think the best way forward is to use this code as an example to implement our own low-level implementation of a http/3 quic server.

https://github.com/aiortc/aioquic/tree/main/src/aioquic

[totaam]

as an example to implement our own low-level implementation of a http/3 quic server

Yes!
This part of the library looks really clean and, at first glance, without too many dependencies, only things like https://github.com/aiortc/pylsqpack
Gio supports UDP so there is no reason why we can't use the GTK main loop for handling packet events.

[totaam]

We need an asyncio event loop:
https://github.com/aiortc/aioquic/blob/444be09157aed3c81881d18647484165dd07139c/src/aioquic/asyncio/server.py#L202

The patches for pygobject are still languishing as a PR and asyncio-glib is unmaintained.

And gbulb has had no update since earlier this year.

Then there's also glibcoro: glibcoro makes the GLib / GTK
event loop compatible with Python’s asyncio

Meanwhile, this answer: https://stackoverflow.com/a/26719999/428751 suggests running the aio part in a separate thread. Downside is: One does need to be careful, however, to use thread-safe functions to communicate between the GUI (main) thread and the asyncio thread. This means using loop.call_soon_threadsafe or asyncio.run_coroutine_threadsafe to submit things to asyncio from GTK, and GLib.idle_add to submit things to GTK from asyncio.

Decisions!

[totaam]

There's still a lot to do:

• try to run with a single aio loop instead of using threading in quic_queue_server? (multiprocessing?)
  • then we don't have to use aioquic.asyncio.server.serve and we can use lower level APIs like QuicServer.datagram_received from GLib IO events?
• reuse list_directory in the "old" http handler
• handle POST requests? HEAD?
• handle quic_socket.close - at least exit quic_queue_server when all sockets are closed
• websocket handler
• add options for QuicConfiguration
  • server_name should come from ssl options?
  • max_data / max_stream_data should be higher?
• redirect quic_logger to our own logging framework (with remote logging)
• simplify http3_server.py, remove wsproto?
• override FrameProtocol._serialize_frame with our faster Cython mask version?
• in WebSocketHandler.send call self.connection.send_data repeatedly instead of concatenating header, mask and data
• hook http authentication - how!?
• add quic client support
• test with a Javascript WebTransport client - see also WebTransport Explainer, Using WebTransport
• move screen updates to a separate lossy stream?
• support different session ticket factories?
• quic can bind to unix socket - not sure what to do with that

---

xpra start --no-daemon -d quic --start=xterm
    --bind-tcp=0.0.0.0:10000
    --bind-quic=localhost:20000 --ssl-cert=./cert.pem --ssl-key=./key.pem

Allows http3 clients to connect:

./examples/http3_client.py "https://localhost:20000/Sessions?hello=foo&bar=2" -k -v --output-dir=./output

---

Apparently, chrome needs to be told to use the QUIC protocol

[totaam]

As of d4c7fc8, I can connect via quic using a self signed cert:

xpra start --bind-quic=localhost:20000 --start=xterm :100 \
    --ssl-cert=./cert.pem --ssl-key=./key.pem

xpra attach quic://localhost:20000/ --ssl-server-verify-mode=none

[totaam]

Looks like asyncio integration with support to await Gio async functions is going to be merged!
But we'll have to keep the threaded implementation around until all the supported distros ship with the new pygobject... which could take many years.

[totaam]

Looks like asyncio will be integrated with pygobject Gio soon: asyncio integration with support to await Gio async functions so we will be able to drop the ugly asyncio_thread workaround in the future.

[totaam]

WebTransport can now be used with the html5 client:
Xpra-org/xpra-html5#143 (comment)
It would be nice to also support a xpra attach webtransport://host:port/ mode for the Python client.

[totaam]

TLDR excerpts from the QUIC is not Quick Enough over Fast Internet paper:

• We find that over fast Internet, the UDP+QUIC+HTTP/3 stack suffers a data rate reduction of up to 45.2% compared to the TCP+TLS+HTTP/2 counterpart
• QUIC Perceives Much More Packets than HTTP/2
• QUIC has a much Higher RTT Dominated by Local Processing
• the root cause to be the pronounced receiver-side processing
  overhead. This overhead manifests in the form of excessive data
  packets observed at Layer 3 and above, as well as QUIC’s distinctive
  user-space ACKs

Fast internet is defined as >500 Mbps
I think this all makes sense. It will get better over time, and we care more about latency than outright maximum throughput.