HTTP/1.1 bidirectional streaming

[graingert]

Checklist

• There are no similar issues or pull requests for this yet. HTTP/2 bi-directional streaming. #1150 discusses http 2 bidirectional streaming
• I discussed this idea on the community chat and feedback is positive.

Is your feature related to a problem? Please describe.

I have an http server implementing a remote shell, think zork or zsh. When a client connects it immediately responds with "200 OK" before consuming the request body

Here's an example echo server:

python
import anyio

from anyio.streams.buffered import BufferedByteReceiveStream
import logging


logger = logging.getLogger(__name__)


async def handle(client):
    try:
        async with client:
            buffer = BufferedByteReceiveStream(client)
            msg = await buffer.receive_until(b"\r\n\r\n", 100_000)
            print(msg)
            await client.send(
                b"HTTP/1.1 200 OK\r\n"
                b"Connection: Keep-Alive\r\n"
                b"Transfer-Encoding: chunked\r\n"
                b"\r\n"
            )
            while True:
                rcv = await buffer.receive()
                await client.send(rcv)
    except Exception:
        logger.exception("failed")


async def serve():
    listener = await anyio.create_tcp_listener(local_port=8888)
    await listener.serve(handle)


anyio.run(serve)

Describe the solution you would like.

I want to be able to able to await for a response resolving when the headers have been received before the body has been sent. I'd also like to be able to send and receive data without passing a generator as the http body

Additional context

some sort of interface like this

async with client.stream_bidirectional(method="GET", url="https://example.com") as request:
    response = await request.receive_headers()
    assert response.status_code == 200
    await request.send(b"echo 'hello'")
    assert await request.receive_until(b"\n", 10_000) == b"hello"
    await request.send(b"exit")
    assert await request.receive_exactly(1) == b"\n"

[tomchristie]

I'd suggest anything here would likely be folded into #1150.

I'm not at all sure that we'd support this over HTTP/1.1 since it's not really designed to be used in that way, and you can't reliably know that servers or intermediaries (proxies, gateways, etc) would support it (they typically won't). In contrast to HTTP/2 which does. But in either case it's not really feasibly on our radar at this point in time.

[florimondmanca]

Closing for housekeeping… I agree with Tom, don't think this could be within our scope right now. Also because this is would be non-standard HTTP/1.1, meaning in any language you'd need to do something custom with most clients anyway.

FWIW, if that's really what you need and you're fine getting your hands dirty, then feel free to look into the HTTPCore source code and duplicate and play with the HTTP/1.1 code to accomodate your use case.

(Or perhaps more realistically, consider updating the server implementation to do simpler TCP socket networking, without any HTTP at all, and then do that on the client as well?)

[tomchristie]

We do now support functionality through the "network_stream" extension.

See also ticket #2599 for documenting this properly in httpx, rather than only in httpcore.