[WIP] Memory Streams

[asmodehn]

A small step, to ultimately address #170
Exposing memory streams similarly to tcp streams to be able to test network code without using any actual sockets.

Still WIP.
No docs and basic tests (check_half_closeable_stream) still blocking around https://github.com/python-trio/trio/blob/master/trio/testing/_check_streams.py#L275

[asmodehn]

It seems to be because :

• basic tests (duplicated from test_highlevel_socket) are trying to fill up the memory (https://github.com/python-trio/trio/blob/master/trio/tests/test_highlevel_socket.py#L133)
• memory streams are implemented with an unbounded queue
  So :

self = <trio.testing._memory_streams._UnboundedByteQueue object at 0x7f24fa5d99b0>
data = bytearray(b'xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx...xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx')

    def put(self, data):
        if self._closed:
            raise _core.ClosedResourceError("virtual connection closed")
>       self._data += data
E       MemoryError

trio/testing/_memory_streams.py:47: MemoryError

Should we bound memory queues ?

[njsmith]

Hi! Thanks for working on this!

Let's start by figuring out what the plan is at a high level. Right now, the idea in this PR seems to be that people will write open_tcp_stream(..., testing=True), or serve_tcp(..., testing=True). But that seems pretty awkward, especially in cases where the person writing the tests doesn't actually control the networking code. (For example, maybe I'm trying to write a test using an HTTP library like requests – I would need to have some way to tell requests that it should pass testing=True.)

So, the idea we've been working on is a bit different. Trio has hooks (documented here) that let you override all networking inside a Trio program, in a single step. It works by overriding how Trio's sockets work, so it works for programs that use requests, it works for programs that use open_tcp_stream, and it works for programs that ignore all those fancy abstractions and use trio.socket directly. I think this is what we want to end up doing. And the nice thing about this is that we don't have to expose a bunch of new public APIs like testing=True... the only public part is with virtual_network(): ... or something, and then all the magic can happen behind the scenes.

Does that make sense?

BTW, depending on how complex this is, it might make sense to split it out as a separate library. We don't have to decide that up front, and cookiecutter-trio makes it pretty quick to set up a new library, but it's something to keep in mind.

[asmodehn]

Very welcome, I m looking forward the day that I can debug most of my network code without setting up an actual network.

Oh interesting... Having network configuration (layout, deploy targets, etc.) as part of the context, definitely makes sense. I ll have a look at how hooks work next, then.

At a higher level, I am playing around with trio-protocol, as the Protocol class (along with Transport) is apparently the interface used for sans-io things. I'm using ohneio currently. It makes sense for me so far, as a stable, IO-agnostic, intermediate API.
Once I get a sensible interface in trio for in-memory streams, I'm planning to have exhaustive testing in trio, and more functional/user level examples addressing miracle2k/trio-protocol#2 in trio-protocol.

I imagine part of the whole feature will be in trio's code, as a generic way to support various abstract, yet related, functionalities, and a specific polished implementation in a separate library. No idea yet of where to draw the line between these two parts however.

[njsmith]

I definitely recommend using the sans-io approach, it's great. (And you'll want to read #796 for more discussion.) But usually when we talk about sans-io, we don't mean asyncio protocols/transports; we mean libraries like h11/h2/wsproto, which use a fairly different API style. And the Trio equivalent of protocols/transports is the Stream API, which is much more friendly to composition than the traditional protocol/transport approach.

If you want to see examples of sans-io libraries hooked up to trio, then I'd suggest looking at:

• this example from the h11 docs, especially the TrioHTTPWrapper that wires together a trio.abc.Stream with an h11.Connection.
• trio-websocket (which is based on wsproto)

[njsmith]

I'm going to mark this closed for now b/c it sounds like you're rethinking things, so there isn't anything to review currently. Feel free to re-open after you do, or open a new PR, or whatever makes most sense :-).