Serial support

[yanzixiang]

Is there any plan to support serial ?
I think UART can also treat as stream,
So can we manage it just as socket ?

[njsmith]

Supporting serial in one way or another would be great, though unfortunately it's not something I'm super familiar with myself. My impression from PySerial is that the code to support this would be somewhat complicated and specialized, so it might be something that makes more sense to keep in a specialized library on top of trio, rather than adding an API directly to trio itself? Certainly it makes sense to make it as similar as possible to a using a socket, for users, and we should reuse the trio.abc.Stream API. But I believe the operating system APIs for serial support are generally their own special things, so we couldn't literally reuse the socket code.

Looking at the code for asyncserial, it looks like on Unix it would be pretty straightforward to write a basic, trio-friendly serial Stream on top of PySerial. On Windows, it would be a bit more difficult, but also doable – the issue is that on Windows you need to use IOCP, and the trio core has support for this in principle but no one has actually used it yet so it probably needs some work.

Serial support isn't something that I anticipate having the time to work on soon myself, but if others want to then hopefully that gives some idea where to start, and of course please ask here or in chat if you need more information or get stuck anywhere.

[yanzixiang]

I got this link,
http://pyserial-asyncio.readthedocs.io/en/latest/shortintro.html

this code use asyncio.get_event_loop()

how can this run at the same with trio.open_nursery() ?

[imrn]

What is different from socket other than setup?

[njsmith]

@yanzixiang that code is very entangled with asyncio's way of doing things. I suppose you could probably use it with trio-asyncio, but then you're basically writing asyncio code instead of trio code. My suggestion was more to look at how asyncserial is implemented, internally, and then implement something similar using trio. This might be challenging as a first project, but otoh you'd certainly learn a lot about how trio works :-)

@imrn after setup, on Unix serial ports use os.read/os.write rather than socket.recv/socket.send, and on Windows the APIs are totally different again.

[njsmith]

@yanzixiang by the way, thanks for the link to pyserial-asyncio, I hadn't seen that one before. I just looked at its source code, and I think asyncserial is probably a better library to look at for inspiration. Pyserial-asyncio uses asyncio's complicated and low-level transport/protocol system, while asyncserial uses a much simpler stream API that's similar to trio's.

[yanzixiang]

@njsmith there is not DOC or DEMOCODE on asyncserial's site,
so i got pyserial-asyncio.

[Joker-vD]

@njsmith IIRC on Windows serial ports use ReadFile/WriteFile just as normal files do, the main trouble is figuring out the right parameters to pass to SetCommState before starting the communication. Also, I vaguely recall hearing stories about some timeout configuration shenanigans being necessary, but I'm not sure.

[njsmith]

@Joker-vD That's for synchronous read/write :-). In an async library, you have to use IOCP.

[Joker-vD]

@njsmith Well, you use them for starting asynchronous reads/writes too :-) The completion is reported via an IOCP. I reckon there also has to be some asynchronous API for watching for RTS/DTS/EOF/whatever other events happening with the COM port.

[shuckc]

Subscribed - I am interested in using serial streams within trio, at least on Linux

[njsmith]

Note that we do have the infrastructure to handle IOCP ReadFile/WriteFile support now. (We use it for talking to subprocesses on Windows, and talking to serial ports is very similar.)

[njsmith]

@bsdis has recently been having some adventures trying to get basic serial support working in trio by first configuring the port with pyserial, and then pulling out the fileno and using it in a trio FdStream, like:

stream = FdStream(os.dup(pyserial_object.fileno())

Along the way, we discovered a weird corner of the serial API: apparently posix serial ports have their own unique form of non-blocking mode, where "no data available" is signaled by read returning 0, like an EOF, instead returning -1 and setting errno to EWOULDBLOCK like you'd expect. And apparently this weird thing is considered so normal in the serial world that PySerial enables it unconditionally (!). Of course FdStream is expecting the EWOULDBLOCK behavior and gets very confused.

As a workaround, I suggested:

async def iterate_over_serial_fdstream(stream):
    while True:
        data = await stream.receive_some(65536) # arbitrary number; sets buffer size
        if not data:
            # spurious EOF, wait until the kernel says data is available and try again
            await trio.hazmat.wait_readable(stream.fileno())
        else:
            yield data

# Usage:
async for data in iterate_over_serial_fdstream(stream):
    print(data)

Apparently this works, so that's great. I guess a more serious approach would be to either bake this into a SerialStream object, or else figure out how to configure the serial port to not do this. Though one of the SO answers below makes it sound like there are lots of real-world serial ports that don't report EOF correctly, and apparently all the existing popular libraries always run in this weird spurious-EOF mode, so maybe that's just the way to go.

References:

• https://stackoverflow.com/questions/25996171/linux-blocking-vs-non-blocking-serial-read
• https://stackoverflow.com/questions/1754621/reading-from-serial-port-fails
• Gitter thread where we debugged this: https://gitter.im/python-trio/general?at=5d56586090bba62a128540f4

[Joker-vD]

Welcome to the wonderful world of General Terminal Interface. Behold the magnificent leftovers from the hectic '70s where talking to a device over a serial line was a special and unique experience, and each device had its own charming ideas about how bits and bytes related to each other.
It truly makes you appreciate how wonderfully elegant and high-level the BSD socket API is in comparison.

[njsmith]

@Joker-vD Huh, yeah, it says it right there in the POSIX spec:

IEEE Std 1003.1-2001 does not specify whether the setting of O_NONBLOCK takes precedence over MIN or TIME settings. Therefore, if O_NONBLOCK is set, read() may return immediately, regardless of the setting of MIN or TIME. Also, if no data is available, read() may either return 0, or return -1 with errno set to [EAGAIN].

i.e.: read don't care, read does what it wants! So maybe the best you can do is to just treat all EOFs as if they were EAGAIN. I would guess that modern systems might accept VMIN=1 + VTIME=0 + O_NONBLOCK as an incantation to mean "please stop the tomfoolery and act like a normal non-blocking file descriptor", but that's not required by the spec.

[Joker-vD]

@njsmith I suppose the semantics are somewhat like sematics of disk files: i.e., if you read and there is no data, that's EOF, not EAGAIN, even though the next read may read some data that wasn't previously there.

[joernheissler]

Hello,
I created a trio-serial package. It works for me so far, but only (partially) tested with Linux. Several features aren't implemented yet, docs could be improved a lot, etc.
So far there is no support for non-posix platforms like Windows. If someone needs this, they'll have to implement it themselves.

Docs: https://trio-serial.1e8.de/
Repo: https://github.com/joernheissler/trio-serial
Pip: https://pypi.org/project/trio-serial/

[raveslave]

did anyone look at libusb as an alternative to pyserial?

[njsmith]

Does libusb have an api for talking to serial drivers? Speaking usb is definitely a useful thing that we might want bindings for, but I thought it was a different thing from serial support.

…

On Tue, Apr 13, 2021, 13:31 david eriksson ***@***.***> wrote: did anyone look at libusb as an alternative to pyserial? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#549 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAEU42GLLBQO2YWW5VVDIQTTISS2LANCNFSM4FFLKYAA> .

[raveslave]

as far as I know, no CDC_ACM wrappers that comes with libusb.
if anything "usb" should be supported in trio, I think libusb is most suitable as it provides a generic direct access to devices, descriptors and endpoints, allowing to run serial CDC as well as mass-storage, or any other device class.

a good example might be when you need to interact with various hardware devices where some might run a proprietary (but documented) device class, being able to provide fairly low amount of glue.

[smurfix]

There are sufficiently many serial ports (e.g. embedded consoles, /dev/ttyACM*) or "ports" (like /dev/pts) that can't be talked to with generic USB-serial commands.

So why would you want to re-implement serial handling for a narrow subset of devices in userspace?

[raveslave]

the benefits of libusb is primarily that we have one way to discover (regardless of class-type) and talk to various usb device-classes by searching for VID and PID rather than a random port-name that is different on various OS'es.

you also reach higher throughputs and better control over async behavious which is highly wanted for CDC or custom bulk transfers where speed is needed.

last is that you don't need to rely on various fairly large libraries, i.e. pyserial (works, but has been fighting with port naming issues, especially on mac the way it uses corefoundation), rtmidi for usb-mid (large library for a simple thing), and of course a way to talk to devices that implement vendor-specific serial-ports over a custom bulk descriptor.

[joernheissler]

If you want to access a usb2serial device through libusb, I think you'll need to disable the /dev/tty* driver for the device and give the user permissions for the usb device. And you'd have to implement the serial specific usb protocols. I think this might be quite a lot of effort. Having generic usb support for trio would be great, though.

[raveslave]

you can do that runtime, and CDC isn't much. I think it's a much better approach to keep a Trio implementation fully async and free from 3rd party libs.
example on a CDC
https://github.com/ARMmbed/DAPLink/blob/master/test/usb_cdc.py

[joernheissler]

My trio-serial (https://trio-serial.1e8.de/) package doesn't depend on any third party libs, aside from the (linux) kernel.

usb_cdc.py uses libusb. If you want to do that without any 3rd-party libs, you'd have to interface the (linux) kernel directly, essentially reinventing libusb in python. Sounds like a nice idea, but might be complicated.

[altendky]

I'm confused. It seems like there is overlap between pyserial-supported-things and those that a theoretical libusb based library would support. Am I wrong and a libusb based system would support everything that pyserial could? If not, it seems straightforward that there are two different libraries to be had (or more, of course). One for serial (that would presumably support some USB devices) and one for USB (that would presumably support some serial devices). Or, have I misunderstood something?

[joernheissler]

Yes, two libs are needed. USB support is a great idea, but it should be dealt with in a different GH issue.