Fix Broken SockFlag::SOCK_NONBLOCK and SockFlag::SOCK_CLOEXEC on MacOS/iOS

[kristate]

SockFlag::SOCK_NONBLOCK and SockFlag::SOCK_CLOEXEC in 0.9.0 broke macOS/iOS support in 0.10.0 (f067e73)

This commit solves the issue and also provides test cases to test for continued support.

• SockFlag::SOCK_NONBLOCK
• SockFlag::SOCK_CLOEXEC
• Tests

ref #244 #861

[kristate]

Note here: nix::sys::socket::socket fn API requires the linux specific SockFlag::SOCK_NONBLOCK and SockFlag::SOCK_CLOEXEC which is not included in macOS' libc -- the work around is to provide a bitflag that mimics such support.

[Susurrus]

Why can't this be fixed in libc? Do these constants not exist on OS X and iOS?

[kristate]

This is fundamentally an implementation concern.

It's because the nix::sys::socket::socket fn is not only assigning a socket, but also setting flags on the socket in the same function to match parity with Linux (which is an improvement). The normal API is defined in socket(2) as:

int socket(int domain, int type, int protocol);

linux diverts from POSIX and BSD by supplying a SOCK_NONBLOCK flag as is defined in linux man socket(2) as:

SOCK_NONBLOCK
Set the O_NONBLOCK file status flag on the new open
file description.  Using this flag saves extra calls
to fcntl(2) to achieve the same result.

SOCK_CLOEXEC
Set the close-on-exec (FD_CLOEXEC) flag on the new
file descriptor.  See the description of the
O_CLOEXEC flag in open(2) for reasons why this may be
useful.

In summery:

• SOCK_NONBLOCK and SOCK_CLOEXEC are in linux since 2.6.27
• Some BSDs have conformed to this new API and such have a define in libc.
• macOS is the exception and sticks to directly calling fcntl(2)
• nix takes the high road and also only calls fcntl(2) with a new fourth parameter on the function call

Because nix provides a new API that does not conform to POSIX or BSD (or directly Linux), this patch creates virtual definitions for macOS and iOS that are backed by their equivalent fcntl flags.

I think this is the best course of action for two reasons:

0. Redefining the function at this stage would break compatibility
1. Defining virtual definitions for SOCK_NONBLOCK and SOCK_CLOEXEC are only used inside of nix and only for the socket fn.

[kristate]

For the sake of conversation, the alternative is:

Removing pub struct SockFlag and having the API user call fcntl directly.

Thus, these function defs need to be changed (breaking downstream code):

pub fn socket<T: Into<Option<SockProtocol>>>(domain: AddressFamily, ty: SockType, flags: SockFlag, protocol: T) -> Result<RawFd> {

/// Create a pair of connected sockets
///
/// [Further reading](http://pubs.opengroup.org/onlinepubs/9699919799/functions/socketpair.html)
pub fn socketpair<T: Into<Option<SockProtocol>>>(domain: AddressFamily, ty: SockType, protocol: T, flags: SockFlag) -> Result<(RawFd, RawFd)> {

/// Accept a connection on a socket
///
/// [Further reading](http://man7.org/linux/man-pages/man2/accept.2.html)
pub fn accept4(sockfd: RawFd, flags: SockFlag) -> Result<RawFd> {

fn accept4_polyfill(sockfd: RawFd, flags: SockFlag) -> Result<RawFd> {

[Susurrus]

I just searched all of /usr/include on my mac, and I don't see any SOCK_NONBLOCK and SOCK_CLOEXEC flags declared. So I'm not certain how async functionality is exposed on Mac, but this doesn't seem to be the right path forward here.

[kristate]

@Susurrus I replied to your comment above here: #863 (comment)

I just searched all of /usr/include on my mac, and I don't see any SOCK_NONBLOCK and SOCK_CLOEXEC flags declared. So I'm not certain how async functionality is exposed on Mac, but this doesn't seem to be the right path forward here.

Please read my above comment and then let me know is the right path forward. I put in the work to support macOS and iOS and am willing to maintain it.

Thanks.

[Susurrus]

Okay, so this is all pretty interesting. The code as it stands at HEAD is incorrect and this code doesn't fix the problem properly as well I don't think.

What we have is that the SOCK_NONBLOCK and SOCK_CLOEXEC options are not properly past to socket, but instead are always emulated on all platforms, even those that support them properly! Basically on a few platforms there is no native support for this. Namely:

• Linux pre-2.6.27 (released in 2008)
• OpenBSD pre-5.7 (released in 2015)
• DragonFlyBSD pre-4.4 (released in 2015)
• NetBSD pre-6.0 (released in 2012)
• FreeBSD pre-2013 (unknown what release, commit adding this is from early 2013)

But what we actually do is instead of allowing the user to use these values directly on support platforms, what nix actually do is emulate them by including calls to fcntl setting the O_NONBLOCK or FD_CLOEXEC flags as appropriate.

So I think this is part of a larger issue here of exposing these APIs. @asomers any input here?

I think what we should do is disable this emulation for all platforms and only emulate it on mac/ios. Additionally I think that on mac/ios we shouldn't reuse the O_NONBLOCK and FD_CLOEXEC values and just use arbitrary ones, since they're discarded anyways. Otherwise it's confusing semantics.

[kristate]

@Susurrus thanks for the reply.

Again to reiterate from my earlier comment (which did not receive a reply):

nix provides a new API that does not conform to POSIX or BSD (or directly Linux), this patch creates virtual definitions for macOS and iOS that are backed by their equivalent fcntl flags, but these definitions have their own typedef/struct that can only be used on socket, so I am not worried about the values being used incorrectly outside of the nix API.

This patch solves the bug and I think it is the best course of action for two reasons:

0. Redefining the function at this stage would break compatibility
1. Defining virtual definitions for SOCK_NONBLOCK and SOCK_CLOEXEC are only used inside of nix and only for the socket fn. Most C network event libraries provide a wrapper layer to create a socket and call fcntl and friends.

That said, it might be smart to also provide access to fcntl,
but is that not already available via libc ??

[asomers]

I think that Nix's purpose is to expose OS functionality in a Rusty way. I don't think we should try to emulate missing OS functionality, especially since we can't do it correctly (the emulated socket flags are not set atomically). So I'm in favor of removing the emulation altogether, even on OSX. And BTW, we already have a wrapper for fcntl. It's in src/fcntl.rs .

[Susurrus]

@kristate I understand your rationale for wanting this feature as I did read your original reply, but I disagree that this functionality should exist. Regarding your two points:
0. We do not guarantee backwards compatibility at this point because we aren't very set on many of our APIs. This is indicated by our pre-1.0 versioning. In fact the Changed section of our CHANGELOG are all APIs we've broken compatibility with as we are willing to break compatibility for correctness or better APIs at this stage of development.

1. We are not trying to provide emulation functionality within this library, especially one like this where it's not obvious we're abstracting something away. By reusing those variable names we're hiding things from our users that can have a clear impact. I could easily see someone posting an issue or PR undoing this or changing the API and/or updating the docs to make it clear that we're hiding things here.

So I agree with @asomers above about removing the emulation entirely. It's probably best to close this PR and submit a new one to do that. @kristate if you'd like to do that, we'd appreciate it, otherwise I'll file one later. I want to resolve this issue for the 0.11.0 release.

[kristate]

@asomers @Susurrus Thanks for the replies.

I really think that the API was trying to be rusty from the beginning. I was just fixing a bug.

The problem is that there are two socket APIs. Linux says why make two syscalls to socket and fcntl, so they added a flag so that it could be done in one call. MacOS cannot do this, but some BSDs have support for it and I think that is the original rationale behind this API.

Also, if people really wanted to handle socket and fcntl directly, there is always libc, right?

edit: the readme for this project even states "The goal is to not provide a 100% unified interface, but to unify what can be while still providing platform specific APIs." -- is this not a unified interface for socket ?

[Susurrus]

I really think that the API was trying to be rusty from the beginning. I was just fixing a bug.

But the code is still wrong even after this fix. All BSDs and Linux support these flags natively, so they don't need the emulation, but it's turned on for all BSDs. I'm not certain if we should add the runtime detection for BSDs or we should actually assume that we're running versions at least as new as what I listed above. 2015 releases are still pretty new, but I'm not certain it's worth the effort when we can just document it as minimum requirements for the lib. I'd also be fine dropping the runtime checks for Linux given it was added 10 years ago. @asomers do you agree?

[Susurrus]

As to address you specific question: "the readme for this project even states "The goal is to not provide a 100% unified interface, but to unify what can be while still providing platform specific APIs." -- is this not a unified interface for socket?"

This is one of those not-100% scenarios. We try to provide a unified interface, but we don't try to emulate or approximate functionality. This code change emulates an atomic API with a non-atomic implementation, which I consider bad form. We have always torn out this code and left the API more lacking in these circumstances.

[kristate]

@Susurrus I think I understand your concern now.

So, C programmers are used to having the socket(2) syscall and those who are used to linux expect to have SOCK_NONBLOCK and SOCK_CLOEXEC available to them. If you are used to macOS, then you understand that fcntl must be called after setting up the socket.

The point I am making is that there needs to be a name and a value for the flags to our rust API. SOCK_NONBLOCK and SOCK_CLOEXEC was previously assigned and map well to the linux socket(2), so that is an improvement and well received by C programmers.

I think that the question we have to first ask ourselves is: Does pub fn socket follow linux's lead and allow SOCK_NONBLOCK and SOCK_CLOEXEC to be called as flags (and take care of fcntl behind the scenes) OR do we simply wrap the libc crate and expect users downstream to again wrap code in cfg macros?

"Emulation" is a poor way to think about this problem. We are creating a Rusty API that helps calm the waters when working with unix. I chose to work with nix because it helps stabilize an API to work with unix systems. Likewise, if nix does not have any benefits over libc, it would seems that programmers would chose libc over nix.

So, I leave it to you: Does pub fn socket follow linux's lead and allow SOCK_NONBLOCK and SOCK_CLOEXEC to be called as flags (and take care of fcntl behind the scenes if required) OR do we simply wrap the libc crate and expect users downstream to again wrap code in cfg macros where applicable?

Thanks.

[kristate]

@Susurrus

This code change emulates an atomic API with a non-atomic implementation, which I consider bad form.

Where in my code does this change emulate an atomic API with a non-atomic implementation?

By the way, it does seem to check if the OS will cloexec atomically which was added by @carllerche ref: c976be5

[asomers]

@kristate it's not your code that lacks atomicity; it's the existing code. The whole reason that socket accepts these flags is so that opening the socket and setting the flags happens atomically. Either both happen, or neither does. The existing Nix code doesn't guarantee that. What happens if the program receives a signal after socket but before fcntl? The socket will exist, but the flags won't be set. There's no way to correctly emulate the behavior of these flags in userland; they require kernel support. That's why @Susurrus and I favor eliminating the emulation entirely. Programs that need to run on OSX will have to handle it themselves. And precisely because it's handled at the application level, the application programmer will be aware that those flags don't get set atomically. The value of the Nix crate is not that it provides a consistent cross-platform API; it's that it provides a Rusy interface to all of those unsafe, pointer-ridden C APIs.

@Susurrus all FreeBSD, NetBSD, and OpenBSD releases that don't support these flags are EoL. I'm not sure what Dragonfly's support policy is, but they've had several releases since adding these flags. Linux 2.6.27 is pretty old, but RHEL 5 uses 2.6.18 and it's on extended support until 2020. And of course, no version of OSX supports these flags. Personally, I don't think it would be too bad if Nix ditched all the emulation. Then socket would return EPROTOTYPE if somebody tried to use those flags on OSX or RHEL5.

[kristate]

@asomers I agree with you in principle, but either way fcntl should be called directly after socket if no atomic behavior exists in the kernel, so does that not mean that we are providing the best practice for downstream users?

We have many years of network programming knowledge that we can pass-on downstream to users who might be less aware. Also, I am thinking about adding more documentation to match this discussion.

Thoughts?

[Susurrus]

should be called directly after socket if no atomic behavior exists in the kernel, so does that not mean that we are providing the best practice for downstream users?

Your implication is that we should be doing this and we've been explaining that we don't believe we should. There are numerous technical reasons why which we've enumerated previously. We all would like if we could have a nice API for this across all platforms, but that's not how things are unfortunately. So what we try to do is make the best of a bad situation.

As for better documentation, I'm all about that, so if you feel you can improve our documentation in this area, we'd welcome it!

Then socket would return EPROTOTYPE if somebody tried to use those flags on OSX or RHEL5.

This wouldn't apply to OSX, where these flags don't exist. It looks like the O_CLOEXEC flag was added in OS X 10.7, which was released in 2011 and was unsupported in 2014.

So I think I agree with you @asomers, let's remove all emulation here. Since the constants aren't even available on macos/ios, then I think this should have minimal impact on downstream codebases.

[kristate]

I guess I feel a little torn here. More or less because the Berkeley socket API sucks in so many ways and by safely normalizing as much as possible, we could build a better ecosystem in Rust.

Better yet, rust's standard library sets CLOEXEC for all fds on unix by default: rust-lang/rust#24034

More discussion here: rust-lang/rust#24237

[Susurrus]

@kristate after #909 is merged, would you like to rebase this and reduce it to just the tests you added? Those would still be very helpful!

[pronebird]

Was this ever addressed in some other PRs?

[SteveLauC]

Was this ever addressed in some other PRs?

I guess no, and I am actually surprised that the socket() interface exposed by Nix does not conform to any implementation, so this is indeed an issue. Git blame shows me that that wrong interface was added 9 years ago, before this commit.

And I think @asomers's thought is right, we should expose the system interface without emulating anything, I will take a look at this and possibly get it fixed.