Add support for --unprivileged mode (MacOS only)

[fujiapple852]

See for background.

The full set of ICMP sockets to support:

Family	Direction	Tracer	Socket	Field
IPv4	Both	ICMP	socket(AF_INET, SOCK_DGRAM, IPPROTO_ICMP)	ip4_txrx_icmp_socket
IPv4	Both	UDP	socket(AF_INET, SOCK_DGRAM, IPPROTO_UDP)	ip4_txrx_udp_socket
IPv6	Both	ICMP	socket(AF_INET6, SOCK_DGRAM, IPPROTO_ICMPV6)	ip6_txrx_icmp_socket
IPv6	Both	UDP	socket(AF_INET6, SOCK_DGRAM, IPPROTO_UDP)	ip6_txrx_udp_socket

[fujiapple852]

For IPv4/UDP we use raw socket so we may set the identification field, which is needed for Dublin strategy which encodes the sequence in that field.

There is no setsockopt to set this field to avoid using a raw socket. That means that elevated permissions are needed for UDP, unlike ICMP where we can use the special "ping sockets" on Unix (Linux, macOS and others).

We may, therefore, need to support both raw and non-raw IPv4/UDP modes, the former being more powerful (supports Dublin strategy) and requiring elevated privileges and the latter being less powerful but not requiring elevated privileges.

[fujiapple852]

For IPv4/ICMP the issue is that the platform byte order probing (PlatformIpv4FieldByteOrder) appears to require a raw socket (does not work with the above non-raw UDP socket on macOS, not needed on Linux, untested for others). Like UDP, we may therefore have to provide raw and non-raw variants for ICMP given the tradeoff. Specifically, raw allow for byte order probing but requires elevated privileges whereas non-raw must specify or assume the byte order but does not require elevated privileges.

Edit: resolved in #721

[fujiapple852]

One further issue to resolve is that the TracerChannel currently always constructs both the ICMP send/recv sockets and the UDP send socket, regardless of the protocol the user has specified. Specifically:

let icmp_send_socket = make_icmp_send_socket(config.source_addr)?;
let udp_send_socket = make_udp_send_socket(config.source_addr)?;
let recv_socket = make_recv_socket(config.source_addr)?;

This limits us to requiring either both or neither to require elevated privileges. The solution to this problem is to only create the UDP send socket if running in UDP protocol mode, and only create the ICMP send socket in ICMP protocol mode. Note that the ICMP recv socket is needed and used by both.

Edit: resolved in #647

[fujiapple852]

We also need a way to be able to know, at startup, if the user is going to need to run with elevated privileges, such that we can error with a sensible message (currently handled in caps.rs).

If we implement fallback logic (try ping socket first, fallback to raw) then we have to move the error handling to later in the process and account for the fact it will occur on a separate thread.

This issue becomes simpler to solve if we force the user to be explicit are raw vs non-raw for both UDP and ICMP protocols.

[fujiapple852]

Running without requiring privileges could require the —unprivileged (-u) flag.

[c-git]

This looks interesting. It would be nice to not need the flag, but if it makes the code materially more complicated without it. I think that's fine to require it.

[fujiapple852]

Three reason why we want the user to be explicit about this:

1. The behaviour of udp tracing differs between privileged and unprivileged modes. Specifically neither the Paris nor Dublin tracing strategies can be used in unprivileged mode as these involve manipulating the udp/ip headers which cannot be done without a raw socket (which requires privileges).
2. The IPPROTO_ICMP socket is only available on a subset of platforms (i.e. most unix-like ones) but is not always enabled and will fail at runtime if not enabled. Falling back to the IPPROTO_RAW socket will fail without privileges.
3. This is a change to the existing behaviour of Trippy and so it should be opt-in. The default will remain privileged mode and Trippy will error out on startup if the user attempts to use it without privileges as it does today.

[crrodriguez]

First, of all thanks for your software, been using it as replacment of mtr for a while with great, clear results.

about 1. That sounds like the kernel is missing an option to do this safely without the need of a privileged socket.. maybe talk to maintainers about it ?
what about requiring privileges only when OS does not support "ping sockets" AND the behaviour will not be the same due to the lack of number 1 item ?

At least on linux the highly desirable behaviour is to run without privileges always whenever possible. (according to the kernel changelog it is even possible to do so with plain sockets in some cases https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=45af29ca761c275e350cca659856bc56f1035ef9)

[c-git]

@crrodriguez if @fujiapple852 is amenable to talking to the maintainers, is that something you would have to contacts to do?

Which ever option we finally go with, we should include in the error message for missing privileges that there is an unprivileged option available.

[fujiapple852]

First, of all thanks for your software, been using it as replacment of mtr for a while with great, clear results.

Thanks @crrodriguez, I'm glad you are finding it useful.

That sounds like the kernel is missing an option to do this safely without the need of a privileged socket.. maybe talk to maintainers about it ?

It's a nice idea. To avoid the need to use a raw socket here we would need a way (i.e. a setsockopt call) to set the IPv4 header identifier field (for Dublin tracing) and the UDP header checksum field (for Paris tracing) on all platforms. Could these be exposed? In theory yes but some issues that come to mind immediately are hardware offloading and packet fragmentation and I imagine a bunch more. I think it would be a long hard road to get access to these across platforms! Saying that i'd be keen to explore this further with anyone who may be in a position to make it a reality.

what about requiring privileges only when OS does not support "ping sockets" AND the behaviour will not be the same due to the lack of number 1 item ?

The trouble is we don't know if the platform supports (or is configured to support) raw sockets until we try and open the socket.

The current logic (0.8.0) used by Trippy is:

• Check if we are privileged, fail if we are not
• Open raw socket, fail if we cannot

Note that the first check is redundant and exists only to allow a clear error message that links to https://github.com/fujiapple852/trippy#privileges vs showing some obscure socket open error.

The logic I am proposing (and have implemented in #638) is:

• Check if we are privileged only if the user has selected privileged mode (default), fail if we are not
• Open either the IPROTO_ICMP (if unprivileged mode) or RAW socket (privileged mode), fail if we can not

The alternative proposed logic that I was considering would be (ignoring Paris and Dublin edge cases):

• Open the IPROTO_ICMP socket
  • If that fails, check if we are privileged
    • If we are privileged, open RAW socket, fail if we cannot
    • If we are unprivileged, fail

Whilst quite awkward to implement in Trippy today I think it would be possible. I not a fan of this approach however, there is a little bit to much "magic" here (try A, if it fails try B) and I feel it could lead to a confusing user experience.

There is also the question of the unsupported Paris and Dublin UDP cases, which would have to be factored into the above logic somehow.

we should include in the error message for missing privileges that there is an unprivileged option available.

This is the approach I have taken in the WIP implementation, where the unsupported cases are detected during parameter validation and we fail:

trip example.com --udp -R dublin -u
Error: Dublin tracing strategy cannot be used in unprivileged mode

The tui header also extends the protocol information to include the privilege level: protocol=udp(v4, classic, unprivileged)

Note that the current implementation does not preclude making it smarter in the future. Internally Trippy now has a PrivilegeMode:

/// The privilege mode.
#[derive(Debug, Copy, Clone, Eq, PartialEq)]
pub enum PrivilegeMode {
    /// Privileged mode.
    Privileged,
    /// Unprivileged mode.
    Unprivileged,
}

This is defaulted to PrivilegeMode::Privileged unless the --unprivileged (-u) flag is passed.

This setup would allow for adding additional PrivilegeMode variants in the future, such as Fallback which behave along the lines of the above. We could then expose an additional cmd line flag such as --privilege-mode <mode> where the fallback or other modes could be supplied.

We may even chose to change the default PrivilegeMode per platform. For example, this feature paves the way for the possibility of having an Android build in the future where unprivileged mode is essential (for Linux and other unixes we have better ways such as CAP_NET_RAW capabilities and setuid bit).

If you were willing it would be great if you could take WIP code (#638) for a spin and see how it feels in practise (note: doesn't support IPv6 yet and hasn't been tested on all platforms, but should be mostly functional). Update: does not work at all on Linux yet.

[fujiapple852]

To my surprise, Linux does not allow setting IP_HDRINCL for IPPROTO_ICMP socket. This means the existing code to handle the icmp protocol will not work out-of-the-box for Linux as it did for MacOS.

Instead we must jump through some hoops to send EchoRequestPacket directly (with no added IP header) and use some additional socket options to extract the identifier/ttl from the responses.

[fujiapple852]

Below is an overview of support for unprivileged icmp across platforms. As a quick explanation of the columns:

IPPROTO_ICMP - A special datagram socket type which allows sending and receiving ICMP packets without requiring a RAW socket
IP_HDRINCL - A socket option which allows the userspace program to pass the IP header as well as the payload. This is normally only allowed for RAW sockets, however it is also supported for the IPPROTO_ICMP socket type on some platforms.
NET_CAP_RAW - A capability which may be assigned to an otherwise unprivileged program to allow the use of RAW sockets.

Platform	IPPROTO_ICMP	IPPROTO_ICMP + IP_HDRINCL	NET_CAP_RAW	Notes
MacOS	Yes	Yes	No	Fully supported
Linux	Yes	No	Yes	Whilst linux supports IPPROTO_ICMP it does not allow using it in conjunction with IP_HDRINCL. Instead Linux employs platform specific techniques as outlined in https://lwn.net/Articles/443051
Windows	No	No	No	Windows does not support IPPROTO_ICMP
NetBSD	No	No	No	Only RAW sockets supported according to https://man.netbsd.org/icmp.4
OpenBSD	No	No	No	Only RAW sockets supported according to https://man.openbsd.org/icmp
FreeBSD	No	No	No	Only RAW sockets supported according to https://man.freebsd.org/cgi/man.cgi?query=icmp&apropos=0&sektion=0&manpath=FreeBSD+15.0-CURRENT&arch=default&format=html

Therefore, at best, we can only support MacOS and Linux. However to support unprivileged icmp in Linux would require platform specific code which extends beyond the current socket abstraction layer and would require a signifiant refactor to support cleanly.

It is worth noting that Linux, unlike MacOS, is the only platform which support the NET_CAP_RAW capability which allows using RAW sockets from unprivileged processes, which goes a long way towards removing the need for this feature.

Therefore, for the initial implementation, this feature will be MacOS only.

[fujiapple852]

Created #741 follow up task to add support for Linux.

[c-git]

I think that makes sense and I'm personally not sure it's worth doing it for linux at all. I don't know how other people feel but I personally feel that just granting the raw sockets permission only made me comfortable using trippy. I don't know how much value add there is for not requiring it at all for linux. IMO where it would matter the most is on Windows where you only have full admin or not and it doesn't work there anyway. So my personal opinion is that it's not worth the work to do it for linux.