RFC: lower network stack rework

[jia200x]

Description

I'm opening this issue to synchronize and keep track of the netif rework.

Motivation and related issues

Mixed transceiver, PHY and MAC logic in netdev implementations and hardware

From the documentation, netdev acts as an interface to "provide a uniform API for network stacks to interacts with network device drivers".
In practice, the network stack interacts with an interface (e.g gnrc_netif) and then there's mixed transceiver, PHY and MAC layer logic.

Some consequences of this:

• Some device drivers already include PHY layer indications. E.g see

  RIOT/cpu/esp32/esp-wifi/esp_wifi_netdev.c

  Line 169 in 03c467a

  _esp_wifi_dev.netdev.event_callback(&_esp_wifi_dev.netdev, NETDEV_EVENT_ISR);

  
  In this case, the device driver already calls a function with the packet. In order to pass it to the network stack, it's necessary to generate a fake ISR event and an extra copy.
• Without a clear distinction between transceiver logic (e.g interrupt from the radio and functions to access the framebuffer) and PHY layer (send and receive data), we need to tell radios how to "read_pkt_size" or "drop" packets. Or sometime "preload" before sending. In practice all radios "preload" and then "trigger send", but we need to add the "send" semantics in the radios. Besides being more complex, it's sometimes inefficient (see [RFC] netdev: change receive mechanism #11652)
• Network device drivers have PHY state changes that should be handled by the MAC layers. E.g at86rf2xx radios go back to the idle_state after sending. This behavior is not valid e.g in TSCH
• We cannot process ACK packets if the radio doesn't support retransmissions (see General 802.15.4/CC2538 RF driver dislikes fast ACKs #7304 (comment))
• And we have to implement ACK handling or software CSMA on each radio, although the logic is the same (see nrf802154: Add ACK handling capabilities #11150 or Software CSMA and Link Retries for AT86RF2XX, and Fix Concurrency Bugs #8332)
• Upper layers expect radios running on Extended Mode (auto ACK, retransmissions, etc). Radios running in Basic Mode won't work properly unless they implement Extended Mode features (see e.g at86rf2xx: Basic mode and NETOPT_AUTOACK  #8213)

Related work:
#7736

There is one thread per (gnrc_)netif

It's needed to allocate one gnrc_netif_t thread per network interface. Besides allocating more resources than needed, this is sometimes problematic for platforms that have several transceivers.

See #10496

Netif relays on IPC messages for handling ISR events, send and receive.

• Race conditions might occur if a single queue process ISR and send events (see tests: add tests for netdev flooding race-condition #11256 )
• Interrupts can be lost using this mechanism (see at86rf2xx: lost interrupts #5486)

Related work:
#9326

Network stacks don't share initialization logic, ISR processing and network interfaces

Each stack needs to handle radio events, auto_init logic and MAC layers.

Some consequences of this:

• We cannot use GNRC on top of LoRaWAN (at least without gnrc_lorawan: add initial support for GNRC based LoRaWAN stack (v2) #11022), or use gnrc_netif_ieee802154 code for LWIP or emb6. It's also not posible to use the OpenThread lower layers (IEEE802.15.4 with security, plus joiner and commissioner roles) with GNRC.
• Some stacks are network device dependent. E.g it's not possible to use other radios than at86rf2xx in OpenThread. It's similar in LWIP.

Outcome

Separate transceiver, PHY and MAC logic

If we can provide interfaces between this components we could benefit of:

• A lot of code re utilization.
• Simpler network device drivers (e.g that only write transceiver logic instead of transceiver+PHY+MAC logic).
• A better testing infrastructure for network devices and their layers.
• A better interface with external code that already include a PHY or MAC layer

For IEEE802.15.4, the PHY layer can be implemented as a SubMAC layer (see #13376 ) in order to take advantage of the interface of most device drivers (MAC hw accelerations already included in the device).

Move auto-initialization and transceiver ISR logic out of the network interface

Moving the initialization and transceiver ISR logic out of the interface allows as to reuse code and immediately extend the support for more radios in LWIP, OpenThread, etc.

With this we could e.g get rid of these lines since the device initialization doesn't depend of the stack.

Remove the need of allocating one thread per interface

Interfaces can be represented with pointers. All events can be handled by only one thread or OS mechanism (e.g see #12474).

Improve support of software MAC layers

We shouldn't worry if a radio doesn't support Auto-ACK, retransmissions, etc. Also, we could have more powerful MAC layers (IEEE802.15.4 with security, pan coordinators, etc).

Write network stack independent network interfaces (see #12688 (comment))

This means the network interface is handled by the OS and not by the network stack. Network stacks can then reuse link layer logic (MAC, upper PHY). With this we can have a more uniform experience between different network stacks.

---

Proposed roadmap

This whole rework can be done in the following phases

1. Improve Link Layer support

• IEEE802.15.4 SubMAC
  • Add support for radio caps (netdev_ieee802154: add radio capabilities #11473)
  • Add support for common link layer features (Auto ACK, retransmissions, etc)
• IEEE802.15.4 MAC
  • L2 security

2. Add required interfaces PHY and HAL interfaces

• Extensibility of netdev (Extensebility of netdev_driver_t API #12469)
  TBD

3. Rework and extend the netif API (TO BE REVISED, see #12688 (comment))

This step is required in order to provide a network stack independent netif.

• Make netif_t a pointer instead of a stack defined type (netif: introduce generic network interface descriptor #11879)
• Define an interface for stack independent packet allocation, data handling and passing data up to the stack
  • Netbuf: net/netbuf: introduce network buffer abstraction (v2.0) #12691
• Extend the netif API to add send/recv operations (analog to sock, but intended to be used from network stacks or applications that send data via an interface)
• Migrate common code from gnrc_netif to netif
  • Refactor MAC layers to make them independent of GNRC (e.g gnrc_pktbuf dependencies in gnrc_netif_xxx functions)
  • Move gnrc_netif events to external event handlers + callbacks.
  • Move gnrc_netif_ops_t ops into netif_ops_t
    I will open a Github project to be able synchronize better.

Useful links

#11483
#11879
#11473

This is intended to be addressed to: #4876

[jia200x]

@RIOT-OS/maintainers there have been some offline conversation with some maintainers (@haukepetersen, @bergzand, @kaspar030, @miri64, @leandrolanzieri, @PeterKietzmann).

Unfortunately we didn't have time to discuss this during the RIOT summit, so I open this issue for syncing.

All kind of feedback on the fundamentals or roadmap is more than welcome!

[maribu]

Also related to this: #12469

[jia200x]

@maribu added to the road map

[miri64]

Define an interface for stack independent packet allocation, data handling and passing data up to the stack
[…]

• Remove GNRC dependencies in MAC layers (e.g gnrc_pktbuf dependencies in gnrc_netif_xxx functions)

I thought we figured last week, that this makes things more complicated than it need be…

[jia200x]

I thought we figured last week, that this makes things more complicated than it need be…

What I meant with that was: make the MAC layer component independent of GNRC

[jia200x]

@miri64 rephrased :)

[roberthartung]

#12128 would probably benefit from this as well.

[jia200x]

I renamed it to "lower network stack" rework, since the network interface are only one of the components involved

[jia200x]

I would like to share some thoughts about the "Write network stack independent network interfaces" part:

After some time, I think it's different to distinguish these 2 elements:

1. The OS representation of a network interface
2. The glue code between a network stack and the OS.

For 1., it's clear that the representation of a network interface should be agnostic to the network stack (e.g running ifconfig should work with all network stacks, same as sock). We already have most of this functions (' netif_iter, netif_get_opt`, etc).

However, 2. doesn't necessarily need a unique entry point (e.g netdev_recv and netif_send function) or a global allocator (netbuf). Here are some of the reasons:

1. Some network stacks require access to the radio (OpenWSN), some other to the PHY/SubMAC layer (OpenThread, LWIP) or even on top of the Link Layer. So, not all network stacks would make use of the netif_recv and netif_send function. Only access to a dedicated layer is needed.
2. Some MAC layers and drivers already have Framebuffers. Those who don't, it's always possible to design MAC layer allocators (e.g for Ethernet) or simply write one function per stack to allocate a packet (in the RECV function of the HAL).
3. Device ISR handling can be done by an (optional) event_thread module (sys: add shared event threads #12474 ), and this logic can be used by all Network Stacks. This would simplify much more the integration of network stacks

And last but not least, solving the netif problem first doesn't solve the problems of radios with different caps and lack of MAC layers, so I would probably give more priority to the devices/link layer rework.

Any comments?

[jia200x]

To make this easier to follow, I will split this rework in 2 independent reworks:

1. Improve GNRC Netif
2. Network stack independent link layers (this also includes HALs and mechanisms to process IRQs)

[jia200x]

this is reorganized in #13771 and #13763 , so I will close the issue.