Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

sys/net/gnrc/netif: make use of confirm_send #15821

Closed
wants to merge 3 commits into from

Conversation

maribu
Copy link
Member

@maribu maribu commented Jan 20, 2021

Contribution description

  • adapt gnrc_netif.c to make use of netdev_driver_t::confirm_send, if not `NULL
    • As the new API is explicitly non-blocking, this requires now IRQs to be processed while waiting for TX completion.
    • While TX is in progress, IPC messages (which might contain additional data to send) must not be received. To implement this core/thread_flags is used to wait for TX completion and IRQs
    • As a result, IRQs are no longer been lost. Hence, it provides the feature gnrc_netif_events provides, but with less ROM. Consequently, gnrc_netif_events is dropped
    • As nice side effect, now all events can be reported from IRQ context. This can be beneficial for peripheral radios
    • If confirm_send() is provided, the pkgsnip is now released by gnrc_netif.c, rather by each adaption layer individually. This is needed, as the driver might still operate on the buffer during TX - e.g. peripheral network interfaces with DMA that supports scatter/gather, no copy of the outgoing frame is needed.
  • adapt gnrc_netif_ethernet.c to only release the packet buffer if confirm_send is not provided
  • implement confirm_send for the STM32 Ethernet driver

Testing procedure

  • There should be no regressions on drivers not using confirm_send()
  • There should be no regressions on the STM32 Ethernet driver

Issues/PRs references

@maribu maribu added Type: enhancement The issue suggests enhanceable parts / The PR enhances parts of the codebase / documentation Area: network Area: Networking State: WIP State: The PR is still work-in-progress and its code is not in its final presentable form yet labels Jan 20, 2021
@maribu maribu requested review from miri64, jia200x and benpicco January 20, 2021 16:53
@maribu
Copy link
Member Author

maribu commented Jan 20, 2021

@benpicco: Maybe you can give this a look. I have a very strange regression on the at86rf215 that I cannot track down. (I only tested for STM32 Ethernet and at86rf215, so it might apply to all network devices not implementing confirm_send().) I used the sub GHz interface, but I guess this is be reproducible on the 2.4 GHz interface as well:

  1. flash examples/gnrc_networking
  2. ifconfig works
  3. ping6 some reachable node. This will work fine.
  4. ifconfig still works
  5. ping6 some non-reachable node. This reports lost pings, as expected
  6. Now ping6 to the reachable node no longer works, and ifconfig blocks waiting for the IPC reply of the interface. The details of the 2.4 GHz interface still show up, so only the sub GHz network interface thread deadlocked somehow.

Do you have any clue what the issue could be?

@benpicco
Copy link
Contributor

My suspicion is that this has to do with how ACK timeouts are handled.

Indeed I get

2021-01-20 18:14:58,078 # gnrc_netif: get index of fe80::e4ea:aff8:af5d:eae4 from interface 7
2021-01-20 18:14:58,080 # gnrc_netif: message 4660
2021-01-20 18:14:58,086 # gnrc_netif: unknown message type 0x1234(no message handler defined)

0x1234 is NETDEV_MSG_TYPE_EVENT which is used to trigger a call to _isr().

The 'proper' solution would be porting this to the new radio HAL that takes care of ACK timeouts in a generic way. But that will also require some 802.15.4g plumbing inside the radio HAL.

@maribu maribu added the State: waiting for other PR State: The PR requires another PR to be merged first label Jan 20, 2021
@maribu
Copy link
Member Author

maribu commented Jan 20, 2021

My suspicion is that this has to do with how ACK timeouts are handled.

Thanks for the pointer. This helped to uncover quite a monster of a bug :-) It works now as expected :-)

@jia200x
Copy link
Member

jia200x commented Jan 21, 2021

The 'proper' solution would be porting this to the new radio HAL that takes care of ACK timeouts in a generic way. But that will also require some 802.15.4g plumbing inside the radio HAL.

We should get there soon :). There are already some traces (pages, 16bit channels, etc).

@jia200x
Copy link
Member

jia200x commented Jan 21, 2021

I like the idea. Specially the fact that we use thread flags now for processing events here. I also think in the future this could be written in a network-stack independent manner, so we can reuse the same mechanism for OpenThread, LWIP or even custom applications.

This:

As the new API is explicitly non-blocking, this requires now IRQs to be processed while waiting for TX completion

And this:

While TX is in progress, IPC messages (which might contain additional data to send) must not be received. To implement this core/thread_flags is used to wait for TX completion and IRQs

I'm afraid these might introduce a regression in some cases. For IEEE 802.15.4 for instance, this could easily take 50~70 ms in some configurations (long frames=~4ms ToA plus IFS, CSMA-CA backoff is 20 symbols = 320 us and the backoff exponent can be 7 => up to 2^7*320us = up to 40ms, several retransmissions).

This would affect the upper layers trying to fetch information for the interfaces and any existing MAC layer that requires some housekeeping (GoMACH, lwmac, gnrc_lorawan and the upcoming IEEE 802.15.4 MAC).

I would suggest to simply use one loop (as it is now). We can safely rely on this:

If confirm_send() is provided, the pkgsnip is now released by gnrc_netif.c, rather by each adaption layer individually. This is needed, as the driver might still operate on the buffer during TX - e.g. peripheral network interfaces with DMA that supports scatter/gather, no copy of the outgoing frame is needed.

And then we simply go back to the main loop until we get a TX done.

If the concerns go in the direction of TX timestamps, I'm gathering through the SDR quite a lot of evidence that this should be handled by the driver (some even provide helpers).

Also, regarding:

As nice side effect, now all events can be reported from IRQ context. This can be beneficial for peripheral radios

I really think we should aim to do that in all radios. Even SPI radios have mechanisms to do that quite fast (e.g the at86rf2xx requires only one SPI instruction to fetch the event). Of course fetching the frame from the radio would be done from thread context. But besides solving synchronization issues (and giving some determinism to the radios), this would help us in the long run to reduce some overhead (e.g we can simply rely on semaphores, mutex, etc).

@jia200x
Copy link
Member

jia200x commented Jan 21, 2021

I really think we should aim to do that in all radios. Even SPI radios have mechanisms to do that quite fast (e.g the at86rf2xx requires only one SPI instruction to fetch the event). Of course fetching the frame from the radio would be done from thread context. But besides solving synchronization issues (and giving some determinism to the radios), this would help us in the long run to reduce some overhead (e.g we can simply rely on semaphores, mutex, etc).

We would need an asynchronous SPI API though, But I'm sure it could be a mid-term goal

@maribu
Copy link
Member Author

maribu commented Jan 21, 2021

I'm afraid these might introduce a regression in some cases. For IEEE 802.15.4 for instance, this could easily take 50~70 ms in some configurations (long frames=~4ms ToA plus IFS, CSMA-CA backoff is 20 symbols = 320 us and the backoff exponent can be 7 => up to 2^7*320us = up to 40ms, several retransmissions).

This would affect the upper layers trying to fetch information for the interfaces and any existing MAC layer that requires some housekeeping (GoMACH, lwmac, gnrc_lorawan and the upcoming IEEE 802.15.4 MAC).

I see. I think we have the following options here:

  1. If any only if gnrc_netif_pktq is used, fetch IPC messages also during TX. During TX, send requests will be queued up, rather than being processed. Some network devices will then hard depend on gnrc_netit_pktq. Without pktq, IPC messages will not be received during TX.
  2. Adapt the upper layers to not generate send requests while TX is still ongoing. I have started some preliminary work in this direction with sys/net/gnrc/tx_sync: new module #15694 - but this PR only blocks at the very top (the SOCK API), so any user directly interacting with GNRC would not sync at all. So that (still WIP PR) is only a step in that direction, not the full solution.
  3. Fetch msgs during TX, but re-enqueue send requests (by sending these msgs to one self). Once a single send request enters the message queue, this will become a busy waiting loop :-/
  4. Fetch msgs during TX, but drop send requests. This would break L2 fragmentation, as all but the first fragment would be dropped. I'd say we can rule this one out.

@jia200x
Copy link
Member

jia200x commented Jan 21, 2021

Hmmm IMO:

  1. If any only if gnrc_netif_pktq is used, fetch IPC messages also during TX. During TX, send requests will be queued up, rather than being processed. Some network devices will then hard depend on gnrc_netit_pktq. Without pktq, IPC messages will not be received during TX.

Is the more reasonable thing to do. But what speak against having simply one loop? (instead of a separate loop for TX)
If the upper layer needs a synchronous send, this could be done from an asynchronous netif event (e.g simply unlock the upper event on TX_DONE).

In fact, several radios using the old send API are still blocking and non-blocking at the same time (e.g the at86rf2xx only blocks if the radio is busy trying to send another frame).
E.g this line https://github.com/RIOT-OS/RIOT/pull/15821/files#diff-2ad823469e86d3772efecaa0f4bb5afcc1d6d039af9cc28db3535fcd292e28a6R1585 will be executed before the TX DONE if the radio is not busy.

Having one loop IMO would help us to synchronize all cases

@jia200x
Copy link
Member

jia200x commented Jan 21, 2021

As a result, IRQs are no longer been lost. Hence, it provides the feature gnrc_netif_events provides, but with less ROM. Consequently, gnrc_netif_events is dropped

BTW, there are some MAC layers that get some benefits when using such a mechanism. E.g I plan to port GNRC LoRaWAN to use an event queue for all MAC timeouts (what I did with #15038).
If you are going to remove that feature, I would at least consider a way to extend the thread flags here (hooks, etc)

@maribu
Copy link
Member Author

maribu commented Jan 21, 2021

As a result, IRQs are no longer been lost. Hence, it provides the feature gnrc_netif_events provides, but with less ROM. Consequently, gnrc_netif_events is dropped

BTW, there are some MAC layers that get some benefits when using such a mechanism.

Sure. But the IRQ should remain at higher priority, as e.g. some radios have an RX/TX FIFO that is smaller than the maximum length of the PDU. Hence, an ISR needs to feed into the TX FIFO while sending, or drain the RX FIFO while receiving - this has very strict time requirements.

Hence, the current implementation cannot be used to also hook in MAC events. Since recently, we do have priority support in the event queue mechanism by providing one event queue per priority. But IMO we should for now just go with core/thread_flags until a second user becomes ready.

Generally speaking, unless all use of core/msg would be replaced by sys/event in the network interface, we need to core/thread_flags anyway - this is the only mechanism that can wait for both messages and events. So there is (for now) no RAM/ROM penalty in just also using this core/thread_flags directly for signaling IRQs or TX_DONE and the like events. So it might turn out that even when other users of sys/event in the network interface pop up, keeping core/thread_flags remains the lowest overhead option for ISRs and low level "events" like signaling TX_DONE.

@maribu
Copy link
Member Author

maribu commented Jan 21, 2021

In fact, several radios using the old send API are still blocking and non-blocking at the same time (e.g the at86rf2xx only blocks if the radio is busy trying to send another frame).

Right after this get's in, I will convert that one to the new API. This semi-blocking behavior is blowing my mind every time.

@jia200x
Copy link
Member

jia200x commented Jan 21, 2021

Hence, the current implementation cannot be used to also hook in MAC events. Since recently, we do have priority support in the event queue mechanism by providing one event queue per priority. But IMO we should for now just go with core/thread_flags until a second user becomes ready.

I agree. The IRQ offloading should have a higher priority than the interface

@maribu
Copy link
Member Author

maribu commented Jan 22, 2021

Because this PR also includes #15783, I refer to "pre-PR" as to master + the first two commits in this PR, which are borrowed from #15783

pre PR: make RIOT_CI_BUILD=1 BOARD=nucleo-f767zi -C examples/gnrc_networking
   text	   data	    bss	    dec	    hex	filename
  76644	    180	  18224	  95048	  17348	.../gnrc_networking.elf
pre PR: USEMODULE=gnrc_netif_events make RIOT_CI_BUILD=1 BOARD=nucleo-f767zi -C examples/gnrc_networking
   text	   data	    bss	    dec	    hex	filename
  77120	    180	  18240	  95540	  17534	.../gnrc_networking.elf
PR: make RIOT_CI_BUILD=1 BOARD=nucleo-f767zi -C examples/gnrc_networking
   text	   data	    bss	    dec	    hex	filename
  77056	    176	  18224	  95456	  174e0	.../gnrc_networking.elf

Conclusion

Pros:
  • fixes lost IRQ issues, but with less ROM than gnrc_netif_events
  • drivers can now be non-blocking, which is especially for drivers that need to handle IRQs during TX a great simplification
  • drivers now can pass up events from IRQ context, which is allows leaner implementation for peripheral network devices with memory mapped interfaces, or SPI attached interfaces that have multiple IRQ pins (so that knowing which IRQ line triggered the IRQ is enough to classify the event).
  • passing auxiliary info up the stack (e.g. data pending, TX timestamps, number of retransmissions, time before channel clear assessment succeeded, ...) is now possible via send_confirm()
Cons:
  • Requires more ROM

@maribu maribu removed the State: WIP State: The PR is still work-in-progress and its code is not in its final presentable form yet label Jan 22, 2021
@maribu maribu changed the title WIP: sys/net/gnrc/netif: make use of confirm_send sys/net/gnrc/netif: make use of confirm_send Jan 22, 2021
@jia200x
Copy link
Member

jia200x commented Jan 22, 2021

Requires more ROM

... which we automatically get back when directly using the submac (instead of netdev_ieee802154_submac) :)

This goes in a good direction IMO

@maribu maribu removed the State: waiting for other PR State: The PR requires another PR to be merged first label Jan 22, 2021
@maribu
Copy link
Member Author

maribu commented Mar 31, 2021

Rebased to include the change that TX/RX start/end events are now generated unconditionally and again since I accidentally pushed a commit that I didn't meant to push.

@maribu
Copy link
Member Author

maribu commented Mar 31, 2021

I split out the adaption of the Ethernet driver on STM32 into its own PR. Especially since this would break lwIP until the glue code between netdev and lwIP also is changed to handle confirm_send, this makes IMO sense. I will now adapt at86rf215 driver instead and @fabian18 already adapted the at86rf2xx driver.

Btw: I cannot find the "semi-blocking" code in at86rf2xx anymore. A year ago the driver was busy waiting on TX when another transmission was still being pushed out the air. Was this removed?

@fabian18
Copy link
Contributor

I noticed that the nrf24l01p_ng blocks in an endless loop here,
if the receive callback just sets a thread flag and does not call the receive routine.

Copy link
Contributor

@fabian18 fabian18 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I believe that a release is missing, if pktq is used and the driver has no confirm_send().

}
#endif /* IS_USED(MODULE_GNRC_NETIF_PKTQ) */
/* remove previously held packet */
gnrc_pktbuf_release_error(pkt, -error_code);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should the release be inside the

#if IS_USED(MODULE_GNRC_NETIF_PKTQ)
...
#endif /* IS_USED(MODULE_GNRC_NETIF_PKTQ) */

because the call to gnrc_pktbuf_hold() is also inside of an

#if IS_USED(MODULE_GNRC_NETIF_PKTQ)
...
#endif /* IS_USED(MODULE_GNRC_NETIF_PKTQ) */

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, once the the frame was passed over to gnrc_netif, it has to release it eventually. With the legacy API, this is done on success by the netdev adaption layer on success. But on failure it has to be freed by gnrc_netif. In case of the new API, gnrc_netif will always be in charge of releasing it, even on success. (But only after TX was signaled as completed.)

if (!netif->dev->driver->confirm_send) {
/* hold in case device was busy to not having to rewrite *all* the link
* layer implementations in case `gnrc_netif_pktq` is included */
gnrc_pktbuf_hold(pkt, 1);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we increment the reference counter of pkt here, do we not have to release it also in _tx_succeeded() and not only in _tx_failed()?
I am experiencing error: packet buffer full with nrf24l01p_ng when I try ping6 fe80::a0cc:a5ff:fe6b:aa%7 -c 255.
But when I remove gnrc_pktbuf_hold(pkt, 1), the error is gone.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should be fixed. You tested with the upstream nrf, so no confirm_send, right?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes I tested with the upstream nrf first. The release does fix it.

#if IS_USED(MODULE_GNRC_TX_SYNC)
gnrc_pktbuf_release(tx_sync);
#endif
return _tx_succeeded(netif, res);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should there be a release to drop the previously held packet?

Suggested change
return _tx_succeeded(netif, res);
#if IS_USED(MODULE_GNRC_NETIF_PKTQ)
if (!netif->dev->driver->confirm_send) {
/* remove previously held packet */
gnrc_pktbuf_release(pkt);
}
#endif
return _tx_succeeded(netif, res);

Or the packet is passed to, and released in _tx_succedded(), because in _process_events_await_msg() the packet is released right after tx_succeeded() was called.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, but only when pktq is used. With the legacy API the netdev (or rather the netdev adaption layer) is in charge of releasing the packet.

dev->driver->isr(dev);
}
if (flags & THREAD_FLAG_RX_DONE) {
gnrc_pktsnip_t *pkt = netif->ops->recv(netif);
Copy link
Contributor

@fabian18 fabian18 Mar 31, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you think it is reasonable to have something like this:

if (flags & THREAD_FLAG_RX_DONE) {
    gnrc_pktsnip_t *pkt;
    while(flags & THREAD_FLAG_RX_DONE) {
        if ((pkt = netif->ops->recv(netif))) {
           _process_receive_stats(netif, pkt);
            _pass_on_packet(pkt);
        }
        if (!tx_pkt) {
            tx_pkt = _send_queued_pkt(netif);
        }
    }
}

The nrf24l01p_ng has an internal queue.
Inside the recv() routine in the driver I would again call the netdev callback with RX_DONE to reset the thread flag.
Else, I guess a send call could override frames in the internal receive queue.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This will not work so easy. The issue is that thread_flags_wait_any(THREAD_FLAG_RX_DONE) needs to be called in the loop. But that will block until the flag is set.

But I see no reason to extend the API to provide a non-blocking function that would to just allow that. But for now it might be the best if the driver would just return -EBUSY on TX if the receive queue is not yet drained.

Copy link
Contributor

@benpicco benpicco left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Well neighbor stats are currently not used yet, so we are not losing much.

Since this PR doesn't contain any driver implementations, there is no expected change with existing drivers besides that, right?
I'll give that a try. Is this the final form of this PR yet?

(My only grudge is that this will create two classes of drivers and converting a driver to the new way seems all but trivial, but the progress is real here)

Comment on lines +1488 to +1490
if (error_code == -EBUSY) {
result = NETSTATS_NB_BUSY;
} else {
result = NETSTATS_NB_NOACK;
netdev_t *dev = netif->dev;
dev->driver->get(dev, NETOPT_TX_RETRIES_NEEDED, &retries, sizeof(retries));
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is no driver that uses -ECOMM to indicate no-ACK

sys/net/gnrc/netif/gnrc_netif.c Outdated Show resolved Hide resolved
Comment on lines +1493 to +1520
#if IS_USED(MODULE_GNRC_NETIF_PKTQ)
if (error_code == -EBUSY) {
int put_res;

/* Lower layer was busy.
* Since "busy" could also mean that the lower layer is currently
* receiving, trying to wait for the device not being busy any more
* could run into the risk of overriding the received packet on send
* Rather, queue the packet within the netif now and try to send them
* again after the device completed its busy state. */
if (push_back) {
put_res = gnrc_netif_pktq_push_back(netif, pkt);
}
else {
put_res = gnrc_netif_pktq_put(netif, pkt);
gnrc_netif_pktq_sched_get(netif);
}
if (put_res == 0) {
DEBUG("gnrc_netif: (re-)queued pkt %p\n", (void *)pkt);
return; /* early return to not release */
}

LOG_ERROR("gnrc_netif: can't queue packet for sending\n");
/* If we got here, it means the device was busy and the pkt queue
* was full. The packet should be dropped here anyway */
error_code = -ENOMEM;
}
#endif /* IS_USED(MODULE_GNRC_NETIF_PKTQ) */
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we move both of those conditional blocks to static inline functions to enhance readability?

@fabian18
Copy link
Contributor

Since this PR doesn't contain any driver implementations, there is no expected change with existing drivers besides that, right?
I'll give that a try. Is this the final form of this PR yet?

I guess every driver that calls netdev->event_callback(netdev, NETDEV_EVENT_RX_COMPLETE); in a loop is going to face unintended behaviour (at worst endless loop), because RX is not handles synchronous anymore.
I found: esp_wifi, enc28j60, encx24j600, nrf24l01p_ng, and w5100 which do that.

@fabian18
Copy link
Contributor

Can there for example be a NETDEV_EVENT_RX_CONTINUE which does provide the previous functionality?

Co-authored-by: benpicco <benpicco@googlemail.com>
@maribu
Copy link
Member Author

maribu commented Apr 24, 2021

I guess every driver that calls netdev->event_callback(netdev, NETDEV_EVENT_RX_COMPLETE); in a loop is going to face unintended behaviour (at worst endless loop), because RX is not handles synchronous anymore.

This is indeed an issue that needs to be addressed first. Technically, it is however better to not handle the event right away. The call graph in the current code is like this:

gnrc_netif --> isr()
    |           |
    |           +---> event_callback()
    |           |           |
    |           |           +---> recv()
    |           |           |      |
    |           |           |<-----+
    |           |           |
    |           |<----------+
    |           |
    |<----------+
    |        

The additional call depths results in more stack memory being used. In addition, it makes it more difficult to implement the network interface as an event queue handler, that just posts events in the event queue. But such an implementation could share a single thread for all network interfaces. Not only for board routers but e.g. also for the AT86RF215 there are inherently more than one netdevs - and cutting the stacks required to handle them in half would be really beneficial.

Bottom line: I really would like to keep the new behavior for netdevs that implement confirm_send(). I'll try if I can hack in the old behavior for the legacy interface, so that things don't break. Once all drivers are converted, we can drop the compatibility code again.

@benpicco benpicco requested a review from dylad May 8, 2021 20:57
@MrKevinWeiss MrKevinWeiss added this to the Release 2021.07 milestone Jun 21, 2021
@MrKevinWeiss MrKevinWeiss removed this from the Release 2021.07 milestone Jul 15, 2021
@stale
Copy link

stale bot commented Mar 2, 2022

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. If you want me to ignore this issue, please mark it with the "State: don't stale" label. Thank you for your contributions.

@stale stale bot added the State: stale State: The issue / PR has no activity for >185 days label Mar 2, 2022
@benpicco
Copy link
Contributor

I guess every driver that calls netdev->event_callback(netdev, NETDEV_EVENT_RX_COMPLETE); in a loop is going to face unintended behaviour (at worst endless loop), because RX is not handles synchronous anymore.
I found: esp_wifi, enc28j60, encx24j600, nrf24l01p_ng, and w5100 which do that.

So to push this forward, those drivers would need to be fixed right?
Am I under the correct assumption that just dropping the loop would be enough with this PR?

@fabian18
Copy link
Contributor

I remember for nrf24l01p_ng, simply changing the while to an if was not sufficient to make it work again.
Right now I have none of them at home to test again. I don´t know about the others. There may also be side effects.

@stale stale bot removed the State: stale State: The issue / PR has no activity for >185 days label Apr 11, 2022
@maribu
Copy link
Member Author

maribu commented May 25, 2022

Rebasing is futile, master has been changed to much. I think I'd rather start again from scratch.

@maribu maribu closed this May 25, 2022
@maribu maribu deleted the confirm_send branch July 22, 2022 08:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Area: network Area: Networking CI: ready for build If set, CI server will compile all applications for all available boards for the labeled PR Type: enhancement The issue suggests enhanceable parts / The PR enhances parts of the codebase / documentation
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants