EVFILT_LIO notifications don't rewaken a future after #4157

[asomers]

Version
tokio v1.13.0 or later

Platform
FreeBSD 14.0-CURRENT amd64

Description

An lio_listio operation, when complete, will deliver an EVFILT_LIO notification via kqueue. Due to resource limitations, sometimes the kernel will only accept a portion of the submitted AIO operations. When that happens, it will deliver notification when the accepted operations are all complete, and then the tokio-file crate will resubmit the remaining operations. Notably, it does not create a separate Future for the resubmitted operations; it just reuses the original Future.

I tried this code:
https://github.com/asomers/tokio-file/blob/0fb990be87a4fe07a066b88bae738349772d1837/tests/lio_listio_incomplete.rs

I expected to see this happen:
Prior to Tokio-1.13.0, after resubmitting with lio_listio, the kernel sends another EVFILT_LIO event to Tokio, which polls the tokio_file::WritevAt Future again. The test program will usually have to resubmit multiple times, and each time the Future gets polled again after receiving the EVFILT_LIO event.

Instead, this happened:
With Tokio-1.13.0, second and subsequent EVFILT_LIO events get ignored. The Future never gets polled again, tokio-file never collects the AIO operations' status with aio_return, so EVFILT_LIO even never gets cleared, and Tokio spins around kevent.

git bisect identified #4157 as the introduction of the problem.
downstream bug: asomers/tokio-file#27

[Darksonn]

If #4157 made this fail, then it should never have worked in a spawned task because that PR is about the block_on future being polled too often. Does it work inside spawned tasks?

[Darksonn]

Are you calling poll_ready again after your call to clear_ready? If you don't do that and just return Poll::Pending, then there is no guarantee you are ever woken up again.

[asomers]

@Darksonn this is all happening inside my implementation of Future::poll. That method of course starts out with tokio::io::bsd::Aio::poll_ready. Are you saying that I need to call poll_ready a second time after resubmitting to the OS?

[Darksonn]

Yes, you must call poll_ready again after clearing readiness. This is because wakeups are only guaranteed if you have called a Tokio method that returned Poll::Pending. If you get Poll::Ready and then call clear_ready, then you are not in that situation because clear_ready does not return a Poll::Pending (it doesn't return anything).

The AsyncFd examples handle this by wrapping the various methods in loops.

[asomers]

That explanation makes sense. Thanks Alice!