Tracking Issue for #![feature(async_iterator)]

[yoshuawuyts]

This is a tracking issue for the RFC "2996" (rust-lang/rfcs#2996).
The feature gate for the issue is #![feature(async_iterator)].

About tracking issues

Tracking issues are used to record the overall progress of implementation.
They are also used as hubs connecting to other relevant issues, e.g., bugs or open design questions.
A tracking issue is however not meant for large scale discussion, questions, or bug reports about a feature.
Instead, open a dedicated issue for the specific matter and add the relevant feature gate label.

Steps

• Implement the RFC (cc @rust-lang/XXX -- can anyone write up mentoring
  instructions?)
• Adjust documentation (see instructions on rustc-dev-guide)
• Stabilization PR (see instructions on rustc-dev-guide)

Unresolved Questions

• Clarify the panic behavior of Stream and Iterator Add core::stream::Stream #79023 (comment)
  • Add a panic section to Iterator, clarifying panic behavior. The panic behavior between Stream and Iterator should be consistent.
• Restore Stream::next. This was removed from the RFC because it prevents dynamic dispatch, and subsequently removed from the implementation. This should be resolved before stabilizing.
  • Investigate whether we can move the trait from fn poll_next to async fn next once we can use async in traits.
• Investigate as part of keyword-generics whether we can merge Iterator and AsyncIterator into a single trait which is generic over "asyncness".
• Should we name this API AsyncIterator instead? Tracking Issue for #![feature(async_iterator)] #79024 (comment)

Implementation history

• PR introducing core::stream::{Stream, Next}
• Rename from Stream to AsyncIterator

[kaimast]

Is there a plan to include a core::stream::StreamExt (similar to the one in the futures crate) as well?

[yoshuawuyts]

Is there a plan to include a core::stream::StreamExt (similar to the one in the futures crate) as well?

The plan is to add methods directly onto Stream much like methods exist on Iterator, but we want to do so in a way that won't cause ambiguities with ecosystem-defined methods in order to not accidentally break existing codebases when upgrading to newer Rust versions.

[joshtriplett]

Some discussion on Zulip raised the question of naming. With full acknowledgement to the fact that the name Stream has a long history in the async ecosystem, multiple people observed that something like AsyncIterator or similar might be much more evocative for new users. Such a name would allow people to map their existing understanding of iterators. "I understand Iterator, and I understand async, and this is an async version of Iterator".

[brainstorm]

As a new, inexperienced user, I find AsyncIterator way more foreign than Stream, to be honest... perhaps I'm not too involved in compiler discussions and I don't see how the jargon clicks together though :-S ... also, blogposts are already being written about Stream, so changing the name mid-flight will only breed confusion, I reckon?

[yoshuawuyts]

Prior art on "iterator" and "async iterator" naming schemes in other languages:

• JavaScript: Symbol.Iterator, Symbol.AsyncIterator
• C#: IEnumerable, IAsyncEnumerable
• Python: __iter__, __aiter__
• Swift: Sequence, AsyncSequence

[bbros-dev]

Context: We're prototyping some simple CLI app functionality. We're following the mini-redis example code where we can.

We've bumped out head on streams, and the need for the crates async-streams, and parallel-streams.

In our experience some *Iterator terminology would have helped clarify what streams are.

Given the need for the crates we cited, especially the async-streams RFC, we wonder if there isn't a need for:

• Iterator
• ConcurrentIterator
• ParallelIterator

Or is the intention that async-stream and parallel-stream crate efforts all able to converge into a stream that covers the concurrent and parallel use cases?

[benkay86]

Rayon provides a whole ecosystem of parallel iterators on top of a work-stealing threadpool, and is currently the de facto Rust standard for parallel iteration. But I don't foresee a parallel iterator trait getting into the Rust standard library anytime soon.

Streams are supposed to cover the use case of concurrent iterators only (per my understanding). Hopefully once streams are stabilized into the Rust standard library we can have some syntax to use them in concurrent for loops just like we currently use synchronous iterators in for loops. However, there are still some issues to work out like what to do if a a stream panics or is dropped. Settling on a name (Stream vs AsyncIterator vs ConcurrentIterator) will be the easy part! 😜

Unfortunately, it's difficult to combine parallel and concurrent iteration at the moment. This would require Rayon to support a way to move tasks from worker threads to an async executor thread, or for an async executor like Tokio to support parallel thread pools in a more sophisticated way than tokio::task::spawn_blocking(). Until that happens, most programmers try to get all their data into memory first on a concurrent executor-driven threadpool and then offload the synchronous computation to a parallel threadpool (e.g. managed by Rayon).

[yoshuawuyts]

multiple people observed that something like AsyncIterator or similar might be much more evocative for new users. Such a name would allow people to map their existing understanding of iterators. "I understand Iterator, and I understand async, and this is an async version of Iterator".

I've filed a PR to the RFCs repo, updating the "streams" terminology to "async iterator" instead: rust-lang/rfcs#3208.

[noelzubin]

why did .next make it. where can i get more info on this ?

[clarfonthey]

One potential concern re: methods on AsyncIterator, which is less of a concern and more of a justification for requiring them, is methods that use internal versus external iteration.

For example, it can be very efficient to perform a fold on an iterator which is composed of several chains, but it can be much less efficient to directly call next on said iterators.

My concern is that without parity between the methods on Iterator and AsyncIterator, much of these optimisations that already exist for Iterator will be lost in the conversion to AsyncIterator. Since, as it stands, there's not really a way to run a for_each or fold on a regular iterator if the loop contains async operations.

I know for a fact that the current async ecosystem with futures::stream::Stream does not have parity with Iterator, with some notable surprises including the fact that try_fold requires the iterator item to also be composed of results, rather than just the return type of the function.

It would be extremely detrimental to the idea that "this is just the async version of an iterator" if there weren't parity there IMHO, since users might notice slowdowns in code that simply calls std::async_iter::from_iter without adding any extra async code at all.

[yoshuawuyts]

@clarfonthey yes, definitely. The concerns you raise are valid, and the working group is currently actively investigating how we can ensure parity between sync and async Rust APIs. We don't yet (but should) have guidelines on how methods on async traits should be translated from sync to async, but that likely needs us to land async closures / async traits first.

[joshtriplett]

Given the current trend of the async working group, I wouldn't be surprised if this can become async fn next rather than fn poll_next.

[yoshuawuyts]

@joshtriplett ah yes, that's definitely something we've been discussing within the working group, and we're currently working towards enabling that. The other thing we're currently researching is keyword-generics, which may allow us to merge the separate Iterator and AsyncIterator traits into a single Iterator trait which is generic over "asyncness".

I'll update the tracking issue to reflect both these items.

[withoutboats]

(NOT A CONTRIBUTION)

Given the current trend of the async working group, I wouldn't be surprised if this can become async fn next rather than fn poll_next.

ah yes, that's definitely something we've been discussing within the working group, and we're currently working towards enabling that.

I think this is not the right direction for this feature. I hold this view very strongly. AsyncIterator::poll_next enables library authors to write explicit, low-level async iteration primitives for unsafe optimizations. Relying on the compiler generated futures of an async function will make the layout of these types much less predictable or controllable and will take this away from users.

For ease of use where this fine-grained laying control is not desirable, generators are the play (async or otherwise), rather than having to write next methods (async or otherwise) at all.

You need to have the low level APIs (Future::poll, Iterator::next and AsyncIterator::poll_next) for hand-rolled optimized code that the compiler can't be relied on to generate. You need to have the high level syntax (async functions, generators, and async generators) for when people just want to get things done and don't care about these kinds of optimizations. You need both. An async next method would be doing each side of it only halfway (low-level iterator, high-level asynchrony), and would basically trap Rust in a local maxima that looks appealing from where we are now but would not be the best final state.

EDIT: What I mean when I say that "generators are the play" and "local maxima" is that I think because Iterator::next has always existed and has a superficially simple API (ie no Pin, no Context), its not as obvious that for ease of use implementing an iterator with a next method is actually an awful experience for users. Yielding from generators would be much easier. So when you want the ease-of-use story, you want generators, and those can be made async just as easily as functions can. But generators can't guarantee the representation that gets the codegen from for loops over slices looking so good, and similarly won't guarantee the optimizations some async code will want as well. You should be thinking of implementing Iterator::next as really as low-level as implementing Future::poll.

---

EDIT2: I think the counterargument to this is that mixing-and-matching high-level and low-level is also desirable. IE next + async is desirable when you want control over iteration but don't care about control over asynchrony. Analogously, there must be a hypothetical API that's control over asynchrony but compiler generated iteration - a polling generator(??). I think there could be a case to be made that users do want to be able to drop down into fine control over one aspect of their control flow but not the other, but then that should be an additional, third (and fourth?) option in addition to full control or full ease of use, you can't get rid of the full-control option, which is poll_next.

[madsmtm]

For the people that haven't closely followed along on the "blogosphere", I'll link to a few excellent blog posts about it (in chronological order) (authored by people in this thread) (by no means exhaustive):

• https://without.boats/blog/async-iterator/
• https://blog.yoshuawuyts.com/async-iterator-trait/
• https://without.boats/blog/poll-next/

---

Another argument in favour of fn poll_next that I've not seen explicitly mentioned: we could still provide a default async fn next impl, to somewhat improve the user experience of manually calling next:

async fn next(mut self: Pin<&mut Self>) -> Option<Self::Item> {
    poll_fn(|cx| self.as_mut().poll_next(cx)).await
}

// Or

async fn next(&mut self) -> Option<Self::Item>
where
    Self: Unpin,
{
    poll_fn(|cx| Pin::new(&mut *self).poll_next(cx)).await
}

Certainly, this is not as clean as the simple async fn next(&mut self), see this playground link for an example of how it might be more verbose, because we're allowing the iterator to be self-referential, but might serve to strike enough of a balance?

Whether Rust then goes with self: Pin<&mut Self> or Self: Unpin mostly depends on what the plans are in the future for making Iterator able to be self-referential, which I think is still an open question.

[the8472]

@withoutboats for your latest blog post, can you give an approximate loop desugaring like in #118847 (comment)
I'm not sure if I'm parsing the ascii charts properly.

Anyway, I'm reraising a concern here that I already mentioned in the PR and that's similar to clarfonthey's:

I think the current proposed interface is terrible for performance when iterating on small types (e.g. u8s) because the poll_next interface returns a Poll<Option<Item>> that conveys two states at the same time, readiness and end-of-iteration.
If the loop body contains no await points, just munching some bytes for example, then that loop body would ideally optimize to a single induction variable based on next's internals. I.e. just branching on "do we still have more data to process".
Only when reaching the end of available data it should poll. This also requires a separation of progress information and getting the next item(s), but it tries to solve a different problem than boat's proposal.

One option is to make poll_next not-async (basically just next) but only return items when the iterator has made progress, which would be polled by a separate method. poll_progress would then return a Poll<bool> I guess to indicate more items / end of iteration.

Another approach is returning an I where I: IntoIterator<IntoIter=IN>, IN: ExactSizeIterator from poll_next. Option fulfills these bounds but an async iter that has an internal buffer can instead choose to return some iterable with more than one item, which allows the loop body to process them on a single induction variable of that iterable without polling.

My async understanding is limited, I'm coming from the sync Iterators side. So I may have misunderstood something.

[jmjoy]

Given the current trend of the async working group, I wouldn't be surprised if this can become async fn next rather than fn poll_next.

After rust 1.75 released, trait async func is stable, async fn next is more user friendly.

[tesaguri]

Given the current trend of the async working group, I wouldn't be surprised if this can become async fn next rather than fn poll_next.

After rust 1.75 released, trait async func is stable, async fn next is more user friendly.

In the rest of the thread, withoutboats has argued that we need async generators for ergonomics, rather than async fn next(), which is less fine-grained than fn poll_next() and less ergonomic than async generators in their opinion. Then, what is your rationale for promoting async fn next() over async generators?

[RalfJung]

Looking at this through the lens of algebraic effects, I would say that

• Future corresponds to a (suspended ongoing invocation of a) function that can trigger the "pending" effect. The effect has argument and return type () (and eventually the function returns Future::Output).
• Iterator corresponds to a (suspended ongoing invocation of a) function that can trigger the "yield" effect. The effect has argument type Iterator::Item and return type () (and eventually the function returns ()).

This analogy is not quite perfect: futures can be polled again after returning "ready" (and similar for iterators); futures have this "context" argument; iterators are not pinned. But I would argue all of those are concessions to how we model these kinds of computations in Rust -- the abstract concept we want to model doesn't have them, but the imperfect realization of that concept in Rust does have them.

An AsyncIterator would then be a (suspended ongoing invocation of a) function that can trigger both the "pending" and the "yield" effect, with the same types as above (and eventually the function returns ()).

We don't have algebraic effects in Rust, but futures have shown how we can model one particular algebraic effect. If we follow that same paradigm, then the fn poll_next-based encoding seems to be the most direct way to represent an AyncIterator.

In contrast, the async fn next-based encoding seems to model something different: it represents a function that can trigger a "yield" effect, where the argument type of this effect (i.e., the data being passed from the function to the handler) is a function that can trigger a "pending" effect. If we view an async iterator as triggering a sequence of yield and pending, then this does seem equal in abstract expressivity to fn poll_next (split up the sequence after each yield, to obtain a sequence of subsequences; then each of these subsequences corresponds to one of the functions that is being yielded). However, it is a much less direct encoding of what actually happens, involving a seemingly unnecessary "thunk" (the functions being yielded). Sure, in trivial cases this can be optimized away, but I wouldn't bet much on the claim that such optimizations will always work. It certainly does not seem to fit the usual Rust philosophy of avoiding unnecessary overhead in the most basic abstractions.

So I guess what I am saying is, I tend to agree with boats. Mind you, I'm not an expert in async, I am taking a 10,000 foot view of this problem.

[bionicles]

speaking of https://doc.rust-lang.org/nightly/std/async_iter/index.html#laziness --- can the basic async iterator has a usage example? right now these docs show how to define the counter and implement the async iterator trait, but not how to actually invoke it or use it as an end user.

is this because to run this async iterator we need externals like tokio? if so, how do we ensure std has what it needs to run async iterators without needing an external runtime

here's the part i mean, the code isn't really showing what to do with the counter, so it's hard to look at this page of rustdoc and know fully how to get rolling with a minimalist async iterator, which tbh is likely to be much more rewarding for all of us than futures, just a hunch, async iterators are the move, because they promote categorical thinking [about i/o] and especially for input streams

seems like a no brainer to have the example here be a perfectly optimized copypasta for std-only async iterator over some particular user defined type of messages from a generic tcp stream output buffer of bytes, for example

as an aside, i wonder if, in the context of async, some subtle fundamental limitation in Poll<T> causes Pin<&mut self>

Which makes me wonder if

pub enum Ratchet<A, B> {
 Pending(A),
 Ready(B),
}

Could give us a different design avenue for async/await besides Poll<B> if I make sense.

Ratchet<A, B>::try_forward could be a better option than Poll<B> with Pin<&mut A>

because

the Ratchet::Pending(A) can take the place of Pin<&mut A> and be potentially better aligned with necessity to preserve referenceability of the owned pending state (giving it the term "A" instead of the disembodied "self" on some other detached thing)

but maybe

Ratchet<A, B> is too tightly coupled between the "self" of the "Future" (A) and the Future::Output (B)? just thinking out loud, sorry, carry on

[traviscross]

@rustbot labels -I-libs-api-nominated +T-lang +WG-async

This was discussed briefly in the libs-api call today. This work is going to be driven from the lang side on the basis of work done in WG-async, as we had discussed in the meeting on 2024-05-14. On that basis, the decision was to unnominate.

[RalfJung]

meeting on 2024-05-14

Where can I find the minutes of that meeting? In the t-lang/meetings Zulip channel, I can only find the meeting on 2024-05-15, but that was about a different topic (match ergonomics).

EDIT: Oh, that was a libs-api meeting, not a lang meeting. oops

[YuhanLiin]

One thing from Yosh's blog post that hasn't been mentioned here is that poll_next() and async fn next() have slightly different semantics, in that poll_next() requires the iterator to be pinned while async fn next() doesn't. Can't say for sure which one is better. I've never actually had to move an iterator after calling next() on it, but maybe that's a legitimate use case.

Another thing is that there's a lot of comparisons being made about the performance of poll_next() versus async fn next(). The easiest way to resolve this would be to implement an async iterator using both APIs, iterate over them using whatever for loop desugaring we're planning to ship, and compare the compiled assembly/LLVM IR. We should do this for non-trivial iterators and also mix in some combinators. Specifically, we can compare the compiled output of:

fn poll_next(self: Pin<&mut Self>, cx: &mut Context<'_>) -> Poll<Option<T>> {
    // Do something
    // Return Poll<Option<T>>
}

with the output of:

async fn next(&mut self) -> Option<T> {
    future::poll_fn(|cx| /* -> Poll<Option<T>> */ {
        // Do something
        // Return Poll<Option<T>>
    }).await
}

Since future::poll_fn allows async functions to "drop down" to lower level polling operations, we can put (mostly) the same code in the bodies of both implementations so that they perform the same operations, leading to a fair comparison. As a bonus, this experiment can also discover perf issues with the loop desugaring and combinator implementations.

[conradludgate]

poll_next() requires the iterator to be pinned while async fn next() doesn't

This makes poll_next more powerful. A Stream like https://docs.rs/tokio-util/latest/tokio_util/codec/struct.FramedRead.html currently works with any ! Unpin AsyncRead type. If we moved to async fn next(), the AsyncRead would need to be Unpin.

Similar story is for https://docs.rs/futures-concurrency/latest/futures_concurrency/stream/trait.Merge.html which would now need to cache the futures of each async fn next(). Since they are all !Unpin, Merge is now forced to box these next futures

[YuhanLiin]

Similar story is for https://docs.rs/futures-concurrency/latest/futures_concurrency/stream/trait.Merge.html which would now need to cache the futures of each async fn next(). Since they are all !Unpin, Merge is now forced to box these next futures

The main issue here is that with poll_next(), the iterator itself must contain the async state machine. Whereas with async fn next(), the async state machine is transiently created by the iterator. Self-contained state machines are much easier to cache and poll in an arbitrary order, which is what Merge does. Managing transient state machines created by async functions is a pain point of async traits in general.

[yoshuawuyts]

Similar story is for https://docs.rs/futures-concurrency/latest/futures_concurrency/stream/trait.Merge.html which would now need to cache the futures of each async fn next(). Since they are all !Unpin, Merge is now forced to box these next futures

I would be hesitant to take this as an inherent constraint. I'm fairly certain that with unsafe binders we can formulate an implementation of Merge based on async fn next which does not require any allocations. We'll need to play around a little with the traits and bounds, but I believe it should be possible.

Unsafe binders are going to be a great help for async traits regardless of their shape. They are the same feature required to e.g. make fn poll_next work with async closures, which is something we need for example for AsyncIterator::filter.

[YuhanLiin]

If we use async fn next, Merge will have to store the futures generated by each call to next. That means Merge will need to be pinned. I don't see how unsafe lifetime binders can get rid of the pinning requirement. The example of unsafe binders from the design doc also uses Pin<&mut Self>.

[yoshuawuyts]

I believe there might be a miscommunication: I didn't mean to comment on the pinning requirement, I meant to say it shouldn't be necessary to allocate. While immovability may be inherent to self-references, allocation should not be.

[conradludgate]

Right. I suggested box-pinning because at present the async fn next(&mut self) does not present a pinning interface. That is unless we expect to implement

impl AsyncIterator for Pin<&mut Merge<...>> { ... }

otherwise we'd need to instead offer

trait AsyncIterator {
    async fn next(self: Pin<&mut Self>) -> Option<Self::Item>;
}