lib: performance improvement on readline async iterator #41276

Using a direct approach to create the readline async iterator allowed an iteration over 20 to 58% faster. **BREAKING CHANGE**: With that change, the async iteterator obtained from the readline interface doesn't have the property "stream" any longer. This happened because it's no longer created through a Readable, instead, the async iterator is created directly from the events of the readline interface instance, so, if anyone is using that property, this change will break their code. Also, the Readable added a backpressure control that is fairly compensated by the use of FixedQueue + monitoring its size. This control wasn't really precise with readline before, though, because it only pauses the reading of the original stream, but the lines generated from the last message received from it was still emitted. For example: if the readable was paused at 1000 messages but the last one received generated 10k lines, but no further messages were emitted again until the queue was lower than the readable highWaterMark. A similar behavior still happens with the new implementation, but the highWaterMark used is fixed: 1024, and the original stream is resumed again only after the queue is cleared. Before making that change, I created a package implementing the same concept used here to validate it. You can find it [here](https://github.com/Farenheith/faster-readline-iterator) if this helps anyhow.

Some Changes I just sent wasn't related to the PR in question, so I'm undoing then (in events.js and the addition of linkedMap.js) Also, fixed the [SymbolAsyncIterator] function returned by eventsToAsyncIteratorFactory, that was referencing a no longer available variable

After seeing how once method works on EventEmitter, it was clear that it doesn't make sense to use it into the waitNext method, as the callback will already remove the events

Adding different lines multipliers is good for this bench because we can see how well the change performs in different scales. That way we can find out if there some curve with some point where the benefits starts or ends

…actory

Using a direct approach to create the readline async iterator allowed an iteration over 20 to 58% faster. **BREAKING CHANGE**: With that change, the async iteterator obtained from the readline interface doesn't have the property "stream" any longer. This happened because it's no longer created through a Readable, instead, the async iterator is created directly from the events of the readline interface instance, so, if anyone is using that property, this change will break their code. Also, the Readable added a backpressure control that is fairly compensated by the use of FixedQueue + monitoring its size. This control wasn't really precise with readline before, though, because it only pauses the reading of the original stream, but the lines generated from the last message received from it was still emitted. For example: if the readable was paused at 1000 messages but the last one received generated 10k lines, but no further messages were emitted again until the queue was lower than the readable highWaterMark. A similar behavior still happens with the new implementation, but the highWaterMark used is fixed: 1024, and the original stream is resumed again only after the queue is cleared. Before making that change, I created a package implementing the same concept used here to validate it. You can find it [here](https://github.com/Farenheith/faster-readline-iterator) if this helps anyhow.

Some Changes I just sent wasn't related to the PR in question, so I'm undoing then (in events.js and the addition of linkedMap.js) Also, fixed the [SymbolAsyncIterator] function returned by eventsToAsyncIteratorFactory, that was referencing a no longer available variable

After seeing how once method works on EventEmitter, it was clear that it doesn't make sense to use it into the waitNext method, as the callback will already remove the events

…actory

Co-authored-by: Antoine du Hamel <duhamelantoine1995@gmail.com>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

lib: performance improvement on readline async iterator #41276

lib: performance improvement on readline async iterator #41276

Commits on Oct 18, 2022