Add a streaming mode when running under pipe #1313
Assignees
Labels
Comments
davidliudev
changed the title
|
I also find this pretty bizarre, and would argue the current behavior is straight up incorrect.
|
|
It is definitely possible for |
|
@blueforesticarus for that particular use case there is the For some context on why it behaves this way, see #736 and #885 |
tavianator
added a commit
to tavianator/fd
that referenced
this issue
Dec 13, 2023
The previous behaviour was designed to mimic the output buffering of typical UNIX tools: line-buffered if stdout is a TTY, and fully-buffered otherwise. More precicely, when printing to a terminal, fd would flush explicitly after printing any buffered results, then flush after every single result once streaming mode started. When not printing to a terminal, fd never explicitly flushed, so writes would only happen as the BufWriter filled up. The new behaviour actually unifies the TTY and non-TTY cases: we flush after printing the buffered results, then once we start streaming, we flush after each batch, but *only when the channel is empty*. This provides a good balance: if the channel is empty, the receiver thread might as well flush before it goes to sleep waiting for more results. If the channel is non-empty, we might as well process those results before deciding to flush. For TTYs, this should improve performance by consolidating write() calls without sacrificing interactivity. For non-TTYs, we'll be flushing more often, but only when the receiver would otherwise have nothing to do, thus improving interactivity without sacrificing performance. This is particularly handy when fd is piped into another command (such as head or grep): with the old behaviour, fd could wait for the whole traversal to finish before printing anything. With the new behaviour, fd will print those results soon after they are received. Fixes sharkdp#1313.
sharkdp
pushed a commit
that referenced
this issue
Dec 18, 2023
The previous behaviour was designed to mimic the output buffering of typical UNIX tools: line-buffered if stdout is a TTY, and fully-buffered otherwise. More precicely, when printing to a terminal, fd would flush explicitly after printing any buffered results, then flush after every single result once streaming mode started. When not printing to a terminal, fd never explicitly flushed, so writes would only happen as the BufWriter filled up. The new behaviour actually unifies the TTY and non-TTY cases: we flush after printing the buffered results, then once we start streaming, we flush after each batch, but *only when the channel is empty*. This provides a good balance: if the channel is empty, the receiver thread might as well flush before it goes to sleep waiting for more results. If the channel is non-empty, we might as well process those results before deciding to flush. For TTYs, this should improve performance by consolidating write() calls without sacrificing interactivity. For non-TTYs, we'll be flushing more often, but only when the receiver would otherwise have nothing to do, thus improving interactivity without sacrificing performance. This is particularly handy when fd is piped into another command (such as head or grep): with the old behaviour, fd could wait for the whole traversal to finish before printing anything. With the new behaviour, fd will print those results soon after they are received. Fixes #1313.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Currently when not running under an interactive shell, it will buffer its output and output everything in one go.
However, I am writing a GUI front end and need this "streaming" mode similar to when running under an interactive shell so users can see the result in a faster manner.
Expected behaviour is that the user can instantly see the initial result with more items being added along the way.
Am I missing something or is there any switch to customize this behavior?
The text was updated successfully, but these errors were encountered: