pipeline into IOBuffer broken on Windows

[simeonschaub]

MWE:

in a fresh session:

julia> sprint() do io
           run(pipeline(`echo foo`; stdout=io))
       end
""

julia> sprint() do io
           run(pipeline(`echo bar`; stdout=io))
       end
"bar\n"

I reduced this using a GitHub Actions Windows runner, but apparently this happens in Git Bash as well. I have seen this on master, as well as 1.5.

[vtjnash]

I think obviously that workaround is fundamentally flawed and should never be trusted. However, what's possibly less clear is whether we should remove the ability to set stdout to an IOBuffer, since it doesn't work correctly on any OS. The challenge is that since we're managing this buffer in Julia, we don't know to flush (or even close?) the buffer before the run command returns, so you can see old state.

[simeonschaub]

Is there a proper way to pipe stdout to somewhere in memory on Windows? Could we allocate an unnamed pipe that's not managed by Julia, but which Julia could then read from without excessive copies? (Sorry if this is a dumb question, I am not too familiar with how exactly pipes are internally handled by the OS.) I haven't had any problems so far on Linux, but since you removed the Windows tag, is this something that may currently be broken on Unix as well?
It would be quite unfortunate if we had to remove the ability to pass IOBuffers as stdout without a good alternative, as I find this very useful in a lot of situations.

[vtjnash]

Yeah, Base.BufferStream (perhaps we should export that?)

[vtjnash]

And possibly (particularly after #39544), using open do instead of run pipeline.

[simeonschaub]

Could you clarify how Base.BufferStream should be used properly in this instance? Does buffer_writes need to be set to true and do I need to take care of locking myself?

[vtjnash]

It is a drop-in replacement for IOBuffer in this case with the default constructor, etc.

[simeonschaub]

Hmm, that does not seem to fix the issue: https://github.com/simeonschaub/FIFOStreams.jl/runs/2768490500

[vtjnash]

Violating the invariants of any object by inspecting a field won’t get you anything useful. You need to call ‘read’.

[simeonschaub]

Ah, I was wondering about that. Is there a way to seek a BufferStream? Or would I need to start reading asynchronously before I start to write to the stream?

[vtjnash]

mark/reset should be defined, but the stream has flow control disabled by default so you don't need to read asynchronously