do chunk boundaries have to be preserved through to underlying sink?

[dbaron]

https://streams.spec.whatwg.org/#ws-model says:

Chunks are written to the stream via its public interface, and are passed one at a time to the stream’s underlying sink.

This seems to imply that chunk boundaries have to be preserved when passing to the underlying sink. But that feels like a slightly odd requirement. Was it intended? If it was, perhaps it could be clearer, or if not, perhaps it should be reworded?

(I got here from w3ctag/design-reviews#163 .)

[ricea]

To a generic stream, chunks are opaque. ReadableStream has a type: 'bytes' option which causes chunks to be interpreted as sequences of bytes. WritableStream does not have a byte option yet, but it is planned: #495.

Application code can rely on streams not to modify chunks when the 'bytes' option is not set. In particular, a TransformStream may or may not respect chunk boundaries, depending on what semantics are appropriate.

[domenic]

In particular, even for byte streams, the underlying source sees the same chunks. It's not clear what else would work; would the stream machinery start slicing and dicing somehow between the two steps, instead of just being a pass-through?

[ricea]

When writing to a file or a network socket, large chunks are more efficient than small ones. But maybe we are more likely to do writev-style semantics than concatenating chunks inside WritableStream.

writev-style semantics are discussed in #27 along with a lot of discussion of cork. Personally I feel that cork is a niche use case compared to "please let me write all queued data in one go".

On the ReadableStream size, the source already has the ability to do appropriately-sized reads via desiredSize. So probably nothing to do there?

When pipeTo() is used it would nice if the underlying sink was able to simultaneously write all the data from the writable queue and the readable queue. But I have no idea if that can be standardised in a sane way.

[tyoshino]

Agreed with @ricea regarding non-byte streams.

---

WritableByteStreamController may have something like coalescing/splitting that the ReadableByteStreamController has though I'm not sure what's appropriate, yet.

E.g. IdentityTransformByteStream (Identity might be misleading though) may have a ReadableByteStream at the readable side and WritableByteStream at the writable side whose writer has .beginWrite() (see #329 (comment)) so that buffers provided to BYOB reader can be passed to the writer side as-is (but after transfer). WritableByteStream may also be able to vend a default writer which takes buffers and they are used to fill the buffer provided to the BYOB reader.

---

If we introduce writev()/cork style interface, yes, we basically want to pass them to the underlying sink as-is as @ricea said as the underlying sink may be able to do scatter-and-gather in efficient way.

If coalescing helper is worth being a part of the Streams API, we could add it. That could be e.g. in the form of .read(buffer) style API added to WritableByteStreamController though this needs diverting underlying sink API as currently we're pushing chunks to it by calling underlyingSink.write() (though with backpressure respected).