batched TransformStream transform()?

[isonmad]

#551 (comment) proposing some kind of lowWaterMark got me thinking. Would it be worth it to change transformer.transform to be passed an array of chunks, instead of a single chunk?

By default with only one element, but with options that would enable

• specifying a maximum total size of the batched chunks (the maximum desired value of writableStrategy.size(chunks[0])+writableStrategy.size(chunks[1])+... )
• specifying a minimum total size of the batched chunks, such that transform() calls are delayed until sufficient chunks have been enqueued in the writable end (this would, for obvious reasons, be required to be lower than highWaterMark)

Could putting that kind of logic into TransformStream instead of having the transformer implement its own, third, internal queue and batching logic be useful?

[ricea]

In the case of fixed-block-size transforms, like encryption, I don't think it helps that much. Even if we provide a minimum total size, the transformer still has to deal with the case where they receive data that is not an exact multiple of the block size. So what they'll end up doing is storing an array of chunks from the last call to transform() that haven't been used yet (or keep a copy of the data in their own buffer). Once you've got that data structure it's trivial to add new chunks of data to it if you don't have enough to process a block yet. So the minimal total size ends up being not that helpful.

A way to say "feed me exactly 1024 bytes at a time, no more, no less" would solve this problem but seems a bit exotic. What I mean by "exotic" is that it solves a specific problem for one particular use-case but is useless for everyone else.

I think it would be natural to look at this again when #27 ("the writev problem") is resolved. If we have batched writes then the case for batched transforms become a lot more compelling.