Add TextEncoder.prototype.encodeInto

[RReverser]

Is your feature request related to a problem? Please describe.

TextEncoder.prototype.encodeInto is a new method available in latest Chrome and Firefox that allows efficient encoding of strings into a given buffer, avoiding intermediate allocation and copying.

Its primary usecase is a more efficient way to pass strings to WASM (one of the most expensive things one has to do when communicating with JavaScript) and it's already leveraged by Rust wasm-bindgen where available. wasm-bindgen has support for a Node.js target, so it would be great to speed things up here too.

Describe the solution you'd like

Adding native encodeInto method to the TextEncoder class already available in util module.

Please describe the desired behavior.

MDN: https://developer.mozilla.org/en-US/docs/Web/API/TextEncoder/encodeInto

Describe alternatives you've considered

Sending PR to wasm-bindgen to wrap memory into a Node-specific Buffer and use Buffer.prototype.write instead.

It has very similar semantics, and produces similar speed-ups (up to 35% on some of my string-interaction-heavy WASM libraries), but I wanted to see if it's possible to solve this on the Node.js side first so that the same method could be used for all platforms.

cc @jasnell as the author of the original implementation of WHATWG Encoding API in Node.js (#13644)

[jasnell]

Should be fairly straightforward to implement. Interested in giving it a try?

[RReverser]

Potentially, but currently pretty busy with another higher-level project (in which I found out this perf differentiator between running my WASM in different environments).

If nobody takes this before, can give it a try sometime later.

Do you think it would be reasonable if I put good first issue label on this?

[jasnell]

Labeling it good first issue should be fine but it will require a bit deeper knowledge than a beginner might have, and would likely require a bit of C++

[RReverser]

Yeah, depends whether that targets beginners in general or just beginners to the project... I think I can put "good first issue" + "C++" to make it more clear.

[RReverser]

One difference with Buffer::write I should mention is that encodeInto additionally returns number of string code units written, not just bytes. This is important to allow partial writes. (In theory people using Buffer::write for partial writes would need something like that too, but somehow they got away without it?)

[JungMinu]

going to take care of this

[zandaqo]

This would be really helpful for anyone dealing with binary. As much as I like Buffer, it's not an option for isomorphic code. Is there any ETA on this one yet?

[AtticusYang]

@RReverser ,I tried to implement this task and I did it. but I am confused by the test case in
https://github.com/nodejs/node/blob/master/test/fixtures/wpt/encoding/encodeInto.any.js,
when input string is "input": "\u{1D306}", the expected read is 2, but the actual is 4, i also did a simple test as followed, the byte length of "\u{1D306}" is 4.

let a = "\u{1D306}";
console.log(a);
console.log(Buffer.byteLength(a, 'utf8'));

and the dest is a uint8array, the byteOffset is 0 and the length is 4.

so I wonder the test case is right or wrong, thanks!

[zandaqo]

@AtticusYang TextEncoder#encodeInto returns the number of code points not the byte length--"\u{1D306}" is encoded with two code points in UTF-16, and indeed four bytes in UTF-8.

[AtticusYang]

@zandaqo thank for your advice.

[zandaqo]

I was playing with the latest nightly build (Sep 23) and noticed an odd behavior when non-string is used as a source:

const encoder = new TextEncoder();
const destination = new Uint8Array(10);

encoder.encodeInto(null, destination);
//=> TypeError: The "src" argument must be of type string. Received type object

While the spec does say that the source should be a USVString, its current implementations in both Chrome and Firefox opt for converting the source into a string first instead of throwing an error. So, are we sticking with the current implementation, that is, Node.js going to differ from browsers and throw if a non-string source is provided?

Also, this throwing makes less sense considering that we are ok with non-string values in TextEncoder#encode:

const encoder = new TextEncoder();
encoder.encode(null)
//=> Uint8Array [ 110, 117, 108, 108 ]