buffer: add buffer.transcode #9038

jasnell · 2016-10-11T21:16:00Z

Checklist

make -j8 test (UNIX), or vcbuild test nosign (Windows) passes
tests and/or benchmarks are included
documentation is changed or added
commit message follows commit guidelines

Affected core subsystem(s)

buffer

Description of change

Add buffer.transcode(source, from, to) method. Primarily uses ICU to transcode a buffer's content from one of Node.js' supported encodings to another.

Originally part of a proposal to add a new unicode module. Decided to refactor the approach towrds individual PRs without a new module.

Refs: #8075
/cc @trevnorris @addaleax

jasnell · 2016-10-11T21:16:52Z

CI: https://ci.nodejs.org/job/node-test-pull-request/4476/

trevnorris · 2016-10-11T23:35:22Z

lib/internal/buffer.js

+const Buffer = require('buffer').Buffer;
+const normalizeEncoding = require('internal/util').normalizeEncoding;
+
+if (process.binding('config').hasIntl) {


what about possibly placing this in lib/internal/buffer-transcode.js and conditionally require()'ing it. purely cosmetic though, to prevent an extra level of indent. or you could just return early. :)

if (!process.binding('config').hasIntl) return;

trevnorris · 2016-10-11T23:36:33Z

lib/internal/buffer.js

+  // Buffer instance.
+  exports.transcode = function transcode(source, from_enc, to_enc) {
+    if (!source || !(source.buffer instanceof ArrayBuffer))
+      throw new TypeError('"source" argument must be a Buffer');


We going to have a complaint about not supporting SharedArrayBuffer for this?

Eventually, perhaps. Not too worried about that for now.

jasnell · 2016-10-12T00:03:50Z

Updated

addaleax · 2016-10-11T21:26:51Z

doc/api/buffer.md

+Returns a new `Buffer` instance.
+
+Throws if transcoding is not possible or if one of the specified encodings is
+invalid or unknown.


Maybe give an example of transcoding is not possible?

addaleax · 2016-10-11T22:54:05Z

src/node_i18n.cc

+  return e;
+}
+
+#define THROW_ICU_ERROR(env, status, msg)                                     \


Hm – this could be a ThrowICUError function, right?

addaleax · 2016-10-11T22:55:53Z

src/node_i18n.cc

+
+MaybeLocal<Object> AsBuffer(Isolate* isolate,
+                            MaybeStackBuffer<char>* buf,
+                            size_t len) {


This is fine but at some point this might become a member of the MaybeStackBuffer class? I realize that would conflict a bit with the MaybeStackBuffer<UChar> overload, maybe leave a TODO here?

addaleax · 2016-10-12T08:58:22Z

lib/internal/buffer.js

+const icu = process.binding('icu');
+
+// Maps the supported transcoding conversions. The top key is the from_enc,
+// the child key is the to_enc. The value is the transcoding function to.


Is that final to residue from editing?

Yeah, slight brain malfunction there I think ;-)

addaleax · 2016-10-12T08:58:44Z

lib/internal/buffer.js

+      return source.toString('base64');
+    },
+    'hex': (source) => {
+      return source.toString('hex');


Mhhh this returns a string or a Buffer depending on the target encoding? I don’t think binary-to-text encodings should be allowed here, .toString() is the right method for them.

Yeah, you're right. I'll pull these back out.

jasnell · 2016-10-12T14:35:02Z

@addaleax ... updated! PTAL

addaleax

LGTM, nice!

addaleax · 2016-10-12T21:26:19Z

src/node_i18n.cc

+      const uint8_t hi = static_cast<uint8_t>(ts_obj_data[n + 0]);
+      const uint8_t lo = static_cast<uint8_t>(ts_obj_data[n + 1]);
+      swapspace[i] = (hi << 8) | lo;
+    }


Could this use SwapBytes16?

addaleax · 2016-10-12T21:27:43Z

src/node_i18n.cc

+      const uint8_t hi = static_cast<uint8_t>(ts_obj_data[n + 0]);
+      const uint8_t lo = static_cast<uint8_t>(ts_obj_data[n + 1]);
+      swapspace[i] = (hi << 8) | lo;
+    }


addaleax · 2016-10-12T21:32:49Z

src/node_i18n.cc

+  MaybeStackBuffer<char> buf;
+  int32_t len;
+
+  u_strToUTF8(*buf, 1024, &len, source, length, &status);


The 1024 seem kind of magic here, although I realize that is largely my fault. 😄 (Not sure if there’s anything to do about that)

Should be fixed now!

bnoordhuis · 2016-10-13T05:23:13Z

lib/internal/buffer.js

+
+// Transcodes the Buffer from one encoding to another, returning a new
+// Buffer instance.
+exports.transcode = function transcode(source, from_enc, to_enc) {


Style: s/from_enc/fromEncoding/ and s/to_enc/toEncoding/. Ditto for cnv_from and cnv_to.

bnoordhuis · 2016-10-13T05:28:42Z

src/node_i18n.cc

+    msg = "Unspecified ICU Exception";
+  Local<String> cons =
+      String::Concat(estring, FIXED_ONE_BYTE_STRING(env->isolate(), ", "));
+  cons = String::Concat(cons, OneByteString(env->isolate(), msg));


I realize you adapted this code from elsewhere but using snprintf() to format the error message will be much more efficient.

bnoordhuis · 2016-10-13T05:29:47Z

src/node_i18n.cc

+      String::Concat(estring, FIXED_ONE_BYTE_STRING(env->isolate(), ", "));
+  cons = String::Concat(cons, OneByteString(env->isolate(), msg));
+
+  Local<Value> e = Exception::Error(cons);


Needs a if (e.Empty()) return Local<Value>();.

bnoordhuis · 2016-10-13T05:32:58Z

src/node_i18n.cc

+                            size_t len) {
+  if (buf->IsAllocated()) {
+    MaybeLocal<Object> ret = Buffer::New(isolate, buf->out(), len);
+    buf->Release();


I think this should be if (!ret.Empty()) buf->Release(); - it's leaking memory now when the buffer can't be created.

bnoordhuis · 2016-10-13T05:33:43Z

src/node_i18n.cc

+  if (buf->IsAllocated()) {
+    MaybeLocal<Object> ret =
+        Buffer::New(isolate, reinterpret_cast<char*>(buf->out()), len);
+    buf->Release();


bnoordhuis · 2016-10-13T05:46:49Z

src/node_i18n.cc

+  UChar* source = nullptr;
+  MaybeStackBuffer<UChar> swapspace;
+  if (IsLittleEndian()) {
+    source = reinterpret_cast<UChar*>(ts_obj_data);


Same as above: strict aliasing violation and prone to crashing.

bnoordhuis · 2016-10-13T05:49:22Z

src/node_i18n.cc

+  } else {
+    ThrowICUError(env, status, "Unable to transcode buffer");
+  }
+}


It seems like there is ample opportunity to share code between Ucs2FromUtf8 and Utf8FromUcs2, they are 80% identical.

Perhaps. For now I'm more inclined to keep these separate as it makes finding and tweaking bugs a bit easier. I'll take another pass in a separate PR to condense things down.

bnoordhuis · 2016-10-13T05:50:25Z

src/util.h

+  }
+
+  void Release() {
+    buf_ = buf_st_;


Shouldn't this also reset length_?

bnoordhuis · 2016-10-13T05:51:54Z

src/util.h

+      return env->ThrowTypeError("argument should be a Buffer");            \
+  } while (0)
+
+#define SPREAD_ARG(val, name)                                                 \


If you move this into a common header, you might want to give it a slightly less generic name; e.g. SPREAD_BUFFER_ARG.

bnoordhuis · 2016-10-13T05:52:37Z

tools/icu/icu-generic.gyp

@@ -21,7 +21,7 @@
      'toolsets': [ 'target' ],
      'direct_dependent_settings': {
        'defines': [
-          'UCONFIG_NO_CONVERSION=1',
+          #'UCONFIG_NO_CONVERSION=1',


Just remove the 'defines' block instead of commenting it out.

jasnell · 2016-10-13T15:13:04Z

~~New CI: https://ci.nodejs.org/job/node-test-pull-request/4503/~~
Issues on big endian... trying again: https://ci.nodejs.org/job/node-test-pull-request/4504/

jasnell · 2016-10-13T18:01:47Z

~~Another CI run after cleanups: https://ci.nodejs.org/job/node-test-pull-request/4507/~~ ... that last run was less than successful....
Trying another: https://ci.nodejs.org/job/node-test-pull-request/4508/

jasnell · 2016-10-13T20:40:57Z

CI looks good. @bnoordhuis PTAL... LGTY?

jasnell · 2016-10-17T17:17:12Z

ping @bnoordhuis

trevnorris · 2016-10-17T22:36:07Z

src/node_i18n.cc

+                       buf, v8::NewStringType::kNormal,
+                       len).ToLocalChecked());
+  if (e.IsEmpty())
+    return Local<Value>();


if you're returning empty handles as an indicator it's probably be more "V8-ish" have the return signature as a MaybeLocal<Value> instead. been trying to do that in other locations myself.

trevnorris · 2016-10-17T22:39:02Z

src/node_i18n.cc

+  char buf[kStorageSize];
+  int len = snprintf(buf, sizeof(buf), "%s [%s]", msg, u_errorName(status));
+
+  Local<Value> e =


creating Local<Value>'s but there's no HandleScope. if this callback is expected to always be called within an existing HandleScope (like MakeCallback), mind putting a comment at the top. also like MakeCallback (see src/node.h).

trevnorris · 2016-10-17T22:39:55Z

src/node_i18n.cc

+                       len).ToLocalChecked());
+  if (e.IsEmpty())
+    return Local<Value>();
+  Local<Object> obj = e->ToObject(env->isolate());


if we know this is a v8::Object then can use e.As<Object>(). that's also more explicit that no extra handle is being created.

trevnorris · 2016-10-17T22:41:13Z

src/node_i18n.cc

+  obj->Set(env->code_string(),
+           String::NewFromUtf8(env->isolate(),
+                               u_errorName(status), v8::NewStringType::kNormal)
+                                 .ToLocalChecked());


The new v8::Maybe<T> API for v8::Object::Set() is annoying and ugly, but if if we're going to use some of the new API might as well use all of it.

trevnorris · 2016-10-17T22:44:43Z

src/node_i18n.cc

+    if (!ret.IsEmpty()) buf->Release();
+    return ret;
+  }
+  return Buffer::Copy(isolate, dst, len);


if you're manipulating the original memory, why bother take a copy?

trevnorris · 2016-10-17T22:46:50Z

src/node_i18n.cc

+  if (U_SUCCESS(status)) {
+    len = target - *buf;
+    args.GetReturnValue().Set(
+        AsBuffer(env->isolate(), &buf, len).ToLocalChecked());


if this operation fails, do we want to abort or throw?

trevnorris · 2016-10-17T22:50:08Z

src/node_i18n.cc

+    for (size_t n = 0, i = 0; i < length; n += 2, i += 1) {
+      const uint8_t hi = static_cast<uint8_t>(ts_obj_data[n + 0]);
+      const uint8_t lo = static_cast<uint8_t>(ts_obj_data[n + 1]);
+      swapspace[i] = (lo << 8) | hi;


for future performance enhancement, detect the alignment of the pointer and perform as many swaps that can be done in a single go.

trevnorris · 2016-10-17T22:53:52Z

lib/internal/buffer.js

+const conversions = {
+  'ascii': {
+    'latin1': (source) => {
+      return Buffer.from(source);


i'm a little confused by this whole object, but right here if we're converting ascii to latin1 shouldn't we be passing 'latin1' as the encoding argument to Buffer.from()?

jasnell · 2016-10-18T21:44:18Z

@bnoordhuis ... ok, reworked the implementation with an eye towards simplification and reducing duplication. PTAL
@trevnorris and @addaleax ... if I could trouble each of you to take another look also, I'd appreciate it.

jasnell · 2016-10-21T17:11:03Z

ping @bnoordhuis @trevnorris @addaleax

jasnell · 2016-10-24T17:33:32Z

Thank you for the follow up review @addaleax. Will get this landed tomorrow if there are no further objections.

Add buffer.transcode(source, from, to) method. Primarily uses ICU to transcode a buffer's content from one of Node.js' supported encodings to another. Originally part of a proposal to add a new unicode module. Decided to refactor the approach towrds individual PRs without a new module. Refs: nodejs#8075

jasnell · 2016-10-25T16:12:56Z

New CI run after squashing: https://ci.nodejs.org/job/node-test-pull-request/4665/

jasnell · 2016-10-25T17:10:46Z

green except for unrelated failures. landing

PR updated after view

Add buffer.transcode(source, from, to) method. Primarily uses ICU to transcode a buffer's content from one of Node.js' supported encodings to another. Originally part of a proposal to add a new unicode module. Decided to refactor the approach towrds individual PRs without a new module. Refs: #8075 PR-URL: #9038 Reviewed-By: Anna Henningsen <anna@addaleax.net>

jasnell · 2016-10-25T17:13:50Z

Landed in e8eaaa7

Add buffer.transcode(source, from, to) method. Primarily uses ICU to transcode a buffer's content from one of Node.js' supported encodings to another. Originally part of a proposal to add a new unicode module. Decided to refactor the approach towrds individual PRs without a new module. Refs: #8075 PR-URL: #9038 Reviewed-By: Anna Henningsen <anna@addaleax.net>

addaleax · 2016-12-21T05:00:51Z

If this is backported to any of the other release lines, it needs to come with #9838

jasnell added the buffer Issues and PRs related to the buffer subsystem. label Oct 11, 2016

nodejs-github-bot added build Issues and PRs related to build files or the CI. c++ Issues and PRs that require attention from people who are familiar with C++. tools Issues and PRs related to the tools directory. labels Oct 11, 2016

jasnell added the semver-minor PRs that contain new features and should be released in the next minor version. label Oct 11, 2016

trevnorris reviewed Oct 11, 2016

View reviewed changes

addaleax reviewed Oct 12, 2016

View reviewed changes

jasnell force-pushed the buffer-transcode branch from 5b65f74 to 62423a0 Compare October 12, 2016 14:18

addaleax approved these changes Oct 12, 2016

View reviewed changes

bnoordhuis requested changes Oct 13, 2016

View reviewed changes

jasnell force-pushed the buffer-transcode branch 3 times, most recently from 9e0464b to 2da087e Compare October 13, 2016 15:11

jasnell force-pushed the buffer-transcode branch from eecad5e to d7340ec Compare October 13, 2016 20:09

trevnorris reviewed Oct 17, 2016

View reviewed changes

jasnell force-pushed the buffer-transcode branch 3 times, most recently from 6f70820 to 65144bb Compare October 18, 2016 21:43

addaleax approved these changes Oct 21, 2016

View reviewed changes

jasnell force-pushed the buffer-transcode branch from 65144bb to 4d7472b Compare October 25, 2016 16:12

jasnell closed this Oct 25, 2016

srl295 mentioned this pull request Oct 25, 2016

intl: Add more versions from ICU #9266

Closed

4 tasks

evanlucas mentioned this pull request Nov 3, 2016

v7.1.0 proposal - 2016-11-08 #9438

Merged

ChALkeR mentioned this pull request Nov 25, 2016

Feature proposal: buffer.convert(value[, fromEnc], toEnc) #9797

Closed

addaleax mentioned this pull request Dec 21, 2016

buffer: fix transcode for single-byte enc to ucs2 #9838

Closed

3 tasks

MylesBorins added lts-agenda labels Jan 6, 2017

MylesBorins mentioned this pull request Jan 6, 2017

Potential Semver Minor Backports nodejs/Release#177

Closed

9 tasks

MylesBorins added dont-land-on-v4.x and removed lts-watch-v4.x labels Jan 16, 2017

buffer: add buffer.transcode #9038

buffer: add buffer.transcode #9038

Conversation

jasnell commented Oct 11, 2016 • edited Loading

Checklist

Affected core subsystem(s)

Description of change

jasnell commented Oct 11, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jasnell commented Oct 12, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

addaleax Oct 11, 2016 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jasnell commented Oct 12, 2016

addaleax left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jasnell commented Oct 13, 2016 • edited Loading

jasnell commented Oct 13, 2016 • edited Loading

jasnell commented Oct 13, 2016

jasnell commented Oct 17, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jasnell commented Oct 18, 2016

jasnell commented Oct 21, 2016

jasnell commented Oct 24, 2016

jasnell commented Oct 25, 2016

jasnell commented Oct 25, 2016

jasnell commented Oct 25, 2016

addaleax commented Dec 21, 2016

jasnell commented Oct 11, 2016 •

edited

Loading

addaleax Oct 11, 2016 •

edited

Loading

jasnell commented Oct 13, 2016 •

edited

Loading

jasnell commented Oct 13, 2016 •

edited

Loading