lib: externalized messages support #4311

jasnell · 2015-12-16T17:56:23Z

@nodejs/ctc @nodejs/collaborators ...

externalizing the error and output messages in lib

the approach taken here is derived from the template
string based mechanism described here
this is an alternative approach to the ICU based approach
that I am working on in src: initial localization support (work in progress) #3413

This introduces a new internal/messages module that exports a
template string tag function. The externalized messages are in
internal/messages_en.js for now but can be moved to a different
location.

Within the lib src, we can use this mechanism using tagged template
strings:

const I18N = require('internal/messages');

const foo = 'testing';
throw new Error(I18N`Example error: ${foo}`);

In the messages_en.js file, we'd have something like:

{
  ...,
  "Example error: {0}": {
    "key": "ERR000009"
  },
  ...
}

The template string would expand into:

ERR000009: Example Error: testing

Later on, when we get into actually translating messages, we
would have other message_{locale}.js files (or however else we'd
like to separate it out) that would look something like:

{
  ...,
  "Example error: {0}": {
    "key": "ERR000009",
    "value": "Ejemplo de error: {0}"
}

I'm not yet convinced this approach is ideal, but it's functional.
There are a few gotchas with template strings currently that are less
than ideal (e.g. line wrapping issues, lookup is a bit inefficient).

Getting this out now, tho, to solicit input on the approach.

This is still a work in progress

Fishrock123 · 2015-12-16T18:17:44Z

lib/internal/messages_en.js

+  "Incorrect value for stdio stream: {0}": {
+    "key": "ERR000010"
+  },
+


What are the spaces for?

This isn't JSON, so we could add comments.

Just visual separation for now. As I said, it's a work in progress ;-) I'm not in love with the idea of stuffing all these into a single js file

Qard · 2015-12-16T18:41:28Z

It might be neat if the mechanisms behind this could be usable in userland. Just a thought.

indutny · 2015-12-16T19:20:31Z

@jasnell what do you think about precompiling this as we currently do for debug(). Do our users need to swap locales when using the same node.js binary?

What I don't like about this approach is that it adds runtime cost to the most (if not all) textual reporting in core. Ideally we may use some kind of macro in C++ or whenever to replace this I18N... with their translation either during startup, or during compilation of node.js binary.

Fishrock123 · 2015-12-16T19:28:37Z

lib/internal/messages.js

+'use strict';
+
+// adapted from the approach documented at:
+// http://jaysoo.ca/2014/03/20/i18n-with-es6-template-strings/


Should we maybe note why we are doing it like this a bit more here?

Also since this won;t be in the API docs it may be useful to have a quick API-doc thing here. (i.e. what to put in, what comes out)

Fishrock123 · 2015-12-16T19:29:27Z

I think the approach is pretty nice, but I also agree with @indutny.

jasnell · 2015-12-16T19:40:08Z

just to be certain, I'm definitely not sold on this approach yet.

@Fishrock123 @indutny ... we could certainly use compiled variants to improve performance. That does get a bit difficult to manage (one binary for each supported language) but it's certainly not out of the question (it's just a bit more work for the build and release team).

I'd be favorable of that approach but we'd need to be clear about the impact and we'd still need to have the clear ID to message mapping. A pre-processor that runs during make wouldn't be too difficult to build.

domenic · 2015-12-16T19:52:06Z

Are we really worried about runtime costs in error situations anyway?

jasnell · 2015-12-16T20:00:53Z

likely not, but there's a trade off. I'd rather not beef up the binary size to hold a bunch of translated strings if we can avoid it.

mcollina · 2015-12-16T23:40:54Z

That's great seeing progress in this area, as a non-native English speaker I rejoice. Even though I like my errors to be in English (mainly because they are easier to search, and I am lazy), translated errors were really helpful when my English was poor.

A couple of things: packagers (deb, rpm) usually package lang files separately. There are reasons for doing this: lots of languages, not requiring a release for adding a locale, ecc. However, if these are not in the main bundle, it is unlikely that a newbie would install them in the first place. On Windows, we can possibly show a nice dialog. But on Mac and Linux? We need to have a clear story for packagers.

Regarding performance: what is the impact? Most platforms were performance is important (e.g. iOS, Android) have externalized errors/strings.

Really really dumb question: why can't we just have functions? Basically we would have a i18n_en.js that is looked up and exposed by i18n at startup.

const ERR000009 = I18N.ERR000009

throw new Error(ERR000009('a', 'b'))

Last thing to consider: a lot of userland modules might want/would benefit of similar functionality.

Fishrock123 · 2015-12-17T00:29:23Z

Last thing to consider: a lot of userland modules might want/would benefit of similar functionality.

At very least, it would probably be a good standard to have.

@jasnell Maybe consider prefixing with NODE_?

jasnell · 2015-12-17T00:54:37Z

@mcollina .. the main issue with the function form (which I considered also) is that we'd have to create the function objects for each message -- which isn't out of the question but that ends up being a lot of additional objects kept around. Not sure if it's enough of an impact to worry about tho.

The main challenge with this is that there are several approaches we can take to the API and it's easy to bike shed. For the most part, we simply need to come up with one approach that everyone can at least work with. Once that's settled I can get it implemented and run performance benchmarks.

@Fishrock123 ... the other question I have is whether there should be a single generic naming pattern or if we should tie it to the module... for instance instead of NODE_ERR0001, NODE_ERR0002 it would be like NODE_CHILDPROC_ERR1, NODE_STREAMS_ERR1 that kind of thing.

Fishrock123 · 2015-12-17T00:55:59Z

@jasnell ooh, that could be even better.

jasnell · 2015-12-17T00:59:09Z

@mcollina ... btw, as part of this effort, once the approach is settled on, I'm planning to implement a command line tool that can accept one of the message identifiers and output the explanation for it, preferably with localization as well. In other words, if you get an ERR_CHILDPROC_ERR1 you would be able to do $ node-explain ERR_CHILDPROC_ERR1 and it would output a description of the error and other relevant information. Obviously that's a much larger chunk of work tho.

YurySolovyov · 2015-12-17T07:47:25Z

@jasnell
I don't quite get why

we'd have to create the function objects for each message

Can you explain?

mcollina · 2015-12-17T08:35:56Z

@mcollina .. the main issue with the function form (which I considered also) is that we'd have to create the function objects for each message -- which isn't out of the question but that ends up being a lot of additional objects kept around. Not sure if it's enough of an impact to worry about tho.

I am ok for trading a bit a of memory for speed. It would probably be negligible, given all the functions return strings that would also need to be maintained in memory. However, it is something to worry about for low-memory devices (IoT?).

The main challenge with this is that there are several approaches we can take to the API and it's easy to bike shed. For the most part, we simply need to come up with one approach that everyone can at least work with. Once that's settled I can get it implemented and run performance benchmarks.

I agree. We should possibly iterate on it between releases.

@Fishrock123 ... the other question I have is whether there should be a single generic naming pattern or if we should tie it to the module... for instance instead of NODE_ERR0001, NODE_ERR0002 it would be like NODE_CHILDPROC_ERR1, NODE_STREAMS_ERR1 that kind of thing.

👍, much better.

@mcollina ... btw, as part of this effort, once the approach is settled on, I'm planning to implement a command line tool that can accept one of the message identifiers and output the explanation for it, preferably with localization as well.

Yes, but it is really needed. There are some incomplete/bad explanations around the net.

jasnell · 2016-01-07T22:27:17Z

@Fishrock123 @mcollina @indutny @Qard @nodejs/ctc @nodejs/collaborators ... Ok, I've taken a step back and did a major refactoring on this and force pushed and update. The new approach is documented in the new commit log in 68792d5.

The changes are:

Messages are now moved into the compiled node binary under src/messages/en/messages.h (previously they were in a *.js file)
A process.binding('messages') object and an internal/messages module are provided to give access to these messages from within lib.
Message keys are defined as integer constants exposed by process.binding('messages')
This does not use the template string approach due to that simply being too inefficient of a model. The new approach significantly outperforms the use of the template string model.

Please take a look and let me know if this is heading in the right direction.

jasnell · 2016-01-07T22:28:26Z

I would note that this is a work in progress! The list of messages is not complete. Fortunately, we do not have a large number of messages that we output.

Fishrock123 · 2016-01-08T00:28:57Z

Interesting, will think about it more tomorrow.

mcollina · 2016-01-08T08:52:26Z

I am generally 👍 regarding this, the API is simple enough and we can possibly change the implementation later.

cjihrig · 2016-01-08T15:21:07Z

lib/_debugger.js

-    return this.error('Breakpoint not found on line ' + line);
-  }
+  if (breakpoint === undefined)
+    return this.error(I18N(I18N.NODE_DEBUGGER_BREAKPOINT_NOT_FOUND,line));


Space after the comma.

Refactor to allow the `new` keyword and proper callsite reporting for the externalized error objects.

bnoordhuis · 2016-01-21T16:15:53Z

lib/internal/messages.js

+// I18N(key[, ...args])
+// Returns: string
+function message() {
+  const key = arguments[0];


If the function always has one argument it's better to declare it as function message(key) {, that avoids an ArgumentsAdaptorTrampoline stack frame.

jasnell · 2016-03-22T05:55:05Z

This is likely to go a different direction, currently pending @srl295's work to make ICU always available as a dependency. Closing this for now, will reopen when it's closer to a full solution.

jasnell added wip Issues and PRs that are still a work in progress. semver-major PRs that contain breaking changes and should be released in the next major version. lib / src Issues and PRs related to general changes in the lib or src directory. labels Dec 16, 2015

jasnell self-assigned this Dec 16, 2015

Fishrock123 reviewed Dec 16, 2015
View reviewed changes

srl295 added the i18n-api Issues and PRs related to the i18n implementation. label Dec 17, 2015

jasnell force-pushed the l10n-first branch from fd7c759 to 2895f2f Compare December 18, 2015 01:29

Trott force-pushed the master branch from 1e896a6 to 082cc8d Compare December 27, 2015 02:01

jasnell force-pushed the l10n-first branch from 2895f2f to 68792d5 Compare January 7, 2016 22:21

jasnell force-pushed the l10n-first branch from 68792d5 to 72b4d9d Compare January 8, 2016 04:21

cjihrig reviewed Jan 8, 2016
View reviewed changes

jasnell added 24 commits January 11, 2016 15:59

src: externalize repl.js messages

5f4861d

src: externalize string_decoder.js messages

67185ec

src: externalize sys.js messages

f0c5782

src: externalized timers.js messages

497329f

src: externalize tls.js messages

e142e8a

src: externalize url.js messages

678e8e8

src: externalize util.js messages

0a197c8

src: externalize zlib.js messages

cd4db97

src: externalize internal/child_process.js messages

61240ea

src: externalize internal/repl.js messages

259a36d

src: externalize internal/socket_list.js messages

96aa61f

src: externalize internal/v8_prof_polyfill.js messages

b885af3

src: externalize additional messages missed

924aa9d

src: avoid runtime calls

086bd73

src: code cleanup

46fcb2d

src: refactor I18N.Error, I18N.TypeError and I18N.RangeError

edcb53b

Refactor to allow the `new` keyword and proper callsite reporting for the externalized error objects.

src: internal/messages.js code cleanup

42b7c7f

src: begin externalizing src/* messages

21b1d95

src: externalize more native strings

5545855

src: support varargs in messages, externalize src/node_dtrace.cc

08866a3

src: refactor macros to make env explicit, externalize node_file.cc

44cc074

src: externalize additional src/*.cc files

a5d97de

src: externalize remaining errors in src/*.cc

9fc6ec1

src: externalize node.cc messages and print help, cleanups

92df8e6

jasnell force-pushed the l10n-first branch from 7cb9a01 to 92df8e6 Compare January 11, 2016 23:59

bnoordhuis reviewed Jan 21, 2016
View reviewed changes

jasnell closed this Mar 22, 2016

Fishrock123 mentioned this pull request May 5, 2016

[WIP] errors: add internal/errors module #6573

Closed

4 tasks

kunalspathak mentioned this pull request Mar 15, 2017

lib: Use regex to compare error message #11854

Closed

2 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

lib: externalized messages support #4311

lib: externalized messages support #4311

jasnell commented Dec 16, 2015

Fishrock123 Dec 16, 2015

jasnell Dec 16, 2015

Qard commented Dec 16, 2015

indutny commented Dec 16, 2015

Fishrock123 Dec 16, 2015

Fishrock123 commented Dec 16, 2015

jasnell commented Dec 16, 2015

domenic commented Dec 16, 2015

jasnell commented Dec 16, 2015

mcollina commented Dec 16, 2015

Fishrock123 commented Dec 17, 2015

jasnell commented Dec 17, 2015

Fishrock123 commented Dec 17, 2015

jasnell commented Dec 17, 2015

YurySolovyov commented Dec 17, 2015

mcollina commented Dec 17, 2015

jasnell commented Jan 7, 2016

jasnell commented Jan 7, 2016

Fishrock123 commented Jan 8, 2016

mcollina commented Jan 8, 2016

cjihrig Jan 8, 2016

bnoordhuis Jan 21, 2016

jasnell commented Mar 22, 2016

lib: externalized messages support #4311

lib: externalized messages support #4311

Conversation

jasnell commented Dec 16, 2015

Fishrock123 Dec 16, 2015

Choose a reason for hiding this comment

jasnell Dec 16, 2015

Choose a reason for hiding this comment

Qard commented Dec 16, 2015

indutny commented Dec 16, 2015

Fishrock123 Dec 16, 2015

Choose a reason for hiding this comment

Fishrock123 commented Dec 16, 2015

jasnell commented Dec 16, 2015

domenic commented Dec 16, 2015

jasnell commented Dec 16, 2015

mcollina commented Dec 16, 2015

Fishrock123 commented Dec 17, 2015

jasnell commented Dec 17, 2015

Fishrock123 commented Dec 17, 2015

jasnell commented Dec 17, 2015

YurySolovyov commented Dec 17, 2015

mcollina commented Dec 17, 2015

jasnell commented Jan 7, 2016

jasnell commented Jan 7, 2016

Fishrock123 commented Jan 8, 2016

mcollina commented Jan 8, 2016

cjihrig Jan 8, 2016

Choose a reason for hiding this comment

bnoordhuis Jan 21, 2016

Choose a reason for hiding this comment

jasnell commented Mar 22, 2016