Globalization of Node Error Messages

[jasnell]

Opening the issue for tracking: Need to explore globalization and general improvement of node error messages (log and user).

[misterdjules]

I would suggest separating actual globalization of error messages from other improvements (like wrapping lower level error messages within higher level ones, etc.).

[srl295]

Today's discussion: https://gist.github.com/srl295/4b11872ca4d6e26b69c1

[jasnell]

Following up on today's discussion: https://gist.github.com/jasnell/1ee1e23a30875683dca2 (slightly revised version)

/cc @tjfontaine @bmeck @srl295

[caridy]

ok, I'm trying to get my head around this proposal, but it seems that it is mixing too many concepts.

We have to have a clear differentiation between contextualization and provision. I think contextualizing by locales only is very limiting, it should be extensible enough to encompass region, device type, bucket information, etc, things that belong to the user land.

In the other hand, provisioning is in the loader realm (getting data from files available in the runtime), and this is the most confusing part for me here. How will this map to the client runtime (browser)? How will this work with ES6 Loader? I'm worry that we will be introducing a complete new way of provisioning, sort of a black magic, instead of just providing an imperative api that allow users to require/provision data and other resources.

Can someone provide the ultimate goal here? It is not very clear from the notes and proposal.

[misterdjules]

@srl295 @jasnell How does globalization of error messages relate to https://gist.github.com/srl295/4b11872ca4d6e26b69c1 and https://gist.github.com/jasnell/1ee1e23a30875683dca2?

Is the idea to implement globalization of error messages with resources and bundles? If so, I would suggest to first discuss these items in separate issues (as they would be used more generally, not only to globalize error messages), and to mention that globalizing error messages depends on them.

Otherwise I'm concerned the relationship between these concepts may be confusing.

[jasnell]

@misterdjules yes, there are really three separate items here that need to be split out.

1. Externalization of Node's own error messages in both the native and JS code
2. Locale-aware resource loading within a module
3. Locale-aware resource bundles within a module

If implemented properly, these will overlap somewhat in that the js api used by node's own core library ought to be identical to that used by module developers -- although the internals of each will differ.

For node's own messages, before I can go much further, a decision ought to be made: are we going to make ICU mandatory or not. The reason is because ICU exposes a very efficient message bundle API already that can be directly leveraged by node. At compile time we would generate an ICU dat file containing all of nodes own messages and would use that within the node src and lib. If ICU is going to be optional, we would need to introduce an abstraction api or use a different mechanism such as gettext. I can go either direction but I'd need to know which the core team would prefer.

[jasnell]

@caridy keep in mind that this is still a very early proposal. I don't yet have answers for how this maps to the browser, how it works with the ES6 loader, etc. We will be working through those bits, however. I will be making sure those questions are answered and I'm working with one of our (ibm's) rep's to emca-402 ( @srl295 ) to ensure that whatever direction we go here will work with ES6. Suggestions are definitely welcome if you have any.

Specifically, this proposal deals with locale-aware resource loading only. That is, any module might have a collection of resources (files such as templates, message bundles, images) that are locale specific (a favicon intended for en-US users may not be appropriate for fr-CA users, etc). I know there are many other ways that can be divided up (as you suggest) but to keep things simple, this proposal focuses specifically on locale / bcp47.

Right now, module developers are required to use fs to access these resources. If the developer wishes to incorporate locale awareness, locale-fallback, caching, etc they either implement it themselves or rely on npm modules. The intent of the Resource proposal is simply to provide that same mechanism within the core lib. Resource.load would handle the default locale, locale fallback, caching, etc. In a way, Resource would somewhat straddle the line between module and fs.

The Bundle part of the proposal simply layers a message bundle API over the Resource.load (A message bundle is Just Another Resource).

Right now, the point of putting this proposal up is to solicit feedback early in the design process. Is this heading the right direction? If it's not, what are we missing? What would be a better approach? Is there something fundamental that I'm overlooking? Or, perhaps more importantly: is there a reason why this absolutely should not be in the node core lib?

[jasnell]

@misterdjules ... let's keep this issue open and focused on the localization of nodes own error messages and I will create a separate issue for the Resource/Bundle API and make sure the two link back to one another

[jzaefferer]

Looking at #9258 and this ticket, along with the proposals linked above, and based on the experience @rxaviers and me gathered working on Globalize on top of CLDR, my recommendation is to keep it very simple and private: Implement a private module to translate node's error messages. Don't try to expose anything as public APIs. Once Ecma-402 supports ICU message format, use that for JS modules. To make that migration easier, use ICU message format in that private module - definitely don't invent your own syntax (like "hello %s").

[jasnell]

//cc @joyent/node-coreteam ... Core team: when you get a moment I'd appreciate a look here: a6e66cc ... it's not yet fully baked and does not build on Windows or Ubuntu yet but it demonstrates the basic direction on externalizing nodes own messages. The tl/dr version is:

(1) If --with-intl is enabled, a new static dependency lib is compiled that contains the externalized resources. These are simple printf style template strings stored within an ICU dat file compiled statically into a lib linked into node itself. Right now, only en, en_US, and ROOT locales are supported.
(2) If --with-intl is disabled, the L10N macro falls back to simple inline printf statements and the static dependency is not built.
(3) Within the C/C++ code, first do an #import "node_l10n.h", then use one of the variants of the L10N macro.. see example:

#include "node_l10n.h"
//...
// use default buffer size (currently 200)
printf(L10N("KEY", "This is the fallback"));
// use buffer size of 30
printf(L10Nn("KEY", "This is the fallback", 30));
// use printf formatting with default buffer size
printf(L10Nf("KEY", "This is the %s", "fallback"));
// use printf formatting with specified buffer size
printf(L10Nfn("KEY", "This is the %s", 30, "fallback"));

(All of Node's own messages are quite simple so printf templates here are more than sufficient. Things get a bit too complicated very quickly if we start trying to do full ICU messageformat support. The idea is to keep it simple)

(4) From Within the JS Core Library code, do a var bundle = require("_bundle");. Then you can do stuff like, console.log(bundle('KEY', 'This is the fallback')). This uses the L10N macros under the covers but uses util.format to do the printf expansion for simplicity. The _bundle.js is intended to be used only from within node's own source. It is NOT a generalized resource bundle mechanism for node apps.

(5) Next steps are to make sure this is building correctly on all our supported platforms which I'll be working on today and tomorrow. In the meantime, I'd appreciate any comments / feedback.