Add Resolved Values and Function Handler sections to formatting

[eemeli]

This PR is effectively an extended comment on the ongoing discussion in #515 / #645 / #646. It also closes #678.

The intent with the change proposed here is to keep our formatting spec close to its current shape, but more clearly define what "resolved values" are, and via a non-normative example guide the reader towards a mental model of what a "formattable" or "selectable" value might look like.

I think this could be included already in LDML45 as its doesn't actually change anything, but that's of course presuming a lot of consensus.

[aphillips]

Thanks for taking at stab at this.

[catamorphism]

I know that this is non-normative text, but my conclusion after reading it is that there should be a design doc.

Some of it should be normative: specifically, the question of what functions take and return, which has to be answered (IMO) for the registry spec to make sense. How to describe that takes some thought, as I said in one of the comments. The task is to design a foreign function interface that's abstract enough to be adapted to a range of implementation languages, while concrete enough to make the spec something that can be implemented in a way that affords confidence that an implementation conforms to the spec. That's not easy.

Given the amount of discussion the question of "what is the domain and range of a function?" has generated, and how closely coupled that question is with "resolved values", I think it's worth a design doc. This could be #645 or someone could start afresh.

[macchiati]

I like the direction this is going, but I think we really never want to use the naked 'value' term. There are different kinds of resolved values, and it is important to keep them separate to prevent confusion. The value of {|3| :number minSignificantDigits=4} is **not** 3, nor is it 3.000. But high bit, this is a difficult issue, and if we rush it it is not unlikely that the spec for tech preview will be worsened. At this point, I think we would almost be better off having a NOTE explaining that we will be revamping this section to clarify the language in the following ways: ...

…

On Mon, Mar 18, 2024 at 2:11 PM Tim Chevalier ***@***.***> wrote: ***@***.**** commented on this pull request. I know that this is non-normative text, but my conclusion after reading it is that there should be a design doc. Some of it *should be* normative: specifically, the question of what functions take and return, which has to be answered (IMO) for the registry spec to make sense. How to describe that takes some thought, as I said in one of the comments. The task is to design a foreign function interface that's abstract enough to be adapted to a range of implementation languages, while concrete enough to make the spec something that can be implemented in a way that affords confidence that an implementation conforms to the spec. That's not easy. Given the amount of discussion the question of "what is the domain and range of a function?" has generated, and how closely coupled that question is with "resolved values", I think it's worth a design doc. This could be #645 <#645> or someone could start afresh. ------------------------------ In spec/formatting.md <#728 (comment)> : > > -> Alternatively, it could be an instance of an ICU4J `FormattedNumber`, -> or some other locally appropriate value. +> With this approach: +> - An _expression_ could be used as a _placeholder_ if +> calling the `formatToString()` or `formatToX()` method of its _resolved value_ +> did not throw an error. "Throw an error" is not terminology that's used elsewhere in the spec (as far as I can recall), and sounds implementation-language-specific. My intention is not to nitpick about the terminology, but to ask how to structure this exposition in a way that doesn't favor one specific implementation language. For example, in some cases, it might be more appropriate for the MessageValue interface to include isFormattable() and isSelectable() (and so on) as predicates. ------------------------------ In spec/formatting.md <#728 (comment)> : > +> did not throw an error. +> - An _expression_ could be used as a _selector_ _expression_ if +> calling the `selectKeys(keys)` method of its _resolved value_ +> did not throw an error. +> - Using a _variable reference_, the resolved value of an _expression_ +> could be used as an _operand_ or _option_ value if +> calling the `getValue()` method of its _resolved value_ did not throw an error. +> In this use case, the `resolvedOptions()` method could also +> provide a set of option values that could be taken into account by the called function. +> +> Extensions of the base `MessageValue` interface could be provided for different data types, +> such as numbers or strings, +> for which the `unknown` return types of `getValue()` and `resolvedOptions()` +> could be narrowed appropriately. +> An implementation could also allow `MessageValue` values to be passed in as input variables, +> or automatically wrap each variable as a `MessageValue` to provide a uniform interface I think this is missing an explanation of what the *value* (or resolved value?) of an input variable is. It's implementation-specific, *and* doesn't necessarily have to be the same as the "resolved value" that can appear in various parts of a message as explained above. ------------------------------ In spec/formatting.md <#728 (comment)> : > -_Expressions_ are used in _declarations_, _selectors_, and _patterns_. -_Markup_ is only used in _patterns_. +This specification allows for the same value to be used for: +- formatting in a _placeholder_, +- selection in a _selector_ _expression_, +- as the _operand_ of another _expression_, or +- as the _option_ value of another _expression_, I think this language is mixing phases. A "resolved value" is a runtime concept. Syntactically, what can be the operand of another expression is either a literal or a variable reference. At runtime (while formatting, when the values of input variables are known) some process maps a literal or variable reference onto a resolved value. I have a branch where I tried writing up an operational semantics <https://github.com/catamorphism/message-format-wg/blob/opsem/exploration/operational-semantics.md>, which would answer the question of "what are the relations that map terms onto values?" (A term being an instance of the data model.) This would be a way of providing a declarative spec, which forces one to answer the question of "what are the values that terms map onto?" I wrote this to try to validate my intuitions for #645 <#645> and wasn't intending to submit this as a PR or even share it, so it might not be readable by anyone except me. At any rate, a different way to organize this text might be to say something like: "Resolved values appear in - Literal resolution - Variable resolution - Function resolution - Option resolution" which would at least be describing a runtime process. Or even something like: "Resolved values appear in - Literal resolution (resolution maps a literal onto a value) - Variable resolution (resolution maps a variable, along with context, onto a value) - Function resolution (functions map values onto values) - Option resolution (resolution maps the right-hand side of an option onto a value)" (Using "right-hand side" instead of "value" in the last bullet point to avoid using "value" with two different meanings in the same sentence.) — Reply to this email directly, view it on GitHub <#728 (review)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACJLEMBC74VQZGIAC6GD35DYY5J7NAVCNFSM6AAAAABEVFFBS6VHI2DSMVQWIX3LMV43YUDVNRWFEZLROVSXG5CSMV3GSZLXHMYTSNBUGEZDEMRXGQ> . You are receiving this because your review was requested.Message ID: ***@***.***>

[aphillips]

In to 2024-03-25 call we agreed to work on this in the Tech Preview period.

[aphillips]

In the 2024-05-06 call, we asked @mihnita to review this before discussing further.

[mihnita]

Overall the direction seems OK.
But I am uneasy with specifying this interface.
Because:

• in "intertwines" even more the formatters and selectors (which I always maintained are different concepts, with different interfaces)
• it is just specified, without explaining how to use it, how it works.

So it feels over-specified.

[mihnita]

I think this update strongly conflicts with #645 ("[DESIGN] dataflow for composability")

That one proposes to throw away "resolved values" as too ambiguous ("In the current spec, a "resolved value" can be a nameable value, formattable value, or preformatted value, depending on context.")

This PR seems to make resolved values even more ambiguous, while in the same time restricting the ability of implementations to implement things in a different (cleaner) way.

[catamorphism]

I experimented with something like the suggested MessageValue interface in the ICU4C implementation, in a branch. Despite my previous comments, I think it's pretty good except for: resolvedOptions(): { [key: string]: unknown }. I think it should be made clearer that an option can be a value with options. I realize that in TypeScript, that's already implicit, but someone coming from a different implementation language reading this might assume that unknown is any implementation-specific type and not notice that it has to be something with options (in order to make examples like the one from #515 work).

So I suggest that instead of unknown, it should be: resolvedOptions(): { [key: string]: MessageValue }.

[catamorphism]

I tried changing ICU4C to use a value representation that's similar to this, and it works. The change in the PR shows what machinery needs to be there in order to implement examples like the one in #515. In an impl that uses the proposed MessageValue interface, it should be possible to write custom functions that use options that have options, and so on. Maybe we can get to a more normative version of this in the future, but I think this is a good step.

[stasm]

That's not right, I think. The result of formatting is a string or some kind of a "formatted part" type. When talking about resolved values, I think we want to identify the intermediate type that's the result of evaluating the placeholder, before it's formatted.

[aphillips]

I think the key thing here is the difference between the output of an expression and the evaluation of an expression. The "resolved value" is only important (or even visible) when it will be consumed by another expression later.

I think @stasm is correct that the output of formatting (or selection) is not the "resolved value". In the case of formatting, it is the string representation (or "formatted foo" parts representation), which is different from the "resolved value". Similarly, the output of selection is a filtered and stack ranked list of patterns (which is definitely not the "resolved value").

The "resolved value", to me, means "what is the value of the operand after the expression has been evaluated".

If I have a message You have {$x :integer} wildebeest, the value of $x is constrained to be an implementation-defined numeric type (it might not be an integer type, even though the formatting function formats it like one), but the output of the expression is a string.

I think that each function needs to explicitly say whether it "covers up" (changes) the operand's resolved value and what it changes it to be. Using my example:

Function: :integer

• Operand: an implementation-defined type or a literal whose contents match the number-literal production in the ABNF
• Format output: a string
• Selection output: number selector (see design number-selection)
• Resolved value: an implementation-defined numeric type

Notice that this eliminates the string representation between the operand (input) and resolved value. It also, probably, allows the implementation defined type to be changed by the function.

[macchiati]

Operand: an implementation-defined type or a literal whose contents match the number-literal production in the ABNF

There is a third kind of entity that can be an operand.

An operand is a defined type, or literal, or the "result of a previous functional application" (call that a function-result for now). A function-result is (logically) not just an implementation-defined datatype. If so, it wouldn't have access to the information that any composing function would need.

Instead, I think the best conceptual model of a function-result is that of an object with methods to get information. Such as (of course, the names can be chosen to protect the innocent):

• getSourceFunction() —the function used to create it
• getSourceOperand() — the operand used to create it
• getsourceOptions() — the options used to create it
• getFormattedString()
• getFormatToPartsStructure()
• getSelector()
• getResolvedValue()

Internally, it can lazy-evaluate. For example, it only needs to generate the formatted string if the method getFormattedString() is called. If the function-result is only used for selection, that never needs to be generated. If the function-result is passed to an :uppercase function, then the getResolvedValue() would never need to be called, etc.

In particular a function can look at its operand, and if it is a function-result, see what options it was passed and use them if the type of the function is known.

The description of a function needs to specify what its inputs are, and how its function-result behaves. Part of that involves specifying which functions it can compose with, and how it treats that function's operand and options.

[stasm]

I may have missed some discussions about this, but why wouldn't we want to be more specific here about the parts?

Suggested change

	> formatToX(): X // where X is an implementation-defined type
	> formatToParts(): IterableIterator<MessagePart>; // MessagePart is implementation-defined.

[eemeli]

As this is a non-normative example and we've not reached consensus on formatted parts, I'd rather not introduce them here implicitly and somewhat out of context, when the key aspect of the formatToX() inclusion is to remind the reader that a string is not the only possible formatting target.

[stasm]

(My review comes in late. I'm OK with its not being considered blocking this PR. I can file a new PR if you prefer to discuss my suggestions elsewhere.)

[eemeli]

@aphillips As discussed yesterday, I've updated this PR to incorporate the consensus we reached a few weeks ago:

A function MUST define its resolved value. The resolved value MAY be different from the value of the operand of the function. It MAY be an implementation specific type. It is not required to be the same type as the operand.

A function MUST define its resolved options. The resolved options MAY be different from the options of the function.

After some thought, I found the best place for this to be a new section defining function handlers as the real-world functions or methods that are called to get a resolved value. This now incorporates quite a bit of the language that was in step 4 of the Function Resolution process.

I opted for "handler" as we tend to use "implementation" to mean the whole of an MF2 formatter.

The exact MUSTard that's here is a little bit different from the above, as the requirements that a function "MUST define its resolved value" and "MUST define its resolved options" are not easily enforceable unless we were to define what "define" here means, and that's really rather tricky.

[aphillips]

Looks good. Doing this on a cell phone, so no suggestions... sorry.

[aphillips]

Should this be optional? * might match?

[eemeli]

If a selector does not support pattern selection, then the * variant is selected. A selector is never asked about *, that's handled by the selection algorithm directly.

The fact that you need to consider the combined effects of Resolve Preferences 2.ii.b, Filter Variants 2.i.b, and Sort Variants 5.iii.c to see that is what I tried to mean by arguing that "best choice" is too complicated.

[aphillips]

I went to the context and read the section carefully. I guess I have a few questions.

This text suggests that the fallback state is transitive--that implementations must track if a fallback has occurred. That probably needs to be stated more clearly, if so. Otherwise, the fallback value looks just like a string i.e. a literal, which is what lead to my comment.

Let's use my bad example:

.local $a = {0xbeef :num}       // this is Bad Operand with a fallback of `|0xbeef|`
.local $b = {$a :uppercase}     // this might work on `0xbeef`? It's probably Bad Operand if the first line isn't
.match $b                       // let's say :uppercase is a type of string selector too...
48879 {{Definitely not this}}
0xBEEF {{Matches?}}
* {{Could match if above doesn't}}

My point is, how does .match $b know that its resolved value is a fallback value?

[eemeli]

That's a good point. Some of the current text is using "fails to resolve" in a way that's not clear. I'll apply some updates to improve this.

[eemeli]

I have a set of further changes fixing this that's now in ac7cdc5, a commit not included in this PR, which improve the text around fallback values. As this PR now has some approval for its current state, I'll hold off on including those changes here, and propose them separately after this (hopefully!) lands.

[macchiati]

I'm confused. Can you give an example of a selector that supports selection where its resolved value doesn't?

That is, if I have
.input {$num :number}
.match $num
...
How could :number ever have a resolved value that didn't support selection? I think the notion of 'resolved value' doesn't help.

Suggested change

	with a _resolved value_ which does not support selection.
	that does not support selection.

[eemeli]

How could :number ever have a resolved value that didn't support selection?

Going with the example you posted above, if that message is formatted without a num argument, then the first step of Function Resolution applies:

If the expression includes an operand, resolve its value. If this fails, use a fallback value for the expression.

And then this part of Fallback Resolution applies:

Pattern selection is not supported for fallback values.

And so we end up emitting two errors when formatting:

1. Unresolved Variable
2. Bad Selector

[aphillips]

I think @macchiati's finger is on the right thing. It's not a technical problem. This is just hard to understand. Perhaps:

Suggested change

	with a _resolved value_ which does not support selection.
	A **_<dfn>Bad Selector</dfn>_** error is an error that occurs when a _message_
	includes a _selector_ whose _resolved value_
	contains a _function handler_ which does not support selection,
	is missing a _function handler_,
	or consists of a _fallback value_.

[eemeli]

Could we spin this suggestion off into a separate PR? As is, the only change that is being applied here is linkifying resolved value.

[aphillips]

Works for me as a solution.

[macchiati]

I couldn't find a definition of what a "function handler" is. Which file is that in?

[eemeli]

It's added by this PR to spec/formatting.md.

[macchiati]

I don't see a definition; just references.

[eemeli]

Here's a link to the section in this PR's branch: https://github.com/unicode-org/message-format-wg/blob/define-resolved-values/spec/formatting.md#function-handler

[eemeli]

Replying to @macchiati in #728 (comment):

I find this all somewhat muddled. Except in function composition, "resolved value" is not a useful abstraction.

That is, outside of function composition, the relevant features are

• a formatted value, and
• a selection value

A resolved value is also useful when considering the processing of a message like this:

.input {$num :number}
.match $num
1 {{You have a thing}}
* {{You have {$num} things}}

Because we're using $num for both selection and formatting, it's useful to be able to say that it has a resolved value that supports both selection and formatting.

[eemeli]

I've rephrased the resolved value definition a bit, and clarified that it applies to all message parts.

[aphillips]

(chair hat 🎩 on)

I just re-requested reviews from everyone who'd left reviews on the old PR so that the status is clear.

[aphillips]

Note my comment about the resolved value definition

[macchiati]

A resolved value is also useful when considering the processing of a message like this:

.input {$num :number}
.match $num
1 {{You have a thing}}

• {{You have {$num} things}}

Because we're using $num for both selection and formatting, it's useful to be able to say that it has a resolved value that supports both selection and formatting.

I really find this extremely confusing.

It appears to me that the core concept is the following:

• Each variable (such as $num) references an entity that that performs selection and formatting.
• And that you are calling that entity a "resolved value".

Is that what you mean, and I'm just confused by the terminology that you are using?

[eemeli]

It appears to me that the core concept is the following:

• Each variable (such as $num) references an entity that that performs selection and formatting.
• And that you are calling that entity a "resolved value".

Is that what you mean, and I'm just confused by the terminology that you are using?

Yes, that's pretty much it. With the very slight caveat that we do not require that the resolved value entity itself performs selection and formatting, as it should be possible for it to be a pure-data thing with no methods but enough information such that a separate, external processor can do the appropriate selection and formatting.

[aphillips]

Yes, that's pretty much it. With the very slight caveat that we do not require that the resolved value entity itself performs selection and formatting, as it should be possible for it to be a pure-data thing with no methods but enough information such that a separate, external processor can do the appropriate selection and formatting.

To use a concrete example from ICU4J, think of the resolved value of a :number as being a FormattedNumber object. It contains the number itself, as well as all of the resolved options for formatting the number. You can call toString() on it to get the formatted representation. If that class implemented MF2 selection, it might add a matchSelectorKeys method that you could call for selection.

[macchiati]

BTW, since at least part of the motivation for 'resolved value' is in Pattern Selection, I took a look at that section. I think the spec could be much cleaner (with the same results) with a revision: I proposed an outline of that in #898