In the third data model proposal, I tried to provide extension points on the runtime via the following RuntimeValue interface (thanks to @mihnita for suggesting adding match):

export abstract class RuntimeValue<T> {
	public value: T;
	abstract formatToString(ctx: FormattingContext): string;
	abstract formatToParts(ctx: FormattingContext): IterableIterator<FormattedPart | OpaquePart>;
	abstract match(ctx: FormattingContext, key: VariantKey): boolean;
}

This way users can define custom runtime values, with custom formatting and selection logic. At the same time the underlying data model doesn't change at all.

Handling errors and fallbacks

I would actually argue that from a tooling point of view, a model with types+functions can provide at least an equivalent if not better experience that one with functions only. To start, let's consider a hypothetical element, which could be implemented in one of two ways (simplifying literals to strings):

• { type: 'element', name: 'a', tag: 'start', attr: { href: 'http://example.com/' } }
• { type: 'function', func: 'element', args: ['a', 'start'], attr: { href: 'http://example.com/' } }

If the tooling does not know what to do with this element, it'll need to in both cases either fail or use some sort of a fallback value when representing this part of the message. Next, consider also a missing function, for example:

• { type: 'function', func: 'list', args: ['foo', 'bar', 'baz'], attr: { type: 'disjunction' } }

This would be rendered the same in either sort of system, but with one key difference: With a types+functions model we can know that this would be performing some operation on just the args and attr values. With a functions-only model, we can't know that this isn't doing something like term and looking up and entire message that ought to be included here, or indicating the start of some formatting element.

I would argue that need to use a fallback should be considered a different level of failure in the two above examples, and that a functions-only model can't differentiate between them. And that in neither case does the data model actually change when it starts supporting an element, it's just packaged differently.

Formatting function signatures

Another way of putting this is that we're talking about the signature of a function call, which I'm arguing should be as simple and limited as possible, whereas you'd like it to have greater access:

stas/third:

ts_eemeli (slightly updated here, but still pretty much the same):

The key difference here is that with types+functions we don't need to give a formatting function unrestricted access to the runtime context, which functions-only effectively requires in order to make it possible for e.g. term to access the current message resource. This makes reasoning about the function's behaviour much easier, and provides a significantly smaller attack surface for exploiting any vulnerabilities in custom formatter code.

Accessing separate parts of the context

With my current implementation, each of its types is distinguishable based on the specific part of the context that it accesses:

• literal requires no context
• variable gets the runtime scope, i.e. the parameters that the formatting function was called with
• function uses the function registry
• term builds a getter function for other messages

Eventually, element would probably get access to a yet-separate set of source components for it to overlay its contents, which need is in fact a decent argument for its separate implementation from the generic function.

I really really hope that we aren't proposing any interpretation nor direct handling of HTML, XML, JSON, CSS, SSML or something else like that. I'm really hoping to focus on what has been proposed as the data model in this group. From my experience, such mappings can be fragile to maintain. I'd rather focus on passing that formatting/decoration information through unmodified, because I consider that a solved problem. I'm hoping that we can focus more on getting words into grammatical agreement.

Though I will admit that our implementation formats 2 representations at once. One is for the printed form, and the other is the spoken form. We're basically creating 2 strings in 1 shot. For example, I may want to print "1 mujer" in Spanish, but pronounce it as "una mujer". Both spans are the same by default.

@grhoten Absolutely agreed on not defining specific interfaces for HTML, its DOM, SSML, React, or anything else. However, I do think that there's value in considering the needs that such interfaces present, and how they might be best accommodated in the interfaces that we do define, so that a real-world implementation wouldn't be forced to e.g. output a message as stringified HTML or SSML and then immediately re-parse that with a different parser in order to actually use it.

This is particularly relevant in a live context such as React, which may use a complex non-stringifiable object as a value that's to be included in the middle of its output. Similarly, Fluent does really rather interesting things with its DOM overlays, which allow for an existing HTML DOM element to be used as a template or enriched by data coming from a message.

Hence e.g. the proposed element type, and indeed the desire that I share with you that we don't get stuck with the discussions on its details continuing to block progress on the core data model. If we could agree that the data model will support at least some set X of possible pattern element types, we could then agree separately on what that data model should look like, and what each of those pattern element types look like. That's what I mean by splitting up the spec.

I think there might be some misunderstanding here. @stasm's RuntimeValue is a wrapper around a value that has a generic type T. It provides three different methods with concrete types for when that value needs to be formatted to a string, formatted to parts, or compared against a select key.

Why should this kind of encapsulation automatically be rejected in code review? For example, an implementation of this interface around a number could include its formatting options, and be able to use those both when formatting as well as when determining a match against a CLDR plural category.

1. What they look like in "the registry"

[...] Think of it as a header file in C / C++

Yes, this will need to be decided. I would think that it'd be easiest to express this in the spec as the shape of the arguments with which each formatting function will be called, along with the type of its return value.

You've argued previously that the data model should only include the literal and function pattern elements, and that all of the other functions could be implemented via function. Is this still so?

The examples DATE and INTERVAL you give are relatively simple, as they only operate on their direct input values. Still, they'll need access to at least the current locale when they're called. How about message references? What would such a function need to have access to in order to work?

2. What does the model look like when parsed

This might be different than the model ready to use when rendering.
It it the model used for lint, refactoring, IDE plugins, export / import to XLIFF, tooling in general.

I don't think I understand this. Why or how would the model be "parsed"? I at least have understood the data model to be a description of a structure that is alrady parsed. When considering various use cases, it's of course likely that not all of them will make use of all parts of the data model, but do we really need to be explicit about which parts are and are not used for various tasks?

3. What the model looks like at runtime, and how it the rendering done

At this point we need real implementations for the various functions, and we also need to "bring in" somehow the various inputs.

msg = MessageFormat2.parse(...) // parsing some serialized form
msg.format(parameters)

What is the type of parameters?

I think it should be a Record<string, unknown>.

What are the types of the various runtime values that we want to format?

I'm pretty sure that we should not generally limit the types of runtime values.

And how do we support more than one placeholder-function (see the examples above).

Is this a question about how pattern elements in a message may refer to more than one runtime value, or something else? I'm not sure that I understand the concern here.

TLDR: looking at the examples above, and following the full "lifecycle", I think it is pretty clear that:

A. The data-model should stop at "parse level" (opinion)

As in, the data model should only contain stringifiable data?

B. We need a description of the rendering (runtime?) behavior

Yes, for both string and parts outputs.

C. The model can't specify they types of the inputs for the functions.

I agree that we should not limit what types of values might be passed in as formatting function arguments at runtime and passed to formatting functions that use them. However, we do need to clarify at least how the formatting function is told what the current locale is, and we need to have a solid story for how message references can work.

Furthermore, we can (and probably should) wrap all formatting function arguments in something like RuntimeValue or Formattable. Consider for instance your DATE and INTERVAL, and how they might handle format-to-parts output. If their input values are wrapped up as implementation-specific FormattableDate objects which encapsulate the code for formatting a date to a list of parts, implementing those functions becomes significantly simpler and duplicates far less code.

D. Would be "unclean" for the model to force on all platforms that "functions" must be class-like at rendering time

I should be free to use function pointers / references, lambdas, something else, whatever. It's an implementation detail.

Agreed. But we should still be able to use object-oriented concepts in the spec when it makes sense to do so. Because we're really talking about implementation internals, and those can of course be implemented in a number of different ways.

E. A simple [] would not be enough for the arguments, we need something with names

Why? I don't see how this follows from your previous arguments.

The examples DATE and INTERVAL you give are relatively simple, as they only operate on their direct input values. Still, they'll need access to at least the current locale when they're called. How about message references? What would such a function need to have access to in order to work?

I would expect that all functions will need to have some kind of "context".
But that can't really be part of the spec.

Ah, but there I think we disagree, on both counts. First, I assert that the only part of the message/resource/formatting-invocation specific context that formatting functions should have access to is the current locale. Yes, a formatting function could make use of static tools such as the JS Intl.ListFormat, but it should not be able to behave differently depending on the context in which it's called, except so far as that context defines the explicit arguments and options passed to the function.

Second, I further assert that we should include in the spec an explicit method for pattern element formatters to be able to define the context that they require to operate. See here for more on this: #190 (reply in thread).

I don't think I understand this. Why or how would the model be "parsed"? I at least have understood the data model to be a description of a structure that is alrady parsed.

You are right, that is not well described.

I'll try to say it differently: there would be one model the is the output of the parser.
And one that can be used for "rendering" (to string, or to parts).

Should we stop to only specify the result of the parse model?

I think I'm still lost on this. Does this "rendering model" merge in some additional data from some other source, or is it a definition of some operations on the "parse model"?

I think it should be a Record<string, unknown>.

Is that the same thing as a Map?

Pretty much, yes. Here's the TS definition: https://www.typescriptlang.org/docs/handbook/utility-types.html#recordkeystype

Am I only confused by the model (as proposed by Stas) defining all kind of types (string, decimal, boolean, etc).
Are they just for the options?

If you're referring to the Parameter type in Staś's data model, then yes, that is purely a description of the contents of the stringifiable data model. During formatting, runtime values are wrapped in a RuntimeValue, which may encapsulate a value of any type.

And are we still talking the Stas model, or are you proposing a forth model?

I'm not sure that it makes sense to really see this as a conversation about a specific model, but about the general structure of the eventual MF2 spec. My preferred model continues to be the one available and implemented here: https://github.com/messageformat/messageformat/tree/mf2/packages/messageformat

That is based on the EZ model, but has been updated to incorporate e.g. the recent consensus of only allowing literals and variables as function arguments.

Because the RuntimeValue there seems quite confusing to me.

It is used both as return of a formatting function:

interface RegistryFunc<T> {
	(ctx: FormattingContext, args: Array<Argument>, opts: Record<string, Parameter>): RuntimeValue<T>
}

How can I represent "format to parts" with the few basic types defined?

In case you're not familiar with the syntax, the <T> in the interface name indicates that it's a generic, and that this type T is used to further define the return value RuntimeValue<T>. That, in turn, is a generic abstract class that wraps a value: T, while requiring its implementations to provide methods such as formatToParts(ctx). If you look then at e.g. NumberValue, one of its concrete implementations, you'll see that it includes a member opts: Intl.NumberFormatOptions, which is used by the formatting methods.

So I can't have a ResourceManager in context, because it is not a RuntimeValue.

Would a ResourceManager be something that you'd like to format? If not, then it probably doesn't make sense to wrap it inside a RuntimeValue.

Or RuntimeValue only means some "unknown type packed with type info"?
So I can write my own DateValue or PersonValue that extend RuntimeValue?

Yes, you can write your own implementations of RuntimeValue. That's actually exactly what you'd be expected to do in order to provide support for a new data type. In Staś's implementation, the base class is abstract, so a custom implementation is always needed. In my implementation, the corresponding class is Formattable, and the base class already provides an implementation that's sufficient for most generic data types: https://github.com/messageformat/messageformat/blob/mf2/packages/messageformat/src/formattable/formattable.ts

If that is all there is, do we even need it, if the programming language I'm using has RTTI?

It's needed to allow for formatting options to be included with the value.

And how do we support more than one placeholder-function (see the examples above).

Is this a question about how pattern elements in a message may refer to more than one runtime value, or something else?

Yes.

I'm not sure that I understand the concern here.

How do we get to them "at render time?"

Because I don't see explained anywhere how things are rendered.
Are there 2 steps, or not?

parse => gives me a model
model.format( arguments ) => gives me a String / parts (as in "format to parts", TBD the name / type)

Basically I don't think it is clear how functions and "variables" (or whatever the name is now) come together, and when.

I think I'm starting to understand a possible source of your confusion here. Each of the proposed models handles this rather differently, and stasm/third is lazier than the others, forcing each formatting function to internally resolve the values of its arguments and options.

Also, it might be easier if we could agree not to refer to a message or resource object as a "model", given that it has methods, and the data model only contains pure data. We should not overload the term with multiple meanings.

B. We need a description of the rendering (runtime?) behavior

Yes, for both string and parts outputs.

Sure, we agree on this.
But what I mean is: in order to be able to discuss what the functions look like we need the runtime part described.
We can't go ahead to "let's discuss functions, do you agree this is good" without explaining how they work.

Good. I just want to ensure that we continue to keep both the format-to-string and format-to-parts outputs in mind, as the need to provide more than one form of output has significant effects on the design of the runtime.

Furthermore, we can (and probably should) wrap all formatting function arguments in something like RuntimeValue or Formattable.

I think that only makes things "clunky"
If the programming language has RTTI there is no need to wrap things.
Instead of doing something like
if (runtimeVal.getType == "date") { ... }
one would do if (runtimeVal instanceof Date) { ... }

As mentioned previously, a wrapper is needed in order to include formatting options with the value. Without such a wrapper, partially formatted values can't be passed in as formatting parameters and building generic formatting functions becomes rather burdensome.

Worse, we might need a RuntimeDateValue. RuntimeCalendarValue, RuntimeInstantValue

We should not burden the spec (and the devs) with extra abstractions , unless they are really needed.

I agree that the specification of e.g. date-based runtime values should be beyond the scope of the spec. I do think that defining a generic RuntimeValue or Formattable interface is within the scope of the spec.

E. A simple [] would not be enough for the arguments, we need something with names

Why? I don't see how this follows from your previous arguments.

If the arguments come as an array (as proposed) there it no way to extend things.

All the functions that take 2 RuntimeValues are "merged" into one.
When this happens in classic APIs the way people go about it is they add yet another parameter.
And some of them are null, and we ignore them. And it's all a mess.

For example, if we have a formatRange( int startIndex, int endIndex ) in this data model they will be an array with 2 IntegerLiterals.
Now I want formatRange( int startIndex, int rangeLength ). Can't do it.

That's why people like named parameters.

I challenge this assertion. While in general I do agree that there are situations in which named parameters are good, there are plenty of situations where a list of arguments works really well.

To use your chosen example, as far as I know every real-world function in every programming language for formatting a range of numbers or dates uses [start, end] arguments. Or can you provide a counterexample?

I think there might be some misunderstanding here. @stasm's RuntimeValue is a wrapper around a value that has a generic type T.
It provides three different methods with concrete types for when that value needs to be formatted to a string, formatted to parts,
or compared against a select key.

So who own the methods? The RuntimeValue?
Art the 3 methods members of the RuntimeValue?
That is not what the current code (from Stas) seems to show.

In Staś's code they are methods of RuntimeValue. In my code they are methods of Formattable. If this is not clear, could you point to specific parts of either implementation that are confusing to you?

But the core question here is:
Are the formatting functions members of the object to be formatted (OO)? foo.toString(options)
Or we are functional? fooToString(foo, options)

The code seems to indicate functional.
You answer seems to indicate OO.

Please be specific about which parts of the code (either Staś's or mine) implies that either his RuntimeValue or my Formattable does not provide methods for formatting its contents.

Sorry, looking through it again, and it feels very argumentative.
That is not my intention.

It's just that I don't see how things come together.
"Look at the code" is not a good answer, we can't discuss these issues in the whole group by asking everybody to read the code.

The main reason is not "people can't read code" (of course they can)
But by looking at the code it is impossible to tell what is intentional, what is implementation detail, what is workaround for programming language limitations, what is a shortcut for cleaner code.

For example in the Stas's code there is stasm\third\impl\context.ts

And at the end the toRuntimeValue has a switch on node.type with cases for "StringLiteral" and so on.

What happens if I add "DateRuntimeValue"? TypeError
Do I have to modify the core implementation to add a new type?

If you have a Date as a runtime value, that will be a VariableReference in the data model. Hence when resolving a Parameter in toRuntimeValue(), you may observe that the value this.vars[node.name] is returned. Then, we may look at the type of this.vars (Record<string, RuntimeValue<unknown>>) and from this deduce that when this function is called, the runtime date value will be returned as a RuntimeValue<unknown>, i.e. just providing the default methods. If a custom formatting function needs to do something special with the wrapped date's value or formatting options, it could use an instanceof check to verify that it has a DateRuntimeValue.

Something very much like this is demonstrated here: https://github.com/unicode-org/message-format-wg/blob/experiments/experiments/stasm/third/example/example_opaque.ts

I think that proof of concept implementation should simulate the "real world": I get ICUNext, or FluentNext, or ECMAScript.Intl.MessageFormat, and WITHOUT changing the code I add my own types to format, or to select on.

This is already now possible with Staś's and my implementations.

The specific and entire extent of what I'm looking to bring to the group for discussion tomorrow is contained in the "Formatting function signatures" part of this comment above: #190 (reply in thread)

Here's the text of it, in case it's difficult to find:

Another way of putting this is that we're talking about the signature of a function call, which I'm arguing should be as simple and limited as possible, whereas you'd like it to have greater access: [snip]

The key difference here is that with types+functions we don't need to give a formatting function unrestricted access to the runtime context, which functions-only effectively requires in order to make it possible for e.g. term to access the current message resource. This makes reasoning about the function's behaviour much easier, and provides a significantly smaller attack surface for exploiting any vulnerabilities in custom formatter code.

I have prepared a presentation clarifying the above, covering the current state of the art, how the different proposals currently address this, and why it's relevant to our design work. In general, the preceding discussion in this thread has not touched on this specific topic at all, or at most only in passing.

I'm sorry, I don't really know what to say here, so I'll try repeating what I've said before.

As I mention in my preceding comment, the relevant part in this thread to what I'd like us to discuss in the meeting is contained in the "Formatting function signatures" section of this comment, and I even went and block-quoted its relevant text.

The part that you quoted and ask about is not in that section. It is of course relevant to the wider discussion here, but not specifically to formatting function signatures. I am not intending to cover it in the meeting.

If you find that it's unclear what I mean by "Formatting function signatures" or why I think they're rather important, I would invite you to look at my presentation on this ahead of the meeting; it's linked from the meeting's agenda.

For me the part that is unclear is the variable one.
Can't really display a variable without a function.
If the variable is a Date, then it needs to be handled by the DATE function.

While it's true that a Date needs specific handling when formatting, this is not in fact provided by a custom formatting function in either the EZ proposal or Staś's third. Instead, both of these approaches provide the actual formatting via a wrapper around the Date. Staś doesn't have this implemented yet, but in EZ it's FormattableDateTime.

With this approach, the role of a DATE formatting function is to return a FormattableDateTime that includes any corresponding options.

At some point "something" should take the count = 42 or date(2021-12-31) and "merge" it with the data model
(AST, whatever Fluent calls this) in order to render things (to a DOM, or whatever).
The fact that is not called .format or that the user does not see that because it is done by the framework, deep-down, is not relevant.

So, what's it "the thing" that brings the_answer=42 and "{count} is the answer" together?
Is that happening in one step (at parse?), or is a separate call?

If I understand right, pattern element formatters are indeed here proposed as the "things" that connect e.g. variable references with runtime variable values.

So if we presume that we start with a message being first resolved into a list of pattern elements:

[
  { type: 'variable', var_path: ['count'] },
  { type: 'literal', value: ' is the answer' }
]

then it is the pattern element formatters that turn this into a list of Formattable objects by combining each with any appropriate context values:

[
  new FormattableNumber(42),
  new Formattable(' is the answer')
]

Leaving the concatenation of these formattables' toString() or toParts() values to be task of the actual format() or formatToParts() call.

If I understand right, pattern element formatters are indeed here proposed as the "things"

I don't know. It is your proposal, and I am trying to understand it.
So if you don't understand it right, who does?

"If I understand right" was a reference to understanding what your were trying to say. I could not be certain that I had understood right what you meant by "the thing" in your previous message.

the role of a DATE formatting function is to return a FormattableDateTime that includes any corresponding options.

I'll try to map things to a different set of concepts, to see if I get it:

Would it be fair to say that "FormattableDateTime" is a formatter in ICU meaning?
That works like this:FormattableDateTime df = new FormattableDateTime(options, locale, ... more?)
But does not encapsulate the actual value of a date, that comes later on.
So it can be used (df.format(date) or equivalent) again and again, for different dates?
(for example if I write a logger message, like "Log entry: {date} {time} {component}::{class}: {tag} ::: {message}")

And would that make DATE a factory? That takes options and other "stuff" and builds a "formatter" (or Formattable*)
(not arguing about names right now, just trying to understand)

No. It would perhaps be easiest if you could allow yourself to become "tainted" (as you put it on Slack) by studying the actual implementations of Fluent, the EZ model and/or Staś's proposal. Each of them has an equivalent of this type of wrapper (Fluent: FluentValue, EZ: Formattable, third: RuntimeValue) that wraps a value and its formatting options.

For example, the actual FormattableDateTime constructor takes the arguments value, locales and options. It explicitly includes the actual value being formatted, so that its toParts() and toString() methods do not set or change the initially set value.

So there is another "secret function" that takes something like

[
  { type: 'variable', var_path: ['count'] },
  { type: 'literal', value: ' is the answer' }
]

and takes the current value of count (where from) and produces

[
  new FormattableNumber(42),
  new Formattable(' is the answer')
]

Is that the case?

There's no need to use an appellation like "secret function". You are describing a pattern element formatter. This is in fact explicitly stated in the paragraph between those to code blocks which you elided:

then it is the pattern element formatters that turn this into a list of Formattable objects by combining each with any appropriate context values

If you would like to find out more about how a pattern element formatter might work, you are invited to study the implementations available here: https://github.com/messageformat/messageformat/tree/mf2/packages/messageformat/src/pattern

I would be very happy to answer any questions you might have about the code, in case it's hard to understand.

you could allow yourself to become "tainted" (as you put it on Slack) by studying the actual implementations

There are reasons I don't want to do that.

• In code it is not possible to say what is accidental, implementation detail, shortcut, tech stack limitations, vs intentional, relevant to the data model.
• The code does not tell you "why", only "how", does not tell you the big picture.
• The code does not answer questions like "how is X handled" if there is not already some unit test for it.
• Specs are standalone. Even when there is a reference implementation, the spec does not point to the implementation, but the other way around
• Because I already looked at it, and it is unclear (that's why I'm asking about INTENT)

Based on the answers here (and what I got from the code) the intention to have that construct (variable without function) is convenience.
So that as a developer I can say "You have a meeting in {count} days, at {time}" without being forced to say "...{NUMBER(count)}..."

ICU has the same thing, I can say "...{dueDate}..." or "...{dueDate, DATE}..." or "...{dueDate, DATE, ::yMMMd}..."

Is that the case?

I think that if the goal is not just "let's put what Fluent does in a standard", but make things better where possible.

So when we design something we should also look at alternate ways, to see if there might be something better.
And be open if someone else suggest something. Might be better, or not, TBD.

What am thinking goes back to some of the core "good practices" that I try to follow (and I acknowledge this was not voted / agreed)

For me:

All functions and types should be equal, and behave the same

There should be no "second class citizens".
A function I write should be able to replace a "stock" one.
A type I add should be able to be default for a type, replacing the stock one.

I should not have to say "...{dueDate}..." works, because Date is a well known type, but "...{price}..." does not work, because the type of price is Amount, and that is your custom data type.

All functions should behave as they are custom

In that the the core implementation should not know about any types, and the "well known types" (and formatters) should be in a different folder. The core should never do if (foo.type == "date")

The only difference between the currently recognized Fluent / ICU types and formatters (long, date, etc) should be that:

• most popular implementations of the core spec will also implement these functions out of the box
• they are in the "standard registry" already (both the non-runtime registry (schema?) and the runtime one). So I don't need a company registry / schema, and don't need to add register them at runtime (in the runtime reg)

But basically if you use a library implementing the spec in 5 years you should not be able to feel any difference between the way numbers and dates (and plurals) work and lists, or person, or amount (types added 2 years later).

After this big detour, this is how I would handle this...

Values are nothing special, they are still formatted with functions.
The "save some typing" is relevant for syntax, not for data model.

It is like defining a function with default parameters (def greet(name, msg="Good morning!"): ...)

When I say "...{dueDate}..." it is still a function there, and there is still a default style (for example LONG)

But I would "register" those function in the runtime registry, associated with the type (as key).
(and I can also replace "standard" ones)

The core implementation never does if (type == "date") ..., it does runtimeRegisty.getFormatterForType(type) ...

So there is no difference between variables and formatting functions, they are all functions.
But there is no need to bother to tell you the function if you already know how to get it from the type.

There might be a difference between solving the type at rendering time vs at parse time (which is why making the distinction is important)

And that might result in treating them differently in the data model.
But should be a deliberate decision, with documented pros and cons, what a design document does.

We should not drag in "the Fluent way" or "the ICU way" without any discussion.

It is very likely that the people who designed Fluent DID all this (I would be worry if they didn't)
And there was a deliberate decision to do things a certain way.

But those documents:

• Are not accessible to outsiders
• What was good in for browser / JS / custom localization tooling / company standard might be different for a server / C++ / standard l10n industry / universal standard
• Some decisions might be mistaken. People design stuff and then learn better when the rubber hits the road

So we can't just take some code that emulates what Fluent does and "enshrine" it as a Unicode standard.

you could allow yourself to become "tainted" (as you put it on Slack) by studying the actual implementations

There are reasons I don't want to do that.

Alas. I can only speak from personal experience, in that I have myself learned much by studying how others have solved similar problems to ones that I've faced.

Based on the answers here (and what I got from the code) the intention to have that construct (variable without function) is convenience.
So that as a developer I can say "You have a meeting in {count} days, at {time}" without being forced to say "...{NUMBER(count)}..."

ICU has the same thing, I can say "...{dueDate}..." or "...{dueDate, DATE}..." or "...{dueDate, DATE, ::yMMMd}..."

Is that the case?

Convenience is certainly one reason, but I would posit that there is value in separating different operations from each other, in this case:

1. Determine the value of a variable.
2. Get the result of calling a formatting function.

A key facet of this proposal, though, is to realise that we may use a higher level of abstraction when dealing with various pattern elements. It isn't specifically about how we deal with a variable or a function, but it's about the fact that each of those (along with e.g. literal values) are pattern elements, and when formatting a message we may use the abstraction of a "pattern element formatter" to answer the question of how we deal with each one.

I think that if the goal is not just "let's put what Fluent does in a standard", but make things better where possible.

So when we design something we should also look at alternate ways, to see if there might be something better.
And be open if someone else suggest something. Might be better, or not, TBD.

Oh, absolutely agreed. For instance, the abstraction of pattern element formatters proposed here has no Fluent equivalent whatsoever.

What am thinking goes back to some of the core "good practices" that I try to follow (and I acknowledge this was not voted / agreed)

For me:

All functions and types should be equal, and behave the same

There should be no "second class citizens".
A function I write should be able to replace a "stock" one.
A type I add should be able to be default for a type, replacing the stock one.

I should not have to say "...{dueDate}..." works, because Date is a well known type, but "...{price}..." does not work, because the type of price is Amount, and that is your custom data type.

Agreed. I actually think the MF2 spec does not need to define the actual formatting for numbers or dates either, leaving those entirely as implementation details. With such an approach, the explicit formatters of numbers and datetime objects would only need to be defined in terms of the options that they might receive.

All functions should behave as they are custom

In that the the core implementation should not know about any types, and the "well known types" (and formatters) should be in a different folder. The core should never do if (foo.type == "date")

Agreed. I think there is a specific case to be made for the spec to make reference to the matching of CLDR plural categories and string representations of integers to number values, and for the concatenated-string output to be somehow defined in addition to the formatted-parts output. Other than those specific cases and their resulting requirements, I do not believe the spec should concern itself with the actual types of values.

After this big detour, this is how I would handle this...

Values are nothing special, they are still formatted with functions.
The "save some typing" is relevant for syntax, not for data model.

It is like defining a function with default parameters (def greet(name, msg="Good morning!"): ...)

When I say "...{dueDate}..." it is still a function there, and there is still a default style (for example LONG)

But I would "register" those function in the runtime registry, associated with the type (as key).
(and I can also replace "standard" ones)

The core implementation never does if (type == "date") ..., it does runtimeRegisty.getFormatterForType(type) ...

So there is no difference between variables and formatting functions, they are all functions.
But there is no need to bother to tell you the function if you already know how to get it from the type.

There might be a difference between solving the type at rendering time vs at parse time (which is why making the distinction is important)

And that might result in treating them differently in the data model.
But should be a deliberate decision, with documented pros and cons, what a design document does.

Perhaps it would help to consider things from a slightly higher level of abstraction? We have all sorts of values that might be coming in as runtime arguments, or as the results of formatting functions. Rather than considering how to format each type separately, let's assume that we can build a wrapper around each { value, options } tuple that itself knows how to format itself as a string, or as a list of formatted parts, and how to match itself againt selector keys.

With this abstraction, we may consider the output of a formatting function to be such a wrapped value. So for instance something like the formatting function corresponding to {dueDate, DATE, ::yMMMd} would know how to construct such a wrapper around a date object, and would be able to parse the skeleton given to it, should the native formatter require some different shape of options.

One benefit of such an abstraction is that the spec could then be written in a way that requires all runtime values to use this wrapper, allowing for the spec itself to be completely agnostic on the type of a value. Some specific consideration would probably need to be made regarding the behaviour of a wrapper around a number, and how it behaves when compared to selector keys.

How does that sound? In my implementation, I use the term "Formattable" to refer to such a wrapper.

We should not drag in "the Fluent way" or "the ICU way" without any discussion.

I do not think anything at all is getting dragged in without thorough and exhaustive discussion.

It is very likely that the people who designed Fluent DID all this (I would be worry if they didn't)
And there was a deliberate decision to do things a certain way.

But those documents:

* Are not accessible to outsiders

* What was good in for browser / JS / custom localization tooling / company standard might be different for a server / C++ / standard l10n industry / universal standard

* Some decisions might be mistaken. People design stuff and then learn better when the rubber hits the road

So we can't just take some code that emulates what Fluent does and "enshrine" it as a Unicode standard.

I'm not really sure to what you're referring to here? This proposal is rather separate from anything done in Fluent, in case that's what you're implying.

If you're interested in the development history of Fluent, it's actually all been done in the open, and is accessible to anyone. If you're interested, this is the repository where you can find all this: https://github.com/projectfluent/fluent. I would in fact encourage you to do so, as the discussions there have covered quite a bit of the same ground as we have in the MF2 working group, but have often used a different approach.

Thankfully, though, some of the original developers of Fluent (namely @stasm and @zbraniecki) are with us in this working group, and have brought with them the benefit of their experiences while designing that language, and then seeing its adoption by real-world users. This allows us to build the MF2 spec with the benefit of their hindsight, which should be considered invaluable. Would you agree?

I value the experience of someone who used a system for a long time over the person who designed it.

We're in a lucky position to have a very tight feedback loop with the engineers using Fluent today in their day job building localizable Web interfaces in HTML, JS, WebComponents and React.

We just completed the migration of the main UI fully to Fluent and we have around 5500 Fluent messages in use in the product used by tens of millions of users. We also have a good paper trail of the work of hundreds of front end engineers who work on new features and maintain the codebase that uses Fluent over the span of the last 3 years.
And that's just one product, we have multiple that we can test, some of them purely "web", some of them fully "react" based etc.

We have 3 localization project managers who maintain those projects and can share relevant experience.
We also have access to the developer community of the Web based CAT tool which is used to localize Firefox and other Mozilla products to Fluent.

Finally, we have over 100 volunteer localization teams from all around the World who work with the CAT tool on localization of all of those products.

All of that work is public, open source and accessible to all of us for any incidental, or statistical analysis.

Are there any questions that we could ask of those groups that would help us formulate objective position on the questions we are facing about viability of the solutions implemented in Fluent?

I have the nagging feeling that we talk about the same thing, but somehow we don't understand each other.

So I keep rephrasing the same question in all kind of ways, to maybe click at some point...

Here is another way for me to ask it...

Does Formattable contains the value to format, or not?

And I keep trying to rephrase my affirmative answer to this. Let me try one more time: Yes, a Formattable contains the value to format, along with any relevant formatting options.

For more on this, I invite you to review my previous answers in this thread or look at PR #198, which adds the relevant section to the spec.

When I parse the syntax (whatever that is) and I have a model, we end up with a PatternElement[] Are some of them Formattable, at some point? Or not?

If the syntax was "...{dueDate, DATE, ::yMMMd}...", what would the PatternElement[] look like? Is there a Formattable there? Does it contain the value to be formatter, or not?

This too seems like a question that gets repeated. This is the PatternElement that I would use to represent your example:

{
  type: 'function',
  func: 'DATE',
  args: [{ type: 'variable', var_path: [{ type: 'literal', value: 'dueDate' }] }],
  options: { skeleton: { type: 'literal', value: 'yMMMd' } }
}

That is not a Formattable, it's a PatternElement. Those are different abstractions.

The PatternElement contains e.g. a reference to the dueDate variable. During formatting, the appropriate Pattern Element Formatter for type: 'function' is used to format it. A method of this Pattern Element Formatter is then called with the current Formatting Context and this PatternElement as its arguments. It would then:

1. Call the type: 'variable' Pattern Element Formatter to resolve the value of args[0] into a Formattable. During that process, the type: 'variable' Pattern Element Formatter would call the type: 'literal' Pattern Element Formatter to resolve the var_path[0] value into a Formattable.
2. Call the type: 'literal' Pattern Element Formatter to resolve the value of options.skeleton first into a Formattable, and then using its getValue() method, into the actual value.
3. Identify the DATE formatting function in the registry.
4. Call the DATE formatting function with the resolved args and options, as well as the current locale(s), and receive a Formattable representation of the resulting value.
5. Depending on the desired form of the output, either call the toString() or toParts() method of the returned Formattable to determine its own output value.
6. Return either a string or a list of MessageFormatPart objects (which are yet again different from either PatternElement and Formattable).

As you may observe, in this flow Formattable is only used internally to wrap runtime values, and does not show up either in the formatter's input arguments or its output.

Can I do

parserMessage.format ( params : {dueDate: {2021, 12, 17} }
parserMessage.format ( params : {dueDate: {2022, 1, 27} }
parserMessage.format ( params : {dueDate: {2023, 7, 3} }

and get different results?

Yes you can.

That means that the value to be formatted is not part of the Formattable PatternElement, it "comes in" later, based on an argument name (dueDate)

Or not?

The value to be formatted is not part of the PatternElement, but it is at runtime included in the Formattable wrapper around the dueDate value. PatternElement and Formattable are different things.

So the Formattable contains the value to be formatted? Or the name of the parameter that later will contain the value to be formatted?

Yes, Formattable contains the value to be formatted.

I think I am starting to (slowly?) understand...

So there is a model after parse, that does not contain any Formattable. The things that will be formatted at some point (like dueDate) are at this point just some kind of identifier.

But then at some later point there is a model where (some of?) the "functions" are replaced by Formattables, which now contain the real dueDate, which is (probably) a wrapper around Date (DateRuntimeValue?)

Is this acurate?

Yes, with the specification that the "some later point" that you refer to is during the format() or formatToParts() call.

If this is the case, then these are the areas that I think are getting in the way of understanding:

There are in fact 2 models:

* One with functions but not formattables, and no real values, just names of values. That is the one from parse?

Could you clarify what specifically you mean by "functions" here?

* The second model is derived from the first, with variable names like `dueDate` & functions replaced by `RuntimeValue<?>`, which are `Formattable`s

We can call these 2 models, or the same model transformed, whatever, tbd

I'm not sure that it's really beneficial to call the second representation a "model", but sure.

This is not explicit anywhere.

And I think that trying to describe 2 models (or 1 model with transformations in time) is what makes things confusing. That is the part that I did not clearly understand until now.

So the runtime workflow is something like this?

model1 = parser( some syntax )
// model1 has functions + variable names
model2 = model1.secretFunction( map of variables ? )
// model2 has Formattables, which wrap real values (RuntimeValue)
model2.format(???) => string
model2.formatToParts(???) => ???

I called the secretFunction that because there is no explicit mentioning of it that I can find. But ignoring the names, is this what's happening, more or less?

Because at least for me this was not at all clear (still not sure that's the case)

The reason you're not seeing any reference to this secretFunction method is because it doesn't exist. There is no model2. It's more like this:

const msgRes = parseMessageResource(str) // pure data, with messages represented as PatternElement sequences
const mf = new MessageFormat(locale) // an object that has formatting methods
mf.addResource(msgRes) // and now it has messages too
mf.format(msgId, variables) // returns a string

Inside that mf.format() call, the following happens:

1. The right message is found from the available resources
2. Its select case is resolved (if necessary)
3. Pattern element formatters turn each of its pattern element values into string values
4. Those string values are concatenated.

In all of the above, Formattable values only exist and are relevant in step 3 of the above. Please see my preceding comment in this thread for an explicit step-by-step walkthrough of what's happening inside such a pattern element formatter.

This is what I asked in the "whiteboard document", the "9. Various “states” for in which the model can be" section.

I think reading that document might help reduce all this back and forth, as I was trying there to understand the model proposed by Stas, which is what I thing all of these PRs are trying to implement.

While the models that Staś and I are proposing are similar, they are not the same. It is unfortunate that you find it so challenging to read others' code, as that might be the most direct way of understanding what they're doing, and how they differ.

It did not upset me at all.

My intention was to emphasize that we do operate in a (lucky for most standard WGs) scenario where we have access to prior art authors, and multiple classes of users of the system so if we have a question like "will CAT tools be able to X" or "will developers want to use Y" or "will localizers benefit from Z" we can ask those questions directly and get answers.

It was my response to your comment that you value the experience of someone who used a system for a long time over the person who designed it.

So it seems like we can get those opinions and reduce the he-said-she-said if we can formulate the questions that would unblock us.

Could you clarify what specifically you mean by "functions" here?

I was thinking that this is what you describe as "function" (things that are registered, like "DATE")

Okay, with this understanding this description that you provided of the "first model" does not match mine:

• One with functions but not formattables, and no real values, just names of values. That is the one from parse?

Specifically, I do not think functions are included within any such model. In my view, the model may include e.g. a type: 'function' pattern element, which is purely data. As described in this comment, when that pattern element is being formatted, the registered function is identified in the registry and called with appropriate arguments.

In my understanding, the data model is at all times a model that only contains data. In fact, I don't think there's ultimately any reason for it to contain anything other than maps, lists, and string values.

My main question then is: if this "second state" exists only inside the format call, why expose it? Why make it part of the official data model at all? Why not an implementation detail? What is the benefit in exposing it?

Could you clarify how you see it being exposed? Later on in this comment I provide a step-by-step walkthrough of the internals of a format() call, and this "second state" is never reached, as each pattern element is formatted directly to a string.

In brief, I'm trying to figure out why you think that this "second state" exists, and if perhaps there's something unclear about my previous comments where I've tried to explain how the formatting would happen? Please do ask if I've left something unclear, I've tried to be quite thorough.

When introducing Formattable to this discussion in this comment, I tried to clarify that it's primarily a wrapper around runtime values. Was that not clear? When formatting a message, there usually isn't any reason to represent it or its parts directly as Formattables, even if Formattables are used internally in the pattern element formatters.

Specifically, I do not think functions are included within any such model. In my view, the model may include e.g. a type: 'function' pattern element, which is purely data.

I am perfectly happy with that, and I would like to see if we can agree on the implications.

For example this means:

* Let's not talk about `Formattables`, it's an implementation detail

* The interface that defines the select/format/formatToParts is irrelevant

I have difficulty seeing how this follows, i.e. how agreeing on the data model only containing data leads to conclusions about discussion on other parts of the specification. I agree that Formattables and the interfaces of format/formatToParts calls don't really relate to the data model, but whether or not they're otherwise relevant to the whole of the MF2 spec is an entirely separate question.

What is important is the signatures of the 3 methods, and describing what they do. It is not relevant if the "runtime registry" is implemented as a map from string (function name) to "Function" (class+3 methods), or there are 2 classes, a "format" one and a "select" one, or if there are 3, one for each method type.

We specify the methods exposed by an implementation of the "runtime registry" and that's good enough (that is working on terms of interfaces).

First question that arises to me from this: Do you see a reason to expose an external selectMessageBody method/function in the public API? I would've presumed that this would be strictly internal.

Second, do you see how Formattables may be used as an abstraction for defining e.g. the "runtime registry" function interfaces? As in, we need a way to encapsulate the behaviour of a formatting function in at least the three places that you've mentioned:

1. When the function's value is used to select between message cases.
2. When the function's value is a part of a message that is formatted to a string.
3. When the function's value is a part of a message that is formatted to a sequence of parts.

Formattable is an abstraction for a value that allows for a single function to implement each of the above requirements. It is not a mandated implementation. If you do not think that it is a useful abstraction, could you e.g. write up a description of how you would specify the runtime registry methods? If you'd find it easier to do so in Java or TypeScript rather than text, I'd be happy to look at those as well.

In my preferred implementation, the MessageFormat constructor accepts a runtime option that may override the default datetime and number formatting functions. This runtime function registry is then included in the formatting context, and from there used by the function pattern element formatter when needed. When called, the arguments of a registered function are all wrapped up as Formattables, as is its return value. This means that when called, the registered formatting function doesn't need to know if its output is being e.g. formatted to parts vs. formatted to string.

So if you do not think that Formattables should be included in the spec, then please do indeed describe what these registry methods ought to look like.

Could you clarify how you see it being exposed?

There is a PR adding it to the spec: #198 I call that "exposing it"

Did you perhaps not notice the description of the PR? It starts with this paragraph:

As the proposal's text describes, a Formattable is a wrapper around values and their formatting options. It may be thought of as an abstraction that allows us to define the rest of the specification in a manner that's independent of the actual type of any runtime value.

A Formattable is an abstraction. It is a tool that allows us to talk about values being passed to and from functions, such that the function does not need to care about how it is being called. This does not in fact mean that it becomes a required part of each implementation.

This is the reason I use "fuzzy lingo" when I talk about implementation details, and I put things in quotes, or give slash-separated alternate names (function/formatter/lambda/callable), and use pseudo-code. Not because I can't decide on a specific term.

So that nobody confuses possible implementation with spec.

On the other hand, using "fuzzy lingo" means that we can't really be sure we're talking about the same thing, or that we're not mixing up different things. This is why I'm doing my best to refer to each thing every time with the same name as before.

I do not buy the argument that this clarity should be avoided because it might make someone mix up what's in the spec and what isn't. That is very clearly answered by looking at the spec, and seeing if it's there or not.

The short answer is "yes, such situations will be supported". I would envision that both examples could/would be handled by (custom) formatters, so in syntax you'd end up with something like You have an appointment { locative($temporal } and Finding things { locative($place) }, with the locative() formatter handling the complicated magic.

Perhaps a more fundamental question here would be whether the "formatted" output ought to be static and fully resolved, or whether it could contain functions/methods that expect additional parameters to be given. So e.g. a "formatted" date could still expect to be passed an argument determining whether it's to be displayed on screen or read out loud. And this is where we connect with the questions about how deeply we define the MF2 runtime; do we assert a specific answer to this question, or do we only define the general shape of what a formatter might do with a message?