Thanks to visit codestin.com
Credit goes to github.com

Skip to content

Clarify fallback resolution #539

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged

Conversation

gibson042
Copy link
Collaborator

  • Consistently phrase "fail(s) to resolve" and italicize "fallback value".
  • Improve examples by showing both input and output.
  • Include previously missing cases:
    • non-function annotations (e.g., {@reserved …}, {|…| @reserved}, {$var @reserved})
    • literals that need escaping (e.g., {|C:\\| :func})
    • non-annotated variables (e.g., {$noSuchVar})
    • transitive fallback (e.g., .local $bad = {:noSuchFunc} {{{$bad}}})
  • Remove unused "otherwise" case.

Also includes a syntax change to require a function-like identifier for every annotation, in support of defining the fallback value for expressions like {@reserved} and {^private}.

@gibson042 gibson042 requested a review from aphillips November 28, 2023 22:36
Copy link
Collaborator

@eemeli eemeli left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I like the fallback resolution changes, but not the reserved-body changes. See inline comment reply for more.

@aphillips
Copy link
Member

Also includes a syntax change to require a function-like identifier for every annotation, in support of defining the fallback value for expressions like {@reserved} and {^private}.

We explicitly did not add this the first time and, in particular, allow a lot of freedom for private use. If people want to use private use for comments, it's their funeral...

Copy link
Member

@aphillips aphillips left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some comment. Good start.

Comment on lines 278 to 279
> In a context where `:now` fails to resolve but `:datetime` does not,
> `.local $t = {:now format=iso8601} .local $pretty_t = {$t :datetime} {{{$pretty_t}}}` resolves to the _fallback value_ `:now`.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Two comments:

  1. This is tricky. If $t is evaluated eagerly (which we allow), then $pretty_t might contain a different sort of error (since $t contains a string that isn't a representative of a date/time value). Is this what we want here?

  2. I find the examples hard to read vs. what we had before. I think the whitespace and multiline presentation of the original example (which is fine to delete) could be beneficial to replicate in the new examples.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The second point is now handled, but the first seems like a consequence of logic we already have—I'm just highlighting its transitive nature:

  1. :now fails to resolve, so .local $t = {:now format=iso8601} resolves to the fallback value ":now".
  2. Therefore .local $pretty_t = {$t :datetime} resolves to the same fallback value ":now", even in the current text—"When an error occurs in an expression with a variable operand and the variable refers to a local declaration, the fallback value is formatted based on the expression on the right-hand side of the declaration…", followed by an explicit example, and also "If a variable would resolve to a fallback value,
    this MUST also be considered a failure" text in Variable Resolution
  3. Therefore {$pretty_t} resolves to the same fallback value ":now" for the same reason.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The main intent with the current transitive logic is to prevent the name of a local variable from ever showing up in the output, either during fallback or as the source of a formatted part.

Local variables should be considered message-internal implementation details that should never leak outside it.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

...which I believe is demonstrated in the new example.

Copy link
Member

@aphillips aphillips left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some minor wording tweaks.

on the right-hand side of the _declaration_,
rather than the _expression_ in the _selector_ or _pattern_.
- _expression_ with _variable_ _operand_ (with or without an _annotation_):
- If the _variable_ fails to resolve, or resolves to a value that is not a _fallback value_:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I found this confusing. You have to read everything here, including the next case, to understand how a fallback value could get into the variable. Unless you know about the transitive case, this appears to say "all variables". I would suggest putting the variable fails to resolve case first, then the transitive case (starting line 291) and end with an "otherwise" case for variables that resolve correctly but whose expressions fail.

Yes, that will be slightly repetitive in the last case.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have refactored into "local reference" vs. "other" cases, which better maps to the "prevent the name of a local variable from ever showing up in the output" goal.

Comment on lines 255 to 273
- If the _annotation_ consists of a _sigil_-prefixed _identifier_
optionally followed by a whitespace-prefixed tail:
the _annotation_ starting sigil followed by that _identifier_
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This sounds an awful lot like someone setting up a follow-up change to reserved syntax. 😇

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Indeed. It would alternatively be possible to revert this privileging of non-function annotations that use function-like syntax (instead having them always fall back to ), but to me that seems unnecessarily future-hostile.

@gibson042 gibson042 force-pushed the 2023-11-clarify-fallback-resolution branch 2 times, most recently from eb70ae8 to a546af2 Compare November 30, 2023 21:56
Copy link
Member

@aphillips aphillips left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Close, but I think some further structuring will make this easier to read/implement


- _expression_ with _variable_ _operand_: U+0024 DOLLAR SIGN `$`
followed by the _variable_ _name_ of the _operand_
- If the non-_function_ _annotation_ consists of a _sigil_-prefixed _identifier_
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A "non-function annotation" is the same as reserved-annotation+private-use-annotation. Is there some reason to be elliptical about it?

Suggested change
- If the non-_function_ _annotation_ consists of a _sigil_-prefixed _identifier_
If the _annotatation_ is a _reserved annotation_ or a _private-use annotation_
and the `reserved` or `private-use` sigil is immediately followed by a character
sequence that is an _identifier_,
make the _fallback value_ the sigil followed by that _identifier_

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I figure it's pretty much "function" vs. "other", because there's no differentiation here between private-use vs. reserved annotations. But your suggestion does more than just rephrase, because the instances of the latter do not necessarily start with a sigil-prefixed identifier (e.g., consider {@ reserved} and {>} and {&|quoted|}).

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Those instances would not match my text and they would emit the logo. What's our intention here? Do we want to pass reserved/private garbage unmolested? Or turn them into the logo? My reading of the current text is that we make them into the logo unless they look like {^identifier something something} where we emit {^identifier} for some reason.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Current text is incoherent—the predicate of "expression with no operand: the function starting sigil followed by its identifier" covers non-function expressions, but the consequent cannot be applied to them. That can be fixed by always having such expressions fall back to the logo, or (as I have done here) by applying function-like treatment to those that use function-like syntax. But I consider it important for "function-like" to be defined narrowly enough to exclude e.g. {@spaceless|quoted|}.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's almost coherent except for the possibility of reserved/private... which is what we're fixing here. Because reserved-annotation and private-use-annotation are permissive about content and because we don't have a construct for parsing inside them in the ABNF (nor, I think, should we...) I think emitting the logo makes some sense.

Putting myself in the shoes of a user, this is the rendered output of a message containing some placeholders that my implementation doesn't understand. If we emit their contents, yes, that might assist in debugging. But if shown to users it might contain code internal stuff. The logo basically is an admission that your implementation has no idea what this placeholder means. Does that make sense?

Note: it's probably important that we emit something as a way of preventing sneak attacks such as {{<scr{@omit me}ipt href=...>}}

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yep, I agree. But as I said elsewhere, I think always falling back to {�} even for function-like expressions is unnecessarily future-hostile.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There's a bit further down in the spec that probably ought to be highlighted here as well:

Resolved values cannot always be formatted by a given implementation.
When such an error occurs during _formatting_,
an implementation SHOULD emit a _formatting error_ and produce a
_fallback value_ for the _placeholder_ that produced the error.
A formatting function MAY substitute a value to use instead of a _fallback value_.

The intent with that is that even though our default fallback must guard against e.g. {$user :name} having a failure mode where a custom stringifier of the $user value gets called, dumping personal data into the output, there are still plenty of places where a better fallback than the default can and should be defined.

But it does mean that the space of fallback values cannot be considered exhaustive, and so a reserved expression producing {�} fallback now should not be considered a restriction on what might happen if we introduce a new meaning for its sigil later. Also, we should add language somewhere allowing an implementation to define separately the fallback behaviour for private-use sigils.

Put together, I think we could default to {�} fallback for reserved and private-use, but make it clear(er) that later spec versions or implementations supporting private-use can and should define something better.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I also think that it is important to indicate to the user that a fallback happened : that is, what is displayed to the user is not the intended result. One possibility is prefixing and/or suffixing with �.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@macchiati Perhaps? Presumably the presence of curly-bracketed garbage is an indicator. But having � present consistently would make it easier to find and easier to check for. If we did this, I'd make it a prefix so that we'd have fallbacks like:

{�}
{�$var}
{�|42.7 some literal|}
{�:function}
{�^unsupported}

Comment on lines 283 to 322
- _expression_ with _literal_ _operand_:
U+007C VERTICAL LINE `|`
followed by the value of the _literal_
with escaping applied to U+005C REVERSE SOLIDUS `\` and U+007C VERTICAL LINE `|`,
and then by U+007C VERTICAL LINE `|`.
The same representation is used for both _quoted_ and _unquoted_ values.

_Option_ identifiers and values are not included in the _fallback value_.
> Examples:
> In a context where `:func` fails to resolve,
> `{42 :func}` resolves to the _fallback value_ `|42|` and
> `{|C:\\| :func}` resolves to the _fallback value_ `|C:\\|`.
> In any context, `{|| @reserved}` resolves to the _fallback value_ `||`.

When an error occurs in an _expression_ with a _variable_ _operand_
and the _variable_ refers to a local _declaration_,
the _fallback value_ is formatted based on the _expression_
on the right-hand side of the _declaration_,
rather than the _expression_ in the _selector_ or _pattern_.
- _expression_ with _variable_ _operand_ (with or without an _annotation_):
- If the _variable_ refers to a local _declaration_:
the _value_ to which it resolves (which may already be a _fallback value_)

> For example,
> in a context in which the function `:func` fails to resolve,
> attempting to format either of the following messages:
>
> ```
> {{
> local $var = {|horse| :func}
> {{The value is {$var}.}}
> }}
> ```
>
> ```
> {{
> local $var = {|horse|}
> {{The value is {$var :func}.}}
> }}
> ```
>
> would in both cases result in the _pattern_ _expression_
> resolving to a _fallback value_ of `|horse|`.
> Examples:
> In a context where `:func` fails to resolve,
> the _pattern_'s _expression_ in `.local $var={|val|} {{{$val :func}}}`
> resolves to the _fallback value_ `|val|` and the message formats to `{|val|}`.
> In a context where `:now` fails to resolve but `:datetime` does not,
> the _pattern_'s _expression_ in
> ```
> .local $t = {:now format=iso8601}
> .local $pretty_t = {$t :datetime}
> {{{$pretty_t}}}
> ```
> (transitively) resolves to the _fallback value_ `:now` and
> the message formats to `{:now}`.

- Otherwise: U+0024 DOLLAR SIGN `$` followed by the _name_ of the _variable_

> Examples:
> In a context where `$var` fails to resolve, `{$var}` and `{$var :number}` and `{$var @reserved}`
> all resolve to the _fallback value_ `$var`.
> In a context where `:func` fails to resolve,
> the _pattern_'s _expression_ in `.input $arg {{{$arg :func}}}`
> resolves to the _fallback value_ `$arg` and
> the message formats to `{$arg}`.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this whole block should be moved to about line 262 so that the block is effectively structured:

The fallback value depends on the contents of the expression:

  • expression with literal operand
    • emit the literal
  • expression with variable operand
    • emit the variable
  • expression with no operand but with a function
    • emit the function name
  • expression with no operand but with reserved/private
    • if sigil+identifier, emit that, else fall through to:
  • otherwise emit the logo

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That also would be a behavior change, because not every variable operand is represented as a variable (specifically, a variable referencing a .local declaration is transparently treated as its resolved value).

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fair enough, but the above is a short representation of the whole. Do you disagree that reordering the text would make it clearer?

The fallback value depends on the contents of the expression:

  • expression with literal operand
    • emit the literal
  • expression with variable operand
    • if the variable references a .local emit the resolved value
    • else emit the variable
  • expression with no operand but with a function
    • emit the function name
  • expression with no operand but with reserved/private
    • if sigil+identifier, emit that, else fall through to:
  • otherwise emit the logo

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think a single-level list of cases with no nested branches is clearer, so I've updated accordingly (basically just promoting "variable referencing .local" vs. "variable not referencing .local" into distinct cases).

Copy link
Member

@aphillips aphillips left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looking good. Minor formatting nit.

@macchiati
Copy link
Member

macchiati commented Dec 3, 2023 via email

@aphillips
Copy link
Member

In the 2023-12-04 teleconference we resolved to not use the U+FFFD character except as the ultimate fallback and to use only the private-use/reserved sigil (no groping) as fallback for those. This makes the patterns:

{�}
{$var}                 // some variable
{|42.7 some literal|}  // some literal
{:function}            // some function
{^}                    // reserved/private-use exposes the sigil used, ^ is only an example

gibson042 and others added 11 commits December 4, 2023 15:33
* Consistently phrase "fail(s) to resolve" and italicize "fallback value".
* Improve examples by showing both input and output.
* Include previously missing cases:
  * non-function annotations (e.g., `{@… …}`, `{|…| @…}`, `{$var @…}`)
  * literals that need escaping (e.g., `{|C:\\| :func}`)
  * non-annotated variables (e.g., `{$noSuchVar}`)
  * transitive fallback (e.g., `.local $bad = {:noSuchFunc} {{{$bad}}}`)
* Remove unused "otherwise" case.
Co-authored-by: Addison Phillips <[email protected]>
@gibson042 gibson042 force-pushed the 2023-11-clarify-fallback-resolution branch from 8501195 to 92c855e Compare December 4, 2023 20:39
@gibson042
Copy link
Collaborator Author

In the 2023-12-04 teleconference we resolved to not use the U+FFFD character except as the ultimate fallback and to use only the private-use/reserved sigil (no groping) as fallback for those. This makes the patterns:

{�}
{$var}                 // some variable
{|42.7 some literal|}  // some literal
{:function}            // some function
{^}                    // reserved/private-use exposes the sigil used, ^ is only an example

These changes are complete, along with a case for supported private-use expression falling back to the sigil plus optional imlementation-specific details and a note that is not used for expression fallback values in this revision (since the cases are exhaustive).

Copy link
Member

@aphillips aphillips left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good. Thanks!

@aphillips
Copy link
Member

As agreed in teleconference, merging these changes.

@aphillips aphillips merged commit 59d84f7 into unicode-org:main Dec 4, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants