[Translation] allow using the ICU message format using domains with the "+intl-icu" suffix #28952

nicolas-grekas · 2018-10-22T17:32:14Z

Q	A
Branch?	4.2
Bug fix?	no
New feature?	yes
BC breaks?	no (unless merged after 4.2)
Deprecations?	no
Tests pass?	yes
Fixed tickets	-
License	MIT
Doc PR	-

This PR replaces the FallbackFormatter logic by explicit opt-in.
This allows triggering accurate exceptions when one wants to use the ICU message format but the extension (or the polyfill) is missing.
The way to opt-in is to put messages in a domain with the +intl-icu suffix at loading/declaration time.
E.g. translations in messages+intl-icu.en.yaml will be read as part of the messages domain , but will be parsed by intl's MessageFormatter instead of the default Symfony one.
To make it seamless to adopt the ICU format, %s are trimmed from parameter keys (idea borrowed from https://github.com/webfactory/WebfactoryIcuTranslationBundle)

This is for 4.2 as it changes a feature that is not released yet.
ping @Nyholm - we drafted this together.

jvasseur · 2018-10-22T18:54:47Z

There is one downside to this compared to the FallbackFormatter, you won't be able to use ICU formatted messages with the validator component since the domain is configured for the whole component. But maybe this is something that should be fixed in the Validator component.

nicolas-grekas · 2018-10-22T18:58:46Z

~~@jvasseur we should work on this separately I suppose when preparing 4.3.~~ see resolution below

javiereguiluz · 2018-10-23T09:51:08Z

The current proposal (adding +intl-icu suffix) looks the best one to me ... but I wonder if this could break things (for naming files like messages+intl-icu.en.xlf instead of messages.en.xlf). But I can't think of concrete examples where things will break. It's just a thought.

nicolas-grekas · 2018-10-23T11:42:01Z

I wonder if this could break things (for naming files like messages+intl-icu.en.xlf instead of messages.en.xlf)

nothing I can think of: a + is a legal char in a filename, and also in a domain name, so all good.

stof · 2018-10-23T13:14:50Z

This requires updating all places using the translator to use an explicit domain though, making the migrating painful.
Could we rely on some catalogue metadata instead of relying on the domain name instead ? Adding some metadata about which keys are Intl ones would require cooperation from loaders (to fill the info based on some opt-in for the resource), but that's doable as we implement them (and custom loaders would keep registering their keys as non-ICU until they support this opt-in, but I think that's fine as there are much less projects writing custom loaders than projects using the translator).
This way, we would not need to allow changing the translation domain in many places (to be able to switch between the normal one and the +intl-icu one).

stof · 2018-10-23T13:16:16Z

the question of course would be: what is the way for loaders to decide whether the resource they load is using the Intl-ICU format or the Symfony one ?
For domain-based opt-in, the domain name is the opt-in.

stof · 2018-10-23T13:20:38Z

hmm, actually, looking at the implementation, the +intl-icu suffix is handled internally in the translator, by checking the ICU catalogue first and then the normal one. So there is no change in places using the translator (and so this fixes the concerns of @jvasseur as well).

nicolas-grekas · 2018-10-23T13:27:23Z

looking at the implementation, the +intl-icu suffix is handled internally in the translator

Yes! Sorry if that wasn't clear. I updated the PR description.

src/Symfony/Bundle/FrameworkBundle/DependencyInjection/Configuration.php

stof · 2018-10-23T14:09:28Z

src/Symfony/Component/Translation/Formatter/IntlMessageFormatter.php

+                throw new LogicException('Cannot parse message translation: please install the "intl" PHP extension or the "symfony/polyfill-intl-messageformatter" package.');
+            }
+            try {
+                $this->cache[$locale][$message] = $formatter = new \MessageFormatter($locale, $message);


should we actually memoize all the formatters ? This increases memory usage for each message being formatted. In my experience, a page is much more likely to translate lots of different messages than lots of messages being the same.
What is the benefit in terms of speed when skipping the instantiation the second time (when using the intl extension. I don't care about the performance of the polyfill here, as the way to improve performance in such case would be to install ext-intl). Does it actually justify such memory leak ?

I benched this with the extension:
\MessageFormatter::formatMessage('Hello {name}', 'en', array('name' => 'Bob'));
vs
$formatter->format(array('name' => 'Bob'));

the later is 3 times faster than the former and each $formatter object takes 226 bytes.
With that in mind, I think it's worth doing it: for run-once requests we don't care, and for long-running ones, the benefit is worth it IMHO.

stof · 2018-10-23T14:11:51Z

src/Symfony/Component/Translation/Formatter/IntlMessageFormatter.php

+        foreach ($parameters as $key => $value) {
+            if ('%' === ($key[0] ?? null)) {
+                unset($parameters[$key]);
+                $parameters[trim($key, '%')] = $value;


should we also support the {{ name }} convention, which is used in the Validator component for instance ?

good idead, updated

src/Symfony/Component/Translation/Formatter/IntlMessageFormatter.php

…he "+intl-icu" suffix

Nyholm · 2018-10-25T15:17:42Z

src/Symfony/Component/Translation/Tests/Formatter/IntlFormatterTest.php

+        $this->assertInstanceof(IntlFormatterInterface::class, $formatter);
+        $this->assertSame('Hello Fab', $formatter->formatIntl('Hello {name}', 'en', array('name' => 'Fab')));
+        $this->assertSame('Hello Fab', $formatter->formatIntl('Hello {name}', 'en', array('%name%' => 'Fab')));
+        $this->assertSame('Hello Fab', $formatter->formatIntl('Hello {name}', 'en', array('{{ name }}' => 'Fab')));


Nyholm · 2018-10-25T15:20:06Z

src/Symfony/Component/Translation/Translator.php

+            if ($catalogue->defines($id, $domain)) {
+                break;
+            }
+            if ($cat = $catalogue->getFallbackCatalogue()) {


Are we really sure this always will return null eventually?

I suppose: this is the exact same logic as in transChoice. Not having it here was a bug actually (with no side effects since trans wasn't locale sensitive before)

nicolas-grekas · 2018-10-28T10:56:03Z

Ping @symfony/deciders

fabpot · 2018-10-28T18:22:31Z

Thank you @nicolas-grekas.

@Nyholm

… domains with the "+intl-icu" suffix (nicolas-grekas) This PR was merged into the 4.2-dev branch. Discussion ---------- [Translation] allow using the ICU message format using domains with the "+intl-icu" suffix | Q | A | ------------- | --- | Branch? | 4.2 | Bug fix? | no | New feature? | yes | BC breaks? | no (unless merged after 4.2) | Deprecations? | no | Tests pass? | yes | Fixed tickets | - | License | MIT | Doc PR | - This PR replaces the `FallbackFormatter` logic by explicit opt-in. This allows triggering accurate exceptions when one wants to use the ICU message format but the extension (or the polyfill) is missing. The way to opt-in is to put messages in a domain with the `+intl-icu` suffix *at loading/declaration time*. E.g. translations in `messages+intl-icu.en.yaml` will be read as part of the `messages` domain , but will be parsed by intl's `MessageFormatter` instead of the default Symfony one. To make it seamless to adopt the ICU format, `%`s are trimmed from parameter keys (idea borrowed from https://github.com/webfactory/WebfactoryIcuTranslationBundle) This is for 4.2 as it changes a feature that is not released yet. ping @Nyholm - we drafted this together. Commits ------- d95cc4d [Translation] allow using the ICU message format using domains with the "+intl-icu" suffix

nicolas-grekas added this to the 4.2 milestone Oct 22, 2018

nicolas-grekas force-pushed the translation-msg-format branch from 5c0f147 to a4fb360 Compare October 22, 2018 17:36

carsonbot added Status: Needs Review Translation Feature labels Oct 22, 2018

nicolas-grekas force-pushed the translation-msg-format branch from a4fb360 to 6e544b4 Compare October 23, 2018 08:37

nicolas-grekas force-pushed the translation-msg-format branch from 6e544b4 to 1e9b58d Compare October 23, 2018 11:41

stof reviewed Oct 23, 2018

View reviewed changes

nicolas-grekas force-pushed the translation-msg-format branch 2 times, most recently from f221f4a to 326dd08 Compare October 24, 2018 10:42

[Translation] allow using the ICU message format using domains with t…

d95cc4d

…he "+intl-icu" suffix

nicolas-grekas force-pushed the translation-msg-format branch from 326dd08 to d95cc4d Compare October 24, 2018 10:50

carsonbot removed the Status: Needs Review label Oct 26, 2018

stof approved these changes Oct 26, 2018

View reviewed changes

carsonbot added the Status: Reviewed label Oct 26, 2018

Nyholm reviewed Oct 26, 2018

View reviewed changes

chalasr approved these changes Oct 28, 2018

View reviewed changes

ogizanagi approved these changes Oct 28, 2018

View reviewed changes

fabpot approved these changes Oct 28, 2018

View reviewed changes

fabpot merged commit d95cc4d into symfony:master Oct 28, 2018

nicolas-grekas deleted the translation-msg-format branch October 29, 2018 11:48

javiereguiluz mentioned this pull request Oct 30, 2018

[Translation] allow using the ICU message format using domains with the "+intl-icu" suffix symfony/symfony-docs#10604

Closed

This was referenced Nov 3, 2018

Release v4.2.0-BETA1 #29072

Closed

Release v4.2.0-BETA1 #29073

Merged

ogizanagi mentioned this pull request Nov 6, 2018

Translation broke after upgrading to Symfony 4.2 beta1 symfony/demo#889

Closed

gharlan mentioned this pull request Nov 8, 2018

Symfony 4.2: support translation file suffix "+intl-icu" Haehnchen/idea-php-symfony2-plugin#1251

Closed

digilist mentioned this pull request Jan 6, 2019

Add support for the IntlMessageFormatter Haehnchen/idea-php-symfony2-plugin#1269

Merged

nicolas-grekas mentioned this pull request Feb 24, 2019

[Translator] Document intl format symfony/symfony-docs#10264

Closed

yceruto mentioned this pull request Dec 2, 2019

The translation:update command should be aware of the +intl-icu variant #34713

Closed

nicolas-grekas mentioned this pull request May 5, 2020

[Form] Add the html5 option to ColorType to validate the input #36302

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Translation] allow using the ICU message format using domains with the "+intl-icu" suffix #28952

[Translation] allow using the ICU message format using domains with the "+intl-icu" suffix #28952

nicolas-grekas commented Oct 22, 2018 •

edited

Loading

jvasseur commented Oct 22, 2018

nicolas-grekas commented Oct 22, 2018 •

edited

Loading

javiereguiluz commented Oct 23, 2018

nicolas-grekas commented Oct 23, 2018

stof commented Oct 23, 2018

stof commented Oct 23, 2018

stof commented Oct 23, 2018

nicolas-grekas commented Oct 23, 2018

stof Oct 23, 2018

nicolas-grekas Oct 24, 2018 •

edited

Loading

stof Oct 23, 2018

nicolas-grekas Oct 24, 2018

Nyholm Oct 25, 2018

Nyholm Oct 25, 2018

nicolas-grekas Oct 26, 2018 •

edited

Loading

nicolas-grekas commented Oct 28, 2018

fabpot commented Oct 28, 2018

[Translation] allow using the ICU message format using domains with the "+intl-icu" suffix #28952

[Translation] allow using the ICU message format using domains with the "+intl-icu" suffix #28952

Conversation

nicolas-grekas commented Oct 22, 2018 • edited Loading

jvasseur commented Oct 22, 2018

nicolas-grekas commented Oct 22, 2018 • edited Loading

javiereguiluz commented Oct 23, 2018

nicolas-grekas commented Oct 23, 2018

stof commented Oct 23, 2018

stof commented Oct 23, 2018

stof commented Oct 23, 2018

nicolas-grekas commented Oct 23, 2018

stof Oct 23, 2018

Choose a reason for hiding this comment

nicolas-grekas Oct 24, 2018 • edited Loading

Choose a reason for hiding this comment

stof Oct 23, 2018

Choose a reason for hiding this comment

nicolas-grekas Oct 24, 2018

Choose a reason for hiding this comment

Nyholm Oct 25, 2018

Choose a reason for hiding this comment

Nyholm Oct 25, 2018

Choose a reason for hiding this comment

nicolas-grekas Oct 26, 2018 • edited Loading

Choose a reason for hiding this comment

nicolas-grekas commented Oct 28, 2018

fabpot commented Oct 28, 2018

nicolas-grekas commented Oct 22, 2018 •

edited

Loading

nicolas-grekas commented Oct 22, 2018 •

edited

Loading

nicolas-grekas Oct 24, 2018 •

edited

Loading

nicolas-grekas Oct 26, 2018 •

edited

Loading