[Translation][TwigBridge] Implementing variables extraction in "translation:update" #38884

liarco · 2020-10-29T21:56:22Z

Q	A
Branch?	5.x
Bug fix?	maybe, see below
New feature?	yes
Deprecations?	no
Tickets
License	MIT
Doc PR	symfony/symfony-docs#...

To do:

feature implementation
create a demo project for testing
document strange behaviors found while working on this (possible bugs?)
fix regressions (e.g. some valid t() syntaxes now are broken when domain is set with trans_default_domain)
update existing tests
new tests
documentation?
update CHANGELOG.md

Required feedback:

is it ok to use hard-coded strings? (see [Translation][TwigBridge] Implementing variables extraction in "translation:update" #38884 (comment))
is there any preference for the "category" field value? (see [Translation][TwigBridge] Implementing variables extraction in "translation:update" #38884 (comment))
how to stop CS Fixer from editing tests fixtures? (see below: "Fabbot sill wants me to change array() to [] inside the tests fixtures")
is new documentation needed in this case? Do you have any suggestion about where to put it?

About this PR

This PR adds available variables to each translation extracted using translation:update. This can be useful to both developers and translators when editing key-based translations directly from the extracted files.

Here is an output example:
PHP code:

$translator->trans('php_with_params_with_whitespaces_with_trailing_comma', [
    'var1' => 'val1',
    'var2' => 'val2',
], 'my-domain');

XLIFF 2.0 output:

<?xml version="1.0" encoding="utf-8"?>
<xliff xmlns="urn:oasis:names:tc:xliff:document:2.0" version="2.0" srcLang="en" trgLang="en">
  <file id="messages.en">
    <unit id="ya16Z3i" name="php_with_params_with_whitespaces_with_trailing_comma">
      <notes>
        <note category="symfony-extractor-variables">Available variables: var1, var2</note>
      </notes>
      <segment>
        <source>php_with_params_with_whitespaces_with_trailing_comma</source>
        <target>__php_with_params_with_whitespaces_with_trailing_comma</target>
      </segment>
    </unit>
  </file>
</xliff>

Example with Poedit:

New features

extracting variables from Twig/PHP source and adding them to catalogue metadata

supporting Twig t() functions when used without |trans() filter (e.g. as variable value, see example below)

{% set my_var = t('my_translation_key', {'var1': 'val1', 'var2': 'val2'}, 'custom-domain') %}

{{ my_var|trans }}

updating variables metadata when source changes (even if translation already exists)
always reporting the highest variables count found in code if the same key is used multiple times

How to test it

I created a demo project at liarco/translation-variables-extractor-demo.
You can clone it and clone this PR, then run:

cd $DEMO_CLONE_PATH
composer install

cd $PR_CLONE_PATH
php link $DEMO_CLONE_PATH
# now you are using code from this PR...

cd $DEMO_CLONE_PATH
bin/console translation:update --force --xliff-version=2.0 en

The latest command will generate translations from the source files:

src/Controller/TestController.php: php syntax examples (with and without explicit domain)
templates/test/index.html.twig: twig syntax examples (without explicit domain)
templates/test/domain.html.twig: twig syntax examples (with trans_default_domain)

You can check the translations running:

symfony serve
# now check out `/` and `/domain` to see all translation results...

Strange behaviors in current Symfony code

While working on this PR I found some strange behaviors that I think may be bugs in the current code. I think I fixed them with my code, but they need to be checked. I'm gonna update this section with documentation about the stuff I found.

If my new feature will be rejected I would be happy to revert it while keeping the bug fixes (if there is any).

Possibly broken `--clean`

If you test my demo project with original Symfony code:

cd $PR_CLONE_PATH
php link --rollback $DEMO_CLONE_PATH

cd $DEMO_CLONE_PATH
composer install
# now you are using original Symfony code...

bin/console translation:update --force --xliff-version=2.0 en --clean
# first run seems to be ok (translation files are created with Intl domain suffix), run it again...
bin/console translation:update --force --xliff-version=2.0 en --clean
# new files are created for each custom domain with no Intl suffix... my code fixes that

Update 2021-02-16: my fix was the same as #40050, I merged it to resolve conflicts.

Test that I didn't understand, maybe there's an error?

// src/Symfony/Component/Translation/Tests/Catalogue/TargetOperationTest.php (Original Symfony code)
public function testGetResultWithMixedDomains()
    {
        $this->assertEquals(
            new MessageCatalogue('en', [
                'messages+intl-icu' => ['a' => 'old_a'],
             // 'messages' => ['a' => 'old_a'], <- here is my version
            ]),
            $this->createOperation(
                new MessageCatalogue('en', ['messages' => ['a' => 'old_a']]),
                new MessageCatalogue('en', ['messages+intl-icu' => ['a' => 'new_a']])
            )->getResult()
        );
    }

While I think this test can be useful (I copied it to MergeOperationTest.php) I can't understand why it should override the domain while keeping the old message. This would mean loading the old value using a different standard from the one which the translation was written for.
This test was introduced in b72b7d3, but I think that it is based on the wrong behavior of --clean that I explained above.

Current Symfony code doesn't support all syntaxes

~~Running the extraction tool on the demo project extracts less keys compared to my PR.~~
I was testing this using 5.1 in demo project (t() function was not supported) so it couldn't extract those keys, but 5.x works fine. Now I updated the demo and added more syntax examples.

Update 2020-11-01: I added support for t() functions used without |trans() filter which are not supported by the original implementation.

Some test cases have been broken by CS fixes

Original tests were performed against both array() and [] syntaxes: 78c0ec5#diff-296e65f0a4669683ef3d74c37eb9983fcf963a0900e1111ce0d497e3f7e0d065

Then they have been changed to fix CS: a23330b#diff-296e65f0a4669683ef3d74c37eb9983fcf963a0900e1111ce0d497e3f7e0d065

Now only the short syntax is tested, but the entries are still there... I reimplemented them.

Fabbot sill wants me to change `array()` to `[]` inside the tests fixtures

symfony/.php_cs.dist

Line 25 in 176f52d

->notPath('#/Fixtures/#')

It seems like the fixtures folder should be capitalized to be filtered properly by CS Fixer. But I tried to rename the folder locally and the ->notPath('#/Fixtures/#') line is ignored anyway... I don't know why. (Side note: changing it to ->notPath('#Fixtures#') skips the folder successfully, I know that it can't be a solution, but it means that the method is not completely ignored)

src/Symfony/Component/Translation/Extractor/PhpExtractor.php

liarco · 2020-10-31T02:17:51Z

I think that this PR may be ready for review, please take a look at the required feedback section on top of description.

Thank you for your time.

src/Symfony/Bridge/Twig/Translation/TwigExtractor.php

liarco · 2020-11-26T12:45:51Z

Hi @jderusse, I'm sorry to bother you. I just wanted to ask if there's anything that I can do to have the feedback I'm missing in the main PR message in order to finalize the development of this feature.

It's not much stuff, but I would like to complete this in order to leave you a better PR to review.

Thank you for your time.

liarco · 2021-02-16T12:45:44Z

Hi @yceruto, my PR is having conflicts with your #40050 since I was facing the same issue while implementing the feature. May I ask you if you would suggest resolving the conflict keeping your changes or using mine? (I think they are almost the same except for a more explicit variable naming on my side, but I don't know if that's considered too verbose)

Thank you for your time.

yceruto · 2021-02-16T18:46:15Z

Hi @liarco, keep your changes (the new naming looks good to me) while tests passed it's ok.

Nyholm · 2021-03-09T09:34:06Z

Cool. Can we focus on finishing this PR this week or next?

@liarco, could you rebase your PR and make sure Travis (2/3) and Fabbot.io is green?

Nyholm · 2021-03-09T09:35:32Z

Oops. Sorry. I see your comment about the CI... Good. Dont make fabbot green =)

Could you rebase please?

…efault_domain

…tor)

…ctor)

Nyholm

Well done. Here are some small things I noticed.

src/Symfony/Bridge/Twig/NodeVisitor/TranslationNodeVisitor.php

src/Symfony/Bridge/Twig/Translation/TwigExtractor.php

src/Symfony/Component/Translation/Extractor/PhpExtractor.php

liarco · 2021-03-09T09:48:32Z

Cool. Can we focus on finishing this PR this week or next?

@liarco, could you rebase your PR and make sure Travis (2/3) and Fabbot.io is green?

Thank you, it would be a pleasure! :)

I'm going to update the repo and check CI (I hope nothing broke badly, but I'll take care of that). Please feel free to tell me if you see anything that can be fixed/improved.

src/Symfony/Bridge/Twig/NodeVisitor/TranslationNodeVisitor.php

nicolas-grekas · 2022-07-29T13:17:45Z

/cc @welcoMattic would you have some time to check this one please? Maybe it conflicts with your PR related to using an AST? Or maybe you're just the right person to review? 👼

welcoMattic · 2022-07-30T11:17:54Z

src/Symfony/Component/Translation/Extractor/PhpExtractor.php

+                        // Update old variables note (if any)
+                        if (isset($metadata['notes'])) {
+                            foreach ($metadata['notes'] as $index => $note) {
+                                if (isset($note['category']) && MessageCatalogue::METADATA_AVAILABLE_VARIABLES_KEY === $note['category']) {
+                                    // Keep the highest variables count
+                                    if (\count($variables) > substr_count($note['content'], ',')) {
+                                        $metadata['notes'][$index] = $variablesNote;
+                                    }
+
+                                    break;
+                                }
+                            }
+                        } else {
+                            $metadata['notes'][] = $variablesNote;
+                        }
+                    }


@liarco this code is repeated several times, shouldn't we factorize it in a separate method, in a new file?

welcoMattic · 2022-07-30T11:24:34Z

Hi @liarco, thanks a lot for this PR. I've some notes to share,

Test that I didn't understand, maybe there's an error?
// src/Symfony/Component/Translation/Tests/Catalogue/TargetOperationTest.php (Original Symfony code)
public function testGetResultWithMixedDomains()
    {
        $this->assertEquals(
            new MessageCatalogue('en', [
                'messages+intl-icu' => ['a' => 'old_a'],
             // 'messages' => ['a' => 'old_a'], <- here is my version
            ]),
            $this->createOperation(
                new MessageCatalogue('en', ['messages' => ['a' => 'old_a']]),
                new MessageCatalogue('en', ['messages+intl-icu' => ['a' => 'new_a']])
            )->getResult()
        );
    }
While I think this test can be useful (I copied it to MergeOperationTest.php) I can't understand why it should override the domain while keeping the old message. This would mean loading the old value using a different standard from the one which the translation was written for. This test was introduced in b72b7d3, but I think that it is based on the wrong behavior of --clean that I explained above.

About these tests, they were fixed in 2 more recent commits (84b48fe#diff-b554ece6142fcdb8b04a4afee7ad8e95400ec5bbc8dd65e5bda8e0196a79f64fR74 and 1802488#diff-b554ece6142fcdb8b04a4afee7ad8e95400ec5bbc8dd65e5bda8e0196a79f64fR75), now I guess it fixes the confusion. Can you rebase your branch please?

Beside this, your PR is adding variables extraction feature in PhpExtractor which we are about to deprecate in 6.2 (see #46161). After merging of these 2 branches we will have to port the variable extraction into the new PhpAstExtractor. I could help you if needed, feel free to ping me when the time comes.

A last thing, I think we have to add more tests on Dumpers, to ensure variables notes are correctly written in translation files (not sure that category is handled by all dumpers, to be checked).

welcoMattic · 2023-03-31T08:13:13Z

@liarco Hi, any news on this feature? As we now have a brand new PhpAstExtractor in 6.2, it could help to achieve this PR.

welcoMattic · 2023-09-20T09:44:18Z

@liarco Any news here? I like this feature, coupled with the new translation:extract command and Translation Providers, it could be powerful for translation management!

fabpot · 2024-02-03T18:07:29Z

Closing as there is no more activity.

carsonbot added Status: Needs Review Feature labels Oct 29, 2020

jderusse added this to the 5.x milestone Oct 29, 2020

jderusse added Translation TwigBridge labels Oct 29, 2020

liarco commented Oct 30, 2020

View reviewed changes

src/Symfony/Component/Translation/Extractor/PhpExtractor.php Outdated Show resolved Hide resolved

liarco force-pushed the translation-variables-extractor branch from 35713fe to 8ac5e2a Compare October 30, 2020 23:43

liarco marked this pull request as ready for review October 31, 2020 02:15

liarco commented Oct 31, 2020

View reviewed changes

src/Symfony/Bridge/Twig/Translation/TwigExtractor.php Outdated Show resolved Hide resolved

liarco marked this pull request as draft October 31, 2020 19:16

liarco marked this pull request as ready for review October 31, 2020 23:33

liarco force-pushed the translation-variables-extractor branch from 74cba18 to c0a6eff Compare December 18, 2020 22:56

liarco force-pushed the translation-variables-extractor branch from e2a0f8d to be7c114 Compare February 16, 2021 21:19

liarco mentioned this pull request Mar 9, 2021

Suggestion on the translation system #40418

Closed

liarco added 10 commits March 9, 2021 10:41

Implementing variables extraction in "translation:update"

06fba65

Applying patch from fabbot

e2934b5

Fixing regression: Twig t() now works when domain is set with trans_d…

bedb0be

…efault_domain

Fixing variables metadata overwriting previously set data (PHP extrac…

87d542f

…tor)

Updating existing tests to include variables

8142f75

Applying patch from fabbot

a333f13

Supporting both long and short array syntax (PHP extractor)

38d9e55

Improving/fixing existing tests and adding variables tests (PHP extra…

edb5aa9

…ctor)

Adding variables tests (Twig extractor)

31f96a0

Clearer naming

5c2e782

liarco force-pushed the translation-variables-extractor branch from be7c114 to 2a10f37 Compare March 9, 2021 09:41

Nyholm requested changes Mar 9, 2021

View reviewed changes

carsonbot added Status: Needs Work and removed Status: Needs Review labels Mar 9, 2021

carsonbot changed the title ~~Implementing variables extraction in "translation:update"~~ [Translation][TwigBridge] Implementing variables extraction in "translation:update" Mar 9, 2021

liarco added 3 commits March 9, 2021 12:54

Minor fixes

0a5366f

Using constants for metadata key and prefix

46d26ec

Fixing CI

1612c08

Nyholm reviewed Mar 9, 2021

View reviewed changes

src/Symfony/Bridge/Twig/NodeVisitor/TranslationNodeVisitor.php Outdated Show resolved Hide resolved

liarco added 2 commits March 9, 2021 15:37

Improving code quality

8ec6086

Fixing static analysis (Psalm)

0025c52

liarco commented Mar 17, 2021

View reviewed changes

src/Symfony/Bridge/Twig/NodeVisitor/TranslationNodeVisitor.php Show resolved Hide resolved

fabpot modified the milestones: 5.4, 6.1 Oct 30, 2021

fabpot modified the milestones: 6.1, 6.2 May 20, 2022

welcoMattic reviewed Jul 30, 2022

View reviewed changes

nicolas-grekas modified the milestones: 6.2, 6.3 Nov 5, 2022

nicolas-grekas modified the milestones: 6.3, 6.4 May 23, 2023

nicolas-grekas modified the milestones: 6.4, 7.1 Nov 15, 2023

fabpot closed this Feb 3, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Translation][TwigBridge] Implementing variables extraction in "translation:update" #38884

[Translation][TwigBridge] Implementing variables extraction in "translation:update" #38884

liarco commented Oct 29, 2020 •

edited

Loading

liarco commented Oct 31, 2020

liarco commented Nov 26, 2020 •

edited

Loading

liarco commented Feb 16, 2021

yceruto commented Feb 16, 2021

Nyholm commented Mar 9, 2021

Nyholm commented Mar 9, 2021 •

edited

Loading

Nyholm left a comment

liarco commented Mar 9, 2021

nicolas-grekas commented Jul 29, 2022

welcoMattic Jul 30, 2022

welcoMattic commented Jul 30, 2022

Test that I didn't understand, maybe there's an error?

welcoMattic commented Mar 31, 2023

welcoMattic commented Sep 20, 2023

fabpot commented Feb 3, 2024

[Translation][TwigBridge] Implementing variables extraction in "translation:update" #38884

[Translation][TwigBridge] Implementing variables extraction in "translation:update" #38884

Conversation

liarco commented Oct 29, 2020 • edited Loading

About this PR

New features

How to test it

Strange behaviors in current Symfony code

Possibly broken --clean

Test that I didn't understand, maybe there's an error?

Current Symfony code doesn't support all syntaxes

Some test cases have been broken by CS fixes

Fabbot sill wants me to change array() to [] inside the tests fixtures

liarco commented Oct 31, 2020

liarco commented Nov 26, 2020 • edited Loading

liarco commented Feb 16, 2021

yceruto commented Feb 16, 2021

Nyholm commented Mar 9, 2021

Nyholm commented Mar 9, 2021 • edited Loading

Nyholm left a comment

Choose a reason for hiding this comment

liarco commented Mar 9, 2021

nicolas-grekas commented Jul 29, 2022

welcoMattic Jul 30, 2022

Choose a reason for hiding this comment

welcoMattic commented Jul 30, 2022

Test that I didn't understand, maybe there's an error?

welcoMattic commented Mar 31, 2023

welcoMattic commented Sep 20, 2023

fabpot commented Feb 3, 2024

liarco commented Oct 29, 2020 •

edited

Loading

Possibly broken `--clean`

Fabbot sill wants me to change `array()` to `[]` inside the tests fixtures

liarco commented Nov 26, 2020 •

edited

Loading

Nyholm commented Mar 9, 2021 •

edited

Loading