Some ideas to improve performance of the Translator component #13948

javiereguiluz · 2015-03-17T08:28:23Z

The problem

Recently some people have raised concerns about Translator component performance (see #13676 for example). I was inspecting the dumped container for production environment to get more information about this component and I saw the following problems:

1. All translation loaders and dumpers are instantiated and loaded, no matter which ones you use:

protected function getTranslation_LoaderService()
{
    $a = $this->get('translation.loader.xliff');
    $this->services['translation.loader'] = $instance = new \Symfony\Bundle\FrameworkBundle\Translation\TranslationLoader();
    $instance->addLoader('php', $this->get('translation.loader.php'));
    $instance->addLoader('yml', $this->get('translation.loader.yml'));
    $instance->addLoader('xlf', $a);
    $instance->addLoader('xliff', $a);
    $instance->addLoader('po', $this->get('translation.loader.po'));
    $instance->addLoader('mo', $this->get('translation.loader.mo'));
    $instance->addLoader('ts', $this->get('translation.loader.qt'));
    $instance->addLoader('csv', $this->get('translation.loader.csv'));
    $instance->addLoader('res', $this->get('translation.loader.res'));
    $instance->addLoader('dat', $this->get('translation.loader.dat'));
    $instance->addLoader('ini', $this->get('translation.loader.ini'));
    $instance->addLoader('json', $this->get('translation.loader.json'));
    return $instance;
}

protected function getTranslation_Dumper_QtService()
{
    return $this->services['translation.dumper.qt'] = new \Symfony\Component\Translation\Dumper\QtFileDumper();
}

protected function getTranslation_Dumper_ResService()
{
    return $this->services['translation.dumper.res'] = new \Symfony\Component\Translation\Dumper\IcuResFileDumper();
}

// ...

Same thing happens with the translation dumpers:

protected function getTranslation_WriterService()
{
    $this->services['translation.writer'] = $instance = new \Symfony\Component\Translation\Writer\TranslationWriter();
    $instance->addDumper('php', $this->get('translation.dumper.php'));
    $instance->addDumper('xlf', $this->get('translation.dumper.xliff'));
    $instance->addDumper('po', $this->get('translation.dumper.po'));
    $instance->addDumper('mo', $this->get('translation.dumper.mo'));
    $instance->addDumper('yml', $this->get('translation.dumper.yml'));
    $instance->addDumper('ts', $this->get('translation.dumper.qt'));
    $instance->addDumper('csv', $this->get('translation.dumper.csv'));
    $instance->addDumper('ini', $this->get('translation.dumper.ini'));
    $instance->addDumper('json', $this->get('translation.dumper.json'));
    $instance->addDumper('res', $this->get('translation.dumper.res'));
    return $instance;
}

2. All resources for all locales are loaded, no matter which ones you use:

protected function getTranslator_DefaultService()
{
    // ...

    $instance->setFallbackLocales(array(0 => 'es'));
    $instance->addResource('xlf', ($this->targetDirs[3].'/vendor/symfony/symfony/src/Symfony/Component/Validator/Resources/translations/validators.af.xlf'), 'af', 'validators');
    $instance->addResource('xlf', ($this->targetDirs[3].'/vendor/symfony/symfony/src/Symfony/Component/Validator/Resources/translations/validators.ar.xlf'), 'ar', 'validators');
    $instance->addResource('xlf', ($this->targetDirs[3].'/vendor/symfony/symfony/src/Symfony/Component/Validator/Resources/translations/validators.az.xlf'), 'az', 'validators');
    $instance->addResource('xlf', ($this->targetDirs[3].'/vendor/symfony/symfony/src/Symfony/Component/Validator/Resources/translations/validators.bg.xlf'), 'bg', 'validators');
    $instance->addResource('xlf', ($this->targetDirs[3].'/vendor/symfony/symfony/src/Symfony/Component/Validator/Resources/translations/validators.ca.xlf'), 'ca', 'validators');
    $instance->addResource('xlf', ($this->targetDirs[3].'/vendor/symfony/symfony/src/Symfony/Component/Validator/Resources/translations/validators.cs.xlf'), 'cs', 'validators');
    $instance->addResource('xlf', ($this->targetDirs[3].'/vendor/symfony/symfony/src/Symfony/Component/Validator/Resources/translations/validators.cy.xlf'), 'cy', 'validators');
    // ... hundreds of additional lines
}

The solution

1. We could solve this problem with a technique similar to the templating engines selector:

framework:
    # ...
    templating:
        engines: ['twig', 'php']

In the case of translator:

framework:
    # ...
    translator:
        loaders: ['yml', 'xliff']
        dumpers: ['php']

To maintain backwards compatibility, these options could be null by default, meaning that all loaders/dumpers should be loaded.

2. We could solve this problem defining a new active_locales option to explicitly define the list of locales which will be used in the application:

framework:
    # ...
    default_locale: 'en'
    active_locales: ['es', 'en', 'fr', 'de']

Again, to maintain backwards compatibility, this new option would default to null to load all resources for all locales.

The benchmark

In my tests, I modified the generated container commenting/removing the lines that instantiate all the unused loaders/dumpers and I also removed all the translation resources files except the ones for the active locale. A quick before/after comparison made with Blackfire gave me around 10% performance increase in CPU time, 2.64% memory consumption reduction and about 600 less PHP function calls.

The text was updated successfully, but these errors were encountered:

aitboudad · 2015-03-17T10:21:03Z

great thoughts so I'll work on it :)
just FYI All resources for all locales are loaded was already fixed by bf5c583

mpdude · 2015-04-02T19:34:02Z

Are you sure that just adding the loaders/dumpers/resources is a problem in itself?

At least when it comes to memory usage, I'd guess that the registration alone does not (much) harm.

However, once you need a single message from a single domain, the entire catalogue for that locale (comprising all message domains) will be loaded – and possibly the catalogues for all fallback domains as well.

mpdude · 2015-04-07T22:12:25Z

#14265 might be interesting.

aitboudad · 2015-04-07T23:42:49Z

As @mpdude say previously adding the loaders/dumpers/resources or even limiting loaded resources active_locales doesn't solve the issue.

In 2.7 we made many improvement I'd prefer to list them here:

mpdude · 2015-04-08T11:46:43Z

The tricky part probably is getting realistic scenarios that we can evaluate the optimizations against.

I mean, for every change or tweak there will be setups where the change brings a big improvement, does not significantly change things or is even detrimental.

Possibly influential factors are:

The number of locales in total
The number of different locales used within a single request
The number of fallback locales
The size of the message catalogue for a particular domain and locale (without fallback merges)
The number of domains
The probability that a given message is not present in the catalogue and that we need to fall back to the next level

How can we figure out realistic numbers and come up with a "reference" dataset that we agree on for optimizations?

jpauli · 2015-04-28T10:03:38Z

Hi,

I come back to you after having profiled, with BlackFire, a very huge app.

This app uses 2.6 , and the Translator service is pointed as beeing very slow, for every request.

The addResource() method is called 2100 times, and this one calls for preg_match(), 2100 times so.
This is true overkill in perfs, as it happens in every single HTTP request.

I saw that 2.7 tried to fix that.
The fix is good, as it trashes away the 2100 addResource() calls, however, it still passes every catalogue to the service constructor, into a big PHP array.
For the app I target, that means creating a 2100 entry array, which is obviously better than calling 2100 times the addResource() method, but still not "perfect" for performances (creating a huge PHP array takes time as well).

Isnt there a way to simply remove the load of the catalogues, knowing that anyway, in prod, every translation/catalogue is cached (and if not, you then failed your deploy) ?

aitboudad · 2015-04-28T10:17:05Z

@jpauli see #13986

jpauli · 2015-04-28T10:35:25Z

@aitboudad thx

marcosdsanchez · 2015-08-11T14:58:14Z

Guys, I think that this is a must. In our application, we have a lot of translation files, with thousands of strings to translate, but we support the translation of the application in just a few languages for now.

I created a compiler pass that removes the resources for locales that we don't support and we got a 3 minute decrease in our build times. I think that a lot of applications can benefit with something like this.

javiereguiluz · 2015-10-05T10:52:40Z

I'm closing this issue because it's too generic and proposes no specific solutions. @aitboudad is taking good care of this and a lot of improvements have already been implemented.

aitboudad added the Translation label Mar 19, 2015

aitboudad mentioned this issue Mar 19, 2015

[2.7][Translation] added message cache. #13986

Closed

2 tasks

mpdude mentioned this issue Apr 8, 2015

[RFC] Translator tweaks #14265

Closed

mpdude mentioned this issue Sep 24, 2015

[Translation] Remove TranslatorBagInterface to allow for optimized caching in 3.0 #14530

Closed

aitboudad mentioned this issue Sep 30, 2015

[2.8][Translation] added DoctrineMessageCatalogue. #16036

Closed

javiereguiluz closed this as completed Oct 5, 2015

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Some ideas to improve performance of the Translator component #13948

Some ideas to improve performance of the Translator component #13948

javiereguiluz commented Mar 17, 2015

aitboudad commented Mar 17, 2015

mpdude commented Apr 2, 2015

mpdude commented Apr 7, 2015

aitboudad commented Apr 7, 2015

mpdude commented Apr 8, 2015

jpauli commented Apr 28, 2015

aitboudad commented Apr 28, 2015

jpauli commented Apr 28, 2015

marcosdsanchez commented Aug 11, 2015

javiereguiluz commented Oct 5, 2015

Some ideas to improve performance of the Translator component #13948

Some ideas to improve performance of the Translator component #13948

Comments

javiereguiluz commented Mar 17, 2015

The problem

The solution

The benchmark

aitboudad commented Mar 17, 2015

mpdude commented Apr 2, 2015

mpdude commented Apr 7, 2015

aitboudad commented Apr 7, 2015

mpdude commented Apr 8, 2015

jpauli commented Apr 28, 2015

aitboudad commented Apr 28, 2015

jpauli commented Apr 28, 2015

marcosdsanchez commented Aug 11, 2015

javiereguiluz commented Oct 5, 2015