[Form] [DoctrineBridge] optimized LazyChoiceList and DoctrineChoiceLoader #18359

HeahDude · 2016-03-30T08:47:32Z

Q	A
Branch?	master
Bug fix?	no
New feature?	no
BC breaks?	no
Deprecations?	yes
Tests pass?	yes
Fixed tickets	~
License	MIT
Doc PR	~

Problem

Actually we got a circular dependency:

Object	Return
`DefaultChoiceListFactory::createListFromLoader()` (decorated)	`LazyChoiceList` with loader and resolved `$value`
`LazyChoiceList::get*`	`DoctrineChoiceLoader` with resolved `$value`
`DoctrineChoiceLoader::loadChoiceList()`	(decorated) `DefaultChoiceListFactory` with loaded choices and resolved `$value`
`DefaultChoiceListFactory::createListFromChoices()`	`ArrayChoiceList` with `resolved`$value`

With this refactoring, the DoctrineChoiceLoader is no longer dependant to the factory, the ChoiceLoaderInterface::loadChoiceList() must return a ChoiceListInterface but should not need a decorated factory while $value is already resolved. It should remain lazy IMHO.

Solution

Object	Return
`DefaultChoiceListFactory::createListFromLoader()`	`LazyChoiceList` with loader and resolved `$value`
`LazyChoiceList::get*()`	`DoctrineChoiceLoader` with resolved `$value`
`DoctrineChoiceLoader::loadChoiceList()`	`ArrayChoiceList` with resolved `$value`.

Since choiceListFactory is a private property, this change should be safe regarding BC.

To justify this change, I've made some blackfire profiling.
You can see my branch of SE here and the all in file test implementation.

Basically it loads a form with 3 EntityType fields with different classes holding 50 instances each.

(INIT events are profiled with an empty cache)

When	What	Diff (SE => PR)
INIT (1)	build form (load types)	see
INIT (2)	build view (load choices)	see
CACHED	build form (load types)	see
CACHED	build view (load choices)	see
SUBMIT	build form (load types)	see
SUBMIT	handle request (load choices)	see
SUBMIT	build view (load values)	see

(1):

(2):

It can seem like 1 and 2 balance each other but it comes clear when comparing values:

-	Build form	Build view
wall time	-88 ms	+9.71 ms
blocking I/O	-40 ms	+3.67 ms
cpu	-48 ms	+13.4 ms
memory	-4.03 MB	+236 kB
network	-203 B	+2.21 kB

linaori · 2016-03-30T09:17:49Z

src/Symfony/Bridge/Doctrine/Form/ChoiceList/DoctrineChoiceLoader.php

+        if (null !== $this->factory) {
+            @trigger_error(sprintf('Passing a ChoiceListFactoryInterface to %s is deprecated since version 3.1 and will no longer be supported in 4.0. You should either call "%s::loadChoiceList" or override it to return a ChoiceListInterface.', __CLASS__, __CLASS__));
+            $this->factory = $factory;
+        }


This deprecation means that in the future versions it will always have to be called with null as first argument. It's a pain to move the argument to the end of the list but I think that's the only solution you can do for 4.0

The proper deprecation way would be to allow moving all args to the left, not to allow passing null as same argument (i.e. allowing using the right Symfony 4 API)

Ok will do that, thanks guys for the review :)

HeahDude · 2016-03-30T14:41:38Z

Comments addressed.

HeahDude · 2016-03-30T14:49:41Z

Actually the more relevant of all the profile is cached app and form types loading profile 3:

HeahDude · 2016-03-30T14:50:55Z

It should also be the most used (just on display) with a warmup cache

HeahDude · 2016-03-30T18:35:54Z

This might be a first step to do better but can be improved even more IMHO, so let's do this for 3.2.

Status: Needs Work

webmozart · 2016-03-31T06:42:58Z

src/Symfony/Bridge/Doctrine/Form/ChoiceList/DoctrineChoiceLoader.php

     * @param null|EntityLoaderInterface $objectLoader The objects loader
     */
-    public function __construct(ChoiceListFactoryInterface $factory, ObjectManager $manager, $class, IdReader $idReader = null, EntityLoaderInterface $objectLoader = null)
+    public function __construct($manager, $class, $idReader = null, $objectLoader = null, $factory = null)


This is a BC break. We cannot reorder the arguments of the constructor.

Ah I just realized what you are doing below.

Yes I'm just feel bad about removing the type hints but @iltar and @stof were suggesting that change for good reasons I guess.

HeahDude · 2016-03-31T06:53:40Z

Should it be done for 3.1 ?

webmozart · 2016-03-31T06:56:14Z

src/Symfony/Bridge/Doctrine/Form/ChoiceList/DoctrineChoiceLoader.php

        // Optimize performance in case we have an object loader and
        // a single-field identifier
-        $optimize = null === $value || is_array($value) && $value[0] === $this->idReader;
+        $optimize = is_array($value) && $this->idReader === $value[0];


Why did you remove the null === $value check?

Because I read it as null === $value && is_array($value) 😞

Even if I was the one "fixing" it in the first place and introducing a regression at the same type...

However it's clear to me now that choice_value in entity type should better not be used on big collections to keep this optimisation efficient.

What I think of now is that actually $value is always resolved by factories when stored in the cached LazyChoiceList which passes it to the loader each time.

What about creating another ChoiceListInterface depending on this logic while moving it in a Symfony\Bridge\Doctrine\Form\ChoiceList\Factory\SingleIdFactoryDecorator ?

Do you think it's doable to cache a map id => value when $value is not null somewhere ?

What about possibilities added by the new Cache component ?
ids should be immutable, right ? But their corresponding values may need to be updated but must not change during the lifecycle of a form since we rely on it to select choices.

How to generate a good hash based on the class and $value in that case ? that should not be a problem if $value is a string property path but if it is a closure ?

One way I see this could be implemented is by adding a Symfony\Bridge\Doctrine\Form\ChoiceList\LazyEntityList which would be basically a LazyChoiceList holding the idReader for mapping and using $value as well.

1. Current WorkFlow

Decorated factory creates and stores the LazyChoiceList storing the DoctrineChoiceLoader and $value = null

LazyChoiceList calls the loader and passes null as value.

DoctrineChoiceLoader resolves $value with its idReader and load the choices

DoctrineChoiceLoader calls the decorated factory with resolved value and loaded choices

Decorated factory creates and stores the ArrayChoiceList will never call it back.

DoctrineChoiceLoader stores the ArrayChoiceList and will never call it back.

LazyChoiceList stores the ArrayChoiceList while storing the loader.

Also, if another EntityType is used in the same form, another DoctrineChoiceLoader is created by the DoctrineType and the same process happens again.

When the same em, class and 'query_builderoptions are used for theEntityTypethe same loader is return, so the factory returns the sameLazyChoiceListstoring the loadedArrayChoiceList`.

Workflow a bit improved by this PR

Removes line 5 from above.

3. Ideal Workflow

IdReader decorated factory resolves $value, $idReader and $entityLoader (instead of DoctrineType) then creates a LazyEntityList storing the DoctrineChoiceLoader and the resolved arguments,

LazyEntityList calls the loader and passes it the resolved arguments.

DoctrineChoiceLoader load the choices with optimisation

DoctrineChoiceLoader creates an EntityList with an extra map ids by values

DoctrineChoiceLoader stores the EntityList.

What do you think ? Thanks.

HeahDude · 2016-03-31T07:19:35Z

Comments addressed.

Status: Needs Review

HeahDude · 2016-03-31T12:33:45Z

Should I deprecate the caching of LazyChoiceList in that PR too ? (ref #18332 (comment))

HeahDude · 2016-03-31T12:34:43Z

@javiereguiluz I guess this one could have a "Enhancement" label :)

HeahDude · 2016-04-03T17:08:08Z

I pushed the LazyChoiceList optimization, with both it's even better, here some profiles from previous optimization of the loader to both loader and LazyChoiceList with this new [commit](:

So with both optimizations from current to 4.0 (without BC layer):

Finally profiles with deprecation layer (the final look after that PR):

3 entity fields with 50 entities each:
- no cache
- cache
- submit
1 entity field with 150 entities:
- display
- submit

And that will be even better in 4.0!

Don't forget to look at the real values, they are more impacted than the percentages :)

Note that actually the code as it works by default has not triggered any deprecations for my tests, it should only happen for custom implementations and that's good.

HeahDude · 2016-04-03T17:14:08Z

Failures are unrelated.

javiereguiluz · 2016-04-03T17:17:22Z

@HeahDude nice comparisons 👏 And thanks for using realistic benchmarks (it's "easy" to improve performance when doing a benchmark with 1 million entities ... but that's no real)

HeahDude · 2016-04-03T17:20:26Z

Thanks @javiereguiluz, still I guess it would be nice if someone could test it in a real app too to confirm it does not mess with anything...

HeahDude · 2016-04-25T13:41:46Z

What's the status of this PR? is this still to be in 3.1?

Allied to #18332 we can get a great performance improvement.

I tested it with symfony-demo by quickly adding some EntityType fields in some forms, and it worked like a charm :)

Anyone for a final review?

fabpot · 2016-04-28T10:16:56Z

Thank you @HeahDude.

…octrineChoiceLoader (HeahDude) This PR was merged into the 3.1-dev branch. Discussion ---------- [Form] [DoctrineBridge] optimized LazyChoiceList and DoctrineChoiceLoader | Q | A | ------------- | --- | Branch? | master | Bug fix? | no | New feature? | no | BC breaks? | no | Deprecations? | yes | Tests pass? | yes | Fixed tickets | ~ | License | MIT | Doc PR | ~ Problem ====== Actually we got a circular dependency: | Object | Return |----------|-------------| |`DefaultChoiceListFactory::createListFromLoader()` (decorated) | `LazyChoiceList` with loader and resolved `$value` | `LazyChoiceList::get*` | `DoctrineChoiceLoader` with resolved `$value` `DoctrineChoiceLoader::loadChoiceList()` | (decorated) `DefaultChoiceListFactory` with loaded choices and resolved `$value` `DefaultChoiceListFactory::createListFromChoices()` | `ArrayChoiceList` with `resolved `$value` With this refactoring, the `DoctrineChoiceLoader` is no longer dependant to the factory, the `ChoiceLoaderInterface::loadChoiceList()` must return a `ChoiceListInterface` but should not need a decorated factory while `$value` is already resolved. It should remain lazy IMHO. Solution ====== | Object | Return | |----------|-----------| | `DefaultChoiceListFactory::createListFromLoader()` | `LazyChoiceList` with loader and resolved `$value` | `LazyChoiceList::get*()` | `DoctrineChoiceLoader` with resolved `$value` | `DoctrineChoiceLoader::loadChoiceList()` | `ArrayChoiceList` with resolved `$value`. Since `choiceListFactory` is a private property, this change should be safe regarding BC. To justify this change, I've made some blackfire profiling. You can see my [branch of SE here](https://github.com/HeahDude/symfony-standard/tree/test/optimize-doctrine_choice_loader) and the [all in file test implementation](https://github.com/HeahDude/symfony-standard/blob/test/optimize-doctrine_choice_loader/src/AppBundle/Controller/DefaultController.php). Basically it loads a form with 3 `EntityType` fields with different classes holding 50 instances each. (INIT events are profiled with an empty cache) When | What | Diff (SE => PR) --------|-------|------ INIT (1) | build form (load types) | [see](https://blackfire.io/profiles/compare/061d5d28-15c6-4e01-b8c0-3edc9cb8daf0/graph) INIT (2) | build view (load choices) | [see](https://blackfire.io/profiles/compare/04f142a8-d886-405a-be4d-636ba82d8acd/graph) CACHED | build form (load types) | [see](https://blackfire.io/profiles/compare/293b27b6-aa58-42ae-bafb-655513201505/graph) CACHED | build view (load choices) | [see](https://blackfire.io/profiles/compare/e5b37dfe-cc9e-498f-b98a-7448830ad190/graph) SUBMIT | build form (load types) | [see](https://blackfire.io/profiles/compare/7f3baea9-0d27-46b6-8c24-c577742382dc/graph) SUBMIT | handle request (load choices) | [see](https://blackfire.io/profiles/compare/8644ebfb-4397-495b-8f3d-1a6e1d7f8476/graph) SUBMIT | build view (load values) | [see](https://blackfire.io/profiles/compare/89c3cc7c-ea07-4300-91b3-99004cb58ea1/graph) (1): ![build_form-no_cache](https://cloud.githubusercontent.com/assets/10107633/14136166/b5a85eb8-f661-11e5-8556-3e0dcbfaf404.jpg) (2): ![build_view-no_cache](https://cloud.githubusercontent.com/assets/10107633/14136240/1162f3ee-f662-11e5-834a-1ed1e519dc83.jpg) It can seem like 1 and 2 balance each other but it comes clear when comparing values: | - | Build form | Build view | |-----|---------|--------------| | wall time | -88 ms | +9.71 ms | blocking I/O | -40 ms | +3.67 ms | cpu | -48 ms | +13.4 ms | memory | -4.03 MB | +236 kB | network | -203 B | +2.21 kB Commits ------- 98621f4 [Form] optimized LazyChoiceList 86b2ff1 [DoctrineBridge] optimized DoctrineChoiceLoader

follows symfony#18359.

…(HeahDude) This PR was merged into the 3.1-dev branch. Discussion ---------- [Form] [DoctrineBridge] updated changelogs after #18359 | Q | A | ------------- | --- | Branch? | master | Bug fix? | no | New feature? | no | BC breaks? | no | Deprecations? | no | Tests pass? | yes | Fixed tickets | ~ | License | MIT | Doc PR | ~ follows #18359. Commits ------- 1c8dc9d [Form] [DoctrineBridge] updated changelogs after #18359

This PR was merged into the 3.1 branch. Discussion ---------- [DoctrineBridge] Remove dead code in DoctrineType | Q | A | ------------- | --- | Branch? | 3.1 | Bug fix? | no | New feature? | no | BC breaks? | no (Only the last arguments of a method may be removed [[3]](http://symfony.com/doc/current/contributing/code/bc.html#id13) ) | Deprecations? | no | Tests pass? | yes | Fixed tickets | - | License | MIT | Doc PR | - After optimization of DoctrineChoiceLoader (86b2ff1 in #18359) this code died :) Commits ------- 9b49723 remove dead code

This PR was merged into the 3.1 branch. Discussion ---------- [DoctrineBridge] Remove dead code in DoctrineType | Q | A | ------------- | --- | Branch? | 3.1 | Bug fix? | no | New feature? | no | BC breaks? | no (Only the last arguments of a method may be removed [[3]](http://symfony.com/doc/current/contributing/code/bc.html#id13) ) | Deprecations? | no | Tests pass? | yes | Fixed tickets | - | License | MIT | Doc PR | - After optimization of DoctrineChoiceLoader (symfony/symfony@86b2ff1 in symfony/symfony#18359) this code died :) Commits ------- 9b49723 remove dead code

…n (yceruto) This PR was submitted for the master branch but it was merged into the 3.2 branch instead (closes #21493). Discussion ---------- [DoctrineBridge] Remove dead code in DoctrineOrmExtension | Q | A | ------------- | --- | Branch? | 3.2 | Bug fix? | no | New feature? | no | BC breaks? | no | Deprecations? | no | Tests pass? | yes | License | MIT After optimization in #20312 and #18359 Commits ------- 0cd8bf8 Remove dead code

… (HeahDude) This PR was squashed before being merged into the 4.4 branch. Discussion ---------- [Form] Fix same choice loader with different choice values | Q | A | ------------- | --- | Branch? | 4.4 | Bug fix? | yes | New feature? | no  | Deprecations? | no  | Tickets | #44655 | License | MIT | Doc PR | ~ It appears that deprecating the caching in the `LazyChoiceList` (cf #18359) was a mistake. The bug went under the radar because in practice every choice field has its own loader instance. However, optimizations made in #30994 then revealed the flaw (cf #42206) as the loaders were actually shared across many fields. While working on a fix I ended up implementing something similar to what's proposed in #44655. I'll send a PR for 5.4 as well. Commits ------- 65cbf18 [Form] Fix same choice loader with different choice values

carsonbot added the Status: Needs Review label Mar 30, 2016

linaori reviewed Mar 30, 2016
View reviewed changes

javiereguiluz added Deprecation DoctrineBridge labels Mar 30, 2016

HeahDude force-pushed the minor-optimize-doctrine_choice_loader branch from d082b38 to 6e3fd0b Compare March 30, 2016 14:41

carsonbot added Status: Needs Work and removed Status: Needs Review labels Mar 30, 2016

HeahDude mentioned this pull request Mar 31, 2016

[Form] added CallbackChoiceLoader and refactored ChoiceType's children #18332

Merged

2 tasks

webmozart reviewed Mar 31, 2016
View reviewed changes

HeahDude force-pushed the minor-optimize-doctrine_choice_loader branch 2 times, most recently from 78124ff to e690da1 Compare March 31, 2016 07:18

carsonbot added Status: Needs Review and removed Status: Needs Work labels Mar 31, 2016

HeahDude force-pushed the minor-optimize-doctrine_choice_loader branch from e690da1 to 1c553eb Compare March 31, 2016 08:55

javiereguiluz added the Enhancement label Mar 31, 2016

HeahDude changed the title ~~[RFC] [DoctrineBridge] Optimize DoctrineChoiceLoader~~ [Form] [DoctrineBridge] Optimize LazyChoiceList and DoctrineChoiceLoader Apr 2, 2016

HeahDude changed the title ~~[Form] [DoctrineBridge] Optimize LazyChoiceList and DoctrineChoiceLoader~~ [Form] [DoctrineBridge] optimized LazyChoiceList and DoctrineChoiceLoader Apr 2, 2016

HeahDude force-pushed the minor-optimize-doctrine_choice_loader branch from 1c553eb to 5c9d90a Compare April 3, 2016 17:05

HeahDude added 2 commits April 3, 2016 19:07

[DoctrineBridge] optimized DoctrineChoiceLoader

86b2ff1

[Form] optimized LazyChoiceList

98621f4

HeahDude force-pushed the minor-optimize-doctrine_choice_loader branch from 5c9d90a to 98621f4 Compare April 3, 2016 17:07

fabpot merged commit 98621f4 into symfony:master Apr 28, 2016

HeahDude added a commit to HeahDude/symfony that referenced this pull request Apr 28, 2016

[Form] [DoctrineBridge] updated changelogs after symfony#18359

1392c8f

follows symfony#18359.

HeahDude added a commit to HeahDude/symfony that referenced this pull request Apr 28, 2016

[Form] [DoctrineBridge] updated changelogs after symfony#18359

0618b69

follows symfony#18359.

HeahDude mentioned this pull request Apr 28, 2016

[Form] [DoctrineBridge] updated changelogs after #18359 #18664

Merged

HeahDude added a commit to HeahDude/symfony that referenced this pull request Apr 28, 2016

[Form] [DoctrineBridge] updated changelogs after symfony#18359

7fad426

follows symfony#18359.

HeahDude deleted the minor-optimize-doctrine_choice_loader branch April 28, 2016 17:16

HeahDude added a commit to HeahDude/symfony that referenced this pull request Apr 29, 2016

[Form] [DoctrineBridge] updated changelogs after symfony#18359

1c8dc9d

follows symfony#18359.

fabpot mentioned this pull request May 13, 2016

Release v3.1.0-BETA1 #18776

Merged

yceruto mentioned this pull request Oct 26, 2016

[DoctrineBridge] Remove dead code in DoctrineType #20312

Merged

yceruto mentioned this pull request Feb 1, 2017

[DoctrineBridge] Remove dead code in DoctrineOrmExtension #21493

Closed

HeahDude mentioned this pull request Apr 18, 2022

[Form] Fix same choice loader with different choice values #46098

Merged

Uh oh!

[Form] [DoctrineBridge] optimized LazyChoiceList and DoctrineChoiceLoader #18359

[Form] [DoctrineBridge] optimized LazyChoiceList and DoctrineChoiceLoader #18359

Uh oh!

Conversation

HeahDude commented Mar 30, 2016

Problem

Solution

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

HeahDude commented Mar 30, 2016

Uh oh!

HeahDude commented Mar 30, 2016

Uh oh!

HeahDude commented Mar 30, 2016

Uh oh!

HeahDude commented Mar 30, 2016

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

HeahDude commented Mar 31, 2016

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

1. Current WorkFlow

3. Ideal Workflow

Uh oh!

HeahDude commented Mar 31, 2016

Uh oh!

HeahDude commented Mar 31, 2016

Uh oh!

HeahDude commented Mar 31, 2016

Uh oh!

HeahDude commented Apr 3, 2016

Uh oh!

HeahDude commented Apr 3, 2016

Uh oh!

javiereguiluz commented Apr 3, 2016

Uh oh!

HeahDude commented Apr 3, 2016

Uh oh!

HeahDude commented Apr 25, 2016

Uh oh!

fabpot commented Apr 28, 2016

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants