[Config] Fix slow service discovery for large excluded directories #28200

gonzalovilaseca · 2018-08-14T09:43:20Z

Q	A
Branch?	master
Bug fix?	no
New feature?	no
BC breaks?	no
Deprecations?	no
Tests pass?	no
Fixed tickets	#26736
License	MIT
Doc PR

Not sure if this is a bug fix or not, is more an improvement.
Please for all detail follow the conversation here:
#26736

stloyd · 2018-08-14T09:49:23Z

src/Symfony/Component/Config/Resource/GlobResource.php

@@ -95,6 +99,13 @@ public function getIterator()

        if (0 !== strpos($this->prefix, 'phar://') && false === strpos($this->pattern, '/**/') && (\defined('GLOB_BRACE') || false === strpos($this->pattern, '{'))) {
            foreach (glob($this->prefix.$this->pattern, \defined('GLOB_BRACE') ? GLOB_BRACE : 0) as $path) {
+                if (is_dir($path) && $this->forExclusion) {


if (is_dir($path)) { if ($this->forExclusion) { yield $path => new \SplFileInfo($path); continue; } if (array_key_exists($path, $this->excludedPrefixes)) { continue; } }

stof · 2018-08-14T10:07:48Z

src/Symfony/Component/Config/Resource/GlobResource.php

+                        continue;
+                    }
+
+                    if (array_key_exists($path, $this->excludedPrefixes)) {


this requires normalizing the directory separators in the path before using it as a key, as done in FileLoader

stof · 2018-08-14T10:08:55Z

src/Symfony/Component/Config/Tests/Resource/GlobResourceTest.php

+    public function testIteratorSkipsFoldersForGivenExcludedPrefixes()
+    {
+        $dir = \dirname(__DIR__).\DIRECTORY_SEPARATOR.'Fixtures';
+        $resource = new GlobResource($dir, '/*Exclude*', true, false, array($dir.\DIRECTORY_SEPARATOR.'Exclude' => true));


please also add a test using / in the exclude path rather than \DIRECTORY_SEPARATOR, to ensure it works. It will be quite common for the config files to use / all the time (much easier in XML or YAML)

nicolas-grekas

Thanks for working on this!
Tests are failing right now, could you have a look?

nicolas-grekas · 2018-08-14T15:57:17Z

src/Symfony/Component/Config/Resource/GlobResource.php

+                        continue;
+                    }
+
+                    if (array_key_exists(str_replace('\\', '/', $path), $this->excludedPrefixes)) {


Is the str_replace needed for Windows? If yes, I'd suggest to use str_replace(\DIRECTORY_SEPARATOR, '/', $path) instead.

Check @stof comment #28200 (comment)
So I just copied how it's done in https://github.com/symfony/dependency-injection/blob/master/Loader/FileLoader.php#L117

Do you still want me to change it?

yes please, \ is a legal character in a path on Linux (even though unusual of course)
that line could be changed also for sure

Ok, should I also do another PR to change https://github.com/symfony/dependency-injection/blob/master/Loader/FileLoader.php#L117 too?

if you're up to yes :) please check the full code base in case there are any other places that need such change (on branch 2.8 since it's the lowest maintained branch)

gonzalovilaseca · 2018-08-14T16:08:07Z

Tests are fine in unix, will check on windows.

nicolas-grekas · 2018-08-14T16:09:53Z

Tests are fine in unix, will check on windows.

Not on Travis, see e.g. https://travis-ci.org/symfony/symfony/jobs/415858572

gonzalovilaseca · 2018-08-14T16:14:41Z

It's complaining about something I haven't changed PHP Fatal error: Class 'Symfony\Component\DependencyInjection\Tests\Fixtures\Prototype\BadClasses\MissingClass' not found in /home/travis/build/symfony/symfony/src/Symfony/Component/DependencyInjection/Tests/Fixtures/Prototype/BadClasses/MissingParent.php on line 5
Will check anyway but might be a glitch.

nicolas-grekas · 2018-08-14T16:17:55Z

This test is directly related actually. We need to figure out why and how. But that's not a glitch nor a transient test. Only this PR has it for now.

fabpot · 2018-09-04T17:01:58Z

@gonzalovilaseca Do you need help here?

gonzalovilaseca · 2018-09-04T19:32:50Z

@fabpot Sorry, got distracted with other friends projects. I'm resuming this today, managed to debug it but haven't found the issue yet. Will ask for help if I can't fix it in the following days.
Thanks though!

gonzalovilaseca · 2018-09-29T20:45:22Z

PR updated!

nicolas-grekas · 2018-09-29T22:10:44Z

src/Symfony/Component/Config/Resource/GlobResource.php

@@ -95,11 +99,27 @@ public function getIterator()

        if (0 !== strpos($this->prefix, 'phar://') && false === strpos($this->pattern, '/**/') && (\defined('GLOB_BRACE') || false === strpos($this->pattern, '{'))) {
            foreach (glob($this->prefix.$this->pattern, \defined('GLOB_BRACE') ? GLOB_BRACE : 0) as $path) {
+                if (is_dir($path)) {
+                    if ($this->forExclusion && \in_array($path, glob($this->prefix.$this->pattern, GLOB_BRACE | GLOB_ONLYDIR))) {


GLOB_BRACE should be use conditionally as done just above
in_array should also be provided with its 3rd argument (true = strict)

What's the deal with GLOB_BRACE? Is it because The GLOB_BRACE flag is not available on some non GNU systems, like Solaris. ?

But then, shouldn't we throw an error message if the { is in the pattern and GLOB_BRACE is not defined?
Otherwise it won't behave as expected and user won't be aware of it.

That's already dealt with, see L100. But this also means we might have forgotten the case below this "if" that falls back to using Finder. Does it need to know about excluded paths also?

Initially it wasn't aware of excluded prefixes, but then it made some tests fail eg:
src/Symfony/Component/DependencyInjection/Tests/Loader/YamlFileLoaderTest.php for testPrototype.
In that test '../Prototype/{OtherDir,BadClasses}' is excluded, so on first foreach loop as src/Symfony/Component/DependencyInjection/Tests/Fixtures/Prototype is not in excluded prefixes, it gets to line 113, so anything under that folder is not longer parsed by the initial glob foreach, but by the recursive iterator.
It's taken me quite a lot of time to figure all this out, in fact this class is getting quite complex, so maybe it's worth a try to refactor it into something easier to follow?

nicolas-grekas · 2018-09-29T22:13:12Z

src/Symfony/Component/Config/Resource/GlobResource.php

@@ -132,7 +152,7 @@ function (\SplFileInfo $file) { return '.' !== $file->getBasename()[0]; }

        $prefixLen = \strlen($this->prefix);
        foreach ($finder->followLinks()->sortByName()->in($this->prefix) as $path => $info) {
-            if (preg_match($regex, substr('\\' === \DIRECTORY_SEPARATOR ? str_replace('\\', '/', $path) : $path, $prefixLen)) && $info->isFile()) {
+            if (\preg_match($regex, \substr('\\' === \DIRECTORY_SEPARATOR ? \str_replace('\\', '/', $path) : $path, $prefixLen)) && $info->isFile()) {


changes to be reverted

nicolas-grekas · 2018-09-30T07:01:28Z

src/Symfony/Component/Config/Resource/GlobResource.php

@@ -95,11 +99,27 @@ public function getIterator()

        if (0 !== strpos($this->prefix, 'phar://') && false === strpos($this->pattern, '/**/') && (\defined('GLOB_BRACE') || false === strpos($this->pattern, '{'))) {
            foreach (glob($this->prefix.$this->pattern, \defined('GLOB_BRACE') ? GLOB_BRACE : 0) as $path) {
+                if (is_dir($path)) {
+                    if ($this->forExclusion && \in_array($path, glob($this->prefix.$this->pattern, (\defined('GLOB_BRACE') ? GLOB_BRACE : 0) | GLOB_ONLYDIR), true)) {


Shouldn't this glob be moved out of the foreach ?

ostrolucky · 2018-10-01T19:50:05Z

src/Symfony/Component/Config/Resource/GlobResource.php

+                            continue;
+                        }
+
+                        if (array_key_exists(str_replace(\DIRECTORY_SEPARATOR, '/', $path), $this->excludedPrefixes)) {


You can replace these with issets, will be faster. Map of values has been reversed (and true used for value) for precisely this reason.

OR we can use \in_array which has been optimized into opcode in PHP 7.2 already and is even faster than isset, but you need to reverse the map. It's good opportunity to do this now, because as you added that array into constructor of this class, we won't be able to reverse it later without breaking BC. Obvious con is it will make code slightly slower in PHP 7.1. But IMO worth it, as it's only one PHP version and all future versions will be faster with in_array, plus it's more readable (intent is more obvious). WDYT @nicolas-grekas ?

in_array cannot be faster than isset, because it still has to do a table scan vs an O(1) hashmap lookup for isset, isn't it?

It will reverse it internally, see https://phpinternals.net/articles/optimising_internal_functions_via_new_opcode_instructions#the-in_array-function

gonzalovilaseca · 2018-10-01T20:20:59Z

I’m happy to change it to what both of you decide

gonzalovilaseca · 2018-10-11T20:05:32Z

@nicolas-grekas @stof Can you have nother look please?

nicolas-grekas · 2018-10-11T21:14:20Z

src/Symfony/Component/Config/Resource/GlobResource.php

+                                }
+                            ),
+                            \RecursiveIteratorIterator::LEAVES_ONLY
+                        ));


this whole block has one extra indentation level that should be removed.

nicolas-grekas

(Thanks! FYI, I push-forced some changes on your fork)

gonzalovilaseca · 2018-10-11T22:43:39Z

Thxs!

nicolas-grekas · 2018-10-23T07:20:04Z

Thank you @gonzalovilaseca.

…irectories (gonzalovilaseca) This PR was squashed before being merged into the 4.2-dev branch (closes #28200). Discussion ---------- [Config] Fix slow service discovery for large excluded directories | Q | A | ------------- | --- | Branch? | master | Bug fix? | no | New feature? | no | BC breaks? | no | Deprecations? | no | Tests pass? | no | Fixed tickets | #26736 | License | MIT | Doc PR | Not sure if this is a bug fix or not, is more an improvement. Please for all detail follow the conversation here: #26736 Commits ------- fa731e5 [Config] Fix slow service discovery for large excluded directories

@gonzalovilaseca

This PR was squashed before being merged into the 4.2-dev branch (closes #29062). Discussion ---------- Fix GlobResource serialization | Q | A | ------------- | --- | Branch? | master  | Bug fix? | yes | New feature? | no  | BC breaks? | no  | Deprecations? | no  | Tests pass? | no  | Fixed tickets | #28200  | License | MIT | Doc PR |  Since @gonzalovilaseca improvement in PR #28200, I noticed that the container is being compiled systematically. This is caused by the two added properties (`forExclusion`, `excludedPrefixes`) not being serialized, and lead to wrong hash computing. I updated the `serialize` and `unserialize` methods in this PR. ping @nicolas-grekas Commits ------- 6ce7f07 Fix GlobResource serialization

…-grekas) This PR was merged into the 4.2 branch. Discussion ---------- [Config] fix path exclusion during glob discovery | Q | A | ------------- | --- | Branch? | 4.2 | Bug fix? | yes | New feature? | - | BC breaks? | no | Deprecations? | no | Tests pass? | yes | Fixed tickets | - | License | MIT | Doc PR | - Something we missed in #28200 - reported on Slack. Commits ------- 4ada4dc [Config] fix path exclusion during glob discovery

carsonbot added the Status: Needs Review label Aug 14, 2018

stloyd suggested changes Aug 14, 2018

View reviewed changes

carsonbot added Status: Needs Work and removed Status: Needs Review labels Aug 14, 2018

stof requested changes Aug 14, 2018

View reviewed changes

nicolas-grekas requested changes Aug 14, 2018

View reviewed changes

nicolas-grekas added this to the next milestone Aug 19, 2018

nicolas-grekas reviewed Sep 29, 2018

View reviewed changes

nicolas-grekas reviewed Sep 30, 2018

View reviewed changes

xabbuh added DependencyInjection Performance labels Oct 1, 2018

ostrolucky suggested changes Oct 1, 2018

View reviewed changes

nicolas-grekas reviewed Oct 11, 2018

View reviewed changes

nicolas-grekas changed the title ~~Autowiring is very slow for large excluded directories~~ [Config] Fix slow service discovery for large excluded directories Oct 11, 2018

nicolas-grekas approved these changes Oct 11, 2018

View reviewed changes

carsonbot added Status: Reviewed and removed Status: Needs Work labels Oct 11, 2018

nicolas-grekas modified the milestones: next, 4.2 Oct 20, 2018

dunglas approved these changes Oct 23, 2018

View reviewed changes

stloyd approved these changes Oct 23, 2018

View reviewed changes

[Config] Fix slow service discovery for large excluded directories

fa731e5

nicolas-grekas merged commit fa731e5 into symfony:master Oct 23, 2018

alexismarquis mentioned this pull request Nov 2, 2018

Fix GlobResource serialization #29062

Merged

nicolas-grekas mentioned this pull request Nov 30, 2018

[Config] fix path exclusion during glob discovery #29394

Merged

Uh oh!

[Config] Fix slow service discovery for large excluded directories #28200

[Config] Fix slow service discovery for large excluded directories #28200

Uh oh!

Conversation

gonzalovilaseca commented Aug 14, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

stof Aug 14, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

nicolas-grekas left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gonzalovilaseca Aug 14, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gonzalovilaseca commented Aug 14, 2018

Uh oh!

nicolas-grekas commented Aug 14, 2018

Uh oh!

gonzalovilaseca commented Aug 14, 2018

Uh oh!

nicolas-grekas commented Aug 14, 2018

Uh oh!

fabpot commented Sep 4, 2018

Uh oh!

gonzalovilaseca commented Sep 4, 2018

Uh oh!

gonzalovilaseca commented Sep 29, 2018

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

nicolas-grekas Sep 30, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gonzalovilaseca Sep 30, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gonzalovilaseca commented Oct 1, 2018

Uh oh!

gonzalovilaseca commented Oct 11, 2018

Uh oh!

Choose a reason for hiding this comment

Uh oh!

nicolas-grekas left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

gonzalovilaseca commented Aug 14, 2018 •

edited

Loading

stof Aug 14, 2018 •

edited

Loading

gonzalovilaseca Aug 14, 2018 •

edited

Loading

nicolas-grekas Sep 30, 2018 •

edited

Loading

gonzalovilaseca Sep 30, 2018 •

edited

Loading

nicolas-grekas left a comment •

edited

Loading