[VarDumper] Dont use Stub objects for arrays - lower GC pressure #23644
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
d030c4e to
5cd7a75
Compare
nicolas-grekas
commented
Jul 24, 2017
| switch ($zval['type']) { | ||
| case 'string': | ||
| if (isset($v[0]) && !preg_match('//u', $v)) { | ||
| switch (true) { |
There was a problem hiding this comment.
Using root namespaced \is_*() functions is noticeably faster than comparing the string result of gettype() on PHP 7. Since this is a hot path, I've also added a \ to all root functions in the loop.
a3be3f4 to
985a11a
Compare
stof
reviewed
Jul 24, 2017
| $r = new \ReflectionClass(Stub::class); | ||
| $stubConstIndexes = array_flip(array_values($r->getConstants())); | ||
| $stubConstValues = array_flip($stubConstIndexes); | ||
| if (!$item || $item instanceof Stub || !is_array($item)) { |
There was a problem hiding this comment.
should we fully-qualify it here as well or no ?
| $r = new \ReflectionClass(Stub::class); | ||
| $stubConstIndexes = array_flip(array_values($r->getConstants())); | ||
| $stubConstValues = array_flip($stubConstIndexes); | ||
| if (!$item || $item instanceof Stub || !is_array($item)) { |
There was a problem hiding this comment.
$item instanceof Stub is useless here. It is covered by !is_array($item)
There was a problem hiding this comment.
prefix added, and instanceof check removed, thanks
8b37388 to
17ea931
Compare
nicolas-grekas
commented
Jul 24, 2017
| @@ -279,57 +279,6 @@ public function dump(DumperInterface $dumper) | |||
| } | |||
|
|
|||
| /** | |||
| * @internal | |||
| */ | |||
| public function serialize() | |||
There was a problem hiding this comment.
and we neither need implementing Serializable anymore: that adds to consumed CPU cycles, and the real issues (GC & size of serialized string) comes from arrays really.
17ea931 to
654c151
Compare
nicolas-grekas
commented
Jul 24, 2017
| @@ -16,19 +16,19 @@ | |||
| * | |||
| * @author Nicolas Grekas <[email protected]> | |||
| */ | |||
| class Stub | |||
| class Stub implements \Serializable | |||
There was a problem hiding this comment.
let's implement Serializable on Stub instead of Data
42b8329 to
5db35eb
Compare
5db35eb to
3ca9a7b
Compare
3ca9a7b to
0d5012d
Compare
fabpot
approved these changes
Jul 26, 2017
|
Thank you @nicolas-grekas. |
fabpot
added a commit
that referenced
this pull request
Jul 26, 2017
…essure (nicolas-grekas) This PR was merged into the 3.3 branch. Discussion ---------- [VarDumper] Dont use Stub objects for arrays - lower GC pressure | Q | A | ------------- | --- | Branch? | 3.3 | Bug fix? | yes | New feature? | no | BC breaks? | no | Deprecations? | no | Tests pass? | yes | Fixed tickets | #23644 | License | MIT | Doc PR | - Several recent profiles have shown that VarDumper triggers the garbage collector quite often, leading to high CPU usage. The reason for this is that internally, VarCloner creates one Stub object per array+object+resource. This PR removes the need for Stub objects for each arrays, replacing them with stub arrays. This should almost remove the GC pressure, since the number of Stub objects now has the same magnitude than the number of dumped objects. Meanwhile, this PR removes any use of the `symfony_debug` extension, which is mostly useless anyway. This helps make the code simpler (really :) ), thus helps maintenance (eg merging up to master.) I also changed the values of the constants defined in the Stub class, and removed the corresponding Data::mapStubConsts() method. Since the serialized format has changed (and we have to do it as there is no other way to fix this GC issue), there is no need to keep any sort of compat mapping there. Commits ------- 0d5012d [VarDumper] Dont use Stub objects for arrays
fabpot
added a commit
that referenced
this pull request
Jul 28, 2017
…grekas) This PR was merged into the 3.3 branch. Discussion ---------- [VarDumper] Keep and reuse array stubs in memory | Q | A | ------------- | --- | Branch? | 3.3 | Bug fix? | yes | New feature? | no | BC breaks? | no | Deprecations? | no | Tests pass? | yes | Fixed tickets | - | License | MIT | Doc PR | - (to be reviewed [ignoring whitespaces](https://github.com/symfony/symfony/pull/23683/files?w=1)) As highlighted by @vtsykun in #23620, the patch in #23644 improves performance but increases the memory usage. The issue is that even a small `array($k => $v)` consumes more memory than a `new Stub()`. That's a shame, but since our small arrays have a low variety, we can keep them in a cache and leverage php's COW mechanism to reduce memory. Effectively, this creates a memory leak. But the leak is upper bounded by the data you had already in memory, since that's what is cloned. Looking at the numbers, it looks worth it: | | 3.3.5 | +#23644 | +this PR | --- | --- | --- | --- | Wall Time | 39.4s | 26.1s | ~~18.6s~~ 17.3s | Memory | 391MB | 539MB | ~~217MB~~ 216MB https://blackfire.io/profiles/compare/846b58bc-7863-4502-9ca2-f82eebd4173f/graph Commits ------- 92fa55d [VarDumper] Keep and reuse array stubs in memory
Merged
nicolas-grekas
added a commit
that referenced
this pull request
Apr 20, 2018
…fferent sub-requests (vtsykun) This PR was merged into the 4.1-dev branch. Discussion ---------- [HttpKernel] LoggerDataCollector: splitting logs on different sub-requests | Q | A | ------------- | --- | Branch? | 4.1 | Bug fix? | yes | New feature? | no | BC breaks? | no | Deprecations? | no | Tests pass? | - | Fixed tickets | #23620 | License | MIT | Doc PR | - This PR fixed performance problem in dev mode and prevent logs serialize several times for each sub-request. STR: 1) Install any symfony application v2.8-3.4 2) Create a simple page with 20-50 sub-requests and 5k-10k logs 3) Open page in dev mode **Actual** - debug toolbar not open (404 not found) - sets of logs (in the logger debug panel) for different requests are same - log processing takes about 20-30s - gc also run about 50% of execution time **Expected** - Debug toolbar should be open - As developer, I want to see in logger panel only those logs that are relevant to the given sub-request ### Profile and performance Tested on page with 7k logs and 70 sub-request. Comparison: - v3.3.5 - this patch: https://blackfire.io/profiles/compare/b1d410b3-c4a7-4778-9b6d-514f72193e28/graph - v3.3.5 - patch #23644 https://blackfire.io/profiles/compare/d0715b04-7834-4981-8da2-9f1dcb2ef90c/graph Commits ------- d0cb1de [HttpKernel] LoggerDataCollector: splitting logs on different sub-requests
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Several recent profiles have shown that VarDumper triggers the garbage collector quite often, leading to high CPU usage. The reason for this is that internally, VarCloner creates one Stub object per array+object+resource.
This PR removes the need for Stub objects for each arrays, replacing them with stub arrays. This should almost remove the GC pressure, since the number of Stub objects now has the same magnitude than the number of dumped objects.
Meanwhile, this PR removes any use of the
symfony_debugextension, which is mostly useless anyway. This helps make the code simpler (really :) ), thus helps maintenance (eg merging up to master.)I also changed the values of the constants defined in the Stub class, and removed the corresponding Data::mapStubConsts() method. Since the serialized format has changed (and we have to do it as there is no other way to fix this GC issue), there is no need to keep any sort of compat mapping there.