-
-
Notifications
You must be signed in to change notification settings - Fork 9.6k
[VarDumper] Dont use Stub objects for arrays - lower GC pressure #23644
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
d030c4e
to
5cd7a75
Compare
switch ($zval['type']) { | ||
case 'string': | ||
if (isset($v[0]) && !preg_match('//u', $v)) { | ||
switch (true) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Using root namespaced \is_*()
functions is noticeably faster than comparing the string result of gettype()
on PHP 7. Since this is a hot path, I've also added a \
to all root functions in the loop.
a3be3f4
to
985a11a
Compare
$r = new \ReflectionClass(Stub::class); | ||
$stubConstIndexes = array_flip(array_values($r->getConstants())); | ||
$stubConstValues = array_flip($stubConstIndexes); | ||
if (!$item || $item instanceof Stub || !is_array($item)) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
should we fully-qualify it here as well or no ?
$r = new \ReflectionClass(Stub::class); | ||
$stubConstIndexes = array_flip(array_values($r->getConstants())); | ||
$stubConstValues = array_flip($stubConstIndexes); | ||
if (!$item || $item instanceof Stub || !is_array($item)) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
$item instanceof Stub
is useless here. It is covered by !is_array($item)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
prefix added, and instanceof check removed, thanks
8b37388
to
17ea931
Compare
@@ -279,57 +279,6 @@ public function dump(DumperInterface $dumper) | |||
} | |||
|
|||
/** | |||
* @internal | |||
*/ | |||
public function serialize() |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
and we neither need implementing Serializable anymore: that adds to consumed CPU cycles, and the real issues (GC & size of serialized string) comes from arrays really.
17ea931
to
654c151
Compare
@@ -16,19 +16,19 @@ | |||
* | |||
* @author Nicolas Grekas <[email protected]> | |||
*/ | |||
class Stub | |||
class Stub implements \Serializable |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
let's implement Serializable on Stub instead of Data
42b8329
to
5db35eb
Compare
5db35eb
to
3ca9a7b
Compare
3ca9a7b
to
0d5012d
Compare
Thank you @nicolas-grekas. |
…essure (nicolas-grekas) This PR was merged into the 3.3 branch. Discussion ---------- [VarDumper] Dont use Stub objects for arrays - lower GC pressure | Q | A | ------------- | --- | Branch? | 3.3 | Bug fix? | yes | New feature? | no | BC breaks? | no | Deprecations? | no | Tests pass? | yes | Fixed tickets | #23644 | License | MIT | Doc PR | - Several recent profiles have shown that VarDumper triggers the garbage collector quite often, leading to high CPU usage. The reason for this is that internally, VarCloner creates one Stub object per array+object+resource. This PR removes the need for Stub objects for each arrays, replacing them with stub arrays. This should almost remove the GC pressure, since the number of Stub objects now has the same magnitude than the number of dumped objects. Meanwhile, this PR removes any use of the `symfony_debug` extension, which is mostly useless anyway. This helps make the code simpler (really :) ), thus helps maintenance (eg merging up to master.) I also changed the values of the constants defined in the Stub class, and removed the corresponding Data::mapStubConsts() method. Since the serialized format has changed (and we have to do it as there is no other way to fix this GC issue), there is no need to keep any sort of compat mapping there. Commits ------- 0d5012d [VarDumper] Dont use Stub objects for arrays
…grekas) This PR was merged into the 3.3 branch. Discussion ---------- [VarDumper] Keep and reuse array stubs in memory | Q | A | ------------- | --- | Branch? | 3.3 | Bug fix? | yes | New feature? | no | BC breaks? | no | Deprecations? | no | Tests pass? | yes | Fixed tickets | - | License | MIT | Doc PR | - (to be reviewed [ignoring whitespaces](https://github.com/symfony/symfony/pull/23683/files?w=1)) As highlighted by @vtsykun in #23620, the patch in #23644 improves performance but increases the memory usage. The issue is that even a small `array($k => $v)` consumes more memory than a `new Stub()`. That's a shame, but since our small arrays have a low variety, we can keep them in a cache and leverage php's COW mechanism to reduce memory. Effectively, this creates a memory leak. But the leak is upper bounded by the data you had already in memory, since that's what is cloned. Looking at the numbers, it looks worth it: | | 3.3.5 | +#23644 | +this PR | --- | --- | --- | --- | Wall Time | 39.4s | 26.1s | ~~18.6s~~ 17.3s | Memory | 391MB | 539MB | ~~217MB~~ 216MB https://blackfire.io/profiles/compare/846b58bc-7863-4502-9ca2-f82eebd4173f/graph Commits ------- 92fa55d [VarDumper] Keep and reuse array stubs in memory
…fferent sub-requests (vtsykun) This PR was merged into the 4.1-dev branch. Discussion ---------- [HttpKernel] LoggerDataCollector: splitting logs on different sub-requests | Q | A | ------------- | --- | Branch? | 4.1 | Bug fix? | yes | New feature? | no | BC breaks? | no | Deprecations? | no | Tests pass? | - | Fixed tickets | #23620 | License | MIT | Doc PR | - This PR fixed performance problem in dev mode and prevent logs serialize several times for each sub-request. STR: 1) Install any symfony application v2.8-3.4 2) Create a simple page with 20-50 sub-requests and 5k-10k logs 3) Open page in dev mode **Actual** - debug toolbar not open (404 not found) - sets of logs (in the logger debug panel) for different requests are same - log processing takes about 20-30s - gc also run about 50% of execution time **Expected** - Debug toolbar should be open - As developer, I want to see in logger panel only those logs that are relevant to the given sub-request ### Profile and performance Tested on page with 7k logs and 70 sub-request. Comparison: - v3.3.5 - this patch: https://blackfire.io/profiles/compare/b1d410b3-c4a7-4778-9b6d-514f72193e28/graph - v3.3.5 - patch #23644 https://blackfire.io/profiles/compare/d0715b04-7834-4981-8da2-9f1dcb2ef90c/graph Commits ------- d0cb1de [HttpKernel] LoggerDataCollector: splitting logs on different sub-requests
Several recent profiles have shown that VarDumper triggers the garbage collector quite often, leading to high CPU usage. The reason for this is that internally, VarCloner creates one Stub object per array+object+resource.
This PR removes the need for Stub objects for each arrays, replacing them with stub arrays. This should almost remove the GC pressure, since the number of Stub objects now has the same magnitude than the number of dumped objects.
Meanwhile, this PR removes any use of the
symfony_debug
extension, which is mostly useless anyway. This helps make the code simpler (really :) ), thus helps maintenance (eg merging up to master.)I also changed the values of the constants defined in the Stub class, and removed the corresponding Data::mapStubConsts() method. Since the serialized format has changed (and we have to do it as there is no other way to fix this GC issue), there is no need to keep any sort of compat mapping there.