[HttpKernel] LoggerDataCollector: splitting logs on different sub-requests #23659

vtsykun · 2017-07-24T22:47:55Z

Q	A
Branch?	4.1
Bug fix?	yes
New feature?	no
BC breaks?	no
Deprecations?	no
Tests pass?	-
Fixed tickets	#23620
License	MIT
Doc PR	-

This PR fixed performance problem in dev mode and prevent logs serialize several times for each sub-request.

STR:

Install any symfony application v2.8-3.4
Create a simple page with 20-50 sub-requests and 5k-10k logs
Open page in dev mode

Actual

debug toolbar not open (404 not found)
sets of logs (in the logger debug panel) for different requests are same
log processing takes about 20-30s
gc also run about 50% of execution time

Expected

Debug toolbar should be open
As developer, I want to see in logger panel only those logs that are relevant to the given sub-request

Profile and performance

Tested on page with 7k logs and 70 sub-request.

Comparison:

nicolas-grekas · 2017-07-25T12:26:16Z

Should the profile for the master request get all logs? At least yes when framework.profiler.only_master_requests is set.
Thinking about the toolbar/panel, having the total count for logs of all sub requests looks desirable to me also. On the implementation side, a simple implementation would require cloning values from collector twice. A better implem would looks for profiles of all subrequests and aggregate them for the master request.

javiereguiluz · 2017-07-26T07:51:03Z

src/Symfony/Bridge/Monolog/Processor/DebugProcessor.php

+        if (func_num_args() === 1) {
+            $requestHash = func_get_arg(0);
+            if (isset($this->recordsRequest[$requestHash]['records'])
+                && $requestHash !== $this->masterRequestHash


We usually don't split these long if() conditions, so this should be one line.

javiereguiluz · 2017-07-26T07:51:59Z

src/Symfony/Bridge/Monolog/Processor/DebugProcessor.php

@@ -45,6 +66,15 @@ public function __invoke(array $record)
     */
    public function getLogs()
    {
+        if (func_num_args() === 1) {


In Symfony code we use Yoda conditions, so this should be 1 === func_num_args () instead.

javiereguiluz · 2017-07-26T07:52:29Z

src/Symfony/Bridge/Monolog/Processor/DebugProcessor.php

+        if (func_num_args() === 1) {
+            $requestHash = func_get_arg(0);
+            if (isset($this->recordsRequest[$requestHash]['errorCount'])
+                && $requestHash !== $this->masterRequestHash


same with this if() condition

vtsykun · 2017-07-26T09:07:02Z

I guess that profile should for the master request get all logs

…1598) - added local cache for a logger collector - overwrite symfony`s data_collector.logger class - added tests

nicolas-grekas · 2017-08-07T07:18:05Z

I guess that profile should for the master request get all logs

@vtsykun same to me. OK to finish this PR then?

nicolas-grekas · 2017-08-31T14:34:45Z

ping @vtsykun

vtsykun · 2017-08-31T19:57:18Z

Hi @nicolas-grekas, this done, please review
Yes, I updated PR and squashed commit on Jul 29. So in this implementation I show all logs for master request.

ro0NL

nice 👍

ro0NL · 2017-09-13T18:55:31Z

src/Symfony/Component/HttpKernel/DataCollector/LoggerDataCollector.php

@@ -25,6 +25,7 @@ class LoggerDataCollector extends DataCollector implements LateDataCollectorInte
 {
    private $logger;
    private $containerPathPrefix;
+    private $splRequestHash;


$currentRequestHash? (compared to $masterRequestHash above)

ro0NL · 2017-09-13T19:04:01Z

src/Symfony/Bridge/Monolog/Processor/DebugProcessor.php

+    }
+
+    /**
+     * @param GetResponseEvent $event


can be removed i guess

ro0NL · 2017-09-13T19:10:57Z

src/Symfony/Bridge/Monolog/Processor/DebugProcessor.php

+            return;
+        }
+
+        $this->masterRequestHash = spl_object_hash($event->getRequest());


might store the current request hash as well (to use if not passed on the calling side)

is passing it on the calling side really needed?

I thinking the better move it on LoggerDataCollector side.

ro0NL · 2017-09-13T19:15:06Z

src/Symfony/Bridge/Monolog/Processor/DebugProcessor.php

 use Symfony\Component\HttpKernel\Log\DebugLoggerInterface;

 class DebugProcessor implements DebugLoggerInterface
 {
-    private $records = array();
-    private $errorCount = 0;


what about array for both? in favor of array_column

ro0NL

almost

ro0NL · 2017-09-15T19:20:15Z

src/Symfony/Component/HttpKernel/DataCollector/LoggerDataCollector.php

+        if (null !== $this->requestStack && $this->requestStack->getMasterRequest() !== $request) {
+            $this->currentRequestHash = spl_object_hash($request);
+        } else {
+            //gets all logs for the master request


space missing + gets > get

ro0NL · 2017-09-15T19:21:01Z

src/Symfony/Bridge/Monolog/Processor/DebugProcessor.php

+        }
+
+        if (1 === func_num_args()) {
+            $requestHash = func_get_arg(0);


collector might pass null here, should we be explicit? the key would be ""

fabpot · 2017-09-17T15:09:53Z

src/Symfony/Bridge/Monolog/Logger.php

@@ -27,7 +27,7 @@ class Logger extends BaseLogger implements DebugLoggerInterface
    public function getLogs()
    {
        if ($logger = $this->getDebugLogger()) {
-            return $logger->getLogs();
+            return call_user_func_array(array($logger, 'getLogs'), func_get_args());


Could be replace with: $logger->getLogs(...func_get_args()); which is easier to read and faster.

not php 5.5 compat

Yes, it not supported on php < 5.6 http://sandbox.onlinephpfunctions.com/code/3ba4ee65a58af1267040434818d752f26bc885a1

fabpot · 2017-09-17T15:09:57Z

src/Symfony/Bridge/Monolog/Logger.php

@@ -39,7 +39,7 @@ public function getLogs()
    public function countErrors()
    {
        if ($logger = $this->getDebugLogger()) {
-            return $logger->countErrors();
+            return call_user_func_array(array($logger, 'countErrors'), func_get_args());


ro0NL · 2017-09-21T17:43:22Z

From #24275; do events have this same issue?

nicolas-grekas · 2018-04-09T21:36:02Z

@vtsykun thank you for this PR, and sorry I didn't manage to review it earlier. I just rebased it on top of the latest version of branch 3.4, with a few minor changes. The code looks to good to me. It sill misses a few unit tests. Would you be up for adding them? If yes, don't forget to fetch+rebase --hard on your remote patch-splitting-logs branch. If not, I'm sure someone else can take over. Please advise.

nicolas-grekas

(I just pushed some unit tests)

fabpot

Looks like something for 4.1/master intead

nicolas-grekas · 2018-04-17T11:17:33Z

Looks like something for 4.1/master instead

It could go on 4.1 for sure. The only thing that made me keep it for 3.4 is that the reported behavior affects 3.4 (sub-requests heavy pages suffer from a significant penalty because of log duplication.)

…uests

nicolas-grekas · 2018-04-20T09:45:36Z

Rebased on master, ready for 4.1.

nicolas-grekas · 2018-04-20T09:46:55Z

Thank you @vtsykun.

…fferent sub-requests (vtsykun) This PR was merged into the 4.1-dev branch. Discussion ---------- [HttpKernel] LoggerDataCollector: splitting logs on different sub-requests | Q | A | ------------- | --- | Branch? | 4.1 | Bug fix? | yes | New feature? | no | BC breaks? | no | Deprecations? | no | Tests pass? | - | Fixed tickets | #23620 | License | MIT | Doc PR | - This PR fixed performance problem in dev mode and prevent logs serialize several times for each sub-request. STR: 1) Install any symfony application v2.8-3.4 2) Create a simple page with 20-50 sub-requests and 5k-10k logs 3) Open page in dev mode **Actual** - debug toolbar not open (404 not found) - sets of logs (in the logger debug panel) for different requests are same - log processing takes about 20-30s - gc also run about 50% of execution time **Expected** - Debug toolbar should be open - As developer, I want to see in logger panel only those logs that are relevant to the given sub-request ### Profile and performance Tested on page with 7k logs and 70 sub-request. Comparison: - v3.3.5 - this patch: https://blackfire.io/profiles/compare/b1d410b3-c4a7-4778-9b6d-514f72193e28/graph - v3.3.5 - patch #23644 https://blackfire.io/profiles/compare/d0715b04-7834-4981-8da2-9f1dcb2ef90c/graph Commits ------- d0cb1de [HttpKernel] LoggerDataCollector: splitting logs on different sub-requests

…0NL) This PR was merged into the 4.1 branch. Discussion ---------- [MonologBridge] Return empty list for unknown requests | Q | A | ------------- | --- | Branch? | 4.1 | Bug fix? | yes | New feature? | no | BC breaks? | no  | Deprecations? | no | Tests pass? | yes  | Fixed tickets | #...  | License | MIT | Doc PR | symfony/symfony-docs#...  Small continuation of #23659. Currently if the specified request is unknown, you'll get all available requests merged. This fixes it. Commits ------- 69be8e6 [MonologBridge] Return empty list for unknonwn requests

This PR was merged into the 3.4 branch. Discussion ---------- [EventDispatcher] Unwrap wrapped listeners internally | Q | A | ------------- | --- | Branch? | 3.4 | Bug fix? | yes | New feature? | no | BC breaks? | no  | Deprecations? | no | Tests pass? | yes  | Fixed tickets | partially #24275 | License | MIT | Doc PR | symfony/symfony-docs#...  I believe what happens is we re-register the listeners on the wrapped dispatcher, so in case of sub request we get those and wrap it again. The profiler seems to confirm this in case of an exception (default 404 here) and thus creates a sub-request. ## Before: (main request) No exception: (OK) ![image](https://user-images.githubusercontent.com/1047696/48968597-295ec180-eff2-11e8-982c-756e9019e107.png) Exception: (KO) ![image](https://user-images.githubusercontent.com/1047696/48968612-4abfad80-eff2-11e8-8556-37bfbd51bc3b.png) ## After (main request): No exception: same Exception: (OK) ![image](https://user-images.githubusercontent.com/1047696/48968636-9f632880-eff2-11e8-983e-b4099133600b.png) I haven't furhter investigated if we should split events per request as done in #23659 for logs. It seems somewhere we deduplicate events.. so im not sure the profiler actually shows correct data. Commits ------- 448e2e2 [EventDispatcher] Unwrap wrapped listeners internally

This PR was squashed before being merged into the 4.3-dev branch (closes #29312). Discussion ---------- [EventDispatcher] Split events across requests | Q | A | ------------- | --- | Branch? | master | Bug fix? | no | New feature? | yes | BC breaks? | no  | Deprecations? | no | Tests pass? | yes  | Fixed tickets | #24275 | License | MIT | Doc PR | symfony/symfony-docs#...  Split events per request, as currently a profiled sub-request includes all events. Follows same approach how logs are split in #23659. Commits ------- c3477ba [EventDispatcher] Split events across requests

carsonbot added Status: Needs Review HttpKernel Bug Feature labels Jul 24, 2017

nicolas-grekas added this to the 3.3 milestone Jul 25, 2017

javiereguiluz reviewed Jul 26, 2017

View reviewed changes

nicolas-grekas mentioned this pull request Jul 26, 2017

[VarDumper] Keep and reuse array stubs in memory #23683

Merged

vtsykun referenced this pull request in oroinc/platform Jul 26, 2017

BAP-14658: LoggerDataCollector called over 500k times in dev mode (#1…

915723a

…1598) - added local cache for a logger collector - overwrite symfony`s data_collector.logger class - added tests

vtsykun force-pushed the patch-splitting-logs branch from c0a300d to c02cedd Compare July 29, 2017 00:09

vtsykun changed the title ~~[WIP] [HttpKernel] LoggerDataCollector: splitting logs on different sub-requests~~ [HttpKernel] LoggerDataCollector: splitting logs on different sub-requests Jul 29, 2017

ro0NL reviewed Sep 13, 2017

View reviewed changes

ro0NL reviewed Sep 15, 2017

View reviewed changes

ro0NL approved these changes Sep 17, 2017

View reviewed changes

fabpot requested changes Sep 17, 2017

View reviewed changes

nicolas-grekas modified the milestones: 3.3, 3.4 Oct 9, 2017

nicolas-grekas force-pushed the patch-splitting-logs branch from 4684570 to 1ec0653 Compare October 12, 2017 16:38

vtsykun changed the base branch from 3.3 to 3.4 October 13, 2017 15:48

nicolas-grekas force-pushed the patch-splitting-logs branch from 1ec0653 to 840525d Compare April 9, 2018 21:33

nicolas-grekas removed the Feature label Apr 9, 2018

nicolas-grekas mentioned this pull request Apr 9, 2018

Events profiler and logs get confused when an exception is thrown in one of listeners #24275

Closed

nicolas-grekas force-pushed the patch-splitting-logs branch from 840525d to a124770 Compare April 17, 2018 11:07

nicolas-grekas approved these changes Apr 17, 2018

View reviewed changes

carsonbot added Status: Reviewed and removed Status: Needs Review labels Apr 17, 2018

fabpot approved these changes Apr 17, 2018

View reviewed changes

[HttpKernel] LoggerDataCollector: splitting logs on different sub-req…

d0cb1de

…uests

nicolas-grekas changed the base branch from 3.4 to master April 20, 2018 09:44

nicolas-grekas force-pushed the patch-splitting-logs branch from a124770 to d0cb1de Compare April 20, 2018 09:45

nicolas-grekas modified the milestones: 3.4, 4.1 Apr 20, 2018

nicolas-grekas merged commit d0cb1de into symfony:master Apr 20, 2018

fabpot mentioned this pull request May 7, 2018

Release v4.1.0-BETA1 #27181

Merged

This was referenced Nov 24, 2018

[EventDispatcher] Unwrap wrapped listeners internally #29305

Merged

[MonologBridge] Return empty list for unknown requests #29310

Merged

[EventDispatcher] Split events across requests #29312

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[HttpKernel] LoggerDataCollector: splitting logs on different sub-requests #23659

[HttpKernel] LoggerDataCollector: splitting logs on different sub-requests #23659

vtsykun commented Jul 24, 2017 •

edited by nicolas-grekas

Loading

nicolas-grekas commented Jul 25, 2017 •

edited

Loading

javiereguiluz Jul 26, 2017

vtsykun Jul 26, 2017

javiereguiluz Jul 26, 2017

vtsykun Jul 26, 2017

javiereguiluz Jul 26, 2017

vtsykun commented Jul 26, 2017

nicolas-grekas commented Aug 7, 2017

nicolas-grekas commented Aug 31, 2017

vtsykun commented Aug 31, 2017

ro0NL left a comment

ro0NL Sep 13, 2017

ro0NL Sep 13, 2017

ro0NL Sep 13, 2017

ro0NL Sep 13, 2017

vtsykun Sep 14, 2017

ro0NL Sep 13, 2017

ro0NL left a comment

ro0NL Sep 15, 2017 •

edited

Loading

ro0NL Sep 15, 2017 •

edited

Loading

vtsykun Sep 17, 2017

fabpot Sep 17, 2017

ro0NL Sep 17, 2017

vtsykun Sep 19, 2017

fabpot Sep 17, 2017

ro0NL commented Sep 21, 2017

nicolas-grekas commented Apr 9, 2018 •

edited

Loading

nicolas-grekas left a comment

fabpot left a comment

nicolas-grekas commented Apr 17, 2018

nicolas-grekas commented Apr 20, 2018

nicolas-grekas commented Apr 20, 2018

[HttpKernel] LoggerDataCollector: splitting logs on different sub-requests #23659

[HttpKernel] LoggerDataCollector: splitting logs on different sub-requests #23659

Conversation

vtsykun commented Jul 24, 2017 • edited by nicolas-grekas Loading

Profile and performance

nicolas-grekas commented Jul 25, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

vtsykun commented Jul 26, 2017

nicolas-grekas commented Aug 7, 2017

nicolas-grekas commented Aug 31, 2017

vtsykun commented Aug 31, 2017

ro0NL left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ro0NL left a comment

Choose a reason for hiding this comment

ro0NL Sep 15, 2017 • edited Loading

Choose a reason for hiding this comment

ro0NL Sep 15, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ro0NL commented Sep 21, 2017

nicolas-grekas commented Apr 9, 2018 • edited Loading

nicolas-grekas left a comment

Choose a reason for hiding this comment

fabpot left a comment

Choose a reason for hiding this comment

nicolas-grekas commented Apr 17, 2018

nicolas-grekas commented Apr 20, 2018

nicolas-grekas commented Apr 20, 2018

vtsykun commented Jul 24, 2017 •

edited by nicolas-grekas

Loading

nicolas-grekas commented Jul 25, 2017 •

edited

Loading

ro0NL Sep 15, 2017 •

edited

Loading

ro0NL Sep 15, 2017 •

edited

Loading

nicolas-grekas commented Apr 9, 2018 •

edited

Loading