Huge performance degradation of ReflectionClassResource and FileResource on big projects inside Docker For Mac #25999

dmifedorenko · 2018-01-31T23:15:29Z

Q	A
Bug report?	no
Feature request?	yes
BC Break report?	no
RFC?	no
Symfony version	3.4.3

Hello.

ReflectionClassResource::isFresh and FileResource::isFresh methods have performance issue on huge projects and slow file system. For example I'm work with php project with about 22K source files in Docker For Mac.

Method isFresh can makes system call to file system twice for file_exists and filemtime but where are no need to make two calls. You can call only filemtime in all cases and check return value:

 public function isFresh($timestamp)
 {
	if (null === $this->hash) {
		$this->hash = $this->computeHash();
		$this->loadFiles($this->classReflector);
	}
	foreach ($this->files as $file => $v) {
		$filemtime = @filemtime($file);
		if (!$filemtime) {
			return false;
		}
		if (@$filemtime > $timestamp) {
			return $this->hash === $this->computeHash();
		}
	}
	return true;
}

nicolas-grekas · 2018-02-01T05:36:41Z

Can you measure any difference with that change?
PHP normally has a stats cache so this shouldn't make any difference.
Docker on Mac is known to be extra slow sometimes. I would suggest you look for solutions to that instead.

dmifedorenko · 2018-02-01T05:50:56Z

Please check out the attached picture. This is profiler's run result and we can see what both functions inside methods take time to execute. I use Docker For Mac Edge 18.02 and PHP 7.2.1 with php-fpm inside Docker's container.

Docker For Mac have massive file system performance difference between container and host os. Without docker-sync container's hard disk works 100x time slower than host.

I see huge difference between original and patched code.

nicolas-grekas · 2018-02-01T06:05:45Z

Can you double check that php's stats cache is working? (Note that I don't know exactly how to do that. Maybe by forcibly disabling it a see any diff?)

nicolas-grekas · 2018-02-01T06:08:21Z

Oh, can you also use eg Blackfire to profile the performance difference with your patch? A flammegraph is a statistical tool so it looses information. Maybe a more precise picture could help better understand the situation.

dmifedorenko · 2018-02-01T06:14:21Z

Sorry, I cant you Blackfire locally.

I have checked php' stats cache. It is working, call if http://php.net/manual/en/function.stat.php function returns meta data after file_exists call. Then the filemtime call is executed with full stat cache.

I wil write synthetic test case on clear php later so you can check it on your system.

dmifedorenko · 2018-02-01T06:30:56Z

Check the code from attached file please test.txt
Edit directory path in first line to proper local folder on you disk.

I run it on php-fpm (not cli) and this is my result:

683 ms for 3000 files for file_exists + filemtime
50 ms for 3000 files for only filemtime

As you see two calls file_exists + filemtime is 13x slower than one filemtime call.

nicolas-grekas · 2018-02-01T07:00:10Z

The call to stat() near file_exists() completely defeats the comparison, because stat() is the heaviest call. It cannot be used to check if the cache is warm or not.

dmifedorenko · 2018-02-01T07:02:20Z

Ok, have you any idea how can i "double check that php's stats cache is working"?

dmifedorenko · 2018-02-01T07:19:34Z

Second question -- if my configuration broken and stat cache not works are where other users with same issues?

And one more -- is where any reason to make call to file_exists + filemtime if one can call only filemtime?

dmifedorenko · 2018-02-01T11:14:39Z

I checked all several time -- the issue reproduces only on Docker For Mac native volume. If I start php server via docker-sync or read file not from shared Docker's volume but from internal directory -- where are very small difference between patcher and original code.

But then I tried to check difference on Docker's volume I see huge performance degradation.

It is famous issue with Docker For Mac FS, one of relative topic https://forums.docker.com/t/file-access-in-mounted-volumes-extremely-slow-cpu-bound/8076/169

Another clear test case with only one file -- test.txt

My result:

526 ms for 3000 files for file_exists + filemtime
2 ms for 3000 files for only filemtime
Patched version much much fasted then original.

Can you please remove file_exists call to increase symfony performance inside Docker For Mac containers? Our big project takes much time to check is container fresh on each request.

nicolas-grekas · 2018-02-01T11:21:39Z

Would you like to submit a PR? I think doing this on branch 2.7 would be possible (we'll then merge it into 3.4)

dmifedorenko · 2018-02-01T11:24:56Z

I dont know what is PR mean. Can you explain how I can help you please?

theofidry · 2018-02-01T11:35:28Z

PR = Pull Request, i.e. submitting a patch for it. You can find some help on how to do it here and ping us on Slack.

Sorry for the acronyms, we may be a bit too use to GitHub :)

Once done Nicolas, I or other could try your patch and measure if there is any difference.

sroze · 2018-02-01T11:39:26Z

I've very interested in such improvement as well. I'd love to see your patch.

dmifedorenko · 2018-02-01T11:39:30Z

Thank you for help. I cant work today any more and will send you pull request in several days.

dmifedorenko · 2018-02-02T00:13:27Z

@nicolas-grekas, where is not way to patch 2.7. ReflectionClassResource class doesnt exist in that version.

I made pull request to 3.4 #26013, can you check it please?

…mifedorenko) This PR was merged into the 3.4 branch. Discussion ---------- [Config] Only using filemtime to check file freshness | Q | A | ------------- | --- | Branch? | 3.4 | Bug fix? | yes | New feature? | no | BC breaks? | no | Deprecations? | no | Tests pass? | yes | Fixed tickets | #25999 | License | MIT Commits ------- 52c9cb4 [Config] Only using filemtime to check file freshness

dmifedorenko changed the title ~~Bad perfomance of ReflectionClassResource and FileResource on huge project~~ Hude performance degradation of ReflectionClassResource and FileResource on big project inside Docker For Mac Feb 1, 2018

javiereguiluz added DependencyInjection Feature Performance labels Feb 1, 2018

dmifedorenko changed the title ~~Hude performance degradation of ReflectionClassResource and FileResource on big project inside Docker For Mac~~ Hude performance degradation of ReflectionClassResource and FileResource on big projects inside Docker For Mac Feb 1, 2018

javiereguiluz changed the title ~~Hude performance degradation of ReflectionClassResource and FileResource on big projects inside Docker For Mac~~ Huge performance degradation of ReflectionClassResource and FileResource on big projects inside Docker For Mac Feb 1, 2018

This was referenced Feb 3, 2018

[Config] Only using filemtime to check file freshness #26031

Merged

Ticket 25999 #26013

Closed

nicolas-grekas closed this as completed Feb 4, 2018

This was referenced Apr 17, 2020

[Http-Kernel] [Cache] Warning on deploy with empty cache #36472

Closed

[HttpKernel] Avoid unsuppressed include #36473

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

Huge performance degradation of ReflectionClassResource and FileResource on big projects inside Docker For Mac #25999

Huge performance degradation of ReflectionClassResource and FileResource on big projects inside Docker For Mac #25999

dmifedorenko commented Jan 31, 2018 •

edited by javiereguiluz

Loading

nicolas-grekas commented Feb 1, 2018

Uh oh!

dmifedorenko commented Feb 1, 2018

Uh oh!

nicolas-grekas commented Feb 1, 2018

Uh oh!

nicolas-grekas commented Feb 1, 2018

Uh oh!

dmifedorenko commented Feb 1, 2018

Uh oh!

dmifedorenko commented Feb 1, 2018

Uh oh!

nicolas-grekas commented Feb 1, 2018

Uh oh!

dmifedorenko commented Feb 1, 2018

Uh oh!

dmifedorenko commented Feb 1, 2018

Uh oh!

dmifedorenko commented Feb 1, 2018

Uh oh!

nicolas-grekas commented Feb 1, 2018

Uh oh!

dmifedorenko commented Feb 1, 2018

Uh oh!

theofidry commented Feb 1, 2018

Uh oh!

sroze commented Feb 1, 2018

Uh oh!

dmifedorenko commented Feb 1, 2018

Uh oh!

dmifedorenko commented Feb 2, 2018

Uh oh!

Uh oh!

Huge performance degradation of ReflectionClassResource and FileResource on big projects inside Docker For Mac #25999

Huge performance degradation of ReflectionClassResource and FileResource on big projects inside Docker For Mac #25999

Comments

dmifedorenko commented Jan 31, 2018 • edited by javiereguiluz Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

nicolas-grekas commented Feb 1, 2018

Uh oh!

dmifedorenko commented Feb 1, 2018

Uh oh!

nicolas-grekas commented Feb 1, 2018

Uh oh!

nicolas-grekas commented Feb 1, 2018

Uh oh!

dmifedorenko commented Feb 1, 2018

Uh oh!

dmifedorenko commented Feb 1, 2018

Uh oh!

nicolas-grekas commented Feb 1, 2018

Uh oh!

dmifedorenko commented Feb 1, 2018

Uh oh!

dmifedorenko commented Feb 1, 2018

Uh oh!

dmifedorenko commented Feb 1, 2018

Uh oh!

nicolas-grekas commented Feb 1, 2018

Uh oh!

dmifedorenko commented Feb 1, 2018

Uh oh!

theofidry commented Feb 1, 2018

Uh oh!

sroze commented Feb 1, 2018

Uh oh!

dmifedorenko commented Feb 1, 2018

Uh oh!

dmifedorenko commented Feb 2, 2018

Uh oh!

dmifedorenko commented Jan 31, 2018 •

edited by javiereguiluz

Loading