I would rename the issue as: create a more realistic benchmark for containers.
The current one measures if there is one more "require" or "isset" on the code path, which we do, since we implement a few more interfaces + handle a few more features (esp. aliases + deprecations).
I looked at the code of each container: that's the only significant difference between the fastest. Which is what makes me state we need another benchmark.

Thanks for the insights. This benchmark literally starts saying "we're making this benchmark because a while ago someone published a benchmark which was wrong". So maybe this benchmark is wrong too to measure real-life container performance.

One case that is not handled well by the current benchmark is the "unused services" one: because we're in PHP, running in short lived cycles, we don't need all services all the time. That's a significant aspect of the performance of a PHP container: will it be fast also when a serious number of services is defined, but only a few of them are actually used per request?

Regarding the tests themselves:

• cold test is useless maybe to compare how the containers fair in a dev environment
• the "big" test is about retrieve a simple graph of objects which is unlikely to be really indicative of anything: it should be a more complex scenario with more dependencies between each services not a simple tree
• the containers are not tested with aliases: I think that's an important feature that should be tested
• the containers are "clean": they only go the services that are going to be retrieved. Once big improvement with Symfony is that you can use a lot of aliases and services, they are going to be optimised behind. This would also reflect more what happens in a real-life application.

Without those, this benchmark is a bit like the JIT tests done in PHP: awesome results for the tests and then realising it doesn't change anything in real-life applications, because that's not how things are used.

On Symfony side, from what I understood the slowness comes from the aliases which we are keeping, so we have an additional check to see if the service is an alias or not. Apparently this is not trivial to remove in the compiled container, but that would bring better performances for sure.

Closing because now it's clear that this benchmark is too synthetic and it won't help us improving the performance of real apps. Thanks!

@javiereguiluz Let's please reopen this. I believe assumptions of two previous posters are incorrect. This needs more investigation at least.

will it be fast also when a serious number of services is defined, but only a few of them are actually used per request?

the containers are "clean": they only go the services that are going to be retrieved. Once big improvement with Symfony is that you can use a lot of aliases and services, they are going to be optimised behind. This would also reflect more what happens in a real-life application.

I modified the benchmark to call ->get() for single \DiContainerBenchmarks\Fixture\Class10::class only, which uses only 10 dependencies out of available 100 services and results didn't change.

cold test is useless maybe to compare how the containers fair in a dev environment

No it isn't. You assume cold test of this benchmark compiles the container. It doesn't. It only creates new instance of already compiled container, $this->container = new CompiledPrototypeContainer(); I also don't like how many people here repeat that performance in dev environment is irrelevant. That's what we, developers look at all day. PHP's ability to instantly see results after save + F5 is one of it's greatest advantages. Speed of compilation is very important too. But this benchmark doesn't measure that. That's why Symfony has one of best positions in cold tests with low iteration.

the "big" test is about retrieve a simple graph of objects which is unlikely to be really indicative of anything: it should be a more complex scenario with more dependencies between each services not a simple tree

That's assumption which isn't based on facts. What makes you think results would be different when hierarchy of services is distributed more randomly?

the containers are not tested with aliases: I think that's an important feature that should be tested

It's me who concluded this (and also @nicolas-grekas later), but after further investigation I found we were wrong. Commenting out this section in Container.php had no significant impact. There is unknown reason why is Symfony slower and I would like you guys to help find it.

Now, I've forked the benchmark and upgraded it to use Symfony 4.0 DI component. And I reduced the impact zone by reducing benchmark to this small simple script. When lines in Container.php linked above get commented, logical assumption (and debugger confirms) that both Containers execute same things. Despite that, Symfony DI takes 3x as much time as Yaco container. It's especially obvious when xdebug is disabled. So let's find out why.

Can you try swapping the tested containers, are the results similar? I noticed that the order of such microbench can have a significant effect sometimes.

Ok I swapped them, results still seem similar :/

I think you can reclaim performance by removing most of the inheritance tree, like the interfaces. They're not needed for the bench. You can also remove the "invalid behavior" argument.
Last time I played with the bench, I think that's what made the difference.
If that's confirmed, it'd be to me definitive proof that the bench is not relevant.

@nicolas-grekas About which instance of "invalid behavior" are you talking about? I see that gets aren't using it and even if this shouldn't affect the performance.

I would also like to know. I think you didn't look at my minimalistic benchmark script? Please clone and take a look. There is nothing more there to remove.

I talking about the second argument of Container::get(): if you remove it, you'll see a perf improvement.

@nicolas-grekas I'm looking at the code of tests and I see second parameter is not used: https://github.com/kocsismate/php-di-container-benchmarks/blob/master/src/Container/Symfony/Test1.php

Throwing surely takes time and using ContainerInterface::NULL_ON_INVALID_REFERENCE should be faster, but are we even supposed to use the 2nd parameter for get()? The behavior of constants is not documented anywhere (rel.: symfony/symfony-docs#8759).

I can confirm removing second argument in Symfony\Component\DependencyInjection\Container::get brings substantial improvement. Difference is now within 15% instead of 3x. It has much more impact than additional isset for aliases. So mystery solved. Results of investigation: Symfony DI is slower than competition in tests with lot of iteriations, because it has following additional features:

• aliases
• configurable behaviour on invalid reference

Now we can close this.

Thank you @ostrolucky for the thorough investigation! I am open to suggestions in order to improve my benchmark. :) And @javiereguiluz I'll change the line saying "my test is better than the last one was" because it sounds a bit arrogant. :S

My idea to measure "cold", "semi-warm" and "warm" performance came after Ocramius' suggestion (kocsismate/php-di-container-benchmarks#4 (comment)). Basically, these test suites do the following:

• The "cold" test suites try to simulate a complete request (using Singleton objects): containers are first bootstrapped and then the object graph (10 or 100 objects) is fetched 10, 100 and 1000 times. I chose these numbers because I thought that they can be likely at a real-world app (ok, maybe the object graph is too small). All measurements are repeated 30 times (so opcache will be fed after the first run), and the median of the results is counted.

• The "semi-warm" test suites instantiate a Prototype object graph (10 or 100 objects) 10, 100 and 1000 times. The bootstrap time of the container is excluded from the results so that only the $container->get() calls and instantiation time of the retrieved objects are measured ( including their autoloading time).

• The "warm" test suites instantiate a Singleton object graph (10 or 100 objects again) 100, 1000 and 10 000 times. Basically they only measure the $container->get() calls without any autoloading time.

Of course it's useful to improve "warm" performance, but I think bootstrap and autoloading time of the container can also be very important, especially when the retrieved object graph is not big or when you don't call the $container->get() method many times (and I believe it shouldn't be needed hundreds of times per request).

All in all, I believe "cold" test suites are the most informative ones because they simulate a complete request with workloads which are similar to what "real life" apps might have. But I am honestly curious about your opinion, as you probably know more about the topic.

@kocsismate I just copied your literal words from your benchmark page, where you say that the previous benchmark was bad and yours is better. I don't think that's arrogant: some things are better than others. You are not insulting the other one:

Yeah, I know they are mine :) But I didn't like them at all when I saw them again. Maybe the sentence is not arrogant, but inelegant, so thanks for bringing it up!

sorry for poking old issue. just wanted to share DI related perf issue i faced myself.

even with APP_ENV=prod, and APP_DEBUG=false, ~69% of wall time is used by DI only in a symfony 5.0.7 app running on PHP7.4.4 (with xhprof).