Codestin Search App

mdeboer · 2025-04-08T18:58:57Z

This PR adds support for running benchmarks using phpbench as discussed in #612

I have included both a KernelBench and UnitBench class that can serve as a base for a benchmark
class.

The KernelBench base class compares to the KernelTestCase for integration tests where an actual application kernel is booted. Whereas the UnitBench base class compares to a regular test case class used in unit tests; it boots foundry but not a whole Symfony kernel.

I have included a sample benchmark that uses KernelBench and uses the 'reset database' functionality.

Unfortunately phpbench does not have a similar system regarding hooks as phpunit, hence I could not solve the 'reset database' functionality with a trait that automatically makes sure some code is being run before the tests. Who knows I might knock on their door next to have that improved as this is wonky at best, but I knew of no other way to fix this.

If you have a KernelBench benchmark that does not require the 'reset database' functionality, you could leave out the class attributes on your benchmark.

nikophil · 2025-04-09T08:04:55Z

Hi!

thanks for this!

I'll handle the PHPStan problem, I have an idea to handle this nicely

Correct me if I'm wrong, to run this comparison stuff, you should do this:

# run this on the "old" code, to create a baseline
$ bin/phpbench run tests/Benchmark --tag=main

# run this on the "new" code, and compare against "main"
$ bin/phpbench run tests/Benchmark --ref=main --report=aggregate

I think, with this, we will be able to show the output in the CI

the ultimate goal being to have the CI fail when a serious performance problem occurs. I've manage to have an assertion check doing the following: --assert="mode(variant.time.avg) <= mode(baseline.time.avg) +/- 5%" (I think this is a very naive approach, since you can define assertions thanks to attributes).

My main problem here, is that in the case of your fix on random(), the case with "1 entity" is less performant than the original one, (which is normal since we're performing one more db query)

I need to deep dive into the assertion mechanism to understand better the granularity of those assertions

mdeboer · 2025-04-09T08:58:06Z

Hi @nikophil!

Correct me if I'm wrong, to run this comparison stuff, you should do this:

Yes that is entirely correct! But for CI you want both commands to run with e.g. 5 iterations using --iterations=5. But thinking of it, it might be better to define the assertions per benchmark using attributes. One benchmark might take a few seconds allowing you to run it with more iterations while others might take longer and run with less iterations.

My main problem here, is that in the case of your fix on random(), the case with "1 entity" is less performant than the original one, (which is normal since we're performing one more db query)

Ah yes, well I included the 1 entity case for verbosity but this is not a realistic case I think. I think 5 would have been a better more realistic test case, which should perform equally or better. It can easily be changed in the "data provider" here.

nikophil · 2025-04-09T09:21:42Z

it seems that in my benchmark, even with 50 entities, performing a count() still introduces an overhead

I'm totally fine with it since we're talking about ~2ms, but I'm wondering how to not fail the CI (I need to test it with --iterations=5 though)

nikophil · 2025-04-09T20:47:42Z

OK, I think I got something that is ok-ish :)

I've added a bench for "random" for ODM + another bench for "create" and "create many" for 1 ODM class, 2 ORM class (on one class, there is a one-to-many being provided as well)

I still cannot run the true "baseline" run, because phpbench is still not present on 2.x branch, I'll need another PR for that, I'll work on the assertions in this further PR

here is the output of my CI run

 +--------------------------------+-------------------+-----+------+-----+-----------+-----------+---------+
| benchmark                      | subject           | set | revs | its | mem_peak  | mode      | rstdev  |
+--------------------------------+-------------------+-----+------+-----+-----------+-----------+---------+
| GenericEntityFactoryBench      | bench_random      | 50  | 10   | 5   | 36.926mb  | 908.306μs | ±23.04% |
| GenericEntityFactoryBench      | bench_random      | 100 | 10   | 5   | 38.403mb  | 2.104ms   | ±19.14% |
| GenericEntityFactoryBench      | bench_random      | 500 | 10   | 5   | 47.576mb  | 7.492ms   | ±29.06% |
| GenericEntityFactoryBench      | bench_random_set  | 50  | 10   | 5   | 40.798mb  | 10.270ms  | ±1.90%  |
| GenericEntityFactoryBench      | bench_random_set  | 100 | 10   | 5   | 42.017mb  | 11.126ms  | ±4.50%  |
| GenericEntityFactoryBench      | bench_random_set  | 500 | 10   | 5   | 51.370mb  | 17.133ms  | ±12.50% |
| ContactFactoryBench            | bench_create      |     | 10   | 5   | 37.189mb  | 7.443ms   | ±5.73%  |
| ContactFactoryBench            | bench_create_many | 1   | 10   | 5   | 37.193mb  | 7.764ms   | ±8.67%  |
| ContactFactoryBench            | bench_create_many | 10  | 10   | 5   | 42.256mb  | 113.655ms | ±2.20%  |
| ContactFactoryBench            | bench_create_many | 50  | 10   | 5   | 64.628mb  | 1.438s    | ±0.83%  |
| CategoryFactoryBench           | bench_create      |     | 10   | 5   | 38.034mb  | 7.541ms   | ±1.50%  |
| CategoryFactoryBench           | bench_create_many | 1   | 10   | 5   | 38.037mb  | 7.832ms   | ±6.19%  |
| CategoryFactoryBench           | bench_create_many | 10  | 10   | 5   | 52.820mb  | 120.368ms | ±3.62%  |
| CategoryFactoryBench           | bench_create_many | 50  | 10   | 5   | 118.511mb | 1.849s    | ±2.36%  |
| GenericDocumentFactoryBench    | bench_random      | 50  | 10   | 5   | 36.145mb  | 607.601μs | ±2.24%  |
| GenericDocumentFactoryBench    | bench_random      | 100 | 10   | 5   | 37.145mb  | 739.223μs | ±1.29%  |
| GenericDocumentFactoryBench    | bench_random      | 500 | 10   | 5   | 44.396mb  | 1.954ms   | ±0.89%  |
| GenericDocumentFactoryBench    | bench_random_set  | 50  | 10   | 5   | 36.145mb  | 602.931μs | ±2.75%  |
| GenericDocumentFactoryBench    | bench_random_set  | 100 | 10   | 5   | 37.145mb  | 732.714μs | ±1.72%  |
| GenericDocumentFactoryBench    | bench_random_set  | 500 | 10   | 5   | 44.396mb  | 1.945ms   | ±0.39%  |
| PersistentDocumentFactoryBench | bench_create      |     | 10   | 5   | 35.127mb  | 1.768ms   | ±1.24%  |
| PersistentDocumentFactoryBench | bench_create_many | 1   | 10   | 5   | 35.128mb  | 1.758ms   | ±1.88%  |
| PersistentDocumentFactoryBench | bench_create_many | 10  | 10   | 5   | 36.825mb  | 11.933ms  | ±1.54%  |
| PersistentDocumentFactoryBench | bench_create_many | 50  | 10   | 5   | 44.005mb  | 101.046ms | ±0.59%  |
+--------------------------------+-------------------+-----+------+-----+-----------+-----------+---------+

it's strange how rstdev ("relative standard deviation") is high on the random benchs: maybe it is inherent to random... I think it will be hard to have consistent assertions with such a deviation between tests. I'll see if fixing the seed can resolve the problem

I'd take a review from you before merging 🙏

thanks for your help on this!

nikophil · 2025-04-09T20:53:25Z

        "test-main": "./phpunit --testsuite main",
-        "test-reset-database": "./phpunit --testsuite reset-database --bootstrap tests/bootstrap-reset-database.php"
+        "test-reset-database": "./phpunit --testsuite reset-database --bootstrap tests/bootstrap-reset-database.php",
+        "post-install-cmd": ["@composer bin phpstan install", "@composer bin phpbench install"]


this way, when we'll do "composer install", both phpstan and phpbench will be installed, and no more problem with phpstan locally because it would not know about phpbench

nikophil · 2025-04-09T20:53:48Z

+      - name: Install PHPBench
+        run: composer bin phpbench install


same than https://github.com/zenstruck/foundry/pull/866/files#r2036127527 but for CI

nikophil · 2025-04-09T20:54:29Z

@@ -0,0 +1,170 @@
+<?php
+
+namespace Zenstruck\Foundry\Tests\Benchmark;


I've moved this class an the other one in the tests directory

nikophil · 2025-04-09T20:55:52Z

+
+#[BeforeClassMethods(['_resetDatabaseBeforeFirstBench'])]
+#[BeforeMethods(['_bootFoundry', '_resetDatabaseBeforeEachBench'])]
+abstract class PersistentFactoryBench extends KernelBench


I had to split into two different base classes, because, performing all these static::factory()->many($params['count'])->create(); in _setup_bench_random() was really time consuming

kbond

@mdeboer, thank you so much for getting this going!

Looks great, I want to see this in action!

nikophil · 2025-04-10T06:36:07Z

I just don't understand those results for the random method: I've removed all "randomness" and only used deterministic stuff:

added a #[Warmup(1)] to warmup Symfony/Doctrine caches
removed shuffle() just for the need of the test
used mt_rand() in RepositoryDecorator::random()
seeded mt_rand() with a fixed seed: mt_srand(1)
added #[Revs(100)] so it will repeat each case x100 times per iteration

and I still have very volatile results, only for GenericEntityFactoryBench 🤔

+--------------------------------+-------------------+-----+------+-----+-----------+-----------+---------+
| benchmark                      | subject           | set | revs | its | mem_peak  | mode      | rstdev  |
+--------------------------------+-------------------+-----+------+-----+-----------+-----------+---------+
| GenericEntityFactoryBench      | bench_random      | 50  | 100  | 5   | 37.030mb  | 594.751μs | ±33.41% |
| GenericEntityFactoryBench      | bench_random      | 100 | 100  | 5   | 38.507mb  | 1.684ms   | ±21.23% |
| GenericEntityFactoryBench      | bench_random      | 500 | 100  | 5   | 47.680mb  | 6.211ms   | ±30.38% |
| GenericEntityFactoryBench      | bench_random_set  | 50  | 100  | 5   | 40.902mb  | 9.587ms   | ±2.56%  |
| GenericEntityFactoryBench      | bench_random_set  | 100 | 100  | 5   | 42.121mb  | 10.529ms  | ±3.87%  |
| GenericEntityFactoryBench      | bench_random_set  | 500 | 100  | 5   | 51.474mb  | 15.268ms  | ±12.37% |
| ContactFactoryBench            | bench_create      |     | 10   | 5   | 37.252mb  | 8.519ms   | ±3.67%  |
| ContactFactoryBench            | bench_create_many | 1   | 10   | 5   | 37.256mb  | 8.865ms   | ±5.60%  |
| ContactFactoryBench            | bench_create_many | 10  | 10   | 5   | 42.769mb  | 133.940ms | ±0.87%  |
| ContactFactoryBench            | bench_create_many | 50  | 10   | 5   | 67.214mb  | 1.721s    | ±0.38%  |
| CategoryFactoryBench           | bench_create      |     | 10   | 5   | 38.222mb  | 7.534ms   | ±4.55%  |
| CategoryFactoryBench           | bench_create_many | 1   | 10   | 5   | 38.226mb  | 7.391ms   | ±4.30%  |
| CategoryFactoryBench           | bench_create_many | 10  | 10   | 5   | 54.335mb  | 137.723ms | ±1.10%  |
| CategoryFactoryBench           | bench_create_many | 50  | 10   | 5   | 125.898mb | 2.080s    | ±1.07%  |
| GenericDocumentFactoryBench    | bench_random      | 50  | 100  | 5   | 36.301mb  | 437.121μs | ±1.99%  |
| GenericDocumentFactoryBench    | bench_random      | 100 | 100  | 5   | 37.304mb  | 560.563μs | ±1.59%  |
| GenericDocumentFactoryBench    | bench_random      | 500 | 100  | 5   | 44.743mb  | 1.777ms   | ±0.54%  |
| GenericDocumentFactoryBench    | bench_random_set  | 50  | 100  | 5   | 36.301mb  | 444.953μs | ±0.75%  |
| GenericDocumentFactoryBench    | bench_random_set  | 100 | 100  | 5   | 37.304mb  | 559.929μs | ±1.23%  |
| GenericDocumentFactoryBench    | bench_random_set  | 500 | 100  | 5   | 44.743mb  | 1.788ms   | ±1.28%  |
| PersistentDocumentFactoryBench | bench_create      |     | 10   | 5   | 35.145mb  | 905.363μs | ±1.76%  |
| PersistentDocumentFactoryBench | bench_create_many | 1   | 10   | 5   | 35.146mb  | 922.373μs | ±3.01%  |
| PersistentDocumentFactoryBench | bench_create_many | 10  | 10   | 5   | 36.986mb  | 11.342ms  | ±2.62%  |
| PersistentDocumentFactoryBench | bench_create_many | 50  | 10   | 5   | 44.969mb  | 112.313ms | ±0.53%  |
+--------------------------------+-------------------+-----+------+-----+-----------+-----------+---------+

nikophil · 2025-04-10T07:12:24Z

I've added the option --retry-threshold=10 which will retry until the rstdev gets lower than 10%

They warn on their website that this can increase the duration of the run

and indeed, without a threshold the duration was 6min and now it is 9min

But I think it is worth it... In the doc they advertise that for a "baseline", the threshold should be 5. they actually say :

everything above 2 should be treated as suspicious

And this does not affect the rest of the CI...

Maybe if we feel that it takes too much time, in the future, I'll run only once the baseline when merging, and I'll store it in cache using action/cache

I'm merging this PR, since results are now ok-ish

don't hesitate to make any remarks even after this has been merged

mdeboer · 2025-04-10T11:52:53Z

Looks great guys, I'll have another in-depth look when I have a bit more time ❤️

Co-authored-by: Nicolas PHILIPPE <[email protected]>

Maarten de Boer added 2 commits April 8, 2025 20:47

feat: add support for running benchmarks using phpbench

692261d

feat: add orm generic entity factory benchmark for random item

c67a1e2

mdeboer mentioned this pull request Apr 8, 2025

feat: optimize performance of repository::random() #867

Merged

nikophil force-pushed the feat/add-phpbench branch 9 times, most recently from e03e4a0 to 56f7318 Compare April 9, 2025 20:37

nikophil reviewed Apr 9, 2025

View reviewed changes

nikophil requested a review from kbond April 9, 2025 20:56

kbond approved these changes Apr 9, 2025

View reviewed changes

Comment thread .github/workflows/benchmarks.yml

nikophil force-pushed the feat/add-phpbench branch from 56f7318 to 12cbf9c Compare April 10, 2025 06:26

feat: create a workflow GitHub Action

2fec28a

nikophil force-pushed the feat/add-phpbench branch from 3874f09 to 2fec28a Compare April 10, 2025 06:51

nikophil merged commit 5ccbe51 into zenstruck:2.x Apr 10, 2025

nikophil added a commit to nikophil/foundry that referenced this pull request Apr 13, 2025

feat: add support for benchmarks using phpbench (zenstruck#866)

0ed06f2

Co-authored-by: Nicolas PHILIPPE <[email protected]>

nikophil added a commit to nikophil/foundry that referenced this pull request Apr 13, 2025

feat: add support for benchmarks using phpbench (zenstruck#866)

0d70b89

Co-authored-by: Nicolas PHILIPPE <[email protected]>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat: add support for benchmarks using phpbench#866

feat: add support for benchmarks using phpbench#866
nikophil merged 3 commits into
zenstruck:2.xfrom
mdeboer:feat/add-phpbench

mdeboer commented Apr 8, 2025

Uh oh!

nikophil commented Apr 9, 2025

Uh oh!

mdeboer commented Apr 9, 2025

Uh oh!

nikophil commented Apr 9, 2025

Uh oh!

nikophil commented Apr 9, 2025 •

edited

Loading

Uh oh!

nikophil Apr 9, 2025

Uh oh!

nikophil Apr 9, 2025

Uh oh!

nikophil Apr 9, 2025

Uh oh!

nikophil Apr 9, 2025

Uh oh!

kbond left a comment

Uh oh!

Uh oh!

nikophil commented Apr 10, 2025 •

edited

Loading

Uh oh!

nikophil commented Apr 10, 2025 •

edited

Loading

Uh oh!

mdeboer commented Apr 10, 2025

Uh oh!

Reviewers

Assignees

Labels

Milestone

Development

Uh oh!

3 participants

		@@ -0,0 +1,170 @@
		<?php

		namespace Zenstruck\Foundry\Tests\Benchmark;

Uh oh!

Conversation

mdeboer commented Apr 8, 2025

Uh oh!

nikophil commented Apr 9, 2025

Uh oh!

mdeboer commented Apr 9, 2025

Uh oh!

nikophil commented Apr 9, 2025

Uh oh!

nikophil commented Apr 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

nikophil Apr 9, 2025

Choose a reason for hiding this comment

Uh oh!

nikophil Apr 9, 2025

Choose a reason for hiding this comment

Uh oh!

nikophil Apr 9, 2025

Choose a reason for hiding this comment

Uh oh!

nikophil Apr 9, 2025

Choose a reason for hiding this comment

Uh oh!

kbond left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

nikophil commented Apr 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

nikophil commented Apr 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mdeboer commented Apr 10, 2025

Uh oh!

Reviewers

Assignees

Labels

Milestone

Development

Uh oh!

3 participants

nikophil commented Apr 9, 2025 •

edited

Loading

nikophil commented Apr 10, 2025 •

edited

Loading

nikophil commented Apr 10, 2025 •

edited

Loading