Given the most forgiving level already gives us 1000 baseline errors, I think we should at least add some text in the contributing guide about adding new baseline errors (as it's very likely that new BC layers will result in new false positive errors).

It would also be good to see whether this doesn't add more work load (dealing with false positives) rather than reducing it (by catching a few mistakes that are fixed before stable releases). This doesn't mean I'm against this PR btw, the only way to test this is by trying it out in real life.

Given the most forgiving level already gives us 1000 baseline errors, I think we should at least add some text in the contributing guide about adding new baseline errors (as it's very likely that new BC layers will result in new false positive errors).

Yes, thank you. Just a note: "BC layer" means when we try to support both doctrine 2 and doctrine 3 etc. Since this job will install doctrine 3, it will complain about missing doctrine 2 classes.

only way to test this is by trying it out in real life.

Exactly. Let's do a 6 month experiment if this is a good thing or not.

Given the most forgiving level already gives us 1000 baseline errors, I think we should at least add some text in the contributing guide about adding new baseline errors (as it's very likely that new BC layers will result in new false positive errors).

The way I see the baseline, it is not the right place to deal with known false positives. The baseline is a big TODO list and, once introduced to the codebase, should hardly see any additions. We need to agree on a way to document WONTFIX errors like false positives, compat code for old PHP/dependency versions, Psalm bugs etc. This could be done inline via annotation or using dedicated configuration files.

So, the general rule for PRs would be: sizeof(<baseline of the target branch>) >= sizeof(<PR baseline>). You are welcome to resolve baseline entries, but don't create new ones.

We could maybe run psalm twice: once with deps=high, twice with deps=low,
then we would exclude failures that don't happen on both runs.

From my experience with introducing PHPStan and Psalm to existing codebases in the past: The errors reported by the starter levels (8 in Psalm, 0 in PHPStan) are mainly problems that need to be fixed so that the tool is able to understand the codebase. The tool won't be helpful at that level.

The project I'm using PHPStan in has reached Level 5 now and here PHPStan finds actual bugs and started to become really helpful when changing the codebase. The project I'm using Psalm in is way smaller. Here, we've reached Level 4 with a similar effect.

If we use Psalm, we should start on level 8, but not remain there:

• iteratively kill entries from the baseline
• lower the level by one and generate a new baseline
• repeat

As discussed on Slack, we could consider the codebase itself as the baseline.

In practice, that'd mean that we would run psalm only on PRs, and for each PR, we would first run psalm on the fork point from the target branch to generate the baseline, then run psalm again with the patch from the PR applied.

Doing this, we could try a higher level than 8.

Increasing the level of the codebase itself could be done separately, running psalm manually from time to spot things.

We can also try suppressing some known errors (e.g. class X only exists with low deps, so let's suppress "class X doesn't exists" in psalm.xml).

I've update the PR to run twice. First on the base to generate a baseline, then on the patch.
The job currently fails because there is no psalm.xml on the 5.x branch. See a job on my fork: https://github.com/Nyholm/symfony/runs/1969878925

This solution is great because we dont have to generate/maintain the baseline. However, you cannot run psalm locally... or you can, but you have no baseline so you will get plenty of errors.

... but that is also true for PHPUnit and php-cs-fixer. I get errors when I run them on the full project too.

In practice, that'd mean that we would run psalm only on PRs, and for each PR, we would first run psalm on the fork point from the target branch to generate the baseline, then run psalm again with the patch from the PR applied.

Or we could just dump the baseline into a file that we commit because that's how the tool is designed to work.

I fear that we unnecessarily create our own special way of using a standard tool here, a way that only works in our CI. This will make it incredibly hard for contributors to understand and fix problems before creating a PR.

Or we could just dump the baseline into a file that we commit because that's how the tool is designed to work.

I think this could be useful for making the codebase more psalm-readable. But this PR is trying to achieve another goal, which is to spot trivial bugs that are hard to catch by human review (e.g. missing use or \).

I fear that we unnecessarily create our own special way of using a standard tool here, a way that only works in our CI. This will make it incredibly hard for contributors to understand and fix problems before creating a PR.

I don't anticipate this hardness at all. Ppl will just see reports about things they just contributed to.

About "the standard way to use psalm", this relates to my previous sentence: psalm-readability is not the goal of this PR.

Baby steps FTW.

I think Im done with this PR. Baby steps.

I will work on a way to help contributors to run psalm locally, but that will go in a separate PR. This PR is to add some more automated review comments.

If we dont like the experience or if it will be too many false positives, we have to reconfigure or reconsider.

Baby steps FTW.

The baby step for me would be: Introduce and use the tool as documented, plain and simple. Don't invent anything around it. If we see that this does not work for us, iterate.

Here is an example PR: Nyholm#16

For the record, I ran psalm on the String component, all reports are false-positives to me.

I'm fearing a bit that adding these checks to our review process might end up with more noise than signal, forcing eg contributors to write code in special ways just to please psalm, or ending up with a stream of "please ignore" comments from core-team members.

But I'm still 👍 to give it a try.

About making the codebase psalm-compliant: if we take the String component as an example, I do understand the theory, but in practice, this is not ready. I feel like achieving this target should be a joint effort with the authors of psalm, because at least some of those false positives should be fixed into psalm - aka we should not work around them.

I think it will find issues like this:

The is no RawMessage::getHeaders(). Either the doc block is wrong or this is a bug.

Thank you @Nyholm.

Thank you all for the input an reviews

WOW. I was about to offer Psalm + PHP Stan full scan and fixes on the whole framework...
I guess someone had the same ideas ^^

I raised readability optimizations recently, and scanned with both tools and found out hundreds of things to do. That, and PHP 8 usage, could lead to massive upgrades. That's good news.

"It might not be the perfect code, but damn it's getting close to it".

Massive thumbs up to that initiative.

🧙‍♀️ By the way, I know this sounds crazy, but I've always thought I could do a whole Psalm / PHPStan full scan and correction (after reports and thoughts exchanges) on the whole framework.
I've already done such a thing, up to one million changes, on a repository. With regexes and PHPStorm, that is close to something a human can achieve without falling into madness.

Do you think (in addition to the proposal made here) that such an adventure on, let's say, 4.4+ (or 5.0+, IDK), would be a lost cause or a huge step forward? This represents over 50K edits on the repo, it's titanic, but would pobably be amazing. Considering the number of concurrent PRs, this would be done step by step, as a (very) long process, to avoid conflicts as much as possible.

All edits would be strictly silent, and overrun by a later PHP-CS-Fixer fix command and this repo usually does.

Is this totally foolish and am I out of my mind, or do think there's any chance that this is feasible?
(hope this is not out off-topic, too)

@DocFX the general rule has been reminded to us by Fabien in #40312 (comment)

So: if there are bugs, we welcome fixes.
Otherwise, changing for the sake of pleasing a static analyzer is out of the rule.

@DocFX I appreciate your enthusiasm, but we're still in the process of figuring out how to use this tool.

Symfony's codebase is probably endgame for static code analyzers. Symfony contains quite some optimized low-level code that we don't want to change just to make a tool happy. In addition to that, the codebase contains BC layer code that will error a lot because it uses (allegedly) missing classes/interfaces and checks for seemingly impossible cases.

We want the 4.4 branch to remain stable, so we should only change code there if we actually fix a problem with the change.

The good news is that for new features, accessibility for static code analyzers will probably improve because the Psalm analysis is taken into account as CI check.

Yeah, no worries. I had that in mind, too. But I just wanted to see if there could be some choices made (like "pick this one specific rule and just this one cause it's useful". That's all I had in mind: see if rules are relevant and if there was a gain. :)

But yeah, got the point and expected the outcome. Plus, too many PRs currently waiting. :)

There's a bit of noise on Twitter and Reddit about this being some sort of seismic event.

Symfony is obviously already amazing, and I doubt Psalm will have a big impact on this project, but I think it might help improve the code quality of new PRs on this repo, just as Psalm has helped improve the code quality of PRs in PHPUnit's repository (and other similar open-source projects).

Good luck with this, and don't hesitate to ask questions. If you get this running with a baseline I'll add Symfony to the list of projects Psalm checks in CI to avoid typechecker regressions in future Psalm releases.