-
-
Notifications
You must be signed in to change notification settings - Fork 9.6k
PHP 7.4 breaks resiliency when loading classes with missing parents #32995
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Comments
My first silly alternative: a subprocess. If it fatals out, class exists 😂 use Symfony\Component\Process\PhpProcess;
require_once $autoloadScript = __DIR__.'/vendor/autoload.php';
function _class_exists(string $autoloadScript, string $class, bool $autoload = true): bool
{
$process = new PhpProcess(sprintf(
'<?php require_once %s; exit(class_exists(%s, %s) ? 0 : 1);',
var_export($autoloadScript, true),
var_export($class, true),
var_export($autoload, true)
));
return 1 !== $process->run();
}
var_dump(_class_exists($autoloadScript, Fatality\InvalidClass::class)); |
The problematic part of the current covariance implementation is step 5: Class To me, this does not sound like a completely unsolvable problem, but I know too little about the php core to actually contribute to a better solution here. 😕 |
One solution could be for php to make a difference between "is_a", "is_subclass_of" kind of functions and the regular "new" operator. At the start of an "is_a" or ""is_subclass_of" function php could "bookmark" its state (regarding classes loaded) and then revert to that state if it runs into that kind of fatal error, and convert the fatal error into a regular php exception. Just my thoughts about what could be done. |
@vudaltsov that'd work but would be very slow. An alternative might be to use |
This sounds possible, in principle. I'm a bit wary because it also makes the behavior unreliable -- you may or may not get an error depending on pretty random changes in the class hierarchy (such as swapping the order of implementation of two interfaces).
Can you point to some of the use cases? My gut reaction here is that Symfony is doing something ... not great and should (in the long term at least) move away from doing it. |
Sure, thanks for asking! Here is what I gathered:
All these cases must be resilient to missing parents since it's just fine ignoring the failure and skipping the class. Missing parents happen all the time when optional dependencies are not installed in the |
If the behavior stays in PHP itself, I think we can resolve our usage in Symfony by actually creating the missing classes / interfaces / traits on the fly in an autoload function, mark them with a special interface / special trait, and then check through reflection if the loaded class / interface / trait implements or use the marker. Something like that (use nikic/php-parser) : spl_autoload_register(function ($class): void {
/*if ($class === $this->resource) {
return;
}*/
$needingClass = null;
foreach (debug_backtrace() as $backtrace) {
if ('spl_autoload_call' === $backtrace['function'] && $backtrace['args'][0] !== $class) {
$needingClass = $backtrace['args'][0];
}
}
if (!$needingClass) {
return;
}
$traverser = new NodeTraverser();
$traverser->addVisitor(new class($class) extends NodeVisitorAbstract {
private $class;
public function __construct(string $class)
{
$this->class = $class;
}
public function enterNode(Node $node) {
if ($node instanceof Class_) {
foreach ([
$node->implements,
$node->extends ?: [],
] as $i => $items) {
foreach ($items as $item) {
if ((string) $item !== $this->class) {
continue;
}
if (0 === $i) {
$code = $this->getInterfaceCode();
} else {
$code = 'class %s implements \Symfony\Component\Config\Resource\MarkerInterface { }';
}
break;
}
}
// If the needing class does not implements or extends the wanted class, it has to be a trait.
$code = $this->getTraitCode();
} elseif ($node instanceof Interface_) {
$code = $this->getInterfaceCode();
} elseif ($node instanceof Trait_) {
$code = $this->getTraitCode();
} else {
return;
}
if (false !== $len = strrpos($this->class, '\\')) {
$code = 'namespace '.substr($this->class, 0, $len).'; '.$code;
$class = substr($this->class, $len + 1);
}
eval(sprintf($code, $class));
return NodeTraverser::STOP_TRAVERSAL;
}
private function getInterfaceCode(): string
{
return 'interface %s extends \Symfony\Component\Config\Resource\MarkerInterface { }';
}
private function getTraitCode(): string
{
return 'trait %s { use \Symfony\Component\Config\Resource\MarkerTrait; }';
}
});
$traverser->traverse((new ParserFactory())->create(ParserFactory::ONLY_PHP7)->parse(file_get_contents((new \ReflectionClass($needingClass))->getFileName())));
});
$hasMarkerTrait = static function (\ReflectionClass $refl) use (&$hasMarkerTrait): bool {
foreach ($refl->getTraits() as $trait) {
if (MarkerTrait::class === $trait->getName()) {
return true;
}
return $hasMarkerTrait($trait);
}
return false;
};
class_exists(ClassWithNotExistingParentOrInterfaceOrTrait::class);
$refl = new \ReflectionClass(ClassWithNotExistingParentOrInterfaceOrTrait::class);
if ($refl->implementsInterface(MarkerInterface::class) || $hasMarkerTrait($refl)) {
throw new \ReflectionException();
} |
Let me abstract away from the PHP language and look from a logical point of view. To me there is a contradiction between the fact that class
If we get back to PHP, I understand that internally |
@fancyweb that would be very fragile, because then the related behavior of standalone components (eg cache miss instead of fatal error) would be tightly coupled to this very special autoloader. |
I think there's a few steps that could be improved to lower the bug from happening.
OK this one is tricky, and this is why I don't love annotations and where I work we don't use them (at all).
The human friendly reporting is useful and nice, but it's not necessary. Developers are developers, they can debug themselves. Moreover, auto-wiring tends to create false positives with errors and enforce to strict file naming which sometime can be a pain. From my perspective, auto-wiring should only be used for final application code, not within bundles (they never be using it, especially to be much more strict and resilient to this king of magic-triggered errors), considering that opinion, if a parent class is missing, just let it crash.
In that case, class_exists() can be called on prior classes in the hierarchy, and most of this bug occurrences could be solved that way, I think.
Isn't this failing because of the two previous points ?
A manual cache clear when delivering should actually solve that. We do force manual cache clear on every delivery on every project and I think that most people also do it. Even if Symfony is very resilient with cache, we still experience bad bugs or edge case behaviours quite often not doing so. Trying to let the framework auto-obsoleting its cache payload on delivery seems a very dangerous thing to do. In any case, if you are attempting a hotfix or critical prod bugfix and don't want to interfere with runtime performance during that time, or avoid unnecessary downtime, your developer must write a proper, finer deploy procedure that manually drops what needs to be dropped. |
@pounard that'd mean undoing the progress of many years of contributions. We built all these behaviors because users reported they would be desired. We should try harder to me. |
The problem (in a nutshell) is that you can't safely call class_exists() on a class without in some special cases risk running into a fatal error. That happens in this case if the class you are testing does kind of exist, but is invalid because it is missing an interface. It would be much better to be able to get a false result without getting a fatal error. You cannot catch fatal errors. |
What are you asking for then, that is less than https://bugs.php.net/78351? I am afraid I missed that. |
This sounds very much like an issue that php should be solving, as it's also php changing this behavior. Until then, it sounds semi-reasonable to have the container first collect the class definitions and in a sub-process and tries to autoload them. While not an ideal solution, I can't think of another sandboxed way of attempting to load it without bringing php in a state where it can't continue. If php could solve it, class unloading would probably what it needs, but I'm no expert on this. |
I think you're exaggerating a bit, it's not about removing everything of auto-wiring or caching, just refining a bit a few bits to be more resilient. I think doing more specialized class_exists() calls that start with classes prior in the hierarchy wouldn't hurt at all usability, it would just need bundle and core developers to be a bit more careful, and I guess this would get rid 90% of sides effects due to this PHP change. All that happened on my projects actually could have been avoided doing so. |
Possibly a stupid suggestion: Instead of trying to catch missing interfaces during inheritance, why not avoid them in the first place?
This seems like a more robust, clearer and faster solution than trying to declare the class and then recovering from a failure. It clearly shows that the class is optional and will correctly work with a vanilla autoloader and class_exists(). I think that independently of the use of the class loading resiliency mechanism for specific purposes (like cache warmup), what Symfony imho does wrong and should preferably stop doing as soon as feasible, is to bake this functionality into the general class_exists() functionality. class_exists() should not implicitly and silently recover from a missing interface, for much the same reason it should not silently recover from a parse error in the loaded file. A priori, a missing interface is just a missing interface (say a typo) and should never be silently ignored. Symfony is hijacking basic language behavior here, in a way that creates incorrect expectations that do not hold anywhere outside the Symfony ecosystem. Using something like this internally is fine, but exposing it to users in such a manner is not. |
@nikic I'm not a native english speaker, I think that it is exactly what I was suggesting.
Actually it does bring a lot of goodness for lazy developers, it allows very quick Symfony app setup and it's very nice to use. But I agree, this is black magic. Symfony dependency-injection component should be more resilient and more careful, I'm sure there are solutions that would allow the same level of comfort for users with less magic. @nicolas-grekas I agree that having PHP be a bit more robust regarding this behaviour would be much easier for Symfony, and probably some other magic-based frameworks, but it doesn't prevent you from preparing for the case where PHP developers would mark this as a wont-fix. To be honest, I'm glad that PHP doesn't allow broken code to be loaded, and VM kept in a broken state. |
Because we don't know the parents. Tight coupling to the class hierarchy is fragile and doesn't work across all versions of dependencies since hierarchies change. And even if we were, that wouldn't fix the other use cases.
The Symfony ecosystem is the PHP ecosystem, see how many rely on the components. Many packages rely on the behavior seamlessly and would learn about the topic in a bad way. Having the engine kill itself on a class_exists check is equally wrong to me and that's why we're asking for help here.
Resiliency when a fatal error occurs is not possible. Nobody is asking to load broken code in the engine btw. |
What about something similar than https://www.php.net/manual/en/reflectionclass.isinstantiable.php? |
That would be doable, however that would decrease the DX as well. In this example, when attempting to use the The code new Foo(); would normally result in
and with your change applied, the error would be
That error message would rather confuse people, I'm afraid. |
I don't understand what you mean here. To be clear, what I have in mind is a check when the class is declared, not when someone is trying to use it. If a class has optional dependencies, then it is an optional class itself -- it should only be declared if the dependencies exist.
That sounds worse: People should not be implicitly pulling in non-standard behavior for class_exists() just because they happen to use a Symfony component. |
To be clear: We're still talking about internal logic of the DI component that is executed during cache warmup. Userland code still gets the vanilla behavior of |
When a bundle auto-configures itself to optionally use classes from another packages, if the case arise, it's a good thing to check deeper in the class hierarchy. It seems to be a sane approach. Moreover, by checking some interface or class higher in the class hierarchy, if it disappear since a version, or was introduced at a specific version, it also covers the versions you didn't anticipated and deactivate gracefully the feature in non anticipated version constraints usages. |
Oh sorry I read too fast. This could work only for code we have control over. But there are many third-party packages that will never accept to add these checks. Also, from my experience, PHP (maybe opcache?) ignores such early return statements sometimes and loads the declarations before executing the code itself. That's why we always wrap such conditional classes inside "if"s. Anyway, doing this for every single class would be ugly and would be a workaround for an issue that lies in the engine really. Ideally, the engine should never "fatal error", because that means taking control away from userland. That's exactly why |
I didn't understood this as well, sorry. |
To be fair, I don't think avoiding all panics is possible, no matter the language, the VM, nor how hard you try. It's sad Symfony actually stumbles upon this one. |
You can take #32396 as an concrete example. The class \Symfony\Bridge\Doctrine\Form\DoctrineOrmTypeGuesser implements \Symfony\Component\Form\FormTypeGuesserInterface But if you in your composer.json do not have symfony/form and it is not pulled in by any other package you use, then the Form classes and interfaces will not exist in the vendor directory. Then the symfony DI compenent needs to make sure that \Symfony\Bridge\Doctrine\Form\DoctrineOrmTypeGuesser is also not available as a service. But the problem is that it cannot do so without stumbling into a fatal error. I myself got this error when using ApiBundle, where we have no use of Forms. Having Symfony just remove the redundant service \Symfony\Bridge\Doctrine\Form\DoctrineOrmTypeGuesser would have been expected behaviour. |
@terjebraten-certua this is a real question, I don't understand at which point you really cannot check for the interface existence before registering the service. Can't you do a |
This is not true. You should study #32396 as an example.
The service When the package There are many other examples of packages providing optional services to other packages that may or may not be there. No auto-wire logic is involved. |
Really, if the bundle lets the service be in the configuration when the class does not exists, it seems to be the bundle responsibility to remove it before compiling. You cannot just let an undefined service in the container and expect it to clean up your mess. In my opinion, the doctrine bundle plays a dangerous game letting incomplete or non existing definitions in the container. The container cannot be held responsible for it if it breaks. The bundle lacks robustness. |
Yes, that was a beautiful feature of the dependency container that worked for a long time. It is a sad thing that a resilient and good dependency container can no longer have a feature it used to have, because of a BC break in the way PHP works. |
I think that in this example, the doctrine bundle, it was more like a nice side effect. As for fixing the doctrine bundle, it would be extremely simple: put every service form related within a Resources/config/form.xml file, and load it conditionally in the extension class. It would be a single xml file and a two-liner patch, easy, and backward compatible with older versions of Symfony. |
To fix the doctrine bundle, you will have to split the form related services out from And this is just one example of many. When/if the dependency container must do without its above mentioned resilience feature, there will be many places and bundles that will need to be updated. |
I guess that they'll need to be fixed. I'm not sure this bug will trigger that often, considering that it's conceptually extremely dangerous to have multiple level of optional class in a single inheritance tree, when it happens, it highlight, in my opinion, very complex and out of control code. Sometime, it takes a tiny change a behaviour in PHP to give an opportunity to developers to make their code more robust :) |
@TerjeBr I am sorry if sometime I can be rude, I do not mean too, I'm not a native english speaker and sometime can be a bit rough in my words. I never meant to be insulting or mean. I know that, for Doctrine maintainers and all others, that contributing to open source is not always easy and maintaining packages is hard. That said I did a PoC for Doctrine bundle: doctrine/DoctrineBundle#1013 |
Wouldn't all problems be solved if services (classes) are only added to the container if you intend on initializing them? If you try to initialize them with missing dependencies, it rightfully crashes. The error message is a bit confusing for this though. In the end the responsibility of the service registration lies with bundles (or your application), which means it can all be solved in bundles by doing feature checks, such as class or interface exists. For optional dependencies, you have to check whether the optionally present class or interface is present and that should be enough. I don't exactly like the way PHP handles it, yet in the end it's an error that is caused by us forgetting to check the existence of the optional dependency, not PHP. |
I did a few benchmarks recently, and I was honestly surprised by the results on container compile phase, it was, on the project I benched, Twig warmup taking up to 50% of the time, and property access component doing whatever it does taking another 20% to 30%, in conclusion, container compilation was decently fast (I didn't suspected those results honestly). Considering https://github.com/Roave/BetterReflection it would actually be a legit use case for it, and I expect it to be decently fast as well, hoping there's some kind of cache mechanism within (and if there's not, it would still be possible in last resort implementing one around it). There's one thing to consider thought, it brings a few (usually dev only) dependencies to production releases, amongst them |
For cross-component conflicts, that's not an issue. We can easily ensure that both components support a common version. External conflicts are a much bigger issue for the ecosystem (as we control only part of the conflicting packages). |
@stof you're absolutely right. It still would add one complexity further in maintenance then it needs to coordinate property-info active maintainers with the container one in case of any change. As it exists, I could install the 4.3 version of the property-info component and any version from 2.x to actual of the container component, without going fullstack, and without creating a dependency problem, they don't share common dependencies (at least not in their actual composer.json files, except maybe for dev dependencies). But that's probably a negligible problem. |
Sorry, I see I've misunderstood the issue. If a class is defined with a missing parent, we've always got an error of this nature during autoload:
Which is great. This issue only concerns the special case of throwing an exception from the autoloader, in order to be able to "recover" from the above error, which I now see is a very dubious loophole indeed. Everything I've said before is irrelevant. |
What can we do to move forward here? php 7.4 has reached RC stage and the behavior described above is still present. I would assume that 7.4.0 will ship with it. |
@nikic can we hope for a patch from your side on php-src to implement this? Because on ours, we're out of luck. Yes, we can check the parent on classes we've control over, but that's never going to solve all items listed in #32995 (comment) Loosely related: as explained, My main concern is BC of course. Thanks for your understanding and help! |
See https://github.com/symfony/symfony/pull/33539/files#diff-4d4aa4c7bcd6435d73fdfc6f6c010e4b for what we could do on code we have control over. |
This is a fix for symfony/symfony#32995. The behavior is: * Throwing exception when loading parent/interface is allowed (and we will also throw one if the class is simply not found). * If this happens, the bucket key for the class is reset, so it's possibly to try registering the same class again. * However, if the class has already been used due to a variance obligation, the exception is upgraded to a fatal error, as we cannot safely unregister the class stub anymore.
Closed by php/php-src#4697, thank you @nikic! |
That's good news, at least it shuts down this discussion and everyone is happy. |
Wow, thanks a lot, @nikic! |
Thanks @nikic, your work on this is highly appreciated! ❤️ |
…-grekas) This PR was merged into the 3.4 branch. Discussion ---------- Re-enable previously failing PHP 7.4 test cases | Q | A | ------------- | --- | Branch? | 3.4 | Bug fix? | no | New feature? | no | Deprecations? | no | Tickets | Fix #32995 | License | MIT | Doc PR | - The remaining PHP 7.4 issue has been fixed by Nikita in php/php-src#4697 This should be green once Travis updates their php7.4snapshot Commits ------- 9e4e191 Re-enable previously failing PHP 7.4 test cases
This issue is a follow up of #32395, so that we can focus on the underlying cause and skip the forum it has become. The closest related PHP bug report is https://bugs.php.net/78351, but it has been described as a feature request, while this is really a blocker for supporting PHP 7.4 as of now in Symfony.
PHP 7.4 support is almost done - all remaining issues have pending PRs. This one currently triggers phpunit warnings so that we can spot the affected test cases easily. Here is an example job that has the warnings: https://travis-ci.org/symfony/symfony/jobs/568248854, search for
PHP 7.4 breaks this test, see https://bugs.php.net/78351
.In case anyone wonders why we built this resiliency, the related failing test cases highlight an interesting variety of use cases, summarized in #32995 (comment)
See #32995 (comment) for a reproducer.
@nikic described what happens in PHP7.4 with this example:
<quote>
What happens now is the following:
C
. The classC
is registered provisionally.B
. The classB
is registered provisionally.A
. The classA
is registered provisionally.A
is verified (trivially) and registered fully.B
is verified and registered fully (and may already be used). During the verification it makes use of the fact thatC
is a subclass ofB
, otherwise the return types would not be covariant.DoesNotExist
, which throws an exception.After these steps have happened ... what can we do now? We can't fully register the class
C
, because the interface it implements does not exist. We can't remove the provisionally registered classC
either, because then a new classC
that is not a subclass ofB
could be registered, and thus violate the variance assumption we made above.</quote>
This leaves little room but @ausi suggested this:
<quote>
Would it be possible to mark a provisionally registered class as “in use” as soon as it is referenced somewhere?
This way we could determine in an exception case if it is OK to remove the provisionally registered class and gracefully recover or if we have to throw a fatal error.
In your example this would mean that once class B is verified it would mark class C as “in use”. When loading DoesNotExist throws the exception then, we would throw a fatal error because class C is “in use”.
This way we could gracefully recover for most cases while resulting in a fatal error for the impossible cases, which sounds like a perfect trade-off to me.
</quote>
Alternatively, @smoench proposed to use https://github.com/Roave/BetterReflection to do the check we need. But I fear this would have an unacceptable performance impact. Compiling the container is already slow enough, we cannot add a preflight parsing of every service class I fear.
I'm stuck, help needed :)
The text was updated successfully, but these errors were encountered: