Thanks to visit codestin.com
Credit goes to github.com

Skip to content

[Serializer] Normalizer incorrectly caching allowedAttributes when typed properties are used. #36594

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
fl0cke opened this issue Apr 27, 2020 · 12 comments · Fixed by #43469
Closed

Comments

@fl0cke
Copy link

fl0cke commented Apr 27, 2020

Symfony version(s) affected: 5.0.5

Description
When using typed properties in a class, the serializer correctly skips those that are not initialized yet. However, when serializing a collection of objects that are instances of the same class, the serializer's attribute caching kicks in, which may lead to "Typed property must not be accessed before initialization" errors, e.g. when property "a" of object 1 is initialized, but on object 2 it is not.

How to reproduce

class Test {
    public string $a;
    public string $b;
}

$test1 = new Test();
$test1->a = "a";
$test1->b = "b";

$test2 = new Test();
$test2->a = "a";

$objects = [$test1, $test2];

$serializer->serialize($objects, "json");

(where $serializer is the default serializer service that is configured by Symfony when enabling the serializer component)

Possible Solution
The "allowed" status of typed properties must not be cached and checked every time an object is serialized.

@fl0cke fl0cke added the Bug label Apr 27, 2020
@fl0cke fl0cke changed the title Normalizer incorrectly caching allowedAttributes when typed properties are used. [Serializer] Normalizer incorrectly caching allowedAttributes when typed properties are used. Apr 27, 2020
@carsonbot
Copy link

Hey, thanks for your report!
There has not been a lot of activity here for a while. Is this bug still relevant? Have you managed to find a workaround?

@carsonbot
Copy link

Just a quick reminder to make a comment on this. If I don't hear anything I'll close this.

@tihomir-stefanov
Copy link

I have a similar problem. Even if we don't serialize them at once, but one after another, UninitializedPropertyException is thrown.

class Test {
    public string $a;
    public string $b;
}

$test1 = new Test();
$test1->a = "a";
$test1->b = "b";
$serializer->serialize($test1, 'json');

$test2 = new Test();
$test2->a = "a";

$serializer->serialize($test2, 'json');

@carsonbot carsonbot removed the Stalled label Mar 18, 2021
@carsonbot
Copy link

Hey, thanks for your report!
There has not been a lot of activity here for a while. Is this bug still relevant? Have you managed to find a workaround?

@juuuuuu
Copy link
Contributor

juuuuuu commented Sep 19, 2021

I had the same problem few months earlier. Let me check again if the issue is still there.

@carsonbot carsonbot removed the Stalled label Sep 19, 2021
@ivannemets-sravniru
Copy link

ivannemets-sravniru commented Oct 5, 2021

This is still an issue and I think it's quite critical... probably this is not so spread just because not much people use PHP 8 yet in production application

Preconditions:

  • Symfony v5.2.14
  • PHP 8.0.11

Description:

So in my use case I have a queue processed by a consumer configured using symfony messenger component
bin/console messenger:consume some_name --limit=10 // allows processing 10 messages by this worker

A simplified example - each message in queue contains reference (ID) to a Doctrine entity, and MessageHandler uses serializer to serialize a DTO object to JSON, like this:

class Dto
{
    public array $requiredData; // MUST be set
    public array $optionalData; // CAN be not initialized (expected to be ignored by serializer when not initialized)
}

class Message
{
    public int $entityId;
}

class MessageHandler
{
    public function __construct(private SerializerInterface $serializer, private EntityRepository $repository)
    {
    }

    public function __invoke(Message $message)
    {
        $entity = $this->repository->find($message->entityId);
        $dto = new Dto();
        $dto->requiredData = [
            'foo' => $entity->getFoo(),
            'bar' => $entity->getBar(),
        ];
        if ($entity->getBaz()) {
            $dto->optionalData = [
                'baz' => $entity->getBaz(),
                'jaz' => 42,
            ];
        }
        $json = $this->serializer->serialize($dto, 'json');
        // the rest business logic goes here...
    }
}

So what actually happens...
When two messages processed within the same consumer (worker) and the first message's entity has not empty baz property ($entity->getBaz()), on the first call serializer caches allowed attributes for Dto class to be like these: ['requiredData', 'optionalData']
(see \Symfony\Component\Serializer\Normalizer\AbstractObjectNormalizer::getAttributes)

but if the second message's entity has empty baz property $dto->optionalData will not be initialized and this will cause type error: "Typed property Dto::$optionalData must not be accessed before initialization" since while normalizing the second $dto normalizer already has the allowed attributes cached in \Symfony\Component\Serializer\Normalizer\AbstractObjectNormalizer::$attributesCache so normalizer tries to get value from $dto->$optionalData property which is not initialized

I hope my description helps someone...

UPD: Temporary workaround we are using now is to force workers to process only 1 message per run so that cache will not be used bin/console messenger:consume some_name --limit=10 but this is poor solution.. this should definitely be fixed in Symfony source code

@lyrixx
Copy link
Member

lyrixx commented Oct 5, 2021

Do someone want to submit a PR for this issue?

@ivannemets-sravniru
Copy link

ivannemets-sravniru commented Oct 5, 2021

@lyrixx the question is - what is the proper fix for this issue..

The only solution I can think of so far - is to just remove usage of \Symfony\Component\Serializer\Normalizer\AbstractObjectNormalizer::$attributesCache property, don't know if this may cause a notable performance decrease though... But as long as \Symfony\Component\Serializer\Normalizer\ObjectNormalizer::extractAttributes and \Symfony\Component\Serializer\Normalizer\PropertyNormalizer::extractAttributes ignore uninitialized properties...

if (!$reflProperty->isInitialized($object)) {
    unset($attributes[$reflProperty->name]);
    continue;
}

... caching attributes per class is wrong since attributes list may differ for different instances of the same class or even for the same object after it's updated. Imagine a use case when you may need to serialize the same object twice - the second time after setting a single property that wasn't initialized on the first call, for example:

$object = new Dummy();
$object->foo = 42;
$json = $serializer->serialize($object, 'json');
// send it somewhere..
$object->bar = 'baz';
$json = $serializer->serialize($object, 'json');
// send json with additional `bar` property somewhere else...

in this case if attributes are cached on the first call, the second json string will not contain bar property (as I would expect) since only foo attribute was cached in \Symfony\Component\Serializer\Normalizer\AbstractObjectNormalizer::$attributesCache on the first call

So, here is the patch I'm currently using in my project and it works fine for me...

diff --git a/vendor/symfony/serializer/Normalizer/AbstractObjectNormalizer.php b/vendor/symfony/serializer/Normalizer/AbstractObjectNormalizer.php
index fbb8b86..c62d67b 100644
--- a/vendor/symfony/serializer/Normalizer/AbstractObjectNormalizer.php
+++ b/vendor/symfony/serializer/Normalizer/AbstractObjectNormalizer.php
@@ -94,7 +94,6 @@ abstract class AbstractObjectNormalizer extends AbstractNormalizer
 
     private $propertyTypeExtractor;
     private $typesCache = [];
-    private $attributesCache = [];
 
     private $objectClassResolver;
 
@@ -234,25 +233,15 @@ abstract class AbstractObjectNormalizer extends AbstractNormalizer
     }
 
     /**
-     * Gets and caches attributes for the given object, format and context.
+     * Gets attributes for the given object, format and context.
      *
      * @return string[]
      */
     protected function getAttributes(object $object, ?string $format, array $context)
     {
-        $class = $this->objectClassResolver ? ($this->objectClassResolver)($object) : \get_class($object);
-        $key = $class.'-'.$context['cache_key'];
-
-        if (isset($this->attributesCache[$key])) {
-            return $this->attributesCache[$key];
-        }
-
         $allowedAttributes = $this->getAllowedAttributes($object, $context, true);
 
         if (false !== $allowedAttributes) {
-            if ($context['cache_key']) {
-                $this->attributesCache[$key] = $allowedAttributes;
-            }
 
             return $allowedAttributes;
         }
@@ -263,10 +252,6 @@ abstract class AbstractObjectNormalizer extends AbstractNormalizer
             array_unshift($attributes, $mapping->getTypeProperty());
         }
 
-        if ($context['cache_key'] && \stdClass::class !== $class) {
-            $this->attributesCache[$key] = $attributes;
-        }
-
         return $attributes;
     }
 

also added a test to avoid problems in future since it's pretty important in my project

class SerializerTest extends KernelTestCase
{
    /**
     * Test isolated serialization calls (new serializer instance for each assert).
     */
    public function testIsolatedCalls(): void
    {
        self::bootKernel();
        $this->assertAllPropertiesSerialized();
        self::bootKernel(); // reboot to reset serializer instance
        $this->assertUninitializedPropertiesIgnored();
    }

    /**
     * Test that first serialization with all properties being set
     * does not break the second serialization with uninitialized property
     * since extra attributes should be serialized on the first call.
     *
     * @depends testIsolatedCalls
     */
    public function testTwoConsecutiveCalls(): void
    {
        self::bootKernel(); // both asserts use same serializer instance
        $this->assertAllPropertiesSerialized();
        $this->assertUninitializedPropertiesIgnored();
    }

    /**
     * Test that first serialization with uninitialized property
     * does not break the second serialization with all properties being set
     * since extra attributes should be serialized on the second call.
     *
     * @depends testIsolatedCalls
     */
    public function testTwoReverseConsecutiveCalls(): void
    {
        self::bootKernel(); // both asserts use same serializer instance
        $this->assertUninitializedPropertiesIgnored();
        $this->assertAllPropertiesSerialized();
    }

    private function assertAllPropertiesSerialized(): void
    {
        $user = new DummyUser();
        $user->firstName = 'foo';
        $user->middleName = 'bar';
        $user->lastName = 'baz';
        $this->assertSame(
            '{"firstName":"foo","middleName":"bar","lastName":"baz"}',
            self::$container->get('serializer')->serialize($user, 'json')
        );
    }

    private function assertUninitializedPropertiesIgnored(): void
    {
        $user = new DummyUser();
        $user->firstName = 'foo';
        $user->lastName = 'baz';
        $this->assertSame(
            '{"firstName":"foo","lastName":"baz"}',
            self::$container->get('serializer')->serialize($user, 'json')
        );
    }
}

class DummyUser
{
    public string $firstName;
    public string $middleName;
    public string $lastName;
}

If this solutions looks acceptable, I can create a PR

@lyrixx
Copy link
Member

lyrixx commented Oct 5, 2021

I already noticed this issue, and IMHO, it should be fixed. But we must be very aware that some very minor could hurt performance a lot. And, while I did not bench your patch, I think it'll decrease performance a lot.

I don't have a better patch ATM, and I'm running out of time for working on this issue :/

(note: I have edited your comment to add some color)

@ivannemets-sravniru
Copy link

@lyrixx
On the other hand - is it actually correct to rely on normalizer ignoring uninitialized attributes?

As another potential solution, instead of ignoring uninitialized properties while collecting attributes in \Symfony\Component\Serializer\Normalizer\AbstractObjectNormalizer::extractAttributes, we probably could check property initialization while building normalized data array in \Symfony\Component\Serializer\Normalizer\AbstractObjectNormalizer::normalize - when $this->getAttributeValue(...) is called and just ignore uninitialized attributes. This approach would allow using attributes cache per class, which is good from performance standpoint... but needs to be implemented and carefully tested

@lyrixx
Copy link
Member

lyrixx commented Oct 5, 2021

I think I remember another issue about this topic. But there is already some support for what your say :

public const SKIP_UNINITIALIZED_VALUES = 'skip_uninitialized_values';

But you idea seems good. Wanna give a try?

@ivannemets-sravniru
Copy link

Thanks for sharing the link to SKIP_UNINITIALIZED_VALUES flag (I didn't know it will be released with v5.4)!
It looks pretty similar to the suggestion I'v explained in my previous comment, but it does not solve the attributes caching issue itself (which I described here)

I do wanna give it a try, but I don't have enough time at the moment, so I'll keep that in mind and see if I can get to it in the next few weeks..

fabpot pushed a commit to ivannemets-sravniru/symfony that referenced this issue Oct 16, 2021
@fabpot fabpot closed this as completed in 732acf5 Oct 16, 2021
derrabus added a commit that referenced this issue Oct 19, 2021
* 5.4: (35 commits)
  fix: Improve FR validators translation
  [Notifier] Add push channel to notifier
  Fix CS
  [Lock] Split PdoStore into DoctrineDbalStore
  [Cache] Split PdoAdapter into DoctrineDbalAdapter
  Add swedish translation for issue #43458
  [HttpClient] fix collecting debug info on destruction of CurlResponse
  Fix CS
  added missing thai translations
  Add missing translations for Chinese (zh_TW)
  [DependencyInjection] fix "url" env var processor
  update translation
  [Serializer] #36594 attributes cache breaks normalization
  Remove untranslated translation for Afrikaans
  [Validator] Add missing validator polish translation
  [Security,Validator] Added missing Latvian translations #41053
  Add the missing translations for Indonesian (id)
  [Validator] Add missing Lithuanian translation
  [Validator] Add missing Czech translation
  replace "ispravna" with "važeća" in translating "valid HTML/CSS"
  ...
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

7 participants