[Serializer] Add FlattenExceptionNormalizer / Fix messenger used with the serializer component #33650

skalpa · 2019-09-20T13:08:01Z

Q	A
Branch?	4.3
Bug fix?	yes
New feature?	yes
Deprecations?	no
Tickets	Fix #32719
License	MIT

This adds a new normalizer for FlattenException objects.

It is against 4.3 and marked as a bug fix, because the Serializer component's current inability to deserialize FlattenException objects breaks Messenger when used with the component (see #32719 for details).

The new FlattenException class of the ErrorRenderer component suffers from the same problem, so once merged it will probably have to be amended on 4.4 to support the ErrorRenderer as well.

skalpa · 2019-09-20T13:56:43Z

The failing test on 7.3 is due to the fact that Travis loads the deps from 4.4.x-dev and can't find the new class there, so I guess that's to be expected.

mleczakm · 2019-09-20T22:00:32Z

What with traceAsString field? Is it properly serialized/deserialized?

skalpa · 2019-09-21T18:10:24Z

@mleczakm That's fixed. Nice catch.

mleczakm · 2019-09-21T19:40:15Z

This fix would also help in #33049

weaverryan

Thanks for the PR!

I think this is the correct approach. I originally chose to use FlattenException because the purpose of it is to be a "serializable" exception. This, of course, works with Messenger's PhpSerializer. I think it makes sense to make it also work with Symfony's serializer. An alternative approach would be for Messenger to store the exception information in a simpler way (without using FlattenException), but that feels like reinventing the wheel: this is the purpose of FlattenException.

dunglas · 2019-09-24T08:02:32Z

Wouldn't it better to serialize the error following/extending the formats defined by RFC 7807 or Hydra (as we do for validation errors: #22150)?
I feel like we're duplicating logic introduced in the new ErrorRenderer component.

Maybe that a better approach would be to:

convert the error renderers as simple normalizers, in the serializer component (as we've done for validation errors)
use the Serializer component as a dependency of the new ErrorHandler component
remove the ErrorRenderer component (it's not too late, it hasn't been tagged yet)

There are several benefits to this approach:

this use case with Messenger is fixed
API Platform will automatically support the new ErrorHandler component (API Platform already has normalizers for \Exception and \FlattenException for most modern JSON error formats including RFC 7807, Hydra and JSON:API)
it removes yet another component dealing with serialization

At least, the error renderers should implement Serializer's interfaces (and these interfaces should probably moved to symfony/contracts).

/cc @yceruto

skalpa · 2019-09-24T14:14:10Z

@dunglas I'm not sure I follow how you'd expect the same set of classes to satisfy the needs of both Messenger and ErrorRenderer as the use cases are totally different.

Also, this PR is about fixing bugs in 4.3, so anything that involves components that haven't been released yet probably won't apply.

In Messenger, objects have to be serialized then deserialized with no loss of information (this PR is actually more about fixing deserialization than serialization).
For error reporting, serialization is a one-way street, potentially destructive, and context dependent (i.e: AFAIK the behaviour of both the api-platform normalizers and ErrorRenderer depend on the value of debug).

As for following RFC 7807 more closely, I'm all for it. However as some fields in the RFC only make sense in an HTTP context, there's not much we could do I guess. Would you have anything else than the following in mind ?

type, title and instance will have to be omitted (that's ok if I understand the RFC)
Rename status_code to status
Rename message to detail

yceruto · 2019-09-24T15:03:16Z

@dunglas I failed to see as well the approach described and its relationship with this PR + ErrorRenderer component in 4.4.

The ErrorRenderer component is not (and should not be) concerned with the full normalization of the FlattenException object or its denormalization at all, but standardized error responses in different formats, included the HTML error/exception page. It's more like (https://github.com/filp/whoops)

I don't know how all those features would fit into the Serializer component instead.

dunglas · 2019-09-24T17:21:44Z

@skalpa

the needs of both Messenger and ErrorRenderer as the use cases are totally different

They don't look that different to me. Also, the benefit of using JSON with Messenger (instead of the native PHP serialize() format) is to be able to consume the messages with other programming languages. It's exactly there are standards for error representations. And converting an internal data structure (such as \Exception) to a public (or even better, standard) representation that can be consumed by another system is exactly the responsibility of the Serialize component.

In Messenger, objects have to be serialized then deserialized with no loss of information

It can be added to extra keys, as done by API Platform (such information could also be useful for developers trying to debug). See https://github.com/api-platform/core/blob/master/src/Hydra/Serializer/ErrorNormalizer.php#L58-L60 for instance.

For error reporting, serialization is a one-way street

Normalizers in API Platform currently only support normalization, but there are no technical reasons preventing to support deserialization.

potentially destructive, and context dependent (i.e: AFAIK the behaviour of both the api-platform normalizers and ErrorRenderer depend on the value of debug).

We have all the infrastructure we need in the Serializer to deal with this. An option could be passed to the context (and we can now set default context options easily) to choose if debug informations for instance should be serialized or not.

As for following RFC 7807 more closely, I'm all for it. However as some fields in the RFC only make sense in an HTTP context, there's not much we could do I guess. Would you have anything else than the following in mind ?

Great. I indeed had something like that in mind.

@yceruto

but standardized error responses in different formats

For instance, in API Platform we already support generating standard error response for RFC 7807, JSON:API and Hydra using normalizers.

If the ErrorHandler component could use any class implementing the SerializerInterface or (the NormalizerInterface), it would allow to reuse those normalizers very easily. Also, FOSRest comes with similar normalizers, and I'm such we can find others. To me it's the responsibility of the Serializer to transform a PHP object (\Exception here) in a standard open format (RFC 7807...).

included the HTML error/exception page. It's more like (https://github.com/filp/whoops)

For (advanced) HTML like this agree. Creating something like whoops is totally out of scope of the Symfony Serializer and should stay in ErrorRenderer. However, it's not the same case for non-HTML renderers. They should be moved to the Serializer component.
Even if it doesn't belong to the Serializer component, the HTML component could also easily implement the SerializerInterface (it still converts an \Exception to a string).

skalpa · 2019-09-24T18:43:45Z

@dunglas I renamed status_code and message as discussed. As the RFC states that status should be a number if present, it is not in the normalized array if the exception has no status code set.

Regarding your other points, I now understand your point of view better, thank you. However, I believe this is a different discussion and don't think this should keep us from merging this PR, which is only for the FlattenException class of the Debug component at this point.

We'll still be able to add support for the ErrorRenderer class and/or use the Serializer in the new components on 4.4 once this is done.

fabpot · 2019-09-27T05:36:03Z

ping @dunglas

dunglas · 2019-09-27T07:14:53Z

We can merge this one, but can we mark it as @experimental in case we have to modify it for ErrorHandler/ErrorRenderer?

skalpa · 2019-09-27T09:37:06Z

@dunglas Depending on what you decide to do, it might be best to create another normalizer for ErrorRenderer, and just retire this one with the Debug component in 5.0.

Anyway, I added the @experimental tag as it can't hurt.

…s (dunglas) This PR was merged into the 4.4 branch. Discussion ---------- [ErrorRenderer] Security fix: hide sensitive error messages | Q | A | ------------- | --- | Branch? | 4.4 | Bug fix? | yes | New feature? | no  | Deprecations? | no  | Tickets | n/a | License | MIT | Doc PR | n/a This PR fixes a security issue. Exception messages must not be displayed except when debugging, because they can contain sensitive data including credentials. For instance, PDO and Doctrine throw exception with message such as `The details are: SQLSTATE[HY000] [1045] Access denied for user 'root'@'db.example.com' (using password: NO)` revealing internal details about the infrastructure usful for an attacker. Also, I still think that ErrorRenderer should be removed in favor of using the Serializer directly (see #33650 (comment)). I'll try to open some PRs to do that in tomorrow. Commits ------- d7d7f22 [ErrorRenderer] Security fix: hide sensitive error messages

nicolas-grekas · 2019-11-28T11:32:53Z

Does #34312 change the approach proposed here? Can you please have a look for 4.4?

monteiro · 2020-05-27T15:34:17Z

I suppose the only thing needed to merge this PR is to change this PR to use the new error handler component (and remove symfony/debug old component) right and put this agaisn't 4.4?

@weaverryan This fix will close #32719 . I tested this locally (to use the new error-handler component), but if I execute the command bin/console messenger:failed:retry -vv I get:

In FlattenException.php line 360:
                                                                                                                                   
  [TypeError]                                                                                                                      
  Return value of Symfony\Component\ErrorHandler\Exception\FlattenException::getTraceAsString() must be of the type string, null   
  returned

BTW, this exception does not discard the event from the queue, but without applying this PR fix, it discarded.

What I'm using to test:

Transport: AMQP (RabbitMQ)
Serializer: My custom serializer (I'm using a lot of code from the Symfony messenger serializer, specially the encoding/decoding stamps).
symfony version: 5.0.8

monteiro · 2020-05-28T08:47:34Z

I found the problem with the error. For some reason the Flatten Normalizer in this PR was not being called when normalizing the FlattenException object.

You see there is a missing trace_as_string parameter, that's why the error.
If I omit the priority on the Flatten Normalizer (in this PR) everything works as expected and the command bin/console messenger:failed:retry -vv does not fail.

monteiro · 2020-05-28T09:02:14Z

src/Symfony/Bundle/FrameworkBundle/Resources/config/serializer.xml

@@ -31,6 +31,11 @@
        <service id="Symfony\Component\Serializer\Mapping\ClassDiscriminatorResolverInterface" alias="serializer.mapping.class_discriminator_resolver" />

        <!-- Normalizer -->
+        <service id="serializer.normalizer.flatten_exception" class="Symfony\Component\Serializer\Normalizer\FlattenExceptionNormalizer">
+            <!-- Run before serializer.normalizer.object -->
+            <tag name="serializer.normalizer" priority="-915" />


If I remove the priority the normalize() method is called (in this PR with priority is not).
denormalize is called with our without the priority.

With this PR (the custom normalizer of this PR was not being called, that's why the trace_as_string was not being set). Result in xdebug:

(just did a throw new \InvalidArgumentException() in an handler to force failing)

monteiro · 2020-05-28T09:02:46Z

src/Symfony/Component/Serializer/composer.json

@@ -25,6 +25,7 @@
        "symfony/property-access": "~3.4|~4.0",
        "symfony/http-foundation": "~3.4|~4.0",
        "symfony/cache": "~3.4|~4.0",
+        "symfony/debug": "~3.4|~4.0",


switch to symfony/error-handler since this component is deprecated?

nicolas-grekas · 2020-06-04T09:25:07Z

Continued in #37087, thank you @skalpa

@skalpa

This PR was squashed before being merged into the 5.2-dev branch. Discussion ---------- [Messenger] Add FlattenException Normalizer | Q | A | ------------- | --- | Branch? | master | Bug fix? | no | New feature? | yes  | Deprecations? | no  | Tickets | Fix #32719 ...  | License | MIT | Doc PR | No docs ...   I have just duplicated the PR #33650 (from @skalpa) but using the new component ErrorHandler and removing the priority. Project that reproduces the bug on Symfony 5.2: https://github.com/monteiro/serializer-pr37087 (all steps are on the README). Since this adds a new class and changes behavior, we add this new feature on the 5.2 branch. Commits ------- 78fbd0a [Messenger] Add FlattenException Normalizer

skalpa requested a review from dunglas as a code owner September 20, 2019 13:08

carsonbot added Status: Needs Review Serializer Bug Feature labels Sep 20, 2019

skalpa force-pushed the flatten-exception-denormalizer branch from 574746d to 482551d Compare September 20, 2019 13:17

nicolas-grekas added this to the 4.3 milestone Sep 21, 2019

skalpa force-pushed the flatten-exception-denormalizer branch from 482551d to ef0b8b3 Compare September 21, 2019 18:06

Added FlattenExceptionNormalizer

733e404

skalpa force-pushed the flatten-exception-denormalizer branch from ef0b8b3 to 733e404 Compare September 21, 2019 18:07

weaverryan approved these changes Sep 23, 2019

View reviewed changes

carsonbot added Status: Reviewed and removed Status: Needs Review labels Sep 23, 2019

maxhelias approved these changes Sep 24, 2019

View reviewed changes

yceruto self-requested a review September 24, 2019 15:09

carsonbot added Status: Needs Review and removed Status: Reviewed labels Sep 24, 2019

Renamed normalized properties names according to RFC 7807

c704b36

Added experimental tag

bfa2ff8

dunglas mentioned this pull request Oct 28, 2019

[ErrorRenderer] Security fix: hide sensitive error messages #34158

Merged

yceruto mentioned this pull request Nov 8, 2019

[ErrorRenderer] Refactor and delegate serializing responsibility to the Serializer component #34288

Closed

nicolas-grekas modified the milestones: 4.3, 4.4 Feb 1, 2020

Tobion mentioned this pull request Mar 23, 2020

[Messenger] Error when throwing exception on MessageHandler #34098

Closed

weaverryan mentioned this pull request Apr 5, 2020

[Messenger] Message retried in failure transport dropped on exception #32719

Closed

monteiro reviewed May 28, 2020

View reviewed changes

monteiro mentioned this pull request Jun 3, 2020

[Messenger] Add FlattenException Normalizer #37087

Merged

nicolas-grekas closed this Jun 4, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Serializer] Add FlattenExceptionNormalizer / Fix messenger used with the serializer component #33650

[Serializer] Add FlattenExceptionNormalizer / Fix messenger used with the serializer component #33650

skalpa commented Sep 20, 2019

skalpa commented Sep 20, 2019

mleczakm commented Sep 20, 2019

skalpa commented Sep 21, 2019

mleczakm commented Sep 21, 2019

weaverryan left a comment

dunglas commented Sep 24, 2019 •

edited

Loading

skalpa commented Sep 24, 2019 •

edited

Loading

yceruto commented Sep 24, 2019 •

edited

Loading

dunglas commented Sep 24, 2019 •

edited

Loading

skalpa commented Sep 24, 2019

fabpot commented Sep 27, 2019

dunglas commented Sep 27, 2019

skalpa commented Sep 27, 2019

nicolas-grekas commented Nov 28, 2019

monteiro commented May 27, 2020 •

edited

Loading

monteiro commented May 28, 2020 •

edited

Loading

monteiro May 28, 2020 •

edited

Loading

monteiro May 28, 2020 •

edited

Loading

nicolas-grekas commented Jun 4, 2020

[Serializer] Add FlattenExceptionNormalizer / Fix messenger used with the serializer component #33650

[Serializer] Add FlattenExceptionNormalizer / Fix messenger used with the serializer component #33650

Conversation

skalpa commented Sep 20, 2019

skalpa commented Sep 20, 2019

mleczakm commented Sep 20, 2019

skalpa commented Sep 21, 2019

mleczakm commented Sep 21, 2019

weaverryan left a comment

Choose a reason for hiding this comment

dunglas commented Sep 24, 2019 • edited Loading

skalpa commented Sep 24, 2019 • edited Loading

yceruto commented Sep 24, 2019 • edited Loading

dunglas commented Sep 24, 2019 • edited Loading

skalpa commented Sep 24, 2019

fabpot commented Sep 27, 2019

dunglas commented Sep 27, 2019

skalpa commented Sep 27, 2019

nicolas-grekas commented Nov 28, 2019

monteiro commented May 27, 2020 • edited Loading

What I'm using to test:

monteiro commented May 28, 2020 • edited Loading

monteiro May 28, 2020 • edited Loading

Choose a reason for hiding this comment

monteiro May 28, 2020 • edited Loading

Choose a reason for hiding this comment

nicolas-grekas commented Jun 4, 2020

dunglas commented Sep 24, 2019 •

edited

Loading

skalpa commented Sep 24, 2019 •

edited

Loading

yceruto commented Sep 24, 2019 •

edited

Loading

dunglas commented Sep 24, 2019 •

edited

Loading

monteiro commented May 27, 2020 •

edited

Loading

monteiro commented May 28, 2020 •

edited

Loading

monteiro May 28, 2020 •

edited

Loading

monteiro May 28, 2020 •

edited

Loading