[Messenger] add missing log context when sending to failure transport #46450

ajardin · 2022-05-24T15:51:20Z

Q	A
Branch?	6.2
Bug fix?	no
New feature?	yes
Deprecations?	no
Tickets	N/A
License	MIT
Doc PR	N/A

Why?

Logs generated by SendFailedMessageToFailureTransportListener::onMessageFailed do not include the message in the context, unlike those generated by SendFailedMessageForRetryListener::onMessageFailed.

Before

INFO      [messenger] Rejected message App\Message\MyCustomMessage will be sent to the failure transport Symfony\Component\Messenger\Bridge\Doctrine\Transport\DoctrineTransport.
 [
   "class" => "App\Message\MyCustomMessage",
   "transport" => "Symfony\Component\Messenger\Bridge\Doctrine\Transport\DoctrineTransport"
 ]
 [
   "request_id" => false,
   "uid" => "537143b"
 ]

After

INFO      [messenger] Rejected message App\Message\MyCustomMessage will be sent to the failure transport Symfony\Component\Messenger\Bridge\Doctrine\Transport\DoctrineTransport.
 [
   "message" => App\Message\MyCustomMessage {
     -foo: true
     -bar: false
   },
   "class" => "App\Message\MyCustomMessage",
   "transport" => "Symfony\Component\Messenger\Bridge\Doctrine\Transport\DoctrineTransport"
 ]
 [
   "request_id" => false,
   "uid" => "537143b"
 ]

carsonbot · 2022-05-24T15:51:24Z

Hey!

I see that this is your first PR. That is great! Welcome!

Symfony has a contribution guide which I suggest you to read.

In short:

Always add tests
Keep backward compatibility (see https://symfony.com/bc).
Bug fixes must be submitted against the lowest maintained branch where they apply (see https://symfony.com/releases)
Features and deprecations must be submitted against the 6.2 branch.

Review the GitHub status checks of your pull request and try to solve the reported issues. If some tests are failing, try to see if they are failing because of this change.

When two Symfony core team members approve this change, it will be merged and you will become an official Symfony contributor!
If this PR is merged in a lower version branch, it will be merged up to all maintained branches within a few days.

I am going to sit back now and wait for the reviews.

Cheers!

Carsonbot

chalasr · 2022-05-25T08:57:39Z

Thanks for the PR. This change looks like a new feature to me, or is there something that breaks without it?

ajardin · 2022-05-25T09:35:28Z

Hello and thanks for the reply! 🙂

You're right, it's not broken. I was also hesitating between a bug and a feature...
Basically, it makes the log less useful as the relationship between the message and the event is not visible, which makes me think it's a missing element rather than a new. What do you think?

fabpot · 2022-05-25T09:55:25Z

That’s a new feature 😀

ajardin · 2022-05-25T11:08:40Z

I have updated the pull request accordingly. 😉

lyrixx · 2022-05-25T11:58:14Z

Message content can be very huge. This can also overflow some third party service.

But I like the idea. I'm my application, I usually log raw object, then I add a custom monolog processor to keep only what I need. (I gave a talk about it)

We could do the same here

Log with the raw envelop
Add a default monolog processor that
- Remove eveything and keep only the class.
- Or keep only the 1024 first bytes of the content (arbitrary number)

So if people want more or less information in the log, they can decorate / replace the default processor in the DIC.

WDYT?

(I actually really like the idea. I think it could be generalized to the framework, when applicable)

ajardin · 2022-05-25T12:39:49Z

Your idea goes further than my initial thoughts, but I really like it! 🙂

Instead of adding an additional risk of overflow (the message is already logged in multiple places), we would fix a minor lack and improve the component monitoring. Moreover, the possibility to replace/decorate the processor would be great.

I can try to work on this approach in the coming days if nobody else is on it.
In the meantime, what do you want to do with this pull request?

lyrixx · 2022-05-25T12:46:26Z

I can try to work on this approach in the coming days if nobody else is on it.

Please, wait for more approval from the core team, I'm not sure everything is OK with that

ping @symfony/mergers

Nyholm

I agree with @lyrixx. This is a great feature. But messages can be huge. Messages can also include sensitive data (passwords) that should not end up in the logs.

I suggest to remove the message. Adding the message class is little but enough info to add more logging in your app if you need to know the exact contents of the message.

Nyholm · 2022-05-25T12:51:46Z

src/Symfony/Component/Messenger/EventListener/SendFailedMessageToFailureTransportListener.php

        $this->logger?->info('Rejected message {class} will be sent to the failure transport {transport}.', [
-            'class' => \get_class($envelope->getMessage()),
+            'message' => $message,


I suggest ro remove this key.

Nyholm · 2022-05-25T12:51:54Z

src/Symfony/Component/Messenger/EventListener/SendFailedMessageToFailureTransportListener.php

        $this->logger?->info('Rejected message {class} will be sent to the failure transport {transport}.', [
-            'class' => \get_class($envelope->getMessage()),
+            'message' => $message,
+            'class' => \get_class($message),


Suggested change

'class' => \get_class($message),

'message_class' => \get_class($message),

ajardin · 2022-05-25T13:55:42Z

@Nyholm removing this key is basically closing the pull request. 😅

Joke aside, what is your opinion on the processor approach proposed by @lyrixx? We could only keep the message class by default and let users decide if they want to add further data through decorators.

Because even if we decide to not add the message within SendFailedMessageToFailureTransportListener::onMessageFailed logs, the message is already available (at least) in:

\Symfony\Component\Messenger\TraceableMessageBus::dispatch
\Symfony\Component\Messenger\Worker::ack
\Symfony\Component\Messenger\EventListener\SendFailedMessageForRetryListener::onMessageFailed
\Symfony\Component\Messenger\Middleware\HandleMessageMiddleware::handle
\Symfony\Component\Messenger\Middleware\SendMessageMiddleware::handle

ajardin · 2022-06-21T14:58:07Z

Hello @lyrixx and @Nyholm,

Did you have the opportunity to discuss the changes mentioned above? Thanks for your time.

lyrixx · 2022-06-21T15:17:22Z

Adding the raw message is a no go unfortunately. IMHO the custom processor is the way to go.

ajardin · 2022-06-21T16:02:08Z

Thanks for your feedback. In that case, I'm closing this pull request.

But while I understand the reason why adding the message could cause problems (e.g. leak of sensitive data or logs overflow), I don't understand why it's a problem here and not in the other parts of the component. Am I missing something?

Nyholm · 2022-06-21T16:08:27Z

You are correct. Maybe we did miss something in previous prs.

I would be super happy if you want to help by removing those instances to.
I definitely understand if you don’t, but please tell us so someone else can work on that.

ajardin · 2022-06-21T16:21:22Z

No problem, I'll submit a PR with the removal of other instances.

But I would love to have a workaround idea for the application I'm working on since we use the message details to monitor exactly what's going on. 😉

Should we subscribe to Messenger events and add our own log entries?

Nyholm · 2022-06-21T16:48:15Z

Should we subscribe to Messenger events and add our own log entries?

Yes,
You can also add your custom middleware. That will give you maximum control.

@Nyholm

This PR was merged into the 5.4 branch. Discussion ---------- [Messenger] Do not log the message object itself In order to avoid the leak of sensitive data (e.g. credentials) or the overflow of third-party services. | Q | A | ------------- | --- | Branch? | 5.4 | Bug fix? | yes | New feature? | no | Deprecations? | no | Tickets | | License | MIT | Doc PR | This is a follow-up of #46450 where we had a discussion with @Nyholm about the problems related to the logging of the message object. I'm targeting the `5.4` branch as we see this change as a (security ?) fix rather than a new feature. Commits ------- f1604e6 [Messenger] Do not log the message object itself

carsonbot added Status: Needs Review Bug Messenger labels May 24, 2022

carsonbot added this to the 4.4 milestone May 24, 2022

ajardin force-pushed the messenger-logs branch from cd23b9d to 575192c Compare May 25, 2022 10:05

ajardin requested review from lyrixx, yceruto, wouterj, chalasr, dunglas and xabbuh as code owners May 25, 2022 10:05

ajardin changed the base branch from 4.4 to 6.1 May 25, 2022 10:05

chalasr modified the milestones: 4.4, 6.2 May 25, 2022

[Messenger] add missing log context when sending to failure transport

ba4b5c3

chalasr changed the base branch from 6.1 to 6.2 May 25, 2022 10:07

ajardin force-pushed the messenger-logs branch from 575192c to ba4b5c3 Compare May 25, 2022 10:07

Nyholm requested changes May 25, 2022

View reviewed changes

carsonbot added Status: Needs Work and removed Status: Needs Review labels May 25, 2022

ajardin closed this Jun 21, 2022

ajardin mentioned this pull request Jun 22, 2022

[Messenger] Do not log the message object itself #46735

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Messenger] add missing log context when sending to failure transport #46450

[Messenger] add missing log context when sending to failure transport #46450

ajardin commented May 24, 2022 •

edited

Loading

carsonbot commented May 24, 2022

chalasr commented May 25, 2022

ajardin commented May 25, 2022

fabpot commented May 25, 2022

ajardin commented May 25, 2022

lyrixx commented May 25, 2022 •

edited

Loading

ajardin commented May 25, 2022

lyrixx commented May 25, 2022

Nyholm left a comment

Nyholm May 25, 2022

Nyholm May 25, 2022

ajardin commented May 25, 2022

ajardin commented Jun 21, 2022

lyrixx commented Jun 21, 2022

ajardin commented Jun 21, 2022

Nyholm commented Jun 21, 2022

ajardin commented Jun 21, 2022

Nyholm commented Jun 21, 2022

	'class' => \get_class($message),
	'message_class' => \get_class($message),

[Messenger] add missing log context when sending to failure transport #46450

[Messenger] add missing log context when sending to failure transport #46450

Conversation

ajardin commented May 24, 2022 • edited Loading

Why?

Before

After

carsonbot commented May 24, 2022

chalasr commented May 25, 2022

ajardin commented May 25, 2022

fabpot commented May 25, 2022

ajardin commented May 25, 2022

lyrixx commented May 25, 2022 • edited Loading

ajardin commented May 25, 2022

lyrixx commented May 25, 2022

Nyholm left a comment

Choose a reason for hiding this comment

Nyholm May 25, 2022

Choose a reason for hiding this comment

Nyholm May 25, 2022

Choose a reason for hiding this comment

ajardin commented May 25, 2022

ajardin commented Jun 21, 2022

lyrixx commented Jun 21, 2022

ajardin commented Jun 21, 2022

Nyholm commented Jun 21, 2022

ajardin commented Jun 21, 2022

Nyholm commented Jun 21, 2022

ajardin commented May 24, 2022 •

edited

Loading

lyrixx commented May 25, 2022 •

edited

Loading