[Messenger] base64_encoding inside PhpSerializer to avoid null characters#30814
Merged
fabpot merged 1 commit intosymfony:masterfrom Apr 1, 2019
Merged
Conversation
fabpot
approved these changes
Apr 1, 2019
Member
fabpot
left a comment
There was a problem hiding this comment.
The only downside is obfuscation of the message that is not as readable as before, but we probably don't have a choice here.
Contributor
Author
|
That's exactly what I was thinking :/. The messages were "readable" before, now it would require a tool. But I also can't see another way around that. |
Contributor
|
I think we don't have another choice here :/ But if we want something readable we can still use the symfony serializer 😉 |
nicolas-grekas
approved these changes
Apr 1, 2019
3 tasks
Member
|
Thank you @weaverryan. |
fabpot
added a commit
that referenced
this pull request
Apr 1, 2019
…null characters (weaverryan) This PR was merged into the 4.3-dev branch. Discussion ---------- [Messenger] base64_encoding inside PhpSerializer to avoid null characters | Q | A | ------------- | --- | Branch? | master | Bug fix? | yes | New feature? | no | BC breaks? | no | Deprecations? | no | Tests pass? | yes | Fixed tickets | #30805 | License | MIT | Doc PR | not needed Hi! As pointed out in #30805, the `PhpSerializer` creates strings with null bytes. This apparently causes problems on at least some database systems (I didn't notice, but @vincenttouzet did). I also read that, for example, SQS doesn't like null characters. And, in general, because we're sending this data over a transport, `base64_encoding` data is pretty standard. Does anyone see any downsides? Cheers! Commits ------- fe7ad81 base64_encoding inside PhpSerializer to avoid null characters
symfony-splitter
pushed a commit
to symfony/messenger
that referenced
this pull request
Apr 1, 2019
This PR was merged into the 4.3-dev branch. Discussion ---------- [Messenger] bug fixes in Doctrine Transport | Q | A | ------------- | --- | Branch? | master | Bug fix? | yes | New feature? | no | BC breaks? | no | Deprecations? | no | Tests pass? | yes | Fixed tickets | | License | MIT | Doc PR | Just tested the new Doctrine transport and I've see 3 bugs so far : - [x] The message is not return by the transport - [x] The headers column must be of type TEXT and not just STRING - [ ] When using the PhpSerializer the message is truncated (PR: symfony/symfony#30814) The body in database looks like this : ``` O:36:"Symfony\Component\Messenger\Envelope":2:{s:44:" ``` The body given by the serializer is the following : ``` O:36:"Symfony\Component\Messenger\Envelope":2:{s:44:"Symfony\Component\Messenger\Envelopestamps";a:3:{s:49:"Symfony\Component\Messenger\Stamp\SerializerStamp";a:1:{i:0;O:49:"Symfony\Component\Messenger\Stamp\SerializerStamp":1:{s:58:"Symfony\Component\Messenger\Stamp\SerializerStampcontext";a:0:{}}}s:46:"Symfony\Component\Messenger\Stamp\BusNameStamp";a:1:{i:0;O:46:"Symfony\Component\Messenger\Stamp\BusNameStamp":1:{s:55:"Symfony\Component\Messenger\Stamp\BusNameStampbusName";s:21:"messenger.bus.default";}}s:43:"Symfony\Component\Messenger\Stamp\SentStamp";a:1:{i:0;O:43:"Symfony\Component\Messenger\Stamp\SentStamp":2:{s:56:"Symfony\Component\Messenger\Stamp\SentStampsenderClass";s:64:"Symfony\Component\Messenger\Transport\Doctrine\DoctrineTransport";s:56:"Symfony\Component\Messenger\Stamp\SentStampsenderAlias";s:16:"environment.stop";}}}s:45:"Symfony\Component\Messenger\Envelopemessage";O:34:"App\Message\EnvironmentStopMessage":1:{s:51:"App\Message\AbstractEnvironmentMessageenvironment";O:22:"App\Entity\Environment":5:{s:26:"App\Entity\Environmentid";s:36:"3bade252-b7a9-4188-82bd-3e68129e0da7";s:37:"App\Entity\EnvironmentrepositoryUrl";s:6:"string";s:30:"App\Entity\Environmentbranch";s:6:"string";s:33:"App\Entity\EnvironmenthostNames";a:1:{i:0;N;}s:27:"App\Entity\Environmentenv";a:2:{s:7:"APP_ENV";s:4:"prod";s:7:"APP_VAR";s:13:"example value";}}}} ``` Commits ------- 27466498d0 [Messenger] Fix get in Doctrine Transport
fabpot
added a commit
to fabpot/symfony
that referenced
this pull request
Apr 1, 2019
…ttouzet) This PR was merged into the 4.3-dev branch. Discussion ---------- [Messenger] bug fixes in Doctrine Transport | Q | A | ------------- | --- | Branch? | master | Bug fix? | yes | New feature? | no | BC breaks? | no | Deprecations? | no | Tests pass? | yes | Fixed tickets | | License | MIT | Doc PR | Just tested the new Doctrine transport and I've see 3 bugs so far : - [x] The message is not return by the transport - [x] The headers column must be of type TEXT and not just STRING - [ ] When using the PhpSerializer the message is truncated (PR: symfony#30814) The body in database looks like this : ``` O:36:"Symfony\Component\Messenger\Envelope":2:{s:44:" ``` The body given by the serializer is the following : ``` O:36:"Symfony\Component\Messenger\Envelope":2:{s:44:"Symfony\Component\Messenger\Envelopestamps";a:3:{s:49:"Symfony\Component\Messenger\Stamp\SerializerStamp";a:1:{i:0;O:49:"Symfony\Component\Messenger\Stamp\SerializerStamp":1:{s:58:"Symfony\Component\Messenger\Stamp\SerializerStampcontext";a:0:{}}}s:46:"Symfony\Component\Messenger\Stamp\BusNameStamp";a:1:{i:0;O:46:"Symfony\Component\Messenger\Stamp\BusNameStamp":1:{s:55:"Symfony\Component\Messenger\Stamp\BusNameStampbusName";s:21:"messenger.bus.default";}}s:43:"Symfony\Component\Messenger\Stamp\SentStamp";a:1:{i:0;O:43:"Symfony\Component\Messenger\Stamp\SentStamp":2:{s:56:"Symfony\Component\Messenger\Stamp\SentStampsenderClass";s:64:"Symfony\Component\Messenger\Transport\Doctrine\DoctrineTransport";s:56:"Symfony\Component\Messenger\Stamp\SentStampsenderAlias";s:16:"environment.stop";}}}s:45:"Symfony\Component\Messenger\Envelopemessage";O:34:"App\Message\EnvironmentStopMessage":1:{s:51:"App\Message\AbstractEnvironmentMessageenvironment";O:22:"App\Entity\Environment":5:{s:26:"App\Entity\Environmentid";s:36:"3bade252-b7a9-4188-82bd-3e68129e0da7";s:37:"App\Entity\EnvironmentrepositoryUrl";s:6:"string";s:30:"App\Entity\Environmentbranch";s:6:"string";s:33:"App\Entity\EnvironmenthostNames";a:1:{i:0;N;}s:27:"App\Entity\Environmentenv";a:2:{s:7:"APP_ENV";s:4:"prod";s:7:"APP_VAR";s:13:"example value";}}}} ``` Commits ------- 2746649 [Messenger] Fix get in Doctrine Transport
This was referenced Apr 2, 2019
fabpot
added a commit
that referenced
this pull request
Apr 15, 2019
…verryan) This PR was merged into the 4.3-dev branch. Discussion ---------- [Messenger] Remove base64_encode & use addslashes | Q | A | ------------- | --- | Branch? | master | Bug fix? | no | New feature? | yes | BC breaks? | no | Deprecations? | no | Tests pass? | yes | Fixed tickets | none | License | MIT | Doc PR | already covered by existing issue In #30814, we base64_encoded messages because some transports (specifically DoctrineTransport + Postgresql & SQS) do not allow binary data. The downside is that the messages become unreadable, which makes it much less convenient to debug your messages with 3rd party monitoring tools, for example. This PR replaces base64_encode with addslashes. Another alternative (that I first tried in this PR) was to use a blob type, which Drupal does in its code (https://www.drupal.org/project/drupal/issues/690746). But, it still meant that binary data could cause problems with other transports, like SQS. I also put all the serializer config under a nice, neat `serializer` key under messenger. Best seen with `?w=1`. Cheers! Commits ------- 70b448d Reorganizing messenger serializer config and replacing base64_encode with addslashes
symfony-splitter
pushed a commit
to symfony/messenger
that referenced
this pull request
Jan 28, 2020
This PR was merged into the 4.3-dev branch. Discussion ---------- [Messenger] bug fixes in Doctrine Transport | Q | A | ------------- | --- | Branch? | master | Bug fix? | yes | New feature? | no | BC breaks? | no | Deprecations? | no | Tests pass? | yes | Fixed tickets | | License | MIT | Doc PR | Just tested the new Doctrine transport and I've see 3 bugs so far : - [x] The message is not return by the transport - [x] The headers column must be of type TEXT and not just STRING - [ ] When using the PhpSerializer the message is truncated (PR: symfony/symfony#30814) The body in database looks like this : ``` O:36:"Symfony\Component\Messenger\Envelope":2:{s:44:" ``` The body given by the serializer is the following : ``` O:36:"Symfony\Component\Messenger\Envelope":2:{s:44:"Symfony\Component\Messenger\Envelopestamps";a:3:{s:49:"Symfony\Component\Messenger\Stamp\SerializerStamp";a:1:{i:0;O:49:"Symfony\Component\Messenger\Stamp\SerializerStamp":1:{s:58:"Symfony\Component\Messenger\Stamp\SerializerStampcontext";a:0:{}}}s:46:"Symfony\Component\Messenger\Stamp\BusNameStamp";a:1:{i:0;O:46:"Symfony\Component\Messenger\Stamp\BusNameStamp":1:{s:55:"Symfony\Component\Messenger\Stamp\BusNameStampbusName";s:21:"messenger.bus.default";}}s:43:"Symfony\Component\Messenger\Stamp\SentStamp";a:1:{i:0;O:43:"Symfony\Component\Messenger\Stamp\SentStamp":2:{s:56:"Symfony\Component\Messenger\Stamp\SentStampsenderClass";s:64:"Symfony\Component\Messenger\Transport\Doctrine\DoctrineTransport";s:56:"Symfony\Component\Messenger\Stamp\SentStampsenderAlias";s:16:"environment.stop";}}}s:45:"Symfony\Component\Messenger\Envelopemessage";O:34:"App\Message\EnvironmentStopMessage":1:{s:51:"App\Message\AbstractEnvironmentMessageenvironment";O:22:"App\Entity\Environment":5:{s:26:"App\Entity\Environmentid";s:36:"3bade252-b7a9-4188-82bd-3e68129e0da7";s:37:"App\Entity\EnvironmentrepositoryUrl";s:6:"string";s:30:"App\Entity\Environmentbranch";s:6:"string";s:33:"App\Entity\EnvironmenthostNames";a:1:{i:0;N;}s:27:"App\Entity\Environmentenv";a:2:{s:7:"APP_ENV";s:4:"prod";s:7:"APP_VAR";s:13:"example value";}}}} ``` Commits ------- 27466498d0 [Messenger] Fix get in Doctrine Transport
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Hi!
As pointed out in #30805, the
PhpSerializercreates strings with null bytes. This apparently causes problems on at least some database systems (I didn't notice, but @vincenttouzet did). I also read that, for example, SQS doesn't like null characters. And, in general, because we're sending this data over a transport,base64_encodingdata is pretty standard.Does anyone see any downsides?
Cheers!