[Messenger] Add SQS transport #32454

jderusse · 2019-07-09T09:06:55Z

Q	A
Branch?	master
Bug fix?	no
New feature?	yes
BC breaks?	no
Deprecations?	no
Tests pass?	yes
Fixed tickets	N/A
License	MIT
Doc PR	TODO

This PR add the AWS SQS transport in messenger.

It also add a DisconnectedReceiverInterface that allows the worker to release not-proceeded message (which are not automatically released in SQS and have to wait a TTL). Tell me if you prefer to move it in a dedicated PR.

accepted DNS:

sqs://default/accountId/queueName
sqs://default/queueName
sqs://default/queueName?region=us-east-2
sqs://my_custome_endpoint:12345/queueName?sslmode=disabled

To reduce AWS costs, the implementation performs a long polling call and prefetch several messages.
TO get ~real time worker, one could use ./bin/console messenger:consume --sleep 0.001

javiereguiluz

Very nice PR and a very useful feature. Thanks Jérémy. I left some very minor comments.

src/Symfony/Component/Messenger/Transport/SqsExt/Connection.php

Tobion · 2019-07-11T01:49:55Z

To me we should prioritize finalizing the messenger API for 4.4 and fixing all the open messenger problems. So adding another transport should come after that.

weaverryan · 2019-07-17T15:37:31Z

To me we should prioritize finalizing the messenger API for 4.4 and fixing all the open messenger problems. So adding another transport should come after that.

I don't want that to block this PR... as we won't be 100% sure of API changes until feature freeze for 4.4. I think we should push this forward.

src/Symfony/Component/Messenger/Transport/SqsExt/Connection.php

fabpot · 2019-08-09T07:18:11Z

I'm also for merging this PR rather sooner than later, so that when we evolve the code for 4.4 stabilization, it can be included as well in the changes.

src/Symfony/Component/Messenger/Transport/Receiver/DisconnectedReceiverInterface.php

src/Symfony/Component/Messenger/Transport/SqsExt/Connection.php

src/Symfony/Component/Messenger/Tests/Transport/SqsExt/SqsExtIntegrationTest.php

bendavies · 2019-08-09T10:46:24Z

sqs has first class support for 'failures', dead letter queues. is it possible to support them somehow?

jderusse · 2019-08-09T11:06:19Z

@bendavies In this PR I included the minimal features required by Messenger. I think that behaviors like FIFO, Deadletter, could be added in dedicated PR

src/Symfony/Component/Messenger/Transport/SqsExt/Connection.php

src/Symfony/Component/Messenger/Worker.php

Nyholm

Im happy with this PR. Great work!

Could you rebase your PR to get rid of the Doctrine related changes?

src/Symfony/Component/Messenger/Bridge/AmazonSqs/Transport/AmazonSqsSender.php

src/Symfony/Component/Messenger/Bridge/AmazonSqs/Transport/Connection.php

fabpot · 2020-02-03T17:35:37Z

@jderusse Can you finish this one so that it can be merged? Ping me when it's done :) Thank you.

chalasr · 2020-02-09T17:14:01Z

Looks good to me.

fabpot · 2020-02-10T08:07:33Z

@jderusse Is it ready on your side? Perhaps a last rebase before merging?

jderusse · 2020-02-10T08:20:11Z

yes it's ok for me

bendavies · 2020-02-10T11:47:36Z

I noticed that the namespaces are at odds with Mailer here, where the namespace is Amazon.
https://github.com/symfony/symfony/tree/master/src/Symfony/Component/Mailer/Bridge/Amazon/Transport

This namesapce could be Amazon instead of AmazonSqs here too, with all classes renamed to include Sqs*, just like mailer does.

Nyholm · 2020-02-10T11:50:04Z

Excellent point. However, I think AmazonSqs is good because it could be a future to include a SNS transport. If so, it would be in a separate package. So it make sense to have AmazonSqs and AmazonSns.

bendavies · 2020-02-10T12:02:27Z

Sure, but my point is that, that's not how mailer has done it. all impls are in one Amazon namespace:
https://github.com/symfony/symfony/tree/master/src/Symfony/Component/Mailer/Bridge/Amazon/Transport
You could argue they are all SES, but...

jderusse · 2020-02-10T12:08:15Z

In my opinion,the difference is AWS provides a single mailing service but several messaging services. (SQS, SNS, Kinesis).

fabpot · 2020-02-10T12:30:57Z

Thank you @jderusse.

This PR was merged into the 5.1-dev branch. Discussion ---------- [Messenger] Add SQS transport | Q | A | ------------- | --- | Branch? | master | Bug fix? | no | New feature? | yes | BC breaks? | no | Deprecations? | no | Tests pass? | yes | Fixed tickets | N/A | License | MIT | Doc PR | TODO This PR add the AWS SQS transport in messenger. It also add a `DisconnectedReceiverInterface` that allows the worker to release not-proceeded message (which are not automatically released in SQS and have to wait a TTL). Tell me if you prefer to move it in a dedicated PR. accepted DNS: - `sqs://default/accountId/queueName` - `sqs://default/queueName` - `sqs://default/queueName?region=us-east-2` - `sqs://my_custome_endpoint:12345/queueName?sslmode=disabled` To reduce AWS costs, the implementation performs a long polling call and prefetch several messages. TO get ~real time worker, one could use `./bin/console messenger:consume --sleep 0.001` Commits ------- c226479 [Messenger] Add SQS transport

tomcoonen · 2020-06-04T07:06:14Z

How are we on the docs on this? Can I do anything to help?

Nyholm · 2020-06-04T07:21:18Z

Yes please. There is an issue symfony/symfony-docs#13093 for that.

I'll be happy to help reviewing if you create content.

Let's discuss further on symfony/symfony-docs#13093

EDIT: I see that there is already an open PR: symfony/symfony-docs#13328

@tomcoonen could you help review that?

jderusse requested a review from sroze as a code owner July 9, 2019 09:06

carsonbot added Status: Needs Review Messenger Feature labels Jul 9, 2019

jderusse force-pushed the messenger-sqs branch 5 times, most recently from 453a19d to 6d4de47 Compare July 9, 2019 10:09

javiereguiluz reviewed Jul 9, 2019

View reviewed changes

jderusse force-pushed the messenger-sqs branch from 6d4de47 to 3c07d2e Compare July 9, 2019 11:14

jderusse changed the base branch from master to 4.4 July 9, 2019 11:14

nicolas-grekas added this to the next milestone Jul 10, 2019

deguif reviewed Jul 10, 2019

View reviewed changes

src/Symfony/Component/Messenger/Transport/SqsExt/Connection.php Outdated Show resolved Hide resolved

stof reviewed Jul 17, 2019

View reviewed changes

src/Symfony/Component/Messenger/Transport/SqsExt/Connection.php Outdated Show resolved Hide resolved

jderusse force-pushed the messenger-sqs branch 2 times, most recently from 4949046 to 8907adc Compare August 7, 2019 12:01

fabpot reviewed Aug 9, 2019

View reviewed changes

jderusse force-pushed the messenger-sqs branch from 8907adc to 5e9073a Compare August 9, 2019 07:49

jderusse commented Aug 9, 2019

View reviewed changes

src/Symfony/Component/Messenger/Tests/Transport/SqsExt/SqsExtIntegrationTest.php Outdated Show resolved Hide resolved

jderusse force-pushed the messenger-sqs branch 2 times, most recently from ac6e397 to bdafae5 Compare August 9, 2019 08:20

medinae reviewed Sep 17, 2019

View reviewed changes

src/Symfony/Component/Messenger/Transport/SqsExt/Connection.php Outdated Show resolved Hide resolved

medinae reviewed Sep 17, 2019

View reviewed changes

src/Symfony/Component/Messenger/Worker.php Outdated Show resolved Hide resolved

jderusse force-pushed the messenger-sqs branch from bdafae5 to 722315d Compare September 17, 2019 06:16

Nyholm approved these changes Jan 29, 2020

View reviewed changes

src/Symfony/Component/Messenger/Bridge/AmazonSqs/Transport/AmazonSqsSender.php Outdated Show resolved Hide resolved

jderusse force-pushed the messenger-sqs branch 2 times, most recently from bf7f60c to 86ffde5 Compare January 29, 2020 13:16

jderusse commented Jan 29, 2020

View reviewed changes

src/Symfony/Component/Messenger/Bridge/AmazonSqs/Transport/Connection.php Outdated Show resolved Hide resolved

jderusse force-pushed the messenger-sqs branch from 86ffde5 to 5c7d8fb Compare February 3, 2020 21:14

[Messenger] Add SQS transport

c226479

jderusse force-pushed the messenger-sqs branch from 5c7d8fb to c226479 Compare February 4, 2020 10:49

fabpot approved these changes Feb 10, 2020

View reviewed changes

fabpot mentioned this pull request Feb 10, 2020

[Messenger] Add SQS transport symfony/symfony-docs#13093

Closed

fabpot merged commit c226479 into symfony:master Feb 10, 2020

jderusse deleted the messenger-sqs branch March 5, 2020 20:03

Nyholm mentioned this pull request May 3, 2020

[Messenger] Redis Transport removes message from stream on rejection #36619

Closed

nicolas-grekas modified the milestones: next, 5.1 May 4, 2020

fabpot mentioned this pull request May 5, 2020

Release v5.1.0-BETA1 #36713

Merged

RocKordier mentioned this pull request May 21, 2020

[Messenger] Add suggestion for Amazon SQS transport and make check more readable #36890

Closed

vsilvestre-accenture mentioned this pull request May 25, 2020

Use new SQS transport brefphp/examples#22

Closed

vasilvestre mentioned this pull request May 25, 2020

Use new SQS transport in Messenger brefphp/examples#23

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Messenger] Add SQS transport #32454

[Messenger] Add SQS transport #32454

jderusse commented Jul 9, 2019 •

edited

Loading

javiereguiluz left a comment

Tobion commented Jul 11, 2019

weaverryan commented Jul 17, 2019

fabpot commented Aug 9, 2019

bendavies commented Aug 9, 2019

jderusse commented Aug 9, 2019

Nyholm left a comment

fabpot commented Feb 3, 2020

chalasr commented Feb 9, 2020

fabpot commented Feb 10, 2020

jderusse commented Feb 10, 2020

bendavies commented Feb 10, 2020 •

edited

Loading

Nyholm commented Feb 10, 2020

bendavies commented Feb 10, 2020 •

edited

Loading

jderusse commented Feb 10, 2020

fabpot commented Feb 10, 2020

tomcoonen commented Jun 4, 2020

Nyholm commented Jun 4, 2020 •

edited

Loading

[Messenger] Add SQS transport #32454

[Messenger] Add SQS transport #32454

Conversation

jderusse commented Jul 9, 2019 • edited Loading

javiereguiluz left a comment

Choose a reason for hiding this comment

Tobion commented Jul 11, 2019

weaverryan commented Jul 17, 2019

fabpot commented Aug 9, 2019

bendavies commented Aug 9, 2019

jderusse commented Aug 9, 2019

Nyholm left a comment

Choose a reason for hiding this comment

fabpot commented Feb 3, 2020

chalasr commented Feb 9, 2020

fabpot commented Feb 10, 2020

jderusse commented Feb 10, 2020

bendavies commented Feb 10, 2020 • edited Loading

Nyholm commented Feb 10, 2020

bendavies commented Feb 10, 2020 • edited Loading

jderusse commented Feb 10, 2020

fabpot commented Feb 10, 2020

tomcoonen commented Jun 4, 2020

Nyholm commented Jun 4, 2020 • edited Loading

jderusse commented Jul 9, 2019 •

edited

Loading

bendavies commented Feb 10, 2020 •

edited

Loading

bendavies commented Feb 10, 2020 •

edited

Loading

Nyholm commented Jun 4, 2020 •

edited

Loading