Thanks to visit codestin.com
Credit goes to github.com

Skip to content

[Messenger] Add SQS transport #32454

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 1 commit into from
Feb 10, 2020
Merged

[Messenger] Add SQS transport #32454

merged 1 commit into from
Feb 10, 2020

Conversation

jderusse
Copy link
Member

@jderusse jderusse commented Jul 9, 2019 •
edited
Loading

Q A
Branch? master
Bug fix? no
New feature? yes
BC breaks? no
Deprecations? no
Tests pass? yes
Fixed tickets N/A
License MIT
Doc PR TODO

This PR add the AWS SQS transport in messenger.

It also add a DisconnectedReceiverInterface that allows the worker to release not-proceeded message (which are not automatically released in SQS and have to wait a TTL). Tell me if you prefer to move it in a dedicated PR.

accepted DNS:

  • sqs://default/accountId/queueName
  • sqs://default/queueName
  • sqs://default/queueName?region=us-east-2
  • sqs://my_custome_endpoint:12345/queueName?sslmode=disabled

To reduce AWS costs, the implementation performs a long polling call and prefetch several messages.
TO get ~real time worker, one could use ./bin/console messenger:consume --sleep 0.001

@jderusse jderusse requested a review from sroze as a code owner July 9, 2019 09:06
@jderusse jderusse force-pushed the messenger-sqs branch 5 times, most recently from 453a19d to 6d4de47 Compare July 9, 2019 10:09
javiereguiluz
javiereguiluz reviewed Jul 9, 2019
View reviewed changes
Copy link
Member

@javiereguiluz javiereguiluz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Very nice PR and a very useful feature. Thanks Jérémy. I left some very minor comments.

@jderusse jderusse changed the base branch from master to 4.4 July 9, 2019 11:14
@nicolas-grekas nicolas-grekas added this to the next milestone Jul 10, 2019
@Tobion
Copy link
Contributor

Tobion commented Jul 11, 2019

To me we should prioritize finalizing the messenger API for 4.4 and fixing all the open messenger problems. So adding another transport should come after that.

@weaverryan
Copy link
Member

To me we should prioritize finalizing the messenger API for 4.4 and fixing all the open messenger problems. So adding another transport should come after that.

I don't want that to block this PR... as we won't be 100% sure of API changes until feature freeze for 4.4. I think we should push this forward.

@jderusse jderusse force-pushed the messenger-sqs branch 2 times, most recently from 4949046 to 8907adc Compare August 7, 2019 12:01
@fabpot
Copy link
Member

fabpot commented Aug 9, 2019

I'm also for merging this PR rather sooner than later, so that when we evolve the code for 4.4 stabilization, it can be included as well in the changes.

@jderusse jderusse force-pushed the messenger-sqs branch 2 times, most recently from ac6e397 to bdafae5 Compare August 9, 2019 08:20
@bendavies
Copy link
Contributor

sqs has first class support for 'failures', dead letter queues. is it possible to support them somehow?

@jderusse
Copy link
Member Author

jderusse commented Aug 9, 2019

@bendavies In this PR I included the minimal features required by Messenger. I think that behaviors like FIFO, Deadletter, could be added in dedicated PR

Nyholm
Nyholm approved these changes Jan 29, 2020
View reviewed changes
Copy link
Member

@Nyholm Nyholm left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Im happy with this PR. Great work!

Could you rebase your PR to get rid of the Doctrine related changes?

@jderusse jderusse force-pushed the messenger-sqs branch 2 times, most recently from bf7f60c to 86ffde5 Compare January 29, 2020 13:16
@fabpot
Copy link
Member

fabpot commented Feb 3, 2020

@jderusse Can you finish this one so that it can be merged? Ping me when it's done :) Thank you.

@chalasr
Copy link
Member

chalasr commented Feb 9, 2020

Looks good to me.

@fabpot
Copy link
Member

fabpot commented Feb 10, 2020

@jderusse Is it ready on your side? Perhaps a last rebase before merging?

@jderusse
Copy link
Member Author

yes it's ok for me

@bendavies
Copy link
Contributor

bendavies commented Feb 10, 2020 •
edited
Loading

I noticed that the namespaces are at odds with Mailer here, where the namespace is Amazon.
https://github.com/symfony/symfony/tree/master/src/Symfony/Component/Mailer/Bridge/Amazon/Transport

This namesapce could be Amazon instead of AmazonSqs here too, with all classes renamed to include Sqs*, just like mailer does.

@Nyholm
Copy link
Member

Nyholm commented Feb 10, 2020

Excellent point. However, I think AmazonSqs is good because it could be a future to include a SNS transport. If so, it would be in a separate package. So it make sense to have AmazonSqs and AmazonSns.

@bendavies
Copy link
Contributor

bendavies commented Feb 10, 2020 •
edited
Loading

Sure, but my point is that, that's not how mailer has done it. all impls are in one Amazon namespace:
https://github.com/symfony/symfony/tree/master/src/Symfony/Component/Mailer/Bridge/Amazon/Transport
You could argue they are all SES, but...

@jderusse
Copy link
Member Author

In my opinion,the difference is AWS provides a single mailing service but several messaging services. (SQS, SNS, Kinesis).

@fabpot
Copy link
Member

fabpot commented Feb 10, 2020

Thank you @jderusse.

@fabpot fabpot mentioned this pull request Feb 10, 2020
Closed
@fabpot fabpot merged commit c226479 into symfony:master Feb 10, 2020
@jderusse jderusse deleted the messenger-sqs branch March 5, 2020 20:03
@Nyholm Nyholm mentioned this pull request May 3, 2020
Closed
@nicolas-grekas nicolas-grekas modified the milestones: next, 5.1 May 4, 2020
@fabpot fabpot mentioned this pull request May 5, 2020
Merged
@RocKordier RocKordier mentioned this pull request May 21, 2020
Closed
@vsilvestre-accenture vsilvestre-accenture mentioned this pull request May 25, 2020
Closed
@vasilvestre vasilvestre mentioned this pull request May 25, 2020
Closed
@tomcoonen
Copy link

How are we on the docs on this? Can I do anything to help?

@Nyholm
Copy link
Member

Nyholm commented Jun 4, 2020 •
edited
Loading

Yes please. There is an issue symfony/symfony-docs#13093 for that.

I'll be happy to help reviewing if you create content.

Let's discuss further on symfony/symfony-docs#13093

EDIT: I see that there is already an open PR: symfony/symfony-docs#13328

@tomcoonen could you help review that?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@javiereguiluz javiereguiluz javiereguiluz left review comments

@nicolas-grekas nicolas-grekas nicolas-grekas left review comments

@stof stof stof left review comments

@sroze sroze sroze left review comments

@deguif deguif deguif left review comments

@Nyholm Nyholm Nyholm approved these changes

@gmponos gmponos gmponos left review comments

@medinae medinae medinae left review comments

@fabpot fabpot fabpot approved these changes

@chalasr chalasr chalasr approved these changes

Assignees
No one assigned
Labels
Feature Messenger Status: Reviewed
Projects
None yet
Milestone
5.1
Development

Successfully merging this pull request may close these issues.