refactor SNS async publishing & ASF data models #7267

bentsku · 2022-11-30T18:00:51Z

This PR refactors SNS publishing and the underlying data model.
The ASF provider was still making use of the data model used by the ProxyListener (using query string parameters).

This PR splits the publishing in different class depending on the target: Lambda, SQS, Firehose, SMS, Platform Endpoint (mobile push notification system), email&email-json, and http/https.

It makes uses of a ThreadPoolExecutor, in the same way as S3 notifications.

Previously, while using publish_batch, it would send the messages sequentially and synchronously (maybe in order to keep the FIFO order right). This is now all asynchronous.
Also, while batch publishing to an SQS queue, the new publisher will make use of SQS SendMessageBatch instead of making individual calls (the batching size and limitations are the same, so it must use it under the hood).

Different validations were added along the way, for example to SetSubscriptionAttributes, and different parameters to Publish, with corresponding AWS validated tests.

Also, I've put a feature allowing us to directly publish to the real GCM/FCM mobile notification platform behind a feature flag, for now called LEGACY_SNS_GCM_PUBLISHING but maybe in need of a new name. This feature was not advertised, but was blocking some other users as they would need real world credentials to GCM to be able to use LocalStack.

Limitations: while testing, I've realised that messages sent to FIFO topics would not properly be directed to the DLQ in case of issue, most probably because we don't pass down the MessageGroupId and such necessary parameters. This will be treated as the next iteration, an xfail test is already in place and I know how to fix the issue, but the PR was getting a bit out of scope.

note: CI run failed on a fluke, should be green

fixes #6863

github-actions · 2022-11-30T18:58:31Z

LocalStack integration with Pro

      3 files +    1       3 suites +1 1h 38m 51s ⏱️ + 9m 45s
1 607 tests +    6 1 315 ✔️ +    5 292 💤 +    1 0 ❌ ±0
2 259 runs +340 1 685 ✔️ +202 574 💤 +138 0 ❌ ±0

Results for commit 1cd0b80. ± Comparison against base commit 6ed9c4d.

♻️ This comment has been updated with latest results.

coveralls · 2022-12-01T17:00:32Z

Coverage: 84.905% (-0.02%) from 84.93% when pulling 1cd0b80 on sns-refactor-publish into 6ed9c4d on master.

localstack/services/sns/publisher.py

baermat

This looks awesome! Finally this is getting the overhaul it needed and deserved. I have a few comments, nothing major.

localstack/services/sns/models.py

localstack/services/sns/provider.py

localstack/services/sns/publisher.py

thrau

This is a fantastic set of changes, really well done with the class hierarchy and the async delivery logic! Encapsulation into classes vs functions is also very nicely balanced.

In terms of architecture I only have one comment regarding try/except pattern in a contract, which in this case I feel is debatable (see inline).

Two things that I think should be addressed are:

More documentation: It would be great if we could add some pydoc all-around, but especially for top-level classes/interfaces.
Logging: proper logging delivery of messages can have a huge impact on DevX for this service, and I think we can make some improvements there

thrau · 2022-12-21T21:13:49Z

localstack/services/sns/publisher.py

+        for subscriber in subscriptions:
+            if self._should_publish(ctx.store, ctx.message, subscriber):
+                notifier = self.topic_notifiers[subscriber["Protocol"]]
+                LOG.debug("Submitting task to the executor for notifier %s", notifier)


nit: will this produce a useful output? it seems __repr__ and __str__ aren't implemented for whatever notifier is.
generally, good logging can be very useful here, so maybe we could improve it to show a concise log message about what's going on: source X publishing to topic Y via protocol Z, maybe?

this would apply to all logging statements that have similar significance.

Agreed! Will work on this! 👍

localstack/services/sns/publisher.py

thrau · 2022-12-24T21:08:42Z

localstack/services/sns/publisher.py

+    def publish(self, context: SnsPublishContext, subscriber: SnsSubscription):
+        try:
+            self._publish(context=context, subscriber=subscriber)
+        except Exception:
+            LOG.exception(
+                "An internal error occurred while trying to send the SNS message %s",
+                context.message,
+            )
+            return


I'm not sure about swallowing exceptions at this level of abstraction (the base class/contract). The try/except pattern is generally something i feel the caller should be taking care of, not the callee (which is what we're doing here). But I understand we're only invoking this method asynchronously via an executor, and this is the most obvious/convenient way of making sure the exception at least gets logged.

Maybe it makes more sense to wrap the executor.submit(publisher.publish) with some inline method that does the exception wrapping, which is a bit more inconvenient for the caller, but keeps the publisher free from this logic.

But debatable 🤷

I'm not quite sure I understand fully the concept between caller/callee in this case.

The only way the caller could manage this would be by checking the Future result, or wrapping like you said.
I guess the PublishDispatcher could theoretically provide a wrapper method, something like _wrap_and_log() or something similar, and dispatch it that way? executor.submit(self._wrap_and_log(publisher.publish)), is that what you meant?
We could almost provide an helper method and use it as a decorator around the publish methods? I'll wait for your answer on this one as I'm not sure I fully understood what you meant! But thanks, agree that the logging is a bit weird in the publisher that way.

Update: we discussed and we will tackle this in a next PR, this will do for now. Will add a comment and a todo to clean up this.

thrau · 2022-12-24T21:09:17Z

localstack/services/sns/publisher.py

+    request_headers: Dict[str, str]
+
+
+class TopicPublisher(abc.ABC):


If this is the interface/contract, a bit of doc could help :-)

thrau · 2022-12-24T21:10:08Z

localstack/services/sns/publisher.py

+class LambdaTopicPublisher(TopicPublisher):
+    def _publish(self, context: SnsPublishContext, subscriber: SnsSubscription):


It would be great if we could add a bit of pydoc to every publisher and a link to the relevant AWS docs for the integration.

localstack/services/sns/publisher.py

baermat

LGTM, with one small nit :)
Also I feel that the added comments now sufficiently describe the classes and that the EmailJsonTopic naming is ok now.

localstack/services/sns/publisher.py

thrau

looks great now, thanks for addressing all the comments and adding docs. logging is really good as well! tested it locally.

one more thing is to update the docs with the new config variable!

bentsku · 2023-01-03T20:29:44Z

I'm not sure I want to update the docs about the config variable when the feature was not widely advertised and mostly blocking. Maybe I could comment on the feature request that this is now behind the flag?

bentsku temporarily deployed to localstack-ext-tests November 30, 2022 18:01 Inactive

bentsku temporarily deployed to localstack-ext-tests November 30, 2022 18:08 Inactive

bentsku temporarily deployed to localstack-ext-tests November 30, 2022 21:38 Inactive

bentsku force-pushed the sns-refactor-publish branch from a563f3a to 9b8f18d Compare December 1, 2022 15:36

bentsku temporarily deployed to localstack-ext-tests December 1, 2022 15:36 Inactive

bentsku temporarily deployed to localstack-ext-tests December 12, 2022 11:38 — with GitHub Actions Inactive

bentsku force-pushed the sns-refactor-publish branch from 4940492 to 0415a22 Compare December 12, 2022 12:22

bentsku temporarily deployed to localstack-ext-tests December 12, 2022 12:22 — with GitHub Actions Inactive

bentsku commented Dec 12, 2022

View reviewed changes

localstack/services/sns/publisher.py Outdated Show resolved Hide resolved

bentsku requested a review from baermat December 12, 2022 17:17

bentsku marked this pull request as ready for review December 12, 2022 17:17

bentsku requested a review from thrau as a code owner December 12, 2022 17:17

bentsku changed the title ~~WIP: refactor SNS async publishing & ASF data models~~ refactor SNS async publishing & ASF data models Dec 12, 2022

baermat reviewed Dec 13, 2022

View reviewed changes

bentsku temporarily deployed to localstack-ext-tests December 14, 2022 16:53 — with GitHub Actions Inactive

bentsku temporarily deployed to localstack-ext-tests December 19, 2022 09:55 — with GitHub Actions Inactive

thrau requested changes Dec 25, 2022

View reviewed changes

bentsku temporarily deployed to localstack-ext-tests December 28, 2022 16:18 — with GitHub Actions Inactive

baermat approved these changes Dec 29, 2022

View reviewed changes

localstack/services/sns/publisher.py Outdated Show resolved Hide resolved

bentsku force-pushed the sns-refactor-publish branch from 4677778 to 4a999af Compare December 30, 2022 12:19

bentsku temporarily deployed to localstack-ext-tests December 30, 2022 12:19 — with GitHub Actions Inactive

bentsku added 7 commits January 2, 2023 20:23

refactor SNS publishing and data models

11e0ce0

refactor batch for not only FIFO but SQS in general

58c572f

fix nits

df316c9

change todo

014d768

add env var to docker white list

0bacb87

add logging and docs

d5c18e4

fix typo + subscription_filter_policy assign

6a6500a

add a TODO and document

1cd0b80

bentsku force-pushed the sns-refactor-publish branch from 4a999af to 1cd0b80 Compare January 2, 2023 19:41

bentsku temporarily deployed to localstack-ext-tests January 2, 2023 19:41 — with GitHub Actions Inactive

amitassaraf mentioned this pull request Jan 3, 2023

fix: messages are sent out of order in SNS fifo topics -> SQS fifo queues #7418

Closed

bentsku requested a review from thrau January 3, 2023 11:52

thrau approved these changes Jan 3, 2023

View reviewed changes

thrau merged commit 59bfd5a into master Jan 3, 2023

thrau deleted the sns-refactor-publish branch January 3, 2023 18:20

whummer pushed a commit that referenced this pull request Jan 4, 2023

refactor SNS async publishing & ASF data models (#7267)

9cab529

bentsku mentioned this pull request Jan 4, 2023

bug: cannot receive SNS published push notification via GCM to Android Emulator #6238

Closed

1 task

bentsku mentioned this pull request Feb 13, 2023

fix: Unable to forward SNS message to SQS: An error occurred (AWS.SimpleQueueService.EmptyBatchRequest) when calling the SendMessageBatch #7663

Closed

		request_headers: Dict[str, str]


		class TopicPublisher(abc.ABC):

		class LambdaTopicPublisher(TopicPublisher):
		def _publish(self, context: SnsPublishContext, subscriber: SnsSubscription):

Uh oh!

refactor SNS async publishing & ASF data models #7267

refactor SNS async publishing & ASF data models #7267

Uh oh!

Conversation

bentsku commented Nov 30, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Nov 30, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

LocalStack integration with Pro

Uh oh!

coveralls commented Dec 1, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

baermat left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

thrau left a comment

Choose a reason for hiding this comment

Uh oh!

thrau Dec 21, 2022

Choose a reason for hiding this comment

Uh oh!

bentsku Dec 28, 2022

Choose a reason for hiding this comment

Uh oh!

Uh oh!

thrau Dec 24, 2022

Choose a reason for hiding this comment

Uh oh!

bentsku Dec 28, 2022

Choose a reason for hiding this comment

Uh oh!

bentsku Jan 2, 2023

Choose a reason for hiding this comment

Uh oh!

thrau Dec 24, 2022

Choose a reason for hiding this comment

Uh oh!

thrau Dec 24, 2022

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

baermat left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

thrau left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bentsku commented Jan 3, 2023

Uh oh!

Uh oh!

bentsku commented Nov 30, 2022 •

edited

Loading

github-actions bot commented Nov 30, 2022 •

edited

Loading

coveralls commented Dec 1, 2022 •

edited

Loading

thrau left a comment •

edited

Loading