add AWS json protocol support for SQS developer endpoints #11726

thrau · 2024-10-22T17:20:17Z

Motivation

There was a long-standing TODO for the sqs developer endpoints /_aws/sqs/messages to also support the modern x-amz-json-1.0 protocol introduced in #8268. This PR adds this functionality.

The implementation is not pretty but the best I could come up with without reworking the entire serialization mechanism. Basically through the use of the @aws_response_serializer decorator the protocol used to serialize the response was hard-coded, but needed to be dynamic per-request.

It's still a bit awkward because we documented that when calling this endpoint with application/json content type, it will return a JSON serialization of the query protocol response. So basically we have three protocols now: query (XML), x-amz-json-1.0 (for modern clients), and our weird/wrong json protocol that just returns the XML responses as json. That needs to be addressed at some point but is technically a breaking change.

Changes

When making AWS SQS client requests in x-amz-json-1.0 format to the endpoint localhost:4566/_aws/sqs/messages , the endpoint now correctly returns the JSON response.

TODO

What's left to do:

Update OpenAPI spec to contain the new input and output shape for x-amz-json-1.0

github-actions · 2024-10-22T18:02:51Z

LocalStack Community integration with Pro

2 files ± 0 2 suites ±0 1h 5m 31s ⏱️ - 36m 45s
2 490 tests - 1 019 2 267 ✅ - 829 223 💤 - 190 0 ❌ ±0
2 492 runs - 1 019 2 267 ✅ - 829 225 💤 - 190 0 ❌ ±0

Results for commit 131770e. ± Comparison against base commit 56e0b77.

This pull request removes 1037 and adds 18 tests. Note that renamed tests count towards both.

tests.aws.scenario.bookstore.test_bookstore.TestBookstoreApplication ‑ test_lambda_dynamodb
tests.aws.scenario.bookstore.test_bookstore.TestBookstoreApplication ‑ test_opensearch_crud
tests.aws.scenario.bookstore.test_bookstore.TestBookstoreApplication ‑ test_search_books
tests.aws.scenario.bookstore.test_bookstore.TestBookstoreApplication ‑ test_setup
tests.aws.scenario.kinesis_firehose.test_kinesis_firehose.TestKinesisFirehoseScenario ‑ test_kinesis_firehose_s3
tests.aws.scenario.lambda_destination.test_lambda_destination_scenario.TestLambdaDestinationScenario ‑ test_destination_sns
tests.aws.scenario.lambda_destination.test_lambda_destination_scenario.TestLambdaDestinationScenario ‑ test_infra
tests.aws.scenario.loan_broker.test_loan_broker.TestLoanBrokerScenario ‑ test_prefill_dynamodb_table
tests.aws.scenario.loan_broker.test_loan_broker.TestLoanBrokerScenario ‑ test_stepfunctions_input_recipient_list[step_function_input0-SUCCEEDED]
tests.aws.scenario.loan_broker.test_loan_broker.TestLoanBrokerScenario ‑ test_stepfunctions_input_recipient_list[step_function_input1-SUCCEEDED]
…

tests.aws.services.sqs.test_sqs_backdoor.TestSqsDeveloperEndpoints ‑ test_fifo_list_messages_as_botocore_endpoint_url[json-domain]
tests.aws.services.sqs.test_sqs_backdoor.TestSqsDeveloperEndpoints ‑ test_fifo_list_messages_as_botocore_endpoint_url[json-path]
tests.aws.services.sqs.test_sqs_backdoor.TestSqsDeveloperEndpoints ‑ test_fifo_list_messages_as_botocore_endpoint_url[json-standard]
tests.aws.services.sqs.test_sqs_backdoor.TestSqsDeveloperEndpoints ‑ test_fifo_list_messages_as_botocore_endpoint_url[query-domain]
tests.aws.services.sqs.test_sqs_backdoor.TestSqsDeveloperEndpoints ‑ test_fifo_list_messages_as_botocore_endpoint_url[query-path]
tests.aws.services.sqs.test_sqs_backdoor.TestSqsDeveloperEndpoints ‑ test_fifo_list_messages_as_botocore_endpoint_url[query-standard]
tests.aws.services.sqs.test_sqs_backdoor.TestSqsDeveloperEndpoints ‑ test_list_messages_as_botocore_endpoint_url[json-domain]
tests.aws.services.sqs.test_sqs_backdoor.TestSqsDeveloperEndpoints ‑ test_list_messages_as_botocore_endpoint_url[json-path]
tests.aws.services.sqs.test_sqs_backdoor.TestSqsDeveloperEndpoints ‑ test_list_messages_as_botocore_endpoint_url[json-standard]
tests.aws.services.sqs.test_sqs_backdoor.TestSqsDeveloperEndpoints ‑ test_list_messages_as_botocore_endpoint_url[query-domain]
…

♻️ This comment has been updated with latest results.

bentsku

I'll only comment as I've updated the PR a bit, let me know what you think @thrau! I've tried to remove the logic of the protocol outside of the method to avoid having nested functions, we still need it for the parser though. Happy to get feedback!

bentsku · 2024-10-22T22:27:42Z

localstack-core/localstack/services/sqs/provider.py

@@ -652,20 +690,14 @@ def list_messages(self, request: Request) -> ReceiveMessageResult:
        return self._get_and_serialize_messages(request, region, account_id, queue_name)

    @route("/_aws/sqs/messages/<region>/<account_id>/<queue_name>")


q: not sure if it's normal that this method accept all HTTP methods?

Perhaps we make this the same as the /_aws/sqs/messages route?

Suggested change

@route("/_aws/sqs/messages/<region>/<account_id>/<queue_name>")

@route("/_aws/sqs/messages/<region>/<account_id>/<queue_name>", methods=["GET", "POST"])

bentsku · 2024-10-22T22:28:54Z

localstack-core/localstack/services/sqs/provider.py

+            # only parse the request using a parser if it comes from an AWS client
+            protocol = get_sqs_protocol(request)
+            operation, service_request = create_parser(
+                load_service("sqs", protocol=protocol)


nit: I've updated it, as we're moving away from the sqs-query service name to the "service name" + "protocol" way of creating/selecting serializer and parser.

bentsku · 2024-10-22T22:29:57Z

localstack-core/localstack/services/sqs/provider.py

+    return "json" if content_type == "application/x-amz-json-1.0" else "query"
+
+
+def sqs_auto_protocol_aws_response_serializer(service_name: str, operation: str):


this is a bit verbose and copies the logic of aws_response_serializer to get the request out of it, I'd like to make it cleaner but not sure how 🤔 there's no way of getting the proper protocol before we're inside the method itself.

thrau · 2024-11-03T18:12:09Z

thanks for your changes @bentsku! merging this PR to unblock @webdev51 for the SQS resource browser improvements. we still need to add the x-amz-json format to the openapi spec so we can re-enable this check here

localstack/tests/aws/services/sqs/test_sqs_backdoor.py

Line 29 in b5823f8

# @pytest.mark.usefixtures("openapi_validate")

thrau added 2 commits October 22, 2024 19:18

refactor sqs developer endpoints to support multiple protocols

570e67b

fix tests

9eb7c58

thrau changed the title ~~Sqs dev endpoints json~~ add AWS json protocol support for SQS developer endpoints Oct 22, 2024

thrau added the semver: minor Non-breaking changes which can be included in minor releases, but not in patch releases label Oct 22, 2024

create auto serializer

131770e

bentsku reviewed Oct 22, 2024

View reviewed changes

thrau marked this pull request as ready for review November 3, 2024 18:08

thrau requested a review from baermat as a code owner November 3, 2024 18:08

thrau merged commit b5823f8 into master Nov 3, 2024
34 checks passed

thrau deleted the sqs-dev-endpoints-json branch November 3, 2024 18:08

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

add AWS json protocol support for SQS developer endpoints #11726

add AWS json protocol support for SQS developer endpoints #11726

Uh oh!

thrau commented Oct 22, 2024 •

edited

Loading

Uh oh!

github-actions bot commented Oct 22, 2024 •

edited

Loading

Uh oh!

bentsku left a comment

Uh oh!

bentsku Oct 22, 2024

Uh oh!

gregfurman Oct 23, 2024

Uh oh!

bentsku Oct 22, 2024

Uh oh!

bentsku Oct 22, 2024

Uh oh!

Uh oh!

thrau commented Nov 3, 2024

Uh oh!

Uh oh!

		@@ -652,20 +690,14 @@ def list_messages(self, request: Request) -> ReceiveMessageResult:
		return self._get_and_serialize_messages(request, region, account_id, queue_name)

		@route("/_aws/sqs/messages/<region>/<account_id>/<queue_name>")

		return "json" if content_type == "application/x-amz-json-1.0" else "query"


		def sqs_auto_protocol_aws_response_serializer(service_name: str, operation: str):

Uh oh!

add AWS json protocol support for SQS developer endpoints #11726

add AWS json protocol support for SQS developer endpoints #11726

Uh oh!

Conversation

thrau commented Oct 22, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

Changes

TODO

Uh oh!

github-actions bot commented Oct 22, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

LocalStack Community integration with Pro

Uh oh!

bentsku left a comment

Choose a reason for hiding this comment

Uh oh!

bentsku Oct 22, 2024

Choose a reason for hiding this comment

Uh oh!

gregfurman Oct 23, 2024

Choose a reason for hiding this comment

Uh oh!

bentsku Oct 22, 2024

Choose a reason for hiding this comment

Uh oh!

bentsku Oct 22, 2024

Choose a reason for hiding this comment

Uh oh!

Uh oh!

thrau commented Nov 3, 2024

Uh oh!

Uh oh!

thrau commented Oct 22, 2024 •

edited

Loading

github-actions bot commented Oct 22, 2024 •

edited

Loading