add support for xml to json responses #9102

calvernaz · 2023-09-09T06:20:17Z

Motivation

Rework this PR to instead of forcing xml to json format exchange, it adds parity to AWS by defaulting SQS responses to JSON. It also works if the request parameter is set to application/xml, in that case, the response from the integration is not JSON but XML.

Changes

Applies the same defaults as AWS, the default response being JSON.
It also adds a basic implementation of update_integration and update_integration_response just to comply with the tests. A follow-up PR will improve the implementation.
Snapshot tests and exercise multiple response formats

coveralls · 2023-09-09T07:57:12Z

coverage: 79.758% (+0.04%) from 79.72% when pulling f66acfc on 9101-bug-response-templates-not-rendering-sqs-response-from-xml-to-json into ca27593 on master.

github-actions · 2023-09-09T08:12:49Z

LocalStack Community integration with Pro

      2 files       2 suites 1h 20m 23s ⏱️
2 181 tests 1 698 ✔️ 483 💤 0 ❌
2 182 runs 1 698 ✔️ 484 💤 0 ❌

Results for commit f66acfc.

♻️ This comment has been updated with latest results.

bentsku

Thanks a lot for adding this!

I’m not sure what was the behaviour of AWS before SQS was able to return JSON, and if this is a result of it, or if they also convert the XML under the hood.  See #8267 (comment) and #8268

Basically, I’m wondering if AWS uses JSON under the hood for the SQS response instead of parsing XML, or if their implementation of VTL is able to take XML as input. I suppose could test it with an HTTP integration and trying to do the same, and see if that works.  

I think it doesn’t hurt to be able to do it now, but I’m not sure if this should be only for SQS until we implement JSON, or if it should be part of every template rendering logic.

  After giving it a try in the AWS console, it seems AWS is unable to do so, it can’t parse XML as input from the integration response, I tried with http://httpbin.org/xml and the template was not being applied nor parsed.

This was the API I tried it on with the nice test console.  

openapi: "3.0.1"
info:
  title: "test-api-xml"
  version: "2023-09-09T15:26:38Z"
servers:
- url: "/{basePath}"
  variables:
    basePath:
      default: "test"
paths:
  /test:
    get:
      responses:
        "200":
          description: "200 response"
          content:
            application/json:
              schema:
                $ref: "#/components/schemas/Empty"
      x-amazon-apigateway-integration:
        httpMethod: "GET"
        uri: "http://httpbin.org/xml"
        responses:
          default:
            statusCode: "200"
            responseTemplates:
              application/json: "#set($responseBody = $input.path('$.slideshow'))\n\
                #set($requestId = $input.path('$.slideshow.slide[0].title'))\n#set($messageId\
                \ = $slideshow.slide[1].title)\n{\n    \"title1\": \"$requestId\"\
                ,\n    \"title2\": \"$messageId\"\n}\n"
        passthroughBehavior: "when_no_match"
        type: "http"
components:
  schemas:
    Empty:
      title: "Empty Schema"
      type: "object"

The logs from the test console seemed to show it wasn't parsing the response, but if you'd like to give it a try too, feel free as I'm not 100% sure if I missed a step. I tried to look online and it seems people went to great length to be able to just only pass XML from a lambda.

I also tried an SQS API, and you can see API Gateway is sending the Accept=application/json header, so SQS should respond in JSON too.  
Logs from the test console:

Sat Sep 09 15:57:46 UTC 2023 : Sending request to https://sqs.us-east-1.amazonaws.com/<account-id>/queue-518a623e
Sat Sep 09 15:57:46 UTC 2023 : Received response. Status: 200, Integration latency: 34 ms
Sat Sep 09 15:57:46 UTC 2023 : Endpoint response headers: {x-amzn-RequestId=315726e7-2463-5735-80bc-7b91bc59149e, X-Amzn-Trace-Id=Root=1-64fc95fa-630fac6d04ebca8e4d4f74ae, Date=Sat, 09 Sep 2023 15:57:46 GMT, Content-Type=application/json, Content-Length=312, connection=keep-alive}
Sat Sep 09 15:57:46 UTC 2023 : Endpoint response body before transformations: {"SendMessageResponse":{"ResponseMetadata":{"RequestId":"315726e7-2463-5735-80bc-7b91bc59149e"},"SendMessageResult":{"MD5OfMessageAttributes":null,"MD5OfMessageBody":"0c38abe3676e5cf7048b3e40018ea418","MD5OfMessageSystemAttributes":null,"MessageId":"a8e25ba3-2165-45d0-8e71-db475f6a4714","SequenceNumber":null}}}

So I’d be in favour to not have this as part of the global template rendering, but just in the SQS integration for now, until we can force the Content-Type to application/json to SQS so that it can return JSON natively so we could remove this workaround. What do you think? I don't know if other services using XML are possible to test?

Also, awesome test, thanks, it made it super easy to test it 👍

localstack/services/apigateway/templates.py

calvernaz · 2023-09-09T18:54:18Z

Thanks a lot for adding this!

Starting from the bottom just for the sake of reproducibility.

I also tried an SQS API, and you can see API Gateway is sending the Accept=application/json header, so SQS should respond in JSON too.   Logs from the test console:

Sat Sep 09 15:57:46 UTC 2023 : Sending request to https://sqs.us-east-1.amazonaws.com/<account-id>/queue-518a623e
Sat Sep 09 15:57:46 UTC 2023 : Received response. Status: 200, Integration latency: 34 ms
Sat Sep 09 15:57:46 UTC 2023 : Endpoint response headers: {x-amzn-RequestId=315726e7-2463-5735-80bc-7b91bc59149e, X-Amzn-Trace-Id=Root=1-64fc95fa-630fac6d04ebca8e4d4f74ae, Date=Sat, 09 Sep 2023 15:57:46 GMT, Content-Type=application/json, Content-Length=312, connection=keep-alive}
Sat Sep 09 15:57:46 UTC 2023 : Endpoint response body before transformations: {"SendMessageResponse":{"ResponseMetadata":{"RequestId":"315726e7-2463-5735-80bc-7b91bc59149e"},"SendMessageResult":{"MD5OfMessageAttributes":null,"MD5OfMessageBody":"0c38abe3676e5cf7048b3e40018ea418","MD5OfMessageSystemAttributes":null,"MessageId":"a8e25ba3-2165-45d0-8e71-db475f6a4714","SequenceNumber":null}}}

You just said it there, ApiGW does send the "Accept" header as "application/json". In order to reproduce it, you would have to set the header to "application/xml" and you would get the following response:

request:

Sat Sep 09 18:28:24 UTC 2023 : Endpoint request headers: {Authorization=*****************************************************************************************************************************************************************************************************************************************
*****************************77efbc, X-Amz-Date=20230909T182824Z, x-amzn-apigateway-api-id=g44647ybs7, 
Accept=application/xml, User-Agent=AmazonAPIGateway_g44647ybs7, X-Amz-Security-Token=IQoJb3JpZ2luX2VjEOv///////

response:

<?xml version="1.0"?><SendMessageResponse xmlns="http://queue.amazonaws.com/doc/2012-11-05/">
<SendMessageResult><MessageId>0a772d86-0a15-45f8-872d-fb41018b411c</MessageId>
<MD5OfMessageBody>dcdf2bdd90cfccde643defa7df7c75a2</MD5OfMessageBody></SendMessageResult>
<ResponseMetadata><RequestId>8c6c3c94-673e-54d4-9db6-61e2ddb02bb4</RequestId></ResponseMetadata></SendMessageResponse>

I’m not sure what was the behaviour of AWS before SQS was able to return JSON, and if this is a result of it, or if they also convert the XML under the hood.  See #8267 (comment) and #8268

Basically, I’m wondering if AWS uses JSON under the hood for the SQS response instead of parsing XML, or if their implementation of VTL is able to take XML as input. I suppose could test it with an HTTP integration and trying to do the same, and see if that works.  

I think it doesn’t hurt to be able to do it now, but I’m not sure if this should be only for SQS until we implement JSON, or if it should be part of every template rendering logic.

  After giving it a try in the AWS console, it seems AWS is unable to do so, it can’t parse XML as input from the integration response, I tried with http://httpbin.org/xml and the template was not being applied nor parsed.

I’m not sure what was the behaviour of AWS before SQS was able to return JSON, and if this is a result of it, or if they also convert the XML under the hood.  See #8267 (comment) and #8268

Basically, I’m wondering if AWS uses JSON under the hood for the SQS response instead of parsing XML, or if their implementation of VTL is able to take XML as input. I suppose could test it with an HTTP integration and trying to do the same, and see if that works.

So it seems this was a waste of time :) The problem stems from the issues you mentioned above and I can confirm AWS doesn't support xml transformations, so I wonder if this should be implemented it all.
 

bentsku

Interesting that we actually JSON from SQS! Looking at the PR of Alex I've linked earlier, it seems this point was marked as done:

Implement a parser and serializer facade which can dynamically select the right parser/serializer based on the content type / accept headers of the request.

So content negotiation was in fact implemented! Thanks a lot for finding this out, this simplifies the PR a lot!

However, I'm a bit worried about the UpdateIntegration and UpdateIntegrationResponse changes, could this break some user setup? What is the current state of those operation? Is it done in moto? Because how I see it, they would now throw exception for most of the calls? Where maybe before it would not have?

I think we have some patching going on for Integration (apigateway_response_integrations in patches.py), so I feel like the fix should either go there instead of implementing something not entirely which may break some Terraform setup, even though I know the current implementation is not great, or then be tested and moved to our provider.

It seems like UpdateIntegrationResponse might not be implemented at all, so I'm not sure about this one: if they are no tests for it, I'd more in favour to not have it not raise specific exceptions if we're not certain they're being raised. So it would do what you want it do to, but if someone tries to do something that works against AWS but we raise exceptions, that could be a bit blocking.

As an example, I think UpdateIntegration actually supports more: /cacheKeyParameters, /requestParameters and /requestTemplates. But now we would be raising exception if it's not /requestParameters only.

bentsku · 2023-09-10T21:56:25Z

localstack/services/apigateway/provider.py

+            if "/requestParameters" not in path:
+                raise BadRequestException(
+                    f"Invalid patch path  '{path}' specified for op '{op}'. Must be one of: [/requestParameters]"
+                )
+            if op != "add":
+                raise BadRequestException(
+                    f"Invalid patch path  '{path}' specified for op '{op}'. Please choose supported operations"
+                )


I think this is too strict, and would raise Exceptions wen AWS doesn't.

too strict because because we don't implement other operations. I followed the same idea of update_resource you did given your experience in the patch zone.

The implementation is good, it's just that the exceptions raised in update_resource are raised by AWS too, and I'm worried it's not the case here.

bentsku · 2023-09-10T21:57:04Z

localstack/services/apigateway/provider.py

+            if "/responseTemplates" not in path:
+                raise BadRequestException(
+                    f"Invalid patch path  '{path}' specified for op '{op}'. Must be one of: [/requestParameters]"
+                )
+            if op != "remove":
+                raise BadRequestException(
+                    f"Invalid patch path  '{path}' specified for op '{op}'. Please choose supported operations"
+                )


Same with this, this is too strict and could break user setup

yep it's strict because this is just a basic implementation to make to test pass. And does not implement any other operation.

localstack/services/apigateway/provider.py

calvernaz · 2023-09-11T07:30:21Z

Interesting that we actually JSON from SQS! Looking at the PR of Alex I've linked earlier, it seems this point was marked as done:

Implement a parser and serializer facade which can dynamically select the right parser/serializer based on the content type / accept headers of the request.

So content negotiation was in fact implemented! Thanks a lot for finding this out, this simplifies the PR a lot!

However, I'm a bit worried about the UpdateIntegration and UpdateIntegrationResponse changes, could this break some user setup? What is the current state of those operation? Is it done in moto? Because how I see it, they would now throw exception for most of the calls? Where maybe before it would not have?

I think we have some patching going on for Integration (apigateway_response_integrations in patches.py), so I feel like the fix should either go there instead of implementing something not entirely which may break some Terraform setup, even though I know the current implementation is not great, or then be tested and moved to our provider.

We do, and I will double check thanks for point that out, but will have to understand why did not pick up the case I had.

It seems like UpdateIntegrationResponse might not be implemented at all, so I'm not sure about this one: if they are no tests for it, I'd more in favour to not have it not raise specific exceptions if we're not certain they're being raised. So it would do what you want it do to, but if someone tries to do something that works against AWS but we raise exceptions, that could be a bit blocking.

As an example, I think UpdateIntegration actually supports more: /cacheKeyParameters, /requestParameters and /requestTemplates. But now we would be raising exception if it's not /requestParameters only.

It does support more, but we don't I notice this by actually applying this in the test and figure there was no implementation.

Interesting that we actually JSON from SQS! Looking at the PR of Alex I've linked earlier, it seems this point was marked as done:

Implement a parser and serializer facade which can dynamically select the right parser/serializer based on the content type / accept headers of the request.

So content negotiation was in fact implemented! Thanks a lot for finding this out, this simplifies the PR a lot!

However, I'm a bit worried about the UpdateIntegration and UpdateIntegrationResponse changes, could this break some user setup? What is the current state of those operation? Is it done in moto? Because how I see it, they would now throw exception for most of the calls? Where maybe before it would not have?

I think we have some patching going on for Integration (apigateway_response_integrations in patches.py), so I feel like the fix should either go there instead of implementing something not entirely which may break some Terraform setup, even though I know the current implementation is not great, or then be tested and moved to our provider.

It seems like UpdateIntegrationResponse might not be implemented at all, so I'm not sure about this one: if they are no tests for it, I'd more in favour to not have it not raise specific exceptions if we're not certain they're being raised. So it would do what you want it do to, but if someone tries to do something that works against AWS but we raise exceptions, that could be a bit blocking.

Overall, we want to move away from these patches as much as possible. Hard to follow and it's just builds on top of a model we are splitting way anyway

As an example, I think UpdateIntegration actually supports more: /cacheKeyParameters, /requestParameters and /requestTemplates. But now we would be raising exception if it's not /requestParameters only.

It does support more, no questions about that, but we don't. I figure this my implementing a test that was updating the integration and integration response https://github.com/localstack/localstack/pull/9102/files#diff-357303ef2a04cdb9ff3ebd56f3a8293cd23b26e77a5e8ef0fd72095547076d78R243-R256

bentsku · 2023-09-11T09:20:34Z

Overall, we want to move away from these patches as much as possible. Hard to follow and it's just builds on top of a model we are splitting way anyway

Totally agree, was suggesting the patch route as there was already an implementation, might not be working but I figured that'd be faster for you.

Overall I just don't want to raise exception if we don't support it at the moment and we did not raise exception before.
I know it'd be better to fail if we don't support it, especially for UpdateIntegration, but as we already have something there that does not raise exception, I'm worried of breaking existing user setup (that might not be working perfectly I agree) and blocking them when deploying. It would be ok to just do nothing in that case if we don't support it.

bentsku

Thanks a lot for removing the exceptions raised! Just a minor comment regarding the new patch to integration_responses, I think this might not be needed as we're using the provider for it, but other than that we're good!

bentsku · 2023-09-11T14:04:05Z

localstack/services/apigateway/patches.py

+    @patch(APIGatewayResponse.integration_responses)
+    def apigateway_response_integration_responses(fn, self, request, *args, **kwargs):
+        result = fn(self, request, *args, **kwargs)
+        if self.method not in ["PATCH"]:
+            return result
+
+        url_path_parts = self.path.split("/")
+        function_id = url_path_parts[2]
+        resource_id = url_path_parts[4]
+        method_type = url_path_parts[6]
+
+        integration = self.backend.get_integration(function_id, resource_id, method_type)
+        if not integration:
+            return result
+
+        if self.method == "PATCH":
+            patch_operations = self._get_param("patchOperations")
+            apply_json_patch_safe(integration.to_json(), patch_operations, in_place=True)
+            return 200, {}, json.dumps(integration.to_json())
+


q: do we need this? or is it implemented with update_integration_response in the provider? I'm not sure I follow as it seems we're patching the whole integration here and not the integration_response, and we're patching only the result of to_json() and not the entity in itself.
It seems from moto this method would be more like:

integration_response = self.backend.get_integration_response( function_id, resource_id, method_type, status_code )

I think this might not be called as we would use the provider operation instead.

🤦 no sorry, I fixed the integration and was trying to get the patch for the integration response either but stumbled in a different issue, so I'll remove this.

bentsku

LGTM! Thanks for addressing all the comments! 🚀

whummer

Looks great, glad we were able to get to the bottom of this and come up with a solid fix, after a few iterations. 🚀 Kudos for keeping track of all the conversation in detail, which made it easy to follow the process 🙌

localstack/services/apigateway/helpers.py

calvernaz linked an issue Sep 9, 2023 that may be closed by this pull request

bug: Response templates not rendering SQS response from xml to json #9101

Closed

1 task

calvernaz marked this pull request as draft September 9, 2023 06:20

calvernaz added the semver: minor Non-breaking changes which can be included in minor releases, but not in patch releases label Sep 9, 2023

calvernaz self-assigned this Sep 9, 2023

calvernaz marked this pull request as ready for review September 9, 2023 06:52

calvernaz marked this pull request as draft September 9, 2023 07:12

calvernaz marked this pull request as ready for review September 9, 2023 11:23

calvernaz requested review from whummer and bentsku September 9, 2023 11:23

bentsku reviewed Sep 9, 2023

View reviewed changes

localstack/services/apigateway/templates.py Outdated Show resolved Hide resolved

calvernaz force-pushed the 9101-bug-response-templates-not-rendering-sqs-response-from-xml-to-json branch from 3f01af3 to 1239f9c Compare September 10, 2023 09:22

bentsku reviewed Sep 10, 2023

View reviewed changes

add support to xml or json responses from SQS

babc8ab

calvernaz force-pushed the 9101-bug-response-templates-not-rendering-sqs-response-from-xml-to-json branch from 80b6439 to babc8ab Compare September 11, 2023 11:52

bentsku reviewed Sep 11, 2023

View reviewed changes

add support for xml to json responses

535e699

bentsku approved these changes Sep 11, 2023

View reviewed changes

whummer approved these changes Sep 11, 2023

View reviewed changes

localstack/services/apigateway/helpers.py Outdated Show resolved Hide resolved

calvernaz added 3 commits September 11, 2023 21:35

add support for xml to json responses

6f484af

add support for xml to json responses

dd6954e

add support for xml to json responses

f66acfc

calvernaz merged commit 16d3c9d into master Sep 12, 2023

calvernaz deleted the 9101-bug-response-templates-not-rendering-sqs-response-from-xml-to-json branch September 12, 2023 07:19

Uh oh!

add support for xml to json responses #9102

add support for xml to json responses #9102

Uh oh!

Conversation

calvernaz commented Sep 9, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

Changes

Uh oh!

coveralls commented Sep 9, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Sep 9, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

LocalStack Community integration with Pro

Uh oh!

bentsku left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

calvernaz commented Sep 9, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

bentsku left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bentsku Sep 10, 2023

Choose a reason for hiding this comment

Uh oh!

calvernaz Sep 11, 2023

Choose a reason for hiding this comment

Uh oh!

bentsku Sep 11, 2023

Choose a reason for hiding this comment

Uh oh!

bentsku Sep 10, 2023

Choose a reason for hiding this comment

Uh oh!

calvernaz Sep 11, 2023

Choose a reason for hiding this comment

Uh oh!

Uh oh!

calvernaz commented Sep 11, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

bentsku commented Sep 11, 2023

Uh oh!

bentsku left a comment

Choose a reason for hiding this comment

Uh oh!

bentsku Sep 11, 2023

Choose a reason for hiding this comment

Uh oh!

calvernaz Sep 11, 2023

Choose a reason for hiding this comment

Uh oh!

bentsku left a comment

Choose a reason for hiding this comment

Uh oh!

whummer left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

calvernaz commented Sep 9, 2023 •

edited

Loading

coveralls commented Sep 9, 2023 •

edited

Loading

github-actions bot commented Sep 9, 2023 •

edited

Loading

bentsku left a comment •

edited

Loading

calvernaz commented Sep 9, 2023 •

edited

Loading

bentsku left a comment •

edited

Loading

calvernaz commented Sep 11, 2023 •

edited

Loading