Thanks to visit codestin.com
Credit goes to github.com

Skip to content

feat: integration test suite #51

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 29 commits into from
Sep 14, 2021
Merged

feat: integration test suite #51

merged 29 commits into from
Sep 14, 2021

Conversation

erunion
Copy link
Member

@erunion erunion commented Sep 10, 2021

This-Is-Fine jpg

This completely overhauls the existing integration suite that HTTPSnippet has in place in requests.js for running generated snippets for a handful of languages to determine they're sending the data that they should be.

🏚 Why doesn't this current iteration work for us?

  1. There aren't enough tests. Currently it only runs snippets for Node native, PHP curl and Python python3. And of those 3 client targets, only 9 of the 17 total request fixtures that we have tests in place are being executed.
  2. The tests currently hit https://mockbin.com/har which while great in that it turns the incoming request back into a HAR, it doesn't give the clearest picture of what the payload of incoming data actually looks like.
  3. The recent issue that cropped up with cURL snippets adding an unnecessary boundary (RM-2107) has me a bit spooked because without individually auditing each individual client, their targets, and the different test cases we just don't know if any others have this (or other problems).
    • I ran into a similar thing with the new Guzzle target while fixing it up where it just didn't work at all and if I hadn't run the thing myself manually its non-functional snippets would have made it out into production.

We can do better.

🏥 So what the hell did I do here?

Over the past year while we've been building up the reference redesign I've found that https://httpbin.org has been absolutely indispensable for debugging all kinds of API request quirks because it returns your data back in ways that are easy to quickly parse and test against.

For example:

$ curl --request POST \
  --url 'https://mockbin.com/har?foo=bar&foo=baz&baz=abc&key=value' \
  --header 'accept: application/json' \
  --header 'content-type: application/x-www-form-urlencoded' \
  --cookie 'foo=bar; bar=baz' \
  --data foo=bar
{
  "args": {
    "baz": "abc",
    "foo": [
      "bar",
      "baz"
    ],
    "key": "value"
  },
  "data": "",
  "files": {},
  "form": {
    "foo": "bar"
  },
  "headers": {
    "Accept": "application/json",
    "Content-Length": "7",
    "Content-Type": "application/x-www-form-urlencoded",
    "Cookie": "foo=bar; bar=baz",
    "Host": "httpbin.org",
    "User-Agent": "curl/7.54.0",
    "X-Amzn-Trace-Id": "Root=1-613be8db-21ff3cb559a8ebb55d951108"
  },
  "json": null,
  "method": "POST",
  "origin": "REDACTED",
  "url": "https://httpbin.org/anything?foo=bar&foo=baz&baz=abc&key=value"
}

Here it's really easy to see how my query parameters were parsed, x-www-form-urlencoded data was sent as form data, and cookies were sent as a proper Cookie header.

Now while this same information is also available with mockbin, I don't need to parse a HAR (again) to check request.postData.mimeType for application/x-www-form-urlencoded and then see if request.post.params has my data. This easy response from HTTPBin also makes it really easy for us to flesh out the HAR fixtures that this library builds tests around to have full response information in them. This full HAR can now also be used to quickly do assertions.

So with that I've completely rewritten the existing requests.js into a new full integration suite that will now let us run a handful of tests locally when running npm test but also a full blown Docker suite to run clients and targets that we can easily containerize.

💁‍♂️ How does it work?

In #49 I updated all of the existing HAR fixtures to store data in the responses section that's the data that we get back from an https://httpbin.org/anything, https://httpbin.org/headers, or https://httpbin.org/cookies request.

The integration suite now picks up existing snippets from the test suite (stored in __tests__/__fixtures/output/:client/:target/:snippettest) and runs those depending on a new config in the client informing us of how they should be run (Node scripts are run with node, PHP with php, Shell scripts just run as themselves, etc).

These snippets hit one of the available endpoints on https://httpbin.org for its test and then we check to see if the response matches what we have already stored in the HAR fixture. If it doesn't match, the integration test fails.

That's it! :easy:

🏄 tl;dr

  • Overhauled the existing integration suite to be easier to configure and extend.
  • Swapped out calls to https://mockbin.com for https://httpbin.org.
  • Renamed the http test to http-insecure because Python snippets were attempting to load the http.py test instead of the http module.
  • Changed a few targets to no longer send query parameters separately from the URL as we encounter double encoding issues in some cases.
  • Constructed a Docker framework, and Github Workflows, for running generated fixture code snippets, that need their own suite of dependencies, within a container.

Note: Since this work depends on HTTPBin we can't upstream this work, but we can upstream any fixes to client targets that come from it.

🧰 Additional Fixes

Included within this work are also a heap of fixes for some client targets that I discovered while running these tests, they include:

  • node + fetch
    • Promoted the useObjectBody option that we have for node-fetch to be the default behavior. Within the example fixture for application-json we have a "f\"oo" string that was getting sent to the server as "f"oo" causing HTTPBin to not see it as JSON.
    • No longer sends a Content-Type header if the payload is multipart/form-data because we push that data into an instance of the form-data module and that takes care of the Content-Type header automatically. The custom header that was getting set there was causing data from these requests to get to the server at all.
  • node + request
    • Snippets were packaging cookies into an instance of request.jar() but then sending that cookie har as 'jar' instead of the variable -- resulting in cookies never being sent.
    • No longer sends query parameters with the qs option, instead relying on sending them as part of the URL. Made this change because query parameters were sometimes either being double encoded, or array params weren't being sent as a proper array.
  • node + axios
    • Changed all var instances over to const.
    • For x-www-form-urlencoded requests params are now being sent to Axios within an instance of URLSearchParams.
    • No longer sending query parameters with the params object because of encoding issues.
  • python + requests
    • No longer sends query parameters with the params option as encoded values were being encoded again with its default form encoding. We're instead supplying query params in the URL.

🧬 QA + Testing

Clients and targets that are now being tested within this new suite:

  • Node
    • axios
    • http (aka "native")
    • node-fetch
    • request
  • PHP
    • curl
    • guzzle
  • Python
    • python3
    • request
  • Shell
    • curl

How do I run the integration tests?

There's two forms of the integration test suite (__tests__/integration.test.js):

  1. Local (via npm test)
    • This is also what runs in our standard CI workflow.
  2. Containers

Running the local instance you can do with either running npm test or npx jest __test__/integration.test.js and that'll run integration tests for Node, PHP, and Python across a single target for each (http for Node, curl for PHP, and python3 for Python).

The containerized suite is where the full boar of tests are happening because we want to contain all of the necessary dependencies for each of these clients within that container (Composer, Guzzle, requests, all of the NPM packages for each target, etc.). Running this suite is as easy as running the following:

docker-compose run integration_php

This'll build out the container and its dependencies and then execute npx jest __test__/integration.test.js with a few environment variables to tell the suite to only run tests for PHP.

Note that after you're done you'll probably want to run docker-compose down to clean everything up.

@erunion erunion added the enhancement New feature or request label Sep 10, 2021
@@ -52,7 +52,6 @@ module.exports = {
hello: 'world',
},
headers: {
Accept: '*/*',
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removed all these Accept: */* headers from the response configs because I originally built these response configs from data coming back from a cURL request and cURL adds that Accept header to the request (and response since HTTPBin shows incoming headers here) where other targets don't.

@erunion erunion added the refactor Issues about tackling technical debt label Sep 10, 2021
method: 'POST',
url: 'https://httpbin.org/anything',
headers: {'content-type': 'application/x-www-form-urlencoded'},
data: {foo: 'bar', hello: 'world'}
data: encodedParams
Copy link
Member Author

@erunion erunion Sep 13, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Made this change because this data was getting to HTTPBin actually as the following:

{
  args: {},
  data: '',
  files: {},
  form: { '{"foo":"bar","hello":"world"}': '' },
  headers: {
    Accept: 'application/json, text/plain, */*',
    'Content-Length': '29',
    'Content-Type': 'application/x-www-form-urlencoded',
    Host: 'httpbin.org',
    'User-Agent': 'axios/0.21.4',
    'X-Amzn-Trace-Id': 'Root=1-613fc81c-33b383d80f53e0750840173b'
  },
  json: null,
  method: 'POST',
  origin: 'REDACTED',
  url: 'https://httpbin.org/anything'
}

With this fix:

{
  args: {},
  data: '',
  files: {},
  form: { foo: 'bar', hello: 'world' },
  headers: {
    Accept: 'application/json, text/plain, */*',
    'Content-Length': '19',
    'Content-Type': 'application/x-www-form-urlencoded',
    Host: 'httpbin.org',
    'User-Agent': 'axios/0.21.4',
    'X-Amzn-Trace-Id': 'Root=1-613fc852-1b54e94f1b77ddac478d20a4'
  },
  json: null,
  method: 'POST',
  origin: 'REDACTED',
  url: 'https://httpbin.org/anything'
}

method: 'POST',
url: 'https://httpbin.org/anything',
params: {foo: ['bar', 'baz'], baz: 'abc', key: 'value'},
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Made this change because query params were getting to HTTPBin as the following:

  args: { baz: 'abc', 'foo[]': [ 'bar', 'baz' ], key: 'value' },

With this fix:

  args: { baz: 'abc', foo: [ 'bar', 'baz' ], key: 'value' },

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

interesting, so foo[] is an old way people used to identify that multiple identical parameters should be treated as an array. I haven't seen it for quite some time though. In fact it doesn't really even seem to be supported as an OAS style.

I'm going to poke around axios a bit and see why they're still using this pattern by default.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To get this working right with URLSearchParams we'll need to run through arrays and add them in separately:

const params = new URLSearchParams();
params.append('foo', 'bar');
params.append('foo', 'baz');
params.append('baz', 'abc');
params.append('key', 'value');

I'm going to leave it with having query params in the URL because it'll be less noisy.

@@ -4,7 +4,14 @@ const url = 'https://httpbin.org/anything';
const options = {
method: 'POST',
headers: {'content-type': 'application/json'},
body: '{"number":1,"string":"f\"oo","arr":[1,2,3],"nested":{"a":"b"},"arr_mix":[1,"a",{"arr_mix_nested":{}}],"boolean":false}'
Copy link
Member Author

@erunion erunion Sep 13, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Prior to this fix, see json in this returned payload:

{
  args: {},
  data: '{"number":1,"string":"f"oo","arr":[1,2,3],"nested":{"a":"b"},"arr_mix":[1,"a",{"arr_mix_nested":{}}],"boolean":false}',
  files: {},
  form: {},
  headers: {
    Accept: '*/*',
    'Accept-Encoding': 'gzip,deflate',
    'Content-Length': '117',
    'Content-Type': 'application/json',
    Host: 'httpbin.org',
    'User-Agent': 'node-fetch/1.0 (+https://github.com/bitinn/node-fetch)',
    'X-Amzn-Trace-Id': 'Root=1-613fc993-7f43a7fc522f744c69c69531'
  },
  json: null,
  method: 'POST',
  origin: 'REDACTED',
  url: 'https://httpbin.org/anything'
}

And now:

{
  args: {},
  data: '{"number":1,"string":"f\\"oo","arr":[1,2,3],"nested":{"a":"b"},"arr_mix":[1,"a",{"arr_mix_nested":{}}],"boolean":false}',
  files: {},
  form: {},
  headers: {
    Accept: '*/*',
    'Accept-Encoding': 'gzip,deflate',
    'Content-Length': '118',
    'Content-Type': 'application/json',
    Host: 'httpbin.org',
    'User-Agent': 'node-fetch/1.0 (+https://github.com/bitinn/node-fetch)',
    'X-Amzn-Trace-Id': 'Root=1-613fc98a-181e57cc5b875d915032bfc7'
  },
  json: {
    arr: [ 1, 2, 3 ],
    arr_mix: [ 1, 'a', [Object] ],
    boolean: false,
    nested: { a: 'b' },
    number: 1,
    string: 'f"oo'
  },
  method: 'POST',
  origin: 'REDACTED',
  url: 'https://httpbin.org/anything'
}

Note: This is because of promoting the useObjectBody option to be the default behavior. We already ran this library with this option turned on in @readme/oas-to-snippet so there's going to be no change to our customers.

Also ignore the [Object] in there, that's from console.log's max depth in the snippet not a side effect of this work.

@@ -6,10 +6,7 @@ const formData = new FormData();
formData.append('foo', fs.createReadStream('__tests__/__fixtures__/files/hello.txt'));

const url = 'https://httpbin.org/anything';
const options = {
method: 'POST',
headers: {'content-type': 'multipart/form-data; boundary=---011000010111000001101001'}
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Prior to this fix check files in this returned payload:

{
  args: {},
  data: '',
  files: {},
  form: {},
  headers: {
    Accept: '*/*',
    'Accept-Encoding': 'gzip,deflate',
    'Content-Type': 'multipart/form-data; boundary=---011000010111000001101001',
    Host: 'httpbin.org',
    'Transfer-Encoding': 'chunked',
    'User-Agent': 'node-fetch/1.0 (+https://github.com/bitinn/node-fetch)',
    'X-Amzn-Trace-Id': 'Root=1-613fca42-5d12267c4f32fd05379c5114'
  },
  json: null,
  method: 'POST',
  origin: 'REDACTED',
  url: 'https://httpbin.org/anything'
}

And now:

{
  args: {},
  data: '',
  files: { foo: 'Hello World\n' },
  form: {},
  headers: {
    Accept: '*/*',
    'Accept-Encoding': 'gzip,deflate',
    'Content-Type': 'multipart/form-data;boundary=--------------------------939643642041348712808456',
    Host: 'httpbin.org',
    'Transfer-Encoding': 'chunked',
    'User-Agent': 'node-fetch/1.0 (+https://github.com/bitinn/node-fetch)',
    'X-Amzn-Trace-Id': 'Root=1-613fca3a-2893e2fc3bceaa1e7d42a3d2'
  },
  json: null,
  method: 'POST',
  origin: 'REDACTED',
  url: 'https://httpbin.org/anything'
}

@@ -4,7 +4,7 @@ const jar = request.jar();
jar.setCookie(request.cookie('foo=bar'), 'https://httpbin.org/cookies');
jar.setCookie(request.cookie('bar=baz'), 'https://httpbin.org/cookies');

const options = {method: 'GET', url: 'https://httpbin.org/cookies', jar: 'JAR'};
const options = {method: 'GET', url: 'https://httpbin.org/cookies', jar: jar};
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cookies here just weren't being sent at all. Before:

{
  "cookies": {}
}

After:

{
  "cookies": {
    "bar": "baz", 
    "foo": "bar"
  }
}

@@ -6,14 +6,13 @@ jar.setCookie(request.cookie('bar=baz'), 'https://httpbin.org/anything');

const options = {
method: 'POST',
url: 'https://httpbin.org/anything',
qs: {foo: ['bar', 'baz'], baz: 'abc', key: 'value'},
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Prior to this fix args was coming back from HTTPBin as the following:

  "args": {
    "baz": "abc", 
    "foo[0]": "bar", 
    "foo[1]": "baz", 
    "key": "value"
  }, 

And now:

  "args": {
    "baz": "abc", 
    "foo": [
      "bar", 
      "baz"
    ], 
    "key": "value"
  }, 

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

you might also be able to use URLSearchParams here too, I haven't checked. Also, unrelated, but I noticed the other day that request is VERY deprecated: https://github.com/request/request#deprecated

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Request doesn't support URLSearchParams for query params it seems:

const request = require('request');

const jar = request.jar();
jar.setCookie(request.cookie('foo=bar'), 'https://httpbin.org/anything');
jar.setCookie(request.cookie('bar=baz'), 'https://httpbin.org/anything');

const qs = new URLSearchParams();
qs.append('foo', 'bar');
qs.append('foo', 'baz');
qs.append('baz', 'abc');
qs.append('key', 'value');

const options = {
  method: 'POST',
  url: 'https://httpbin.org/anything',
  qs,
  headers: {
    accept: 'application/json',
    'content-type': 'application/x-www-form-urlencoded'
  },
  form: {foo: 'bar'},
  jar: jar
};

request(options, function (error, response, body) {
  if (error) throw new Error(error);

  console.log(body);
});
{
  "args": {}, 
  "data": "", 
  "files": {}, 
  "form": {
    "foo": "bar"
  }, 
  "headers": {
    "Accept": "application/json", 
    "Content-Length": "7", 
    "Content-Type": "application/x-www-form-urlencoded", 
    "Cookie": "foo=bar; bar=baz", 
    "Host": "httpbin.org", 
    "X-Amzn-Trace-Id": "Root=1-6140e835-7dd0f312641363ff0e30bc4b"
  }, 
  "json": null, 
  "method": "POST", 
  "origin": "REDACTED", 
  "url": "https://httpbin.org/anything"
}

@@ -1,73 +1 @@
module.exports = function (HTTPSnippet, fixtures) {
test('should respond with stringify when useObjectBody is enabled', function () {
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These tests are no longer needed since useObjectBody: true is now the default and only behavior for the node-fetch target.

}

code.push("require_once('vendor/autoload.php');").blank();
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I mostly added this into the snippet because running php path/to/snippet.php in these integration tests was failing because the Composer autoloader wasn't getting picked up and there isn't an easy way to tell PHP where to look for it without dealing a custom php.ini file that I have zero interest in messing around with.

@erunion erunion marked this pull request as ready for review September 14, 2021 02:02
@erunion erunion added the bug Something isn't working label Sep 14, 2021
Copy link

@Dashron Dashron left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks great, and will be a great framework to expand upon

@erunion erunion merged commit bbb7fa2 into main Sep 14, 2021
@erunion erunion deleted the feat/integration-tests branch September 14, 2021 19:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working enhancement New feature or request refactor Issues about tackling technical debt
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants