-
Notifications
You must be signed in to change notification settings - Fork 6
feat: integration test suite #51
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
@@ -52,7 +52,6 @@ module.exports = { | |||
hello: 'world', | |||
}, | |||
headers: { | |||
Accept: '*/*', |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Removed all these Accept: */*
headers from the response configs because I originally built these response configs from data coming back from a cURL request and cURL adds that Accept
header to the request (and response since HTTPBin shows incoming headers here) where other targets don't.
…uble encoding issues
method: 'POST', | ||
url: 'https://httpbin.org/anything', | ||
headers: {'content-type': 'application/x-www-form-urlencoded'}, | ||
data: {foo: 'bar', hello: 'world'} | ||
data: encodedParams |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Made this change because this data was getting to HTTPBin actually as the following:
{
args: {},
data: '',
files: {},
form: { '{"foo":"bar","hello":"world"}': '' },
headers: {
Accept: 'application/json, text/plain, */*',
'Content-Length': '29',
'Content-Type': 'application/x-www-form-urlencoded',
Host: 'httpbin.org',
'User-Agent': 'axios/0.21.4',
'X-Amzn-Trace-Id': 'Root=1-613fc81c-33b383d80f53e0750840173b'
},
json: null,
method: 'POST',
origin: 'REDACTED',
url: 'https://httpbin.org/anything'
}
With this fix:
{
args: {},
data: '',
files: {},
form: { foo: 'bar', hello: 'world' },
headers: {
Accept: 'application/json, text/plain, */*',
'Content-Length': '19',
'Content-Type': 'application/x-www-form-urlencoded',
Host: 'httpbin.org',
'User-Agent': 'axios/0.21.4',
'X-Amzn-Trace-Id': 'Root=1-613fc852-1b54e94f1b77ddac478d20a4'
},
json: null,
method: 'POST',
origin: 'REDACTED',
url: 'https://httpbin.org/anything'
}
method: 'POST', | ||
url: 'https://httpbin.org/anything', | ||
params: {foo: ['bar', 'baz'], baz: 'abc', key: 'value'}, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Made this change because query params were getting to HTTPBin as the following:
args: { baz: 'abc', 'foo[]': [ 'bar', 'baz' ], key: 'value' },
With this fix:
args: { baz: 'abc', foo: [ 'bar', 'baz' ], key: 'value' },
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
interesting, so foo[]
is an old way people used to identify that multiple identical parameters should be treated as an array. I haven't seen it for quite some time though. In fact it doesn't really even seem to be supported as an OAS style.
I'm going to poke around axios a bit and see why they're still using this pattern by default.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks like you can use URLSearchParams
: https://stackoverflow.com/questions/42898009/multiple-fields-with-same-key-in-query-params-axios-request
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
To get this working right with URLSearchParams
we'll need to run through arrays and add them in separately:
const params = new URLSearchParams();
params.append('foo', 'bar');
params.append('foo', 'baz');
params.append('baz', 'abc');
params.append('key', 'value');
I'm going to leave it with having query params in the URL because it'll be less noisy.
@@ -4,7 +4,14 @@ const url = 'https://httpbin.org/anything'; | |||
const options = { | |||
method: 'POST', | |||
headers: {'content-type': 'application/json'}, | |||
body: '{"number":1,"string":"f\"oo","arr":[1,2,3],"nested":{"a":"b"},"arr_mix":[1,"a",{"arr_mix_nested":{}}],"boolean":false}' |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Prior to this fix, see json
in this returned payload:
{
args: {},
data: '{"number":1,"string":"f"oo","arr":[1,2,3],"nested":{"a":"b"},"arr_mix":[1,"a",{"arr_mix_nested":{}}],"boolean":false}',
files: {},
form: {},
headers: {
Accept: '*/*',
'Accept-Encoding': 'gzip,deflate',
'Content-Length': '117',
'Content-Type': 'application/json',
Host: 'httpbin.org',
'User-Agent': 'node-fetch/1.0 (+https://github.com/bitinn/node-fetch)',
'X-Amzn-Trace-Id': 'Root=1-613fc993-7f43a7fc522f744c69c69531'
},
json: null,
method: 'POST',
origin: 'REDACTED',
url: 'https://httpbin.org/anything'
}
And now:
{
args: {},
data: '{"number":1,"string":"f\\"oo","arr":[1,2,3],"nested":{"a":"b"},"arr_mix":[1,"a",{"arr_mix_nested":{}}],"boolean":false}',
files: {},
form: {},
headers: {
Accept: '*/*',
'Accept-Encoding': 'gzip,deflate',
'Content-Length': '118',
'Content-Type': 'application/json',
Host: 'httpbin.org',
'User-Agent': 'node-fetch/1.0 (+https://github.com/bitinn/node-fetch)',
'X-Amzn-Trace-Id': 'Root=1-613fc98a-181e57cc5b875d915032bfc7'
},
json: {
arr: [ 1, 2, 3 ],
arr_mix: [ 1, 'a', [Object] ],
boolean: false,
nested: { a: 'b' },
number: 1,
string: 'f"oo'
},
method: 'POST',
origin: 'REDACTED',
url: 'https://httpbin.org/anything'
}
Note: This is because of promoting the useObjectBody
option to be the default behavior. We already ran this library with this option turned on in @readme/oas-to-snippet so there's going to be no change to our customers.
Also ignore the [Object]
in there, that's from console.log
's max depth in the snippet not a side effect of this work.
@@ -6,10 +6,7 @@ const formData = new FormData(); | |||
formData.append('foo', fs.createReadStream('__tests__/__fixtures__/files/hello.txt')); | |||
|
|||
const url = 'https://httpbin.org/anything'; | |||
const options = { | |||
method: 'POST', | |||
headers: {'content-type': 'multipart/form-data; boundary=---011000010111000001101001'} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Prior to this fix check files
in this returned payload:
{
args: {},
data: '',
files: {},
form: {},
headers: {
Accept: '*/*',
'Accept-Encoding': 'gzip,deflate',
'Content-Type': 'multipart/form-data; boundary=---011000010111000001101001',
Host: 'httpbin.org',
'Transfer-Encoding': 'chunked',
'User-Agent': 'node-fetch/1.0 (+https://github.com/bitinn/node-fetch)',
'X-Amzn-Trace-Id': 'Root=1-613fca42-5d12267c4f32fd05379c5114'
},
json: null,
method: 'POST',
origin: 'REDACTED',
url: 'https://httpbin.org/anything'
}
And now:
{
args: {},
data: '',
files: { foo: 'Hello World\n' },
form: {},
headers: {
Accept: '*/*',
'Accept-Encoding': 'gzip,deflate',
'Content-Type': 'multipart/form-data;boundary=--------------------------939643642041348712808456',
Host: 'httpbin.org',
'Transfer-Encoding': 'chunked',
'User-Agent': 'node-fetch/1.0 (+https://github.com/bitinn/node-fetch)',
'X-Amzn-Trace-Id': 'Root=1-613fca3a-2893e2fc3bceaa1e7d42a3d2'
},
json: null,
method: 'POST',
origin: 'REDACTED',
url: 'https://httpbin.org/anything'
}
@@ -4,7 +4,7 @@ const jar = request.jar(); | |||
jar.setCookie(request.cookie('foo=bar'), 'https://httpbin.org/cookies'); | |||
jar.setCookie(request.cookie('bar=baz'), 'https://httpbin.org/cookies'); | |||
|
|||
const options = {method: 'GET', url: 'https://httpbin.org/cookies', jar: 'JAR'}; | |||
const options = {method: 'GET', url: 'https://httpbin.org/cookies', jar: jar}; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Cookies here just weren't being sent at all. Before:
{
"cookies": {}
}
After:
{
"cookies": {
"bar": "baz",
"foo": "bar"
}
}
@@ -6,14 +6,13 @@ jar.setCookie(request.cookie('bar=baz'), 'https://httpbin.org/anything'); | |||
|
|||
const options = { | |||
method: 'POST', | |||
url: 'https://httpbin.org/anything', | |||
qs: {foo: ['bar', 'baz'], baz: 'abc', key: 'value'}, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Prior to this fix args
was coming back from HTTPBin as the following:
"args": {
"baz": "abc",
"foo[0]": "bar",
"foo[1]": "baz",
"key": "value"
},
And now:
"args": {
"baz": "abc",
"foo": [
"bar",
"baz"
],
"key": "value"
},
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
you might also be able to use URLSearchParams
here too, I haven't checked. Also, unrelated, but I noticed the other day that request is VERY deprecated: https://github.com/request/request#deprecated
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Request doesn't support URLSearchParams
for query params it seems:
const request = require('request');
const jar = request.jar();
jar.setCookie(request.cookie('foo=bar'), 'https://httpbin.org/anything');
jar.setCookie(request.cookie('bar=baz'), 'https://httpbin.org/anything');
const qs = new URLSearchParams();
qs.append('foo', 'bar');
qs.append('foo', 'baz');
qs.append('baz', 'abc');
qs.append('key', 'value');
const options = {
method: 'POST',
url: 'https://httpbin.org/anything',
qs,
headers: {
accept: 'application/json',
'content-type': 'application/x-www-form-urlencoded'
},
form: {foo: 'bar'},
jar: jar
};
request(options, function (error, response, body) {
if (error) throw new Error(error);
console.log(body);
});
{
"args": {},
"data": "",
"files": {},
"form": {
"foo": "bar"
},
"headers": {
"Accept": "application/json",
"Content-Length": "7",
"Content-Type": "application/x-www-form-urlencoded",
"Cookie": "foo=bar; bar=baz",
"Host": "httpbin.org",
"X-Amzn-Trace-Id": "Root=1-6140e835-7dd0f312641363ff0e30bc4b"
},
"json": null,
"method": "POST",
"origin": "REDACTED",
"url": "https://httpbin.org/anything"
}
@@ -1,73 +1 @@ | |||
module.exports = function (HTTPSnippet, fixtures) { | |||
test('should respond with stringify when useObjectBody is enabled', function () { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
These tests are no longer needed since useObjectBody: true
is now the default and only behavior for the node-fetch
target.
} | ||
|
||
code.push("require_once('vendor/autoload.php');").blank(); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I mostly added this into the snippet because running php path/to/snippet.php
in these integration tests was failing because the Composer autoloader wasn't getting picked up and there isn't an easy way to tell PHP where to look for it without dealing a custom php.ini
file that I have zero interest in messing around with.
…lying query params
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
looks great, and will be a great framework to expand upon
This completely overhauls the existing integration suite that HTTPSnippet has in place in requests.js for running generated snippets for a handful of languages to determine they're sending the data that they should be.
🏚 Why doesn't this current iteration work for us?
native
, PHPcurl
and Pythonpython3
. And of those 3 client targets, only 9 of the 17 total request fixtures that we have tests in place are being executed.We can do better.
🏥 So what the hell did I do here?
Over the past year while we've been building up the reference redesign I've found that https://httpbin.org has been absolutely indispensable for debugging all kinds of API request quirks because it returns your data back in ways that are easy to quickly parse and test against.
For example:
Here it's really easy to see how my query parameters were parsed,
x-www-form-urlencoded
data was sent asform
data, and cookies were sent as a properCookie
header.Now while this same information is also available with mockbin, I don't need to parse a HAR (again) to check
request.postData.mimeType
forapplication/x-www-form-urlencoded
and then see ifrequest.post.params
has my data. This easy response from HTTPBin also makes it really easy for us to flesh out the HAR fixtures that this library builds tests around to have full response information in them. This full HAR can now also be used to quickly do assertions.So with that I've completely rewritten the existing requests.js into a new full integration suite that will now let us run a handful of tests locally when running
npm test
but also a full blown Docker suite to run clients and targets that we can easily containerize.💁♂️ How does it work?
In #49 I updated all of the existing HAR fixtures to store data in the
responses
section that's the data that we get back from an https://httpbin.org/anything, https://httpbin.org/headers, or https://httpbin.org/cookies request.The integration suite now picks up existing snippets from the test suite (stored in
__tests__/__fixtures/output/:client/:target/:snippettest
) and runs those depending on a new config in the client informing us of how they should be run (Node scripts are run withnode
, PHP withphp
, Shell scripts just run as themselves, etc).These snippets hit one of the available endpoints on https://httpbin.org for its test and then we check to see if the response matches what we have already stored in the HAR fixture. If it doesn't match, the integration test fails.
That's it! :easy:
🏄 tl;dr
http
test tohttp-insecure
because Python snippets were attempting to load thehttp.py
test instead of thehttp
module.🧰 Additional Fixes
Included within this work are also a heap of fixes for some client targets that I discovered while running these tests, they include:
node
+fetch
useObjectBody
option that we have fornode-fetch
to be the default behavior. Within the example fixture forapplication-json
we have a"f\"oo"
string that was getting sent to the server as"f"oo"
causing HTTPBin to not see it as JSON.Content-Type
header if the payload ismultipart/form-data
because we push that data into an instance of theform-data
module and that takes care of theContent-Type
header automatically. The custom header that was getting set there was causing data from these requests to get to the server at all.node
+request
request.jar()
but then sending that cookie har as'jar'
instead of the variable -- resulting in cookies never being sent.qs
option, instead relying on sending them as part of the URL. Made this change because query parameters were sometimes either being double encoded, or array params weren't being sent as a proper array.node
+axios
var
instances over toconst
.x-www-form-urlencoded
requests params are now being sent to Axios within an instance ofURLSearchParams
.params
object because of encoding issues.python
+requests
params
option as encoded values were being encoded again with its default form encoding. We're instead supplying query params in the URL.🧬 QA + Testing
Clients and targets that are now being tested within this new suite:
axios
http
(aka "native")node-fetch
request
curl
guzzle
python3
request
curl
How do I run the integration tests?
There's two forms of the integration test suite (
__tests__/integration.test.js
):npm test
)Running the local instance you can do with either running
npm test
ornpx jest __test__/integration.test.js
and that'll run integration tests for Node, PHP, and Python across a single target for each (http
for Node,curl
for PHP, andpython3
for Python).The containerized suite is where the full boar of tests are happening because we want to contain all of the necessary dependencies for each of these clients within that container (Composer, Guzzle,
requests
, all of the NPM packages for each target, etc.). Running this suite is as easy as running the following:This'll build out the container and its dependencies and then execute
npx jest __test__/integration.test.js
with a few environment variables to tell the suite to only run tests for PHP.