I'd like to suggest a goal for the project: automated tests of the vendor badges.

As Thaddée mentioned in #411:

Testing is of course imperfect (as in, manual). The CLI and backend engine have tests, but the server does not. What would be helpful however isn't tests here; it would be automated error reporting. We know that a list of badges should work. Having a program that goes through them and reports what the vendor server sent back if the badge failed would be great. However, it should absolutely not complexify the existing logic of the server.

I agree that tests like these would be valuable for end users. They would detect API problems when they occur, rather than hoping/waiting someone will eventually show up and report them.

They'd be relatively simple to write. They would help badge developers , and push more of the testing responsibility from the reviewer to the contributor.

In PR reviews, I spend a significant amount of time manually testing various combinations of inputs. After requesting changes I really should start all over again. Tests would make this faster and much more reliable.

I'm skilled enough to spot many common problems in this type of code, however unlike Thaddée I'm not an expert in this codebase. I don't have the eye to recognize all the problematic patterns that he would. I can't tell the code is obviously right, or obviously wrong code, just by looking at it.

Given how easy it is to write vendor badges, and what's sure to be endless growth of available repositories and tools, PR review will remain a bottleneck. With automatic tests, testing responsibility can be scaled out to a wide net of contributors. Writing tests for existing vendors is another easy way for contributors to put love into the project, even those who are content with the existing feature set.

Some downsides:

1. The tests would be slow, due to slow external servers. We'd want to easily run a subset.
2. The tests would be somewhat flaky due to flaky external servers. When running the tests, we'd want to include some amount of retrying.
3. Since assertions depend on external servers and external data, they would need to be somewhat flexible. A valid regex for semver, e.g., or for a build which may be passing, failing, or errored. We risk making the assertions so complex that they allow false negatives: assertions which pass when they should fail.
4. They would cover something pretty fundamental that can’t be tested any other way: changes to a vendor's API.

There are a few important attributes of tests like these:

• Easy to read and write
• Simple assertions – no false negatives
• Retriable – so it’s possible to run automatically
• Non-invasive – they should not add complexity to the server

We could call these integration tests.

There’s another, more ambitious kind of test I'd also like to see: tests which record vendor responses and play them back later. They would inject mock vendor responses to handle cases like malformed requests, which can’t be tested any other way. Fundamentally, they could test that the server code is doing exactly what it is supposed to, regardless of whether a package still exists or a vendor server is temporarily down.

Important attributes:

• Easy to debug – simple flow control, minimal magic, good error messages
• Blazingly fast – whole suite runs in a few seconds at most

We could run the whole suite of tests on every PR. That ensures non-breakage if other features due to a change, speeds up reviews, and lets us merge with confidence.

Running the whole suite is especially useful for refactoring: developing other, potentially significant changes to the implementation which should not affect most of the server’s behavior.

The tool I've been experimenting with is Nock Back. Working tests are reliable and it's pretty good overall. However it doesn't have the most active maintenance, I've run into some bugs (nock/nock#870), and I've found failing tests to be tricky to debug (nock/nock#781, nock/nock#869).

We could call these recorded tests.

Checking code coverage is important not just for improving the code quality, but also the quality of the service. In my experience writing just a few of these, I found behaviors which should be improved, code paths that do not make any callback, and other errors in error-handling code.

However, recording responses is a really big bite to take, especially with some uncertainty about the current state of the tooling. So I'd suggest we leave recording for another day.

For now, the goals should be the following:

• Badge developers can easily test their work
• They can commit those tests so reviewers can see them
• Reviewers can run a subset of the tests (or CI can infer the affected services and do this automatically)
• Developers can TDD bug fixes
• Full suite runs periodically and/or on pushes to master, and we find out when badges have broken

This would include:

1. Tests using real vendor servers (true integration tests)
2. Tests of error handling using mock responses (functional tests)
3. Optional grouping by vendor
4. Retrying transient failures
5. Maybe down the line…
   a. Automatically inferring which vendors to run
   b. Reporting code coverage

Thoughts? Concerns?

And importantly, would anyone else like to join in the fun? 😀