Hypermodern Python Chapter 2 - Testing Claudio Jolowicz
Hypermodern Python Chapter 2 - Testing Claudio Jolowicz
Claudio Jolowicz
Blog
Music
In this second installment of the Hypermodern Python series, I’m going to discuss how to add automated testing to your project, and how to teach the random fact
generator foreign languages.1 Previously, we discussed How to set up a Python project. (If you start reading here, you can also download the code for the previous
chapter.)
Chapter 1: Setup
Chapter 2: Testing (this article)
Chapter 3: Linting
Chapter 4: Typing
Chapter 5: Documentation
Chapter 6: CI/CD
This guide has a companion repository: cjolowicz/hypermodern-python. Each article in the guide corresponds to a set of commits in the GitHub repository:
View changes
Download code
https://cjolowicz.github.io/posts/hypermodern-python-02-testing/#test-automation-with-nox 1/12
6/1/24, 3:15 PM Hypermodern Python Chapter 2: Testing · Claudio Jolowicz
Unit tests, as the name says, verify the functionality of a unit of code, such as a single function or class. While the unittest framework is part of the Python standard
library, pytest has become somewhat of a de facto standard.
Let’s add this package as a development dependency, using Poetry’s --dev option:
.
├── src
└── tests
├── __init__.py
└── test_console.py
2 directories, 2 files
The file __init__.py is empty and serves to declare the test suite as a package. While this is not strictly necessary, it allows your test suite to mirror the source
layout of the package under test, even when modules in different parts of the source tree have the same name. Furthermore, it gives you the option to import modules
from within your tests package.
The file test_console.py contains a test case for the console module, which checks whether the program exits with a status code of zero.
# tests/test_console.py
import click.testing
def test_main_succeeds():
runner = click.testing.CliRunner()
result = runner.invoke(console.main)
assert result.exit_code == 0
Click’s testing.CliRunner can invoke the command-line interface from within a test case. Since this is likely to be needed by most test cases in this module, let’s
turn it into a test fixture. Test fixtures are simple functions declared with the pytest.fixture decorator. Test cases can use a test fixture by including a function
parameter with the same name as the test fixture.
# tests/test_console.py
import click.testing
import pytest
@pytest.fixture
def runner():
return click.testing.CliRunner()
https://cjolowicz.github.io/posts/hypermodern-python-02-testing/#test-automation-with-nox 2/12
6/1/24, 3:15 PM Hypermodern Python Chapter 2: Testing · Claudio Jolowicz
def test_main_succeeds(runner):
result = runner.invoke(console.main)
assert result.exit_code == 0
tests/test_console.py . [100%]
Code coverage is a measure of the degree to which the source code of your program is executed while running its test suite. The code coverage of Python programs can
be determined using a tool called Coverage.py. Install it with the pytest-cov plugin, which integrates Coverage.py with pytest:
You can configure Coverage.py using the pyproject.toml configuration file, provided it was installed with the toml extra as shown above. Update this file to
inform the tool about your package name and source tree layout. The configuration also enables branch analysis and the display of line numbers for missing coverage:
# pyproject.toml
[tool.coverage.paths]
source = ["src", "*/site-packages"]
[tool.coverage.run]
branch = true
source = ["hypermodern_python"]
[tool.coverage.report]
show_missing = true
tests/test_console.py . [100%]
The reported code coverage is 100%. This number does not imply that your test suite has meaningful test cases for all uses and misuses of your program. Code
coverage only tells you that all lines and branches in your code base were hit. (In fact, our test case achieved full coverage without checking the functionality of the
https://cjolowicz.github.io/posts/hypermodern-python-02-testing/#test-automation-with-nox 3/12
6/1/24, 3:15 PM Hypermodern Python Chapter 2: Testing · Claudio Jolowicz
program at all, only its exit status.)
Nevertheless, aiming for 100% code coverage is good practice, especially for a fresh codebase. Anything less than that implies that some part of your code base is
definitely untested. And to quote Bruce Eckel, “If it’s not tested, it’s broken.” Later, we will see some tools that help you achieve extensive code coverage.
You can configure Coverage.py to require full test coverage (or any other target percentage) using the fail_under option:
# pyproject.toml
[tool.coverage.report]
fail_under = 100
One of my personal favorites, Nox is a successor to the venerable tox. At its core, the tool automates testing in multiple Python environments. Nox makes it easy to run
any kind of job in an isolated environment, with only those dependencies installed that the job needs.
# noxfile.py
import nox
@nox.session(python=["3.8", "3.7"])
def tests(session):
session.run("poetry", "install", external=True)
session.run("pytest", "--cov")
This file defines a session named tests, which installs the project dependencies and runs the test suite. Poetry is not a part of the environment created by Nox, so we
specify external to avoid warnings about external commands leaking into the isolated test environments.
Nox creates virtual environments for the listed Python versions (3.8 and 3.7), and runs the session inside each environment:
$ nox
https://cjolowicz.github.io/posts/hypermodern-python-02-testing/#test-automation-with-nox 4/12
6/1/24, 3:15 PM Hypermodern Python Chapter 2: Testing · Claudio Jolowicz
Nox recreates the virtual environments from scratch on each invocation (a sensible default). You can speed things up by passing the --reuse-existing-virtualenvs (-r)
option:
nox -r
Sometimes, you need to pass additional options to pytest, for example to select specific test cases. Change the session to allow overriding the options passed to
pytest, via the session.posargs variable:
# noxfile.py
import nox
@nox.session(python=["3.8", "3.7"])
def tests(session):
args = session.posargs or ["--cov"]
session.run("poetry", "install", external=True)
session.run("pytest", *args)
Now you can run a specific test module inside the environments:
nox -- tests/test_console.py
Unit tests should be fast, isolated, and repeatable. The test for console.main is neither of these:
It is not fast, because it takes a full round-trip to the Wikipedia API to complete.
It does not run in an isolated environment, because it sends out an actual request over the network.
It is not repeatable, because its outcome depends on the health, reachability, and behavior of the API. In particular, the test fails whenever the network is down.
The unittest.mock standard library allows you to replace parts of your system under test with mock objects. Use it via the pytest-mock plugin, which integrates the
library with pytest:
The plugin provides a mocker fixture, which functions as a thin wrapper around the standard mocking library. Use mocker.patch to replace the requests.get
function by a mock object. The mock object will be useful for any test case involving the Wikipedia API, so let’s create a test fixture for it:
# tests/test_console.py
@pytest.fixture
def mock_requests_get(mocker):
return mocker.patch("requests.get")
If you run Nox now, the test fails because click expects to be passed a string for console output, and receives a mock object instead. Simply “knocking out”
requests.get is not quite enough. The mock object also needs to return something meaningful, namely a response with a valid JSON object.
When a mock object is called, or when an attribute is accessed, it returns another mock object. Sometimes this is sufficient to get you through a test case. When it is not,
you need to configure the mock object. To configure an attribute, you simply set the attribute to the desired value. To configure the return value for when the mock is
called, you set return_value on the mock object as if it were an attribute.
https://cjolowicz.github.io/posts/hypermodern-python-02-testing/#test-automation-with-nox 5/12
6/1/24, 3:15 PM Hypermodern Python Chapter 2: Testing · Claudio Jolowicz
with requests.get(API_URL) as response:
response.raise_for_status()
data = response.json()
The code above uses the response as a context manager. The with statement is syntactic sugar for the following slightly simplified pseudocode:
context = requests.get(API_URL)
response = context.__enter__()
try:
response.raise_for_status()
data = response.json()
finally:
context.__exit__(...)
data = requests.get(API_URL).__enter__().json()
Rewrite the fixture, and mirror this call chain when you configure the mock:
@pytest.fixture
def mock_requests_get(mocker):
mock = mocker.patch("requests.get")
mock.return_value.__enter__.return_value.json.return_value = {
"title": "Lorem Ipsum",
"extract": "Lorem ipsum dolor sit amet",
}
return mock
Mocking not only speeds up your test suite, or lets you hack offline on a plane or train. By virtue of having a fixed, or deterministic, return value, the mock also enables
you to write repeatable tests. This means you can, for example, check that the title returned by the API is printed to the console:
Additionally, mocks can be inspected to see if they were called, using the mock’s called attribute. This provides you with a way to check that requests.get was
invoked to send a request to the API:
# tests/test_console.py
def test_main_invokes_requests_get(runner, mock_requests_get):
runner.invoke(console.main)
assert mock_requests_get.called
Mock objects also allow you to inspect the arguments they were called with, using the call_args attribute. This allows you to check the URL passed to
requests.get:
# tests/test_console.py
def test_main_uses_en_wikipedia_org(runner, mock_requests_get):
runner.invoke(console.main)
args, _ = mock_requests_get.call_args
assert "en.wikipedia.org" in args[0]
You can configure a mock to raise an exception instead of returning a value by assigning the exception instance or class to the side_effect attribute of the mock. Let’s
check that the program exits with a status code of 1 on request errors:
# tests/test_console.py
def test_main_fails_on_request_error(runner, mock_requests_get):
mock_requests_get.side_effect = Exception("Boom")
result = runner.invoke(console.main)
assert result.exit_code == 1
You should generally have a single assertion per test case, because more fine-grained test cases make it easier to figure out why the test suite failed when it does.
Tests for a feature or bugfix should be written before implementation. This is also known as “writing a failing test". The reason for this is that it provides confidence
that the tests are actually testing something, and do not simply pass because of a flaw in the tests themselves.
https://cjolowicz.github.io/posts/hypermodern-python-02-testing/#test-automation-with-nox 6/12
6/1/24, 3:15 PM Hypermodern Python Chapter 2: Testing · Claudio Jolowicz
The great thing about a good test suite is that it allows you to refactor your code without fear of breaking it. Let’s move the Wikipedia client to a separate module.
Create a file src/hypermodern-python/wikipedia.py with the following contents:
# src/hypermodern-python/wikipedia.py
import requests
API_URL = "https://en.wikipedia.org/api/rest_v1/page/random/summary"
def random_page():
with requests.get(API_URL) as response:
response.raise_for_status()
return response.json()
# src/hypermodern-python/console.py
import textwrap
import click
@click.command()
@click.version_option(version=__version__)
def main():
"""The hypermodern Python project."""
data = wikipedia.random_page()
title = data["title"]
extract = data["extract"]
click.secho(title, fg="green")
click.echo(textwrap.fill(extract))
$ nox -r
...
nox > Ran multiple sessions:
nox > * tests-3.8: success
nox > * tests-3.7: success
https://cjolowicz.github.io/posts/hypermodern-python-02-testing/#test-automation-with-nox 7/12
6/1/24, 3:15 PM Hypermodern Python Chapter 2: Testing · Claudio Jolowicz
If you run the example application without an Internet connection, your terminal will be filled with a long traceback. This is what happens when the Python interpreter
is terminated by an unhandled exception. For common errors such as this, it would be better to print a friendly, informative message to the screen.
Let’s express this as a test case, by configuring the mock to raise a RequestException. (The requests library has more specific exception classes, but for the
purposes of this example, we will only deal with the base class.)
# tests/test_console.py
import requests
The simplest way to get this test to pass is by converting the RequestException into a ClickException. When click encounters this exception, it prints the
exception message to standard error and exits the program with a status code of 1. You can reuse the exception message by converting the original exception to a string.
# src/hypermodern-python/wikipedia.py
import click
import requests
API_URL = "https://en.wikipedia.org/api/rest_v1/page/random/summary"
def random_page():
try:
with requests.get(API_URL) as response:
response.raise_for_status()
return response.json()
except requests.RequestException as error:
message = str(error)
raise click.ClickException(message)
https://cjolowicz.github.io/posts/hypermodern-python-02-testing/#test-automation-with-nox 8/12
6/1/24, 3:15 PM Hypermodern Python Chapter 2: Testing · Claudio Jolowicz
In this section, we add a command-line option to select the language edition of Wikipedia.
Wikipedia editions are identified by a language code, which is used as a subdomain below wikipedia.org. Usually, this is the two-letter or three-letter language code
assigned to the language by ISO 639-1 and ISO 639-3. Here are some examples:
As a first step, let’s add an optional parameter for the language code to the wikipedia.random_page function. When an alternate language is passed, the API
request should be sent to the corresponding Wikipedia edition. The test case is placed in a new test module named test_wikipedia.py:
# tests/test_wikipedia.py
from hypermodern_python import wikipedia
def test_random_page_uses_given_language(mock_requests_get):
wikipedia.random_page(language="de")
args, _ = mock_requests_get.call_args
assert "de.wikipedia.org" in args[0]
The mock_requests_get fixture is now used by two test modules. You could move it to a separate module and import from there, but Pytest offers a more
convenient way: Fixtures placed in a conftest.py file are discovered automatically, and test modules at the same directory level can use them without explicit
import. Create the new file at the top-level of your tests package, and move the fixture there:
# tests/conftest.py
import pytest
@pytest.fixture
def mock_requests_get(mocker):
mock = mocker.patch("requests.get")
mock.return_value.__enter__.return_value.json.return_value = {
"title": "Lorem Ipsum",
"extract": "Lorem ipsum dolor sit amet",
}
return mock
To get the test to pass, we turn API_URL into a format string, and interpolate the specified language code into the URL using str.format:
# src/hypermodern-python/wikipedia.py
import click
import requests
https://cjolowicz.github.io/posts/hypermodern-python-02-testing/#test-automation-with-nox 9/12
6/1/24, 3:15 PM Hypermodern Python Chapter 2: Testing · Claudio Jolowicz
API_URL = "https://{language}.wikipedia.org/api/rest_v1/page/random/summary"
def random_page(language="en"):
url = API_URL.format(language=language)
try:
with requests.get(url) as response:
response.raise_for_status()
return response.json()
except requests.RequestException as error:
message = str(error)
raise click.ClickException(message)
As the second step, we make the new functionality accessible from the command line, adding a --language option. The test case mocks the
wikipedia.random_page function, and uses the assert_called_with method on the mock to check that the language specified by the user is passed on to the
function:
# tests/test_console.py
@pytest.fixture
def mock_wikipedia_random_page(mocker):
return mocker.patch("hypermodern_python.wikipedia.random_page")
We are now ready to implement the new functionality using the click.option decorator. Without further ado, here is the final version of the console module:
# src/hypermodern-python/console.py
import textwrap
import click
@click.command()
@click.option(
"--language",
"-l",
default="en",
help="Language edition of Wikipedia",
metavar="LANG",
show_default=True,
)
@click.version_option(version=__version__)
def main(language):
"""The hypermodern Python project."""
data = wikipedia.random_page(language=language)
title = data["title"]
extract = data["extract"]
click.secho(title, fg="green")
click.echo(textwrap.fill(extract))
You now have a polyglot random fact generator, and a fun way to test your language skills (and the Unicode skills of your terminal emulator).
Using fakes
https://cjolowicz.github.io/posts/hypermodern-python-02-testing/#test-automation-with-nox 10/12
6/1/24, 3:15 PM Hypermodern Python Chapter 2: Testing · Claudio Jolowicz
Mocks help you test code units depending on bulky subsystems, but they are not the only technique to do so. For example, if your function requires a database
connection, it may be both easier and more effective to pass an in-memory database than a mock object. Fake implementations are a good alternative to mock objects,
which can be too forgiving when faced with wrong usage, and too tightly coupled to implementation details of the system under test (witness the
mock_requests_get fixture). Large data objects can be generated by test object factories, instead of being replaced by mock objects (check out the excellent
factoryboy package).
Implementing a fake API is out of the scope of this tutorial, but we will cover one aspect of it: How to write a fixture which requires tear down code as well as set up
code. Suppose you have written the following fake API implementation:
class FakeAPI:
url = "http://localhost:5000/"
@classmethod
def create(cls):
...
def shutdown(self):
...
@pytest.fixture
def fake_api():
return FakeAPI.create()
The API needs to be shut down after use, to free up resources such as the TCP port and the thread running the server. You can do this by writing the fixture as a
generator:
@pytest.fixture
def fake_api():
api = FakeAPI.create()
yield api
api.shutdown()
Pytest takes care of running the generator, passing the yielded value to your test function, and executing the shutdown code after it returns. If setting up and tearing
down the fixture is expensive, you may also consider extending the scope of the fixture. By default, fixtures are created once per test function. Instead, you could create
the fake API server once per test session:
@pytest.fixture(scope="session")
def fake_api():
api = FakeAPI.create()
yield api
api.shutdown()
End-to-end testing
https://cjolowicz.github.io/posts/hypermodern-python-02-testing/#test-automation-with-nox 11/12
6/1/24, 3:15 PM Hypermodern Python Chapter 2: Testing · Claudio Jolowicz
Testing against the live production server is bad practice for unit tests, but there is nothing like the confidence you get from seeing your code work in a real
environment. Such tests are known as end-to-end tests, and while they are usually too slow, brittle, and unpredictable for the kind of automated testing you would want
to do on a CI server or in the midst of development, they do have their place.
Let’s reinstate the original test case, and use Pytest’s markers to apply a custom mark. This will allow you to select or skip them later, using Pytest’s -m option.
# tests/test_console.py
@pytest.mark.e2e
def test_main_succeeds_in_production_env(runner):
result = runner.invoke(console.main)
assert result.exit_code == 0
Register the e2e marker using the pytest_configure hook, as shown below. The hook is placed in the conftest.py file, at the top-level of your tests package.
This ensures that Pytest can discover the module and use it for the entire test suite.
# tests/conftest.py
def pytest_configure(config):
config.addinivalue_line("markers", "e2e: mark as end-to-end test.")
Finally, exclude end-to-end tests from automated testing by passing -m "not e2e" to Pytest:
# noxfile.py
import nox
@nox.session(python=["3.8", "3.7"])
def tests(session):
args = session.posargs or ["--cov", "-m", "not e2e"]
session.run("poetry", "install", external=True)
session.run("pytest", *args)
You can now run end-to-end tests by passing -m e2e to the Nox session, using a double dash (--) to separate them from Nox’s own options. For example, here’s how
you would run end-to-end tests inside the testing environment for Python 3.8:
1. The images in this chapter come from Émile-Antoine Bayard’s Illustrations for From the Earth to the Moon (De la terre à la lune) by Jules Verne (1870) (source:
Internet Archive via The Public Domain Review) ↩︎
https://cjolowicz.github.io/posts/hypermodern-python-02-testing/#test-automation-with-nox 12/12