Thanks to visit codestin.com
Credit goes to github.com

Skip to content

Commit f957a39

Browse files
JukkaLJukkaL
authored
Add scripts for running mypy and mypyc tests inside a Docker container (#14454)
This speeds up running `pytest mypyc/test/test_run.py` by over 3x on a Mac mini M1. Mypy test performance is fairly similar to running natively. Example of how to use these on macOS: ``` $ colima start -c 8 # Start VM $ python3 misc/docker/build.py # Build Ubuntu container with test dependencies $ misc/docker/run.sh pytest mypyc/test/test_run.py # Run tests in container ``` Also add a README that documents how to use these. Closes #14453.
1 parent e88f4a4 commit f957a39

5 files changed

Lines changed: 187 additions & 0 deletions

File tree

‎misc/docker/Dockerfile‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,12 @@
1+
FROM ubuntu:latest
2+
3+
WORKDIR /mypy
4+
5+
RUN apt-get update
6+
RUN apt-get install -y python3 python3-pip clang
7+
8+
COPY mypy-requirements.txt .
9+
COPY test-requirements.txt .
10+
COPY build-requirements.txt .
11+
12+
RUN pip3 install -r test-requirements.txt

‎misc/docker/README.md‎

Lines changed: 101 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,101 @@
1+
Running mypy and mypyc tests in a Docker container
2+
==================================================
3+
4+
This directory contains scripts for running mypy and mypyc tests in a
5+
Linux Docker container. This allows running Linux tests on a different
6+
operating system that supports Docker, or running tests in an
7+
isolated, predictable environment on a Linux host operating system.
8+
9+
Why use Docker?
10+
---------------
11+
12+
Mypyc tests can be significantly faster in a Docker container than
13+
running natively on macOS.
14+
15+
Also, if it's inconvient to install the necessary dependencies on the
16+
host operating system, or there are issues getting some tests to pass
17+
on the host operating system, using a container can be an easy
18+
workaround.
19+
20+
Prerequisites
21+
-------------
22+
23+
First install Docker. On macOS, both Docker Desktop (proprietary, but
24+
with a free of charge subscription for some use cases) and Colima (MIT
25+
license) should work as runtimes.
26+
27+
You may have to explicitly start the runtime first. Colima example
28+
(replace '8' with the number of CPU cores you have):
29+
30+
```
31+
$ colima start -c 8
32+
33+
```
34+
35+
How to run tests
36+
----------------
37+
38+
You need to build the container with all necessary dependencies before
39+
you can run tests:
40+
41+
```
42+
$ python3 misc/docker/build.py
43+
```
44+
45+
This creates a `mypy-test` Docker container that you can use to run
46+
tests.
47+
48+
You may need to run the script as root:
49+
50+
```
51+
$ sudo python3 misc/docker/build.py
52+
```
53+
54+
If you have a stale container which isn't up-to-date, use `--no-cache`
55+
`--pull` to force rebuilding everything:
56+
57+
```
58+
$ python3 misc/docker/build.py --no-cache --pull
59+
```
60+
61+
Now you can run tests by using the `misc/docker/run.sh` script. Give
62+
it the pytest command line you want to run as arguments. For example,
63+
you can run mypyc tests like this:
64+
65+
```
66+
$ misc/docker/run.sh pytest mypyc
67+
```
68+
69+
You can also use `-k <filter>`, `-n0`, `-q`, etc.
70+
71+
Again, you may need to run `run.sh` as root:
72+
73+
```
74+
$ sudo misc/docker/run.sh pytest mypyc
75+
```
76+
77+
You can also use `runtests.py` in the container. Example:
78+
79+
```
80+
$ misc/docker/run.sh ./runtests.py self lint
81+
```
82+
83+
Notes
84+
-----
85+
86+
File system changes within the container are not visible to the host
87+
system. You can't use the container to format code using Black, for
88+
example.
89+
90+
On a mac, you may want to give additional CPU to the VM used to run
91+
the container. The default allocation may be way too low (e.g. 2 CPU
92+
cores). For example, use the `-c` option when starting the VM if you
93+
use Colima:
94+
95+
```
96+
$ colima start -c 8
97+
```
98+
99+
Giving access to all available CPUs to the Linux VM tends to provide
100+
the best performance. This is not needed on a Linux host, since the
101+
container is not run in a VM.

‎misc/docker/build.py‎

Lines changed: 46 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,46 @@
1+
"""Build a "mypy-test" Linux Docker container for running mypy/mypyc tests.
2+
3+
This allows running Linux tests under a non-Linux operating system. Mypyc
4+
tests can also run much faster under Linux that the host OS.
5+
6+
NOTE: You may need to run this as root (using sudo).
7+
8+
Run with "--no-cache" to force reinstallation of mypy dependencies.
9+
Run with "--pull" to force update of the Linux (Ubuntu) base image.
10+
11+
After you've built the container, use "run.sh" to run tests. Example:
12+
13+
misc/docker/run.sh pytest mypyc/
14+
"""
15+
16+
import argparse
17+
import os
18+
import subprocess
19+
import sys
20+
21+
22+
def main() -> None:
23+
parser = argparse.ArgumentParser(
24+
description="""Build a 'mypy-test' Docker container for running mypy/mypyc tests. You may
25+
need to run this as root (using sudo)."""
26+
)
27+
parser.add_argument("--no-cache", action="store_true", help="Force rebuilding")
28+
parser.add_argument("--pull", action="store_true", help="Force pulling fresh Linux base image")
29+
args = parser.parse_args()
30+
31+
dockerdir = os.path.dirname(os.path.abspath(__file__))
32+
dockerfile = os.path.join(dockerdir, "Dockerfile")
33+
rootdir = os.path.join(dockerdir, "..", "..")
34+
35+
cmdline = ["docker", "build", "-t", "mypy-test", "-f", dockerfile]
36+
if args.no_cache:
37+
cmdline.append("--no-cache")
38+
if args.pull:
39+
cmdline.append("--pull")
40+
cmdline.append(rootdir)
41+
result = subprocess.run(cmdline)
42+
sys.exit(result.returncode)
43+
44+
45+
if __name__ == "__main__":
46+
main()

‎misc/docker/run-wrapper.sh‎

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,13 @@
1+
#!/bin/bash
2+
# Internal wrapper script used to run commands in a container
3+
4+
# Copy all the files we need from the mypy repo directory shared with
5+
# the host to a local directory. Accessing files using a shared
6+
# directory on a mac can be *very* slow.
7+
echo "copying files to the container..."
8+
cp -R /repo/{mypy,mypyc,test-data,misc} .
9+
cp /repo/{pytest.ini,conftest.py,runtests.py,pyproject.toml,setup.cfg} .
10+
cp /repo/{mypy_self_check.ini,mypy_bootstrap.ini} .
11+
12+
# Run the wrapped command
13+
"$@"

‎misc/docker/run.sh‎

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,15 @@
1+
#!/bin/bash
2+
# Run mypy or mypyc tests in a Docker container that was built using misc/docker/build.py.
3+
#
4+
# Usage: misc/docker/run.sh <command> <arg>...
5+
#
6+
# For example, run mypyc tests like this:
7+
#
8+
# misc/docker/run.sh pytest mypyc
9+
#
10+
# NOTE: You may need to run this as root (using sudo).
11+
12+
SCRIPT_DIR=$(cd -- "$( dirname -- "${BASH_SOURCE[0]}" )" &> /dev/null && pwd)
13+
MYPY_DIR="$SCRIPT_DIR/../.."
14+
15+
docker run -ti --rm -v "$MYPY_DIR:/repo" mypy-test /repo/misc/docker/run-wrapper.sh "$@"

0 commit comments

Comments
 (0)