Touchups to tests/REGRESSION.md (#12826)

AlexWaygood · web-flow · commit c9d3ca6f489a · 2024-10-17T00:43:39.000+02:00
diff --git a/tests/REGRESSION.md b/tests/REGRESSION.md
@@ -1,10 +1,11 @@
 ## Regression tests for typeshed
 
 Regression tests for the standard library stubs can be found in the
-`stdlib/@tests/test_cases` directory. Stubs for third-party libraries that do
-have test cases can be found in `@tests/test_cases` subdirectories for each
-stubs package. For example, the test cases for `requests` can be found in the
-`stubs/requests/@tests/test_cases` directory.
+`stdlib/@tests/test_cases` directory. Not all third-party-library stub
+packages in typeshed have test cases, and not all of them need test cases --
+but for those that do, their test cases can be found in `@tests/test_cases`
+subdirectories for each stubs package. For example, the test cases for
+`requests` can be found in the `stubs/requests/@tests/test_cases` directory.
 
 **Regression test cases should only be written for functions and classes which
 are known to have caused problems in the past, where the stubs are difficult to
@@ -14,34 +15,45 @@ multiple other mechanisms for spotting errors in the stubs.
 
 ### The purpose of these tests
 
-Different test cases in this directory serve different purposes. For some stubs in
-typeshed, the type annotations are complex enough that it's useful to have
+Different test cases in typeshed serve different purposes. For some typeshed
+stubs, the type annotations are complex enough that it's useful to have
 sanity checks that test whether a type checker understands the intent of
 the annotations correctly. Examples of tests like these are
-`builtins/check_pow.py` and `asyncio/check_gather.py`.
+`stdlib/@tests/test_cases/builtins/check_pow.py` and
+`stdlib/@tests/test_cases/asyncio/check_gather.py`.
 
-Other test cases, such as the samples for `ExitStack` in `check_contextlib.py`
-and the samples for `LogRecord` in `check_logging.py`, do not relate to
+Other test cases, such as the samples for `ExitStack` in
+`stdlib/@tests/test_cases/check_contextlib.py` and the samples for `LogRecord`
+in `stdlib/@tests/test_cases/check_logging.py`, do not relate to
 stubs where the annotations are particularly complex, but they *do* relate to
 stubs where decisions have been taken that might be slightly unusual. These
 test cases serve a different purpose: to check that type checkers do not emit
 false-positive errors for idiomatic usage of these classes.
 
 ## Running the tests
 
-To verify the test cases in this directory pass with mypy, run `python tests/regr_test.py stdlib`
-from the root of the typeshed repository. This assumes that the development
-environment has been set up as described in the [CONTRIBUTING.md](../CONTRIBUTING.md)
-document.
+To verify the stdlib test cases pass with mypy, run
+`python tests/regr_test.py stdlib` from the root of the typeshed repository.
+This assumes that the development environment has been set up as described in
+the [CONTRIBUTING.md](../CONTRIBUTING.md) document.
+
+For third-party-library stubs, pass the name of the runtime library the stubs
+are for. For example, to run the tests for our `requests` stubs, run
+`python tests/regr_test.py requests`.
+
+Run `python tests/regr_test.py -h` for the full range of CLI options this script
+supports. There is no equivalent script for pyright currently; for pyright, the
+tests are checked in CI using a GitHub Action.
 
 ### How the tests work
 
-The code in this directory is not intended to be directly executed. Instead,
-type checkers are run on the code, to check that typing errors are
-emitted at the correct places.
+The code in typeshed's test cases is not intended to be directly executed.
+Instead, type checkers are run on the code, to check that type checker
+diagnostics are emitted at the correct locations.
 
-Some files in this directory simply contain samples of idiomatic Python, which
-should not (if the stubs are correct) cause a type checker to emit any errors.
+Some files in these directories simply contain samples of idiomatic Python,
+which should not (if the stubs are correct) cause a type checker to emit any
+diagnostics.
 
 Many test cases also make use of
 [`assert_type`](https://docs.python.org/3.11/library/typing.html#typing.assert_type),
@@ -54,8 +66,8 @@ mypy's
 setting and pyright's
 [`reportUnnecessaryTypeIgnoreComment`](https://github.com/microsoft/pyright/blob/main/docs/configuration.md#type-check-diagnostics-settings)
 setting) to test instances where a type checker *should* emit some kind of
-error, if the stubs are correct. Both settings are enabled by default for the entire
-subdirectory.
+error, if the stubs are correct. Both settings are enabled by default for
+all `@tests/test_cases/` subdirectories in typeshed.
 
 For more information on using `assert_type` and
 `--warn-unused-ignores`/`reportUnnecessaryTypeIgnoreComment` to test type
@@ -65,21 +77,21 @@ provides a useful guide.
 
 ### Naming convention
 
-Use the same top-level name for the module / package you would like to test.
+Use the same top-level name for the module or package you would like to test.
 Use the `check_${thing}.py` naming pattern for individual test files.
 
 By default, test cases go into a file with the same name as the stub file, prefixed with `check_`.
-For example: `check_contextlib.py`.
+For example: `stdlib/@tests/test_cases/check_contextlib.py`.
 
 If that file becomes too big, we instead create a directory with files named after individual objects being tested.
-For example: `builtins/check_dict.py`.
+For example: `stdlib/@tests/test_cases/builtins/check_dict.py`.
 
 ### Differences to the rest of typeshed
 
-Unlike the rest of typeshed, this directory largely contains `.py` files. This
-is because the purpose of this folder is to test the implications of typeshed
-changes for end users, who will mainly be using `.py` files rather than `.pyi`
-files.
+Unlike the rest of typeshed, the `@tests/test_cases` directories largely
+contain `.py` files. This is because the purpose of these directories is to
+test the implications of typeshed changes for end users, who will mainly be
+using `.py` files rather than `.pyi` files.
 
 Another difference to the rest of typeshed
 (which stems from the fact that the test-case files are all `.py` files
@@ -109,7 +121,8 @@ with a specific Python version passed on the command line to the `tests/regr_tes
 To mark a test-case file as being skippable on lower versions of Python,
 append `-py3*` to the filename.
 For example, if `foo` is a stdlib feature that's new in Python 3.11,
-test cases for `foo` should be put in a file named `check_foo-py311.py`.
+test cases for `foo` should be put in a file named
+`stdlib/@tests/test_cases/check_foo-py311.py`.
 This means that mypy will only run the test case
 if `--python-version 3.11`, `--python-version 3.12`, etc.
 is passed on the command line to `tests/regr_test.py`,
@@ -118,4 +131,4 @@ is passed on the command line.
 
 However, `if sys.version_info >= (3, target):` is still required for `pyright`
 in the test file itself.
-Example: [`check_exception_group-py311.py`](../stdlib/@tests/test_cases/builtins/check_exception_group-py311.py)
+Example: [`stdlib/@tests/test_cases/check_exception_group-py311.py`](../stdlib/@tests/test_cases/builtins/check_exception_group-py311.py)
