Thanks to visit codestin.com
Credit goes to github.com

Skip to content

Fix unit tests for Windows #537

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 10 commits into from
Closed

Conversation

EliahKagan
Copy link

@EliahKagan EliahKagan commented Jul 13, 2023

Fixes #536

This fixes the four unit tests that were inadvertently incompatible with Windows, so that they now work on all systems, by:

  1. Using the tmp_path pytest fixture instead of NamedTemporaryFile. It is possible to fix this while still using NamedTempoaryFile, but doing so involves suppressing automatic deletion, manually closing the file, and then taking care of deletion. In contrast, tmp_path works on all systems and, in pytest tests, it is a more idiomatic way to deal with temporary files. It does work differently, creating a directory inside which one creates whatever files are desired, but this turned out to work just as well for expressing the test logic (in my opinion).

    In one of the places where I made this change, I also replaced manual monkey patching with the monkeypatch pytest fixture. I did this because the combination of those two changes allowed the code to be simplified (otherwise it might have become more complicated).

  2. Passing a list of separate arguments to subprocess.run and not using shell=True. Other approaches, like passing a string instead of a list and keeping shell=True, also work. But this way is more similar to the other uses of subprocess functions in this project's unit tests.

  3. Having subprocess.run use the default system encoding for child processes' standard streams, when accessing them in text mode. Because the child processes are also Python--they are scripts provided by this project--their stdin, stdout, and stderr automatically use the system's default encoding, as detected by Python. Therefore, while omitting encoding="utf-8" in a call to any function that accepts it feels wrong, it fixes the problem here and I believe is more robust.

    Of course, that is only for subprocess.run. Accessing named files on disk should generally specify encoding="utf-8", which I have not removed (and I added it in a couple places where it was missing).

This only modifies test code; other code is unchanged. I've checked that no new mypy or black --check warnings are introduced.

I have tried to take care to avoid creating new incompatibilities, either with Windows or other operating systems. One thing I've done to avoid that is to make a number of small, self-contained changes in separate commits, and to describe each. (In one case, I did temporarily introduce a new incompatibility, due to python/cpython#85919. A subsequent commit fixes it.) I assume this is okay, because pull requests in this project are usually merged using "squash and merge." However, I'd be pleased to make any requested changes (including abridging commit messages if necessary). Maintainers can also push changes to my branch.

The other thing I've done to avoid introducing new problems is to test these changes on all 15 combinations of supported Python version (3.7, 3.8, 3.9, 3.10, 3.11) and major supported operating system (Ubuntu and macOS to check that I wasn't breaking them, and Windows to make sure this really does fix the incompatibilities without introducing new ones). I tested Windows and Ubuntu, with all five Python versions, both locally and on CI, and macOS just on CI. I used a CI workflow based on @tuliren's workflow at tuliren#1, with Windows and macOS platforms added (and a few other changes). With #535 unfixed, tests fail, but in the correct and expected way. With #535 fixed (see #406), all tests pass on all operating systems.

Please note that this PR does not itself address #535, which is conceptually unrelated to the changes here. I just needed to test it in both ways to make sure I was really making the tests compatible with Windows, in the case of test_long_examples_validator.

On Windows (only), a file created by NamedTemporaryFile can't be
opened again while it is already open. That causes some tests to
fail with PermissionError even when the code under test is correct.

To fix it without making the test code longer or more complicated,
we can use the tmp_path pytest fixture, which makes a temporary
directory (separate for each test and each run) where we can create
whatever file(s) we need, and which handles cleanup. This has the
further benefit of being a more idiomatic way to work with
temporary files in pytest tests.

This commit makes that change for test_file_cli.
As in test_file_cli, these failed with PermissionError on Windows
by trying to open a file created by NamedTemporary file while it is
already open. This likewise switches to using the pytest tmp_path
fixture. The api_key_file fixture is modified to request and use
it, and to provide a Path object, which test case functions call
write_text on instead of printing and flushing.

Since this was a major change to the api_key_file fixture, I've
also taken the opportunity to have it delegate patching and
automatically unpatching openai.api_key_path to another pytest
fixture, allowing its code to be shorter and simpler.
(It's unused because of how openai.api_key_path now gets patched.)
This fixes one of the problems running test_long_examples_validator
on Windows: the command run by subprocess.run giving a "The
filename, directory name, or volume label syntax is incorrect"
error.

It happens because the command to run is specified as a list of one
element that is a string for the shell to interpret, rather than
that string itself. This ends up working on Unix-like systems, but
on Windows it results in extra quotation (adjacent `"` characters)
that cmd.exe doesn't understand.

This changes the subprocess.run call to what was probably intended,
but it may make sense to modify it further to make it more
consistent with how the subprocess module is used in other tests.

Note that test_long_examples_validator is not fully fixed; now it
raises PermissionError on Windows (the NamedTemporaryFile issue).
This makes the subprocess.run call in test_long_examples_validator
more similar to how other tests use the subprocess module, by
passing a list of separate command-line arguments instead of using
a shell to parse a single string.
This fixes the Windows-specific PermissionError that happened
because of how NamedTemporaryFile was used. As in other such tests,
the pytest tmp_path fixture is now used instead.
In test_long_examples_validator, subprocess.run was treating the
child process's standard streams as UTF-8 encoded. It's usually
good practice to specify UTF-8, but in this case, the child
process uses a different encoding for its standard streams on
Windows.

On many Windows systems, this is CP-1252 (Windows-1252), and the
curly apostrophe in the "After you’ve fine-tuned a model..."
message triggers a UnicodeDecodeError. Although the message could
be changed, the test's correcteness probably shouldn't rely on
knowledge of the output besides what we specifically test.

This has subprocess.run use the system default encoding for the
child process's standard streams. Since the child process is itself
Python, this should match in nearly all circumstances.
- Code style (variable names, keyword argument order)
- File encoding (specify as UTF-8 in all, not just some, cases)
- Child process standard stream encoding (system default)

(The encoding of the temporary files, and of stdin/stdout for the
child processes, are separate. Even when text from the temporary
files is output by a child process, it is encoded with whatever the
stream's encoding is, which for standard streams is the default
system encoding.)
When using Python 3.7 on Windows (but not on other systems, and not
on Windows with any later version of Python), subprocess.run and
related functions don't support Path objects as command line
arguments, due to python/cpython#85919.

This explicitly converts them to str in those calls. This change
can be undone whenever this codebase drops Python 3.7 support.
@rattrayalex
Copy link
Collaborator

Thanks for this!

We've since rewritten the library entirely, so this change is no longer relevant. I'm sorry we didn't get to it sooner.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Tests with temporary files or subprocesses are broken on Windows
2 participants