-
Notifications
You must be signed in to change notification settings - Fork 3.9k
Fix unit tests for Windows #537
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Closed
Closed
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
On Windows (only), a file created by NamedTemporaryFile can't be opened again while it is already open. That causes some tests to fail with PermissionError even when the code under test is correct. To fix it without making the test code longer or more complicated, we can use the tmp_path pytest fixture, which makes a temporary directory (separate for each test and each run) where we can create whatever file(s) we need, and which handles cleanup. This has the further benefit of being a more idiomatic way to work with temporary files in pytest tests. This commit makes that change for test_file_cli.
For improved readability.
As in test_file_cli, these failed with PermissionError on Windows by trying to open a file created by NamedTemporary file while it is already open. This likewise switches to using the pytest tmp_path fixture. The api_key_file fixture is modified to request and use it, and to provide a Path object, which test case functions call write_text on instead of printing and flushing. Since this was a major change to the api_key_file fixture, I've also taken the opportunity to have it delegate patching and automatically unpatching openai.api_key_path to another pytest fixture, allowing its code to be shorter and simpler.
(It's unused because of how openai.api_key_path now gets patched.)
This fixes one of the problems running test_long_examples_validator on Windows: the command run by subprocess.run giving a "The filename, directory name, or volume label syntax is incorrect" error. It happens because the command to run is specified as a list of one element that is a string for the shell to interpret, rather than that string itself. This ends up working on Unix-like systems, but on Windows it results in extra quotation (adjacent `"` characters) that cmd.exe doesn't understand. This changes the subprocess.run call to what was probably intended, but it may make sense to modify it further to make it more consistent with how the subprocess module is used in other tests. Note that test_long_examples_validator is not fully fixed; now it raises PermissionError on Windows (the NamedTemporaryFile issue).
This makes the subprocess.run call in test_long_examples_validator more similar to how other tests use the subprocess module, by passing a list of separate command-line arguments instead of using a shell to parse a single string.
This fixes the Windows-specific PermissionError that happened because of how NamedTemporaryFile was used. As in other such tests, the pytest tmp_path fixture is now used instead.
In test_long_examples_validator, subprocess.run was treating the child process's standard streams as UTF-8 encoded. It's usually good practice to specify UTF-8, but in this case, the child process uses a different encoding for its standard streams on Windows. On many Windows systems, this is CP-1252 (Windows-1252), and the curly apostrophe in the "After you’ve fine-tuned a model..." message triggers a UnicodeDecodeError. Although the message could be changed, the test's correcteness probably shouldn't rely on knowledge of the output besides what we specifically test. This has subprocess.run use the system default encoding for the child process's standard streams. Since the child process is itself Python, this should match in nearly all circumstances.
- Code style (variable names, keyword argument order) - File encoding (specify as UTF-8 in all, not just some, cases) - Child process standard stream encoding (system default) (The encoding of the temporary files, and of stdin/stdout for the child processes, are separate. Even when text from the temporary files is output by a child process, it is encoded with whatever the stream's encoding is, which for standard streams is the default system encoding.)
When using Python 3.7 on Windows (but not on other systems, and not on Windows with any later version of Python), subprocess.run and related functions don't support Path objects as command line arguments, due to python/cpython#85919. This explicitly converts them to str in those calls. This change can be undone whenever this codebase drops Python 3.7 support.
Thanks for this! We've since rewritten the library entirely, so this change is no longer relevant. I'm sorry we didn't get to it sooner. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fixes #536
This fixes the four unit tests that were inadvertently incompatible with Windows, so that they now work on all systems, by:
Using the
tmp_path
pytest fixture instead ofNamedTemporaryFile
. It is possible to fix this while still usingNamedTempoaryFile
, but doing so involves suppressing automatic deletion, manually closing the file, and then taking care of deletion. In contrast,tmp_path
works on all systems and, in pytest tests, it is a more idiomatic way to deal with temporary files. It does work differently, creating a directory inside which one creates whatever files are desired, but this turned out to work just as well for expressing the test logic (in my opinion).In one of the places where I made this change, I also replaced manual monkey patching with the
monkeypatch
pytest fixture. I did this because the combination of those two changes allowed the code to be simplified (otherwise it might have become more complicated).Passing a list of separate arguments to
subprocess.run
and not usingshell=True
. Other approaches, like passing a string instead of a list and keepingshell=True
, also work. But this way is more similar to the other uses ofsubprocess
functions in this project's unit tests.Having
subprocess.run
use the default system encoding for child processes' standard streams, when accessing them in text mode. Because the child processes are also Python--they are scripts provided by this project--their stdin, stdout, and stderr automatically use the system's default encoding, as detected by Python. Therefore, while omittingencoding="utf-8"
in a call to any function that accepts it feels wrong, it fixes the problem here and I believe is more robust.Of course, that is only for
subprocess.run
. Accessing named files on disk should generally specifyencoding="utf-8"
, which I have not removed (and I added it in a couple places where it was missing).This only modifies test code; other code is unchanged. I've checked that no new
mypy
orblack --check
warnings are introduced.I have tried to take care to avoid creating new incompatibilities, either with Windows or other operating systems. One thing I've done to avoid that is to make a number of small, self-contained changes in separate commits, and to describe each. (In one case, I did temporarily introduce a new incompatibility, due to python/cpython#85919. A subsequent commit fixes it.) I assume this is okay, because pull requests in this project are usually merged using "squash and merge." However, I'd be pleased to make any requested changes (including abridging commit messages if necessary). Maintainers can also push changes to my branch.
The other thing I've done to avoid introducing new problems is to test these changes on all 15 combinations of supported Python version (3.7, 3.8, 3.9, 3.10, 3.11) and major supported operating system (Ubuntu and macOS to check that I wasn't breaking them, and Windows to make sure this really does fix the incompatibilities without introducing new ones). I tested Windows and Ubuntu, with all five Python versions, both locally and on CI, and macOS just on CI. I used a CI workflow based on @tuliren's workflow at tuliren#1, with Windows and macOS platforms added (and a few other changes). With #535 unfixed, tests fail, but in the correct and expected way. With #535 fixed (see #406), all tests pass on all operating systems.
Please note that this PR does not itself address #535, which is conceptually unrelated to the changes here. I just needed to test it in both ways to make sure I was really making the tests compatible with Windows, in the case of
test_long_examples_validator
.