-
-
Notifications
You must be signed in to change notification settings - Fork 31.9k
gh-115154: Fix untokenize handling of unicode named literals #115171
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
Signed-off-by: Pablo Galindo <[email protected]>
This failed twice on Windows x64 with the same random file (test_peepholer.py)
and similarly on Windows x64 (freethreading) on test_xml_etree.py
|
Doc/Doctest failed with "python: ./Include/internal/pycore_typeobject.h:105: _PyType_GetModuleState: Assertion `et->ht_module' failed. Aborted (core dumped)". I don't know what this is. Windows / build and test (x64) had the same failure on 'test_peepholer' as I showed above for the same binary. I ran test_tokenize with free-threading debug and test_random_files failed with test_format.py, test_os.py, test_peepholer.py, and test_xml_etree.py. EDIT I ran test.test_tokenize, so that The patch does fix the targeted f-string failure. With a better fix, we can implement
|
To determine where the patch iinitially fails in each file, I wrote this test program. from tokenize import generate_tokens, untokenize
def test(file):
toks = list(generate_tokens(file.readline))
code = untokenize(toks)
readline = iter(code.splitlines(keepends=True)).__next__
toks2 = list(generate_tokens(readline))
print('tok lengths:', len(toks), len(toks2))
for t1, t2 in zip(toks, toks2):
if t1[0:2] != t2[0:2]:
for t in t1, t2:
print(f'{t[0]}, {t[1]}, {t[2]}, {t[3]}\n{t[4]}', end='')
print()
break
for name in ('format', 'os', 'peepholer', 'xml_etree'):
filename = f'f:/dev/3x/Lib/test/test_{name}.py'
print(filename)
with open(filename) as f:
test(f) It prints
The failures are all about '{{' in code becoming '{' when tokenized and untokenized. |
Thanks a lot @terryjreedy, this is extremely useful. I will try to adapt the patch today to cover this case and I will add @serhiy-storchaka tests cases to the PR |
I changed the strategy but I think this should do the trick. Also, now that I was at it I added some fixes to get |
Misc/NEWS.d/next/Core and Builtins/2024-02-08-16-01-18.gh-issue-115154.ji96FV.rst
Outdated
Show resolved
Hide resolved
Co-authored-by: Alex Waygood <[email protected]>
python -m test.test_tokenize, testing all 388 (with test _fstring no longer deleted) test_x modules, passes on my machine. |
It also passes for me, on a macbook (I passed the |
Thanks @pablogsal for the PR 🌮🎉.. I'm working now to backport this PR to: 3.12. |
…ythonGH-115171) (cherry picked from commit ecf16ee) Co-authored-by: Pablo Galindo Salgado <[email protected]>
GH-115662 is a backport of this pull request to the 3.12 branch. |
Signed-off-by: Pablo Galindo [email protected]