gh-115154: Fix untokenize handling of unicode named literals#115171
gh-115154: Fix untokenize handling of unicode named literals#115171pablogsal merged 8 commits intopython:mainfrom
Conversation
Signed-off-by: Pablo Galindo <[email protected]>
|
This failed twice on Windows x64 with the same random file (test_peepholer.py) and similarly on Windows x64 (freethreading) on test_xml_etree.py |
|
Doc/Doctest failed with "python: ./Include/internal/pycore_typeobject.h:105: _PyType_GetModuleState: Assertion `et->ht_module' failed. Aborted (core dumped)". I don't know what this is. Windows / build and test (x64) had the same failure on 'test_peepholer' as I showed above for the same binary. I ran test_tokenize with free-threading debug and test_random_files failed with test_format.py, test_os.py, test_peepholer.py, and test_xml_etree.py. EDIT I ran test.test_tokenize, so that The patch does fix the targeted f-string failure. With a better fix, we can implement |
|
To determine where the patch iinitially fails in each file, I wrote this test program. from tokenize import generate_tokens, untokenize
def test(file):
toks = list(generate_tokens(file.readline))
code = untokenize(toks)
readline = iter(code.splitlines(keepends=True)).__next__
toks2 = list(generate_tokens(readline))
print('tok lengths:', len(toks), len(toks2))
for t1, t2 in zip(toks, toks2):
if t1[0:2] != t2[0:2]:
for t in t1, t2:
print(f'{t[0]}, {t[1]}, {t[2]}, {t[3]}\n{t[4]}', end='')
print()
break
for name in ('format', 'os', 'peepholer', 'xml_etree'):
filename = f'f:/dev/3x/Lib/test/test_{name}.py'
print(filename)
with open(filename) as f:
test(f)It prints The failures are all about '{{' in code becoming '{' when tokenized and untokenized. |
|
Thanks a lot @terryjreedy, this is extremely useful. I will try to adapt the patch today to cover this case and I will add @serhiy-storchaka tests cases to the PR |
|
I changed the strategy but I think this should do the trick. Also, now that I was at it I added some fixes to get |
Misc/NEWS.d/next/Core and Builtins/2024-02-08-16-01-18.gh-issue-115154.ji96FV.rst
Outdated
Show resolved
Hide resolved
Co-authored-by: Alex Waygood <[email protected]>
|
python -m test.test_tokenize, testing all 388 (with test _fstring no longer deleted) test_x modules, passes on my machine. |
It also passes for me, on a macbook (I passed the |
|
Thanks @pablogsal for the PR 🌮🎉.. I'm working now to backport this PR to: 3.12. |
…ythonGH-115171) (cherry picked from commit ecf16ee) Co-authored-by: Pablo Galindo Salgado <[email protected]>
|
GH-115662 is a backport of this pull request to the 3.12 branch. |
Signed-off-by: Pablo Galindo [email protected]