Thanks to visit codestin.com
Credit goes to github.com

Skip to content

stubtest crash when a referenced module is a DLL #13822

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
Avasam opened this issue Oct 5, 2022 · 2 comments · Fixed by #13848
Closed

stubtest crash when a referenced module is a DLL #13822

Avasam opened this issue Oct 5, 2022 · 2 comments · Fixed by #13848
Labels
crash topic-stubtest

Comments

@Avasam
Copy link
Contributor

Avasam commented Oct 5, 2022 •
edited
Loading

Crash Report

stubtest crashes when a referenced module is a DLL, as it seemingly tries to parse the .dll's content as text.

Traceback

PS C:\Users\Avasam\Documents\Git\typeshed> python .\tests\stubtest_third_party.py pywin32
pywin32... fail

C:\Users\Avasam\AppData\Local\Temp\tmp3yxfa81_\Scripts\pip.exe install pywin32[]==304.* mypy==0.982
MYPYPATH=C:\Users\Avasam\Documents\Git\typeshed\stubs\pywin32 C:\Users\Avasam\AppData\Local\Temp\tmp3yxfa81_\Scripts\python.exe -m mypy.stubtest --custom-typeshed-dir C:\Users\Avasam\Documents\Git\typeshed pythonwin win32 win32comext pythoncom _win32typing --allowlist C:\Users\Avasam\Documents\Git\typeshed\stubs\pywin32\@tests\stubtest_allowlist.txt

Traceback (most recent call last):
  File "C:\Program Files\Python39\lib\tokenize.py", line 330, in find_cookie
    line_string = line.decode('utf-8')
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x90 in position 2: invalid start byte

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Program Files\Python39\lib\runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "C:\Program Files\Python39\lib\runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "C:\Users\Avasam\AppData\Local\Temp\tmp3yxfa81_\lib\site-packages\mypy\stubtest.py", line 1713, in <module>
    sys.exit(main())
  File "C:\Users\Avasam\AppData\Local\Temp\tmp3yxfa81_\lib\site-packages\mypy\stubtest.py", line 1709, in main
    return test_stubs(parse_options(sys.argv[1:]))
  File "C:\Users\Avasam\AppData\Local\Temp\tmp3yxfa81_\lib\site-packages\mypy\stubtest.py", line 1605, in test_stubs
    print(error.get_description(concise=args.concise))
  File "C:\Users\Avasam\AppData\Local\Temp\tmp3yxfa81_\lib\site-packages\mypy\stubtest.py", line 138, in get_description
    runtime_line = inspect.getsourcelines(self.runtime_object)[1]
  File "C:\Program Files\Python39\lib\inspect.py", line 1006, in getsourcelines
    lines, lnum = findsource(object)
  File "C:\Program Files\Python39\lib\inspect.py", line 831, in findsource
    lines = linecache.getlines(file, module.__dict__)
  File "C:\Program Files\Python39\lib\linecache.py", line 46, in getlines
    return updatecache(filename, module_globals)
  File "C:\Program Files\Python39\lib\linecache.py", line 136, in updatecache
    with tokenize.open(fullname) as fp:
  File "C:\Program Files\Python39\lib\tokenize.py", line 394, in open
    encoding, lines = detect_encoding(buffer.readline)
  File "C:\Program Files\Python39\lib\tokenize.py", line 371, in detect_encoding
    encoding = find_cookie(first)
  File "C:\Program Files\Python39\lib\tokenize.py", line 335, in find_cookie
    raise SyntaxError(msg)
SyntaxError: invalid or missing encoding declaration for 'C:\\Users\\Avasam\\AppData\\Local\\Temp\\tmp3yxfa81_\\lib\\site-packages\\pywin32_system32\\pywintypes39.dll'

To Reproduce

The easiest way to reproduce would be to use the pywin32 stubs from typeshed as an example. It has references to pywintypes39.dll and pythoncom39.dll. Make sure that METADATA.toml has:

[tool.stubtest]
ignore_missing_stub = false
skip = false

python .\tests\stubtest_third_party.py pywin32

Your Environment

  • Mypy version used: 0.981 (compiled: yes)
  • Mypy command-line flags: (see above)
  • Mypy configuration options from mypy.ini (and other config files): (see above)
  • Python version used: 3.9.13
  • Operating system and version: Windows 10.0.19044 Build 19044
@Avasam Avasam added the crash label Oct 5, 2022
@Avasam
Copy link
Contributor Author

Avasam commented Oct 5, 2022 •
edited
Loading

Related, very similar error in stubgen as well:
stubgen -p pythoncom

Results in:
Critical error during semantic analysis: mypy: can't decode file '[...]\.venv\Lib\site-packages\pywin32_system32\pythoncom39.dll': 'utf-8' codec can't decode byte 0x90 in position 2: invalid start byte

@JelleZijlstra
Copy link
Member

This feels like a CPython bug, since stubtest is just calling inspect.getsourcelines. I think the relevant lines are here: https://github.com/python/cpython/blob/b44372e03c5461b6ad3d89763a9eb6cb82df07a4/Lib/inspect.py#L946

Basically, inspect doesn't try to find a source file if the object's __file__ has one of the importlib.machinery.EXTENSION_SUFFIXES. But in this case, I suppose pywin32 exposes an object with a __file__ that points to a .dll.

Where should this be fixed?

  • Maybe CPython should try harder to avoid loading binary files as source files. But I don't know if that's going to be fully reliable, and people do load Python code from files with non-.py extension files.
  • Maybe pywin32 shouldn't set a __file__ pointing to a file that's not a valid extension module.
  • Maybe stubtest should be more resilient to getsourcelines() failing. That's not very principled, but it's good for stubtest to be resilient to odd failures, so I think we should do that.

@Avasam Avasam mentioned this issue Oct 7, 2022
Merged
hauntsaninja added a commit to hauntsaninja/mypy that referenced this issue Oct 9, 2022
@hauntsaninja
stubtest: catch SyntaxError from inspect.getsourcelines
90cda6d 
Fixes python#13822
@hauntsaninja hauntsaninja mentioned this issue Oct 9, 2022
Merged
JelleZijlstra pushed a commit that referenced this issue Oct 9, 2022
@hauntsaninja
stubtest: catch SyntaxError from inspect.getsourcelines (#13848)
1fbbe91 
Fixes #13822
This was referenced Nov 7, 2022
Closed
Closed
@Avasam Avasam mentioned this issue Jan 27, 2024
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
crash topic-stubtest
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

stubtest: catch SyntaxError from inspect.getsourcelines hauntsaninja/mypy
3 participants
@JelleZijlstra @Avasam @AlexWaygood