Thanks to visit codestin.com
Credit goes to github.com

Skip to content

BUG: FIxed an issue where running mypy could raise an AssertionError #17376

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 3 commits into from

Conversation

BvB93
Copy link
Member

@BvB93 BvB93 commented Sep 25, 2020

Closes #17316.

Fixes an issue where certain import cycles could crash mypy.
That, and np.core shouldn't have been imported here in the first place (as it's not a public module).

@BvB93
Copy link
Member Author

BvB93 commented Sep 25, 2020

@charris and @mattip can you confirm whether or not this resolves the issues you were previously experiencing?

@charris
Copy link
Member

charris commented Sep 25, 2020

I'm still getting 36 errors with python runtests.py -t numpy/typing/tests. Mypy looks current: 0.782.

@BvB93
Copy link
Member Author

BvB93 commented Sep 26, 2020

Does something similar happen when running python ./runtests.py --mypy numpy?

@charris
Copy link
Member

charris commented Sep 26, 2020

That seems to work.

charris@fc [numpy.git ((b45068860c...))]$ python ./runtests.py --mypy numpy
Building, see build.log...
    ... build in progress
    ... build in progress
Build OK
Success: no issues found in 376 source files

OTOH

python ./runtests.py --mypy numpy/typing/tests

Fails. Indeed, any path ending in tests fails with a long traceback and assertion error. But python ./runtests.py --mypy numpy/typing works just fine.

@BvB93
Copy link
Member Author

BvB93 commented Sep 26, 2020

That seems to work.

I suspect there is an older preexisting mypy cache which is causing problems. If my suspicion is correct
then b9724f1 should fix the issue.

Note that this shouldn't affect the CI execution speed, as those start without any cached mypy files anyway.

@charris
Copy link
Member

charris commented Sep 26, 2020

Doesn't help here.

@mattip
Copy link
Member

mattip commented Sep 26, 2020

Likewise, I am still getting 36 failures. Here is the traceback for the last one, they are all similar.

_________________________________________________________________________________________ test_reveal[ndarray_shape_manipulation.py] _________________________________________________________________________________________

path = '/home/matti/pypy_stuff/numpy/build/testenv/lib/python3.8/site-packages/numpy/typing/tests/data/reveal/ndarray_shape_manipulation.py'

    @pytest.mark.skipif(NO_MYPY, reason="Mypy is not installed")
    @pytest.mark.parametrize("path", get_test_cases(REVEAL_DIR))
    def test_reveal(path):
>       stdout, stderr, exitcode = api.run([
            "--config-file",
            MYPY_INI,
            "--cache-dir",
            CACHE_DIR,
            path,
        ])

path       = '/home/matti/pypy_stuff/numpy/build/testenv/lib/python3.8/site-packages/numpy/typing/tests/data/reveal/ndarray_shape_manipulation.py'

numpy/typing/tests/test_typing.py:112: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
/home/matti/miniconda3/envs/numpy/lib/python3.8/site-packages/mypy/api.py:69: in run
    return _run(lambda stdout, stderr: main(None, args=args,
        args       = ['--config-file', '/home/matti/pypy_stuff/numpy/build/testenv/lib/python3.8/site-packages/numpy/typing/tests/data/mypy...py_stuff/numpy/build/testenv/lib/python3.8/site-packages/numpy/typing/tests/data/reveal/ndarray_shape_manipulation.py']
        main       = <function main at 0x7f4d32081e50>
/home/matti/miniconda3/envs/numpy/lib/python3.8/site-packages/mypy/api.py:58: in _run
    main_wrapper(stdout, stderr)
        main_wrapper = <function run.<locals>.<lambda> at 0x7f4c5c3098b0>
        stderr     = <_io.StringIO object at 0x7f4c577c09d0>
        stdout     = <_io.StringIO object at 0x7f4c577c0820>
/home/matti/miniconda3/envs/numpy/lib/python3.8/site-packages/mypy/api.py:69: in <lambda>
    return _run(lambda stdout, stderr: main(None, args=args,
        args       = ['--config-file', '/home/matti/pypy_stuff/numpy/build/testenv/lib/python3.8/site-packages/numpy/typing/tests/data/mypy...py_stuff/numpy/build/testenv/lib/python3.8/site-packages/numpy/typing/tests/data/reveal/ndarray_shape_manipulation.py']
        main       = <function main at 0x7f4d32081e50>
        stderr     = <_io.StringIO object at 0x7f4c577c09d0>
        stdout     = <_io.StringIO object at 0x7f4c577c0820>
/home/matti/miniconda3/envs/numpy/lib/python3.8/site-packages/mypy/main.py:89: in main
    res = build.build(sources, options, None, flush_errors, fscache, stdout, stderr)
        args       = ['--config-file', '/home/matti/pypy_stuff/numpy/build/testenv/lib/python3.8/site-packages/numpy/typing/tests/data/mypy...py_stuff/numpy/build/testenv/lib/python3.8/site-packages/numpy/typing/tests/data/reveal/ndarray_shape_manipulation.py']
        blockers   = False
        flush_errors = <function main.<locals>.flush_errors at 0x7f4c42424c10>
        formatter  = <mypy.util.FancyFormatter object at 0x7f4c38be85e0>
        fscache    = <mypy.fscache.FileSystemCache object at 0x7f4c38be8430>
        messages   = []
        options    = Options({'allow_redefinition': False,
 'allow_untyped_globals': False,
 'always_false': [],
 'always_true': [],
 'baze...,
 'warn_return_any': False,
 'warn_unreachable': False,
 'warn_unused_configs': False,
 'warn_unused_ignores': False})
        res        = None
        script_path = None
        serious    = False
        sources    = [<BuildSource path='/home/matti/pypy_stuff/numpy/build/testenv/lib/python3.8/site-packages/numpy/typing/tests/data/rev...=False base_dir=/home/matti/pypy_stuff/numpy/build/testenv/lib/python3.8/site-packages/numpy/typing/tests/data/reveal>]
        stderr     = <_io.StringIO object at 0x7f4c577c09d0>
        stdout     = <_io.StringIO object at 0x7f4c577c0820>
        t0         = 1601152619.1519773
/home/matti/miniconda3/envs/numpy/lib/python3.8/site-packages/mypy/build.py:180: in build
    result = _build(
        alt_lib_path = None
        default_flush_errors = <function build.<locals>.default_flush_errors at 0x7f4c4c802ca0>
        extra_plugins = []
        flush_errors = <function main.<locals>.flush_errors at 0x7f4c42424c10>
        fscache    = <mypy.fscache.FileSystemCache object at 0x7f4c38be8430>
        messages   = []
        options    = Options({'allow_redefinition': False,
 'allow_untyped_globals': False,
 'always_false': [],
 'always_true': [],
 'baze...,
 'warn_return_any': False,
 'warn_unreachable': False,
 'warn_unused_configs': False,
 'warn_unused_ignores': False})
        sources    = [<BuildSource path='/home/matti/pypy_stuff/numpy/build/testenv/lib/python3.8/site-packages/numpy/typing/tests/data/rev...=False base_dir=/home/matti/pypy_stuff/numpy/build/testenv/lib/python3.8/site-packages/numpy/typing/tests/data/reveal>]
        stderr     = <_io.StringIO object at 0x7f4c577c09d0>
        stdout     = <_io.StringIO object at 0x7f4c577c0820>
/home/matti/miniconda3/envs/numpy/lib/python3.8/site-packages/mypy/build.py:252: in _build
    graph = dispatch(sources, manager, stdout)
        alt_lib_path = None
        cache_dir_existed = True
        cached_read = <bound method FileSystemCache.read of <mypy.fscache.FileSystemCache object at 0x7f4c38be8430>>
        data_dir   = '/home/matti/miniconda3/envs/numpy/lib/python3.8/site-packages/mypy'
        errors     = <mypy.errors.Errors object at 0x7f4c3cced490>
        extra_plugins = []
        flush_errors = <function main.<locals>.flush_errors at 0x7f4c42424c10>
        fscache    = <mypy.fscache.FileSystemCache object at 0x7f4c38be8430>
        manager    = <mypy.build.BuildManager object at 0x7f4c3cced3d0>
        options    = Options({'allow_redefinition': False,
 'allow_untyped_globals': False,
 'always_false': [],
 'always_true': [],
 'baze...,
 'warn_return_any': False,
 'warn_unreachable': False,
 'warn_unused_configs': False,
 'warn_unused_ignores': False})
        plugin     = <mypy.plugins.default.DefaultPlugin object at 0x7f4c3ccedc70>
        reports    = None
        search_paths = SearchPaths(python_path=('/home/matti/pypy_stuff/numpy/build/testenv/lib/python3.8/site-packages/numpy/typing/tests/da.../home/matti/miniconda3/envs/numpy/lib/python3.8/site-packages/mypy/typeshed/third_party/2and3', '/usr/local/lib/mypy'))
        snapshot   = {}
        source_set = <mypy.build.BuildSourceSet object at 0x7f4c38be8c40>
        sources    = [<BuildSource path='/home/matti/pypy_stuff/numpy/build/testenv/lib/python3.8/site-packages/numpy/typing/tests/data/rev...=False base_dir=/home/matti/pypy_stuff/numpy/build/testenv/lib/python3.8/site-packages/numpy/typing/tests/data/reveal>]
        stderr     = <_io.StringIO object at 0x7f4c577c09d0>
        stdout     = <_io.StringIO object at 0x7f4c577c0820>
        t0         = 1601152634.4924617
/home/matti/miniconda3/envs/numpy/lib/python3.8/site-packages/mypy/build.py:2626: in dispatch
    process_graph(graph, manager)
        graph      = {'__future__': <mypy.build.State object at 0x7f4c4a49a220>, '_ast': <mypy.build.State object at 0x7f4cd428c760>, '_bisect': <mypy.build.State object at 0x7f4c4dde50a0>, '_compression': <mypy.build.State object at 0x7f4c6996e850>, ...}
        manager    = <mypy.build.BuildManager object at 0x7f4c3cced3d0>
        sources    = [<BuildSource path='/home/matti/pypy_stuff/numpy/build/testenv/lib/python3.8/site-packages/numpy/typing/tests/data/rev...=False base_dir=/home/matti/pypy_stuff/numpy/build/testenv/lib/python3.8/site-packages/numpy/typing/tests/data/reveal>]
        stdout     = <_io.StringIO object at 0x7f4c577c0820>
        t0         = 1601152619.156744
        t1         = 1601152620.644785
/home/matti/miniconda3/envs/numpy/lib/python3.8/site-packages/mypy/build.py:2949: in process_graph
    process_stale_scc(graph, scc, manager)
        ascc       = frozenset({'_pytest.assertion', '_pytest.assertion.rewrite', '_pytest.assertion.truncate', '_pytest.cacheprovider', '_pytest.capture', '_pytest.config', ...})
        deps       = {'_ast', '_curses', '_dummy_thread', '_importlib_modulespec', '_pytest', '_pytest._code', ...}
        fresh      = False
        fresh_msg  = 'inherently stale (_pytest.assertion _pytest.assertion.rewrite _pytest.assertion.truncate _pytest.cacheprovider _pytes...mpy.testing._private.noseclasses numpy.testing._private.nosetester numpy.testing._private.utils pytest pytest.collect)'
        fresh_scc_queue = []
        graph      = {'__future__': <mypy.build.State object at 0x7f4c4a49a220>, '_ast': <mypy.build.State object at 0x7f4cd428c760>, '_bisect': <mypy.build.State object at 0x7f4c4dde50a0>, '_compression': <mypy.build.State object at 0x7f4c6996e850>, ...}
        id         = 'hypothesis.extra.django'
        manager    = <mypy.build.BuildManager object at 0x7f4c3cced3d0>
        newest_in_deps = 0
        oldest_in_scc = 1601152074
        prev_scc   = ['packaging.requirements']
        scc        = ['_pytest.config.findpaths', '_pytest.hookspec', 'numpy.f2py.cfuncs', 'numpy.f2py.diagnose', 'numpy.distutils.ccompiler_opt', 'numpy.distutils.misc_util', ...]
        scc_str    = '_pytest.config.findpaths _pytest.hookspec numpy.f2py.cfuncs numpy.f2py.diagnose numpy.distutils.ccompiler_opt numpy.d...mpl _pytest.helpconfig _pytest.cacheprovider _pytest.terminal _pytest.warnings _pytest.capture hypothesis.extra.django'
        sccs       = [frozenset({'_ast', '_importlib_modulespec', 'abc', 'ast', 'builtins', 'codecs', ...}), frozenset({'fileinput'}), frozenset({'email.mime'}), frozenset({'_markupbase'}), frozenset({'html'}), frozenset({'numpy.f2py.__version__'}), ...]
        size       = 186
        stale_deps = set()
        stale_scc  = {'_pytest.assertion', '_pytest.assertion.rewrite', '_pytest.assertion.truncate', '_pytest.cacheprovider', '_pytest.capture', '_pytest.debugging', ...}
        undeps     = set()
        viable     = set()
/home/matti/miniconda3/envs/numpy/lib/python3.8/site-packages/mypy/build.py:3066: in process_stale_scc
    graph[id].write_cache()
        graph      = {'__future__': <mypy.build.State object at 0x7f4c4a49a220>, '_ast': <mypy.build.State object at 0x7f4cd428c760>, '_bisect': <mypy.build.State object at 0x7f4c4dde50a0>, '_compression': <mypy.build.State object at 0x7f4c6996e850>, ...}
        id         = 'numpy.core.numeric'
        manager    = <mypy.build.BuildManager object at 0x7f4c3cced3d0>
        scc        = ['_pytest.config.findpaths', '_pytest.hookspec', 'numpy.f2py.cfuncs', 'numpy.f2py.diagnose', 'numpy.distutils.ccompiler_opt', 'numpy.distutils.misc_util', ...]
        stale      = ['_pytest.config.findpaths', '_pytest.hookspec', 'numpy.f2py.cfuncs', 'numpy.f2py.diagnose', 'numpy.distutils.ccompiler_opt', 'numpy.distutils.misc_util', ...]
        unfinished_modules = set()
/home/matti/miniconda3/envs/numpy/lib/python3.8/site-packages/mypy/build.py:2228: in write_cache
    new_interface_hash, self.meta = write_cache(
        dep_lines  = [1, 2, 3, 4, 5, 6, ...]
        dep_prios  = [10, 10, 10, 10, 10, 10, ...]
        is_errors  = False
        self       = <mypy.build.State object at 0x7f4c5187cf70>
/home/matti/miniconda3/envs/numpy/lib/python3.8/site-packages/mypy/build.py:1442: in write_cache
    data = tree.serialize()
        _          = None
        bazel      = False
        data_json  = 'numpy/core/numeric.data.json'
        dep_lines  = [1, 2, 3, 4, 5, 6, ...]
        dep_prios  = [10, 10, 10, 10, 10, 10, ...]
        dependencies = ['functools', 'itertools', 'operator', 'sys', 'warnings', 'numbers', ...]
        id         = 'numpy.core.numeric'
        ignore_all = False
        manager    = <mypy.build.BuildManager object at 0x7f4c3cced3d0>
        meta_json  = 'numpy/core/numeric.meta.json'
        metastore  = <mypy.metastore.FilesystemMetadataStore object at 0x7f4c3ccedb80>
        old_interface_hash = ''
        path       = '/home/matti/pypy_stuff/numpy/build/testenv/lib/python3.8/site-packages/numpy/core/numeric.py'
        source_hash = '51a4ba28d5e7135880adef8218e8ef5940231ba9a777f15bad925ea78daa5fa5'
        suppressed = []
        tree       = <mypy.nodes.MypyFile object at 0x7f4c43450f90>
/home/matti/miniconda3/envs/numpy/lib/python3.8/site-packages/mypy/nodes.py:302: in serialize
    'names': self.names.serialize(self._fullname),
        self       = <mypy.nodes.MypyFile object at 0x7f4c43450f90>
/home/matti/miniconda3/envs/numpy/lib/python3.8/site-packages/mypy/nodes.py:3097: in serialize
    data[key] = value.serialize(fullname, key)
        data       = {'.class': 'SymbolTable', 'ALLOW_THREADS': {'.class': 'SymbolTableNode', 'kind': 'Gdef', 'node': {'.class': 'Var', 'fl...'flags': ['is_suppressed_import', 'is_ready'], 'fullname': 'numpy.core.numeric.BUFSIZE', 'name': 'BUFSIZE', ...}}, ...}
        fullname   = 'numpy.core.numeric'
        key        = 'number'
        self       = {'__builtins__': <mypy.nodes.SymbolTableNode object at 0x7f4c4b791b20>, '__name__': <mypy.nodes.SymbolTableNode object...y.nodes.SymbolTableNode object at 0x7f4cba8b89a0>, 'geterrcall': <mypy.nodes.SymbolTableNode object at 0x7f4cba8b8a00>}
        value      = <mypy.nodes.SymbolTableNode object at 0x7f4cba8cf220>
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <mypy.nodes.SymbolTableNode object at 0x7f4cba8cf220>, prefix = 'numpy.core.numeric', name = 'number'

    def serialize(self, prefix: str, name: str) -> JsonDict:
        """Serialize a SymbolTableNode.
    
        Args:
          prefix: full name of the containing module or class; or None
          name: name of this object relative to the containing object
        """
        data = {'.class': 'SymbolTableNode',
                'kind': node_kinds[self.kind],
                }  # type: JsonDict
        if self.module_hidden:
            data['module_hidden'] = True
        if not self.module_public:
            data['module_public'] = False
        if self.implicit:
            data['implicit'] = True
        if self.plugin_generated:
            data['plugin_generated'] = True
        if isinstance(self.node, MypyFile):
            data['cross_ref'] = self.node.fullname
        else:
            assert self.node is not None, '%s:%s' % (prefix, name)
            if prefix is not None:
                fullname = self.node.fullname
                if (fullname is not None and '.' in fullname
                        and fullname != prefix + '.' + name
                        and not (isinstance(self.node, Var)
                                 and self.node.from_module_getattr)):
>                   assert not isinstance(self.node, PlaceholderNode)
E                   AssertionError

data       = {'.class': 'SymbolTableNode', 'kind': 'Gdef', 'module_public': False}
fullname   = 'numpy.core.fromnumeric.number'
name       = 'number'
prefix     = 'numpy.core.numeric'
self       = <mypy.nodes.SymbolTableNode object at 0x7f4cba8cf220>

/home/matti/miniconda3/envs/numpy/lib/python3.8/site-packages/mypy/nodes.py:3033: AssertionError

@BvB93
Copy link
Member Author

BvB93 commented Sep 26, 2020

Does removing the sub-module imports change anything?

numpy/numpy/__init__.pyi

Lines 44 to 60 in c7aa92c

# Ensures that the stubs are picked up
from . import (
char,
compat,
ctypeslib,
emath,
fft,
lib,
linalg,
ma,
matrixlib,
polynomial,
random,
rec,
testing,
version,
)

@charris
Copy link
Member

charris commented Sep 26, 2020

No, still fails. Just to be clear, what command are we testing now?

@BvB93
Copy link
Member Author

BvB93 commented Sep 26, 2020

No, still fails. Just to be clear, what command are we testing now?

The python runtests.py -t numpy/typing/tests command, as the other one appears to be working?
Still, I'm at a loss what's triggering the issue and why it so incredibly inconsistent...

@charris
Copy link
Member

charris commented Sep 27, 2020

@BvB93 Do you have any related environmental variables set?

@BvB93
Copy link
Member Author

BvB93 commented Sep 28, 2020

@BvB93 Do you have any related environmental variables set?

No, none whatsoever.

@BvB93
Copy link
Member Author

BvB93 commented Sep 28, 2020

Continuing the theme of removing imports:
Does removing the imports from numpy.core have any effect?

numpy/numpy/__init__.pyi

Lines 62 to 154 in c7aa92c

from numpy.core.function_base import (
linspace,
logspace,
geomspace,
)
from numpy.core.fromnumeric import (
take,
reshape,
choose,
repeat,
put,
swapaxes,
transpose,
partition,
argpartition,
sort,
argsort,
argmax,
argmin,
searchsorted,
resize,
squeeze,
diagonal,
trace,
ravel,
nonzero,
shape,
compress,
clip,
sum,
all,
any,
cumsum,
ptp,
amax,
amin,
prod,
cumprod,
ndim,
size,
around,
mean,
std,
var,
)
# Add an object to `__all__` if their stubs are defined in an external file;
# their stubs will not be recognized otherwise.
# NOTE: This is redundant for objects defined within this file.
__all__ = [
"linspace",
"logspace",
"geomspace",
"take",
"reshape",
"choose",
"repeat",
"put",
"swapaxes",
"transpose",
"partition",
"argpartition",
"sort",
"argsort",
"argmax",
"argmin",
"searchsorted",
"resize",
"squeeze",
"diagonal",
"trace",
"ravel",
"nonzero",
"shape",
"compress",
"clip",
"sum",
"all",
"any",
"cumsum",
"ptp",
"amax",
"amin",
"prod",
"cumprod",
"ndim",
"size",
"around",
"mean",
"std",
"var",
]

@charris
Copy link
Member

charris commented Sep 28, 2020

That helps:

FAILED numpy/typing/tests/test_typing.py::test_fail[fromnumeric.py] - assert 0 != 0
FAILED numpy/typing/tests/test_typing.py::test_fail[linspace.py] - assert 0 != 0
FAILED numpy/typing/tests/test_typing.py::test_reveal[fromnumeric.py] - assert 'numpy.bool_' in "numpy/typing/tests/data/reveal/fromn...
FAILED numpy/typing/tests/test_typing.py::test_reveal[linspace.py] - assert 'numpy.ndarray' in "numpy/typing/tests/data/reveal/linspa...
4 failed, 47 passed, 1 warning in 48.68s

@BvB93
Copy link
Member Author

BvB93 commented Sep 29, 2020

The errors are to be expected, considering a number of functions were suddenly ripped out of the main numpy space.

Still, while it is good to know that the problem can be fixed, the required solution is less then ideal: moving the functions defined in sub-modules (back) to the main __init__.pyi file. I fear this will have an averse affect on the maintainability due to the files' sheer size, especially as more and more functions will be added...

Would it be an idea to be bring this up in Wednesday's meeting?

@BvB93
Copy link
Member Author

BvB93 commented Sep 30, 2020

Closing this per our discussion in todays' meeting, i.e. mark the typing tests as slow as a band aid for avoiding (local) mypy crashes.
I'll create a follow up either tomorrow or a bit later this week.

@BvB93 BvB93 closed this Sep 30, 2020
@mattip
Copy link
Member

mattip commented Sep 30, 2020

Let's leave this open until the PR is merged. You can make that automatic by putting something like closes gh-17376 in the PR.

@mattip mattip reopened this Sep 30, 2020
@BvB93
Copy link
Member Author

BvB93 commented Sep 30, 2020

Let's leave this open until the PR is merged. You can make that automatic by putting something like closes gh-17376 in the PR.

That's pretty neat actually, I never realized you could close a PR with another PR.

@mattip
Copy link
Member

mattip commented Sep 30, 2020

Whoops, my bad. I thought this was an issue, not a PR. No, that won't work. Closing again.

@mattip mattip closed this Sep 30, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

mypy test errors when using python runtests.py
3 participants