Add all the files which pytype fails on during import. #1720

x3k6a2 · 2017-11-08T12:53:29Z

All of these files can not be used with pytype at HEAD. They fail during import with various error messages.

matthiaskramm · 2017-11-08T14:06:45Z

tests/pytype_blacklist.txt

+stdlib/3/io.pyi
+stdlib/3/calendar.pyi
+stdlib/3/fcntl.pyi
+stdlib/3/gzip.pyi


How come we didn't have to blacklist these before? The existing travis tests do check the pyi parsing step.

This change blacklists all files which will kill PyType if actually used. The above listed files depend on stdlib/3/builtins.pyi which contains line 241:
"def center(self, width: int, fillchar: str = ' ') -> str: ..."

ParseError: File: "pytype/typeshed/stdlib/3/builtins.pyi", line 241
def center(self, width: int, fillchar: str = ' ') -> str: ...
^
ParseError: syntax error, unexpected ')', expecting NAME or NUMBER or ELLIPSIS

The travis tests just try to parse each file in isolation and can therefore not see this issue.

That particular error is because of the ' ' default, right? That seems easy to fix at least.

Created #1721 to change the defaults.

We don't actually want to use typeshed's builtins.pyi with pytype, though, since we have our own (__bulitin__.pytd). We're probably just missing a symlink builtins.pytd -> __builtin__.pytd in pytype/pytd/builtins.

It becomes a BadDependency Error once builtins.pyi is changed. As the pytype typing module does not have all the things needed.

gvanrossum · 2017-11-08T16:06:48Z

You can fix that in builtins.pyi though. Replace ' ' with ...

…

On Nov 8, 2017 7:58 AM, "Sebastian Steenbuck" ***@***.***> wrote: ***@***.**** commented on this pull request. ------------------------------ In tests/pytype_blacklist.txt <#1720 (comment)>: > # pytype doesn't yet support aliases with implicit type parameters # (e.g., here, FutureT = Future[T]) stdlib/3.4/asyncio/tasks.pyi + +# ParseError: syntax error, unexpected ')', expecting NAME or NUMBER or +# ELLIPSIS +stdlib/2and3/locale.pyi +stdlib/3/io.pyi +stdlib/3/calendar.pyi +stdlib/3/fcntl.pyi +stdlib/3/gzip.pyi This change blacklists all files which will kill PyType if actually used. The above listed files depend on stdlib/3/builtins.pyi which contains line 241: "def center(self, width: int, fillchar: str = ' ') -> str: ..." ParseError: File: "pytype/typeshed/stdlib/3/builtins.pyi", line 241 def center(self, width: int, fillchar: str = ' ') -> str: ... ^ ParseError: syntax error, unexpected ')', expecting NAME or NUMBER or ELLIPSIS The travis tests just try to parse each file in isolation and can therefore not see this issue. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#1720 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ACwrMlaU0IhWl3C7XY-ogUbaI4KCJSxiks5s0c-jgaJpZM4QWVW-> .

gvanrossum · 2017-11-08T16:54:00Z

Oh, BTW, we should probably get rid of symlinks, not add more. They don't work on Windows (at least, not without Administrator privileges). See python/mypy#4143 (comment)

JelleZijlstra · 2017-11-08T17:08:46Z

tests/pytype_blacklist.txt

@@ -1,13 +1,116 @@
-# Pytype blacklist. Files will not be tested with pytype.
+# Pytype blacklist. Files will not be tested with pytype. Notice that the parse
+# test will work for some of the files. Loading the files will fail, but is not


Is it possible to make typeshed's pytype tests also load these files? It's not ideal if the typeshed project has to maintain a blacklist file without a test to ensure the blacklist is correct.

I agree. This will take a few more days. The code path to easily test this is not yet in PyType's github repo.

Fixes a pytype issue from #1720.

This change will make the test script actually load the files and all dependencies of it, instead of just verifying that it can be parsed.

x3k6a2 · 2017-11-10T12:00:32Z

I updated the pytype_test script to use pytype instead of pytd and updated the blacklist accordingly.

This is not a perfect change, it would be nicer to actually test with Python 3.6 for the stdlib/3.X files but that is too much for this change and should be done in a different one.

x3k6a2 · 2017-11-10T15:08:29Z

A green travis run is at https://travis-ci.org/x3k6a2/typeshed/builds/300136808 . Afaik I can't request a rerun for the failed integration check.
This works now because of the pytype commit google/pytype@0964db4

JelleZijlstra · 2017-11-10T16:31:40Z

Just restarted the Travis build.

* don't import enum in Python 2 Fixes a pytype issue from #1720. * move enums together

This list was generated by running pytype with more options. 1: typeshed-location was pointed to the current typeshed instead of the one shipped with pytype. 2: --module-name=<foo> was provided as a parameter. Without this parameter wildcard imports "from _ast import *" misbehave. Files with a "# parse only" comment will only be parsed and not loaded.

This change adds the ability to either run pytype or pytd, based on a "# parse only" comment in pytype_blacklist.pytd

x3k6a2 · 2017-11-14T10:04:18Z

As discussed offline:
1: Added a "# parse only" comment to the 3.x files in the blacklist. Which will make those files be parsed but not loaded.
2: Checked all the stdlib/2.x/ files with exceptions. Most of the exceptions could be removed by providing the --module-name=foo flag. Without it wildcard imports are not handled correctly. The alias for ".*" is not removed by the LookupExternalTypes.VisitTypeDevlUnit visitor. This behavior is a bit surprising.
The only 2.x files added left to be added to the blacklist is os.init.pyi which imports builtins. Builtins is not defined for python2.

The files tries to load builtins which are not defined for python2.

* Replace " with ' everywhere were it was used. * Move the multi line parameters around to allow for something the linter will accept.

matthiaskramm · 2017-11-15T00:57:08Z

tests/pytype_test.py

        sys.exit(1)

+_PARSE_ONLY_REGEX = r'\s*#\s*parse only\s*'


Do the re.compile right here?

Obsolete, done in another place.

matthiaskramm · 2017-11-15T00:58:23Z

tests/pytype_test.py

+    parse_only_re = re.compile(_PARSE_ONLY_REGEX)
+    for f in lines:
+        if parse_only_re.search(f):
+            parse_only.append(f.split('#')[0].strip())


Make the regexp so that you can do

match = parse_only_re.match(f) if match: parse_only.append(match.group(1))

?

Refactored so that the regexes are independent of each other.

matthiaskramm · 2017-11-15T00:59:37Z

tests/pytype_test.py

-    wanted = re.compile(r"stdlib/.*\.pyi$")
-    skipped = re.compile("(%s)$" % "|".join(load_blacklist()))
-    files = []
+    blacklist, parse_only = load_blacklist()


"parse_only" is from the blacklist, too, so that variable name is a bit confusing.

Rename blacklist to skip?

matthiaskramm · 2017-11-15T01:01:05Z

tests/pytype_test.py

+                    ['pytype',
+                     '--typeshed-location=%s' % os.getcwd(),
+                     '--module-name=%s' % _get_module_name(f),
+                     '--convert-to-pickle=%s' % temp.name,


Do you need the temporary file? Or could you just use os.devnull?

matthiaskramm · 2017-11-15T01:01:52Z

tests/pytype_test.py

    return max_code, runs

+def _get_module_name(filename):


Could you move this above the location where it's called?

…s per list.

Add all the files which pytype fails on during import.

fba951e

All of these files can not be used with pytype at HEAD. They fail during import with various error messages.

matthiaskramm suggested changes Nov 8, 2017

View reviewed changes

Update pytype_blacklist.txt

98215f7

JelleZijlstra reviewed Nov 8, 2017

View reviewed changes

JelleZijlstra added a commit that referenced this pull request Nov 8, 2017

don't import enum in Python 2

003afa5

Fixes a pytype issue from #1720.

JelleZijlstra mentioned this pull request Nov 8, 2017

don't import enum in Python 2 #1724

Merged

x3k6a2 added 3 commits November 9, 2017 15:01

PyType will ignore the PreconditionErrors in the latest version.

7242b76

Update pytype_blacklist.txt

81f1d09

Update the pytype_test script to use pytype and not pytd

846102c

This change will make the test script actually load the files and all dependencies of it, instead of just verifying that it can be parsed.

Update pytype_test.py

88cccb7

matthiaskramm pushed a commit that referenced this pull request Nov 10, 2017

don't import enum in Python 2 (#1724)

97be335

* don't import enum in Python 2 Fixes a pytype issue from #1720. * move enums together

x3k6a2 added 2 commits November 14, 2017 10:55

Add a switch to run pytd or pytype

edc721a

This change adds the ability to either run pytype or pytd, based on a "# parse only" comment in pytype_blacklist.pytd

x3k6a2 added 5 commits November 14, 2017 11:06

Add parse only comment to stdlib/2/os/__init__.pyi

130e233

The files tries to load builtins which are not defined for python2.

Make indentation a multiple of four.

06dd71d

Make indentation a multiple of four.

52ae616

Fix two more linter issues.

e06d71c

Some more linting

8de94a3

* Replace " with ' everywhere were it was used. * Move the multi line parameters around to allow for something the linter will accept.

matthiaskramm reviewed Nov 15, 2017

View reviewed changes

matthiaskramm suggested changes Nov 15, 2017

View reviewed changes

x3k6a2 added 4 commits November 15, 2017 12:20

Use os.devnull instead of a temp file.

06a9ad9

Move _get_module_name above call location.

c24fb66

Rename blacklist to skip.

0c5d43a

Refactor the skip, parse_only list creation to use independent regexe…

ea9adbb

…s per list.

x3k6a2 added 2 commits November 15, 2017 12:41

Remove unused tempfile import.

6946a42

Make whitespaces a multiple of four

9206008

matthiaskramm merged commit 8b56c1e into python:master Nov 15, 2017

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add all the files which pytype fails on during import. #1720

Add all the files which pytype fails on during import. #1720

x3k6a2 commented Nov 8, 2017

matthiaskramm Nov 8, 2017

x3k6a2 Nov 8, 2017

JelleZijlstra Nov 8, 2017

x3k6a2 Nov 8, 2017 •

edited

Loading

matthiaskramm Nov 8, 2017

x3k6a2 Nov 8, 2017

gvanrossum commented Nov 8, 2017 via email

gvanrossum commented Nov 8, 2017 via email

JelleZijlstra Nov 8, 2017

x3k6a2 Nov 9, 2017

x3k6a2 commented Nov 10, 2017

x3k6a2 commented Nov 10, 2017

JelleZijlstra commented Nov 10, 2017

x3k6a2 commented Nov 14, 2017

matthiaskramm Nov 15, 2017

x3k6a2 Nov 15, 2017

matthiaskramm Nov 15, 2017

x3k6a2 Nov 15, 2017

matthiaskramm Nov 15, 2017

x3k6a2 Nov 15, 2017

matthiaskramm Nov 15, 2017

x3k6a2 Nov 15, 2017

matthiaskramm Nov 15, 2017

x3k6a2 Nov 15, 2017

Add all the files which pytype fails on during import. #1720

Add all the files which pytype fails on during import. #1720

Conversation

x3k6a2 commented Nov 8, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

x3k6a2 Nov 8, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gvanrossum commented Nov 8, 2017 via email

gvanrossum commented Nov 8, 2017 via email

Choose a reason for hiding this comment

Choose a reason for hiding this comment

x3k6a2 commented Nov 10, 2017

x3k6a2 commented Nov 10, 2017

JelleZijlstra commented Nov 10, 2017

x3k6a2 commented Nov 14, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

x3k6a2 Nov 8, 2017 •

edited

Loading