make io.pyi import everything from _io.py(i), like io.py does #1395

matthiaskramm · 2017-06-07T02:21:53Z

No description provided.

Also, as far as possible, try to simplify, by moving methods into base classes.

matthiaskramm · 2017-06-10T00:06:50Z

This PR got larger than expected. The basic motivation is to allow code like

io.BufferedReader(fp)

to pass type-checking. (Which it didn't before, since io didn't have a BufferedReader)

gvanrossum · 2017-06-10T01:22:38Z

Do you have code that directly imports _io? If not, maybe we should get rid of it completely, like we did in the PY3 stubs.

Anyway, I tested this without our internal code and it found one issue, io.BufferedIOBase is supposed to define __enter__ and __exit__.

I am also still worried about some classes not inheriting from BinaryIO where they should. BinaryIO is abstract, we must ensure the abstractness or concreteness of various classes matches those in the stdlib.

matthiaskramm · 2017-06-12T15:59:46Z

Do you have code that directly imports _io?

We do, unfortunately. Some in internal code, but I'm also seeing occurrences in third party packages like pymock and astroid.

I am also still worried about some classes not inheriting from BinaryIO where they should. BinaryIO is abstract, we must ensure the abstractness or concreteness of various classes matches those in the stdlib.

Right. Made _IOBase inherit from BinaryIO. (Note that _TextIOBase inherits from _IOBase, too, but is a TextIO. So you have a class that inherits both from BinaryIO and TextIO, which is a bit awkward. But that's also how it was modelled before this PR.)

gvanrossum · 2017-06-12T16:25:34Z

Seeing all the # type: ignore lines I wonder if it wouldn't be better to ignore the runtime/implementation inheritance between _TextIOBase and _IOBase and just claim (to the type system) that _TextIOBase doesn't inherit from _IOBase? Or perhaps construct a new fictitious class/type that represents the common methods like seek()?

matthiaskramm · 2017-06-12T17:58:01Z

I don't see why not. Let me look into it.

JelleZijlstra

Thanks for all the cleanup! I left some comments from comparing this with the C code and (mostly) the docs.

JelleZijlstra · 2017-06-13T03:26:45Z

stdlib/2/_io.pyi

    def seek(self, offset: int, whence: int = ...) -> int: ...
    def seekable(self) -> bool: ...
    def tell(self) -> int: ...
    def truncate(self, size: Optional[int] = ...) -> int: ...
    def writable(self) -> bool: ...
-    def write(self, s: bytes) -> int: ...
+    def __exit__(self, type, value, traceback) -> bool: ...
+    # This just returns self:


Then why not use a self type?

JelleZijlstra · 2017-06-13T03:27:28Z

stdlib/2/_io.pyi

    def seek(self, offset: int, whence: int = ...) -> int: ...
    def seekable(self) -> bool: ...
    def tell(self) -> int: ...
    def truncate(self, size: Optional[int] = ...) -> int: ...
    def writable(self) -> bool: ...
-    def write(self, s: bytes) -> int: ...
+    def __exit__(self, type, value, traceback) -> bool: ...


Why did you remove the argument types here?

My bad. The types got under the wheel when I shifted __exit__ back and fro.

That said, I'm not sure what we get out of having types on __exit__. This function is invoked by the type-checker, not user code (nobody ever does a manual x.__exit__(a, b,c)), and the parameters are not originating from user code, either.

Either way, I added types.

JelleZijlstra · 2017-06-13T03:28:16Z

stdlib/2/_io.pyi

    def detach(self) -> "_BufferedIOBase": ...
+    def __enter__(self) -> '_BufferedIOBase': ...


Looks odd to have double quotes on one line and single on the next. Since this is a stub, we can just remove all quotes anyway.

JelleZijlstra · 2017-06-13T03:29:18Z

stdlib/2/_io.pyi


-class FileIO(_RawIOBase):
+class FileIO(_RawIOBase, BytesIO):  # type: ignore  # for __enter__


Would this not be necessary if you used self types for these __enter__ methods?

JelleZijlstra · 2017-06-13T03:30:59Z

stdlib/2/io.pyi

@@ -7,98 +7,28 @@
 from typing import List, BinaryIO, TextIO, IO, overload, Iterator, Iterable, Any, Union, Optional
 import _io

-DEFAULT_BUFFER_SIZE = 0
+from _io import (DEFAULT_BUFFER_SIZE, BlockingIOError, UnsupportedOperation,


These need x as x imports to make clear that they're re-exported.

JelleZijlstra · 2017-06-13T03:47:42Z

stdlib/2/_io.pyi


 class BufferedRWPair(_BufferedIOBase):
-    def peek(self, n: int = ...) -> str: ...
+    def __init__(self, reader: _RawIOBase, writer: _RawIOBase) -> None: ...


This also takes buffer_size and max_buffer_size arguments.

JelleZijlstra · 2017-06-13T03:48:17Z

stdlib/2/_io.pyi

-class _TextIOBase(_IOBase):
+
+# Note: In the actual _io.py, _TextIOBase inherits from _IOBase.
+class _TextIOBase(TextIO):
    errors = ...  # type: Optional[str]
    newlines = ...  # type: Union[str, unicode]


According to the docs it can also be None or a tuple of strings.

Actually, on _TextIOBase, this always returns None:

https://github.com/python/cpython/blob/2.7/Modules/_io/textio.c#L104

overloaded below.

JelleZijlstra · 2017-06-13T03:49:21Z

stdlib/2/_io.pyi

+    def _checkSeekable(self) -> None: ...
+    def _checkWritable(self) -> None: ...
+    def close(self) -> None: ...
+    def detach(self) -> None: ...


This returns the underlying BufferedIOBase, not None.

Had to make this IO, since _TextIOBase doesn't inherit from BufferedIOBase, and the underlying stream might be either.
buffer is IO below, too.

JelleZijlstra · 2017-06-13T03:50:05Z

stdlib/2/_io.pyi

    _CHUNK_SIZE = ...  # type: int
-
-def open(file: Union[int, str], mode: str = ...) -> _IOBase: ...
+    def __init__(self, buffer: IO, encoding: unicode = ...,


I think this should be IO[bytes].

No, this works:

_io.TextIOWrapper(_io.StringIO(u"hello world")).read()

JelleZijlstra · 2017-06-13T03:50:37Z

stdlib/2/_io.pyi

-
-def open(file: Union[int, str], mode: str = ...) -> _IOBase: ...
+    def __init__(self, buffer: IO, encoding: unicode = ...,
+                 errors: unicode = ..., newline: unicode = ...,


Further up you had encoding and errors attributes as str on _TextIOBase.

Done. These parameters are actually more permissive. They allow None, as well.

matthiaskramm · 2017-06-13T15:34:45Z

Thanks for the thorough review!

matthiaskramm · 2017-06-13T22:21:19Z

Is there a way to tell github that (you think) you addressed all the requested changes on a PR? It seems we occasionally have dangling PRs because it's not clear whether the author is still working on it or not.

gvanrossum · 2017-06-13T22:27:31Z

I don't think so, the convention we follow is for the contributor to send a comment to the PR saying they're done addressing the feedback.

JelleZijlstra · 2017-06-13T22:30:08Z

That's also something I've missed in GitHub's workflow (similar to "request review" in Phabricator).

I'll take another look at this PR in a few days.

gvanrossum · 2017-06-13T22:37:13Z

Well, "request review" exists -- it's just that "request re-review" doesn't. :-)

gvanrossum

I personally think this is good to go, but I'll leave it up to @JelleZijlstra to approve and merge.

JelleZijlstra · 2017-06-22T03:04:32Z

I guess I was stretching the definition of "a few days" :)

Thanks!

make io.pyi import _io.py(i), like io.py does

57e0a21

matthiaskramm force-pushed the io branch from 2652b1e to 57e0a21 Compare June 7, 2017 02:22

matthiaskramm added 2 commits June 6, 2017 19:27

make write/writelines take 'Any', on _IOBase

34ce366

fix mypy warnings

44371bb

matthiaskramm force-pushed the io branch from fae720e to 44371bb Compare June 9, 2017 18:42

matthiaskramm added 2 commits June 9, 2017 14:37

Add missing constructors, fix inconsistencies.

0acb51a

Also, as far as possible, try to simplify, by moving methods into base classes.

fix lint+mypy warnings

58cc0ae

matthiaskramm force-pushed the io branch from 9d44c36 to 58cc0ae Compare June 9, 2017 22:29

matthiaskramm changed the title ~~WIP: make io.pyi import everything from _io.py(i), like io.py does~~ make io.pyi import everything from _io.py(i), like io.py does Jun 10, 2017

add missing __enter__ methods

7dc8830

matthiaskramm force-pushed the io branch from 568659c to 7dc8830 Compare June 12, 2017 15:49

make _IOBase inherit from BinaryIO

4388527

make _TextIOBase not subclass _IOBase

ffff66a

JelleZijlstra requested changes Jun 13, 2017

View reviewed changes

address review comments

b75b47a

remove trailing whitespace

986a33f

gvanrossum approved these changes Jun 21, 2017

View reviewed changes

JelleZijlstra merged commit c4e0580 into master Jun 22, 2017

JelleZijlstra deleted the io branch June 22, 2017 03:04

gvanrossum mentioned this pull request Jun 23, 2017

Make HTTPError instantiable. #1379

Closed

JelleZijlstra mentioned this pull request Jul 2, 2017

io classes incomplete in Python 2 #694

Closed

		def detach(self) -> "_BufferedIOBase": ...
		def __enter__(self) -> '_BufferedIOBase': ...


		class FileIO(_RawIOBase):
		class FileIO(_RawIOBase, BytesIO): # type: ignore # for __enter__

Uh oh!

make io.pyi import everything from _io.py(i), like io.py does #1395

make io.pyi import everything from _io.py(i), like io.py does #1395

Uh oh!

Conversation

matthiaskramm commented Jun 7, 2017

Uh oh!

matthiaskramm commented Jun 10, 2017

Uh oh!

gvanrossum commented Jun 10, 2017

Uh oh!

matthiaskramm commented Jun 12, 2017

Uh oh!

gvanrossum commented Jun 12, 2017

Uh oh!

matthiaskramm commented Jun 12, 2017

Uh oh!

JelleZijlstra left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

matthiaskramm commented Jun 13, 2017

Uh oh!

matthiaskramm commented Jun 13, 2017

Uh oh!

gvanrossum commented Jun 13, 2017 via email

Uh oh!

JelleZijlstra commented Jun 13, 2017

Uh oh!

gvanrossum commented Jun 13, 2017

Uh oh!

gvanrossum left a comment

Choose a reason for hiding this comment

Uh oh!

JelleZijlstra commented Jun 22, 2017

Uh oh!

Uh oh!