Delay closing SubprocessStreamProtocol's Transport until all pipes are closed. #485
Conversation
Delays the closing for `SubprocessStreamProtocol._transport` until all pipes have been closed and the process has exited rather than just the process exiting. This allows for reading data from the subprocess that is still pending in the pipe after the process has exited.
|
As pretty much the sole author of |
|
cc @asvetlov. Although without a test there's nothing to review :) |
|
@1st1 Yes that's true, it'd be nice to get a nod to say this is actually a sane thing to do though. Doesn't break any other tests but I don't know what other untested assumptions I might be breaking! :P |
It looks sane. Asking for a review without tests isn't productive though, because you're essentially asking someone to stop doing what they are doing, understand (or remember, doesn't matter) your patch and all code that it touches, and make an opinion if your patch works. PRs without tests are usually reviewed the last if at all. |
|
Sorry, won't do it again. I added a simple test that fails on old build and passes with the proposed change. I presume it's okay to have a test that doesn't assert anything, just to make sure that no errors are thrown? The test is taken pretty much 1-to-1 with the attached issue. I'm sure I can come up with a few more tests if more tests are required. :) |
|
I can confirm that this pull request fixes my issue #484 |
asyncio/subprocess.py
Outdated
| def _maybe_close_transport(self, fd): | ||
| if fd in self._pipe_fds: | ||
| self._pipe_fds.remove(fd) | ||
| if len(self._pipe_fds) == 0: |
There was a problem hiding this comment.
Shouldn't this if len(...) be dedented?
There was a problem hiding this comment.
I guess that's true. My logic is we know that if nothing gets removed it won't be closed.
...removal from fd list does not occur.
asyncio/subprocess.py
Outdated
| @@ -24,6 +24,7 @@ def __init__(self, limit, loop): | |||
| self._limit = limit | |||
| self.stdin = self.stdout = self.stderr = None | |||
| self._transport = None | |||
| self._pipe_fds = [-1] | |||
There was a problem hiding this comment.
I should probably add a comment. -1 is just the token for "process_exited" meaning we won't close a Transport until the process exits and all the pipes are done.
There was a problem hiding this comment.
Should I use a better sentinel object than -1?
There was a problem hiding this comment.
Ah, interesting. I'd add a separate boolean attribute for that.
asyncio/subprocess.py
Outdated
| @@ -85,10 +88,17 @@ def pipe_connection_lost(self, fd, exc): | |||
| reader.feed_eof() | |||
| else: | |||
| reader.set_exception(exc) | |||
| self._maybe_close_transport(fd) | |||
There was a problem hiding this comment.
I'd simply always call this method in pipe_connection_lost. Makes the code simpler to follow, and the logic with if statements should be (as they currently are) in _maybe_close_transport.
There was a problem hiding this comment.
Sure, just dedent that line then?
There was a problem hiding this comment.
Alright, just wanted to make sure I was understanding that comment correctly. :)
..._pipe_fds and change interface of _maybe_close_transport to not take an fd.
|
Can you please close this PR and make a new one to http://github.com/python/cpython (with a link to this discussion and review). It would make merging this way easier for me. |
|
So I made another change to |
|
@1st1 Oh, sure! Can I do this when I'm home from work? Don't have my actual environment all setup here, making do with the GitHub web interface. :) |
|
I like it now. Thanks for opening the new PR! I'll close this one. |
I believe this change fixes #484.
Delays the closing for
SubprocessStreamProtocol._transportuntil all pipes have been closed and the process has exited rather than just the process exiting. This allows for reading data from the subprocess that is still pending in the pipe after the process has exited.I haven't run tests or created a test for this exact issue but I will create a few tests later.