Thanks to visit codestin.com
Credit goes to github.com

Skip to content

Conversation

timvaillancourt
Copy link
Contributor

@timvaillancourt timvaillancourt commented Jun 19, 2016

Fix to resolve #8.

  • Parse output from Popen PIPE every poll to make sure the pipe buffer never fills, which stalls the shell command. Before we only checked after the command execution, which can easily fill the PIPE buffer to 100%. From the subprocess docs:

"Warning This will deadlock when using stdout=PIPE and/or stderr=PIPE and the child process generates enough output to a pipe such that it blocks waiting for the OS pipe buffer to accept more data.

  • Moved Popen thread polling from 500ms down to 100ms so we empty the buffers often. The stall takes minutes/hours on a large deployment so this should be plenty.
  • Moved output to an array to facilitate the change to calling the output parser N times. Changed logging to accept an array for output and display it the same as before.

Logging output is unchanged for this PR to keep changes small, look out for a future PR for realtime thread logging.

Functional tests are passing with this change.

Tim Vaillancourt added 3 commits June 19, 2016 19:31
@dbmurphy dbmurphy merged commit 8f80732 into Percona-Lab:master Jun 20, 2016
@timvaillancourt timvaillancourt deleted the subprocess_popen_stall branch June 20, 2016 10:08
dbmurphy pushed a commit that referenced this pull request Feb 22, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Mongodump commands hang on larger data sets
2 participants