-
Notifications
You must be signed in to change notification settings - Fork 553
Socket recv handling #1605
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Socket recv handling #1605
Conversation
|
We need to decide what to do when the socket is closed but < the expected number of bytes is received. |
|
I don't have much input for this, just wanted to note that there's two more places in our code base with a
Also, this might even be considered a bug fix? (Which would qualify it for +TESTS:clients.earthworm,clients.seedlink (to make Travis run those network modules' tests as well) |
|
@megies: I checked those clients (neic, seedlinkconnection) but it appeared they are using nonblocking sockets, which should generate an exception instead of fail silently. When I put in the pr, I checked but didn't find the maintenance branch. Let me know if you need me to update the pr. |
OK. Also, there should be a tick box for you on this page to the right
Alright. Like I said, I'm no expert on sockets so I'll leave this to @QuLogic and @krischer and whoever feels qualified to review this. |
|
Looks good to me and I don't think the change can have any negative side effects! Thanks a lot! I guess the memory leak happened because it kept on looping and adding empty things to the Can you please make your changes to the
It will probably just raise some kind of parsing error later down the line which is fine IMHO. |
7a33c1e to
701bdb6
Compare
|
Rebasing ended up being a bad idea... Changes updated to start from maintenance_1.0.x. |
|
Yea rebasing does not work here. You could have cherry picked the commits on top of a branch off of maintenance. But whatever you did - looks good now :-) |
|
Can be merged depending on what CI says. |
|
@megies: Are the seedlink and earthworm tests supposed to be executed everywhere? I don't see them anywhere. |
|
@krischer, no, just Travis picks that up currently (see #1408 (comment)). My Windows-fu wasn't up to adding it to Appveyor too, and the docker scripts are pretty messy already, so I didn't want to add more mess on top. Now, don't ask me why Travis didn't run this PR.. (maybe it got confused by the rebase) |
|
I see. I'm merging in any case - I'm fairly sure this works based on the changes + the socket module documentation. Additionally I tested it locally for Python 2 and 3. @jmfee-usgs: Thanks for this! |
|
I ran this PR through our docker images, looking good. 👍 http://tests.obspy.org/?git=701bdb68fd&node=docker- Thanks @jmfee-usgs! |
What does this PR do?
Check the return value from socket.recv before buffering the (empty) response and looping forever.
Why was it initiated? Any relevant Issues?
I couldn't find any existing issues for this specific problem.
On a heavily loaded Scientific Linux 7 (RHEL flavor) system, we run many concurrent processes using the earthworm client to access data. Intermittently (<1%, but every few hours) the process would spike CPU and leak memory until the kernel killed the process because the system ran out of memory. Logs indicated this was while fetching data using the earthworm obspy client. Using a copy of the earthworm client with these modifications has thus far eliminated this leak.
From https://docs.python.org/2/howto/sockets.html#using-a-socket
PR Checklist
This is not reproducible under normal conditions.
CHANGELOG.txt.CONTRIBUTORS.txt.