BrokenPipeErrors while using datastore (dev) #1811
Description
We've recently started to see more BrokenPipeErrors while using gcloud-python. We started noticing them more when we upgraded to Kubernetes 1.2.4 (vs 1.1), if that makes any difference. But, I think this was just a coincidence.
Anyway, I delved deeper and saw that the dev datastore version was sending FIN ACK, and then RST ACK packets after some inactivity. The datastore client machine sees these packets as well. The next time the client tries to do a query against datastore, we get a broken pipe error. It would be nice if the gcloud-python / httplib2 did some pre-emptive checking to see if the connection was closed retried the query itself.
Some code snippets:
This is one way to detect if the socket on the other side has closed a connection.
possibly_disconnected = False
data_waiting = bytes()
try:
data_waiting = conn.recv(BUFFER_SIZE, socket.MSG_DONTWAIT|socket.MSG_PEEK)
except BlockingIOError:
# There is no data available... and the socket still seems good
pass
else:
possibly_disconnected = len(data_waiting) == 0
Here is a stack trace of the exception
File "/root/pyenvs/my_env/lib/python3.5/site-packages/gcloud/datastore/client.py", line 250, in get
deferred=deferred)
File "/root/pyenvs/my_env/lib/python3.5/site-packages/gcloud/datastore/client.py", line 291, in get_multi
transaction_id=transaction and transaction.id,
File "/root/pyenvs/my_env/lib/python3.5/site-packages/gcloud/datastore/client.py", line 123, in _extended_lookup
transaction_id=transaction_id,
File "/root/pyenvs/my_env/lib/python3.5/site-packages/gcloud/datastore/connection.py", line 203, in lookup
_datastore_pb2.LookupResponse)
File "/root/pyenvs/my_env/lib/python3.5/site-packages/gcloud/datastore/connection.py", line 121, in _rpc
data=request_pb.SerializeToString())
File "/root/pyenvs/my_env/lib/python3.5/site-packages/gcloud/datastore/connection.py", line 93, in _request
method='POST', headers=headers, body=data)
File "/root/pyenvs/my_env/lib/python3.5/site-packages/oauth2client/client.py", line 621, in new_request
redirections, connection_type)
File "/root/pyenvs/my_env/lib/python3.5/site-packages/httplib2/__init__.py", line 1314, in request
(response, content) = self._request(conn, authority, uri, request_uri, method, body, headers, redirections, cachekey)
File "/root/pyenvs/my_env/lib/python3.5/site-packages/httplib2/__init__.py", line 1064, in _request
(response, content) = self._conn_request(conn, request_uri, method, body, headers)
File "/root/pyenvs/my_env/lib/python3.5/site-packages/httplib2/__init__.py", line 988, in _conn_request
conn.request(method, request_uri, body, headers)
File "/usr/local/lib/python3.5/http/client.py", line 1083, in request
self._send_request(method, url, body, headers)
File "/usr/local/lib/python3.5/http/client.py", line 1128, in _send_request
self.endheaders(body)
File "/usr/local/lib/python3.5/http/client.py", line 1079, in endheaders
self._send_output(message_body)
File "/usr/local/lib/python3.5/http/client.py", line 911, in _send_output
self.send(msg)
File "/usr/local/lib/python3.5/http/client.py", line 885, in send
self.sock.sendall(data)
BrokenPipeError: [Errno 32] Broken pipe