crash not detected by tools because blocking on a socket #6
Description
when client crashes and waits in gdb, some call from the monitoring thread might block on a rpc socket call -> need to add a timeout to local rpc calls too:
(crash happened ~ 12:52)
2014-10-02 13:08:25,101 WARNING [bitshares_delegate_tools.rpcutils:get_streak:195] -- get_streak() failed with: <class 'ValueError'>(Unterminated string starting at: line 1 column 656634 (char 656633))
2014-10-02 13:08:25,102 ERROR [bitshares_delegate_tools.rpcutils:get_streak:196] -- Unterminated string starting at: line 1 column 656634 (char 656633)
Traceback (most recent call last):
File "/home/admin/.virtualenvs/bitshares_delegate_tools/lib/python3.3/site-packages/bitshares_delegate_tools/rpcutils.py", line 187, in get_streak
cached=cached)[::-1]
File "/home/admin/.virtualenvs/bitshares_delegate_tools/lib/python3.3/site-packages/bitshares_delegate_tools/rpcutils.py", line 130, in call
return self.rpc_call(funcname, *args, cached=cached)
File "/home/admin/.virtualenvs/bitshares_delegate_tools/lib/python3.3/site-packages/bitshares_delegate_tools/rpcutils.py", line 147, in rpc_call
result = self._rpc_call(funcname, *args)
File "/home/admin/.virtualenvs/bitshares_delegate_tools/lib/python3.3/site-packages/bitshares_delegate_tools/rpcutils.py", line 94, in local_call
funcname, *args)
File "/home/admin/.virtualenvs/bitshares_delegate_tools/lib/python3.3/site-packages/bitshares_delegate_tools/rpcutils.py", line 70, in rpc_call
r = response.json()
File "/home/admin/.virtualenvs/bitshares_delegate_tools/lib/python3.3/site-packages/requests/models.py", line 756, in json
return json.loads(self.content.decode(encoding), **kwargs)
File "/usr/lib/python3.3/json/__init__.py", line 316, in loads
return _default_decoder.decode(s)
File "/usr/lib/python3.3/json/decoder.py", line 351, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/usr/lib/python3.3/json/decoder.py", line 367, in raw_decode
obj, end = self.scan_once(s, idx)
ValueError: Unterminated string starting at: line 1 column 656634 (char 656633)
2014-10-02 13:08:25,193 ERROR [bitshares_delegate_tools.monitor:monitoring_thread:173] -- An exception occurred in the monitoring thread:
2014-10-02 13:08:25,193 ERROR [bitshares_delegate_tools.monitor:monitoring_thread:174] -- 'NoneType' object has no attribute 'cpu_percent'
Traceback (most recent call last):
File "/home/admin/.virtualenvs/bitshares_delegate_tools/lib/python3.3/site-packages/bitshares_delegate_tools/monitor.py", line 155, in monitoring_thread
s = StatsFrame(cpu=p.cpu_percent(),
AttributeError: 'NoneType' object has no attribute 'cpu_percent'
2014-10-02 13:08:40,259 WARNING [bitshares_delegate_tools.monitor:monitoring_thread:95] -- Delegate just went offline...
2014-10-02 13:08:41,632 INFO [bitshares_delegate_tools.notification:send_notification_email:60] -- Done sending email notification: Delegate just went offline...
2014-10-02 13:08:51,100 INFO [bitshares_delegate_tools.notification:send_notification_apns:114] -- Done sending APNS notification: Delegate just went offline...