
Description
every few days, sometimes a week, one of our subscribers in a while True
loop waiting on future.result(timeout=1)
is getting excepted, exception string is None Stream removed
. this might happen maybe every thousand received messages (I can count better if it's needed).
originally we were on grpcio-1.8.1
, and after searching, we seemed to find some indication that this error might be solved by an RPC library upgrade. the reference is a totally different language/application but using the same google RPC library, they say upgrade to grpcio-1.8.4
fixed their issue. recent comment seems to indicate that the nodejs library would normally retry this RPC but wasn't getting the right return value, which grpcio-1.8.4
fixed for them.
moving to later grpcio-1.8.4
for us however did not help with the Python library, we still see this error.
we don't think our application should see this, the library should be retrying it based on the RPC result, just like they do in the referenced cloud client library.
environment is ubuntu16 on a gce node, relevant libs:
- grpcio: 1.8.4
- google-api-core: 0.1.4
- google-cloud-core: 0.28.0
- google-cloud-pubsub: 0.30.1