Closed
Description
Hey guys,
I am seeing the pubsub client receive a 503 unavailable and instead of retrying it just exits. And it can't receive any more messages. The subscriber starts and works for a few minutes before throwing the error. I can reproduce it consistently with multiple different pubsub subscriptions.
Package: google-cloud-pubsub==1.4.2
OS: Mac Catalina, also tried on an Ubuntu VM.
Python: Python 3.7.3
Here is an example test case where I see the issue followed by a stack trace:
import os
import json
from time import sleep
from datetime import datetime
import logging
from google.cloud import pubsub_v1
from config import Config
logging.basicConfig()
logging.getLogger().setLevel(logging.DEBUG)
def _callback(message):
try:
print('message', message)
except Exception:
print('exception')
if __name__ == '__main__':
subscription = pubsub_v1.SubscriberClient()
subscription_path = subscription.subscription_path(
Config.PROJECT_ID,
Config.PUBSUB_SUB_NAME)
future = subscription.subscribe(
subscription_path,
_callback,
pubsub_v1.types.FlowControl(max_messages=Config.MAX_MESSAGES))
try:
future.result()
except KeyboardInterrupt:
future.cancel()
logging.error("Keyboard Interrupt")
except Exception:
raise
And stack trace -
DEBUG:google.cloud.pubsub_v1.subscriber._protocol.leaser:Snoozing lease management for 8.538204 seconds.
DEBUG:google.api_core.bidi:Thread-ConsumeBidirectionalStream caught error 503 The service was unable to fulfill your request. Please try again. [code=8a75] and will exit. Generally this is due to the RPC itself being cancelled and the error will be surfaced to the calling code.
Traceback (most recent call last):
File "/Users/tameem/pubsub-test/env/lib/python3.7/site-packages/google/api_core/grpc_helpers.py", line 144, in error_remapped_callable
return _StreamingResponseIterator(result)
File "/Users/tameem/pubsub-test/env/lib/python3.7/site-packages/google/api_core/grpc_helpers.py", line 72, in __init__
self._stored_first_result = six.next(self._wrapped)
File "/Users/tameem/pubsub-test/env/lib/python3.7/site-packages/grpc/_channel.py", line 416, in __next__
return self._next()
File "/Users/tameem/pubsub-test/env/lib/python3.7/site-packages/grpc/_channel.py", line 706, in _next
raise self
grpc._channel._MultiThreadedRendezvous: <_MultiThreadedRendezvous of RPC that terminated with:
status = StatusCode.UNAVAILABLE
details = "The service was unable to fulfill your request. Please try again. [code=8a75]"
debug_error_string = "{"created":"@1587006128.267570000","description":"Error received from peer ipv6:[]:443","file":"src/core/lib/surface/call.cc","file_line":1056,"grpc_message":"The service was unable to fulfill your request. Please try again. [code=8a75]","grpc_status":14}"
>
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/Users/tameem/pubsub-test/env/lib/python3.7/site-packages/google/api_core/bidi.py", line 637, in _thread_main
self._bidi_rpc.open()
File "/Users/tameem/pubsub-test/env/lib/python3.7/site-packages/google/api_core/bidi.py", line 280, in open
call = self._start_rpc(iter(request_generator), metadata=self._rpc_metadata)
File "/Users/tameem/pubsub-test/env/lib/python3.7/site-packages/google/cloud/pubsub_v1/gapic/subscriber_client.py", line 1059, in streaming_pull
requests, retry=retry, timeout=timeout, metadata=metadata
File "/Users/tameem/pubsub-test/env/lib/python3.7/site-packages/google/api_core/gapic_v1/method.py", line 143, in __call__
return wrapped_func(*args, **kwargs)
File "/Users/tameem/pubsub-test/env/lib/python3.7/site-packages/google/api_core/retry.py", line 286, in retry_wrapped_func
on_error=on_error,
File "/Users/tameem/pubsub-test/env/lib/python3.7/site-packages/google/api_core/retry.py", line 184, in retry_target
return target()
File "/Users/tameem/pubsub-test/env/lib/python3.7/site-packages/google/api_core/timeout.py", line 214, in func_with_timeout
return func(*args, **kwargs)
File "/Users/tameem/pubsub-test/env/lib/python3.7/site-packages/google/api_core/grpc_helpers.py", line 146, in error_remapped_callable
six.raise_from(exceptions.from_grpc_error(exc), exc)
File "<string>", line 3, in raise_from
google.api_core.exceptions.ServiceUnavailable: 503 The service was unable to fulfill your request. Please try again. [code=8a75]
INFO:google.api_core.bidi:Thread-ConsumeBidirectionalStream exiting
INFO:google.cloud.pubsub_v1.subscriber._protocol.leaser:Thread-LeaseMaintainer exiting.
INFO:google.cloud.pubsub_v1.subscriber._protocol.heartbeater:Thread-Heartbeater exiting.