Closed
Description
Possibly related: #5410
I'm getting intermittent status code 503 responses from the service, when the logging client tries to log logs. When that happens, the problem is logged into the logger (of same name as the module) and I can't do anything about it not even, programmatically, resend it to stackdriver.
Additionally, I have no intentions of using the sync API due to how much it slows down the whole system.
Environment details
- Linux (default OS used in google cloud flexible)
- google cloud flexible python 3.6.x
- google-cloud-logging==1.7.0
Steps to reproduce
- Generate a large amount of logs and it intermittently happens
Stack trace
ERROR:google.cloud.logging.handlers.transports.background_thread:Failed to submit 1 logs.
Traceback (most recent call last): File "/env/lib/python3.6/site-packages/google/api_core/grpc_helpers.py", line 59, in error_remapped_callable return callable_(*args, **kwargs) File "/env/lib/python3.6/site-packages/grpc/_channel.py", line 533, in __call__ return _end_unary_response_blocking(state, call, False, None) File "/env/lib/python3.6/site-packages/grpc/_channel.py", line 467, in _end_unary_response_blocking raise _Rendezvous(state, None, None, deadline) grpc._channel._Rendezvous: <_Rendezvous of RPC that terminated with:
status = StatusCode.UNAVAILABLE
details = "Socket closed"
debug_error_string = "{"created":"@1542965282.988857618","description":"Error received from peer","file":"src/core/lib/surface/call.cc","file_line":1017,"grpc_message":"Socket closed","grpc_status":14}"
>
The above exception was the direct cause of the following exception:
Traceback (most recent call last): File "/env/lib/python3.6/site-packages/google/cloud/logging/handlers/transports/background_thread.py", line 118, in _safely_commit_batch batch.commit() File "/env/lib/python3.6/site-packages/google/cloud/logging/logger.py", line 612, in commit client.logging_api.write_entries(entries, **kwargs) File "/env/lib/python3.6/site-packages/google/cloud/logging/_gapic.py", line 124, in write_entries partial_success=partial_success) File "/env/lib/python3.6/site-packages/google/cloud/logging_v2/gapic/logging_service_v2_client.py", line 341, in write_log_entries request, retry=retry, timeout=timeout, metadata=metadata) File "/env/lib/python3.6/site-packages/google/api_core/gapic_v1/method.py", line 139, in __call__ return wrapped_func(*args, **kwargs) File "/env/lib/python3.6/site-packages/google/api_core/retry.py", line 260, in retry_wrapped_func on_error=on_error, File "/env/lib/python3.6/site-packages/google/api_core/retry.py", line 177, in retry_target return target() File "/env/lib/python3.6/site-packages/google/api_core/timeout.py", line 206, in func_with_timeout return func(*args, **kwargs) File "/env/lib/python3.6/site-packages/google/api_core/grpc_helpers.py", line 61, in error_remapped_callable six.raise_from(exceptions.from_grpc_error(exc), exc) File "<string>", line 3, in raise_from google.api_core.exceptions.ServiceUnavailable: 503 Socket closed