PubSub publish hangs in completion_queue.poll() when used inside ProcessPoolExecutor #3925
Description
- OS type and version
Linux, various flavours (problem observed on Arch with 4.12.8 and Ubuntu with 4.4.0)
- Python version and virtual environment information
python --version
3.5.2
- google-cloud-python version
pip show google-cloud,pip show google-<service>orpip freeze
appdirs==1.4.3
asn1crypto==0.22.0
astroid==1.4.9
beautifulsoup4==4.5.3
bots==3.3.1.dev0
cachetools==2.0.0
certifi==2017.7.27.1
cffi==1.10.0
chardet==3.0.4
cheroot==5.8.3
CherryPy==10.2.2
click==6.7
cryptography==2.0.3
decorator==4.1.2
dill==0.2.7.1
Django==1.8
dropbox==7.2.1
enum34==1.1.6
et-xmlfile==1.0.1
fake-useragent==0.1.4
ftputil==3.3.1
future==0.16.0
gapic-google-cloud-datastore-v1==0.15.3
gapic-google-cloud-error-reporting-v1beta1==0.15.3
gapic-google-cloud-logging-v2==0.91.3
gapic-google-cloud-pubsub-v1==0.15.4
gapic-google-cloud-spanner-admin-database-v1==0.15.3
gapic-google-cloud-spanner-admin-instance-v1==0.15.3
gapic-google-cloud-spanner-v1==0.15.3
geocoder==1.20.1
google-auth==1.0.2
google-auth-httplib2==0.0.2
google-cloud==0.27.0
google-cloud-bigquery==0.26.0
google-cloud-bigtable==0.26.0
google-cloud-core==0.26.0
google-cloud-datastore==1.2.0
google-cloud-dns==0.26.0
google-cloud-error-reporting==0.26.0
google-cloud-happybase==0.22.0
google-cloud-language==0.27.0
google-cloud-logging==1.2.0
google-cloud-monitoring==0.26.0
google-cloud-pubsub==0.27.0
google-cloud-resource-manager==0.26.0
google-cloud-runtimeconfig==0.26.0
google-cloud-spanner==0.26.0
google-cloud-speech==0.28.0
google-cloud-storage==1.3.2
google-cloud-translate==1.1.0
google-cloud-videointelligence==0.25.0
google-cloud-vision==0.26.0
google-gax==0.15.14
google-resumable-media==0.2.3
googleapis-common-protos==1.5.2
grpc-google-cloud-datastore-v1==0.14.0
grpc-google-cloud-logging-v2==0.90.0
grpc-google-cloud-pubsub-v1==0.14.0
grpc-google-iam-v1==0.11.1
grpcio==1.6.0
httplib2==0.10.3
hvac==0.2.17
idna==2.6
ipdb==0.10.2
ipython==5.3.0
ipython-genutils==0.2.0
isort==4.2.15
jdcal==1.3
lazy-object-proxy==1.3.1
lxml==3.7.3
magento==2.1
mccabe==0.6.1
monotonic==1.3
oauth2client==3.0.0
openpyxl==2.4.4
packaging==16.8
paramiko==2.1.1
pexpect==4.2.1
pickleshare==0.7.4
ply==3.8
portend==2.1.2
prompt-toolkit==1.0.14
proto-google-cloud-datastore-v1==0.90.4
proto-google-cloud-error-reporting-v1beta1==0.15.3
proto-google-cloud-logging-v2==0.91.3
proto-google-cloud-pubsub-v1==0.15.4
proto-google-cloud-spanner-admin-database-v1==0.15.3
proto-google-cloud-spanner-admin-instance-v1==0.15.3
proto-google-cloud-spanner-v1==0.15.3
protobuf==3.4.0
psycopg2==2.6.2
ptyprocess==0.5.1
py==1.4.34
pyasn1==0.3.3
pyasn1-modules==0.1.1
pycparser==2.18
Pygments==2.2.0
pyinotify==0.9.6
pylint==1.6.5
pyparsing==2.2.0
pytest==3.0.6
python-amazon-mws==0.6
python-dateutil==2.6.0
pytz==2017.2
PyYAML==3.12
ratelim==0.1.6
requests==2.18.4
retrying==1.3.3
rsa==3.4.2
selenium==3.0.2
simplegeneric==0.8.1
six==1.10.0
SQLAlchemy==1.1.5
suds-jurko==0.6
tempora==1.9
tenacity==4.4.0
traitlets==4.3.2
typing==3.6.2
ujson==1.35
urllib3==1.22
wcwidth==0.1.7
winpdb==1.3.6
wrapt==1.10.11
xlrd==1.0.0
- Stacktrace if available
completion_queue.poll() hangs on this line
- Steps to reproduce
Run code sample - hangs in topic.publish(...)
- Code example
from concurrent.futures import ProcessPoolExecutor
from time import sleep
from google.cloud import pubsub
def start_publishing(task):
client = pubsub.Client()
topic = client.topic('my-topic')
for i in range(5):
print('task {}, publishing {}'.format(task, i))
topic.publish('this is a test ' + str(i))
print('task {}, published {}'.format(task, i))
sleep(1)
def done(future):
r = future.result()
print(r)
if __name__ == '__main__':
executor = ProcessPoolExecutor(max_workers=4)
for t in range(4):
future = executor.submit(start_publishing, t)
future.add_done_callback(done)