-
Notifications
You must be signed in to change notification settings - Fork 4.3k
use blob.exists to check the GCS file #34818
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: master
Are you sure you want to change the base?
Conversation
Checks are failing. Will not request review until checks are succeeding. If you'd like to override that behavior, comment |
Assigning reviewers. If you would like to opt out of this review, comment R: @shunping for label python. Available commands:
The PR bot will only process comments in the main thread (not review comments). |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for attempting to fix the issue.
I see you are trying to replace bucket.get_blob, called in _gcs_object, with blob.exists.
I took a look at the two function calls.
- bucket.get_blob
- calling
blob.reload
inherited from_PropertyMixin
at https://github.com/googleapis/python-storage/blob/v2.18.2/google/cloud/storage/_helpers.py#L216- calling
client._get_resource
at https://github.com/googleapis/python-storage/blob/v2.18.2/google/cloud/storage/_helpers.py#L300
- calling
- calling
- blob.exists
- calling
client._get_resource
at https://github.com/googleapis/python-storage/blob/v2.18.2/google/cloud/storage/blob.py#L734
- calling
It seems that under the hood, both apis are calling the same client._get_resource
, though the query parameters may be different. I am not sure if the change could make any performance improvement for the use case here.
Adding @cojenco and @andrewsg from GCS client library to chime in regarding any performance difference between bucket.get_blob
and blob.exists
.
I ran some tests on both apis, it does have some difference, though not big from google.cloud import storage
import timeit
storage_client = storage.Client()
bucket = storage_client.bucket("shunping-test")
time_taken = timeit.timeit(
lambda : bucket.get_blob("wordcount-output-00000-of-00001"),
number=100)
print(time_taken)
blob = bucket.blob("wordcount-output-00000-of-00001")
time_taken = timeit.timeit(blob.exists, number=100)
print(time_taken)
|
Thanks for tagging! That's right, both |
Fixes #34792
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
addresses #123
), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, commentfixes #<ISSUE NUMBER>
instead.CHANGES.md
with noteworthy changes.See the Contributor Guide for more tips on how to make review process smoother.
To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md
GitHub Actions Tests Status (on master branch)
See CI.md for more information about GitHub Actions CI or the workflows README to see a list of phrases to trigger workflows.