Add Google Cloud Storage adapter. #2000
Conversation
…or which doesn't relies on method.
…oogle client classes.
|
@gbanasiak This is ready for review. |
gbanasiak
left a comment
There was a problem hiding this comment.
Looks good, I left small comments below.
Local testing with small files were OK, but tests with larger downloads show data corruption during decompression. pzstd reports Unknown frame descriptor while standard library reports zstd.ZstdError: zstd decompress error: Data corruption detected. This is easily reproducible. Legacy download works fine.
gbanasiak
left a comment
There was a problem hiding this comment.
This works fine in testing now, I've checked both Linux and MacOS.
Can you add relevant tests to test_adapter_registry_get() in adapter_test.py?
Unrelated: when testing directly with esrally-storage I need to pass the bucket URL as base with --base-url, example:
esrally-storage --base-url=gs://<bucket> -v get gs://<bucket>/<something>/<file>
Is this avoidable?
I'll do.
There are some issues (including this) that I saw in the CLI that should have been fixed in the incoming CL (please have a look at PR #1959 for it).
|
This adds the new
GSAdapterclass to download files from Google Cloud buckets using authentication.It allows to download bucket blobs via URLs like the following:
gs://{bucket_name}/{blob_name}https://storage.cloud.google.com/{bucket_name}/{blob_name}For downloading files publicly available from a Google Cloud bucket, the following type of URLs is supported by the already existing
HTTPAdapter:https://storage.googleapis.com/{bucket_name}/{blob_name}