-
Notifications
You must be signed in to change notification settings - Fork 1.1k
Description
Hi,
I'm trying to use LocalStorageHelper for a test. I do this (scala)
val request = CopyRequest
.newBuilder
.setSource(BlobId.of(sourceBucketName, sourceKey))
.setTarget(BlobId.of(destinationBucketName, destinationKey))
.build
gcsClient.copy(request).getResult
val obj = gcsClient.get(BlobId.of(destinationBucketName, destinationKey))
val content: Array[Byte] = obj.getContent(BlobSourceOption.generationMatch)
So that's a copy followed by a read action on the new file (gscClient is the LocalStorageHelper in this case).
However, during the copy action here:
https://github.com/googleapis/google-cloud-java/blob/master/google-cloud-clients/google-cloud-contrib/google-cloud-nio/src/main/java/com/google/cloud/storage/contrib/nio/testing/FakeStorageRpc.java#L379
The target is stored in metadata without a generation, so the read throws a NPE attempting to perform a generationMatch.
I'm hoping to learn:
- Do you agree this is a bug or at least "not optimal behavior" ?
- Would you be open to a PR that does the following:
a) on copy, checks if destination exists, if it exists, gets the existing generation + 1
b) if it doesn't exist gets 0
c) uses that resolved generation in the target in the rewrite request target that is stored in themetadatain memory store.
The currently implemented generational match logic seems to be checking the destination:
https://github.com/googleapis/google-cloud-java/blob/master/google-cloud-clients/google-cloud-contrib/google-cloud-nio/src/main/java/com/google/cloud/storage/contrib/nio/testing/FakeStorageRpc.java#L366
Since generations are always updated on modification, I think my proposed changes are still good, but this precondition check in the code felt a little awkward to me as I read it.
Environment details
- Specify the API at the beginning of the title (for example, "BigQuery: ...")
General, Core, and Other are also allowed as types: FakeStorageRpc - OS type and version: osx 10.14.5
- Java version:
scala 2.11.12 - google-cloud-java version(s):
0.99.0-alpha
Steps to reproduce
- use the LocalStorageHelper to copy a file
- use the LocalStorageHelper to read the file that was just copied
Code example
val request = CopyRequest
.newBuilder
.setSource(BlobId.of(sourceBucketName, sourceKey))
.setTarget(BlobId.of(destinationBucketName, destinationKey))
.build
gcsClient.copy(request).getResult
val obj = gcsClient.get(BlobId.of(destinationBucketName, destinationKey))
val content: Array[Byte] = obj.getContent(BlobSourceOption.generationMatch)
Stack trace
java.lang.NullPointerException
at com.google.cloud.storage.Blob$BlobSourceOption.toSourceOptions(Blob.java:99)
at com.google.cloud.storage.Blob$BlobSourceOption.toSourceOptions(Blob.java:196)
at com.google.cloud.storage.Blob.getContent(Blob.java:478)
at com.datadog.spark.util.GCSUtil$.readObject(GCSUtil.scala:110)
External references such as API reference guides used
Any additional information below
Making sure to follow these steps will guarantee the quickest resolution possible.
Thanks!