Thanks to visit codestin.com
Credit goes to github.com

Skip to content

FakeStorageRpc generation support for copy fails on read #5705

@philipsdoctor

Description

@philipsdoctor

Hi,
I'm trying to use LocalStorageHelper for a test. I do this (scala)

    val request = CopyRequest
      .newBuilder
      .setSource(BlobId.of(sourceBucketName, sourceKey))
      .setTarget(BlobId.of(destinationBucketName, destinationKey))
      .build
    gcsClient.copy(request).getResult

    val obj =  gcsClient.get(BlobId.of(destinationBucketName, destinationKey))
    val content: Array[Byte] = obj.getContent(BlobSourceOption.generationMatch)

So that's a copy followed by a read action on the new file (gscClient is the LocalStorageHelper in this case).

However, during the copy action here:
https://github.com/googleapis/google-cloud-java/blob/master/google-cloud-clients/google-cloud-contrib/google-cloud-nio/src/main/java/com/google/cloud/storage/contrib/nio/testing/FakeStorageRpc.java#L379

The target is stored in metadata without a generation, so the read throws a NPE attempting to perform a generationMatch.

I'm hoping to learn:

  1. Do you agree this is a bug or at least "not optimal behavior" ?
  2. Would you be open to a PR that does the following:
    a) on copy, checks if destination exists, if it exists, gets the existing generation + 1
    b) if it doesn't exist gets 0
    c) uses that resolved generation in the target in the rewrite request target that is stored in the metadata in memory store.

The currently implemented generational match logic seems to be checking the destination:
https://github.com/googleapis/google-cloud-java/blob/master/google-cloud-clients/google-cloud-contrib/google-cloud-nio/src/main/java/com/google/cloud/storage/contrib/nio/testing/FakeStorageRpc.java#L366

Since generations are always updated on modification, I think my proposed changes are still good, but this precondition check in the code felt a little awkward to me as I read it.

Environment details

  1. Specify the API at the beginning of the title (for example, "BigQuery: ...")
    General, Core, and Other are also allowed as types: FakeStorageRpc
  2. OS type and version: osx 10.14.5
  3. Java version: scala 2.11.12
  4. google-cloud-java version(s): 0.99.0-alpha

Steps to reproduce

  1. use the LocalStorageHelper to copy a file
  2. use the LocalStorageHelper to read the file that was just copied

Code example

    val request = CopyRequest
      .newBuilder
      .setSource(BlobId.of(sourceBucketName, sourceKey))
      .setTarget(BlobId.of(destinationBucketName, destinationKey))
      .build
    gcsClient.copy(request).getResult

    val obj =  gcsClient.get(BlobId.of(destinationBucketName, destinationKey))
    val content: Array[Byte] = obj.getContent(BlobSourceOption.generationMatch)

Stack trace

java.lang.NullPointerException
	at com.google.cloud.storage.Blob$BlobSourceOption.toSourceOptions(Blob.java:99)
	at com.google.cloud.storage.Blob$BlobSourceOption.toSourceOptions(Blob.java:196)
	at com.google.cloud.storage.Blob.getContent(Blob.java:478)
	at com.datadog.spark.util.GCSUtil$.readObject(GCSUtil.scala:110)

External references such as API reference guides used

Any additional information below

Making sure to follow these steps will guarantee the quickest resolution possible.

Thanks!

Metadata

Metadata

Assignees

Labels

api: storageIssues related to the Cloud Storage API.type: questionRequest for information or clarification. Not an issue.

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions