Storage: Deprecate DeleteRules and add LifecycleRules to accommodate SetStorageClass actions #3795

JesseLovelace · 2018-10-09T23:40:15Z

(Note: I'm holding on off Javadoc until the general class structure is approved)

This deprecates DeleteRules and establishes LifecycleRules as the standard to support object lifecycle management. The DeleteRules approach had the following problems:

It didn't have a base "Rule" class, meaning Delete was the only supported action
Instead of the condition being a field, all conditions were subclasses, which meant a single Delete rule could only have one condition, whereas the API specifies that a rule can have any number of conditions as long as it's at least one.
Following this pattern would have meant creating a new amount of subclasses equal to the number of conditions for each new action (i.e. to support SetStorageClass we would need to add AgeSetStorageClassRule, CreatedBeforeSetStorageClassRule, etc.)

Instead of following this pattern I've established a new pattern that more closely mirrors the API spec. Now rules are described by LifecycleRules, which have a LifecycleAction (currently Delete or SetStorageClass) and a LifecycleCondition(At least one of age, created before, num newer versions, etc). This way if a new action is created we only have to add one new subclass of LifecycleAction (instead of an amount equal to the number of conditions) and if a new condition is created we just add one new field to LifecycleCondition.

In addition to the new approach, this PR adds support for SetStorageClass lifecycle actions and MatchesStorageClass lifecycle conditions.

…etStorageClass actions

chingor13

If you set delete rules and lifecycle rules, won't the lifecycle rules overwrite the delete ones?

Should we merge the underlying representation of all the rules to be lifecycle rules?

…etStorageClass actions

frankyn

Left one request in comment for now and have three follow-up requests.

Could you add an integration test that exercises this against the service?
Also the code should keep the existing unit tests for the Deprecated method until it's removed entirely. At the moment the new feature removes support for the Deprecated setDeleteRules().
Could you format your code using this tool https://github.com/google/google-java-format?

...le-cloud-clients/google-cloud-storage/src/main/java/com/google/cloud/storage/BucketInfo.java

              }));
+      builder.setDeleteRules(
+              transform(
+                      bucketPb.getLifecycle().getRule(),


...ud-clients/google-cloud-storage/src/test/java/com/google/cloud/storage/it/ITStorageTest.java


+  @Test
+  public void testGetBucketLifecycleRules() {
+    Bucket remoteBucket = storage.get(BUCKET, Storage.BucketGetOption.fields(BucketField.values()));


...ud-clients/google-cloud-storage/src/test/java/com/google/cloud/storage/it/ITStorageTest.java

    storage = remoteStorageHelper.getOptions().getService();
    storage.create(
        BucketInfo.newBuilder(BUCKET)
-            .setDeleteRules(Collections.singleton(new BucketInfo.AgeDeleteRule(1)))


...le-cloud-clients/google-cloud-storage/src/main/java/com/google/cloud/storage/BucketInfo.java

+      Rule rule = new Rule();
+
+      Rule.Action action = new Rule.Action().setType(lifecycleAction.getActionType());
+      if (lifecycleAction instanceof SetStorageClassLifecycleAction) {


...le-cloud-clients/google-cloud-storage/src/main/java/com/google/cloud/storage/BucketInfo.java

+
+      switch (action.getType()) {
+        case DeleteLifecycleAction.TYPE:
+          {


...le-cloud-clients/google-cloud-storage/src/main/java/com/google/cloud/storage/BucketInfo.java

+        return createdBefore;
+      }
+
+      public LifecycleCondition setCreatedBefore(DateTime createdBefore) {


frankyn

Builder feedback and a few other nits. Thanks @JesseLovelace!

...loud-clients/google-cloud-storage/src/test/java/com/google/cloud/storage/BucketInfoTest.java

+  @Test
+  public void testLifecycleRules() {
+    Rule deleteLifecycleRule = new LifecycleRule(new LifecycleRule.DeleteLifecycleAction(),
+            new LifecycleRule.LifecycleCondition.Builder().setAge(10).build()).toPb();


...ud-clients/google-cloud-storage/src/test/java/com/google/cloud/storage/it/ITStorageTest.java

+                                                    .setCreatedBefore(new DateTime(System.currentTimeMillis()))
+                                                    .setMatchesStorageClass(ImmutableList.of(StorageClass.COLDLINE))
+                                                    .build())))
+                    .build());


...ud-clients/google-cloud-storage/src/test/java/com/google/cloud/storage/it/ITStorageTest.java

+                                                    .setNumberOfNewerVersions(3)
+                                                    .setIsLive(false)
+                                                    .setCreatedBefore(new DateTime(System.currentTimeMillis()))
+                                                    .setMatchesStorageClass(ImmutableList.of(StorageClass.COLDLINE))


...ud-clients/google-cloud-storage/src/test/java/com/google/cloud/storage/it/ITStorageTest.java

  private static final String KMS_KEY_ONE_NAME = "gcs_kms_key_one";
  private static final String KMS_KEY_TWO_NAME = "gcs_kms_key_two";
  private static final boolean IS_VPC_TEST = System.getenv("GOOGLE_CLOUD_TESTS_IN_VPCSC") != null
          && System.getenv("GOOGLE_CLOUD_TESTS_IN_VPCSC").equalsIgnoreCase("true");


...le-cloud-clients/google-cloud-storage/src/main/java/com/google/cloud/storage/BucketInfo.java

+      }
+    }
+
+    public static class SetStorageClassLifecycleAction extends LifecycleAction {


...loud-clients/google-cloud-storage/src/test/java/com/google/cloud/storage/BucketInfoTest.java

+            new LifecycleRule.SetStorageClassLifecycleAction(StorageClass.COLDLINE),
+            new LifecycleRule.LifecycleCondition.Builder().setIsLive(true).setNumberOfNewerVersions(10).build()).toPb();
+
+    assertEquals("COLDLINE", setStorageClassLifecycleRule.getAction().getStorageClass());


...loud-clients/google-cloud-storage/src/test/java/com/google/cloud/storage/BucketInfoTest.java

+    Rule deleteLifecycleRule = new LifecycleRule(new LifecycleRule.DeleteLifecycleAction(),
+            new LifecycleRule.LifecycleCondition.Builder().setAge(10).build()).toPb();
+
+    assertEquals("Delete", deleteLifecycleRule.getAction().getType());


frankyn

I think these are my last nits.

...le-cloud-clients/google-cloud-storage/src/main/java/com/google/cloud/storage/BucketInfo.java

      switch (action.getType()) {
        case DeleteLifecycleAction.TYPE:
-          lifecycleAction = new DeleteLifecycleAction();
+          lifecycleAction = LifecycleAction.newDeleteLifecycleAction();


...le-cloud-clients/google-cloud-storage/src/main/java/com/google/cloud/storage/BucketInfo.java

        case SetStorageClassLifecycleAction.TYPE:
          lifecycleAction =
-              new SetStorageClassLifecycleAction(StorageClass.valueOf(action.getStorageClass()));
+              LifecycleAction.newSetStorageClassLifecycleAction(StorageClass.valueOf(action.getStorageClass()));


...loud-clients/google-cloud-storage/src/test/java/com/google/cloud/storage/BucketInfoTest.java

-    assertEquals("COLDLINE", setStorageClassLifecycleRule.getAction().getStorageClass());
+    assertEquals(StorageClass.COLDLINE, setStorageClassLifecycleRule.getAction().getStorageClass());
    assertTrue(setStorageClassLifecycleRule.getCondition().getIsLive());
    assertEquals(10, (long)setStorageClassLifecycleRule.getCondition().getNumNewerVersions());


...loud-clients/google-cloud-storage/src/test/java/com/google/cloud/storage/BucketInfoTest.java

+            .toPb();
+
+    assertEquals(LifecycleRule.DeleteLifecycleAction.TYPE, deleteLifecycleRule.getAction().getType());
+    assertEquals(10, (long)deleteLifecycleRule.getCondition().getAge());


frankyn

Summary comments:

Reduce length of class names for Actions by removing Lifecycle.
Try to use more of cloud.google.com/storage/docs documentation on this feature with your method javadocs.
Add short snippets of how to use the new feature in the javadoc sparingly. We will have samples, but in the mean time it will help provide direction.
One question about a possible breaking change.

Looks good! Minor requests.

...le-cloud-clients/google-cloud-storage/src/main/java/com/google/cloud/storage/BucketInfo.java

+    /**
+     * Condition for a Lifecycle rule, specifies under what criteria an Action should be executed
+     *
+     * @see <a href="https://cloud.google.com/storage/docs/lifecycle">Object Lifecycle Management</a>


...le-cloud-clients/google-cloud-storage/src/main/java/com/google/cloud/storage/BucketInfo.java

+   * Lifecycle rule for a bucket. Allows supported Actions, such as deleting and changing storage class,
+   * to be executed when certain Conditions are met
+   *
+   * @see <a href="https://cloud.google.com/storage/docs/lifecycle">Object Lifecycle Management</a>


...le-cloud-clients/google-cloud-storage/src/main/java/com/google/cloud/storage/BucketInfo.java

   * @see <a href="https://cloud.google.com/storage/docs/lifecycle">Object Lifecycle Management</a>
+   *
+   * @deprecated Use a {@code LifecycleRule} with a {@code DeleteLifecycleAction} and use
+   * {@code LifecycleCondition.Builder.setAge} instead


...le-cloud-clients/google-cloud-storage/src/main/java/com/google/cloud/storage/BucketInfo.java

    @Override
    void populateCondition(Rule.Condition condition) {
-      condition.setCreatedBefore(new DateTime(timeMillis));
+      condition.setCreatedBefore(new DateTime(true, timeMillis, 0));


...le-cloud-clients/google-cloud-storage/src/main/java/com/google/cloud/storage/BucketInfo.java

+     * Sets the bucket's lifecycle configuration as a number of lifecycle rules, consisting of an action and a
+     * condition.
+     *
+     * @see <a href="https://cloud.google.com/storage/docs/lifecycle">Lifecycle Management</a>


...le-cloud-clients/google-cloud-storage/src/main/java/com/google/cloud/storage/BucketInfo.java

+
+        /** Sets whether a Live Blob meets the Condition for an Action to be executed. If this is set to {@code false},
+         * the condition is met by Archived Blobs.
+         */


...le-cloud-clients/google-cloud-storage/src/main/java/com/google/cloud/storage/BucketInfo.java

+          return this;
+        }
+
+        /** Sets a list of Storage Classes ({@code StorageClass} objects) for a Blob to match for an Action to be executed **/


...le-cloud-clients/google-cloud-storage/src/main/java/com/google/cloud/storage/BucketInfo.java

+
+    /**
+     * Base class for the Action to take when a Lifecycle Condition is met. Specific Actions are expressed as subclasses
+     * of this class, accessed by static factory methods


...le-cloud-clients/google-cloud-storage/src/main/java/com/google/cloud/storage/BucketInfo.java

+       *
+       * @param storageclass The {@code StorageClass} to set Blobs that meet the condition to
+       */
+      public static SetStorageClassLifecycleAction newSetStorageClassAction(StorageClass storageclass) {


...le-cloud-clients/google-cloud-storage/src/main/java/com/google/cloud/storage/BucketInfo.java

+      /** Creates a new {@code SetStorageClassLifecycleAction}. Blobs that meet the Condition associated with this
+       * action will have their Storage Class changed to the one specified
+       *
+       * @param storageclass The {@code StorageClass} to set Blobs that meet the condition to


...le-cloud-clients/google-cloud-storage/src/main/java/com/google/cloud/storage/BucketInfo.java

-   *
-   * @deprecated Use a {@code LifecycleRule} with a {@code DeleteLifecycleAction} and use
-   * {@code LifecycleCondition.Builder.setAge} instead
+   * @deprecated Use a {@code LifecycleRule} with a {@code DeleteLifecycleAction} and use {@code


frankyn

One last nit, mainly spacing and then I'll approve.

Thanks @JesseLovelace

frankyn

Add a new line between deprecation message and the example.
Found another missing period I missed on the last review.

...le-cloud-clients/google-cloud-storage/src/main/java/com/google/cloud/storage/BucketInfo.java

+    }
+
+    /**
+     * Condition for a Lifecycle rule, specifies under what criteria an Action should be executed


frankyn

One missed period. I think that's the last of my comments. Double checking on the generated javadocs. What do you think about the @deprecation section rendered?

...le-cloud-clients/google-cloud-storage/src/main/java/com/google/cloud/storage/BucketInfo.java


+  /**
+   * Lifecycle rule for a bucket. Allows supported Actions, such as deleting and changing storage class,
+   * to be executed when certain Conditions are met


frankyn

LGTM, One additional AI discussed offline is to format the code with the java formatter.

frankyn

Comments are now within column limits. LGTM Thanks @JesseLovelace!

Storage: Deprecate DeleteRules and add LifecycleRules to accomodate S…

d273f4e

…etStorageClass actions

JesseLovelace requested review from chingor13 and frankyn October 9, 2018 23:40

JesseLovelace requested a review from pongad as a code owner October 9, 2018 23:40

googlebot added the cla: yes This human has signed the Contributor License Agreement. label Oct 9, 2018

chingor13 changed the title ~~Storage: Deprecate DeleteRules and add LifecycleRules to accommodate SetStorageClass actions~~ WIP: Storage: Deprecate DeleteRules and add LifecycleRules to accommodate SetStorageClass actions Oct 10, 2018

chingor13 requested a review from garrettjonesgoogle October 10, 2018 21:08

chingor13 reviewed Oct 11, 2018

View reviewed changes

JesseLovelace added 2 commits October 12, 2018 11:16

Storage: Deprecate DeleteRules and add LifecycleRules to accomodate S…

8742ecf

…etStorageClass actions

Merge branch 'master' of github.com:JesseLovelace/google-cloud-java

a360f0e

JesseLovelace requested a review from a team as a code owner October 12, 2018 18:21

frankyn suggested changes Oct 12, 2018

View reviewed changes

...le-cloud-clients/google-cloud-storage/src/main/java/com/google/cloud/storage/BucketInfo.java Show resolved Hide resolved

chingor13 reviewed Oct 15, 2018

View reviewed changes

...le-cloud-clients/google-cloud-storage/src/main/java/com/google/cloud/storage/BucketInfo.java Outdated

}));

builder.setDeleteRules(

transform(

bucketPb.getLifecycle().getRule(),

This comment was marked as spam.

Sign in to view

This comment was marked as spam.

Sign in to view

JustinBeckwith and others added 3 commits October 15, 2018 21:24

Merge branch 'master' into master

ae4db50

Merge branch 'master' of github.com:JesseLovelace/google-cloud-java

126bdc0

Merge branch 'master' of github.com:JesseLovelace/google-cloud-java

2ddef75

frankyn suggested changes Oct 17, 2018

View reviewed changes

JesseLovelace added 3 commits October 17, 2018 12:42

Merge branch 'master' of github.com:JesseLovelace/google-cloud-java

00a21e2

Merge branch 'master' of github.com:JesseLovelace/google-cloud-java

25f0dd4

Merge remote-tracking branch 'upstream/master'

b8ef6cb

frankyn suggested changes Oct 17, 2018

View reviewed changes

JesseLovelace added 2 commits October 18, 2018 13:49

Merge remote-tracking branch 'upstream/master'

7ec9765

Merge branch 'master' of github.com:JesseLovelace/google-cloud-java

ee8c3c4

frankyn suggested changes Oct 18, 2018

View reviewed changes

JesseLovelace and others added 3 commits October 22, 2018 12:42

Merge branch 'master' of github.com:JesseLovelace/google-cloud-java

e5ac097

Merge branch 'master' of github.com:JesseLovelace/google-cloud-java

c844086

Merge branch 'master' into master

843c6ac

frankyn suggested changes Oct 23, 2018

View reviewed changes

JesseLovelace added 2 commits October 23, 2018 14:23

Merge branch 'master' of github.com:JesseLovelace/google-cloud-java

ce974f3

Merge branch 'master' of github.com:JesseLovelace/google-cloud-java

b17dfdb

frankyn reviewed Oct 23, 2018

View reviewed changes

frankyn suggested changes Oct 23, 2018

View reviewed changes

...le-cloud-clients/google-cloud-storage/src/main/java/com/google/cloud/storage/BucketInfo.java Outdated

}

/**

* Condition for a Lifecycle rule, specifies under what criteria an Action should be executed

This comment was marked as spam.

Sign in to view

JesseLovelace added 2 commits October 23, 2018 15:47

Merge branch 'master' of github.com:JesseLovelace/google-cloud-java

2114eb5

Merge branch 'master' of github.com:JesseLovelace/google-cloud-java

477f5df

frankyn suggested changes Oct 23, 2018

View reviewed changes

JesseLovelace added 2 commits October 23, 2018 15:58

Merge branch 'master' of github.com:JesseLovelace/google-cloud-java

9b59e0c

Merge branch 'master' of github.com:JesseLovelace/google-cloud-java

3ee4257

frankyn approved these changes Oct 23, 2018

View reviewed changes

JesseLovelace added 2 commits October 23, 2018 16:02

Merge branch 'master' of github.com:JesseLovelace/google-cloud-java

8a503c9

Merge branch 'master' of github.com:JesseLovelace/google-cloud-java

1b5abc3

frankyn approved these changes Oct 23, 2018

View reviewed changes

JesseLovelace added the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Oct 23, 2018

kokoro-team removed the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Oct 23, 2018

Merge branch 'master' into master

6aa7b91

JesseLovelace added the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Oct 24, 2018

kokoro-team removed the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Oct 24, 2018

JesseLovelace merged commit 114701f into googleapis:master Oct 24, 2018

JesseLovelace mentioned this pull request Oct 25, 2018

Release v0.68.0 #3857

Merged

JesseLovelace changed the title ~~WIP: Storage: Deprecate DeleteRules and add LifecycleRules to accommodate SetStorageClass actions~~ Storage: Deprecate DeleteRules and add LifecycleRules to accommodate SetStorageClass actions Feb 25, 2021

Storage: Deprecate DeleteRules and add LifecycleRules to accommodate SetStorageClass actions #3795

Storage: Deprecate DeleteRules and add LifecycleRules to accommodate SetStorageClass actions #3795

Uh oh!

Conversation

JesseLovelace commented Oct 9, 2018

Uh oh!

chingor13 left a comment

Choose a reason for hiding this comment

Uh oh!

frankyn left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

frankyn left a comment

Choose a reason for hiding this comment

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

frankyn left a comment

Choose a reason for hiding this comment

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

frankyn left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

frankyn left a comment •

edited

Loading

frankyn left a comment •

edited

Loading