US66693: Refactor policy evaluation decision caching to allow eviction of keys #115

FrankGasparovic · 2017-04-10T20:21:58Z

Signed-off-by: Anubhav [email protected]

FrankGasparovic · 2017-04-10T20:23:53Z

Notes on new caching:
https://devcloud.swcoe.ge.com/devspace/display/RMOCM/ACS+Decision+Caching

irinaepshteyn

Review is still in progress. These are the initial comments...

irinaepshteyn · 2017-04-13T21:19:15Z

...e/src/main/java/com/ge/predix/acs/service/policy/evaluation/PolicyEvaluationServiceImpl.java

+            } catch (Exception e) {
+                LOGGER.error(String.format(
+                        "Unable to set cache key '%s' to value '%s' due to redis exception", key, result), e);
+            }


Error message should not be tied to redis. Could have other cache implementations.

irinaepshteyn · 2017-04-13T21:23:23Z

...e/src/main/java/com/ge/predix/acs/service/policy/evaluation/PolicyEvaluationServiceImpl.java

+        } catch (Exception e) {
+            LOGGER.error(String.format("Unable to get cache key '%s'", key), e);
+        }
+


Add a unit test that policy evaluation returns expected result if cache get or set fails

irinaepshteyn · 2017-04-13T22:31:29Z

...e/src/main/java/com/ge/predix/acs/policy/evaluation/cache/AbstractPolicyEvaluationCache.java

+
+        String zone = key.getZoneId();
+        values.addAll(multiGet(cachedResult.getResolvedResourceUris().stream()
+                .map(resourceUri -> resourceKey(zone, resourceUri)).collect(Collectors.toList())));


Have you thought of including resolved URIs as part of evaluation result key to avoid another trip to the cache store? Something like:
zoneId:resourceId:resolvedURIList:...
Are we limited on the length of the key?

Add a unit test that verifies that get returns null if one of the resolved resources change. Resolved URI of the changed does not match resource URI in the request.

Add a unit test that verifies that get returns null if timestamps for any entity including resolved URIs are not present.

I take it back. Resolved UIRs could not be part of the eval result key as they are not known at the time of the request.

irinaepshteyn · 2017-04-13T22:33:41Z

...e/src/main/java/com/ge/predix/acs/policy/evaluation/cache/AbstractPolicyEvaluationCache.java

+        // Only need to assemble keys for policy sets, the subject, and the policy evaluation result.
+        // Resource-related information will be captured in the resolved resource URIs from the cached
+        // policy evaluation result.
        List<String> keys = new ArrayList<>();


If resolvedURI were part of evaluation result key we could assemble resource keys for resolved URIs here.

irinaepshteyn · 2017-04-13T23:32:09Z

...e/src/main/java/com/ge/predix/acs/policy/evaluation/cache/AbstractPolicyEvaluationCache.java

-    }
+    private void setEntityTimestamps(final PolicyEvaluationRequestCacheKey key, final PolicyEvaluationResult result) {
+        // This will lower the overall hit rate for policy evaluation but is required to ensure that if
+        // the cached timestamp for any entity involved in this decision is aged out, the cached decision will be used


Did you mean WILL NOT be used?

I do mean will, but I think the message is confusing. I rewrote it.

irinaepshteyn · 2017-04-14T00:35:38Z

...e/src/main/java/com/ge/predix/acs/policy/evaluation/cache/AbstractPolicyEvaluationCache.java

+        String key = getKey.apply(zoneId, entityId);
+        setIfNotExists(key, timestampValue());
+    }
+


Is there a difference between setEntityIfNotExists and resetForEntity?
When eval result is being cached we could call resetForEntity to set the timestamp. It will create a entity key if doesn't exist.

setEntityIfNotExists will set the timestamp for the object if the object is not in the cache. resetForEntity will reset the timestamp regardless if the value if the value is in the cache

irinaepshteyn · 2017-04-14T00:45:22Z

...e/src/main/java/com/ge/predix/acs/policy/evaluation/cache/AbstractPolicyEvaluationCache.java

-        multisetForResources(zoneId, resourceEntities.stream().map(resource -> resource.getResourceIdentifier())
+        multisetForResources(zoneId, resourceEntities.stream().map(ResourceEntity::getResourceIdentifier)
                .collect(Collectors.toList()));
    }


We are collecting resource Ids into a list and then build a map of keys inside multisetForResources. Could we combine it all in one call? Map streamed resourceEntities to resourceKey and collect into a map with the timestamp logging debug message for each... :-)

irinaepshteyn · 2017-04-14T00:47:58Z

...e/src/main/java/com/ge/predix/acs/policy/evaluation/cache/AbstractPolicyEvaluationCache.java

        multisetForSubjects(zoneId,
-                subjectEntities.stream().map(subject -> subject.getSubjectIdentifier()).collect(Collectors.toList()));
+                subjectEntities.stream().map(SubjectEntity::getSubjectIdentifier).collect(Collectors.toList()));
    }


Same comment as for resetForResources above. Could we combine resetForSubjects and multisetForSubjects in one long beautiful fluent java 8 call? :-)

irinaepshteyn · 2017-04-14T00:53:54Z

...e/src/main/java/com/ge/predix/acs/policy/evaluation/cache/AbstractPolicyEvaluationCache.java

+    boolean havePrivilegeServiceAttributesChanged(final List<String> values, final DateTime policyEvalTimestampUTC) {
+        for (String value : values) {
+            if (null == value) {
+                return true;


Values were already tested for nulls by isRequestCached before calling this method.

irinaepshteyn · 2017-04-14T01:01:53Z

...e/src/main/java/com/ge/predix/acs/policy/evaluation/cache/AbstractPolicyEvaluationCache.java

+                            "Privilege service attributes have timestamp '%s' which is after "
+                                    + "policy evaluation timestamp '%s'",
+                            invalidationTimestampUTC, policyEvalTimestampUTC);
+                }


Since you are using lazy string construction we could rely on debug to check the debug level. No need for extra isDebugEnabled call. Will message formatting work with %s or should we use {}?

irinaepshteyn

More comments...

irinaepshteyn · 2017-04-14T21:10:39Z

...e/src/main/java/com/ge/predix/acs/policy/evaluation/cache/AbstractPolicyEvaluationCache.java

+        // all entities involved in the request.
+        values.addAll(multiGet(cachedResult.getResolvedResourceUris().stream()
+                .map(resourceUri -> resourceKey(zone, resourceUri)).collect(Collectors.toList())));



One optimization we could do here is to compare resource from request with resolved resource. If policy doesn't contain any template URIs then request resource and resolved resource are the same. In that case we don't have to perform a multiGet again. In that case when assembling keys for the first call to multiGet we need to include a resource from request again.

irinaepshteyn · 2017-04-14T21:38:11Z

service/src/main/java/com/ge/predix/acs/policy/evaluation/cache/PolicyEvaluationCache.java


    void resetForPolicySet(String zoneId, String policySetId);

+    void setPolicySetIfNotExists(String zoneId, String policySetId);


setPolicySetIfNotExists, setResourceIfNotExists, setSubjectIfNotExists do not need to be part of cache interface. These methods are implemented and used by AbstractPolicyEvaluationCache only. However, setIfNotExists should be defined by interface and implemented by each cache implementation.

I agree that setPolicySetIfNotExists, setResourceIfNotExists and setSubjectIfNotExists should not be part of the cache interface (because these are only used internally in the AbstractPolicyEvaluationCache). But why should setIfNotExists be apart of the interface if it is also only use internally?

irinaepshteyn · 2017-04-14T22:29:29Z

service/src/main/java/com/ge/predix/acs/policy/evaluation/cache/RedisPolicyEvaluationCache.java

-            }
-        });
+    void setIfNotExists(final String key, final String value) {
+        this.redisTemplate.boundValueOps(key).setIfAbsent(value);


Should we set an expiration timeout on the key similar to what's done in line #89? The same comment applies to set function above. Timeout is only set on the key for eval result and not for any other.

Timeout should not be set for anything other then Policy Evaluation. This is because we would then be unnecessarily reevaluating requests when entities have no changed.

irinaepshteyn · 2017-04-14T23:21:29Z

...c/test/java/com/ge/predix/acs/policy/evaluation/cache/InMemoryPolicyEvaluationCacheTest.java

-        assertEquals(evalCache.size(), 0);
-        Map<String, String> timestampCache = (Map<String, String>) getInternalState(this.cache, "timestampCache");
-        assertEquals(timestampCache.size(), 1);
+        assertEquals(evalCache.size(), 1);


May be rename the test to testSetPolicySetChangedTimestamp?

Current implementation of the in-memory cache stores eval result and timestamps in the same map, not in different maps as before. There is no need to have different tests for setting eval result and setting timestamps.

I do feel like there is value in this test because of the logic to check the key format.

irinaepshteyn · 2017-04-14T23:51:44Z

...c/test/java/com/ge/predix/acs/policy/evaluation/cache/AbstractPolicyEvaluationCacheTest.java

    }

    @Test
    public void testGetWithCacheHit() {


Add a test to assert that get returns cached result if eval result is cached but not timestamps for entities.

If eval is cached but timestamps for entities are not, then get should return null

irinaepshteyn · 2017-04-14T23:55:48Z

...c/test/java/com/ge/predix/acs/policy/evaluation/cache/AbstractPolicyEvaluationCacheTest.java

+        this.cache.resetForResource(ZONE_NAME, XFILES_ID);
+        this.cache.resetForPolicySet(ZONE_NAME, POLICY_ONE.getName());
+        this.cache.resetForPolicySet(ZONE_NAME, POLICY_TWO.getName());
        request.setAction(ACTION_GET);


Do we really need to call resets for each entity i every test? Get will return cached result if no timestamps.

We do, get will return null if no timestamps.

irinaepshteyn

Done reviewing.

FrankGasparovic · 2017-05-10T17:54:45Z

Siva ran performance tests and he saw a 3ms increase in Policy Evaluation calls

…n of keys Signed-off-by: Anubhav <[email protected]>

sanjeevchopra · 2017-05-16T01:32:17Z

@anubhavi25 @FrankGasparovic - pls review my 2nd commit

FrankGasparovic · 2017-05-16T16:24:05Z

...e/src/main/java/com/ge/predix/acs/policy/evaluation/cache/AbstractPolicyEvaluationCache.java

+        Set<String> cachedResolvedResourceUris = cachedEvalResult.getResolvedResourceUris();
+        //is requested resource id same as resolved resource uri ?
+        if (cachedResolvedResourceUris.size() == 1
+                && cachedResolvedResourceUris.toArray()[0].equals(key.getResourceId())) {


Use cachedResolvedResourceUris.iterator().next() vs toArray() so that we don't have to copy contents to array.

FrankGasparovic

Looks good.

anubhavi25

Looks good for the most part 👍

anubhavi25 · 2017-05-16T17:40:10Z

...e/src/main/java/com/ge/predix/acs/policy/evaluation/cache/AbstractPolicyEvaluationCache.java

+            return entryValues.get(lastValueIndex - 2);
+        }
+
+        String getRequestedResourceLastModified() {


Changing the name to just getResourceLastModified should suffice (and is also consistent with the naming convention used for the other get*LastModified methods).

…us code cleanup.

sanjeevchopra · 2017-05-31T17:07:16Z

Performance tests show no regression. A second test was run which only writes resource and subject keys (which have no TTL) to confirm that eviction is working as expected.

FrankGasparovic assigned sanjeevchopra and anurag-gujral Apr 10, 2017

FrankGasparovic force-pushed the US95911 branch from a058d10 to b2a2b41 Compare April 11, 2017 17:56

anubhavi25 changed the base branch from US95911 to develop April 11, 2017 22:25

FrankGasparovic force-pushed the US66693 branch from 3b1079f to 295c645 Compare April 12, 2017 17:41

sanjeevchopra removed their assignment Apr 12, 2017

irinaepshteyn reviewed Apr 14, 2017

View reviewed changes

irinaepshteyn self-assigned this Apr 14, 2017

irinaepshteyn reviewed Apr 14, 2017

View reviewed changes

irinaepshteyn requested changes Apr 14, 2017

View reviewed changes

FrankGasparovic force-pushed the US66693 branch 3 times, most recently from 6eb663c to b4ac16e Compare April 18, 2017 17:37

FrankGasparovic force-pushed the US66693 branch from b4ac16e to cd4ca5b Compare April 24, 2017 22:16

US66693: Refactor policy evaluation decision caching to allow evictio…

6886e24

…n of keys Signed-off-by: Anubhav <[email protected]>

FrankGasparovic force-pushed the US66693 branch from cd4ca5b to 6886e24 Compare May 12, 2017 17:39

sanjeevchopra assigned anubhavi25 May 16, 2017

FrankGasparovic commented May 16, 2017

View reviewed changes

anubhavi25 reviewed May 16, 2017

View reviewed changes

Added a structured, immutable type for batch result set from cache pl…

a7bb09c

…us code cleanup.

sanjeevchopra force-pushed the US66693 branch from d1b964e to a7bb09c Compare May 16, 2017 18:13

sanjeevchopra merged commit 9596557 into develop May 31, 2017

sanjeevchopra deleted the US66693 branch May 31, 2017 17:08


		void resetForPolicySet(String zoneId, String policySetId);

		void setPolicySetIfNotExists(String zoneId, String policySetId);

US66693: Refactor policy evaluation decision caching to allow eviction of keys #115

US66693: Refactor policy evaluation decision caching to allow eviction of keys #115

Uh oh!

Conversation

FrankGasparovic commented Apr 10, 2017

Uh oh!

FrankGasparovic commented Apr 10, 2017

Uh oh!

irinaepshteyn left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

irinaepshteyn left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

irinaepshteyn Apr 14, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

irinaepshteyn Apr 14, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

irinaepshteyn Apr 14, 2017 •

edited

Loading

irinaepshteyn Apr 14, 2017 •

edited

Loading

irinaepshteyn Apr 14, 2017 •

edited

Loading

FrankGasparovic commented May 10, 2017 •

edited

Loading