US87160: Caching-related abstractions for resource/subject attribute readers #93

anubhavi25 · 2017-03-02T00:15:24Z

No description provided.

FrankGasparovic

Looks good

FrankGasparovic · 2017-03-02T17:25:16Z

acs-integration-tests/pom.xml

                                <NO_READ_ONLY_USER_PASSWORD>${NO_READ_ONLY_USER_PASSWORD}</NO_READ_ONLY_USER_PASSWORD>
                                <ZAC_CLIENT_SECRET>${ZAC_CLIENT_SECRET}</ZAC_CLIENT_SECRET>
                            </systemPropertyVariables>
                            <excludes>


Add ACSMetering until we figure out how to fix that test?

Was added as part of DE22023

FrankGasparovic · 2017-03-02T17:27:04Z

...on-tests/src/test/java/com/ge/predix/integration/test/ExternalResourceAttributeReaderIT.java

+    private static final String ZONE = "test-zone";
+    private static final String RESOURCE_ID = "aircraftPart/13a71359-db68-4602-aac5-a8fa401c3194";
+    private static final String ASSET_ADAPTER_URI = "http://localhost:8080/v1/attribute/";
+    private static final String ASSET_ATTRIBUTES_JSON =


Store JSON in src/test/resources like we do for other integration tests

Done (however this file has been moved out to another branch)

FrankGasparovic · 2017-03-02T18:49:15Z

.../com/ge/predix/acs/service/policy/evaluation/PolicyEvaluationWithAttributeConnectorTest.java

-import com.ge.predix.acs.attribute.connectors.ResourceAttributeConnector;
-import com.ge.predix.acs.attribute.connectors.SubjectAttributeConnector;
+import com.ge.predix.acs.attribute.readers.ExternalResourceAttributeReader;
+import com.ge.predix.acs.attribute.readers.ExternalSubjectAttributeReader;


Should we stick to calling everything connectors or readers? We have these ExternalResourceAttributeReader and ExternalSubjectAttributeReader, but the test is called PolicyEvaluationWithAttributeConnectorTest.java, and in the JPA story we created the AttributeConnectorEntity @sanjeevchopra. Just seems like we are making it confusing for no reason.

point taken, @FrankGasparovic lets discuss naming tomorrow.

As discussed, we're sticking to the Reader suffix for connector-related internal abstractions and the Connector suffix for external abstractions

irinaepshteyn

These are my initial comments. I will continue reviewing...

irinaepshteyn · 2017-03-02T19:05:58Z

acs-integration-tests/pom.xml

        </profile>

+        <profile>
+            <id>redis</id>


Since we added another profile, please update (or add a story to update) https://github.build.ge.com/predix/acs-ci-config/blob/master/AcsTestsReadme.md

In the past it was decided to place test reports outside of target directory so reports are not removed by 'mvn clean' and retained by Jenkins for each profile. Refer to reportsDirectory and summaryFile in the failsafe plugin configuration for, for example, public profile.
It might not be relevant any longer since each profile is run by the separate Jenkins job now.
We need to decide if we still want to maintain custom output directories and be consistent in every profile.

Added changes to AcsTestsReadme.md in the master branch of the acs-ci-config Git repo. Also these changes to add the redis Maven profile have been pulled out to another branch but when I re-introduce it I'll look into how to standardize the test output directories.

irinaepshteyn · 2017-03-02T19:21:44Z

acs-integration-tests/pom.xml

                            <excludes>
+                                <exclude>**/ExternalResourceAttributeReaderIT.java</exclude>
                                <exclude>**/NonHierarchical*IT.java</exclude>
                            </excludes>


Since for acceptance profile we only execute single ACSAcceptanceIT test, excludes section is not needed. Please remove the whole section.

irinaepshteyn · 2017-03-02T19:52:15Z

service/src/main/java/com/ge/predix/acs/attribute/readers/ExternalResourceAttributeReader.java

+    }
+
+    public Set<Attribute> getAttributesFromAdapter(final String zoneId, final String resourceId) {
+        // TODO: Add an inline call to the Asset Adapter based on configuration


Could we change it to private?

Made it package-private to allow testing

irinaepshteyn · 2017-03-02T19:52:27Z

service/src/main/java/com/ge/predix/acs/attribute/readers/ExternalResourceAttributeReader.java

+    }
+
+    public String getZoneId() {
+        return ((ZoneOAuth2Authentication) SecurityContextHolder.getContext().getAuthentication()).getZoneId();


Could we change it to private?

Made it package-private to allow testing

irinaepshteyn

Additional review comments.

irinaepshteyn · 2017-03-03T01:35:38Z

...e/src/test/java/com/ge/predix/acs/attribute/readers/ExternalResourceAttributeReaderTest.java

+import com.ge.predix.acs.policy.evaluation.cache.InMemoryPolicyEvaluationCache;
+
+public class ExternalResourceAttributeReaderTest {
+


This test is written to test mostly InMemoryAttributeCache and not ExternalResourceAttributeReader. It has to be renamed to com.ge.predix.acs.attribute.cache.InMemoryAttributeCacheTest.
For testing ExternalResourceAttributeReader it should be sufficient to mock cache to return no attributes and mock getAttributesFromAdapter to return a set of attributes and compare what's returned from actual getAttributes.

irinaepshteyn · 2017-03-03T01:42:18Z

...e/src/test/java/com/ge/predix/acs/attribute/readers/ExternalResourceAttributeReaderTest.java

+        Set<Attribute> cachedAttributes = this.externalResourceAttributeReader.getAttributes(RESOURCE_ID);
+
+        for (Attribute cachedAttribute : cachedAttributes) {
+            Assert.assertTrue(attributes.contains(cachedAttribute));


Instead of 'for' loop could use Assert.assertTrue(attributes.containsAll(cachedAttributes)) or Assert.assertEquals(attributes, cachedAttributes);

irinaepshteyn · 2017-03-03T02:16:41Z

...on-tests/src/test/java/com/ge/predix/integration/test/ExternalResourceAttributeReaderIT.java

+        Long ttl = this.redisTemplate.getExpire(resourceAttributesKey, TimeUnit.SECONDS);
+        Assert.assertTrue(ttl > 0L && ttl <= this.attributeCache.getConfiguredTtl());
+    }
+}


We need to add another IT: PolicyEvalCachingWithExternalResourceAttributeReaderIT to assert that policy decision is reevaluated if attributes cache is expired and attributes are refreshed.

This will be added as part of US87156

sanjeevchopra · 2017-03-03T01:36:44Z

service/src/main/java/com/ge/predix/acs/attribute/cache/AbstractAttributeCache.java

+import com.ge.predix.acs.policy.evaluation.cache.AbstractPolicyEvaluationCache;
+
+@Component
+public abstract class AbstractAttributeCache implements AttributeCache {


This class does not need to know the difference between subject of resource. Just (id, attributes)

Both resource and subject connector will have one instance of each of this cache

On that note most of the connector implementation will be subject/resource agnostic.

Discussed this with @anubhavi25 . Added notes in redis caching story.

this pull request will be limited to in-memory caching only

initially a common cache across zones is ok, however we need to limit the storage impact on ACS. Please add 2 env vars - MAX_ATTRIBUTE_CACHE_ENTRIES (default 10K), MAX_ATTRIBUTES_PER_ID (default 100?)

we should adjust the mem allocation to ACS accordingly ~ +1gb (1k per attribute x 100 attributes per user/resource x 10,000 users/resources)

key should be zone+id. One cache each for attribute and subject connectors.

Done. We can make the relevant config changes to bump up the app memory in acs-ci-config as part of US87168. Also, as discussed, Redis-related changes will be pulled out to the US87168 branch.

sanjeevchopra · 2017-03-03T01:40:32Z

service/src/main/java/com/ge/predix/acs/attribute/cache/AttributeCache.java

+
+import com.ge.predix.acs.model.Attribute;
+
+public interface AttributeCache {


Dont see much value in this interface. Just make this the abstract class?

Removed in lieu of AbstractAttributeCache

sanjeevchopra · 2017-03-03T01:49:38Z

service/src/main/java/com/ge/predix/acs/attribute/cache/AbstractAttributeCache.java

+
+    public abstract long getExpiration(String key);
+
+    public abstract long getConfiguredTtl();


please add time unit to the method name

sanjeevchopra · 2017-03-03T01:50:46Z

service/src/main/java/com/ge/predix/acs/attribute/cache/AbstractAttributeCache.java

+
+    public abstract Set<Attribute> get(String key);
+
+    public abstract long getExpiration(String key);


please add time unit to the method name

Done (renamed this to getRemainingTtlInSeconds)

sanjeevchopra · 2017-03-03T01:51:49Z

service/pom.xml

            <artifactId>jedis</artifactId>
            <version>${jedis.version}</version><!--$NO-MVN-MAN-VER$-->
        </dependency>
+        <dependency>


Please check with Anurag on process to get new dependencies approved.

Now using EhCache which, according to Anurag, is under Apache licensing

sanjeevchopra · 2017-03-03T02:04:34Z

...e/src/main/java/com/ge/predix/acs/policy/evaluation/cache/AbstractPolicyEvaluationCache.java

 import java.util.Set;
 import java.util.stream.Collectors;

+import org.codehaus.jackson.map.ObjectMapper;


pls revert changes to this package

sanjeevchopra · 2017-03-03T02:08:32Z

...e/src/test/java/com/ge/predix/acs/attribute/readers/ExternalResourceAttributeReaderTest.java

+        Mockito.doReturn(ZONE).when(this.externalResourceAttributeReader).getZoneId();
+        Set<Attribute> cachedAttributes = this.externalResourceAttributeReader.getAttributes(RESOURCE_ID);
+
+        for (Attribute cachedAttribute : cachedAttributes) {


use equals()

sanjeevchopra · 2017-03-03T02:08:50Z

...e/src/test/java/com/ge/predix/acs/attribute/readers/ExternalResourceAttributeReaderTest.java

+
+    @Test
+    public void testGetAttributes() throws Exception {
+        Set<Attribute> attributes = Collections.singleton(


expectedAttributes

sanjeevchopra · 2017-03-03T02:09:09Z

...e/src/test/java/com/ge/predix/acs/attribute/readers/ExternalResourceAttributeReaderTest.java

+        Mockito.doReturn(attributes).when(this.externalResourceAttributeReader)
+                .getAttributesFromAdapter(ZONE, RESOURCE_ID);
+        Mockito.doReturn(ZONE).when(this.externalResourceAttributeReader).getZoneId();
+        Set<Attribute> cachedAttributes = this.externalResourceAttributeReader.getAttributes(RESOURCE_ID);


actualAttributes

sanjeevchopra · 2017-03-03T02:25:43Z

.../com/ge/predix/acs/service/policy/evaluation/PolicyEvaluationWithAttributeConnectorTest.java

-import com.ge.predix.acs.attribute.connectors.ResourceAttributeConnector;
-import com.ge.predix.acs.attribute.connectors.SubjectAttributeConnector;
+import com.ge.predix.acs.attribute.readers.ExternalResourceAttributeReader;
+import com.ge.predix.acs.attribute.readers.ExternalSubjectAttributeReader;


point taken, @FrankGasparovic lets discuss naming tomorrow.

Addressed feedback and replied to comments

anubhavi25 · 2017-03-06T18:16:06Z

@sanjeevchopra @irinaepshteyn please review again as there were non-trivial changes in the last commit (i.e. removed Redis-related changes, moved to EhCache and incorporated cache-related constraints)

…readers Signed-off-by: Sanjeev <[email protected]> Signed-off-by: Frank <[email protected]>

anubhavi25 assigned sanjeevchopra, irinaepshteyn, FrankGasparovic and anurag-gujral Mar 2, 2017

FrankGasparovic force-pushed the US87160 branch from 9a572b6 to 12e57ba Compare March 2, 2017 16:14

FrankGasparovic reviewed Mar 2, 2017

View reviewed changes

irinaepshteyn requested changes Mar 2, 2017

View reviewed changes

irinaepshteyn previously requested changes Mar 3, 2017

View reviewed changes

sanjeevchopra previously requested changes Mar 3, 2017

View reviewed changes

FrankGasparovic force-pushed the US87160 branch from 12e57ba to 3e9e939 Compare March 6, 2017 17:35

anubhavi25 changed the title ~~US87160: In-memory and Redis cache integration for resource attribute…~~ US87160: In-memory cache integration for resource attribute… Mar 6, 2017

FrankGasparovic force-pushed the US87160 branch from 3e9e939 to c3b442c Compare March 6, 2017 18:11

FrankGasparovic force-pushed the US87160 branch from ad3597f to d7ef139 Compare March 6, 2017 23:51

US87160: Caching-related abstractions for resource/subject attribute …

d7ef139

…readers Signed-off-by: Sanjeev <[email protected]> Signed-off-by: Frank <[email protected]>

anubhavi25 changed the title ~~US87160: In-memory cache integration for resource attribute…~~ US87160: Caching-related abstractions for resource/subject attribute readers Mar 6, 2017

sanjeevchopra merged commit fc76e59 into develop Mar 7, 2017

sanjeevchopra deleted the US87160 branch March 7, 2017 00:18

irinaepshteyn mentioned this pull request Nov 6, 2017

DE50536: ACS: policy evaluation against non-existent acs zone causes 500 response code #174

Merged

		import com.ge.predix.acs.policy.evaluation.cache.InMemoryPolicyEvaluationCache;

		public class ExternalResourceAttributeReaderTest {


		import com.ge.predix.acs.model.Attribute;

		public interface AttributeCache {


		public abstract long getExpiration(String key);

		public abstract long getConfiguredTtl();


		public abstract Set<Attribute> get(String key);

		public abstract long getExpiration(String key);

US87160: Caching-related abstractions for resource/subject attribute readers #93

US87160: Caching-related abstractions for resource/subject attribute readers #93

Uh oh!

Conversation

anubhavi25 commented Mar 2, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

FrankGasparovic left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

anubhavi25 Mar 6, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

FrankGasparovic Mar 2, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

irinaepshteyn left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

irinaepshteyn left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sanjeevchopra Mar 3, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

anubhavi25 commented Mar 2, 2017 •

edited

Loading

anubhavi25 Mar 6, 2017 •

edited

Loading

FrankGasparovic Mar 2, 2017 •

edited

Loading

sanjeevchopra Mar 3, 2017 •

edited

Loading