-
Notifications
You must be signed in to change notification settings - Fork 29
SEAB-5762: Implement pagination to getWorkflowVersions endpoint #6001
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## develop #6001 +/- ##
=============================================
+ Coverage 74.44% 74.46% +0.02%
- Complexity 5495 5502 +7
=============================================
Files 381 381
Lines 19788 19831 +43
Branches 2044 2051 +7
=============================================
+ Hits 14731 14768 +37
- Misses 4076 4078 +2
- Partials 981 985 +4
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. |
dockstore-webservice/src/main/java/io/dockstore/webservice/jdbi/WorkflowVersionDAO.java
Outdated
Show resolved
Hide resolved
| if (user.isEmpty() && !workflow.getIsPublished()) { | ||
| LOG.error("Unpublished workflow requires authentication"); | ||
| throw new CustomWebApplicationException("Unpublished workflow requires authentication", HttpStatus.SC_BAD_REQUEST); | ||
| } |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can use checkCanRead which checks if it's published or if the user has access to it and throws an error if not:
Lines 91 to 99 in cec35ec
| /** | |
| * Check if a user is allowed to read the specified entry. | |
| * If not, throw a {@link CustomWebApplicationException}. | |
| * @param user the user to be checked, not set if the user is not logged in | |
| * @param entry entry to be checked | |
| */ | |
| default void checkCanRead(Optional<User> user, Entry<?, ?> entry) { | |
| checkCanRead(user.orElse(null), entry); | |
| } |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Related comment:
Looks like previously, this method called checkCanExamine, which confirms the access necessary to see fields that only owner-type users should be able to see. If we switch to confirming access via checkCanRead or similar, this implies that the endpoint is changing so that it provides access to all users, and perhaps, in certain situations, we'd need to filter out things that everyone shouldn't be able to see, like hidden versions.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
checkCanRead looks to be sufficient as it returns true if the entry is either published or the canExamine function.
Line 209 in 081a733
| return isPublished(entry) || canExamine(user, entry); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes, checkCanRead works fine, but if you use it, the endpoint will return Versions to anyone if the entry is published. If that's a change from how the endpoint worked before (and it appears like the endpoint previously checked for a user with owner-type access to the entry), we might need to make sure that it doesn't show versions that shouldn't be public, like those that are hidden. (I think.)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thank you for your comment. I added changes to remove hidden versions from being seen to those without access
dockstore-webservice/src/main/java/io/dockstore/webservice/resources/WorkflowResource.java
Outdated
Show resolved
Hide resolved
dockstore-webservice/src/main/java/io/dockstore/webservice/resources/WorkflowResource.java
Outdated
Show resolved
Hide resolved
| if (user.isEmpty() && !workflow.getIsPublished()) { | ||
| LOG.error("Unpublished workflow requires authentication"); | ||
| throw new CustomWebApplicationException("Unpublished workflow requires authentication", HttpStatus.SC_BAD_REQUEST); | ||
| } |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Related comment:
Looks like previously, this method called checkCanExamine, which confirms the access necessary to see fields that only owner-type users should be able to see. If we switch to confirming access via checkCanRead or similar, this implies that the endpoint is changing so that it provides access to all users, and perhaps, in certain situations, we'd need to filter out things that everyone shouldn't be able to see, like hidden versions.
coverbeck
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There are several comments already; I'll wait to take a look at the PR after you have responded to them. Please re-request review then.
coverbeck
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Approved, but please do add @Max/min as pointed out by Steve.
dockstore-webservice/src/main/java/io/dockstore/webservice/resources/WorkflowResource.java
Outdated
Show resolved
Hide resolved
dockstore-webservice/src/main/java/io/dockstore/webservice/resources/WorkflowResource.java
Outdated
Show resolved
Hide resolved
kathy-t
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Might be good to add a test to make sure the permissions are working as expected/hidden versions are hidden since we are now using this endpoint publicly and privately
dockstore-webservice/src/main/java/io/dockstore/webservice/resources/WorkflowResource.java
Outdated
Show resolved
Hide resolved
| checkNotNullEntry(workflow); | ||
| checkCanExamine(user, workflow); | ||
| checkCanRead(user, workflow); | ||
| response.addHeader(X_TOTAL_COUNT, String.valueOf(versionDAO.getVersionsCount(workflowId))); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is this count accurate if hidden versions are removed?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I added another query to get the public version count
|
|
||
| List<WorkflowVersion> versions = this.workflowVersionDAO.getWorkflowVersionsByWorkflowId(workflow.getId(), VERSION_PAGINATION_LIMIT, 0); | ||
| List<WorkflowVersion> versions = this.workflowVersionDAO.getWorkflowVersionsByWorkflowId(workflow.getId(), limit, offset); | ||
| if (!canExamine(user.orElse(null), workflow)) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hmm, I think this means that if a workflow author is viewing the public page of their own workflow, then they will be able to see their hidden versions. Currently, the behaviour is that the hidden versions are hidden for workflow authors as well in the public view.
Not sure if this is worth changing, but noting it.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Think this is worth changing, @hyunnaye please add a follow-up ticket
denis-yuen
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Added one comment, hard to review on phone but agree that it is weird to see hidden versions on the public page. With minor exceptions, the public page is meant to be the same for everyone
dockstore-webservice/src/main/java/io/dockstore/webservice/resources/WorkflowResource.java
Outdated
Show resolved
Hide resolved
|
|
||
| List<WorkflowVersion> versions = this.workflowVersionDAO.getWorkflowVersionsByWorkflowId(workflow.getId(), VERSION_PAGINATION_LIMIT, 0); | ||
| List<WorkflowVersion> versions = this.workflowVersionDAO.getWorkflowVersionsByWorkflowId(workflow.getId(), limit, offset); | ||
| if (!canExamine(user.orElse(null), workflow)) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Think this is worth changing, @hyunnaye please add a follow-up ticket
dockstore-webservice/src/main/java/io/dockstore/webservice/jdbi/WorkflowVersionDAO.java
Outdated
Show resolved
Hide resolved
dockstore-webservice/src/main/java/io/dockstore/webservice/jdbi/WorkflowVersionDAO.java
Outdated
Show resolved
Hide resolved
dockstore-webservice/src/main/java/io/dockstore/webservice/jdbi/WorkflowVersionDAO.java
Show resolved
Hide resolved
| if ("desc".equalsIgnoreCase(sortOrder)) { | ||
| query.orderBy(cb.desc(version.get("versionMetadata").get("publicAccessibleTestParameterFile")), cb.desc(version.get("id"))); | ||
| } else { | ||
| query.orderBy(cb.asc(version.get("versionMetadata").get("publicAccessibleTestParameterFile")), cb.asc(version.get("id"))); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nit-picky, and you don't have to change, but there's a lot of duplication throughout this method with column names and method invocations. It increases the odds of a typo (each time you type in a name, it is a new chance to spell it wrong, lol). Plus if we ever change anything in the future, it makes it easier to modify if it's only in one place.
You could introduce variables, constants, and/or add method(s) to reduce this.
| } else { | ||
| query.orderBy(cb.desc(sortPath), cb.desc(version.get("id"))); | ||
| } | ||
| } |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Would recommend trying to streamline this method to reduce the amount of duplicate/similar code. One way would be to calculate a sort path from the various sortCol values, then apply the sort path and order to the query at the end. Something like:
private List<Predicate> processQuery(String sortCol, String sortOrder, CriteriaBuilder cb, CriteriaQuery query, Root<WorkflowVersion> version) {
List<Predicate> predicates = new ArrayList<>();
if (!Strings.isNullOrEmpty(sortCol)) {
Path<Object> sortPath;
if ("open".equalsIgnoreCase(sortCol)) {
sortPath = version.get("versionMetadata").get("publicAccessibleTestParameterFile");
} else if ("descriptorTypeVersions".equals(sortCol)) {
sortPath = version.get("versionMetadata").get("descriptorTypeVersions");
} else {
boolean hasSortCol = version.getModel()
.getAttributes()
.stream()
.map(Attribute::getName)
.anyMatch(sortCol::equals);
if (!hasSortCol) {
LOG.error(INVALID_SORTCOL_MESSAGE);
throw new CustomWebApplicationException(INVALID_SORTCOL_MESSAGE,
HttpStatus.SC_BAD_REQUEST);
}
sortPath = version.get(sortCol);
}
if ("asc".equalsIgnoreCase(sortOrder)) {
query.orderBy(cb.asc(sortPath), cb.asc(version.get("id")));
} else {
query.orderBy(cb.desc(sortPath), cb.desc(version.get("id")));
}
}
return predicates;
}
}
Simplifying the method will make it easier to understand and modify.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done! Thank you
svonworl
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Might add some additional tests that confirm that hidden versions are being excluded, and that the column sort functionality is working correctly.
dockstore-webservice/src/main/java/io/dockstore/webservice/jdbi/WorkflowVersionDAO.java
Outdated
Show resolved
Hide resolved
coverbeck
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
A couple of minor suggestions... BTW, in case you're unaware, it's really easy to safely introduce with IntelliJ -- just use the "Introduce Variable" refactoring.
dockstore-webservice/src/main/java/io/dockstore/webservice/jdbi/WorkflowVersionDAO.java
Outdated
Show resolved
Hide resolved
dockstore-webservice/src/main/java/io/dockstore/webservice/jdbi/WorkflowVersionDAO.java
Outdated
Show resolved
Hide resolved
kathy-t
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Would be nice to have some tests to check the hidden and sorting functionality
dockstore-webservice/src/main/java/io/dockstore/webservice/resources/WorkflowResource.java
Outdated
Show resolved
Hide resolved
dockstore-webservice/src/main/java/io/dockstore/webservice/resources/WorkflowResource.java
Outdated
Show resolved
Hide resolved
dockstore-webservice/src/main/java/io/dockstore/webservice/jdbi/WorkflowVersionDAO.java
Outdated
Show resolved
Hide resolved
| name = "workflowId", description = "id of the worflow", required = true, in = ParameterIn.PATH) @PathParam("workflowId") Long workflowId, | ||
| @QueryParam("limit") @Min(1) @Max(MAX_PAGINATION_LIMIT) @DefaultValue(PAGINATION_LIMIT) Integer limit, | ||
| @QueryParam("offset") @Min(0) @DefaultValue("0") Integer offset, | ||
| @DefaultValue("dbCreateDate") @QueryParam("sortCol") String sortCol, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
|
Tests added for the hidden and sorting |
dockstore-integration-testing/src/test/java/io/dockstore/client/cli/OpenAPIGeneralIT.java
Outdated
Show resolved
Hide resolved
dockstore-integration-testing/src/test/java/io/dockstore/client/cli/OpenAPIGeneralIT.java
Outdated
Show resolved
Hide resolved
dockstore-integration-testing/src/test/java/io/dockstore/client/cli/OpenAPIGeneralIT.java
Show resolved
Hide resolved
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Was a long-running PR but thanks for sticking with it!
(some minor comments/questions)
|
|
||
|
|
||
| // Refresh a single version | ||
| workflow = workflowsOpenApi.refreshVersion(workflow.getId(), "master", false); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Interesting, I didn't remember that you can refresh specifically one (two) versions of a workflow.
This is a nice way of isolating test cases
dockstore-webservice/src/main/java/io/dockstore/webservice/jdbi/WorkflowVersionDAO.java
Show resolved
Hide resolved
dockstore-webservice/src/main/java/io/dockstore/webservice/resources/WorkflowResource.java
Outdated
Show resolved
Hide resolved
|
Description
This PR adds the
offsetandlimitparameters and the total number of versions into thegetWorkflowVersionsendpoint.Review Instructions
Review that the default limit is 200 and the pagination returns correctly.
Issue
https://ucsc-cgl.atlassian.net/browse/SEAB-5762
Security and Privacy
If there are any concerns that require extra attention from the security team, highlight them here and check the box when complete.
e.g. Does this change...
Please make sure that you've checked the following before submitting your pull request. Thanks!
mvn clean install@RolesAllowedannotation