Thanks to visit codestin.com
Credit goes to github.com

Skip to content

Conversation

@hyunnaye
Copy link
Contributor

@hyunnaye hyunnaye commented Sep 25, 2024

Description
This PR adds the offset and limit parameters and the total number of versions into the getWorkflowVersions endpoint.

Review Instructions
Review that the default limit is 200 and the pagination returns correctly.

Issue
https://ucsc-cgl.atlassian.net/browse/SEAB-5762

Security and Privacy

If there are any concerns that require extra attention from the security team, highlight them here and check the box when complete.

  • Security and Privacy assessed

e.g. Does this change...

  • Any user data we collect, or data location?
  • Access control, authentication or authorization?
  • Encryption features?

Please make sure that you've checked the following before submitting your pull request. Thanks!

  • Check that you pass the basic style checks and unit tests by running mvn clean install
  • Ensure that the PR targets the correct branch. Check the milestone or fix version of the ticket.
  • Follow the existing JPA patterns for queries, using named parameters, to avoid SQL injection
  • If you are changing dependencies, check the Snyk status check or the dashboard to ensure you are not introducing new high/critical vulnerabilities
  • Assume that inputs to the API can be malicious, and sanitize and/or check for Denial of Service type values, e.g., massive sizes
  • Do not serve user-uploaded binary images through the Dockstore API
  • Ensure that endpoints that only allow privileged access enforce that with the @RolesAllowed annotation
  • Do not create cookies, although this may change in the future
  • If this PR is for a user-facing feature, create and link a documentation ticket for this feature (usually in the same milestone as the linked issue). Style points if you create a documentation PR directly and link that instead.

@hyunnaye hyunnaye self-assigned this Sep 25, 2024
@codecov
Copy link

codecov bot commented Sep 25, 2024

Codecov Report

Attention: Patch coverage is 83.01887% with 9 lines in your changes missing coverage. Please review.

Project coverage is 74.46%. Comparing base (4f79d9c) to head (8c92cff).
Report is 18 commits behind head on develop.

Files with missing lines Patch % Lines
.../dockstore/webservice/jdbi/WorkflowVersionDAO.java 75.67% 5 Missing and 4 partials ⚠️
Additional details and impacted files
@@              Coverage Diff              @@
##             develop    #6001      +/-   ##
=============================================
+ Coverage      74.44%   74.46%   +0.02%     
- Complexity      5495     5502       +7     
=============================================
  Files            381      381              
  Lines          19788    19831      +43     
  Branches        2044     2051       +7     
=============================================
+ Hits           14731    14768      +37     
- Misses          4076     4078       +2     
- Partials         981      985       +4     
Flag Coverage Δ
bitbuckettests 26.60% <5.66%> (-0.06%) ⬇️
hoverflytests 27.99% <49.05%> (+0.05%) ⬆️
integrationtests 56.69% <58.49%> (-0.01%) ⬇️
languageparsingtests 11.03% <1.88%> (-0.02%) ⬇️
localstacktests 21.52% <1.88%> (-0.05%) ⬇️
toolintegrationtests 30.35% <73.58%> (+0.32%) ⬆️
unit-tests_and_non-confidential-tests 25.73% <1.88%> (-0.06%) ⬇️
workflowintegrationtests 38.09% <58.49%> (+0.03%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@hyunnaye hyunnaye changed the title openapi SEAB-5762: Implement pagination to getWorkflowVersions endpoint Oct 1, 2024
@hyunnaye hyunnaye marked this pull request as ready for review October 1, 2024 16:03
Comment on lines 455 to 458
if (user.isEmpty() && !workflow.getIsPublished()) {
LOG.error("Unpublished workflow requires authentication");
throw new CustomWebApplicationException("Unpublished workflow requires authentication", HttpStatus.SC_BAD_REQUEST);
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can use checkCanRead which checks if it's published or if the user has access to it and throws an error if not:

/**
* Check if a user is allowed to read the specified entry.
* If not, throw a {@link CustomWebApplicationException}.
* @param user the user to be checked, not set if the user is not logged in
* @param entry entry to be checked
*/
default void checkCanRead(Optional<User> user, Entry<?, ?> entry) {
checkCanRead(user.orElse(null), entry);
}

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Related comment:
Looks like previously, this method called checkCanExamine, which confirms the access necessary to see fields that only owner-type users should be able to see. If we switch to confirming access via checkCanRead or similar, this implies that the endpoint is changing so that it provides access to all users, and perhaps, in certain situations, we'd need to filter out things that everyone shouldn't be able to see, like hidden versions.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

checkCanRead looks to be sufficient as it returns true if the entry is either published or the canExamine function.

Copy link
Contributor

@svonworl svonworl Oct 3, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, checkCanRead works fine, but if you use it, the endpoint will return Versions to anyone if the entry is published. If that's a change from how the endpoint worked before (and it appears like the endpoint previously checked for a user with owner-type access to the entry), we might need to make sure that it doesn't show versions that shouldn't be public, like those that are hidden. (I think.)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for your comment. I added changes to remove hidden versions from being seen to those without access

Comment on lines 455 to 458
if (user.isEmpty() && !workflow.getIsPublished()) {
LOG.error("Unpublished workflow requires authentication");
throw new CustomWebApplicationException("Unpublished workflow requires authentication", HttpStatus.SC_BAD_REQUEST);
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Related comment:
Looks like previously, this method called checkCanExamine, which confirms the access necessary to see fields that only owner-type users should be able to see. If we switch to confirming access via checkCanRead or similar, this implies that the endpoint is changing so that it provides access to all users, and perhaps, in certain situations, we'd need to filter out things that everyone shouldn't be able to see, like hidden versions.

Copy link
Collaborator

@coverbeck coverbeck left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There are several comments already; I'll wait to take a look at the PR after you have responded to them. Please re-request review then.

Copy link
Collaborator

@coverbeck coverbeck left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved, but please do add @Max/min as pointed out by Steve.

Copy link
Contributor

@kathy-t kathy-t left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Might be good to add a test to make sure the permissions are working as expected/hidden versions are hidden since we are now using this endpoint publicly and privately

checkNotNullEntry(workflow);
checkCanExamine(user, workflow);
checkCanRead(user, workflow);
response.addHeader(X_TOTAL_COUNT, String.valueOf(versionDAO.getVersionsCount(workflowId)));
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this count accurate if hidden versions are removed?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I added another query to get the public version count


List<WorkflowVersion> versions = this.workflowVersionDAO.getWorkflowVersionsByWorkflowId(workflow.getId(), VERSION_PAGINATION_LIMIT, 0);
List<WorkflowVersion> versions = this.workflowVersionDAO.getWorkflowVersionsByWorkflowId(workflow.getId(), limit, offset);
if (!canExamine(user.orElse(null), workflow)) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm, I think this means that if a workflow author is viewing the public page of their own workflow, then they will be able to see their hidden versions. Currently, the behaviour is that the hidden versions are hidden for workflow authors as well in the public view.

Not sure if this is worth changing, but noting it.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Think this is worth changing, @hyunnaye please add a follow-up ticket

Copy link
Member

@denis-yuen denis-yuen left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added one comment, hard to review on phone but agree that it is weird to see hidden versions on the public page. With minor exceptions, the public page is meant to be the same for everyone


List<WorkflowVersion> versions = this.workflowVersionDAO.getWorkflowVersionsByWorkflowId(workflow.getId(), VERSION_PAGINATION_LIMIT, 0);
List<WorkflowVersion> versions = this.workflowVersionDAO.getWorkflowVersionsByWorkflowId(workflow.getId(), limit, offset);
if (!canExamine(user.orElse(null), workflow)) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Think this is worth changing, @hyunnaye please add a follow-up ticket

if ("desc".equalsIgnoreCase(sortOrder)) {
query.orderBy(cb.desc(version.get("versionMetadata").get("publicAccessibleTestParameterFile")), cb.desc(version.get("id")));
} else {
query.orderBy(cb.asc(version.get("versionMetadata").get("publicAccessibleTestParameterFile")), cb.asc(version.get("id")));
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit-picky, and you don't have to change, but there's a lot of duplication throughout this method with column names and method invocations. It increases the odds of a typo (each time you type in a name, it is a new chance to spell it wrong, lol). Plus if we ever change anything in the future, it makes it easier to modify if it's only in one place.

You could introduce variables, constants, and/or add method(s) to reduce this.

} else {
query.orderBy(cb.desc(sortPath), cb.desc(version.get("id")));
}
}
Copy link
Contributor

@svonworl svonworl Oct 31, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would recommend trying to streamline this method to reduce the amount of duplicate/similar code. One way would be to calculate a sort path from the various sortCol values, then apply the sort path and order to the query at the end. Something like:

    private List<Predicate> processQuery(String sortCol, String sortOrder, CriteriaBuilder cb, CriteriaQuery query, Root<WorkflowVersion> version) {
        List<Predicate> predicates = new ArrayList<>();

        if (!Strings.isNullOrEmpty(sortCol)) {
            Path<Object> sortPath;
            if ("open".equalsIgnoreCase(sortCol)) {
                sortPath = version.get("versionMetadata").get("publicAccessibleTestParameterFile");
            } else if ("descriptorTypeVersions".equals(sortCol)) {
                sortPath = version.get("versionMetadata").get("descriptorTypeVersions");
            } else {
                boolean hasSortCol = version.getModel()
                    .getAttributes()
                    .stream()
                    .map(Attribute::getName)
                    .anyMatch(sortCol::equals);
                if (!hasSortCol) {
                    LOG.error(INVALID_SORTCOL_MESSAGE);
                    throw new CustomWebApplicationException(INVALID_SORTCOL_MESSAGE,
                            HttpStatus.SC_BAD_REQUEST);

                }
                sortPath = version.get(sortCol);
            }
            if ("asc".equalsIgnoreCase(sortOrder)) {
                query.orderBy(cb.asc(sortPath), cb.asc(version.get("id")));
            } else {
                query.orderBy(cb.desc(sortPath), cb.desc(version.get("id")));
            }
        }
        return predicates;
    }
}

Simplifying the method will make it easier to understand and modify.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done! Thank you

Copy link
Contributor

@svonworl svonworl left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Might add some additional tests that confirm that hidden versions are being excluded, and that the column sort functionality is working correctly.

Copy link
Collaborator

@coverbeck coverbeck left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A couple of minor suggestions... BTW, in case you're unaware, it's really easy to safely introduce with IntelliJ -- just use the "Introduce Variable" refactoring.

Copy link
Contributor

@kathy-t kathy-t left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would be nice to have some tests to check the hidden and sorting functionality

name = "workflowId", description = "id of the worflow", required = true, in = ParameterIn.PATH) @PathParam("workflowId") Long workflowId,
@QueryParam("limit") @Min(1) @Max(MAX_PAGINATION_LIMIT) @DefaultValue(PAGINATION_LIMIT) Integer limit,
@QueryParam("offset") @Min(0) @DefaultValue("0") Integer offset,
@DefaultValue("dbCreateDate") @QueryParam("sortCol") String sortCol,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this expected to work with all the columns in the Versions table?
image

@hyunnaye hyunnaye requested review from kathy-t and svonworl November 12, 2024 18:38
@hyunnaye
Copy link
Contributor Author

Tests added for the hidden and sorting

Copy link
Member

@denis-yuen denis-yuen left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Was a long-running PR but thanks for sticking with it!
(some minor comments/questions)



// Refresh a single version
workflow = workflowsOpenApi.refreshVersion(workflow.getId(), "master", false);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Interesting, I didn't remember that you can refresh specifically one (two) versions of a workflow.
This is a nice way of isolating test cases

@sonarqubecloud
Copy link

@hyunnaye hyunnaye merged commit b604742 into develop Nov 12, 2024
18 of 19 checks passed
@hyunnaye hyunnaye deleted the feature/SEAB-5762 branch November 12, 2024 22:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants