feat: Implement unified pagination and add template versions support #1308

mafredri · 2022-05-05T12:18:21Z

This PR adds support for pagination to the /api/v2/templates/:id/versions endpoint whilst breaking out some of the common functionality that can be shared between endpoints that will support pagination. As such, the /api/v2/users endpoint also received a small refactor.

After a discussion with the team, I settled on using embedded structs for the pagination options. We may want to consider using pointers &codersdk.Pagination{} for being explicit when they're not set, but since the zero values work well it should compose better in more use cases (e.g. req := codersdk.MyRequest{}; req.Limit = 10). Among other API designs considered were functional options and and optional arguments inside optional structs.

I decided to omit support for search in the template versions endpoint since I was uncertain if this will be needed. If we want to support it, we may want to consider support for defining the column/field to be searched (e.g. should search be applied to name, description or both).

Questions:

@Emyrk Currently using both offset and after_id are supported, as per your design. What are your thoughts about splitting these up? We could for instance create two separate types: type PaginateOffset struct and type PaginateID struct, an endpoint can choose to support either or both by embedding (example):

type MyEndpointRequest struct {
        MyParam int
        PaginateOffset
        PaginateID
}

Benefits of splitting them up would be to allow an endpoint to use a lesser feature set, but maybe there's a use-case for always supporting both?

Misc thoughts:

In codersdk it would make sense to me to rename all .*Request structs to .*Params since they define the parameters that can be used for the Client method(s). My reasoning is that we're doing a request by invoking the client method, the params only define the arguments.
Should we start to consider adding support for ordering by a specified column as well? (This would be left up to future exploration)

Fixes #53

codecov · 2022-05-05T12:21:00Z

Codecov Report

Merging #1308 (aab400c) into main (dc115b8) will increase coverage by 0.01%.
The diff coverage is 59.48%.

@@            Coverage Diff             @@
##             main    #1308      +/-   ##
==========================================
+ Coverage   66.81%   66.82%   +0.01%     
==========================================
  Files         284      282       -2     
  Lines       18633    18493     -140     
  Branches      235      235              
==========================================
- Hits        12449    12358      -91     
+ Misses       4903     4871      -32     
+ Partials     1281     1264      -17

Flag	Coverage Δ
unittest-go-macos-latest	`54.07% <52.58%> (+0.13%)`	⬆️
unittest-go-postgres-	`65.59% <54.31%> (+0.07%)`	⬆️
unittest-go-ubuntu-latest	`56.36% <52.58%> (-0.03%)`	⬇️
unittest-go-windows-2022	`?`
unittest-js	`73.67% <ø> (ø)`

Impacted Files	Coverage Δ
scripts/apitypings/main.go	`0.00% <0.00%> (ø)`
coderd/pagination.go	`51.35% <51.35%> (ø)`
coderd/templates.go	`64.24% <76.92%> (+1.95%)`	⬆️
coderd/users.go	`61.44% <88.88%> (+0.81%)`	⬆️
coderd/database/queries.sql.go	`77.90% <100.00%> (+0.06%)`	⬆️
codersdk/client.go	`59.74% <100.00%> (ø)`
codersdk/pagination.go	`100.00% <100.00%> (ø)`
codersdk/templates.go	`60.37% <100.00%> (ø)`
codersdk/users.go	`63.79% <100.00%> (-0.62%)`	⬇️
cli/configssh.go	`62.58% <0.00%> (-6.48%)`	⬇️
... and 15 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update dc115b8...aab400c. Read the comment docs.

coderd/templates_test.go

kylecarbs

Looks good to me! Just two minor comments 😎

coderd/database/queries/templateversions.sql

coderd/httpapi/httpapi.go

coderd/templates.go

codersdk/pagination.go

presleyp · 2022-05-05T14:42:19Z

site/src/api/typesGenerated.ts

  readonly status: string
+  readonly pagination: Pagination


@BrunoQuaresma just want to make sure you see this and get a chance to discuss it if needed

presleyp · 2022-05-05T14:49:36Z

site/src/api/typesGenerated.ts

+export interface Pagination {
+  readonly after_id: string
+  readonly limit: number
+  readonly offset: number


Does this mean we send either after_id or limit and offset? I think typescript will complain if we leave anything out of this type, so can we send 0 or empty string for the ones we're not using?

@presleyp The way this is (currently) implemented means you can set any combination of them (including setting all or none). I don't know if we want to keep it that way though.

@Emyrk do you know if there's a way we can make the generator express the above? (E.g. would setting ? all of these, like readonly after_id?: string, work?)

Yeah I'm concerned that if the backend expects "just omit the ones you don't want to use" but the frontend enforces "these are all required" then we might have trouble sending well formed requests.

Yes I agree, that would be super inconvenient! I see pointers and a few other null types are marked as optional by the TypeScript generator, but not others. I could take a look at improving that. Perhaps using the json omitempty option (Go specific) could also be marked optional in TypeScript.

I don't think both after_id and offset make sense together. And, I'll argue that we should drop offset entirely in favor of after_id.

The problem with offset is that it is not safe to race conditions that add or delete rows. Say we have {a0, a1, a2, b0, b1, b2} as our data. I paginate with limit=3 offset=0 and get {a0, a1, a2}. Now someone else deletes a1. Then I call paginate with limit=3 offset=3. I get {b1, b2}!

You might argue that in some use cases, it doesn't matter if we duplicate or omit data, so why not give both options? The problem is that it's really hard to be sure we've communicated the danger about offset to potential users of this API. They might not read the docstrings. This API might end up being offered to end users outside of Coder. That offset is so tempting because the code is easier to write than after_id, but we're setting users up to shoot themselves in the foot.

I agree with this. It's also why I suggested separating the options (offset and after_id) into two structs in my PR description. But since adding both was deliberate (#1057 (comment)) I don't think we should remove it in this PR (at least not for the users endpoint). Personally I don't see much of a usability difference between the two and would prefer only having after_id (or another solution entirely).

coderd/database/queries/templateversions.sql

spikecurtis · 2022-05-05T16:59:53Z

codersdk/pagination.go

+// Pagination sets pagination options for the endpoints that support it.
+type Pagination struct {
+	// AfterID returns all or up to Limit results after the given
+	// UUID. This option can be used with or as an alternative to


I don't understand the use case for using both AfterID and Offset. I would have thought they should be mutually exclusive.

I agree, I'd be in favor of dropping offset if there's no use-case for it.

I found some context in an older PR: #1057 (comment)

@presleyp @BrunoQuaresma Can the frontend just use cursor based pagination? How much harder would it be to support over offset

I wouldn't argue for offset-based pagination over cursor-based pagination based on how hard it is to implement on the frontend, but...we don't need pagination on the frontend in CE so we are currently ignoring it, sending no limit or offset params, which I believe works because the offset-based pagination defaults are the same as if it wasn't paginated. Could cursor-based pagination be similarly ignored?

Cursor-based pagination is similarly ignored if not set in the implementation I'm reading.

I opened up an issue/feature for this: #1358

spikecurtis · 2022-05-05T17:12:28Z

site/src/api/typesGenerated.ts

+export interface Pagination {
+  readonly after_id: string
+  readonly limit: number
+  readonly offset: number


I don't think both after_id and offset make sense together. And, I'll argue that we should drop offset entirely in favor of after_id.

The problem with offset is that it is not safe to race conditions that add or delete rows. Say we have {a0, a1, a2, b0, b1, b2} as our data. I paginate with limit=3 offset=0 and get {a0, a1, a2}. Now someone else deletes a1. Then I call paginate with limit=3 offset=3. I get {b1, b2}!

You might argue that in some use cases, it doesn't matter if we duplicate or omit data, so why not give both options? The problem is that it's really hard to be sure we've communicated the danger about offset to potential users of this API. They might not read the docstrings. This API might end up being offered to end users outside of Coder. That offset is so tempting because the code is easier to write than after_id, but we're setting users up to shoot themselves in the foot.

spikecurtis · 2022-05-05T17:14:26Z

codersdk/pagination.go

+	// Offset for better performance. To use it as an alternative,
+	// set AfterID to the last UUID returned by the previous
+	// request.
+	AfterID uuid.UUID `json:"after_id"`


This locks us into having a UUID for every database table that we want to paginate. Have we fully considered the consequences?

I had the same thought, but at least all existing tables seem to be using UUID for id.

@coder/backend Does anyone else have input on this? Steven seems to be on PTO so can't check in with him.

This can easily be changed to a string or something if we ever want to support different ID types in the future so I think it's fine for now

spikecurtis · 2022-05-05T17:16:26Z

coderd/database/queries/templateversions.sql

+					FROM
+						template_versions
+					WHERE
+						id = @after_id


What happens if the after_id doesn't exist? Either because user modified a URL, or because someone deleted the row?

No results, which is acceptable in the former scenario. As for the latter, good catch. Not sure if this has been thought of.

I opened up an issue for this #1357 (also relevant #1358).

deansheather

LGTM

…rsdk sturct

…gs (#1318) * feat: Add support for json omitempty to apitypings * feat: Express embedded structs via extends in TypeScript * Handle unembedding via json field in apitypings * Add scripts/apitypings/main.go to Makefile

…1308) * feat: Implement pagination for template versions * feat: Use unified pagination between users and template versions * Sync codepaths between users and template versions * Create requestOption type in codersdk and add test * Fix created_at edge case for pagination cursor in queries * feat: Add support for json omitempty and embedded structs in apitypings (#1318) * Add scripts/apitypings/main.go to Makefile

mafredri requested a review from a team as a code owner May 5, 2022 12:18

mafredri self-assigned this May 5, 2022

mafredri requested review from BrunoQuaresma, coadler, kylecarbs, spikecurtis, a team and Emyrk May 5, 2022 12:18

mafredri commented May 5, 2022

View reviewed changes

coderd/templates_test.go Show resolved Hide resolved

kylecarbs approved these changes May 5, 2022

View reviewed changes

coderd/database/queries/templateversions.sql Outdated Show resolved Hide resolved

coderd/httpapi/httpapi.go Outdated Show resolved Hide resolved

coadler approved these changes May 5, 2022

View reviewed changes

coderd/templates.go Outdated Show resolved Hide resolved

codersdk/pagination.go Outdated Show resolved Hide resolved

presleyp reviewed May 5, 2022

View reviewed changes

spikecurtis reviewed May 5, 2022

View reviewed changes

deansheather approved these changes May 5, 2022

View reviewed changes

mafredri mentioned this pull request May 6, 2022

feat: Add support for json omitempty and embedded structs in apitypings #1318

Merged

mafredri added 13 commits May 10, 2022 07:29

feat: Implement pagination for template versions

475dcf9

Use codersdk.Pagination in users endpoint

01a52d1

Avoid user sort side-effect in databasefake

d1478a9

Return sql.ErrNoRows for GetUsers in databasefake

7a36610

Sync codepaths in databasefake

61c60c6

Fix test with better handling of sql.ErrNoRows in coderd

62fa273

Remove an unused require.NoError

878d633

Return empty list for sql.ErrNoRows in coderd users endpoint

5c840fd

Add t.Parallel() to sub tests

07e590c

Reinit tt due to parallel test

be8ba6c

Remove unused variable

5ab9ad5

Fix copy pasta in query comments

e2a3741

Move ParsePagination from httpapi to coderd and unexport, return code…

61410a5

…rsdk sturct

mafredri added 7 commits May 10, 2022 07:29

codersdk: Create requestOption type

8a67746

codersdk: Add test for Pagination.asRequestOption

55028d2

coderd: Handle http response errors in parsePagination

ea2371e

codersdk: Use parallel test

66af4ec

Fix created_at edge case for pagination cursor in queries

067f912

Fix copy paste issue

b8be7a8

Run make gen

13fc406

mafredri force-pushed the mafredri/feat-paginate-template-versions-endpoint branch from 04b3b1f to 13fc406 Compare May 10, 2022 07:30

mafredri enabled auto-merge (squash) May 10, 2022 07:42

mafredri merged commit 2d3dc43 into main May 10, 2022

mafredri deleted the mafredri/feat-paginate-template-versions-endpoint branch May 10, 2022 07:44

mafredri changed the title ~~feat: Implement pagination for template versions~~ feat: Implement unified pagination and add template versions support May 10, 2022

This was referenced May 10, 2022

Bug: Pagination breaks if the row used as cursor is deleted #1357

Closed

Feat: Use only one standardized method for pagination across the API #1358

Closed

misskniss added the V2 BETA label May 15, 2022

misskniss added this to the V2 Beta milestone May 15, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: Implement unified pagination and add template versions support #1308

feat: Implement unified pagination and add template versions support #1308

mafredri commented May 5, 2022 •

edited

Loading

codecov bot commented May 5, 2022 •

edited

Loading

kylecarbs left a comment

presleyp May 5, 2022

presleyp May 5, 2022

mafredri May 5, 2022

presleyp May 5, 2022

mafredri May 5, 2022

spikecurtis May 5, 2022

mafredri May 5, 2022 •

edited

Loading

spikecurtis May 5, 2022

mafredri May 5, 2022

mafredri May 5, 2022

deansheather May 5, 2022

presleyp May 6, 2022

spikecurtis May 6, 2022

mafredri May 10, 2022

spikecurtis May 5, 2022

spikecurtis May 5, 2022

mafredri May 5, 2022

deansheather May 5, 2022

spikecurtis May 5, 2022

mafredri May 5, 2022

mafredri May 10, 2022

deansheather left a comment

feat: Implement unified pagination and add template versions support #1308

feat: Implement unified pagination and add template versions support #1308

Conversation

mafredri commented May 5, 2022 • edited Loading

codecov bot commented May 5, 2022 • edited Loading

Codecov Report

kylecarbs left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mafredri May 5, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

deansheather left a comment

Choose a reason for hiding this comment

mafredri commented May 5, 2022 •

edited

Loading

codecov bot commented May 5, 2022 •

edited

Loading

mafredri May 5, 2022 •

edited

Loading