fix: add distributed protection for external auth refresh - #27479
Conversation
84a83c7 to
73c7af1
Compare
73c7af1 to
b499ccc
Compare
|
/coder-agents-review |
|
Chat: Review posted | View chat Review history
deep-review v0.9.0 | Round 1 | Last posted: Round 1, 6 findings (1 P2, 1 P3, 3 Nit, 1 Note), COMMENT. Review Finding inventoryFinding inventory: PR 27479Findings
Round logRound 1Netero-only (P2 present, panel deferred). 1 P2, 1 P3, 3 Nit, 1 Note. Reviewed against 2f34e1a..b499ccc. About deep-reviewCRF = Coder Review Finding (P0-P4, Nit, Note)
|
There was a problem hiding this comment.
This is a first-pass review only: these are mechanical findings from Netero, the full review panel has not yet reviewed this PR. The panel will review after these findings are addressed.
The lease design is solid: the acquire/release queries have accurate conditional-update semantics, the deferred errors.Join on release surfaces release failures (and is tested), all new code uses dbtime.Now(), and the removal of UpdateExternalAuthLinkRefreshToken left no stale references across dbauthz, dbcrypt, dbmock, and dbmetrics. Test density is 64.6% with real DB-state assertions.
Severity count: 1 P2, 1 P3, 3 Nit, 1 Note.
A quote from the first pass: "The failure the winner recorded is silently converted into a false success for every waiting replica."
🤖 This review was automatically generated with Coder Agents.
Before refreshing, the caller has to put a lease on the row to prevent any other replicas from trying to refresh using the same token. Callers unable to get a lease enter a holding pattern waiting for the results. Also use this same lease query to re-read the link. This more or less functions the same as the re-read we used to have after a refresh failure, except it also allows us to avoid making the failed request in the first place. The downside is that we have to refetch the link every time even if the caller just fetched it, but since we have to make a database call anyway...
b174d96 to
7741c7e
Compare
If we have to refresh, we do still validate while holding the lease, since we have it anyway. But if we do not need to refresh, validate without getting a lease. The assumption is that the overhead to get the lease is greater than the overhead of concurrent validations.
Previously if the token changed (including getting removed) we just returned it, but it could have errored.
This lets us distinguish between the row not existing and not being able to take the lease because something else has it.
7741c7e to
b59ef04
Compare
|
@jeremyruppel to give an overview of the work and changes so far:
|
|
@code-asher I discussed this with an agent and we found two race conditiony things. I'm going to have the agent post the comments here, but I did discuss these at length so I'm really using this to pass on the excruciating details |
Also switch to a SQL function instead of the lock pattern.
New migrations have since come in on main.
|
I fixed the clock drift issue by moving the lease computation into the query and and I added the extra lease guard for the release step just in case. While I was refactoring those queries, I went in a bit of a different direction with the lock. The reason for the lock is to differentiate between something already having the lease and the row no longer existing, so I created a sql function instead to do that work. No need for the lock anymore. Postgres's default isolation level ensures that two |
d312290 to
4c9fe79
Compare
4c9fe79 to
5a1de17
Compare
Before refreshing, the caller has to put a lease on the row to prevent any other replicas from trying to refresh using the same token. Callers unable to get a lease enter a holding pattern waiting for the results.
Also use this same lease query to re-read the link. This more or less functions the same as the re-read we used to have after a refresh failure, except it also allows us to avoid making the failed request in the first place. The downside is that we have to refetch the link every time even if the caller just fetched it, but since we have to make a database call anyway...