perf: cap count queries and emit native UUID comparisons for audit/connection logs - #23835
Conversation
594783e to
a9036cc
Compare
a9036cc to
53fe52a
Compare
…nnection logs When audit_logs and connection_logs tables grow to tens of millions of rows, they start causing two performance bottlenecks: 1. COUNT(*) queries become extremely slow due to full sequential scans. Solution: wrap count queries in a subquery with LIMIT 2001 so PostgreSQL stops scanning early. The frontend infers capping from count > 2000 and displays "of 2,000+". 2. Authorized queries (both COUNT and SELECT) are slowed down by RBAC authorization that emits text-based UUID comparisons like 'uuid' = COALESCE(organization_id::text, '') which prevents index usage. Solution: add a UUIDVarMatcher type that emits `organization_id = 'uuid'::uuid` instead, allowing PostgreSQL to use indexes on the organization_id column. Related to: https://linear.app/codercom/issue/PLAT-31/connectionaudit-log-performance-issue
53fe52a to
aacd95e
Compare
|
I like the creativity here. Losing the total count is unfortunate. But probably not a big deal? |
I honestly don't see a way around it. COUNT(*) queries are going to be slow on lots of rows, no matter what. |
|
The frontend count-capping logic is generic - it will display "of 2,000+" for any paginated query returning count > 2000, even if the backend count isn't capped. In practice this only affects audit logs and connection logs since no other table approaches that size? But if that feels too broad, I can add a EDIT: pushed an update that makes count capping opt-in via |
aslilac
left a comment
There was a problem hiding this comment.
I think we'll really want some stories to look at to make sure that the math on the frontend is correct, but the backend changes look good
Also add a test for hasPreviousPage when count is capped.
The frontend used a hardcoded COUNT_CAP = 2000 constant to detect capped counts, which would incorrectly cap any paginated endpoint returning >2000 records. Instead, have the backend include count_cap in the response so the frontend only activates capping when the endpoint explicitly signals it. The SQL LIMIT is also parameterized so the cap value flows from a named Go constant through to the query.
…(fixes GroupPage story crash)
When audit_logs and connection_logs tables grow to tens of millions of rows, two performance bottlenecks can cause page timeouts:
Unbounded count queries: COUNT(*) queries do full sequential scans.
Solution: wrap count queries in a subquery with
LIMIT count_cap + 1so PostgreSQL stops scanning early. The backend includescount_capin the response; the frontend detects capping whencount > count_capand displays "... of N+".Text-based UUID comparisons in RBAC filters: authorized queries (both COUNT and SELECT) are slowed by RBAC emitting
'uuid' = COALESCE(organization_id::text, '')which prevents index usage.Solution: add a
UUIDVarMatchertype that emitsorganization_id = 'uuid'::uuidinstead, allowing PostgreSQL to use indexes on the organization_id column. This improvement is most significant at medium offsets where the index saves substantial work; at small offsets PG finds rows quickly regardless, and at very large offsets PG switches to a sequential scan anyway.Audit/connection log page load times without filters (tested with 100M+ rows in PG16 on Core Ultra 7 268V, 32GB, SSD):
Related to:
https://linear.app/codercom/issue/PLAT-31/connectionaudit-log-performance-issue