perf(coderd/rbac): collapse org authorization to a set-membership test - #27244
Merged
Conversation
jeremyruppel
commented
Jul 14, 2026
jeremyruppel
left a comment
Contributor
Author
There was a problem hiding this comment.
Nice, surgical fix. Keeping the unknown org_owner in a single positive set-membership test (and folding org-deny into a ground set difference) is exactly the discipline POLICY.md prescribes, and it makes the O(N) partial-eval fan-out structurally impossible rather than merely unlikely. The not org = -1 gates were preserved rather than deleted, keeping the diff auditable, and the new test genuinely exercises both the full and partial/SQL paths.
Independent verification across reviewers found no over-authorization: the full/partial-eval equivalence holds across all nine {org, member} vote combinations, and the intermediate -1 -> 0 change is not observable outside rego (authz.go only queries data.authz.allow).
Findings are all P3 / non-blocking, across 4 inline comments: an untested scope-level deny fold, a now-implicit "known-org org vote is never -1" invariant (plus POLICY.md drift), a stale doc comment, and a few high-value test rows to add.
Two notes that don't map to changed lines:
- P3 (Database Reviewer): the list endpoints this PR targets use
StringVarMatcher("organization_id :: text", ...)(coderd/rbac/regosql/configs.go), so the residual compiles toorganization_id :: text = ANY(ARRAY[...]). The left-side cast defeats a UUID index onorganization_id, and the array is now larger. This is pre-existing (the old N-residual form had the same cast) and still a large net win, but it caps the DB-side ceiling for exactly these endpoints; worth a follow-up to evaluate a UUID matcher or expression index. - Resolved during cross-check: the Edge Case concern that
org_membercallingorg_ids_with_votetwice compounds the O(N^2) map build is neutralized — the Performance Analyst verified OPA v1.18.1 memoizes ground-arg function calls viavirtualCache, socheck_all_org_permissions(roles, "org")runs once. The remaining O(N^2) build is the known, already-tracked follow-up.
Member
|
Ran some benchmarks on this branch vs TL;DR: big wins at high org counts (
Memory follows the same shape: Full benchstat output (sec/op, B/op, allocs/op)Methodology: Coder Agents on behalf of @Emyrk. |
Member
|
Follow-up with the memory side of the comparison (same runs as above).
(All p=0.002, n=6.) Same pattern as runtime: Coder Agents on behalf of @Emyrk. |
Emyrk
added a commit
that referenced
this pull request
Jul 15, 2026
<!-- Authored with Coder Agents on behalf of @Emyrk --> Adds `BenchmarkRBACManyOrgs` to measure `Authorize`, `Prepare` (partial evaluation), and `Prepare`+`CompileToSQL` as a subject's org-membership count grows (1, 5, 10, 50, 100 orgs). - Written to evaluate the org set-membership rewrite in #27244, where partial-eval cost scales with org count. - Subject uses pre-expanded cached roles (`WithCachedASTValue`), member + per-org `organization-member` roles, `ScopeAll`; authorizer has no cache so each iteration measures a real evaluation. Results comparing `main` vs #27244 are posted on that PR. <sub>Coder Agents on behalf of @Emyrk.</sub>
jeremyruppel
force-pushed
the
jeremy/devex-608-org-setmembership
branch
from
July 15, 2026 19:08
8f4080e to
b4d0eb9
Compare
Contributor
Author
|
Memory side of the comparison, re-run after the rebase (
(All p=0.002, n=6.) After memoizing the org vote maps, the earlier single-org Coder Agents on behalf of @jeremyruppel. |
Contributor
Author
|
Re-ran the runtime benchmarks on the rebased branch vs TL;DR: big wins at high org counts (
Memory follows the same shape: Full benchstat output (sec/op, B/op, allocs/op)Coder Agents on behalf of @jeremyruppel. |
The known-org authorization path indexed an N-entry vote map by the object's org id, which is unknown during partial evaluation. Indexing by an unknown forces OPA to emit one residual query per org membership, so Prepare did O(N) work per request (~4s and up for users in hundreds of orgs), driving slow list endpoints for many-org users. Rewrite the known-org check to test the object's org id for membership in a set that is fully known at partial-evaluation time, which collapses to a single 'organization_id = ANY(ARRAY[...])' residual. The known-org clause now only votes to allow; org-level denies are folded into the org-member level as a set difference (member-allow minus org-deny), preserving exact semantics while keeping the unknown org id out of any map index. Add an OrgDenyBlocksMember case to TestAuthorizeLevels covering the org-level deny gate on both the full- and partial-evaluation paths.
…asymmetry Address review feedback on the org set-membership change: - Expand OrgDenyBlocksMember with member-level deny-wins, org-level allow overriding member-level deny, and action-scoped org deny cases. - Add ScopeOrgDenyBlocksMember to exercise the scope_org_member deny fold, which built-in scopes never reach. - Document the known-org asymmetry in POLICY.md (known-org org level never votes -1; deny is folded into org_member) and annotate the now-vestigial not org = -1 gates. - Refresh the stale check_all_org_permissions doc comment that described the removed org-id-keyed lookup.
…ression The set-membership rewrite emitted an unsatisfiable `org_owner in set()` residual for every org/scope org-level branch whose allow set was empty (e.g. an org-member role with no org-level permission, or a site-wildcard scope). OPA does not fold membership in an empty set to false, so each dead branch still cost a PrepareForEval, inflating Prepare at low org counts (+115% at 1 org) even though high org counts improved. Guard each membership test with a ground `count(set) > 0`: when the set is empty the guard is ground-false and partial evaluation drops the branch instead of emitting a dead residual; when non-empty it is ground-true and drops out, leaving just the membership residual. Residual count for a workspace read is now a flat 5 regardless of org count (was 6 -> 505 on main, 21 flat before this guard). Benchmarks (this branch vs main): Prepare/orgs=1 +115% -> +10%, Prepare/orgs=100 still -78% (allocs -90%).
Hoist the per-org vote maps into memoized zero-arg complete rules (role_org_votes, role_member_votes, scope_org_votes, scope_member_votes) instead of recomputing them through parametrized functions at every call site. OPA evaluates and caches complete rules once per query, while a function is re-evaluated on each call, so the org, member, and scope paths previously rebuilt the same vote map on every authorization check. The known-org clauses now read from these maps. This removes the single-org partial-evaluation regression (Prepare/Compile at orgs=1 go from slower than main to faster) while keeping the large wins at high org counts, and leaves the set-membership partial-eval shape unchanged.
jeremyruppel
force-pushed
the
jeremy/devex-608-org-setmembership
branch
from
July 16, 2026 19:20
b4d0eb9 to
14b02e3
Compare
Contributor
Author
This stack of pull requests is managed by Graphite. Learn more about stacking. |
This was referenced Jul 16, 2026
jeremyruppel
marked this pull request as ready for review
July 16, 2026 20:46
johnstcn
removed their request for review
July 20, 2026 13:32
Contributor
|
closed my PR in favour of this one this is more performant across the board by reducing the fanout for queries to a single query, while mine only reduces the O(n^2) map building to be linear it may be possible to stack ours for an additional perf. improvement at 50+ orgs for |
johnstcn
approved these changes
Jul 24, 2026
Member
There was a problem hiding this comment.
Noting that the tests also pass against the policy from origin/main 👍 This gives me more signal that this does not change the core logic.
The comment referenced role_member_allowed_orgs, which does not exist. Describe the actual construct: the allowed set is a ground set difference of the member-allow and org-deny vote sets.
Emyrk
approved these changes
Aug 5, 2026
jeremyruppel
added a commit
that referenced
this pull request
Aug 6, 2026
`POST /api/v2/authcheck` evaluated every check with a full policy evaluation in a serial loop. A subject in many organizations (100+) produced hundreds of full evaluations, taking seconds on a cold cache (DEVEX-608). Group the checks by `(action, resource type)` and authorize each group with the existing `rbac.Filter`, which amortizes a single partial evaluation across the group once it is large enough. Each check is wrapped in a small value struct that carries its response key, so `Filter`'s returned subset maps back to keys by reading a field rather than relying on element identity. `Filter` now takes an explicit `prepareThreshold`; existing callers pass the new `rbac.DefaultFilterThreshold` (10), and `checkAuthorization` passes 50, above the ~35-group crossover measured for this workload, so subjects with few objects of a given type keep the per-object path and cannot regress. ## Stacking This is stacked on top of #27244. `Filter` runs `Prepare` (partial evaluation), and those residuals are only compact once #27244's set-membership residuals land. On plain `main` the existing O(N) residual fanout means batching can regress at high org counts, so this change should land with or after #27244. <details> <summary>Decision log</summary> ### Bottleneck - `site/src/modules/permissions/organizations.ts` defines ~14 permission checks per org; `organizationsPermissions()` flattens them across all orgs into one `POST /api/v2/authcheck`. A 100-org request is ~1400 checks. - `checkAuthorization` looped serially, calling `Authorizer.Authorize` (full eval) once per check. - The endpoint's `maxFetch = 10` only caps checks that carry a `resource_id`, not total checks, so it does not bound this workload. ### Approach - Group checks by `(action, resource type)` and run each group through `rbac.Filter`, which does one partial evaluation (`Prepare`) and reuses it across the group. - Carry the response key as data in a small value struct implementing `RBACObject()`, so allowed results map back to keys without pointer identity: ```go type authorizeCheck struct { key string object rbac.Object } func (c authorizeCheck) RBACObject() rbac.Object { return c.object } ``` - `Filter` takes a required `prepareThreshold int` (no functional options). Generic callers pass `rbac.DefaultFilterThreshold = 10`; `/authcheck` passes 50 because the measured crossover for this workload is ~35 groups. ### Alternatives rejected - **Bounded `errgroup` parallelism**: reduced wall time at high org counts but not aggregate work (allocations flat). Discarded in favor of reducing work via partial evaluation. - **Symmetric-deny Rego simplification** (on the #27244 branch): replacing the known-org deny-fold with symmetric `org := -1` / `scope_org := -1` rules failed existing SQL-compile tests. A `-1` known-org vote gated by `not org = -1` produces a negated membership test over the unknown org id, which OPA emits as an unconvertible support rule. #27244's fold (`member_allow - org_deny`, a positive set-difference membership test) is therefore load-bearing, not incidental. </details> --- Authored with Coder Agents. --------- Co-authored-by: Steven Masley <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Authorization for users who belong to many organizations is slow. On the list
endpoints (
/api/v2/organizations,/users,/groups) a user in hundreds oforgs saw multi-second page loads
(DEVEX-608
/ #21890 / Pylon #2758). This is partial-evaluation bound:
rbac.Preparescales with the number of org-scoped roles the subject carries.
Root cause
The known-org path in
check_org_permissionsindexed an N-entry vote map by theobject's org id:
input.object.org_owneris unknown during partial evaluation. Indexing a map byan unknown key cannot reduce to a single expression, so OPA emits one residual
query per org membership, and
newPartialAuthorizerthen callsPrepareForEvalonce per residual, making
PrepareO(N) in org count. The list endpointsintentionally use partial eval; the fan-out is in partial eval itself.
Change
Test the object's org id for membership in a set that is fully known at
partial-evaluation time, so the query collapses to a single
organization_id = ANY(ARRAY[...])residual instead of N residuals:org_owner in org_ids_with_vote(role_org_votes, 1).difference (
member_allow - org_deny), so the unknown org id appears in onlyone positive membership test and the decision never branches on it.
(
role_org_votes,role_member_votes,scope_org_votes,scope_member_votes) instead of through parametrized functions that OPAre-evaluates at every call site.
role_allow/scope_allow, theany_orgpath, and full evaluation areunchanged in behavior.
Semantics are unchanged (see the equivalence argument below). The only
representational change is that a denied known org's intermediate
orgvote isnow
0instead of-1, compensated by the set difference and not observable inthe final
allowdecision.Results
Measured with
BenchmarkRBACManyOrgs(added onmainin #27270). Full tables:B/op and allocs/op.
Prepare/PrepareAndCompilememory changes from < />quadratic growth onmain(176 MiB, 7.08M allocs per op at 100 orgs) to near-linear (6.5 MiB, 258k
allocs), a < />96% reduction at 100 orgs, with similar wins in time.
Preparenow allocates < />7% fewer bytes and < />9% fewer objects thanmain.Authorize(full evaluation) memory is marginally higher (+1-8%, largest at1 org) and time-neutral. This is the inherent cost of the set-membership form
that keeps partial evaluation from fanning out; full evaluation builds an
allow set it would not otherwise need.
go test ./coderd/rbac/...passes, includingTestAuthorizeDomain(full- vspartial-eval equivalence) and the regosql suite.
A second, independent bottleneck remains (out of scope here): the vote map is
still built in O(N^2) in
check_all_org_permissions(
roles[_].by_org_id[org_id]scans all roles per org). Fixing it meanspre-merging roles'
by_org_idinto one org->perms map in the OPA input, and istracked as a follow-up.
Testing
OrgDenyBlocksMember(TestAuthorizeLevels): an org-level deny blocks amember-allowed action on an owned in-org object, while a clean org is allowed,
including an action-scoped deny.
ScopeOrgDenyBlocksMember(TestAuthorizeScope): the same fold at the scopelevel.
result compiles to SQL with zero support rules.
Decision log and equivalence argument
Why not deny-via-set-membership
The first attempt expressed deny as a second set-membership clause (
:= -1 if org_owner in deny_set). That makesorg/scope_orgmulti-valued, and thenot org = -1checks inrole_allow/scope_allowthen cause OPA to emit adata.partial.__not__support rule that regosql cannot compile (TestAuthorizeDomain/UserACLListfailed). It failed even when the deny set was empty, purely because the-1clause exists.Why not deny-via-enumeration
A follow-up enumerated only the (usually empty) deny set. It compiled and passed, but it branches on the unknown org id (one ground residual per denied org), which violates the "do not branch on the unknown" rule in
coderd/rbac/POLICY.md.Final approach: allow-only + ground set difference
The known-org clause votes only to allow, and the org-level deny gate is moved into the org-member level as
member_allow - org_deny, a set difference over fully-known sets. The unknown org id is used only in positiveintests, so there is no enumeration, no negated membership, and no branching on the unknown.Empty-set residual pruning
A naive set-membership left unsatisfiable residuals (
org_owner in set()) for levels with no matching permissions (e.g. the org level for an org-member role, or scope-org forScopeAll), each still costing aPrepareForEval. Guarding each membership with a groundcount(...) > 0lets OPA drop those branches, flattening the residual count across org sizes.Memoized vote maps
Profiling the single-org path showed the cost was repeated function evaluation: the parametrized helpers rebuilt the same vote map for the org, member, and scope paths on every check. Hoisting the maps into memoized zero-arg complete rules (which OPA evaluates once per query) removed that overhead and eliminated the single-org
Prepareregression, while composition keeps the policy readable.Equivalence (known-org path,
site != -1)org == 1<=>org_owner in org_allow(unchanged).org != -1 and member == 1<=>org_owner not in org_deny and org_owner in member_allow<=>org_owner in (member_allow - org_deny)= neworg_member == 1.The critical case (
orgdenies, member allows): old blocks it vianot org = -1; new blocks it becauseorg_owneris removed frommember_allow - org_deny. Same outcome. Deny-wins aggregation is intact becausecheck_all_org_permissionsstill nets an org to-1viato_vote, landing it inorg_deny.This PR was generated by Coder Agents on behalf of @jeremyruppel.