Thanks to visit codestin.com
Credit goes to github.com

Skip to content

test(coderd/rbac): add many-orgs authorization benchmark - #27270

Merged
Emyrk merged 1 commit into
mainfrom
emyrk/rbac-manyorgs-bench
Jul 15, 2026
Merged

test(coderd/rbac): add many-orgs authorization benchmark#27270
Emyrk merged 1 commit into
mainfrom
emyrk/rbac-manyorgs-bench

Conversation

@Emyrk

@Emyrk Emyrk commented Jul 15, 2026

Copy link
Copy Markdown
Member

Adds BenchmarkRBACManyOrgs to measure Authorize, Prepare (partial evaluation), and Prepare+CompileToSQL as a subject's org-membership count grows (1, 5, 10, 50, 100 orgs).

Results comparing main vs #27244 are posted on that PR.

Coder Agents on behalf of @Emyrk.

@Emyrk
test(coderd/rbac): add many-orgs authorization benchmark
3c52bc1 
Benchmarks Authorize, Prepare, and Prepare+CompileToSQL for subjects
with 1/10/50/100 org memberships. Partial evaluation is O(N) in org
count on main (see #21890); this benchmark makes before/after
comparisons of policy changes like #27244 reproducible.
@Emyrk Emyrk mentioned this pull request Jul 15, 2026
Merged
@Emyrk Coder

Emyrk commented Jul 15, 2026 •
edited
Loading

Copy link
Copy Markdown
Member Author

Results from this benchmark comparing main (f958727887) against #27244 (8f4080e0e2 + this benchmark cherry-picked), also posted on that PR: #27244 (comment)

Summary: #27244's partial-eval (Prepare) is dramatically faster at high org counts (-66% at 50 orgs, -74% at 100 orgs, allocs -88%) but regresses at low org counts (+115% at 1 org, +45% at 5 orgs). Crossover is between 5 and 10 orgs. Full-eval Authorize is flat to slightly better across the board.

sec/op, main → #27244:

Benchmark main #27244 vs base
Authorize/orgs=1 284.1µ 332.0µ +16.84% (p=0.004)
Authorize/orgs=5 868.7µ 898.5µ ~ (p=0.589)
Authorize/orgs=10 1.662m 1.622m ~ (p=0.132)
Authorize/orgs=50 7.809m 7.445m -4.67%
Authorize/orgs=100 16.53m 15.89m -3.88%
Prepare/orgs=1 6.760m 14.534m +115.00%
Prepare/orgs=5 13.55m 19.65m +45.02%
Prepare/orgs=10 27.95m 25.82m -7.62%
Prepare/orgs=50 303.6m 103.6m -65.89%
Prepare/orgs=100 1021.2m 261.8m -74.36%
PrepareAndCompile/orgs=1 6.368m 15.116m +137.39%
PrepareAndCompile/orgs=5 14.96m 20.54m +37.31%
PrepareAndCompile/orgs=10 30.70m 27.63m -10.01%
PrepareAndCompile/orgs=50 293.1m 107.5m -63.34%
PrepareAndCompile/orgs=100 1038.3m 265.6m -74.42%

Memory follows the same shape: Prepare/orgs=100 drops 176 MiB → 20.4 MiB per op (7.08M → 803k allocs), while Prepare/orgs=1 grows 1.06 MiB → 2.79 MiB (32.8k → 77.7k allocs).

Full benchstat output (sec/op, B/op, allocs/op) 
goos: linux
goarch: amd64
pkg: github.com/coder/coder/v2/coderd/rbac
cpu: AMD EPYC 9454P 48-Core Processor               
                                           │ /tmp/bench-main-10s.txt │         /tmp/bench-pr-10s.txt         │
                                           │         sec/op          │    sec/op      vs base                │
RBACManyOrgs/Authorize/orgs=1-96                        284.1µ ± 11%    332.0µ ±  7%   +16.84% (p=0.004 n=6)
RBACManyOrgs/Prepare/orgs=1-96                          6.760m ±  2%   14.534m ±  6%  +115.00% (p=0.002 n=6)
RBACManyOrgs/PrepareAndCompile/orgs=1-96                6.368m ±  7%   15.116m ±  3%  +137.39% (p=0.002 n=6)
RBACManyOrgs/Authorize/orgs=5-96                        868.7µ ± 13%    898.5µ ± 15%         ~ (p=0.589 n=6)
RBACManyOrgs/Prepare/orgs=5-96                          13.55m ±  6%    19.65m ±  6%   +45.02% (p=0.002 n=6)
RBACManyOrgs/PrepareAndCompile/orgs=5-96                14.96m ± 10%    20.54m ±  4%   +37.31% (p=0.002 n=6)
RBACManyOrgs/Authorize/orgs=10-96                       1.662m ± 10%    1.622m ±  4%         ~ (p=0.132 n=6)
RBACManyOrgs/Prepare/orgs=10-96                         27.95m ±  7%    25.82m ±  6%    -7.62% (p=0.015 n=6)
RBACManyOrgs/PrepareAndCompile/orgs=10-96               30.70m ±  7%    27.63m ±  3%   -10.01% (p=0.002 n=6)
RBACManyOrgs/Authorize/orgs=50-96                       7.809m ±  2%    7.445m ±  3%    -4.67% (p=0.002 n=6)
RBACManyOrgs/Prepare/orgs=50-96                         303.6m ±  4%    103.6m ±  2%   -65.89% (p=0.002 n=6)
RBACManyOrgs/PrepareAndCompile/orgs=50-96               293.1m ±  3%    107.5m ±  1%   -63.34% (p=0.002 n=6)
RBACManyOrgs/Authorize/orgs=100-96                      16.53m ± 15%    15.89m ±  2%    -3.88% (p=0.026 n=6)
RBACManyOrgs/Prepare/orgs=100-96                       1021.2m ±  3%    261.8m ±  1%   -74.36% (p=0.002 n=6)
RBACManyOrgs/PrepareAndCompile/orgs=100-96             1038.3m ±  7%    265.6m ±  1%   -74.42% (p=0.002 n=6)
geomean                                                 20.18m          16.98m         -15.87%

                                           │ /tmp/bench-main-10s.txt │        /tmp/bench-pr-10s.txt         │
                                           │          B/op           │     B/op      vs base                │
RBACManyOrgs/Authorize/orgs=1-96                        59.05Ki ± 0%   64.17Ki ± 0%    +8.66% (p=0.002 n=6)
RBACManyOrgs/Prepare/orgs=1-96                          1.059Mi ± 0%   2.787Mi ± 0%  +163.16% (p=0.002 n=6)
RBACManyOrgs/PrepareAndCompile/orgs=1-96                1.085Mi ± 0%   2.899Mi ± 0%  +167.14% (p=0.002 n=6)
RBACManyOrgs/Authorize/orgs=5-96                        145.7Ki ± 0%   151.9Ki ± 0%    +4.27% (p=0.002 n=6)
RBACManyOrgs/Prepare/orgs=5-96                          2.440Mi ± 0%   3.484Mi ± 0%   +42.80% (p=0.002 n=6)
RBACManyOrgs/PrepareAndCompile/orgs=5-96                2.497Mi ± 0%   3.644Mi ± 0%   +45.96% (p=0.002 n=6)
RBACManyOrgs/Authorize/orgs=10-96                       254.9Ki ± 0%   262.9Ki ± 0%    +3.15% (p=0.002 n=6)
RBACManyOrgs/Prepare/orgs=10-96                         4.858Mi ± 0%   4.388Mi ± 0%    -9.67% (p=0.002 n=6)
RBACManyOrgs/PrepareAndCompile/orgs=10-96               4.955Mi ± 0%   4.621Mi ± 0%    -6.73% (p=0.002 n=6)
RBACManyOrgs/Authorize/orgs=50-96                       1.099Mi ± 0%   1.119Mi ± 0%    +1.87% (p=0.002 n=6)
RBACManyOrgs/Prepare/orgs=50-96                         51.11Mi ± 0%   11.51Mi ± 0%   -77.48% (p=0.002 n=6)
RBACManyOrgs/PrepareAndCompile/orgs=50-96               51.51Mi ± 0%   12.30Mi ± 0%   -76.12% (p=0.002 n=6)
RBACManyOrgs/Authorize/orgs=100-96                      2.161Mi ± 0%   2.198Mi ± 0%    +1.72% (p=0.002 n=6)
RBACManyOrgs/Prepare/orgs=100-96                       176.39Mi ± 0%   20.44Mi ± 0%   -88.41% (p=0.002 n=6)
RBACManyOrgs/PrepareAndCompile/orgs=100-96             177.22Mi ± 0%   21.82Mi ± 0%   -87.69% (p=0.002 n=6)
geomean                                                 3.325Mi        2.469Mi        -25.76%

                                           │ /tmp/bench-main-10s.txt │        /tmp/bench-pr-10s.txt         │
                                           │        allocs/op        │  allocs/op    vs base                │
RBACManyOrgs/Authorize/orgs=1-96                         1.727k ± 0%    1.845k ± 0%    +6.83% (p=0.002 n=6)
RBACManyOrgs/Prepare/orgs=1-96                           32.77k ± 0%    77.72k ± 0%  +137.14% (p=0.002 n=6)
RBACManyOrgs/PrepareAndCompile/orgs=1-96                 33.29k ± 0%    79.85k ± 0%  +139.84% (p=0.002 n=6)
RBACManyOrgs/Authorize/orgs=5-96                         4.986k ± 0%    5.143k ± 0%    +3.15% (p=0.002 n=6)
RBACManyOrgs/Prepare/orgs=5-96                           83.06k ± 0%   107.19k ± 0%   +29.05% (p=0.002 n=6)
RBACManyOrgs/PrepareAndCompile/orgs=5-96                 84.00k ± 0%   109.59k ± 0%   +30.47% (p=0.002 n=6)
RBACManyOrgs/Authorize/orgs=10-96                        9.058k ± 0%    9.266k ± 0%    +2.30% (p=0.002 n=6)
RBACManyOrgs/Prepare/orgs=10-96                          173.9k ± 0%    144.2k ± 0%   -17.09% (p=0.002 n=6)
RBACManyOrgs/PrepareAndCompile/orgs=10-96                175.4k ± 0%    146.9k ± 0%   -16.23% (p=0.002 n=6)
RBACManyOrgs/Authorize/orgs=50-96                        41.56k ± 0%    42.13k ± 0%    +1.38% (p=0.002 n=6)
RBACManyOrgs/Prepare/orgs=50-96                         2010.5k ± 0%    437.0k ± 0%   -78.27% (p=0.002 n=6)
RBACManyOrgs/PrepareAndCompile/orgs=50-96               2016.1k ± 0%    442.1k ± 0%   -78.07% (p=0.002 n=6)
RBACManyOrgs/Authorize/orgs=100-96                       82.17k ± 0%    83.20k ± 0%    +1.25% (p=0.002 n=6)
RBACManyOrgs/Prepare/orgs=100-96                        7083.7k ± 0%    802.7k ± 0%   -88.67% (p=0.002 n=6)
RBACManyOrgs/PrepareAndCompile/orgs=100-96              7094.1k ± 0%    810.8k ± 0%   -88.57% (p=0.002 n=6)
geomean                                                  118.4k         82.88k        -30.00%

Methodology: go test -run '^$' -bench 'BenchmarkRBACManyOrgs' -benchmem -benchtime 10s -count 6 ./coderd/rbac/ on linux/amd64, AMD EPYC 9454P 48-Core, compared with benchstat.

Coder Agents on behalf of @Emyrk.

@Emyrk
Emyrk marked this pull request as ready for review July 15, 2026 17:57
@Emyrk
Emyrk requested a review from jeremyruppel July 15, 2026 17:57
@Emyrk
Emyrk merged commit 3b72a3e into main Jul 15, 2026
55 of 56 checks passed
@Emyrk
Emyrk deleted the emyrk/rbac-manyorgs-bench branch July 15, 2026 18:04
@github-actions github-actions Bot locked and limited conversation to collaborators Jul 15, 2026
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Reviewers

@jeremyruppel jeremyruppel jeremyruppel approved these changes

Assignees

@Emyrk Emyrk

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@Emyrk @jeremyruppel