-
-
Notifications
You must be signed in to change notification settings - Fork 406
test(bench): add benchmarking #4415
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Conversation
|
|
This pull request is automatically built and testable in CodeSandbox. To see build info of the built libraries, click here or the icon next to each commit SHA. |
|
very interesting... i wonder what the right strategy would be here. there are certain, specific things that are worth creating specific benchmarks for, but it might also be a good idea to benchmark every existing test even if it takes a long time (we can just run it manually after changes to the internals if it becomes a problem in CI). i think using beforeEach is gonna be a requirement here to avoid big noisy codebase changes, and we probably also want to make it optional, disabled by default and enabled through an env variable maybe? ultimately we'll also want tools/scripts to compare bench results between versions, e.g. in PRs but also solid vs. react and such. thanks for working on this! let's keep iterating, cc @diegohaz |
|
Thanks for the feedback @DaniGuardiola! Notes
Enabling vitest hooksI agree, it is a little noisy. Here's what it takes to resolve the issue. Requirements
This feels like a sensible first step. Running tests as benchmarksRunning normal tests in the bench runner isn't on the roadmap yet, but being considered. I think if we're going to use benchmarks as a state of measurement, it seems important to not have assertions we only need for normal (validation) tests. Requirements
|
examples/menu-values-test/test.ts
Outdated
| const spyOnLog = () => vi.spyOn(console, "log").mockImplementation(() => {}); | ||
| // sets up the console.log spy before the component renders, | ||
| // so we can see the spy being called as it renders. | ||
| vi.setConfig({ sequence: { hooks: 'parallel' } }) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We're currently exposed to a race condition; the loader is slower than the setup of this spy. If we ever speed up the loader, this test may fail.
Whilst thinking about this, it's apparent that in our performance test we're not testing how fast it is to render the component.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Whilst thinking about this, it's apparent that in our performance test we're not testing how fast it is to render the component.
Is this still the case?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There's no metric recorded for how long the component takes to render.
The current performance tests test how long it takes to run the test many, many times and finds average latency and throughput values from that.
WalkthroughAdds benchmarking support: direnv/flake files, Nix flake config, Bencher GitHub Actions workflow, vitest benchmark runner and npm bench scripts, .gitignore updates, and numerous test adjustments (beforeEach hooks or conditional test.skipIf gating) for benchmark/React-version runs. Changes
Sequence Diagram(s)sequenceDiagram
autonumber
participant GH as GitHub Actions
participant Repo as Repository
participant Runner as Job Runner
participant Node as Node/npm
participant Bencher as Bencher CLI
note right of GH `#DCEEF9`: Workflow triggers on push to main (matrix)
GH->>Repo: checkout repository
GH->>Runner: provision devShell / environment (flake, node)
Runner->>Node: npm ci / install deps
Runner->>Bencher: ensure bencher available (flake devShell)
Runner->>Node: run `bencher run` (projectId, token, branch, testbed, adapter=json)
Node->>Node: npm run bench variant (sets ARIAKIT_BENCH, LOADER)
Node->>Node: test runner executes tests (beforeEach / test.skipIf gating)
Node->>Bencher: emit JSON benchmark results
Bencher->>GH: upload/submit results
sequenceDiagram
autonumber
participant TestRunner as Test Runner
participant Hook as beforeEach
participant DOM as Document
participant TestCase as Test Case
note right of Hook `#F7F3DE`: per-test lifecycle via beforeEach
TestRunner->>Hook: invoke beforeEach
Hook->>DOM: insert/setup fixtures (returns cleanup)
TestRunner->>TestCase: execute test body (assertions)
TestCase-->>TestRunner: complete
Hook->>DOM: perform teardown (from returned cleanup)
Estimated code review effort🎯 4 (Complex) | ⏱️ ~45 minutes
Pre-merge checks and finishing touches❌ Failed checks (1 warning)
✅ Passed checks (2 passed)
✨ Finishing touches
🧪 Generate unit tests (beta)
Comment |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Actionable comments posted: 1
♻️ Duplicate comments (1)
examples/menu-nested/test.ts (1)
109-123: Clarify benchmark skip rationale and align comments with guidelines.The new comments before both
test.skipIf(process.env.ARIAKIT_BENCH === "1")blocks still mostly state what happens (“Benchmark fails…”) rather than why/how benchmark mode breaks these specific tests. Prior reviews already requested more specific rationale tied to benchmark double rendering, timing/focus instability, and whether this is a temporary workaround.Per the repo’s style rules and retrieved learnings, comments should explain the underlying reason (double render, hover/typeahead timing, focus /
data-active-itemflakiness) so future maintainers understand why these tests are gated byARIAKIT_BENCH. As written, someone debugging benchmark failures won’t know what behavior is incompatible. Based on learnings, ...You can make this more explicit with something like:
-// Benchmark fails when running the cycle (test) twice, -// even with setup/teardown applied. +// Skip in benchmark mode: running this interaction cycle twice +// (benchmark's double render) disrupts typeahead timing and focus, +// causing flaky assertions on the active item. test.skipIf(process.env.ARIAKIT_BENCH === "1")( "typeahead on submenu", async () => { @@ -// Benchmark fails when running the cycle (test) twice, -// even with setup/teardown applied. +// Skip in benchmark mode: hover/blur timing with disabled submenu +// items is not stable under double render, causing flaky assertions +// on focus and data-active-item state. test.skipIf(process.env.ARIAKIT_BENCH === "1")( "blur submenu button on mouse leave after hovering over disabled submenu item", async () => {Feel free to tweak the wording, but capturing the double-render / timing behavior and the kind of assertions that become flaky will satisfy the guidelines and the earlier review feedback.
Also applies to: 125-137
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (2)
contributing.md(2 hunks)examples/menu-nested/test.ts(1 hunks)
🧰 Additional context used
📓 Path-based instructions (2)
**/*.{js,ts,jsx,tsx}
📄 CodeRabbit inference engine (.cursor/rules/style.mdc)
**/*.{js,ts,jsx,tsx}: Always wrapwhileandforloop bodies in braces{}.
Do not use single-line loop bodies without braces.
Default: wrapifbodies in braces{}, with allowed single-line exceptions only for early-exit statements (if (cond) continue;,if (cond) break;,if (cond) return;, orif (cond) return <primitive>;).
Prefer early returns to reduce nesting. For multiple early exits, write oneifper line instead of combining with||for early exits.
Multi-line comments should wrap at 80 characters.
Explain the WHY and HOW in comments, not the WHAT (the WHAT should be clear from good names).
Place comments above the code they describe, not at the end of the line.
Use JSDoc for functions, parameters, and types when documenting top-level functions, APIs, or non-obvious behavior.
If logic feels complex, extract a well-named helper that explains the WHAT; add a short rationale comment if useful.
If the comment is not JSDoc, it should start with//. If it's multi-line comment, each line should start with//.
Use thefunctionkeyword for module-level/top-level functions.
Inside functions, declare nested helpers withconstand arrow syntax (e.g.,const helper = () => { ... };).
Avoid deep nesting; prefer early returns and small helpers.
Avoid abbreviated variable names except for conventional indicesi,j,k. Use descriptive, intention-revealing names.
When indexing arrays, guard againstundefined. Do not use non-null assertions with indexed access. Check withif (!item)orif (item == null)before using the indexed element.
Eliminate repetition by extracting shared logic into small, well-named functions.
Choose function names that convey intent so readers rarely need comments to understand WHAT a function does.
Keep complex conditions readable; prefer early returns or well-named variables to long boolean chains.
Maintain the existing indentation style and width of any file you edit.
Files:
examples/menu-nested/test.ts
**/*.{ts,tsx}
📄 CodeRabbit inference engine (.cursor/rules/style.mdc)
**/*.{ts,tsx}: Type assertions and casts in TypeScript: Use safe, explicit operators that preserve type intent. Allowed:as constfor narrowing literals, DOM-specific casts with runtime checks, andsatisfiesfor validation. Disallowed:as anyandunknown as Tto silence errors. Prefer proper typing with generics, discriminated unions, and type guards.
Preferinterfaceovertypefor object shapes. Usetypeonly when interfaces cannot express the shape (unions, tuples, conditional or mapped types).
Files:
examples/menu-nested/test.ts
🧠 Learnings (3)
📚 Learning: 2025-11-24T22:53:38.597Z
Learnt from: CR
Repo: ariakit/ariakit PR: 0
File: .cursor/rules/style.mdc:0-0
Timestamp: 2025-11-24T22:53:38.597Z
Learning: Applies to **/*.{js,ts,jsx,tsx} : Explain the WHY and HOW in comments, not the WHAT (the WHAT should be clear from good names).
Applied to files:
examples/menu-nested/test.ts
📚 Learning: 2025-11-24T22:53:38.597Z
Learnt from: CR
Repo: ariakit/ariakit PR: 0
File: .cursor/rules/style.mdc:0-0
Timestamp: 2025-11-24T22:53:38.597Z
Learning: Applies to **/*.{js,ts,jsx,tsx} : If logic feels complex, extract a well-named helper that explains the WHAT; add a short rationale comment if useful.
Applied to files:
examples/menu-nested/test.ts
📚 Learning: 2025-11-24T22:53:38.597Z
Learnt from: CR
Repo: ariakit/ariakit PR: 0
File: .cursor/rules/style.mdc:0-0
Timestamp: 2025-11-24T22:53:38.597Z
Learning: Applies to **/*.{js,ts,jsx,tsx} : Place comments above the code they describe, not at the end of the line.
Applied to files:
examples/menu-nested/test.ts
🧬 Code graph analysis (1)
examples/menu-nested/test.ts (3)
packages/ariakit-test/src/click.ts (1)
click(128-169)packages/ariakit-test/src/query.ts (1)
q(165-165)packages/ariakit-test/src/hover.ts (1)
hover(15-73)
🪛 LanguageTool
contributing.md
[grammar] ~273-~273: Ensure spelling is correct
Context: ... between commits and frameworks. We use exising tests to create benchmarks via `vitest-...
(QB_NEW_EN_ORTHOGRAPHY_ERROR_IDS_1)
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Actionable comments posted: 2
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (1)
contributing.md(2 hunks)
🔇 Additional comments (2)
contributing.md (2)
17-24: List item renumbering complete.The basic tutorial steps have been correctly renumbered to accommodate the new "Benchmarking the example" section. The table of contents now reflects the updated structure.
273-273: Typo correction verified.The typo "exising" has been correctly updated to "existing", addressing the prior review comment.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Actionable comments posted: 2
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
⛔ Files ignored due to path filters (1)
package-lock.jsonis excluded by!**/package-lock.json
📒 Files selected for processing (1)
package.json(3 hunks)
🔇 Additional comments (1)
package.json (1)
128-128: Clarify whether benchmarking workflow requires browser testing.
@vitest/browseris an optional, experimental package required only when running benchmarks in real browsers (via Playwright, WebDriver, etc.). Standard Node.js benchmarking does not require it. Verify whether the benchmark workflow uses browser-based testing; if it runs only in Node.js, this dependency should be removed.
424e1c3 to
29115d4
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Actionable comments posted: 1
♻️ Duplicate comments (1)
flake.nix (1)
15-15: Informal license comment previously flagged.This was already discussed in a prior review where you noted these flake files may be removed.
🧹 Nitpick comments (1)
flake.nix (1)
51-53: Remove or implement the postInstall placeholder.This block contains only commented-out code and serves no purpose. Either remove it entirely or implement the intended functionality.
- postInstall = '' - #mv $out/bin/api $out/bin/bencher-api - '';
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
⛔ Files ignored due to path filters (1)
package-lock.jsonis excluded by!**/package-lock.json
📒 Files selected for processing (2)
flake.nix(1 hunks)package.json(3 hunks)
🚧 Files skipped from review as they are similar to previous changes (1)
- package.json
🔇 Additional comments (2)
flake.nix (2)
71-77: Development dependencies look appropriate.The tooling selection (direnv, Node.js 24, bencher, corepack) aligns well with the benchmarking infrastructure goals. The TODO for Playwright support is noted.
78-87: LGTM!Clean output structure exposing the bencher package and dev shell. The telemetry opt-out with the reference comment is good practice.
| on: | ||
| push: | ||
| branches: [main] |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why not on pull_request? If we merge this PR, this workflow will fail, right?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why not on
pull_request?
Because posting benchmarks to bencher on PR's seems redundant since it doesn't apply diffs to PR's.
If we want to ensure the benchmarks run on each PR so they don't fail on main, we can split running benchmarks and posting benchmarks.
if we merge this PR, this workflow will fail, right?
Yes, it will fail because there's no Bencher token.
.github/workflows/benchmark.yaml
Outdated
| - name: Checkout repository | ||
| uses: actions/checkout@v5 | ||
| with: | ||
| token: ${{ secrets.ARIAKIT_BOT_PAT }} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Token not needed here.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'll remove.
.github/workflows/benchmark.yaml
Outdated
| - uses: bencherdev/bencher@main | ||
| - name: Track base branch benchmarks with Bencher | ||
| run: | | ||
| bencher run \ | ||
| --project e63bc897-b5fa-43e4-87d9-f294fcab62b9 \ | ||
| --token '${{ secrets.BENCHER_API_TOKEN }}' \ | ||
| --branch main \ | ||
| --testbed '${{ matrix.machine }}' \ | ||
| --err \ | ||
| --adapter json \ | ||
| --github-actions '${{ secrets.GITHUB_TOKEN }}' \ | ||
| npm run --silent '${{ matrix.command }}' -- --reporter="vitest-runner-benchmark/bmf-reporter" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm wondering if there's another way to run this without requiring secrets, since otherwise it won't work on pull requests from forks.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Interesting, I did not know about this limitation.
I'm not sure there is, that is that way it authorizes.
If we send benchmarks on main only there's no issue.
| inputs = { | ||
| nixpkgs.url = "github:nixos/nixpkgs/nixos-unstable"; | ||
| flake-utils.url = "github:numtide/flake-utils"; | ||
| }; | ||
|
|
||
| outputs = { | ||
| nixpkgs, | ||
| flake-utils, | ||
| ... | ||
| }: | ||
| flake-utils.lib.eachDefaultSystem ( | ||
| system: let | ||
| pkgs = import nixpkgs {inherit system;}; | ||
| # UNFREE check the license fam | ||
| # Derived from https://github.com/NixOS/nixpkgs/pull/456672 | ||
| bencher = pkgs.rustPlatform.buildRustPackage (finalAttrs: { | ||
| pname = "bencher"; | ||
| version = "0.5.8"; | ||
|
|
||
| # When updating, also make sure to update npmDeps.hash in bencher-console! | ||
| src = pkgs.fetchFromGitHub { | ||
| owner = "bencherdev"; | ||
| repo = "bencher"; | ||
| tag = "v${finalAttrs.version}"; | ||
| hash = "sha256-Nz+j8Iwjy4Ziw/L8D7SK3AFRIWP4QQyu63mQnc7dh4o="; | ||
| }; | ||
|
|
||
| cargoHash = "sha256-3jiBz1gWO9klTeXMqVL16qczJptPf9HVksitiGversI="; | ||
|
|
||
| cargoBuildFlags = pkgs. lib.cli.toGNUCommandLine {} { | ||
| package = ["bencher_cli"]; | ||
| }; | ||
|
|
||
| # The default features include `plus` which has a custom license | ||
| buildNoDefaultFeatures = true; | ||
|
|
||
| # does dlopen() libfontconfig during tests and at runtime | ||
| RUSTFLAGS = pkgs.lib.optionalString pkgs. stdenv.targetPlatform.isElf "-C link-arg=-Wl,--add-needed,${pkgs.fontconfig.lib}/lib/libfontconfig.so"; | ||
|
|
||
| nativeBuildInputs = with pkgs; [ | ||
| # .cargo/config.toml selects mold | ||
| mold | ||
| pkg-config | ||
| ]; | ||
|
|
||
| buildInputs = with pkgs; [ | ||
| fontconfig | ||
| ]; | ||
|
|
||
| postInstall = '' | ||
| #mv $out/bin/api $out/bin/bencher-api | ||
| ''; | ||
|
|
||
| meta = { | ||
| description = "Suite of continuous benchmarking tools"; | ||
| homepage = "https://bencher.dev"; | ||
| changelog = "https://github.com/bencherdev/bencher/releases/tag/v${finalAttrs.version}"; | ||
| license = [ | ||
| pkgs. lib.licenses.asl20 | ||
| pkgs.lib.licenses.mit | ||
| ]; | ||
| maintainers = with pkgs. lib.maintainers; [ | ||
| flokli | ||
| ]; | ||
| platforms = pkgs.lib.platforms.unix; | ||
| mainProgram = "bencher"; | ||
| }; | ||
| }); | ||
|
|
||
| # todo: playwright support | ||
| nativeBuildInputs = with pkgs; [ | ||
| direnv | ||
| nodejs_24 | ||
| bencher | ||
| corepack_24 | ||
| ]; | ||
| in { | ||
| packages = {inherit bencher;}; | ||
| devShells.default = pkgs.mkShell { | ||
| inherit system nativeBuildInputs; | ||
| # opt out of nextjs telemetry | ||
| # https://nextjs.org/telemetry | ||
| NEXT_TELEMETRY_DISABLED = 1; | ||
| }; | ||
| } | ||
| ); | ||
| } |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The description says to ignore Nix-related stuff. Will these files be removed?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes.
examples/menu-values-test/test.ts
Outdated
| const spyOnLog = () => vi.spyOn(console, "log").mockImplementation(() => {}); | ||
| // sets up the console.log spy before the component renders, | ||
| // so we can see the spy being called as it renders. | ||
| vi.setConfig({ sequence: { hooks: 'parallel' } }) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Whilst thinking about this, it's apparent that in our performance test we're not testing how fast it is to render the component.
Is this still the case?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Actionable comments posted: 2
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
⛔ Files ignored due to path filters (1)
package-lock.jsonis excluded by!**/package-lock.json
📒 Files selected for processing (2)
.github/workflows/benchmark.yaml(1 hunks)package.json(3 hunks)
🚧 Files skipped from review as they are similar to previous changes (1)
- .github/workflows/benchmark.yaml
🔇 Additional comments (1)
package.json (1)
129-129: The package "vitest-runner-benchmark" does not exist. No npm package or GitHub repository with this name was found.@vitest/runner4.0.14 is compatible with Vitest 4's benchmark system, but the review comment references a non-existent package. Benchmark runners integrate with Vitest through the standard Runner API (extendingBenchmarkNodeRunner) and are configured viavitest.config.ts, not through a separate "vitest-runner-benchmark" package. If this PR is adding benchmark support, verify the actual package/integration being used instead of the stated package name.Likely an incorrect or invalid review comment.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Actionable comments posted: 1
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
⛔ Files ignored due to path filters (1)
package-lock.jsonis excluded by!**/package-lock.json
📒 Files selected for processing (1)
package.json(3 hunks)
🔇 Additional comments (2)
package.json (2)
128-128: Correct infrastructure dependency.
@vitest/runnerv4.0.14 is the official Vitest runner package and aligns with the Vitest upgrade. This change is appropriate.
44-47: I'll verify the benchmark scripts issue by examining the current state of the package.json and understanding how Vitest handles benchmark commands.
<function_calls>
#!/bin/bashRead the package.json lines around 44-47 to see current state
cat -n package.json | sed -n '40,52p'
</function_calls>Now let me search for how ARIAKIT_BENCH is used in the codebase to understand the intended pattern:
<function_calls>
#!/bin/bashSearch for ARIAKIT_BENCH usage to understand the pattern
rg "ARIAKIT_BENCH" -B2 -A2
</function_calls>Let me also check the test configuration to see if there's a mechanism to filter benchmarks:
<function_calls>
#!/bin/bashLook for vitest configuration files
fd -e "vitest.config" -e "vitest.setup" -e "vite.config" 2>/dev/null | head -20
</function_calls><function_calls>
Vitest bench command separate mode from test mutually exclusive
</function_calls>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Actionable comments posted: 1
♻️ Duplicate comments (1)
vitest.config.ts (1)
42-46: Update comment to reflect actual configuration.The comment states "1 cycle of warmup and benchmark respectively" but
minCyclesis set to 2. As discussed in previous reviews, running 2 cycles is intentional for added confidence.Consider updating the comment to clarify the intent:
- // 1 cycle of warmup and benchmark respectively is enough to catch errors. + // 2 cycles of benchmark and 1 of warmup is enough to catch errors locally.
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
⛔ Files ignored due to path filters (1)
package-lock.jsonis excluded by!**/package-lock.json
📒 Files selected for processing (3)
.github/workflows/benchmark.yaml(1 hunks)package.json(2 hunks)vitest.config.ts(2 hunks)
🚧 Files skipped from review as they are similar to previous changes (1)
- package.json
🧰 Additional context used
📓 Path-based instructions (2)
**/*.{js,ts,jsx,tsx}
📄 CodeRabbit inference engine (.cursor/rules/style.mdc)
**/*.{js,ts,jsx,tsx}: Always wrapwhileandforloop bodies in braces{}.
Do not use single-line loop bodies without braces.
Default: wrapifbodies in braces{}, with allowed single-line exceptions only for early-exit statements (if (cond) continue;,if (cond) break;,if (cond) return;, orif (cond) return <primitive>;).
Prefer early returns to reduce nesting. For multiple early exits, write oneifper line instead of combining with||for early exits.
Multi-line comments should wrap at 80 characters.
Explain the WHY and HOW in comments, not the WHAT (the WHAT should be clear from good names).
Place comments above the code they describe, not at the end of the line.
Use JSDoc for functions, parameters, and types when documenting top-level functions, APIs, or non-obvious behavior.
If logic feels complex, extract a well-named helper that explains the WHAT; add a short rationale comment if useful.
If the comment is not JSDoc, it should start with//. If it's multi-line comment, each line should start with//.
Use thefunctionkeyword for module-level/top-level functions.
Inside functions, declare nested helpers withconstand arrow syntax (e.g.,const helper = () => { ... };).
Avoid deep nesting; prefer early returns and small helpers.
Avoid abbreviated variable names except for conventional indicesi,j,k. Use descriptive, intention-revealing names.
When indexing arrays, guard againstundefined. Do not use non-null assertions with indexed access. Check withif (!item)orif (item == null)before using the indexed element.
Eliminate repetition by extracting shared logic into small, well-named functions.
Choose function names that convey intent so readers rarely need comments to understand WHAT a function does.
Keep complex conditions readable; prefer early returns or well-named variables to long boolean chains.
Maintain the existing indentation style and width of any file you edit.
Files:
vitest.config.ts
**/*.{ts,tsx}
📄 CodeRabbit inference engine (.cursor/rules/style.mdc)
**/*.{ts,tsx}: Type assertions and casts in TypeScript: Use safe, explicit operators that preserve type intent. Allowed:as constfor narrowing literals, DOM-specific casts with runtime checks, andsatisfiesfor validation. Disallowed:as anyandunknown as Tto silence errors. Prefer proper typing with generics, discriminated unions, and type guards.
Preferinterfaceovertypefor object shapes. Usetypeonly when interfaces cannot express the shape (unions, tuples, conditional or mapped types).
Files:
vitest.config.ts
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Actionable comments posted: 0
♻️ Duplicate comments (1)
vitest.config.ts (1)
42-46: Update comment to reflect the intentional 2-cycle design.The comment still says "1 cycle" but the previous discussion confirmed 2 cycles is intentional for extra confidence. The comment should be updated as suggested in the earlier review thread.
Apply this diff to update the comment:
- // 1 cycle of warmup and benchmark respectively is enough to catch errors. + // 2 cycles of benchmark (1 would suffice) and 1 of warmup for extra confidence locally. benchmark: { minCycles: 2, minMs: 0,
🧹 Nitpick comments (1)
vitest.config.ts (1)
28-28: Consider explicit boolean conversion for CI flag.The
CIvariable accepts any truthy string value. While this works for typical CI environments (which setCI="true"orCI="1"), explicit conversion improves clarity.Apply this diff for explicit boolean handling:
-const CI = process.env.CI; +const CI = process.env.CI === "true" || process.env.CI === "1";
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
⛔ Files ignored due to path filters (1)
package-lock.jsonis excluded by!**/package-lock.json
📒 Files selected for processing (2)
package.json(2 hunks)vitest.config.ts(3 hunks)
🚧 Files skipped from review as they are similar to previous changes (1)
- package.json
🧰 Additional context used
📓 Path-based instructions (2)
**/*.{js,ts,jsx,tsx}
📄 CodeRabbit inference engine (.cursor/rules/style.mdc)
**/*.{js,ts,jsx,tsx}: Always wrapwhileandforloop bodies in braces{}.
Do not use single-line loop bodies without braces.
Default: wrapifbodies in braces{}, with allowed single-line exceptions only for early-exit statements (if (cond) continue;,if (cond) break;,if (cond) return;, orif (cond) return <primitive>;).
Prefer early returns to reduce nesting. For multiple early exits, write oneifper line instead of combining with||for early exits.
Multi-line comments should wrap at 80 characters.
Explain the WHY and HOW in comments, not the WHAT (the WHAT should be clear from good names).
Place comments above the code they describe, not at the end of the line.
Use JSDoc for functions, parameters, and types when documenting top-level functions, APIs, or non-obvious behavior.
If logic feels complex, extract a well-named helper that explains the WHAT; add a short rationale comment if useful.
If the comment is not JSDoc, it should start with//. If it's multi-line comment, each line should start with//.
Use thefunctionkeyword for module-level/top-level functions.
Inside functions, declare nested helpers withconstand arrow syntax (e.g.,const helper = () => { ... };).
Avoid deep nesting; prefer early returns and small helpers.
Avoid abbreviated variable names except for conventional indicesi,j,k. Use descriptive, intention-revealing names.
When indexing arrays, guard againstundefined. Do not use non-null assertions with indexed access. Check withif (!item)orif (item == null)before using the indexed element.
Eliminate repetition by extracting shared logic into small, well-named functions.
Choose function names that convey intent so readers rarely need comments to understand WHAT a function does.
Keep complex conditions readable; prefer early returns or well-named variables to long boolean chains.
Maintain the existing indentation style and width of any file you edit.
Files:
vitest.config.ts
**/*.{ts,tsx}
📄 CodeRabbit inference engine (.cursor/rules/style.mdc)
**/*.{ts,tsx}: Type assertions and casts in TypeScript: Use safe, explicit operators that preserve type intent. Allowed:as constfor narrowing literals, DOM-specific casts with runtime checks, andsatisfiesfor validation. Disallowed:as anyandunknown as Tto silence errors. Prefer proper typing with generics, discriminated unions, and type guards.
Preferinterfaceovertypefor object shapes. Usetypeonly when interfaces cannot express the shape (unions, tuples, conditional or mapped types).
Files:
vitest.config.ts
🔇 Additional comments (2)
vitest.config.ts (2)
6-6: LGTM: Side-effect import for type augmentation.The side-effect import correctly augments Vitest types with benchmark runner support.
86-91: LGTM: Benchmark runner integration.The conditional
runnerassignment andprovideblock correctly wire up the benchmark runner whenBENCHis enabled, while making the configuration available to all tests.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Actionable comments posted: 2
📜 Review details
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
⛔ Files ignored due to path filters (1)
package-lock.jsonis excluded by!**/package-lock.json
📒 Files selected for processing (1)
package.json(2 hunks)
| "bench": "ARIAKIT_BENCH=1 npm run test --", | ||
| "bench-solid": "ARIAKIT_BENCH=1 npm run test-solid --", | ||
| "bench-react17": "ARIAKIT_BENCH=1 npm run test-react17 --", | ||
| "bench-reactnext": "ARIAKIT_BENCH=1 npm run test-reactnext --", |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
🧩 Analysis chain
🏁 Script executed:
npm view @waynevanson/[email protected] 2>&1Repository: ariakit/ariakit
Length of output: 174
🏁 Script executed:
# Check if vitest and @vitest/runner version mismatch is documented as compatible
npm view [email protected] peerDependencies 2>&1
npm view @vitest/[email protected] description 2>&1Repository: ariakit/ariakit
Length of output: 174
🏁 Script executed:
# Search for how ARIAKIT_BENCH is actually used in the codebase
rg "ARIAKIT_BENCH" --type-add 'config:*.{json,js,ts,mjs,cjs}' -iRepository: ariakit/ariakit
Length of output: 174
🏁 Script executed:
# Check vitest config to see if it handles the ARIAKIT_BENCH flag
fd 'vitest' -t f | head -20Repository: ariakit/ariakit
Length of output: 174
🌐 Web query:
@waynevanson/vitest-benchmark npm package 0.2.6
💡 Result:
I couldn't find @waynevanson/[email protected] on the npm registry. The @waynevanson npm profile exists (lists other packages) but there is no published package named @waynevanson/vitest-benchmark or a 0.2.6 release on npm. [1][2]
Sources:
- @waynevanson npm profile (package list). [1]
- Scoped packages index for @waynevanson (no vitest-benchmark entry). [2]
If you want, I can:
- search GitHub for a vitest-benchmark repo by Wayne Van Son, or
- check an alternative registry (GitHub Packages, GitLab, or a private registry) — tell me which.
🌐 Web query:
vitest @vitest/runner version compatibility 4.0.14 4.0.15
💡 Result:
Short answer: Yes — 4.0.14 → 4.0.15 are patch releases within Vitest v4 and are intended to be compatible. However, Vitest’s packages (vitest core and @vitest/* scoped packages) are meant to be used with aligned v4.x versions, so it’s best to keep all vitest and @vitest/* deps at the same 4.0.x patch to avoid peer/dependency resolution issues. (vitest.dev)
Citations:
🌐 Web query:
ARIAKIT_BENCH vitest environment variable benchmark
💡 Result:
Short answer: there’s no public/documented ARIAKIT_BENCH environment variable in Ariakit or Vitest — I couldn’t find any reference to ARIAKIT_BENCH in the Ariakit repo or docs. To run benchmarks for a project like Ariakit use Vitest’s benchmark mode (vitest bench / vitest --mode benchmark) and the benchmark config options in vitest.config (or the CLI) instead. [Vitest docs cover benchmark mode and config; Ariakit’s repo contains the vitest config but does not expose an ARIAKIT_BENCH var]. [1][2]
How to run benchmarks with Vitest (example)
- Add benchmark files (/*.bench.ts or *.benchmark.ts) using bench(...) in your tests.
- Run: npx vitest bench --run (or configure benchmark.include/benchmark.outputJson in vitest.config.ts).
- Save/compare results: vitest bench --outputJson main.json and vitest bench --compare main.json. [1]
If you want, I can:
- search Ariakit’s repo for any internal CI env var names (deeper search),
- or show exact vitest.config snippets / example bench file and CLI commands for your project. Which would you prefer?
Sources:
- Vitest benchmark & config docs. [Vitest benchmark/config]. (vitest.dev)
- Ariakit repository (contains vitest.config.ts; no ARIAKIT_BENCH reference found). (github.com)
Citations:
@waynevanson/[email protected] is not published on npm and cannot be installed.
The package @waynevanson/vitest-benchmark does not exist on the public npm registry under the @waynevanson profile. Additionally, the ARIAKIT_BENCH=1 environment variable pattern is not a documented or standard approach in Vitest—benchmarks should be run using Vitest's native vitest bench command or --mode benchmark flag instead.
Also update @vitest/runner from 4.0.14 to 4.0.15 to align with the vitest version.
🤖 Prompt for AI Agents
In package.json around lines 44 to 47, the scripts use a non‑existent
@waynevanson/vitest-benchmark package and an undocumented ARIAKIT_BENCH env var;
replace those scripts to use Vitest's built-in benchmark command (e.g., use
"vitest bench" or "vitest --mode benchmark" instead of setting ARIAKIT_BENCH)
and remove the dependency on @waynevanson/vitest-benchmark; additionally update
the @vitest/runner dependency version from 4.0.14 to 4.0.15 to match the vitest
version.
package.json
Outdated
| "@vitest/runner": "4.0.14", | ||
| "@waynevanson/vitest-benchmark": "0.2.6", |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
🧩 Analysis chain
🌐 Web query:
@vitest/runner 4.0.14 vitest 4.0.15 compatibility
💡 Result:
Short answer: Generally compatible — @vitest/runner 4.0.14 (a v4 patch) should work with vitest 4.0.15, but the safest approach is to keep Vitest core and its @vitest/* packages aligned to the same patch version. If you see problems, either upgrade the runner to 4.0.15 or pin vitest to 4.0.14. [1][2]
Sources:
- Vitest latest package info (shows vitest 4.0.15 available). [1]
- Vitest subpackage release listings / release notes indicating v4.x subpackages (examples of @vitest/* 4.0.14). [2]
Align @vitest/runner to match vitest patch version.
@vitest/runner 4.0.14 is compatible with vitest 4.0.15, but best practice is to keep @vitest/* subpackages aligned to the same patch version as vitest core. Update @vitest/runner to 4.0.15 to match vitest.
🤖 Prompt for AI Agents
In package.json around lines 128-129, update the @vitest/runner dependency
version from "4.0.14" to "4.0.15" so it matches the vitest core patch version;
edit the dependency entry to use "4.0.15" and run npm/yarn install (or update
lockfile) to ensure the change is applied.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Actionable comments posted: 0
♻️ Duplicate comments (1)
vitest.config.ts (1)
56-58: Update the comment to match the configured value.The comment states "1 cycle of warmup and benchmark respectively" but
minCyclesis set to 2. Based on past discussion, 2 cycles is intentional for extra confidence. Update the comment to reflect this:- // 1 cycle of warmup and benchmark respectively is enough to catch errors. + // 2 cycles of benchmark and 1 of warmup for extra confidence locally. benchmark: { minCycles: 2,
🧹 Nitpick comments (1)
vitest.config.ts (1)
101-103: Consider conditionally providing benchrunner.The
benchrunnerobject is always provided to tests regardless of whetherBENCHmode is enabled. If tests only needbenchrunnerduring benchmark runs, consider making the provide conditional to reduce overhead:runner: BENCH ? "@waynevanson/vitest-benchmark/runner" : undefined, - provide: { - benchrunner, - }, + provide: BENCH ? { benchrunner } : {},Alternatively, if tests rely on
benchrunnereven in normal test mode, the current approach is fine.
📜 Review details
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
⛔ Files ignored due to path filters (1)
package-lock.jsonis excluded by!**/package-lock.json
📒 Files selected for processing (2)
package.json(2 hunks)vitest.config.ts(3 hunks)
🚧 Files skipped from review as they are similar to previous changes (1)
- package.json
🧰 Additional context used
📓 Path-based instructions (2)
**/*.{js,ts,jsx,tsx}
📄 CodeRabbit inference engine (.cursor/rules/style.mdc)
**/*.{js,ts,jsx,tsx}: Always wrapwhileandforloop bodies in braces{}.
Do not use single-line loop bodies without braces.
Default: wrapifbodies in braces{}, with allowed single-line exceptions only for early-exit statements (if (cond) continue;,if (cond) break;,if (cond) return;, orif (cond) return <primitive>;).
Prefer early returns to reduce nesting. For multiple early exits, write oneifper line instead of combining with||for early exits.
Multi-line comments should wrap at 80 characters.
Explain the WHY and HOW in comments, not the WHAT (the WHAT should be clear from good names).
Place comments above the code they describe, not at the end of the line.
Use JSDoc for functions, parameters, and types when documenting top-level functions, APIs, or non-obvious behavior.
If logic feels complex, extract a well-named helper that explains the WHAT; add a short rationale comment if useful.
If the comment is not JSDoc, it should start with//. If it's multi-line comment, each line should start with//.
Use thefunctionkeyword for module-level/top-level functions.
Inside functions, declare nested helpers withconstand arrow syntax (e.g.,const helper = () => { ... };).
Avoid deep nesting; prefer early returns and small helpers.
Avoid abbreviated variable names except for conventional indicesi,j,k. Use descriptive, intention-revealing names.
When indexing arrays, guard againstundefined. Do not use non-null assertions with indexed access. Check withif (!item)orif (item == null)before using the indexed element.
Eliminate repetition by extracting shared logic into small, well-named functions.
Choose function names that convey intent so readers rarely need comments to understand WHAT a function does.
Keep complex conditions readable; prefer early returns or well-named variables to long boolean chains.
Maintain the existing indentation style and width of any file you edit.
Files:
vitest.config.ts
**/*.{ts,tsx}
📄 CodeRabbit inference engine (.cursor/rules/style.mdc)
**/*.{ts,tsx}: Type assertions and casts in TypeScript: Use safe, explicit operators that preserve type intent. Allowed:as constfor narrowing literals, DOM-specific casts with runtime checks, andsatisfiesfor validation. Disallowed:as anyandunknown as Tto silence errors. Prefer proper typing with generics, discriminated unions, and type guards.
Preferinterfaceovertypefor object shapes. Usetypeonly when interfaces cannot express the shape (unions, tuples, conditional or mapped types).
Files:
vitest.config.ts
🔇 Additional comments (2)
vitest.config.ts (2)
6-6: Confirm the side-effect import is required by the benchmark runner library.The side-effect import of
@waynevanson/vitest-benchmark/runnerfollows the standard Vitest pattern for benchmark runner initialization—the import causes module setup code to execute at load time. Verify that this import is mandated by the@waynevanson/vitest-benchmarklibrary documentation or remove it if the runner works without it.
30-65: Verify benchrunner object shape inconsistency.The
benchrunnerobject has different shapes in CI vs. non-CI mode: the CI branch includes aresultsproperty (lines 40-53) but the non-CI branch omits it. Without access to howbenchrunneris used downstream or the benchmark runner's type expectations, it's unclear whether this will cause type errors or runtime issues. Check if the benchmark library requires theresultsproperty in all configurations or if it gracefully handles its absence.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Actionable comments posted: 0
♻️ Duplicate comments (1)
vitest.config.ts (1)
30-65: Keep benchrunner comment in sync with configuredminCyclesThe non‑CI branch comment still says “1 cycle of warmup and benchmark respectively is enough to catch errors.” but the config deliberately uses 2 benchmark cycles and 1 warmup cycle:
benchmark: { minCycles: 2, minMs: 0, }, warmup: { minCycles: 1, minMs: 0, },Since you intentionally run the benchmark state twice, it’d be good for the comment to reflect that choice. For example:
- // 1 cycle of warmup and benchmark respectively is enough to catch errors. + // 2 benchmark cycles (1 would suffice) and 1 warmup cycle are enough to + // catch local errors while keeping runs fast.This documents the intentional “extra confidence” while keeping the configuration and comment aligned.
🧹 Nitpick comments (2)
vitest.config.ts (2)
25-29: Align loader guard with brace style guidelines and consider stricter CI flag parsingTwo small points here:
- Brace style on the loader guard
Per the repo’s guideline to wrap
ifbodies in braces (except for early exits), the loader validation should use braces around thethrow:-if (!ALLOWED_TEST_LOADERS.includes(LOADER)) - throw new Error(`Invalid loader: ${LOADER}`); +if (!ALLOWED_TEST_LOADERS.includes(LOADER)) { + throw new Error(`Invalid loader: ${LOADER}`); +}
- Optional: tighten
CIandLOADERparsingRight now
CIis truthy for any defined value, andLOADERuses a type assertion with a runtime guard later. That’s totally workable, but if you ever want stricter behavior:
- You could normalize
CIto a boolean, e.g.const CI = process.env.CI === "1" || process.env.CI === "true";.- You could derive
LOADERas a plain string and then narrow it via a small assertion/helper instead of the upfrontas AllowedTestLoader, so TypeScript reflects that the runtime check is what guarantees the type.These would modestly improve clarity and type soundness but are not strictly required.
As per coding guidelines for
ifbraces.Also applies to: 71-72
114-125: Confirmbmfreporter/provide usage for non‑bench runs; consider gating byBENCHRight now:
runneris gated byBENCH, butprovide: { benchrunner }andreporters: [["bmf", { prefix: createPrefix() }]]are always set, even for normal test runs.If
bmfandbenchrunnerare only meaningful in benchmarking mode, you might want to avoid wiring them in regularvitestruns to keep the default UX (and avoid any surprises if the reporter/runner semantics change). For example:- runner: BENCH ? "@waynevanson/vitest-benchmark/runner" : undefined, - provide: { - benchrunner, - }, - reporters: [ - [ - "bmf", - { - prefix: createPrefix(), - }, - ], - ], + runner: BENCH ? "@waynevanson/vitest-benchmark/runner" : undefined, + provide: BENCH ? { benchrunner } : undefined, + reporters: BENCH + ? [ + [ + "bmf", + { + prefix: createPrefix(), + }, + ], + ] + : undefined,This keeps benchmark‑specific wiring behind
ARIAKIT_BENCHand lets non‑bench test runs behave like a standard Vitest config.Please double‑check against the current Vitest and
@waynevanson/vitest-benchmarkdocs thatprovide/reporterscan safely beundefinedand thatbmfis intended only for bench mode; adjust the exact shape accordingly.
📜 Review details
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
⛔ Files ignored due to path filters (1)
package-lock.jsonis excluded by!**/package-lock.json
📒 Files selected for processing (2)
package.json(2 hunks)vitest.config.ts(4 hunks)
🚧 Files skipped from review as they are similar to previous changes (1)
- package.json
🧰 Additional context used
📓 Path-based instructions (2)
**/*.{js,ts,jsx,tsx}
📄 CodeRabbit inference engine (.cursor/rules/style.mdc)
**/*.{js,ts,jsx,tsx}: Always wrapwhileandforloop bodies in braces{}.
Do not use single-line loop bodies without braces.
Default: wrapifbodies in braces{}, with allowed single-line exceptions only for early-exit statements (if (cond) continue;,if (cond) break;,if (cond) return;, orif (cond) return <primitive>;).
Prefer early returns to reduce nesting. For multiple early exits, write oneifper line instead of combining with||for early exits.
Multi-line comments should wrap at 80 characters.
Explain the WHY and HOW in comments, not the WHAT (the WHAT should be clear from good names).
Place comments above the code they describe, not at the end of the line.
Use JSDoc for functions, parameters, and types when documenting top-level functions, APIs, or non-obvious behavior.
If logic feels complex, extract a well-named helper that explains the WHAT; add a short rationale comment if useful.
If the comment is not JSDoc, it should start with//. If it's multi-line comment, each line should start with//.
Use thefunctionkeyword for module-level/top-level functions.
Inside functions, declare nested helpers withconstand arrow syntax (e.g.,const helper = () => { ... };).
Avoid deep nesting; prefer early returns and small helpers.
Avoid abbreviated variable names except for conventional indicesi,j,k. Use descriptive, intention-revealing names.
When indexing arrays, guard againstundefined. Do not use non-null assertions with indexed access. Check withif (!item)orif (item == null)before using the indexed element.
Eliminate repetition by extracting shared logic into small, well-named functions.
Choose function names that convey intent so readers rarely need comments to understand WHAT a function does.
Keep complex conditions readable; prefer early returns or well-named variables to long boolean chains.
Maintain the existing indentation style and width of any file you edit.
Files:
vitest.config.ts
**/*.{ts,tsx}
📄 CodeRabbit inference engine (.cursor/rules/style.mdc)
**/*.{ts,tsx}: Type assertions and casts in TypeScript: Use safe, explicit operators that preserve type intent. Allowed:as constfor narrowing literals, DOM-specific casts with runtime checks, andsatisfiesfor validation. Disallowed:as anyandunknown as Tto silence errors. Prefer proper typing with generics, discriminated unions, and type guards.
Preferinterfaceovertypefor object shapes. Usetypeonly when interfaces cannot express the shape (unions, tuples, conditional or mapped types).
Files:
vitest.config.ts
🔇 Additional comments (2)
vitest.config.ts (2)
6-6: Bench runner side‑effect import looks goodThe side‑effect import of
@waynevanson/vitest-benchmark/runnerlines up with the conditionalrunnersetting below and ensures the runner hooks are registered when BENCH is enabled. No changes needed here.
81-93: Prefix helper cleanly encodes loader/React version into results
createPrefix()nicely captures the loader and React major version into a short prefix (react,react-17,react-next,solid), which should make cross‑framework and cross‑version benchmark comparisons much easier to read in reports. No issues spotted here.
This PR aims to add benchmarks using existing non-browser tests and comparing benchmarks between versions.
Summary of Changes
vitest-runner-benchmark, a Vitest runner created specifically for this feature.{react,solid}.Requires an API token for bencher. I believe I should be able to add ya'll to the organisation in bencher.
Todo:
BMF from benchmarks
Summary by CodeRabbit
New Features
Tests
Documentation
Chores
✏️ Tip: You can customize this high-level summary in your review settings.