Thanks to visit codestin.com
Credit goes to github.com

Skip to content

Conversation

@waynevanson
Copy link

@waynevanson waynevanson commented Feb 19, 2025

This PR aims to add benchmarks using existing non-browser tests and comparing benchmarks between versions.

Please ignore the nix stuff unless you'd like to have it in AriaKit.

Summary of Changes

  1. Integrate vitest-runner-benchmark, a Vitest runner created specifically for this feature.
  2. Upgrade Vitest to major version 4.0.12, for above.
  3. Move test setup/teardown into Vitest lifecycle hooks.
  4. Add scripts.
  5. Test both component flavors {react,solid}.
  6. Comparing benchmarks between versions.
    1. Use bencher.dev to record benchmarks.
    2. Update GitHub actions and scripts.

Requires an API token for bencher. I believe I should be able to add ya'll to the organisation in bencher.

Todo:

  1. Run benchmarks in matrix, import using artifacts and upload all reports at once to bencher.
BMF from benchmarks
{
  "react # examples/button/test.ts > markup": {
    "Latency Average": { "value": 8.407643500000063 },
    "Throughput Average": { "value": 0.11893939128127787 },
    "Latency Minimum": { "value": 6.978161 },
    "Latency Maximum": { "value": 9.837126000000126 },
    "Throughput Minimum": { "value": 0.10165570716487592 },
    "Throughput Maximum": { "value": 0.14330423158766328 },
    "Latency P10": { "value": 7.264057500000013 },
    "Latency P50": { "value": 8.407643500000063 },
    "Latency P90": { "value": 9.551229500000112 },
    "Throughput P10": { "value": 0.13913937914538457 },
    "Throughput P50": { "value": 0.12247996937626959 },
    "Throughput P90": { "value": 0.10582055960715465 }
  },
  "react # examples/button-as-div/test.ts > markup": {
    "Latency Average": { "value": 2.400882000000024 },
    "Throughput Average": { "value": 0.4165135979194271 },
    "Latency Minimum": { "value": 2.358608000000004 },
    "Latency Maximum": { "value": 2.4431560000000445 },
    "Throughput Minimum": { "value": 0.40930665090562446 },
    "Throughput Maximum": { "value": 0.42397888924314614 },
    "Latency P10": { "value": 2.3670628000000082 },
    "Latency P50": { "value": 2.400882000000024 },
    "Latency P90": { "value": 2.4347012000000405 },
    "Throughput P10": { "value": 0.422511665409394 },
    "Throughput P50": { "value": 0.4166427700743853 },
    "Throughput P90": { "value": 0.41077387473937665 }
  }
}

Summary by CodeRabbit

  • New Features

    • Added end-to-end benchmarking: local dev shell/direnv support, Nix flake, CLI tooling exposure, and a GitHub Actions benchmark workflow.
  • Tests

    • Tests updated to use framework lifecycle hooks and environment/version-based conditional skips for more stable benchmark runs.
  • Documentation

    • Added a benchmarking guide and integrated it into the contribution tutorial.
  • Chores

    • Added bench scripts, benchmark runner/configuration, dependency updates, and new ignore patterns.

✏️ Tip: You can customize this high-level summary in your review settings.

@changeset-bot
Copy link

changeset-bot bot commented Feb 19, 2025

⚠️ No Changeset found

Latest commit: 56bf2af

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

@codesandbox-ci
Copy link

codesandbox-ci bot commented Feb 19, 2025

This pull request is automatically built and testable in CodeSandbox.

To see build info of the built libraries, click here or the icon next to each commit SHA.

@DaniGuardiola
Copy link
Member

DaniGuardiola commented Feb 19, 2025

very interesting... i wonder what the right strategy would be here. there are certain, specific things that are worth creating specific benchmarks for, but it might also be a good idea to benchmark every existing test even if it takes a long time (we can just run it manually after changes to the internals if it becomes a problem in CI).

i think using beforeEach is gonna be a requirement here to avoid big noisy codebase changes, and we probably also want to make it optional, disabled by default and enabled through an env variable maybe?

ultimately we'll also want tools/scripts to compare bench results between versions, e.g. in PRs but also solid vs. react and such.

thanks for working on this! let's keep iterating, cc @diegohaz

@waynevanson
Copy link
Author

Thanks for the feedback @DaniGuardiola!

Notes

  1. To run benchmarks, the command is vitest bench, which sets env variable MODE="bench" and only runs bench callbacks, throwing when test is called.
  2. Here's a list of related issues on the experimental bench API.
    https://github.com/vitest-dev/vitest/issues?q=is%3Aissue%20state%3Aopen%20benchmark
  3. Noted your points about versioned benchmarks, I'll have a think about it.

Enabling vitest hooks

I agree, it is a little noisy. Here's what it takes to resolve the issue.

Requirements

  1. We could patch/fork this until it is merged upstream?
    feat(vitest): support hooks in bench runner vitest-dev/vitest#5076

This feels like a sensible first step.

Running tests as benchmarks

Running normal tests in the bench runner isn't on the roadmap yet, but being considered.
vitest-dev/vitest#5766

I think if we're going to use benchmarks as a state of measurement, it seems important to not have assertions we only need for normal (validation) tests.

Requirements

  1. Vitest hooks.
  2. Refactor some existing tests to setup/teardown in hooks rather than test execution.
  3. Rollup plugin to remove anything used only in expect() assertions.

@waynevanson waynevanson changed the title test(bench): add benchmarking test(bench): add benchmarking [skip ci] Feb 22, 2025
const spyOnLog = () => vi.spyOn(console, "log").mockImplementation(() => {});
// sets up the console.log spy before the component renders,
// so we can see the spy being called as it renders.
vi.setConfig({ sequence: { hooks: 'parallel' } })
Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We're currently exposed to a race condition; the loader is slower than the setup of this spy. If we ever speed up the loader, this test may fail.

Whilst thinking about this, it's apparent that in our performance test we're not testing how fast it is to render the component.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Whilst thinking about this, it's apparent that in our performance test we're not testing how fast it is to render the component.

Is this still the case?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There's no metric recorded for how long the component takes to render.

The current performance tests test how long it takes to run the test many, many times and finds average latency and throughput values from that.

@coderabbitai
Copy link
Contributor

coderabbitai bot commented Nov 15, 2025

Walkthrough

Adds benchmarking support: direnv/flake files, Nix flake config, Bencher GitHub Actions workflow, vitest benchmark runner and npm bench scripts, .gitignore updates, and numerous test adjustments (beforeEach hooks or conditional test.skipIf gating) for benchmark/React-version runs.

Changes

Cohort / File(s) Summary
Direnv / Nix flake / ignore
\.envrc, flake.nix, \.gitignore
Adds .envrc with use flake; introduces flake.nix (nixpkgs & flake-utils inputs, exports bencher package and devShells.default, nativeBuildInputs incl. direnv/nodejs/bencher/corepack, NEXT_TELEMETRY_DISABLED=1); adds .direnv and .benchmark to .gitignore.
CI: GitHub Actions (Bencher)
\.github/workflows/benchmark.yaml
Adds "Benchmark via Bencher" workflow (pushes to main) with matrix job running multiple bench commands, checks out repo, installs deps, uses bencher action and runs bencher run with project ID, secret token, branch/testbed, JSON adapter and npm bench variants.
Package manifest & scripts
package.json
Adds bench scripts (bench, bench-solid, bench-react17, bench-reactnext) that set ARIAKIT_BENCH=1; updates dependencies (@vitest/runner -> 4.0.14, @waynevanson/vitest-benchmark -> 0.8.2).
Vitest config
vitest.config.ts
Integrates @waynevanson/vitest-benchmark/runner; introduces BENCH (ARIAKIT_BENCH), LOADER (ARIAKIT_TEST_LOADER), CI detection, benchrunner config (CI-aware cycles/warmup), loader validation, sets runner when BENCH enabled, provides benchrunner and adds reporters with dynamic prefix; removes prior test.sequence parallel setting.
Docs: contributing
contributing.md
Adds a "Benchmarking the example" section and renumbers/reorders tutorial steps to include benchmarking guidance.
Tests — conditional skips for benchmarks
examples/composite-hover-virtual-focus-events/test.ts, examples/menu-item-checkbox/test.ts, examples/menu-nested/test.ts, examples/separator/test.ts, site/src/examples/combobox-group/test.ts, site/src/examples/separator/_component/test.ts
Wraps selected tests with test.skipIf(...) gating (commonly process.env.ARIAKIT_BENCH === "1" and/or React version checks) so they are skipped during benchmark runs; test bodies unchanged.
Tests — migrate to beforeEach hooks
examples/focus-trap-region/test.ts, examples/focus-trap/test.ts, examples/select-combobox-focus-within/test.ts, examples/toolbar-active-id-disabled/test.ts, examples/toolbar-active-id-stale/test.ts
Replaces manual setup()/explicit teardown with beforeEach() hooks that prepare DOM fixtures (returning cleanup); tests no longer call manual teardown.
Tests — config / spies
examples/menu-values-test/test.ts
Adds vi.setConfig({ sequence: { hooks: "parallel" } }) before spy setup to allow parallel hook execution for the test.
Examples / minor adjustments
examples/*, site/src/examples/*
Minor formatting and structural edits to accommodate hooks and conditional skips; no public API changes beyond flake outputs.

Sequence Diagram(s)

sequenceDiagram
    autonumber
    participant GH as GitHub Actions
    participant Repo as Repository
    participant Runner as Job Runner
    participant Node as Node/npm
    participant Bencher as Bencher CLI

    note right of GH `#DCEEF9`: Workflow triggers on push to main (matrix)
    GH->>Repo: checkout repository
    GH->>Runner: provision devShell / environment (flake, node)
    Runner->>Node: npm ci / install deps
    Runner->>Bencher: ensure bencher available (flake devShell)
    Runner->>Node: run `bencher run` (projectId, token, branch, testbed, adapter=json)
    Node->>Node: npm run bench variant (sets ARIAKIT_BENCH, LOADER)
    Node->>Node: test runner executes tests (beforeEach / test.skipIf gating)
    Node->>Bencher: emit JSON benchmark results
    Bencher->>GH: upload/submit results
Loading
sequenceDiagram
    autonumber
    participant TestRunner as Test Runner
    participant Hook as beforeEach
    participant DOM as Document
    participant TestCase as Test Case

    note right of Hook `#F7F3DE`: per-test lifecycle via beforeEach
    TestRunner->>Hook: invoke beforeEach
    Hook->>DOM: insert/setup fixtures (returns cleanup)
    TestRunner->>TestCase: execute test body (assertions)
    TestCase-->>TestRunner: complete
    Hook->>DOM: perform teardown (from returned cleanup)
Loading

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~45 minutes

  • Areas to inspect closely:
    • flake.nix: inputs, cargoHash, build flags, outputs.packages (bencher) and devShells.default composition.
    • .github/workflows/benchmark.yaml: bencher invocation args, secrets usage, and matrix-to-script mapping.
    • vitest.config.ts: BENCH/LOADER/CI handling, benchrunner config, runner/provide injection, and reporters/prefix logic.
    • Tests updated to beforeEach() and test.skipIf(): verify skip conditions and that teardown still occurs as expected.
    • package.json: bench scripts and dependency version updates.

Pre-merge checks and finishing touches

❌ Failed checks (1 warning)
Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 33.33% which is insufficient. The required threshold is 80.00%. You can run @coderabbitai generate docstrings to improve docstring coverage.
✅ Passed checks (2 passed)
Check name Status Explanation
Title check ✅ Passed The pull request title 'test(bench): add benchmarking' clearly and concisely describes the main change: adding benchmarking infrastructure to the test suite.
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
✨ Finishing touches
  • 📝 Generate docstrings
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Post copyable unit tests in a comment

Comment @coderabbitai help to get the list of available commands and usage tips.

Copy link
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

♻️ Duplicate comments (1)
examples/menu-nested/test.ts (1)

109-123: Clarify benchmark skip rationale and align comments with guidelines.

The new comments before both test.skipIf(process.env.ARIAKIT_BENCH === "1") blocks still mostly state what happens (“Benchmark fails…”) rather than why/how benchmark mode breaks these specific tests. Prior reviews already requested more specific rationale tied to benchmark double rendering, timing/focus instability, and whether this is a temporary workaround.

Per the repo’s style rules and retrieved learnings, comments should explain the underlying reason (double render, hover/typeahead timing, focus / data-active-item flakiness) so future maintainers understand why these tests are gated by ARIAKIT_BENCH. As written, someone debugging benchmark failures won’t know what behavior is incompatible. Based on learnings, ...

You can make this more explicit with something like:

-// Benchmark fails when running the cycle (test) twice,
-// even with setup/teardown applied.
+// Skip in benchmark mode: running this interaction cycle twice
+// (benchmark's double render) disrupts typeahead timing and focus,
+// causing flaky assertions on the active item.
 test.skipIf(process.env.ARIAKIT_BENCH === "1")(
   "typeahead on submenu",
   async () => {
@@
-// Benchmark fails when running the cycle (test) twice,
-// even with setup/teardown applied.
+// Skip in benchmark mode: hover/blur timing with disabled submenu
+// items is not stable under double render, causing flaky assertions
+// on focus and data-active-item state.
 test.skipIf(process.env.ARIAKIT_BENCH === "1")(
   "blur submenu button on mouse leave after hovering over disabled submenu item",
   async () => {

Feel free to tweak the wording, but capturing the double-render / timing behavior and the kind of assertions that become flaky will satisfy the guidelines and the earlier review feedback.

Also applies to: 125-137

📜 Review details

Configuration used: CodeRabbit UI

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 83b4e56 and 75514dd.

📒 Files selected for processing (2)
  • contributing.md (2 hunks)
  • examples/menu-nested/test.ts (1 hunks)
🧰 Additional context used
📓 Path-based instructions (2)
**/*.{js,ts,jsx,tsx}

📄 CodeRabbit inference engine (.cursor/rules/style.mdc)

**/*.{js,ts,jsx,tsx}: Always wrap while and for loop bodies in braces {}.
Do not use single-line loop bodies without braces.
Default: wrap if bodies in braces {}, with allowed single-line exceptions only for early-exit statements (if (cond) continue;, if (cond) break;, if (cond) return;, or if (cond) return <primitive>;).
Prefer early returns to reduce nesting. For multiple early exits, write one if per line instead of combining with || for early exits.
Multi-line comments should wrap at 80 characters.
Explain the WHY and HOW in comments, not the WHAT (the WHAT should be clear from good names).
Place comments above the code they describe, not at the end of the line.
Use JSDoc for functions, parameters, and types when documenting top-level functions, APIs, or non-obvious behavior.
If logic feels complex, extract a well-named helper that explains the WHAT; add a short rationale comment if useful.
If the comment is not JSDoc, it should start with //. If it's multi-line comment, each line should start with //.
Use the function keyword for module-level/top-level functions.
Inside functions, declare nested helpers with const and arrow syntax (e.g., const helper = () => { ... };).
Avoid deep nesting; prefer early returns and small helpers.
Avoid abbreviated variable names except for conventional indices i, j, k. Use descriptive, intention-revealing names.
When indexing arrays, guard against undefined. Do not use non-null assertions with indexed access. Check with if (!item) or if (item == null) before using the indexed element.
Eliminate repetition by extracting shared logic into small, well-named functions.
Choose function names that convey intent so readers rarely need comments to understand WHAT a function does.
Keep complex conditions readable; prefer early returns or well-named variables to long boolean chains.
Maintain the existing indentation style and width of any file you edit.

Files:

  • examples/menu-nested/test.ts
**/*.{ts,tsx}

📄 CodeRabbit inference engine (.cursor/rules/style.mdc)

**/*.{ts,tsx}: Type assertions and casts in TypeScript: Use safe, explicit operators that preserve type intent. Allowed: as const for narrowing literals, DOM-specific casts with runtime checks, and satisfies for validation. Disallowed: as any and unknown as T to silence errors. Prefer proper typing with generics, discriminated unions, and type guards.
Prefer interface over type for object shapes. Use type only when interfaces cannot express the shape (unions, tuples, conditional or mapped types).

Files:

  • examples/menu-nested/test.ts
🧠 Learnings (3)
📚 Learning: 2025-11-24T22:53:38.597Z
Learnt from: CR
Repo: ariakit/ariakit PR: 0
File: .cursor/rules/style.mdc:0-0
Timestamp: 2025-11-24T22:53:38.597Z
Learning: Applies to **/*.{js,ts,jsx,tsx} : Explain the WHY and HOW in comments, not the WHAT (the WHAT should be clear from good names).

Applied to files:

  • examples/menu-nested/test.ts
📚 Learning: 2025-11-24T22:53:38.597Z
Learnt from: CR
Repo: ariakit/ariakit PR: 0
File: .cursor/rules/style.mdc:0-0
Timestamp: 2025-11-24T22:53:38.597Z
Learning: Applies to **/*.{js,ts,jsx,tsx} : If logic feels complex, extract a well-named helper that explains the WHAT; add a short rationale comment if useful.

Applied to files:

  • examples/menu-nested/test.ts
📚 Learning: 2025-11-24T22:53:38.597Z
Learnt from: CR
Repo: ariakit/ariakit PR: 0
File: .cursor/rules/style.mdc:0-0
Timestamp: 2025-11-24T22:53:38.597Z
Learning: Applies to **/*.{js,ts,jsx,tsx} : Place comments above the code they describe, not at the end of the line.

Applied to files:

  • examples/menu-nested/test.ts
🧬 Code graph analysis (1)
examples/menu-nested/test.ts (3)
packages/ariakit-test/src/click.ts (1)
  • click (128-169)
packages/ariakit-test/src/query.ts (1)
  • q (165-165)
packages/ariakit-test/src/hover.ts (1)
  • hover (15-73)
🪛 LanguageTool
contributing.md

[grammar] ~273-~273: Ensure spelling is correct
Context: ... between commits and frameworks. We use exising tests to create benchmarks via `vitest-...

(QB_NEW_EN_ORTHOGRAPHY_ERROR_IDS_1)

Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
Copy link
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

📜 Review details

Configuration used: CodeRabbit UI

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 75514dd and 6efcce4.

📒 Files selected for processing (1)
  • contributing.md (2 hunks)
🔇 Additional comments (2)
contributing.md (2)

17-24: List item renumbering complete.

The basic tutorial steps have been correctly renumbered to accommodate the new "Benchmarking the example" section. The table of contents now reflects the updated structure.


273-273: Typo correction verified.

The typo "exising" has been correctly updated to "existing", addressing the prior review comment.

Copy link
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

📜 Review details

Configuration used: CodeRabbit UI

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 16f0db6 and cdbcd50.

⛔ Files ignored due to path filters (1)
  • package-lock.json is excluded by !**/package-lock.json
📒 Files selected for processing (1)
  • package.json (3 hunks)
🔇 Additional comments (1)
package.json (1)

128-128: Clarify whether benchmarking workflow requires browser testing.

@vitest/browser is an optional, experimental package required only when running benchmarks in real browsers (via Playwright, WebDriver, etc.). Standard Node.js benchmarking does not require it. Verify whether the benchmark workflow uses browser-based testing; if it runs only in Node.js, this dependency should be removed.

Copy link
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

♻️ Duplicate comments (1)
flake.nix (1)

15-15: Informal license comment previously flagged.

This was already discussed in a prior review where you noted these flake files may be removed.

🧹 Nitpick comments (1)
flake.nix (1)

51-53: Remove or implement the postInstall placeholder.

This block contains only commented-out code and serves no purpose. Either remove it entirely or implement the intended functionality.

-          postInstall = ''
-            #mv $out/bin/api $out/bin/bencher-api
-          '';
📜 Review details

Configuration used: CodeRabbit UI

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 424e1c3 and 29115d4.

⛔ Files ignored due to path filters (1)
  • package-lock.json is excluded by !**/package-lock.json
📒 Files selected for processing (2)
  • flake.nix (1 hunks)
  • package.json (3 hunks)
🚧 Files skipped from review as they are similar to previous changes (1)
  • package.json
🔇 Additional comments (2)
flake.nix (2)

71-77: Development dependencies look appropriate.

The tooling selection (direnv, Node.js 24, bencher, corepack) aligns well with the benchmarking infrastructure goals. The TODO for Playwright support is noted.


78-87: LGTM!

Clean output structure exposing the bencher package and dev shell. The telemetry opt-out with the reference comment is good practice.

Comment on lines +3 to +5
on:
push:
branches: [main]
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why not on pull_request? If we merge this PR, this workflow will fail, right?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why not on pull_request?

Because posting benchmarks to bencher on PR's seems redundant since it doesn't apply diffs to PR's.

If we want to ensure the benchmarks run on each PR so they don't fail on main, we can split running benchmarks and posting benchmarks.

if we merge this PR, this workflow will fail, right?

Yes, it will fail because there's no Bencher token.

- name: Checkout repository
uses: actions/checkout@v5
with:
token: ${{ secrets.ARIAKIT_BOT_PAT }}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Token not needed here.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll remove.

Comment on lines 35 to 46
- uses: bencherdev/bencher@main
- name: Track base branch benchmarks with Bencher
run: |
bencher run \
--project e63bc897-b5fa-43e4-87d9-f294fcab62b9 \
--token '${{ secrets.BENCHER_API_TOKEN }}' \
--branch main \
--testbed '${{ matrix.machine }}' \
--err \
--adapter json \
--github-actions '${{ secrets.GITHUB_TOKEN }}' \
npm run --silent '${{ matrix.command }}' -- --reporter="vitest-runner-benchmark/bmf-reporter"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm wondering if there's another way to run this without requiring secrets, since otherwise it won't work on pull requests from forks.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Interesting, I did not know about this limitation.

I'm not sure there is, that is that way it authorizes.

If we send benchmarks on main only there's no issue.

Comment on lines +2 to +88
inputs = {
nixpkgs.url = "github:nixos/nixpkgs/nixos-unstable";
flake-utils.url = "github:numtide/flake-utils";
};

outputs = {
nixpkgs,
flake-utils,
...
}:
flake-utils.lib.eachDefaultSystem (
system: let
pkgs = import nixpkgs {inherit system;};
# UNFREE check the license fam
# Derived from https://github.com/NixOS/nixpkgs/pull/456672
bencher = pkgs.rustPlatform.buildRustPackage (finalAttrs: {
pname = "bencher";
version = "0.5.8";

# When updating, also make sure to update npmDeps.hash in bencher-console!
src = pkgs.fetchFromGitHub {
owner = "bencherdev";
repo = "bencher";
tag = "v${finalAttrs.version}";
hash = "sha256-Nz+j8Iwjy4Ziw/L8D7SK3AFRIWP4QQyu63mQnc7dh4o=";
};

cargoHash = "sha256-3jiBz1gWO9klTeXMqVL16qczJptPf9HVksitiGversI=";

cargoBuildFlags = pkgs. lib.cli.toGNUCommandLine {} {
package = ["bencher_cli"];
};

# The default features include `plus` which has a custom license
buildNoDefaultFeatures = true;

# does dlopen() libfontconfig during tests and at runtime
RUSTFLAGS = pkgs.lib.optionalString pkgs. stdenv.targetPlatform.isElf "-C link-arg=-Wl,--add-needed,${pkgs.fontconfig.lib}/lib/libfontconfig.so";

nativeBuildInputs = with pkgs; [
# .cargo/config.toml selects mold
mold
pkg-config
];

buildInputs = with pkgs; [
fontconfig
];

postInstall = ''
#mv $out/bin/api $out/bin/bencher-api
'';

meta = {
description = "Suite of continuous benchmarking tools";
homepage = "https://bencher.dev";
changelog = "https://github.com/bencherdev/bencher/releases/tag/v${finalAttrs.version}";
license = [
pkgs. lib.licenses.asl20
pkgs.lib.licenses.mit
];
maintainers = with pkgs. lib.maintainers; [
flokli
];
platforms = pkgs.lib.platforms.unix;
mainProgram = "bencher";
};
});

# todo: playwright support
nativeBuildInputs = with pkgs; [
direnv
nodejs_24
bencher
corepack_24
];
in {
packages = {inherit bencher;};
devShells.default = pkgs.mkShell {
inherit system nativeBuildInputs;
# opt out of nextjs telemetry
# https://nextjs.org/telemetry
NEXT_TELEMETRY_DISABLED = 1;
};
}
);
}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The description says to ignore Nix-related stuff. Will these files be removed?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes.

const spyOnLog = () => vi.spyOn(console, "log").mockImplementation(() => {});
// sets up the console.log spy before the component renders,
// so we can see the spy being called as it renders.
vi.setConfig({ sequence: { hooks: 'parallel' } })
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Whilst thinking about this, it's apparent that in our performance test we're not testing how fast it is to render the component.

Is this still the case?

Copy link
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

📜 Review details

Configuration used: CodeRabbit UI

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 29115d4 and 8fd99a9.

⛔ Files ignored due to path filters (1)
  • package-lock.json is excluded by !**/package-lock.json
📒 Files selected for processing (2)
  • .github/workflows/benchmark.yaml (1 hunks)
  • package.json (3 hunks)
🚧 Files skipped from review as they are similar to previous changes (1)
  • .github/workflows/benchmark.yaml
🔇 Additional comments (1)
package.json (1)

129-129: The package "vitest-runner-benchmark" does not exist. No npm package or GitHub repository with this name was found. @vitest/runner 4.0.14 is compatible with Vitest 4's benchmark system, but the review comment references a non-existent package. Benchmark runners integrate with Vitest through the standard Runner API (extending BenchmarkNodeRunner) and are configured via vitest.config.ts, not through a separate "vitest-runner-benchmark" package. If this PR is adding benchmark support, verify the actual package/integration being used instead of the stated package name.

Likely an incorrect or invalid review comment.

Copy link
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

📜 Review details

Configuration used: CodeRabbit UI

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 8fd99a9 and 9b0aa51.

⛔ Files ignored due to path filters (1)
  • package-lock.json is excluded by !**/package-lock.json
📒 Files selected for processing (1)
  • package.json (3 hunks)
🔇 Additional comments (2)
package.json (2)

128-128: Correct infrastructure dependency.

@vitest/runner v4.0.14 is the official Vitest runner package and aligns with the Vitest upgrade. This change is appropriate.


44-47: I'll verify the benchmark scripts issue by examining the current state of the package.json and understanding how Vitest handles benchmark commands.
<function_calls>


#!/bin/bash

Read the package.json lines around 44-47 to see current state

cat -n package.json | sed -n '40,52p'


</function_calls>

Now let me search for how ARIAKIT_BENCH is used in the codebase to understand the intended pattern:
<function_calls>


#!/bin/bash

Search for ARIAKIT_BENCH usage to understand the pattern

rg "ARIAKIT_BENCH" -B2 -A2


</function_calls>

Let me also check the test configuration to see if there's a mechanism to filter benchmarks:
<function_calls>


#!/bin/bash

Look for vitest configuration files

fd -e "vitest.config" -e "vitest.setup" -e "vite.config" 2>/dev/null | head -20


</function_calls>

<function_calls>


Vitest bench command separate mode from test mutually exclusive


</function_calls>

Copy link
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

♻️ Duplicate comments (1)
vitest.config.ts (1)

42-46: Update comment to reflect actual configuration.

The comment states "1 cycle of warmup and benchmark respectively" but minCycles is set to 2. As discussed in previous reviews, running 2 cycles is intentional for added confidence.

Consider updating the comment to clarify the intent:

-        // 1 cycle of warmup and benchmark respectively is enough to catch errors.
+        // 2 cycles of benchmark and 1 of warmup is enough to catch errors locally.
📜 Review details

Configuration used: CodeRabbit UI

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 9b0aa51 and b4e0326.

⛔ Files ignored due to path filters (1)
  • package-lock.json is excluded by !**/package-lock.json
📒 Files selected for processing (3)
  • .github/workflows/benchmark.yaml (1 hunks)
  • package.json (2 hunks)
  • vitest.config.ts (2 hunks)
🚧 Files skipped from review as they are similar to previous changes (1)
  • package.json
🧰 Additional context used
📓 Path-based instructions (2)
**/*.{js,ts,jsx,tsx}

📄 CodeRabbit inference engine (.cursor/rules/style.mdc)

**/*.{js,ts,jsx,tsx}: Always wrap while and for loop bodies in braces {}.
Do not use single-line loop bodies without braces.
Default: wrap if bodies in braces {}, with allowed single-line exceptions only for early-exit statements (if (cond) continue;, if (cond) break;, if (cond) return;, or if (cond) return <primitive>;).
Prefer early returns to reduce nesting. For multiple early exits, write one if per line instead of combining with || for early exits.
Multi-line comments should wrap at 80 characters.
Explain the WHY and HOW in comments, not the WHAT (the WHAT should be clear from good names).
Place comments above the code they describe, not at the end of the line.
Use JSDoc for functions, parameters, and types when documenting top-level functions, APIs, or non-obvious behavior.
If logic feels complex, extract a well-named helper that explains the WHAT; add a short rationale comment if useful.
If the comment is not JSDoc, it should start with //. If it's multi-line comment, each line should start with //.
Use the function keyword for module-level/top-level functions.
Inside functions, declare nested helpers with const and arrow syntax (e.g., const helper = () => { ... };).
Avoid deep nesting; prefer early returns and small helpers.
Avoid abbreviated variable names except for conventional indices i, j, k. Use descriptive, intention-revealing names.
When indexing arrays, guard against undefined. Do not use non-null assertions with indexed access. Check with if (!item) or if (item == null) before using the indexed element.
Eliminate repetition by extracting shared logic into small, well-named functions.
Choose function names that convey intent so readers rarely need comments to understand WHAT a function does.
Keep complex conditions readable; prefer early returns or well-named variables to long boolean chains.
Maintain the existing indentation style and width of any file you edit.

Files:

  • vitest.config.ts
**/*.{ts,tsx}

📄 CodeRabbit inference engine (.cursor/rules/style.mdc)

**/*.{ts,tsx}: Type assertions and casts in TypeScript: Use safe, explicit operators that preserve type intent. Allowed: as const for narrowing literals, DOM-specific casts with runtime checks, and satisfies for validation. Disallowed: as any and unknown as T to silence errors. Prefer proper typing with generics, discriminated unions, and type guards.
Prefer interface over type for object shapes. Use type only when interfaces cannot express the shape (unions, tuples, conditional or mapped types).

Files:

  • vitest.config.ts

Copy link
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 0

♻️ Duplicate comments (1)
vitest.config.ts (1)

42-46: Update comment to reflect the intentional 2-cycle design.

The comment still says "1 cycle" but the previous discussion confirmed 2 cycles is intentional for extra confidence. The comment should be updated as suggested in the earlier review thread.

Apply this diff to update the comment:

-      // 1 cycle of warmup and benchmark respectively is enough to catch errors.
+      // 2 cycles of benchmark (1 would suffice) and 1 of warmup for extra confidence locally.
       benchmark: {
         minCycles: 2,
         minMs: 0,
🧹 Nitpick comments (1)
vitest.config.ts (1)

28-28: Consider explicit boolean conversion for CI flag.

The CI variable accepts any truthy string value. While this works for typical CI environments (which set CI="true" or CI="1"), explicit conversion improves clarity.

Apply this diff for explicit boolean handling:

-const CI = process.env.CI;
+const CI = process.env.CI === "true" || process.env.CI === "1";
📜 Review details

Configuration used: CodeRabbit UI

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between b4e0326 and d115c9b.

⛔ Files ignored due to path filters (1)
  • package-lock.json is excluded by !**/package-lock.json
📒 Files selected for processing (2)
  • package.json (2 hunks)
  • vitest.config.ts (3 hunks)
🚧 Files skipped from review as they are similar to previous changes (1)
  • package.json
🧰 Additional context used
📓 Path-based instructions (2)
**/*.{js,ts,jsx,tsx}

📄 CodeRabbit inference engine (.cursor/rules/style.mdc)

**/*.{js,ts,jsx,tsx}: Always wrap while and for loop bodies in braces {}.
Do not use single-line loop bodies without braces.
Default: wrap if bodies in braces {}, with allowed single-line exceptions only for early-exit statements (if (cond) continue;, if (cond) break;, if (cond) return;, or if (cond) return <primitive>;).
Prefer early returns to reduce nesting. For multiple early exits, write one if per line instead of combining with || for early exits.
Multi-line comments should wrap at 80 characters.
Explain the WHY and HOW in comments, not the WHAT (the WHAT should be clear from good names).
Place comments above the code they describe, not at the end of the line.
Use JSDoc for functions, parameters, and types when documenting top-level functions, APIs, or non-obvious behavior.
If logic feels complex, extract a well-named helper that explains the WHAT; add a short rationale comment if useful.
If the comment is not JSDoc, it should start with //. If it's multi-line comment, each line should start with //.
Use the function keyword for module-level/top-level functions.
Inside functions, declare nested helpers with const and arrow syntax (e.g., const helper = () => { ... };).
Avoid deep nesting; prefer early returns and small helpers.
Avoid abbreviated variable names except for conventional indices i, j, k. Use descriptive, intention-revealing names.
When indexing arrays, guard against undefined. Do not use non-null assertions with indexed access. Check with if (!item) or if (item == null) before using the indexed element.
Eliminate repetition by extracting shared logic into small, well-named functions.
Choose function names that convey intent so readers rarely need comments to understand WHAT a function does.
Keep complex conditions readable; prefer early returns or well-named variables to long boolean chains.
Maintain the existing indentation style and width of any file you edit.

Files:

  • vitest.config.ts
**/*.{ts,tsx}

📄 CodeRabbit inference engine (.cursor/rules/style.mdc)

**/*.{ts,tsx}: Type assertions and casts in TypeScript: Use safe, explicit operators that preserve type intent. Allowed: as const for narrowing literals, DOM-specific casts with runtime checks, and satisfies for validation. Disallowed: as any and unknown as T to silence errors. Prefer proper typing with generics, discriminated unions, and type guards.
Prefer interface over type for object shapes. Use type only when interfaces cannot express the shape (unions, tuples, conditional or mapped types).

Files:

  • vitest.config.ts
🔇 Additional comments (2)
vitest.config.ts (2)

6-6: LGTM: Side-effect import for type augmentation.

The side-effect import correctly augments Vitest types with benchmark runner support.


86-91: LGTM: Benchmark runner integration.

The conditional runner assignment and provide block correctly wire up the benchmark runner when BENCH is enabled, while making the configuration available to all tests.

Copy link
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

📜 Review details

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 2e825e3 and ae37bcd.

⛔ Files ignored due to path filters (1)
  • package-lock.json is excluded by !**/package-lock.json
📒 Files selected for processing (1)
  • package.json (2 hunks)

Comment on lines +44 to +47
"bench": "ARIAKIT_BENCH=1 npm run test --",
"bench-solid": "ARIAKIT_BENCH=1 npm run test-solid --",
"bench-react17": "ARIAKIT_BENCH=1 npm run test-react17 --",
"bench-reactnext": "ARIAKIT_BENCH=1 npm run test-reactnext --",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🔴 Critical

🧩 Analysis chain

🏁 Script executed:

npm view @waynevanson/[email protected] 2>&1

Repository: ariakit/ariakit

Length of output: 174


🏁 Script executed:

# Check if vitest and @vitest/runner version mismatch is documented as compatible
npm view [email protected] peerDependencies 2>&1
npm view @vitest/[email protected] description 2>&1

Repository: ariakit/ariakit

Length of output: 174


🏁 Script executed:

# Search for how ARIAKIT_BENCH is actually used in the codebase
rg "ARIAKIT_BENCH" --type-add 'config:*.{json,js,ts,mjs,cjs}' -i

Repository: ariakit/ariakit

Length of output: 174


🏁 Script executed:

# Check vitest config to see if it handles the ARIAKIT_BENCH flag
fd 'vitest' -t f | head -20

Repository: ariakit/ariakit

Length of output: 174


🌐 Web query:

@waynevanson/vitest-benchmark npm package 0.2.6

💡 Result:

I couldn't find @waynevanson/[email protected] on the npm registry. The @waynevanson npm profile exists (lists other packages) but there is no published package named @waynevanson/vitest-benchmark or a 0.2.6 release on npm. [1][2]

Sources:

If you want, I can:

  • search GitHub for a vitest-benchmark repo by Wayne Van Son, or
  • check an alternative registry (GitHub Packages, GitLab, or a private registry) — tell me which.

🌐 Web query:

vitest @vitest/runner version compatibility 4.0.14 4.0.15

💡 Result:

Short answer: Yes — 4.0.14 → 4.0.15 are patch releases within Vitest v4 and are intended to be compatible. However, Vitest’s packages (vitest core and @vitest/* scoped packages) are meant to be used with aligned v4.x versions, so it’s best to keep all vitest and @vitest/* deps at the same 4.0.x patch to avoid peer/dependency resolution issues. (vitest.dev)

Citations:


🌐 Web query:

ARIAKIT_BENCH vitest environment variable benchmark

💡 Result:

Short answer: there’s no public/documented ARIAKIT_BENCH environment variable in Ariakit or Vitest — I couldn’t find any reference to ARIAKIT_BENCH in the Ariakit repo or docs. To run benchmarks for a project like Ariakit use Vitest’s benchmark mode (vitest bench / vitest --mode benchmark) and the benchmark config options in vitest.config (or the CLI) instead. [Vitest docs cover benchmark mode and config; Ariakit’s repo contains the vitest config but does not expose an ARIAKIT_BENCH var]. [1][2]

How to run benchmarks with Vitest (example)

  • Add benchmark files (/*.bench.ts or *.benchmark.ts) using bench(...) in your tests.
  • Run: npx vitest bench --run (or configure benchmark.include/benchmark.outputJson in vitest.config.ts).
  • Save/compare results: vitest bench --outputJson main.json and vitest bench --compare main.json. [1]

If you want, I can:

  • search Ariakit’s repo for any internal CI env var names (deeper search),
  • or show exact vitest.config snippets / example bench file and CLI commands for your project. Which would you prefer?

Sources:

  1. Vitest benchmark & config docs. [Vitest benchmark/config]. (vitest.dev)
  2. Ariakit repository (contains vitest.config.ts; no ARIAKIT_BENCH reference found). (github.com)

Citations:


@waynevanson/[email protected] is not published on npm and cannot be installed.

The package @waynevanson/vitest-benchmark does not exist on the public npm registry under the @waynevanson profile. Additionally, the ARIAKIT_BENCH=1 environment variable pattern is not a documented or standard approach in Vitest—benchmarks should be run using Vitest's native vitest bench command or --mode benchmark flag instead.

Also update @vitest/runner from 4.0.14 to 4.0.15 to align with the vitest version.

🤖 Prompt for AI Agents
In package.json around lines 44 to 47, the scripts use a non‑existent
@waynevanson/vitest-benchmark package and an undocumented ARIAKIT_BENCH env var;
replace those scripts to use Vitest's built-in benchmark command (e.g., use
"vitest bench" or "vitest --mode benchmark" instead of setting ARIAKIT_BENCH)
and remove the dependency on @waynevanson/vitest-benchmark; additionally update
the @vitest/runner dependency version from 4.0.14 to 4.0.15 to match the vitest
version.

package.json Outdated
Comment on lines 128 to 129
"@vitest/runner": "4.0.14",
"@waynevanson/vitest-benchmark": "0.2.6",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor

🧩 Analysis chain

🌐 Web query:

@vitest/runner 4.0.14 vitest 4.0.15 compatibility

💡 Result:

Short answer: Generally compatible — @vitest/runner 4.0.14 (a v4 patch) should work with vitest 4.0.15, but the safest approach is to keep Vitest core and its @vitest/* packages aligned to the same patch version. If you see problems, either upgrade the runner to 4.0.15 or pin vitest to 4.0.14. [1][2]

Sources:

  • Vitest latest package info (shows vitest 4.0.15 available). [1]
  • Vitest subpackage release listings / release notes indicating v4.x subpackages (examples of @vitest/* 4.0.14). [2]

Align @vitest/runner to match vitest patch version.

@vitest/runner 4.0.14 is compatible with vitest 4.0.15, but best practice is to keep @vitest/* subpackages aligned to the same patch version as vitest core. Update @vitest/runner to 4.0.15 to match vitest.

🤖 Prompt for AI Agents
In package.json around lines 128-129, update the @vitest/runner dependency
version from "4.0.14" to "4.0.15" so it matches the vitest core patch version;
edit the dependency entry to use "4.0.15" and run npm/yarn install (or update
lockfile) to ensure the change is applied.

Copy link
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 0

♻️ Duplicate comments (1)
vitest.config.ts (1)

56-58: Update the comment to match the configured value.

The comment states "1 cycle of warmup and benchmark respectively" but minCycles is set to 2. Based on past discussion, 2 cycles is intentional for extra confidence. Update the comment to reflect this:

-      // 1 cycle of warmup and benchmark respectively is enough to catch errors.
+      // 2 cycles of benchmark and 1 of warmup for extra confidence locally.
       benchmark: {
         minCycles: 2,
🧹 Nitpick comments (1)
vitest.config.ts (1)

101-103: Consider conditionally providing benchrunner.

The benchrunner object is always provided to tests regardless of whether BENCH mode is enabled. If tests only need benchrunner during benchmark runs, consider making the provide conditional to reduce overhead:

     runner: BENCH ? "@waynevanson/vitest-benchmark/runner" : undefined,
-    provide: {
-      benchrunner,
-    },
+    provide: BENCH ? { benchrunner } : {},

Alternatively, if tests rely on benchrunner even in normal test mode, the current approach is fine.

📜 Review details

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between ae37bcd and 394266c.

⛔ Files ignored due to path filters (1)
  • package-lock.json is excluded by !**/package-lock.json
📒 Files selected for processing (2)
  • package.json (2 hunks)
  • vitest.config.ts (3 hunks)
🚧 Files skipped from review as they are similar to previous changes (1)
  • package.json
🧰 Additional context used
📓 Path-based instructions (2)
**/*.{js,ts,jsx,tsx}

📄 CodeRabbit inference engine (.cursor/rules/style.mdc)

**/*.{js,ts,jsx,tsx}: Always wrap while and for loop bodies in braces {}.
Do not use single-line loop bodies without braces.
Default: wrap if bodies in braces {}, with allowed single-line exceptions only for early-exit statements (if (cond) continue;, if (cond) break;, if (cond) return;, or if (cond) return <primitive>;).
Prefer early returns to reduce nesting. For multiple early exits, write one if per line instead of combining with || for early exits.
Multi-line comments should wrap at 80 characters.
Explain the WHY and HOW in comments, not the WHAT (the WHAT should be clear from good names).
Place comments above the code they describe, not at the end of the line.
Use JSDoc for functions, parameters, and types when documenting top-level functions, APIs, or non-obvious behavior.
If logic feels complex, extract a well-named helper that explains the WHAT; add a short rationale comment if useful.
If the comment is not JSDoc, it should start with //. If it's multi-line comment, each line should start with //.
Use the function keyword for module-level/top-level functions.
Inside functions, declare nested helpers with const and arrow syntax (e.g., const helper = () => { ... };).
Avoid deep nesting; prefer early returns and small helpers.
Avoid abbreviated variable names except for conventional indices i, j, k. Use descriptive, intention-revealing names.
When indexing arrays, guard against undefined. Do not use non-null assertions with indexed access. Check with if (!item) or if (item == null) before using the indexed element.
Eliminate repetition by extracting shared logic into small, well-named functions.
Choose function names that convey intent so readers rarely need comments to understand WHAT a function does.
Keep complex conditions readable; prefer early returns or well-named variables to long boolean chains.
Maintain the existing indentation style and width of any file you edit.

Files:

  • vitest.config.ts
**/*.{ts,tsx}

📄 CodeRabbit inference engine (.cursor/rules/style.mdc)

**/*.{ts,tsx}: Type assertions and casts in TypeScript: Use safe, explicit operators that preserve type intent. Allowed: as const for narrowing literals, DOM-specific casts with runtime checks, and satisfies for validation. Disallowed: as any and unknown as T to silence errors. Prefer proper typing with generics, discriminated unions, and type guards.
Prefer interface over type for object shapes. Use type only when interfaces cannot express the shape (unions, tuples, conditional or mapped types).

Files:

  • vitest.config.ts
🔇 Additional comments (2)
vitest.config.ts (2)

6-6: Confirm the side-effect import is required by the benchmark runner library.

The side-effect import of @waynevanson/vitest-benchmark/runner follows the standard Vitest pattern for benchmark runner initialization—the import causes module setup code to execute at load time. Verify that this import is mandated by the @waynevanson/vitest-benchmark library documentation or remove it if the runner works without it.


30-65: Verify benchrunner object shape inconsistency.

The benchrunner object has different shapes in CI vs. non-CI mode: the CI branch includes a results property (lines 40-53) but the non-CI branch omits it. Without access to how benchrunner is used downstream or the benchmark runner's type expectations, it's unclear whether this will cause type errors or runtime issues. Check if the benchmark library requires the results property in all configurations or if it gracefully handles its absence.

Copy link
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 0

♻️ Duplicate comments (1)
vitest.config.ts (1)

30-65: Keep benchrunner comment in sync with configured minCycles

The non‑CI branch comment still says “1 cycle of warmup and benchmark respectively is enough to catch errors.” but the config deliberately uses 2 benchmark cycles and 1 warmup cycle:

benchmark: {
  minCycles: 2,
  minMs: 0,
},
warmup: {
  minCycles: 1,
  minMs: 0,
},

Since you intentionally run the benchmark state twice, it’d be good for the comment to reflect that choice. For example:

-      // 1 cycle of warmup and benchmark respectively is enough to catch errors.
+      // 2 benchmark cycles (1 would suffice) and 1 warmup cycle are enough to
+      // catch local errors while keeping runs fast.

This documents the intentional “extra confidence” while keeping the configuration and comment aligned.

🧹 Nitpick comments (2)
vitest.config.ts (2)

25-29: Align loader guard with brace style guidelines and consider stricter CI flag parsing

Two small points here:

  1. Brace style on the loader guard

Per the repo’s guideline to wrap if bodies in braces (except for early exits), the loader validation should use braces around the throw:

-if (!ALLOWED_TEST_LOADERS.includes(LOADER))
-  throw new Error(`Invalid loader: ${LOADER}`);
+if (!ALLOWED_TEST_LOADERS.includes(LOADER)) {
+  throw new Error(`Invalid loader: ${LOADER}`);
+}
  1. Optional: tighten CI and LOADER parsing

Right now CI is truthy for any defined value, and LOADER uses a type assertion with a runtime guard later. That’s totally workable, but if you ever want stricter behavior:

  • You could normalize CI to a boolean, e.g. const CI = process.env.CI === "1" || process.env.CI === "true";.
  • You could derive LOADER as a plain string and then narrow it via a small assertion/helper instead of the upfront as AllowedTestLoader, so TypeScript reflects that the runtime check is what guarantees the type.

These would modestly improve clarity and type soundness but are not strictly required.

As per coding guidelines for if braces.

Also applies to: 71-72


114-125: Confirm bmf reporter/provide usage for non‑bench runs; consider gating by BENCH

Right now:

  • runner is gated by BENCH, but
  • provide: { benchrunner } and reporters: [["bmf", { prefix: createPrefix() }]] are always set, even for normal test runs.

If bmf and benchrunner are only meaningful in benchmarking mode, you might want to avoid wiring them in regular vitest runs to keep the default UX (and avoid any surprises if the reporter/runner semantics change). For example:

-    runner: BENCH ? "@waynevanson/vitest-benchmark/runner" : undefined,
-    provide: {
-      benchrunner,
-    },
-    reporters: [
-      [
-        "bmf",
-        {
-          prefix: createPrefix(),
-        },
-      ],
-    ],
+    runner: BENCH ? "@waynevanson/vitest-benchmark/runner" : undefined,
+    provide: BENCH ? { benchrunner } : undefined,
+    reporters: BENCH
+      ? [
+          [
+            "bmf",
+            {
+              prefix: createPrefix(),
+            },
+          ],
+        ]
+      : undefined,

This keeps benchmark‑specific wiring behind ARIAKIT_BENCH and lets non‑bench test runs behave like a standard Vitest config.

Please double‑check against the current Vitest and @waynevanson/vitest-benchmark docs that provide/reporters can safely be undefined and that bmf is intended only for bench mode; adjust the exact shape accordingly.

📜 Review details

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 394266c and de9b397.

⛔ Files ignored due to path filters (1)
  • package-lock.json is excluded by !**/package-lock.json
📒 Files selected for processing (2)
  • package.json (2 hunks)
  • vitest.config.ts (4 hunks)
🚧 Files skipped from review as they are similar to previous changes (1)
  • package.json
🧰 Additional context used
📓 Path-based instructions (2)
**/*.{js,ts,jsx,tsx}

📄 CodeRabbit inference engine (.cursor/rules/style.mdc)

**/*.{js,ts,jsx,tsx}: Always wrap while and for loop bodies in braces {}.
Do not use single-line loop bodies without braces.
Default: wrap if bodies in braces {}, with allowed single-line exceptions only for early-exit statements (if (cond) continue;, if (cond) break;, if (cond) return;, or if (cond) return <primitive>;).
Prefer early returns to reduce nesting. For multiple early exits, write one if per line instead of combining with || for early exits.
Multi-line comments should wrap at 80 characters.
Explain the WHY and HOW in comments, not the WHAT (the WHAT should be clear from good names).
Place comments above the code they describe, not at the end of the line.
Use JSDoc for functions, parameters, and types when documenting top-level functions, APIs, or non-obvious behavior.
If logic feels complex, extract a well-named helper that explains the WHAT; add a short rationale comment if useful.
If the comment is not JSDoc, it should start with //. If it's multi-line comment, each line should start with //.
Use the function keyword for module-level/top-level functions.
Inside functions, declare nested helpers with const and arrow syntax (e.g., const helper = () => { ... };).
Avoid deep nesting; prefer early returns and small helpers.
Avoid abbreviated variable names except for conventional indices i, j, k. Use descriptive, intention-revealing names.
When indexing arrays, guard against undefined. Do not use non-null assertions with indexed access. Check with if (!item) or if (item == null) before using the indexed element.
Eliminate repetition by extracting shared logic into small, well-named functions.
Choose function names that convey intent so readers rarely need comments to understand WHAT a function does.
Keep complex conditions readable; prefer early returns or well-named variables to long boolean chains.
Maintain the existing indentation style and width of any file you edit.

Files:

  • vitest.config.ts
**/*.{ts,tsx}

📄 CodeRabbit inference engine (.cursor/rules/style.mdc)

**/*.{ts,tsx}: Type assertions and casts in TypeScript: Use safe, explicit operators that preserve type intent. Allowed: as const for narrowing literals, DOM-specific casts with runtime checks, and satisfies for validation. Disallowed: as any and unknown as T to silence errors. Prefer proper typing with generics, discriminated unions, and type guards.
Prefer interface over type for object shapes. Use type only when interfaces cannot express the shape (unions, tuples, conditional or mapped types).

Files:

  • vitest.config.ts
🔇 Additional comments (2)
vitest.config.ts (2)

6-6: Bench runner side‑effect import looks good

The side‑effect import of @waynevanson/vitest-benchmark/runner lines up with the conditional runner setting below and ensures the runner hooks are registered when BENCH is enabled. No changes needed here.


81-93: Prefix helper cleanly encodes loader/React version into results

createPrefix() nicely captures the loader and React major version into a short prefix (react, react-17, react-next, solid), which should make cross‑framework and cross‑version benchmark comparisons much easier to read in reports. No issues spotted here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants