Summary

A small, recurring set of CodSpeed Simulation microbenchmarks report false-positive regressions in nearly every PR, regardless of what the PR changes. This PR gates the worst, provably-unfixable offenders out of CodSpeed simulation (keeping them runnable via local cargo bench), and documents the rest with a keep/remove decision for each.

How I found them

I read CodSpeed's bot comment on 9 recent PRs spanning unrelated areas (scalar ops, SIMD take, GPU/CUDA, session refactors, stats rules, random-access splitting, etc.):
#8345, #8454, #8470, #8478, #8483, #8489, #8496, #8500, #8505, #8511.

A benchmark that shows a large change in a PR that doesn't touch its code is, by definition, noise. The tell is when the same unchanged code reports wildly different numbers depending only on which commit it was built against. Every one of these comparisons also carried CodSpeed's own ⚠️ "Different runtime environments detected" warning.

The flaky set (each row = identical code, different measured value)

Root cause

CodSpeed Simulation estimates CPU cycles from an instruction trace (Cachegrind-style). That trace is deterministic for a fixed binary in a fixed environment, but it includes instructions executed inside glibc — and memcpy/memmove/memset are resolved at runtime via ifunc to a SIMD variant chosen by the runner's CPU/glibc. When the develop baseline and the PR run on different runner images (which CodSpeed explicitly flags here as "different runtime environments"), any benchmark whose hot path is heap allocation + byte copying changes instruction count even though the Vortex code is byte-identical. See CodSpeed's own write-up, "Why glibc is faster on some GitHub Actions Runners".

This predicts exactly which benchmarks flake, and the data confirms it:

• decompress_rd flakes; compress_rd never does. Decode materializes a fresh canonical output buffer (alloc + tight SIMD/memcpy); encode is compute-bound (sampling, dictionary build). Same file, same data — only the copy-bound one is noisy.
• chunked_varbinview_* flakes — canonicalizing chunked strings is concatenating variable-length bytes into one buffer (memcpy-dominated).
• chunked_bool_canonical_into flakes hardest in relative terms — it's also just too small (~16–35 µs → a few hundred layout-sensitive instructions dominate).

The data movement is the thing being measured, so these cannot be made stable under Simulation by tweaking inputs. docs/developer-guide/benchmarking.md already prescribes the remedy: "Use #[cfg(not(codspeed))] for benchmarks that are incompatible with CodSpeed."

Changes (every gated benchmark justified)

All gated benches stay fully available via local cargo bench — only CodSpeed CI stops tracking them.

encodings/alp/benches/alp_compress.rs

• decompress_rd → gated. The Fix build after move #1 offender: moved in 7/9 sampled PRs, bidirectionally, 842↔1025 µs for identical code. It decodes to a canonical array, so its instruction count is dominated by output-buffer alloc + memcpy rather than the ALP-RD decode. compress_rd (encode, compute-bound, never flagged) is kept.

vortex-array/benches/chunk_array_builder.rs

• chunked_varbinview_canonical_into / _into_canonical / _opt_canonical_into / _opt_into_canonical → gated. All four flake across 6/9 PRs; all are memcpy-bound string canonicalization.
• chunked_bool_canonical_into → gated. Worst relative swings (~2×, 16↔35 µs) and below the Simulation noise floor.
• Kept: chunked_opt_bool_canonical_into, chunked_opt_bool_into_canonical, chunked_constant_i32_…, chunked_constant_utf8_… — compute-bound, never flagged.

Deliberately kept (flaky ≠ delete)

• bitpacking_take.rs take_10k_* / patched_take_10k_* — bimodal (±23%), but the in-file comment shows the author tuned the sparse/bitpacked-take thresholds using these exact cases. They measure the core random-access path (real Vortex compute, not glibc), so they're load-bearing. Removing them would lose intended coverage; better to live with the noise floor.
• varbinview_large, chunked_dict_*, encode_varbin*, bench_many_codes_few_values — same memcpy-bound root cause, but several were only observed against a single base (679e2c5), so I left them for a focused follow-up rather than over-reaching here.
• bitwise_not_vortex_buffer_mut[128] — sub-µs noise, but its INPUT_SIZE is shared by ~25 valid vortex-vs-arrow comparison benches; not worth restructuring for a ±30 ns wobble.
• CUDA walltime benches (e.g. cuda/bitpacked_u8/unpack/3bw[100M], swings bidirectionally) — these use the WallTime instrument on non-macro GPU runners (you can't simulate a GPU); that's a separate, intentional trade-off and out of scope.

Verification

Run locally with the pinned nightly + cargo-codspeed, valgrind present:

• ✅ cargo codspeed build (Simulation) succeeds for both vortex-alp and vortex-array (--features _test-harness) — no warnings, no orphaned imports/dead code.
• ✅ cargo codspeed run (Measurement mode: Simulation) executes both suites to completion (exit 0). The gated benches are absent at runtime; compress_rd, chunked_opt_bool_*, and chunked_constant_* still measure.
• ✅ Binary symbol check confirms the gated benches are compiled out under --cfg codspeed and present without it.
• ✅ Local cargo bench build path, cargo +nightly fmt --check, and cargo clippy --no-deps on both bench targets are clean. (Full-workspace clippy hits an unrelated pre-existing lint in vortex-buffer under this environment's toolchain; not touched here.)

Follow-ups for maintainers

1. Archive the gated benches on CodSpeed so they drop off the dashboard instead of showing as "skipped".
2. The broader class (canonicalization/builder/decode-to-canonical) shares this root cause. The durable fix is infrastructure-level: ensure the develop baseline and PR runs use the same pinned runner image (so "different runtime environments" stops firing), or move these to CodSpeed walltime/macro runners. If that lands, several "kept" benches above could return to Simulation.

Generated by Claude Code