Add parallel scaling to the profile-store benchmark - #164

Summary

This PR turns the profile-store benchmark into a concurrency-aware benchmark harness capable of measuring ZoneTree, RocksDB, SQLite, and MySQL consistently from p1 through highly parallel workloads.

Parallelism is now a first-class workload dimension rather than an external variation. A single command can run multiple profile counts and parallelism levels, produce isolated reports for each combination, and compare every engine under the same execution model.

Highlights

• Adds --parallelism with support for comma-separated levels such as 1,4,8,16.
• Introduces dedicated engine workers for concurrent operations.
• Preserves the existing sequential phase order while parallelizing work inside each phase.
• Coordinates worker startup through a shared gate for cleaner timing boundaries.
• Keeps worker creation and initial scheduling outside measured intervals.
• Preserves update ordering by routing repeated updates for the same user to one worker.
• Adds parallelism to JSON, Markdown, SVG charts, output paths, and reference paths.
• Enables Server GC for representative concurrent execution.
• Improves SQLite concurrency with per-worker connections and a busy timeout.
• Rewrites the benchmark documentation around reproducibility and interpretation.

Worker Model

The engine lifecycle and operation execution are now separated:

• IProfileStoreEngine owns initialization, stabilization, settlement, verification, and storage lifetime.
• IProfileStoreEngineWorker owns measured read, query, insert, and update operations.

ZoneTree and RocksDB workers share their engine’s opened storage handles. SQLite and MySQL workers use independent connections and prepared-command sets.

This preserves each engine’s natural concurrency model without forcing a lowest-common-denominator implementation.

Measurement Integrity

Parallel phases wait until every worker reaches a common start gate. The stopwatch starts immediately before all workers are released together.

The benchmark also:

• creates workers before measured phases;
• keeps stabilization outside raw write timings;
• reports stabilized write throughput separately;
• assigns duplicate user updates to the same worker;
• validates checksums between engines at the same parallelism;
• settles, reopens, and verifies persisted data after the workload.

Checksums are intentionally scoped to a specific parallelism level because worker partitioning changes their final combination.

Results and Reports

P1 retains the existing directory layout:

results/profiles-1000000/

Higher parallelism levels receive dedicated directories:

results/profiles-1000000-p16/

Parallelism is included in report configuration, titles, chart descriptions, child-run paths, and curated reference paths.

Example:

dotnet run --project src/ProfileStore.Benchmark.csproj -c Release -- \
  --engine zonetree,rocksdb,sqlite \
  --profiles 100K,1M \
  --parallelism 1,16 \
  --clean

Additional Changes

• Reduces the default query limit from 100 to 50 so profile-fetching queries do not dominate total runtime.
• Adds parallelism-aware result and reference directories.
• Retains backward-compatible p1 output paths.
• Improves Docker instructions and persisted data paths.
• Removes machine-specific credentials and commands from the README.