Mine your own repo's git history into a private, re-runnable coding eval — then race local and frontier models on bugs that actually happened in your codebase.
Public leaderboards tell you how a model does on other people's code. ownbench tells you how it does on yours.
- Coding-model rankings shift monthly; re-evaluating "can a local model handle my repo yet?" by vibes is unreliable.
- Public benchmarks are contaminated and out-of-distribution for your stack, your conventions, your bug shapes.
- Your git history already contains hundreds of labelled examples: a fix commit is a (broken file, instruction, fixed file) triple for free.
- mine — walk history for single-file fix commits with small diffs. Parent version = broken input, commit message = instruction, committed version = reference.
- run — send each broken file + instruction to any OpenAI-compatible endpoint (llama.cpp, vLLM, Ollama, LiteLLM, OpenRouter, or a frontier API).
- score — deterministic, no LLM judge: exact match, or edit-F1 between the model's diff and the true fix. Whole-file rewrites are penalised the way a reviewer would penalise them.
- report — per-model table, per-language breakdown, and the tasks where every model still fails.
pip install ownbench # zero runtime dependencies, Python ≥3.10# 1. Build a private eval from your repo (writes <repo>/.ownbench/tasks.jsonl)
ownbench mine ~/code/myrepo --max-tasks 30
# 2. Race two models on it
export OWNBENCH_BASE_URL=http://localhost:8000/v1 # any OpenAI-compatible URL
export OWNBENCH_API_KEY=sk-...
ownbench run ~/code/myrepo/.ownbench/tasks.jsonl --model qwen3-coder-32b
ownbench run ~/code/myrepo/.ownbench/tasks.jsonl --model gpt-5-mini \
--base-url https://api.openai.com/v1 --api-key $OPENAI_API_KEY
# 3. Score both runs and render the report
ownbench score ~/code/myrepo/.ownbench/tasks.jsonl \
~/code/myrepo/.ownbench/results-*.jsonl
ownbench report ~/code/myrepo/.ownbench/scores.jsonlExample output:
| model | score | n | exact | partial | wrong | unchanged | invalid | error | avg latency |
|-----------------|-----------|----|-------|---------|-------|-----------|---------|-------|-------------|
| gpt-5-mini | **0.612** | 30 | 14 | 9 | 5 | 1 | 1 | 0 | 8.4s |
| qwen3-coder-32b | **0.471** | 30 | 9 | 10 | 8 | 2 | 1 | 0 | 21.7s |
| status | meaning | score |
|---|---|---|
exact |
output ≡ the real fix (modulo trailing whitespace) | 1.0 |
partial / wrong |
edit-F1 between model's changed lines and the true fix's changed lines | 0–1 |
unchanged |
returned the broken file untouched | 0 |
invalid |
no code block in the reply | 0 |
error |
endpoint failure after retries | 0 |
Edit-F1 = mean of F1(added lines) and F1(removed lines), whitespace-normalised. Precision punishes drive-by rewrites; recall punishes missing the fix.
- Contamination: if your repo is public and old, frontier models may have trained on the fix commits. Mine with
--sinceto restrict to recent history, or use a private repo. - Commit messages vary in quality;
minedrops subjects under 12 chars and known noise (typo/lint/format/bump), but garbage-in still applies. - Exact-match + edit-F1 is a proxy, not a test suite. A semantically correct alternative fix scores partial, not perfect. Treat scores as relative between models on the same task set.
- Single-file tasks only. Cross-file refactors are out of scope by design — they don't fit a single-completion eval.
ownbench mine <repo> [--max-tasks 30] [--max-commits 2000] [--max-diff-lines 40] [--since DATE] [--out PATH]
ownbench run <tasks.jsonl> --model NAME [--base-url URL] [--api-key KEY] [--limit N] [--timeout 300] [--out PATH]
ownbench score <tasks.jsonl> <results.jsonl...> [--out PATH]
ownbench report <scores.jsonl> [--out report.md]
Env vars: OWNBENCH_BASE_URL, OWNBENCH_API_KEY.
MIT