CodeEraser applies deterministic computation to the non-deterministic output of language models. A model writing into a long-lived repository drifts toward stacking rather than editing: the same function implemented twice, the same fact restated in a third file, an update that arrives as an append. The tempting fix is to audit that drift with a second model. This is the opposite commitment.
Every verdict below is arithmetic over facts extracted from the tree: token
fingerprints, tree edit distances, graph in-degrees, git-window counts, integer and
rational comparisons against thresholds that are written down. There is
no sampling temperature anywhere between the evidence and the verdict, no floating
point in the judgment layer, and no model in the loop. The same tree yields the same
bytes on any machine at any hour, so a disagreement about code health is settled by
re-running the number and reading the file:line it points at.
Each section below is the short form. The full derivation (every constant traced to the source line that implements it) lives in docs/reference/methodology.md, which every heading here links into.
Three counts of "family" ship across this site and they measure different things. Below there are fifteen booklets, one per judgment whose math is worth its own derivation. On the wire there are twelve capabilities (the stack diagram's count; the join and the split advisory ride inside verdict and structure requests, and the FPR tier ladder is a discipline, not a request). And there are sixteen read-only MCP tools, which follow the report surfaces rather than the wire: churn and graph --sites each get one, erase reaches the dry-run plan and nothing else, doctor answers with machine state rather than a judgment, erase_log reads the applied-erase audit trail, update_check compares this build against the latest release, which is a surface no wire family and no booklet has, and similar_units is the same-role advisor (ce similar's document: the core's order and role bit over similar/1, advisory only).
Finds exact and parameterized duplicate token runs: renamed variables and changed constants are clones, changed syntax is not.
t = window + kgram - 1 = 26 + 25 - 1 = 50 tokens
Any common substring of at least t normalized tokens contains
t - k + 1 = w consecutive k-grams (one complete window) and window
selection depends only on that window's contents, so both copies select the same
minimum. Hence at least one shared fingerprint, always: the Schleimer et al. SIGMOD'03
no-miss lower bound as a correctness contract, not an estimate. The rolling hash is
Rabin-Karp, h = (h - t[i-k]*top)*BASE + t[i], over FNV-1a leaf hashes.
Judges two units whose ASTs are structurally almost the same but whose token streams are not: reshaped and rewritten copies.
TSED(a, b) = (max(n1, n2) - ted(a, b)) / max(n1, n2) clone ⇔ TSED >= 0.85
cloneDecidesWith (num, den) t n1 n2 = (mx - t) * den >= num * mx where mx = max n1 n2
ted is Zhang-Shasha with unit costs (delete = insert = 1,
relabel = 0 on matching kind codes, else 1) and is always integral, so the comparison
is an exact cross-multiplication and the boundary is decidable in both directions: at
max = 100, ted 15 is a clone and ted 16 is not.
Two provably admissible O(1) prefilters cut pairs before any TED runs:
q · tsedDen < tsedNum · max for q ∈ {min(n1,n2), I}, where
I = Σ_label min(c1, c2), using the same 85/100 the judgment will,
which is exactly what makes them admissible.
Decides whether two blocks of documentation text (markdown paragraphs, comment blocks, docstrings) are near-duplicates of each other.
dupDecidesWith num den inter union = inter * den >= num * union dupVerdictWith (num, den, vfloor) inter union run = dupDecidesWith num den inter union || run >= vfloor
Bound in production to (80, 100, 50): Jaccard at or above
0.80, decided exactly in integers, or a verbatim run of at least 50 words. The
core computes inter and union itself from the ascending
deduped shingle sets; raw counts cross the wire, never a ratio, or "the re-check lives
in Haskell" would be an empty claim. MinHash/LSH is only a coarse filter and is
RNG-free: the permutation index is the salt. A run of R shingles spans
R + k - 1 words.
Judges the tree rather than the file: directory geometry, naming distributions, reference locality, documentation coverage, doc staleness, redundancy, modularity.
tsallis2 cs = 1 - Σ (c/N)^2 -- and = 0 when N == 0
tsallis2Norm cs = tsallis2 cs / (1 - 1/n) -- n = nonzero bins, n > 1
chi2 pairs = Σ_{r > 0} (p - q)^2 / q -- p = o/Σo, q = r/Σr
perMille r = floor (r * 1000)
charge_i = floor(scale * v_i / (v_i + N)) -- v = flagged dirs, N = dir total raw = Σ_axes (charge_i * violCost) score = max 0 (scale - raw `div` (structViolCostNeutral * judgedAxisCount))
Shannon entropy and KL divergence need logarithms (irrational, therefore
not exactly decidable), so the family publishes the rational-closed members of the same
families instead: Tsallis-2 diversity and the χ² f-divergence, all over
Data.Ratio. A χ² with observed mass on a zero-reference bin returns
Nothing, a refusal rather than a zero: the report names those directories.
Each axis's violation mass v is its flagged-directory count, charged
through the verdict family's density law into ‰ of scale, so the axis rows carry charges,
not counts, folded through structViolCostNeutral = 10;
judgedAxisCount is 5 to 8
depending on which optional fact tables rode the wire.
Folds seven axes into one 0–1000 score and gates it against a banked baseline that is only allowed to tighten.
raw = sum_i (w_i * p_i * violCost) wTotal = sum_i w_i -- derived, never a literal score = max 0 (scoreScale - raw `div` (violCostNeutral * wTotal))
p(x) = 0 if x <= S
= pMax if H <= S -- degenerate fallback
= pMax * ((x - S) / (H - S))^2 if S < x <= H
= pMax * (1 + 2*(x - H)/(H - S)) if x > H -- C¹ linear arm
m = median(x)
r = median( max(x/m, m/x) ) -- >= 1 by construction
S = clamp(floor(m * r^k), [softMin, softMax])
tolerated(c) = max (c * tolNum `div` tolDen) (c + tolAbs) added = current \ baseline -- non-empty => fail removed = baseline \ current -- informational; drives the shrink
Axis 0 is the only axis that is not a count: a convex penalty on file
size, exact Rational, monotone past the hard line. pMax is the
value at H, not a ceiling. The soft line S is a
statistic of the repository's own frozen LOC distribution, the identity
S = clamp(median + k·MAD, …) re-expressed multiplicatively so no logarithm
is ever taken; it is derived only at establish and then frozen into the baseline. The
fail bit is a disjunction of six named conditions:
ratchet_over, discrete_added, floor,
dedup_budget, knobs_digest, rows_dropped. The reply carries the
list of names that held; the console prints them after FAIL in that order.
Since the rulepack (proto 3.1.0) a continuous row may carry a fourth column, the file's path class:
the 1-based index of the first [[rules.class]]
whose globs match, 0 for none. Beside it rides an additive classKnobs table
whose codes are the ceilings' own 0 / 1 / 2 plus 3 (proto 5.1.0), the
class's own ADR-006 ratchet allowance in lines, and 4 (proto 6.4.0), its own
cognitive-complexity allowance. Those two knobs are the ones whose value 0 is meaningful, so the
table's value floor is judged per code. A classed row is charged against its
class's soft line, hard line and CoC ceiling and falls back to the global line wherever the
class declares none; the charge law itself is untouched. The ratchet reads the fourth column
only to pick that allowance, asked per (class, metric), so code 4 answers cognitive
complexity where declared and code 3 answers the rest; declared, it replaces
both global legs, so 0 means any growth is over. It writes a three-column
baseline back, so a class is a charging
parameter for this run and never a baseline fact. Names and globs stay on the client: only the
index and its knobs cross.
Answers one question ("which files does nothing live reach?") over a reference graph whose node identity is the row index, with no text on the wire.
arcs = { (s,d) | [s,d,kind,rung] ∈ edges, rung <= minRung, kind ∉ inert }
reach = ⋃ { reachable(G, s) | s ∈ entries(entryMask, flags) }
public = testBit flags 0 referenced = indeg >= 1 over kept arcs judged = i ∉ reach code = 1 + public + 2*referenced -- the lookup table is the authority
Four codes, structurally separated so an exported-but-unreferenced API
can never collapse into plain dead: 1 unref_private,
2 unref_public, 3 unreach_private, 4 unreach_public.
Resolution never guesses: a site walks its language's rungs in order and the first rung
producing exactly one in-scope candidate wins; more than one is
Unresolved, and External is a correct terminal answer, not a
miss. Cycles are reported, never judged; a cyclic island with no entry seed is dead by
reachability alone. The kind column is the formula's second filter, not baggage:
inert = {3 asset, 5 unused ref-def}, because an image link renders bytes and an unused
reference definition renders nothing, neither counts as a reference, and the exclusion is
executed by the core itself. Since proto 2.32.0 each dead row carries a confidence column judged from
the per-language site ledger the request ships: 0 unvouched (the language still has
unresolved sites), 1 vacuous, 2 vouched. That column is the erase family's trust boundary, executed
by the family that owns the ledger. Import-edge precision measured
38/40 = 0.95 across five pinned corpora against a ≥ 0.90 gate, on a
sample frozen before any resolver existed.
Combines similarity × graph position × churn into one of four candidate codes, and then deliberately declines to act on it.
(1, sev 2, [1,2,3,4], []) -- merge_candidate: sim + graph + both referenced + distinct SCCs (2, sev 3, [1,2,5], [6]) -- delete_candidate: sim + graph + dead flank, RG10 guard clear (3, sev 1, [1,2,7,8], []) -- churn_hotspot: sim + graph + cochange + rewrite
rewriteHot = total > 0 && (rewrote_a + rewrote_b) * rewriteDen >= total * rewriteNum cochangeHot = cochange >= cochangeFloor legsMask = legSim .|. (if graphBoth then legGraph else 0) .|. legChurn
Priority is data, not guard order: the first row whose required
bits all hold and whose forbidden bits all stay clear wins, else code 0. Making the
order data is what lets the property battery falsify it by rotating the table. Every
gating row requires the graph bit, so a mask of 5 (graph leg absent) can only carry code
0; a missing graph leg refuses to gate rather than pretending indegree 0. Since
proto 2.33.0 every candidate row carries a leg-agreement confidence (how many
present legs contributed a held condition), and the table's severity face ships once
as joinSeverity; ce join judges its pairs over the SAME
verdict/1 road the check gate uses. The join still produces candidates; no
verdict code appears in the fail bit, so ce join never gates on
its own verdicts; it exits 0 unless the run itself fails, and a missing or
refusing core is a run failure (exit 2), not a quiet pass.
Answers "is this long file worth splitting, and where?" with a number instead of a slogan, as an advisory that never contributes to the structure score or the fail bit.
benefitMilli(u) = max 0 (floor (1000 * (p(total) - p(end_u) - p(total - end_u))))
costMilli(u) = crossRefs(u) * roiRefMilli
+ cutClones(end_u) * roiCloneMilli
+ crossChurn(u) * roiChurnMilli
+ roiPhiMilli
viable ⇔ b >= c -- ROI >= 1, evaluated without division
Benefit is the graded-zone penalty a split gives back, computed on the
same convex curve the verdict family judges with, imported rather than
re-derived; because p is convex with p(0) = 0 it is
superadditive, so the bracket is non-negative. Best-seam selection is the exact rational
argmax over ROI, compared by cross-multiplied b % c. A file with
n top-level units yields n - 1 seams. Long-and-cohesive
becomes an exemption with numbers attached; long-and-splittable gets a cut
line.
Reduces every supervised edit to four integer counts per file pair: matched, novel, moved, deleted. That makes "this update was stacked, not applied" a measurement.
siteOpens s n = n * movedCost + s < n * plainCost destFloor = least n with siteOpens siteCostCross n = 2
accepted = isStart && distinctEvidence >= destFloor && anchored anchored = any (\(_,_,w) -> w >= anchorFloor) evidence
The cross-file evidence floor is derived, not tuned:
siteCostCross = 2 makes a single cross line a tie
(1*1 + 2 = 3 = 1*3), ties do not open, so destFloor evaluates
to 2, and that tie is the coincidence rejection. Only
anchorFloor = 19 is decided rather than derived: in the dual-corpus shadow
ablation the invented station's widest anchor measured 16 alnum characters and the
thinnest real anchor 19, so 19 is the top of the window that kills every measured
coincidence while keeping every measured real site. The L2 delta is monotone in one
direction only: plain lines may become moved, never the reverse. Since proto 7.1.0 a
second, coarser stage rides beside the line delta: paired declRem /
declAdd tables of declaration keys that vanish on one side and appear on
exactly one other, accepted when the two spans share at least declFloor = 1
leftover hash; the key pays the cross-site cost itself
(declCredit = siteCostCross), so one shared line opens an edge where a line
would need two. It appends relocations with lines = 0; no line
classification, block, suspicion or score moves.
Answers one question about a repository's history: is the check score going up, flat, or down, and by how much per day.
x_i = ts_i % 86400 -- seconds to days, exact ratio
y_i = (score_i * 1000000) % scale_i
slope = median{ (y_j - y_i) / (x_j - x_i) : x_i ≠ x_j } -- Theil-Sen
slope < -band → 2 (degrading) slope > band → 0 (improving) otherwise → 1 (flat) where band = floorMicro
% here is Data.Ratio's exact-ratio
constructor, not modulo; y renormalizes every commit onto a fixed 10⁶
full-scale grid so rows measured under different scoreScale values are
commensurable. The slope is the median of pairwise slopes (trend/2): one wild point,
a broken commit that still measured, drags a least-squares mean anywhere and cannot
move the median past its neighbors. Row order is deliberately unconstrained: the
judged view sorts by timestamp and caps at the tsWindow most recent points, so rebased
commits are legal input. The steepest single-step fall ships as cliff and
the longest strictly-falling run as declineRun, each naming its commit by
request index; hashes never cross. Below minPoints, or with no
timestamp-distinct pair, the slope is Nothing (absence, never a
fabricated flat) and the fail bit stays false. With the default floor 0, degrading
can be reported and cannot fail.
A deterministic gate over a non-deterministic writer: the guard sits on
PreToolUse for Write|Edit and answers each pending write with
exact arithmetic, and a rule class may only enforce once it has paid for the right.
TIERS = ["observe", "warn", "ask", "deny"] PROMOTED_DEFAULT = "deny"
| tier | permissionDecision | effect |
|---|---|---|
| observe | none, returns before printing | feed line only, no injected text |
| warn | allow | edit proceeds, reason surfaces as a visible warning |
| ask | ask | the user is prompted |
| deny | deny | write is refused, reason points at the existing file:line |
Determinism is bought by replaying the write rather than
estimating it: resulting_lines computes the exact post-write line count,
and any case where the tool call would fail on its own (missing file, ambiguous
non-replace_all match) returns None and the rule stays
silent. The gate never judges a write that will not land. An unrecognized
[guard] mode resolves to an observe (ce.toml ERROR: …) string
rather than being passed through, because a pass-through typo once disarmed every
enforcement path while the session banner still printed the mode as armed.
Why a rule class may not simply be set to deny. The
ladder is a route written into the plan, so the default can neither stay at
warn forever nor start at deny. Admission is quantitative:
the M3 acceptance criterion is ≤ 1 mis-block in 500 real normal edits
with N=1 demonstrations explicitly disallowed, and the M4 main gate is
FPR ≤ 1% over 500 real normal edits on an evaluation set pre-registered
before implementation, ≥ 200 edit samples, ≥ 50% drawn from real agent transcripts.
Sample purity is part of the gate: only observe-mode and pre-guard sessions may be
sampled and edits the guard already intervened in are excluded, otherwise FPR is biased
downward by the guard's own shaping and the deny admission becomes self-certifying.
Exactly two classes have paid: T1/T2 exact duplicate write, and hard-budget breach at
file > its hard line, 750 by default, or the line the file's
[[rules.class]] declares, so the hook denies exactly where the CI wall fails.
Every other rule stays at observe for want of its
own record.
Since CodeEraser 1.2.0 the duplicate-write rule charges novel duplication
only. The K-round replay resurrected the retired instrument over 2,761 real
edit events (this repository's full history plus the pinned requests
tail), and arbitration split the raw intercepts into genuine duplication landings,
re-fires on files already carrying budgeted blocks, and split/fold mid-states: the
write-first ordering of exactly the extract-to-a-leaf refactors a 300-line discipline
produces. The fix subtracts the matches the replaced content already carried
(Write replaces the on-disk file, Edit replaces old_string): a rewrite
that carries its debt is silent, an introduction still denies, and a split to a new
file, indistinguishable from a copy at the instant of the write, still denies while
the denial teaches the ordering that passes (trim the source first). Both framings of
the resulting rate are ledgered in
FPR-REPLAY.md.
The recorded replay treats git linear history as a real edit stream:
probe first, then apply, at the shipped default knobs t = 50 and
min_distinct = 7: 630 events, 35 blocks, 0 false after
arbitration → 0.00 per 500. That was the M3 round; the standing instrument
(cli/tests/it/fpr_replay.rs) re-measured on 2026-09-05: requests 365 events
/ 0 intercepts, and 3164 self-repo events raising 178 landed introductions plus 79
write-first mid-states. Both readings live in FPR-REPLAY.md. The 35 self-repo blocks
arbitrated as true positives and were remediated, stepping the clone-block budget
251 → 211 → 209 → 205 → 202 in lockstep.
cap = lines_for(file).file_lines_fail // the file's class table; 750 for class 0 breach ⇔ cap != 0 && lines > cap permille = (lines - S) * 1000 / (H - S) 0 ..= 249 → observe 250 ..= 750 → warn 751 .. → ask
The graded zone reads its S fallback and its H off the
same per-file table as the hard budget. The map is the same discipline applied to itself:
[guard] zone_tiers is off by default, so the zone rule is
feed-only and injects nothing. Its ledger now exists and it still does not enforce: the
pre-registered rule armed the default only if every corpus measured at or under
1 %, and the first run read self 0.54 % but requests 2.46 %; variant B,
zone_tiers stays false, with
cli/tests/it/fpr_zone_gate.rs holding the shipped default equal to what
that arithmetic licenses. Warns are rate-limited to once per
(rule, file, session) and clipped at a token budget; enforcement is not rate-limited,
because a deny is not context bloat.
Judges erase-plan rows for three provable classes (dead_file,
verbatim_doc, and t1_twin) and refuses every unsafe
row with a named reason code.
class = 0 (retired 4.0.0, refused by name) | 1 verbatim_doc | 2 t1_twin
3 dead_file (confidence road, 2.32.0)
reason = 0 eraseable | 1 language_unresolved | 2 not_full_segment
3 bytes_differ | 4 copy_not_dead | 5 unit_not_covered
6 public_surface (6.1.0)
Rust assembles integer facts from the three source families; Haskell applies the fixed first-failure predicate. Since proto 2.32.0 the dead-file family rides class 3, whose trust fact is the graph family's own per-row confidence; since proto 6.1.0 a public dead verdict (2 unref_public or 4 unreach_public) is refused as public_surface before that confidence is weighed, so an exported API is never eraseable however well vouched, and only then does an unvouched confidence refuse the row. Class 0, the same family on the local-count road, retired at proto 4.0.0 when its grace window closed: the position stays frozen and the core refuses it by name. A degraded over-cap reply authorizes nothing, and the safety surface has no knobs.
Lists, beside the graph family's verdicts, every judged declaration whose name no other file in the tree spells, a negative instrument that reports with a visibility code and never judges.
veto = another file spells it | fold (Rust, ≥2 segments ∧ ≥7 chars)
| the file's own exception regions spell it
row = [node, vis, conv] (integers only — a name never rides the wire)
emit ⇔ vis ⊇ {exported, scope-exported} ∧ conv has no exempt bit (0..10)
code = 1 private > 2 restricted > 3 reexported > 0 public
U is a walk of its own: hidden files enter, .git/.ce
are cut by name, an undeclared nested repository is cut whole while a path the root's
.gitmodules declares stays in U as foreign, where it can spell a name and is never
measured. Only .gitignore and .ceignore are
honoured, so one commit yields one U on any machine, and every parameter of the walk is a
MENTION_REV input. The mentions store keeps 64-bit hashes, never text. Rust
extracts each declaration's mention name and category word and asks the three veto questions
cheapest first; Haskell reads only the visibility word and the mounts row (how many of the
file's mod mounts are private, whether a façade re-exports it, whether its own
package keeps it private), and assigns the frozen total order above. Both tables ride
graph.request together or not at all: a request without them gets the ten-key
reply byte for byte, and the dead set is the same either way. Past unmentionedCap
the core still judges the graph, drops the table and says so. The advisory never turns a gate
red and never enters ce erase.
The families above read a tree. The fourteenth reads a change, the pairs one
edit, one session or one commit wrote, at the PreToolUse hook, at the Stop audit and at
ce precommit / ce commitmsg.
Finds the shape an agent leaves when told to remove X: X is gone from the code, and
the same change writes X back as an absence: a heading (no X), an identifier
without_x, a sentence X is no longer needed.
R = ⋃ names(before) \ ⋃ alive(after) (structural positions only; keys, never text)
row = [kind, marks, names] kind ∈ {0 bracketed, 1 bare, 2 prose}
site ⇔ names ≥ 1 ∧ (kind ≠ 2 ∨ marks ≥ 1) (the conjunction, read per sentence)
over ⇔ budget declared ∧ |sites| > budget (spoken at [tombstone] tier; ships observe)
A name is what a text declares: an identifier outside comments and literals, a
unit name, a heading, a list lead; an inline code span only mentions, and keeps a name alive.
Rust reads the two surfaces the change added (new headings, unit names and file stems; the
added lines of comments, docstrings and paragraphs) and sends one row of three integers per
candidate; Haskell seats the sites and judges the budget, so no name and no path ever crosses
the wire. A document in the changelog role is exempt whole and counted by path convention,
by the shape of a version-indexed ledger, or by [tombstone] ledger, and a quote
run or section that is a ledger by itself exempts only itself. The class ships at
observe and writes every measurement to the feed; the nine-round FPR replay over
two histories, recorded in docs/FPR-TOMBSTONE.md, is what any promotion would
have to argue from.
The clone families need two units to look alike. The fifteenth answers "file 2 now has a
function that plays the same part as one in file 1" when the two share no line, through a sparse
retrieval over the index's own facts, judged as advice at ce similar, the MCP tool
similar_units, the GUI's similar screen and one line of the Stop audit.
Ranks the units most like one unit (or a text) by integer BM25 over six-channel term bags read off facts the index already carries, and asks the core which of them play the query's role: a deterministic, offline "code RAG" with no model and no float, advisory only.
bag = six channels N P C D S L (facts the parse already carries; hashes, never words) score = Σ w · idf · 22·tf·avg / (10·tf·avg + 3·avg + 9·len) (k1 = 6/5, b = 3/4; integer fixed point) widen = top-m PPMI neighbours at ≤ ½ weight (opt-in view; never evidence) role ⇔ (N ≥ 1 ∧ C ≥ 1) ∨ (N ≥ 2 ∧ shapeEqual) (judged in Haskell over similar/1; advisory only)
Every unit of the T3 universe gets one bag: name pieces, shape, callee spellings, its own
doc segment, the node-kind histogram and literal kinds, each word stemmed and hashed with its
channel letter, so a shared name and a shared callee are different evidence. The bags persist as two
tables inside the one content-hash-gated refresh (bag is its own posting list; the pair
table the spec first drew was measured at 7–10× the cold index and not built, so the reader derives
co-occurrence at query time). Ranking is written once against a trait, so the in-memory corpus the
instruments build and the SQL reader the product runs agree on every unit of five corpora. Rust sends
the query bag and one nine-integer row per candidate; Haskell orders them as exact rationals and
applies the role conjunction; names, words and paths never cross the wire. Two arbitrated oracle
generations hold the floors: 60 % on the first sample, 40 % on the holdout that read one step lower
and retired every tuning candidate the first sample had favoured. The family never reddens a gate
and never enters ce erase.
PreToolUse is a behavior-shaping layer, not a
security boundary: an agent can bypass it with Bash: echo >>
or sed -i; the backstop is the Stop audit over
git diff, which is write-tool agnostic, plus the CI gate. And the hook is
deliberately fail-open: any internal failure allows the edit, and the
degraded run lands in the observe feed rather than being silently read as "no
duplicates". Neither property is a gap to be patched later; both are written into the
plan, because a gate that lies about its own reach is worse than one that states it.