fix(memory): reconcile procedures by meaning, not exact name (#1818)#1828
Conversation
Verdict: ApproveThis fixes a real correctness gap in the procedural-memory tier: supersession was keyed on the recipe's exact No security, CLI, API, or docs surface is touched, and the "go-forward only" scope (not cleaning up rows already duplicated by prior drift) is the right call for an off-by-default feature. Nothing blocking — safe to merge. 🔍 Technical detailsVerified contracts
🟢 Minor (optional)
Strengths
|
|
Nice work! One area we may consider as a follow-up to all this is enabling ability for an agent to write code and store it as a tool. This would be a slightly more advanced technique as we would need to leverage two agents that would communicate with eachother. One that is task-dependent and another that is a coding agent that is designed to write python programs as tools. |
Summary
Procedural memory now keeps one proven recipe per goal even when the distiller renames it between synthesis passes — recipes are matched by meaning, not by an exact name string.
Why
The procedural-memory tier (#887) promises "learn a task once, keep one proven recipe, replace it when a better version appears." Supersession is the replace half. But
reconcile_and_storedecided "same recipe?" by exact-string match onname— and the distiller LLM authors a fresh kebab name each pass (DISTILL_TEMPERATURE = 0.1, kebab-validated only, never canonicalized). So a recurring goal that drifts fromsummarize-unread-emailsone pass tosummarize-my-unread-emailsthe next slipped past the name lookup: before, a drifted name produced a second ADD and the supersede branch silently never fired — duplicates accumulated,recall_skill(top_k=2)could surface two near-identical copies into the planner prompt, and a genuinely better recipe coexisted with the stale one instead of replacing it. After, reconcile matches the nearest enabled, non-superseded procedure bywhen_to_useembedding cosine>= SIMILARITY_TAU(0.82), so a drifted name still resolves to the same recipe and supersession fires as designed. Go-forward only — cleaning up rows already duplicated by prior drift is out of scope (the feature is new and off-by-default).Linked issue
Closes #1818
Changes
reconcile_and_storenow finds the prior recipe via a new pure helper_nearest_enabled_procedure(cosine>= similarity_tauoverwhen_to_usevectors), reusing the embedding the call site already computes and passes asembedding=— previously persisted but never used for matching. The supersede-by-success_countdominance check, the insert-new-then-superseded_bylineage, the ADD/UPDATE/NOOP shape, and "never DELETE" are all unchanged; only the match source moved.search_skills(..., with_embedding=True), which sees rows written earlier in the samemax_clusters_per_passloop (put_skillcommits immediately) — the FAISS index isn't refreshed until after reconcile returns, so reusing it could let a same-pass double-ADD through. This also keepsreconcile_and_storea pure function (store-only) that unit-tests without a live backend._RECONCILE_SCAN_LIMIT = 500; saturating it logs awarningrather than silently dropping rows. Degraded inputs (missing/zero-norm/wrong-dim embeddings) are skipped with intent, not swallowed.np.frombuffersoskill_synthesisimports nomemorysymbol, preserving thememory → procedural_memory → skill_synthesisone-way direction.Test plan
python -m pytest tests/unit/test_skill_synthesis.py tests/unit/test_memory_mixin.py tests/unit/test_memory_store.py -q— all green (578 passed locally), including the 4 new pure-fn tests, the e2e two-pass drift test, and the 3 updated pure-fn tests.python util/lint.py --all— black/isort/pylint/flake8 clean (remaining MyPy/Bandit warnings are pre-existing and not in the touched files).reconcile_and_storeto the oldsearch_skills(name=candidate.name, …)lookup and re-run the four meaning tests — they must FAIL on the old code and PASS on the fix, proving they exercise the changed branch (not vacuous).Deviations from the approved sketch
search_skillscosine pass)"_proc_faiss_indexlives on the mixin and is refreshed only afterreconcile_and_storereturns; the function is pure and takes onlystore.search_skillscosine pass — keeps the function pure/unit-testable and catches same-pass ADDs the index wouldn't. No behavior gap vs the sketch.procedural_memory.py:86procedural_memory.py:463and passed asembedding=.test_noop_when_existing_dominatesandtest_update_supersedes_lower_success_countseeded rows with no embedding and relied on name equality.Checklist
Closes #1818).python util/lint.py --all,pytest tests/unit/).Acceptance-criteria proof — #1818
The match key in
reconcile_and_storemoved from exactnametowhen_to_useembedding cosine>= SIMILARITY_TAU(0.82) via a new live-SQLite cosine pass_nearest_enabled_procedure(skill_synthesis.py:580); dominance, lineage, and never-DELETE are unchanged.Every AC is proven by a deterministic test that exercises the changed branch — and each is shown to fail on the old name-match code and pass on the fix (regression proof below), which is what makes the green meaningful.
AC → proof map
when_to_usecosine>= SIMILARITY_TAU, notname; recurring goal → one enabled row under name drifttest_matches_by_meaning_supersedes_under_name_drift(:542),test_distinct_meaning_adds_even_with_same_name(:578)name+ orthogonal vector (cosine 0 < τ) → 2 rows (ADD). Proves the key is meaning, not name.success_countcandidate supersedes (insert-new +superseded_by, never DELETE); dominance unchangedtest_matches_by_meaning_supersedes_under_name_drift(:542),test_lower_success_same_meaning_noops(:597),test_update_supersedes_lower_success_count(:513),test_noop_when_existing_dominates(:494)action=="update",superseded_id==old_id, old row kept withsuperseded_byset (never deleted); weaker cluster under a drifted name →noop(dominance logic intact under the new key).test_name_drift_across_passes_supersedes(test_memory_mixin.py:3331) — two real_synthesize_skillspassessummarize-unread-emails; pass 2 (driftedsummarize-my-unread-emails, highersuccess_count) →stored==1(UPDATE);search_skills()returns 1 row; pass-1 idsuperseded_bythe pass-2 id.Comment #4772627749 refinements — both addressed
1. "Live SQLite + cosine pass, not
_proc_faiss_search" (avoids a stale-index same-pass double-ADD)._nearest_enabled_procedurescansstore.search_skills(enabled_only=True, include_superseded=False, with_embedding=True)— the live table, never the FAISS index. Pinned by a contract-shape spy:2. "Assert the surviving row carries the second candidate's name/body, not just count==1" (catches a wrong-direction supersede). Both the pure-fn and e2e AC tests assert name and body of the survivor:
visible[0]["name"] == "summarize-my-unread-emails"andvisible[0]["markdown_body"] == "# New\n1. better step".visible[0]["name"] == "summarize-my-unread-emails"and"digest" in visible[0]["markdown_body"], plusold[0]["superseded_by"] == visible[0]["id"].Verbatim results
All AC tests pass on the fix:
test_matches_by_meaning_supersedes_under_name_drift PASSED
test_distinct_meaning_adds_even_with_same_name PASSED
test_lower_success_same_meaning_noops PASSED
test_reconcile_queries_store_by_embedding_not_name PASSED
test_update_supersedes_lower_success_count PASSED
test_noop_when_existing_dominates PASSED
test_name_drift_across_passes_supersedes PASSED (e2e, 2 synthesis passes)
test_rerun_is_noop_and_never_deletes PASSED (never-DELETE)
8 passed in 1.28s
Regression proof — the tests catch the bug (temporarily revert the match line to the old
search_skills(name=…)lookup):=== AC tests against OLD exact-name code — MUST FAIL ===
FAILED test_matches_by_meaning_supersedes_under_name_drift
FAILED test_distinct_meaning_adds_even_with_same_name
FAILED test_lower_success_same_meaning_noops
FAILED test_name_drift_across_passes_supersedes
4 failed in 1.05s
=== same 4 against the fix — MUST PASS ===
4 passed in 0.78s
Full suite for the touched modules: 578 passed;
python util/lint.py --all→ ALL QUALITY CHECKS PASSED.