feat(memory): agent memory v2 — second brain with hybrid search, LLM extraction, and observability dashboard#606
Merged
Merged
Conversation
e0eff31 to
068eead
Compare
## Summary - **`gaia init` now installs RAG dependencies** for `chat`, `rag`, and `all` profiles — adds `pip_extras` field to profile definitions and a new `_install_pip_extras()` step that detects editable vs package install, tries `uv pip` first with `pip` fallback - **Added `self.rag` None guards** to 8 RAG tools in `rag_tools.py` that were crashing with `'NoneType' object has no attribute 'index_document'` when RAG deps not installed - **Widened ChatAgent RAG init exception catch** from `ImportError` to `Exception` with warning-level logging and debug traceback - **Updated Agent UI docs** to include `[rag]` in install instructions (`[ui,rag]`) ## Test plan - [x] Lint passing (black, isort, pylint, flake8) - [x] All 1104 unit tests passing - [ ] `gaia init --profile chat` installs RAG deps automatically - [ ] Agent UI document indexing works after `pip install -e ".[rag]"` - [ ] RAG tools return actionable error when deps not installed (instead of crashing) 🤖 Generated with [Claude Code](https://claude.com/claude-code)
C-1: Guard winreg import and all registry-scanning methods in discovery.py
so the module loads cleanly on Linux/macOS where winreg is absent.
Also guard _scan_credential_manager() behind sys.platform check to
avoid subprocess.CREATE_NO_WINDOW AttributeError on non-Windows.
C-3: Replace direct _lock/_conn access in CLI with two new MemoryStore
public methods: get_source_counts() and delete_by_source(source).
delete_by_source() wraps FTS cleanup + DELETE in a single atomic
transaction with rollback, removing the per-ID loop that could
leave knowledge/FTS diverged on partial failure.
C-4: Add close_store() to memory router module; call it from FastAPI
lifespan shutdown so the WAL is checkpointed and the SQLite
connection is released cleanly on server exit.
M-2: list_knowledge endpoint now excludes sensitive items by default.
New include_sensitive=false query param (default false) controls
visibility; sensitive=true still filters to sensitive-only.
M-6: Add append-only comment to conversations FTS trigger block noting
that an AFTER UPDATE trigger would be required if store_turn()
ever changes to update existing rows.
Tests: +9 tests (394 total) covering get_source_counts, delete_by_source
rollback discipline, and all three sensitive filter modes in the router.
- Fix _original_user_input=None fallback bug in _after_process_query (getattr default ignored None; switch to `or` to handle init state) - Extract VALID_CATEGORIES/MAX_CONTENT_LENGTH/MAX_TURN_LENGTH and other magic numbers to named module-level constants in memory_store.py - Import constants in memory.py to eliminate duplicate category sets and ensure truncation limits stay in sync across all call sites - DRY: memory router imports VALID_CATEGORIES from data layer instead of redefining its own copy - Clean up unused imports in test files (F401/F811 flake8 violations) - 394 unit tests passing, flake8 clean
Replace substring `"github.com" in url_lower` with urlparse().hostname comparison to fix CodeQL CWE-20 "Incomplete URL substring sanitization". A crafted URL like http://evil.com/github.com could otherwise bypass the check. Hostname equality/suffix match is unambiguous.
Security: - recall tool now filters out sensitive items before returning results to the LLM — sensitive entries (API keys, credentials) are for internal use only and must not appear in tool output. Performance: - Add get_by_category_contexts() to MemoryStore: single SQL query with WHERE context IN (active, 'global') replaces two separate get_by_category() calls in _get_context_items(), halving DB round-trips per system-prompt build (was 6 queries, now 3). - Replace N+1 correlated subquery in get_sessions() with a LEFT JOIN on MIN(id) per session — scales linearly regardless of session count. Reliability: - Add PRAGMA busy_timeout=5000 so concurrent WAL readers/writers in the same process (dashboard REST singleton + ChatAgent) retry for 5 s instead of failing immediately with SQLITE_BUSY. Correctness: - update_memory tool truncation check now uses MAX_CONTENT_LENGTH constant instead of hardcoded 2000, keeping it in sync with memory_store.py. Testability: - Replace sys.exit(1) in _bootstrap_chat/_bootstrap_discover/_bootstrap_reset helpers with raise RuntimeError; _handle_memory_bootstrap catches and exits, making helpers unit-testable in isolation. Tests (+34): - TestGetByCategoryContexts (5): single-query context+global fetch - TestGetAllKnowledgeSortByValidation (4): sort_by whitelist protection - TestGetSessionsFirstMessageV2 (3): join-based first_message - test_memory_discovery.py (22): _classify_remote, _classify_path, _classify_domain, scan_all structure, Windows guard 428 tests passing, 1 skipped (Windows-only guard on non-Windows).
# Conflicts: # src/gaia/agents/chat/agent.py # src/gaia/apps/webui/src/App.tsx # src/gaia/apps/webui/src/components/ChatView.tsx # src/gaia/ui/server.py
d4fdb90 to
a06f9cc
Compare
Comprehensive rewrite of agent-memory-architecture.md as a single unified design document. Key changes: - Hybrid search: vector (FAISS) + BM25 (FTS5) + RRF fusion + cross-encoder reranking (ms-marco-MiniLM-L-6-v2). No fallback — embeddings are a hard requirement. - Mem0-style LLM extraction: ADD/UPDATE/DELETE/NOOP operations against existing memory, replacing naive extract-and-store. - Zep-inspired fact lineage: superseded_by column preserves history when facts are corrected rather than silently overwriting. - Hindsight-inspired background reconciliation: pairwise similarity check on startup detects contradictions missed at extraction time. - Complexity-aware recall depth: adaptive top_k (3/5/10) based on query complexity heuristics. - Temporal range search: time_from/time_to on all search methods for natural time-based recall. - Conversation consolidation: auto-distill old sessions to durable knowledge before 90-day prune. - Second brain use cases: journaling, meeting notes, PKM, reminders, wake-up scheduling, recurring commitments. - Removed all graceful degradation / silent fallback patterns. - Removed openjarvis-memory-analysis.md (temp analysis doc). Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
…coverage, temporal+superseded filters - POST /api/memory/consolidate, /reconcile, /rebuild-embeddings - GET /api/memory/embedding-coverage - Updated GET /api/memory/knowledge with include_superseded, time_from, time_to - Updated GET /api/memory/stats with embedding coverage and reconciliation stats - 95 tests passing, lint clean Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
…d_by, temporal search, consolidation - Schema v1→v2 migration: embedding BLOB, superseded_by TEXT, consolidated_at TEXT - New methods: store_embedding, get_items_with/without_embeddings, get_unconsolidated_sessions, mark_turns_consolidated, get_items_for_reconciliation - Updated search() with time_from/time_to, superseded_by IS NULL, use_count increment - Updated all query methods with superseded_by IS NULL filter - 275 tests passing, lint clean Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
…LLM extraction, temporal recall Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
… FAISS, API integration Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
…ledge browser, activity timeline, tool stats
6-section dashboard: header stat cards, 30-day activity bar chart,
paginated knowledge browser with entity/category/context/search filters,
tool performance table, conversation history with FTS search,
upcoming & overdue temporal panel.
Features:
- Embedding coverage indicator with progress bar
- Maintenance dropdown: consolidate, rebuild embeddings, reconcile, rebuild FTS
- Click-to-expand knowledge row detail (metadata, timestamps, superseded_by chain)
- Inline actions: edit, delete, toggle sensitive, copy ID
- Superseded entries toggle with server-side filtering
- Toast notification system for all CRUD and maintenance operations
- Brain icon in sidebar for navigation
- Keyboard support: Escape key (layered close), Enter/Space on rows
- ARIA labels, roles, and aria-live for accessibility
- Responsive layout (3 breakpoints)
- Relative date formatting ("in 2 days", "3 days ago")
- API calls aligned with backend router field names
Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
…em0 extraction, consolidation, reconciliation Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
The backend returns metadata as parsed JSON (dict), not a string. Rendering it directly showed [object Object]. Now uses JSON.stringify for object metadata and plain text for strings. Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
…e cases - Strengthen conversation context filtering test with explicit zero-result assertions instead of vacuous loop - Add due_at validation, empty-list consolidation, and history limit tests - Remove dead _past_iso import from API test file - 117 tests, all passing Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
…m0 extraction, consolidation, reconciliation Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
…up scope includes entity, dynamic context always returns time - MemoryStore.search(): corrected from "hybrid" to "FTS5 keyword search" (hybrid is MemoryMixin._hybrid_search) - get_memory_dynamic_context(): fixed "returns empty" claim — always returns current time - store() dedup scope: category+context+entity, not category+context - get_items_with_embeddings(): added missing top_k, time_from, time_to params - _classify_query_complexity: added missing medium/complex signal words - get_entities(): added missing last_updated field in return - Added undocumented update_confidence() and delete_by_source() methods - update(): noted embedding cleared on content change Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
… fixes - memory_store.py: set embedding=NULL when content changes in update() to force re-embedding (stale embedding would return wrong results) - server.py: alphabetize router imports - test fixes: formatting cleanup, mixin test updates from parallel tasks Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
pull Bot
pushed a commit
to bhardwajRahul/gaia
that referenced
this pull request
May 15, 2026
## Why this matters v0.18.0 ships agent memory v2 (hybrid-search second brain with LLM extraction and observability dashboard), ChatAgent split into three composable agents (Chat/FileIO/DocumentQA), parallel tool calls, and a Telegram adapter scaffold — plus fixes the RAG-on-PDF timeout with Gemma 4 that broke document Q&A since v0.17.6 and adds CI gates that enforce RAG quality baselines on every future PR. Full notes: `docs/releases/v0.18.0.mdx`. ## What's New - **Agent memory v2** ([amd#606](amd#606)) — Hybrid semantic + keyword search, LLM extraction, observability dashboard via SSE streaming ([amd#1032](amd#1032)). Per-user isolation enforced; extraction runs async so it doesn't add latency. - **ChatAgent split** ([amd#979](amd#979)) — `ChatAgent`, `FileIOAgent`, and `DocumentQAAgent` replace the monolithic class; each composable via `tools=`. Backward-compatible shim preserved. - **Parallel tool calls** ([amd#946](amd#946)) — Multiple `tool_calls` from a single LLM turn are executed concurrently, cutting round-trips for multi-tool workflows. - **Telegram adapter scaffold, Phase 0** ([amd#951](amd#951)) — `gaia telegram start|stop|status`, per-user session isolation, `[telegram]` extras. Phase 1 (message handling + allowed-users gate) tracked in [amd#889](amd#889). - **Connectors: per-MCP toggle + single-writer enforcement** ([amd#1018](amd#1018), [amd#998](amd#998)) — Disable individual MCP servers without removing them; concurrent writes serialised with actionable errors on contention. - **File navigation, web browsing, and write security** ([amd#495](amd#495)) — `FileSearchToolsMixin`, web browsing tool, and scratchpad mixin in `KNOWN_TOOLS`; write tools check `allowed_paths` before dispatch. - **Email UI and policy alerts** ([amd#995](amd#995), [amd#1039](amd#1039), [amd#952](amd#952)) — Pre-scan triage card, in-chat Connect, policy alert cards, and durable receipts for confirmation-gated actions. ## Bug Fixes - **RAG-on-PDF timeouts on Gemma 4** ([amd#1034](amd#1034), closes [amd#1030](amd#1030)) — Prompt-size budget check added at composition time; CI gates enforce it on every PR ([amd#1040](amd#1040)). - **Envelope-level parse failure crashed SD recovery** ([amd#1047](amd#1047), closes [amd#1023](amd#1023)) — Falls through to a clean recovery path with step-1 context preserved. - **Windows-path tool args corrupted** ([amd#1027](amd#1027)) — Backslash normalisation now happens after argument parsing. - **Blender `send_command` hung** ([amd#1026](amd#1026), closes [amd#1022](amd#1022)) — Read timeout applied to persistent-connection servers. - **`gaia chat init` in post-install banner** ([amd#1029](amd#1029), closes [amd#1024](amd#1024)) — Replaced with the correct `gaia init`. - **Keyring treated as required** ([amd#1028](amd#1028)) — Import guarded; optional on systems without `keyring`. - **electron-builder URLs stale** ([amd#953](amd#953)) — Three doc/installer files updated to current download paths. ## Tooling & Docs - **RAG eval CI gates** ([amd#1040](amd#1040), closes [amd#1033](amd#1033)) — RAG quality baselines + prompt-size budget enforced on every PR. - **Fork-PR authors now receive Claude review** ([amd#932](amd#932)) — `allowed_non_write_users: "*"` with prompt-injection mitigations documented. - **Eval runs mandated before merging** ([amd#1036](amd#1036)) — `CLAUDE.md` requires `gaia eval agent` for LLM-affecting changes. - **GAIA website** ([amd#369](amd#369)) — [amd-gaia.ai](https://amd-gaia.ai) live. - **Custom agent guide reorganised** ([amd#997](amd#997)), Lemonade PPA docs ([amd#801](amd#801)), broken Lemonade CLI URL fixed ([amd#996](amd#996)), WhatsApp adapter evaluation spec ([amd#950](amd#950)). ## Release checklist - [x] `util/validate_release_notes.py docs/releases/v0.18.0.mdx --tag v0.18.0` passes - [x] `src/gaia/version.py` → `0.18.0` - [x] `src/gaia/apps/webui/package.json` → `0.18.0` - [x] Navbar label in `docs/docs.json` → `v0.18.0 · Lemonade 10.2.0` - [x] All 28 commits in range (v0.17.6..HEAD) are represented in the notes - [ ] Review from @kovtcharov-amd addressed
This was referenced May 18, 2026
7 tasks
This was referenced May 29, 2026
kagura-agent
added a commit
to kagura-agent/gaia
that referenced
this pull request
May 29, 2026
…work (amd#1204) PR amd#606 renamed .empty-chat* CSS classes to .empty-task* and grew MemoryDashboard.tsx to ~159KB, but the electron framework tests were not updated. Changes: - test_electron_chat_app.js: update CSS selectors (.empty-chat → .empty-task) and TSX className assertion (empty-chat-chip → empty-task-chip) - test_electron_chat_installer.js: add allowlist with 200KB cap for known-large dashboard files while keeping 100KB default for others
kagura-agent
added a commit
to kagura-agent/gaia
that referenced
this pull request
May 29, 2026
Address review feedback from @itomek: 1. Fix remaining aria-label assertions in test_electron_chat_app.js (lines 1086-1087): 'Rename chat' → 'Rename task', 'Export chat' → 'Export task' — same amd#606 rename caught higher up. 2. Align electron major version across all packages to ^42.2.0: - root package.json: ^40.6.1 → ^42.2.0 - src/gaia/electron: ^40.6.1 → ^42.2.0 - src/gaia/apps/webui: ^40.6.1 → ^42.2.0 This resolves the 'consistent electron major version' test failure (expected 40, received 42) in test_electron_framework_integration.js:360. The example, jira, and emr-dashboard apps were already on ^42.2.0.
pull Bot
pushed a commit
to bhardwajRahul/gaia
that referenced
this pull request
May 29, 2026
…work (amd#1204) (amd#1210) ## Summary Fix the permanently-red `Test Electron Framework` CI job caused by stale test assertions after PR amd#606 reworked the empty-state UX. ## Why Since amd#606 merged on 2026-05-08, every PR touching electron-related paths fails CI — forcing reviewers to manually triage "is this red mine?" for unrelated changes. ## Linked issue Closes amd#1204 ## Changes ### `test_electron_chat_app.js` - Update CSS selector assertions: `.empty-chat` → `.empty-task`, `.empty-chat-title` → `.empty-task-title`, `.empty-chat-chip` → `.empty-task-chip` (PR amd#606 renamed these classes) - Update TSX className assertion: `empty-chat-chip` → `empty-task-chip` ### `test_electron_chat_installer.js` - Add allowlist for known-large dashboard files (`MemoryDashboard.tsx`, ~159KB since amd#606 added the full observability dashboard) - Allowlisted files get a 200KB cap; all other components keep the 100KB default - This follows acceptance criteria option 2-C from the issue ## Test plan - The assertions now match the actual CSS class names in `ChatView.css` (`.empty-task`, `.empty-task-title`, `.empty-task-chip`) and the actual className in `ChatView.tsx` (`empty-task-chip`) - `MemoryDashboard.tsx` at 159KB passes the 200KB allowlist cap - All other component files still enforced at 100KB - No production code changed — test-only fix --- 🤖 **Disclosure:** This PR was authored by [Kagura](https://github.com/kagura-agent), an AI agent. Open source contribution is one of the things I do — you can see my work history [here](https://github.com/kagura-agent/github-contribution). If you'd prefer not to receive AI-authored PRs, just let me know and I'll stop — no hard feelings. --------- Co-authored-by: Tomasz Iniewicz <[email protected]>
kovtcharov-amd
pushed a commit
that referenced
this pull request
Jun 3, 2026
Resolve conflicts in CLAUDE.md and docs/reference/cli.mdx by taking main's versions: PRs #1337, #1340, #1364 already synced both files to the code more recently, and the PR's stale side reintroduced removed commands (mcp add/remove) and the broken /guides/summarize link that was failing the internal cross-reference CI check. Keep the PR's still-valid accuracy fixes (Gemma-4-E4B-it default in chat/llm/emr docs, agent-ui Lemonade port 13305 + #install anchor, code agent Python-gen still supported, telegram.mdx removal, #606 stale-note removal). Fix one contradiction the merge surfaced: main's CLAUDE.md and emr/cli.py listed the EMR VLM as Qwen3-VL-4B, but agent.py and all --vlm-model defaults are Gemma-4-E4B-it-GGUF. Aligned both with the code and emr.mdx.
pull Bot
pushed a commit
to bhardwajRahul/gaia
that referenced
this pull request
Jun 3, 2026
…tStore (amd#1368) ## Why this matters Clicking the **Memory Dashboard** button in the sidebar (or the Memory icon in the chat-toolbar) had no effect when the **Settings** page was currently open. The user had to close Settings first before Memory Dashboard would respond. The reverse direction "worked" but left a stale `showMemoryDashboard=true` in the store, so closing Settings via the back arrow could re-surface the Memory Dashboard from a previous session. The Agent UI is not router-based. "Routing" between top-level views is a stateful ternary in `App.tsx:531-535` that gives `showSettings` unconditional priority over `showMemoryDashboard`, and the two store setters never cleared each other's flag. ## What changed Single-source-of-truth fix in `src/gaia/apps/webui/src/stores/chatStore.ts:262-265`: the two setters are now mutually exclusive — opening one view always closes the other. This is better than patching all three call sites (sidebar × 2, chat-toolbar × 1) because future call sites cannot reintroduce the bug. ```diff - setShowSettings: (show) => set({ showSettings: show }), - setShowMemoryDashboard: (show) => set({ showMemoryDashboard: show }), + setShowSettings: (show) => + set(show ? { showSettings: true, showMemoryDashboard: false } : { showSettings: false }), + setShowMemoryDashboard: (show) => + set(show ? { showMemoryDashboard: true, showSettings: false } : { showMemoryDashboard: false }), ``` `onHome` in `App.tsx:524` still works correctly — it sets both flags to `false` explicitly, and the `false` path of each setter does not touch the other flag. ## Test plan - [ ] Manual: open Settings from the sidebar, then click the Memory Dashboard button — Memory Dashboard renders (not Settings). - [ ] Manual: open Memory Dashboard, then click Settings — Settings renders (no change there, but verify flag state is clean). - [ ] Manual: from each view, hit the back arrow — return to the chat / welcome view, not to whatever was opened "behind" it. - [ ] Add Playwright regression covering the "open Settings, click Memory Dashboard, verify Memory Dashboard is the rendered view" scenario and its symmetric counterpart. ## Notes - Bug introduced by amd#606 (commit `74f637a4`) which added `MemoryDashboard` and the `showMemoryDashboard` flag without making the two flags mutually exclusive. - Triage confirmed the fix location on the issue. Fixes amd#1367
kovtcharov-amd
pushed a commit
that referenced
this pull request
Jun 4, 2026
Memory is a user-controlled setting, so the dynamic tool loader must inherit that switch explicitly: when memory is off, the loader reverts to the legacy behavior of exposing every registered tool — identical to a build without this feature. Adds a dedicated section making this a documented off-state (not a degradation), with a table collapsing the three "load everything" conditions — user-disabled memory, loader toggle off, and embedder failure — onto the same safe floor so a user can never lose tool access by toggling memory. Cross-referenced from the Part 1 build steps and the #606 dependency note.
4 tasks
pull Bot
pushed a commit
to bhardwajRahul/gaia
that referenced
this pull request
Jun 18, 2026
…1732) ## Why this matters PR amd#606 shipped three of CoALA's four memory tiers (working / episodic / semantic) and left the **procedural** tier — distilling recurring successful tool sequences into reusable skills — as a follow-up (amd#887). The issue's own design notes predate amd#691 locking and the current code, so a contributor building straight from them would rebuild against a stale picture — exactly how amd#688's spec drifted across four unmerged PRs. This lands one canonical, **code-accurate** spec at `docs/plans/skill-synthesis.mdx`, grounded line-by-line on the merged memory layer and reconciled against the format it *emits* (amd#691) and the tool-loader that *consumes* it (amd#1451), so design and implementation can't diverge before the first line is written. ## Linked issue Refs amd#887 <!-- design/spec only — the synthesis pipeline implementation lands separately --> ## Changes - **Separate `procedures` table + additive v2→v3 migration** — procedural memory diverges from `knowledge` on every lifecycle axis (empirical track-record vs belief-strength, multi-step body vs one sentence, its own goal-trigger embedding index); sharing the table would `NULL` every procedure-only column on existing rows and pollute the recall index. - **`recall_skill` is an internal method, not a sixth `@tool`** — the planner calls it programmatically at turn start, so the agent gains the capability without enlarging the five-tool memory registry it must reason about. - **Emits the locked amd#691 format by reference (the emit/inject split)** — the LLM emits only the fields it *derives*; `Skill.parse()` injects the fixed constants (`license`, `version`) and maps to `metadata.gaia.tools_required`, so synthesized files validate against the format instead of re-inventing it. - **Fail-loud off-states + content-trust boundary** — Lemonade-down / malformed-output / embedder-fail each skip-and-log or re-raise (no silent smaller-model swap, per CLAUDE.md); injected bodies are internal-corpus-only, capped at `MAX_RECALL_BODY_CHARS`, never auto-exported. - **Corrects the issue's drifted framing** — re-pins the synthesis hook to the real `_run_memory_post_init`, config to the existing `memory_settings.json`, and retracts the "new tool" / intermediate-frontmatter-on-disk framing the pre-amd#691 pseudocode implied. ## Test plan - [x] `python -c "import json; json.load(open('docs/docs.json'))"` — nav JSON valid - [x] New page `plans/skill-synthesis` registered under **Ecosystem** in `docs.json` - [x] Mintlify preview renders `docs/plans/skill-synthesis.mdx` — tables, `<Info>`/`<Note>`/`<Warning>`, the `<AccordionGroup>` examples, and all internal anchor links resolve - [ ] Maintainer sign-off on the design direction before implementation begins (the five items under **Open questions** are explicitly left as PR-time confirmations, not blockers) Co-authored-by: Alexey Tyurin <>
This was referenced Jun 19, 2026
alexey-tyurin
added a commit
to alexey-tyurin/gaia
that referenced
this pull request
Jun 22, 2026
amd#1794) ## Summary Today a GAIA agent re-plans every multi-step task from scratch — solve the same kind of goal ten times and it derives the tool sequence from zero all ten times, with no memory that it already found a working recipe. This PR adds the **procedural** memory tier (the one amd#606 deliberately left out): when the same kind of goal succeeds ≥3 times with the same shape of work, GAIA distils that successful tool sequence into a reusable, `SKILL.md`-shaped procedure, stores it, and — the next time a similar goal appears — injects the proven recipe into the planner so it reuses it instead of re-planning. It is automatic (no command, no install) and **off by safe default**: with zero procedures or memory disabled, the planner prompt is byte-identical to today. ## Why amd#606 shipped CoALA-style working/episodic/semantic memory over a SQLite `MemoryStore` with a `tool_history` table recording every tool call + outcome, but left the *procedural* tier as a follow-up. amd#887 closes that loop so repeated work gets cheaper over time — the agent learns its own recipes from its own successful runs. This is a learning loop, **not** a marketplace: synthesis only *creates* procedures into a local `procedures` table from the agent's own history; browsing/installing/sharing (amd#647) and the on-disk loader/CLI (amd#691) are out of scope. ## Linked issue Closes amd#887 ## Changes - **Synthesis pipeline (new `skill_synthesis.py`).** Detect successful ≥3-step tool spans, cluster by embedded user-goal at cosine ≥ 0.82, and distil clusters that recur ≥3× at ≥80% success into a procedure — at most 10 clusters per pass. Reuses the existing nomic-768 embedder and the `chat.send_messages` seam the extraction loop already uses — **no new model or client**. Runs as one extra step inside amd#606's first-query maintenance pass, off the request hot path. - **`procedures` store + dedicated recall index.** Additive **v2→v3** migration adds a `procedures` table with its **own** FAISS index (the `knowledge` index is untouched); reconcile only **ADD / UPDATE / supersede — never DELETE**. `recall_skill(goal, top_k=2)` vector-searches that index and injects the matched body (capped at 1500 chars) into the planner's system prompt. `recall_skill` is an **internal method, not a 6th `@tool`** — the five-tool memory registry is unchanged. - **Off by safe default, fail-loud.** `GAIA_MEMORY_DISABLED=1` and `"skill_synthesis": {"enabled": false}` both short-circuit synthesis *and* recall; a disabled/superseded row is never recalled. Embedder failure **re-raises with context**; Lemonade-down and malformed distill **skip + log**, producing zero rows — no silent fallback. - **Observability + docs.** `gaia memory status` prints a synthesized-procedures count; `docs/guides/memory.mdx` gains a "Procedural memory (skills)" section covering the thresholds, off-switches, and supersede-not-delete lineage. - **Structural (code-review follow-up).** The procedural orchestration (proc-FAISS index + recall + synthesis) is extracted from `memory.py` into a new `ProceduralMemoryMixin` (`procedural_memory.py`), so `memory.py` doesn't grow ~450 lines. Behavior-preserving — the memory suite is the net. ## Deviations from the approved spec — please ratify The ratified spec (`docs/plans/skill-synthesis.mdx`) was written against an assumed schema; these are where the code diverges to fit what's actually on `main`. Each is intentional: - **Goal source.** `tool_history` has no goal column, so the cluster goal is the first `role='user'` turn of the session, derived via a `conversations` JOIN inside the new `iter_sessions(since, min_steps)` — **plain SQL, no LLM, one row per session**. (Spec assumed an embeddable `user_goal` per row.) - **`iter_sessions` is new.** No per-session successful-span iterator existed; added as a single read-only grouped query (avoids N per-session reads). - **Distill client.** Reuses `self.chat.send_messages` at low temperature rather than introducing an abstract `llm` — same client the extraction pass uses. - **Recall-injection seam.** `_compose_system_prompt` is cached and MemoryMixin didn't contribute to it, so recall is wired through the existing `_get_mixin_prompts` auto-discovery (`get_recalled_skills_system_prompt`) with a per-turn `_refresh_recalled_skills` that recomposes only when the recalled set changes. - **Config home.** The 5 thresholds live on the existing `~/.gaia/memory_settings.json` (`skill_synthesis` section), not a new `config.toml` section (which doesn't exist). - **Emit/inject split.** The LLM emits only `name` / `when_to_use` / `tools_required` / body; `Skill.parse` injects the fixed `license: MIT` / `version: 1.0.0` (the maintainer's pseudocode put `version` in the LLM output — moved to injection because it's fixed, not derived). - **Deterministic clustering.** Goal clustering is order-independent so a given history always yields the same procedures. - **Store-layer extraction deferred.** Only the `memory.py` mixin was extracted; splitting the `memory_store.py` procedural code (schema string + woven migration) is a higher-risk, separate follow-up. **For maintainer:** the v2→v3 additive `procedures` migration needs ratification. ## Test plan All commands run from the repo root against the `.venv`. Results are from this branch tip. - [x] **Lint** — `.venv/bin/python util/lint.py --all` → **ALL QUALITY CHECKS PASSED — Ready for PR submission** (Black / isort / Pylint / Flake8 / imports / dependabot / doc-versions all PASS; 3 non-blocking warnings, both pre-existing and unrelated to this PR — see Proof). - [x] **Full unit suite** — `.venv/bin/python -m pytest tests/unit/ -q -p no:cacheprovider 2>&1 | grep -E "FAILED|ERROR|passed|failed|error" | tail -30` → **5294 passed, 77 skipped**; 6 failures, all **pre-existing on `main`** and outside this PR's diff (proof below). The 563 procedural-memory tests (`test_skill_synthesis.py` + `test_memory_store.py` + `test_memory_mixin.py`) all pass. - [x] **Observability** — `.venv/bin/gaia memory status` → prints `Procedures (skills): 0 synthesized` (0 is the correct cold-start floor on a DB with no qualifying history). - [x] **Behavioral eval (LLM-affecting gate; serial — one `gaia eval agent` at a time):** ```bash # Terminal 1 .venv/bin/python -m gaia.ui.server --port 4200 --host 127.0.0.1 # Terminal 2 — confirm nothing else is evaluating, then run ps aux | grep "gaia eval" | grep -v grep | wc -l # must print 0 .venv/bin/gaia eval agent --category rag_quality --agent-type doc ``` → **7/8 passed (100% judged), avg 9.5/10** — no regression vs the committed Gemma baseline (9.47 vs 9.43). The one non-pass is an `INFRA_ERROR`, not a regression (proof below). ## Verification & Proof ### Lint (`util/lint.py --all`) | Code Formatting (Black) | PASS | | Critical Errors (Pylint) | PASS | | Import Sorting (isort) | PASS | | Style Compliance (Flake8)| PASS | | Import Validation | PASS | | Dependabot / Doc-Version | PASS | Total: 10 checks · Passed 7 · Failed 0 · Warnings 3 (non-blocking) [SUCCESS] ALL QUALITY CHECKS PASSED — Ready for PR submission The 3 warnings are not from this PR: a Bandit **B608** (Low/Medium, ML false positive) on a fully-parameterized `IN (?,?,…)` query — the same accepted pattern as the existing `mark_consolidated` — and a pre-existing `DocumentQAAgent` tool-isolation soft-warning in a file this PR does not touch. ### Full unit suite — the 6 failures are pre-existing, not amd#887 6 failed, 5294 passed, 77 skipped, 10 warnings in 453.75s None of the 6 touch this PR's diff, and all 6 **fail identically on `main` (`894d9783`)** with this branch's source removed — confirmed by checking out `main` and re-running the same node IDs: | Failing test | Why (unrelated to procedural memory) | |---|---| | `test_parse_error_recovery.py` ×2 | Agent context-overflow trim/retry in `agent.py` — untouched here | | `test_init_command.py` ×3 | Lemonade-installer platform paths (linux/windows/CI) — macOS-local mock quirk | | `test_memory_router.py::…faiss_missing` | Asserts 503 when faiss is absent, but faiss **is** installed in the local `.venv` → 200 | The procedural-memory suite itself is green: `pytest tests/unit/test_skill_synthesis.py tests/unit/test_memory_store.py tests/unit/test_memory_mixin.py -q` → **563 passed**. ### `gaia memory status` — the new procedures surface renders === GAIA Agent Memory === Knowledge entries: 0 Conversations: 18 turns across 9 sessions Tool calls: 0 (0 unique tools) Procedures (skills): 0 synthesized ← new line (amd#887) Database size: 148.0 KB ### Behavioral eval — no RAG regression (`rag_quality`, doc agent) RUN: eval-20260619-193456 — 7/8 passed (88% all, 100% judged) — avg 9.5/10 budget_query 10.0 · cross_section_rag 9.7 · hallucination_resistance 9.7 negation_handling 9.0 · safety_handbook_water 9.3 · simple_factual_rag 9.7 table_extraction 8.9 · csv_analysis INFRA_ERROR | Metric | Baseline `gemma-4-e4b-d71cd914` | This run | |---|---|---| | Judged pass rate | 100% | **100%** | | Failures | 0 | **0** | | Avg score | 9.43 | **9.47** | The lone `csv_analysis → INFRA_ERROR` is **environmental, not a regression**: that scenario pins `agent_type: data` (AnalystAgent), which was not registered in the running UI server (`chat, doc, file, builder` only — server logged `requested unknown agent_type 'data'`), and it isn't part of the 7-scenario rag_quality baseline. The procedural layer initialized cleanly in every session (`procedures FAISS index rebuilt: 0 vectors`, `registered 5 memory tools (v2)` — recall stays internal), with no error from any new module. ## Checklist - [x] I have linked a GitHub issue above (`Closes amd#887`). - [x] I have described **why** this change is being made, not just what changed. - [x] I have run linting and tests locally (`python util/lint.py --all`, `pytest tests/unit/`). - [x] I have updated documentation if user-visible behavior changed (`docs/guides/memory.mdx`, `gaia memory status`). --------- Co-authored-by: Alexey Tyurin <>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Before: GAIA had no persistent memory — every session started from zero. The agent couldn't remember your name, preferences, project context, or past conversations.
After: GAIA has a second brain. It stores facts, preferences, notes, reminders, and conversation history across sessions using hybrid search (vector + BM25 + cross-encoder reranking). The agent recalls what matters, learns from corrections, and builds contact profiles — all stored locally on the user's machine. Memory is disabled by default (beta) and requires explicit opt-in from the Memory Dashboard.
Key decisions:
X-Gaia-UI: 1headerTest plan
pytest tests/unit/test_memory_store.py tests/unit/test_memory_mixin.py tests/unit/test_memory_router.py tests/unit/test_memory_discovery.py— 588 passpython util/lint.py --all --fix— cleancd src/gaia/apps/webui && npm run build— cleancurl -X PUT .../memory/settingswithout header → 403