Thanks to visit codestin.com
Credit goes to github.com

Skip to content

feat(memory): agent memory v2 — second brain with hybrid search, LLM extraction, and observability dashboard#606

Merged
kovtcharov-amd merged 69 commits into
mainfrom
feature/agent-memory
May 11, 2026
Merged

feat(memory): agent memory v2 — second brain with hybrid search, LLM extraction, and observability dashboard#606
kovtcharov-amd merged 69 commits into
mainfrom
feature/agent-memory

Conversation

@kovtcharov

@kovtcharov kovtcharov commented Mar 20, 2026

Copy link
Copy Markdown
Collaborator

Summary

Before: GAIA had no persistent memory — every session started from zero. The agent couldn't remember your name, preferences, project context, or past conversations.

After: GAIA has a second brain. It stores facts, preferences, notes, reminders, and conversation history across sessions using hybrid search (vector + BM25 + cross-encoder reranking). The agent recalls what matters, learns from corrections, and builds contact profiles — all stored locally on the user's machine. Memory is disabled by default (beta) and requires explicit opt-in from the Memory Dashboard.

Key decisions:

  • Opt-in by default — memory, system discovery, and MCP access all start OFF with beta confirmation dialogs
  • CSRF protection — all mutating memory endpoints require X-Gaia-UI: 1 header
  • Dual consent for system scanning — hardware/software discovery requires explicit toggle, never runs at startup without consent
  • Re-initialize button — one-click wipe + rebuild from the Memory Dashboard

Test plan

  • pytest tests/unit/test_memory_store.py tests/unit/test_memory_mixin.py tests/unit/test_memory_router.py tests/unit/test_memory_discovery.py — 588 pass
  • python util/lint.py --all --fix — clean
  • cd src/gaia/apps/webui && npm run build — clean
  • Memory Dashboard: toggle Memory ON → beta popup appears → confirm → toggle shows On
  • Memory Dashboard: toggle Memory OFF → instant, no popup
  • Memory Dashboard: System discovery ON → popup → confirm → system facts collected
  • Memory Dashboard: Re-initialize → confirmation → clears all + re-collects system
  • Chat: with memory ON, tell agent a fact → new session → agent recalls it
  • Chat: with memory OFF, tell agent a fact → new session → agent does NOT recall
  • CSRF: curl -X PUT .../memory/settings without header → 403

@github-actions github-actions Bot added documentation Documentation changes dependencies Dependency updates agents cli CLI changes tests Test changes electron Electron app changes labels Mar 20, 2026
Comment thread src/gaia/agents/base/discovery.py Fixed
@kovtcharov kovtcharov force-pushed the feature/agent-memory branch from e0eff31 to 068eead Compare March 21, 2026 23:13
itomek and others added 6 commits March 21, 2026 18:10
## Summary

- **`gaia init` now installs RAG dependencies** for `chat`, `rag`, and
`all` profiles — adds `pip_extras` field to profile definitions and a
new `_install_pip_extras()` step that detects editable vs package
install, tries `uv pip` first with `pip` fallback
- **Added `self.rag` None guards** to 8 RAG tools in `rag_tools.py` that
were crashing with `'NoneType' object has no attribute 'index_document'`
when RAG deps not installed
- **Widened ChatAgent RAG init exception catch** from `ImportError` to
`Exception` with warning-level logging and debug traceback
- **Updated Agent UI docs** to include `[rag]` in install instructions
(`[ui,rag]`)

## Test plan

- [x] Lint passing (black, isort, pylint, flake8)
- [x] All 1104 unit tests passing
- [ ] `gaia init --profile chat` installs RAG deps automatically
- [ ] Agent UI document indexing works after `pip install -e ".[rag]"`
- [ ] RAG tools return actionable error when deps not installed (instead
of crashing)

🤖 Generated with [Claude Code](https://claude.com/claude-code)
C-1: Guard winreg import and all registry-scanning methods in discovery.py
     so the module loads cleanly on Linux/macOS where winreg is absent.
     Also guard _scan_credential_manager() behind sys.platform check to
     avoid subprocess.CREATE_NO_WINDOW AttributeError on non-Windows.

C-3: Replace direct _lock/_conn access in CLI with two new MemoryStore
     public methods: get_source_counts() and delete_by_source(source).
     delete_by_source() wraps FTS cleanup + DELETE in a single atomic
     transaction with rollback, removing the per-ID loop that could
     leave knowledge/FTS diverged on partial failure.

C-4: Add close_store() to memory router module; call it from FastAPI
     lifespan shutdown so the WAL is checkpointed and the SQLite
     connection is released cleanly on server exit.

M-2: list_knowledge endpoint now excludes sensitive items by default.
     New include_sensitive=false query param (default false) controls
     visibility; sensitive=true still filters to sensitive-only.

M-6: Add append-only comment to conversations FTS trigger block noting
     that an AFTER UPDATE trigger would be required if store_turn()
     ever changes to update existing rows.

Tests: +9 tests (394 total) covering get_source_counts, delete_by_source
       rollback discipline, and all three sensitive filter modes in the router.
- Fix _original_user_input=None fallback bug in _after_process_query
  (getattr default ignored None; switch to `or` to handle init state)
- Extract VALID_CATEGORIES/MAX_CONTENT_LENGTH/MAX_TURN_LENGTH and other
  magic numbers to named module-level constants in memory_store.py
- Import constants in memory.py to eliminate duplicate category sets
  and ensure truncation limits stay in sync across all call sites
- DRY: memory router imports VALID_CATEGORIES from data layer instead
  of redefining its own copy
- Clean up unused imports in test files (F401/F811 flake8 violations)
- 394 unit tests passing, flake8 clean
Replace substring `"github.com" in url_lower` with urlparse().hostname
comparison to fix CodeQL CWE-20 "Incomplete URL substring sanitization".
A crafted URL like http://evil.com/github.com could otherwise bypass the
check. Hostname equality/suffix match is unambiguous.
Security:
- recall tool now filters out sensitive items before returning results
  to the LLM — sensitive entries (API keys, credentials) are for
  internal use only and must not appear in tool output.

Performance:
- Add get_by_category_contexts() to MemoryStore: single SQL query with
  WHERE context IN (active, 'global') replaces two separate
  get_by_category() calls in _get_context_items(), halving DB round-trips
  per system-prompt build (was 6 queries, now 3).
- Replace N+1 correlated subquery in get_sessions() with a LEFT JOIN on
  MIN(id) per session — scales linearly regardless of session count.

Reliability:
- Add PRAGMA busy_timeout=5000 so concurrent WAL readers/writers in the
  same process (dashboard REST singleton + ChatAgent) retry for 5 s
  instead of failing immediately with SQLITE_BUSY.

Correctness:
- update_memory tool truncation check now uses MAX_CONTENT_LENGTH constant
  instead of hardcoded 2000, keeping it in sync with memory_store.py.

Testability:
- Replace sys.exit(1) in _bootstrap_chat/_bootstrap_discover/_bootstrap_reset
  helpers with raise RuntimeError; _handle_memory_bootstrap catches and
  exits, making helpers unit-testable in isolation.

Tests (+34):
- TestGetByCategoryContexts (5): single-query context+global fetch
- TestGetAllKnowledgeSortByValidation (4): sort_by whitelist protection
- TestGetSessionsFirstMessageV2 (3): join-based first_message
- test_memory_discovery.py (22): _classify_remote, _classify_path,
  _classify_domain, scan_all structure, Windows guard

428 tests passing, 1 skipped (Windows-only guard on non-Windows).
# Conflicts:
#	src/gaia/agents/chat/agent.py
#	src/gaia/apps/webui/src/App.tsx
#	src/gaia/apps/webui/src/components/ChatView.tsx
#	src/gaia/ui/server.py
@kovtcharov kovtcharov force-pushed the feature/agent-memory branch from d4fdb90 to a06f9cc Compare April 1, 2026 16:31
Comprehensive rewrite of agent-memory-architecture.md as a single
unified design document. Key changes:

- Hybrid search: vector (FAISS) + BM25 (FTS5) + RRF fusion + cross-encoder
  reranking (ms-marco-MiniLM-L-6-v2). No fallback — embeddings are a hard
  requirement.
- Mem0-style LLM extraction: ADD/UPDATE/DELETE/NOOP operations against
  existing memory, replacing naive extract-and-store.
- Zep-inspired fact lineage: superseded_by column preserves history when
  facts are corrected rather than silently overwriting.
- Hindsight-inspired background reconciliation: pairwise similarity check
  on startup detects contradictions missed at extraction time.
- Complexity-aware recall depth: adaptive top_k (3/5/10) based on query
  complexity heuristics.
- Temporal range search: time_from/time_to on all search methods for
  natural time-based recall.
- Conversation consolidation: auto-distill old sessions to durable
  knowledge before 90-day prune.
- Second brain use cases: journaling, meeting notes, PKM, reminders,
  wake-up scheduling, recurring commitments.
- Removed all graceful degradation / silent fallback patterns.
- Removed openjarvis-memory-analysis.md (temp analysis doc).

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
@kovtcharov kovtcharov changed the title feat(memory): persistent agent memory system with dashboard UI feat(memory): agent memory v2 — second brain with hybrid search, LLM extraction, and observability dashboard Apr 1, 2026
Karim13014 and others added 7 commits April 1, 2026 15:48
…coverage, temporal+superseded filters

- POST /api/memory/consolidate, /reconcile, /rebuild-embeddings
- GET /api/memory/embedding-coverage
- Updated GET /api/memory/knowledge with include_superseded, time_from, time_to
- Updated GET /api/memory/stats with embedding coverage and reconciliation stats
- 95 tests passing, lint clean

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
…d_by, temporal search, consolidation

- Schema v1→v2 migration: embedding BLOB, superseded_by TEXT, consolidated_at TEXT
- New methods: store_embedding, get_items_with/without_embeddings, get_unconsolidated_sessions, mark_turns_consolidated, get_items_for_reconciliation
- Updated search() with time_from/time_to, superseded_by IS NULL, use_count increment
- Updated all query methods with superseded_by IS NULL filter
- 275 tests passing, lint clean

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
…LLM extraction, temporal recall

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
… FAISS, API integration

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
…ledge browser, activity timeline, tool stats

6-section dashboard: header stat cards, 30-day activity bar chart,
paginated knowledge browser with entity/category/context/search filters,
tool performance table, conversation history with FTS search,
upcoming & overdue temporal panel.

Features:
- Embedding coverage indicator with progress bar
- Maintenance dropdown: consolidate, rebuild embeddings, reconcile, rebuild FTS
- Click-to-expand knowledge row detail (metadata, timestamps, superseded_by chain)
- Inline actions: edit, delete, toggle sensitive, copy ID
- Superseded entries toggle with server-side filtering
- Toast notification system for all CRUD and maintenance operations
- Brain icon in sidebar for navigation
- Keyboard support: Escape key (layered close), Enter/Space on rows
- ARIA labels, roles, and aria-live for accessibility
- Responsive layout (3 breakpoints)
- Relative date formatting ("in 2 days", "3 days ago")
- API calls aligned with backend router field names

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
…em0 extraction, consolidation, reconciliation

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
The backend returns metadata as parsed JSON (dict), not a string.
Rendering it directly showed [object Object]. Now uses
JSON.stringify for object metadata and plain text for strings.

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
Karim13014 and others added 4 commits April 1, 2026 15:52
…e cases

- Strengthen conversation context filtering test with explicit zero-result
  assertions instead of vacuous loop
- Add due_at validation, empty-list consolidation, and history limit tests
- Remove dead _past_iso import from API test file
- 117 tests, all passing

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
…m0 extraction, consolidation, reconciliation

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
…up scope includes entity, dynamic context always returns time

- MemoryStore.search(): corrected from "hybrid" to "FTS5 keyword search" (hybrid is MemoryMixin._hybrid_search)
- get_memory_dynamic_context(): fixed "returns empty" claim — always returns current time
- store() dedup scope: category+context+entity, not category+context
- get_items_with_embeddings(): added missing top_k, time_from, time_to params
- _classify_query_complexity: added missing medium/complex signal words
- get_entities(): added missing last_updated field in return
- Added undocumented update_confidence() and delete_by_source() methods
- update(): noted embedding cleared on content change

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
… fixes

- memory_store.py: set embedding=NULL when content changes in update()
  to force re-embedding (stale embedding would return wrong results)
- server.py: alphabetize router imports
- test fixes: formatting cleanup, mixin test updates from parallel tasks

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
@itomek itomek mentioned this pull request May 14, 2026
6 tasks
pull Bot pushed a commit to bhardwajRahul/gaia that referenced this pull request May 15, 2026
## Why this matters

v0.18.0 ships agent memory v2 (hybrid-search second brain with LLM
extraction and observability dashboard), ChatAgent split into three
composable agents (Chat/FileIO/DocumentQA), parallel tool calls, and a
Telegram adapter scaffold — plus fixes the RAG-on-PDF timeout with Gemma
4 that broke document Q&A since v0.17.6 and adds CI gates that enforce
RAG quality baselines on every future PR.

Full notes: `docs/releases/v0.18.0.mdx`.

## What's New

- **Agent memory v2** ([amd#606](amd#606)) —
Hybrid semantic + keyword search, LLM extraction, observability
dashboard via SSE streaming
([amd#1032](amd#1032)). Per-user isolation
enforced; extraction runs async so it doesn't add latency.
- **ChatAgent split** ([amd#979](amd#979)) —
`ChatAgent`, `FileIOAgent`, and `DocumentQAAgent` replace the monolithic
class; each composable via `tools=`. Backward-compatible shim preserved.
- **Parallel tool calls** ([amd#946](amd#946))
— Multiple `tool_calls` from a single LLM turn are executed
concurrently, cutting round-trips for multi-tool workflows.
- **Telegram adapter scaffold, Phase 0**
([amd#951](amd#951)) — `gaia telegram
start|stop|status`, per-user session isolation, `[telegram]` extras.
Phase 1 (message handling + allowed-users gate) tracked in
[amd#889](amd#889).
- **Connectors: per-MCP toggle + single-writer enforcement**
([amd#1018](amd#1018),
[amd#998](amd#998)) — Disable individual MCP
servers without removing them; concurrent writes serialised with
actionable errors on contention.
- **File navigation, web browsing, and write security**
([amd#495](amd#495)) — `FileSearchToolsMixin`,
web browsing tool, and scratchpad mixin in `KNOWN_TOOLS`; write tools
check `allowed_paths` before dispatch.
- **Email UI and policy alerts**
([amd#995](amd#995),
[amd#1039](amd#1039),
[amd#952](amd#952)) — Pre-scan triage card,
in-chat Connect, policy alert cards, and durable receipts for
confirmation-gated actions.

## Bug Fixes

- **RAG-on-PDF timeouts on Gemma 4**
([amd#1034](amd#1034), closes
[amd#1030](amd#1030)) — Prompt-size budget
check added at composition time; CI gates enforce it on every PR
([amd#1040](amd#1040)).
- **Envelope-level parse failure crashed SD recovery**
([amd#1047](amd#1047), closes
[amd#1023](amd#1023)) — Falls through to a
clean recovery path with step-1 context preserved.
- **Windows-path tool args corrupted**
([amd#1027](amd#1027)) — Backslash
normalisation now happens after argument parsing.
- **Blender `send_command` hung**
([amd#1026](amd#1026), closes
[amd#1022](amd#1022)) — Read timeout applied
to persistent-connection servers.
- **`gaia chat init` in post-install banner**
([amd#1029](amd#1029), closes
[amd#1024](amd#1024)) — Replaced with the
correct `gaia init`.
- **Keyring treated as required**
([amd#1028](amd#1028)) — Import guarded;
optional on systems without `keyring`.
- **electron-builder URLs stale**
([amd#953](amd#953)) — Three doc/installer
files updated to current download paths.

## Tooling & Docs

- **RAG eval CI gates** ([amd#1040](amd#1040),
closes [amd#1033](amd#1033)) — RAG quality
baselines + prompt-size budget enforced on every PR.
- **Fork-PR authors now receive Claude review**
([amd#932](amd#932)) —
`allowed_non_write_users: "*"` with prompt-injection mitigations
documented.
- **Eval runs mandated before merging**
([amd#1036](amd#1036)) — `CLAUDE.md` requires
`gaia eval agent` for LLM-affecting changes.
- **GAIA website** ([amd#369](amd#369)) —
[amd-gaia.ai](https://amd-gaia.ai) live.
- **Custom agent guide reorganised**
([amd#997](amd#997)), Lemonade PPA docs
([amd#801](amd#801)), broken Lemonade CLI URL
fixed ([amd#996](amd#996)), WhatsApp adapter
evaluation spec ([amd#950](amd#950)).

## Release checklist

- [x] `util/validate_release_notes.py docs/releases/v0.18.0.mdx --tag
v0.18.0` passes
- [x] `src/gaia/version.py` → `0.18.0`
- [x] `src/gaia/apps/webui/package.json` → `0.18.0`
- [x] Navbar label in `docs/docs.json` → `v0.18.0 · Lemonade 10.2.0`
- [x] All 28 commits in range (v0.17.6..HEAD) are represented in the
notes
- [ ] Review from @kovtcharov-amd addressed
kagura-agent added a commit to kagura-agent/gaia that referenced this pull request May 28, 2026
Address review feedback from @itomek:
- "Rename chat" → "Rename task" (line 1086)
- "Export chat" → "Export task" (line 1087)

These match ChatView.tsx:1320,1365 which were renamed in PR amd#606.
kagura-agent added a commit to kagura-agent/gaia that referenced this pull request May 29, 2026
…work (amd#1204)

PR amd#606 renamed .empty-chat* CSS classes to .empty-task* and grew
MemoryDashboard.tsx to ~159KB, but the electron framework tests
were not updated.

Changes:
- test_electron_chat_app.js: update CSS selectors (.empty-chat → .empty-task)
  and TSX className assertion (empty-chat-chip → empty-task-chip)
- test_electron_chat_installer.js: add allowlist with 200KB cap for
  known-large dashboard files while keeping 100KB default for others
kagura-agent added a commit to kagura-agent/gaia that referenced this pull request May 29, 2026
Address review feedback from @itomek:

1. Fix remaining aria-label assertions in test_electron_chat_app.js
   (lines 1086-1087): 'Rename chat' → 'Rename task',
   'Export chat' → 'Export task' — same amd#606 rename caught higher up.

2. Align electron major version across all packages to ^42.2.0:
   - root package.json: ^40.6.1 → ^42.2.0
   - src/gaia/electron: ^40.6.1 → ^42.2.0
   - src/gaia/apps/webui: ^40.6.1 → ^42.2.0
   This resolves the 'consistent electron major version' test failure
   (expected 40, received 42) in test_electron_framework_integration.js:360.
   The example, jira, and emr-dashboard apps were already on ^42.2.0.
pull Bot pushed a commit to bhardwajRahul/gaia that referenced this pull request May 29, 2026
…work (amd#1204) (amd#1210)

## Summary

Fix the permanently-red `Test Electron Framework` CI job caused by stale
test assertions after PR amd#606 reworked the empty-state UX.

## Why

Since amd#606 merged on 2026-05-08, every PR touching electron-related
paths fails CI — forcing reviewers to manually triage "is this red
mine?" for unrelated changes.

## Linked issue

Closes amd#1204

## Changes

### `test_electron_chat_app.js`
- Update CSS selector assertions: `.empty-chat` → `.empty-task`,
`.empty-chat-title` → `.empty-task-title`, `.empty-chat-chip` →
`.empty-task-chip` (PR amd#606 renamed these classes)
- Update TSX className assertion: `empty-chat-chip` → `empty-task-chip`

### `test_electron_chat_installer.js`
- Add allowlist for known-large dashboard files (`MemoryDashboard.tsx`,
~159KB since amd#606 added the full observability dashboard)
- Allowlisted files get a 200KB cap; all other components keep the 100KB
default
- This follows acceptance criteria option 2-C from the issue

## Test plan

- The assertions now match the actual CSS class names in `ChatView.css`
(`.empty-task`, `.empty-task-title`, `.empty-task-chip`) and the actual
className in `ChatView.tsx` (`empty-task-chip`)
- `MemoryDashboard.tsx` at 159KB passes the 200KB allowlist cap
- All other component files still enforced at 100KB
- No production code changed — test-only fix

---

🤖 **Disclosure:** This PR was authored by
[Kagura](https://github.com/kagura-agent), an AI agent. Open source
contribution is one of the things I do — you can see my work history
[here](https://github.com/kagura-agent/github-contribution). If you'd
prefer not to receive AI-authored PRs, just let me know and I'll stop —
no hard feelings.

---------

Co-authored-by: Tomasz Iniewicz <[email protected]>
@kovtcharov-amd kovtcharov-amd mentioned this pull request Jun 1, 2026
6 tasks
kovtcharov-amd pushed a commit that referenced this pull request Jun 3, 2026
Resolve conflicts in CLAUDE.md and docs/reference/cli.mdx by taking main's
versions: PRs #1337, #1340, #1364 already synced both files to the code more
recently, and the PR's stale side reintroduced removed commands (mcp
add/remove) and the broken /guides/summarize link that was failing the
internal cross-reference CI check.

Keep the PR's still-valid accuracy fixes (Gemma-4-E4B-it default in
chat/llm/emr docs, agent-ui Lemonade port 13305 + #install anchor, code
agent Python-gen still supported, telegram.mdx removal, #606 stale-note
removal).

Fix one contradiction the merge surfaced: main's CLAUDE.md and emr/cli.py
listed the EMR VLM as Qwen3-VL-4B, but agent.py and all --vlm-model
defaults are Gemma-4-E4B-it-GGUF. Aligned both with the code and emr.mdx.
pull Bot pushed a commit to bhardwajRahul/gaia that referenced this pull request Jun 3, 2026
…tStore (amd#1368)

## Why this matters

Clicking the **Memory Dashboard** button in the sidebar (or the Memory
icon in the chat-toolbar) had no effect when the **Settings** page was
currently open. The user had to close Settings first before Memory
Dashboard would respond. The reverse direction "worked" but left a stale
`showMemoryDashboard=true` in the store, so closing Settings via the
back arrow could re-surface the Memory Dashboard from a previous
session.

The Agent UI is not router-based. "Routing" between top-level views is a
stateful ternary in `App.tsx:531-535` that gives `showSettings`
unconditional priority over `showMemoryDashboard`, and the two store
setters never cleared each other's flag.

## What changed

Single-source-of-truth fix in
`src/gaia/apps/webui/src/stores/chatStore.ts:262-265`: the two setters
are now mutually exclusive — opening one view always closes the other.
This is better than patching all three call sites (sidebar × 2,
chat-toolbar × 1) because future call sites cannot reintroduce the bug.

```diff
-    setShowSettings: (show) => set({ showSettings: show }),
-    setShowMemoryDashboard: (show) => set({ showMemoryDashboard: show }),
+    setShowSettings: (show) =>
+        set(show ? { showSettings: true, showMemoryDashboard: false } : { showSettings: false }),
+    setShowMemoryDashboard: (show) =>
+        set(show ? { showMemoryDashboard: true, showSettings: false } : { showMemoryDashboard: false }),
```

`onHome` in `App.tsx:524` still works correctly — it sets both flags to
`false` explicitly, and the `false` path of each setter does not touch
the other flag.

## Test plan

- [ ] Manual: open Settings from the sidebar, then click the Memory
Dashboard button — Memory Dashboard renders (not Settings).
- [ ] Manual: open Memory Dashboard, then click Settings — Settings
renders (no change there, but verify flag state is clean).
- [ ] Manual: from each view, hit the back arrow — return to the chat /
welcome view, not to whatever was opened "behind" it.
- [ ] Add Playwright regression covering the "open Settings, click
Memory Dashboard, verify Memory Dashboard is the rendered view" scenario
and its symmetric counterpart.

## Notes

- Bug introduced by amd#606 (commit `74f637a4`) which added
`MemoryDashboard` and the `showMemoryDashboard` flag without making the
two flags mutually exclusive.
- Triage confirmed the fix location on the issue.

Fixes amd#1367
kovtcharov-amd pushed a commit that referenced this pull request Jun 4, 2026
Memory is a user-controlled setting, so the dynamic tool loader must inherit that
switch explicitly: when memory is off, the loader reverts to the legacy behavior
of exposing every registered tool — identical to a build without this feature.

Adds a dedicated section making this a documented off-state (not a degradation),
with a table collapsing the three "load everything" conditions — user-disabled
memory, loader toggle off, and embedder failure — onto the same safe floor so a
user can never lose tool access by toggling memory. Cross-referenced from the
Part 1 build steps and the #606 dependency note.
pull Bot pushed a commit to bhardwajRahul/gaia that referenced this pull request Jun 18, 2026
…1732)

## Why this matters

PR amd#606 shipped three of CoALA's four memory tiers (working / episodic /
semantic) and left the **procedural** tier — distilling recurring
successful tool sequences into reusable skills — as a follow-up (amd#887).
The issue's own design notes predate amd#691 locking and the current code,
so a contributor building straight from them would rebuild against a
stale picture — exactly how amd#688's spec drifted across four unmerged
PRs. This lands one canonical, **code-accurate** spec at
`docs/plans/skill-synthesis.mdx`, grounded line-by-line on the merged
memory layer and reconciled against the format it *emits* (amd#691) and the
tool-loader that *consumes* it (amd#1451), so design and implementation
can't diverge before the first line is written.

## Linked issue

Refs amd#887 <!-- design/spec only — the synthesis pipeline implementation
lands separately -->

## Changes

- **Separate `procedures` table + additive v2→v3 migration** —
procedural memory diverges from `knowledge` on every lifecycle axis
(empirical track-record vs belief-strength, multi-step body vs one
sentence, its own goal-trigger embedding index); sharing the table would
`NULL` every procedure-only column on existing rows and pollute the
recall index.
- **`recall_skill` is an internal method, not a sixth `@tool`** — the
planner calls it programmatically at turn start, so the agent gains the
capability without enlarging the five-tool memory registry it must
reason about.
- **Emits the locked amd#691 format by reference (the emit/inject split)**
— the LLM emits only the fields it *derives*; `Skill.parse()` injects
the fixed constants (`license`, `version`) and maps to
`metadata.gaia.tools_required`, so synthesized files validate against
the format instead of re-inventing it.
- **Fail-loud off-states + content-trust boundary** — Lemonade-down /
malformed-output / embedder-fail each skip-and-log or re-raise (no
silent smaller-model swap, per CLAUDE.md); injected bodies are
internal-corpus-only, capped at `MAX_RECALL_BODY_CHARS`, never
auto-exported.
- **Corrects the issue's drifted framing** — re-pins the synthesis hook
to the real `_run_memory_post_init`, config to the existing
`memory_settings.json`, and retracts the "new tool" /
intermediate-frontmatter-on-disk framing the pre-amd#691 pseudocode
implied.

## Test plan

- [x] `python -c "import json; json.load(open('docs/docs.json'))"` — nav
JSON valid
- [x] New page `plans/skill-synthesis` registered under **Ecosystem** in
`docs.json`
- [x] Mintlify preview renders `docs/plans/skill-synthesis.mdx` —
tables, `<Info>`/`<Note>`/`<Warning>`, the `<AccordionGroup>` examples,
and all internal anchor links resolve
- [ ] Maintainer sign-off on the design direction before implementation
begins (the five items under **Open questions** are explicitly left as
PR-time confirmations, not blockers)

Co-authored-by: Alexey Tyurin <>
alexey-tyurin added a commit to alexey-tyurin/gaia that referenced this pull request Jun 22, 2026
amd#1794)

## Summary

Today a GAIA agent re-plans every multi-step task from scratch — solve
the same
kind of goal ten times and it derives the tool sequence from zero all
ten times,
with no memory that it already found a working recipe. This PR adds the
**procedural** memory tier (the one amd#606 deliberately left out): when
the same
kind of goal succeeds ≥3 times with the same shape of work, GAIA distils
that
successful tool sequence into a reusable, `SKILL.md`-shaped procedure,
stores it,
and — the next time a similar goal appears — injects the proven recipe
into the
planner so it reuses it instead of re-planning. It is automatic (no
command, no
install) and **off by safe default**: with zero procedures or memory
disabled,
the planner prompt is byte-identical to today.

## Why

amd#606 shipped CoALA-style working/episodic/semantic memory over a SQLite
`MemoryStore` with a `tool_history` table recording every tool call +
outcome,
but left the *procedural* tier as a follow-up. amd#887 closes that loop so
repeated
work gets cheaper over time — the agent learns its own recipes from its
own
successful runs. This is a learning loop, **not** a marketplace:
synthesis only
*creates* procedures into a local `procedures` table from the agent's
own
history; browsing/installing/sharing (amd#647) and the on-disk loader/CLI
(amd#691)
are out of scope.

## Linked issue

Closes amd#887

## Changes

- **Synthesis pipeline (new `skill_synthesis.py`).** Detect successful
≥3-step
tool spans, cluster by embedded user-goal at cosine ≥ 0.82, and distil
clusters
that recur ≥3× at ≥80% success into a procedure — at most 10 clusters
per pass.
Reuses the existing nomic-768 embedder and the `chat.send_messages` seam
the
extraction loop already uses — **no new model or client**. Runs as one
extra
step inside amd#606's first-query maintenance pass, off the request hot
path.
- **`procedures` store + dedicated recall index.** Additive **v2→v3**
migration
adds a `procedures` table with its **own** FAISS index (the `knowledge`
index
is untouched); reconcile only **ADD / UPDATE / supersede — never
DELETE**.
`recall_skill(goal, top_k=2)` vector-searches that index and injects the
matched
body (capped at 1500 chars) into the planner's system prompt.
`recall_skill` is
an **internal method, not a 6th `@tool`** — the five-tool memory
registry is
  unchanged.
- **Off by safe default, fail-loud.** `GAIA_MEMORY_DISABLED=1` and
`"skill_synthesis": {"enabled": false}` both short-circuit synthesis
*and*
  recall; a disabled/superseded row is never recalled. Embedder failure
**re-raises with context**; Lemonade-down and malformed distill **skip +
log**,
  producing zero rows — no silent fallback.
- **Observability + docs.** `gaia memory status` prints a
synthesized-procedures
count; `docs/guides/memory.mdx` gains a "Procedural memory (skills)"
section
covering the thresholds, off-switches, and supersede-not-delete lineage.
- **Structural (code-review follow-up).** The procedural orchestration
(proc-FAISS index + recall + synthesis) is extracted from `memory.py`
into a
new `ProceduralMemoryMixin` (`procedural_memory.py`), so `memory.py`
doesn't
  grow ~450 lines. Behavior-preserving — the memory suite is the net.

## Deviations from the approved spec — please ratify

The ratified spec (`docs/plans/skill-synthesis.mdx`) was written against
an
assumed schema; these are where the code diverges to fit what's actually
on
`main`. Each is intentional:

- **Goal source.** `tool_history` has no goal column, so the cluster
goal is the
first `role='user'` turn of the session, derived via a `conversations`
JOIN
inside the new `iter_sessions(since, min_steps)` — **plain SQL, no LLM,
one row
  per session**. (Spec assumed an embeddable `user_goal` per row.)
- **`iter_sessions` is new.** No per-session successful-span iterator
existed;
added as a single read-only grouped query (avoids N per-session reads).
- **Distill client.** Reuses `self.chat.send_messages` at low
temperature rather
than introducing an abstract `llm` — same client the extraction pass
uses.
- **Recall-injection seam.** `_compose_system_prompt` is cached and
MemoryMixin
  didn't contribute to it, so recall is wired through the existing
`_get_mixin_prompts` auto-discovery
(`get_recalled_skills_system_prompt`) with a
per-turn `_refresh_recalled_skills` that recomposes only when the
recalled set
  changes.
- **Config home.** The 5 thresholds live on the existing
  `~/.gaia/memory_settings.json` (`skill_synthesis` section), not a new
  `config.toml` section (which doesn't exist).
- **Emit/inject split.** The LLM emits only `name` / `when_to_use` /
`tools_required` / body; `Skill.parse` injects the fixed `license: MIT`
/
`version: 1.0.0` (the maintainer's pseudocode put `version` in the LLM
output —
  moved to injection because it's fixed, not derived).
- **Deterministic clustering.** Goal clustering is order-independent so
a given
  history always yields the same procedures.
- **Store-layer extraction deferred.** Only the `memory.py` mixin was
extracted;
  splitting the `memory_store.py` procedural code (schema string + woven
  migration) is a higher-risk, separate follow-up.

**For maintainer:** the v2→v3 additive `procedures` migration needs
ratification.

## Test plan

All commands run from the repo root against the `.venv`. Results are
from this branch tip.

- [x] **Lint** — `.venv/bin/python util/lint.py --all`
→ **ALL QUALITY CHECKS PASSED — Ready for PR submission** (Black / isort
/ Pylint /
Flake8 / imports / dependabot / doc-versions all PASS; 3 non-blocking
warnings,
      both pre-existing and unrelated to this PR — see Proof).
- [x] **Full unit suite** —
`.venv/bin/python -m pytest tests/unit/ -q -p no:cacheprovider 2>&1 |
grep -E "FAILED|ERROR|passed|failed|error" | tail -30`
→ **5294 passed, 77 skipped**; 6 failures, all **pre-existing on
`main`** and outside
      this PR's diff (proof below). The 563 procedural-memory tests
(`test_skill_synthesis.py` + `test_memory_store.py` +
`test_memory_mixin.py`) all pass.
- [x] **Observability** — `.venv/bin/gaia memory status`
→ prints `Procedures (skills): 0 synthesized` (0 is the correct
cold-start floor on a DB
      with no qualifying history).
- [x] **Behavioral eval (LLM-affecting gate; serial — one `gaia eval
agent` at a time):**
      ```bash
      # Terminal 1
      .venv/bin/python -m gaia.ui.server --port 4200 --host 127.0.0.1
      # Terminal 2 — confirm nothing else is evaluating, then run
      ps aux | grep "gaia eval" | grep -v grep | wc -l   # must print 0
      .venv/bin/gaia eval agent --category rag_quality --agent-type doc
      ```
→ **7/8 passed (100% judged), avg 9.5/10** — no regression vs the
committed Gemma
baseline (9.47 vs 9.43). The one non-pass is an `INFRA_ERROR`, not a
regression (proof below).

## Verification & Proof

### Lint (`util/lint.py --all`)
| Code Formatting (Black) | PASS | | Critical Errors (Pylint) | PASS |
| Import Sorting (isort) | PASS | | Style Compliance (Flake8)| PASS |
| Import Validation | PASS | | Dependabot / Doc-Version | PASS |
Total: 10 checks · Passed 7 · Failed 0 · Warnings 3 (non-blocking)
[SUCCESS] ALL QUALITY CHECKS PASSED — Ready for PR submission


The 3 warnings are not from this PR: a Bandit **B608** (Low/Medium, ML
false positive) on a
fully-parameterized `IN (?,?,…)` query — the same accepted pattern as
the existing
`mark_consolidated` — and a pre-existing `DocumentQAAgent`
tool-isolation soft-warning in a
file this PR does not touch.

### Full unit suite — the 6 failures are pre-existing, not amd#887
6 failed, 5294 passed, 77 skipped, 10 warnings in 453.75s


None of the 6 touch this PR's diff, and all 6 **fail identically on
`main` (`894d9783`)** with
this branch's source removed — confirmed by checking out `main` and
re-running the same node IDs:

| Failing test | Why (unrelated to procedural memory) |
|---|---|
| `test_parse_error_recovery.py` ×2 | Agent context-overflow trim/retry
in `agent.py` — untouched here |
| `test_init_command.py` ×3 | Lemonade-installer platform paths
(linux/windows/CI) — macOS-local mock quirk |
| `test_memory_router.py::…faiss_missing` | Asserts 503 when faiss is
absent, but faiss **is** installed in the local `.venv` → 200 |

The procedural-memory suite itself is green:
`pytest tests/unit/test_skill_synthesis.py
tests/unit/test_memory_store.py tests/unit/test_memory_mixin.py -q` →
**563 passed**.

### `gaia memory status` — the new procedures surface renders
=== GAIA Agent Memory ===
Knowledge entries:   0
Conversations:       18 turns across 9 sessions
Tool calls:          0 (0 unique tools)
Procedures (skills): 0 synthesized          ← new line (amd#887)
Database size:       148.0 KB



### Behavioral eval — no RAG regression (`rag_quality`, doc agent)
RUN: eval-20260619-193456 — 7/8 passed (88% all, 100% judged) — avg
9.5/10
budget_query 10.0 · cross_section_rag 9.7 · hallucination_resistance 9.7
negation_handling 9.0 · safety_handbook_water 9.3 · simple_factual_rag
9.7
table_extraction 8.9 · csv_analysis INFRA_ERROR


| Metric | Baseline `gemma-4-e4b-d71cd914` | This run |
|---|---|---|
| Judged pass rate | 100% | **100%** |
| Failures | 0 | **0** |
| Avg score | 9.43 | **9.47** |

The lone `csv_analysis → INFRA_ERROR` is **environmental, not a
regression**: that scenario
pins `agent_type: data` (AnalystAgent), which was not registered in the
running UI server
(`chat, doc, file, builder` only — server logged `requested unknown
agent_type 'data'`), and it
isn't part of the 7-scenario rag_quality baseline. The procedural layer
initialized cleanly in
every session (`procedures FAISS index rebuilt: 0 vectors`, `registered
5 memory tools (v2)` —
recall stays internal), with no error from any new module.


## Checklist

- [x] I have linked a GitHub issue above (`Closes amd#887`).
- [x] I have described **why** this change is being made, not just what
changed.
- [x] I have run linting and tests locally (`python util/lint.py --all`,
`pytest tests/unit/`).
- [x] I have updated documentation if user-visible behavior changed
(`docs/guides/memory.mdx`, `gaia memory status`).

---------

Co-authored-by: Alexey Tyurin <>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

agents chat Chat SDK changes cli CLI changes consumer Blocks consumer adoption — must ship for the v0.20.0 consumer launch window dependencies Dependency updates devops DevOps/infrastructure changes documentation Documentation changes electron Electron app changes eval Evaluation framework changes mcp MCP integration changes performance Performance-critical changes security Security-sensitive changes stop-the-line PR is foundational; do not merge changes to its frozen paths until it lands tests Test changes

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants