Purpose of the change

Motivation:

• current DeclarativeMemory does not handle multimodal content
• current DeclarativeMemory does not handle large messages or text items well (no chunking).
• current DeclarativeMemory produces high fan out in number of Neo4j queries
• current DeclarativeMemory operations do not tolerate failures
• VectorGraphStore is difficult to implement using other databases
• we have a new VectorStore interface defined previously to move toward a solution
• tangential: current DeclarativeMemory may face problems with top-k redundancy from vector search
• current DeclarativeMemory has no way of distinguishing name from identity
  • top-level API from server does not provide enough information as there is no concept of temporal context
• add encryption support

Description

Based on #1375.
#1205 based on this.

Changes:

• define new extensible data models to support multimodal content and chunking
• define new APIs to allow for more efficient and atomic operations
• API: limit is now vector limit for transparency and to avoid throwing away computations -- it is trivial for client to limit/threshold and transform into usable context and this allows much more flexible iterative expansion without requerying
• uses new API to avoid breaking existing memory for now -- although I would prefer an overhaul of the server due to its many problems [Feat]: Server tech debt resolution wishlist #1297
• introduce derivative eviction system for duplicate-heavy online-ingested data (cannot pre-dedup, need to maintain index quality)

each segment comes from exactly one episode
each derivative comes from exactly one segment

alternative considered: many-to-many segments-to-derivatives

• slows growth of vector index
• handles HNSW index degradation due to ((potentially) near) duplicates
• rejected for complexity and inability to filter in vector store
• consolidation is about as effective as increasing the search limit proportionally
• requires reference counting, ownership transfer, or similar

Approach to deletion:

• purging state necessary as a lock to allow deleting from external DB (vector DB)

Scoping:

• by partition key: should allow sharding and horizontal scaling

• this memory will be parallel to DeclarativeMemory

Decisions to make:

• naming

POC:

• Current Neo4j implementation on a single MacBook Pro (M3 Pro, 18GB) cannot handle even 30 concurrent search queries with search limit 100 (latency explodes).
• Qdrant + PostgreSQL can handle 50+ concurrent search queries with acceptable latency.

Type of change

Will eventually be breaking or require new API.

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• Refactor (does not change functionality, e.g., code style improvements, linting)
• Documentation update
• Project Maintenance (updates to build scripts, CI, etc., that do not affect the main project)
• Security (improves security without changing functionality)

How Has This Been Tested?

APIs and data models not tested.

• Unit Test
• Integration Test
• End-to-end Test
• Test Script (please provide)
• Manual verification (list step-by-step instructions)

Checklist

• I have signed the commit(s) within this pull request
• My code follows the style guidelines of this project (See STYLE_GUIDE.md)
• I have performed a self-review of my own code
• I have commented my code
• I have made corresponding changes to the documentation
• My changes generate no new warnings
• I have added unit tests that prove my fix is effective or that my feature works
• New and existing unit tests pass locally with my changes
• Any dependent changes have been merged and published in downstream modules
• I have checked my code and corrected any misspellings

Maintainer Checklist

• Confirmed all checks passed
• Contributor has signed the commit(s)
• Reviewed the code
• Run, Tested, and Verified the change(s) work as expected