Codestin Search App

edwinyyyu · 2026-03-10T00:55:00Z

Purpose of the change

Motivation:

current DeclarativeMemory does not handle multimodal content
current DeclarativeMemory does not handle large messages or text items well (no chunking).
current DeclarativeMemory produces high fan out in number of Neo4j queries
current DeclarativeMemory operations do not tolerate failures
VectorGraphStore is difficult to implement using other databases
we have a new VectorStore interface defined previously to move toward a solution
tangential: current DeclarativeMemory may face problems with top-k redundancy from vector search
current DeclarativeMemory has no way of distinguishing name from identity
- top-level API from server does not provide enough information as there is no concept of temporal context
add encryption support (initial infrastructure)

Description

For wiring into server, #1304 is required.

Choices:

JSON/JSONB allows faster upserts. Filters are already scoped to queries so queries are fast. GIN indexes are not so useful right now. Another alternative is EAV properties table which requires more complexity but can be better for range filters.

Changes:

define new extensible data models to support multimodal content and chunking
define new APIs to allow for more efficient and atomic operations
API: limit is now vector limit for transparency and to avoid throwing away computations -- it is trivial for client to limit/threshold and transform into usable context and this allows much more flexible iterative expansion without requerying
uses new API to avoid breaking existing memory for now -- although I would prefer an overhaul of the server due to its many problems [Feat]: Server tech debt resolution wishlist #1297
~~- introduce derivative eviction system for duplicate-heavy online-ingested data (cannot pre-dedup, need to maintain index quality)~~

each segment comes from exactly one episode
each derivative comes from exactly one segment

alternative considered: many-to-many segments-to-derivatives

slows growth of vector index
handles HNSW index degradation due to ((potentially) near) duplicates
rejected for complexity and inability to filter in vector store
consolidation is about as effective as increasing the search limit proportionally
requires reference counting, ownership transfer, or similar

Approach to deletion:

purging state necessary as a lock to allow deleting from external DB (vector DB)

Scoping:

by partition key: should allow sharding and horizontal scaling
this memory will be parallel to DeclarativeMemory

Decisions to make:

naming

POC:

Current Neo4j implementation on a single MacBook Pro (M3 Pro, 18GB) cannot handle even 30 concurrent search queries with search limit 100 (latency explodes).
Qdrant + PostgreSQL can handle 50+ concurrent search queries with acceptable latency.

Type of change

[Please delete options that are not relevant.]

Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Breaking change (fix or feature that would cause existing functionality to not work as expected)
Refactor (does not change functionality, e.g., code style improvements, linting)
Documentation update
Project Maintenance (updates to build scripts, CI, etc., that do not affect the main project)
Security (improves security without changing functionality)

How Has This Been Tested?

Checklist

Maintainer Checklist

Confirmed all checks passed
Contributor has signed the commit(s)
Reviewed the code
Run, Tested, and Verified the change(s) work as expected

edwinyyyu · 2026-03-18T19:15:53Z

Still needs some performance fixes.

edwinyyyu · 2026-03-19T00:19:40Z

Maybe lazy ownership transfer is more performant.

Signed-off-by: Edwin Yu <[email protected]>

…iness Signed-off-by: Edwin Yu <[email protected]>

Signed-off-by: Edwin Yu <[email protected]>

edwinyyyu · 2026-05-04T18:35:29Z

Wiring for SQLAlchemySegmentStore will be added in #1304 if this is merged.

malatewang · 2026-05-04T17:34:16Z

+class MessageContext(BaseModel):
+    """The content is communicated by a source."""
+
+    type: Literal["message"] = "message"


It is clearer to use messagecontext

Updated to be ProducerContext and context_type: "producer". Removed CitationContext (no use case yet).

Having "context" in the value for the discriminator is redundant and more error-prone than "context" in the discriminator key.

malatewang · 2026-05-04T18:33:30Z

+    segment_uuid: UUID
+    timestamp: datetime
+    context: Context = Field(default_factory=NullContext)
+    text: str


It is better to use block instead of plain str here

The original intent was for Derivatives to be text-only since multimodal embedding models are uncommon. The solution for supporting multimodal embeddings is probably to make Derivatives text if the embedding reports no support for multimodality.

malatewang · 2026-05-04T23:17:58Z

+                f"{', '.join(sorted(reserved_fields))}"
+            )
+
+        missing_fields = (


This has been checked in the constructor

This checks that input events for ingestion are not missing fields, which cannot be done at construction time.

Unless you mean it is checked at Event construction time? In that case I think it's safer to check close to where the condition is required.

I do not think the code is working that way. It does not check the property of the incoming events.

Sorry, I think that (or checking that the schema includes all fields required by events) was the behavior at some point but it changed and it was not deduplicated. I will remove.

malatewang · 2026-05-04T23:19:03Z

+
+        events = sorted(
+            events,
+            key=lambda event: (event.timestamp, event.uuid),


Since the UUID is random, there is no sense to sort by uuid here. Or do we require special UUID generator?

The implied behavior is that UUID can be used to order events that have the same timestamp, so behavior with UUIDv7 vs. UUIDv4 works differently.

This is not documented, so I can remove it.

I think the original motivation for sorting at all was to make temporal derivative eviction easier -- simulating the order of occurrence.

malatewang · 2026-05-05T00:48:08Z

+    """Event memory system."""
+
+    # System-defined metadata field names. Reserved.
+    _SEGMENT_UUID_FIELD_NAME = "_segment_uuid"


Are they enough? Event source should be indexed

Event source (along with all Context fields) is LLM-friendly private data that should not be indexed in a multi-tenant system where we may have encryption. If the user/consumer feels that they need to filter on this and it is not sensitive information, they may add it to the properties schema.

It is safer to completely prohibit indexing anything in Context because Context can reveal things like the user's name, interests (if adding content from something like a book), sensitive information like location and time they were there (which may reveal where they live), etc in plaintext (which may be required for most/all filters to work).

If it needs to be system-defined, the application may add _producer_id as a non-LLM-friendly system-reserved field in properties (EventMemory is not responsible for reserved prefix validation, and it allows for creating upper-level system-reserved fields.). In this proposed case, producer_id is a property that exposes no sensitive information (it is just a user or agent id like user_123 or a UUID or something).

Another challenge that we avoid by not indexing this is that semantically different fields (potentially with different types) with the same name do not collide. Because Context is a discriminated union, enforcing this would be complicated and each Context implementation would have to consider what other implementations have already reserved.

Originally I agreed with this idea. But the above considerations made me reconsider. See 41f2fc1, 7582233 for changes. Also the code was a lot more complicated when supporting filtering by context.

Signed-off-by: Edwin Yu <[email protected]>

malatewang · 2026-05-06T17:51:33Z

+                f"{', '.join(sorted(reserved_fields))}"
+            )
+
+        missing_fields = (


I do not think the code is working that way. It does not check the property of the incoming events.

malatewang · 2026-05-06T18:21:33Z

+            t_embed - t_derive,
+            t_seg_store - t_embed,
+            t_v_store - t_seg_store,
+            t_v_store - t_start,


Add the stats to metrics

Done. Renamed/reorganized.

Signed-off-by: Edwin Yu <[email protected]>

malatewang · 2026-05-06T20:53:11Z

+                label_names=("phase",),
+            )
+
+        self._text_splitter = RecursiveCharacterTextSplitter(


In our discussion, I think we agreed to move this out of the event memory

Signed-off-by: Edwin Yu <[email protected]>

edwinyyyu · 2026-05-07T00:35:02Z

Before merge, I would like to do another manual verification run on LoCoMo/LongMemEval, since there have been quite a few changes since last runs.

edwinyyyu · 2026-05-07T19:50:44Z

LoCoMo score is good enough. Used SQLite for both segment store and vector store (sqlite-vec). Happy path works.

edwinyyyu · 2026-05-07T19:51:51Z

No breaking changes since 0.3.7 so I'm changing the milestone to 0.3.8. Wiring determines if 0.4.0 is needed.

Signed-off-by: Edwin Yu <[email protected]>

edwinyyyu force-pushed the sqlalchemy_segment_linker branch 5 times, most recently from a724574 to ced2263 Compare March 10, 2026 19:00

edwinyyyu mentioned this pull request Mar 10, 2026

Sqlalchemy snapshot store #1015

Closed

26 tasks

edwinyyyu force-pushed the sqlalchemy_segment_linker branch 6 times, most recently from 5498137 to e540bde Compare March 13, 2026 00:05

edwinyyyu requested review from jealous, malatewang and o-love March 13, 2026 17:10

edwinyyyu force-pushed the sqlalchemy_segment_linker branch from e540bde to a6954a9 Compare March 13, 2026 17:35

edwinyyyu mentioned this pull request Mar 13, 2026

Improve vector store API #1201

Merged

edwinyyyu force-pushed the sqlalchemy_segment_linker branch 10 times, most recently from 5d451f6 to 29d090d Compare March 18, 2026 03:30

edwinyyyu force-pushed the sqlalchemy_segment_linker branch from 29d090d to e3961e2 Compare March 20, 2026 00:49

edwinyyyu added 7 commits May 4, 2026 09:59

Delete extraneous file

9c2c9dc

Signed-off-by: Edwin Yu <[email protected]>

Specify Context field types

b4c5ddc

Signed-off-by: Edwin Yu <[email protected]>

Context fields may expose sensitive information if used in filtering

7582233

Signed-off-by: Edwin Yu <[email protected]>

Support NullContext and use LargeBinary for future encryption-friendl…

4dee680

…iness Signed-off-by: Edwin Yu <[email protected]>

Support encryption

54ead9c

Signed-off-by: Edwin Yu <[email protected]>

Refactor codec loading

8208d63

Signed-off-by: Edwin Yu <[email protected]>

Remove encryption

ba130a6

Signed-off-by: Edwin Yu <[email protected]>

malatewang requested changes May 5, 2026

View reviewed changes

edwinyyyu and others added 6 commits May 5, 2026 12:52

Reorganize data models

311c28e

Signed-off-by: Edwin Yu <[email protected]>

Generalize Derivative content type

d2eafb9

Signed-off-by: Edwin Yu <[email protected]>

Pluggable Deriver

b5e1604

Signed-off-by: Edwin Yu <[email protected]>

Define Block clearly

44f8cb2

Signed-off-by: Edwin Yu <[email protected]>

Add metrics

bfd2229

Signed-off-by: Edwin Yu <[email protected]>

Merge branch 'main' into sqlalchemy_segment_linker

91423d9

edwinyyyu mentioned this pull request May 6, 2026

Payload codecs #1375

Closed

malatewang requested changes May 6, 2026

View reviewed changes

edwinyyyu added 2 commits May 6, 2026 12:38

Remove redundant check

01c2ebb

Signed-off-by: Edwin Yu <[email protected]>

Add phase timing to metrics

f82b044

Signed-off-by: Edwin Yu <[email protected]>

malatewang reviewed May 6, 2026

View reviewed changes

edwinyyyu added 3 commits May 6, 2026 16:12

Pluggable Segmenter

acdfeab

Signed-off-by: Edwin Yu <[email protected]>

Unify redundant methods

9232a34

Signed-off-by: Edwin Yu <[email protected]>

Inline method

db778df

Signed-off-by: Edwin Yu <[email protected]>

malatewang approved these changes May 7, 2026

View reviewed changes

Merge branch 'main' into sqlalchemy_segment_linker

5a1ab4b

edwinyyyu mentioned this pull request May 7, 2026

Wire event memory #1304

Closed

26 tasks

Remove unused metadata field

6bac119

Signed-off-by: Edwin Yu <[email protected]>

Conversation

edwinyyyu commented Mar 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Purpose of the change

Description

Type of change

How Has This Been Tested?

Checklist

Maintainer Checklist

Uh oh!

edwinyyyu commented Mar 18, 2026

Uh oh!

edwinyyyu commented Mar 19, 2026

Uh oh!

edwinyyyu commented May 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

edwinyyyu May 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

edwinyyyu May 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

edwinyyyu May 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

edwinyyyu commented May 7, 2026

Uh oh!

edwinyyyu commented May 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

edwinyyyu commented May 7, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

edwinyyyu commented Mar 10, 2026 •

edited

Loading

edwinyyyu commented May 4, 2026 •

edited

Loading

edwinyyyu May 5, 2026 •

edited

Loading

edwinyyyu May 6, 2026 •

edited

Loading

edwinyyyu May 5, 2026 •

edited

Loading

edwinyyyu commented May 7, 2026 •

edited

Loading