Codestin Search App

reffdev · 2026-04-14T23:08:04Z

Purpose of the change

Adds Apache AGE (openCypher over PostgreSQL, Apache-2.0) as a third supported graph backend behind VectorGraphStore, alongside Neo4j and NebulaGraph. Enables a single-Postgres deployment where episodic memory (graph) and semantic memory (pgvector) share one database service, and provides an Apache-2.0-compatible alternative to Neo4j that's operationally lighter than NebulaGraph's 3-role distributed topology.

Opened in response to #1324.

Description

MemMachine already requires Postgres + pgvector for semantic memory. AGE ships the age and vector extensions in the same Postgres, so with this PR a minimal AGE deployment collapses what was previously two database services (Neo4j + pgvector Postgres) into one. NebulaGraph already covers Apache-2.0 licensing for users at scale, but bundling it means running a 3-role distributed system; AGE fits single-instance and small-cluster deployments that want to drop Neo4j without adopting a new product.

What's added:

AgeVectorGraphStore + age_utils: full VectorGraphStore implementation against AGE + pgvector. Graph structure lives in AGE; per-embedding similarity goes to pgvector side tables keyed by vertex uid, tracked via a per-graph registry table. Supports the full pgvector 1..16000 dimension range. Asserts AGE >= 1.6.0 on first use so older versions fail fast instead of hitting the cryptic "SET clause expects a map" runtime error.
AgeConf + DatabaseManager plumbing symmetric with Neo4jConf. /resources status endpoint surfaces AGE backends (and, while touching that code, the pre-existing nebula_graph gap).
deployments/docker/postgres-age/Dockerfile layering postgresql-16-pgvector on apache/age:release_PG16_1.6.0. Published as memmachine/postgres-age:pg16-1.6.0 via a new build job in .github/workflows/docker-image.yml.
docker-compose.age.yml: self-contained single-Postgres AGE stack (one command). Base docker-compose.yml keeps postgres-age under an opt-in age profile for the "run AGE alongside a pip-installed server" flow.
Helm chart (bumped 0.1.0 -> 0.2.0): new top-level episodicBackend: neo4j | age toggle, orthogonal to per-backend *.enabled flags. In AGE mode the standalone pgvector postgres Deployment is automatically skipped and all relational workloads ride on the AGE-enabled Postgres. Validation guard fails cleanly on invalid values.
Installation wizard + installer: memmachine-configure prompts for backend choice and routes to the existing Neo4j auto-install or the wizard's interactive AGE prompts. Wizard emits a paired AgeConf + co-located SqlAlchemyConf so semantic memory's pgvector store shares the AGE Postgres.
Also fixes a pre-existing bug where the wizard's generated semantic_memory.database pointed at NEO4J_DB_ID (a graph backend, not a relational one).
Sample config sample_configs/episodic_memory_config.age.sample.
Tests: unit coverage for age_utils, AgeConf lifecycle tests in test_database_manager, wizard tests for AGE and invalid-backend paths, and a testcontainer-backed integration test built from the shipped Dockerfile.

Dependencies: none added - asyncpg, pgvector, sqlalchemy, and alembic are already pinned in packages/server/pyproject.toml.

Fixes/Closes

Fixes #(issue number)

Type of change

New feature (non-breaking change which adds functionality)

How Has This Been Tested?

Unit Test
Integration Test
Manual verification

Unit tests (no Docker required):

pytest packages/server/server_tests/memmachine_server/common/test_age_utils.py \
       packages/server/server_tests/memmachine_server/common/resource_manager/test_database_manager.py \
       packages/server/server_tests/memmachine_server/installation/

Integration test (requires Docker; built from the shipped deployments/docker/postgres-age/Dockerfile so the test environment matches prod):

pytest -m integration \
  packages/server/server_tests/memmachine_server/common/vector_graph_store/test_age_vector_graph_store.py

Helm template verified on all four deployment scenarios plus invalid-input guard:

cd deployments/helm
helm template memmachine-test .                                                   # default: in-cluster Neo4j
helm template memmachine-test . --set neo4j.enabled=false                         # external Neo4j
helm template memmachine-test . \
    --set episodicBackend=age --set postgresAge.enabled=true --set neo4j.enabled=false   # in-cluster AGE single-stack
helm template memmachine-test . \
    --set episodicBackend=age --set postgresAge.enabled=false \
    --set postgresAge.host=my-age.example.com --set neo4j.enabled=false           # external AGE
helm template memmachine-test . --set episodicBackend=foobar                      # must fail

Docker Compose verified:

docker compose -f docker-compose.yml config --quiet     # default Neo4j stack
docker compose -f docker-compose.age.yml config --quiet # single-Postgres AGE stack

Test Results:

Helm scenario 3 (AGE single-stack): exactly one Postgres Deployment (postgres-age), db_postgres and db_age both target it, one wait-for-postgres init container, POSTGRES_PASSWORD sourced from postgres-age-secret. No postgres-deployment, no neo4j-deployment.
Helm scenario 5: aborts with Error: ... episodicBackend must be one of: neo4j, age (got "foobar") - expected.
All Python files pass ast.parse; all YAML files pass docker compose config --quiet.

Checklist

I have signed the commit(s) within this pull request
My code follows the style guidelines of this project
I have performed a self-review of my own code
I have commented my code
I have made corresponding changes to the documentation
My changes generate no new warnings
I have added unit tests that prove my fix is effective or that my feature works
New and existing unit tests pass locally with my changes
I have checked my code and corrected any misspellings

Maintainer Checklist

Confirmed all checks passed
Contributor has signed the commit(s)
Reviewed the code
Run, Tested, and Verified the change(s) work as expected

Screenshots/Gifs

N/A - backend + deployment change

Further comments

Helm chart version bumped 0.1.0 -> 0.2.0 because templates and values changed materially.
Behavior change in wizard-generated configs (Neo4j mode): semantic_memory.database was previously set to NEO4J_DB_ID - a pre-existing bug since SemanticMemoryConf.database expects a relational backend. It's now left unset in Neo4j mode so SemanticMemoryConf's existing _auto_disable_when_incomplete validator fires cleanly; in AGE mode it targets the companion SqlAlchemyConf. Users who relied on the broken behavior will see semantic memory cleanly disabled instead of failing obscurely at runtime.
Helm AGE mode collapses two Postgres services into one: when episodicBackend=age, postgres.enabled is ignored because the postgres-age image already ships vector. This is intentional and documented in values.yaml / deployments/helm/README.md.
memmachine/postgres-age:pg16-1.6.0 doesn't exist in the registry yet - the new CI job publishes it on the first release cut with this PR merged. Until then, helm deployments will need to build and push locally (documented in values.yaml).
AGE 1.6.0-specific workaround: SET n += $param inside cypher() fails with "SET clause expects a map"; aliased via WITH $param AS p before SET. The version probe enforces AGE >= 1.6.0 with a clear error.
postgres-age image build is pinned to linux/amd64: apache/age has had inconsistent arm64 availability historically. Easy flip to multi-arch once confirmed upstream.

Happy to split any subsystem out into a follow-up if reviewers prefer a tighter scope (e.g. the NebulaGraph /resources status addition, or the wizard's semantic_memory.database fix), but they're small enough I've kept them bundled.

Add Apache AGE (openCypher over PostgreSQL) as a third VectorGraphStore implementation alongside Neo4j and NebulaGraph. MemMachine already requires Postgres + pgvector for semantic memory; AGE ships ``age`` and ``vector`` in one Postgres instance, letting episodic + semantic memory share a single database service. NebulaGraph already covers Apache-2.0 licensing for users at scale, but bundling it means running a 3-role distributed system; AGE is the natural fit for single-instance and small-cluster deployments that want to drop Neo4j without adopting a new product. Backend implementation: - AgeVectorGraphStore + age_utils: full VectorGraphStore against AGE + pgvector. Graph structure lives in AGE; per-embedding similarity goes to pgvector side tables keyed by vertex uid, tracked via a per-graph registry table. Supports pgvector's full 1..16000 dimension range rather than inheriting Neo4j's tighter cap. Asserts AGE >= 1.6.0 on first use so older versions fail with a clear message instead of "SET clause expects a map". - AgeConf + DatabaseManager plumbing mirroring Neo4jConf. The connect- event hook is extracted to a named ``_register_age_connect_hook`` so tests patch a stable seam instead of SQLAlchemy internals. - /resources status endpoint surfaces age (and, while touching that code, the pre-existing nebula_graph gap). Deployment (Docker Compose): - deployments/docker/postgres-age/Dockerfile layers postgresql-16- pgvector on apache/age:release_PG16_1.6.0. Published as memmachine/postgres-age:pg16-1.6.0 via a new build-postgres-age job in .github/workflows/docker-image.yml. - docker-compose.age.yml: self-contained single-Postgres AGE stack (``docker compose -f docker-compose.age.yml up``). The base docker-compose.yml keeps postgres-age under an opt-in ``age`` profile for the "run AGE alongside a pip-installed server" flow. Deployment (Helm, chart bumped 0.1.0 -> 0.2.0): - New top-level ``episodicBackend: neo4j | age`` toggle, orthogonal to the per-backend ``*.enabled`` flags (which keep their meaning of "deploy in-cluster vs use external host"). Validation guard fails cleanly on an invalid value. - In AGE mode, ``postgres.enabled`` is ignored: the standalone pgvector Deployment/Service/PVC are skipped, and db_postgres + db_age both target the AGE-enabled Postgres (which ships vector too). Single database service for session_manager, episode_store, semantic memory, config_database, and the graph store. The ``wait-for-postgres`` init container and ``POSTGRES_PASSWORD`` env-var source follow the active backend. - postgres-age-secret renames its key from POSTGRES_AGE_PASSWORD to POSTGRES_PASSWORD so the MemMachine Deployment sources the same env var name from either postgres-secret or postgres-age-secret. - postgresAge.* values expose SQLAlchemy pool knobs (pool_size, max_overflow, pool_timeout, pool_recycle, pool_pre_ping) for parity with postgres.* in Neo4j mode. Installation: - memmachine-configure prompts for graph backend choice and routes accordingly: Neo4j mode keeps the existing auto-install flow; AGE mode falls through to the wizard's interactive connection prompts (users stand up their own AGE-enabled Postgres via docker-compose.age.yml, helm, or existing infra). - Configuration wizard emits a paired AgeConf + SqlAlchemyConf at the same Postgres so semantic memory's pgvector store rides on the AGE database. AGE connection defaults consolidated into an ``AgeDefaults`` dataclass. - Fixes a pre-existing bug where semantic_memory.database in generated configs pointed at NEO4J_DB_ID (a graph backend, not a relational one). Now left unset in Neo4j mode so SemanticMemoryConf's existing auto-disable validator fires cleanly, and targets the co-located Postgres in AGE mode. Tests: - Unit: test_age_utils (agtype round-trip, cypher wrapping, sanitizer, parameter encoding); AgeConf lifecycle tests in test_database_manager mirror the Neo4jConf ones; wizard tests cover AGE interactive mode and invalid-backend rejection. - Integration: test_age_vector_graph_store exercises the store end-to- end against a testcontainer built from the shipped postgres-age Dockerfile, so the test environment matches prod. Docs: README, USAGE, DOCKER_COMPOSE_README, and the helm README updated with AGE sections; sample_configs/episodic_memory_config.age.sample ships a complete single-Postgres config example. Notes: - AGE 1.6.0 rejects ``SET n += $param`` inside cypher() ("SET clause expects a map") even when the parameter resolves to a map; aliasing via ``WITH $param AS p`` before SET sidesteps it. - Side-table names are SHA1-hashed because Postgres' 63-byte identifier limit cannot fit the full (graph, kind, collection, embedding_name) tuple; the registry table preserves the reverse mapping for cleanup and hydration. - The docker-image.yml postgres-age build targets linux/amd64 only; apache/age has had inconsistent arm64 availability historically. Flip to multi-arch once confirmed upstream.

Copilot

Pull request overview

Adds Apache AGE (openCypher over PostgreSQL) as an additional VectorGraphStore backend, enabling a “single-Postgres” deployment where episodic (graph) and semantic (pgvector) storage can share one database service, plus corresponding installer and deployment support.

Changes:

Introduce AgeVectorGraphStore and age_utils, plus AgeConf/DatabaseManager plumbing to configure and run AGE as a graph backend.
Add deployment artifacts for AGE (Dockerfile + Compose + Helm chart toggles/templates) and CI publishing for a postgres-age image.
Update installer/wizard flows, sample configs, docs, and tests (unit + integration) to cover AGE and the “single-Postgres” path.

Reviewed changes

Copilot reviewed 32 out of 32 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
sample_configs/episodic_memory_config.age.sample	New sample configuration for an AGE-backed stack.
packages/server/src/memmachine_server/server/api_v2/config_service.py	Surface AGE (and NebulaGraph) DB configs in `/resources` status output.
packages/server/src/memmachine_server/installation/utilities.py	Add default prompt values for AGE connection details.
packages/server/src/memmachine_server/installation/memmachine_configure.py	Add backend selection prompt and route installer flow based on backend.
packages/server/src/memmachine_server/installation/configuration_wizard.py	Add AGE wizard mode and emit paired `AgeConf` + `SqlAlchemyConf` for single-Postgres deployments.
packages/server/src/memmachine_server/common/vector_graph_store/age_vector_graph_store.py	New AGE-backed `VectorGraphStore` implementation using AGE + pgvector side tables.
packages/server/src/memmachine_server/common/resource_manager/database_manager.py	Add AGE engine lifecycle, validation, and graph store construction.
packages/server/src/memmachine_server/common/configuration/database_conf.py	Add `AgeConf`, `SupportedDB.AGE`, and `DatabasesConf.age_confs` parsing/serialization.
packages/server/src/memmachine_server/common/age_utils.py	New helpers for agtype parsing, Cypher wrapping, identifier sanitization, and per-connection AGE setup.
packages/server/server_tests/memmachine_server/installation/test_memmachine_configure.py	Update installer tests for the new backend selection prompt.
packages/server/server_tests/memmachine_server/installation/test_configuration_wizard.py	Add wizard tests for AGE mode and invalid backend values.
packages/server/server_tests/memmachine_server/common/vector_graph_store/test_age_vector_graph_store.py	New Docker-backed integration tests for `AgeVectorGraphStore`.
packages/server/server_tests/memmachine_server/common/test_age_utils.py	New unit tests for `age_utils` pure helpers.
packages/server/server_tests/memmachine_server/common/resource_manager/test_database_manager.py	Add unit tests for AGE engine/store construction and pool kwarg forwarding.
docker-compose.yml	Add opt-in `postgres-age` service under an `age` profile.
docker-compose.age.yml	New self-contained Compose stack running MemMachine against AGE.
deployments/helm/values.yaml	Add `episodicBackend` toggle and `postgresAge` values to support AGE deployments.
deployments/helm/templates/secrets.yaml	Add `postgres-age-secret` for AGE Postgres password wiring.
deployments/helm/templates/pvc.yaml	Skip standalone Postgres PVC in AGE mode; add PVC for postgres-age when enabled.
deployments/helm/templates/postgres-service.yaml	Skip standalone Postgres service in AGE mode.
deployments/helm/templates/postgres-deployment.yaml	Skip standalone Postgres deployment in AGE mode; document behavior.
deployments/helm/templates/postgres-age-service.yaml	New service for in-cluster postgres-age.
deployments/helm/templates/postgres-age-deployment.yaml	New deployment for in-cluster postgres-age.
deployments/helm/templates/memmachine-deployment.yaml	Adjust init containers and secrets/env wiring for AGE vs Neo4j mode.
deployments/helm/templates/memmachine-configmaps.yaml	Validate `episodicBackend` and emit AGE vs Neo4j database resources accordingly.
deployments/helm/README.md	Document backend selection semantics and AGE single-Postgres mode.
deployments/helm/Chart.yaml	Bump chart version and update description.
deployments/docker/postgres-age/Dockerfile	New image layering pgvector onto the apache/age base image.
USAGE.md	Update docs to reflect multiple episodic graph backends.
README.md	Update storage backend description to include AGE.
DOCKER_COMPOSE_README.md	Document AGE compose options and configuration shape.
.github/workflows/docker-image.yml	Add CI job to build/push the postgres-age companion image.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2026-04-15T20:07:04Z

+    async def _label_exists(self, sanitized_label: str) -> bool:
+        """Return True if the AGE label has a backing table in the graph schema.
+
+        Reads use this to bail out early with empty results when a label has
+        never been written to — recent AGE releases raise "label does not
+        exist" on ``MATCH (n:Label)`` for unknown labels, so we cannot rely on
+        Cypher returning zero rows.
+        """
+        await self._ensure_graph_initialized()
+        async with self._engine.connect() as conn:
+            result = await conn.exec_driver_sql(
+                "SELECT 1 FROM pg_tables WHERE schemaname = $1 AND tablename = $2",
+                (self._graph_name, sanitized_label),
+            )
+            return result.first() is not None
+
+    async def _ensure_label(self, sanitized_label: str, *, kind: str) -> None:
+        """Create an AGE vertex or edge label if it does not already exist.
+
+        AGE creates each label as a table in the graph's schema, so we detect
+        existence via ``pg_tables`` rather than relying on ``ag_catalog``
+        internals (which vary across AGE versions). The actual creation uses
+        AGE's built-in ``create_vlabel`` / ``create_elabel`` helpers.
+        """
+        entity_type = EntityType.NODE if kind == "v" else EntityType.EDGE
+        key = (entity_type, sanitized_label)
+        if key in self._ensured_labels:
+            return
+        async with self._ensured_labels_lock:
+            if key in self._ensured_labels:
+                return
+            await self._ensure_graph_initialized()
+            async with self._engine.begin() as conn:
+                exists = (
+                    await conn.exec_driver_sql(
+                        "SELECT 1 FROM pg_tables "
+                        "WHERE schemaname = $1 AND tablename = $2",
+                        (self._graph_name, sanitized_label),
+                    )
+                ).first()
+                if exists is None:
+                    creator = "create_vlabel" if kind == "v" else "create_elabel"
+                    await conn.exec_driver_sql(
+                        f"SELECT {creator}($1, $2)",
+                        (self._graph_name, sanitized_label),


AGE label existence is detected via pg_tables(tablename = sanitized_label). PostgreSQL identifiers are limited to 63 bytes; if sanitize_identifier(collection/relation) produces a label longer than that, the underlying table name will be truncated by Postgres/AGE, causing _label_exists() to return false and ensure_label() to repeatedly try create_vlabel/create_elabel (likely failing). Consider validating/truncating/hashing sanitized_label to <=63 bytes (and using the same value consistently for create*label and pg_tables lookups).

Copilot · 2026-04-15T20:07:04Z

+    @staticmethod
+    def _ask_graph_backend() -> str:
+        """Prompt the user for the episodic-memory graph backend."""
+        choice = (
+            input(
+                "Which graph backend would you like to use for episodic "
+                "memory? (neo4j/age) [neo4j]: "
+            )
+            .strip()
+            .lower()
+        )
+        if choice in ("age", "a"):
+            return "age"
+        return "neo4j"
+
    def install(self, prompt: bool = True) -> None:
        """Install and configure MemMachine."""
+        graph_backend = self._ask_graph_backend()
        neo4j_started_by_installer = False


Installer.install() always calls _ask_graph_backend() (input prompt) even when install(prompt=False). Since prompt is forwarded to ConfigurationWizard to control interactive behavior, install() should also respect it (e.g., default to "neo4j" when prompt is False) to avoid blocking non-interactive runs/tests.

Copilot · 2026-04-15T20:07:05Z

+        # embedding name for hydration.
+        kind = "node" if entity_type is EntityType.NODE else "edge"
+        key = f"{kind}|{sanitized_collection_or_relation}|{sanitized_embedding_name}"
+        digest = hashlib.sha1(key.encode("utf-8")).hexdigest()[:16]


_vector_table_name() uses hashlib.sha1() without the usedforsecurity=False flag. On some FIPS-enabled Python builds this can raise at runtime; the same file already uses usedforsecurity=False for index-name hashing, so side-table hashing should do the same for consistency and FIPS compatibility.

Suggested change

digest = hashlib.sha1(key.encode("utf-8")).hexdigest()[:16]

digest = hashlib.sha1(

key.encode("utf-8"), usedforsecurity=False

).hexdigest()[:16]

sscargal requested review from Copilot, edwinyyyu and malatewang April 15, 2026 20:00

Copilot started reviewing on behalf of sscargal April 15, 2026 20:03 View session

Copilot AI reviewed Apr 15, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: add Apache AGE as an alternative graph backend#1326

feat: add Apache AGE as an alternative graph backend#1326
reffdev wants to merge 1 commit into
MemMachine:mainfrom
reffdev:feat/age-support

reffdev commented Apr 14, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI Apr 15, 2026

Uh oh!

Copilot AI Apr 15, 2026

Uh oh!

Copilot AI Apr 15, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

reffdev commented Apr 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Purpose of the change

Description

Fixes/Closes

Type of change

How Has This Been Tested?

Checklist

Maintainer Checklist

Screenshots/Gifs

Further comments

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Copilot AI Apr 15, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 15, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 15, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

reffdev commented Apr 14, 2026 •

edited

Loading