Below is the technical design document I spent today crafting and plan on implementing to address: #12 (comment)

Objective

Enable users to safely soft-delete one or more knowledge graph nodes within a specified project using an MCP tool, rebuild a clean graph, optionally recalculate affected summary nodes based on referenced node IDs, and create a timestamped backup of the existing graph before recompilation. If dependent (child) nodes exist, return their details to the actor for user confirmation before proceeding with deletion. This addresses user needs to remove accidental, junk, or experimental entries (e.g., typos, irrelevant thoughts, or temporary spikes) while preserving data for recovery and maintaining graph integrity.

User Problems Addressed

• Accidental/Junk Entries: Clean up erroneous nodes (e.g., typos, irrelevant thoughts).
• Experimental Spikes: Remove temporary nodes from experimental work if discarded.
• Graph Management: Keep the knowledge graph concise and relevant without permanent data loss.
• Dependency Awareness: Ensure users are informed of dependent nodes and can decide whether to delete them.

Requirements

1. Soft Deletes Only: Move deleted nodes to knowledge_graph.deleted.ndjson for recovery.
2. Dependency Handling: Identify dependent nodes (via getDependentNodes), log warnings, and return their details in the JSON response. The actor presents an overview and prompts the user to decide on deleting them.
3. Clean Graph Rebuild: Create a new knowledge_graph.ndjson excluding deleted nodes.
4. Summary Recalculation: If recalculateSummaries is true, identify summary nodes referencing deleted node IDs, queue their recalculation, and recompile the graph.
5. Backup: Save a timestamped copy of the existing knowledge_graph.ndjson in ./data/backup/ before recompilation.
6. Project Isolation: Scope deletions to the project inferred from projectContext.
7. Auditability: Log deletions with metadata (e.g., reason, timestamp) in knowledge_graph.deleted.ndjson and via logger.
8. Accessibility: Expose functionality via an MCP tool (delete_nodes).
9. API Reuse: Leverage existing KnowledgeGraphManager methods (getNode, streamDagNodes, allDagNodes, appendEntity, parseDagNode) for efficiency and consistency.

System Context

• Knowledge Graph: A directed acyclic graph (DAG) stored in knowledge_graph.ndjson (append-only NDJSON). Nodes have a nodeId, project, thought, tags, artifacts, parents, children, createdAt, role (actor, critic, summary), and optional fields like verdict, target, summarizedSegment. No branch labels; summaries reference node IDs via summarizedSegment.
• Architecture: Built on the Model Context Protocol (MCP) server with KnowledgeGraphManager, ActorCriticEngine, and agents (Actor, Critic, SummarizationAgent).

Implementation Details

1. Input Schema (DeleteNodeSchema)

Validates user input for the delete_nodes MCP tool.

import { z } from 'zod';

const DeleteNodeSchema = z.object({
  nodeIds: z
    .array(z.string().min(1, 'Node ID cannot be empty'))
    .min(1, 'At least one node ID is required'),
  projectContext: z
    .string()
    .describe(
      'Full path to the currently open directory in the code editor. Used to infer the project name from the last item in the path.',
    ),
  reason: z.string().optional(),
  recalculateSummaries: z.boolean().default(false),
});

• Example:

  {
    "nodeIds": ["node123", "node124"],
    "projectContext": "/home/user/projects/my-project",
    "reason": "junk entry",
    "recalculateSummaries": true
  }

2. Output Schema (DeletedNodeSchema)

Defines the structure of deleted node records in knowledge_graph.deleted.ndjson.

import { z } from 'zod';

const DagNodeSchema = z.object({
  id: z.string().min(1),
  project: z.string().min(1),
  projectContext: z.string(),
  thought: z.string().optional(),
  role: z.enum(['actor', 'critic', 'summary']),
  tags: z.array(z.string()).optional(),
  artifacts: z
    .array(
      z.object({
        name: z.string(),
        path: z.string(),
        uri: z.string().optional(),
        hash: z.string().optional(),
        contentType: z.string().optional(),
      }),
    )
    .optional(),
  parents: z.array(z.string()),
  children: z.array(z.string()),
  createdAt: z.string().datetime(),
  verdict: z.enum(['approved', 'needs_revision', 'reject']).optional(),
  verdictReason: z.string().optional(),
  verdictReferences: z.array(z.string()).optional(),
  target: z.string().optional(),
  summarizedSegment: z.array(z.string()).optional(),
});

const DeletedNodeSchema = z.object({
  nodeId: z.string().min(1),
  project: z.string().min(1),
  reason: z.string().optional(),
  timestamp: z.string().datetime(),
  nodeData: DagNodeSchema,
});

• Example Entry:

  {
    "nodeId": "node123",
    "project": "my-project",
    "reason": "junk entry",
    "timestamp": "2025-05-17T19:12:00-04:00",
    "nodeData": {
      "id": "node123",
      "project": "my-project",
      "projectContext": "/home/user/projects/my-project",
      "thought": "Add login feature",
      "role": "actor",
      "tags": ["feature", "auth"],
      "artifacts": [{ "name": "login.ts", "path": "src/login.ts" }],
      "parents": ["node122"],
      "children": ["node125"],
      "createdAt": "2025-05-17T10:00:00-04:00"
    }
  }

3. Response Schema (DeleteResultSchema)

Defines the JSON response for the delete_nodes tool, including dependent nodes if present.

import { z } from 'zod';

const DependentNodeSchema = z.object({
  nodeId: z.string().min(1),
  thought: z.string().optional(),
  tags: z.array(z.string()).optional(),
  role: z.enum(['actor', 'critic', 'summary']),
  createdAt: z.string().datetime(),
});

const DeleteResultSchema = z.object({
  success: z.boolean(),
  deletedNodeIds: z.array(z.string()),
  dependentNodes: z.array(DependentNodeSchema).optional(),
  warnings: z.array(z.string()),
  backupPath: z.string().nullable(),
});

• Example Response (with dependent nodes):

  {
    "success": false,
    "deletedNodeIds": [],
    "dependentNodes": [
      {
        "nodeId": "node125",
        "thought": "Implement login validation",
        "tags": ["auth", "task"],
        "role": "actor",
        "createdAt": "2025-05-17T11:00:00-04:00"
      }
    ],
    "warnings": ["Node node123 has 1 dependent node"],
    "backupPath": null
  }

• Example Response (no dependent nodes, deletion complete):

  {
    "success": true,
    "deletedNodeIds": ["node123", "node124"],
    "dependentNodes": [],
    "warnings": [],
    "backupPath": "./data/backup/knowledge_graph_20250517-191200.ndjson"
  }

4. Knowledge Graph Enhancements (src/engine/KnowledgeGraph.ts)

Update KnowledgeGraphManager to reuse existing APIs and add new methods:

• deleteNodes(input: z.infer<typeof DeleteNodeSchema>): Promise<z.infer<typeof DeleteResultSchema>>:

  • Validates input with DeleteNodeSchema.
  • Infers project from projectContext (existing codebase logic).
  • Creates a backup of the existing knowledge_graph.ndjson using createBackup.
  • Checks for dependent nodes using getDependentNodes. If any exist:
    • Logs warnings: logger.warn({ nodeId, dependents }, 'Node has dependent nodes');.
    • Calls getNodeDetails to fetch details (nodeId, thought, tags, role, createdAt).
    • Returns { success: false, deletedNodeIds: [], dependentNodes, warnings, backupPath }.
  • If no dependent nodes or after user confirmation:
    • Uses getNode to fetch each node.
    • Creates DeletedNodeSchema-compliant records and appends to knowledge_graph.deleted.ndjson using appendEntity-style logic with file locking.
    • Calls rebuildGraph to exclude nodeIds.
    • If recalculateSummaries, uses getAffectedSummaries and queueSummaryRecalculation, then recompiles the graph.
    • Returns { success: true, deletedNodeIds, dependentNodes: [], warnings, backupPath }.

• getDependentNodes(nodeId: string): Promise<string[]>:

  • Uses streamDagNodes to iterate over nodes in the project.
  • Returns IDs of nodes where parents includes nodeId.

• getNodeDetails(nodeIds: string[]): Promise<z.infer<typeof DependentNodeSchema>[]>:

  • Uses getNode to fetch each node.
  • Extracts nodeId, thought, tags, role, createdAt for the response.

• rebuildGraph(excludeNodeIds: string[]): Promise<void>:

  • Reuses streamDagNodes to read knowledge_graph.ndjson.
  • Writes a temp file excluding excludeNodeIds.
  • Uses proper-lockfile for safe renaming, similar to appendEntity.

• getSummaryNodesByProject(project: string): Promise<string[]>:

  • Uses allDagNodes to fetch nodes in the project.
  • Filters for nodes with role: "summary".

• getAffectedSummaries(nodeIds: string[], project: string): Promise<string[]>:

  • Uses getSummaryNodesByProject to get summary nodes.
  • Uses getNode to check summarizedSegment for any nodeIds.

• queueSummaryRecalculation(summaryNodeIds: string[], project: string): Promise<void>:

  • Queues updates via SummarizationAgent, processing sequentially.

• createBackup(): Promise<string>:

  • Saves a timestamped copy of the current knowledge_graph.ndjson to ./data/backup/knowledge_graph_20250517-191200.ndjson.
  • Ensures ./data/backup/ exists (fs.mkdirSync).
  • Uses fs.copyFile for efficient copying.

• restoreNodes(nodeIds: string[]): Promise<RestoreResult>:

  • Reads knowledge_graph.deleted.ndjson using readline.
  • Reuses appendEntity to restore nodes to knowledge_graph.ndjson.

• Persistence:

  • knowledge_graph.deleted.ndjson: Append-only, using appendEntity-style logic.
  • File locking via proper-lockfile for all NDJSON operations.

• API Reuse:

  • getNode: Fetch individual nodes for deletion and details.
  • streamDagNodes: Stream nodes for getDependentNodes and rebuildGraph.
  • allDagNodes: Fetch summary nodes in getSummaryNodesByProject.
  • appendEntity: Append deleted nodes to knowledge_graph.deleted.ndjson and restored nodes to knowledge_graph.ndjson.
  • parseDagNode: Parse NDJSON lines for consistency.
  • DagNodeSchema: Validate node data.

5. MCP Tool Integration (src/index.ts)

Add delete_nodes tool:

• Description:

  Soft-delete nodes in the specified project, moving them to knowledge_graph.deleted.ndjson. Backs up the existing graph before changes. If dependent nodes exist, returns their details for user confirmation. Rebuilds the graph, optionally recalculates summaries based on referenced node IDs. Project is inferred from projectContext.
  

• Implementation:

  • Register with @modelcontextprotocol/sdk.
  • Validate input with DeleteNodeSchema.
  • Call KnowledgeGraphManager.deleteNodes.
  • Return: JSON conforming to DeleteResultSchema.

• Example Input:

  {
    "tool": "delete_nodes",
    "input": {
      "nodeIds": ["node123", "node124"],
      "projectContext": "/home/user/projects/my-project",
      "reason": "experimental spike cleanup",
      "recalculateSummaries": true
    }
  }

• Actor Handling:

  • If dependentNodes is non-empty, the actor presents an overview (e.g., node ID, thought, tags, role, creation date) to the user and prompts: “Node node123 has 1 dependent node. Delete it? [y/N]”.
  • If the user confirms, the actor calls delete_nodes again, including the dependent node IDs.
  • If declined, the actor aborts or proceeds with only the non-dependent nodes.

6. Summarization Integration (src/agents/Summarize.ts)

Update SummarizationAgent:

• Enhance checkAndTriggerSummarization:
  • Accept a summary node ID and project.
  • Recalculate summary using referenced node IDs (via KnowledgeGraphManager.getNode), excluding deleted nodes.
• Queueing:
  • Process sequentially with an in-memory array.
  • Log: logger.info({ nodeId }, 'Recalculated summary node');.

7. Auditing and Logging (src/logger.ts)

• Deletions:

  logger.info({ nodeIds, project, reason }, 'Soft-deleted nodes');

• Dependent Nodes:

  logger.warn({ nodeId, dependents }, 'Node has dependent nodes');

• Summaries:

  logger.info({ nodeId }, 'Recalculated summary node');

• Backups:

  logger.info({ path }, 'Created backup at %s', path);

8. Documentation Updates

• docs/OVERVIEW.md:

  - **Node Deletion**: Soft-delete nodes using the `delete_nodes` MCP tool. Backs up the existing graph before changes. If dependent nodes exist, their details are returned for user confirmation. Nodes move to `knowledge_graph.deleted.ndjson` with metadata. Optionally recalculate summaries. Project is inferred from projectContext.
  

• docs/INSTALL_GUIDE.md:

  ## MCP Tools
  - Delete nodes:
    ```json
    {
      "tool": "delete_nodes",
      "input": {
        "nodeIds": ["node123", "node124"],
        "projectContext": "/home/user/projects/my-project",
        "reason": "junk entry",
        "recalculateSummaries": true
      }
    }
  

Development Steps

1. Implement KnowledgeGraphManager methods: deleteNodes, rebuildGraph, createBackup, getAffectedSummaries, getNodeDetails, getDependentNodes, getSummaryNodesByProject, queueSummaryRecalculation, and restoreNodes.
   • Reuse getNode, streamDagNodes, allDagNodes, appendEntity, and parseDagNode.
   • Add dependency checking and response formatting in deleteNodes.
   • Update createBackup to copy knowledge_graph.ndjson before recompilation.
   • Test NDJSON streaming and file locking for rebuildGraph and createBackup.
2. Update SummarizationAgent to support queueSummaryRecalculation.
   • Enhance checkAndTriggerSummarization for targeted summary recalculation.
3. Implement the delete_nodes MCP tool in src/index.ts.
   • Register with @modelcontextprotocol/sdk and use DeleteResultSchema.
   • Test integration with the MCP server and actor prompting for dependent nodes.
4. Add logging for deletions, dependent nodes, summaries, and backups in logger.ts.
   • Update documentation in docs/OVERVIEW.md and docs/INSTALL_GUIDE.md.
5. Validate the feature with Jest tests (pnpm test).
   • Test end-to-end workflow, including dependent node handling and user confirmation.
   • Verify restore functionality with restoreNodes and backup integrity.

Risks and Mitigations

• Risk: Missing dependent nodes in response.
  • Mitigation: Test getDependentNodes and getNodeDetails with complex DAGs.
• Risk: Slow rebuild for large graphs.
  • Mitigation: Use streamDagNodes and readline; test with 10,000+ nodes.
• Risk: Backup directory clutter.
  • Mitigation: Plan future cleanup policy (e.g., prune backups >90 days).

Future Considerations

• Node ID Querying: Add an MCP tool to list nodeIds by project or tags.
• Restore Tool: Implement restore_nodes MCP tool for recovery.
• Summary Optimization: Cache summary-to-node mappings.