Summary

Implement agentic chunking that uses an LLM to decide optimal chunk boundaries based on content understanding, producing higher-quality chunks than fixed-size or even semantic splitting.

Current State

The IngestionPipeline uses fixed-size chunking. Issue #215 adds semantic chunking. This goes a step further with LLM-driven boundaries.

Technical Design

1. Send document to LLM with instruction: "Split this into coherent, self-contained chunks"
2. LLM returns structured output with chunk boundaries and topic labels
3. Each chunk includes: content, topic label, key entities, summary
4. Expensive — use as "premium" chunking mode for high-value documents

Acceptance Criteria

• Given a complex document, when agentic chunking runs, then chunks are coherent and self-contained
• Given each chunk, when labeled, then it includes a topic and key entities
• Given the mode is configurable, when selected, then agentic chunking runs instead of fixed/semantic

Implementation Guide

1. Create AgenticChunker with LLM-based boundary detection using structured output (📋 Structured JSON output mode for LLM responses #214)
2. Add as ChunkingStrategy.AGENTIC enum option in IngestionPipeline
3. Use per-document: best for long, complex docs (> 10 pages)
4. Build: mvn clean compile -pl spector-ingestion

References

• IngestionPipeline.java
• Related: 🧠 Knowledge Graph Explorer in Cortex UI #215 (Semantic chunking), 📋 Structured JSON output mode for LLM responses #214 (Structured output)