A scalable, distributed system for document chunking and processing in Retrieval Augmented Generation (RAG) applications. This repository contains both the high-level architecture and a working implementation of an AI troubleshooting assistant powered by RAG.
RAG-Chunking provides a comprehensive solution for:
- Efficient document ingestion and chunking
- Parallel processing of large document collections
- Semantic vector embedding generation
- Vector-based similarity search
- Intelligent query handling with context awareness
Perfect for building domain-specific AI assistants that can leverage technical documentation, manuals, and knowledge bases.
The system consists of several key components:
- Document Intake Service: Processes documents from multiple sources
- Document Splitter: Intelligently chunks documents with semantic awareness
- Embedding Generator: Creates vector representations of document chunks
- Vector Database: Stores and indexes chunks for fast retrieval
- Query Processor: Handles user queries and retrieves relevant context
- Response Generator: Creates AI responses based on retrieved context
- ✅ Parallel Processing: Process documents concurrently for maximum throughput
- ✅ Smart Chunking: Preserve semantic meaning with intelligent document splitting
- ✅ Scalable Architecture: Add processing nodes to handle larger document volumes
- ✅ Multi-Format Support: Process PDFs, HTML, text files, and more
- ✅ Conversation Context: Maintain dialogue history for improved responses
- ✅ Extensive Logging: Comprehensive monitoring and observability
- Azure account
- Python 3.10+
# Clone the repository
git clone https://github.com/ptbdnr/rag-demo.git
cd rag-demo
# Create and activate virtual environment
python -m venv venv
source venv/bin/activate # On Windows: venv\Scripts\activate
# Install dependencies
pip install -r requirements.txtSee quick README.md in /back and /front directories.
For deployment on Azure see README.md in /infra directory.
This repository includes a complete implementation of a troubleshooting assistant for household appliances. The assistant:
- Processes technical service manuals and user guides
- Understands error codes and common problems
- Provides step-by-step troubleshooting instructions
- Maintains conversation context for follow-up questions
See examples/TODO for a complete demonstration.
The system is designed for horizontal scaling:
- Each component can be deployed as a separate microservice
- Processing can be distributed across multiple nodes
- Document processing is parallelized for maximum throughput
- Supports multi-tenant operation with resource isolation
This project is licensed under the GNU GPLv3 License - see the LICENSE file for details.