AnyDocs MCP Server

Transform any website's documentation section into an MCP-compatible server using the Python MCP SDK.

🚀 Overview

AnyDocs MCP Server is a comprehensive solution that turns any website's documentation into an interactive, AI-accessible knowledge base through the Model Context Protocol (MCP). It can scrape, index, and serve documentation from any website - from modern API docs to legacy documentation portals.

Key Features

🌐 Universal Website Scraping: Turn ANY website's documentation into an interactive knowledge base
🔌 Universal Adapter System: Support for GitBook, Notion, Confluence, and custom documentation platforms
🔍 Advanced Search: Full-text search with SQLite FTS and semantic search capabilities
🔐 Robust Authentication: API Key, OAuth2, and JWT-based authentication
⚡ High Performance: Async/await architecture with caching and rate limiting
🎛️ Web Management Interface: FastAPI-based admin panel for configuration and monitoring
📊 Real-time Monitoring: Health checks, metrics, and logging
🐳 Docker Ready: Complete containerization with development and production configurations
🔄 Auto-sync: Automatic content synchronization with source documentation

📋 Requirements

Python 3.11+ (recommended: 3.11 or 3.12)
SQLite 3.35+ (for FTS5 support)
Optional: Redis (for caching)
Optional: PostgreSQL/MySQL (for production)

🛠️ Installation

Using uvx (Recommended)

The easiest way to run AnyDocs MCP Server is using uvx, which automatically manages dependencies and virtual environments:

# Run directly with uvx (no installation needed)
uvx anydocs-mcp-server

# Run with custom configuration
uvx anydocs-mcp-server --config config.yaml

# Run in debug mode
uvx anydocs-mcp-server --debug

# Install globally with uvx for repeated use
uvx install anydocs-mcp-server

# Then run anytime with:
anydocs-mcp-server --config config.yaml

Quick Start

# Clone the repository
git clone https://github.com/funky1688/anydocs-mcp.git
cd anydocs-mcp

# Install dependencies using uv (recommended)
pip install uv
uv pip install -e .

# Copy environment configuration
cp .env.example .env

# Copy and customize configuration
cp config.yaml my-config.yaml

# Edit configuration (add your API keys and settings)
nano .env

# Start the server (hybrid mode - MCP + Web interface)
uv run python start.py

Manual Installation

# Create virtual environment (recommended)
python -m venv venv
source venv/bin/activate  # On Windows: venv\Scripts\activate

# Install dependencies
uv pip install -e .  # for production
# OR for development:
uv pip install -e .[dev]

# Copy and configure environment
cp .env.example .env

# Edit configuration files
nano .env

# Initialize and start
uv run python start.py --mode hybrid --debug

Running as a Python Module

After installation, you can also run the server as a Python module:

# Run as module
python -m anydocs_mcp

# With configuration
python -m anydocs_mcp --config config.yaml

# Debug mode
python -m anydocs_mcp --debug

Docker Installation

# Development environment
docker-compose -f docker-compose.dev.yml up -d

# Production environment
docker-compose up -d

🚀 Usage

Starting the Server

AnyDocs MCP Server supports 3 startup modes:

1. Hybrid Mode (Default - Recommended)

Starts both MCP server and web management interface simultaneously:

uv run python start.py
# or explicitly:
uv run python start.py --mode hybrid

MCP Server: Available at http://localhost:8000 (handles MCP protocol communication)
Web Interface: Available at http://localhost:8080 for management
Best for: Most users who want both MCP functionality and web management

Important: Always use uv run to ensure the correct virtual environment is used.

2. MCP Server Only

Starts only the MCP server without web interface:

uv run python start.py --mode mcp

Use case: Production deployments where only MCP protocol is needed
Lighter resource usage: No web interface overhead
Best for: Headless servers, CI/CD environments

3. Web Interface Only

Starts only the web management interface:

uv run python start.py --mode web

Use case: Administrative tasks, configuration management
Web Interface: Available at http://localhost:8080
Best for: Configuration, monitoring, and testing without MCP protocol

Additional Options

# Debug mode with auto-reload
uv run python start.py --debug

# Custom configuration file
uv run python start.py --config custom-config.yaml

# Skip dependency check (faster startup)
uv run python start.py --no-deps-check

# Kill occupied ports before starting
uv run python start.py --kill-ports

# Skip database initialization
uv run python start.py --no-db-init

Command Line Options

uv run python start.py --help

# Available options:
#   --mode {mcp,web,hybrid}     Startup mode (default: hybrid)
#   --config CONFIG             Configuration file path (default: config.yaml)
#   --debug                     Enable debug mode
#   --no-deps-check            Skip dependency check
#   --no-db-init               Skip database initialization
#   --kill-ports               Kill occupied ports before starting

Web Management Interface

Access the web interface at http://localhost:8080 to:

Configure document sources
Manage users and API keys
Monitor system health
View logs and metrics
Test MCP endpoints

MCP Client Integration

from mcp import ClientSession, StdioServerParameters
from mcp.client.stdio import stdio_client

# Connect to AnyDocs MCP Server
async with stdio_client(StdioServerParameters(
    command="python",
    args=["main.py"]
)) as (read, write):
    async with ClientSession(read, write) as session:
        # Initialize the connection
        await session.initialize()
        
        # List available tools
        tools = await session.list_tools()
        
        # Search documents
        result = await session.call_tool(
            "search_documents",
            arguments={"query": "authentication", "limit": 10}
        )

📚 Documentation Adapters

Supported Platforms

Platform	Status	Features
Any Website	✅	Universal scraper for any documentation site
GitBook	✅	Full API integration, real-time sync
Notion	✅	Database and page content, webhooks
Confluence	✅	Space and page management, attachments
GitHub	✅	Repository documentation, wikis
GitLab	✅	Project documentation, wikis
SharePoint	✅	Document libraries, lists
Slack	✅	Channel messages, knowledge base
File System	✅	Local markdown files, watch mode
Custom	🔧	Extensible adapter framework

Adding a New Adapter

from anydocs_mcp.adapters.base import BaseDocumentAdapter

class CustomAdapter(BaseDocumentAdapter):
    """Custom documentation adapter implementation."""
    
    async def fetch_documents(self) -> List[Document]:
        """Fetch documents from your platform."""
        # Implementation here
        pass
    
    async def get_document_content(self, doc_id: str) -> str:
        """Get specific document content."""
        # Implementation here
        pass

🔧 Configuration

Troubleshooting

Common Installation Issues

Python-Jose Import Error: If you encounter No module named 'jose' error:

# Always use uv run to ensure correct virtual environment
uv run python start.py

# If the issue persists, reinstall python-jose with cryptography extras
uv pip uninstall python-jose
uv pip install "python-jose[cryptography]"

Dependency Check Failures: If you see errors about missing pyyaml or beautifulsoup4:

# The dependency check has been fixed to use correct import names
# Ensure you're using the latest version
git pull origin main
uv pip install -e .

Configuration Attribute Errors: If you see 'AppConfig' object has no attribute 'server_host':

# This is fixed in the current version - ensure you have the latest code
git pull origin main

Virtual Environment Issues: If packages seem installed but imports fail:

# Always use 'uv run' to ensure correct environment
uv run python start.py

# Check if you're in the right environment
which python  # Should point to your project's Python
uv pip list    # Should show installed packages

Setup.py Conflicts: If you encounter conflicts with multiple setup files:

# The redundant root setup.py has been removed
# Use pyproject.toml for package management
uv pip install -e .

Environment Setup Best Practices

Always use uv run for executing Python scripts to ensure correct environment
Use uv pip install instead of uv install for package installation
Check virtual environment with uv pip list if imports fail
Pull latest changes if you encounter configuration issues

Port Conflicts

If ports 8000 or 8080 are occupied:

# Kill processes on required ports (Windows)
netstat -ano | findstr :8000
taskkill /F /PID <PID>

# Or use the built-in option
uv run python start.py --kill-ports

Environment Variables

# Server Configuration
ANYDOCS_HOST=localhost
ANYDOCS_PORT=8000
ANYDOCS_WEB_PORT=8080
ANYDOCS_DEBUG=false

# Database
DATABASE_URL=sqlite:///data/anydocs.db
# DATABASE_URL=postgresql://user:pass@localhost/anydocs

# Authentication
JWT_SECRET_KEY=your-secret-key
API_KEY_PREFIX=anydocs_

# Document Adapters
GITBOOK_API_TOKEN=your-gitbook-token
NOTION_API_TOKEN=your-notion-token
CONFLUENCE_API_TOKEN=your-confluence-token

# Cache (Optional)
REDIS_URL=redis://localhost:6379/0

# Monitoring
ENABLE_METRICS=true
LOG_LEVEL=INFO

YAML Configuration

# config.yaml
server:
  host: localhost
  port: 8000
  web_port: 8080
  debug: false

database:
  url: sqlite:///data/anydocs.db
  pool_size: 10
  echo: false

auth:
  jwt_secret: ${JWT_SECRET_KEY}
  token_expire_minutes: 1440
  
  api_key:
    prefix: anydocs_
    length: 32

adapters:
  gitbook:
    api_token: ${GITBOOK_API_TOKEN}
    base_url: https://api.gitbook.com
    rate_limit: 100
  
  notion:
    api_token: ${NOTION_API_TOKEN}
    version: "2022-06-28"
    rate_limit: 3

🔍 MCP Tools

AnyDocs MCP Server provides the following tools:

Core Tools

search_documents - Search documents with full-text and semantic search
get_document - Retrieve a specific document by ID
list_sources - List all configured document sources
summarize_content - Summarize document content
ask_question - Ask questions about document content

AI-Powered Tools

generate_documentation - AI-assisted documentation generation
translate_content - Multi-language content translation
extract_insights - Extract insights and analytics from documentation
suggest_improvements - AI-powered content enhancement suggestions

🧪 Development

Setup Development Environment

# Install development dependencies
uv pip install -e .[dev]

# Setup pre-commit hooks
uv run pre-commit install

# Run tests
uv run pytest

# Run with coverage
uv run pytest --cov=src/anydocs_mcp

# Code formatting
uv run black src/ tests/
uv run isort src/ tests/

# Type checking
uv run mypy src/

# Security checks
uv run bandit -r src/

Project Structure

anydocs-mcp/
├── src/anydocs_mcp/          # Main package
│   ├── adapters/             # Document adapters
│   ├── auth/                 # Authentication
│   ├── config/               # Configuration management
│   ├── content/              # Content processing
│   ├── database/             # Database models and operations
│   ├── utils/                # Utilities and helpers
│   ├── web/                  # Web interface
│   └── server.py             # MCP server implementation
├── tests/                    # Test suite
├── docs/                     # Documentation
├── scripts/                  # Utility scripts
│   └── setup.py              # Development environment setup
├── pyproject.toml            # Package configuration (modern Python packaging)
├── start.py                  # Main startup script
└── main.py                   # MCP server entry point

Note: The project uses pyproject.toml for package configuration following modern Python packaging standards. The redundant root setup.py has been removed to avoid conflicts.

Running Tests

# All tests
uv run pytest

# Unit tests only
uv run pytest tests/unit/

# Integration tests only
uv run pytest tests/integration/

# With coverage
uv run pytest --cov=src/anydocs_mcp --cov-report=html

# Performance tests
uv run pytest tests/performance/

📊 Monitoring & Observability

Health Checks

# Check service health
curl http://localhost:8080/health

# Detailed health check
curl http://localhost:8080/health/detailed

Metrics

Metrics are available at /metrics endpoint in Prometheus format:

Request count and duration
Database connection pool status
Document sync statistics
Error rates and types
Cache hit/miss ratios

Logging

Structured logging with configurable levels:

# Application logs
tail -f anydocs_mcp.log

# Check logs in real-time
uv run python start.py --debug

🐳 Docker Deployment

Development

# Start development environment
docker-compose -f docker-compose.dev.yml up -d

# View logs
docker-compose logs -f anydocs-mcp-dev

# Access shell
docker-compose exec anydocs-mcp-dev bash

Production

# Build and start production environment
docker-compose up -d

# Scale services
docker-compose up -d --scale anydocs-mcp=3

# Update services
docker-compose pull && docker-compose up -d

Monitoring Stack

# Start with monitoring
docker-compose --profile monitoring up -d

# Access services
# Grafana: http://localhost:3001 (admin/admin)
# Prometheus: http://localhost:9090

🔒 Security

Authentication Methods

API Keys: Simple token-based authentication
JWT Tokens: Stateless authentication with expiration
OAuth2: Integration with external providers

Security Best Practices

All API endpoints require authentication
Rate limiting on all endpoints
Input validation and sanitization
SQL injection prevention
CORS configuration
Security headers
Audit logging

Security Scanning

# Run security checks
uv run bandit -r src/

# Dependency vulnerability scan
uv run safety check

# SAST scanning
uv run bandit -r src/

🚀 Performance

Optimization Features

Async/Await: Non-blocking I/O operations
Connection Pooling: Efficient database connections
Caching: Redis-based caching with TTL
Rate Limiting: Prevent API abuse
Batch Processing: Efficient bulk operations
Lazy Loading: On-demand content loading

Performance Monitoring

# Performance testing
uv run pytest tests/performance/

# Memory profiling
uv run python -m memory_profiler start.py

🤝 Contributing

We welcome contributions! Please see our Contributing Guide for details.

Development Workflow

Fork the repository
Create a feature branch
Make your changes
Add tests
Run the test suite
Submit a pull request

Code Standards

Follow PEP 8 style guide
Use type hints
Write comprehensive tests
Document public APIs
Use meaningful commit messages

📄 License

This project is licensed under the MIT License - see the LICENSE file for details.

🙏 Acknowledgments

Model Context Protocol for the MCP specification
Python MCP SDK for the SDK implementation
All contributors and maintainers

📞 Support

📧 Email: [email protected]
🐛 Issues: GitHub Issues
📖 Documentation: Full Documentation

Made with ❤️ by funky1688

Name		Name	Last commit message	Last commit date
Latest commit History 18 Commits
.github/workflows		.github/workflows
monitoring		monitoring
scripts		scripts
src/anydocs_mcp		src/anydocs_mcp
.claude-mcp.json		.claude-mcp.json
.env.example		.env.example
.gitignore		.gitignore
.python-version		.python-version
CODE_OF_CONDUCT.md		CODE_OF_CONDUCT.md
CONTRIBUTING.md		CONTRIBUTING.md
Dockerfile		Dockerfile
Dockerfile.dev		Dockerfile.dev
LICENSE		LICENSE
Makefile		Makefile
README.md		README.md
SECURITY.md		SECURITY.md
docker-compose.dev.yml		docker-compose.dev.yml
docker-compose.yml		docker-compose.yml
main.py		main.py
pyproject.toml		pyproject.toml
start.bat		start.bat
start.py		start.py
start.sh		start.sh
uv.lock		uv.lock

License

funky1688/AnyDocs-MCP

Folders and files

Latest commit

History

Repository files navigation