Thanks to visit codestin.com
Credit goes to Github.com

Skip to content

A complete REST API server using the DDGS library for metasearch functionality.

Notifications You must be signed in to change notification settings

Jayanth-MKV/ddgs-server

Repository files navigation

DDGS Server - Complete Implementation

A production-ready server with DDGS metasearch integration and MCP (Model Context Protocol) support. A production-ready server with DDGS metasearch integration and MCP (Model Context Protocol) support.


πŸ“¦ Project Structure

ddgs-server/
β”œβ”€β”€ main.py                 # FastAPI app initialization and router registration
β”œβ”€β”€ mcp_server.py          # MCP server with search tools for LLMs
β”œβ”€β”€ mcp_server.py          # MCP server with search tools for LLMs
β”œβ”€β”€ models/                 # Data models and schemas
β”‚   β”œβ”€β”€ __init__.py
β”‚   └── schemas.py         # Pydantic models and enums
β”œβ”€β”€ controllers/           # Business logic layer
β”‚   β”œβ”€β”€ __init__.py
β”‚   β”œβ”€β”€ text_controller.py
β”‚   β”œβ”€β”€ image_controller.py
β”‚   β”œβ”€β”€ video_controller.py
β”‚   β”œβ”€β”€ news_controller.py
β”‚   β”œβ”€β”€ book_controller.py
β”‚   └── unified_controller.py  # Searches all sources at once
β”œβ”€β”€ routes/                # API route definitions
β”‚   β”œβ”€β”€ __init__.py
β”‚   β”œβ”€β”€ text_routes.py
β”‚   β”œβ”€β”€ image_routes.py
β”‚   β”œβ”€β”€ video_routes.py
β”‚   β”œβ”€β”€ news_routes.py
β”‚   β”œβ”€β”€ book_routes.py
β”‚   └── unified_routes.py      # Unified search endpoint
└── public/

πŸš€ Quick Start

1. Installation

# Create project directory
mkdir ddgs-server
cd ddgs-server

# Install dependencies
pip install -r requirements.txt

2. Run the Server

# Save the main.py file from the generated code
# Then run:
uvicorn main:app --reload --host 0.0.0.0 --port 8000

3. Access Documentation


πŸ€– MCP Server Integration

This server includes an integrated MCP (Model Context Protocol) server that provides LLM-friendly search tools. The MCP server is mounted at /mcp and exposes the following tools:

MCP Tools Available:

  • search_web - Search the web for text content
  • search_images - Search for images with size/color filters
  • search_videos - Search for videos with resolution/duration filters
  • search_news - Search for news articles with time limits
  • search_books - Search for books
  • search_everything - Search all sources at once (parallel)
  • fetch_content - Extract content from a single URL
  • fetch_multiple_contents - Extract content from multiple URLs (max 10)

Using MCP Server:

The MCP server reuses all existing controller logic, making it efficient and maintainable. It's accessible at /mcp/mcp for LLM clients that support the MCP protocol.


πŸ€– MCP Server Integration

This server includes an integrated MCP (Model Context Protocol) server that provides LLM-friendly search tools. The MCP server is mounted at /mcp and exposes the following tools:

MCP Tools Available:

  • search_web - Search the web for text content
  • search_images - Search for images with size/color filters
  • search_videos - Search for videos with resolution/duration filters
  • search_news - Search for news articles with time limits
  • search_books - Search for books
  • search_everything - Search all sources at once (parallel)

Using MCP Server:

The MCP server reuses all existing controller logic, making it efficient and maintainable. It's accessible at /mcp for LLM clients that support the MCP protocol.


πŸ”Œ API Endpoints

Overview

Endpoint Method Description
/ GET API information
/health GET Health check
/api/search/text GET Web/text search
/api/content/fetch GET Fetch content from URL
/api/content/fetch-multiple POST Fetch content from multiple URLs
/api/search/images GET Image search
/api/search/videos GET Video search
/api/search/news GET News search
/api/search/books GET Book search

πŸ“ Endpoint Details

1. Text/Web Search

Endpoint: GET /api/search/text

Parameters:

  • q (required): Search query
  • region (optional): Region code (default: us-en)
  • safesearch (optional): on/moderate/off (default: moderate)
  • timelimit (optional): d/w/m/y
  • max_results (optional): 1-100 (default: 10)
  • page (optional): Page number (default: 1)
  • backend (optional): Search backend (default: auto)

Example:

curl "http://localhost:8000/api/search/text?q=python%20programming&max_results=5"

Response:

{
  "success": true,
  "query": "python programming",
  "results_count": 5,
  "results": [
    {
      "title": "Python Programming Language",
      "href": "https://www.python.org/",
      "body": "Official Python website..."
    }
  ]
}

2. Image Search

Endpoint: GET /api/search/images

Additional Parameters:

  • size: Small/Medium/Large/Wallpaper
  • color: color/Monochrome/Red/Orange/Yellow/Green/Blue/Purple/Pink
  • type_image: photo/clipart/gif/transparent/line
  • layout: Square/Tall/Wide

Example:

curl "http://localhost:8000/api/search/images?q=butterfly&size=Large&color=color"

3. Video Search

Endpoint: GET /api/search/videos

Additional Parameters:

  • resolution: high/standard
  • duration: short/medium/long
  • license_videos: creativeCommon/youtube

Example:

curl "http://localhost:8000/api/search/videos?q=tutorial&resolution=high&duration=medium"

4. News Search

Endpoint: GET /api/search/news

Parameters:

  • Same as text search
  • timelimit limited to: d/w/m

Example:

curl "http://localhost:8000/api/search/news?q=technology&timelimit=w&max_results=20"

5. Book Search

Endpoint: GET /api/search/books

Parameters:

  • q (required): Search query
  • max_results (optional): 1-100 (default: 10)
  • page (optional): Page number

Example:

curl "http://localhost:8000/api/search/books?q=machine%20learning&max_results=15"

6. Content Fetching

Endpoint: GET /api/content/fetch

Parameters:

  • url (required): URL to fetch content from
  • timeout (optional): Request timeout 5-30 seconds (default: 10)

Example:

curl "http://localhost:8000/api/content/fetch?url=https://example.com/article"

Response:

{
  "url": "https://example.com/article",
  "title": "Article Title",
  "description": "Meta description",
  "content": "Main article text...",
  "content_length": 5420,
  "status_code": 200
}

Endpoint: POST /api/content/fetch-multiple

Body:

{
  "urls": ["https://example.com/page1", "https://example.com/page2"],
  "timeout": 10
}

Example:

curl -X POST "http://localhost:8000/api/content/fetch-multiple" \
  -H "Content-Type: application/json" \
  -d '{"urls": ["https://example.com/article1", "https://example.com/article2"]}'

🐍 Python Client Usage

import requests

# Text search
response = requests.get(
    "http://localhost:8000/api/search/text",
    params={
        "q":  tutorial",
        "region": "us-en",
        "max_results": 10
    }
)

data = response.json()
print(f"Found {data['results_count']} results")

for result in data['results']:
    print(f"- {result['title']}: {result['href']}")

🌐 JavaScript/Fetch Usage

// Image search
fetch('http://localhost:8000/api/search/images?q=nature&size=Large&max_results=20')
  .then(response => response.json())
  .then(data => {
    console.log(`Found ${data.results_count} images`);
    data.results.forEach(img => {
      console.log(img.title, img.image);
    });
  });

πŸ§ͺ Testing with cURL

Text Search

curl -X GET "http://localhost:8000/api/search/text?q=python&max_results=5" \
  -H "accept: application/json"

Image Search with Filters

curl -X GET "http://localhost:8000/api/search/images?q=sunset&size=Large&color=Orange" \
  -H "accept: application/json"

News Search (Last Week)

curl -X GET "http://localhost:8000/api/search/news?q=AI&timelimit=w&max_results=10" \
  -H "accept: application/json"

🐳 Docker Deployment

Using Docker

# Build image
docker build -t ddgs-api .

# Run container
docker run -p 8000:8000 ddgs-api

Using Docker Compose

# Start service
docker-compose up -d

# View logs
docker-compose logs -f

# Stop service
docker-compose down

βš™οΈ Configuration

Environment Variables

Create a .env file:

HOST=0.0.0.0
PORT=8000
LOG_LEVEL=info
DDGS_TIMEOUT=10
DDGS_PROXY=  # Optional proxy

Using Proxy

Modify the DDGS initialization in main.py:

ddgs = DDGS(
    proxy="socks5h://127.0.0.1:9150",  # Tor browser
    timeout=10
)

πŸ”’ Security Features

  1. Input Validation: Pydantic models validate all inputs
  2. Rate Limit Handling: Proper HTTP 429 responses
  3. Error Handling: Comprehensive exception handling
  4. Type Safety: Full type hints throughout
  5. Logging: Request logging for monitoring

πŸ“Š Response Format

Success Response

{
  "success": true,
  "query": "search term",
  "results_count": 10,
  "results": [...]
}

Error Response

{
  "success": false,
  "error": "Error message",
  "details": "Additional error details"
}

🚦 HTTP Status Codes

Code Description
200 Success
400 Bad Request (invalid parameters)
429 Rate Limit Exceeded
504 Gateway Timeout
500 Internal Server Error

πŸ› οΈ Advanced Features

CORS Configuration

Add CORS middleware for cross-origin requests:

from.middleware.cors import CORSMiddleware

app.add_middleware(
    CORSMiddleware,
    allow_origins=["*"],
    allow_credentials=True,
    allow_methods=["*"],
    allow_headers=["*"],
)

Rate Limiting

Install and add rate limiting:

pip install slowapi
from slowapi import Limiter, _rate_limit_exceeded_handler
from slowapi.util import get_remote_address

limiter = Limiter(key_func=get_remote_address)
app.state.limiter = limiter

@app.get("/api/search/text")
@limiter.limit("30/minute")
async def search_text(...):
    ...

πŸ“ˆ Production Tips

  1. Use multiple workers:

    uvicorn main:app --workers 4 --host 0.0.0.0 --port 8000
  2. Set up reverse proxy (Nginx example):

    location / {
        proxy_pass http://localhost:8000;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
    }
  3. Enable HTTPS using Certbot/Let's Encrypt

  4. Monitor with logging:

    • Use structured logging
    • Send logs to monitoring service
  5. Use environment-based configuration


πŸ“š Full Code Files

Dockerfile

FROM python:3.11-slim

WORKDIR /app

COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

COPY main.py .

EXPOSE 8000

CMD ["uvicorn", "main:app", "--host", "0.0.0.0", "--port", "8000"]

docker-compose.yml

version: '3.8'

services:
  ddgs-api:
    build: .
    ports:
      - "8000:8000"
    environment:
      - LOG_LEVEL=info
    restart: unless-stopped
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:8000/health"]
      interval: 30s
      timeout: 10s
      retries: 3

πŸ” Testing the Server

Manual Testing

# Health check
curl http://localhost:8000/health

# API root
curl http://localhost:8000/

# Text search
curl "http://localhost:8000/api/search/text?q=test&max_results=3"

🎯 Features Summary

βœ… 5 Complete Search Endpoints

  • Text/Web Search
  • Image Search with filters
  • Video Search with filters
  • News Search
  • Book Search

βœ… Production Ready

  • Auto-generated API documentation
  • Input validation with Pydantic
  • Comprehensive error handling
  • Logging and monitoring
  • Docker support

βœ… Developer Friendly

  • Type hints throughout
  • Clear endpoint documentation
  • Example client code
  • Easy to extend

πŸ“ License

MIT License


🀝 Contributing

Contributions welcome! Please:

  1. Fork the repository
  2. Create a feature branch
  3. Submit a pull request

πŸ“§ Support

For issues:


πŸŽ‰ Ready to Use!

Your DDGS server is now complete with:

  • Full implementation in main.py
  • Test client in test_client.py
  • Docker configuration
  • Comprehensive documentation
  • Production-ready features

Start the server and visit http://localhost:8000/docs to explore the interactive API documentation!

About

A complete REST API server using the DDGS library for metasearch functionality.

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published