Distributed Task Queue built from scratch on Redis Streams.
JobForge is a production-quality, fully async task queue system demonstrating how to build reliable distributed infrastructure with Redis 7 Streams (XADD, XREADGROUP, XACK, XCLAIM). It features priority queues, exponential-backoff retries, a dead-letter queue, job dependencies, per-job-type rate limiting, a real-time React dashboard, and a chaos mode for resilience testing.
┌──────────────────────────────────────────────────────────────────┐
│ Redis 7 Streams │
│ │
│ jobforge:queue:high ──┐ │
│ jobforge:queue:normal ──┼──► Consumer Group "workers" │
│ jobforge:queue:low ──┘ │ │
│ ▼ │
│ jobforge:retry:scheduled (ZSET) ←── backoff scheduler │
│ jobforge:queue:dlq (Stream) ←── permanently failed jobs │
│ jobforge:workers:* (Hash) ←── heartbeat registry │
│ jobforge:jobs:* (Hash) ←── job metadata │
│ jobforge:deps:* (Set) ←── dependency tracking │
│ jobforge:rate:* (String)←── sliding-window counters │
└──────────────────────────────────────────────────────────────────┘
▲ │
│ XADD │ XREADGROUP (claim)
│ ▼
┌────────┴────────┐ ┌─────────────────────┐
│ FastAPI API │ │ Async Workers (×3) │
│ :8000 │ │ concurrency=4 each │
│ + WebSocket │ │ heartbeat every 10s│
└─────────────────┘ └─────────────────────┘
│ │
│ WS push (1/sec) │ XCLAIM (stale jobs)
▼ ▼
┌─────────────────┐ ┌─────────────────────┐
│ React Dashboard │ │ Health Monitor │
│ :3000 │ │ (every 15s) │
└─────────────────┘ └─────────────────────┘
| Decision | Rationale |
|---|---|
| Redis Streams + XREADGROUP | Atomic consumer-group claiming prevents double-processing without distributed locks |
| Separate streams per priority | True priority isolation — high-priority stream is always checked first with no head-of-line blocking |
| Separate scheduler for retries | Decouples retry delays from worker processes; workers don't sleep |
| ZSET for scheduled retries | ZRANGEBYSCORE makes finding due-retries O(log N) regardless of queue size |
| Hash per job | Full metadata (retry count, worker ID, timestamps) persisted independently of stream entries |
| XAUTOCLAIM for reclaim | Redis 6.2+ atomic ownership transfer; falls back to XCLAIM for older Redis |
| Sliding window rate limiting | Two-bucket approximation gives accurate rate limiting without Lua scripting |
| Layer | Technology |
|---|---|
| Queue | Redis 7 — Streams, Sorted Sets, Hashes |
| Workers | Python 3.12, asyncio, plugin architecture |
| API | FastAPI 0.109, uvicorn, Pydantic v2 |
| Scheduler | asyncio daemon with ZSET scan |
| Dashboard | React 18, TypeScript, Recharts, WebSocket |
| Observability | structlog (JSON output) |
| Testing | pytest-asyncio, fakeredis |
| Packaging | Docker Compose — 1 API, 1 scheduler, 3 workers, 1 Redis |
┌─────────┐
enqueue │ PENDING │
──────► └────┬────┘
│ XREADGROUP (worker claims)
▼
┌─────────┐
│ RUNNING │
└────┬────┘
┌─────────┴──────────┐
│ success │ failure
▼ ▼
┌──────────┐ ┌────────┐
│COMPLETED │ │ FAILED │
└──────────┘ └───┬────┘
│ retry_count < max_retries
│ (after backoff: 30s/2m/10m)
▼
┌─────────┐
│ PENDING │ (re-enqueued)
└─────────┘
│ retry_count >= max_retries
▼
┌────────┐
│ DEAD │ → DLQ stream
└────────┘
│ manual replay
▼
┌─────────┐
│ PENDING │ (retry_count reset to 0)
└─────────┘
Special state:
WAITING — job has unresolved depends_on; promoted when all parents complete
- Docker and Docker Compose
- (Optional) Python 3.12+ for local development
git clone https://github.com/rohanmukka/JobForge.git
cd JobForge
cp .env.example .env
# Start all services
docker compose up -d
# Scale workers (if needed)
docker compose up -d --scale worker=5Services:
- API + Swagger UI: http://localhost:8000/docs
- Dashboard: http://localhost:3000
- Redis: localhost:6379
# Install Python dependencies
pip install -r requirements.txt
# Start Redis
docker run -d -p 6379:6379 redis:7-alpine
# Start the API (in one terminal)
python -m jobforge.api.main
# Start a worker (in another terminal)
python -m jobforge.worker.runner
# Start the scheduler (in another terminal)
python -m jobforge.scheduler.daemon
# Start the dashboard
cd dashboard && npm install && npm startBase URL: http://localhost:8000
POST /jobs
Content-Type: application/json
{
"job_type": "send_email",
"payload": {"to": "[email protected]", "subject": "Hello"},
"priority": "high",
"max_retries": 3,
"depends_on": []
}Response 201:
{
"id": "550e8400-e29b-41d4-a716-446655440000",
"job_type": "send_email",
"status": "pending",
"stream_id": "1705312800000-0",
...
}GET /jobs/{job_id}GET /jobs?status=running&limit=50Status values: pending, running, completed, failed, dead, waiting
GET /jobs/{job_id}/depsGET /workersReturns all registered workers with heartbeat timestamps, job counts, and liveness status.
GET /dlq?count=50POST /dlq/replay
{"dlq_stream_id": "1705312800000-0", "priority": "high"}DELETE /dlqGET /stats{
"queue": {
"high": {"queued": 5, "pending": 1},
"normal": {"queued": 23, "pending": 3},
"low": {"queued": 100, "pending": 0},
"total": {"queued": 128, "pending": 4}
},
"workers": {"total": 3, "alive": 3, "dead": 0},
"jobs": {"total": 1543, "by_status": {"completed": 1400, "failed": 43, ...}}
}# Configure rate limit
POST /rate-limits/send_email
{"limit": 100, "window_seconds": 60}
# View all limits
GET /rate-limits# Kill a random worker (demonstrates recovery)
POST /chaos/kill-worker
# Flood queue with N jobs
POST /chaos/flood
{"count": 50, "job_type": "noop", "priority": "low"}
# Add always_fail jobs to populate DLQ
POST /chaos/dlq-flood?count=5
# Enable automatic chaos (kills worker every 10s)
POST /chaos/enable
{"interval": 10}
# Stop chaos
POST /chaos/disableConnect to ws://localhost:8000/ws. Receives a push every second:
{
"type": "stats",
"timestamp": "2024-01-15T10:30:00.123Z",
"queue": { ... },
"workers": {
"total": 3,
"alive": 3,
"dead": 0,
"details": [...]
}
}from jobforge.queue.models import Job
from jobforge.worker.base_worker import Worker, WorkerContext
import redis.asyncio as aioredis
redis_client = get_redis()
worker = Worker(redis_client, concurrency=4)
@worker.register("send_email")
async def handle_send_email(job: Job, ctx: WorkerContext) -> None:
"""Handler must be an async function. Raise any exception to trigger retry."""
to = job.payload["to"]
subject = job.payload["subject"]
# ... send email
print(f"Email sent to {to}: {subject}")
# Run the worker
import asyncio
asyncio.run(worker.start())# All tests (uses fakeredis — no real Redis needed)
pytest
# With coverage
pytest --cov=jobforge --cov-report=html
# Specific test file
pytest tests/integration/test_retry_logic.py -vAll configuration via environment variables (see .env.example):
| Variable | Default | Description |
|---|---|---|
REDIS_URL |
redis://localhost:6379/0 |
Redis connection URL |
WORKER_CONCURRENCY |
4 |
Async slots per worker process |
WORKER_HEARTBEAT_INTERVAL |
10 |
Seconds between heartbeats |
WORKER_VISIBILITY_TIMEOUT |
60 |
Seconds before job is reclaimed |
WORKER_DEAD_THRESHOLD |
30 |
Seconds without HB = worker dead |
RETRY_DELAYS |
[30, 120, 600] |
Backoff schedule in seconds |
RATE_LIMIT_DEFAULT |
100 |
Default jobs/window per type |
RATE_LIMIT_WINDOW |
60 |
Window duration in seconds |
SCHEDULER_POLL_INTERVAL |
5 |
Seconds between retry scans |
LOG_FORMAT |
json |
json or pretty |
JobForge/
├── jobforge/
│ ├── config.py # Pydantic-settings configuration
│ ├── logging_config.py # structlog JSON/pretty setup
│ ├── chaos.py # Chaos injection module
│ ├── queue/
│ │ ├── models.py # Job, JobSpec, Priority, JobStatus
│ │ ├── connection.py # Redis connection pool
│ │ ├── job_queue.py # Core: enqueue, claim, ack, list
│ │ ├── priority_queue.py # Three-lane priority system + stats
│ │ ├── dlq.py # Dead-letter queue operations
│ │ ├── dependency_manager.py # Job dependency resolution
│ │ └── rate_limiter.py # Sliding-window rate limiting
│ ├── worker/
│ │ ├── base_worker.py # Async worker with plugin registry
│ │ ├── heartbeat.py # Worker registration + heartbeat
│ │ ├── retry_engine.py # Exponential backoff retry logic
│ │ ├── health_monitor.py # XAUTOCLAIM dead-worker recovery
│ │ └── runner.py # Worker process entry point
│ ├── api/
│ │ ├── main.py # FastAPI app factory + lifespan
│ │ ├── routes.py # All REST + WebSocket endpoints
│ │ ├── models.py # API request/response models
│ │ └── websocket_manager.py # WS connection manager + broadcaster
│ ├── scheduler/
│ │ └── daemon.py # Retry re-enqueue scheduler
│ └── plugins/
│ └── builtin.py # noop, sleep, fail_once, always_fail, echo
├── dashboard/ # React + TypeScript frontend
│ ├── src/
│ │ ├── App.tsx # Main layout
│ │ ├── types.ts # Shared TypeScript types
│ │ ├── hooks/
│ │ │ ├── useWebSocket.ts # WS hook with auto-reconnect
│ │ │ └── useApi.ts # Typed fetch wrapper
│ │ └── components/
│ │ ├── StatusBadge.tsx
│ │ ├── MetricCard.tsx
│ │ ├── QueueDepthChart.tsx # Recharts AreaChart (60s rolling)
│ │ ├── WorkerTable.tsx
│ │ └── DLQViewer.tsx
│ └── Dockerfile
├── tests/
│ ├── conftest.py # fakeredis fixtures
│ ├── unit/
│ │ └── test_models.py # Job model unit tests
│ └── integration/
│ ├── test_queue_operations.py
│ ├── test_retry_logic.py
│ ├── test_priority_ordering.py
│ ├── test_dependencies.py
│ └── test_rate_limiting.py
├── config/
│ └── redis.conf # Redis config (AOF, maxmemory, etc.)
├── docker-compose.yml
├── Dockerfile
├── requirements.txt
└── pyproject.toml
MIT