Queue Chat

A chat app where every message is a Celery task on RabbitMQ, answered by an offline LLM running in Ollama. The UI shows your request's live position in the queue counting down, then Generating response…, then the answer streaming in token by token.

Everything runs locally — no API keys, no network calls after the first model pull.

docker compose up

Then open http://localhost:8000.

The first run downloads the model (qwen2.5:0.5b, ~400 MB) into a Docker volume; the workers wait for that to finish, so the UI may report "no workers online yet" for a minute. Subsequent runs start in seconds.

What to try

1. Send a message. With an idle queue it goes straight to Generating response….
2. Hit Flood queue +25 in the right-hand panel, then immediately send a message. Your bubble shows position 26, 25, 24… ticking down as three workers chew through the backlog, then flips to generating and streams the reply.
3. Watch the Workers panel: each of the three replicas lights up green with the prompt it is currently working on.
4. Reset everything purges the RabbitMQ backlog, terminates jobs the workers are mid-way through, and deletes all stored task results from Redis — both the app's own queue state and Celery's result backend, which is what AsyncResult and Flower read. Flower's task list is a separate in-memory event log; clear it with docker compose restart flower.

The flood button enqueues real (short) prompts, not fake sleeps — the workers are genuinely busy, which is why your message actually has to wait.

Ports

URL	What
http://localhost:8000	Chat UI + JSON API (/api/docs)
http://localhost:5555	Flower — Celery task/worker monitor
http://localhost:15672	RabbitMQ management UI (guest / guest)
http://localhost:11434	Ollama API

Services

Service	Role
rabbitmq	Broker. Holds the task messages.
redis	Celery result backend + the queue-position bookkeeping the UI reads.
ollama	The offline model server.
ollama-pull	One-shot: pulls the model, then exits. Workers gate on it.
web	FastAPI — serves the UI, publishes tasks, pushes state over a WebSocket.
worker	3 replicas of a Celery worker, --concurrency=1 --prefetch-multiplier=1.
flower	Celery monitoring UI.

Knobs

Set these in the environment or a .env file next to docker-compose.yml:

OLLAMA_MODEL=smollm2:135m     # any Ollama tag; this one is the smallest (~270 MB)
WORKER_REPLICAS=5             # more consumers drain the queue faster
OLLAMA_NUM_PREDICT=192        # token budget for chat replies
OLLAMA_FILLER_NUM_PREDICT=64  # token budget for flood-button jobs
WEB_PORT=8000
FLOWER_PORT=5555

To scale workers without editing anything:

docker compose up -d --scale worker=6

WORKER_REPLICAS also sets OLLAMA_NUM_PARALLEL, so Ollama serves as many concurrent generations as there are workers rather than quietly re-queueing them.

Useful commands

docker compose logs -f worker      # watch tasks being consumed
docker compose ps                  # service health
curl localhost:8000/api/queue      # queue snapshot as JSON
curl -X POST localhost:8000/api/flood -H 'content-type: application/json' -d '{"count":40}'
docker compose down                # stop (keeps the model volume)
docker compose down -v             # stop and delete the downloaded model

GPU (optional)

CPU is fine at this model size. To use an NVIDIA GPU, add to the ollama service:

    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]

See ARCHITECTURE.md for how the queue position is tracked.