local-llm-orchestrator

Local OpenAI-compatible LLM orchestrator for servers with several GPUs and several services using local models.

The goal is to make one controlled entry point for all internal LLM traffic:

• Queue proxy is the public API endpoint for internal services.
• Queue proxy controls per-model concurrency, queue size, queue timeout, and token budget.
• Queue proxy accepts durable task queues, executes stored OpenAI-compatible payloads, and can route through the lifecycle backend registry.
• LiteLLM handles OpenAI-compatible routing and provider abstraction.
• LM Studio runs locally on the host and serves the model for the first backend.
• Postgres and Redis are available for LiteLLM state.
• Healthcheck verifies LM Studio and the full queue proxy -> LiteLLM -> backend path.
• GPU inventory exposes GPU/VRAM state for scheduling.
• Lifecycle service plans GPU placement, starts/warms/stops owned backends, reconciles live LM Studio state, and explains placement/reload decisions.
• Prometheus and Grafana are wired for service metrics.

The first backend is LM Studio because it is convenient locally. For heavier multi-GPU serving, the intended migration path is to keep this orchestrator and replace or extend the backend with vLLM/SGLang instances.

Architecture

Service A / Service B / OpenAI SDK compatible client
  -> Queue Proxy :4100
      -> per-model token budget
      -> per-model queue
      -> per-model active request limiter
      -> optional durable task store/executor
      -> Lifecycle registry/allocation :4300
      -> ready backend or LiteLLM Proxy :4000
          -> LM Studio OpenAI-compatible API on the host :1234
          -> Postgres
          -> Redis
  -> GPU Inventory :4200
      -> nvidia-smi / fake inventory
  -> Lifecycle :4300
      -> scheduler
      -> backend registry
      -> Healthcheck service :8080
      -> Prometheus :9090
      -> Grafana :3000

LM Studio is not exposed by this compose file. It should run on the host and be reachable from Docker through host.docker.internal.

LiteLLM is still published on :4000 for debugging, but internal services should use the queue proxy on :4100.

Prerequisites

• Docker Desktop or Docker Engine with Compose.
• NVIDIA Container Toolkit / Docker GPU passthrough for real vLLM GPU serving.
• LM Studio installed.
• A local model downloaded in LM Studio.
• Optional: LM Studio CLI lms.

For a repeatable local preparation run that installs Python dependencies, pulls Docker images, builds services, runs tests, and verifies GPU passthrough, see Real Run Preparation.

If you already have models downloaded in LM Studio, use LM Studio Models to discover model ids and map them into the orchestrator.

All callers must use the strict cross-project request envelope and concurrency semantics described in the Unified Task Protocol.

The roadmap for durable task queues, context planning, LM Studio shape selection, and graceful reloads is in Task Context Orchestration Implementation Plan.

Applications can request any allowed LM Studio model dynamically through the queue proxy; see Dynamic Model Allocation.

Dynamic smoke test:

docker compose up -d --build queue-proxy
powershell.exe -NoProfile -ExecutionPolicy Bypass -File .\scripts\smoke_dynamic_allocation.ps1

CLI:

.\.venv\Scripts\pip install -e ".[dev]"
llmoctl models
llmoctl allocate mistralai/ministral-3-3b --gpu auto --lms-gpu max
llmoctl chat mistralai/ministral-3-3b "Return exactly: ok" --max-tokens 8
llmoctl tasks --tenant elvis --state queued
llmoctl task task_123 --tenant elvis
llmoctl explain-plan --tenant elvis
llmoctl cleanup

Configure LM Studio

Start the LM Studio server on port 1234.

With the desktop app, enable the local server from the Developer/API panel.

With the CLI:

lms server start --port 1234
lms ls
lms load <model-key> --identifier local-main

Use the model identifier you load as LMSTUDIO_MODEL_ID.

Dynamic LM Studio profiles can also let lifecycle call lms load <model-key> --identifier <model-key> --yes and lms unload <model-key> automatically. This works when lifecycle runs where the lms CLI is available.

In Docker on Windows the Linux lifecycle container cannot execute the host lms.exe directly (it is host-bound and has no --host option). Without help, cli-if-available falls back to already reachable LM Studio API models, and LM Studio then JIT-loads at its default context (e.g. 4096), ignoring the profile's lms_context_length. To make lifecycle load with the profile context, run the host lms bridge and let the container shim forward lms ... calls to it:

# On the host (keep running; add to Task Scheduler at logon for persistence):
$env:LMS_BINARY="C:\Users\<you>\.lmstudio\bin\lms.exe"
python services\lms_bridge\lms_bridge.py --port 4399

docker-compose.yml mounts services/lms_bridge/lms-shim as the container's /usr/local/bin/lms, sets LMS_BRIDGE_URL=http://host.docker.internal:4399, and defaults LIFECYCLE_DRY_RUN=false. The lifecycle load logic is unchanged: it still runs lms load --context-length <N> --parallel <M>; the shim transparently executes it on the host, so a cold allocation loads with the configured context (verified at 32768/parallel 2 for zotero-html-translate). The bridge only allows whitelisted lms subcommands.

To discover already downloaded models:

powershell.exe -NoProfile -ExecutionPolicy Bypass -File .\scripts\discover_lmstudio_models.ps1

Configure This Orchestrator

Create a local environment file:

Copy-Item .env.example .env

Edit .env and set at least:

LMSTUDIO_MODEL_ID=local-main
LITELLM_MODEL=openai/local-main
LITELLM_MASTER_KEY=sk-change-this-local-key

Model orchestration policy lives in:

config/orchestrator.yaml

For each public model you can set:

• max_active_requests: how many requests may run at once.
• max_queued_requests: how many requests may wait.
• queue_timeout_seconds: how long a request may wait for a slot.
• default_max_output_tokens: output budget when the caller does not specify one.
• max_input_tokens: maximum estimated input size.
• max_output_tokens: maximum output budget.
• max_total_tokens: input estimate plus output budget.
• lifecycle.estimated_vram_gb: VRAM reservation for scheduler placement.
• lifecycle.safety_margin_gb: extra VRAM headroom.
• lifecycle.preferred_gpus: auto or explicit GPU ids such as gpu0.
• lifecycle.min_replicas / lifecycle.max_replicas: desired model replica bounds.

For Docker Desktop on Windows and macOS, the default backend URL usually works:

LMSTUDIO_OPENAI_BASE_URL=http://host.docker.internal:1234/v1

On Linux, keep the compose extra_hosts entry or set a host address such as:

LMSTUDIO_OPENAI_BASE_URL=http://172.17.0.1:1234/v1

Start

docker compose up -d --build

Useful URLs:

• Queue proxy: http://localhost:4100
• LiteLLM debug endpoint: http://localhost:4000
• GPU inventory: http://localhost:4200/gpus
• Lifecycle registry: http://localhost:4300/registry
• Healthcheck: http://localhost:8080/ready
• Metrics: http://localhost:8080/metrics
• Prometheus: http://localhost:9090
• Grafana: http://localhost:3000

Smoke Test

PowerShell:

.\scripts\smoke_test.ps1

Bash:

./scripts/smoke_test.sh

Expected result: HTTP 200 from LiteLLM and a valid OpenAI-compatible chat completion.

The smoke test goes through the queue proxy by default.

Runtime Behavior

For a request to /v1/chat/completions, /v1/responses, /v1/completions, or /v1/embeddings, the queue proxy:

1. Reads the requested model.
2. Resolves model policy from config/orchestrator.yaml.
3. Estimates input tokens with a configurable chars-per-token heuristic.
4. Sets a default output token limit if the caller omitted one.
5. Clamps oversized output token requests unless the policy says to reject.
6. Rejects too-large input or total token budget with 413.
7. Admits the request into the per-model queue.
8. Rejects queue overflow or queue timeout with 429.
9. Resolves a ready backend through lifecycle when registry routing is enabled.
10. Falls back to the configured LiteLLM upstream when policy allows fallback.

This gives immediate protection when several internal services call the same local model at the same time.

For long-running batch work, clients should prefer POST /tasks/queue. Queue proxy stores tenant-scoped tasks, renders employer-owned payload templates, claims work fairly between employer groups, records results/errors, and asks lifecycle to reconcile capacity from the queue's context plan. When the durable queue becomes empty, queue proxy schedules a delayed reconcile so lifecycle can unload owned idle LM Studio models after the configured TTL.

GPU Control Plane

The GPU management layer can run in either dry-run or real Docker mode. Dry-run mode is the default and records planned backend instances without starting containers. Real Docker mode is enabled with LIFECYCLE_DRY_RUN=false and uses the Docker vLLM adapter.

For the external contract other programs should use to submit LLM work, request model startup, and pass GPU/task constraints, see the Unified Task Protocol. Lower-level lifecycle allocation details are in Resource Request API.

The implemented dynamic request path is documented in Dynamic Model Allocation.

For real vLLM container launching, Docker socket deployment, healthcheck/warmup, and idle stop behavior, see Docker vLLM Runtime Adapter.

For the current module boundaries and refactoring map, see Architecture Notes.

GPU inventory:

Invoke-RestMethod http://localhost:4200/gpus

Lifecycle placement plan:

Invoke-RestMethod `
  -Method Post `
  -Uri http://localhost:4300/plan `
  -ContentType "application/json" `
  -Body '{"queue_lengths":{"local-main":1}}'

Dry-run reconcile creates a registry entry for the planned backend instance:

Invoke-RestMethod `
  -Method Post `
  -Uri http://localhost:4300/reconcile `
  -ContentType "application/json" `
  -Body '{"queue_lengths":{"local-main":1}}'

On a machine without NVIDIA drivers, set GPU_INVENTORY_FAKE_GPU_INVENTORY_JSON in .env:

{"gpus":[{"id":"gpu0","index":0,"name":"fake","memory_total_mb":24576,"memory_used_mb":2048}]}

Queue proxy can use ready HTTP backends from the lifecycle registry:

ENABLE_BACKEND_REGISTRY_ROUTING=true

When registry routing is enabled and no ready backend exists for a requested dynamic model, queue proxy calls lifecycle POST /allocations before forwarding the request.

For LM Studio dynamic models, lifecycle can auto-estimate VRAM from lms ls --json metadata. A caller may still send orchestration.estimated_vram_gb to override that reservation for a specific task.

By default it falls back to UPSTREAM_LITELLM_BASE_URL when the registry has no ready HTTP backend. To force registry-only routing:

REQUIRE_BACKEND_REGISTRY_BACKEND=true

When registry routing is enabled, queue proxy leases the selected backend before forwarding the request and releases it when the upstream response finishes. That keeps active_requests in the lifecycle registry current enough for least-active routing decisions.

The lifecycle service now has a runtime adapter layer. Dry-run mode records the command that would be used. For a vLLM model profile, the generated command is shaped like:

docker run -d `
  --name llm-<instance> `
  --gpus device=0 `
  -p 8100:8000 `
  vllm/vllm-openai:latest `
  --model /models/qwen `
  --served-model-name qwen `
  --host 0.0.0.0 `
  --port 8000

Real Docker launching is intentionally opt-in:

LIFECYCLE_DRY_RUN=false

For real container launching, the lifecycle image includes the Docker CLI and the compose file mounts /var/run/docker.sock. A production vLLM profile should include explicit model volumes and runtime settings:

models:
  qwen-14b:
    public_name: qwen-14b
    backend_model: qwen-14b
    lifecycle:
      runtime: vllm
      artifact: D:/models/qwen-14b
      runtime_image: vllm/vllm-openai:latest
      host_port_start: 8100
      container_port: 8000
      public_host: host.docker.internal
      volumes:
        - host_path: D:/models/qwen-14b
          container_path: /models/qwen-14b
          mode: ro
      environment:
        HF_HOME: /root/.cache/huggingface
      runtime_extra_args:
        - --max-model-len
        - "8192"
      healthcheck_path: /v1/models
      startup_timeout_seconds: 120
      healthcheck_interval_seconds: 2
      warmup_enabled: true
      warmup_prompt: "Return exactly: ok"
      warmup_max_tokens: 8
      estimated_vram_gb: 16
      safety_margin_gb: 2
      min_replicas: 0
      max_replicas: 2
      idle_ttl_seconds: 120
      preferred_gpus: [gpu0, gpu1]

When LIFECYCLE_DRY_RUN=false, lifecycle starts the container, waits for /v1/models, sends a warmup chat completion, then marks the backend ready. Idle ready instances above min_replicas are marked draining, stopped with docker stop, and then marked stopped. Use min_replicas: 0 with idle_ttl_seconds: 120 when the model should unload after one to two idle minutes.

Lifecycle also exposes POST /cleanup and :4300/metrics for LM Studio/vLLM backend operations. Queue proxy schedules a delayed reconcile after durable queues become empty, so owned LM Studio loads can be unloaded after the idle TTL without manual cleanup. Cleanup stops idle dynamic LM Studio allocations and purges old stopped/failed LM Studio records after dynamic_models.registry_cleanup_ttl_seconds.

Development

Run the healthcheck service locally:

python -m venv .venv
.\.venv\Scripts\pip install -e ".[dev]"
.\.venv\Scripts\uvicorn app.main:app --app-dir services/healthcheck --reload --port 8080

Run tests:

python -m pytest

Integration tests start a fake OpenAI-compatible backend and a real queue proxy on temporary local ports:

python -m pytest tests\integration

You can also start the fake backend through Compose for manual debugging:

docker compose --profile test up -d --build fake-backend

Current Scope

Implemented now:

• Compose and LiteLLM configuration.
• Queue proxy for per-model concurrency, queueing, token budget enforcement, and strict llmo.task.v1 validation.
• Durable task queue admission through POST /tasks/queue, tenant-scoped task status/cancel APIs, and optional in-process task execution.
• Employer-provided payload_template rendering for durable tasks.
• Equal-priority fair task claiming across (tenant, project, service, task, priority, model) groups.
• Fake OpenAI-compatible backend for integration tests.
• Integration tests for non-streaming, streaming, token rejection, queue overflow, queue timeout, and upstream failure.
• GPU inventory service with nvidia-smi parser and fake inventory mode.
• Lifecycle scheduler with backend registry, VRAM-aware placement, context-plan driven LM Studio shape selection, and plan explanation.
• Registry-aware queue proxy routing.
• Active request lease/release accounting between queue proxy and lifecycle registry.
• Lifecycle runtime adapter framework with Docker vLLM command generation.
• Lifecycle support for already-running LM Studio/OpenAI-compatible backends.
• Dynamic model allocation from request payloads through lifecycle POST /allocations.
• LM Studio dynamic loading/unloading through lms load/unload when the CLI is available.
• LM Studio VRAM auto-estimation from lms ls --json metadata.
• LM Studio live-state reconciliation, external-load reservation, reload hysteresis, and idle unload after empty queues.
• llmoctl CLI for model catalog, registry, allocation, chat, embeddings, durable task status/cancel, plan explanation, cleanup, and metrics.
• Postgres durable task store with startup schema version checks.
• Prometheus metrics for lifecycle GPU/model/allocation/reload state and queue proxy queue/request state.
• Registry cleanup TTL for old LM Studio allocation records.
• Production-oriented Docker vLLM lifecycle: model volumes, Docker socket/CLI launch, healthcheck, warmup, starting -> ready, and idle drain/stop.
• Real-run preparation script and documentation.
• Environment-driven settings.
• Smoke test scripts.
• Basic FastAPI healthcheck with Prometheus metrics.

Next phases:

• Formal Postgres migration tooling and real-container multi-worker tests.
• Dashboards and richer structured task/reload logs.
• Zotero worker-side migration to submit executable payloads/templates.
• Optional SGLang runtime adapter.
• Additional compatibility coverage for Responses API edge cases and provider-specific backend failures.
• Reverse proxy and TLS for controlled non-local access.