A durable task execution system. Persistent workers, multiprocess isolation, comprehensive observability.
$ weft run echo "hello world"
hello world
$ weft run --spec task.json --wait
Task completed: 1234567890
$ weft status
Manager: Running (PID 1234, idle timeout: 600s)
Tasks: 3 total (2 running, 1 completed)Weft is a queue-based task execution system focused on enabling interaction between AI agents, user-provided functions, and existing CLI tools. It combines the simplicity of direct command execution with the power of durable task queues, multiprocess isolation, and comprehensive state tracking.
# Initialize project
$ weft init
Initialized weft in .weft/
# Run a simple command
$ weft run echo "hello world"
hello world
# Run with resource limits
$ weft run --memory 100 --cpu 50 python script.py
# Run and wait for completion
$ weft run --wait --timeout 30 ./long-task.sh
# Run Python function
$ weft run --function mymodule:process_data --arg input.csv
# Check system status
$ weft status
Manager: Running (PID 12345)
Tasks: 5 total (3 running, 2 completed)
# Get task result
$ weft result 1234567890
{"status": "completed", "return_code": 0, "output": "..."}Weft uses .weft/ directories for project isolation, similar to git repositories:
myproject/
.weft/
broker.db # SimpleBroker database
autostart/ # Auto-starting task templates
outputs/ # Large output spillover
logs/ # Centralized logging
...
Run weft commands from anywhere in the project tree - it searches upward to find .weft/.
Every task receives a unique 19-digit TID:
- Full TID:
1837025672140161024(Unique task ID) - Short TID:
0161024(last 10 digits for convenience) - Used for correlation across queues and process titles
- Chronologically ordered, format-compatible with time.time_ns()
Each task gets its own queues:
T{tid}.inbox # Work messages to process
T{tid}.reserved # Messages being processed (reservation pattern)
T{tid}.outbox # Results and output
T{tid}.ctrl_in # Control commands (STOP, PAUSE, etc.)
T{tid}.ctrl_out # Status responses
weft.tasks.log # Global state log (all tasks)
weft.spawn.requests # Task spawn requests to manager
weft.workers.registry # Manager liveness tracking
Weft implements inbox -> reserved -> outbox flow for reliable message processing:
- Reserve: Move message from inbox to reserved
- Process: Execute work while message is in reserved
- Complete: Write output to outbox, delete from reserved (or apply policy)
Configurable policies (keep, requeue, clear) control reserved queue behavior on errors.
Persistent worker processes that:
- Monitor
weft.spawn.requestsfor new tasks - Launch child task processes
- Track process lifecycle
- Auto-terminate after idle timeout (default 600 seconds)
- Launch autostart tasks on boot
Tasks update their process title for observability:
$ ps aux | grep weft
weft-proj-0161024:mytask:running
weft-proj-0161025:worker:completedFormat: weft-{context}-{short_tid}:{name}:{status}
# Initialize new project
weft init [--autostart/--no-autostart]
# Show system status
weft status [TID] [--all] [--json] [--watch]
# Dump/load database
weft dump [-o FILE]
weft load [-i FILE]# Run command
weft run COMMAND [args...]
weft run --spec taskspec.json
weft run --function module:func [--arg VALUE] [--kw KEY=VALUE]
# Execution options
--wait # Wait for completion
--timeout N # Timeout in seconds
--memory N # Memory limit in MB
--cpu N # CPU limit (percentage)
--interactive, -i # Interactive mode (stdin/stdout)
--stream-output # Stream output in real-time
--tag KEY=VALUE # Add metadata tags
# Get results
weft result TID [--timeout N] [--stream] [--json]
weft result --all [--peek]# Direct queue access
weft queue read QUEUE [--json] [--all]
weft queue write QUEUE MESSAGE
weft queue peek QUEUE [--json] [--all]
weft queue move SOURCE DEST [--all]
weft queue list [--pattern PATTERN]
weft queue watch QUEUE [--json] [--peek]
# Broadcast and aliases
weft queue broadcast MESSAGE [--pattern GLOB]
weft queue alias add ALIAS TARGET
weft queue alias remove ALIAS
weft queue alias list [--target QUEUE]Template files in .weft/autostart/*.json are automatically launched when the manager starts:
# Create autostart template
$ cat > .weft/autostart/monitor.json <<EOF
{
"name": "queue-monitor",
"spec": {
"type": "function",
"function_target": "monitoring.watch_queues",
"timeout": null
},
"io": {
"inputs": {},
"outputs": {"metrics": "monitoring.queue.metrics"}
}
}
EOF
# Next manager start will launch it automatically
$ weft run --wait echo "trigger manager"Control autostart behavior:
weft init --no-autostart- Skip autostart directory creationweft run --no-autostart- Skip launching autostart tasksWEFT_AUTOSTART_TASKS=false- Disable via environment
Tasks are configured with JSON specifications:
{
"name": "process-data",
"spec": {
"type": "command",
"process_target": ["python", "process.py"],
"timeout": 300,
"limits": {
"memory_mb": 512,
"cpu_percent": 75,
"max_fds": 100
},
"env": {"LOG_LEVEL": "debug"},
"stream_output": true,
"cleanup_on_exit": true
},
"io": {
"inputs": {"data": "input.queue"},
"outputs": {"results": "output.queue"}
},
"metadata": {
"tags": ["production", "critical"]
}
}Spec fields:
type:"command"or"function"process_target: Command array (for commands)function_target: Module:function string (for functions)timeout: Seconds (null for no timeout)limits: Resource constraintsenv: Environment variablesstream_output: Enable output streamingcleanup_on_exit: Delete queues on completion
IO fields:
inputs: Map of name -> queue for readingoutputs: Map of name -> queue for writing
All state changes are logged to weft.tasks.log:
{
"event": "work_completed",
"tid": "1837025672140161024",
"tid_short": "0161024",
"status": "completed",
"timestamp": 1705329000123456789,
"taskspec": {...},
"task_pid": 12345,
"return_code": 0
}Events include:
task_initialized- Task startupwork_started- Processing beginswork_completed- Successwork_failed- Execution errorwork_timeout- Timeout exceededwork_limit_violation- Resource limit hitcontrol_*- Control message handling
Tasks track resource usage with psutil:
# Resource limits in TaskSpec
"limits": {
"memory_mb": 512, # Max memory
"cpu_percent": 75, # Max CPU (0-100)
"max_fds": 100, # Max file descriptors
"max_connections": 50 # Max network connections
}Violations trigger work_limit_violation events and task termination.
0 - Success
1 - General error
2 - Not found / Queue empty
124 - Timeout# Launch task without waiting
$ weft run ./background-job.sh
# Check status later
$ weft status
Tasks: 1 running
# Get result when ready
$ weft result <tid># Task chain via queues
$ weft run --spec extract.json # Writes to "raw.data"
$ weft run --spec transform.json # Reads "raw.data", writes "clean.data"
$ weft run --spec load.json # Reads "clean.data"
# Or use queue operations directly
$ weft queue write input.queue "data.csv"
$ weft run --spec processor.json
$ weft queue read output.queue# Create autostart watcher
$ cat > .weft/autostart/file-watcher.json <<EOF
{
"name": "file-watcher",
"spec": {
"type": "function",
"function_target": "watchers.watch_directory",
"timeout": null
}
}
EOF
# Starts automatically with manager
$ weft run echo "start"
# Verify running
$ ps aux | grep weft
weft-proj-1234567:file-watcher:running# Limit memory and CPU
$ weft run --memory 100 --cpu 25 ./memory-intensive.py
# With timeout
$ weft run --timeout 60 --memory 500 ./task.sh- TaskSpec: Validated task configuration with partial immutability
- Manager: Persistent worker process for task spawning
- Consumer: Task executor with reservation pattern
- BaseTask: Abstract base providing queue wiring and state tracking
- TaskRunner: Multiprocess execution wrapper with timeout/monitoring
- ResourceMonitor: psutil-based resource tracking and limit enforcement
1. CLI: weft run COMMAND
2. Manager auto-started if needed
3. TaskSpec created and validated
4. Queued to weft.spawn.requests
5. Manager spawns Consumer process
6. Consumer reserves work from inbox
7. TaskRunner executes in child process
8. Output written to outbox
9. State logged to weft.tasks.log
10. CLI retrieves result
Tasks execute in separate processes using multiprocessing.spawn:
- Clean process environment
- No inherited state from parent
- Resource monitoring per process
- Crash isolation
- Timeout enforcement
# Install development dependencies
uv sync --all-extras
# Run tests
uv run pytest
uv run pytest tests/tasks/
uv run pytest tests/cli/
# Linting
uv run ruff check weft tests
uv run ruff format weft tests
uv run mypy weft
# Build
uv buildEnvironment variables:
WEFT_MANAGER_LIFETIME_TIMEOUT- Manager idle timeout (default: 600s)WEFT_MANAGER_REUSE_ENABLED- Keep manager running (default: true)WEFT_AUTOSTART_TASKS- Enable autostart (default: true)
MIT