The Privacy-First Alternative to Ollama

🔒 Local AI Without the Lock-in 🚀

Shimmy will be free forever. No asterisks. No "free for now." No pivot to paid.

Shimmy is a 5.1MB single-binary that provides 100% OpenAI-compatible endpoints for GGUF models. Point your existing AI tools to Shimmy and they just work — locally, privately, and free.

# 1) Install + run
cargo install shimmy --features huggingface
shimmy serve &

# 2) See models and pick one
shimmy list

# 3) Smoke test the OpenAI API
curl -s http://127.0.0.1:11435/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
        "model":"REPLACE_WITH_MODEL_FROM_list",
        "messages":[{"role":"user","content":"Say hi in 5 words."}],
        "max_tokens":32
      }' | jq -r '.choices[0].message.content'

No code changes needed - just change the API endpoint:

• VSCode Extensions: Point to http://localhost:11435
• Cursor Editor: Built-in OpenAI compatibility
• Continue.dev: Drop-in model provider
• Any OpenAI client: Python, Node.js, curl, etc.

• Node.js (openai v4)

import OpenAI from "openai";

const openai = new OpenAI({
  baseURL: "http://127.0.0.1:11435/v1",
  apiKey: "sk-local", // placeholder, Shimmy ignores it
});

const resp = await openai.chat.completions.create({
  model: "REPLACE_WITH_MODEL",
  messages: [{ role: "user", content: "Say hi in 5 words." }],
  max_tokens: 32,
});

console.log(resp.choices[0].message?.content);

• Python (openai>=1.0.0)

from openai import OpenAI

client = OpenAI(base_url="http://127.0.0.1:11435/v1", api_key="sk-local")

resp = client.chat.completions.create(
    model="REPLACE_WITH_MODEL",
    messages=[{"role": "user", "content": "Say hi in 5 words."}],
    max_tokens=32,
)

print(resp.choices[0].message.content)

• Auto-discovers models from Hugging Face cache, Ollama, local dirs
• Auto-allocates ports to avoid conflicts
• Auto-detects LoRA adapters for specialized models
• Just works - no config files, no setup wizards

• Privacy: Your code never leaves your machine
• Cost: No API keys, no per-token billing
• Speed: Local inference, sub-second responses
• Reliability: No rate limits, no downtime

# RECOMMENDED: Use pre-built binary (no build dependencies required)
curl -L https://github.com/Michael-A-Kuykendall/shimmy/releases/latest/download/shimmy.exe -o shimmy.exe

# OR: Install from source (requires LLVM/Clang)
# First install build dependencies:
winget install LLVM.LLVM
# Then install shimmy:
cargo install shimmy --features huggingface

⚠️ Windows Notes:

• Pre-built binary recommended to avoid build dependency issues
• If Windows Defender flags the binary, add an exclusion or use cargo install
• For cargo install: Install LLVM first to resolve libclang.dll errors

# Install from crates.io
cargo install shimmy --features huggingface

Shimmy auto-discovers models from:

• Hugging Face cache: ~/.cache/huggingface/hub/
• Ollama models: ~/.ollama/models/
• Local directory: ./models/
• Environment: SHIMMY_BASE_GGUF=path/to/model.gguf

# Download models that work out of the box
huggingface-cli download microsoft/Phi-3-mini-4k-instruct-gguf --local-dir ./models/
huggingface-cli download bartowski/Llama-3.2-1B-Instruct-GGUF --local-dir ./models/

# Auto-allocates port to avoid conflicts
shimmy serve

# Or use manual port
shimmy serve --bind 127.0.0.1:11435

Point your AI tools to the displayed port — VSCode Copilot, Cursor, Continue.dev all work instantly.

• Rust: cargo install shimmy
• VS Code: Shimmy Extension
• npm: npm install -g shimmy-js (coming soon)
• Python: pip install shimmy (coming soon)

• GitHub Releases: Latest binaries
• Docker: docker pull shimmy/shimmy:latest (coming soon)

Full compatibility confirmed! Shimmy works flawlessly on macOS with Metal GPU acceleration.

# Install dependencies
brew install cmake rust

# Install shimmy
cargo install shimmy

✅ Verified working:

• Intel and Apple Silicon Macs
• Metal GPU acceleration (automatic)
• Xcode 17+ compatibility
• All LoRA adapter features

{
  "github.copilot.advanced": {
    "serverUrl": "http://localhost:11435"
  }
}

{
  "models": [{
    "title": "Local Shimmy",
    "provider": "openai", 
    "model": "your-model-name",
    "apiBase": "http://localhost:11435/v1"
  }]
}

Works out of the box - just point to http://localhost:11435/v1

I built Shimmy to retain privacy-first control on my AI development and keep things local and lean.

This is my commitment: Shimmy stays MIT licensed, forever. If you want to support development, sponsor it. If you don't, just build something cool with it.

Shimmy saves you time and money. If it's useful, consider sponsoring for $5/month — less than your Netflix subscription, infinitely more useful.

• GET /health - Health check
• POST /v1/chat/completions - OpenAI-compatible chat
• GET /v1/models - List available models
• POST /api/generate - Shimmy native API
• GET /ws/generate - WebSocket streaming

shimmy serve                    # Start server (auto port allocation)
shimmy serve --bind 127.0.0.1:8080  # Manual port binding
shimmy list                     # Show available models  
shimmy discover                 # Refresh model discovery
shimmy generate --name X --prompt "Hi"  # Test generation
shimmy probe model-name         # Verify model loads

• Rust + Tokio: Memory-safe, async performance
• llama.cpp backend: Industry-standard GGUF inference
• OpenAI API compatibility: Drop-in replacement
• Dynamic port management: Zero conflicts, auto-allocation
• Zero-config auto-discovery: Just works™

• 🐛 Bug Reports: GitHub Issues
• 💬 Discussions: GitHub Discussions
• 📖 Documentation: docs/ • Engineering Methodology • OpenAI Compatibility Matrix • Benchmarks (Reproducible)
• 💝 Sponsorship: GitHub Sponsors

📦 5 MB single binary
🌟 stars and climbing fast
⏱ <1s startup
🦀 100% Rust, no Python

🔥 Hacker News • Front Page Again • IPE Newsletter

🚀 If Shimmy helps you, consider sponsoring — 100% of support goes to keeping it free forever.

Sponsorship Tiers:

• $5/month: Coffee tier - My eternal gratitude + sponsor badge
• $25/month: Bug prioritizer - Priority support + name in SPONSORS.md
• $100/month: Corporate backer - Logo on README + monthly office hours
• $500/month: Infrastructure partner - Direct support + roadmap input

🎯 Become a Sponsor | See our amazing sponsors who make Shimmy possible! 🙏

Companies: Need invoicing? Email [email protected]

Shimmy maintains high code quality through comprehensive testing:

• Comprehensive test suite with property-based testing
• Automated CI/CD pipeline with quality gates
• Runtime invariant checking for critical operations
• Cross-platform compatibility testing

See our testing approach for technical details.

MIT License - forever and always.

Philosophy: Infrastructure should be invisible. Shimmy is infrastructure.

Testing Philosophy: Reliability through comprehensive validation and property-based testing.

Forever maintainer: Michael A. Kuykendall
Promise: This will never become a paid product
Mission: Making local AI development frictionless