AI Makerspace Technical Report - Comprehensive documentation of integrating 10 advanced AI techniques (Vector Memory, Multi-Agent Swarms, MCP/A2A Protocols, etc.) into Luanti Voyager. Includes complete technical details, prompts, models used, and reproduction guide.

Imagine an AI that starts knowing nothing about the voxel world, then gradually learns to chop trees, craft tools, build shelters, and maybe even create things we never imagined. That's Luanti Voyager - a wild experiment in letting language models loose in the open-source Luanti sandbox.

Inspired by MineDojo's Voyager but built on fully open foundations, this project is whatever we make it together!

✨ Agents that learn by doing - No pre-programmed behaviors, just curiosity and code 🧠 Any LLM, any approach - OpenAI, Anthropic, local models, or something completely new 📚 Growing skill libraries - Each agent builds on what came before 🔓 Radically open - Fork it, break it, make it weird 🚀 Fast experimentation - Craftium gives us thousands of steps per second

• Voyager showed the way - LLMs can actually play and learn
• Luanti is ready - Mature, moddable, and truly open
• Craftium unlocked speed - Research-grade performance in a voxel world
• The community is hungry - People want to build crazy AI experiments

• No corporate overlords - We own this completely
• Mod anything - The engine itself is our playground
• Run anywhere - Your laptop, a Pi cluster, or the cloud
• Share everything - Skills, worlds, and wild ideas

Luanti World ←→ Craftium Magic ←→ Your Agent's Brain

The agent sees the world, thinks about what to do (using whatever LLM you want), tries stuff, and remembers what worked. That's it!

🌍 World Interface - Your agent sees blocks, entities, inventory 🤔 Brain Options - Any LLM, any prompting strategy, any crazy idea 🎮 Actions - Move, dig, place, craft, and whatever else we dream up 💾 Memory - Save skills, share with others, build on previous work

3D Viewer showing agent exploring terrain with static test blocks

The real-time 3D viewer displays the agent's world view using structured data re-rendering (similar to Mindcraft's approach). It shows blocks, agent position, and exploration in real-time without screen capture.

# Clone and run with one command!
git clone --recursive https://github.com/toddllm/luanti-voyager.git
cd luanti-voyager
./tools/quick_start_ollama.sh

📖 See INSTALL.md for complete setup instructions

Includes:

• ✅ Step-by-step guide for macOS
• 🔄 Linux/Ubuntu instructions (community help needed!)
• 🔄 Windows WSL instructions (community help needed!)
• 🎮 Luanti server setup
• 🤖 Agent configuration
• 🧠 LLM integration options

• Change the world: Flat, mountains, custom, whatever
• Change the brain: Different LLMs, prompting strategies, or even non-LLM approaches
• Change the goals: Survival, building, art, chaos
• Change everything: This is YOUR experiment

🤖 LLM Agents: Fully functional with live proof of gameplay

• Real-time 3D viewer - Working Three.js visualization with WebSocket streaming

• Agent exploration - Bot moves, teleports, and explores world via file-based commands

• Block detection - Agent perceives nearby blocks and reports world state

• Web interface - Live agent monitoring and 3D visualization at localhost:8090

• Screenshot workflow - Paste interface for rapid documentation (paste → describe → save)

• LLM integration - OpenAI/Anthropic/Ollama support with PROVEN real gameplay 🎯

• Basic survival - Health monitoring with emergency responses 🩸

• Skill memory - File-based learning system that remembers strategies 💭

🚧 Known Issues (help wanted!):

• Terrain generation - Replace void world with actual Luanti terrain (Issues #1-3)

Comprehensive test suite now available:

# Run all tests
python -m pytest tests/

# Run specific test categories
python -m pytest tests/unit/          # Component tests
python -m pytest tests/integration/   # System tests
python -m pytest tests/behaviors/     # Agent behavior tests
python -m pytest tests/skills/        # Skill safety tests

# Example output:
============================= test session starts ==============================
tests/unit/test_memory.py::TestSkillMemory::test_remember_skill PASSED   [100%]
========================= 1 passed, 1 warning in 0.01s =========================

Test coverage includes:

• Memory persistence - Skills and strategies saved/loaded correctly
• LLM mocking - Test without API calls
• Agent behaviors - Void escape, survival, exploration
• Skill safety - Sandboxing and code validation

• Advanced LLM integration - Multi-step reasoning and planning (Issue #4)
• Skill sharing between agents - Multi-agent collaboration (Issue #5)
• Learning from failure - Adaptive behavior improvement (Issue #6)
• Community challenges - Competitions and showcases (Issue #7)

We're inspired by Mindcraft, an impressive Minecraft agent framework that pioneered multi-agent coordination in voxel worlds. While Mindcraft focuses on Minecraft, we're adapting their best ideas for the open-source Luanti ecosystem.

• A Minecraft AI agent framework by Kolby Nottingham
• Features multi-agent collaboration and flexible LLM integration
• Uses Mineflayer to control bots in Minecraft servers
• Research focus: "Collaborating Action by Action" (paper coming 2025)

We've done a comprehensive analysis of their approach and identified key innovations to bring to Luanti:

1. Multi-Agent Coordination 🤖🤖

   • Agents that work together on complex tasks
   • Communication protocols for sharing information
   • Team-based goal decomposition

2. Profile-Based Behaviors 📋

   • JSON-configurable agent personalities
   • Easy experimentation with different agent types
   • Role specialization (builder, explorer, defender)

3. Flexible LLM Support 🧠

   • Support for OpenAI, Anthropic, Google, and local models
   • Hot-swappable AI backends
   • Model-agnostic design

4. Safety-First Design 🔒

   • Sandboxed code execution
   • Security warnings for public servers
   • Careful handling of LLM-generated code

We're implementing these Mindcraft-inspired features for Luanti:

• Integration Proposal
• Research Report

The goal: Combine Mindcraft's multi-agent expertise with Luanti Voyager's open-ended learning for unprecedented AI capabilities in open-source voxel worlds!

We've successfully implemented multi-agent coordination inspired by Mindcraft. Here's proof it works:

# Test basic communication between agents
python tests/test_multi_agent_simple.py

# See personality-based behaviors  
python tests/test_multi_agent_mock_llm.py

🤖 MULTI-AGENT PERSONALITY DEMO
====================================
📋 Agent Profiles:
  Scout: adventurous and excitable
  Constructor: careful and methodical

💎 Scenario 1: Major Discovery
Scout discovers: massive underground cave with glowing crystals
  Scout's reaction: "WOW! This is AMAZING! We could build an underground crystal palace here!"
  Constructor's response: "Interesting find. We should survey for stability before proceeding."

🚨 Scenario 2: Danger Approaching  
Alert: Large monster approaching the base
  Scout's reaction: "Let's fight it! I'll distract while you attack!"
  Constructor's reaction: "Fall back to defensive positions. Activate base defenses."

Full examples and demos →

# 20 lines of working multi-agent code!
import asyncio
from luanti_voyager.multi_agent import AgentCommunication

async def minimal_multi_agent():
    # Create two agents with different roles
    builder = AgentCommunication("BuilderBot", "team_alpha")
    explorer = AgentCommunication("ExplorerBot", "team_alpha") 
    
    # Connect them
    builder.register_listener("ExplorerBot", explorer)
    explorer.register_listener("BuilderBot", builder)
    
    # Explorer makes a discovery and shares it
    await explorer.share_discovery("Found diamonds at coordinates (100, -58, 200)!", importance=9)
    
    # Builder receives and processes the message
    messages = await builder.process_messages()
    print(f"BuilderBot received: {messages[0].content}")
    # Output: BuilderBot received: Discovery: Found diamonds at coordinates (100, -58, 200)!

asyncio.run(minimal_multi_agent())

• Multi-agent societies
• Agents that mod the game
• Completely new architectures
• Things we haven't imagined yet

• Wild experiments
• Silly projects
• Unexpected discoveries
• Community first

Test the proven LLM integration in 2 steps:

# 1. One-command setup and run (handles everything!)
./tools/quick_start_ollama.sh

# 2. Watch your agent make intelligent decisions!
# - Visit http://localhost:8090/viewer for 3D visualization
# - Check logs for LLM decision making

📖 Need help? See INSTALL.md for detailed setup instructions

# Start basic exploration agent
python -m luanti_voyager --name MyBot

# With test server on port 40000
python -m luanti_voyager --name MyBot --port 40000

🎯 WORKING NOW - Live Proof of LLM Gameplay

One-Command Start:

# Auto-setup and run with Ollama (handles everything!)
./tools/quick_start_ollama.sh

Manual Setup:

# 1. Install Ollama: https://ollama.ai/
# 2. Pull a model
ollama pull llama3.1

# 3. Start Ollama server
ollama serve

# 4. Run agent with local LLM (no API keys needed!)
python -m luanti_voyager --llm ollama --name LocalBot

# Set up API keys
export OPENAI_API_KEY=your_key_here
export ANTHROPIC_API_KEY=your_key_here

# Run with OpenAI GPT
python -m luanti_voyager --llm openai --name SmartBot

# Run with Anthropic Claude
python -m luanti_voyager --llm anthropic --name ClaudeBot

# Test all LLM integrations
python tests/test_llm_integration.py

# Copy and configure environment
cp .env.example .env
# Edit .env with your preferences

• 3D Viewer: http://localhost:8090/viewer
• Screenshot Paste: http://localhost:8090/paste

Check out our Deep Research Prompts - copy-paste ready prompts for exploring:

• Neurosymbolic skill learning
• Emergent communication
• Multi-agent coordination
• Open-ended evaluation
• And much more!

luanti-voyager/
├── luanti_voyager/      # The main code
├── worlds/              # Interesting places to explore
├── skills/              # What agents have learned
├── experiments/         # Your wild ideas
└── whatever/            # Seriously, add whatever you want

Literally anything helps:

• 💡 Share a crazy idea
• 🐛 Break something and tell us about it
• 🎨 Make it beautiful
• 🧪 Try weird experiments
• 📹 Record your agent doing something cool
• 🗣️ Tell your friends

No contribution is too small or too weird!

• Real-time visualization works - 3D viewer shows agent movement and world state ✅
• Agent moves autonomously - Bot explores and responds to environment ✅
• Development workflow is smooth - Screenshot paste and documentation tools work ✅
• It survives longer than 5 minutes - Basic survival and safety behaviors
• It learns something new - Demonstrates skill acquisition and memory
• It surprises us - Emergent behaviors we didn't program
• Someone says "whoa, I didn't expect that" - The ultimate test
• The community gets excited

We'll figure out proper benchmarks later. For now, let's just build cool stuff!

• GitHub: This is home base - star, fork, break things
• Discussions: Share your experiments and ideas
• Matrix: #luanti-voyager:matrix.org for real-time chat
• Your Blog/Twitter/YouTube: Show off what you build!

We're standing on the shoulders of giants:

• Voyager - Showed us LLMs can play
• Craftium - Made it fast
• Luanti - Gave us the perfect playground
• You - For being here and making this real

Understanding how Luanti Voyager approaches visual perception

Following Mindcraft's proven approach, we implement structured data re-rendering rather than screen capture:

Key insight: This is not screen capture of the game client - it's a real-time 3D re-render built from protocol data the agent already receives.

✅ Symbolic-First Architecture

• Rich game state: position, inventory, nearby blocks (5x5x5 radius)
• WebSocket streaming to Three.js viewer at http://localhost:8090/viewer
• Real-time agent visualization with block rendering
• Follow-cam and free-look camera controls

✅ Why This Works

• Keeps prompts small (JSON not JPG)
• Avoids vision model latency/cost
• Provides contributors with live 3D browser window
• Identical to how Minecraft Voyager operates (no visual input)

When ready for multimodal capabilities:

• Option A: Enhance current viewer to capture frames programmatically
• Option B: Add Luanti client screenshot API integration
• Option C: Integrate with GPT-4V/LLaVA-Next for pixel understanding

The architecture supports both approaches seamlessly - visual input becomes just another observation_space key.

# Start the integrated experience
python -m luanti_voyager.main --port 40000

# Open browser to see live 3D view
open http://localhost:8090/viewer

Watch your agent explore in real-time 3D while staying symbol-first for efficiency.

Luanti Voyager now includes a native UDP connection implementation for direct server communication:

from luanti_voyager import UDPLuantiConnection
import asyncio

async def connect():
    conn = UDPLuantiConnection(host="localhost", port=30000)
    await conn.connect()
    await conn.send_chat_message("Hello from AI!")
    await conn.disconnect()

asyncio.run(connect())

See Network API Documentation for complete details.

• UDP Connection Example - Basic connection and chat
• More examples coming soon!

MIT License - basically do whatever you want! See LICENSE for the formal version.