Supported OS:
Latest Version:    

Lybic platform placeholder - comprehensive AI platform for building and deploying intelligent agents

Lybic GUI Agent is an open-source framework that enables developers and businesses to create intelligent computer-use agents,mobile-use agents, and intelligent agents that can understand and interact with graphical user interfaces across Windows, macOS, Linux and Android(via lybic Android Sandbox) platforms.

• 2025/09/14: The paper has been accepted by arxiv
• 2025/09/09: We achieved the world's first place in the 50-step length of OS-world!
• 2025/08/08: Released v0.1.0 of Lybic GUI Agent library, with support for Windows, Mac, Ubuntu and Lybic API!

1. 💡 Introduction
2. 🛠️ Installation & Setup
3. 🚀 Usage
4. 🔧 Troubleshooting
5. 💬 Citations

• Multiple LLMs providers: OpenAI, Anthropic, Google, xAI , AzureOpenAI, DeepSeek, Qwen, Doubao, ZhipuGLM
  • Aggregation Model Provider: Bedrock, Groq, Monica, OpenRouter, SiliconFlow
• RAG: We support RAG, and this capability is provided as an extension
• Cross-Platform GUI Control: Windows, Linux, macOS, Android Supported
• Observability: Supported
• Local Deployment: Supported
• Cloud Sandbox Environment: Supported

🎉 Agents Online Demo

🎯 Current Results

Fig. Lybic GUI Agent system structure

You can install Lybic GUI Agent by using pip:

pip install lybic-guiagents

You can use UV (a modern Python package manager) version 0.8.5 for installation:

# 1. Install UV if not already installed
# macOS and Linux
curl -LsSf https://astral.sh/uv/0.8.5/install.sh | sh
# Windows
powershell -ExecutionPolicy ByPass -c "irm https://astral.sh/uv/0.8.5/install.ps1 | iex"

# testing uv installation, version should be 0.8.5
uv --version

# 2. Install the python 3.12
uv python install 3.12.11

# 3. Create a virtual environment
uv venv -p 3.12.11

# 4. Activate the virtual environment
# macOS and Linux
source .venv/bin/activate
# Windows
.venv\Scripts\activate

# 5. Install dependencies (using locked versions)
uv sync

# 6. Install the package locally in development mode
uv pip install -e .

The simplest way to configure API keys is to:

1. Copy gui_agents/.env.example to gui_agents/.env
2. Edit the .env file and add your API keys

We provide two pre-configured tool settings:

• tools_config_en.json: Configured for English language models (Gemini, Exa)
• tools_config_cn.json: Configured for Chinese language models (Doubao, bocha)

The agent uses tools_config.json by default. You can:

• Copy either tools_config_en.json or tools_config_cn.json to tools_config.json
• Or create your own custom configuration

If you are using tools_config_cn.json and use pyautogui backend, the environment variable only ARK_API_KEY should be set.

If you are using tools_config_en.json and use pyautogui backend, you should set those 3 environment variables:

GEMINI_ENDPOINT_URL=https://generativelanguage.googleapis.com/v1beta/openai/
GEMINI_API_KEY=your_gemini_api_key
ARK_API_KEY=your_ark_api_key

# For English models
cp gui_agents/tools/tools_config_en.json gui_agents/tools/tools_config.json

# For Chinese models
cp gui_agents/tools/tools_config_cn.json gui_agents/tools/tools_config.json

Note: Our recommended configuration uses doubao-1-5-ui-tars-250428 for "tool_name": "grounding" or "fast_action_generator" and claude-sonnet-4-20250514 or doubao-seed-1-6-250615 for other tools such as "tool_name": "action_generator". You can customize the model configuration in the tools configuration files. Do not modify the "tool_name" in tools_config.json file. To change the "provider" and "model_name" in tools_config.json file, see model.md

Run Lybic GUI Agent with python in the command-line interface:

python gui_agents/cli_app.py [OPTIONS]

This will show a user query prompt where you can enter your instructions and interact with the agent.

• --backend [lybic|pyautogui|pyautogui_vmware]: Specifies the backend to use for controlling the GUI. Defaults to lybic.

• --query "YOUR_QUERY": Optional, can be input during the runtime; if provided, the agent will execute the query and then exit.

• --max-steps NUMBER: Sets the maximum number of steps the agent can take. Defaults to 50.

• --mode [normal|fast]: (Optional) Selects the agent mode. normal runs the full agent with detailed reasoning and memory, while fast mode executes actions more quickly with less reasoning overhead. Defaults to normal.

• --enable-takeover: (Optional) Enables user takeover functionality, allowing the agent to pause and request user intervention when needed. By default, user takeover is disabled.

• --disable-search: (Optional) Disables web search functionality. By default, web search is enabled.

Run in interactive mode with the lybic backend:

python gui_agents/cli_app.py --backend lybic

Run a single query with the pyautogui backend and a maximum of 20 steps:

python gui_agents/cli_app.py --backend pyautogui --query "Find the result of 8 × 7 on a calculator" --max-steps 20

Run in fast mode with the pyautogui backend:

python gui_agents/cli_app.py --backend pyautogui --mode fast

The simplest way to configure Lybic Sandbox is still to edit the .env file and add your API keys, as mentioned in the API Key Configuration section.

LYBIC_API_KEY=your_lybic_api_key
LYBIC_ORG_ID=your_lybic_org_id
LYBIC_MAX_LIFE_SECONDS=3600

Note: If you want to use a precreated Lybic Sandbox in Lybic Dashboard, you need to set the LYBIC_PRECREATE_SID to the precreated Sandbox ID.

LYBIC_PRECREATE_SID=SBX-XXXXXXXXXXXXXXX

After installing lybic-guiagents, you can run it as a service.

Main Components:

• AgentService: High-level service interface (recommended for most users)
• AgentS2, AgentSFast: Core agent implementations
• HardwareInterface: Hardware abstraction layer
• ServiceConfig: Configuration management

Quick Start:

from gui_agents import AgentService
service = AgentService()
result = service.execute_task("Take a screenshot")
print(f"Task completed: {result.status}")

To use PyAutoGUI with VMware, you need to install VMware Workstation Pro (on Windows) and create a virtual machine.

Next, you need to download the Windows-x86.zip and Ubuntu-x86.zip from Hugging Face. Then unzip them into ./vmware_vm_data/Windows-x86 and ./vmware_vm_data/Ubuntu-x86 directory.

Finally, you need to edit the .env file and set the USE_PRECREATE_VM environment variable to the name of the virtual machine. USE_PRECREATE_VM support Windows and Ubuntu on x86 arch computer.

USE_PRECREATE_VM=Ubuntu

Problem: KeyError or authentication errors when running the agent.

Solution:

• Ensure your .env file is properly configured with valid API keys
• Check that environment variables are set correctly:

  # For English models
  export GEMINI_API_KEY=your_gemini_api_key
  export ARK_API_KEY=your_ark_api_key
  
  # For Chinese models  
  export ARK_API_KEY=your_ark_api_key

• Verify API key permissions and quotas

Problem: ModuleNotFoundError or package import errors.

Solution:

• Ensure you're using Python 3.12.11 as specified
• Activate the virtual environment:

  # macOS/Linux
  source .venv/bin/activate
  # Windows
  .venv\Scripts\activate

• Reinstall dependencies:

  uv sync
  uv pip install -e .
  # uv pip install -e . -i  https://pypi.tuna.tsinghua.edu.cn/simple

Problem: Connection timeout or Sandbox creation failed.

Solution:

• Check network connectivity to Lybic servers
• Verify LYBIC_ORG_ID and LYBIC_API_KEY are correct
• Ensure sufficient quota in your Lybic account
• Try increasing LYBIC_MAX_LIFE_SECONDS if sandbox times out

Problem: Virtual machine fails to start or control.

Solution:

• Ensure VMware Workstation Pro is properly installed
• Check that VM files are extracted to correct directories:
  • ./vmware_vm_data/Windows-x86/
  • ./vmware_vm_data/Ubuntu-x86/
• Verify VMware service is running
• Set correct USE_PRECREATE_VM environment variable

Problem: Slow response times or poor grounding accuracy.

Solution:

• Use recommended models for better performance:
  • Visual grounding: doubao-1-5-ui-tars-250428
  • Action generation: claude-sonnet-4-20250514
• Switch to --mode fast for quicker execution
• Reduce --max-steps for shorter tasks

If you encounter issues not covered here:

1. Check the GitHub Issues for similar problems
2. Review the Lybic Documentation
3. Create a new issue with:
   • Your operating system and version
   • Python version and environment details
   • Complete error messages
   • Steps to reproduce the issue

If you find this codebase useful, please cite:

@misc{guo2025agenticlybicmultiagentexecution,
      title={Agentic Lybic: Multi-Agent Execution System with Tiered Reasoning and Orchestration}, 
      author={Liangxuan Guo and Bin Zhu and Qingqian Tao and Kangning Liu and Xun Zhao and Xianzhe Qin and Jin Gao and Guangfu Hao},
      year={2025},
      eprint={2509.11067},
      archivePrefix={arXiv},
      primaryClass={cs.AI},
      url={https://arxiv.org/abs/2509.11067}, 
}

This project is distributed under Apache 2.0 License. Therefore, you can modify the source code and release it commercially.