guicli

A Python CLI tool for "human-like" UI automation on both desktop and web applications. guicli simulates natural user behavior through randomization, jitter, and timing delays, making automated interactions appear more realistic.

Features

• Desktop Automation: Control mouse and keyboard on desktop applications (PyAutoGUI + pynput)
• Web Automation: Automate browser interactions with Playwright (Chrome, Firefox, Edge)
• Computer Vision: Find and click UI elements using template matching (OpenCV)
• Human-like Behavior: Natural mouse movements with easing curves, click jitter, and typing delays
• Script Execution: Define complex workflows in YAML/JSON format
• Cross-platform: Works on Windows, macOS, and Linux
• Comprehensive Logging: Detailed JSONL logs with screenshots on errors
• Dry-run Mode: Preview actions before execution

Installation

Prerequisites

• Python 3.8 or higher
• pip package manager

From Source

# Clone the repository
git clone https://github.com/yourusername/guicli.git
cd guicli

# Install dependencies
pip install -e .

# Install Playwright browsers (for web automation)
playwright install chromium

Using pip (when published)

pip install guicli
playwright install chromium

Binary Distribution

Download pre-built binaries for your platform from the Releases page:

• Windows: guicli-windows.exe (~150MB with browser, ~40MB without)
• macOS: guicli-macos (~150MB with browser, ~40MB without)
• Linux: guicli-linux (~150MB with browser, ~40MB without)

No Python installation required for binary distributions.

Platform-Specific Setup

Windows

1. No special permissions required for basic desktop automation
2. For web automation, ensure Windows Defender doesn't block the browser executable
3. Run from Command Prompt or PowerShell:

   guicli --help

macOS

1. Grant Accessibility Permission (required for desktop automation):

   • Open System Preferences → Security & Privacy → Privacy → Accessibility
   • Click the lock icon to make changes
   • Add Terminal (or your terminal app) to the list
   • Check the box to enable access

2. Grant Screen Recording Permission (for screenshots):

   • Open System Preferences → Security & Privacy → Privacy → Screen Recording
   • Add Terminal to the list and enable

3. Run from Terminal:

   ./guicli --help

Linux

1. Install X11 dependencies (if not already installed):

   # Ubuntu/Debian
   sudo apt-get install python3-tk python3-dev scrot
   
   # Fedora
   sudo dnf install python3-tkinter python3-devel scrot

2. For Wayland users: Some features may require XWayland compatibility layer

3. Run from terminal:

   ./guicli --help

Quick Start

Basic Desktop Commands

# Click at coordinates (500, 400)
guicli click --x 500 --y 400

# Move mouse to coordinates
guicli move --x 800 --y 600

# Type text
guicli type --text "Hello World"

# Press hotkey combination
guicli hotkey --keys "ctrl+c"

# Take screenshot
guicli screenshot --output screenshot.png

# Wait for 2 seconds
guicli wait --seconds 2

Template Matching

# Find and click a UI element using an image template
guicli find --template buttons/ok_button.png --threshold 0.85

Web Automation

# Navigate to URL and click element
guicli web --url https://example.com --click "text=Login"

# Type into web form
guicli web --url https://example.com --type "#username" "myuser"

Script Execution

# Run a YAML/JSON script
guicli run --script my_workflow.yaml

# Dry-run to preview actions
guicli run --script my_workflow.yaml --dry-run

# Run with logging
guicli run --script my_workflow.yaml --log execution.jsonl

Configuration

Create a config file at ~/.guicli/config.yaml:

defaults:
  jitter: 3              # Click jitter in pixels
  speed: 1.0             # Mouse movement speed multiplier
  delay_min: 0.05        # Minimum typing delay (seconds)
  delay_max: 0.2         # Maximum typing delay (seconds)
  timeout: 10            # Default timeout (seconds)
  retries: 3             # Number of retries for operations

web:
  browser: chromium      # Browser: chromium, firefox, webkit
  headless: false        # Run browser in headless mode

logging:
  console: true          # Log to console
  file: null             # Log file path (null = no file logging)
  log_text: true         # Log sensitive text (set false for security)
  screenshot_on_error: true  # Capture screenshot on errors

Global Flags

All commands support these global flags:

• --speed N: Mouse movement speed (0.1-10.0, default 1.0)
• --jitter N: Click jitter in pixels (default 2)
• --delay-min N: Minimum typing delay in seconds (default 0.05)
• --delay-max N: Maximum typing delay in seconds (default 0.2)
• --timeout N: Operation timeout in seconds (default 10)
• --retries N: Number of retries (default 3)
• --log PATH: Log to JSONL file
• --no-log-text: Don't log sensitive text
• --dry-run: Preview actions without executing
• --highlight: Highlight click areas (dry-run mode)

Documentation

• Usage Guide - Detailed command reference and examples
• Scripting Guide - How to write automation scripts
• Troubleshooting - Common issues and solutions

Examples

See the examples directory for sample scripts:

• examples/desktop_workflow.yaml - Desktop automation example
• examples/web_login.yaml - Web form automation
• examples/mixed_workflow.yaml - Combined desktop and web automation

Development

# Install development dependencies
pip install -e ".[dev]"

# Run tests
pytest

# Run property-based tests
pytest tests/property/

# Build binary
python build.py

License

MIT License - see LICENSE file for details

Contributing

Contributions are welcome! Please read CONTRIBUTING.md for guidelines.

Support

• Issues: GitHub Issues
• Discussions: GitHub Discussions
• Documentation: Wiki

Acknowledgments

Built with:

• PyAutoGUI - Desktop automation
• pynput - Keyboard/mouse control
• Playwright - Web automation
• OpenCV - Computer vision
• Click - CLI framework