A Python library for model-based GUI automation with intelligent state management and visual recognition.
Based on Model-based GUI Automation published in Springer's Software and Systems Modeling journal (October 2025).
The research provides:
- Mathematical proof of complexity reduction (exponential → polynomial)
- First testable approach to GUI automation (unit tests, integration tests)
- Formal framework for robust visual APIs for RL agents
- Enables reliable dataset generation for AI training
Qontinui enables building robust GUI automation through:
- Model-based state management using MultiState | Docs
- Visual recognition with OpenCV template matching
- JSON configuration for defining automation workflows
- Cross-platform support (Windows, macOS, Linux)
Qontinui is a Python port of Brobot, a Java library for GUI automation (2018-2024).
pip install qontinui# Clone the repository
git clone https://github.com/jspinak/qontinui.git
cd qontinui
# Install with Poetry
poetry install
# Or with pip
pip install -e .Qontinui requires:
- MultiState - Multi-state state management
- OpenCV - Image template matching
- PyAutoGUI/pynput - Input control
Create an automation configuration in JSON:
{
"version": "1.0",
"states": [
{
"name": "LoginScreen",
"stateImages": [
{
"imageId": "login_button",
"threshold": 0.9
}
]
}
],
"processes": [
{
"name": "Login",
"actions": [
{
"type": "CLICK",
"target": {"type": "image", "imageId": "login_button"}
},
{
"type": "TYPE",
"text": "[email protected]"
}
]
}
]
}from qontinui.json_executor import JSONRunner
# Initialize runner
runner = JSONRunner()
# Load configuration
runner.load_configuration("automation_config.json")
# Execute automation
success = runner.run(monitor_index=0)Use Qontinui Runner for a GUI interface to create and run automations.
qontinui/
├── src/qontinui/
│ ├── json_executor/ # JSON configuration execution
│ ├── model/ # State, Transition, Image models
│ ├── hal/ # Hardware Abstraction Layer
│ └── multistate_adapter/ # MultiState integration
- ✅ JSON-based automation configuration
- ✅ Template-based image matching
- ✅ Multi-state state management
- ✅ Process and state machine execution modes
- ✅ Cross-platform input control (PyAutoGUI, pynput)
- ✅ Hardware abstraction layer for multiple backends
- ✅ Self-healing system with action caching, visual validation, and optional LLM assistance
- ✅ AWAS integration for structured web automation via AI action manifests
- 🔄 AI-enhanced visual recognition (SAM, CLIP)
- 🔄 Domain-specific language (DSL)
- 🔄 Advanced Brobot migration tools
- 🔄 Cloud execution via qontinui-web
Qontinui includes an intelligent self-healing system that automatically recovers from element lookup failures:
- Action Caching - Remembers successful element locations for instant replay
- Visual Search - Finds elements at lower thresholds and multiple scales when exact matching fails
- LLM Assistance - Optionally uses vision models (local Ollama or cloud APIs) to locate elements by description
from qontinui.actions.find import FindAction, FindOptions
from qontinui.healing import HealingConfig, configure_healing
# Enable local LLM healing (optional)
configure_healing(HealingConfig.with_ollama())
# Find with self-healing enabled
options = FindOptions(
similarity=0.85,
enable_healing=True,
healing_context_description="Submit button",
use_cache=True,
store_in_cache=True,
)
result = await FindAction().find(pattern, options)See Self-Healing Documentation for complete configuration options and API reference.
Qontinui includes AWAS support for structured web automation. AWAS enables websites to expose machine-readable action manifests that AI agents can discover and execute.
qontinui/src/qontinui/awas/
├── types.py # Pydantic models for manifests and actions
├── discovery.py # Manifest discovery with caching
├── executor.py # HTTP action execution
└── extractor.py # Web extraction strategy
from qontinui.awas.discovery import AwasDiscoveryService
from qontinui.awas.executor import AwasExecutor
# Discover AWAS manifest
discovery = AwasDiscoveryService()
manifest = await discovery.discover("https://example.com")
# List available actions
for action in manifest.actions:
print(f"{action.method} {action.endpoint}: {action.intent}")
# Execute an action
executor = AwasExecutor()
result = await executor.execute(
manifest=manifest,
action_id="list_items",
params={"limit": 10}
)- 10-100x faster than vision-based automation
- No visual templates to maintain
- Typed parameters with validation
- Structured responses for reliable parsing
See AWAS Integration Guide for complete API reference.
Run the test suite:
# Run all tests
poetry run pytest
# Run with coverage
poetry run pytest --cov=src/qontinui
# Run specific test file
poetry run pytest tests/json_executor/test_json_runner.pyWe welcome contributions! Please see CONTRIBUTING.md for guidelines.
Please note that this project is released with a Code of Conduct. By participating in this project you agree to abide by its terms.
Qontinui is a faithful port of Brobot. When contributing, please preserve Brobot's architecture and behavior.
- GitHub: github.com/qontinui/qontinui
- Issues: GitHub Issues
- Self-Healing Guide: docs/self-healing.md
- MultiState Docs: qontinui.github.io/multistate
- Research Paper: Springer SoSyM
Licensed under the GNU Affero General Public License v3.0 or later (AGPL-3.0-or-later). See LICENSE for full terms.
- Brobot - The original Java library
- MultiState - Multi-state state management
- PyAutoGUI - GUI automation
- OpenCV - Computer vision
- qontinui-runner - Desktop application (Rust/TypeScript)
- qontinui-web - Web-based visual builder (launching Feb 2026)
- multistate - State management library | Docs
- Brobot - Original Java implementation (2018-2024)