Open Resource Broker

A cloud provider integration plugin for IBM Spectrum Symphony Host Factory, enabling dynamic provisioning of compute resources with a REST API interface and structured architecture implementation.

Overview

The Open Resource Broker provides integration between IBM Spectrum Symphony Host Factory and cloud providers, implementing industry-standard patterns including Domain-Driven Design (DDD), Command Query Responsibility Segregation (CQRS), and structured architecture principles.

Currently Supported Providers:

• AWS - Amazon Web Services (RunInstances, EC2Fleet, SpotFleet, Auto Scaling Groups)
  • Context field support for EC2Fleet, SpotFleet, and Auto Scaling Groups

Key Features

Core Functionality

• HostFactory Compatible Output: Native compatibility with IBM Symphony Host Factory requirements
• Extensible Architecture: Extensible provider system with AWS support
• REST API Interface: REST API with OpenAPI/Swagger documentation
• Configuration-Driven: Dynamic provider selection and configuration through centralized config system

Key Architecture Features

• Clean Architecture: Domain-driven design with clear separation of concerns
• CQRS Pattern: Command Query Responsibility Segregation for scalable operations
• Event-Driven Architecture: Domain events with optional event publishing for template operations
• Dependency Injection: Comprehensive DI container with automatic dependency resolution
• Strategy Pattern: Pluggable provider strategies with runtime selection
• Resilience Patterns: Built-in retry mechanisms, circuit breakers, and error handling

Output Formats and Compatibility

• Flexible Field Control: Configurable output fields for different use cases
• Multiple Output Formats: JSON, YAML, Table, and List formats
• Legacy Compatibility: Support for camelCase field naming conventions
• Professional Tables: Rich Unicode table formatting for CLI output

Quick Start

PyPI Installation (Recommended)

# Minimal install (CLI only, 10 dependencies)
pip install orb-py

# With colored CLI output
pip install orb-py[cli]

# With API server (for REST API mode)
pip install orb-py[api]

# With monitoring (OpenTelemetry, Prometheus)
pip install orb-py[monitoring]

# Everything (all features)
pip install orb-py[all]

# Initialize configuration
orb init

# Generate example templates
orb templates generate

# List available templates
orb templates list

# Request machines
orb requests create --template-id EC2FleetInstant --count 3

Docker Deployment

# Clone repository
git clone https://github.com/awslabs/open-resource-broker.git
cd open-resource-broker

# Configure environment
cp .env.example .env
# Edit .env with your configuration

# Start services
docker-compose up -d

# Verify deployment
curl http://localhost:8000/health

Installation

Package Installation (Recommended)

# Minimal install (CLI only, 10 dependencies)
pip install orb-py

# With colored CLI output
pip install orb-py[cli]

# With API server (for REST API mode)
pip install orb-py[api]

# With monitoring (OpenTelemetry, Prometheus)
pip install orb-py[monitoring]

# Everything (all features)
pip install orb-py[all]

# Initialize configuration (required after installation)
orb init

# Verify installation
orb --version
orb --help

System-Wide Installation (Production)

# Auto-detects best location (no sudo needed if not available)
make install-system

# Or install to custom directory (requires sudo if needed)
ORB_INSTALL_DIR=/opt/orb make install-system

# Installation will output the actual location and next steps
# Add to PATH as instructed by the installer
orb --version

Local Development Installation

# Clone repository
git clone https://github.com/awslabs/open-resource-broker.git
cd open-resource-broker

# Install local development environment
make dev-install

# Or full development workflow (recommended)
make dev

Installation Comparison

Method	Location	Use Case	Command
PyPI	System Python	End users	pip install orb-py
System	/usr/local/orb/ or ~/.local/orb/	Production deployment	make install-system
Local	./.venv/	Development	make dev-install

Fast Development Setup with UV (Advanced)

For faster dependency resolution and installation, use uv:

# Install uv (if not already installed)
pip install uv

# Clone repository
git clone https://github.com/awslabs/open-resource-broker.git
cd open-resource-broker

# Fast development setup with uv
make dev-install

# Generate lock files for reproducible builds
make uv-lock

# Sync with lock files (fastest)
make uv-sync-dev

Or manually

pip install -e ".[dev]"

## Usage Examples

### MCP Server Mode (AI Assistant Integration)

The plugin provides a Model Context Protocol (MCP) server for AI assistant integration:

```bash
# Start MCP server in stdio mode (recommended for AI assistants)
orb mcp serve --stdio

# Start MCP server as TCP server (for development/testing)
orb mcp serve --port 3000 --host localhost

# Configure logging level
orb mcp serve --stdio --log-level DEBUG

Available MCP Tools

The MCP server exposes all CLI functionality as tools for AI assistants:

• Provider Management: check_provider_health, list_providers, get_provider_config, get_provider_metrics
• Template Operations: list_templates, get_template, validate_template
• Infrastructure Requests: request_machines, get_request_status, list_return_requests, return_machines

Available MCP Resources

Access domain objects via MCP resource URIs:

• templates:// - Available compute templates
• requests:// - Provisioning requests
• machines:// - Compute instances
• providers:// - Cloud providers

AI Assistant Prompts

Pre-built prompts for common infrastructure tasks:

• provision_infrastructure - Guide infrastructure provisioning workflows
• troubleshoot_deployment - Help diagnose deployment issues
• infrastructure_best_practices - Provide deployment best practices

Integration Examples

Claude Desktop Configuration:

{
  "mcpServers": {
    "open-resource-broker": {
      "command": "orb",
      "args": ["mcp", "serve", "--stdio"]
    }
  }
}

Python MCP Client:

import asyncio
from mcp import ClientSession, StdioServerParameters

async def use_hostfactory():
    server_params = StdioServerParameters(
        command="orb",
        args=["mcp", "serve", "--stdio"]
    )

    async with ClientSession(server_params) as session:
        # List available tools
        tools = await session.list_tools()

        # Request infrastructure
        result = await session.call_tool(
            "request_machines",
            {"template_id": "EC2FleetInstant", "count": 3}
        )

CLI Flag Patterns

The CLI supports both positional arguments and flags for consistent usage:

Template Operations:

# Both patterns supported
orb templates show template-id               # Positional
orb templates show --template-id template-id # Flag

orb templates delete template-id             # Positional  
orb templates delete --template-id template-id # Flag

orb templates update template-id --file new.json    # Positional + flag
orb templates update --template-id template-id --file new.json # All flags

Request Operations:

# Single request ID
orb requests status req-123                  # Positional
orb requests status --request-id req-123     # Flag
orb requests status -r req-123               # Short flag

# Multiple request IDs
orb requests status req-123 req-456          # Multiple positional
orb requests status -r req-123 -r req-456    # Multiple flags
orb requests status req-123 --request-id req-456 # Mixed patterns

Machine Operations:

# Request machines
orb machines request template-id 5           # Positional
orb machines request --template-id template-id --count 5 # Flags

# Show machine details
orb machines show machine-id                 # Positional
orb machines show --machine-id machine-id    # Flag

Global Provider Override:

# Works with any command
orb --provider aws_prod_us-west-2 [any-command]
orb --provider aws_dev_eu-west-1 templates list
orb --provider aws_prod_us-east-1 machines request template-id 3

Command Line Interface

Initial Setup

# Initialize ORB configuration (required after pip install)
orb init

# Interactive setup with prompts
orb init --interactive

# Non-interactive with defaults
orb init --non-interactive --scheduler default --provider aws --region us-east-1

# Custom configuration location
orb init --config-dir /path/to/config

Template Management (Full CRUD Operations)

# Generate example templates (after orb init)
orb templates generate

# Multi-provider template generation
orb templates generate --all-providers      # Generate for all active providers
orb templates generate --provider aws-prod  # Generate for specific provider instance
orb templates generate --provider-api EC2Fleet  # Generate for specific handler

# List available templates
orb templates list
orb templates list --long                    # Detailed information
orb templates list --format table           # Table format

# Provider override for template operations
orb --provider aws-prod templates list      # Use specific provider instance
orb --provider aws-dev templates show template-id  # Override provider for command

# Show specific template (supports both positional and flag arguments)
orb templates show TEMPLATE_ID              # Positional argument
orb templates show --template-id TEMPLATE_ID  # Flag argument

# Create new template
orb templates create --file template.json
orb templates create --file template.yaml --validate-only

# Update existing template (supports both argument patterns)
orb templates update TEMPLATE_ID --file updated-template.json  # Positional
orb templates update --template-id TEMPLATE_ID --file updated-template.json  # Flag

# Delete template (supports both argument patterns)
orb templates delete TEMPLATE_ID            # Positional
orb templates delete --template-id TEMPLATE_ID  # Flag
orb templates delete TEMPLATE_ID --force    # Force without confirmation

# Validate template configuration
orb templates validate --file template.json

# Refresh template cache
orb templates refresh
orb templates refresh --force               # Force complete refresh

Machine and Request Management

# Request machines (supports both positional and flag arguments)
orb requests create --template-id my-template --count 5
orb machines request TEMPLATE_ID COUNT       # Positional arguments
orb machines request --template-id TEMPLATE_ID --count COUNT  # Flag arguments

# Check request status (supports both patterns)
orb requests status REQUEST_ID               # Positional argument
orb requests status --request-id REQUEST_ID  # Flag argument
orb requests status -r REQUEST_ID            # Short flag

# Multiple request IDs (both patterns supported)
orb requests status req-123 req-456 req-789  # Multiple positional
orb requests status --request-id req-123 --request-id req-456  # Multiple flags
orb requests status -r req-123 -r req-456    # Multiple short flags

# Provider override for requests
orb --provider aws-prod machines request template-id 5
orb --provider aws-dev requests status req-123

# List active machines
orb machines list

# Show specific machine (supports both patterns)
orb machines show MACHINE_ID                 # Positional argument
orb machines show --machine-id MACHINE_ID    # Flag argument

# Return machines
orb requests return --request-id req-12345

Storage Management

orb storage list                    # List available storage strategies
orb storage show                    # Show current storage configuration
orb storage health                  # Check storage health
orb storage validate                # Validate storage configuration
orb storage test                    # Test storage connectivity
orb storage metrics                 # Show storage performance metrics

REST API

# Get available templates
curl -X GET "http://localhost:8000/api/v1/templates"

# Create machine request
curl -X POST "http://localhost:8000/api/v1/requests" \
  -H "Content-Type: application/json" \
  -d '{"templateId": "my-template", "maxNumber": 5}'

# Check request status
curl -X GET "http://localhost:8000/api/v1/requests/req-12345"

Architecture

The plugin implements Clean Architecture principles with the following layers:

• Domain Layer: Core business logic, entities, and domain services
• Application Layer: Use cases, command/query handlers, and application services
• Infrastructure Layer: External integrations, persistence, and technical concerns
• Interface Layer: REST API, CLI, and external interfaces

Design Patterns

• Domain-Driven Design (DDD): Rich domain models with clear bounded contexts
• CQRS: Separate command and query responsibilities for scalability
• Ports and Adapters: Hexagonal architecture for testability and flexibility
• Strategy Pattern: Pluggable provider implementations
• Factory Pattern: Dynamic object creation based on configuration
• Repository Pattern: Data access abstraction with multiple storage strategies
• Clean Architecture: Strict layer separation with dependency inversion principles

Configuration

Environment Configuration

# Provider configuration
PROVIDER_TYPE=aws
AWS_REGION=us-east-1
AWS_PROFILE=default

# API configuration
API_HOST=0.0.0.0
API_PORT=8000

# Storage configuration
STORAGE_TYPE=dynamodb
STORAGE_TABLE_PREFIX=hostfactory

# Scheduler directory configuration
# HostFactory scheduler
HF_PROVIDER_WORKDIR=/path/to/working/directory
HF_PROVIDER_CONFDIR=/path/to/config/directory
HF_PROVIDER_LOGDIR=/path/to/logs/directory

# Default scheduler
DEFAULT_PROVIDER_WORKDIR=/path/to/working/directory
DEFAULT_PROVIDER_CONFDIR=/path/to/config/directory
DEFAULT_PROVIDER_LOGDIR=/path/to/logs/directory

Provider Configuration

# config/providers.yml
providers:
  - name: aws-primary
    type: aws
    config:
      region: us-east-1
      profile: default
      handlers:
        default: ec2_fleet
        spot_fleet:
          enabled: true
        auto_scaling_group:
          enabled: true
    template_defaults:

Provider Naming Convention

Provider instances follow the pattern: {type}_{profile}_{region}

Examples:

• aws_default_us-east-1 - AWS default profile in us-east-1
• aws_prod_us-west-2 - AWS production profile in us-west-2
• aws_dev_eu-west-1 - AWS development profile in eu-west-1

Usage with Provider Override:

# Use specific provider instance for any command
orb --provider aws_prod_us-west-2 templates list
orb --provider aws_dev_eu-west-1 machines request template-id 3

# List available provider instances
orb providers list

Multi-Provider Operations

# Generate templates for all active providers
orb templates generate --all-providers

# Generate templates for specific provider instance
orb templates generate --provider aws_prod_us-west-2

# Generate templates for specific provider API
orb templates generate --provider-api EC2Fleet

# Override provider for any command
orb --provider aws_dev_eu-west-1 system health
orb --provider aws_prod_us-west-2 requests status req-123

Development

Prerequisites

• Python 3.10+ (tested on 3.10, 3.11, 3.12, 3.13, 3.14)
• Docker and Docker Compose
• AWS CLI (for AWS provider)

Development Setup

# Clone repository
git clone https://github.com/awslabs/open-resource-broker.git
cd open-resource-broker

# Create virtual environment
python -m venv .venv
source .venv/bin/activate

# Install development dependencies
pip install -r requirements-dev.txt

# Install in development mode
pip install -e .

# Run tests
make test

# Format code (Ruff replaces Black + isort)
make format

# Check code quality
make lint

# Run before committing (replaces pre-commit hooks)
make pre-commit

Testing

# Run all tests
make test

# Run with coverage
make test-coverage

# Run integration tests
make test-integration

# Run performance tests
make test-performance

Project Health & Metrics

These badges show real-time project health metrics including workflow success rates, test coverage, code quality indicators, and performance metrics. Dynamic badges are populated by automated workflows and may show "resource not found" until the first workflow runs complete.

Release Workflow

The project uses semantic-release for automated version management:

# Create a new release
git commit -m "release: add new features and bug fixes"
git push origin main

Release Process:

• Uses conventional commits for version calculation
• feat: → minor version bump
• fix: → patch version bump
• BREAKING CHANGE: → major version bump
• Commit with "release:" prefix triggers semantic-release
• Automatically publishes to PyPI, builds containers, and deploys documentation

See Release Management Guide for complete documentation.

Documentation

Comprehensive documentation is available at:

• Architecture Guide: Understanding the system design and patterns
• API Reference: Complete REST API documentation
• Configuration Guide: Detailed configuration options
• Developer Guide: Contributing and extending the plugin
• Deployment Guide: Production deployment scenarios

HostFactory Integration

The plugin is designed for seamless integration with IBM Spectrum Symphony Host Factory:

• API Compatibility: Full compatibility with HostFactory API requirements
• Attribute Generation: Automatic CPU and RAM specifications based on AWS instance types
• Output Format Compliance: Native support for expected output formats with accurate resource specifications
• Configuration Integration: Easy integration with existing HostFactory configurations
• Monitoring Integration: Compatible with HostFactory monitoring and logging

Resource Specifications

The plugin generates HostFactory attributes based on AWS instance types:

{
  "templates": [
    {
      "templateId": "t3-medium-template",
      "maxNumber": 5,
      "attributes": {
        "type": ["String", "X86_64"],
        "ncpus": ["Numeric", "2"],
        "nram": ["Numeric", "4096"]
      }
    },
    {
      "templateId": "m5-xlarge-template",
      "maxNumber": 3,
      "attributes": {
        "type": ["String", "X86_64"],
        "ncpus": ["Numeric", "4"],
        "nram": ["Numeric", "16384"]
      }
    }
  ]
}

Supported Instance Types: Common AWS instance types with appropriate CPU and RAM mappings

Support and Contributing

Getting Help

• Documentation: Comprehensive guides and API reference
• Issues: GitHub Issues for bug reports and feature requests
• Discussions: Community discussions and questions

Contributing

We welcome contributions! Please see our Contributing Guide for details on:

• Code style and standards
• Testing requirements
• Pull request process
• Development workflow

License

This project is licensed under the Apache License 2.0 - see the LICENSE file for details.

Security

For security concerns, please see our Security Policy for responsible disclosure procedures.