Guardian

Guardian is a monitoring and auto-remediation platform for a Linux fleet. It combines Prometheus, Alertmanager, Grafana, Ansible, and controlled webhook runbooks to detect unhealthy conditions, notify operators, and execute bounded recovery actions.

Live Links

• Public Grafana: https://grafana.guardian.abhinash.dev/
• Architecture: docs/ARCHITECTURE.md
• Deployment guide: docs/DEPLOYMENT.md
• Product scope: docs/PRD.md

What Guardian Does

• Collects host-level and application-level metrics from a Linux fleet.
• Uses Prometheus and Alertmanager for rule evaluation and routing.
• Sends live alerts to Slack and supports bounded webhook remediation paths.
• Executes whitelisted runbooks for safe, bounded recovery actions.
• Uses Ansible for inventory-driven host onboarding and control-plane rollout.

How It Works

1. node_exporter and the custom Python exporter expose host and application metrics.
2. Prometheus scrapes those metrics and evaluates alert rules.
3. Alertmanager routes alerts to Slack and, when allowed, to the remediation webhook.
4. The webhook selects a mapped runbook and executes it against the intended target.
5. Grafana shows the fleet before, during, and after recovery.

Active Alerts

The live deployment currently evaluates and routes these alert sources:

• TargetDown fires when a monitored scrape target stays down for more than 2 minutes.
• HighCPU fires when sustained host CPU usage exceeds 85% for more than 2 minutes.
• Watchdog is always firing internally as an Alertmanager heartbeat and is intentionally not sent to Slack.

In steady state, TargetDown and HighCPU both notify Slack. The recorded demo shows the same alerting pipeline with a temporary drill host attached to prove the remediation loop end to end.

Demo Flow

Guardian includes a controlled failure-injection path for demonstration and validation:

1. A temporary drill host is placed under sustained synthetic CPU load with stress-ng.
2. The HighCPU alert fires after the configured hold period.
3. Alertmanager sends the event to Slack and the remediation webhook.
4. The webhook runs a drill-safe runbook that stops the synthetic load.
5. CPU usage returns to normal and the alert clears.

The live fleet no longer keeps a permanent drill host attached. The demo shown here was recorded against a temporary drill-safe host, and the current HighCPU remediation path is retired from active deployment. In a real workload, the same pipeline would invoke a host-role-specific recovery action instead of a generic process kill.

Guardian.Demo.Final.mp4

Local Run

1. Copy the environment template:

   cp .env.example .env

2. Set SLACK_WEBHOOK_URL in .env.
3. Start the stack:

   docker compose up --build

4. Access the local services:
   • Prometheus: http://localhost:9090
   • Alertmanager: http://localhost:9093
   • Grafana (direct): http://localhost:3000
   • Grafana via Caddy TLS: https://grafana.localtest.me
   • Webhook via Caddy TLS: https://webhook.localtest.me/remediate

Test and Validation

Run local checks from the repo root:

• Exporter tests: uv run --project exporter --extra dev pytest exporter/tests
• Webhook tests: uv run --project webhook --extra dev pytest webhook/tests
• Python lint: uv run --with ruff ruff check exporter webhook
• Ansible lint: uv run --with ansible-core --with ansible-lint ansible-lint ansible/site.yml
• Playbook syntax check: uv run --with ansible-core ansible-playbook --syntax-check -i ansible/inventory/hosts.ini ansible/site.yml

Subproject uv.lock files are intentionally not tracked. Use uv run --project ... instead of creating committed per-subproject lockfiles.

Deployment

Guardian is currently deployed manually with Ansible. Inventory is the source of truth for monitored hosts and control-plane rollout.

Short version:

1. Configure ansible/inventory/hosts.ini.
2. Ensure target hosts have SSH access, Python, and the required sudo permissions.
3. Provide runtime secrets to the Ansible runner.
4. Validate the playbook:

   uv run --with ansible-core ansible-playbook --syntax-check -i ansible/inventory/hosts.ini ansible/site.yml

5. Apply the deployment:

   uv run --with ansible-core ansible-playbook -i ansible/inventory/hosts.ini ansible/site.yml

The full operator workflow, runtime variable requirements, and live deploy procedure are documented in docs/DEPLOYMENT.md.

Repository Layout

ansible/             Fleet and control-plane provisioning playbooks/roles
exporter/            Custom Python exporter service and tests
webhook/             Remediation webhook service, runbooks, tests
prometheus/          Scrape config and alert rules
alertmanager/        Alert routing config
grafana/             Datasource and dashboard provisioning
runbooks/            Human-readable runbooks
scripts/             Drill and operator helper scripts
docs/                PRD, architecture, deployment, SLO, MTTR, postmortem template

Documentation

• docs/PRD.md for scope and functional requirements
• docs/ARCHITECTURE.md for system design and data flow
• docs/DEPLOYMENT.md for setup and operator workflow
• docs/SLO.md for service-level targets
• docs/MTTR.md for drill timing and recovery measurements
• docs/POSTMORTEM_TEMPLATE.md for incident writeups