Monitoring Stack Deployment (Prometheus, Node Exporter, Docker Volumes Metrics)

This repository contains all configuration files, scripts, and automation necessary to deploy and maintain a monitoring stack based on Prometheus, Node Exporter, and custom Docker volumes metrics collection.
It also includes a GitHub Actions CI/CD pipeline for syntax validation, lint checks, and safe dry-run execution.

---

Components Overview

1. Prometheus Server

• Installed manually (non-Docker).
• Configured to scrape:
  • Local Prometheus instance
  • Remote Node Exporter endpoints
• Config file: /etc/prometheus/prometheus.yml

2. Node Exporter

• Installed manually on both Prometheus server and monitored nodes.
• Configured to expose standard Linux metrics.
• Installed as a systemd service.

3. Docker Volumes Metrics Script

• Collects the size (in bytes) of Docker volumes matching pattern sremgas-*.
• Exports metrics in Prometheus textfile format to be scraped by Node Exporter.
• Runs periodically via systemd timer.

4. Watchdog Scripts

• Monitors health of Prometheus and Node Exporter services.
• Automatically restarts services if they are not running.
• Implemented as Bash scripts + systemd timers.

---

Repository Structure

ansible/                 # Ansible automation for deployment
  hosts                  # Ansible inventory file
  playbook.yml           # Main Ansible playbook
  roles/
    prometheus/
    node_exporter/
    docker_metrics/
    watchdogs/
.github/
  workflows/
    ansible-ci.yml       # GitHub Actions CI/CD workflow
scripts/                 # Utility scripts (manual install / debug)

---

Installation & Deployment

1. Manual Installation Steps

Prometheus

# Download & extract
wget https://github.com/prometheus/prometheus/releases/download/v3.5.0/prometheus-3.5.0.linux-amd64.tar.gz
tar xvf prometheus-3.5.0.linux-amd64.tar.gz
sudo mv prometheus-3.5.0.linux-amd64 /opt/prometheus

# Create Prometheus user
sudo useradd --no-create-home --shell /bin/false prometheus
sudo mkdir /etc/prometheus /var/lib/prometheus
sudo chown prometheus:prometheus /etc/prometheus /var/lib/prometheus

# Copy binaries
sudo cp /opt/prometheus/prometheus /opt/prometheus/promtool /usr/local/bin/

# Create systemd service
sudo nano /etc/systemd/system/prometheus.service

Service example:

[Unit]
Description=Prometheus Monitoring
Wants=network-online.target
After=network-online.target

[Service]
User=prometheus
ExecStart=/usr/local/bin/prometheus   --config.file=/etc/prometheus/prometheus.yml   --storage.tsdb.path=/var/lib/prometheus   --web.console.templates=/opt/prometheus/consoles   --web.console.libraries=/opt/prometheus/console_libraries
Restart=always

[Install]
WantedBy=multi-user.target

---

Node Exporter

wget https://github.com/prometheus/node_exporter/releases/download/v1.8.1/node_exporter-1.8.1.linux-amd64.tar.gz
tar xvf node_exporter-1.8.1.linux-amd64.tar.gz
sudo mv node_exporter-1.8.1.linux-amd64/node_exporter /usr/local/bin/

# Create service
sudo nano /etc/systemd/system/node_exporter.service

Example:

[Unit]
Description=Node Exporter
After=network.target

[Service]
User=node_exporter
ExecStart=/usr/local/bin/node_exporter   --collector.textfile.directory=/var/lib/node_exporter/textfile
Restart=always

[Install]
WantedBy=multi-user.target

---

Docker Volumes Metrics

Script: /usr/local/bin/docker-volumes-metrics.sh

#!/usr/bin/env bash
set -euo pipefail
out="/var/lib/node_exporter/textfile/docker_volumes.prom"
tmp="$(mktemp)"
echo "# HELP docker_volume_size_bytes Size of Docker named volumes" > "$tmp"
echo "# TYPE docker_volume_size_bytes gauge" >> "$tmp"

while IFS= read -r vol; do
  path="/var/lib/docker/volumes/${vol}/_data"
  if [ -d "$path" ]; then
    size_bytes=$(du -sb "$path" 2>/dev/null | awk '{print $1}')
    size_bytes=${size_bytes:-0}
    echo "docker_volume_size_bytes{volume=\"${vol}\"} $size_bytes" >> "$tmp"
  fi
done < <(docker volume ls --format '{{.Name}}' | grep '^sremgas-')

mv "$tmp" "$out"

Systemd timer runs it periodically.

---

🔄 GitHub Actions CI/CD

• Lints Ansible playbooks.
• Runs syntax check.
• Performs dry-run execution to verify changes.

Workflow file: .github/workflows/ansible-ci.yml

---

Monitoring Integration

• Prometheus scrapes:
  • Local Prometheus instance (localhost:9090)
  • Node Exporters (<IP>:9100)
• Docker metrics exposed via node_exporter textfile collector.

---

Notes

• Ensure firewall allows Prometheus and Node Exporter ports.
• In production, secure endpoints via TLS or reverse proxy.
• This repo’s playbook is idempotent – safe to re-run.

---

Contact

Maintainer: Darko Nedic
Email: [email protected]