Monitoring & Logging - DevOps Masterclass

Learning Objectives
●​ Define monitoring and logging in the context of DevOps​

●​ Distinguish between metrics and logs​

●​ Understand how Prometheus and Grafana work together​

●​ Deploy Prometheus and Grafana using Docker Compose​

​

1. What is Monitoring?
Monitoring is the collection, processing, and analysis of metrics to ensure the health and
performance of systems.

Common Monitoring Metrics

●​ CPU usage​

●​ Memory usage​

●​ Disk space​

●​ Request rate​

●​ Uptime​

Benefits
 ●​ Detect failures early​

●​ Observe trends over time​

●​ Enable performance tuning​

2. What is Logging?
Logging is the process of recording textual data about system events, errors, and operations.

Examples

●​ Error messages​

●​ User activity logs​

●​ HTTP request logs​

Benefits

●​ Debugging and root cause analysis​

●​ Security auditing​

●​ Operational transparency​

3. Monitoring vs Logging
Feature Monitoring Logging

Data Type Numeric metrics Text-based messages

Purpose System health, Troubleshooting, audit trail

alerting
 Visualization Dashboards (Grafana) Log viewers (e.g., Loki, Kibana)

Example Tool Prometheus ELK Stack / Loki

4. Tools Overview

Prometheus

●​ Pull-based metrics collection system​

●​ Built-in query language: PromQL​

●​ Stores time-series data​

Grafana

●​ Dashboard visualization tool​

●​ Connects to Prometheus and other data sources​

●​ Customizable panels and alerts​

5. Prometheus & Grafana with Docker Compose

Step 1: Project Setup

mkdir monitoring-stack && cd monitoring-stack

Step 2: Create docker-compose.yml

version: '3'
services:
prometheus:
image: prom/prometheus
container_name: prometheus
 volumes:
- ./prometheus.yml:/etc/prometheus/prometheus.yml
ports:
- "9090:9090"

grafana:
image: grafana/grafana
container_name: grafana
ports:
- "3000:3000"
volumes:
- grafana-storage:/var/lib/grafana

volumes:
grafana-storage:

Step 3: Create Prometheus Configuration File prometheus.yml

global:
scrape_interval: 15s

scrape_configs:
- job_name: 'prometheus'
static_configs:
- targets: ['localhost:9090']

Step 4: Launch the Monitoring Stack

docker-compose up -d

Step 5: Access the Interfaces

●​ Prometheus UI: http://localhost:9090​

●​ Grafana UI: http://localhost:3000 (default login: admin/admin)​

 6. Configure Grafana to Use Prometheus
1.​ Log into Grafana​

2.​ Go to Settings → Data Sources​

3.​ Click Add data source > Select Prometheus​

4.​ Set the URL to: http://prometheus:9090​

5.​ Click Save & Test​

●​ Recreate the entire setup​

●​ Configure Prometheus & Grafana​

●​ Connect data source and build a dashboard​

●​ Troubleshoot any issues​

Takeaways
●​ Monitoring is for system health, logging is for deep insights.​

●​ Prometheus pulls metrics, Grafana visualizes them.​

●​ Docker Compose makes local setup simple and consistent.​

●​ Metrics are essential for observability, alerts, and SLA monitoring.​

​
Resources
●​ Prometheus Docs: https://prometheus.io/docs/​

●​ Grafana Docs: https://grafana.com/docs/​

●​ Docker Compose: https://docs.docker.com/compose/​

Portainer and Jenkins CI/CD

Setting Up and Using Portainer (Docker Management UI)

What is Portainer?

Portainer is a lightweight, open-source management UI that makes it easier to manage Docker

environments visually. It helps users create and manage containers, images, volumes, and
networks using a graphical interface.

Why Use Portainer?

●​ Simplifies Docker management for beginners​

●​ Avoids command-line complexity​

●​ Offers container health and activity monitoring​

●​ Role-based access control for teams (in paid version)​

Step-by-Step: Installing Portainer with Docker Compose

1. Create a directory and YAML file:
mkdir portainer && cd portainer
touch docker-compose.yml

2. Add the following content into docker-compose.yml:

version: '3.3'

services:
portainer:
image: portainer/portainer-ce:latest
container_name: portainer
ports:
- "9000:9000"
volumes:
- "/var/run/docker.sock:/var/run/docker.sock"
- portainer_data:/data
restart: unless-stopped

volumes:
portainer_data:

3. Deploy Portainer:
docker compose up -d

4. Access the Portainer UI:

●​ Open browser and go to http://localhost:9000​

●​ Set an admin password​

●​ Choose Local Docker Environment​

●​ Click Connect​

Navigating Portainer UI:

Main Dashboard:

●​ Overview of container count, volume usage, images, etc.​

Containers Tab:

●​ Click on Containers in the left menu to see all running/stopped containers​

●​ Options:​

○​ Start/Stop​

○​ View Logs​

○​ Inspect settings​

○​ Create a new container​

Images:

●​ View all available Docker images​

●​ Pull new images directly from Docker Hub​

Volumes:

●​ Inspect data volumes used by containers​

🔍 Tip: Use Portainer to observe how Jenkins and other Docker services behave
after you set them up.

Part 2: Jenkins CI/CD with GitHub or GitLab

What is Jenkins?
Jenkins is an automation server used to build, test, and deploy software automatically. It is
often used in CI/CD pipelines.

Step-by-Step: Jenkins Setup with Docker Compose

1. Create a Jenkins directory and YAML file:

mkdir jenkins && cd jenkins
touch docker-compose.yml

2. Add this to docker-compose.yml:

version: '3'

services:
jenkins:
image: jenkins/jenkins:lts
container_name: jenkins
user: root
ports:
- "8080:8080"
- "50000:50000"
volumes:
- jenkins_home:/var/jenkins_home
- /var/run/docker.sock:/var/run/docker.sock
restart: unless-stopped

volumes:
jenkins_home:

3. Start Jenkins:
docker compose up -d

4. Initial Setup:

●​ Open http://localhost:8080​
 ●​ Run this to get the setup password:​

docker exec jenkins cat /var/jenkins_home/secrets/initialAdminPassword

●​ Paste the password, proceed with setup​

●​ Install suggested plugins​

Integrate Jenkins with GitHub/GitLab

For GitHub:

1.​ In GitHub, create a Personal Access Token (Settings → Developer Settings → Tokens)​

2.​ In Jenkins → Manage Jenkins → Credentials → Add new credential:​

○​ Type: Username & Password​

○​ Username: GitHub username​

○​ Password: Token​

For GitLab:

1.​ Create GitLab access token (Settings → Access Tokens)​

2.​ Add to Jenkins Credentials (same method as GitHub)​

3.​ Install GitLab Plugin in Jenkins​

4.​ Configure GitLab connection in Manage Jenkins → Configure System → GitLab​

Create a Simple CI/CD Pipeline (Jenkins + Git Repo)

1. Create a Freestyle Job in Jenkins:

●​ Click New Item → ci-cd-demo → Freestyle project​

2. Connect to GitHub/GitLab repo:

●​ Select Git for Source Code Management​

●​ Add repo URL and choose credentials​

3. Add Build Steps:

●​ Select Execute Shell​

echo "Linting Code (simulated)..."

echo "Running Unit Tests..."
chmod +x hello.sh
./hello.sh
echo "Build Step (Simulated)"
echo "Deploy Step Done"

4. Trigger:

●​ Set Build Triggers to poll SCM or webhook from GitHub/GitLab (recommended for live
sync)​

5. Save and Click Build Now

You’ve successfully run your first CI/CD pipeline!