Prometheus minio_cluster_capacity_* metrics do not work #11422
Description
When configuring Prometheus monitoring using provided docs https://docs.min.io/docs/how-to-monitor-minio-using-prometheus.html, the value of metrics minio_cluster_capacity_raw_free_bytes, minio_cluster_capacity_raw_total_bytes, minio_cluster_capacity_usable_free_bytes and minio_cluster_capacity_usable_total_bytes is 0. Other metrics work fine for me.
Expected Behavior
Metrics minio_cluster_capacity_raw_free_bytes, minio_cluster_capacity_raw_total_bytes, minio_cluster_capacity_usable_free_bytes and minio_cluster_capacity_usable_total_bytes should show real values of cluster capacity and free space
Current Behavior
Metrics minio_cluster_capacity_raw_free_bytes, minio_cluster_capacity_raw_total_bytes, minio_cluster_capacity_usable_free_bytes and minio_cluster_capacity_usable_total_bytes show 0 value.
Tested on versions RELEASE.2021-02-01T22-56-52Z and RELEASE.2021-01-30T00-20-58Z
Possible Solution
Steps to Reproduce (for bugs)
Run Minio distributed setup on versions RELEASE.2021-02-01T22-56-52Z or RELEASE.2021-01-30T00-20-58Z and try to collect Prometheus metrics on /minio/v2/metrics/cluster endpoint
Context
I'm trying to test Minio deployment before production and wanted to explore metrics and create dashboards. I would also like to explore how minio_cluster_capacity_raw_free_bytes metric changes as we expand distributed setup with a new pool of servers.
Regression
Your Environment
- Minio version used: RELEASE.2021-02-01T22-56-52Z and RELEASE.2021-01-30T00-20-58Z
- Operating System and version: Ubuntu 18.04 LTS - 15.0-122-generic Add json config reader/writer #124-Ubuntu
- Docker version: 20.10.3
- Prometheus version: 2.19.2
- Setup description:
4 VMs with Ubuntu 18.04 LTS and 2 disks each, using XFS:
mkfs.xfs /dev/sdc
mkfs.xfs /dev/sdd
mount -t xfs /dev/sdc /data/pool1-disk1
mount -t xfs /dev/sdd /data/pool1-disk2
Create env file /etc/minio.env for minio configuration:
MINIO_ROOT_USER=minio
MINIO_ROOT_PASSWORD=minio123
MINIO_PROMETHEUS_AUTH_TYPE="public"
Start Minio instances on each VM:
docker run -dit --rm --name minio1 \
--network=host \
--env-file /etc/minio.env \
-v /data/pool1-disk1:/data1 \
-v /data/pool1-disk2:/data2 \
minio/minio:RELEASE.2021-01-30T00-20-58Z server --address :9001 http://minio{1...4}:9001/data{1...2}
Configure Prometheus:
- job_name: minio-job-test
honor_timestamps: true
scrape_interval: 15s
scrape_timeout: 10s
metrics_path: /minio/v2/metrics/cluster
scheme: http
static_configs:
- targets:
- 10.10.10.10:9001
curl example using token:
$ curl -s -H 'Accept: application/json' -H "Authorization: Bearer eyJhbGciOiJIUzUxMiIsInR5cCI6IkpXVCJ9.eyJleHAiOjQ3NjU5NDg5NDksImlzcyI6InByb21ldGhldXMiLCJzdWIiOiJtaW5pbyJ9.d3u2HYCEbTF-ZP4i6b8X8ty6yY1PalXO8RwzaDpAfMSuFDoeSWY4GB6qyS6gtndexz4IUUa0ogJrK-TwmNPcww" http://10.10.10.10:9001/minio/v2/metrics/cluster | grep cluster
# HELP minio_cluster_capacity_raw_free_bytes Total free capacity online in the cluster.
# TYPE minio_cluster_capacity_raw_free_bytes gauge
minio_cluster_capacity_raw_free_bytes{server="minio3:9001"} 0
# HELP minio_cluster_capacity_raw_total_bytes Total capacity online in the cluster.
# TYPE minio_cluster_capacity_raw_total_bytes gauge
minio_cluster_capacity_raw_total_bytes{server="minio3:9001"} 0
# HELP minio_cluster_capacity_usable_free_bytes Total free usable capacity online in the cluster.
# TYPE minio_cluster_capacity_usable_free_bytes gauge
minio_cluster_capacity_usable_free_bytes{server="minio3:9001"} 0
# HELP minio_cluster_capacity_usable_total_bytes Total usable capacity online in the cluster.
# TYPE minio_cluster_capacity_usable_total_bytes gauge
minio_cluster_capacity_usable_total_bytes{server="minio3:9001"} 0
# HELP minio_cluster_disk_offline_total Total disks offline.
# TYPE minio_cluster_disk_offline_total gauge
minio_cluster_disk_offline_total{server="minio3:9001"} 0
# HELP minio_cluster_disk_online_total Total disks online.
# TYPE minio_cluster_disk_online_total gauge
minio_cluster_disk_online_total{server="minio3:9001"} 16
# HELP minio_cluster_nodes_offline_total Total number of MinIO nodes offline.
# TYPE minio_cluster_nodes_offline_total gauge
minio_cluster_nodes_offline_total{server="minio3:9001"} 0
# HELP minio_cluster_nodes_online_total Total number of MinIO nodes online.
# TYPE minio_cluster_nodes_online_total gauge
minio_cluster_nodes_online_total{server="minio3:9001"} 8