First of all thanks for a great piece of software and the willingness to make it better. We currently have multiple Kubernetes clusters in our environment and wanted to share some improvements I would like to see in how netdata detects and monitors Kubernetes/docker containers and how that relates to cgroups.
First improvement I believe is that netdata should have it's own "Kubernetes" application section in the UI with sub groups/charts for PODs and containers and not pollute the right side menu bar. There is also enough concept differences between Docker and Docker running on K8s that warrants this as well in my opinion. So the way I see it working is you would have a K8s application, under that you have a list of POD names, and then under the PODs you have your list of containers. I'm not sure if netdata can support this or not.
Second improvement is netdata adds unnecessary containers/PODs that do not need to be monitored and pollutes the UI as well as external systems like prometheus that pulls data from it.
Third improvement revolves around how netdata is naming K8s objects (PODs, containers, etc).
CGROUP and K8s PODs/Containers
So when correlating containers/pods to cgroups, it's important to understand QoS classes that K8s assigns to PODs: Guaranteed, Burstable, or BestEffort. This directly relates to the full path a containers cgroup will live in. For example, let's assume I'm in the following working directory /sys/fs/cgroup/cpu/kubepods and I do an ls:
drwxr-xr-x. 2 root root 0 Mar 2 15:31 besteffort
drwxr-xr-x. 5 root root 0 Mar 8 10:38 burstable
-rw-r--r--. 1 root root 0 Mar 2 15:17 cgroup.clone_children
-rw-r--r--. 1 root root 0 Mar 2 15:17 cgroup.procs
-r--r--r--. 1 root root 0 Mar 2 15:17 cpuacct.stat
-rw-r--r--. 1 root root 0 Mar 2 15:17 cpuacct.usage
-r--r--r--. 1 root root 0 Mar 2 15:17 cpuacct.usage_percpu
-rw-r--r--. 1 root root 0 Mar 2 15:17 cpu.cfs_period_us
-rw-r--r--. 1 root root 0 Mar 2 15:17 cpu.cfs_quota_us
-rw-r--r--. 1 root root 0 Mar 2 15:17 cpu.rt_period_us
-rw-r--r--. 1 root root 0 Mar 2 15:17 cpu.rt_runtime_us
-rw-r--r--. 1 root root 0 Mar 6 13:12 cpu.shares
-r--r--r--. 1 root root 0 Mar 2 15:17 cpu.stat
-rw-r--r--. 1 root root 0 Mar 2 15:17 notify_on_release
drwxr-xr-x. 4 root root 0 Mar 8 09:41 pod69ef92e5-1e6a-11e8-ab3b-06a6795d02ec
drwxr-xr-x. 4 root root 0 Mar 8 09:41 pod87bbe673-1e6a-11e8-ab3b-06a6795d02ec
drwxr-xr-x. 5 root root 0 Mar 2 15:31 pod9cb7c6f6-029b-11e8-8ba7-063fdff58aa0
drwxr-xr-x. 4 root root 0 Mar 8 09:41 podffedb595-22dd-11e8-be53-06a6795d02ec
-rw-r--r--. 1 root root 0 Mar 2 15:17 tasks
You'll notice the two directories besteffort and burstable and any pod[a-f][0-9] sub-directories under those have that QoS class applied to it. Any pod directories currently listed in the output above have the guaranteed QoS applied to it.
Issue 1: The issue right now is that netdata adds besteffort and burstable as a container to monitor and that should not be the case. We only care about the POD's and the containers that live under them and that's it. So what we want to do is monitor the cgroup folders with just pod[a-f][0-9] in the folder name which gives us an aggregated view of performance of all containers running under that pod and then the container ID's in sub-folders gives us the actual metrics for those specific containers.
So let's cd into the burstable directory and see if we have any pod directories in there:
-rw-r--r--. 1 root root 0 Mar 2 16:51 cgroup.clone_children
-rw-r--r--. 1 root root 0 Mar 2 16:51 cgroup.procs
-r--r--r--. 1 root root 0 Mar 2 16:51 cpuacct.stat
-rw-r--r--. 1 root root 0 Mar 2 16:51 cpuacct.usage
-r--r--r--. 1 root root 0 Mar 2 16:51 cpuacct.usage_percpu
-rw-r--r--. 1 root root 0 Mar 2 16:51 cpu.cfs_period_us
-rw-r--r--. 1 root root 0 Mar 2 16:51 cpu.cfs_quota_us
-rw-r--r--. 1 root root 0 Mar 2 16:51 cpu.rt_period_us
-rw-r--r--. 1 root root 0 Mar 2 16:51 cpu.rt_runtime_us
-rw-r--r--. 1 root root 0 Mar 8 10:49 cpu.shares
-r--r--r--. 1 root root 0 Mar 2 16:51 cpu.stat
-rw-r--r--. 1 root root 0 Mar 2 16:51 notify_on_release
drwxr-xr-x. 4 root root 0 Mar 2 17:38 pod6a837235-1e6a-11e8-ab3b-06a6795d02ec
drwxr-xr-x. 5 root root 0 Mar 8 10:38 podb7d6aa9f-22e6-11e8-be53-06a6795d02ec
-rw-r--r--. 1 root root 0 Mar 2 16:51 tasks
So we have two pod directories pod6a837235-1e6a-11e8-ab3b-06a6795d02ec and podb7d6aa9f-22e6-11e8-be53-06a6795d02ec so let's cd into pod6a837235-1e6a-11e8-ab3b-06a6795d02ec:
drwxr-xr-x. 2 root root 0 Mar 8 10:56 98a2905a85d282046fb5c65580c6e066044bd5be740c3909132607763a9eaf1b
-rw-r--r--. 1 root root 0 Mar 8 09:59 cgroup.clone_children
-rw-r--r--. 1 root root 0 Mar 8 09:59 cgroup.procs
-r--r--r--. 1 root root 0 Mar 8 09:59 cpuacct.stat
-rw-r--r--. 1 root root 0 Mar 8 09:59 cpuacct.usage
-r--r--r--. 1 root root 0 Mar 8 09:59 cpuacct.usage_percpu
-rw-r--r--. 1 root root 0 Mar 8 09:59 cpu.cfs_period_us
-rw-r--r--. 1 root root 0 Mar 8 09:59 cpu.cfs_quota_us
-rw-r--r--. 1 root root 0 Mar 8 09:59 cpu.rt_period_us
-rw-r--r--. 1 root root 0 Mar 8 09:59 cpu.rt_runtime_us
-rw-r--r--. 1 root root 0 Mar 2 17:38 cpu.shares
-r--r--r--. 1 root root 0 Mar 8 09:59 cpu.stat
drwxr-xr-x. 2 root root 0 Mar 8 10:56 e0407748d982456c509fc579676ada5a5953c48507c40642d1c67df788d63e7d
-rw-r--r--. 1 root root 0 Mar 8 09:59 notify_on_release
-rw-r--r--. 1 root root 0 Mar 8 09:59 tasks
So we have two containers running in this POD identified by 98a2905a85d282046fb5c65580c6e066044bd5be740c3909132607763a9eaf1b and e0407748d982456c509fc579676ada5a5953c48507c40642d1c67df788d63e7d. These are the actual IDs for the containers and should be used when calling the K8s api or the docker sock and brings up multiple issues with how netdata implements this:
- Issue 1: Netdata by default uses the docker classic call to try to retrieve information about a container and this should never happen when dealing with K8s docker containers. We need to change this so that whenever kubepods is in the cgroup file path or some other method to detect it's a k8s container, that it always uses the API/docker sock like you have defined.
- Issue 2: The evaluation expression here, https://github.com/firehol/netdata/blob/master/plugins.d/cgroup-name.sh#L129, does not work for all names. I added
set -x to the top of that script and ran the following:
This works:
[root@localhost] /usr/libexec/netdata/plugins.d/cgroup-name.sh kubepods_burstable_pod6a837235-1e6a-11e8-ab3b-06a6795d02ec_51dfbb5816137555bbe237aec3ceaf54f1e7e570bd9c0005b79e1b3d93278c55
+ CGROUP=kubepods_burstable_pod6a837235-1e6a-11e8-ab3b-06a6795d02ec_51dfbb5816137555bbe237aec3ceaf54f1e7e570bd9c0005b79e1b3d93278c55
+ NAME=
+ '[' -z kubepods_burstable_pod6a837235-1e6a-11e8-ab3b-06a6795d02ec_51dfbb5816137555bbe237aec3ceaf54f1e7e570bd9c0005b79e1b3d93278c55 ']'
+ '[' -f /usr/libexec/netdata/plugins.d/../../../../etc/netdata/cgroups-names.conf ']'
+ '[' -z '' ']'
+ [[ kubepods_burstable_pod6a837235-1e6a-11e8-ab3b-06a6795d02ec_51dfbb5816137555bbe237aec3ceaf54f1e7e570bd9c0005b79e1b3d93278c55 =~ ^.*docker[-_/\.][a-fA-F0-9]+[-_\.]?.*$ ]]
+ [[ kubepods_burstable_pod6a837235-1e6a-11e8-ab3b-06a6795d02ec_51dfbb5816137555bbe237aec3ceaf54f1e7e570bd9c0005b79e1b3d93278c55 =~ ^.*kubepods[_/].*[_/]pod[a-fA-F0-9-]+[_/][a-fA-F0-9]+$ ]]
++ echo kubepods_burstable_pod6a837235-1e6a-11e8-ab3b-06a6795d02ec_51dfbb5816137555bbe237aec3ceaf54f1e7e570bd9c0005b79e1b3d93278c55
++ sed 's|^.*kubepods[_/].*[_/]pod[a-fA-F0-9-]\+[_/]\([a-fA-F0-9]\+\)$|\1|'
+ DOCKERID=51dfbb5816137555bbe237aec3ceaf54f1e7e570bd9c0005b79e1b3d93278c55
+ echo DOCKERID=51dfbb5816137555bbe237aec3ceaf54f1e7e570bd9c0005b79e1b3d93278c55
DOCKERID=51dfbb5816137555bbe237aec3ceaf54f1e7e570bd9c0005b79e1b3d93278c55
+ '[' '!' -z 51dfbb5816137555bbe237aec3ceaf54f1e7e570bd9c0005b79e1b3d93278c55 -a '(' 64 -eq 64 -o 64 -eq 12 ')' ']'
+ docker_get_name_api 51dfbb5816137555bbe237aec3ceaf54f1e7e570bd9c0005b79e1b3d93278c55
+ local id=51dfbb5816137555bbe237aec3ceaf54f1e7e570bd9c0005b79e1b3d93278c55
This Does Not Work:
[root@localhost] /usr/libexec/netdata/plugins.d/cgroup-name.sh kubepods_pod69ef92e5-1e6a-11e8-ab3b-06a6795d02ec_70494ddb81ed7f0833570af95ef6fd584694b73b54cc472e1ab3b4bef9732166
+ CGROUP=kubepods_pod69ef92e5-1e6a-11e8-ab3b-06a6795d02ec_70494ddb81ed7f0833570af95ef6fd584694b73b54cc472e1ab3b4bef9732166
+ NAME=
+ '[' -z kubepods_pod69ef92e5-1e6a-11e8-ab3b-06a6795d02ec_70494ddb81ed7f0833570af95ef6fd584694b73b54cc472e1ab3b4bef9732166 ']'
+ '[' -f /usr/libexec/netdata/plugins.d/../../../../etc/netdata/cgroups-names.conf ']'
+ '[' -z '' ']'
+ [[ kubepods_pod69ef92e5-1e6a-11e8-ab3b-06a6795d02ec_70494ddb81ed7f0833570af95ef6fd584694b73b54cc472e1ab3b4bef9732166 =~ ^.*docker[-_/\.][a-fA-F0-9]+[-_\.]?.*$ ]]
+ [[ kubepods_pod69ef92e5-1e6a-11e8-ab3b-06a6795d02ec_70494ddb81ed7f0833570af95ef6fd584694b73b54cc472e1ab3b4bef9732166 =~ ^.*kubepods[_/].*[_/]pod[a-fA-F0-9-]+[_/][a-fA-F0-9]+$ ]]
+ [[ kubepods_pod69ef92e5-1e6a-11e8-ab3b-06a6795d02ec_70494ddb81ed7f0833570af95ef6fd584694b73b54cc472e1ab3b4bef9732166 =~ machine.slice[_/].*\.service ]]
+ [[ kubepods_pod69ef92e5-1e6a-11e8-ab3b-06a6795d02ec_70494ddb81ed7f0833570af95ef6fd584694b73b54cc472e1ab3b4bef9732166 =~ machine.slice_machine.*-qemu ]]
+ [[ kubepods_pod69ef92e5-1e6a-11e8-ab3b-06a6795d02ec_70494ddb81ed7f0833570af95ef6fd584694b73b54cc472e1ab3b4bef9732166 =~ machine_.*\.libvirt-qemu ]]
+ [[ kubepods_pod69ef92e5-1e6a-11e8-ab3b-06a6795d02ec_70494ddb81ed7f0833570af95ef6fd584694b73b54cc472e1ab3b4bef9732166 =~ qemu.slice_([0-9]+).scope ]]
+ [[ kubepods_pod69ef92e5-1e6a-11e8-ab3b-06a6795d02ec_70494ddb81ed7f0833570af95ef6fd584694b73b54cc472e1ab3b4bef9732166 =~ lxc_([0-9]+) ]]
Filtering out the pause container
So when detecting cgroups and k8s containers, there's one container type that needs to be filtered out from being detected and monitored and that's the K8s pause container. More information about it can be found here https://www.ianlewis.org/en/almighty-pause-container. In summary, the pause container in K8s serves as the "parent container" for all of the containers in your pod. It's basically the PID 1 for each pod and reaps zombie processes, that's it. We do not want to monitor it and only adds confusion to the list of containers in netdata.
So using an example, here's how you would detect this. Let's assume we are still currently in the pod6a837235-1e6a-11e8-ab3b-06a6795d02ec cgroup directory:
drwxr-xr-x. 2 root root 0 Mar 8 10:56 98a2905a85d282046fb5c65580c6e066044bd5be740c3909132607763a9eaf1b
-rw-r--r--. 1 root root 0 Mar 8 09:59 cgroup.clone_children
-rw-r--r--. 1 root root 0 Mar 8 09:59 cgroup.procs
-r--r--r--. 1 root root 0 Mar 8 09:59 cpuacct.stat
-rw-r--r--. 1 root root 0 Mar 8 09:59 cpuacct.usage
-r--r--r--. 1 root root 0 Mar 8 09:59 cpuacct.usage_percpu
-rw-r--r--. 1 root root 0 Mar 8 09:59 cpu.cfs_period_us
-rw-r--r--. 1 root root 0 Mar 8 09:59 cpu.cfs_quota_us
-rw-r--r--. 1 root root 0 Mar 8 09:59 cpu.rt_period_us
-rw-r--r--. 1 root root 0 Mar 8 09:59 cpu.rt_runtime_us
-rw-r--r--. 1 root root 0 Mar 2 17:38 cpu.shares
-r--r--r--. 1 root root 0 Mar 8 09:59 cpu.stat
drwxr-xr-x. 2 root root 0 Mar 8 10:56 e0407748d982456c509fc579676ada5a5953c48507c40642d1c67df788d63e7d
-rw-r--r--. 1 root root 0 Mar 8 09:59 notify_on_release
-rw-r--r--. 1 root root 0 Mar 8 09:59 tasks
Using the docker.sock, we are going to make an api call and return the .Config.Image json value for container id 98a2905a85d282046fb5c65580c6e066044bd5be740c3909132607763a9eaf1b:
echo -e "GET /containers/98a2905a85d282046fb5c65580c6e066044bd5be740c3909132607763a9eaf1b/json HTTP/1.0\r\n" | nc -U /var/run/docker.sock |grep '^{.*' | jq '.Config.Image'
"gcr.io/google_containers/pause-amd64:3.0"
Any container that contains this Image name needs to be filtered out and prevented from being monitored by netdata.
POD and Container Naming
It appears that you are trying to use the .Name json field for the container name but I would suggest a slightly better alternative. Using the container ID in our previous examples, here's the .Name field that netdata tries to use:
echo -e "GET /containers/51dfbb5816137555bbe237aec3ceaf54f1e7e570bd9c0005b79e1b3d93278c55/json HTTP/1.0\r\n" | nc -U /var/run/docker.sock |grep '^{.*' | jq '.Name'
"/k8s_app_example-app-5b8d7c86f8-ht678_example-app_6a837235-1e6a-11e8-ab3b-06a6795d02ec_32"
This name field is fine for the chart label but not for the actual container name as it's too long and confusing when viewing in the netdata UI. Let's look at the .Config.Labels:
echo -e "GET /containers/51dfbb5816137555bbe237aec3ceaf54f1e7e570bd9c0005b79e1b3d93278c55/json HTTP/1.0\r\n" | nc -U /var/run/docker.sock |grep '^{.*' | jq '.Config.Labels'
{
"annotation.io.kubernetes.container.hash": "87d9370c",
"annotation.io.kubernetes.container.preStopHandler": "{\"exec\":{\"command\":[\"sleep\",\"15\"]}}",
"annotation.io.kubernetes.container.restartCount": "32",
"annotation.io.kubernetes.container.terminationMessagePath": "/dev/termination-log",
"annotation.io.kubernetes.container.terminationMessagePolicy": "File",
"annotation.io.kubernetes.pod.terminationGracePeriod": "60",
"build-date": "20180107",
"io.kubernetes.container.logpath": "/var/log/pods/6a837235-1e6a-11e8-ab3b-06a6795d02ec/app_32.log",
"io.kubernetes.container.name": "app",
"io.kubernetes.docker.type": "container",
"io.kubernetes.pod.name": "example-app-5b8d7c86f8-ht678",
"io.kubernetes.pod.namespace": "example-app",
"io.kubernetes.pod.uid": "6a837235-1e6a-11e8-ab3b-06a6795d02ec",
"io.kubernetes.sandbox.id": "98a2905a85d282046fb5c65580c6e066044bd5be740c3909132607763a9eaf1b",
"license": "GPLv2",
"name": "CentOS Base Image",
"vendor": "CentOS"
}
Let's break this down into the different names we need:
- POD Name: When showing a POD name in the UI it should be using the
"io.kubernetes.pod.name". We also need "io.kubernetes.pod.uid" but that should only get included in the chart label for prometheus.
- Namespace Name: This is obviously the
"io.kubernetes.pod.namespace" field. This needs to be added to the chart label for prometheus.
- Container Name: Use the
"io.kubernetes.container.name" field.
The .Name field is actually a combination of these with a prefix of k8s. So for example, this is how netdata currently is naming things from a prometheus viewpoint in the chart label:
netdata_cgroup_mem_usage_MB_average{chart="cgroup_kubepods_pod87bbe673_1e6a_11e8_ab3b_06a6795d02ec.mem_usage",family="mem",dimension="ram"}
What I would like to see for the chart label is more like this: {container_name}_{pod_name}_{namespace}_{pod_uid}
Which would then translate to:
app_example-app-5b8d7c86f8-ht678_example-app_6a837235-1e6a-11e8-ab3b-06a6795d02ec
We also need to change the metric name as well if we are going to create a netdata application for K8s. So the full metric name with prometheus labels would look something like this:
netdata_k8s_mem_usage_MB_average{chart="app_example-app-5b8d7c86f8-ht678_example-app_6a837235-1e6a-11e8-ab3b-06a6795d02ec",family="mem",dimension="ram"}
If for some unknown reason to me that changing the metric name is not feasible, than I would suggest pre-fixing the chart name with a k8s which gets us back to what the .Name field almost provides us minus the "annotation.io.kubernetes.container.restartCount" at the end:
netdata_cgroup_mem_usage_MB_average{chart="k8s_app_example-app-5b8d7c86f8-ht678_example-app_6a837235-1e6a-11e8-ab3b-06a6795d02ec",family="mem",dimension="ram"}
I know this is a lot of information and hopefully most of it makes sense. Please let me know if you need any additional information and would love to work with you in making netdata better about monitoring K8s. Thanks!
First of all thanks for a great piece of software and the willingness to make it better. We currently have multiple Kubernetes clusters in our environment and wanted to share some improvements I would like to see in how netdata detects and monitors Kubernetes/docker containers and how that relates to cgroups.
First improvement I believe is that netdata should have it's own "Kubernetes" application section in the UI with sub groups/charts for PODs and containers and not pollute the right side menu bar. There is also enough concept differences between Docker and Docker running on K8s that warrants this as well in my opinion. So the way I see it working is you would have a K8s application, under that you have a list of POD names, and then under the PODs you have your list of containers. I'm not sure if netdata can support this or not.
Second improvement is netdata adds unnecessary containers/PODs that do not need to be monitored and pollutes the UI as well as external systems like prometheus that pulls data from it.
Third improvement revolves around how netdata is naming K8s objects (PODs, containers, etc).
CGROUP and K8s PODs/Containers
So when correlating containers/pods to cgroups, it's important to understand QoS classes that K8s assigns to PODs: Guaranteed, Burstable, or BestEffort. This directly relates to the full path a containers cgroup will live in. For example, let's assume I'm in the following working directory
/sys/fs/cgroup/cpu/kubepodsand I do anls:You'll notice the two directories besteffort and burstable and any
pod[a-f][0-9]sub-directories under those have that QoS class applied to it. Any pod directories currently listed in the output above have the guaranteed QoS applied to it.Issue 1: The issue right now is that netdata adds besteffort and burstable as a container to monitor and that should not be the case. We only care about the POD's and the containers that live under them and that's it. So what we want to do is monitor the cgroup folders with just
pod[a-f][0-9]in the folder name which gives us an aggregated view of performance of all containers running under that pod and then the container ID's in sub-folders gives us the actual metrics for those specific containers.So let's
cdinto the burstable directory and see if we have any pod directories in there:So we have two pod directories
pod6a837235-1e6a-11e8-ab3b-06a6795d02ecandpodb7d6aa9f-22e6-11e8-be53-06a6795d02ecso let's cd intopod6a837235-1e6a-11e8-ab3b-06a6795d02ec:So we have two containers running in this POD identified by
98a2905a85d282046fb5c65580c6e066044bd5be740c3909132607763a9eaf1bande0407748d982456c509fc579676ada5a5953c48507c40642d1c67df788d63e7d. These are the actual IDs for the containers and should be used when calling the K8s api or the docker sock and brings up multiple issues with how netdata implements this:docker_get_name_api "${DOCKERID}".set -xto the top of that script and ran the following:This works:
This Does Not Work:
.Namefield is not the best one to use and I make suggestions below in my "POD and Container Naming" sectionFiltering out the pause container
So when detecting cgroups and k8s containers, there's one container type that needs to be filtered out from being detected and monitored and that's the K8s pause container. More information about it can be found here https://www.ianlewis.org/en/almighty-pause-container. In summary, the pause container in K8s serves as the "parent container" for all of the containers in your pod. It's basically the PID 1 for each pod and reaps zombie processes, that's it. We do not want to monitor it and only adds confusion to the list of containers in netdata.
So using an example, here's how you would detect this. Let's assume we are still currently in the
pod6a837235-1e6a-11e8-ab3b-06a6795d02eccgroup directory:Using the docker.sock, we are going to make an api call and return the
.Config.Imagejson value for container id98a2905a85d282046fb5c65580c6e066044bd5be740c3909132607763a9eaf1b:Any container that contains this Image name needs to be filtered out and prevented from being monitored by netdata.
POD and Container Naming
It appears that you are trying to use the .Name json field for the container name but I would suggest a slightly better alternative. Using the container ID in our previous examples, here's the
.Namefield that netdata tries to use:This name field is fine for the chart label but not for the actual container name as it's too long and confusing when viewing in the netdata UI. Let's look at the
.Config.Labels:Let's break this down into the different names we need:
"io.kubernetes.pod.name". We also need"io.kubernetes.pod.uid"but that should only get included in the chart label for prometheus."io.kubernetes.pod.namespace"field. This needs to be added to the chart label for prometheus."io.kubernetes.container.name"field.The
.Namefield is actually a combination of these with a prefix ofk8s. So for example, this is how netdata currently is naming things from a prometheus viewpoint in the chart label:What I would like to see for the chart label is more like this:
{container_name}_{pod_name}_{namespace}_{pod_uid}Which would then translate to:
app_example-app-5b8d7c86f8-ht678_example-app_6a837235-1e6a-11e8-ab3b-06a6795d02ecWe also need to change the metric name as well if we are going to create a netdata application for K8s. So the full metric name with prometheus labels would look something like this:
If for some unknown reason to me that changing the metric name is not feasible, than I would suggest pre-fixing the chart name with a
k8swhich gets us back to what the.Namefield almost provides us minus the"annotation.io.kubernetes.container.restartCount"at the end:I know this is a lot of information and hopefully most of it makes sense. Please let me know if you need any additional information and would love to work with you in making netdata better about monitoring K8s. Thanks!