cgmgr: create cgroups for systemd cgroup driver for dropped infra pods #6856

haircommander · 2023-04-28T20:32:40Z

 The history here is a bit convoluted. Originally, runc created the cgroup for the infra container.
cAdvisor was built to assume the cgroup for the infra container would be created, and it uses
this to find the network metrics for the pod. When we dropped the infra container, cri-o needed to make this
cgroup so cAdvisor could still find the network metrics.

However, systemd didn't like the way we did it, and would remove the cgroup mid pod creation, which was fixed in
https://github.com/cri-o/cri-o/pull/6196. This actually caused the cgroup to not be created at all, which then caused
the networking metrics to not be gathered at all.

Thus, we do need to create a cgroup underneath the systemd cgroup. Attempt to use a slice for this, as systemd won't require a
process be underneath it.

What type of PR is this?

/kind bug

What this PR does / why we need it:

reverts #6196
fixes #6657

Which issue(s) this PR fixes:

Special notes for your reviewer:

Does this PR introduce a user-facing change?

Fix a bug where network metrics collection is broken with systemd cgroup driver and dropped infra containers.

haircommander · 2023-04-28T20:34:43Z

/hold

I think there are some cgroupfs issues here and I haven't tested thoroughly with cgroupv2 yet either

codecov · 2023-04-28T20:55:26Z

Codecov Report

Merging #6856 (660b63b) into main (1dcce7a) will decrease coverage by 0.10%.
The diff coverage is 5.66%.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #6856      +/-   ##
==========================================
- Coverage   49.73%   49.64%   -0.10%     
==========================================
  Files         127      127              
  Lines       14984    15040      +56     
==========================================
+ Hits         7453     7466      +13     
- Misses       6644     6686      +42     
- Partials      887      888       +1

kolyshkin · 2023-05-01T16:04:12Z

internal/config/cgmgr/cgmgr.go

 	// CreateSandboxCgroup takes the sandbox parent, and sandbox ID.
 	// It creates a new cgroup for that sandbox, which is useful when spoofing an infra container.
 	CreateSandboxCgroup(sbParent, containerID string) error
+	// CreateSandboxCgroup takes the sandbox parent, and sandbox ID.


typo: s/CreateSandboxCgroup/RemoveSandboxCgroup/.

Perhaps we should use a linter to catch this. Revive would do (alas enabling it for the whole repo will cause tons of warnings)

good catch!

internal/config/cgmgr/cgmgr.go

internal/config/cgmgr/systemd.go

haircommander · 2023-05-01T19:35:01Z

/hold cancel

haircommander · 2023-05-02T16:12:05Z

after a conversation with @kolyshkin offline, we opted to have crio create a slice for the infra container, as systemd won't require a process be underneath it.

test/pod.bats

internal/config/cgmgr/systemd.go

kolyshkin · 2023-05-02T19:46:14Z

test/cgroups.bats

+	ls "$cgroup_base"/"$parent"/crio-"$pod_id"*
+
+	crictl rmp -fa
+	! ls "$cgroup_base"/"$parent"/crio-"$pod_id"*


Offtopic, but that reminds I recently learned that this form won't work, unless this is the last statement in a function. I'll send a fix.

kolyshkin · 2023-05-02T19:47:08Z

/lgtm

kolyshkin · 2023-05-02T19:48:45Z

So, I would like to have it tested before the merge, to make sure

the issue (Networking metrics missing with cri-o 1.26 #6657) is solved
the older bug (1.24.3 leaks pidns with cgroup v2 #6349) is not reintroduced

kolyshkin

LGTM

haircommander · 2023-05-02T19:55:20Z

/hold

to double check

* the issue ([Networking metrics missing with cri-o 1.26 #6657](https://github.com/cri-o/cri-o/issues/6657)) is solved

* the older bug ([1.24.3 leaks pidns with cgroup v2 #6349](https://github.com/cri-o/cri-o/issues/6349)) is not reintroduced

haircommander · 2023-05-02T20:35:09Z

/retest

The history here is a bit convoluted. Originally, runc created the cgroup for the infra container. cAdvisor was built to assume the cgroup for the infra container would be created, and it uses this to find the network metrics for the pod. When we dropped the infra container, cri-o needed to make this cgroup so cAdvisor could still find the network metrics. However, systemd didn't like the way we did it, and would remove the cgroup mid pod creation, which was fixed in cri-o#6196. This actually caused the cgroup to not be created at all, which then caused the networking metrics to not be gathered at all. Thus, we do need to create a cgroup underneath the systemd cgroup. Attempt to use a slice for this, as systemd won't require a process be underneath it. Signed-off-by: Peter Hunt <[email protected]>

- skip test for cgroupfs - remove skip for runc 1.0.0-rc11 (very old now) - drop removal of cgroup parent (not required) Signed-off-by: Peter Hunt <[email protected]>

since there will be no processes in this cgroup, it won't cause any harm, but it will enable cpu load balancing through cgroups. Signed-off-by: Peter Hunt <[email protected]>

haircommander · 2023-05-03T18:35:17Z

/hold cancel

the issue (Networking metrics missing with cri-o 1.26 #6657) is solved

I now see the network metrics

the older bug ( 1.24.3 leaks pidns with cgroup v2 #6349) is not reintroduced

I don't see a pid cgroup being created. I think using libcontainer has fixed this

I also have monitored the cgroup for a bit and i don't see systemd cleaning it up. I think we're all set

kolyshkin

/lgtm

openshift-ci · 2023-05-03T20:18:05Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: haircommander, kolyshkin

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

~~OWNERS~~ [haircommander,kolyshkin]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

haircommander · 2023-05-03T20:29:46Z

/cherry-pick release-1.27

openshift-cherrypick-robot · 2023-05-03T20:29:48Z

@haircommander: once the present PR merges, I will cherry-pick it on top of release-1.27 in a new PR and assign it to you.

Details

In response to this:

/cherry-pick release-1.27

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-cherrypick-robot · 2023-05-03T20:34:54Z

@haircommander: new pull request created: #6873

Details

In response to this:

/cherry-pick release-1.27

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

haircommander requested a review from mrunalp as a code owner April 28, 2023 20:32

openshift-ci bot added release-note Denotes a PR that will be considered when it comes time to generate release notes. kind/bug Categorizes issue or PR as related to a bug. dco-signoff: yes Indicates the PR's author has DCO signed all their commits. labels Apr 28, 2023

openshift-ci bot requested review from QiWang19 and klihub April 28, 2023 20:32

haircommander mentioned this pull request Apr 28, 2023

Networking metrics missing with cri-o 1.26 #6657

Closed

openshift-ci bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Apr 28, 2023

openshift-ci bot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Apr 28, 2023

haircommander force-pushed the infraless-create-cgroup branch 3 times, most recently from 5198c6b to 0d34440 Compare May 1, 2023 15:09

kolyshkin reviewed May 1, 2023

View reviewed changes

haircommander force-pushed the infraless-create-cgroup branch from 0d34440 to c1e34b9 Compare May 1, 2023 17:16

kolyshkin requested changes May 1, 2023

View reviewed changes

internal/config/cgmgr/cgmgr.go Outdated Show resolved Hide resolved

openshift-ci bot assigned kolyshkin May 1, 2023

haircommander force-pushed the infraless-create-cgroup branch from c1e34b9 to 8a3bd18 Compare May 1, 2023 18:06

haircommander commented May 1, 2023

View reviewed changes

internal/config/cgmgr/systemd.go Outdated Show resolved Hide resolved

haircommander force-pushed the infraless-create-cgroup branch 2 times, most recently from 18668e8 to aabcedb Compare May 1, 2023 19:00

openshift-ci bot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label May 1, 2023

haircommander force-pushed the infraless-create-cgroup branch from aabcedb to 65985d2 Compare May 2, 2023 16:10

haircommander force-pushed the infraless-create-cgroup branch 4 times, most recently from 9971ad9 to c9baba7 Compare May 2, 2023 16:32

haircommander force-pushed the infraless-create-cgroup branch from d2793af to 54468b2 Compare May 2, 2023 16:55

kolyshkin reviewed May 2, 2023

View reviewed changes

test/pod.bats Outdated Show resolved Hide resolved

haircommander force-pushed the infraless-create-cgroup branch from 54468b2 to 258bffa Compare May 2, 2023 19:28

kolyshkin reviewed May 2, 2023

View reviewed changes

internal/config/cgmgr/systemd.go Outdated Show resolved Hide resolved

kolyshkin reviewed May 2, 2023

View reviewed changes

openshift-ci bot added the lgtm Indicates that a PR is ready to be merged. label May 2, 2023

kolyshkin approved these changes May 2, 2023

View reviewed changes

openshift-ci bot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label May 2, 2023

haircommander added 3 commits May 3, 2023 14:10

test/pod.bats: update to current setup

5a1707e

- skip test for cgroupfs - remove skip for runc 1.0.0-rc11 (very old now) - drop removal of cgroup parent (not required) Signed-off-by: Peter Hunt <[email protected]>

cgmgr: set sched_load_balance to disabled on sandbox cgroup

660b63b

since there will be no processes in this cgroup, it won't cause any harm, but it will enable cpu load balancing through cgroups. Signed-off-by: Peter Hunt <[email protected]>

haircommander force-pushed the infraless-create-cgroup branch from 258bffa to 660b63b Compare May 3, 2023 18:15

openshift-ci bot removed the lgtm Indicates that a PR is ready to be merged. label May 3, 2023

openshift-ci bot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label May 3, 2023

kolyshkin approved these changes May 3, 2023

View reviewed changes

openshift-ci bot added the lgtm Indicates that a PR is ready to be merged. label May 3, 2023

openshift-merge-robot merged commit 58ab503 into cri-o:main May 3, 2023

openshift-cherrypick-robot mentioned this pull request May 3, 2023

[release-1.27] cgmgr: create cgroups for systemd cgroup driver for dropped infra pods #6873

Merged

This was referenced May 17, 2023

[1.25] cgmgr: create cgroups for systemd cgroup driver for dropped infra pods #6930

Merged

[1.24] cgmgr: create cgroups for systemd cgroup driver for dropped infra pods #6949

Merged

cgmgr: create cgroups for systemd cgroup driver for dropped infra pods #6856

cgmgr: create cgroups for systemd cgroup driver for dropped infra pods #6856

Uh oh!

Conversation

haircommander commented Apr 28, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What type of PR is this?

What this PR does / why we need it:

Which issue(s) this PR fixes:

Special notes for your reviewer:

Does this PR introduce a user-facing change?

Uh oh!

haircommander commented Apr 28, 2023

Uh oh!

codecov bot commented Apr 28, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

kolyshkin May 1, 2023

Choose a reason for hiding this comment

Uh oh!

haircommander May 1, 2023

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

haircommander commented May 1, 2023

Uh oh!

haircommander commented May 2, 2023

Uh oh!

Uh oh!

Uh oh!

kolyshkin May 2, 2023

Choose a reason for hiding this comment

Uh oh!

kolyshkin commented May 2, 2023

Uh oh!

kolyshkin commented May 2, 2023

Uh oh!

kolyshkin left a comment

Choose a reason for hiding this comment

Uh oh!

haircommander commented May 2, 2023

Uh oh!

haircommander commented May 2, 2023

Uh oh!

haircommander commented May 3, 2023

Uh oh!

kolyshkin left a comment

Choose a reason for hiding this comment

Uh oh!

openshift-ci bot commented May 3, 2023

Uh oh!

haircommander commented May 3, 2023

Uh oh!

openshift-cherrypick-robot commented May 3, 2023

Uh oh!

openshift-cherrypick-robot commented May 3, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

haircommander commented Apr 28, 2023 •

edited

Loading

codecov bot commented Apr 28, 2023 •

edited

Loading