Bug 1977100: retry setting CPU load balancing #5786

cynepco3hahue · 2022-04-06T13:31:12Z

What type of PR is this?

/kind bug

What this PR does / why we need it:

It is possible that the kernel will rebuild sched_domain related files and
because of it enabling or disabling CPU load balancing for container CPUs
will fail with different file errors:

lstat /proc/sys/kernel/sched_domain/cpu22/domain1/flags: no such file or directory
readdirent /proc/sys/kernel/sched_domain/cpu66/domain0: no such file or directory

Add retry logic around setting CPU load balancing values to reduce the possibility of such errors.

NONE

Signed-off-by: Artyom Lukianov [email protected]

openshift-ci · 2022-04-06T13:31:23Z

Hi @cynepco3hahue. Thanks for your PR.

I'm waiting for a cri-o member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

cynepco3hahue · 2022-04-06T13:32:56Z

@haircommander Can you please review it?

haircommander · 2022-04-06T13:35:26Z

internal/runtimehandlerhooks/high_performance_hooks.go

+	// TODO: re-visit once we will have some more acceptable cgroups hierarchy to disable CPU load balancing
+	// correctly via cgroups, see -https://bugzilla.redhat.com/show_bug.cgi?id=1946801
+	return wait.PollImmediate(time.Second, 5*time.Second, func() (bool, error) {
+		if err := setCPUSLoadBalancing(c, enable, schedDomainDir); err != nil {


should we only do this if os.IsNotExist(err)?

yes it makes sense

It possible that the kernel will rebuild sched_domain related files and because of it enabling or disabling CPU load balancing for container CPUs will fail with different file errors: 1. lstat /proc/sys/kernel/sched_domain/cpu22/domain1/flags: no such file or directory 2. readdirent /proc/sys/kernel/sched_domain/cpu66/domain0: no such file or directory Add retry logic around setting CPU load balancing values to reduce possibility of such errors. Signed-off-by: Artyom Lukianov <[email protected]>

haircommander · 2022-04-06T18:53:31Z

/approve

LGTM
@cri-o/cri-o-maintainers PTAL

cynepco3hahue · 2022-04-07T08:21:05Z

@haircommander Can you please add ok-to-test?

haircommander · 2022-04-07T13:41:04Z

/ok-to-test

cynepco3hahue · 2022-04-11T08:11:26Z

/retest

cynepco3hahue · 2022-04-12T12:12:02Z

/retest

cynepco3hahue · 2022-04-12T12:12:22Z

@haircommander Do you know whom can I ask for an additional review?

saschagrunert

Good one!
/lgtm

openshift-ci · 2022-04-12T14:14:41Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: cynepco3hahue, haircommander, saschagrunert

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

~~OWNERS~~ [haircommander,saschagrunert]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

cynepco3hahue · 2022-04-12T14:33:39Z

Good one!
/lgtm

Thanks!

saschagrunert · 2022-04-12T14:42:19Z

/test e2e-gcp

openshift-bot · 2022-04-12T18:35:07Z

/retest-required

Please review the full test history for this PR and help us cut down flakes.

kolyshkin · 2022-05-31T18:01:17Z

/cherry-pick release-1.21

openshift-cherrypick-robot · 2022-05-31T18:02:16Z

@kolyshkin: new pull request created: #5919

Details

In response to this:

/cherry-pick release-1.21

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

kolyshkin · 2022-05-31T18:09:24Z

/cherry-pick release-1.22
/cherry-pick release-1.23

openshift-cherrypick-robot · 2022-05-31T18:10:04Z

@kolyshkin: new pull request created: #5920

Details

In response to this:

/cherry-pick release-1.22
/cherry-pick release-1.23

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

kolyshkin · 2022-05-31T18:14:07Z

/cherry-pick release-1.23

openshift-cherrypick-robot · 2022-05-31T18:14:46Z

@kolyshkin: new pull request created: #5921

Details

In response to this:

/cherry-pick release-1.23

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

cynepco3hahue requested review from mrunalp and runcom as code owners April 6, 2022 13:31

openshift-ci bot requested review from klihub and wgahnagl April 6, 2022 13:31

cynepco3hahue force-pushed the retry_setting_cpu_load_balancing branch from c7c874d to 5502351 Compare April 6, 2022 13:32

haircommander reviewed Apr 6, 2022

View reviewed changes

cynepco3hahue force-pushed the retry_setting_cpu_load_balancing branch from 5502351 to 1098cc9 Compare April 6, 2022 16:28

openshift-ci bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Apr 6, 2022

openshift-ci bot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Apr 7, 2022

saschagrunert approved these changes Apr 12, 2022

View reviewed changes

openshift-ci bot assigned saschagrunert Apr 12, 2022

openshift-ci bot added the lgtm Indicates that a PR is ready to be merged. label Apr 12, 2022

openshift-merge-robot merged commit 9ed9393 into cri-o:main Apr 12, 2022

openshift-cherrypick-robot mentioned this pull request May 31, 2022

[release-1.21] Bug 1977100: retry setting CPU load balancing #5919

Closed

openshift-cherrypick-robot mentioned this pull request May 31, 2022

[release-1.22] Bug 1977100: retry setting CPU load balancing #5920

Closed

openshift-cherrypick-robot mentioned this pull request May 31, 2022

[release-1.23] Bug 1977100: retry setting CPU load balancing #5921

Merged

Bug 1977100: retry setting CPU load balancing #5786

Bug 1977100: retry setting CPU load balancing #5786

Uh oh!

Conversation

cynepco3hahue commented Apr 6, 2022

What type of PR is this?

What this PR does / why we need it:

Uh oh!

openshift-ci bot commented Apr 6, 2022

Uh oh!

cynepco3hahue commented Apr 6, 2022

Uh oh!

haircommander Apr 6, 2022

Choose a reason for hiding this comment

Uh oh!

cynepco3hahue Apr 6, 2022

Choose a reason for hiding this comment

Uh oh!

cynepco3hahue Apr 6, 2022

Choose a reason for hiding this comment

Uh oh!

haircommander commented Apr 6, 2022

Uh oh!

cynepco3hahue commented Apr 7, 2022

Uh oh!

haircommander commented Apr 7, 2022

Uh oh!

cynepco3hahue commented Apr 11, 2022

Uh oh!

cynepco3hahue commented Apr 12, 2022

Uh oh!

cynepco3hahue commented Apr 12, 2022

Uh oh!

saschagrunert left a comment

Choose a reason for hiding this comment

Uh oh!

openshift-ci bot commented Apr 12, 2022

Uh oh!

cynepco3hahue commented Apr 12, 2022

Uh oh!

saschagrunert commented Apr 12, 2022

Uh oh!

openshift-bot commented Apr 12, 2022

Uh oh!

kolyshkin commented May 31, 2022

Uh oh!

openshift-cherrypick-robot commented May 31, 2022

Uh oh!

kolyshkin commented May 31, 2022

Uh oh!

openshift-cherrypick-robot commented May 31, 2022

Uh oh!

kolyshkin commented May 31, 2022

Uh oh!

openshift-cherrypick-robot commented May 31, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants