handle irqbalance service #4441

pperiyasamy · 2020-12-17T13:00:48Z

This would manipulate irqbalance service if it's already running on the system to exclude assigned pod CPUs from IRQ balancing.

Signed-off-by: Periyasamy Palanisamy [email protected]

What type of PR is this?

/kind api-change
/kind bug
/kind ci
/kind cleanup
/kind dependency-change
/kind deprecation
/kind design
/kind documentation
/kind failing-test
/kind feature
/kind flake
/kind other

What this PR does / why we need it:

The crio currently runs irqbalance --oneshot command to exclude pod cpus from handling interrupts, but this may not be a right choice if irqbalance service is already running on the system. This PR aims to address that concern by restarting the service after updating irqbalance config file with required settings for IRQBALANCE_BANNED_CPUS parameter.

Which issue(s) this PR fixes:

None

Special notes for your reviewer:

This change is not yet tested, after a initial round of reviews would go for testing out this change.

Does this PR introduce a user-facing change?

When using high performance hooks, CRI-O now restarts the irqbalance service after updating the irqbalance config file. rather than calling `irqbalance --oneshot`. A new config value `irqbalance_config_file` has been introduced to configure the file to update with `IRQBALANCE_BANNED_CPUS` settings. The default of this config value is `/etc/sysconfig/irqbalance`, but must be set to `/etc/default/irqbalance` for Ubuntu-like distributions.

openshift-ci-robot · 2020-12-17T13:00:56Z

Hi @pperiyasamy. Thanks for your PR.

I'm waiting for a cri-o member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

pperiyasamy · 2020-12-17T13:04:44Z

/cc @JanScheurich

TomSweeneyRedHat · 2020-12-17T14:51:24Z

@pperiyasamy looks like you've some lint issues.

TomSweeneyRedHat · 2020-12-17T15:01:54Z

Any tests to add or at least to modify?

pperiyasamy · 2020-12-17T15:38:43Z

@TomSweeneyRedHat Thanks, golint issues are fixed now, I'm waiting for the initial feedback on the changes, if that is finalised, then I'm planning to test this on my local setup and add/modify test cases.

haircommander · 2020-12-17T15:45:13Z

@jianzzha @cynepco3hahue PTAL

haircommander · 2020-12-17T15:46:28Z

internal/runtimehandlerhooks/high_performance_hooks.go

 // HighPerformanceHooks used to run additional hooks that will configure a system for the latency sensitive workloads
-type HighPerformanceHooks struct{}
+type HighPerformanceHooks struct {
+	irqBalanceConfigFile string


why have this file as a member of the Hooks, when the other file locations are hardcoded?

@haircommander This file location is platform dependent. For example, in Ubuntu, the config file is /etc/default/irqbalance. Hence making it to configure from a RuntimeConfig property. Can you check it now ?

codecov · 2020-12-17T16:32:58Z

Codecov Report

Merging #4441 (77f1b7c) into master (3a90a17) will increase coverage by 0.06%.
The diff coverage is 45.19%.

@@            Coverage Diff             @@
##           master    #4441      +/-   ##
==========================================
+ Coverage   40.03%   40.10%   +0.06%     
==========================================
  Files         116      116              
  Lines        9391     9481      +90     
==========================================
+ Hits         3760     3802      +42     
- Misses       5224     5256      +32     
- Partials      407      423      +16

jianzzha · 2021-01-05T19:20:00Z

@jianzzha @cynepco3hahue PTAL

@haircommander I will get someone to validate this in centos

haircommander · 2021-01-05T20:36:20Z

/ok-to-test

haircommander · 2021-01-05T20:38:05Z

by the way, thanks for taking this on @pperiyasamy, and thanks @jianzzha for the testing help !

jianzzha · 2021-01-06T15:25:03Z

internal/runtimehandlerhooks/utils.go

+	lines := strings.Split(string(input), "\n")
+	found := false
+	for i, line := range lines {
+		if strings.Contains(line, irqBalanceBannedCpus+"=") {


make sure this is a true config line, not in a comment line, e.g. no leading non-space characters

correct, now changed to use HasPrefix method, i guess that should be ok.

jianzzha · 2021-01-06T18:30:20Z

internal/runtimehandlerhooks/high_performance_hooks.go

-		logrus.Warnf("irqbalance binary not found: %v", err)
-		return nil
+
+	if !fileExists(irqBalanceConfigFile) {


can we think of a better way to test if irqbalance service is enabled rather than test the config file existence? I mean it is possible that the service config file exists but the service is disabled. The original idea was that the irqbalance service was disabled via external mechanism outside crio or k8s, for example via deployment orchestration. Not sure if any deployment has taken that route but it is always a good idea to be careful.

yes, good point! added is-enabled check now.

jianzzha · 2021-01-06T19:15:44Z

internal/runtimehandlerhooks/high_performance_hooks.go

 }
+
+// RestoreIrqBalanceConfig restores irqbalance service with original banned cpu mask settings
+func RestoreIrqBalanceConfig(irqBalanceConfigFile, irqBannedCPUConfigFile string) error {


Thanks for adding this restore function. Can we check if the irqbalance service is enabled before restart it?

sure, done.

looks good to me

jianzzha · 2021-01-07T03:09:14Z

@haircommander integration test on bare metal (centos k8s) works fine; some minor comments posted inline
@cynepco3hahue on the openshift side - I see no extra work to adopt this
@pperiyasamy can you take a look of the inline comments

haircommander · 2021-01-07T14:48:46Z

docs/crio.conf.5.md

 **apparmor_profile**=""
  Used to change the name of the default AppArmor profile of CRI-O. The default profile name is "crio-default".

+**irqbalance_config_file**="/etc/sysconfig/irqbalance"


can you add a blurb about where this file should be for which distro (only for the most popular: ubuntu/centos, maybe debian/rhel if they differ at all)

Yes! done, I was able to confirm the file locations only in centos/suse and ubuntu platforms. so added it only for those cases.

internal/runtimehandlerhooks/high_performance_hooks_test.go

pperiyasamy · 2021-01-07T16:29:18Z

@pperiyasamy: The following tests failed, say /retest to rerun all failed tests:

Test name Commit Details Rerun command
ci/openshift-jenkins/integration_crun e14e386 link /test integration_crun
ci/prow/e2e-gcp e14e386 link /test e2e-gcp
ci/prow/e2e-agnostic e14e386 link /test e2e-agnostic
Full PR test history. Your PR dashboard.

@haircommander thanks, while addressing your review comments, any idea while these checks are failing ? Looks it's not related to this PR.

haircommander · 2021-01-07T17:29:15Z

the ci/prow/e2e* are generally flaky and I believe integration_crun was a provisioning flake
/retest

Since cri-o#4441 , to support the dynamic irqbalance configuration in turn needed by the high performance hooks, CRI-O runs the restore code at startup. In some configuration scenarios it's preferred to have more control over this step. Some configuration wants to have more control over the mask to be restored, or to disable the restore flow entirely. For this reason, we add an option to provide a user-supplied restore file. Users wishing to exert more control can provide their data, or they can disable the flow entirely setting the path to the restore file to empty string (""). The new option defaults are meant to ensure full backward compatibility. Signed-off-by: Francesco Romani <[email protected]>

Since cri-o#4441 , to support the dynamic irqbalance configuration in turn needed by the high performance hooks, CRI-O runs the restore code at startup. In some configuration scenarios it's preferred to have more control over this step. Some configuration wants to have more control over the mask to be restored, or to disable the restore flow entirely. For this reason, we add an option to provide a user-supplied restore file. Users wishing to exert more control can provide their data, or they can disable the flow entirely setting the path to the restore file "disable". The new option defaults are meant to ensure full backward compatibility. Signed-off-by: Francesco Romani <[email protected]>

pperiyasamy requested review from mrunalp and runcom as code owners December 17, 2020 13:00

openshift-ci-robot added the dco-signoff: yes Indicates the PR's author has DCO signed all their commits. label Dec 17, 2020

openshift-ci-robot requested review from fidencio and nalind December 17, 2020 13:00

openshift-ci-robot added the do-not-merge/release-note-label-needed Indicates that a PR should not merge because it's missing one of the release note labels. label Dec 17, 2020

openshift-ci-robot added the needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. label Dec 17, 2020

pperiyasamy force-pushed the irq branch from b3b5120 to 2007914 Compare December 17, 2020 13:23

pperiyasamy force-pushed the irq branch from 2007914 to 548cfed Compare December 17, 2020 15:34

haircommander reviewed Dec 17, 2020

View reviewed changes

pperiyasamy force-pushed the irq branch 2 times, most recently from 856b7f0 to b98fea6 Compare January 5, 2021 12:35

openshift-ci-robot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Jan 5, 2021

jianzzha reviewed Jan 6, 2021

View reviewed changes

haircommander reviewed Jan 7, 2021

View reviewed changes

internal/runtimehandlerhooks/high_performance_hooks_test.go Outdated Show resolved Hide resolved

jlojosnegros mentioned this pull request Nov 22, 2022

improve the irqbalance restore flow #6388

Merged

handle irqbalance service #4441

handle irqbalance service #4441

Uh oh!

Conversation

pperiyasamy commented Dec 17, 2020 • edited by haircommander Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What type of PR is this?

What this PR does / why we need it:

Which issue(s) this PR fixes:

Special notes for your reviewer:

Does this PR introduce a user-facing change?

Uh oh!

openshift-ci-robot commented Dec 17, 2020

Uh oh!

pperiyasamy commented Dec 17, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

TomSweeneyRedHat commented Dec 17, 2020

Uh oh!

TomSweeneyRedHat commented Dec 17, 2020

Uh oh!

pperiyasamy commented Dec 17, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

haircommander commented Dec 17, 2020

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

codecov bot commented Dec 17, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

jianzzha commented Jan 5, 2021

Uh oh!

haircommander commented Jan 5, 2021

Uh oh!

haircommander commented Jan 5, 2021

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jianzzha commented Jan 7, 2021

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

pperiyasamy commented Jan 7, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

haircommander commented Jan 7, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

10 participants

pperiyasamy commented Dec 17, 2020 •

edited by haircommander

Loading

pperiyasamy commented Dec 17, 2020 •

edited

Loading

pperiyasamy commented Dec 17, 2020 •

edited

Loading

codecov bot commented Dec 17, 2020 •

edited

Loading

pperiyasamy commented Jan 7, 2021 •

edited

Loading