Allow runtimes to be added and default runtime to be modified on config reload #6056

elezar · 2022-07-14T09:59:08Z

What type of PR is this?

/kind feature

What this PR does / why we need it:

This PR adds support for reloading runtime configs on SIGHUP. This allows for runtimes to be added dynamically and the default runtimes to be adjusted.

This is useful for cases where runtimes are added / configured after crio is started. One example of this is the NVIDIA Container Runtime which is managed by the NVIDIA GPU Operator. Without this change, adding a runtime to the config requires a node restart which is disruptive.

Which issue(s) this PR fixes:

Fixes #6036

Special notes for your reviewer:

Does this PR introduce a user-facing change?

Allow for a reload to add additional runtimes and change the default runtime

openshift-ci · 2022-07-14T09:59:24Z

Hi @elezar. Thanks for your PR.

I'm waiting for a cri-o member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

TomSweeneyRedHat · 2022-07-14T14:42:51Z

The code LGTM for what you're intending to do. However, I'll have to lean on @giuseppe , @haircommander and @saschagrunert on the safeness of swapping out the runtime like this.

pkg/config/reload.go

pkg/config/reload_test.go

haircommander · 2022-07-14T14:55:01Z

/ok-to-test

thanks @elezar ! I have a couple of notes, but this is definitely in the right direction

elezar · 2022-07-14T16:03:43Z

The code LGTM for what you're intending to do. However, I'll have to lean on @giuseppe , @haircommander and @saschagrunert on the safeness of swapping out the runtime like this.

Note that this only adds new runtimes and does not replace existing ones. The exception is that the name of the default runtime can be changed.

Any insights into caveats here would be appreciated.

haircommander · 2022-07-15T13:17:27Z

pkg/config/config.go

+	// check we do have at least a runtime
+	if _, ok := c.Runtimes[c.DefaultRuntime]; !ok {
+		// Set the default runtime to "runc" if default_runtime is not set
+		if c.DefaultRuntime == "" {


nit, can we invert this condition and compress the else?

if c.DefaultRuntime != "" { return } ...

I suppose this was already there, but doesn't hurt to update now 🙂

OK. I have updated this in 493eb02

saschagrunert

The unit test failures may be legit. Can we also add a test to https://github.com/cri-o/cri-o/blob/main/test/reload_config.bats, maybe by injecting a fake runtime?

elezar · 2022-07-18T08:49:58Z

The unit test failures may be legit. Can we also add a test to https://github.com/cri-o/cri-o/blob/main/test/reload_config.bats, maybe by injecting a fake runtime?

OK. I will have a look.

haircommander · 2022-09-09T12:16:40Z

@mrunalp PTAL, if this gets merged, in which OCP version can we expect this to land?

main is currently feeding into 1.26, so 4.13 would be the target

@saschagrunert PTAL

saschagrunert

Just one tiny error wrapping nit, otherwise LGTM

saschagrunert · 2022-09-12T06:52:48Z

pkg/config/reload.go

+	}
+
+	if err := c.ValidateRuntimes(); err != nil {
+		return fmt.Errorf("unabled to reload runtimes: %v", err)


Suggested change

return fmt.Errorf("unabled to reload runtimes: %v", err)

return fmt.Errorf("unabled to reload runtimes: %w", err)

Thanks. I will rebase and make this change tomorrow.

Updated and rebased.

elezar · 2022-09-15T12:37:06Z

/retest-required

haircommander · 2022-09-16T19:59:12Z

thanks for your patience as we went through a CI migration and multiple outages

/lgtm

haircommander · 2022-09-16T20:38:09Z

oh bother
/hold

elezar · 2022-09-19T09:15:07Z

/retest

haircommander · 2022-09-19T15:42:54Z

/hold cancel

Since we want to validate the default runtime separately when reloading the runtime config we factor out default runtime validation into a ValidateDefaultRuntime method. Signed-off-by: Evan Lezar <[email protected]>

This change cleans up the conditional checks in ValidateDefaultRuntime for readability. Signed-off-by: Evan Lezar <[email protected]>

This change includes the update of runtime configs on a config reload. This is currently limited to: * Changing the default runtime * Adding new runtimes Signed-off-by: Evan Lezar <[email protected]>

Signed-off-by: Evan Lezar <[email protected]>

elezar · 2022-09-21T11:46:09Z

/retest

elezar · 2022-09-21T11:47:11Z

@haircommander thanks for giving me the heads-up regarding the CI issues. I have rebased the changes off main again. Are CI issues still expected?

haircommander · 2022-09-21T15:44:40Z

yeah that one is expected

/override ci/prow/ci-e2e-conmonrs
/lgtm

openshift-ci · 2022-09-21T15:44:48Z

@haircommander: Overrode contexts on behalf of haircommander: ci/prow/ci-e2e-conmonrs

Details

In response to this:

yeah that one is expected

/override ci/prow/ci-e2e-conmonrs
/lgtm

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-ci · 2022-09-21T16:05:12Z

@elezar: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/openshift-jenkins/integration_fedora	`ae54757`	link	true	`/test integration_fedora`
ci/openshift-jenkins/integration_crun	`ae54757`	link	true	`/test integration_crun`
ci/openshift-jenkins/integration_crun_cgroupv2	`ae54757`	link	false	`/test integration_cgroupv2`
ci/openshift-jenkins/integration_rhel	`ae54757`	link	true	`/test integration_rhel`

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

haircommander · 2022-09-21T18:23:30Z

/retest

openshift-ci bot requested review from klihub and wgahnagl July 14, 2022 09:59

openshift-ci bot added the needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. label Jul 14, 2022

elezar mentioned this pull request Jul 14, 2022

Support reload of cri-o config to add runtimes #6036

Closed

elezar changed the title ~~Reload runtimes~~ Allow runtimes to be added and default runtime to be modified on config reload Jul 14, 2022

haircommander reviewed Jul 14, 2022

View reviewed changes

pkg/config/reload.go Outdated Show resolved Hide resolved

haircommander reviewed Jul 14, 2022

View reviewed changes

pkg/config/reload.go Show resolved Hide resolved

haircommander reviewed Jul 14, 2022

View reviewed changes

pkg/config/reload_test.go Show resolved Hide resolved

openshift-ci bot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Jul 14, 2022

elezar requested a review from haircommander July 15, 2022 12:56

elezar marked this pull request as ready for review July 15, 2022 12:56

elezar requested review from mrunalp and runcom as code owners July 15, 2022 12:56

openshift-ci bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jul 15, 2022

openshift-ci bot requested a review from QiWang19 July 15, 2022 12:56

haircommander reviewed Jul 15, 2022

View reviewed changes

saschagrunert reviewed Jul 18, 2022

View reviewed changes

elezar marked this pull request as draft July 18, 2022 08:45

openshift-ci bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jul 18, 2022

elezar force-pushed the reload-runtimes branch from 2115034 to 1e3b0b2 Compare July 18, 2022 09:01

saschagrunert reviewed Sep 12, 2022

View reviewed changes

elezar force-pushed the reload-runtimes branch from c5b6e2f to 90e7bff Compare September 14, 2022 14:09

openshift-ci bot assigned haircommander Sep 16, 2022

openshift-ci bot added the lgtm Indicates that a PR is ready to be merged. label Sep 16, 2022

openshift-ci bot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Sep 16, 2022

openshift-ci bot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Sep 19, 2022

elezar added 5 commits September 21, 2022 11:45

Move default runtime validation to its own function

2ba6ee2

Since we want to validate the default runtime separately when reloading the runtime config we factor out default runtime validation into a ValidateDefaultRuntime method. Signed-off-by: Evan Lezar <[email protected]>

Invert conditional check in ValidateDefaultRuntime

29bff15

This change cleans up the conditional checks in ValidateDefaultRuntime for readability. Signed-off-by: Evan Lezar <[email protected]>

Reload runtime configs on reload

a3fb007

This change includes the update of runtime configs on a config reload. This is currently limited to: * Changing the default runtime * Adding new runtimes Signed-off-by: Evan Lezar <[email protected]>

Add notes on runtime reload support to documentation

877b5fb

Signed-off-by: Evan Lezar <[email protected]>

Add basic integration tests for runtime reload

9db3e8e

Signed-off-by: Evan Lezar <[email protected]>

elezar force-pushed the reload-runtimes branch from 90e7bff to 9db3e8e Compare September 21, 2022 09:45

openshift-ci bot removed the lgtm Indicates that a PR is ready to be merged. label Sep 21, 2022

openshift-ci bot added the lgtm Indicates that a PR is ready to be merged. label Sep 21, 2022

openshift-merge-robot merged commit 5ddc1fe into cri-o:main Sep 21, 2022

elezar mentioned this pull request Sep 22, 2022

Allow runtime configs to be updated completely #6248

Merged

haircommander mentioned this pull request Oct 4, 2022

[1.25] Allow runtimes to be reloaded #6270

Merged

elezar deleted the reload-runtimes branch October 11, 2022 18:14

	return fmt.Errorf("unabled to reload runtimes: %v", err)
	return fmt.Errorf("unabled to reload runtimes: %w", err)

Allow runtimes to be added and default runtime to be modified on config reload #6056

Allow runtimes to be added and default runtime to be modified on config reload #6056

Conversation

elezar commented Jul 14, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What type of PR is this?

What this PR does / why we need it:

Which issue(s) this PR fixes:

Special notes for your reviewer:

Does this PR introduce a user-facing change?

Uh oh!

openshift-ci bot commented Jul 14, 2022

Uh oh!

TomSweeneyRedHat commented Jul 14, 2022

Uh oh!

Uh oh!

Uh oh!

Uh oh!

haircommander commented Jul 14, 2022

Uh oh!

elezar commented Jul 14, 2022

Uh oh!

haircommander Jul 15, 2022

Choose a reason for hiding this comment

Uh oh!

haircommander Jul 15, 2022

Choose a reason for hiding this comment

Uh oh!

elezar Jul 18, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

saschagrunert left a comment

Choose a reason for hiding this comment

Uh oh!

elezar commented Jul 18, 2022

Uh oh!

haircommander commented Sep 9, 2022

Uh oh!

saschagrunert left a comment

Choose a reason for hiding this comment

Uh oh!

saschagrunert Sep 12, 2022

Choose a reason for hiding this comment

Uh oh!

elezar Sep 13, 2022

Choose a reason for hiding this comment

Uh oh!

elezar Sep 14, 2022

Choose a reason for hiding this comment

Uh oh!

elezar commented Sep 15, 2022

Uh oh!

haircommander commented Sep 16, 2022

Uh oh!

haircommander commented Sep 16, 2022

Uh oh!

elezar commented Sep 19, 2022

Uh oh!

haircommander commented Sep 19, 2022

Uh oh!

elezar commented Sep 21, 2022

Uh oh!

elezar commented Sep 21, 2022

Uh oh!

haircommander commented Sep 21, 2022

Uh oh!

openshift-ci bot commented Sep 21, 2022

Uh oh!

openshift-ci bot commented Sep 21, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

haircommander commented Sep 21, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

elezar commented Jul 14, 2022 •

edited

Loading

elezar Jul 18, 2022 •

edited

Loading

openshift-ci bot commented Sep 21, 2022 •

edited

Loading