Multi storage support and selection using the runtime class #5624

alicefr · 2022-02-11T10:18:07Z

What type of PR is this?

/kind feature

What this PR does / why we need it:

Extend cri-o to configure multiple storage drivers at the same time and select the storage driver based on the runtime.

Today, cri-o supports a single storage driver at the time and the suggested/default is overlayfs for standard containers. For VM-based runtimes, the current solution is to share the container filesystem mount point using 9p or virtiofs. However, this doesn't offer the best performance, and passing a disk would be the ideal solution.

The benefits of this enhancement are that we avoid the conflicts between standard and VM-based containers. We could get the best performance for both kind of workloads. Giving an example, kata-container could use device-mapper instead of overlays for passing the container filesystem as a device.

In the long term, this extension could also facilitate the introduction of new storage driver types for confidential computing. These new technologies come with a lot of challenges and requirements that filesystem-based storage drivers probably cannot satisfy.

Which issue(s) this PR fixes:

Fixes #5578

Special notes for your reviewer:

Missing pieces:

if we download the same image for different storage drivers, it is listed multiple times? Do we need to filter it
Display the storage information for image inspection

Right now, I've been testing these changes by:
Starting cri-o with overlay and devicemapper:

$ bin/crio -l debug \
	--internal-wipe \
	--root /var/lib/containers-test/storage \
	--runroot /var/lib/containers-test/storage \
	--cgroup-manager systemd \
	--runtimes runc0:/usr/bin/runc:/run/runc0:oci:false \
	--runtime-storage runc0:devicemapper:dm.directlvm_device=/dev/loop0:dm.fs=ext4:dm.directlvm_device_force=true

Does this PR introduce a user-facing change?

Support for multi storage drivers and selection using the runtimeclass

openshift-ci · 2022-02-11T10:18:16Z

Hi @alicefr. Thanks for your PR.

I'm waiting for a cri-o member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

haircommander · 2022-02-11T15:51:30Z

/ok-to-test

thanks @alicefr !

alicefr · 2022-02-17T13:57:56Z

Update PR with 2 new interfaces for the multi store support. The unit test are passing locally. However, I needed to put a couple of AnyTimes(). So probably we want to get rid of those but at least I wanted to check the entire testsuite. Now, I'd like to check what is passing/failing in the upstream CI

sonatype-lift · 2022-02-23T14:26:29Z

pkg/config/config.go

+	storageDrivers := c.Storage
+	storageOpts := copyStorageOptions(c.StorageOptions)

 	data, err := ioutil.ReadFile(path)


G304: Potential file inclusion via variable
(at-me in a reply with help or ignore)

Generate mock for cri-o/cri-o/internal/storage.MultiStoreServer and cri-o/pkg/config.MultiStore Signed-off-by: Alice Frosi <[email protected]>

With the introduction of the multi storage driver support, the tests need to be adjusted to reflect the new structures and interfaces. Signed-off-by: Alice Frosi <[email protected]>

Signed-off-by: Alice Frosi <[email protected]>

Add documentation for the new option --runtime-storage Signed-off-by: Alice Frosi <[email protected]>

Replace errrors.Wraps with fmt.Errors Signed-off-by: Alice Frosi <[email protected]>

Signed-off-by: Alice Frosi <[email protected]>

The image can be present on multiple storage driver and needs to be remove from all of them. Signed-off-by: Alice Frosi <[email protected]>

Every instance of the iterator is a local copy and concurrent calls to the same function don't conflicts. Signed-off-by: Alice Frosi <[email protected]>

The new iterator behavior changed the sequence of the mocked calls. Signed-off-by: Alice Frosi <[email protected]>

Include fix github.com/containers/storage#1298 Signed-off-by: Alice Frosi <[email protected]>

Signed-off-by: Alice Frosi <[email protected]>

Genereting mock function for GetImageServerForContainer Signed-off-by: Alice Frosi <[email protected]>

Adjust the new unit tests with the checkpoint/restore and multi-store support. Signed-off-by: Alice Frosi <[email protected]>

openshift-merge-robot · 2022-09-17T22:52:57Z

@alicefr: PR needs rebase.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

github-actions · 2022-10-18T00:06:22Z

A friendly reminder that this PR had no activity for 30 days.

openshift-ci-robot · 2023-05-06T01:18:24Z

@alicefr: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/kata-jenkins	`daff36c`	link	true	`/test kata-containers`

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

openshift-ci · 2023-05-25T16:09:53Z

@alicefr: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/openshift-jenkins/integration_crun	`fcb0dd1`	link	true	`/test integration_crun`
ci/openshift-jenkins/integration_crun_cgroupv2	`fcb0dd1`	link	false	`/test integration_cgroupv2`
ci/prow/ci-critest	`daff36c`	link	true	`/test ci-critest`
ci/prow/ci-integration	`daff36c`	link	true	`/test ci-integration`
ci/prow/ci-rhel-integration	`daff36c`	link	true	`/test ci-rhel-integration`
ci/prow/ci-images	`daff36c`	link	true	`/test ci-images`
ci/prow/e2e-aws-ovn	`daff36c`	link	true	`/test e2e-aws-ovn`
ci/prow/images	`daff36c`	link	true	`/test images`
ci/prow/periodics-images	`daff36c`	link	true	`/test periodics-images`
ci/prow/ci-e2e-conmonrs	`daff36c`	link	true	`/test ci-e2e-conmonrs`
ci/prow/ci-cgroupv2-e2e-crun	`daff36c`	link	true	`/test ci-cgroupv2-e2e-crun`
ci/prow/ci-cgroupv2-e2e	`daff36c`	link	true	`/test ci-cgroupv2-e2e`
ci/prow/ci-e2e	`daff36c`	link	true	`/test ci-e2e`
ci/prow/ci-crun-e2e	`daff36c`	link	true	`/test ci-crun-e2e`
ci/prow/ci-cgroupv2-e2e-features	`daff36c`	link	true	`/test ci-cgroupv2-e2e-features`
ci/prow/ci-rhel-critest	`daff36c`	link	true	`/test ci-rhel-critest`
ci/prow/ci-fedora-critest	`daff36c`	link	true	`/test ci-fedora-critest`
ci/prow/ci-fedora-integration	`daff36c`	link	true	`/test ci-fedora-integration`
ci/prow/ci-cgroupv2-integration	`daff36c`	link	true	`/test ci-cgroupv2-integration`
ci/prow/ci-rhel-e2e	`daff36c`	link	true	`/test ci-rhel-e2e`
ci/prow/e2e-gcp-ovn	`daff36c`	link	true	`/test e2e-gcp-ovn`
ci/kata-jenkins	`daff36c`	link	true	`/test kata-containers`
ci/prow/ci-e2e-evented-pleg	`daff36c`	link	true	`/test ci-e2e-evented-pleg`

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

alicefr · 2023-05-30T07:26:22Z

Unfortunately, I'm not working on this anymore. However, I think in the future we might need to be able to differentiate the storage driver based on the runtime or kind of workload and this PR can be used as reference.

Closing it.

alicefr requested review from mrunalp and runcom as code owners February 11, 2022 10:18

openshift-ci bot requested review from fidencio and rhatdan February 11, 2022 10:18

openshift-ci bot added the needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. label Feb 11, 2022

openshift-ci bot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Feb 11, 2022

alicefr force-pushed the multi-storage branch from a8bced8 to f37492a Compare February 17, 2022 13:49

alicefr force-pushed the multi-storage branch 5 times, most recently from a946cff to 8c7912e Compare February 23, 2022 14:03

sonatype-lift bot reviewed Feb 23, 2022

View reviewed changes

alicefr force-pushed the multi-storage branch 2 times, most recently from f11da2b to 2ff8f6a Compare February 24, 2022 16:14

openshift-ci bot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Feb 26, 2022

alicefr force-pushed the multi-storage branch from 2ff8f6a to 4e711f0 Compare February 28, 2022 11:11

openshift-ci bot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Feb 28, 2022

alicefr force-pushed the multi-storage branch 4 times, most recently from 68fdb27 to 3e64a4b Compare March 2, 2022 13:53

alicefr added 19 commits September 8, 2022 13:46

generate mock for multi store interfaces

1d50be0

Generate mock for cri-o/cri-o/internal/storage.MultiStoreServer and cri-o/pkg/config.MultiStore Signed-off-by: Alice Frosi <[email protected]>

adjust tests

6ff27c1

With the introduction of the multi storage driver support, the tests need to be adjusted to reflect the new structures and interfaces. Signed-off-by: Alice Frosi <[email protected]>

bash completions generation

b71ebc5

Signed-off-by: Alice Frosi <[email protected]>

test: add unit test for multi store

0fe9d4d

Signed-off-by: Alice Frosi <[email protected]>

add runtime handler for critest

5ffa254

Signed-off-by: Alice Frosi <[email protected]>

add test lane for critest and multi storage

b586e51

Signed-off-by: Alice Frosi <[email protected]>

add integration test for multi storage

64fd9c1

Signed-off-by: Alice Frosi <[email protected]>

additional cleanup for lvm

3f3155a

Signed-off-by: Alice Frosi <[email protected]>

docs generation

483f074

Add documentation for the new option --runtime-storage Signed-off-by: Alice Frosi <[email protected]>

switch to native error wrapping

e1eb549

Replace errrors.Wraps with fmt.Errors Signed-off-by: Alice Frosi <[email protected]>

update vendor

e564c02

Signed-off-by: Alice Frosi <[email protected]>

fix wrong type for info

1dcda5c

Signed-off-by: Alice Frosi <[email protected]>

fix image deletion

17516b6

The image can be present on multiple storage driver and needs to be remove from all of them. Signed-off-by: Alice Frosi <[email protected]>

fix iterator for MultiStore and MultiStoreServer

e520341

Every instance of the iterator is a local copy and concurrent calls to the same function don't conflicts. Signed-off-by: Alice Frosi <[email protected]>

fix unit tests

e05c980

The new iterator behavior changed the sequence of the mocked calls. Signed-off-by: Alice Frosi <[email protected]>

update vendor

cb2af51

Include fix github.com/containers/storage#1298 Signed-off-by: Alice Frosi <[email protected]>

add multi store to checkpoint/restore

b02d066

Signed-off-by: Alice Frosi <[email protected]>

make mockgen

9fcafc1

Genereting mock function for GetImageServerForContainer Signed-off-by: Alice Frosi <[email protected]>

fix unit tests

daff36c

Adjust the new unit tests with the checkpoint/restore and multi-store support. Signed-off-by: Alice Frosi <[email protected]>

alicefr force-pushed the multi-storage branch from 2b7b47f to daff36c Compare September 8, 2022 11:46

openshift-merge-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Sep 17, 2022

github-actions bot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Oct 18, 2022

alicefr mentioned this pull request Nov 2, 2022

[RFE] Select the storage driver based on the workload/runtime type #5578

Closed

littlejawa mentioned this pull request Mar 10, 2023

[WIP] Setting up a per-container imageserver mapping #6724

Closed

alicefr closed this May 30, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Multi storage support and selection using the runtime class #5624

Multi storage support and selection using the runtime class #5624

Uh oh!

alicefr commented Feb 11, 2022 •

edited

Loading

Uh oh!

openshift-ci bot commented Feb 11, 2022

Uh oh!

haircommander commented Feb 11, 2022

Uh oh!

alicefr commented Feb 17, 2022

Uh oh!

sonatype-lift bot Feb 23, 2022

Uh oh!

openshift-merge-robot commented Sep 17, 2022

Uh oh!

github-actions bot commented Oct 18, 2022

Uh oh!

openshift-ci-robot commented May 6, 2023

Uh oh!

openshift-ci bot commented May 25, 2023

Uh oh!

alicefr commented May 30, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants

Multi storage support and selection using the runtime class #5624

Multi storage support and selection using the runtime class #5624

Uh oh!

Conversation

alicefr commented Feb 11, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What type of PR is this?

What this PR does / why we need it:

Which issue(s) this PR fixes:

Special notes for your reviewer:

Does this PR introduce a user-facing change?

Uh oh!

openshift-ci bot commented Feb 11, 2022

Uh oh!

haircommander commented Feb 11, 2022

Uh oh!

alicefr commented Feb 17, 2022

Uh oh!

sonatype-lift bot Feb 23, 2022

Choose a reason for hiding this comment

Uh oh!

openshift-merge-robot commented Sep 17, 2022

Uh oh!

github-actions bot commented Oct 18, 2022

Uh oh!

openshift-ci-robot commented May 6, 2023

Uh oh!

openshift-ci bot commented May 25, 2023

Uh oh!

alicefr commented May 30, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants

alicefr commented Feb 11, 2022 •

edited

Loading