Support Intel RDT #4830

marquiz · 2021-04-30T08:57:05Z

OCI runtime-spec has supported Intel RDT (resctrl pseudo-filesystem) for a while already. This RFC PR adds two independent parts of RDT support to CRI-O.

The first one is integration with github.com/intel/goresctrl which enables flexible class-based configuration mechanism. The concept of the integration is partly inspired by seccomp et. al. CRI-O config only specifies a path to an external configuration file, and the file structure and logic of applying the configuration is owned by the library. This integration will give users an easy option for configuring RDT in tandem with the container runtime.

The second part adds support for container annotation for controlling the RDT class/CLOS of containers. The idea is that Kubelet can utilize this annotation before the CRI API properly supports RDT. There's also a patch that adds respective Pod annotation for controlling the RDT class (per-pod and per-container) – this is for testing/demonstration purposes of K8s integration.

My take is that the next steps – e.g. how to split this PR, how to proceed with CRI API etc – depend on how this PR is received.

What type of PR is this?

/kind feature

/kind api-change
/kind bug
/kind ci
/kind cleanup
/kind dependency-change
/kind deprecation
/kind design
/kind documentation
/kind failing-test
/kind flake
/kind other

What this PR does / why we need it:

The OCI runtime spec has had support for RDT for a long time, already. This PR opens up the path to CRI support. Integration with goresctrl is a light and simple extension point for better user experience.

Which issue(s) this PR fixes:

Special notes for your reviewer:

This (RFC) PR consists of two separate pieces (support RDT annotation(s) and integration with goresctrl) that could be split into separate PRs.

Related runc PR:
opencontainers/runc#2920

Sibling PR against containerd:
containerd/containerd#5439

/hold

Does this PR introduce a user-facing change?

Support `io.kubernetes.cri.rdt-class` container annotation for specifying RDT class.

Add rdt_config_file config file option (and corresponding --rdt-config-file for command line) for configuring the resctrl pseudo-filesystem.

openshift-ci-robot · 2021-04-30T08:57:10Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: marquiz
To complete the pull request process, please assign haircommander after the PR has been reviewed.
You can assign the PR to them by writing /assign @haircommander in a comment when ready.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

openshift-ci-robot · 2021-04-30T08:57:13Z

Hi @marquiz. Thanks for your PR.

I'm waiting for a cri-o member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

vendor/github.com/intel/goresctrl/pkg/rdt/config.go

haircommander · 2021-04-30T18:11:42Z

cool! thanks for doing this work @marquiz.

It is my understanding that this currently doesn't totally work (the rdt found in the annotation doesn't seem to be going anywhere). that's fine to start, just wanted to check.

Is it standard for all containers to have the same rdt profile for a host? or would they want the configuration to look differently? (I don't know anything about rdt)

If they may look different, it may behoove us to have mutliple profiles, found in a directory, rather than a single file. seccomp does this currently, where if you specify localhost/filename it looks for the seccomp config file in the filename in a specific directory.

I have a few review comments and nits, and we'll likely want some tests to go along with this one day. I would be interested in adding support independent of corresponding CRI changes. our allowed_annotations mechanism allows admins to specify annotations on the runtime class they want to allow. This means they can opt-out of this behavior, so we don't risk anything by adding it

haircommander · 2021-04-30T18:11:49Z

/ok-to-test

marquiz · 2021-05-03T11:17:07Z

Thanks for taking a look at this @haircommander !

It is my understanding that this currently doesn't totally work (the rdt found in the annotation doesn't seem to be going anywhere). that's fine to start, just wanted to check.

It does work. We inspect the annotation(s) and set Spec.Linux.IntelRdt.ClosID field accordingly. There is a small catch, though, as runc does not correctly support the ClosID field. That's what the related PR I noted (opencontainers/runc#2920) fixes – I haven't taken a look at crun, yet.

Is it standard for all containers to have the same rdt profile for a host? or would they want the configuration to look differently? (I don't know anything about rdt)

If they may look different, it may behoove us to have mutliple profiles, found in a directory, rather than a single file. seccomp does this currently, where if you specify localhost/filename it looks for the seccomp config file in the filename in a specific directory.

RDT enables QoS control of cache and/or memory bandwidth by providing class-based allocation of these resources. IOW, it allows a set of Classes-of-Service (CLOSes) that cache lines and/or memory bandwidth is allocated. PIDs are then assigned to one of those classes limiting their resource usage. The maximum number of classes is fairly limited (by HW) e.g. to 16.

The OCI container runtime (runc) does not read any configuration file. It just assigns the container (PIDs) to the class/clos specified in the container runtime spec. With this PR, classes/closes are pre-configured at CRI-O startup by goresctrl. Or probably better, "may be configured", as the class/clos configuration could be done out-of-band, manually or with some other tool as well.

I'm not sure of the aspect of independently configured profiles of e.g. seccomp/apparmor is applicable here. At least not in the case of goresctrl. The classes are inter-dependent as you may e.g. specify that reserve 50% to class A exclusively and remaining 50% to B and C so that C only gets 60% of that share. The set of available classes/closes are specified in a single configuration file.

Some references:

I have a few review comments and nits, and we'll likely want some tests to go along with this one day.

I didn't add any, yet, as the integration is so "thin". We could basically test the config parsing error cases and parsing of the annotation(s) but that's about it.

I would be interested in adding support independent of corresponding CRI changes. our allowed_annotations mechanism allows admins to specify annotations on the runtime class they want to allow. This means they can opt-out of this behavior, so we don't risk anything by adding it

Yeah, I added the container annotation to the list of allowed_annotations.

I think that the last patch (adding Pod annotations) really should not go into CRI-O. I'll submit that against K8s/kubelet if this approach looks feasible to the container runtimes (I submitted a PR against containerd as well: containerd/containerd#5439)

haircommander · 2021-05-05T16:15:47Z

/retest

codecov · 2021-05-05T16:19:57Z

Codecov Report

Merging #4830 (79e2a85) into master (78f06f2) will decrease coverage by 0.13%.
The diff coverage is 20.77%.

❗ Current head 79e2a85 differs from pull request most recent head d0baf9c. Consider uploading reports for the commit d0baf9c to get more accurate results

@@            Coverage Diff             @@
##           master    #4830      +/-   ##
==========================================
- Coverage   44.27%   44.13%   -0.14%     
==========================================
  Files         112      113       +1     
  Lines       11563    11640      +77     
==========================================
+ Hits         5119     5137      +18     
- Misses       5957     6013      +56     
- Partials      487      490       +3

marquiz · 2021-05-07T04:03:26Z

Rebased

saschagrunert

Looks good from my point of view. Can we test this somehow in our CI?

pkg/config/config.go

pkg/config/template.go

TomSweeneyRedHat · 2021-05-07T18:27:26Z

Other than @saschagrunert 's comments, LGTM

saschagrunert · 2021-05-10T07:19:19Z

@marquiz please rebase again 🙃

Use goresctrl for parsing RDT related container and pod annotations. In practice from the users' perspective, this patchs adds support for a container annotation and two pod annotations for controlling the RDT class on CRI level. Container annotation can be used by the CRI client: "io.kubernetes.cri.rdt-class" Pod annotations for specifying the RDT class in the K8s podspec level: "rdt.resources.beta.kubernetes.io/pod" (pod-wide default for all containers) "rdt.resources.beta.kubernetes.io/container.<container_name>" (container-specific overrides) Annotations are intended as an intermediate step before the CRI API supports RDT. Signed-off-by: Markus Lehtonen <[email protected]>

Signed-off-by: Markus Lehtonen <[email protected]>

marquiz · 2021-08-10T10:22:55Z

I'm sorry we have to rebase again after the k8s bump.

Np, this was an easy one.

Hmm, seems like sonatype is not happy. I think managing the indirect dependencies manually in goresctrl would be unmaintainable and error prone (e.g. go mod tidy would clean those away). Also, I checked that the listed deps above are already (indirect) dependencies of cri-o at present. I'd suggest to ignore the comments (especially as cri-o already depends on those vulnerabilities).

Yes, I agree.

👍

marquiz · 2021-08-10T11:22:12Z

/retest

marquiz · 2021-08-12T13:03:04Z

/retest

haircommander · 2021-08-17T13:09:13Z

/retest
/lgtm

haircommander · 2021-08-17T13:09:32Z

/assign @saschagrunert
for the final approval

saschagrunert

/lgtm
/unhold

openshift-ci · 2021-08-17T15:02:26Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: marquiz, saschagrunert

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

~~OWNERS~~ [saschagrunert]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

openshift-bot · 2021-08-17T15:03:08Z

/retest-required

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2021-08-17T15:14:54Z

/retest-required

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2021-08-17T15:26:57Z

/retest-required

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2021-08-17T15:38:51Z

/retest-required

Please review the full test history for this PR and help us cut down flakes.

openshift-ci · 2021-08-17T16:54:36Z

@marquiz: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Rerun command
ci/openshift-jenkins/e2e_crun_cgroupv2	`d0baf9c`	link	`/test e2e_cgroupv2`

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

marquiz · 2021-08-17T17:23:55Z

/retest

haircommander · 2021-08-17T18:18:19Z

/override ci/prow/e2e-gcp

agnostic passed

openshift-ci · 2021-08-17T18:18:24Z

@haircommander: Overrode contexts on behalf of haircommander: ci/prow/e2e-gcp

Details

In response to this:

/override ci/prow/e2e-gcp

agnostic passed

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

marquiz requested review from mrunalp and runcom as code owners April 30, 2021 08:57

openshift-ci-robot requested a review from sboeuf April 30, 2021 08:57

openshift-ci-robot added the needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. label Apr 30, 2021

sonatype-lift bot reviewed Apr 30, 2021

View reviewed changes

vendor/github.com/intel/goresctrl/pkg/rdt/config.go Outdated Show resolved Hide resolved

sonatype-lift bot reviewed Apr 30, 2021

View reviewed changes

vendor/github.com/intel/goresctrl/pkg/rdt/config.go Outdated Show resolved Hide resolved

marquiz mentioned this pull request Apr 30, 2021

Add support for Intel RDT containerd/containerd#5439

Merged

openshift-ci-robot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Apr 30, 2021

marquiz force-pushed the devel/goresctrl branch from c5693c8 to ad9f196 Compare May 3, 2021 07:08

openshift-ci bot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label May 6, 2021

marquiz force-pushed the devel/goresctrl branch from ad9f196 to 965d0ea Compare May 7, 2021 04:03

openshift-ci bot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label May 7, 2021

saschagrunert reviewed May 7, 2021

View reviewed changes

pkg/config/config.go Outdated Show resolved Hide resolved

pkg/config/template.go Show resolved Hide resolved

askervin mentioned this pull request May 7, 2021

Support for cgroups blockio #4873

Merged

openshift-ci bot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label May 10, 2021

marquiz added 2 commits August 10, 2021 13:15

Update dependencies

d0baf9c

Signed-off-by: Markus Lehtonen <[email protected]>

marquiz force-pushed the devel/goresctrl branch from fafbef7 to d0baf9c Compare August 10, 2021 10:18

haircommander added this to the 1.22 milestone Aug 12, 2021

openshift-ci bot assigned haircommander Aug 17, 2021

openshift-ci bot added the lgtm Indicates that a PR is ready to be merged. label Aug 17, 2021

openshift-ci bot assigned saschagrunert Aug 17, 2021

saschagrunert approved these changes Aug 17, 2021

View reviewed changes

openshift-ci bot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Aug 17, 2021

openshift-ci bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Aug 17, 2021

openshift-ci bot merged commit 71ec863 into cri-o:master Aug 17, 2021

marquiz mentioned this pull request Aug 18, 2021

libcontainer/intelrdt: support ClosID parameter opencontainers/runc#2920

Merged

marquiz deleted the devel/goresctrl branch September 24, 2021 12:45

askervin mentioned this pull request Jan 12, 2022

iops limit for pod/pvc/pv kubernetes/kubernetes#92287

Closed

marquiz mentioned this pull request May 16, 2022

Unconference Topics - KCEU Contributor Summit 2022 Valencia kubernetes/community#6633

Closed

pacoxu mentioned this pull request May 26, 2022

KEP-3008: QoS-class resources kubernetes/enhancements#3004

Closed

Support Intel RDT #4830

Support Intel RDT #4830

Uh oh!

Conversation

marquiz commented Apr 30, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What type of PR is this?

What this PR does / why we need it:

Which issue(s) this PR fixes:

Special notes for your reviewer:

Does this PR introduce a user-facing change?

Uh oh!

openshift-ci-robot commented Apr 30, 2021

Uh oh!

openshift-ci-robot commented Apr 30, 2021

Uh oh!

Uh oh!

Uh oh!

haircommander commented Apr 30, 2021

Uh oh!

haircommander commented Apr 30, 2021

Uh oh!

marquiz commented May 3, 2021

Uh oh!

haircommander commented May 5, 2021

Uh oh!

codecov bot commented May 5, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

marquiz commented May 7, 2021

Uh oh!

saschagrunert left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

TomSweeneyRedHat commented May 7, 2021

Uh oh!

saschagrunert commented May 10, 2021

Uh oh!

marquiz commented Aug 10, 2021

Uh oh!

marquiz commented Aug 10, 2021

Uh oh!

marquiz commented Aug 12, 2021

Uh oh!

haircommander commented Aug 17, 2021

Uh oh!

haircommander commented Aug 17, 2021

Uh oh!

saschagrunert left a comment

Choose a reason for hiding this comment

Uh oh!

openshift-ci bot commented Aug 17, 2021

Uh oh!

openshift-bot commented Aug 17, 2021

Uh oh!

openshift-bot commented Aug 17, 2021

Uh oh!

openshift-bot commented Aug 17, 2021

Uh oh!

openshift-bot commented Aug 17, 2021

Uh oh!

openshift-ci bot commented Aug 17, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

marquiz commented Aug 17, 2021

Uh oh!

haircommander commented Aug 17, 2021

Uh oh!

openshift-ci bot commented Aug 17, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

marquiz commented Apr 30, 2021 •

edited

Loading

codecov bot commented May 5, 2021 •

edited

Loading

openshift-ci bot commented Aug 17, 2021 •

edited

Loading