Support for cgroups blockio #4873

askervin · 2021-05-07T13:56:33Z

This PR implements support for the cgroup blockio controller.

In user perspective the blockio support works as follows.

User configures blockio controller parameters for their blockio classes, for example: ThrottledIO and LowLatency. Class names are not restricted, and the number of classes is not limited.
Each class contains sets of devices (list of filenames, wildcards allowed) and blockio parameters for each set. For example:
```
Classes:
  ThrottledIO:
  - Devices:
    - /dev/vd[a-z]
    ThrottleReadBps: 50M
    ThrottleWriteBps: 10M
```
User launches CRI-O with --blockio-config-file commandline parameter or corresponding configuration file option pointing to her configuration file.
User annotates container(s) in a pod with a blockio class name in the configuration. Example:
blockio.resources.beta.kubernetes.io/pod: ThrottledIO.
When a container is created, CRI-O assigns blockio parameters based its class configuration.

How it is implemented:

The goresctrl library understands blockio class configurations, block devices in the system and the OCI BlockIO format.
CRI-O sets effective user-defined blockio class configuration to goresctrl.
CRI-O looks for container's blockio class from pod and container annotations. The annotation syntax is similar to container RDT class annotations in PR Support Intel RDT #4830.
If CRI-O finds a class, it uses goresctrl to get corresponding OCI BlockIO data and includes it in the OCI runtime-spec.

What type of PR is this?

/kind feature

What this PR does / why we need it:

This patch implements OCI runtime-spec/BlockIO support, and makes it immediately usable through annotations.

In Kubernetes point of view this enables users to build cluster-specific QoS for block devices.

Which issue(s) this PR fixes:

Special notes for your reviewer:

Does this PR introduce a user-facing change?

Support `io.kubernetes.cri.blockio-class` container annotation for specifying blockio class.
Support `blockio.resources.beta.kubernetes.io/pod` pod annotation for specifying the default blockio class to all containers in the pod.
Support `blockio.resources.beta.kubernetes.io/container.NAME` pod annotation for specifying the blockio class of the NAME container in the pod.

Add blockio_config_file config file option (and corresponding --blockio-config-file for command line) for configuring blockio classes and their cgroups blockio controller parameters.

openshift-ci-robot · 2021-05-07T13:56:35Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: askervin
To complete the pull request process, please assign kolyshkin after the PR has been reviewed.
You can assign the PR to them by writing /assign @kolyshkin in a comment when ready.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

openshift-ci-robot · 2021-05-07T13:56:41Z

Hi @askervin. Thanks for your PR.

I'm waiting for a cri-o member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

haircommander · 2021-05-07T14:31:34Z

WOW this is awesome
/ok-to-test

thank you very much for the submission @askervin !
@giuseppe @kolyshkin PTAL

TomSweeneyRedHat · 2021-05-07T18:30:38Z

All kinds of test unhappiness, but LGTM otherwise. Definitely want @giuseppe and/or @kolyshkin to give this a good eyeball.

askervin · 2021-05-10T06:08:11Z

@haircommander , @TomSweeneyRedHat , thanks for having a look this quickly!

Rebased to fix test unhappiness...

askervin · 2021-06-04T06:11:00Z

Dear Reviewers,

I'll update this PR to match the design of the RDT PR. That is, annotation parsing will be moved from cri-o to goresctrl.

This PR will be in the Draft state until necessary pieces are in the goresctrl main branch, and I've updated this PR accordingly.

sonatype-lift · 2021-08-20T06:15:52Z

go.mod

 	github.com/google/uuid v1.3.0
 	github.com/grpc-ecosystem/go-grpc-middleware v1.3.0
-	github.com/intel/goresctrl v0.0.0-20210623080121-be5fa857f4b4
+	github.com/intel/goresctrl v0.1.0


Critical OSS Vulnerability:

pkg:golang/golang.org/x/[email protected]

5 Critical, 0 Severe, 0 Moderate and 0 Unknown vulnerabilities have been found in a transitive dependency of pkg:golang/github.com/intel/[email protected]

CRITICAL Vulnerabilities (5)

CVE-2018-17143

[CVE-2018-17143] Improper Input Validation

The html package (aka x/net/html) through 2018-09-17 in Go mishandles <template><tBody><isindex/action=0>, leading to a "panic: runtime error" in inBodyIM in parse.go during an html.Parse call.

CVSS Score: 7.5

CVSS Vector: CVSS:3.0/AV:N/AC:L/PR:N/UI:N/S:U/C:N/I:N/A:H

CVE-2018-17848

[CVE-2018-17848] Data Handling

The html package (aka x/net/html) through 2018-09-25 in Go mishandles <math><template><mn><b></template>, leading to a "panic: runtime error" (index out of range) in (*insertionModeStack).pop in node.go, called from inHeadIM, during an html.Parse call.

CVSS Score: 7.5

CVSS Vector: CVSS:3.0/AV:N/AC:L/PR:N/UI:N/S:U/C:N/I:N/A:H

CVE-2018-17847

[CVE-2018-17847] Improper Input Validation

The html package (aka x/net/html) through 2018-09-25 in Go mishandles <svg><template><desc><t><svg></template>, leading to a "panic: runtime error" (index out of range) in (*nodeStack).pop in node.go, called from (*parser).clearActiveFormattingElements, during an html.Parse call.

CVSS Score: 7.5

CVSS Vector: CVSS:3.0/AV:N/AC:L/PR:N/UI:N/S:U/C:N/I:N/A:H

CVE-2018-17142

[CVE-2018-17142] Improper Input Validation

The html package (aka x/net/html) through 2018-09-17 in Go mishandles <math><template><mo><template>, leading to a "panic: runtime error" in parseCurrentToken in parse.go during an html.Parse call.

CVSS Score: 7.5

CVSS Vector: CVSS:3.0/AV:N/AC:L/PR:N/UI:N/S:U/C:N/I:N/A:H

CVE-2018-17846

[CVE-2018-17846] Resource Management Errors

The html package (aka x/net/html) through 2018-09-25 in Go mishandles <table><math><select><mi><select></table>, leading to an infinite loop during an html.Parse call because inSelectIM and inSelectInTableIM do not comply with a specification.

CVSS Score: 7.5

CVSS Vector: CVSS:3.0/AV:N/AC:L/PR:N/UI:N/S:U/C:N/I:N/A:H

(at-me in a reply with help or ignore)

sonatype-lift · 2021-08-20T06:15:53Z

go.mod

 	github.com/google/uuid v1.3.0
 	github.com/grpc-ecosystem/go-grpc-middleware v1.3.0
-	github.com/intel/goresctrl v0.0.0-20210623080121-be5fa857f4b4
+	github.com/intel/goresctrl v0.1.0


Critical OSS Vulnerability:

pkg:golang/golang.org/x/[email protected]

5 Critical, 0 Severe, 0 Moderate and 0 Unknown vulnerabilities have been found in a transitive dependency of pkg:golang/github.com/intel/[email protected]

CRITICAL Vulnerabilities (5)

CVE-2018-17143

[CVE-2018-17143] Improper Input Validation

The html package (aka x/net/html) through 2018-09-17 in Go mishandles <template><tBody><isindex/action=0>, leading to a "panic: runtime error" in inBodyIM in parse.go during an html.Parse call.

CVSS Score: 7.5

CVSS Vector: CVSS:3.0/AV:N/AC:L/PR:N/UI:N/S:U/C:N/I:N/A:H

CVE-2018-17848

[CVE-2018-17848] Data Handling

The html package (aka x/net/html) through 2018-09-25 in Go mishandles <math><template><mn><b></template>, leading to a "panic: runtime error" (index out of range) in (*insertionModeStack).pop in node.go, called from inHeadIM, during an html.Parse call.

CVSS Score: 7.5

CVSS Vector: CVSS:3.0/AV:N/AC:L/PR:N/UI:N/S:U/C:N/I:N/A:H

CVE-2018-17847

[CVE-2018-17847] Improper Input Validation

The html package (aka x/net/html) through 2018-09-25 in Go mishandles <svg><template><desc><t><svg></template>, leading to a "panic: runtime error" (index out of range) in (*nodeStack).pop in node.go, called from (*parser).clearActiveFormattingElements, during an html.Parse call.

CVSS Score: 7.5

CVSS Vector: CVSS:3.0/AV:N/AC:L/PR:N/UI:N/S:U/C:N/I:N/A:H

CVE-2018-17142

[CVE-2018-17142] Improper Input Validation

The html package (aka x/net/html) through 2018-09-17 in Go mishandles <math><template><mo><template>, leading to a "panic: runtime error" in parseCurrentToken in parse.go during an html.Parse call.

CVSS Score: 7.5

CVSS Vector: CVSS:3.0/AV:N/AC:L/PR:N/UI:N/S:U/C:N/I:N/A:H

CVE-2018-17846

[CVE-2018-17846] Resource Management Errors

The html package (aka x/net/html) through 2018-09-25 in Go mishandles <table><math><select><mi><select></table>, leading to an infinite loop during an html.Parse call because inSelectIM and inSelectInTableIM do not comply with a specification.

CVSS Score: 7.5

CVSS Vector: CVSS:3.0/AV:N/AC:L/PR:N/UI:N/S:U/C:N/I:N/A:H

(at-me in a reply with help or ignore)

sonatype-lift · 2021-08-20T06:15:54Z

go.mod

 	github.com/google/uuid v1.3.0
 	github.com/grpc-ecosystem/go-grpc-middleware v1.3.0
-	github.com/intel/goresctrl v0.0.0-20210623080121-be5fa857f4b4
+	github.com/intel/goresctrl v0.1.0


Severe OSS Vulnerability:

pkg:golang/golang.org/x/[email protected]

0 Critical, 1 Severe, 0 Moderate and 0 Unknown vulnerabilities have been found in a transitive dependency of pkg:golang/github.com/intel/[email protected]

SEVERE Vulnerabilities (1)

[CVE-2019-11840] Use of Insufficiently Random Values

An issue was discovered in supplementary Go cryptography libraries, aka golang-googlecode-go-crypto, before 2019-03-20. A flaw was found in the amd64 implementation of golang.org/x/crypto/salsa20 and golang.org/x/crypto/salsa20/salsa. If more than 256 GiB of keystream is generated, or if the counter otherwise grows greater than 32 bits, the amd64 implementation will first generate incorrect output, and then cycle back to previously generated keystream. Repeated keystream bytes can lead to loss of confidentiality in encryption applications, or to predictability in CSPRNG applications.

CVSS Score: 5.9

CVSS Vector: CVSS:3.0/AV:N/AC:H/PR:N/UI:N/S:U/C:H/I:N/A:N

(at-me in a reply with help or ignore)

sonatype-lift · 2021-08-20T06:15:56Z

go.mod

 	github.com/google/uuid v1.3.0
 	github.com/grpc-ecosystem/go-grpc-middleware v1.3.0
-	github.com/intel/goresctrl v0.0.0-20210623080121-be5fa857f4b4
+	github.com/intel/goresctrl v0.1.0


Severe OSS Vulnerability:

pkg:golang/golang.org/x/[email protected]

0 Critical, 1 Severe, 0 Moderate and 0 Unknown vulnerabilities have been found in a transitive dependency of pkg:golang/github.com/intel/[email protected]

SEVERE Vulnerabilities (1)

[CVE-2019-11840] Use of Insufficiently Random Values

An issue was discovered in supplementary Go cryptography libraries, aka golang-googlecode-go-crypto, before 2019-03-20. A flaw was found in the amd64 implementation of golang.org/x/crypto/salsa20 and golang.org/x/crypto/salsa20/salsa. If more than 256 GiB of keystream is generated, or if the counter otherwise grows greater than 32 bits, the amd64 implementation will first generate incorrect output, and then cycle back to previously generated keystream. Repeated keystream bytes can lead to loss of confidentiality in encryption applications, or to predictability in CSPRNG applications.

CVSS Score: 5.9

CVSS Vector: CVSS:3.0/AV:N/AC:H/PR:N/UI:N/S:U/C:H/I:N/A:N

(at-me in a reply with help or ignore)

askervin · 2021-08-20T12:04:55Z

/retest

Signed-off-by: Antti Kervinen <[email protected]>

- Load class configuration file if available. - Activate configuration in the goresctrl library. - Depend on goresctrl blockio. Signed-off-by: Antti Kervinen <[email protected]>

Use goresctrl for parsing blockio class from container and pod annotations. Add Linux.Resources.BlockIO information to a container being created based on blockio class configuration. Container annotation can be used by the CRI client: "io.kubernetes.cri.blockio-class" Pod annotations for the blockio class in the K8s podspec level: "blockio.resources.beta.kubernetes.io/pod" (pod-wide default for all containers in the pod) "blockio.resources.beta.kubernetes.io/container.<container_name>" (container-specific overrides) Signed-off-by: Antti Kervinen <[email protected]>

askervin · 2021-08-23T12:40:16Z

@haircommander , I reran make docs-generation and make completions-generation, this time as root. It seems that running them as user makes changes that should not be. Fixes amended accordingly. Hoping that now the last errors are gone.

haircommander · 2021-08-23T13:38:11Z

/retest
/lgtm

haircommander · 2021-08-23T13:38:28Z

thanks @askervin !

haircommander · 2021-08-24T16:37:37Z

/approve

openshift-ci · 2021-08-24T16:37:41Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: askervin, haircommander

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

~~OWNERS~~ [haircommander]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

haircommander · 2021-08-24T19:17:19Z

/override ci/prow/e2e-gcp

openshift-ci · 2021-08-24T19:17:22Z

@haircommander: Overrode contexts on behalf of haircommander: ci/prow/e2e-gcp

Details

In response to this:

/override ci/prow/e2e-gcp

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-ci · 2021-08-24T19:57:00Z

@askervin: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Rerun command
ci/openshift-jenkins/e2e_crun_cgroupv2	`f74d274`	link	`/test e2e_cgroupv2`

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

haircommander · 2021-08-24T21:27:49Z

/retest-required

haircommander · 2021-08-24T21:48:38Z

/override ci/openshift-jenkins/integration_rhel

openshift-ci · 2021-08-24T21:48:42Z

@haircommander: Overrode contexts on behalf of haircommander: ci/openshift-jenkins/integration_rhel

Details

In response to this:

/override ci/openshift-jenkins/integration_rhel

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

askervin requested review from mrunalp and runcom as code owners May 7, 2021 13:56

openshift-ci-robot added release-note Denotes a PR that will be considered when it comes time to generate release notes. kind/feature Categorizes issue or PR as related to a new feature. dco-signoff: yes Indicates the PR's author has DCO signed all their commits. labels May 7, 2021

openshift-ci-robot requested a review from umohnani8 May 7, 2021 13:56

openshift-ci bot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label May 7, 2021

openshift-ci-robot added the needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. label May 7, 2021

openshift-ci-robot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels May 7, 2021

askervin force-pushed the 5CH_blockio branch from cc79b21 to b13bdc6 Compare May 10, 2021 06:06

openshift-ci bot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label May 10, 2021

kad mentioned this pull request May 10, 2021

Support Intel RDT #4830

Merged

openshift-ci bot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label May 11, 2021

askervin marked this pull request as draft June 4, 2021 06:10

openshift-ci bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jun 4, 2021

askervin force-pushed the 5CH_blockio branch from b13bdc6 to 5757106 Compare June 7, 2021 08:24

openshift-ci bot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jun 7, 2021

askervin marked this pull request as ready for review June 7, 2021 08:25

openshift-ci bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jun 7, 2021

askervin mentioned this pull request Jun 9, 2021

libcontainer/cgroups/fs/blkio: support BFQ weight[_device] opencontainers/runc#3010

Merged

openshift-ci bot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jun 9, 2021

askervin force-pushed the 5CH_blockio branch from 5757106 to 9a7950e Compare June 21, 2021 05:42

openshift-ci bot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jun 21, 2021

sonatype-lift bot reviewed Aug 20, 2021

View reviewed changes

askervin added 3 commits August 23, 2021 15:38

blockio: enable setting blockio class configuration file

d7444c8

Signed-off-by: Antti Kervinen <[email protected]>

blockio: handle class configuration file if set

7b3f68f

- Load class configuration file if available. - Activate configuration in the goresctrl library. - Depend on goresctrl blockio. Signed-off-by: Antti Kervinen <[email protected]>

askervin force-pushed the 5CH_blockio branch from 082bf8a to f74d274 Compare August 23, 2021 12:38

openshift-ci bot assigned haircommander Aug 23, 2021

openshift-ci bot added the lgtm Indicates that a PR is ready to be merged. label Aug 23, 2021

openshift-ci bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Aug 24, 2021

openshift-merge-robot merged commit 5d5a754 into cri-o:master Aug 24, 2021

askervin mentioned this pull request Jan 12, 2022

iops limit for pod/pvc/pv kubernetes/kubernetes#92287

Closed

pacoxu mentioned this pull request Feb 22, 2022

Add blkio for iops limit support in the future kubernetes/system-validators#29

Merged

pacoxu mentioned this pull request May 5, 2022

Add block io in the container stats of CRI #108575 kubernetes/kubernetes#109660

Closed

pacoxu mentioned this pull request Sep 16, 2022

iops limit for pod/pvc/pv kubernetes/enhancements#3520

Closed

4 tasks

llamerada-jp mentioned this pull request Sep 20, 2022

feat: add disk io throttling support topolvm/topolvm#564

Closed

m-yosefpor mentioned this pull request Dec 8, 2022

librbd QoS settings for RBD based PVs ceph/ceph-csi#521

Closed

Support for cgroups blockio #4873

Support for cgroups blockio #4873

Uh oh!

Conversation

askervin commented May 7, 2021

What type of PR is this?

What this PR does / why we need it:

Which issue(s) this PR fixes:

Special notes for your reviewer:

Does this PR introduce a user-facing change?

Uh oh!

openshift-ci-robot commented May 7, 2021

Uh oh!

openshift-ci-robot commented May 7, 2021

Uh oh!

haircommander commented May 7, 2021

Uh oh!

TomSweeneyRedHat commented May 7, 2021

Uh oh!

askervin commented May 10, 2021

Uh oh!

askervin commented Jun 4, 2021

Uh oh!

sonatype-lift bot Aug 20, 2021

Choose a reason for hiding this comment

pkg:golang/golang.org/x/[email protected]

[CVE-2018-17143] Improper Input Validation

[CVE-2018-17848] Data Handling

[CVE-2018-17847] Improper Input Validation

[CVE-2018-17142] Improper Input Validation

[CVE-2018-17846] Resource Management Errors

Uh oh!

sonatype-lift bot Aug 20, 2021

Choose a reason for hiding this comment

pkg:golang/golang.org/x/[email protected]

[CVE-2018-17143] Improper Input Validation

[CVE-2018-17848] Data Handling

[CVE-2018-17847] Improper Input Validation

[CVE-2018-17142] Improper Input Validation

[CVE-2018-17846] Resource Management Errors

Uh oh!

sonatype-lift bot Aug 20, 2021

Choose a reason for hiding this comment

pkg:golang/golang.org/x/[email protected]

[CVE-2019-11840] Use of Insufficiently Random Values

Uh oh!

sonatype-lift bot Aug 20, 2021

Choose a reason for hiding this comment

pkg:golang/golang.org/x/[email protected]

[CVE-2019-11840] Use of Insufficiently Random Values

Uh oh!

askervin commented Aug 20, 2021

Uh oh!

askervin commented Aug 23, 2021

Uh oh!

haircommander commented Aug 23, 2021

Uh oh!

haircommander commented Aug 23, 2021

Uh oh!

haircommander commented Aug 24, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

openshift-ci bot commented Aug 24, 2021

Uh oh!

haircommander commented Aug 24, 2021

Uh oh!

openshift-ci bot commented Aug 24, 2021

Uh oh!

openshift-ci bot commented Aug 24, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

haircommander commented Aug 24, 2021

Uh oh!

haircommander commented Aug 24, 2021

Uh oh!

openshift-ci bot commented Aug 24, 2021

Uh oh!

Reviewers

Assignees

Labels

haircommander commented Aug 24, 2021 •

edited

Loading

openshift-ci bot commented Aug 24, 2021 •

edited

Loading