kubelet & kube-proxy: ignore sysctl errors and rlimit errors when running in UserNS (for rootless) #92863

AkihiroSuda · 2020-07-07T12:51:40Z

What type of PR is this?

/kind feature

What this PR does / why we need it:
Ignore sysctl errors and rlimit errors when running in UserNS.

These changes are required for Rootless mode. See KEP for the further information.

Special notes for your reviewer:

How to test rootless mode with this PR:

Run make kubelet kube-proxy
Git clone https://github.com/rootless-containers/usernetes and run make
Replace the kubelet and kube-proxy binaries inside the bin directory of the Usernetes to test the codes in this PR. To just try rootless mode, no need to replace the binaries.
Configure the host to use cgroup v2: https://github.com/rootless-containers/usernetes#enable-cgroup-v2
Run the Usernetes installer: ./install.sh
export KUBECONFIG="$HOME/.config/usernetes/master/admin-localhost.kubeconfig"
Run kubectl

To run the smoke test suite, run ./hack/smoketest-binaries.sh instead of ./install.sh ....

After this PR gets merged, I'm planning to add kind tests with Rootless Docker (or Podman).

Does this PR introduce a user-facing change?:

Adds feature gate `KubeletInUserNamespace` which enables support for running kubelet in a user namespace.

The user namespace has to be created before running kubelet.
All the node components such as CRI need to be running in the same user namespace.

When the feature gate is enabled, kubelet ignores errors that happens during setting the following sysctl values: `vm.overcommit_memory`, `vm.panic_on_oom`, `kernel.panic`, `kernel.panic_on_oops`, `kernel.keys.root_maxkeys`, `kernel.keys.root_maxbytes`. (These sysctl values for the host, not for the containers)

kubelet also ignores an error during opening `/dev/kmsg`.
This feature gate also allows kube-proxy to ignore an error during setting `RLIMIT_NOFILE`.

This feature gate is especially useful for running Kubernetes inside Rootless Docker/Podman with `kind` or `minikube`.

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

- [KEP]: https://github.com/kubernetes/enhancements/tree/master/keps/sig-node/2033-kubelet-in-userns-aka-rootless

The patches are from Usernetes project: https://github.com/rootless-containers/usernetes/tree/master/src/patches/kubernetes

KEP: https://github.com/kubernetes/enhancements/tree/master/keps/sig-node/2033-kubelet-in-userns-aka-rootless
Tracking issue: kubernetes/enhancements#2033
Documentation PR: kubernetes/website#28827

kubelet changes

Ignore errors that happens during setting the following sysctl values:

vm.overcommit_memory
vm.panic_on_oom
kernel.panic
kernel.panic_on_oops
kernel.keys.root_maxkeys
kernel.keys.root_maxbytes

Also ignores an error during opening /dev/kmsg.

Gated by KubeletInUserNamespace feature gate.

kube-proxy changes

Ignore an error during setting RLIMIT_NOFILE

k8s-ci-robot · 2020-07-07T12:51:49Z

Hi @AkihiroSuda. Thanks for your PR.

I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

AkihiroSuda · 2020-07-07T12:52:13Z

cc @giuseppe @cyphar @rhatdan @KentaTada FYI

hakman · 2020-07-07T13:25:14Z

/ok-to-test

fejta-bot · 2020-07-07T13:51:47Z

This PR may require API review.

If so, when the changes are ready, complete the pre-review checklist and request an API review.

Status of requested reviews is tracked in the API Review project.

pkg/kubelet/cm/cgroup_manager_linux.go

AkihiroSuda · 2020-07-08T03:00:06Z

/retest

AkihiroSuda · 2020-07-08T03:11:22Z

This PR may require API review.

If so, when the changes are ready, complete the pre-review checklist and request an API review.

Status of requested reviews is tracked in the API Review project.

This PR doesn't change any API other than KubeletConfiguration, which seems out of the scope of "API change"

AkihiroSuda · 2020-07-08T04:11:37Z

CI failure doesn't seem related

pkg/kubelet/cm/cgroup_manager_linux.go

test/e2e_node/node_container_manager_test.go

AkihiroSuda · 2021-07-05T07:45:34Z

CI failure is unrelated

I0705 07:27:49.411] NAME                 STATUS      MESSAGE                                                                                       ERROR
I0705 07:27:49.412] scheduler            Unhealthy   Get "http://127.0.0.1:10251/healthz": dial tcp 127.0.0.1:10251: connect: connection refused   
I0705 07:27:49.412] etcd-0               Healthy     {"health":"true","reason":""}                                                                 
I0705 07:27:49.412] etcd-1               Healthy     {"health":"true","reason":""}                                                                 
I0705 07:27:49.413] controller-manager   Healthy     ok                                                                                            
I0705 07:27:49.415] Validation returned one or more failed components. Cluster is probably broken.

https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/92863/pull-kubernetes-e2e-gce-alpha-features/1411942353094053888/

/retest

AkihiroSuda · 2021-07-05T09:58:05Z

/retest

aojea · 2021-07-05T14:52:56Z

/approve
sig-network changes

neolit123 · 2021-07-05T15:48:20Z

Added support for rootless mode

i recommend amending the release note to be more specific of what the change is about and what components are being modified.

AkihiroSuda · 2021-07-06T04:40:27Z

@neolit123

Updated.

Adds feature gate `KubeletInUserNamespace` which enables support for running kubelet in a user namespace.

The user namespace has to be created before running kubelet.
All the node components such as CRI need to be running in the same user namespace.

When the feature gate is enabled, kubelet ignores errors that happens during setting the following sysctl values: `vm.overcommit_memory`, `vm.panic_on_oom`, `kernel.panic`, `kernel.panic_on_oops`, `kernel.keys.root_maxkeys`, `kernel.keys.root_maxbytes`. (These sysctl values for the host, not for the containers)

kubelet also ignores an error during opening `/dev/kmsg`.
This feature gate also allows kube-proxy to ignore an error during setting `RLIMIT_NOFILE`.

This feature gate is especially useful for running Kubernetes inside Rootless Docker/Podman with `kind` or `minikube`.

/retest

AkihiroSuda · 2021-07-06T12:24:54Z

/retest

Enables support for running kubelet in a user namespace. The user namespace has to be created before running kubelet. All the node components such as CRI need to be running in the same user namespace. See kubernetes/enhancements PR 1371 (merged) and issue 2033. Signed-off-by: Akihiro Suda <[email protected]>

Errors during setting the following sysctl values are ignored: - vm.overcommit_memory - vm.panic_on_oom - kernel.panic - kernel.panic_on_oops - kernel.keys.root_maxkeys - kernel.keys.root_maxbytes Signed-off-by: Akihiro Suda <[email protected]>

Ignore an error during setting RLIMIT_NOFILE. Signed-off-by: Akihiro Suda <[email protected]>

oomwatcher.NewWatcher returns "open /dev/kmsg: operation not permitted" error, when running with sysctl value `kernel.dmesg_restrict=1`. The error is negligible for KubeletInUserNamespace. Signed-off-by: Akihiro Suda <[email protected]>

AkihiroSuda · 2021-07-07T05:27:41Z

Rebased

AkihiroSuda · 2021-07-07T05:33:28Z

@kubernetes/sig-node-pr-reviews Can we get this reviewed before v1.22 Code Freeze on July 8th? 🙏

k8s-ci-robot · 2021-07-07T06:00:08Z

@AkihiroSuda: The following test failed, say /retest to rerun all failed tests:

Test name	Commit	Details	Rerun command
pull-kubernetes-e2e-gce-alpha-features	`26e83ac`	link	`/test pull-kubernetes-e2e-gce-alpha-features`

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

dims · 2021-07-07T11:01:04Z

/skip

dims · 2021-07-07T11:04:28Z

/hold cancel

the KEP is ready for 1.22

ehashman

/lgtm

k8s-ci-robot · 2021-07-07T23:26:19Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: AkihiroSuda, aojea, dims, kolyshkin, mrunalp

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

AkihiroSuda · 2021-07-08T01:13:13Z

Thanks all for reviewing and approval!

Opened a doc PR: kubernetes/website#28827

k8s-ci-robot added the needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. label Jul 7, 2020

AkihiroSuda mentioned this pull request Jul 7, 2020

sig-node: Kubelet-in-UserNS, aka Rootless mode kubernetes/enhancements#1371

Merged

k8s-ci-robot requested review from nicksardo and Random-Liu July 7, 2020 12:53

k8s-ci-robot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Jul 7, 2020

tedyu reviewed Jul 8, 2020

View reviewed changes

pkg/kubelet/cm/cgroup_manager_linux.go Outdated Show resolved Hide resolved

AkihiroSuda force-pushed the rootless-pr branch from c3a39db to 2f21248 Compare July 8, 2020 03:56

giuseppe reviewed Jul 8, 2020

View reviewed changes

pkg/kubelet/cm/cgroup_manager_linux.go Outdated Show resolved Hide resolved

SergeyKanzhelev reviewed Jul 9, 2020

View reviewed changes

test/e2e_node/node_container_manager_test.go Outdated Show resolved Hide resolved

AkihiroSuda force-pushed the rootless-pr branch from 15159c5 to 4d24b0d Compare July 5, 2021 06:57

AkihiroSuda force-pushed the rootless-pr branch from 4d24b0d to e80e4f2 Compare July 6, 2021 11:12

k8s-ci-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jul 6, 2021

AkihiroSuda added 4 commits July 7, 2021 14:22

kube-proxy: allow running in userns

192790c

Ignore an error during setting RLIMIT_NOFILE. Signed-off-by: Akihiro Suda <[email protected]>

AkihiroSuda force-pushed the rootless-pr branch from e80e4f2 to 26e83ac Compare July 7, 2021 05:24

k8s-ci-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jul 7, 2021

AkihiroSuda mentioned this pull request Jul 7, 2021

Add KubeletInUserNamespace feature gate kubernetes/website#28827

Merged

k8s-ci-robot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jul 7, 2021

ehashman reviewed Jul 7, 2021

View reviewed changes

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Jul 7, 2021

mrunalp approved these changes Jul 7, 2021

View reviewed changes

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jul 7, 2021

k8s-ci-robot merged commit ebbe63f into kubernetes:master Jul 8, 2021

mengjiao-liu mentioned this pull request Jul 15, 2021

Batch update feature gates for 1.22 kubernetes/website#28849

Merged

kubelet & kube-proxy: ignore sysctl errors and rlimit errors when running in UserNS (for rootless) #92863

kubelet & kube-proxy: ignore sysctl errors and rlimit errors when running in UserNS (for rootless) #92863

Uh oh!

Conversation

AkihiroSuda commented Jul 7, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

kubelet changes

kube-proxy changes

Uh oh!

k8s-ci-robot commented Jul 7, 2020

Uh oh!

AkihiroSuda commented Jul 7, 2020

Uh oh!

hakman commented Jul 7, 2020

Uh oh!

fejta-bot commented Jul 7, 2020

Uh oh!

Uh oh!

AkihiroSuda commented Jul 8, 2020

Uh oh!

AkihiroSuda commented Jul 8, 2020

Uh oh!

AkihiroSuda commented Jul 8, 2020

Uh oh!

Uh oh!

Uh oh!

AkihiroSuda commented Jul 5, 2021

Uh oh!

AkihiroSuda commented Jul 5, 2021

Uh oh!

aojea commented Jul 5, 2021

Uh oh!

neolit123 commented Jul 5, 2021

Uh oh!

AkihiroSuda commented Jul 6, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

AkihiroSuda commented Jul 6, 2021

Uh oh!

AkihiroSuda commented Jul 7, 2021

Uh oh!

AkihiroSuda commented Jul 7, 2021

Uh oh!

k8s-ci-robot commented Jul 7, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dims commented Jul 7, 2021

Uh oh!

dims commented Jul 7, 2021

Uh oh!

ehashman left a comment

Choose a reason for hiding this comment

Uh oh!

k8s-ci-robot commented Jul 7, 2021

Uh oh!

AkihiroSuda commented Jul 8, 2021

Uh oh!

Uh oh!

AkihiroSuda commented Jul 7, 2020 •

edited

Loading

AkihiroSuda commented Jul 6, 2021 •

edited

Loading

k8s-ci-robot commented Jul 7, 2021 •

edited

Loading